\documentclass{article}

\usepackage{times}
\usepackage{graphicx} % more modern
\usepackage{natbib}
\usepackage{algorithm, algorithmic}
\usepackage{hyperref}
\usepackage{amssymb, mathtools}
\usepackage{xcolor,colortbl}
\newcommand{\theHalgorithm}{\arabic{algorithm}}
\usepackage{subcaption} % for "subfigure" environment
\usepackage{geometry}

\usepackage[accepted]{icml2017}

% The \icmltitle you define below is probably too long as a header.
% Therefore, a short form for the running title is supplied here:
\icmltitlerunning{Globally Induced Forest}

% ============================== CONFIG =================================== %
\graphicspath{{./images/}}
\captionsetup[figure]{size=footnotesize}
% ============================== COLORS =================================== %
\definecolor{orange}{HTML}{FFA500}
\definecolor{dodgerblue}{HTML}{1E90FF}
\definecolor{deepgreen}{HTML}{0AF191}
\definecolor{purplish}{HTML}{E21173}

% ============================== COMMANDS =================================== %
\DeclareMathOperator*{\argmin}{arg\,min}
\DeclareMathOperator*{\argmax}{arg\,max}

\newcommand{\best}{\cellcolor{lightgray}}
\newcommand{\bestA}{\cellcolor{orange}}
\newcommand{\bestB}{\cellcolor{dodgerblue}}

\begin{document}
\twocolumn[
\icmltitle{Globally Induced Forest: A Prepruning Compression
Scheme\\Supplementary material}

% It is OKAY to include author information, even for blind
% submissions: the style file will automatically remove it for you
% unless you've provided the [accepted] option to the icml2017
% package.

% list of affiliations. the first argument should be a (short)
% identifier you will use later to specify author affiliations
% Academic affiliations should list Department, University, City, Region,
%Country
% Industry affiliations should list Company, City, Region, Country

% you can specify symbols, otherwise they are numbered in order
% ideally, you should not use this facility. affiliations will be numbered
% in order of appearance and this is the preferred way.
\icmlsetsymbol{equal}{*}

\begin{icmlauthorlist}
\icmlauthor{Jean-Michel Begon}{ulg}
\icmlauthor{Arnaud Joly}{ulg}
\icmlauthor{Pierre Geurts}{ulg}
\end{icmlauthorlist}

\icmlaffiliation{ulg}{Department of Electrical Engineering and Computer Science
University of Li\`ege, Li\`ege, Belgium}

\icmlcorrespondingauthor{Jean-Michel Begon}{jm.begon@ulg.ac.be}
%\icmlcorrespondingauthor{Arnaud Joly}{a.joly@ulg.ac.be}
\icmlcorrespondingauthor{Pierre Geurts}{p.geurts@ulg.ac.be}

% You may provide any keywords that you
% find helpful for describing your paper; these are used to populate
% the "keywords" metadata in the PDF but will not be shown in the document

\icmlkeywords{Decision tree, Random forest, Extremely randomized trees,
pruning, node budget, memory constraint, compression, growing algorithm, greedy
selection}

\vskip 0.3in
]

\printAffiliationsAndNotice{}

\section{GIF algorithm}
Figure \ref{fig:gif-algo} illustrates visually the inner loop of the GIF
building algorithm: a subset of the candidates nodes is chosen uniformely at
random. The contribution of each node is evaluated and the one which reduces
the error the most is added to the model. Its children are then built and added
to the candidate list.

\begin{figure*}[ht]
  \begin{subfigure}[b]{0.5\linewidth}
    \centering
    \includegraphics[height=0.25\textheight]{gif_algo1}
    \caption{Current forest at time $t$}
    \label{fig:gif-algo1}
  \end{subfigure}
  \hspace{\fill}  %% maximize space between adjacent subfigures
  \begin{subfigure}[b]{0.5\linewidth}
    \centering
    \includegraphics[height=0.25\textheight]{gif_algo2}
    \caption{A subset of candidates $C_t$  is drawn uniformely at random from
    the set of candidates $C$ (step 8)}
    \label{fig:gif-algo2}
  \end{subfigure}

  \vspace{4ex}  %% extra vertical space
  \begin{subfigure}[b]{0.5\linewidth}
    \centering
    \includegraphics[height=0.25\textheight]{gif_algo3}
    \caption{The error reduction is computed for all candidates of $C_t$ (step
    9)}
    \label{fig:gif-algo3}
  \end{subfigure}
  \hspace{\fill}
  \begin{subfigure}[b]{0.5\linewidth}
    \centering
    \includegraphics[height=0.25\textheight]{gif_algo4}
    \caption{The best node (highest error reduction) is selected (step 9)}
    \label{fig:gif-algo4}
  \end{subfigure}

    \vspace{4ex}
  \begin{subfigure}[b]{0.5\linewidth}
    \centering
    \includegraphics[height=0.25\textheight]{gif_algo5}
    \caption{The chosen node is introduced in the model (step 10)}
    \label{fig:gif-algo5}
  \end{subfigure}
  \hspace{\fill}
  \begin{subfigure}[b]{0.5\linewidth}
    \centering
    \includegraphics[height=0.25\textheight]{gif_algo6}
    \caption{The children of the chosen node are computed (step 11) and added
    to the candidate list (step 12)}
    \label{fig:gif-algo6}
  \end{subfigure}

\caption{Illustration of the GIF regression building algorithm ($T=3$, $CW=3$)}
\label{fig:gif-algo}
\end{figure*}

\section{Optimization problem}

We are building an additive model by inserting progressively nodes in the
forest.
At time $t$, we are trying to find the best node $j^{(t)}$ from the candidate
list $C_t$ and its associated optimal weight $w_j^{(t)}$:
\begin{align}
j^{(t)},w_j^{(t)} =\argmin_{j\in C_t, w\in \mathbb{R}^K} \sum_{i=1}^{N} L
\left(y_i,
\hat{y}^{(t-1)}(x_i) + w z_j(x_i) \right)
\end{align}
where $(x_i, y_i)_{i=1}^N$ is the learning sample, $\hat{y}^{(t-1)}()$ is the
model at time $t-1$, $z_j()$ is the node indicator functions, meaning that it
is $1$ if its argument reaches node $j$ and $0$ otherwise.

This problem is solved in two steps. First a node $j$ is selected from $C_t$
and the corresponding optimal weight, alongside the error reduction, are
computed. This is repeated for all nodes and the one achieving the best
improvement is selected.

\paragraph{Regression}
For regression, we used the L2-norm:
\begin{align}
w_j^{(t)} = \argmin_{w\in \mathbb{R}} \sum_{i=1}^{N} L \left(y_i,
\hat{y}^{(t-1)}(x_i) + w z_j(x_i) \right)^2
\end{align}
and the solution is given by
\begin{align}\label{eq:L2Solution}
w_j^{(t)} = \frac{1}{|Z_j|} \sum_{i \in Z_j} r_i^{(t-1)}
\end{align}
where $r_i^{(t-1)} = y_i - \hat{y}^{(t-1)}(x_i)$ is the residual at time $t-1$
for the $i$th training instance and $Z_j = \{1 \leq i \leq N | z_j(x_i)=1\}$ is
the subset of instances reaching node $j$.

\paragraph{Classification}
For classification we used the multi-exponential loss
\cite{zhu2009multiadaboost}. First, we need to encode the labels so that
\begin{align}\label{eq:MEencode}
y_i^{(k)} = \begin{cases}
1, &\text{ if the class of } y_i \text{ is } k \\
-\frac{1}{K-1}, &\text{otherwise}
\end{cases}
\end{align}
where $K$ is the number of classes. Notice that $\sum_{k=1}^{K} y_i^{(k)} = 0$.
The optimization then becomes
\begin{align}\label{eq:MEmin}
w_j^{(t)} &=  \argmin_{w \in \mathbb{R}^K} \sum_{i=1}^N \exp
\left(\frac{-1}{K} y_i^T \left(\hat{y}^{(t-1)}(x_i) + w z_j(x_i) \right)\right)
\\
&= \argmin_{w \in \mathbb{R}^K} F_j^{(t-1)}(w)
\end{align}
Solving for $\nabla F_j^{(t-1)}(w) = 0$ yields
\begin{align}\label{eq:MErawSol}
\alpha_j^{(t-1, k)}\phi^{(k)}(w) = \frac{1}{K} \sum_{l=1}^{K} \alpha_j^{(t-1,
l)}\phi^{(l)}(w)
\end{align}
for $1 \leq k\leq K$, where
\begin{align}
\alpha_j^{(t-1, k)} &\triangleq \sum_{i \in Z_j^{(k)}} \exp \left( -
\mu_i^{(t-1)} \right) \\
\mu_i^{(t-1)} &\triangleq \frac{1}{K} \sum_{k=1}^{K} y_i \hat{y}^{(t-1,
k)}(x_i) \\
\phi^{(k)}(w) &\triangleq \exp \left( - \frac{1}{K} \psi^{(k)}(w) \right) \\
\psi^{(k)}(w) &\triangleq -w^{(k)} + \frac{1}{K-1} \sum_{l=1, l\neq k}^{K}
w^{(l)}
\end{align}
where $Z_j^{(k)} = \{1 \leq i \leq N | z_{i,j} = 1 \wedge y_i^{(k)} = 1 \}$ is
the subset of learning instances of class $k$ reaching node $j$. In words,
$\mu_i^{(t-1)}$ is the hyper-margin of instance $i$ at time $t-1$ and
$\alpha_j^{(t-1, k)}$ is the class error of label $k$ for node $j$ at time
$t-1$.

Equation \ref{eq:MErawSol} is equivalent to
\begin{align}\label{eq:MEequation}
\alpha_j^{(t-1, k)}\phi^{(k)}(w) &= \alpha_j^{(t-1, l)}\phi^{(l)}(w) \quad 1
\leq k,l \leq K
\end{align}

In keeping with the output representation (Equation \ref{eq:MEencode}), we can
impose a zero-sum constraint on the prediction to get a unique solution for the
$k$th component of $w_j^{(t)}$. If it is imposed at each stage, it means that
\begin{align}\label{eq:MEzeroSum}
\sum_{k=1}^{K} \hat{y}^{(t-1, k)} = \sum_{k=1}^{K}
\hat{y}^{(t, k)} = 0 = \sum_{k=1}^{K} w^{(k)}
\end{align}
and this is not impacted by the learning rate.

The corresponding solution is
\begin{align}
\phi^{(k)}(w) &= \exp \left(-\frac{1}{K-1} w^{(k)}\right)\\
\label{eq:MEClsErrZS}
\alpha_j^{(t-1, k)} &= \sum_{i \in Z_j^{(k)}} \exp \left( -\frac{1}{K-1}
\hat{y}^{(t-1, k)}(x_i) \right) \\ \label{eq:MEsolution}
w_j^{(t,k)} &= \frac{K-1}{K}  \sum_{l=1}^{K} \log \frac{\alpha_j^{(t-1,
k)}}{\alpha_j^{(t-1, l)}}
\end{align}

\section{Equivalence of GIF and the underlying tree}\label{app:Equiv}
In the case of a single tree ($T=1$) and a unit learning rate ($\lambda=1$),
both the square loss in regression and the multiexponential loss in
classification produce the same predictions as the underlying tree.
This is due to the fact that, when examining the weight to give to node $j$ at
time $t$, the prediction of time $t-1$ relates to the parent node $\pi_j$ of
$j$. It is thus independent of $t$ and is also the same for all instances
reaching that node.

Consequently, we will adopt the following slight change in notation:
\begin{align}
\hat{y}_j = \hat{y}_{(\pi_j)} + w_j
\end{align}
meaning that the prediction associated to any object reaching node $j$ is the
weight of $j$ plus the prediction associated to its parent $\pi_j$. With
$\hat{y}_{(\pi_1)} = 0$, the prediction of the root's pseudo-parent.

\subsection{Regression}
In regression, the tree prediction $Tr_j$ of any leaf $j$ is the average of the
learning set's outputs reaching that node: $Tr_j = \frac{1}{|Z_j|}\sum_{i \in
Z_j} y_i$. We need to show that the GIF prediction is:
\begin{align}\label{eq:EquivL2Cond}
\hat{y}_{j} = \frac{1}{|Z_j|}\sum_{i \in Z_j} y_i
\end{align}

The prediction of node $j$ is
\begin{align}\label{eq:EquivL2Solution}
\hat{y}_j &= \hat{y}_{\pi_j} + w_j \\
&= \hat{y}_{\pi_j} +  \frac{1}{|Z_j|} \sum_{i \in Z_j} \left(y_i -
\hat{y}_{\pi_j}\right) \\
&= \hat{y}_{\pi_j} + \frac{1}{|Z_j|} \sum_{i \in Z_j} \left( y_i \right) -
\hat{y}_{\pi_j} \\
&= \frac{1}{|Z_j|} \sum_{i \in Z_j}  y_i
\end{align}

The first step is how the additive model is built. The second is the optimal
weight value of node $j$ derived in Equation \ref{eq:L2Solution}, the third
step is due to the fact that the prediction at $\pi_j$ is constant since there
is only one tree.

\subsection{Classification}
In order to have the same prediction as the underlying tree, we must
demonstrate that the probability of being in class $l$ associated to node $j$
will be $\frac{|Z_j^{(l)}|}{|Z_j|}$.

Under the zero-sum constraint, we have
\begin{align}
\exp \left(  \frac{1}{K-1} w_j^{(l)}\right) &= \frac{1}{c_j}
\alpha_{\pi_j}^{(l)} \\
&=  \frac{1}{c_j} \sum_{i \in Z_j^{(l)}} \exp \left(-\frac{1}{K-1}
\hat{y}_{\pi_j}^{(l)}\right)\\
&= \frac{1}{c_j} |Z_j^{(l)}| \exp \left(-\frac{1}{K-1}
\hat{y}_{\pi_j}^{(l)}\right) \\
\exp \left(\frac{1}{K-1} \hat{y}_j^{(l)} \right) &= \exp \left(\frac{1}{K-1}
\hat{y}_{\pi_j}^{(l)} \right) \exp \left(\frac{1}{K-1} w_j^{(l)}\right) \\
&= \frac{1}{c_j} |Z_j^{(l)}| \\
P_j(l) &= \frac{\exp \left(\frac{1}{K-1} \hat{y}_j^{(l)}
\right)}{\sum_{k=1}^K\exp \left(\frac{1}{K-1} \hat{y}_j^{(k)} \right)} =
\frac{|Z_j^{(l)}|}{|Z_j|}
\end{align}
where $c_j = \left(\prod_{k=1}^K \alpha_j^{(k)}\right)^{\frac{1}{K}}$ is a
constant. The first equality is a consequence of the value of $w_j^{(l)}$
(Equation \ref{eq:MEsolution}). The second is a due to the definition of
$\alpha_j^{(l)}$ (Equation \ref{eq:MEClsErrZS}). The third is a consequence of
having a single tree: the prediction of the parent is the same for all
instances.

Notice that, in both regression and classification, the equivalence also holds
for an internal node: the prediction is the one the tree would have yielded if
that node had been a leaf.

\section{Datasets}

Table \ref{tab:datasets} sums up the main characteristics of the datasets we
used. Abalone, CT slice, California data housig (Cadata), Musk2, Vowel and
Letter come from the UCI Machine Learning Repository \cite{uci}. Ringnorm,
Twonorm and Waveform are described in \cite{breiman1998arcing}. Hwang F5 comes
from the DELVE repository \footnote{http://www.cs.utoronto.ca/∼delve}.
The noise parameter of the Friedman1 dataset \cite{friedman11991} has
been set to $1$. Hastie is described in \cite{hastie2009}. Out of the 500
features of Madelon \cite{guyon2004madelon}, 20 are informative and 50 are
redundant; the others are noise.
Mnist8vs9 is the Mnist dataset \cite{lecun1998mnist} of which only the $8$ and
$9$ digits have been kept. Binary versions of the Mnist, Letter and Vowel
datasets have been created as well by grouping the first half and second half
classes together.

\begin{table}[th]
\caption{Characteristics of the datasets. $N$ is the learning sample size, TS
stands for testing set, and $p$ is the number of features.}
\label{tab:datasets}
\begin{center}
\begin{footnotesize}
\begin{sc}
\begin{tabular}{l|cccc}
\hline
Dataset & $N$ & $|TS|$ & $p$ & \# classes\\
\hline
Friedman1 & 300 & 2000 & 10 & - \\
Abalone & 2506 & 1671 & 10 & - \\
CT slice & 2000 & 51500 & 385 & - \\
Hwang F5 & 2000 & 11600 & 2 & - \\
Cadata & 12384 & 8256 & 8 & - \\
Ringnorm & 300 & 7100 & 20 & 2 \\
Twonorm & 300 & 7100 & 10 & 2 \\
Hastie & 2000 & 10000 & 10 & 2 \\
Musk2 & 2000 & 4598 & 166 & 2 \\
Madelon & 2200 & 2200 & 500 & 2 \\
Mnist8vs9 & 11800 & 1983 & 784 & 2 \\
Waveform & 3500 & 1500 & 40 & 3 \\
Vowel & 495 & 495 & 10 & 11 \\
Mnist & 50000 & 10000 & 784 & 10 \\
Letter & 16000 & 4000 & 8 & 26 \\
\hline
\end{tabular}
\end{sc}
\end{footnotesize}
\end{center}
\vskip -0.2in
\end{table}

\section{Comparison with local baseline algorithms}
We have tested three deepening algorithm for decision forest relying on
non-global metrics, meaning that the choice of the best candidate is not made
according to how well the forest, as a whole, performs. These algorithms share
that the final model is exactly a sub-forest of the un-pruned forest: contrary
to GIF, no internal weights are fitted and the predictions of at the leaves are
the usual tree predictions.

\paragraph{Breadth first deepening}
This variant consist in adding the nodes level after level, from left to right,
producing a heaped forest. As a consequence, all trees have the same (order of)
height, implying that the forest can be quite wide but usually shallow.

\paragraph{Random deepening}
This variant consist in first choosing a tree and then choosing one of its
leaves to transform to a decision nodes. Both choices are made uniformly at
random so that the trees are expected to have approximately the same number of
nodes. The depth, however, might vary significantly.

\paragraph{Best first deepening}
This variant consist in choosing, among all leaves which could be turned into a
internal node, the one which reduces its local impurity the most.
Let $N_c$, $N_l$ and $N_r$ be the number of instances
reaching the candidate node, candidate left child and candidate right child
respectively.
Let also $I_c$, $I_l$ and $I_r$ be the impurity (gini index in classification,
variance in regression) of the instances reaching the candidate node, candidate
left child and candidate right child respectively.
Then, for $N$ learning instances, the local impurity reduction is
defined as:

\begin{align}
\Delta I_c \triangleq \frac{N_c}{N} \left[ I_c - \left( \frac{N_l}{N_c} I_l +
\frac{N_r}{N_c} I_r \right)\right]
\end{align}

Since the fraction of learning instances reaching the candidate is accounted
for in the reduction of impurity, this approach will naturally favor higher
nodes in the trees.

\begin{table*}[t]
\caption{Average mean square error for local baselines at $1\%$ and $10\%$
budgets  ($T=1000$, $m=p$).}
\vskip -0em
\label{tab:baseline-reg}
\begin{center}
\begin{footnotesize}
\begin{sc}
\begin{tabular}{l|ccc|ccc}
\hline
Dataset & Breadth First$_{10\%}$ & Random$_{10\%}$ & Best First$_{10\%}$ &
Breadth First$_{1\%}$ & Random$_{1\%}$ & Best First$_{1\%}$ \\
\hline
Friedman1 & 6.02 $\pm$ 0.28 & 6.80 $\pm$ 0.34 & 15.00 $\pm$ 0.39 & 11.73 $\pm$
0.46 & 12.52 $\pm$ 0.47 & 15.29 $\pm$ 0.42  \\
Abalone & \best 4.72 $\pm$ 0.23 & 4.77 $\pm$ 0.23 & 6.82 $\pm$ 0.33 & 5.42
$\pm$ 0.27 & 5.55 $\pm$ 0.27 & 6.82 $\pm$ 0.33 \\
CT slice &  30.39 $\pm$ 1.90 & 36.19 $\pm$ 1.84 & 310.87 $\pm$ 4.79 & 82.19
$\pm$ 2.41 & 97.24 $\pm$ 1.90 & 313.84 $\pm$ 4.64  \\
Hwang F5 \hfill {\tiny $\times 10^{-2}$} & \best 6.73 $\pm$ 0.07 & 6.83 $\pm$
0.06 & 56.57 $\pm$ 6.03 & 8.52 $\pm$ 0.24 & 13.17 $\pm$ 0.44 & 56.60 $\pm$ 6.07
\\
Cadata \hfill {\tiny $\times 10^{-2}$} & 29.24 $\pm$ 0.73 & 31.08 $\pm$ 0.74 &
75.23 $\pm$ 0.95  & 43.40 $\pm$ 1.18 & 47.47 $\pm$ 1.02 & 75.48 $\pm$ 0.95 \\
\hline
\end{tabular}
\end{sc}
\end{footnotesize}
\end{center}
\vskip -0.2in
\end{table*}

\begin{table*}[t]
\caption{Error rate ($\%$) for local baselines at $1\%$ and $10\%$ budgets
($T=1000$, $m=\sqrt{p}$). The six first datasets are binary classification. The
last three are multiclass. The three in the middle are their binary versions.}
\vskip -1em
\label{tab:baseline-cls}
\begin{center}
\begin{footnotesize}
\begin{sc}
\begin{tabular}{l|ccc|ccc}
\hline
Dataset & Breadth First$_{10\%}$ & Random$_{10\%}$ & Best First$_{10\%}$ &
Breadth First$_{1\%}$ & Random$_{1\%}$ & Best First$_{1\%}$ \\
\hline
Ringnorm & 4.25 $\pm$ 1.24 & 4.08 $\pm$ 1.12 & 8.38 $\pm$ 6.94 &  8.94 $\pm$
7.45 & 8.53 $\pm$ 7.04 & 8.94 $\pm$ 7.41 \\
Twonorm & 3.51 $\pm$ 0.26 & 3.53 $\pm$ 0.30 & 5.59 $\pm$ 1.85  & 5.91 $\pm$
3.03 & 6.52 $\pm$ 4.28 & 7.28 $\pm$ 4.34 \\
Hastie & 11.30 $\pm$ 1.20 & 11.18 $\pm$ 1.16 & 21.24 $\pm$ 7.11  & 13.92 $\pm$
2.93 & 14.29 $\pm$ 3.20 & 21.24 $\pm$ 7.12 \\
Musk2 & 7.01 $\pm$ 0.40 & 7.63 $\pm$ 0.43 & 15.42 $\pm$ 0.23 & 15.42 $\pm$ 0.23
& 15.42 $\pm$ 0.23 & 15.42 $\pm$ 0.23 \\
Madelon & 11.68 $\pm$ 0.67 & 11.92 $\pm$ 0.65 & 19.12 $\pm$ 1.94 & 16.26 $\pm$
0.97 & 16.70 $\pm$ 1.07 & 20.14 $\pm$ 2.41 \\
Mnist8vs9 & 2.20 $\pm$ 0.38 & 2.37 $\pm$ 0.39 & 6.17 $\pm$ 0.73  & 4.53 $\pm$
0.48 & 4.84 $\pm$ 0.51 & 6.67 $\pm$ 0.69 \\
\hline
Bin. Vowel &8.99
$\pm$ 1.96 & 8.85 $\pm$ 2.03 & 16.57 $\pm$ 3.02  & 18.73 $\pm$ 3.08 & 19.90
$\pm$ 3.71 & 21.80 $\pm$ 4.38 \\
Bin. Mnist & 4.46 $\pm$ 0.25 & 4.91 $\pm$ 0.27 & 21.71 $\pm$ 0.30  & 10.09
$\pm$ 0.25 & 11.78 $\pm$ 0.32 & 22.50 $\pm$ 0.35\\
Bin. Letter & 5.91 $\pm$ 0.43 & 5.71 $\pm$ 0.40 & 26.16 $\pm$ 0.86  & 17.91
$\pm$ 0.77 & 18.05 $\pm$ 0.78 & 26.19 $\pm$ 0.88 \\
\hline
Waveform & 14.74 $\pm$ 0.63 & 14.83 $\pm$ 0.76 & 20.25 $\pm$ 2.22 & 16.75 $\pm$
1.26 & 17.13 $\pm$ 1.25 & 20.45 $\pm$ 2.21 \\
Vowel & 14.26 $\pm$ 2.41 & 13.21 $\pm$ 2.33 & 41.49 $\pm$ 5.45 & 42.40 $\pm$
4.33 & 40.28 $\pm$ 4.62 & 50.44 $\pm$ 5.81 \\
Mnist & 4.63 $\pm$ 0.27 & 4.96 $\pm$ 0.26 & 28.54 $\pm$ 0.59 & 8.60 $\pm$ 0.35
& 9.76 $\pm$ 0.31 & 29.72 $\pm$ 0.61 \\
Letter & 7.06 $\pm$ 0.29 & 6.39 $\pm$ 0.20 & 36.92 $\pm$ 1.80 & 22.11 $\pm$
0.59 & 20.90 $\pm$ 0.55 & 37.27 $\pm$ 1.78 \\
\hline
\end{tabular}
\end{sc}
\end{footnotesize}
\end{center}
\vskip -0.2in
\end{table*}

\paragraph{Experiment}
We conducted the same experiment as for GIF: the three algorithms were tested
on ten folds with different learning sample/testing sample splits and were
subjected to the $1\%$ and $10\%$ node constraints. We started with a pool of
$T=1000$ roots and no restriction was imposed regarding the depth. All of the
$m=p$ the features were examined in regression and $m=\sqrt{p}$ in
classification, as suggested in \cite{extratrees}. Table \ref{tab:baseline-reg}
holds the average mean square error for the five regression problems and Table
\ref{tab:baseline-cls} holds the average misclassification rate for the
classification problems.

\paragraph{Regression}
The trend is quite clear: both at $1\%$ and $10\%$, the breadth first algorithm
is the best and the best first is (largely) the worst.
There are two instances where the local baselines are able to beat GIF: on
Abalone and Hwang F5 at $10\%$. Interestingly, these are the same cases on
which GIF was beaten by a small forest of Extremely randomized trees. The
$10\%$ Hwang F5 case aside, the local baselines always underperform the smaller
fully-developed forest. Overall, such variants do not seem adequate for
regression.

\paragraph{Classification}
In classification, the breadth first and random baselines tend to perform
similarly, one beating the other on some problems. Once again, the best first
approach seems to be lagging behind on some datasets. At $10\%$, the local
baselines cannot rival with the other methods. Only on Waveform are they able
to reach the other performances---a setting where all methods seems to produce
close results. At $1\%$, the breadth first and/or the random methods surpass
the ET$_{10\%}$ on Twonorm, Hastie, Madelon and Waveform. Those datasets
correspond to cases where ET was under-performing significantly compared to GIF.
All in all, the local baselines are never able to beat GIF, even in the
multiclass setting, which is particularly defavorable for GIF. Once again, the
conclusion is against the purely local baselines.

We believed the poor performances of the baselines are due to the building
mechanism of traditional ensemble methods. Although the trees are built
independently and with randomization, there remains an important redundancy
between them, which is especially defavorable to pruning. A global approach is
better able to avoid redundancy and can thus better exploit the node budget.
This would also explain why the best first variant performs worst in both
regression and classification: it is prone at picking redundant nodes, which
will usually offer the same kind of impurity reduction.

\bibliography{gif}
\bibliographystyle{icml2017}

\end{document}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
% From a template maintained at https://github.com/jamesrobertlloyd/cbl-tikz-poster
%
% Code near the top should be fairly standard and not need to be changed
%  - except for the document class
% Code lower down is more likely to be customised
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\documentclass[landscape,a0b,final,a4resizeable]{include/a0poster}

\usepackage{multicol}
\usepackage{color}
\usepackage{morefloats}
\usepackage[pdftex]{graphicx}
\usepackage{rotating}
\usepackage{amsmath, amsthm, amssymb, bm}
\usepackage{array}
\usepackage{booktabs}
\usepackage{multirow}
\usepackage{hyperref}

\usepackage{include/picins}
\usepackage{tikz}
\usetikzlibrary{shapes.geometric,arrows,chains,matrix,positioning,scopes,calc}
\tikzstyle{mybox} = [draw=white, rectangle]
\definecolor{darkblue}{rgb}{0,0.08,0.45}
\definecolor{blue}{rgb}{0,0,1}

\usepackage{dsfont}

\input{include/jlposter.tex}

\input{include/preamble.sty}
\newcommand{\vv}{\mathbf{v}}

\begin{document}
\begin{poster}

% Potentially add some space at the top of the poster
\vspace{0\baselineskip}

%%% Header
\begin{center}
\begin{pcolumn}{0.99}

\newcommand{\logowidth}{0.11\textwidth}

\pbox{0.99\textwidth}{}{linewidth=2mm,framearc=0.3,linecolor=camdarkblue,fillstyle=gradient,gradangle=0,gradbegin=white,gradend=white,gradmidpoint=1.0,framesep=1em}{
%
%%% Cambridge Logo
\begin{minipage}[c]{\logowidth}
  \begin{center}
    \includegraphics[width=20cm]{badges/hips-logo.png}
  \end{center}
\end{minipage}
%
%%% Title
\begin{minipage}[c][9cm][c]{0.76\textwidth}
  \begin{center}
    {\sffamily \VeryHuge \textbf{Convolutional Networks on Graphs\\[4mm] for Learning Molecular Fingerprints}}\\[10mm]
    {\huge\sffamily \Huge David Duvenaud*, Dougal Maclaurin*, Jorge Aguilera-Iparraguirre \\ Rafael G\'omez-Bombarelli, Timothy Hirzel, Al\'an Aspuru-Guzik, Ryan P. Adams\\[7.5mm]
    %\texttt{\{ti242, dkd23, zoubin\}@cam.ac.uk}
    }
  \end{center}
\end{minipage}
%
%
% Harvard logo
\begin{minipage}[c]{\logowidth}
  \begin{flushright}
    \includegraphics[width=9cm,trim=2em 0em 2em 2em, clip]{badges/harvard}
  \end{flushright}
\end{minipage}
%
}
\end{pcolumn}
\end{center}

\vspace*{3cm}

\Large

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%% Beginning of Document
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{multicols}{3}

\mysection{How to do regression on graphs?}

%\vspace{-0.5in}

\begin{tabular}{cc}
\begin{minipage}[c]{0.45\columnwidth}
\begin{itemize}
  \item Input can be any size or shape
  \item Hard to turn into fixed-length vector
  \item In our case, graphs represent molecules
  \item Applications to photovoltaics, organic LEDS, flow batteries and pharmaceuticals
\end{itemize}
\end{minipage} &
\begin{minipage}[c]{0.55\columnwidth}
%\includegraphics[width=\columnwidth]{../talks/talkfigs/learning_curves_3.pdf}
\centerline{\includegraphics[width=1.0\columnwidth, clip, trim=4mm 0mm 4mm 4mm]{figures/how-fingerprints.png}}
\end{minipage}
\end{tabular}

%\vspace{-0.5in}

\vspace{1.5in}

\mysection{Circular fingerprints}

%Also called Morgan fingerprints, or ECFP

\vspace{0.5in}

\begin{tabular}{cc}
\begin{minipage}[c]{0.5\columnwidth}
\begin{itemize}
  \item Maps variable-sized molecular graph to fixed-length binary vector
  %\item Does this by hashing self with neighbors iteratively
  \item Binary features indicate presence of substructures
\end{itemize}

\vspace{0.5in}

Can be efficiently computed using local operations:

\begin{itemize}
  \item At each layer, hash the features of each atom and its neighbors/bonds
  \item More layers correspond to increasing radius of substructures
  \item Interpret each hash as integer and set that entry to one
\end{itemize}
\end{minipage} &
\begin{minipage}[c]{0.5\columnwidth}
%\includegraphics[width=\columnwidth]{../talks/talkfigs/learning_curves_3.pdf}
\centerline{\includegraphics[width=0.9\columnwidth, clip, trim=4mm 12mm 4mm 4mm]{figures/fig_1}}
\end{minipage}
\end{tabular}

\vspace{0.5in}

Was state-of-the-art for large-scale regression and classification.

\newpage %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\mysection{Convolutional neural nets on graphs}

\vspace{0.5in}

\begin{tabular}{ccc}
\begin{minipage}[c]{0.45\columnwidth}

How to make circular fingerprints differentiable?
Replacing ops:
\vspace{0.5in}

\begin{center}
\begin{tabular}{rcl}
Hash  & $\rightarrow$ & Neural net \\
Index & $\rightarrow$ & Softmax \\
Write & $\rightarrow$ & Add
\end{tabular}
\end{center}

\vspace{0.5in}

Gives end-to-end differentiable convolutional network.

\vspace{0.5in}

Can be trained to adapt to particular tasks.

\end{minipage}
& \qquad &
\begin{minipage}[c]{0.45\columnwidth}
\begin{center}
Information flow graph:
\centerline{\includegraphics[width=\columnwidth, clip, trim=0mm 0mm 4mm 4mm]{figures/3d-nets/net1.png}}
\end{center}

Message passing between neighbors, then final pooling step

\end{minipage}
\end{tabular}

\vspace{1.5in}

\mysection{Neural fingerprints are interpretable}

\newcommand{\mywidtha}{8cm}
\newcommand{\mywidthb}{10.3cm}

When fed into linear layer, can see how fragments affect prediction:

\vspace{-0.5in}
\begin{center}
\begin{tabular}{>{\centering}m{\mywidthb} >{\centering}m{\mywidtha} >{\centering}m{\mywidtha} >{\centering\arraybackslash}m{\mywidtha}}
Fragments predictive of solubility &
%\molfeature{15}{0}{3.3cm} & \molfeature{15}{3}{3.3cm} & \molfeature{15}{2}{2.5cm}\\
\includegraphics[width=\mywidtha, clip, trim = 2mm 3mm 2mm 6mm]{figures/fig_5.pdf} &
\includegraphics[width=\mywidtha, clip, trim = 2mm 3mm 2mm 6mm]{figures/fig_6.pdf} &
\includegraphics[width=6cm, clip, trim = 2mm 3mm 2mm 6mm]{figures/fig_7.pdf} \\
\midrule
Fragments predictive of insolubility &
%\molfeature{18}{4}{3.3cm} & \molfeature{18}{1}{3.3cm} & \molfeature{18}{2}{3.3cm}
\includegraphics[width=\mywidtha, clip, trim = 2mm 3mm 2mm 6mm]{figures/fig_8.pdf} &
\includegraphics[width=\mywidtha, clip, trim = 2mm 3mm 2mm 6mm]{figures/fig_9.pdf} &
\includegraphics[width=\mywidtha, clip, trim = 2mm 3mm 2mm 6mm]{figures/fig_10.pdf} \\
%\end{tabular}
%\end{center}
\midrule
%\subsection{Toxicity features}
%We trained the same model architecture to predict toxicity in two different datasets.
%Shows fragments which maximally activate the feature most predictive of toxicity, in two separate datasets.
%\newcommand{\molfeaturetox}[2]{\includegraphics[width=3.4cm, clip, trim = 1mm 3mm 1mm 3mm]{fig_11.pdf}}%
%\begin{figure}[h]
%\begin{center}
%\begin{tabular}{>{\centering}m{\mywidthb} >{\centering}m{\mywidtha} >{\centering}m{\mywidtha} >{\centering\arraybackslash}m{\mywidtha}}
%\vspace{0.5}
\vspace{0.5in}Fragments predictive of toxicity on SR-MMP dataset\vspace{0.5in}
& \includegraphics[width=7cm]{figures/jorge-figures/7.png}
& \includegraphics[width=\mywidtha]{figures/jorge-figures/8.png}
& \includegraphics[width=\mywidtha]{figures/jorge-figures/9.png}\\
\midrule
\vspace{0.5in}
Fragments predictive of toxicity on NR-AHR dataset
\vspace{0.5in}
%& \molfeaturetox{7}{14} & \molfeaturetox{7}{6} & \molfeaturetox{7}{5}
& \includegraphics[width=6.5cm]{figures/jorge-figures/10.png}
& \includegraphics[width=\mywidtha]{figures/jorge-figures/11.png}
& \includegraphics[width=7cm]{figures/jorge-figures/12.png}\\
\end{tabular}
%\caption{
\end{center}

%Visualizing fingerprints optimized for predicting toxicity.
%Shown here are representative samples of molecular fragments (highlighted in red) which most activate the feature most predictive of toxicity.

%\emph{Top row:} the most predictive feature identifies groups containing a sulphur atom attached to an aromatic ring.
%\emph{Bottom row:} the most predictive feature identifies fused aromatic rings, also known as polycyclic aromatic hydrocarbons, a well-known carcinogen.

\newpage %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\mysection{Neural graph fingerprints\\generalize circular fingerprints}

Large random weights give similar behavior to circular fingeprints:
\vspace{0.5em}

\begin{tabular}{cc}
\begin{minipage}[c]{0.48\columnwidth}
\includegraphics[width=\columnwidth]{figures/fig_2.pdf}
\end{minipage} &
\begin{minipage}[c]{0.48\columnwidth}
\begin{center}
\vspace{0.5cm}\includegraphics[width=\columnwidth]{figures/fig_3.pdf}
\end{center}
\end{minipage}
\end{tabular}

\vspace{0.5em}

Small random weights already much better than circular fingerprints!

Can do even better by optimizing for given task.

\vspace{1in}

\mysection{Predictive accuracy}

Neural graph fingerprints fed to neural net generalizes state of the art:%, we can't not win:

\vspace{0.5em}

\begin{center}
\begin{tabular}{r|lll}
Dataset                      &   Solubility  & Drug efficacy & Photovoltaic \\
 & & & efficiency \\
\midrule
Units                        &   log Mol/L                            & EC$_{50}$ in nM                        & percent \\
\midrule
Predict mean                 & 2.07 $\pm$ 0.10        & 1.21 $\pm$ 0.03         & 2.53 $\pm$ 0.02 \\
Circular FPs + linear layer  & 1.31 $\pm$ 0.05        & \bf{1.06} $\pm$ 0.01    & 1.62 $\pm$ 0.03 \\
Circular FPs + neural net    & 1.18 $\pm$ 0.05        & 1.16 $\pm$ 0.04         & 1.41 $\pm$ 0.03 \\
Neural FPs + linear layer    & 0.87 $\pm$ 0.06        & \bf{1.07} $\pm$ 0.01    & 1.61 $\pm$ 0.06 \\
Neural FPs + neural net      & \bf{0.72} $\pm$ 0.05   & \bf{1.08} $\pm$ 0.01    & \bf{1.20} $\pm$ 0.04
\end{tabular}

\vspace{0.5em}

%RMSE Mean predictive accuracy of neural fingerprints compared to standard circular fingerprints.
\end{center}

\vspace{0.5em}

\mysection{Conclusion}

\begin{itemize}
\item Can learn graph features end-to-end!
\item Works on other types of graphs too
\item Code at \url{github.com/HIPS/neural-fingerprint}
\item Autodiff package that works on standard Numpy code:\\ \url{github.com/HIPS/autograd}
\end{itemize}

\end{multicols}
\end{poster}

\end{document}


\subsection{Preliminary experiments: encoding hierarchical rules}
Before diving into the FET task, KENN has been tested in a hierarchical classification scenario. Since the goal of this preliminary phase is to familiarize with KENN rather than achieve the best performance, we do not dwell on the architecture, training setup, and results. We focus, instead, on how hierarchical relations can be encoded with KENN. Different heuristics to represent a hierarchy through logical rules are proposed within these preliminary studies.

\subsubsection{Dataset}
% The baseline model implemented for these experiments has a simple architecture. It uses DistilBERT as encoder, followed by a dense fully connected layer and a final classification layer that applies the sigmoid activation function. The choice of this activation function is due to the fact that in a multilabel task we want as output the probability of each label. Starting from this architecture, KENN is placed between the fully connected layer and the classification layer. Clause weights are set as learnable parameters with an initial value of 0.5.

The dataset used for this experiment is called DBpedia Classes\footnote{https://www.kaggle.com/danofer/dbpedia-classes}. It is composed of approximately 240k labeled Wikipedia articles, where the only feature is the span of text. The types are organized as a forest of 9 trees, each of them representing a 3-level hierarchy. The type set counts 298 types: 9 at top-level, 70 at middle-level, and 219 at low-level. Each example is labeled with exactly 3 classes from the full path of a tree (i.e., one class per level). For example, if we consider the \textit{Species} tree shown in Figure~\ref{fig:dbpedia_classes}, an instance of the dataset could present $ label_{1} = Species $, $ label{_2} = Animal $, and $ label_{3} = Bird $.

\begin{figure}[H]
    \centering
    \includegraphics[width=.7\linewidth]{dbpedia_classes.png}
    \caption{Example of a tree of DBpedia Classes}
    \label{fig:dbpedia_classes}
\end{figure}

\subsubsection{Knowledge definition} \label{knowledge_generation}
The first step to integrate KENN is to create the logical KB. We can start from the tree in Figure~\ref{fig:dbpedia_classes} and then generalize different strategies for defining logical rules. The most intuitive and straightforward way consists in following the \textit{is-a} relations, which are implicitly encoded in the tree structure. Considering the picture, it is possible to state that if an instance is a \texttt{Fish}, then it is an \texttt{Animal}. In the same way, we can then proceed with the next level of the hierarchy and state that if an instance is an \texttt{Animal}, then it is a \texttt{Species}. This mechanism can be extended to all the other subtrees to create a logical clause per edge. The resulting KB will be a set of logical implications between subtypes and supertypes. Even if it seems a natural way to describe the hierarchy, it doesn't necessarily mean it is the most effective solution. For this reason, we should take a step back and analyze the alternatives offered by the hierarchy. Figure~\ref{fig:hierarchy_example} shows a generic hierarchy and highlights two groups of constraints we can distinguish:
\begin{enumerate}
    \item \textbf{Vertical constraints:} based on the paths of a tree, they can be used to represent specialization relations
    \item \textbf{Horizontal constraints:} based on pairs of nodes from different branches, they can be used to represent disjointness relations
\end{enumerate}
\begin{figure}[H]
    \centering
    \includegraphics[width=.7\linewidth]{hierarchy_example.png}
    \caption{Examples of a generic hierarchy. In green the vertical constraints, in red the horizontal constraints.}
    \label{fig:hierarchy_example}
\end{figure}
For each group of constraints, it is possible to implement different strategies to define the KB. In the rest of the thesis, these strategies will be referred to as \textit{KB modes}. Note that in the definitions below we will use unary predicates to represent the belonging of an instance $X$ to a type.

\paragraphn{Vertical Constraints}
Using the unary predicates $Subtype$ and $Supertype$ such that the type $Subtype$ is a specialization of the type $Supertype$, we can define three main strategies to build vertical constraints:
\begin{itemize}
    \item \textbf{Bottom Up:}
        \begin{itemize}
            \item FOL: $ \forall X, Subtype(X) \to Supertype(X) $
            \item KENN: $ \neg Subtype \vee Supertype $
            \item Defined by following every path of the tree from the leaves to the root. It represents the logic dependency between types, following the most widely used semantics in the definition of subclass.
            \item Open World Assumption: a clause is satisfied even when an instance is assigned only to supertypes
        \end{itemize}
    \item \textbf{Top Down:}
        \begin{itemize}
            \item FOL: $ \forall X, Supertype(X) \to (Subtype_{1}(X) \vee ... \vee Subtype_{n}(X)) $
            \item KENN: $ \neg Supertype \vee Subtype_{1} \vee ... \vee Subtype_{n}  $
            \item Defined by following every path of the tree from the root to the leaves
            \item Closed World Assumption: a clause is not satisfied when an instance is assigned to a supertype without any of its subtypes
        \end{itemize}
    \item \textbf{Hybrid:}
    \begin{itemize}
        \item Composed of a mix of \textit{Bottom Up} and \textit{Top Down} clauses
    \end{itemize}
\end{itemize}
Note that if a model that uses this kind of knowledge is confident about a given type, a \textit{Bottom Up} clause propagates this certainty towards the root of the hierarchy. Conversely, a \textit{Top Down} clause only guarantees that one of the subtypes is appropriate, but provides no information as to which one. The reason, of course, is the tree-like hierarchy, where each type has only one parent, but potentially many descendants.

The presented KB modes can be considered as the starting point to create new strategies by adding some variations. In Table~\ref{tab:kb_modes} we can find a summary that comprehends every KB mode, each of them accompanied by examples of clauses from the hierarchy in Figure~\ref{fig:dbpedia_classes}. Before proceeding with the explanation of the variants, we have to make clear why they have been introduced: the conflicts. If we look at the table, we can see that in the examples of the pure \textit{Bottom Up} and \textit{Top Down} modes there are conflicts: a middle-level label appears both as positive and negative literal. This fact derives from the translation of the logical implication into a disjunction since an intermediate node is at the same time antecedent and consequent of different logical implications. This behavior still hold for the \textit{Hybrid} mode.

Now that the possible problems of the starting strategies are known, it is possible to introduce their conflict-free variants. These variants avoid conflict situations in different ways. The ``Skip" ones rely on the transitivity of a relation by linking top-level and low-level nodes by skipping those middle-level nodes that would create a conflict (i.e., a middle-level node cannot be the antecedent of an implication rule). These solutions provide a more shallow representation of the knowledge, especially for the \textit{Top Down Skip}. In this case, a logical rule that represents the relation between top-level and low-level lose effectiveness because the implication may have too many consequents. The \textit{Hybrid In} and \textit{Hybrid Out} modes, instead, avoid conflicts by fixing the middle-level types as consequent or antecedent of an implication rule, respectively. In this way, the deltas produced by each CE for the same literal will always have the same sign.

Note that the proposed variants are valid for this dataset, but some of them do not apply to every context since they require a hierarchy with at least 3 levels.

\begin{table}
\centering
\caption{Strategies for defining logical clauses based on the hierarchy in Figure~\ref{fig:dbpedia_classes}}
\label{tab:kb_modes}
\begin{tabular}{c|l|}
\cline{2-2}
                                              & \multicolumn{1}{c|}{\textbf{Clauses}}                                                                                \\ \hline
\multicolumn{1}{|c|}{\textbf{Bottom Up}}      & \begin{tabular}[c]{@{}l@{}}$ c_{1}: \neg Fish \vee Animal $\\ $ c_{2}: \neg Bird \vee Animal $\\ $ c_{3}: \neg Animal \vee Species $\\ ...\end{tabular}                                                                           \\ \hline
\multicolumn{1}{|c|}{\textbf{Top Down}}       & \begin{tabular}[c]{@{}l@{}}$ c_{1}: \neg Species \vee Animal \vee ... \vee Plant $\\ $ c_{2}: \neg Animal \vee Fish \vee ... \vee Bird $\\ $ c_{3}: \neg Plant \vee Fem \vee ... \vee Cycad $\\ ...\end{tabular}                  \\ \hline
\multicolumn{1}{|c|}{\textbf{Hybrid}}         & \begin{tabular}[c]{@{}l@{}}$ c_{1}: \neg Species \vee Animal \vee ... \vee Plant $\\ $ c_{2}: \neg Animal \vee Species $\\ $ c_{3}: \neg Plant \vee Species $\\ ...\end{tabular}                                                  \\ \hline
\multicolumn{1}{|c|}{\textbf{Bottom Up Skip}} & \begin{tabular}[c]{@{}l@{}}$ c_{1}: \neg Fish \vee Animal $\\ $ c_{2}: \neg Fish \vee Species $\\ $ c_{3}: \neg Bird \vee Animal $\\ $ c_{4}: \neg Bird \vee Species $\\ ...\end{tabular}                                         \\ \hline
\multicolumn{1}{|c|}{\textbf{Top Down Skip}}  & \begin{tabular}[c]{@{}l@{}}$ c_{1}: \neg Species \vee Animal \vee ... \vee Plant $\\ $ c_{2}: \neg Species \vee Fish \vee ... \vee Bird \vee Fem \vee Cycad $\\ ...\end{tabular}                                                  \\ \hline
\multicolumn{1}{|c|}{\textbf{Hybrid In}}      & \begin{tabular}[c]{@{}l@{}}$ c_{1}: \neg Species \vee Animal \vee ... \vee Plant $\\ $ c_{2}: \neg Fish \vee Animal $\\ $ c_{3}: \neg Bird \vee Animal $\\ $ c_{4}: \neg Fem \vee Plant $\\ ...\end{tabular}                      \\ \hline
\multicolumn{1}{|c|}{\textbf{Hybrid Out}}     & \begin{tabular}[c]{@{}l@{}}$ c_{1}: \neg Animal \vee Species $\\ $ c_{2}: \neg Animal \vee Fish \vee ... \vee Bird $\\ $ c_{3}: \neg Plant \vee Species $\\ $ c_{4}: \neg Plant \vee Fem \vee ... \vee Cycad $\\ ...\end{tabular} \\ \hline
\end{tabular}
\end{table}

\paragraphn{Horizontal Constraints}
These constraints aim to avoid the co-occurrence of types whose instances form disjoint sets (e.g., if an example is labeled as \texttt{Person} it cannot be labeled as \texttt{Location} and vice versa). Using $T_i$ to indicate an arbitrary type, the logical rule that best represents this constraint is:
\begin{gather*}
    \forall X, T_{1}(X) \to (\neg T_{2}(X) \wedge ... \wedge \neg T_{n}(X))
\end{gather*}
With the following steps, we can derive an equivalent formula without implication and conjunctions:
\begin{align*}
    & T_{1} \to (\neg T_{2} \wedge ... \wedge \neg T_{n}) = \\
    & = \neg T_{1} \vee (\neg T_{2} \wedge ... \wedge \neg T_{n}) = \\
    & = \neg T_{1} \vee \neg (T_{2} \vee ... \vee T_{n})
\end{align*}
Unfortunately, the derived formula cannot be furthermore decomposed due to the presence of parentheses. We can opt for two alternative solutions to bypass the problem:
\begin{enumerate}
    \item \textbf{Approximate the rule with a softer constraint:} if an instance does not belong to \texttt{T$_1$}, then it should belong to another type. \\
    The logical formula for this statement is:
    \begin{gather*}
            \forall X, \neg T_{1}(X) \to (T_{2}(X) \vee ... \vee T_{n}(X))
    \end{gather*}
    that can be translated into KENN's language with the following steps:
    \begin{align*}
        & \neg T_{1} \to (T_{2} \vee ... \vee T_{n}) = \\
        & = T_{1} \vee (T_{2} \vee ... \vee T_{n}) = \\
        & = T_{1} \vee T_{2} \vee ... \vee T_{n}
    \end{align*}
    From a logical point of view, the resulting formula does not seem very helpful because it simply states that an instance belongs at least to one class. Indeed, if we repeat the previous steps using any other type as the antecedent of the implication, the derived clause will be identical. Furthermore, the action of KENN on such clauses would not lead to mutual exclusivity because every literal is positive and none of the involved predictions will receive a negative boost.

    \item \textbf{Split the rule into multiple clauses:} define a clause for each pair of disjoint types.
    Given two disjoint types, the logical formula to express the mutual exclusivity is:
    \begin{gather*}
        \forall X, T_{1}(X) \to \neg T_{2}(X)
    \end{gather*}
    If we consider $n$ disjoint pairs in which appears $T_{1}$, the resulting KB in KENN's language will be the following:
    \begin{align*}
        & c_{1}: \neg T_{1} \vee \neg T_{2}\\
        & ...\\
        & c_{n}: \neg T_{1} \vee \neg T_{n}
    \end{align*}
    Unlike strategy 1, this one preserves the soundness of the starting formula. However, there are still some issues to take into account. The first concerns the number of clauses necessary to cover the possible combinations of pairs: having $ n $ mutual independent types will result in $ \frac{n!}{2!(n-2)!} $ logical clauses. The second issue regards the action of KENN: while in strategy 1 there were only positive literals, in this strategy there are only negative literals, thus meaning that none of the predictions will ever receive a positive boost.
\end{enumerate}

\paragraphn{Horizontal constraints - Alternative usage}
Another use of horizontal constraints that is not related to this dataset, but could be useful in other contexts, is to encourage the co-occurrence between different types. Here the purpose is exactly the opposite with respect to the previous rules. Even if this goal can be reached by applying a simple inference on the final predictions without the use of KENN, in some tasks (e.g., domain adaptation) it may be useful to leave the decision to the network. If we want the model to learn a mapping between two types \texttt{A} and \texttt{B}, we can define the logical rule
\begin{gather*}
    (A \to B) \wedge (B \to A)
\end{gather*}
that can be expressed in KENN using the following clauses:
\begin{align*}
    & c1: \neg A \vee B \\
    & c2: \neg B \vee A
\end{align*}
The first thing we can notice is the presence of conflicts, but they will not have a negative impact on the final predictions. The preactivation of the literals belonging to the two clauses are the same with opposite signs, so a  literal cannot be dominant in both clauses. In other words, the final predictions always benefit from the knowledge enhancement since the effect of KENN will be one of the following:
\begin{itemize}
    \item \textbf{positive preactivations:} a positive aggregated delta is produced for each literal
    \item \textbf{negative preactivations:} a negative aggregated delta is produced for each literal
    \item \textbf{discord preactivations:} an aggregated delta with the sign of the highest\footnote{in absolute value} preactivation is produced for each literal
\end{itemize}

\subsubsection{Results}
As anticipated, the results are not the focus of this study since we are not dealing with the target problem of this thesis. To see KENN in action, the KB modes of Bottom Up, Top Down, Hybrid In, and Hybrid Out have been evaluated.
The baseline model implemented for these experiments has a simple architecture. It uses DistilBERT as encoder, followed by a dense fully connected layer and a final classification layer that applies the sigmoid activation function. The choice of this activation function is due to the fact that in a multilabel task we want as output the probability of each label. Starting from this architecture, KENN is placed between the fully connected layer and the classification layer. Clause weights are set as learnable parameters with an initial value of 0.5.

In Table \ref{tab:performance_dbpedia} are reported the results obtained by the baseline and KENN-based models. Hybrid Out is the configuration that brought more benefits to the baseline model with a $+0.0157$ on the F1 score, thanks to its highest recall. Even Top Down brought improvements with an increase of $+0.0097$. This did not happen with the other configurations.

An interesting fact that emerged regards the clause weights. By inspecting the final weights learned by the models, it was found out that the models that performed the best are the ones that finished the train with higher clause weights. In particular, some clauses have increased their starting weight by 10 times. On the contrary, the worst models are the ones whose final weights are smaller. This fact means that KENN helped the baseline model the most when it gained more influence on the final predictions. From these results, we can say that the higher the weight, the better the performance. This result may lead to think that it could be reasonable to set higher initial clause weights to anticipate the learning process. However, we have to take this conclusion with a grain of salt because it could be strictly related to this dataset.

\begin{table}[H]
\centering
\caption{Comparison between the baseline and KENN-based models on DBpedia Classes in terms of \textit{macro f1 classes}}
\label{tab:performance_dbpedia}
\begin{tabular}{|c|ccc|}
\hline
\textbf{KB mode} & \multicolumn{1}{c|}{\textbf{P}} & \multicolumn{1}{c|}{\textbf{R}} & \textbf{F1}     \\ \hline
- (baseline)       & 0.9265                          & 0.8644                          & 0.8897          \\ \hline
Bottom Up      & 0.924                           & 0.8685                          & 0.8901          \\ \hline
Top Down       & \textbf{0.93}                   & 0.8788                          & 0.8994          \\ \hline
Hybrid In      & 0.9186                          & 0.87                            & 0.8884          \\ \hline
Hybrid Out     & 0.927                           & \textbf{0.8913}                 & \textbf{0.9054} \\ \hline
\end{tabular}
\end{table}


\section{\texorpdfstring{Random number generators}{Random number generators}}
\vspace{5mm}
\large

\begin{definition}
	Secure RNG - even knowing complete past output, next bit is unpredictable $\Rightarrow$ statistically uniform.
\end{definition}

We have 2 type of random number generators:\\
a) PRNG: e.g. block cipher in CTR mode. Key should be changed on regular basis.\\
b) physical randomness (we have no proof such thing exists).
Examples:

\begin{itemize}
	\item Thermal noise or resister/diode
	\item Radioactive decay (count \# of particles)
	\item Radio noise
	\item Lava lamp (chaotic movement)
	\item Ring oscillator (attacker can observer)
	\item Precise timings of keystrokes, mouse movements or network packages (attacker can record or flood packages)
\end{itemize}

\begin{note}
	Entropy in physical processes is not as high as we can expect.
	Therefore, a combination of PRNG and physical randomness is the best choice.
\end{note}

Problem with PRGN + rekey: if a key or state of the PRNG is compromised, adding 1 bit or physical entropy does not help. Attacher can try both options (guess) and track the state. Therefore entropy should be mixed in large batches.

\subsection{Fortuna RNG}
\begin{definition}
	Fortuna RGN has a \emph{Generator} and \emph{Accumulator}.

	Generator is based on block cipher (AES) which encrypts a 128-bit counter (never overflows). After $2^{16}$ block we rekey and using the next 2 blocks as key.

	Trick: do not reset the counter to avoid short cycles.

	Accumulator: gets entropy from other sources and resets Generator periodically.
	Has $p_0, ..., p_{31}$ pods of entropy. External RNG is mixed to the pods in Round-Robin order: assign RNG to $p_i$, move to the next.
	Such process distributes RNG among the pods.

	After 100ms we mix pods into Generator. In the $j$-th mixing only selected pods are used, e.g. with index $2^j/i$.
\end{definition}

Assume that $\rho = $ rate of input entropy, measured in bits per 100ms.
If $\rho \geq 128 * 32 \Rightarrow p_0$ is enough for recovery. As at least 128 bit of entropy were put in $p_0$.
If $ \rho \geq 128 * 32/2^i \Rightarrow p_i$ is enough to recover.

To conclude, for any given rate of entropy Fortuna can recover after some time. Speed of recovery depends on entropy rate.

\subsection{Secure channel}
We use symmetric encryption to establish secure channel. Assume Alice and Bob share unique random key $k$.
Alice sends $\{ m_i \}_0^n$ sequence of messages, Bob receives a subsequence and he is still able to know which one he got.
On the other hand, attacker have no such info, except for length of the sequence.

We use
\begin{itemize}
	\item Random IV for every message (public).
	\item AES in CTR mode for encryption (alternatively ChaCha20).
	\item MAC after encrypt.
	\item Sequence numbers (inside MAC).
	\item Key derivation function which generates encryption and MAC keys for both directions.
		\[ h(k | id) \]
\end{itemize}


\section{Preliminary Results}\label{section:results}

As a criteria of the project is to compare the performance of different frameworks against the performance of ROOT based \texttt{HistFactory} benchmarking of the different backends was carried out.
As a preliminary benchmark of the frameworks a one point $\text{CL}_{s}$ test\footnote{$\text{CL}_{s}$ is defined as the ratio of the signal + background hypothesis $p$-value to $1$ minus the background only hypothesis $p$-value: $\text{CL}_{s} = p_{s+b}/\left(1-p_{b}\right)$.~\cite{Cowan:2010js}} was performed, in which the $p$-values of test statistics from the data and the ``Asimov'' data set are compared given the parameter of interest.~\cite{Cowan:2010js}
The model used is a simple one in which every bin has the same content to ensure that the fit will complete.
In lieu of model complexity large numbers of bins are given to simulate difficult conditions for the fit as each bin is uncorrelated and so represents an additional nuisance parameter.
The results are shown against each other in Figure~\ref{fig:benchmark_backends}.
As a pure MXNet optimizer has not been fully implemented in pyhf at this time it has not yet been benchmarked, and so only backends that do have language specific optimizers have been compared to ROOT based \texttt{HistFactory}.\\

The preliminary results show that the computational backends that support automatic differentiation and built in parallelism show promising scaling behavior as the nature of the statistical fits becomes more computational difficult, as seen in Figure~\ref{fig:relative_time}.
The optimizer for the NumPy pyhf backend is based on optimizers in SciPy, and while the NumPy backend optimizer does not currently have automatic differentiation support it does show very good timing for lower computational difficulty problems.
Given that pyhf allows for fluidity in the choice of backend, this demonstrates a possible use case for automatically switching from the low overhead NumPy backend to a graph based backend as computational difficulty rises.

\begin{figure}
 \centering
 \includegraphics[width=0.9\linewidth]{benchmark_times.pdf}
 \caption{Comparison of the mean time needed to complete a one point $\text{CL}_{s}$ test for ROOT \texttt{HistFactory} and the NumPy, TensorFlow, and PyTorch pyhf backends vs. the number of bins in the associated fit.
  The binning choices (number of nuisance parameters) used are $n_{\text{bins}} \in \left\{1, 10, 50, 100, 200, 500, 800, 1000\right\}$.
  For each binning choice the fit is repeated 5 times.
  For ROOT \texttt{HistFactory} only binning choices up to 200 are used as runtime became too long afterwards.
  The minimum and maximum run times time for each binning choice are shown as open shapes corresponding to the shapes of the markers for each backend, and an uncertainty bar corresponding to 1 standard deviation is drawn (though it may be obscured by the markers).
 }\label{fig:benchmark_backends}
\end{figure}

\begin{figure}[htbp]
 \centering
 \subfigure[ROOT \texttt{HistFactory}]{%
  \includegraphics[width=0.5\textwidth]{relative_times_root_log.eps}\label{fig:relative_time_root_HF}%
 }\hfil
 \subfigure[NumPy backend]{%
  \includegraphics[width=0.5\textwidth]{relative_times_numpy_log.eps}\label{fig:relative_time_numpy}%
 }
 \subfigure[TensorFlow backend]{%
  \includegraphics[width=0.5\textwidth]{relative_times_tensorflow_log.eps}\label{fig:relative_time_tensorflow}%
 }\hfil
 \subfigure[PyTorch backend]{%
  \includegraphics[width=0.5\textwidth]{relative_times_pytorch_log.eps}\label{fig:relative_time_pytorch}%
 }
 \caption{Comparison of the mean time needed to complete a one point $\text{CL}_{s}$ test for ROOT \texttt{HistFactory} and the pyhf backends for a number of bins in the associated fit relative to the time for a single bin.
  The binning choices (number of nuisance parameters) used are $n_{\text{bins}} \in \left\{1, 10, 50, 100, 200, 500, 800, 1000\right\}$.
  For ROOT \texttt{HistFactory} only binning choices up to 200 are used as runtime became too long afterwards.
 }\label{fig:relative_time}
\end{figure}


% From Robert_Harper@terrier.fox.cs.cmu.edu  Wed Jun 19 00:07:29 1996
%Return-Path: Robert_Harper@terrier.fox.cs.cmu.edu
%To: tofte@diku.dk
%Subject: revised appendix 5 (tex source, no references)
%Date: Tue, 18 Jun 1996 18:07:12 -0400
%From: Robert Harper <Robert_Harper@terrier.fox.cs.cmu.edu>
%
\section{Appendix: The Development of ML}
\label{story-app}

This Appendix records the main stages in the development of ML, and the people
principally involved.  The main emphasis is upon the design of the language;
there is also a section devoted to implementation. On the other hand, no
attempt is made to record work on applications of the language.
\note{\thenewpreface}{This appendix has been revised and extended in
several ways. A detailed list of changes is not available.}
\subsection*{Origins}

ML and its semantic description have evolved over a period of about
twenty years.  It is a fusion of many ideas from many people;  in this
appendix we try to record and to acknowledge the important precursors
of its ideas, the important influences upon it, and the important
contributions to its design, implementation and semantic description.

ML, which stands for {\em meta language}, was conceived as a medium for
finding and performing proofs in a formal logical system.  This application
was the focus of the initial design effort, by Robin Milner in collaboration
first with Malcolm Newey and Lockwood Morris, then with Michael Gordon and
Christopher Wadsworth~\cite{GMMNW}. The intended application to proof affected
the design considerably.  Higher order functions in full generality seemed
necessary for programming proof tactics and strategies, and also a robust type
system (see below).  At the same time, imperative features were important for
practical reasons; no-one had experience of large useful programs written in a
pure functional style. In particular, an exception-raising mechanism was
highly desirable for the natural presentation of tactics.

The full definition of this first version of ML was included in a
book~\cite{GMW} which describes LCF, the proof system which ML was designed to
support.  The details of how the proof application exerted an influence on
design is reported by Milner~\cite{Mil2}.  Other early influences were the
applicative languages already in use in Artificial Intelligence, principally
LISP~\cite{McC}, ISWIM~\cite{Lan} and POP2~\cite{BP}.

\subsection*{Polymorphic types}

The polymorphic type discipline and the associated type-assignment algorithm
were\linebreak
 prompted by the need for security;  it is vital to know that when
a program produces an object which it claims to be a theorem, then it
is indeed a theorem.  A type discipline provides the security, but a
polymorphic discipline also permits considerable flexibility.

The key ideas of the type discipline were evolved in combinatory logic by
Haskell Curry and Roger Hindley, who arrived at different but equivalent
algorithms for computing principal type schemes.  Curry's~\cite{Cur} algorithm
was by equation-solving; Hindley~\cite{Hin} used the unification algorithm of
Alan Robinson~\cite{Rob} and also presented the precursor of our type
inference system.  James Morris~\cite{Mor} independently gave an
equation-solving algorithm very similar to Curry's.  The idea of an algorithm
for finding principal type schemes is very natural and may well have been
known earlier. Roger Hindley has pointed out that Carew
Meredith's inference rule for propositional logic called Condensed Detachment,
defined in the early 1950s, clearly suggests that he knew such an algorithm
\cite{Mer}.

Milner~\cite{Mil1}, during the design of ML, rediscovered principal types and
their calculation by unification, for a language (slightly richer than
combinatory logic) containing local declarations.  He and Damas~\cite{DM}
presented the ML type inference systems following Hindley's style.  Damas
\cite{Dam}, using ideas from Michael Gordon, also devised the first
mathematical treatment of polymorphism in the presence of references and
assignment.  Tofte~\cite{Tof-a} produced a different scheme employing
so-called {\em imperative types}, which was adopted in the original version of
the language.  This approach has been superseded in the present language by a
simpler scheme, suggested by Tofte~\cite{Tof-a}, Andrew Wright~\cite{wright95}, and
Xavier Leroy~\cite{Ler1}, according to which polymorphic bindings are
restricted to non-expansive expressions.

\subsection*{Refinement of the Core Language}

Two movements led to the re-design of ML.  One was the work of Rod
Burstall and his group on specifications, crystallised in the specification
language CLEAR~\cite{BG} and in the functional programming language HOPE
\cite{BMS};  the latter was for expressing executable specifications.  The
outcome of this work which is relevant here was twofold.  First, there were
elegant programming features in HOPE, particularly pattern matching and
clausal function definitions; second, there were ideas on modular construction
of specifications, using signatures in the interfaces.  A smaller but
significant movement was by Luca Cardelli, who extended the data-type
repertoire in ML by adding named records and variant types.

In 1983, Milner (prompted by Bernard Sufrin) wrote the first draft of a
standard form of ML attempting to unite these ideas; over the next three years
it evolved into the Standard ML core language.  Notable here was the harmony
found among polymorphism, HOPE patterns and Cardelli records, and the nice
generalisations of ML exceptions due to ideas from Alan Mycroft, Brian Monahan
and Don Sannella.  A simple stream-based I/O mechanism was developed from
ideas of Cardelli by Milner and Harper.  The Standard ML core language is
described in detail in a composite report~\cite{HMM} which also contains a
description of the I/O mechanism and MacQueen's proposal for program modules
(see later for discussion of this). Since then only few changes to the core
language have occurred.  Milner proposed equality types, and these were added,
together with a few minor adjustments~\cite{Mil3}.  The
last development before the 1990 Definition was in the exception mechanism, by MacQueen using an idea
from Burstall~\cite{AMMT}; it harmonized the ideas of exception and data type
construction.

\subsection*{Modules}

Besides contributory ideas to the core language, HOPE~\cite{BMS} contained a
simple notion of program module.  The most important and original feature of
ML modules, however, stems from the work on parameterised specifications in
CLEAR~\cite{BG}.  MacQueen, who was a member of Burstall's group at the time,
designed~\cite{Mac} a new parametric module feature for HOPE inspired by the
CLEAR work.  He later extended the parameterisation ideas by a novel method of
specifying sharing of components among the structure parameters of a functor,
and produced a draft design which accommodated features already present in ML
-- in particular the polymorphic type system.  This design was discussed in
detail at Edinburgh, leading to MacQueen's first report on modules~\cite{HMM}.

Thereafter, the design came under close scrutiny through a draft operational
static semantics and prototype implementation of it by Harper, through Kevin
Mitchell's implementation of the evaluation, through a denotational semantics
written by Don Sannella, and then through further work on operational
semantics by Harper, Milner, and Tofte.  (More is said about this in the later
section on Semantics.)  In all of this work the central ideas withstood
scrutiny, while it also became clear that there were gaps in the design and
ambiguities in interpretation.  (An example of a gap was the inability to
specify sharing between a functor argument structure and its result structure;
an example of an ambiguity was the question of whether sharing exists in a
structure over and above what is specified in the signature expression which
accompanies its declaration.)

Much discussion ensued; it was possible for a wider group to comment on
modules through using Harper's prototype implementation, while Harper, Milner
and Tofte gained understanding during development of this semantics.  In
parallel, Sannella and Tarlecki explored the implications of modules for the
methodology of program development~\cite{ST}.  Tofte, in his thesis
\cite{tofte88}, proved several technical properties of modules in a skeletal
language, which generated considerable confidence in this design. A key point
in this development was the proof of the existence of principal signatures,
and, in the careful distinction between the notion of {\it enrichment} of
structures, which allows more polymorphism and more components, and {\it
realisation} which allows more sharing.

At a meeting in Edinburgh in 1987 a choice of two designs was presented,
hinging upon whether or not a functor application should coerce its actual
argument to its argument signature.  The meeting chose coercion, and
thereafter the production of Section~\ref{statmod-sec} of this report -- the
static semantics of modules -- was a matter of detailed care.  That section is
undoubtedly the most original and demanding part of this semantics, just as
the ideas of MacQueen upon which it is based are the most far-reaching
extension to the original design of ML.

Considerable experience was gained in implementing, programming with, and
teaching the language during the nearly ten years since the definition was
first published.  Based on this experience a number of design decisions were
revisited at a meeting of the authors in Cambridge at the end of 1995.  At
this meeting it was decided to make several modest, but significant, changes
to the language in order to simplify the semantics and to correct some
shortcomings that had come to light.  The most important of these changes was
the replacement of the imperative type discipline by the so-called value
restriction (discussed above), the elimination of structure sharing as a
separate concept from type sharing, and the introduction of the closely
connected mechanisms of opaque signature matching and type abbreviations in
signatures.  An important impetus for these changes to the modules language
was the work of Leroy~\cite{leroy94}, and Harper and
Lillibridge~\cite{HL} on the type-theoretic interpretation of
modules (described below).

\subsection*{Implementation}

The first implementation of ML was by Malcolm Newey, Lockwood Morris and Robin
Milner in 1974, for the DEC10.  Later Mike Gordon and Chris Wadsworth joined;
their work was mainly in specialising ML towards machine-assisted reasoning.
Around 1980 Luca Cardelli implemented a version on VAX; his work was later
extended by Alan Mycroft, Kevin Mitchell and John Scott.  This version
contained one or two new data-type features, and was based upon the {\em
Functional Abstract Machine (FAM)}, a virtual machine which has been a
considerable stimulus to later implementation.  By providing a reasonably
efficient implementation, this work enabled the language to be taught to
students; this, in turn, prompted the idea that it could become a useful
general purpose language.

In Gothenburg, an implementation was developed by Lennart Augustsson and
Thomas Johnsson in 1982, using lazy evaluation rather than call-by-value; the
result was called {\em Lazy ML} and is described in~\cite{Aug}.  This work is
part of continuing research in many places on implementation of lazy
evaluation in pure functional languages.  But for ML, which includes
exceptions and assignment, the emphasis has been mainly upon strict evaluation
(call-by-value).

In Cambridge, in the early 1980s, Larry Paulson made considerable improvements
to the Edinburgh ML compiler, as part of his wider programme of improving {\em
Edinburgh LCF} to become {\em Cambridge LCF}~\cite{Pau}.  This system has
supported larger proofs than the Edinburgh system, and with greater
convenience; in particular, the compiled ML code ran four to five times
faster.

Around the same time G\'{e}rard Huet at INRIA (Versailles) adapted ML to
Maclisp on Multics, again for use in machine-assisted proof.  There was close
collaboration between INRIA and Cambridge in this period.  ML has undergone a
separate development in the group at INRIA on the {\em CAML}
language~\cite{CCM}.  Work on {\em CAML} included the development of several
extensions to the core language, notably updatable fields in record types,
values with dynamic types, support for lazy evaluation, and handling of
embedded languages with user-defined syntax.  It did not, however, include
modules.

The first implementation of the Standard ML core language was by Mitchell,
Mycroft and Scott at Edinburgh, around 1984.  The prototype implementation of
modules, before that part of the language settled down, was done in 1985-6;
Mitchell dealt with evaluation, while Harper tackled the elaboration (or
`signature checking') which raised problems of a kind not previously
encountered.  Harper's implementation employed a form of unification that was
later adopted in the static semantics of modules.

At around the same time the {\em Poly/ML} implementation began with a
suggestion from Mike Gordon that an interesting application of Matthews' Poly
language would be to implement Standard ML.  Important experience was gained
through Matthews' early implementation of the core language, followed by
several versions of the modules language as they were devised.  {\em Poly/ML}
features arbitrary precision arithmetic, a process package, and a windowing
system.  Considerable experience has been gained with the compiler, notably by
Larry Paulson at Cambridge and by Abstract Hardware Limited (AHL).

  The {\em Standard ML of New Jersey (SML/NJ)} system has been in active development
  since 1986~\cite{am87,am91}.
  Initially started by David MacQueen at Bell Laboratories and Andrew Appel at Princeton University,
  the project has also benefited from significant contributions
  by Matthias Blume, Emden Gansner, Lal George, John Reppy and Zhong Shao.
  {\em SML/NJ} is a robust and complete environment for Standard ML that supports
  the implementation of large software systems and generates efficient code for a
  number of different hardware and software platforms.  {\em SML/NJ} also serves as
  a laboratory for compiler research: in implementations of module systems for ML;
  code optimization based on continuation-passing style; efficient pattern matching;
  and very fast heap allocation and garbage collection.  Dozens of researchers
  have contributed to the development of the compiler, in such areas as
  efficient closure representations, first-class continuations, type-directed
  compilation, concurrent programming, portable code generators, separate
  compilation, and register allocation.
  {\em SML/NJ} has also been widely used to explore extending SML with
  concurrency features.

%In 1986 Andrew Appel and David MacQueen began work on the {\em Standard ML of
%New Jersey (SML/NJ)} compiler~\cite{am87}.  {\em SML/NJ} is a robust and
%complete environment for Standard ML that supports the implementation of large
%software systems and generates efficient code for a number of different
%hardware and software platforms.  {\em SML/NJ} also serves as a laboratory for
%compiler research: in implementations of module systems for ML; code
%optimization based on continuation-passing style; efficient pattern matching;
%and very fast heap allocation and garbage collection.  Dozens of researchers
%have contributed to the development of the compiler, in such areas as
%efficient closure representations, first-class continuations, type-directed
%compilation, concurrent programming, portable code generators, separate
%compilation, and register allocation.

In 1989, Mads Tofte, Nick Rothwell and David N. Turner started work on the
{\em ML Kit Compiler} in Edinburgh. The {\em ML Kit} is a direct translation
of the 1990 Definition into a collection of Standard ML modules, emphasis being
on clarity rather than efficiency. During 1992 and 1993, Version~1 of the {\em
ML Kit} was completed, mostly through the work of Nick Rothwell at Edinburgh
and Lars Birkedal at DIKU\cite{BRTT}. In 1994, region inference was added to
the {\em ML Kit}, by Mads Tofte. Lars Birkedal wrote a region-based C-code
generator and a runtime system in C.  In 1995, Martin Elsman and Niels
Hallenberg extended this work to generate native code for the HP PA-RISC
architecture.

Harlequin Ltd. began the implementation of a commercial compiler in 1990.  The
{\em MLWorks} system is a fully-featured graphical programming environment,
including an interactive debugger, inspector, browser, extensive profiling
facilities, separate compilation and delivery, a foreign-language interface,
and libraries for threads and windowing systems.
%Harlequin was a major
%partner in the effort to standardize the revised Standard ML basis library.

{\em Caml Light}, a lightweight reimplementation of {\em CAML} released in
1991, added a simple module system in the style of Modula-2, targeted towards
separate compilation of modules: structures and signatures are identified with
files, functors and multiple views of a structure are not supported.  These
were added in the {\em Caml Special Light} implementation in 1995, while
preserving the support for separate compilation. {\em Caml Special Light} and
the present version of Standard ML share several important simplifications,
such as the value restriction on polymorphism, type definitions in signatures,
and the lack of support for structure sharing.  The static semantics for {\em
Caml Special Light} is based on the type-theoretic properties of dependent
function types (functor signatures) and manifest types (type definitions in
signatures)~\cite{leroy94}.

{\em Moscow ML} is an implementation of core Standard ML, created in 1994 by
Sergei Romanenko in Moscow and Peter Sestoft in Copenhagen.  The {\em Caml
Light} system was used to implement the dynamic semantics, and the ML Kit
guided the implementation of the static semantics.  The result is a compact
and robust implementation, suitable for teaching.

The {\em TIL (Typed Intermediate Languages)} compiler developed at Carnegie
Mellon
University by Greg Morrisett, David Tarditi, Perry Cheng, Chris Stone,
Robert Harper,
and Peter Lee demonstrates the use of
types in compilation.  All but the last few stages of {\em TIL} are expressed
as type-directed and type-preserving transforms.  Types are used at run time
to support unboxed, untagged data representations and natural calling
conventions in the presence of variable types and garbage collection.  {\em
TIL} employs a wide variety of conventional functional language optimizations
found in other SML compilers, as well as a set of loop-oriented optimizations.
A description of the compiler and an analysis of its performance appears
in~\cite{Tar}.

Other currently active implementations are by Michael Hedlund at the
Rutherford-Appleton Laboratory, by Robert Duncan, Simon Nichols and Aaron
Sloman at the University of Sussex ({\em POPLOG}) and by Malcolm Newey and his
group at the Australian National University.

\subsection*{Semantics}

The description of the first version of ML~\cite{GMW} was informal, and in an
operational style; around the same time a denotational semantics was written,
but never published, by Mike Gordon and Robin Milner.  Meanwhile structured
operational semantics, presented as an inference system, was gaining credence
as a tractable medium.  This originates with the reduction rules of
$\lambda$-calculus, but was developed more widely through the work of Plotkin
\cite{Plo}, and also by Milner.  This was at first only used for dynamic
semantics, but later the benefit of using inference systems for both static
and dynamic semantics became apparent.  This advantage was realised when
Gilles Kahn and his group at INRIA were able to execute early versions of both
forms of semantics for the ML core language using their Typol system
\cite{Des}.  The static and dynamic semantics of the core language reached a
final form mostly through work by Tofte and Milner.

The modules of ML presented little difficulty as far as dynamic semantics is
concerned, but the static semantics of modules was a concerted effort by
several people.  MacQueen's original informal description~\cite{HMM} was the
starting point; Sannella wrote a denotational semantics for several versions,
which showed that several issues had not been settled by the informal
description.  Robert Harper, while writing the first implementation of
modules, made the first draft of the static semantics.  Harper's version made
clear the importance of structure names; work by Milner and Tofte introduced
further ideas including realisation; thereafter a concerted effort by all
three led to several suggestions for modification of the language, and a small
range of alternative interpretations; these were assessed in discussion with
MacQueen, and more widely with the principal users of the language, and an
agreed form was reached.

Concurrently with the formulation of the Definition of Standard ML, Harper and
Mitchell took up the challenge adumbrated by MacQueen~\cite{Mac2} to find
a type-theoretic interpretation of Standard ML~\cite{HM}.  This work led to
the formulation of the XML language, an explicitly-typed $\lambda$-calculus
that captured many aspects of Standard ML.  Although incomplete, their
approach formed the basis for a number of subsequent studies, including the
work of Harper and Lillibridge~\cite{HL} and Leroy~\cite{leroy94} on the
type-theoretic interpretation of modules.  This work influenced the decision
to revise the language, and culminated in a type-theoretic interpretation of
the present language by Harper and Stone~\cite{SH}.  The TIL/ML compiler
(described above) is based directly on this interpretation.

There is no doubt that the interaction between design and semantic description
of modules has been one of the most striking phases in the entire language
development, leading (in the opinion of those involved) to a high degree of
confidence both in the language and in the semantics.

\subsection*{Program Libraries}
%During 1989-1991, Dave Berry produced the first program library for
%Standard ML\cite{mllib91,berry93}. Subsequently, a partnership between
%the originators of {\em SML/NJ}, {\em MLWorks} and
%{\em Moscow ML} was formed, with the goal of creating an industrial strength
%initial basis for Standard ML. The resulting SML Basis
%Library\cite{mllib96} is a much improved and extended
%replacement of the initial basis defined in the 1990 Definition of
%Standard ML.
 During 1989-1991, Dave Berry produced the first program library for
 Standard ML\cite{mllib91,berry93}.
 The {\em SML/NJ} system is distributed with a rich library organised by
 Emden Gansner and John Reppy; this library was the starting point for
 the SML Basis Library~.
 The {\em SML Basis Library\/}\cite{mllib96} has been developed
 over the past three years in a
 partnership between the SML/NJ effort, {\em MLWorks}, and {\em Moscow ML}.
 The resulting library is a much improved and extended
 replacement of the initial basis defined in the 1990 Definition of
 Standard ML.


\chapter{Conclusion \& Perspectives}         % ne pas numéroter
%\phantomsection\addcontentsline{toc}{chapter}{Conclusion \& Perspectives} % dans TdM

\section{General conclusions}
The objectives of the present thesis were, first, to deploy the infrared thermography technique in the procedure of maintaining the ``cold food chain'', especially in insulated vehicles of ATP standards. The application of infrared thermography aims to identify thermal insulation anomalies, which the standard ATP test cannot localize.

The preliminary work focused on mapping the heat flux on the external surface of an insulated roll-container using the Infrared thermography technique. The ATP standard measurement was performed to obtain the experimental results, meanwhile IR images of the roll-container were obtained when the steady condition was reached, in order to analyze and compute the corresponding heat flux on the entire surface. A simple thermal resistance model was applied to conduct the computation. Final temperature figures showed a good uniform distribution, and several defects in the structure such as thermal bridges or air leakages were identified. A reference zone of the external wall was measured by a thermal flux meter, then with that reference the entire surface heat flux map was determined. In addition, for a better view of the heat flux map, the homography, one of the computer vision techniques, was performed in the raw images by applying a bilinear interpolation with the projective transformation matrix. The final corrected heat flux map was demonstrated for each surface, in which the right surface showed a smaller value than the others.  Due to the air streaming, temperatures in the lateral surfaces were a little smaller than other surfaces, thus this lead to the smaller heat flux values.

When implemented into the internal surface of the insulated vehicle, a panoramic view was needed, since the field of view (FOV) of the infrared camera could not capture the entire surface of insulated vehicle. With the help of an infrared camera mounted on a pan-tilt head and automatically driven by a suitable software, a series of thermal images of the inner walls of the vehicle under steady condition have been captured. Proper computer vision techniques such as inverse spherical projection and stitching images by translation helped to generate the final panorama. The same thermal resistance model was utilized to compute the corresponding heat flux map. Results demonstrated a good performance of the algorithm, though the manual creation of the panoramic view required more time for completion. Compared with the standard ATP test, the final K-value obtained by infrared thermography showed a good accuracy (0.87\% of error).

Then based on the previous favorable results, the second aspect of this project was to explore cold approaches (such as compressed air, liquid nitrogen, etc.) in infrared thermography for Non-Destructive Testing \& Evaluation. The first interesting idea of detection of insulation flaws and thermal bridges in insulated truck box panels then emerged. As it is not convenient to heat the entire vehicle panels for the detection, cooling them by compressed air therefore can be a better solution. The study then focused on the cooling approach for the truck box panels inspection by infrared thermography. Both heating and cooling methods were applied by lamp and compressed air respectively. Numerical simulations under COMSOL Multiphysics{\textregistered} platform were conducted as well. For a comprehensive analysis, passive thermography detection in computational models has been presented at the same time. Results demonstrate that the compressed air spray is more rapid than the traditional heating method in providing successful detection.

A consideration of replacing compressed air by liquid nitrogen was then explored in more detail. Thus a study was performed, in which a steel specimen was used to test three different stimulations for thermal images and also Receiver Operating Characteristic (ROC) analysis comparison. Results showed that all techniques highlighted part of the flaws in the sample, whereas the liquid nitrogen technique represented the defects only at the beginning; this may be due to the high conductivity of steel. In thermal results, the PCT post-processing method displayed better results for all procedures. More defects were exhibited in Flash stimulation with PCT processing. ROC curve analysis has elucidated a straightforward classification comparison, in which the best curve was obtained using the Flash technique with PCT processing.

\section{Future perspectives}
Generally, the infrared thermography technique has been applied with promising results in ATP standard insulated vehicle tests, for the goal of maintaining the ``cold food chain". The results have demonstrated that the benefits of time-saving and the accuracy in the determination of the K-value could be applied for assessment at a commercial level.  For the panoramic view of the insulated vehicle internal surface, due to the repeated structure on the internal surface of vehicle, algorithms of automatic creation which have better feature detection and comparison remain to be explored.  A suitable software package may be created to simplify the post-processing of thermal images. This will facilitate the computation of the K-value.

On the other hand, the exploration of cold approaches in infrared thermography for Non-Destructive Testing and Evaluation has also shown several advantages. Compressed air cooling can be a good replacement of heating in the detection of insulation flaws for truck box panels. The strategy of heating one side and cooling another side can be deployed in practice, since ideal cases in simulation show favorable results.

The use of ROC curves to compare the different methods in infrared thermography is an interesting approach. This technique would benefit from more complete application and discussion, which would favor a more harmonious implementation in traditional techniques of NDT.


\documentclass[../../main]{subfiles}
\graphicspath{{images/rubiks-exp/}}

\begin{document}

This project is a continuation of the previous experiment described in Section \ref{subsec:sudoku_experiment}. Ideas developed for Sudoku were extrapolated on a Rubik's cube face detection problem as they are assumed to be similar in some sense \cite{rubiks_cube_zakharov}.

\subsubsection{Contour and corner detection}

Contour detection was covered in the previous step but in this case there were some issues to be solved. From the beginning, it was an issue that edges of the cube are not very clear thus face extraction with the biggest contour is not an option. On the contrary, small contours of face pieces were detected with higher success rate. Therefore, they were used for face detection. So as the next step, the closed contours with proper approximation of 4 corners were picked. These contours were supposed to be diffused square pieces of cube's face. The mentioned operations still were not able to filter all duplicates. They occur from time to time if \ac{OpenCV} detects a couple of different contours for the same object. To reject unnecessary contours, all of those with almost the same center of mass were removed but one. In order to find a face it is not enough to find contours. Diagonal of these small contours were computed and if 9 diagonals exist and located at the same angle to horizon their contours assumed to be face's pieces. The following action is to find the extremums or top-left, top-right, bottom-left, bottom-right points. The found extremums are expected to be the face's corners.

\subsubsection{Extraction}

The corner detection is covered in the previous subsection thus at this point 4 detected corners should to be passed to the next stage to apply linear transformation. As a result, plane image of the face will be stored.
This approach has the limitation for an input. It is not possible to detect the face corners if an image or a video quality is poor or contours are broken. These circumstances will fail face extraction. However, if the source quality is sufficient then this algorithm performs well because of its simplicity and performance.

\subsubsection{Transforming Image}

At this stage the same approach as in the Sudoku experiment was used. The same set of equations was used to transform the image. The system of equations from figure \ref{eq:persptransform} makes mapping of points from distorted image to a flat square image possible. To achieve this \ac{OpenCV}'s $GetPerspectiveTransform$ and $WarpPerspective$ methods were used for applying perspective transformation to the source.

\subsubsection{Identifying unique faces}

There are many possible options to check images for similarities like feature matching, color histograms, cross correlation, euclidean distance. Looking at a cube it is clear that there are no specific features on it, only colored pieces. The other case is the observation problem, the cube can be shown in different orientations. From the observations made, color histograms was chosen as a best choice to distinguish faces.
All uniquely detected faces were stored in the list and compared with every newly detected face. In order to achieve this images were converted to \ac{HSV} to calculate the histogram and normalize the results. To compare images correlation was used and performed well. For example, if the correlation is more than 0.6 it is assumed that faces are too similar thus the new one has to be rejected as already found.
This algorithm is a simple one however the empirical research showed that it works as expected.

\subsubsection{Color detection}

Color detection is nearly the easiest part of the program. In order to detect \ac{RGB} color representation for each piece of a cube face it should be segmented into $n$ equal squares, since the 3 by 3 Rubik's cube was used. The segmentation operation produces nine pieces and this approach cannot handle larger cubes.
After the piece of the face is cut out, the center pixel \ac{RGB} data and intensity are taken. The extracted data is collected into specific data structure and this is the end of the process. Extracted \ac{RGB} colors are mapped to the new data structure in order to convert it to the Rubik's cube 3D model for solving and rendering purposes.

\subsubsection{Conclusion}

This project was a continuation of the Sudoku ideas projected onto the Rubik's cube. At this stage single faces were extracted correctly but some faces extraction might fail due to illumination conditions. In fact, cube's surface may be different and it means that when the light falls on the surface it may be reprojected and create color distortions thus it brakes edge detection. Under the good illumination conditions corners of faces and pieces colors are extracted correctly.  From the figure \ref{fig:rubiks_cube_face_detected} it is clearly seen that this approach is applicable for the given problem.

\begin{figure} [ht]
    \begin{center}
        \includegraphics[width=150pt]{detected_face}
        \includegraphics[width=150pt]{extracted_face}
        \caption{Detected and extracted face sample with distorted colors.}
        \label{fig:rubiks_cube_face_detected}
    \end{center}
\end{figure}

\end{document}