\section{Training of the model}

This section will elaborate on the training implementaiton of a model used for this project. For the implementation python 2.7 and Jupyter is used. The implementation of the training section is from the github repository\footnote{\url{https://github.com/awjuliani/TF-Tutorials}}.

The training is started by initializing all the variables to a gaussian distribution and setting the batch size. The initialization is done to give the weights some ballast to train on, and so often in the beginning of the training the correction of the training loss will have a steep reduction. The batch size determines the amount of images which must go through the model, both forward and backward propagation, before the update parameter function ADAM is enabled. The batch size regress the model and it is a hyperparameter, but it can also cause a large amount of memory in usage. The batch size of 64 is used, with a total of 20000 steps. Each step consist of 64 images from one of the six folds of the cifar-10 dataset. The cifar-10 dataset is divided into five random folds for training and one fold for testing.
\begin{lstlisting}[language=Python, label=lst:trainingLoop, caption=for loop which finds the 64 training images for training the model]
init = tf.global_variables_initializer()
batch_size = 64
currentCifar = 1
total_steps = 20000
l = []
a = []
aT = []
saver = tf.train.Saver()
sess = tf.Session()
sess.run(init)
i = 0
draw = range(10000)
while i < total_steps:
	if i % (10000/batch_size) != 0:
		batch_index = np.random.choice(draw,size=batch_size,replace=False)
	else:
		draw = range(10000)
		if currentCifar == 5:
			currentCifar = 1
			print "Switched CIFAR set to " + str(currentCifar)
		else:
			currentCifar = currentCifar + 1
			print "Switched CIFAR set to " + str(currentCifar)
		cifar = unpickle('./cifar10/data_batch_'+str(currentCifar))
		batch_index = np.random.choice(draw,size=batch_size,replace=False)
	x = ConvertImages(cifar['data'][batch_index])
	y = np.reshape(np.array(cifar['labels'])[batch_index],[batch_size,1])
\end{lstlisting}

The images are forwarded through the network as shown in \ref{lst:Parameter_updating}, while the loss and accuracy are saved. Together with the current loss are the probabilities for each category being saved. The estimated label is compared to the true label of the image, and if the comparison is true the accuracy is updated.

For each tenth step the training accuracy is printed, while for each 100 step a small part of the test data is forwarded through the model. The test data does not update the model, but it only shows how well the model is progressing through the training. When the training is done, the model is saved for later analysis through saver.save.

\begin{lstlisting}[language=Python, label=lst:Parameter_updating, caption=Parameter updating: loss and accuracy are saved and they are tested every 100 steps]
	_,lossA,yP,LO = sess.run([update,loss,output,label_oh],feed_dict=
	{input_layer:x,label_layer:np.hstack(y)})
	accuracy = np.sum(np.equal(np.hstack(y),np.argmax(yP,1)))/float(len(y))
	l.append(lossA)
	a.append(accuracy)
	if i % 10 == 0: print "Step: " + str(i) + " Loss: " + str(lossA) +
	 " Accuracy: " + str(accuracy)
	if i % 100 == 0:
		point = np.random.randint(0,10000-500)
		xT = ConvertImages(cifarT['data'][point:point+500])
		yT = np.reshape(np.array(cifarT['labels'])[point:point+500],[500])
		lossT,yP = sess.run([loss,output],feed_dict=
		{input_layer:xT,label_layer:yT})
		accuracy = np.sum(np.equal(yT,np.argmax(yP,1)))/float(len(yT))
		aT.append(accuracy)
		print "Test set accuracy: " + str(accuracy)
	i+= 1
save_path = saver.save(sess, "./trainingmodels/RegularNet/
model_regularnet.ckpt")
print "Model saved in file: " + str(save_path)
sess.close()
\end{lstlisting}


\section{Low-Voltage Induced Random Bit Errors in Quantized DNN Weights}
\label{sec:errors}

We assume the quantized DNN weights to be stored  on multiple memory banks, \eg, SRAM in the case of on-chip scratchpads or DRAM for off-chip memory. As shown in \cite{GanapathyDAC2017,KimDATE2018,ChandramoorthyHPCA2019}, the probability of memory bit cell failures increases exponentially as operating voltage is scaled below $\Vmin$, \ie, the minimal voltage required for reliable operation, see \figref{fig:introduction}. This is done intentionally to reduce energy consumption, \eg, \cite{ChandramoorthyHPCA2019,KimDATE2018,KoppulaMICRO2019}, or adversarially by an attacker, \eg, \cite{TangUSENIX2017}. Process variation during fabrication causes a variation in the vulnerability of individual bit cells. As shown in \figref{fig:errors} (left), for a specific memory array, bit cell failures are typically approximately random and independent of each other \cite{GanapathyDAC2017}. We also consider chips showing other error patterns as in \figref{fig:errors} (right). Nevertheless, there is generally an ``inherited'' distribution of bit cell failures across voltages \cite{GanapathyHPCA2019}, if a bit error occurred at a given voltage, it is likely to occur at lower voltages, as made explicit in \figref{fig:errors}. However, across different SRAM arrays in a chip or different chips, the patterns or spatial distribution of bit errors is usually different and can be assumed random \cite{ChandramoorthyHPCA2019}. Throughout the paper, we use the following bit error model:

\textbf{Random Bit Error Model:}
\textit{The probability of a bit error is $p$ (in \%) for all weight values and bits. For a fixed memory array, bit errors are persistent across supply voltages, \ie, bit errors at probability $p'{\leq}p$ also occur at probability $p$. A bit error flips the currently stored bit. We denote random bit error injection by $\biterror_p$.}

\begin{figure*}
    \vspace*{-0.15cm}
    \centering
    \includegraphics[width=0.85\textwidth]{main_training_flow4.pdf}
    \vspace*{-6px}
    \caption{\textbf{Random Bit Error Training (\Random).} We illustrate the data-flow for \Random as in \algref{alg:training}. Here, $\biterror_p$ injects random bit errors in the \red{quantized weights} $\red{v^{(t)}} = Q(\magenta{w^{(t)}})$, resulting in $\red{\tilde{v}^{(t)}}$, while the forward pass is performed on the \blue{de-quantized perturbed weights} $\blue{\tilde{w}_q^{(t)}} = Q^{-1}(\red{\tilde{v}^{(t)}})$, \ie, fixed-point arithmetic is not emulated. The weight update during training is not affected by bit errors and computed in \magenta{floating point}.}
    \label{fig:flowchart}
    \vspace*{-0.2cm}
\end{figure*}

This error model realistically captures the nature of low-voltage induced bit errors, from both SRAM and DRAM as confirmed in \cite{ChandramoorthyHPCA2019,KimDATE2018,KoppulaMICRO2019}. However, our approach in \secref{sec:robustness} is model-agnostic: the error model can be refined if extensive memory characterization results are available for individual chips. For example, faulty bit cells with $1$-to-$0$ or $0$-to-$1$ flips might not be equally likely. Similarly, as in \cite{KoppulaMICRO2019}, bit errors might be biased towards alignment along rows or columns of the memory array. The latter case is illustrated in \figref{fig:errors} (right). However, estimating these specifics requires testing infrastructure and detailed characterization of individual chips.
More importantly, it introduces the risk of overfitting to few specific memories/chips.
Furthermore, we demonstrate that the robustness obtained using our uniform error model generalizes to bit error distributions with strong spatial biases as in \figref{fig:errors} (right).

We assume the quantized weights to be mapped linearly to the memory. This is the most direct approach and, in contrast to \cite{KoppulaMICRO2019}, does not require knowledge of the exact spatial distribution of bit errors. This also means that we do not map particularly vulnerable weights to more reliable memory cells, and therefore no changes to the hardware or the application are required. Thus, in practice, for $W$ weights and $m$ bits per weight value, we sample uniformly $u \sim U(0, 1)^{W \times m}$. Then, the $j$-th bit in the quantized weight $v_i = Q(w_i)$ is flipped iff $u_{ij} \leq p$.
Our model assumes that the flipped bits at lower probability $p' \leq p$ are a subset of the flipped bits at probability $p$ and that bit flips to $1$ and $0$ are equally likely. The noise pattern of random bit errors is illustrated in \figref{fig:quantization}:
for example a single bit flip in the most-significant bit (MSB) of the signed integer $v_i$ can result in a change of roughly half of the quantized range (also \cf \secref{subsec:robustness-quantization}).

\section{Towards Robustness Against Random Bit Errors}
\label{sec:robustness}

We address robustness against random bit errors in three steps: First, we analyze the impact of fixed-point quantization schemes on bit error robustness. This has been neglected both in prior work on low-voltage DNN accelerators \cite{KimDATE2018,KoppulaMICRO2019} and in work on quantization robustness \cite{MurthyARXIV2019,MerollaARXIV2016,SungARXIV2015}. This yields our \textbf{robust quantization} (\secref{subsec:robustness-quantization}). On top, we propose aggressive \textbf{weight clipping} as regularization during training (\secref{subsec:robustness-clipping}).
Weight clipping enforces a more uniformly distributed, \ie, redundant, weight distribution, improving robustness. We show that this is due to minimizing the cross-entropy loss, enforcing large logit differences.
Finally, in addition to robust quantization and weight clipping, we perform \textbf{random bit error training (\Random)} (\secref{subsec:robustness-training}): in contrast to the fixed bit error patterns in \cite{KimDATE2018,KoppulaMICRO2019}, we train on completely \emph{random} bit errors and, thus, generalize across chips and voltages.
Generalization is measured using \emph{average robust test error (\RTE)}, the test error after injecting bit errors, \wrt to our error model from \secref{sec:errors} as well as real, profiled bit error patterns. % but also the corresponding standard deviation.
Robustness against bit error rate $p$ has to induce robustness for $p' \leq p$ (\ie, higher voltage), as well.

\subsection{Robust Fixed-Point Quantization}
\label{subsec:robustness-quantization}

We consider quantization-aware training \cite{JacobCVPR2018,KrishnamoorthiARXIV2018} using a generic, deterministic fixed-point quantization scheme commonly used in DNN accelerators \cite{ChandramoorthyHPCA2019}. However, we focus on the impact of quantization schemes on robustness against random bit errors, mostly neglected so far \cite{MurthyARXIV2019,MerollaARXIV2016,SungARXIV2015}. We find that quantization affects robustness significantly, even if accuracy is largely unaffected.

\textbf{Fixed-Point Quantization:} Let $f(x; w)$ be a DNN taking an example $x \in [0, 1]^D$, \eg, an image, and weights $w \in \mathbb{R}^W$ as input. Quantization determines how weights are represented in memory, \eg, on SRAM. In a \emph{fixed-point quantization} scheme, $m$ bits allow to represent $2^m$ distinct values.
A weight $w_i \in [-\qmax, \qmax]$
is represented by a signed $m$-bit integer $v_i = Q(w_i)$ corresponding to the underlying bits. Here, $[-\qmax, \qmax]$ is the \emph{symmetric} quantization range and signed integers use two's complement representation. Then, $Q: [-\qmax, \qmax] \mapsto \{-2^{m - 1} - 1, \ldots, 2^{m - 1} - 1\}$ is defined as
\begin{align}
    Q(w_i) = \left\lfloor \frac{w_i}{\Delta}\right\rfloor,\text{  }
    Q^{-1}(v_i) = \Delta v_i,\text{  }
    \Delta = \frac{\qmax}{2^{m - 1} - 1}
    \label{eq:quantization}
\end{align}
Flipping the most significant bit (MSB, \ie, sign bit) leads to an absolute error of half the quantization range, \ie, $\qmax$ ({\color{yellow!75!black!}yellow} in \figref{fig:quantization}).
Flipping the least significant bit (LSB) incurs an error of $\Delta$, \cf \eqnref{eq:quantization}. Thus, the impact of bit errors ``scales with'' $\qmax$.

\textbf{Global and Per-Layer Quantization:} $\qmax$ can be chosen to accommodate all weights, \ie, $\qmax = \max_i |w_i|$. This is called \emph{global} quantization. However, it has become standard to apply quantization \textit{per-layer} allowing to adapt $\qmax$ to each layer. As in PyTorch \cite{PaszkeNIPSWORK2017}, we consider weights and biases of each layer separately. By reducing the quantization range for each layer individually, the errors incurred by bit flips are automatically minimized, \cf \figref{fig:quantization}. The
\textbf{per-layer, symmetric quantization is our default reference}, referred to as \Normal. However, it turns out that it is further beneficial to consider arbitrary quantization ranges $[\qmin, \qmax]$ (allowing $\qmin > 0$). In practice, we
first map $[\qmin, \qmax]$ to $[-1,1]$ and then quantize $[-1,1]$ using \eqnref{eq:quantization}.
Overall, per-layer asymmetric quantization has the finest granularity, \ie, lowest $\Delta$ and approximation error. Nevertheless it is not the most robust
quantization.

\begin{figure}[t]
	\centering
	\vspace*{-0.1cm}
	\hspace*{-0.3cm}
	\begin{subfigure}{0.24\textwidth}
		\vspace*{3px}

		\includegraphics[height=1.35cm]{c10_q81auunfp_nt_original_logits.pdf}
	\end{subfigure}
	\begin{subfigure}{0.12\textwidth}
		\vspace*{0px}

		\includegraphics[height=1.425cm]{c10_q81auunfp_nt_original_confidences.pdf}
	\end{subfigure}
	\begin{subfigure}{0.12\textwidth}
		\vspace*{0px}

		\includegraphics[height=1.425cm]{c10_q81auunfp_nt_original_weights.pdf}
	\end{subfigure}

	\hspace*{-0.4cm}
	\begin{subfigure}{0.24\textwidth}
		\vspace*{3px}

		\includegraphics[height=1.35cm]{c10_q801auunfp_nt_original_logits.pdf}
	\end{subfigure}
	\begin{subfigure}{0.12\textwidth}
		\vspace*{0px}

		\includegraphics[height=1.425cm]{c10_q801auunfp_nt_original_confidences.pdf}
	\end{subfigure}
	\begin{subfigure}{0.12\textwidth}
		\vspace*{3px}

		\includegraphics[height=1.425cm]{c10_q801auunfp_nt_original_weights.pdf}
	\end{subfigure}

	\hspace*{-0.3cm}
	{\color{black!25!white}\rule{0.5\textwidth}{0.5px}}

	\hspace*{-0.4cm}
	\begin{subfigure}{0.24\textwidth}
		\vspace*{0px}

		\includegraphics[height=1.825cm]{c10_q81auunrfp_sawt_bit_random_g001_pop1_logits.pdf}
	\end{subfigure}
	\begin{subfigure}{0.12\textwidth}
		\vspace*{0px}

		\includegraphics[height=1.7cm]{c10_q81auunrfp_sawt_bit_random_g001_pop1_confidences.pdf}
	\end{subfigure}
	\begin{subfigure}{0.12\textwidth}
		\vspace*{3px}

		\includegraphics[height=1.7cm]{c10_q81auunrfp_sawt_bit_random_g001_pop1_weights.pdf}
	\end{subfigure}

	\vspace*{-8px}
	\caption{\textbf{Effect of Weight Clipping.} On \CifarT, weight clipping constraints the weights (right), thereby implicitly limiting the possible range for logits (left, {\color{colorbrewer2}blue}).
	However, even for $\wmax = 0.1$ the DNN is able to produce high confidences (middle, {\color{colorbrewer2}blue}), suggesting that more weights are used to obtain these logits. Furthermore, the impact of random bit errors, $p = 1\%$, on the logits/confidences ({\color{colorbrewer1}red}) is reduced significantly. \Random (trained with $p = 1\%$, w/o weight clipping), increases the range of weights and is less effective at preserving logit/confidence distribution. %Logit/confidence histograms normalized over $10\cdot 128$ examples.
	}
	\label{fig:clipping}
	\vspace*{-0.2cm}
\end{figure}

\textbf{Robust Quantization:} Quantization as in \eqnref{eq:quantization} does \emph{not} provide optimal robustness against bit errors. First, the floor operation $\lfloor \nicefrac{w_i}{\Delta}\rfloor$ is commonly implemented as float-to-integer conversion. Using proper rounding $\lceil\nicefrac{w_i}{\Delta}\rfloor$ instead has negligible impact on accuracy, even though approximation error improves slightly. In stark contrast, bit error robustness is improved considerably. During training, DNNs can compensate the differences in approximation errors, even for small precision $m < 8$. However, at test time,
rounding decreases the impact of bit errors considerably. Second, \eqnref{eq:quantization} uses signed integers for symmetric quantization. For asymmetric quantization, with arbitrary $[\qmin, \qmax]$, we found quantization into \emph{unsigned} integers to improve robustness, \ie, $Q : [\qmin, \qmax] \mapsto \{{\color{red}0}, \ldots, {\color{red}2^m - 1}\}$. This is implemented using an additive term of $2^{m-1}-1$ in \eqnref{eq:quantization}. While accuracy is not affected, the effect of bit errors in the sign bit changes: in symmetric quantization, the sign bit mirrors the sign of the weight value. For asymmetric quantization, an unsigned integer representation is more meaningful. Overall, our \textbf{robust fixed-point quantization (\Quant)}  uses per-layer, asymmetric quantization into unsigned integers with rounding.
These seemingly \emph{small differences} have little to no impact on accuracy, while having tremendous impact on robustness against bit errors, see \secref{subsec:experiments-quantization} and \appref{sec:supp-implementation}.
\revision{They are simple to implement, do not add training complexity or hyper-parameters and demonstrate the importance of robustness in developing DNN quantization schemes.}

\begin{algorithm}[t]
\caption{\textbf{Random Bit Error Training (\Random).} The forward passes are performed using \blue{de-quantized weights (blue)}. Perturbed weights are obtained by injecting bit errors in the \red{quantized weights (in red)}. The update, averaging gradients from both forward passes, is performed in \magenta{floating-point (magenta)}. Also see \figref{fig:flowchart}.}
\label{alg:training}
\begin{algorithmic}[1]
\small
\Procedure{RandBET}{$p$}
    \State initialize \magenta{$w^{(0)}$}
	\For{$t = 0, \ldots, T - 1$}
    	\State sample batch $\{(x_b, y_b)\}_{b = 1}^B$
    	\State \{element-wise clipping:\}
    	\State $\magenta{w^{(t)}} = \min(\wmax, \max(-\wmax, \magenta{w^{(t)}}))$
    	\State \{quantization:\}
    	\State $\red{v^{(t)}} = Q(\magenta{w^{(t)}})$
        \State $\blue{w_q^{(t)}} = Q^{-1}(\red{v^{(t)}})$
        \State \{clean forward and backward pass:\}
        \State $\Delta^{(t)} = \nabla_w \sum_{b = 1}^B \mathcal{L}(f(x_b; \blue{w_q^{(t)}}), y_b)$
        \State \{\emph{perturbed} forward and backward pass:\}
        \State $\blue{\tilde{w}_q^{(t)}}{\hskip 1px=\hskip 1px}Q^{-1}(\biterror_p(\red{v^{(t)}}))$\label{line:attack} \{inject random bit errors\}
        \State $\tilde{\Delta}^{(t)} = \nabla_w \sum_{b = 1}^B \mathcal{L}(f(x_b; \blue{\tilde{w}_q^{(t)}}), y_b)$
        \State \{average gradients and weight update:\}
        \State $\magenta{w^{(t + 1)}} = \magenta{w^{(t)}} - \gamma(\Delta^{(t)} + \tilde{\Delta}^{(t)})$
    \EndFor
    \State \textbf{return} $\blue{w_q^{(T)}} = Q^{-1}(Q(\magenta{w^{(T)}}))$
\EndProcedure
\end{algorithmic}
\end{algorithm}

\subsection{Training with Weight Clipping as Regularization}
\label{subsec:robustness-clipping}

\textbf{Weight clipping} refers to constraining the weights to $[-\wmax, \wmax]$ \emph{during training}, where $\wmax$ is a hyper-parameter. Generally, $\wmax$ is independent of the quantization range(s) which always adapt(s) to the weight range(s) at hand. However, weight clipping limits the maximum possible quantization range (\cf \secref{subsec:robustness-quantization}), \ie, $\qmax \leq \wmax$.
It might seem that weight clipping with small $\wmax$ automatically improves robustness against bit errors as the absolute errors are reduced. However, the \emph{relative} errors are not influenced by rescaling. As the DNN's decision is usually invariant to rescaling, reducing the scale of the weights does not impact robustness. In fact, the mean relative error of the weights in \figref{fig:quantization} (right) increased with clipping at $\wmax = 0.1$. Thus, weight clipping does \emph{not} ``trivially'' improve robustness by reducing the scale of weights. Nevertheless, we found that weight clipping actually improves robustness considerably on top of our robust quantization.

The interplay of weight clipping and minimizing the the cross-entropy loss during training is the key. High confidences can only be achieved by large differences in the logits. Because the weights are limited to $[-\wmax, \wmax]$, large logits can only be achieved using more weights in each layer to produce larger outputs. This is illustrated in \figref{fig:clipping} (right): using $\wmax = 0.1$, the weights are (depending on the layer) up to $5$ times smaller. Considering deep NNs, the ``effective'' scale factor for the logits is significantly larger, scaling exponentially with the number of layers. Thus, using $\wmax = 0.1$ is a significant constraint on the DNNs ability to produce large logits. As result, weight clipping produces a much more uniform weight distribution.
\figref{fig:clipping} (left and middle) shows that a DNN constrained at $\wmax = 0.1$ can produce similar logit and confidence distributions (in {\color{colorbrewer2}blue}) as the unclipped DNN. At the same time, random bit errors, have a significantly smaller impact on the logits and confidences (in {\color{colorbrewer1}red}). \figref{fig:clipping} (right column) also shows the induced redundancy in the weight distribution. Weight clipping leads to more weights being utilized, \ie, less weights are zero (note log-scale, marked in {\color{colorbrewer1}red}, on the y-axis). Also, more weights reach large values, relative to the maximum absolute weight. Overall, we found weight clipping to be an easy-to-use but effective measure to improve weight robustness. We use \Clipping[$\wmax{=}0.1$] to refer to, \eg, weight clipping with $\wmax = 0.1$. For more evidence supporting our argumentation, see \tabref{tab:clipping-robustness}. For example, we show that DNNs loose robustness when using label smoothing, \ie, not enforcing high confidences/logits during training.
\revision{Finally, weight clipping is straight-forward to implement (\cf \algref{alg:training}, line 6) and adds negligible training cost. The additional hyper-parameter, \ie, $\wmax$, can easily be tuned based on constraints on clean (or robust) performance, \cf \secref{subsec:experiments-clipping}.}

\subsection{Random Bit Error Training (\Random)}
\label{subsec:robustness-training}

In \emph{addition to} weight clipping and robust quantization, we inject random bit errors with probability $p$ during training to further improve robustness. This results in the following learning problem, which we optimize as illustrated in \figref{fig:flowchart}:
\begin{align}
	\begin{split}
    	&\min_w \mathbb{E}[\mathcal{L}(f(x; \tilde{w}), y) + \mathcal{L}(f(x; w), y)]\\
    	\text{s.t.}&\quad v = Q(w),\, \tilde{v} = \biterror_p(v),\, \tilde{w} = Q^{-1}(\tilde{v}).
   	\end{split}\label{eq:random-training-average}
\end{align}
where $(x, y)$ are labeled examples, $\mathcal{L}$ is the cross-entropy loss and $v = Q(w)$ denotes the (element-wise) quantized weights $w$ which are to be learned. $\biterror_p(v)$ injects random bit errors with rate $p$ in $v$. Note that we consider both the loss on clean weights and weights with bit errors. This is desirable to avoid an increase in (clean) test error and stabilizes training compared to training only on bit errors in the weights. Note that bit error rate $p$ implies, in expectation, $pmW$ bit errors. Following \algref{alg:training}, we use stochastic gradient descent to optimize \eqnref{eq:random-training-average}, by performing the gradient computation using the perturbed weights $\tilde{w} = Q^{-1}(\tilde{v})$ with $\tilde{v} = \biterror_p(v)$, while applying the gradient update on the (floating-point) clean weights $w$. In spirit, this is similar to data augmentation, however, the perturbation is applied on the weights instead of the inputs. As we found that introducing bit errors right from the start may prevent the DNN from converging, we apply bit errors as soon as the (clean) cross-entropy loss is below $1.75$. Interestingly, weight clipping and \Random have somewhat orthogonal effects, which allows to combine them easily in practice: While weight clipping encourages redundancy in weights by constraining them to $[-\wmax,\wmax]$, % and activations,
\Random (w/o weight clipping) causes the DNN to have larger tails in the weight distribution, as shown in \figref{fig:clipping} (bottom). However, considering logits and confidences, especially with random bit errors (in {\color{colorbrewer1}red}), \Random alone performs slightly worse than \Clipping[$0.1$]. Thus, \Random becomes particularly effective when combined with weight clipping, as we make explicit using the notation \Random[$\wmax$] in \algref{alg:training}.
\revision{While \Random increases training complexity (\cf \algref{alg:training}), inference is \emph{not} affected. This is in stark contrast to hardware- or redundancy-based bit error mitigation strategies which usually impact inference time and energy consumption. We also note that the additional hyper-parameter, \ie, $p$, is easily chosen according to the target bit error rate.}


\section{Scheduling}

\paragraph{Motivation}
\begin{itemize}
  \item \( K \) jobs ready to run, \( K > N \geq 1 \) CPUs available
  \item \textbf{Scheduling Problem}:
  \begin{itemize}
    \item Which jobs should kernel assign to which CPUs?
    \item When should it make decision?
  \end{itemize}
\end{itemize}

\paragraph{Dispatcher}
\begin{itemize}
  \item \textbf{Dispatcher}: performs actual process switch
  \begin{itemize}
    \item mechanism
    \item save/restore process context
    \item switch to user mode
  \end{itemize}
  \item \textbf{Scheduler}: selects next process to run based on \emph{policy}
\end{itemize}

\paragraph{Voluntary Yielding vs. Preemption}
\begin{itemize}
  \item kernel responsible for CPU switch
  \item kernel doesn't always run \( \to \) can only dispatch different process when invoked
  \item \textbf{cooperative multitasking}: running process performs \emph{yield} syscall
  \begin{itemize}
    \item[$ \to $] kernel switches process
  \end{itemize}
  \item \textbf{preemptive scheduling}:
  \begin{itemize}
    \item kernel invoked in certain time intervals
    \item kernel makes scheduling decisions after every time-slice
  \end{itemize}
\end{itemize}
\begin{figure}[H]\centering\label{ProcessSwitching}\includegraphics[width=0.33\textwidth]{ProcessSwitching}\end{figure}

\paragraph{Scheduling --- Process States}
\begin{itemize}
  \item \textbf{new}: process was created but did not run yet
  \item \textbf{running}: instructions are currently being executed
  \item \textbf{waiting}: process is waiting for some event
  \item \textbf{ready}: process is waiting to by assigned a processor
  \item \textbf{terminated}: process has finished execution
\end{itemize}

\paragraph{Scheduling --- long-term vs. short-term}
\begin{itemize}
  \item \textbf{Short-term scheduler} (CPU Scheduler, focused on in this lecture):
  \begin{itemize}
    \item selects process to run next, allocates CPU
    \item invoked frequently (ms) \( \leadsto \) must be fast
  \end{itemize}
  \item \textbf{Long-term scheduler} (job scheduler):
  \begin{itemize}
    \item selects process to be brought into ready queue
    \item invoked very infrequently (s, m) \( \leadsto \) can be slow
    \item controls degree of \emph{multiprogramming}
  \end{itemize}
\end{itemize}

\paragraph{Scheduling queues}
\begin{itemize}
  \item \textbf{job queue}: set of all processes in system
  \item \textbf{ready queue}: process in main memory, ready or waiting
  \item \textbf{device queue}: processes waiting for I/O device
\end{itemize}

\paragraph{Scheduling Policies --- Categories}
\begin{itemize}
  \item \textbf{batch scheduling}:
  \begin{itemize}
    \item still widespread in business (payroll, inventory,\dots)
    \item no users waiting for quick response
    \item non-preemptive algorithms acceptable \( \to \) less switches \( \to \) less overhead
  \end{itemize}
  \item \textbf{interactive scheduling}:
  \begin{itemize}
    \item need to optimize for response time
    \item preemption essential to keep processes from hogging CPU
  \end{itemize}
  \item \textbf{real-time scheduling}:
  \begin{itemize}
    \item guarantee job completion within time constraints
    \item need to be able to plan when which process runs + how long
    \item preemption not always needed
  \end{itemize}
\end{itemize}

\paragraph{Scheduling Policies --- Goals}
\begin{itemize}
  \item \textbf{General}:
  \begin{itemize}
    \item \emph{fairness}: give each process fair share of CPU
    \item \emph{balance}: keep all parts of system busy
  \end{itemize}
  \item \textbf{batch scheduling}:
  \begin{itemize}
    \item \emph{throughput}: number of processes that complete per time unit
    \item \emph{turnaround time}: time from job submission to job completion
    \item \emph{CPU utilization}: keep CPU as busy as possible
  \end{itemize}
  \item \textbf{interactive scheduling}:
  \begin{itemize}
    \item \emph{waiting time}: reduce time a process waits in waiting queue
    \item \emph{response time}: time from request to first response
  \end{itemize}
  \item \textbf{real-time scheduling}:
  \begin{itemize}
    \item \emph{meeting deadlines}: finishing jobs in time
    \item \emph{predictability}: minimize jitter
  \end{itemize}
\end{itemize}

\paragraph{Scheduling Policies --- first come first served}
\begin{itemize}
  \item intuitively clear
  \item \textbf{Example}: 3 processes arrive at time 0 in the order \( P_1, P_2, P_3 \)
    \begin{center}
      \begin{tabular}{|c|c|c|}
        \textbf{Process} & \textbf{Burst time} & \textbf{Turnaround time} \\
        \hline
        \( P_1 \) & 24 & 24 \\
        \( P_2 \) & 3 & 27 \\
        \( P_3 \) & 3 & 30
      \end{tabular}
    \end{center}
  \item \( \leadsto \) average turnaround time 27 \( \to \) \emph{can we do better?}
  \item \textbf{Conclusion}: if processes would arrive in order \( P_2 \), \( P_3 \), \( P_1 \), average turnaround time would be 13 \\*
    \( \leadsto \) \emph{good scheduling can reduce turnaround time}
\end{itemize}

\paragraph{Scheduling Policies --- shortest job first}
\begin{itemize}
  \item \textbf{Benefits}: optimal average turnaround/waiting/response time
  \item \textbf{Challenge}: cannot know job lengths in advance
  \item \textbf{Solution}: predict length of next CPU burst for each process \\*
    \( \leadsto \) schedule process with shortest burst next
  \item \textbf{Burst Estimation}: \emph{exponential averaging} \\*
    - \( \tau_{n+1} = \alpha t_n + (1-\alpha)\tau_n \) \\* \phantom{-} (\( t_n \): actual length of \( n \)-th CPU burst, \( \tau_{n+1} \): predicted length of next CPU burst, \( 0 \leq \alpha \leq 1 \))
\end{itemize}

\paragraph{Process Behavior --- CPU bursts}
\begin{itemize}
  \item CPU bursts exists because processes wait for I/O
  \item \textbf{CPU-bound processes}: spends more time doing computations \\*
    \( \leadsto \) few very long CPU bursts
  \item \textbf{I/O-bound processes}: spends more time doing I/O \\*
    \( \leadsto \) many short CPU bursts
\end{itemize}

\paragraph{Scheduling Policies --- preemptive shortest-job-first}
\begin{itemize}
  \item SJF optimizes waiting/response time \\*
    \( \leadsto \) what about throughput?
  \item \textbf{Problem}: CPU-bound jobs hold CPU until exit or I/O \( \to \) \emph{poor I/O utilization}
  \item \textbf{Idea}: SJF, but preempt periodically to make new scheduling decision
  \begin{itemize}
    \item each time slice: schedule job with shortest remaining time next
    \item alternatively: schedule job with shortest next CPU burst
  \end{itemize}
\end{itemize}

\paragraph{Scheduling Policies --- round robin}
\begin{itemize}
  \item \textbf{Problem}: batch schedulers suffer from starvation and don't provide fairness
  \item \textbf{Idea}: each process runs for small CPU time unit
  \begin{itemize}
    \item \emph{time quantum}/\emph{time slice} length: usually 10-100ms
    \item preempt processes that have not blocked by end of time slice
    \item append current thread to end of run queue, run next thread
  \end{itemize}
  \item \textbf{Caution}: time slice length needs to balance interactivity and overhead!
  \begin{itemize}
    \item[$ \to $] if time slice length in the area of dispatch time, 50\% of CPU time wasted for process switching
  \end{itemize}
\end{itemize}

\paragraph{Scheduling Policies --- virtual round robin}
\begin{itemize}
  \item \textbf{Problem}: RR is unfair for I/O-bound jobs: they block before using up time quantum
  \item \textbf{Idea}: put jobs that didn't use up their quantum in additional queue
  \begin{itemize}
    \item store share of unused time-slice
    \item give those jobs additional queue priority
    \item put them back into normal queue afterwards
  \end{itemize}
\end{itemize}

\paragraph{Scheduling Policies --- (strict) priority scheduling}
\begin{itemize}
  \item \textbf{Problem}: not all jobs are equally important \\*
    \( \leadsto \) different priorities (e.g., 4)
  \item \textbf{Solution}: associate priority number with each process
  \begin{itemize}
    \item RR for each priority
    \item \emph{aging}: increase priority of low priority processes over time
  \end{itemize}
\end{itemize}

\paragraph{Scheduling Policies --- multi-level feedback queue}
\begin{itemize}
  \item \textbf{Problem}: context switching expensive \\*
    \( \leadsto \) trade-off between interactivity and overhead?
  \item \textbf{Goals}:
  \begin{itemize}
    \item higher priority for I/O jobs (usually don't use up quantum)
    \item low priority for CPU jobs (rather run them longer)
  \end{itemize}
  \item \textbf{Idea}: different queues with different priorities and time slice lengths
  \begin{itemize}
    \item schedule queues with (static) priority scheduling
    \item double time slice length in each next-lower priority
    \item process to higher priority when they don't use up quantum repetitively
    \item process to lower priority when they use up quantum repetitively
  \end{itemize}
\end{itemize}

\paragraph{Scheduling Principles --- priority donation}
\begin{itemize}
  \item \textbf{Problem}: Process B (higher priority) waits for process A (lower priority) \\*
    \( \leadsto \) B has now effectively lower priority
  \item \textbf{Solution}: \emph{priority donation}
  \begin{itemize}
    \item give A priority of B as long as B waits for A
    \item if C, D, E wait for B \( \to \) A gets highest priority of B, C, D, E
  \end{itemize}
\end{itemize}

\paragraph{Scheduling Policies --- lottery scheduling}
\begin{itemize}
  \item issue number of lottery tickets to processes (amount depending on priority)
  \item amount of tickets controls average proportion of CPU for each process
  \item \textbf{Scheduling}: scheduler draws random number \( N \), process with \( N \)-th ticket is executed
  \item processes can transfer tickets to other processes if they wait for them
\end{itemize}

\begin{summary}
  \begin{itemize}
    \item \textbf{phases}: processes have phases of communication and waiting for I/O
    \begin{itemize}
      \item[$ \to $] appropriate switching between processes increases computing system utilization
    \end{itemize}
    \item \textbf{goal-based}: scheduler decides what appropriate means based on goals
    \begin{itemize}
      \item \emph{long-term scheduler}: degree of multiprogramming
      \item \emph{short-term scheduler}: which process to run next
    \end{itemize}
    \item \textbf{dispatching}: only happens when OS is invoked
    \begin{itemize}
      \item \emph{cooperative scheduling}: currently running thread yields (syscall)
      \item \emph{preemptive scheduling}: OS is called periodically to switch threads
    \end{itemize}
  \end{itemize}
\end{summary}


%!TEX TS-program = xelatex
%!TEX encoding = UTF-8 Unicode

\documentclass[12pt,letterpaper]{article}
\usepackage{fontspec}
\usepackage{geometry}
\geometry{letterpaper, textwidth=6.5in, textheight=9.5in, marginparsep=20pt, marginparwidth=2in}
\setlength\parindent{0in}
\usepackage[usenames,dvipsnames]{color}

% math packages
\usepackage{amsmath}
\usepackage{arevmath}
\usepackage{enumerate}

% for sans serif math
%\usepackage{cmbright}

\usepackage{cancel}
% alp's math bold command: Use it like \mb{\lambda} or \mb{X}.
\DeclareRobustCommand{\mb}[1]{\ensuremath{\boldsymbol{\mathbf{#1}}}}

% cond. independence
\newcommand\independent{\protect\mathpalette{\protect\independenT}{\perp}}
\def\independenT#1#2{\mathrel{\rlap{$#1#2$}\mkern2mu{#1#2}}}

\usepackage{xunicode}
\usepackage{xltxtra}
\defaultfontfeatures{Mapping=tex-text,Ligatures=Common, Scale=1}
%\setromanfont [Ligatures={Common}, Numbers={OldStyle}, Variant=01]{Hoefler Text Roman}
%\setromanfont{Whitney HTF Light}
%\setmainfont{Whitney HTF}
\setmainfont[
 BoldFont={Whitney HTF Medium},
 ItalicFont={Whitney HTF Light Italic},
 BoldItalicFont={Whitney HTF Medium Italic}
 ]{Whitney HTF}
\usepackage[]{hyperref}
\usepackage[
    style=numeric-comp,
    sorting=none,
    doi=false,
    isbn=false,
    url=true,
    eprint=false,
    natbib=true,
    maxnames=99
]{biblatex}
\addbibresource{Untitled.bib}

\usepackage{dsfont}

%% Commands
\newcommand{\be}{\begin{equation*}}
\newcommand{\ee}{\end{equation*}}
\newcommand{\ba}{\begin{align*}}
\newcommand{\ea}{\end{align*}}
\newcommand{\indicator}{\mathds{1}}

\title{User-artist-song Model (draft)}
\author{Jaan Altosaar, Laurent Charlin, David Blei}

\begin{document}

\makeatletter

\def\maketitle{%
\begin{centering}
\textbf{\large\@title}\\%
\@author\\%
{\today}\\
\end{centering}
\par}

\makeatother
\maketitle
\vspace{10mm}

\section{Motivation}

Listening to music is different than reading articles. The artist of a song is of higher importance than the author of a scientific paper for a given user perusing their library.

We aim to capture this intuition using latent variables representing artist-level topics $\beta_a$, song-level topics $\beta_s$, and user preferences $x_u$ as shown in Figure \ref{fig:uasmodel}.  This model can be extended, for example by connecting the artist-level $\beta_a$ node to the song-level $\beta_s$ node, or the artist-level rating $r_{ua}$ to the song-level rating $r_{uai}$.

\input{./src/uasmodelfig}
\clearpage
\section{Generative Process}

The generative process for the observed ratings is partially inspired by content-based Poisson factorization \parencite{Gopalana}:

\begin{enumerate}
\item Draw user preferences $x_u \sim \textrm{\bf Gamma}(b,c)$
\item Draw artist-level topics $\beta_a \sim \textrm{\bf Gamma}(d,e)$
\item Draw song-level topics $\beta_i \sim \textrm{\bf Gamma}(f,g)$
\item Draw user-artist rating $r_{ua} \sim \textrm{\bf Poisson}(x_u^T \beta_a)$
\item Draw user-song rating $r_{uai} \sim \textrm{\bf Poisson}(x_u^T(\beta_a + \beta_i))$
\end{enumerate}

\section{Inference}

Structured Stochastic Variational Inference (SSVI) was recently introduced \parencite{Hoffman2014} as a method to add some dependencies between global and local parameters (as opposed to mean-field inference where there are no such dependencies). Here we derive SSVI first for Poisson factorization \citep{Gopalan2013}, and then for the user-artist-song model (Figure \ref{fig:uasmodel}).

\section{Experiments}

SSVI was used to fit Poisson factorization to the million song data set \citep{Liang2014}. The precision was 8.5\% (compared to 5.2\% for content-based Poisson factorization from last semester).

\printbibliography

\end{document}

%%% Local Variables:
%%% mode: tex-pdf
%%% TeX-master: t
%%% End:


\section{Fault Tree Model}
\label{sec:FTModel}

This model is designed to read the structure of the fault tree (FT) from the file and to import such Boolean logic structures as a RAVEN model.
The FT must be specified in the OpenPSA format (\href{<url>}{https://github.com/open-psa}).
As an example, the FT of Figure~\ref{fig:FT} is translated in the OpenPSA as shown below:

\begin{lstlisting}[style=XML,morekeywords={anAttribute},caption=FT in OpenPSA format., label=lst:FTModel]
<opsa-mef>
    <define-fault-tree name="FT">
        <define-gate name="TOP">
            <or>
                <gate name="G1"/>
                <gate name="G2"/>
                <gate name="G3"/>
            </or>
        </define-gate>
        <define-component name="A">
            <define-gate name="G1">
                <and>
                    <basic-event name="BE1"/>
                    <basic-event name="BE2"/>
                </and>
            </define-gate>
            <define-gate name="G2">
                <and>
                    <basic-event name="BE1"/>
                    <basic-event name="BE3"/>
                </and>
            </define-gate>
            <define-basic-event name="BE1">
                <float value="1.2e-3"/>
            </define-basic-event>
            <define-component name="B">
                <define-basic-event name="BE2">
                    <float value="2.4e-3"/>
                </define-basic-event>
                <define-basic-event name="BE3">
                    <float value="5.2e-3"/>
                </define-basic-event>
            </define-component>
        </define-component>
        <define-component name="C">
            <define-gate name="G3">
                <and>
                    <basic-event name="BE1"/>
                    <basic-event name="BE4"/>
                </and>
            </define-gate>
            <define-basic-event name="BE4">
                <float value="1.6e-3"/>
            </define-basic-event>
        </define-component>
    </define-fault-tree>
</opsa-mef>
\end{lstlisting}

\begin{lstlisting}[style=XML,morekeywords={anAttribute},caption=FT model input example., label=lst:FT_InputExample]
  <Models>
    ...
    <ExternalModel name="FT" subType="FTModel">
      <variables>
        statusBE1,statusBE2,statusBE3,statusBE4,TOP
      </variables>
      <topEvents>TOP</topEvents>
      <map var="statusBE1">BE1</map>
      <map var="statusBE2">BE2</map>
      <map var="statusBE3">BE3</map>
      <map var="statusBE4">BE4</map>
    </ExternalModel>
    ...
  </Models>
\end{lstlisting}

\begin{figure}
    \centering
    \centerline{\includegraphics[scale=0.5]{FT.pdf}}
    \caption{Example of FT.}
    \label{fig:FT}
\end{figure}

The FT of Figure~\ref{fig:FT} described in Listing~\ref{lst:FTModel} can be defined in the RAVEN input file as
shown in Listing~\ref{lst:FT_InputExample}

All the specifications of the FT model are given in the
\xmlNode{ExternalModel} block.
Inside the \xmlNode{ExternalModel} block, the XML
nodes that belong to this models are:
\begin{itemize}
  \item  \xmlNode{variables}, \xmlDesc{string, required parameter}, a list containing the names of both the input and output variables of the model
  \item  \xmlNode{topEvents}, \xmlDesc{string, required parameter}, the name of the alias Top Event
  \item  \xmlNode{map}, \xmlDesc{string, required parameter}, the name ID of the FT basic events
	  \begin{itemize}
	    \item \xmlAttr{var}, \xmlDesc{required string attribute}, the ALIAS name ID of the FT basic events
	  \end{itemize}
\end{itemize}

Provided this definition and the FT model of Figure~\ref{fig:FT} described in Listing~\ref{lst:FT_InputExample},
the resulting model in RAVEN is characterized by these variables:
\begin{itemize}
	\item Input variables: statusBE1, statusBE2, statusBE3, statusBE4
	\item Output variable: TOP
\end{itemize}

\subsection{FT Model Reference Tests}
\begin{itemize}
	\item SR2ML/tests/test\_FTmodel.xml
	\item SR2ML/tests/test\_FTmodel\_TD.xml
\end{itemize}


%! Author = tstreule

\section{Amperometric Sensors}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Electrochemistry}
%
\formbox{Overpotential}{\eta \equiv \Delta\phi\ped{appl} {-} \Delta\phi^0 \equiv \phi_s {-} \phi_m \equiv E {-} E^0}
{\hfill\scriptsize $E^0$: Nernst}%
\formula{~}{\eta = 0}
\quad (\textbf{equilibrium} i.e. no net current)

\formula{\textbf{Faraday's law}}{
    \textcolor{gray}{
        \textit{1st: } n\propto Q \textit{,\quad 2nd: } \textrm{(equiv. weigth) } W\ped{eq} = M/z
    }
}
\formbox{\textbf{of electrolysis}}{m = \frac{Q}{z\;F}\;M = \frac{I\;t}{z\;F}\;M = n\cdot M}
\enskip \textcolor{gray}{\scriptsize $[m] {=} \unit{g}$, $[n] {=} \unit{mol}$}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Butler-Volmer equation \textnormal{Effect of $\eta$ on barrier heigth $G$}}
\label{sec:butler-volmer}
%
\begin{tabular}{r@{:\quad}l}
    Let
    $\phi_s$	& potential of ions in solution\\
    $\phi_m$	& potential of $e^-$ in (metallic) electrode\\
    $z$			& valency of oxidized species\\
    $n$			& \#of transferred $e^-$\\
    $z-n$		& valency of reduced species
\end{tabular}
\formula{\textbf{transfer coeff. $\alpha$}}{nF\eta = n(1-\alpha)F\eta+n\alpha F\eta}
%		\formula{@equilibrium ($\eta=0$)}{k_0 \equiv k\ped{red}=k\ped{ox}}
\scalebox{.9}{%
    \formbox{\fbox{$f{\equiv}\frac{F}{RT} {\overset{@\unit[298]{K}}{=}} \frac{1}{\SI{25.69}{\milli\volt}}$}}{%
        k\ped{red} {=} k_0\;\eu^{-n\alpha f\eta}, \enskip
        k\ped{ox}  {=} k_0\;\eu^{n(1-\alpha)f\eta}
    }	\shortstack[l]{
        @equi. ($\eta=0$)\\
        $k_0 {\equiv} k\ped{red} {=} k\ped{ox}$
    }
}

The Butler-Vomer eq. relates what we measure (the \textbf{current}) with what we would like to determine (the \textbf{concentration} of an analyte):
\scalebox{.9}{%
    \formbox{\textbf{Butler-Volmer}}{j = \underbrace{nFAk_0\vspace{-1mm}}_{\vspace{-2mm}j_0} \big( C\ped{ox}(0,t)\;\eu^{-\alpha nf\eta} - C\ped{red}(0,t)\;\eu^{(1-\alpha)nf\eta} \big)}}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Cyclic Voltammetry}
%
\begin{minipage}{.4\columnwidth}
    \includegraphics[width=.9\columnwidth]{Electrochemistry_Cyclic_Voltammetry}
\end{minipage}%
\begin{minipage}{.6\columnwidth}
    Offers information on the mechanism of the ec reactions occuring at an electrode.
    \begin{itemize}
        \item May be irreversible\\
        $\to$ \#upper peaks $\neq$ \#lower peaks\\
        $\to$ irreversible reaction fast
        \item Sweep rate\\
        $\to$ irreversible reaction slow
    \end{itemize}
\end{minipage}
\formtex{\textbf{Water electrolysis}}{High potentials/volt's}
%$\to$ affects $I$ %(disturbance, bad)
\formtex{~}{$\to$ affects $I$ (disturbance, bad)}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Amperometric Sensors}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Clark \textnormal{(Oxygen)} Electrode \hfill\textnormal{$\to$ Measure \ce{O2}}}
%
\begin{itemize}
    \item test solution \ce{->[O2 passes][membrane]} \ce{Pt} cathode \ce{->[reduction]} current
    \item \ce{Ag} anode is in \ce{KCl} solution ($\to$ ``enough'' \ce{Cl-} for oxidation)
\end{itemize}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{1st and 2nd Generation}
%
$	\textrm{analyte of interest}
\;\ldots\;
\underbrace{\textrm{redox enzyme}\vspace{-1.5mm}}_{\vspace{-2.5mm}\textrm{catalize}}
\;\underbrace{\ldots\vspace{-.5mm}}_{\vspace{-2.5mm}\textrm{P1}}\;
\underbrace{\textrm{mediator}\vspace{-1mm}}_{\vspace{-2.5mm}\textrm{P2}}
\;\ldots\;
\textrm{Clark Electrode}
$
\formtex{problem P1 (1st)}{\ce{O2} may be consumed, when not measured}
\formtex{problem P2 (2nd)}{need to have ``enough'' of them}
\formtex{Mediator has to be}{$\bullet$ reversible \quad $\bullet$ not toxic \quad $\bullet$ no side reactions}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{3rd Generation}
%
\textit{Immobilization}/fixation of a redox enzyme on electrode surface

$\to$ free-diffusing redox mediators are not necessary\\
$\to$ in vivo measurments allowed since immobilized

\formtex{problem P3}{Efficient electron transfer (Marcus theory)}\vspace{-1mm}
\formtex{~}{$\to$ may be overcome by mediators}\vspace{-1mm}
\formtex{~}{$\to$ minimize ET distance}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Three Electrode Cell}
%
\begin{minipage}{.5\columnwidth}
    \includegraphics[width=\columnwidth]{Electrochemistry_Three_Electrode_Cell}
\end{minipage}%
\hspace{1cm}
\begin{minipage}{.5\columnwidth-1cm}
    $\frac{U\ped{REF}}{R_2} = - \frac{U\ped{in}}{R_1}$
    \par
    $I = \frac{U\ped{out}}{R_3}$
\end{minipage}


\documentclass{article}

\usepackage{times}
\usepackage{graphicx} % more modern
\usepackage{natbib}
\usepackage{algorithm, algorithmic}
\usepackage{hyperref}
\usepackage{amssymb, mathtools}
\usepackage{xcolor,colortbl}
\newcommand{\theHalgorithm}{\arabic{algorithm}}
\usepackage{subcaption} % for "subfigure" environment
\usepackage{geometry}

\usepackage[accepted]{icml2017}

% The \icmltitle you define below is probably too long as a header.
% Therefore, a short form for the running title is supplied here:
\icmltitlerunning{Globally Induced Forest}

% ============================== CONFIG =================================== %
\graphicspath{{./images/}}
\captionsetup[figure]{size=footnotesize}
% ============================== COLORS =================================== %
\definecolor{orange}{HTML}{FFA500}
\definecolor{dodgerblue}{HTML}{1E90FF}
\definecolor{deepgreen}{HTML}{0AF191}
\definecolor{purplish}{HTML}{E21173}

% ============================== COMMANDS =================================== %
\DeclareMathOperator*{\argmin}{arg\,min}
\DeclareMathOperator*{\argmax}{arg\,max}

\newcommand{\best}{\cellcolor{lightgray}}
\newcommand{\bestA}{\cellcolor{orange}}
\newcommand{\bestB}{\cellcolor{dodgerblue}}

\begin{document}
\twocolumn[
\icmltitle{Globally Induced Forest: A Prepruning Compression
Scheme\\Supplementary material}

% It is OKAY to include author information, even for blind
% submissions: the style file will automatically remove it for you
% unless you've provided the [accepted] option to the icml2017
% package.

% list of affiliations. the first argument should be a (short)
% identifier you will use later to specify author affiliations
% Academic affiliations should list Department, University, City, Region,
%Country
% Industry affiliations should list Company, City, Region, Country

% you can specify symbols, otherwise they are numbered in order
% ideally, you should not use this facility. affiliations will be numbered
% in order of appearance and this is the preferred way.
\icmlsetsymbol{equal}{*}

\begin{icmlauthorlist}
\icmlauthor{Jean-Michel Begon}{ulg}
\icmlauthor{Arnaud Joly}{ulg}
\icmlauthor{Pierre Geurts}{ulg}
\end{icmlauthorlist}

\icmlaffiliation{ulg}{Department of Electrical Engineering and Computer Science
University of Li\`ege, Li\`ege, Belgium}

\icmlcorrespondingauthor{Jean-Michel Begon}{jm.begon@ulg.ac.be}
%\icmlcorrespondingauthor{Arnaud Joly}{a.joly@ulg.ac.be}
\icmlcorrespondingauthor{Pierre Geurts}{p.geurts@ulg.ac.be}

% You may provide any keywords that you
% find helpful for describing your paper; these are used to populate
% the "keywords" metadata in the PDF but will not be shown in the document

\icmlkeywords{Decision tree, Random forest, Extremely randomized trees,
pruning, node budget, memory constraint, compression, growing algorithm, greedy
selection}

\vskip 0.3in
]

\printAffiliationsAndNotice{}

\section{GIF algorithm}
Figure \ref{fig:gif-algo} illustrates visually the inner loop of the GIF
building algorithm: a subset of the candidates nodes is chosen uniformely at
random. The contribution of each node is evaluated and the one which reduces
the error the most is added to the model. Its children are then built and added
to the candidate list.

\begin{figure*}[ht]
  \begin{subfigure}[b]{0.5\linewidth}
    \centering
    \includegraphics[height=0.25\textheight]{gif_algo1}
    \caption{Current forest at time $t$}
    \label{fig:gif-algo1}
  \end{subfigure}
  \hspace{\fill}  %% maximize space between adjacent subfigures
  \begin{subfigure}[b]{0.5\linewidth}
    \centering
    \includegraphics[height=0.25\textheight]{gif_algo2}
    \caption{A subset of candidates $C_t$  is drawn uniformely at random from
    the set of candidates $C$ (step 8)}
    \label{fig:gif-algo2}
  \end{subfigure}

  \vspace{4ex}  %% extra vertical space
  \begin{subfigure}[b]{0.5\linewidth}
    \centering
    \includegraphics[height=0.25\textheight]{gif_algo3}
    \caption{The error reduction is computed for all candidates of $C_t$ (step
    9)}
    \label{fig:gif-algo3}
  \end{subfigure}
  \hspace{\fill}
  \begin{subfigure}[b]{0.5\linewidth}
    \centering
    \includegraphics[height=0.25\textheight]{gif_algo4}
    \caption{The best node (highest error reduction) is selected (step 9)}
    \label{fig:gif-algo4}
  \end{subfigure}

    \vspace{4ex}
  \begin{subfigure}[b]{0.5\linewidth}
    \centering
    \includegraphics[height=0.25\textheight]{gif_algo5}
    \caption{The chosen node is introduced in the model (step 10)}
    \label{fig:gif-algo5}
  \end{subfigure}
  \hspace{\fill}
  \begin{subfigure}[b]{0.5\linewidth}
    \centering
    \includegraphics[height=0.25\textheight]{gif_algo6}
    \caption{The children of the chosen node are computed (step 11) and added
    to the candidate list (step 12)}
    \label{fig:gif-algo6}
  \end{subfigure}

\caption{Illustration of the GIF regression building algorithm ($T=3$, $CW=3$)}
\label{fig:gif-algo}
\end{figure*}

\section{Optimization problem}

We are building an additive model by inserting progressively nodes in the
forest.
At time $t$, we are trying to find the best node $j^{(t)}$ from the candidate
list $C_t$ and its associated optimal weight $w_j^{(t)}$:
\begin{align}
j^{(t)},w_j^{(t)} =\argmin_{j\in C_t, w\in \mathbb{R}^K} \sum_{i=1}^{N} L
\left(y_i,
\hat{y}^{(t-1)}(x_i) + w z_j(x_i) \right)
\end{align}
where $(x_i, y_i)_{i=1}^N$ is the learning sample, $\hat{y}^{(t-1)}()$ is the
model at time $t-1$, $z_j()$ is the node indicator functions, meaning that it
is $1$ if its argument reaches node $j$ and $0$ otherwise.

This problem is solved in two steps. First a node $j$ is selected from $C_t$
and the corresponding optimal weight, alongside the error reduction, are
computed. This is repeated for all nodes and the one achieving the best
improvement is selected.

\paragraph{Regression}
For regression, we used the L2-norm:
\begin{align}
w_j^{(t)} = \argmin_{w\in \mathbb{R}} \sum_{i=1}^{N} L \left(y_i,
\hat{y}^{(t-1)}(x_i) + w z_j(x_i) \right)^2
\end{align}
and the solution is given by
\begin{align}\label{eq:L2Solution}
w_j^{(t)} = \frac{1}{|Z_j|} \sum_{i \in Z_j} r_i^{(t-1)}
\end{align}
where $r_i^{(t-1)} = y_i - \hat{y}^{(t-1)}(x_i)$ is the residual at time $t-1$
for the $i$th training instance and $Z_j = \{1 \leq i \leq N | z_j(x_i)=1\}$ is
the subset of instances reaching node $j$.

\paragraph{Classification}
For classification we used the multi-exponential loss
\cite{zhu2009multiadaboost}. First, we need to encode the labels so that
\begin{align}\label{eq:MEencode}
y_i^{(k)} = \begin{cases}
1, &\text{ if the class of } y_i \text{ is } k \\
-\frac{1}{K-1}, &\text{otherwise}
\end{cases}
\end{align}
where $K$ is the number of classes. Notice that $\sum_{k=1}^{K} y_i^{(k)} = 0$.
The optimization then becomes
\begin{align}\label{eq:MEmin}
w_j^{(t)} &=  \argmin_{w \in \mathbb{R}^K} \sum_{i=1}^N \exp
\left(\frac{-1}{K} y_i^T \left(\hat{y}^{(t-1)}(x_i) + w z_j(x_i) \right)\right)
\\
&= \argmin_{w \in \mathbb{R}^K} F_j^{(t-1)}(w)
\end{align}
Solving for $\nabla F_j^{(t-1)}(w) = 0$ yields
\begin{align}\label{eq:MErawSol}
\alpha_j^{(t-1, k)}\phi^{(k)}(w) = \frac{1}{K} \sum_{l=1}^{K} \alpha_j^{(t-1,
l)}\phi^{(l)}(w)
\end{align}
for $1 \leq k\leq K$, where
\begin{align}
\alpha_j^{(t-1, k)} &\triangleq \sum_{i \in Z_j^{(k)}} \exp \left( -
\mu_i^{(t-1)} \right) \\
\mu_i^{(t-1)} &\triangleq \frac{1}{K} \sum_{k=1}^{K} y_i \hat{y}^{(t-1,
k)}(x_i) \\
\phi^{(k)}(w) &\triangleq \exp \left( - \frac{1}{K} \psi^{(k)}(w) \right) \\
\psi^{(k)}(w) &\triangleq -w^{(k)} + \frac{1}{K-1} \sum_{l=1, l\neq k}^{K}
w^{(l)}
\end{align}
where $Z_j^{(k)} = \{1 \leq i \leq N | z_{i,j} = 1 \wedge y_i^{(k)} = 1 \}$ is
the subset of learning instances of class $k$ reaching node $j$. In words,
$\mu_i^{(t-1)}$ is the hyper-margin of instance $i$ at time $t-1$ and
$\alpha_j^{(t-1, k)}$ is the class error of label $k$ for node $j$ at time
$t-1$.

Equation \ref{eq:MErawSol} is equivalent to
\begin{align}\label{eq:MEequation}
\alpha_j^{(t-1, k)}\phi^{(k)}(w) &= \alpha_j^{(t-1, l)}\phi^{(l)}(w) \quad 1
\leq k,l \leq K
\end{align}

In keeping with the output representation (Equation \ref{eq:MEencode}), we can
impose a zero-sum constraint on the prediction to get a unique solution for the
$k$th component of $w_j^{(t)}$. If it is imposed at each stage, it means that
\begin{align}\label{eq:MEzeroSum}
\sum_{k=1}^{K} \hat{y}^{(t-1, k)} = \sum_{k=1}^{K}
\hat{y}^{(t, k)} = 0 = \sum_{k=1}^{K} w^{(k)}
\end{align}
and this is not impacted by the learning rate.

The corresponding solution is
\begin{align}
\phi^{(k)}(w) &= \exp \left(-\frac{1}{K-1} w^{(k)}\right)\\
\label{eq:MEClsErrZS}
\alpha_j^{(t-1, k)} &= \sum_{i \in Z_j^{(k)}} \exp \left( -\frac{1}{K-1}
\hat{y}^{(t-1, k)}(x_i) \right) \\ \label{eq:MEsolution}
w_j^{(t,k)} &= \frac{K-1}{K}  \sum_{l=1}^{K} \log \frac{\alpha_j^{(t-1,
k)}}{\alpha_j^{(t-1, l)}}
\end{align}

\section{Equivalence of GIF and the underlying tree}\label{app:Equiv}
In the case of a single tree ($T=1$) and a unit learning rate ($\lambda=1$),
both the square loss in regression and the multiexponential loss in
classification produce the same predictions as the underlying tree.
This is due to the fact that, when examining the weight to give to node $j$ at
time $t$, the prediction of time $t-1$ relates to the parent node $\pi_j$ of
$j$. It is thus independent of $t$ and is also the same for all instances
reaching that node.

Consequently, we will adopt the following slight change in notation:
\begin{align}
\hat{y}_j = \hat{y}_{(\pi_j)} + w_j
\end{align}
meaning that the prediction associated to any object reaching node $j$ is the
weight of $j$ plus the prediction associated to its parent $\pi_j$. With
$\hat{y}_{(\pi_1)} = 0$, the prediction of the root's pseudo-parent.

\subsection{Regression}
In regression, the tree prediction $Tr_j$ of any leaf $j$ is the average of the
learning set's outputs reaching that node: $Tr_j = \frac{1}{|Z_j|}\sum_{i \in
Z_j} y_i$. We need to show that the GIF prediction is:
\begin{align}\label{eq:EquivL2Cond}
\hat{y}_{j} = \frac{1}{|Z_j|}\sum_{i \in Z_j} y_i
\end{align}

The prediction of node $j$ is
\begin{align}\label{eq:EquivL2Solution}
\hat{y}_j &= \hat{y}_{\pi_j} + w_j \\
&= \hat{y}_{\pi_j} +  \frac{1}{|Z_j|} \sum_{i \in Z_j} \left(y_i -
\hat{y}_{\pi_j}\right) \\
&= \hat{y}_{\pi_j} + \frac{1}{|Z_j|} \sum_{i \in Z_j} \left( y_i \right) -
\hat{y}_{\pi_j} \\
&= \frac{1}{|Z_j|} \sum_{i \in Z_j}  y_i
\end{align}

The first step is how the additive model is built. The second is the optimal
weight value of node $j$ derived in Equation \ref{eq:L2Solution}, the third
step is due to the fact that the prediction at $\pi_j$ is constant since there
is only one tree.

\subsection{Classification}
In order to have the same prediction as the underlying tree, we must
demonstrate that the probability of being in class $l$ associated to node $j$
will be $\frac{|Z_j^{(l)}|}{|Z_j|}$.

Under the zero-sum constraint, we have
\begin{align}
\exp \left(  \frac{1}{K-1} w_j^{(l)}\right) &= \frac{1}{c_j}
\alpha_{\pi_j}^{(l)} \\
&=  \frac{1}{c_j} \sum_{i \in Z_j^{(l)}} \exp \left(-\frac{1}{K-1}
\hat{y}_{\pi_j}^{(l)}\right)\\
&= \frac{1}{c_j} |Z_j^{(l)}| \exp \left(-\frac{1}{K-1}
\hat{y}_{\pi_j}^{(l)}\right) \\
\exp \left(\frac{1}{K-1} \hat{y}_j^{(l)} \right) &= \exp \left(\frac{1}{K-1}
\hat{y}_{\pi_j}^{(l)} \right) \exp \left(\frac{1}{K-1} w_j^{(l)}\right) \\
&= \frac{1}{c_j} |Z_j^{(l)}| \\
P_j(l) &= \frac{\exp \left(\frac{1}{K-1} \hat{y}_j^{(l)}
\right)}{\sum_{k=1}^K\exp \left(\frac{1}{K-1} \hat{y}_j^{(k)} \right)} =
\frac{|Z_j^{(l)}|}{|Z_j|}
\end{align}
where $c_j = \left(\prod_{k=1}^K \alpha_j^{(k)}\right)^{\frac{1}{K}}$ is a
constant. The first equality is a consequence of the value of $w_j^{(l)}$
(Equation \ref{eq:MEsolution}). The second is a due to the definition of
$\alpha_j^{(l)}$ (Equation \ref{eq:MEClsErrZS}). The third is a consequence of
having a single tree: the prediction of the parent is the same for all
instances.

Notice that, in both regression and classification, the equivalence also holds
for an internal node: the prediction is the one the tree would have yielded if
that node had been a leaf.

\section{Datasets}

Table \ref{tab:datasets} sums up the main characteristics of the datasets we
used. Abalone, CT slice, California data housig (Cadata), Musk2, Vowel and
Letter come from the UCI Machine Learning Repository \cite{uci}. Ringnorm,
Twonorm and Waveform are described in \cite{breiman1998arcing}. Hwang F5 comes
from the DELVE repository \footnote{http://www.cs.utoronto.ca/∼delve}.
The noise parameter of the Friedman1 dataset \cite{friedman11991} has
been set to $1$. Hastie is described in \cite{hastie2009}. Out of the 500
features of Madelon \cite{guyon2004madelon}, 20 are informative and 50 are
redundant; the others are noise.
Mnist8vs9 is the Mnist dataset \cite{lecun1998mnist} of which only the $8$ and
$9$ digits have been kept. Binary versions of the Mnist, Letter and Vowel
datasets have been created as well by grouping the first half and second half
classes together.

\begin{table}[th]
\caption{Characteristics of the datasets. $N$ is the learning sample size, TS
stands for testing set, and $p$ is the number of features.}
\label{tab:datasets}
\begin{center}
\begin{footnotesize}
\begin{sc}
\begin{tabular}{l|cccc}
\hline
Dataset & $N$ & $|TS|$ & $p$ & \# classes\\
\hline
Friedman1 & 300 & 2000 & 10 & - \\
Abalone & 2506 & 1671 & 10 & - \\
CT slice & 2000 & 51500 & 385 & - \\
Hwang F5 & 2000 & 11600 & 2 & - \\
Cadata & 12384 & 8256 & 8 & - \\
Ringnorm & 300 & 7100 & 20 & 2 \\
Twonorm & 300 & 7100 & 10 & 2 \\
Hastie & 2000 & 10000 & 10 & 2 \\
Musk2 & 2000 & 4598 & 166 & 2 \\
Madelon & 2200 & 2200 & 500 & 2 \\
Mnist8vs9 & 11800 & 1983 & 784 & 2 \\
Waveform & 3500 & 1500 & 40 & 3 \\
Vowel & 495 & 495 & 10 & 11 \\
Mnist & 50000 & 10000 & 784 & 10 \\
Letter & 16000 & 4000 & 8 & 26 \\
\hline
\end{tabular}
\end{sc}
\end{footnotesize}
\end{center}
\vskip -0.2in
\end{table}

\section{Comparison with local baseline algorithms}
We have tested three deepening algorithm for decision forest relying on
non-global metrics, meaning that the choice of the best candidate is not made
according to how well the forest, as a whole, performs. These algorithms share
that the final model is exactly a sub-forest of the un-pruned forest: contrary
to GIF, no internal weights are fitted and the predictions of at the leaves are
the usual tree predictions.

\paragraph{Breadth first deepening}
This variant consist in adding the nodes level after level, from left to right,
producing a heaped forest. As a consequence, all trees have the same (order of)
height, implying that the forest can be quite wide but usually shallow.

\paragraph{Random deepening}
This variant consist in first choosing a tree and then choosing one of its
leaves to transform to a decision nodes. Both choices are made uniformly at
random so that the trees are expected to have approximately the same number of
nodes. The depth, however, might vary significantly.

\paragraph{Best first deepening}
This variant consist in choosing, among all leaves which could be turned into a
internal node, the one which reduces its local impurity the most.
Let $N_c$, $N_l$ and $N_r$ be the number of instances
reaching the candidate node, candidate left child and candidate right child
respectively.
Let also $I_c$, $I_l$ and $I_r$ be the impurity (gini index in classification,
variance in regression) of the instances reaching the candidate node, candidate
left child and candidate right child respectively.
Then, for $N$ learning instances, the local impurity reduction is
defined as:

\begin{align}
\Delta I_c \triangleq \frac{N_c}{N} \left[ I_c - \left( \frac{N_l}{N_c} I_l +
\frac{N_r}{N_c} I_r \right)\right]
\end{align}

Since the fraction of learning instances reaching the candidate is accounted
for in the reduction of impurity, this approach will naturally favor higher
nodes in the trees.

\begin{table*}[t]
\caption{Average mean square error for local baselines at $1\%$ and $10\%$
budgets  ($T=1000$, $m=p$).}
\vskip -0em
\label{tab:baseline-reg}
\begin{center}
\begin{footnotesize}
\begin{sc}
\begin{tabular}{l|ccc|ccc}
\hline
Dataset & Breadth First$_{10\%}$ & Random$_{10\%}$ & Best First$_{10\%}$ &
Breadth First$_{1\%}$ & Random$_{1\%}$ & Best First$_{1\%}$ \\
\hline
Friedman1 & 6.02 $\pm$ 0.28 & 6.80 $\pm$ 0.34 & 15.00 $\pm$ 0.39 & 11.73 $\pm$
0.46 & 12.52 $\pm$ 0.47 & 15.29 $\pm$ 0.42  \\
Abalone & \best 4.72 $\pm$ 0.23 & 4.77 $\pm$ 0.23 & 6.82 $\pm$ 0.33 & 5.42
$\pm$ 0.27 & 5.55 $\pm$ 0.27 & 6.82 $\pm$ 0.33 \\
CT slice &  30.39 $\pm$ 1.90 & 36.19 $\pm$ 1.84 & 310.87 $\pm$ 4.79 & 82.19
$\pm$ 2.41 & 97.24 $\pm$ 1.90 & 313.84 $\pm$ 4.64  \\
Hwang F5 \hfill {\tiny $\times 10^{-2}$} & \best 6.73 $\pm$ 0.07 & 6.83 $\pm$
0.06 & 56.57 $\pm$ 6.03 & 8.52 $\pm$ 0.24 & 13.17 $\pm$ 0.44 & 56.60 $\pm$ 6.07
\\
Cadata \hfill {\tiny $\times 10^{-2}$} & 29.24 $\pm$ 0.73 & 31.08 $\pm$ 0.74 &
75.23 $\pm$ 0.95  & 43.40 $\pm$ 1.18 & 47.47 $\pm$ 1.02 & 75.48 $\pm$ 0.95 \\
\hline
\end{tabular}
\end{sc}
\end{footnotesize}
\end{center}
\vskip -0.2in
\end{table*}

\begin{table*}[t]
\caption{Error rate ($\%$) for local baselines at $1\%$ and $10\%$ budgets
($T=1000$, $m=\sqrt{p}$). The six first datasets are binary classification. The
last three are multiclass. The three in the middle are their binary versions.}
\vskip -1em
\label{tab:baseline-cls}
\begin{center}
\begin{footnotesize}
\begin{sc}
\begin{tabular}{l|ccc|ccc}
\hline
Dataset & Breadth First$_{10\%}$ & Random$_{10\%}$ & Best First$_{10\%}$ &
Breadth First$_{1\%}$ & Random$_{1\%}$ & Best First$_{1\%}$ \\
\hline
Ringnorm & 4.25 $\pm$ 1.24 & 4.08 $\pm$ 1.12 & 8.38 $\pm$ 6.94 &  8.94 $\pm$
7.45 & 8.53 $\pm$ 7.04 & 8.94 $\pm$ 7.41 \\
Twonorm & 3.51 $\pm$ 0.26 & 3.53 $\pm$ 0.30 & 5.59 $\pm$ 1.85  & 5.91 $\pm$
3.03 & 6.52 $\pm$ 4.28 & 7.28 $\pm$ 4.34 \\
Hastie & 11.30 $\pm$ 1.20 & 11.18 $\pm$ 1.16 & 21.24 $\pm$ 7.11  & 13.92 $\pm$
2.93 & 14.29 $\pm$ 3.20 & 21.24 $\pm$ 7.12 \\
Musk2 & 7.01 $\pm$ 0.40 & 7.63 $\pm$ 0.43 & 15.42 $\pm$ 0.23 & 15.42 $\pm$ 0.23
& 15.42 $\pm$ 0.23 & 15.42 $\pm$ 0.23 \\
Madelon & 11.68 $\pm$ 0.67 & 11.92 $\pm$ 0.65 & 19.12 $\pm$ 1.94 & 16.26 $\pm$
0.97 & 16.70 $\pm$ 1.07 & 20.14 $\pm$ 2.41 \\
Mnist8vs9 & 2.20 $\pm$ 0.38 & 2.37 $\pm$ 0.39 & 6.17 $\pm$ 0.73  & 4.53 $\pm$
0.48 & 4.84 $\pm$ 0.51 & 6.67 $\pm$ 0.69 \\
\hline
Bin. Vowel &8.99
$\pm$ 1.96 & 8.85 $\pm$ 2.03 & 16.57 $\pm$ 3.02  & 18.73 $\pm$ 3.08 & 19.90
$\pm$ 3.71 & 21.80 $\pm$ 4.38 \\
Bin. Mnist & 4.46 $\pm$ 0.25 & 4.91 $\pm$ 0.27 & 21.71 $\pm$ 0.30  & 10.09
$\pm$ 0.25 & 11.78 $\pm$ 0.32 & 22.50 $\pm$ 0.35\\
Bin. Letter & 5.91 $\pm$ 0.43 & 5.71 $\pm$ 0.40 & 26.16 $\pm$ 0.86  & 17.91
$\pm$ 0.77 & 18.05 $\pm$ 0.78 & 26.19 $\pm$ 0.88 \\
\hline
Waveform & 14.74 $\pm$ 0.63 & 14.83 $\pm$ 0.76 & 20.25 $\pm$ 2.22 & 16.75 $\pm$
1.26 & 17.13 $\pm$ 1.25 & 20.45 $\pm$ 2.21 \\
Vowel & 14.26 $\pm$ 2.41 & 13.21 $\pm$ 2.33 & 41.49 $\pm$ 5.45 & 42.40 $\pm$
4.33 & 40.28 $\pm$ 4.62 & 50.44 $\pm$ 5.81 \\
Mnist & 4.63 $\pm$ 0.27 & 4.96 $\pm$ 0.26 & 28.54 $\pm$ 0.59 & 8.60 $\pm$ 0.35
& 9.76 $\pm$ 0.31 & 29.72 $\pm$ 0.61 \\
Letter & 7.06 $\pm$ 0.29 & 6.39 $\pm$ 0.20 & 36.92 $\pm$ 1.80 & 22.11 $\pm$
0.59 & 20.90 $\pm$ 0.55 & 37.27 $\pm$ 1.78 \\
\hline
\end{tabular}
\end{sc}
\end{footnotesize}
\end{center}
\vskip -0.2in
\end{table*}

\paragraph{Experiment}
We conducted the same experiment as for GIF: the three algorithms were tested
on ten folds with different learning sample/testing sample splits and were
subjected to the $1\%$ and $10\%$ node constraints. We started with a pool of
$T=1000$ roots and no restriction was imposed regarding the depth. All of the
$m=p$ the features were examined in regression and $m=\sqrt{p}$ in
classification, as suggested in \cite{extratrees}. Table \ref{tab:baseline-reg}
holds the average mean square error for the five regression problems and Table
\ref{tab:baseline-cls} holds the average misclassification rate for the
classification problems.

\paragraph{Regression}
The trend is quite clear: both at $1\%$ and $10\%$, the breadth first algorithm
is the best and the best first is (largely) the worst.
There are two instances where the local baselines are able to beat GIF: on
Abalone and Hwang F5 at $10\%$. Interestingly, these are the same cases on
which GIF was beaten by a small forest of Extremely randomized trees. The
$10\%$ Hwang F5 case aside, the local baselines always underperform the smaller
fully-developed forest. Overall, such variants do not seem adequate for
regression.

\paragraph{Classification}
In classification, the breadth first and random baselines tend to perform
similarly, one beating the other on some problems. Once again, the best first
approach seems to be lagging behind on some datasets. At $10\%$, the local
baselines cannot rival with the other methods. Only on Waveform are they able
to reach the other performances---a setting where all methods seems to produce
close results. At $1\%$, the breadth first and/or the random methods surpass
the ET$_{10\%}$ on Twonorm, Hastie, Madelon and Waveform. Those datasets
correspond to cases where ET was under-performing significantly compared to GIF.
All in all, the local baselines are never able to beat GIF, even in the
multiclass setting, which is particularly defavorable for GIF. Once again, the
conclusion is against the purely local baselines.

We believed the poor performances of the baselines are due to the building
mechanism of traditional ensemble methods. Although the trees are built
independently and with randomization, there remains an important redundancy
between them, which is especially defavorable to pruning. A global approach is
better able to avoid redundancy and can thus better exploit the node budget.
This would also explain why the best first variant performs worst in both
regression and classification: it is prone at picking redundant nodes, which
will usually offer the same kind of impurity reduction.

\bibliography{gif}
\bibliographystyle{icml2017}

\end{document}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
% From a template maintained at https://github.com/jamesrobertlloyd/cbl-tikz-poster
%
% Code near the top should be fairly standard and not need to be changed
%  - except for the document class
% Code lower down is more likely to be customised
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\documentclass[landscape,a0b,final,a4resizeable]{include/a0poster}

\usepackage{multicol}
\usepackage{color}
\usepackage{morefloats}
\usepackage[pdftex]{graphicx}
\usepackage{rotating}
\usepackage{amsmath, amsthm, amssymb, bm}
\usepackage{array}
\usepackage{booktabs}
\usepackage{multirow}
\usepackage{hyperref}

\usepackage{include/picins}
\usepackage{tikz}
\usetikzlibrary{shapes.geometric,arrows,chains,matrix,positioning,scopes,calc}
\tikzstyle{mybox} = [draw=white, rectangle]
\definecolor{darkblue}{rgb}{0,0.08,0.45}
\definecolor{blue}{rgb}{0,0,1}

\usepackage{dsfont}

\input{include/jlposter.tex}

\input{include/preamble.sty}
\newcommand{\vv}{\mathbf{v}}

\begin{document}
\begin{poster}

% Potentially add some space at the top of the poster
\vspace{0\baselineskip}

%%% Header
\begin{center}
\begin{pcolumn}{0.99}

\newcommand{\logowidth}{0.11\textwidth}

\pbox{0.99\textwidth}{}{linewidth=2mm,framearc=0.3,linecolor=camdarkblue,fillstyle=gradient,gradangle=0,gradbegin=white,gradend=white,gradmidpoint=1.0,framesep=1em}{
%
%%% Cambridge Logo
\begin{minipage}[c]{\logowidth}
  \begin{center}
    \includegraphics[width=20cm]{badges/hips-logo.png}
  \end{center}
\end{minipage}
%
%%% Title
\begin{minipage}[c][9cm][c]{0.76\textwidth}
  \begin{center}
    {\sffamily \VeryHuge \textbf{Convolutional Networks on Graphs\\[4mm] for Learning Molecular Fingerprints}}\\[10mm]
    {\huge\sffamily \Huge David Duvenaud*, Dougal Maclaurin*, Jorge Aguilera-Iparraguirre \\ Rafael G\'omez-Bombarelli, Timothy Hirzel, Al\'an Aspuru-Guzik, Ryan P. Adams\\[7.5mm]
    %\texttt{\{ti242, dkd23, zoubin\}@cam.ac.uk}
    }
  \end{center}
\end{minipage}
%
%
% Harvard logo
\begin{minipage}[c]{\logowidth}
  \begin{flushright}
    \includegraphics[width=9cm,trim=2em 0em 2em 2em, clip]{badges/harvard}
  \end{flushright}
\end{minipage}
%
}
\end{pcolumn}
\end{center}

\vspace*{3cm}

\Large

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%% Beginning of Document
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{multicols}{3}

\mysection{How to do regression on graphs?}

%\vspace{-0.5in}

\begin{tabular}{cc}
\begin{minipage}[c]{0.45\columnwidth}
\begin{itemize}
  \item Input can be any size or shape
  \item Hard to turn into fixed-length vector
  \item In our case, graphs represent molecules
  \item Applications to photovoltaics, organic LEDS, flow batteries and pharmaceuticals
\end{itemize}
\end{minipage} &
\begin{minipage}[c]{0.55\columnwidth}
%\includegraphics[width=\columnwidth]{../talks/talkfigs/learning_curves_3.pdf}
\centerline{\includegraphics[width=1.0\columnwidth, clip, trim=4mm 0mm 4mm 4mm]{figures/how-fingerprints.png}}
\end{minipage}
\end{tabular}

%\vspace{-0.5in}

\vspace{1.5in}

\mysection{Circular fingerprints}

%Also called Morgan fingerprints, or ECFP

\vspace{0.5in}

\begin{tabular}{cc}
\begin{minipage}[c]{0.5\columnwidth}
\begin{itemize}
  \item Maps variable-sized molecular graph to fixed-length binary vector
  %\item Does this by hashing self with neighbors iteratively
  \item Binary features indicate presence of substructures
\end{itemize}

\vspace{0.5in}

Can be efficiently computed using local operations:

\begin{itemize}
  \item At each layer, hash the features of each atom and its neighbors/bonds
  \item More layers correspond to increasing radius of substructures
  \item Interpret each hash as integer and set that entry to one
\end{itemize}
\end{minipage} &
\begin{minipage}[c]{0.5\columnwidth}
%\includegraphics[width=\columnwidth]{../talks/talkfigs/learning_curves_3.pdf}
\centerline{\includegraphics[width=0.9\columnwidth, clip, trim=4mm 12mm 4mm 4mm]{figures/fig_1}}
\end{minipage}
\end{tabular}

\vspace{0.5in}

Was state-of-the-art for large-scale regression and classification.

\newpage %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\mysection{Convolutional neural nets on graphs}

\vspace{0.5in}

\begin{tabular}{ccc}
\begin{minipage}[c]{0.45\columnwidth}

How to make circular fingerprints differentiable?
Replacing ops:
\vspace{0.5in}

\begin{center}
\begin{tabular}{rcl}
Hash  & $\rightarrow$ & Neural net \\
Index & $\rightarrow$ & Softmax \\
Write & $\rightarrow$ & Add
\end{tabular}
\end{center}

\vspace{0.5in}

Gives end-to-end differentiable convolutional network.

\vspace{0.5in}

Can be trained to adapt to particular tasks.

\end{minipage}
& \qquad &
\begin{minipage}[c]{0.45\columnwidth}
\begin{center}
Information flow graph:
\centerline{\includegraphics[width=\columnwidth, clip, trim=0mm 0mm 4mm 4mm]{figures/3d-nets/net1.png}}
\end{center}

Message passing between neighbors, then final pooling step

\end{minipage}
\end{tabular}

\vspace{1.5in}

\mysection{Neural fingerprints are interpretable}

\newcommand{\mywidtha}{8cm}
\newcommand{\mywidthb}{10.3cm}

When fed into linear layer, can see how fragments affect prediction:

\vspace{-0.5in}
\begin{center}
\begin{tabular}{>{\centering}m{\mywidthb} >{\centering}m{\mywidtha} >{\centering}m{\mywidtha} >{\centering\arraybackslash}m{\mywidtha}}
Fragments predictive of solubility &
%\molfeature{15}{0}{3.3cm} & \molfeature{15}{3}{3.3cm} & \molfeature{15}{2}{2.5cm}\\
\includegraphics[width=\mywidtha, clip, trim = 2mm 3mm 2mm 6mm]{figures/fig_5.pdf} &
\includegraphics[width=\mywidtha, clip, trim = 2mm 3mm 2mm 6mm]{figures/fig_6.pdf} &
\includegraphics[width=6cm, clip, trim = 2mm 3mm 2mm 6mm]{figures/fig_7.pdf} \\
\midrule
Fragments predictive of insolubility &
%\molfeature{18}{4}{3.3cm} & \molfeature{18}{1}{3.3cm} & \molfeature{18}{2}{3.3cm}
\includegraphics[width=\mywidtha, clip, trim = 2mm 3mm 2mm 6mm]{figures/fig_8.pdf} &
\includegraphics[width=\mywidtha, clip, trim = 2mm 3mm 2mm 6mm]{figures/fig_9.pdf} &
\includegraphics[width=\mywidtha, clip, trim = 2mm 3mm 2mm 6mm]{figures/fig_10.pdf} \\
%\end{tabular}
%\end{center}
\midrule
%\subsection{Toxicity features}
%We trained the same model architecture to predict toxicity in two different datasets.
%Shows fragments which maximally activate the feature most predictive of toxicity, in two separate datasets.
%\newcommand{\molfeaturetox}[2]{\includegraphics[width=3.4cm, clip, trim = 1mm 3mm 1mm 3mm]{fig_11.pdf}}%
%\begin{figure}[h]
%\begin{center}
%\begin{tabular}{>{\centering}m{\mywidthb} >{\centering}m{\mywidtha} >{\centering}m{\mywidtha} >{\centering\arraybackslash}m{\mywidtha}}
%\vspace{0.5}
\vspace{0.5in}Fragments predictive of toxicity on SR-MMP dataset\vspace{0.5in}
& \includegraphics[width=7cm]{figures/jorge-figures/7.png}
& \includegraphics[width=\mywidtha]{figures/jorge-figures/8.png}
& \includegraphics[width=\mywidtha]{figures/jorge-figures/9.png}\\
\midrule
\vspace{0.5in}
Fragments predictive of toxicity on NR-AHR dataset
\vspace{0.5in}
%& \molfeaturetox{7}{14} & \molfeaturetox{7}{6} & \molfeaturetox{7}{5}
& \includegraphics[width=6.5cm]{figures/jorge-figures/10.png}
& \includegraphics[width=\mywidtha]{figures/jorge-figures/11.png}
& \includegraphics[width=7cm]{figures/jorge-figures/12.png}\\
\end{tabular}
%\caption{
\end{center}

%Visualizing fingerprints optimized for predicting toxicity.
%Shown here are representative samples of molecular fragments (highlighted in red) which most activate the feature most predictive of toxicity.

%\emph{Top row:} the most predictive feature identifies groups containing a sulphur atom attached to an aromatic ring.
%\emph{Bottom row:} the most predictive feature identifies fused aromatic rings, also known as polycyclic aromatic hydrocarbons, a well-known carcinogen.

\newpage %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\mysection{Neural graph fingerprints\\generalize circular fingerprints}

Large random weights give similar behavior to circular fingeprints:
\vspace{0.5em}

\begin{tabular}{cc}
\begin{minipage}[c]{0.48\columnwidth}
\includegraphics[width=\columnwidth]{figures/fig_2.pdf}
\end{minipage} &
\begin{minipage}[c]{0.48\columnwidth}
\begin{center}
\vspace{0.5cm}\includegraphics[width=\columnwidth]{figures/fig_3.pdf}
\end{center}
\end{minipage}
\end{tabular}

\vspace{0.5em}

Small random weights already much better than circular fingerprints!

Can do even better by optimizing for given task.

\vspace{1in}

\mysection{Predictive accuracy}

Neural graph fingerprints fed to neural net generalizes state of the art:%, we can't not win:

\vspace{0.5em}

\begin{center}
\begin{tabular}{r|lll}
Dataset                      &   Solubility  & Drug efficacy & Photovoltaic \\
 & & & efficiency \\
\midrule
Units                        &   log Mol/L                            & EC$_{50}$ in nM                        & percent \\
\midrule
Predict mean                 & 2.07 $\pm$ 0.10        & 1.21 $\pm$ 0.03         & 2.53 $\pm$ 0.02 \\
Circular FPs + linear layer  & 1.31 $\pm$ 0.05        & \bf{1.06} $\pm$ 0.01    & 1.62 $\pm$ 0.03 \\
Circular FPs + neural net    & 1.18 $\pm$ 0.05        & 1.16 $\pm$ 0.04         & 1.41 $\pm$ 0.03 \\
Neural FPs + linear layer    & 0.87 $\pm$ 0.06        & \bf{1.07} $\pm$ 0.01    & 1.61 $\pm$ 0.06 \\
Neural FPs + neural net      & \bf{0.72} $\pm$ 0.05   & \bf{1.08} $\pm$ 0.01    & \bf{1.20} $\pm$ 0.04
\end{tabular}

\vspace{0.5em}

%RMSE Mean predictive accuracy of neural fingerprints compared to standard circular fingerprints.
\end{center}

\vspace{0.5em}

\mysection{Conclusion}

\begin{itemize}
\item Can learn graph features end-to-end!
\item Works on other types of graphs too
\item Code at \url{github.com/HIPS/neural-fingerprint}
\item Autodiff package that works on standard Numpy code:\\ \url{github.com/HIPS/autograd}
\end{itemize}

\end{multicols}
\end{poster}

\end{document}


\subsection{Preliminary experiments: encoding hierarchical rules}
Before diving into the FET task, KENN has been tested in a hierarchical classification scenario. Since the goal of this preliminary phase is to familiarize with KENN rather than achieve the best performance, we do not dwell on the architecture, training setup, and results. We focus, instead, on how hierarchical relations can be encoded with KENN. Different heuristics to represent a hierarchy through logical rules are proposed within these preliminary studies.

\subsubsection{Dataset}
% The baseline model implemented for these experiments has a simple architecture. It uses DistilBERT as encoder, followed by a dense fully connected layer and a final classification layer that applies the sigmoid activation function. The choice of this activation function is due to the fact that in a multilabel task we want as output the probability of each label. Starting from this architecture, KENN is placed between the fully connected layer and the classification layer. Clause weights are set as learnable parameters with an initial value of 0.5.

The dataset used for this experiment is called DBpedia Classes\footnote{https://www.kaggle.com/danofer/dbpedia-classes}. It is composed of approximately 240k labeled Wikipedia articles, where the only feature is the span of text. The types are organized as a forest of 9 trees, each of them representing a 3-level hierarchy. The type set counts 298 types: 9 at top-level, 70 at middle-level, and 219 at low-level. Each example is labeled with exactly 3 classes from the full path of a tree (i.e., one class per level). For example, if we consider the \textit{Species} tree shown in Figure~\ref{fig:dbpedia_classes}, an instance of the dataset could present $ label_{1} = Species $, $ label{_2} = Animal $, and $ label_{3} = Bird $.

\begin{figure}[H]
    \centering
    \includegraphics[width=.7\linewidth]{dbpedia_classes.png}
    \caption{Example of a tree of DBpedia Classes}
    \label{fig:dbpedia_classes}
\end{figure}

\subsubsection{Knowledge definition} \label{knowledge_generation}
The first step to integrate KENN is to create the logical KB. We can start from the tree in Figure~\ref{fig:dbpedia_classes} and then generalize different strategies for defining logical rules. The most intuitive and straightforward way consists in following the \textit{is-a} relations, which are implicitly encoded in the tree structure. Considering the picture, it is possible to state that if an instance is a \texttt{Fish}, then it is an \texttt{Animal}. In the same way, we can then proceed with the next level of the hierarchy and state that if an instance is an \texttt{Animal}, then it is a \texttt{Species}. This mechanism can be extended to all the other subtrees to create a logical clause per edge. The resulting KB will be a set of logical implications between subtypes and supertypes. Even if it seems a natural way to describe the hierarchy, it doesn't necessarily mean it is the most effective solution. For this reason, we should take a step back and analyze the alternatives offered by the hierarchy. Figure~\ref{fig:hierarchy_example} shows a generic hierarchy and highlights two groups of constraints we can distinguish:
\begin{enumerate}
    \item \textbf{Vertical constraints:} based on the paths of a tree, they can be used to represent specialization relations
    \item \textbf{Horizontal constraints:} based on pairs of nodes from different branches, they can be used to represent disjointness relations
\end{enumerate}
\begin{figure}[H]
    \centering
    \includegraphics[width=.7\linewidth]{hierarchy_example.png}
    \caption{Examples of a generic hierarchy. In green the vertical constraints, in red the horizontal constraints.}
    \label{fig:hierarchy_example}
\end{figure}
For each group of constraints, it is possible to implement different strategies to define the KB. In the rest of the thesis, these strategies will be referred to as \textit{KB modes}. Note that in the definitions below we will use unary predicates to represent the belonging of an instance $X$ to a type.

\paragraphn{Vertical Constraints}
Using the unary predicates $Subtype$ and $Supertype$ such that the type $Subtype$ is a specialization of the type $Supertype$, we can define three main strategies to build vertical constraints:
\begin{itemize}
    \item \textbf{Bottom Up:}
        \begin{itemize}
            \item FOL: $ \forall X, Subtype(X) \to Supertype(X) $
            \item KENN: $ \neg Subtype \vee Supertype $
            \item Defined by following every path of the tree from the leaves to the root. It represents the logic dependency between types, following the most widely used semantics in the definition of subclass.
            \item Open World Assumption: a clause is satisfied even when an instance is assigned only to supertypes
        \end{itemize}
    \item \textbf{Top Down:}
        \begin{itemize}
            \item FOL: $ \forall X, Supertype(X) \to (Subtype_{1}(X) \vee ... \vee Subtype_{n}(X)) $
            \item KENN: $ \neg Supertype \vee Subtype_{1} \vee ... \vee Subtype_{n}  $
            \item Defined by following every path of the tree from the root to the leaves
            \item Closed World Assumption: a clause is not satisfied when an instance is assigned to a supertype without any of its subtypes
        \end{itemize}
    \item \textbf{Hybrid:}
    \begin{itemize}
        \item Composed of a mix of \textit{Bottom Up} and \textit{Top Down} clauses
    \end{itemize}
\end{itemize}
Note that if a model that uses this kind of knowledge is confident about a given type, a \textit{Bottom Up} clause propagates this certainty towards the root of the hierarchy. Conversely, a \textit{Top Down} clause only guarantees that one of the subtypes is appropriate, but provides no information as to which one. The reason, of course, is the tree-like hierarchy, where each type has only one parent, but potentially many descendants.

The presented KB modes can be considered as the starting point to create new strategies by adding some variations. In Table~\ref{tab:kb_modes} we can find a summary that comprehends every KB mode, each of them accompanied by examples of clauses from the hierarchy in Figure~\ref{fig:dbpedia_classes}. Before proceeding with the explanation of the variants, we have to make clear why they have been introduced: the conflicts. If we look at the table, we can see that in the examples of the pure \textit{Bottom Up} and \textit{Top Down} modes there are conflicts: a middle-level label appears both as positive and negative literal. This fact derives from the translation of the logical implication into a disjunction since an intermediate node is at the same time antecedent and consequent of different logical implications. This behavior still hold for the \textit{Hybrid} mode.

Now that the possible problems of the starting strategies are known, it is possible to introduce their conflict-free variants. These variants avoid conflict situations in different ways. The ``Skip" ones rely on the transitivity of a relation by linking top-level and low-level nodes by skipping those middle-level nodes that would create a conflict (i.e., a middle-level node cannot be the antecedent of an implication rule). These solutions provide a more shallow representation of the knowledge, especially for the \textit{Top Down Skip}. In this case, a logical rule that represents the relation between top-level and low-level lose effectiveness because the implication may have too many consequents. The \textit{Hybrid In} and \textit{Hybrid Out} modes, instead, avoid conflicts by fixing the middle-level types as consequent or antecedent of an implication rule, respectively. In this way, the deltas produced by each CE for the same literal will always have the same sign.

Note that the proposed variants are valid for this dataset, but some of them do not apply to every context since they require a hierarchy with at least 3 levels.

\begin{table}
\centering
\caption{Strategies for defining logical clauses based on the hierarchy in Figure~\ref{fig:dbpedia_classes}}
\label{tab:kb_modes}
\begin{tabular}{c|l|}
\cline{2-2}
                                              & \multicolumn{1}{c|}{\textbf{Clauses}}                                                                                \\ \hline
\multicolumn{1}{|c|}{\textbf{Bottom Up}}      & \begin{tabular}[c]{@{}l@{}}$ c_{1}: \neg Fish \vee Animal $\\ $ c_{2}: \neg Bird \vee Animal $\\ $ c_{3}: \neg Animal \vee Species $\\ ...\end{tabular}                                                                           \\ \hline
\multicolumn{1}{|c|}{\textbf{Top Down}}       & \begin{tabular}[c]{@{}l@{}}$ c_{1}: \neg Species \vee Animal \vee ... \vee Plant $\\ $ c_{2}: \neg Animal \vee Fish \vee ... \vee Bird $\\ $ c_{3}: \neg Plant \vee Fem \vee ... \vee Cycad $\\ ...\end{tabular}                  \\ \hline
\multicolumn{1}{|c|}{\textbf{Hybrid}}         & \begin{tabular}[c]{@{}l@{}}$ c_{1}: \neg Species \vee Animal \vee ... \vee Plant $\\ $ c_{2}: \neg Animal \vee Species $\\ $ c_{3}: \neg Plant \vee Species $\\ ...\end{tabular}                                                  \\ \hline
\multicolumn{1}{|c|}{\textbf{Bottom Up Skip}} & \begin{tabular}[c]{@{}l@{}}$ c_{1}: \neg Fish \vee Animal $\\ $ c_{2}: \neg Fish \vee Species $\\ $ c_{3}: \neg Bird \vee Animal $\\ $ c_{4}: \neg Bird \vee Species $\\ ...\end{tabular}                                         \\ \hline
\multicolumn{1}{|c|}{\textbf{Top Down Skip}}  & \begin{tabular}[c]{@{}l@{}}$ c_{1}: \neg Species \vee Animal \vee ... \vee Plant $\\ $ c_{2}: \neg Species \vee Fish \vee ... \vee Bird \vee Fem \vee Cycad $\\ ...\end{tabular}                                                  \\ \hline
\multicolumn{1}{|c|}{\textbf{Hybrid In}}      & \begin{tabular}[c]{@{}l@{}}$ c_{1}: \neg Species \vee Animal \vee ... \vee Plant $\\ $ c_{2}: \neg Fish \vee Animal $\\ $ c_{3}: \neg Bird \vee Animal $\\ $ c_{4}: \neg Fem \vee Plant $\\ ...\end{tabular}                      \\ \hline
\multicolumn{1}{|c|}{\textbf{Hybrid Out}}     & \begin{tabular}[c]{@{}l@{}}$ c_{1}: \neg Animal \vee Species $\\ $ c_{2}: \neg Animal \vee Fish \vee ... \vee Bird $\\ $ c_{3}: \neg Plant \vee Species $\\ $ c_{4}: \neg Plant \vee Fem \vee ... \vee Cycad $\\ ...\end{tabular} \\ \hline
\end{tabular}
\end{table}

\paragraphn{Horizontal Constraints}
These constraints aim to avoid the co-occurrence of types whose instances form disjoint sets (e.g., if an example is labeled as \texttt{Person} it cannot be labeled as \texttt{Location} and vice versa). Using $T_i$ to indicate an arbitrary type, the logical rule that best represents this constraint is:
\begin{gather*}
    \forall X, T_{1}(X) \to (\neg T_{2}(X) \wedge ... \wedge \neg T_{n}(X))
\end{gather*}
With the following steps, we can derive an equivalent formula without implication and conjunctions:
\begin{align*}
    & T_{1} \to (\neg T_{2} \wedge ... \wedge \neg T_{n}) = \\
    & = \neg T_{1} \vee (\neg T_{2} \wedge ... \wedge \neg T_{n}) = \\
    & = \neg T_{1} \vee \neg (T_{2} \vee ... \vee T_{n})
\end{align*}
Unfortunately, the derived formula cannot be furthermore decomposed due to the presence of parentheses. We can opt for two alternative solutions to bypass the problem:
\begin{enumerate}
    \item \textbf{Approximate the rule with a softer constraint:} if an instance does not belong to \texttt{T$_1$}, then it should belong to another type. \\
    The logical formula for this statement is:
    \begin{gather*}
            \forall X, \neg T_{1}(X) \to (T_{2}(X) \vee ... \vee T_{n}(X))
    \end{gather*}
    that can be translated into KENN's language with the following steps:
    \begin{align*}
        & \neg T_{1} \to (T_{2} \vee ... \vee T_{n}) = \\
        & = T_{1} \vee (T_{2} \vee ... \vee T_{n}) = \\
        & = T_{1} \vee T_{2} \vee ... \vee T_{n}
    \end{align*}
    From a logical point of view, the resulting formula does not seem very helpful because it simply states that an instance belongs at least to one class. Indeed, if we repeat the previous steps using any other type as the antecedent of the implication, the derived clause will be identical. Furthermore, the action of KENN on such clauses would not lead to mutual exclusivity because every literal is positive and none of the involved predictions will receive a negative boost.

    \item \textbf{Split the rule into multiple clauses:} define a clause for each pair of disjoint types.
    Given two disjoint types, the logical formula to express the mutual exclusivity is:
    \begin{gather*}
        \forall X, T_{1}(X) \to \neg T_{2}(X)
    \end{gather*}
    If we consider $n$ disjoint pairs in which appears $T_{1}$, the resulting KB in KENN's language will be the following:
    \begin{align*}
        & c_{1}: \neg T_{1} \vee \neg T_{2}\\
        & ...\\
        & c_{n}: \neg T_{1} \vee \neg T_{n}
    \end{align*}
    Unlike strategy 1, this one preserves the soundness of the starting formula. However, there are still some issues to take into account. The first concerns the number of clauses necessary to cover the possible combinations of pairs: having $ n $ mutual independent types will result in $ \frac{n!}{2!(n-2)!} $ logical clauses. The second issue regards the action of KENN: while in strategy 1 there were only positive literals, in this strategy there are only negative literals, thus meaning that none of the predictions will ever receive a positive boost.
\end{enumerate}

\paragraphn{Horizontal constraints - Alternative usage}
Another use of horizontal constraints that is not related to this dataset, but could be useful in other contexts, is to encourage the co-occurrence between different types. Here the purpose is exactly the opposite with respect to the previous rules. Even if this goal can be reached by applying a simple inference on the final predictions without the use of KENN, in some tasks (e.g., domain adaptation) it may be useful to leave the decision to the network. If we want the model to learn a mapping between two types \texttt{A} and \texttt{B}, we can define the logical rule
\begin{gather*}
    (A \to B) \wedge (B \to A)
\end{gather*}
that can be expressed in KENN using the following clauses:
\begin{align*}
    & c1: \neg A \vee B \\
    & c2: \neg B \vee A
\end{align*}
The first thing we can notice is the presence of conflicts, but they will not have a negative impact on the final predictions. The preactivation of the literals belonging to the two clauses are the same with opposite signs, so a  literal cannot be dominant in both clauses. In other words, the final predictions always benefit from the knowledge enhancement since the effect of KENN will be one of the following:
\begin{itemize}
    \item \textbf{positive preactivations:} a positive aggregated delta is produced for each literal
    \item \textbf{negative preactivations:} a negative aggregated delta is produced for each literal
    \item \textbf{discord preactivations:} an aggregated delta with the sign of the highest\footnote{in absolute value} preactivation is produced for each literal
\end{itemize}

\subsubsection{Results}
As anticipated, the results are not the focus of this study since we are not dealing with the target problem of this thesis. To see KENN in action, the KB modes of Bottom Up, Top Down, Hybrid In, and Hybrid Out have been evaluated.
The baseline model implemented for these experiments has a simple architecture. It uses DistilBERT as encoder, followed by a dense fully connected layer and a final classification layer that applies the sigmoid activation function. The choice of this activation function is due to the fact that in a multilabel task we want as output the probability of each label. Starting from this architecture, KENN is placed between the fully connected layer and the classification layer. Clause weights are set as learnable parameters with an initial value of 0.5.

In Table \ref{tab:performance_dbpedia} are reported the results obtained by the baseline and KENN-based models. Hybrid Out is the configuration that brought more benefits to the baseline model with a $+0.0157$ on the F1 score, thanks to its highest recall. Even Top Down brought improvements with an increase of $+0.0097$. This did not happen with the other configurations.

An interesting fact that emerged regards the clause weights. By inspecting the final weights learned by the models, it was found out that the models that performed the best are the ones that finished the train with higher clause weights. In particular, some clauses have increased their starting weight by 10 times. On the contrary, the worst models are the ones whose final weights are smaller. This fact means that KENN helped the baseline model the most when it gained more influence on the final predictions. From these results, we can say that the higher the weight, the better the performance. This result may lead to think that it could be reasonable to set higher initial clause weights to anticipate the learning process. However, we have to take this conclusion with a grain of salt because it could be strictly related to this dataset.

\begin{table}[H]
\centering
\caption{Comparison between the baseline and KENN-based models on DBpedia Classes in terms of \textit{macro f1 classes}}
\label{tab:performance_dbpedia}
\begin{tabular}{|c|ccc|}
\hline
\textbf{KB mode} & \multicolumn{1}{c|}{\textbf{P}} & \multicolumn{1}{c|}{\textbf{R}} & \textbf{F1}     \\ \hline
- (baseline)       & 0.9265                          & 0.8644                          & 0.8897          \\ \hline
Bottom Up      & 0.924                           & 0.8685                          & 0.8901          \\ \hline
Top Down       & \textbf{0.93}                   & 0.8788                          & 0.8994          \\ \hline
Hybrid In      & 0.9186                          & 0.87                            & 0.8884          \\ \hline
Hybrid Out     & 0.927                           & \textbf{0.8913}                 & \textbf{0.9054} \\ \hline
\end{tabular}
\end{table}