\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{multicol}
\usepackage[formats]{listings}
\lstloadaspects{formats}
\usepackage{verbatim}
\usepackage{color}
\usepackage{geometry}
\usepackage{float}
\usepackage{amsmath}
\usepackage{caption}
\usepackage{pdflscape}

\usepackage{hyperref}
\setlength{\belowcaptionskip}{-10pt}
\setlength{\abovecaptionskip}{-30pt}
\floatstyle{boxed}
\restylefloat{figure}
\usepackage{graphicx}
\definecolor{codegreen}{rgb}{0,0.6,0}
\definecolor{codegray}{rgb}{0.5,0.5,0.5}
\definecolor{codepurple}{rgb}{0.58,0,0.82}
\definecolor{backcolour}{rgb}{0.95,0.95,0.92}
\lstdefinestyle{mystyle}{
	backgroundcolor=\color{backcolour},
	commentstyle=\color{codegreen},
	keywordstyle=\color{blue},
	numberstyle=\tiny\color{codegray},
	stringstyle=\color{codepurple},
	basicstyle=\footnotesize,
	breakatwhitespace=false,
	breaklines=true,
	captionpos=b,
	keepspaces=true,
	numbers=left,
	numbersep=5pt,
	showspaces=false,
	showstringspaces=false,
	showtabs=false,
	tabsize=2
}
\lstset{style=mystyle}
\title{Data Mining\\
		Home work 10\\Machine Learning III Regression Analysis }
\author{Aqeel Labash\\ \textbf{Lecturer:} Jaak Vilo}
\date{13 April 2016}
\geometry{
	a4paper,
	total={170mm,257mm},
	left=10mm,
	top=5mm,
}
\begin{document}
	\maketitle
\section*{First Question}
For this task I used python and here is the code :
\begin{lstlisting}[language=Python]
# #Question 1
import numpy as np
get_ipython().magic(u'matplotlib inline')
import matplotlib.pyplot as plt
points = [(9,3,1),(2,4,1),(3,3,1),(4,1,1),(1,6,1),(3,9,0),(5,6,0),(6,4,0),(6,2,0),(3,7,0)]
#Draw the figure
plt.figure('points.jpg')
plt.plot([x[0] for x in points],[x[1] for x in points],'bo')
plt.plot([3,4],[5,6],'rv')
plt.ylabel('Y')
plt.xlabel('X')
plt.title('Points')
plt.savefig('points.jpg')

#Print Probabilities for classes
def GetClosePoints(centerpoint,k=1):
    indx =0
    distances={}
    for point in points:

        #Calculate Eucludean distance
        distance = np.linalg.norm(np.array(centerpoint)-np.array((point[0],point[1])))

        #Store all points with the same distance under the same
        if distance in distances.keys():
            distances[distance].append(indx)
        else:
            distances[distance] = []
            distances[distance].append(indx)
        indx+=1

    #Sort list by distance
    keys = distances.keys()
    keys.sort(key = lambda x:x,reverse=False)
    #print the list
    #for key in keys:
    #    print key , distances[key]

    #Get Points in K distance
    pointindx=[]
    for i in range (0,k):
        for index in distances[keys[i]]:
            pointindx.append(index)

    #Print Selected Points indexs
    #print pointindx

    #Calculate Probabilities
    Totalpoints = len(pointindx)
    Class0=0
    Class1=0
    for i in pointindx:
        if points[i][2]:
            Class1+=1
        else:
            Class0+=1
    print 'For Point ({},{}) with K={},Probabilities is Class 0:{},Class 1:{}'.format(centerpoint[0],centerpoint[1],k,
    Class0/float(Totalpoints),Class1/float(Totalpoints))
p1 = (3,5)
p2 = (4,6)
GetClosePoints(p1,1)
GetClosePoints(p1,2)
GetClosePoints(p1,3)
GetClosePoints(p2,1)
GetClosePoints(p2,2)
GetClosePoints(p2,3)
\end{lstlisting}
What  I did in the previous code is:
\begin{enumerate}
	\item Store the indexes of points with same distance from our point under same hash label.
	\item Select the indexes depending on K
	\item Count total points,points in class0 , points in class1
	\item Print the probabilities
\end{enumerate}
Here is an image showing how the points look like :
\begin{figure}[H]
\includegraphics[scale=1]{points.jpg}
\caption{Points in the plane.}
\end{figure}
And here I report the output of the previous code :\\
For Point (3,5) with K=1,Probabilities is Class0=0.0,Class1=1.0\\
For Point (3,5) with K=2,Probabilities is Class0=0.333333333333,Class1=0.666666666667\\
For Point (3,5) with K=3,Probabilities is Class0=0.4,Class1=0.6\\
For Point (4,6) with K=1,Probabilities is Class0=1.0,Class1=0.0\\
For Point (4,6) with K=2,Probabilities is Class0=1.0,Class1=0.0\\
For Point (4,6) with K=3,Probabilities is Class0=0.75,Class1=0.25
\section*{Second Question}
For this question I build my own function to calculate the measurements from confusion matrix , here is the code:
\begin{lstlisting}[language=R]
measuresprint<-function(cm)
{
  Accuracy<-(cm[2,2]+cm[1,1])/sum(cm)
  Precision<-(cm[2,2])/(cm[2,2]+cm[1,2])
  Recall<-cm[2,2]/(cm[2,2]+cm[2,1])
  F1<-2*(Precision*Recall)/(Precision+Recall)
  print(c(Accuracy,Precision,Recall,F1))
}
###### Second Question ########
diabetes <- read.csv('pima-indians-diabetes.data',header = FALSE)
colnames(diabetes)<-c('PregnantTimes','glucos_constr','Blood_pressure','Triceps','insulin','BMI','diabts_pedigree','Age','Class')
#Shuffle The list so we pick randomly
diabetes<- diabetes[sample(nrow(diabetes)),]
traindata<-diabetes[seq(1,floor(nrow(diabetes)*0.8),1),]
test<-diabetes[seq(floor(nrow(diabetes)*0.8)+1,nrow(diabetes)),]
module <- glm(Class~.,data = traindata)
summary(module)
png('correlationhm.png')
heatmap(cor(diabetes),symm = TRUE, Colv=NA, Rowv=NA,col=colorRampPalette(c("red", "yellow", "green"))(n = 299))
dev.off()

#prediction_prop<-
prediction_bin<-ifelse(predict(module, test)<=0.5,0,1)
measuresprint(table(real=test$Class, predictions=prediction_bin))
\end{lstlisting}
The summary of the module output was :
\begin{lstlisting}
Coefficients:
                  Estimate Std. Error t value Pr(>|t|)
(Intercept)     -0.8270580  0.0952931  -8.679  < 2e-16 ***
PregnantTimes    0.0227378  0.0055950   4.064 5.46e-05 ***
glucos_constr    0.0058791  0.0005769  10.192  < 2e-16 ***
Blood_pressure  -0.0027814  0.0009006  -3.088  0.00211 **
Triceps         -0.0001970  0.0012530  -0.157  0.87511
insulin         -0.0002141  0.0001702  -1.258  0.20893
BMI              0.0135468  0.0022537   6.011 3.19e-09 ***
diabts_pedigree  0.1493781  0.0511308   2.921  0.00361 **
Age              0.0024483  0.0017115   1.430  0.15310
---
\end{lstlisting}
From the summary we can notice that "Plasma glucose concentration" which equals to "glucos\_constr" in table , that it's high significant to predict the class. Also we can see that it's highly correlated with the class.
\begin{figure}[H]
\includegraphics[scale=0.9]{correlationhm.png}
\caption{Heatmap for correlation}
\end{figure}
From the previous figure we can see that plasma glucos is the most correlated to the class.\\
For "diabetes pedigree" we can see from the table that it's 2 stars significant, and from the correlation picture we can see it's less correlated than plasma glucos.\\
To do the calcuation I printed out the confusion matrix \\
\begin{tabular}{|c|c|c|}
\hline
real\_ predic&0&1\\ \hline
0& 84 & 15 \\ \hline
   1 &22 &33\\ \hline
\end{tabular}
The rules for the measurements :
\[Accuracy = \frac{TP+TN}{Total} \]
\[Precision = \frac{TP}{TP+FP}\]
\[Recall = \frac{TP}{TP+FN} \]
F1 "is the harmonic mean of precision and sensitivity" \cite{1}
\[F1 =\frac{2TP}{2TP+FP+FN}\]
Another way to calculate it
\[F1 =2.\frac{precision*recall}{precision+recall}\]
The measurements :\[ Accuracy=0.7597403,Precision= 0.6875000,Recall= 0.6000000,F1= 0.6407767\]
\section*{Third Question}
For this question I used the function knn in R and here is the code :
\begin{lstlisting}[language=R]
####### Third Question#####
library(class)
knn1prediction<-knn(traindata,test = test,cl=traindata$Class)
measuresprint(table(real=test$Class, predictions=knn1prediction))
knn3prediction<-knn(traindata,test = test,cl=traindata$Class,k = 3)
measuresprint(table(real=test$Class, predictions=knn3prediction))
\end{lstlisting}
From the previous code we get the measurements for k1 and k3 and in the following table I put all the measurements for k1,k3 and glm to compare them\\
\begin{tabular}{|l|*{5}{c|}}
\hline
Method&Accuracy&Precision&Recall&F1\\ \hline
glm&0.7597403&0.6875000&0.6000000&0.6407767\\ \hline
K1&0.6948052&0.5769231&0.5454545&0.5607477\\ \hline
K3&0.7272727&0.6181818&0.6181818&0.6181818\\ \hline
\end{tabular}\\
From the previous table we can see that logit regression got the best Accuracy and the highest F1 score.
\section*{Fourth Question}
For this question I used the following code :
\begin{lstlisting}[language=R]
########## Fourth Question ###########
rm(list=ls())
setwd('/home/aqeel/Study/DM/HW10/')
library(ggplot2)
data("diamonds")
diamonds<- diamonds[sample(nrow(diamonds)),]
trainset<-diamonds[seq(1,floor(0.8*nrow(diamonds))),]
testset<-diamonds[seq(floor(0.8*nrow(diamonds))+1,nrow(diamonds)),]
module1<-lm(price~.,data = trainset)
module2<-lm(price~.+poly(carat,2)+poly(depth,2)-carat-depth,data = trainset)
module3<-lm(price~.+poly(carat,3)+poly(depth,3)-carat-depth,data = trainset)
module4<-lm(price~.+poly(carat,3)+poly(depth,3)+poly(x,2)+poly(y,2)+poly(z,2)-carat-depth-x-y-z,data = trainset)
module1predtrn<-predict(module1, trainset)
module2predtrn<-predict(module2, trainset)
module3predtrn<-predict(module3, trainset)
module4predtrn<-predict(module4, trainset)
module1predtst<-predict(module1, testset)
module2predtst<-predict(module2, testset)
module3predtst<-predict(module3, testset)
module4predtst<-predict(module4, testset)
#install.packages("qpcR")
library(qpcR)
trainRMSE<-c(sqrt(sum((module1predtrn-trainset$price)^2)/length(trainset$price)),
sqrt(sum((module2predtrn-trainset$price)^2)/length(trainset$price)),
sqrt(sum((module3predtrn-trainset$price)^2)/length(trainset$price)),
sqrt(sum((module4predtrn-trainset$price)^2)/length(trainset$price)))

testRMSE<-c(sqrt(sum((module1predtst-testset$price)^2)/length(testset$price)),
sqrt(sum((module2predtst-testset$price)^2)/length(testset$price)),
sqrt(sum((module3predtst-testset$price)^2)/length(testset$price)),
sqrt(sum((module4predtst-testset$price)^2)/length(testset$price)))
numbers<-seq(1,4,1)
png('train_test.png')
ggplot(data.frame(cbind(numbers,trainRMSE)),aes(numbers, trainRMSE))+geom_line(col="red") +
  geom_line(aes(numbers,testRMSE),data =data.frame(cbind(numbers,testRMSE)),col="blue" )+ xlab("Module")+
  ylab("RSME")
\end{lstlisting}

The previous code will output the following picture which represent the Train vs Test RMSE value.
\begin{figure}[H]
\includegraphics[scale=1]{train_test.png}
\caption{Train RMSE in Red , Test in blue}
\end{figure}
I would detect overfiting when the training RMSE keep decreasing while the testing RMSE start increasing. I believe we can see that from the plot after doing many iteration for the same module.With only one iteration it's hard to decide if we have overfiting mmmm maybe we can if the RMSE was pretty low for training and pretty high for testing.\\
Also I would notice from the plot that module 4 performed the best :).
\section*{Fifth Question}
For this question I used straight forward solution and here is the code :
\begin{lstlisting}[language=R]
rm(list=ls())
setwd('/home/aqeel/Study/DM/HW10/')
train <- read.csv('train.csv',header = TRUE)
module<-lm(data = train,target~.)
summary(train)
test<-read.csv('test.csv',header = TRUE)
head(test[,c(1,2)])
result<-predict(module,test)
output<-cbind(result)
colnames(output)<-c('ID','target')

write.csv(output,file='submit 001')
\end{lstlisting}
I got  	score 1.95523 for this code :)

\section*{Sixth Question}
\section*{Seventh Question}

The code for this question :
\begin{lstlisting}[language=R]
################ Seventh Question ############
library(MASS)
module4ridge<- lm.ridge(price~.+poly(carat,3)+poly(depth,3)+poly(x,2)+poly(y,2)+poly(z,2)-carat-depth-x-y-z,data = trainset)
module4ridge.trn.prd = as.matrix(model.matrix(price~.+poly(carat,3)+poly(depth,3)+poly(x,2)+poly(y,2)+poly(z,2)-carat-depth-x-y-z,trainset))%*% coef(module4ridge)
module4ridge.tst.prd = as.matrix(model.matrix(price~.+poly(carat,3)+poly(depth,3)+poly(x,2)+poly(y,2)+poly(z,2)-carat-depth-x-y-z,testset))%*% coef(module4ridge)

#install.packages("lars")
library(lars)
module4lasso <- lars(
  model.matrix(price~.+poly(carat,3)+poly(depth,3)+poly(x,2)+poly(y,2)+poly(z,2)-carat-depth-x-y-z,trainset),
  trainset$price, type="lasso",trace = TRUE, max.steps=20)
module4lasso.trn.prd <- predict(module4lasso,
                        model.matrix(price~.+poly(carat,3)+poly(depth,3)+poly(x,2)+poly(y,2)+poly(z,2)-carat-depth-x-y-z,trainset),
                        s=module4lasso$df[which.min(module4lasso$RSS)], type="fit")$fit

module4lasso.tst.prd <- predict(module4lasso,
                                model.matrix(price~.+poly(carat,3)+poly(depth,3)+poly(x,2)+poly(y,2)+poly(z,2)-carat-depth-x-y-z,testset),
                                s=module4lasso$df[which.min(module4lasso$RSS)], type="fit")$fit

trainRMSE<-c(sqrt(sum((module1predtrn-trainset$price)^2)/length(trainset$price)),
             sqrt(sum((module2predtrn-trainset$price)^2)/length(trainset$price)),
             sqrt(sum((module3predtrn-trainset$price)^2)/length(trainset$price)),
             sqrt(sum((module4predtrn-trainset$price)^2)/length(trainset$price)),
             sqrt(sum(( module4lasso.trn.prd- trainset$price)^2)/length(trainset$price)),
             sqrt(sum((module4ridge.trn.prd- trainset$price)^2)/length(trainset$price)))

testRMSE<-c(sqrt(sum((module1predtst-testset$price)^2)/length(testset$price)),
            sqrt(sum((module2predtst-testset$price)^2)/length(testset$price)),
            sqrt(sum((module3predtst-testset$price)^2)/length(testset$price)),
            sqrt(sum((module4predtst-testset$price)^2)/length(testset$price)),
            sqrt(sum(( module4lasso.tst.prd- testset$price)^2)/length(testset$price)),
            sqrt(sum((module4ridge.tst.prd- testset$price)^2)/length(testset$price)))
numbers<-seq(1,6,1)
png('all_modules.png')
ggplot(data.frame(cbind(numbers,trainRMSE)),aes(numbers, trainRMSE))+geom_line(col="red") +
  geom_line(aes(numbers,testRMSE),data =data.frame(cbind(numbers,testRMSE)),col="blue" )+ xlab("Module")+
  ylab("RSME")
dev.off()
\end{lstlisting}
The RMSE for lasso and ridge was quite high so it won't be much clear in the graph but here is the graph : \\
\begin{figure}[H]
\includegraphics[scale=1]{all_modules.png}
\caption{Last two values are lasso and ridge}
\end{figure}

\textbf{Please note:}Faiz helped a lot in this question.
\textbf{Note: }All code , ipython , images , etc.. exist on \href{https://github.com/aqeel13932/DM/tree/master/HW10}{github}
\begin{center}
\textbf{E.O.F}
\end{center}
\begin{thebibliography}{9}
	\bibitem{1}
	\href{https://en.wikipedia.org/wiki/Precision_and_recall}{Precision and Recall}
\end{thebibliography}
\end{document}


\chapter{Boosted Variational Bayesian Monte Carlo}

In this chapter, it is presented a modification of the Variational Bayesian Monte Carlo approach presented in Section \ref{vbmc_section}, using ideas of boosting presented in \ref{boostedvi_section}, along with some other minor modifications of the previous approach.

\section{Boosting Variational Bayesian Monte Carlo}

The idea underlying this approach is rather simple: instead of using a GP surrogate model for doing variational inference with mixtures of Gaussians as in \ref{vbmc_section}, use it for variational inference with the gradient boosting approach discussed in section \ref{boostedvi_section}.

To expand on this, consider again the algorithm in Figure \ref{vbalgorithm}. In line 5, the objective \eqref{relbogaussian} (here called \textit{$f$-step}) consists of three terms
\begin{displaymath}
 \begin{split}
 \text{RELBO}(\mu_i,\Sigma_i) = & \int \log(\gu(\theta)) \mathcal{N}(\theta|\mu,\Sigma) d\theta \\
 & - \int \log(q_{i-1}(\theta)) \mathcal{N}(\theta|\mu,\Sigma) d\theta \\
 & + \frac{\lambda}{4} \log |\Sigma|,
 \end{split}
\end{displaymath}
while in line 6, for the objective \eqref{boosting_objective_alpha} (here called \textit{$w$-step}), the gradient $\mathcal{L}'_i(w_i)$ consists of
\begin{displaymath}
\begin{split}
 \mathcal{L}'_i(w_i) = & \int \log (\gu(\theta)) f_{i}(\theta) \\
 & -\int \log(\gu(\theta)) q_{i-1}(\theta)\\
 & + \int \log((1-w_{i}) q_{i-1}(\theta) + w_{i} f_{i}(\theta)) (f_i(\theta) - q_{i-1}(\theta)) d\theta
\end{split}
\end{displaymath}

Then, in the same vein of the VBMC method, set a $GP(m,k)$ prior for $\log \gu$, and given evaluations $\mathcal{D}_0 = \{(x_i,f(x_i))\}_{i=1}^N$, substitute $\log \gu$ for $\Ev \log \gu_\mathcal{D}$. Since all the terms involving $\log \gu$ are expectations of either Gaussian distributions or mixture of Gaussian distributions, then one can approximate those by Bayesian Monte Carlo, as seen in \eqref{evvarbmcgaussian} and \eqref{bmcmixgaussians}, resulting in the maximization objective for the $f$-step:
\begin{equation}\label{relbo_vbmc}
\begin{split}
\text{RELBO}_\mathcal{D}(\mu_i,\Sigma_i) = & \int \Ev[\log \gu_\mathcal{D}(\theta)] \mathcal{N}(\theta|\mu_i,\Sigma_i) d\theta - \\ &\int \log(q_{i-1}(\theta)) \mathcal{N}(\theta|\mu_i,\Sigma_i) d\theta
+ \frac{\lambda}{4} \log |\Sigma_i|,
\end{split}
\end{equation}
and the maximization objective for the $w$-step
\begin{equation}\label{lalpha_vbmc}
\begin{split}
 \mathcal{L}_{i,\mathcal{D}}(w) = & \int \log \gu_\mathcal{D}(\theta)
 ((1-w_{i}) q_{i-1}(\theta) + w_{i} f_{i}(\theta)) d\theta - \\
 & \int \log ((1-w_{i}) q_{i-1}(\theta) + w_{i} f_{i}(\theta)) ((1-w_{i}) q_{i-1}(\theta) + w_{i} f_{i}(\theta)) d\theta \\
\end{split}
\end{equation}
with gradient
\begin{equation}
\begin{split}
\mathcal{L}'_{i,\mathcal{D}}(w) =& \int \Ev [\log \gu_\mathcal{D}(\theta)] f_{i}(\theta) - \int \Ev [\log \gu_\mathcal{D}(\theta)] q_{i-1}(\theta) + \\
& \int \log((1-w_{i}) q_{i-1}(\theta) + w_{i} f_{i}(\theta)) (f_i(\theta) - q_{i-1}(\theta)) d\theta,
\end{split}
\end{equation}
where
\begin{equation}
f_i(\theta) = \mathcal{N}(\theta|\mu_i,\Sigma_i), \quad q_{i-1}(\theta) = \sum_{k=0}^{i-1} w_k \mathcal{N}(\theta|\mu_k,\Sigma_k)
\end{equation}
The integrals not involving $\log \gu_\mathcal{D}$ can be easily approximated by the reparameterization trick, while $\log |\Sigma|$ is trivial to compute. This readily results in an algorithm for variational inference, given fixed evaluations points $\{x_n\}_{n=1}^N$, shown in Figure \ref{naivebvbmc}. Here it is named \textit{Naive Boosted Variational Bayesian Monte Carlo} (Naive BVBMC).
\begin{Algorithm}
\begin{algorithmic}[1]
	\Procedure{NaiveBVBMC}{$\log \gu,\mu_0$,$\Sigma_0$,$\{x\}_{n=1}^N$}
	\LineComment{$\mu_0,\Sigma_0$ the are initial boosting values}
	\For{$n=1,...,N$}
	\State $y_n := \log \gu(x_n)$
	\EndFor
	\State $\mathcal{D} := \{(x_n,y_n)\}_{i=1}^N$
	\State GPModel := PosteriorGP($m$,$k_{RBF}$,$\mathcal{D}$)
	\State GPModel.MaximizeLogLikelihood() \Comment{Using \eqref{loglikelihoodGP}}
	\State $\Ev[\log \gu_\mathcal{D}(\theta)]$ := GPModel.$m_\mathcal{D}$
	\State $w_0 := 1.0$
	\For{$t=1,...,T$}
	\LineComment{Using BMC and reparameterization}
	\State $\mu_{t},\Sigma_{t} := \argmax RELBO_\mathcal{D}(\mu_{t},\Sigma_{t})$
	\State $w_{t} := \argmax \mathcal{L}_{t,\mathcal{D}}(w_i)$ \Comment{Using $\mathcal{L}'_{t,\mathcal{D}}(w_t)$ for gradient descent}
	\For{$j=0,...,t-1$}
	\State $w_{j} \gets (1-w_t)w_j$
	\EndFor
	\EndFor
	\State \Return $\{(\mu_t,\Sigma_t,w_t)\}_{t=1}^T$
	\EndProcedure
\end{algorithmic}
\caption{\label{naivebvbmc} Naive boosted variational bayesian monte carlo}
\end{Algorithm}
\subsection{Practical issues}
The algorithm in Figure \ref{naivebvbmc} run into some practical issues, described below, that requires fixes of an heuristic nature. Here we present the heuristics that were used to make the algorithm stable. The resulting algorithm in found in \ref{bvbmcalgorithm}.

\subsubsection{RELBO stabilization}
By letting $r_\mathcal{D}(\theta) := \exp \Ev \log_\mathcal{D}(\theta)$, consider again the maximization objective \eqref{relbo_vbmc}, rewritten as
\begin{equation}
 \text{RELBO}_\mathcal{D}(\mu_i,\Sigma_i) = \int \log \left(\frac{r_\mathcal{D}(\theta)}{q_{i-1}(\theta)} \right) \mathcal{N}(\theta;\mu_i,\Sigma_i) d \theta + \log |\Sigma_i|.
\end{equation}
Assume $\Sigma_i$ fixed for now, so that the only term being optimized is $\mu_i$. Then, what the maximizer "wants" to do is set $\mu_i$ in a place that $\mathcal{N}(\theta;\mu_i,\Sigma_i)$ allocates probability mass where $\log r_\mathcal{D}(\theta)/q_{i-1}(\theta)$ is large. Now, consider the tail behavior of this quantity. We have that $q_{i-1}(\theta) \approx C_1 \exp(-||A_1(\theta-c_1)||_2^2)$ on the tail, while either $r_\mathcal{D}(\theta) \approx \exp(-C)$, if $m(\theta) = -C$, or $r_\mathcal{D}(\theta) \approx \exp(-||A_2(\theta-c_2)||_2^2)$, in case of $m(\theta) = -||A_2(\theta-c_2)||_2^2$. In the first case, then $\log r_\mathcal{D}(\theta)/q_{i-1}(\theta) \to \infty$ as $\theta \to \infty$, while in the second case, whether this happens depends on the relation between $A_1$ and $A_2$. However, if $\log r_\mathcal{D}(\theta)/q_{i-1}(\theta) \to \infty$ as $\theta \to \infty$, then the maximizer will try to get $\mu_i$ to $\infty$, thus resulting in bad proposals, that ends up with negligible weights $w_i$.

One approach to deal with this problem is to replace the term $\log r_\mathcal{D}/q_{i-1}(\theta)$ by $\log r_\mathcal{D}/(q_{i-1}(\theta)+\delta_D)$, where $\delta_D$ is a small positive constant. This approach is similar to the one in \cite{Guo_2016}, where a regularizer is added to both numerator and denominator, although it is done in the approximate heuristic proposed there. Thus, the $f$-step maximization objective becomes

\begin{equation}
\text{RELBO}^{\delta_D}_\mathcal{D}(\mu_i,\Sigma_i) = \int \log \left(\frac{r_\mathcal{D}(\theta)}{q_{i-1}(\theta)+\delta_D} \right) \mathcal{N}(\theta;\mu_i,\Sigma_i) d \theta + \log |\Sigma_i|.
\end{equation}

In the current work, $\delta_D = e^{-20}$ by default.

\subsubsection{Output scaling}\label{outputscaling}
One issue that arises with unnormalized log-densities is that one has no control on its scale. Fortunately, scaling the output can be done easily. Given data $\mathcal{D} = \{(x_i,y_i)\}_{i=1}^N$, suppose one makes a linear transformation $\tilde{y}_i = \frac{y_i-b}{a}$, and, letting $\tilde{\mathcal{D}} = \{x_i,\tilde{y}_i\}$, consider the GP posterior $f_{\tilde{\mathcal{D}}} \sim GP(m_{\tilde{\mathcal{D}}},k_{\tilde{\mathcal{D}}})$. Then, the random function $a f_{\tilde{\mathcal{D}}} + b$ is GP distributed, and
\begin{equation}
\int (a f_{\tilde{\mathcal{D}}} + b)(x) p(x) dx
\end{equation}
is a GP random variable, with
\begin{equation}
\Ev \left[\int (a f_{\tilde{\mathcal{D}}} + b)(x) p(x) dx \right] = a \Ev \left[ \int f_{\tilde{\mathcal{D}}}(x) p(x) dx \right] + b,
\end{equation}
and variance
\begin{equation}
\Var \left(\int (a f_{\tilde{\mathcal{D}}} + b)(x) p(x) dx \right) = a^2 \Var \left( \int f_{\tilde{\mathcal{D}}}(x) p(x) dx \right).
\end{equation}
This implies that one can do BMC, hence also VBMC and BVBMC using affinely scaled output variables, without difficulty.

Two heuristic for scaling were implemented, the \textit{normalize} heuristic, given by
\begin{itemize}
	\item Letting $y_i = \log g(x_i)$, calculating the sample mean $m_y = \frac{1}{N} \sum_{i=1}^N y_i$ and the (unbiased) sample standard deviation $\sigma_y = \sqrt{\frac{1}{N-1} \sum_{i=1}^N (y_i-m_y)}$.
	\item Transforming the output $\tilde{y}_i = (y_i-m_y)/\sigma_y$.
	\item Train a GP (along with hyperparameters) on $\tilde{\mathcal{D}} = \{x_i,\tilde{y}_i\}_{i=1}^N$.
	\item Substituting, when needed, $\log g_\mathcal{D}(x)$ (and their integrals) for $\sigma_y \log g_\mathcal{\tilde{D}}(x) + \mu_y$
\end{itemize}
and the \textit{zeromax} heuristic, where
\begin{itemize}
	\item Letting $y_i = \log g(x_i)$, calculating the sample maximum $m_y = \max\{y_i\}_{i=1}^N$ and making $\sigma_y = 1$.
	\item Do the other steps as in \textit{normalize} heuristic.
\end{itemize}
In toy problems presented in the next section, both heuristics worked well, but for more complex problems the \textit{zeromax} heuristic was found to be more stable.

\subsubsection{Component initialization}
A question that still wasn't addressed is on how to choose the first boosting component. Two options were tested:
\begin{itemize}
	\item Initializing the first mixture component with zero mean and a large covariance.
	\item Choosing the first component by maximizing the ELBO between the component and GP surrogate.
\end{itemize}
In general, the second method was chosen in this work, although performance in both cases was comparable.

\subsubsection{Component pruning}
When running the BVBMC algorithm, many mixture components may end up with negligible weights. This may both increase the computational cost of the algorithm, and hinder the performance of joint parameter updating (explained in \ref{jointparameterupdating}). In order to diminish this problem, an weight threshold $\beta$ may be established, so that, if $(w_1,\ldots,w_N)$ are the current mixture weights, every $i$-th mixture components with $w_i < \beta$ is removed, with the remainder weights being renormalized to one. This pruning may be done either every iteration, or only when doing joint parameter updating (for which pruning proved necessary).

\subsubsection{Mean functions}\label{meansection}
As discussed in Section \ref{quadmeanfnsection}, one option for the mean function set $m(\theta) = m_Q(\theta;l,c)$ as in \eqref{vbmc_quadratic_mean}. However, one issue that arose in this work is that optimizing $l,c$ by maximizing the log-likelihood often resulted in very large values (in absolute value) for $l$ and $c$, destabilizing the algorithm.

In this light, a different approach for the mean is proposed, by letting
\begin{equation}
m_F(\theta) = C
\end{equation}
where $C$ is a large negative constant, that is \textit{fixed before hyperparameter optimization}. Strictly speaking, in this case $\exp \Ev[\log \gu_\mathcal{D}(\theta)]$ is not anymore a probability distribution. However, if $C$ is sufficiently low, it resembles a probability distribution enough so that the algorithm works well in practice.

Since $C$ is not set by optimizing the log-likelihood, one must decide on how to set it. If the data was scaled beforehand using \textit{zeromax}, some good options that worked well in practice were $C = -10,-20,-30$. For scaling using \textit{normalize}, the following way to set $C$ is proposed:
\begin{equation}
C = \tilde{y}_{\min} - K_{C}(\tilde{y}_{\max} - \tilde{y}_{\min}),
\end{equation}
where $\tilde{y}_{\max}$ and $\tilde{y}_{\min}$ are the maximum and minimum of the normalized output values, and $K_C > 0$ is a constant. In this work, $K_C = 1$ was found to be a good choice.

\subsubsection{Periodic joint parameter updating}\label{jointparameterupdating}
Sometimes, boosting may get \enquote{stuck}, in the sense that the variational proposal takes too long to improve. In that case, one may desire to periodically update all variational parameters in the sense of the original VBMC algorithm. However, since this optimization is expensive, it should be used sparsely, for example every 10 boosting steps, or every 3 steps when boosting does not show improvement, when checking the ELBO.

\subsubsection{Other kernel functions}
As discussed in Section \ref{tensorprodbmc}, tensor product of one-dimensional kernels can be easily integrated with Bayesian Monte Carlo with diagonal covariance Gaussian distributions. In this light, by letting $k_{\text{Matern},\nu}(|x-x'|;l)$ be the 1-d Matérn kernel, the product of Matérn kernels is defined
\begin{equation}
 k_{\text{PMat},\nu}(x,x';\theta,l) = \theta \prod_{d=1}^D k_{\text{Matern},\nu}(|x_i-x_i'|;l_d).
\end{equation}
It must be noted that the product of Matérn kernels is \text{not} the Matérn kernel for $d > 1$, although it is a stationary kernel.

In experiments, the product of Matérn kernels tended to be more stable than the squared exponential kernel, particularly for $\nu = 1/2,3/2$, although for $\nu = 5/2$ the cases where it became unstable were rare. This may be due to the fact that the squared exponential kernels assumes far too much smoothness in its defined GP, hence estimating large oscillations outside the domain of interest, resulting in spurious multimodality.

\subsection{Other acquisition functions for active evaluation}
One problem that may arises in prospective prediction is that the term $\exp(m_\mathcal{D}(\theta_{N+1}))$ may become unstable for high values. Hence, one change proposed in this work is to substitute $\exp$ for the function $\text{softplus}(x) = \log(1+\exp(x))$, resulting in the \textit{soft prospective prediction}
\begin{equation}\label{soft_prospective_vbmc}
\alpha^\mathcal{D}_{PP}(\theta_{N+1}) = k_\mathcal{D}(\theta_{N+1},\theta_{N+1}) \text{softplus}(m_\mathcal{D}(\theta_{N+1}))q_k(\theta_{N+1};\lambda)^2.
\end{equation}
Another option is to disregard the current proposal altogether, and instead noticing that, ideally, the proposal is going to approximate the unnormalized posterior $\gu = \exp \log \gu$. Hence, one can take as an inspiration the warping approach in Section \ref{positivebmc} and use the \textit{moment-matched log transform} objective
\begin{equation}\label{mmlt_vbmc}
\alpha^m_{MMLT}(x_{m+1}) = e^{2 m_\mathcal{D}(x) + k_\mathcal{D}(x,x)} \left(e^{k_\mathcal{D}(x,x')}-1\right).
\end{equation}
Finally, one can adapt the uncertainty sampling approach to the warped approach, resulting in the \textit{prospective moment-matched log transform} objective
\begin{equation}\label{mmltprop_vbmc}
\alpha^m_{MMLT_P}(x_{m+1}) = e^{2 m_\mathcal{D}(x) + k_\mathcal{D}(x,x)} \left(e^{k_\mathcal{D}(x,x')}-1\right)q_k(\theta_{N+1};\lambda)^2.
\end{equation}
In general, the best performing acquisition functions were \eqref{prospective_vbmc} and \eqref{mmlt_vbmc}, and best results were obtained alternating between the two, either cyclically or randomly.

\begin{Algorithm}
\begin{algorithmic}[1]
	\Procedure{BoostedVBMC$_0$}{$\log \gu,\mu_0$,$\Sigma_0$,$\{x\}_{n=1}^N$,$\delta_D$}
	\LineComment{$\mu_0,\Sigma_0$ the are initial boosting values}
	\For{$n=1,...,N$}
	\State $y_n := \log \gu(x_n)$
	\EndFor
	\State $\mathcal{D}_0 := \{(x_n,y_n)\}_{i=1}^N$
	\State Make $\tilde{\mathcal{D}}$ from $\mathcal{D}$. Hold $m_y$ and $\sigma_y$. \Comment{Section \ref{outputscaling}}
	\State GPModel := PosteriorGP($m$,$k$,$\mathcal{D}$)
	\State GPModel.MaximizeLogLikelihood() \Comment{Using \eqref{loglikelihoodGP}}
	\State $\Ev[\log \gu_\mathcal{D}(\theta)]$ := GPModel.$m_\mathcal{D}$
	\State $\alpha_0 := 1.0$
	\State $(\mu_0,\Sigma_0)\ \gets \argmax \mathcal{L}_\mathcal{D}(\lambda)$ \Comment{Section \ref{outputscaling}}
	\For{$t=1,...,T$}
		\State $x' := \argmax \alpha^m(x)$ \Comment{Some acquisition function}
		\State $y' = \log \gu(x')$
		\State $\tilde{y}' = \frac{y - m_y}{\sigma_y}$
		\State $\tilde{\mathcal{D}} \gets \tilde{\mathcal{D}} \cup \{(x',\tilde{y}')\}$
		\LineComment{Using BMC and reparameterization}
		\State $\mu_{t},\Sigma_{t} := \argmax RELBO^{\delta_D}_\mathcal{D}(\mu_{t},\Sigma_{t})$
		\State $w_{t} := \argmax \mathcal{L}_{t,\mathcal{D}}(w_i)$ \Comment{Using $\mathcal{L}'_{t,\mathcal{D}}(w_t)$ for gradient descent}
		\For{$j=0,...,t-1$}
			\State $w_{j} \gets (1-w_t)w_j$
		\EndFor
		\If{conditions met}
			\State $\lambda = \{(\mu_j,\Sigma_j,w_j)\}_{j=1}^t \gets \argmax \mathcal{L}_\mathcal{D}(\lambda)$ \Comment{Section \ref{jointparameterupdating}}
		\EndIf
	\EndFor
	\State \Return $\{(\mu_t,\Sigma_t,w_t)\}_{t=1}^T$
	\EndProcedure
	\caption{\label{bvbmcalgorithm}Boosted Variational Bayesian Monte Carlo}
\end{algorithmic}
\end{Algorithm}

\section{Implementation}
The algorithm was implemented in Python, mainly using the PyTorch package \cite{Paszke_2017}. The reason for using PyTorch is that it combines strong automatic differentiation capabilities, allowing to avoid computing derivatives by hand, with easy usability when compared to other packages with similar strenghts such as TensorFlow. Since derivatives are necessary for the many inner optimization procedures in the algorithm,automatic differentiation greatly facilitated development.

Code can be found in \url{https://github.com/DFNaiff/BVBMC}. An example usage of the package is shown in Figure \ref{bvbmcpackexample}, with associated densities shown in \ref{bvbmcpackfigs}. \footnote{The author intends to change the package name used in development, \textit{variational\_boosting\_bmc}, to something akin to \textit{bvbmc}, but the change wasn't made at this document writing.}

\begin{figure}\label{bvbmcpackexample}
	\begin{lstlisting}
	#Import necessary packages
	import torch #PyTorch package
	from variational_boosting_bmc import VariationalBoosting #BVBMC package

	#Approximating unnormalized 2-d Cauchy
	def logjoint(theta):
		return torch.sum(-torch.log(1+theta**2))

	#Set up parameters
	dim=2 #Dimension of problem
	samples = torch.randn(20,dim) #Initial samples
	mu0 = torch.zeros(dim) #Initial mean
	cov0 = 20.0*torch.ones(dim) #Initial covariance
	acquisition = "prospective" #Acquisition function

	#Initialize algorithm
	vb = VariationalBoosting(dim,logjoint,samples,mu0,cov0)
	vb.optimize_bmc_model() #Optimize GP model
	vb.update_full() #Fit first component

	#Training loop
	for i in range(100):
		_ = vb.update() #Choose new boosting component
		vb.update_bmcmodel(acquisition=acquisition) #Choose new evaluation
		vb.cutweights(1e-3) #Weights prunning
		if ((i+1)%20) == 0:
			vb.update_full(cutoff=1e-3) #Joint parameter updating

	vb.save_distrib("finaldistrib") #Save distribution
	\end{lstlisting}
	\caption[Usage of the BVBMC Python package.]{\label{bvbmcpackexample}Usage of the BVBMC Python package, approximating a product of Cauchy distributions.}
\end{figure}

\subsection{Backpropagation}
Usually, in an algorithm with so many optimization steps, one would either have to worry about carefully keeping control of gradients, or resort to gradient free optimization. However, backpropagation \cite{Goodfellow_2016}, a technique from the class of automatic differentiation algorithms, allows the computer to provide arbitrary derivatives of functions, provided the forward history of the function is tracked.

A formal discussion on backpropagation is beyond the scope of this work. A good reference can be found in \cite{Goodfellow_2016}. Greatly simplifying, backpropagation hinges on the fact that almost every function in a computer comes from composition, so the chain rule can be applied. So, if the forward history of the composition is stored in a suitable data structure (a \textit{computational graph}), then, given the derivatives of those unit compositions, chain rule can be applied to get any derivative, without ever needing for then to being calculated by hand.

Backpropagation is ubiquitous in deep learning , and almost all packages dealing with neural networks implement some form of it. In fact, most of then are backpropagation packages, in which neural networks are build on top of it. Some of the most popular are Theano \cite{Theano_2016}, TensorFlow \cite{Abadi_2015} and PyTorch \cite{Paszke_2017}. Of those, PyTorch is the one with easiest usability, so it was chosen to implement the BVBMC package.

The usefulness of backpropagation and their related packages in general escapes the community outside deep learning. An important exception is in development of probability programming languages (PPL), whose many are built on top of deep learning packages, as in PyMC3 \cite{Salvatier_2016}, Edward \cite{Tran_2016} and Pyro \cite{Bingham_2018}, built on top of Theano, TensorFlow and PyTorch, respectively.

\begin{figure}
	\centering
	\subfloat[True density]{\label{fig11a}\includegraphics[width=0.4\linewidth]
		{figs/examplebvbmctrue.png}}
	\subfloat[Estimated density]{\label{fig11b}\includegraphics[width=0.4\linewidth]
	{figs/examplebvbmcestimated.png}}

	\caption[\label{bvbmcpackfigs}True density and estimated density found by running code in Figure \ref{bvbmcpackexample}]{\label{bvbmcpackfigs}True density (left) and estimated density (right), found by running code in Figure \ref{bvbmcpackexample}. Generating code can be found in \url{https://github.com/DFNaiff/Dissertation/blob/master/illustrations_dissertation/examplebvbmc.py}.}
\end{figure}


\documentclass{article}

% math stuff
\usepackage{amsmath}
\usepackage{amsthm}
\usepackage{amssymb}
\usepackage{xcolor}

\usepackage{mathtools}

\usepackage{float}
\usepackage{subcaption}

% to insert images
\usepackage{graphicx}

% to correctly insert stressed characters
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}

\usepackage{multirow}

% Bibliography
% \usepackage[style=alphabetic]{biblatex}
% \usepackage[nottoc]{tocbibind}
% \usepackage{bibentry}
% \setcounter{biburllcpenalty}{9000}
% \usepackage{nameref}

% to put links in table of contents
\usepackage{hyperref}
\hypersetup{colorlinks=false, %set true if you want colored links
	linktoc=all,     %set to all if you
}

% Add symbols
% \usepackage{textcomp}

% Add command for Real and Z sets
% \usepackage{dsfont}
% \newcommand{\Rset}{$\mathds{R}$}
% \newcommand{\Zset}{$\mathds{Z}$}

% Code highlighting
% \usepackage{minted}
% \usemintedstyle{perldoc}
% \setminted{
%     frame=single,
%     breaklines,
% }

% tikz figures
\usepackage{tikz}
\usepackage{tikzit}
\input{style.tikzstyles}
\usetikzlibrary{positioning}

\newtheorem{theorem}{Theorem}
\newtheorem{claim}[theorem]{Claim}

\begin{document}

\section*{Hardness of Echo Chamber Problem}%
\label{sec:np_hardness_of_echo_chamber_problem}

\begin{theorem}
	\label{th:hardness}
	Solving exactly the Echo Chamber Problem is $\mathcal{NP}$-hard.
\end{theorem}

\begin{proof}
	We show this by presenting a direct reduction from Maximum Clique, which is
	well-known to have the mentioned hardness factor.

	\bigskip
	Let $G_{1}  = (V_{1} ,E_{1} )$ be an undirected and unweighted graph and
	$\lambda \geq \frac{\alpha }{1 - \alpha }$, $\lambda \in \mathbb{N}$.
	We construct the \emph{interaction} graph ${G}_{2}  = (V_{2} , E^{+}_{2} , E
			^{-}_{2} ) $ as follows

	\begin{itemize}
		\item for each vertex $v_{i}  \in V_{1} $ we add a vertex in $G_{2} $
		\item for each edge $e_{ij}  \in
			      E_{1} $ add a positive edge between $v_{i} $ and $v_{j} $
		\item for each edge $e_{ij} \in V_{1} \times V_{1}, e_{ij}  \not\in
			      E_{1} $ add $\lambda n^{2}_{1}  $ negative edges between $v_{i} $ and $v_{j} $.
		\item add a vertex $v_x$ and $\lambda n_{1} $ negative edges between $v_x$
		      and each other vertex $v_i$ in $G_2$.
	\end{itemize}

	Furthermore, all the edges in $G_{2} $ are associated to the same content
	$C$ and the same thread $T \in \mathcal{T}_{C}  $.
	An illustration of the conversion can be found in \autoref{fig:construction}.

	\begin{figure}[hbt]
		\begin{center}
			\begin{subfigure}[b]{0.4\textwidth}
				\centering
				\caption{$G_{1}$, undirected graph}
				\tikzfig{tikz/hardness1}
				\label{fig:g1_example}
			\end{subfigure}
			\begin{subfigure}[b]{0.4\textwidth}
				\centering
				\tikzfig{tikz/hardness2}
				\caption{$G_{2}$, directed signed graph, for $\lambda = 1$}
				\label{fig:g2_example}
			\end{subfigure}
		\end{center}
		\caption{Example construction of the interaction graph $G_{2} $ from
			$G_{1} $, for $\alpha = \frac{1}{2} $}
		\label{fig:construction}
	\end{figure}

	\begin{claim}
		\label{th:claim-controversial}
		Content $C$ is controversial.
	\end{claim}
	\begin{proof}
		Let $m_{2}^{+} $ be the number of positive edges in $G_{2} $.

		By construction $m_{2}^{+} = m _{1} $ and $m_{2}^{-} \geq
			\lambda n_{1}^{2}  $ so
		\begin{align}
			\eta(C) = \frac{m_{2}^{-} }{m_{2}^{-} +
				m_{2}^{+} } \geq \frac{\lambda n_{1}^{2}}{\lambda n_{1}^{2}
				+ m_{1} } \geq \frac{\lambda n_{1}^{2}}{\lambda n_{1}^{2}
				+ n_{1}(n _{1} -1 )/2  } \geq \frac{\lambda n_{1}^{2}}{\lambda n_{1}^{2}
			+ n_{1}^{2} } = \\
			= \frac{\lambda }{\lambda + 1} =
			\frac{ \frac{\alpha }{1 - \alpha }  }{ \frac{\alpha }{1 - \alpha }
				+ 1 } \geq \alpha
		\end{align}
	\end{proof}

	This reduces the Echo Chamber Problem on $G_2$ to the maximization of

	\begin{equation}
		\label{eq:score}
		\xi(U) = \sum^{}_{T \in S_{C}(U) } | T[U] |
	\end{equation}

	\begin{claim}
		\label{th:claim-complete}
		The solution of the Echo Chamber Problem for $G_2$ is a set of vertices
		$\{ v_{ia} \} _{i \in I}  \subseteq V_{2} $ which is a clique
		in $G_{1} $.
	\end{claim}

	\begin{proof}
		Let $U \coloneqq \{ v_{ia} \} _{i \in I}  \subseteq V_{2} $ be the
		solution of the Echo Chamber Problem on $G_2$.

		We can assume $\xi(U) > 0$ (otherwise the proof is
		trivial)\footnote{In this case any subset of $V_{2} $ maximizes the echo
			chamber score, and this would clearly
			violate Claim~\ref{th:claim-complete}. For simplicity we can
			assume that in this case the algorithm returns a
			\emph{singleton}.}.
		It is also easy to see that $U$ does not contain $v_x$\footnote{Similarly
			to the proof of Claim~\ref{th:claim-controversial} it can be shown
			that if $v_x \in U$ then $T$ becomes controversial}.

		Now suppose that $U$ does not induce a complete subgraph on $G_1$. This
		means that there is at least one missing edge $e_{ij} \in V_1 \times
			V_1 $, $x_{ij} \not\in E_1$ and consequently at
		least $\lambda n^{2}_{1}  $ negative edges in $T[U]$. Let $n_U
			\coloneqq |U|$, then

		\begin{equation}
			\eta(T) \geq \frac{\lambda n_{1} ^{2} }{\lambda n_{1} ^{2} + n_U
				(n_U -1)/2 }
			\geq \frac{\lambda n_{1} ^{2} }{\lambda n_{1} ^{2} + n_{1} ^{2}  }
			= \frac{\lambda }{\lambda + 1} \geq \alpha
		\end{equation}

		Therefore, thread $T$ is controversial and does not
		contribute to the score $\implies \xi(U) = 0 \implies$ \emph{contradiction}.
	\end{proof}

	\begin{claim}
		\label{th:max-clique}
		The solution of the Echo Chamber Problem for $G_2$ is a set of vertices
		$U$ associated to a \emph{maximum clique} of $G_1$.
	\end{claim}

	\begin{proof}
		Suppose there is a set of vertices $\tilde{U} \neq U, |\tilde{U}| >
			|U|$ which is a clique for $G_1$. Then by construction it will
		contain only positive edge in $G_2$ and will be
		non-controversial. Also, being $|T[\tilde{U}]| > |T[U]| \implies
			\xi(\tilde{U}) > \xi(U) \implies contradiction$.
	\end{proof}

	\begin{claim}
		The set of vertices defining a maximum clique on $G_1$
		% corresponds to a  of the Echo Chamber Problem on $G_2$.
		corresponds to a solution of the Echo Chamber Problem on $G_2$.
	\end{claim}

	\begin{proof}
		Let $U \subseteq V_1$ be a set of nodes defining a maximum clique on
		$G_1$ and $n_{U} = |U|$. By construction $T[U]$ will not be controversial and

		\begin{equation}
			\xi(U) = |T[U]| = n_{U} (n_{U} -1 )/2
		\end{equation}

		Now suppose $\exists \tilde{U} \subseteq V_2 \; s.t. \; \xi(\tilde{U})
			> \xi(U)$. Due to Claim~\ref{th:claim-complete} $U_2$ induces a
		clique on $G_1$; consequently $T[\tilde{U}]$ has only positive
		edges and $\xi(\tilde{U}) = |T[\tilde{U}]| = n_{\tilde{U}}
			(n_{\tilde{U}} - 1)/2$.

		And since $\xi(\tilde{U}) > \xi({U}) \implies n_{\tilde{U}} > n_{U}
			\implies contradiction$.
	\end{proof}

	This concludes the proof of \autoref{th:hardness}.
\end{proof}

\end{document}


\chapter{Empirical Analysis}

\fbox{
    \parbox{\textwidth}
    {
        Chapter Overview
        \begin{itemize}
            \item Applying Bayesian \acp{cnn} for the task of Image Recognition on MNIST, CIFAR-10, CIFAR-100 and STL-10 datasets.
            \item Comparison of results of Bayesian \acp{cnn} with Normal \ac{cnn} architectures on similar datasets.
            \item Regularization effect of Bayesian Network with dropouts.
            \item Distribution of mean and variance in Bayesian \ac{cnn} over time.
            \item Parameters comparison before and after model pruning.
        \end{itemize}
    }
}

\pagebreak

\section{Experimentation Methodology} \label{experiments}

\subsubsection{Activation Function}

The originally chosen activation functions in all architectures are \textit{ReLU}, but we must introduce another, called \textit{Softplus}, see \eqref{softplus}, because of our method to apply two convolutional or fully-connected operations. As aforementioned, one of these is determining the mean $\mu$, and the other the variance $\alpha \mu^2$. Specifically, we apply the \textit{Softplus} function because we want to ensure that the variance $\alpha \mu^2$ never becomes zero. This would be equivalent to merely calculating the MAP, which can be interpreted as equivalent to a maximum likelihood estimation (MLE), which is further equivalent to utilising single point-estimates, hence frequentist inference. The \textit{Softplus} activation function is a smooth approximation of \textit{ReLU}. Although it is practically not influential, it has the subtle and analytically important advantage that it never becomes zero for $x \rightarrow -\infty$, whereas \textit{ReLU} becomes zero for $x \rightarrow -\infty$.
\\
\begin{equation}\label{softplus}
     \text{Softplus}(x) = \frac{1}{\beta} \cdot \log \big ( 1 + \exp(\beta \cdot x) \big )
\end{equation}
\\
where $\beta$ is by default set to $1$.
\newline All experiments are performed with the same hyper-parameters settings as stated in the Appendix.

\subsubsection{Network Architecture}

For all conducted experiments, we implement the foregoing description of Bayesian \acp{cnn} with variational inference in LeNet-5 \cite{lecun1998gradient} and AlexNet \cite{krizhevsky2012imagenet}. The exact architecture specifications can be found in the Appendix and in our GitHub repository\footnote{\url{https://github.com/kumar-shridhar/PyTorch-BayesianCNN}}.

\subsubsection{Objective Function}
To learn the objective function, we use \textit{Bayes by Backprop} \cite{graves2011practical, blundell2015weight}, which is a variational inference method to learn the posterior distribution on the weights $w \sim q_{\theta}(w|\mathcal{D})$ of a neural network from which weights $w$ can be sampled in backpropagation.
It regularises the weights by minimising a compression cost, known as the variational free energy or the expected lower bound on the marginal likelihood.

 We tackled the problem of intractability in Chapter 2 and consequently, we arrive at the tractable cost function \eqref{cost} which is aimed to be optimized, i.e. minimised w.r.t. $\theta$, during training:
\begin{equation} \label{cost}
    \mathcal{F}(\mathcal{D}, \theta)\approx \sum_{i=1}^n \log q_{\theta}(w^{(i)}|\mathcal{D})-\log p(w^{(i)})-\log p(\mathcal{D}|w^{(i)})
\end{equation}
%
where $n$ is the number of draws.

Let's break the Objective Function \eqref{cost} and discuss in more details.
\subsubsection{Variational Posterior }

The first term in the equation \eqref{cost} is the variational posterior. The variational posterior is taken as Gaussian distribution centred around mean $\mu$ and variance as $\sigma^2$.

\begin{equation}
    q_{\theta}(w^{(i)}|\mathcal{D})= \prod_{i} \mathcal{N}(w_{i} | \mu,\sigma^2)
\end{equation}

We will take the log and the log posterior is defined as :

\begin{equation}
    log(q_{\theta}(w^{(i)}|\mathcal{D}))= \sum_{i}log \mathcal{N}(w_{i} | \mu,\sigma^2)
\end{equation}

\subsubsection{Prior}

The second term in the equation \eqref{cost} is the prior over the weights and we define the prior over the weights as a product of individual Gaussians :

\begin{equation}
    p(w^{(i)})= \prod_{i} \mathcal{N}(w_{i} | 0,\sigma_{p}^2)
\end{equation}

We will take the log and the log prior is defined as:

\begin{equation}
    log (p(w^{(i)}))= \sum_{i} log \mathcal{N}(w_{i} | 0,\sigma_{p}^2)
\end{equation}

\subsubsection{ Likelihood }

The final term of the equation \eqref{cost} $\log p(\mathcal{D}|w^{(i)})$ is the likelihood term and is computed using the softmax function.

\subsubsection{Parameter Initialization}

We use a Gaussian distribution and we store mean and variance values instead of just one weight. The way mean $\mu$ and variance $\sigma$ is computed is defined in the previous chapter. Variance cannot be negative and it is ensured by using \textit{softplus} as the activation function. We express variance $\sigma$ as $\sigma_{i}=softplus(\rho_{i})$ where $\rho$ is an unconstrained parameter. \\

We take the Gaussian distribution and initialize mean $\mu$ as 0 and variance $\sigma$ (and hence $\rho$) randomly. We observed mean centred around 0 and a variance starting with a big number and gradually decreasing over time. A good initialization can also be to put a restriction on variance and initialize it small. However, it might be data dependent and a good method for variance initialization is still to be discovered. We perform gradient descent over $\theta$ = ($\mu$, $\rho$), and individual weight $w_{i} \sim \mathcal{N} (w_{i} | \mu_{i}, \sigma_{i}$).

\subsubsection{Optimizer}

For all our tasks, we take Adam optimizer \cite{kingma2014adam} to optimize the parameters. We also perform the local reparameterization trick as mentioned in the previous section and take the gradient of the combined loss function with respect to the variational parameters ($\mu$, $\rho$).

\subsubsection{Model Pruning}

We take the weights of all the layers of the network, apply an L1 norm over it and for all the weights value as zero or below a defined threshold are removed and the model is pruned. \\ Also, since the Bayesian \acp{cnn} has twice the number of parameters ($\mu$, $\sigma$) compared to a frequentist network (only 1 weight), we reduce the size of our network to half (AlexNet and LeNet- 5) by reducing the number of filters to half. The architecture used is mentioned in the Appendix.

\textit{Please note that it can be argued that reducing the number of filters to be half is a method for pruning or not. It can be seen as a method that reduces the number of overall parameters and hence can be thought of a pruning method in some sense. However, it is a subject to argument.}

\section{Case Study 1: Small Datasets (MNIST, CIFAR-10)}

We train the networks with the MNIST dataset of handwritten digits \cite{lecun1998gradient}, and CIFAR-10 dataset \cite{krizhevsky2009learning} since these datasets serve widely as benchmarks for \acp{cnn}' performances.

\subsection{Datasets}
\newline
\subsubsection{MNIST}
The MNIST database \cite{lecun-mnisthandwrittendigit-2010} of handwritten digits have a training set of 60,000 examples and a test set of 10,000 examples. It is a subset of a larger set available from NIST. The digits have been size-normalized and centred in a fixed-size image of 28 by 28 pixels. Each image is grayscaled and is labelled with its corresponding class that ranges from zero to nine.
\newline

\subsubsection{CIFAR-10}
The CIFAR-10 are labelled subsets of the 80 million tiny images dataset \cite{Torralba:2008:MTI:1444381.1444403}. The CIFAR-10 dataset has a training dataset of 50,000 colour images in 10 classes, with 5,000 training images per class, each image 32 by 32 pixels large. There are 10000 images for testing.
\newline

\subsection{Results}
First, we evaluate the performance of our proposed method, Bayesian \acp{cnn} with variational inference. Table \ref{tab:results} shows a comparison of validation accuracies (in percentage) for architectures trained by two disparate Bayesian approaches, namely variational inference, i.e. \textit{Bayes by Backprop} and Dropout as proposed by Gal and Ghahramani \cite{gal2015bayesian}.\\

We compare the results of these two approaches to frequentist inference approach for both the datasets. Bayesian \acp{cnn} trained by variational inference achieve validation accuracies comparable to their counter-architectures trained by frequentist inference. On MNIST, validation accuracies of the two disparate Bayesian approaches are comparable, but a Bayesian LeNet-5 with Dropout achieves a considerable higher validation accuracy on CIFAR-10, although we were not able to reproduce these reported results.
\begin{table}[H]
\tiny
    \centering
    \renewcommand{\arraystretch}{1.5}
    \resizebox{\linewidth}{!}{
    \begin{tabular}{ l  c  c  c  c }
     \hline
      \empty & MNIST & CIFAR-10   \\ [0.75ex]
     \hline
     Bayesian AlexNet (with VI) & 99 & 73   \\

     Frequentist AlexNet & 99 & 73   \\
     \hdashline
     Bayesian LeNet-5 (with VI) & 98 & 69   \\

     Frequentist LeNet-5 & 98 & 68   \\
     \hdashline
     Bayesian LeNet-5 (with Dropout) & 99.5 & 83  \\
     \hline \\
    \end{tabular}}
    \renewcommand{\arraystretch}{1.5}
    \caption{Comparison of validation accuracies (in percentage) for different architectures with variational inference (VI), frequentist inference and Dropout as a Bayesian approximation as proposed by Gal and Ghahramani \cite{gal2015bayesian} for MNIST, and CIFAR-10.}
    \label{tab:results}
\end{table}

\begin{figure}[t!]
\begin{center}
\includegraphics[width=\linewidth]{Chapter5/Figs/results_mnist_CIFAR10.png}
\caption{Comparison of Validation Accuracies of Bayesian AlexNet and LeNet-5 with frequentist approach on MNIST and CIFAR-10 datasets}
\label{fig:MnistCIFAR10reesults}
\end{center}
\end{figure}

\newline Figure \ref{fig:MnistCIFAR10reesults} shows the validation accuracies of Bayesian vs Non-Bayesian \acp{cnn}. One thing to observe is that in initial epochs, Bayesian \acp{cnn} trained by variational inference start with a low validation accuracy compared to architectures trained by frequentist inference. This must deduce from the initialization of the variational posterior probability distributions $q_{\theta}(w|\mathcal{D})$ as uniform distributions, while initial point-estimates in architectures trained by frequentist inference are randomly drawn from a standard Gaussian distribution. (For uniformity, we changed the initialization of frequentist architectures from Xavier initialization to standard Gaussian). The latter initialization method ensures the initialized weights are neither too small nor too large. In other words, the motivation of the latter initialization is to start with weights such that the activation functions do not let them begin in saturated or dead regions. This is not true in case of uniform distributions and hence, Bayesian \acp{cnn}' starting validation accuracies can be comparably low.

\pagebreak
\newline Figure \ref{fig:std_CNN} displays the convergence of the standard deviation $\sigma$ of the variational posterior probability distribution $q_{\theta}(w|\mathcal{D})$ of a random model parameter over epochs. As aforementioned, all prior probability distributions $p(w)$ are initialized as uniform distributions. The variational posterior probability distributions $q_{\theta}(w|\mathcal{D})$ are approximated as Gaussian distributions which become more confident as more data is processed - observable by the decreasing standard deviation over epochs in Figure \ref{fig:std_CNN}. Although the validation accuracy for MNIST on Bayesian LeNet-5 has already reached 99\%, we can still see a fairly steep decrease in the parameter's standard deviation. In Figure \ref{fig:distribution}, we plot the actual Gaussian variational posterior probability distributions $q_{\theta}(w|\mathcal{D})$ of a random parameter of LeNet-5 trained on CIFAR-10 at some epochs.
%
\begin{figure}[H]
\begin{center}
\includegraphics[width=\linewidth]{Chapter5/Figs/std_CNN.png}
\caption{Convergence of the standard deviation of the Gaussian variational posterior probability distribution $q_{\theta}(w|\mathcal{D})$ of a random model parameter at epochs 1, 5, 20, 50, and 100. MNIST is trained on Bayesian LeNet-5.}
\label{fig:std_CNN}
\end{center}
\end{figure}
%

\begin{figure}[H]
\begin{center}
\includegraphics[width=\linewidth]{Chapter5/Figs/distribution.png}
\caption{Convergence of the Gaussian variational posterior probability distribution $q_{\theta}(w|\mathcal{D})$ of a random model parameter at epochs 1, 5, 20, 50, and 100. CIFAR-10 is trained on Bayesian LeNet-5.}
\label{fig:distribution}
\end{center}
\end{figure}

\newline Figure \ref{fig:distribution} displays the convergence of the Gaussian variational probability distribution of a weight taken randomly from the first layer of LeNet-5 architecture. The architecture is trained on CIFAR-10 dataset with uniform initialization.

\section{Case Study 2: Large Dataset (CIFAR-100)}
\subsection{Dataset}

\subsubsection{CIFAR-100}
This dataset is similar to the CIFAR-10 and is a labelled subset of the 80 million tiny images dataset \cite{Torralba:2008:MTI:1444381.1444403}. The dataset has 100 classes containing 600 images each. There are 500 training images and 100 validation images per class. The images are coloured with a resolution of 32 by 32 pixels.

\subsection{Results}

\begin{table}[H]
\tiny
    \centering
    \renewcommand{\arraystretch}{1.5}
    \resizebox{\linewidth}{!}{
    \begin{tabular}{ l  c  c  }
     \hline
      \empty & CIFAR-100  \\ [0.75ex]
     \hline
     Bayesian AlexNet (with VI)  & 36  \\

     Frequentist AlexNet & 38  \\

     Bayesian LeNet-5 (with VI) &  31  \\

     Frequentist LeNet-5  & 33  \\
     \hline \\
    \end{tabular}}
    \renewcommand{\arraystretch}{1.5}
    \caption{Comparison of validation accuracies (in percentage) for different architectures with variational inference (VI), frequentist inference and Dropout as a Bayesian approximation as proposed by Gal and Ghahramani \cite{gal2015bayesian} for MNIST, CIFAR-10, and CIFAR-100.}
    \label{tab:resultsCIFAR-100}
\end{table}

In Figure \ref{fig:regularization}, we show how Bayesian networks incorporate naturally effects of regularization, exemplified on AlexNet. While an AlexNet trained by frequentist inference without any regularization overfits greatly on CIFAR-100, an AlexNet trained by Bayesian inference on CIFAR-100 does not. This is evident from the high value of training accuracy for frequentist approach with no dropout or 1 layer dropout. Bayesian CNN performs equivalently to an AlexNet trained by frequentist inference with three layers of Dropout after the first, fourth, and sixth layers in the architecture.
Another thing to note here is that the Bayesian CNN with 100 samples overfits slightly lesser compared to Bayesian CNN with 25 samples. However, a higher sampling number on a smaller dataset didn't prove useful and we stuck with 25 as the number of samples for all other experiments.

\begin{figure}[H]
\begin{center}
\includegraphics[width=\linewidth]{Chapter5/Figs/results_train_test_cifar100.png}
\caption{Comparison of Training and Validation Accuracies of Bayesian AlexNet and LeNet-5 with frequentist approach with and without dropouts on CIFAR-100 datasets}
\label{fig:regularization}
\end{center}
\end{figure}

Table \ref{tab:tableCIFAR100} shows a comparison of the training and validation accuracies for AlexNet with Bayesian approach and frequentist approach. The low gap between the training and validation accuracies shows the robustness of Bayesian approach towards overfitting and shows how Bayesian approach without being regularized overfits lesser as compared to frequentist architecture with no or one dropout layer. The results are comparable with AlexNet architecture with 3 dropout layers.

\begin{table}[H]
\tiny
    \centering
    \renewcommand{\arraystretch}{1.5}
    \resizebox{\linewidth}{!}{
    \begin{tabular}{ l  c  c  c  c }
     \hline
      \empty & Training Accuracy & Validation Accuracy   \\ [0.75ex]
     \hline
     Frequentist AlexNet (No dropout) & 83 & 38   \\

     Frequentist AlexNet (1 dropout layer) & 72 & 40   \\

      Frequentist AlexNet (3 dropout layer) & 39 & 38   \\
     \hdashline
     Bayesian AlexNet (25 num of samples) & 54 & 37   \\

     Bayesian AlexNet (100 num of samples) & 48 & 37   \\
     \hline \\
    \end{tabular}}
    \renewcommand{\arraystretch}{1.5}
    \caption{Comparison of training and validation accuracies (in percentage) for AlexNet architecture with variational inference (VI) and frequentist inference for CIFAR-100.}
    \label{tab:tableCIFAR100}
\end{table}

\section{Uncertainity Estimation}

\newline Finally, Table \ref{tab:uncertainty} compares the means of aleatoric and epistemic uncertainties for a Bayesian LeNet-5 with variational inference on MNIST and CIFAR-10. The aleatoric uncertainty of CIFAR-10 is about twenty times as large as that of MNIST. Considering that the aleatoric uncertainty measures the irreducible variability and depends on the predicted values, a larger aleatoric uncertainty for CIFAR-10 can be directly deduced from its lower validation accuracy and may be further due to the smaller number of training examples. The epistemic uncertainty of CIFAR-10 is about fifteen times larger than that of MNIST, which we anticipated since epistemic uncertainty decreases proportionally to validation accuracy.
\begin{table}[H]
\tiny
    \centering
    \renewcommand{\arraystretch}{1.5}
    \resizebox{\linewidth}{!}{
    \begin{tabular}{ l  c  c  c  }
     \hline
      \empty & Aleatoric uncertainty &  Epistemic uncertainty  \\ [0.75ex]
     \hline
     Bayesian LeNet-5 (MNIST) & 0.0096 & 0.0026   \\

     Bayesian LeNet-5 (CIFAR-10) & 0.1920 & 0.0404   \\
     \hline \\
    \end{tabular}}
    \renewcommand{\arraystretch}{1.5}
    \caption{Aleatoric and epistemic uncertainty for Bayesian LeNet-5 calculated for MNIST and CIFAR-10, computed as proposed by Kwon et al. \cite{kwon2018uncertainty}.}
    \label{tab:uncertainty}
\end{table}

\section{Model Pruning}

\subsubsection{Halving the Number of Filters}

For every parameter for a frequentist inference network, Bayesian \acp{cnn} has two parameters ($\mu$, $\sigma$). Halving the number of parameters of Bayesian AlexNet ensures the number of parameters of it is comparable with a frequentist inference network. The number of filters of ALexNet is halved and a new architecture called AlexNetHalf is defined in Figure 5.4.

\begin{table}[h!]
    \centering
    \renewcommand{\arraystretch}{2}
    \begin{tabular}{c c c c c c}
 \hline
 layer type & width & stride & padding & input shape & nonlinearity \\ [0.5ex]
 \hline
 convolution ($11\times11$) & 32 & 4 & 5 & $M\times3\times32\times32$ & Softplus \\

 max-pooling ($2\times2$) & \empty & 2 & 0 & $M\times32\times32\times32$ & \empty \\

 convolution ($5\times5$) & 96 & 1 & 2 & $M\times32\times15\times15$ & Softplus \\

 max-pooling ($2\times2$) & \empty & 2 & 0 & $M\times96\times15\times15$ & \empty \\

 convolution ($3\times3$) & 192 & 1 & 1 & $M\times96\times7\times7$ & Softplus \\

 convolution ($3\times3$) & 128 & 1 & 1 & $M\times192\times7\times7$ & Softplus \\

 convolution ($3\times3$) & 64 & 1 & 1 & $M\times128\times7\times7$ & Softplus \\

 max-pooling ($2\times2$) & \empty & 2 & 0 & $M\times64\times7\times7$ & \empty \\

 fully-connected & 64 & \empty & \empty & $M\times64$ & \empty \\ [1ex]
 \hline
\end{tabular}
\renewcommand{\arraystretch}{1.5}
\label{tab:AlexNetHalfArchitecture}
\caption{AlexNetHalf with number of filters halved compared to the original architecture.}
\end{table}

The AlexNetHalf architecture was trained and validated on the MNIST, CIFAR10 and CIFAR100 dataset and the results are shown in Table \ref{tab:resultsAlexNetHalf}. The accuracy of pruned AlexNet with only half the number of filters compared to the normal architecture shows an accuracy gain of 6 per cent in case of CIFAR10 and equivalent performance for MNIST and CIFAR100 datasets. A lesser number of filters learn the most important features which proved better at inter-class classification could be one of the explanations for the rise in accuracy. However, upon visualization of the filters, no distinct clarification can be made to prove the previous statement. \\
Another possible explanation could be the model is generalizing better after the reduction in the number of filters ensuring the model is not overfitting and validation accuracy is comparatively higher. CIFAR-100 higher validation accuracy on ALexNetHalf and a lower training accuracy than Bayesian AlexNet proves the theory. Using a lesser number of filters further enhances the regularization effect and makes the model more robust against overfitting. Similar results have been achieved by Narang \cite{DBLP:journals/corr/NarangDSE17} in his work where a pruned model achieved better accuracy compared to the original architecture in a speech recognition task. Suppressing or removing the weights that have lesser or no contribution to the prediction makes the model rely its prediction on the most prominent and unique features and hence improves the prediction accuracy.

\begin{table}[H]
\tiny
    \centering
    \renewcommand{\arraystretch}{1.5}
    \resizebox{\linewidth}{!}{
    \begin{tabular}{ l  c  c  c  c }
     \hline
      \empty & MNIST & CIFAR-10 & CIFAR-100 \\ [0.75ex]
     \hline
     Bayesian AlexNet (with VI) & 99 & 73 & 36 \\

     Frequentist AlexNet & 99 & 73 & 38  \\

     Bayesian AlexNetHalf (with VI) & 99 & 79 & 38 \\

     \hline \\
    \end{tabular}}
    \renewcommand{\arraystretch}{1.5}
    \caption{Comparison of validation accuracies (in percentage) for AlexNet with variational inference (VI), AlexNet with frequentist inference and AlexNet with half number of filters halved for MNIST, CIFAR-10 and CIFAR-100 datasets.}
    \label{tab:resultsAlexNetHalf}
\end{table}

\subsubsection{Applying L1 Norm}

L1 norm induces sparsity in the trained model parameters and sets some values to zero. We trained a model to some epochs (number of epochs differs across datasets as we applied early stopping when validation accuracy remains unchanged for 5 epochs). We removed the zero-valued parameters of the learned weights and keep the non-zero parameters for a trained Bayesian AlexNet on MNIST and CIFAR-10 datasets. We pruned the model to make the number of parameters in a Bayesian Network comparable to the number of parameters in the point-estimate architecture. \\ Table \ref{tab:resultsL1Norm} shows the comparison of validation accuracies of the applied L1 Norm AlexNet Bayesian architecture with Bayesian AlexNet architecture and with AlexNet frequentist architecture. We got comparable results on MNIST and CIFAR10 with the experiments and the results are shown in Table \ref{tab:resultsL1Norm}

\begin{table}[H]
\tiny
    \centering
    \renewcommand{\arraystretch}{1.5}
    \resizebox{\linewidth}{!}{
    \begin{tabular}{ l  c  c  c  }
     \hline
      \empty & MNIST & CIFAR-10  \\ [0.75ex]
     \hline
     Bayesian AlexNet (with VI) & 99 & 73  \\

     Frequentist AlexNet & 99 & 73   \\

     Bayesian AlexNet with L1 Norm (with VI) & 99 & 71  \\

     \hline \\
    \end{tabular}}
    \renewcommand{\arraystretch}{1.5}
    \caption{Comparison of validation accuracies (in percentage) for AlexNet with variational inference (VI), AlexNet with frequentist inference and BayesianAlexNet with L1 norm applied for MNIST and CIFAR-10 datasets.}
    \label{tab:resultsL1Norm}
\end{table}

One thing to note here is that the numbers of parameters of Bayesian Network after applying L1 norm is not necessarily equal to the number of parameters in the frequentist AlexNet architecture. It depends on the data size and the number of classes. However, the number of parameters in the case of MNIST and CIFAR-10 are pretty comparable and there is not much reduction in the accuracy either. Also, the early stopping was applied when there is no change in the validation accuracy for 5 epochs and the model was saved and later pruned with the application of L1 norm.

\section{Training Time}

Training time of a Bayesian \acp{cnn} is twice of a frequentist network with similar architecture when the number of samples is equal to one. In general, the training time of a Bayesian \acp{cnn}, $T$ is defined as:
\begin{align}
T = 2 * number\_of\_samples * t
\end{align}
where $t$ is the training time of a frequentist network.
The factor of 2 is present due to the double learnable parameters in a Bayesian CNN network i.e. mean and variance for every single point estimate weight in the frequentist network.

However, there is no difference in the inference time for both the networks.

\ifpdf
    \graphicspath{{Chapter2/Figs/Raster/}{Chapter2/Figs/PDF/}{Chapter2/Figs/}}
\else
    \graphicspath{{Chapter2/Figs/Vector/}{Chapter2/Figs/}}
\fi


\section{Overlay Mesh}

\subsection{Self–Optimisation}
Nodes in the Mitosis network constantly try to improve their own connections. As discussed in \vref{sec:design-peers}, this should lead to a more stable system in general.

In this experiment, the positioning of peers in relation to the router is measured. Running a simulation with $n=200$ nodes for $t=1000$ ticks, the distance from each peer to the router is put in relation to the simulated network configuration of the peer. In this scenario, connection latency is configured to a random value in the range of $0.125<l<0.5$ ticks, connection establishment delay falls in the range of $0.1<1.0$ ticks and the drop probability for messages on the unreliable channel \cref{sec:mit-connections} is set to $0.0<s<5.0$ percent. For simplicity, every peer is assigned values within these ranges at instantiation and those values remain static throughout the simulation.

As peers strive to acquire connections to peers with a good router link quality and a good connection quality, the hypothesis is that peers with good connection qualities will \say{bubble} towards the router. For verification, at the end of the scenario, every peer reports its network configuration parameters along with its distance to the router node. Distances are calculated in hops using the Dijkstra algorithm and treating the mesh as an undirected and unweighted graph.
\cref{fig:connection-quality-per-distance} shows the histogram of network quality settings binned by router distance. For comparability, the settings are normalised to percentages corresponding to the aforementioned ranges, with higher percentages representing the \textit{better} end of the range.

\begin{figure}[htb!]
\centering
\includegraphics[width=1.0\textwidth]{graphics/analysis/connection-quality-per-distance.pdf}
\caption{Network quality of nodes in relation to their router distance}
\label{fig:connection-quality-per-distance}
\end{figure}

As to be expected, establishment delays do not show a dependency on router distance. Nodes only begin metering the quality of a connection after it has been opened. In fact, this is a desirable behaviour, since \gls{webrtc} connections can take a while to generate \gls{ice} candidates and to establish. Yet, the grade of the \gls{webrtc} connection does not relate to how long the establishment took.

The distribution of average connection latency per router distance is in line with the hypothesis. Nodes with below average latency (circa $l<0.27$) have acquired direct connections to the \router node, while nodes with only half as responsive connections have been pushed outwards to be leaf nodes.

The stability for the unreliable channel shows a distribution proximate to the latency, but not as clear. Nodes with low message drop probabilities, accumulate to distance levels one and two, yet the falloff is not as distinctive. This can be explained by looking back at the utilisation of the unreliable channel in \vref{par:webrtc-data-measure-quality}: The \gls{tq} of the connection meter is intended to access the duplexity of the connection and the responsiveness of the peer. However, in the simulation, the drop probability is applied to both incoming and outgoing messages.

\subsection{Ingress Rates}

For a \gls{p2p} network to reach its potential scale, not only does it need to retain a high number of peers, it needs to take them in at a high rate. In real–world applications users can join and exit in unpredictable bursts, which can cause queues at the entry point or leaf gaps in the middle of the mesh.

The Mitosis mesh design envisions a clustered landscape \cref{par:scaling-cluster}, where peers are associated to a router by a signal server. These router peers are the only way into the mesh and naturally have a maximum rate at which they can absorb new peers and forward them into the mesh. In real–world applications, the ingress rate would depend on the network and computing capabilities of the router peer, as well as the size and structure of its mesh. For the purposes of identifying the ingress rate under optimal conditions, these factors are intentionally left untouched.

A new mesh is created by spawning nodes at a static rate and metering how fast they are connected to at least one peer of the mesh (excluding the signal server). Spawn rates were set to $3.2<r<12.5$, while the maximum node amount was set to $n=100$. \Vref{fig:ingress-rates} shows three distinct runs of this experiment, where the dotted lines show the number of spawned nodes and the solid line shows the number of successfully absorbed nodes. A clear disparity can be seen between the lines for $r=12.5$. This hints towards multiple nodes queuing up at the entry point. For $r=3.2$, the network's ingress seems to lag only a few nodes behind the spawn rate. These lines run in parallel, indicating that apart from nodes currently in transition, no queue is forming. So, for the default configuration of the \textit{Mitosis} core library and in a flawless network setting, this would be the optimal value.

\begin{figure}[htb!]
\centering
\includegraphics[width=1.0\textwidth]{graphics/analysis/ingress-final.pdf}
\caption{Ingress rates in relation to join rates in nodes per tick}
\label{fig:ingress-rates}
\end{figure}

This experiment shows how an important factor of mesh scalability can be measured. The actual numerical outcome, however, is to be taken with a grain of salt. Instead, signal servers would need to detect this rate during runtime and adjust the number of clusters according to demand.

\subsection{Connection Limits}

One other important facet of the mesh configuration, is the range connection goal and maximum connection limit as introduced by \vref{sec:design-optimising}. These influence the mesh density and rigidity and help it take on more peers and better recover from connection loss. In this experimental run, the simulation is started goal range configurations from $g_{min}=1$ to $g_{max}=8$ with a range width of $g_{max}-g_{min}=2$. The maximum connection limit is set to $m=(g_{min}+g_{max})/2$. The node join rate is kept at a low value of $r=1$ for $t=500$ ticks and network conditions remain at stable settings.

The prediction is, that low connection limits will cause the network's connectedness to suffer but higher connection goals will bring proportionally more traffic overhead.
To verify this predicted behaviour of the mesh implementation, the total mesh size is measured in peers and the total network traffic per tick is calculated.
The mesh size is calculated by finding all \glspl{scc} in the graph and counting all nodes in the largest component. The network traffic is aggregated per tick across nodes by the simulator's message delivery system.

\Vref{fig:connection-limits-largest-component} shows the node count of the largest mesh component, when spawning a new cluster with different connection limits. It is easily visible, how a goal of $g\leq4$ causes the mesh to fail at taking in peers at a stable rate and growing past a size of $n\approx100$. The $3 \leq g \leq 5, m=8$ configuration, also shows one incident of unintentional mesh separation. All higher connection limits are able to gain and maintain their mesh size proportionally to the spawn rate.

\begin{figure}[htb!]
\centering
\includegraphics[width=1.0\textwidth]{graphics/analysis/connection-limit-largest-component.pdf}
\caption{Mesh size in nodes in relation to connection limits}
\label{fig:connection-limits-largest-component}
\end{figure}

\Vref{fig:connection-limits-total-io} details the growth in traffic the network has to handle collectively per tick in megabytes. This diagram paints the opposite picture, as configurations with lower connection limits cause significantly less traffic overhead. The extra traffic is due to more Ping/Pong and \peerUpdate messages being sent to keep the network alive and peers informed.

However, total traffic is just one indicator, how higher connection limits do not imply a more performant network. With more connections, nodes will take on more neighbours and their routing table will grow exponentially. Routing will therefore become more complex and finding the best path to the router or the best peering candidate more ambiguous.

\begin{figure}[htb!]
\centering
\includegraphics[width=1.0\textwidth]{graphics/analysis/connection-limit-total-io.pdf}
\caption{Network–wide traffic per tick in relation to connection limits}
\label{fig:connection-limits-total-io}
\end{figure}