\documentclass[./standalone.tex]{subfiles}
%\documentclass[../../../CR/pac.tex]{subfiles}

\begin{document}

	%% =======================================
	\section{Algorithms}
	For this labwork I have implemented 2 metaheuristics algorithms. The first one being the steepest hill algorithm with restarts and the second one being the tabu algorithm. \\

	\subsection{Steepest Hill with restarts}

Below is the steepest hill algorithm without restarts. We won't detail here the auxiliary functions such as the \textit{getDistance} method from the TSP class or the \textit{debugSteepestRun} which only allows a few debugging prints. Furthermore some methods will be detailed in the section \ref{neighb}.
	\begin{lstlisting}[style=Java, caption={Source code of the steepest hill algorithm without restarts}]
public static int[] run(City[] cities, int[] citySequence, int maxMoves){
	int[] tmpCitySeq, res = citySequence;
	int nbMoves = 0; boolean stop = false;

	//Not essential to the algorithm but checks coherence of the algorithm:
	System.out.println("Random initial city sequence is: ");
	System.out.print("[ ");
	for(int cityID : citySequence) System.out.print(cityID + " ");
	System.out.println("]");

	//Essential to the algorithm:
	while(!stop && nbMoves < maxMoves){
		tmpCitySeq = bestNeighb(cities, res);

		if(TSP.getDistance(cities, tmpCitySeq) < TSP.getDistance(cities, res))
		res = tmpCitySeq;
		else
		stop = true;

		nbMoves++;
	}

	//Not essential to the algorithm but checks coherence of the algorithm:
	debugSteepestRun(nbMoves, res);
	return res;
}
	\end{lstlisting}
The steepest hill algorithm consists in always choosing the best neighbor and always going towards a better solution. If a better solution can't be found then the algorithm stops. The number of moves acts as a fail-safe for this algorithm (eg. if it's always possible to go up). This is actually the whole meaning of the loop quoted above from line 12 to 21. The restart version just adds another while loop over this algorithm and compares the best solutions found at each iteration between themselves to select only the best solution over the best solutions. This is a strategy to get out of local optimums.

\newpage
	\subsection{Tabu}
\begin{lstlisting}[style=Java, caption={Source code of the tabu algorithm}]
public static int[] run(City[] cities, int[] citySequence, int maxMoves, int tabuSize){
	LinkedList<int []> tabu = new LinkedList<>();
	int[] tmpBestCitySeq = null, bestNonTabuNeighb, currentCitySeq = citySequence;
	int nbMoves = 0; boolean stop = false;

	while(nbMoves < maxMoves && !stop){
	  bestNonTabuNeighb = bestNonTabuNeighb(tabu, cities, currentCitySeq);
		if(bestNonTabuNeighb == null)
		stop = true;
		else{
			addTabu(currentCitySeq, tabu, tabuSize);
			currentCitySeq = bestNonTabuNeighb;

			if(isBetterNeighb(cities, currentCitySeq, tmpBestCitySeq))
			tmpBestCitySeq = currentCitySeq;

			nbMoves++;
		}
	}

	//Not essential to the algorithm but checks coherence of the algorithm:
	debugTabu(nbMoves, currentCitySeq, tabu);
	return tmpBestCitySeq;
}
\end{lstlisting}
The steepest hill algorithm uses the random restarts strategy to get out of local optimums. The tabu algorithm on the other hand uses another philosophy: it allows itself to not always pick up a better solution. But it does always pick up the best possible neighbor around (or the least worse). In order to not backtrack and get caught in an infinite loop of backtracking back and forth to the local optimum it uses a tabu list in which already explored solutions are stored so that they are not explored again (as long as the tabu list, implemented as a FIFO, doesn't overflow in which case we drop the most ancient step). The longest the tabu list, the more freedom we give to our tabu execution to roam around the local optimum hoping to find a better local optimum. Again, as with the steepest hill, the while loop reflects this metaheuristics concept (from line 6 to line 17).\\

\underline{NB:} If no non tabu neighbour is found then in order to avoid a \textit{null} value as the \textit{currentCitySeq} (cf. line 12) we stop the algorithm and return the best solution found so far. This is the meaning of lines 8 to 9 above.

	%% =======================================
	\newpage
	\section{Neighbours and Neighbourdhoods}
	\label{neighb}

	\begin{lstlisting}[style=Java, caption={Source code of the neighbourhood generation in the Tabu class}]
public static LinkedList<int[]> genNeighbourhood(int[] citySeq){
	int nbCities = citySeq.length;
	LinkedList<int[]> res = new LinkedList<>();

	for(int i = 0; i < nbCities; i++) {
		for (int j = i + 1; j < nbCities; j++) {
			int[] tmp = new int[nbCities];
			swap(citySeq, i, j);
			System.arraycopy(citySeq, 0, tmp, 0, nbCities);
			res.add(tmp);
			swap(citySeq, i, j);
		}
	}
	return res;
}
	\end{lstlisting}
While this piece of code comes from the TSP class it is actually very similar to the \textit{bestNeighb} static method found in the SteepestHill class. This stems from the fact that the said method kills two birds in one stone by generating the whole neighborhood of a give solution while selecting the best neighbor.\\

What the above function shows is that \underline{in the context of the TSP problem}, a neighborhood consists in the swapping of two cities from one initial sequence of cities. We can generate all the possible neighbors by generating all the possible swaps with one value then by generating all the possible swaps of the other values without including the swaps with the value already computed. Here we do this in ascending order: we generate all the possible swaps from the first city then we do this with the second city excluding the first city, etc.\\

	%% =======================================
	\newpage
	\section{Instances and results}

	\subsection{tsp5.txt instance}

	\subsubsection{Steepest Hill algorithm with restarts}
	\begin{lstlisting}[style=Java, caption={Bash output of the execution of the steepest hill algorithm on the tsp5.txt instance}]
chuxclub @ CRex2 ~/projects/wip/2021_m1csa_algo_metaheuristics (dev)
$ java -jar tsp.jar res/tsp5.txt steepest

Input maxMoves: 10
Input maxTrials: 3

######### TESTING STEEPEST HILL ALGORITHM (with restarts) ##########
>>> INITIAL SEQUENCE OF CITIES:
=====================================================
Vector #DataStructures.Vector@3567135c has length: 5
Vector #DataStructures.Vector@3567135c contains:
3     4     5     2     1
=====================================================
Distance: 202.72643264132455 km

>>> STEEPEST HILL ALGORITHM EXECUTION (with restarts):
Trial #1 :
Random initial city sequence is:
[ 1 2 3 5 4 ]
2 moves required to reach the following solution:
[ 1 2 4 5 3 ]

Trial #2 :
Random initial city sequence is:
[ 4 2 3 5 1 ]
3 moves required to reach the following solution:
[ 1 2 4 5 3 ]

Trial #3 :
Random initial city sequence is:
[ 2 3 1 5 4 ]
3 moves required to reach the following solution:
[ 1 3 2 4 5 ]

>>> BEST SEQUENCE OF CITIES :
=====================================================
Vector #DataStructures.Vector@54a097cc has length: 5
Vector #DataStructures.Vector@54a097cc contains:
1     2     4     5     3
=====================================================
Distance: 194.04052963659356 km
####################################################################
	\end{lstlisting}
With 10 maximum moves and 3 maximum trials the best solution was reached in a fraction of a second.

\newpage
\subsubsection{Tabu algorithm}
\begin{lstlisting}[style=Java, caption={Bash output of the execution of the tabu algorithm on the tsp5.txt instance}]
chuxclub @ CRex2 ~/projects/wip/2021_m1csa_algo_metaheuristics (dev)
$ java -jar tsp.jar res/tsp5.txt tabu

Input maxMoves: 10
Input tabuSize: 5

############################# TESTING TABU ALGORITHM ###########################
>>> INITIAL SEQUENCE OF CITIES:
=====================================================
Vector #DataStructures.Vector@3567135c has length: 5
Vector #DataStructures.Vector@3567135c contains:
4     1     5     2     3
=====================================================
Distance: 258.8852276594047 km

>>> TABU ALGORITHM EXECUTION:
Number of moves used to reach last solution: 10
Last solution reached: [ 4 2 5 3 1 ]
City sequences in tabu list:
[ 3 5 4 2 1 ]
[ 3 5 2 4 1 ]
[ 3 4 2 5 1 ]
[ 5 4 2 3 1 ]
[ 2 4 5 3 1 ]

>>> BEST SEQUENCE OF CITIES:
=====================================================
Vector #DataStructures.Vector@3551a94 has length: 5
Vector #DataStructures.Vector@3551a94 contains:
3     5     4     2     1
=====================================================
Distance: 194.04052963659356 km
################################################################################
\end{lstlisting}
As with the steepest hill algorithm above, here with 10 maximum moves and a tabu list size of 5 the best solution was reached in a fraction of a second.\\

On the tsp5.txt we have two possible best solutions as shown above: 3     5     4     2     1  and  1     2     4     5     3  with a distance of 194.04052963659356 km. We notice that one solution is the inverted version of the other solution which trivially means that the same distance is traveled whether we do the traveling salesman circuit in one way or the other.\\

	\newpage
	\subsection{tsp101.txt instance}
Given the amount of cities (101 of them) I couldn't find no best local optimum despite the algorithm used and the amount of moves/trials/tabu size involved. But I managed once to go down to around 994km with the tabu algorithm with 1000 moves and tabu list size of 500. Recently, the steepest hill algorithm provided a solution with 1023.92 km of traveled distance with 400 moves and 100 trials.  So we can conclude that both of these algorithms find more or less the same solutions. One algorithm take more memory space (the tabu algorithm) while the other seems to take more computing time (the steepest hill algorithm) meaning that the choice of one or the other algorithm depends on the application they are used for.\\

We also notice that the time to find local optimums given the parameters mentioned above usually spanned from 15s to 30s.\\

\end{document}


\documentclass[runningheads,a4paper]{llncs}

\usepackage{amssymb}
\setcounter{tocdepth}{3}
\usepackage{graphicx}

%%%%%%%%%ADDED PACKAGES%%%%%%%%%%
\usepackage{amsmath}
\usepackage{graphicx}
\usepackage{booktabs}
\usepackage[caption=false]{subfig}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\usepackage{url}
\urldef{\mailsa}\path||
\urldef{\mailsb}\path||
\urldef{\mailsc}\path|simon.andermatt@unibas.ch|
\newcommand{\keywords}[1]{\par\addvspace\baselineskip
\noindent\keywordname\enspace\ignorespaces#1}

\DeclareMathOperator*{\argmax}{arg\,max}
\begin{document}

\mainmatter  % start of an individual contribution

% first the title is needed
\title{Multi-Dimensional Gated Recurrent Units\\
for the Segmentation of Biomedical 3D-Data}

% a short form should be given in case it is too long for the running head
\titlerunning{Multi-Dimensional GRU for the Segmentation of Biomedical Data}

% the name(s) of the author(s) follow(s) next
%
% NB: Chinese authors should write their first names(s) in front of
% their surnames. This ensures that the names appear correctly in
% the running heads and the author index.
%
\author{Simon Andermatt \and Simon Pezold \and Philippe Cattin}
% index{Andermatt, Simon}
% index{Pezold, Simon}
% index{Cattin, Philippe}
%
\authorrunning{Multi-Dimensional GRU for the Segmentation of Biomedical Data}

% the affiliations are given next; don't give your e-mail address
% unless you accept that it will be published
\institute{Department of Biomedical Engineering,
University of Basel,
Switzerland
\mailsc\\
}

%
% NB: a more complex sample for affiliations and the mapping to the
% corresponding authors can be found in the file "llncs.dem"
% (search for the string "\mainmatter" where a contribution starts).
% "llncs.dem" accompanies the document class "llncs.cls".
%

\toctitle{Multi-Dimensional Gated Recurrent Units for the Segmentation of Volumetric Biomedical Data}
\tocauthor{Simon Andermatt}
\maketitle

\begin{abstract}
We present a supervised deep learning method to automatically segment 3D volumes of biomedical image data. The presented method takes advantage of a neural network with the main layers consisting of multi-dimensional gated recurrent units. We apply an on-the-fly data augmentation technique which allows for accurate estimations without the need for either a huge amount of training data or advanced data pre- or postprocessing. We show that our method performs amongst the leading techniques on a popular brain segmentation challenge dataset in terms of speed, accuracy and memory efficiency. We describe in detail advantages over a similar method which uses the well-established long~short-term~memory.

\keywords{deep learning, GRU, multi-dimensional RNN, segmentation}
\end{abstract}

\section{Introduction}
With the rapid advancements of imaging technologies, their ubiquitous availability and dropping prices, vast amounts of data are collected. This is particularly true for medical imaging. Accurate segmentation and delineation of e.g. pathologies in this medical data, however, pose real challenges as this is still mainly a manual process. In late phase drug studies with thousands of patients, multiple 3d datasets with different MR sequences are often collected per patient. If quantitative analysis of the immense amount of data is required, the time that has to be spent on the data by trained experts is enormous. A successful automated segmentation technique would decrease manual work to a minimum, cutting the costs and time spent on developing new treatments.

Automatic segmentation of biomedical volumetric data is, however, a challenging problem due to its high dimensionality, imaging noise, artifacts and other factors. Recent advances in the field of deep learning, especially the enabling effect of modern GPUs along with the advent of general purpose GPU computing, led to a revival of convolutional neural networks~\cite{krizhevsky_imagenet_2012}. These feed-forward networks show great promise, but need a large number of layers to solve a difficult task accurately. A recurrent neural network (RNN), in contrast, can become arbitrarily deep due to its additional temporal dimension. Each timestep computed in an RNN corresponds roughly to one layer in a feed-forward network, with the weights in one RNN being the same for each timestep. This property allows defining substantially more complexity very elegantly without the need for a huge number of layers or parameters.

The multi-dimensional Long Short-Term Memory (MD-LSTM) proposed by Stollenga et al.~\cite{stollenga_parallel_2015}, called \textit{PyraMiD-LSTM}, applied these insights to the Long Short-Term Memory (LSTM)~\cite{hochreiter1997long}. It defines two LSTMs for each spatial dimension, using said spatial dimension as temporal dimension. The first one processes the data along that dimension, the second one in the opposite direction. In order to make full use of the spatial information, not only the direct predecessor along the temporal direction is taken into account, but also its local neighborhood. This can be neatly expressed using convolutions.

A relatively new RNN called Gated Recurrent Unit (GRU)~\cite{cho_learning_2014} grew popular in recent years and became a strong competitor for the LSTM. It can be seen as a simplified version of the LSTM, which uses an update gate instead of a forget and input gate and combines the hidden and cell state~\cite{olah_understanding_2015}. It has been shown that it performs comparably to the LSTM in the task of sequence modeling~\cite{chung_empirical_2014}. Another study suggests that GRU and LSTM report similar performance on selected tasks~\cite{greff_lstm:_2015}. An empirical search among more than 10\,000 RNN architectures showed that on the selected tasks, although not the best performing RNN on every task, the GRU outperformed the standard LSTM architecture~\cite{jozefowicz_empirical_2015}.
A larger time dimension in an RNN can mean that larger time dependencies can be represented. The lower memory requirement of the GRU means that larger volumes can be fed into the network and larger networks can be designed for the same volume size.

For all these reasons, a modification of the GRU to be able to process volumetric data seems compelling. We propose the multi-dimensional GRU (MD-GRU), which is capable of accurate segmentation of 3d data. We hint at the theoretical memory savings compared to the MD-LSTM and show that the performance of MD-GRU is comparable if not superior. Furthermore, we show that its convergence rate, computation time and combination of fewer gates favor the MD-GRU. We apply our method on a popular brain segmentation challenge dataset, achieving a score among the top 3 best performing methods.

\section{Methods}
\subsection{Data}
\begin{figure}
 \centering
 \begin{minipage}{.24\textwidth}
 \includegraphics[width=.99\linewidth]{./images/t1.png}
 \end{minipage}
  \begin{minipage}{.24\textwidth}
 \includegraphics[width=.99\linewidth]{./images/t1-ir.png}
  \end{minipage}
  \begin{minipage}{.24\textwidth}
 \includegraphics[width=.99\linewidth]{./images/t2-flair.png}
  \end{minipage}

   \begin{minipage}{.24\textwidth}
 \includegraphics[width=.99\linewidth]{./images/t1-gs.png}
 \end{minipage}
  \begin{minipage}{.24\textwidth}
 \includegraphics[width=.99\linewidth]{./images/t1-ir-gs.png}
  \end{minipage}
  \begin{minipage}{.24\textwidth}
 \includegraphics[width=.99\linewidth]{./images/t2-flair-gs.png}
  \end{minipage}
  \caption{Slice 19 of the 5th training sample. \emph{Top row (left to right):} T1, T1\_IR and T2\_FLAIR. \emph{Bottom row:} respective highpass filtered versions.}
  \label{data}
\end{figure}
We used the publicly available MrBrainS~\cite{mendrik2015mrbrains} challenge dataset, which was one of the datasets used to evaluate the PyraMiD-LSTM. The MrBrainS challenge data consists of 5 labeled samples and 15 testing samples, where each sample has a T1 weighted, T1 inversion recovery and a FLAIR scan. The additional high-resolution T1 scan was not used, as the labeling was performed on the low resolution data. The training data contained two different label maps, one for training and one for testing. The training map consists of classes for cortical gray matter (GM), basal ganglia, white matter (WM), WM lesions, cerebrospinal fluid (CSF), ventricles, cerebellum, brainstem and background. The testing map only defines classes for GM, WM and CSF, the respective classes of the training map are merged. Brainstem and cerebellum are not included in the evaluation and do therefore not appear labeled in the testing map.

\subsection{Convolutional Gated Recurrent Unit}\label{reordering}

The standard GRU as proposed in~\cite{cho_learning_2014} is defined as
\begin{align}
 r^j &= \sigma([W_r x]^j + [U_r h_{t-1}]^j),\\
 z^j &= \sigma([W_z x]^j + [U_z h_{t-1}]^j),\\
 \tilde{h}^j_t &= \phi([Wx]^j + [U(r \odot h_{t-1})]^j),\\
 h^j_{t} &= z^j\odot h^j_{t-1} + (1-z^j)\odot \tilde{h}^j_{t},
\end{align}
where $x$ is the input data, $r^j$ is the reset gate, $z^j$ is the update gate of the hidden unit $j$ and the activation is performed in $h^j$. The operator $\odot$ represents an elementwise multiplication. The functions $\sigma(\cdot)$ and $\phi(\cdot)$ stand for the logistic function and the hyperbolic tangent. $W$ and $U$ are the weight matrices for the current input and last step's output data respectively. Along the lines of Stollenga et al.~\cite{stollenga_parallel_2015}, we adapt these equations to be able to process 3D volumes and introduce our convolutional GRU (C-GRU):
\begin{align}\label{eq:cgru1}
 r^j &= \sigma \left( \sum\limits_i^I (x^i*w_r^{i,j}) + \sum\limits_k^J ( h_{t-1}^k*u_r^{k,j} ) +b^j_r\right),\\\label{eq:cgru2}
 z^j &= \sigma \left( \sum\limits_i^I (x^i*w_z^{i,j}) + \sum\limits_k^J ( h_{t-1}^k*u_z^{k,j} ) +b^j_z\right),\\\label{eq:cgru3}
 \tilde{h}^j_t &= \phi \left( \sum\limits_i^I (x^i*w^{i,j}) + r^j \odot \sum\limits_k^J ( h_{t-1}^k * u^{k,j} ) +b^j\right),\\
  h^j_{t} &= z^j\odot h^j_{t-1} + (1-z^j)\odot\tilde{h}^j_{t},
\end{align}
where $*$ represents a convolution. Compared to the vanilla GRU, we introduced slight changes. We decided to use a bias $b$ on each gate. We factored $r^j$ out of the convolution operation between $u$ and $h_{t-1}$. This change was motivated by the fact that an additional convolution would require $r$ to have twice the support it needs now because of the chained convolution. Moreover, we reorder the data for each C-RNN such that the two spatial dimensions are closest to memory, and the temporal dimension is ordered according to the temporal direction, as explained in the next paragraph. We motivated that decision with faster possible processing speeds on the GPU, since all convolutions now require data that lies close in memory. The computations of one C-GRU are visualized as a computational graph in Fig.~\ref{fig:network}a.

The MD-GRU consists of two times $D$ C-GRUs, where $D$ is the dimensionality of the image data and we need one C-GRU for each of the two directions. We set the input data of channel $i$ as $x^i \in \mathbb{R}^{S_1\times \cdots\times S_D}$. For each spatial dimension $d$, we create the copies $x^{i,d,-1}, x^{i,d,+1} \in \mathbb{R}^{S_d\times S_1\times \cdots\times S_D}$ of $x$ and apply the following data transformations:
\begin{align}
 x^{i,d,+1}(s_d, s_1, \dots, s_{D}) &= x^i(s_1, \dots, s_d,\dots, s_{D}),\\
 x^{i,d,-1}(S_d-s_d, s_1, \dots, s_{D}) &= x^i(s_1,\dots , s_d,\dots, s_{D}),
\end{align}
where $s_d$ is the index of the assigned dimension of the C-GRU and $S_d$ is the size of dimension $d$. The inverse operation is applied to $h^{j,d,+1},h^{j,d,-1} \in \mathbb{R}^{S_d\times S_1\times \cdots\times S_D}$ to gather the final output $h^j$:
\begin{align}
h^j(s_1,\dots, s_{D}) &= \sum\limits_{d=1}^D \left ( h^{j,d,+1}(s_d, s_1, \dots, s_{D}) + h^{j,d,-1}(S_d-s_d, s_1, \dots, s_{D}) \right ).
\end{align}

Figure~\ref{fig:network}b details this process for the MD-GRU. We apply the same technique for our implementation of the MD-LSTM.
\subsection{Experiments}
\subsubsection{Network}
We model our network similar to~\cite{stollenga_parallel_2015}. We include three multi-dimensional RNN (MD-RNN) layers of 16, 32 and 64 channels which are connected with pixelwise fully connected hidden layers of 25 and 45 channels respectively, each followed by a hyperbolic tangent activation function. The last MD-RNN is attached to a pixelwise fully connected layer with $c$ channels, the same number as classes in the data. We estimate the probabilities for each class using a softmax in the last layer and consequently choose the multinomial logistic loss for the training of our network. Figure~\ref{fig:network}c shows our network setup for the case of MD-GRU.

\begin{figure}
 \centering
 \subfloat[][C-GRU]{
          \includegraphics[width=0.35\textwidth]{images/cgrunotitle3.pdf}
 }\qquad
 \subfloat[][MD-GRU]{
          \includegraphics[width=0.48\textwidth]{images/mdgrunotitle2.pdf}
 }\quad

  \subfloat[][Network Architecture]{
          \includegraphics[width=0.99\textwidth]{images/network.pdf}
 }
 \caption{(a) Directed graph denoting the computations in one C-GRU. The variables $x^{d,o}$, $h^{d,o}$ with $o \in \{-1,+1\}$ represent the input and output data across all $I$ and $J$ channels respectively. The $\circledast$ operator denotes here the sum per channel $j$ over the convolutions with each channel $i$ or $k$, as used in equations (\ref{eq:cgru1})--(\ref{eq:cgru3}). (b) Proposed arrangement of 6 C-GRUs in a MD-GRU for three-dimensional data. (c) Setup of our network.} \label{fig:network}
\end{figure}

\subsubsection{Setting}\label{setting}
All experiments were calculated on an NVIDIA GTX Titan X GPU with 12 GB global memory. Our implementation of MD-LSTM and MD-GRU relied on the fast convolution routines provided by NVIDIA's cuDNN~\cite{chetlur_cudnn:_2014}. For other layers, the already available implementations of the CAFFE\footnote{version 1.0.0-rc3, commit 9c46289} framework~\cite{jia2014caffe} were used.

\subsubsection{Preprocessing}
For all volumes, unsharp masking was done using a Gaussian smoothed image ($\sigma=5$~voxels) which was then subtracted from the original images to produce highpass filtered volumes. The original images and the highpass filtered images were normalized to $\sigma=1$ and $\mu=0$, assuming normally distributed values. In this way we followed a procedure similar to~\cite{stollenga_parallel_2015}, but omitted the histogram equalization. Figure \ref{data} shows the original and preprocessed data for training sample 5 at slice 19.

\subsubsection{Data augmentation}
In the training stage, at each iteration, a random location in the training data was selected and a deformation field was generated and applied to the subvolumes, which were then fed into the network. We used a procedure similar to~\cite{ronneberger_u-net:_2015}, but made the grid size dependent on the data. We did not use random deformations in the feasibility study mentioned in Sec.~\ref{feasibilitystudylabel}. For the testing phase, no deformations were applied.

\subsubsection{Training}
In three training steps we iteratively increase the subvolume size from $64\times64\times8$~voxels to $128\times128\times12$ and finally to $200\times200\times15$, keeping the third dimension smaller to account for the anisotropic MR volume resolution. We relied on AdaDelta~\cite{zeiler_adadelta_2012} to omit the manual tuning of a learning rate. For the challenge, we additionally used DropConnect~\cite{icml2013_wan13} of 0.5 on the input connections of each C-GRU to prevent overfitting. Training took around two days.

\subsubsection{Testing}
In the testing phase, we divided the volume into a grid of equally sized subvolumes of $120\times120\times8$, which were padded by 50, 50 and 4 voxels respectively on all sides of the volume. The padding was later used to stitch the results together using a Gaussian ($\mu=0$, $\sigma=(10,10,0.8)$) to produce interpolation weights, since the borders contain starting artifacts from the individual RNNs and do not contain adequate results. Since we trained for nine classes, but only four classes were needed for the final evaluation, we simply combined the binary labels for the CSF with the ventricles, the cortical GM with the basal ganglia and the WM with the WM lesions. Everything else was considered background. Testing one volume of the MRBrainS data required 32 iterations, which needed around two minutes.

\section{Results}

\subsection{Feasibility Study}
\label{feasibilitystudylabel}
\begin{table}
\caption{Feasibility study. Dice coefficients in percent for gray and white matter (GM/WM), cerebrospinal fluid (CSF) and intracranial volume (ICV).}% The case with more memory was only trained until iteration 28000}
\begin{center}
\begin{tabular}{lllll}\toprule
× & GM & WM & CSF & ICV\\ \midrule
MD-LSTM  & \textbf{88.09}  & 90.08 & 82.62  & 97.56 \\
MD-GRU & 87.88 & \textbf{90.15 } & \textbf{83.19} & \textbf{97.73 }\\ \bottomrule
\end{tabular}
\label{feasibilityquantitative}
\end{center}
\end{table}

\begin{figure}
 \centering
 \begin{minipage}{.24\textwidth}
 \includegraphics[width=.98\linewidth]{./images/lstmfeas_new_new.png}
 \end{minipage}
  \begin{minipage}{.24\textwidth}
 \includegraphics[width=.98\linewidth]{./images/grusfeas_new_new.png}
  \end{minipage}
  \begin{minipage}{.24\textwidth}
 \includegraphics[width=.98\linewidth]{./images/Traindata5atslice19-Testlabel.png}
  \end{minipage}

  \begin{minipage}{0.80\textwidth}
    \includegraphics[width=.9\linewidth]{./images/convergence_rate_new10000_mediumlines_withtitle6.pdf}
  \end{minipage}
  \caption{Feasibility study. \emph{Top row:} slice 19 of the 5th training volume used for the evaluation. The images from left to right represent the results of the MD-LSTM, the MD-GRU and the manual labeling. \emph{Bottom row:} convergence rates for the feasibility study of both MD-GRU and MD-LSTM.}
\label{feasibilityqualitative}

\end{figure}
To point out differences between the MD-GRU and the MD-LSTM, we ran the same setup with the multi-dimensional RNN layers either being an MD-GRU or an MD-LSTM. We used the first four volumes in the training set of the MrBrainS challenge and trained both networks for 3\,000 iterations on the largest possible resolution which was feasible for both (limited to $192\times192\times14$ by our MD-LSTM implementation). On average, one training iteration for MD-GRU and MD-LSTM took 9.1 and 12.8 seconds, respectively. The Dice coefficients for CSF, GM, WM and ICV between the computed segmentation of the 5th training volume and the provided reference segmentation are shown in Table~\ref{feasibilityquantitative} for both the MD-GRU and MD-LSTM. Slice 19 of the computed segmentations and the reference segmentation are displayed in Fig.~\ref{feasibilityqualitative} together with a plot of a running average of 100 iterations of the loss function for each iteration of the training procedure.

\subsection{MD-GRU on MRBrainS}
In our attempt to beat the highscore of the MRBrainS challenge, we used our described data augmentation method. Each subvolume was deformed randomly throughout all three training phases. We used all provided low resolution volumes and their highpass filtered versions. Table~\ref{mrbrainsres} lists our performance according to the Dice coefficients, 95th-percentile  of the Hausdorff distance and average volume difference of the GM, WM, CSF and ICV. Nine measures were relevant for the final evaluation: Dice, modified Hausdorff distance and average volume distance in each of the categories GM, WM and CSF. The sum of the ranks in these nine categories is used as the performance score and determines the final rank.
Figure~\ref{visualresultschallenge} shows the computed segmentation at slice 19 of samples 5, 10 and 15 of the test data.
\begin{table}
\begin{center}
 \caption{MrBrainS challenge. Results of the six best performing methods for GM, WM, CSF and ICV of all three used metrics (Dice, 95th-percentile of the Hausdorff distance (HD) and average volume difference (AVD)). A bold number means best out of these six. The results reflect the state on August 12, 2016.}
\label{mrbrainsres}
 \scriptsize
\begin{tabular}{l c rcl rcl rcl rcl rcl}
\toprule
Team name &  Rank &\multicolumn{3}{c}{GM} & \multicolumn{3}{c}{WM} & \multicolumn{3}{c}{CSF} &  \multicolumn{3}{c}{ICV}\\%\hline\hline
×& × & Dice & HD & AVD & Dice & HD & AVD & Dice & HD & AVD & Dice & HD & AVD \\\midrule
CU\_DL2  & \textbf{1}& \textbf{86.15} & \textbf{1.45} & 6.60 & \textbf{89.46} & \textbf{1.94} & 6.05 & \textbf{84.25}& 2.19 & 7.69 &  98.10 & 2.75 & 1.54\\
CU\_DL& 2  & 86.12 & 1.47 & 6.42 & 89.39 & \textbf{1.94} & \textbf{5.84} & 83.96 & 2.28 & 7.44 &  97.99 & 3.16 & 1.83 \\
MD-GRU [proposed] & 3 & 85.40 & 1.55 & \textbf{6.09} & 88.98 & 2.02 & 7.69 & 84.13 & 2.17 & 7.44 & \textbf{98.15} & \textbf{2.37} & 0.86\\
PyraMiD-LSTM2 & 4 & 84.89 & 1.67 & 6.35 & 88.53 & 2.07 & 5.93 & 83.05 & 2.30 & 7.17 &  98.04 & 2.86 & \textbf{0.69}\\
FBI/LMB Freiburg~\cite{cicek_3d_2016}& 5 & 85.44 & 1.58 & 6.60 & 88.86 & 1.95 & 6.47 & 83.47 & 2.22 & 8.63 &  97.98 & 2.51 & 1.06\\
IDSIA~\cite{stollenga_parallel_2015} & 6 & 84.82 & 1.70 & 6.77 & 88.33 & 2.08 & 7.05& 83.72 & \textbf{2.14} & \textbf{7.09}& \textbf{98.15} & 2.44 & 0.95\\\bottomrule
\end{tabular}
\end{center}
 \label{resultschallenge}
\end{table}

\begin{figure}
 \centering
 \begin{minipage}{.24\textwidth}
 \includegraphics[width=.99\linewidth]{./images/sl19-v5-test-challres_new.png}
 \end{minipage}
  \begin{minipage}{.24\textwidth}
 \includegraphics[width=.99\linewidth]{./images/sl19-v5-test-t1.png}
  \end{minipage}
  \begin{minipage}{.24\textwidth}
 \includegraphics[width=.99\linewidth]{./images/sl19-v5-test-t1-ir.png}
  \end{minipage}
  \begin{minipage}{.24\textwidth}
 \includegraphics[width=.99\linewidth]{./images/sl19-v5-test-t2-flair.png}
  \end{minipage}

  \vspace{0.3em}
   \begin{minipage}{.24\textwidth}
 \includegraphics[width=.99\linewidth]{./images/sl19-v10-test-challres_new.png}
 \end{minipage}
  \begin{minipage}{.24\textwidth}
 \includegraphics[width=.99\linewidth]{./images/sl19-v10-test-t1.png}
  \end{minipage}
  \begin{minipage}{.24\textwidth}
 \includegraphics[width=.99\linewidth]{./images/sl19-v10-test-t1-ir.png}
  \end{minipage}
  \begin{minipage}{.24\textwidth}
 \includegraphics[width=.99\linewidth]{./images/sl19-v10-test-t2-flair.png}
  \end{minipage}

  \vspace{0.3em}
   \begin{minipage}{.24\textwidth}
 \includegraphics[width=.99\linewidth]{./images/sl19-v15-test-challres_new.png}
 \end{minipage}
  \begin{minipage}{.24\textwidth}
 \includegraphics[width=.99\linewidth]{./images/sl19-v15-test-t1.png}
  \end{minipage}
  \begin{minipage}{.24\textwidth}
 \includegraphics[width=.99\linewidth]{./images/sl19-v15-test-t1-ir.png}
  \end{minipage}
  \begin{minipage}{.24\textwidth}
 \includegraphics[width=.99\linewidth]{./images/sl19-v15-test-t2-flair.png}
  \end{minipage}
  \caption{MrBrainS challenge. \emph{Rows (top to bottom):} 5th, 10th and 15th test sample. \emph{Columns (left to right):} slice 19 of our segmentation results, T1, T1\_IR and T2\_FLAIR.}
   \label{visualresultschallenge}

\end{figure}

\section{Discussion}

The feasibility study has shown that MD-GRU has great potential for the segmentation of volumetric images, since it achieved comparable results to the MD-LSTM in less time with the same settings.

Using deformation as a data augmentation strategy and DropConnect for regularization in the challenge, we ranked 3rd out of 37. Unfortunately, none of the results in the top five of the challenge highscore are published so far. The 4th and 6th entries are both incarnations of the already discussed MD-LSTM, where only the latter was described in~\cite{stollenga_parallel_2015} and the former likely contains unpublished improvements to their method. In contrast to \cite{stollenga_parallel_2015}, we did not omit the original T1\_IR images. Yet some obvious misclassifications could be traced back to strong bias field artifacts in the T1\_IR images. Given the small training size, using the T1\_IR images leads to apparent fitting to the bias field. Furthermore, we were not able to replicate the training volume size of Stollenga et al.~\cite{stollenga_parallel_2015} due to a higher memory requirement of our implementation, since we decided to copy the input and output data for each RNN layer, as detailed in Sec.~\ref{reordering}. This has to be kept in mind when comparing the two approaches. Relationships between areas that are located at a certain distance in the data could therefore not be modeled in our network, where~\cite{stollenga_parallel_2015} was able to use the full spatial context in two dimensions as well as a larger third dimension. In their last training step more than half of the data was covered while we could only fit a bit more than a fifth in our memory.

The contribution on rank five was computed using the 3D U-Net~\cite{cicek_3d_2016}. It consists of a hierarchical convolutional neural network with shortcut connections, which is trained using various on-the-fly data augmentation techniques, including the deformation strategy used in this paper. The challenge results and corresponding adaptations of the algorithm to fit the challenge data are, however, not yet published. We believe that data augmentation is key for successful applications to problems with such a small training size.

 \subsubsection{Conclusion}
 With the MD-GRU, we combined the enormous expressive power of RNNs with a highly beneficial data augmentation strategy, resulting in a powerful supervised automatic segmentation technique. With a memory-savvy implementation that omits the initial reordering of the data, results surpassing the state of the art should be possible with MD-GRU.

\bibliographystyle{splncs03}
\bibliography{manuallib}
\end{document}


\section{Static Obstacle Avoidance}
\label{chap:StaticObstacleAvoidance}
For the static obstacle avoidance task, the controller must drive the vehicle in order to avoid collision with any still obstacle detected by a set of sensors mounted on the vehicle.\\
Before implementing the algorithm for obstacle avoidance, we have made a set of assumptions with the aim of simplifying this task:
\begin{itemize}
    \item we have a set of sensor able to determine the position of the obstacle and to place it on the reference map, but the detection is performed only when the distance from the obstacle is less than 200 $m$;
    \item it is always possible to avoid the obstacle by moving to the left lane;
    \item the obstacle is never placed at less than 200 $m$ from the starting point, therefore it is always possible to avoid it without violating the zones (described below);
    \item dimensions of the obstacle are not relevant.
\end{itemize}
To be sure that the controller tries to avoid obstacles, we have decided to implement the avoidance algorithm by giving to the controller a set of constraints on the output, in order to define a \textit{``forbidden zone"}, as described in subsection \ref{sec:Safety_distance}.\\
To be precise, we have defined 5 zones nearby each obstacle:
\begin{itemize}
    \item \textbf{Zone 1}: the obstacle is detected, but the vehicle is still too far from it, so it stays in its lane;
    \item \textbf{Zone 2}: the vehicle is close enough to the obstacle to start the overtaking maneuver, so it can start the lane change;
    \item \textbf{Zone 3}: the vehicle is in the left lane, in the so called \textit{``Safe Zone"};
    \item \textbf{Zone 4}: the obstacle is passed and the vehicle can come back to the right lane;
    \item \textbf{Zone 5}: the vehicle is in its lane and the obstacle has been already passed, so we are in the same condition of no obstacle detected.
\end{itemize}
These zones are defined, starting from the obstacle position, in the following way:
\begin{itemize}
    \item \textbf{Zone 1} starts when the obstacle is detected, so when the distance between the vehicle and the obstacle is less than 200 $m$, as specified in the assumptions made at the start of this section;
    \item \textbf{Zone 2} starts 40 meters before the Zone 3. We have decided to define this zone in this way to be symmetric with respect to Zone 4, where the other lane change maneuver is performed;
    \item \textbf{Zone 3} is defined to be compliant with the Safety Distance described in \ref{sec:Safety_distance}, according to equation \ref{eq:safetyDistance}: $SafetyDistance [m] = \left(\frac{v[\sfrac{km}{h}]}{10}\right)^2$, so it is proportional to the square of the speed. Zone 3 ends 10 meters after the obstacle, to be compliant with the requirement n° 4 in section \ref{System_Requirements};
    \item \textbf{Zone 4} is defined to be compliant with requirement n° 4 in Section \ref{System_Requirements}, so it starts 10 meters after the obstacle and ends 50 meters after it, resulting in a space of 40 meters for the re-entering maneuver, thus it is symmetric with respect to the maneuver performed in Zone 2 as previously stated;
    \item \textbf{Zone 5} starts right after Zone 4 and it is kept until the distance from the obstacle becomes greater then 200 $m$.
\end{itemize}
The distance of 200 $m$ from the obstacle is evaluated as the Euclidean distance between the vehicle and the obstacle, while the other distances involved in the definition of the zones are evaluated as projection on the reference path, and assuming constant speed, they result in fixed points on the reference map.
\begin{figure}[H]
    \centering
    \includegraphics[width=1\textwidth]{Figures/Zones.png}
    \caption{Zones defined for obstacle avoidance}
      \label{fig:Zones}
\end{figure}
In Figure \ref{fig:Zones} the aforementioned zones are depicted on a straight line scenario with a reference speed of 100 $km/h$.\\\\
In Zones 1 and 5, as previously said, the obstacle is too far from the vehicle to be a problem, so no procedure is applied to the controller.\\\\
In Zone 2, the yellow one in the figure, the vehicle should pass in the left lane. To do so, we have defined two points, represented in the figure respectively with a green circle, which is called the \textbf{\textit{Detection Point}}, and a red circle that is the \textbf{\textit{Safe Point}}.
The latter is defined as the projection on the left lane of the point in the reference trajectory at a distance equal to the safety distance. \\To let the controller move on the path defined by these two points, we have exploited the \textit{Custom Constraint} of the MPC block in Simulink.\\\\
As briefly explained in Section \ref{chap:Controller}, the linear combination of inputs and states can be set as constraints for the controller. To accomplish our task, we have defined 2 constraints on the \textit{x} and \textit{y} states, which result in two lines representing an upper and a lower bound for the vehicle position during the overtaking maneuver. Looking at the example in Figure \ref{fig:Zones}, the lower bound becomes the border of the polygon defined by Zones 2, 3 and 4, while the upper bound is the left limit of the road, that is to say the left border of the left lane (straight line $y=6$ in Figure \ref{fig:Zones}).\\
For what concerns the lower bound for constraints in Zone 2, it is completely defined by the line passing from the two aforementioned points (\textit{Detection Point} and \textit{Safe Point}) .\\\\
In Zone 3, the vehicle should stay in the left lane. To define constraints in this zone, we have set as lower bound the projection of the reference trajectory on the left lane instead of the straight line connecting points from the start (red circle) to the end (blue cross), since the former method allows to track both straight and curved paths. One limitation of the \textit{linear} constraints is that they can only define \textit{linear} relation between states, thus when the path is curved, we have to approximate it to a straight line, but this can cause errors because the controller tries to satisfy the constraint for all the timesteps in the prediction horizon, so resulting in a reduced or increased distance with respect to the points in the reference map. To limit this effect, we have inserted a correction factor to the constraint generated.
\begin{figure}[H]
    \centering
    \includegraphics[width=1\textwidth]{Figures/ConstraintCorrection.png}
    \caption{Constraints generated in a curved path with 300 $m$ radius at 50 $km/h$, with and without the correction factor}
      \label{fig:CorrectionConstraint}
\end{figure}
In Figure \ref{fig:CorrectionConstraint} a case where the reference road is a left curve with 300 meters radius is represented. The dashed red line is the projection of the road approximated with the slope of the reference point where the controller is called, while the green dashed line is the projection evaluated with the correction factor. As it can be seen, the original projection "collapses" on the reference path before the corrected one does, resulting in a reduction of the distance from the reference during the predicted horizon and a consequent reduction of distance between the trajectory and starting lane, occupied by the obstacle.\\
This corrected constraint is evaluated according to the following procedure:
\begin{enumerate}
    \item an approximation of the \textit{curvature} of the road is evaluated as the mean value of the reference angle in the prediction horizon: $curvature = \frac{\sum_{i=0}^{p}\theta_i-\theta_0}{p}$;
    \item the correction factor is: $416\cdot\frac{|curvature|}{V}$, where the coefficient 416 has been found empirically by simulations;
    \item the \textit{CorrectedSlope} is given by the road slope ($tan(\theta_r)$) plus the correction factor;
    \item the intercept for the constraint line is given by:\\
    $CorrectedIntercept = y_r + Lw/cos(\theta_r) - CorrectedSlope\times x_r$\\
    where $x_r$, $y_r$, and $\theta_r$ are the reference trajectory values at the first timestep of the prediction horizon, and $Lw$ is the lane width;
    \item the constraint is applied as $y > CorrectedSlope\times x + CorrectedIntercept$.
\end{enumerate}
Obviously, when the reference path is straight, the correction function becomes 0 and the constraint is the simple projection of the road.\\\\
In Zone 4, the situation is analogous to the one in Zone 2, with the only difference that here we are \textit{``relaxing"} the constraints, allowing the vehicle to come back to its original position on the right lane. The bounds here are defined by the blue cross point, called \textbf{\textit{End Point}} and the yellow diamond point, called \textbf{\textit{Entry Point}} (Figure \ref{fig:Zones}).\\
To define these bounds in a suitable form for the controller, we have transposed the information in a matrix form, describing two line equations:
\begin{equation}
    y > m_{lower}x + q_{lower}
\end{equation}
\begin{equation}
    y < m_{upper}x + q_{upper}
\end{equation}
where $m$ is the slope of the constraint and $q$ is the intercept of the constraint.
The slope of the constraint can be evaluated by calculating the tangent of the road orientation when we are considering the upper bound of the Zone 3, while it is evaluated as the slope between the points defined above in Zones 2 and 4.\\
The intercept is given by the projection of the reference trajectory, on the left lane center line for Zone 3 and on the left road limit for the upper bound, or it is evaluated by inverting the line equation in the changing lane maneuvers.\\
To let our algorithm work on all the quadrants, we make a check of the reference trajectory orientation and we adapt the constraint generation to the case we are in. This check projects the constraints on the $x$ axis rather than $y$ if the road inclination is closer to the vertical line in the X-Y plane, moreover it defines which are the relations between the bounds and the reference trajectory, intended as \textit{``lower than"} or \textit{``greater than"}, according to the direction of the vehicle and its left.
\subsection{Multiple obstacles avoidance}
The procedure described until now shows how the controller sets constraints when an obstacle is detected, but when multiple obstacles are present, things can become harder.\\
If two obstacles are very far one from the other, at least 400 $m$, they can be treated as single obstacles in sequence, but if they are close, a new definition of the zones is needed. In detail, we have decided to control the position of the next obstacle when we reach the \textit{End Point} of the previous. In this case, we can decide whether to come back in the right lane or to stay in the left lane and continue the overtaking maneuver. The procedure follows the rules below:
\begin{itemize}
    \item if the next obstacle is more than 400 $m$ far from the previous one, it is treated as an independent single obstacle;
    \item if the next obstacle is in the range 400 $m$ and 210 $m$ from the previous obstacle, it is not detected when the vehicle is in the \textit{End Point} of the previous obstacle. Here the vehicle simply comes back to the right lane and when the next obstacle is detected, it is treated as a single one;
    \item if the next obstacle is closer than 210 $m$ from the previous one, it is detected when the vehicle is in the \textit{End Point} of the latter, so here a decision is taken:
    \begin{enumerate}
        \item the next \textit{Detection Point} is after the previous \textit{Entry Point} $\xrightarrow{}$ the vehicle reenters in the right lane and starts the new overtaking maneuver when it reaches the next \textit{Detection Point};
        \item the next \textit{Detection Point} is before the previous \textit{Entry Point} $\xrightarrow{}$ the vehicle stays in the left lane and sets as target the next obstacle, so then it repeats this procedure when the next \textit{End Point} is reached.
    \end{enumerate}
\end{itemize}
To implement this control strategy in the simulation environment, we have used a counter on the obstacles, which takes into account if an obstacle has already been passed and in that case the detection algorithm neglects it, considering the next obstacle.\\
In real-world scenarios, a counter is not a feasible solution since it is impossible to know how many obstacles will be detected during a trip, but advanced navigation and sensor systems can detect and classify obstacles, and then can understand whether they have been passed or not, so our assumption is justified.


\section{object.size() and Strings}

\subsection{Comparison to object.size()}

\R contains a handy tool for telling you how big an already allocated
object is, namely the \code{object.size()} function from the \code{utils}
package.  The functions in this package are essentially an extension of that
function for unallocated, dense objects, provided your objects are numeric (more
on this later).

So say we have the vector \code{x <- 1.0}.  This should be using 8 bytes to
store that \code{1.0} as a double, right?  Well\dots
\begin{lstlisting}[language=rr]
object.size(1.0)
## 48 bytes
\end{lstlisting}

So where is all that extra space coming from?  Simply put, \R objects
are more than just their data.  They contain a great deal of very useful
metadata, which is where all the nice abstraction comes from.  Whenever you
create a vector, \R keeps track of, for example, its length.  If you
do not appreciate this convenience, go learn \proglang{C} and then get back to
me.

For vectors, this overhead is 40 bytes, regardless of the type of data.
Matrices, unsurprisingly cost more, clocking in at 200 bytes overhead.  It is
worth noting that this overhead does not scale; it is on a per-object basis.  So
we don't need 40 bytes for each element of a vector when just 8 would do (in the
case of double precision values).  We need 40 plus 8 per element:
\begin{lstlisting}[language=rr]
# 2 elements
40+8*2
## [1] 56
object.size(rnorm(2))
## 56 bytes

# 100.000 elements
40+1e5*8
## [1] 800040
object.size(rnorm(1e5))
## 800040 bytes
\end{lstlisting}

The story is slightly more complicated for integer data (and a lot more
complicated for strings; see the following section).  On my machine (and
probably yours, but not necessarily), \code{int}s costs 4 bytes.  However,
\R does some aggressive allocation:
\begin{lstlisting}[language=rr]
object.size(1L:3L)
## 56 bytes
object.size(1L:4L)
## 56 bytes
\end{lstlisting}
Here we see \R allocating more bytes than it needs for integer vectors
sometimes, choosing to allocate in 16 byte chunks rather than 8 byte chunks.

The \pkg{memuse} package does not adjust for this overhead, because it
honestly just doesn't matter.  This overhead is really paltry, and when you
think about all the abstraction it buys you, it's a hell of a bargain.  If you
have a million R objects stored, you're wasting less than one MiB ($1024^2$
bytes); so you would need a billion objects to use just about a GiB ($1024^3$
bytes) on overhead.  And if you're doing that kind of silly shit, my advice
would be to learn how to properly use data structures.

All that said, the main \code{memuse()} (\code{mu()} for short) function
will offer a better version of \code{object.size()}.  In previous
version, the package overwrote the default functionality, but as of
package version 3.0.0, that is no longer the case.

The \code{object.size()} function returns an object of class
\code{object_size}, which is a not-so-useful S3 class provided by core
\R.  If you instead use \code{memuse()}, then you will get a much more useful
\code{memuse} object.  So revisiting the code above:
\begin{lstlisting}[language=rr]
memuse(rnorm(1e5))
## 781.289 KiB
\end{lstlisting}

\subsection{Strings}

String objects have been avoided up until this point because they are much more
difficult to describe in general, unless they have a great deal of regularity
imposed on them.  In \R, strings by default are allocated to use 56
bytes (not counting overhead), unless they need more.  I'm not sure why this
value was chosen, but 56 byte strings will allow for the storage of 7 chars
(like \code{a} but not \code{aa}).  Each char costs 1 byte, so there's some fat
overhead for the strings here, and almost certainly an additional byte held out
for the null terminator.  So for example, recall that a vector allocates 40
bytes of overhead, so the vector string \code{letters} should use $56\times 26 +
40$ bytes.  We can easily verify that this is the case:
\begin{lstlisting}[language=rr]
56*26+40
## [1] 1496
memuse(letters)
## 1.461 KiB
\end{lstlisting}

If you have a string with more than 7 chars, \R will allocate extra
space in 8-16 byte blocks.  After the initial 8 byte allocation (7 chars $+$
null terminator), if you need more you get an additional 8 bytes (in reality
this is probably a contiguous 16 byte allocation; I have not bothered to check).
 Beyond that, storage is allocate in 16 byte blocks for each string.  For
example:
\begin{lstlisting}[language=rr]
memuse(c(paste(rep("a", 7), collapse=""), "a"))
## 152 B

  memuse(c(paste(rep("a", 7+1), collapse=""), "a"))
## 160 B

  memuse(c(paste(rep("a", 7+8+1), collapse=""), "a"))
## 176 B

  memuse(c(paste(rep("a", 7+8+16+1), collapse=""), "a"))
## 192 B
\end{lstlisting}

If you have a vector of strings with them of varying lengths, the allocation of
individual elements is handled on a case-by-case basis.  Consider the
following:
\begin{lstlisting}[language=rr]
  memuse(c(paste(rep("a", 7+8+16+1), collapse=""), "a"))
## 192 B
\end{lstlisting}

This object (the vector of 2 elements with first element
``aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'' and second element ``a'') is using 40
bytes for the vector, $56+8+16+16$ bytes for the first element, and 56 bytes for
the second.

For all of these reasons, and given the fact that I almost never (ever) deal
with character data, I have not bothered to make any attempt to extend, for
example, \code{howmany()} or \code{howbig()}, to incorporate strings.  Deal
with it, nerd.


\documentclass[11pt,a4paper,oneside]{report}             % Single-side

\usepackage{t1enc}
\usepackage[utf8]{inputenc}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{enumerate}
\usepackage{graphics}
\usepackage{listings}
\usepackage{color}
\usepackage{fancyvrb}
\usepackage{anysize}
\usepackage{setspace}
\usepackage[center]{caption}

\usepackage{titlesec}
	\titleformat{\chapter}[hang]
	{\normalfont\huge\bfseries}{\chaptertitlename\ \thechapter:}{1em}{}

\usepackage[hidelinks]{hyperref}
\usepackage{textcomp}
\usepackage{graphicx}
\usepackage{svg}

\onehalfspacing

\def\doubleunderline#1{\underline{\underline{#1}}}
\def\dul#1{\doubleunderline{#1}}
\def\ul#1{\underline{#1}}
\newcommand{\vect}[2]{\begin{bmatrix} #1 \\ #2 \end{bmatrix}}
\setcounter{MaxMatrixCols}{20}

\begin{document}

% Title page ==================================================================================
\begin{titlepage}
\begin{center}
{\huge Fast decoding of systematic Golay-24 code } \\
\vspace{0.5cm}
{\small Description of the algorithm used on board the SMOG-P, ATL-1 and SMOG-1 satellites } \\
{\large Timur Kristóf } \\

\vfill
{\large \today}
\end{center}
\end{titlepage}

\chapter{Introduction}

The Golay forward error correction code was invented by Marcel Golay in 1949. Its most well-known
use was onboard the NASA Voyager satellites where they used the Golay(24, 12, 8) code. These days
we know codes with much better error correction capabilities, but the Golay code is still a good
choice in applications where the size of each packet is small and/or the available hardware
resources are too modest, so there is no chance to use more complicated coding schemes.

The topic of this study is the \emph{systematic Golay(24, 12, 8)} code, called \emph{Golay-24} for short.
The aim is to present a new algorithm which can quickly and efficiently
decode this coding scheme even on machines with limited compute capabilities,
such as embedded systems.

This study not only presents the algorithm but also proves its correctness.

The algorithm presented here is a \emph{hard-decision} algorithm, since most of the embedded hardware
that I've worked with make a hard decision about the received bits, so the aim was to make decoding
efficient on this class of hardware.
Further research needs to be done to determine if the ideas presented here can be utilized in soft-decision algorithms.

\section{Definitions}

\doubleunderline{A} is the Golay matrix.

\[
    \doubleunderline{A} =
    \begin{bmatrix}
            1 & 0 & 0 & 1 & 1 & 1 & 1 & 1 & 0 & 0 & 0 & 1 \\
            0 & 1 & 0 & 0 & 1 & 1 & 1 & 1 & 1 & 0 & 1 & 0 \\
            0 & 0 & 1 & 0 & 0 & 1 & 1 & 1 & 1 & 1 & 0 & 1 \\
            1 & 0 & 0 & 1 & 0 & 0 & 1 & 1 & 1 & 1 & 1 & 0 \\
            1 & 1 & 0 & 0 & 1 & 0 & 0 & 1 & 1 & 1 & 0 & 1 \\
            1 & 1 & 1 & 0 & 0 & 1 & 0 & 0 & 1 & 1 & 1 & 0 \\
            1 & 1 & 1 & 1 & 0 & 0 & 1 & 0 & 0 & 1 & 0 & 1 \\
            1 & 1 & 1 & 1 & 1 & 0 & 0 & 1 & 0 & 0 & 1 & 0 \\
            0 & 1 & 1 & 1 & 1 & 1 & 0 & 0 & 1 & 0 & 0 & 1 \\
            0 & 0 & 1 & 1 & 1 & 1 & 1 & 0 & 0 & 1 & 1 & 0 \\
            0 & 1 & 0 & 1 & 0 & 1 & 0 & 1 & 0 & 1 & 1 & 1 \\
            1 & 0 & 1 & 0 & 1 & 0 & 1 & 0 & 1 & 0 & 1 & 1
    \end{bmatrix}
\]

\doubleunderline{G} is the generator matrix of the systematic Golay(24, 12, 8) code.

\[
    \doubleunderline{G} =
    \begin{bmatrix}
        \doubleunderline{I} \\
        \doubleunderline{A}
    \end{bmatrix}
\]

If \underline{m} is the message (which we want to encode), and the parity vector is \underline{p},
then the encoded code word is \underline{x}.
The Hamming weight of the code word is $w\{\ul{x}\} = w\{\ul{m}\} + w\{\ul{p}\}$.

\[
    \underline{x} = \begin{bmatrix}
        \underline{m} \\
        \underline{p}
    \end{bmatrix} = \doubleunderline{G} \cdot \underline{m} = \begin{bmatrix}
        \doubleunderline{I} \\
        \doubleunderline{A}
    \end{bmatrix} \cdot \underline{m} = \begin{bmatrix}
        \doubleunderline{I} \cdot \underline{m} \\
        \doubleunderline{A} \cdot \underline{m}
    \end{bmatrix} = \begin{bmatrix}
        \underline{m} \\
        \doubleunderline{A} \cdot \underline{m}
    \end{bmatrix}
\]

If the \underline{x} code word is transmitted through a noisy channel, the received word might
contain errors. The error vector is \underline{e} (the error vector in the received message
is $\underline{e}_m$ and the error vector in the received parity is $\underline{e}_p$)
and the received word is \underline{r}.

\[
    \ul{r} = \ul{x} + \ul{e} = \vect{\ul{r}_m}{\ul{r}_p} = \vect{\ul{m} + \ul{e}_m}{\ul{p} + \ul{e}_p}
\]

The Golay-24 code is capable of correcting 3 or fewer
bit errors, and detecting 4 bit errors. So the cases we care about the most are when $w\{\ul{e}\} \leq 3$
or $w\{\ul{e}\} = 4$.

\chapter{Algorithm}

This chapter presents the decoding algorithm. We will make the assumption that $w\{\ul{e}\} \leq 4$,
so the received vector contains at most 4 bit errors. This is reasonable because the Golay(24, 12, 8)
code is only capable to correct at most 3 bit errors, and detect 4 bit errors. Since the minimal
Hamming distance between each code word is 8, in case of 4 bit errors it is possible to create the
received vector from multiple different code words, therefore it is not possible to tell which was
the original.

\section{Basics}

First a few basic considerations are presented on which the algorithm is based.

\subsubsection{Inverse and transpose of the Golay matrix}

The \dul{A} Golay matrix is its own inverse and transpose.

\[
    \dul{A}^T = \dul{A}
\]
\[
    \dul{A}^{-1} = \dul{A}
\]

Consequences:
\begin{itemize}
    \item \dul{A} is an invertable linear transformation, so if $\ul{p} = \dul{A} \cdot \ul{m}$, then
          $\dul{A} \cdot \ul{p} = \dul{A} \cdot \dul{A} \cdot \ul{m} = \dul{A} \cdot \dul{A}^{-1} \cdot \ul{m} = \ul{m}$.
          In other words, the message can be restored from the parity vector.
    \item We only need to store the \dul{A} matrix in memory, and we don't need to ever invert or transpose it.
\end{itemize}

\subsubsection{Parity of messages with 1 Hamming weight}

If $\ul{m} = \ul{I}_{i}$, then $\ul{p} = \dul{A} \cdot \ul{m} = \dul{A} \cdot \ul{I}_{i} = \ul{A}_{i}$.
So if the message is the same as the $i$th row of the identity matrix $\dul{I}$, then the parity vector is the
same as the $i$-th row of the Golay matrix $\dul{A}$.

Consequences:
\begin{itemize}
    \item When changing a single bit at $\ul{I}_{i}$ in the message, the equivalent change to the parity is $\ul{A}_{i}$.
    \item When changing a single bit at $\ul{I}_{i}$ in the parity, the equivalent change to the message is $\ul{A}_{i}$.
\end{itemize}

\subsubsection{Hamming weight of code words from non-zero Hamming weight messages}

If $\ul{m} \neq \ul{0}$, then $w\{ \ul{x} \} = w\{ \ul{m} \} + w\{ \ul{p} \} \geq 8$.

Proof:
We already know that the Hamming distance between every possible code word is at least 8.
(Not proven here, but it's a well-known fact about the Golay code.)
The $\ul{m} = \ul{0}$ message is encoded to the code word $\ul{x} = \ul{0}$, so
the minimal Hamming distance of every possible code word can only be at least 8 if
in case of $\ul{m} \neq \ul{0}$ the $w\{ \ul{x} \} \geq 8$, otherwise there
would be a code word $\ul{x}$ whose Hamming distance from the $\ul{0}$ code word is less than 8.

Consequence:
\begin{itemize}
    \item $\forall \ul{m} \neq \ul{0} : w\{ \dul{A} \cdot \ul{m} \} \geq 8$
\end{itemize}

\subsubsection{Lemma 1.}

If $w\{\ul{e}\} \leq 4$ and $w\{\dul{A} \cdot \ul {r_m} + \ul{r}_p\} \leq 3$,
then $w\{\ul{e}_m\} = 0$ (which is equivalent to $\ul{e}_m = \ul{0}$).
So, assuming 4 or fewer bit errors, if we calculate the parity from the received message, and
the Hamming distance between the calculated parity and the received parity is less than
or equal to 3, then the received message is error-free.

Proof:

Let's suppose that the statement is false, and $\ul{e}_m \neq \ul{0}$.
The given expression looks like:

\[
    \dul{A} \cdot \ul{r}_m + \ul{r}_p =
    \dul{A} \cdot ( \ul{m} + \ul{e}_m ) + ( \ul{p} + \ul{e}_p ) =
    \dul{A} \cdot \ul{m} + \dul{A} \cdot \ul{e}_m + \dul{A} \cdot \ul{m} + \ul{e}_p =
    \dul{A} \cdot \ul{e}_m + \ul{e}_p
\]

Consider the code word generated from $\ul{e}_m$. The minimum Hamming distance between every possible code word is 8, therefore
in case of $\forall \ul{e}_m \neq \ul{0}$ the following is true:
$w\{ \ul{e}_m \} +  w\{ \dul{A} \cdot \ul{e}_m \} \geq 8$.
(See "Hamming weight of code words from non-zero Hamming weight messages" above.)

Let's define $w\{ \ul{e}_m \} = N$, then because $w\{ \ul{e} \} \leq 4$ the following
will also be true: $w\{ \ul{e}_p \} \leq 4 - N$.
So, $w\{ \dul{A} \cdot \ul{e}_m \} \geq 8 - N$.

So according the above, we get this:
\[
    w\{ \dul{A} \cdot \ul{e}_m + \ul{e}_p \} \geq (8 - N) - (4 - N) = 4
\]

What we got is that when $\ul{e}_m \neq \ul{0}$, then
$w\{ \dul{A} \cdot \ul{e}_m + \ul{e}_p \} \geq 4$ which is a contradiction, because
we started with the condition that it's $\leq 3$.

So we can conclude that the statement is true, and $\ul{e}_m = \ul{0}$.

\subsubsection{Lemma 2.}

If $2 \leq w\{\ul{e}\} \leq 4$ and $w\{\dul{A} \cdot \ul{r}_m + \ul{A}_i + \ul{r}_p\} \leq 2$,
then $\ul{e}_m = \ul{I}_i$. So if we assume the number of errors to be at least 2 but at most 4,
and we correct a 1-bit error $\ul{e}_{m, i} = \ul{I}_i$ in the received message, then we calculate
the parity of the corrected message, and the Hamming distance between the calculated parity and
the received parity is less than or equal to 2, then the received message contains exactly 1 error,
and that error is the $i$th row of the identity matrix, which is what was corrected.

Proof.

Consider the $\ul{r}_2 = \vect{\ul{r}_{m,2}}{\ul{r}_{p,2}} = \vect{\ul{r}_m + \ul{I}_i}{\ul{r}_p}$ received word.

Because of Lemma 1, we know that
if $w\{ \dul{A} \cdot \ul{r}_{m,2} + \ul{r}_{p,2} \} \leq 3$, then $\ul{e}_{m,2} = \ul{0}$.

The above formula of $\ul{r}_2$ yields the following result:

\[
    \dul{A} \cdot \ul{r}_{m,2} + \ul{r}_{p,2} =
    \dul{A} \cdot ( \ul{r}_m + \ul{I}_i ) + \ul{r}_p =
    \dul{A} \cdot \ul{r}_m + \dul{A} \cdot \ul{I}_i + \ul{r}_p =
    \dul{A} \cdot \ul{r}_m + \ul{A}_i + \ul{r}_p
\]

The above conditions contain $w \{ \dul{A} \cdot \ul{r}_m + \ul{A}_i + \ul{r}_p \} \leq 2$,
so for the received word $\ul{r}_2$ the following is true: $\ul{e}_{m,2} = \ul{0}$.

Since $\ul{r}$ only differs from $\ul{r}_2$ by only 1 message bit, the error of the parity is
$\ul{e}_p = \ul{e}_{p,2}$. The differing message bit is at $\ul{I}_i$.
So if $\ul{e}_{m,2} = \ul{0}$, then $\ul{e}_m = \ul{I}_i$.

\subsubsection{Lemma 3.}

Same as Lemma 1, but the other way around:

If $w\{\ul{e}\} \leq 4$ and $w\{\dul{A} \cdot \ul {r_p} + \ul{r}_m\} \leq 3$,
then $w\{\ul{e}_p\} = 0$  (which is equivalent to $\ul{e}_p = \ul{0}$).
So, if we calculate the message from the received parity, and
the Hamming distance between the calculated message and the received message is less than
or equal to 3, then the received parity is error-free. (And the received message is
$\dul{A} \cdot \ul {r_p}$.)

This can be proved in the exact same way as Lemma 1.

\subsubsection{Lemma 4.}

Same as Lemma 2, but the other way around:

If $2 \leq w\{\ul{e}\} \leq 4$ and $w\{\dul{A} \cdot \ul{r}_p + \ul{A}_i + \ul{r}_m\} \leq 2$,
then $\ul{e}_p = \ul{I}_i$. So if we assume the number of errors to be at least 2 but at most 4,
and we correct a 1-bit error $\ul{e}_{p, i} = \ul{I}_i$ in the received parity, then we calculate
the message of the corrected parity, and the Hamming distance between the calculated message and
the received message is less than or equal to 2, then the received parity contains exactly 1 error,
and that error is the $i$th row of the identity matrix, which is what we corrected.

This can be proved in the exact same way as Lemma 2.

\section{Steps of the algorithm}

\subsubsection{Step 1.}

Consider the $\ul{r}$ received vector.
Split $\ul{r}$ into received message and received parity:

\[
    \ul{r} = \vect{\ul{r}_m}{\ul{r}_p}
\]

\subsubsection{Step 2.}

Calculate the parity of the received message part: $\ul{p}_2 = \dul{A} \cdot \ul{r}_m$.

\subsubsection{Step 3.}

If $\ul{p}_2 = \dul{A} \cdot \ul{r}_m = \ul{p}$ then the received word contains no error, so we're done.

\subsubsection{Step 4.}

When the received message is error-free but the parity contains $w\{\ul{e}_p\} \leq 3$ errors, then according to
\emph{Lemma 1} we can tell this by looking at the Hamming distance between the received parity $\ul{r}_p$ and
the calculated parity $\ul{p}_2$ which is $d = w\{\ul{r}_p + \ul{p}_2\}$.

If $d \leq 3$ then the received message is error-free, and only the received parity contained at most 3 errors.

\subsubsection{Step 5.}

Calculate the message from the received parity, which will be $\ul{m_2} = \dul{A} \cdot \ul{r}_p$.

\subsubsection{Step 6.}

When the received parity is error-free but the received message has $w\{\ul{e}_m\} \leq 3$ errors, then according to
\emph{Lemma 2} we can tell this by looking at the Hamming distance between the received message $\ul{r}_m$
and the calculated message $\ul{m_2}$ which is $d = w\{\ul{r}_m + \ul{m_2}\}$.

If $d \leq 3$ then the received parity is error-free, and only the received message contained at most 3 errors.
So the decoded message is $\ul{m_2}$.

\subsubsection{Step 7.}

So far, the algorithm has covered every possible 1-bit error and all the 2 and 3-bit errors which were
limited to only the message or the parity. Let's now deal with the case where the received message contains 1 error,
and the received parity contains 1 or 2 errors.

For $\forall i : 1 \leq i \leq 12, \ul{e}_m = \ul{I}_i$ cases, let's look at the possible 1-bit errors of the
received message, and correct them in the following manner:

\[
    \ul{r}_{corr, i} = \vect{\ul{r}_{m, corr, i}}{\ul{r}_{p, corr, i}} = \vect{\ul{r}_m + \ul{I}_i}{\ul{r}_p + \ul{A}_i}
\]

Let's also calculate the parity of the corrected message:

\[
    \ul{p}_{i} = \dul{A} \cdot \ul{r}_{m, corr, i}
\]

There are 12 possible corrections, the right one is the case when:

\[
    w\{\ul{p}_{i} + \ul{r}_{p, corr, i}\} \leq 2
\]

Thanks to \emph{Lemma 2}, whichever $\ul{r}_{corr, i}$ satisfies the above formula, the correspoding $\ul{r}_{m, corr, i}$ will be
the decoded message.

\subsubsection{Step 8.}

Let's deal with the cases when the received parity has 1 error, and the received message has 1 or 2 errors.

For $\forall i : 1 \leq i \leq 12, \ul{e}_p = \ul{I}_i$ cases, let's look at the possible 1-bit errors of the
received parity, and correct them in the following manner:

\[
    \ul{r}_{corr, i} = \vect{\ul{r}_{m, corr, i}}{\ul{r}_{p, corr, i}} = \vect{\ul{r}_m + \ul{A}_i}{\ul{r}_p + \ul{I}_i}
\]

Let's calculate the message from the corrected parity:

\[
    \ul{m}_{i} = \dul{A} \cdot \ul{r}_{p, corr, i}
\]

There are 12 possible corrections, the right one is the case when:

\[
    w\{\ul{m}_{i} + \ul{r}_{m, corr, i}\} \leq 2
\]

Thanks to \emph{Lemma 4}, whichever $\ul{r}_{corr, i}$ satisfies the above formula, the corresponding $\ul{m}_{i}$  will be
the decoded message.

\subsubsection{Step 9.}

If the algorithm reached this point, it means that we've already covered all possible 1, 2 and 3 bit errors.
So we've detected a 4-bit error which we can't correct.

\end{document}


\subsection{Solvent Accessible Surface}\index{SAS|ff}
\index{Solvent Accessible Surface|ff}
The solvent accessible surface is a continuous surface of the molecule which
can be reached by the center of charge of a solvent molecule.  The calculation
of the SAS is carried out as follows:

\begin{itemize}
\item Each atom is assigned a van der Waals' radius.  VdW radii used in COSMO
are given in Table~\ref{vdw}.
\begin{table}
\caption{\label{vdw} Van der Waals radii (\AA ) used in COSMO}
\begin{center}
\begin{tabular}{llllllllllllll}
\hline
 I  & R    & II & R    &III& R    &IV & R    & V & R    &VI & R    &VII& R \\
\hline
 H  & 1.08 & \\
 Li & 1.80 &    &      &   &      & C & 1.53 & N & 1.48 & O & 1.36 & F & 1.30\\
 Na & 2.30 &    &      & Al& 2.05 & Si& 2.10 & P & 1.75 & S & 1.70 & Cl& 1.65\\
 K  & 2.80 & Ca & 2.75 &   &      &   &      &   &      &   &      &Br & 1.80\\
    &      &    &      &   &      &   &      &   &      &   &      &I  &  2.05\\
\hline
\end{tabular}
\end{center}
\end{table}

\item To each radius is added a distance equal to the radius of the solvent. By
default, this is 1.0\AA, but may be changed by the user using
\comp{RSOLV=$n.nn$}. This gives the distance from the nucleus to the center of
a solvent molecule.
\item A set of points is generated on this surface. These points produce a
basic grid.
\item All points which are inside the surface of any other atom are excluded.
\item The remaining points are moved towards the center of the atom.  The
distance moved is equal to the distance of the center of charge of the solvent
molecule from the center of the solvent molecule.  By default, this distance is
set to \comp{RSOLV}, but may be set explicitly by keyword \comp{RSOLV=$n.nn$}.
\item Each of the remaining points represents a small area of the solvent
accessible surface.  The total SAS is calculated from the number of points.
\end{itemize}

From this definition of the SAS we see that the SAS of each atom is a  surface
of radius equal to the van der Waals' radius plus the radius of the solvent
molecule minus the distance of the center of charge of the solvent molecule to
the center of the solvent molecule.  In other words, the radius is the VdW
radius plus the distance from the surface of the solvent molecule to the center
of charge of the solvent molecule.  By default, this extra distance is zero.
Only that part of the atom surface which can be touched by the solvent molecule
is used.  This means that only those atoms on the surface of the molecule can
contribute to the SAS.  Of those atoms  that are on the surface of the molecule
there will be parts of the surface which cannot be reached by the solvent
because the solvent molecule is too bulky.


\chapter{Generation of Band Structures}
\section{Program BZ}
\subsection{Description} \index{BZ!description of}
Unlike molecules, whose eigenvalue spectrum consists of infinitely sharp lines
(Kronecker-$\delta$ type), the eigenvalues of solids form bands.  These bands
are conventionally represented by Brillouin Zones (BZ).  For a polymer, the BZ
consists of a line, for a layer structure, a surface, and for a solid, a
three-dimensional shape.  The dimensions of the BZ are inverse distance: if two
polymers have unit cells of 10 and 8 \AA, then the ratio of their unit cell
translation vectors is 1.25:1, and the ratio of their BZ's is 1:1.25.

Although MOPAC can generate all the information necessary  to generate
band-structures, for ease of use this job is given to a special program called
BZ. \  MOPAC is designed to run in batch mode, but drawing band-structures is
an interactive operation.

Before BZ can be run, a MOPAC calculation must first be run in order to
generate the data needed by BZ. \  Most of this data is invisible to the user,
but the user must supply to MOPAC one or two data which BZ will use.  These
data are:

\begin{description}
\item[\comp{MERS=($n_1$,$n_2$,$n_3$)}]~\\
(See Chapter~\ref{makpol:mers}  and Section~\ref{mopac:mers})  BZ needs to know how
the Fock matrix is organized, and  \comp{MERS} gives the number of unit cells
in each direction.  MOPAC uses the keyword \comp{MERS} as the prompt to write
out a file for use by BZ.  Within MOPAC, \comp{MERS} has no other use.
\index{MERS}
\item[\comp{BCC} \normalfont{(optional)}]~\\
Like \comp{MERS}, \comp{BCC} is not used by MOPAC. \comp{BCC} (see
Chapter~\ref{makpol:bcc})  indicates that the system is Body-Centered-Cubic; in
other words, that every odd unit cell is missing.  If the keyword \comp{BCC} is
present, MOPAC informs BZ that the system is Body-Centered-Cubic.
\end{description}

BZ is an interactive utility program.  It can perform three types of
calculation:
\begin{itemize}
\item  Points in $k$-space are to be studied.
\item  Lines in $k$-space are to be drawn.
\item  Surfaces in $k$-space are to be drawn.
\end{itemize}

\subsection{Data input for BZ}\index{Data! for BZ}
Once the MOPAC calculation has been completed, the band structure can be
generated. The instruction to generate band structures is:

\comp{bz $<$filename$>$}

where \comp{$<$filename$>$} is the name of the file. This must be the same as
that used for the MOPAC calculation.

When BZ starts, it invites the user to specify the type
of calculation to be performed. The allowed types are:
\begin{description}
\item[0] Points in $k$-space are to be studied.
\item[1] Lines in $k$-space are to be drawn.
\item[2] Surfaces in $k$-space are to be drawn.
\end{description}

Instead of supplying data at the prompt, a data set can be supplied, using the
 command:

\comp{bz $<$filename$>$ $<$filename2$>$}

A second file, called \comp{$<$filename2$>$.bz}, must exist.  This file will
contain the data that would otherwise be supplied by the prompt.

\index{Help! with BZ}
In addition, the user is invited to enter the number `3' to request help. Help
describes diagnostic key-words which allow an interested user to follow the
calculation.

Some of the operations in BZ are very sensitive to perturbations in the Fock
matrix, specifically any perturbations which lower the symmetry.  The first
thing BZ does, therefore, is to restore symmetry.  To do this, BZ reads in a
list of symmetry operations and symmetrizes the Fock matrix.

At the present time, the user must supply all operations of the group.
Eventually this will be done automatically by the program, but for now the job
of defining symmetry is given to the user.

\subsubsection{Structure of $<$filename$>$.ops}
\index{Space group!operations}
This file holds the definitions of the symmetry operations for the space-group.
The format is `free format' - one or more spaces are treated as one space. The
first line should contain one of the following two words: ``Cartesian'' or
``Crystallographic''.  If ``Cartesian'' is selected, then the rotation axis is
relative to the Cartesian coordinates of the atoms; if ``Crystallographic'' is
selected, then the rotation axis is relative to the crystal coordinates of the
atoms.  (In practice, ``Cartesian'' is {\em much} easier to use.)

Each operation is defined by one line and is composed of 12 data.
In order, these data are:
\begin{enumerate}
\item A `1' or `0'.  `1' if the operation is composed of the inversion operation times
a rotation, `0' if the operation is a pure rotation.
\item The non-primitive translations.  These are given as their reciprocals, thus
a (0.5,0.5,0.0) translation would be entered as `2~2~0'. The absence of
a non-primitive translation is indicated by a zero. (three numbers)
\item The rotation expressed as a fraction of a circle.
\item The axis about which the operation is performed. Units are
Cartesian coordinates, or, if ``Crystallographic'' is selected, the crystal
coordinates. (three numbers)
\item The center about which the operation is performed. Units are
Cartesian coordinates, in Angstroms. (three numbers)
\item A text string describing the operation.
\end{enumerate}

The operations list is terminated by an operation which has a `2' for the
inversion option.

Examples of these operations are illustrated in Table~\ref{sgo}
by the 10 classes of operation for the O$_h$ group in O$_h^7$.

The Fock matrix will be symmetrized if, and only if, every operation of
the group is represented once.  For O$_h$, this would involve a total of 49
lines, 48 for the operations of O$_h$, and one line to terminate the operations
file.

% 24 lines, including this one
\begin{table}
\caption{\label{sgo}
 Space-Group Operations of O$_h^7$}
\index{oh7@{$O_h^7$}|ff}
\begin{center}
\begin{tabular}{lllllllrrrll}\\ \hline
i& \multicolumn{3}{c}{N.P.T}  & Rotation & \multicolumn{3}{c}{Axis of Rotation} &
 \multicolumn{3}{c}{Center of Operation}& Name\\ \hline
0& 0& 0& 0& 0.0     &  0.0&0.0 &1.0 &  0.0& 0.0& 0.0 &  Identity  \\
0& 0& 0& 0& 0.3333  &  1.0&1.0 &1.0 &  0.0& 0.0& 0.0 &  C$_3$  \\
0& 0& 0& 0& 0.5     &  1.0&0.0 &0.0 &  0.0& 0.0& 0.0 &  C$_2$=C$_4^2$  \\
0& 2& 2& 2& 0.5     &  1.0&0.0 &-1.0 & 0.0& 0.0& 0.0 &  C$_2$  \\
0& 2& 2& 2& 0.25    &  0.0&0.0 &1.0 &  0.0& 0.0& 0.0 &  C$_4$  \\
1& 2& 2& 2& 0.0     &  0.0&0.0 &1.0 &  0.0& 0.0& 0.0 &  Inversion  \\
1& 0& 0& 0& 0.25    &  0.0&0.0 &1.0 &  0.0& 0.0& 0.0 &  S$_4$  \\
1& 0& 0& 0& 0.5     &  1.0&1.0 &0.0 &  0.0& 0.0& 0.0 &  $\sigma_d$  \\
1& 2& 2& 2& 0.333333&  1.0&1.0 &1.0 &  0.0& 0.0& 0.0 &  S$_6$  \\
1& 2& 2& 2& 0.5     &  0.0&0.0 &1.0 &  0.0& 0.0& 0.0 &  $\sigma_h$  \\
2& 0& 0& 0& 0.0     &  0.0&0.0 &1.0 &  0.0& 0.0& 0.0 &  End  \\ \hline
\end{tabular}

N.P.T.: Non-Primitive Translation \end{center}
\end{table}

\subsection{Calculating Points in $k$-space}
\index{ksp@{$k$-space}!calculating points in}

A point in $k$-space is specified by one number for a polymer, two
numbers for a layer system, and three numbers for a solid.   All symmetry
points and lines for diamond are shown in Table~\ref{sod} and Figure~\ref{oh7}.
% 21 lines, including this one
\begin{figure}
\begin{makeimage}
\end{makeimage}
% \setlength{\unitlength}{0.1cm}
\begin{center}
% \begin{picture}(100,100)
% \put(62,47){$\Gamma$}
% \put(106,42){X}
% \put(108,68){W}
% \put(107.5,58){$\longleftarrow$Z}
% \put(96.5,54){S$\longrightarrow$}
% \put(84,44){$\Delta$}
% \put(100.2,59.5){U}
% \put(76,71){L}
% \put(92,71){Q}
% \put(56,80){K}
% \put(61,63){$\Sigma$}
% \put(69,59){$\Lambda$}
% %\put(14,38){\psfig{figure=oh7.ps}}
% \put(14,38){\includegraphics{oh7}}
% \end{picture}
\includegraphics{oh7}
\end{center}
\caption{\label{oh7}Brillouin zone for Diamond}
\end{figure}
A ``random'' point is indicated by the number 0.123.  (Note: 0.377 = 0.500 -
0.123)

These high-symmetry points have associated subgroup symmetries, thus $\Gamma $
has the symmetry O$_h$, $\Lambda $ has the symmetry C$_{3v}$, and X has the
symmetry D$_{4h}$.

\subsubsection{Little Groups}\index{Little groups|(}
The symmetry operations supplied in \comp{ $<$filename$>$.ops} are used to
generate a little group of the space group.  A ``little group'' is the
sub-group of the space-group at a specific point in $k$-space.  There are too
many little groups to allow all the irreducible representations to be labeled.
In addition, there is no generally-accepted convention for describing them.
Because of this, the little group is given as a group-table.  Thus the little
group for $\Gamma$ would have the form shown in Table~\ref{lgd}.

% 22 lines, including this line
\begin{table}
\begin{center}
\caption{\label{sod} Symmetry Points in $k$-space for Diamond}
\begin{tabular}{lllcc} \\ \hline
\multicolumn{3}{c}{Point} & Little Group* & Symbol\\ \hline
  0.000& 0.000& 0.000  & O$_h$ &  $\Gamma$\\
  0.500& 0.000& 0.000  & (D$_{4h}$)&X\\
  0.500& 0.250& 0.000  & (C$_{2v}$)&W\\
  0.500& 0.125& 0.125  & (C$_{2v}$)&U\\
  0.375& 0.375& 0.000  & (C$_{2v}$)&K\\
  0.250& 0.250& 0.250  & D$_{3d}$ &L\\
  0.123& 0.000& 0.000  & C$_{4v}$ &$\Delta $\\
  0.500& 0.123& 0.000  & (C$_{2v}$) &Z\\
  0.500& 0.045& 0.045  & (C$_{2v}$)&S\\
  0.123& 0.123& 0.000  & C$_{2v}$ &$\Sigma $\\
  0.123& 0.250& 0.377  &  C$_{2}$ &Q\\
  0.123& 0.123& 0.123  & C$_{3v}$ &$\Lambda $\\ \hline
\end{tabular}

* Point groups in parentheses only approximate little groups.\end{center}\end{table}
\begin{table}
\begin{center}
\caption{\label{lgd} Little Group for $\Gamma$ for Diamond}
\begin{tabular}{lrrrrrrrrrrrr}\\ \hline
Level & Energy & $\Gamma$ & E & C$_3$ & C$_2$=C$_4^2$ & C$_2$ & C$_4$
 & I & S$_4$ & $\sigma_d$ & S$_6$
&$\sigma_h$ \\ \hline
  1& -58.0408& 1& 1& 1& 1& 1 &1&1&1&1&1&1 \\
  2&  -9.4687& 2& 3& 0&-1& 1&-1&3&-1&1&0&-1 \\
  3&   0.9807& 3& 3& 0&-1&-1&1&-3&-1&1&0&1 \\
  4&   4.1933& 4& 1& 1& 1&-1&-1&-1&1&1&-1&-1\\ \hline
\end{tabular}\end{center} \end{table}

From this we see that there are four different irreducible representations: 1,
2, 3, and 4. These representations can be equated with those of the O$_h$
point-group, so 1 and 4, being non-degenerate, become $a$, while 2 and 3, being
3-fold degenerate, become $t$. So far we have ($a$, $t$, $t$, $a$).  The effect
of inversion on the eigenfunctions allows us to assign {\em gerade} and  {\em
ungerade}, thus ($a_g$, $t_g$, $t_u$, $a_u$), while C$_4$ resolves the
``1''-``2'' nature of the representations: ($a_{1g}$, $t_{2g}$, $t_{1u}$,
$a_{2u}$).

Inspection of the eigenvectors reveals, as expected, that the two
$a$ representations can be identified with the $s$ atomic orbitals, and the two
$t$ representations, with the six $p$ atomic orbitals.

For points other than $\Gamma$, the little groups become complex. Consider a
point on $\Delta$, the line connecting $\Gamma$ and X, or (0,0,0) and
(1/2,0,0). The little group for (0.123,0,0) has the form shown in
Table~\ref{dig}.
\begin{table}
\begin{center}
\caption{\label{dig} Little Group for $\Delta$ for Diamond}
 \index{Diamond!little group|(}
\begin{tabular}{lrlrrrrr} \\ \hline
Level&Energy&$\Gamma$&E &C$_2$=C$_4^2$ &C$_4$&$\sigma_d$
&$\sigma_h$  \\  \hline
  1& -54.2893 &1 & 1& 1& $\theta$    &   1& $\theta$   \\
  2& -10.2758 &2 & 2&-2& 0           &   0& 0              \\
  3& -10.1336 &3 & 1& 1&-$\theta$    &   1& -$\theta$   \\
  4&   0.9087 &1 & 1& 1& $\theta$    &   1& $\theta$   \\
  5&   1.5003 &2 & 2&-2& 0           &   0& 0  \\
  6&   3.7878 &3 & 1& 1&-$\theta$    &   1& -$\theta$  \\ \hline
\end{tabular}\end{center}\end{table}
Here $\theta$ = $e^{-ik/2\pi}$.

As is seen, complications arise for those operations which involve
\index{Characters!complex}\index{Complex characters} non-primitive
translations,\index{Translations!non-primitive} \index{Non-primitive
translations} e.g.\  C$_{4}$ and $\sigma_h$, when points other than $\Gamma$ are
studied. The complex phase-factor can cause interpretation of the group
characters to be difficult.  One option is to correct for the phase-factor by
multiplying the character by $e^{ik/2\pi}$. This would make the little group
for $\Delta$ easier to read (Table~\ref{mdig}).

\begin{table}
\begin{center}
\caption{\label{mdig} Modified Little Group for $\Delta$ for Diamond}
\begin{tabular}{lrlrrrrr} \\ \hline
Level&Energy&$\Gamma$&E &C$_2$=C$_4^2$ &C$_4$&$\sigma_d$
&$\sigma_h$  \\ \hline
  1& -54.2893 &1 & 1& 1& 1     &   1& 1    \\
  2& -10.2758 &2 & 2&-2& 0     &   0& 0              \\
  3& -10.1336 &3 & 1& 1&-1     &   1& -1    \\
  4&   0.9087 &1 & 1& 1& 1     &   1& 1    \\
  5&   1.5003 &2 & 2&-2& 0     &   0& 0  \\
  6&   3.7878 &3 & 1& 1&-1     &   1& -1   \\ \hline
\end{tabular}\end{center}\end{table}

However, for some little groups, this procedure may cause complications. Thus,
at $L$, the inversion operation becomes sensitive to the point chosen, e.g.
(0.25,0.25,0.25) will give a different character for the $I$ operation than
(0.25,0.25,-0.25).

A third option is to assume that the lowest-lying level represents the totally
symmetric representation.  (Note: this is different from transforming as the
totally symmetric representation.  Only one point, the $\Gamma$-point, can ever
contain a totally symmetric representation.) By making that assumption, all
characters can be related to those of the  lowest-lying level.

Users are given key-words which select these alternatives to the basic
character tables when the $k$-point is specified.  The key-words are:

\begin{description}
\index{One! use in BZ}\index{BZ!use of ``one''}
\item[\comp{one}] Assume the lowest energy level is totally symmetric.  All operations
involving non-primitive translations are then multiplied by a (complex) factor
which will make the lowest energy level totally symmetric. Example: \comp{0.123 0 0 one}.

\index{Byk! use in BZ}\index{BZ!use of ``byk''}
\item[\comp{byk}] Multiply all operations which involve non-primitive translations by
a phase factor (``byk''---``multiply by k'')  which depends on the current value
of $k$. Example: \comp{0.123 0 0 byk}.
\end{description}

Using the points option, assignment of the symmetry of lines and points in
$k$-space becomes very easy.

\subsection{Lines in $k$-space}

A line  in $k$-space is specified by two numbers for a polymer, four numbers
for a layer system, and six numbers for a solid.  The number of points
calculated for each line is proportional to the length of the line in
$k$-space. For a line of length 1.0, for example $\Gamma$ to $\Gamma$, 300
points would be calculated. \index{Walks in
$k$-space}\index{ksp@{$k$-space}!walks} The data needed to specify a
band-structure `walk' from $\Gamma $ to X to  W to L to $\Gamma $ is given in
Table~\ref{walk}, and for diamond, the band-structure for this `walk' is given
in Figure~\ref{walk-pic}.

\begin{table}
\begin{center}
\caption{\label{walk}A ``Walk'' in $k$-Space ($\Gamma $ to X to W to L to $\Gamma $)}
\begin{tabular}{ccccccccc} \\ \hline
Start && Stop&\multicolumn{3}{c}{Start} & \multicolumn{3}{c}{Stop} \\ \hline
$\Gamma$&-&X & 0.000& 0.000& 0.000& 0.500& 0.000& 0.000  \\
X       &-&W        & 0.500& 0.000& 0.000& 0.500& 0.250& 0.000  \\
W       &-&L        & 0.500& 0.250& 0.000& 0.250& 0.250& 0.250  \\
L       &-&$\Gamma$ & 0.250& 0.250& 0.250& 0.000& 0.000& 0.000  \\ \hline
\end{tabular}
\end{center}
\end{table}
\begin{figure}
\begin{makeimage}
\end{makeimage}
\begin{center}
\includegraphics{diamondbs}
\end{center}
\caption{\label{walk-pic}Band structure for Diamond}
\label{pbt}
\end{figure}
\index{Little groups|)}
\index{Diamond!little group|)}

\subsection{Surfaces in $k$-space}
\index{ksp@{$k$-space}!surfaces in}\index{Surfaces in $k$-space} A surface is
defined by seven data.  Three numbers are used to specify the center of the
surface, three are used to define a perpendicular, and one number is used to
define the length of an edge.

For example, to generate the hexagonal face of the Brillouin Zone for diamond,
the specification would be (0.25 0.25 0.25 1 1 1 0.7071), the center of the
plot being (0.25 0.25 0.25), and (1 1 1) defining the direction.

Two dimensional surfaces use 80 points in each direction, and an interpolation
procedure is used to double this number to 160 points in each direction.  The
length of a side in a two-dimensional plot is defined by the user at
run-time.

The cost of generating the surface is quite large.  Because of this, the data
are stored on disk, once they are generated.

The user is prompted for input.

\subsection{Files used by BZ}
\index{Files used by BZ}\index{BZ!files used by}
BZ uses the following files:
\begin{description}
\item[\comp{ $<$filename$>$.ops}] a symmetry file.
\item[\comp{ $<$filename$>$.brz}] a Fock matrix file made by MOPAC.
\end{description}

The file \comp{$<$filename$>$.brz} {\em must} exist.  The file
\comp{$<$filename$>$.ops}
{\em should}, but does not have to, exist.    If \comp{$<$filename$>$.ops} is
written,  the first two lines should look like this:
\begin{verbatim}
0 0 0 0 0.0       0.0       0.0       1.0  0.0 0.0 0.0  'Identity'
2 0 0 0 0.0       0.0       0.0       0.0  0.0 0.0 0.0  'Terminus'
\end{verbatim}
More lines can be added, as needed,  to define the other symmetry operations.


% !TEX encoding = UTF-8
% !TEX program = pdflatex
% !TEX spellcheck = en_US
% !TEX root = roma-demo.tex

\section{Introduction}

\subsection{Original problem}

\begin{frame}[t]{Code Example}
	\begin{lstlisting}[language=C++, caption=Multiplication example]
		byte mult(byte in_1, byte in_2){
			byte mask, result, piv;
			mask = 0x01;
			result = 0x00;
			piv = in_1;
			for(int i=0; i<8; i++){
				if(in_2 & mask) result^=piv;
				mask = mask << 1;
				piv = xtime(piv);
			}
			return result;
		}
	\end{lstlisting}
\end{frame}

\begin{frame}[t]{Assumptions}

	Let us consider the usual linear regression model defined as: \[y = \beta_*^t \mathbf{x} + \epsilon \text{ with }\]

	\begin{enumerate}[i.]
		\item $\mathbf{x} \in \mathbb{R}^p$ a vector of random variables normally called the input vector.
		\item $\epsilon \in \mathbb{R}$ the random noise defined as a Gaussian random variable with expectation zero and variance $\sigma^2$, this is, $\epsilon \sim \mathcal{N}(\mu=0,\,\sigma^{2})$
		\item $y \in \mathbb{R}$ a random variable that depends linearly on $\mathbf{x}$.
		\item $\beta_* \in \mathbb{R}^p$ is the optimal model.
	\end{enumerate}

\end{frame}

\begin{frame}[t]{Problem Formulation}

	The problems we need to solve to estimate $\beta_*$ are written as follow:

	\begin{mini!}|s|[2]<b>
		{\beta, e}{ f(e)+ \lambda g(\beta)}{}{P(s, \lambda) \quad}
		\addConstraint{e}{= \mathbf{y} - X \beta \quad \label{const:c1ex1}}{e \in \mathbb{R}^m, \beta \in \mathbb{R}^p }
		\addConstraint{h(\beta)}{\leq s}{ s \in \mathbb{R}, s \geq 0}
		\addConstraint{\mathbf{L}\leq }{A\beta \leq \mathbf{U}}{\mathbf{L,U} \in \mathbb{R}^m, A \in \mathbb{R}^{q x p}}
	\end{mini!}

\end{frame}

\begin{frame}[t]{Examples}
	\begin{itemize}

	\item $f$ is the error function. Examples: $f(\mathbf{e}) = {\| \mathbf{e} \|}_1$ y $f(\mathbf{e}) =\frac{1}{2} {\| \mathbf{e} \|}^2_2$, among others.

	\item $h$ and $g$ are the complexity functions of the model. Examples: $g(\mathbf{\beta}) = {\| \mathbf{\beta} \|}_1$ y $g(\mathbf{\beta}) = {\| \mathbf{\beta} \|}^2_2$, $h(\mathbf{\beta}) = {\| \mathbf{\beta} \|}_1$ y $h(\beta) = {\| \beta \|}_0 = \vert \lbrace j: \beta_j \neq 0,\;\;j \in [n] \rbrace \vert$, among  others.

	\item  $A$,$\mathbf{L}$ and $\mathbf{U}$ allow the modelling of linear constraints over the regressors.

	\end{itemize}
\end{frame}

\subsection{The Holistic Regression Problem}

\begin{frame}[t]{The Holistic Regression problem}
\[(R(\lambda,s))\;\;\underset{\beta,\mathbf{e},\mathbf{z}}{min}\;\;f(\mathbf{e}) + \lambda g(\beta)\;\;s.t.\]
\[\mathbf{y} - X \beta = \mathbf{e}\]
\[h(\beta) \leq s\]
\[\mathbf{L} \leq A\beta \leq \mathbf{U}\]
\[(\beta,\mathbf{z}) \in H\]
\[\beta \in \mathbb{R}^n,\;\;\mathbf{e} \in \mathbb{R}^m,\;\;\mathbf{z} \in {\lbrace 0,1 \rbrace}^n\]

	\begin{itemize}
		\item The structure of $R(\lambda,s)$ is too general, so the algorithms designed for $P(s, \lambda)$ cannot be used (Specially because $H$).
		\item For the usual optins of $f$,$g$,$h$ and for $H$ defined as affine equations and inequations in $(\beta,\mathbf{z})$ $R(\lambda,s)$ is as (0-1-MICQP).

\end{itemize}
\end{frame}

\begin{frame}[t]{Huber Function $\rho_\gamma(s)$}
	\begin{center}
		$\rho_\gamma(s) =
		\begin{cases}
		\frac{1}{2} s^2  & 0 \leq  s  \leq \gamma\\
		\gamma  s - \frac{1}{2} \gamma^2  &  s \geq \gamma
		\end{cases}
		$

		\vspace{0.3cm}

		\resizebox{0.58\textwidth}{!}{
			\begin{tikzpicture}
			\begin{axis}[
			xlabel=$s$,
			ylabel={$\rho_\gamma(s)$},
			legend pos=north west
			]
			% p_2(x)
			\addplot[green, ultra thick, domain=0:3]{0.5*x^2};
			\addlegendentry{$\gamma=3$}
			\addplot[blue, ultra thick, domain=0:2]{0.5*x^2};
			\addlegendentry{$\gamma=2$}
			\addplot[red, ultra thick, domain=0:0.5]{0.5*x^2};
			\addlegendentry{$\gamma=0.5$}

			\addplot[green, ultra thick, domain=3:6]{3*abs(x)-0.5*(3)^2};

			\addplot[blue, ultra thick, domain=2:6]{2*abs(x)-0.5*(2)^2};

			\addplot[red, ultra thick, domain=0.5:6]{0.5*abs(x)-0.5*(0.5)^2};

			\end{axis}
			\end{tikzpicture}
		}
	\end{center}
\end{frame}

\begin{frame}[t]{The $\epsilon$-insensitive Huber Function $g_\gamma^\epsilon(t)$}
	\begin{center}

		$f^{\epsilon}_\gamma(\mathbf{e}) = \underset{i \in [m]}{\sum} g^{\epsilon}_\gamma(\bar{\mathbf{e}}_i)$ with: \\

		$g^{\epsilon}_\gamma(t) = \left\{\begin{array}{lcc} 0 & si & \vert t \vert \leq \epsilon\\\rho_\gamma(\vert t \vert - \epsilon)   & si & \vert t \vert \geq \epsilon\\
		\end{array}
		\right.$
	\end{center}

\end{frame}

\section{The study case}

\begin{frame}[t]{Study case}

	\[(R(\lambda,k))\;\;\underset{\beta,\mathbf{e},\mathbf{z}}{min}\;\;f^\epsilon_\gamma (\mathbf{e}) + \lambda {\parallel \beta \parallel}_1\;\;s.t.\;\;(1)\]
	\[\mathbf{y} - X \beta = \mathbf{e}\;\;(2)\]
	\[\mbox{If}\;\;\mathbf{z}_j = 1\;\;\mbox{then}\;\;\beta_j = 0\;\;\forall j \in [n]\;\;(3)\]
	\[Co(\mathbf{z})\leq k\;\;(4)\]
	\[\underset{j \in J1_i}{\sum} \mathbf{z}_j \leq 1\;\;\forall i \in [n_1]\;\;(5)\]
	\[\underset{j \in J2_i}{\sum} \mathbf{z}_j = 1\;\;\forall i \in [n_2]\;\;(6)\]
	\[\mathbf{z}_{j_1} = \mathbf{z}_{j_2}\;\;\forall j_1,j_2 \in B_i\;\;\forall i \in [n_B]\;\;(7)\]
	\[\beta_j \geq 0\;\;\forall j \in J^+,\;\;\beta_j \leq 0\;\;\forall j \in J^-\;\;(8)\]

\end{frame}

\begin{frame}[t]{Study case (II)}

	\[(R(\lambda,k))\;\;\underset{\beta,\mathbf{e},\mathbf{z}}{min}\;\;f^\epsilon_\gamma (\mathbf{e}) + \lambda {\parallel \beta \parallel}_1\;\;s.t.\;\;(1)\]
	\[ \cdots \]
	\[\mathbf{z}_{j_1} + \mathbf{z}_{j_2} \leq 1\;\;\forall (j_1,j_2) \in Jc\;\;(9)\]
\[\beta \in \mathbb{R}^n,\;\;\mathbf{e} \in \mathbb{R}^m,\;\;\mathbf{z} \in {\lbrace 0,1 \rbrace}^n\;\;(10)\]

\end{frame}

\section{Roadmap}

\begin{frame}[t]{Road}
\textbf{Objective:} Find a set of solutions with values close to the optimal, this is {\it a path of Approximate Solutions}

\begin{enumerate}
	\item Formulation of $ R(\lambda, k)$.
	\item Obtaining valid values for BigM values.
		BigMs too big $\implies$ numerical problems, inefficiency of algorithms.
		BigMs too small $ ~ implies $ good solutions removed.
		\item {\it Local Holistic Searches}: Distances, Neighborhoods and Complexity function are presented on $\mathbf{z}$, which we will call Holistic, that adapt naturally to the case study, to obtain solutions locally optimal.
		\item Algorithms for the construction of the Approximate Solutions 	Path combining (2.) and (3.)
\end{enumerate}
\end{frame}