\chapter{Image Segmentation}
\label{ch:image_segmentation}
Initially, the notion of segmentation stands for the process of classifying pixels into groups which corresponds to the same type or class \cite{Margaritondo2011}. The segmentation allows to obtain more meaningful and potentially hidden information from original images, what let to bring indeed numerous insights for various business problems.

For instance, in medicine analysis segmentation invokes various uses cases such as to measure volume of an organ, to render a 3D view of an organ, surface-based registration and many others. Before moving forward I will explicitly distinguish segmentation types from general point of view.

\begin{figure}[h]
    \centering
    \includegraphics[width=10cm]{images/semantic_instance_segmenattion.png}
    \caption{An example of semantic and instance segmentation.}
    \label{fig:image_segmentation}
\end{figure}

Moving from the left to the right side on the Figure \ref{fig:image_segmentation}, we in deed can transparently notice 2 main types of segmentation. From the very begging there is a original 3D CT image, after applying particular segmentation method, on the second image we see the resulting image where every pixel belongs to a certain class (either blue: cervical, green: thoracic, red: lumbar). Pixels which belongs the same class are represented by the same color. Such type of segmentation is called "semantic segmentation".

The very last image has assigned a particular class to each pixel of the image as well. However, different objects of the same class have different colors. Such type of segmentation is called "instance segmentation".

\section{Classical Segmentation Methods}
Now we are good to go with classical segmentation methods section. I will cover basic thresholding method, then modified version of thresholding so called region growing. Afterwards I will speak about edge detection segmentation and clustering based segmentation. On top of that I will demonstrate the performance of each dedicated technique within the following baseline Figure \ref{fig:sample_vertebrae} of vertebrae.

\begin{figure}[h]
    \centering
    \includegraphics[width=2cm]{images/sample_vertebrae.jpeg}
    \caption{Sample CT scan of vertebrae.}
    \label{fig:sample_vertebrae}
\end{figure}

\subsection{Region Based Segmentation}
Sometimes region based segmentation method is mentioned as thresholding. Actually, it one of the simplest methods for classifying pixels based on solely on their intensity values \cite{Butchiraju2019}. In thresholding we have options to choose either upper and lower bound values or to choose just one threshold value which can be for instance the mean of the original image values.

Mathematically, the algorithm can be defined as:
\[
    f(x, y)=
\begin{cases}
    0 & \text{, if } f(x, y) > T \\
    255 & \text{, otherwise}
\end{cases}
,\]
where $T$ is threshold value, $x$ and $y$ certain coordinates of pixel.

After applying proposed algorithm with the mean threshold on the baseline Figure \ref{fig:sample_vertebrae}, I had obtained following segmented image depicted on Figure \ref{fig:sample_vertebrae_thresholding}.

\begin{figure}[h]
    \centering
    \includegraphics[width=2cm]{images/sample_vertebrae_thresholding.png}
    \caption{Applied thresholding algorithm for sample CT scan of vertebrae.}
    \label{fig:sample_vertebrae_thresholding}
\end{figure}

The problem within simple thresholding is that it does not take into consideration any spatial information of an image, meaning each pixel is evaluated no matter where it is located. Beyond, each time we have to find best way to define either range of values or value for threshold.

\subsection{Region Growing Segmentation}
Unlike basic thresholding, region growing segmentation indeed takes into consideration spatial information starting with the small set of seed pixels and then growing out from them \cite{Khaloo2017}.

Let me assume following example. There is set of pixels $4 \cdot 4$ which should be colored (segmented). The threshold range is between $10$ and $100$. Some of the cells are already filled with red color, another cells are potential candidates to be segmented. The task is to fill (segment) the "non red" cells with a red color based on predefined thresholding. Mathematically region growing can be defined as following equitation:
\[
    f(x, y) =
\begin{cases}
    \text{segment as red} & \text{, if } 10 \leq \text{intensity} \leq 100 \\
    \text{keep same} & \text{, otherwise}
\end{cases}
,\]
where intensity is the pixel value with $x$ and $y$ coordinates.

All it all, region growing is an iterative method used to extract similar parts of an image. One or several points are chosen as a start. The region then grows until it is finally blocked by the stop criteria. This criteria is generally an inside/outside region comparison such as size or others.

Exactly the same manner growing region thresholding method can be applied for the previously chosen baseline Figure \ref{fig:sample_vertebrae}. The resulting segmentation shown on Figure \ref{fig:sample_vertebrae_grwoing}.

\begin{figure}[h]
    \centering \includegraphics[width=2cm]{images/sample_vertebrae_regiongrowing.png}
    \caption {Applied region growing algorithm for sample CT scan of vertebrae.}
    \label{fig:sample_vertebrae_grwoing}
\end{figure}

There is a huge potential room for improvements making the method more sophisticated by coming up with different conditions for inclusion or exclusion of pixels.

\subsection{Edge Detection Segmentation}
The detection of edges approach outputs meaningful semantic information that facilitate the understanding of an image. It is often used as auxiliary function or filter to perform segmentation. Edge detection can be used to directly segment edges and regions of images but it is particularly useful when we want to use that as part of another segmentation algorithm.

As Muthukrishnan \cite{Muthukrishnan2011} described, there are four possible sources of edges in an image: surface normal discontinuity (surface changes direction sharply), depth discontinuity (one surface behind another), surface color discontinuity (single surface changes color), illumination discontinuity (shadows or lighting).

In order to perform edge detection usually it is applied high-pass filters such as Laplacian, Sobel, Prewitt or Canny whereas the corresponding output is so called filter mask which shows the transition values. After filtering it is usually applied thresholding to retrieve edges.

\subsubsection{Edge Basics}
Edges occur in images when the magnitude of the gradient is high. In order to find the gradient, we firstly have to find the derivatives in both the $x$ and $y$ directions. These actions can be mathematically considered as:
\[ \frac{\partial f (x)}{\partial x} = \lim_{\delta{f} \to 0} \dfrac{f(x) - f(x - \delta x)}{\delta x}  = f'(x); \]
\[ \frac{\partial f (x)}{\partial x} = \dfrac{f(x) - f(x - 1)}{1}  = f'(x); \]
\[ \frac{\partial f (x)}{\partial x} = f(x) - f(x - 1) = f'(x); \]

Also it is possible to take the derivative in three different ways and represent them as filter (convoluting the filter with the image gives the derivative) as:
\begin{align*}
Backward: f'(x) = f(x) - f(x - 1) \to [0, 1, -1]; \\
Forward:  f'(x) = f(x + 1) - f(x) \to [-1, 1, 0]; \\
Central:  f'(x) = \dfrac{(x + 1) - f(x - 1)}{2} \to [1, 0, -1];
\end{align*}

The gradient ($\nabla f$), gradient magnitude ($|\nabla f(x, y)|$) and gradient angle ($\Theta$) can be accordingly calculated as:
\begin{align*}
\nabla f(x, y) = \left[ \frac{\partial{f (x, y)}}{\partial{x}}, \frac{\partial{f (x, y)}}{\partial{y}} \right]  = [f_x, f_y]; \\
|\nabla f(x, y)| = \sqrt{f^2_x + f^2_y}; \\
\Theta = \tan^{-1} \frac{f_y}{f_x};
\end{align*}

\subsubsection{Convolution}
Convolution itself is a mathematical operation on two functions ($f$ and $g$) that produces a third function ($f \cdot g$) that expresses how the shape of one is modified by the other \cite{Pang2018}. Considering, mathematical notion of convolutional operation is:
\[(f \ast g)(t) := \int_{-\infty}^{+\infty} f(\tau)g(t - \tau)\partial\tau \]

All it all, convolution operation can be described as: while a filter is sliding or convolving around an input image, it has been multiplying the values in the filter with the original pixel values of the image (computing element wise multiplications).

To show convolutional mathematical background I will use Figure \ref{fig:sample_1_channel_image}.
\begin{figure}[h]
    \centering \includegraphics[width=4cm]{images/1_channel.jpg}
    \caption {Sample 1 channel image.}
    \label {fig:sample_1_channel_image}
\end{figure}

Suppose we have 1 channel (gray scale) Figure \ref{fig:sample_1_channel_image} image with size of $5 \cdot 5$. Also there is a filter (kernel) which I had defined as:
\[ \begin{pmatrix} 1 & 0 & -1 \\ 0 & 1 & 0 \\ -1 & 0 & 1 \end{pmatrix} \]

Now I need to calculate the convolution with filter (kernel) which I had defined above. The size of filter is $3 \cdot 3$, padding is $1 \cdot 1$ and stride $1$.

To be aware in the correctness of the result, I would like to know the output size of convolution operation. In mathematical terms it is calculated as: \[ \text{size of output volume} = \frac{W-F+2 \cdot P}{S+1} \] where $W$ - is the input volume size, $F$ - is the receptive field size, $S$ - the stride with which they are applied and $P$ - is the padding.

\begin{figure}[h]
    \centering \includegraphics[width=10cm]{images/convolution_operation.jpeg}
    \caption {Example of convolution operation.}
    \label{fig:convolution}
\end{figure}

On the Figure \ref{fig:convolution} I partially depicted step-wise convolution operation. Mathematically, red colored cell on the resulting convolution \ref{fig:convolution} can be calculated as:
\[ \begin{pmatrix} 0 & 0 & 0 \\ 0 & 5 & 2 \\ 0 & 3 & 1 \end{pmatrix} \odot \begin{pmatrix} 1 & 0 & -1 \\ 0 & 1 & 0 \\ -1 & 0 & 1 \end{pmatrix} = \]
\[ = 0 \cdot 1 + 0 \cdot 0 + 0 \cdot (-1) + 0 \cdot 0 + 5 \cdot 1 + 2 \cdot 0 + 0 \cdot (-1) + 3 \cdot 0 + 1 \cdot 1 = \]
\[ = 0 + 0 + 0 + 0 + 5 + 0 + 0 + 0 + 1= 6 \]

\subsubsection{Sobel Filter}
Now I can emphasize that each filter utilizes various functionalities described above. The Sobel filter performs a 2-D spatial gradient measurement on an image and so emphasizes regions of high spatial frequency that correspond to edges. Typically it is used to find the approximate absolute gradient magnitude at each point in an input gray scale image. This is how Sobel filter performs on baseline Figure \ref{fig:sample_vertebrae}.

\begin{figure}[h]
\centering
\includegraphics[width=2cm]{images/sample_vertebrae_sobel.jpeg}
    \caption {Applied Sobel filter for sample CT scan of vertebrae.}
    \label{fig:sobel}
\end{figure}

In general, each edge detector can be either separate convolution operation within special filter and threshold or combination such as: suppressing noise, computing gradient magnitude and direction, applying non-maximum suppression, applying hysteresis thresholding and finally utilizing connectivity analysis to detect edges (Canny Edge Detector).

By the reason of edge detection is based on derivatives it can easily degrade with variations and noise. But issues which should be considered are size of the neighborhood as well as what exactly represents a transition on the image.

\subsection{Segmentation Based on Clustering}
Another was to approach segmentation is clustering. The obvious question I can ask myself whether I can use clustering techniques to divide images into segments. And the answer is - yes, I can! All it all, the aim is to group objects (instances) into so-called clusters, such that objects in the same cluster are (or, at least, should be) more similar to each other than to the objects belonging to other clusters \cite{Dhanachandra2017}.

Within clustering there are multiple algorithms to be issued. Few of them are
Dynamic Time Warping, Hierarchical Agglomerate Clustering and k-Means \cite{Dhanachandra2017}. Beyond it should be intuition behind what is similarity in terms of clustering, but I will not dig around this part, instead focus on most popular and in the public eye algorithm so called k-Means.

k-Means algorithm is a common and simple clustering method whereas usually the data set is represented as the a scatter plot in some feature space.

The Figure \ref{fig:triangle} and Figure \ref{fig:clustering} helps to understand how indeed images data can be represented in some feature space. On the Figure  \ref{fig:clustering} depicted the representation of colored triangle in 3D space, where each dimension represents each color.

\begin{figure}[h]
    \centering
    \begin{minipage}[b]{0.4\textwidth}
    \includegraphics[width=\textwidth]{images/k_mean_triangle.png}
    \caption{Colorful triangle.}
    \label{fig:triangle}
    \end{minipage}
    \hfill
    \begin{minipage}[b]{0.4\textwidth}
    \includegraphics[width=\textwidth]{images/k_mean_triangle_clustered.jpg}
    \caption{Feature space representation.}
    \label{fig:clustering}
    \end{minipage}
\end{figure}

Considering k-Means mathematically, given $D \subseteq D_1 \times D_2 \times ... \times D_m$, a distance measure $d$ (or similarity measure $s$) and the number of clusters $k$, where $k \ll n$. The goal is to find cluster centers $c_1, c_2, ..., c_k$ and a mapping $p: D \rightarrow {1, 2, .., k} $ such that \[ \sum_{i=1}^{n} d(x_i, c_p_{x_i}) - \text{is minimal}\]

The algorithm itself can be assumed as:
\begin{itemize}
    \item Initialize $c_1, c_2, ..., c_k$ such that for all $i = {1, 2, .., k}$
    \subitem $c_i \in D$ (random initialization), or
    \subitem or $c_i = \frac{\sum_{x, p(x) = x^X}}{\sum_{x, p(x) = x^1}}$
    \item compute $p$ such that
    \subitem $\sum_{i=1}^n d(x_i, c_p_{(x_i)})$ is minimal
    \item update $c_i$ for all $i = {1,2, ..., k}$ where changed then goto step 2
    \item return $p$ and $c_1, c_2, ..., c_k$
\end{itemize}

To demonstrate the method performance I will proceed with the same baseline CT spine Figure \ref{fig:sample_vertebrae}. As for regressors number (number of clusters) I had chosen $k=3$.

\begin{figure}[h]
    \centering \includegraphics[width=2.5cm]{images/sample_vertebrae_kmeans.png}
    \caption {Applied k-Means algorithm for sample CT scan of vertebrae.}
    \label{fig:k_menas}
\end{figure}

Eventually, as it can be noticed on Figure \ref{fig:k_menas}, clustering is a really "good to go" approach for image segmentation, but there are various disadvantages such as tackling with number of clusters, being dependent on initial values, clustering outliers and scaling with number of dimensions.

\section{Modern Segmentation Methods}
At present, I have wrapped up classical-wise methods for image segmentation. There are numerous amount of another techniques as well, but now I would like to narrate about recent techniques which live under umbrella of deep learning family. Before exploring the deep learning methods I have to relate basic mathematical operations, notions and general intuition for neural networks.

\subsection{Deep Learning Basics}
For simplicity I will represent neuron's mathematical model graphically on Figure \ref{fig:neuron}. In the mathematical model of the neuron, the body of the neuron, where the input signals accumulates, is denoted by a summarizing neuron. In addition, the biological neuron also has axons and dendrites, which get the inputs and send the outputs.

\begin{figure}[h]
    \centering \includegraphics[width=10cm]{images/neuron_math_model.jpeg}
    \caption {Mathematical model of neuron.}
    \label{fig:neuron}
\end{figure}

Accordingly, in our mathematical model of the neuron, we will add inputs and outputs to the summarizing neuron. It is also worth to keep in mind the biological neuron applies some actions with the signals that come into it, namely, it accumulates charge until it reaches some threshold and only after that forwards it further. This is what we will do in the mathematical model of the neuron using the activation functions. Mathematical definition of neuron model is written as:
\begin{align*}
y = f(z) = f(w_0 \cdot x_0+w_1 \cdot x_1+w_2 \cdot x_2+b) = f(\sum\limits_{i=0}^{N-1} w_i \cdot x_i+b) = f(\langle w, x \rangle + b),
\end{align*}
where: $x_0, x_1, x_2$ - inputs, $w_0, w_1, w_2$ - weights,  $b$ - some bias for increasing non-linearity, $f(z)$ - some activation function.

Inputs can be either single number or vector of numbers. As for weights, we mainly should remember they are tuned parameters. Moreover it should be considered 2 potential situations to manage with.

The first one is initialization of the weights. Here we have multiple options whereas we literally can define them either randomly or apply special initialisation techniques such as 'He' initialization or 'Xavier' initialization and others.

The second situation we should be aware of is to somehow change the weights during model fitting, meaning weights should be changed in iterative manner from epoch to epoch. To approach it we will apply backpropagation algorithm \cite{Benvenuto1992}.

Concerning the bias, which is like a weights, meaning tuned parameter, we can consider it allows to shift the activation function by adding a constant to the input of neuron. Hence, the last but not least component is activation function. In simplified terms, activation function is used to determine the output of neuron in a manner: 'yes' or 'no'. It maps the resulting values in range between 0 to 1 or -1 to 1.

\subsubsection{Activation Function}
There is a significant number of various activation functions. Assuming the very basic one named "step function" and figure out from general point of view how it works in both mathematical and geometrical sense.
The "step function" is defined as:
\begin{align*}
f(x) = \begin{cases} 0, & \mbox{if } x\mbox{ $\leq$ 0} \\ 1, & \mbox{if } x\mbox{ $>$ 0} \end{cases}
\end{align*}

The function has 2 strongly distinguished values: $0$ and $1$. The place where the function changes it's value from $0$ to $1$ is named as dividing surface \cite{Apicella2021}. Hence, the dividing surface place is where the argument of activation function is equal $0$.

\begin{figure}[h]
    \centering \includegraphics[width=7cm]{images/step_function.jpeg}
    \caption {Geometrical representation of step function.}
\end{figure}

The geometrical representation of step function can be derived as following. There is the neuron formula:
\begin{align*}
y = f(\langle w, x \rangle + b)
\end{align*}

The neuron does some linear operation, which is denoted by 2 parameters: $w$ (vector of weights) and $b$ (bias).
Accordingly, the dividing surface is defined by following equitation which denotes equitation of straight line:
\[\langle w, x \rangle + b = 0\]

On one hand the value of step function is equal $1$ and on the other hand $0$. The activation function is equal $1$ that side of dividing surface where the vector $w$ points out. From geometrical point of view it is represented on Figure \ref{fig:dividing_surface}.

\begin{figure}[h]
    \centering \includegraphics[width=6cm]{images/dividing_surface.jpeg}
    \caption {Sample geometrical representation of dividing surface for single neuron.}
    \label{fig:dividing_surface}
\end{figure}

As it was mentioned earlier, there are many more functions. For instance sigmoid activation function showed on Figure \ref{fig:sigmoid} which unlike step function does not have discontinuity point at zero.

\begin{figure}[h]
    \centering \includegraphics[width=10cm]{images/sigmoid_function.jpeg}
    \caption {Sigmoid activation function.}
    \label{fig:sigmoid}
\end{figure}

The sigmoid activation function formed as:
\begin{align*}
\sigma(x) = \dfrac{1}{1+e^x}
\end{align*}

\[ \sigma(x) = \begin{cases} 1, & \mbox{if } x\mbox{$\xrightarrow{} + \infty$} \\ 0, & \mbox{if } x\mbox{$\xrightarrow{} - \infty$} \end{cases} \]

One more activation function is rectified linear activation function \cite{Eckle2019}. For short it is a piece-wise linear function which outputs input directly if it is positive otherwise outputs zero. It has become the common-wise activation function for many types of neural networks.

\subsubsection{From Neuron to Neural Network}
So far I have covered the properties and functional of just a single neuron, which as a result outcomes a linear divide surface as shown on Figure \ref{fig:dividing_surface}. Unlike, multiple neurons as shown on Figure \ref{fig:dividing_surface_2}, meaning when we concatenate the multiple neurons in some architecture we may achieve some nonlinear divide surfaces.

\begin{figure}[h]
    \begin{center}
        \includegraphics[width=10cm]{images/neuron_to_neural_net.jpeg}
        \caption {Sample geometrical representation of dividing surface for single neuron versus multiple neurons.}
        \label{fig:dividing_surface_2}
    \end{center}
\end{figure}

Now the question is how do we get the nonlinear dividing surface. Simply put, define 3 neurons with linear activation function as shown on Figure \ref{fig:1_layer_net}.

\begin{figure}[h]
    \centering \includegraphics[width=6cm]{images/3_neurons_net.jpeg}
    \caption {Sample 1 layer neural net with linear activation functions.}
    \label{fig:1_layer_net}
\end{figure}

Mathematically, results of performing the net can be considered as:
\[ y_3 = f(w_2^3 \cdot y_2+w_1^3 \cdot y_1+b^3) = \]
By the reason $f$ is simple linear function, equation can be derived as:
\[ = w_2^3 \cdot y_2+w_1^3 \cdot y_1+b^3 = \]
\[ = w_2^3 \cdot f(w_2^1\cdot x_1+w_2^2 \cdot x_2+b^2) + w_1^3 \cdot f(w_1^1 \cdot x_1+w_1^2 \cdot x_2+b^1) + b^3 = \]
\[ = w_2^3 \cdot w_2^1 \cdot x_1+w_2^3 \cdot w_2^2 \cdot x_2+b^2 + w_1^3 \cdot w_1^1 \cdot x_1+w_1^3 \cdot w_1^2 \cdot x_2+b^1+b^3 = \]
\[ = x_1[w_2^3 \cdot w_1^2 + w_1^3 \cdot w_1^1] + x_2[w_2^3 \cdot w_2^2 + w_1^3 \cdot w_2^1] + [w_2^3 \cdot b^2+w_1^3 \cdot b^1+b^3] = \]
\[ =  x_1\cdot \tilde{w_1} + x_2\cdot \tilde{w_2} + \tilde{b} \]
Meaning the obtained equitation is just linear combination of inputs into the neural net.

\subsubsection{Loss function}
Characteristically for neural networks, we keen to minimize the error of net predictions. There are many ways (functions) that could be potentially used to estimate an error of a neural network. Usually it comes to make use of so called loss functions or cost functions \cite{Janocha2016}. Below I considered few popular losses.

Mean Squared Error (MSE) is the average of the squared error that is used as the loss function for least squares regression. To streamline, MSE is the sum over all the data points of the square of the difference between the predicted and actual target variables divided by the number of data points. Mathematically, it is defined as:
\begin{align*}
\text{MSE} = \frac{{1}}{n} \sum_{i=1}^{n} (y_i - \hat{y_i})^2,
\end{align*}
where $y$ is the ground truth value, and $\hat{y_i}$ is neural network prediction.

Frequently, loss functions are utilized in combination with regularization terms \cite{Chen2019}. The purpose of regularization terms is establish additional rules for penalizing the loss function for errors. Meaning the regularization will manage whether the weights are important (good) or not. One of such regularization techniques is $l2$ regularization or so called weight decay. The definition is as:
\[ \lambda \cdot \sum_{i=1}^{n} \alpha_i^2 \]

In terms of mean squared error loss function, the addition of regularization considered as:
\[ \sum_{i=1}^n(\hat{y}-y_i)^2 + \lambda \cdot \sum_{i=1}^{n} \alpha_i^2\]

One more commonly used function is so called cross-entropy loss or log loss. It measures the performance of a model whose output is a probability value between 0 and 1. Cross-entropy loss increases as the predicted probability diverges from the actual label \cite{Boudiaf2020}. For discrete probability distributions $p$ and $q$ with the same support $X$ over a given set it is defined as follows:
\[H(p,q) = - \sum_{x \in X} p(x) \log q(x), \]
where $p$ and $q$ are discrete probability distributions.

\subsubsection{Gradient Descent}
Optimization of neural net is the key component in terms of training any model. The optimization of the net should be considered as the on-the-spot training. There are variety of optimization algorithms but the pillar algorithm in the list is gradient descent \cite{Zou2020}.

Suppose a function which is defined by some level lines as shown on Figure \ref{fig:gradient}. The function has 2 minimums. It worth to understand by distancing from the minimums the function value increases.
So far, we have a neural net with some parameters (weights and biases) which outputs some value of loss function (error). As we remember the task is to decrease the value of loss function. Hence, the question is how to achieve that?

\begin{figure}[h]
    \centering \includegraphics[width=8cm]{images/gradient_descent.jpeg}
    \caption {Sample gradient descent visualization.}
    \label{fig:gradient}
\end{figure}

Well, the very first provision is as follows. The vector of function surface will be defined by $w_0$, where $w_0$ denotes vector of all weights and biases of the net. It will be represented as:
\[ w_0 = [w_1^1, w_1^2,...,w_1^n, w_2^1,...,w_2^n,...,w_n^n, b1^1, b1^2,...,b1^n, b2^1,...,b2^n,...,b_n^n] \]

Accordingly, we need to take a derivative of loss function (meaning gradient, which is vector, which consists of derivatives per each coordinate of the function). It is performed as:
\[ \delta{f} = \left[ \frac{\partial{f}}{\partial{w_0}}, \frac{\partial{f}}{\partial{w_1}}, ... ,\frac{\partial{f}}{\partial{w_n}} \right] \]
As the result, we have calculated the gradient of loss function at the point ($w_0$) where we currently located. The gradient of loss function points out at the direction of greater loss function growth. But, oppositely, we on demand of lower loss function growth, meaning we need to make an opposite step against calculated gradient descent. It can be considered as:
\begin{align*}
w_1 = w_0 - \alpha \cdot \delta{f(w_0)},
\end{align*}
where $w_0$ - the initial vector of weights and biases of neural net, $\alpha$ - parameter for regulating the speed of training of neural net.

These steps could be performed calculating gradient of $w_n$ vector until it would not converge or reach the stop criteria as:
\begin{align*}
w_n = w_{n-1} - \alpha \cdot \delta{f(w_{n-1})}
\end{align*}

As well, there are different variations of gradient descent within their scopes, pros and cons. Some of them are stochastic gradient descent, batch gradient descent and mini-batch gradient descent.

\subsection{Convolutional neural networks (CNN)}
So far I have wrapped up the very basics behind neural networks fundamentals. Within these basics now I can introduce convolutional neural networks. The cons of such kind of neural networks are that they can perfectly find the latent consistencies within structured data such as images or videos. Using CNNs we can solve various computer vision tasks such as object detection, object localisation, image segmentation or even more complex as face verification and recognition or neural style transfer \cite{Jing2020}. As Baicen Xiao concluded \cite{Hosseini2017}, CNNs have
achieved state-of-the-art performance on a variety of computer vision tasks, particularly visual classification problems, where new algorithms reported to achieve or even surpass the human performance.

To present simple distinguish example, suppose, there is an RGB image sized $224 \cdot 224$ where a dog is located in the middle of image. The goal is to classify whether the dog is on the image. The usual (for instance sigmoid feed forward) neural network will come up with the mask of the dog based on the image. But once new image will be revealed and the dog will not be located in the middle of it, the net will not be able to find and classify the dog. Whilst convolutional neural networks could be generalized independently of the location of the dog, because the convolution itself is invariant operation.

There are few more essential properties within CNNs such as pooling, dropout and others, but I'm aim to focus on following architectures and their performance.

\subsection{Unet}
The U-net is convolutional network architecture for fast and precise segmentation of images. It was proposed by Olaf Ronneberger, Philipp Fischer, and Thomas Brox at 2015 \cite{Ronneberger2015}.

\begin{figure}[h]
    \centering \includegraphics[width=10cm]{images/unet.png}
    \caption {Unet architecture.}
    \label{fig:unet}
\end{figure}

Using original terminology of the authors \cite{Ronneberger2015} the net consist of a contracting path (left side) and an expansive path (right side) as shown on Figure \ref{fig:unet}. The contracting path follows the typical architecture of a convolutional network. In simplified terms it can be considered as downsampling or encoder which formed as
a five repeats of double $3 \cdot 3$ convolutions (unpadded convolutions) with rectified linear unit and $2 \cdot 2$ max pooling operation with stride 2. Thus expansive path which can be considered as upsampling or decoder consists of 4 repeats of double $2 \cdot 2$ convolutions (up-convolution) which halve number of feature channels. In total the network has 23 convolutional layers.

U-net proved itself multiple times and still is one of the main backbone architectures for medical image analysis. As author \cite{Ahmed2020} observed in his work "Comparison results of segmentation models using top view person data set", U-net still remains one of the best ones. Within Table \ref{tab:classical_versus_modern} it is shown the comparison across classical and modern methods from the original U-net paper \cite{Ronneberger2015}.

\begin{table}[h]
\centering
\begin{tabular}{|l|c|c|c|c|c|}
\hline
Method & Precision & Recall & F1-score & Pixel Accuracy \\
\hline \hline
Otsu & 50\% & 80\% & 70\% & 78\%  \\
\hline
Watershred & 52\% & 82\% & 74\% & 80\% \\
\hline
Gaussian mixture-based model & 60\% & 82\% & 76\% & 82\%  \\
\hline
Background subtraction-based model & 65\% & 84\% & 74\% & 84\% \\
\hline
RE-weighted HOG & 68\% & 82\% & 75\% & 82\%  \\
\hline
FCN & 62\% & 92\% & 76\% & 91\%  \\
\hline
U-net & 74\% & 92\% & 81\% & 92\% \\
\hline
DeepLabV3 & 80\% & 96\% & 83\% & 93\% \\
\hline
\end{tabular}
\caption{Table of comparison "Classical versus Modern segmentation methods"}
\label{tab:classical_versus_modern}
\end{table}

\subsection{DoubleU-Net}
The doubleU-net is enhancement wrapper over U-net, which make the architecture less flexible but more scope-wise related. It consists of 2 U-nets, pretrained (encoder) and 2 non pretrained (decoders) accordingly as shown on Figure \ref{fig:double_unet}.

\begin{figure}[h]
    \centering \includegraphics[width=8cm]{images/DoubleU-Net.png}
    \caption {DoubleU-Net architecture.}
    \label{fig:double_unet}
\end{figure}

The encoder used in the network is pretrained VGG-19, which is trained on ImageNet. Additionally, it uses Atrous Spatial Pyramid Pooling (ASPP). The output from encoder (OUTPUT 1) is represented as binary attention mask which retrieves important and less important pixels denoted as $1$ and $0$ accordingly. The output from decoder (OUTPUT 2) is represented as binary attention mask as well but the results are obtained in a different way. The final output architecture output is a concatenation of OUTPUT 1 and OUTPUT 2.

Looking more closely on Figure \ref{fig:double_unet}, first network (NETWORK 1) is the use of VGG-19 marked in yellow, ASPP marked in blue, and decoder block marked in light green. The squeeze-and-excite block is used in the encoder of NETWORK 1 and decoder blocks of NETWORK 1 and NETWORK 2. An element-wise multiplication is performed between the output of NETWORK 1 with the input of the same network. The difference between DoubleU-Net and U-Net in the second network (NETWORK 2) is only the use of ASPP and squeeze-and-excite block. All other components remain the same.

At the original paper \cite{Jha2020} it was concluded the experimental results shows that doubleU-net achieved a Dice score of $0.7649$ and a mIoU of $0.6255$ what is better in comparison with U-net.

\section{Summary}
The applications of the classical techniques and traditional machine learning methods, such as region growing, edge detection based algorithms and clustering in medical image classification began long ago. However, each certain method has advantages and disadvantages which I had narrated above.

To sum them up, the performance is far from the practical standard and the developing of them is quite slow in recent years. Likewise, the feature extracting and selection are time-consuming and vary according to different objects. The advent of deep neural networks, especially the CNNs hugely impacted in changing image classification tasks and had achieved significant performance since 2012.

Some research on medical image classification by CNN has achieved performances rivaling human experts. It was perfectly demonstrated in the recent research work \cite{Anwar2018} that indeed the CNN based methods can achieve highly precise accuracy in various tasks such as classification and segmentation.

Additionally, as it was emphasised in one of  \href{https://www.itnonline.com/content/deep-learning-medical-imaging-create-300-million-market-2021}{\color{blue}"Image Technology News"} journal articles: ``Deep learning is a truly transformative technology and the longer-term impact on the radiology market should not be underestimated. It’s more a question of when, not if, machine learning will be routinely used in imaging diagnosis``.


\begin{landscape}
\begin{multicols}{2}
\chapter{Code Listing}

In the following section we include the code that implements the unsupervised deep learning models used.

\begingroup

\lstset{
    frame=none,
    breaklines=true,
    breakautoindent=true,
    postbreak={},
    upquote=true
}

\renewcommand{\thesubsection}{\arabic{subsection}}

\renewcommand{\addcontentsline}[3]{}% Do nothing

\subsection{Autoencoder}

\begin{lstlisting}[language=Python]
"""
Implements a simple stacked autoencoder.

Hyperparameters to tune:
------------------------
- Learning rate
- Activation function (sigmoid, ReLU, atan)
- Amount of neurons in each layer
- Learning function (GradientDescentOptimizer, RMSProp, AdamOptimizer)
- Batch size
"""
"""
Implements a simple stacked autoencoder.

Hyperparameters to tune:
------------------------
- Learning rate
- Activation function (sigmoid, ReLU, atan)
- Amount of neurons in each layer
- Learning function (GradientDescentOptimizer, RMSProp, AdamOptimizer)
- Batch size
"""
import numpy as np
import tensorflow as tf

from sys import stdout, path
from os import path as ospath

from sklearn.preprocessing import MinMaxScaler

path.append(ospath.dirname(ospath.dirname(ospath.abspath(__file__))))
import helpers

class AutoEncoder():
    """
    Implements an autoencoder that tries to learn a representation for web page traces.

    Atrributes:
        - activation_func is a tensorflow function, representing the activation function used.
            *(Often found in `tf.nn`)*
        - encoder is a computation, representing the encoder layers
        - decoder is another computtational graph, representing the decoder layers
        - loss is the operation for the mean squared error (MSE)
        - train_op is the train operation (`RMSProp`)
        - layers is a list of integerrs, determining the amount of layers and their size
        - is_training is a boolean representing whether you are training the autoencoder or not *(used in the batch_norm layer)*.
        - batch_size
        - learning_rate
    """

    def __init__(self, layers, batch_size, activation_func=tf.nn.sigmoid, saved_graph=None, sess=None, learning_rate=0.0001, batch_norm=False):
        """
        @param layers is a list of integers, determining the amount of layers and their size
            starting with the input size
        """
        if len(layers) < 2:
            print("Amount of layers must be greater than 1")
            exit(0)

        self.batch_size = batch_size
        self.learning_rate = learning_rate
        self.activation_func = activation_func
        self.batch_norm = batch_norm

        self.is_training = True

        # Use this in data preprocessing
        self.layers = layers

        self._make_graph(layers)

        if saved_graph is not None and sess is not None:
            self.import_from_file(sess, saved_graph)

    def _make_graph(self, layers):
        """
        Constructs the computational graph

        @param layers is a list of integers, determining the size of the layers
        """
        self._init_placeholders(layers[0])

        self.encoder = self._init_encoder(layers)
        self.decoder = self._init_decoder(layers)

        self._init_train()

    def _init_placeholders(self, first_layer):
        """
        The main placeholders for input and output data
        """
        self.encoder_inputs = tf.placeholder(tf.float32, [self.batch_size, first_layer])

        # We could technically use the same value as encoder_inputs but we do not
        # for future possible extensions
        self.decoder_targets = tf.placeholder(tf.float32, [self.batch_size, first_layer])

    def _get_layer(self, layer_input, size_last_layer, size_current_layer):
        """
        Returns a layer with a batch normalized input, depending on the `batch_norm flag`

        @param layer_input is the value used as an input to the layer.
        @param size_last_layer is the size of the last layer (used in weight) or the size of the input
        @param size_current_layer is the size of the current layer (used in weight and bias)
        """
        weight = tf.Variable(tf.random_normal([size_last_layer, size_current_layer]))
        bias = tf.Variable(tf.random_normal([size_current_layer]))

        if not self.batch_norm:
            return self.activation_func(tf.add(tf.matmul(layer_input, weight), bias))

        layer_input = tf.contrib.layers.batch_norm(layer_input,
                         center=True, scale=True,
                         is_training=self.is_training,
                         scope='bn{}-{}'.format(size_last_layer, size_current_layer))

        return self.activation_func(tf.add(tf.matmul(layer_input, weight), bias))

    def _init_encoder(self, layers):
        """
        Creates the layers of the decoder and returns the last layer.
        """
        previous_layer = None

        # We don't want to enumerate over the last one
        for i in range(len(layers) - 1):
            current_layer = None
            if previous_layer is None:
                current_layer = self._get_layer(self.encoder_inputs, layers[i], layers[i + 1])
            else:
                current_layer = self._get_layer(previous_layer, layers[i], layers[i + 1])

            previous_layer = current_layer

        # Will be the last layer
        return previous_layer

    def _init_decoder(self, layers):
        """
        Creates the decoder graph and returns the last layer
        """
        previous_layer = None

        # We don't want to enumerate over the last one
        for i in range(len(layers) - 1, 0, -1):
            current_layer = None
            if previous_layer is None:
                current_layer = self._get_layer(self.encoder, layers[i], layers[i - 1])
            else:
                current_layer = self._get_layer(previous_layer, layers[i], layers[i - 1])

            previous_layer = current_layer

        # Will be the last layer
        return previous_layer

    def _init_train(self):
        """
        Create the train operation
        """
        self.loss = tf.reduce_sum(tf.square(self.decoder_targets - self.decoder))

        # Which optimizer to use? `GradientDescentOptimizer`, `AdamOptimizer` or `RMSProp`?
        self.train_op = tf.train.AdamOptimizer(self.learning_rate).minimize(self.loss)

    def _init_batch_norm(self):
        """
        Adds a batch normalization layer.
        """
        self.decoder = tf.contrib.layers.batch_norm(self.decoder,
                        center=True, scale=True,
                        is_training=self.is_training,
                        scope='bn')

    def set_is_training(is_training):
        """
        Sets the `is_training` class variable, used in th batch normalization layer.
        If `batch_norm == False`, this does not make a difference but if it is true, this variable should be set to false after training.
        """
        self.is_training = is_training

    def _process_trace(self, trace, n):
        """
        Cuts the traces after `n` steps or pads them such that they are of length `n`.
        """
        features = []

        for packet in trace:
            # Either positive or negative depending on whether its incoming or outgoing.
            features.append(packet[0] * packet[1])

            if len(features) == n:
                break

        for i in range(len(features), n):
            features.append(0)

        return features

    def next_batch(self, batches, in_memory):
        """
        Returns the next batch in some fixed-length representation.
        Currently we use Panchenko et al.'s cumulative traces

        @param batches an iterator with all of the batches (
            if in_memory == True:
                in batch-major form without padding
            else:
                A list of paths to the files
        )
        @param in_memory is a boolean value

        @return if in_memory is False, returns a tuple of (dict, [paths]) where paths is a list of paths for each batch
            else it returns a dict for training
        """
        batch = next(batches)
        data_batch = batch

        if not in_memory:
            data_batch = [helpers.read_cell_file(path) for path in batch]

        data_batch = [self._process_trace(trace, self.layers[0]) for trace in data_batch]

        min_max_scaler = MinMaxScaler()
        data_batch = min_max_scaler.fit_transform(data_batch)

        encoder_inputs_ = data_batch
        decoder_targets_ = data_batch

        train_dict = {
            self.encoder_inputs: encoder_inputs_,
            self.decoder_targets: decoder_targets_,
        }

        if not in_memory:
            return (train_dict, batch)
        return train_dict

    def save(self, sess, file_name):
        """
        Save the model in a file

        @param sess is the session
        @param file_name is the file name without the extension
        """
        saver = tf.train.Saver()
        saver.save(sess, file_name)

    def import_from_file(self, sess, file_name):
        """
        Imports the graph from a file

        @param sess is the session
        @param file_name is a string that represents the file name
            without the extension
        """

        # Get the graph
        saver = tf.train.Saver()

        # Restore the variables
        saver.restore(sess, file_name)

def train_on_copy_task(sess, model, data,
                       batch_size=100,
                       max_batches=None,
                       batches_in_epoch=1000,
                       verbose=False):
    """
    Train the `AutoEncoder` on a copy task

    @param sess is a tensorflow session
    @param model is the autoencoder model
    @param data is the data (in batch-major form and not padded or a list of files (depending on `in_memory`))
    """
    batches = helpers.get_batches(data, batch_size=batch_size)

    loss_track = []

    batches_in_data = len(data) // batch_size
    if max_batches is None or batches_in_data < max_batches:
        max_batches = batches_in_data - 1

    try:
        for batch in range(max_batches):
            print("Batch {}/{}".format(batch, max_batches))
            fd, _ = model.next_batch(batches, False)
            _, l = sess.run([model.train_op, model.loss], fd)

            loss_track.append(l)

            if batch == 0 or batch % batches_in_epoch == 0:
                model.save(sess, 'autoencoder_model')
                helpers.save_object(loss_track, 'loss_track.pkl')

                if verbose:
                    stdout.write('  minibatch loss: {}\n'.format(sess.run(model.loss, fd)))
                    predict_ = sess.run(model.decoder_outputs, fd)
                    for i, (inp, pred) in enumerate(zip(fd[model.encoder_inputs].swapaxes(0, 1), predict_.swapaxes(0, 1))):
                        stdout.write('  sample {}:\n'.format(i + 1))
                        stdout.write('    input     > {}\n'.format(inp))
                        stdout.write('    predicted > {}\n'.format(pred))
                        if i >= 0:
                            break
                    stdout.write('\n')

    except KeyboardInterrupt:
        stdout.write('training interrupted')
        model.save(sess, 'autoencoder_model')
        exit(0)

    model.save(sess, 'autoencoder_model')
    helpers.save_object(loss_track, 'loss_track.pkl')

    return loss_track

def get_vector_representations(sess, model, data, save_dir,
                       batch_size=100,
                       max_batches=None,
                       batches_in_epoch=1000,
                       extension=".cell"):
    """
    Given a trained model, gets a vector representation for the traces in batch

    @param sess is a tensorflow session
    @param model is the autoencoder model
    @param data is the data (in batch-major form and not padded or a list of files (depending on `in_memory`))
    """
    batches = helpers.get_batches(data, batch_size=batch_size)

    batches_in_data = len(data) // batch_size
    if max_batches is None or batches_in_data < max_batches:
        max_batches = batches_in_data - 1

    try:
        for batch in range(max_batches):
            print("Batch {}/{}".format(batch, max_batches))
            fd, paths = model.next_batch(batches, False)
            l = sess.run(model.encoder, fd)

            file_names = [helpers.extract_filename_from_path(path, extension) for path in paths]

            for file_name, features in zip(file_names, list(l)):
                helpers.write_to_file(features, save_dir, file_name, new_extension=".cellf")

    except KeyboardInterrupt:
        stdout.write('Interrupted')
        exit(0)

    return results

\end{lstlisting}

\subsection{Sequence-to-sequence model}

\begin{lstlisting}[language=Python]
"""
This file implements a RNN encoder-decoder model (also known as sequence-to-sequence models).

We made the choice not to implement an attention mechanism (which means that the decoder is allowed to have a 'peak' at the input).
The reason why is because we are not trying to maximize the output of the decoder but instead the feature selection process.
(http://suriyadeepan.github.io/2016-06-28-easy-seq2seq/)

We will use batch-major rather than time-major even though time-major is slightly more efficient
since it makes the feature extraction process a lot easier.

We will not be using bucketing because traces of the same webpage will have the same length.
Therefore every batch, we will most likely be training the seq2seq model on one webpage

! Does encoder share weights with decoder or not (Less computation vs natural (https://arxiv.org/pdf/1409.3215.pdf))
! Reverse traces (https://arxiv.org/pdf/1409.3215.pdf)

Hyperparameters to tune:
------------------------
- Learning rate
- Which cell to use (GRU vs LSTM) or a deep RNN architecture using `MultiRNNCell`
- Reversing traces
- Bidirectional encoder
- Other objective functions (such as MSE,...)
- Amount of encoder and decoder hidden states
"""
import numpy as np
import tensorflow as tf

from sys import stdout, path
from os import path as ospath

from tensorflow.contrib.rnn import LSTMStateTuple

path.append(ospath.dirname(ospath.dirname(ospath.abspath(__file__))))
import helpers

class Seq2SeqModel():
    """
    Implements a sequence to sequence model for real values

    Attributes:
        - encoder_cell is the cell that will be used for encoding
            (Should be part of `tf.nn.rnn_cell`)
        - decoder cell is the cell used for decoding
            (Should be part of `tf.nn.rnn_cell`)

        - seq_width shows how many features each input in the sequence has
            (For website fingerprinting this is only 2 (packet_size, incoming))
        - batch_size

        - bidirectional is a boolean value that determines whether the encoder is bidirectional or not
        - reverse is also a boolean value that when if true, reversed the traces for training
    """

    def __init__(self, encoder_cell, decoder_cell, seq_width, batch_size=100, bidirectional=False, reverse=False, saved_graph=None, sess=None, learning_rate=0.0006):
        """
        @param saved_graph is a string, representing the path to the saved graph
        """
        # Constants
        self.PAD = 0
        self.EOS = -1

        self.reverse = reverse
        self.seq_width = seq_width
        self.batch_size = batch_size
        self.learning_rate = learning_rate

        self.bidirectional = bidirectional

        self.encoder_cell = encoder_cell
        self.decoder_cell = decoder_cell

        self._make_graph()

        if saved_graph is not None and sess is not None:
            self.import_from_file(sess, saved_graph)

    def _make_graph(self):
        """
        Construct the graph
        """

        self._init_placeholders()

        self._init_encoder()
        self._init_decoder()

        self._init_train()

    def _init_placeholders(self):
        """
        The main placeholders used for the input data, and output
        """
        # The usual format is: `[self.batch_size, max_sequence_length, self.seq_width]`
        # But we define `max_sequence_length` as None to make it dynamic so we only need to pad
        # each batch to the maximum sequence length
        self.encoder_inputs = tf.placeholder(tf.float32,
            [self.batch_size, None, self.seq_width])

        self.encoder_inputs_length = tf.placeholder(tf.int32, [self.batch_size])

        self.decoder_targets = tf.placeholder(tf.float32,
            [self.batch_size, None, self.seq_width])

    def _init_encoder(self):
        """
        Creates the encoder attributes

        Attributes:
            - encoder_outputs is shaped [max_sequence_length, batch_size, seq_width]
                (since time-major == True)
            - encoder_final_state is shaped [batch_size, encoder_cell.state_size]
        """
        if not self.bidirectional:
            with tf.variable_scope('Encoder') as scope:
                self.encoder_outputs, self.encoder_final_state = tf.nn.dynamic_rnn(
                    cell=self.encoder_cell,
                    dtype=tf.float32,
                    sequence_length=self.encoder_inputs_length,
                    inputs=self.encoder_inputs,
                    time_major=False)
        else:
            ((encoder_fw_outputs,
              encoder_bw_outputs),
             (encoder_fw_final_state,
              encoder_bw_final_state)) = (
                tf.nn.bidirectional_dynamic_rnn(cell_fw=self.encoder_cell,
                    cell_bw=self.encoder_cell,
                    inputs=self.encoder_inputs,
                    sequence_length=self.encoder_inputs_length,
                    dtype=tf.float32, time_major=False)
                )

            self.encoder_outputs = tf.concat((encoder_fw_outputs, encoder_bw_outputs), 2)

            if isinstance(encoder_fw_final_state, LSTMStateTuple):
                encoder_final_state_c = tf.concat(
                    (encoder_fw_final_state.c, encoder_bw_final_state.c), 1)

                encoder_final_state_h = tf.concat(
                    (encoder_fw_final_state.h, encoder_bw_final_state.h), 1)

                self.encoder_final_state = LSTMStateTuple(
                    c=encoder_final_state_c,
                    h=encoder_final_state_h
                )

            else:
                self.encoder_final_state = tf.concat(
                    (encoder_fw_final_state, encoder_bw_final_state), 1)

    def _init_decoder(self):
        """
        Creates decoder attributes.
        We cannot simply use a dynamic_rnn since we are feeding the outputs of the
        decoder back into the inputs.
        Therefore we use a raw_rnn and emulate a dynamic_rnn with this behavior.
        (https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/ops/rnn.py)
        """
        # EOS token added
        self.decoder_inputs_length = self.encoder_inputs_length + 1

        def loop_fn_initial(time, cell_output, cell_state, loop_state):
            elements_finished = (time >= self.decoder_inputs_length)

            # EOS token (0 + self.EOS)
            initial_input = tf.zeros([self.batch_size, self.decoder_cell.output_size], dtype=tf.float32) + self.EOS
            initial_cell_state = self.encoder_final_state
            initial_loop_state = None  # we don't need to pass any additional information

            return (elements_finished,
                    initial_input,
                    initial_cell_state,
                    None,  # cell output is dummy here
                    initial_loop_state)

        def loop_fn(time, cell_output, cell_state, loop_state):
            if cell_output is None:  # time == 0
                return loop_fn_initial(time, cell_output, cell_state, loop_state)

            cell_output.set_shape([self.batch_size, self.decoder_cell.output_size])

            emit_output = cell_output

            next_cell_state = cell_state

            elements_finished = (time >= self.decoder_inputs_length)
            finished = tf.reduce_all(elements_finished)

            next_input = tf.cond(
                finished,
                lambda: tf.zeros([self.batch_size, self.decoder_cell.output_size], dtype=tf.float32), # self.PAD
                lambda: cell_output # Use the input from the previous cell
            )

            next_loop_state = None

            return (
                elements_finished,
                next_input,
                next_cell_state,
                emit_output,
                next_loop_state
            )

        decoder_outputs_ta, decoder_final_state, _ = tf.nn.raw_rnn(self.decoder_cell, loop_fn)
        self.decoder_outputs = decoder_outputs_ta.stack()
        self.decoder_outputs = tf.transpose(self.decoder_outputs, [1, 0, 2])

        with tf.variable_scope('DecoderOutputProjection') as scope:
            self.decoder_outputs = self.projection(self.decoder_outputs, self.seq_width, scope)

    def _init_train(self):
        self.loss = tf.reduce_sum(tf.square(self.decoder_targets - self.decoder_outputs))

        # Which optimizer to use? `GradientDescentOptimizer`, `AdamOptimizer` or `RMSProp`?
        self.train_op = tf.train.AdamOptimizer(self.learning_rate).minimize(self.loss)

    def projection(self, inputs, projection_size, scope):
        """
        Projects the input with a known amount of features to a `projection_size amount of features`

        @param inputs is shaped like [time, batch, input_size] or [batch, input_size]
        @param projection_size int32
        @param scope outer variable scope
        """
        input_size = inputs.get_shape()[-1].value

        with tf.variable_scope(scope) as scope:
            W = tf.get_variable(name='W', shape=[input_size, projection_size],
                                dtype=tf.float32)

            b = tf.get_variable(name='b', shape=[projection_size],
                                dtype=tf.float32,
                                initializer=tf.constant_initializer(0, dtype=tf.float32))

        input_shape = tf.unstack(tf.shape(inputs))

        if len(input_shape) == 3:
            time, batch, _ = input_shape  # dynamic parts of shape
            inputs = tf.reshape(inputs, [-1, input_size])

        elif len(input_shape) == 2:
            batch, _depth = input_shape

        else:
            raise ValueError("Weird input shape: {}".format(inputs))

        linear = tf.add(tf.matmul(inputs, W), b)

        if len(input_shape) == 3:
            linear = tf.reshape(linear, [time, batch, projection_size])

        return linear

    def next_batch(self, batches, in_memory, max_time_diff=float("inf")):
        """
        Returns the next batch.

        @param batches an iterator with all of the batches (
            if in_memory == True:
                in batch-major form without padding
            else:
                A list of paths to the files
        )
        @param in_memory is a boolean value
        @param max_time_diff **(should only be defined if `in_memory == False`)**
            specifies what the maximum time different between the first packet in the trace and the last one should be

        @return if in_memory is False, returns a tuple of (dict, [paths], max_length) where paths is a list of paths for each batch
            else it returns a dict for training
        """
        batch = next(batches)
        data_batch = batch

        if not in_memory:
            data_batch = [helpers.read_cell_file(path, max_time_diff=max_time_diff) for path in batch]
            for i, cell in enumerate(data_batch):
                data_batch[i] = [packet[0] * packet[1] for packet in cell]

        data_batch, encoder_input_lengths_ = helpers.pad_traces(data_batch, reverse=self.reverse, seq_width=self.seq_width)
        encoder_inputs_ = data_batch

        decoder_targets_ = helpers.add_EOS(data_batch, encoder_input_lengths_)

        train_dict = {
            self.encoder_inputs: encoder_inputs_,
            self.encoder_inputs_length: encoder_input_lengths_,
            self.decoder_targets: decoder_targets_,
        }

        if not in_memory:
            return (train_dict, batch, max(encoder_input_lengths_))
        return train_dict

    def save(self, sess, file_name):
        """
        Save the model in a file

        @param sess is the session
        @param file_name is the file name without the extension
        """
        saver = tf.train.Saver()
        saver.save(sess, file_name)
        # saver.export_meta_graph(filename=file_name + '.meta')

    def import_from_file(self, sess, file_name):
        """
        Imports the graph from a file

        @param sess is the session
        @param file_name is a string that represents the file name
            without the extension
        """

        # Get the graph
        saver = tf.train.Saver()

        # Restore the variables
        saver.restore(sess, file_name)

def train_on_copy_task(sess, model, data,
                       batch_size=100,
                       max_batches=None,
                       batches_in_epoch=1000,
                       max_time_diff=float("inf"),
                       verbose=False):
    """
    Train the `Seq2SeqModel` on a copy task

    @param sess is a tensorflow session
    @param model is the seq2seq model
    @param data is the data (in batch-major form and not padded or a list of files (depending on `in_memory`))
    """
    batches = helpers.get_batches(data, batch_size=batch_size)

    loss_track = []

    batches_in_data = len(data) // batch_size
    if max_batches is None or batches_in_data < max_batches:
        max_batches = batches_in_data - 1

    try:
        for batch in range(max_batches):
            print("Batch {}/{}".format(batch, max_batches))
            fd, _, length = model.next_batch(batches, False, max_time_diff)
            _, l = sess.run([model.train_op, model.loss], fd)
            loss_track.append(l / length)

            if batch == 0 or batch % batches_in_epoch == 0:
                model.save(sess, 'seq2seq_model')
                helpers.save_object(loss_track, 'loss_track.pkl')

                if verbose:
                    stdout.write('  minibatch loss: {}\n'.format(sess.run(model.loss, fd)))
                    predict_ = sess.run(model.decoder_outputs, fd)
                    for i, (inp, pred) in enumerate(zip(fd[model.encoder_inputs].swapaxes(0, 1), predict_.swapaxes(0, 1))):
                        stdout.write('  sample {}:\n'.format(i + 1))
                        stdout.write('    input     > {}\n'.format(inp))
                        stdout.write('    predicted > {}\n'.format(pred))
                        if i >= 0:
                            break
                    stdout.write('\n')

    except KeyboardInterrupt:
        stdout.write('training interrupted')
        model.save(sess, 'seq2seq_model')
        exit(0)

    model.save(sess, 'seq2seq_model')
    helpers.save_object(loss_track, 'loss_track.pkl')

    return loss_track

def get_vector_representations(sess, model, data, save_dir,
                       batch_size=100,
                       max_batches=None,
                       batches_in_epoch=1000,
                       max_time_diff=float("inf"),
                       extension=".cell"):
    """
    Given a trained model, gets a vector representation for the traces in batch

    @param sess is a tensorflow session
    @param model is the seq2seq model
    @param data is the data (in batch-major form and not padded or a list of files (depending on `in_memory`))
    """
    batches = helpers.get_batches(data, batch_size=batch_size)

    batches_in_data = len(data) // batch_size
    if max_batches is None or batches_in_data < max_batches:
        max_batches = batches_in_data - 1

    try:
        for batch in range(max_batches):
            print("Batch {}/{}".format(batch, max_batches))
            fd, paths, _ = model.next_batch(batches, False, max_time_diff)
            l = sess.run(model.encoder_final_state, fd)

            # Returns a tuple, so we concatenate
            if isinstance(l, LSTMStateTuple):
                l = np.concatenate((l.c, l.h), axis=1)

            file_names = [helpers.extract_filename_from_path(path, extension) for path in paths]

            for file_name, features in zip(file_names, list(l)):
                helpers.write_to_file(features, save_dir, file_name, new_extension=".cellf")

    except KeyboardInterrupt:
        stdout.write('Interrupted')
        exit(0)
\end{lstlisting}

\subsection{Batch-normalized LSTM cell}

\begin{lstlisting}[language=Python]
"""
Implements batch normalized LSTM cells described in https://arxiv.org/pdf/1510.01378.pdf.
"""

from tensorflow.contrib.rnn import LSTMCell, LSTMStateTuple

from tensorflow.python.framework import ops
from tensorflow.python.framework import tensor_shape
from tensorflow.python.framework import tensor_util
from tensorflow.python.ops import array_ops
from tensorflow.python.ops import clip_ops
from tensorflow.python.ops import embedding_ops
from tensorflow.python.ops import init_ops
from tensorflow.python.ops import math_ops
from tensorflow.python.ops import nn_ops
from tensorflow.python.ops import partitioned_variables
from tensorflow.python.ops import random_ops
from tensorflow.python.ops import variable_scope as vs

from tensorflow.python.ops.math_ops import sigmoid
from tensorflow.python.ops.math_ops import tanh
from tensorflow.python.ops.rnn_cell_impl import _RNNCell as RNNCell

from tensorflow.python.platform import tf_logging as logging
from tensorflow.python.util import nest

from tensorflow.contrib.layers import batch_norm

import contextlib

_BIAS_VARIABLE_NAME = "biases"
_WEIGHTS_VARIABLE_NAME = "weights"

@contextlib.contextmanager
def _checked_scope(cell, scope, reuse=None, **kwargs):
  if reuse is not None:
    kwargs["reuse"] = reuse
  with vs.variable_scope(scope, **kwargs) as checking_scope:
    scope_name = checking_scope.name
    if hasattr(cell, "_scope"):
      cell_scope = cell._scope  # pylint: disable=protected-access
      if cell_scope.name != checking_scope.name:
        raise ValueError(
            "Attempt to reuse RNNCell %s with a different variable scope than "
            "its first use.  First use of cell was with scope '%s', this "
            "attempt is with scope '%s'.  Please create a new instance of the "
            "cell if you would like it to use a different set of weights.  "
            "If before you were using: MultiRNNCell([%s(...)] * num_layers), "
            "change to: MultiRNNCell([%s(...) for _ in range(num_layers)]).  "
            "If before you were using the same cell instance as both the "
            "forward and reverse cell of a bidirectional RNN, simply create "
            "two instances (one for forward, one for reverse).  "
            "In May 2017, we will start transitioning this cell's behavior "
            "to use existing stored weights, if any, when it is called "
            "with scope=None (which can lead to silent model degradation, so "
            "this error will remain until then.)"
            % (cell, cell_scope.name, scope_name, type(cell).__name__,
               type(cell).__name__))
    else:
      weights_found = False
      try:
        with vs.variable_scope(checking_scope, reuse=True):
          vs.get_variable(_WEIGHTS_VARIABLE_NAME)
        weights_found = True
      except ValueError:
        pass
      if weights_found and reuse is None:
        raise ValueError(
            "Attempt to have a second RNNCell use the weights of a variable "
            "scope that already has weights: '%s'; and the cell was not "
            "constructed as %s(..., reuse=True).  "
            "To share the weights of an RNNCell, simply "
            "reuse it in your second calculation, or create a new one with "
            "the argument reuse=True." % (scope_name, type(cell).__name__))

    # Everything is OK.  Update the cell's scope and yield it.
    cell._scope = checking_scope  # pylint: disable=protected-access
    yield checking_scope

class BNLSTMCell(LSTMCell):
  """Long short-term memory unit (LSTM) recurrent network cell.
  The default non-peephole implementation is based on:
    http://deeplearning.cs.cmu.edu/pdfs/Hochreiter97_lstm.pdf
  S. Hochreiter and J. Schmidhuber.
  "Long Short-Term Memory". Neural Computation, 9(8):1735-1780, 1997.
  The peephole implementation is based on:
    https://research.google.com/pubs/archive/43905.pdf
  Hasim Sak, Andrew Senior, and Francoise Beaufays.
  "Long short-term memory recurrent neural network architectures for
   large scale acoustic modeling." INTERSPEECH, 2014.
  The class uses optional peep-hole connections, optional cell clipping, and
  an optional projection layer.
  """

  def __init__(self, num_units, input_size=None,
               use_peepholes=False, cell_clip=None,
               initializer=None, num_proj=None, proj_clip=None,
               num_unit_shards=None, num_proj_shards=None,
               forget_bias=1.0, state_is_tuple=True,
               activation=tanh, is_training=True, batch_norm=True):
    """Initialize the parameters for an LSTM cell.
    Args:
      num_units: int, The number of units in the LSTM cell
      input_size: Deprecated and unused.
      use_peepholes: bool, set True to enable diagonal/peephole connections.
      cell_clip: (optional) A float value, if provided the cell state is clipped
        by this value prior to the cell output activation.
      initializer: (optional) The initializer to use for the weight and
        projection matrices.
      num_proj: (optional) int, The output dimensionality for the projection
        matrices.  If None, no projection is performed.
      proj_clip: (optional) A float value.  If `num_proj > 0` and `proj_clip` is
        provided, then the projected values are clipped elementwise to within
        `[-proj_clip, proj_clip]`.
      num_unit_shards: Deprecated, will be removed by Jan. 2017.
        Use a variable_scope partitioner instead.
      num_proj_shards: Deprecated, will be removed by Jan. 2017.
        Use a variable_scope partitioner instead.
      forget_bias: Biases of the forget gate are initialized by default to 1
        in order to reduce the scale of forgetting at the beginning of
        the training.
      state_is_tuple: If True, accepted and returned states are 2-tuples of
        the `c_state` and `m_state`.  If False, they are concatenated
        along the column axis.  This latter behavior will soon be deprecated.
      activation: Activation function of the inner states.
      is_training: Python boolean describing whether or not you are currently
        training. Should only be changed if `batch_norm == True`
      batch_norm: Python boolean that indicated whether or not the cell is
        batch normalized
    """
    self._is_training = is_training
    self._batch_norm = batch_norm

    super().__init__(num_units, input_size=input_size,
                 use_peepholes=use_peepholes, cell_clip=cell_clip,
                 initializer=initializer, num_proj=num_proj, proj_clip=proj_clip,
                 num_unit_shards=num_unit_shards, num_proj_shards=num_proj_shards,
                 forget_bias=forget_bias, state_is_tuple=state_is_tuple,
                 activation=activation)

  def __call__(self, inputs, state, scope=None):
    """Run one step of LSTM.
    Args:
      inputs: input Tensor, 2D, batch x num_units.
      state: if `state_is_tuple` is False, this must be a state Tensor,
        `2-D, batch x state_size`.  If `state_is_tuple` is True, this must be a
        tuple of state Tensors, both `2-D`, with column sizes `c_state` and
        `m_state`.
      scope: VariableScope for the created subgraph; defaults to "lstm_cell".
    Returns:
      A tuple containing:
      - A `2-D, [batch x output_dim]`, Tensor representing the output of the
        LSTM after reading `inputs` when previous state was `state`.
        Here output_dim is:
           num_proj if num_proj was set,
           num_units otherwise.
      - Tensor(s) representing the new state of LSTM after reading `inputs` when
        the previous state was `state`.  Same type and shape(s) as `state`.
    Raises:
      ValueError: If input size cannot be inferred from inputs via
        static shape inference.
    """
    num_proj = self._num_units if self._num_proj is None else self._num_proj

    if self._state_is_tuple:
      (c_prev, m_prev) = state
    else:
      c_prev = array_ops.slice(state, [0, 0], [-1, self._num_units])
      m_prev = array_ops.slice(state, [0, self._num_units], [-1, num_proj])

    dtype = inputs.dtype
    input_size = inputs.get_shape().with_rank(2)[1]
    if input_size.value is None:
      raise ValueError("Could not infer input size from inputs.get_shape()[-1]")
    with _checked_scope(self, scope or "lstm_cell",
                        initializer=self._initializer) as unit_scope:
      if self._num_unit_shards is not None:
        unit_scope.set_partitioner(
            partitioned_variables.fixed_size_partitioner(
                self._num_unit_shards))
      # i = input_gate, j = new_input, f = forget_gate, o = output_gate
      lstm_matrix = _linear([inputs, m_prev], 4 * self._num_units, bias=True)
      i, j, f, o = array_ops.split(
          value=lstm_matrix, num_or_size_splits=4, axis=1)
      # Diagonal connections
      if self._use_peepholes:
        with vs.variable_scope(unit_scope) as projection_scope:
          if self._num_unit_shards is not None:
            projection_scope.set_partitioner(None)
          w_f_diag = vs.get_variable(
              "w_f_diag", shape=[self._num_units], dtype=dtype)
          w_i_diag = vs.get_variable(
              "w_i_diag", shape=[self._num_units], dtype=dtype)
          w_o_diag = vs.get_variable(
              "w_o_diag", shape=[self._num_units], dtype=dtype)

      if self._use_peepholes:
        res = (sigmoid(f + self._forget_bias + w_f_diag * c_prev) * c_prev +
             sigmoid(i + w_i_diag * c_prev) * self._activation(j))
        if self._batch_norm:
          c = batch_norm(res,
                         center=True, scale=True,
                         is_training=self._is_training,
                         scope='bn')
        else:
          c = res
      else:
        res = (sigmoid(f + self._forget_bias) * c_prev + sigmoid(i) *
             self._activation(j))
        if self._batch_norm:
          c = batch_norm(res,
                         center=True, scale=True,
                         is_training=self._is_training,
                         scope='bn')
        else:
          c = res

      if self._cell_clip is not None:
        # pylint: disable=invalid-unary-operand-type
        c = clip_ops.clip_by_value(c, -self._cell_clip, self._cell_clip)
        # pylint: enable=invalid-unary-operand-type
      if self._use_peepholes:
        m = sigmoid(o + w_o_diag * c) * self._activation(c)
      else:
        m = sigmoid(o) * self._activation(c)

      if self._num_proj is not None:
        with vs.variable_scope("projection") as proj_scope:
          if self._num_proj_shards is not None:
            proj_scope.set_partitioner(
                partitioned_variables.fixed_size_partitioner(
                    self._num_proj_shards))
          m = _linear(m, self._num_proj, bias=False)

        if self._proj_clip is not None:
          # pylint: disable=invalid-unary-operand-type
          m = clip_ops.clip_by_value(m, -self._proj_clip, self._proj_clip)
          # pylint: enable=invalid-unary-operand-type

    new_state = (LSTMStateTuple(c, m) if self._state_is_tuple else
                 array_ops.concat([c, m], 1))
    return m, new_state

def _linear(args, output_size, bias, bias_start=0.0):
  """Linear map: sum_i(args[i] * W[i]), where W[i] is a variable.
  Args:
    args: a 2D Tensor or a list of 2D, batch x n, Tensors.
    output_size: int, second dimension of W[i].
    bias: boolean, whether to add a bias term or not.
    bias_start: starting value to initialize the bias; 0 by default.
  Returns:
    A 2D Tensor with shape [batch x output_size] equal to
    sum_i(args[i] * W[i]), where W[i]s are newly created matrices.
  Raises:
    ValueError: if some of the arguments has unspecified or wrong shape.
  """
  if args is None or (nest.is_sequence(args) and not args):
    raise ValueError("`args` must be specified")
  if not nest.is_sequence(args):
    args = [args]

  # Calculate the total size of arguments on dimension 1.
  total_arg_size = 0
  shapes = [a.get_shape() for a in args]
  for shape in shapes:
    if shape.ndims != 2:
      raise ValueError("linear is expecting 2D arguments: %s" % shapes)
    if shape[1].value is None:
      raise ValueError("linear expects shape[1] to be provided for shape %s, "
                       "but saw %s" % (shape, shape[1]))
    else:
      total_arg_size += shape[1].value

  dtype = [a.dtype for a in args][0]

  # Now the computation.
  scope = vs.get_variable_scope()
  with vs.variable_scope(scope) as outer_scope:
    weights = vs.get_variable(
        _WEIGHTS_VARIABLE_NAME, [total_arg_size, output_size], dtype=dtype)
    if len(args) == 1:
      res = math_ops.matmul(args[0], weights)
    else:
      res = math_ops.matmul(array_ops.concat(args, 1), weights)
    if not bias:
      return res
    with vs.variable_scope(outer_scope) as inner_scope:
      inner_scope.set_partitioner(None)
      biases = vs.get_variable(
          _BIAS_VARIABLE_NAME, [output_size],
          dtype=dtype,
          initializer=init_ops.constant_initializer(bias_start, dtype=dtype))
    return nn_ops.bias_add(res, biases)
\end{lstlisting}

\subsection{Batch-normalized GRU cell}
\begin{lstlisting}[language=Python]
"""
Implements a batch normalized GRU cell.
Unfortunately, there aren't any research papers that examine how to exactly implement this.

But most of it is based on https://arxiv.org/pdf/1510.01378.pdf
"""

from tensorflow.contrib.rnn import GRUCell

from tensorflow.python.framework import ops
from tensorflow.python.framework import tensor_shape
from tensorflow.python.framework import tensor_util
from tensorflow.python.ops import array_ops
from tensorflow.python.ops import clip_ops
from tensorflow.python.ops import embedding_ops
from tensorflow.python.ops import init_ops
from tensorflow.python.ops import math_ops
from tensorflow.python.ops import nn_ops
from tensorflow.python.ops import partitioned_variables
from tensorflow.python.ops import random_ops
from tensorflow.python.ops import variable_scope as vs

from tensorflow.python.ops.math_ops import sigmoid
from tensorflow.python.ops.math_ops import tanh
from tensorflow.python.ops.rnn_cell_impl import _RNNCell as RNNCell

from tensorflow.python.platform import tf_logging as logging
from tensorflow.python.util import nest

from tensorflow.contrib.layers import batch_norm

import contextlib

_BIAS_VARIABLE_NAME = "biases"
_WEIGHTS_VARIABLE_NAME = "weights"

@contextlib.contextmanager
def _checked_scope(cell, scope, reuse=None, **kwargs):
  if reuse is not None:
    kwargs["reuse"] = reuse
  with vs.variable_scope(scope, **kwargs) as checking_scope:
    scope_name = checking_scope.name
    if hasattr(cell, "_scope"):
      cell_scope = cell._scope  # pylint: disable=protected-access
      if cell_scope.name != checking_scope.name:
        raise ValueError(
            "Attempt to reuse RNNCell %s with a different variable scope than "
            "its first use.  First use of cell was with scope '%s', this "
            "attempt is with scope '%s'.  Please create a new instance of the "
            "cell if you would like it to use a different set of weights.  "
            "If before you were using: MultiRNNCell([%s(...)] * num_layers), "
            "change to: MultiRNNCell([%s(...) for _ in range(num_layers)]).  "
            "If before you were using the same cell instance as both the "
            "forward and reverse cell of a bidirectional RNN, simply create "
            "two instances (one for forward, one for reverse).  "
            "In May 2017, we will start transitioning this cell's behavior "
            "to use existing stored weights, if any, when it is called "
            "with scope=None (which can lead to silent model degradation, so "
            "this error will remain until then.)"
            % (cell, cell_scope.name, scope_name, type(cell).__name__,
               type(cell).__name__))
    else:
      weights_found = False
      try:
        with vs.variable_scope(checking_scope, reuse=True):
          vs.get_variable(_WEIGHTS_VARIABLE_NAME)
        weights_found = True
      except ValueError:
        pass
      if weights_found and reuse is None:
        raise ValueError(
            "Attempt to have a second RNNCell use the weights of a variable "
            "scope that already has weights: '%s'; and the cell was not "
            "constructed as %s(..., reuse=True).  "
            "To share the weights of an RNNCell, simply "
            "reuse it in your second calculation, or create a new one with "
            "the argument reuse=True." % (scope_name, type(cell).__name__))

    # Everything is OK.  Update the cell's scope and yield it.
    cell._scope = checking_scope  # pylint: disable=protected-access
    yield checking_scope

class BNGRUCell(GRUCell):
  """Gated Recurrent Unit cell (cf. http://arxiv.org/abs/1406.1078)."""

  def __init__(self, num_units, input_size=None, activation=tanh, is_training=True, batch_norm=True):
    self._is_training = is_training
    self._batch_norm = batch_norm

    super().__init__(num_units, input_size, activation)

  def __call__(self, inputs, state, scope=None):
    """Gated recurrent unit (GRU) with nunits cells."""
    with _checked_scope(self, scope or "gru_cell"):
      with vs.variable_scope("gates"):  # Reset gate and update gate.
        # We start with bias of 1.0 to not reset and not update.
        value = sigmoid(_linear(
          [inputs, state], 2 * self._num_units, True, 1.0))
        r, u = array_ops.split(
            value=value,
            num_or_size_splits=2,
            axis=1)
      with vs.variable_scope("candidate"):
        res = self._activation(_linear([inputs, r * state],
                                     self._num_units, True))

        if self._batch_norm:
          c = batch_norm(res,
                         center=True, scale=True,
                         is_training=self._is_training,
                         scope='bn1')
        else:
          c = res

      new_h = u * state + (1 - u) * c
    return new_h, new_h

def _linear(args, output_size, bias, bias_start=0.0):
  """Linear map: sum_i(args[i] * W[i]), where W[i] is a variable.
  Args:
    args: a 2D Tensor or a list of 2D, batch x n, Tensors.
    output_size: int, second dimension of W[i].
    bias: boolean, whether to add a bias term or not.
    bias_start: starting value to initialize the bias; 0 by default.
  Returns:
    A 2D Tensor with shape [batch x output_size] equal to
    sum_i(args[i] * W[i]), where W[i]s are newly created matrices.
  Raises:
    ValueError: if some of the arguments has unspecified or wrong shape.
  """
  if args is None or (nest.is_sequence(args) and not args):
    raise ValueError("`args` must be specified")
  if not nest.is_sequence(args):
    args = [args]

  # Calculate the total size of arguments on dimension 1.
  total_arg_size = 0
  shapes = [a.get_shape() for a in args]
  for shape in shapes:
    if shape.ndims != 2:
      raise ValueError("linear is expecting 2D arguments: %s" % shapes)
    if shape[1].value is None:
      raise ValueError("linear expects shape[1] to be provided for shape %s, "
                       "but saw %s" % (shape, shape[1]))
    else:
      total_arg_size += shape[1].value

  dtype = [a.dtype for a in args][0]

  # Now the computation.
  scope = vs.get_variable_scope()
  with vs.variable_scope(scope) as outer_scope:
    weights = vs.get_variable(
        _WEIGHTS_VARIABLE_NAME, [total_arg_size, output_size], dtype=dtype)
    if len(args) == 1:
      res = math_ops.matmul(args[0], weights)
    else:
      res = math_ops.matmul(array_ops.concat(args, 1), weights)
    if not bias:
      return res
    with vs.variable_scope(outer_scope) as inner_scope:
      inner_scope.set_partitioner(None)
      biases = vs.get_variable(
          _BIAS_VARIABLE_NAME, [output_size],
          dtype=dtype,
          initializer=init_ops.constant_initializer(bias_start, dtype=dtype))
    return nn_ops.bias_add(res, biases)
\end{lstlisting}

\endgroup

\end{multicols}
\end{landscape}


\documentclass[a4paper]{article}
\usepackage[a4paper, total={6.5in, 10in}]{geometry}
\usepackage[english]{babel}
\usepackage[utf8]{inputenc}
\usepackage{amsmath}
\usepackage{graphicx}
\usepackage{multirow}
\usepackage{graphicx}
\usepackage{caption}
\usepackage{subcaption}
\usepackage{floatrow}
\restylefloat{table}
\usepackage[colorinlistoftodos]{todonotes}
\usepackage{float}
\title{Vector-Valued Image Regularization with PDEs}

\author{Meet Kathiriya, Shreyas Pimpalgaonkar, Saiteja Talluri \\ 160050001,160050024,160050098}

\date{\today}

\begin{document}
\maketitle

\begin{abstract}

We focussed on techiniques for vector-valued image regularization, based on variational methods and PDEs. We performed denoising, image reconstruction, inpainting, magnification and flow visualization on color images using these techinques and presented our results and conclusions.
\end{abstract}

\section{Theory}
\label{sec:theory}
\hspace{5mm}The problem of image regularisation can be solved by three interconnected models of functional \\ minimisation, divergence expressions and oriented laplacians.

The structure tensor is used here, and it's eigen vectors represent directions of maximum variation. Therefore, the problem can be formulated as minimising the generalised functional representing the variation in the image using the eigen vectors as follows
$$\min_{I:\Omega\to\mathcal{R}^{n}} E(I) = \int_{\Omega} \psi(\lambda_{+},\lambda_{-})\,d\Omega$$

 The above equation can be converted to the divergence of a diffusion matrix with gradient of the Image channel by using Euler-Lagrange theorem, heat equations, orthogonality of eigen vectors and simple partial differential equation chain rules as done in the appendix 1 of the original paper.
        $$\frac{\partial I_i}{\partial t} = div ( D \nabla I )          \qquad  (i = 1..n) $$
        $$ \textbf{D} = \frac{\partial \psi}{\partial \lambda_+} \theta_+ \theta_+^T  +  \frac{\partial \psi}{\partial \lambda_-} \theta_- \theta_-^T$$
    where $n$ is the number of channels and $\theta_+,\theta_-$ are the eigen vectors of the structure tensor. If a $\psi$ exists, eigenvalues of the divergence tensor D can be seen it's gradient and obtaining $I_i$ is easy for every channel.

The oriented laplacian equation can be written as the following PDE
$$\frac { \partial I _ { i } } { \partial t } = c _ { 1 } I _ { i \epsilon \varepsilon } + c _ { 2 } I _ { i m } = \operatorname { trace } \left( \mathbf { T } H _ { i } \right)$$
 $$ I _ { i ( t ) } = I _ { i _ { ( t = 0 ) } } * G ^ { ( \mathbf { T } , t ) }$$
where $*$ stands for the convolution with the oriented Gaussian kernel
$$G ^ { ( \mathbf { T } , t ) } ( \mathbf { x } ) = \frac { 1 } { 4 \pi t } \exp \left( - \frac { \mathbf { x } ^ { T } \mathbf { T } ^ { - 1 } \mathbf { x } } { 4 t } \right) \quad \text { with } \quad \mathbf { x } = ( x \quad y ) ^ { T }$$

$$\frac { \partial I _ { i } } { \partial t } = \operatorname { trace } \left( \mathbf { T } \mathbf { H } _ { i } \right) \quad ( i = 1 . . n )$$

where $\textbf{T}$ is the tensor field defined pointwise as

$$\mathbf { T } = f _ { - } \left( \sqrt { \lambda _ { + } ^ { * } + \lambda _ { - } ^ { * } } \right) \theta _ { - } ^ { * } \theta _ { - } ^ { * T } + f _ { + } \left( \sqrt { \lambda _ { + } ^ { * } + \lambda _ { - } ^ { * } } \right) \theta _ { + } ^ { * } \theta _ { + } ^ { * } T$$

where $\lambda_{\pm}^*$ and $\theta_{\pm}^*$ are defined to be the spectral elements of $G_\sigma = G * G_{\sigma'}$ , a Gaussian smoothed version of the structure tensor G, allowing us to retrieve a more coherent vector geometry giving a better approximation of the vector discontinuity directions.

For our experiments we choose, $f _ { + } ( s ) = \frac { 1 } { 1 + s ^ { 2 } } .\text { and } f _ { - } ( s ) = \frac { 1 } { \sqrt { 1 + s ^ { 2 } } }$ \newline

Finally, a high level overview of the algorithm is to convolve the image with a gaussian that solves the PDE of Oriented Laplacians with $\textbf{T}$ and $f$ as in the above section. This is equivalent to using the trace formula in section 2.3 and also is proved in the original paper.

\pagebreak
\section{Results and Inferences}

\subsection{Image Inpainting}

\begin{figure}[h]
\centering
\includegraphics[width=1\textwidth]{glasses.png}
\caption{\label{fig:data} Inpainting with 21x21 neighbourhood, t=5 and 10 iterations}
\end{figure}

\begin{figure}[h]
\centering
\includegraphics[width=1\textwidth]{parrot1.png}
\caption{\label{fig:data} Inpainting with 10x10 neighbourhood, t=2 and 20 iterations}
\end{figure}

\begin{figure}[H]
\centering
\includegraphics[width=1\textwidth]{redparrot.png}
\caption{\label{fig:data} Inpainting with 20x20 neighbourhood, t=3 and 20 iterations Also the black prints at boundaries and some darker regions in image is due to the fact that the mask we are using is imperfect as it contains more white region at some locations than the cage obstacle in real image.}
\end{figure}

\pagebreak
\subsection{Image Denoising}

\begin{figure}[h]
\centering
\includegraphics[width=1\textwidth]{baby.png}
\caption{\label{fig:data} From L to R : Noisy image, image filtered with gaussian smoothing with 5x5 kernel, image filtered with PDE regularisation with t=3, 3x3 neighbourhood and 5 iterations - better performance and lower noise especially on left and top parts of the image }
\end{figure}

\begin{figure}[h]
\centering
\includegraphics[width=0.8\textwidth,height=0.4\textwidth]{face.png}
\caption{\label{fig:data} From L to R : Noisy image, image filtered with PDE with t=3, 3x3 neighbourhood and 5 iterations }
\end{figure}

\subsection{Image Magnification}

\begin{figure}[H]
\centering
\includegraphics[width=0.8\textwidth]{jerry.png}
\caption{\label{fig:data}
    From L to R : Original Image. This image was downsampled by 4 to serve as input for the two algorithms. Magnified image using bi-linear interpolation, magnified image using PDE regularisation with t=3, 3x3 neighbourhood and 10 iterations.
}

\end{figure}

\pagebreak

\subsection{Flow Visualisation}

\begin{table}[H]
\caption{Input images, flow vectors and output images, with dt = 0.01, 100 iterations and sobel gradient vectors for Hessian Matrix. Paralallised code, runs much faster compared to others}
\begin{tabular}{c c c}
\includegraphics[width=0.33\textwidth,height=0.4\textwidth]{1} & \includegraphics[width=0.33\textwidth,height=0.4\textwidth]{5} &
\includegraphics[width=0.33\textwidth,height=0.4\textwidth]{2}\\
\includegraphics[width=0.33\textwidth,height=0.4\textwidth]{4} & \includegraphics[width=0.33\textwidth,height=0.4\textwidth]{6} &
\includegraphics[width=0.33\textwidth,height=0.4\textwidth]{3}
\end{tabular}
\end{table}

\subsection{Image Restoration}

\begin{figure}[H]
\centering
\includegraphics[width=1\textwidth]{obama.png}
\caption{\label{fig:data}
    From L to R : Input Image, image with 50 \% pixels removed in 2x2 granularity, output image with SSIM of 0.995, better than results with median filter. Also, median filter will become less accurate on increasing the box sizes. }

\end{figure}

\begin{figure}[H] \label{frow}
 \begin{floatrow}
 {\includegraphics[width=0.5\textwidth]{l1}}
 {\includegraphics[width=0.5\textwidth]{l2}}
 \end{floatrow}
 \caption{\label{fig:data} from L to R : image with 50 \% pixels removed and SSIM = 0.23, restored image with t=4, 10 iterations with 3x3 neighbourhood, SSIM = 0.966 with original image}
\end{figure}

\section{Experimental Observations and Conclusions}

\begin{enumerate}
    \item This technique is hard to parallelise in MATLAB as we have to calculate the eigen vectors and eigen values of the  structure tensor at each pixel (The matrix $\textbf{T}$). The MATLAB eig function doesn't provide this functionality. We tried to do this using parfor but it takes too much time to setup the threads and ultimately is slower.

    \item Image Inpainting fails drastically as the size of the width of continuous portion of mask increases. Inpainting by this framework is only suitable for removing shapes like cylinders/spheres with small radii. On increasing the width, the lost information is harder to recover due to the high variability in diffusion. Some techniques that specialise in hole filling on the basis of similar patterns like used in Mean Shift Segmentation can be used to give better results. Also, machine learning models can be trained to predict the missing parts, rather than regularisation.

    \item This technique performs almost similar to bilateral filtering in denoising images, with a lot more computation overhead.

    \item Algorithm follows three main parameters neighbourhood,number of iterations and time. From our experimentation we observed that neighbourhood size highly affects the regularization and it determines amount of smoothing (directly proportional) and edge preservation (indirectly proportional), number of iteration also determines above mentioned image characteristics but they characteristic change start to converge as the number of iteration increases. the time parameter we observed to be least affecting on result as in a very large t worked as a mean filter and very small t there was no significant regularization on image.

    \item While using PDE for image magnification we didn't obtain desired results when the down sampling ratio is high. The possible reason for this could be that we are applying PDE on image magnified with bi-linear interpolation which doesn't retain global features like edges for high down sampling ratios.

    \item Flow visualization with PDEs doesn't require calculation of Tensor field at each point unlike denoising and inpainting so with vector manipulations in MATLAB we could very efficiently compute it if the Field vector is not dependent on local variations at each pixel.

\end{enumerate}

\begin{thebibliography}{9}
D. Tschumperle ́ and R. Deriche, “Vector-valued image regularization with PDE’s: A common framework for different applications,” IEEE Trans. PAMI, vol. 27, no. 4, pp. 506–517, 2005.
\end{thebibliography}
\end{document}