\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage[backend=biber]{biblatex}
\usepackage{amssymb}
\usepackage{amsmath}
\usepackage{dsfont}
\addbibresource{bib.bib}
\setlength{\parindent}{0em}
\bibliography{bib}
\setlength{\parskip}{6pt}
\usepackage[margin=1.0in]{geometry}
\usepackage{graphicx}
\usepackage{caption}
\usepackage{subcaption}
\usepackage{wrapfig}
\usepackage{url}

\title{Intro to deep learning with PyTorch}
\author{Miguel A. Saavedra-Ruiz}
\date{Abril 2020}
\linespread{1.0}

\nocite{*}

\begin{document}

\maketitle

\section*{Introduction to Neural Networks}

Deep learning is everywhere, applications in games such as Go or jeopardy, detecting spam in emails, forecasting stock prices, recognizing images in a picture, and diagnosing illnesses sometimes with more precision than doctors are just few examples.

The heart of Deep learning are object called \textbf{neural networks}. Neural networks vaguely mimic the process of how the brain operates, with neurons that fire bits of information. The next image shows a neural network \ref{fig:f1}.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.35\textwidth,height=0.35\textheight,keepaspectratio]{images/nn.PNG}
    \captionsetup{justification=centering}
    \caption{A Neural network}
    \label{fig:f1}
\end{figure}

A neural network is a function approximator which is capable to split data. Given some data in the form of blue or red points, the neural network will look for the best line that separates the data \ref{fig:f2}..

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.2\textwidth,height=0.2\textheight,keepaspectratio]{images/data.PNG}
    \captionsetup{justification=centering}
    \caption{A Neural network splitting data}
    \label{fig:f2}
\end{figure}

The separation line is called a model and its job is to split the data. A model might not be perfect, but it most be as accurate as possible. Imagine the next example where we plot the acceptance rate of students based on their grades and test scores. The blue dots are accepted and red rejected. Furthermore, the line plotted is "the model" Fig. \ref{fig:f3}.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.5\textwidth,height=0.5\textheight,keepaspectratio]{images/example_1.PNG}
    \captionsetup{justification=centering}
    \caption{Neural network model}
    \label{fig:f3}
\end{figure}

Based on that, it is safe to predict that if a point is over the line the student gets accepted and if it's under the line then the student gets rejected.

Taking into account that Test is label as \(x_1\) and Grades as \(x_2\), the \textbf{boundary line} that separates the blue and the red points is going to have a linear equation. The equation of the one drawn above is describes in \eqref{eq:1}.

\begin{equation}
2 x_1 + x_2 - 18 = 0 \label{eq:1}
\end{equation}

The equation \eqref{eq:1} means that the method for accepting or rejecting students says the following: take this equation as our score where \(score =2.Test + Grades - 18\), if a student has \(score > 0\) the he gets accepted, otherwise he is rejected. This process is called a prediction. Additionally, by convention it is possible to say that if the score is 0, the student will get accepted although this won't matter much at the end. \textbf{The linear equation is the model}.

In the more general case, a boundary will be an equation of the following form \eqref{eq:2}.

\begin{equation}
wx_1+w_2x_2+b=0 \label{eq:2}
\end{equation}

The last equation can be abbreviate in vector notation as \eqref{eq:3}.

\begin{equation}
wx+b=0 \label{eq:3}
\end{equation}

Where w and x are vectors as shown below:

\[ w = (w_1, w_2)\]
\[ x = (x_1, x_2)\]

X can be refereed as the input, w as the weights and b as the bias. For a student coordinates \(x_1, x_2\), a label denoted as Y will be used as the value to predict. So if the student gets accepted, namely the point is blue,
then the label is \( y = 1\), And if the student gets rejected, namely the point is red and then the label is \( y = 0\)

The prediction made by our algorithm is going to be called \(\hat{y}\) and it will be what the algorithm predicts that the label will be \eqref{eq:4}. Where one means accepted and zero rejected.

\begin{equation}
\label{eq:4}
\hat{y} =
  \begin{cases}
    1, & \text{if } Wx + b \geq 0 \\
    0, & \text{if } Wx + b < 0 \\
  \end{cases}
\end{equation}

So, to summarize, the points above the line have \( \hat{y} = 1\) and the points below the line have \( \hat{y} = 0\). And, the blue points have \( y = 1\) and the red points have \( y = 0\).

Subsequently, if we have more data columns so not just testing grades, but maybe something else like the ranking of the student in the class it will turn the problem into a three-dimensional space. The only difference now is that the problem won't be working in two dimensions but three. So now, the three axis are: \(x_1\) for the test, \(x_2\) for the grades and \(x_3\) for the class ranking. The data will looks like Fig. \ref{fig:f4}.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.5\textwidth,height=0.5\textheight,keepaspectratio]{images/example_2.PNG}
    \captionsetup{justification=centering}
    \caption{3D data example}
    \label{fig:f4}
\end{figure}

The equation won't be a line in two dimension, but a plane in three dimensions with a similar equation as before. Now, the equation would be \eqref{eq:5} which will separate this space into two regions.

\begin{equation}
w_1x_1 + w_2x_2 + w_3x_3 + b = 0, \label{eq:5}
\end{equation}

This equation \eqref{eq:5} can still be abbreviated by \eqref{eq:3} (\(wx+b=0\)) except our vectors will now have three entries instead of two. The prediction will be the same as described by \eqref{eq:4}.

If we have many columns like say n of them as shown in Fig. \ref{fig:f5}, the data just leaps in n-dimensional space. Here the points are just things with n coordinates called \(x_1, x_2, x_3, \dots, x_n  \) with the labels being \(y\).

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.5\textwidth,height=0.5\textheight,keepaspectratio]{images/multiple_dimension.PNG}
    \captionsetup{justification=centering}
    \caption{Multi-dimensional data}
    \label{fig:f5}
\end{figure}

Therefore, the boundaries just an \(n-1\) dimensional hyperplane, and the equation of this \(n-1\) dimensional hyperplane is going to be the one shown in \eqref{eq:6} or the abbreviated form shown in \eqref{eq:3} and the predictions still being the same as shown by equation \eqref{eq:4}.

\begin{equation}
w_1x_1 + w_2x_2 + \dots + w_n x_n + b = zero, \label{eq:6}
\end{equation}

e.g Given the table in the Fig. \ref{fig:f5}, the dimensions for input features (x), the weights (W), and the bias (b) to satisfy (Wx + b) would be \( W(1xn), x(nx1) \text{and} b(1x1)\) for a single student.

Now it is time to introduce the notion of a \textbf{Perceptron}, which is the building block of neural networks. It is possible to show an example based on Fig. \ref{fig:f3}, where it is possible to encode the previous equations into a small graph. Let's fit The data and boundary line inside a node. Add small nodes for the inputs which in this case, they are the test and grades. The Fig. \ref{fig:f6} presents the mention before, with a specific value of test and grades. The Perceptron checks if the point is in the positive or negative area. If the point is in the positive area, then it returns a yes, otherwise a no.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.5\textwidth,height=0.5\textheight,keepaspectratio]{images/perceptron_1.PNG}
    \captionsetup{justification=centering}
    \caption{Perceptron example with student admission}
    \label{fig:f6}
\end{figure}

Recall that the prediction consists of accepting the student based on \(score =2.Test + Grades - 18\), if the \( score \geq 0\) he gets accepted, and rejecting them if the \(score < 0\)

The weights \(2\), \(1\) and \(-18\) are what define the linear equation, and so they are used as labels in the graph Fig. \ref{fig:f6}. Another way to graph this node is to consider the bias as part of the input.

Since \(w_1\) gets multiplied by \(x_1\) and \(w_2\) by \(x_2\), it's natural to think that \(b\) gets multiplied by a one. So the bias will have the B labeling and and edge coming from a one. Then what the node does is it multiplies the values coming from the incoming nodes by the values and the corresponding edges. Then it adds them and finally, it checks if the result is greater than or equal to zero. If it is, then the node returns a yes or a value of one, and if it isn't then the node returns a no or a value of zero as shown in Fig. \ref{fig:f7}.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.45\textwidth,height=0.45\textheight,keepaspectratio]{images/perceptron_2.PNG}
    \captionsetup{justification=centering}
    \caption{Perceptron with weights and bias}
    \label{fig:f7}
\end{figure}

In the general case, a node looks like the one presented in Fig. \ref{fig:f8}.The node has inputs coming in with values \(x_1\) up to \(x_n\) and one (for the bias), and edges with weights \(w_1\) up to \(w_n\), and \(b\) corresponding to the bias unit. Then the node calculates the linear equation \eqref{eq:3}, which is this case is \(\sum_{i=1}^n w_i x_i + B\). This node then checks if the value is zero or bigger, and if it is, then the node returns a value of one for yes and if not, then it returns a value of zero for no.

\[wx+b = \sum_{i=1}^n w_i x_i + b\]

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.5\textwidth,height=0.5\textheight,keepaspectratio]{images/general_perceptron.PNG}
    \captionsetup{justification=centering}
    \caption{General Perceptron}
    \label{fig:f8}
\end{figure}

In the last Figure, an implicit function is being used, this is called a step function. What the step function does is it returns a one if the input is positive or zero if the input is negative Fig. \ref{fig:f9}.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.4\textwidth,height=0.4\textheight,keepaspectratio]{images/step.PNG}
    \captionsetup{justification=centering}
    \caption{Step function}
    \label{fig:f9}
\end{figure}

\[
y =
  \begin{cases}
    1, & \text{if } x \geq 0 \\
    0, & \text{if } x < 0 \\
  \end{cases}
\]

In reality, the perceptron can be seen as a combination of nodes, where the first node calculates a linear equation based on the inputs and weights. The second node applies the step function to the result, giving the final outcome of the perceptron Fig. \ref{fig:f10}. The \(\sum\) represents a linear function in the first node and the drawing represents a step function in the second node. Nevertheless, we can use different step functions or generally, activation functions in this node.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.5\textwidth,height=0.5\textheight,keepaspectratio]{images/simple_perceptron.PNG}
    \captionsetup{justification=centering}
    \caption{The simple perceptron}
    \label{fig:f10}
\end{figure}

The name "neural network" comes from the fact that perceptrons look like neurons in the brain. In the Fig. \ref{fig:f11} can be seen a perceptron in the Left with four inputs, these inputs are the dendrites in a Neuron and they receive nervous impulses. Similarly, the computation is made in the node of the perceptron or nucleus. Finally, the perceptron outputs the final result as a neuron does through its axon, which is the mechanism a neuron use to outputs nervous impulses. \textbf{The neural networks mimic the concept of neuron and brain} by taking the output from one neuron and turning it into the input for another one.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.5\textwidth,height=0.5\textheight,keepaspectratio]{images/perceotron_neuron.png}
    \captionsetup{justification=centering}
    \caption{The perceptron as a neuron}
    \label{fig:f11}
\end{figure}

Perceptrons can be used as logic operators (AND, OR, XOR). E.g the ND operator can be represented as a two inputs perceptron with 1 output, where the inputs can be 1 or 0 as well as the output. To recall the true table of the perceptron, it is presented in Fig. \ref{fig:f12}. The perceptron presented in the same figure shows how it splits the points with a boundary line, where the dots in the red area corresponds to a zero and the point in the blue area corresponds to a one. In the file called \textit{introduction/and\_perceptron.py} can be seen an example of this which use the general equation of a boundary \eqref{eq:2} (\(wx_1+w_2x_2+b=0\)).

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.5\textwidth,height=0.5\textheight,keepaspectratio]{images/and_table.png}
    \captionsetup{justification=centering}
    \caption{True table of the AND operator}
    \label{fig:f12}
\end{figure}

It is possible to represent logic operators as AND or OR with simple perceptrons, nevertheless, a problem arises when the problem has non-linear boundaries.

To find the boundary line for a simple logic operator the process is really straightforward. Imagine a classification problem as shown by Fig. \ref{fig:f13}, the goal is to classify both the blue and red points correctly. However, there are one blue point and red point misclassified. To solve this, one logic approach would be to make the line closer to the misclassified points (located in the blue region) and hence change its label.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.25\textwidth,height=0.25\textheight,keepaspectratio]{images/split_data.png}
    \captionsetup{justification=centering}
    \caption{Splitting the data example}
    \label{fig:f13}
\end{figure}

Now imagine the line plotted in Fig. \ref{fig:f14}, where the line equation is given by \(3x_1+ 4x_2 - 10 = 0\) and out labels are: blue area if \(\hat{y} > 0\) or red area if \(\hat{y} < 0\). Suppose that the line has coordinates \((4,5)\), based on the approach described above, the point coordinates should be subtracted to the line equation as shown by the equation below, where \(x_1 = 4, x_2 = 5 \text{ and } b = 1\).

\[
\begin{bmatrix}  \text{new } w_1 \\
                 \text{new } w_2 \\
                \text{new } b \end{bmatrix} = \begin{bmatrix}
                                    3 - 4 \\
                                    4 - 5 \\
                                    -10 - 1
                                    \end{bmatrix} = \begin{bmatrix}
                                                    -1 \\
                                                    -1 \\
                                                    -11
                                                    \end{bmatrix}
\]

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.25\textwidth,height=0.25\textheight,keepaspectratio]{images/moving_line.png}
    \captionsetup{justification=centering}
    \caption{Making the line closer example}
    \label{fig:f14}
\end{figure}

With the new parameters, the equation will be \(-1x_1+ 4-1x_2 - 11 = 0\). Nevertheless, this might be a very drastic change in the equation, leading to misclassification of other well-classified points. The idea is to make the line take small moves towards the misclassified point. To solve this, it is neccesary to take small steps towards the point, therefore, a new concept needs to be introduced. This concepts is called \(\alpha\) or \textbf{learning rate}. The learning rate is a small number between zero and one and its function is to minimize the change in the model equation by multiplying its value to the values of the point. E.g, using a learning rate of \(0.1\) and the same example as before, the result will be the shown in the equation and Fig. \ref{fig:f15}

\[
\begin{bmatrix}  \text{new } w_1 \\
                 \text{new } w_2 \\
                \text{new } b \end{bmatrix} = \begin{bmatrix}
                                    3 - 4*(0.1) \\
                                    4 - 5*(0.1) \\
                                    -10 - 1*(0.1)
                                    \end{bmatrix} = \begin{bmatrix}
                                                    2.6 \\
                                                    3.5 \\
                                                    -10.1
                                                    \end{bmatrix}
\]

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.25\textwidth,height=0.25\textheight,keepaspectratio]{images/new_line.png}
    \captionsetup{justification=centering}
    \caption{Moving the line closer to the point}
    \label{fig:f15}
\end{figure}

Similarly, if the misclassified point is located in the red area, instead of subtracting the point to the model equation it needs to be added. As a thumb rule, the \eqref{eq:7} shows the process of adding or substracting based on the point's location.

\begin{equation}
\label{eq:7}
  \begin{cases}
    Substract, & \text{if } \text{point is located in region } > 0 \\
    Add, & \text{if } \text{point is located in region } < 0 \\
  \end{cases}
\end{equation}

Based on the technique given above, it is possible to determine the perceptron algorithm. This one is presented in the Fig. \ref{fig:f16} and shows and iterative process that tries to classify correctly all the misclassified points. The process works iteratively and can run until all the points are correctly classified or a minimum of misclassified points are obtained. A code example of this is given in \textit{introduction/perceptron\_algorithm.py}

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.2\textwidth,height=0.2\textheight,keepaspectratio]{images/perceotron_algorithm.png}
    \captionsetup{justification=centering}
    \caption{The perceptron Pseudocode}
    \label{fig:f16}
\end{figure}

Let's look more carefully at the model presented in Fig. \ref{fig:f3} for accepting and rejecting students. Imagine that the student four got nine in the test but only one on the grades. According to the model this student gets accepted since it's placed in the positive region of this line. Nevertheless, this is not accurate. A more precise model is presented in Fig. \ref{fig:f7}, however, this is non-linear separable. To create an accurate model a different boundary should be used, two lines, a circle or a polynomial function are just few examples of this. Unfortunately, \textbf{the perceptron algorithm won't work well for non-linear problems}. Therefore, something more complex needs to be applied to redefine the simple perceptron algorithm for a line in a way that it'll generalize to other types of curves. Nevertheless, this problem will be tackle later.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.35\textwidth,height=0.35\textheight,keepaspectratio]{images/non_linear.png}
    \captionsetup{justification=centering}
    \caption{Non-linear area}
    \label{fig:f17}
\end{figure}

To tackle Deep learning problems, it is necessary to introduce the concept of \textbf{Error function}. This is simply something that tells how far is the current model from the solution. An example of a simple error function used in the last example would be a function that tells how far is a point from its good classification region (distance).

\textbf{Gradient descent} is a method to reduce the error of a problem and find the global optimum using known metrics. This algorithm calculates the error and then takes a step towards the direction that minimizes the error in a problem. As an example for the Fig. \ref{fig:f13} it is possible to construct an error function which consists of the sum of the distance of the points to the boundary lines, where the misclassified points have a big penalty while the well classifies have a low penalty. This function can be used to guide the boundary line towards a direction that minimizes the error and thus, find the optimal model to solve the problem. \textbf{Note:} the error function should be differentiable and continuous.

Recall that a discrete prediction might be something like a 1 or zero, whereas a continuous prediction would be a number, normally between zero and one. As an example, see the  Fig. \ref{fig:f18} where we have students with discrete predictions (left) and continuous predictions(right). In the discrete plot, the algorithm will tell if a student is accepted (1 - blue) or rejected (0 - red). In the continuous plot, the farther the point is from the black line, the more drastic these probabilities are. Points that are well into the blue area get very high probabilities And points that are well into the red region are given very low probabilities. The points over the line are all given a 50\% probability. The probability is a function of the distance from the line.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.5\textwidth,height=0.5\textheight,keepaspectratio]{images/discrete_continuous.png}
    \captionsetup{justification=centering}
    \caption{Discrete and continuous predictions}
    \label{fig:f18}
\end{figure}

To change from discrete to continuous predictions it is necessary to use a \textbf{Sigmoid function} (Fig. \ref{fig:f19}) instead of a unitary step as the activation function. This activation will guarantee a continuous functions instead of a discrete one, allowing the usage of gradient descent techniques. The equation of the Sigmoid function is given in the Equation \eqref{eq:8}. Before the model consisted of a line with a positive region and a negative region. With the Sigmoid function now it consists of an entire probability space for each point in the plane.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.35\textwidth,height=0.35\textheight,keepaspectratio]{images/sigmoid.png}
    \captionsetup{justification=centering}
    \caption{Sigmoid function}
    \label{fig:f19}
\end{figure}

\begin{equation}
\label{eq:8}
\sigma(x) = \frac{1}{1 - e^{-x}}
\end{equation}

Using the Sigmoid function as the activation function to predict \(\hat{y}\), the output of the perceptron will be as follows Eq. \eqref{eq:9}. The simple perceptron seen in Fig. \ref{fig:f8} with a step function can be modified as shown by Fig. \ref{fig:f20} to work with a Sigmoid function as the activation function.  Before it used to say the student got accepted or not, with the Sigmoid function it says the probability of the student got accepted is this much.

\begin{equation}
\label{eq:9}
\hat{y} = \sigma(Wx + b)
\end{equation}

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.30\textwidth,height=0.30\textheight,keepaspectratio]{images/perceptron_sigmoid.png}
    \captionsetup{justification=centering}
    \caption{Simple perceptron with a sigmoid activation function}
    \label{fig:f20}
\end{figure}

\textbf{The softmax function} is used when the problem has more than two classes and gives a probability of zero or one being zero as "not the class" and one as "object of this class". This is the equivalent of the sigmoid activation function, but when the problem has 3 or more classes. It is important to point out that the probability of all classes must add to one and there could not be any negative probability. To define the Softmax function let's imagine that there are N classes and a linear model that gives the following scores \( Z_1, Z_2, \dots Z_n\), each score for each of the classes in the problem. The softmax probability of the class \(Z_i\) is defined by the equation \eqref{eq:10}.

\begin{equation}
\label{eq:10}
P(\text{class i}) = \frac{e^{Z_i}}{e^{Z_i} + \dots + e^{Z_n}}  = \frac{e^{Z_i}}{\sum_k^N e^{Z_k}}
\end{equation}

When there are multiple output classes, it is necessary to use a specific output format called \textbf{one-hot encoding}. This format assign one variable per class and outputs a one in the specific class that is selected.  Imagine that the classes are Duck, Beaver and Walrus. Here it is necessary to assign one variable for each of the classes. In the Fig. \ref{fig:f21} That's one variable for Duck, one for Beaver and one for Walrus And, each one has its corresponding column. E.g., if the input is a duck then the variable for duck is 1 and the variables for beaver and walrus are 0.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.45\textwidth,height=0.45\textheight,keepaspectratio]{images/one_hot.png}
    \captionsetup{justification=centering}
    \caption{One-hot encoding}
    \label{fig:f21}
\end{figure}

\textbf{Maximum likelihood} is when the model picked gives the existing labels the highest probability. Thus, by maximizing the probability, it is possible to pick the best possible model. To use an example of how maximum likelihood works, see the model presented in Fig. \ref{fig:f22}, where the value presented in each dot is the probability of being blue or red (determined by the color of the number). To determine which model is better, it is necessary to compute the probability of the four points, this probability is called the \textbf{probability space} and will tell if the points are of the colors that they actually are. For the left model, the probability space is \(0.1 x 0.6 x 0.7 x 0.2 = 0.0084\) which is a very small value, whereas for the right model the probability space is \(0.7 x 0.8 x 0.9 x 0.6 = 0.3024\). This value means that the probability that the points of the left model are of these colors is 0.0084 and the right model is 0.3024, therefore, it confirms that the model on the right is better because it makes the arrangement of the points much more likely to have those colors. If the likelihood or probability space of the model is improved, hence the model will be better/

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.45\textwidth,height=0.45\textheight,keepaspectratio]{images/likelihood.png}
    \captionsetup{justification=centering}
    \caption{Maximum likelihood example}
    \label{fig:f22}
\end{figure}

Maximizing the probability is equivalent to minimizing the error function and thus, improve the model. To start with this, it is important to transform the products of the probability of the whole arrangement into a sum, this is possible by passing the individual probabilities in a logarithmic function owing to \(\ln{ab} = \ln{a} + \ln{b}\). Applying logarithms to the example shown in Fig. \ref{fig:f22}, and taking into account that the logarithm of a number between zero and one is always negative, it is possible to use the sum of the negatives of the logarithms instead of a positive sum. Therefore, the model of the left gives \(\ln{0.6} - \ln{0.2} - \ln{0.1} - \ln{0.7} = 4.8\) ad \(\ln{0.8} - \ln{0.6} - \ln{0.7} - \ln{0.9} = 1.2\). This concepts of the negative sum of logarithms is called \textbf{Cross-entropy} and it gives a high value for bad models (left) and low values for good models (right) which is a measurement of how good the model is.

Using the Cross Entropy concept in the last example the result can be seen in Fig. \ref{fig:f23}. The Cross-Entropy of misclassified points have really high values whereas the well classified points have low values. Thus it is possible to think of the negatives of these logarithms as errors at each point. Points that are correctly classified will have small errors and points that are misclassified will have large errors. With this it is feasible to concluded that the cross-entropy will tell us if a model is good or bad. The goal has changed from maximizing a probability to minimizing a cross-entropy in order to get from the model in left to the model in the right in Fig. \ref{fig:f23}. \textbf{The error-function is the cross-entropy}.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.45\textwidth,height=0.45\textheight,keepaspectratio]{images/cross_entrophy.png}
    \captionsetup{justification=centering}
    \caption{Using Cross Entropy example}
    \label{fig:f23}
\end{figure}

The formula of the Cross-entropy is given in Eq. \eqref{eq:11} where \(y_i\) is the true label of the class being 1 for blue and 0 for red (see Fig. \ref{fig:f23}). If \(y_i\) is one, then the probability is \(p_i\), otherwise the probability will be \((1 - p_i)\) which is the probability for \(y_i = 0\).

\begin{equation}
\label{eq:11}
Cross-Entropy = - \sum_{i = 1}^m y_i \ln{p_i} + (1 - y_i) \ln{(1 - p_i)}
\end{equation}

When there are more than two classes, the equation of cross-entropy has to change and find a different expressions to compute these probabilities. In the Fig. \ref{fig:f24} it is possible to see an example of this. There are three different animal classes and three different "doors". Recall that the probability of each column or door must add up to 1. E.g, the probability to obtain a duck from the door two is equals to \(P_{12}\).

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.3\textwidth,height=0.3\textheight,keepaspectratio]{images/multiple_classes.png}
    \captionsetup{justification=centering}
    \caption{Multiple classes example}
    \label{fig:f24}
\end{figure}

Expressing this mathematically as shown by Fig. \ref{fig:f25} the output \(y_{ij}\) is one if a specific class is obtained from the door \(j\). Therefore, the equation for \textbf{multi-class entropy} is given by Eq. \eqref{eq:12} where m is the number of classes.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.3\textwidth,height=0.3\textheight,keepaspectratio]{images/multiple_notation.png}
    \captionsetup{justification=centering}
    \caption{Multiple classes notation}
    \label{fig:f25}
\end{figure}

\begin{equation}
\label{eq:12}
Cross-Entropy = - \sum_{i = 1}^m \sum_{j=1}^n y_{ij}  \ln{p_{ij}}
\end{equation}

Recall the Fig. \ref{fig:f23} where there are two models and the cross-entrophy was calculated. The cross-entropy calculation gave 4.8 for the left model and 1.2 for the right model. The expression given in Eq \eqref{eq:13} can be used as an error given the following data. Consider that if \(y = 1\) which is blue class then \(P(blue) = \hat{y}\), therefore \(Error = -\ln{(\hat{y})}\). Now, analizing the red classification if \(y = 0\) (red class) then \(P(red) = 1 - P(blue) = 1 - \hat{y}\), thus the error for the red class would be \(Error = -\ln{(1 - \hat{y})}\).

The last analizis leads to the next Error expression.

\begin{equation}
\label{eq:13}
Error = -(1 - y)(\ln{(1 - \hat{y})} - y\ln{(\hat{y})}
\end{equation}

The expression given in Eq \eqref{eq:13} can be used as an error, so changing the notation the error function and averaging by m gives the Equation in \eqref{eq:14}.

\begin{equation}
\label{eq:14}
Error = - \frac{1}{m} \sum_{i=1}^m (1 - y_i) (\ln{(1 - \hat{y})}) + y_i\ln{(\hat{y})}
\end{equation}

Where \(\hat{y}\) is the output of the network. Transforming \(\hat{y}\) in terms of Eq. \eqref{eq:9}, the Eq. \eqref{eq:14} will be as follows (Eq. \eqref{eq:15}).

\begin{equation}
\label{eq:15}
E(W,b) = - \frac{1}{m} \sum_{i=1}^m (1 - y_i) (\ln{(1 - \sigma(Wx^{(i)} + b))}) + y_i\ln{(\sigma(Wx^{(i)} + b))}
\end{equation}

The goal is to minimize the Error function given by Eq. \eqref{eq:15}. Now if there are more than two classes, the only change would be to add the additional summation as shown by Eq. \eqref{eq:12} to the error function. The Error function for more than one class is given as follows Eq. \eqref{eq:16}.

\begin{equation}
\label{eq:16}
E(W,b) = - \frac{1}{m} \sum_{i=1}^m \sum_{j=1}^n y_{ij}\ln{(\sigma(Wx^{(ij)} + b))}
\end{equation}

Minimizing this error leads to an algorithm called \textbf{Logistic regression } which is very popular in Machine Learning and AI. The procedure of this algorithm is given as follows:

\begin{itemize}
  \item Take the data
  \item Pick a random model
  \item Calculate the error
  \item Minimize the error, and obtain a better model
\end{itemize}

Now, to accomplish the last step of the items and minimize the error it is necessary to use the gradient descent algorithm which will allow the minimization of the error and pass from \(E(W,b)\) to \(E(W',b')\) where \(E(W',b') < E(W,b)\).

Imagine a person somewhere in a Mount where the objective is to go down Fig. \ref{fig:f26}. The inputs of the functions are \(W_1\) and \(W_2\), the error function is given by E. The gradient of E (\(\nabla E\)) is given by the vector sum of the partial derivatives \(\frac{\partial E}{\partial w_1}\) and \(\frac{\partial E}{\partial w_2}\). This gradient actually tells the direction the person has to move if he want to increase the error function the most (go up). Thus, taking the negative of the gradient,leads to decrease the error function the most. Taking the negative direction of the gradient will eventually take the person to the bottom of the mountain and decrease the error to a minimum expression. Additinally, taking steps in the direction of the negative of the gradient will also change the Weights and Bias values. This is basically the \textbf{gradient descent} algorithm.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.45\textwidth,height=0.45\textheight,keepaspectratio]{images/gradient.png}
    \captionsetup{justification=centering}
    \caption{Gradient descent example}
    \label{fig:f26}
\end{figure}

To calculate the gradient the next equations are used. First of all the initial prediction is given by the next equation.

\[\hat{y} = \sigma(Wx + b)\]

Writing the prediction in terms of the vectors W and B gives.

\[\hat{y} = \sigma(W_1X_1 + \dots + W_nX_n + b)\]

The gradient of the error function is the vector formed by the partial derivative of the error function with respect to the weights and the bias.

\[\nabla E = (\frac{\partial E}{\partial W_1}, \dots, \frac{\partial E}{\partial W_2}, \frac{\partial E}{\partial b})\]

To avoid any drastic changes, a small learning rate alpha is introduced.

\[\alpha = 0.1\]

Multiplying the gradient by the learning rate \(\alpha\) and taking a step is the negative direction of the gradient is the same thing as updating the weights and the bias as follows, where the weight \(W_i\) will now become \(W_i'\)

\[w_i' \leftarrow w_i - \alpha \frac{\partial E}{\partial W_i}\]

The bias will now become:

\[b_i' \leftarrow b - \alpha \frac{\partial E}{\partial b}\]

The full derivation of the gradient descent is as follows. First of all, it is necessary to compute the derivative of the sigmoid function (supposing that this is our activation function) which is Eq. \eqref{eq:17}

\begin{equation}
\label{eq:17}
\sigma'(x) = \sigma(x)(1 - \sigma(x))
\end{equation}

The demonstration of the last result is shown below.

\[\sigma'(x) = \frac{\partial}{\partial x} \frac{1}{1 + e ^{-x}}\]
\[\sigma'(x) = \frac{e ^{-x}}{(1 + e ^{-x})^2}\]
\[\sigma'(x) = \frac{1}{1 + e ^{-x}} \frac{e ^{-x}}{1 + e ^{-x}}\]
\[\sigma'(x) = \sigma(x)(1 - \sigma(x)) \]

Recall the Fig. \ref{fig:f23} where there are m points labeled \(X^{(1)}, X^{(2)}, \dots, X^{(m)} \), thus the error equation would be Eq. \eqref{eq:14}

\[E = - \frac{1}{m} \sum_{i=1}^m y_i\ln{(\hat{y})} + (1 - y_i) (\ln{(1 - \hat{y})})\]

The goal is to calculate \(\nabla E\), at a point \(x = (x_1, x_2,\dots,x_n)\)), given by the partial derivatives

\[\nabla E = \frac{\partial E}{\partial w_1}, \dots, \frac{\partial E}{\partial w_n}, \frac{\partial E}{\partial b}\]

To simplify the calculations, think of the error that each point produces, and calculate the derivative of this error. The total error, then, is the average of the errors at all the points. The error produced by each point is:

\[ E = -y_i\ln{(\hat{y})} - (1 - y_i) (\ln{(1 - \hat{y})})\]

In order to calculate the derivative of this error with respect to the weights, first calculate \(\frac{\partial \hat{y}}{\partial w_j}\), recalling that \(\hat{y} = \sigma(Wx + b) = \sigma(\theta)\). The

\[\frac{\partial \hat{y}}{\partial w_j} = \frac{\partial}{\partial w_j} \sigma(Wx + b) \]
\[\frac{\partial \hat{y}}{\partial w_j} = \sigma(Wx + b)(1 - \sigma(Wx + b)) \frac{\partial}{\partial w_j}Wx + b \]
\[\frac{\partial \hat{y}}{\partial w_j} = \hat{y}(1 - \hat{y}) \frac{\partial}{\partial w_j}(w_1x_1+\dots+w_ix_i+\dots+w_nx_n, + b) \]
\[\frac{\partial \hat{y}}{\partial w_j} = \hat{y}(1 - \hat{y}) x_i \]

The last equality is because the only term in the sum which is not a constant with respect to \(w_j\) is \( w_jx_j\) which clearly has derivative \(x_j\).

Reall that the gradient descent is based on the chain rule, which would be the one described in Eq. \eqref{eq:18} where \(\theta = Wx + b\)

\begin{equation}
\label{eq:18}
\frac{\partial E}{\partial w_j} = \frac{\partial E}{\partial \hat{y}} \frac{\partial \hat{y}}{\partial w_j} =  \frac{\partial E}{\partial \hat{y}} \frac{\partial \sigma(\theta)}{\partial \theta} \frac{\partial \theta}{\partial w_j}
\end{equation}

Now, To calculate the derivative of the error \(E\) at a point \(x\) with respect to the weight \(w_j\) is as follows.

\[\frac{\partial E}{\partial w_j} = \frac{\partial}{\partial w_j} [-y\ln{(\hat{y})} - (1 - y) (\ln{(1 - \hat{y})})] \]
\[\frac{\partial E}{\partial w_j} = -y \frac{\partial}{\partial w_j} \ln{(\hat{y})} - (1 - y) \frac{\partial}{\partial w_j} \ln{(1 - \hat{y})} \]
\[\frac{\partial E}{\partial w_j} = -y \frac{1}{\hat{y}} \frac{\partial}{\partial w_i} \hat{y} - (1 - y) \frac{1}{1 - \hat{y}} \frac{\partial}{\partial w_i} (1 - \hat{y}) \]
\[\frac{\partial E}{\partial w_j} = -y \frac{1}{\hat{y}} \hat{y}(1 - \hat{y}) x_i - (1 - y) \frac{1}{1 - \hat{y}} (-1)(\hat{y}(1 - \hat{y}) x_i) \]
\[\frac{\partial E}{\partial w_j} = -y (1 - \hat{y}) x_i + (1 - y)\hat{y}x_i \]
\[\frac{\partial E}{\partial w_j} = -(y - \hat{y})x_i \]

The Eq. \eqref{eq:19} gives the gradient of the weights.

\begin{equation}
\label{eq:19}
\frac{\partial E}{\partial w_j} = -(y - \hat{y})x_i
\end{equation}

Similarly, the result for the bias is Eq. \eqref{eq:20}

\[ \frac{\partial E}{\partial b} = \frac{\partial E}{\partial \hat{y}} \frac{\partial \hat{y}}{\partial b} =  \frac{\partial E}{\partial \hat{y}} \frac{\partial \sigma(\theta)}{\partial \theta} \frac{\partial \theta}{\partial b} \]

\begin{equation}
\label{eq:20}
\frac{\partial E}{\partial b} = -(y - \hat{y})
\end{equation}

In summary the gradient is:

\[\nabla E = -(y - \hat{y})(x_1,\dots ,x_n,1)\]

Therefore, a small gradient means that it changes the coordinates by a little bit, and a large gradient means it changes the coordinates by a lot.

\textbf{The gradient descent step} rule to update the weights is as follows Eq. \eqref{eq:21}

\begin{equation}
\label{eq:21}
w_i' \leftarrow w_i - \alpha-(y - \hat{y})x_i = w_i + \alpha(y - \hat{y})x_i
\end{equation}

To update the bias the equation is the next:

\begin{equation}
\label{eq:22}
b' \leftarrow b + \alpha(y - \hat{y})
\end{equation}

Note: Since the average of the errors were taken, the term added should be \(\frac{1}{m}*\alpha\) instead of \(\alpha\) but as \(\alpha\) is a constant, then in order to simplify calculations, it is possible to take \(\frac{1}{m}*\alpha\) to be the learning rate, and abuse the notation by just calling it \(\alpha\).

The gradient descent algorithm is given in the Fig. \ref{fig:f27}.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.45\textwidth,height=0.45\textheight,keepaspectratio]{images/gradient_algorithm.png}
    \captionsetup{justification=centering}
    \caption{Gradient descent algorithm}
    \label{fig:f27}
\end{figure}

Recall the problem presented in Fig. \ref{fig:f17}, where the problem was not linear separable. To tackle this problem it is necessary to generate a non-linear probability distribution which accurately classified the blue and red points.
\textbf{Nonlinear models} can be generated by the combination of two linear models. To make it possible, see the Fig. \ref{fig:f28} where in the left are the linear models to be combined, in the middle is the computation of one single point of the model and finally, the result after the application of the sigmoid function.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.6\textwidth,height=0.6\textheight,keepaspectratio]{images/adding_perceptrons.png}
    \captionsetup{justification=centering}
    \caption{Adding two perceptrons}
    \label{fig:f28}
\end{figure}

Imagine taking the point highlighted in the Fig. \ref{fig:f28}, which has a probability of \(0.7\) in the first linear model and \(0.8\) in the second model. Adding some weights to this model is possible and hence, modify its influence on the result. In this particular example, the first model has a weight of \(7\) and the second of \(5\). Therefore, the linear equation for the point highlighted gives the next linear combination \(7*0.7 + 5*0.8 - 6 = 2.9\), where \(b = -0.6\) and \(P(x_i) = 2.9\) is the probability of that point being blue. The final step is apply the sigmoid function to approximate the probability to 1, with this the final result of the point is \(0.95\) which is the probability of being blue in the resulting probability space.

The overall steps to generate nonlinear models with linear ones is:

\begin{itemize}
  \item Calculate the probability for one of the points
  \item calculate the probability for the other models
  \item Add the probabilities times its weights and bias
  \item Apply the sigmoid function
\end{itemize}

This might result very simple but this is at the heart of how neural networks work.

The following image (Fig. \ref{fig:f29}) shows the representation of the example in Fig. \ref{fig:f28} but as a combination of nodes and weights. This representation clearly shows how the perceptrons are combined to produce and new non-linear output.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.25\textwidth,height=0.25\textheight,keepaspectratio]{images/adding_perceptrons_2.png}
    \captionsetup{justification=centering}
    \caption{The perceptrons as a neural network}
    \label{fig:f29}
\end{figure}

The Fig. \ref{fig:f30} is an even more simplified representation of Fig. \ref{fig:f29} where all the input nodes connects with the middle nodes and these connects to a unique output. Additionally, this representation has the bias in each layer and shows the activation function needed (sigmoid) to obtain a non-linear function in the input.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.4\textwidth,height=0.4\textheight,keepaspectratio]{images/adding_perceptrons_3.png}
    \captionsetup{justification=centering}
    \caption{The simplified representation of multiple perceptrons in a neural network}
    \label{fig:f30}
\end{figure}

Neural networks have a certain special \textbf{architecture} with layers. The first layer is called the input layer, which contains the inputs. The next layer is called the hidden layer, which is a set of linear models created with this first input layer. Then the final layer is called the output layer, where the linear models get combined to obtain a nonlinear model. It is possible to create different architectures with various inputs, hidden layers or outputs.

\textbf{The number of input} noted defined the dimension of the problem. Two input nodes turn the second layer into a two-dimensional representation whereas three inputs generates three dimensional representations or planes in the hidden layer.

\textbf{Adding hidden layers} increase the amount of representations that can be generated with the neural network and create more complex boundaries than with only two layers.

Finally, \textbf{adding more outputs} creates multiple categories and opens the possiblity to classify different things like cat, dogs or birds. \textbf{Note:} if the model has multiple outputs, it is necessary to change the activation in the last function for a Softmax activation function and add as many outputs as classes are present in the problem. This activation function will guarantee a probability distribution in the last layer and will show which of the classes is the selected one with the highest probability.

\textbf{Feedforward} is the process neural networks use to turn the input into an output, passing the whole data throughout the network. In the Fig. \ref{fig:f31} is an example of this with a network with two input layers, two hidden units and one output. The Weights \(W^{1}\) corresponds to the weights between the input layer and the hidden layer, for a matter of simplicity, the bias was added to this set of weights. Similarly, the weights \(W^{2}\) are between the hidden layer and the output layer. Mathematically, the feedforward process can be expressed as shown by Eq. \eqref{eq:23}. \textbf{Note:} the bias are now denoted as \(W_{31}^{(1)}, W_{32}^{(1)}, W_{31}^{(2)}\)

\begin{equation}
\label{eq:23}
\hat{y} = \sigma(\sigma (\begin{bmatrix}
           x_{1} \\
           x_{2} \\
           1
         \end{bmatrix} \cdot \begin{pmatrix}
                                W_{11}^{(1)} & W_{12}^{(1)} \\
                                W_{21}^{(1)} & W_{22}^{(1)} \\
                                W_{31}^{(1)} & W_{32}^{(1)}
                                \end{pmatrix} ) \cdot \begin{bmatrix}
                                                           W_{11}^{(2)} \\
                                                           W_{21}^{(2)} \\
                                                           W_{31}^{(2)}
                                                         \end{bmatrix})
\end{equation}

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.4\textwidth,height=0.4\textheight,keepaspectratio]{images/feedforward.png}
    \captionsetup{justification=centering}
    \caption{Feedforward in a neural network}
    \label{fig:f31}
\end{figure}

Equation \eqref{eq:23} can be simplied into Eq. \eqref{eq:24}. As a rule of thumb, feedforward takes the input vector and then apply a sequence of linear models and sigmoid functions or whatever activation function is used in the network. The feedforward process is used to obtain the prediction from the input vector. These steps can be extended into more deep networks as the procedure is always the same.

\begin{equation}
\label{eq:24}
\hat{y} = \sigma(\sigma(W^{(1)} \cdot X) \cdot W^{(2)})
\end{equation}

The error function for a multi-layer perceptron is exactly the same used for the perceptron algorithm which is Eq. \eqref{eq:14} or the Cross-entropy. Similarly, \(\hat{y} = \sigma(Wx + b)\) but for a neural netowrk with multiple layers, the only difference is that \(\hat{y}\) is more complex and is a combination of multiplications and sigmoid functions as shown here \[\hat{y} = \sigma W^{(3)} \cdot \sigma W^{(2)} \cdot \sigma W^{(1)} \cdot X \]

\textbf{Backpropagation} is the method used to train multilayer perceptrons or deep neural networks. At a glance, this is the opposite process of feeding forward a neural network, to propagate the error backwards and correct the value of the weights towards a good model or solution. The steps for this algorithm are:

\begin{itemize}
  \item Doing a feedforward operation.
  \item Comparing the output of the model with the desired output.
  \item Calculating the error.
  \item Running the feedforward operation backwards (backpropagation) to spread the error to each of the weights.
  \item Use this to update the weights, and get a better model.
  \item Continue this until we have a model that is good.
\end{itemize}

To start, recall that for the perceptron algorithm, the prediction was given by Eq. \eqref{eq:9}, the error is the cross-entropy (Eq. \eqref{eq:14}) and the objective is to calculate the gradient of the error function \(\nabla E = \frac{\partial E}{\partial w_1}, \dots, \frac{\partial E}{\partial w_n}, \frac{\partial E}{\partial b}\).

For a multilayer perceptron as the one presented in Fig. \ref{fig:f31}, The prediction is given by Eq. \eqref{eq:25}, the error function is exactly the same as in the perceptron algorithm Eq. \eqref{eq:14} and the gradients of the error are the same but with way more calculations Eq. \eqref{eq:26}.

\begin{equation}
\label{eq:25}
\hat{y} = \sigma W^{(3)} \cdot \sigma W^{(2)} \cdot \sigma W^{(1)} \cdot X
\end{equation}

\begin{equation}
\label{eq:26}
\nabla E = (\dots, \frac{\partial E}{\partial w_j^i}, \dots)
\end{equation}

Writing this more formally, the prediction is a composition of matrix multiplications and sigmoid functions.

\[\hat{y} = \sigma W^{(2)} \cdot \sigma W^{(1)} \cdot x\]

Where:

\[ W^{(1)} = \begin{pmatrix}
                                W_{11}^{(1)} & W_{12}^{(1)} \\
                                W_{21}^{(1)} & W_{22}^{(1)} \\
                                W_{31}^{(1)} & W_{32}^{(1)}
                                \end{pmatrix} \]

\[ W^{(2)} = \begin{bmatrix}
               W_{11}^{(2)} \\
               W_{21}^{(2)} \\
               W_{31}^{(2)}
             \end{bmatrix}\]

With a gradient that can be formalized as (in reality \(\nabla E\) is a long vector:

\[ \nabla E = \begin{pmatrix}
                                \frac{\partial E}{\partial W_{11}^{(1)}} & \frac{\partial E}{\partial W_{12}^{(1)}} & \frac{\partial E}{\partial W_{11}^{(2)}} \\
                                \frac{\partial E}{\partial W_{21}^{(1)}} & \frac{\partial E}{\partial W_{22}^{(1)}} & \frac{\partial E}{\partial W_{11}^{(2)}} \\
                                \frac{\partial E}{\partial W_{31}^{(1)}} & \frac{\partial E}{\partial W_{32}^{(1)}} & \frac{\partial E}{\partial W_{11}^{(2)}}
                                \end{pmatrix} \]

The update step for the backpropagation would be Eq. \eqref{eq:27}:

\begin{equation}
\label{eq:27}
W_{ij}^{'k} \leftarrow W_{ij}^{k} + \alpha \frac{\partial E}{\partial W_{ij}^{(k)}}
\end{equation}

At this point is important to recall the chain rule, which is presented in the Fig. \ref{fig:f32}.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.5\textwidth,height=0.5\textheight,keepaspectratio]{images/chain_rule.png}
    \captionsetup{justification=centering}
    \caption{The chain rule}
    \label{fig:f32}
\end{figure}

\[\frac{\partial B}{\partial x} = \frac{\partial B}{\partial A} \frac{\partial A}{\partial x}\]

To start with the calculation of the backgropagation algorithm, the first step is to do a forward pass. The hidden nodes and the output nodes have been labeled \(h_1, h_2, h\) respectively Fig. \ref{fig:f33}. The equation of these nodes is given as follows.

\[h_1 = W_{11}^{(1)}x_1 +  W_{21}^{(1)}x_2 + W_{31}^{(1)}\]
\[h_2 = W_{12}^{(1)}x_1 +  W_{22}^{(1)}x_2 + W_{32}^{(1)}\]
\[h = W_{11}^{(2)}\sigma(h_1) + W_{21}^{(2)}\sigma(h_2) + + W_{31}^{(2)}\]

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.4\textwidth,height=0.4\textheight,keepaspectratio]{images/feedforward_pass.png}
    \captionsetup{justification=centering}
    \caption{A feedforward pass in a MLP}
    \label{fig:f33}
\end{figure}

Similarly, the prediction is:

\[\hat{y} = \sigma(h)\]

\[\hat{y} = \sigma W^{(2)} \cdot \sigma W^{(1)} \cdot x\]

To backpropagate the error it is neccesary to calculate the gradients of the error function Eq. \eqref{eq:14} \(E(W) = - \frac{1}{m} \sum_{i=1}^m y_i\ln{(\hat{y})} + (1 - y_i) (\ln{(1 - \hat{y})})\) using the chain rule.

Since the error function is function of the prediction and the prediction is function of the weights the error function can be written as a function of the weights \(W_{ij}^{k}\) as follows:

\[E(W) = E(W_{11}^{(1)}, W_{12}^{(1)}, \dots, W_{31}^{(2)})\]

Therefore, the gradient is the vector formed by all the partial derivatives of the error function E with respect to each of the weights.

\[\nabla E = (\frac{\partial E}{\partial W_{11}^{(1)}}, \dots, \frac{\partial E}{\partial W_{31}^{(2)}})\]

With this information, it is possible to do an example and calculate the partial derivative of the error with respect to \(W_{11}^{(1)}\), this partial derivative is given by the next equation Eq. \eqref{eq:28}.

\begin{equation}
\label{eq:28}
\frac{\partial E}{\partial W_{11}^{(1)}} = \frac{\partial E}{\partial \hat{y}} \frac{\partial \hat{y}}{\partial h} \frac{\partial h}{\partial h_1} \frac{\partial h_1}{\partial W_{11}^{(1)}}
\end{equation}

Where most of the partial derivatives are already know from the perceptron algorithm Eq. \eqref{eq:19}, the only new term is: \(\frac{\partial h}{\partial h_1} \). The partial derivative of this term is giving by Eq. \eqref{eq:29} and is based on the partial derivative of the sigmoid function in Eq. \eqref{eq:17}.

\begin{equation}
\label{eq:29}
\frac{\partial h}{\partial h_1} = W_{11}^{(2)} \sigma(h_1)(1 - \sigma(h_1))
\end{equation}

This process has to be repeated for each of the weights in the neural network but it is mainly the same. If the activation function is changed, then the derivative will change. Nevertheless, the process still being the same and the gradient descent algorithm can be easy applied to different neural networks with multiple layers.

Now that the backpropagtion algorithm has been introduced, it is time to talk about testing. \textbf{Testing} is the process of evaluating a model with two different tests, the training set and the test set. In the Fig. \ref{fig:f34} the solid color points are the training set and the points with the white inside are the testing set. The training set is used to train the model without looking the test set and once the model is completely trained, the test set is used to evaluate how good the model is actually doing.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.4\textwidth,height=0.4\textheight,keepaspectratio]{images/testing.png}
    \captionsetup{justification=centering}
    \caption{Splitting the data into training and test sets}
    \label{fig:f34}
\end{figure}

Oversimplifying the problem means trying a solution that is too simple and won't do the job. In machine learning, this is called \textbf{underfitting}. On the other hand, when a solution is overly complicated it will probably leads to bad solutions and extra complexity when a simpler solution was just the answer. In machine learning, this is called \textbf{overfitting}. Generally, an overfitting model fits the data well but it fails when generalizing (error due to variance), whereas an underfitting model generalizes too much and tends to get the dataset bad (error due to bias).

In the Fig. \ref{fig:f35} can be seen an example of how a underfitting, just right and overfitting model looks like.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.6\textwidth,height=0.6\textheight,keepaspectratio]{images/under_over.png}
    \captionsetup{justification=centering}
    \caption{Underfitting and Overfitting models}
    \label{fig:f35}
\end{figure}

To try to solve this problem, it is better to always try to go for a slightly overfitting model. The idea is to approach as close as possible to the solution and then use some techniques to avoid overfitting.

One technique to solve the overfitting issue is called \textbf{early stopping}, and this is presented in Fig. \ref{fig:f36}. The plot is called the model complexity graph. In the Y-axis, there is a measure of the error and in the X-axis is a measure of the complexity of the model. In this case, the measure of complexity is the number of epochs. It is possible to see in the figure that in the left are high testing and training error, hence the model is underfitting. In the right, there are a high testing error and low training error, therefore the model is overfitting. Nevertheless in the middle is the right spot to obtain the best train and test error, the main idea of this plot is to determines the number of epochs needed.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.5\textwidth,height=0.5\textheight,keepaspectratio]{images/early.png}
    \captionsetup{justification=centering}
    \caption{Early stopping}
    \label{fig:f36}
\end{figure}

Another technique to stop overfitting is called \textbf{regularization}. To explain it, first see the Fig. \ref{fig:f37} which presents two models with exactly the same boundary line. The only difference is that the weights of the model in the right are bigger. This difference in the weights will create a bigger estimate value which being pass through the activation function (sigmoid) will result in values very close to zero and one. However, the left model creates values between zero and one with more disparity.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.6\textwidth,height=0.6\textheight,keepaspectratio]{images/regularization.png}
    \captionsetup{justification=centering}
    \caption{Early stopping}
    \label{fig:f37}
\end{figure}

This particular situation might leads the thought "The right model is the best" despite this is not true. The right model will generate a very well known problem when the gradient descent step is applied which will generate very similar values (0 or 1) when the training is being executed.

One way to prevent this is use L1 (Eq. \eqref{eq:30}) and L2 regularization (Eq. \eqref{eq:31}), which is a technique to prevent the drastic increase in the weights of the network.

\[\text{Large Coefficientes} \rightarrow \text{Overfitting}\]

\begin{equation}
\label{eq:30}
\text{L1 Error function} = - \frac{1}{m} \sum_{i=1}^m y_i\ln{(\hat{y})} + (1 - y_i) (\ln{(1 - \hat{y})} + \lambda (|w_1|+\dots+|w_n|))
\end{equation}

\begin{equation}
\label{eq:31}
\text{L2 Error function} = - \frac{1}{m} \sum_{i=1}^m y_i\ln{(\hat{y})} + (1 - y_i) (\ln{(1 - \hat{y})} + \lambda (w_1^2+\dots+w_n^2))
\end{equation}

The main difference between L1 and L2 is that L1 is good for feature selection because at the end the model end up with sparse vectors \([0,1,1,0,0]\) this means that small weights then to go to zero and big ones to one. However, L2 is normally better for training models because it tries to maintain all the weights homogeneously small \([0.2, 0.5, 0.7 ,0.9, 0.2]\).

\textbf{Dropout} is another technique to prevent overfitting. This technique randomly turn off some of the nodes and prevents the data to pass through there. In that case, the other nodes have to pick up the slack and take more part in the training. This process improves the training process a guarantee a good training of all the nodes in the network. Each epoch different nodes are turned off and this is governed by a probability, let's say \(0.2\) this probability means that per each epoch, each node has a probability of 20\% to be turned off.

There is a problem related to the application of the backpropagation algorithm to minimize the error and it is the \textbf{local minimums}. A local minimum is a point that doesn't have more apparently logic direction to move which will decrease the error, e.g, Fig. \ref{fig:f38} presents an example of this. The person is located in a local minimum (yellow point) which for the algorithm seems like the global optimum (green point), nevertheless, this is not.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.6\textwidth,height=0.6\textheight,keepaspectratio]{images/minima.png}
    \captionsetup{justification=centering}
    \caption{Local minima}
    \label{fig:f38}
\end{figure}

One way to solve the local minimum issue is to do \textbf{random restarts} as a very simple approach. The model starts from a few different random places and do gradient descend from all of them. This increases the probability that the model gets to the global minimum, or at least a pretty good local minimum.

\textbf{Vanishing gradient} is caused when the derivative of the activation function is small. The sigmoid function is an example of this, the derivative of the regions that are close to one or zero are almost zero, leading to a very small gradient and complicating the propagation of the error backwards (tiny values of gradient leads to small changes in the weights during the training process). This problem gets even worse when more layers are added to the neural network, increasing the number of activation functions and hence, reducing the gradient of the error towards zero.

A solution to the vanishing gradient problem is the use of different activation functions. One of these is the hyperbolic tangent function Eq. \eqref{eq:32} or the rectified linear unit (ReLU) Eq. \eqref{eq:33}

\begin{equation}
\label{eq:32}
tanh(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}}
\end{equation}

\begin{equation}
\label{eq:33}
relu(x) = \begin{cases}
            x & \text{if } x \geq 0  \\
            0 & \text{if } x < 0
            \end{cases}
\end{equation}

The ReLU activation function can be seen in Fig. \ref{fig:f39}.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.5\textwidth,height=0.5\textheight,keepaspectratio]{images/relu.png}
    \captionsetup{justification=centering}
    \caption{ReLU activation function}
    \label{fig:f39}
\end{figure}

\textbf{Stochastic gradient} descent is a technique to save computation time and find the optimum quicker when the amount of data is huge. The idea behind stochastic gradient descent is simply take small subsets of data, run them through the neural network, calculate the gradient of the error function based on those points and then move one step in that direction. Now, to use all the data it is necessary to split the data into several batches. These batches are then passed through the neural network, one batch per step, until all the batches are passed through the neural network.

This method will eventually converge to a good solution in the model, not as accurate as the one got with the full batch of data. However, in practice it's much better to take a bunch of slightly inaccurate steps than to take one good one in a lot of time.

The learning rate is very important during the training of the model. A very high learning rate will make the model oscillate in a local minimum, whereas a small learning rate will take longer to find the minimum but it will eventually find it. As a rule of thumb, it is always better to work with tiny learning rates Fig. \ref{fig:f40}

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.6\textwidth,height=0.6\textheight,keepaspectratio]{images/learning_rate.png}
    \captionsetup{justification=centering}
    \caption{The impact of learning rates}
    \label{fig:f40}
\end{figure}

Finally, another way to solve the local minimum problem is with the use of the Momentum. The idea is to walk a bit fast with momentum and determination in a way that if the model gets stuck in a local minimum, it can sort it off and get over the hump to look for a lower minimum Fig. \ref{fig:f41}.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.75\textwidth,height=0.75\textheight,keepaspectratio]{images/momentum.png}
    \captionsetup{justification=centering}
    \caption{Momentum}
    \label{fig:f41}
\end{figure}

Momentum is a constant \(\beta\) between 0 and 1 that attaches to the steps as follows: the previous step gets multiplied by 1, the one before, by beta, the one before, by beta squared, the one before, by beta cubed, etc (Eq. \eqref{eq:34}). In this way, the steps that happened a long time ago will matter less than the ones that happened recently.

\begin{equation}
\label{eq:34}
step(n) = step(n) + \beta(step_{n-1}) + \beta^2(step_{n-2}) + \dots
\end{equation}

\printbibliography

\end{document}


\documentclass{article}  %Need this.

\usepackage{amsmath,amsthm,amssymb}

\usepackage[margin=1in]{geometry}

\newtheorem*{thm}{Theorem}
\newtheorem*{cnj}{Conjecture}
\newtheorem*{lem}{Lemma}
\newtheorem*{cor}{Corollary}
\newtheorem*{prop}{Proposition}

\newcommand{\N}{\mathbb{N}}
\newcommand{\Z}{\mathbb{Z}}
\newcommand{\R}{\mathbb{R}}

\title{I Heart \LaTeX}
\author{Me!}

\begin{document}
\maketitle

Remember your $\$$ signs!  $x^2+y^2=1$, $\frac{a}{b}$.

\section{Lists}
%%%%%%%%%%%%%%%
\begin{enumerate}
\item {\bf First Point (Bold Face)}
\item {\em Second Point (Italic)}
\item {\Large Third Point (Large Font)}
    \begin{enumerate}
        \item {\small First Subpoint (Small Font)}
        \item {\tiny Second Subpoint (Tiny Font)}
        \item {\Huge Third Subpoint (Huge Font)}
    \end{enumerate}
\item[$\bullet$] {\sf Bullet Point (Sans Serif)}
\item[$\circ$] {\sc Circle Point (Small Caps)}
\end{enumerate}

\section{Equations}
%%%%%%%%%%%%%%%%%%%

\subsection{Binomial Theorem}
\begin{thm}[Binomial Theorem]
For any nonnegative integer $n$, we have
$$(1+x)^n = \sum_{i=0}^n {n \choose i} x^i$$
\end{thm}

\subsection{Sets}

\begin{thm}
For any sets $A$, $B$ and $C$, we have
$$ (A\cup B)-(C-A) = A \cup (B-C)$$
\end{thm}

\begin{proof}
\begin{eqnarray*}
(A\cup B)-(C-A) &=& (A\cup B) \cap (C-A)^c\\
&=& (A\cup B) \cap (C \cap A^c)^c \\
&=& (A\cup B) \cap (C^c \cup A) \\
&=& A \cup (B\cap C^c) \\
&=& A \cup (B-C)
\end{eqnarray*}
\end{proof}

\end{document}


\section{Framework description}
\label{sec:method}

In this section, the proposed artificial neural networks-evolutionary algorithm (\gls{ann-ea}) based method for prognostics is presented. Our method uses a multi-layer perceptron (\gls{mlp}) as the main regressor for estimating the \gls{rul} of the engines at each subset of the \gls{cmaps} dataset. For the training sets, the feature vectors are generated by using a strided time window while the labels vectors are generated using a constant \gls{rul} for the early cycles of the simulation and then linearly decreasing the number of remaining cycles, this is the so-called piecewise linear degradation model \citep{Ramasso2014}. For the test set, a time window is taken from the last sensors readings of the engine and used to predict the \gls{rul} of the engine.

The window-size $n_w$, window-stride $n_s$, and early-\gls{rul} $R_e$ data-related parameters, which for the sake of clarity and formalism in this study are considered as components of a vector $\nu \in \mathbb{Z}^3$ such that $\nu = (n_w, n_s, R_e)$, have a considerable impact on the quality of the predictions made by the regressor. Handpicking the best parameters, i.e. $\nu$,  is time consuming, furthermore, grid search approaches as the ones used for hyperparameter tuning in neural networks are computationally expensive given the dimension of the search spaces of the data-related parameters. In this paper, we propose the use of an evolutionary algorithm to fine tune the data-related parameters. The optimization framework here proposed allows for the use of a simple neural network architecture while attaining better results in terms of the quality of the predictions made by the other methods in the current literature.

\subsection{The Neural Network Architecture}

For this study we propose to use a rather simple \gls{mlp} architecture and the structure of the network remained consistent for all the four subsets. All the implementations were done in python using the Keras/Tensorflow environment, the source code is publicly available at the git repository \url{https://github.com/dlaredo/NASA_RUL_-CMAPS-}.

The choice of the network architecture was made using an iterative process; comparing 6 different architectures, training each for $100$ iterations using a mini-batch size of $512$ and averaging their results over $10$ different runs. Two objectives were pursued: that the architecture was compact, e.g. in terms of layers and neurons within each layer, and that the performance indicators presented in Section \ref{sec:rul_dataset} were minimized.

The process for choosing the network architecture is as follows: First, chose a fixed $\nu$, for the following experiment let $\nu = (30, 1, 140)$. Next, six different \gls{ann} architectures are defined, details of the architectures are provided in Appendix \ref{sec:appendices}. For each of the six different architectures, the performance is assessed using a cross-validation set from subset 1 of \gls{cmaps}. Table \ref{table:tested_architectures_100} summarizes the results for each tested architectures while Table \ref{table:proposed_nn} presents the architecture chosen for the remainder of this work. The chosen architecture provided the best compromise between compactness and performance among the rest of the tested architectures.

\begin{table}[!htb]
\centering

\begin{tabular}{l | r r r r | r r r r}
	\hline
	& \multicolumn{4}{| c}{RMSE} & \multicolumn{4}{| c}{RHS} \\
	Tested Architecture & Min. & Max. & Avg. & STD & Min. & Max. & Avg. & STD\\
  	\hline
  	Architecture 1 & 15.86 & 17.26 & 16.47 & 0.43 & 5.98 & 10.06 & 7.33 & 1.11\\
  	Architecture 2 & 15.56 & 17.15 & 16.35 & 0.65 & 6.52 & 20.11 & 4.50 & 4.50\\
  	Architecture 3 & 16.07 & 19.18 & 17.67 & 1.12 & 6.91 & 19.18 & 12.78 & 4.72\\
  	Architecture 4 & 15.32 & 19.99 & 17.63 & 1.48 & 5.93 & 24 & 13.54 & 6.28\\
  	Architecture 5 & 15.70 & 17.24 & 16.37 & 0.49 & 4.84 & 8.57 & 6.35 & 1.25\\
  	Architecture 6 & 15.58 & 16.92 & 15.95 & 0.39 & 5.44 & 7.65 & 6.38 & 0.68\\
  	\hline
\end{tabular}

\caption{Results for different architectures for subset 1, 100 epochs}
\label{table:tested_architectures_100}
\end{table}

\begin{table}[!htb]
\centering
\begin{tabular}{l l l l}
	\hline
	Layer & Shape & Activation & Additional Information\\
  	\hline
  	Hidden layer & 20 & ReLU & L2 regularization factor = 0.2\\
  	Output layer & 1 & Linear & \\
  	\hline
\end{tabular}
\caption{Proposed Neural Network architecture}
\label{table:proposed_nn}
\end{table}

\subsection{Shaping the data}

This section covers the data preprocessing applied to the raw sensors readings in each of the datasets. Although the original datasets contains $21$ different sensors readings, some of the sensors do not present much variance or convey redundant information, such sensors are therefore discarded. In the end, only $14$ sensors out of the $21$ are considered for this study, their indices in the \gls{cmaps} dataset are $\left\lbrace 2, 3, 4, 7, 8, 9, 11, 12, 13, 14, 15, 17, 20, 21 \right\rbrace$. The raw measurements are then used to create the strided time windows with window-size $n_w$ and window-stride $n_s$. For the training labels, $R_e$ is used at the early stages and then the \gls{rul} is linearly decreased. The data is also normalized to be within the range $\left[ -1,1 \right]$ using the min-max normalization.

\begin{equation}
\hat{x}_i = 2* \frac{x_i - min(x_i)}{max(x_i) - min(x_i)} - 1,
\label{eq:min_max_norm}
\end{equation}
where $x_i$ denotes the $m$-dimensional vector whose components are all the readings for the \textit{i-th} sensor and $\hat{x}_i$ is the normalized $x_i$ vector.

\subsubsection{Time Window Processing}

In multivariate time-series based problems such as \gls{rul}, more information can be generally obtained from the temporal sequence of data as compared with the multivariate data point at a single time stamp. Let $n_w$ denote the size of the time window, for a time window with a stride $n_s = 1$, all the past sensors values within the time window are collected and put together to form a feature vector $\mathbf{x}$. This approach has successfully been tested in \citep{Li2018, Lim2016} where the authors propose the use of a moving window with values raging from 20 to 30. In this paper we propose not only the use of a moving time window, but also a \textit{strided} time window that updates $n_s$ elements at the time instead of $1$. A graphical depiction of the strided time window is shown in Figure \ref{fig:time_window}.

\begin{figure}[!htb]
\centering
\includegraphics[width=0.9\textwidth]{../img/time_window.png}
\caption{Graphical depiction of the time window used in this framework.}
\label{fig:time_window}
\end{figure}

The use of a \textit{strided time window} allows for the regressor to take advantage not only of the previous information available, but also to control the ratio at which the algorithm is fed with new information. With the usual time window approach only one point is updated for every new time window, on the contrary, the strided time window allows for updating $n_s$ points at the time, allowing for the algorithm to catch newer information with fewer iterations, furthermore, the information contained in the strided time window is likely more rich than the one contained in a time window with stride of one.

\subsubsection{Piecewise linear degradation model}

Different from common regression problems, the desired output value of the input data is difficult to determine for a \gls{rul} problem. It is usually impossible to evaluate the precise health condition and estimate the \gls{rul} of the system at each time step without an accurate physics based model. For this kind of applicatios, a piece-wise linear degradation model has been proposed in \citep{Ramasso2014}. The piece-wise linear degradation model assumes that the engines have a constant \gls{rul} label in the early cycles and then the \gls{rul} starts degrading linearly until it reaches 0 as shown in Figure \ref{fig:piecewise_model}. The piecewise linear degradation approach is used for this work, in here we denote the value for the \gls{rul} at the early stages as $R_e$.

\begin{figure}[!htb]
\centering
\includegraphics[width=0.7\textwidth]{../img/test_engine.png}
\caption{Piecewise linear degradation for \gls{rul}.}
\label{fig:piecewise_model}
\end{figure}

\subsection{Choosing optimal data-related parameters}
\label{sec:choosing_otimal_data-related_params}

As mentioned in the previous sections, the choice of the data-related parameters $\nu$ has a large impact on the performance of the regressor, i.e. the \gls{mlp}. In this section we present a framework for picking the optimal combination of the data-related parameters $n_w$, $n_s$ and $R_e$ while being computationally efficient.

Recall that $\nu = (n_w, n_s, R_e)$, where $n_w \in \left[1, b\right]$, $n_s \in \left[1, 10\right]$, and $R_e \in \left[90, 140 \right]$ are the specific boundaries for the \gls{cmaps} dataset and all the intervals are integer. The value of $b$ is dependent upon the specific subset, Table \ref{table:b_values} presents the different values $b$ can take for each dataset.

\begin{table}[!htb]
\centering
\begin{tabular}{l | l l l l}
	\hline
	 & FD001 & FD002 & FD003 & FD004\\
  	\hline
  	$b$ & 30 & 20 & 30 & 18\\
  	\hline
\end{tabular}
\caption{Allowed values for $b$ per subset}
\label{table:b_values}
\end{table}

Let also $X(\nu)$ be the training/cross-val/test sets parametrized by $\nu$ and used by the \gls{mlp} to perform the \gls{rul} estimation. Finally, let $f(\nu)=e_{rms}(X(\nu))$, recall from Equation (\ref{eq:rmse}) that $d = \hat{y} - y$ and that $\hat{y}$ depends on $X(\nu)$, also note that one function evaluation of $f(\nu)$ implies training the \gls{mlp} and computing the result of Equation (\ref{eq:rmse}). Here we propose to fine tune $\mathbf{\nu}$, formally speaking

\begin{equation}
\begin{aligned}
& \underset{\nu \in \mathbb{Z}^3}{\text{min}}
& & f(\nu) \\
\end{aligned}
\label{eq:optimization_problem}
\end{equation}

Given the nature of the problem at hand; namely that no analytical form of the problem is given, gradient information is unavailable, and the integer nature of the function variables, an evolutionary algorithm is the natural choice for the optimization process.

\subsubsection{Obtaining the true optimal data-related parameters}

The size of \gls{cmaps} dataset and the search space of $\nu$ allows for an exhaustive search to be performed in order to find the true optimal data-related parameters. We would like to emphasize tough, that although exhaustive search is a possibility for \gls{cmaps} dataset it is in no way a possibility in a more general setting, therefore the use of the evolutionary algorithm (\gls{ea}) adopted in this framework. Nevertheless, the possibility to perform exhaustive search on the \gls{cmaps} dataset can be exploited to demonstrate the accuracy of the chosen \gls{ea} and of the framework overall. Taking subsets FD001 and FD002 an exhaustive search is performed to find the true optimal values for $\nu$. The \gls{mlp} is trained for only $20$ epochs as in this experiment we are only interested in comparing the effect of different combinations for $\nu$ instead of obtaining the best performance. Table \ref{table:true_optimal_data_params} shows the optimal as well as the worst combinations of data-related parameters $\nu$ and the total number of function evaluations used by the exhaustive search.

\begin{table}[!htb]
\centering
\begin{tabular}{l | c r c r r l}
	\hline
	 Dataset & argmin $\nu$ & min $f(\nu)$ & argmax $\nu$ & max $f(\nu)$ & Function evals.\\
  	\hline
  	FD001 & $\left[ 30, 1, 125 \right]$ & $17.43$ & $\left[ 19, 1, 97 \right]$ & $80.92$ & 7500\\
  	FD002 & $\left[ 20, 1, 135 \right]$ & $36.89$ & $\left[ 19, 10, 109 \right]$ & $76.80$ & 2500\\
  	\hline
\end{tabular}
\caption{Exhaustive search results for subsets FD001 and F002.}
\label{table:true_optimal_data_params}
\end{table}

\subsubsection{Evolutionary algorithms for obtaining the optimal data-related parameters}
\label{sec:ea_optimization_process}

Evolutionary algorithms/meta-heuristics are a family of methods that optimize a problem by iteratively trying to improve a set of candidate solutions with regard to a given measure of quality. The methods do not make any assumptions about the problem, treating it as a black box that merely provides a measure of quality given a candidate solution. Furthermore \glspl{ea} do not require the gradient of the problem being optimized, making them very suitable for applications such as neural networks. Among the drawbacks of this kind of methods are that they usually require considerable computing effort to converge to a solution.

For this particular application, differential evolution (\gls{de}) \citep{Storn1997} is chosen as the optimization algorithm. Though in principle any meta-heuristic capable of handling integer variables is suitable for this application, \gls{de} has been stablished itself as one of the most reliable, robust and easy to use \glspl{ea}. Furthermore, a ready to use python implementation of \gls{de} is available through the scipy package \citep{scipy}. Although \gls{de} does not have special operators for treating integer variables a very simple modification to the algorithm, consisting on rounding every component of a candidate solution $\nu'$ to its nearest integer, is used for this work.

As mentioned earlier, evolutionary algorithms such as \gls{de} tend to use several function evaluations for obtaining the optimal solutions, recall that for this application one function evaluations implies retraining the  neural network from scratch. This is not a desirable scenario, as obtaining the optimal data-related parameters $\nu$ would entail an extensive use of computational power. Instead of running for \gls{de} for several iterations and with a large population size we propose to run it just for $30$ iterations (generations in the literature of evolutionary computation) and using a population size of $12$, which seems reasonable given the size of the search space of $\nu$.

Furthermore, during the optimization process the \gls{mlp} is not trained for  $100$ epochs but for just $20$ instead, this is done mainly for two reasons: the use of the mini-batch in the training process allows for a speed up in the convergence, therefore it can be assumed that the algorithm will most likely be very close to its optima after just a couple of iterations, second and most important is the assumption that parameters that lead to lower score values in the early stages of the \gls{mlp} training process are more likely to provide better performance when trained for a larger number of epochs. Given the similarities between subsets FD001/FD003 and FD002/FD004 we have decided to just tune $\nu$ for subsets FD001 and FD002 and then use the obtained results on sets FD003 and FD004 respectively. Details for the use of \gls{de} in finding the optimum data-related parameters are described in Table \ref{table:de_hyperparams}.

\begin{table}[!htb]
\centering
\begin{tabular}{l l l l l}
	\hline
	 Population Size & Generations & Strategy & \gls{mlp} epochs\\
  	\hline
  	12 & 30 & Best1Bin & 20\\
  	\hline
\end{tabular}
\caption{Differential Evolution hyper-parameters.}
\label{table:de_hyperparams}
\end{table}

\begin{comment}
\begin{table}[!htb]
\centering
\begin{tabular}{l l l l l}
	\hline
	 Dataset & Window Size $n_w$ & Window Stride $n_s$ & Early RUL $R_e$\\
  	\hline
  	FD001 & 26 & 2 & 100\\
  	FD002 & 16 & 2 & 91\\
  	FD003 & 30 & 2 & 97\\
  	FD004 & 16 & 2 & 92\\
  	\hline
\end{tabular}
\caption{Optimal data-related parameters for each subset.}
\label{table:optimal_data_params}
\end{table}
\end{comment}

The optimal data-related parameters for each of the subsets found by \gls{de} are shown in Table \ref{table:optimal_data_params}. As can be observed the results obtained by \gls{de} are in fact very close to the real optima (displayed in Table \ref{table:true_optimal_data_params}) for both datasets, nevertheless the computational burden is reduced by one order of magnitude when using \gls{de}. From the results in Table \ref{table:optimal_data_params} it can be observed that the maximum allowable time window is always preferred while, on the contrary, small window strides yield better results, for the case of early RUL it can be observed that large $R_e$ improve the results.

\begin{table}[!htb]
\centering
\begin{tabular}{l | c r r l}
	\hline
	 Dataset & argmin $\nu$ & min $f(\nu)$ & Function evals.\\
  	\hline
  	FD001 & $\left[ 30, 1, 128 \right]$ & $17.78$ & 372\\
  	FD002 & $\left[ 20, 2, 134 \right]$ & $37.65$ & 372\\
  	\hline
\end{tabular}
\caption{Data-related parameters for each subset obtained with Differential Evolution.}
\label{table:optimal_data_params}
\end{table}

\subsection{The ANN-EA RUL estimation Framework}

Having described the major building blocks of the proposed method, we now introduce the complete framework in Algorithm \ref{alg:rul_framework}.

\setcounter{algorithm}{0}
\begin{algorithm}[H]
\caption{\gls{ann}-\gls{ea} \gls{rul} estimation Framework}\label{alg:rul_framework}
\textbf{Input:} Initial set of data-related parameters $\nu \in \mathbb{Z}^n$, Raw training/testing data $X$ and training labels $y$\\
\textbf{Output:} Optimal set of data-related parameters $\nu^*$
	\begin{algorithmic}[1]
		\State Choose regressor architecture (\gls{ann}, \gls{svm}, linear/logistic regression, etc).
		\State Define $f(\nu)$ as in Section\ref{sec:choosing_otimal_data-related_params}.
		\State Optimize $f(\nu)$ using the preferred evolutionary algorithm, i.e. differential evolution, evolutionary strategies, genetic algorithm, etc, using the proposed guidelines from Section \ref{sec:ea_optimization_process}.
		\State Use $\nu^*$ to train the regressor for as many epochs as needed.
	\end{algorithmic}
\end{algorithm}


\par
\section{Data Structure}
\par
There are four typed objects.
\begin{itemize}
\item
{\tt MSMD} : the main object.
\item
{\tt MSMDinfo} : an object that communicate parameter choices from
the caller to the {\tt MSMD} object and information and statistics
from the {\tt MSMD} object to the caller.
\item
{\tt MSMDstageInfo} : an object that contains statistics for a
stage of elimination, e.g., number of steps, number of vertices
eliminated, weight of vertices eliminated, etc.
\item
{\tt MSMDvtx} : an object that models a vertex.
\end{itemize}
A user needs to understand the {\tt MSMDinfo} object,
so this is where we will start our description.
\par
%-----------------------------------------------------------------------
\par
\subsection{{\tt MSMDinfo} : define your algorithm}
\par
\begin{itemize}
\item
{\tt int compressFlag} -- define initial and subsequent compressions of
the graph.
\par
We compress a graph using a checksum technique.
At some point in the elimination, vertices in the reach set (those
adjacent to vertices just eliminated) have a checksum based on
their adjacencies computed, and then vertices with the same
checksum are compared to see if they are indistinguishable.
This operation has a cost, and there are classes of vertices for
which there is a large amount of compression, and for other
classes there is little.
Compression is a powerful tool, but we need a way to control it.
\begin{itemize}
\item
{\tt compressFlag \% 4 == 0}
--- do not perform any compression after each elimination step.
\item
{\tt compressFlag \% 4 == 1}
--- after each elimination step, consider only those nodes that are
{\it 2-adjacent}, adjacent to two eliminated subtrees and having no
uncovered adjacent edges.
\item
{\tt compressFlag \% 4 == 2}
--- after each elimination step, consider all nodes.
\item
{\tt compressFlag / 4 >= 1} --- compress at stage zero before any
elimination.
\end{itemize}
The default value is {\tt 1}, no initial compression and
consider only 2-adjacent nodes after each elimination step.
\item
{\tt int prioType} ---
define the priority to choose a vertex to eliminate.
\begin{itemize}
\item
{\tt prioType == 0}
--- zero priority
\item
{\tt prioType == 1}
--- exact external degree for each vertex
\item
{\tt prioType == 2}
--- approximate external degree for each vertex
(${\hat d}$ from \cite{ame96-amd})
\item
{\tt prioType == 3}
--- exact external degree for 2-adjacent vertices, approximate
external degree for the others
\end{itemize}
The default value is {\tt 1}, exact external degree for each vertex.
\item
{\tt double stepType} --- define the elimination steps.
\begin{itemize}
\item
{\tt stepType == 0}
--- only one vertex of minimum priority is eliminated at each step,
    e.g., as used in SPARSPAK's {\tt GENQMD}, YSMP's ordering,
    and {\tt AMD} \cite{ame96-amd}.
\item
{\tt stepType == 1}
--- an independent set of vertices of minimum priority is eliminated
at each step, e.g., as used in GENMMD, multiple minimum degree.
\item
{\tt stepType > 1}
--- an independent set of vertices is eliminated whose priorities
lie between the minimum priority and the minimum priority
multiplied by {\tt stepType}.
\end{itemize}
The default value is {\tt 1}, multiple elimination of vertices with
minimum priority.
\item
{\tt int seed} --- a seed used for a random number generator, this
introduces a necessary random element to the ordering.
\item
{\tt int msglvl} -- message level for statistics, diagnostics and
monitoring. The default value is zero, no statistics.
Set {\tt msglvl} to one and get elimination monitoring.
Increase {\tt msglvl} slowly to get more mostly debug information.
\item
{\tt FILE *msgFile} -- message file, default is {\tt stdout}.
\item
{\tt int maxnbytes} -- maximum number of bytes used during the ordering.
\item
{\tt int nbytes} -- present number of bytes used during the ordering.
\item
{\tt int istage} -- present stage of elimination.
\item
{\tt int nstage} -- number of stages of elimination.
\item
{\tt MSMDstageInfo *stageInfo} -- pointer to vector of
{\tt MSMDstageInfo} structures that hold information about each
stage of the elimination.
\item
{\tt double totalCPU} -- total CPU to find the ordering.
\end{itemize}
%-----------------------------------------------------------------------
\par
\subsection{{\tt MSMD} : driver object}
\par
A user need not know anything about the internals of this object,
just the methods to initialize it, order the graph, and extract the
permutation vectors and/or a front tree.
\par
\begin{itemize}
\item
{\tt int nvtx} --- number of internal vertices in the graph.
\item
{\tt IIheap *heap} --
pointer to a {\tt IIheap} object that maintains the priority queue.
\item
{\tt IP *baseIP} -- pointer to the base {\tt IP} objects, used to hold
subtree lists
\item
{\tt IP *freeIP} -- pointer to the list of free {\tt IP} objects
\item
{\tt int incrIP} -- integer that holds the increment factor for the
{\tt IP} objects.
\item
{\tt MSMDvtx *vertices} -- pointer to vector of {\tt MSMDvtx}
objects that represent the vertices.
\item
{\tt IV ivtmpIV} -- {\tt IV} object that holds an integer temporary
vector.
\item
{\tt IV reachIV} -- {\tt IV} object that holds the reach vector.
\end{itemize}
\par
%-----------------------------------------------------------------------
\par
\subsection{{\tt MSMDstageInfo} : statistics object for a stage of
the elimination}
\par
This object stores information about the elimination process at a
stage of the elimination.
\par
\begin{itemize}
\item
{\tt int nstep} --- number of elimination steps in this stage
\item
{\tt int nfront} --- number of fronts created at this stage
\item
{\tt int welim} --- weight of the vertices eliminated at this stage
\item
{\tt int nfind} --- number of front indices
\item
{\tt int nzf} ---
number of factor entries (for a Cholesky factorization)
\item
{\tt double ops} ---
number of factor operations (for a Cholesky factorization)
\item
{\tt int nexact2} ---
number of exact degree updates to 2-adjacent vertices
\item
{\tt int nexact3} --- number of exact degree updates
to non-2-adjacent vertices
\item
{\tt int napprox} --- number of approximate degree updates
\item
{\tt int ncheck} --- number of comparisons of vertices with the same
checksum during the process to find indistinguishable nodes
\item
{\tt int nindst} --- number of indistinguishable nodes that were
detected.
\item
{\tt int noutmtch} --- number of nodes that were outmatched
\item
{\tt double cpu} --- elapsed CPU time for this stage of the elimination.
\end{itemize}
\par
%-----------------------------------------------------------------------
\par
\subsection{{\tt MSMDvtx} : vertex object}
\par
This object stores information for a vertex during the elimination.
\par
\begin{itemize}
\item
{\tt int id} --- id of the vertex, in range {\tt [0,nvtx)}
\item
{\tt char mark} --- character mark flag, {\tt 'O'} or {\tt 'X'}
\item
{\tt char status} --- character status of the vertex
\begin{itemize}
\item {\tt 'L'} -- eliminated leaf vertex
\item {\tt 'E'} -- eliminated interior vertex
\item {\tt 'O'} -- outmatched vertex
\item {\tt 'D'} -- vertex on degree (priority) heap
\item {\tt 'R'} -- vertex on reach set
\item {\tt 'I'} -- vertex found to be indistinguishable to another
\item {\tt 'B'} -- boundary vertex, to be eliminated in another stage
\end{itemize}
\item
{\tt int stage} --- stage of the vertex. Stage 0 nodes are
eliminated before stage 1 nodes, etc.
\item
{\tt int wght} --- weight of the vertex
\item
{\tt int nadj} --- size of the {\tt adj} vector
\item
{\tt int *adj} ---
for an uneliminated vertex, {\tt adj} points to a list
of uncovered adjacent edges; for an eliminated vertex, {\tt adj}
points points to a list of its boundary vertices (only valid when
the vertex is a leaf of the elimination tree or a root of a subtree
of uneliminated vertices).
\item
{\tt int bndwght} --- for an eliminated vertex, the weight of the
vertices on its boundary.
\item
{\tt MSMDvtx *par} --- for an eliminated vertex,
{\tt par} points to its parent vertex in the front tree
({\tt NULL} if the vertex is the root of a subtree).
For an indistinguishable vertex, {\tt par} points to its
representative vertex (which may have also been found to be
indistinguishable to another).
\item
{\tt IP *subtrees} --- pointer to a list of {\tt IP} objects to store
the adjacent subtrees, valid only for uneliminated vertices.
\end{itemize}


\subsection{Pre-image attacks}

Given hash value h, can we find message m?

\subsection{Defence from pre-image attacks}


\chapter{Equivalent current Dipole fitting}
\label{ch:eeg_ecd}

This little chapter demonstrates how to use the ECD (Equivalent Current Dipole) routines with the multimodal dataset available on the FIL website. The aim is to fit a single dipole on the N170 wave visible in the 3 conditions.
I will briefly describe how to analyse the dataset. For more details about the implementation, please refer to the help bit of and comments in the routines themselves.

\section{Necessary data}
Before proceeding any further, we have to make sure that we have all the necessary data in the right format
We need
\begin{itemize}
\item the {\it amri.img/hdr} structural MRI of the subject. It will be used to build the head model and display the results in the subject's anatomical space.\\
\item the {\it mae\_eeg.dat/mat} EEG data files. These are the fully processed data with one ERP per condition.
\item the coordinates of the sensors, fiducial markers and scalp points (headshape) in 3 distinct {\it *.mat} files.
\end{itemize}

In the dataset provided on the web, the raw {\it *.pol} files are available. It is necessary to prepare these files to use them with the source reconstruction routines. This is a crucial step as the registration between the "EEG space" and "patient/image space" relies entirely on these files!
To prepare these files, use the little script {\it create\_fid\_files.m} distributed with SPM5. A copy is also available at the end of this chapter.

Once we have all the files ready, we can proceed with the 3 main steps: building the model, fitting the dipole and displaying the results. To launch the GUI, press "3D source reconstruction" in the main window of SPM.

\section{Model building}
After selecting the data file {\it mae\_eeg.mat} and the method "ECD", the first step is building the meshes for the scalp and inner skull volume. This is done automatically through the "Meshes" button. Select the structural MRI to use ({\it amri.img} here) and wait...

This step takes some time as the MRI is normalised and segmented. The normalisation parameters are saved in the {\it amri\_vbm\_inv\_sn.mat} file and will be used later to map coordinates between the template and subject spaces. With the segmentation, the brain and scalp binary volumes are built ({\it amri\_iskull.img} and {\it amri\_oscalp.img} ). These are used to build the outer scalp and inner skull surface meshes. These are saved in the {\it model\_head\_amri.mat} file with other information. The scalp mesh is also saved in the file {\it amri\_scVert.mat}.

Once the head model is ready, we can co-register the EEG space with subject/image space. Use the "Data Reg." button and decide if the registration should be based on the fiducials only (which is quite approximate) or the fiducials and the scalp surface (which should be more precise). Then select the appropriate files: {\it fid\_eeg.mat}, {\it fid\_MRI.mat}, {\it headshape\_orig.mat}, {\it amri\_scVert.mat} and let the routine work.

To prepare the model for the forward solution, simply press "ForwardComp." and "individual" to use the subject's own MRI. The forward model uses a spherical approximation. The best fitting sphere are adjusted on the scalp surface and 2 other spheres are added to model the scalp and skull outer surfaces. Obviously the head is not spherical and there will be a mismatch between the scalp/brain surfaces and their respective spheres. We have used the idea proposed by Spinelli et al., 2000 \cite{Spinelli2000}, where the brain volume is warped into a sphere. This allows us to use an analytical formula to calculate the forward solution for each dipole location while preserving some anatomical characteristics: superficial (resp. deep) sources remain superficial (resp. deep) in the spherical head model.

At this last step, the electrodes are also introduced in the head
model and positioned relative to the subject head, as in the MRI. The
{\it model\_head\_amri.m} at contains the information about the fitted spheres and electrodes. Dipole fitting of the data is now possible.

\section{Dipole fitting}
By pressing the "Inverse Sol." you launch the dipole fitting procedure. A number of questions have to be answered in order to specify the kind of solution you want:
\begin{itemize}
\item "Condition to use", select which condition is used to fit the dipole(s). So far, it is not possible to fit multiple conditions (or linear combinations of them) at the same time. For example, for differences between conditions, you should pre-calculate this difference before trying to fit ECDs. \\
\item "Time window", define the time window in ms on which the ECDs should be fitted. With the N170 demo data, a good window is 150 to 180.\\
\item "Number of dipoles", this is the crucial questions. How many dipoles should be used? It's up to you to decide... With the demo data, from the look of the EEG scalp map, 1 ECD should be enough.\\
\item "Number of random seeds". In order to avoid being trapped in a local minimum during the optimisation process because of a peculiar starting point. The algorithm can be launched from multiple random starting 'seeds'. If they all converge to approximately the same solution, then we'll have most surely reached the local optimum.\\
\item "Orientation of the dipoles". The location of the ECD will be constant throughout the time window but its orientation can be left free or be fixed as well. Leaving the orientation free allows the dipoles to rotate over time. To fix the orientation, we can use the (weighted according to the EEG power)) mean over the time window or use the orientation of the ECD fitting the time instant with maximum EEG power.\\
\item "File name". File names are suggested but feel free to change it!
\end{itemize}

After fitting the N random seeds, the routine tries to group them in clusters of similar ECDs according to their location and signal variance explained. Eventually, these 'grouped' ECDs are displayed on the subject anatomy. The result of this clustering is saved in a mat file starting by {\it res\_} and finishing with the name you entered.

\section{Result display}

Results can be redisplayed with the routine {\it spm\_eeg\_inv\_ecd\_DrawDip.m}. The routine asks you to select the solution file you want to display and the MR image to be used.

\section{Preparing the *.pol files}
\begin{quote}
% Just quick and dirty programming to extract the information from the .pol
% files: fiducials, sensors & headshape in EEG space.

% Actually, I just opened the ascii files and copy the coord of the EEG
% fiducials here... Much easier.

% Order of the fiducials is: LE, RE, Na
fid\_eeg = ([-0.0587687  6.79448 -0.00636311 ; ...
            0.0352661   -6.78906    -0.00369206 ; ...
            9.3675  0.0260009   0.00481311] + ...
           [-0.0328487  6.78991 0.00636288 ; ...
            0.0563513   -6.79533    0.00369206 ; ...
            9.45206 -0.0260009  -0.00481297])/2 ...
            * 10 ; % To convert cm into mm

% These are coordinates picked by hand on the sMRI.
% So it's quite an approximation of where the fiducials are really...
fid\_mri = [-71.8 3.5 -58.8 ; ...
            71.3 -6  -62.5 ; ...
            0   90.6 -28.4]        ;

% I edited the .pol files to REMOVE the first few lines with
% fiducial information.
sensors = load('sensors\_noFid.pol','-ASCII')*10;
headshape = load('headshape\_noFid.pol','-ASCII')*10;
    % Again, multiply by 10 to get the measures in mm instead of cm

% ATTENTION !!!
% Polhemus, uses the axes in a different orientation!
% It's still a right handed system but axes are:
% - x: from back to front (versus left to right)
% - y: from right to left (versus back to front)
% - z: from bottom to top
%
% To make coord systems compatible, it is necessary to rotate clockwise the
% coord by 90 degree around the z axis.

Rot = spm\_matrix([0 0 0 0 0 -pi/2]); Rot = Rot(1:3,1:3);
fid\_eeg = (Rot*fid\_eeg')';
sensors = (Rot*sensors')';
headshape = (Rot*headshape')';

save fid\_eeg fid\_eeg
save fid\_mri fid\_mri
save sensors\_orig sensors
save headshape\_orig headshape

\end{quote}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%