\chapter{Artificial Neural Networks}
\label{ch:Artificial_Neural_Networks}
%\section{Introduction}
%\label{sec:AAN_intro}
%A neural network is an interconnected assembly of simple processing elements, units or nodes, whose functionality is loosely based on the animal neuron. The processing ability of the network is stored in the interunit connection strengths, or weights, obtained by a process of adaptation to, or learning from, a set of training patterns.

\acfp{ANN} are computational systems which elaborate information in a way that is loosely inspired by the operation of biological neural networks (brains).
Biological networks are superior in performance to computers and extremely more efficient in difficult tasks such as classification (e.g. image recognition), prediction (e.g. stock market prediction), and creation (e.g. new text generation).
The underlying idea is to copy some of their mechanisms and exploit them for computational applications.
These systems are: intrinsically parallel in operation, able to modify their behavior through a learning process, and sharing similar dynamics to solve different tasks.

Topologically, \acsp{ANN} are made of collection of interconnected nodes, each one of them elaborating information.
\acsp{ANN} can be either simulated on computers or physically built on hardware designed ad hoc.
At this time, the improvement of software implemented ANN is withheld by limitation in computing power and efficiency.
Training a complex artificial neural network with computers at the state of art might take even weeks.
With the aim of improving operating performance from simulated networks, research on hardware architectures is surging: digital, analog, electrical, and optical devices are being developed.

\section{History}
\label{sec:History}
It is widely acknowledged that the opening work of this research field was made in 1943 by Warren McCulloch and Walter Pitts, a neurophysiologist and a mathematician respectively.
In their research work, they described the operating mechanism of biological neurons by modeling a simple electronic circuit \cite{McCulloch1943}.

The following important work was made by Donald O. Hebb, who hypothesized the neural plasticity in 1949 \cite{hebb1949organization}.
He pointed out the fact that neural pathways are strengthened each time they are used, a concept known as \textit{Hebbian learning}.

During the late 1950s and the early 1960s, as computers became more powerful, many promising works were published.
Some simulated operation of artificial networks on calculators, e.g. Farley and Clark \cite{Farley1954} and Rochester \cite{Rochester1956}.
Other produced circuitry that implemented on hardware such networks, e.g. Rosenblatt \cite{frank1957perceptron,Rosenblatt1958}.
However, despite the early successes of neural networks, the traditional computing architecture (von Neumann architecture) was chosen as the preferred computing architecture.
%As a result, funding and therefore also research diminished drastically in the following decades.

The reason why this happened, probably, was due to several concurring facts.
First of all, at the time the single-layer perceptron was the only trainable network, not the multiple- or deep-layered networks.
Moreover, in the same time period (1969) a research paper by Minksi and Papert \cite{minski1969perceptrons} showed that the linear classfiers (e.g the basic perceptron) were not able to execute the \ac{XOR} operation and with it a wide class of real-world problems.
Thus, unlike the logical circuits at the base of the von Neumann architecture, \ac{ANN} were considered insufficient for demanding applications.

%were impossible to train due to the vanishing gradient problem.
%Moreover, the research stated the fact that  %(at that time) because of the lack of adequate processing power.

%who discovered two key issues with the computational machines that processed neural networks.
%The first was that basic perceptrons were incapable of processing the exclusive-or circuit.
%The second was that computers didn't have enough processing power to effectively handle the work required by large neural networks.
%in the same time period a research paper wrongly suggested that there could not be any multiple-layered neural network.

In addition to this research, the early successes of some works on neural networks pointed to an overestimation of the artificial neural networks potential, also held back by the technological capacity of the time.
Finally, important questions of philosophical nature came to light, such as the fear fueled debate on the impact on our society of a class of computers able to think.
This very controversy, i.e. the \ac{AI} problem, is discussed still today \cite{stanford.edu}.

During the 1980s the interest for this computing method was reinvigorated by the fact that it was discovered a method to train multi-layered networks by distributing pattern recognition errors through all the layers in the network.
This method is now called \textit{backpropagation}.

In today's research and technology, artificial neural networks are used in numerous applications: from the most serious ones, such as medical image classification \cite{li2014medical}, to the lighter ones, like photo enhancement \cite{ignatov2017wespe} and generation of polyphonic music \cite{johnson2017generating, briot2017deep}.
%However, the development is made slowly, due to technological limitation in computational power of present processors.

\section{Basis of Neural Networks}
\label{sec:Basis_of_Neural_Networks}

A neural network is a collection of processing elements, or nodes, interconnected with arbitrary topology.
From its input nodes, the network accepts information, which distribute across inner nodes through the interconnections and will get elaborated at each node.
%At the end of the network, there will be a number of output nodes, with the task of reading a portion of the inner nodes.
A layer of output nodes reads to outcome of the inner (hidden) nodes computation and transfer it to the outside.
Inner nodes are also called hidden, because they are not meant to be accessible to the external world.
A generic scheme of such network is shown in \autoref{fig:generic_NN}.

\begin{figure}[ht]
	\centering
	\input{tikz/GenericNN.tex}
	\caption{
		Generic scheme of a neural network.
		Triangles (red) are input nodes, circles (grey) are inner nodes,
		and squares (blue) are output nodes.
		Interconnections among nodes are represented by arrows.
		%continuous when both elements are drawn, and dotted otherwise.
		This figure uses an uncommon representation
		(see \autoref{sec:standard_representation}).
		%(see \hyperref[sec*:standard_representation]{next Section}).
	}
	\label{fig:generic_NN}
\end{figure}

Nodes can all implement the same function or behave differently, depending on the type.
%Each node can operate in the same way of the others or in a completely different manner, depending on the type of neural network.
The operation of nodes resembles that of neurons: various input gets collected and elaborated together to obtain an output, which will become one of the many inputs for subsequent neurons/nodes.
Specifically, the most used model for neurons is the McCulloch–Pitts (MCP) neuron.
It is divided into two parts, as shown in \autoref{fig:generic_node}: the first part is a weighted sum of the inputs, while the second part is given by the so called activation function.

\begin{figure}[ht]
	\centering
	\input{tikz/GenericNode.tex}
	\caption{Generic node representation. $x$-values are inputs, $y$-values are outputs, $w_0$ is the bias.}
	\label{fig:generic_node}
\end{figure}
The node is described mathematically by \cref{eq:Generic_node_function}
\begin{equation}
y = f_a \left(  w_0 + \sum_{i=1}^{n} w_i x_i \right),
\label{eq:Generic_node_function}
\end{equation}
where $f_a$ is the activations function, evaluated on the sum of the input $x_i$ weighted with $w_i$, plus a bias $w_0$.

\newpage
Each node accepts values at its inputs and produces an output accordingly.
However, in addition to the input, the output depends also on the node's parameters: the weights and the bias (see \autoref{sec:Working_Principles_of_ANNs}).
Generally these parameters are modified during the training of the network, but not while it is operating.

The activation function $f_a\left(\cdot\right)$ must met a few requirements: firstly, it must be nonlinear, because otherwise a collection of nodes will result in just a weighted sum of its inputs.
Moreover, in order to apply the backpropagation algorithm on the network, it must also be continuous and derivable.
In addition, depending on the problem that needs to be solved, limitedness is also a desirable property. \cite{duda2012pattern}.

\begin{figure}[ht]
	\begin{subfigure}[b]{0.49\textwidth}
		\centering
		\input{tikz/HeavisideThetaFunction.tex}
		\caption{}
		\label{fig:activation_function_example_1}
  \end{subfigure}
  \begin{subfigure}[b]{0.49\textwidth}
  		\centering
		\input{tikz/LogisticFunction.tex}
		\caption{}
		\label{fig:activation_function_example_2}
  \end{subfigure}
  \caption{
		Examples of activation function:
		(\ref{fig:activation_function_example_1}) is the step function,
		or Heaviside $\Theta$,
		(\ref{fig:activation_function_example_2}) depicts few Logistic functions.
		The former is a discontinuous function, while the latter is a continuous and derivable function.}
  	\label{fig:activation_function_examples}
\end{figure}

Hence, nonlinear activation functions are distinguished for their properties, such as continuity, limitedness, and existence of the derivative.
\autoref{fig:activation_function_examples} show two examples of nonlinear activation function: \ref{fig:activation_function_example_1} is discontinuous and bounded between \numlist{0;1}, while \ref{fig:activation_function_example_2} is continuous, derivable, and saturates for $|x|\rightarrow\infty$, i.e it is limited.

%One can distinguish at least three type of nodes in every neural network: input, inner/hidden, and output nodes.
%Input nodes take one input value, from the outside of the neural network, and pass it on to the inner nodes unchanged.
%Inner/hidden nodes take many inputs and generate an output through the activation function.
%Output nodes, similarly to input nodes, take one input value, from the inside of the \acs{ANN}, and pass it on to the outside.

\newpage
\subsection{Standard Representation}
\label{sec:standard_representation}
%This is not the orthodox description, but I claim that it is more consistent than the standard representation with the idea of functional \textit{black box}, in which input and output are the only visible nodes, while the other are hidden inside.
In the thesis I depict a generic neuromorphic network in a slightly different way compared to ICT literature.
%The way I depicted a generic neuromorphic network in \autoref{fig:generic_NN} is not the standard representation used in books and research papers.
Usually weights are represented on the connection between the nodes, which are then designated to apply only the activation function.
Moreover the input layer is linear as it feeds the inner nodes with the input data, while the output layer is actually given by the last nonlinear layer of the inner nodes.
A generic network is shown in \autoref{fig:standardNNdesc}.

\begin{figure}[ht]
	\begin{subfigure}[b]{0.49\textwidth}
		\centering
		\input{tikz/standardNNdesc.tex}
		\caption{Standard way of representing networks}
		\label{fig:standardNNdesc}
  \end{subfigure}
  \begin{subfigure}[b]{0.49\textwidth}
  		\centering
		\input{tikz/BlackBoxNN.tex}
		\caption{Representation of the black-box concept}
		\label{fig:black_box_NN}
  \end{subfigure}
  \caption{}
  	\label{fig:description_comparison}
\end{figure}

I consider the inner nodes as the only place where any kind of elaboration on the data happens.
Inner nodes have a number of inputs, which are weighted and summed together to be entered as argument in the activation function.
This leads to a natural separation between input/output nodes, which acquire the task of providing data from/to the outside, and inner nodes, which is where the weighted sum and/or the activation function are carried out.

This non-standard description is consistent with the idea of functional \textit{black box}, in which input and output are the only visible nodes, while the other are hidden inside, as shown in \autoref{fig:black_box_NN}.
%Therefore I can easily distinguish the nodes which will use a nonlinear activation function.
%Questa non è la visione standard/ortodossa, ma è consistente con un'idea di 'black box' in cui gli input e output sono gli unici nodi visibili. mentre gli altri sono all'interno della scatola nera.

\subsection{Applications of ANNs}
\label{ssec:Applications_of_ANNs}
Conventional computers are extremely fast and efficient in executing simple algebraic tasks and they can perform more complex activities if provided with the correct series of instructions.
Hence algorithms can solve arbitrary difficult problems, provided that the necessary steps are known.

\aclp{ANN} take a different path in the solution of a given problem.
In fact, when \acsp{ANN} are designed to solve a problem, they learn from the examples that are provided to them.
\acsp{ANN} cannot be programmed, instead they exploit their internal complexity to minimize the error and hence automatically solve the problem.
After a network has been prepared, it is able to operate much faster than the complex algorithms on conventional computers.

\acsp{ANN} have proven to be extremely good at recognizing patterns, which can be used to solve problems of several classes.
For example, classification problems are solved by the recognition of attributes of the input element.
Likewise, clustering is obtained when a sequence of examples is grouped according to their features.
Moreover, regression analysis and time series prediction can be obtained, when a sequence of data is fed to the network.
% Regression: Predicting a continuous variable\\
% Classification: Predicting a variable with finite possible values\\
% Clustering: Grouping data\\
% https://cs.stackexchange.com/questions/58131/are-neural-networks-a-type-of-reinforcement-learning-or-are-they-different

\newpage
\section{Working Principles of ANNs}
\label{sec:Working_Principles_of_ANNs}
\acsp{ANN} are implemented with different topologies in order to solve different problems.
However, the behavior of a single network changes depending on the value of the weights applied at each node.
For this reason, before a neural network is considered ready to perform a task, it must go through a preparation phase, or training, which it is usually composed by three processes: learning, validation, and test.
Each one of these procedure is meant to prepare the network to work as required from the designer.

\subsection{Learning Process}
\label{ssec:Learning_Process}
During the learning process the neural network is run on a set of inputs ${x}$, each paired with the correct answer ${y}$, or target, in a second set of data.
This procedure is called \textit{supervised learning}.
The neural network will produce at the output its \textit{prediction}, a third set ${\hat{y}}$, which should be as close as possible to the correct answers, when the network works properly.
%However this happens rarely, if ever, and a change in the way data is elaborated becomes necessary.
%The usual \ref{} way is to keep the same topology, but tweak the weights that connect the hidden nodes together.
%On account of this need, one have to quantify the distance of the predicted result of the artificial neural network from the correct answer.
%This is made by means of the loss function.
Typically weights among nodes are initialized randomly and the distance of the network outcome from the target function is measured through a loss function.
Then weights are modified in order to minimize the loss function.% to its lowest possible value. %up to a tolerance limit is achieved.
Every time the network is trained on the full dataset, an \textit{epoch} is completed.

\subsubsection{Loss function}
\label{sssec:Loss_function}
The loss function $L(y, \hat{y})$ evaluates the difference between the predicted and the correct answer.
Usually, this quantity is linked to the geometrical distance between the predicted output and the target $\left| \hat{y}-y \right|$.

The most common loss function is the \acfi{MSE}.
Assuming to have an input set of $N$ examples paired with the same number of targets, and that the outputs and the targets are composed by $C$ values, or classes, the function becomes:
\begin{equation}
	L(y, \hat{y}) = f_{MSE}(y, \hat{y}) = \frac{1}{N} \sum_{n=1}^N \sum_{i=1}^C \left( \hat{y}_{n,i} - y_{n,i} \right)^2,
\end{equation}
where each example in the set is subtracted to its target and then squared.
Finally the mean of all squares gives the expected result.

Another commonly used function is the \acfi{CEL} (also known as negative log likelihood),
\begin{equation}
	L(y, \hat{y}) = f_{CEL}(y, \hat{y}) = - \frac{1}{N} \sum_{n=1}^N \sum_{i=1}^C y_{n,i} \log \left( \hat{y}_{n,i} \right),
\end{equation}
which probabilities at the input, i.e. positive values between \numlist{0;1}.
Hence the error $-y\log \left( \hat{y} \right)$, quantified for each element in each example, is always a positive number.
The mean over all examples in the set returns the results.

%Alternatively, variations of the previous methods are given by taking the sum of the examples in place of the mean, or by calculating a loss function for each example instead of evaluating it for the whole set.

Depending on the class of the problem, different loss functions are chosen.
In fact, since the loss function drives the weights update process, it is important to choose the correct one.
For example, in tasks that expect outputs to take continuous values, the \acs{MSE} is favored.
On the other hand, in classification tasks, where the answer of each output node is boolean, the preferred function is \acs{CEL}.

\subsubsection{Weights Update Process}
\label{sssec:Weights_Update_Process}

The weights update process is a difficult task, and probably the most computationally expensive one in running a neural network.
There is a variety of methods to chose from, depending on the type of \acs{ANN} and on the resources available.

A widely used algorithm is the gradient descent, from its most simple version to more complex variations such as \acfi{SGD}.
This method updates the weights by subtracting a value proportional to the gradient of the loss function in respect to the weights themselves times a positive factor called \textit{learning rate}, as shown below.
\begin{equation}
	\left.w_i\right|_{n+1} = \left.w_i\right|_n - lr \cdot \frac{\partial L}{\partial \left.w_i\right|_n}
\end{equation}
where $\left.w_i\right|_{n}$ are the current weights, $lr$ is the learning rate, $\dfrac{\partial L}{\partial \left.w_i\right|_n}$ is the first derivative of the loss function in respect to the i-th weight at the current step, and $\left.w_i\right|_{n+1}$ are the updated weights.
%Hence it is necessary to calculate the gradient $\bigtriangledown_w L$.
This method is equivalent to minimize the error on the loss function, by following the gradient $\nabla_w L$.
This vector lives in the multidimensional space of the loss function $L:\mathbb{R}^W \mapsto \mathbb{R}$, where $W$ is the total number of weights in the network.

The most used algorithm is called \textit{backpropagation}
%, or \textit{chain rule}
: it computes the first derivative of the loss function $L$ in respect to all the parameters of the network, the weights
It starts from the last layer of the \acs{ANN} and goes backward toward the input, hence its name.
%Since the number of connections between nodes might be order of magnitude greater than the number of nodes, it is easy to understand how large networks are computationally expensive to train.

In early days, this algorithm was used together with the Logistic activation function.
Due to the saturating behavior of the function, the gradient is repeatedly multiplied by small values at each layer.
Hence, the effect of backpropagation becomes negligible for the first layers in deep networks.
This problem is known as \textit{vanishing gradient} problem.

Thanks to the introduction of non saturating activation functions, the training of deep neural network has become gradually possible \cite{krizhevsky2012imagenet}.
An example of such function is the \acf{ReLU}.

%\paragraph{Other types of learning processes} are used, e.g. unsupervised/reinforced.

\subsection{Validation Process}
\label{ssec:Validation_Process}
The validation process is carried out at the same time of the learning process and consists on testing the neural network on a new set of examples.
Differently from the learning process, during validation the output predicted by the network are compared to the examples, but the weights are not updated.
Instead, the loss function is used as a control parameter to prevent overfitting.

Overfitting is the phenomenon in which a neural network recognizes features that are specific of the sampled data instead of the more general ones.
It occurs when the network is trained over and over on the same set of data, and it starts to learn details which characterize the test dataset.
%Since the specific features are the ones that characterize precisely the test dataset, they should not characterize other dataset.
This typically worsen the \acs{ANN} prediction capability on other datasets.
Therefore, by periodically testing the network on a novel set, i.e. the validation set, one can avoid the problem.

The validation process happens repeatedly throughout the learning process. %, depending on the resources available and the size of the dataset.
The learning process is stopped at a minimum of the error on the validation set.

\subsection{Testing Process}
\label{ssec:Testing_Process}
At the end of the learning and validation processes, the ANN have to be tested on never seen dataset: the test data.
Predicted outputs are compared to the correct answers and an overall value of the correctness is evaluated, often expressed in percentage.

\autoref{fig:LVTcurves} shows an example of the evolution of the loss function on the three dataset as function of the epochs.

\begin{figure}[htbp]
	\centering
	\input{tikz/LVTcurves.tex}
	\caption{The learning curves are the value of the loss function as a function of the training iterations.
	Both the validation and the test error are usually larger than the errors on the training set.
	Sometimes, to avoid overfitting, the learning procedure is stopped at a minimum of the validation learning curve \cite{duda2012pattern}.
	}
	\label{fig:LVTcurves}
\end{figure}

\subsection{Datasets}
\label{ssec:Datasets}
The training process of an \acs{ANN} is heavily conditioned by the characteristics of the dataset.
For example, the number of attributes of the examples in the dataset defines the number of input nodes and similarly the features of the answers define the output nodes.
This is a constraint on the network size, which obviously affects the training.
Moreover, the size of the dataset influences the length of the training and the presence of effects such as overfitting.
For instance, smaller datasets are more prone to overfitting than larger datasets.
Hence, the collection of the data and the choice of its attributes are problems that might account for a large part of the time in the development of an \acs{ANN}.

In addition, considering a given dataset, there is also the problem of defining the subset used for learning, validation, and testing.
The most naive division is in three equal parts, since three are the phases of preparation for any \acs{ANN}.
However, as shown later in \autoref{ssec:PyDataset}, when the resources dictate otherwise, other subdivisions can be implemented.
In my case, the decision was to follow the suggestions of the authors of the dataset and divide the examples into a learning set and a test set only, with a ratio between the two close to \SI{50}{\percent}.

%\textsc{CHANGE ALL THIS!!!}
%Creating a dataset and splitting it into subsets is another problem to deal with.
%It is not so simple and straightforward as it appears.
%For example a dataset which is too big will lead to a longer training time for the network, whereas a too little set will cause a poor training.
%
%The most naive division is in three equal parts, since three are the phases of preparation for any artificial neural network.
%However, as shown later in \autoref{ssec:PyDataset}, when the resources dictates otherwise, other subdivision can be implemented.
%In my case, the decision was to follow the suggestions of the authors of the dataset and divide the examples into a training set and a test set only, with a ratio between the two close to 50\%.

\section{Feed Forward ANN}
\label{sec:Feed_Forward_ANN}

The first and most simple network type is the \ac{FFNN}.
In this kind of \acs{ANN}, nodes are divided into groups called \textit{layers}.
A layer is a collection of nodes that accepts inputs from a preceding group and generate as many outputs as the number of nodes in the layer.
Each layer of a \acs{FFNN} is connected in series with the others, except for the input layer at the beginning and the output layer at the end.
As for the single nodes, the inner layer are called hidden, because usually not accessible.

The information travels from the input to the output and gets elaborated from each hidden layer: there are neither connection between nodes of the same layer, nor loops or feedback between layers.
%Depending on the topology of the network, there might be more or less layers, each composed by the same or a different number of nodes.
The number of hidden layers and the number of nodes they contain depends on the network size.
Moreover the connection between the layers might be complete, i.e. each node in the layer accepts each input of the preceding layer, in that case the layer is said to be \textit{fully connected}, or sparse as in the case of convolutional layers (see \autoref{par:Convolutional}).

\subsection{Perceptron}
\label{ssec:Perceptron}

The most naive topology of a \acs{FFNN} is given by the so called \textit{Perceptron}.
The Perceptron dates back to the 1957, when the homonym \textit{Perceptron algorithm} was software implemented by Frank Rosenblatt on a computer (IBM 704) and subsequently in hardware as the \textit{Mark 1 perceptron} \cite{frank1957perceptron,Rosenblatt1958}.
The graph of a generic (single layer) perceptron \acs{ANN} is shown in \autoref{fig:Perceptron}.

\begin{figure}[ht]
	\centering
	\input{tikz/PerceptronNN.tex}
	\caption{%
		Perceptron type neural network: in this representation the perceptron has $n$ inputs and $m$ outputs as well as a hidden layer with $m$ nodes. %
%		Colors, shape and styles are the same as in \autoref{fig:generic_NN} \vpageref{fig:generic_NN}.%
		}
	\label{fig:Perceptron}
\end{figure}

By adding more than one hidden perceptron layer to the neural network, one obtain the so called \acfi{MLP}.
This allows for more computational complexity, e.g. \acs{MLP} can solve the \acs{XOR} problem, whereas single layer perceptron cannot.
When the number of hidden layers is more than two, the network is called \textit{deep}.
A deep \acs{MLP} is shown in \autoref{fig:deepMLP}.

\begin{figure}[ht]
	\centering
	\input{tikz/MLPNN.tex}
	\caption{	Deep \acf{MLP}, fully connected.}
	\label{fig:deepMLP}
\end{figure}

\acsp{MLP} might show heavily different topology, i.e. each layer could have a different number of nodes, however often the layers at the beginning are wider than the layer at the end of the network (see following section).
Moreover, in literature with the term perceptron one almost always refers to fully connected \acs{FFNN}.

\subsection{Other Feed Forward \acsp{ANN}}
\label{ssec:Other_Feed_Forward_NNs}

\acsp{FFNN} are a large family that includes many other types besides the perceptron one.
A few names are autoencoder, time delay, and convolutional neural networks.
Autoencoders \acsp{ANN} are \acsp{FFNN} with the same number of input and output nodes, whose purpose is to reconstruct its own inputs.
For this reason autoencoders employ unsupervised learning.
Time delay networks have a \acs{FFNN} structure and their purpose is to analyze sequence of signals in search of specific patterns.

\subsubsection{Convolutional Neural Networks}
\label{par:Convolutional}
%\acp{CNN} are inspired to the visual cortex, in which neurons are not fully connected all the inputs but only to a restricted region.
\ac{CNN} is a type of \acs{FFNN} conceived to recognize images without being misled by distortions such as translation, skewing, or scaling.
Its input is often represented as a 2D matrix, instead of a 1D vector.
This kind of network is usually composed by many layers: the most recurring is the convolutional one, but other types can be mixed together too.

A convolutional layer performs a two-dimensional convolution over the input matrix of a second 2D matrix of weights, called \textit{feature map}.
Thus, each node of the layer operates on a restricted region to understand if a feature is present or not.
The operating regions are commonly overlapping and the feature map is shared among the nodes in the same layer.
Often several convolutional layers operate in parallel to extract different features, hence they use different feature maps.
Due to the specific operation of convolutional layers, the number of nodes per layer usually decreases with each layer.
\autoref{fig:convolutionalNN} shows the possible scheme of a convolutional layer, operating a $3\times 3$ convolution on the input.

\begin{figure}[ht]
	\centering
	\input{tikz/ConvNN.tex}
	\caption{%
		Pictorial representation of a convolutional layer.
		Each node acts on a restricted region of the inputs, in this examples a $3\times 3$ region, to extract a feature.
		Several convolutional layers might be placed side by side to extract different features from the same input.
		}
	\label{fig:convolutionalNN}
\end{figure}

\acsp{CNN} are nowadays widely used in image recognition with outstanding results and they improve with a steady pace.
A technological application of this kind of network is the real time recognition of obstacles in a vehicle path, for safety and automation purposes in the automotive industry.

\section{Other Types of \acsp{ANN}}
\label{sec:Other_Types_of_NNs}
By changing the topology of the nodes distribution and their connections, one obtain other networks that cannot be catalogued under the class of \acs{FFNN}.
%Moreover, those different types of network are not a niche, but they are widely studied as a different approaches to the same or additional problems.

\subsection{Recurrent ANN}
\label{ssec:Recurrent_ANN}
\acp{RNN} are a kind of network in which a portion of the input of a node depends on the (past) output of the same node.% or nodes of subsequent layers.
That is information does not propagates only forward like in the \acs{FFNN}, but can propagate also backward, for example in loops or in feedbacks.
As a result, the time-dependent dynamic of the node is usually very important in the study of \acsp{RNN}.
\autoref{fig:RecurrentNN} shows a recurrent node, in which the output is connected to the input of the node.

\begin{figure}[ht]
	\centering
	\input{tikz/RecurrentNN.tex}
	\caption{%
		Representation of a recurrent node.
		One of the inputs is given by the output itself.
		This output to input connection could be mediated by a delay device, so that for example the output at $t-1$ becomes the input at $t$.
		Depending on the structure of the network, there might be recurrent nodes and/or recurrent groups of nodes, i.e. loops.
		}
	\label{fig:RecurrentNN}
\end{figure}

\acsp{RNN} have found large use in time series analysis \cite{duda2012pattern}.
However \acsp{RNN} are occasionally employed to obtain the same classification of \acs{FFNN}.
%In this case, the former are equivalent to the latter, if ``unfolded'' in time.
%That is the expansion of the recurrent architecture in time is the same as the topology of the \acs{FFNN} one in space.

\subsection{Reservoir Computing}
\label{ssec:Reservoir_Computing}
\ac{RC} describes a learning method that differs from the deep learning commonly used by \acs{MLP} (\acs{FFNN}) and \acs{RNN}.
%In fact, the topology of the network could be exactly the same as that of a deep multi-layer perceptron or that of a recurrent network.
%However, the reservoir computing differs in approach in respect to deep learning.
It claims that it is not necessary to learn all the weights of the network, as in deep learning, but it is sufficient to train only the last layer of the network.
\acs{RC} was initially implemented due to the vanishing gradient problem, for which the last layer was the only one that showed significant change in the weights.

\begin{figure}[ht]
	\centering
	\input{tikz/ReservoirNN.tex}
	\caption{A reservoir NN is topologically equivalent to deep networks. However only the last layer is trained. In this picture the trained node are represented outside the box, while those inside are initialized with random weights which are then left unchanged.}
	\label{fig:reservoirNN}
\end{figure}

\autoref{fig:reservoirNN} shows a generic topology that could implement the \acs{RC} method: only the nodes outside the darker region are trained, whereas those inside provide the computational power.
This kind of network can be trained much faster than its respective deep-learning counterpart, i.e. deep \acs{FFNN} and deep \acs{RNN}.
The question over which training method is more effective is still debated and literature does not provide clear answers yet.

Significant results for \acs{RC} implemented with photonics have been obtained mainly with the \acs{RNN}, therefore in this field \acs{RC} is typically associated to the \acs{RNN} topology.

%\subsubsection{Modular NN}
%\label{sec:Modular_NN}

%\subsubsection{Spiking NN}
%\label{sec:Spiking_NN}
%Spiking artificial networks are the most different kind in respect to all the other networks until now described.
%In this class of ANNs, information is not coded only in the intensity of the signal, but also in the rate of signals, e.g. a high value will be encoded as a signal with high repetition rate, whereas a low value as a signal with low repetition rate.
%This way of encoding information is more alike the mechanism of biological neural networks, such as our brain \ref{}.
%
%\vspace{1em}
%\noindent LOOK INTO other type of neuron model: Hodgkin–Huxley (H-H) model\\
%\url{ https://en.wikipedia.org/wiki/Binding_neuron }

%\section{Real-Life Examples}
%\label{sec:Real-Life_Examples}
%\noindent\uppercase{\large{? Should I keep this section ?}}
%\normalsize

\newpage
\section{FFNN Simulation}
\label{sec:FFNN_Simulation}
In the field of \acs{ANN} simulation, there are several platforms that are independently developed.
Among them there are TensorFlow, Theano, Caffe, Keras, Torch, and PyTorch.
Unfortunately, each one of them have its strength and weakness, therefore the choice of the framework becomes very difficult.

To make a choice, the key factors considered were the language used, the features, the flexibility, and the level of diffusion of the framework in the machine learning community.
The language used is a crucial factor, because it takes time to learn the peculiarities of a library written for a known language, but even more time to learn to use an unknown programming language.
However also the features and the flexibility in the implementation are important traits, because any feature that is not natively implemented needs new coding.
In turn, it is difficult to integrate new coding if the framework is not flexible enough to accommodate extensions or custom definitions.
Last, but not least, it is easier to find support for a widespread framework in respect to a limitedly used one.
In regards to this, \autoref{fig:GoogleTrendsPyTorch} shows the number of internet searches for some frameworks in the past few years.

In light of the facts above, PyTorch was chosen as framework.
PyTorch is an open source machine learning library for python, which development has started only recently and it is based upon the older framework Torch  \cite{PyTorch.org}.
It provides a high-level platform for the deep learning ecosystem and integrates acceleration libraries that allow fast and lean operation, both on common \acsp{CPU} and \acsp{GPU}.
It is based on a backpropagation algorithm called \textit{Reverse-mode auto-differentiation}, which allows versatile execution.

To summarize, the library was chosen for its language, its flexibility, and the growing interest for it within the machine learning community.
Nevertheless, its features make it a very powerful framework.
%However, considering the need to keep the simulated system as simple as possible, this last characteristic slipped in the background.
%The need to simulate a simple network comes from the fact that the ultimate goal of this project is to implement an \acs{ANN} with a (photonic) hardware architecture.

\begin{figure}[htbp]
	\centering
	\input{tikz/GoogleTrendsPyTorch.tex}
	\caption{Google Search statistics for different keywords in the \textit{machine learning and artificial intelligence} field.
		Numbers represent search interest relative to the highest point on the chart for the given region and time.
		A value of 100 is the peak popularity for the term.}
	\label{fig:GoogleTrendsPyTorch}
\end{figure}

%The resources and the time available allowed me to implement physically only the activation function of one node.
%Hence, to test this hardware implementation as I will show in \autoref{sec:Test_of_a_Trained ANN} of \autoref{ch:experiments}, I had first to simulate and train \textit{offline} a specific neural network.
%To do so, I chose a programming language, \textit{Python}, and a library, \textit{PyTorch}, which helped me in this task.

\subsection{PyTorch}
\label{ssec:PyTorch}
PyTorch is a Python package which provides a powerful framework for deep learning.
It is versatile in the sense that allow customizations at almost every level of operation.
For the purposes of this work, a neural network implemented in PyTorch is composed by the definition of three main components and a few lines of code to implement the operation.

The most important part of the network is the so called \textit{model}, which defines the topology of the network by setting parameters such as the number of nodes in each layer and the connections among them.
Moreover, it defines also the activation function for each node, separately or in groups.
The activation function can be coded from scratch, but the library provides several functions and allows to combine them easily together.
%can also be one of the functions provided by the library or a composition of them.
%While the first offers almost unlimited flexibility, the second choice consents to exploit the functions already given.
%The second option is the best choice, if the needs of the task support it, because it allows to save much time.

Another important part of the network is the so called \textit{criterion}, which is nothing else than the loss function.
The set of functions provided by PyTorch can accommodate almost any necessity.
Moreover, their operation can usually be adjusted with some internal parameters.

Similarly to the criterion, the last piece of the system is given by the \textit{optimizer}.
Optimizers are a class of algorithms that provides the optimization of the weights during the learning phase.
Most commonly used methods are already supported and they provide parameters to fine tune their execution.

%Following the official tutorials and the immense package documentation,
I implemented a generic model of a fully connected \acf{MLP}, in which I could choose the overall structure of the network by setting some parameters to specific values (see \autoref{ssec:Simulated_ANN_operation}).
The optimizer chosen is called \acfi{SGD}, which is parametrized just by the learning rate, at least in its vanilla implementation.
It was then improved with the addition of the momentum and a learning rate scheduler.
The former allows to decrease the sensitivity of local variation in the gradient, while the latter decreases the learning rate with the number of iterations.
The criterion used was initially the \acs{MSE}, however it was eventually replaced by the \acfi{CEL}, which is more suited to the classification task.
%More details on the model, optimizer, and criterion will follow in \autoref{ssec:Simulated_ANN_operation}.

%I tested the model with it with randomly generated numbers until its operation seemed correct.

\subsection{Dataset}
\label{ssec:PyDataset}
Initial tests and trials on artificial neural networks are often made by using randomly generated sets of numbers.
Exploiting the finite number of the examples, it is possible to train a network to successfully correlate the input to the output.
However, once the system has been successfully set up, to compare different working parameters such as number of nodes and number of layers, a common standardized dataset should be employed.
In this way, it is possible to set side by side different network implementations and analyze their performances.

In this preliminary work, the need to keep the system as simple as possible, drove my attention towards datasets of middle to small sizes.
With the help of the \textit{UC Irvine Machine Learning Repository} \cite{UCIMLR}, I selected one with slightly less than a thousand entries.
The dataset is called \textit{Connectionist Bench (Vowel Recognition - Deterding Data) Data Set}.
It is composed by \num{990} entries of \num{10} attributes, i.e. input values, grouped in \num{11} classes.

Each entry is a vector of \num{10} real numbers plus an integer number between \num{0} and \num{10}.
In view of the fact the activation function of the photonic node is a real-positive-valued function (see \autoref{sec:Characterization_of_the_Activation_Function}), I renormalized the database.
The normalization is independently applied to each one of the attributes in such a way that is described by a real number in $[0,1]$.
The normalization is applied before the dataset is divided in the learning and testing sets.

The authors suggest to divide the dataset in a learning set of \num{528} entries and a testing set of \num{462} entries.
Since the absolute number of entries is low, compared to other datasets, the validation set is not defined and the test set is used in its place.

\subsection{Simulated ANN operation}
\label{ssec:Simulated_ANN_operation}
The model that I defined allows to modify the number of input ($D_{in}$) and output nodes ($D_{out}$), the number of hidden layers ($n_{H}$), and the number of nodes in each hidden layer ($D_{H}$).
Whereas the $D_{in}$ and $D_{out}$ is fixed by the problem, i.e. the dataset, the other are free parameters.
Another degree of freedom that I implemented is whether the last hidden layer applies the nonlinear activation function or just the weighted sum, since I observed that in many examples of network the last hidden layer was linear.
The scheme of the network is shown in \autoref{fig:ANN_model}.

\begin{figure}[htbp]
	\centering
	\input{tikz/PT_MLP.tex}
	\caption{Topology of the model implemented in PyTorch. The number of input ($D_{in}$) ad output nodes ($D_{out}$), as well as the number of nodes in the last hidden layer is determined by the dataset.
	The number of nodes ($D_{H}$) in the other hidden layers and the number of hidden layers itself ($n_{H}$) is governed by a parameter.}
	\label{fig:ANN_model}
\end{figure}

The choice of the structure of the neural network is purely heuristic, since up to now no satisfactory method to design the topology has been proposed in literature.
However, given the problem (dataset), at least the input and output layer have a fixed amount of nodes.
Specifically, the input nodes are \num{10}, the number of attributes, while the output nodes are \num{11}, the number of classes.

Using the chosen dataset, I tested the neural network operation on the model defined before, with two different kinds of activation function.
Initially I implemented a \acfi{ReLU} and a \textit{Logistic} (sigmoid) activation functions.
%%The \acs{ReLU} function is a standard activation function and is defined as follows:
%Using the chosen dataset, I tested the neural network operation on the model defined before, with the nonlinearity provided by the \acfi{ReLU} activation function.
The \acs{ReLU} function is a standard activation function defined as follows:
\begin{equation}
f_{ReLU}(x) =
\begin{cases}
	0 & \qquad \mathrm{for}~ x\leq 0\\
	x & \qquad \mathrm{for}~ x\geq 0
\end{cases}~.
\label{eq:relu}
\end{equation}
The Logistic function, which has already been shown in \autoref{fig:activation_function_example_2}, is defined by
\begin{equation}
f_{Logistic}(x) = \frac{1}{1+e^{-k\left(x-x_0\right)}},
\end{equation}
with the parameters $k\in \mathcal{R}^+$ and $x_0\in \mathcal{R}$.
Both functions are widely used in literature, however recently \acs{ReLU} has become the preferred choice in the \ac{ICT} community.
Unfortunately, the results for the Logistic function were almost the same as a random choice.
For this reason in the next section I will describe the results for the \acs{ReLU} only.
I will discuss in \autoref{sec:Optical_Activation_Function} the results of the same network model, with the nonlinear activation function given by the microring resonator.

\subsubsection{Learning}
After the definition of the model, I had to train the networks.
All the examples in the dataset are fed at once into the model, then the overall loss is evaluated, and finally the weights are upgraded.
This method is called \textit{batch} operation; others are \textit{online} operation, which evaluate the loss and change the weights at each example, and \textit{minibatch} operation, which uses a random subset of the full dataset.

I provided the \ac{SGD} algorithm with a learning rates $l_r$ and a momentum $m$, which should stabilize the evolution.
Moreover, I used a scheduler to reduce the learning rate with the iterations.
There are several algorithm for the learning rate scheduler: the one I used is called \textit{StepLR} and decays the value of $l_r$ by $\gamma$ every $T$ epochs.
Specifically the learning rate is initially set to $l_{r|ReLU} = \num{0.05}$ and the momentum to $m=0.25$, while $\gamma = 0.6$ and $T=1000$.

I trained the models for \num{4000} epochs, in order to see the different dynamics of the training procedure: \autoref{fig:PyTorch_learning} shows the typical evolution of the loss on the train and the validation datasets for each model.
The network loss (solid lines) on the training dataset decays towards near zero and shows some particular behaviors.
During the training, there are times at which the network loss does not improve much, \textit{plateau}, and times at which it improves more rapidly.
Moreover, some models show \textit{ripples} during the evolution: this is due either to the learning rate being too high or to the complex shape of the loss function in the parameters space.
Note that each time the learning rate scheduler changes $l_r$ the ripples disappear.

The evolution of the loss function on the validation dataset is different.
It shows a minimum usually in the second quarter of the total training period, which means that in the second half of the training the network is being overfitted on the data.
In fact, I observed that usually the percentage of correct answers decreases slightly toward the end of the \num{4000} epochs due to overfitting.

%Every time the network is trained on the full dataset, an \textit{epoch} is completed.
%I trained the models defined above for \num{4000} epochs, in order to observe the widest variety of dynamics.
%For example, smaller networks tend to train faster, because there are less free parameters.
%On the other hand, larger networks often train slower, but can obtain better results thanks to their complexity.
%At the beginning the network improves its operation, but going on with the training the validation set shows worse
%
%Moreover, I defined a different learning rates $l_r$ for each activation function, in order to provide the best parameter for different models but still be able to compare the performance of similar ones.
%Specifically the learning rate for the \acs{ReLU} is $l_{r|ReLU} = \num{5e-2}$, while for the Logistic function is $l_{r|Logistic}= \num{5e-3}$

\begin{figure}[htbp]
	\centering
	\input{tikz/relu_evolution.tex}
	\caption{Typical evolution of the loss criterion (solid lines) throughout the epochs.
		Points represent the values of the loss criterion on the validation dataset, carried out repeatedly during the training.
		Dotted lines are the scheduled change in the learning rate.
	}
	\label{fig:PyTorch_learning}
\end{figure}

%\autoref{fig:PyTorch_learning} shows the evolution of the loss and the validation of each model during the \num{4000} epochs of training.
%From the evolution of the loss function we can see that every model has

\newpage
\subsubsection{Collection of results}
Considering that the number of classes is \num{11}, the probability of giving a correct answer at random is $p_{rnd}\simeq \SI{9.09}{\percent}$.
Hence, values of correct answers above that mean that the neural network is working.
On the other hand, values around or even below $p_{rnd}$ mean that the current neural network is not working at all.

Since in \num{4000} epochs the \acp{ANN} are become overfitted on the dataset, I select as correctness the maximum value of correct answer given during the training.
It is the same as stopping the learning process on the network when a minimum of the validation set is reached.

%\autoref{tab:PyResults} collects all the results for the different topologies.
%The same shape has been tested with both $f_{ReLU}$ and $f_{Logistic}$ to compare the performance.
\autoref{tab:PyResults} collects all the results for the different topologies, where the activation function was provided by a $f_{ReLU}$ function.
We can observe that the width of the hidden layers seems to influence the correctness of the network, at least marginally.
Moreover the depth of the network, i.e. the number of hidden layers ($n_H$), seems to be correlated to the deviation of the correctness.

\begin{table}[htbp]
	\centering
	\begin{tabular}{c c c c r}
	\toprule
	activation	& no. hidden 	& no. nodes	& last hidden	& Percent \\
	function		& layers 			& per layer	& layer				& correct \\
	\midrule
	$f_{ReLU}$ 			& 2 & 11 & linear & \SI{48 +- 6}{\percent} \\
	$f_{ReLU}$ 			& 3 & 11 & linear & \SI{40 +- 11}{\percent} \\
	$f_{ReLU}$ 			& 2 & 22 & linear & \SI{54 +- 2}{\percent} \\
	$f_{ReLU}$ 			& 3 & 22 & linear & \SI{51 +- 4}{\percent} \\
	\bottomrule
	\end{tabular}
	\caption{Results of the several network topologies.
	The statistics is made on \num{10} different initializations of the same model: the error on the percentage is the standard deviation.
	}
	\label{tab:PyResults}
\end{table}

The overall results obtained for the $f_{ReLU}$-based models are satisfactory because they all well over the $p_{rnd}$ threshold.

%+-------------------------+--------+---------+---------+
%|                         | no. of | no.     | percent |
%|       Classifier        | hidden | correct | correct |
%|                         | units  |         |         |
%+-------------------------+--------+---------+---------+
%| Single-layer perceptron |  -     | 154     | 33      |
%| Multi-layer perceptron  | 88     | 234     | 51      |
%| Multi-layer perceptron  | 22     | 206     | 45      |
%| Multi-layer perceptron  | 11     | 203     | 44      |
%| Modified Kanerva Model  | 528    | 231     | 50      |
%| Modified Kanerva Model  | 88     | 197     | 43      |
%| Radial Basis Function   | 528    | 247     | 53      |
%| Radial Basis Function   | 88     | 220     | 48      |
%| Gaussian node network   | 528    | 252     | 55      |
%| Gaussian node network   | 88     | 247     | 53      |
%| Gaussian node network   | 22     | 250     | 54      |
%| Gaussian node network   | 11     | 211     | 47      |
%| Square node network     | 88     | 253     | 55      |
%| Square node network     | 22     | 236     | 51      |
%| Square node network     | 11     | 217     | 50      |
%| Nearest neighbour       |  -     | 260     | 56      |
%+-------------------------+--------+---------+---------+


\documentclass[]{BasiliskReportMemo}
\usepackage{AVS}

\newcommand{\submiterInstitute}{Autonomous Vehicle Simulation (AVS) Laboratory,\\ University of Colorado}

\newcommand{\ModuleName}{test\textunderscore unitThrusterDynamics}
\newcommand{\subject}{Sunline Switch-EKF Module and Test}
\newcommand{\status}{Initial document}
\newcommand{\preparer}{T. Teil}
\newcommand{\summary}{This module implements and tests a Switch Extended Kalman Filter in order to estimate the sunline direction.}

\begin{document}

\makeCover

%
%	enter the revision documentation here
%	to add more lines, copy the table entry and the \hline, and paste after the current entry.
%
\pagestyle{empty}
{\renewcommand{\arraystretch}{2}
\noindent
\begin{longtable}{|p{0.5in}|p{4.5in}|p{1.14in}|}
\hline
{\bfseries Rev}: & {\bfseries Change Description} & {\bfseries By} \\
\hline
Draft & Initial Revision & T. Teil \\
\hline

\end{longtable}
}

\newpage
\setcounter{page}{1}
\pagestyle{fancy}

\tableofcontents
~\\ \hrule ~\\

%\begin{figure}[htb]
%	\centerline{
%	\includegraphics[]{Figures/Fig1}
%	}
%	\caption{Sample Figure Inclusion.}
%	\label{fig:Fig1}
%\end{figure}

\section{Introduction}
The Switch Extended Kalman filter (SEKF) in the AVS Basilisk simulation is a sequential
filter implemented to give the best estimate of the desired states.
In this method we estimate the sun heading as well as it's rate of change along the observable axes.
The SEKF reads in the message written by the coarse sun sensor, and writes a message
containing the sun estimate.

This document summarizes the content of the module, how to use it, and the test that
was implemented for it. More information on the filter derivation can be found in Reference [\citenum{Teil:2018fe}], and more information on the EKF can be found in Reference [\citenum{Born}].

\section{Filter Set-up, initialization, and I/O}

\subsection{Filter Derivation} %%%

The Switch-EKF attempts to avoid subtracting any terms from the state, while still removing the unobservable component of the rate. In order to do this, an appropriate frame must be defined.
In order to not track the rate component alongside the sunline direction, that vector needs to be one of the basis vectors of the frame. It is decided to be the first vector for the frame, and therefore in that frame, $\omega_1$ the component of the rotation rate can be removed from the states.
This frame is called $\mathcal{S}_1 = \{\hat{\bm s}_1 = \frac{\bm d}{|\bm d|}, \hat{\bm s}_2, \hat{\bm s}_3 \}$. This is seen in Figure \ref{fig:Switches}, where the dotted line represents the $30 \dg$ threshold cone before switching frames.

\begin{figure}[t]
	\centering
	\includegraphics[]{./Figures/Switches}
	\caption{Frame built off the body frame for Switch filters}
	\label{fig:Switches}
\end{figure}

The second vector of the frame must be created using only $\bm d$, and the body frame vectors. The first intuitive decision, is to use $\hat{\bm b}_1$ of the body frame and define $\bm s_2$ in Equation \eqref{eq:s2}. The third vector $\hat{\bm s}_3$ of the $\mathcal{S}_1$ frame, is naturally created from the first two.
\begin{equation}\label{eq:s2}
\hat{\bm s}_2 = \frac{\hat{\bm s}_1 \times \hat{\bm b}_1}{|\hat{\bm s}_1 \times \hat{\bm b}_1|} \hspace{2cm} \hat{\bm s}_3 = \frac{\hat{\bm s}_1 \times \hat{\bm s}_2}{|\hat{\bm s}_1 \times \hat{\bm s}_2|}
\end{equation}
The problem that arises is the singularity that occurs when $\hat{\bm b}_1$ and $\bm d$ become aligned: this frame becomes undefined. In order to counteract this, using a similar process as the shadow set used for Modified Rodrigues Parameters [\citenum{schaub}], a second frame is created. This frame $\mathcal{S}_2 =  \{\hat{\bar{\bm s}}_1 = \hat{\bm s}_1, \hat{\bar{\bm s}}_2 , \hat{\bar{\bm s}}_3 \}$ is created with the same first vector, but constructs $\hat{ \bar{\bm s}}_2$ using $\hat{\bm b}_2$ of the body frame as in Equation \eqref{eq:s2bar}. The last vector, once again, finishes the orthonormal frame.
\begin{equation}\label{eq:s2bar}
\hat{\bar{\bm s}}_2 = \frac{\hat{\bar{\bm s}}_1 \times \hat{\bm b}_2}{|\hat{\bar{\bm s}}_1 \times \hat{\bm b}_2|}
\end{equation}
With both these frames, $\mathcal{S}_1$ and $\mathcal{S}_2$, the singularities can always be avoided. Indeed, $\mathcal{S}_1$ becomes singular when $\bm d$ approches $\hat{\bm b}_1$, while $\mathcal{S}_2$ becomes singular when the sunheading approaches $\hat{\bm b}_2$. By changing frames, whenever the sunline gets within a safe cone of $30 \dg$ (a modifiable value) of $\hat{\bm b}_1$, the frame is rotated into $\mathcal{S}_2$, which is not singular. Similarly, when $\bm d$ approches $\hat{\bm b}_2$ the frame is switched back to $\mathcal{S}_1$.

Because the two frames share the sunline vector $\bm d$, this vector is the same in both frames. This is a clear advantage as this is the vector we desire to estimate, and not having to rotate it avoids numerical issues.
The rotation of the rates is done by computing the following DCMs, of which all the vectors are known.
\begin{equation}\label{eq:DCMs}
 [\mathcal{B}\mathcal{S}_1] = \begin{bmatrix} \leftexp{B}{\hat{\bm s}_1} & \leftexp{B}{\hat{\bm s}_2} & \leftexp{B}{\hat{\bm s}_3}\end{bmatrix} \hspace{1cm}  [\mathcal{B}\mathcal{S}_2] = \begin{bmatrix} \leftexp{B}{\hat{\bar{\bm s}}_1} & \leftexp{B}{\hat{\bar{\bm s}}_2} & \leftexp{B}{\hat{\bar{\bm s}}_3}\end{bmatrix} \hspace{1cm}  [\mathcal{S}_2\mathcal{S}_1] = [\mathcal{B}\mathcal{S}_2] ^T [\mathcal{B}\mathcal{S}_1]
\end{equation}

\subsection{Filter Dynamics} %%%

The filter is therefore derived with the states being $\bm X =\begin{bmatrix}\leftexp{B}{\bm d} & \omega_2 & \omega_3 \end{bmatrix}^{T}$, given that $\bm \omega_{\mathcal{S}/\mathcal{B}} = \leftexp{S}{\begin{bmatrix} \omega_1 & \omega_2 &\omega_3\end{bmatrix}}^T$. The rates of $\mathcal{S}$ relative to the body and inertial frame are related as such: $\bm \omega_{\mathcal{S}/\mathcal{N}} - \bm \omega_{\mathcal{S}/\mathcal{B}}  = \bm \omega_{\mathcal{B}/\mathcal{N}}$. Since $\omega_1$ is unknown, it is set to zero. Furthermore, since the sun heading is considered to be constant in the inertial frame over the period of time required for attitude determination and control, the equation becomes  $- \bar{\bm \omega}_{\mathcal{S}/\mathcal{B}}  = \bar{\bm \omega}_{\mathcal{B}/\mathcal{N}}$.

${\bm \omega}_{\mathcal{S}/\mathcal{B}}$ is estimated directly by the filter, and its skew matrix can be computed by setting $\omega_1$ to zero (in the absence of information). This defines $\tilde{\bm \omega}_{\mathcal{B}/\mathcal{N}}$ as a function of known parameters. The dynamics are therefore given by Equations \eqref{eq:dynSwitch} and \eqref{eq:dynmatSwitch}, where $ \tilde{[\bm d]}(2,3)$ corresponds to the $2^{\text{nd}}$ and $3^{\text{rd}}$ columns of the $ \tilde{[\bm d]}$ matrix.
\begin{align}
\label{eq:dynSwitch}
\bm X' = \bm F(\bm X) &=  \begin{bmatrix}  \leftexp{B}{ \bm d'} \\  \omega_2' \\ \omega_3' \end{bmatrix} =   \begin{bmatrix}  -\leftexp{B}{\bar{\bm \omega}_{\mathcal{B}/\mathcal{N}}} \times \leftexp{B}{\bm d}\\ 0 \\ 0\end{bmatrix}  =   \begin{bmatrix} [\mathcal{B}\mathcal{S}]\leftexp{S}{\begin{bmatrix} 0 \\ \omega_2 \\ \omega_3\end{bmatrix}} \times \leftexp{B}{\bm d}\\ 0 \\ 0\end{bmatrix}
\\\label{eq:dynmatSwitch}
[A]&= \begin{bmatrix} \frac{\partial \bm F (\bm d, t_i)}{\partial \bm X}  \end{bmatrix}  =   \begin{bmatrix} [ \leftexp{B}{\tilde{\bar{\bm \omega}}_{\mathcal{S}/\mathcal{B}}}] & - \tilde{[\bm d]}[\mathcal{B}\mathcal{S}](2,3) \\  [0]_{2\times 3} &  [0]_{2\times 2} \end{bmatrix}
\end{align}
This formulation leads to simple dynamics, much simpler than those of the filter which subtracts the unobservable states, yet can actually estimate the observable of the rate, instead of using past estimates of $\bm d$.

\subsection{Switching Frames} %%%

When switching occurs, the switch matrix $[W]$ can be computed in Equation \eqref{eq:switchMat} using the previously computed DCMs. This equation assumes the switch is going from frame 1 to frame 2 (the reciprocal is equivalent), and $[\mathcal{S}_2 \mathcal{S}_1](2,3)$ corresponds to the $2^{\text{nd}}$ and $3^{\text{rd}}$ columns of the $[\mathcal{S}_2 \mathcal{S}_1]$ matrix.
\begin{equation}\label{eq:switchMat}
[W] = \begin{bmatrix} [I]_{3\times 3} & [0]_{3 \times 2} \\ [0]_{2 \times 3} & [\mathcal{S}_2 \mathcal{S}_1](2,3)\end{bmatrix}
\end{equation}
The new states $\bm X$ and covariance [P] after the switch are therefore given in Equation \eqref{eq:switchEq}
\begin{equation}\label{eq:switchEq}
\bar{\bm X} = [W] \bm X \hspace{2cm} [\bar{P}] = [W] [P] [W]^T
\end{equation}
When writing out the values of the state and covariance, it is necessary to bring it back into the body frame, using the $[\mathcal{B}\mathcal{S}]$ DCM ($\mathcal{S}$ representing the current frame in use).
%
\subsection{Process Noise for Switch-EKF} %%%

The time update of the error covariance matrix from time $t_k$ to $t_{k+1}$ ($\Delta t = t_{k+1} - t_k$)is given in equation \eqref{eq:SNCTime}. The process noise matrix $[Q]$ is added via the $[\Gamma]$ matrix defined in equation \eqref{eq:GammaDef}. Process noise is only added on the accelerations, meaning that $[B] = \begin{bmatrix} [0]_{3 \times 3} \\ [I]_{3 \times 3}\end{bmatrix}$ when there are 6 states.

\begin{equation}\label{eq:SNCTime}
[P]_{k+1} = [\Phi](t_{k+1}, t_k) [P]_k  [\Phi](t_{k+1}, t_k)^T +  [\Gamma](t_{k+1}, t_k) [Q]  [\Gamma](t_{k+1}, t_k) ^T
\end{equation}

\begin{equation}\label{eq:GammaDef}
  [\Gamma](t_{k+1}, t_k) = \int_{t_k}^{t_{k+1}}  [\Phi](t_{k+1}, \tau) [B](\tau) \D \tau
\end{equation}

In others filters (the EKF and the UKF), the second half of the state vector is a direct derivative of the sun heading vector. Regarding state noise compensation, this allowed the approximation in equation \eqref{eq:GammaApprox}, along with the fact that measurements are received frequently with regard to the evolution of the dynamics.

\begin{equation}\label{eq:GammaApprox}
  [\Gamma](t_{k+1}, t_k) =\Delta t \begin{bmatrix} \frac{\Delta t}{2}[I]_{3 \times 3} \\ [I]_{3 \times 3}\end{bmatrix}
\end{equation}

This is not the case for this filter. Indeed, $[\Phi]$ is a 5 by 5 matrix, expanded in equation \eqref{eq:STM} using the fact that $[\Phi](t_{k+1}, \tau) = \frac{\partial \bm X(t_{k+1})}{\partial \bm X(\tau)}$, and that $\bm X = \begin{bmatrix} \bm d & \bar{\bm \omega} \end{bmatrix}^T$. With this, equation \eqref{eq:GammaApprox} can be re-written as equation \eqref{eq:Gamma2}.

\begin{equation}\label{eq:STM}
[\Phi](t_{k+1}, \tau) =    \begin{bmatrix} [\Phi_{1}]_{3\times 3} &  [\Phi_2]_{3 \times 2} \\  [\Phi_3]_{2 \times 3} &  [\Phi_3]_{2 \times 2}\end{bmatrix}=   \begin{bmatrix} \frac{\partial \bm d (t_{k+1})}{\partial \bm d(\tau)} &   \frac{\partial \bm d (t_{k+1})}{\partial \bar{\bm \omega}(\tau)} \\  \frac{\partial \bar{\bm \omega}(t_{k+1})}{\partial \bm d(\tau)} &   \frac{\partial \bar{\bm \omega}(t_{k+1})}{\partial \bar{\bm \omega}(\tau)} \end{bmatrix}
\end{equation}

\begin{equation}\label{eq:Gamma2}
[\Gamma](t_{k+1}, t_k) = \int_{t_k}^{t_{k+1}}  \ \begin{bmatrix} [\Phi_{1}]_{3\times 3} &  [\Phi_2]_{3 \times 2} \\  [\Phi_3]_{2 \times 3} &  [\Phi_3]_{2 \times 2}\end{bmatrix}\begin{bmatrix} [0]_{3 \times 3} \\ [I]_{3 \times 3}\end{bmatrix} \D \tau =  \int_{t_k}^{t_{k+1}}  \begin{bmatrix} [\Phi_2]_{3 \times 2} \\    [\Phi_4]_{2 \times 2}\end{bmatrix} \D \tau
\end{equation}

These submatrices of the state transition matrix now need to be approximated. As before, assuming dense tracking data, $[\Phi_4]_{2 \times 2} =   \frac{\partial \bar{\bm \omega}(t_{k+1})}{\partial \bar{\bm \omega}(\tau)} \approx [I]_{2\times 2}$.
In order to approximate $[\Phi_2]_{3 \times 2}= \frac{\partial \bm d (t_{k+1})}{\partial \bar{\bm \omega}(\tau)}$, the discrete state update is used as seen in equation \eqref{eq:ApproxPhi2}, where $\bm d_i$ is the sun heading at time $\tau$. It is reminded that $\bar{\bm \omega} = \bar{\bm \omega}_{\mathcal{S}/ \mathcal{B}}$, and that $\bar{\bm \omega}_{\mathcal{B}/ \mathcal{N}} = - \bar{\bm \omega} = -\bar{\bm \omega}_{\mathcal{S}/ \mathcal{B}}$.

\begin{align}\label{eq:ApproxPhi2}
&\bm d_{k+1} = \bm d_i - (t_{k+1} - \tau) \tilde{[\bm d_i]} \bar{\bm \omega}_{\mathcal{B}/ \mathcal{N}} \\
&\bm d_{k+1} = \bm d_i + (t_{k+1} - \tau) \tilde{[\bm d_i]} \bar{\bm \omega} \\
\Rightarrow &  \frac{\partial \bm d (t_{k+1})}{\partial \bar{\bm \omega}(\tau)} =  (t_{k+1} - \tau) \tilde{[\bm d_i]}
\end{align}

Therefore, assuming the state does not vary over the time between two updates,  $[\Phi_2]_{3 \times 2}= \frac{\partial \bm d (t_{k+1})}{\partial \bar{\bm \omega}(\tau)}=   (t_{k+1} - \tau) \tilde{[\bm d_i]}$. This leads to the new $[\Gamma]$ matrix in equation \eqref{eq:NewGamma}, which is used for state noise compensation.

\begin{equation}\label{eq:NewGamma}
[\Gamma](t_{k+1}, t_k) = \int_{t_k}^{t_{k+1}}  \begin{bmatrix} [\Phi_2]_{3 \times 2} \\    [\Phi_4]_{2 \times 2}\end{bmatrix} \D \tau = \Delta t \begin{bmatrix} \frac{\Delta t}{2}\tilde{[\bm d_k]} (2,3)\\ [I]_{2 \times 2}\end{bmatrix}
\end{equation}

 \subsection{Measurements}

The measurement model is given in equation \ref{eq:meas}, and the $H$ matrix defined as $H = \left[\frac{\partial \bm G (\bm X, t_i)}{\partial \bm X}\right]^{*}$ is given in equation $\ref{eq:Hmat}$.

In this filter, the only measurements used are from the coarse sun sensor. For the $i^\mathrm{th}$ sensor, the measurement is simply given by the dot product of the sunline heading and the normal to the sensor. This yields easy partial derivatives for the H matrix, which is a matrix formed of the rows of transposed normal vectors (only for those which received a measurement). Hence the $H$ matrix has a changing size depending on the amount of measurements.

\begin{equation}\label{eq:meas}
\bm G_i(\bm X) = \bm n_i \cdot \bm d
\end{equation}

\begin{equation}\label{eq:Hmat}
\bm H(\bm X) = \begin{bmatrix} \bm n_1^T \\ \vdots \\ \bm n_i^T \end{bmatrix}
\end{equation}

\subsection{User initialization}

In order for the filter to run, the user must set a few parameters:

\begin{itemize}
\item The angle threshold under which the coarse sun sensors do not read the measurement:  \\
\texttt{FilterContainer.sensorUseThresh = 0.}
\item The process noise value, for instance: \\
 \texttt{FilterContainer.qProcVal = 0.001}
\item The measurement noise value, for instance: \\
 \texttt{FilterContainer.qObsVal = 0.001}
\item The threshold in the covariance norm leading to the switch from the EKF update to the linear Kalman Filter update (discussed more closely in the Measurement update part):
 \texttt{FilterContainer.ekfSwitch = 5}
\item The initial covariance: \\
 \texttt{Filter.covar =} \\
  \texttt{ [0.4, 0., 0., 0., 0., 0.,\\
   0., 0.4, 0., 0., 0., 0., \\
   0., 0., 0.4, 0., 0., 0.,\\
    0., 0., 0., 0.004, 0., 0.,\\
    0., 0., 0., 0., 0.004, 0., \\
    0., 0., 0., 0., 0., 0.004]}
 \item The initial state :\\
 \texttt{Filter.state =[0.0, 0.0, 1.0, 0.0, 0.0]}
\end{itemize}

The messages must also be set as such:

\begin{itemize}
\item    \texttt{ filterObject.navStateOutMsgName = "sunline$\_$state$\_$estimate"}
\item    \texttt{ filterObject.filtDataOutMsgName = "sunline$\_$filter$\_$data"}
\item   \texttt{ filterObject.cssDataInMsgName = "css$\_$sensors$\_$data"}
\item   \texttt{ filterObject.cssConfInMsgName = "css$\_$config$\_$data"}
\end{itemize}

\subsection{Inputs and Outputs}

The EKF reads in the measurements from the coarse sun sensors. These are under the form of a list of cosine values. Knowing the normals to each of the sensors, we can therefore use them to estimate sun heading.

\section{Filter Algorithm}

Once the filter has been properly setup in the python code, it can go through it's algorithm. This is done according to the algorithms derived in Reference [\citenum{Born}].

\subsubsection*{Initialization}

First the filter is initialized. This can be done at any time during a simulation in order to reset
the filter.

\begin{itemize}
\item Time is set to $t_0$
\item The state $\bm X^*$ is set to the initial state  $\bm X_0^*$
\item The state error $\bm x$ is set to it's initial value $\bm x_0$
\item The covariance $P$ is set to the initial state  $P_0$
\end{itemize}

\subsubsection*{Time Update}

At some time $t_i$, if the update filter method is called, a time update will first be executed.

\begin{itemize}
\item The state is propagated using the dynamics $\bm F$ with initial conditions $\bm X^*(t_{i-1})$
\item Compute the dynamics matrix $A(t) = \left[\frac{\partial \bm F (\bm X, t)}{\partial \bm X}\right]^{*}$ which is evaluated on the reference trajectory
\item Integrate the STM, $\dot{\Phi}(t, t_{i-1}) = A(t) \Phi (t, t_{i-1})$ with initial conditions $\Phi(t_{i-1}, t_{i-1}) = I$
\end{itemize}

This gives us $\bm X^*(t_i)$ and $\Phi(t_{i}, t_{i-1})$.

\subsubsection*{Observation read in}

\underline{If no measurement is read in at time $t_i$:}

\begin{itemize}
\item $\bm X^*(t_i)$ previously computed becomes the most recent reference state
\item $\bm x_i = \bm \bar{x}_i = \Phi(t_{i}, t_{i-1}) x_{i-1}$ is the new state error
\item $P_i = \bar{P}_i =\Phi(t_{i}, t_{i-1}) P_{i-1} \Phi^T(t_{i}, t_{i-1})$ becomes the updated covariance
\end{itemize}
\underline{If a measurement is read in}, the algorithm computes the observation, the observation state matrix, and the Kalman Gain.

\begin{itemize}
\item The observation ($\bm Y_i$) is compared to the observation model, giving the innovation: $\bm y_i = \bm Y_i - G(\bm X_i^*, t_i)$
\item Compute the observation matrix along the reference trajectory: $\tilde{H}_i = \left[\frac{\partial \bm G (\bm X, t_i)}{\partial \bm X}\right]^{*}$
\item Compute the Kalman Gain $K_i = \bar{P}_i \tilde{H_i}^T\left(\tilde{H_i} \bar{P}_i \tilde{H_i}^T + R_i \right)^{-1}$
\end{itemize}

\subsubsection*{Measurement Update}

Depending on the covariance, the filter can either update as a classic, linear Kalman Filter, or as the Extended Kalman filter.
This is done in order to assure robust and fast filter convergence. Indeed in a scenario with a very large initial covariance, the EKF's
change in reference trajectory could delay or inhibit the convergence. In order to remedy this, a few linear updates are performed if the
maximum value in the covariance is greater than a user-set threshold.

\underline{Linear update:}

\begin{itemize}
\item The state error is updated using the time updated value: $\bm x_i =  \bm \bar{x}_i + K_i\left[\bm y_i - \tilde{H}_i \bm \bar{x}_i \right]$
\item The covariance is updated using the Joseph form of the covariance update equation:
$P_i = \left( I - K_i \tilde{H}_i\right) \bar{P}_{i} \left( I - K_i \tilde{H}_i\right)^T + K_i R_i K_i^T$
\item The reference state stays the same, and it's propagated value $\bm X^*(t_i)$ becomes $\bm X^*(t_{i-1})$
\end{itemize}

 \underline{EKF update:}

\begin{itemize}
\item The state error is updated using the innovation and the Kalman Gain: $\bm x_i =   K_i\bm y_i $
\item The reference state is changed by the state error: $\bm X^*(t_i) =  \bm X^*(t_i) + \bm x_i $
\item The covariance is updated using the Joseph form of the covariance update equation:
$P_i = \left( I - K_i \tilde{H}_i\right) \bar{P}_{i} \left( I - K_i \tilde{H}_i\right)^T + K_i R_i K_i^T$
\item The new reference state is now used $\bm X^*(t_i)$ becomes $\bm X^*(t_{i-1})$
\end{itemize}

\section{Test Design}
The unit test for the sunlineEKF module is located in:\\

\noindent
{\tt fswAlgorithms/attDetermination/sunlineEKF/$\_$UnitTest/test$\_$SunlineEKF.py} \\

As well as another python file containing plotting functions:

\noindent
{\tt fswAlgorithms/attDetermination/sunlineEKF/$\_$UnitTest/SunlineEKF$\_$test$\_$utilities.py} \\

The test is split up into 4 subtests, the last one is parametrized in order to test different scenarios. The first test creaks up all of the individual filter methods and tests them individually. The second test verifies that in the case where the state is zeroed out from the start of the simulation, it remains at zero. The third test verifies the behavior of the time update in a general case. The final test is a full filter test.

\subsection{\texttt{sunline$\_$individual$\_$test}}

In each of these individual tests, random inputs are fed to the methods and their values are computed in parallel in python. These two values are then compared to assure that the correct computations are taking place.
\begin{itemize}
\item \underline{Dynamics Matrix}: This method computes the dynamics matrix $A$. Tolerance to absolute error $\epsilon = 10^{-10}$.

\textcolor{ForestGreen}{Passed}
\item \underline{State and STM propagation}: This method propagates the state using the $\bm F$ function as well as the STM using $\dot{\Phi} = A \Phi$. Tolerance to absolute error $\epsilon = 10^{-10}$.

\textcolor{ForestGreen}{Passed}
\item \underline{$H$ and $y$ propagation}: This method computes the $H$ matrix, and compares the measurements to the expected measurements given the state. Tolerance to absolute error $\epsilon = 10^{-10}$.

\textcolor{ForestGreen}{Passed}
\item \underline{Kalman gain}: This method computes the $K$ matrix. Tolerance to absolute error $\epsilon = 10^{-10}$.

\textcolor{ForestGreen}{Passed}
\item \underline{EKF update}: This method performs the measurement update in the case of an EKF. Tolerance to absolute error $\epsilon = 10^{-10}$.

\textcolor{ForestGreen}{Passed}
\item \underline{Linear Update}: This method performs the measurement update in the linear case. Tolerance to absolute error $\epsilon = 10^{-10}$.

\textcolor{ForestGreen}{Passed}
\end{itemize}

\subsection{\texttt{StatePropStatic}}

This test runs the filter with no measurements. It initializes with a zeroed state, and assures that at the end of the simulation all values are still at zero. Plotted results are seen in Figure \ref{fig:StatesExpected}.

Tolerance to absolute error: $\epsilon = 10^{-10}$

\textcolor{ForestGreen}{Passed}

\input{AutoTeX/StatesExpected.tex}

\subsection{\texttt{StatePropVariable}}

This test also takes no measurements in, but gives a random state with rate of change. It then tests that the states and covariance are as expected throughout the time of simulation. Plotted results are seen in Figure \ref{fig:StatesCompare}. We indeed see that the state and covariance for the test and the code overlap perfectly.

Tolerance to absolute error: $\epsilon = 10^{-10}$

\textcolor{ForestGreen}{Passed}
\input{AutoTeX/StatesCompare.tex}

\subsection{\texttt{Full Filter test}}

This test the filter working from start to finish. No measurements are taken in for the first 20 time steps. Then a heading is given through the CSS message. Halfway through the simulation, measurements stop, and 20 time steps later a different heading is read. The filter must be robust and detect this change. This test is parametrized for different test lengths, different initial conditions, different measured headings, and with or without measurement noise. All these are successful.

\vspace{0.2cm}
Tolerance to absolute error without measurement noise: $\epsilon = 10^{-10}$

Tolerance to absolute error with measurement noise: $\epsilon = 10^{-2}$

\textcolor{ForestGreen}{Passed}

Plotted results are seen in Figures \ref{fig:StatesPlot}, \ref{fig:StatesTarget}, and \ref{fig:PostFit}. Figure \ref{fig:StatesPlot} shows the state error and covariance over the run. We see the covariance initially grow, then come down quickly as measurements are used. It grows once again as the measurements stop before bringing the state error back to zero with a change in sun heading.

Figure \ref{fig:StatesTarget} shows the evolution of the state vector compared to the true values. The parts were there is a slight delay is due to the fact that no observations are read in.

Figure \ref{fig:PostFit} shows the post fit residuals for the filter, with the $3\sigma$ measurement noise values. We see that the observations are read in well an that the residuals are brought back down to noise. We do observe a slight bias in the noise. This could be due to the equations of motion, and is not concerning.

\input{AutoTeX/StatesPlot.tex}
\input{AutoTeX/StatesTarget.tex}
\input{AutoTeX/PostFit.tex}

\bibliographystyle{AAS_publication}   % Number the references.
\bibliography{references}   % Use references.bib to resolve the labels.

\end{document}