\documentclass[12pt,a4paper]{amsart}
\usepackage[margin=1in]{geometry}
\usepackage[latin1]{inputenc}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{graphicx}
\usepackage{listings}
\usepackage{hyperref}
\usepackage{url}
\begin{document}
	\title{Deformable Objects Using Fast Lattice Shape Matching}
	\author{Marwan Kallal}
	\date{December 17, 2017}
	\maketitle

	\section{Introduction}
	I attempt to use the results of Rivers and James \cite{rivers} on Fast Lattice Shape Matching (FLSM) to make our already fuzzy bunny deformable as well. I have omitted collision and other physics in the interest of time and am focusing on  deformations upon movements of single particle masses. I will also be using the unoptimized version SLSM (Slow LSM), to focus on the dynamics of these deformations.

	\section{Explanation of FLSM}
	\subsection{Constructing the Lattice}
	The first step to FLSM is to build a lattice, or grid, that encloses the mesh. We start by creating a bounding box around the mesh. From here we round the dimensions of the bounding box up to fit an integral number of grid squares. Each grid square has an associated particle which will be used for movement and other calculations.

	\subsection{Creating Particles}
	From here we need to check which grid squares are actually in contact with the inside of the mesh. We can do this using a standard triangle mesh voxelization algorithm \cite{rosen}. Each particle will have a mass associated with it to properly simulate physical responses. To differentiate between inner and outer particles, we can set the particle mass of outside particles to 0.

	\subsubsection{Assigning Vertices}
	Now that we have our grid and particles, we can assign vertices to their associated particles. This allows us to move the vertices of the mesh as the particles move, essentially sharing the transform of the particle with that vertex. To do this, we can iterate through the vertices of the mesh, and calculate which grid square they lie in. We can then assign that vertex to the particle in that grid square.

	\subsection{Creating Regions}
	Shape matching regions make up the building blocks of the deformation system. We start by specifying a region half-width $w$. This region size will also change the dynamic properties of the mesh, with larger regions making the mesh stiffer. This is because the way that region center of mass is calculated, and therefore the final rotations of the regions will be smaller with the displaced particles having less weight relative to the total. Using masses of 0 for particles outside the mesh is important for making sure that their potential movement doesn't affect the deformation of the object. The regions will also overlap to make sure that the object doesn't flop around and fall apart. We can make a region centered around every particle as long as no part of the region exceeds the range of the bounding box (and thus the lattice we constructed).

	\subsection{Dynamics}

	\subsubsection{Particle Movement}
	Particles are affected by outside forces such as gravity and collisions. This creates movement among the particles, which will then move regions, affecting the final resting position of the particles for that timestep.

	\subsubsection{Region Movement}
	When particles move, this changes the center of mass of the regions. In order to move the region properly and assign goal positions, we need to know both the original center of mass of the region $\mathbf{c}_r^0$ and the newly deformed one $\mathbf{c}_r$. We can calculate them both for each region using:

	\[ \frac{1}{Region Total Mass} \sum_{i \in Region Particles} \tilde{m}_i \mathbf{x}_i\]

	Now we must find a way to rotate and translate the regions in a way that preserves the shape of the particles within the region as much as possible.\cite{rivers} proposes an approximation of the least squares rotation given the position of the deformed particles relative to the original region. This method uses an intermediate matrix value $\mathbf{A_r}$ (shown below) and then a polar decomposition to extract the rotational component.

	\[ \mathbf{A_r} = \sum_{i \in Region} \tilde{m}_i ( \mathbf{x}_i - \mathbf{c}_r ) ( \mathbf{x}_i^0 - \mathbf{c}_r^0 )^\top \]

	From here we can take the polar decomposition of $\mathbf{A_r}$ to obtain the rotational component, $\mathbf{R_{r}}$. To get the final transformation we need for the region, we use the two centers of mass we found earlier, $\mathbf{c}_r^0$ and $\mathbf{c}_r$, and our newly obtained $\mathbf{A_r}$, as shown below:

	\[ \mathbf{T}_r = [\mathbf{R}_r (\mathbf{c_r} - \mathbf{R}_r \mathbf{c}_r^0) ] \]

	\subsubsection{Setting Particle Positions}

	Because we know the regions which each particle belongs to, we can place a goal position $\mathbf{g}$ using the average of the transformations for the regions containing it. We can then use the relative distance between original and goal positions of the particles, as well as the outside forces mentioned above, to find the velocities and then final positions of the particles as shown below.

	\[ \mathbf{v}(t+h) = \mathbf{v}(t) + \frac{\mathbf{g}(t) - \mathbf{x}(t)}{h} + h \frac{\mathbf{Forces}(t)}{m} \]

	\[ \mathbf{x}(t+h) = \mathbf{x}(t) + h \mathbf{v}(t + h) \]

	Now that we have the final positions of the particles, we can move the associated vertices with the same transformations as the particles to make the mesh itself deform realistically.

	\section{What I Learned}
	I learned quite a bit about graphics and deformable objects from this projects. Having worked with the bunny and the dynamic fur before this, I had an idea of how the dynamics might work, with spring like coefficients and the like. I had originally thought of doing a 3 dimensional grid of springs and point masses, like some of the cloth dynamics demos, but wanted to explore a deeper problem. I read through some other papers as well, such as \cite{JamesDoug1999Aart} which described much more complicated, but realistic, ways of deforming objects with volume preservation and many tunable parameters. The problem was that I couldn't fully understand the math they were using, so I moved on. \\
	\indent I finally found the \cite{rivers} paper, which seemed to be much more in my reach mathematically. I first set out to understand the basic building blocks of the problem. It seems similar at first glance to the spring problem, with particles trying to stay within their regions, but it gives you more tunability for floppiness and area of effect when deformed than a straight up spring lattice will.
	\indent One of the more interesting parts of the paper was the fast summation they used to precompute and reuse as many values as possible, and is why the paper is called FLSM. I will cover the way this works in Section 3. This was also good to learn, showing a way that we can significantly reduce processor time when doing summations by finding ways to reuse previously calculated values.

	\section{Optimizations}
	There are many potential optimizations we can make, many of them included in the \cite{rivers} paper.

	\subsection{Fast Summation}
	The main breakthrough of the paper was the introduction of a fast summation algorithm that stores as much information as possible to be reused in future computations.

	\section{My Implementation}
	I began by using the bunny mesh we had been using before. I generated a bounding box and rounded it up to the nearest grid square so that we could have integral particles. You can see this process in \texttt{lattice.h::get\_lattice\_bounds()}. I then assigned each grid square a particle and to that particle a set of vertices to share their transforms with. This is done in \texttt{Lattice::create\_lattice()} which then calls \texttt{Particle::setPosition()}. Next I started to assign regions. For simplicity I ignored the mesh when constructing particles and regions, and just used the expanded bounding box as our object. I created regions with half-width $\mathbf{w} = 1$, as shown in \texttt{particle.h:Region()}. All of the necessary values have been set in the particles. All that's left now is to find the centers of mass of the new regions, use the least squares rotation of the regions and apply those transforms to the particles and associated vertices.
	\subsection{State of Affairs}
	The code is not currently fully functional. I have created what I stated in Section 5, and have some comments detailing the approach for the rest in the code, marked with \texttt{//TODO:}. The most important pieces of pseudocode can be found in \texttt{Lattice::deform\_timestep()}, detailing the procedures to take and values to set in order to deform the bunny.

	\bibliographystyle{plainnat}
	\bibliography{references}

\end{document}


% chap1.tex

\chapter{Introduction and Overview}\label{chap:intro}
\noindent Deep neural networks have demonstrated excellent results in many machine learning tasks [REFERENCES:TBD] and became a default choice for machine learning researchers irrespective of the end result they achieved.
Computer vision , Natural language and Speech processing tasks have scaled to the next level of accuracies using these networks.
\paragraph{Structure:} The core structure of Neural network is the Neurons arranged in a layer manner also known as hidden layer and stack of these layers with interconnected Neurons provides depth to the network. This arrangement of layers is known as network architecture. A simple Neural Network is shown in Figure~\ref{fig:nn_simple_dep}.
Several networks are proposed till date for the range of machine learning tasks.[REFERENCES:TBD]
\paragraph{Training:} Network architecture remains passive until it gets trained for the specific task for which network parameters are estimated for the desired level of prediction performance on data samples outside training data set, also known as Generalization performance.
The most important part is the training strategy which encompass choosing the hyper-parameters to intermediate updates along with learning algorithm.
The number of hyper-parameters termed as annoying knobs to be adjusted [Bengio 2012: Practical recommendations] and famously known as nuisance parameters are quite high. On top of this range of choices for these hyper-parameters make exhaustive search impractical.
Once these parameters are chosen, training can be proceed and the algorithm, which is more often than not is Stochastic gradient descent [References] along with Back propagation [Reference: Rumelhart] as a default choice allows network to evolve from untrained to train network. Each training process has stopping criterion, which is more intuition than rules which guides training progress including stopping point.

\begin{figure}[H]
\centering
	\subfigure{
	\centering
	\includegraphics[scale=0.50]{Images/nn_simple_dep}
}
	\caption{\label{fig:nn_simple_dep} A Simple Neural Network}
	\medskip
	\small
	\begin{flushleft}
	\textit{Four layer fully connected Neural network is shown in \ref{fig:nn_simple_dep}. All Neurons in successive layers are connected to each other, while within layer they are not connected.\\
	Input feeds the input features and propagation of features undergoes transformation or encoding as it passes via layers.
	Intermediate layers are known as hidden layers. Output layer gets the final output in desired format. \\
	In case of classification task it could be class vector representing class probabilities, in case of regression task it is the numerical value and in case of unsupervised task it can be input vector itself. Network parameters gets updated via famous technique called back propagation formulating this problem as optimization problem in which loss occured is minimized iteratively.}
	\end{flushleft}

\end{figure}

Considering all these background the \textbf{Training Life Cycle} of DNN's has 3 main stages as shown in Figure~\ref{fig:dnn_lifecycle}

\begin{enumerate}
	\item Choosing network structure
	\item Selection of hyper parameters and training algorithm
	\item Network updates during training and stopping criterion
\end{enumerate}

\begin{figure}[H]
\centering
	\subfigure{
	\centering
	\includegraphics[scale=0.50]{Images/dnn_training_lifecycle}
}
	\caption{\label{fig:dnn_lifecycle} DNN Training Lifecycle}
	\medskip
	\small
	\begin{flushleft}
	\textit{Training lifecycle is shown in \ref{fig:dnn_lifecycle}. Firstly network choice needs to be made, which would be type of network such as CNN, LSTM, ANN, RNN, Auto encoder and so on, once network type is finalized network configuration has to be decided, such as number of layers, number of neurons in layers, their interconnections, filter size in case of CNN and so on.\\
	secondly comes choice of hyperparameters, which starts with some standard initial choices and during learning gets tuned for better performance.\\
Lastly choice of learning algorithm and update rule for back propagation is to be decided, Here stopping criterion also is crucial which governs the network final learnt parameters.\\
This also allows fine tuning of network structure or hyperparameters as more and more knowledge about underlying data and its performance in current selected environment is known.}
	\end{flushleft}

\end{figure}
\section{Chapters Organization}
This study is divided into 6 chapters, Chapter 2, discusses \textbf{Network architecture}, Chapter 3 discusses \textbf{Hyper-parameters}, Chapter 4 discusses \textbf{Training the Network}, Chapter 5 discusses the \textbf{Insights and Recommendations} from this study. Chapter 6 \textbf{concludes} and poses some \textbf{open questions} for future work.

\subsection{Network types}
Chapter \ref{chap:nwstruct} details different architectures, which can be seen as small survey on the state of the art networks used in deep learning.We will keep them for study purposes.

As we studied the details and interdependencies among the training parameters,so we have used our own network, we call it \textbf{ GsNet}. We recommend to use them for comparative study of this kind.
Table \ref{tab:gsnet} shows the layers configuration of GsNet-2,GsNet-3 and GsNet-5.
\begin{table}[!htbp]
	\centering
	\caption{\textbf{GsNet architectures}}
	\label{tab:gsnet}
	\vspace{2mm}
	\begin{tabular}{|l|l|l|}
		\hline
		\hline
		\textbf{GsNet-2} & \textbf{GsNet-3} & \textbf{GsNet-5} 		\\
		\hline
		\hline
		conv 3x3x64 & conv 3x3x64 	& conv 3x3x64 		\\
		pool 2x2    & pool 2x2 		& pool 2x2			\\
		\hline
		conv 3x3x64 & conv 3x3x64 	& conv 3x3x64 		\\
		pool 2x2    & pool 2x2 		& pool 2x2 			\\
		\hline

		& conv 3x3x64 	& conv 3x3x64 		\\
		& pool 2x2 		& pool 2x2 			\\
		\hline
		&        		& conv 3x3x64 		\\
		&  				& pool 2x2 			\\
		\hline
		&  				& conv 3x3x64 		\\
		&  				& pool 2x2 			\\
		\hline
		dense,128 	& dense,128 	& dense,128  		\\
		\hline
		softmax,c 	& softmax,c 	& softmax,c  		\\
		\hline
		\hline
	\end{tabular}
\end{table}

We have used these networks for our experiments and their analysis. This may bring how depth affects learning. Different architectures in practice are described, which can be seen as small survey on state of the art network architectures in chapter \ref{chap:nwstruct}.

\subsection{Hyper-parameters}
Hyper-parameters are discussed in chapter \ref{chap:hyperparams}.Main parameters, which we studied are as following
\begin{enumerate}
	\item Batch Size
	\item Optimizations
	\item Initializations
\end{enumerate}

Firstly we describe all the different prescribed available techniques for these parameters which can be seen as a small survey of the available studies, experimental results and techniques.

Secondly we present results of almost exhaustive set of parameters configuration. Then best of parameters and configurations are chosen for the next set of experiments.

\subsection{Training the network}

Chapter \ref{chap:training} discusses training the network and study which describes different techniques used in training.We will also explain our training set up which is used for our experiments.

\subsection{Insights and Recommendations}
Chapter \ref{chap:recommendations} provides all insights and analysis of our results. This includes well performing strategies as well as strategies which may didn't  perform well.
Based on these we will describe our recommendations. Also we will explain novel technique which perform well and provide more stability to the learning system.

\subsection{Conclusion}
Chapter \ref{chap:conclusions} concludes with the summary of our study and future direction of this work.

%\iffalse
\section{Notations used}
This section explains the notations used through out this study.
\subsection{DNN Setting}
DNN goal is to approximate a target function $g^*$ for the unknown distribution input $X^*=(x_1,x_2,......x_d)^T \in \mathbb{R}^d$. The target value is $Y^*=(y_1,y_2,........y_s) \in \mathbb{R}^s$. if $\theta^*$ is the parameters associated, then
\begin{equation}
Y^*=g^*(X^*, \theta^*)
\end{equation}

$X^*$ represents entire input data for the underlying input distribution.
Getting $X^*$ is almost impossible, so generally $g^*$ is approximate using the representative input $X$ of size $N$ which is sampled from $X^*$ and hoped to have same distribution as the original input distribution. Let $Y$ is the target value for $X$.

So DNN problem reduces to approximating $ g^*$ using $(X,Y)$ of size $N$. $\theta=(\theta_1,\theta_2...\theta_m) \in \mathbb{R}^m$ represents the parameters which gives best approximation for target function. Finally the DNN has to learn the best $\theta$  such that $g(X,\theta) \sim g^*$.

$g$ represents a chained function in context of DNN as it flows from input to output via hidden layers as shown in \ref{fig:nn_simple_dep}. $g$ as chained function flowing via hidden layers can  be written as
\begin{equation}
g(X)=g^K(g^{K-1}(g^{K-2}......(g^2(g^1(X))).......))
\end{equation}
where $K$ represents total number of hidden layers and $\{g^k, k=1....K\}$ is output of $k^{th}$ layer.Let $\hat{Y}=g(X)$ then lets define a loss function $\mathcal{L}(\hat{Y},Y)$ as the cost it incurs using $g(X)$ to approximate $g^*(X)$ and hence it is also known as cost function.

The gradient of $\mathcal{L}$ with respect to $\theta$ is denoted as $\nabla_{\theta_{t}}\mathcal{L}(\theta_{t})$ at $t^{th}$ iteration. For simplicity we denote this as $\nabla_{\theta_{t}}$, where $\theta_{t}$ denotes the network parameters at iteration/time $t$.

\paragraph{Network Parameters}
%\fi
%\input{Results/cifar100results.tex}
%\input{Results/mnistresults.tex}
%\input{Results/cifar100results.tex}
%\input{Results/cifar10results.tex}

\section{Experimental set up}
We have performed exhaustive set of experiment using standard datasets on Nvidia-Tesla K80 GPU.Regularly results are analyzed to reduce the experiment space and become basis for the next set of experiments.
\subsection{Databases}
Following is the list of datasets used in the experiments:
\begin{enumerate}
	\item MNIST \cite{lecun-mnisthandwrittendigit-2010}
	\item CIFAR-10 \cite{Krizhevsky09learningmultiple}
	\item CIFAR 100 \cite{Krizhevsky09learningmultiple}
\end{enumerate}
\subsection{MNIST}
MNIST dataset has 60000 training samples and 10000 testing samples.It is database of handwritten digits. Sample examples are shown in fig.\ref{fig:mnist_example}

\begin{figure}[H]\label{fig:mnist_example}
	\centering
	\includegraphics[scale=1.0]{Images/mnist}
	\caption{MNIST dataset input images example}
		\medskip
		\small
		MNIST sample images having handwritten digits from 0-9, image size is 28x28 and images are grayscale, shown samples are randomly chosen, 10 for each class.
\end{figure}

\subsection{CIFAR10}
MNIST dataset has50000 training samples and 10000 testing samples.It is database of different objects present in the images. Sample examples are shown in fig.\ref{fig:cifar10_example}

\begin{figure}[H]\label{fig:cifar10_example}
	\centering
	\includegraphics[scale=0.8]{Images/cifar10}
	\caption{CIFAR-10 dataset input images example}
	\medskip
	\small
	CIFAR-10 sample images having different objects present in the images, image size is 32x32 and images are color
\end{figure}

\subsection{CIFAR100}
CIFAR100 dataset has 50000 training samples and 10000 testing samples.It is database of 100 different object classes. Sample examples are shown in fig.\ref{fig:cif00_example}

\begin{figure}[H]\label{fig:cif00_example}
	\centering
	\includegraphics[scale=0.8]{Images/cifar_100}
	\caption{CIFAR-100 dataset input images example}
	\medskip
	\small
	CIFAR-100 sample images consist of database with 100 different object classes, image size is 32x32 and images are color
\end{figure}

\subsection{Software}
Keras \cite{chollet2015} is mainly used for almost all the experiments. Theano \cite{2016arXiv160502688short} is used as main backend for Keras.Python is used as main programming language.

%%%%%%%%%%%%%INITIALIZE VARIABLES%%%%%%%%%%%%%%%%%%
\input{init_variable.tex}


\section{Introduction}
\label{sec:introduction}

Given a mathematical description of a mathematical structure (constants,
functions, relations, and axioms), what should a computer
implementation look like?

For simple cases,
\iflong
the answer is obvious. A
group would have a type whose values represent group elements, as well
as a binary operation that is associative, a constant
neutral element, and a unary inverse operator.

\else % \iflong
like groups, the answer is obvious.
\fi % \iflong
%
But for more interesting structures, especially those arising in
constructive mathematical analysis, the answer is less clear. How do we
implement the real numbers (a Cauchy-complete Archimedean ordered
field)? Or choose the operations for a compact metric
space or a space of smooth functions? Significant research goes into
finding satisfactory representations~\cite{Wei00,TZ98,Bla97,edalat04:_domain_theor_differ_calcul_funct}, while
implementations of exact real arithmetic~\cite{muller01,Lam05a} show that
theory can be put into practice quite successfully.

\iflong
The theory of realizability provides guidance in development of
computable mathematics.  Our work shows that realizability is
not only  a
fundamental tool in logic and computability, but also
has direct application to the design and implementation of programs:
it can
\else % \iflong
Realizability theory can be used to
\fi % \iflong
produce a description of the data structure (a code interface)
directly corresponding to a mathematical specification.
%
\iflong

\fi % \iflong
%
\iflong
However, doing this by hand quickly grows tedious.
Worse, different
but logically equivalent sets of axioms correspond to different,
although interdefinable, interfaces for code. One might then want to
compare several variations, since some interfaces will be more useful
than others in practice.
   % this line intentionally left blank
And
\else
But
\fi % \iflong
few programmers --- even those with strong backgrounds in
mathematics and classical logic --- are familiar with constructive logic or
realizability.
\iflong
Programmers are used to language constructs
describing interfaces (e.g., C++ header files, ML signatures, or Java
interfaces) and to using logical assertions (e.g., preconditions
and postconditions).
\fi

%\bigskip

We have therefore implemented a system, called RZ, to serve as a
bridge between the logical world and the programming
world.\footnote{RZ is publicly available for download at
  \url{http://math.andrej.com/rz/}, together with \iflong
  the abridged\else an extended\fi\ version of this paper
  \iflong presented at CiE 2007 \cite{bauer+:cie07}\fi.} RZ translates
specifications in constructive logic into standard interface code in a
programming language (currently Objective Caml~\cite{ocaml}, but other
languages could be used).

The constructive part of the original specification turns into
interface code, listing types and values to be implemented. The rest
becomes assertions about these types and values. As these assertions have
no computational content, so their constructive and classical meanings
agree, and they can be understood by programmers and mathematicians
accustomed to classical logic.

%\bigskip

RZ was designed as a lightweight system
supporting a rich input language. Although transforming complete
proofs into complete code is possible~\cite{komagata+:tr95}, we have not
implemented this. Other excellent systems, including Coq~\cite{coqart} and
Minlog~\cite{benl98:_proof_theor_work}, can
extract programs from proofs. But they work best managing the entire
task, from specification to code generation. In contrast, interfaces generated by RZ can be
implemented in any fashion as long as the assertions are satisfied.
Code can be written by hand, using imperative, concurrent, and other language features
rather than a ``purely functional'' subset.
\iflong
At the other
extreme, the output of RZ can be viewed as a possible \emph{input} to
a program extraction tool, where the the distinction between
computational and non-computational parts (in Coq these are
$\mathtt{Set}$ and $\mathtt{Prop}$, respectively) has been
automatically determined; a corresponding implementation would then be
provided via theorem-proving and program extraction.
\else %\iflong
Or, the output can serve as a basis for theorem-proving and code
extraction using another system.
\fi
%\bigskip

\iflong
The paper is organized as follows. In
Section~\ref{sec:typed-realizability} we present a version of
realizability which is most suitable for our purposes.
Sections~\ref{sec:spec-sign-assert} and~\ref{sec:input-language}
describe the input and the output language of RZ, while in
Section~\ref{sec:translation} we explain how RZ translates from one to
the other. Various implementation issues are discussed in
Section~\ref{sec:implementation}, and examples of RZ at work are shown
in Section~\ref{sec:examples}. We conclude with remarks on related
work in Section~\ref{sec:related-work}.
\fi %\iflong

An earlier description of our RZ work appears in \cite{bauer+:clase05};
since then, the input syntax and underlying implementation has been
significantly revised and improved, and the support for dependent types and
hoisting is completely new.

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "cie"
%%% End:


%!TEX root = paper.tex
%
% GeoClaw
%
% Lead currently:  Randy LeVeque
%

\subsection{\geoclaw}

The \geoclaw branch of \clawpack was developed to solve the
two-dimensional shallow water equations over topography
for modeling tsunami generation, propagation, and inundation.
The \amrclaw code formed the starting point but it was necessary to make many
modifications to support the requirements of this application, as described
briefly below.  This code originated with the work of George
\cite{dgeorge:masters, dgeorge:phd, dgeorge:jcp}
and was initially called
{\sc TsunamiClaw}.  Later it became clear that many other
geophysical flow applications have similar requirements and the code was
generalized as \geoclaw.

One of the major issues is the treatment of wetting and
drying of grid cells at the margins of the flow. The handling of dry
states in a Riemann solver is difficult to handle robustly, and has gone
through several iterations.
\geoclaw must also be well-balanced in order to preserve steady states, in
particular the ``ocean at rest''.
To achieve this, the source terms in the momentum equations arising from
variations in topography are incorporated into the Riemann solver rather than
using a fractional step splitting approach. This is critical for modeling
waves that have very small amplitudes relative to the variations in the depth
of the ocean.
See \cite{rjl:wbfwave10} for a general discussion of such methods and
\cite{dgeorge:phd, dgeorge:jcp} for details of the Riemann solver used in
\geoclaw.  Other features of
\geoclaw include the ability to solve the equations in latitude--longitude
coordinates on the surface of the sphere, and the incorporation of source terms
modeling bottom friction using a Manning formulation. More details about the
code and tsunami modeling applications can be found in
\cite{BergerGeorgeLeVequeMandli:awr11, LeVequeGeorgeBerger:an11}. In 2011, a
significant effort took place to verify and validate \geoclaw against the US
National Tsunami Hazard Mitigation Program (NTHMP) benchmarks
\cite{GonzalezLeVequeEtAl2011}.
NTHMP approval of the code allows \geoclaw to be used in hazard mapping
projects that are funded by this program or other federal and state agencies,
e.g.
\cite{GonzalezLeVequeEtAl2013a,GonzalezLeVequeEtAl2014}.
One such project is illustrated in \cref{fig:ocosta}.

In addition to a variety of tsunami modeling applications,
\geoclaw has been used to solve dam break problems in steep terrain
\cite{George:Malpasset}, storm surge problems \cite{Mandli:ws} (see \cref
{fig:surge} and \cref{tab:surge_timing}),
and submarine landslides \cite{jhkim:phd}.  The code also
formed the basis for solving the
multi-layer shallow water equations for storm surge modeling
\cite{mandli:phd, Mandli:2013it},
and is currently being extended further to handle debris flow modeling in the
packages D-Claw
\cite{Iverson:2014dc,George:2014gh} (see \cref{fig:meager,fig:oso}).

Nearly one quarter of the files in the \amrclaw source library have to be
modified for \geoclaw. There are currently 113 files in the \amrclaw 2D
library, of which 26 are
replaced by a \geoclaw-specific files of the same name
in the \geoclaw 2D library. For example, to preserve a flat sea surface when
interpolating, it is necessary to interpolate the surface elevation
(topography plus water depth) rather than simply interpolating the depth
component of the solution vector as would normally be done in \amrclaw.
An additional 24 files in the \geoclaw shallow water equations
library handle other complications introduced by the need to model tsunamis
and storm surge.

Several other substantial improvements in the
algorithms implemented in \geoclaw have been made between versions 4.6 and
5.3.0, including:

\begin{itemize}
\item In depth-averaged flow, the wave speed and therefore the CFL
condition depends on the depth.  As a result, flows in shallow water
that have been refined spatially may not need to be refined in time.
This ``variable-time-stepping'' was easily added along with the anisotropic
capabilities that were added to \amrclaw.
\item The ability to specify topography via a set of {\tt
topo} files that may cover overlapping regions at different resolutions has been added.
 The finite volume method requires cell averages of topography, computed by
integrating a piecewise bilinear function constructed from the input {\tt
topo} files over each grid cell.  In \clawpack 5.1.0, this was improved to
allow an arbitrary number of nested {\tt topo} grids.
When adaptive mesh refinement is used,
regridding may take place every few time steps.  Improvements were made
in 5.2.0 so that topography could be copied rather than always being
recomputed in regions where there is an existing old grid.

\item The user can now provide multiple  {\tt dtopo} files that
specify changes to the
initial topography at a series of times.  This is used to specify sea-floor
motion during a tsunamigenic earthquake, but can also be used to specify
submarine landslide motion or a failing dam, for example.
%Major changes were
%made to these algorithms in 5.1.0 to fix a problem in earlier versions that
%resulted in incorrect topography motion in some cases, also resulting in a much
%more robust handling of multiple  {\tt dtopo} files.

\item A number of new Python modules has been developed to assist the user
in working with {\tt topo} and {\tt dtopo} files.  These are documented in
the \clawpack documentation and several of them are illustrated with Jupyter
notebooks found in the \clawpack Gallery.

\item New capabilities were added in 5.0.0 to monitor the maximum of
various flow quantities over a specified time range of a simulation.
This capability is crucial for many applications where the maximum
flow depth at each point, maximum current velocities in a harbor, or
maximum momentum flux (a measure of the hydrodynamic force that would
be exerted by the flow on a structure) is desired.  Arrival time of
the first wave at each point can also be monitored.  Such
capabilities were included in the 4.x version of the code, but were
more limited and did not always perform properly near the edges of
refinement patches.  In Version 5.2 these routines were further
improved and extended.  The user can specify a grid of points on which
to monitor values, and the new code is more flexible in allowing
one-dimensional grids (e.g. a transect), two-dimensional rectangular
grids, or an arbitrary set of points\footnote{Described in
\url{http://www.clawpack.org/fgmax.html}}.

\end{itemize}

\begin{figure}
\hfil\includegraphics[width=0.9\textwidth]{final_submission/Fig4}
\caption{\label{fig:ocosta}
Gray's Harbor showing Westport, WA on southern peninsula.
\revised{(Google map data and image, 2016.)}
(b) Simulation of a potential magnitude 9
Cascadia Subduction Zone event, 40 minutes after the earthquake.
(c) Design for new Ocosta Elementary School in Westport, based in part on
\geoclaw simulations \cite{GonzalezLeVequeEtAl2013a}.
\revised{Image courtesy of TCF Architecture.}
  }
\end{figure}

\begin{figure}[t]
    \centering
    \includegraphics[width=\textwidth]{final_submission/Fig5}
    \caption{(a) A snapshot of a \geoclaw storm surge simulation of
Hurricane Ike at landfall.  (b) Tide gauge data computed from \geoclaw
and {\sc adcirc} along with observed data at the same location.
(c) Computational effort and timings for \geoclaw and {\sc adcirc}.  From
\cite{Mandli:ws}. \label{fig:surge}}
\end{figure}

\begin{table}
    \centering
    \begin{tabular}{|c|c|c|c|}
        \hline
        {\bf Package} & {\bf Threads} & {\bf Wall Time} & {\bf Core Time} \\
        \hline
        {\sc ADCIRC} & 4000 & 35 minutes & 2333 hours \\
        \geoclaw & 4 & 2 hours & 8 hours \\
        \hline
    \end{tabular}
    \caption{
    Computational effort and timings for \geoclaw and {\sc adcirc}.  From
\cite{Mandli:ws}.} \label{tab:surge_timing}
\end{table}

\begin{figure}[t]
\hfil\includegraphics[height=1.9in]{final_submission/Fig6}
%\hfil\includegraphics[height=1.9in]{meagerview.png}
%\hskip 5pt
%\includegraphics[height=1.9in]{meager200_labeled.png}\hfil
\caption{\label{fig:meager}
(a) Photograph of the 2010 Mt.\ Meager debris-flow deposit, from \cite{Allstadt2013}.
(b) Simulated debris flow, from D.\ George.
}
\end{figure}

\begin{figure}[t]
\centerline{\includegraphics[height=2.2in]{final_submission/Fig7}}
\caption{\label{fig:oso}
Observed (yellow line) and computed (blue) landslide at Oso, WA in
2014 \cite{IversonGeorgeEtAl2015}.}
\end{figure}


\chapter{Literature Review}

In induction machine, a power converter and a controller are the three major components of an induction motor drive system. Some of the disciplines related to these components are electric machine design, electric machine modeling, sensing and measurement techniques, signal processing, power electronic design and electric machine control. It is beyond the scope of this research to address all of these areas: it will primarily focus on the issue related to the induction machine control. A conventional low cost volts per hertz or a high performance field oriented controller can be used to control the machine. This chapter reviews the principles of the field orientation control of the induction machines and outline major problems in its design and implementation.

\section{Induction machine control}

The controllers required for induction motor drives can be divided into two major types: a conventional low cost volts per hertz v/f controller and torque controller \cite{Vas}-\cite{Lipo}. In v/f control, the magnitudes of the voltage and frequency are kept in proportion. The performance of the v/f control is not satisfactory, because the rate of change of voltage and frequency has to be low. A sudden acceleration or deceleration of the voltage and frequency can cause a transient change in the current, which can result in drastic problems. Some efforts were made to improve v/f control performance, but none of these improvements could yield a v/f torque controlled drive systems and this made DC motors a prominent choice for variable speed applications. This began to change when the theory of field orientation was introduced by Hasse and Blaschke. Field orientation control is considerably more complicated than DC motor control. The most popular class of the successful controllers uses the vector control technique because it controls both the amplitude and phase of AC excitation. This technique results in an orthogonal spatial orientation of the electromagnetic field and torque, commonly known as Field Oriented Control (FOC).

\section{Field orientation control (FOC) of induction machine}

The concept of field orientation control is used to accomplish a decoupled control of flux and torque. This concept is identical to a dc machine direct torque control that has following requirements \cite{Lipo}:
\begin{itemize}
\item{An independent control of armature current to overcome the effects of armature winding resistance, leakage inductance and induced voltage}
\item{An independent control of constant value of flux}
\end{itemize}

If all of these requirements are met at every instant of time, the torque will follow the current, allowing an immediate torque control and decoupled flux and torque regulation.\\

Next, a two phase d-q model of an induction machine rotating at the synchronous speed is introduced which will help to carry out this decoupled control concept to the induction machine. This model can be summarized by the following equations (see chapter 4 for detail):

\begin{align}
V_{ds}^{\omega}&=R_{s}i_{ds}^{\omega}+\frac{d}{dt}\psi_{ds}^{\omega}-\omega_{e}\psi_{qs}^{\omega}\\
V_{qs}^{\omega}&=R_{s}i_{qs}^{\omega}+\frac{d}{dt}\psi_{qs}^{\omega}+\omega_{e}\psi_{qs}^{\omega}\\
0&=R_{r}i_{dr}^{\omega}+\frac{d}{dt}\psi{dr}^{\omega}-(\omega_{e}-\omega_{r})\psi_{qr}^{\omega}\\
0&=R_{r}i_{qr}^{\omega}+\frac{d}{dt}\psi{qr}^{\omega}+(\omega_{e}-\omega_{r})\psi_{dr}^{\omega}\\
T_{e}&=3p\frac{L_{m}}{L_{r}}(\psi_{dr}^{\omega}i_{qs}^{\omega}-\psi_{qr}^{\omega}i_{ds}^{\omega})
\end{align}

\begin{figure}[h]
\centering
\includegraphics[scale=0.65]{chapter0/phasordig}
\caption{Phasor diagram of the field oriented drive system}
\label{phasor}
\end{figure}

and it is quite significant to synthesize the concept of field-oriented control. In this model it can be seen from the torque expression (2.5) that, if the flux along the q-axis
can be made zero then all the flux is aligned along the d-axis and, therefore, the torque can be instantaneously controlled by controlling the current along q-axis. Then the question will be how it can be guaranteed that all the flux is aligned along the d-axis of the machine. When three-phase voltages are applied to the machine, they produce three-phase fluxes both in the stator and the rotor. The three phase fluxes can be represented in a two phase stationary ($\alpha$-$\beta$) frame. If these two phase fluxes along ($\alpha$-$\beta$) axes are represented by a single-vector then all the machine flux will be aligned along that vector. This vector is commonly specified as d-axis which makes an angle $\theta_{e}$ with the stationary frame $\alpha$-axis, as shown in \autoref{phasor}. The q-axis is set perpendicular to the d-axis. The flux along the q-axis in this case will be obviously zero. The phasor diagram \autoref{phasor} presents these axes. When the machine input currents change sinusoidally in time, the angle $\theta_{e}$ keeps changing. Thus the problem is to know the angle $\theta_{e}$ accurately, so that the d-axis of the d-q frame is locked with the flux vector.\\

The control inputs can be specified in two phase synchronously rotating d-q frame as $i_{ds}^{\omega}$ and $i_{qs}^{\omega}$ such that $i_{ds}^{\omega}$ being aligned with the d-axis or the flux vector. These two phase synchronous control inputs are converted into two phase stationary quantities and then to three phase stationary control inputs. To accomplish this the flux angle $\theta_{e}$ must be known precisely. The angle $\theta_{e}$ can be found either by Direct Field Orientation control (DFO) or by Indirect Field Orientation control (IFO). The controller implemented in this fashion that can achieve a decoupled control of the flux and the torque is known as field oriented controller. The block diagram is shown in the \autoref{foc} In the field-oriented controller the flux can be regulated in the stator, air-gap or rotor flux orientation \cite{Vas}-\cite{Lipo}.

\begin{figure}[h]
\centering
\includegraphics[scale=0.73]{chapter0/foc}
\caption{Field oriented induction motor drive system}
\label{foc}
\end{figure}

The control algorithm for calculation of the rotor flux angle $\theta_{e}$ using IFO control is shown in the \autoref{ifoc}. This algorithm is based on the assumption that, the flux along the q-axis is zero, which forces the command slip velocity to be $\omega_{sl}=i_{qs}^{\omega}/(\tau_{r}i_{ds}^{\omega})$ as a necessary and sufficient condition to guarantee that all the flux is aligned with d-axis and the flux along q-axis is zero. The angle $\theta_{e}$ can then be determined as the sum of the slip and the rotor angles after integrating the respective velocities. This slip angle includes the necessary and sufficient condition for decoupled control of flux and torque. The rotor speed can be measured directly by using an encoder or can be estimated. In case the rotor speed is estimated, the control technique is known as sensorless control. This concept will be studied in detail in the following chapters. \autoref{dfo} shows the control algorithm block diagram for DFO control. In this technique the flux angle $\theta_{e}$ is classically calculated by sensing the air-gap flux through the use of flux sensing coils, or can be calculated by estimating the flux along the d-q axes using the voltage and current signals.

\begin{figure}[h]
\centering
\includegraphics[scale=0.85]{chapter0/ifoc}
\caption{Indirect field oriented drive system}
\label{ifoc}
\end{figure}

\begin{figure}[h]
\centering
\includegraphics[scale=0.8]{chapter0/dfo}
\caption{Direct field oriented drive system}
\label{dfo}
\end{figure}

\subsection{Direct field orientation (DFO)}

The DFO control and sensorless control rely heavily on accurate flux estimation. DFOC is most often used for sensorless control, because the flux observer used to estimate the synchronous speed or angle can also be used to estimate the machine speed. Investigation of ways to estimate the flux and speed of the induction machine has also been extensively studied in the past two decades. Classically, the rotor flux was measured by using a special sensing element, such as Hall effect sensors placed in the air-gap. An advantage of this method is that additional required parameters, $L_{lr}$, $L_{m}$, and $L_{r}$ are not significantly affected by changes in temperature and flux level. However, the disadvantage of this method is that a flux sensor is expensive and needs special installation and maintenance. Another flux and speed estimation technique is saliency based with fundamental or high frequency signal injection. One advantage of saliency technique is that the saliency is not sensitive to actual motor parameters, but this method fails at low and zero speed level. When applied with high frequency signal injection \cite{jansen}, the method may cause torque ripples, and mechanical problems.\\

Gabriel \cite{Gabrie} avoided the special flux sensors and coils by estimating the rotor flux from the terminal quantities (stator voltages and currents). This technique requires the knowledge of the stator resistance along with the stator, rotor leakage inductances and magnetizing inductance. This method is commonly known as the Voltage Model Flux Observer (VMFO). The stator flux in the stationary frame estimated by the equations:

\begin{align}
\psi_{ds}^s&=V_{ds}^s-R_{s}i_{ds}^s\\
\psi_{qs}^s&=V_{qs}^s-R_{s}i_{qs}^s
\end{align}
Then the rotor flux can be expressed as:
\begin{align}
\psi_{dr}^s&=\frac{L_{r}}{L_{m}}(\psi_{ds}^s-\sigma i_{ds}^s)\\
\psi_{qr}^s&=\frac{L_{r}}{L_{m}}(\psi_{qs}^s-\sigma i_{qs}^s)
\end{align}

In this model, integration of the low frequency signals, dominance of stator resistance voltage drop at low speed and leakage inductance variation result in a less precise flux estimation. Integration at low frequency is studied by \cite{jun} and three different alternatives are given. Estimation of rotor flux from the terminal quantities depends on parameters such as stator resistance and leakage inductance. The study of parameter sensitivity shows that the leakage inductance can significantly affect the system performance such as stability, dynamic response, and utilizations of the machine and the inverter.\\

The Current Model Flux Observer (CMFO) is an alternative approach to overcome the problems of leakage inductance and stator resistance at low speed. In this model flux can be estimated as:
\begin{align}
\psi_{dr}^s=-\frac{1}{\tau_{r}}\psi_{dr}^s-\omega_{r}\psi_{qr}^s+\frac{L_{m}}{\tau_{r}}i_{ds}^s\\
\psi_{qr}^s=-\frac{1}{\tau_{r}}\psi_{qr}^s-\omega_{r}\psi_{dr}^s+\frac{L_{m}}{\tau_{r}}i_{qs}^s
\end{align}

However, it does not work well at high speed due to its sensitivity to the rotor resistance. Jansen \cite{pl} did an extensive study on VMFO and CMFO based direct field orientation control, discussed the design and accuracy assessment of various flux observers, compared them, and analyzed the alternative flux observers. To further improve the observer performance, closed-loop rotor flux observers are proposed which use the estimated stator current error \cite{pl,gc} or the estimated stator voltage error \cite{gc} to estimate the rotor flux. Furthermore, Lennart \cite{len} proposed reduced order observers for this task.

\subsection{Indirect field orientation control (IFOC)}

In indirect field orientation, the synchronous speed $\omega_{e}$ is the same as the instantaneous speed of the rotor flux vector $\psi_{dr}^{\omega}$ and the d-axis of the d-q coordinate system is exactly locked on the rotor flux vector (rotor flux vector orientation). This facilities the flux control through the magnetizing current $i_{ds}^{\omega}$ by aligning all the flux with the d-axis while aligning the torque producing component of the current with the q-axis. After decoupling the rotor flux and torque producing component of the current components, the torque can be instantaneously controlled by controlling the current $i_{qs}^{\omega}$. The requirement to align the rotor flux with the d-axis of the d-q coordinate system means that the flux along the q-axis must be zero. This means that the current through the q-axis of the mutual inductance is zero.\\

Based on this restriction $\omega_{sl}$ is :

\begin{align}
\omega_{sl}=\frac{i_{qs}^{\omega}}{(\tau_{r}i_{ds}^{\omega})}
\end{align}

These relations suggest that flux and torque can be controlled independently by specifying d-q axis currents provided the slip frequency is satisfied (2.12) at all instants.\\

The concept of indirect field oriented control developed in the past has been widely studied by researchers during the last two decades. The rotor flux orientation is both the original and usual choice for the indirect orientation control. Also the IFO control can be implemented in the stator and air-gap flux orientation as well. De Doncker \cite{doncker} introduced this concept in his universal field oriented controller. In the air-gap flux the slip and flux relations are coupled equations and the d-axis current does not independently control the flux as it does in the rotor flux orientation. For the constant air-gap flux orientation, the maximum of the produced torque is \%20 less than that of the other two methods \cite{sul}. In the stator flux orientation, the transient reactance is a coupling factor and it varies with the operating conditions of the machine. In addition, Mircea \cite{mirc} shows that among these methods, rotor flux oriented control has linear torque curve. Therefore, the most commonly used choice for IFO is the rotor flux orientation.\\

The IFOC is an open loop, feed forward control in which the slip frequency is fed forward guaranteeing the field orientation. This feed forward control is very sensitive to the rotor open circuit time constant $\tau_{r}$. Therefore, $\tau_{r}$ must be known in order to achieve a decoupled control of torque and flux components by controlling $i_{ds}^{\omega}$ and $i_{qs}^{\omega}$ , respectively. When $\tau_{r}$ is not set correctly, the machine is said to be detuned and the performance will become sluggish due to loss of decoupled control of torque and flux. The measurement of the rotor time constant, its effects on the system performance and its adaptive tuning to the variations resulting during the operation of the machine have been studied extensively in the literature \cite{rl}-\cite{krisn}. Lorenz, Krishnan and Novotny \cite{rl}-\cite{krisn} studied the effect of temperature and saturation level on the rotor time constant and concluded that it can reduce the torque capability of the machine and torque/amps of the machine. The detuning effect becomes more severe in the field-weakening region. Also, it results in a steady-state error and, transient oscillations in the rotor flux and torque. Some of the advanced control techniques such as estimation theory tools and adaptive control tools are also studied to estimate rotor time constant and other motor parameters \cite{ca}-\cite{jm}.\\

\section{Variable speed control using advance control algorithms}

There are two issues in motion control using field oriented controlled (FOC) induction machine drives. One is to make the resulting drive system and the controller robust against parameter deviations and disturbances. The other is to make the system intelligent to adjust the control system itself to environment changes and task requirements. If the speed regulation loop fails to produce the command current correctly, than the desired torque response will not be produced by the induction machine. In addition, such a failure may cause the degradation of slip command. As a result, a satisfactory speed regulation is extremely important not only to produce desired torque performance from the induction machine but also to guarantee the decoupling between control of torque and flux.\\

Conventionally, a PI controller has been used for the speed regulation to generate a command current for last two decades, and accepted by industry because of its simplicity. Even though, a well tuned PI controller performs satisfactorily for a field oriented induction machine during steady state. The speed response of the machine at transient, especially for the variable speed tracking, may sometimes be problematic. In last two decades, alternative control algorithms for the speed regulation were investigated. Among these, fuzzy logic, sliding mode, and adaptive nonlinear control algorithms gained much attention.\\

A traditional rotor flux oriented induction machine drive offers a better control performance but it often requires additional sensors on the machine. This adds to the cost and complexity of the drive system. To avoid these sensors on the machine, many different algorithms are proposed for the last three decades to estimate the rotor flux vector and rotor shaft speed. The recent trend in field oriented control is to use such algorithms based on the terminal quantities of the machine for the estimation of the fluxes and speed. They can easily be applied to any induction machine. Therefore, our focus in this study is also on these algorithms.\\

Before looking into individual approaches, the common problems of the speed and flux estimation are discussed briefly for general field orientation and state estimation algorithms.
 \begin{itemize}
 \item{\textbf{Parameter sensitivity:} One of the important problems of the sensorless control algorithms for the field oriented induction machine drives is the insufficient information about the machine parameters which yield the estimation of some machine parameters along with the sensorless structure. Among these parameters stator resistance, rotor resistance and rotor time constant play more important role than the other parameters since these values are more sensitive to temperature changes. The knowledge of the correct stator resistance $R_{s}$ is important to widen the
operation region toward the lower speed range. Since at low speeds the induced voltage is low and stator resistance voltage drop becomes dominant, a mismatching stator resistance induces instability in the system. On the other hand, errors made in determining the actual value of the rotor resistance $R_{r}$, may cause both instability of the system and speed estimation error proportional to $R_{r}$ \cite{gy}. Also, correct $\tau_{r}$ value is vital decoupling factor in IFOC}

\item{\textbf{Pure Integration:} The other important issue regarding many of the topologies is the integration process inherited from the induction machine dynamics where an integration process is needed to calculate the state variables of the system. However, it is difficult both to decide on the initial value, and prevent the drift of the output of a pure integrator. Usually, to overcome this problem a low-pass filter replaces the integrator.}

\item{\textbf{Overlapping loop Problems:} In a sensorless control system, the control loop and the speed estimation loop may overlap and these loops influence each other. As a result, outputs of both of these loops may not be designed independently,  in some bad cases this dependency may influence the stability or performance of the overall system.}
\end{itemize}
\vspace{1cm}
The algorithms, where terminal quantities of the machine are used to estimate the fluxes and speed of the machine are categorized in two basic groups. First one is ``the open loop observers," in a sense that the on-line model of the machine does not use the feedback correction. Second one is ``the closed loop observers" where the feedback correction is used along with the machine model itself to improve the estimation accuracy. These two basic groups can also be divided further into subgroups based on the control method used. These can be summarized as:\\

Open loop observers based on:\\
- Current model\\
- Voltage model\\
- Full-order observer\\

Closed loop observers based on:\\
- Model Reference Adaptive Systems (MRAS)\\
- Kalman filter techniques\\
- Adaptive observers based on both voltage and current model\\
- Neural network flux and speed estimators\\
- Sliding mode flux and speed estimators\\

Current model based open loop observers use the measured stator currents and rotor velocity. The velocity dependency of the current model is very important since this means that although using the estimated flux eliminates the flux sensor, the position sensor is still required. On the other hand, voltage model based open loop observers \cite{M}-\cite{gc} use the measured stator voltage and current as inputs. A full-order open loop observer can be formed using only the measured stator voltage and rotor velocity as inputs where the stator current appears as an estimated quantity. Because of its dependency on the stator current estimation, the full order observer will not exhibit better performance than the current model. Furthermore, parameter sensitivity and observer gain are the problems to be tuned in a full order observer design \cite{br}. These open-loop observer structures are all based on the induction machine model, and they do not employ any feedback. Therefore, they are quite sensitive to parameter variations, which yield the estimation of some machine parameters along with the sensorless structure.\\
\vspace{1cm}\\

To produce more robust structures to parameter variations some kind of feedback may be helpful. For this purpose many closed loop topologies are proposed using different induction machine models and control methods. Among these MRAS attracts attention and several different algorithms are produced. In MRAS a comparison is made between the outputs of two estimators. The estimator which does not contain the quantity to be estimated can be considered as a reference model of the induction machine. The error between these two estimators is used as an input to an adaptation mechanism. The estimated rotor speed in the adjustable model is changed in such a way that the difference between two estimators converges to zero asymptotically, and the estimated rotor speed will be equal to actual rotor speed. The basics of the analysis and design of MRAS are discussed in \cite{Vas, booksul}. In \cite{gy, J}. In \cite{cs} similar speed estimators are proposed based on the MRAS, and a secondary variable is introduced as the reference quantity by letting the rotor flux through a first-order delay instead of a pure integration to nullify the offset. However, their algorithms produce inaccurate estimated speed if the excitation frequency goes below certain level. In addition these algorithms suffer from the machine parameter uncertainties since the parameter variation in the reference model cannot be corrected. Zhen \cite{lz} proposed an interesting MRAS structure that is built with two mutual MRAS schemes. In this structure, the reference model and the adjustable models are interchangeable. For rotor speed estimation, one model is used as reference model and other model is used as adjustable model. The pure integration is removed from reference model. \cite{me} supported the MRAS scheme with ANN using its training and modeling of non-linear systems. MRAS scheme is also used for the on-line adaptation of the motor parameters in field oriented control techniques \cite{ca, kubota}.\\

Kalman filter (KF) is another method employed to identify the speed and rotor flux of an induction machine based on the measured quantities such as stator current and voltage \cite{kim}. Kalman filter approach is based on the system model and a mathematical model describing the induction motor dynamics for the use of Kalman filter application. Parameter deviations and measurement disturbance are taken into consideration in KF covariance matrices of the KF must be properly initialized. KF works for linear systems and for non linear induction motor model extended Kalman filter (EKF) is used. However, KF approach is computationally intensive and depends on the accuracy of the model of the motor. In the EKF model proposed by  one can estimate rotor fluxes and rotor speed which makes the field orientation. EKF is also used for online parameter estimation of induction motor \cite{Vas, lcz}. Reduced order models are also proposed to shorten and speed up the complex EKF algorithm \cite{ekf}. A new KF technique for non-linear systems, Unscented Kalman Filter (UKF), is applied to induction machine state estimatio, which is a derivative free KF technique which avoids costly calculation of Jacobian matrix, linearization of the estimates \cite{ukf}.\\

Another method used for the sensorless control of induction motor is the neural network technique, which is based on a learning process. It has the advantage of tolerating machine parameter uncertainties. For speed estimation, a two-layered neural network, based on back propagation technique, is used and the neural network outputs are compared with the actual measurement values and error then back- propagated to adjust the weights such that the estimated speed converges to actual one. The neural network based sensorless control algorithms have the advantages of fault-tolerant characteristics. However, because of the neural network learning process these algorithms may suffer from the computational intensity.\\

Another approach is sliding mode control for FOC of induction machine. In the sliding mode technique, the control action is very strong and being switched into either ``on" or ``off" at high frequency. The command signals control directly the power devices. This type of control is also favorable because ``on-off" is the only admissible mode of operation for the power converters. Therefore, it seems more natural to employ the algorithm towards discontinuous control.\\

In addition to the algorithms mentioned above, some of the proposed work is hard to classify because of their combined structure. In \cite{J}, a nonlinear high- gain observer structure is proposed, and it is claimed that with the exact knowledge of stator resistance, flux and speed estimation convergence is guaranteed.

\section{Conclusion}

The literature review of DFOC, IFOC, flux, position and speed estimation and speed control can be summarized as:
\begin{itemize}

\item{The DFOC and IFOC are the methods for instantaneous torque and speed control of an induction motor drive system. These methods can be implemented with or without a speed sensor. An IFOC is synthesized by properly controlled slip frequency which is necessary for the field-orientation}\\
\item{The main problem of an IFO drive system is the rotor time constant deviation. The drive system torque control performance decreases if the rotor time constant is not set precisely. Therefore, on-line estimation is necessary and is one of the main challenges for better performance of an IFOC. Most of the techniques proposed so far either need some special hardware or are very complex with respect to the software and require intensive calculations which put extra burden on the processor.}\\
%\item{The main problem in DFO control is precise rotor flux or position observation. This observation from terminal quantities is more desirable than the one including additional hardware.}\\
\item{Voltage model and current model flux observers are the two most common ways to estimate the flux using the terminal quantities. The voltage model flux observer is dominated by stator IR drop at low speed, whereas the current model flux observer has problems of rotor time constant variations. Also the current model flux observer requires the rotor speed. Therefore, if the flux observer is being used for the sensorless control, an error in the estimated speed will be fed back in to the system. Thus will affect the observer accuracy.}\\
\item{The proposed open-loop observers can be simple in the structure but they are susceptible to variety of errors that become specially detrimental at low stator frequencies, including measurement, noise digital approximation errors, parameter detuning and DC offset in measurements, which ultimately may drive the observer instability.}\\
\item{For the time varying system model problems, closed-loop observers are proposed here feedback correction is used along with the machine model itself to improve the estimation accuracy. The algorithmic complexity and calculation intensity looks higher when compared with former solutions but the recent processors are fast enough to solve these algorithms in real-time applications. They also require a strong mathematical background to deal with.}
\end{itemize}


%% !TEX root = manual.tex

\section{Spyplot Diagrams}
\label{sec:tutorials:spyplot}

Spyplots visualize communication matrices, showing either the number of messages or number of bytes sent between two network endpoints.
They are essentially contour diagrams, where instead of a continuous function $F(x,y)$ we are plotting the communication matrix $M(i,j)$.
An example spyplot is shown for a simple application that only executes an MPI\_Allreduce (Figure \ref{fig:spyplot}).
Larger amounts of data (red) are sent to nearest neighbors while decreasing amounts (blue) are sent to MPI ranks further away.

\begin{figure}[h]
\centering
\includegraphics[width=0.4\textwidth]{figures/spyplot/mpi_spyplot.png}
\caption{Spyplot of Bytes Transferred Between MPI Ranks for MPI\_Allreduce}
\label{fig:spyplot}
\end{figure}

Various spyplots can be activated by boolean parameters in the input file.
The most commonly used are the MPI spyplots, for which you must add

\begin{ViFile}
mpi_spyplot = <fileroot>
\end{ViFile}

After running there will be a .csv and .png file in the folder, with e.g. \inlineshell{fileroot = test}

\begin{ShellCmd}
example> ls
test.png
test.csv
\end{ShellCmd}
\inlineshell{test.png} shows the number of bytes exchanged between MPI ranks.
To extend the analysis you can specify

\begin{ViFile}
network_spyplot = <fileroot>
\end{ViFile}
A new csv/png will appear showing the number of bytes exchanged between physical nodes,
accumulating together all MPI ranks sharing the same node.
This gives a better sense of spatial locality when many MPI ranks are on the same node.


\documentclass[a4paper]{article}

\usepackage[english]{babel}
\usepackage[utf8]{inputenc}
\usepackage{amsmath}
\usepackage{graphicx}
\usepackage[colorinlistoftodos]{todonotes}
\usepackage{hyperref}
\usepackage{listings}
\usepackage[numbers]{natbib}

\usepackage{graphicx}
\usepackage{babel,blindtext}

\usepackage{algorithm}
\usepackage[noend]{algpseudocode}

\usepackage{booktabs} % To thicken table lines

\title{Train a Smartcab How to Drive}

\author{Uirá Caiado}

\date{\today}

\begin{document}

\maketitle

\begin{abstract}
A smartcab is a self-driving car from the not-so-distant future that ferries people from one arbitrary location to another. In this project, I will design the AI driving agent for the smartcab using reinforcement learning. This area of machine learning\footnote{Source: \url{https://en.wikipedia.org/wiki/Reinforcement_learning}} is inspired by behaviorist psychology and consists in training the agent by reward and punishment without needing to specify how the task is to be achieved. The agent should learn an optimal policy for driving on city roads, obeying traffic rules correctly, and trying to reach the destination within a goal time.
\end{abstract}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% INTRODUCTION
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Introduction}
\label{sec:introduction}
In this section, I will present a brief introduction to reinforcement learning and to the problem addressed by this project.

\subsection{Reinforcement Learning}
As explained by \cite{Mohri_2012}, reinforcement learning is the study of planning and learning in a scenario where a learner (or agent) actively interacts with the environment to achieve a particular goal. The achievement of the agent's goal is typically measured by the reward he receives from the environment and which he seeks to maximize.

\cite{Kaelbling_1996} state that the most significant difference between reinforcement learning and supervised learning is that there is no presentation of input/output pairs. Instead, they explained that after choosing an action, the agent is told the immediate reward and the following state, but is not told which action would have been in its best long-term interests. It is necessary for the agent to gather useful experience about the possible system states, actions, transitions and rewards actively to act optimally.

Defining a general formulation of the problem based on a Markov Decision Process (MDP), as proposed by \cite{Mitchell}, the agent can perceive a set $S$ os distinct states of its environment and has a set $A$ of actions that it can perform. So, at each discrete time step $t$, the agent senses the current state $s_t$ and choose to take an action $a_t$. The environment responds by giving the agent a reward $r_t=r(s_t, a_t)$ and by producing the succeeding state $s_{t+1}=\delta(s_t, a_t)$. The functions $r$ and $\delta$ only depend on the current state and action (it is memoryless\footnote{Source: \url{https://en.wikipedia.org/wiki/Markov_process}}), are part of the environment and are not necessarily known to the agent.

The task of the agent is to learn a policy $\pi$ that maps each state to an action ($\pi: S \rightarrow A$), selecting its next action $a_t$ based solely on the current observed state $s_t$, that is $\pi(s_t)=a_t$. The optimal policy, or control strategy, is the one that produces the greatest possible cumulative reward over time. So, stating that:

$$V^{\pi}(s_t)= r_t + \gamma r_{t+1} + \gamma^2 r_{t+1} + ... = \sum_{i=0}^{\infty} \gamma^{i} r_{t+i}$$

Where $V^{\pi}(s_t)$ is also called the discounted cumulative reward and it represents the cumulative value achieved by following an policy $\pi$ from an initial state $s_t$ and $\gamma \in [0, 1]$ is a constant that determines the relative value of delayed versus immediate rewards. If we set $\gamma=0$, only immediate rewards is considered. As $\gamma \rightarrow 1$, future rewards are given greater emphasis relative to immediate reward. The optimal policy $\pi^{*}$ that will maximizes $V^{\pi}(s_t)$ for all states $s$ can be written as:

$$\pi^{*} = \underset{\pi}{\arg \max} \, V^{\pi} (s)\,\,\,\,\,, \,\, \forall s$$

As learning $\pi^{*}: S \rightarrow A$ directly is difficult because the available training data does not provide training examples of the form $(s, a)$, in the section \ref{sec:implement_q_learning} I will implement the Q-learning algorithm for estimating the optimal policy.

\subsection{What Will Be Done}

The goal of this project\footnote{Source: \url{https://goo.gl/BZdyLo}} is to design the AI driving agent for the smartcab, that operates in an idealized grid-like city.

The smartcab is able to sense whether the traffic light is green for its direction of movement and whether there is a car at the intersection on each of the incoming roadways (and which direction they are trying to go). In addition to this, each trip has an associated timer that counts down every time step. If the timer is at 0 and the destination has not been reached, the trip is over, and a new one may start.

It should receive the inputs mentioned above at each time step t, and generate an output move, that consists on to stay put at the current intersection, move one block forward, one block left, or one block right (no backward movement).

The smartcab also should receive a reward for each successfully completed trip. A trip is considered “successfully completed” if the passenger is dropped off at the desired destination within a pre-specified time bound. It also gets a smaller reward for each correct move executed at an intersection. It gets a minor penalty for a wrong move and a larger penalty for violating traffic rules and/or causing an accident.

Based on the rewards and penalties it gets, the agent should learn an optimal policy for driving on city roads, obeying traffic rules correctly, and trying to reach the destination within a goal time.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% IMPLEMENT A BASIC DRIVING AGENT
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Implement a Basic Driving Agent}
\label{sec:implement_driving_agent}
In this section, I will implement a basic driving agent that accepts specified inputs ans produces a valid output.

\subsection{My Grid-Like World}
Let's start by testing the output produced by my current \textit{'world'} that will be used to analyze the results of this project. This \textit{'world'} is a grid-like city, with roads going North-South and East-West. Other vehicles may be present on the streets, but no pedestrians. There is a traffic light at each intersection that can be in one of two states: North-South open or East-West open. US right-of-way rules apply: On a green light, you can turn left only if there is no oncoming traffic at the intersection coming straight. On a red light, you can turn right if there is no oncoming traffic turning left or traffic from the left going straight.

We are told to assume that a higher-level planner assigns a route to the smartcab, splitting it into waypoints at each intersection. The time in this world is quantized and at any instant, the smartcab is at some intersection. Therefore, the next waypoint is always either one block straight ahead, one block left, one block right, one block back or exactly there (reached the destination).Below is a sample of the log file generated by the simulator.

% code snipet
\begin{lstlisting}
<Time>;Environment.reset():Trial set up with start=(4, 3)
                      ,destination=(8, 4), deadline = 25
<Time>;RoutePlanner.route_to(): destination = (8, 4)
<Time>;LearningAgent.update(): deadline = 25,
    inputs = {'light': 'green', 'oncoming': None,
              'right': 'forward', 'left': None},
    action = None, reward = 0.0
...
<Time>;Environment.step(): Primary agent ran out of time!
Trial aborted.
\end{lstlisting}

\subsection{The Basic Agent}
As mentioned before, I am going to implement a Smartcab that processes the following inputs at each time step:

\begin{itemize}
\item \textit{Next waypoint location}: about its current position and heading
\item \textit{Intersection state}: traffic light and presence of cars
\item \textit{Current deadline value}: time steps remaining
\end{itemize}

For the purposes of this project, the agent should produce just some random move, like \textit{'None'}, \textit{'forward'}, \textit{'left'} and \textit{'right'}. I will not implement the correct strategy because it is precisely what my agent is supposed to learn.

For the purposes of this project, the first implementation of the agent should produce just some random move at each time $t$, such that $a_t \in (None,\, forward,\,$
$ left,\, right)$. I will not implement the correct strategy because it is precisely what my agent is supposed to learn. Below I am going to simulate $100$ different trials with the \textit{'enforce\_deadline'} set to False and will save the logs produced by the agent to observe how it performs. In this mode, the agent is given unlimited time to reach the destination.

% code snipet
\begin{lstlisting}
Number of Trials: 100
Times that the agent reached the target location: 69
Times the agent reached the hard deadline: 31
Times the agent successfully reached the target: 21
\end{lstlisting}

As can be seen above, even using just random moves, the agent still was able to reach the target destination roughly $20\%$ of the trials at the deadline stipulated by the Planner. If we considering all the times it reached the location, it was able to complete the route almost $70\%$ of the trials. Now, I am going to produce same basic statistics about the times that it reached the target location.

% \begin{figure}[ht!]
% \centering
% \includegraphics[width=0.75\textwidth]{images/steps_takes.png}
% \caption{\label{fig:boxplot}Number of Steps Used to Reached The Destination}
% \end{figure}

\begin{table}[ht!]
\centering
\begin{tabular}{l|r}
\multicolumn{2}{ c }{Number Of Steps} \\
\midrule
mean  &         59.00 \\
std   &         36.06 \\
min   &          2.00 \\
25\%   &         23.75 \\
50\%   &         60.00 \\
75\%   &         91.00 \\
max   &        130.00 \\
\end{tabular}
\caption{\label{tab:basic_stats}Basic Agent Simulation Statistics}
\end{table}

According to the table \ref{tab:basic_stats}, It took 59 steps on average, and the variation was huge. The agent has taken from 2 steps to 130 to finish the route. In the section \ref{sec:enhance_driving_agent}, I will try to improve that.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% IDENTIFY AND UPDATE STATE
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Identify and Update State}
To complete the implementation of my Basic Agent, I still need to decide how it will represent its state internally. Considering the inputs that it receives before performs any action and, consequently, receives a reward, I initially will use a tuple using all of them to represent the current state of my agent, in the form $(inputs,\, next\_waypoint,\, deadline)$. I believe that it will represent a reasonable set of states that my agent could use to gather more information from the environment.

$Inputs$ is a dictionary with the state of the traffic light for the agents' direction of movement and the direction of the random agents on each of the incoming roadways (if any). $next\_waypoint$ is the direction of the target location and $deadline$, a timer.

I believe that $next\_waypoint$ and $inputs$ are natural choices, given that the first gives the direction of the target and the second provides information about the environment state. The inclusion of $deadline$ might help the agent choose to explore the environment when it has more time available, for example. Later on, we will see how significant this single decision is when dealing with Reinforcement Learning problems.

So, let's count the number of stated that I get when I enforce deadline in my simulation on $100$ trials.

\begin{lstlisting}
number of states in the state space: 891
\end{lstlisting}

\begin{figure}[ht!]
\centering
\includegraphics[width=0.75\textwidth]{images/basic_agent_behaviour.png}
\caption{\label{fig:hist_basic_agent}Count Of Times That Each State Appeared}
\end{figure}

The figure \ref{fig:hist_basic_agent} presents the frequency of the counting of each state. For instance, around $600$ states appeared just two times, more than $100$ appeared four times and so on. Curiously the agent's behavior distribution looks like a log-normal distribution.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% IMPLEMENT Q-LEARNING
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Implement Q-Learning}
\label{sec:implement_q_learning}
In this section, I will briefly explained the concept of Q-learning and implement an algorithm to learn the Q function.

\subsection{The Q-Function}
As mention before, to learn an optimal policy $\pi^{*}$, I can't learn a function $\pi^{*}: S \rightarrow A$ that maps a state to the optimal action directly. There is no such information upfront to be used as training data. Instead, as \cite{Mitchell} explained, the only available information is the sequence of immediate rewards $r(s_i, a_i)$ for $i=1,\, 2,\, 3,\,...$

So, as we are trying to maximize the cumulative rewards $V^{*}(s_t)$ for all states $s$, the agent should prefer $s_1$ over $s_2$ wherever $V^{*}(s_1) > V^{*}(s_2)$. Given that the agent must choose among actions and not states, and it isn't able to perfectly predict the immediate reward and immediate successor for every possible state-action transition, we also must learn $V^{*}$ indirectly.

To solve that, we define a function $Q(s, \, a)$ such that its value is the maximum discounted cumulative reward that can be achieved starting from state $s$ and applying action $a$ as the first action. So, we can write:

$$Q(s, \, a) = r(s, a) + \gamma V^{*}(\delta(s, a))$$

As $\delta(s, a)$ is the state resulting from applying action $a$ to state $s$ (the successor) chosen by following the optimal policy, $V^{*}$ is the cumulative value of the immediate successor state discounted by a factor $\gamma$. Thus,  what we are trying to achieve is

$$\pi^{*}(s) = \underset{a}{\arg \max} Q(s, \, a)$$

Thus, the optimal policy can be obtained even if the agent just uses the current action $a$ and state $s$ and chooses the action that maximizes $Q(s,\, a)$. Also, it is important to notice that the function above implies that the agent can select optimal actions even when it has no knowledge of the functions $r$ and $\delta$. In the next subsection, we will see how \cite{Mitchell} defined a reliable way to estimate training values for $Q$, given only a sequence of immediate rewards $r$.

\subsection{Learning Q}
As we have seen, learning the Q function corresponds to learning the optimal policy. According to \cite{Mohri_2012}, the optimal state-action value function $Q^{*}$ is defined for all $(s, \, a) \in S \times A$ as the expected return for taking the action $a \in A$ at the state $s \in S$, following the optimal policy. So, it can be written as \cite{Mitchell} suggested:

$$V^{*}(s) = \underset{a'}{\arg \max} Q(s, \, a')$$

Using this relationship, we can write a recursive definition of Q function, such that:

$$Q(s, \, a) = r(s, a) + \gamma \, \underset{a'}{\max} \, Q(\delta(s,\, a), \, a')$$

The recursive nature of the function above implies that our agent doesn't know the actual $Q$ function. It just can estimate $Q$, that we will refer as $\hat{Q}$. It will represents is hypothesis $\hat{Q}$ as a large table that attributes each pair $(s\, , \, a)$ to a value for $\hat{Q}(s,\, a)$ - the current hypothesis about the actual but unknown value $Q(s, \, a)$. I will initialize this table with zeros, but it could be filled with random numbers, according to \cite{Mitchell}. Still according to him, the agent repeatedly should do the following:

\begin{algorithm}
\caption{Update Q-table}\label{alg:qtable}
\begin{algorithmic}[1]
\Loop{ Observe the current state $s$ and:}
  \State Choose some action $a$ and execute it
  \State Receive the immediate reward $r = r(s, a)$
  \State initialize the table entry $\hat{Q}(s, \, a)$ to zero if there is no entry $(s, \, a)$
  \State Observe the new state $s' = \delta(s, \,a)$
  \State Updates the table entry for $\hat{Q}(s, \, a)$ following:

    \do $\hat{Q}(s, \, a) \leftarrow r + \gamma \, \underset{a'}{\max}\, \hat{Q}(s', \, a')$
  \State $s \leftarrow s'$
\EndLoop
\end{algorithmic}
\end{algorithm}

Note this training rule is suited to a deterministic Markov decision process (where $r(s,\, a)$ and $\delta(s,\, a)$ are deterministic) and uses the agent's current $\hat{Q}$ values for the new state $s'$ to refine its estimate of $\hat{Q}(s, \,a)$ for the previous state $s$. It is tricky. Let's take a look at its performance.

\begin{lstlisting}
Number of Trials: 100
Times that the agent reached the target location: 55
Times the agent reached the hard deadline: 45
Times the agent successfully reached the target: 55
\end{lstlisting}

The agent is reaching its destination much more frequently than before. When the agent selected its action randomly, it reached its destination just 21\% of the times. Now, it is reaching 55\%. Now the agent is always taking the "safest action":  takes the action with the biggest reward. It is not bad, but looking at the number of explored actions by state in the Q-table produced by this simulation, we can notice something curious

\begin{table}[ht!]
\centering
\begin{tabular}{l|r}
\multicolumn{2}{ c }{Number of Explored Actions} \\
\midrule
1                          &  56.28\% \\
2                          &  21.10\% \\
3                          &  15.83\% \\
4                          &  6.78\% \\
\end{tabular}
\caption{\label{tab:explored_actions}Explored Actions in The Q-Table}
\end{table}

The agent explored just one possible action in $56\%$ of the states in the Q-Table. According to \cite{Mitchell}, there are some conditions to ensure that the algorithm converges toward a $\hat{Q}$ equal to the true $Q$ function. One of them is that the system should be a deterministic MDP. Another is that the agent must select actions in a way that it visits every possible state-action pair infinitely often. Looking at the number of times this implementation of the agent explored all $4$ possibles actions (less than $7\%$ of the states), I would say that it isn't true. In the next section, I will try to improve te agent's performance.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% ENHANCE DRIVING AGENT
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Enhance the Driving Agent}
\label{sec:enhance_driving_agent}
In this section, I will try to improve the driving agent to make it consistently reaches the destination within allotted time and the net reward positive.

\subsection{Experimentation Strategies}
One of the issues of the current strategy is that the agent could over-commit to actions that presented positive $\hat{Q}$ values early in the simulation, failing to explore other actions that could present even higher values. \cite{Mitchell} proposed to use a probabilistic approach to select actions, assigning higher probabilities to action with high $\hat{Q}$ values, but given to every action at least a nonzero probability. So, I will implement the following relation:

$$P(a_i\, | \,s ) = \frac{k ^{\hat{Q}(s, a_i)}}{\sum_j k^{\hat{Q}(s, a_j)}}$$

Where $P(a_i\, | \,s )$ is the probability of selecting the action $a_i$ given the state $s$. The constant $k$ is positive and determines how strongly the selection favors action with high $\hat{Q}$ values. In the figure \ref{fig:exploint_explore} we can see the number of times the algorithm choose to explore when we vary the value of the $k$.

Large values of $k$ assigned a higher probability of the agent exploits what it already has learned. Small values have made the agent favor the exploration of new actions. In the figure \ref{fig:success_runs} we can check how the different values of $k$ impacted on the times the agent completed the routes.

\begin{figure}[ht!]
\minipage{0.49\textwidth}
  \includegraphics[width=\linewidth]{images/exploint_explore.png}
  \caption{\label{fig:exploint_explore}Exploration vs Exploitation}
\endminipage
\minipage{0.49\textwidth}
  \includegraphics[width=\linewidth]{images/success_runs.png}
  \caption{\label{fig:success_runs}On Time Ratio}
\endminipage
\end{figure}

The agent presented a very steep improvement as $k$ was approaching $3$. Above this value, the completed trips ratio just showed a slight growth. Now, let's check how the number of trials changed the average reward received by the agent.

\begin{figure}[ht!]
\centering
\includegraphics[width=0.75\textwidth]{images/cum_reward_by_k.png}
\caption{\label{fig:cum_reward_by_k}Average Cumulative Reward By Value of $k$}
\end{figure}

Clearly, small values of $k$ are worst than large values, as can be seen at the figure \ref{fig:cum_reward_by_k}. As I want that my agent has some probability of exploring new actions, and it seems that $k=3$ is the best choice (the "slope" is not so steep, and the start value is pretty high), I will use this parameter to perform the next simulations.

\subsection{Non-deterministic Rewards and Actions}
The last modification that I want to introduce is when the environment is non-deterministic. As I said before, the algorithm used is appropriate to the deterministic case. On a nondeterministic environment, the reward function $r(s,\,a)$ and action transition function $\delta(s,\,a)$ may have probabilistic outcomes. As there are other agents in the Smartcab world, I believe that it is a proper assumption.

As explained by \cite{Mitchell}, to handle nondeterministic MDPs, it is needed to redefine the value $V^{*}$ to be expressed as a expected value. So, jumping to the $Q$ function, we can re-express $Q$ recursively as:

\begin{equation}
\begin{aligned}
Q(s, \, a) &= E \left [r(s, a) \right ] + E \left [\gamma \, \underset{a'}{\max} \, Q(\delta(s,\, a), \, a')\right ]\\
&= E \left [r(s, a) \right ] + \gamma \,  E \left [\underset{a'}{\max} \, Q(\delta(s,\, a), \, a')\right ]\\
&= E \left [r(s, a) \right ] + \gamma \,  \sum_{s'} P\left(s' \,| \,s, a   \right) \underset{a'}{\max} \, Q(\delta(s,\, a), \, a')
\end{aligned}
\end{equation}

Where $P\left(s' \,| \,s, a   \right)$ is the probability that taking action $a$ in state $s$ will produce the next state $s'$. To implement that, we need to assume that the $\hat{Q}$ values in our table may be wrong, due to the non-deterministic behavior of our environment. So, \cite{Mitchell} suggested to modifying the training rule by introducing a decaying weighted average factor to the current $\hat{Q}$ value and to the revised estimate, such that:

$$\hat{Q}(s, \, a) \leftarrow  (1 - \alpha_{n})\, \hat{Q}_{n-1}(s, \, a) +  \alpha_{n}\left[ r + \gamma \underset{a'}{\max} \hat{Q}_{n-1}(s', \, a') \right ]$$

Where $\alpha_{n} = \frac{1}{1 + visits_{n}(s, \,a)}$ is the decaying factor and $visits_{n}$ is the total number of times this state-action $(s, \, a)$ pair has been visited. So, let's put it in practice.

\begin{lstlisting}
Number of Trials: 100
Times that the agent reached the target location: 58
Times the agent reached the hard deadline: 42
Times the agent successfully reached the target: 58
\end{lstlisting}

Looking just at the times the agent has completed the route, it doesn't look like we made much progress. I will compare the performance of all the agents created so far in the last section.

\newpage
\subsection{The last parameter to Tune: Gamma}
There is just one remain parameter of the model that was not explored yet:  the $\gamma$. As already explained, the $\gamma$ in the $\hat{Q}$ function is a factor used to discount the immediate successor state to be used to update $\hat{Q}$ table. As $\gamma$ approaches $0$, less "memory" the agent has. So, let's check what happens when we vary this parameter about the average cumulative reward of the learning agent:

\begin{figure}[ht!]
\centering
\includegraphics[width=0.75\textwidth]{images/cum_reward_by_gamma.png}
\caption{\label{fig:avg_cum_rwd}Average Cumulative Reward By Value of $\gamma$}
\end{figure}

In the last section, I will use $\gamma = 0.5$ to perform the final tests.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% CONCLUSION
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Conclusion}
\label{sec:conclusion}
In this section, I will compare the performance of the algorithms implemented so far and then I will show how a little change can make a huge difference in a reinforcement learning problem.

\subsection{Final Remarks}
The goal of the whole project was to build an agent that obeys traffic rules correctly and reachs the destination within an allotted time. So, the "perfect" agent would be the one that always come to the destination in the goal time without any penalty.

However, as our environment is non-deterministic, to always reach the destination on time is not reasonable. Also, as the environment penalizes the agent even when it makes a correct move, but in a distinct direction from the target location, it isn't fair to expect that the perfect agent never incurs any penalty.

So, the best one would be the one that learns a reasonable policy fast and receives the maximum average cumulative reward possible. Average because we should weight the rewards received by the number of steps used to avoid discrepancies. In this sense, let's first compare how the modifications made on this project reflected on our agent when we ensure deadline. The improved agent will be set with $\gamma=0.5$ and $k=3$.

\begin{figure}[ht!]
\centering
\includegraphics[width=0.75\textwidth]{images/performance_comparition.png}
\caption{\label{fig:perf_compr}Cumulative Reward By Agent}
\end{figure}

As can be seen above, the improved agent did better than the other implementations. It reached the destination $64\%$ of the trials on time and the final average reward by action was greater than the others. It was achieved by tuning the parameters, including an experimentation strategy and changing the $\hat{Q}$ function to deal with the non-deterministic environment, as explained before. The random agent reached the destination on time just $42\%$ of the trials and the agent that used the deterministic version of $\hat{Q}$ function, $55\%$. Although the final agent is still incurring penalties, it is closer to a perfect agent than it was when using just random moves. In the next subsection, I will show how to change the behavior of the agent radically.

\subsection{Blowing Up Everything: Changing The Intern State}
As pointed out by the Udacity Reviewer, something that I should have taken into account was the size of state space resulted by my choice of agents' intern state representation. I included an Experimentation Strategy and addressed a non-deterministic MDP environment exactly due to the number of states that the agent would need to learn. It turned out being the right choice, as demonstrated in the last subsection.

However, I believe that I underestimated this single decision. My current state is represented by the tuple $(inputs,\, next\_waypoint,\, deadline)$. The $deadline$ is assigned by the Environment in a non-deterministic way, what makes my state space unlikely to learn in $100$ trials. In the figure \ref{fig:perf_compr_2} is possible to see what happens If I discard this single variable in the performance of all my $LearningAgent$.

\begin{figure}[ht!]
\centering
\includegraphics[width=0.75\textwidth]{images/performance_comparition_2.png}
\caption{\label{fig:perf_compr_2}Cumulative Reward By Agent}
\end{figure}

Using this inter-state representation, the final version of the agent is much closer to a perfect agent. It reached the destination at $90\%$ of the trials and learned most of what need to learn in less than $60$ trials.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% REFLECTION
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Reflection}
\label{sec:reflection}
Something really valuable that I learned from reviewers is the importance of the state space representation and how it affects the performance of the learning agent.

Still, it's hard to say how better would be the performance of a perfect agent to contrast to the current implementation, given the stochastic environment. An option would be to implement a learning agent that already knows the traffic rules and just need to learn how to find his way to the destination on time.

\bibliographystyle{plain}
% or try abbrvnat or unsrtnat
\bibliography{bibliography/biblio.bib}
\end{document}