\subsection{Super elasticty of demand}

If elasticity is constant super elasticity is 0.


\chapter{Solving Polynomials}

\section{Unit 3}


\documentclass[12pt,a4paper]{amsart}
\usepackage[margin=1in]{geometry}
\usepackage[latin1]{inputenc}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{graphicx}
\usepackage{listings}
\usepackage{hyperref}
\usepackage{url}
\begin{document}
	\title{Deformable Objects Using Fast Lattice Shape Matching}
	\author{Marwan Kallal}
	\date{December 17, 2017}
	\maketitle

	\section{Introduction}
	I attempt to use the results of Rivers and James \cite{rivers} on Fast Lattice Shape Matching (FLSM) to make our already fuzzy bunny deformable as well. I have omitted collision and other physics in the interest of time and am focusing on  deformations upon movements of single particle masses. I will also be using the unoptimized version SLSM (Slow LSM), to focus on the dynamics of these deformations.

	\section{Explanation of FLSM}
	\subsection{Constructing the Lattice}
	The first step to FLSM is to build a lattice, or grid, that encloses the mesh. We start by creating a bounding box around the mesh. From here we round the dimensions of the bounding box up to fit an integral number of grid squares. Each grid square has an associated particle which will be used for movement and other calculations.

	\subsection{Creating Particles}
	From here we need to check which grid squares are actually in contact with the inside of the mesh. We can do this using a standard triangle mesh voxelization algorithm \cite{rosen}. Each particle will have a mass associated with it to properly simulate physical responses. To differentiate between inner and outer particles, we can set the particle mass of outside particles to 0.

	\subsubsection{Assigning Vertices}
	Now that we have our grid and particles, we can assign vertices to their associated particles. This allows us to move the vertices of the mesh as the particles move, essentially sharing the transform of the particle with that vertex. To do this, we can iterate through the vertices of the mesh, and calculate which grid square they lie in. We can then assign that vertex to the particle in that grid square.

	\subsection{Creating Regions}
	Shape matching regions make up the building blocks of the deformation system. We start by specifying a region half-width $w$. This region size will also change the dynamic properties of the mesh, with larger regions making the mesh stiffer. This is because the way that region center of mass is calculated, and therefore the final rotations of the regions will be smaller with the displaced particles having less weight relative to the total. Using masses of 0 for particles outside the mesh is important for making sure that their potential movement doesn't affect the deformation of the object. The regions will also overlap to make sure that the object doesn't flop around and fall apart. We can make a region centered around every particle as long as no part of the region exceeds the range of the bounding box (and thus the lattice we constructed).

	\subsection{Dynamics}

	\subsubsection{Particle Movement}
	Particles are affected by outside forces such as gravity and collisions. This creates movement among the particles, which will then move regions, affecting the final resting position of the particles for that timestep.

	\subsubsection{Region Movement}
	When particles move, this changes the center of mass of the regions. In order to move the region properly and assign goal positions, we need to know both the original center of mass of the region $\mathbf{c}_r^0$ and the newly deformed one $\mathbf{c}_r$. We can calculate them both for each region using:

	\[ \frac{1}{Region Total Mass} \sum_{i \in Region Particles} \tilde{m}_i \mathbf{x}_i\]

	Now we must find a way to rotate and translate the regions in a way that preserves the shape of the particles within the region as much as possible.\cite{rivers} proposes an approximation of the least squares rotation given the position of the deformed particles relative to the original region. This method uses an intermediate matrix value $\mathbf{A_r}$ (shown below) and then a polar decomposition to extract the rotational component.

	\[ \mathbf{A_r} = \sum_{i \in Region} \tilde{m}_i ( \mathbf{x}_i - \mathbf{c}_r ) ( \mathbf{x}_i^0 - \mathbf{c}_r^0 )^\top \]

	From here we can take the polar decomposition of $\mathbf{A_r}$ to obtain the rotational component, $\mathbf{R_{r}}$. To get the final transformation we need for the region, we use the two centers of mass we found earlier, $\mathbf{c}_r^0$ and $\mathbf{c}_r$, and our newly obtained $\mathbf{A_r}$, as shown below:

	\[ \mathbf{T}_r = [\mathbf{R}_r (\mathbf{c_r} - \mathbf{R}_r \mathbf{c}_r^0) ] \]

	\subsubsection{Setting Particle Positions}

	Because we know the regions which each particle belongs to, we can place a goal position $\mathbf{g}$ using the average of the transformations for the regions containing it. We can then use the relative distance between original and goal positions of the particles, as well as the outside forces mentioned above, to find the velocities and then final positions of the particles as shown below.

	\[ \mathbf{v}(t+h) = \mathbf{v}(t) + \frac{\mathbf{g}(t) - \mathbf{x}(t)}{h} + h \frac{\mathbf{Forces}(t)}{m} \]

	\[ \mathbf{x}(t+h) = \mathbf{x}(t) + h \mathbf{v}(t + h) \]

	Now that we have the final positions of the particles, we can move the associated vertices with the same transformations as the particles to make the mesh itself deform realistically.

	\section{What I Learned}
	I learned quite a bit about graphics and deformable objects from this projects. Having worked with the bunny and the dynamic fur before this, I had an idea of how the dynamics might work, with spring like coefficients and the like. I had originally thought of doing a 3 dimensional grid of springs and point masses, like some of the cloth dynamics demos, but wanted to explore a deeper problem. I read through some other papers as well, such as \cite{JamesDoug1999Aart} which described much more complicated, but realistic, ways of deforming objects with volume preservation and many tunable parameters. The problem was that I couldn't fully understand the math they were using, so I moved on. \\
	\indent I finally found the \cite{rivers} paper, which seemed to be much more in my reach mathematically. I first set out to understand the basic building blocks of the problem. It seems similar at first glance to the spring problem, with particles trying to stay within their regions, but it gives you more tunability for floppiness and area of effect when deformed than a straight up spring lattice will.
	\indent One of the more interesting parts of the paper was the fast summation they used to precompute and reuse as many values as possible, and is why the paper is called FLSM. I will cover the way this works in Section 3. This was also good to learn, showing a way that we can significantly reduce processor time when doing summations by finding ways to reuse previously calculated values.

	\section{Optimizations}
	There are many potential optimizations we can make, many of them included in the \cite{rivers} paper.

	\subsection{Fast Summation}
	The main breakthrough of the paper was the introduction of a fast summation algorithm that stores as much information as possible to be reused in future computations.

	\section{My Implementation}
	I began by using the bunny mesh we had been using before. I generated a bounding box and rounded it up to the nearest grid square so that we could have integral particles. You can see this process in \texttt{lattice.h::get\_lattice\_bounds()}. I then assigned each grid square a particle and to that particle a set of vertices to share their transforms with. This is done in \texttt{Lattice::create\_lattice()} which then calls \texttt{Particle::setPosition()}. Next I started to assign regions. For simplicity I ignored the mesh when constructing particles and regions, and just used the expanded bounding box as our object. I created regions with half-width $\mathbf{w} = 1$, as shown in \texttt{particle.h:Region()}. All of the necessary values have been set in the particles. All that's left now is to find the centers of mass of the new regions, use the least squares rotation of the regions and apply those transforms to the particles and associated vertices.
	\subsection{State of Affairs}
	The code is not currently fully functional. I have created what I stated in Section 5, and have some comments detailing the approach for the rest in the code, marked with \texttt{//TODO:}. The most important pieces of pseudocode can be found in \texttt{Lattice::deform\_timestep()}, detailing the procedures to take and values to set in order to deform the bunny.

	\bibliographystyle{plainnat}
	\bibliography{references}

\end{document}


\chapter{Method}\label{cha:method}
This chapter describes the implementation and application of the theory in more detail.
As such, it will build upon the literature in order to more precisely describe how the project was implemented.
The chapter will also give a description of how the evaluation of the different algorithms was performed.

\section{Implementation}
The full source code of the project can be found at \cite{source-code}.
The implementation was done in C++ with OpenGL and CUDA.
CUDA handled the voxelization and OpenGL was used to render the results of it.
The project was implemented in Linux, but can of course be ported to other platforms.
The implementation details which were not covered in the literature will be explained in the following sections.

\subsection{CUDA-OpenGL Interoperability}
During development, some form of visual feedback was required in order to verify and debug the results of the voxelization.
As such, a CUDA-OpenGL interoperation had to be implemented.
This meant creating a 3D texture in OpenGL and binding it to CUDA in order to write to it.
A simplified source code of this can be seen in \appref{app:cuda-opengl-inter}, which is adapted from~\cite{cuda-opengl-Interoperability}.
This method links the memory of the 3D texture to a CUDA array, meaning no data is duplicated.
This is required, as texture sizes of up to 8 GB were used.

\vfill

\subsection{Floating-Point Voxelization}
The first implementation detail that needs to be explained is how the scanline direction, $d_{sl}$, as shown in \figref{fig:scanline-distance}, was calculated.
Let's call the three vertices of a triangle $v_1$, $v_2$ and $v_3$, which are sorted in the most dominant axis (assumed to be the z-axis).
The scanline direction was calculated in two different ways, depending on how the vertices were positioned.
First, if all vertices were in the same z-slice, any direction would sufficed.
In the implementation, the edge between $v_1$ and $v_3$ was chosen.
If the vertices were not in the same z-slice, the gradient of the triangle with respect to $z$ was used.
This was calculated by first solving the plane equation for $z$:
$$ D = n_x x + n_y y + n_z z \rightarrow z = \frac{D - n_x x - n_y y}{n_z},$$
where $n$ is the normal of the triangle and $D$ describes the position of the plane.
The gradient was then the partial derivatives of the equation with respect to $x$ and $y$:
$$d = (\frac{-n_x}{n_z}, \frac{-n_y}{n_z})$$
Which was the resulting scanline direction.

In both cases, the direction also needed to be normalized, meaning the final direction was
$$d_{sl} = \frac{d}{|d|}.$$

Each iteration of the algorithm, two scanline endpoints were calculated by reverse projecting the scanline direction onto the triangle's edges.
The scanline direction was scaled by $l_i$, which was increased by $l$ each iteration.
Recall from the theory that $l = |d_{sl,x}| + |d_{sl,y}|$.
Using the reverse projection, the endpoints were calculated as
\begin{equation}
v_p = v_1 + d_e\frac{l_i}{d_e \cdot d_{sl}},
\label{eq:inverse-projection}
\end{equation}
where $d_e$ is the normalized direction of the edge it projects to.
Note that this only works with the edges from $v_1$ to $v_2$ and $v_1$ to $v_3$.
So a special case was needed for the edge from $v_2$ to $v_3$.
This was solved by recalculating $v_2$ such that
\begin{equation*}
  \left\{
    \begin{aligned}
      0 &= (v^\prime_2 - v_1) \cdot d_{sl} \\
      v^\prime_2 - v_2 &= (v_3 - v_2)t
    \end{aligned}
  \right.
  .
\end{equation*}
That is, $v^\prime_2 - v_1$ is perpendicular to $d_{sl}$ and $v'_2 - v_2$ is parallel with $v_3 - v_2$.
An example of this can be seen in \figref{fig:triangle-v2}.
Solving the equation resulted in
$$
v^\prime_2 = v_2 + (v_3 - v_2)\frac{(v_1 - v_2) \cdot d_{sl}}{(v_3 - v_2) \cdot d_{sl}}.
$$
This was used instead of $v_1$ in \equref{eq:inverse-projection}, when projecting to the edge between $v_2$ and $v_3$.
\input{fig/triangle-v2.fig}

\subsection{Integer Voxelization}\label{ss:meth-integer-vox}
In the integer version there were two edge cases which created holes in the voxelization.
In order to solve these problems first recall the values $C_{lower}$, $C_{upper}$ and $C_k$ from the theory in \secref{ss:integer-optimal-scanline}.

The first problem occurred when a new slice was started and an example is shown to the left in \figref{fig:integer-miss}.
More specifically, it occurred when the next voxel was behind the current scanline, meaning it should have been included in it.
This was resolved by checking if the value of $C_1$ was on the other side of $C_0$ relative to $C_{end}$.
That is, if $C_0 < C_1 < C_{end}$ or $C_{end} < C_1 < C_0$.
Here $C_{end}$ is defined as the $C_k$ of the last voxel on the triangle's edge.
To test if the two values were on different sides, following equation was evaluated:
$$ (C_1 - C_0) * (C_{end} - C_0) \leq 0.
$$
If this condition was met, either $C_{upper}$ or $C_{lower}$ was set to $C_0$ depending on if $C_{end}$ was greater or less than $C_0$ respectively.
This was done in the same step as when the next endpoint for the scanline is calculated.

The other case where holes occurred was when $(X_1, Y_1)$ for both edges were on different sides of the scanline.
This sometimes happened for triangles which had a very acute angle at $v_1$.
An example of this can be seen to the right in \figref{fig:integer-miss}.
This problem was similarly detected as before by
$$ (C^a_1 - C_0) * (C^b_1 - C_0) \leq 0,$$
where $C^a_1$ and $C^b_1$ are $C_1$ for the two edges.
When this was the case, $\Delta X$ and $\Delta Y$ were set similarly to how the scanline direction was chosen for the floating-point version.
That is, the gradient of the triangle was calculated, but the division by $n_z$ was removed to avoid floating-point operations.
However, this resulted in a scanline direction and not a difference between scanline endpoints.
So it was also rotated by 90$\degree$, as the scanline should always be perpendicular to the scanline direction.
The result was $\Delta X = N_y$ and $\Delta Y = -N_x$, where $N$ is the unnormalized normal of the triangle.
These values were then used when determining the next scanline endpoints.

\input{fig/integer-voxel-miss.fig}

\subsection{Bresenham Algorithm}\label{ss:bresenham-problems}
Worth noting is that Bresenham only works when $\Delta X$ is positive and greater than $\Delta Y$ and $\Delta Z$.
To allow for negative directions, three changes were required.
First, whenever $x$, $y$ or $z$ increased by one, they were instead decreased by one if the difference was negative.
Another change was to set $\Delta X$, $\Delta Y$ and $\Delta Z$ to their absolute value.
Finally, the iteration was terminated when the x-position of the voxel equals $X_1$, where $X_1$ is the last x-position on the line.
This would however miss the last voxel on the line, which was resolved by setting the last voxel after the loop.

To solve the problem where $\Delta X$ has to be greater than the other differences, a variable which keeps track of this axis was introduced.
This variable will be called $A$ and was initialized to
\begin{equation*}
\left\{
\begin{aligned}
  A = 0 \hspace{1cm} &, max(\Delta X, \Delta Y, \Delta Z) = \Delta X \\
  A = 1 \hspace{1cm} &, max(\Delta X, \Delta Y, \Delta Z) = \Delta Y \\
  A = 2 \hspace{1cm} &, max(\Delta X, \Delta Y, \Delta Z) = \Delta Z \\
\end{aligned}
\right.
.
\end{equation*}
Then whenever the $\Delta X$ or $x$ was needed, instead the variable was indexed using $A$.
For example $\Delta [A]$, would give the greatest difference.
The indices of the other two axes was calculated by increasing $A$ by one and two respectively, and then taking the modulus 3 of them.

Another thing missing from the theory is that Bresenham is not 6-connected.
This can be seen in \figref{fig:bresenham-8}.
As the optimal scanline requires 6-connected lines in order to fill the interior, the Bresenham algorithm had to be modified to support this.
The first intuitive way to do this would be to add a voxel at $(x,y,z)$ whenever $y$ or $z$ increases in the algorithm.
This however, caused the voxelization to not follow the line correctly, as seen to the left in \figref{fig:bresenham-8}.
This was solved by checking if the error in $y$ was greater than $\Delta Y$, then instead of voxelizing the right voxel, voxelize the top voxel.
The result can be seen to the right in \figref{fig:bresenham-8}.
The same check was performed on the z-axis.
This modification of the algorithm in \appref{app:3dbresen} can be seen in \appref{app:3dbresen-4}.

The final problem with the Bresenham algorithm was that the integer version of the scanline algorithm required being able to get the next voxel in the line.
The problem was that the 6-connected Bresenham could generate up to three voxels each iteration.
One way to solve the problem would be to run one iteration of Bresenham and store all voxels generated in that iteration in a list.
Then the next time a new voxel is needed, return an unused voxel from the list.
The way it was solved for this thesis however, was to keep track of where the last voxel was returned in the iteration.
Then the next time a voxel was needed, the iteration resumed where it left off last.

\input{fig/bresenham-8.fig}

\input{fig/bresenham-4.fig}

\subsection{Voxel Rendering}\label{ss:voxel_rendering}
In order to test the results of the optimal scanline, some form of rendering was needed.
This thesis only considers rendering which results in cubes in a uniform grid.

One approach would be to iterate through the 3D texture on the CPU and render a cube for every existing voxel.
This would be bad for numerous reasons.
First off, the data created on the GPU would need to be copied over to the CPU, which creates significant overhead.
Especially since the data could get up towards 8 GB in size for the highest resolution.
Secondly, rendering all voxels (even occluded ones) would slow down the rendering significantly.

Another way would be to write each voxel coordinate to a list on the GPU when voxelizing the model.
The data could then be used in a geometry shader, where each coordinate is transformed into a cube.
This avoids the copying to the CPU and requires less data to store the voxelization, since only occupied voxels are stored.
It can, however, render the same voxel multiple times if the voxelization does not keep track of which voxels are already occupied.

Therefore, rendering the scene using raymarching was both deemed easier and a potentially faster method.
The data was not transferred between the CPU and GPU and only the visible voxels were rendered.

To perform the raymarching, it used the RLV algorithm to traverse the voxels.
In this case however, the line was not terminated when it reached the endpoint of the line segment, as it does not exist.
Instead it terminated when the line intersected with an existing voxel or when it had exited the voxel grid.

\section{Evaluation}
When performing the comparisons of the different line algorithms, several variable changes were considered in the experiments.

Firstly, multiple models were tested on, with ranging levels of detail.
The models that were used included the Blender monkey (also called Suzanne)~\cite{blender-monkey}, the Stanford bunny~\cite{stanford-rep} and the Stanford dragon~\cite{stanford-rep}.
These can be seen in \figref{fig:models}.
The triangle count of each model was 3936, 69451 and 871414 respectively.
Suzanne was also subdivided once in Blender~\cite{blender}, in order to increase the triangle count to 3936.

Secondly, the resolution of the voxel grid was varied for each model.
The resolution varied between 128-2048, incremented by powers of 2.
Due to GPU memory limits, resolutions greater than 2048 were not possible.

\input{fig/models.fig}

\subsection{Performance Analysis}
To profile the performance of the algorithms, CUDA events were used~\cite{cuda-profiling}.
To use the CUDA events, two timestamp events were created.
One of them was initialized before the kernel ran using the CUDA function \texttt{cuEventRecord}.
Then after the kernel ran, another event was initialized.
These timestamps were entirely handled by the GPU, meaning the CPU would not wait for the timestamps to be executed.
The function \texttt{cuEventSynchronize}, was used in order to wait for the events to be executed.
The time between two timestamps was then determined by the CUDA call \texttt{cuEventElapsedTime}.

\newpage

\subsection{Error Analysis}
The error was calculated for each of the resolutions, models and combinations of line voxelizations.
Meaning RLV was compared to both ILV and Bresenham, but ILV was also compared to Bresenham.

The comparison was done by first voxelizing using the first algorithm with the voxel value of 1.
Then the voxelization ran again with the other algorithm (with the same 3D texture).
This time the voxel value was determined depending on the current voxel value in that position.
If the value at that position was 1 or 2, the value 2 was written, otherwise the value 3 was written.
The result was a voxelization where the intersection had value 2, the voxels only in the first algorithm had value 1 and the voxels only in the second had value 3.

To calculate the error, a simple CUDA kernel was created which iterated through the whole voxel space and summed up the amount of each of the values.
With these values the error could be calculated using the formulas in \secref{s:error}.


% chap1.tex

\chapter{Introduction and Overview}\label{chap:intro}
\noindent Deep neural networks have demonstrated excellent results in many machine learning tasks [REFERENCES:TBD] and became a default choice for machine learning researchers irrespective of the end result they achieved.
Computer vision , Natural language and Speech processing tasks have scaled to the next level of accuracies using these networks.
\paragraph{Structure:} The core structure of Neural network is the Neurons arranged in a layer manner also known as hidden layer and stack of these layers with interconnected Neurons provides depth to the network. This arrangement of layers is known as network architecture. A simple Neural Network is shown in Figure~\ref{fig:nn_simple_dep}.
Several networks are proposed till date for the range of machine learning tasks.[REFERENCES:TBD]
\paragraph{Training:} Network architecture remains passive until it gets trained for the specific task for which network parameters are estimated for the desired level of prediction performance on data samples outside training data set, also known as Generalization performance.
The most important part is the training strategy which encompass choosing the hyper-parameters to intermediate updates along with learning algorithm.
The number of hyper-parameters termed as annoying knobs to be adjusted [Bengio 2012: Practical recommendations] and famously known as nuisance parameters are quite high. On top of this range of choices for these hyper-parameters make exhaustive search impractical.
Once these parameters are chosen, training can be proceed and the algorithm, which is more often than not is Stochastic gradient descent [References] along with Back propagation [Reference: Rumelhart] as a default choice allows network to evolve from untrained to train network. Each training process has stopping criterion, which is more intuition than rules which guides training progress including stopping point.

\begin{figure}[H]
\centering
	\subfigure{
	\centering
	\includegraphics[scale=0.50]{Images/nn_simple_dep}
}
	\caption{\label{fig:nn_simple_dep} A Simple Neural Network}
	\medskip
	\small
	\begin{flushleft}
	\textit{Four layer fully connected Neural network is shown in \ref{fig:nn_simple_dep}. All Neurons in successive layers are connected to each other, while within layer they are not connected.\\
	Input feeds the input features and propagation of features undergoes transformation or encoding as it passes via layers.
	Intermediate layers are known as hidden layers. Output layer gets the final output in desired format. \\
	In case of classification task it could be class vector representing class probabilities, in case of regression task it is the numerical value and in case of unsupervised task it can be input vector itself. Network parameters gets updated via famous technique called back propagation formulating this problem as optimization problem in which loss occured is minimized iteratively.}
	\end{flushleft}

\end{figure}

Considering all these background the \textbf{Training Life Cycle} of DNN's has 3 main stages as shown in Figure~\ref{fig:dnn_lifecycle}

\begin{enumerate}
	\item Choosing network structure
	\item Selection of hyper parameters and training algorithm
	\item Network updates during training and stopping criterion
\end{enumerate}

\begin{figure}[H]
\centering
	\subfigure{
	\centering
	\includegraphics[scale=0.50]{Images/dnn_training_lifecycle}
}
	\caption{\label{fig:dnn_lifecycle} DNN Training Lifecycle}
	\medskip
	\small
	\begin{flushleft}
	\textit{Training lifecycle is shown in \ref{fig:dnn_lifecycle}. Firstly network choice needs to be made, which would be type of network such as CNN, LSTM, ANN, RNN, Auto encoder and so on, once network type is finalized network configuration has to be decided, such as number of layers, number of neurons in layers, their interconnections, filter size in case of CNN and so on.\\
	secondly comes choice of hyperparameters, which starts with some standard initial choices and during learning gets tuned for better performance.\\
Lastly choice of learning algorithm and update rule for back propagation is to be decided, Here stopping criterion also is crucial which governs the network final learnt parameters.\\
This also allows fine tuning of network structure or hyperparameters as more and more knowledge about underlying data and its performance in current selected environment is known.}
	\end{flushleft}

\end{figure}
\section{Chapters Organization}
This study is divided into 6 chapters, Chapter 2, discusses \textbf{Network architecture}, Chapter 3 discusses \textbf{Hyper-parameters}, Chapter 4 discusses \textbf{Training the Network}, Chapter 5 discusses the \textbf{Insights and Recommendations} from this study. Chapter 6 \textbf{concludes} and poses some \textbf{open questions} for future work.

\subsection{Network types}
Chapter \ref{chap:nwstruct} details different architectures, which can be seen as small survey on the state of the art networks used in deep learning.We will keep them for study purposes.

As we studied the details and interdependencies among the training parameters,so we have used our own network, we call it \textbf{ GsNet}. We recommend to use them for comparative study of this kind.
Table \ref{tab:gsnet} shows the layers configuration of GsNet-2,GsNet-3 and GsNet-5.
\begin{table}[!htbp]
	\centering
	\caption{\textbf{GsNet architectures}}
	\label{tab:gsnet}
	\vspace{2mm}
	\begin{tabular}{|l|l|l|}
		\hline
		\hline
		\textbf{GsNet-2} & \textbf{GsNet-3} & \textbf{GsNet-5} 		\\
		\hline
		\hline
		conv 3x3x64 & conv 3x3x64 	& conv 3x3x64 		\\
		pool 2x2    & pool 2x2 		& pool 2x2			\\
		\hline
		conv 3x3x64 & conv 3x3x64 	& conv 3x3x64 		\\
		pool 2x2    & pool 2x2 		& pool 2x2 			\\
		\hline

		& conv 3x3x64 	& conv 3x3x64 		\\
		& pool 2x2 		& pool 2x2 			\\
		\hline
		&        		& conv 3x3x64 		\\
		&  				& pool 2x2 			\\
		\hline
		&  				& conv 3x3x64 		\\
		&  				& pool 2x2 			\\
		\hline
		dense,128 	& dense,128 	& dense,128  		\\
		\hline
		softmax,c 	& softmax,c 	& softmax,c  		\\
		\hline
		\hline
	\end{tabular}
\end{table}

We have used these networks for our experiments and their analysis. This may bring how depth affects learning. Different architectures in practice are described, which can be seen as small survey on state of the art network architectures in chapter \ref{chap:nwstruct}.

\subsection{Hyper-parameters}
Hyper-parameters are discussed in chapter \ref{chap:hyperparams}.Main parameters, which we studied are as following
\begin{enumerate}
	\item Batch Size
	\item Optimizations
	\item Initializations
\end{enumerate}

Firstly we describe all the different prescribed available techniques for these parameters which can be seen as a small survey of the available studies, experimental results and techniques.

Secondly we present results of almost exhaustive set of parameters configuration. Then best of parameters and configurations are chosen for the next set of experiments.

\subsection{Training the network}

Chapter \ref{chap:training} discusses training the network and study which describes different techniques used in training.We will also explain our training set up which is used for our experiments.

\subsection{Insights and Recommendations}
Chapter \ref{chap:recommendations} provides all insights and analysis of our results. This includes well performing strategies as well as strategies which may didn't  perform well.
Based on these we will describe our recommendations. Also we will explain novel technique which perform well and provide more stability to the learning system.

\subsection{Conclusion}
Chapter \ref{chap:conclusions} concludes with the summary of our study and future direction of this work.

%\iffalse
\section{Notations used}
This section explains the notations used through out this study.
\subsection{DNN Setting}
DNN goal is to approximate a target function $g^*$ for the unknown distribution input $X^*=(x_1,x_2,......x_d)^T \in \mathbb{R}^d$. The target value is $Y^*=(y_1,y_2,........y_s) \in \mathbb{R}^s$. if $\theta^*$ is the parameters associated, then
\begin{equation}
Y^*=g^*(X^*, \theta^*)
\end{equation}

$X^*$ represents entire input data for the underlying input distribution.
Getting $X^*$ is almost impossible, so generally $g^*$ is approximate using the representative input $X$ of size $N$ which is sampled from $X^*$ and hoped to have same distribution as the original input distribution. Let $Y$ is the target value for $X$.

So DNN problem reduces to approximating $ g^*$ using $(X,Y)$ of size $N$. $\theta=(\theta_1,\theta_2...\theta_m) \in \mathbb{R}^m$ represents the parameters which gives best approximation for target function. Finally the DNN has to learn the best $\theta$  such that $g(X,\theta) \sim g^*$.

$g$ represents a chained function in context of DNN as it flows from input to output via hidden layers as shown in \ref{fig:nn_simple_dep}. $g$ as chained function flowing via hidden layers can  be written as
\begin{equation}
g(X)=g^K(g^{K-1}(g^{K-2}......(g^2(g^1(X))).......))
\end{equation}
where $K$ represents total number of hidden layers and $\{g^k, k=1....K\}$ is output of $k^{th}$ layer.Let $\hat{Y}=g(X)$ then lets define a loss function $\mathcal{L}(\hat{Y},Y)$ as the cost it incurs using $g(X)$ to approximate $g^*(X)$ and hence it is also known as cost function.

The gradient of $\mathcal{L}$ with respect to $\theta$ is denoted as $\nabla_{\theta_{t}}\mathcal{L}(\theta_{t})$ at $t^{th}$ iteration. For simplicity we denote this as $\nabla_{\theta_{t}}$, where $\theta_{t}$ denotes the network parameters at iteration/time $t$.

\paragraph{Network Parameters}
%\fi
%\input{Results/cifar100results.tex}
%\input{Results/mnistresults.tex}
%\input{Results/cifar100results.tex}
%\input{Results/cifar10results.tex}

\section{Experimental set up}
We have performed exhaustive set of experiment using standard datasets on Nvidia-Tesla K80 GPU.Regularly results are analyzed to reduce the experiment space and become basis for the next set of experiments.
\subsection{Databases}
Following is the list of datasets used in the experiments:
\begin{enumerate}
	\item MNIST \cite{lecun-mnisthandwrittendigit-2010}
	\item CIFAR-10 \cite{Krizhevsky09learningmultiple}
	\item CIFAR 100 \cite{Krizhevsky09learningmultiple}
\end{enumerate}
\subsection{MNIST}
MNIST dataset has 60000 training samples and 10000 testing samples.It is database of handwritten digits. Sample examples are shown in fig.\ref{fig:mnist_example}

\begin{figure}[H]\label{fig:mnist_example}
	\centering
	\includegraphics[scale=1.0]{Images/mnist}
	\caption{MNIST dataset input images example}
		\medskip
		\small
		MNIST sample images having handwritten digits from 0-9, image size is 28x28 and images are grayscale, shown samples are randomly chosen, 10 for each class.
\end{figure}

\subsection{CIFAR10}
MNIST dataset has50000 training samples and 10000 testing samples.It is database of different objects present in the images. Sample examples are shown in fig.\ref{fig:cifar10_example}

\begin{figure}[H]\label{fig:cifar10_example}
	\centering
	\includegraphics[scale=0.8]{Images/cifar10}
	\caption{CIFAR-10 dataset input images example}
	\medskip
	\small
	CIFAR-10 sample images having different objects present in the images, image size is 32x32 and images are color
\end{figure}

\subsection{CIFAR100}
CIFAR100 dataset has 50000 training samples and 10000 testing samples.It is database of 100 different object classes. Sample examples are shown in fig.\ref{fig:cif00_example}

\begin{figure}[H]\label{fig:cif00_example}
	\centering
	\includegraphics[scale=0.8]{Images/cifar_100}
	\caption{CIFAR-100 dataset input images example}
	\medskip
	\small
	CIFAR-100 sample images consist of database with 100 different object classes, image size is 32x32 and images are color
\end{figure}

\subsection{Software}
Keras \cite{chollet2015} is mainly used for almost all the experiments. Theano \cite{2016arXiv160502688short} is used as main backend for Keras.Python is used as main programming language.

%%%%%%%%%%%%%INITIALIZE VARIABLES%%%%%%%%%%%%%%%%%%
\input{init_variable.tex}


\section{Introduction}
\label{sec:introduction}

Given a mathematical description of a mathematical structure (constants,
functions, relations, and axioms), what should a computer
implementation look like?

For simple cases,
\iflong
the answer is obvious. A
group would have a type whose values represent group elements, as well
as a binary operation that is associative, a constant
neutral element, and a unary inverse operator.

\else % \iflong
like groups, the answer is obvious.
\fi % \iflong
%
But for more interesting structures, especially those arising in
constructive mathematical analysis, the answer is less clear. How do we
implement the real numbers (a Cauchy-complete Archimedean ordered
field)? Or choose the operations for a compact metric
space or a space of smooth functions? Significant research goes into
finding satisfactory representations~\cite{Wei00,TZ98,Bla97,edalat04:_domain_theor_differ_calcul_funct}, while
implementations of exact real arithmetic~\cite{muller01,Lam05a} show that
theory can be put into practice quite successfully.

\iflong
The theory of realizability provides guidance in development of
computable mathematics.  Our work shows that realizability is
not only  a
fundamental tool in logic and computability, but also
has direct application to the design and implementation of programs:
it can
\else % \iflong
Realizability theory can be used to
\fi % \iflong
produce a description of the data structure (a code interface)
directly corresponding to a mathematical specification.
%
\iflong

\fi % \iflong
%
\iflong
However, doing this by hand quickly grows tedious.
Worse, different
but logically equivalent sets of axioms correspond to different,
although interdefinable, interfaces for code. One might then want to
compare several variations, since some interfaces will be more useful
than others in practice.
   % this line intentionally left blank
And
\else
But
\fi % \iflong
few programmers --- even those with strong backgrounds in
mathematics and classical logic --- are familiar with constructive logic or
realizability.
\iflong
Programmers are used to language constructs
describing interfaces (e.g., C++ header files, ML signatures, or Java
interfaces) and to using logical assertions (e.g., preconditions
and postconditions).
\fi

%\bigskip

We have therefore implemented a system, called RZ, to serve as a
bridge between the logical world and the programming
world.\footnote{RZ is publicly available for download at
  \url{http://math.andrej.com/rz/}, together with \iflong
  the abridged\else an extended\fi\ version of this paper
  \iflong presented at CiE 2007 \cite{bauer+:cie07}\fi.} RZ translates
specifications in constructive logic into standard interface code in a
programming language (currently Objective Caml~\cite{ocaml}, but other
languages could be used).

The constructive part of the original specification turns into
interface code, listing types and values to be implemented. The rest
becomes assertions about these types and values. As these assertions have
no computational content, so their constructive and classical meanings
agree, and they can be understood by programmers and mathematicians
accustomed to classical logic.

%\bigskip

RZ was designed as a lightweight system
supporting a rich input language. Although transforming complete
proofs into complete code is possible~\cite{komagata+:tr95}, we have not
implemented this. Other excellent systems, including Coq~\cite{coqart} and
Minlog~\cite{benl98:_proof_theor_work}, can
extract programs from proofs. But they work best managing the entire
task, from specification to code generation. In contrast, interfaces generated by RZ can be
implemented in any fashion as long as the assertions are satisfied.
Code can be written by hand, using imperative, concurrent, and other language features
rather than a ``purely functional'' subset.
\iflong
At the other
extreme, the output of RZ can be viewed as a possible \emph{input} to
a program extraction tool, where the the distinction between
computational and non-computational parts (in Coq these are
$\mathtt{Set}$ and $\mathtt{Prop}$, respectively) has been
automatically determined; a corresponding implementation would then be
provided via theorem-proving and program extraction.
\else %\iflong
Or, the output can serve as a basis for theorem-proving and code
extraction using another system.
\fi
%\bigskip

\iflong
The paper is organized as follows. In
Section~\ref{sec:typed-realizability} we present a version of
realizability which is most suitable for our purposes.
Sections~\ref{sec:spec-sign-assert} and~\ref{sec:input-language}
describe the input and the output language of RZ, while in
Section~\ref{sec:translation} we explain how RZ translates from one to
the other. Various implementation issues are discussed in
Section~\ref{sec:implementation}, and examples of RZ at work are shown
in Section~\ref{sec:examples}. We conclude with remarks on related
work in Section~\ref{sec:related-work}.
\fi %\iflong

An earlier description of our RZ work appears in \cite{bauer+:clase05};
since then, the input syntax and underlying implementation has been
significantly revised and improved, and the support for dependent types and
hoisting is completely new.

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "cie"
%%% End:


%!TEX root = paper.tex
%
% GeoClaw
%
% Lead currently:  Randy LeVeque
%

\subsection{\geoclaw}

The \geoclaw branch of \clawpack was developed to solve the
two-dimensional shallow water equations over topography
for modeling tsunami generation, propagation, and inundation.
The \amrclaw code formed the starting point but it was necessary to make many
modifications to support the requirements of this application, as described
briefly below.  This code originated with the work of George
\cite{dgeorge:masters, dgeorge:phd, dgeorge:jcp}
and was initially called
{\sc TsunamiClaw}.  Later it became clear that many other
geophysical flow applications have similar requirements and the code was
generalized as \geoclaw.

One of the major issues is the treatment of wetting and
drying of grid cells at the margins of the flow. The handling of dry
states in a Riemann solver is difficult to handle robustly, and has gone
through several iterations.
\geoclaw must also be well-balanced in order to preserve steady states, in
particular the ``ocean at rest''.
To achieve this, the source terms in the momentum equations arising from
variations in topography are incorporated into the Riemann solver rather than
using a fractional step splitting approach. This is critical for modeling
waves that have very small amplitudes relative to the variations in the depth
of the ocean.
See \cite{rjl:wbfwave10} for a general discussion of such methods and
\cite{dgeorge:phd, dgeorge:jcp} for details of the Riemann solver used in
\geoclaw.  Other features of
\geoclaw include the ability to solve the equations in latitude--longitude
coordinates on the surface of the sphere, and the incorporation of source terms
modeling bottom friction using a Manning formulation. More details about the
code and tsunami modeling applications can be found in
\cite{BergerGeorgeLeVequeMandli:awr11, LeVequeGeorgeBerger:an11}. In 2011, a
significant effort took place to verify and validate \geoclaw against the US
National Tsunami Hazard Mitigation Program (NTHMP) benchmarks
\cite{GonzalezLeVequeEtAl2011}.
NTHMP approval of the code allows \geoclaw to be used in hazard mapping
projects that are funded by this program or other federal and state agencies,
e.g.
\cite{GonzalezLeVequeEtAl2013a,GonzalezLeVequeEtAl2014}.
One such project is illustrated in \cref{fig:ocosta}.

In addition to a variety of tsunami modeling applications,
\geoclaw has been used to solve dam break problems in steep terrain
\cite{George:Malpasset}, storm surge problems \cite{Mandli:ws} (see \cref
{fig:surge} and \cref{tab:surge_timing}),
and submarine landslides \cite{jhkim:phd}.  The code also
formed the basis for solving the
multi-layer shallow water equations for storm surge modeling
\cite{mandli:phd, Mandli:2013it},
and is currently being extended further to handle debris flow modeling in the
packages D-Claw
\cite{Iverson:2014dc,George:2014gh} (see \cref{fig:meager,fig:oso}).

Nearly one quarter of the files in the \amrclaw source library have to be
modified for \geoclaw. There are currently 113 files in the \amrclaw 2D
library, of which 26 are
replaced by a \geoclaw-specific files of the same name
in the \geoclaw 2D library. For example, to preserve a flat sea surface when
interpolating, it is necessary to interpolate the surface elevation
(topography plus water depth) rather than simply interpolating the depth
component of the solution vector as would normally be done in \amrclaw.
An additional 24 files in the \geoclaw shallow water equations
library handle other complications introduced by the need to model tsunamis
and storm surge.

Several other substantial improvements in the
algorithms implemented in \geoclaw have been made between versions 4.6 and
5.3.0, including:

\begin{itemize}
\item In depth-averaged flow, the wave speed and therefore the CFL
condition depends on the depth.  As a result, flows in shallow water
that have been refined spatially may not need to be refined in time.
This ``variable-time-stepping'' was easily added along with the anisotropic
capabilities that were added to \amrclaw.
\item The ability to specify topography via a set of {\tt
topo} files that may cover overlapping regions at different resolutions has been added.
 The finite volume method requires cell averages of topography, computed by
integrating a piecewise bilinear function constructed from the input {\tt
topo} files over each grid cell.  In \clawpack 5.1.0, this was improved to
allow an arbitrary number of nested {\tt topo} grids.
When adaptive mesh refinement is used,
regridding may take place every few time steps.  Improvements were made
in 5.2.0 so that topography could be copied rather than always being
recomputed in regions where there is an existing old grid.

\item The user can now provide multiple  {\tt dtopo} files that
specify changes to the
initial topography at a series of times.  This is used to specify sea-floor
motion during a tsunamigenic earthquake, but can also be used to specify
submarine landslide motion or a failing dam, for example.
%Major changes were
%made to these algorithms in 5.1.0 to fix a problem in earlier versions that
%resulted in incorrect topography motion in some cases, also resulting in a much
%more robust handling of multiple  {\tt dtopo} files.

\item A number of new Python modules has been developed to assist the user
in working with {\tt topo} and {\tt dtopo} files.  These are documented in
the \clawpack documentation and several of them are illustrated with Jupyter
notebooks found in the \clawpack Gallery.

\item New capabilities were added in 5.0.0 to monitor the maximum of
various flow quantities over a specified time range of a simulation.
This capability is crucial for many applications where the maximum
flow depth at each point, maximum current velocities in a harbor, or
maximum momentum flux (a measure of the hydrodynamic force that would
be exerted by the flow on a structure) is desired.  Arrival time of
the first wave at each point can also be monitored.  Such
capabilities were included in the 4.x version of the code, but were
more limited and did not always perform properly near the edges of
refinement patches.  In Version 5.2 these routines were further
improved and extended.  The user can specify a grid of points on which
to monitor values, and the new code is more flexible in allowing
one-dimensional grids (e.g. a transect), two-dimensional rectangular
grids, or an arbitrary set of points\footnote{Described in
\url{http://www.clawpack.org/fgmax.html}}.

\end{itemize}

\begin{figure}
\hfil\includegraphics[width=0.9\textwidth]{final_submission/Fig4}
\caption{\label{fig:ocosta}
Gray's Harbor showing Westport, WA on southern peninsula.
\revised{(Google map data and image, 2016.)}
(b) Simulation of a potential magnitude 9
Cascadia Subduction Zone event, 40 minutes after the earthquake.
(c) Design for new Ocosta Elementary School in Westport, based in part on
\geoclaw simulations \cite{GonzalezLeVequeEtAl2013a}.
\revised{Image courtesy of TCF Architecture.}
  }
\end{figure}

\begin{figure}[t]
    \centering
    \includegraphics[width=\textwidth]{final_submission/Fig5}
    \caption{(a) A snapshot of a \geoclaw storm surge simulation of
Hurricane Ike at landfall.  (b) Tide gauge data computed from \geoclaw
and {\sc adcirc} along with observed data at the same location.
(c) Computational effort and timings for \geoclaw and {\sc adcirc}.  From
\cite{Mandli:ws}. \label{fig:surge}}
\end{figure}

\begin{table}
    \centering
    \begin{tabular}{|c|c|c|c|}
        \hline
        {\bf Package} & {\bf Threads} & {\bf Wall Time} & {\bf Core Time} \\
        \hline
        {\sc ADCIRC} & 4000 & 35 minutes & 2333 hours \\
        \geoclaw & 4 & 2 hours & 8 hours \\
        \hline
    \end{tabular}
    \caption{
    Computational effort and timings for \geoclaw and {\sc adcirc}.  From
\cite{Mandli:ws}.} \label{tab:surge_timing}
\end{table}

\begin{figure}[t]
\hfil\includegraphics[height=1.9in]{final_submission/Fig6}
%\hfil\includegraphics[height=1.9in]{meagerview.png}
%\hskip 5pt
%\includegraphics[height=1.9in]{meager200_labeled.png}\hfil
\caption{\label{fig:meager}
(a) Photograph of the 2010 Mt.\ Meager debris-flow deposit, from \cite{Allstadt2013}.
(b) Simulated debris flow, from D.\ George.
}
\end{figure}

\begin{figure}[t]
\centerline{\includegraphics[height=2.2in]{final_submission/Fig7}}
\caption{\label{fig:oso}
Observed (yellow line) and computed (blue) landslide at Oso, WA in
2014 \cite{IversonGeorgeEtAl2015}.}
\end{figure}


\chapter{Literature Review}

In induction machine, a power converter and a controller are the three major components of an induction motor drive system. Some of the disciplines related to these components are electric machine design, electric machine modeling, sensing and measurement techniques, signal processing, power electronic design and electric machine control. It is beyond the scope of this research to address all of these areas: it will primarily focus on the issue related to the induction machine control. A conventional low cost volts per hertz or a high performance field oriented controller can be used to control the machine. This chapter reviews the principles of the field orientation control of the induction machines and outline major problems in its design and implementation.

\section{Induction machine control}

The controllers required for induction motor drives can be divided into two major types: a conventional low cost volts per hertz v/f controller and torque controller \cite{Vas}-\cite{Lipo}. In v/f control, the magnitudes of the voltage and frequency are kept in proportion. The performance of the v/f control is not satisfactory, because the rate of change of voltage and frequency has to be low. A sudden acceleration or deceleration of the voltage and frequency can cause a transient change in the current, which can result in drastic problems. Some efforts were made to improve v/f control performance, but none of these improvements could yield a v/f torque controlled drive systems and this made DC motors a prominent choice for variable speed applications. This began to change when the theory of field orientation was introduced by Hasse and Blaschke. Field orientation control is considerably more complicated than DC motor control. The most popular class of the successful controllers uses the vector control technique because it controls both the amplitude and phase of AC excitation. This technique results in an orthogonal spatial orientation of the electromagnetic field and torque, commonly known as Field Oriented Control (FOC).

\section{Field orientation control (FOC) of induction machine}

The concept of field orientation control is used to accomplish a decoupled control of flux and torque. This concept is identical to a dc machine direct torque control that has following requirements \cite{Lipo}:
\begin{itemize}
\item{An independent control of armature current to overcome the effects of armature winding resistance, leakage inductance and induced voltage}
\item{An independent control of constant value of flux}
\end{itemize}

If all of these requirements are met at every instant of time, the torque will follow the current, allowing an immediate torque control and decoupled flux and torque regulation.\\

Next, a two phase d-q model of an induction machine rotating at the synchronous speed is introduced which will help to carry out this decoupled control concept to the induction machine. This model can be summarized by the following equations (see chapter 4 for detail):

\begin{align}
V_{ds}^{\omega}&=R_{s}i_{ds}^{\omega}+\frac{d}{dt}\psi_{ds}^{\omega}-\omega_{e}\psi_{qs}^{\omega}\\
V_{qs}^{\omega}&=R_{s}i_{qs}^{\omega}+\frac{d}{dt}\psi_{qs}^{\omega}+\omega_{e}\psi_{qs}^{\omega}\\
0&=R_{r}i_{dr}^{\omega}+\frac{d}{dt}\psi{dr}^{\omega}-(\omega_{e}-\omega_{r})\psi_{qr}^{\omega}\\
0&=R_{r}i_{qr}^{\omega}+\frac{d}{dt}\psi{qr}^{\omega}+(\omega_{e}-\omega_{r})\psi_{dr}^{\omega}\\
T_{e}&=3p\frac{L_{m}}{L_{r}}(\psi_{dr}^{\omega}i_{qs}^{\omega}-\psi_{qr}^{\omega}i_{ds}^{\omega})
\end{align}

\begin{figure}[h]
\centering
\includegraphics[scale=0.65]{chapter0/phasordig}
\caption{Phasor diagram of the field oriented drive system}
\label{phasor}
\end{figure}

and it is quite significant to synthesize the concept of field-oriented control. In this model it can be seen from the torque expression (2.5) that, if the flux along the q-axis
can be made zero then all the flux is aligned along the d-axis and, therefore, the torque can be instantaneously controlled by controlling the current along q-axis. Then the question will be how it can be guaranteed that all the flux is aligned along the d-axis of the machine. When three-phase voltages are applied to the machine, they produce three-phase fluxes both in the stator and the rotor. The three phase fluxes can be represented in a two phase stationary ($\alpha$-$\beta$) frame. If these two phase fluxes along ($\alpha$-$\beta$) axes are represented by a single-vector then all the machine flux will be aligned along that vector. This vector is commonly specified as d-axis which makes an angle $\theta_{e}$ with the stationary frame $\alpha$-axis, as shown in \autoref{phasor}. The q-axis is set perpendicular to the d-axis. The flux along the q-axis in this case will be obviously zero. The phasor diagram \autoref{phasor} presents these axes. When the machine input currents change sinusoidally in time, the angle $\theta_{e}$ keeps changing. Thus the problem is to know the angle $\theta_{e}$ accurately, so that the d-axis of the d-q frame is locked with the flux vector.\\

The control inputs can be specified in two phase synchronously rotating d-q frame as $i_{ds}^{\omega}$ and $i_{qs}^{\omega}$ such that $i_{ds}^{\omega}$ being aligned with the d-axis or the flux vector. These two phase synchronous control inputs are converted into two phase stationary quantities and then to three phase stationary control inputs. To accomplish this the flux angle $\theta_{e}$ must be known precisely. The angle $\theta_{e}$ can be found either by Direct Field Orientation control (DFO) or by Indirect Field Orientation control (IFO). The controller implemented in this fashion that can achieve a decoupled control of the flux and the torque is known as field oriented controller. The block diagram is shown in the \autoref{foc} In the field-oriented controller the flux can be regulated in the stator, air-gap or rotor flux orientation \cite{Vas}-\cite{Lipo}.

\begin{figure}[h]
\centering
\includegraphics[scale=0.73]{chapter0/foc}
\caption{Field oriented induction motor drive system}
\label{foc}
\end{figure}

The control algorithm for calculation of the rotor flux angle $\theta_{e}$ using IFO control is shown in the \autoref{ifoc}. This algorithm is based on the assumption that, the flux along the q-axis is zero, which forces the command slip velocity to be $\omega_{sl}=i_{qs}^{\omega}/(\tau_{r}i_{ds}^{\omega})$ as a necessary and sufficient condition to guarantee that all the flux is aligned with d-axis and the flux along q-axis is zero. The angle $\theta_{e}$ can then be determined as the sum of the slip and the rotor angles after integrating the respective velocities. This slip angle includes the necessary and sufficient condition for decoupled control of flux and torque. The rotor speed can be measured directly by using an encoder or can be estimated. In case the rotor speed is estimated, the control technique is known as sensorless control. This concept will be studied in detail in the following chapters. \autoref{dfo} shows the control algorithm block diagram for DFO control. In this technique the flux angle $\theta_{e}$ is classically calculated by sensing the air-gap flux through the use of flux sensing coils, or can be calculated by estimating the flux along the d-q axes using the voltage and current signals.

\begin{figure}[h]
\centering
\includegraphics[scale=0.85]{chapter0/ifoc}
\caption{Indirect field oriented drive system}
\label{ifoc}
\end{figure}

\begin{figure}[h]
\centering
\includegraphics[scale=0.8]{chapter0/dfo}
\caption{Direct field oriented drive system}
\label{dfo}
\end{figure}

\subsection{Direct field orientation (DFO)}

The DFO control and sensorless control rely heavily on accurate flux estimation. DFOC is most often used for sensorless control, because the flux observer used to estimate the synchronous speed or angle can also be used to estimate the machine speed. Investigation of ways to estimate the flux and speed of the induction machine has also been extensively studied in the past two decades. Classically, the rotor flux was measured by using a special sensing element, such as Hall effect sensors placed in the air-gap. An advantage of this method is that additional required parameters, $L_{lr}$, $L_{m}$, and $L_{r}$ are not significantly affected by changes in temperature and flux level. However, the disadvantage of this method is that a flux sensor is expensive and needs special installation and maintenance. Another flux and speed estimation technique is saliency based with fundamental or high frequency signal injection. One advantage of saliency technique is that the saliency is not sensitive to actual motor parameters, but this method fails at low and zero speed level. When applied with high frequency signal injection \cite{jansen}, the method may cause torque ripples, and mechanical problems.\\

Gabriel \cite{Gabrie} avoided the special flux sensors and coils by estimating the rotor flux from the terminal quantities (stator voltages and currents). This technique requires the knowledge of the stator resistance along with the stator, rotor leakage inductances and magnetizing inductance. This method is commonly known as the Voltage Model Flux Observer (VMFO). The stator flux in the stationary frame estimated by the equations:

\begin{align}
\psi_{ds}^s&=V_{ds}^s-R_{s}i_{ds}^s\\
\psi_{qs}^s&=V_{qs}^s-R_{s}i_{qs}^s
\end{align}
Then the rotor flux can be expressed as:
\begin{align}
\psi_{dr}^s&=\frac{L_{r}}{L_{m}}(\psi_{ds}^s-\sigma i_{ds}^s)\\
\psi_{qr}^s&=\frac{L_{r}}{L_{m}}(\psi_{qs}^s-\sigma i_{qs}^s)
\end{align}

In this model, integration of the low frequency signals, dominance of stator resistance voltage drop at low speed and leakage inductance variation result in a less precise flux estimation. Integration at low frequency is studied by \cite{jun} and three different alternatives are given. Estimation of rotor flux from the terminal quantities depends on parameters such as stator resistance and leakage inductance. The study of parameter sensitivity shows that the leakage inductance can significantly affect the system performance such as stability, dynamic response, and utilizations of the machine and the inverter.\\

The Current Model Flux Observer (CMFO) is an alternative approach to overcome the problems of leakage inductance and stator resistance at low speed. In this model flux can be estimated as:
\begin{align}
\psi_{dr}^s=-\frac{1}{\tau_{r}}\psi_{dr}^s-\omega_{r}\psi_{qr}^s+\frac{L_{m}}{\tau_{r}}i_{ds}^s\\
\psi_{qr}^s=-\frac{1}{\tau_{r}}\psi_{qr}^s-\omega_{r}\psi_{dr}^s+\frac{L_{m}}{\tau_{r}}i_{qs}^s
\end{align}

However, it does not work well at high speed due to its sensitivity to the rotor resistance. Jansen \cite{pl} did an extensive study on VMFO and CMFO based direct field orientation control, discussed the design and accuracy assessment of various flux observers, compared them, and analyzed the alternative flux observers. To further improve the observer performance, closed-loop rotor flux observers are proposed which use the estimated stator current error \cite{pl,gc} or the estimated stator voltage error \cite{gc} to estimate the rotor flux. Furthermore, Lennart \cite{len} proposed reduced order observers for this task.

\subsection{Indirect field orientation control (IFOC)}

In indirect field orientation, the synchronous speed $\omega_{e}$ is the same as the instantaneous speed of the rotor flux vector $\psi_{dr}^{\omega}$ and the d-axis of the d-q coordinate system is exactly locked on the rotor flux vector (rotor flux vector orientation). This facilities the flux control through the magnetizing current $i_{ds}^{\omega}$ by aligning all the flux with the d-axis while aligning the torque producing component of the current with the q-axis. After decoupling the rotor flux and torque producing component of the current components, the torque can be instantaneously controlled by controlling the current $i_{qs}^{\omega}$. The requirement to align the rotor flux with the d-axis of the d-q coordinate system means that the flux along the q-axis must be zero. This means that the current through the q-axis of the mutual inductance is zero.\\

Based on this restriction $\omega_{sl}$ is :

\begin{align}
\omega_{sl}=\frac{i_{qs}^{\omega}}{(\tau_{r}i_{ds}^{\omega})}
\end{align}

These relations suggest that flux and torque can be controlled independently by specifying d-q axis currents provided the slip frequency is satisfied (2.12) at all instants.\\

The concept of indirect field oriented control developed in the past has been widely studied by researchers during the last two decades. The rotor flux orientation is both the original and usual choice for the indirect orientation control. Also the IFO control can be implemented in the stator and air-gap flux orientation as well. De Doncker \cite{doncker} introduced this concept in his universal field oriented controller. In the air-gap flux the slip and flux relations are coupled equations and the d-axis current does not independently control the flux as it does in the rotor flux orientation. For the constant air-gap flux orientation, the maximum of the produced torque is \%20 less than that of the other two methods \cite{sul}. In the stator flux orientation, the transient reactance is a coupling factor and it varies with the operating conditions of the machine. In addition, Mircea \cite{mirc} shows that among these methods, rotor flux oriented control has linear torque curve. Therefore, the most commonly used choice for IFO is the rotor flux orientation.\\

The IFOC is an open loop, feed forward control in which the slip frequency is fed forward guaranteeing the field orientation. This feed forward control is very sensitive to the rotor open circuit time constant $\tau_{r}$. Therefore, $\tau_{r}$ must be known in order to achieve a decoupled control of torque and flux components by controlling $i_{ds}^{\omega}$ and $i_{qs}^{\omega}$ , respectively. When $\tau_{r}$ is not set correctly, the machine is said to be detuned and the performance will become sluggish due to loss of decoupled control of torque and flux. The measurement of the rotor time constant, its effects on the system performance and its adaptive tuning to the variations resulting during the operation of the machine have been studied extensively in the literature \cite{rl}-\cite{krisn}. Lorenz, Krishnan and Novotny \cite{rl}-\cite{krisn} studied the effect of temperature and saturation level on the rotor time constant and concluded that it can reduce the torque capability of the machine and torque/amps of the machine. The detuning effect becomes more severe in the field-weakening region. Also, it results in a steady-state error and, transient oscillations in the rotor flux and torque. Some of the advanced control techniques such as estimation theory tools and adaptive control tools are also studied to estimate rotor time constant and other motor parameters \cite{ca}-\cite{jm}.\\

\section{Variable speed control using advance control algorithms}

There are two issues in motion control using field oriented controlled (FOC) induction machine drives. One is to make the resulting drive system and the controller robust against parameter deviations and disturbances. The other is to make the system intelligent to adjust the control system itself to environment changes and task requirements. If the speed regulation loop fails to produce the command current correctly, than the desired torque response will not be produced by the induction machine. In addition, such a failure may cause the degradation of slip command. As a result, a satisfactory speed regulation is extremely important not only to produce desired torque performance from the induction machine but also to guarantee the decoupling between control of torque and flux.\\

Conventionally, a PI controller has been used for the speed regulation to generate a command current for last two decades, and accepted by industry because of its simplicity. Even though, a well tuned PI controller performs satisfactorily for a field oriented induction machine during steady state. The speed response of the machine at transient, especially for the variable speed tracking, may sometimes be problematic. In last two decades, alternative control algorithms for the speed regulation were investigated. Among these, fuzzy logic, sliding mode, and adaptive nonlinear control algorithms gained much attention.\\

A traditional rotor flux oriented induction machine drive offers a better control performance but it often requires additional sensors on the machine. This adds to the cost and complexity of the drive system. To avoid these sensors on the machine, many different algorithms are proposed for the last three decades to estimate the rotor flux vector and rotor shaft speed. The recent trend in field oriented control is to use such algorithms based on the terminal quantities of the machine for the estimation of the fluxes and speed. They can easily be applied to any induction machine. Therefore, our focus in this study is also on these algorithms.\\

Before looking into individual approaches, the common problems of the speed and flux estimation are discussed briefly for general field orientation and state estimation algorithms.
 \begin{itemize}
 \item{\textbf{Parameter sensitivity:} One of the important problems of the sensorless control algorithms for the field oriented induction machine drives is the insufficient information about the machine parameters which yield the estimation of some machine parameters along with the sensorless structure. Among these parameters stator resistance, rotor resistance and rotor time constant play more important role than the other parameters since these values are more sensitive to temperature changes. The knowledge of the correct stator resistance $R_{s}$ is important to widen the
operation region toward the lower speed range. Since at low speeds the induced voltage is low and stator resistance voltage drop becomes dominant, a mismatching stator resistance induces instability in the system. On the other hand, errors made in determining the actual value of the rotor resistance $R_{r}$, may cause both instability of the system and speed estimation error proportional to $R_{r}$ \cite{gy}. Also, correct $\tau_{r}$ value is vital decoupling factor in IFOC}

\item{\textbf{Pure Integration:} The other important issue regarding many of the topologies is the integration process inherited from the induction machine dynamics where an integration process is needed to calculate the state variables of the system. However, it is difficult both to decide on the initial value, and prevent the drift of the output of a pure integrator. Usually, to overcome this problem a low-pass filter replaces the integrator.}

\item{\textbf{Overlapping loop Problems:} In a sensorless control system, the control loop and the speed estimation loop may overlap and these loops influence each other. As a result, outputs of both of these loops may not be designed independently,  in some bad cases this dependency may influence the stability or performance of the overall system.}
\end{itemize}
\vspace{1cm}
The algorithms, where terminal quantities of the machine are used to estimate the fluxes and speed of the machine are categorized in two basic groups. First one is ``the open loop observers," in a sense that the on-line model of the machine does not use the feedback correction. Second one is ``the closed loop observers" where the feedback correction is used along with the machine model itself to improve the estimation accuracy. These two basic groups can also be divided further into subgroups based on the control method used. These can be summarized as:\\

Open loop observers based on:\\
- Current model\\
- Voltage model\\
- Full-order observer\\

Closed loop observers based on:\\
- Model Reference Adaptive Systems (MRAS)\\
- Kalman filter techniques\\
- Adaptive observers based on both voltage and current model\\
- Neural network flux and speed estimators\\
- Sliding mode flux and speed estimators\\

Current model based open loop observers use the measured stator currents and rotor velocity. The velocity dependency of the current model is very important since this means that although using the estimated flux eliminates the flux sensor, the position sensor is still required. On the other hand, voltage model based open loop observers \cite{M}-\cite{gc} use the measured stator voltage and current as inputs. A full-order open loop observer can be formed using only the measured stator voltage and rotor velocity as inputs where the stator current appears as an estimated quantity. Because of its dependency on the stator current estimation, the full order observer will not exhibit better performance than the current model. Furthermore, parameter sensitivity and observer gain are the problems to be tuned in a full order observer design \cite{br}. These open-loop observer structures are all based on the induction machine model, and they do not employ any feedback. Therefore, they are quite sensitive to parameter variations, which yield the estimation of some machine parameters along with the sensorless structure.\\
\vspace{1cm}\\

To produce more robust structures to parameter variations some kind of feedback may be helpful. For this purpose many closed loop topologies are proposed using different induction machine models and control methods. Among these MRAS attracts attention and several different algorithms are produced. In MRAS a comparison is made between the outputs of two estimators. The estimator which does not contain the quantity to be estimated can be considered as a reference model of the induction machine. The error between these two estimators is used as an input to an adaptation mechanism. The estimated rotor speed in the adjustable model is changed in such a way that the difference between two estimators converges to zero asymptotically, and the estimated rotor speed will be equal to actual rotor speed. The basics of the analysis and design of MRAS are discussed in \cite{Vas, booksul}. In \cite{gy, J}. In \cite{cs} similar speed estimators are proposed based on the MRAS, and a secondary variable is introduced as the reference quantity by letting the rotor flux through a first-order delay instead of a pure integration to nullify the offset. However, their algorithms produce inaccurate estimated speed if the excitation frequency goes below certain level. In addition these algorithms suffer from the machine parameter uncertainties since the parameter variation in the reference model cannot be corrected. Zhen \cite{lz} proposed an interesting MRAS structure that is built with two mutual MRAS schemes. In this structure, the reference model and the adjustable models are interchangeable. For rotor speed estimation, one model is used as reference model and other model is used as adjustable model. The pure integration is removed from reference model. \cite{me} supported the MRAS scheme with ANN using its training and modeling of non-linear systems. MRAS scheme is also used for the on-line adaptation of the motor parameters in field oriented control techniques \cite{ca, kubota}.\\

Kalman filter (KF) is another method employed to identify the speed and rotor flux of an induction machine based on the measured quantities such as stator current and voltage \cite{kim}. Kalman filter approach is based on the system model and a mathematical model describing the induction motor dynamics for the use of Kalman filter application. Parameter deviations and measurement disturbance are taken into consideration in KF covariance matrices of the KF must be properly initialized. KF works for linear systems and for non linear induction motor model extended Kalman filter (EKF) is used. However, KF approach is computationally intensive and depends on the accuracy of the model of the motor. In the EKF model proposed by  one can estimate rotor fluxes and rotor speed which makes the field orientation. EKF is also used for online parameter estimation of induction motor \cite{Vas, lcz}. Reduced order models are also proposed to shorten and speed up the complex EKF algorithm \cite{ekf}. A new KF technique for non-linear systems, Unscented Kalman Filter (UKF), is applied to induction machine state estimatio, which is a derivative free KF technique which avoids costly calculation of Jacobian matrix, linearization of the estimates \cite{ukf}.\\

Another method used for the sensorless control of induction motor is the neural network technique, which is based on a learning process. It has the advantage of tolerating machine parameter uncertainties. For speed estimation, a two-layered neural network, based on back propagation technique, is used and the neural network outputs are compared with the actual measurement values and error then back- propagated to adjust the weights such that the estimated speed converges to actual one. The neural network based sensorless control algorithms have the advantages of fault-tolerant characteristics. However, because of the neural network learning process these algorithms may suffer from the computational intensity.\\

Another approach is sliding mode control for FOC of induction machine. In the sliding mode technique, the control action is very strong and being switched into either ``on" or ``off" at high frequency. The command signals control directly the power devices. This type of control is also favorable because ``on-off" is the only admissible mode of operation for the power converters. Therefore, it seems more natural to employ the algorithm towards discontinuous control.\\

In addition to the algorithms mentioned above, some of the proposed work is hard to classify because of their combined structure. In \cite{J}, a nonlinear high- gain observer structure is proposed, and it is claimed that with the exact knowledge of stator resistance, flux and speed estimation convergence is guaranteed.

\section{Conclusion}

The literature review of DFOC, IFOC, flux, position and speed estimation and speed control can be summarized as:
\begin{itemize}

\item{The DFOC and IFOC are the methods for instantaneous torque and speed control of an induction motor drive system. These methods can be implemented with or without a speed sensor. An IFOC is synthesized by properly controlled slip frequency which is necessary for the field-orientation}\\
\item{The main problem of an IFO drive system is the rotor time constant deviation. The drive system torque control performance decreases if the rotor time constant is not set precisely. Therefore, on-line estimation is necessary and is one of the main challenges for better performance of an IFOC. Most of the techniques proposed so far either need some special hardware or are very complex with respect to the software and require intensive calculations which put extra burden on the processor.}\\
%\item{The main problem in DFO control is precise rotor flux or position observation. This observation from terminal quantities is more desirable than the one including additional hardware.}\\
\item{Voltage model and current model flux observers are the two most common ways to estimate the flux using the terminal quantities. The voltage model flux observer is dominated by stator IR drop at low speed, whereas the current model flux observer has problems of rotor time constant variations. Also the current model flux observer requires the rotor speed. Therefore, if the flux observer is being used for the sensorless control, an error in the estimated speed will be fed back in to the system. Thus will affect the observer accuracy.}\\
\item{The proposed open-loop observers can be simple in the structure but they are susceptible to variety of errors that become specially detrimental at low stator frequencies, including measurement, noise digital approximation errors, parameter detuning and DC offset in measurements, which ultimately may drive the observer instability.}\\
\item{For the time varying system model problems, closed-loop observers are proposed here feedback correction is used along with the machine model itself to improve the estimation accuracy. The algorithmic complexity and calculation intensity looks higher when compared with former solutions but the recent processors are fast enough to solve these algorithms in real-time applications. They also require a strong mathematical background to deal with.}
\end{itemize}


%\label{sec:limitsetting} % uncomment if label used.

The limit setting phase is performed in case no significant deviations from the
background-only hypothesis are reported in the search phase. While the search
phase only depends on the BKG
distribution and its (dis)agreement with data, the limit setting phase performs
exclusion limits given certain theory models. In the following a more detailed
description on the statistical machinery used to set limits will be given.

\subsection{Limit setting using the dijet mass distribution}

Exclusion limits were calculated using the CL$_\mathrm{s}$ method with a binned profile likelihood ratio as the test
statistic using the \texttt{HistFitter} \cite{HistFitter:2014} framework.
The limit is extracted from adjusting the normalization $\mu_{s}$ of a template that represents the shape of the nominal signal model in question.
By scanning over values of $\mu_{s}$ until a CL$_\mathrm{s}$ p-value of 0.05 is found,
a 95\% confidence upper limit on this normalization parameter can be found.
The obtained upper limit on signal strength is then converted into an upper limit of cross section
corresponds to given luminosity and the observed data.
The limit on $\sigma\times {\cal A} \times BR$ from data is interpolated logarithmically
between mass points to create a continuous curve in \mjj{}.
The exclusion limit on the mass (or energy scale) of the
given new physics resonant signal occurs at the value of \mjj{} where the limit
on $\sigma\times {\cal A} \times BR$ from data is the same as the
theoretical value, which is derived from the interpolation between
the generated mass values.

Expected limits on the signal model are calculated by using an asymptotic approximation. A major benefit
of using a binned likelihood approach such as used with this \texttt{HistFitter} approach is it's
computational speed, particularly due to the asymptotic approximation.
The \texttt{HistFitter} method has calculated expected and observed limit for each mass point around 30 seconds.
This is compared to hours or days using the Bayesian approach for very similar results.
%
%
%\subsection{Limit setting using the dijet mass distribution}
%
%For each benchmark process under study in the resonance analysis, Monte Carlo samples have been
%simulated at a number of selected mass points, $m_{\textrm{R}}$.
%The Bayesian method documented in ref.~\cite{Aad:2011aj}
%is applied to data at these same mass points to set a
%95\% CL limit on the cross section times acceptance times branching ratio,
%$\sigma\times {\cal A} \times BR$, for the new physics resonant signal as a function of
%$m_{\textrm{R}}$, using a prior constant in signal strength.  The one exception is QBH, which has limits set on $\sigma\times {\cal A}$.
%The limit on $\sigma\times {\cal A} \times BR$ from data is interpolated logarithmically
%between mass points to create a continuous curve in \mjj{}.
%The exclusion limit on the mass (or energy scale) of the
%given new physics resonant signal occurs at the value of \mjj{} where the limit
%on $\sigma\times {\cal A} \times BR$ from data is the same as the
%theoretical value, which is derived from the interpolation between
%the generated mass values.

This form of analysis is applicable to all resonant phenomena
where the new physics resonance couplings are strong compared to the scale of perturbative QCD
at the signal mass, so that interference with QCD terms can be neglected.
The acceptance calculation includes all reconstruction steps and analysis cuts.
%is described in Section~\ref{sec:res_selection}.

The background prediction used in this calculation is obtained from a
simultaneous S+B fit.

The effects of the systematic uncertainties on the data-driven background together with the systematic uncertainties
on the signal samples
%, described in Section~\ref{sec:systematicUncertainties},
are considered in the limit-setting procedure.
These uncertainties are incorporated into the LH formalism by varying all sources according
to Gaussian probability distributions.
%The templates are linearly interpolated
%between 1$\sigma$ and -1$\sigma$ and exptrapolated to reach 3$\sigma$ with an exponential function
%for points above that interval according to the recomandation of the stat.forum.
Confidence intervals are then calculated from the resulting profile of
the parameter-of-interest of the LH.

In order to validate the limit setting machinery, results have been compared
with
the Bayesian limits used in previous iteration of the analysis. In addition, a
validation of the asymptotic approximation has been checked comparing limits
derived from toys.
%Both studies are described in Appendix~\ref{sec:HF_val}.


%% !TEX root = manual.tex

\section{Spyplot Diagrams}
\label{sec:tutorials:spyplot}

Spyplots visualize communication matrices, showing either the number of messages or number of bytes sent between two network endpoints.
They are essentially contour diagrams, where instead of a continuous function $F(x,y)$ we are plotting the communication matrix $M(i,j)$.
An example spyplot is shown for a simple application that only executes an MPI\_Allreduce (Figure \ref{fig:spyplot}).
Larger amounts of data (red) are sent to nearest neighbors while decreasing amounts (blue) are sent to MPI ranks further away.

\begin{figure}[h]
\centering
\includegraphics[width=0.4\textwidth]{figures/spyplot/mpi_spyplot.png}
\caption{Spyplot of Bytes Transferred Between MPI Ranks for MPI\_Allreduce}
\label{fig:spyplot}
\end{figure}

Various spyplots can be activated by boolean parameters in the input file.
The most commonly used are the MPI spyplots, for which you must add

\begin{ViFile}
mpi_spyplot = <fileroot>
\end{ViFile}

After running there will be a .csv and .png file in the folder, with e.g. \inlineshell{fileroot = test}

\begin{ShellCmd}
example> ls
test.png
test.csv
\end{ShellCmd}
\inlineshell{test.png} shows the number of bytes exchanged between MPI ranks.
To extend the analysis you can specify

\begin{ViFile}
network_spyplot = <fileroot>
\end{ViFile}
A new csv/png will appear showing the number of bytes exchanged between physical nodes,
accumulating together all MPI ranks sharing the same node.
This gives a better sense of spatial locality when many MPI ranks are on the same node.