\documentclass[conference]{IEEEtran}
\usepackage{cite}
\usepackage{amsmath,amssymb,amsfonts}
\usepackage{algorithmic}
\usepackage{graphicx}
\usepackage{textcomp}
\usepackage{array}
\usepackage{mathrsfs}
\usepackage{graphicx}
\usepackage{mathtools}
\def\BibTeX{{\rm B\kern-.05em{\sc i\kern-.025em b}\kern-.08em
    T\kern-.1667em\lower.7ex\hbox{E}\kern-.125emX}}
\begin{document}

\title{Deep Quaternion Networks}

\author{\IEEEauthorblockN{Chase J. Gaudet}
\IEEEauthorblockA{\textit{School of Computing \& Informatics} \\
\textit{University of Lousiana at Lafayette}\\
Lafayette, USA \\
cjg7182@louisiana.edu}
\and
\IEEEauthorblockN{Anthony S. Maida}
\IEEEauthorblockA{\textit{School of Computing \& Informatics} \\
\textit{University of Lousiana at Lafayette}\\
Lafayette, USA \\
maida@louisiana.edu}
}

\maketitle

\begin{abstract}
The field of deep learning has seen significant advancement in recent years.
However, much of the existing work has been focused on real-valued numbers.
Recent work has shown that a deep learning system using the complex numbers can be deeper for a fixed parameter budget compared to its real-valued counterpart.
In this work, we explore the benefits of generalizing one step further into the hyper-complex numbers, quaternions specifically, and provide the architecture components needed to build deep quaternion networks.
We develop the theoretical basis by reviewing quaternion convolutions, developing a novel quaternion weight initialization scheme, and developing novel algorithms for quaternion batch-normalization.
These pieces are tested in a classification model by end-to-end training on the CIFAR-10 and CIFAR-100 data sets and a segmentation model by end-to-end training on the KITTI Road Segmentation data set.
These quaternion networks show improved convergence compared to real-valued and complex-valued networks, especially on the segmentation task, while having fewer parameters.
\end{abstract}

\begin{IEEEkeywords}
quaternion, complex, neural networks, deep learning
\end{IEEEkeywords}

\section{Introduction}
There have been many advances in deep neural network architectures in the past few years.
One such improvement is a normalization technique called batch normalization \cite{ioffe2015batch} that standardizes the activations of layers inside a network using minibatch statistics.
It has been shown to regularize the network as well as provide faster and more stable training.
Another improvement comes from architectures that add so called shortcut paths to the network.
These shortcut paths connect later layers to earlier layers typically, which allows for the stronger gradients to propagate to the earlier layers.
This method can be seen in Highway Networks \cite{srivastava2015training} and Residual Networks  \cite{he2016deep}.
Other work has been done to find new activation functions with more desirable properties.
One example is the exponential linear unit (ELU) \cite{clevert2015fast}, which attempts to keep activations standardized.
All of the above methods are combating the vanishing gradient problem \cite{hochreiter1991untersuchungen} that plagues deep architectures.
With solutions to this problem appearing it is only natural to move to a system that will allow one to construct deeper architectures with as low a parameter cost as possible.

Other work in this area has explored the use of complex and hyper-complex numbers, which are a generalization of the complex, such as quaternions.
Using complex numbers in recurrent neural networks (RNNs) has been shown to increase learning speed and provide a more noise robust memory retrieval mechanism \cite{arjovsky2016unitary, danihelka2016associative, wisdom2016full}.
The first formulation of complex batch normalization and complex weight initialization is presented by \cite{trabelsi2017deep} where they achieve some state of the art results on the MusicNet data set.
Hyper-complex numbers are less explored in neural networks, but have seen use in manual image and signal processing techniques \cite{bulow1999hypercomplex, sangwine2000colour, bulow2001hypercomplex}.
Examples of using quaternion values in networks is mostly limited to architectures that take in quaternion inputs or predict quaternion outputs, but do not have quaternion weight values \cite{rishiyur2006neural, kendall2015posenet}.
There are some more recent examples of building models that use quaternions represented as real-values.
In \cite{parcollet2016quaternion} they used a quaternion multi-layer perceptron (QMLP) for document understanding and \cite{minemoto2017feed} uses a similar approach in processing multi-dimensional signals.

Building on \cite{trabelsi2017deep} our contribution in this paper is to formulate and implement quaternion convolution, batch normalization, and weight initialization \footnote{Source code located at https://github.com/gaudetcj/DeepQuaternionNetworks}.
There arises some difficulty over complex batch normalization that we had to overcome as their is no analytic form for our inverse square root matrix.

\section{Motivation and Related Work}
The ability of quaternions to effectively represent spatial transformations and analyze multi-dimensional signals makes them promising for applications in artificial intelligence.

One common use of quaternions is for representing rotation into a more compact form.
PoseNet \cite{kendall2015posenet} used a quaternion as the target output in their model where the goal was to recover the $6-$DOF camera pose from a single RGB image.
The ability to encode rotations may make a quaternion network more robust to rotational variance.

Quaternion representation has also been used in signal processing.
The amount of information in the phase of an image has been shown to be sufficient to recover the majority of information encoded in its magnitude by Oppenheim and Lin \cite{oppenheim1981importance}.
The phase also encodes information such as shapes, edges, and orientations.
Quaternions can be represented as a 2~x~2 matrix of complex numbers, which gives them a group of phases potentially holding more information compared to a single phase.

Bulow and Sommer \cite{bulow2001hypercomplex} used the higher complexity representation of quaternions by extending Gabor's complex signal to a quaternion one which was then used for texture segmentation.
Another use of quaternion filters is shown in \cite{sangwine2000colour} where they introduce a new class of filter based on convolution with hyper-complex masks, and present three color edge detecting filters.
These filters rely on a three-space rotation about the grey line of RGB space and when applied to a color image produce an almost greyscale image with color edges where the original image had a sharp change of color.
More quaternion filter use is shown in \cite{shi2007quaternion} where they show that it is effective in the context of segmenting color images into regions of similar color texture.
They state the advantage of using quaternion arithmetic is that a color can be represented and analyzed as a single entity (by assigning each color channel to an imaginary axis), which we will see holds for quaternion convolution in a convolutional neural network architecture as well in Section \ref{s:qc}.

A quaternionic extension of a feed forward neural network, for processing multi-dimensional signals, is shown in \cite{minemoto2017feed}.
They expect that quaternion neurons operate on multi-dimensional signals as single entities, rather than real-valued neurons that deal with each element of signals independently.
A convolutional neural network (CNN) should be able to learn a powerful set of quaternion filters for more impressive tasks.

Another large motivation is discussed in \cite{trabelsi2017deep}, which is that complex numbers are more efficient and provide more robust memory mechanisms compared to the reals \cite{bulow1999hypercomplex, sangwine2000colour, bulow2001hypercomplex}.
They continue that residual networks have a similar architecture to associative memories since the residual shortcut paths compute their residual and then sum it into the memory provided by the identity connection.
Again, given that quaternions can be represented as a complex group, they may provide an even more efficient and robust memory mechanisms.

\section{Quaternion Network Components}
This section will include the work done to obtain a working deep quaternion network.
Some of the longer derivations are given in the Appendix.

\subsection{Quaternion Representation}
In 1833 Hamilton proposed complex numbers $\mathbb{C}$ be defined as the set $\mathbb{R}^2$ of ordered pairs $(a, b)$ of real numbers.
He then began working to see if triplets $(a,b,c)$ could extend multiplication of complex numbers.
In 1843 he discovered a way to multiply in four dimensions instead of three, but the multiplication lost commutativity.
This construction is now known as quaternions.
Quaternions are composed of four components, one real part, and three imaginary parts.
Typically denoted as
\begin{equation}
\mathbb{H} = \{a + b\textit{i} + c\textit{j} + d\textit{k}~:~a,b,c,d \in \mathbb{R}\}
\label{eq:quaternion1}
\end{equation}
where $a$ is the real part, $(i,j,k)$ denotes the three imaginary axis, and $(b,c,d)$ denotes the three imaginary components.
Quaternions are governed by the following arithmetic:
\begin{equation}
i^2=j^2=k^2=ijk=-1
\label{eq:quarternion2}
\end{equation}
which, by enforcing distributivity, leads to the noncommutative multiplication rules
\begin{equation}
ij=k,~jk=i,~ki=j,~ji=-k,~kj=-i,~ik=-j
\label{eq:quarternion3}
\end{equation}

Since we will be performing quaternion arithmetic using reals it is useful to embed $\mathbb{H}$ into a real-valued representation.
There exists an injective homomorphism from $\mathbb{H}$ to the matrix ring $M(4,\mathbb{R})$ where $M(4,\mathbb{R})$ is a 4x4 real matrix.
The 4~x~4 matrix can be written as
\begin{align}
\begin{bmatrix}
 a & -b & -c & -d \\
 b & a & -d & c \\
 c & d & a & -b \\
 d & -c & b & a
\end{bmatrix}= &~~a
\begin{bmatrix}
 1 & 0 & 0 & 0 \\
 0 & 1 & 0 & 0 \\
 0 & 0 & 1 & 0 \\
 0 & 0 & 0 & 1
\end{bmatrix}
\nonumber \\ &+ b
\begin{bmatrix}
 0 & -1 & 0 & 0 \\
 1 & 0 & 0 & 0 \\
 0 & 0 & 0 & -1 \\
 0 & 0 & 1 & 0
\end{bmatrix}
\nonumber \\ &+ c
\begin{bmatrix}
 0 & 0 & -1 & 0 \\
 0 & 0 & 0 & 1 \\
 1 & 0 & 0 & 0 \\
 0 & -1 & 0 & 0
\end{bmatrix}
\nonumber \\ &+ d
\begin{bmatrix}
 0 & 0 & 0 & -1 \\
 0 & 0 & -1 & 0 \\
 0 & 1 & 0 & 0 \\
 1 & 0 & 0 & 0
\end{bmatrix}.
\label{eq:m4r}
\end{align}
This representation of quaternions is not unique, but we will stick to the above in this paper.
It is also possible to represent $\mathbb{H}$ as $M(2,\mathbb{C})$ where $M(2,\mathbb{C})$ is a 2~x~2 complex matrix.

With our real-valued representation a quaternion real-valued $2D$ convolution layer can be expressed as follows.
Say that the layer has $N$ feature maps such that $N$ is divisible by 4.
We let the first $N/4$ feature maps represent the real components, the second $N/4$ represent the $i$ imaginary components, the third $N/4$ represent the $j$ imaginary components, and the last $N/4$ represent the $k$ imaginary components.

\subsection{Quaternion Differentiability}
In order for the network to perform backpropagation the cost function and activation functions used must be differentiable with respect to the real, $i$, $j$, and $k$ components of each quaternion parameter of the network.
As the complex chain rule is shown in \cite{trabelsi2017deep}, we provide the quaternion chain rule which is given in the Appendix section \ref{a:diff}.

\subsection{Quaternion Convolution}\label{s:qc}
Convolution in the quaternion domain is done by convolving a quaternion filter matrix $\textbf{W}=\textbf{A}+\textit{i}~\textbf{B}+\textit{j}~\textbf{C}+\textit{k}~\textbf{D}$ by a quaternion vector $\textbf{h}=\textbf{w}+\textit{i}~\textbf{x}+\textit{j}~\textbf{y}+\textit{k}~\textbf{z}$.
Here $\textbf{A}$, $\textbf{B}$, $\textbf{C}$, and $\textbf{D}$ are real-valued matrices and $\textbf{w}$, $\textbf{x}$, $\textbf{y}$, and $\textbf{z}$ are real-valued vectors.
Performing the convolution by using the distributive property and grouping terms one gets
\begin{align}
\textbf{W}\ast \textbf{h} = &~(\textbf{A}\ast\textbf{w}-\textbf{B}\ast\textbf{x}-\textbf{C}\ast\textbf{y}-\textbf{D}\ast\textbf{z}) + \nonumber \\
&\textit{i}(\textbf{A}\ast\textbf{x}+\textbf{B}\ast\textbf{w}+\textbf{C}\ast\textbf{z}-\textbf{D}\ast\textbf{y}) + \nonumber \\
&\textit{j}(\textbf{A}\ast\textbf{y}-\textbf{B}\ast\textbf{z}+\textbf{C}\ast\textbf{w}+\textbf{D}\ast\textbf{x}) + \nonumber \\
&\textit{k}(\textbf{A}\ast\textbf{z}+\textbf{B}\ast\textbf{y}-\textbf{C}\ast\textbf{x}+\textbf{D}\ast\textbf{w}).
\end{align}
Using a matrix to represent the components of the convolution we have:
\begin{equation}
\begin{bmatrix}
 \mathscr{R}(\textbf{W}\ast \textbf{h}) \\
 \mathscr{I}(\textbf{W}\ast \textbf{h}) \\
 \mathscr{J}(\textbf{W}\ast \textbf{h}) \\
 \mathscr{K}(\textbf{W}\ast \textbf{h})
\end{bmatrix}
=
\begin{bmatrix}
 \textbf{A} & -\textbf{B} & -\textbf{C} & -\textbf{D}\\
 \textbf{B} & \textbf{A} & -\textbf{D} & \textbf{C} \\
 \textbf{C} & \textbf{D} & \textbf{A} & -\textbf{B} \\
 \textbf{D} & -\textbf{C} & \textbf{B} & \textbf{A} \\
\end{bmatrix}
\ast
\begin{bmatrix}
 \textbf{w} \\
 \textbf{x} \\
 \textbf{y} \\
 \textbf{z}
\end{bmatrix}
\label{eq:qconvolve2}
\end{equation}

An example is shown in Fig.~\ref{f:quatconv}, which is useful to visualize one of the main motivational factors of quaternions for CNNs.
Notice that the result of the quaternion convolution produces a unique linear combination of each axis per the result of a single axis.
This comes from the structure of quaternion multiplication and is forcing each axis of the kernel to interact with each axis of the image.
Real-valued convolution simply multiplies each channel of the kernel with the corresponding channel of the image.
The quaternion convolution is similar to a mixture of standard convolution and depthwise separable convolution from \cite{chollet2016xception}.
Depthwise separable convolution is where first a flat convolution kernel (no depth to match the depth of the feature image) is applied separately to each feature map.
This is only giving spatial context on each feature map individually.
Then a $1\times1$ convolution is applied to the results of the previous operation to get a linear interaction of the feature maps, projecting them into a new feature map space.

The quaternion network's reuse of filters on every axis and combination may help extract texture information across channels as seen in \cite{shi2007quaternion}.
One can think in terms of a RGB image where the greyscale of the image can be the real axis and the RGB channels individually can be the $i, j, k$ axes.
Then a quaternion kernel convolved against this quaternion image will view the colors as a single entity, unlike standard real-valued convolution.
Since a quaternion can be thought of as a vector, the quaternion kernels and feature maps can be thought of as vectors as well.

\begin{figure*}
	\centering
		\includegraphics[width=1.0\textwidth]{quatconv.png}
	\caption{An illustration of quaternion convolution.}
	\label{f:quatconv}
\end{figure*}

\subsection{Quaternion Batch-Normalization}
Batch-normalization \cite{ioffe2015batch} is used by the vast majority of all deep networks to stabilize and speed up training.
It works by keeping the activations of the network at zero mean and unit variance.
The original formulation of batch-normalization only works for real-values.
Applying batch normalization to complex or hyper-complex numbers is more difficult, one can not simply translate and scale them such that their mean is 0 and their variance is 1.
This would not give equal variance in the multiple components of a complex or hyper-complex number.
To overcome this for complex numbers a whitening approach is used \cite{trabelsi2017deep}, which scales the data by the square root of their variances along each of the two principle components.
We use the same approach, but must whiten 4D vectors.

However, an issue arises in that there is no nice way to calculate the inverse square root of a 4~x~4 matrix.
It turns out that the square root is not necessary and we can instead use the Cholesky decomposition on our covariance matrix.
The details of why this works for whitening given in the Appendix section \ref{a:whitening}.
Now our whitening is accomplished by multiplying the \textbf{0}-centered data ($\textbf{x} - \mathbb{E}[\textbf{x}]$) by \textbf{W}:
\begin{equation}
\tilde{x} = \textbf{W}(\textbf{x} - \mathbb{E}[\textbf{x}])
\label{eq:white4d}
\end{equation}
where \textbf{W} is one of the matrices from the Cholesky decomposition of $\textbf{V}^{-1}$ where \textbf{V} is the covariance matrix given by:
\begin{align*}
\textbf{V}
=&
\begin{bmatrix}
 V_{rr} & V_{ri} & V_{rj} & V_{rk} \\
 V_{ir} & V_{ii} & V_{ij} & V_{ik} \\
 V_{jr} & V_{ji} & V_{jj} & V_{jk} \\
 V_{kr} & V_{ki} & V_{kj} & V_{kk}
\end{bmatrix}
\label{eq:V4d}
\end{align*}
where each $V$ is the covariance between its two subscripts which represent the real, $i$, $j$, and $k$ components of $\textbf{x}$ respectively.

Real-valued batch normalization also uses two learned parameters, $\beta$ and $\gamma$.
Our shift parameter {\boldmath$\beta$} must shift a quaternion value so it is a quaternion value itself with real, $i$, $j$, and $k$ as learnable components.
The scaling parameter {\boldmath$\gamma$} is a symmetric matrix of size matching $\textbf{V}$ given by:
\begin{equation}
\mathbf{\gamma}
=
\left(
\begin{array}{cccc}
\gamma_{rr} & \gamma_{ri} & \gamma_{rj} & \gamma_{rk} \\
\gamma_{ri} & \gamma_{ii} & \gamma_{ij} & \gamma_{ik} \\
\gamma_{rj} & \gamma_{ij} & \gamma_{jj} & \gamma_{jk} \\
\gamma_{rk} & \gamma_{ik} & \gamma_{jk} & \gamma_{kk}
\end{array}
\right)
\label{eq:gamma}
\end{equation}
Because of its symmetry it has only ten learnable parameters.
The variance of the components of input $\tilde{\textbf{x}}$ are variance 1 so the diagonal of {\boldmath$\gamma$} is initialized to $1/\sqrt{4}$ in order to obtain a modulus of 1 for the variance of the normalized value.
The off diagonal terms of {\boldmath$\gamma$} and all components of {\boldmath$\beta$} are initialized to 0.
The quaternion batch normalization is defined as:
\begin{equation}
\mbox{BN}(\tilde{\textbf{x}}) = \mathbf{\gamma}\tilde{\textbf{x}} + \mathbf{\beta}
\label{eq:qbn}
\end{equation}

\subsection{Quaternion Weight Initialization}
The proper initialization of weights is vital to convergence of deep networks.
In this work we derive our quaternion weight initialization using the same procedure as Glorot and Bengio \cite{glorot2010understanding} and He et al. \cite{he2015delving}.

To begin we find the variance of a quaternion weight:
\begin{align}
W = &~|W|e^{(\mbox{cos}\phi_1 \textit{i} + \mbox{cos}\phi_2 \textit{j} + \mbox{cos}\phi_3 \textit{k})\theta} \nonumber \\
= &~\mathscr{R}\{W\} + \mathscr{I}\{W\} + \mathscr{J}\{W\} + \mathscr{K}\{W\}.
\label{eq:quaternion_weight}
\end{align}
where $|W|$ is the magnitude, $\theta$ and $\phi$ are angle arguments, and $\mbox{cos}^2\phi_1 + \mbox{cos}^2\phi_2 + \mbox{cos}^2\phi_3 = 1$ \cite{turner2002}.

Variance is defined as
\begin{equation}
\mbox{Var}(W) = \mathbb{E}[|W|^2] - (\mathbb{E}[W])^2,
\label{eq:variance}
\end{equation}
but since $W$ is symmetric around 0 the term $(\mathbb{E}[W])^2$ is 0.
We do not have a way to calculate $\mbox{Var}(W) = \mathbb{E}[|W|^2]$ so we make use of the magnitude of quaternion normal values $|W|$, which follows an independent normal distribution with four degrees of freedom (DOFs).
We can then calculate the expected value of $|W|^2$ to find our variance
\begin{equation}
\mathbb{E}[|W|^2] = \int_{-\infty}^\infty x^2 f(x) ~dx = 4\sigma^2
\label{eq:expected}
\end{equation}
where $f(x)$ is the four DOF distribution given in the Appendix.

And since $\mbox{Var}(W) = \mathbb{E}[|W|^2]$, we now have the variance of $W$ expressed in terms of a single parameter $\sigma$:
\begin{equation}
\mbox{Var}(W) = 4\sigma^2.
\label{eq:variance_sigma}
\end{equation}

To follow the Glorot and Bengio \cite{glorot2010understanding} initialization we have $\mbox{Var}(W) = 2/(n_{in}+n_{out})$, where $n_{in}$ and $n_{out}$ are the number of input and output units respectivly.
Setting this equal to \eqref{eq:variance_sigma} and solving for $\sigma$ gives $\sigma = 1/\sqrt{2(n_{in}+n_{out})}$.
To follow He et al. \cite{he2015delving} initialization that is specialized for rectified linear units (ReLUs) \cite{nair2010rectified}, then we have $\mbox{Var}(W) = 2/n_{in}$, which again setting equal to \eqref{eq:variance_sigma} and solving for $\sigma$ gives $\sigma = 1/\sqrt{2n_{in}}$.

As shown in \eqref{eq:quaternion_weight} the weight has components $|W|$, $\theta$, and $\phi$.
We can initialize the magnitude $|W|$ using our four DOF distribution defined with the appropriate $\sigma$ based on which initialization scheme we are following.
The angle components are initialized using the uniform distribution between $-\pi$ and $\pi$ where we ensure the constraint on $\phi$.

\section{Experimental Results}
Our experiments covered image classification using both the CIFAR-10 and CIFAR-100 benchmarks \cite{krizhevsky2009learning} and image segmentation using the KITTI Road Estimation benchmark \cite{Fritsch2013ITSC}.
The CIFAR datasets are $32\times32$ color images of 10 and 100 classes receptively.
Each image only contains one class and labels are provided.
The KITTI dataset is large color images of varying sizes depicting roads as seen from a driver's perspective.
Each image has a corresponding label image in which each pixel is an integer value relating to a class of road or not road.
We chose CIFAR because it is a extremely common benchmark task making it a good sanity check.
The KITTI dataset was chosen because it is a fairly common, color segmentation benchmark and has binary classes which made it a simple test.
All training was done on a single Nvidia 980Ti.

\subsection{Classification}
We use the same architecture as the large model in \cite{trabelsi2017deep}, which is a 110 layer Residual model similar to the one in \cite{he2016deep}.
There is one difference between the real-valued network and the ones used for both the complex and hyper-complex valued networks.
Because the datasets are all real-valued the network must learn the imaginary or quaternion components.
We use the same technique as \cite{trabelsi2017deep} where there is an additional residual block immediately after the input which will learn the hyper-complex components
\begin{equation*}
BN \rightarrow ReLU \rightarrow Conv \rightarrow BN \rightarrow ReLU \rightarrow Conv.
\end{equation*}
One of these blocks exist per imaginary component and are concatenated with the original input image.
Another possible choice if using color images is to use the gray scale image as the real axis and then use the red, green, and blue channels as the $i, j,$ and $k$ axis respectively.
With this choice it is not necessary to use the above block after input to learn the imaginary components.

To maintain the same parameter budget among the three network types we divided the number of filters per layer of the real network by a factor of two for the complex, and by a factor of four for the quaternion.

The architecture for all models consists of 3 stages of repeating residual blocks,
\begin{equation*}
BN \rightarrow ReLU \rightarrow Conv \rightarrow BN \rightarrow ReLU \rightarrow Conv,
\end{equation*}
where at the end of each stage the images are downsized by a strided convolution.
For these classification experiments we ran a shallow network where the stages contained 2, 1, and 1 residual blocks respectively and a deep network where the stages contained 10, 9, and 9 residual blocks respectively.
Each stage also doubles the previous stage's number of convolution kernels.
For example the real model has 32 kernels in the first stage, 64 in the second, and finally 124 in the last.
The last two layers are a global average pooling layer followed by a single fully connected layer with a softmax function used to classify the input as either one of the 10 classes in CIFAR-10 or one of the 100 classes in CIFAR-100.

We also followed their training procedure of using the backpropagation algorithm with Stochastic Gradient Descent with Nesterov momentum \cite{nesterov1983method} set at 0.9.
The norm of the gradients were clipped to 1 and a custom learning rate scheduler was used.
The learning scheduler was the same used in \cite{trabelsi2017deep} for a direct comparison in performance.
The learning rate was initially set to 0.01 for the first 10 epochs and then set to 0.1 from epoch 11-100 and then cut by a factor of 10 at epochs 120 and 150.
Table~\ref{t:results1} presents our results alongside the real and complex valued networks.
Our quaternion models outperform the real and complex networks on both datasets with a smaller parameter count.
The quaternion models do take roughly 50\% longer to train due to the computationally intense operations of quaternion batch normalization.

\begin{table}[h]
	\centering
		\begin{tabular}{l c c c c}
			\hline
			Architecture & Params & CIFAR-10 & CIFAR-100 \\
			\hline
			Shallow Real & 508,932 & 6.82 & 32.02 \\
			Shallow Complex & 257,412 & 6.91 & 31.65 \\
			Shallow Quaternion & 133,560 & \textbf{6.77} & \textbf{30.59} \\
			\hline
			Deep Real & 3,619,844 & 6.37 & 28.07 \\
			Deep Complex & 1,823,620 & 5.60 & 27.09 \\
			Deep Quaternion & 932,792 & \textbf{5.44} & \textbf{26.01} \\
			\hline
		\end{tabular}
	\caption{Classification error on CIFAR-10 and CIFAR-100. Params is the total number of parameters.}
	\label{t:results1}
\end{table}

\subsection{Segmentation}
For this experiment we used the same model as the above, but cut the number of residual blocks out of the model for memory reasons given that the KITTI data is large color images about $1200 \times 375$ pixels in size.
We only use one model due to resource limitations for this experiment.
It is the same as the small model from the classification experiments which has 2, 1, and 1 blocks at the three stages, but it does not perform any strided convolutions.
It also does not have the global average pooling layer or the fully connected layer.

The last layer is a $1 \times 1$ convolution with a sigmoid output so we are getting a heatmap prediction the same size as the input.
The training procedure is also as above, but the learning rate is scheduled differently.
Here we begin at 0.01 for the first 10 epochs and then set it to 0.1 from epoch 11-50 and then cut by a factor of 10 at 100 and 150.
Table~\ref{t:results2} presents our results along side the real and complex valued networks where we used Intersection over Union (IOU) for performance measure.
Quaternion outperformed the other two by a larger margin compared to the classification tasks and again, with a smaller parameter count.

\begin{table}[h]
	\centering
		\begin{tabular}{l c c}
			\hline
			Architecture & Params & KITTI \\
			\hline
			Real & 507,029 & 0.747 \\
			Complex & 254,037 & 0.769 \\
			Quaternion & 128,701 & \textbf{0.827}
		\end{tabular}
	\caption{IOU on KITTI Road Estimation benchmark.}
	\label{t:results2}
\end{table}

\section{Conclusions}
We have extended upon work looking into complex valued networks by exploring quaternion values.
We presented the building blocks required to build and train deep quaternion networks and used them to test residual architectures on two common image classification benchmarks.
We show that they have competitive performance by beating both the real and complex valued networks with less parameters.
Future work will be needed to test quaternion networks for more segmentation datasets and for audio processing tasks.

\section{Acknowledgment}
We would like to thank James Dent of the University of Louisiana at Lafayette Physics Department for helpful discussions.
We also thank Fugro for research time on this project.

\bibliography{bib}{}
\bibliographystyle{ieeetr}

\section{Appendix}
\subsection{The Generalized Quaternion Chain Rule for a Real-Valued Function}\label{a:diff}
We start by specifying the Jacobian.
Let $L$ be a real valued loss function and $q$ be a quaternion variable such that $q = a+\textit{i}~b+\textit{j}~c+\textit{k}~d$ where $a,b,c,d \in \mathbb{R}$ then,
\begin{align}
\nabla_L(q) &= \frac{\partial L}{\partial q} = \frac{\partial L}{\partial a} + \textit{i}~\frac{\partial L}{\partial b} + \textit{j}~\frac{\partial L}{\partial c} + \textit{k}~\frac{\partial L}{\partial d} \\ \nonumber
&= \frac{\partial L}{\partial \mathbb{R}(q)} + \textit{i}~\frac{\partial L}{\partial \mathbb{I}(q)} + \textit{j}~\frac{\partial L}{\partial \mathbb{J}(q)} + \textit{k}~\frac{\partial L}{\partial \mathbb{K}(q)} \\ \nonumber
&= \mathbb{R}(\nabla_L(q)) + i~\mathbb{I}(\nabla_L(q)) + j~\mathbb{J}(\nabla_L(q)) + k~\mathbb{K}(\nabla_L(q))
\label{eq:diff1}
\end{align}
Now let $g = m+\textit{i}~n+\textit{j}~o+\textit{k}~p$ be another quaternion variable where $q$ can be expressed in terms of $g$ and $m,n,o,p \in \mathbb{R}$ we then have,
\begin{align}
\nabla_L(q) &= \frac{\partial L}{\partial g} = \frac{\partial L}{\partial m} + \textit{i}~\frac{\partial L}{\partial n} + \textit{j}~\frac{\partial L}{\partial o} + \textit{k}~\frac{\partial L}{\partial p} \\ \nonumber
&= \frac{\partial L}{\partial a}\frac{\partial a}{\partial m} + \frac{\partial L}{\partial b}\frac{\partial b}{\partial m} + \frac{\partial L}{\partial c}\frac{\partial c}{\partial m} + \frac{\partial L}{\partial d}\frac{\partial d}{\partial m} \\ \nonumber
&~~+ \textit{i}~\left( \frac{\partial L}{\partial a}\frac{\partial a}{\partial n} + \frac{\partial L}{\partial b}\frac{\partial b}{\partial n} + \frac{\partial L}{\partial c}\frac{\partial c}{\partial n} + \frac{\partial L}{\partial d}\frac{\partial d}{\partial n} \right) \\ \nonumber
&~~+ \textit{j}~\left( \frac{\partial L}{\partial a}\frac{\partial a}{\partial o} + \frac{\partial L}{\partial b}\frac{\partial b}{\partial o} + \frac{\partial L}{\partial c}\frac{\partial c}{\partial o} + \frac{\partial L}{\partial d}\frac{\partial d}{\partial o} \right) \\ \nonumber
&~~+ \textit{k}~\left( \frac{\partial L}{\partial a}\frac{\partial a}{\partial p} + \frac{\partial L}{\partial b}\frac{\partial b}{\partial p} + \frac{\partial L}{\partial c}\frac{\partial c}{\partial p} + \frac{\partial L}{\partial d}\frac{\partial d}{\partial p} \right) \\ \nonumber
&= \frac{\partial L}{\partial a} \left( \frac{\partial a}{\partial m} + \textit{i}~\frac{\partial a}{\partial n} + \textit{j}~\frac{\partial a}{\partial o} + \textit{k}~\frac{\partial a}{\partial p} \right) \\ \nonumber
&~~+ \frac{\partial L}{\partial b} \left( \frac{\partial b}{\partial m} + \textit{i}~\frac{\partial b}{\partial n} + \textit{j}~\frac{\partial b}{\partial o} + \textit{k}~\frac{\partial b}{\partial p} \right) \\ \nonumber
&~~+ \frac{\partial L}{\partial c} \left( \frac{\partial c}{\partial m} + \textit{i}~\frac{\partial c}{\partial n} + \textit{j}~\frac{\partial c}{\partial o} + \textit{k}~\frac{\partial c}{\partial p} \right) \\ \nonumber
&~~+ \frac{\partial L}{\partial d} \left( \frac{\partial d}{\partial m} + \textit{i}~\frac{\partial d}{\partial n} + \textit{j}~\frac{\partial d}{\partial o} + \textit{k}~\frac{\partial d}{\partial p} \right) \\ \nonumber
&= \frac{\partial L}{\partial \mathbb{R}(q)} \left( \frac{\partial a}{\partial m} + \textit{i}~\frac{\partial a}{\partial n} + \textit{j}~\frac{\partial a}{\partial o} + \textit{k}~\frac{\partial a}{\partial p} \right) \\ \nonumber
&~~+ \frac{\partial L}{\partial \mathbb{I}(q)} \left( \frac{\partial b}{\partial m} + \textit{i}~\frac{\partial b}{\partial n} + \textit{j}~\frac{\partial b}{\partial o} + \textit{k}~\frac{\partial b}{\partial p} \right) \\ \nonumber
&~~+ \frac{\partial L}{\partial \mathbb{J}(q)} \left( \frac{\partial c}{\partial m} + \textit{i}~\frac{\partial c}{\partial n} + \textit{j}~\frac{\partial c}{\partial o} + \textit{k}~\frac{\partial c}{\partial p} \right) \\ \nonumber
&~~+ \frac{\partial L}{\partial \mathbb{K}(q)} \left( \frac{\partial d}{\partial m} + \textit{i}~\frac{\partial d}{\partial n} + \textit{j}~\frac{\partial d}{\partial o} + \textit{k}~\frac{\partial d}{\partial p} \right) \\ \nonumber
&= \mathbb{R}(\nabla_L(q)) \left( \frac{\partial a}{\partial m} + \textit{i}~\frac{\partial a}{\partial n} + \textit{j}~\frac{\partial a}{\partial o} + \textit{k}~\frac{\partial a}{\partial p} \right) \\ \nonumber
&~~+ \mathbb{I}(\nabla_L(q)) \left( \frac{\partial b}{\partial m} + \textit{i}~\frac{\partial b}{\partial n} + \textit{j}~\frac{\partial b}{\partial o} + \textit{k}~\frac{\partial b}{\partial p} \right) \\ \nonumber
&~~+ \mathbb{J}(\nabla_L(q)) \left( \frac{\partial c}{\partial m} + \textit{i}~\frac{\partial c}{\partial n} + \textit{j}~\frac{\partial c}{\partial o} + \textit{k}~\frac{\partial c}{\partial p} \right) \\ \nonumber
&~~+ \mathbb{K}(\nabla_L(q)) \left( \frac{\partial d}{\partial m} + \textit{i}~\frac{\partial d}{\partial n} + \textit{j}~\frac{\partial d}{\partial o} + \textit{k}~\frac{\partial d}{\partial p} \right)
\label{eq:diff2}
\end{align}

\subsection{Whitening a Matrix}\label{a:whitening}
Let $\textbf{X}$ be an $n$~x~$n$ matrix and $\mbox{cov}(\textbf{X}) = \mathbf{\Sigma}$ is the symmetric covariance matrix of the same size.
Whitening a matrix linearly decorrelates the input dimensions, meaning that whitening transforms $\textbf{X}$ into $\textbf{Z}$ such that $\mbox{cov}(\textbf{Z}) = \textbf{I}$ where $\textbf{I}$ is the identity matrix \cite{kessy2017optimal}.
The matrix $\textbf{Z}$ can be written as:
\begin{equation}
\textbf{Z} = \textbf{W}(\textbf{X} - \mu)
\label{eq:white1}
\end{equation}
where $\textbf{W}$ is an $n$~x~$n$ `whitening' matrix. Since $\mbox{cov}(\textbf{Z}) = \textbf{I}$ it follows that:
\begin{align}
&\mathbb{E}[\mathbf{Z}\mathbf{Z}^T] = \mathbf{I} \nonumber \\
&\mathbb{E}[\mathbf{W}(\mathbf{X - \mu})(\mathbf{W}(\mathbf{X} - \mu))^T] = \mathbf{I} \nonumber \\
&\mathbb{E}[\mathbf{W}(\mathbf{X - \mu})(\mathbf{X} - \mu)^T\mathbf{W}^T] = \mathbf{I} \nonumber \\
&\mathbf{W}\Sigma\mathbf{W}^T = \mathbf{I} \nonumber \\
&\mathbf{W}\Sigma\mathbf{W}^T\mathbf{W} = \mathbf{W} \nonumber \\
&\mathbf{W}^T \mathbf{W} = \mathbf{\Sigma}^{-1} \label{eq:white2}
\end{align}
From \eqref{eq:white2} it is clear that the Cholesky decomposition provides a suitable (but not unique) method of finding $\textbf{W}$.

\subsection{Cholesky Decomposition}
Cholesky decomposition is an efficient way to implement LU decomposition for symmetric matrices, which allows us to find the square root.
Consider $\textbf{A}\textbf{X} = \textbf{b}$, $\textbf{A}=[a_{ij}]_{n\times n}$, and $a_{ij} = a_{ji}$, then the Cholesky decomposition of $\textbf{A}$ is given by $\textbf{A} = \textbf{L}\textbf{L}'$ where
\begin{equation}
\textbf{L}=
\begin{bmatrix}
 l_{11} & 0 & \ldots & 0 \\
 l_{21} & l_{22} & \ldots & \vdots \\
 \vdots & \vdots & \ddots & 0 \\
 l_{n1} & l_{n2} & ... & l_{nn} \\
\end{bmatrix}
\label{eq:cholesky1}
\end{equation}
Let $l_{ki}$ be the $k^{th}$ row and $i^{th}$ column entry of $\textbf{L}$, then

\[
   l_{ki} =
	 \begin{cases}
      0, & k < i \\
      \sqrt{a_{ii} - \sum_{j=1}^{i-1}l^2_{kj}}, & k=i \\
      \frac{1}{l_{ii}} (a_{ki} - \sum_{j=1}^{i-1}l_{ij}l_{kj}), & i < k
   \end{cases}
\]

\subsection{4 DOF Independent Normal Distribution}
Consider the four-dimensional vector $\textbf{Y} = (S,T,U,V)$ which has components that are normally distributed, centered at zero, and independent.
Then $S$, $T$, $U$, and $V$ all have density functions
\begin{equation}
f_S(x;\sigma) = f_T(x;\sigma) = f_U(x;\sigma) = f_V(x;\sigma) = \frac{e^{-x^2/(2\sigma^2)}}{\sqrt{2\pi\sigma^2}}.
\label{eq:single_dists}
\end{equation}
Let $\textbf{X}$ be the length of $\textbf{Y}$, which means $\textbf{X} = \sqrt{S^2+T^2+U^2+V^2}$.
Then $\textbf{X}$ has the cumulative distribution function
\begin{equation}
F_X(x;\sigma) = \int \!\!\!\int \!\!\!\int \!\!\!\int_{H_x} \!\!f_S(\mu;\sigma)f_T(\mu;\sigma)f_U(\mu;\sigma)f_V(\mu;\sigma) ~dA,
\label{eq:cumdist}
\end{equation}
where $H_x$ is the four-dimensional sphere
\begin{equation}
H_x = \left\{(s,t,u,v)~:~\sqrt{s^2+t^2+u^2+v^2} < x \right\}.
\label{eq:4dsphere}
\end{equation}
We then can write the integral in polar representation
\begin{align}
F_X(x;\sigma) = & ~\frac{1}{4\pi^2\sigma^4} \!\int_0^\pi \!\!\!\!\int_0^\pi \!\!\!\!\int_0^{2\pi} \!\!\!\!\!\int_0^x \!r^3e^{\frac{-r^2}{2\sigma^2}} \mbox{sin}(\theta) \mbox{sin}(\phi) \mbox{cos}(\psi) ~dr d\theta d\phi d\psi \nonumber \\
= & ~\frac{1}{2\sigma^4} \int_0^x r^3e^{-r^2/(2\sigma^2)} ~dr.
\label{eq:polarint}
\end{align}
The probability density function of $\textbf{X}$ is the derivative of its cumulative distribution function so we use the funamental theorem of calculus on \eqref{eq:polarint} to finally arrive at
\begin{equation}
f_X(x;\sigma) = \frac{d}{dx}F_X(x;\sigma) =  ~\frac{1}{2\sigma^4} x^3e^{-x^2/(2\sigma^2)}.
\label{eq:finaldist}
\end{equation}

\end{document}


\documentclass[a4paper,11pt]{article}
\usepackage{preamble}

\title{Machine Learning Course\\\large{Notebook}}
\author{Mariana Jó}
\date{\today}

\begin{document}
  \maketitle
  \thispagestyle{fancy}

  \section{What is Machine Learning?}
  \begin{itemize}
    \item Definitions of ML \\
    \textit{"A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T, as measured by P, improves with experience E."}
    \item Kinds of ML algorithms
  \end{itemize}

  \section{Supervised learning}
  Some definitions:
  \begin{itemize}
    \item Supervised Learning: We give the dataset the "right answers"
    \item Regression: predict continous valued output
    \item Classification: discrete valued output
    \item Support Vector Machines: an algorithm that can deal with an infinite number of features
  \end{itemize}
  \section{Unsupervised learning}
  \textit{"Unsupervised learning allows us to approach problems with little or no idea what our results should look like. We can derive structure from data where we don't necessarily know the effect of the variables. With unsupervised learning there is no feedback based on the prediction results."}
  \begin{itemize}
    \item Clustering: groups things that are somehow similar and/or related
    \item Non-clustering: finds structure. Ex.: cocktail party algorithm.
  \end{itemize}

  \section{Model representation}
  \textbf{Notation:}
  \begin{itemize}
    \item $m=$ Number of training examples
    \item $x=$ "input" variable / features
    \item $y=$ "output" variable / "target" variable
    \item $(x,y)=$ one training example
    \item $\theta=$parameter
  \end{itemize}
  About hypothesis $h$:
  \begin{itemize}
    \item Hypothesis $h$: takes the input $x$ and outputs the estimated value $y$, i.e., $h$ maps from $x$'s to $y$'s.
    \item How do we represent $h$? \\
    We choose an initial choice
    \begin{equation}
    h_\theta (x)=\theta_0 +\theta_1 x  ,
    \label{eq:hypothesis}
    \end{equation}
    which is a linear function.
    \item Linear Regression with one variable $=$ Univariate Linar Regression
  \end{itemize}
  Cost function:
  \begin{itemize}
    \item We try to minimize $\theta_0 \theta_1$
    \item So we try to minimize
    \begin{equation}
    J(\theta_0, \theta_1)=\frac{1}{2m}\sum_{i=1}^m \left(h_\theta (x^i) - y^i\right)^2
  \end{equation}
  where $h_\theta (x)$ is given by \ref{eq:hypothesis} and $J(\theta_0, \theta_1)$ is called the \textbf{cost function}. This function is also called \textbf{Squared error function} or \textbf{Mean Squared Error (MSE)}.
  \item The mean is halved $(\frac{1}{2})$ as a convenience for the computation of the gradient descent, as the derivative term of the square function will cancel out the $\frac{1}{2}$ term.
  \end{itemize}

  \section{Parameter Learning}
  Gradient descent
  \begin{itemize}
    \item Given $J(\theta_0, \theta_1)$, we want to minimize it.
    \item Start with some $\theta_0, \theta_1$
    \item Keep changing $\theta_0, \theta_1$ to reduce $J(\theta_0, \theta_1)$
    \item Gradient descent algorithm: \\
    Repeat until convergence:
    \begin{equation}
      \theta_j := \theta_j - \alpha \frac{\partial}{\partial \theta_j}
      J(\theta_0, \theta_1), \quad (\text{for} \ j=0 \ \text{and} \ j=1)
      \label{eq:gradient_descent}
    \end{equation}
    where $\alpha$ is the \textbf{learning rate} and $\alpha \geqslant   0$.
    \item It is importante to make the simultaneous update of $\theta_0$ and $\theta_1$.
    \item "Batch" Gradient descent means that each step of gradient descent uses all the training examples, i.e., all the $m$ examples.
  \end{itemize}
  For Linear Regression:
  \begin{itemize}
    \item $J(\theta_0, \theta_1)$ will always be a Convex Function, i.e., a "bowl-shaped function"
    \item This implies that it does not have any local optima, only the global one.
  \end{itemize}

  \section{Prediction and Inference}

  \begin{itemize}
    \item Prediction: you use variables to predict \textbf{values}
    \item Inference: you use variables to understand behaviors, find structures, etc
  \end{itemize}

  \subsection*{Prediction accuracy vs. Model interpretability}

  Usually, when you want to \textit{predict}, you don't care very much about \textbf{interpretability}, because what matters is the accuracy and not how variables are correlated, and then you can have more \textbf{flexibility}. On the other hand, when you want to make an \textit{inference}, you also want to \textbf{interpret} the algorithm's outcomes, therefore you will end up using less \textbf{flexible} methods.

  \begin{quotation}
    \textit{"In general, as the flexibility of a method increases, its interpretability decreases."} - ISLR-Gareth \cite{gareth}.
  \end{quotation}

\begin{figure}[h]
  \includegraphics[width=0.75\textwidth]{trade-off_garneth}
  \caption{How the learning algorithms are classified in terms of flexibility vs. interpretability, from ISLR-Gareth \cite{gareth}.}
  \label{}
\end{figure}

\section{Multivariate Linear Regression}

Let
\begin{equation}
  \mathbf{x}=
  \begin{bmatrix}
    x_0 \\ x_1 \\ x_2 \\ \vdots \\ x_n
  \end{bmatrix} \in \mathbb{R}^{n+1}
  \quad \text{and} \quad
  \theta=
  \begin{bmatrix}
    x_0 \\ x_1 \\ x_2 \\ \vdots \\ x_n
  \end{bmatrix} \in \mathbb{R}^{n+1}
\end{equation}

then the Multivariate Linear Regression is defined by
\begin{equation}
  h_\theta(x)=\theta_0 x_0 + \theta_1 x_1 + \dots + \theta_n x_n = \theta^T \mathbf{x}
\end{equation}

\subsection*{Gradient descent}
In the case $n\geq 1$ case, you should repeat:
\begin{equation}
  \theta_j := \theta_j - \alpha\frac{1}{m}\sum_{i=1}^m (h_\theta (x^{(i)})-y^{(i)})x_{j}^{(i)}
\end{equation}

There are some tricks to help the Gradient Descent converge more quickly:
\begin{itemize}
  \item Feature scaling
    \subitem get every feature into approximately a $-1 \leq x \leq 1$ range
    \subitem mean normalization: $\frac{x_i-\mu_i}{s_i}$ where $\mu_i$ is the mean of the $x_i$ and $s_i$ is the range.
  \item Debugging: your $J(\theta)$ should descrease at every iteration. If it's increasing, or if it shows an "weird" behavior (like descreasing and increasing over and over) maybe you should use a smaller $\alpha$. But if $alpha$ is too small, gradient descent can be slow to converge.
\end{itemize}

\subsection*{Normal equation}
It is a method to solve for $\theta$ "analytically". In this case, you don't need to use feature scaling.

\begin{equation}
  \theta=(X^T X)^{-1}X^T y
\end{equation}

Gradient descent
\begin{itemize}
  \item Need choose $\alpha$
  \item Needs many itrations
  \item Works well even when $n$ is large
\end{itemize}

Normal equation
\begin{itemize}
  \item No need to chosse $\alpha$
  \item Don't need to iterate
  \item Need to compute $(X^TX)^{-1}$
  \item Slow if $n$ is very large (usually for $n\geq 10000$)
\end{itemize}

$X^TX$ could be non-invertible if
\begin{itemize}
  \item The features are linear dependente
  \item You have too many features. You should delete some features or use regulatization.
\end{itemize}

\clearpage
\begin{thebibliography}{99}
  \bibitem{gareth}
  Gareth James, Daniela Witten, Trevor Hastie and Robert Tibshirani,
  \textit{Introduction to Statistical Learning with Applications in R},
  Springer,
  2017.

\end{thebibliography}

\end{document}


\section{Tricks}
\begin{enumerate}

\item
The last result is stored in the symbol \verb$last$.

\item
In a script, setting \verb$trace=1$
causes each line to be printed just before it is evaluated.
Useful for debugging.

\item
Use \verb$contract(A)$ to get the mathematical trace of matrix \verb$A$.

\item
Calculations in a script can span multiple lines.
The trick is to arrange things so the parser will keep going.
For example, if a calculation ends with a plus sign, the parser will go to the next line to get another term.
Also, the parser will keep going when it expects a close parenthesis.

\item
Use \verb$binding(s)$ to get the unevaluated binding of symbol \verb$s$.

\item
Use \verb$s=quote(s)$ to clear symbol \verb$s$.

\item
Use \verb$float(pi)$ to get the floating point value of $\pi$.

\end{enumerate}


\chapter{\texttt{Mesh} class}

\section{Motivation}
To understand, the design behind the Mesh class, we would like to begin by discussing one of the key problems currently faced in \texttt{dgswem}. The issue comes from Steven Brus' dissertation \cite{brusDiss}. Brus goes on to show that the addition of curvilinear elements plays a significant role in obtaining high-order accuracy for real world problems. Simultaneously, these curvilinear elements incur a significantly higher computational cost. Thus in the interior of the finite element domain, where the solution doesn't exhibit mesh geometry to more computationally effient standard affine elements suffice. While computationally straight forward, this approach provides a software engineering challenge. Namely, something like the volume kernel now needs to be split up into two loops, i.e.
\begin{lstlisting}[language=c++, caption={Na\"ive implementation of curved and linear finite element kernel}, label=lst:badloops]
for ( auto& elt : linear_elements ) {
  //Evaluate volume kernel for each linear element
}

for ( auto& elt : curved_elements ) {
  //Evaluate volume kernel for each curved element
}
\end{lstlisting}
This can quickly provide a software engineering headache. For instance, the addition of quadrilateral elements, would suddenly require that each time step cover 4 loops. The addition of more features will lead to more code bloat ultimately resulting in maintainable code. The main goal of the Mesh class is to address specifically the problem, mentioned above.

\section{The Element Class}
The object-oriented solution approach is based on the fact that the discontinuous Galerkin algorithm ultimately, simply applies integrals over the elements, remaining mathematically agnostic to the actually implementation of the approximation. This lends itself to a polymorphic implementation. The idea behind polymorphism being that functionally all elements should behave identically. Thus, if we can agree to an interface that we would like to expose. We should be able to accomplish everything in one loop over the elements without having to worry about what's happening under the hood.

Regardless of the type of element, we assume that for any given element, the approximated solution $u^h$ is given in the form
\begin{equation*}
u^h(x,t) = \sum_{i=0}^{N} u_n(t) \phi_n(x),
\end{equation*}
where $\phi_n$ is some basis. The approximated solution is determined by the evolution of the functions $u_n(t)$. For the Galerkin Method, this evolution is ultimately determined by ensuring that the residuals of the approximate solution is orthogonal to the space spanned by $\{\phi_n\}$. Thus the key functionality that every element must possess is that ability to compute integrals of the form
\begin{equation*}
\int_{\Omega} f \phi_n \d x
\end{equation*}
for an arbitrary function $f$. In practice these integrations are approximated by quadrature rules. However, this provides the basis for the interface, we would like to expose from a generic element class.
\begin{lstlisting}[language=c++, caption={Generic Element API}]
class Element {

  //Compute the value of F at the Gauss-points
  template <typename F>
  void ComputeFgp(const F& f, std::vector<double>& f_gp);

  //Compute the value of  a function u given by
  // basis coefficients u at the Gauss-Points
  void ComputeUgp(const std::vector<double>& u,
                  std::vector<double>& u_gp);

  //Compute the gradient of a function given by
  // basis coefficients u at the Gauss Points
  void ComputeDUgp(const uint dir,
                   const std::vector<double>& u,
                   std::vector<double>& du_gp);

  //Compute the integral of u
  double Integration(const std::vector<double>& u_gp);

  //Test u against basis function dof
  double IntegrationPhi(const uint dof,
                        const std::vector<double>& u_gp);

  //Test u against the derivative in direction dir
  // of basis function dof
  double IntegrationDPhi(const uint dir,
                         const uint dof,
                         const std::vector<double>& u_gp);

  //...
};
\end{lstlisting}
That is to say if every element satisfied this API, we would be able to write the discontinuous Galerkin kernel with only one loop regardless of the element.

\subsection{Strong Typing}
C++ is a strongly typed language. Thus, in exposing any kind of polymorphism there are two options. (1) Dynamic polymorphism and (2) Static polymorphism. Dynamic polymorphism is typically achieved through the use of the  \lstinline[language=c++]{virtual} keyword. This approach is typically considered the most readable, and typically be preferred in a first attempt at an implementation. However, virtual objects typically can't be resolved by the compiler at runtime. This means that when the executable is running, the compiler maintains a virtual look-up table, which it uses to determine which implementation to call, when provided one of these virtual calls. The cost of this is that there is a virtual overhead associated with each of these calls. Additionally, the compiler may have trouble inlining and optimizing virtual function calls. Typically, when the function execution is large enough this overhead may be treated as negligible. But for our applications, these function calls are expensive enough that virtualization would seriously degrade performance.

The other typical approach is the use of static polymorphism. The idea being that we still maintain a unified API exposed by the class. However, now we specify the order in which we loop through the elements. That way the compiler is able to determine each of the function calls. Although ultimately, we will be able to get away with still only having one loop. The binary code generated should be equivalent to the code in Listing \ref{lst:badloops}. Thus, static polymorphism provides us with the usage of dynamic polymorphism without the additional overhead. The downsides include a code base that is more difficult to maintain. However, given the performance critical nature of these function calls, this is a downside we've decided to accept.

\subsection{Class Hierachy of Element}
\begin{figure}
\centering
\begin{tikzpicture}[edge from parent/.style={draw,<-,>=latex}, sibling distance=10em,
  every node/.style = {shape=rectangle, rounded corners,
    draw, align=center,
    top color=white, bottom color=blue!20}]]
  \node {Element}
    child { node {Shape} }
    child { node {Master}
      child { node {Integration} }
      child { node {Basis} } };
\end{tikzpicture}
\caption{Composition of the Element classes.}
\label{fig:eltcomp}
\end{figure}
The last subsection describes the exact decomposition of the element. In designing \texttt{dgswem-v2}, we attempted to think of all possible features that one might want to implement and provide encapsulation to allow for adding of new features without having to be familiar with the entire code base. Figure \ref{fig:eltcomp} describes the major constituents of the element class.
\begin{itemize}
\item \textbf{Shape}: The shape class deals with deformations from the master element to the actual orientation within the mesh. Here features such as curvature of the mesh (e.g. solving in spherical coordinates) or the element (e.g. isoparametric or isogeometric) should be implemented.
\item \textbf{Master}: The master element is rather similar to the generic element class. It exposes integration hooks over the master element.
\item \textbf{Basis}: This class encapsulates all basis information. Potentially additional choices of basis include Bernstein, modal, and bases on quadrilateral elements.
\item \textbf{Quadrature}: This class contains all information required to approximate an integral over the master element.
\end{itemize}
In addition to adding features by providing new instances of these classes. There also still remains the ability to add classes through template specialization and SFINAE. These techniques allow for special implementations to be written in certain element compositions. For example, one of the large advantages of a Bernstein basis is the fast matrix inversion formula. Since this formula drastically deviates from linear elements, one might want to write a specific implementation for linear triangles using the Bernstein basis.
\section{\texttt{Mesh} Class}
With the decision to strongly type the elements in the Mesh class, the next task is to develop a container, which represents the mesh. This requires the ability to iterate over interfaces, boundaries, elements, and distributed boundaries. The key object in allowing this to are the heterogeneous containers in the \textbf{Utilities} namespace. To explain, what in particular is happening in these containers assume we have three types of elements in our mesh: \lstinline{EltA, EltB, EltC}.
Morally, the heterogeneous vector can be written out as:
\begin{lstlisting}[language=c++]
using HeterogeneousVector<EltA,EltB,EltC> = tuple<vector<EltA>,
                                                  vector<EltB>,
                                                  vector<EltC>>;
\end{lstlisting}
Now assuming, we have a function we would like to execute on each element regardless of type. We emulate the \lstinline[language=c++]{std::for_each} API. So we will define a hook in the Mesh class, which will execute that function for every element in the HeterogeneousVector. We have demonstrated what specifically we would like in Listing \ref{lst:foreach}.
\begin{lstlisting}[language=c++,
                   caption={Moral implementation of iterating over HeterogeneousVector},
                   label=lst:foreach]
template<typename Element>
void SomeKernel(Element& e);

template<typename F>
void ForEachImpl( HeterogeneousVector<EltA,EltB,EltC>& v,
                   const F& f ) {

  vector<EltA>& vA = get<0>(v);
  for_each(vA.begin(),vA.end(), f);

  vector<EltB>& vB = get<1>(v);
  for_each(vB.begin(), vB.end(), f);

  vector<EltC>& vC = get<2>(v);
  for_each(vC.begin(), vC.end(), f);
}

//Apply SomeKernel to every element in v
MoralForImpl(v, SomeKernel);
\end{lstlisting}
In \texttt{dgswem-v2}, this type of implementation can be generalized to arbitrary Hetereogeneous vectors through the use of template metaprogramming. However, ultimately, the code is effectively executing the above code. Allowing the compiler to generate these loops is precisely the issue we wanted to address. The definition of the kernels to be passed into the mesh class are problem specific and will be addressed in the next chapter. This mechanism is used for boundaries, internal interfaces, and distributed interfaces, in addition to elements. In understanding, this key concept the Mesh class becomes nothing more than a container. Detailed API descriptions are provided in the doxygen documentation.


\title{Computer Architecture - CS 301} % You may change the title if you want.

\author{Rishit Saiya - 180010027, Assignment - 12}

\date{\today}

\documentclass[12pt]{article}
\usepackage{fullpage}
\usepackage{enumitem}
\usepackage{amsmath,mathtools}
\usepackage{amssymb}
\usepackage[super]{nth}
\usepackage{textcomp}
\usepackage{hyperref}
\hypersetup{
    colorlinks=true,
    linkcolor=blue,
    filecolor=magenta,
    urlcolor=cyan,
}
\begin{document}
\maketitle

%----------------------------------------------------------------

\section{}
This is a direct application to find the CPI as mentioned in the video lecture. Figure 1 shows the formula we will be using which is given in lecture.

\begin{figure}
    \centering
    \includegraphics[width = 13cm, height = 2cm]{Assignment-12/Formula1.png}
    \caption{CPI Calculation}
    % \label{fig:my_label}
\end{figure}
In here, 1 is the $L1_{Hit \, Time}$. So, formula might change accordingly. \\
To get an overview, we will try to list down all the given values to us. They are as follows: \\
Baseline IPC = 0.8 \\
Fraction of Memory Operations = 30 \% \\
L1 Cache Hit Rate = 100 \% \\
L1 Cache Hit Time = 1 Cycle \\
L1 Data Cache Miss Rate = 5 \% \\
L2 Cache Miss Rate = 50 \% \\
L2 Cache Hit Time = 10 cycles \\
L2 Miss Penalty = 100 cycles \\

\begin{equation*}
    CPI_{ideal} = \frac{1}{IPC_{baseline}}
\end{equation*}
\begin{equation*}
    CPI_{ideal} = \frac{1}{0.8} = 10/8 = 1.25
\end{equation*}
We are given the $f_{mem}$ as follows: \\
\begin{equation*}
    f_{mem} = 30 \% = 0.3
\end{equation*}
So, we calculate the AMAT variable value. We are given in the lecture that AMAT value is as given in Figure 2. With that reference, we will calculate AMAT here.
\begin{figure}
    \centering
    \includegraphics[width = 15cm, height = 2cm]{Assignment-12/Formula2.png}
    \caption{AMAT Calculation}
\end{figure}
\begin{equation*}
    AMAT = L1_{Hit \, Time} + L1_{Miss \, Rate} \times (L2_{Hit \, Time} + (L2_{Miss \, Rate} \times L2_{Miss \, Penalty}))
\end{equation*}
\begin{equation*}
    AMAT = 1 + 0.05 \times (10 + (0.5 \times 100)) = 4
\end{equation*}
So, finally we calculate CPI.
\begin{equation*}
    CPI = 1.25 + 0.3 \times (4 - 1) = 2.15
\end{equation*}
So, final IPC will be as follows:
\begin{equation*}
    IPC = \frac{1}{CPI}
\end{equation*}
\begin{equation*}
    IPC = \frac{1}{2.15} = 0.4651162790
\end{equation*}
%----------------------------------------------------------------

\section{}

Before we attempt to understand the motivation of Prefetching, we need to understand the aspect and environment where it will prove to be effective. As mentioned in the lecture, we have following types of misses:
\begin{itemize}
    \item \textbf{Compulsory Miss:}
    These misses occur when a data stream is read first time. Since it is read first time, it compulsory has to miss and hence the name. The following are some of the suggested mitigation techniques to decrease number of Compulsory Misses:
    \begin{enumerate}
        \item As mentioned in the lecture, we can increase the Block Size, in order to store more data blocks at single read and further more this will lead to usage of spatial locality. With this the number of Compulsory Misses will reduce.
        \item Another proposed mitigation is prefetching the memory locations that are to be used in the near future. This is done by generating an algorithm which can wisely and effectively guess upcoming cache locations based on usage history.
    \end{enumerate}
    \item \textbf{Conflict Miss:}
    These misses occur when there is limited amount of associativity in a direct mapped cache or set associative cache. The following are some of the suggested mitigation techniques to decrease number of Conflict  Misses:
    \begin{enumerate}
        \item A simple way to mitigate is to write algorithms/codes which have cache involvement. Basically, trying to put together code which is cache independent to maximum possible extent.
        \item Another way is to use a smaller fully associative cache which is meant to store the line which is thrown out of main cache so as to use immediately next time instead of fetching. This is generally called as Victim Cache.
        \item We can also increase the associativity of the cache. As mentioned in lecture, this comes with a trade off cost of high consumption of power and latency also increases resulting in slower computation time.
    \end{enumerate}
    \item \textbf{Capacity Miss:}
    These misses occur due to limited size of Cache blocks. The following are some of the suggested mitigation techniques to decrease number of Capacity  Misses:
    \begin{enumerate}
        \item A simple fix would be straight away increase the size of Cache blocks.
        \item Another mitigation technique would be produce a novel and wise technique for prefetching in order to accommodate more such misses.
    \end{enumerate}
\end{itemize}

As clearly mentioned above, we see that two of the misses' mitigation involves Prefetching and making a novel and wise algorithm to predict the memory locations that will be used in the near future would help us to mitigate Compulsory and Capacity Misses. This would increase the overall efficiency output of the Cache and lead to increase in more hits than before. This constitutes the motivation to integrate \textbf{\textit{Prefetching}} in modern caches. \\

In the following sections, we will explain different types of Prefetching and its processes.
\begin{itemize}
    \item \textbf{Hardware Prefetching:} A hardware prefetcher is a dedicated hardware unit that predicts the memory accesses in the near future, and fetches them from the lower levels of the memory system. \\

    In the Figure 3, we can see the basic look of Hardware Prefetcher as mentioned in the video.
    \begin{figure}
        \centering
        \includegraphics[width=10cm , height=8cm]{Assignment-12/Hardware_Prefetch.png}
        \caption{Hardware Prefetcher}
    \end{figure}
In here, the prefetcher checks the hits/misses at the L1 cache. Based on results from L1 cache, now it decides whether the lists of addresses are likely to be accessed in the near future or not. It then sends the data to L2 cache to get in those lines to L1 cache in a such a way that when there is a call for those data, there is a hit in L1 cache and it can easily fetch there onwards.
    \item \textbf{Software Prefetching:} A software prefetcher uses the compiler and predicts future cache misses and inserts a prefetch instruction based on the miss penalty and execution time of the instructions. \\
Below we will consider 2 different code snippets taken from the lecture to explain software Prefetching. Let us consider following Code\_Snippet-1:
\begin{verbatim}
int addAll(int data[], int vals[]) {
      int i, sum = 0;
      for (i = 0; i < N; i++) {
         sum += data[vals[i]];
      }
      return sum;
}
\end{verbatim}
Let us consider following Code\_Snippet-2:
\begin{verbatim}
int addAllP(int data[], int vals[]) {
      int i, sum = 0;
      for (i = 0; i < N; i++) {
         __builtin_prefetch(& data[vals[i + 100]]);
         sum += data[vals[i]];
      }
    return sum;
}
\end{verbatim}
Since \textit{vals} array can have discrete values and can be stored in the sequence, \textit{vals} array possess spatial locality whereas \textit{data} array doesn't has any spatial locality. In such case, we integrate the above function in Code\_Snippet-1 with \textit{\_\_builtin\_prefetch} function which helps us to prefetch the address of data which are to be accessed after such 100 iterations. This helps us to increase the Memory Fetch efficiency of cache and also to reduce the MA time.
\end{itemize}

As much as Prefetching techniques are helping us to mitigate compulsory and capacity misses, there also exists some disadvantages of using it. Some of which are as follows:
\begin{itemize}
    \item Addition of Extra Complexity to the Code Execution
    \item Addition of Risk of displacing useful/frequently used data from the caches.
\end{itemize}
Source used: Reference Book, Video Lecture,  \href{https://en.wikipedia.org/wiki/Cache_prefetching}{Wiki Page}
%----------------------------------------------------------------
\end{document}