The previous chapters focused on using camera models to identify the relationship between points in a 3D scene and their projections onto the camera image, as well as how to leverage those models to reconstruct 3D scene structure from 2D images.
Alternatively, this chapter begins to look at methods for extracting other types of information through \textit{image processing}, for example to answer the question ``what object am I seeing?'' rather than ``how far away is this object?''.
Extracting this type of visual content from raw images is important for mobile robots to be able to intelligently interpret their surroundings. In fact, it can have a major impact on the ability of the robot to perform several tasks including localization and mapping or decision making. This chapter focuses on some of the more commonly used tools in image processing including image filtering, feature detection, and feature description\cite{SiegwartNourbakhshEtAl2011}\cite{Moravec1977}.

\notessection{Image Processing}
Image processing is a form of signal processing where the input signal is an image (such as a photo or a video) and the output is either an image or a set of parameters associated with the image. While a large number of image processing techniques exist, this chapter focuses on some of the more fundamental methods that are relevant for robotics. In particular, these methods will be related to image filtering, feature detection, and feature description\footnote{The software library OpenCV implements a number of useful image filtering algorithms: \url{https://docs.opencv.org}.}.

In the following methods, grayscale images are treated as functions $I$: $[a,b]\times[c,d] \rightarrow [0,L]$, where $I(x,y)$ represents the grayscale pixel intensity at $(x,y)$.
For a color image, $I$ is a vector valued function with three components, one each for the red, green, and blue color channels of the image.

\subsection{Image Filtering}
Image filtering is one of the principal tasks in image processing. The terminology ``filter'' comes from frequency domain signal processing and refers to the process of accepting or rejecting certain frequency components of a signal (e.g. eliminating high-frequency noise).

Perhaps the most common type of image filtering is \textit{spatial filtering}.
The basic principle of spatial filtering is that a particular pixel is modified in the filtered image based only on the pixels in the immediate spatial neighborhood (see Figure \ref{fig:spatial_filter_concept_fig}). To be more specific, a spatial filter for an image $I(x,y)$ consists of:
\begin{enumerate}
    \item A neighborhood $S_{xy}$ of pixels around a particular point $(x,y)$ under examination, typically rectangular.
    \item A predefined operation $F$ that is performed on the image pixels encompassed by the neighborhood $S_{xy}$.
\end{enumerate}
Once the operation $F$ has been applied to all pixels $(x,y)$ in the image $I$ a new image $I'(x,y)$ is defined.
\begin{figure}[ht]
  \centering
  \includegraphics[width=.75\textwidth]{tex/figs/ch10_figs/spatialfiltering.png}
    \caption{Illustration of the concept of spatial filtering. The spatial filter operates on a neighborhood $S_{xy}$ of each point in the original image to produce a new pixel in the filtered image.}
    \label{fig:spatial_filter_concept_fig}
\end{figure}

In general filters can be linear or nonlinear, but many of the most fundamental filters are linear and can be expressed mathematically as:
\begin{equation}
  \label{eq:correlation}
    I'(x,y) = F \circ I = \sum_{i=-N}^N \sum_{j=-M}^M F(i,j)I(x+i,y+j),
\end{equation}
where $N$ and $M$ are integers that define the width and height of a rectangular neighborhood $S_{xy}$. Based on the size of this neighborhood, it is said that this filter is of size $(2N+1) \times (2M+1)$. Additionally, the filter operation $F$ is usually called a \textit{mask} or \textit{kernel}. Broadly speaking, filters expressed by \eqref{eq:correlation} are referred to as \textit{correlation filters}.

Another type of linear filters that are commonly used are referred to as \textit{convolution} filters. Convolution filters are similar to correlation filters but use reverse image indices (in fact correlation and convolution filters are identical when the filter mask is symmetric in both the horizontal and vertical directions). In particular, these filters are expressed mathematically as:
\begin{equation}
  \label{eq:convolution}
    I'(x,y) = F \ast I = \sum_{i=-N}^N \sum_{j=-M}^M F(i,j)I(x-i,y-j).
\end{equation}
Convolution filters are associative, meaning that for two different filter masks $F$ and $G$ it is true that $F*(G*I) = (F*G)*I$. One example of how the associative property is useful is for smoothing an image \textit{before} taking applying a differentiation filter. Suppose the mask $F$ implemented a derivative filter and $G$ implemented a smoothing filter, then sequentially applying these filters would result in $F*(G*I)$. However, because of the associative property the masks can be convolved together \textit{first} such that only one filter needs to be applied to the image (i.e. $(F*G)*I$).

Note that in both the correlation and convolution filters the boundaries of the image need some special care because of the width and height of the mask. For example, Figure \ref{fig:nopaddingexample} shows how the filtered image is smaller than the original due to the width and height of the mask. Some possible options to handle this include padding the image, cropping it, extending it, or wrapping it. However, as images are generally quite large the exact approach likely won't vary the final result significantly.
\begin{figure}[ht]
  \centering
  \includegraphics[width=.65\textwidth]{tex/figs/ch10_figs/centerfilter_nopadding.png}
    \caption{Due to the width and height of the mask, the filtered image may be smaller than the original. However this can be fixed with several techniques, such as padding.}
    \label{fig:nopaddingexample}
\end{figure}

\begin{example}[Practical Considerations for Image Filtering] \label{ex:padding}
\theoremstyle{definition}
Implementation of correlation and convolution filters typically leverages some additional ``tricks'' to make things easier to implement. In this example two such tricks will be introduced: zero-padding and a change in indexing.

First, to more simply accommodate varying sizes of filters (including even and odd sized filters) the indexing is often changed such that the coordinate of interest is associated with the top-left element in the window rather than the center. In particular, for a correlation filter this would correspond to:
\begin{equation}
  \label{eq:correlation_newindex}
    I'(x,y) = F \circ I = \sum_{i=1}^K \sum_{j=1}^L F(x,y)I(x+i-1,y+j-1),
\end{equation}
where $K$ and $L$ are integers that define the width and height of the filter and the pixel $(x,y)$ is at row $x$ and column $y$. However, note that with this formulation the output image $I'$ will be shifted up and to the left. To see this consider the pixel at $x=1$ and $y=1$ in the new image $I'$, which would correspond to the top-left pixel $I'$. This new pixel value is generated by applying the filter $F$ over the pixels in the original image $I$ at rows $\{1,\dots,K\}$ and columns $\{1,\dots,L\}$ (which is not centered at $(1,1)$ in the original image $I$). Therefore it will appear as if the image has been shifted! But in practice this isn't an issue as long as you always index with respect to the top-left corner. An example of top-left indexing is shown in Figure \ref{fig:topleftfilter}
\begin{figure}[ht]
  \centering
  \includegraphics[width=.65\textwidth]{tex/figs/ch10_figs/topleftfilter_nopadding.png}
    \caption{Top-left indexing is typically easier to implement than center indexing. Notice that when top-left indexing, it appears as if the filtered image has shifted with respect to when center indexing is used.}
    \label{fig:topleftfilter}
\end{figure}

Zero-padding (also commonly referred to as \textit{same padding}) is another simple trick that can be used to ensure that the output filtered image $I'$ has the same dimension as the input image $I$. In this approach the left and right boundaries of the image are \textit{each} padded by $\lfloor K/2 \rfloor$ columns of zeros, and the top and bottom boundaries are padded by $\lfloor L/2 \rfloor$ rows of zeros ($\lfloor \cdot \rfloor$ denotes the ``floor'' operation). For example the image:
\begin{equation*}
I = \begin{bmatrix}
    1 & 2 & 3 \\
    4 & 5 & 6 \\
    7 & 8 & 9 \\
    \end{bmatrix},
\end{equation*}
would become
\begin{equation*}
I_\text{padded} = \begin{bmatrix}
    0 & 0 & 0 & 0 & 0 \\
    0 & 1 & 2 & 3 & 0 \\
    0 & 4 & 5 & 6 & 0 \\
    0 & 7 & 8 & 9 & 0 \\
    0 & 0 & 0 & 0 & 0 \\
    \end{bmatrix},
\end{equation*}
for filters $F \in \R^{3\times 3}$, $F \in \R^{2 \times 2}$, $F \in \R^{2 \times 3}$ and $F \in \R^{3 \times 2}$. When using this padding rule with the correlation filter \eqref{eq:correlation_newindex} and a filter $F$ with $K = 2,3$ and $L= 2,3$, the new image $I'$ can be defined for values $x \in \{1,2,3\}$ and $y \in \{1,2,3\}$, resulting in $I'$ being the same dimension as the original image $I$. The use of padding (along with top-left indexing) is also shown graphically in Figure \ref{fig:paddingfilter}
\begin{figure}[ht]
  \centering
  \includegraphics[width=.7\textwidth]{tex/figs/ch10_figs/topleftfilter_padding.png}
    \caption{Image padding is a commonly used technique to ensure that the size of the filtered image is the same size as the original.}
    \label{fig:paddingfilter}
\end{figure}
\end{example}

\subsubsection{Moving Average Filter}
The moving average filter returns the average of pixels in the mask, which achieves a smoothing effect (i.e. removes sharp features in the image). For example, a moving average filter with a normalized $3 \times 3$ mask is defined with the operation $F$ in \eqref{eq:correlation} chosen as:
\begin{equation*}
    F = \frac{1}{9}\begin{bmatrix}
    1 & 1 & 1 \\
    1 & 1 & 1 \\
    1 & 1 & 1 \\
    \end{bmatrix}.
\end{equation*}
Note that due to symmetry of the mask, the correlation \eqref{eq:correlation} and convolution \eqref{eq:convolution} filters will be identical. Additionally, the normalization is used to maintain the overall brightness of the image.

\subsubsection{Gaussian Smoothing Filter}
Gaussian smoothing filters are similar to the moving average filer, but instead of weighting all of the pixels evenly they are weighted by the Gaussian function:
\begin{equation*}
G_\sigma(x,y) = \frac{1}{2\pi\sigma^2} \exp \bigg(-\frac{x^2 + y^2}{2\sigma^2} \bigg).
\end{equation*}
This function is used to obtain the mask operation $F$ by sampling the function about the center pixel (i.e. for the center pixel with $i=j=0$ in \eqref{eq:correlation}, sample $G_\sigma(0,0)$). For example, for a normalized $3\times3$ mask with $\sigma$ = 0.85 this filter is approximately defined by:
\begin{equation*}
F = \frac{1}{16}
\begin{bmatrix}
1 & 2 & 1\\
2 & 4 & 2\\
1 & 2 & 1
\end{bmatrix}.
\end{equation*}
Like the moving average filter, this filter mask is symmetric and therefore yields identical results with respect to the correlation \eqref{eq:correlation} or convolution \eqref{eq:convolution} filters. The advantage of the Gaussian filter is that it provides more weight to the neighboring pixels that are closer.  An example of this filter is shown in Figure \ref{fig:gaussianfilter}.
\begin{figure}[ht]
  \centering
  \includegraphics[width=.8\textwidth]{tex/figs/ch10_figs/gaussianfilter.png}
    \caption{Example of a Gaussian smoothing filter, which produces a smoothing (blurring) effect on the filtered image.}
    \label{fig:gaussianfilter}
\end{figure}

\subsubsection{Separable Masks}
A mask $F$ is called \textit{separable} if it can be broken down into the convolution of two kernels $F = F_1 \ast F_2$. If a mask is separable into ``smaller'' masks, then it is often cheaper to apply $F_1$ followed by $F_2$, rather than by $F$ directly. One special case of this is when the mask can be represented as an outer product of two vectors (meaning it is equivalent to the 2D convolution of those two vectors). If the mask is of shape $M\times M$, and the input image has size $w\times h$, then the computational complexity of directly performing the convolution is $O(M^2wh)$. However, by separating the masks the computational cost is $O(2Mwh)$, which is linear in $M$ rather than quadratic. As an example, consider the moving average filter mask from before:
\begin{equation*}
F = \frac{1}{9}
\begin{bmatrix}
1 & 1 & 1\\
1 & 1 & 1\\
1 & 1 & 1
\end{bmatrix} = \frac{1}{9}
\begin{bmatrix}
1 \\
1 \\
1
\end{bmatrix}
\begin{bmatrix}
1 & 1 & 1\\
\end{bmatrix}.
\end{equation*}
As another example, note that the Gaussian smoothing filter mask is also separable. To see why this is, note that the Gaussian weighting function can be decomposed as:
\begin{equation*}
\begin{split}
G_\sigma(x,y) &= \frac{1}{2\pi\sigma^2} \exp \bigg(-\frac{x^2 + y^2}{2\sigma^2} \bigg), \\
    &= \frac{1}{\sqrt{2\pi}\sigma}\exp \bigg(-\frac{x^2}{2\sigma^2}\bigg)\frac{1}{\sqrt{2\pi}\sigma} \exp \bigg(-\frac{y^2}{2\sigma^2}\bigg), \\
    &= g_\sigma(x) \cdot g_\sigma(y).
\end{split}
\end{equation*}

\subsubsection{Image Differentiation Filters}
Taking the derivative of an image can be used to identify certain features, such as edges. On a basic level, the derivative of an image quantifies changes in pixel intensity in both the vertical and horizontal direction. However, since images are represented as functions defined over a discrete domain the traditional method for differentiating continuous functions can not be used. Instead it is more common to just compute differences between pixels, such as using a central difference method:
\begin{equation} \label{eq:cendiff}
\begin{split}
 \frac{\partial I} {\partial x} &= \frac{I(x+1,y) - I(x-1,y)}{2},\\
\frac{\partial I} {\partial y} &= \frac{I(x,y+1) - I(x,y-1)}{2}.
\end{split}
\end{equation}
where $\partial I/\partial x$ is the derivative in the horizontal direction and $\partial I/\partial y$ is the derivative in the vertical direction. It is of course also possible to define the derivatives using just one side, for example $\frac{\partial I}{\partial x} = I(x+1,y) - I(x,y)$.

It is also possible to differentiate an image using convolution filters. In particular, one common approach is to use a convolution filter \eqref{eq:convolution} defined with a mask $F$ called a \textit{Sobel mask} (also referred to as simply a Sobel operator). For the $x$ direction this mask is denoted as $S_x$ and for the $y$ direction as $S_y$:
\begin{equation}
S_x = \begin{bmatrix}
1 & 0 & -1\\
2 & 0 & -2\\
1 & 0 & -1
\end{bmatrix}, \quad S_y =
\begin{bmatrix}
1 & 2 & 1\\
0 & 0 & 0\\
-1 & -2 & -1
\end{bmatrix}
\end{equation}
Sobel masks are similar to the central difference method but use more neighboring pixels when calculating the derivative (i.e. they also consider the rows above and below to compute the difference). Note that Sobel masks are also separable.

\subsubsection{Similarity Measures}
Filtering can also be used to find similar features in different images, which can be useful for solving the correspondence problem in stereo vision or structure-from-motion techniques.
In particular, the similarity between the pixel $(x,y)$ in image $I_1$ and pixel $(x', y')$ in image $I_2$ can be computed by:
\begin{equation} \label{eq:similarity}
\begin{split}
SAD &= \sum_{i=-N}^N \sum_{j=-M}^M \rvert I_1(x+i,y+j)-I_2(x^\prime+i,y^\prime+j)\lvert, \\
SSD &= \sum_{i=-N}^N \sum_{j=-M}^M [I_1(x+i,y+j)-I_2(x^\prime+i,y^\prime+j)]^2,
\end{split}
\end{equation}
where SAD is an acronym for ``sum of absolute differences'', SSD is an acronym for ``sum of squared differences'', and $N$ and $M$ define the size of the window around the pixels that is considered.

\subsection{Image Feature Detection}
A local feature (also sometimes referred to as interest points, interest regions, or keypoints) in an image is a pattern that differs from its immediate neighborhood in terms of intensity, color, or texture.
Local features can generally be categorized in several ways, for example whether they provide semantic content or not. For example, features that may provide semantic content include edges or other geometric shapes (e.g. lanes of a road or blobs corresponding to blood cells in medical images). These types of features were some of the first for which feature detectors were proposed in the image processing literature. Features that do not provide semantic content may also be useful, for example in feature tracking, camera calibration, 3D reconstruction, image mosaicing, and panorama stitching. In these cases it may be more important that the feature be able to be located accurately and robustly over time. A third category of features are those that may not have semantic interpretations individually, but may have meaning as a collection.
For instance, a scene could be recognized by counting the number of feature matches between the observed scene and a query image. In this case only the number of matches is relevant and not the location or type of feature. Applications where these types of features are important include texture analysis, scene classification, video mining, and
image retrieval.

In this section several feature detection strategies will be discussed. While many strategies exist for different types of features, the focus here will be on two common features that are often useful in robotics: edges and corners.

\subsubsection{Edge Detection}
An \textit{edge} in an image is a region where there is a significant change in intensity values along one direction, and negligible change along the orthogonal direction. In one dimension an edge corresponds to a point where there is a sharp change in intensity, which mathematically can be thought of as a point of a function having a large first derivative and a small second derivative. Many edge detectors rely on this concept by differentiating images and looking for spikes in the derivative.
An edge detector can be evaluated based on several criteria for robustness and performance, including accuracy, localization, and single response. Good accuracy implies few false positives or negatives (missed edges), good localization implies that the detected edge should be exactly where the true edge is in the image, and a single response implies \textit{only} one edge is detected for each real edge. In practice, noise and discretization can make edge detection challenging.

Most edge detection methods rely on two key steps: smoothing and differentiation. Differentiation is performed in both the vertical and horizontal directions to find locations in the image with high intensity gradients. However, differentiation alone is vulnerable to false positives due to image noise, which is why many algorithms will first smooth the image.

\paragraph{Edge Detection in 1D:}
An example of how noise can corrupt image differentiation is given in Figure \ref{fig:noisy}.
\begin{figure}[ht]
  \centering
  \includegraphics[width=0.5\textwidth]{tex/figs/ch10_figs/edge_detection.png}
    \caption{Differentiation of signal (e.g. for edge detection) with noise can be particularly challenging, which can be addressed by first smoothing the signal.}
    \label{fig:noisy}
\end{figure}
Notice that in this case it is impossible to identify the jump in the signal due to the noise levels.
Smoothing filters, such as the Gaussian smoothing filter discussed earlier, can help remedy this problem. In particular, suppose the original signal in Figure \ref{fig:noisy} is defined by $I(x)$. Then a smoothed version can be defined by applying a smoothing convolution filter:
\begin{equation*}
s(x) = g_\sigma(x) \ast I(x),
\end{equation*}
where $g_\sigma(x)$ represents a Gaussian smoothing filter, and then by applying the differentiation filter:
\begin{equation*}
s'(x)=\frac{d}{dx}\ast s(x).
\end{equation*}
This process is shown in Figure \ref{fig:gauss}.
\begin{figure}[ht!]
  \centering
  \includegraphics[width=0.55\textwidth]{tex/figs/ch10_figs/edge_detection_smooth.png}
    \caption{Edge detection through convolution with a Gaussian smoothing filter, followed by a differentiation filter.}
    \label{fig:gauss}
\end{figure}
Note however that since these filters are convolutions, the associativity property can be leveraged to actually combine them into a single filter:
\begin{equation*}
s'=(\frac{d}{dx} * g_\sigma) * I.
\end{equation*}

\paragraph{Edge Detection in 2D:}
Edge detection in a two-dimensional image is quite similar to the example previously discussed in 1D. Let the smoothing filter be the Gaussian smoothing filter from before, and a differentiation filter such as the Sobel filter. The gradient of the smoothed image in both the $x$ and $y$ directions can be written as:
\begin{equation*}
\nabla S= \begin{bmatrix}
\frac{\partial}{\partial x} * G_\sigma * I \\ \frac{\partial}{\partial y} * G_\sigma * I \end{bmatrix}= \begin{bmatrix}
G_{\sigma,x} * I\\G_{\sigma,y} * I
\end{bmatrix}=\begin{bmatrix}
S_x\\S_y
\end{bmatrix},
\end{equation*}
where $I$ is the original image and the associativity properties of the smoothing and differentiation convolution filters is used to define the combined filters $G_{\sigma,x}$ and $G_{\sigma,y}$. The magnitude of the gradient can then be computed by:
\begin{equation*}
\lvert\nabla S\rvert =\sqrt{S_x^2+S_y^2},
\end{equation*}
which can be used to check against a predefined threshold value for edge detection. To guarantee thin edges it is also possible to filter out points whose gradient magnitude are above the threshold but are not local maxima. Examples of this process are shown in Figures \ref{fig:sobel} and \ref{fig:canny}.
\begin{figure}[ht]
  \centering
  \includegraphics[width=.8\textwidth]{tex/figs/ch10_figs/sobel.png}
    \caption{Edge detection using the ``Sobel'' edge detector.}
    \label{fig:sobel}
\end{figure}
\begin{figure}[ht]
  \centering
  \includegraphics[width=.8\textwidth]{tex/figs/ch10_figs/canny.png}
    \caption{Edge detection using the ``Canny'' edge detector.}
    \label{fig:canny}
\end{figure}

\subsubsection{Corner Detection}
A \textit{corner} in an image is defined as an intersection of two or more edges, and also sometimes as a point where there is a large intensity variation in every direction.
Important properties of corner detectors include repeatability and distinctiveness. Repeatability quantifies how well the same features can be found in multiple images even under geometric and photometric transformations. Distinctiveness refers to whether the information carried by the patch surrounding the feature is distinctive, which can be used to reliably produce correspondences. Both of these properties are particularly important in applications such as panorama stitching and 3D reconstruction.

Generally corner detection can be thought of in a similar way to edge detection, except that instead of looking for change along one direction there should be changes in all directions. One well known corner detector is known as the Harris detector \cite{Harris1988}, which has the useful property that the detection is invariant to rotations and linear intensity changes (i.e. geometric and photometric invariance). However the Harris detector is not invariant to scale changes or geometric affine changes, which has led to the development of scale-invariant detectors such as the Harris-Laplacian detector or the scale-invariant feature transform (SIFT) detector.

\subsection{Image Descriptors}
Image \textit{descriptors} describe features so that they can be compared across images, or used for object detection and matching. Similar to image detectors, it is also desirable for image descriptors to be repeatable (i.e. invariant with respect to pose, scale, illumination, etc.) and distinct. Perhaps the simplest example of a descriptor is an $n\times m$ window of pixel intensities centered at the feature, which can be normalized to be illumination invariant. However, such a descriptor is not invariant to pose or scale and is not distinctive, and therefore is generally not useful in practice.
Alternative detectors/descriptors that have become popular include SIFT, SURF, FAST, BRIEF, ORB, and BRISK.

\subsection{Exercises}
\subsubsection{Linear Filtering}
Complete \textit{Problem 3: Linear Filtering} located in the online repository:

\vspace{\baselineskip}

\url{https://github.com/PrinciplesofRobotAutonomy/AA274A_HW3},

\vspace{\baselineskip}

where you will explore the use of linear filters for image processing.


\section*{Ex.34.2-5}
\subsection*{Show that any language in NP can be decided by an algorithm running in time $2^{O(n^k)}$ for some constant $k$}

First note, that if we can verify a language, then we can also decide it.
\\
Now, we know that we can verify any language L $\in$ NP in polynomial time (top of page 1064, CLRS), with a two-input algorithm. As input, the algorithm takes $x\in L$ and a certificate $y$. Since the length of any certificate is bounded by $O(|x|^c)$, the input $x$ can at most be $O(|x|^c)$ long.
\\
If we run the algorithm on every possible input $x\in {0,1}^*$, with that max length, it will take time $2^{O(n^k)}$.


\documentclass[class=report, float=false, crop=false]{standalone}
\usepackage[subpreambles=true]{standalone}

\input{preamble}

\graphicspath{{figures/images/}}

% \begin{cbunit}

\begin{document}

\chapter{Ellipsoids}
\label{appendix:ellipsoids}

\section{Definition}

An ellipsoid is a surface that may be obtained from a sphere by deforming it by means of directional scalings, or more generally, of an affine transformation \cite{wiki:Ellipsoid}.\\

Within a Cartesian coordiante system in which the origin is the center of the ellipsoid and the coordinate axes are axes of the ellipsoid, a vector \(\vec{r} = \begin{pmatrix} x \\ y \\ z \end{pmatrix} \in \mathbb{R}^3\) belongs to the surface of the ellispoid if and only if
\begin{equation}
\frac{x^2}{a^2} + \frac{y^2}{b^2} + \frac{z^2}{c^2} = 1
\label{ellipsoid_cartesian}
\end{equation}
where $a$, $b$ and $c \in \mathbb{R}$ are the the semi-axes of the ellipsoid.\\

Equivalently to equation \ref{ellipsoid_cartesian}, an ellipsoid $\mathcal{A}$ can be defined by a positive definite matrix $A \in \mathcal{M}_3(\mathbb{R})$ and a vector $\vec{v} \in \mathbb{R}^3$ such that
\begin{equation}
\boxed{\forall \vec{r} \in \mathbb{R}^3, \vec{r} \in \bar{\mathcal{A}} \Leftrightarrow (\vec{r}-\vec{v})^TA(\vec{r}-\vec{v}) = 1}
\label{ellipsoid_matrix}
\end{equation}
The eigenvectors of $A$ then define the principal axes of $\mathcal{A}$ and the associated eigenvalues are the reciprocals of the squares of the semi-axes. The vector $\vec{v}$ defines the center of the ellipsoid.

\section{Homogeneous coordinates}

\subsection{Quick reminder}

A vector \(\vec{r} = \begin{pmatrix} x \\ y \\ z \end{pmatrix} \in \mathbb{R}^3\) is equivalent to its homogeneous form \(\begin{pmatrix} x \\ y \\ z \\ 1 \end{pmatrix} \in E^3\) where $E^3$ is the Euclidean 3D projective space.\\

Moreover, we have that in $E^3$, $\forall \lambda \in \mathbb{R}^*$ and $\forall (x,y,z) \in \mathbb{R}^3$, \(\begin{pmatrix} x \\ y \\ z \\ 1 \end{pmatrix}\) and \(\begin{pmatrix} \frac{x}{\lambda} \\ \frac{y}{\lambda} \\ \frac{z}{\lambda} \\ 1 \end{pmatrix}\) represent the exact same point. Therefore, we will always assume that the last coordinate of our vectors in $E^3$ is 1.\\

From the property above, you naturally have that any point in $E^3$ whose last coordinate would be $0$ is then infinitely far from the origin \(\begin{pmatrix} 0 \\ 0 \\ 0 \\ 1 \end{pmatrix}\).

\subsection{Useful affine transformations}

\subsubsection{Translation}

We define in $\mathbb{R}^3$ the translation of vector $\vec{u} \in \mathbb{R}^3$ as
\begin{align*}
\mathcal{T}_{\vec{u}} \colon &\mathbb{R}^3 \to \mathbb{R}^3\\     &\phantomarrow{\mathbb{R}^3}{\vec{v}} \vec{v} + \vec{u}
\end{align*}

In $E^3$, this function $\mathcal{T}_{\vec{u}}$ is represented by the matrix $T_{\vec{u}}$:
\begin{equation}
\boxed{T_{\vec{u}} = \begin{pmatrix} \mathbbm{1}_3 & \vec{u} \\ \vec{0}^T & 1 \end{pmatrix}}
\label{translation_matrix}
\end{equation}

\subsubsection{Dilatation}

We define in $\mathbb{R}^3$ the dilatation of factor $a \in \mathbb{R}$ as
\begin{align*}
\mathcal{X}_{a} \colon &\mathbb{R}^3 \to \mathbb{R}^3\\     &\phantomarrow{\mathbb{R}^3}{\vec{v}} a\vec{v}
\end{align*}

In $E^3$, this function $\mathcal{X}_{a}$ is represented by the matrix $X_{a}$:
\begin{equation}
\boxed{X_{a} = \begin{pmatrix} a\mathbbm{1}_3 & \vec{0} \\ \vec{0}^T & 1 \end{pmatrix}}
\label{dilatation_matrix}
\end{equation}

\subsubsection{Rotation}

We showed in part \ref{action_rotation} that every 3D rotation can be expressed as the action of some unit quaternion.\\

The product of quaternions being bilinear, we can associate the action of a quaternion to a linear function in a vectorial space and therefore to a matrix \cite{shoemake}.\\

Consider $q = [\vec{q},q_0],p = [\vec{p},p_0] \in \mathbb{H}$. According to part \ref{quat_properties}, we have $qp = [\underbrace{\vec{q}\times\vec{p}}_{Ap} + \underbrace{q_0\vec{p}}_{Bp} + \underbrace{p_0\vec{q}}_{Cp},\underbrace{q_0p_0}_{Dp}~\underbrace{-\vec{q}\cdot\vec{p}}_{Ep}]$, with
\begin{align*}
A = \begin{pmatrix} \vec{q} \times \bigcdot & \vec{0} \\ \vec{0}^T & 0 \end{pmatrix}, B = \begin{pmatrix} q_0 \mathbbm{1}_3 & \vec{0} \\ \vec{0}^T & 0 \end{pmatrix}, C = \begin{pmatrix} 0_3 & \vec{q} \\ \vec{0}^T & 0 \end{pmatrix}, D = \begin{pmatrix} 0_3 & \vec{0} \\ \vec{0}^T & q_0 \end{pmatrix}, E = \begin{pmatrix} 0_3 & \vec{0} \\ -\vec{q}^T & 0 \end{pmatrix}
\end{align*}
Therefore the left multiplication by $q$ in $\mathbb{H}$ can be represented by the matrix
\begin{align*}
L_q &= A + B + C + D + E\\
&= \begin{pmatrix} \vec{q} \times \bigcdot + q_0\mathbbm{1}_3 & \vec{q} \\ -\vec{q}^T & q_0 \end{pmatrix}
\end{align*}

Similarly, we have $pq^* = [\underbrace{\vec{q}\times\vec{p}}_{Ap} + \underbrace{q_0\vec{p}}_{Bp}~\underbrace{-p_0\vec{q}}_{-Cp},\underbrace{q_0p_0}_{Dp} + \underbrace{\vec{q}\cdot\vec{p}}_{-Ep}]$, therefore the right multiplication by $q^*$ can be represented by the matrix
\begin{align*}
R_{q^*} &= A + B - C + D - E\\
&= \begin{pmatrix} \vec{q} \times \bigcdot + q_0\mathbbm{1}_3 & -\vec{q} \\ \vec{q}^T & q_0 \end{pmatrix}
\end{align*}

Consequently, the matrix representing the action of the unit quaternion $q$ in $\mathbb{H}$ is
\begin{align*}
Q_q = L_qR_{q^*} = R_{q^*}L_q = \begin{pmatrix} \mathcal{Q}_q & \vec{0} \\ \vec{0}^T & \underbrace{N^2(q)}_{=1} \end{pmatrix}
\end{align*}
where
\begin{equation}
\boxed{\mathcal{Q}_q = (\vec{q}\times\bigcdot + q_0\mathbbm{1}_3)^2 + \vec{q}\vec{q}^T}
\label{rotation_matrix_R3}
\end{equation}
is the matrix representing the rotation associated to the unit quaternion $q$ in $\mathbb{R}^3$.\\

Therefore, the matrix reprensenting the action of $q$ in $\mathbb{H}$ and the rotation associated to $q$ in $E^3$ are the same and we will note the latter
\begin{equation}
\boxed{Q_q = \begin{pmatrix} \mathcal{Q}_q & \vec{0} \\ \vec{0}^T & 1 \end{pmatrix}}
\label{rotation_matrix}\\
\end{equation}

We can develop equation \ref{rotation_matrix_R3} to get an expression of the rotation matrix $\mathcal{Q}_q$ in $E^3$. With
\begin{align*}
  \left(\vec{e}_1 \equiv (1, 0, 0)^T, \vec{e}_2 \equiv (0, 1, 0)^T, \vec{e}_3 \equiv (0, 0, 1)^T\right)
\end{align*}
the canonical basis of $E^3$, we have
\begin{align*}
  \vec{q} \times \vec{e}_1 = \begin{pmatrix} 0 \\ q_3 \\ -q_2 \end{pmatrix}, \vec{q} \times \vec{e}_2 = \begin{pmatrix} -q_3 \\ 0 \\ q_1 \end{pmatrix}, \vec{q} \times \vec{e}_3 = \begin{pmatrix} q_2 \\ -q_1 \\ 0 \end{pmatrix}
\end{align*}
such that
\begin{align*}
  \vec{q} \times \bigcdot = \begin{pmatrix} 0 & -q_3 & q_2 \\ q_3 & 0 & -q_1 \\ -q_2 & q_1 & 0 \end{pmatrix} \text{ and } \vec{q} \times \bigcdot + q_0 \mathbbm{1}_3 = \begin{pmatrix} q_0 & -q_3 & q_2 \\ q_3 & q_0 & -q_1 \\ -q_2 & q_1 & q_0 \end{pmatrix}
\end{align*}
and then
\begin{align*}
  (\vec{q} \times \bigcdot + q_0 \mathbbm{1}_3)^2 =
  \begin{pmatrix}
    q_0^2 - q_3^2 - q_2^2 & q_1 q_2 - 2 q_0 q_3 & q_1 q_3 + 2 q_0 q_2 \\
    q_1 q_2 + 2 q_0 q_3 & q_0^2 - q_3^2 - q_1^2 & q_2 q_3 - 2 q_0 q_1 \\
    q_1 q_3 - 2 q_0 q_2 & q_2 q_3 + 2 q_0 q_1 & q_0^2 - q_1^2 - q_2^2
  \end{pmatrix}
\end{align*}
thus leading to
\begin{align*}
  \mathcal{Q}_q = (\vec{q} \times \bigcdot + q_0 \mathbbm{1}_3)^2 + \vec{q} \vec{q}^T =
  \begin{pmatrix}
    q_0^2 + q_1^2 - q_3^2 - q_2^2 & 2 q_1 q_2 - 2 q_0 q_3 & 2 q_1 q_3 + 2 q_0 q_2 \\
    2 q_1 q_2 + 2 q_0 q_3 & q_0^2 + q_2^2 - q_3^2 - q_1^2 & 2 q_2 q_3 - 2 q_0 q_1 \\
    2 q_1 q_3 - 2 q_0 q_2 & 2 q_2 q_3 + 2 q_0 q_1 & q_0^2 + q_3^2 - q_1^2 - q_2^2
  \end{pmatrix}
\end{align*}
where we can note that $q$ is an unit quaternion, \textit{i.e.} $N^2(q) = \sum_{i=0}^4 q_i^2 = 1$, and therefore
\begin{equation}
  \boxed{\mathcal{Q}_q =
    \begin{pmatrix}
      1 - 2 (q_2^2 + q_3^2) & 2 (q_1 q_2 - q_0 q_3) & 2 (q_1 q_3 + q_0 q_2) \\
      2 (q_1 q_2 + q_0 q_3) & 1 - 2 (q_1^2 + q_3^2) & 2 (q_2 q_3 - q_0 q_1) \\
      2 (q_1 q_3 - q_0 q_2) & 2 (q_2 q_3 + q_0 q_1) & 1 - 2 (q_1^2 + q_2^2)
    \end{pmatrix}
  }
\label{rotation_matrix_R3_expression}
\end{equation}
in accordance with \cite{wiki:quaternions}.

\section{Belonging matrix}

\subsection{Definition}

We want to find for any ellipsoid $\mathcal{A}$ a matrix $B$ acting on homogeneous coordinates with the following properties
\begin{equation}
\forall \vec{r} \in E^3, \begin{cases} \vec{r}^T B \vec{r} < 0 &\text{ if } \vec{r} \in \mathcal{A} \setminus \bar{\mathcal{A}} \\ \vec{r}^T B \vec{r} = 0 &\text{ if } \vec{r} \in \bar{\mathcal{A}} \\ \vec{r}^T B \vec{r} > 0 &\text{ if } \vec{r} \notin \mathcal{A} \end{cases}
\label{belonging_definition}
\end{equation}
which we will call the \textit{belonging matrix} of $\mathcal{A}$.

\subsection{Expression}
\label{belonging_exp}

We have seen with equation \ref{ellipsoid_matrix} that the belonging to the surface of an ellispoid could be expressed with a matrix whose eigenvectors define the principal axes of the ellipsoid and whose eigenvalues define the reciprocals of the squares of the semi-axes. Therefore, if we denote $(R_i)_{i=1:3}$ the semi-axes of the ellipsoid, this matrix can be written as $A = P^{-1}\text{diag}(R_1^{-2},R_2^{-2},R_3^{-2})P$.\\

The semi-axes of an ellipsoid define an orthogonal base of $\mathbb{R}^3$. Therefore P is the transition matrix from the orthogonal base formed by the principal axes of the ellipsoid to the original Euclidean base $(\vec{e_i})_{i=1:3}$, and inversely for $P^{-1}$.\\

Consequently, we have that $\forall i \in \llbracket1,3\rrbracket, \vec{v} + R_iP^{-1}\vec{e_i} \in \bar{\mathcal{A}}$ and so
\begin{align*}
\forall i \in \llbracket1,3\rrbracket,&(P^{-1}R_i\vec{e_i})^TA(P^{-1}R_i\vec{e_i}) = 1\\ \Leftrightarrow&(P^{-1}R_i\vec{e_i})^TP^{-1}\text{diag}(R_j^{-2})_{j=1:3}P(P^{-1}R_i\vec{e_i}) = 1\\
\Leftrightarrow &R_i\vec{e_i}^T(P^{-1})^TP^{-1}\text{diag}(R_j^{-2})_{j=1:3}\underbrace{PP^{-1}}_{\mathbbm{1}_3}R_i\vec{e_i} = 1\\
\Leftrightarrow &\vec{e_i}^T(P^{-1})^TP^{-1}\underbrace{\text{diag}(R_j^{-2})_{j=1:3}R_i^2\vec{e_i}}_{\vec{e_i}} = 1\\
\Leftrightarrow &\vec{e_i}^T(P^{-1})^TP^{-1}\vec{e_i} = 1
\end{align*}
Thus, the norm of every column vector of $P^{-1}$ equals to 1. Therefore, $P^{-1}$, and equivalently $P$, is an orthonormal matrix.\\

If $P$ is an orthonormal matrix, then the principal axes of the ellipsoids are derived from the original Euclidean base through rotations and permutations of axis. Without loss of generality, we can consider that there are no permutations of axis, leading to $P = \mathcal{Q}_q^{-1} = \mathcal{Q}_q^T$ with $q$ the quaternion describing the orientation of the ellipsoid. Equation \ref{ellipsoid_matrix} thus becomes:
\begin{align*}
\forall \vec{r} \in \mathbb{R}^3, \vec{r} \in \bar{\mathcal{A}} \Leftrightarrow (\vec{r}-\vec{v})^T\mathcal{Q}_q\text{diag}(R_i^{-2})_{i=1:3}\mathcal{Q}_q^T(\vec{r}-\vec{v}) = 1
\end{align*}

We can get rid of the $\vec{v}$ by using homogeneous coordinates. In $E^3$, $\vec{r} - \vec{v}$ becomes $T_{-\vec{v}}\vec{r}$ and, to conserve the scalar product, $\text{diag}(R_i^{-2})_{i=1:3}$ becomes $\text{diag}(R_i^{-2},0)_{i=1:3}$.\\

We always assume that the last coordinate of our vectors in $E^3$ is 1, therefore we can notice that $\forall \vec{u} \in E^3, \vec{u}^T\text{diag}(0,0,0,-1)\vec{u} = -1$. We can then rewrite equation \ref{ellipsoid_matrix}:
\begin{align*}
\forall \vec{r} \in E^3, \vec{r} \in \bar{\mathcal{A}} \Leftrightarrow &\vec{r}^TT_{-\vec{v}}^TQ_q\text{diag}(R_i^{-2},0)_{i=1:3}Q_q^TT_{-\vec{v}}\vec{r} = 1\\
\Leftrightarrow&\vec{r}^T\underbrace{T_{-\vec{v}}^TQ_q\text{diag}(R_i^{-2},-1)_{i=1:3}Q_q^TT_{-\vec{v}}}_{\mathcal{C}}\vec{r} = 0
\end{align*}
The matrix $\mathcal{C}$ is a good candidate for the belonging matrix of $\mathcal{A}$, we then have to understand how it works.\\

The $T_{-\vec{v}}$ translation and the $Q_q^T$ rotation bring back the principal axes of the ellipsoid to the original Euclidean base and origin. Without loss of generality, we can then consider that the principal axes of the ellipsoids are along $(\vec{e_1},\vec{e_2},\vec{e_3})$.\\

For a vector \(\vec{r} = \begin{pmatrix} x \\ y \\ z \end{pmatrix} \in \mathbb{R}^3\), we have
\begin{align*}
&\frac{x^2}{R_1^2} + \frac{y^2}{R_2^2} + \frac{z^2}{R_3^2} = \mu^2\\
\Leftrightarrow & \frac{x^2}{(\mu R_1)^2} + \frac{y^2}{(\mu R_2)^2} + \frac{z^2}{(\mu R_3)^2} = 1
\end{align*}
therefore, $\mu$ can be construed as the rescaling factor that has to be applied to the ellipsoid for $\vec{r}$ to be on its surface. Then, $\mu < 1$ if $\vec{r} \in \mathcal{A}\setminus\bar{\mathcal{A}}$, $\mu = 1$ if $\vec{r} \in \bar{\mathcal{A}}$, and $\mu > 1$ otherwise.\\

Since we have that $\forall \vec{r} \in E^3, \vec{r}^T\mathcal{C}\vec{r} = \mu^2 - 1$, $\mathcal{C}$ is the matrix we have been looking for, and we can define
\begin{equation}
\boxed{B(\vec{v},q,(R_i)_{i=1:3}) \equiv T_{-\vec{v}}^TQ_q\text{diag}(R_i^{-2},-1)_{i=1:3}Q_q^TT_{-\vec{v}}}
\label{belonging_matrix}
\end{equation}

One must notice that such a matrix is symmetric. We can also define the \textit{belonging function} of the ellipsoid $\mathcal{A}$
\begin{equation}
\boxed{\begin{aligned}\mathcal{F}_{\mathcal{A}} \colon &E^3 \to \mathbb{R}\\     &\phantomarrow{E^3}{\vec{r}} \vec{r}^TB(\vec{v},q,(R_i)_{i=1:3})\vec{r}\end{aligned}}
\label{belonging_function}
\end{equation}
such that $\forall \vec{r} \in E^3$, $\vec{r} \in \mathcal{A} \Leftrightarrow \mathcal{F}_{\mathcal{A}}(\vec{r}) \leq 0$ and
\begin{equation}
\boxed{\mathcal{F}_{\mathcal{A}}(\vec{r}) = \mu^2(\vec{r}) - 1}
\end{equation}
with $\mu^2(\vec{r})$ the rescaling factor that has to be applied to the ellipsoid for $\vec{r}$ to be on its surface.

\subsection{Reduced belonging matrix}

The equation \ref{belonging_matrix} is theoretically relevant, however it is not computationally efficient.\\

On the one side, resorting to homogeneous coordinates and expressing the translation with a translation matrix (equation \ref{translation_matrix}) leads to more operations than to express the rotation in $\mathbb{R}^3$ directly. On the other side, we have seen in part \ref{quaternions_interest} that the rotation of a vector was quicker performed when converting the quaternion associated to the rotation to a rotation matrix.\\

We then introduce, for an ellipsoid $\mathcal{A}$ whose centre is located in $\vec{v}$, whose orientation is described by the unit quaternion $q$ and whose semi-axes are $(R_i)_{i=1:3}$, the associate \textit{reduced belonging matrix}
\begin{equation}
\boxed{\bar{B}(q,(R_i)_{i=1:3}) \equiv \mathcal{Q}_q\text{diag}(R_i^{-2})_{i=1:3}\mathcal{Q}_q^T}
\label{reduced_belonging_matrix}
\end{equation}
with $\mathcal{Q}_q$ the $\mathbb{R}^3$ rotation matrix associated to $q$ (equation \ref{rotation_matrix_R3}). One must notice that the reduced belonging matrix is also symmetric. Furthermore, this reduced belonging matrix has the following properties
\begin{equation}
\forall \vec{r} \in \mathbb{R}^3, \begin{cases} (\vec{r} - \vec{v})^T \bar{B} (\vec{r} - \vec{v}) < 1 &\text{ if } \vec{r} \in \mathcal{A} \setminus \bar{\mathcal{A}} \\ (\vec{r} - \vec{v})^T \bar{B} (\vec{r} - \vec{v}) = 1 &\text{ if } \vec{r} \in \bar{\mathcal{A}} \\ (\vec{r} - \vec{v})^T \bar{B} (\vec{r} - \vec{v}) > 1 &\text{ if } \vec{r} \notin \mathcal{A} \end{cases}
\label{reduced_belonging_definition}
\end{equation}
according to equation \ref{belonging_definition}, which leads to the following expression for the belonging function:
\begin{equation}
\boxed{\begin{aligned}\mathcal{F}_{\mathcal{A}} \colon &\mathbb{R}^3 \to \mathbb{R}\\     &\phantomarrow{\mathbb{R}^3}{\vec{r}} (\vec{r} - \vec{v})^T\bar{B}(q,(R_i)_{i=1:3})(\vec{r}-\vec{v}) - 1\end{aligned}}
\label{belonging_function_reduced}
\end{equation}

\section{Unit normal vectors}

Since the surface of the ellipsoid -- the locus of points where $\mathcal{F}_{\mathcal{A}}(\vec{r}) = 0$ -- is an isosurface of the function $\mathcal{F}_{\mathcal{A}}$, we have that $\forall \vec{r_S} \in \bar{\mathcal{A}}, \vec{\nabla}\mathcal{F}_{\mathcal{A}}(\vec{r_S})$ is a non-unit outward-facing normal vector to the ellipsoid in $\vec{r_S}$.\\

Let us note $\forall \vec{r_S} \in \bar{\mathcal{A}}$, $\vec{n}(\vec{r_S})$ the unit outward-facing normal vector to the ellipsoid in $\vec{r_S}$. Then $\vec{n}(\vec{r_S}) // \vec{\nabla}\mathcal{F}_{\mathcal{A}}(\vec{r_S})$, we thus have to find the direction of this gradient.

\subsection{Expression with the belonging matrix}
\label{normal_vector_belonging}

We have that
\begin{align*}
\forall \vec{r} \in E^3, \mathcal{F}_{\mathcal{A}}(\vec{r} + \vec{dr}) &= (\vec{r} + \vec{dr})^TB(\vec{r} + \vec{dr})\\
&= \underbrace{\vec{r}^TB\vec{r}}_{\mathcal{F}_{\mathcal{A}}(\vec{r})} + \underbrace{\vec{r}^TB\vec{dr} + \vec{dr}^TB\vec{r}}_{2\vec{dr}^TB\vec{r}} + \vec{dr}^TB\vec{dr}
\end{align*}
therefore we can show that
\begin{align*}
\forall \vec{r} \in E^3, \forall i \in \llbracket1,3\rrbracket,~ &\mathcal{F}_{\mathcal{A}}(\vec{r} + dr_i\vec{e_i}) - \mathcal{F}_{\mathcal{A}}(\vec{r}) = 2dr_i\vec{e_i}^TB\vec{r} + dr_i^2\vec{e_i}B\vec{e_i}\\
\Rightarrow&\frac{\partial}{\partial e_i}\mathcal{F}_{\mathcal{A}}(\vec{r}) = \lim_{dr_i \to 0} \frac{\mathcal{F}_{\mathcal{A}}(\vec{r} + dr_i\vec{e_i}) - \mathcal{F}_{\mathcal{A}}(\vec{r})}{dr_i} = 2\vec{e_i}^TB\vec{r}\\
\Rightarrow&\vec{\nabla}\mathcal{F}_{\mathcal{A}}(\vec{r}) = \sum_{i=1}^3\left(\frac{\partial}{\partial e_i}\mathcal{F}_{\mathcal{A}}(\vec{r})\right)\vec{e_i} = 2\underbrace{\sum_{i=1}^3(\vec{e_i}^TB\vec{r})\vec{e_i}}_{B\vec{r}}
\end{align*}
\textit{i.e.} $\vec{\nabla}\mathcal{F}_{\mathcal{A}}(\vec{r}) = 2B\vec{r}$. Consequently, we finally have that
\begin{equation}
\boxed{\forall \vec{r_S}\in \bar{\mathcal{A}}, \vec{n}(\vec{r_S}) = \frac{B(\vec{v},q,(R_i)_{i=1:3})\vec{r_S}}{|B(\vec{v},q,(R_i)_{i=1:3})\vec{r_S}|}}
\end{equation}

\subsection{Expression with the reduced belonging matrix}

With the same notations while assuming that the centre of the ellipsoid is located in $\vec{v}$, we can notice that replacing $\vec{r}$ by $(\vec{r} - \vec{v})$ in the demonstration of part \ref{normal_vector_belonging} leads to the following expression
\begin{align*}
\forall \vec{r} \in E^3, \forall i \in \llbracket1,3\rrbracket,~ &\mathcal{F}_{\mathcal{A}}(\vec{r} + dr_i\vec{e_i}) - \mathcal{F}_{\mathcal{A}}(\vec{r}) = 2dr_i\vec{e_i}^T\bar{B}(\vec{r} - \vec{v}) + dr_i^2\vec{e_i}\bar{B}\vec{e_i}
\end{align*}
and thus, by analogy, the following result
\begin{equation}
\boxed{\forall \vec{r_S}\in \bar{\mathcal{A}}, \vec{n}(\vec{r_S}) = \frac{\bar{B}(q,(R_i)_{i=1:3})(\vec{r_S} - \vec{v})}{|\bar{B}(q,(R_i)_{i=1:3})(\vec{r_S} - \vec{v})|}}
\label{surface_vec_reduced}
\end{equation}

% \input{references/biblio}

\end{document}

% \end{cbunit}


%%%% 3D Skewness Example %%%%%%

\subsection{Dependence on Dimension}\label{ex:3dmap}
To further illustrate that relationship between skewness and accuracy holds as we move towards higher dimensions, we extend the numerical investigation to a three--dimensional parameter space.
Generally, we have fewer QoI than number of uncertain model parameters, so we assume that the potential QoI maps are defined by the $2\times 3$ matrices
\begin{equation}\label{eq:qmap3}
\qspace_S := \left \lbrace \qoi^{(s)} =  \mat{ccc}{1 & 0 & 0\\ \sqrt{s^2 - 1}& 1 & 0} \right \rbrace_{s\in S}.
\end{equation}
Here, as in the previous example, the index $s$ indicates the magnitude of skewness.
Furthermore, the results of Example~\ref{ex:rotation} justify the restriction of the maps to this form since any linear map of skewness $s$ is simply a rotation of maps of this form.

%Now, the generalized contours for inverses of maps from $\RR^3 \to \RR^2$ will be isomorphic to 2\--dimensional contour events in that the inverse sets will be columns in 3\-space orthogonal to the aforemntioned plane.
%
%IMAGE DEMONSTRATING THIS WOULD HELP.
%We define
%
%which is just the map from \eqref{eq:qmap2} appended with zeros in the third column.
%We make this choice solely for convience and are justified in doing so owing to Proposition~\ref{prop:rot_invariance} and the fact of generalized contours of maps from $\RR^3 \to \RR^2$ being parallel columns.
%The rotational invariance naturally extends to the third dimension.

%We note that $\bar{N}$ is much higher since we kept the convention of 200 grid cells per dimension in our reference.
%However, we kept the same number of random samples $N$, so we should expect higher errors due to the overresolved regular grid.
%Fortunately, we find that the results still generalize.
%We present the case where $M=1$:

\begin{figure}[h]
\begin{table}[H]
\begin{tabular}{ c | c | c | c }
\nsamps & $\qoiA$ & $\qoiB$ & $\qoiC$\\ \hline \hline
$200$ & $3.33E-01$ & $4.56E-01$ & $6.10E-01$\\ \hline

$400$ & $2.78E-01$ & $3.51E-01$ & $4.97E-01$\\ \hline

$800$ & $2.19E-01$ & $2.95E-01$ & $4.10E-01$\\ \hline

$1600$ & $1.72E-01$ & $2.37E-01$ & $3.35E-01$\\ \hline

$3200$ & $1.36E-01$ & $1.89E-01$ & $2.64E-01$\\ \hline

$6400$ & $1.09E-01$ & $1.47E-01$ & $2.09E-01$\\ \hline
\end{tabular}
\end{table}

\includegraphics[width=0.45\linewidth]{./images/Plot-reg_BigN_8000000_reg_M_1_rand_I_100000.png}

\caption{The results of $d^2_\text{TV}(\PP_{\pspace, \ndiscs, \nsamps}, \PP_{\pspace, \ndiscs, \bar{\nsamps}})$ for $\ndiscs = 1, \bar{\nsamps} = 8,000,000$, with $a, b, c = 1, 2, 4$ in three dimensions.}
\label{fig:M1_3d}
\end{figure}
\FloatBarrier
In Figure~\ref{fig:M1_3d}, it appears that the effect of skewness is even more pronounced in higher dimensions, and that the number of samples required to achieve similar levels of accuracy between two maps with a ratio of skewness 2 is now quadrupled.
The analysis of \cite{BGE+15} suggested a dependence of accuracy related to the skewness raised to a power related to the dimension of the data space.


\addcontentsline{toc}{section}{Exercise 2.1 Bayes rule for medical diagnosis}
\section*{Exercise 2.1 Bayes rule for medical diagnosis}
\import{sections/}{exercise 2-1.tex}
\newpage
\addcontentsline{toc}{section}{Exercise 2.2 Legal reasoning}
\section*{Exercise 2.2 Legal reasoning}
\import{sections/}{exercise 2-2.tex}
\newpage
\addcontentsline{toc}{section}{Exercise 2.3 Probabilities are sensitive to the form of the question that was used to generate the answer}
\section*{Exercise 2.3 Probabilities are sensitive to the form of the question that was used to generate the answer}
\import{sections/}{exercise 2-3.tex}
\newpage
\addcontentsline{toc}{section}{Exercise 2.4 Deriving the posterior predictive density for the healthy levels game}
\section*{Exercise 2.4 Deriving the posterior predictive density for the healthy levels game}
\import{sections/}{exercise 2-4.tex}


\section{Finding Periodic Solutions}
\label{sec:initial}

A separate problem from evaluating, optimizing and continuing periodic solutions is finding an initial candidate solution.
Because continuation is a crucial part of this project, having just one single periodic solution might enable to find many others, through tracing and switching solution branches.
Because the systems considered in this work have stable periodic solutions, we focus on this case.
When this is not the case, as mentioned in the section about continuation methods (\autoref{sec:cont}), a homotopy between a trivial system and the target system, combined with continuation methods, might be a promising approach.

Starting from a point in the periodic solution's basin of attraction, one can simply forward integrate such a system.
There are several possible problems involved:
\begin{itemize}
	\item Forward integration can accumulate errors.
	\item Even if the starting point lies exactly on the periodic trajectory, the sampling interval would probably not be an integer fraction of the period of the periodic trajectory.
		Periodicity in the sequence of points are not directly related to periodicity in the continuous system.
	\item Given the nature of the project, it is very likely that period doubling bifurcations are encountered.
		These provoke situations where two periodic solutions exist which are difficult to distinguish.
\end{itemize}

Forward integration yields a sequence of points in phase space $(\textbf{s}_i)_{i \in \N}$.
To obtain more manageable data only intersections of the trajectory with a hyperplane $p(\textbf{x}) = \langle \textbf{n}, \textbf{x}-\textbf{x}_0 \rangle = 0$ in a single direction are considered (so called Poincaré sections).
To find these, $\textbf{s}_i, \textbf{s}_{i+1} = \textbf{s}_i + h_0 \cdot \textbf{f}(\textbf{s}_i)$ with $p(s_i) \le 0 < p(s_{i+1})$ are searched.
Because of continuity, there needs to exists $h \in [0,h_0]$ such that $p(\textbf{s}_i + h \cdot \textbf{f}(\textbf{s}_i)) = 0$, which is found via bisection.
Let $(\textbf{u}_i)_{i \in \N}$ be the sequence of intersections.

To find the number of intersections per period, the intersections are partitioned into $k \in \N$ disjoint clusters $V_{k,i} = \{ \textbf{u}_m\ |\ m \in k\N+i \}$ for $i \in \N$, $i \le k$.
The relative quality of the $i$-th cluster can then be assessed using the within-cluster variance $\sum_{v \in V_{k,i} } ||v - \E(V_{k,i})||^2$.
The correct number $k_{\text{min}} \in \N$ of intersections in one period is then taken to be the one minimizing the sum of within-cluster variances:
\[
	k_\text{min} \coloneqq \argmin_{k \in \N} \sum_{i = 1}^k \sum_{v \in V_{k,i} } ||v - \E(V_{k,i})||^2 \text.
\]
For further information about these measures see for example \cite{halkidi2001clustering}. %TODO measure...

In this form the criterion might at best work if the intersection sequence is infinite.
When dealing with finite sequences increasing the number of clusters inevitably leads to lower total within-cluster variances.
It is thus necessary to constrain the available values for $k$ and discourage the method of overestimating the number of clusters.
Trivially an upper limit for $k$ needs to be introduced.
Furthermore, the minimality criterion needs to be relaxed: Suppose there are $k_\text{min}$ intersections per period, then all multiples of $k_\text{min}$ yield equal or lower ratings.
One thus wants to choose the minimum $k$ which is in some sense still almost optimal.


\section{SCC}

%%%%%%%%%%%%%%%%%%%%
\begin{frame}{Digraph as DAG}
  \begin{exampleblock}{Digraph as DAG (Problem 5.3)}
    Every digraph is a dag of its SCCs.
  \end{exampleblock}

  \begin{alertblock}{Remark}
    Two tiered structure of digraphs:
    \begin{itemize}
	  \item digraph $\equiv$ a dag of SCCs
      \item SCC: equivalence class over reachability
    \end{itemize}
  \end{alertblock}
\end{frame}
%%%%%%%%%%%%%%%%%%%%
\begin{frame}{SCC}
  \begin{exampleblock}{Kosaraju SCC algorithm, 1978}
	\begin{quote}
	  ``SCCs can be topo-sorted in decreasing order of their highest finish time.''
	\end{quote}

	\pause
	\centerline{The vertice with the highest finish time is in a source SCC.}
  \end{exampleblock}

  \pause
  \begin{alertblock}{Remark}
    \begin{itemize}
      \item DFS on $G$; \; DFS/BFS on $G^{T}$
      \item DFS on $G^{T}$; DFS/BFS on $G$
    \end{itemize}
  \end{alertblock}
\end{frame}
%%%%%%%%%%%%%%%%%%%%
\begin{frame}{SCC}
  \begin{exampleblock}{Kosaraju SCC algorithm, 1978 (Problem 5.4)}
    \begin{itemize}
      \item 1st DFS $\xLongrightarrow{?}$ BFS
      \item 2nd DFS $\xLongrightarrow{?}$ BFS
    \end{itemize}
  \end{exampleblock}
\end{frame}
%%%%%%%%%%%%%%%%%%%%
\begin{frame}{One-to-all reachability}
  \begin{exampleblock}{One-to-all reachability (Problem 5.12)}
	Digraph $G = (V, E)$:
    \begin{itemize}
      \item given $v: v \leadsto^{?} \forall u$
      \item $\exists?\; v: v \leadsto \forall u$
    \end{itemize}
  \end{exampleblock}

  \pause
  \[
	\text{SCC; } \exists! \text{source vertex } v \iff v \leadsto \forall u
  \]

  \pause
  \begin{proof}
	\begin{itemize}
	  \item $\Longleftarrow$: (1) source (2) $\exists !$
		\pause
	  \item $\implies$: By contradiction.
		% \pause
		% \[
		%   \exists u: v \not\leadsto u \land \text{in}[u] > 0 \implies \exists u' \to u \land v \nrightarrow u' \implies \exists \text{ cycle}
		% \]
	\end{itemize}
  \end{proof}
\end{frame}
%%%%%%%%%%%%%%%%%%%%
\begin{frame}{Impacts of vertices}
  \begin{exampleblock}{Impacts of vertices (Problem 5.13)}
	Digraph $G$:
	\[
	  \text{impact}(v) = |\set{w: v \leadsto w}|
	\]

	\begin{itemize}
	  \item $\argmin_{v} \text{impact}(v)$
	  \item $\argmax_{v} \text{impact}(v)$
	\end{itemize}
  \end{exampleblock}

  \pause
  \vspace{0.50cm}
  \centerline{$\argmin_{v} \text{impact}(v) \in \text{SCC of smallest cardinality}$}

  \pause
  \begin{alertblock}{Question}
	$\forall v:$ computing $\text{impact}(v)$.
  \end{alertblock}
\end{frame}
%%%%%%%%%%%%%%%%%%%%
\begin{frame}{One-way streets}
  \begin{exampleblock}{One-way streets (Problem 5.15)}
	Digraph $G$ for city:
    \begin{enumerate}
      \item $\forall u,v: u \leftrightsquigarrow v$
      \item $s: s \leadsto v \leadsto s$
    \end{enumerate}
  \end{exampleblock}

  \pause
  \[
	(2)\; \set{v \mid s \leadsto v} \text{ is an SCC}
  \]
\end{frame}
%%%%%%%%%%%%%%%%%%%%
\begin{frame}{Connectivity}
  \begin{exampleblock}{Connectivity (Problem 5.7)}
	\begin{description}[Example:]
	  \item[Prove:] connected undirected graph $G$:
		\[
		  \exists v: G \setminus v \text{ is still connected}
		\]
	  \item[Example:] strongly connected digraph $G$:
		\[
		  \exists v: G \setminus v \text{ is not strongly connected}
		\]
	  \item[Example:] digraph $G$ with 2 SCCs:
		\[
		  (G + e) \text{ is not strongly connected}
		\]
	\end{description}
  \end{exampleblock}
\end{frame}
%%%%%%%%%%%%%%%%%%%%
\begin{frame}{2SAT}
  \begin{exampleblock}{2SAT (Problem 5.17)}
	\[
	  I: (x_1 \lor \overline{x_2}) \land (\overline{x_1} \lor \overline{x_3}) \land (x_1 \lor x_2) \land (\overline{x_3} \lor x_4) \land (\overline{x_1} \lor x_4)
	\]
  \end{exampleblock}

  \pause
  \[
	\alpha \lor \beta \equiv \overline{\alpha} \to \beta \equiv \overline{\beta} \to \alpha
  \]

  \pause
  \begin{center}
	Implication graph $G_I$.
  \end{center}

  \pause
  \begin{theorem}
	\[
	  \exists \text{ SCC } \exists x: v_x \in \text{SCC} \land v_{\overline{x}} \in \text{SCC} \iff I \text{ is not satisfiable}.
	\]
  \end{theorem}

  \pause
  \begin{alertblock}{Reference}
	\begin{itemize}
	  \pause
	  \item ``A Linear-time Algorithm for Testing the Truth of Certain Quantified Boolean Formulas'' by Bengt Aspvall, Michael Plass, and Robert Tarjan, 1979.
	\end{itemize}
  \end{alertblock}
\end{frame}
%%%%%%%%%%%%%%%%%%%%
\begin{frame}{2SAT}
  % \begin{gather*}
  %   (x_0\lor x_2)\land(x_0\lor\lnot x_3)\land(x_1\lor\lnot x_3)\land(x_1\lor\lnot x_4)\land\\
  %   (x_2\lor\lnot x_4)\land (x_0\lor\lnot x_5)\land (x_1\lor\lnot x_5)\land (x_2\lor\lnot x_5)\land\\
  %   (x_3\lor x_6)\land (x_4\lor x_6)\land (x_5\lor x_6)
  % \end{gather*}
  \begin{columns}
	\column{0.50\textwidth}
	  % \fignocaption{width = 0.80\textwidth}{figs/2sat-implication-graph-wiki.png}
	  \fignocaption{width = 0.80\textwidth}{figs/2sat-implication-graph.pdf}
	\column{0.50\textwidth}
	  \pause
	  \fignocaption{width = 0.80\textwidth}{figs/2sat-implication-graph.png}
  \end{columns}
\end{frame}
%%%%%%%%%%%%%%%%%%%%
% \begin{frame}{Odd cycle in digraph}
%   \begin{exampleblock}{Odd cycle in digraph (Additional Problem)}
%     Find an odd cycle in a digraph $G$.
%   \end{exampleblock}
%
%   \pause
%   \begin{lemma}
%     A digraph $G$ has an odd directed cycle $\iff$ $\exists \text{scc}: $ scc is non-bipartite (when treated undirected).
%   \end{lemma}
%
%   \pause
%   \begin{alertblock}{Question}
% 	To prove the lemma and design an algorithm.
%   \end{alertblock}
%   % \begin{proof}
%   %   $\Longleftarrow$: undirected $C$; oriented
%   %     \begin{itemize}
%   %   	\item odd directed cycle
%   %   	\item choose a direction $\forall u \to v$: $\text{Len}(v \leadsto u)$
%   %     \end{itemize}
%   % \end{proof}
% \end{frame}
%%%%%%%%%%%%%%%%%%%%


\section{Determinants of \texorpdfstring{$2\times 2$}{2x2}- and \texorpdfstring{$3\times 3$}{3x3}-matrices}

\begin{outcome}
  \begin{enumerate}
    \item Calculate the determinant of $2\times 2$-matrices and
      $3\times 3$-matrices.
  \end{enumerate}
\end{outcome}

Let $A$ be an $n\times n$-matrix. The \textbf{determinant}%
\index{determinant} of $A$, denoted by $\det(A)$, is a very important
number which we will explore throughout this chapter.

The determinant of a$2\times 2$-matrix is given by the following
formula.

\begin{definition}{Determinant of a $2\times 2$-matrix}{two-by-two-determinant}
  Let $A=\begin{mymatrix}{rr}
    a & b \\
    c & d
  \end{mymatrix}$. Then
  \begin{equation*}
    \det(A) = ad-bc.
  \end{equation*}
\end{definition}

\begin{example}{A $2\times 2$ determinant}{two-by-two-determinant}
  Find $\det(A)$ for the matrix
  $A =  \begin{mymatrix}{rr}
    2 & 4 \\
    -1 & 6
  \end{mymatrix}$.
\end{example}

\begin{solution}
  We have $\det(A) = 2\cdot 6 - (-1)\cdot 4 = 12 + 4 = 16$.
\end{solution}

The determinant is also often denoted by enclosing the matrix with two
vertical lines. Thus
\begin{equation*}
  \det \begin{mymatrix}{cc}
    a & b \\
    c & d
  \end{mymatrix} =\begin{absmatrix}{cc}
    a & b \\
    c & d
  \end{absmatrix}
  = ad - bc.
\end{equation*}

\begin{definition}{Determinant of a $3\times 3$-matrix}{3-by-3-determinant}
  Let $A=\begin{mymatrix}{ccc}
    a_{11} & a_{12} & a_{13} \\
    a_{21} & a_{22} & a_{23} \\
    a_{31} & a_{32} & a_{33} \\
  \end{mymatrix}$. Then
  \begin{equation*}
    \det(A)
    = a_{11}a_{22}a_{33}
    + a_{12}a_{23}a_{31}
    + a_{13}a_{21}a_{32}
    - a_{31}a_{22}a_{13}
    - a_{32}a_{23}a_{11}
    - a_{33}a_{21}a_{12}.
  \end{equation*}
\end{definition}

The following picture may help in memorizing the formula:
\begin{equation*}
  \begin{tikzpicture}[yscale=0.6,xscale=0.9]
    \draw[ultra thick,red!30] (1,-3) -- (3,-1);
    \draw[ultra thick,red!30] (2,-3) -- (4,-1);
    \draw[ultra thick,red!30] (3,-3) -- (5,-1);
    \draw[ultra thick,blue!60] (1,-1) -- (3,-3);
    \draw[ultra thick,blue!60] (2,-1) -- (4,-3);
    \draw[ultra thick,blue!60] (3,-1) -- (5,-3);
    \foreach\i in {1,2,3} {
      \foreach\j in {1,2,3} {
        \path (\j,-\i) node {$a_{\i\j}$};
      }
    }
    \foreach\i in {1,2,3} {
      \foreach\j in {1,2} {
        \path (\j,-\i) + (3,0) node {$a_{\i\j}$};
      }
    }
    \draw(0.5,-2) node {$\left[\rule{0mm}{1.0cm}\right.$};
    \draw(3.5,-2) node {$\left.\rule{0mm}{1.0cm}\right]$};
  \end{tikzpicture}
\end{equation*}
Here, we have written down the matrix $A$, then repeated the first two
columns next to it. The blue lines correspond to the positive terms of
the determinant: $\color{blue}a_{11}a_{22}a_{33}$,
$\color{blue}a_{12}a_{23}a_{31}$, and
$\color{blue}a_{13}a_{21}a_{32}$. The pink lines correspond to the
negative terms: $\color{red}a_{31}a_{22}a_{13}$,
$\color{red}a_{32}a_{23}a_{11}$, and $\color{red}a_{33}a_{21}a_{12}$.

\begin{example}{A $3\times 3$ determinant}{3-by-3-determinant}
  Find $\det(A)$, where
  \begin{equation*}
    A = \begin{mymatrix}{rrr}
      0 & 1 & 2 \\
      3 & 1 & 0 \\
      1 & 1 & -1 \\
    \end{mymatrix}.
  \end{equation*}
\end{example}

\begin{solution}
  We have
  \begin{equation*}
    \det(A) ~=~
    \begin{absmatrix}{rrr}
      0 & 1 & 2 \\
      3 & 1 & 0 \\
      1 & 1 & -1 \\
    \end{absmatrix}
    ~=~
    0\cdot 1\cdot (-1)
    + 1\cdot 0 \cdot 1
    + 2 \cdot 3 \cdot 1
    - 1\cdot 1\cdot 2
    - 1\cdot 0\cdot 0
    - (-1)\cdot 3\cdot 1
    ~=~ 7.
  \end{equation*}
\end{solution}


\lab{Logistic Regression}{Logistic Regression}
\objective{Understand the basics of Logistic Regression, and apply to the Titanic problem.}

\subsection*{Binary Logistic Regression}
A \emph{Logistic Regression Model} is a probability model that can be used to predict outcomes for a set of data.  Usually ``Logistic Regression" refers to what is more appropriately called \emph{binary logistic regression}.  This is a model which can assign data points to one of two sets, and is used in many different fields.  One common medical example is predicting whether or not a patient has a particular disease.  Based upon several factors, which may be both continuous (age, height, weight) and categorical (gender, race), we can quantify the probability of infection.  This probability is computed by way of the \emph{logistic function}, which, given the contributing factors, will return a probability value between $0$ and $1$.  This probability will then be used to assign a label to our input data (`infected' or `not infected', for example) by using some cut-off value, and will depend on the need for accuracy in the specific application.  Success corresponds to a label of $1$, and failure to a label of $0$.

The logistic function takes in as input any real number and returns a value between $0$ and $1$, and is defined explicitly as
\begin{equation}
\phi(t) = \frac{1}{1 + e^{-s}},
\end{equation}
where $s$ is some combination of the input variables $x_1, \cdots, x_n$.  The graph of this function can be seen in Figure ???.   $\phi(x)$ can then be interpreted as the probability of success.  In some cases it is not possible to find a closed-form expression for the correct combination of the input variables.  However, in many cases we can acheive reasonable accuracy by using a linear combination of $x_i, \cdots, x_n$, i.e.
\begin{equation*}
s = c_0 + c_1 x_1 + \cdots + c_n x_n.
\end{equation*}
This can be written more compactly as
\begin{equation*}
s = \bf{c}^{T}\bf{x},
\end{equation*}
where $\bf{x}$ $= (1, x_1, \cdots , x_n)^{T}$ and $\bf{c}$ $\in \mathbb{R}^{n+1}$.

Given a training set of labeled data points, a Logistic Regression Model will find the optimal $\bf{c}$ for the data, and can then be used to predict labels for further data points.

\subsection*{The Titanic Problem}
The Titanic dataset is especially useful for Logistic Regression.  This dataset is composed of actual data obtained concerning the passengers on the ill-fated Titanic voyage, given in the bulleted list below.  Using logistic regression, we can predict whether or not a passenger survived based on this data.  We do so by training a model on a portion of the dataset, the training set, and predicting labels for the remaining data, the test set.

Before beginning our classification, however, we will first need to process our data.  The Titanic dataset contains much more information than is currently relevant for our purposes.  You can obtain this data in Excel Spreadsheet form at [Insert link here].  We recommend using pandas to read in and process the data.  The columns are as follows:
\begin{itemize}
\item \li{pclass}: An integer in $\{1, 2, 3\}$ which describes the class the passenger was in.
\item \li{survived}:  The dependent variable.  $1$ indicates survival, and $0$ death.
\item \li{name}: A string containing the passenger's name.
\item \li{sex}: A string, either `male' or `female'.
\item \li{age}: Either an integer or a float.
\item \li{sibsp}: An integer giving the number of siblings and/or spouse who embarked with the passenger.
\item \li{parch}: An integer giving the number of parents and/or children who embarked with the passenger.
\item \li{ticket}: A string containing the transaction code for the ticket(s) purchased.
\item \li{fare}: A float giving the cost of the ticket purchased.
\item \li{cabin}: A string giving the assigned sleeping cabin for the passenger (note that the majority of this column is blank).
\item \li{embarked}: A string in $\{S, C, Q\}$ corresponding to the location of the passenger's embarkment, Southampton (UK), Cherbourg (France), or Queenstown (Ireland), respectively.
\item \li{boat}: An int or string for those who survived giving which life boat they rode in.
\item \li{body}: An int giving the number of body for those who died who were found and identified.
\item \li{home.dest}: A string giving the home of or location to which the passenger was headed.
\end{itemize}

\begin{problem}
Create a function called \li{initialize} which will process the Titanic data set into useable format by doing the following:
\begin{enumerate}
\item Choose the coulmns that you believe will be relevant in predicting the survival of the passengers, and drop the other columns.  You may not use \li{boat} or \li{body}, as these are dependent on whether or not the passenger survived.  Be sure to include \li{survived}, which will be separated later as the independent variable, as well as \li{sex} and \li{pclass}.
\item Since \li{sex} is really a binary variable, make it one explicitly by changing ``female" and ``male" to be binary values.
\item Drop the rows that contain missing values.  Make sure you have a significant number of rows left.  If you have too few, you may need to choose fewer columns to keep before deleting the incomplete rows.
\item Because the \li{pclass} column is an integer in $\{1, 2, 3\}$, it will be treated as a ranked variable instead of simply a categorical variable.  It may be useful to rank this variable, or it may mess up our classification.  Include a keyword argument \li{pclass_change} with default \li{True}.  If it is set to \li{True}, eliminate this ranking by dividing \li{pclass} into two binary columns.  Make one column a boolean for being $1^{st}$ class and the other a boolean for being $2^{nd}$ class. (This means that a value of $1$ would correspond to $[1, 0]$, $2$ to $[0, 1]$, and $3$ to $[0, 0]$.)
\item Split the remaining rows into a training and a test set using a $60/40$ split.  Be sure and pick random rows for each group and not rows in any particular order.
\end{enumerate}
Have your function return the training set and the test set, in that order.
\end{problem}

\subsection*{Model Evaluation}
Now that we have our training set, we can train a model, which can be used to obtain the probability of success for each data point.  The label chosen depends heavily on the probability cut-off value mentioned previously, which we represent as $\tau$.  In the simplest manner, we can simply pick a value of $\tau = 0.5$, which will then assign the label with the highest likelihood.  However, it is often beneficial to choose a different value of $\tau$.  In regards to the probability of infection for serious diseases, it might be best to give a patient medicine if they have even a $10$ percent chance of infection.  So how can we find the ``best" cut-off value?

In order to determine this, we need to discuss how to measure the accuracy of the labels predicted.  Say that we have picked a cut-off value $\tau$, and have assigned labels to the test set.  Using the predicted labels and the actual labels, we can obtain four important values: the number of \emph{true positives}, \emph{false positives}, \emph{true negatives}, and \emph{false positives}, which are abbreviated TP, FP, TN, and FP, respectively.  These values are integers which together sum to the number of labels predicted.  You can see the definition of these in Table ??? (until the figure is up, true positives are the points with predicted label 1 and true label 1, false positives have predicted label 1 true label 0, true negatives predicted label 0 true label 0, and false negatives true label 1 predicted label 0).  We can now use these to report our accuracy in various metrics:
\begin{itemize}
\item \emph{Prediction accuracy} is defined as $\frac{TP+TN}{TP+FN+FP+TN}$, and is the percentage of correctly predicted cases.
\item \emph{Sensitivity}, also known as the \emph{true positive rate} or $TPR$, is given by the fraction of correctly predicted cases where the actual outcome is $1$, $\frac{TP}{TP+FN}$.
\item \emph{Specificity}, the \emph{true negative rate}, or $TNR$ is the proportion of correctly predicted cases where the true outcome is $0$, $\frac{TN}{FP+TN}$.
\item \emph{False Positive Rate}, or $FPR$, is the proportion of incorrectly predicted cases where the true outcome is $0$, and is given by $\frac{FP}{FP+TN}$.
\end{itemize}
All of these depend strongly on the value chosen for $\tau$.

A \emph{roc curve} is useful in measuring the accuracy of a model.  To make a roc curve, we pick many values for $\tau$, and obtain the False Positive Rate and the True Positive Rate for each.  Then we plot the data points  $(FPR_{\tau},TPR_{\tau})$ for each value of $\tau$ chosen and connect them into a curve.  A completely random label assignment would result in a nearly-linear roc curve, while a more accurate assignment would result in a more steeply-rising curve (see Figure ???).  A good choice for $\tau$ is the one that intersects the family of lines $y=x+b$ at only one point, which intuitively is the point closest to the vertex $(0, 1)$.  Mathematically, this is given by
\begin{equation}
\operatorname*{arg\,max}_{\tau} (TPR_{\tau} - FPR_{\tau}).
\end{equation}

\begin{problem}
Use the function declaration below to find the best value for $\tau$.  You should use evenly spaced values from $0$ to $1$, exclusive.
\begin{lstlisting}
def best_tau(predicted_labels, true_labels, n_tau=100, plot=True):
    """
    Parameters
    ----------
    predicted_labels : ndarray of shape (n,)
        The predicted labels for the data
    true_labels : ndarray of shape (n,)
        The actual labels for the data
    n_tau : int
        The number of values to try for tau
    plot : boolean
        Whether or not to plot the roc curve

    Returns
    -------
    best_tau : float
        The optimal value for tau for the data.
    """
    pass
\end{lstlisting}
\end{problem}

Now that we have a good value for $\tau$, we can quantify the accuracy of a model. We will do so for a few different types of models for the Titanic data you initialized previously.  For the first two, we will use the logistic classifier found in \li{sklearn.linear_model.LogisticRegression}.  The first model we use will be our ``Unchanged Logistic Classifier", which will use our Titanic data with \li{pclass} unchanged.  The second model is the ``Changed Logistic Classifier", which use the data with \li{pclass} changed.

When using this package to create a classifier, we need to input a keyword argument \li{C}.  This value represents the inverse of the regularization strength.  Different values will yield different results.  A better value for \li{C} will yield a more steeply-rising roc curve.  The model can find the coefficients (the vector $\bf{c}$), along with the probabilities of failure and success for each label.  With these in hand, we can use the function \li{sklearn.metrics.roc_curve} to obtain the False Positive Rates, the True Positive Rates, and the optimal value for $\tau$.  You will need to pass in the test data, the probability of success, and the keyword argument \li{pos_label} $= 1$.  The accuracy of the model with input value \li{C} can then be obtained using \li{sklearn.metrics.auc}, which will give the area under the curve.  The larger the area, the more steeply-rising curve we have, and the better the model.  Note that we create a single roc curve using one value for \li{C} and multiple values for $\tau$.

\begin{problem}
Use the following function declaration to return the auc score for the two Logistic Regression models described.
\begin{lstlisting}
def auc_scores(unchanged_logreg, changed_logreg):
    """
    Parameters
    ----------
    unchanged_logreg : float in (0,1)
        The value to use for C in the unchanged model
    changed_logreg : float in (0,1)
        The value to use for C in the changed model

    Returns
    -------
    unchanged_auc : float
        The auc for the unchanged model
    changed_auc : float
        The auc for the changed model
    """
    pass
\end{lstlisting}
\end{problem}

We can test a Naive Bayes model against our Logistic Regression model for both the unchanged and changed models to see the comparative accuracy.  Use the model \li{MultinomialNB}, found in \li{sklearn.naive_bayes}.  You can use it in the same manner as \li{LogisticRegression}, except instead of passing in the keyword argument \li{C}, you will pass in a keyword argument \li{alpha} corresponding to a smoothing parameter.
\begin{problem}
Add input variables \li{unchanged_bayes} and \li{changed_bayes} to your function from the previous problem to obtain the auc for each of these models.  Your function should return all four areas, unchanged logistic regression, changed logistic regression, unchanged Bayes, and changed Bayes, in that order.
\end{problem}

Different values for \li{C} and \li{alpha} will yield different results.  We seek to find those that will maximize the area under the curve.  One way to do so is to pick a number of evenly-spaced points between $0$ and $1$, exclusive, and try each one in turn, keeping the value that yields the greatest accuracy.
\begin{problem}
Use the function declaration below to find the optimal values for \li{C} and $alpha$ as described.
\begin{lstlisting}
def find_best_parameters(choices):
    """
    Parameters
    ----------
    choices : int
        The number of values to try for C and alpha

    Returns
    -------
    best : list of length 4
        The best values for C for the unchanged and changed logistic
         regression models, and the best values for alpha for the
         unchanged and changed Naive Bayes models, respectively.
    """
    pass
\end{lstlisting}
\end{problem}

Now that we have found the optimal inputs for these functions, we can test them against one another.
\begin{problem}
Create a function called \li{results} which will graph of the roc curves for each of the methods, and will print out the names of the models with their corresponding areas, in numerically descending order.
\end{problem}


\documentclass[12pt, a4paper]{report}

\usepackage{amsfonts}
\usepackage[usenames,dvipsnames,svgnames,table]{xcolor}
\usepackage{titlesec}
\usepackage{hyperref}
\hypersetup{
	colorlinks,
	citecolor=Fuchsia,
	filecolor=black,
	linkcolor=Sepia,
	urlcolor=blue
}

\titleformat{\chapter}{\normalfont\huge\bfseries}{\thechapter.}{10pt}{\huge\bfseries}

% Title Page
\title{Fourier-Motzkin extension to Multivariate Polynomial Integer Constraints}

\author{Diogo Sampaio}

\begin{document}

\maketitle

\begin{tabular}[c]{|c|c|p{0.5\linewidth}|}
	\hline
	2017 - 12 - 07 & Diogo Sampaio & First version\\
	\hline
\end{tabular}

\tableofcontents

\chapter{Introduction}
Quantifier elimination is the process of removing existential variables of a given first-order formula, obtaining one that is simpler in the number of variables, and that implies the original formula.

A very well known algorithm is the Fourier-Motzkin elimination process, that given a system (or formula) of inequalities removes one quantified variable by combining all of it's upper and lower bounds.

For example, let the system:
$$\exists x ~ | ~ 0 < y \land z < 1 \land y < x \land x < z$$
In such formula, $x$ is a existential quantified variable, and $y, z$ are parameters. Using FME to remove variable $x$ from this formula requires to find it's upper and lower bounds. In this case we have the upper bound $$x < z$$ and the lower bound $$y < x.$$ For last $x$ is eliminated by isolating $x$ of each formula and combining lower with upper bounds such as:
$$y < x < z \Rightarrow y < z$$
generating the new system:
$$0 < y \land z < 1 \land y < z.$$

This algorithm is designed for linear systems, where all coefficients of the variable being eliminated are numeric values, and the inequality can be classified as either a upper or lower bound.

When dealing with polynomials, variable coefficients might be symbolic expressions. In such case, all possible signs of the coefficient (positive, negative, or zero) must be explored.
e.g. imagine the system:
$$\exists x ~ | ~ a - 1 > 0 \land -b > 0 \land ax > 0 \land bx > 0$$
when eliminating the variable $x$ it is required to classify the terms $ax > 0$ and $bx > 0$ as either upper or lower bounds of $x$. Not knowing the sign of the coefficients ($a, b$) requires to evaluate nine different combinations:
$$a > 0 \land b > 0$$
$$a > 0 \land b = 0$${\color{red}$$a > 0 \land b < 0$$}$$a = 0 \land b > 0$$
$$a = 0 \land b = 0$$
$$a = 0 \land b < 0$$
$$a < 0 \land b > 0$$
$$a < 0 \land b = 0$$
$$a < 0 \land b < 0$$
To avoid this branching we use an implementation of a theorem used in the positiveness test algorithm, proposed by Markus Schweighofer (\url{https://doi.org/10.1016/S0022-4049(01)00041-X}), to retrieve symbolic coefficient signs. Using such algorithm it is possible to infer that only the red assumption holds correct.

The same positiveness test algorithm is of major importance when resolving system over integer variables, instead of reals. It is used in many other techniques required to preserve the precision of the simplified formula, such as extending the normalization (\url{https://doi.org/10.1145/125826.125848}) ~ technique to symbolic expressions, perform convex hull detection and remove redundant constraints.

Our C++ implementation uses GiNaC (\url{https://www.ginac.de/}) to manipulate symbolic expressions and GLPK to implement the positiveness test (\url{https://www.gnu.org/software/glpk}).

For further details please refer to \emph{Profile Guided Hybrid Compilation} \S2.7 and \S5. (\url{https://hal.archives-ouvertes.fr/tel-01428425})

\chapter{Access to it}
\section{Code access and description}
The code is publicly available for download at:

\url{https://gitlab.inria.fr/nunessam/pghc.git}

Source-code folders contents:
\begin{itemize}
	\item doc: this document.
	\item FM: the FME and a stand-alone positiveness tester.
	\item converters: distinct converters to generate input to other quantifier-elimination tools, such as
	\begin{itemize}
	\item QEPCAD: \url{https://github.com/PetterS/qepcad}
	\item reduce/redlog: \url{https://sourceforge.net/projects/reduce-algebra/}
	\end{itemize}
	as well as input to barvinok \url{http://barvinok.gforge.inria.fr/} where it tries to discover if it is possible to prove if a system of constraints define a empty / absurd space. For such task it computes maximum and minimum values of polynomials inside a convex polyhedron space (see \url{https://icps.u-strasbg.fr/upload/icps-2006-173.pdf}).
	\item packages: Dockerfile for a painless process of install.
\end{itemize}
\section{Compiling requirements}
\begin{itemize}
	\item C++11 compatible compiler
	\item GNU make, GiNaC, GLPK
\end{itemize}
{\color{red}OBS: These tools where tested solely in \textbf{Linux}.}

\chapter{How to use}
\section{Fourier-Motzkin elimination - Text IO}
\subsection{Inputs}
The application can either read a input text file or it can read from the standard input stream. Example files can be seen with extension \texttt{.in} inside the folder FM. The binary file it self is called {\color{red}\textbf{\texttt{Simplifier}}}.
\subsection{Input format}
The input formula must be given in a disjunction of systems, where each system is a conjunction of terms. In a bottom up manner we have that:
\begin{itemize}
	\item A \emph{term} is a INTEGER VALUED equality or inequality, such as:	\begin{verbatim}4*x^2 - 5*y < 0
x > 0
y <= s
N*h - 5*z == 0
	\end{verbatim}
	\item A system of terms declares all quantified variables, parameter such as the example in the file \texttt{FM/POL.in}:
	\begin{verbatim}[n]->[jj,j,k,kk,ii,i]: -kk^2+j+j^2-kk-2*ii+2*i = 0 &
		k >= 0 & -1-k+n >= 0 &
		i >= 0 & -1+k-i >= 0 &
		-1+j-k >= 0 & -1-j+n >= 0 &
		kk >= 0 & -1-kk+n >= 0 &
		ii >= 0 & -1+kk-ii >= 0 &
		-1+jj-kk >= 0 & -1-jj+n >= 0 &
		-1-k+kk >= 0 & -1-ii+i >= 0 &
		n >= 0;
		\end{verbatim}
		The distinct parts of the system are:
		\begin{itemize}
		\item A system starts with a list, comma separated, of parameters, e.g.: \texttt{[n]}, \texttt{[nn,n, z, b]}, \texttt{[]}.
		\item An arrow \texttt{->} describing "mapping from".
		\item A list of the quantified variables to be eliminated, comma separated e.g.: \texttt{[jj,j,k,kk,ii,i]}, \texttt{[a, bc,c]}, \texttt{[]}.
		\item A list of polynomial terms, initialized by a colon(\texttt{:}) and separated by the logic and symbol (\texttt{\&}) and terminated by a semi-colon (\texttt{;}).
	\end{itemize}

	\item Multiple systems can be represented in a single input file, being interpreted as an \texttt{logic or} of each one of the systems:
	\begin{verbatim}
	[]->[x]: x < 1 & x > 0;
	[x]->[a]: a < x & a > 3*x-4;
	\end{verbatim}
	is interpreted as $\{\exists x \in \mathbb{Z} | x < 1 \land x > 0\} \lor \{\exists a \in \mathbb{Z} | a < x \land 3x-4 < a\}$.
\end{itemize}

\section{Fourier-Motzlin elimination - Python Interface}

In the folder

\section{Schweighofer Tester}
In the same FM folder, the binary file \texttt{st} allows to interactively play with a single system of constraints. It can read from input files or iteratively add expressions to the system. It allows the user to:
\begin{itemize}
	\item ~[A]ppend or [D]elete a constraint.
	\item ~[T]est if an expression is implied by the system.
	\item ~Obtain the [R]oot expressions of the system.
	\item ~[S]how the generated expression of the Schweighofer Tester.
	\item ~Show the Schweighofer Tester [m]onomials generated.
	\item ~[C]hange the degree (number of multiplications) the Schweighofer Tester multiply all system constraints against all others.
\end{itemize}

\end{document}