\chapter{Time Series Data Mining For Visual Analysis}
Even though most of the data around us have a temporal nature, we often transform them into spatial data by either fixing time or ignoring time completely. It is not because the time aspect is not essential, though it makes all tasks significantly more challenging as it increases the number of data points and adding relations between them.  For example, it is straightforward to compare temperatures in two places at a fixed time. However, once we want to compare their temperature profiles throughout the year, we cannot omit the temporal aspect of the data. Because the temporal aspect adds relations between points and increases the dimensionality of the data, the analysis becomes significantly more difficult. This effect is multiplied once we work with a dataset of time series \cite{met:vis-data-help}. Mostly there, we see an opportunity where visual exploration could help.

This chapter will discuss the right choices for measuring the distance between time series and examine how to effectively apply them in the subsequent visual analysis. Then it goes through the clustering and anomaly detection methods available for large datasets. However, before we dive into the chapter, we will clarify the basic terminology.

Having a set of objects $X$, we define the \textit{distance function} as:
\begin{equation}
    d: X \times X \rightarrow R^{+}_0
\end{equation}
The resulting non-negative real value is called a \textit{distance} or \textit{dissimilarity}. We call a distance function a \textit{metric} if and only if it satisfies the following three axioms:
\begin{enumerate}
    \item $\forall x, y \in X: d(x, y) = 0 \Leftrightarrow x = y $ (identity of indiscernibles)
    \item $\forall x, y \in X: d(x, y) = d(y, x)$ (symmetry)
    \item $\forall x, y, z \in X: d(x, y) + d(y, z) \ge d(x, z)$ (triangle inequality)
\end{enumerate}
A \textit{semi-metric} is a distance function that satisfies the first two axioms, but it does not guarantee the triangle inequality.
\textit{A time series} is then defined as a sequence of data points ($X_t$) indexed by time ($t$).

\section{Distance Measures for Time Series}
In a typical scenario, a distance metric computes the distance for one specific data type (categorical, spatial, numerical). However, time series are a combination of multiple data types as each point has an order in time, and we have to take it into account when computing the distance.

To the best of our knowledge, there are over 30 used distance measures for time series today. As there is no "universally best" metric \cite{met:universal}, we will go through the different types of distance measures used for time series, following the categorization proposed by Esling and Agon \cite{met:classification}.

\subsection{Shape-Based Distance Measures}
Shape-based distance measures compare the overall shape of the time series \cite{met:classification}. They are the closest derivation of commonly used distance measures adjusted for the time series data, making them easy to implement and understand. Regardless of their simple nature, they generally yield very competitive results and are considered a gold standard of time series distance measures \cite{met:comparison-new, met:dtw-best-0, met:dtw-best-1, met:universal, met:fDTW}.

\subsubsection{Euclidian Distance and Other $L_p$ Norms}
The most basic and widely used distance measures for spatial and temporal data are Euclidian distance (ED) and other $L_p$ norms \cite{met:lp-norm, met:lp-norm-usage}.

A distance measure on a set of objects $X$ is a function $d:X\times~X\rightarrow[0; \infty)$.
Given the two time series $X$ and $Y$, where $X=(x_1; x_2; ...; x_n)$ and $Y=(y_1; y_2;...; y_m)$, we define $L_p$ norms as listed in Table~\ref{tab:lp-norms}.
\begin{table}[!htbp]
\begin{tabular}{ll}
\textbf{Norm}          & \textbf{Definition}                               \\ \hline
$L_1$ - Manhattan      & $\sum^{n}_{i=1}|x_i - y_i|$                       \\
$L_2$ - Euclidian      & $\sqrt{\sum^{n}_{i=1}(x_i - y_i)^2}$              \\
$L_p$ - Minkowski      & $\sqrt[p]{\sum^{n}_{i=1}(x_i - y_i)^\frac{1}{p}}$ \\
$L_\infty$ - Infinite  & $\max_{i=1..n}|x_i - y_i|$
\end{tabular}
\caption{$L_p$ norms for time series.}
\label{tab:lp-norms}
\end{table}

Although the $L_p$ norms perform well, especially on larger datasets \cite{met:universal, met:comparison-new, met:met-comparison}, they require time series of equal length and tend to be fragile to the noise, shifts, and different speeds in time series \cite{met:classification}.

\subsubsection{Dynamic Time Warping}
An elegant solution to the above-mentioned limitations of the $L_p$ norms are elastic measures, especially the Dynamic Time Warping (DTW) \cite{met:dtw}. DTW is a method to compute a distance between a pair of time series. When computing DTW distance between time series $X$ and $Y$, it finds the best alignment between them as a minimal cost warping path ($W=w_1 ,...,w_k ,...,w_K$) in $m \times n$ matrix. Elements of the matrix represent the cost to align two corresponding points ($(x_i - y_j)^2$). The warping path starts in the bottom-left corner and ends in the upper-right one (Fig. \ref{fig:dtw_warping}). It is formally defined as:
\begin{equation}
    DTW(X; Y) = \argmin_{W=w_1 ,...,w_k ,...,w_K} \sqrt{ \sum^{K}_{k=1; w_k=(i; j)} (x_i - y_j)^2}
\end{equation}

When comparing DTW with the Euclidian distance, DTW finds the best possible alignment between time series (Fig. \ref{fig:Euclid_vs_DTW}). That allows us to reasonably compare time series that could have different lengths, speeds, and shift \cite{met:dtw, met:dtw-window}.
\begin{figure}[h]
     \centering
     \begin{subfigure}[b]{0.495\textwidth}
        \centering
        \includesvg[width=\textwidth]{img/dtw_warping_path.svg}
        \caption{}
        \label{fig:dtw_warping}
     \end{subfigure}
     \hfill
     \begin{subfigure}[b]{0.495\textwidth}
        \centering
        \includesvg[width=\textwidth]{img/dtw_vs_euclid.svg}
        \caption{}
        \label{fig:Euclid_vs_DTW}
     \end{subfigure}
    \caption{
        (a) displays the optimal warping path between two time series. The darkness of color encodes the distance between the corresponding data points (darker is further away). (b) shows the alignment of data points between two time series using the Euclidian distance and DTW.
    }
\end{figure}

Dynamic time warping is considered one of the best distance measures for time series and regularly outperforms other more sophisticated approaches \cite{met:universal, met:comparison-new, met:dtw-best-1, met:dtw-best-0}, but its use comes with two main concerns:
\begin{enumerate}
    \item In practice, DTW is solved by dynamic programming with $O(n^2)$ time complexity.
    \item DTW is a semi-metric -- it does not guarantee triangle inequality, limiting the number of optimization techniques and structures we can use.
\end{enumerate}

The time complexity of this algorithm becomes an issue when operating with long time series and their large amounts. As a solution to the DTW algorithm's time complexity, it is possible to either use lower and upper bounding constraints on the maximal allowed warping path \cite{met:dtw-koegh-rata} or use a temporal constraint on the warping window size \cite{met:dtw-window}. Among the most used are Sakoe-Chiba \cite{met:dtw-window}, Itakura \cite{met:dtw-itakura}, and upper bound to discard complex paths \cite{met:DTW-zheng}. Another option is to use an approximate algorithm proposed by Salvador and Chan \cite{met:FastDTW} called FastDTW, where the warping path is recursively projected and optimized from a lower resolution. This algorithm has linear time complexity. Using the bounding or approximate algorithm comes with faster computational speed and could help with better accuracy due to generalization \cite{met:fDTW}.

DTW is a semi-metric that makes it rather problematic to use with many machine learning algorithms that require a metric space. As there does not exist any complete solution to this issue, there are only partial answers, like Feature DTW from \textcite{met:fDTW}. As we want to use a wide variety of algorithms for visual exploration, we will discuss this in more detail in the following section.

\subsubsection{SpADe, DISSIM, and OSB}
In 2007, three new distance measures for time series were proposed. The first one is the Spatial Assembling Distance (SpADe), a distance measure used for streaming data that recognizes matching patterns concerning shifting and scaling in temporal and amplitude axes \cite{met:spade}.

For measuring similarity between trajectories with different sampling rates, Frentzos et al. \cite{met:dissim} proposed the DISSIM as an approximation of the integral of Euclidian distance.

Because data tend to be noisy and include many outliers, Latecki et al. \cite{met:osb} proposed Optimal Subsequence Bijection (OSB) for similarity search. This method automatically finds the most suitable subsequence for a sensible comparison. Its disadvantage is a higher computational cost.

\subsection{Other Distance Measures}
Except for the shape-based metrics used for time series data, other methods could be used for the distance calculation. These types of measures focus on specific properties shared among time series.

\textit{Edit-based} distances were originally designed for string comparison. The idea is to find the minimal number of basic string operations (insertion, deletion, and substitution) by which we can transform one string to another.  The most popular are techniques measuring the similarity by comparing the longest common subsequences (LCSS) \cite{met:LCSS0, met:LCSS1, met:swale}. They are robust to outliers and noise in the data. Similar options are the Edit Distance on Real Sequence (EDR) \cite{met:edr} and Edit Distance with a Real Penalty (ERP) \cite{met:erp}. EDR finds the minimal number of edit operations to convert time series and penalizes unmatched regions' gaps by their lengths. On the other hand, ERP is a combination of DTW and EDR approaches. It uses the Manhattan distance as a penalization for local shifting. Edit-based distances tend to perform well, but generally are outperformed by DTW-like measures \cite{met:dtw-best-0, met:dtw-best-1}.

\textit{Feature-based} distance measures use features extracted from the original time series, such as correlation \cite{met:coss-col} or coefficients from discrete Fourier transformation and discrete wavelet transformation \cite{met:dft-dwt}. It is possible to use them in specific applications but they overly show worse results than other techniques \cite{met:comparison-new}.

\textit{Model-based} measures are an option for very long sequences, but they require prior knowledge of the time series generating process \cite{met:classification}. After choosing and fitting the parametric temporal model on time series, the distance measure is a likelihood that time series came from the same model. Most standard models are using Hidden Markov models or ARMA models \cite{met:hmm}.

\textit{Compression-based} strategies have been succesfully applied in bioinformatics and medical data applications \cite{met:cdm, met:cdm-santos, met:cdm-espoti}. This approach uses the fact that concatenation and compression of similar time series should produce a higher compression ratio than for very diverse ones.

Many of these techniques can outperform more common shape-based measures, particularly in specific domains, but lack wide usage and universality of shape-based approaches. Even though we would primarily use the shape-based measure in this thesis, the proposed techniques for time series analysis can use any underlying distance measure.

\section{Artificial Feature Space Representation}
From the previous section, we can presume that shape-based distance measures, especially Dynamic Time Warping and its derivatives, are robust and accurate. The original DTW algorithm's time complexity can be reduced significantly by using bounding or approximation. However, we still need to consider its semi-metric nature because the triangle inequality is required for many machine learning algorithms \cite{cluster:decade-review}. It significantly reduces the number of available algorithms that we could use, primarily as datasets' size increases. Partial answer to this problem is building a feature space representation, which we could consider to be in a metric space.

\subsection{Feature DTW Transformation}
\textcite{met:fDTW} introduced an idea to build a feature space representation by using DTW. He transforms the original dataset of time series into a new feature space on which it is possible to apply any technique that is used in a metric space.

The transformation works as follows. Having dataset $X$ consisting of time series $(X_1; X_2; ...; X_n)$, we define the Feature DTW transformation as function
\begin{equation}
    f_{fDTW}: X \times X \rightarrow M
\end{equation}
where $M=(m_{ij}) \in \mathbb{R}_{\ge 0}^{n \times n}$ is a distance matrix with $m_{ij}=DTW(X_i; X_j)$.

 In his paper, Kate sees an improved accuracy in classification tasks to widely used techniques using DTW directly. Further, he shows that using DTW with bonding and approximation increases the accuracy probably due to better generalization, and points out that any distance measure could be used instead of DTW. Another notable advantage of using the Feature DTW transformation is that it is possible to combine DTW obtained features with other measures and features. In his work, \textcite{met:fDTW} combined DTW features with Euclidian ones, which led to an additional approvement in terms of accuracy. This technique's main drawback is space complexity as the distance matrix $M$ size is $n^2$ regarding the number of time series.

\subsection{Prototyped Feature DTW Transformation}
The result of Feature DTW transformation on two similar time series should produce very similar and correlated feature vectors. A statistical point suggests that we are adding very little new information while increasing the number of dimensions in the new feature space. An increase in dimensionality leads to a rise in computational and space requirements, while having multiple significantly correlated features is a problem for many machine learning algorithms. Iwana et al. \cite{met:protofDTW} showed that using only a fraction of the original distance matrix $M$ could obtain comparable or better results than using the whole matrix.

Having the dataset $X=(X_1; X_2; ...; X_n)$ and a set of time series $P~ \subseteq~X,$ $P=(P_1; P_2; ...; P_m)$ called \textit{prototypes}, we define the transformation as function
\begin{equation}
    f_{Iwana-fDTW}: X \times X \rightarrow K
\end{equation}
where $K=(k_{ij}) \in \mathbb{R}_{\ge 0}^{n \times m}$ is a distance matrix with $k_{ij}=DTW(X_i; P_j)$.

As this method strongly depends on prototypes, their selection is crucial for the method's success. They proposed a supervised technique using the \textit{AdaBoost} algorithm and weak learners on one feature at a time. Using this technique, they were able to use a smaller number of components while increasing the classification accuracy. Because this method is available only for supervised problems, they briefly discussed multiple statistical strategies to chose possible prototypes.

All of the approaches described in their work require prior computation of the whole distance matrix, which is problematic for larger datasets. In this thesis, we will propose methods that do not require a full distance matrix in advance but rather compute it interactively.

\section{Clustering}
Another part of our work is to determine the clusters in a dataset. Because we are using the Feature DTW transformation, we are not limited to clustering specific only for time series, but we can use a wide variety of clustering algorithms used for spatial data. Because we will be working with large datasets in our analysis, we will list only algorithms that are scalable to a large number of samples and features.

We will separate the clustering algorithms into three categories:
\begin{enumerate}
    \item Distance-based methods
    \item Methods using the nearest-neighbor graph
    \item Density-based methods
\end{enumerate}

\subsection{Distance-based Methods}
Distance-based methods are directly using the distance between the data points to determine their affiliations to the clusters. The two main representatives of this category, that scale well  with dataset's size, are \textit{K-means} and \textit{Agglomerative Hierarchical Clustering} methods.

\subsubsection{K-means}
The \textit{K-means} method \cite{vis:kmeans} is a simple clustering algorithm that separates the data into $n$ clusters, minimizing the within-cluster sum of squares. It produces convex and isotropic clusters, and it is easily scalable to large datasets. The disadvantages of this method are that we have to define the number of clusters in advance, and in some cases, the clusters in data are neither convex nor isotropic. Nevertheless,  it is a solid starting point for any cluster analysis.

\subsubsection{Agglomerative Hierarchical Clustering}
The \textit{Agglomerative Hierarchical Clustering} approach \cite{vis:agg-cluster, vis:kmeans} uses a bottom-up strategy to join the two most similar data points or clusters into a single bigger one. The primary representation is a tree-like structure, where leaves are single data points, nodes represent clusters, and the root is a single unified cluster. The algorithm selects two closest nodes based on the merge strategy in each iteration and joins them into a single one, until the given number of clusters is met.

There are four main merge strategies called \textit{linkages}:
\begin{enumerate}
    \item \textit{Single linkage} -- the minimal distance among all tuples of data points from two clusters.
    \item \textit{Complete linkage} -- the maximal distance among all tuples of data points from two clusters.
    \item \textit{Average linkage} -- the average distance between all pairs of data points from two clusters.
    \item \textit{Ward linkage} -- minimization of the within-cluster sum of squares, similar to the K-means.
\end{enumerate}
This clustering method is scalable with the increasing number of samples and, based on the linkage type, provides different cluster types. The ward linkage produces the most even-sized clusters among the merging strategies, but is usable only with Euclidian metrics, while also having much higher time complexity. If we want to use non-Euclidian metrics, the average linkage is a feasible alternative. Both single linkage and complete linkage are very efficient and scale well on large datasets. The drawback is that they are fragile to noise in the data, as they are using only the distance between two points from clusters.

\subsection{Methods Using the Nearest Neighbor Graph}
Another family of clustering algorithms is using the pre-constructed affiliation graph from data points. Usually, it uses the nearest neighbor graph. The two main representatives are the Affinity Propagation and Spectral clustering.
\subsubsection{Affinity Propagation}
The \textit{Affinity Propagation} \cite{vis:affini} sends messages between data points in the affiliation graph to determine cluster centers and the number of clusters. The disadvantage is that it is usable only for smaller datasets because of its quadratic time complexity, making it unusable for our datasets of interest.

\subsubsection{Spectral Clustering}
The \textit{Spectral Clustering} \cite{vis:spectral} starts with the nearest neighbor graph represented as a similarity matrix, where each row represents a data point and its graph distance to the other points. After selecting the number of clusters $n$, it computes the optimal graph cuts to create a connected component for every cluster. The drawback of spectral clustering is that it can be efficiently computed only for a small number of clusters.

\subsection{Density-based Methods}
The last group of clustering algorithms, density-based methods, uses the difference between dense and sparse regions to determine the optimal clustering.
The main challenge of these methods is to determine the correct density estimate of the dataset, which is problematic for high-dimensional datasets. As the number of dimensions increases, the volume of the space rises exponentially, thus making the dataset quickly sparse. This phenomenon is also known as the \textit{Curse of Dimensionality} \cite{exp:curse-of-dim}.

Due to the expected size of datasets of our interest, and usage of the Feature DTW transformation, we will be interest and the usage with high-dimensional data. Therefore, we are not considering the most common density-based techniques such as \textit{Gaussian Mixture Models} \cite{vis:gauss-mixt} and \textit{Mean Shift} \cite{vis:mean-shift}, as they make assumptions about the underlying data distribution, or computationally expensive.

\subsubsection{DBSCAN}
The first density-based clustering algorithm in our work is the \textit{Density-Based Spatial Clustering of Applications With Noise (DBSCAN)} \cite{vis:dbscan}. This algorithm separates the dataset into dense clusters divided by sparse regions and outlying data points, considered as noise. Firstly it separates data points into three groups: core points, non-core points close to the core points, and noise.

We say that a point is a core point if and only if in its surrounding area with radius $\epsilon$ there are at least $X$ other data points (the minimal number of data points). The non-core points close to the core points have a core point within the radius $\epsilon$, yet they do not meet the minimal data points requirement within $\epsilon$. Noise points do not have any core points in their surrounding area. Hence, the DBSCAN algorithm automatically detects the number of clusters and only requires the radius $\epsilon$ and the minimal number of samples $X$. The disadvantage of DBSCAN is that it cannot detect clusters with very different density, as the radius is same for each point.

\subsubsection{OPTICS}
The \textit{Ordering Points To Identify the Clustering Structure (OPTICS)} \cite{vis:optics} is a partial solution of the DBSCAN's problem with clustering regions with different densities. OPTICS uses the minimal number of neighbors $X$, and instead of a single radius like in DBSCAN, it uses the maximal radius to consider $\epsilon$. For each point, it computes the core distance, the minimal radius, in which is the point is considered as a core point, and reachability distance to every other point, which is either the core distance of the other point or the distance between them, whichever of them is bigger. If two neighboring points have a reachability distance smaller than the core distance, we consider them to be a part of the same cluster.

Because OPTICS computes both core distance and reachability, it has a much higher computational cost than DBSCAN. With the usage of spatial indexing trees, we could avoid a costly computation of the full distance matrix, making it applicable to larger datasets.

\subsubsection{HDBSCAN}
The \textit{Hierarchical Density-based Spatial Clustering of Applications with Noise (HDBSCAN)} \cite{vis:hdbscan} is another clustering algorithm originating in the DBSCAN. Like DBSCAN and OPTICS, it starts by computing the core distance of every point, which is a minimum radius in which there are at least $X$ data points (minimal number of data points) and the mutual reachability distance for every pair of data points -- the largest value among their core distances and their mutual distance.

Afterwards, it finds the minimal spanning tree of a weighted graph, where data points are vertices, and edges between them have the weight of their mutual reachability distance (Fig.~\ref{fig:hdbscan_tree}).
\begin{figure}[h]
     \centering
     \begin{subfigure}[b]{0.495\textwidth}
        \centering
        \includesvg[width=\textwidth]{img/hdbscan_dataset.svg}
        \caption{}
        \label{fig:hdbscan_dataset}
     \end{subfigure}
     \hfill
     \begin{subfigure}[b]{0.495\textwidth}
        \centering
        \includesvg[width=\textwidth]{img/hdbscan_tree.svg}
        \caption{}
        \label{fig:hdbscan_tree}
     \end{subfigure}
    \caption{The minimal spanning tree from HDBSCAN (a) using the mutual reachability distance on an artificially generated dataset (b).
    }
\end{figure}

The next step clusters the tree vertices by single-linkage hierarchical clustering using the edge weights. In Fig.~\ref{fig:hdbscan_hier}, we can see the dendrogram visualizing such clustering, where on the $y$ axis we can see the \textit{mutual reachability distance} at which the nodes merge.
\begin{figure}[h]
     \centering
     \begin{subfigure}[b]{0.48\textwidth}
        \centering
        \includesvg[width=\textwidth]{img/hdbscan_hier.svg}
        \caption{}
        \label{fig:hdbscan_hier}
     \end{subfigure}
     \hfill
     \begin{subfigure}[b]{0.48\textwidth}
        \centering
        \includesvg[width=\textwidth]{img/hdbscan_cons.svg}
        \caption{}
        \label{fig:hdbscan_cons}
     \end{subfigure}
    \caption{HDBSCAN's single linkage hierarchy of connected components (a) and the most stable clusters in condensed cluster tree (b).
    }
\end{figure}

The next step converts the tree into a hierarchy of connected components by sorting the edges and merging them in an ascending order (Fig.~\ref{fig:hdbscan_hier}). To determine the number of clusters in the data, DBSCAN computes the stability of each cluster. As a measure, we are using the $\lambda$ value, which is the reversed \textit{mutual reachability distance} ($\lambda = \frac{1}{distance}$). For every cluster, HDBSCAN defines the distance when it is created as $\lambda_c$ and the distance when it splits into multiple clusters as $\lambda_d$. Then for every point $p$ in the cluster, it computes the value $\lambda_p$, which is the value when the point leaves the cluster either by cluster split or complete separation. The stability of a cluster $C$ is then defined as:
\begin{equation}
    \sum_{p \in C}(\lambda_p - \lambda_c)
\end{equation}
Finally, HDBSCAN travels through the tree, starting from the leaves going up to the root node. For every node (cluster), if its stability is greater than the sum of its children's stabilities, HDBSCAN marks the cluster as a real cluster and unmarks all its descendants. Once the algorithm reaches the root node, it stops and returns all marked clusters as final data clustering (Fig.~\ref{fig:hdbscan_cons}).

Even though HDBSCAN consists of multiple computational steps, there are highly optimized implementations \cite{vis:hdbscan-imp}, allowing HDBSCAN to be effectively used for large datasets.

\section{Anomaly Detection}
In the last part of our analysis, we want to detect and study the anomalous data points within a dataset. Here we will focus on two methods, \textit{Isolation Forest} and \textit{Lightweight On-line Detector of Anomalies}, which are widely used and efficient even for large high-dimensional datasets.

\subsection{Isolation Forest}
One of the widely used anomaly detection algorithms for large datasets is the \textit{Isolation Forest} \cite{vis:isoforest}. The central concept of this approach is that anomalous data points can be isolated faster than the rest of the dataset.

To find the anomalies, it constructs the forest of binary isolation trees. A binary isolation tree is a simple decision tree with restricted depth, where each node of the tree selects the random feature and random value in its range as a separator. The anomaly score of a data point is the average depth it reached in trees in the isolation forest.

Because of its simple nature, the Isolation Forest scales very well with the number of samples and number of features, while also achieving competitive results.

\subsection{Lightweight On-line Detector of Anomalies}
The \textit{Lightweight On-line Detector of Anomalies (LODA)} is an anomaly detection method designed to process huge high-dimensional real-time data streams. LODA uses a combination of randomly generated sparse projections and histograms to determine the anomaly score of each data point.

In the first step, LODA generates $X$ random projection vectors with only $\sqrt{d}$ non-zero taken from $N(0, 1)$ values, where $d$ is input data's dimensionality. Then, it constructs a histogram on every projection vector. The final anomaly score of each point is the negative average logarithm of probability obtained throughout all histograms.

Because LODA is using sparse projections and histograms as very simple density estimators, it has very fast training and execution time. Another advantage is that through the vectors' sparsity, it is possible to discover which feature contributed most to the data points anomaly score. We compute the \textit{one-tailed two-sample t-test} between probabilities from histograms on projections with and without a specific feature. The higher the value of the \textit{t-test}, the stronger is the feature significance.

\section{Chapter Summary}
In this chapter, we discussed distance measures for time series. Among them, shaped-based distances, particularly the Euclidian distance and Dynamic Time Warping (DTW) seem like the most promising ones. The main drawback of DTW is its semi-metric nature. To use the advantages of DTW in a metric space, we can use Feature DTW transformation to create a feature space representation of DTW. This representation then forms the input for other machine learning algorithms and can be combined with other distance measures and feature extraction algorithms.

There are several options for clustering of large datasets:
\begin{itemize}
    \item \textit{K-means} is suitable when we know the number of clusters and want evenly sized convex clusters. It is a good starting technique as it scales very well with the dataset size.
    \item \textit{Hierarchical agglomerative clustering} is suitable for medium to large datasets based on the chosen linkage. \textit{Ward linkage} produces the most evenly sized clusters but has the highest complexity and is defined only in the Euclidian space. For non-Euclidian cases, the \textit{average linkage} is a good alternative. While the \textit{single linkage} is very efficient and can be used on large datasets, with the capability to produce non-convex clusters it is fragile to noise in the data.
    \item \textit{OPTICS} and \textit{HDBSCAN} are suitable for cases when we do not know the number of clusters in our data but want to specify only the minimal number of clusters. As these methods are density-based, we have to be aware of the curse of dimensionality, and consider transforming the dataset into a low-dimensional representation.
\end{itemize}

In the field of anomaly detection, we recommend the \textit{Isolation Forest} and \textit{LODA} approaches. Both methods are ensembles of simple estimators, giving them high-speed performance regardless of the dataset size. \textit{LODA} also provides us with the capability for scoring the features based on their significance in the anomaly score.


\section{Circles}

\subsection{Defining a circle}
Circle $(O,r)$ is the set of points at distance exactly $r$ from a point $O=(x_0,y_0)$.

We can also define it by equation
\[(x-x_0)^2 + (y-y_0)^2 = r^2\]
or parametrically as the set of all points
\[(x_0+r\cos\theta, y_0+r\sin\theta)\]
with $\theta$ in $[0,2\pi)$, for example.

\centerFig{circle0}

\subsection{Circumcircle}
The \term{circumcircle} of a triangle $ABC$ is the circle that passes through all three points $A$, $B$ and $C$.

\centerFig{circle1}

It is undefined if $A$, $B$, $C$ are aligned, and unique otherwise. We can compute its center $O$ this way:
\begin{lstlisting}
pt circumCenter(pt a, pt b, pt c) {
    b = b-a, c = c-a; // consider coordinates relative to A
    assert(cross(b,c) != 0); // no circumcircle if A,B,C aligned
    return a + perp(b*sq(c) - c*sq(b))/cross(b,c)/2;
}
\end{lstlisting}

The radius can then be found by taking the distance to any of the three points, or directly taking the length of \lstinline|perp(b*sq(c) - c*sq(b))/cross(b,c)/2|, which represents vector $\vv{AO}$.

\begin{mathy}
The formula can be easily interpreted as the intersection point of the line segment bisectors of segments $[AB]$ and $[AC]$, when considering coordinates relative to $A$. Consider the bisector of $[AB]$. It is perpendicular to $[AB]$, so its direction vector is $\perpop(\vv{AB})$, and it passes through the middle point $\frac{1}{2}\vv{AB}$, so its constant term (variable \lstinline|c| in the \lstinline|line| structure) is $\perpop(\vv{AB}) \times \frac{1}{2}\vv{AB} = -\frac{1}{2}|AB|^2$. Similarly, the bisector of $[AC]$ is defined by direction vector $\perpop(\vv{AC})$ and constant term $-\frac{1}{2}|AC|^2$. We then just plug those into the formula for line intersection found in section~\ref{line-line}:
\begin{align*}
\vv{AO} &= \frac{(-\frac{1}{2}|AB|^2)\perpop(\vv{AC}) - (-\frac{1}{2}|AC|^2)\perpop(\vv{AB})}{\perpop(\vv{AB}) \times \perpop(\vv{AC})} \\
&= \frac{\perpop(|AC|^2\vv{AB} - |AB|^2\vv{AC})}{2\ \crossv{AB}{AC}}
\end{align*}
\end{mathy}

\subsection{Circle-line intersection}
A circle $(O,r)$ and a line $l$ have either 0, 1, or 2 intersection points.

\centerFig{circle2}

Let's assume there are two intersection points $I$ and $J$. We first find the midpoint of $[IJ]$. This happens to be the projection of $O$ onto the line $l$, which we will call $P$.

\centerFig{circle3}

Once we have found $P$, to find $I$ and $J$ we need to move along the line by a certain distance $h$. By the Pythagorean theorem, $h=\sqrt{r^2-d^2}$ where $d$ is the distance from $O$ to $l$.

This gives the following implementation (note that we have to divide by $\norm{\dirof{l}}$ so that we move by the correct distance). It returns the number of intersections, and places them in \lstinline|out|. If there is only one intersection, \lstinline|out.first| and \lstinline|out.second| are equal.
\begin{lstlisting}
int circleLine(pt o, double r, line l, pair<pt,pt> &out) {
    double h2 = r*r - l.sqDist(o);
    if (h2 >= 0) { // the line touches the circle
        pt p = l.proj(o); // point P
        pt h = l.v*sqrt(h2)/abs(l.v); // vector parallel to l, of length h
        out = {p-h, p+h};
    }
    return 1 + sgn(h2);
}
\end{lstlisting}

\subsection{Circle-circle intersection}
Similarly to the previous section, two circles $(O_1,r_1)$ and $(O_2,r_2)$ can have either 0, 1, 2 or an infinity of intersection points (in case the circles are identical).

\centerFig{circle4}

As before, we assume there are two intersection points $I$ and $J$ and we try to find the midpoint of $[IJ]$, which we call $P$.

\centerFig{circle5}

Let $d=|O_1O_2|$. We know from the law of cosines on $O_1O_2J$ that
\[\cos(\measuredangle{O_2O_1J}) = \frac{d^2 + r_1^2 - r_2^2}{2dr_1}\]
and since $O_1PJ$ is a right triangle,
\[|O_1P| = r_1\cos(\measuredangle{O_2O_1J}) = \frac{d^2 + r_1^2 - r_2^2}{2d}\]
which allows us to find $P$.

Now to find $h=|PI|=|PJ|$, we apply the Pythagorean theorem on triangle $O_1PJ$, which gives $h=\sqrt{r_1^2 - |O_1P|^2}$.

This gives the following implementation, which works in a very similar way to the code in the previous section. It aborts if the circles are identical.
\begin{lstlisting}
int circleCircle(pt o1, double r1, pt o2, double r2, pair<pt,pt> &out) {
    pt d=o2-o1; double d2=sq(d);
    if (d2 == 0) {assert(r1 != r2); return 0;} // concentric circles
    double pd = (d2 + r1*r1 - r2*r2)/2; // = |O_1P| * d
    double h2 = r1*r1 - pd*pd/d2; // = h^2
    if (h2 >= 0) {
        pt p = o1 + d*pd/d2, h = perp(d)*sqrt(h2/d2);
        out = {p-h, p+h};
    }
    return 1 + sgn(h2);
}
\end{lstlisting}

\begin{mathy}
Let's check that if $d \neq 0$ and variable \lstinline|h2| in the code is nonnegative, there are indeed 1 or 2 intersections (the opposite is clearly true: if \lstinline|h2| is negative, the length $h$ cannot exist). The value of \lstinline|h2| is
\begin{align*}
r_1^2 &- \frac{(d^2 + r_1^2 - r_2^2)^2}{4d^2}\\
&= \frac{4d^2r_1^2 - (d^2 + r_1^2 - r_2^2)^2}{4d^2}\\
&= \frac{-d^4-r_1^4-r_2^4+2d^2r_1^2+2d^2r_2^2+2r_1^2r_2^2}{4d^2}\\
&= \frac{(d+r_1+r_2)(d+r_1-r_2)(d+r_2-r_1)(r_1+r_2-d)}{4d^2}
\end{align*}
Let's assume this is nonnegative. Thus an even number of those conditions are false:
\[d+r_1 \geq r_2 \qquad d+r_2 \geq r_1 \qquad r_1+r_2 \geq d\]
Since $d,r_1,r_2 \geq 0$, no two of those can be simultaneously false, so they must all be true. As a consequence, the triangle inequalities are verified for $d,r_1,r_2$, showing the existence of a point at distance $r_1$ from $O_1$ and distance $r_2$ from $O_2$.
\end{mathy}

\subsection{Tangent lines}
We say that a line is tangent to a circle if the intersection between them is a single point. In this case, the ray going from the center to the intersection point is perpendicular to the line.

\centerFig{circle6}

Here we will try and find a line which is tangent to two circles $(O_1,r_1)$ and $(O_2,r_2)$. There are two types of such tangents: outer tangents, for which both circles are on the same side of the line, and inner tangents, for which the circles are on either side.

\centerFig{circle7}

We will study the case of outer tangents. Our first goal is to find a unit vector parallel to the rays $[O_1P_1]$ and $[O_2P_2]$, in other words, we want to find $\vv{v} = \vv{O_1P_1}/r_1$. To do this we will try to find angle $\alpha$ marked on the figure.

\centerFig{circle8}

If we project $O_2$ onto the ray from $O_1$, this forms a right triangle with hypothenuse $d=|O_1O_2|$ and adjacent side $r_1-r_2$, which means $\cos\alpha = \frac{r_1-r_2}{d}$. By the Pythagorean theorem, the third side is $h = \sqrt{d^2-(r_1-r_2)^2}$, and we can compute $\sin\alpha = \frac{h}{d}$.

From this we find $\vv{v}$ in terms of $\vv{O_1O_2}$ and $\perpop{\left(\vv{O_1O_2}\right)}$ as
\begin{align*}
\vv{v} &= \cos\alpha \left(\vv{O_1O_2}\,/d\right) \pm \sin\alpha \left(\perpop{\left(\vv{O_1O_2}\right)}/d\right)\\
&= \frac{(r_1-r_2)\,\vv{O_1O_2} \pm h \perpop{\left(\vv{O_1O_2}\right)}}{d^2}
\end{align*}
where the $\pm$ depends on which of the two outer tangents we want to find.

We can then compute $P_1$ and $P_2$ as
\[P_1 = O_1 + r_1\vv{v} \quad \mbox{and} \quad P_2 = O_2 + r_2\vv{v}\]

\exo{
    Study the case of the inner tangents and show that it corresponds exactly to the case of the outer tangents if $r_2$ is replaced by $-r_2$. This will allow us to write a function that handles both cases at once with an additional argument \lstinline|bool inner| and this line:
    \lstinputlisting{code/bits/tangent-inner.cpp}
}

This gives the following code. It returns the number of tangents of the specified type. Besides,
\begin{itemize}
\item if there are 2 tangents, it fills \lstinline|out| with two pairs of points: the pairs of tangency points on each circle $(P_1,P_2)$, for each of the tangents;
\item if there is 1 tangent, the circles are tangent to each other at some point $P$, \lstinline|out| just contains $P$ 4 times, and the tangent line can be found as \lstinline|line(o1,p).perpThrough(p)| (see \ref{perpThrough});
\item if there are 0 tangents, it does nothing;
\item if the circles are identical, it aborts.
\end{itemize}
\begin{lstlisting}
int tangents(pt o1, double r1, pt o2, double r2, bool inner, vector<pair<pt,pt>> &out) {
    if (inner) r2 = -r2;
    pt d = o2-o1;
    double dr = r1-r2, d2 = sq(d), h2 = d2-dr*dr;
    if (d2 == 0 || h2 < 0) {assert(h2 != 0); return 0;}
    for (double sign : {-1,1}) {
        pt v = (d*dr + perp(d)*sqrt(h2)*sign)/d2;
        out.push_back({o1 + v*r1, o2 + v*r2});
    }
    return 1 + (h2 > 0);
}
\end{lstlisting}

Conveniently, the same code can be used to find the tangent to a circle passing through a point by setting \lstinline|r2| to 0 (in which case the value of \lstinline|inner| doesn't matter).


\documentclass[11pt]{article}
\usepackage[a4paper,margin= 2cm]{geometry}
\usepackage{graphicx}
\usepackage{caption}
\usepackage{subcaption}

\title{\LARGE{\bf{Task 3 - Root Finding}}}
\author{\Large{\bf{Kirtan Patel - AE19B038}}}
\date{}

\begin{document}

\maketitle

\section{Introduction}
This week's Task is:\\
Given a function F(x), find a value x\textsubscript{i} for which F(x\textsubscript{i}) = 0. We will use a 3 methods to solve the problem.
\begin{enumerate}
	\item Interval Bisection Method
	\item Newton-Raphson Method
	\item Secant Method
\end{enumerate}

\subsection{Interval Bisection Method}
This method is an algorithm to find a zero of a function by repeated bisection of an interval and determination of the subinterval where the zero must be found. It is simple and reliable, but relatively slow.The essence of the bisection method lies in the fact that the sign of a function f(x) changes on opposite sides of a root.

Consider an interval [a,b] and a function f that has different signs at the boundaries of that interval: \[f(a).f(b)<0\]

Therefore, for a continuous function f, there must be atleast one zero within the interval [a,b]. To numerically approach a zero x\textsubscript{i} of f, the midpoint of the interval is taken as first approximation (with a1 = a , b1=b):
\[ c1 = a1 + \frac{b1-a1}{2} = \frac{a1 + b1}{2}\]

If the function value at the midpoint is still larger than the permissible error $\epsilon$, one has to repeat the bisection for the subiterval where the zero must be found. This can be determined from a change of sign of f,which can occur for one of the subintervals only (either for [a1,c1] or for [c1,b1]).\\

To determine which subinterval to continue the search in, we check the following:
\[if~~f(a1).f(c1)<0 \Rightarrow a2=a1, b2=c1\]
\[if~~f(b1).f(c1)<0 \Rightarrow a2=c1, b2=b1\]

In this new interval, we again determine the midpoint \[c2 = \frac{a2+b2}{2}\] and check the error at this point. \\

Continue this till we find a midpoint(root) which has an error less than the permissible error.\\
We can even set an upper-bound on the number of times we bisect the interval.\\

The code applied in the task takes both the above mentioned values as input and stops at whichever condition is met first.

\begin{figure} [h!]
	\centering
		\centering
		\includegraphics[width=0.5\linewidth]{bsm1}
		\caption{Graphical Representation of Root Finding using Interval Bisection Method}
		\label{fig1}
\end{figure}

\subsubsection{Remarks}
\begin{enumerate}
\item \textbf {Non-detectable zeros}\\
The interval bisection method relies on a change of the sign of . However, might have a zero where the sign does not change, i.e. where the graph does not cross the axis but just touches it. Such a second-order zero cannot be found by the interval bisection method.
\item \textbf {Multiple zeros}\\
The interval bisection method always converges to a zero. However, the given range might have several zeros in it . This will not be noticed by the interval bisection method, which therefore will find only one of them — but it is not clear which one will be found.
\item \textbf {Numerical Stability}\\
To detect a change of the sign of , the interval bisection algorithm is formulated above in terms of a product of two function values. Since a zero of is approached, these values can become very small, so that their product may lead to numerical underflow. This can furthermore be confused with the case that accidentally the midpoint is the sought-after zero, i.e. not the product but already a single function value equals zero. A robust numerical implementation must take care of these possibilities and should not use products of function values.
\item \textbf {Numerical Accuracy}\\
In general, the best accuracy that can be reached for the value of the zero does not equal the machine precision. This is related to the influence of the limited precision of floating-point arithmetic. Because of that, only the relative error can be used as a measure of numerical accuracy. In other words, the machine precision determines the smallest relative error reachable. Specifying an for the relative error smaller than the machine precision does not make sense and should be prevented. \\
\item \textbf {Numerical Reliablity}\\
Additionally, not only the deviation with respect to the value might be of interest: It may happen that the function has a rather steep slope at the zero, so that despite of a small relative error, , a function value may occur that is still large. Therefore it is advisable to check this after a zero has been determined numerically. \\
On the other hand, if the function has a rather low slope at the zero, the interval of very small function values can be rather large, which implies that the problem of numerical underflow described in remark 3 above might occur already before the prescribed accuracy has been reached. Then, the algorithm will most likely exit with an unreliable value. Obviously, in such a case it does not make sense to specify too small an for the relative error.
\end{enumerate}

\pagebreak
\subsection{Newton-Raphson Method}
The Newton-Raphson Method is a simple algorithm to find an approximate solution for the root of a real-valued function f(x)=0. If the function f satisfies sufficient assumptions then after repeative steps the x\textsubscript{n+1} given by:\[x_{n+1} = x_n -\frac{f(x_n)}{f'(x_n)}\] will be a good approximation to the root.\\

We can set 2 control parameters to control the root search:
\begin{itemize}
	\item error $\epsilon$ \\
	If f($x_n$) $<$ $\epsilon$, then we stop the iterations
	\item N$\_$max \\
	If the number of iterations crosses an upper-bound, then we stop the iterations and print the answer
\end{itemize}
If either of the conditions is met, then the iterations are stopped and the root is printed

\begin{figure}[h!]
	\centering
	\centering
	\includegraphics[width=0.8\linewidth]{nrm1}
	\caption{Graphical Representation of Root Finding using Newton-Raphson Method}
	\label{fig2}
\end{figure}
\vspace{1cm}
\subsubsection{Remarks}
Newton's method is only guaranteed to converge if certain conditions are satisfied. If the assumptions made in the proof of quadratic convergence are met, the method will converge. Failure of the method to converge indicates that the assumptions made in the proof were not met.
\begin{enumerate}
	\item \textbf {Bad Starting Points}\\
	In some cases the conditions on the function that are necessary for convergence are satisfied, but the point chosen as the initial point is not in the interval where the method converges. In such cases a different method, such as bisection, should be used to obtain a better estimate for the zero to use as an initial point.

	\item \textbf{Iteration point is stationary}\\
	Consider a function \[ f(x) = 1 - x^2\]
	It has a maximum at x = 0 and solutions of f(x) = 0 at x = $\pm$1. If we start iterating from the stationery point $x_0 = 0$ (where derivative is zero), $x_1$ will be undefined, since the tangent at (0,1) is parallel to the x-axis.\\

	The same issue occurs if, instead of the starting point, any iteration point is stationery. Even if the derivative is small but not zero, the next iteration will be a far worse approximation

	\item \textbf{Starting Point enters a cycle}\\
	For some functions, some starting points may enter an infinite cycle, preventing convergence. Let  \[ f(x) = x^3 -2x + 2\]
	and take 0 as the starting point. The first iteration produces 1 and the second iteration returns to 0 so the sequence will alternate between the two without converging to a root. In fact, this 2-cycle is stable: there are neighborhoods around 0 and around 1 from which all points iterate asymptotically to the 2-cycle (and hence not to the root of the function).
	\item \textbf{Derivative Issues}\\
	There can be several issues related to the derivative of the function.If the function is not continuously differentiable in a neighborhood of the root then it is possible that Newton's method will always diverge and fail, unless the solution is guessed on the first try.

	Such cases include, derivative no being defined at the root and discontinuous derivatives.
	\item \textbf{Multiple Solutions}\\
	The algorithm found a solution but it does not mean that this solution is unique. Actually it found the closest. In case of multiple roots, it finds only one
\end{enumerate}
\pagebreak

\subsection{Secant Method}
The secant method is very similar to the bisection method except instead of dividing each interval by choosing the midpoint the secant method divides each interval by the secant line connecting the endpoints. The secant method always converges to a root of f(x)=0 provided that f(x) is continuous on [a,b] and f(a).f(b)$<$0.\\

The secant method procedure is almost identical to the bisection method. The only difference it how we divide each subinterval.

With an initial interval satisfying the initial conditions(a1=a, b1=b), we first compute f($x_o$), where $x_o$ is given by the secant line \[ x_o = a_o - f(a_o).\frac{b_o - a_o}{f(b_o) - f(a_o)}\]

To determine the next subinterval [$a_1, b_1$]:
\[if~~ f(a_o).f(x_o)<0 \Rightarrow a_1=a_o, b_1=x_o\]
\[if~~ f(b_o).f(x_o)<0 \Rightarrow a_1=x_o, b_1=b_o\]

We then find $x_1$ using the secant line formula and repeat the following steps till we reach son interval [$a_N, b_N$] which returns the value $x_N$, the x-intercept of the Nth subinterval\\

Here too, we can set 2 control parameters as mentioned before.

\begin{figure}[h!]
	\centering
	\centering
	\includegraphics[width=0.6\linewidth]{secm1}
	\caption{Graphical Representation of Root Finding using Newton-Raphson Method}
	\label{fig3}
\end{figure}

\subsubsection{Remarks}
Secant Method is faster when compared to Bisection and Newton-Raphson methods as the order of convergence is higher in Secant Method. But there are some drawbacks too as follow:
\begin{enumerate}
	\item \textbf{It may not converge}

	\item \textbf{Derivative issues}\\
	It is likely to have difficulty if f´(a) = 0 i.e the x-axis is tangent to the graph y=f(x) at x = a.
\end{enumerate}

\section{Results and Analysis}
The following results are obtained by keeping the permissible error $\epsilon$ = 0.001, and varying the maximum number of iterations \textit{N} for different root-finding methods.

Here
\[ f1(x) = x^3 - 3x^2 - x + 9\]
\[ f2(x) = (x^3 - 3x^2 - x + 9).e^x\]

For each method, the results have been obtained for 2 sets of starting conditions. The below plots show the comparison between the results obtained.
\subsection{Interval Bisection Method}
\begin{figure} [h!]
	\centering
	\begin{subfigure}{0.55\textwidth}
		\centering
		\includegraphics[width=\linewidth]{bsm-f1-abs}
		\caption{$\|$f1(x)$\|$ at the roots vs Max Number of Iterations}
		\label{fig1:sub1}
	\end{subfigure}%
	\begin{subfigure}{0.55\textwidth}
		\centering
		\includegraphics[width=\linewidth]{bsm-f2-abs}
		\caption{$\|$f2(x)$\|$ at roots vs Max Number of Iterations}
		\label{fig1:sub2}
	\end{subfigure}
	\caption{Results obtained using Bisection Method}
\end{figure}

\subsection{Newton-Raphson Method}
\begin{figure} [h!]
	\centering
	\begin{subfigure}{0.5\textwidth}
		\centering
		\includegraphics[width=\linewidth]{nrm-f1-abs}
		\caption{$\|$f1(x)$\|$ at the roots vs Max Number of Iterations}
		\label{fig2:sub1}
	\end{subfigure}%
	\begin{subfigure}{0.5\textwidth}
		\centering
		\includegraphics[width=\linewidth]{nrm-f2-abs}
		\caption{$\|$f2(x)$\|$ at the roots vs Max Number of Iterations}
		\label{fig2:sub2}
	\end{subfigure}
	\caption{Results obtained using Newton-Ralphson Method}
\end{figure}

\subsection{Secant Method}
\begin{figure} [h!]
	\centering
	\begin{subfigure}{0.55\textwidth}
		\centering
		\includegraphics[width=\linewidth]{secm-f1-abs}
		\caption{$\|$f1(x)$\|$ at the roots vs Max Number of Iterations}
		\label{fig3:sub1}
	\end{subfigure}%
	\begin{subfigure}{0.55\textwidth}
		\centering
		\includegraphics[width=\linewidth]{secm-f2-abs}
		\caption{$\|$f2(x)$\|$ at roots vs Max Number of Iterations}
		\label{fig3:sub2}
	\end{subfigure}
	\caption{Results obtained using Secant Method}
\end{figure}

\subsection{Comparison of Methods}
\begin{figure} [h!]
	\centering
	\begin{subfigure}{0.55\textwidth}
		\centering
		\includegraphics[width=\linewidth]{all-f1-abs}
		\caption{$\|$f1(x)$\|$ at the roots vs Max Number of Iterations}
		\label{fig4:sub1}
	\end{subfigure}%
	\begin{subfigure}{0.55\textwidth}
		\centering
		\includegraphics[width=\linewidth]{all-f2-abs}
		\caption{$\|$f2(x)$\|$ at roots vs Max Number of Iterations}
		\label{fig4:sub2}
	\end{subfigure}
	\caption{Comparison of Results obtained}
\end{figure}
\pagebreak
\section{Special Case}
Taking, \[f_3(x) = x^3 - 2x + 2~~~and~~~x_o = 0\]
using the Newton-Raphson Method to find the root, we get the following result.

\begin{figure}[h!]
	\centering
	\centering
	\includegraphics[width=\linewidth]{special_case_newton-raphson}
	\caption{Graphical Representation Special Case of Root Finding using Newton-Raphson Method}
	\label{fig4}
\end{figure}

Here we see one of the cases where the Newton-Raphson Method fails to converge to the root. The Starting Point is a part of a cycle.

Using,\[x_{n+1} = x_n -\frac{f(x_n)}{f'(x_n)}\] and the fact that \[f_3(x) = x^3 - 2x + 2 ~~ and ~~ f_3'(x) = 3x^2 - 2\]

For $x_o = 0$, f$_3$(0) = 2 and f'$_3$(0) = -2, and thus we get \[x_1 = 0 - \frac{2}{-2} = 1\]

For $x_1 = 1$, f$_3$(1) = 1 and f'$_3$(1) = 1, and thus we get \[x_2 = 1 - \frac{1}{1} = 0~~...\]
This cycle goes on and we are not able to obtain the root. This is mentioned in the section 1.2.1 .

\section{Inference and Conclusion}
It is very rarely the case that one particular method is better than another in all instances. Typically different methods have different strengths and weaknesses which make them good for some problems but bad for others. This is why in numerical analysis, we like to keep a broad range of tools in our toolboxes.\\

Consider, for example, a comparison of the bisection method and Newton’s method. When Newton’s method converges, it typically does so much faster than the Bisection method. We get the answer that we need with minimal computation, so it would seem that Newton’s method is better than the Bisection method.

On the other hand, we need to remember that Newton’s method does not always work. Since the iterative formula involves a division by the derivative of a function, the method will completely fail if that derivative is 0 for any particular iteration. The nice thing about the Bisection method is that as long as you start with an interval that contains the root, you are guaranteed to converge! So with a problem that makes Newton’s method fail, Bisection is clearly a better alternative.\\

Evaluating the 3 methods of root-finding, we can see that while Bisection is very slow, it is always guaranteed to find a root once the process begins.
Compare this to Newton’s Method, which doesn’t have such guarantees.This is pretty typical behavior for the two algorithms since Bisection has linear convergence while Newton’s method quadratic convergence.

Bisection only requires function evaluations, but Newton’s Method also requires a first deriva-
tive. You may take this for granted in an academic setting, but in practical situations finding
the derivative can be impractical or expensive compared to a function evaluation. One way
to get around finding exact expressions for the derivative is to use a finite difference ap-
proximation for the derivative. The workaround is to use the Secant Method (Poor Man’s Newton Method), which approximates the derivative by the slope of a secant line. But this requires two starting guesses which are hopefully close to the root.\\

A more efficient method though is a combination of the two — simply, start with the bisection until you get a rough estimate of the root, then use Newton’s method to refine your estimate.\\

We can even tweak either of these methods to make them work better, and thus there are many more variations, but again, each has its own strengths and weaknesses.\\

\section{References}
\begin{enumerate}
	\item  https://x-engineer.org/undergraduate-engineering/advanced-mathematics/numerical-methods/the-bisection-method-for-root-finding/
	\item https://silviaximenametodos.blogspot.com/2010/06/bisection-method$\_$29.html
	\item https://www.tf.uni-kiel.de/matwis/amat/comp$\_$math/kap$\_$1/backbone/r$\_$se19.html
	\item https://predictivehacks.com/newton-raphson-method-in-python/
	\item https://en.wikipedia.org/wiki/Newton's$\_$method
	\item https://www.quora.com/What-is-the-Newton-Raphson-method
	\item https://secure.math.ubc.ca/~pwalls/math-python/roots-optimization/secant/
	\item https://en.wikipedia.org/wiki/Secant$\_$method
	\item http://www.numericmethod.com/About-numerical-methods/roots-of-equations/secant-method
\end{enumerate}

\end{document}


\subsection{Strong rules}

\subsubsection{Strong rules}

We use frequent patterns to generate strong rules, \(R\).

An example of a strong rule is \(a\rightarrow b\).

We can look at this by comparing the support of \(a\) to \(a\land b\).

\(supp(A\rightarrow B)=P(A\land B)\)

\(conf(A\rightarrow B)=P(B|A)\)

\(conf(A\rightarrow B)=\dfrac{P(A\land B)}{P(A)}\)


\lab{Applications}{Riemann Sphere and Mobius Transformations}{Riemann Sphere and Mobius Transformations}

\objective{Understand the Riemann Sphere in graphics applications.}

We have now examined several applications of complex numbers and functions.
In this lab we extend several of these ideas and develop some intuition about the Riemann sphere using visualization techniques in Python.

Recall that the complex numbers are an extension of the real numbers that include the imaginary numbers.
We extend them even further in this section and examine the extended complex numbers.
Similar to the extended real numbers, the extended complex numbers are our regular set of complex numbers with the addition of a point at infinity.
This allows us to examine, for example, certain quotients that are undefined on the standard complex numbers.

The Riemann sphere construction allows a compact and intuitive construction of the extended complex numbers.
Consider the standard complex plane, only now in 3-space instead of a 2 dimensional plane.
Let the $x$-axis correspond to the real part of the complex numbers and let the $y$-axis correspond to the imaginary part.
Let $z=0$ for all $x$ and $y$.

Now consider a the unit sphere centered at $(0,0,1)$ combined with the complex numbers in 3-space.
We may map every point of the extended complex plane onto this sphere.
Let infinity correspond to the point $(0,0,2)$ at the very top of the sphere.
For all other points on the complex plane consider the line between the point on the complex plane and the point at the top of the sphere.
Let each point of the complex plane map to the point where this line intersects with the surface of the sphere.
Note that for any complex number, there is exactly one such point.
This construction is illustrated in Figure \ref{riemann:stereographic}

\begin{figure}
\includegraphics[width=\textwidth]{StereographicIllustration.png}
\caption{Stereographic projection of a point in the complex plane onto the unit sphere.}
\label{riemann:stereographic}
\end{figure}

\begin{problem}  Write a function that, given a complex number, returns the coordinates of the corresponding point on the riemann sphere in 3-space.
\end{problem}

\begin{problem}
Write a function that, given the coordinates of a point on the Riemann Sphere, returns the corresponding point on the complex plane.
If the input point is $(0,0,2)$ have your function return "infinity."
\end{problem}

\section*{Mobius Transformations}

Recall from the previous chapter that a conformal mapping is a complex function that preserves angles between lines.
We now examine a special kind of conformal mapping called a Mobius transformation (not to be confused with the Mobius Transform), or a fractional linear transformation.
We first give the definition of a fractional linear transformation, and then examine what sorts of transformations we can do with them.

\begin{definition} A fractional linear transformation is a any complex function of the form
\[
f(z) = \frac{az + b}{cz + d}
\]
 Where a, b, c, and d are complex numbers with the restriction that $ad \neq bc$.
\end{definition}

Fractional linear transformations are versatile enough to include translations, rotations, magnifications, or any combination of the same, of shapes in the complex plane.
For example, if we wish to magnify the unit disk by a factor of 2 and then translate it up 2 along the imaginary axis, we would use the following transformation:
\[
f(z)=2z+2i
\]

\begin{figure}
\includegraphics[width=\textwidth]{mobius1.png}
\caption{The unit disk and its image under the transformation $2z+2i$.}
\label{riemann:mobius1}
\end{figure}

\begin{problem} Write a Python function that accepts arguments $a,b,c,$ and $d$.
Perform the corresponding fractional linear transformation on a grid inside the box $[-1,1]\times[-i,i]$, and then visualize the box before and after the transformation on the same plot, but using different colors.
\end{problem}

We now will mention some of the important properties of fractional linear transformations.

With some manipulation, it can be shown that:

$$\frac{az+b}{cz+d}=\frac{a}{c}+\frac{bc-ad}{c} \frac{1}{cz+d}$$

This implies that any fractional linear transformation can be represented as the composition of the functions $\alpha x$, $x+\beta$, and $\frac{1}{x}$ where $\alpha$ and $\beta$ are complex.

This allows us to determine some things about the properties of fractional linear transformations.
Since each of those functions is holomorphic wherever it is defined, we may say the same of their composition.
In other words, our fractional linear transformation is holomorphic except at $z=-\frac{d}{c}$, or if $c=0$, the transformation is holomorphic at all points in the complex plane.

Since fractional linear transformations are holomorphic, they are also conformal.
This can also be readily derived by considering a fractional linear transformation as the composition of the functions above.
If each of the functions above is conformal on a domain, their composition will also be conformal on that domain.

We will now consider the effects of fractional linear transformations on points of the Riemann Sphere.
Figures \ref{riemann:mobius2} and \ref{riemann:mobius3} show a grid on the complex plane, it's projection onto the Riemann Sphere, the image of the grid under the transformation $\frac{1}{z}$, and it's projection onto the Riemann Sphere.

\begin{figure}
\includegraphics[width=\textwidth]{mobius2.png}
\caption{The Riemann Sphere witha $3\times 3$ grid projected onto it.}
\label{riemann:mobius2}
\end{figure}

\begin{figure}
\includegraphics[width=\textwidth]{mobius3.png}
\caption{The images of the Riemann Sphere and the $3\times 3$ grid shown in Figure \ref{riemann:mobius2} under the transformation $\frac{1}{z}$.}
\label{riemann:mobius3}
\end{figure}

Notice how both circles and lines on the complex plane are circles on the Riemann Sphere.
A line on the complex plane is really just a circle on the Riemann Sphere that passes through the point at infinity.
An interesting property of fractional linear transformations is that they are circle preserving.
This means that the image of a circle on the Riemann Sphere under any fractional linear transformation is also a circle on the Riemann Sphere.
In terms of the standard complex plane, we may say that the image of any circle or line is a circle or line.

This can be proven analytically by considering the equation of a circle on the complex plane.
Let $z=a=bi$.
For some circle centered at a point $E=\alpha + \beta i$ with radius $r$, $a$ and $b$ will satisfy the equation $(a-\alpha)^2+(b-\beta)^2=r^2$.
Expanding this expression and simplifying we have:
$$a^2-b^2-2a\alpha-2b\beta+\alpha^2+\beta^2-r^2=0$$
Letting $D=\alpha^2+\beta^2-r^2$, we may rewrite the above equation as follows:
$$z\bar{z}+E\bar{z}+\bar{E}z+D=0$$
This gives motivation for the general equation of a circle in the complex plane.
$$Az\bar{z}+E\bar{z}+\bar{E}z+D=0$$
Where $A$ and $D$ are real and $E$ is complex.
If $A\neq 0$ and $E\bar{E}-AD>0$ then this is the equation of a circle.
Note that if $A=1$, $E$ is the center of the circle.
If $A=0$ then this is the equation of a line (or a circle through infinity on the Riemann Sphere).

Now to see that a fractional linear transformation is circle preserving, recall that it can be represented as the composition of the functions $\alpha x$, $x+\beta$, and $\frac{1}{x}$ where $\alpha$ and $\beta$ are complex.
If each of these functions is circle preserving, their composition will also be circle preserving, and we will have the desired result.

The function $x+\beta$ is a translation and will clearly be circle preserving.

Letting $z=\alpha w$, we have:
$$A\alpha \bar{\alpha} w\bar{w}+\bar{E}\alpha z+E\bar{\alpha} \bar{z}+D=0$$
which may be rewritten as:
$$A|\alpha|^2 w\bar{w}+\bar{(E\bar{\alpha})}z+E\bar{\alpha} \bar{z}+D=0$$
which is still the equation of a circle, so the function $\alpha z$ is also circle preserving.

Now, letting $z=\frac{1}{w}$, we have:
$$\frac{A}{w\bar{w}}+\frac{\bar{E}}{w}+\frac{E}{\bar{w}}+D=0$$
which we may rewrite as:
$$Dw\bar{w}+\bar{E}\bar{w}+Ew+A=0$$
which is, again, the equation of a circle.

Since each of these functions is circle preserving, any composition of these functions will also be circle preserving, so all fractional linear transformations are circle preserving.

\begin{problem}
Write a python function which accepts the constants $a$, $b$, $c$, and $d$ and returns the constants $A$, $E$, and $D$ of the general equation of the image of the unit circle under the corresponding fractional linear transformation.
You may say, without loss of generality, that $A=1$.
\end{problem}

Fractional linear transformations have varied effects on the Riemann Sphere.
A simple example is the transformation $e^{\frac{\pi i}{4}}z$ which rotates the entire complex plane 45 degrees.
This transformation is shown in Figure \ref{riemann:mobius4}.

\begin{figure}
\includegraphics[width=\textwidth]{mobius4.png}
\caption{The images of the Riemann Sphere and the $3\times 3$ grid shown in Figure \ref{riemann:mobius2} under the transformation $e^{\frac{\pi i}{4}}z$.}
\label{riemann:mobius4}
\end{figure}

We can also rotate the Riemann Sphere in other directions.
The transformation $\frac{2z+4}{-iz+2i}$ rotates the Riemann sphere 90 degrees in the x-z plane.
Figure \ref{riemann:mobius5} is a plot of the same grid under that transformation.

\begin{figure}
\includegraphics[width=\textwidth]{mobius5.png}
\caption{The images of the Riemann Sphere and the $3\times 3$ grid shown in Figure \ref{riemann:mobius2} under the transformation $\frac{2z+4}{-iz+2i}$.}
\label{riemann:mobius5}
\end{figure}

\begin{problem}
The matrix representation of the fracional linear transformation $\frac{az+b}{cz+d}$ is:
\[
\begin{pmatrix}
a&b\\
c&d
\end{pmatrix}
\]
Prove that for two fractional linear transformation $f$ and $g$ with matrix representations $F$ and $G$ that the matrix representation of $f(g(z))$ is $FG$.
\end{problem}

\begin{problem}
It turns out that we can define a fractional linear transformation by specifying 3 points and their images under the transformation.
Write a python function that accepts two lists of three complex numbers and returns the matrix representation of the fractional linear transformation that maps the numbers of the first list to the numbers of the second list.
\end{problem}


\subsection{The Auxiliary Equation}
\noindent
It's not a coincidence that the coefficients of the polynomial that we had to find the 0's of in the above example matched the coefficients of the differential equation. We call this polynomial the auxiliary equation, and it can help us solve linear, homogeneous differential equations with constant coefficients of any order.
\begin{definition}
	A nth order, linear, homogeneous differential equation with constant coefficients has the form
	\begin{equation*}
		a_ny^{(n)} + a_{n-1}y^{(n-1)} + \ldots + a_1y' + a_0y = 0
	\end{equation*}
	The corresponding auxiliary equation is
	\begin{equation*}
		a_nr^n + a_{n-1}r^{n-1} + \ldots + a_1r + a_0 = 0
	\end{equation*}
\end{definition}

\noindent
We now have a method for solving these equations with the roots of the auxiliary equation are all unique.
\begin{theorem}
	Let $\left\{r_1, \ldots, r_n\right\}$ be the set of unique roots to an auxiliary equation corresponding to a nth order, linear, homogeneous differential equation with constant coefficients. The set of fundamental solutions are $\left\{C_1e^{r_1x}, \ldots, C_ne^{r_nx}\right\}$, and the general solution is
	\begin{equation*}
		y = C_1e^{r_1x} + \ldots + C_ne^{r_nx}
	\end{equation*}
\end{theorem}

\noindent
We can easily extend this method to deal with roots of higher multiplicities.
\begin{theorem}
	Let $\alpha$ be a root with multiplicity $k$ to an auxiliary equation corresponding to a nth order, linear, homogeneous differential equation with constant coefficients. Then $e^{\alpha x}, xe^{\alpha x}, \ldots, x^{k-1}e^{\alpha x}$ are fundamental solutions.
\end{theorem}

\input{./higherOrder/constCoeffs/complexRoots.tex}


\chapter{Logarithms}

After the world had created exponents, it needed the opposite. We
could talk about the quantity $? = 2^3$, that is, ``What is the
product of 2 multiplied by itself three times?''  We needed some way
to talk about $2^? = 8$, that is ``2 to the what is 8?'' So we
developed the logarithm.\index{logarithm} \index{log}

Here is an example:

$$\log_{2}8 = 3$$

In English, you would say ``The logarithm base 2 of 8 is 3.''

The base (2, in this case) can be any positive number. The argument
(8, in this case) can also be any positive number.

Try this one: What is the logarthim base 2 of 1/16?

You know that $2^{-4} = \frac{1}{16}$, so $\log_{2} \frac{1}{16} = -4$.

\section{Logarithms in Python}

Most calculators have pretty limited logarithm capabilities, but
python has a nice \pyfunction{log} function that lets you specify both
the argument and the base.  Start python, import the math module, and try taking a few logarithms:\index{log!in python}

\begin{Verbatim}
>>> import math
>>> math.log(8,2)
3.0
>>> math.log(1/16, 2)
-4.0
\end{Verbatim}

Let's say that a friend offers you 5\% interest per year on your
investment for as long as you want. And you wonder, ``How many years
before my investment is 100 times as large?'' You can solve this problem with logarithms:

\begin{Verbatim}
>>> math.log(100, 1.05)
94.3872656381287
\end{Verbatim}

If you leave your investment with your friend for 94.4 years, the
investment will be worth 100 times what you put in.

\section{Logarithm Identities}

The logarithm is defined this way:\index{logarithm!identities}

$$\log_b a = c \iff b^c = a$$

Notice that the logarithm of 1 is always zero, and $\log_b b = 1$.

The logarithm of a product:

$$\log_b a c = \log_b a + \log_b c$$

This follows from the fact that $b^{a + c} = b^a b^c$. What about a quotient?

$$\log_b \frac{a}{c} = \log_b a - \log_b c$$

Exponents?

$$\log_b \left(a^c\right) = c \log_b a$$

Notice that because logs and exponents are the opposite of each other, they can cancel each other out:

$$b^{\log_b a} = a$$

and

$$\log_b \left(b^a\right) = a$$

\section{Changing Bases}

I mentioned that most calculators have pretty limited logarithm
capabilities. Most calculators don't allow you to specify what base
you want to work with. All scientific calculators have a button for
``log base 10''.  So you need to know how to use that button to get
logarithms for other bases.  Here is the change-of-base identity:\index{logarithm!change of base}

$$\log_b a = \frac {\log_c a}{\log_c b}$$

So, for example, if you wanted to find $\log_2 8$, you would ask the
calculator for $\log_{10} 8$ and then divide that by $\log_{10} 2$.
You should get 3.

\section{Natural Logarithm}

When you learn about circles, you are told that the circumference of a
circle is about 3.141592653589793 times its diameter.  Because we use
this unweildy number a lot, we give it a name: We say ``The
circumference of a circle is $\pi$ times its diameter.''

There is a second unweildy number that we will eventually use a lot in
solving problems.  It is about 2.718281828459045 (but the digits
actually go on forever, just like $\pi$). We call this number $e$. (I'm
not going to tell you why $e$ is special now, but soon...)\index{e}\index{logarithm!natural}

Most calculators have a button labeled ``ln''.  That is the
\textit{natural logarithm} button. It takes the log in base $e$.\index{ln}

Similarly, in python, if you don't specify a base, the logarithm is done in base $e$:

\begin{Verbatim}
>>> math.log(10)
2.302585092994046
>>> math.log(math.e)
1.0
\end{Verbatim}

\section{Logarithms in Spreadsheets}

Spreadsheets have three log functions:
\begin{itemize}
\item \pyfunction{LOG} takes both the argument and the base. \pyfunction{LOG(8,2)} returns 3.
\item \pyfunction{LOG10} takes just the argument and uses 10 as the base.
\item \pyfunction{LN} takes just the argument and uses $e$ as the base.
\end{itemize}

Here is a plot from a spreadsheet of a graph of $y = LOG(x, 2)$.

\includegraphics[width=0.8\textwidth]{log_graph.png}

Spreadsheets also have the function \pyfunction{EXP(x)} which returns
$e^x$.  For example, \pyfunction{EXP(2)} returns 7.38905609893065.


\section{Analysis}

\subsection{Analytical}

This section will detail the time complexity of each method.\\

\begin{table}[t]
\centering
\begin{tabular}[t]{| l | l |}
	\hline
	\multicolumn{2}{| c |}{\code{RedBlackTree}} \\ \hline
	\textbf{Method} & \textbf{Time complexity} \\ \hline
	\code{isEmpty()} & O(1) \\ \hline
	\code{contains(Object)} & O(log n) \\ \hline
	\code{hasPredecessor(Object)} & O(1) \\ \hline
	\code{hasSuccessor(Object)} & O(1) \\ \hline
	\code{predecessor(Object)} & O(log n) \\ \hline
	\code{successor(Object)} & O(log n) \\ \hline
	\code{min()} & O(1) \\ \hline
	\code{max()} & O(1) \\ \hline
	\code{add(Object)} & O(log n) \\ \hline
	\code{delete(Object)} & O(log n) \\ \hline
	\code{iterator()} & O(1) \\ \hline
	\code{iterator(Object)} & O(log n) \\ \hline
	\code{toString()} & O(n) \\ \hline
\end{tabular}
\begin{tabular}[t]{| l | l |}
	\hline
	\multicolumn{2}{| c |}{\code{TreeIterator}} \\ \hline
	\textbf{Method} & \textbf{Time complexity} \\ \hline
	\code{hasNext()} & O(1) \\ \hline
	\code{next()} & O(log n) \\ \hline
	\code{remove()} & O(log n) \\ \hline
\end{tabular}
\caption{Summary of time complexity of each method.}
\end{table}

Red-black trees have five main properties:
\begin{enumerate}
\item A node is either black or red.
\item The root node is black.
\item Every leaf node is black.
\item A red node has black children.
\item Every path from a node to a leaf descendent contains the same number of black nodes.
\end{enumerate}

Since the height of the tree is referenced in demonstrating the worst-case time complexity of almost every method in the class, I first prove the following limit on the height of a red-black tree \cite{clrs}:
\begin{equation*}
h \leq 2\log_2(n+1)
\end{equation*}
\begin{proof}
To begin, we prove that a subtree with root node $n$ has a minimum of $2^{h_b(n)} - 1$ internal nodes (where $b_h(n)$, the `black-height' of $n$, is the number of black nodes on any path from the node $n$ to any leaf, not including $n$): we do this by induction. For the base case - if the black height of $n$ is zero, then $n$ must be a leaf - and so $2^0 - 1 = 0$ is the number of internal nodes. So it holds for $b_h(n) = 0$. Now we assume that a node $n$ has a positive height and two children (i.e. $n$ is an internal node). Then each child must have a height of $b_h(n)$ or $b_h(n) - 1$, depending on the colour of the child. We can then apply the inductive hypothesis to the children - then each child has $2^{b_h(n) - 1} - 1$ internal nodes. Therefore we can simply sum the number of internal nodes of the two child subtrees to prove the claim - the subtree rooted at $n$ will have at least $(2^{b_h(n) - 1} - 1) + (2^{b_h(n) - 1} - 1) + 1 = 2^{b_h(n)} - 1$ internal nodes.

With the above combined with red-black tree properties, we can complete the proof: if we have a tree of height $h$, then according to the fourth property of red-black trees, at least half of the nodes on any path from the root to a given leaf (not including the root) are black. So $b_h(r) \geq \frac{h}{2}$, where $r$ is the root node of the tree. Then $n \geq 2^{\frac h 2} - 1$, and $h \leq 2\log_2(n + 1)$ follows.
\end{proof}

\subsubsection{\code{isEmpty()}}
This method performs a single null-check of the root node and is independent of the number of elements in the tree. Thus \code{isEmpty()} runs in constant time O(1).

\subsubsection{\code{contains(Object)}}
This method calls the helper method \code{locate(Node)} which is linear in the height of the tree. \code{locate(Node)} performs a basic iterative binary tree search, moving down the tree and comparing the node with the argument, then moving to the left or right child depending on the result of \code{compareTo}. A binary tree search is linear in the height of the tree (O(h)) and since the height of a red-black tree is guaranteed to be logarithmic, the time complexity of this method is thus O(log n).

\subsubsection{\code{hasPredecessor(Object)} and \code{hasSuccessor(Object)}}
These methods make a single comparison with the current minimum/maximum of the dictionary, to ensure that there is a smaller/larger element in the dictionary. Since the extremum of the dictionary can be accessed in constant time, this comparison is independent of the size of the dictionary, so these methods are O(1).

\subsubsection{\code{predecessor(Object)} and \code{successor(Object)}}
These methods call the internal methods \code{below(Node)} and \code{above(Node)} respectively: I will show that \code{below(Node)} runs in logarithmic time, and then \code{above(Node)} is symmetric to this. \code{below(Node)} starts at the root and then navigates down the tree in a similar fashion to \code{locate(Node)}, generally moving to the right child if the argument is larger than the current node, and moving left if it's less than or equal to the current node. Since it moves strictly down the tree, it can iterate at most $h$ times - therefore being logarithmic.
Once it reaches some point in the tree - (where the right/left child is \code{nil}) then it will, in the worst case, need to return call the \code{predecessor(Node)} method and return the result. Since the \code{predecessor(Node)} method also runs in O(log n) time (it either returns the maximum node of the left subtree or moves up the tree until it finds a node which is a right child), then the total time is on the order of O(log n + log n) = O(log n).

\subsubsection{\code{min()} and \code{max()}}
These two methods run in constant time as fields referencing the current minimum and maximum are kept updated upon insertions and deletions to the dictionary. So \code{min()} and \code{max()} simply access these fields, making them O(1).

\subsubsection{\code{add(Object)}}
\code{add(Object)} first creates a new \code{Node} and calls the \code{insert(Node)} helper method, which is analogous to \code{RB-INSERT(T, z)} in CLRS. A summary of the non-trivial parts of the method follows:
first, the place to insert the new node in the tree is located - this takes at O(h) time since it moves strictly down the tree each iteration, and does constant time work inside the loop. After this is done, the node is made red and then the tree properties are restored using \code{fixInsert(Node)}. \code{fixInsert(Node)} is analogous to \code{RB-INSERT-FIXUP(T, z)} in CLRS. The while loop in \code{fixInsert(Node)} only repeats if the node's uncle is red, and if this occurs we move two levels up (or possibly terminate the loop). Everything inside the while loop takes constant time (in particular note that \code{rotateLeft(Node)} and \code{rotateRight(Node)} do constant work - they have O(1) time complexity). Therefore \code{fixInsert(Node)} is O(h) $\implies$ the whole insertion procedure takes O(log n) time.

Note that if an item is successfully inserted into the dictionary, it is then compared with the minimum and maximum fields to check if we need to update them. This is a constant time procedure and doesn't affect the time complexity of \code{add(Object)}.

\subsubsection{\code{delete(Object)}}
\code{delete(Object)} was also implemented using CLRS as a foundation. Firstly note that the \code{transplant(Node)} method runs in constant time since it simply replaces one subtree as a child of its parent with another subtree by modifying references, and is independent of the size of the tree. With this, we can see that the internal method \code{delete(Node)}, at worst, takes O(log n) time when it has to find the minimum node of the right child's subtree. Then at the end of the method \code{fixDelete(Node)} is called, which also takes O(log n) time: the while loop in the method only repeats when node's sibling is black and both of the sibling's children are black - in which case each iteration we move up the tree by assigning \code{node = node.parent}. This can only happen at most O(h) times, so the overall complexity of \code{delete(Object)} is O(log n).

NB: if the element that was deleted was the minimum or maximum element in the dictionary, then \code{minimum(Node)}/\code{maximum(Node)} will be called with the root as the argument, in order to update the minimum/maximum references. This does not affect the time complexity, as O(2 log n) = O(log n).

\subsubsection{\code{iterator()}}
Firstly: the constructor for the internal class \code{TreeIterator} runs in O(1) time, since it is passed the start node as an argument and stores this reference in a field in preparation for the first call to \code{next()}. Since for this method we start at the minimum element in the dictionary and iterate through to the maximum, and we can access the minimum element in constant time and simply pass it as a parameter to the \code{TreeIterator} constructor, the whole method takes O(1) time.

\subsubsection{\code{iterator(Object)}}
This method, although similar in functionality to \code{iterator()} does \textit{not} take constant time: it first needs to find the least node with key greater than the input argument. The method makes a call to the private method \code{ceiling(Node)}. The method \code{ceiling(Node)} is very similar in functionality to \code{above(Node)}, which is used in \code{successor(Object)} - \code{ceiling(Node)} just has a different edge case, and so it also runs in O(log n).

In addition to the time complexity analysis of returning a new iterator, I will also briefly detail the time complexity of the methods in the \code{TreeIterator} class: clearly \code{hasNext()} takes O(1) since it simply performs a `nil-check'; \code{remove()} takes O(log n) since it calls \code{delete(Node)}, which is shown to be O(log n) above; \code{next()} also takes O(log n) since it calls the \code{successor(Node)} method, and as above \code{successor(Node)} runs in O(log n). Therefore iterating through the dictionary takes time O(n log n).

I expect there is a more efficient way to do iteration - e.g. using a stack and pushing the left subtree onto the stack/pushing right child and its left subtree onto the stack with each call to \code{next()}. Although I had this strategy working for iteration over the full tree, I could not find a way to successfully iterate in this fashion when given some start node that is not the minimum.

\subsubsection{\code{getLogString()}}
\code{getLogString()} retrieves the string value of the \code{StringBuilder} log field and then resets the log. This is independent of the number of elements in the dictionary, therefore running in O(1) time.

\subsubsection{\code{toString()}}
This method was adapted from a \href{http://stackoverflow.com/}{StackOverflow} post \cite{stackoverflow}. The interface method \code{toString()} calls the \code{toString()} method of the root node, which is a recursive method that prints out the node itself and the string representation of its two children. This method will therefore consider each node in the dictionary only once, making the method O(n).

\subsection{Empirical}

This section gives experimental data on the efficiency of the implementation. The efficiency is quantified by the number of comparisons made when performing an operation in the \code{Dictionary} interface.

For demonstration of experimental correctness, refer to the JUnit tests which can be found in the \code{/test/java/DictionaryTest.java} file. JUnit 4.12 was used, and all tests passed at the time of project submission.

Experimental data was obtained by repeatedly running a given method and averaging the result. Each run, the elements are randomly shuffled before being inserted into the dictionary to allow for various internal structures. MATLAB R2014a was used for data visualisation in the figures for \code{add}, \code{delete} and \code{contains}.

For \code{add}, \code{delete} and \code{contains}, plot data was averaged over 100 trials. Each trial recorded the number of comparisons made on dictionaries of size ranging from 0 to 10,000.

Plots of the growth of the number of comparisons for adding, deleting and searching are below. Both linear-linear and log-linear graphs are provided to demonstrate the logarithmic growth of the functions.

For the other methods required by the \code{Dictionary} interface, the average number of comparisons made on a 10,000-element dictionary is given in the table below. These results were averaged over 100 trials. For all tests, a new dictionary was created each time and the 10,000 elements were inserted in random order. This allows for a range of different possible tree structures to even out any possible biases in any individual trial.

\begin{table}[!htbp]
\centering
\begin{tabular}{| l | l | l |}
	\hline
	\textbf{Method} & \multicolumn{2}{c |}{\textbf{Comparisons}} \\ \hline
	& \textbf{Item in dictionary} & \textbf{Item not in dictionary} \\ \hline
	\code{isEmpty()} & \multicolumn{2}{l |}{0.0}  \\ \hline
	\code{contains(Object)} & 12.8 & 13.2 \\ \hline
	\code{hasPredecessor(Object)} & \multicolumn{2}{l |}{1.0} \\ \hline
	\code{hasSuccessor(Object)} & \multicolumn{2}{l |}{1.0}  \\ \hline
	\code{predecessor(Object)} & 14.3 & 13.6 \\ \hline
	\code{successor(Object)} & 14.1 & 14.0 \\ \hline
	\code{min()} & \multicolumn{2}{l |}{0.0}  \\ \hline
	\code{max()} & \multicolumn{2}{l |}{0.0}  \\ \hline
	\code{add(Object)} & 12.8 & 15.3 \\ \hline
	\code{delete(Object)} & 13.4 & 13.1 \\ \hline
	\code{iterator()} & \multicolumn{2}{l |}{0.0} \\ \hline
	\code{iterator(Object)} & 12.9 & 14.1 \\ \hline
	\code{toString()} & \multicolumn{2}{l |}{0.0} \\
	\hline
\end{tabular}
\caption{Mean number of comparisons made on a dictionary of 10,000 elements, averaged over 100 trials and split into two columns depending on whether the argument is in the dictionary or not.}
\end{table}

Table 3 is consistent with the analysis section. The constant time methods take 0 or 1 comparisons. \code{contains(Object)} takes slightly longer if the element is not in the dictionary (since it has to search down the full height of the tree). \code{add(Object)} takes around 2-3 comparisons more if the item is not already in the dictionary, since it also needs to check if it is necessary to update the \code{min}/\code{max} fields. \code{delete(Object)} takes more comparisons if the element is not in the dictionary, for the same reason as \code{contains(Object)}.

\begin{figure}[!htbpp]
    \centering
    \includegraphics[width=0.49\textwidth]{resources/add}
    \includegraphics[width=0.49\textwidth]{resources/add_log}
    \caption{The number of comparisons made when adding a random distinct element to a dictionary of size \textit{n} (linear and log-linear scale)}

\end{figure}

\begin{figure}[!htbp]
    \centering
    \includegraphics[width=0.49\textwidth]{resources/del}
    \includegraphics[width=0.49\textwidth]{resources/del_log}
    \caption{The number of comparisons made when deleting a random element from a dictionary of size \textit{n} (linear and log-linear scale)}
\end{figure}

\begin{figure}[!htbp]
    \centering
    \includegraphics[width=0.49\textwidth]{resources/search}
    \includegraphics[width=0.49\textwidth]{resources/search_log}
    \caption{The number of comparisons made when searching for a random element in a dictionary of size \textit{n} (linear and log-linear scale)}
\end{figure}