\chapter{Reduced memory consumption}

The initial implementation has a couple of problems, one of which is it's memory consumption. This excessive memory consumption is mostly caused by the implementation of the memory preallocation for the matrices \texttt{L}, \texttt{D} and \texttt{U}. Using \texttt{zeros(m,n)} results in full matrices for \texttt{L}, \texttt{D} and \texttt{U}, while they are often far from full.\\

\noindent The painful part is the fact that matrices \texttt{L}, \texttt{D} and \texttt{U} grow dynamically with each iteration of the \texttt{for}-loop. Adding to the problem is that fill-in makes it impossible to know a priory by just how much they grow each iteration. Memory preallocation is a real pig in these type of situations and resorting to a full \texttt{zeros} matrix is often the only way that doesn't involve very complicated or convoluted code.\\

\noindent Dynamic growth of a matrix in a \texttt{for}-loop is evil, mostly because matrices require a contiguous memory block. Adding data to a matrix involves allocating an entirely new contiguous block of memory for the matrix, copying the old values from the previous block to the new, then releasing the old block for potential reuse. These memory operations are time consuming and should be avoided in \texttt{for}-loops. The most common way to deal with this problem is to use memory preallocation, but the other possibility is to use a data structure that drops the contiguous memory range requirement.\\

\noindent With an eye on making the implementation more memory efficient for a variety or reasons, an effort was undertaken to tackle the problem. As stated earlier, preallocation is difficult to achieve. With this in mind, the plan was to NOT use memory preallocation, but instead write the code in such a way that the negative effects of not using memory preallocation are largely avoided. MATLAB has a data structure known as a "cell array"\autocite[]{math_doc_cell}, the most obvious differentiating qualities between it and a matrix data structure are:

\begin{itemize}
    \item Can contain different kind of elements. A collection of an "int", a matrix and a string can all be stored in the same cell array.
    \item No contiguous memory range required. The separate entries of the cell array may require a contiguous memory range, but the collection of the entries might be stored separately.
\end{itemize}

\noindent The most obvious quality that cell arrays and matrices share is their ability to be indexed, something that a "struct" lacks\autocite[]{math_doc_struct}. The improved implementation uses cell arrays to store the additions of the matrices \texttt{L}, \texttt{D} and \texttt{U}, with each element of the cell array containing a COO representation of the \texttt{for}-loop additions\autocite[]{wiki_coo}.\\

\section{The code}

\lstinputlisting{code/MLDU_Simple_cell.m}

\section{Discussion of the code}

This implementation creates one new nested function, two new local functions and increases the number of lines from $47$ to $91$, so the reduced memory consumption does come at the cost of code complexity.

\newpage

\section{Profiler results}

\noindent The profiler results below are generated by running:\\

\noindent \texttt{[ E ] = Test\_Function\_1( 100, @MLDU\_Simple\_cell )}\\

\begin{figure}[h!]
    \includegraphics[width=\linewidth]{figures/Profile_MLDU_Simple_cell_1.eps}
    \centering
\end{figure}

\noindent This is exactly the same as the previous situation. The error is also acceptable:\\

\noindent \texttt{E =}\\
\\
\noindent \texttt{   1.1287e-13}\\

\noindent Apart from the targeted result that the memory consumption has decreased, a couple of other noteworthy positive changes occurred as well.\\

\noindent One of the positive aspect is that calculating the error has a vastly reduced runtime, from about $10$ seconds to just under one second. This is caused by the fact that matrices \texttt{L}, \texttt{D} and \texttt{U} are now sparse, making the calculation a lot faster.

\begin{figure}[h!]
    \includegraphics[width=\linewidth]{figures/Profile_MLDU_Simple_cell_2.eps}
    \centering
\end{figure}

\noindent The runtime required to save the matrices \texttt{L}, \texttt{D} and \texttt{U} has actually decreased. The initial implementation spends $2.243$ seconds at lines $19$ through $21$ while the cell array implementation takes a total of $1.187$ seconds at lines $19, 20, 21$ and $33, 34, 35$.\\

\noindent Lastly and maybe the most significant side effect of the cell array implementation is the runtime reduction of line $26$, from about $17$ seconds to about $13$ seconds, a reduction of about $25 \%$. This is probably caused by the reduced memory consumption itself, allowing the cache of the CPU to operate more effectively.


\theoremstyle{plain}
\newtheorem{Theorem}{Theorem}[section]
\newtheorem{Lemma}[Theorem]{Lemma}
\newtheorem{Corollary}[Theorem]{Corollary}
\newtheorem*{With Space}{With Space}
\newtheorem{Proposition}{Proposition}[section]
\newtheorem{Conjecture}[Proposition]{Conjecture}
\newtheorem{WithoutSpace}{WithoutSpace}[section]
\newtheorem{KL}{Klein’s Lemma}[section]
\theoremstyle{definition}
\newtheorem{Definition}{Definition}[section]
\theoremstyle{remark}
\newtheorem{Case}{Case}[section]

\hypertarget{demo}{%
\chapter{Demo}\label{demo}}

\begin{Theorem}[within parenthesis]
plain theoremstyle \emph{here}

We can use pandoc-crossref style \ref{simplestEquation} and
\eqref{simplestEquation} and cite before definition.
\end{Theorem}

\begin{Theorem}\label{simplestEquation}
\leavevmode\vadjust pre{\hypertarget{simplestEquation}{}}%
Label and reference:

\[E=mc^2\]
\end{Theorem}

From the \ref{simplestEquation}, we see that\ldots{} Or
\eqref{simplestEquation}, \ldots{}

\begin{With Space}[\textbf{This} is \emph{markdown}.]
Environment name has a space, and is unnumbered.
\end{With Space}

\begin{Lemma}[can cite \ref{simplestEquation}]
This one share counter with Theorem.
\end{Lemma}

\begin{Definition}[pandoc-crossref style cite \ref{simplestEquation}]
definition theoremstyle here
\end{Definition}

\begin{Case}[within parenthesis]
remark theoremstyle here
\end{Case}

\begin{proof}[Proof of the Main Theorem]
Predefined proof theoremstyle here
\end{proof}

\begin{proof}[Proof of the \emph{little} theorem]
Predefined proof theoremstyle here with markdown info.
\end{proof}

\begin{proof}
Bare proof here.
\end{proof}

\begin{KL}
Klein's Lemma from amsthm doc.
\end{KL}

\begin{Definition}
\begin{verbatim}
code here
\end{verbatim}
\end{Definition}

\hypertarget{counter-test}{%
\chapter{Counter test}\label{counter-test}}

\begin{Theorem}
some theorem
\end{Theorem}

\begin{Theorem}
some theorem
\end{Theorem}

\hypertarget{next-level}{%
\section{Next level}\label{next-level}}

\begin{Theorem}
some theorem
\end{Theorem}

\begin{Theorem}
some theorem
\end{Theorem}

\hypertarget{level-3}{%
\subsection{Level 3}\label{level-3}}

\begin{Theorem}
some theorem
\end{Theorem}

\begin{Theorem}
some theorem
\end{Theorem}

\hypertarget{level-4}{%
\subsubsection{Level 4}\label{level-4}}

\begin{Theorem}
some theorem
\end{Theorem}

\begin{Theorem}
some theorem
\end{Theorem}

\hypertarget{level-5}{%
\paragraph{Level 5}\label{level-5}}

\begin{Theorem}
some theorem
\end{Theorem}

\begin{Theorem}
some theorem
\end{Theorem}

\hypertarget{level-6}{%
\subparagraph{Level 6}\label{level-6}}

\begin{Theorem}
some theorem
\end{Theorem}

\begin{Theorem}
some theorem
\end{Theorem}


% When using TeXShop on the Mac, let it know the root document. The following must be one of the first 20 lines.
% !TEX root = ../design.tex

\chapter[Generalized Linear Models]{Generalized Linear Models}

\begin{moduleinfo}
\item[Author] \href{mailto:lpei@gopivotal.com}{Liquan Pei}
\item[Author] \href{mailto:lhuang@pivotal.io}{Lei Huang}
\item[History]
    \begin{modulehistory}
        \item[v0.1] Initial version
        \item[v0.2] Extention to multivariate response and ordinal response case
    \end{modulehistory}
\end{moduleinfo}

% Abstract. What is the problem we want to solve?

\section{Introduction}

Linear regression model assumes that the dependent variable $Y$ is equal to a linear combination $\vec{X}^\top \vec{\beta}$ and a normally distributed error term
\begin{align*}
Y = \vec{X}^\top \vec{\beta} + \epsilon
\end{align*}
where $\vec{\beta} = (\beta_{1}, \dots, \beta_{m})^\top$ is a vector of unknown parameters and $\vec{X} = (X_1, \dots, X_{m})^\top$ is a vector of independent variables.

In a generalized linear model (GLM), the distribution of dependent variable $Y$ is a member from the \emph{exponential family} and the mean $\mu = \mathbf{E}(Y)$ depends on the independent variables $\vec{X}$ through
\begin{align*}
\mu = \mathbf{E}(Y) = g^{-1}(\eta) =g^{-1}(\vec{X}^\top \beta)
\end{align*}
where $\eta = \vec{X}^\top \beta$ is the \emph{linear predictor} and $g$ is the \emph{link function}.

In what follows, we denote $G(\eta) = g^{-1}(\eta)$ as the inverse link function.

\subsection{Exponential Family}
A random variable $Y$ is a member from the exponantial family if its probability function or its density function has the form
\begin{align*}
f(y,\theta, \psi) = \exp \left\{ \frac{y\theta - b(\theta)}{a(\psi)} + c(y, \psi)\right\}
\end{align*}
where $\theta$ is the canonical parameter.
The mean and variance of the exponential family are
\begin{itemize}
\item $\mathbf{E}(Y) = \mu = b'(\theta)$
\item $\mathbf{Var}(Y) = V(\mu)a(\psi) = b''(\theta)a(\psi)$
\end{itemize}

\subsection{Linear Predictor}
The linear predictor $\eta$ incorporates the information about the independent variables into the model. It is related to the expected value of the data through link functions.

\subsection{Link Function}
The link function provides the relationship between $\eta$, the linear predictor and $\mu$, the mean of the distribution function. There are many commonly used link functions, and their choice can be somewhat arbitrary. It makes sense to try to match the domain of the link function to the range of the distribution function's mean.

For canonical parameter $\theta$, the canonical link function is the function that expresses $\theta$ in terms of $\eta = g(\mu)$. In what follows we treat $\theta = \theta(\eta) = h(g(\mu))$. If we choose $h$ to be an identical function, then $\theta = \eta$
and $\mu = G(\eta) = \mu(\theta)$.

\section{Parameter Estimation}
We estimate unknown parameters $\beta$ by maximizing the log-likelihood of a GLM. Given the examples $\vec{Y} = (Y_1, \dots, Y_n)^\top$ and denote their mean as $\vec{\mu} = (\mu_1, \dots, \mu_{n})^\top$, the log-likelihood for a GLM is

\begin{align*}
l(\vec{Y}, \vec{\mu}, \psi) & =\sum_{i=1}^{n} \log f(Y_i, \theta_{i}, \psi) \\
& =\sum_{i=1}^{n}\left\{ \frac{Y_i \theta_{i} - b\left(\theta_{i}\right)}{a\left(\psi\right)} - c(Y_i, \psi)\right\}
\end{align*}
where $\theta_{i} = \theta(\eta_{i}) = \theta(x_i^\top \beta)$

Note that $a(\psi)$ and $c(Y_i, \psi)$ dose not depend on $\beta$, we then maximize
\begin{align}
\label{likelihood}
\tilde{l}(\vec{Y}, \vec{\mu}) = \sum_{i = 1}^{n} \left\{ Y_i \theta_{i} - b(\theta_{i}) \right\}
\end{align}
with respect to $\vec{\beta}$.
In what follows, we denote $\vec{x_i}$ to be the vector of values of independent variables for $Y_i$.
\subsection{Iterative Reweighted Least Squares Algorithm}
We use iterative reweighted least squares (IRLS) algorithm to find $\vec{\beta}$ that maximize $\tilde{l}(\vec{Y},\vec{\mu})$. Specifically, we use Fisher scoring algorithm which updates $\vec{\beta}$ at step $k+1$ using
\begin{align*}
\vec{\beta}^{k+1} = \vec{\beta}^{k} + \left\{\mathbf{E}[H(\beta^{k})]\right\}^{-1} \nabla_{\vec{\beta}}\tilde{l}(\vec{\beta}^{k})
\end{align*}
where $\mathbf{E}[H]$ is the mean of Hessian over examples $\vec{Y}$ and $\nabla_{\vec{\beta}}\tilde{l}$ is the gradient.
For GLM, the gradient is
\begin{align*}
\nabla_{\vec{\beta}}\tilde{l} &= \sum_{i = 1}^{n}\left\{Y_i - b'(\theta_{i})\right\}\nabla_{\vec{\beta}}\theta_{i} \\
\end{align*}
Note that $\mu_{i} = G(\eta_{i}) = G(\vec{x_i}^\top \vec{\beta}) = b'(\theta_{i})$, we have
\begin{align*}
\nabla_{\vec{\beta}}\theta_{i} &= \frac{G'(\eta_{i})}{V(\mu_{i})} \vec{x_i}
\end{align*}
%and
%\begin{align*}
%\frac{\partial^2 \theta_{i}}{\partial \beta \partial \beta^{T}} = \frac{G''(\eta_{i})V(\mu_{i}) - G'(\eta_{i})^2 V'(\mu_i)}{V(\mu_{i})^2}x_i x_i^T
%\end{align*}
then
\begin{align*}
\nabla_{\vec{\beta}}\tilde{l} &= \sum_{i = 1}^{n}\left\{Y_i - \mu_{i}\right\}\frac{G'(\eta_{i})}{V(\mu_{i})}\vec{x_i}
\end{align*}
The Hessian is
\begin{align*}
H(\beta) &= \sum_{i = 1}^{n} \left\{-b''(\theta_{i}) \nabla_{\vec{\beta}}\theta_{i} {\nabla_{\vec{\beta}}\theta_{i}}^\top - \left\{Y_i - b'(\theta_{i})\right\}\nabla_{\vec{\beta}}^2 \theta_{i}\right\} \\
& = \sum_{i = 1}^{n} \left\{ \frac{G'(\eta_{i})^2}{V(\mu_{i})} - \left\{Y_i - \mu_{i}\right\}\nabla_{\vec{\beta}}^2 \theta_{i}\right\}\vec{x_i} \vec{x_i}^\top
\end{align*}
Note that $\mathbf{E}[Y_i] = \mu_{i}$, we have
\begin{align*}
\mathbf{E}[H(\beta)] = \sum_{i = 1}^{n} \left\{\frac{G'(\eta_{i})^2}{V(\mu_{i})}\right\}\vec{x_i} \vec{x_i}^\top
\end{align*}
Define the weight matrix
\begin{align*}
\vec{W} = \text{diag}\left( \frac{G'(\eta_{1})^2}{V(\mu_{1})}, \dots, \frac{G'(\eta_{n})^2}{V(\mu_{n})} \right)
\end{align*}
and define
\begin{align*}
\vec{\tilde{Y}} = \left(\frac{Y_1 - \mu_1}{G'(\eta_{1})}, \dots, \frac{Y_n - \mu_{n}}{G'(\eta_{n})}\right)^\top
\end{align*}
and the design matrix
\begin{align*}
\vec{X}^\top = \left(\vec{x_1}, \dots, \vec{x_n}\right)
\end{align*}
Finally, the update rule for GLM is
\begin{align*}
\vec{\beta}^{k+1}  &= \vec{\beta}^{k} + (\vec{X}^\top \vec{W} \vec{X})^{-1}\vec{X}^\top \vec{W}\tilde{\vec{Y}} \\
&= (\vec{X}^\top \vec{W} \vec{X})^{-1}\vec{X^\top WZ}
\end{align*}
where  $\vec{Z} = (Z_1, \dots, Z_n)$ is a vector of \emph{adjusted dependent variables}
\begin{align*}
Z_i = \vec{x_i}^\top \vec{\beta}^{k} + \frac{Y_i - \mu_{i}}{G'(\eta_{i})}
\end{align*}
Note that each step is the result of a weighted least square regression on the adjusted variables $Z_i$ on $x_i$ and this the reason that this algorithm is called iterative reweighted least squares.

The IRLS algorithm for GLM is as follows
\begin{algorithm}
\alginput{$\vec{X}$, $\vec{Y}$, inverse link function $G(\eta)$, dispersion function $V(\mu)$ and initial values $\vec{\beta}^0$}
\algoutput{$\vec{\beta}$ that maximize $\tilde{l}(\vec{Y}, \vec{\mu})$}
\begin{algorithmic}[1]
    \State $k \leftarrow 0$
    \Repeat
        \State Compute $\vec{\mu}$ where $\mu_{i} = G(\eta_{i}) = G(\vec{x_i}^\top \vec{\beta}^{k})$
        \State Compute $\vec{Z}$ where $Z_i = \vec{x_i}^\top \vec{\beta}^{k} + \frac{Y_i - \mu_{i}}{G'(\eta_{i})}$
        \State Compute $\vec{W}$ where $W_{ii} = \frac{G'(\eta_{i})^2}{V(\mu_{i})}$
        \State $\vec{\beta}^{k+1} = \vec{(X^\top W X)}^{-1} \vec{X^\top WZ}$
    \Until{$\vec{\beta}^{k+1}$ converges}
\end{algorithmic}
\label{alg:IRLS}
\end{algorithm}

\subsection{Functions for contructing the exponential families}
Table \ref{tab:glm_func} \cite{fox2008applied} provides functions $a(), b()$ and $c()$ to contruction the exponential families,

\begin{table}[h]
\centering
\begin{tabular}{cccc}
Family & $a(\psi)$ & $b(\theta)$ & $\c(y, \psi)$ \\
\hline
Gaussian & $\psi$ & $\theta^2/2$ & $-\frac{1}{2}\left[y^2/\psi+\log_e(2\pi\psi)\right]$ \\
Binomial & 1/n & $\log_e(1+e^\theta)$ & $\log_eC^n_{ny}$ \\
Poisson & 1 & $e^\theta$ & $-log_ey!$ \\
Gamma & $\psi$ & $-\log_e(-\theta)$ & $\psi^{-1}\log_e(y/\psi) - \log_ey - \log_e\Gamma(\psi^{-1})$ \\
Inverse-Gaussian & $-\psi$ & $\sqrt{2\theta}$ & $-\frac{1}{2}\left[\log_e(\pi\psi y^3)^3 + 1/(\psi y) \right]$ \\
\end{tabular}
\caption{Functions for constructing the exponential families}
\label{tab:glm_func}
\end{table}

\subsection{Multivariate response family}
\label{subsec::multinomreg}
Instead of a single scalar number, some response variable follows a multivariate distribution (i.e. the response variable is a vector instead of a scalar number). One example is multinomial GLM where the response variable is an indictor vector containing zeros and ones to dummy code the corresponding categories. For illustration purpose, in this section, we are discussing multinomial GLM. However, other distributions can be easily extended.

Let $J$ denote the number of categories, $y_i=(y_{i1}, y_{i2}, ..., y_{i(J-1)})^T$ be the indicator vector for $i$th subject where each $y_{ij}$ is the binary indicator whether subject $i$ is in categories $j$, $\mu_{ij}$ will be the probabilty subject $i$ is in category $j$. Therefore, we can have the log likelihood as below,

\begin{align*}
l & = \sum_{i=1}^{I} \left(\sum_{j=1}^{J-1} y_{ij} \log{\frac{\mu_{ij}}{1-\sum_{j=1}^{J}\mu_{ij}}} + \log(1-\sum_{j=1}^{J-1}) \right)\\
 & = \sum_{i=1}^I \left( \sum_{j=1}^{J-1} y_{ij} \theta_{ij} - \log(1+\sum_{j=1}^{J-1}\exp\theta_{ij}) \right)
\end{align*}

Define $b(\theta_i) = \log(1+\sum_{j=1}^{J-1}\exp\theta_{ij})$, then it can be showed $\triangledown b(\theta_i) = \mu_i$ and
\[
\triangledown \triangledown^T b(\theta_i) = \left( \begin{array}{cccc}
\mu_{i1}(1-\mu_{i1}) & -\mu_{i1}\mu_{i2} & ... & -\mu_{i1}\mu_{i(J-1)} \\
-\mu_{i2}\mu_{i1} & \mu_{i2}(1-\mu_{i2}) & ... & -\mu_{i2}\mu_{i(J-1)} \\
\vdots & \vdots & \vdots & \vdots \\
-\mu_{i(J-1)}\mu_{i1} & -\mu_{i(J-1)}\mu_{i2} & ... & \mu_{i(J-1)}\mu_{i(J-1)}
\end{array} \right)
\]
We set $V = \triangledown \triangledown^T b(\theta_i)$

Let $\eta_{ij} = g_j(\mu_i)$ and $g() = (g_1(), g_2(), ..., g_{J-1}())^T$ be the link map, which is $\Re^{J-1}$ to $\Re^{J-1}$. Also we have $\mu_i = G(\eta_i)$ be its inverse map. We define the derivative of $G$ to be $G^\prime = \left(\frac{\partial\mu_i}{\partial\eta_{i1}}, \frac{\partial\mu_i}{\partial\eta_{i2}}, ..., \frac{\partial\mu_i}{\partial\eta_{i(J-1)}} \right)$. For example, in multinomial logistic regression, $\eta_{ij} = \theta_{ij}$, then $G^\prime = V$.

Denote the coefficient to be $\beta_{kj}$ where $k$ stands for the $k$th predictor and $j$ stands for the $j$th category. Then we have
\begin{align*}
\frac{\partial l_i}{\partial \beta_{kj}} & = (y_i - \triangledown b(\theta_i) )^T \frac{\partial \theta_i}{\partial \mu_i} \frac{\partial \mu_i}{\partial \eta_{ij}} \frac{\partial \eta_{ij}}{\partial \beta_{kj}} \\
& = (y_i - \mu_i )^T V^{-1} G^\prime_j x_{ik}
\end{align*}
where $G^\prime_j$ is the $j$th column of $G^\prime$.
\begin{align*}
\frac{\partial^2 l_i}{\partial\beta_{kj} \partial\beta_{lh}} &= -x_{il} (G^\prime_h)^T V^{-1} \triangledown \triangledown^T b(\theta_i) V^{-1} G^\prime_j x_{ik} \\
& = -x_{il} (G^\prime_h)^T V^{-1} G^\prime_j x_{ik}
\end{align*}

As a entire vector $\beta$,
\begin{align*}
\triangledown_\beta l_i &= \left((y_i - \mu_i)^T V^{-1} G^\prime(\mu_i) \right)^T \otimes X_i \\
E\left[\triangledown \triangledown^T_\beta l_i\right] & = - \left( [G^\prime(\mu_i)]^T V^{-1} [G^\prime(\mu_i)] \right) \otimes (X_i X_i^T)
\end{align*}
where $X_i = (x_{i1}, x_{i2}, ... , x_{ip})^T$.

Finally, we can use the below update equation for Newton-Raphson method,
\[
\beta^{k+1} = \beta^k + \left\{ \sum_{i=1}^I\left([G^\prime(\mu_i)]^T V^{-1} [G^\prime(\mu_i)] \right) \otimes (X_i X_i^T)\right\}^{-1}\sum_{i=1}^I\left\{\left((y_i - \mu_i)^T V^{-1} G^\prime(\mu_i) \right)^T \otimes X_i\right\}
\]

\subsection{Ordinal logistic regression}
In statistics, the ordered logit model (also ordered logistic regression or proportional odds model), is a regression model for ordinal dependent variables. For example, if one question on a survey is to be answered by a choice among "poor", "fair", "good", "very good", and "excellent", and the purpose of the analysis is to see how well that response can be predicted by the responses to other questions, some of which may be quantitative, then ordered logistic regression may be used. It can be thought of as an extension of the logistic regression model that applies to dichotomous dependent variables, allowing for more than two (ordered) response categories.

The model we implement here only applies to data that meet the proportional odds assumption, the meaning of which can be exemplified as follows.

\[
\log\left( \frac{\Pr(Y_i \le j)}{1-\Pr(Y_i \le j)} \right) = \alpha_j - \beta_1 x_{i1} - \beta_2 x_{i2} - ... - \beta_p x_{ip}
\]
where $\Pr(Y_i \le j)$ is the cumulative probability that the $i$th subject belongs to the first $j$th categories; $\alpha_j$ is category-specific intercept and $\beta_k$ is feature-specific coefficient. Using the notation in the subsection \ref{subsec::multinomreg}, the link function is
\[
g\left([\mu_1, \mu_2, ..., \mu_{J-1}]^T\right) = \left[\log\left(\frac{\mu_1}{1-\mu_1}\right), \log\left(\frac{\mu_1+\mu_2}{1-\mu_1-\mu_2}\right), ..., \log\left(\frac{\mu_1+\mu_2+...+\mu_{J-1}}{1-\mu_1-\mu_2-...-\mu_{J-1}}\right)\right]^T
\]
Then the inverse of link function G is,
\begin{align*}
G\left([\eta_1, \eta_2, ..., \eta_{J-1}]^T\right) & = \left[\frac{\exp(\eta_1)}{1+\exp(\eta_1)}, \frac{\exp(\eta_2)}{1+\exp(\eta_2)} - \frac{\exp(\eta_1)}{1+\exp(\eta_1)},\right. \\
						  &    \left. ..., \frac{\exp(\eta_{J-1})}{1+\exp(\eta_{J-1})} - \frac{\exp(\eta_{J-2})}{1+\exp(\eta_{J-2})}\right]^T
\end{align*}
Its derivative matrix $G^\prime$ is
\begin{align*}
G^\prime & = \left[\frac{\partial \mu}{\partial \eta_1}, \frac{\partial \mu}{\partial \eta_2}, ..., \frac{\partial \mu}{\partial \eta_{J-1}}\right] \\
& = \left[ \begin{array}{ccccc}
	      \frac{\exp(\eta_1)}{(1+\exp(\eta_1))^2} & 0 & 0 & ... & 0 \\
	     -\frac{\exp(\eta_1)}{(1+\exp(\eta_1))^2} & \frac{\exp(\eta_2)}{(1+\exp(\eta_2))^2} & 0 & ... & 0 \\
	     0 & -\frac{\exp(\eta_2)}{(1+\exp(\eta_2))^2} & \frac{\exp(\eta_3)}{(1+\exp(\eta_3))^2} & ... & 0 \\
	     \vdots & \vdots & \vdots & ... & \vdots \\
	     0 & 0 & 0 & ... & \frac{\exp(\eta_{J-1})}{(1+\exp(\eta_{J-1}))^2}
	   \end{array} \right]
\end{align*}

Define $h_i = \sum_{j=1}^{J-1} \frac{\partial \mu_i}{\partial \eta_{ij}}$, then as a entire coefficient vector $\gamma = [\alpha_1, \alpha_2, ..., \alpha_{J-1}, \beta_1, \beta_2, ..., \beta_p]^T$,
\begin{align*}
\triangledown_\gamma l_i &= \left[ \begin{array}{c}
					\left\{(y_i - \mu_i)^T V^{-1} G_i^{\prime} \right\}^T \\
					-(y_i - \mu_i)^T V^{-1} h_i X_i
				   \end{array} \right] \\
E\left[\triangledown \triangledown^T_\gamma l_i\right] & = - \left[ \begin{array}{cc}
							(G_i^\prime)^T V^{-1} G_i^\prime & -(G_i^\prime)^T V^{-1} h_i X_i^T \\
							-X_i h_i^T V^{-1} G_i & h_i^T V^{-1} h_i X_iX_i^T
						      \end{array}
						\right]
\end{align*}
where $X_i = (x_{i1}, x_{i2}, ... , x_{ip})^T$, $G_i^\prime = G^\prime(\eta_{i1},\eta_{i2},...,\eta_{iJ-1})$. We can then implement Newton-Raphson method using above gradient and Hessian matrix.

\subsection{Ordinal probit model}
In the ordinal probit model, instead of logistic link function used in ordinal logistic regression, the probit function $\Phi(x)$ is used. Therefore the model becomes,
\[
\Phi^{-1}\left( \Pr(Y_i \le j) \right) = \alpha_j - \beta_1 x_{i1} - \beta_2 x_{i2} - ... - \beta_p x_{ip}
\]
and the link function is
\[
g\left([\mu_1, \mu_2, ..., \mu_{J-1}]^T\right) = \left[\Phi^{-1}(\mu_1), \Phi^{-1}(\mu_1+\mu_2), ..., \Phi^{-1}(\mu_1+\mu_2+...+\mu_{J-1})\right]^T
\]
Then the inverse of link function G is,
\[
G\left([\eta_1, \eta_2, ..., \eta_{J-1}]^T\right)  = \left[\Phi(\eta_1), \Phi(\eta_2) - \Phi(\eta_1), ..., \Phi(\eta_{J-1}) - \Phi(\eta_{J-2})\right]^T
\]
Its derivative matrix $G^\prime$ is
\begin{align*}
G^\prime & = \left[\frac{\partial \mu}{\partial \eta_1}, \frac{\partial \mu}{\partial \eta_2}, ..., \frac{\partial \mu}{\partial \eta_{J-1}}\right] \\
& = \left[ \begin{array}{ccccc}
	      \phi(\eta_1) & 0 & 0 & ... & 0 \\
	     -\phi(\eta_1) & \phi(\eta_2) & 0 & ... & 0 \\
	     0 & -\phi(\eta_2) & \phi(\eta_3) & ... & 0 \\
	     \vdots & \vdots & \vdots & ... & \vdots \\
	     0 & 0 & 0 & ... & \phi(\eta_{J-1})
	   \end{array} \right]
\end{align*}
where $\Phi(x)$ and $\phi(x)$ are the cumulative probability and density probability of standard normal distribution.


\documentclass[]{article}

\usepackage[]{graphicx}

\title{CS 636 Assignment 4: Image Classification}

\author{Derek Jones}

\begin{document}

\maketitle

\begin{abstract}
	I train a random forest and a logistic regression classifier on a random sample of 500 example images from each of the classes. I chose to try two methods in order to obtain extra credit for the assignment as well as get an idea of the performance of each with respect to the parameter of interest in my experiments, feature type while I use $k$-fold cross validation to asses how well each model can generalize.. My results show that the bag of words features, used in field such as NLP and computer vision, are suboptimal compared to the features generated by the AlexNet deep convolutional neural network. Modern advances in image classification make better use of raw image data, generating more useful features as a byproduct.
\end{abstract}

\section{Bag of Words}

For the bag of words implementation, I aggregate a random subset of the surf features (throwing out examples in which the number of features is less than a threshold specified as a parameter in the make\_codebook() function) and then perform k-means (50 clusters, maximum of 300 iterations) to solve for the code book. Then for each of the training examples, I iterate over each surf feature and predict the cluster that the given feature belongs to. I form a histogram over all of the possible clusters, of which there are 50 total as specified in the assignment, then normalize this (as the number of features varies between examples, normalization makes this less relevant) and use this as my feature vector.

\subsection{Random Forest}

 I train a random forest classifier which is provided via the scikit-learn library with $n = 100$ decision stumps, using default settings for all other parameters. I choose this number of trees in order to decrease computation time, I've also found limited marginal gain by increasing the size of the forest (the largest I've tried is $n=100$ decision stumps). My results give a $k$-fold cross validation (on the training set, $k = 5$) average accuracy of 34.7\%. The random forest is also able to perfectly classify the full training set of 500 examples. Finally the test accuracy of this method is rather disappointing at $5.0$\%. The confusion matrix is given below:

\includegraphics[scale=0.10]{bow_random_forest_confusion_results.jpg}

\subsection{Logistic Regression (Extra Credit)}

I train a logistic regression classifier provided via the scikit-learn library with default settings. My results give a $k$-fold cross validaton (on the training set, $k$ =5) average accuracy of 27\%. The logistic regression also gets about a 25\% training accuracy on the full training set. Finally, the test accuracy of this method is $1.6$\% accuracy. The confusion matrix is given below:

\includegraphics[scale=0.10]{bow_logistic_confusion_results.jpg}

\section{Alexnet}
Foor the alexnet implementation, I simply use the provided feature vector which is the output of a convolutional layer in the network.

\subsection{Random Forest}
I train a random forest with $n = 500$ decision stumps. The random forest achieves an average cross validation accuracy of $71$\% accuracy, while perfectly classifying the full training set, and achieves about $33$\% accuracy on the test set, which is vastly more impressive than the bag of words counterpart (although I did use a large number of estimators).

\includegraphics[scale=0.10]{alex_net_random_forest_confusion_results.jpg}

\subsection{Logistic Regression(Extra Credit)}
I train a logistic regression classifier with default parameters. The logistic classifier achieves an average cross validation accuracy of 81\% accuracy, perfectly classifies the full training set, and achieves a testing accuracy of approximately 57.6\% accuracy, by far the best result.

\includegraphics[scale=0.10]{alex_net_logistic_confusion_results.jpg}

\section{Dilated Convolutional Network: AlexNet(Extra Extra Credit)}
Finally, for extra extra credit, I use Keras to implement a convolutional network on top of the alexNet features with a 1D dilated convolution layer followed by a 1D convolution layer (no dilation) followed by a dropout layer, then finally with a fully connected softmax layer. I use tanh activations for the convolution layers. I use categorical cross entropy as my loss and ADAM with a learning rate of $\alpha = 1e-5$. I train using 5235 images as my training set and 1047 as holdout test during my training. I train the network for 1000 epochs.  My results for this implementation vastly outperform the previously mentioned methods with a test accuracy of approximately 85\%.

\includegraphics[scale=0.10]{deep_dilated_conv_alex_net_confusion_results.jpg}

\section{Conclusions}
Overall, I found the combination of a logistic regression classifier trained with AlexNet features to be the most accurate predictor of the image class. AlexNet is able to generate informative features that are perhaps better at incorporating global information better than the bag of words features, which is a local feature aggregation method. Moreover, logistic regession is a powerful classification tool that can fit many non-linear decision problems and be quite successful.

However with modern deep learning methods, one can easily take the output of a layer in a more powerful network and use as input to another network in order to leverage the features for a different task or to add more complexity without needing to retrain the part of the network that does the heavy lifting in terms of extracting good features. My final implementation of convolutional network with dilated convolution layer vastly outperforms each of my previous methods by a large margin.

\end{document}


%!TeX encoding=utf8

\section{Bose Einstein Condensate}

\subsection{Questions}
\begin{itemize}
        \item $\int \frac{1}{r^{n}} \mathrm{d^{3}r}$ divergent just for $n \leq 3 $ ?
        \item What is an s-wave?
        \item if $l < \frac{n-3}{2}$,
        \item and like $k^{n-2}$ otherwise (Landau and Lishitz, 1977). For a van der
        Waals-like potential $(n = 6)$, only $l = 0$ (s-wave) matters at low energies. Whats with $l=0$? Lifschitz is a typo?
        \item In the notes: eq. 1.8 to 1.9 the commutator $[\psi, \psi^{\dagger}]$ was used, but from 1.9 to 1.10 Bogoliubov aproximation was used, why not directly on 1.8 ?
        \item What is the derivation of 1.11? Its a FFT, but what are the xact steps?
        \item What are the python packages to use operators like $\hat{\psi}$? There is some sympy implementation, but probably there is a better one?
        \item how to get from 1.11 to 1.12? Is it a commutator expansion? Why $q$ is disappearing?
        \item In 1.17 $\omega_{\rho}$ part has a factor 2, but $\omega_{z}$ not, despite beeing symmetric in $\psi$. Why?
        \item What is variable $a$? Why should $a > 0$ as repulsive short-range interactions stabilize the BEC (p.10)?
        \item ``When the atomic density grows due to the attractive interaction, three-body losses predominantly occur in the high-density region. '' What does three-body losses mean?
        \item ``As the collapse occurs mainly in the x-y direction due to anisotropy of the DDI (in the absence of inelastic losses, the condensate would indeed become an infinitely thin cigar-shaped cloud along z),
        and therefore the condensate explodes essentially radially, producing the anisotropic
        shape of the cloud.'' Why is the collaps not along z axis?
        \item How are the regions stable, metastable, unstable derived in Figure 1.5, here Figure \ref{fig:acrit}?
\end{itemize}

\begin{figure}[H]
    \centering
    \includegraphics[width=0.7\textwidth]{IMAGE/acrit.png}\\
    \caption{Logo}
    \textsc{Santos}, \emph{title} (year)
    \label{fig:acrit}
\end{figure}

\begin{itemize}
        \item Typo in ``we obtain a 1D equation similar to the a GP equation'', just the or a
        \item ``ground-state wave-function is independent of the in-plane coordinates '' Why?
        \item 1.26 to 1.27, where does the $U_{dd}$ go?
        \item Typo If: ``roton momentum. if this were so''
        \item Why should a modulation with a finite wavelength allow superfluids?
        \item Typo repeatance: ``the width of the width''
        \item What are the spin-F matrizes?
        \item Is the occurance of these spin textures in Figure \ref{fig:helical} special?
\end{itemize}

\begin{figure}[H]
    \centering
    \includegraphics[width=0.7\textwidth]{IMAGE/helical.png}\\
    \caption{Is the occurance of these textures special?}
    \textsc{Santos}, \emph{title} (year)
    \label{fig:helical}
\end{figure}

\subsection{Summary}
\begin{itemize}
        \item dipol-dipol interaction (DDI): \begin{equation}
          U(r) = \underbrace{g \delta (r)}_{\frac{4 \pi \hbar^{2} a(d) \delta(r)}{m}} + \underbrace{U_{dd}(r)}_{\frac{C_{dd}}{4 \pi} \frac{(e_{1} \cdot e_{2}) r^{2} - 3(e_{1} \cdot r) (e_{2} \cdot r)}{r^{5}}}
        \end{equation}
        \item Use pseudo potential as dipol-dipol interaction is anisotropic and all partial wave (different l) mix
        \item coupling of different channels generates short-range contribution in the s-channel $s=0$ $\Rightarrow$ by changing DDI strength $a$ gets modified too $\Rightarrow$ shape resonances $\Rightarrow$ virtual state transform into a new ground state
        \item for fermions s-channel does not exists, so just long-range
        \item FFT of $U_{dd}$ using sperical harmonics $Y_{lm}$ gives:
        \begin{equation}
          \tilde{U_{dd}}(k) = \int \mathrm{d^{r} r} U_{dd}(r) e^{-ik \cdot r}= \frac{C_{dd}}{3}\left(3 \cos^{2}(\theta_{k}) - 1 \right)
        \end{equation}
        \item Use DDI in Gross-Pitajevski Equation, FFT, approximate to 2nd order, diagonalize with Bogoliubov transform
        \item As a result the square root can be imaginary, so the BEC gets dynamically unstable for long-wave length (phonon-instability):
            \begin{align}
              \epsilon(p) &= \sqrt{\frac{p^{2}}{2m} \left[\frac{p^{2}}{2m} + 2 n_{0} \left( g + \tilde{U_{dd}(p)} \right)\right]} \\
              &= p c_{s} \sqrt{1 + \epsilon_{dd} \left( 3 \cos^{2} \theta_{p} - 1 \right)} \\
              &\underset{p \rightarrow 0}{=} p c_{s} \sqrt{1 - \epsilon_{dd}}
            \end{align}
        \item For dipolar BEC the trap geometry is crucial (for non-dipolar not)
        \item ``pancake traps'' can stabilize the phonon-instability
        \item qualitative features for $a_{crit}(\lambda)$ by gaussian ansatz, for exact numerical solution non-local Gross-Pitaevskii Equation needed
\end{itemize}

\begin{figure}[H]
    \centering
    \includegraphics[width=0.7\textwidth]{IMAGE/stability.png}\\
    \caption{Logo}
    \textsc{Santos}, \emph{title} (year)
    \label{fig:stability}
\end{figure}

\begin{itemize}
        \item for sufficiently strong interactions, we may neglect quantum pressure, and consider the Thomas-Fermi (TF) regime
        \item TF solution for the trapped BEC has the same inverted parabola shape (as in non-dipolar case)
        \item BEC is prolat for $0 < \kappa < 1$ and $1 < \kappa$ oblat
        \item Bogoliubov-de Gennes Equation shows that the nonlocal character of the DDI causes a momentum dependend coupling constant, leading to a roton-like dispersion law, leading to dynamically instability, when the roton $\beta = \frac{g_{d}}{g}$ touches zero (experimetally not obeserved yet)
        \item by varying the density, the frequency of the confinement, and the short-range coupling,
        one can control the spectrum (roton minimum deeper/shallower)
        \item sequence of the non-local non-linearity 2D bright solitary waves may become stable
under appropriate conditions (Pedri and Santos, 2005)
        \item two instability regions for 2D solitons (against col-
        lapse and against unlimited expansion)
  \item $\tilde{g}_{cr}(\beta) \equiv \frac{g N_{cr}}{2 \pi l_{z}}$, so stable 2D anisotropic self-localised solitons exists just for $N < N_{cr}$
        \item non-dipolar BECs scatter elastically, the scattering of dipolar solitons is inelastic due to the lack integrability
        \item The solitons may transfer centre-of-mass energy into internal vibrational modes,
        resulting in intriguing scattering properties:

        \begin{itemize}
          \item including soliton fusion (Fig. 1.8)
          \item appearance of strong inelastic resonances
          \item possibility of observing 2D- soliton spiraling as that already observed in photo-refractive materials
        \end{itemize}

        \item Dipolar effects in spinor condensates
        \begin{itemize}
            \item spinor BECs: we focus on an effect which resembles the Einstein-de Haas effect
            \item Because of Zeeman sub-levels short-range interactions may occur in different s-wave scattering channels with different total angular momentum (for bosons even number) ( spin-1 bosons we have just F = 0 and F = 2)
            \item Each scattering channel has an associated s-wave scattering length $a_{F}$
            \item short-range interactions necessarily preserve the spin projection Sz
            \item DDI does not necessarily conserve the spin projection along the quantisation axis as DDI is anisotropic
            \item for initially maximally stretched state ($m_{F} =  - F$)
            \item short-range interactions cannot induce any spinor dynamics (due to conservation of total magnetisation $S_{z}$
            \item DDI may induce a transfer to $m_{F} + 1$
            \item for cylindrical symmetry around the quantisation axis, this violation of the spin projection is accompanied by a transfer of angular momentum to the centre of mass, resembling the well known Einstein-de Haas effect $\Rightarrow$ initially spin-polarised dipolar condensate can generate dynamically vorticity
            \item Einstein-de Haas effect is destroyed by weak magnetic fields (1 mG)
            \item the dominant Larmor precession, and invoking rotating-wave-approximation arguments, the physics must be constrained to manifolds of preserved magnetisation (2D optical lattices could help)
            \item Effect of DDI could be even observable under conserved $S_{z}$ (alkali spinor condensates)
            \item spin-changing collisions: collisions that conserve $S_{z}$, but do not conserve the relative population of the different Zeeman components
            \item Spin-changing collisions are characterised by an energy scale proportional
to the difference between scattering lengths at different channels
            \item this difference is very small, so can be significantly modified by the presence of other small energy scales (DDI) $\Rightarrow$ helical spin textures
        \end{itemize}
\end{itemize}

\begin{figure}[H]
    \centering
    \includegraphics[width=0.7\textwidth]{IMAGE/acrit.png}\\
    \caption{Logo}
    \textsc{Santos}, \emph{title} (year)
    \label{fig:acrit}
\end{figure}

\begin{itemize}
        \item Typo in ``we obtain a 1D equation similar to the a GP equation'', just the or a
        \item ``ground-state wave-function is independent of the in-plane coordinates '' Why?
        \item 1.26 to 1.27, where does the $U_{dd}$ go?
        \item Typo If: ``roton momentum. if this were so''
        \item Why should a modulation with a finite wavelength allow superfluids?
        \item Typo repeatance: ``the width of the width''
        \item What are the spin-F matrizes?
        \item Is the occurance of these spin textures in Figure \ref{fig:helical} special?
\end{itemize}

\section{Supersolids}

\begin{itemize}
    \item supersolid: features both the crystalline structure of a solid and the frictionless flow of a superfluid
In this state, every constituent atom is part of the solid and the superfluid simultaneously
    \item direct observation was limited to systems where the structure formation was mediated by external light fields
    \item beyond mean-field approximation leads to corrections to the ground
state energy stemming from quantum fluctuations of the collective modes in a BEC (LHY-correction)
    \item In 2018 quantum droplets in a Bose-Bose mixture were observed
    \item mean- field energy depends on the difference of the two coupling constants $\delta(g) = |g_{rep}| - |g_{att}|$ \item LHY-correction depends on the individual coupling constants
    \item For weakly attractive combination of interactions, a repulsive beyond mean-field correction can stabilize the BEC
    \item after a peak density increasing the number of particles only leads to an increase in the
size of the droplet
    \item eGPE: kinetic energy, external trapping, and two-body interactions, LHY
    \item beyond mean-field correction has only been calculated for a homogeneous system and can therefore only be included within a local-density approximation
    \item QMC calculations in full many-body system verified the formation
    \item intra-species scattering lengths $a_{11}$ and $a_{22}$ lead to different equilibrium densities $n^{(i)}_{0}$ for the two components of the mixture.
    \item droplet forms an intrinsic imbalance in the atom numbers of the two components ($\frac{N1}{N2} = \sqrt{\frac{a_{22}}{a_{11}}}$ )
    \item larger density than in original BEC increases the rate of three-body loss $\Rightarrow$ extra term in eGPE
\end{itemize}

\begin{figure}[H]
    \centering
    \includegraphics[width=0.7\textwidth]{IMAGE/droplet.png}\\
    \caption{Peak density of the droplet saturates in z-driection}
    \textsc{Santos}, \emph{title} (year)
    \label{fig:droplet}
\end{figure}


% !TeX root = ../main.tex

\section{Data set}
The Epinions data set \citep{Massa} is a very good case to demonstrate the Recursive
K-Nearest Neighbors algorithm because of its sparsity. As demonstrated in \autoref{chap:3},
this algorithm manages to overcome the limitations of the conventional KNN algorithm, which
cannot make predictions for ratings with no direct associations to other users or items.
This dataset consists of 40163 users and 139738 items and 664824 ratings.
As we mentioned previously, the sparsity percentage is 0,99988154.
The structure of the dataset is shown in the table below:
\begin{table}[H]
\centering
\caption{Epinions Dataset Sample}
\label{table:Epinions_sample}
\begin{tabular}{ |c|c|c| }
\hline
\textbf{user} & \textbf{item} & \textbf{Rating} \\
\hline
36153 & 62461 & 5 \\
\hline
427 & 38005 & 5  \\
\hline
751 & 53361 & 4 \\
\hline
11001 & 118950 & 4 \\
\hline
1169 & 66176 & 5 \\
\hline
9808 & 84459 & 2 \\
\hline
85 & 7446 & 4 \\
\hline
14717 & 3397 & 2 \\
\hline
\end{tabular}
\end{table}
The user and item columns contain the ids of users and items respectively, and the rating
column contains the rating of a user to an item. To test the accuracy of
KNN and Recursive-KNN algorithms this ratings matrix was split in a
train set and a test set. The train set is used so the algorithms can learn the
patterns of the data. Either how users rate or how items are being rated.
The similarity metrics discussed in \autoref{chap:2} will be computed based on
the train set and the rating predictions will be calculated based on this set's
ratings. The rating predictions will be both user-based and item based.
The test set is used to evaluate the rating predictions produced by the trained algorithms
to this set using an aggregating error function between the predictions
and the truth values. The train set consists of 520203
ratings(78\%) and the test set consists of 144621 ratings(22\%). The train
set is stratified by users, which means that it contains a proportion of each
user's ratings. Below there are some descriptive information about the Epinions dataset and the splitting method.
\begin{table}[H]
\centering
\caption{Epinions Descriptive}
\label{table:epinions_descriptive}
\begin{tabular}{ |c|c|c|c| }
\hline
&\textbf{Ratings Matrix} & \textbf{Train} & \textbf{Test}\\
\hline
count & 664824 & 520203 & 144621\\
\hline
mean & 3.9917 & 3.99 & 3.9975\\
\hline
std & 1.2068 & 1.2072 & 1.2053\\
\hline
min & 1 & 1 & 1\\
\hline
25\% & 3 & 3 & 3\\
\hline
50\% & 4 & 4 & 4\\
\hline
75\% & 5 & 5 & 5\\
\hline
max & 5 & 5 & 5\\
\hline
\end{tabular}
\end{table}

In the subsections below we present some basic descriptive information about each of the similarity
methods and in \autoref{sec:4.3} we will present the best result from the
rating predictions.

\subsection{Cosine Similarity}

\begin{figure}[H]
\centering
\includegraphics[width=0.7\textwidth]{chapter_4/boxplots/cosine/cosine_boxplot.jpg}
\caption{Cosine Similarity Boxplot (Whiskers= -+1.5IQR)}
\label{figure:cosine_similarity_boxplot}
\end{figure}

Both item and user similarity values have a minimum value near zero.The items' right whisker value is
around 0.8 and the users' around 0.18. Both items and users similarity outliers seem to reach the value of 1.
Also the number of outliers in item cosine appears significantly lower than that of the users.
At least 75\% of users similarities are lower than 0.1. In contrast, at least 50\% of
item similarities are larger than 0.1 and at least 25\% are larger than 0.35.

\begin{table}[H]
\centering
\caption{Cosine Similarity Descriptive}
\label{table:cosine_similarity_descriptive}
\begin{tabular}{|c|c|c|}
\hline
		  & \textbf{users} & \textbf{items} \\ \hline
count     & 14614164       & 18672616       \\ \hline
mean      & 0.063717709    & 0.2590515005   \\ \hline
std       & 0.0852396246   & 0.2948759498   \\ \hline
min       & 0.0001118907   & 0.0001876405   \\ \hline
25\%      & 0.0184512236   & 0.0467473495   \\ \hline
50\%      & 0.0389925088   & 0.1290322581   \\ \hline
75\%      & 0.0758098044   & 0.3638034376   \\ \hline
max       & 1              & 1              \\ \hline
max count & 26742          & 1519527        \\ \hline
min count & 1              & 1              \\ \hline
\end{tabular}
\end{table}

\begin{table}[H]
\centering
\caption{Cosine Similarity Count Descriptive}
\label{table:cosine_similarity_count_descriptive}
\begin{tabular}{|c|c|c|}
\hline
          & \textbf{users}  & \textbf{items} \\ \hline
count     & 39156           & 121140         \\ \hline
mean      & 746.4584737971  & 308.281591547  \\ \hline
std       & 1174.5013754664 & 666.5799806645 \\ \hline
min       & 1               & 1              \\ \hline
25\%      & 40              & 37             \\ \hline
50\%      & 246             & 120            \\ \hline
75\%      & 944             & 338            \\ \hline
max       & 13843           & 31057          \\ \hline
max count & 1               & 1              \\ \hline
min count & 793             & 545            \\ \hline
\end{tabular}
\end{table}

\subsection{Modified Cosine Similarity}

\begin{figure}[H]
\centering
\includegraphics[width=0.9\textwidth]{chapter_4/boxplots/modified_cosine/modified_cosine_boxplot.jpg}
\caption{Modified Cosine Similarity Boxplot (Whiskers= -+1.5IQR)}
\label{figure:modified_cosine_similarity_boxplot}
\end{figure}

Modified cosine similarity seems to have the same effect in the values for both items
and users. Both these boxplots have the majority of their values at 1. Their minimum values
are a little less than 0.4. Any value other than 1 can be interpreted as an outlier.

\begin{table}[H]
\centering
\caption{Modified Cosine Similarity Descriptive}
\label{table:modified_cosine_similarity_descriptive}
\begin{tabular}{|c|c|c|}
\hline
          & \textbf{users} & \textbf{items} \\ \hline
count     & 14614164       & 18672616       \\ \hline
mean      & 0.9923727737   & 0.9970178776   \\ \hline
std       & 0.0345699768   & 0.0205710501   \\ \hline
min       & 0.3846153846   & 0.3846153846   \\ \hline
25\%      & 1              & 1              \\ \hline
50\%      & 1              & 1              \\ \hline
75\%      & 1              & 1              \\ \hline
max       & 1              & 1              \\ \hline
max count & 12887451       & 17728374       \\ \hline
min count & 1170           & 263            \\ \hline
\end{tabular}
\end{table}

\begin{table}[H]
\centering
\caption{Modified Cosine Similarity Count Descriptive}
\label{table:modified_cosine_similarity_count_descriptive}
\begin{tabular}{|c|c|c|}
\hline
          & \textbf{users}  & \textbf{items} \\ \hline
count     & 39156           & 121140         \\ \hline
mean      & 746.4584737971  & 308.281591547  \\ \hline
std       & 1174.5013754664 & 666.5799806645 \\ \hline
min       & 1               & 1              \\ \hline
25\%      & 40              & 37             \\ \hline
50\%      & 246             & 120            \\ \hline
75\%      & 944             & 338            \\ \hline
max       & 13843           & 31057          \\ \hline
max count & 1               & 1              \\ \hline
min count & 793             & 545            \\ \hline
\end{tabular}
\end{table}

\subsection{Adjusted Cosine Similarity}

\begin{figure}[H]
\centering
\includegraphics[width=0.8\textwidth]{chapter_4/boxplots/adjusted_cosine/adjusted_cosine_boxplot.jpg}
\caption{Adjusted Cosine Similarity Boxplot (Whiskers= -+1.5IQR)}
\label{figure:adjusted_cosine_similarity_boxplot}
\end{figure}

Adjusted cosine similarity seems to have distributed the values both for users and items very well.
There are no outliers for either of these boxplots. Although, items have at least 50\% of their
values at 1, users have at least 50\% of their values over 0.7. They both have their minimum
value at -1.

\begin{table}[H]
\centering
\caption{Adjusted Cosine Similarity Descriptive}
\label{table:adjusted_cosine_similarity_descriptive}
\begin{tabular}{|c|c|c|}
\hline
          & \textbf{users} & \textbf{items} \\ \hline
count     & 14547343       & 18578462       \\ \hline
mean      & 0.073856558    & 0.09784622     \\ \hline
std       & 0.9611564664   & 0.9793750112   \\ \hline
min       & -1             & -1             \\ \hline
25\%      & -1             & -1             \\ \hline
50\%      & 0.7255719661   & 1              \\ \hline
75\%      & 1              & 1              \\ \hline
max       & 1              & 1              \\ \hline
max count & 6998746        & 9729527        \\ \hline
min count & 5753489        & 7845237        \\ \hline
\end{tabular}
\end{table}

\begin{table}[H]
\centering
\caption{Adjusted Cosine Similarity Count Descriptive}
\label{adjusted_cosine_similarity_count_descriptive}
\begin{tabular}{|c|c|c|}
\hline
          & \textbf{users}  & \textbf{items} \\ \hline
count     & 38302           & 119634         \\ \hline
mean      & 759.6127095191  & 310.5883277329 \\ \hline
std       & 1179.0168839062 & 666.8475091836 \\ \hline
min       & 1               & 1              \\ \hline
25\%      & 45              & 39             \\ \hline
50\%      & 260             & 123            \\ \hline
75\%      & 968             & 340            \\ \hline
max       & 13798           & 30920          \\ \hline
max count & 1               & 1              \\ \hline
min count & 569             & 398            \\ \hline
\end{tabular}
\end{table}

\subsection{Modified Adjusted Cosine Similarity}

\begin{figure}[H]
\centering
\includegraphics[width=0.8\textwidth]{chapter_4/boxplots/modified_adjusted_cosine/modified_adjusted_cosine_boxplot.jpg}
\caption{Modified Adjusted Cosine Similarity Boxplot (Whiskers= -+1.5IQR)}
\label{figure:modified_adjusted_cosine_similarity_boxplot}
\end{figure}

The modification used in adjusted cosine similarity seems to have enclosed the values near zero.
Their median is aligned very close to zero. The whiskers for items are around -0.4 and 0.4
for left and right respectively. For users, the whiskers are around -0.12 and 0.12 for left and right
respectively. Items similarities behave like a wider version of users similarities, probably
because items are more than twice the size of users which allows them to form a wider range of
different similarities.

\begin{table}[H]
\centering
\caption{Modified Adjusted Cosine Similarity Descriptive}
\label{table:modified_adjusted_cosine_similarity_descriptive}
\begin{tabular}{|c|c|c|}
\hline
          & \textbf{users} & \textbf{items} \\ \hline
count     & 14547343       & 18578462       \\ \hline
mean      & 0.0004873924   & 0.0165982167   \\ \hline
std       & 0.1313835099   & 0.398039276    \\ \hline
min       & -1             & -1             \\ \hline
25\%      & -0.0280684821  & -0.0701974943  \\ \hline
50\%      & 0.0012377237   & 0.0033406502   \\ \hline
75\%      & 0.029188451    & 0.096874534    \\ \hline
max       & 1              & 1              \\ \hline
max count & 17005          & 863028         \\ \hline
min count & 11891          & 679602         \\ \hline
\end{tabular}
\end{table}

\begin{table}[H]
\centering
\caption{Modified Adjusted Cosine Similarity Count Descriptive}
\label{table:modified_adjusted_cosine_similarity_count_descriptive}
\begin{tabular}{|c|c|c|}
\hline
          & \textbf{users}  & \textbf{items} \\ \hline
count     & 38302           & 119634         \\ \hline
mean      & 759.6127095191  & 310.5883277329 \\ \hline
std       & 1179.0168839062 & 666.8475091836 \\ \hline
min       & 1               & 1              \\ \hline
25\%      & 45              & 39             \\ \hline
50\%      & 260             & 123            \\ \hline
75\%      & 968             & 340            \\ \hline
max       & 13798           & 30920          \\ \hline
max count & 1               & 1              \\ \hline
min count & 569             & 398            \\ \hline
\end{tabular}
\end{table}

\subsection{Pearson Correlation Coefficient}

\begin{figure}[H]
\centering
\includegraphics[width=0.8\textwidth]{chapter_4/boxplots/pearson/pearson_boxplot.jpg}
\caption{Pearson Correlation Coefficient Boxplot (Whiskers= -+1.5IQR)}
\label{figure:pearson_corr_coef_boxplot}
\end{figure}

The similarities formed using the Pearson correlation coefficient seem well distributed across
the range -1 to 1. No outliers exist in both users and items. In the users' boxplot at least 50\% of the
values appear to be near 1. In the items' boxplot at least 50\% of the values are over 0.8. The
minimum value for both of the similarities is at -1.

\begin{table}[H]
\centering
\caption{Pearson Correlation Coefficient Descriptive}
\label{table:pearson_corr_coef_descriptive}
\begin{tabular}{|c|c|c|}
\hline
          & \textbf{users} & \textbf{items} \\ \hline
count     & 12801832       & 9114131        \\ \hline
mean      & 0.2437009881   & 0.0973284577   \\ \hline
std       & 0.9285318435   & 0.9625193963   \\ \hline
min       & -1             & -1             \\ \hline
25\%      & -1             & -1             \\ \hline
50\%      & 1              & 0.940706152    \\ \hline
75\%      & 1              & 1              \\ \hline
max       & 1              & 1              \\ \hline
max count & 6825146        & 4489219        \\ \hline
min count & 4163467        & 3678485        \\ \hline
\end{tabular}
\end{table}

\begin{table}[H]
\centering
\caption{Pearson Correlation Coefficient Count Descriptive}
\label{table:pearson_corr_coef_boxplot}
\begin{tabular}{|c|c|c|}
\hline
          & \textbf{users}  & \textbf{items} \\ \hline
count     & 26166           & 40409          \\ \hline
mean      & 978.5089046855  & 451.0941126977 \\ \hline
std       & 1227.2905909627 & 753.2523780236 \\ \hline
min       & 1               & 1              \\ \hline
25\%      & 148             & 90             \\ \hline
50\%      & 507             & 222            \\ \hline
75\%      & 1353            & 500            \\ \hline
max       & 12486           & 17515          \\ \hline
max count & 1               & 1              \\ \hline
min count & 89              & 16             \\ \hline
\end{tabular}
\end{table}

\subsection{Pearson Modification 1}

\begin{figure}[H]
\centering
\includegraphics[width=0.8\textwidth]{chapter_4/boxplots/modified_pearson_1/modified_pearson_1_boxplot.jpg}
\caption{Modified Pearson 1 Boxplot (Whiskers= -+1.5IQR)}
\label{figure:modified_pearson_1_boxplot}
\end{figure}

The minimum value for both users and items similarities is at -1. Users have at least 50\% of their
values over 0.8 and at least 25\% at 1. Items' median is around 0.4. Items also seem to have
a 25\% of their value at 1. Both have their minimum at -1. At least 25\% of the similarities
for both users and items are negative.

\begin{table}[H]
\centering
\caption{Modified Pearson 1 Descriptive}
\label{table:modified_pearson_1_descriptive}
\begin{tabular}{|c|c|c|}
\hline
          & \textbf{users} & \textbf{items} \\ \hline
count     & 1161526        & 560066         \\ \hline
mean      & 0.3619158933   & 0.1343440987   \\ \hline
std       & 0.8153387654   & 0.8305601334   \\ \hline
min       & -1             & -1             \\ \hline
25\%      & -0.5           & -0.8703882798  \\ \hline
50\%      & 0.8947368421   & 0.3942210015   \\ \hline
75\%      & 1              & 1              \\ \hline
max       & 1              & 1              \\ \hline
max count & 529709         & 184408         \\ \hline
min count & 227880         & 133046         \\ \hline
\end{tabular}
\end{table}

\begin{table}[H]
\centering
\caption{Modified Pearson 1 Count Descriptive}
\label{modified_pearson_1_count_descriptive}
\begin{tabular}{|c|c|c|}
\hline
          & \textbf{users} & \textbf{items} \\ \hline
count     & 19566          & 25208          \\ \hline
mean      & 118.7290197281 & 44.4355760076  \\ \hline
std       & 266.6999446542 & 153.5700823656 \\ \hline
min       & 1              & 1              \\ \hline
25\%      & 5              & 2              \\ \hline
50\%      & 23             & 6              \\ \hline
75\%      & 105            & 24             \\ \hline
max       & 5003           & 5401           \\ \hline
max count & 1              & 1              \\ \hline
min count & 2066           & 5234           \\ \hline
\end{tabular}
\end{table}

\subsection{Pearson Modification 2}
\begin{figure}[H]
\centering
\includegraphics[width=0.8\textwidth]{chapter_4/boxplots/modified_pearson_2/modified_pearson_2_boxplot.jpg}
\caption{Modified Pearson 2 Boxplot (Whiskers= -+1.5IQR)}
\label{figure:modified_pearson_2_boxplot}
\end{figure}

At a first glance, these boxplots look very similar to \autoref{figure:modified_adjusted_cosine_similarity_boxplot}.
This is probably because the modification in both cases was to include the entire vectors in the
denominator (see \autoref{eq:modified_adjusted_cosine} and \autoref{eq:pearson_2}). Both boxplots have
their median near zero. They both have many outliers that spread both left and right of the boxes with
minimum value at -1 and maximum at 1. The whiskers are around -0.2 to -0.15 from the left
side and 0.15 to 0.2 from the right side.

\begin{table}[H]
\centering
\caption{Modified Pearson 2 Descriptive}
\label{table:modified_pearson_2_descriptive}
\begin{tabular}{|c|c|c|}
\hline
          & \textbf{users} & \textbf{items} \\ \hline
count     & 12801832       & 9114131        \\ \hline
mean      & 0.0218434042   & 0.0101521404   \\ \hline
std       & 0.089326644    & 0.1385987076   \\ \hline
min       & -1             & -1             \\ \hline
25\%      & -0.0093795851  & -0.0251329949  \\ \hline
50\%      & 0.0076098072   & 0.0037092894   \\ \hline
75\%      & 0.0367446145   & 0.0372677996   \\ \hline
max       & 1              & 1              \\ \hline
max count & 3              & 710            \\ \hline
min count & 2              & 412            \\ \hline
\end{tabular}
\end{table}

\begin{table}[H]
\centering
\caption{Modified Pearson 2 Count Descriptive}
\label{table:modified_pearson_2_count_descriptive}
\begin{tabular}{|c|c|c|}
\hline
          & \textbf{users}  & \textbf{items} \\ \hline
count     & 26166           & 40409          \\ \hline
mean      & 978.5089046855  & 451.0941126977 \\ \hline
std       & 1227.2905909627 & 753.2523780236 \\ \hline
min       & 1               & 1              \\ \hline
25\%      & 148             & 90             \\ \hline
50\%      & 507             & 222            \\ \hline
75\%      & 1353            & 500            \\ \hline
max       & 12486           & 17515          \\ \hline
max count & 1               & 1              \\ \hline
min count & 89              & 16             \\ \hline
\end{tabular}
\end{table}

\subsection{MSD}
\begin{figure}[H]
\centering
\includegraphics[width=0.8\textwidth]{chapter_4/boxplots/msd/msd_boxplot.jpg}
\caption{MSD Boxplot (Whiskers= -+1.5IQR)}
\label{figure:msd_boxplot}
\end{figure}

The minimum similarity value for both users and items is very close to zero. Their median
is around 1 and 75\% of the similarities are below 2. They both have a lot of outliers that
reach a score around 15. Item similarity has some more extreme outliers that are very close to 25.

\begin{table}[H]
\centering
\caption{MSD Descriptive}
\label{table:msd_descriptive}
\begin{tabular}{|c|c|c|}
\hline
          & \textbf{users} & \textbf{items} \\ \hline
count     & 9805108        & 12434536       \\ \hline
mean      & 0.6661864961   & 0.6544694584   \\ \hline
std       & 0.5256923021   & 0.4628208708   \\ \hline
min       & 0.0625         & 0.0625         \\ \hline
25\%      & 0.25           & 0.25           \\ \hline
50\%      & 1              & 1              \\ \hline
75\%      & 1              & 1              \\ \hline
max       & 13             & 25             \\ \hline
max count & 1              & 1              \\ \hline
min count & 598720         & 800571         \\ \hline
\end{tabular}
\end{table}

\begin{table}[H]
\centering
\caption{MSD Count Descriptive}
\label{msd_count_descriptive}
\begin{tabular}{|c|c|c|}
\hline
          & \textbf{users} & \textbf{items} \\ \hline
count     & 38442          & 120382         \\ \hline
mean      & 510.1247593778 & 206.5846388995 \\ \hline
std       & 850.5223750077 & 484.7004513454 \\ \hline
min       & 1              & 1              \\ \hline
25\%      & 25             & 22             \\ \hline
50\%      & 156            & 72             \\ \hline
75\%      & 613            & 213            \\ \hline
max       & 11014          & 23589          \\ \hline
max count & 1              & 1              \\ \hline
min count & 1182           & 1563           \\ \hline
\end{tabular}
\end{table}

\subsection{MAD}

\begin{figure}[H]
\centering
\includegraphics[width=0.8\textwidth]{chapter_4/boxplots/mad/mad_boxplot.jpg}
\caption{MAD Boxplot (Whiskers= -+1.5IQR)}
\label{figure:mad_boxplot}
\end{figure}

As expected, mean absolute difference boxplots are very similar to that of mean squared difference
\autoref{figure:msd_boxplot}. It is clear that the boxes have shrunk, due to the
fact that MAD uses the absolute values instead of the squared ones and this modification
lowers the standard deviation between the similarities.

\begin{table}[H]
\centering
\caption{MAD Descriptive}
\label{table:mad_descriptive}
\begin{tabular}{|c|c|c|}
\hline
          & \textbf{users} & \textbf{items} \\ \hline
count     & 9805108        & 12434536       \\ \hline
mean      & 0.7962500349   & 0.7668765592   \\ \hline
std       & 0.4338056858   & 0.3639417506   \\ \hline
min       & 0.25           & 0.25           \\ \hline
25\%      & 0.5            & 0.5            \\ \hline
50\%      & 1              & 1              \\ \hline
75\%      & 1              & 1              \\ \hline
max       & 13             & 25             \\ \hline
max count & 1              & 1              \\ \hline
min count & 598720         & 800571         \\ \hline
\end{tabular}
\end{table}

\begin{table}[H]
\centering
\caption{MAD Count Descriptive}
\label{table:mad_count_descriptive}
\begin{tabular}{|c|c|c|}
\hline
          & users          & items          \\ \hline
count     & 38442          & 120382         \\ \hline
mean      & 510.1247593778 & 206.5846388995 \\ \hline
std       & 850.5223750077 & 484.7004513454 \\ \hline
min       & 1              & 1              \\ \hline
25\%      & 25             & 22             \\ \hline
50\%      & 156            & 72             \\ \hline
75\%      & 613            & 213            \\ \hline
max       & 11014          & 23589          \\ \hline
max count & 1              & 1              \\ \hline
min count & 1182           & 1563			\\ \hline
\end{tabular}
\end{table}

\subsection{Jaccard Coefficient}
\begin{figure}[H]
\centering
\includegraphics[width=0.8\textwidth]{chapter_4/boxplots/jaccard/jaccard_boxplot.jpg}
\caption{Jaccard Boxplot (Whiskers= -+1.5IQR)}
\label{figure:jaccard_boxplot}
\end{figure}

Both item and user similarity values have a minimum value near zero. The items' right whisker value is
around 0.45 and for the users around 0.04. Both items and users similarity outliers appear to reach the value
1. Also the number of outliers in item cosine seems significantly lower than users. At least 75\% of
users similarities are lower than 0.1. In contrast, at least 25\% of item similarities are larger than 0.2.
Users' boxplot is very narrow which indicates that 50\% of the values are very close to each other. It
covers a range of merely 0.02. On the other hand, the items' boxplot is relatively wider. It covers a range of about 0.19,
which confirms the wider spread of the values.

\begin{table}[H]
\centering
\caption{Jaccard Descriptive}
\label{table:jaccard_descriptive}
\begin{tabular}{|c|c|c|}
\hline
          & \textbf{users} & \textbf{items} \\ \hline
count     & 14614164       & 18672616       \\ \hline
mean      & 0.0307346429   & 0.1733928584   \\ \hline
std       & 0.0544456926   & 0.2779653358   \\ \hline
min       & 0.0006485084   & 0.0006060606   \\ \hline
25\%      & 0.0112359551   & 0.0153846154   \\ \hline
50\%      & 0.0196078431   & 0.0476190476   \\ \hline
75\%      & 0.0344827586   & 0.2            \\ \hline
max       & 1              & 1              \\ \hline
max count & 26746          & 1522328        \\ \hline
min count & 1              & 1              \\ \hline
\end{tabular}
\end{table}

\begin{table}[H]
\centering
\caption{Jaccard Count Descriptive}
\label{table:jaccard_count_descriptive}
\begin{tabular}{|c|c|c|}
\hline
          & \textbf{users}  & \textbf{items} \\ \hline
count     & 39156           & 121140         \\ \hline
mean      & 746.4584737971  & 308.281591547  \\ \hline
std       & 1174.5013754664 & 666.5799806645 \\ \hline
min       & 1               & 1              \\ \hline
25\%      & 40              & 37             \\ \hline
50\%      & 246             & 120            \\ \hline
75\%      & 944             & 338            \\ \hline
max       & 13843           & 31057          \\ \hline
max count & 1               & 1              \\ \hline
min count & 793             & 545            \\ \hline
\end{tabular}
\end{table}

\section{Evaluation Metrics}
The accuracy of the predictions is one of the most important goals in recommender
systems as it gives more chances that the prediction will be relevant to the
user. As discussed in \autoref{chap:1} other goals include the novelty of the
recommendations, serendipity and diversity. In this thesis the focus is
on the accuracy of the predictions. Four metrics will be discussed to evaluate the
Recursive Nearest Neighbors algorithm on the Epinions dataset.
\subsection{Root Mean Squared Error (RMSE)}
RMSE is one of the most popular accuracy estimators.
It sums the squared error between the true and estimated value divided by the
number of the predictions the model was able to extract. Finally it is square
rooted so that the error corresponds to units of ratings instead of square
units. \citep{Ricci}
$$RMSE = \sqrt{\frac{\sum_{(u,i) \in \mathcal{T}(r_{u,i} - \hat{r}_{u,i})^2}}{n}}$$
where,
\begin{itemize}
	\item[] $\mathcal{T}$ is the test set
	\item[] $r_{u,i}$ is the true rating of user $u$ to item $i$ in $\mathcal{T}$
	\item[] $\hat{r}_{u,i}$ is the rating prediction for user $u$ to item $i$
	\item[] $n$ is the number of predictions
\end{itemize}

\subsection{Mean Absolute Error (MAE)}
MAE is another popular metric for accuracy estimation. It sums the absolute
value between the difference between the true and estimated ratings
divided by the number of predictions. \citep{Ricci}
$$MAE = \frac{\sum_{(u,i) \in \mathcal{T}\left|r_{u,i} - \hat{r}_{u,i}\right|}}{n}$$
where,
\begin{itemize}
	\item[] $\mathcal{T}$ is the test set
	\item[] $r_{u,i}$ is the true rating of user $u$ to item $i$ in $\mathcal{T}$
	\item[] $\hat{r}_{u,i}$ is the rating prediction for user $u$ to item $i$
	\item[] $n$ is the number of predictions
\end{itemize}

\subsection{Mean Absolute User Error (MAUE)}
While RMSE and MAE estimate the global error to the system MAUE first
calculates the MAE score for each user then averages all user errors to find
how much the system diverges on average from each user \citep{Massa}.
$$MAUE = \frac{1}{n}\sum_{u \in \mathcal{T}}\frac{\sum_{i \in \mathcal{I}_u\mathopen|r_{u,i} - \hat{r}_{u,i}\mathclose|}}{n_u}$$
where,
\begin{itemize}
	\item[] $\mathcal{T}$ is the test set
	\item[] $r_{u,i}$ is the true rating of user $u$ to item $i$ in $\mathcal{T}$
	\item[] $\hat{r}_{u,i}$ is the rating prediction for user $u$ to item $i$
	\item[] $n_u$ is the number of predictions for user $u$
	\item[] $n$ is the number of users for whom predictions were made for
\end{itemize}
\subsection{Root Mean Squared User Error (RMSUE)}
Like in MAUE, the equation for RMSE can be used to produce a per user rmse estimate.
In this metric first the RMSE for every user is computed. Then the RMSEs of
the users are averaged to form the RMSUE score with is an estimate in the
error of the model per user of the dataset.
$$RMSUE = \frac{1}{n}\sum_{u \in \mathcal{T}}\sqrt{\frac{\sum_{i \in \mathcal{I}_u(y_{u,i} - \hat{y}_{u,i})^2}}{n_u}}$$
where,
\begin{itemize}
	\item[] $\mathcal{T}$ is the test set
	\item[] $r_{u,i}$ is the true rating of user $u$ to item $i$ in $\mathcal{T}$
	\item[] $\hat{r}_{u,i}$ is the rating prediction for user $u$ to item $i$
	\item[] $n_u$ is the number of predictions for user $u$
	\item[] $n$ is the number of users for whom predictions were made for
\end{itemize}

\section{Evaluation Results and Benchmarks}\label{sec:4.3}
For this experiment item-based and user-based similarities discussed in \autoref{chap:2} where used.
Any similarity metrics that could contain
negative similarities, had those negative similarities excluded from the
computations.
For the K-Nearest Neighbors, the following 22 different values
of $\mathcal{K}$ were used:
$$\mathcal{K} = [1, 3, 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100]$$
For the Recursive K-Nearest Neighbors, as defined in the
algorithmic steps of \autoref{chap:3}, 22 different values of $\mathcal{M}$
were also used:
$$\mathcal{M} = [1, 3, 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100]$$

Below, the best outcome based on each evaluation metric for user-based and
item-based CF will be presented. The combined errors for KNN and Recursive-KNN
for RMSE and MAE below are computed using these formulas:
\begin{itemize}
	\item[] \textbf{Total RMSE:}
	$$RMSE_{Total} = \sqrt{\frac{n_{KNN}*RMSE_{KNN}^2 + n_{R-KNN}*RMSE_{R-KNN}^2}{n_{KNN} + n_{R-KNN}}}$$
	where,
	\begin{itemize}
		\item[] $n_{KNN}$ is the number of predictions made using the KNN algorithm
		\item[] $RMSE_{KNN}$ is the RMSE score using the KNN algorithm
		\item[] $n_{R-KNN}$ is the number of predictions made using the Recursive KNN algorithm
		\item[] $RMSE_{R-KNN}$ is the RMSE score using the Recursive KNN algorithm
	\end{itemize}
	\item[] \textbf{Total MAE:}
	$$MAE_{Total} = \frac{n_{KNN}*MAE_{KNN} + n_{R-KNN}*MAE_{R-KNN}}{n_{KNN} + n_{R-KNN}}$$
	where,
	\begin{itemize}
		\item[] $n_{KNN}$ is the number of predictions made using the KNN algorithm
		\item[] $MAE_{KNN}$ is the MAE score using the KNN algorithm
		\item[] $n_{R-KNN}$ is the number of predictions made using the Recursive KNN algorithm
		\item[] $MAE_{R-KNN}$ is the MAE score using the Recursive KNN algorithm
	\end{itemize}
\end{itemize}

\autoref{chap:appendix_src} has a link to
the code written in Python(version 3.6) using the Anaconda distribution \citep{anaconda}
that calculated the similarities, rating predictions
and the errors. It also contains the train and test data, the \LaTeX{} structure
that this thesis was build on and the numerical results.
\autoref{chap:appendix_results} contains all the plots produced by the
numerical results. All plots in this thesis were produced using the Matplotlib library
\citep{Hunter:2007}.

\subsection{User-Based}
\begin{figure}[H]
\centering
\includegraphics[width=1\textwidth]{chapter_4/evaluation/evaluation_user_rmse.eps}
\caption{User-based Total RMSE Best Score}
\label{figure:user_best_total_rmse}
\end{figure}
The best RMSE score for user-based CF was produced using $\mathcal{K}=100$ and $\mathcal{M}=3$.
The exact RMSE score is 1.1604146071 and it was calculated based on 124433 rating predictions(\autoref{table:prediction_counters_by_sim})
using the modified cosine similarity(\autoref{eq:modified_cosine})
for the 144621 ratings in the test set(\autoref{table:epinions_descriptive}).

\begin{figure}[H]
\centering
\includegraphics[width=1\textwidth]{chapter_4/evaluation/evaluation_user_mae.eps}
\caption{User-based Total MAE Best Score}
\label{figure:user_total_mae}
\end{figure}

The best MAE score for user-based CF was produced using $\mathcal{K}=50$ and $\mathcal{M}=3$.
The exact MAE score is 0.854066176 and it was calculated based on 124433 rating predictions(\autoref{table:prediction_counters_by_sim})
using the jaccard coefficient(\autoref{eq:jaccard})
for the 144621 ratings in the test set(\autoref{table:epinions_descriptive}).

\begin{figure}[H]
\centering
\includegraphics[width=1\textwidth]{chapter_4/knn/User_RMSUE_KNN.eps}
\caption{User-based KNN RMSUE Scores}
\label{figure:User_knn_rmsue}
\end{figure}

The best KNN RMSUE score for user-based CF was produced using $\mathcal{K}=100$.
The exact KNN RMSUE score is 1.0031145695 and it was calculated based on 100311 rating
predictions(\autoref{table:prediction_counters_by_sim})
using the modified cosine similarity(\autoref{eq:modified_cosine})
for the 144621 ratings in the test set(\autoref{table:epinions_descriptive}).

\begin{figure}[H]
\centering
\includegraphics[width=1\textwidth]{chapter_4/evaluation/evaluation_user_rmsue.eps}
\caption{User-based Recursive-KNN RMSUE Best Score}
\label{figure:User_rknn_rmsue}
\end{figure}

The best Recursive-KNN RMSUE score for user-based CF was produced using $\mathcal{K}=100$ and $\mathcal{M}=3$.
The exact Recursive-KNN RMSUE score is 0.9089525549 and it was calculated based on 24122 rating
predictions(\autoref{table:prediction_counters_by_sim})
using the modified cosine similarity(\autoref{eq:modified_cosine})
out of the 25061 pairs for which KNN was unable to find neighbors(\autoref{table:users_log}).

\begin{figure}[H]
\centering
\includegraphics[width=1\textwidth]{chapter_4/knn/User_MAUE_KNN.eps}
\caption{User-based KNN MAUE Scores}
\label{figure:User_knn_maue}
\end{figure}

The best KNN MAUE score for user-based CF was produced using $\mathcal{K}=50$.
The exact KNN MAUE score is 0.8819077974 and it was calculated based on 100311 rating predictions(\autoref{table:prediction_counters_by_sim})
using the cosine similarity(\autoref{eq:cosine})
for the 144621 ratings in the test set(\autoref{table:epinions_descriptive}).

\begin{figure}[H]
\centering
\includegraphics[width=1\textwidth]{chapter_4/evaluation/evaluation_user_maue.eps}
\caption{User-based Recursive-KNN MAUE Best Score}
\label{figure:User_rknn_maue}
\end{figure}

The best Recursive-KNN MAUE score for user-based CF was produced using $\mathcal{K}=75$ and \\$\mathcal{M}=20$.
The exact Recursive-KNN MAUE score is 0.8558869566 and it was calculated based on 34392 rating predictions(\autoref{table:prediction_counters_by_sim})
using the adjusted cosine similarity(\autoref{eq:adjusted_cosine})
out of the 36278 pairs for which KNN was unable to find neighbors(\autoref{table:users_log}).

\subsection{Item-based}
\begin{figure}[H]
\centering
\includegraphics[width=1\textwidth]{chapter_4/evaluation/evaluation_item_rmse.eps}
\caption{Item-based Total RMSE Best Score}
\label{figure:Item_knn_rmse}
\end{figure}

The best RMSE score for item-based CF was produced using $\mathcal{K}=100$ and $\mathcal{M}=3$.
The exact RMSE score is 1.3155259043 and it was calculated based on 123115 rating predictions(\autoref{table:prediction_counters_by_sim})
using the adjusted cosine similarity(\autoref{eq:adjusted_cosine})
for the 144621 ratings in the test set(\autoref{table:epinions_descriptive}).

\begin{figure}[H]
\centering
\includegraphics[width=1\textwidth]{chapter_4/evaluation/evaluation_item_mae.eps}
\caption{Item-based Total MAE Best Score}
\label{figure:Item_knn_mae}
\end{figure}

The best MAE score for item-based CF was produced using $\mathcal{K}=100$ and $\mathcal{M}=3$.
The exact MAE score is 0.9707211649 and it was calculated based on 123115 rating predictions(\autoref{table:prediction_counters_by_sim})
using the adjusted cosine similarity(\autoref{eq:adjusted_cosine})
for the 144621 ratings in the test set(\autoref{table:epinions_descriptive}).

\begin{figure}[H]
\centering
\includegraphics[width=1\textwidth]{chapter_4/knn/Item_RMSUE_KNN.eps}
\caption{Item-based KNN RMSUE Scores}
\label{figure:Item_knn_rmsue}
\end{figure}

The best KNN RMSUE score for item-based CF was produced using $\mathcal{K}=100$.
The exact KNN RMSUE score is 1.0328968801 and it was calculated based on 89800 rating predictions(\autoref{table:prediction_counters_by_sim})
using the adjusted cosine similarity(\autoref{eq:adjusted_cosine})
for the 144621 ratings in the test set(\autoref{table:epinions_descriptive}).

\begin{figure}[H]
\centering
\includegraphics[width=1\textwidth]{chapter_4/evaluation/evaluation_item_rmsue.eps}
\caption{Item-based Recursive-KNN RMSUE Best Score}
\label{figure:Item_rknn_rmsue}
\end{figure}

The best Recursive-KNN RMSUE score for item-based CF was produced using $\mathcal{K}=100$ and $\mathcal{M}=3$.
The exact Recursive-KNN RMSUE score is 1.0367245756 and it was calculated based on 33315 rating predictions(\autoref{table:prediction_counters_by_sim})
using the adjusted cosine similarity(\autoref{eq:adjusted_cosine})
out of the 35280 pairs for which KNN was unable to find neighbors(\autoref{table:items_log}).

\begin{figure}[H]
\centering
\includegraphics[width=1\textwidth]{chapter_4/knn/Item_MAUE_KNN.eps}
\caption{Item-based KNN MAUE Scores}
\label{figure:Item_knn_maue}
\end{figure}

The best KNN MAUE score for item-based CF was produced using $\mathcal{K}=30$.
The exact KNN MAUE score is 0.9318609947 and it was calculated based on 89800 rating predictions(\autoref{table:prediction_counters_by_sim})
using the modified adjusted cosine similarity(\autoref{eq:modified_adjusted_cosine})
for the 144621 ratings in the test set(\autoref{table:epinions_descriptive}).

\begin{figure}[H]
\centering
\includegraphics[width=1\textwidth]{chapter_4/evaluation/evaluation_item_maue.eps}
\caption{Item-based KNN MAUE Best Score}
\label{figure:Item_rknn_maue}
\end{figure}

The best Recursive-KNN MAUE score for item-based CF was produced using $\mathcal{K}=90$ and $\mathcal{M}=3$.
The exact Recursive-KNN MAUE score is 0.9864005988 and it was calculated based on 33315 rating predictions(\autoref{table:prediction_counters_by_sim})
using the adjusted cosine similarity(\autoref{eq:adjusted_cosine})
out of the 35280 pairs for which KNN was unable to find neighbors(\autoref{table:items_log}).

\subsection{Benchmarks}
\begin{table}[H]
\centering
\caption{Rating Predicted with KNN and Recursive-KNN}
\label{table:prediction_counters_by_sim}
\begin{tabular}{ccc|ccc|c}
\multicolumn{3}{c|}{\textbf{USERS}}            & \multicolumn{3}{c|}{\textbf{ITEMS}}            &                          \\ \cline{1-6}
\textbf{KNN} & \textbf{R-KNN} & \textbf{TOTAL} & \textbf{KNN} & \textbf{R-KNN} & \textbf{TOTAL} & \textbf{SIMILARITY}      \\ \hline
88379        & 34392          & 122771         & 89800        & 33315          & 123115         & Adjusted Cosine          \\
\textbf{100311}       & 24122          & \textbf{124433}         & \textbf{100311}       & 24122          & \textbf{124433}         & Cosine                   \\
\textbf{100311}       & 24122          & \textbf{124433}         & \textbf{100311}       & 24122          & \textbf{124433}         & Jaccard                  \\
93924        & 29747          & 123671         & 94528        & 29203          & 123731         & MAD                      \\
93924        & 29747          & 123671         & 94528        & 29203          & 123731         & MSD                      \\
88379        & 34392          & 122771         & 89800        & 33315          & 123115         & Modified Adjusted Cosine \\
\textbf{100311}       & 24122          & \textbf{124433}         & \textbf{100311}       & 24122          & \textbf{124433}         & Modified Cosine          \\
59017        & \textbf{40470}          & 99487          & 54644        & \textbf{34096}          & 88740          & Modified Pearson 1       \\
89936        & 28183          & 118119         & 84511        & 24357          & 108868         & Modified Pearson 2       \\
89936        & 28183          & 118119         & 84511        & 24357          & 108868         & Pearson
\end{tabular}
\end{table}

Cosine, Jaccard and Modified Cosine were able to produce the most rating predictions, 124433 in total,
both for user-based and item-based. Modified Pearson 1 was the one with the least similarities
(\autoref{table:modified_pearson_1_descriptive}) and therefore the one with the least rating
predictions, even though, it managed to predict a total of 99487 ratings user-based
and 88740 item-based. The outcome however was very poor for this similarity formula for any
evaluation metric. The average boost in rating predictions using the Recursive-KNN method was
about 25\% for user-based CF and 24\% for item-based CF.
\newpage
The two tables below are the logs produced for user-based(\autoref{table:users_log})
and item-based(\autoref{table:items_log}) KNN and Recursive-KNN algorithms.
The first three columns refer to the KNN algorithm and present the number of pairs that KNN
algorithm was unable to predict. The first column indicates the number of pairs
(user, item) for which relevant nearest neighbors did not exist. The second column
indicates the number of pairs that either the item in user-based or user in
item-based did not exist in the train set. The third column indicates the number
of pairs that either the user in user-based or item in
item-based did not exist in the train set and therefore KNN could not find similarities.
The test was for 22 different values of $\mathcal{K}$. The total compute time for each similarity
metric for all $\mathcal{K}$'s is indicated in the fourth column.
The next two columns refer to Recursive-KNN algorithm. The fifth column indicates
the number of pairs that could not be predicted due to no further neighbors
after the completion of the Recursive-KNN algorithm.
The sixth column is the time the Recursive-KNN took for completing all the available combinations
484 in count between $\mathcal{K}$'s and $\mathcal{M}$'s (i.e. ($\mathcal{M}$=1, $\mathcal{K}$=1),
($\mathcal{M}$=1, $\mathcal{K}$=3), ($\mathcal{M}$=3, $\mathcal{K}$=1) e.t.c.).

The computations were run on a cloud computing platform.
The specifications of the machine were the following:
\begin{itemize}
	\item[] \textbf{OS:} Ubuntu Server 16.04.3
	\item[] \textbf{CPU:} Intel Xeon E5-2697A v4 @ 2.60GHz
	\item[] \textbf{RAM:} 3GB DDR4 + 1GB SWAP
\end{itemize}

\begin{table}[H]
\centering
\caption{User-based KNN and Recursive-KNN Log}
\label{table:users_log}
\begin{tabular}{|c|c|c|c|c|c|c|}
\cline{1-6}
\multicolumn{4}{|c|}{\textbf{KNN}}                                               & \multicolumn{2}{c|}{\textbf{Recursive-KNN}} & \multicolumn{1}{l}{}                          \\ \hline
\textbf{No NN} & \textbf{Not in Train} & \textbf{No Similarity} & \textbf{Time} & \textbf{R-No NN} & \textbf{R-Time}		& \multicolumn{1}{c|}{\textbf{Similarity}}      \\ \hline
36278          & 18939                 & 1025                  & 0:3:58        & 1886             & 4:3:9           		& \multicolumn{1}{c|}{adjusted cosine}          \\ \hline
25061          & 18939                 & 310                   & 0:5:17        & 939              & 4:1:57          		& \multicolumn{1}{c|}{cosine}                   \\ \hline
25061          & 18939                 & 310                   & 0:3:41        & 939              & 2:57:58         		& \multicolumn{1}{c|}{jaccard}                  \\ \hline
31200          & 18939                 & 558                   & 0:4:16        & 1453             & 3:2:29          		& \multicolumn{1}{c|}{mad}                      \\ \hline
31200          & 18939                 & 558                   & 0:4:9         & 1453             & 3:9:12          		& \multicolumn{1}{c|}{msd}                      \\ \hline
36278          & 18939                 & 1025                  & 0:4:13        & 1886             & 4:3:56          		& \multicolumn{1}{c|}{modified adjusted cosine} \\ \hline
25061          & 18939                 & 310                   & 0:4:31        & 939              & 2:42:42         		& \multicolumn{1}{c|}{modified cosine}          \\ \hline
48886          & 18939                 & 17779                 & 0:3:8         & 8416             & 1:33:26         		& \multicolumn{1}{c|}{modified pearson 1}       \\ \hline
29450          & 18939                 & 6296                  & 0:3:18        & 1267             & 3:4:17          		& \multicolumn{1}{c|}{modified pearson 2}       \\ \hline
29450          & 18939                 & 6296                  & 0:3:8         & 1267             & 2:48:48         		& \multicolumn{1}{c|}{pearson}                  \\ \hline
\end{tabular}
\end{table}

\begin{table}[H]
\centering
\caption{Item-based KNN and Recursive-KNN Log}
\label{table:items_log}
\begin{tabular}{|c|c|c|c|c|c|c}
\cline{1-6}
\multicolumn{4}{|c|}{\textbf{KNN}}                                               & \multicolumn{2}{c|}{\textbf{Recursive-KNN}} & \multicolumn{1}{l}{}                          \\ \hline
\textbf{No NN} & \textbf{Not in  Train} & \textbf{No Similarity} & \textbf{Time} & \textbf{R-No NN}      & \textbf{R-Time}     & \multicolumn{1}{c|}{\textbf{Similarity}}      \\ \hline
35280          & 0                      & 19541                  & 0:3:2         & 1965                  & 2:18:35             & \multicolumn{1}{c|}{adjusted cosine}          \\ \hline
25188          & 0                      & 19122                  & 0:3:43        & 1066                  & 2:26:56             & \multicolumn{1}{c|}{cosine}                   \\ \hline
25188          & 0                      & 19122                  & 0:3:37        & 1066                  & 2:12:56             & \multicolumn{1}{c|}{jaccard}                  \\ \hline
30827          & 0                      & 19266                  & 0:3:22        & 1624                  & 1:54:18             & \multicolumn{1}{c|}{mad}                      \\ \hline
30827          & 0                      & 19266                  & 0:3:28        & 1624                  & 1:58:50             & \multicolumn{1}{c|}{msd}                      \\ \hline
35280          & 0                      & 19541                  & 0:3:10        & 1965                  & 2:32:4              & \multicolumn{1}{c|}{modified adjusted cosine} \\ \hline
25188          & 0                      & 19122                  & 0:3:37        & 1066                  & 2:11:18             & \multicolumn{1}{c|}{modified cosine}          \\ \hline
41037          & 0                      & 48940                  & 0:2:14        & 6941                  & 0:39:1              & \multicolumn{1}{c|}{modified pearson 1}       \\ \hline
25332          & 0                      & 34778                  & 0:2:58        & 975                   & 1:49:35             & \multicolumn{1}{c|}{modified pearson 2}       \\ \hline
25332          & 0                      & 34778                  & 0:2:52        & 975                   & 1:46:24             & \multicolumn{1}{c|}{pearson}                  \\ \hline
\end{tabular}
\end{table}


\subsubsection{ValueDuration}
\label{ValueDurationPP}
The \xmlNode{ValueDuration} PostProcessor is a tool to construct a particular kind of histogram, where the
independent variable is the number of times a variable exceeds a particular value, and the dependent variable
is the values themselves.  An example of this is the Load Duration Curve in energy modeling. This approach is
similar to that used in Lebesgue integration. Note that for each realization in the input
\xmlNode{HistorySet}, a separate load duration curve will be created for each target.

The \xmlNode{ValueDuration} PostProcessor can only act on \xmlNode{HistorySet} data objects, and generates a
\xmlNode{HistorySet} in return.  Two output variables are created for each \xmlAttr{target}:
\xmlString{counts\_x} and \xmlString{bins\_x}, where \xmlString{x} is replaced by the name of the target.
These must be specified in the output data object in order to be collected.

To plot a traditional Load Duration Curve, the x-axis should be the bins variable, and the y-axis should be
the counts variable.

\ppType{ValueDuration}{ValueDuration}
%
\begin{itemize}
  \item \xmlNode{target}, \xmlDesc{comma separated strings, required field}, specifies the names of the
    target(s) for which Value Duration histograms should be generated.
  \item \xmlNode{bins}, \xmlDesc{integer, required field}, specifies the number of bins that the values of the
    targets should be counted into.
\end{itemize}

\textbf{Example:}

\begin{lstlisting}[style=XML]
<Simulation>
 ...
  <Models>
    ...
    <PostProcessor name="pp" subType="ValueDuration">
      <target>x, y</target>
      <bins>100</bins>
    </PostProcessor>
    ...
  </Models>
 ...
</Simulation>
\end{lstlisting}


\section{Graphs and Paths}

\subsection{Graphs}

\subsubsection{Overview}
A (directed) graph is a set of nodes and each node has a list of pointers to their neighbour nodes. You can use such a graph for path finding - within your scene there's a graph with way nodes and possible directed ways between them, meaning that there may be cases you can go from node A to node B, but not from B to A - then for instance a \emph{find the shortest way from node A to node D} query can be performed on this graph and you will receive a path with a list of nodes you have to pass if you want to walk from the source to the target node. How you find this source and target graph nodes in another story... for this, you need to find the graph node which is nearest to your source world position, the same for the target. You can do this by iterating over all graph nodes and finding the closest graph node, or by using some kind of hierarchy to make this task more performant...

All different graphs are managed like textures, meshes etc. through a manager and resource handlers. The graph manager is accessed through \emph{GraphManager::GetInstance()}.

Here's an example how such a graph may look like:

\begin{lstlisting}[caption=Graph example]
Test graph:                  Shortest paths: ('source'->
                             'target' = nodes to pass)

+------5------+              A->A = A
|             |              A->B = A->B
|  /-2--B--3--C--5-\         A->C = A->D->C
| /     |    /|     \        A->D = A->D
+-A     2 /3/ 1      F       A->E = A->D->E
  \     |/    |     /        A->F = A->D->E->F
   \-1--D--1--E--2-/

\end{lstlisting}

Normally for path finding, only the \emph{distances} between the nodes are taken into account - not their \emph{real} positions within the world.

\subsubsection{File Format}
PixelLight graph files have the extension \emph{graph}.

Note that the distance attribute of a neighbour node is optional, if this attribute is not given or has a negative value, the distance is calculated automatically. Normally you can ignore this distance attribute - manipulate it only for special tasks by hand!

\subsection{Paths}

\subsubsection{Overview}
A path consists of a set of nodes at different world coordinates. An actor can for example move along this path automatically. If the path is closed which can be set using the \emph{GraphPath} function \emph{SetClosed()}, actors will move to the first node if the last one was reached.

The most important path function is \emph{GetPosByNodeIndex()} which will return the path world position at a given time. The first parameter is the node index were 1.5 will be between two nodes. With the second parameter you can define whether the position is interpolated in a linear and rough way or if you want to receive positions which lies on a smooth curve. If the position isn't linear interpolated the first and last node are skipped because they will be used to define the behaviour of the curve.

Here's an example how such a path may look like:

\begin{lstlisting}[caption=Path example]
  /B-------------C\
 /                 \
A                   D
 \                 /
  \F-------------E/
\end{lstlisting}

An actor can be moved along a path for example in the way shown below:

\begin{lstlisting}[caption=Moving along path]
fTimer += PLCore::Timing::GetInstance()->GetTimeDifference();
pMyActor->SetPosition(pMyPath->GetPosByNodeIndex(fTimer));
\end{lstlisting}

Normally you only use a simple timer to move on the path. The example above shows the simplest way to move an actor along a path, this isn't useful if you also want to perform e.g. collision detection. In this way you should use the path points only as \emph{move to} targets were the actor should try to move to and then working with for instance forces to the target.

\begin{figure}
  \begin{center}
    \includegraphics{pics/Paths.eps}
  \end{center}
  \caption{Actor walking on a path}
  \label{fig:Actor walking on a path}
\end{figure}

Paths can be build by hand or automatically as result of for instance a graph query (path finding).

\subsubsection{Create Paths by Hand}
It's possible to build a path by hand during runtime. Call the graph path managers creation function \emph{GraphPathManager::Create()} to generate a new path. The function will return a pointer to the new path and now it's up to you to fill up this path with nodes.

New nodes are added using the path function \emph{GraphPath::AddNode()} which will take a path node as parameter. Note that this path node is just linked. The path node itself is in the simplest case only a world coordinate. \emph{GraphNode} also offers some tool functions like to compute the distance to another node using \emph{GetDistance()}.

Example:

\begin{lstlisting}[caption=Path creation example]
//	Test path:
//	  /B-------------C\
//	 /                 \
//	A                   D
//	 \                 /
//	  \F-------------E/

// Build path visualized above
GraphPath cPath;
GraphNode *pNode;

// A
pNode = new GraphNode("A");
cPath.AddNode(*pNode);
pNode->SetPos(0.0f, 0.0f, 2.0f);
// B
pNode = new GraphNode("B");
cPath.AddNode(*pNode);
pNode->SetPos(2.0f, 0.0f, 0.0f);
// C
pNode = new GraphNode("C");
cPath.AddNode(*pNode);
pNode->SetPos(16.0f, 0.0f, 0.0f);
// D
pNode = new GraphNode("D");
cPath.AddNode(*pNode);
pNode->SetPos(19.0f, 0.0f, 2.0f);
// E
pNode = new GraphNode("E");
cPath.AddNode(*pNode);
pNode->SetPos(16.0f, 0.0f, 4.0f);
// F
pNode = new GraphNode("F");
cPath.AddNode(*pNode);
pNode->SetPos(2.0f, 0.0f, 4.0f);

// The path should be closed
cPath.SetClosed(true);
\end{lstlisting}

\subsubsection{File Format}
PixelLight path files have the extension \emph{path}.