\subsubsection{\lst{ByIndex} operation (OpCode 178)}

\noindent
\(\begin{array}{| l | l | l | l |}
    \hline
    \bf{Slot} & \bf{Format} & \bf{\#bytes} & \bf{Description} \\
    \hline
    \lst{input} & \lst{Value} & variable &  \\
    \hline
    \multicolumn{4}{l}{\lst{optional default}} \\
    \hline
    ~~\lst{tag} & \lst{Byte} & 1 & \text{0 - no value; 1 - has value} \\
    \hline
    \multicolumn{4}{l}{~~\lst{when tag == 1}} \\
    \hline
    ~~~~\lst{value} & \lst{Value} & variable &  \\
    \hline
    \multicolumn{4}{l}{\lst{end optional}} \\
 \end{array}\)

\subsubsection{\lst{EQ} operation (OpCode 147)}

\noindent
\(\begin{array}{| l | l | l | l |}
    \hline
    \bf{Slot} & \bf{Format} & \bf{\#bytes} & \bf{Description} \\
    \hline
    \multicolumn{4}{l}{\lst{match (left, right)}} \\
    \multicolumn{4}{l}{~~\lst{with (Constant(left, SBoolean), Constant(right, SBoolean))}} \\
    \hline
    ~~~~\lst{opCode} & \lst{Byte} & 1 & \lst{ConcreteCollectionBooleanConstantCode}  \\
    \hline
    ~~~~\lst{(left,right)} & \lst{Bits} & 1 & \text{two bits in 1 byte} \\
    \hline
    \multicolumn{4}{l}{~~\lst{otherwise}} \\
    \hline
    ~~~~\lst{left} & \lst{Value} & variable &  \\
    \hline
    ~~~~\lst{right} & \lst{Value} & variable &  \\
    \hline
    \multicolumn{4}{l}{\lst{end match}} \\
 \end{array}\)

 \subsubsection{\lst{Tuple} operation (OpCode 134)}

 \noindent
 \(\begin{array}{| l | l | l | l |}
     \hline
     \bf{Slot} & \bf{Format} & \bf{\#bytes} & \bf{Description} \\
     \hline
     \lst{numItems} & \lst{UByte} & 1 &  \\
     \hline
     \multicolumn{4}{l}{\lst{for i=1 to numItems}} \\
     \hline
     \lst{item} & \lst{Value} & variable &  \\
     \hline
     \multicolumn{4}{l}{\lst{end for}} \\
  \end{array}\)

 \subsubsection{\lst{Fold} operation (OpCode 176)}

 \noindent
 \(\begin{array}{| l | l | l | l |}
     \hline
     \bf{Slot} & \bf{Format} & \bf{\#bytes} & \bf{Description} \\
     \hline
     \lst{this} & \lst{Value} & variable &  \\
     \hline
     \lst{zero} & \lst{Value} & variable &  \\
     \hline
     \lst{op} & \lst{Value} & variable &  \\
     \hline
  \end{array}\)

%   SerScope(Fold$(176), ArrayBuffer((this,DataScope(DataInfo(ArgInfo(this,this
%   instance),ValueFmt))), (zero,DataScope(DataInfo(ArgInfo(zero,a starting
%   value),ValueFmt))), (op,DataScope(DataInfo(ArgInfo(op,the binary
%   operator),ValueFmt)))))


\section{Related Work} \label{sec:RelatedWork}

Most of the parallelization of LCS on most multi-core platforms
focuses on {\em wavefront} parallelism.  The wavefront method keeps
the computation as a wavefront that sweep through the entire dynamic
programming tables.  The wavefront computation is {\em not}
cache-friendly, i.e., the wavefront algorithm cannot effectively keep
the required data in cache.  To address this cache issue, Maleki et
al.~\cite{Maleki2016EfficientPU} developed a technique to exploit more
parallelism in the dynamic programming.

The alternative to wavefront method is the traditional row-by-row
approach, in which the dynamic programming tables is built in a
row-by-row manner.  For example, Peng~\cite{Peng2011TheLC} gives a
$O(nm \alpha(n))$ VGLCS algorithm that is easy to implement and an
asymptotically better $O(nm)$ algorithm, where $\alpha$ is the inverse
of Ackermann's function~\cite{Banachowski1980ACT}.

It is difficult to parallelize traditional row-by-row approach for VGLCS
due to the difficulty in efficient suffix and range query in a parallel
environment.  Peng's sequential VGLCS algorithm uses disjoint sets by
Gabow~\cite{Gabow1983ALA} and Tarjan~\cite{Tarjan1975EfficiencyOA} for
suffix maximum query.  We instead use {\em sparse
table}~\cite{Berkman1993RecursiveSP} to support incremental suffix/range
maximum queries in our VGLCS algorithm.  The sparse table is simple to
implement and provides sufficient parallelism for good performance in a
parallel environment.

Fischer~\cite{Fischer2006TheoreticalAP} proposed blocked sparse table
for better performance than the unblocked sparse
table~\cite{Berkman1993RecursiveSP}.  We also adopted blocked sparse
table and tested its implementation in our experiments.  Fischer's
algorithm~\cite{Fischer2006TheoreticalAP} builds least ancestor tables
for answering range maximum query.  We instead use a {\em
rightmost-pops} encoding for Cartesian trees.

Demaine~\cite{Demaine2009OnCT} also proposed {\em cache-aware}
operations on Cartesian tree~\cite{Vuillemin1980AUL}, to address the
cache miss issues in Fischer's least common ancestor table
building~\cite{Fischer2006TheoreticalAP}.  Masud~\cite{Hasan2010CacheOA}
presents a new encoding method that reduces the number of instructions.
Our rightmost-pops encoding for Cartesian trees also reduces cache
misses, and with a much simpler implementation than Demaine's encoding.

Finally the authors would like to point out that to the best of our
knowledge, we are not aware of any {\em dynamic} encoding for Cartesian
trees.  All previous works are {\em off-line}, i.e., they assume all
data are given in advance, as a result, they cannot cope with
incrementally added data.  In contrast, our dynamic Catalan index
computation technique in Section~\ref{sec:dynamic} does support
efficient range query on an incremental data sets.


\section{Background on LDA}
\label{sec:background}

%------------------------------------------------

Topic modeling, a form of latent variable modeling, is an unsupervised machine learning method which attempts to recreate the distribution of so-called ``topics'' an author used to generate a corpus of documents.
The term topic is used to describe a frequency distribution of terms within a vocabulary.
In this use, a topic can be understood to represent an academic concept covered within the context of a course.
This is based on the assumption that the words used in a course description when introducing the course's topics are the same words used within descriptions of the topic itself.
The topics discovered in a corpus can be used to categorize documents and provide structure to an otherwise unknown dataset.

%------------------------------------------------

\acf{lda} is a specific type of topic modeling which assumes that a mixture of multiple topics exist within a single document in some proportion (\ie\ were used to generate that document)~\cite{Blei2003}.
\ac{lda} assumes a generative process where, for each word in the document, the algorithm selects a distribution over topics, selects a topic, and then selects a vocabulary term~\cite{Blei2003}.
Reversing this generative process is significantly more difficult because the topic distributions are unknown; these unknown information is what the ``hidden model'' or ``latent model'' refers to.

%------------------------------------------------

The computation \ac{lda} performs is the determination of the topic distributions over a set of documents.
Given the set of documents as input, generating the corpus topics is a probabilistic process.
Taking the variables $\theta_{d,k}$ (topic proportion for topic $k$ in document $d$), $\beta_{1:k}$ (topic $k$), $z_{d,n}$ (topic assignment for word $n$ in document $d$), and $w_{d,n}$ (the $n^{th}$ word in document $d$), \ac{lda} estimates the posterior probability in \eref{eq:posterior}~\cite{Blei2012}.

%------------------------------------------------

\begin{equation}
p(\beta_{1:K}, \theta_{1:D},z_{1:D} | w_{1:D}) = \frac{\beta_{1:K},
\theta_{1:D},z_{1:D}, w_{1:D}}{w_{1:D}}
\label{eq:posterior}
\end{equation}

%------------------------------------------------

\noindent
Gibbs Sampling is used to estimate the denominator (\ie\ the evidence)~\cite{Blei2003}.
Running \ac{lda} over a document set results in a usable set of vocabulary frequency distributions or topics for each document.

%------------------------------------------------

A graphical ``plate'' diagram of \ac{lda}'s generative process is given in \fref{fig:lda-plates}, adapted from~\cite{Blei2003}.
Circular nodes represent random variables while rectangular plates represent duplication.
The shaded node is the only observed (\ie\ evidence) variable, words from the document set.

%------------------------------------------------

\begin{figure}
  \centering
  \includegraphics[width=0.45\textwidth]{figures/lda-plates}
  \caption{LDA graphical diagram adapted from~\cite{Blei2012}\label{fig:lda-plates}.}
\end{figure}

%------------------------------------------------

\subsection{Related Work}
\label{sec:related-work}

%------------------------------------------------

Our research complements other efforts within Computer Science education that are directed towards categorization of content to improve pedagogy, \eg\ Hubwieser et al.~\cite{hubwieser2013}.
We believe that our project contributes both, by identifying content (\ie\ topics) being taught across institutions and by identifying gaps and unique contributions.
This information can be compared to teacher competencies and used to design assessment and instruments to measure them.
Another area in which this work can assist is in identification of concepts and their classification, especially ``threshold concepts''~\cite{ShinnersKennedyFincher2013}.
Overall, we believe our data-driven approach complements other qualitative efforts by building on them and by automating some aspects of the research.
The work of~\cite{airasian2001taxonomy} discusses a taxonomy related to learning breaking the process down along the cognitive process and knowledge dimensions.
Using a richer source of course information that course descriptions our proposed method could be adopted to learn concepts along these dimensions.

%------------------------------------------------

Other work has also attempted to extract concept information from course data.
Yang et al.\ employ four distinct techniques to map courses into a conceptual space and then learn prerequisite relationships between similar courses~\cite{Yang2015}.
Two of their conceptual mapping techniques generate latent features, which have the downside of not being human-readable as in \ac{lda}.
The remaining two techniques generate human-readable topics, but rely either on an outside source (Wikipedia) or simply represent concepts as the vocabulary of the document.
The benefit of \ac{lda} as an information retrieval tool is its ability to generate pseudo human-readable topics while acting in a fully unsupervised manner on a single, large data set.
Our approach naively targets a dataset of fixed universities and customizes web scrapers specifically for their computer science departments.
Effland et al.\ introduces a robust web crawler system to automatically search for, identify, and extract course descriptions from disparate locations on the Internet~\cite{Effland2015}.
Application of similar technology in this work was considered, and would greatly improve the scale of the analyzed data.

%------------------------------------------------


% !TEX root = ../00_thesis.tex

%-------------------------------------------------------------------------------
\subsection{Simulated Worst-Case Performance}
\label{subsec:simulation}

In \cref{subsec:perf_model}, we derived the optimal performance achievable according to our \DRP's model.
However, this model is based on a worst-case analysis of message latency throughout the system.
Because such an analysis is inherently pessimistic, it is important to estimate how pessimist the analysis is. In other words, how tight are the latency bounds given by the model?
In this section, we investigate this question using a discrete event simulation.

\fakepar{Procedure}
We simulate the run-time behavior of \DRP using the values and parameters from our implementations~(\cref{table:simulation_parameters}).
The simulation framework tracks the latency of each individual message through the entire system, \ie all \APs, \CPs, \bolt and the wireless communication network.
Concretely, the simulation is implemented using Matlab scripts (openly available -- \cref{append:drp_artifacts}).

\blink computes the round schedules assuming that the first message of each flow is available for communication at $t = 0\s$. The actual epoch at which the  \APs write the first packet of each flow is randomized between $0\s$ and the flow's minimal message interval $T$; subsequent packets are sent with period $T$.
The random seed is fixed for reproduciblility.

\squarepar{%
	\fakepar{Scenario}
	Node 1 acts as the sink and communicates with all other nodes in the network. As described in \cref{sec:designDetailed}, \DRP is initialized with a set of control flows $\mathcal{F}_{control}$, which is necessary in order to register subsequent flows%
}
\[
\mathcal{F}_{control} =
	\left\lbrace
	\begin{tabular}{@{\,}l@{\,}}
	(1\,, n\,, \periodany = 10\s\,, \jitterany = 0\s\,, \deadlineany = 30\s )\\
	(n\,, 1\,, \periodany = 10\s\,, \jitterany = 0\s\,, \deadlineany = 30\s )\\
	\end{tabular}
	\right\rbrace
\]
for $n \in ($2$ .. $20$)$. In practice, such flows can also be used to send low-priority data (\eg status data) regularly to the sink.

An event from the environment (\eg a rock crack~\cite{meyer2019IPSN}) is co-detected by nodes 2 to 5, which consequently emit a request for a new flow to the sink node. In order to transfer the event data as fast as possible, the message interval is chosen as small as possible
(\ie equal to $T_f^s$, the flushing interval of $\cp$ -- Refer to \eqref{eq:design_flush_period_source}, \eqref{eq:design_round_period} and \eqref{eq:ndeadline_constraint_period}),
\[
\mathcal{F}_{new} =
	\left\lbrace
	\begin{tabular}{@{}l@{}}
	(n\,, 1\,, \periodany = 1.074\s\,, \jitterany = 0\s\,, \deadlineany = 10\s )\\
	\end{tabular}
	\right\rbrace
\]
for $n \in ($2$ .. $5$)$. We record the end-to-end latency of all packets during two minutes, during which about 900 messages are transmitted through the system.

\fakepar{Results}
\cref{fig:latency} shows the distribution of end-to-end latency of messages, shown as percentage of the analytical worst-case latency (given by \cref{thm:delta}).
We see that a few messages indeed experience a latency up to 97\,\% of the analytic worst-case bound.
The simulation also indicates that, in many cases, the worst-case buffer sizes of $\cp$ and \bolt are reached.
Overall, these results support our analysis of \DRP. They show that our worst-case bounds are tight; therefore, we can conclude that the performance derived using \DRP's model~(\cref{subsec:perf_model}) is representative of the performance that can be truly guaranteed by the system.

\begin{figure}
	\centering
	\href{\drpfig{Figure-11}}{%
	\includegraphics[scale=1]{lat_simu}}
	\caption{Distribution of end-to-end latency of messages, shown as percentage of the analytical worst-case latency.
		\capt{Some messages experience a latency very close to their worst-case bound (97\,\%), which demonstrates the tightness of the analysis.}
	 }
	\label{fig:latency}
\end{figure}

%-------------------------------------------------------------------------------
\subsection{Real-World Performance}
\label{subsec:flocklab}

\begin{table}
	\caption{Flow sets used in the end-to-end latency evaluation of \DRP.
	\capt{The \blink utilization is computed assuming a strictly periodic release of messages.}}
	{\smaller\input{\PathTab/flow_set.csv}}
	\label{tab:flow_set}
\end{table}

% Some intro about why we do real tests
We now consider the performance of our implementation of \DRP on embedded hardware: We use the first-generation DPP, which features a TI MSP432P401R as \AP and a TI CC430F5147 as \CP~(\cref{append:dpp}).
The software is based on the publicly available implementation of LWB~\cite{Code_LWB}; it is written in C and uses Contiki 2.7~\cite{contiki} as operating system.
The implementation of \blink on the \AP is built upon~\cite{acevedo2016Realtime}.
%
We discuss our implementation performance in terms of memory usage, computation workload, and message latency.

\squarepar{%
	\fakepar{Memory usage}
	\DRP requires both \AP and \CP to store some state information related to the currently running flows, as well message queues and buffers. The available RAM on both processors is shown in \cref{table:memory}.%
}

The 64\kB of the \AP are largely sufficient; it would supports hundreds of flows. The \CP is more limited: With a payload size of 32\bytes, \CP is capped to a maximum for 40 flows~(\cref{table:memory}). For a regular node, this will likely be sufficient for most applications; however, on the host node, this seriously limits the scalability of the system.

One possible solution is to use the embedded external memory (128\kB); this would solve the memory limitation issue, but it may also introduce additional delays, which are currently not accounted for. Or we could use another processor as \CP with more than 4\kB of RAM.

\begin{table}
	\centering
	\caption{Memory available and required for our implementation of \DRP.
	\capt{%
		The difference between ``total'' and ``available''  memory corresponds to the memory taken by the firmware only. $L$ is the message payload.}
	}
	\label{table:memory}
	{\smaller\input{\PathTab/memory.csv}}
\end{table}

\pagebreak

\squarepar{%
	\fakepar{Computation}
	The most extensive computations in \DRP are the computations of the \blink schedules and admission tests. The evaluation of these computations on embedded hardware is discussed in depth in~\cite{zimmerling2017Blink}.%
}

In addition to \blink computations, the \APs must perform \DRP admission tests. These are simple operations~(\cref{thm:CP} and \ref{thm:AP}) which can be implemented efficiently. In our experiments, an admission test takes typically around 30\ms to complete (maximum observed execution time: 130\ms).

\DRP admission tests are performed only once per flow (when a new flow is requested). Thus, we can conclude that the computational workload induced by \DRP (in addition to \blink) is negligible.

\afterpage{
\begin{figure}
	\centering
		\begin{subfigure}{\linewidth}
				\captionsetup{labelformat=empty}
				\href{\drpfig{Figure-12}}{%
	      \includegraphics[scale=1]{lat_real_low}}
				\caption{}
			\end{subfigure}\\[10pt]
			\begin{subfigure}{\linewidth}
					\captionsetup{labelformat=empty}
					\href{\drpfig{Figure-12}}{%
		      \includegraphics[scale=1]{lat_real_medium}}
					\caption{}
				\end{subfigure}\\[10pt]
				\begin{subfigure}{\linewidth}
						\captionsetup{labelformat=empty}
						\href{\drpfig{Figure-12}}{%
			      \includegraphics[scale=1]{lat_real_high}}
						\caption{}
					\end{subfigure}
  \caption{%
	Distribution of end-to-end latency of messages, shown as percentage of the analytical worst-case latency for a \DRP run using different flow sets~(\cref{tab:flow_set}).
	\capt{%
	Top -- Low utilization.
	Middle -- Medium utilization.
	Bottom -- High utilization.}
  }
  \label{fig:lat_real_all}
\end{figure}
}

\fakepar{End-to-end latency}
We investigate the experienced end-to-end latency of messages. We use a network of 10 source nodes and one host, and run experiments on the FlockLab testbed~\cite{lim2013FlockLab}. In addition to the control flows, each source node request a data flow toward the host with a pseudo-random period and end-to-end deadline.
Once the flow is admitted, the source \APs release new packets periodically.
The different flow sets used are listed in \cref{tab:flow_set}. \DRP is configured with a deadline ratio $r=0.5$, a round length $\rlength = 1\s$, and a maximum of $\nslotsmax = 5$ slots per round.

\begin{table}
	\centering
	\caption{Experienced latency of messages, expressed as percentage of the flow's end-to-end deadline.
	\capt{The tightness correspond to the ratio of the experienced latency with the analytical upper-bound given by~\cref{thm:delta}.}}
	{\smaller\input{\PathTab/latency.csv}}
	\label{tab:latency}
\end{table}

The results are summarized in \cref{tab:latency} and \cref{fig:lat_real_all}, reporting data from one run for each flow set.
The first observation is that all messages that are successfully transmitted over the wireless network do meet their end-to-end deadline.
However, compared to the simulation experiment~(\cref{subsec:simulation}), we do not encounter so much analytical corner cases: the observed latency is often much smaller than the analytical upper-bound~(\cref{fig:lat_real_all}).
On the other hand, this means that the actual runtime performance is better that what is guaranteed: even with large end-to-end deadlines (around 60\s), the experience message latency is most of the time between 5\s to 15\s.

Such ``short'' average latency can be explained by the nature of the flow set. The \emph{network} deadline, enforced by \blink, must be smaller than the flow period~\cref{sec:designDetailed}. Since the period are small compared to the \emph{end-to-end deadlines}, these end-to-end deadlines do not constraint the \DRP contracts: the experience latency correlates with the flow period.

It is interesting to observe that the average end-to-end latency is smaller for the high utilization flow set than for the low and middle ones. Again, this is due to the flow set. To meet the network deadline, \blink schedule at least one round per period. Thus, a flow set with shorter period results in more frequent rounds. Once a round is scheduled, it is filled with any message ready for transmission. Thus, flows are opportunistically served earlier than necessary to meet their end-to-end deadline. Conclusion: having a flow with a small period reduces the average latency experienced by all flows in the system. This is an interesting (and unforeseen) consequence of \DRP mechanism.

\squarepar{%
	\fakepar{Conclusions}
	The performance evaluation presented in this section validates the design of \DRP and our implementation: we showcased that we can run a wireless \CPS that meet end-to-end deadlines between distributed applications~(\cref{subsec:simulation} and \ref{subsec:flocklab}).
	In addition, we derived the theoretical optimal performance achievable based on \DRP's model~(\cref{subsec:perf_model}).%
}

A more thorough investigation of the actual system performance across different scenarios and environments remains to be performed.
For such a performance evaluation, using \triscale~(\cref{ch:triscale}) would be natural.
%
Chronologically, \triscale is the last piece of work of this dissertation. In hindsight, our evaluation of \DRP appears a bit naive and simple. Still, we argue that it successfully demonstrates the soundness of \DRP's design.% and supports our claims.


\documentclass{report}
\usepackage{graphicx}

\usepackage{amsmath}
\DeclareMathOperator*{\argmax}{argmax}

\usepackage{hyperref}
\hypersetup{
    colorlinks=true,
    linkcolor=blue,
    filecolor=magenta,
    urlcolor=cyan,
}

\begin{document}

\title{Banana World Project Report}
\author{Denis Sergeev}

\section*{Problem definition}

The goal of the project was to create a RL Agent which is able to train from scratch by interacting with the Reacher (robot arm) environment for a series of episodes. During the episode the agent receives the environment state. The agent is able to perform some actions which may change the environment state.

\subsection*{Environment description}

In this environment, a double-jointed arm can move to target locations. A reward of \(+0.1\) is provided for each step that the agent's hand is in the goal location. Thus, the goal of the agent is to maintain its position at the target location for as many time steps as possible.

The observation space consists of 33 variables corresponding to position, rotation, velocity, and angular velocities of the arm. Each action is a vector with four numbers, corresponding to torque applicable to two joints. Every entry in the action vector should be a number between \(-1\) and \(1\).

The barrier for solving the problem is that all agents must get an average score of +30 (over 100 consecutive episodes). Specifically, after each episode, we add up the rewards that each agent received (without discounting), to get a score for each agent. This yields 20 (potentially different) scores. We then take the average of these 20 scores. This yields an average score for each episode (where the average is over all 20 agents).

\section*{Solution}
\subsection*{Deep Deterministic Policy Gradient Algorithm}
\subsubsection*{Network}
The DDPG algorithm uses two neural networks, an actor and a critic. Each one has two copies, a local and a target one. The Actor is trained to give the best possible action, given the current state of the environment. \(s => a\). The Critic is trained to give an estimate of the reward that'd be obtained if an action A is applied at the state \(s\). \((s, a) => V\) Local networks are trained to get the "labels" given by the target.

Both networks consist of fully connected units with batch normalization and ReLu activation. The input to the actor network is the environment state, which goes through 3 fully connected layers. The final activation function is \(tanh\), since the action value should be in-between \(-1\) and \(1\).

The input to the critic is the environment state, which goes through the first fully connected layer. After batch normalization and ReLu activation it is concatenated with the action tensor and goes through the remaining 2 layers.

\subsection*{Training}
\subsubsection*{Loss function and optimizing method}

DDGP algorithm uses following loss function for critic network \ref{critic-loss}:
\begin{equation} \label{critic-loss}
L \equiv (R_t + \gamma Q(S_{t+1}, a_{t+1}; \theta'_t) - Q(S_t, a_t; \theta_t))^{2}
\end{equation}

Where \(Q(S_t, a_t; \theta_t)\) is output (value of action) of the critic network with internal parameters \(\theta_t\) given input \(S_t\) and \(a_t\). \(a_{t+1}\) is obtained from \(\pi(S_t; \eta'_t)\) (target actor) prediction with internal parameters \(\eta'_t\).

Actor loss function is following \ref{actor-loss}:
\begin{equation} \label{actor-loss}
L \equiv -(Q(S_t, \pi(S_t; \eta_t); \theta_t))
\end{equation}

Both loss functions used 2 sets of network parameters: target \(\theta'_t\), \(\eta'_t\) and local \(\theta_t\), \(\eta_t\). Target network parameters are updating through soft update \(\theta'_t = (1 - \tau) \theta'_t + \tau \theta_t\).
As optimizing method for both networks I used Adam with learning rate \(\alpha\).

\subsubsection*{Experience replay}
The input for training did not come directly from episodes. Instead I used experience replay technique (see \href{https://storage.googleapis.com/deepmind-media/dqn/DQNNaturePaper.pdf}{Human-level control through deep reinforcement learning}).

\subsubsection*{Exploratory noise}
In order to make the agent explore the environment DDPG algorithm adds noise in it actions during training. I used \href{https://en.wikipedia.org/wiki/Ornstein-Uhlenbeck_process}{Ornstein–Uhlenbeck process} for this purpose.

\section*{Results}

I came up using following training hyper parameters:
\begin{itemize}
	\item Buffer size: 10e6
	\item Batch size: 128
	\item \(\gamma\): 0.95
	\item \(\tau\): 0.001
	\item \(\alpha_{actor}\): 0.0001
	\item \(\alpha_{critic}\): 0.001
\end{itemize}
They were used across all training experiments below.

\subsection*{Training DDPG 128x256 128x256}

DDPG actor: 128x256, critic 128x(256+31).

Training history:

Episode 10.	Average Score: 1.29.	Time elapsed: 4:41

Episode 20.	Average Score: 2.01.	Time elapsed: 9:22

Episode 30.	Average Score: 3.98.	Time elapsed: 14:12

Episode 40.	Average Score: 8.78.	Time elapsed: 19:10

Episode 50.	Average Score: 13.40.	Time elapsed: 24:40

Episode 60.	Average Score: 16.83.	Time elapsed: 29:54

Episode 70.	Average Score: 19.30.	Time elapsed: 35:04

Episode 80.	Average Score: 21.39.	Time elapsed: 40:21

Episode 90.	Average Score: 22.95.	Time elapsed: 46:02

Episode 100.	Average Score: 24.32.	Time elapsed: 51:54

Episode 110.	Average Score: 27.81.	Time elapsed: 57:52

Environment solved in 17 episodes!	Average Score: 30.10.	Time elapsed: 62:04. See \ref{fig:DDPG_128x256}.

\begin{figure}
	\includegraphics[width=0.9\linewidth]{res/ddpg_128x256/score.png}
	\caption{DDPG 128x256: Rewards per episode}
	\label{fig:DDPG_128x256}
\end{figure}

\subsection*{Training DDPG 128x128 128x128}

DDPG actor: 128x128, critic 128x(128+31).

Episode 10.	Average Score: 0.48.	Time elapsed: 2:00

Episode 20.	Average Score: 0.95.	Time elapsed: 4:26

Episode 30.	Average Score: 1.47.	Time elapsed: 7:16

Episode 40.	Average Score: 2.20.	Time elapsed: 10:27

Episode 50.	Average Score: 3.02.	Time elapsed: 13:59

Episode 60.	Average Score: 4.29.	Time elapsed: 17:43

Episode 70.	Average Score: 5.77.	Time elapsed: 21:24

Episode 80.	Average Score: 7.87.	Time elapsed: 25:11

Episode 90.	Average Score: 9.93.	Time elapsed: 29:02

Episode 100.	Average Score: 12.19.	Time elapsed: 33:03

Episode 110.	Average Score: 15.77.	Time elapsed: 37:09

Episode 120.	Average Score: 19.19.	Time elapsed: 41:21

Episode 130.	Average Score: 22.43.	Time elapsed: 45:05

Episode 140.	Average Score: 25.19.	Time elapsed: 48:39

Episode 150.	Average Score: 27.81.	Time elapsed: 52:17

Episode 160.	Average Score: 29.93.	Time elapsed: 56:03

Environment solved in 61 episodes!	Average Score: 30.14.	Time elapsed: 56:25. See \ref{fig:DDPG_128x128}.

\begin{figure}
	\includegraphics[width=0.9\linewidth]{res/ddpg_128x128/score.png}
	\caption{DDPG 128x128: Rewards per episode}
	\label{fig:DDPG_128x128}
\end{figure}

\subsection*{Conclusion}

From the above results we can summarize, that actor: 128x256, critic 128x(256+31) networks architecture have better performance and learn faster than actor: 128x128, critic 128x(128+31). In overall DDPG algorithm performs very good in this environment.

\section*{Areas of improvement}

I also tried to implement \href{https://arxiv.org/pdf/1707.06347.pdf}{PPO algorithm}. I used some code form \href{https://github.com/ShangtongZhang/DeepRL}{Shangtong Zhang DeepRL github repository} to build more intuition about PPO and be more confident in correct implementation. But implemented algorithm almost didn't perform. Seems like something is still wrong. PPO algorithm is said to be more stable and perform better.

Another improvement could be implementing \href{https://arxiv.org/abs/1511.05952}{prioritized experience replay} in DDPG algorithm.

\end{document}


%
%
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\chapter{Fundamental Components in CPPINTS}
%
%
%
In this chapter the fundamental component classes in CPPINTS will be discussed in
detail. These classes forms foundation for building the recurrence
relation(RR).

\section{Basis Set}
%
% 1  what's the form of basis sets?
% 2  why in basis set class we only store l,m,n information?
%
\label{bs}

In quantum chemistry, the basis set functions is the fundamental
unit for practical calculations\cite{Davidson_Feller_CR_86_681_1986}.
Primarily the basis set functions can be divided into radial part
and angular part. The radial part for the basis set function we discuss
in CPPINTS is formed by Gaussian functions, and the angular part could
be further divided into two groups in terms of its function form.
One group uses the spherical harmonics:
\begin{equation}
 \chi = Y_{m}^{L}(\theta,\phi)e^{-\alpha r^{2}}
\end{equation}
The other group uses the Cartesian function:
\begin{equation}\label{basis_set_cart_form}
 \chi = x^{l}y^{m}z^{n}e^{-\alpha r^{2}}
\end{equation}
In this program, we only focus on the Cartesian type
Gaussian basis set functions.

Generally the basis set function $\psi$ is a linear combination of
Gaussian functions, and each such Gaussian function is also termed
as primitive function:
\begin{equation}\label{program_contract_basis_set}
	\psi = \sum_{\mu}d_{\mu}\chi_{\mu}
\end{equation}
$d$ is pre-optimized contraction coefficients.
$\chi_{\mu}$ is the primitive
functions as defined in \ref{basis_set_cart_form}.
All of $\chi$ are on the same center as $\psi$, and they all share the
same angular momentum with the basis set function.

For each basis set function, $x^{l}y^{m}z^{n}$ is its angular momentum part,
which is characterized by three number of l, m, and n. The $e^{-\alpha r^{2}}$
is its radial part, so l,m,n combined with exponential factor $\alpha$ and
its coefficient of $d_{\mu}$; that give all of information to get $\psi$.

In the recurrence relation(RR), typically it starts with the bottom integral
\footnote{for VRR, bottom integral is in form of $(00|00)^{(m)}$ etc. For
HRR, bottom integral is in form of $(0e|0f)$. See the chapter discussing RR
for more information} and by raising up angular momentums it reaches the target
integrals. Therefore in the basis set class, we only use $l,m,n$ to represent
the basis set.

In CPPINTS we also define the ``NULL'' basis set which means that
this basis set is physical meaningless (l, m and n are all set to be -1).
This type of basis set is used to complete an integral definition (see the
\ref{integral} for more definition).

The basis set is defined in the file basis.cpp and basis.h.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Shell}
%
% 1  what's the shell? Why shell only have one data member of L?
% 2  what is the basis set order? How can we change it?
%
\label{shell}

In quantum chemistry, shell is an aggregation of same type of basis set
functions; whose total angular momentum is same(sum of l,m,n). For example,
P shell contains 3 basis set functions, their l, m, n are characterized by:
\begin{align}\label{pshell_example}
	P_{x} &\Leftrightarrow (1,0,0) \nonumber \\
	P_{y} &\Leftrightarrow (0,1,0) \nonumber \\
	P_{z} &\Leftrightarrow (0,0,1)
\end{align}
By following the same principle, it's able the form D shell ($L = l+m+n = 2$),
F shell ($L = l+m+n = 3$) etc.

As what we have demonstrated in the basis set functions section, to fulfill
recurrence relation it only requires the angular momentum information.
Therefore, in the shell class it's only the total angular momentum L
is defined.

Here it needs to emphasize that how to arrange the basis set
functions in a given shell, this is called ``basis set order''.
For example, for the P shell, it's able to have:
\begin{equation}
 P_{x} \Rightarrow P_{y} \Rightarrow P_{z}
\end{equation}
or the other order of
\begin{equation}
 P_{z} \Rightarrow P_{y} \Rightarrow P_{x}
\end{equation}

Since different basis sets in a shell are in a same level, therefore
there's not a basis set order prior to the others. It's able to
pick up any basis set order theoretically. In this program, we
pick up the ``libint'' order (which is used by libint program
\footnote{\url{http://sourceforge.net/projects/libint/}}).
In basisutil.h, the arrangement of basis set order is given for
L up to 20. On the other hand, it's able to use the other
type of basis set orders. Therefore we separate the basis set
order information all into the basisutil.h and basisutil.cpp.

In terms of a given basis set order, each basis set occupies an
unique position in the
basis set order array, which refers as ``local index'' of the
basis set in the given shell. For example, in the above P shell
case \ref{pshell_example} $Px$ is 0, $Py$ is 1, $Pz$ is 2. For
each basis set it's able to set up some ``ONE-TO-ONE'' mapping
relation between basis set and it's position in the basis set
order list. Such fundamental relation is used for building the
mapping relation between shell quartet and integral (see
section \ref{mapping_integral_sq} for more details).

If the user wants to employ other type of basis set order other
than the libint order, here is the procedure the user should
follow:
\begin{itemize}
 \item generate all of explicit basis set order array and
	 replace the old content of ``LIBINT\_BASIS\_SET\_ORDER''
	 with the new array (this is in basisutil.h). This step
	 will enable you to get correct basis set by given
	 a basis set index;
 \item Secondly the function of ``getLocalBasisSetIndex'' in
	 basisutil.cpp should be updated for the new basis
	 set order. This step will give the correct local index
	 by an arbitrary input l, m, n value of basis set.
\end{itemize}
Now the mapping between index and angular momentum should
set up. You should get correct integral results without
referring to the other places.

Similar with basis set, we also define ``NULL'' shell whose
L is set to be -1. This type of shell is used to complete
shell quartet definition.

The shell is defined in shell.h and shell.cpp.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Composite Shells}
%
% 1  what is the composite shell?
% 2  how we handle the composite shell?
\label{composite_shell}
%
%
The composite shell is that for each shell it may contain several
type of sub-shells. For example, SP shell has one S shell and one
P shell, and SPD shell has S shell, P shell and a D shell. All of
these sub-shells share the same exponential factors for it's radial part,
and each of them has its own contraction coefficients of $d$ in equation
\ref{program_contract_basis_set}. The most famous basis set library using
composite shell is Pople basis sets (6-31G etc.)

In our program, the shell class is defined for ``pure'' shell(shell that
only corresponds to one $L$ value) and the composite shell situation is
handled elsewhere. You can refer to the section \ref{composite_shell_quartet}
etc. for more details that how CPPINTS handle the composite shell situation.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Integral Type}
%
% how to represent the integral operator?
%
\label{inttype}

In CPPINTS, we have a series of pre-defined integer to represent
the ``OPERATORS'' used in quantum chemistry integrals. For example,
the two body overlap integral(TOV), nuclear-electron attraction integral(NAI)
etc. These integers are defined in the inttype.h and inttype.cpp(it's
name appears as macro defined in general.h). The pre-defined integral
is a data member for both integral and shell quartet class.

For the given integral type, there are some integral properties that
is solely determined by the integral operator itself. For example,
NAI is two body integral, and it requires $(0|0)^{(m)}$ for RR etc.
You can find the details from inttype.cpp that what kind of properties
can be determined only from operator itself.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Integral and Shell Quartets}
%
%
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Integral}
\label{integral}
%
% 1  what is integral?
% 2  how to represent it?
%
For a given operator representing quantum quantity (for example,
the kinetic operator, electron repulsion operator etc.) it's able
to form the integrals based on the basis set functions:
\begin{equation}
 I = \langle \psi_{l_{1}m_{1}n_{1}}(r)\psi_{l_{2}m_{2}n_{2}}(r)|
 \hat{f}(r,r^{'})| \psi_{l_{3}m_{3}n_{3}}(r^{'})
 \psi_{l_{4}m_{4}n_{4}}(r^{'})\rangle
\end{equation}
therefore to define an integral, basically the following
information is needed:
\begin{itemize}
 \item operator;
 \item basis set information for all given integral positions
\end{itemize}
Because recurrence relation also uses the integrals $(ab|cd)^{(m)}$,
an additional $m$ value is defined in both integral and shell
quartet class.

Primarily the integrals could be divided into different categories according
to the number of its basis set function components. In quantum chemistry,
the possible number of basis set functions in the integral is ranging
from 1 to 4. Therefore, integral is ranging from one body integral to
four body integrals.

The four possible basis set positions are called as: ``BRA1'', ``BRA2'',
``KET1'' and ``KET2''. For one body integral, it's only the bra1
position having basis sets(other positions have null basis set);
for the two body integral, it's only bra1 and bra2 positions having
basis sets; for the three body integral,
it's only bra1, bra2, and ket1 positions having basis sets.
Such position definition is used across the whole program, and the
similar definition holds for shell quartet, too.

The integral class always has four position for the basis set (see
integral.h). For the one to three body integrals, we use NULL
basis set to complete the integral definition.

The integral is defined in integral.cpp and integral.h.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Shell Quartet}
%
% how to define shell quartet?
% what is the relation between shell quartet and integral?
%
\label{shell_quartet}
Similar to the relation between basis set and shell, the aggregation
of certain type of integrals will lead to the ``shell quartet''.

For example, for the electron repulsion operator it's able to define
the shell quartet over four shells:
\begin{equation}
 SQ_{P,P,P,P} = \langle P_{bra1}P_{bra2}|
 \frac{1}{r_{12}}|P_{ket1}P_{ket2} \rangle
\end{equation}
This shell quartet includes all of integrals in terms of basis sets
for the given four P shells. Because each P shell have 3 basis set
functions, the shell quartet contains $3^{4} = 81$ ERI integrals.

Similar to the integral class which is defined based on the basis set
class; the shell quartet is constructed based on the shell class. Therefore
for defining a fundamental shell quartet, it needs shell information
on all of four BRA1, BRA2, KET1 and KET2 positions (null shell is used
to complement the shell quartet definition) as well as the operation
information.

The definition of shell quartet could be referred to shellquartet.h
and shellquartet.cpp.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Mapping between Integral and Shell Quartet}
\label{mapping_integral_sq}

For CPPINTS, shell quartet is the basic unit for deriving the
recurrence relation. The integrals are considered to be attached to
its corresponding shell quartet.

Based on the ``ONE-TO-ONE'' mapping relation between basis set
and shell, it's able to construct the ``ONE-TO-ONE'' mapping relation
between integral and its shell quartet. For each integral, it
has a unique position according to the given basis set order.
For example, integrals in the shell quartet $(SPSP|SPSP)$ may
correspond to an order like:
\begin{align}
 (SS|SS)         &\rightarrow  0 \nonumber \\
 (P_{x}S|SS)     &\rightarrow  1 \nonumber \\
 (P_{y}S|SS)     &\rightarrow  2 \nonumber \\
 (P_{z}S|SS)     &\rightarrow  3 \nonumber \\
 (SP_{x}|SS)     &\rightarrow  4 \nonumber \\
 (P_{x}P_{x}|SS) &\rightarrow  5 \nonumber \\
 (P_{y}P_{x}|SS) &\rightarrow  6 \nonumber \\
 (P_{z}P_{x}|SS) &\rightarrow  7 \nonumber \\
                 &\cdots
\end{align}
By giving the shell quartet and this unique position, it's able
to reconstruct the integral; on the other hand, from the given
integral it's able to derive its unique position\footnote{please
refer to integral.cpp for more details that how we do it}.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Representation of Integral in Recurrence Relation}
%
% why we use position ID to reprernt integral in shell quartet?
%
\label{representation_integral_sq}

The ``ONE-TO-ONE'' mapping relation could be used to improve the memory
usage for CPPINTS. Suggest we have a $(II|II)$ ERI shell
quartet for deriving its vertical recurrence relation, to record
all of the information for each integrals requires the following
items:
\begin{itemize}
 \item LHS integral;
 \item RHS integral and its coefficients.
\end{itemize}

For each integral, to characterize it it requires $3\times 4 = 12$
integers to record angular momentum information, plus two integers
for operator and m value. Therefore each integral needs 14 integer
\footnote{we have not considered the division yet, which is related to
composite shell quartet case}.

For the OS VRR framework, the recursive relation has 8 items on the
RHS so there are totally 9 integrals, which needs 126 integers.
Because the $(II|II)$ has $28^4$ integrals, it needs 77446656 integers.
Suggest each integer takes 4 byte memory, the total memory for
storing integer of this shell quartet would be 296mb! So far we did not
count in the coefficients.

However, if we use the ``position ID'' to represent the integrals
inside the shell quartet, then each integral only need one integer
thus it's only 5.3mb memory needed. Therefore, to use the position
ID to represent integrals inside shell quartet is a good way for
memory saving with only marginal CPU cost added. This is the way
we handle integrals in the recurrence relation(RR) process.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Composite Shell Quartet}
%
%
\label{composite_shell_quartet}

In a given shell quartet, if one of its shell is in composite type;
then this shell quartet is composite shell quartet. It contains
several shell quartets in terms of pure shell. For example,
shell quartet $(DD|DSP)$ has two pure shell quartets; namely
$(DD|DS)$ and $(DD|DP)$.

How to process composite shell quartet in the recurrence relation?
It's depending on whether it's HRR or it's in VRR.

For VRR, Let's take ERI as example. Because the recurrence relation
is generally expressed as:
\begin{align}
I(L,m) &= a_{0}I_{0}(L-1,m) + a_{1}I_{1}(L-1,m+1) \nonumber \\
&+ a_{2}I_{2}(L-2,m) - a_{3}I_{3}(L-2,m+1) \nonumber \\
&+ a_{4}I_{4}(L-2,m) - a_{5}I_{5}(L-2,m+1) \nonumber \\
&+ a_{6}I_{6}(L-2,m+1) + a_{7}I_{7}(L-2,m+1)
\end{align}
The above integral expression is on primitive Gaussian function, and
$a_{i}$ is RR coefficient which is independent with the integral.
Because VRR is linear with the contraction coefficients $d$,
the expression above holds for both contracted and un-contracted primitive
Gaussian functions. Therefore, the recurrence relation for VRR is
independent of the contraction information. We can simply get the
integral with contraction coefficients by multiply it to the final
results:
\begin{equation}\label{composte_sq_contraction_coefs}
 I_{contracted}(L,m) = d_{bra1}d_{bra2}d_{ket1}d_{ket2}I(L,m)
\end{equation}
This is what we do for dealing with the composite shell quartets in VRR.
Because in composite shell the sub-shells share the same exponential
factor, the raw integral $I(L,m)$ in \ref{composte_sq_contraction_coefs}
is same between different pure shell quartets; the final integral results
with contraction coefficients could be simply retrieved by performing
\ref{composte_sq_contraction_coefs}. This is called ``contraction'' step
in VRR. The pseudocode is given like this:
\begin{verbatim}
loop over ket side primitive Gaussian pairs:
  loop over bra side primitive Gaussian pairs:
    compute bottom integrals;
    do VRR to derive result integrals;
    perform contraction step;
  end loop
end loop
\end{verbatim}
For pure shell quartet like $(DD|DD)$, there's no need to perform
contraction step because it can simply add contraction information
to the bottom integrals $(00|00)^{(m)}$ so that VRR is performed
for contracted integrals. After VRR is ending, the result integrals
get updated by the result in the primitive integral loops.

However, for HRR the story is different. Because HRR is usually applied on
contracted integral results (it sums over all of integrals on
primitive Gaussian functions), the equation \ref{composite_sq_hrr}
shows that how we can step from
un-contracted integrals to contracted ones in HRR:
\begin{align}\label{composite_sq_hrr}
 [a(b+\iota_{i})|cd]_{k} &= [(a+\iota_{i})b|cd]_{k} +
(A_{i} - B_{i})[ab|cd]_{k} \rightarrow \nonumber \\
\sum_{k}C_{k}[a(b+\iota_{i})|cd]_{k} &= \sum_{k}C_{k}[(a+\iota_{i})b|cd]_{k} +
(A_{i} - B_{i})\sum_{k}C_{k}[ab|cd]_{k} \rightarrow \nonumber \\
(a(b+\iota_{i})|cd) &= ((a+\iota_{i})b|cd) +
(A_{i} - B_{i})(ab|cd)
\end{align}
Here $[ab|cd]$ is used to represent the integrals on un-contracted Gaussian
functions, and $(ab|cd)$ for the contracted integrals; $C_{k}$ is the
combination for all of contraction coefficients of $d$.

It's clear from \ref{composite_sq_hrr} that as long as LHS and RHS share the
same contraction coefficients, HRR can be applied from $(0e|0f)$ until
the final target integral $(ab|cd)$ is derived. Hence for the composite
shell quartets, each pure shell quartet has its own HRR process and they
are independent with each other.

For this reason, we have a data member named as ``division'' on both
integral and shell quartet classes. It's an ID for each pure shell quartet
in the composite shell quartet. For example; $(DSP|DSP)$ has four shell quartets;
$(DS|DS)$, $(DS|DP)$, $(DP|DS)$ and $(DP|DP)$ and each pure shell quartet has it's
own ID number (characterized by division). They will have their own
HRR process.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Sorting Shell Quartets}
%
%
\label{sort_shell_quartet}

After RR(VRR or HRR) path is constructed in detail, a sorting
function is needed to be applied to the whole LHS shell quartets
so that to make sure all of RHS shell quartets are well defined
in the previous content. Therefore, we need sorting function for
building the correct RR.

The inconvertibility character for VRR establishes in section
\ref{optimal_path} in fact enables us to set up sorting facility
on the shell quartet in the RR direction. In other words, in
accumulating all of RR terms along the VRR path; we are able
to sort the shell quartets either from bottom integrals to
the results(this is always from RHS to LHS in terms of RR),
or from the results to the bottom integrals in reverse.

This is what the operator $<$ function in shell quartet class
does. By following the VRR formula\footnote{right now the function
just follows the VRR in OS framework}, it's able to distinguish
the shell quartet on either RHS or LHS. More details can be
found in the comments of this function.

HRR does not hold the inconvertibility character as shown in
section \ref{optimal_path}. However, it's still applicable
to set up some algorithm to sort the shell quartets in the
HRR process. The sorting function for HRR is presented in
hrrCompare function in shellquartet.cpp. Therefore, it's
able to sort the HRR shell quartets so that we print out
the HRR formula in correct order.


\section{Introduction}

Achieving an optimal runtime complexity is a fundamental challenge when it comes to designing scalable solutions for handling a large amount of data\cite{sipser13}. Recently, quantum computing has sparked the interest of both scientists and engineers due to its asymptotical performance gain on some problems compared to classical computing. While the field has witnessed rapid progress in developing new software, hardware, and efficient algorithms, the core principles at play have been relatively stable. In almost every physical or abstract quantum computer, information encoding and processing occur via unitary transformations while the result is written onto some classical storage via projective measurements\cite{Nielsen2009}. In this project, we aim to investigate \emph{measurement-based quantum computing}, a different formulation for quantum computation that provides several advantages over the circuit based model.

Measurement-based quantum computing was first proposed by Briegel and Russeldorf in 2000 \cite{Briegel_2001} as a general framework for universal computation via using the entanglement patterns of two-state particles. They later extended their work to include a formulation based on stabilizers, whose use in quantum mechanics is mainly restricted to error correction \cite{Nielsen2009, quant-ph/9705052}. The structure of a measurement-based quantum computer was further generalized to encompass the use of graph states, which are used to represent qubits with Ising interaction patterns via simple graphs \cite{hein2006}. Recently, several algorithms formulated for measurement-based quantum computing have been suggested \cite{keith2014, debeaudrap2008theory, Fitzsimons2017}.

In this project, we start by reviewing the necessary mathematical background, where we introduce graphs and their relevant properties. We use these graph states as a general framework for studying the entanglement properties of special qubit systems, which can be represented by simple graphs. We use two different mathematical formulations for our exposition of graph states. One such way of formulating these objects is by using their interaction patterns. As graph states interact via specific Ising interactions, their states can be expressed by applying the well-known Ising interaction hamiltonians\cite{ichikawa2013}. The second formulation uses stabilizer formalism to achieve a more compact and transparent representation of the graph state and its entanglement patterns.

In the second phase of the project, we introduce the notion of a \emph{cluster state} which is a regular lattice-like graph state that will be a general substrate for universal quantum computation. Following this introduction, we define an abstract machine called the one-way quantum computer, which takes some measurement patterns and executes them on its cluster. Furthermore, we provide a method to simulate circuit-based quantum computation using this abstract machine to prove its universal computation capability.


\begin{savequote}[90mm]
  {\QuoteFont Praesent metus ligula, auctor vitae, lacinia sed.}
\qauthor{- Chao Chao}
\end{savequote}

\chapter{Example of Math}
\label{chap:chapter_04}

Mauris mollis condimentum risus. Integer ipsum. Quisque
malesuada, erat ac dictum pulvinar, magna nisl fermentum ligula,
quis euismod mauris felis non diam. Nullam sapien turpis, rutrum
vel, condimentum ac, bibendum vulputate, nulla. Vestibulum tortor
ipsum, fermentum egestas, placerat ut, vulputate et, wisi. Aliquam
erat volutpat. Each element is defined as $t_{seek}$,$t_{rot}$, and
$t_{trans}$, respectively. Donec consequat, ligula sit amet tincidunt aliquam,
nunc lorem sagittis nunc, a ullamcorper erat ante ac felis. Donec
eleifend. Nullam quam leo, lobortis non, condimentum at, tempus
consectetuer, orci. Quisque ut lorem.  Vestibulum ante ipsum primis
in faucibus orci luctus et ultrices posuere cubilia Curae; Donec
porta, libero eget feugiat posuere, felis arcu pulvinar odio, vel
dapibus enim dui nec turpis. \begin{equation}
\label{eq:attf}
t_{HDD}(f_{n}) = t_{seek}(f_{n}) +t_{rot}(f_{n}) + t_{trans}(f_{n})
\end{equation}
Suspendisse porta, dolor sed fringilla
ultrices, augue mauris gravida dolor, vel sollicitudin magna dui sit
amet nunc.

Donec nisl. Lorem ipsum dolor sit amet, consectetuer adipiscing elit. More
detailed description of the expressions is found in \cite{miner,CloSpan}. Let
$F=\{ f_{1}, ~ f_{2}, ..., ~ f_{n} \}$ be a set of all files accessed during
launch of an application.  A sequence
$\sigma = < s_{1}, ~ s_{2}, ..., ~ s_{i} >$
is an ordered list and is a subset of $F$ such that
$\{ s_{n} \sqsubseteq F \}$.
Given two sequences
$\alpha = < a_{1}, ~ a_{2}, ..., ~ a_{n} >$ and
$\beta = < b_{1}, ~ b_{2}, ..., ~ b_{m} >$, sub-sequence is defined as follows:
Sequence  $\alpha$ is a sub-sequence of another sequence $\beta$, if and only
if there exist $i_{1}, ~ i_{2}, \ldots, ~ i_{m}$ such that
$1 \le i_{1} < i_{2} < \ldots < i_{m} \le n$  and
$a_{1} \subseteq b_{i1},  ~ a_{2} \subseteq b_{i2} , \ldots and  a_{m} \subseteq b_{im}$.
If $\alpha \neq \beta$ then it is denoted as $ \alpha \subset \beta $.
If files in sequence $\alpha$ are in the
same order as sequence $\beta$, then we call $\beta$ a super-sequence of
$\alpha$.  A sequence database,
$D = \{ \sigma_{1}, ~ \sigma_{2}, ..., ~ \sigma_{i} \}$,
is a set of sequences, and $|D|$ is the number of sequences in
the database $D$. The support of a sequence $\alpha$ in $D$ is the number of
sequences of $D$ which contains $\alpha$ as a subsequence. We define frequent
sub-sequence as sequences with the support greater than or equal to $min\_sup$,
where $min\_sup$ is the minimum support threshold.

{\bfseries \noindent Example 1.} Suppose a database $D$ with $|D| = 5$ has the
following set of sequences $\{ ab, ~ acde, ~ bcdf, ~ abcd, ~ abcf \}$. Then, 2
is the support of $abc$ and 3 is the support of $ac$ in $D$. Suppose
$min\_sup = 3$, then $abc$ is not a frequent sub-sequence but $ac$ is a frequent
sub-sequence. $\blacksquare$

\lipsum[1-4]


\RequirePackage[l2tabu, orthodox]{nag}
\documentclass{article}

\usepackage[letterpaper, margin=1.3cm]{geometry}
\usepackage{booktabs}
\usepackage[final]{pdfpages}

\title{ECE 487 Assignment 1}
\author{Michael Kwok}
\begin{document}

\maketitle
\subsection*{Links requied to connect 15 nodes}
\begin{itemize}
    \item Mesh: \((15 * 14) / 2 = 105\)
    \item Star: 15
    \item Bus: 15
    \item Ring: 15
\end{itemize}

\subsection*{Topology diagram}

Attached on 2nd page.

\subsection*{Match to layer}
\begin{itemize}
    \item Transport
    \item Data Link
    \item Transport
    \item Presentation
\end{itemize}

\subsection*{Data frame headers}
\begin{tabular}{ l  c  c  c  c }
    \toprule
                   & \multicolumn{2}{c}{Data frame from PC2 to the router} & \multicolumn{2}{c}{Data frame from router to PC3}                                        \\
    \midrule
                   & Source address                                        & Destination address                               & Source address & Destination address \\
    \midrule
    Layer 2 Header & 25                                                    & 67                                                & 38             & 46                  \\
    Layer 3 Header & C                                                     & B                                                 & B              & B                   \\
    Layer 4 Header & k                                                     & m                                                 & k              & m                   \\
    \bottomrule
\end{tabular}
\includepdf[pages=-]{StarRing.pdf}
\end{document}


% This file contains the content for a main section
\regularsectionformat	% Change formatting to that of "Introduction" section
%% Modify below this line %%
\chapter{Specification}

\section{Naming conventions}
The encoding of ACES specified in \autoref{sec:ACEScg} shall be known as ACEScg.

\section{Color component value encoding}
ACEScg shall be stored as either 16-bit (IEEE binary16) or 32-bit (IEEE binary32) floating point values.

\section{Color component value range}
The value range for ACEScg color component values is [-65504.0, +65504.0].

The chromaticity coordinates of the defined ACEScg RGB primaries (AP1) form a triangle on the CIE chromaticity diagram. ACEScg RGB values which express visible colors are represented by points within this triangle that also lie within the visual gamut.

The set of valid ACEScg RGB values also includes members whose projection onto the CIE chromaticity diagram falls outside the region of the AP1 primaries. These ACEScg RGB values include those with one or more negative ACEScg color component values; Ideally these values would be preserved through any compositing operations done in ACEScg space but it is recognized that keeping negative values is not always practical, in which case it will be acceptable to replace negative values with zero.

Values well above 1.0 are expected and should not be clamped except as part of the color correction needed to produce a desired artistic intent.

\section{Color component transfer function}
The color component transfer function directly encodes relative exposure values and is defined as
\begin{center}
$R = E_r, \quad G = E_g, \quad B = E_b$
\end{center}
where $E_r$, $E_g$ and $E_b$ represent relative exposure values that would be captured from the scene by the ACES Reference Image Capture Device (RICD) and $R$, $G$ and $B$ are the resulting ACES color component values transformed to ACEScg using the methods specified in section 4.1.6.

\section{Color space chromaticities}
\label{sec:colorspace}
ACEScg uses a different set of primaries than ACES RGB primaries defined in SMPTE ST 2065-1. The CIE 1931 colorimetry of the ACEScg RGB primaries and white are specified below.

\subsection{Color primaries}
The RGB primaries chromaticity values, known as AP1, shall be those found in \autoref{table:AP1rgb}.

\begin{center}
\begin{tabularx}{4.5in}{XlllXll}
        & R       & G       & B       & & CIE x & CIE y \\ \hline
Red     & 1.00000 & 0.00000 & 0.00000 & & 0.713 & 0.293 \\
Green   & 0.00000 & 1.00000 & 0.00000 & & 0.165 & 0.830 \\
Blue    & 0.00000 & 0.00000 & 1.00000 & & 0.128 & 0.044 \\
\end{tabularx}
\captionof{table}{ACEScg RGB primaries chromaticity values}
\label{table:AP1rgb}
\end{center}

\subsection{White Point}
The white point shall be that found in \autoref{table:AP1w}.

\begin{center}
\begin{tabularx}{4.5in}{XlllXll}
        & R       & G       & B       & & CIE x & CIE y \\ \hline
White   & 1.00000 & 1.00000 & 1.00000 & & 0.32168 & 0.33767 \\
\end{tabularx}
\captionof{table}{ACES RGB white point chromaticity values}
\label{table:AP1w}
\end{center}

\note{The ACEScg white point is the same as the white point of ACES 2065-1.}

\section{ACEScg}
\label{sec:ACEScg}
The following functions shall be used to convert between ACES values, encoded according to SMPTE ST 2065-1, and ACEScg.

\subsection{Converting ACES2065-1 RGB values to ACEScg RGB values}
\label{sec:aces2acescg}
ACES $R$, $G$, and $B$ values shall be converted to ACEScg $R$, $G$, and $B$ values using the transformation matrix ($TRA$) calculated and applied using the methods provided in Section 4 of SMPTE RP 177:1993.

\note{Equation \ref{eq:aces2acescg} shows the relationship between ACES $R$, $G$, and $B$ values and ACEScg $R$, $G$, and $B$ values. $TRA_{1}$, rounded to 10 significant digits, is derived from the product of $NPM_{AP1}$ inverse and $NPM_{AP0}$ calculated using methods provided in Section 3.3 of SMPTE RP 177:1993. AP0 are the primaries of ACES specified in SMPTE ST 2065-1:2012. AP1 are the primaries of ACEScg specified in \autoref{sec:colorspace}.}

\begin{floatequ}
\begin{gather}
    \begin{bmatrix}
        R_{ACEScg}\\
        G_{ACEScg}\\
        B_{ACEScg}
    \end{bmatrix}
    =
    TRA_{1}
    \cdot
    \begin{bmatrix}
        R_{ACES}\\
        G_{ACES}\\
        B_{ACES}
    \end{bmatrix} \\
    \\
    TRA_{1} =
    \begin{bmatrix*}[r]
        1.4514393161 & -0.2365107469 & -0.2149285693 \\
       -0.0765537734 &  1.1762296998 & -0.0996759264 \\
        0.0083161484 & -0.0060324498 &  0.9977163014 \\
    \end{bmatrix*} \\
    \\
    TRA_{1} = NPM^{-1}_{AP1} \cdot NPM_{AP0}
\end{gather}
\caption{ACES2065-1 to ACEScg}
\label{eq:aces2acescg}
\end{floatequ}

\subsection{Converting ACEScg RGB values to ACES2065-1 RGB values}
ACEScg $R$, $G$, and $B$ values shall be converted to ACES2065-1 $R$, $G$ and $B$ using the transformation matrix ($TRA$) calculated and applied using the methods provided in Section 4 of SMPTE RP 177:1993.

\note{Equation \ref{eq:acescg2aces} shows the relationship between ACES $R$, $G$, and $B$ values and ACEScg $R$, $G$, and $B$ values. $TRA_{2}$, rounded to 10 significant digits, is derived from the product of $NPM_{AP0}$ inverse and $NPM_{AP1}$ calculated using methods provided in Section 3.3 of SMPTE RP 177:1993. AP0 are the primaries of ACES specified in SMPTE ST 2065-1:2012. AP1 are the primaries of ACEScg specified in \autoref{sec:colorspace}.}

\begin{floatequ}
\begin{gather}
    \begin{bmatrix}
        R_{ACES}\\
        G_{ACES}\\
        B_{ACES}
    \end{bmatrix}
    =
    TRA_{2}
    \cdot
    \begin{bmatrix}
        R_{ACEScg}\\
        G_{ACEScg}\\
        B_{ACEScg}
    \end{bmatrix} \\
    \\
    TRA_{2} =
    \begin{bmatrix*}[r]
        0.6954522414 & 0.1406786965 & 0.1638690622 \\
        0.0447945634 & 0.8596711185 & 0.0955343182 \\
        -0.0055258826 & 0.0040252103 & 1.0015006723 \\
    \end{bmatrix*} \\
    \\
    TRA_{2} = NPM^{-1}_{AP0} \cdot NPM_{AP1}
\end{gather}
\caption{ACEScg to ACES2065-1}
\label{eq:acescg2aces}
\end{floatequ}


\section{Plugin for the RAVEN Code}
\label{sec:RavenPlugin}

The deterministic model can be repeatedly solved by changing the input values (i.e.,
\xmlNode{available\_capitals} [$b_{kt}$] and \xmlNode{net\_present\_values} [$a_{ij}$]).
This will allow for what-if sensitivity analysis to identify the
crucial drivers behind the optimal project selection decision. Monte Carlo simulation permits a
powerful variant of this approach in which we model $a_{ij}$ and $b_{kt}$ as random
variables, sample from their distributions, and perform a form of uncertainty quantification
in terms of the resulting distributions governing the binary decisions selected, $x_{ij}$, and
the overall NPV of the selected portfolio.

Example RAVEN input \xmlNode{ExternalModel} XML:
\begin{lstlisting}[style=XML]
<Models>
  <ExternalModel name="singleKnapsack" subType="LOGOS.CapitalInvestmentModel">
    <variables>available_capitals,i1,i2,i3,i4,i5,i6,i7,i8,i9,i10,
      MaxNPV</variables>
    <ModelData>
      <Sets>
        <investments>
          i1,i2,i3,i4,i5,i6,i7,i8,i9,i10
        </investments>
      </Sets>
      <Parameters>
        <net_present_values index="investments">
          18,20,17,19,25,21,27,23,25,24
        </net_present_values>
        <costs index="investments">
          1,3,7,4,8,9,6,10,2,5
        </costs>
        <available_capitals>
          15
        </available_capitals>
      </Parameters>
      <Settings>
        <solver>glpk</solver>
        <sense>maximize</sense>
      </Settings>
    </ModelData>
  </ExternalModel>
</Models>
\end{lstlisting}

As the name suggests, an external model is an entity that is embedded in the RAVEN
code at run time. This object allows the user to import the LOGOS module that will
be treated as a predefined internal RAVEN object. In other words, the
\textbf{External Model} will be treated by RAVEN as a normal external model.
Check the RAVEN user manual for a more detailed description.
\nb The value for attribute \xmlAttr{subType} should always be \xmlString{LOGOS.CapitalInvestmentModel}.

\xmlNode{variables} specifies a list of variable names that need to match
variables used/defined in the LOGOS model. In the above example, the variable
\xmlString{available\_capitals} is sampled by RAVEN, and its value is used
by LOGOS instead of using the default values specified by \xmlNode{ModelData}.
The decision variables \xmlString{i1,i2,i3,i4,i5,i6,i7,i8,i9,i10} and the
objective variable \xmlString{MaxNPV} are collected from the output of the LOGOS model,
and can be stored in the RAVEN data objects.
The XML node \xmlNode{ModelData} is used to specify the LOGOS optimization problem, which
should be consistent with the LOGOS input XML file.

\subsection{Test Automation}
Automated regression testing is a development methodology generally used
to verify the correctness and performance of software after each modification.
This methodology is integrated directly into GitHub and GitLab for RAVEN and
RAVEN-supported plugins. In this case, testing is performed automatically as part of the
continuous integration system (CIS) process whenever a user commits a change to the
repository. Tests of changes across multiple platforms are executed with each pull
request. Results from each test execution are maintained in an approved records
repository in the CIS database, along with results from the timing executions.

\subsection{Testing System Prerequisites}
The module test system consists of scripts written in Bash shell language
and Python. It may be used on any platform supported by RAVEN (i.e.
Linux, Mac, and Windows). The following conditions must be satisfied for the
module test system to function properly:
\begin{itemize}
  \item RAVEN should be installed and updated. This is because RAVEN is used to
  run the module tests;
  \item The system running the tests must be configured with the software prerequisites
  necessary to build and run RAVEN. These include a Python interpreter, Python
  libraries (h5py, matplotlib, numpy, scipy, and scikit-learn), and development
  tools (C++ compiler, Miniconda package manager for Python, and Git source code control);
  \item RAVEN must be built with the appropriate compiler before it can be used to
  run the tests;
  \item The LOGOS submodule must be initialized and fully updated. Additional prerequisites
  include PYOMO, glpk, and coincbc;
\end{itemize}

\subsection{Test Location and Definition}
LOGOS is a RAVEN-supported plugins and managed by Git. RAVEN plugins afford
an option to associate a workflow or a set of RAVEN external models to RAVEN
without having them included in the RAVEN main repository. The benefits include
modularity, access restriction, and regression testing for compatibility with RAVEN
as it continues to grow. In this case, we are able to treat this repository as
separate, yet still be able to use one from within the other. The main structure
of the LOGOS repository is shown in Figure~\ref{fig:Logos}. The folder \textit{tests} contains the input of
the tests, in addition to the corresponding output (in the gold folder) used for
assurance that the behavior of the code is not changed for new modifications. These
tests can also be collected in subfolders based on their characteristics.

\begin{figure}
    \centering
    \centerline{\includegraphics[scale=0.5]{LogosRepo.jpg}}
    \caption{Folder tree of the Logos repository.}
    \label{fig:Logos}
\end{figure}

The RAVEN repository contains a complete testing system for providing regression
testing for itself and its plugins. The LOGOS module tests are defined in the
same manner as for RAVEN. A single test consists of a RAVEN input file
along with associated data needed to perform that run. This can include input data,
external models, and Python files. These may be placed in the \textit{tests} directory
or a related subdirectories. Each directory that contains tests to be run by
the framework must contain a test specification file named “tests”. The syntax of
these files is defined by the RAVEN test framework, which controls how each test
is run and sets the criteria used to determine whether it passed or not. An example
of a test specification file is presented here:
\begin{lstlisting}[language=python]
[Tests]
  [./skp_optimization]
    type = 'RavenFramework'
    input = 'test_skp.xml'
    UnorderedCsv = 'skp/test_skp.csv'
  [../]
[]
\end{lstlisting}
In the above example, one test, named ``skp\_optimization'', is defined using the RAVEN
test module (defined in the test file as “type = `RavenFramework'”). Comparison
criteria are also defined in the “tests” file. In most cases, one or more output
files generated by running the specified input file with RAVEN are compared
against a gold standard provided by the developer and stored in the repository.
Typically, comparisons are performed on numeric values contained in
CSV files up to a defined tolerance. When these file comparisons are specified
by the test developer, reference files must have the same name and be placed
in the gold subdirectory below that containing the “tests” file.

\subsection{Running the Tests}
The integrated tests can be run separately, as indicated by following this pathway:
\begin{lstlisting}[language=bash]
path/to/raven/raven_framework path/to/LOGOS/tests/TestName
\end{lstlisting}

\subsection{Continuous Integration System (CIVET)}
CIVET, developed at INL, is used for continuous integration, verification,
enhancement, and testing of RAVEN and LOGOS. Each time a developer
proposes modification of the contents of the LOGOS repository, CIVET will cause
the automated tests to be run on the modified version. These tests must all pass
before a proposed change can become part of the official repository. In this way,
the LOGOS project is protected from the accidental introduction of flaws into
software that required such a significant investment of resources to develop.


\section{Geneset analysis}

A hypergeometric test is used to test for the enrichment of GO, KEGG and msigdb genesets amongst the positive marker genes for each cluster. The full results
are available as a separate xlsx document.

\input{\clusterGenesetsDir/cluster.genesets.figure.tex}

\clearpage

\subsection{Top genesets by cluster}

The table lists the top (filtered by nominal p value) genesets by cluster.

\input{\clusterGenesetsDir/cluster.genesets.table.tex}

\clearpage


\section{Conclusions}

We have demonstrated that even a basic \acrshort{ESN} can
predict the evolution of dynamic systems as others have
\cite{pathak_model-free_2018,wikner_combining_2020,bianchi_reservoir_2020}.
Future work will include:
\begin{enumerate}
  \item Identifying ideal input vectors, whether a single value for net demand
  history will suffice or some combination of values (e.g. local weather and
  total demand) will improve predictive power.
  \item Grid searches to tune hyperparameters.
  \item Taking advantage of fast model training to perform uncertainty analysis.
\end{enumerate}
Fairer comparisons on the effectiveness of training length will be done by
fixing the prediction time period for each ESN.
Accurate predictions of chaotic systems, like wind energy production, will
enable nuclear power plants to improve their economic feasibility through
relaxed load following.


% !TeX root = ../main.tex

\section{Neighborhood-based Collaborative Filtering}
\subsection{Introduction}
\begin{frame}
    \frametitle{Introduction}
    \centering
    \underline{\textbf{Collaborative Filtering}}
    \begin{itemize}
        \item \textbf{User-based}
            \begin{itemize}
                \item Many People like "The Godfather" should I watch it too?
                \item Choose a number of users who like things I like and
                      decide based on how much they liked it
                \item If we used to like similar items in the past, we will continue to like similar items in the future
            \end{itemize}
        \item \textbf{Item-based}
            \begin{itemize}
                \item Is "Jurassic Park" a good choice based on movies I usually see?
                \item Choose a number of movies that I have seen and share similar audience with "Jurassic Park",
                then decide based on how much I liked the previous movies
                \item If I liked these type of items in the past, I will probably also like those items
            \end{itemize}
    \end{itemize}
\end{frame}
\begin{frame}[t]
    \frametitle{Introduction}
    \centering
    \underline{\textbf{Advantages of Collaborative Filtering}}
    \begin{center}
        \begin{enumerate}
            \item Simplicity
            \item Justifiability
            \item Efficiency
            \item Stability
        \end{enumerate}
    \end{center}
\end{frame}

\subsection{Similarity Function Variants}
\begin{frame}
    \frametitle{Cosine Similarity}
    \begin{columns}
        \column{0.5\textwidth}
        \centering
        \underline{\textbf{User-based}}
        \begin{equation*}
            cos(u,v) = \frac{\sum_{i \in \mathcal{I}_{uv}}r_{ui}r_{vi}}
        		    {\sqrt{\sum_{i \in \mathcal{I}_{u}}r_{ui}^2}
        		     \sqrt{\sum_{i \in \mathcal{I}_{v}}r_{vi}^2}}
        \end{equation*}
        \tiny
        \begin{itemize}
            \item $\mathcal{I}_{u}:$ The set of items that have been rated by user $u$
            \item $\mathcal{I}_{v}:$ The set of items that have been rated by user $v$
            \item $\mathcal{I}_{uv}:$ The set of items that users $u$ and $v$ rated in common
            \item $r_{ui}:$ The rating that user $u$ gave to item $i$
            \item $r_{vi}:$ The rating that user $v$ gave to item $i$
        \end{itemize}
        \column{0.5\textwidth}
        \centering
        \underline{\textbf{Item-based}}
        \begin{equation*}
            cos(i,j) = \frac{\sum_{u \in \mathcal{U}_{ij}}r_{iu}r_{ju}}
        		    {\sqrt{\sum_{u \in \mathcal{U}_{i}}r_{iu}^2}
        		     \sqrt{\sum_{u \in \mathcal{U}_{j}}r_{ju}^2}}
        \end{equation*}
        \tiny
        \begin{itemize}
            \item $\mathcal{U}_{i}:$ The set of users that have rated item $i$
            \item $\mathcal{U}_{j}:$ The set of users that have rated item $j$
            \item $\mathcal{U}_{ij}:$ The set of users that have both rated items $i$ and $j$
            \item $r_{iu}:$ The rating item $i$ received from user $u$
            \item $r_{ju}:$ The rating item $j$ received from user $u$
        \end{itemize}
    \end{columns}
\end{frame}
\begin{frame}
    \frametitle{Modified Cosine Similarity}
    \begin{columns}
        \column{0.5\textwidth}
        \centering
        \underline{\textbf{User-based}}
        \begin{equation*}
            MC(u,v) = \frac{\sum_{i \in \mathcal{I}_{uv}}r_{ui}r_{vi}}
		   {\sqrt{\sum_{i \in \mathcal{I}_{uv}}r_{ui}^2}
                    \sqrt{\sum_{i \in \mathcal{I}_{uv}}r_{vi}^2}}
        \end{equation*}
        \tiny
        \begin{itemize}
            \item $\mathcal{I}_{uv}:$ The set of items that users $u$ and $v$ rated in common
            \item $r_{ui}:$ The rating that user $u$ gave to item $i$
            \item $r_{vi}:$ The rating that user $v$ gave to item $i$
        \end{itemize}
        \column{0.5\textwidth}
        \centering
        \underline{\textbf{Item-based}}
        \begin{equation*}
            MC(i,j) = \frac{\sum_{u \in \mathcal{U}_{ij}}r_{iu}r_{ju}}
		   {\sqrt{\sum_{u \in \mathcal{U}_{ij}}r_{iu}^2}
                    \sqrt{\sum_{u \in \mathcal{U}_{ij}}r_{ju}^2}}
        \end{equation*}
        \tiny
        \begin{itemize}
            \item $\mathcal{U}_{ij}:$ The set of users that have both rated items $i$ and $j$
            \item $r_{iu}:$ The rating item $i$ received from user $u$
            \item $r_{ju}:$ The rating item $j$ received from user $u$
        \end{itemize}
    \end{columns}
\end{frame}
\begin{frame}
    \frametitle{Adjusted Cosine Similarity}
    \vspace{-0.8cm}
    \begin{columns}
        \column{0.5\textwidth}
        \centering
        \underline{\textbf{User-based}}
        \begin{equation*}
            \small
            \begin{split}
            &AC(u,v) = \frac{\sum_{i \in \mathcal{I}_{uv}}(r_{ui}-\bar{r_{i}})(r_{vi}-\bar{r_{i}})}
        		    {\sqrt{\sum_{i \in \mathcal{I}_{uv}}(r_{ui}-\bar{r_{i}})^2}
                             \sqrt{\sum_{i \in \mathcal{I}_{uv}}(r_{vi}-\bar{r_{i}})^2}} \\\\
            &\bar{r_{i}} = \frac{\sum_{u \in \mathcal{U}_i}r_{iu}}
         		        {\mathopen|\mathcal{U}_i\mathclose|}
            \end{split}
        \end{equation*}
        \tiny
        \begin{itemize}
            \item $\mathcal{I}_{uv}:$ The set of items that users $u$ and $v$ rated in common
            \item $r_{ui}:$ The rating that user $u$ gave to item $i$
            \item $r_{vi}:$ The rating that user $v$ gave to item $i$
            \item $\mathcal{U}_{i}:$ The set of users that have rated item $i$
            \item $\bar{r_{i}}:$ The mean rating of item $i$

        \end{itemize}
        \column{0.5\textwidth}
        \centering
        \underline{\textbf{Item-based}}
        \begin{equation*}
            \small
            \begin{split}
            &AC(i,j) = \frac{\sum_{u \in \mathcal{U}_{ij}}(r_{iu}-\bar{r_{u}})(r_{ju}-\bar{r_{u}})}
        		    {\sqrt{\sum_{u \in \mathcal{U}_{ij}}(r_{iu}-\bar{r_{u}})^2}
                             \sqrt{\sum_{u \in \mathcal{U}_{ij}}(r_{ju}-\bar{r_{u}})^2}} \\\\
            &\bar{r_{u}} = \frac{\sum_{i \in \mathcal{I}_u}r_{ui}}
         		        {\mathopen|\mathcal{I}_u\mathclose|}
            \end{split}
        \end{equation*}
        \tiny
        \begin{itemize}
            \item $\mathcal{U}_{ij}:$ The set of users that have both rated items $i$ and $j$
            \item $r_{iu}:$ The rating item $i$ received from user $u$
            \item $r_{ju}:$ The rating item $j$ received from user $u$
            \item $\mathcal{I}_{u}:$ The set of items that have been rated by user $u$
            \item $\bar{r_{u}}:$ The mean rating of user $u$
        \end{itemize}
    \end{columns}
\end{frame}
\begin{frame}
    \frametitle{Modified Adjusted Cosine Similarity}
    \vspace{-0.6cm}
    \begin{columns}
        \column{0.5\textwidth}
        \centering
        \underline{\textbf{User-based}}
        \begin{equation*}
        \footnotesize
        \begin{split}
        &MAC(u,v) = \frac{\sum_{i \in \mathcal{I}_{uv}}(r_{ui}-\bar{r_{i}})(r_{vi}-\bar{r_{i}})}
                         {\sqrt{\sum_{i \in \mathcal{I}_{u}}(r_{ui}-\bar{r_{i}})^2}
                          \sqrt{\sum_{i \in \mathcal{I}_{v}}(r_{vi}-\bar{r_{i}})^2}} \\\\
        &\bar{r_{i}} = \frac{\sum_{u \in \mathcal{U}_i}r_{iu}}
                            {\mathopen|\mathcal{U}_i\mathclose|}
        \end{split}
        \end{equation*}
        \tiny
        \begin{itemize}
            \item $\mathcal{I}_{u}:$ The set of items that have been rated by user $u$
            \item $\mathcal{I}_{v}:$ The set of items that have been rated by user $v$
            \item $\mathcal{I}_{uv}:$ The set of items that users $u$ and $v$ rated in common
            \item $r_{ui}:$ The rating that user $u$ gave to item $i$
            \item $r_{vi}:$ The rating that user $v$ gave to item $i$
            \item $\mathcal{U}_{i}:$ The set of users that have rated item $i$
            \item $\bar{r_{i}}:$ The mean rating of item $i$
        \end{itemize}
        \column{0.5\textwidth}
        \centering
        \underline{\textbf{Item-based}}
        \begin{equation*}
        \footnotesize
        \begin{split}
        &MAC(i,j) = \frac{\sum_{u \in \mathcal{U}_{ij}}(r_{iu}-\bar{r_{u}})(r_{ju}-\bar{r_{u}})}
                         {\sqrt{\sum_{u \in \mathcal{U}_{i}}(r_{iu}-\bar{r_{u}})^2}
                          \sqrt{\sum_{u \in \mathcal{U}_{j}}(r_{ju}-\bar{r_{u}})^2}} \\\\
        &\bar{r_{u}} = \frac{\sum_{i \in \mathcal{I}_u}r_{ui}}
                            {\mathopen|\mathcal{I}_u\mathclose|}
        \end{split}
        \end{equation*}
        \tiny
        \begin{itemize}
            \item $\mathcal{U}_{i}:$ The set of users that have rated item $i$
            \item $\mathcal{U}_{j}:$ The set of users that have rated item $j$
            \item $\mathcal{U}_{ij}:$ The set of users that have both rated items $i$ and $j$
            \item $r_{iu}:$ The rating item $i$ received from user $u$
            \item $r_{ju}:$ The rating item $j$ received from user $u$
            \item $\mathcal{I}_{u}:$ The set of items that have been rated by user $u$
            \item $\bar{r_{u}}:$ The mean rating of user $u$
        \end{itemize}
    \end{columns}
\end{frame}
\begin{frame}
    \frametitle{Pearson Correlation Coefficient}
    \vspace{-0.8cm}
    \begin{columns}
        \hspace{-7mm}
        \column{0.5\textwidth}
        \centering
        \underline{\textbf{User-based}}
        \begin{equation*}
            \small
            \begin{split}
            &PCC(u,v) = \frac{\sum_{i \in \mathcal{I}_{uv}}(r_{ui}-\bar{r_{u}})(r_{vi}-\bar{r_{v}})}
                             {\sqrt{\sum_{i \in \mathcal{I}_{uv}}(r_{ui}-\bar{r_{u}})^2}
                              \sqrt{\sum_{i \in \mathcal{I}_{uv}}(r_{vi}-\bar{r_{v}})^2}} \\\\
            &\bar{r_{u}} = \frac{\sum_{i \in \mathcal{I}_u}r_{ui}}
                                {\mathopen|\mathcal{I}_u\mathclose|}
        \end{split}
        \end{equation*}
        \tiny
        \begin{itemize}
            \item $\mathcal{I}_{uv}:$ The set of items that users $u$ and $v$ rated in common
            \item $r_{ui}:$ The rating that user $u$ gave to item $i$
            \item $r_{vi}:$ The rating that user $v$ gave to item $i$
            \item $\bar{r_{u}}:$ The mean rating of user $u$

        \end{itemize}
        \column{0.5\textwidth}
        \centering
        \underline{\textbf{Item-based}}
        \begin{equation*}
            \small
            \begin{split}
            &PCC(i,j) = \frac{\sum_{u \in \mathcal{U}_{ij}}(r_{iu}-\bar{r_{i}})(r_{ju}-\bar{r_{j}})}
                             {\sqrt{\sum_{u \in \mathcal{U}_{ij}}(r_{iu}-\bar{r_{i}})^2}
                              \sqrt{\sum_{u \in \mathcal{U}_{ij}}(r_{ju}-\bar{r_{j}})^2}} \\\\
            &\bar{r_{i}} = \frac{\sum_{u \in \mathcal{U}_i}r_{iu}}
                                {\mathopen|\mathcal{U}_i\mathclose|}
        \end{split}
        \end{equation*}
        \tiny
        \begin{itemize}
            \item $\mathcal{U}_{ij}:$ The set of users that have both rated items $i$ and $j$
            \item $r_{iu}:$ The rating item $i$ received from user $u$
            \item $r_{ju}:$ The rating item $j$ received from user $u$
            \item $\bar{r_{i}}:$ The mean rating of item $i$

        \end{itemize}
    \end{columns}
\end{frame}
\begin{frame}
    \frametitle{Modified Pearson Correlation Coefficient 1}
    \vspace{-0.8cm}
    \begin{columns}
        \hspace{-7mm}
        \column{0.5\textwidth}
        \centering
        \underline{\textbf{User-based}}
        \begin{equation*}
            \footnotesize
            \begin{split}
    &MPCC1(u,v) = \frac{\sum_{i \in \mathcal{I}_{uv}}(r_{ui}-\tilde{r_{u}})(r_{vi}-\tilde{r_{v}})}
                       {\sqrt{\sum_{i \in \mathcal{I}_{uv}}(r_{ui}-\tilde{r_{u}})^2}
                        \sqrt{\sum_{i \in \mathcal{I}_{uv}}(r_{vi}-\tilde{r_{v}})^2}} \\\\
      &\tilde{r_{u}} = \frac{\sum_{i \in \mathcal{I}_{uv}}r_{ui}}
                          {\mathopen|\mathcal{I}_{uv}\mathclose|}
            \end{split}
        \end{equation*}
        \tiny
        \begin{itemize}
            \item $\mathcal{I}_{uv}:$ The set of items that users $u$ and $v$ rated in common
            \item $r_{ui}:$ The rating that user $u$ gave to item $i$
            \item $r_{vi}:$ The rating that user $v$ gave to item $i$
            \item $\tilde{r_{u}}:$ The mean rating of user $u$

        \end{itemize}
        \column{0.5\textwidth}
        \centering
        \underline{\textbf{Item-based}}
        \begin{equation*}
            \footnotesize
            \begin{split}
            &MPCC1(i,j) = \frac{\sum_{u \in \mathcal{U}_{ij}}(r_{iu}-\tilde{r_{i}})(r_{ju}-\tilde{r_{j}})}
                             {\sqrt{\sum_{u \in \mathcal{U}_{ij}}(r_{iu}-\tilde{r_{i}})^2}
                              \sqrt{\sum_{u \in \mathcal{U}_{ij}}(r_{ju}-\tilde{r_{j}})^2}} \\\\
            &\tilde{r_{i}} = \frac{\sum_{u \in \mathcal{U}_i}r_{iu}}
                                {\mathopen|\mathcal{U}_i\mathclose|}
        \end{split}
        \end{equation*}
        \tiny
        \begin{itemize}
            \item $\mathcal{U}_{ij}:$ The set of users that have both rated items $i$ and $j$
            \item $r_{iu}:$ The rating item $i$ received from user $u$
            \item $r_{ju}:$ The rating item $j$ received from user $u$
            \item $\tilde{r_{i}}:$ The mean rating of item $i$

        \end{itemize}
    \end{columns}
\end{frame}
\begin{frame}
    \frametitle{Modified Pearson Correlation Coefficient 2}
    \vspace{-0.8cm}
    \begin{columns}
        \hspace{-7mm}
        \column{0.5\textwidth}
        \centering
        \underline{\textbf{User-based}}
        \begin{equation*}
            \footnotesize
            \begin{split}
            &MPCC2(u,v) = \frac{\sum_{i \in \mathcal{I}_{uv}}(r_{ui}-\bar{r_{u}})(r_{vi}-\bar{r_{v}})}
                             {\sqrt{\sum_{i \in \mathcal{I}_{u}}(r_{ui}-\bar{r_{u}})^2}
                              \sqrt{\sum_{i \in \mathcal{I}_{v}}(r_{vi}-\bar{r_{v}})^2}} \\\\
            &\bar{r_{u}} = \frac{\sum_{i \in \mathcal{I}_u}r_{ui}}
                                {\mathopen|\mathcal{I}_u\mathclose|}
        \end{split}
        \end{equation*}
        \tiny
        \begin{itemize}
            \item $\mathcal{I}_{u}:$ The set of items that have been rated by user $u$
            \item $\mathcal{I}_{v}:$ The set of items that have been rated by user $v$
            \item $\mathcal{I}_{uv}:$ The set of items that users $u$ and $v$ rated in common
            \item $r_{ui}:$ The rating that user $u$ gave to item $i$
            \item $r_{vi}:$ The rating that user $v$ gave to item $i$
            \item $\bar{r_{u}}:$ The mean rating of user $u$
        \end{itemize}
        \column{0.5\textwidth}
        \centering
        \underline{\textbf{Item-based}}
        \begin{equation*}
            \footnotesize
            \begin{split}
            &MPCC2(i,j) = \frac{\sum_{u \in \mathcal{U}_{ij}}(r_{iu}-\bar{r_{i}})(r_{ju}-\bar{r_{j}})}
                             {\sqrt{\sum_{u \in \mathcal{U}_{i}}(r_{iu}-\bar{r_{i}})^2}
                              \sqrt{\sum_{u \in \mathcal{U}_{j}}(r_{ju}-\bar{r_{j}})^2}} \\\\
            &\bar{r_{i}} = \frac{\sum_{u \in \mathcal{U}_i}r_{iu}}
                                {\mathopen|\mathcal{U}_i\mathclose|}
        \end{split}
        \end{equation*}
        \tiny
        \begin{itemize}
            \item $\mathcal{U}_{i}:$ The set of users that have rated item $i$
            \item $\mathcal{U}_{j}:$ The set of users that have rated item $j$
            \item $\mathcal{U}_{ij}:$ The set of users that have both rated items $i$ and $j$
            \item $r_{iu}:$ The rating item $i$ received from user $u$
            \item $r_{ju}:$ The rating item $j$ received from user $u$
            \item $\bar{r_{i}}:$ The mean rating of item $i$
        \end{itemize}
    \end{columns}
\end{frame}
\begin{frame}
    \frametitle{Mean Squared Difference}
    \begin{columns}
        \column{0.5\textwidth}
        \centering
        \underline{\textbf{User-based}}
        \begin{equation*}
        MSD(u,v) = \frac{\mathopen|\mathcal{I}_{uv}\mathclose|}
                        {\sum_{i \in \mathcal{I}_{uv}}(r_{ui}-r_{vi})^2}
    \end{equation*}
        \tiny
        \begin{itemize}
            \item $\mathcal{I}_{uv}:$ The set of items that users $u$ and $v$ rated in common
            \item $r_{ui}:$ The rating that user $u$ gave to item $i$
            \item $r_{vi}:$ The rating that user $v$ gave to item $i$
        \end{itemize}
        \column{0.5\textwidth}
        \centering
        \underline{\textbf{Item-based}}
        \begin{equation*}
        MSD(i,j) = \frac{\mathopen|\mathcal{U}_{ij}\mathclose|}
                        {\sum_{u \in \mathcal{U}_{ij}}(r_{iu}-r_{ju})^2}
    \end{equation*}
        \tiny
        \begin{itemize}
            \item $\mathcal{U}_{ij}:$ The set of users that have both rated items $i$ and $j$
            \item $r_{iu}:$ The rating item $i$ received from user $u$
            \item $r_{ju}:$ The rating item $j$ received from user $u$
        \end{itemize}
    \end{columns}
\end{frame}
\begin{frame}
    \frametitle{Mean Absolute Difference}
    \begin{columns}
        \column{0.5\textwidth}
        \centering
        \underline{\textbf{User-based}}
        \begin{equation*}
        MAD(u,v) = \frac{\mathopen|\mathcal{I}_{uv}\mathclose|}
                        {\sum_{i \in \mathcal{I}_{uv}}\mathopen|r_{ui}-r_{vi}\mathclose|}
    \end{equation*}
        \tiny
        \begin{itemize}
            \item $\mathcal{I}_{uv}:$ The set of items that users $u$ and $v$ rated in common
            \item $r_{ui}:$ The rating that user $u$ gave to item $i$
            \item $r_{vi}:$ The rating that user $v$ gave to item $i$
        \end{itemize}
        \column{0.5\textwidth}
        \centering
        \underline{\textbf{Item-based}}
        \begin{equation*}
        MAD(i,j) = \frac{\mathopen|\mathcal{U}_{ij}\mathclose|}
                        {\sum_{u \in \mathcal{U}_{ij}}\mathopen|r_{iu}-r_{ju}\mathclose|}
    \end{equation*}
        \tiny
        \begin{itemize}
            \item $\mathcal{U}_{ij}:$ The set of users that have both rated items $i$ and $j$
            \item $r_{iu}:$ The rating item $i$ received from user $u$
            \item $r_{ju}:$ The rating item $j$ received from user $u$
        \end{itemize}
    \end{columns}
\end{frame}
\begin{frame}
    \frametitle{Jaccard Coefficient}
    \begin{columns}
        \column{0.5\textwidth}
        \centering
        \underline{\textbf{User-based}}
        \begin{equation*}
    J(u,v) = \frac{\mathopen|\mathcal{I}_{uv}\mathclose|}
                  {\mathopen|\mathcal{I}_{u}\mathclose| +
		   \mathopen|\mathcal{I}_{v}\mathclose| -
		   \mathopen|\mathcal{I}_{uv}\mathclose|}
       \end{equation*}
        \tiny
        \begin{itemize}
            \item $\mathcal{I}_{u}:$ The set of items that have been rated by user $u$
            \item $\mathcal{I}_{v}:$ The set of items that have been rated by user $v$
            \item $\mathcal{I}_{uv}:$ The set of items that users $u$ and $v$ rated in common
            \item $\mathopen|\mathcal{I}_{u}\mathclose|:$ The number of items in set $\mathcal{I}_{u}$
            \item $\mathopen|\mathcal{I}_{v}\mathclose|:$ The number of items in set $\mathcal{I}_{v}$
            \item $\mathopen|\mathcal{I}_{uv}\mathclose|:$ The number of items in set $\mathcal{I}_{uv}$
        \end{itemize}
        \column{0.5\textwidth}
        \centering
        \underline{\textbf{Item-based}}
        \begin{equation*}
    J(i,j) = \frac{\mathopen|\mathcal{U}_{ij}\mathclose|}
                  {\mathopen|\mathcal{U}_{i}\mathclose| +
		   \mathopen|\mathcal{U}_{j}\mathclose| -
		   \mathopen|\mathcal{U}_{ij}\mathclose|}
       \end{equation*}
        \tiny
        \begin{itemize}
            \item $\mathcal{U}_{i}:$ The set of users that have rated item $i$
            \item $\mathcal{U}_{j}:$ The set of users that have rated item $j$
            \item $\mathcal{U}_{ij}:$ The set of users that rated items $i$ and $j$ in common
            \item $\mathopen|\mathcal{U}_{i}\mathclose|:$ The number of users in set $\mathcal{U}_{i}$
            \item $\mathopen|\mathcal{U}_{j}\mathclose|:$ The number of users in set $\mathcal{U}_{j}$
            \item $\mathopen|\mathcal{U}_{ij}\mathclose|:$ The number of users in set $\mathcal{U}_{ij}$
        \end{itemize}
    \end{columns}
\end{frame}
\subsection{K-Nearest Neighbors Algorithm}
\begin{frame}
    \frametitle{K-Nearest Neighbors Algorithm}
    \only<1>{
        \vspace{2cm}
        \centering
        \textbf{The K-Nearest Neighbors algorithm}
    }
    \vspace{-1cm}
    \begin{itemize}
	\item[]<2-> \textbf{Step 1:} Select users that have rated $Item_B$.
	\item[]<3-> \textbf{Step 2:} Compute the similarities between $User_A$ and the users that have
	rated $Item_B$.
	\item[]<4-> \textbf{Step 3:}  Sort the similarities in descending order.
	\item[]<5-> \textbf{Step 4:}  Choose how many neighbors will contribute in the rating
	prediction by selecting the top $\mathcal{K}$ out of all the available
	neighbors($\mathcal{K}$ can be in range [1 - $\mathcal{N}$] where $\mathcal{N}$ is all
	the available neighbors).
	\item[]<6-> \textbf{Step 5:} Use an aggregation formula to calculate the rating prediction of
	$User_A$ to $Item_B$. In this case the weighted sum is used.
\begin{equation*}
	\hat{r}(User_A,Item_B) = \frac{\sum_{u \in \mathcal{K}}{similarity(User_A,User_u) * r(User_u,Item_B)}}
						    {\sum_{u \in \mathcal{K}}{\mathopen|similarity(User_A,User_u)\mathclose|}}
\end{equation*}
\end{itemize}
\end{frame}