\chapter{Theoretical Background}

This chapter explains the concepts needed to understand this thesis. The theory behind RL, ANN, DDPG and HER is explained.

\section{Reinforcement Learning}
%% In context of machine learning (supervised,unsupervised)
%% potential to be better than humans

RL is one of the main learning concepts of Machine Learning next to Supervised Learning and Unsupervised Learning \cite{machinelearning}.
\newline
In Supervised Learning some input data is given to the learning agent. The agent is expected to come up with some output which is then compared with the expected output. If the output given by the agent and the expected output matches, then the agent was correct. The agent is trained by calculating the error between expected output and actual output. An use case for Supervised Learning is sorting mails into regular mail and spam mail. The agent is given a mail and it should decide whether the mail is regular or spam based on the content. Supervised Learning is used for classification and regression problems \cite{machinelearning}. It is mainly useful when the expected output is already known, so the learning agent can learn to do recognize these.
\newline
In Unsupervised Learning there is no expected output. The learning agent is fed with data so it can figure out interesting features and similarities between different data. Unsupervised Learning is often used to cluster data, often pictures, based on similarities \cite{machinelearning}.
\newline
In RL the agent is learning through rewards that are given through interaction with an environment. The goal of a task is clear, but the path of actions to reach the goal is not trivial. RL is used to find the best action in each situation. It is often used for games because they are already set up to have a clear task and goal, but the optimal way to reach it is not clear. Games usually provide full information on the environment in contrast to the real world, therefore setting up the learning environment is simple. Also Supervised Learning is limited in performance because the agent can only learn to become as good the expected output that we set. So in games like chess, with Supervised Learning the agent can only become as good as the best players it learns from, but not better. It is limited by the skills of the best humans \cite{nolimit}. However, RL is not restricted by these limitations. The agent can improve on its own only by exploring his options. In games like Go and Chess, engines that use RL have already far surpassed the best human players \cite{alphazero}.

\vspace{0.5cm}

%%general idea
This section will explain the theory behind RL. The post by Lilian Weng is recommended as a resource on more precise explanations on RL \cite{rllilianweng}. RL is usually modeled as a Markov Decision Process. The Markov Decision Process for RL consists of following elements \cite{rlwiki}:

\begin{itemize}
	\item A set of states S
	\item A set of actions A
	\item The transition probability $P_a(s,s')$ from state s to s' under action a
	\item The immediate reward $R_a(s,s')$ of that transition
	\item rules that describe the agents observation
\end{itemize}

The agent and environment are in a state s. The agent chooses an action a from its set of possible actions A to interact with the environment. The environment reacts by transitioning to another state and returning a reward R and an observation to the agent. Depending on the model the transitions might be stochastic or deterministic. The aim of the agent is to earn the maximal total reward possible. In order to reach this aim, the agent interacts with the environment to gain knowledge about the environment through the gained rewards and observations. Through this process, the agent learns in which state which actions are better to gain more reward. This is illustrated by Figure \ref{rl_general}.

\begin{figure} [h]

	\centering
	\includegraphics[width=1\textwidth]{figures/rl_general.pdf}
	\caption{Reinforcement Learning. An agent chooses an action to interact with the environment and gets a reward and an observation of his new state back.
		\label{rl_general}
		\cite{rl_general.jpg}
	}
\end{figure}

\vspace{0.5cm}

%policy
In each state there is an action that the agent considers best due to its current knowledge about the expected rewards of each action. This set of actions is known as the policy
$\pi (s)$.
The goal of getting maximal reward can be interpreted as finding the best actions in each state that give the most reward, which is finding the optimal policy. The policy can also be either deterministic or stochastic, depending on the transition probability of the environment.

\vspace{0.5cm}

%value function
A value function is used to measure how good a state or action is. Two types of value functions are used for the states and the actions. the state value function is denoted as V(s). The value of a state is the expected reward when acting according to the policy
$\pi$ \cite{rlwiki}.
V(s) is defined as follows in equation \ref{eq:state-value-function}.

\begin{equation}
\label{eq:state-value-function}
V^\pi (s) = \mathbb{E} [R | s,\pi]
\end{equation}

The actions value function is denoted as Q(s,a) and is defined In equation \ref{eq:action-value-function}.

\begin{equation}
\label{eq:action-value-function}
Q^\pi (s,a) = \mathbb{E} [R | s,a,\pi]
\end{equation}

When determining the value of a state or action, a discount factor $\gamma$
is used to discount future rewards towards immediate rewards.
The idea is that a state s is not only as good as the reward you get when transitioning to that state. Future rewards from states that are reachable from state s should also be considered. The value of a state consists of the reward that you get by transitioning to that state and the potential rewards that can be gained by transitioning from that state.
Since future rewards are not as certain as immediate rewards, the discount factor is used. The farther a reward is in the future, the more it is discounted.
%bellman equations
The Bellman equations described in equation \ref{eq:V-bellman} and \ref{eq:Q-bellman} are a set of equations that convert the value functions into the immediate and future reward and can be used to update the value-functions \cite{rllilianweng}.

\begin{equation}
\label{eq:V-bellman}
V_\pi (s) = \sum_{a \in A}  \pi(a|s) (R(s,a) + \gamma \sum_{s' \in S} P_{ss'}^a V_\pi (s'))
\end{equation}

\begin{equation}
\label{eq:Q-bellman}
Q_\pi (s,a) = R(s,a) + \gamma \sum_{s' \in S} P_{ss'}^a \sum_{a \in A}  \pi(a'|s') Q_\pi(s',a')
\end{equation}

The goal is to find the actions that return the maximal reward. This is displayed by the Bellman optimality equations (\ref{eq:V-optbellman},\ref{eq:Q-optbellman}) \cite{rllilianweng}:

\begin{equation}
\label{eq:V-optbellman}
V_\ast (s) = \max_{a \in A} (R(s,a) + \gamma \sum_{s' \in S} P_{ss'}^a V_\ast (s'))
\end{equation}

\begin{equation}
\label{eq:Q-optbellman}
Q_\ast (s,a) = R(s,a) + \gamma \sum_{s' \in S} P_{ss'}^a \max_{a \in A} Q_\ast(s',a')
\end{equation}

To calculate the optimal values of each state and action, Dynamic Programming could be used if the entire model is known. But even knowing the entire model is not good enough as usually the main issue lies in the huge state and action space, which makes it impossible to use Dynamic Programming. For RL, artificial neural networks (ANNs) can be used to approximate the value functions \cite{neuralnetpath}.

When following the current policy, the agent will earn the maximal reward that it could earn with current knowledge, but never more than that. To learn, the agent has to deviate from the policy and explore different actions and states. The question is how much deviation is necessary, as the agent also needs to exploit most of the policy path it has learned until now because following most of the policy has a higher probability of earning a high reward. This is known in RL as the exploration vs. exploitation problem. Usually there is a variable $\epsilon$ that determines with which probability the agent will deviate from the policy. It is set rather low to let the agent exploit most of its policy. As an example, an approach is $\epsilon$-greedy \cite{egreedy}. With a high probability of $1-\epsilon$ the agent will choose the policy and with a probability of $\epsilon$ a random action is chosen. This ensures that the agent exploits most of the policy, but also explores more possible actions that might bring more reward.

%%neuronal networks
\section{Artificial Neural Networks}

ANNs are inspired by the human brain.
%\nnbio, blablabla
The ANN consists of layers of neurons. Each neuron is connected to the next layer of neurons. There is one input layer and one output layer at the beginning and end of the layer of neurons. The layers between the input and output layers are called hidden layer. The hidden layer can consist of only one or more layers. The idea is to train the ANN to take inputs and produce outputs. To approximate the value functions the input would be states and actions, the output should be the correct and optimal values of these states and actions. An example of an ANN is shown in Figure \ref{neuralnet}.

\begin{figure} [h]
	\centering
	\includegraphics[width=1\textwidth]{figures/neural_network.pdf}
	\caption{A neural network. \cite{neural_networkpng}}
	\label{neuralnet}
\end{figure}

\vspace{0.5cm}

The learning process of the neuronal network is as follows. Each neuron obtains inputs $x_i$ by the the outputs of the neurons in the layer before it. Each input value is weighted and then the sum of these weighted values is taken. A bias $b$ is also added to support the learning process. After using an activation function on the sum, the value is output to the next layer of neurons. The activation function is a simple function that either reduces the output of the neuron to 0 if the value is below a certain threshold, otherwise the value is output unfiltered.
The training process of the neuron can be seen in Figure \ref{neuron}.
%besser formulieren!

\begin{figure} [h]
	\centering
	\includegraphics[width=1\textwidth]{figures/neuron.pdf}
	\caption{A neuron and how its value is composed \cite{neuron.jpeg}}
	\label{neuron}
\end{figure}

\vspace{0.5cm}

When the ANN outputs a value, a process called Backpropagation is used to improve the ANN \cite{backprop}. When initializing the ANN, the actual right weights for the input values are unknown, so random values for the weights are used. Therefore, the output values will not be right. By using an error function, the amount of error can be computed. The amount of error can be used to figure out how much the weights have to be modified for the output to become right. Backpropagation is the process of going backwards through the ANN and improving the weights of the neurons that caused the output value to be wrong. By repeating the whole process, the weight of each neuron converges towards an optimal value.

\vspace{0.5cm}

\section{Deep Deterministic Policy Gradients}

The paper by Silver et al. is a recommended resource for more detailed explanations \cite{ddpg}. This chapter will give a short summary of DDPG.
The algorithm DDPG learns concurrently a Q-function (the action-value function) and a policy. \cite{ddpg}
Q*(s,a) is used to find the the optimal action in each state.
To compute the Q-values for a discrete action space, the Q-values of each action could be calculated and compared to find the biggest value. But in a continuous action space it is not possible to calculate the Q-value for each action. DDPG uses the fact that the action space is continuous and so Q*(s,a) is expected to be differentiable in respect to the action argument. \cite{ddpg}
This way it is possible to approximate the Q-values and policy.

\vspace{0.5cm}

To learn the Q-values, the Bellman equation for the action value is used. To approximate the Q-values, a mean squared Bellman error function is used.
The idea is that minimizing this function error is equal to approximating the current Q-values to the optimal Q-values.

\vspace{0.5cm}

For DDPG an experience replay buffer is also used. The replay buffer is a set of experiences. This can be used to replay old experiences. When only using new experiences, the ANN might be overfit to those experiences. Being overfit means that for some experiences the ANN will output very good results, but for most other experiences it will perform very bad. Experience replay is useful to prevent that. But a too large buffer can cause the learning process to slow down. The right balance has to be found.

DDPG also uses target networks. Equation \ref{eq:target} is called target \cite{ddpg}.

\begin{equation}
\label{eq:target}
r + \gamma (1-d) \max_{a'} Q_\phi(s',a')
\end{equation}

The target is what is desired for the Q-function to approximate to.
When minimizing the mean squared bellman error function, there is the problem that the target is also dependent on the parameters that are trained. When changing the parameters, the target would also change which is problematic. That is why the target network, a copy of the ANN is used. The update of the target network is delayed to avoid this conflict.
To find the optimal policy, simply gradient ascent can be used to find the maximal Q-values.

\section{Hindsight Experience Replay}

%subsection curriculum learning ?

%read paper, use it
Sparse rewards are a big issue in RL, especially in tasks for robotic arms often the rewards are sparse. Having sparse rewards means that most of the samples used for training will not successful and therefore will not bring any useful reward. For example, the task to move an object to a certain point would have a sparse reward for a robotic arm because very precise movements are needed which the robotic arm has to learn first.

\vspace{0.5cm}

Andrychowicz et al. have shown that HER can be used to deal with this issue for robotic arms \cite{herpaper}
HER can learn efficiently from sparse rewards and can also be combined with any off-policy RL algorithm.
This technique is inspired by the ability of humans to learn from failures as least as much as from successes.

\vspace{0.5cm}

HER works as follows. After an episode of gaining experiences, all transitions between the states in each training sample is stored in a replay buffer, but the goal that was not achieved is extended to a set with a goal that is reached. This can also be further extended to a set of more goals that can be achieved in the terminating state of the training sample. If the goal was to move an object to point x, but it was pushed to point y, the replay buffer would use the same transitions but change the the goal we wanted to achieve to y. So when replaying the same experience, the agent would be successful and earn an useful reward. This does not help the agent learn how to reach the goal it wanted to reach initially, but it learns how to reach other goals. Being able to reach those other already achieved goals might be beneficial in learning how to reach the goal it actually wanted to achieve. HER is mainly used for tasks with multiple goals, but it was shown that it also improves the training of tasks with only a single goal. \cite{herpaper}
Interestingly, Andrychowicz et al. have shown HER performs has problems when using shaped rewards. \cite{herpaper}
%add binary rewards at start of section ?. binary != shaped ?

 %add pseudocode ?


% ============================================================================

\subsection{Shared Bitmanip Extension Functionality}
\label{sec:scalar:bitmanip}

Many of the primitive operations used in symmetric key cryptography
and cryptographic hash functions are well supported by the
RISC-V Bitmanip \cite{riscv:bitmanip:repo} extension
\footnote{
At the time of writing, the Bitmanip extension is still undergoing
standardisation.
Please refer to the Bitmanip draft specification
\cite{riscv:bitmanip:draft}
directly for the
latest information, as it may be slightly ahead of what is described
here.
}.
We propose that the scalar cryptographic extension {\em reuse} a
subset of the instructions from the Bitmanip extension directly.
Specifically, this would mean that
a core implementing
{\em either}
the scalar cryptographic extensions,
{\em or}
the Bitmanip extension,
{\em or}
both,
would be required to implement these instructions.

%
% TODO: Venn diagram of proposed instructions.
%

The following subsections give the assembly syntax of instructions
proposed for inclusion in the scalar crypto extension, along with a
set of use-cases for common algorithms or primitive operations.
For information on the semantics of the instructions, we refer directly
to the Bitmanip draft specification.

\subsubsection{Rotations}
\label{sec:scalar:bitmanip:rotate}

\begin{cryptobitmanipisa}
RV32, RV64:                         RV64 only:
    ror    rd, rs1, rs2                 rorw   rd, rs1, rs2
    rol    rd, rs1, rs2                 rolw   rd, rs1, rs2
    rori   rd, rs1, imm                 roriw  rd, rs1, imm
\end{cryptobitmanipisa}

See \cite[Section 3.1.1]{riscv:bitmanip:draft} for details of
these instructions.
Standard bitwise rotation is a primitive operation in many block ciphers and
hash functions;
it features particularly in the ARX (Add, Rotate, Xor) class of
block ciphers
\footnote{\url{https://www.cosic.esat.kuleuven.be/ecrypt/courses/albena11/slides/nicky_mouha_arx-slides.pdf}}
and stream ciphers.

Algorithms making use of 32-bit rotations:
SHA256, AES (Shift Rows), ChaCha20, SM3.

Algorithms making use of 64-bit rotations:
SHA512, SHA3.

\subsubsection{Bit \& Byte Permutations}
\label{sec:scalar:bitmanip:grev}

\begin{cryptobitmanipisa}
RV32:
    rev.b   rd, rs1 // grevi rd, rs1,  7 - Reverse bits in bytes
    rev8    rd, rs1 // grevi rd, rs1, 24 - Reverse bytes in 32-bit word

RV64:
    rev.b   rd, rs1 // grevi rd, rs1,  7 - Reverse bits in bytes
    rev8    rd, rs1 // grevi rd, rs1, 56 - Reverse bytes in 64-bit word
    rev8.w  rd, rs1 // grevi rd, rs1, 24 - Reverse bytes in 32-bit words
\end{cryptobitmanipisa}

The scalar cryptography extension provides the following instructions for
manipulating the bit and byte endianness of data.
They are all parameterisations of the Generalised Reverse with Immediate
({\tt grevi}) instruction.
The scalar cryptography extension requires {\em only} the above instances
of {\tt grevi} be implemented, which can be invoked via their pseudo-ops.

Reversing bytes in words is very common in cryptography when setting a
standard endianness for input and output data.
Bit reversal within bytes is used for implementing the GHASH  component
of Galois/Counter Mode (GCM)~\cite{nist:gcm}.

Cores which also implement the Bit-manipulation extension {\em must}
implement the complete {\tt grevi} instruction, with all immediate values
supported.
The full specification of the {\tt grevi} instruction is available in
\cite[Section 2.2.2]{riscv:bitmanip:draft}.

\begin{cryptobitmanipisa}
RV32:
    zip     rd, rs1 // shfli   rd, rs1, 15 - Bit interleave
    unzip   rd, rs1 // unshfli rd, rs1, 15 - Bit de-interleave
\end{cryptobitmanipisa}

The {\tt zip} and {\tt unzip} pseudo-ops are specific instances of
the more general {\tt shfli} and {\tt unshfli} instructions.
The scalar cryptography extension requires {\em only} the above instances
of {\tt [un]shfli} be implemented, which can be invoked via their
pseudo-ops.
Only RV32 implementations require these instructions.
They perform a bit-interleave (or de-interleave) operation, and are
useful for implementing the 64-bit rotations in the
SHA3~\cite{nist:fips:202} algorithm on
a 32-bit architecture\footnote{
It is also useful for the ASCON cipher, which is a candidate in the
NIST Lightweight Cryptography competition.
}.
On RV64, the relevant operations in SHA3 can be done natively, so
{\tt zip} and {\tt unzip} are not required.

Cores which also implement the Bit-manipulation extension {\em must}
implement the complete {\tt [un]shfli} instruction, with all immediate values
supported.
The full specification of the {\tt shfli} instruction is available in
\cite[Section 2.2.3]{riscv:bitmanip:draft}.

\subsubsection{Carry-less Multiply}

\begin{cryptobitmanipisa}
RV32, RV64:
    clmul  rd, rs1, rs2
    clmulh rd, rs1, rs2
\end{cryptobitmanipisa}

See \cite[Section 2.6]{riscv:bitmanip:draft} for details of
this instruction.
As is mentioned there, obvious cryptographic use-cases for carry-less
multiply are for Galois Counter Mode (GCM) block cipher operations
\footnote{\url{https://en.wikipedia.org/wiki/Galois/Counter_Mode}}.
GCM is recommended by NIST as a block cipher mode of operation
\cite{nist:gcm}, and is the only {\em required} mode for the TLS 1.3
protocol.

See Section \ref{sec:scalar:timing} for additional implementation
requirements for this instruction, related to data independent
execution latency.

\subsubsection{Logic With Negate}

\begin{cryptobitmanipisa}
RV32, RV64:
    andn rd, rs1, rs2
     orn rd, rs1, rs2
    xnor rd, rs1, rs2
\end{cryptobitmanipisa}

See \cite[Section 2.1.3]{riscv:bitmanip:draft} for details of
these instructions.
These instructions are useful inside hash functions, block ciphers and
for implementing software based side-channel countermeasures like masking.
The {\tt andn} instruction is also useful for constant time word-select
in systems without the ternary Bitmanip {\tt cmov} instruction.

Useful for:
SHA3 Chi step,
bit-sliced function implementations
and
software based power/EM side-channel countermeasures based on masking.

\subsubsection{Packing}

\begin{cryptobitmanipisa}
RV32, RV64:                         RV64:
    pack   rd, rs1, rs2                 packw  rd, rs1, rs2
    packu  rd, rs1, rs2                 packuw rd, rs1, rs2
    packh  rd, rs1, rs2
\end{cryptobitmanipisa}

See \cite[Section 2.1.4]{riscv:bitmanip:draft} for details of
these instructions.
Some lightweight block ciphers
(e.g., SPARX \cite{DPUVGB:16})
use sub-word data types in their primitives.
The Bitmanip pack instructions are useful for performing rotations on
16-bit data elements.
They are also useful for re-arranging halfwords within words, and
generally getting data into the right shape prior to applying transforms.
This is particularly useful for cryptographic algorithms which pass inputs
around as byte strings, but can operate on words made out of those byte
strings.
This occurs for AES when loading blocks and keys (which may not be
word aligned) into registers to perform the round functions.

\subsubsection{Crossbar Permutation Instructions}
\label{sec:xperm}

\begin{cryptobitmanipisa}
RV32, RV64:
    xperm.n rd, rs1, rs2
    xperm.b rd, rs1, rs2
\end{cryptobitmanipisa}

See \cite[Section 2.2.4]{riscv:bitmanip:draft} for a complete
description of this instruction.

The {\tt xperm.n} instruction operates on nibbles.
The \rsone register contains a vector of $\XLEN/4$ $4$-bit elements.
The \rstwo register contains a vector of $\XLEN/4$ $4$-bit indexes.
The result is each element in \rstwo replaced by the indexed element
in \rsone, or zero if the index into \rstwo is out of bounds.

The {\tt xperm.b} instruction operates on bytes.
The \rsone register contains a vector of $\XLEN/8$ $8$-bit elements.
The \rstwo register contains a vector of $\XLEN/8$ $8$-bit indexes.
The result is each element in \rstwo replaced by the indexed element
in \rsone, or zero if the index into \rstwo is out of bounds.

The instruction can be used to implement arbitrary bit
permutations.
For cryptography, they can accelerate bit-sliced implementations,
permutation layers of block ciphers, masking based countermeasures
and SBox operations.

%Figure \ref{fig:example:xperm} shows example implementations of the
%$4$-bit PRINCE SBox using the instructions.
Lightweight block ciphers using $4$-bit SBoxes include
PRESENT\cite{block:present},
Rectangle\cite{block:rectangle},
GIFT\cite{block:gift},
Twine\cite{block:twine},
Skinny, MANTIS\cite{block:skinny},
Midori \cite{block:midori}.

National ciphers using $8$-bit SBoxes include
Camellia\cite{block:camellia} (Japan),
Aria\cite{block:aria} (Korea),
AES\cite{nist:fips:197} (USA, Belgium),
SM4\cite{block:sm4:1} (China)
and Kuznyechik (Russia).
All of these SBoxes can be implemented efficiently, in constant
time, using the {\tt xperm.b} instruction\footnote{
    \url{http://svn.clairexen.net/handicraft/2020/lut4perm/demo02.cc}
}.
Note that this technique is also suitable for masking based
side-channel countermeasures.

%\begin{figure}[h]
%\begin{lstlisting}[style=ASM]
%prince_sbox_rv64:
%    li  t0, 0x4D5E087619CA23FB  // Load the prince block cipher SBox
%    xperm.n a0, t0, a0          // a0.4[i] = t0.4[a0.4[i]]
%    ret
%
%prince_sbox_rv32:
%    li  t0, 0x4D5E0876  // Load last  8 elements of prince sbox
%    li  t1, 0x19CA23FB  // Load first 8 elements of prince sbox
%    li  t2, 0x88888888  // Bit mask for MS bits of index nibbles.
%    xperm.n a1, t1, a0  // a1.4[i] = t1.4[a0.4[i]] if a0.4[i] < 8 else 0
%    xor     a0, a0, t2  // Toggle MS bit of each nibble in input vector
%    xperm.n a0, t0, a0  // a0.4[i] = t1.4[a0.4[i]] if a0.4[i] < 8 else 0
%    or      a0, a0, a1  // Or results together.
%    ret
%\end{lstlisting}
%\caption{
%    Example implementations of the $4$-bit PRINCE\cite{block:prince}
%    block cipher SBox using the \mnemonic{xperm.n} instruction.
%}
%\label{fig:example:xperm}
%\end{figure}


% !TeX spellcheck = en_GB
\section{Surface, Thickness, Wind}
\label{sec:Geop}
%A good overview gives the sea level pressure, \SI{1000}-\SI{500}{\hPa} thickness map and winds at \SI{250}{\hPa}. \Cref{fig:GeopJet} shows, that it combines several important features of the vertical distribution within the atmosphere, for example.\\
%Black contour lines indicate sea level pressure in \SI{}{\hPa} and makes it possible to observe cyclones and anticyclones at the sea surface.
A complementary view of the three-dimensional structure of the atmosphere is also presented:  \SI{250}{\hPa} wind speed (colour shading, \SI{}{\mPs}), mean sea level pressure (black contours, \SI{}{\hPa}), \num{1000}--\SI{500}{\hPa} thickness (dashed contours) and the total precipitable water (black-white shading, \SI{}{\mm}). See \Cref{fig:GP24_pres} for an example.
% %%% Geopot Jet maps %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% \input{images_tex/WeatherSit_GeopotJet}
% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%\noindent
\\
The dashed, coloured contours in \Cref{fig:GP24_pres} show the vertical thickness between the \SI{1000}{\hPa} and \SI{500}{\hPa} surface, every \SI{6}{\deca\meter}. The thickness between two pressure levels can be interpreted via the hypsometric equation (\Cref{eq:hypsometric}), which equates the thickness to the mean temperature of the layer in question. In a relative sense, a larger thickness indicates a warmer air mass. In addition, strong horizontal gradients in the thickness field can be related to frontal boundaries. Specific to the discussion in this thesis, the thickness field also provides useful information regarding the form of precipitation (liquid, frozen).
%This is a relation of the mean temperature of the air between two pressure levels. Thus, high values of thickness mean relative warm, moist air (red, dashed). This can then be associated to rain or snow in mid-latitudes, depending on cold or warm air advection.
\\
%Gray shaded areas describe total precipitable water in the atmosphere in \SI{}{\mm}.
Analysis of the mean sea level pressure can be used to identify cyclones (L) and anticyclones (H) at the surface as well as provide supplementary information regarding frontal boundaries (\Cref{fig:GP24_pres}).
The total precipitable water is a measure of the column integrated moisture. It represents an instantaneous measure of moisture in time and space, which can be useful when assessing the amount of moisture that may fall as precipitation in future time steps.
The \SI{250}{\hPa} wind speeds are used to identify strong upper-level flow (i.e. the jet stream) and can be directly compared to the dynamic tropopause map (\Cref{fig:DT24_pres}).
% It is an indicator for the amount of moisture to supply rainfall, and will be used to identify where moisture was present.
% \\
% Colour shaded contours in \Cref{fig:GeopJet} indicate the mid-latitudal jet streaks at \SI{250}{\hPa}. Warmer colour is associated with higher wind speeds at this level.

% %% Geopot Jet maps %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% \input{images_tex/WeatherSit_GeopotJet}
% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


\begin{document}
\section{Difficulty from Replays}

We are able to extract data from replays, however, they are not linked to the beatmaps themselves. This means that it only has data of what keys and when did the player press it, there's no data on accuracy achieved.

To recap, we managed to decode the replay sent into the following format:

$$ action_{replay} := \lbrace(offset_1, action_1), (offset_2, action_2), ... , (offset_n, action_n)\rbrace $$

Whereby,
$$n \in \lbrace-9, -8, ... , -2, -1, 1, 2, ... , 8, 9\rbrace$$

$offset$ is when the $action$ happens. For $action$, $-n$ means the key \textbf{n} is released, $n$ means the key \textbf{n} is pressed. In this section, we will be discussing how we can make this a suitable output for the neural network to predict.

\subsection{Mapping a Replay to Beatmap}

In this, we match all similar actions in their respective columns.

There are a few things we need to take note of when matching:
\begin{enumerate}
	\item Not all $action_{beatmap}$ will have a matching $action_{replay}$
	\item We put the threshold of this matching as $100ms$, i.e. $action_{beatmap}$ that doesn't have any $action_{replay}$ within $100ms$ will be regarded as a miss.
	\item The nearest $action_{replay}$ will match the $action_{beatmap}$, not the earliest one.
	\item We will deviate on how osu! calculate accuracy due to the above pointers, this allows us to calculate on a more common basis.
\end{enumerate}

We will expect the output of:

$$ deviation := \lbrace(offset_1, deviation_1), (offset_2, deviation_2), ..., (offset_n, deviation_n)\rbrace $$

Where:
$$ n = length(action_{beatmap}) $$

And if there's no match, $deviation > 100$, this is to allow us to understand that it's a \textbf{miss} instead of a $100ms$ hit.

This doesn't proportionally represent accuracy (which will be easier to understand), as its lower value represents a better judgement, so we will adjust to the following:
$$ accuracy := \lbrace(offset_1, accuracy_1), (offset_2, accuracy_2), ..., (offset_n, accuracy_n)\rbrace $$

Where:
$$ accuracy_n = 100 - {deviation_n} $$

Therefore, a \textbf{miss} would simply just be $accuracy = 0$ instead.

So accuracy will only span:
$$ (Miss) 0 \leq accuracy_n \leq (Perfect) 1 $$
$$ where, deviation_n \in [0, 1, 2, ..., 99, 100] $$

\subsection{Replay Soloing}

As discussed in the preface, we need to resolve two issues. \textbf{Multiple Replays} and \textbf{Multiple Players}. We will expect an output similar to a replay, this is where the first major assumption kicks in.

\subsubsection{Assumption of the Top 50}
In this, we assume that if we took the \textbf{median} all top 50 replays, we will end up with a replay that is all-encompassing.

This leads some problems:

\paragraph{Top Player Bias} The neural network will perform worse on easier maps $ \approx 3.0 S.R. $ due to the median in easier maps being too consistently perfect, this leads to amplification of noise (in this case, chokes).
\paragraph{Population Interaction} This assumption will only hold if the beatmaps are old enough such that most of the general playerbase has played it, else it doesn't represent the population well enough due to low participation
\paragraph{Population Decay/Improvement} The beatmaps we check must be ranked within close proximity with each other, this is to avoid the Top 50 median from adjusting too much

\subsubsection{Assumption of the Player}
In this, instead of looking at it \textbf{per beatmap}, we will do it \textbf{per player}. This is much more consistent in data, however it's consistent \textbf{for that player only}.

In this method, we grab the all replays that \textbf{Player X} has played and compare them with each other. However this still leads to assumptions:

\begin{enumerate}
	\item The player played all the beatmaps at a similar time, or the player never improved
	\item The player is representative of the community (bias)
\end{enumerate}

The problem of the machine being \textbf{biased} to only the player can be fixed if we ran an averaging function on the output with hundreds of other players.

\textbf{However,} due to how osu! sends API scores, I couldn't get more than \textbf{50} scores \textbf{per player}. This means that there will be no data for these replays.

\subsection{Choosing an Assumption}

Despite this, we will work with \textbf{Assumption of the Top 50} as it's reasonable enough, and it's easier.

This means that the median of all replays will be saved in a different data file with $<beatmap\_id>.acrv$ extension. \textit{(v represents virtual)}

\subsection{Virtual Player/Replay}

The median of the top 50 creates a \textbf{virtual replay}, where we expect it to be a \textbf{good enough} representation of a \textbf{virtual player}. We can now pivot from this player as we calculate difficulty!

\subsection{Smoothing}

As expected, the median gathered will not be a smooth graph, we will smooth out with an \textbf{moving aggregation of its mean with a window of 30}. This means that it'll grab 30 data points, front and back, and calculate its median, creating a new series/vector.

As player's deviations usually will not be consistent throughout, this will help hammer down large errors, while also affecting its neighbouring data points. In turn, this aids the machine to learn that the error may not only be the issue of that one note, but instead a group of it.

\end{document}


\documentclass{documentation}

\title{Densor Package}

\author{Joshua Maglione}
\address{Universit\"at Bielefeld}
\email{jmaglione@math.uni-bielefeld.de}

\author{James B. Wilson}
\address{Colorado State University}
\email{james.wilson@colostate.edu}

\version{1.0}
\date{\today}
\copyrightyear{2016--2019}

\input{preamble.tex}

%-----------------------------------------------------------------------------
\begin{document}

\frontmatter

\dominitoc
\maketitle
\tableofcontents

\mainmatter

\chapter{Introduction}

\chapter{Subspaces as closures}~

The subspaces in this section are given by Galois connections in \cite{FMW:densors}*{Theorem~A}.
Currently, these are the only closures that are constructible in this package, and they are implemented for 3-tensors.

\index{DerivationClosure}
\begin{intrinsics}
DerivationClosure(T, Delta) : TenSpc, AlgMat -> TenSpc
DerivationClosure(T, Delta) : TenSpc, ModMatFld -> TenSpc
DerivationClosure(T, Delta) : TenSpc, AlgMatLie -> TenSpc
DerivationClosure(T, Delta) : TenSpc, [Mtrx] -> TenSpc
DerivationClosure(T, Delta) : TenSpc, [AlgMatLie] -> TenSpc
\end{intrinsics}

Returns the derivation closure of the given tensor space $T$, with frame $U_2\times U_1\rightarrowtail U_0$, with operators $\Delta\subseteq \text{End}(U_2)\times \text{End}(U_1)\times \text{End}(U_0)$.
If a nontrivial subset of coordinates are fused, then \texttt{DerivationClosure} accepts $\Delta$ input from the smaller ambient space.
For example, if coordinates $2$ and $1$ are fused, then the input for $\Delta$ can be in either $\text{End}(U_2)\times\text{End}(U_1)\times\text{End}(U_0)$ or $\text{End}(U_2)\times\text{End}(U_0)$.
Currently, this only works for tensor spaces of valence 3.
This is the subspace whose tensors' derivation algebra contains $\Delta$.

\index{DerivationClosure}
\begin{intrinsics}
DerivationClosure(T, t) : TenSpc, TenSpcElt -> TenSpc
\end{intrinsics}

Returns the derivation closure of the given tensor space $T$, with frame $U_2\times U_1\rightarrowtail U_0$, whose operators are the derivation algebra of $t$.
Currently, this only works for tensor spaces of valence 3.
This is the subspace whose tensors' derivation algebra contains the derivation algebra of $t$.

\index{UniversalDensorSubspace}
\begin{intrinsics}
UniversalDensorSubspace(t) : TenSpcElt -> TenSpc
\end{intrinsics}

Returns the universal densor subspace of the given tensor $t : U_2\times U_1 \rightarrowtail U_0$ as a subspace of the universal tensor space of $t$, whose operators are the derivation algebra of $t$.
Currently, this only works for tensor spaces of valence 3.
This is equivalent to \texttt{DerivationClosure(Parent(t), t)}.
This tensor subspace is stored with $t$ once computed.

\begin{example}[Rank1Densor]

We illustrate the fact that the tensor for (hyper-)matrix multiplication spans its densor, see \cite{FMW:densors}*{Theorem~G}.
We will construct the derivation closure of $\bra{t}:\mathbb{M}_{2\times 3}(\mathbb{F}_3)\times \mathbb{M}_{3\times 2}(\mathbb{F}_3)\rightarrowtail \mathbb{M}_{2}(\mathbb{F}_3)$, given by matrix multiplcation.
The derivation closure of $t$ in the universal tensor space $T$ is 1-dimensional.
\begin{code}
> Fr := [ KMatrixSpace(GF(3),2,3), KMatrixSpace(GF(3),3,2),
>     KMatrixSpace(GF(3),2,2) ];
> F := func< x | x[1]*x[2] >;
> t := Tensor(Fr, F);
> t;
Tensor of valence 3, U2 x U1 >-> U0
U2 : Full Vector space of degree 6 over GF(3)
U1 : Full Vector space of degree 6 over GF(3)
U0 : Full Vector space of degree 4 over GF(3)
\end{code}

The derivation algebra of $t$ is isomorphic to $(\mathfrak{gl}_2(\mathbb{F}_3)\oplus \mathfrak{gl}_3(\mathbb{F}_3)\oplus \mathfrak{gl}_2(\mathbb{F}_3))/\mathbb{F}_3$.
We do not show this, but we verify that the dimensions match.
\begin{code}
> D := DerivationAlgebra(t);
> Dimension(D) eq 4+9+4-1;
true
\end{code}

Now we verify that $t$ spans its own densor.
\begin{code}
> T := Parent(t);
> T;
Tensor space of dimension 144 over GF(3) with valence 3
U2 : Full Vector space of degree 6 over GF(3)
U1 : Full Vector space of degree 6 over GF(3)
U0 : Full Vector space of degree 4 over GF(3)
> densor := DerivationClosure(T, D);
> densor eq sub< T | t >;
true
\end{code}
\end{example}

\index{NucleusClosure}
\begin{intrinsics}
NucleusClosure(T, Delta, a, b) : TenSpc, AlgMat, RngIntElt, RngIntElt -> TenSpc
NucleusClosure(T, Delta, a, b) : TenSpc, ModMatFld, RngIntElt, RngIntElt -> TenSpc
NucleusClosure(T, Delta, a, b) : TenSpc, [Mtrx], RngIntElt, RngIntElt -> TenSpc
\end{intrinsics}

Returns the nucleus closure of the tensor space $T$, with frame $U_2\times U_1\rightarrowtail U_0$, with operators $\Delta\subseteq \text{End}(U_a)\times \text{End}(U_b)$.
Currently, this only works for tensor spaces of valence 3.
This returns the subspace whose tensors' $\{a,b\}$-nuclues contains $\Delta$.

\index{NucleusClosure}
\begin{intrinsics}
NucleusClosure(T, t, a, b) : TenSpc, TenSpcElt, RngIntElt, RngIntElt -> TenSpc
\end{intrinsics}

Returns the nucleus closure of the tensor space $T$, with frame $U_2\times U_1\rightarrowtail U_0$, whose operators are the $\{a,b\}$-nucleus of $t$.
Currently, this only works for tensor spaces of valence 3.
This returns the subspace whose tensors' $\{a,b\}$-nuclues contains the $\{a,b\}$-nucleus of $t$.

\begin{example}[NucClosure]

We illustrate that if $t$ is the commutator tensor from the Heisenberg group, then the densor of $t$ is $\text{Cen}(t)\cdot t$, 1-dimensional over the centroid.
First, we construct $t$ with frame $\mathbb{F}_5^6\times\mathbb{F}_5^6\rightarrowtail \mathbb{F}_5^3$, so that $t$ is $\mathbb{F}_5$-bilinear.
\begin{code}
> H := ClassicalSylow(GL(3,125), 5);
> t := pCentralTensor(H);
> t;
Tensor of valence 3, U2 x U1 >-> U0
U2 : Full Vector space of degree 6 over GF(5)
U1 : Full Vector space of degree 6 over GF(5)
U0 : Full Vector space of degree 3 over GF(5)
\end{code}

Because $t$ came from the Heisenberg group over $\mathbb{F}_{5^3}$, the centroid of $t$ will be 3-dimensional (over $\mathbb{F}_5$) and isomorphic to $\mathbb{F}_{5^3}$.
After computing the centroid of $t$, we will rewrite $t$ over the centroid $\text{Cen}(t)$, so that $t$ is $\mathbb{F}_{5^3}$-bilinear.
\begin{code}
> C := Centroid(t);
> C;
Matrix Algebra of degree 15 and dimension 3 with 3 generators over GF(5)
> s := TensorOverCentroid(t);
> s;
Tensor of valence 3, U2 x U1 >-> U0
U2 : Full Vector space of degree 2 over GF(5^3)
U1 : Full Vector space of degree 2 over GF(5^3)
U0 : Full Vector space of degree 1 over GF(5^3)
\end{code}

Now we will compute the nucleus closure of both $s$ and $t$ at $\{2,1\}$.
The dimension of the closure of $s$ will be 1-dimensional, while the dimension of the closure for $t$ will not.
\begin{code}
> NucleusClosure(Parent(s), s, 2, 1);
Tensor space of dimension 1 over GF(5^3) with valence 3
U2 : Full Vector space of degree 2 over GF(5^3)
U1 : Full Vector space of degree 2 over GF(5^3)
U0 : Full Vector space of degree 1 over GF(5^3)
>
> NucleusClosure(Parent(t), t, 2, 1);
Tensor space of dimension 9 over GF(5) with valence 3
U2 : Full Vector space of degree 6 over GF(5)
U1 : Full Vector space of degree 6 over GF(5)
U0 : Full Vector space of degree 3 over GF(5)
\end{code}
\end{example}

\backmatter

\begin{bibdiv}
\begin{biblist}

\bib{FMW:densors}{article}{
   author={First, Uriya},
   author={Maglione, Joshua},
   author={Wilson, James B.},
   title={Polynomial identity tensors and their invariants},
   note={in preparation},
}

\end{biblist}
\end{bibdiv}

\printindex

\end{document}


\section{Dynamic and Intrinsic Reaction Coordinates}\index{Coordinates!reaction}
\label{t_irc}\index{Molecular dynamics}\index{Time-dependent phenomena}
The Intrinsic Reaction Coordinate method pioneered and developed by Mark
Gordon, North Dakota State University,\index{gordon@{\bf Gordon, Mark}}   has
been incorporated into MOPAC in a modified form.  As this facility is quite
complicated all the keywords associated with  the IRC have been grouped
together in this section (these can be seen later on  in this section).
\index{DRC|(} \index{DRC!definition}

The Dynamic Reaction Coordinate is the path  followed  by  all  the atoms  in
a  system  assuming  conservation  of  energy;  i.e.,  as the potential energy
changes the kinetic energy of  the  system  changes  in exactly  the  opposite
way  so  that  the  total  energy  (kinetic plus potential) is a constant.  It
is equivalent to the molecular mechanics molecular dynamics calculation, except
that bond-breaking and bond-making are supported, as are all the electronic
phenomena of the semiempirical methods.

If started at a  ground  state  geometry,  no \index{Transition state!use with
DRC} significant  motion should be seen.  Similarly, starting at a transition
state geometry should not produce  any  motion---after  all  it  is  a
stationary point and during the lifetime of a calculation it is unlikely to
accumulate enough momentum to travel far from the starting position.

In order to calculate the DRC path from a transition state,  either an
initial  deflection  is  necessary  or some initial momentum must be supplied.

Because of the time-dependent nature of the DRC  the  time  elapsed since the
start of the reaction is meaningful, and is printed.

\subsection*{Description}
The course of a molecular vibration can be followed by  calculating the
potential  and  kinetic  energy  at  various  times.   Two  extreme conditions
can be identified:  (a) gas phase, in which the total  energy is a constant
through time, there being no damping of the kinetic energy allowed, and (b)
liquid phase, in which kinetic energy is always set  to zero, the motion of the
atoms being infinitely damped.\index{Liquids}

All possible degrees of damping  are  allowed.   In  addition,  the facility
exists  to  dump  energy into the system, appearing as kinetic energy.  As
kinetic energy is a function of velocity, a vector quantity, the  energy
appears  as  energy of motion in the direction in which the molecule would
naturally move.  If the system  is  a  transition  state, then  the  excess
kinetic  energy  is added after the intrinsic kinetic energy has built up to at
least 0.2 kcal/mol.\index{Kinetic energy!damping}

For ground-state systems, the excess energy sometimes  may  not  be added;  if
the  intrinsic kinetic energy never rises above 0.2 kcal/mol then the excess
energy will not be added.

\subsection*{Equations used}
Force acting on any atom:
$$ g(i) + g'(i)t + g''(i)t^2 = \frac{dE}{dx(i)} +
        \frac{d^2E}{dx(i)^2} + \frac{d^3E}{dx(i)^3} $$
Acceleration due to force acting on each atom:
$$ a(i) = \frac{1}{M(i)} (g(i) + g'(i)t + g''(i)t^2) $$
New velocity:
$$ V(o) + \frac{1}{M(i)} \left(\Delta t g(i) + (1/2) \Delta  t^2 g'(i) +
(1/3) \Delta t^3g''(i)\right) $$
or:
$$ V(i) = V(i) + V'(i)t + V''(i)t^2 + V'''(i)t^3 $$
That is, the change in velocity is equal to the integral  over  the
time interval of the acceleration.

New position of atoms:
$$ X(i) = X(o) + V(o)t + (1/2) V't^2 + (1/3) V''t^3 + (1/4) V'''t^4 $$
That is, the change in position is equal to the integral  over  the
time interval of the velocity.

The velocity vector is accurate to the extent that  it  takes  into account
the  previous velocity, the current acceleration, the predicted acceleration,
and the change in predicted  acceleration  over  the  time interval.    Very
little  error  is  introduced  due  to  higher  order contributions to the
velocity; those that do occur  are  absorbed  in  a re-normalization of the
magnitude of the velocity vector after each time interval.

The magnitude of $\Delta t$, the time interval, is determined mainly by the
factor   needed   to   re-normalize  the  velocity  vector.   If  it  is
significantly different from unity, $\Delta t$ will be reduced; if  it  is
very close to unity, $\Delta t$ will be increased.

Even with all this, errors creep in and a system,  started  at  the transition
state,  is  unlikely  to  return precisely to the transition state  unless  an
excess  kinetic  energy  is  supplied,  for   example 0.2 kcal/mol.

The calculation  is  carried  out  in  Cartesian  coordinates,  and converted
into   internal  coordinates  for  display.   All  Cartesian coordinates must
be allowed to vary, in order to  conserve  angular  and translational
momentum.\index{DRC!conservation of momentum}

\subsection*{IRC}
\index{IRC|(}
The Intrinsic Reaction Coordinate is the path followed by  all  the atoms  in
a  system  and assumes that  all kinetic energy is completely lost at every
point; i.e., as the potential energy changes  the  kinetic  energy generated
is  annihilated  so  that  the  total  energy  (kinetic  plus potential) is
always equal to the potential energy only.

The IRC is intended for use in calculations in which the starting geometry is
that of the transition state.  A   normal  coordinate  is  chosen,  usually
the  reaction coordinate, and the system  is  displaced  in  either  the
positive  or negative  direction  along  this  coordinate.   The  internal
modes are obtained by calculating the mass-weighted  Hessian  matrix  in  a
force calculation   and   translating  the  resulting  Cartesian  normal  mode
eigenvectors to conserve  momentum.   That  is,  the  initial  Cartesian
coordinates  are  displaced  by  a  small  amount  proportional  to  the
eigenvector coefficients plus a translational constant; the constant  is
required  to  ensure that the total translational momentum of the system is
conserved as zero.  At the present time there may be  small  residual
rotational  components  which  are not annihilated; these are considered
unimportant, and will not materially affect the calculation.

\subsection*{General description of the DRC and IRC}
As the IRC usually requires a normal coordinate, a  force  constant
calculation  normally  has to be done first.  If IRC is specified on its own, a
normal coordinate is not used and the IRC calculation is performed on the
supplied geometry.

A recommended sequence of operations to start an IRC calculation is
as follows:
\begin{enumerate}
\item Calculate the transition state geometry.  If  the  transition state  is
not first  optimized,  then  the  IRC  calculation  may  give  very misleading
results.  For example, if NH$_3$ inversion  is  defined as  the  planar
system  but  without the N--H bond length being optimized, the first normal
coordinate might be for N--H  stretch rather  than  inversion.   In  that case
the IRC will relax the geometry to the optimized planar structure.

\index{ISOTOPE!use with IRC}
\index{FORCE!use with IRC}
\item Do a normal FORCE calculation, specifying \comp{ISOTOPE} in  order  to
save  the  FORCE  matrices.   (Note: Do  not  attempt  to  run the IRC at this
point directly unless you have confidence that  the FORCE  calculation will
work as expected.  If the IRC calculation is run directly, specify
\comp{ISOTOPE} anyway:  that will save the FORCE matrix and if the
calculation  has  to  be  re-done  then  \comp{RESTART} will work correctly.)

\item Using \comp{IRC=$n$} and \comp{RESTART}, run the IRC calculation.   If
\comp{RESTART} is specified with \comp{IRC=$n$} then the restart is assumed to
be from the FORCE calculation.  If, in an \comp{IRC} calculation,
\comp{RESTART}  is specified, and \comp{IRC=$n$} is {\em not} present,  then
the restart is  assumed to be from an earlier IRC calculation that was  shut
down  before  going  to completion.
\end{enumerate}

A DRC calculation is simpler, in that a force calculation is  not  a
prerequisite;  however,  most  calculations of interest normally involve use of
an internal coordinate.  For this reason IRC=$n$  can  be  combined with  DRC
to  give  a  calculation in which the initial motion (0.3kcal worth of kinetic
energy) is supplied by  the  IRC,  and  all  subsequent motion  obeys
conservation of energy.  The DRC motion can be modified in three ways:
\begin{enumerate}
\item It is possible to calculate the reaction  path  followed  by  a system
in  which  the  generated  kinetic energy decays with a finite half-life.  This
can  be  defined  by  DRC=$n.nnn$,  where $n.nnn$  is  the  half-life in
femtoseconds.  If $n.nn$ is 0.0 this corresponds  to  infinite  damping
simulating  the   IRC.    A limitation  of  the  program is that time only has
meaning when DRC is specified without a half-life.

\item Excess kinetic energy can be added to the calculation by use of
KINETIC=$n.nn$.   After  the  kinetic  energy  has  built  up  to 0.2 kcal/mol
or if IRC=$n$ is used then $n.nn$ kcal/mol of kinetic energy  is  added  to
the  system.   The excess kinetic energy appears as a velocity vector  in  the
same  direction  as  the initial motion.

\index{RESTART!use with IRC/DRC}
\item The RESTART file \verb/<filename>.res/ can be edited to allow the user
to  modify the velocity vector or starting geometry.  This file
is formatted.
\end{enumerate}

Frequently, the  DRC leads to a periodic, repeating orbit.   One  special
type---the  orbit in which the direction of motion is reversed so that the
system retraces its own path---is sensed for  and  if  detected  the
calculation  is  stopped after exactly one cycle.  If the calculation is to be
continued,  the  keyword  \comp{GEO-OK}  will  allow  this  check  to  be
by-passed.

\index{GNORM!use with IRC/DRC}  Sometimes the system will enter a stable state
in which the geometry is always changing, but nothing new is occurring.  One
example would be a system which decomposed into fragments, and the fragments
were moving apart. If all forces acting on the atoms become small, then the
calculation will be stopped.  If the calculation should be continued, then
specify \comp{GNORM=0 LET}.

Due to the potentially very large output files  that  the  DRC  can generate,
extra  keywords  are  provided  to allow selected points to be printed.  Two
types of control are provided:  one controls which points to  print, the other
controls what is printed.

By default, every point calculated is printed.  Often, this is not desirable,
and three keywords are provided to allow printing to be done whenever the
system changes by a preset amount.  These keywords are:

\begin{center}
\begin{tabular}{cll}\hline
          KeyWord &       Default         &   User Specification  \\ \hline
\comp{X-PRIO}  &   0.05 \AA ngstroms   &         \comp{X-PRIORITY=$n.nn$}  \\
\comp{T-PRIO}  &   0.10 Femtoseconds   &      \comp{T-PRIORITY=$n.nn$}  \\
\comp{H-PRIO}  &   0.10 kcal/mol      &      \comp{H-PRIORITY=$n.nn$}\\ \hline
\end{tabular}
\end{center}

By default, only the energies involved are printed (one line per point).  To
allow the geometry to be printed, \hyperref[pageref]{\comp{LARGE} is provided}{, see
p.~}{ for more detail}{large}.  Using \comp{LARGE} a wide range of control
is provided over what is printed.

\subsection*{Option to allow only extrema to be output}
In the geometry specification, if an internal coordinate is  marked for
optimization  then  when that internal coordinate passes through an extremum a
message will be printed and the geometry output.

Difficulties can  arise  from  the  way  internal  coordinates  are
processed.   The  internal  coordinates are generated from the Cartesian
coordinates, so an internal coordinate supplied  may  have  an  entirely
different  meaning  on  output.  In particular the connectivity may have
changed.  For obvious reasons dummy atoms should  not  be  used  in  the
supplied  geometry  specification.   If  there  is  any  doubt about the
internal coordinates or if the starting geometry  contains  dummy  atoms then
run  a  \comp{1SCF} calculation specifying \comp{INT}.  This  will produce an
ARC file with the ``ideal'' numbering---the internal numbering system used  by
MOPAC. \ Use this ARC file to construct a data file suitable for the DRC or
IRC.\index{DRC!dummy atoms in}\index{Dummy atoms!in DRC}

Notes:
\begin{enumerate}
\item Any coordinates marked for optimization  will  result  in  only extrema
being printed.
\item If extrema are being printed then kinetic energy  extrema  will also be
printed.
\end{enumerate}

\subsection*{Keywords for use with the IRC and DRC}
\index{IRC!keywords for}
\label{drckeys}
\begin{enumerate}
\item Setting up the transition state:  \comp{NLLSQ}, \comp{SIGMA}, or \comp{TS}.
\item Constructing the FORCE matrix:  \comp{FORCE} or \comp{IRC=$n$},
\comp{ISOTOPE}, \comp{LET}.
\item Starting an IRC:  \comp{RESTART} and \comp{IRC=$n$},  \comp{X-PRIO}, \comp{H-PRIO}.
\item Starting a DRC:  \comp{DRC} or \comp{DRC=$n.nn$}, \comp{KINETIC=$n.nn$},
\comp{T-PRIO}, etc..
\item Starting a DRC from a transition state:   (\comp{DRC}  or  \comp{DRC=$n$})  and
            \comp{IRC=$n$}, \comp{KINETIC=$n$}.
\item Restarting an IRC:  \comp{RESTART} and \comp{IRC}.
\item Restarting a DRC:  \comp{RESTART} and (\comp{DRC} or \comp{DRC=$n.nn$}).
\item Restarting a DRC starting from a transition state:  \comp{RESTART} and
            (\comp{DRC} or \comp{DRC=$n.nn$}).
\end{enumerate}
Other keywords, such as \comp{T=$nnn$} or \comp{GEO-OK} can be used any time.

\subsection*{Examples of DRC/IRC data}
Use of the IRC/DRC facility is quite complicated.  In the following examples
various `reasonable' options are illustrated for a calculation on water. It is
assumed  that  an  optimized  transition-state  geometry  is available.

Example  1:   Figure~\ref{h2odrc} illustrates a  Dynamic  Reaction   Coordinate
calculation,  starting  at   the transition  state  for  water  inverting, the
initial motion being opposite to the transition normal mode, with 6kcal of
excess kinetic  energy  added  in. Every point calculated is to be printed
(Note all coordinates are marked with a zero, and T-PRIO, H-PRIO and X-PRIO are
all absent).  The results of  an  earlier calculation using the same keywords
is assumed to exist. The earlier calculation would have constructed the force
matrix.   While the  total  cpu  time  is specified, it is in fact redundant in
that the calculation will run to completion in less than 600 seconds.

\index{IRC!example of}
\begin{figure}
\begin{makeimage}
\end{makeimage}
\begin{verbatim}
 KINETIC=6 RESTART  IRC=-1 DRC T=600
 WATER

      H   0.000000 0   0.000000 0   0.000000 0  0 0 0
      O   0.911574 0   0.000000 0   0.000000 0  1 0 0
      H   0.911574 0 180.000000 0   0.000000 0  2 1 0
      0   0.000000 0   0.000000 0   0.000000 0  0 0 0
\end{verbatim}
\caption{\label{h2odrc} Example of DRC calculation}
\end{figure}

Example 2:  Figure~\ref{h2oirc} shows an Intrinsic Reaction Coordinate
calculation.  Here the restart  is from a previous IRC calculation which was
stopped before the minimum was reached.  Recall that RESTART with IRC=$n$
implies  a  restart from  the FORCE calculation.  Since this is a restart from
within an IRC calculation the keyword IRC=$n$ has been replaced by IRC. \  IRC
on its  own (without the ``=$n$'') implies an IRC calculation from the starting
position---here the RESTART position---without initial
displacement.\index{IRC!example of restart}

\begin{figure}
\begin{makeimage}
\end{makeimage}
\begin{verbatim}
 RESTART  IRC  T=600
 WATER

      H   0.000000 0   0.000000 0   0.000000 0  0 0 0
      O   0.911574 0   0.000000 0   0.000000 0  1 0 0
      H   0.911574 0 180.000000 0   0.000000 0  2 1 0
      0   0.000000 0   0.000000 0   0.000000 0  0 0 0
\end{verbatim}
\caption{\label{h2oirc} Example of IRC calculation}
\end{figure}

\subsection*{Output format for IRC and DRC}
The IRC and DRC can produce  several  different  forms  of  output. Because of
the large size of these outputs, users are recommended to use search functions
to extract information.  To facilitate  this,  specific lines  have specific
characters.  Thus, a search for the ``\%'' symbol will summarize the energy
profile while a search  for ``AA'' will  yield  the coordinates of atom 1,
whenever it is printed.  The main flags to use in searches are:

\begin{description}
\item[\comp{\%}] Energies for all points calculated,    excluding extrema
\item[\comp{\%M}] Energies for all turning points
\item[\comp{\%MAX}] Energies for all maxima
\item[\comp{\%MIN}] Energies for all minima
\item[\comp{\%}] Energies for all points calculated
\item[\comp{AA*}] Internal coordinates for atom 1 for every point
\item[\comp{AE*}] Internal coordinates for atom 5 for every point
\item[\comp{123AB*}] Internal coordinates for atom 2 for point 123
\end{description}

As the keywords for the IRC/DRC are interdependent,  the  following list of
keywords illustrates various options.\index{DRC!keyword options}
\index{KINETIC}

\begin{description}
\item[\comp{DRC}] The Dynamic Reaction Coordinate is calculated.
Energy is conserved, and no initial impetus.
\item[\comp{DRC=0.5}] In the DRC kinetic energy is lost with a half-life of
0.5 femtoseconds.
\item[\comp{DRC=1.0}] Energy is put into a DRC with an half-life of
-1.0 femtoseconds, i.e., the system gains   energy.
\item[\comp{IRC}] The Intrinsic Reaction Coordinate is
calculated.  No initial impetus is given.
Energy not conserved.
\item[\comp{IRC=4}] The IRC is run starting with an impetus in the
negative of the 4th normal mode direction. The
impetus is one quantum of vibrational energy.
\item[\comp{IRC1 KINETIC=1}] The first normal mode is used in an IRC, with
the initial impetus being 1.0 kcal/mol.
\item[\comp{DRC KINETIC=5}] In a DRC, after the velocity is defined, 5 kcal
of kinetic energy is added in the direction of
the initial velocity.
\item[\comp{IRC=1 DRC KINETIC=4}] After starting with a 4 kcal impetus in the
direction of the first normal mode, energy is
conserved.
\item[\comp{DRC VELOCITY KINETIC=10}] Follow a DRC trajectory which starts with an
initial velocity read in, normalized to a
kinetic energy of 10 kcal/mol.
\end{description}

Instead of every point being printed, the option  exists  to  print specific
points  determined  by the keywords \comp{T-PRIORITY}, \comp{X-PRIORITY} and
\comp{H-PRIORITY}.  If any one of these words is specified, then the calculated
points  are used to define quadratics in time for all variables normally
printed.  In addition, if the flag for the first atom is set to  ``T''  then
all  kinetic  energy  turning  points  are printed.  If the flag for any other
internal coordinate is set to ``T'' then, when that coordinate  passes through
an extremum, that point will be printed.  As with the PRIORITY's, the point
will be calculated via  a  quadratic  to  minimize  non-linear errors.

N.B.:  Quadratics are unstable in the regions of inflection points; in  these
circumstances linear interpolation will be used.  A result of this is that
points printed in the  region  of  an  inflection  may  not correspond  exactly
to those requested.  This is not an error and should not affect the quality of
the results.

\subsection*{Test of DRC---verification of trajectory path}
Introduction:  Unlike  a  single-geometry  calculation  or  even  a geometry
optimization, verification of a DRC trajectory is not a simple task.  In this
section  a  rigorous  proof  of  the  DRC  trajectory  is presented;  it  can
be used both as a test of the DRC algorithm and as a teaching exercise.  Users
of the DRC are asked to  follow  through  this proof in order to convince
themselves that the DRC works as it should.

\subsection*{The nitrogen molecule}
For the nitrogen molecule (using MNDO) the equilibrium  distance is  $1.103816$
\AA, the heat of formation is 8.25741 kcal/mol and the vibrational frequency is
$2738.8$ cm$^{-1}$.   For  small  displacements, the  energy curve versus
distance is parabolic and the gradient curve is approximately linear, as is
shown in Table~\ref{n2}.         A  nitrogen molecule is thus a good
approximation to a harmonic oscillator.

% 40 lines, including this line
\begin{table}
\caption{\label{n2}Stretching Curve for Nitrogen Molecule}
\begin{center}
\begin{tabular}{rrr}
\multicolumn{1}{c}{N--N DIST} & \multicolumn{1}{c}{$\Delta H_f$}  & \multicolumn{1}{c}{GRADIENT}\\
\multicolumn{1}{c}{(\AA ngstroms)} &\multicolumn{1}{c}{(kcal/mol)} & \multicolumn{1}{c}{(kcal/mol/\AA ngstrom)}\\
\hline
1.11800   &  8.69441  &   60.84599 \\
1.11700   &  8.63563  &   56.70706 \\
1.11600   &  8.58100  &   52.54555 \\
1.11500   &  8.53054  &   48.36138 \\
1.11400   &  8.48428  &   44.15447 \\
1.11300   &  8.44224  &   39.92475 \\
1.11200   &  8.40444  &   35.67214 \\
1.11100   &  8.37091  &   31.39656 \\
1.11000   &  8.34166  &   27.09794 \\
1.10900   &  8.31672  &   22.77620 \\
1.10800   &  8.29611  &   18.43125 \\
1.10700   &  8.27986  &   14.06303 \\
1.10600   &  8.26799  &    9.67146 \\
1.10500   &  8.26053  &    5.25645 \\
1.10400   &  8.25749  &    0.81794 \\
1.10300   &  8.25890  &   -3.64427 \\
1.10200   &  8.26479  &   -8.12993 \\
1.10100   &  8.27517  &  -12.63945 \\
1.10000   &  8.29007  &  -17.17278 \\
1.09900   &  8.30952  &  -21.73002 \\
1.09800   &  8.33354  &  -26.31123 \\
1.09700   &  8.36215  &  -30.91650 \\
1.09600   &  8.39538  &  -35.54591 \\
1.09500   &  8.43325  &  -40.19953 \\
1.09400   &  8.47579  &  -44.87745 \\
1.09300   &  8.52301  &  -49.57974 \\
1.09200   &  8.57496  &  -54.30648 \\
1.09100   &  8.63164  &  -59.05775 \\
1.09000   &  8.69308  &  -63.83363 \\
\end{tabular}
\end{center}
\end{table}

\subsubsection{Period of vibration}
The period of vibration (time taken for the oscillator to undertake one
complete vibration, returning to its original position and velocity) can be
calculated in three ways.  Most direct is  the  calculation  from the  energy
curve; using the gradient constitutes a faster, albeit less direct, method,
while calculating it from the vibrational  frequency  is very  fast  but
assumes  that the vibrational spectrum has already been calculated.

\begin{enumerate}

\item From the energy curve. For a simple harmonic oscillator the period $r$ is
given by: $$ r = 2 \pi \sqrt{\frac{\mu}{k}}  $$ where $k$ is the
force constant.  \index{Reduced mass}\index{Force constant} The
reduced  mass, $\mu$,  (in amu)   of   a   nitrogen  molecule  is
$14.0067/2  =  7.00335$, and  the force-constant, $k$, can be
calculated from: $$E-c = (1/2) k(R-R_o)^2. $$ Given $R_o =
1.1038$, $R = 1.092$, $c = 8.25741$ and $E = 8.57496$~kcal/mol
then:
 \begin{eqnarray*}
k &=& 2*0.31755/(0.0118)^2 \; \mbox{(per mole)}\\
k &=& 4561.2 \mbox{ kcal/mol/A$^2$  (per mole)}\\
k &=& 1.9084*10^{30} \; \mbox{ ergs/cm$^2$ (per mole)}\\
k &=& 31.69*10^5  \; \mbox{ dynes/cm (per molecule)}\\
\end{eqnarray*}

(Experimentally, for N$_2$, k = $23*10^5$ dynes/cm )

Therefore:
$$ r = 2 \times 3.14159 \times \sqrt{\frac{7.0035}{1.9084\times 10^{30}}}
\;{\rm seconds} = 12.037 \times 10^{-15}\;{\rm s} = 12.037\;{\rm fs}. $$
If the frequency is calculated using the other half of the curve ($R=1.118,
E=8.69441$), then $k=12.333$ fs, or $k$, average, = 12.185 fs.

\item From the gradient curve. The force  constant  is  the  derivative  of
the  gradient  wrt distance:
$$ k = \frac{dG}{dx}. $$
Since we are using discrete points,  the  force  constant  is  best
obtained from finite differences:
$$ k = \frac{(G_2-G_1)}{(x_2-x_1)}. $$
For $x_2 = 1.1100$, $G_2 = 27.098$ and for $x_1 =  1.0980$,
$G_1  =  -26.311$,
giving rise to $k = 4450.75$ kcal/mol/\AA$^2$ and a period of $12.185$~fs.

\item From the vibrational frequency. Given a ``frequency'' (wavenumber) of
vibration of N$_2$ of $\bar{\nu}=2738.8$   cm$^{-1}$,  the period of
oscillation, in seconds, is given directly by:
$$ r = \frac{1}{c\bar{\nu}} = \frac{1}{2738.8 \times 2.998 \times 10^{10}} ,$$
or as $12.179$ fs.
\end{enumerate}

Summarizing, by three different methods the period  of  oscillation of N$_2$
is calculated to be $12.1851$, $12.185$ and $12.179$~fs, average $12.183$~fs.

\subsubsection{Initial dynamics of \mbox{N$_{2}$} with N--N distance = 1.094 \AA}
A useful check on the dynamics of N$_2$ is to  calculate  the  initial
acceleration  of  the  two  nitrogen  atoms  after releasing them from a
starting interatomic separation of 1.094 \AA.

At R(N-N) = 1.094 \AA, $G = -44.877$~kcal/mol/\AA\ or $-18.777 \times
10^{19}$~erg/cm. Therefore acceleration, $f = -18.777 \times 10^{19}
/14.0067$~cm/sec/sec, or $-13.405 \times 10^{18}$~cm/s$^2$, which is $ -13.405
\times 10^{15} \times$ Earth surface gravity.

Distance from equilibrium  $= 0.00980$ \AA. After $0.1$ fs, velocity is
$0.1\times  10^{-15} (-13.405 \times  10^{18})$ cm/sec or $1340.5$ cm/s.

In the  DRC  the  time-interval  between  points  calculated  is  a complicated
function of the curvature of the local surface.  By default, the first
time-interval is 0.105fs, so the calculated velocity  at  this time should be
$0.105/0.100 \times 1340.5 = 1407.6$ cm/s, in the DRC calculation the predicted
velocity is $1407.6$ cm/s.

The option is provided to allow sampling of the system at  constant
time-intervals,  the  default being $0.1$~fs.  For the first few points the
calculated velocities are given in Table~\ref{tdrc}.

\begin{table}
\caption{\label{tdrc} Velocities in DRC for N$_2$ Molecule}
\begin{center}
\begin{tabular}{rrrr}\\ \hline
      Time  & Calculated &  Linear &    Diff. in \\
            & Velocity  & Velocity &  Velocity \\ \hline
      0.000 &      0.0  &    0.0   &    0.0  \\
      0.100 &   1340.6  & 1340.5   &   -0.1  \\
      0.200 &   2678.0  & 2681.0   &   -3.0  \\
      0.300 &   4007.0  & 4021.5   &  -14.5  \\
      0.400 &   5325.3  & 5362.0   &  -36.7  \\
      0.500 &   6628.4  & 6702.5   &  -74.1  \\
      0.600 &   7912.7  & 8043.0   & -130.3  \\ \hline
\end{tabular}
\end{center}
\end{table}

As the calculated velocity is  a  fourth-order  polynomial  of  the
acceleration,   and  the  acceleration,  its  first,  second  and  third
derivatives, are all changing, the predicted velocity rapidly becomes  a poor
guide to future velocities.

For simple harmonic motion the velocity at any time is given by:
$$ v = v_0 \sin(2\pi t/r). $$
By fitting the computed velocities to simple harmonic motion, a much better fit
is obtained (Table~\ref{tdrc2}).

\begin{table}
\caption{\label{tdrc2} Modified  Velocities in DRC for N$_2$ Molecule}
\begin{center}
\begin{tabular}{rrrr} \hline
 & Calculated & Simple Harmonic  &    Diff. \\
Time  & Velocity  & 25325.Sin(0.5296t) &\\ \hline
            &           &                   &    \\
     0.000  &     0.0   &       0.0         &    0.0  \\
     0.100  &  1340.6   &    1340.6         &    0.0  \\
     0.200  &  2678.0   &    2677.4         &   +0.6  \\
     0.300  &  4007.0   &    4006.7         &   +0.3  \\
     0.400  &  5325.3   &    5324.8         &   +0.5  \\
     0.500  &  6628.4   &    6628.0         &   +0.4  \\
     0.600  &  7912.7   &    7912.5         &    0.0  \\ \hline
\end{tabular}
\end{center}
\end{table}

The repeat-time required for this  motion  is  $11.86$~fs,  in  good agreement
with  the  three  values calculated using static models.  The repeat time
should not be calculated from the time required to go from a minimum  to  a
maximum and then back to a minimum---only half a cycle. For all real systems
the potential energy is a skewed parabola, so  that the  potential energy
slopes are different for both sides; a compression (as in this case) normally
leads to a higher force-constant, and shorter apparent  repeat  time  (as in
this case).  Only the addition of the two half-cycles is meaningful.

\subsubsection{Conservation of normal coordinate}
So far this analysis has only considered a homonuclear diatomic.  A detailed
analysis  of  a  large  polyatomic  is  impractical,  and  for simplicity a
molecule of formaldehyde will be studied.

In polyatomics, energy can  transfer  between  modes.   This  is  a result  of
the non-parabolic nature of the potential surface.  For small displacements the
surface can be considered as  parabolic.   This  means that  for small
displacements interconversion between modes should occur only very slowly.  Of
the six normal modes, mode 1, at 1209.5~cm$^{-1}$, the in-plane C--H asymmetric
bend, is the most unsymmetric vibration, and is chosen to demonstrate
conservation of vibrational purity.

Mode 1 has a  frequency  corresponding  to  3.46  kcal/mol  and  a predicted
vibrational time of $27.58$~fs.  By direct calculation, using the DRC, the
cycle time is $27.59$~fs.  The rate of decay of this mode  has  an estimated
half-life of a few thousands femtoseconds.

\subsubsection{Rate of decay of starting mode}
For trajectories initiated by an IRC=$n$  calculation,  whenever  the
potential  energy is a minimum the current velocity is compared with the
supplied velocity.  The square of the cosine of the  angle  between  the two
velocity vectors is a measure of the intensity of the original mode in the
current vibration.

\subsubsection{Half-Life for decay of initial mode}
Vibrational purity is assumed to decay according to  zero'th  order kinetics.
The  half-life is thus $-0.6931472t/\log(<\!\psi^2\!>^2)$~fs, where
$<\!\psi^2\!>^2$ is the square of  the overlap integral of the wavefunction for
the  original vibration with that of the current  vibration.   Due to the  very
slow rate of decay of the starting mode, several half-life calculations
should  be  examined.   Only  when successive  half-lives  are  similar  should
any confidence be placed in their value.

\subsubsection{DRC print options}
The amount of output in the DRC is  controlled  by  three  sets  of
options.  These sets are:\index{H--PRIORITY}
\begin{itemize}
\item Equivalent Keywords \comp{H-PRIORITY}, \comp{T-PRIORITY}, and
\comp{X-PRIORITY}.
\item Potential Energy Turning Point option.
\item Geometry Maxima Turning Point options.
\end{itemize}
If \comp{T-PRIORITY} is used then  turning  points  cannot  be  monitored.
%Currently  \comp{H-PRIORITY} and \comp{X-PRIORITY} are not implemented,
%but will be as soon as practical.

\index{``T'' - Optimization flag} To monitor geometry turning points, put  a
``T'' in  place  of  the geometry optimization flag for the relevant geometric
variable. In the example shown in Figure~\ref{t}, the geometry of formaldehyde
would first be optimized, then a \comp{FORCE} calculation run, then a DRC
calculation started, using the first normal mode for the starting velocity.
Whenever the C=O bond length becomes a maximum or a minimum, a message is
printed.

\begin{figure}
\begin{makeimage}
\end{makeimage}
\begin{verbatim}
 IRC=1 DRC T=20
 Formaldehyde
 Monitoring the C=O Bond-length turning points
  O    0.0 0    0.0 0    0.000000 0   0 0 0
  C    1.2 T    0.0 0    0.000000 0   1 0 0
  H    1.0 1  120.0 1    0.000000 0   2 1 0
  H    1.0 1  120.0 1  180.000000 0   2 1 3
  0    0.0 0    0.0 0    0.000000 0   0 0 0
\end{verbatim}
\caption{\label{t} Example of DRC calculation, monitoring a geometric variable}
\end{figure}

To monitor the potential energy turning points, put a ``T'' for  the flag for
atom 1 bond length (Do not forget to put in a bond-length (zero will do)!).

To monitor the geometry, use \comp{LARGE=$n$}.  This will cause the geometry to
be printed once every $n$ steps.

The effect of using these flags together is as follows.
\begin{enumerate}
\item No options:  All calculated points will be printed.  No turning points
will be calculated.

\item Atom 1 bond length flagged with a ``T'': If  \comp{T-PRIO},  etc.\ are
NOT  specified,  then  potential  energy turning points will be printed.

\item Internal coordinate flags set to ``T'':  If \comp{T-PRIO}, etc.  are NOT
specified,  then geometry extrema will be printed.  If only one coordinate is
flagged, then the turning point will be displayed in  chronologic  order; if
several are flagged then all turning points occurring in a given time-interval
will  be  printed  as they  are  detected.   In  other  words,  some  may  be
out of chronologic order.  Note that each coordinate flagged will give rise  to
a different geometry:  minimize flagged coordinates to minimize output.

\item Potential and geometric flags set:  The effect is equivalent to the sum
of the first two options.

\item \comp{T-PRIO} set:  No turning points will be  printed,  but  constant
time-slices  (by  default  $0.1$~fs)  will  be used to control the print.
\end{enumerate}
\index{DRC|)}\index{IRC|)}