\documentclass[a4paper]{article}

\usepackage[english]{babel}
\usepackage[utf8]{inputenc}
\usepackage{amsmath}
\usepackage{graphicx}
\usepackage[colorinlistoftodos]{todonotes}
\usepackage{hyperref}
\usepackage{listings}
\usepackage[numbers]{natbib}

\usepackage{graphicx}
\usepackage{babel,blindtext}

\usepackage{algorithm}
\usepackage[noend]{algpseudocode}

\usepackage{booktabs} % To thicken table lines

\title{Train a Smartcab How to Drive}

\author{Uirá Caiado}

\date{\today}

\begin{document}

\maketitle

\begin{abstract}
A smartcab is a self-driving car from the not-so-distant future that ferries people from one arbitrary location to another. In this project, I will design the AI driving agent for the smartcab using reinforcement learning. This area of machine learning\footnote{Source: \url{https://en.wikipedia.org/wiki/Reinforcement_learning}} is inspired by behaviorist psychology and consists in training the agent by reward and punishment without needing to specify how the task is to be achieved. The agent should learn an optimal policy for driving on city roads, obeying traffic rules correctly, and trying to reach the destination within a goal time.
\end{abstract}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% INTRODUCTION
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Introduction}
\label{sec:introduction}
In this section, I will present a brief introduction to reinforcement learning and to the problem addressed by this project.

\subsection{Reinforcement Learning}
As explained by \cite{Mohri_2012}, reinforcement learning is the study of planning and learning in a scenario where a learner (or agent) actively interacts with the environment to achieve a particular goal. The achievement of the agent's goal is typically measured by the reward he receives from the environment and which he seeks to maximize.

\cite{Kaelbling_1996} state that the most significant difference between reinforcement learning and supervised learning is that there is no presentation of input/output pairs. Instead, they explained that after choosing an action, the agent is told the immediate reward and the following state, but is not told which action would have been in its best long-term interests. It is necessary for the agent to gather useful experience about the possible system states, actions, transitions and rewards actively to act optimally.

Defining a general formulation of the problem based on a Markov Decision Process (MDP), as proposed by \cite{Mitchell}, the agent can perceive a set $S$ os distinct states of its environment and has a set $A$ of actions that it can perform. So, at each discrete time step $t$, the agent senses the current state $s_t$ and choose to take an action $a_t$. The environment responds by giving the agent a reward $r_t=r(s_t, a_t)$ and by producing the succeeding state $s_{t+1}=\delta(s_t, a_t)$. The functions $r$ and $\delta$ only depend on the current state and action (it is memoryless\footnote{Source: \url{https://en.wikipedia.org/wiki/Markov_process}}), are part of the environment and are not necessarily known to the agent.

The task of the agent is to learn a policy $\pi$ that maps each state to an action ($\pi: S \rightarrow A$), selecting its next action $a_t$ based solely on the current observed state $s_t$, that is $\pi(s_t)=a_t$. The optimal policy, or control strategy, is the one that produces the greatest possible cumulative reward over time. So, stating that:

$$V^{\pi}(s_t)= r_t + \gamma r_{t+1} + \gamma^2 r_{t+1} + ... = \sum_{i=0}^{\infty} \gamma^{i} r_{t+i}$$

Where $V^{\pi}(s_t)$ is also called the discounted cumulative reward and it represents the cumulative value achieved by following an policy $\pi$ from an initial state $s_t$ and $\gamma \in [0, 1]$ is a constant that determines the relative value of delayed versus immediate rewards. If we set $\gamma=0$, only immediate rewards is considered. As $\gamma \rightarrow 1$, future rewards are given greater emphasis relative to immediate reward. The optimal policy $\pi^{*}$ that will maximizes $V^{\pi}(s_t)$ for all states $s$ can be written as:

$$\pi^{*} = \underset{\pi}{\arg \max} \, V^{\pi} (s)\,\,\,\,\,, \,\, \forall s$$

As learning $\pi^{*}: S \rightarrow A$ directly is difficult because the available training data does not provide training examples of the form $(s, a)$, in the section \ref{sec:implement_q_learning} I will implement the Q-learning algorithm for estimating the optimal policy.

\subsection{What Will Be Done}

The goal of this project\footnote{Source: \url{https://goo.gl/BZdyLo}} is to design the AI driving agent for the smartcab, that operates in an idealized grid-like city.

The smartcab is able to sense whether the traffic light is green for its direction of movement and whether there is a car at the intersection on each of the incoming roadways (and which direction they are trying to go). In addition to this, each trip has an associated timer that counts down every time step. If the timer is at 0 and the destination has not been reached, the trip is over, and a new one may start.

It should receive the inputs mentioned above at each time step t, and generate an output move, that consists on to stay put at the current intersection, move one block forward, one block left, or one block right (no backward movement).

The smartcab also should receive a reward for each successfully completed trip. A trip is considered “successfully completed” if the passenger is dropped off at the desired destination within a pre-specified time bound. It also gets a smaller reward for each correct move executed at an intersection. It gets a minor penalty for a wrong move and a larger penalty for violating traffic rules and/or causing an accident.

Based on the rewards and penalties it gets, the agent should learn an optimal policy for driving on city roads, obeying traffic rules correctly, and trying to reach the destination within a goal time.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% IMPLEMENT A BASIC DRIVING AGENT
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Implement a Basic Driving Agent}
\label{sec:implement_driving_agent}
In this section, I will implement a basic driving agent that accepts specified inputs ans produces a valid output.

\subsection{My Grid-Like World}
Let's start by testing the output produced by my current \textit{'world'} that will be used to analyze the results of this project. This \textit{'world'} is a grid-like city, with roads going North-South and East-West. Other vehicles may be present on the streets, but no pedestrians. There is a traffic light at each intersection that can be in one of two states: North-South open or East-West open. US right-of-way rules apply: On a green light, you can turn left only if there is no oncoming traffic at the intersection coming straight. On a red light, you can turn right if there is no oncoming traffic turning left or traffic from the left going straight.

We are told to assume that a higher-level planner assigns a route to the smartcab, splitting it into waypoints at each intersection. The time in this world is quantized and at any instant, the smartcab is at some intersection. Therefore, the next waypoint is always either one block straight ahead, one block left, one block right, one block back or exactly there (reached the destination).Below is a sample of the log file generated by the simulator.

% code snipet
\begin{lstlisting}
<Time>;Environment.reset():Trial set up with start=(4, 3)
                      ,destination=(8, 4), deadline = 25
<Time>;RoutePlanner.route_to(): destination = (8, 4)
<Time>;LearningAgent.update(): deadline = 25,
    inputs = {'light': 'green', 'oncoming': None,
              'right': 'forward', 'left': None},
    action = None, reward = 0.0
...
<Time>;Environment.step(): Primary agent ran out of time!
Trial aborted.
\end{lstlisting}

\subsection{The Basic Agent}
As mentioned before, I am going to implement a Smartcab that processes the following inputs at each time step:

\begin{itemize}
\item \textit{Next waypoint location}: about its current position and heading
\item \textit{Intersection state}: traffic light and presence of cars
\item \textit{Current deadline value}: time steps remaining
\end{itemize}

For the purposes of this project, the agent should produce just some random move, like \textit{'None'}, \textit{'forward'}, \textit{'left'} and \textit{'right'}. I will not implement the correct strategy because it is precisely what my agent is supposed to learn.

For the purposes of this project, the first implementation of the agent should produce just some random move at each time $t$, such that $a_t \in (None,\, forward,\,$
$ left,\, right)$. I will not implement the correct strategy because it is precisely what my agent is supposed to learn. Below I am going to simulate $100$ different trials with the \textit{'enforce\_deadline'} set to False and will save the logs produced by the agent to observe how it performs. In this mode, the agent is given unlimited time to reach the destination.

% code snipet
\begin{lstlisting}
Number of Trials: 100
Times that the agent reached the target location: 69
Times the agent reached the hard deadline: 31
Times the agent successfully reached the target: 21
\end{lstlisting}

As can be seen above, even using just random moves, the agent still was able to reach the target destination roughly $20\%$ of the trials at the deadline stipulated by the Planner. If we considering all the times it reached the location, it was able to complete the route almost $70\%$ of the trials. Now, I am going to produce same basic statistics about the times that it reached the target location.

% \begin{figure}[ht!]
% \centering
% \includegraphics[width=0.75\textwidth]{images/steps_takes.png}
% \caption{\label{fig:boxplot}Number of Steps Used to Reached The Destination}
% \end{figure}

\begin{table}[ht!]
\centering
\begin{tabular}{l|r}
\multicolumn{2}{ c }{Number Of Steps} \\
\midrule
mean  &         59.00 \\
std   &         36.06 \\
min   &          2.00 \\
25\%   &         23.75 \\
50\%   &         60.00 \\
75\%   &         91.00 \\
max   &        130.00 \\
\end{tabular}
\caption{\label{tab:basic_stats}Basic Agent Simulation Statistics}
\end{table}

According to the table \ref{tab:basic_stats}, It took 59 steps on average, and the variation was huge. The agent has taken from 2 steps to 130 to finish the route. In the section \ref{sec:enhance_driving_agent}, I will try to improve that.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% IDENTIFY AND UPDATE STATE
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Identify and Update State}
To complete the implementation of my Basic Agent, I still need to decide how it will represent its state internally. Considering the inputs that it receives before performs any action and, consequently, receives a reward, I initially will use a tuple using all of them to represent the current state of my agent, in the form $(inputs,\, next\_waypoint,\, deadline)$. I believe that it will represent a reasonable set of states that my agent could use to gather more information from the environment.

$Inputs$ is a dictionary with the state of the traffic light for the agents' direction of movement and the direction of the random agents on each of the incoming roadways (if any). $next\_waypoint$ is the direction of the target location and $deadline$, a timer.

I believe that $next\_waypoint$ and $inputs$ are natural choices, given that the first gives the direction of the target and the second provides information about the environment state. The inclusion of $deadline$ might help the agent choose to explore the environment when it has more time available, for example. Later on, we will see how significant this single decision is when dealing with Reinforcement Learning problems.

So, let's count the number of stated that I get when I enforce deadline in my simulation on $100$ trials.

\begin{lstlisting}
number of states in the state space: 891
\end{lstlisting}

\begin{figure}[ht!]
\centering
\includegraphics[width=0.75\textwidth]{images/basic_agent_behaviour.png}
\caption{\label{fig:hist_basic_agent}Count Of Times That Each State Appeared}
\end{figure}

The figure \ref{fig:hist_basic_agent} presents the frequency of the counting of each state. For instance, around $600$ states appeared just two times, more than $100$ appeared four times and so on. Curiously the agent's behavior distribution looks like a log-normal distribution.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% IMPLEMENT Q-LEARNING
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Implement Q-Learning}
\label{sec:implement_q_learning}
In this section, I will briefly explained the concept of Q-learning and implement an algorithm to learn the Q function.

\subsection{The Q-Function}
As mention before, to learn an optimal policy $\pi^{*}$, I can't learn a function $\pi^{*}: S \rightarrow A$ that maps a state to the optimal action directly. There is no such information upfront to be used as training data. Instead, as \cite{Mitchell} explained, the only available information is the sequence of immediate rewards $r(s_i, a_i)$ for $i=1,\, 2,\, 3,\,...$

So, as we are trying to maximize the cumulative rewards $V^{*}(s_t)$ for all states $s$, the agent should prefer $s_1$ over $s_2$ wherever $V^{*}(s_1) > V^{*}(s_2)$. Given that the agent must choose among actions and not states, and it isn't able to perfectly predict the immediate reward and immediate successor for every possible state-action transition, we also must learn $V^{*}$ indirectly.

To solve that, we define a function $Q(s, \, a)$ such that its value is the maximum discounted cumulative reward that can be achieved starting from state $s$ and applying action $a$ as the first action. So, we can write:

$$Q(s, \, a) = r(s, a) + \gamma V^{*}(\delta(s, a))$$

As $\delta(s, a)$ is the state resulting from applying action $a$ to state $s$ (the successor) chosen by following the optimal policy, $V^{*}$ is the cumulative value of the immediate successor state discounted by a factor $\gamma$. Thus,  what we are trying to achieve is

$$\pi^{*}(s) = \underset{a}{\arg \max} Q(s, \, a)$$

Thus, the optimal policy can be obtained even if the agent just uses the current action $a$ and state $s$ and chooses the action that maximizes $Q(s,\, a)$. Also, it is important to notice that the function above implies that the agent can select optimal actions even when it has no knowledge of the functions $r$ and $\delta$. In the next subsection, we will see how \cite{Mitchell} defined a reliable way to estimate training values for $Q$, given only a sequence of immediate rewards $r$.

\subsection{Learning Q}
As we have seen, learning the Q function corresponds to learning the optimal policy. According to \cite{Mohri_2012}, the optimal state-action value function $Q^{*}$ is defined for all $(s, \, a) \in S \times A$ as the expected return for taking the action $a \in A$ at the state $s \in S$, following the optimal policy. So, it can be written as \cite{Mitchell} suggested:

$$V^{*}(s) = \underset{a'}{\arg \max} Q(s, \, a')$$

Using this relationship, we can write a recursive definition of Q function, such that:

$$Q(s, \, a) = r(s, a) + \gamma \, \underset{a'}{\max} \, Q(\delta(s,\, a), \, a')$$

The recursive nature of the function above implies that our agent doesn't know the actual $Q$ function. It just can estimate $Q$, that we will refer as $\hat{Q}$. It will represents is hypothesis $\hat{Q}$ as a large table that attributes each pair $(s\, , \, a)$ to a value for $\hat{Q}(s,\, a)$ - the current hypothesis about the actual but unknown value $Q(s, \, a)$. I will initialize this table with zeros, but it could be filled with random numbers, according to \cite{Mitchell}. Still according to him, the agent repeatedly should do the following:

\begin{algorithm}
\caption{Update Q-table}\label{alg:qtable}
\begin{algorithmic}[1]
\Loop{ Observe the current state $s$ and:}
  \State Choose some action $a$ and execute it
  \State Receive the immediate reward $r = r(s, a)$
  \State initialize the table entry $\hat{Q}(s, \, a)$ to zero if there is no entry $(s, \, a)$
  \State Observe the new state $s' = \delta(s, \,a)$
  \State Updates the table entry for $\hat{Q}(s, \, a)$ following:

    \do $\hat{Q}(s, \, a) \leftarrow r + \gamma \, \underset{a'}{\max}\, \hat{Q}(s', \, a')$
  \State $s \leftarrow s'$
\EndLoop
\end{algorithmic}
\end{algorithm}

Note this training rule is suited to a deterministic Markov decision process (where $r(s,\, a)$ and $\delta(s,\, a)$ are deterministic) and uses the agent's current $\hat{Q}$ values for the new state $s'$ to refine its estimate of $\hat{Q}(s, \,a)$ for the previous state $s$. It is tricky. Let's take a look at its performance.

\begin{lstlisting}
Number of Trials: 100
Times that the agent reached the target location: 55
Times the agent reached the hard deadline: 45
Times the agent successfully reached the target: 55
\end{lstlisting}

The agent is reaching its destination much more frequently than before. When the agent selected its action randomly, it reached its destination just 21\% of the times. Now, it is reaching 55\%. Now the agent is always taking the "safest action":  takes the action with the biggest reward. It is not bad, but looking at the number of explored actions by state in the Q-table produced by this simulation, we can notice something curious

\begin{table}[ht!]
\centering
\begin{tabular}{l|r}
\multicolumn{2}{ c }{Number of Explored Actions} \\
\midrule
1                          &  56.28\% \\
2                          &  21.10\% \\
3                          &  15.83\% \\
4                          &  6.78\% \\
\end{tabular}
\caption{\label{tab:explored_actions}Explored Actions in The Q-Table}
\end{table}

The agent explored just one possible action in $56\%$ of the states in the Q-Table. According to \cite{Mitchell}, there are some conditions to ensure that the algorithm converges toward a $\hat{Q}$ equal to the true $Q$ function. One of them is that the system should be a deterministic MDP. Another is that the agent must select actions in a way that it visits every possible state-action pair infinitely often. Looking at the number of times this implementation of the agent explored all $4$ possibles actions (less than $7\%$ of the states), I would say that it isn't true. In the next section, I will try to improve te agent's performance.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% ENHANCE DRIVING AGENT
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Enhance the Driving Agent}
\label{sec:enhance_driving_agent}
In this section, I will try to improve the driving agent to make it consistently reaches the destination within allotted time and the net reward positive.

\subsection{Experimentation Strategies}
One of the issues of the current strategy is that the agent could over-commit to actions that presented positive $\hat{Q}$ values early in the simulation, failing to explore other actions that could present even higher values. \cite{Mitchell} proposed to use a probabilistic approach to select actions, assigning higher probabilities to action with high $\hat{Q}$ values, but given to every action at least a nonzero probability. So, I will implement the following relation:

$$P(a_i\, | \,s ) = \frac{k ^{\hat{Q}(s, a_i)}}{\sum_j k^{\hat{Q}(s, a_j)}}$$

Where $P(a_i\, | \,s )$ is the probability of selecting the action $a_i$ given the state $s$. The constant $k$ is positive and determines how strongly the selection favors action with high $\hat{Q}$ values. In the figure \ref{fig:exploint_explore} we can see the number of times the algorithm choose to explore when we vary the value of the $k$.

Large values of $k$ assigned a higher probability of the agent exploits what it already has learned. Small values have made the agent favor the exploration of new actions. In the figure \ref{fig:success_runs} we can check how the different values of $k$ impacted on the times the agent completed the routes.

\begin{figure}[ht!]
\minipage{0.49\textwidth}
  \includegraphics[width=\linewidth]{images/exploint_explore.png}
  \caption{\label{fig:exploint_explore}Exploration vs Exploitation}
\endminipage
\minipage{0.49\textwidth}
  \includegraphics[width=\linewidth]{images/success_runs.png}
  \caption{\label{fig:success_runs}On Time Ratio}
\endminipage
\end{figure}

The agent presented a very steep improvement as $k$ was approaching $3$. Above this value, the completed trips ratio just showed a slight growth. Now, let's check how the number of trials changed the average reward received by the agent.

\begin{figure}[ht!]
\centering
\includegraphics[width=0.75\textwidth]{images/cum_reward_by_k.png}
\caption{\label{fig:cum_reward_by_k}Average Cumulative Reward By Value of $k$}
\end{figure}

Clearly, small values of $k$ are worst than large values, as can be seen at the figure \ref{fig:cum_reward_by_k}. As I want that my agent has some probability of exploring new actions, and it seems that $k=3$ is the best choice (the "slope" is not so steep, and the start value is pretty high), I will use this parameter to perform the next simulations.

\subsection{Non-deterministic Rewards and Actions}
The last modification that I want to introduce is when the environment is non-deterministic. As I said before, the algorithm used is appropriate to the deterministic case. On a nondeterministic environment, the reward function $r(s,\,a)$ and action transition function $\delta(s,\,a)$ may have probabilistic outcomes. As there are other agents in the Smartcab world, I believe that it is a proper assumption.

As explained by \cite{Mitchell}, to handle nondeterministic MDPs, it is needed to redefine the value $V^{*}$ to be expressed as a expected value. So, jumping to the $Q$ function, we can re-express $Q$ recursively as:

\begin{equation}
\begin{aligned}
Q(s, \, a) &= E \left [r(s, a) \right ] + E \left [\gamma \, \underset{a'}{\max} \, Q(\delta(s,\, a), \, a')\right ]\\
&= E \left [r(s, a) \right ] + \gamma \,  E \left [\underset{a'}{\max} \, Q(\delta(s,\, a), \, a')\right ]\\
&= E \left [r(s, a) \right ] + \gamma \,  \sum_{s'} P\left(s' \,| \,s, a   \right) \underset{a'}{\max} \, Q(\delta(s,\, a), \, a')
\end{aligned}
\end{equation}

Where $P\left(s' \,| \,s, a   \right)$ is the probability that taking action $a$ in state $s$ will produce the next state $s'$. To implement that, we need to assume that the $\hat{Q}$ values in our table may be wrong, due to the non-deterministic behavior of our environment. So, \cite{Mitchell} suggested to modifying the training rule by introducing a decaying weighted average factor to the current $\hat{Q}$ value and to the revised estimate, such that:

$$\hat{Q}(s, \, a) \leftarrow  (1 - \alpha_{n})\, \hat{Q}_{n-1}(s, \, a) +  \alpha_{n}\left[ r + \gamma \underset{a'}{\max} \hat{Q}_{n-1}(s', \, a') \right ]$$

Where $\alpha_{n} = \frac{1}{1 + visits_{n}(s, \,a)}$ is the decaying factor and $visits_{n}$ is the total number of times this state-action $(s, \, a)$ pair has been visited. So, let's put it in practice.

\begin{lstlisting}
Number of Trials: 100
Times that the agent reached the target location: 58
Times the agent reached the hard deadline: 42
Times the agent successfully reached the target: 58
\end{lstlisting}

Looking just at the times the agent has completed the route, it doesn't look like we made much progress. I will compare the performance of all the agents created so far in the last section.

\newpage
\subsection{The last parameter to Tune: Gamma}
There is just one remain parameter of the model that was not explored yet:  the $\gamma$. As already explained, the $\gamma$ in the $\hat{Q}$ function is a factor used to discount the immediate successor state to be used to update $\hat{Q}$ table. As $\gamma$ approaches $0$, less "memory" the agent has. So, let's check what happens when we vary this parameter about the average cumulative reward of the learning agent:

\begin{figure}[ht!]
\centering
\includegraphics[width=0.75\textwidth]{images/cum_reward_by_gamma.png}
\caption{\label{fig:avg_cum_rwd}Average Cumulative Reward By Value of $\gamma$}
\end{figure}

In the last section, I will use $\gamma = 0.5$ to perform the final tests.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% CONCLUSION
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Conclusion}
\label{sec:conclusion}
In this section, I will compare the performance of the algorithms implemented so far and then I will show how a little change can make a huge difference in a reinforcement learning problem.

\subsection{Final Remarks}
The goal of the whole project was to build an agent that obeys traffic rules correctly and reachs the destination within an allotted time. So, the "perfect" agent would be the one that always come to the destination in the goal time without any penalty.

However, as our environment is non-deterministic, to always reach the destination on time is not reasonable. Also, as the environment penalizes the agent even when it makes a correct move, but in a distinct direction from the target location, it isn't fair to expect that the perfect agent never incurs any penalty.

So, the best one would be the one that learns a reasonable policy fast and receives the maximum average cumulative reward possible. Average because we should weight the rewards received by the number of steps used to avoid discrepancies. In this sense, let's first compare how the modifications made on this project reflected on our agent when we ensure deadline. The improved agent will be set with $\gamma=0.5$ and $k=3$.

\begin{figure}[ht!]
\centering
\includegraphics[width=0.75\textwidth]{images/performance_comparition.png}
\caption{\label{fig:perf_compr}Cumulative Reward By Agent}
\end{figure}

As can be seen above, the improved agent did better than the other implementations. It reached the destination $64\%$ of the trials on time and the final average reward by action was greater than the others. It was achieved by tuning the parameters, including an experimentation strategy and changing the $\hat{Q}$ function to deal with the non-deterministic environment, as explained before. The random agent reached the destination on time just $42\%$ of the trials and the agent that used the deterministic version of $\hat{Q}$ function, $55\%$. Although the final agent is still incurring penalties, it is closer to a perfect agent than it was when using just random moves. In the next subsection, I will show how to change the behavior of the agent radically.

\subsection{Blowing Up Everything: Changing The Intern State}
As pointed out by the Udacity Reviewer, something that I should have taken into account was the size of state space resulted by my choice of agents' intern state representation. I included an Experimentation Strategy and addressed a non-deterministic MDP environment exactly due to the number of states that the agent would need to learn. It turned out being the right choice, as demonstrated in the last subsection.

However, I believe that I underestimated this single decision. My current state is represented by the tuple $(inputs,\, next\_waypoint,\, deadline)$. The $deadline$ is assigned by the Environment in a non-deterministic way, what makes my state space unlikely to learn in $100$ trials. In the figure \ref{fig:perf_compr_2} is possible to see what happens If I discard this single variable in the performance of all my $LearningAgent$.

\begin{figure}[ht!]
\centering
\includegraphics[width=0.75\textwidth]{images/performance_comparition_2.png}
\caption{\label{fig:perf_compr_2}Cumulative Reward By Agent}
\end{figure}

Using this inter-state representation, the final version of the agent is much closer to a perfect agent. It reached the destination at $90\%$ of the trials and learned most of what need to learn in less than $60$ trials.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% REFLECTION
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Reflection}
\label{sec:reflection}
Something really valuable that I learned from reviewers is the importance of the state space representation and how it affects the performance of the learning agent.

Still, it's hard to say how better would be the performance of a perfect agent to contrast to the current implementation, given the stochastic environment. An option would be to implement a learning agent that already knows the traffic rules and just need to learn how to find his way to the destination on time.

\bibliographystyle{plain}
% or try abbrvnat or unsrtnat
\bibliography{bibliography/biblio.bib}
\end{document}


\section{Dimension theory}


\chapter[]{Concurrency Problems}
\section{Dining Philosophers}\label{app:dining_phil}
The Dining Philosophers problem is defined as follows:
\bsqoute{Five philosophers spend their lives thinking and eating. The philosophers share a common dining room where there is a circular table surrounded by five chairs, each belonging to one philosopher. In the center of the table there is a large bowl of spaghetti, and the table is laid with five forks (see \bsref{fig:dining_philosophers}). On feeling hungry, a philosopher enters the dining room, sits in his own chair, and picks up the fork on the left of his place. Unfortunately, the spaghetti is so tangled that he needs to pick up and use the fork on his right as well. When he has finished, he puts down both forks, and leaves the room. The room should keep a count of the number of philosophers in it.}{Edgar Dijkstra\cite[p. 673]{hoare1978communicating}}

\begin{figure}[htbp]
\centering
\includegraphics[width=0.7\textwidth]{\rootpath/worksheets/appendix/figures/dining_philosophers}
\caption{The Dining Philosophers}
\label{fig:dining_philosophers}
\end{figure}

\section{The Santa Claus Problem}\label{app:santa}
The Santa Claus problem is defined as follows:
\bsqoute{Santa Claus sleeps in his shop up at the North Pole, and can only be wakened by either all nine reindeer being back from their year long vacation on the beaches of some tropical island in the South Pacific, or by some elves who are having some difficulties making the toys. One elf's problem is never serious enough to wake up Santa (otherwise, he may \b{never} get any sleep), so, the elves visit Santa in a group of three. When three elves are having their problems solved, any other elves wishing to visit Santa must wait for those elves to return. If Santa wakes up to find three elves waiting at his shop's door, along with the last reindeer having come back from the tropics, Santa has decided that the elves can wait until after Christmas, because it is more important to get his sleigh ready as soon as possible. (It is assumed that the reindeer don't want to leave the tropics, and therefore they stay there until the last possible moment. They might not even come back, but since Santa is footing the bill for their year in paradise … This could also explain the quickness in their delivering of presents, since the reindeer can't wait to get back to where it is warm.) The penalty for the last reindeer to arrive is that it must get Santa while the others wait in a warming hut before being harnessed to the sleigh.}{John A. Trono\cite{trono1994new}}

\section{Concurrent Queue}\label{app:queue}
A queue is a \ac{FIFO} data structure, operating like the queue at a cash register\cite[p. 234]{cormen2009introduction}. A queue supports the operations \bscode{Enqueue} and \bscode{Dequeue}. \bscode{Enqueue} inserts an item at the back of the queue while \bscode{Dequeue} gets and removes the item at the front of the queue, per the \ac{FIFO} principle. A queue is commonly implemented using a linked list of nodes\cite[p. 234]{cormen2009introduction}. To be able to enqueue and dequeue, the queue holds a reference to the first and last node in the list, referred to as the head and tail of the queue.
\section{Concurrent HashMap}\label{app:hashmap}
A HashMap is a data structure which allows looking up values based on their associated key and provides the operations Add, Get, and Remove. It benefits from the Get operation having an average time complexity of \textit{O}(1) under reasonable assumptions\cite[p. 256]{cormen2009introduction}. Implementation details of a HashMap may vary, but traditionally they follow the concept described in\cite[p. 256]{cormen2009introduction}. Internally it uses an array for storing a list of key/value pairs. When inserting a value into the HashMap, the  array index is commonly calculated as:
\begin{equation}
Hash(key) \bmod arraylength
\end{equation}
That is, a hash function applied to the key, modulo the size of the backing array. If another key is already stored in the array index calculated, it is said to be a collision. To handle this, the value is not stored directly in the array index, but in a list referred to as a bucket. This bucket is implemented as a linked list as this ensures fast insertions\cite[p. 257]{cormen2009introduction}. Index based lookup and removal is not needed as the index of a particular key is not known. Therefore it is always required to iterate through the bucket list when searching for a particular key. The calculation of the bucket index must distribute the values evenly to lessen the collisions. When a certain percentage of the buckets are filled, the internal array must be resized to keep the risk of collisions low. This internal operation is expensive in terms of both time and space, as a new internal array of an increased size must be allocated, and the key/value pairs must be inserted into the new array.


%
% API Documentation for QSTK
% Module QSTK.qstklearn.fastknn
%
% Generated by epydoc 3.0.1
% [Mon Mar  5 00:49:20 2012]
%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%                          Module Description                           %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

    \index{QSTK \textit{(package)}!QSTK.qstklearn \textit{(package)}!QSTK.qstklearn.fastknn \textit{(module)}|(}
\section{Module QSTK.qstklearn.fastknn}

    \label{QSTK:qstklearn:fastknn}
This package is an implementation of a novel improvement to KNN which
speeds up query times

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%                               Functions                               %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

  \subsection{Functions}

    \label{QSTK:qstklearn:fastknn:adistfun}
    \index{QSTK \textit{(package)}!QSTK.qstklearn \textit{(package)}!QSTK.qstklearn.fastknn \textit{(module)}!QSTK.qstklearn.fastknn.adistfun \textit{(function)}}

    \vspace{0.5ex}

\hspace{.8\funcindent}\begin{boxedminipage}{\funcwidth}

    \raggedright \textbf{adistfun}(\textit{u}, \textit{v})

\setlength{\parskip}{2ex}
\setlength{\parskip}{1ex}
    \end{boxedminipage}

    \label{QSTK:qstklearn:fastknn:dataifywine}
    \index{QSTK \textit{(package)}!QSTK.qstklearn \textit{(package)}!QSTK.qstklearn.fastknn \textit{(module)}!QSTK.qstklearn.fastknn.dataifywine \textit{(function)}}

    \vspace{0.5ex}

\hspace{.8\funcindent}\begin{boxedminipage}{\funcwidth}

    \raggedright \textbf{dataifywine}(\textit{fname})

\setlength{\parskip}{2ex}
\setlength{\parskip}{1ex}
    \end{boxedminipage}

    \label{QSTK:qstklearn:fastknn:testwine}
    \index{QSTK \textit{(package)}!QSTK.qstklearn \textit{(package)}!QSTK.qstklearn.fastknn \textit{(module)}!QSTK.qstklearn.fastknn.testwine \textit{(function)}}

    \vspace{0.5ex}

\hspace{.8\funcindent}\begin{boxedminipage}{\funcwidth}

    \raggedright \textbf{testwine}()

\setlength{\parskip}{2ex}
\setlength{\parskip}{1ex}
    \end{boxedminipage}

    \label{QSTK:qstklearn:fastknn:testspiral}
    \index{QSTK \textit{(package)}!QSTK.qstklearn \textit{(package)}!QSTK.qstklearn.fastknn \textit{(module)}!QSTK.qstklearn.fastknn.testspiral \textit{(function)}}

    \vspace{0.5ex}

\hspace{.8\funcindent}\begin{boxedminipage}{\funcwidth}

    \raggedright \textbf{testspiral}()

\setlength{\parskip}{2ex}
\setlength{\parskip}{1ex}
    \end{boxedminipage}

    \label{QSTK:qstklearn:fastknn:getflatcsv}
    \index{QSTK \textit{(package)}!QSTK.qstklearn \textit{(package)}!QSTK.qstklearn.fastknn \textit{(module)}!QSTK.qstklearn.fastknn.getflatcsv \textit{(function)}}

    \vspace{0.5ex}

\hspace{.8\funcindent}\begin{boxedminipage}{\funcwidth}

    \raggedright \textbf{getflatcsv}(\textit{fname})

\setlength{\parskip}{2ex}
\setlength{\parskip}{1ex}
    \end{boxedminipage}

    \label{QSTK:qstklearn:fastknn:testgendata}
    \index{QSTK \textit{(package)}!QSTK.qstklearn \textit{(package)}!QSTK.qstklearn.fastknn \textit{(module)}!QSTK.qstklearn.fastknn.testgendata \textit{(function)}}

    \vspace{0.5ex}

\hspace{.8\funcindent}\begin{boxedminipage}{\funcwidth}

    \raggedright \textbf{testgendata}()

\setlength{\parskip}{2ex}
\setlength{\parskip}{1ex}
    \end{boxedminipage}

    \label{QSTK:qstklearn:fastknn:test}
    \index{QSTK \textit{(package)}!QSTK.qstklearn \textit{(package)}!QSTK.qstklearn.fastknn \textit{(module)}!QSTK.qstklearn.fastknn.test \textit{(function)}}

    \vspace{0.5ex}

\hspace{.8\funcindent}\begin{boxedminipage}{\funcwidth}

    \raggedright \textbf{test}()

\setlength{\parskip}{2ex}
\setlength{\parskip}{1ex}
    \end{boxedminipage}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%                               Variables                               %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

  \subsection{Variables}

    \vspace{-1cm}
\hspace{\varindent}\begin{longtable}{|p{\varnamewidth}|p{\vardescrwidth}|l}
\cline{1-2}
\cline{1-2} \centering \textbf{Name} & \centering \textbf{Description}& \\
\cline{1-2}
\endhead\cline{1-2}\multicolumn{3}{r}{\small\textit{continued on next page}}\\\endfoot\cline{1-2}
\endlastfoot\raggedright \_\-\_\-p\-a\-c\-k\-a\-g\-e\-\_\-\_\- & \raggedright \textbf{Value:}
{\tt \texttt{'}\texttt{QSTK.qstklearn}\texttt{'}}&\\
\cline{1-2}
\end{longtable}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%                           Class Description                           %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

    \index{QSTK \textit{(package)}!QSTK.qstklearn \textit{(package)}!QSTK.qstklearn.fastknn \textit{(module)}!QSTK.qstklearn.fastknn.FastKNN \textit{(class)}|(}
\subsection{Class FastKNN}

    \label{QSTK:qstklearn:fastknn:FastKNN}
A class which implements the KNN learning algorithm with sped up query
times.

This class follows the conventions of other classes in the qstklearn
module, with a constructor that initializes basic parameters and
bookkeeping variables, an 'addEvidence' method for adding labeled training
data individually or as a batch, and a 'query' method that returns an
estimated class for an unlabled point.  Training and testing data are in
the form of numpy arrays, and classes are discrete.

In order to speed up query times, this class keeps a number of lists which
sort the training data by distance to 'anchor' points.  The lists aren't
sorted until the first call to the 'query' method, after which, the lists
are kept in sorted order. Initial sort is done using pythons 'sort'
(samplesort), and sorted insertions with 'insort' from the bisect module.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%                                Methods                                %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

  \subsubsection{Methods}

    \label{QSTK:qstklearn:fastknn:FastKNN:__init__}
    \index{QSTK \textit{(package)}!QSTK.qstklearn \textit{(package)}!QSTK.qstklearn.fastknn \textit{(module)}!QSTK.qstklearn.fastknn.FastKNN \textit{(class)}!QSTK.qstklearn.fastknn.FastKNN.\_\_init\_\_ \textit{(method)}}

    \vspace{0.5ex}

\hspace{.8\funcindent}\begin{boxedminipage}{\funcwidth}

    \raggedright \textbf{\_\_init\_\_}(\textit{self}, \textit{num\_anchors}, \textit{k})

    \vspace{-1.5ex}

    \rule{\textwidth}{0.5\fboxrule}
\setlength{\parskip}{2ex}
    Creates a new FastKNN object that will use the given number of anchors.

\setlength{\parskip}{1ex}
    \end{boxedminipage}

    \label{QSTK:qstklearn:fastknn:FastKNN:resetAnchors}
    \index{QSTK \textit{(package)}!QSTK.qstklearn \textit{(package)}!QSTK.qstklearn.fastknn \textit{(module)}!QSTK.qstklearn.fastknn.FastKNN \textit{(class)}!QSTK.qstklearn.fastknn.FastKNN.resetAnchors \textit{(method)}}

    \vspace{0.5ex}

\hspace{.8\funcindent}\begin{boxedminipage}{\funcwidth}

    \raggedright \textbf{resetAnchors}(\textit{self}, \textit{selection\_type}={\tt \texttt{'}\texttt{random}\texttt{'}})

    \vspace{-1.5ex}

    \rule{\textwidth}{0.5\fboxrule}
\setlength{\parskip}{2ex}
    Picks a new set of anchors.  The anchor lists will be re-sorted upon
    the next call to 'query'.

    selection\_type - the method to use when selecting new anchor points.
    'random' performs a random permutation of the training points and picks
    the first 'num\_anchors' as new anchors.

\setlength{\parskip}{1ex}
    \end{boxedminipage}

    \label{QSTK:qstklearn:fastknn:FastKNN:addEvidence}
    \index{QSTK \textit{(package)}!QSTK.qstklearn \textit{(package)}!QSTK.qstklearn.fastknn \textit{(module)}!QSTK.qstklearn.fastknn.FastKNN \textit{(class)}!QSTK.qstklearn.fastknn.FastKNN.addEvidence \textit{(method)}}

    \vspace{0.5ex}

\hspace{.8\funcindent}\begin{boxedminipage}{\funcwidth}

    \raggedright \textbf{addEvidence}(\textit{self}, \textit{data}, \textit{label})

    \vspace{-1.5ex}

    \rule{\textwidth}{0.5\fboxrule}
\setlength{\parskip}{2ex}
    Adds to the set of training data. If the anchor lists were sorted
    before the call to this method, the new data will be inserted into the
    anchor lists using 'bisect.insort'

    data - a numpy array, either a single point (1D) or a set of points
    (2D)

    label - the label for data. A single value, or a list of values in the
    same order as the points in data.

\setlength{\parskip}{1ex}
    \end{boxedminipage}

    \label{QSTK:qstklearn:fastknn:FastKNN:query}
    \index{QSTK \textit{(package)}!QSTK.qstklearn \textit{(package)}!QSTK.qstklearn.fastknn \textit{(module)}!QSTK.qstklearn.fastknn.FastKNN \textit{(class)}!QSTK.qstklearn.fastknn.FastKNN.query \textit{(method)}}

    \vspace{0.5ex}

\hspace{.8\funcindent}\begin{boxedminipage}{\funcwidth}

    \raggedright \textbf{query}(\textit{self}, \textit{point}, \textit{k}={\tt None}, \textit{method}={\tt \texttt{'}\texttt{mode}\texttt{'}}, \textit{slow}={\tt False}, \textit{dumdumcheck}={\tt False})

    \vspace{-1.5ex}

    \rule{\textwidth}{0.5\fboxrule}
\setlength{\parskip}{2ex}
    Returns class value for an unlabled point by examining its k nearest
    neighbors. 'method' determines how the class of the unlabled point is
    determined.

\setlength{\parskip}{1ex}
    \end{boxedminipage}

    \index{QSTK \textit{(package)}!QSTK.qstklearn \textit{(package)}!QSTK.qstklearn.fastknn \textit{(module)}!QSTK.qstklearn.fastknn.FastKNN \textit{(class)}|)}
    \index{QSTK \textit{(package)}!QSTK.qstklearn \textit{(package)}!QSTK.qstklearn.fastknn \textit{(module)}|)}


% chapter included in vclmanual.tex
\documentclass[vcl_manual.tex]{subfiles}
\begin{document}

\section{Floating point behavior details}
\label{FloatingPointBehavior}

The Vector Class Library is generally conforming to the new IEEE 754-2019 Standard for Floating-Point Arithmetic, but some compromises have been necessary for the purpose of vector processing and for better performance. The deviations from the standard are discussed below.
\vspacesmall

\begin{description}

\item[Subnormal numbers.]
Subnormal numbers (also called denormal numbers) are numerically extremely small floating point numbers where the exponent is below the normal range. Some microprocessors are handling subnormal numbers in a very inefficient way that is more than a hundred times slower than for normal floating point numbers. You may call the function \codei{no\_subnormals()} to prevent this and treat subnormal numbers as zero. Some of the mathematical functions in VCL always treat subnormal numbers as zero for reasons of performance. This includes logarithm, exponential, and power functions.

\item[Signed zero.]
Signed zero is a controversial issue. The floating point standard defines two different zeroes: +0.0 and -0.0.
The two zeroes are equal, but still distinguishable. Some of the functions may return +0.0 where the standard requires -0.0.\\
You may {} \codei{\#define SIGNED\_ZERO} {} if you want the sign of zero to conform to the
IEEE 754-2019 standard, though this may slow down performance a little.
\codei{SIGNED\_ZERO} may affect several functions, including
\codei{round}, \codei{truncate}, \codei{floor}, \codei{ceil},
\codei{maximum}, \codei{minimum}, \codei{cbrt}, \codei{pow\_ratio}, \codei{expm1}, \codei{log1p}. \\
This feature is not supported in version 1.xx of VCL.

\item[No exception trapping.] \label{NoExceptionTrapping}
Floating point errors are traditionally detected by trapping errors or relying on an \codei{errno} variable. These methods are not well suited for vector processing and out-of-order processing. This is explained in the document \href{https://www.agner.org/optimize/nan_propagation.pdf}{"NAN propagation versus fault trapping in floating point code", Agner Fog, 2019}.
\vspacesmall

The Vector Class Library does not support fault trapping, and it does not indicate exceptions in a variable such as the traditional \codei{errno}. It is not recommended to turn on floating point exceptions because this can cause inconsistent behavior, such as traps for exceptions in not-taken branches. Do not attempt to trap numerical errors in \codei{try/catch} blocks.
\vspacesmall

Instead, the vector class library indicates floating point exceptions by producing INF or NAN codes in the individual vector element that produced the fault.
The INF and NAN codes will propagate to the end result of a series of calculations when certain conditions are satisfied. The most efficient way of detecting floating point errors is to look for INF and NAN codes in the result.
\vspacesmall

Conditions where INF and NAN codes are not propagated are discussed at page  \pageref{FloatingPointErrors}
\vspacesmall

Do not use the compiler options -ffast-math, -ffinite-math-only, or /fp:fast because this may disable the detection of INF and NAN.
\vspacesmall

\item[No signaling NANs.]
Signaling NANs are special codes that will raise an exception when they are loaded from memory. Signaling NANs are rarely used in modern software. Signaling NANs should not be used in VCL because exception trapping is not supported.

\item[NAN payload operations.]
A NAN may contain additional information called a payload. This payload can propagate through a series of calculations to the end result. Some of the mathematical functions in VCL can put a payload into the NAN result in case of an error. This makes it possible to identify which function generated the NAN.
\vspacesmall

The \codei{nan..} and \codei{nan\_code} functions make it possible to set and get NAN payloads. The IEEE 754 standard does not specify what happens to the payload when converting between single and double precision, but experiments show that all microprocessors that use the binary floating point format will left-justify the payload. The \codei{nan..} and \codei{nan\_code} functions treat the NAN payload as a 22-bit left-justified unsigned integer in order to allow conversions between single and double precision. These functions deviate from the IEEE 754-2019 standard.

\item[NAN propagation in maximum and minimum functions.]
The \codei{max} and \codei{min} functions do not propagate NANs according to the 2008 version of the standard. This unfortunate situation is redressed in the  2019 revision of the standard. VCL offers two different versions of these functions:
The \codei{max} and \codei{min} functions are equivalent to
\codei{a > b ? a : b} and
\codei{a < b ? a : b}, respectively. These functions return \codei{b} if \codei{a} is NAN. The slightly less efficient functions \codei{maximum} and \codei{minimum} are sure to propagate NANs, in accordance with the 2019 revision of the standard.

\item[NAN propagation in pow function.]
The standard specifies that pow(NAN,0) and pow(1,NAN) will give the result 1.0. The VCL implementation deviates from this and produces a NAN output in all cases where an input is NAN, in order to support reliable NAN propagation.

\item[Function parameter range.]
Some of the mathematical functions have internal overflow for extreme values of the input parameters. These functions have a limited input range because an extra branch to handle the extreme cases would reduce the overall performance. Limitations of the input range are mentioned in the listing of the individual functions.

\end{description}

\vspacesmall

\end{document}


The horseshoe effect is a phenomenon that has long intrigued ecologists.  Commonly thought to be an artifact of dimensionality reduction, multiple techniques were developed to unravel this phenomenon and simplify interpretation.  Here, we provide evidence that horseshoes arise as a consequence of distance metrics that saturate - a familiar concept in other fields but new to microbial ecology.   This saturation property loses information about community dissimilarity, simply because it cannot discriminate between samples that do not share any common features. The phenomenon illuminates niche differentiation in microbial communities and indicates species turnover along environmental gradients.  Here we propose a rationale to the observed horseshoe effect from multiple dimensionality reduction techniques applied to simulations, soil samples, and samples from postmortem mice.   An intuitive-depth understanding of this phenomenon allows for the targeting of niche differentiation patterns from high-level ordination plots.
\section{Introduction}
Ecological datasets, particularly those observed in microbiome studies, are typically sparse and high-dimensional, frustrating most conventional statistical techniques.  Many numerical ecology software packages make use of distance-based statistics by calculating the distance between ecological communities, to compare various ecosystems to each other over space and time.  One of the most common exploratory analysis techniques is ordination, where the distances between the communities are embedded into a Euclidean space, and then visualized via Principal Components Analysis (\gls{pca}) \cite{numerical_ecology}.  A widely used extension of this technique, where the distance metric can be varied, is called Principal Coordinates Analysis (\gls{pcoa}) \cite{numerical_ecology}.  \par
One phenomenon that commonly occurs in datasets containing ecological gradients is the horseshoe effect or Guttman effect \cite{global_patterns}.  This phenomenon is typified by a linear gradient that appears as a curve in ordination space. The horseshoe effect, or its relative the arch effect \cite{detrended_correspondence_analysis} (where the ends of the gradient do not attract each other along the first principal coordinate as they do in the horseshoe effect), is observed using multiple types of ordinations, including Principal Components Analysis, Principal Coordinates Analysis, Non-Metric Multidimensional Scaling, Correspondence Analysis, and many others \cite{numerical_ecology}.  In 1982, the prevailing view of the horseshoe effect arose, when it was described by Gauch as a mathematical artifact that obscures the underlying ecological gradient.  Soon thereafter, Detrending Correspondence Analysis \cite{detrended_correspondence_analysis} was invented to unbend the horseshoe using reciprocal averaging. Since then, detrending has become a commonly applied practice to ordinations in ecological datasets.   Although these detrending techniques appear to provide a more intuitive visualization, they have been criticized as providing a distorted perspective of the underlying data, relying on many parameter settings that cannot be chosen in a principled way, and obscuring true underlying patterns in the data \cite{trust_dca}.\par
\begin{figure}[H]
        \centering
        \includegraphics[width=1.1\textwidth]{ch2/Figure1.pdf}
        \caption[An explanation of the horseshoe effect arising from distance saturation.]
        {An explanation of the horseshoe effect arising from distance saturation. (a) A band table where the y axis encodes for individual \gls{otu}s and the x axis encodes for samples.  Blocks that are colored black have a value of 1/10 while blocks that are colored white have a value of 0.  (b) The first 2 components from a \gls{pca} of the band table, yielding the typical horseshoe shape.  (c) The Euclidean distance from the point 0 to all of the other points.  (d) An illustration of distance saturation property.\index{SanDiego6}}
        \label{figb1}
\end{figure}
From previous studies, it was shown that horseshoes can arise from band tables \cite{horseshoe_kernel, guttman_effect}.  These tables consist of highly dense, non-zero values along the diagonal of the table, and sparse values everywhere else.  This pattern can be apparent when the rows and columns are sorted in the proper order. Although the idea that band tables lead to horseshoes is not a new idea, it is commonly misunderstood how this concept applies to microbial analyses.  Here we provide some intuition behind the mathematical structure of horseshoes.\par
In Figure \ref{figb1}a, we show a simulated band table, where each vertical band is represented by a sample, and contains 10 non-zero values.  In typical microbiome datasets, these values could reflect \gls{otu} or species counts; for simplicity, here we will to refer to them as species counts, although this concept can also be generalized to multiple data types, such as gene counts, metabolite abundances.  Each sample in the table is shifted by 1 row, creating the band effect.  When \gls{pca} is applied directly to this table, the first 2 eigenvectors yield a horseshoe pattern (Figure \ref{figb1}b).  Here, the band table is parameterized with a band size of 10, since each sample has exactly 10 non-zero values.  \par
For close local points, the Euclidean distance grows linearly along the gradient (Figure \ref{figb1}c).  However, after a certain point, the distance completely saturates. This property has been previously noted with Euclidean distance \cite{detrended_correspondence_analysis}.  The overlap between the first sample in the band table, and sample 10 and beyond disappears, and the distance between these samples is maximized. This can yield unintuitive properties, sample 10 could be less dissimilar than sample 1 compared to sample 20.  For instance, sample 10 could represent a medium pH environment, sample 1 could represent an acidic low pH environment and sample 20 could represent a high pHbasic environment. Sample 1 is expected to be a substantially more different microbial community to Sample 20 than Sample 10.  The acidophiles found in Sample 1 are typically not found in basic environments. Sample 20 is expected to be more different to sample 1 than sample 10, since it contains very different microbes that thrive in high pH environments.  But as far as Euclidean distance is concerned, sample 10 is just as dissimilar to sample 1 as sample 20, just because there are no common bacteria shared between these samples.  It is apparent that the saturation property of Euclidean distance does not capture all of the information about community dissimilarity along a gradient, simply because it cannot discriminate between samples that do not share any common features.  Once the distance is saturated, all samples that do not overlap lie within a ball of radius B where B is the band size lie within a ball of radius where B is the band size and the first point is the center of the ball as shown in Figure \ref{figb1}d.\par
This saturation property has been suggested to give rise to horseshoes in previous studies in other fields \cite{horseshoe_kernel}, and is an unintuitive property that can confound ecological interpretations if not understood properly.  This property also restricts the possible trajectories of samples in the feature space, and gradients cannot be represented by linear trajectories in the real space (Supplemental proof 1). This means that communities in the original high dimensional space do not arrange into linear trajectories in the first place, and when projected to lower dimensions do not fall into linear trajectories.  These trajectories are what we refer to as horseshoes.  The horseshoe phenomenon is analogous to the familiar concept of saturation in molecular evolution, where two randomly evolving sequences saturate at 75\% DNA sequence identity (assuming equal nucleotide frequencies), even if infinite time has elapsed \cite{dna_saturation}. Consequently, distances that reflect a higher degree of molecular change need to be corrected for multiple substitutions in order to recover the molecular clock-like behavior obtained when comparing more similar sequences. This is why corrections according to models such as Jukes-Cantor or the Kimura 2-parameter model are required to obtain distances for reconstructing better phylogenetic trees. Analogous distance corrections are needed in microbial ecology for reconstructing better relationships among microbial communities \cite{evolutionary_distances}.\par
It is important to note that horseshoes do not only arise from \gls{pca}, but also arise in \gls{pcoa} with a variety of distance metrics.  Arch effects have plagued every multidimensional reduction technique we have applied to a wide range of microbial ecology datasets \cite{microbial_patterns}. In the following case studies, we'll show that these distance metrics also have the saturation property.  In addition, if a distance doesn't have this saturation property, there won't be an observed horseshoe artifact (Figure S1).\par
\section*{Case Study 1 - 88 Soils}
In this study, 88 soil samples were obtained from multiple locations across the United States having varying levels of pH \cite{soil_pyro}.  The V4 region of the 16S rRNA gene (16S) within each organism was amplified and sequenced using 454 pyrosequencing to obtain relative abundances of microbial taxa.  A matrix representing abundance values for each taxonomic unit per soil sample was used as input in correspondence analysis (\gls{ca}) \cite{correspondence_analysis}. The resulting ordination showed clear separation of the communities based on pH (Figure \ref{figb2}a), which led to the same conclusion that pH is a major driving factor in soil biogeography, i.e. pH has major impacts on the distribution of bacterial taxonomic units in soil \cite{soil_pyro}.  The \gls{ca} analysis in Figure \ref{figb2}a also shows the classic horseshoe shape.   Here we revisited this study, to better understand the horseshoe shape behind this dataset.\par
To test the effect of another commonly used distance metric on the sample distribution, we analyzed the same soil dataset applying Chi Squared distance (Figure \ref{figb2}b). Similar to what was observed with Euclidean distance, which was applied in the simulation, the Chi Squared distance increased sharply at pH 3 and 4, but began to saturate at pH of 5.  Also the band table similar to what we have observed in Figure \ref{figb2}a can be obtained when sorting.  Also, when the the \gls{otu} table was sorted by sample pH and the mean pH of the samples that the \gls{otu}s were observed in mean pH of the \gls{otu}s (Equation 12), the same band table pattern appeared as we show in Figure \ref{figb2}a.  While the diagonal isn't completely dense, there are more non-zero values compared to the corners of the heatmap. In line with the findings from the original study, this pattern is likely representative of niche differentiation of \gls{otu}s with respect to pH. The organisms that thrive in low pH environments tend not to exist in high pH environments and vice versa.  Low pH and high pH samples are shown in Figure \ref{figb2}c to have few overlapping species, a pattern not observed in the original study as membership was evaluated at coarser levels of taxonomic resolution\cite{soil_pyro}.\par
\begin{figure}[H]
        \centering
        \includegraphics[width=1\textwidth]{ch2/Figure2.pdf}
        \caption[Two case studies show casing how horseshoes can appear in the context of
          soil microbial communities and post-mortem microbial communities.]
        {Two case studies show casing how horseshoes can appear in the context of soil microbial communities and post-mortem microbial communities. (a) Correspondence analysis of 88 soils.  (b) Distance saturation of chi-squared metric, plotting the chi squared distance of the first sample versus all of the other samples. (c) Heatmap of log transformed \gls{otu} counts from the 88 soils with the samples sorted by pH and the \gls{otu}s sorted by mean pH.  (d) Principal Coordinates Analysis of unweighted UniFrac distance.  (e) UniFrac distance of a samples from the last time point versus all of the samples. (f) Heatmap of centred log ratio transformed (Equation 2) \gls{otu} counts sorted by harvest days.\index{SanDiego8}}
        \label{figb2}
\end{figure}
\section*{Case Study 2 - Post Mortem Mice Study}
In this study, 120 mice were sacrificed and allowed to decompose on soil. Mice were destructively sampled over approximately 8 weeks\cite{mammalian_corpse}. 16S sequencing libraries were generated from total DNA extracted from swabs of the skin on the head, and relative abundance values were calculated for each bacterial \gls{otu}. A relative abundance matrix was generated for each library and used as input in \gls{pca}. This analysis generated a clear horseshoe (Figure \ref{figb2}d) using unweighted UniFrac distance \cite{soil_pyro}, with a gradient with respect to the time since death, possibly reflecting a changing skin microbiome during decomposition of the mouse carcass. When the samples were sorted by time since death using a similar strategy as noted above, a band table emerges (Figure \ref{figb2}f). Also, the unweighted UniFrac distance analysis appears to have the same saturation property as observed previously with Euclidean distance and Chi-squared distance.  It is important to note that highest possible UniFrac distance is 1, suggesting that this distance metric can also be saturated.  In Figure \ref{figb2}e, while the distance hasn't completely saturated, these distances are quickly approaching the theoretical maximal UniFrac distance. \par
The striking changes in microbial communities during decomposition are associated with dramatic environmental biochemical changes, including increased pH, ammonia, and total nitrogen, all measured in soil beneath the mice. Correspondingly, microbial communities are predicted to increase in gene abundance of important nitrogen cycling pathways such as amino acid degradation (e.g. glutamate dehydrogenase, lysine decarboxylase, ornithine decarboxylase) and nitrate reduction (e.g. nitrate and nitrite reductase). Bacterial taxa in the families Chromatiaceae (\gls{otu} 46026, 4482362) and Rhizobiaceae (\gls{otu} 4301099) are involved in nitrogen metabolism and become abundant as mouse bodies progress through the stages of decomposition (e.g. Fresh, Active Decay, Advanced Decay).  As shown in Figure S1, all of these \gls{otu}s peak at specific timepoints. The two Chromatiaceae \gls{otu}s peak during Active Decay (bloating and purge of fluids) at 15 days of decomposition. The Rhizobiaceae \gls{otu} peaks during Advanced Decay (sinking and sagging flesh) at 30 days of decomposition and when pH, ammonia, and total nitrogen were measured at their highest levels \cite{mammalian_corpse}.\par
To further validate if saturation leads to horseshoes, a new distance metric \gls{embad} (Earth Mover Band Aware Distance) was engineered to be non-saturating as a proof of concept (Supplemental Methods).  This distance metric uses prior knowledge about the ordering of the band table, and is determined by calculating the flow between two samples. As shown in Figure S1a, sample 1 and sample 2 each have 4 species proportions.  To calculate the distance between sample 1 and sample 2, the probability mass of species 1 and species 2 needs to be shuffled over to species 3 and species 4.  This concept is analogous to computing maximum flow along a pipe, and can be calculated using Earth Mover's distance \cite{emd, unifrac_emd, unifrac}.\par
For the 88 soils (Figure S1b), the \gls{embad} was applied to the pH sorted table. Therefore, even if two samples are not overlapping, samples closer together will have a smaller distance than samples farther apart in the gradient.  This is because the distance is defined to be not saturating and explicitly accounts for the pH gradient.  The same strategy was employed for the postmortem interval mice (Figure S1c), sorting the table by decomposition days. The \gls{pcoa} plots resulting from these applications of \gls{embad} suggest that a non-saturating distance metric could remove the horseshoe effect from lower dimensional projections of these abundances.  This provides further evidence that this saturation property could explain the the horseshoe phenomenon.\par
For the 88 soils study a \gls{permanova} test investigating the difference between soils with a pH less than 3, and soils with a pH greater than 8.  With the \gls{embad} distance metric the \gls{permanova} gave a pseudo F-statistic of 650.5 and a p-value of 0.0003, which has a much larger effect size compared to the original Chi-squared distance metric with a pseudo F-statistic of 3.8 and a p-value of 0.0004 with 9999 permutations.  A similar trend was observed in the post mortem interval mice study when testing the first decomposition day to the last decomposition day using \gls{permanova}.  The \gls{embad} distance metric had a pseudo F-statistic of 439.8 and a p-value of 0.0001 with 9999 permutations, which has a larger effect size than the Unifrac distance metric, which had a pseudo F-statistic of 25.5 and a p-value of 0.0001. This method is relieved from misinterpretations of data due to horseshoes and arches and facilitates the interpretation of taxonomic units along biologically significant gradients that reflect the selective pressure of these factors on the distribution of microbes.  \par
In light of the benefits of engineering a non-saturating distance metric, the \gls{embad} distance metric requires the gradient to be known a priori.  Generalizing this approach in the absence of known gradients is a difficult problem would require an exhaustive using known algorithms.  Specifically, this problem falls under the category of NP-hard problems (Supplemental Proof 2).  In the 88 soils study and the post mortem mice study, we were fortunate to be able to infer the underlying band table with known metadata.  \par
The band patterns we observe here are probably very common in ecology studies investigating species distribution patterns across spatial or temporal gradients. The pattern confirms microbial ecological fundamentals, i.e. bacteria have acquired unique adaptations to the environment and occupy either a broad range or very specific niches. In our case studies of the 88 soils - and the postmortem mice we confirmed that by using a band table pattern analysis approach, bacterial species show different adaptations to pH and bacterial diversity changes over time during decomposition of mice carcasses. The band pattern approach we apply here represents an additional method to visualize differences between microbial communities.\par
Based on our observations here, the horseshoe effect appears in dimensionality reduction techniques due to the saturation property of distance metrics.  While we have only tested a few distance metrics, it is suspected that a vast majority of these distance metrics exhibit the same property, which would also explain why horseshoes are encountered so frequently across many different fields. The saturation property has also been observed in multiple other fields, and other studies from different disciplines have come to similar conclusions \cite{horseshoe_kernel}. In addition, multiple techniques such as self organizing maps \cite{self_organizing_maps} and local linear embedding \cite{local_linear_embedding} attempt to avoid this saturation phenomenon by focusing on distances of nearby points.\par
In spite of the saturation property of distance metrics, identifying horseshoes is still highly useful for identifying patterns concerning niche differentiation.   Properly understanding this phenomenon will motivate the use and development of algorithms such as biclustering \cite{biclustering} to uncover underlying band patterns.  These insights will in effect allow us to identify microbial niches along different environmental conditions.\par
\section{Materials and Methods}
All analyses can be found below on github \\
\url{https://github.com/knightlab-analyses/horseshoe-analyses}.
The mean gradient used for the 2 case studies was calculated as follows.
\begin{equation}
        \overline{g}_{x}=\sum_{i=1}^{N}g_{i}\frac{x_{i}}{\sum_{j=1}^{D}x_{j}}
\end{equation}
Where $x_{i}$ is the proportion of \gls{otu} \textit{x} in sample \textit{i} , $\overline{g}_{x}$ is the mean gradient of \gls{otu} \textit{x}, and $g_{i}$  is the sample gradient at sample i. This calculation can be found in the gneiss package under the function \textbf{mean\_niche\_estimator}. The function used to sort the tables in Figure \ref{figaS1}c used \textbf{niche\_sort}. In the 88 soils study, the table was sorted by sample pH and the mean pH of the samples that the organisms were observed in. In the post mortem mice study, the table was sorted by the days of decomposition and the mean day of the samples of that the organisms were observed in.\par
The heatmap in Figure \ref{figaS2}f and the abundances in Figure S2 were normalized using the centre log ratio \gls{clr} transformation given by the following equation.
\begin{equation}
        clr(x)=\left [ \log \frac{x_{1}}{g(x)},...,\log \frac{x_{D}}{g(x)} \right ]=\log x-\overline{\log x}
\end{equation}
Where $g(x)=\sqrt[n]{\prod_{i=0}^{n}x_{i}}$ is the geometric mean and $\overline{\log x}=\log g(x)=\frac{1}{n}\sum_{i=0}^{n}\log x$ is the average of the log transformed values.  A pseudocount of 1 is added to all of the counts to prevent logarithms of zero occurring. \par
Analyses were performed using Scipy, Numpy, Matplotlib, Seaborn, Scikit-bio and Gneiss.
\section{Acknowledgements}
We thank Noah Fierer for his input on the analysis of the 88 soil samples and Dan Knights for discussion on the \gls{embad} metric. We also acknowledge Amnon Amir and Tomasz Kosciolek for their insights into the horseshoe effect. Finally, we thank Susan Holmes for her insights on previous work understanding the horseshoe effect.

J.T.M. was funded by NSF GRFP DGE-1144086. J.L.M. and R.K. were funded by the Office of Justice Programs National Institute of Justice (award NIJ-2011-DN-BX-K533).

Chapter 2, in full, is a reprint of the material as it appears in
``Uncovering the Horseshoe Effect in Microbial Analyses''
James T. Morton, Liam Toran, Anna Edlund, Jessica L. Metcalf,
Christian Lauber, Rob Knight \emph{mSystems}, 2, 2017.  The dissertation author was the primary investigator and first author of this paper.


%%
%% Template abstract.tex
%%

\chapter*{Abstract}
\label{cha:abstract}
\addcontentsline{toc}{chapter}{Abstract}

Recent sky surveys such as the Sloan Digital Sky Survey and the VST ATLAS Survey have given us a large
amount of data to work with. Spectroscopic labelling, however, is quite expensive, so we only
want to manually label an object if doing so allows us to gain new information. This thesis
explores pool-based active learning and the novel application of prominent active learning heuristics to
the domain of photometric classification.

We begin by applying standard supervised machine learning techniques to two astronomical datasets.
The best-performing classifiers with reliable probability estimates, logistic regression and support
vector machines, are then used to conduct the active learning experiment. Our key original
contribution is the application of Thompson sampling, a Bayesian solution to the exploration vs
exploitation problem, to the selection of six active learning heuristics. To address the problem of
class imbalance, we derive an extension of the posterior balanced accuracy to the
multi-class setting. This is used to evaluate the performance of our algorithms.

The results are very promising. Even under simplistic assumptions like a normally distributed
reward, Thompson sampling manages to automatically identify the optimal heuristic after only 50
examples. In particular, the margin minimisation technique is a clear winner, outperforming random
sampling by as much as 9\% in the VST ATLAS dataset after 300 examples. Being very
computationally efficient, we recommend that astronomers use the margin heuristic with logistic
regression to decide which objects to label next in future sky surveys. To help them with this task,
we have created an open-source and extendable Python package that allows others to easily apply
active learning routines to any dataset and make quick visualisation of photometric data.

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "thesis"
%%% End:


%\documentclass{article}
%\usepackage{fancyvrb}
%\usepackage{perltex}
%\usepackage{xcolor}
%\usepackage{listings}
%\usepackage{longtable}
%\usepackage{multirow}
%\input{src/cmd}
%\begin{document}

\index{poscvt!application writeup}
\section{\emph{poscvt}}
\subsection{Overview}
This application allows the user to convert among different coordinate systems on
the command line. Coordinate systems handled include Cartesian, geocentric, and
geodetic.

\subsection{Usage}
\subsubsection{\emph{poscvt}}
\begin{\outputsize}
\begin{longtable}{lll}
%\multicolumn{3}{c}{\application{poscvt}} \\
\multicolumn{3}{l}{\textbf{Optional Arguments}} \\
\entry{Short Arg.}{Long Arg.}{Description}{1}
\entry{-d}{--debug}{Increase debug level.}{1}
\entry{-v}{--verbose}{Increase verbosity.}{1}
\entry{-h}{--help}{Print help usage.}{1}
\entry{}{--ecef=POSITION}{ECEF ``X Y Z'' in meters.}{1}
\entry{}{--geodetic=POSITION}{Geodetic ``lat lon alt'' in deg, deg, meters.}{1}
\entry{}{--geocentric=POSITION}{Geocentric ``lat lon radius'' in deg, deg, meters.}{1}
\entry{}{--spherical=POSITION}{Spherical ``theta, pi, radius'' in deg, deg, meters.}{1}
\entry{-l}{--list-formats}{List the available format codes for use by the input and output format options.}{2}
\entry{-F}{--output-format=ARG}{Write the position with the given format.}{1}
\end{longtable}
\end{\outputsize}

\subsection{Examples}
\begin{\outputsize}
\begin{verbatim}
> poscvt --ecef="4345070.59253 45619878.26297 803.598856837"

    ECEF (x,y,z) in meters              4345070.5925 45619878.2630 803.5989
    Geodetic (llh) in deg, deg, m       0.00100566 84.55926933 39448197.4795
    Geocentric (llr) in deg, deg, m     0.00100472 84.55926933 45826334.4795
    Spherical (tpr) in deg, deg, m      89.99899528 84.55926933 45826334.4795

\end{verbatim}
\end{\outputsize}
\subsection{Notes}
If no options are given \application{poscvt} assumes XYZ 0 0 0.

%\end{document}


\documentclass{article}
\usepackage{amsmath,mathtools,amssymb}
\usepackage{graphicx}
\usepackage{booktabs}
\usepackage{blkarray}
\usepackage{gensymb}
\usepackage{verbatim}
\usepackage{mathrsfs}
\usepackage{bbm}
\usepackage{braket}
\usepackage{hyperref}
\usepackage{verbatim}
\usepackage{cancel}
\usepackage[margin=1.0in]{geometry}
\newcommand{\ol}{\overline}
\newcommand{\lp}{\left(}
\newcommand{\rp}{\right)}
\newcommand{\eps}{\varepsilon}
\newcommand{\lam}{\lambda}
\newcommand{\h}{\circ}
\newcommand{\p}{\bullet}

\newcommand{\Ezero}{E^{(0)}}
\newcommand{\Phizero}{\Phi^{(0)}}
\newcommand{\Eone}{E^{(1)}}
\newcommand{\Phione}{\Phi^{(1)}}

\newcommand{\Ecorr}{E_{\mathrm{corr}}}
\newcommand{\Hc}{H_{\mathrm{c}}}
\newcommand{\dg}{\ensuremath{^\dagger} }
\def\*#1{\mathbf{#1}}
\DeclarePairedDelimiter\floor{\lfloor}{\rfloor}

\title{Lecture 5: Perturbation Theory I}
\date{April 1, 2020}
\begin{document}
\maketitle
\noindent
To remind you of the current task at hand, we are trying to find good quality approximations
    to electronic wavefunctions in order to obtain the electronic energy of a molecular system.
That is, we seek to solve the Schr{\"o}dinger equation in the best way possible.
\[H \ket{\Psi} = E \ket{\Psi} \]
We already learned that HF theory is a decent and cheap approximation, and full-CI is exact but very expensive.
Perturbation theory is one of many approaches to finding wavefunctions and energies which are of
    reasonable cost and reasonable accuracy.
Recall, however, that the HF wavefunction $\Phi$ gives us a very good starting point, so we are really just interested in solving for the correlation energy,
which is obtained by shifting our Hamiltonian by the HF energy.
\[\Hc \ket{\Psi} = \Ecorr \ket{\Psi} \]
where
\[\Hc = H - \braket{\Phi| H | \Phi} \]
\[\Ecorr = E_{\mathrm{exact}} - E_{\mathrm{HF}}\]

We will start by discussing Rayleigh-Schr{\"o}dinger perturbation theory (RSPT), which applies the ideas of PT to the time-independent
    Schr{\"o}dinger equation.
Next, we will apply the results of RSPT to derive M{\o}ller-Plesset methods (MP$n$) for approximating the electronic correlation energy of molecular systems.

\section{Rayleigh-Schr{\"o}dinger Perturbation Theory}
Perturbation theory (PT), in the most general sense, is applicable in nearly every domain of physics.
If a problem can be separated into some easily solvable piece A and not-so-easily solvable piece B
    (``the perturbation''), one can approach the exact solution of the problem in terms of the solution to A
    plus a series of corrections to account for the influence of perturbation.
Rayleigh-Schr{\"o}dinger Perturbation Theory (RSPT) applies the ideas of PT to the time-independent
    Schr{\"o}dinger equation.
The Hamiltonian is decomposed into a sum of a simple (easy to solve) problem $H_0$ and a perturbation $V$:
\[H = H_0 + V \]
If the perturbation is ``small'' (i.e., does not wildly shift the solutions of the problem) the
    energy levels/eigenstates associated with the \textit{perturbed system} described by $H$ can be expressed
    as sum of the energy levels/eigenstates of  the \textit{simple system} (described by $H_0$) and a series
    of corrections obtained by considering the influence of the perturbation $V$ on the
    energy levels/eigenstates of the simple system.
Each of these corrections contain eigenvalues of $H_0$ and matrix elements of the perturbation
    between the eigenfunctions of $H_0$, as we will see.
In summary, RSPT is applying age-old ideas of general perturbation theory to the
    time independent Schr{\"o}dinger equation.

The eigenvalue problem we wish to solve is the following:
\[H \ket{\Psi_i} = (H_0 + V) \ket{\Psi_i} = E_i \ket{\Psi_i}\]
We are assuming $H_0$ is a simple, easy-to-solve system, for which we have access to all the eigenstates
    and eigenvalues.
\[H_0 \ket{\Phi_i^{(0)}} = E_i^{(0)} \ket{\Phi_i^{(0)}} \]
These eigenstates $\ket{\Phi_i^{(0)}}$ and eigenvalues $E_i$ for the simple system are hopefully very close
    to the exact results  $\ket{\Psi_i}$ and  $E_i$, that is, the perturbation $V$ is small.
Now, we do something a bit weird.
We introduce a parameter $\lam$ in front of $V$, which allows us to toggle the influence of the
    perturbation on the energy levels and eigenstates.
Before saying much else about $\lam$, let's write it down and see what it allows us to do.
\[(H_0 +  \lam V) \ket{\Psi_i} = E_i \ket{\Psi_i}\]
When $\lam = 0$, our Hamiltonian is just $H_0$, and our eigenstates and energy levels become that
    of the simple system $\ket{\Phi_i^{(0)}}$ and $ E_i^{(0)}$.
When $\lam = 1$, our Hamiltonian is the full Hamiltonian $H$, and our eigenstates and energy levels
    are the perturbed system $\ket{\Psi_i^{(0)}}$ and $E_i$.
By doing this, \textit{the energy levels and eigenstates of the perturbed system are a function
    of }$\lam$.
To put it another way, our energy levels and eigenstates depend on lambda;
    we might express this mathematically as  $E_i(\lam)$ and $\Psi_i(\lam)$.
Since these quantities are \textit{functions of} $\lam$, we can write down a Taylor series
expansion of the energy levels and eigenstates of $H$ in terms of $\lam$:
\[E_i =  E_i^{(0)} +  \lam E_i^{(1)} + \lam^2 E_i^{(2)}  + \cdots \]
\[\ket{\Psi_i} =  \Phi_i^{(0)} +  \lam \Phi_i^{(1)} + \lam^2 \Phi_i^{(2)} + \cdots \]
where each term is,
\[ E_i^{(k)} = \left. \frac{1}{k!} \frac{d^k E_i}{d \lam^k} \right\rvert_{\lam = 0} \]
\[ \Phi_i^{(k)} = \left. \frac{1}{k!} \frac{d^k \ket{\Psi_i}}{d \lam^k} \right\rvert_{\lam = 0} \]

We call $E_i^{(0)}$ the 0$^{th}$ order energy, which is the energy of the simple system.
Every additional term in the energy expansion is the $n^{th}$ order energy correction,
    and these are what we must solve for.
Likewise terminology is used for the eigenstate corrections $\ket{\Phi_i^{(k)}}$

Many textbooks and other resources simply say $\lam$ is a
    ``fictitious dummy parameter'' or ``book-keeping device'' that is
    used to keep track of the  ``orders'' of the terms when working with the perturbation expansion.
This is a bit of an oversimplification. The parameter $\lam$ has the utility of turning
the perturbation on/off, and motivates the use of a Taylor expansion in the first place.
Without $\lam$, you have no Taylor expansion, and no perturbation theory.
While it does serve as a useful label for the orders of each term, it's not just a ``dummy'' parameter.
It's the key to all of perturbation theory.

We have a form for the wavefunction and the energy. Plugging these
into the time-independent Schr{\"o}dinger equation, we get
\[
(H_0 + \lam V) (\Phi_i^{(0)} +  \lam \Phi_i^{(1)} + \lam^2 \Phi_i^{(2)} + \cdots)
=  (E_i^{(0)} +  \lam E_i^{(1)} + \lam^2 E_i^{(2)}  + \cdots ) (\Phi_i^{(0)} +  \lam \Phi_i^{(1)} + \lam^2 \Phi_i^{(2)} + \cdots)
 \]

Subtracting from each side so that the left is 0 and grouping like powers of $\lam$ gives
\begin{align*}
&  \left(H_0 \ket{\Phi_i^{(0)}} - E_i^{(0)} \ket{\Phi_i^{(0)}}   \right) \\
&+ \lam \left( H_0\ket{\Phi_i^{(1)}} + V\ket{\Phi_i^{(0)}} - E_i^{(0)}\ket{\Phi_i^{(1)}} - E_i^{(1)}\ket{\Phi_i^{(0)}} \right)  \\
&+ \lam^2 \left( H_0 \ket{\Phi_i^{(2)}} + V \ket{\Phi_i^{(1)}} - E_i^{(0)} \ket{\Phi_i^{(2)}} - E_i^{(1)} \ket{\Phi_i^{(1)}} - E_i^{(2)} \ket{\Phi_i^{(0)}}  \right)  \\
&+ \cdots \\
&= 0
\end{align*}

The only way the above equation can be satisfied for any arbitrary value of $\lam$ is
    if the coefficient in parentheses for each power of $\lam$ is equal to 0.
Therefore, since each term for each power of $\lam$ is equal to 0,
    this equation is fully separable into an infinite series of simultaneous equations
\begin{align*}
&  H_0 \ket{\Phi_i^{(0)}} = E_i^{(0)} \ket{\Phi_i^{(0)}}   \\
&  H_0\ket{\Phi_i^{(1)}} + V\ket{\Phi_i^{(0)}} = E_i^{(0)}\ket{\Phi_i^{(1)}} + E_i^{(1)}\ket{\Phi_i^{(0)}}  \\
&  H_0 \ket{\Phi_i^{(2)}} + V \ket{\Phi_i^{(1)}} = E_i^{(0)} \ket{\Phi_i^{(2)}} + E_i^{(1)} \ket{\Phi_i^{(1)}} + E_i^{(2)} \ket{\Phi_i^{(0)}}  \\
& \cdots \\
\end{align*}
Notice we have now dropped the $\lam$'s since each of the above equations hold whether or not $\lam^n$ is applied to each term.
You could also interpret this as us setting $\lam=1$, and fully ``turning on'' the perturbation.

We will assume henceforth that our reference state is normalized  $\braket{\Phizero_i | \Phizero_i} = 1$
and our exact wavefunction is intermediately normalized $\braket{\Phizero_i | \Psi_i} = 1$.
This yields,
\[\braket{\Phizero_i | \Psi_i} = 1 = \braket{\Phizero_i | \Phizero_i} + \lam \braket{\Phizero_i | \Phione_i} + \lam^2 \braket{\Phizero_i|\Phi_i^{(2)}} + \cdots \]
In order for the above expression to be true for arbitrary $\lam$,
it must be the case that $\braket{\Phizero_i | \Phi_i^{(k)}} = 0 \space \mathrm{for} \space k = 1,2,3,...$

We can now project each equation in our system of equations on the left by $\bra{\Phi_i^{(0)}}$.
We can simplify quite a bit by noting $\braket{\Phizero_i | \Phi_i^{(k)}} = 0 \space \mathrm{for} \space k = 1,2,3,...$
and that since $H_0$ is Hermitian, we can act it act on the left to give
$ \braket{\Phizero_i | H_0 | \Phi_n^{(k)}   } = \Ezero_i \braket{\Phizero_i | \Phi_i^{(k)}}  $.

The final result is:
\begin{align*}
E_i^{(0)} = \braket{\Phi_i^{(0)} | H_0 | \Phi_i^{(0)}} \\
E_i^{(1)} = \braket{\Phi_i^{(0)} | V | \Phi_i^{(0)}} \\
E_i^{(2)} = \braket{\Phi_i^{(0)} | V | \Phi_i^{(1)}} \\
\end{align*}

Notice that $E_i^{(1)}$ depends on $\Phi_i^{(0)}$, and $E_i^{(2)}$ depends on $\Phi_i^{(1)}$.
This trend continues on for higher order energy corrections:
    the $n^{th}$ order correction to the wavefunction is required for finding the $(n+1)^{th}$ order energy correction.

% Note, algebra in Szabo/Ostlund was archaic, instead used this https://www.lct.jussieu.fr/pagesperso/toulouse/enseignement/introduction_pt_cc.pdf
% combined with some other sources
\subsection{The first-order wavefunction and second-order energy corrections}
The third equation above, the second-order energy ($E_i^{(2)}$) correction, requires you to know the 1st order correction to the wavefunction $\Phione_i$.
To obtain it, we take the equation containing our 1st-order terms and rearrange:

\[ H_0\ket{\Phione_i} + V \ket{\Phizero_i} = \Ezero_i \ket{\Phione_i} + \Eone_i \ket{\Phizero_i} \]
Projecting by $\bra{\Phizero_n}$ for some $n \neq i$ gives

\[ \braket{\Phizero_n | H_0 | \Phione_i} + \braket{ \Phizero_n | V | \Phizero_i} = \Ezero_i \braket{\Phizero_n|\Phione_i} + \Eone_i \braket{\Phizero_n | \Phizero_i} \]

To simplify, note that $\bra{\Phizero_n} H_0 = \Ezero_n \bra{\Phizero_n}$, so the first term on the left becomes $\Ezero_n \braket{\Phizero_n | \Phione_i} $
The second term on the right goes to 0 since $\braket{\Phizero_n|\Phizero_i} = 0 $ when $n \neq i$.
The reason is these are both eigenfunctions of $H_0$, which are orthonormal.
We are left with
\[ \Ezero_n \braket{ \Phizero_n | \Phione_i} + \braket{ \Phizero_n | V | \Phizero_i} = \Ezero_i \braket{ \Phizero_n | \Phione_i }    \]
Rearranging we get
\[ \braket{\Phizero_n | \Phione_i } = -\frac{ \braket{ \Phizero_n | V | \Phizero_i} }{ \Ezero_n - \Ezero_i} \]

We want to get $\ket{\Phione_i}$ by itself on the left side.
The left side above is the expansion coefficients of $ \Phione_i$ in the basis of $\Phizero_n$.
To see this, project each side by $ \ket{\Phizero_n} $ and sum over all $n \neq i$
\[ \sum_{n \neq i} \ket{ \Phizero_n} \braket{\Phizero_n | \Phione_i } = \sum_{n \neq i} -\frac{ \braket{ \Phizero_n | V | \Phizero_i} }{ \Ezero_n - \Ezero_i} \ket{\Phizero_n} \]

The left side above would be equal to $\ket{\Phione_i}$ if the sum was over all $n$ (resolution of the identity),
    but if we sum over all $n$, the energy denominator on the right will be 0 for $n = i$.
Fortunately, the expansion coefficient $\braket{\Phizero_n | \Phione_i } $ for $n = i$ is necessarily 0.
Why? Because of intermediate normalization.
\[\braket{\Phizero_i | \Psi_i} = 1 = \braket{\Phizero_i | \Phizero_i} + \lam \braket{\Phizero_i | \Phione_i} + \lam^2 \braket{\Phizero_i|\Phi_i^{(2)}} + \cdots \]

Since our reference wavefunction is normalized $\braket{\Phizero_i | \Phizero_i} = 1$, in order for this expression to be true for arbitrary $\lam$,
it must be the case that $\braket{\Phizero_i | \Phi_i^{(k)} = 0 \mathrm{for k} = 1,2,3,...}   $
This implies that when $n = i$, the expansion coefficient is 0.
Therefore, we can sum over all $n$ (rather than $n \neq i$) on the left side of our expression for $\ket{\Phione_i}$ without penalty; it is the same as adding zero to both sides:
\[ \sum_{n} \ket{ \Phizero_n} \braket{\Phizero_n | \Phione_i } = \sum_{n \neq i} -\frac{ \braket{ \Phizero_n | V | \Phizero_i} }{ \Ezero_n - \Ezero_i} \ket{\Phizero_n} \]
Noting the left side contains the resolution of the identity, we can just remove it:
\[ \ket{\Phione_i} = \sum_{n \neq i} -\frac{ \braket{ \Phizero_n | V | \Phizero_i} }{ \Ezero_n - \Ezero_i} \ket{\Phizero_n} \]

This now enables us to derive an evaluable expression for the second-order energy
\[ E_i^{(2)} = \braket{\Phi_i^{(0)} | V | \Phi_i^{(1)}} \]
Plugging in the definition for $\Phi_i^{(1)}$ above we obtain
\[ E_i^{(2)} = \sum_{n \neq i} -\frac{ \braket{ \Phizero_n | V | \Phizero_i} \braket{ \Phizero_i | V | \Phizero_n} }{ \Ezero_n - \Ezero_i} \ket{\Phizero_n} \]
\[ E_i^{(2)} = \sum_{n \neq i} \frac{ |\braket{ \Phizero_i | V | \Phizero_n} |^2  }{ \Ezero_i - \Ezero_n} \]

One could keep working through the tedious algebra and derive the expressions for the second order wavefunction, third order wavefunction, etc, and
in turn obtain energy corrections $E_i^{(3)}$, $E_i^{(4)}$, etc, but we will stop here.
The above results are sufficient for deriving second-order M{\o}ller--Plesset perturbation theory.

There does exist a more advanced scheme which generalizes the results of all orders of perturbation theory using a mysterious new thing called ``resolvents''.
We will touch on this at the end of these notes and continue down that path after learning diagrams.

RSPT, as described above, is a completely general scheme, applicable to any application of the time-independent Schrodinger equation.
We will use these ideas to come up with a way to obtain electronic energies with RSPT.

\section{M{\o}ller--Plesset Perturbation Theory}
M{\o}ller--Plesset Perturbation Theory (MP$n$) is the direct application of time-independent RSPT to the problem of finding electronic wavefunctions and energies.
In this section, we will derive MP2 by applying the RSPT equations derived above for a specific partition of our electronic Hamiltonian into $H_0$ and $V$.
The MP2 energy which is correct up to second-order.

We first must divide our Hamiltonian into two parts, one which has a simple solution $H_0$ and the perturbation $V$.
\[H = H_0 + V \]
In MP$n$ methods, the $H_0$ part of the Hamiltonian is taken to be the Hartree-Fock Hamiltonian,
\[H_0 = F = (h_p^q + \bar{g}_{pi}^{qi}) a_q^p = f_p^q a_q^p  \]
Note here we are \textbf{not} using the $\Phi$-normal ordered Fock operator $f_p^q \tilde{a}_q^p$.
The perturbation $V$ will just be everything not included in $H_0$.
\[V = H - F \]
\[V =  \frac{1}{4} \bar{g}_{pq}^{rs} a_{rs}^{pq} - \bar{g}_{pi}^{qi} a_{q}^{p}  \]
One could interpret $V$, the perturbation, as being all of the two-electron interactions, except for the mean-field picture already given by the HF Hamiltonian.

Suppose we seek the energy of the ground state of a system (which we will denote by a subscript 0).
The ground state 0th order wavefunction $\Phi_0^{(0)}$ is the Hartree-Fock Slater determinant wavefunction.
The ground state 0th order energy is given by $E_0^{(0)} = \braket{\Phi_0^{(0)} | H_0 | \Phi_0^{(0)}}$
\begin{align*}
E_0^{(0)} &= \braket{\Phi_0^{(0)} | H_0 | \Phi_0^{(0)}} \\
          &= \sum_i h_{ii} + \sum_{ij} \braket{ij||ij}
\end{align*}
We note the above expression, from Hartree-Fock theory, is equal to the sum of the occupied orbital energies $\sum_i \eps_i$
\[E_0^{(0)} = \sum_i h_{ii} + \sum_{ij} \braket{ij||ij} = \sum_i \eps_i  \]

The 1st order energy correction is given by $E_0^{(1)} = \braket{\Phi_0^{(0)} | V | \Phi_0^{(0)}}$
\begin{align*}
E_0^{(1)} &= \braket{\Phi_0^{(0)} | V | \Phi_0^{(0)}} \\
          &= -\frac{1}{2} \sum_{ij} \braket{ij||ij}
\end{align*}

We notice immediately that the sum of the 0th and 1st order energies is the Hartree-Fock energy.
\[ E_0^{(0)} + E_0^{(1)} = \sum_i h_{ii} + \frac{1}{2}\sum_{ij} \braket{ij||ij}   \]

Thus, the first two terms of our energy perturbation expansion is the HF energy.
\[E_0 =  E_0^{(0)} +  E_0^{(1)} + E_0^{(2)} + \cdots \]
\[E_0 =  E_{\mathrm{HF}} + E_0^{(2)} + \cdots \]
It must be that all higher-order terms constitute \textit{correlation energies}, since in principle the expansion should approach the exact
    energy so long as the perturbation is small.

The second-order energy correction to the ground state energy is given by
\[ E_0^{(2)} = \sum_{n \neq 0} \frac{ |\braket{ \Phizero_0 | V | \Phizero_n} |^2  }{ \Ezero_0 - \Ezero_n}  \]
What sort of states are $\Phizero_n$? They can be any excitation from our ground state reference HF determinant $\Phi_0$ (any determinant in our CI expansion).
However, we find that single excitations do not survive:
\[ \braket{\Phi_0 | V | \Phi_i^a } =  \braket{\Phi_0 | H - H_0 | \Phi_i^a } = \braket{\Phi_0 | H | \Phi_i^a } - \braket{\Phi_0 | H_0 | \Phi_i^a }  = 0 \]
Triple and higher excitations do not survive because you cannot form complete contractions, so $\Phizero_n$ can only be the set of unique doubly excited determinants.
What about the $\Ezero_0$ and $\Ezero_n $?
Recall $\Ezero_0$ is the eigenvalue of $H_0$ acting on our Hartree-Fock determinant,
\[H_0 \ket{\Phi_0^{(0)}} = E_0^{(0)} \ket{\Phi_0^{(0)}} \]
 which is just the sum of the occupied orbital energies $E_0^{(0)} = \sum_k \eps_k$,.
The same is true for $\Ezero_n$, but it has a sum over occupied orbital energies which differs by two spin orbital occupations. Thus,
\[\Ezero_0 - \Ezero_n =  \sum_k \eps_k - ( \sum_k \eps_k - (\eps_i + \eps_j - \eps_a - \eps_b)) \]
\[\Ezero_0 - \Ezero_n =  \eps_i + \eps_j - \eps_a - \eps_b \]

So we finally obtain for the second order energy expression,
\[ E_0^{(2)} = \sum_{n \neq 0} \frac{ |\braket{ \Phizero_0 | V | \Phizero_n} |^2  }{ \Ezero_0 - \Ezero_n}  \]
\[ E_0^{(2)} = \sum_{\substack{i < j \\ a < b}} \frac{ |\braket{ \Phi_0 | V | \Phi_{ij}^{ab}} |^2}{ \eps_i + \eps_j - \eps_a - \eps_b  }  \]
Noting that only the first term of $V$ yields a nonzero result,
\[ E_0^{(2)} = \sum_{\substack{i < j \\ a < b}} \frac{ |\braket{ \Phi_0 | \frac{1}{4} \bar{g}_{pq}^{rs} a_{rs}^{pq} | \Phi_{ij}^{ab}} |^2}{ \eps_i + \eps_j - \eps_a - \eps_b  }  \]
\[ E_0^{(2)} = \frac{1}{16} \sum_{\substack{i < j \\ a < b}}  \frac{ |\braket{ ij || ab}|^2}{ \eps_i + \eps_j - \eps_a - \eps_b}\]
\[ E_0^{(2)} = \frac{1}{4} \sum_{ijab} \frac{ |\braket{ ij || ab}|^2}{ \eps_i + \eps_j - \eps_a - \eps_b}\]

\end{document}


\section{Background}
\label{sec:background}

To give context for DSP-PBE, we first explain the traditional concept of programming by example~\cite{cypher93,lieberman01,synasc12}.
Programming by example (PBE) is a synthesis technique that automatically generates programs that coincide with given examples.
An example is specified as a tuple of input and output values.
Given a set $S= \{(i_1, o_1),\ldots, (i_n, o_n)\}$ of input/output examples, the goal is to automatically derive a program $P$ such that for every $j$, $P(i_j) = o_j$.

PBE is in line with one of the often repeated high level goals of functional programming -- to describe \textit{what} a program should do, and not \textit{how} the program should do it.
Instead of writing code, the user provides a list of relevant examples and the synthesis tool automatically generates a program.
In this way, the examples can be seen as an easily readable and understandable specification.
However, even if the synthesized program satisfies all the provided examples, it still might not correspond to the user's intentions.
Examples are, by nature, an incomplete specification.

PBE is a promising research direction that enables easy manipulation of data even for non-programmers~\cite{GulwaniHS12}.
Recent work in this area has focused on manipulating fundamental data types such as strings~\cite{vldb12,icml13} and lists~\cite{FeserCD15,poseraZ15}.
The success and impact of this line of work can be estimated from the fact that PBE ships as part of the popular Flash Fill feature in Excel 2013~\cite{flashfill}.

The core difference between traditional PBE and DSP-PBE is in the application domain of Digital Signal Processing.
Digital Signal Processing (DSP) programming languages provide users with an interface to build signal processing programs in domain specific languages.
Some of these languages provide their own implementations of signal processing primatives, such as SuperCollider~\cite{supercollider}, CSound~\cite{csound}, and PureData~\cite{puredata}.
Other DSP languages provide alternative front-ends to these languages, such as Vivid~\cite{vivid}, which provides Haskell bindings to Supercollider.

Although many DSP languages are full featured enough to write general purpose programs, in this work we focus on the construction of DSP filters.
A DSP filter is, broadly speaking, any program that transforms a digital signal from one form to another.
An example of a DSP filter is a low-pass filter, which takes an input signal and generates an output signal that keeps frequencies below some frequency threshold, but removes frequencies above that threshold.

The most closely related work in audio signal processing is a technique called resynthesis~\cite{masri1996improved}.
Resynthesis is the process of decomposing a sound into its spectrogram, and then building a synthesizer to recreate a similar sound.
The limitation here is that resynthesis builds a generative synthesizer, which does not take into account any information about the components used to create the original sound.
This limitation means that resynthesis cannot be applied in a new context, whereas DSP-PBE allows us to construct a DSP program that can be used with various new input samples to create novel sounds.
For example, DSP-PBE could be given a sample of a trumpet and a trombone, and the generated DSP program could be applied to a violin to hear what a violin sounds like if it was a trumpet that had been turned into a trombone.
In this case we can discover the analogy \textit{trumpet:trombone :: violin:?}.

From a machine learning perspective, the above example use case is closely related to work on learning analogies~\cite{mikolov2013distributed}, where the goal is to discover relations such as \textit{man:king :: woman:queen}.
To do this, words are embedded in a vector space, so that the transformation from \textit{man} to \textit{king}, can be directly applied to \textit{woman}.
There are two keys differences between this approach and DSP-PBE.
The first is that DSP-PBE should produce a human readable transformation.
We would like to generate DSP programs that can be used verbatim, but also inspected and modified by the user.
While program code provides this readability, vector transformations are not comprehensible in the same way.
Second, word embeddings require that the semantics of an object can be embedded into a vector space.
As we will see in Sec~\ref{sec:distance}, a semantic representation of an audio file (what a human perceives) is not immediately recoverable from its direct representation.