\documentclass[lang=en,mode=geye,device=normal,color=blue,14pt]{elegantnote}
\usepackage{amsmath,amssymb}
\usepackage{bbm}
\usepackage{tcolorbox}
\usepackage{graphicx}
\usepackage{booktabs}
 \usepackage{subfigure}

\DeclareMathOperator*{\E}{\mathbb{E}}
\DeclareMathOperator*{\1}{\mathbbm{1}}
\DeclareMathOperator*{\R}{\mathbbm{R}}
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}

\title{Note: Reinforcement Learning - An Introduction}

%\author{Yanqing Wu}
%\institute{Viwistar Robotics}

% \version{0.1.0}
%\date{\today}

\begin{document}
\maketitle

\setlength{\parindent}{0pt}

\newpage

\section{Chap 1: Introduction}
\subsection{Reinforcement Learning}

RL is learning what to do, so as to maximize a numerical reward signal.

RL means taking optimal action with \textbf{long term} results or \textbf{cumulative rewards} in mind.

Two most important distinguishing features of RL: trial-and-error search and delayed reward.

RL methods generally incorporate 3 aspects: sensation, action and goal.

\begin{definition}
\textbf{Supervised Learning} is learning from a training set of labeled examples provided by a knowledgable external supervisor.
\end{definition}

\begin{itemize}
	\item Example: a description of a situation
	\item Label: a specification of the correct action the system should take in a situation, which is often to identify a category to which the situation belongs
	\item Object of Supervised Learning: for the system to extrapolate (generalize) its responses s.t. it acts correctly in unseen situations
	\item Con: inadequate for learning from \textit{interaction}, because it is (often) impractical to obtain examples of desired behavior that are both correct and representative of all the situations in which the agent has to act
\end{itemize}

\begin{definition}
\textbf{Unsupervised Learning} is about finding structure hidden in collections of unlabeled data.
\end{definition}
RL is trying to maximize a reward signal, while Unsupervised Learning is trying to find hidden structure.

Key feature 1: the exploration-exploitation dilemma is one of RL's challenges.
Exploitation: to obtain reward from experienced.
Exploration: to find better actions.

Key feature 2: RL explicitly considers the whole problem of a goal-directed agent interacting with an uncertain environment.

\subsection{Examples}
\subsection{Elements of RL}

Four main subelements of a RL system: a \textit{policy}, a \textit{reward signal}, a \textit{value function}, and a \textit{model} of the environment.

\begin{definition}
\textbf{Policy}, defines the learning agent's way of behaving at a given time.
\end{definition}

A policy is a mapping from perceived states of the environment, to actions to be taken, when in those states.
A policy may be a simple function or a lookup table, or complex computation (e.g. a search process).
Policies may be stochastic, specifying probabilities for each action.

\begin{definition}
\textbf{Reward signal}, defines the goal of a RL problem.
\end{definition}

On each timestep, the environment sends a single number (i.e. \textit{reward}) to the RL agent. The agent's only objective is to maximize the total reward it receives over the long run.

\begin{definition}
\textbf{Value function}, specifies what is good in the long run.
\end{definition}

The \textit{value} of a state is the total amount of reward an agent can expect to acumulate over the future, starting from that state.

!! \textit{Rewards} determine the immediate desirability of states, \textit{values} determines the long-term desirability of states.

State Value function = Expected Return = Discounted sum of all rewards

\begin{definition}
\textbf{Model}, mimics the behavior of the environment.
\end{definition}

The model defines the reward function and transition probabilities. Models are used for \textit{planning}.
Model-based methods: use models and planning to solve RL problems.
Model-free methods: explicitly trial-and-error learner (almost the opposite of planning).

Modern RL spans the spectrum from low-level, trial-and-error learning to high-level, deliberative planning.

\subsection{Limitations and Scope}
\subsection{An Extended Example: Tic-Tac-Toe}
Minimax algorithm \cite{wiki_minimax} (from game theory).

Classical optimization methods are for sequential decision problems (e.g. dynamic programming). We need to estimate an approximate opponent model so to use dynamic programming; in such case, this is similar to RL methods.

During playing, we need to adjust the values of the states as to make more accuracte estimates of the probabilities of winning. The current value of the earlier state is updated to be closer to the value of the later state. This can be done by moving the earlier state's value a fraction of the way toward the value of the later state.

\begin{equation}
V(S_t) \leftarrow V(S_t) + \alpha[V(S_{t+1}) - V(S_t)]
\end{equation}

$S_t$, the state before a move.
$S_{t+1}$, the state after the move.
$V(S_t)$, the update to the estimated value of $S_t$.
$\alpha$, step-size parameter, a small positive fraction, influences the rate of learning.

This update rule is an example of a temporal-difference (TD) learning method. TD means the changes are based on a difference between estimates at two consecutive times.

If $\alpha$ is reduced properly over time, then this method converges (for any fixed opponent, it converges to the true probabilities of winning for each state); if $\alpha$ is not reduced to zero over time, then the agent ``also plays well against opponents that slowly change their way of playing'' (?). My understanding is that, there must have some slow changes, whether the parameter or the opponent model.

\textbf{Evolutionary methods vs. Value Function methods.} Evolutionary methods ignores what happens during the games, and only the final outcome of each game is used (e.g. all of its behavior in a winning game is given credit). In contrast, value function methods evaluate individual states.

In this example, learning started with no prior knowledge except the rules of the game. In reality, prior information can be incorporated into RL in many ways for efficient learning.
RL can be used when we have access to true state, or where some states are hidden, or when different states appear to be the same to the learner.

The player need a model of the game to see the result states in response to actions. While model-free systems cannot know how their environments will change in response to actions.
Model-free methods are building blocks for model-based methods.
Sometimes, it is difficult to construct an accurate environment model, thus model-based methods cannot be used in such cases.

\subsection{Summary}

RL is a computational approach to understanding and automating goal-directed learning and decision making.
It is distinguished from other computational approaches by its emphasis on learning by an agent from direct \textbf{interaction} with its environment, without requiring exemplary supervision or complete models of the environment.

RL uses the formal framework of Markov decision processes (MDP) to defined the interaction between a learning agent and its environment in terms of states, actions, and rewards.

The concepts of value and value function are key to most of the RL methods in this book.
The authors believe that value functions are critical for efficient search in policy space.
The use of value functions distinguishes RL methods from evolutionary methods.

\subsection{Early History of RL}

Three threads:
1. learning by trial and error, originated in the psychology of animal learning. Time from 1850s to early 1980s' revival of RL;
2. optimal control problem and its solution using value functions and dynamic programming. This thread did not involve learning;
3. temporal-difference methods.
These 3 threads came together in late 1980s.
The TD and optimal control threads were fully brought together in 1989 with Chris Watkin's development of Q-learning.

The essential idea of trial-and-error learning: `the drive to achieve some result from the environment, to control the environment toward desired ends and away from undesired ends.'

TD learning methods are driven by the difference between temporally successive estimates of the same quantity.

The authors developed a method for using TD learning + trial-and-error learning, known as the \textit{actor-critic architecture}.

\newpage
\textbf{Part I: Tabular Solution Methods}

Algorithms in this part are described in simplest forms: the state space and action space are small enough for the approximate value functions to be represented as arrays/tables.
In this case, the methods can (often) find exact solutions (i.e. optimal value function and optimal policy).
Algorithms in next part can only find approximate solutions but can be applied to much larger problems.

\textbf{Bandit problems}, RL problem with only a single state. Chap 2 describes general problem formulation (finite MDP) and its main ideas (including Bellman equations and value functions).

Three fundamental classes of methods for solving MDP problems: dynamic programming, Monte Carlo methods and TD learning.
\begin{itemize}
\item DP: well developed, but require a complete and accurate model of the environment
\item MC: no model needed, but not good for incremental computation
\item TD: no model needed and fully incremental, but are complex
\end{itemize}

\newpage
\section{Chap. 2 Multi-armed Bandits}
\begin{itemize}
\item Evaluative: feedback depends solely on the taken action
\item Instructive: feedback is independent of the taken action
\item Nonassociative setting: no learning involved in more than one situation; no need to associate different actions with different situations/states
\item Associative setting: the best action depends on the situation
\end{itemize}

\subsection{A k-armed Bandit Problem}

\textit{k-armed bandit problem}:
You choose one option (action) from \textit{k} different options (actions). The probability of choosing each option (action) is stationary. For each choice, you receive a (probably distinct) numerical reward. Your objective is to maximize the expected total reward over \textit{j} option selections (time steps).

For now, let us define that
\begin{itemize}
\item \textit{Value}: the expected/mean reward of a taken action
\item $A_t$: action selected on time step $t$
\item $R_t$: corresponding reward of $A_t$
\item $q_{*}(a)$:  $q_{*}(a) \doteq \E[R_t|A_t=a]$. The (true) value of an arbitrary action $a$ (i.e. the expected reward given $a$ is selected)
\item $Q_t(a)$: the \textit{estimated} value of action $a$ at time step $t$.
\end{itemize}

If we know each action values, then solving the \textit{k}-armed bandit problem would be always select the action with highest $value$.
If we don't know the action values, then we can estimate them.
Our goal is to have $Q_t(a)$ to be as close to $q_*(a)$ as possible.

\begin{definition}
\textbf{Greedy actions}, action(s) with highest \textbf{estimated} value.
\end{definition}
\textit{exploiting}: select greedy action(s).
\textit{exploring}: select non-greedy action(s). Exploration may improve the estimate value of non-greedy actions.

\subsection{Action-value Methods}

\begin{definition}
\textbf{Action-value Methods}, or Q function, methods for estimating the values of actions, and using the estimates to select actions.
\end{definition}

`Q' means the \textbf{quality} of an action.

Let see a simple action-value method: \textit{sample-average method}, in which action-value is the mean reward for an action.

$$ Q_t(a) \doteq \frac{\sum_{i=1}^{t-1}R_i \cdot \1_{A_{i=a}}}{\sum_{i=1}^{t-1}\1_{A_{i=a}}}$$

\begin{itemize}
	\item $\1$: return 1 if predicate is true, otherwise 0
	\item in the case when denominator is 0 (which makes the fraction mathematically undefined), $Q_t(a)$ is set as default value (e.g. 0)
	\item in the case when denominator $\rightarrow \infty$, by the Law of Large Numbers, $Q_t(a)$ converges to $q_*(a)$
\end{itemize}

\begin{definition}
\textbf{Greedy action selection}, If there are more than one greedy action, select one of them arbitrarily.
$$ A_t \doteq \argmax_{a} Q_t(a) $$
\end{definition}

\begin{definition}
\textbf{$\varepsilon$-greedy action selection} with probability $\varepsilon$, select an action randomly; with probability $1-\varepsilon$, select a greedy action
\end{definition}

\subsection{The 10-armed Testbed}

For Exercise 2.2, note that we should first convert the sequence of actions and rewards to $Q_t(a)$, then we can decide whether the selection is greedy or exploration based on the definition of $\varepsilon$-greedy action selection.

\subsection{Incremental Implementation}

Recall that estimated action value $Q_n$, using sample-average method, is calculated as:
$$ Q_n \doteq \frac{R_1 + \cdots + R_{n-1}}{n-1} $$

We don't want to waste memory and computational resource for each new reward.
Intuitively, to save memory, we can simplify the equation by first reconstruct the previous estimated action value $Q_{n-1}$ and then evaluate $Q_n$ with the new reward $R_n$.
Therefore,
$$ Q_{n+1} = \frac{Q_n \cdot (n-1) + R_n}{n} = \frac{Q_n \cdot n - Q_n + R_n}{n} = Q_n + \frac{1}{n}[R_n - Q_n] $$

The form $Q_{n+1} = Q_n + \frac{1}{n}[R_n - Q_n]$ is an important form occurs in this book. The general form is
$$ NewEstimate \leftarrow OldEstiamt + StepSize\underbrace{[Target - OldEstimate]}_{error} $$

% \begin{tcolorbox}[width=\textwidth,colback={grey},title={A simple bandit algorithm},colbacktitle=black,coltitle=white]}
\begin{tcolorbox}[width=\textwidth,title={A simple bandit algorithm}]
	\begin{align*}
	   & \text{Initialize, for a = 1 to }k: \\
	   & \quad Q(a) \leftarrow 0 \\
	   & \quad N(a) \leftarrow 0 \\
	   & \text{Loop forever: }\\
	   & \quad A \leftarrow \left\{
	    \begin{array}{ll}
	        \argmax_x Q(a) & \text{with prob. } 1 - \varepsilon \text{ (breaking ties randomly) }\\
	        \text{a random action} & \text{with prob. } \varepsilon
	    \end{array}
	\right. \\
	   & \quad R \leftarrow bandit(A) \\
	   & \quad N(A) \leftarrow N(A) + 1 \\
	   & \quad Q(A) \leftarrow Q(A) + \frac{1}{N(A)}[R-Q(A)]
	\end{align*}
\end{tcolorbox}

\subsection{Tacking a Nonstationary Problem}

In previous section, the reward probability distribution is stationary (i.e. doesn't change over time).
For nonstatoinary cases, let say we give more weight to recent rewards than to `long-past' rewards; for doing so, one popular way is to use a constant $\alpha$ (step size).

Not all choices of $\alpha_n(a)$ guarantee convergence.
In \textit{stochastic approximation theory}, two conditions required to assure convergence ``with probability 1''\footnote{i.e. \textit{almost sure}. No distinction between `sure' and `almost sure' in finite sample space, but becomes important in infinite sample space. see \url{https://en.wikipedia.org/wiki/Almost_surely}}:

$$ \sum_{n=1}^\infty \alpha_n(a) = \infty $$
$$ \sum_{n=1}^\infty \alpha_n^2(a) < \infty $$

The 1st condition is required to guarantee that the steps are large enough to eventually overcome any initial conditions or random fluctuations.
The 2nd condition is required to guarantee that the steps eventually become small enough to assure convergence.

Therefore, in sample-average method, $\alpha_n(a) = \frac{1}{n}$ converages, but a constant step-size parameter $\alpha_n(a)=\alpha$ does not converge.

Note that sequences of $\alpha$ that meet the conditions often converge very slowly and often need considerable tunning.
These sequences of $\alpha$ often used in theoretical work, and are seldom used in real-world application.

\subsection{Optimistic Initial Values}

Higher (more optimistic) initial values may result in better performance. More optimistic initial values ($Q_i(a)$) encourages more exploration.
This may only work well on stationary problems; as in nonstationary problems, the drive for exploration is inherently temporary.

\subsection{Upper-Confidence-Bound Action Selection}

In $\varepsilon$-greedy action selection, it is not ideal to just randomly select actions; instead, we want to select among the non-greedy actions according to their potential for actually being optimal. Such taking into account both (1) how close their estimates are to being maximal, and (2) the uncertainties in those estimates.

\begin{definition}
Upper-Confidence-Bound (UCB) Action Selection

\[
A_t \doteq \argmax_a \bigg[ Q_t(a) + c\sqrt{\frac{\text{ln }t}{N_t(a)}} \bigg]
\]
\end{definition}
$N_t(a)$, denotes the number of times that action $a$ has been selected prior to time $t$. If $N_t(a)=0$, then $a$ is considered to be a maximizing action.
$c$, $c > 0$ (confidence level) controls the degree of exploration

The idea of UCB action selection is that the square-root term is a measure of the uncertainty/variance in the estimate of $a$'s value.
Each time $a$ is selected, the uncertainty is (presumably) reduced as $N_t(a)$ increments.
Each time a non-$a$ is selected, $t$ increments but $N_t(a)$ doesn't, thus the uncertainty estimate increases.
All actions will eventually be selected, but actions with lower value estimates or that have been frequently selected, will be selected with decreasing frequency over time.

UCB is useful in bandit problems, but is (usually) not practical in general RL settings. It is difficult to deal with (1) nonstationary problems, and (2) large state spaces.

\subsection{Gradient Bandit Algorithms}
We have considered methods that (1) estimate actions values; (2) select actions based on estimations.
We now consider a new approach for action selection: learning a numerical \textit{preference} for each action \textit{a}, which we denote $H_t(a) \in \R$.
The larger the preference, the more often that action is taken, but the preference has no interpretation regarding rewards.
Only the \textbf{relative} preference of one action over another is important.
The action probabilities is determined by \textit{soft-max distribution}.

\[
P(A_t=a) \doteq \frac{e^{H_t(a)}}{\sum_{b=1}^k e^{H_t(b)}} \doteq \pi_t(a)
\]

$\pi_t(a)$ is the probability of taking action $a$ at time $t$. The initial probability are the same for all actions.

There is a natural learning algorithm in this setting based on the idea of stochastic gradient descent.
This is based on updating the preference values as such, after taking action $A_t$ and obtaining reward $R_t$

$$ H_{t+1}(A_t)\doteq H_t(A_t) + \alpha (R_t - \overline{R_t})(1-\pi_t(A_t)) $$

$$ H_{t+1}(a) \doteq H_t(a) - \alpha(R_t - \overline{R_t})\pi_t(a)\quad\text{for all }a\neq sA_t $$

$\overline{R_t} \in \R $ is the average of the rewards up to but not including time $t$ (with $\overline{R_1} = R_1$)

If the reward is higher than the baseline $\overline{R_t}$, then the probability of taking $A_t$ in the future is increased; and if the reward is below baseline, then the probability is decreased.
The non-selected actions move in the opposite direction.

The above update scheme is equivalent to \textbf{stochastic gradient ascent} with bath size 1.

\subsection{Associative Search (Contextual Bandits)}

For now, we have only considered nonassociative tasks, that is, tasks in which there is no need to associate different actions with different situations.
However, in the general RL task, there is more than one situation (state?), and the goal is to learn a policy, i.e., a mapping from states to optimal actions given the state.

Suppose when a bandit task is selected for you, you are given some distinctive clue about its identity (but not its action values). For instance, if screen=red (actual reward is hidden from you), select arm 1; if screen=green, select arm 2... This is an example of an associative search task, it involves both trial-and-error learning to search for the best actions, and association of these actions with the situations in which they are best. This task is now called \textit{contextual bandits} in the literature.

In this setting, we still don't have actions affecting the next situation as well as the reward. This will add another layer of complexity and lead us to full RL problems.

\begin{itemize}
\item (level 1) $k$-armed bandit problem: select one of $k$ actions that have fixed probabilities. For each selected action, a reward is given. The goal is to maximize the total reward after $j$ selections.
\item (level 2) associative search task (contextual bandits): Same as $k$-armed bandit problem, the action affects only the immediate reward. Unlike $k$-armed bandits problem, what situations/state you are facing are told and it involves learning a policy.
\item (level 3) full RL problem: On top of associative search task, actions affect next situation and reward.
\end{itemize}

\subsection{Summary}

This chapter presented several ways to balance exploration and exploitation:

\begin{itemize}
\item $\varepsilon$-greedy methods choose randomly among actions a small fraction of the time to encourage exploration
\item UCB methods choose deterministically but favouring actions with uncertain estimates
\item Gradient methods don't estimate action values, but preferences, and choose actions probabilistically according to the preferences
\item Optimistic initialization sets optimistic initial $Q$ values in order to encourage a very active exploration in the initial phase, leading to a fast convergence with no bias
\end{itemize}

There is no method among the above that is best. Common ways to assess performance of these kind of algorithms are through graphs:
\begin{itemize}
\item learning curve: shows the performance of an algorithm vs. iterations for a certain parameter setting
\item parameter studies: summarize each algorithm by the average reward over the first 1000 steps, and show such a value for various parameter settings and algorithms.
\end{itemize}

One should also consider the sensitivity to parameter setting, that is an indication of \textbf{robustness}.

Despite the simplicity of methods presented in this chapter, they can be considered state of the art. More sophisticated methods usually impose additional complexity and assumptions.

Another approach to balancing exploration and exploitation in $k$-armed bandit problems is to compute a special kind of action value called \textit{Gittins index}, which is an instance of Bayesian methods. This assume a known initial distribution over the (stationary) action values and then update the distribution exactly after each step.
In general, the update computations are complex, but for \textit{conjugate priors} they are easy.
One way to select the best action at each time step is based on posterior probability.
This method, sometimes called posterior sampling or Thompson sampling.

\newpage
\section{Finite Markov Decision Processes}

\begin{definition}
finite Markov Decision Processes, involves evaluative feedback (as in bandits) and associative settings (choosing different actions in different situations)
\end{definition}

MDPs are a classical formalization of \textit{sequential} decision making, where actions influence both immediate rewards and subsequent situations/states (thus influence future rewards as well).
MDPs involves delayed reward. MDPs need to consider trade-off between immediate reward and delayed reward.

In bandit problems, we estimated the value $q_{*}(a)$ of each action $a$.

In MDPs, we estimate $q_{*}(s,a)$ (value of each action $a$ in each state $s$), or $v_{*}(s)$ (value of each state given optimal action selections).

MDPs are an ideal form of the RL problem where precise theoretical statements can be made.
Key elements of RL's mathematical structure: returns, value functions, and Bellman equations.

\subsection{The Agent-Environment Interface}

\begin{itemize}
\item agent: learner and decision maker
\item environment: everything outside the agent, and which agent interacts with
\end{itemize}

\begin{figure}[!h]
  \includegraphics[width=\linewidth]{MDP_interface.PNG}
  \caption{The agent-environment interaction in a MDP}
  \label{fig:MDP_interface}
\end{figure}

At each time step $t$, the agent receives some representation of the environment's $state$, $S_t$, and on the state selects an $action$, $A_t$. One time step later, the agent receives reward $R_{t+1}$ as a result of the action, and find itself in a new state $S_{t+1}$. The trajectory/sequence is:

$$ S_0, A_0, R_1, S_1, A_1, R_2, S_2, A_2, R_3, \cdots $$

In a \textit{finite} MDP, the state sets, action sets and reward sets are finite.

Following defines the \textit{dynamics} of the MDP.

$$ p(s', r| s, a) \doteq P(S_t = s', R_t = r | S_{t-1} = s, A_{t-1} = a) $$
$$ \sum_{s' \in \mathcal{S}} \sum_{r \in \mathcal{R}} p(s',r|s,a) = 1, \forall s \in \mathcal{S}, a \in \mathcal{A}(s) $$

$p$ specifies a probability distribution for each choice of $s$ and $a$.
The probability of each possible value for $S_t$ and $R_t$ depends on the immediately preceding state and action, $S_{t-1}$ and $A_{t-1}$ (note not including earlier states and actions).

\begin{definition}
Markov property\footnote{Markov property refers to the memoryless property of a stochastic process}, the future only depends on the current state, not the history.
\end{definition}

\begin{definition}
state-transition probabilities,
$$ p(s'|s,a) \doteq P(S_t=s'|S_{t-1}=s, A_{t-1}=a) = \sum_{r\in \mathcal{R}} p(s',r|s,a) $$
\end{definition}
We can express state transition probabilities in an Matrix as shown in Figure \ref{fig:state-trans-prob-eg}. Each row in the matrix represents the probability from moving from our original or starting state to any successor state. Sum of each row is equal to 1.
\begin{figure}[h!]
  \includegraphics[width=\linewidth]{state_transition_probability_example.jpg}
  \caption{An example of state transition probability.}
  \label{fig:state-trans-prob-eg}
\end{figure}

\begin{definition}
expected rewards for state-action pairs

$$ r(s,a) \doteq \E [R_t | S_{t-1}=s, A_{t-1}=a] = \sum_{r\in \mathcal{R}} \sum_{s\in \mathcal{S}} p(s',r|s,a) $$
\end{definition}

\begin{definition}
expected rewards for state-action-nextState,

$$ r(s,a,s') \doteq \E [R_t | S_{t-1}=s, A_{t-1}=a, S_t=s'] = \sum_{r \in \mathcal{R}} r \frac{p(s',r|s,a)}{p(s'|s,a)} $$
\end{definition}

\paragraph{agent-environment}
The agent-environment boundary represents the lit of the agent's absolute control, not of its knowledge. Anything that cannot be changed by the agent.

Any problem of learning goal-directed behavior can be reduced to 3 signals passing back and forth between an agent and its environment:
\begin{itemize}
\item action: the choice made by the agent
\item the basis on which the choice is made
\item the agent's goal
\end{itemize}

\begin{example}
Bioreactor,

\begin{itemize}
\item actions (vector): target temperatures, target sitrring rates
\item states (vector): sensory readings, ingredients input, target chemical
\item rewards (number): production rate of useful chemical (target)
\end{itemize}

\end{example}

\begin{example}
Robotic Arm (pick-and-place task)

\begin{itemize}
\item actions: voltages applied to each joint motor
\item states: readings of joint angles and velocities
\item rewards: +1 for successfully object pick-and-place; small negative reward at each time step (punish jerkiness motion)
\end{itemize}
\end{example}

\subsection{Goals and Rewards}
 In RL, thr purpose/goal of the agent is to maximize \textit{cumulative reward in the long run}.
 It is important that the set-up rewards indicate what we truly want accomplished.
 In particular, rewards are not the place to tell/impart agent prior knowledge about \textit{how} to achieve our goal.
 For example, in a chess game, the agent should only be reward for winning, instead of achieving subgoals like taking opponent's pieces.
 Better places for imparting this kind of prior knowledge in (1) initial policy; or (2) initial value function.

 The reward signal is a way of imparting to the agent \textit{what} you want achieved, not \textit{how} you want it achieved.

\subsection{Returns and Episodes}

A RL agent's goal is to maximize the cumulative reward it receives in the long run.
In general, we seek to maximize the \textit{expected return} $G_t$.

$$ G_t \doteq R_{t+1} + R_{t+2} + \cdots + R_T $$

where $T$ is a final time step.

The final time step marks the end of a subsequences.

\begin{definition}
Episode, a subsequence of the agent-environment interaction. Each episode ends in \textbf{terminal state}, followed by a \textbf{reset} to a standard starting state or to a \textbf{sample} from a standard distribution of starting states. The next episode begins independently of how the previous one ended.
\end{definition}

The set of all nonterminal states, $\mathcal{S}$. All states and the terminal state, $\mathcal{S^+}$.

\begin{definition}
Episodic tasks, tasks with episodes that with terminal state.
\end{definition}

\begin{definition}
continuing tasks, tasks without terminal state.
\end{definition}

This task uses discounted rewards to deal with $T=\infty$.

\begin{definition}
discounted rewards $G_t$. discount rate, $0 \leq \gamma \leq 1$

$$ G_t \doteq R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \cdots = \sum_{k=0}^{\infty} \gamma^k R_{t+k+1} $$
\end{definition}

The discount rate determines the present value of future rewards. As long as the reward sequence {$R_k$} is bounded, the infinite sum $G_t$ has a finite value.
If $\gamma \rightarrow 0$, the agent is myopic; if $\gamma \rightarrow 1$, the agent is farsighted.

If the reward is constant $R$, then we have the \textit{geometric series}

$$ G_t = \sum_{k=0}^{\infty} R\gamma^k = R\frac{1}{1-\gamma} $$

Note that the returns at successive time steps are related to each other (which is important for RL):

$$ G_t \doteq R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \cdots = R_{t+1} + \gamma G_{t+1} $$

\begin{example} pole-balancing
\begin{itemize}
\item terimnal state: if the pole falls past a given angle from vertical or if the cart runs off the track
\item starting state: pole reset to vertical
\item reward (episodic): +1 for each time step before falling
\item reward (continuing): -1 on each falling, 0 for all other times
\end{itemize}
\end{example}

\begin{figure}[!h]
  \includegraphics[width=\linewidth]{MDP_pole.PNG}
  \label{fig:MDP_pole}
\end{figure}

\subsection{Unified Notation for Episodic and Continuing Tasks}

We use $S_{t,i}$ to represents state at time $t$ of episode $i$ (same for $A_{t,i}$, $R_{t,i}$, etc).
In practice, we dropped the $i$ as we normally don't have to distinguish between different episodes.
So, now $S_t$ refers to $S_{t,i}$.

We use an $absorbing state$ to replace the terminate state. The absorbing state transits to itself and have reward = 0.
This way, we can have the same formula of $G_t$ for episodic and continuing tasks.

\subsection{Policies and Value Functions}

Value functions (either V or Q) are \textbf{always} conditional on some policy $\pi$. Sometimes in literature we leave off the $\pi$ or $*$ and just refer to $V$ and $Q$, because it’s implicit in the context, but ultimately, every value function is always with respect to some policies.

\begin{definition}
There are two value functions:
\begin{itemize}
\item value functions of states, or V, estimate how much expected return it is received for the agent to be in a given state;
\item value functions of state-action pairs, or Q, estimate how much expected return it is received to perform a given action in a given state.
\end{itemize}
\end{definition}

\begin{definition}
policy, $\pi(A_t=a|S_t=s)$, a mapping from states to probabilities of selecting each action
\end{definition}

Expectation of $R_{t+1}$ in terms of $\pi$ and the four-argument function $p$:
\[
\E\left[R_{t+1} \middle|S_{t}\right] = \sum_{a \in \mathcal{A}} \pi\left(a \middle| S_t\right) \cdot r(S_t, a) = \sum_{a \in \mathcal{A}} \pi\left(a \middle| S_t\right) \cdot \sum_{r \in \mathcal{R}} r \sum_{s' \in \mathcal{S}} p\left(s', r \middle| S_t, a\right)
\]
My word explanation: the expectation of reward given current state $S_t$= the total sum of the probability of perform action $a$ at state $s$ $\cdot$ the reward $r$ $\cdot$ the probability of getting reward $r$ (when performs action $a$ at state $s$) (?)

\begin{definition}
state-value function for policy $\pi$, (value function of a state $s$ under a policy $\pi$), $v_{\pi}(s)$, is the expected return when starting in $s$ and following $\pi$ thereafter. For MDPs:

$$ v_\pi(s) \doteq \E_\pi[G_t | S_t = s] = \E_\pi [ \sum_{k=0}^{\infty} \gamma^k R_{t+k+1} | S_t = s], \forall s \in \mathcal{S} $$
\end{definition}

\begin{definition}
action-value function for policy $\pi$, (value of taking action $a$ in state $s$ under a policy $\pi$), $q_\pi(s,a)$, as the expected return starting from $s$, taking the action $a$, and \textbf{then} following policy $\pi$:

$$ q_\pi(s,a) \doteq \E_\pi [G_t | S_t = s, A_t = a] = \E_\pi [\sum_{k=0}^{\infty} \gamma^k R_{t+k+1} | S_t = s, A_t = a] $$
\end{definition}

The main difference between $Q$ and $V$ then, is the $Q$-value lets you play a hypothetical of potentially taking a different action in the first time step than what the policy might prescribe and then following the policy from the state the agent winds up in.
\newline

Equation of $v_\pi$ in terms of $q_\pi$ and $\pi$:
$$ v_\pi(s) = \sum_a \pi(a|s)q_\pi(s,a) $$

My word explanation: the value function is the total sum of probability of choosing action $a$ at state $s$ $\times$ the action-value of taking each action.
\newline
Equation of $v_\pi$ in terms of $\pi$ and $p$:
\begin{align*}
v_\pi(s) & \doteq \E_\pi[G_t | S_t = s]\\
& = \E_\pi[R_{t+1} + \gamma G_{t+1} | S_t = s]\\
& = \sum_a \pi(a|s) \sum_{s'}\sum_r p(s',r|s,a) [r+ \gamma \E_\pi[G_{t+1} | S_{t+1}=s']]\\
& = \sum_a \pi(a|s) \sum_{s',r} p(s',r|s,a) [r+ \gamma v_\pi(s')]
\end{align*}

Equation of $q_\pi$ in terms of $v_\pi$ and the four argument $p$
$$ q_\pi(s,a) = \sum_{s', r} p(s',r|s,a) \underbrace{ [r+\gamma v_\pi(s')] }_{G_t?} $$

\begin{definition}
Monte Carlo methods, estimation methods that involve averaging over many random samples of actual returns.
\end{definition}

For example, an agent follows policy $\pi$; and for each state $s$, the agent maintains an average of the actual returns which have followed that state, then the average converges to the state's value $v_\pi(s)$. (Recall the Law of Large Numbers, or sample-average method in Chapter 2).
For another example, if separate averages are kept for each action taken in each state, then these averages will similarly converge to the action values $q_\pi(s,a)$.

\begin{definition}
Bellman equation for $v_\pi$:

$$ v_\pi = \sum_a \pi(a|s) \sum_{s',r} p(s',r|s,a) [r+ \gamma v_\pi(s')] $$
\end{definition}

It expresses a relationship between the value of a state and the values of its successor states.

\begin{figure}[!h]
  \centering
  \includegraphics[]{MDP_backup.PNG}
  \caption{Backup diagram for $v_\pi$}
  \label{fig:MDP_backup}
\end{figure}
Note that in backup diagram, the nodes do not necessarily represent distinct states (e.g. a state might be its own successor).

The value function $v_\pi$ is the unique solution to its Bellman equation (??). My understanding is that since $v_\pi$ is the only solution, meaning that $v_\pi$ must converge to a point?

\begin{definition}
Bellman equation for $q_\pi$:

$$ q_\pi(s,a) \doteq \E_\pi [G_t | S_t = s, A_t = a] = \sum_{s',r} p(s',r|s,a)[r+\gamma \sum_{a'} \pi(a'|s')q_\pi (s',a')] $$
\end{definition}

\begin{figure}[!h]
  \centering
  \includegraphics[]{MDP_backup_Qpi.PNG}
  \caption{Backup diagram for $q_\pi$}
  \label{fig:MDP_backup_qpi}
\end{figure}

\begin{figure}[!h]
    \centering
    \subfigure[]{\includegraphics[width=\textwidth]{MDP_1.PNG}}
    \subfigure[]{\includegraphics[width=\textwidth]{MDP_2.PNG}}
    \caption{ (a) $ v_\pi(s)=\sum_a \pi(a|s)q_\pi(s,a) $ (b) $ q_\pi(s,a)=\sum_{s',r}p(s',r|s,a)[r+\gamma v_\pi(s')] $ }
    \label{fig:MDPMDP}
\end{figure}
The value of a state depends on the values of the actions possible in that state and on how likely each action is to be taken under the current policy.

\subsection{Optimal Policies and Optimal Value Functions}

Solving a RL means finding a policy that achieves a lot of reward over the long run.

\begin{definition}
optimal policy, $\pi_*$

$$ \pi_* \geq \pi' \iff v_\pi(s) \geq v_{\pi'}(s), \quad \forall s \in \mathcal{S} $$
\end{definition}

\begin{definition}
optimal state-value function, $v_*$

$$ v_*(s) \doteq \max_\pi v_\pi(s), \quad \forall s \in \mathcal{S} $$
\end{definition}

\begin{definition}
optimal action-value function, $q_*$

$$ q_*(s) \doteq \max_\pi q_\pi(s,a), \quad \forall s \in \mathcal{s}, a \in \mathcal{A} $$
\end{definition}

We can write $q_*$ in terms of $v_*$:

$$ q_*(s,a) = \E [R_{t+1} + \gamma v_* (S_{t+1}) | S_t = s, A_t = a] $$

\begin{definition}
Bellman optimality equation for $v_*$,
\begin{align*}
v_*(s) & = \max_{a\in\mathcal{A}}(s) q_{\pi_*} (s,a) \\
& = \max_a \E [R_{t+1} + \gamma v_* (S_{t+1}) | S_t = s, A_t = a] \\
& = \max_a \sum_{s',r} p(s',r|s,a)[r+\gamma v_*(s')]
\end{align*}
\end{definition}

\begin{definition}
Bellman optimality equatoin for $q_*$,
\begin{align*}
q_*(s,a) & = \E [R_{t+1} + \gamma \max_{a'} q_*(S_{t+1}, a') | S_t = s, A_t = a] \\
& = \sum_{s',r} p(s',r|s,a)[r+\gamma \max_{a'} q_*(s',a')]
\end{align*}
\end{definition}

\begin{figure}[!h]
  \centering
  \includegraphics[]{MDP_optimal.PNG}
  \caption{Backup diagram for $v_*$ and $q_*$}
  \label{fig:MDP_optimal}
\end{figure}

Explicitly solving the Bellman optimality equation provides one route to finding $\pi_*$, and thus to solving the RL problem.
However, this solution is rarely useful. It needs an exhaustive search, looking ahead at all possibilities, computing their probabilities of occurrence and their desirabilities in terms of expected rewards.
This solution relies on at least 3 assumptions:
\begin{itemize}
\item the dynamics of the environment are accurately known
\item computational resources are sufficient
\item the states have the Markov property
\end{itemize}

In RL, one typically has to settle for approximate solutions. Many RL methods can be taken as approximately solving the Bellman optimality equation.

\paragraph{Exercise} Consider the contining MDP shown, the only decision to be made is in the top state with two actions (left, right).
Find optimal policy when $\gamma$ = 0.9.

\begin{figure}[!h]
  \centering
  \includegraphics[]{ex3-22.PNG}
  \label{fig:ex3-22}
\end{figure}

Recall the definition of $v_\pi(s)$ and $q_\pi(s)$, we get

\begin{itemize}
\item Left Policy
\begin{itemize}
\item $ v(s_0) = 1 + 0.9\cdot v(s_L) $
\item $ v(s_L) = 0 + 0.9\cdot v(s_0) $
\item $ v(s_R) = 2 + 0.9\cdot v(s_0) $
\end{itemize}
\item Right Policy
\begin{itemize}
\item $ v(s_0) = 0 + 0.9\cdot v(s_R) $
\item $ v(s_L) = 0 + 0.9\cdot v(s_0) $
\item $ v(s_R) = 2 + 0.9\cdot v(s_0) $
\end{itemize}
\end{itemize}
Solving above we find that all state values of Right Policy is greater than all state values of Left Policy. Thus, `right' is the optimal policy for $\gamma$ = 0.9.

It is important to note that if we take `left' action in $s_0$, then policy $\pi$ would never take us to state $s_R$ (same for `right' and $s_L$); however, due to the definition of $\pi_*$, we must evaluate the value function for all states, even ones that would not be visited under a policy we are evaluating.

\subsection{Optimality and Approximation}

It is difficult to reach optimal solutions due to constraints from computational resource and memory. We can often approximate the optimal solutions.

\subsection{Summary}

\begin{itemize}
\item RL is about learning from interaction how to behave in order to achieve a goal.
\item RL \textit{agent} and its \textit{environment} interact over a sequence of discrete time steps.
\item \textit{actions}: choices made by the agent, \textit{states}: the basis for making the choices, \textit{rewards}: basis for evaluating the choices.
\item everything inside agent is known and controllable; environment is incompletely controllable, and are partialily known
\item a \textit{policy} is a stochastic rule that the agent selects actions as a function of states
\item agent's objective is to maximize reward it receives in the long run
\item when RL setup with agent, environment, states, actions, rewards, policy and formulated with well-defined transition probabilities it constitutes a Markov decision process (MDP)
\item a finite MDP is an MDP with finite state sets, action sets and reward sets.
\item \textit{return} is the function of future rewards that the agent seeks to maximize (in expected value)
\item undiscounted formulation is appropriate for \textit{episodic tasks}
\item discounted formulation is appropriate for \textit{\textbf{tabular} continuing tasks} (but not for approximate continuing tasks; see chap 10.3-4)
\item \textit{value functions} $v_\pi$ (state) and $q_\pi$ (state-action pair) are the expected return from that state/state-action under policy $\pi$
\item \textit{optimal value functions} $v_*$ and $q_*$ are the largest expected return by any policy
\item A policy whose value functions are optimal is an optimal policy
\item !! A MDP can have many optimal policies, but can have only one unique optimal value function ($v_*$ and $q_*$)
\item Any policy that is \textit{greedy} w.r.t $v_*$ and $q_*$ must be $\pi_*$
\item The Bellman optimality equations are special consistency conditions that the optimal value functions must satisfy
\item In RL most cases, their optimal solutions cannot be found but must be approximated in some way.
\end{itemize}

\newpage
\section{Dynamic Programming}

In RL, DP refers to a collection of algorithms that can be used to compute optimal policies $\pi_*$ given a perfect model of the environment as a Markov decision process (MDP).
DP itself cannot be useful in RL due to its computational cost and requirement of a perfect model, but DP provides an essential foundation for the understanding of other algorithms.
Other algorithms can be treated as attempts to achieve much the same effect as DP, only with less computation and without assuming a perfect model of the environment.

Starting with this chapter, we assume the environment is a finite MDP (i.e. $S$, $A$, $R$ are finite and dynamics are given by a set of probabilities $p(s',r|s,a)$).
A common way of obtaining approximate solutions for tasks with continuous states and actions is to quantize the state and action spaces and then apply finite-state DP methods.
In this chapter we show how DP can be used to compute the value functions defined in Chapter 3. As shown in Chapter 3, we can obtain $\pi_*$ once we have found $v_*$ or $q_*$, which satisfy the Bellman optimality equations:

\begin{align*}
v_*(s) & = \max_a \E [R_{t+1} + \gamma v_*(S_{t+1}) | S_t = s, A_t = a] \\
& = \max_a \sum_{s',r} p(s',r|s,a)[r + \gamma v_*(s')]
\end{align*}

\begin{align*}
q_*(s,a) & = \E [R_{t+1} + \gamma \max_{a'} q_*(S_{t+1},a') | S_t = s, A_t = a] \\
& = \sum_{s',r} p(s',r|s,a)[r+\gamma \max_{a'} q_*(s',a')]
\end{align*}

\subsection{Policy Evaluation (Prediction)}
\begin{definition}
policy evaluation, compute the state-value function $v_\pi$ for an arbitrary policy $\pi$
\end{definition}

Policy evaluation is also called prediction problem.

Recall that $\forall s \in \mathcal{S}$,

\begin{align*}
v_\pi(s) & \doteq \E_\pi [G_t | S_t = s] \\
& = \E_\pi [R_{t+1} + \gamma G_{t+1} | S_t = s] \\
& = \E_\pi [R_{t+1} + \gamma v_\pi (S_{t+1}) | S_t = s] \\
& = \sum_a \pi (a|s) \sum_{s',r} p(s',r|s,a) [r+\gamma v_\pi (s')]
\end{align*}

If the environment's dynamics are completely known, then the last equation is a sysnteim of $|\mathcal{S}|$ simultaneous linear equations.
For this case, iterative solution methods are most suitable. Consider a sequence of approximate value functions $v_0$, $v_1$, $v_2$, $\cdots$, each mapping $\mathcal{S^+}$ to $\R$.
The initial approximation, $v_0$, is chosen arbitrarily, and each successive approximation is obtained by using the Bellman equation $v_\pi$ as an update rule ($\forall s$):

\begin{align*}
v_{k+1} & \doteq \E_\pi [R_{t+1} + \gamma v_k (S_{t+1}) | S_t = s] \\
& = \sum_a \pi (a|s) \sum_{s',r} p(s',r|s,a)[r+\gamma v_k (s')]
\end{align*}

We see that $v_k$ = $v_\pi$ is a fixed point for this update rule as $k \rightarrow \infty$, $v_k$ converages to $v_\pi$. This algorithm is called \textit{iterative policy evaluation}.
The update is called \textit{expected update}. It is called \textit{expected} because the updates are based on an expectation over all possible next states rather than on a sample next state.

In this algorithm, we use one array and update values in place.
\begin{tcolorbox}[width=\textwidth,title={Iterative Policy Evaluation, for estimating $V \approx v_\pi$}]
Input $\pi$ (the policy to be evaluated); $\theta > 0$ threshold determining accuracy of estimation.
Initialize $V(s)$ arbitrarily, for $s \in \mathcal{S}$, and $V$ to 0.

Loop:
\begin{align*}
&\quad\Delta \leftarrow 0 \\
&\quad\text{Loop for each } s \in \mathcal{S} \\
&\quad\quad v \leftarrow V(s) \\
&\quad\quad V(s) \leftarrow \sum_a \pi (a|s) \sum_{s',r} p(s',r|s,a)[r + \gamma V(s')] \\
&\quad\quad \Delta \leftarrow \text{max}(\Delta, |v-V(s)|) \\
&\text{until } \Delta < \theta
\end{align*}
\end{tcolorbox}

Policy evaluation of $q_\pi$:

\begin{align*}
q_\pi(s,a) & \doteq \E_\pi [G_t | S_t = s, A_t = a] \\
& = \E_\pi [R_{t+1} + \gamma G_{t+1} | S_t = s, A_t = a] \\
& = \E_\pi [R_{t+1} + \gamma \sum_{s',a'} q_\pi (s',a') | S_t = s, A_t = a] \\
& = \sum_{s',r} p(s',r|s,a) [ r+ \gamma \sum_{a'} \pi(a'|s')q_\pi(s',a')]
\end{align*}

\begin{align*}
q_{k+1}(s,a) & \doteq \E_\pi [R_{t+1} + \gamma G_{t+1} | S_t = s, A_t = a] \\
& = \sum_{s',r} p(s',r|s,a)[r+\gamma \sum_{a'} \pi (a'|s')q_k(s',a')]
\end{align*}

\subsection{Policy Improvement}

\begin{definition}
policy improvement theorem. Let $\pi$ and $\pi'$ be any pair of deterministic policies, where $\pi'$ is idential to $\pi$ except $\pi'(s) = a \neq \pi(s)$, s.t. $\forall s \in \mathcal{S}$,

$$ q_\pi (s,\pi'(s)) \geq v_\pi(s) $$

Then the policy $\pi'$ must be btter (or as good as) than $\pi$. That is, it must obtain greater or equal expected return from all states $s \in \mathcal{S}$

$$ v_{\pi'}(s) \geq v_\pi (s) $$
\end{definition}

\begin{definition}
policy improvement, the process of making a new policy $\pi'$ that improves on an original policy $\pi$, by making it greedy w.r.t the value function of $\pi$.
\end{definition}

If there are ties in policy improvement steps, each maximizing action can be given a portion of the probability of being selected in the new greedy policy.

\subsection{Policy Iteration}

It is trivial to see that we can ultimately reach an optimal policy through iterative improving policies and value functions \footnote{In large-scale reinforcement learning problems, it is typically impractical to run either of these steps to convergence, and instead the value function and policy are optimized jointly. By Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor}.

\begin{definition}
policy iteration, the way of finding an optimal policy.

$$\pi_0 \xrightarrow{E} v_{\pi_0} \xrightarrow{I} \pi' \xrightarrow{E} v_{\pi'} \xrightarrow{I} \pi'' \xrightarrow{E} \cdots \xrightarrow{I} \pi_* \xrightarrow{E} v_* $$
\end{definition}

\begin{tcolorbox}[width=\textwidth,title={Policy Iteration (using iterative policy evaluation) for estimating $\pi \approx \pi_*$}]
1. Initialization

$\quad V(s) \in\R$ and $\pi(s) \in A(s)$ arbitrarily $\forall s \in \mathcal{S}$; $V(terminal) \doteq 0$

2. Policy Evaluation
\begin{align*}
&\text{Loop:}\\
&\quad\Delta \leftarrow 0\\
&\quad\text{Loop for each } s \in \mathcal{S}\\
&\quad\quad v \leftarrow V(s)\\
&\quad\quad V(s) \leftarrow \sum_a \pi (a|s) \sum_{s',r} p(s',r|s,a)[r + \gamma V(s')]\\
&\quad\quad \Delta \leftarrow \text{max}(\Delta, |v-V(s)|)\\
&\text{until } \Delta < \theta
\end{align*}

3. Policy Improvement
\begin{align*}
&policyStable \leftarrow true\\
&\text{For each } s \in \mathcal{S}\\
&\quad oldAction \leftarrow \pi(s)\\
&\quad \pi(s) \leftarrow \argmax_a \sum_{s',r} p(s',r|s,a)[r+\gamma V(s')]\\
&\quad \text{If } oldAction \notin \{a_i\} \text{, which is the all equal best solutions from $\pi(s)$}\\
& \quad \text{then } policyStable \leftarrow false\\
&\text{If } policyStable \text{, then stop and return } V \approx v_* \text{ and } \pi \approx \pi_*\text{; else go to 2}
\end{align*}
\end{tcolorbox}

\begin{tcolorbox}[width=\textwidth,title={Policy Iteration (using iterative policy evaluation) for estimating $\pi \approx \pi_*$}]
1. Initialization

$\quad Q(s,a) \in\R$ and $\pi(s) \in A(s)$ arbitrarily $\forall s \in \mathcal{S}, a \in \mathcal{A}$

2. Policy Evaluation
\begin{align*}
&\text{Loop:}\\
&\quad\Delta \leftarrow 0\\
&\quad\text{Loop for each } s \in \mathcal{S}, a \in \mathcal{A}\\
&\quad\quad q \leftarrow Q(s,a)\\
&\quad\quad Q(s,a) \leftarrow \sum_{s',r} p(s',r|s,a)[r + \gamma \sum_{a'}\pi(a'|s')Q(s',a')]\\
&\quad\quad \Delta \leftarrow \text{max}(\Delta, |q-Q(s,a)|)\\
&\text{until } \Delta < \theta
\end{align*}

3. Policy Improvement
\begin{align*}
&policyStable \leftarrow true\\
&\text{For each } s \in \mathcal{S}, a \in \mathcal{A}\\
&\quad oldAction \leftarrow \pi(s)\\
&\quad \pi(s) \leftarrow \argmax_a Q(s,a)\\
&\quad \text{If } oldAction \notin \{a_i\} \text{, which is the all equal best solutions from $\pi(s)$}\\
& \quad \text{then } policyStable \leftarrow false\\
&\text{If } policyStable \text{, then stop and return } Q \approx q_* \text{ and } \pi \approx \pi_*\text{; else go to 2}
\end{align*}
\end{tcolorbox}

\begin{definition}
$\varepsilon$-soft, the probability of selecting each action in each state, is at least $\frac{\varepsilon}{A(s)}$
\end{definition}

\subsection{Value Iteration}

The policy evaluation of policy iteration requires multiple sweeps through the state set and convergence occurs only in the limit.
We want to truncate policy evaluation without losing the convergence guarantees of policy iteration. One of the technique is value iteration.

\begin{definition}
value iteration, stop policy evaluation after one sweep (one update after each state).

\begin{align*}
v_{k+1}(s) & \doteq max_a \E [R_{t+1}+\gamma v_k(S_{t+1})|S_t = s, A_t = a]\\
& = max_a \sum_{s',r} p(s',r|s,a)[r+\gamma v_k (s')], \forall s \in \mathcal{S}
\end{align*}
\end{definition}

\newpage
\section{Some Notes}

% Please add the following required packages to your document preamble:
\begin{table}[]
\begin{tabular}{@{}ll@{}}
\toprule
on-policy                       & off-policy                                                                                \\ \midrule
Agent can pick actions          & Agent can't pick actions                                                                                \\
most obvious setup :)           & \begin{tabular}[c]{@{}l@{}}learning with exploration\\ playing without exploration\end{tabular}                             \\
Agent always follows own policy & \begin{tabular}[c]{@{}l@{}}Learning from expert (expert is imperfect)\\ Learning from sessions (recorded data)\end{tabular} \\ \bottomrule
\end{tabular}
\end{table}

% Please add the following required packages to your document preamble:
% \usepackage{booktabs}
\begin{table}[]
\begin{tabular}{@{}lll@{}}
\toprule
             & on-policy                                                                                & off-policy                                                                          \\ \midrule
value based  & \begin{tabular}[c]{@{}l@{}}Monte Carlo Learning\\ TD(0)\\ SARSA\\ Expected SARSA\\ n-Step TD/SARSA\\ TD(\textbackslash{}lamda)\end{tabular} & \begin{tabular}[c]{@{}l@{}}Q-Learning\\ DQN\\ Double DQN\\ Dueling DQN\end{tabular} \\
policy based & \begin{tabular}[c]{@{}l@{}}REINFORCE\\ REINFORCE with Advantage\end{tabular}                                                                &                                                                                \\
actor-critic & \begin{tabular}[c]{@{}l@{}}A3C\\ A2C\\ TRPO\\ PPO\end{tabular}                                                                              & \begin{tabular}[c]{@{}l@{}}DDPG\\ TD3\\ SAC\\ IMPALA\end{tabular}                   \\ \bottomrule
\end{tabular}
\end{table}

\subsection{Gradient Descent vs. Gradient Ascent}

The gradient of a continuous function $f$ = the vector that contains the partial derivatives $\frac{\partial f(p)}{\partial x_i}$ computed at that point $p$.
The gradient is finite and defined if and only if all partial derivatives are also defined and finite.
The gradient formula:

$$ \nabla f(x) = [\frac{\partial f(p)}{\partial x_1}, \frac{\partial f(p)}{\partial x_2}, \cdots, \frac{\partial f(p)}{\partial x_{|x|}}]^T $$

When using the gradient for optimization, we can either conduct gradient descent or gradient ascent.

\paragraph{Gradient Descent}
Gradient Descent is an iterative process through which we optimize the parameters of a ML model. It's particularly used in NN, but also in logistic regression and support vector machines (SVM). It is the most typical method for iterative minimization of a cost function. Its major limitation consists of its guaranteed convergence to a local, not necesarily global, minimum.

A hyperparameter (i.e. pre-defined parameter) $\alpha$ (learning rate), allows the fine-tuning of the process of decent. In particular, we may descent to a global minimum.
The gradient is calculated with respect to a vector of parameters for the model, typically the weight $w$. In NN, the process of applying gradient descent to the weight matrix is called backpropagation of the error.

Backpropagation uses the sign of the gradient to determine whether the weights should increase or decrease. The sign of the gradient allows us to direction of the closet minimum to the cost function. For a given $\alpha$, we iteratively optimize the vector $w$ by computing
$$ w_{n+1} = w_n - \alpha \nabla_w f(w) $$
At step $n$, the weights of the NN are all modified by the product of the hyperparameter $\alpha$ times the gradient of the cost function, computed with those weights.
If the gradient is positive, then we decrease the weights; if the gradient is negative, then we increase the weights.

\paragraph{Gradient Ascent}
Gradient ascent works in the same manner as gradient descent. The only difference is that gradient ascent maximize functions (instead of minimization).
$$ w_{n+1} = w_n + \alpha \nabla_w f(w) $$

Gradient descent works on \textbf{convex functions}, while gradient ascent works on \textbf{concave functions}.

\paragraph{Summary}
\begin{itemize}
\item The gradient is the vector containing all partial derivatives of a function in a point
\item We can apply gradient descent on a convex function, and gradient ascent on a concave function
\item Gradient descent finds the nearest minimum of a function, gradient ascent finds the nearest maximum
\item We can use either form of optimization for the same problem if we can flip the objective function.
\end{itemize}

\paragraph{Gradient vs. Derivative}

\begin{itemize}
\item A directional derivative = a slope in an arbitrary specified direction.
\item A directional derivative is a rate of change of a function in any given direction.
\end{itemize}

\begin{itemize}
\item Gradient = a vector with slope of the function along each of the coordinate axes.
\item Gradient indicates the direction of \textbf{greatest} change of a funciton of more than one variable.
\item Gradient vector can be interpreted as the `direction and rate of fastest increase'.
\end{itemize}

\paragraph{Differential vs. Derivative}
\begin{itemize}
\item Differential is a subfield of calculus that refers to infinitesimal difference in some varying quantity
\item Differential represents an equation that contains a function and one or more derivatives of that function
\item The function which represents the relationsihp between the dependent and the independent variables is unknown
\end{itemize}

\begin{itemize}
\item The derivative of a function is the rate of change of the output value with respect to its input value
\item Derivative represent the instantaneous change in the dependent variable with respect to its independent variable
\item The function which represents the relationship between the variables is known
\end{itemize}

\paragraph{Loss/Cost/Objective function}

\begin{itemize}
\item \textbf{Loss function} is usually a function defined on a data point, prediction and label, and measures the penalty. For example: square loss in linear regression, hinge loss in SVM, 01 loss in theoretical analysis
\item \textbf{Cost function} is usually more general. It might be a sum of loss functions over all training set plus some model complexity penalty. For example: MSE and SVM cost function
\item \textbf{Objective function} is the most general term for any function that you optimize during training. For example, a probability of generating training set in maximum likelihood approach is a well defined objective function, but it is not a loss function nor cost function
\end{itemize}

We may say that, a loss function is a part of cost function which is a type of an objective function.

From wikipedia, in mathematical optimization and decision theory, a loss function or cost function (sometimes also called an error function) is a function that maps an event or values of one or more variables onto a real number intuitively representing some "cost" associated with the event. An optimization problem seeks to minimize a loss function. An objective function is either a loss function or its opposite (in specific domains, variously called a reward function, a profit function, a utility function, a fitness function, etc.), in which case it is to be maximized.

\paragraph{Surrogate vs. Approximation}

I think surrogate and approximation can be used interchangeably.

Surrogate loss function is used when the original loss function is inconvenient for calculation.

\bibliography{references.bib}

\end{document}


\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{hyperref}
\usepackage{amsmath}
\usepackage{graphics}
\usepackage{graphicx}
\usepackage{float}
\graphicspath{ {images/} }
\title{Starlings Murmuration Simulation}
\author{Sankalan Pal Chowdhury, Shreshth Tuli}
\date{April 2018}

\begin{document}

\maketitle

\section{Introduction}
Starlings are a species of small birds that show tendency to migrate in flocks. These flocks are called murmurations and can contain thousands of starlings. Due to their small size and coordinated movements, they can often put up very beautiful displays while moving. In this project, we try to mathematically model this behavior of theirs and run a Computer Simulation of the same
\section{Modeling the Starling}
We seek to model each bird as an individual agent which reacts to its surroundings in order to determine its flying direction. In the most simple model, its environment consists only of other birds. The bird must therefore negotiate three types of constraints in order to determine its flying direction: External forces, Internal forces and Personal limitations.
\subsection{External Forces}
External forces are the forces of mother nature which the bird act on the bird. We assume that the bird does not need to spend energy to generate these forces, and also cannot have any direct control over them. While there are many such forces, we seek to model only the major ones(this selection is similar to what is done for modeling airplanes):
\begin{itemize}
    \item \textbf{Gravity:} Gravity is perhaps the simplest yet most significant force to model. The concept of flight is fascinating essentially since it seems to overcome gravity.
    \item \textbf{Buoyancy:} Buoyancy may not be too important in air, but it is extremely easy to adjust within gravity. It can become significant for birds with very low density, so it cannot harm to model it.
    \item \textbf{Lift:} The  lift force is the main contributor towards overcoming gravity. In fact the bird can maintain its altitude only if the sum of Lift and Buoyancy is same as Gravity.
    \item \textbf{Drag:} The drag force arises due to air resistance which tries to prevent the motion of any body traveling through a fluid. But for this force, the bird could keep flying without applying any effort.
\end{itemize}
Birds, unlike airplanes can flap their wings in order to generate upward force. This force is more like a buoyant force, but needs the bird to spend energy to get it and can also control its magnitude to some extent. Modeling this requires a deep insight of the anatomy of the starlings which is beyond the scope of the model designers, therefore, we do not include it here.
\subsection{Internal Forces}
Once the external forces have been decided, the bird may decide to fly in a certain direction. It is somewhat misleading to call this decision a force, these are actually factors based on which the bird decides what kind of force to apply on itself(actually the bird applies the force on the environment and it gets it force by Newtons Third Law). These, however are all vector quantities and it is rather intuitive to consider them as components of the net force that the bird applies.

In the current model, we consider three forces, which are commonly used to simulate \href{https://en.wikipedia.org/wiki/Flocking_(behavior)}{flocking behaviour}:
\begin{itemize}
    \item \textbf{Cohesion:} The cohesive force is the attractive force that a bird experiences towards other members of the flock. It ensures that the flock stays together and does not start moving in random directions.
\begin{figure}[h]
\centering
\includegraphics[width=10cm]{cohesion600}
\end{figure}
\item \textbf{Separation:} The separative force acts opposite to the cohesive force and ensures that the birds keep a minimum distance between them and don't start colliding with each other
\begin{figure}[h]
\centering
\includegraphics[width=10cm]{separation600}
\end{figure}
    \item \textbf{Alignment:} This force tries to coordinate the direction of motion of the entire flock. Each bird has the urge to fly in the same direction as its neighbors.
\end{itemize}
\begin{figure}[h]
\centering
\includegraphics[width=10cm]{alignment600}
\end{figure}

The above mentioned factors present a somewhat idealistic view of how a certain bird desires to fly. However, there are other factors effecting how the bird really flies.

To start off, a bird is not likely to know about another bird which is flying right behind it. In fact, it can only see those birds which are in its \textbf{field of view}, ie the maximum angle from its line of motion which it can see. This is actually quit large for birds since they have eyes on the sides of their heads. Further, two birds which are far off from each other are less likely to affect each other than two birds that are close by. This phenomenon is modeled by the (near)\textbf{sightedness} of the bird. Finally, some birds may be more \textbf{adventurous} than others, giving them urges to fly off in random directions rather than those specified by the above forces.

Finally, a bird can apply only a limited amount of force, in which it would try to overcome the External forces and move in its desired direction. This limit would be determined both by an overall \textbf{maximum acceleration} and the individual \textbf{strength} of the bird.
\section{Mathematical Formulation}
To begin our mathematical formulation, we need to first fix a coordinate system. We fix the origin at the flock centroid(this is beneficial for bounding box creation during implementation) and the y axis upwards. The x and z axis are arbitrarily chosen since the system has cylindrical symmetry.

The mathematical values of the external forces are more or less standard and do not require much explanation
\begin{center}

    $$\vec{F_{gravity}}=\rho_{bird}V\vec{g}$$
    $$\vec{F_{buoyancy}}=-\rho_{air}V\vec{g}$$
    $$\vec{F_{lift}}=C_{lift}\rho_{air}\vec{v}^2W\hat{j}$$
    $$\vec{F_{drag}}=-\frac{1}{2} C_{drag}\vec{v}^2A\hat{v}$$
    $$\vec{F_{external}}=\vec{F_{gravity}}+\vec{F_{buoyancy}}+\vec{F_{lift}}+\vec{F_{drag}}$$

    $$\rho_{bird},\rho_{air} \text{ are the densities of the bird and air,}$$
    $$C_{lift}, C_{drag} \text{ are the lift and drag coefficients,}$$
    $$g \text{ is the acceleration due to gravity }=0\hat{i}-9.8\hat{j}+0\hat{k}$$
    $$W, A \text{ are wing area and cross sectional area respectively,}$$
    $$\vec{v} \text{ is the current velocity of the bird,}$$
    $$V \text{ is the volume of the bird}$$

\end{center}

The internal forces are more involved. To begin with, field of view simply filters out the birds which are beyond the specified angle, so all statements made hereafter are w.r.t. the rest of the birds.

\subsection{Internal Forces}
Two different approaches are possible to deal with the sightedness issue. One is to say that only a fixed number of nearest neighbors affect the bird. This partially relies on the fact that the bird would not want to over complicate its life and would approximate the world by a fixed number of birds around it. However, the evidence for this is quite empirical and needs to be judge in light of the fact that the number of birds in a certain region around the bird is held more or less constant by the Cohesion and separation forces. This also does not account for the chance that the bird might look at a very distant bird and adjust itself according to it.

The other view is to limit the distance till which a bird can see. This seems more logical, except for the fact that there is unlikely to be a hard boundary at a given distance. Therefore, we choose an exponential decay function to model the sightedness of the birds:
\begin{center}
$$W_i=e^{-sd_i}$$
\[
    d_i \text{ is the distance to the } i^{th} \text{ bird}
    s \text{is the sightedness of the bird}
\]
\end{center}
At this point, it may seem strange that a bird would look at like a thousand other birds before deciding which direction to move, which is clearly illogical. One way of explaining this would be that the bird looks at only a certain number of birds at each moment, and the quantity mentioned here is the over time expectation of looking at each bird.

This done, we can move on to the three forces. We calculate for each bird separately(call this the target bird and then aggregate to get the net forces.

It is intuitive to keep Alignment proportional to the velocity of the target bird. As for Separation and Cohesion, they should logically be proportional to the inverse of the distance between the two birds(Actually, cohesion can be constant, allowing for the sightedness to take care of the distance factor. Ignoring this fact is more of a personal intuition). Also, the separation force needs to grow faster that the cohesion force, as we want to prevent collisions. We chose to take the simplest expressions that satisfy these constraints.
\begin{center}
$$\vec{F_{cohesion,i}}=\frac{C_{coh}}{d_i} \hat{d_i}$$
$$\vec{F_{separation,i}}=-\frac{C_{sep}}{d_i^2} \hat{d_i}$$
$$\vec{F_{align,i}}=C_{ali}\vec{v_{i}}$$

$$\hat{d_i} \text{ is the unit vector towards the } i^{th} \text{ bird }$$
$$\vec{v_{targ}} \text{ is the velocity of } i^{th} \text{ bird }$$
$$C_{coh},C_{sep},C_{ali} \text{ are respective coefficients}$$
\end{center}

Once these forces are calculated due to each bird, we need to aggregate them to get net internal force. Aggregation can be done in many ways, but the two most common ways are summation and averaging. Summation is good for aggregating quantities that are likely to add up destructively. This is clearly true for the separation force. Averaging, on the other hand, make sense for quantities that would add up constructively, like the alignment force. Cohesion force can go either way, but it is somewhat safer to consider averaging since a zero is less dangerous than an infinity. The Weights assigned to the different birds is taken care of here.

\begin{center}
    $$\vec{F_{cohesion}}=\frac{\sum_iW_i\vec{F_{cohesion,i}}}{\sum_iW_i}$$
    $$\vec{F_{alignment}}=\frac{\sum_iW_i\vec{F_{alignment,i}}}{\sum_iW_i}$$
     $$\vec{F_{separation}}=\sum_iW_i\vec{F_{separation,i}}$$
\end{center}

Finally, we need to add the adventure component to these forces to calculate the net internal force on the bird. Now, while an adventurous bird may not want to align itself with the other birds or get attracted to them, it will certainly not want to collide with other birds. Therefore, the random component will only replace the alignment and cohesion forces and not the separation force.
\begin{center}
    $$\vec{F_{internal}}=\vec{F_{separation}}+\frac{1000-Adv}{1000}(\vec{F_{cohesion}}+ \vec{F_{alignment}})+\frac{Adv}{1000}\vec{R}$$

    $$Adv \in \{0,...,1000\} \text{ is the adventurousness parameter }$$
    $$\vec{R} \text{ is a suitable random vector(suitability depends on range of magnitudes)}$$
\end{center}
\subsection{Putting it Together}
Now that we have both the external and internal force vectors, we need to put them together to calculate the acceleration, velocity and power of the bird. It is here that we need to take into account the maximum force that the bird can apply is limited.

If the bird was an inanimate object, it would simply drift in the direction of the external force. What the bird really does is try its best to change this to its desired direction of flight as much as possible. This difference is simply given by $\vec{v_{diff}}=\vec{v}+\vec{F_{external}}-\vec{F_{internal}}$. Since we can only take discrete time, the desired acceleration is $\vec{a_{desired}}=\vec{v_{diff}}/P$ where P is the clock period. The magnitude of  $\vec{a_{desired}}$ needs to be bounded by a $a_{bound}=a_{max} \times strength$ where $a_{max}$ is the global maximum acceleration and strength is a property of the bird.\newline

The bounding mechanism itself is somewhat non trivial. The naive way of doing it would be to cut off any value higher that the required bound. But that would be very unrealistic as in a real scenario, the bird would start facing difficulty gradually till it reaches a point where it can no longer fly faster.
A good function to approximate this is the $tanh$ function, which has derivative of 1 around zero, but is bounded by 1 on the positive side. However, its derivative never falls to 0 for any finite parameter.

Once the bounding is done, we can calculate Power easily using laws of physics.
\begin{center}
    $$\vec{a_{final}}=tanh(\frac{|\vec{a_{desired}}|}{a_{bound}})\frac{a_{bound}}{|\vec{a_{desired}}|}\vec{a_{desired}}$$
    $$\vec{v_{final}}=\vec{v}+\vec{v}+\vec{F_{external}}-\vec{a_{final}}$$
    $$Power=m\vec{a_{final}}.\vec{v_{final}}$$
\end{center}
Position and Energy follow trivially.

\pagebreak
\section{Implementation}
Having designed the model, we attempt to implement it on a computer. The aim of this part is to 1) Be able to visualize the murmurations of the starlings on a computer screen, and 2) approximate some statistics about these murmurations.

The implementation needs to proceed in two parts. First, we look at what kinds of deviations must be made from the original model in order to be able to simulate it. Second, we choose a framework to implement the design. Both of these are dependent on the characteristics of the model and/or the constraints set by the programming environment.
\subsection{Deviations from the Model}
One of the major issues we need to deal with here is the fact that in real life, Starlings are free to move anywhere in the open sky. However, our simulation only lets us look at the starlings from a fixed viewpoint, and there is little point in having the murmurations happening out of the visible region. There are multiple options to solve this:
\begin{itemize}
    \item \textbf{Moving the Viewpoint:} This approach tries to approximate the real motions of the birds as best as possible, by moving the viewpoint automatically along with the birds. The main advantage of this is the model does not need to be tampered with. However, there can be scenarios in which the motion of the view point would cancel out the motion of the birds, making them appear to be static. Also, the flock may split into two, leading to issues on deciding the viewpoint. Alternatively, the shifting of viewpoint can be left upto the user, but that is probably not very user friendly.
    \item \textbf{A Circular World:} To give the feeling of an infinite space, we can make the edges of the view wrap around to the opposite edge. This also would require minimal changes to the model, but its physical effect can hardly be explained. Also, it is not logical anyway to let the $-y$ direction extend to infinity.
    \item \textbf{Adding Imaginary Force}: To prevent the birds from escaping into infinity, we can define a potential like field that forces the birds to remain in a specified region, beyond which this field becomes too large. This can represent pressure zones or simply unfamiliar territory for the birds. However, if not tuned properly, such a field can cause the birds to actually divert to infinity.
    \item \textbf{Limiting Position Directly:} This would be similar to what was done for limiting the maximum acceleration, forcing the birds to stay in a fixed region. The chief downsides of this is the lack of a physical explanation and the fact that the birds might all want to escape in one direction, but get restricted by this force, making them appear as if they are all at the same point.
\end{itemize}
In our current implementation, we choose to go with a combination of the last two approaches, which tend to cancel out some of each others negative points. For the force field, we choose a force that is zero in some region around the origin and then grows as the square root of distance from origin. This choice is kind of empirical, with comparisons performed over multiple regularized polynomial fields. The modified set of equations are given below:
\begin{center}
    $$\vec{v_{positional}}=-\vec{P} * max(\sqrt{\frac{|\vec{P}|}{P_{max}}}-\sqrt{5},0)$$
    $$\vec{v_{diff}}=\vec{v}+\vec{F_{external}}-\vec{F_{internal}}-\vec{v_{positional}}$$
    $$\vec{P_{final}}=tanh(\frac{|\vec{P_{desired}}|}{20*P_{max}})\frac{20*P_{max}}{|\vec{P_{desired}}|}\vec{P_{desired}}$$
    P \text{represents position of the bird, no subscript means current position}
\end{center}
The other equations remain unchanged.
Apart from this, it turns out that the exponential sightedness model can sometimes become too much to calculate(it needs to be done O($n^2$) times in each time step, where n is the number of birds). We therefore keep an alternative of using a constant viewing range(ie a step function). In the same vein, all the other internal forces are also made toggle-able.

\subsection{Frameworks and Design Decisions}
While it has not been stated previously, some decisions are implicit to our model:
\begin{itemize}
    \item Since every starling has been modeled as an entity, and \textbf{Object Oriented Paradigm} is necessary to implement these.
    \item there is a stochastic dimension to this simulation. Every calculation therefore need to be performed on a clock edge.
    \item Interaction between different birds is purely observational, so the all the computations are parallelisable. This makes our program fit for multi-threading.
\end{itemize}
\subsubsection{The Clock}
While it is clear that a clock needs to be used, there are actually multiple options on how the clock gets used. The simplest way to go about it is to use a \textbf{real clock} which calculates the position and updates the display in real time. In such a setting, there would be a trade off between the frequency of the clock and the number of birds. The advantage, however is that no storage needs to be done in order to create a realistic display.

The alternative to this approach is to use a \textbf{logical clock} which can proceed at much slower pace than the actual clock, which only influences the display. This means that we can slow down the Logical clock arbitrarily, allowing us to simulate as many birds as we like(well, kind of). In such an approach, the user cannot dynamically set parameters or visualize what the current setting looks like. The main problem here though, is the fact that the positions of each bird needs to be saved into a file, and this file can get huge.

Suppose we store the position and velocity of each bird at each instant as vectors of single precision floats. If we use a 50 hz clock for 2000 birds for 10 minutes we end up with $50*6*1000*30*32=1.152 Gb$ of data. Note that this data would not make any sense to a human, and if we want to use a text file, the file size blows up even more. Another consideration here is that the display of the birds itself takes up significant computational power, and that needs to be done on a real clock anyway.

Therefore, we choose to stick to a real clock
\subsubsection{Order of Computation with Multi-threading}
When talking of objects, the ideal model is that all instances of an object run their computation concurrently. This would also be closest to the real world.

In a real programming paradigm, however, to run everything concurrently is not possible(unless we have as many logical cores as birds). The closest we can get is to give each bird its own thread, but that would mean a huge number of context switches, and therefore huge slowdown. Intuitively, since the steps taken by each bird in each tick of the clock would be rather small, imposing a fixed order should not harm the model too much.

In our implementation, we use 4 threads(this number can be increased depending on how many cores are available) and within each thread, we provide two options: In one, we impose a fixed order of execution amongst the birds, while in the other, we randomize within each thread. For the non-random approach, a simple segregation scheme is used viz that from the vector of birds thread $t$ gets birds which have index $i$ such that $i\,\%\,4\,=\,t$. Order between the threads is always random, as long as at least 4 cores are available. So there are two levels of randomization that this approach provides:
\begin{enumerate}
\item The inherent scheduler of the operating system and the seemingly random distribution of load on the physical cores, randomize the order in which the computation is done for each bird.
\item An array of numbers 1 to flock size is shuffled and divided into 4 sub-arrays. Each thread gets a sub-array and uses it's values as indices for execution of the computation, thus making it random.
\end{enumerate}

Although the approach used still does not allow concurrent calculation for each bird (which is a limitation of the hardware) but provides the best way to model the system with finite logical cores.
\begin{figure}[H]
\centering
\includegraphics[width=10cm]{Flock_threading}
\caption{Multi-threading flowchart with randomization}
\end{figure}

\subsubsection{Selected tools}
The calculation is implemented in \textbf{C++} because it is perhaps one of the most widely used, rapidly evolving and fastest language for Object Oriented Programming. The display part is done in \textbf{OpenGL} which is said to be the \textit{industry standard for high performance graphics}. Other platforms like $Unity$ and other similar physics engines have not been used as they do not allow tuning at the root level even though they allow users to model the problem much more easily.
\section{Evaluation and Parameter Tuning}
Once we have implemented our design, we need to tune the hyper-parameters in order to get realistic displays. But before that, we need to have some metrics to evaluate the performance.
\subsection{Metrics for Correctness}
In the age of big data and machine learning, one is always talking of differentiable and convex cost functions. Luckily or unluckily, these things have not yet invaded the lives of the starlings, so we need to have heuristic measures of correctness.

The best judgment is obviously given by visualization of the simulation, to judge whether it is realistic or not. However, is is normally better to have a more quantitative measures. Following are some such possibilities:
\begin{itemize}
	\item The most important correctness indicator is the \textbf{visual effect} created by the starlings when in motion. A low number of birds may not give such an effect so relaxation should be for starlings when they are less than say 100.
    \item The starlings have a diet consisting fruits and insects. Based on this, we can estimate how much Energy they can actually spend in flying. Thereafter, we can check if the \textbf{Average Energy} and the \textbf{Maximum Energy} spent by the starling is around this value or not
    \item Based on the energy consumption in a finite time like 10 minutes, the amount of \textbf{food intake} should be less than the typical food intake of a starling in a day which is around 10g. Such measurements are from online sources like \href{https://www.tandfonline.com/doi/pdf/10.1080/00063657309476384}{this}.
    \item The starlings can withstand only so much \textbf{force} on themselves. If it becomes too high, the will probably get squashed.
    \item \textbf{Collisions} never take place in the real setting, so even if two birds get too close, we should rethink our model.
    \item The starlings can not apply a large amount of power while in motion, so the parameters \textbf{Average Power} and \textbf{Maximum Power} should be in an acceptable range.
\end{itemize}
Beyond these, there are things like maximum velocity and acceleration, but that have already been accounted for explicitly in our model, so it makes little sense in measuring them. Finally, we should also see what is the maximum number of birds that we are able to simulate, as that is definitely a measure of performance.

\pagebreak

\subsection{Tuning the Hyper-parameters}

Tuning the hyper-parameters is crucial for a more realistic simulation. Although, these parameters can be determined by complex calculations and analysis of the birds anatomical structure and the cognitive model. This becomes much more complicated when such functions are to be mapped for discrete time. A not so precise but more effective approach would be to augment the model parameters by brute-force and intelligent guesses based on the metrics of correctness as discussed earlier. This approach leads to errors in the model, but the deviation from realism is also because of modeling constraints as discussed. These approximations and relaxations allow us to take such a decision for parameter tuning. \\
For the internal forces, the parameter tuning has been based on a intuitive notion of prioritizing the various forces: separation, alignment and cohesion. As separation must dominate at lower distances and cohesion at higher, it is important that the constants used for cohesion are higher than that of separation but the dependence with distance must rise faster in the latter case. For realistic simulation, it is also important to give some randomness to the forces considering that these random forces do not dominate over the systematic ones used in the model. As these forces are based on a factor that exponentially decays or as a step function with distance, there constants must be considering the distances as well. \\
For the external forces including lift, drag, buoyancy and lift some physical aspects need to be investigated. The cross section area and the drag coefficient in the real scenario change dynamically but for the ease of implementation and their somewhat low effect on the model have been neglected. It is important to observe that the drag force depends on the airflow around the bird. A turbulent airflow is expected most of the times due to the flock around it. Thus, we use a drag coefficient somewhat in between but more the side of that for turbulent air. The decision as to how much the air is turbulent or laminar is based on quantitative data of the Reynolds number on the internet as on \href{https://aip.scitation.org/doi/abs/10.1063/1.4807064}{"this"}  link, and empirical observations.

\section{Conclusion}
In this project we have been able to approximately simulate the murmurations of starlings and also calculate some essential parameters of the motion. The fact that our outputs actually look realistic is definitely evidence in favour of the fact that the real model of flight is somewhat similar to the one that we have defined. This is therefore also a step towards understanding the behavior of starlings.

Finally, our model contains a lot of hyper parameters, thus a lot of flexibility. In future, should someone be able to track the actual motion of starlings, the birds can also be made \textit{Reinforcement Learning} agents.
\end{document}


\subsection{LFSR}

Very primitive peripheral containing free running shift register with
linear feedback that can be used for generating pseudo random numbers.

\subsubsection{Function}

You simply read data register and that is. There is nothing to set-up.
Feedback is $x^{32} + x^{22} + x^{2} + x^{1} + 1$, that is sequence
with 4294967295 states.

\subsubsection{Registers}

There is only one register, called LFSR\_RN, and it have offset $+0$. It
is read only and contain pseudonymous number.

\begin{table}[h]
    \centering
    \begin{tabular}{|l|l|l|}
        \hline
        \textbf{Offset} & \textbf{Name} & \textbf{Purpose}                 \\ \hline
        $+0$            & PSBR          & Last received byte from PS2 bus. \\ \hline
    \end{tabular}
    \caption{LFSR register map}
    \label{tab:lfsr_reg_map}
\end{table}


\newpage{}

\hypertarget{a003---10-categories}{%
\section{A003 - 10 categories}\label{a003---10-categories}}

\hypertarget{description}{%
\subsection{Description}}

To be better than A001 we could use a categorization.
For that we divide all tasks into 10 groups with similarities.
Our target here is to reduce the dispersion within those groups. E.g. we know that there are a lot of tasks below half an hour duration. A001 often claims they are more costly. If we could clearly say that a task would belong to that category we could reduce our error margin here.
But how can we accomplish that?

\subsubsection{A003.1}

We divide our tasks using a linear scale
\begin{enumerate}
\tightlist
\item smaller than 30 minutes,
\item bigger than 30 minutes and less than one hour,
\item bigger than 1 hour and less than 2 hours
\item bigger than 2 hour and less than 3 hours
\item bigger than 3 hour and less than 4 hours
\item bigger than 4 hour and less than 5 hours
\item bigger than 5 hour and less than 6 hours
\item bigger than 6 hour and less than 7 hours
\item bigger than 7 hour and less than 8 hours
\item bigger than 8 heures
\end{enumerate}

Our algorithm tries to identify significant words that are unique par category:
\begin{enumerate}
\tightlist
\item collect all words in a category
\item for all categories:
    \begin{enumerate}
        \tightlist
        \item remove all words that are somehow mentioned in another category
    \end{enumerate}
\end{enumerate}