\section{Classification Performance}
\label{sec:verification_and_benchmark:classification_performance}

The performance of the floating-point \acrshort{cnn} model is verified separately from that of the quantized model.
Afterwards they are compared with each other.

Measuring the performance of a \acrshort{cnn} model requires the definition of a suitable function to quantify the classification accuracy.
Therefore, the Top-$K$ function is specified in a first step.

Furthermore, the determination of the overall accuracy of the models is not sufficient.
The fact that some classes might perform worse than others would not be noticed.
The individual Top-$K$ accuracies of the different classes are therefore also evaluated.
Additionally, the overall accuracy of a test dataset containing only images in which the object is fully visible is computed.

% ------------------------------------------------------------------------------------------------------------------------------
\subsection{Top-K}
\label{subsec:verification_and_benchmark:classification_performance:topk}

A common way to quantify the performance of a single prediction is its Top-$K$ value.
In general, Top-$K$ means that the element with the $K$-th highest probability of the prediction vector corresponds to its true label.
Consider, for example, a prediction with a Top-$K$ value of two.
This means that the element with the second highest probability of the prediction vector corresponds to its true label.
If two or more elements of the prediction vector feature the same probability, there are two or more possible Top-$K$ values.
In such a case, the worst (highest) of the possible Top-$K$ values is chosen.
Otherwise, an equal distribution of probabilities would lead to a Top-1 accuracy, although in reality it is a random guess.

To measure the performance of a \acrshort{cnn} model, the Top-$K$ accuracy is used to quantify the classification accuracy.
The Top-$K$ accuracy is a vector with as many elements as there are classes.
Each element $k$ corresponds to the percentage of the predictions which feature a Top-$K$ value of $k$ or better (lower).

The implementation uses the NumPy function \texttt{np.unique} with the parameter \texttt{return\_counts} set to \texttt{True}, which returns two arrays.
The first array contains the unique elements of the input prediction array sorted in ascending order.
The second array contains the number of occurrences of each of the values from the first array \cite[inf_numpy_unique].
The next step involves finding the position of the probability in the first array which corresponds to the probability at the index of the label in the input array.
The last step is to sum up the elements of the second array from the previously found position to the end of the array.
This value corresponds to the Top-$K$ value of a particular prediction.
All Top-$K$ values of all predictions are then used to determine the Top-$K$ accuracy of the input predictions.
Listing \ref{lst:topk} shows the implementation of the \texttt{top\_k} function in the \texttt{fhnwtoys} package.

\begin{lstlisting}[style=python, caption={Implementation of the Top-K function}, label=lst:topk]
def top_k(predictions, labels, num_classes):
  num_predictions = len(predictions)

  top_k = np.zeros((num_classes,), dtype=np.float64)

  for pred, label in zip(predictions, labels):
    unique_v, unique_c = np.unique(pred, return_counts=True)
    pos = np.asarray(unique_v == pred[label]).nonzero()[0][0]
    idx = sum(unique_c[pos:]) - 1 # top k - 1 (-1: array index)
    top_k[idx:] += 1

  return top_k / num_predictions
\end{lstlisting}

% ------------------------------------------------------------------------------------------------------------------------------
\subsection{Training}
\label{subsec:verification_and_benchmark:classification_performance:training}

The computation of the classification accuracy of the floating-point model is implemented in the Python script \texttt{cnnv.py}.
In a first step, the floating-point model is used to generate predictions for all of the samples in the test dataset.
These predictions are stored in a NumPy array, which is then used to compute the overall Top-$K$ accuracy of the test dataset.
Additionally, the Top-$K$ accuracies of the individual classes are also calculated.

The overall Top-$K$ accuracy of the floating-point model is listed in table \ref{tab:overall_top_k_training} and the Top-$K$ accuracies of the individual classes are listed in table \ref{tab:individual_top_k_training}.

Analyzing the two tables shows that the usage of a test dataset containing only images in which the object is fully visible leads to a neglectable improvement of about \SI{0.3}{\percent}.
Furthermore, it can be seen that the individual classes perform about equally well.

\begin{table}[b]
  \caption{Overall Top-$K$ accuracy of the floating-point model with $K = 1, 2, \dots, 5$}
  \label{tab:overall_top_k_training}
  \centering
  \begin{tabular}{llllll}
    \toprule
     & \textbf{Top-1} & \textbf{Top-2} & \textbf{Top-3} & \textbf{Top-4} & \textbf{Top-5} \\
    \midrule
    \textbf{Entire Dataset} & \num{0.995} & \num{0.999} & \num{0.999} & \num{1.000} & \num{1.000} \\
    \textbf{Fully Visible Only} & \num{0.998} & \num{1.000} & \num{1.000} & \num{1.000} & \num{1.000} \\ % the last three values are not rounded
    \bottomrule
  \end{tabular}
\end{table}

\begin{table}
  \caption{Individual Top-$K$ accuracies of the floating-point model with $K = 1, 2, \dots, 5$}
  \label{tab:individual_top_k_training}
  \centering
  \begin{tabular}{llllll}
    \toprule
    \textbf{Object} & \textbf{Top-1} & \textbf{Top-2} & \textbf{Top-3} & \textbf{Top-4} & \textbf{Top-5} \\
    \midrule
    \textbf{Nerf Dart} & \num{1.000} & \num{1.000} & \num{1.000} & \num{1.000} & \num{1.000} \\
    \textbf{American Football} & \num{0.995} & \num{0.999} & \num{0.999} & \num{0.999} & \num{1.000} \\
    \textbf{Table Tennis Ball} & \num{1.000} & \num{1.000} & \num{1.000} & \num{1.000} & \num{1.000} \\
    \textbf{Shuttlecock} & \num{0.999} & \num{1.000} & \num{1.000} & \num{1.000} & \num{1.000} \\
    \textbf{Sporf} & \num{1.000} & \num{1.000} & \num{1.000} & \num{1.000} & \num{1.000} \\
    \textbf{Arrow} & \num{1.000} & \num{1.000} & \num{1.000} & \num{1.000} & \num{1.000} \\
    \textbf{Hand Featherball} & \num{0.993} & \num{1.000} & \num{1.000} & \num{1.000} & \num{1.000} \\
    \textbf{Floorball} & \num{0.996} & \num{0.998} & \num{0.999} & \num{1.000} & \num{1.000} \\
    \textbf{Spiky Ball} & \num{1.000} & \num{1.000} & \num{1.000} & \num{1.000} & \num{1.000} \\
    \textbf{Tesafilm} & \num{1.000} & \num{1.000} & \num{1.000} & \num{1.000} & \num{1.000} \\
    \textbf{Sponge} & \num{0.986} & \num{0.995} & \num{0.998} & \num{0.999} & \num{0.999} \\
    \textbf{Red Duplo Brick} & \num{0.989} & \num{0.999} & \num{1.000} & \num{1.000} & \num{1.000} \\
    \textbf{Green Duplo Brick} & \num{0.983} & \num{0.998} & \num{0.999} & \num{1.000} & \num{1.000} \\
    \textbf{Duplo Figure} & \num{0.997} & \num{0.999} & \num{0.999} & \num{1.000} & \num{1.000} \\
    \textbf{Foam Die} & \num{0.995} & \num{0.997} & \num{0.997} & \num{0.997} & \num{0.997} \\
    \textbf{Infant Shoe} & \num{0.991} & \num{0.999} & \num{0.999} & \num{0.999} & \num{0.999} \\
    \textbf{Stuffed Bunny} & \num{0.997} & \num{0.997} & \num{0.997} & \num{0.997} & \num{0.998} \\
    \textbf{Goalkeeper Glove} & \num{0.987} & \num{1.000} & \num{1.000} & \num{1.000} & \num{1.000} \\
    \textbf{Hemp Cord} & \num{0.994} & \num{0.999} & \num{0.999} & \num{1.000} & \num{1.000} \\
    \textbf{Paper Ball} & \num{0.996} & \num{0.997} & \num{0.997} & \num{0.999} & \num{0.999} \\
    \textbf{Beer Cap} & \num{0.989} & \num{1.000} & \num{1.000} & \num{1.000} & \num{1.000} \\
    \textbf{Water Bottle} & \num{1.000} & \num{1.000} & \num{1.000} & \num{1.000} & \num{1.000} \\
    \bottomrule
  \end{tabular}
\end{table}

% ------------------------------------------------------------------------------------------------------------------------------
\subsection{Inference}
\label{subsec:verification_and_benchmark:classification_performance:inference}

The computation of the classification accuracy of the quantized model is also implemented in the Python script \texttt{cnnv.py}.
However, the size of the entire test dataset is too large to fit on the SD card.
For this reason the Python script \texttt{inference\_predictions.py} is used to generate all predictions in arbitrary sized batches and store them in a NumPy array.
This array, which contains all predictions, can then be exfiltrated and used to compute the overall Top-$K$ accuracy of the test dataset.
In addition, the Top-$K$ accuracies of the individual classes are also calculated for the quantized model.

The overall Top-$K$ accuracy of the quantized model is listed in table \ref{tab:overall_top_k_inference} and the Top-$K$ accuracies of the individual classes are listed in table \ref{tab:individual_top_k_inference}.

The analysis of these two tables shows that the usage of a test dataset containing only images in which the object is fully visible leads to a significant improvement of the overall Top-$K$ accuracy of about \SI{1.3}{\percent}.
Furthermore, it is noticeable that some of the classes perform somewhat worse than others.

\begin{table}
  \caption{Overall Top-$K$ accuracy of the quantized model with $K = 1, 2, \dots, 5$}
  \label{tab:overall_top_k_inference}
  \centering
  \begin{tabular}{llllll}
    \toprule
     & \textbf{Top-1} & \textbf{Top-2} & \textbf{Top-3} & \textbf{Top-4} & \textbf{Top-5} \\
    \midrule
    \textbf{Entire Dataset} & \num{0.972} & \num{0.983} & \num{0.989} & \num{0.993} & \num{0.995} \\
    \textbf{Fully Visible Only} & \num{0.986} & \num{0.992} & \num{0.996} & \num{0.998} & \num{0.999} \\
    \bottomrule
  \end{tabular}
\end{table}

\begin{table}
  \caption{Individual Top-$K$ accuracies of the quantized model with $K = 1, 2, \dots, 5$}
  \label{tab:individual_top_k_inference}
  \centering
  \begin{tabular}{llllll}
    \toprule
    \textbf{Object} & \textbf{Top-1} & \textbf{Top-2} & \textbf{Top-3} & \textbf{Top-4} & \textbf{Top-5} \\
    \midrule
    \textbf{Nerf Dart} & \num{0.999} & \num{0.999} & \num{0.999} & \num{1.000} & \num{1.000} \\
    \textbf{American Football} & \num{0.971} & \num{0.977} & \num{0.983} & \num{0.991} & \num{0.993} \\
    \textbf{Table Tennis Ball} & \num{0.999} & \num{0.999} & \num{1.000} & \num{1.000} & \num{1.000} \\
    \textbf{Shuttlecock} & \num{0.991} & \num{0.993} & \num{0.993} & \num{0.994} & \num{0.996} \\
    \textbf{Sporf} & \num{0.995} & \num{0.998} & \num{0.999} & \num{0.999} & \num{1.000} \\
    \textbf{Arrow} & \num{0.994} & \num{0.995} & \num{0.997} & \num{0.999} & \num{0.999} \\
    \textbf{Hand Featherball} & \num{0.943} & \num{0.964} & \num{0.980} & \num{0.989} & \num{0.999} \\
    \textbf{Floorball} & \num{0.990} & \num{0.995} & \num{0.996} & \num{0.996} & \num{0.996} \\
    \textbf{Spiky Ball} & \num{0.999} & \num{0.999} & \num{0.999} & \num{0.999} & \num{0.999} \\
    \textbf{Tesafilm} & \num{0.998} & \num{0.999} & \num{0.999} & \num{0.999} & \num{0.999} \\
    \textbf{Sponge} & \num{0.955} & \num{0.973} & \num{0.985} & \num{0.989} & \num{0.992} \\
    \textbf{Red Duplo Brick} & \num{0.956} & \num{0.984} & \num{0.993} & \num{0.999} & \num{0.999} \\
    \textbf{Green Duplo Brick} & \num{0.966} & \num{0.978} & \num{0.987} & \num{0.992} & \num{0.995} \\
    \textbf{Duplo Figure} & \num{0.946} & \num{0.963} & \num{0.972} & \num{0.978} & \num{0.983} \\
    \textbf{Foam Die} & \num{0.989} & \num{0.991} & \num{0.994} & \num{0.995} & \num{0.997} \\
    \textbf{Infant Shoe} & \num{0.897} & \num{0.927} & \num{0.948} & \num{0.969} & \num{0.976} \\
    \textbf{Stuffed Bunny} & \num{0.991} & \num{0.991} & \num{0.992} & \num{0.993} & \num{0.993} \\
    \textbf{Goalkeeper Glove} & \num{0.912} & \num{0.941} & \num{0.958} & \num{0.973} & \num{0.981} \\
    \textbf{Hemp Cord} & \num{0.971} & \num{0.990} & \num{0.995} & \num{0.997} & \num{0.997} \\
    \textbf{Paper Ball} & \num{0.974} & \num{0.983} & \num{0.989} & \num{0.993} & \num{0.997} \\
    \textbf{Beer Cap} & \num{0.966} & \num{0.988} & \num{0.994} & \num{0.997} & \num{0.998} \\
    \textbf{Water Bottle} & \num{0.988} & \num{0.995} & \num{0.997} & \num{0.998} & \num{0.999} \\
    \bottomrule
  \end{tabular}
\end{table}

% ------------------------------------------------------------------------------------------------------------------------------
\subsection{Comparison}
\label{subsec:verification_and_benchmark:classification_performance:comparison}

As expected, the classification performance of the quantized model is slightly worse than that of the floating-point model.
While the floating-point model features a Top-1 accuracy of about \SI{99.5}{\percent}, the Top-1 accuracy of the quantized model is only about \SI{97.2}{\percent}.
This corresponds to a drop in accuracy of about \SI{2.3}{\percent} due to the quantization process.
However, since a throw usually creates more than one frame, the average of the probabilities can be used.
This reduces the impact of a misclassified frame significantly.

More interestingly is the fact that the entire test dataset performs worse than a modified test dataset where the images in which the object is only partially visible were removed.
The Top-1 accuracy of the quantized model is increased by about \SI{1.3}{\percent} when the modified dataset is used.

For this reason, it is highly beneficial to weight the frames according to their occurrence in the throw (see section \ref{subsec:inference:app:weighting}).
Frames from the beginning of the throw, where the object is only partially visible, are weighted less than frames from the middle of the throw, where the object is most likely fully visible.


\chapter{More Advanced Error Analysis}

\section{Objectives}

\begin{enumerate}
    \item To understand the concept of the random error.
    \item To study the propagation of errors from measured to derived quantities.
\end{enumerate}

\section{Introduction}

In the first chapter on errors, in the section on errors on time measurement, we found that the error on time measurement in our lab is almost certainly dominated by the reaction time of the observer. And, to determine this error, one method we suggested was to determine the time period many times and find the spread in the readings, using a measure called the standard deviation. The standard deviation is used in statistics to measure the characteristic spread of a random variable. The question then arises: What is a random variable? We will investigate this question in this chapter.

The other matter we will address ourselves to in this chapter is that of propagation of errors. You will have noticed that in the earlier chapter on errors, we spoke at length on the errors on the observations themselves, e.g. the length $l$, or obtained by a simple operation from the observations, e.g. the time period $T$, but we said nothing about how this may affect quantity derived from these observations, e.g. the acceleration due to gravity $g$. It is clear that how certain we are of the time period and the length will affect how certain we are of $g$. In other words the errors on $l$ and $T$ will \textit{propagate} to $g$. How error propagates from observed quantities to derived quantities is the second subject of this chapter.

\section{Random Errors}

To understand this, let us imagine trying to make a large number of tiles, by a manually operated machine. It is clear that though the tiles will be similar to each other, no two will be identical. The differences will be of various kinds, appearance, mass, edge length, etc. Since we want to look at the tiles through the eye of a physicist, let us concentrate on measurable quantities; we could choose edge length or mass; let us choose mass.

The mass of a tile is what we call a \textit{random variable}, because the mass of one tile is different from that of another in a random manner (in spite of the template which tends of make them similar). If we want to find the characteristic mass of the tiles, we can weigh a number of them and find the average. However, what we are after is how different the tiles are from each other; so the mean won't do. What we could do, to begin with, is weigh a large number of them, and note them down. If we look carefully at the numbers, we will see that there is a spread. We see that numbers close to each other don't indicate much of a difference, and we'd like to think of them together. A powerful way to represent neighbouring readings together while highlighting the difference between distant readings is to draw a histogram.

\subsection{Histogram}

\begin{figure}[!htb]
    \centering
    \includegraphics[scale=0.75]{figs/data-gaussian.png}
    \caption{A typical histogram, for 1200 readings of the heights of individuals. The bell-like shape of the pattern is characteristic. It is described by a famous mathematical function called the Gaussian. }
    \label{fig:data-gaussian}
\end{figure}

In a histogram, we put the readings in a number of bins, i.e. we put all the readings in a certain range in the same bin, and we have a number of such bins to cover all the readings. There is no formula for the number of bins, but what we want a large-enough number of bins to see how the population differs from that in the others as we move across the readings, i.e. we want to see the pattern in the variation of the readings.

What we find is that whenever the variation from one reading to another is caused by a large number of factors unconnected with each other, this shape arises naturally. This is true in our example of the masses of a collection of tiles made with the same template. It is also true of the masses of group of human beings belonging to the same population, e.g. urban Punjabis.

It is clear from the way we have arrived at the histogram that its width is determined by the spread of the readings. The mean does not contain information on this width; in fact what we are looking for is precisely how far the readings are, on the average, from the mean. If we imagine two artisans making the tiles, one a master who is able to achieve greater consistency and the other an apprentice, it ought to be clear that the histogram of the masses of the master's tiles will be narrower than that of the apprentice's tiles.

One way to characterise the spread of the masses might be to find the difference in mass between the heaviest tile and lightest tile. But there is something unsatisfactory about this because we are allowing the outliers to characterise and entire. What we would like to do is use all the readings to characterise the spread. This is done by the \textit{standard deviation} $\sigma$, which is given by the formula

\begin{equation}
    \sigma = \sqrt{\frac{\sum_{i=1}^{n} (x_i - \overline{x})^2}{n}},
\end{equation}

where $x_i$ is a reading, $\overline{x}$ is the mean, and $n$ is the number of readings.

For a Gaussian distribution of readings, i.e. a distribution the envelope of whose histogram is a Gaussian, about $68 \%$ of the readings are found between $x = \overline{x} - \sigma$ and $x = \overline{x} + \sigma$. Thus an arbitrary reading has a 68\% likelihood that an individual measurement will fall within one standard deviation $(\pm\sigma_x)$ of the mean.\footnote{Keep in mind, however, the example of superluminal neutrinos we gave earlier: in that case, the error was \textit{assumed} to be random, and their precise data led them to a false positive. The true error turned out to be systematic, leading to \textit{different (and larger) error bars}!}

\begin{question}
    \paragraph{Question:} In the last paragraph we have gone from the fraction of readings in a certain range to a probability of an arbitrary reading's having a certain value. Go through this carefully and understand the argument.

    \paragraph{Question:} Do some research to find out what fraction of the readings are found in the range $\pm 2\sigma$ and $\pm 3 \sigma$ from the mean.
\end{question}

When the significant source of error is considered to be random and the distribution is a Gaussian, random error associated with the reading is some factor times $\sigma$, the factor depending on how much confidence we wish to associate with our reading. If we choose $\sigma$ as our error -- this is a common choice -- the confidence associated with the reading is $68 \%$.
\newpage
\textbf{A note on statistics:} Consider two sets of measurements of the acceleration due to gravity:

\begin{table}[!htb]
\parbox{.45\linewidth}{
\centering
\begin{tabular}{cc}
\hline
\textbf{S. No.}&$\bm{g}\,\, (ms^{-2})$\\
\hline
1&$9.80\pm0.01$\\
2&$0.70\pm0.01$\\
3&$18.90\pm0.01$\\
4&$15.60\pm0.01$\\
5&$4.10\pm0.01$\\
\hline
Mean&$9.82\pm0.01$\\
\hline
\end{tabular}
\caption{Measurement of $g$ (Set 1)}
}
\hfill
\parbox{.45\linewidth}{
\centering
\begin{tabular}{cc}
\hline
\textbf{S. No.}&$\bm{g}\,\, (ms^{-2})$\\
\hline
1&$9.83\pm0.01$\\
2&$9.80\pm0.01$\\
3&$9.82\pm0.01$\\
4&$9.84\pm0.01$\\
5&$9.83\pm0.01$\\
\hline
Mean&$9.82\pm0.01$\\
\hline
\end{tabular}
\caption{Measurement of $g$ (Set 2)}
}
\end{table}

It would obviously be wrong to go simply by the mean and say that both these sets of data were equally reliable. In fact, stating the mean of the first set as the acceleration due to gravity \textbf{doesn't make sense}.

Suppose we compared the mean of a data set to its \textit{dispersion} about the mean. If this is small, we could then say that there is some ``true'' value and that all the different values we measured occurred due to random fluctuations about this ``true'' value. Since the fluctuations are random, we could assume that they average out to zero, leaving us with a closer estimate to the ``true'' value than any individual reading.

\begin{question}
\paragraph{Question:} In above sets of data (with standard deviations of $7.6\,\,ms^{-2}$ and $0.02\,\,ms^{-2}$ respectively), does the quoted error of $\pm 0.01\,\, ms^{-2}$ make sense
\begin{enumerate}
    \item For Set 1?
    \item For Set 2?
    \item For Both?
    \item For Neither?
\end{enumerate}
Justify your answer quantitatively.
\end{question}

\begin{imp}
The two errors in the lab you will encounter that you cannot remove are \textbf{least count errors} and \textbf{random errors}. Remember to always compare them and \textbf{\textit{take the larger value}}. In Set 1 it makes no sense to say $g$ is specified to $(9.82\pm0.01)\,ms^{-2}$, since all of the values are much farther away than that!
\end{imp}

\begin{question}
\begin{enumerate}
    \item \textbf{Question:} Multiple measurements with the same instrument increases the
    \begin{enumerate}
        \item Accuracy
        \item Precision
        \item Both
    \end{enumerate}

    \item \textbf{Question:} Consider the following data-table~\\
    \begin{tabular}{cccccc}
    \hline
    \textbf{S. No.}&1&2&3&4&5\\
    \hline
    \textbf{Time Period} ($s$)& $1.2\pm0.1$&$1.2\pm0.1$&$1.2\pm0.1$&$1.2\pm0.1$&$1.2\pm0.1$\\
    \hline
    \end{tabular}~\\~\\
    The standard deviation is $0.0$. Would it be right to say that $T_\text{avg} = 1.2\pm 0.0$?
\end{enumerate}
\end{question}

However, keep in mind there are many cases where the mean  \textbf{does not} represent some true value.\footnote{For example, even if the average number of siblings every student has is $1.574$, there is no student who has a non-integer number of siblings.} There are also cases where the standard deviation contains physical information. Such cases are usually a result of statistical phenomena.

\begin{tip}
\begin{enumerate}
    \item In a coin-toss experiment with a large number of tosses, $\sigma$ gives you a measure of the bias of the coin.
    \item In a random walk, $\sigma$ can be a measure of the diffusion coefficient $D$.
    \item In shot-noise -- the statistical fluctuations of current due to the actual number of electrons flowing in the conductor per unit time -- $\sigma$ gives you a measure of Boltzmann's constant $k_B$.
\end{enumerate}
\end{tip}

\subsection{Reporting Errors}
\subsubsection{Significant figures}

The significant figures of a number are the digits in its representation that contribute to the precision of the number. In practice, we assume that all digits used to write a number are significant (except leading zeroes.\footnote{Non-leading zeros are considered to be significant. If you write a number as 1,200, we assume there are four significant digits. If you only mean to have two or three, then it is best to use scientific notation: $1.2 \times 10^3$ or $1.20 \times 10^3$. Leading zeros are not considered significant: 0.55 and 0.023 have just two significant figures.})

\textbf{Results of simple calculations should not increase the number of significant digits}. Calculations transform our knowledge; they do not increase it! The rounding should be performed at the final step of a calculation to prevent rounding errors at intermediate steps from propagating through your work but \textbf{only} one or two
extra digits suffice to prevent this.

If you measure a value on a two-digit digital meter to be 1.0 and another value to be 3.0, it is incorrect to say that the ratio of these measurements is 0.3333333. The two values are not exact numbers with infinite precision. Since they each have two significant digits, the correct number to write down is 0.33.\footnote{If this is an intermediate result, then 0.333 or 0.3333 are preferred, but the final result must have two significant digits.}

\begin{imp}
\textbf{Do not write significant figures beyond the first digit of the error on the quantity}. Giving more precision than this to a value is not only irrelevant, \textit{it is misleading}.

If you're told you're using FAR too many digits, please do not try to use the excuse, ``That's what the computer gave me.'' \textbf{You} are in charge of presenting your results, not the computer!
\end{imp}

\subsubsection{Error propagation}

Experiments worth carrying out rarely measure only one quantity. Typically we measure two or more quantities and then `fold' them together in some equation or equations to determine some other quantity that we believe to depend on them. It is thus imperative that we understand how uncertainties in certain measured quantities ``propagate'' into other derived quantities.

For our analysis, let us assume that

\begin{itemize}
    \item $x$ and $y$ are measured quantities with uncertainties $\delta x$ and $\delta y$ respectively. These errors are considered to be \textbf{uncorrelated}, meaning that $\delta x$ and $\delta y$ are independent.\footnote{For example, the precision of measuring the length of your simple pendulum has no effect on the precision of measuring time.}

    \item $c$ is a constant known to known absolutely precisely (or with negligible uncertainty).

    \item $z$ is a quantity \textit{derived} from $x$ and $y$ and possessing a ``propagated'' uncertainty $\delta z$.
\end{itemize}

The formulae used to compute errors are usually not completely understood: a variety of different formulae are used in different cases, and the reasons \textbf{why} are usually lost on students. It turns out that there is only \textbf{one} way to add (uncorrelated) errors, which one can manipulate to get the rest:

\begin{imp}
If $z = x + y$, their (uncorrelated) errors add \textbf{in quadrature}:
\begin{equation}
    \delta z = \sqrt{\left(\delta x\right)^2+\left(\delta y\right)^2}
    \label{quadrature}
\end{equation}
\end{imp}

The reason for errors adding in quadrature is one that comes from statistics; it is essential that the errors be \textbf{independent} of each other (uncorrelated).\footnote{It also makes sense:  errors could be positive and some negative, simply adding them could conceivably give a smaller number (or even zero!). Thus, the next best thing is to add their \textit{squares}.}

Here is another motivation:\footnote{This is \textbf{not} an explanation of why this is true!} you could imagine two uncorrelated measurements to represent two axes ($x$ and $y$) that are orthogonal to each other (see Figure (\ref{fig:quadrature}). Imagine that you want to specify a point $z = (x,y)$: given that there is an uncertainty in both the coordinates $x$ and $y$, the point $z$ is uncertain \textbf{at most} by $\delta z = \sqrt{\left(\delta x\right)^2+\left(\delta y\right)^2}$.

\begin{figure}
    \centering
    \includegraphics[scale=0.5]{figs/quadrature.png}
    \caption{Two points $(x,0)$ and $(0,y)$, with uncertainties $\delta x$ and $\delta y$ respectively, can be summed to get a point $z=(x,y)$ with uncertainty $\delta z$. }
    \label{fig:quadrature}
\end{figure}

This formula can then be used to get the uncertainties of more complicated relations. For example, consider $z = xy$. In this case, the two quantities are \textbf{multiplied}, and so we can't use the above formula as is. We could, however, take the log, and get $\log{z} = \log{x} + \log{y}$. This is of the form $u = v + w$. We could then apply Equation (\ref{quadrature}), and find that $$\delta \left(\log{z}\right) = \sqrt{\left(\delta \left(\log{x}\right)\right)^2 + \left(\delta \left(\log{y}\right)\right)^2}$$

The uncertainty in $\log{z}$ can easily be related by taking the derivative.\footnote{A differential is by definition the variation of function when its parameter changes by a small amount. We want to find how much $\log{z}$ changes when $z$ changes by $\delta z$. We use $\delta$ instead of $\dd$, since the variation is not truly \textit{infinitesimal}.}

\begin{equation*}
    \dd({\log{z}}) = \frac{\dd z}{z} \quad \implies \quad \delta (\log{z}) = \frac{\delta z}{z}
\end{equation*}

Thus,

\begin{equation}
    \frac{\delta z}{z} = \sqrt{\left(\frac{\delta x}{x}\right)^2 + \left(\frac{\delta y}{y}\right)^2}
\end{equation}

\begin{tip}
The following rules should exhaust most of the common cases that you will be exposed to in your undergraduate labs. Everything here can be generalised simply to a a \textit{set} of measurements $x_i$ with uncertainties $\delta x_i$.

\begin{enumerate}
    \item \textbf{Addition or subtraction by a constant:} If $z = c \pm x$, then
    \begin{equation}
        \delta z = \delta x
    \end{equation}

    \item \textbf{Multiplication by a constant:} If $z = c x$, then
    \begin{equation}
        \delta z = c\delta x
    \end{equation}

    \item \textbf{Addition or subtraction of two measured quantities:} If $z = x \pm y$, then

    \begin{equation}
        \delta z = \sqrt{(\delta x)^2 +(\delta y)^2}
    \end{equation}

    \item \textbf{Multiplication or division of two measured quantities:} If $z = xy$ or $z = \frac{x}{y}$, then

    \begin{equation}
        \frac{\delta z}{z} = \sqrt{\left(\frac{\delta x}{x}\right)^2 + \left(\frac{\delta y}{y} \right)^2}
        \label{relerror}
    \end{equation}

    \item \textbf{A measured quantity raised to a power:} If $z = x^c$, then

    \begin{equation}
        \frac{\delta z}{z} = c \frac{\delta x}{x}
        \label{powerror}
    \end{equation}

\end{enumerate}
\end{tip}

\begin{question}
\paragraph{Question:} Prove Equation (\ref{powerror}).

\paragraph{Question:} If $z = x^2 = x \times x$, we get different answers if we use the Power Rule (Equation (\ref{powerror})) or the Product Rule (Equation (\ref{relerror})). Which of the two is correct? Why?

\paragraph{Question:} Calculate the uncertainty in $z$ if
\begin{enumerate}
    \item $z = \frac{1}{x}$
    \item $z = \frac{x}{1+x}$
    \item $z = \frac{x}{x+y}$
\end{enumerate}
\paragraph{Hint:} The last one is slightly hard. You will first write it as $z = \frac{1}{1 + u}$, where $u = y/x$. Then, calculate $\delta z$ in terms of $\delta u$, and only then $\delta u$ in terms of $\delta x$ and $\delta y$.
\end{question}


\section{Machine Learning based prediction of the next SARS-CoV-2 variants and simulation of vaccine effectiveness} \label{proposal1}

\subsection{Idea and research question}

There is hope for an end of the pandemic due to the vaccines. All vaccines are developed against the wild type of the virus. Through the mutation processes the danger, that a variant occurs, against which the vaccines  are no longer effective, arises (viral escape). This proposal aims to detect this development faster to enable a faster and better response to new variants. So there are two scientific questions (which can also be separated):

\begin{enumerate}
	\item Can the next possible SARS-CoV-2 variants be predicted by a \ac{ML} model?
	\item Based on the  predicted possible next variants can the effectiveness of the vaccines be simulated?
\end{enumerate}

\color{green}
It is more realistic to focus on one of the questions above. For the first the data access is easier.
\color{black}

\subsection{Related work}

Already some works exist in the area of predicting virus mutations using \ac{ML} techniques. Hie et al. \cite{Hie2021} applied methods developed for \ac{NLP}.  According to their work escape mutations look different to the immune system, but have the same viral infectivity. The analogy from the \ac{NLP} area are word changes, which change the meaning of a sentence, but the grammaticality remains. %Through their work, they have managed to generate a connection between natural language and viral evolution.
Even before the SARS-CoV-2 pandemic, research was done in this area, e.g. from Salama et al. \cite{Salama2016}. They used neural networks for predicting new mutations and rough set techniques for detecting patterns in mutations. Furthermore they validated their approach for the Newcastle virus and achieved an accuracy of 75\%. \ac{GAN} achieve great results in image generation. Berman et al. applied a \ac{GAN} in \cite{Berman2020} to generate \ac{DNA} sequences.

\subsection{Approach}

After preprocessing the raw SARS-CoV-2 genomes we would like to train a \ac{ML} model for predicting the probable next mutations. The \ac{ML} model could be a standard neural network (as in  \cite{Salama2016}), a GAN (as in \cite{Berman2020}) or a neural network language model (as in \cite{Hie2021}). Due to a lack of time we haven't decided yet for one \ac{ML} model.
\color{green}
Is clarified after a decision for one proposal.
\color{black}

\subsection{Data sources}

For predicting the next possible SARS-CoV-2 variants the SARS-CoV-2 genome develop\-ment from the past can be used as data source (genomic time series data). Data sources are available through the GISAID initiative \cite{Gisaid2021}. In April 2021 over one million SARS-CoV-2 genomes are available via GISAID \cite{Maxmen2021}. Based on this data the Nextstrain\footnote{https://nextstrain.org/ncov/global?label=clade:19A} project provides a visualization how the SARS-CoV-2 genome evolves.

%In terms of data quality, when working with genomic data one should always have in mind, that genotyping errors are possible.

\subsection{Computational resources}

The needed computational resources and time depends on the concrete chosen \ac{ML} approach. As described above due to a lack of time we haven't decided yet for one \ac{ML} approach. For further research and the decision for one \ac{ML} approach we keep this in mind.

\subsection{Probable difficulties}

Challenges could arise due to the amount of data (one SARS-CoV-2 genome consists of about 30.000 bases => TS size = data instances * 30.000). This can be encountered with less data instances or a compression of SARS-CoV-2 genome e.g. through DNA2vec \cite{Ng2017} (represents pieces of \ac{DNA} as vector).

\newpage


\part{Lecture 07: Planning and Learning with Tabular Methods}
\title[RL Lecture 07]{Lecture 07: Planning and Learning \\with Tabular Methods}
\date{}
\frame{\titlepage}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Recall: RL Agent Taxonomy%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Recap: RL Agent Taxonomy}
\begin{figure}
	\includegraphics[width=5.5cm]{fig/lec01/RL_Agent_Taxonomy.pdf}
	\caption{Main categories of reinforcement learning algorithms\\  \SilverLectureSource}
\end{figure}
\begin{itemize}
	\item Up to now: independent usage of model-free and model-based RL
	\item Today: integrating both strategies (on finite  state \& action spaces)
\end{itemize}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Repetition: Model-based and Model-free RL}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{frame}
\frametitle{Table of Contents}
\tableofcontents
\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Model-based RL %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Model-based RL}
\begin{itemize}
	\item \hl{Plan/predict} value functions and/or policy from a model.
	\item Requires an a priori model or to learn a model from experience.
	\item Solves control problems by planning algorithms such as
	\begin{itemize}
		\item Policy iteration or
		\item Value iteration.
	\end{itemize}
\end{itemize}
\begin{figure}
	\includegraphics[width=5.5cm]{fig/lec07/Markov_Decision_Process.pdf}
	\caption{A model for discrete state and action space problems is generally an MDP (source: \href{https://commons.wikimedia.org/wiki/File:Markov_Decision_Process.svg}{www.wikipedia.org},  by \href{https://commons.wikimedia.org/wiki/User:Waldoalvarez}{Waldoalvarez} \href{https://creativecommons.org/licenses/by-sa/4.0/deed.en}{CC BY-SA 4.0})}
	\label{fig:Markov_Decision_Process}
\end{figure}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% What is a Model? %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{What is a Model?}
\begin{itemize}
	\item A model $\mathcal{M}$ is an MDP tuple $\left\langle\mathcal{X}, \mathcal{U}, \bm{\mathcal{P}}, \mathcal{R}, \gamma \right\rangle$. \pause
	\item In particular, we require the
	\begin{itemize}
		\item state-transition probability
	\begin{equation}
		\bm{\mathcal{P}} = \Pb{\bm{X}_{k+1}=\bm{x}_{k+1}|\bm{X}_k=\bm{x}_k, \bm{U}_k=\bm{u}_k}
	\end{equation}
	\item and the reward probability
	\begin{equation}
		\mathcal{R} = \Pb{R_{k+1}=r_{k+1}|\bm{X}_k=\bm{x}_k, \bm{U}_k=\bm{u}_k}\,.
	\end{equation}
	\end{itemize}\pause
	\item State space $\mathcal{X}$ and action space $\mathcal{U}$ is assumed to be known.\pause
	\item Discount factor $\gamma$ might be given by environment or engineer's choice. \pause
	\item What kind of model is available?
	\begin{itemize}
	\item If $\mathcal{M}$ is perfectly known a priori: \hl{true MDP}.
	\item If $\hat{\mathcal{M}}\approx \mathcal{M}$ needs to be learned: \hl{approximated MDP}.
\end{itemize}
\end{itemize}

}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Model Learning / Identification %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Model Learning / Identification}
\begin{itemize}
	\item Ideally, the model $\mathcal{M}$ is exactly known a priori (e.g., gridworld case). \pause
	\item On the contrary, a model might be too complex to derive or not exactly available (e.g., real physical systems). \pause
	\begin{itemize}
		\item \hl{Objective: estimate model $\hat{\mathcal{M}}$ from experience $\left\{X_0, U_0,R_1,\ldots,X_T\right\}$.} \pause
	\end{itemize}
	\item This is a supervised learning / system identification task:
	\begin{align*}
		\left\{X_0, U_0\right\}&\rightarrow\left\{X_1, R_1\right\}\\
		&\vdots\\
		\left\{X_{T-1}, U_{T-1}\right\}&\rightarrow\left\{X_{T}, R_{T}\right\}
	\end{align*}\pause\vspace{-0.2cm}
	\item Simple tabular / look-up table approach (with $n(x,u)$ visit count):
	\begin{equation}
		\begin{split}
		\hat{p}_{xx'}^u&=\frac{1}{n(x,u)}\sum_{k=0}^{T} 1(X_{k+1}=x'|X_{k}=x, U_k=u),\\
		\hat{\mathcal{R}}_{x}^u&=\frac{1}{n(x,u)}\sum_{k=0}^{T} 1(X_{k}=x|U_k=u) r_{k+1}.
	\end{split}
	\label{eq:dist_model}
	\end{equation}
\end{itemize}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Distribution vs. Sample Models %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Distribution vs. Sample Models}
\begin{itemize}
	\onslide<1->{\item A model based on $\bm{\mathcal{P}}$ and $\mathcal{R}$ is called a \hl{distribution model}.
	\begin{itemize}
		\item Contains descriptions of all possibilities by random distributions.
		\item Has full explanatory power, but is still rather complex to obtain.
	\end{itemize}}
	\onslide<2->{\item Alternatively, use \hl{sample models} to receive realization series.
	\begin{itemize}
		\item Remember black jack examples: easy to sample by simulation but hard to model a full distributional MDP.
	\end{itemize}}
\end{itemize}
\onslide<1->{\begin{figure}
	\includegraphics[width=5.0cm]{fig/lec07/cards_dice.jpg}
	\caption{Depending on the application distribution models are easily available or not (source: Josh Appel on \href{https://unsplash.com/photos/PHwdpTVUlXw}{Unsplash})}
\end{figure}}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Model-free RL %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Model-free RL}
\begin{itemize}
	\item \hl{Learn} value functions and/or policy directly from experience.
	\item Requires no model at all (policy can be considered an implicit model). \pause
	\item Solves control problems by learning algorithms such as
	\begin{itemize}
		\item Monte-Carlo,
		\item Sarsa or
		\item $Q$-learning.
	\end{itemize}
\end{itemize}\pause
\begin{figure}
	\includegraphics[width=4.5cm]{fig/lec07/direct_indirect_RL.pdf}
	\caption{If a perfect a priori model is not available, RL can be realized directly or indirectly (source: R. Sutton and G. Barto, Reinforcement learning: an introduction, 2018, \href{https://creativecommons.org/licenses/by-nc-nd/2.0/}{CC BY-NC-ND 2.0})}
	\label{fig:direct_indirect_RL}
\end{figure}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Advantages and Drawbacks: Model-free vs. Model-based RL %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Advantages \& Drawbacks: Model-free vs. Model-based RL }
\onslide<1->{Pro model-based / indirect RL:
\begin{itemize}
	\item Efficiently uses limited amount of experience (e.g., by replay).
	\item Allows integration of available a priori knowledge.
	\item Learned models might be re-used for other tasks (e.g., monitoring).
\end{itemize}}
\vspace{0.25cm}
\onslide<2->{Pro model-free / direct RL:
\begin{itemize}
	\item Is simpler to implement (only one task, not two consequent ones).
	\item Not affected by model bias / error during model learning.
\end{itemize}}
\onslide<1->{\begin{figure}
	\includegraphics[width=7.0cm]{fig/lec07/crossroads.jpg}
	\caption{What way is better? (source: Mike Kononov on \href{https://unsplash.com/photos/FY0nQ-I3H_U}{Unsplash})}
\end{figure}}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Dyna: Integrated Planning, Acting and Learning}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{frame}
\frametitle{Table of Contents}
\tableofcontents[currentsection]
\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% The General Dyna Architecture (1)%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{The General Dyna Architecture (1)}
\begin{itemize}
	\item Proposed by R. Sutton in 1990's
	\item General framework with many different implementation variants
\end{itemize}
\begin{figure}
	\includegraphics[width=7cm]{fig/lec07/Dyna.pdf}
	\caption{Dyna framework (source: R. Sutton and G. Barto, Reinforcement learning: an introduction, 2018, \href{https://creativecommons.org/licenses/by-nc-nd/2.0/}{CC BY-NC-ND 2.0})}
	\label{fig:Dyna}
\end{figure}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% The General Dyna Architecture (2)%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{The General Dyna Architecture (2)}
\begin{itemize}
	\onslide<1->{\item \hl{Direct RL update}: any model-free algorithm: $Q$-learning, Sarsa, ...}
	\onslide<2->{\item \hl{Model learning}:
	\begin{itemize}
		\item In tabular case: simple distribution estimation as in \eqref{eq:dist_model}
		\item Simple experience buffer to re-apply model-free algorithm
		\item For large or continuous state/action spaces: function approximation by supervised learning / system identification (next lecture)
	\end{itemize}}
	\onslide<3->{\item \hl{Search control}: strategies for selecting starting states and action to generate simulated experience}
\end{itemize}
\onslide<1->{\begin{figure}
	\includegraphics[width=5cm]{fig/lec07/Dyna.pdf}
\end{figure}}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Algorithmic Implementation: Dyna-Q %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Algorithmic Implementation: Dyna-$Q$}

\setlength{\algomargin}{0.5em}
\begin{algorithm}[H]
\footnotesize
\SetKwInput{Input}{input}
\SetKwInput{Output}{output}
\SetKwInput{Init}{init}
\SetKwInput{Param}{parameter}
\Param{$\alpha\in\left\{\mathbb{R}|0<\alpha<1\right\}, \quad n\in\left\{\mathbb{N}|n \geq 1\right\}$ (planning steps per real step)}
\Init{$\hat{q}(x,u)$ arbitrary (except terminal) and $\hat{\mathcal{M}}(x,u)$ $\forall \, \left\{x\in\mathcal{X}, u\in\mathcal{U}\right\}$}
\For{$j=1,2,\ldots$ episodes}{
	Initialize $x_0$\;
	$k \leftarrow 0$\;
	\Repeat{$x_k$ is terminal}{
				Choose $u_k$ from $x_k$ using a soft policy derived from $\hat{q}(x,u)$\;
				Take action $u_k$, observe $r_{k+1}$ and $x_{k+1}$\;
				$\hat{q}(x_k, u_k) \leftarrow \hat{q}(x_k, u_k) + \alpha\left[r_{k+1}+\gamma \max_{u} \hat{q}(x_{k+1}, u) - \hat{q}(x_k, u_k)\right]$\;
				$\hat{\mathcal{M}}(x_k,u_k)\leftarrow \left\{r_{k+1}, x_{k+1}\right\}$ (assuming deterministic env.)\;
				\For{$i=1,2,\ldots$ n}{
						$\tilde{x}_i\leftarrow$ random previously visited state\;
						$\tilde{u}_i\leftarrow$ random previously taken action in $\tilde{x}_i$\;
						$\left\{\tilde{r}_{i+1}, \tilde{x}_{i+1}\right\}\leftarrow\hat{\mathcal{M}}(\tilde{x}_i, \tilde{u}_i)$\;
						$\hat{q}(\tilde{x}_i, \tilde{u}_i) \leftarrow \hat{q}(\tilde{x}_i, \tilde{u}_i) + \alpha\left[\tilde{r}_{i+1}+\gamma \max_{u} \hat{q}(\tilde{x}_{i+1}, u) - \hat{q}(\tilde{x}_i, \tilde{u}_i)\right]$\;
				}
				$k \leftarrow k+1$\;
	}
}
\caption{Dyna with $Q$-learning (Dyna-$Q$)}
\label{algo:Dyna_Q}
\end{algorithm}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Remarks on Dyna-Q Implementation %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Remarks on Dyna-$Q$ Implementation}
The specific Dyna-$Q$ characteristics are:
\begin{itemize}
	\item Direct RL update: $Q$-learning,
	\item Model: simple memory buffer of previous real experience,
	\item Search strategy: random choices from model buffer.
\end{itemize}\pause
\vspace{0.5cm}
Moreover:
\begin{itemize}
	\item Number of Dyna planning steps $n$ is to be delimited from $n$-step bootstrapping (same symbol, two interpretations).\pause
	\item Without the model $\hat{\mathcal{M}}$ one would receive one-step $Q$-learning.\pause
	\item The model-based learning is done $n$ times per real environment interaction:
	\begin{itemize}
		\item Previous real experience is re-applied to $Q$-learning.
		\item Can be considered a \hl{background task}: choose $\max n$ s.t. hardware limitations (prevent turnaround errors).\pause
	\end{itemize}
	\item For stochastic environments: use a distributional model as in \eqref{eq:dist_model}.
	\begin{itemize}
		\item Update rule then may be modified from sample to expected update.
	\end{itemize}
\end{itemize}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Maze Example (1) %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Maze Example (1)}
\begin{columns}[t,onlytextwidth]
\begin{column}{0.6\textwidth}
\begin{minipage}[c]{\linewidth}
\begin{figure}
	\includegraphics[width=7cm]{fig/lec07/Dyna_Simple_Maze.pdf}
	\caption{Applying Dyna-$Q$ with different planning steps $n$ to simple maze (source: R. Sutton and G. Barto, Reinforcement learning: an introduction, 2018, \href{https://creativecommons.org/licenses/by-nc-nd/2.0/}{CC BY-NC-ND 2.0})}
	\label{fig:Dyna_Simple_Maze}
\end{figure}
\end{minipage}
\end{column}
\hfill
\begin{column}{0.39\textwidth}
\begin{minipage}[c]{\linewidth}
	\begin{itemize}
		\item Maze with obstacles (gray blocks)
		\item Start at $S$ and reach $G$
		\item $r_T=+1$ at $G$
		\item Episodic task with $\gamma=0.95$
		\item Step size $\alpha=0.1$
		\item Exploration $\varepsilon=0.1$
		\item Averaged learning curves
	\end{itemize}
\end{minipage}
\end{column}
\end{columns}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Maze Example (2) %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Maze Example (2)}
\begin{itemize}
	\item Blocks without an arrow depict a neutral policy (equal action values).
	\item Black squares indicate agent's position during second episode.
	\item Without planning ($n=0$), each episodes only adds one new item to the policy.
	\item With planning ($n=50$), the available experience is efficiently utilized.
		\item After the third episode, the planning agent found the optimal policy.
\end{itemize}
\begin{figure}
	\includegraphics[width=7.5cm]{fig/lec07/Dyna_Simple_Maze_Updates.pdf}
	\caption{Policies (greedy action) for Dyna-$Q$ agent halfway through second episode (source: R. Sutton and G. Barto, Reinforcement learning: an introduction, 2018, \href{https://creativecommons.org/licenses/by-nc-nd/2.0/}{CC BY-NC-ND 2.0})}
	\label{fig:Dyna_Simple_Maze_Updates}
\end{figure}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% What if the Model is Wrong? %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{What if the Model is Wrong?}
Possible model error sources:
\begin{itemize}
	\item A provided a priori model may be inaccurate (expert knowledge). \pause
	\item Environment behavior changes over time (non-stationary).\pause
	\item Early-stage model is biased due to learning process.\pause
	\item If function approximators are used: generalization error (cf. lecture 08 and following). \pause
\end{itemize}
\vspace{0.5cm}
Consequences:
\begin{itemize}
	\item Model errors are likely to lead to a suboptimal policy. \pause
	\item If lucky: errors are quickly discovered and directly corrected by default, random exploration.\pause
	\item Nevertheless, more intelligent exploration / correction strategies might be useful (compared to random actions as in $\varepsilon$-greedy strategies).
\end{itemize}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% The Blocking Maze Example %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{The Blocking Maze Example}
\begin{columns}[t,onlytextwidth]
\begin{column}{0.6\textwidth}
\begin{minipage}[c]{\linewidth}
\begin{figure}
	\includegraphics[width=6.75cm]{fig/lec07/Dyna_Q_Blocking.pdf}
	\caption{Maze with a changing layout after 1000 steps illustrates a model error (source: R. Sutton and G. Barto, Reinforcement learning: an introduction, 2018, \href{https://creativecommons.org/licenses/by-nc-nd/2.0/}{CC BY-NC-ND 2.0})}
	\label{fig:Dyna_Q_Blocking}
\end{figure}
\end{minipage}
\end{column}
\hfill
\begin{column}{0.39\textwidth}
\begin{minipage}[c]{\linewidth}
	\begin{itemize}
		\item Maze with a changing obstacle line
		\item Start at $S$ and reach $G$
		\item $r_T=+1$ at $G$
		\item Dyna-$Q+$ encourages intelligent exploration (upcoming slides)
		\item Dyna-$Q$ requires more steps in order to overcome the blockade
		\item Averaged learning curves
	\end{itemize}
\end{minipage}
\end{column}
\end{columns}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% The Shortcut Maze Example %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{The Shortcut Maze Example}
\begin{columns}[t,onlytextwidth]
\begin{column}{0.6\textwidth}
\begin{minipage}[c]{\linewidth}
\begin{figure}
	\includegraphics[width=6.75cm]{fig/lec07/Dyna_Q_Shortcut.pdf}
	\caption{Maze with an additional shortcut after 3000 steps (source: R. Sutton and G. Barto, Reinforcement learning: an introduction, 2018, \href{https://creativecommons.org/licenses/by-nc-nd/2.0/}{CC BY-NC-ND 2.0})}
	\label{fig:Dyna_Q_Shortcut}
\end{figure}
\end{minipage}
\end{column}
\hfill
\begin{column}{0.39\textwidth}
\begin{minipage}[c]{\linewidth}
	\begin{itemize}
		\item Maze opens a shortcut after 3000 steps
		\item Start at $S$ and reach $G$
		\item $r_T=+1$ at $G$
		\item Dyna-$Q$ with random exploration is likely not finding the shortcut
		\item Dyna-$Q$+ exploration strategy is able to correct internal model
		\item Averaged learning curves
	\end{itemize}
\end{minipage}
\end{column}
\end{columns}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Dyna-Q+ Extensions %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Dyna-$Q$+ Extensions}
Compared to default Dyna-$Q$ in \algoref{algo:Dyna_Q}, Dyna-$Q$+ contains the following extensions:
\begin{itemize}
	\item Search heuristic: add $\kappa \sqrt{\tau}$ to regular reward.
	\begin{itemize}
		\item $\tau$: is the number of time steps a state-action transition has not been tried.
		\item $\kappa$: is a small scaling factor $\kappa\in\left\{\mathbb{R}|0<\kappa\right\}$.
		\item Agent is encouraged to keep testing all accessible transitions.
	\end{itemize}\pause
	\vspace{0.25cm}
	\item Actions for given states that had never been tried before are allowed for simulation-based planning.
	\begin{itemize}
		\item Initial model for that: actions lead back to same state without reward.
	\end{itemize}
\end{itemize}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Prioritized Sweeping}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{frame}
\frametitle{Table of Contents}
\tableofcontents[currentsection]
\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Background and Idea %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Background and Idea}

\begin{itemize}
	\item Dyna-$Q$ randomly samples from the memory buffer.
	\begin{itemize}
		\item Many planning updates maybe pointless, e.g., zero-valued state updates during early training.
		\item In large state-action spaces: inefficient search since transitions are chosen far away from optimal policies.
	\end{itemize}\pause
	\vspace{0.5cm}
	\item Better: focus on important updates.
	\begin{itemize}
		\item In episodic tasks: \hl{backward focusing} starting from the goal state.
		\item In continuing tasks: \hl{prioritize} according to impact on value updates. \pause
	\end{itemize}
	\vspace{0.5cm}
	\item Solution method is called \hl{prioritized sweeping}.
	\begin{itemize}
		\item Build up a queue of every state-action pair whose value would change significantly.
		\item Prioritize updates by the size of change.
		\item Neglect state-action pairs with only minor impact.
	\end{itemize}
\end{itemize}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Algorithmic Implementation: Prioritized sweeping %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Algorithmic Implementation: Prioritized Sweeping}
\vspace{-0.15cm}
\setlength{\algomargin}{0.5em}
\begin{algorithm}[H]
\footnotesize
\SetKwInput{Input}{input}
\SetKwInput{Output}{output}
\SetKwInput{Init}{init}
\SetKwInput{Param}{parameter}
\Param{$\alpha\in\left\{\mathbb{R}|0<\alpha<1\right\}, \quad n\in\left\{\mathbb{N}|n \geq 1\right\}, \quad \theta\in\left\{\mathbb{R}|\theta \geq 0\right\}$}
\Init{$\hat{q}(x,u)$ arbitrary and $\hat{\mathcal{M}}(x,u)$ $\forall \, \left\{x\in\mathcal{X}, u\in\mathcal{U}\right\}$, empty queue $\mathcal{Q}$}
\For{$j=1,2,\ldots$ episodes}{
	Initialize $x_0$ and $k \leftarrow 0$\;
	\Repeat{$x_k$ is terminal}{
				Choose $u_k$ from $x_k$ using a soft policy derived from $\hat{q}(x,u)$\;
				Take action $u_k$, observe $r_{k+1}$ and $x_{k+1}$\;
				%$\hat{q}(x_k, u_k) \leftarrow \hat{q}(x_k, u_k) + \alpha\left[r_{k+1}+\gamma \max_{u} \hat{q}(x_{k+1}, u) - \hat{q}(x_k, u_k)\right]$\;
				$\hat{\mathcal{M}}(x_k,u_k)\leftarrow \left\{r_{k+1}, x_{k+1}\right\}$ (assuming deterministic env.)\;
				$P \leftarrow \left|r_{k+1}+\gamma \max_{u} \hat{q}(x_{k+1}, u) - \hat{q}(x_k, u_k)\right|$\;
				\lIf{$P>\theta$}{insert $\left\{x_{k}, u_{k}\right\}$ in $\mathcal{Q}$ with priority $P$}
				\For{$i=1,2,\ldots$ n while queue $\mathcal{Q}$ is not empty}{
						$\left\{\tilde{x}_{i}, \tilde{u}_{i}\right\}\leftarrow \argmax_{P}(\mathcal{Q})$\;
						$\left\{\tilde{r}_{i+1}, \tilde{x}_{i+1}\right\}\leftarrow\hat{\mathcal{M}}(\tilde{x}_i, \tilde{u}_i)$\;
						$\hat{q}(\tilde{x}_i, \tilde{u}_i) \leftarrow \hat{q}(\tilde{x}_i, \tilde{u}_i) + \alpha\left[\tilde{r}_{i+1}+\gamma \max_{u} \hat{q}(\tilde{x}_{i+1}, u) - \hat{q}(\tilde{x}_i, \tilde{u}_i)\right]$\;
						\For{$\forall \left\{\overline{x}, \overline{u}\right\}$ predicted to lead to $\tilde{x}_i$}{
						$\overline{r}\leftarrow$ predicted reward for $\left\{\overline{x}, \overline{u}, \tilde{x}_i\right\}$\;
						$P \leftarrow \left|\overline{r}+\gamma \max_{u} \hat{q}(\tilde{x}_i, u) - \hat{q}(\overline{x}, \overline{u})\right|$\;
						\lIf{$P>\theta$}{insert $\left\{\overline{x}, \overline{u}\right\}$ in $\mathcal{Q}$ with priority $P$}
						}
				}
				$k \leftarrow k+1$\;
	}
}
%\caption{Prioritized sweeping}
\label{algo:prior_sweeping}
\end{algorithm}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Remarks on Prioritized Sweeping Implementation %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Remarks on Prioritized Sweeping Implementation}
The specific prioritized sweeping characteristics are:
\begin{itemize}
	\item Direct RL update: $Q$-learning,
	\item Model: simple memory buffer of previous real experience,
	\item \hl{Search strategy}: prioritized updates based on predicted value change.
\end{itemize}\pause
\vspace{0.5cm}
Moreover:
\begin{itemize}
	\item $\theta$ is a hyperparameter denoting the update significance threshold.\pause
	\item Prediction step regarding $\tilde{x}_i$ is a backward search in the model buffer.\pause
	\item For stochastic environments: use a distributional model as in \eqref{eq:dist_model}.
	\begin{itemize}
		\item Update rule then may be modified from sample to expected update.
	\end{itemize}
\end{itemize}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Comparing Against Dyna-$Q$ on Simple Maze Example %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Comparing Against Dyna-$Q$ on Simple Maze Example}
\begin{columns}[t,onlytextwidth]
\begin{column}{0.6\textwidth}
\begin{minipage}[c]{\linewidth}
\begin{figure}
	\includegraphics[width=6.75cm]{fig/lec07/Prior_Sweeping_Dyna.pdf}
	\caption{Comparison of prioritized sweeping and Dyna-$Q$ on simple maze (source: R. Sutton and G. Barto, Reinforcement learning: an introduction, 2018, \href{https://creativecommons.org/licenses/by-nc-nd/2.0/}{CC BY-NC-ND 2.0})}
	\label{fig:Prior_Sweeping_Dyna}
\end{figure}
\end{minipage}
\end{column}
\hfill
\begin{column}{0.39\textwidth}
\begin{minipage}[c]{\linewidth}
	\begin{itemize}
		\item Environment framework as in \figref{fig:Dyna_Simple_Maze}
		\item But: changing maze sizes (number of states)
		\item Both methods can utilize up to $n=5$ planning steps
		\item Prioritized sweeping finds optimal solution 5-10 times quicker
	\end{itemize}
\end{minipage}
\end{column}
\end{columns}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Update Variants}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{frame}
\frametitle{Table of Contents}
\tableofcontents[currentsection]
\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Update Rule Alternatives %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Update Rule Alternatives}
\begin{itemize}
	\item Dyna updates (search strategy) are not bound to $Q$-learning during planning and can be exchanged in many ways (see \figref{fig:Backup_one_step}).
	\item Even evaluating a fixed policy $\pi$ in terms of $v_\pi(x)$ and $q_\pi(x,u)$ is possible.
\end{itemize}
\begin{figure}
	\includegraphics[width=11cm]{fig/lec07/Backup_one_step.pdf}
	\caption{Possible one-step updates: alternatives for Dyna}
	\label{fig:Backup_one_step}
\end{figure}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Advantages and Drawbacks: Expected vs. Sampled Updates %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Advantages \& Drawbacks: Expected vs. Sampled Updates}
Pro expected updates:
\begin{itemize}
	\item Delivers more accurate value estimates (no sampling error).  \pause
\end{itemize}
\vspace{0.25cm}
Pro sample updates:
\begin{itemize}
	\item Is computational cheaper (e.g., distributional model not required). \pause
\end{itemize}
\vspace{0.25cm}
Leads to trade-off:
\begin{itemize}
	\item Estimation accuracy vs. computational burden.\pause
	\item Evaluate decision on given problem, i.e., how many state-action pairs have to be evaluated for a new expected update?\pause
	\item Utilize \hl{branching factor $b$} metric: corresponds to number of possible next states $x'$ with $p(x'|x,u)>0$.
	\begin{itemize}
		\item If expected update is available, this will be roughly as accurate as $b$ samplings.
		\item If only incomplete expected update is available, prefer sampling solution (often applies to large state-action spaces).
	\end{itemize}
\end{itemize}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Example for Sampled vs. Expected Updates %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Example for Sampled vs. Expected Updates}
\begin{itemize}
	\item Artificial prediction task where all $b$ successor states are equally likely
	\item Method initialization such that RMS error is always one
	\item Sample updates perform particularly well for large branching factors $b$ (takeaway message: \hl{if facing large stochastic problems, use sampling})
\end{itemize}
\begin{figure}
	\includegraphics[width=8.5cm]{fig/lec07/Expected_vs_sample_updates_example.pdf}
	\caption{Comparison of expected vs. sampled updates (source: R. Sutton and G. Barto, Reinforcement learning: an introduction, 2018, \href{https://creativecommons.org/licenses/by-nc-nd/2.0/}{CC BY-NC-ND 2.0})}
	\label{fig:Expected_vs_sample_updates_example}
\end{figure}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Alternative on Distributing the Updates %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Alternatives on Distributing the Updates}
Recap:
\begin{itemize}
	\item Dynamic programming: sweep through the entire state(-action) space
	\item Dyna-$Q$: random uniform sampling
	\item Mutual problem: irrelevant updates decrease computational efficiency
\end{itemize}\pause
\vspace{0.5cm}
Alternative: update according to on-policy distribution
\begin{itemize}
	\item Based on sampling along the encountered state(-action) pairs (\hl{trajectory sweeping})\pause
	\item Based on explicit on-policy distribution\pause
	\item In both cases: ignore vast, uninteresting parts of the problem space at the risk of updating same old parts all over again
\end{itemize}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Exemplary Update Distribution Comparison %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Exemplary Update Distribution Comparison}
\begin{columns}[t,onlytextwidth]
\begin{column}{0.62\textwidth}
\begin{minipage}[c]{\linewidth}
\begin{figure}
	\includegraphics[width=4.5cm]{fig/lec07/Trajectory_sampling.pdf}
	\caption{Update distribution comparison (source: R. Sutton and G. Barto, Reinforcement learning: an introduction, 2018, \href{https://creativecommons.org/licenses/by-nc-nd/2.0/}{CC BY-NC-ND 2.0})}
	\label{fig:Trajectory_sampling}
\end{figure}
\end{minipage}
\end{column}
\hfill
\begin{column}{0.39\textwidth}
\begin{minipage}[c]{\linewidth}
Example:
	\begin{itemize}
		\item Two actions per state
		\item Both actions led to $b$ next states
		\item $10\,\%$ probability of transition to terminal state
		\item reward per transition: $\mathcal{N}(\mu=0, \sigma^2=1)$
		\item Task: estimate start state value
		\item 200 randomly generated undiscounted episodic runs
	\end{itemize}
\end{minipage}
\end{column}
\end{columns}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Planning at Decision Time}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{frame}
\frametitle{Table of Contents}
\tableofcontents[currentsection]
\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Background Planning vs. Planning at Decision Time %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Background Planning vs. Planning at Decision Time}
\hl{Background Planning} (discussed so far):
\begin{itemize}
	\item Gradually improves policy or value function if time is available.
	\item Backward view: re-apply gathered experience. \pause
	\item Feasible for fast execution: policy or value estimate are available with low latency (important, e.g., for real-time control). \pause
\end{itemize}
\vspace{0.5cm}
\hl{Planning at decision time}\footnote[1]{\onslide<3->{Can be interpreted as \textit{model predictive control} in an engineering context.}} (not yet discussed alternative):
\begin{itemize}
	\item Select single next future action through planning.
	\item Forward view: predict future trajectories starting from current state. \pause
	\item Typically discards previous planning outcomes (start from scratch after state transition).\pause
	\item If multiple trajectories are independent: easy parallel implementation.\pause
	\item Most useful if fast responses are not required (e.g., turn-based games).
\end{itemize}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Heuristic Search %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Heuristic Search}
\begin{itemize}
	\item Develop \hl{tree-like continuations} from each state encountered.
	\item Approximate value function at leaf nodes (using a model) and back up towards the current state.
	\item Choose action according to predicted trajectory with highest value.
	\item Predictions are normally discarded (new search tree in each state).
\end{itemize}
\begin{figure}
	\includegraphics[width=7.0cm]{fig/lec07/Heuristic_search_tree.pdf}
	\caption{Heuristic search tree with exemplary order of back-up operations (source: R. Sutton and G. Barto, Reinforcement learning: an introduction, 2018, \href{https://creativecommons.org/licenses/by-nc-nd/2.0/}{CC BY-NC-ND 2.0})}
	\label{fig:Heuristic_search_tree}
\end{figure}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Rollout Algorithms %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Rollout Algorithms}
\begin{itemize}
	\item Similar to heuristic search, but: \hl{simulate trajectories following a rollout policy}.
	\item Use Monte Carlo estimates of action value \hl{only for current state} to evaluate on best action.
	\item Gradually improves rollout policy but optimal policy might not be found if rollout sequences are too short.
	\item Predictions are normally discarded (new rollout in each state).
\end{itemize}
\begin{figure}
	\includegraphics[width=6.5cm]{fig/lec07/Rollout.pdf}
	\caption{Simplified processing diagram of rollout algorithms}
	\label{fig:Rollout}
\end{figure}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Monte Carlo Tree Search (MCTS) %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Monte Carlo Tree Search (MCTS)}
\begin{itemize}
	\item Rollout algorithm, but:
	\begin{itemize}
		\item \hl{accumulates values estimates} from former MC simulations,
		\item makes use of an \hl{informed tree policy} (e.g., $\varepsilon$-greedy).
	\end{itemize}
\end{itemize}
\begin{figure}
	\includegraphics[width=8cm]{fig/lec07/MCTS.pdf}
	\caption{Basic building blocks of MCTS algorithms (source: R. Sutton and G. Barto, Reinforcement learning: an introduction, 2018, \href{https://creativecommons.org/licenses/by-nc-nd/2.0/}{CC BY-NC-ND 2.0})}
	\label{fig:MCTS}
\end{figure}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Basic MCTS Procedure %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Basic MCTS Procedure}
Repeat the following steps while prediction time is available:
\begin{enumerate}
	\item \hl{Selection}: Starting at root node, use a tree policy (e.g., $\varepsilon$-greedy) to travel through the tree until arriving at a leaf node.\pause
	\begin{itemize}
		\item The tree policy exploits auspicious tree regions while maintaining some exploration.
		\item It is improved and (possibly) extended in every simulation run. \pause
	\end{itemize}
	\item \hl{Expansion}: Add child node(s) to the leaf node by evaluating unexplored actions (optional step). \pause
	\item \hl{Simulation}: Simulate the remaining full episode using the rollout policy starting from the leaf or child node (if available). \pause
	\begin{itemize}
		\item The rollout policy could be random, pre-trained or based on model-free methods using real experience (if available). \pause
	\end{itemize}
	\item \hl{Backup}: Update the values along the traveled trajectory but only saves those within the tree policy.
\end{enumerate}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Further MCTS Remarks %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Further MCTS Remarks}
What is happening after reaching the feasible simulation runs?
\begin{itemize}
	\item After time is up, MCTS picks an appropriate action regarding the root node, e.g.:
	\begin{itemize}
		\item The action visited the most times during all simulation runs or
		\item The action having the largest action value. \pause
	\end{itemize}
	\item After transitioning to a new state, the MCTS procedure re-starts:
	\begin{itemize}
		\item Either with a new tree incorporating only the root node or
		\item by re-utilizing the applicable parts from the previous tree. \pause
	\end{itemize}
\end{itemize}
\vspace{0.5cm}
Further reading on MCTS:
\begin{itemize}
	\item MCTS-based algorithms are not limited to game applications but were able to achieve outstanding success in this field.
	\begin{itemize}
		\item Famous AlphaGo (cf. \href{https://www.youtube.com/watch?v=Wujy7OzvdJk}{Keynote lecture from D. Silver})\pause
	\end{itemize}
	\item More in-depth lectures on MCTS can be found (among others) here:
	\begin{itemize}
		\item \href{https://www.youtube.com/watch?v=vDF1BYWhqL8}{Stanford Online: CS234}
		\item \href{https://www.youtube.com/watch?v=xmImNoDc9Z4}{MIT OpenCourseWare}
		\item \href{https://www.lri.fr/~sebag/Slides/InvitedTutorial_CP12.pdf}{Extensive slide set from M. Sebag at Universite Paris Sud}
	\end{itemize}
\end{itemize}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Summary %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{frame}
\frametitle{Summary: What You've Learned Today}
\begin{itemize}
	\item Model-free RL is easy to implement and cannot suffer any model learning error while model-based approaches use a limited amount of experience much more efficient. \pause
	\item Integrating these two RL branches can be achieved using the Dyna framework (background planning) incorporating the steps:
	\begin{itemize}
		\item Direct RL updates (any model-free approach, e.g., $Q$-learning),
		\item Model learning: use real experience to improve model predictions,
		\item Search control: strategies on how to generate simulated experience. \pause
	\end{itemize}
	\item The Dyna framework allows many different algorithms such as Dyna-$Q$(+) or prioritized sweeping.
	\begin{itemize}
		\item Learning efficiency is much increased compared to pure model-based/free approaches.
		\item Many degrees of freedom regarding internal update rules exist.\pause
	\end{itemize}
	\item In contrast, planning at decision time predicts future trajectories starting from the current state (forward view).
	\begin{itemize}
		\item Rather computationally expensive leading to high latency responses.
		\item The Monte Carlo tree search rollout algorithm is a well-known example.
	\end{itemize}
\end{itemize}
\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Final Slide %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{The End for Today}
\vspace{-0.25cm}
\begin{figure}
\hspace*{-0.5cm}
\includegraphics[width=10cm]{fig/lec07/dilbert.png}
\end{figure}
\vspace{1cm}
\centering
Thanks for your attention and have a nice week!
}


\documentclass[]{article}
\usepackage{lmodern}
\usepackage{amssymb,amsmath}
\usepackage{ifxetex,ifluatex}
\usepackage{fixltx2e} % provides \textsubscript
\ifnum 0\ifxetex 1\fi\ifluatex 1\fi=0 % if pdftex
  \usepackage[T1]{fontenc}
  \usepackage[utf8]{inputenc}
\else % if luatex or xelatex
  \ifxetex
    \usepackage{mathspec}
    \usepackage{xltxtra,xunicode}
  \else
    \usepackage{fontspec}
  \fi
  \defaultfontfeatures{Mapping=tex-text,Scale=MatchLowercase}
  \newcommand{\euro}{€}
    \setmainfont{PT Serif}
\fi
% use upquote if available, for straight quotes in verbatim environments
\IfFileExists{upquote.sty}{\usepackage{upquote}}{}
% use microtype if available
\IfFileExists{microtype.sty}{%
\usepackage{microtype}
\UseMicrotypeSet[protrusion]{basicmath} % disable protrusion for tt fonts
}{}
\ifxetex
  \usepackage[setpagesize=false, % page size defined by xetex
              unicode=false, % unicode breaks when used with xetex
              xetex]{hyperref}
\else
  \usepackage[unicode=true]{hyperref}
\fi
\hypersetup{breaklinks=true,
            bookmarks=true,
            pdfauthor={David Charte},
            pdftitle={Common multilabel formulae},
            colorlinks=true,
            citecolor=blue,
            urlcolor=blue,
            linkcolor=magenta,
            pdfborder={0 0 0}}
\urlstyle{same}  % don't use monospace font for urls
\setlength{\parindent}{0pt}
\setlength{\parskip}{6pt plus 2pt minus 1pt}
\setlength{\emergencystretch}{3em}  % prevent overfull lines
\setcounter{secnumdepth}{0}

\title{Common multilabel formulae\\\vspace{0.5em}{\large \ldots{}under a common notation}}
\author{David Charte}
\date{}
\newcommand{\func}[1]{\operatorname{\mathit{#1}}} \newcommand{\const}[1]{\mathit{#1}} \newcommand{\abs}[1]{\left\lvert{#1}\right\rvert}

\begin{document}
\maketitle
\begin{abstract}
Multilabel Classification is a branch of Data Mining in which many
different metrics and equations are defined. Each author uses different
notation and basic definitions, thus difficulting the coherence of
future papers where some of these formulas are used. This is a document
where the multilabel scenario is presented with simple basic
definitions, and a list of commonly used formulas are rigurously adapted
to this notation, in an attempt to introduce them in a clear way.
\end{abstract}

\section{Definitions}\label{definitions}

Let \(A_1, A_2, \dots A_f\) be arbitrary sets. We will call them
\emph{input attributes} or simply \emph{attributes}. An instance will
take a certain value on each of these sets, that is, we will be working
with elements of their cartesian product,
\(A_1\times A_2\times\dots\times A_f\).

Let \(L\) be a finite set. This will be the set of all possible labels.
Each instance of a dataset will then have a subset of active labels or
\emph{labelset}, \(y \subset L\).

Let \(D\) be a finite subset of
\(A_1\times A_2\times\dots\times A_f\times\mathcal{P}(L)\), where
\(\mathcal{P}(L)\) is the powerset of \(L\), that is, the set of all
possible combinations of labels. We will call \(D\) a \emph{dataset} and
each of its elements an \emph{instance}: \((x, y)\in D\) where
\(x=(x_1,x_2,\dots x_f)\in A_1\times A_2\times\dots\times A_f\) and
\(y=\{l_1,\dots l_k\} \in\mathcal{P}(L)\).

\textbf{Note}: Since \(D\) is a set, we will be assuming no two
instances are identical.

\section{Basic measures}\label{basic-measures}

In the following we will assume \(D\) is a fixed set. Otherwise, many of
the measures would be dependent on D. Some basic data directly extracted
from the definitions are:

\begin{itemize}
\itemsep1pt\parskip0pt\parsep0pt
\item
  Number of input attributes, \(f\)
\item
  Number of labels, \(\abs L\)
\item
  Number of instances, \(\abs D\)
\item
  Number of distinct labelsets, \(\abs{\{Y:(X, Y)\in D\}}\)
\end{itemize}

Many of the measures specific to multilabel classification are related
to the labels themselves. For instance, we can find out the mean number
of labels in a labelset (\emph{Card}, \ref{eq:Card}) and its proportion
as to the total number of labels (\emph{Dens}, \ref{eq:Dens}):

\begin{equation} \func{Card} = \frac 1 {\abs D} \sum_{(x,y)\in D} \abs y \label{eq:Card}\end{equation}

\begin{equation} \func{Dens} = \frac 1 {\abs D \abs L} \sum_{(x,y)\in D} \abs y \label{eq:Dens}\end{equation}

\section{Imbalance and concurrence}\label{imbalance-and-concurrence}

\end{document}


\chapter{Post-Report Double Implementation}\label{a:modernDouble}
This appendix contains a complete version of Double described in \autoref{teExample}, but modernizes the example by using the improvements discussed in this report.
\begin{tcolorbox}[breakable, toprule at break=0pt, bottomrule at break=0pt]
\begin{minted}{haskell}
module Main(main) where
import Language.Drasil (
  -- Primitive types
  Document, Expr(Int), Sentence(S, E, EmptyS),
  Space(Integer), Symbol(Atomic),
  -- Chunks
  CI, ConceptInstance, ConstrainedChunk, QDefinition,
  QuantityDict, UnitaryConceptDict,
  -- Smart constructors
  cic, commonIdeaWithDict, cn''', fromEqn', nounPhraseSent,
  pn, nw, qw, person, vc,
  -- Sentence functions
  atStart, short, phrase,
  -- Expr combinators
  ($^), sy)

import Database.Drasil (Block, ChunkDB,
  SystemInformation(SI), cdb, rdb, refdb, _authors,
  _concepts, _constants, _constraints,  _datadefs,
  _definitions, _defSequence, _inputs, _kind, _outputs,
  _quants, _sys, _sysinfodb, _usedinfodb)
import Theory.Drasil (DataDefinition, ddNoRefs)
import Utils.Drasil (chgsStart, foldlSent, foldlSent_, for)

import Data.Drasil.Concepts.Documentation (doccon, doccon',
  input_, output_, software, srs, system, value,
  -- Domains
  assumpDom, funcReqDom, likeChgDom, srsDomains)
import Data.Drasil.IdeaDicts (mathematics)
import Data.Drasil.SI_Units (derived, fundamentals, siUnits)

import Drasil.DocLang (
  DocSection(RefSec, IntroSec, SSDSec, ReqrmntSec, LCsSec,
    TraceabilitySec), DerivationDisplay(HideDerivation),
  Field(DefiningEquation, Description, Label, Symbol, Units),
  InclUnits(IncludeUnits), IntroSec(IntroProg),
  RefSec(RefProg), ReqrmntSec(ReqsProg), ReqsSub(FReqsSub),
  SCSSub(Assumptions, DDs), SolChSpec(SCSProg), SRSDecl,
  SSDSec(SSDProg), SSDSub(SSDSolChSpec), TSIntro(TSPurpose),
  TraceabilitySec(TraceabilityProg), Verbosity(Verbose),
  intro, mkDoc, tsymb, traceMatStandard)

import Language.Drasil.Code (Choices(..), CodeSpec,
  Comments(CommentNone), ConstraintBehaviour(Warning),
  ImplementationType(Program),
  Lang(Cpp, CSharp, Java, Python), Logging(LogNone),
  Structure(Bundled), codeSpec)
import Language.Drasil.Printers (DocSpec(DocSpec),
  DocType(SRS, Website), PrintingInformation(PI),
  defaultConfiguration)
import Language.Drasil.Generate (gen, genCode)

symbols :: [QuantityDict]
symbols = [x, qw calcQ]

pname :: String
pname = "Double"

double :: CI
double = commonIdeaWithDict "double" (pn pname) pname
  [mathematics]

assumpNum :: ConceptInstance
assumpNum = cic "assumpNum" (foldlSent [S "This",
  phrase system, S "only considers", phrase input_,
  S "integers between", E $ (-2) $^ 29, S "and",
  E (2 $^ 29)]) "reasonableNumber" assumpDom

reqMul :: ConceptInstance
reqMul = cic "reqMul" (foldlSent [S "The", phrase output_,
  S "shall be twice the", phrase input_, phrase value])
  "mulNum" funcReqDom

chg :: ConceptInstance
chg = cic "chg" (foldlSent [chgsStart assumpNum (S "The"),
  phrase software, S "may be changed to remove the range",
  S "restriction on the", phrase input_, S "to support",
  S "doubling any integer"]) "removeRestriction" likeChgDom

-- Calculations
x :: QuantityDict
x = vc "x"    (cn''' "input value") (Atomic "x") Integer

calc :: Expr
calc = (Int 2) * sy x

calcQ :: QDefinition
calcQ = fromEqn' "y" (nounPhraseSent $ foldlSent_ [
  phrase input_, phrase value, S "doubled"]) EmptyS
  (Atomic "y") Integer calc

y :: DataDefinition
y = ddNoRefs calcQ [{-Derivation-}] "doubledd" [{-Notes-}]

thisSI :: SystemInformation
thisSI = SI {
  _sys = double,
  _kind = srs,
  _authors = [person "Gabriel" "Dalimonte"],
  _quants = symbols,
  _datadefs = [y],
  _inputs = [x],
  _outputs = [qw y],
  _sysinfodb = allSymbols,
  _usedinfodb = allSymbols,
  _defSequence = [] :: [Block QDefinition],
  _constraints = [] :: [ConstrainedChunk],
  _constants = [],
  _concepts = [] :: [UnitaryConceptDict],
  _definitions = [] :: [QDefinition],
  refdb = rdb [] []
}

allSymbols :: ChunkDB
allSymbols = cdb symbols (nw double : map nw symbols ++
  map nw doccon ++ map nw fundamentals ++ map nw derived ++
  map nw doccon') srsDomains siUnits [y] [] [] []
  [assumpNum, reqMul, chg] [] []

pS :: PrintingInformation
pS = PI allSymbols defaultConfiguration

thisSRS :: SRSDecl
thisSRS = [
  RefSec $ RefProg intro [tsymb [TSPurpose]],
    IntroSec $ IntroProg (foldlSent [atStart double,
      S "is a trivial example for demonstrating Drasil's",
      S "capabilities"]) (short double) [],
    SSDSec $ SSDProg [
      SSDSolChSpec $ SCSProg [
        Assumptions,
        DDs [] [Label, Symbol, Units, DefiningEquation,
        Description Verbose IncludeUnits] HideDerivation
      ]],
  ReqrmntSec $ ReqsProg [
    FReqsSub []
  ],
  LCsSec,
  TraceabilitySec $ TraceabilityProg $
    traceMatStandard thisSI]

srsBody :: Document
srsBody = mkDoc thisSRS for thisSI

thisCode :: CodeSpec
thisCode = codeSpec thisSI thisChoices []

thisChoices :: Choices
thisChoices = Choices {
  lang             = [Python, Cpp, CSharp, Java],
  impType          = Program,
  logFile          = "log.txt",
  logging          = LogNone,
  comments         = CommentNone,
  onSfwrConstraint = Warning,
  onPhysConstraint = Warning,
  inputStructure   = Bundled
}

main :: IO ()
main = do
  gen (DocSpec Website $ pname ++ "_SRS") srsBody pS
  gen (DocSpec SRS $ pname ++ "_SRS")     srsBody pS
  genCode thisChoices thisCode
\end{minted}
\end{tcolorbox}