\documentclass[11pt, a4paper]{article}

\usepackage{tikz}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{mathtools}
\usepackage{graphicx}
\usepackage{multicol}
\usepackage{appendix}
\usepackage{pgfplots}
\usepackage{algorithm}
\usepackage{algpseudocode}
\usepackage[utf8]{inputenc}
\usepackage[english]{babel}

\pgfplotsset{compat=1.16}

\date{}
\title{Exoplanets detection using \\ Auto-Tuned Random Forest}

\author{Lorenzo Loconte \\ \small Knowledge Engineering Project \\ \small University of Bari Aldo Moro, Department of Informatics}

\begin{document}

\maketitle
\begin{abstract}
  The \texttt{NASA Kepler} project consists of discovering exoplanets (i.e. planets outside our solar system).
  The data generated by the \texttt{Kepler Space Telescope} is analyzed by humans and algorithms to discover new exoplanets.
  The main problem is that a lot of exoplanets are revealed false positives.
  This work consists of identifying exoplanets using random forest, a \textit{Supervised Machine Learning} model.
  Furthermore, the fitted model is analyzed in order to determine which features are relevant.
  The hyperparameters are automatically optimized with techniques that come from the \texttt{AutoML} research.
  In fact, the hyperparameters of the model are optimized and cross-validated with \texttt{Hyperband}, a simple yet effective and scalable method for hyperparameters optimization.
\end{abstract}

\clearpage

\section{Introduction}
  \paragraph{Kepler Object of Interest}
    Exoplanets are particular planets that are situated outside our solar system which orbit around one star (sometimes even multiple stars).
    The \texttt{NASA Kepler} project aims to discover these objects by looking at some stars and observing their luminosity during time.
    Basically suppose we are observing a star.
    If a rapid decrease and consequently increase of its luminosity occurs it's most likely that an object passed the way between us and the star.
    Analyzing the variation of luminosity for a certain amount of time we can extrapolate features like the radius and the orbit eccentricity of the object.
    Sometimes these object are actually exoplanets but it's common they can be something else like asteroids or even another star.
    Sometimes they are just false positives due to measurements noise.

    The goal of this work is to build a model (using \textit{Supervised Machine Learning}) that is capable to determinate if a hypothetical exoplanet can be confirmed or is just false positive.
    The dataset used is the cumulative list of \texttt{Kepler Object of Interest} (\texttt{KOI}) tables that come from the \texttt{NASA Kepler} project of discovering exoplanets.
    The complete dataset can be found at \texttt{https://exoplanetarchive.ipac.caltech.edu}.
    The dataset is composed by a list of hypothetical exoplanets.
    Each sample have a label indicating if the corresponding exoplanet is candidate, false positive or confirmed.
    The provided dataset consists of a lot of heterogeneous features.
    For the classification task the selected features can be found in Appendix \ref{appendix:features}.

    The candidate exoplanets (i.e. the exoplanets which existence is uncertain) are discarded in order to reduce noise and improve the learning process.
    For simplicity all the samples having null values are discarded from the dataset.
    Furthermore, all the selected features are numeric and the preprocessing method applied to the dataset is the standard normalization.
    The resulting dataset contains \texttt{5860} samples, of which only \textasciitilde \texttt{38\%} are confirmed and the remaining are false positives.

  \paragraph{Random Forests}
    As described before, the classification task consists to determinate if some numerical characteristics can be attributed to an existing exoplanet.
    The model used for the classification task is a random forest, a bagging ensemble of decision trees.
    The prediction of the random forest is computed as the mode of the predictions made by the decision trees.
    Note that each tree of the forest is trained on a subset of the features.
    Random forests correct the habit of decision trees of overfitting on their training set.
    In fact, decision trees tend to overfit their training sets because they have a low bias but an high variance.

    In \textit{Supervised Machine Learning}, bias consists of erroneous assumptions about the data and variance is a measure of sensitivity to the noise in the training data.
    So, random forests generally obtain better performance.

    The law that regulates the bias and variance is called \texttt{Bias-Variance tradeoff} \cite{biasvariance}.
    This law states that models having low bias have an high variance and vice versa.
    In \textit{Machine Learning} bias and variance corresponds to the models behaviors that we want to regulate in order to minimize the error.
    An high bias causes \textit{underfitting} while an high variance causes \textit{overfitting}.
      \[\mathrm{Error}=\mathrm{Bias}^{2}+\mathrm{Variance}+\mathrm{Irreducible\:Error}\]

  \paragraph{Hyperparameters}
    Often it's difficult (or nearly impossible) to select the correct hyperparameters that guarantee good performance on some specific datasets.
    \texttt{AutoML} is a branch of \textit{Machine Learning} that try to simplify the process of model selection and hyperparameters optimization.
    This work focuses on solving the hyperparameters search problem for random forests.
    The algorithm used is \texttt{Hyperband} that, as it'll be shown in the next section, represents an evolution of random search designed especially for hyperparameters optimization.

    For the hyperparameters optimization algorithm chosen, the number of trees in the random forest is not considered an hyperparameter.
    Furthermore, the following hyperparameters are optimized:
    \begin{itemize}
      \item The split criterion (Gini Impurity or Information Gain, see Appendix \ref{appendix:splitcriterions} for details)
      \item The fraction of features to use for each tree
      \item The fraction of samples used to train each tree
      \item The maximum depth of each tree
      \item The minimum number of samples required to split an internal node
      \item The minimum number of samples to be at a leaf node
    \end{itemize}

\clearpage

\section{Hyperparameters Optimization}
  In this section it will be shown a formalization of the hyperparameters optimization task as a search problem.
  In this work we refer to hyperparameters as the ones that cannot be trained (e.g. the ones described in the previous section).
  After that we briefly introduce the hyperparameters optimizer used in this work, \texttt{Hyperband}, as described in \cite{hyperband}.

  Given a training set $T$, a model $L$ and its hyperparameters search space $\Omega$, the hyperparameters optimization task consists to find an hyperparameters configuration $\omega \in \Omega$ such that the resulting model $L\langle\omega\rangle$ maximizes a certain score $S$ using cross-validation on $T$.
  Note that most search algorithms don't converge to the global maximum.
  So, the goodness of the hyperparameters found is dependent of the amount of resources we have.

  There are a lot of different techniques for hyperparameters optimization, some of them are the following:
  \begin{itemize}
    \item Random Search
    \item Grid Search
    \item Hyperband
    \item Bayesian Optimization
    \item Hybrid Approaches (like \texttt{BOHB} \cite{bohb})
  \end{itemize}

  Of the techniques cited above only \texttt{Bayesian Optimization} guarantees the convergence to the global maximum in the hyperparameters search space (see \cite{bayesianoptimization} for details).
  The simplest algorithm is \texttt{Random Search} and it works well if the hyperparameters search space is not too big.
  \texttt{Random Search} is expensive because each evaluation of the score function is done on models completely built from scratch.
  So, each evaluation of the score function on some randomly selected hyperparameters will use the same amount of computational resources, even if some hyperparameters of the search space obtain a very low score.
  The idea behind \texttt{Hyperband} is that it's better to save up resources on these hyperparameters configurations that doesn't obtain a sufficiently good score by gradually building more complex models over time.

  Note that we refer to budget as the computational cost of cross-validating a certain model.
  For example, for neural networks the budget can be the number of training epochs, while for random forests it can be the number of trees.

  Algorithm \ref{algorithm:hyperband} shows an implementation of \texttt{Hyperband} as described now.
  Given a budget $R$, \texttt{Hyperband} consists of random sampling some points (for example using the uniform distribution) in the hyperparameters space and cross-validating them using only a fraction of the budget $R$.

  After that we pick a fraction $1/\eta$ of the best models.
  In this way we discard the models that didn't obtain a sufficiently good score saving up computational resources.

  After that, we strengthen the previous models using a larger portion of $R$ and iterate in this way until we use the complete budget $R$.
  The procedure of consequently eliminating hyperparameters samples that don't obtain a good score is called \texttt{Successive Halving}.
  The fraction of the budget to use for each iteration grows geometrically in respect of $\eta$.
  Furthermore, we execute the discussed algorithm multiple times using less and less initial random points and using an higher starting budget.
  So, initially \textit{exploration} is preferred and \textit{exploitation} is performed later.

  Even if \texttt{Hyperband} is a simple algorithm, it is well scalable on multiple CPUs because we assume that every point in the hyperparameters space is independent from the others.

  In our case the model $L$ is a random forest.
  As said before, the budget $R$ is the maximum number of trees that the final random forest will have.
  The choice of the score $S$ is dependent from the task.
  For simplicity the $F_{1}$ score is used (i.e. we want to maximize both \textit{Precision} and \textit{Recall}).
  In this work the hyperparameters search space can be defined formally as:
  \[\Omega = \overbrace{\{gini, entropy\}}^{\text{Split criterion}} \quad\times \overbrace{(0, 1]}^{\text{Features fraction}} \times \overbrace{(0, 1]}^{\text{Samples fraction}} \times\]
  \[\times \overbrace{\mathbb{N}^{*}}^{\text{Max depth}} \times \overbrace{\mathbb{N}^{*}}^{\text{Min samples to split}} \times \overbrace{\mathbb{N}^{*}}^{\text{Min samples at leaf}}\]
  For obvious reasons and for the hyperparameters optimization algorithm chosen, only a subset of this space can be explored.

  \begin{algorithm}
    \caption{Hyperband algorithm for hyperparameters optimization}
    \label{algorithm:hyperband}
    \begin{algorithmic}[1]
      \Procedure{Hyperband}{$T,L,S,\Omega,R,\eta$}
        \State $C \gets \emptyset$
        \State $s_{max} \gets \lfloor \log_{\eta}{R} \rfloor, \quad B \gets (s_{max} + 1)R$
        \For{$s \in \{s_{max},s_{max}-1,...,0\}$}
          \State $n \gets \lceil \frac{B}{R}\frac{\eta^{s}}{(s+1)} \rceil, \quad r \gets R\eta^{-s}$
          \State $H \gets getRandomHyperparameters(n,\Omega)$
          \For{$i \in \{0,...,s\}$}
            \State $n_{i} \gets \lfloor n\eta^{-i} \rfloor, \quad r_{i} \gets r\eta^{i}$
            \State $H \gets top_{\lfloor n_{i}/\eta \rfloor} \arg sort_{\omega} \{S(L\langle\omega \cup \{r_{i}\}\rangle, T) \mid \omega \in H\}$
          \EndFor
          \State $C \gets C \cup H$
        \EndFor
        \State \textbf{return} $\arg max_{\omega} \{S(L\langle\omega \cup \{R\}\rangle, T) \mid \omega \in C\}$
      \EndProcedure
    \end{algorithmic}
  \end{algorithm}

\clearpage

\section{Conclusion}
  In this section we analyze the results of the model chosen comparing it to other models.
  As you can see from Table \ref{table:benchmark} the auto-tuned random forest used in this work obtained a way better \textit{Recall}.

  Furthermore, as you can see from Figure \ref{figure:confusion} the false positives count (the number of examples, which are predicted as exoplanets, but that are not) and the false negatives count (the number of actual exoplanets not being discovered), are pretty low.

  Models like neural networks and support vector machines are said \textit{black-box} models.
  It means that predictions cannot be explained and the features importance cannot be evaluated easily.
  At contrary, random forests are said \textit{white-box} models.
  So, predictions can be explained deeply and it's possible to evaluate the importance of each feature used.
  In \textit{Artificial Intelligence} the explainability of a model is important because it guarantees transparency.
  Furthermore, in \textit{Machine Learning}, explainable models can reveal hidden knowledge about the data that isn't known at prior.
  In the end we present Figure \ref{figure:importances} that shows the importance of the five most important features.

  \begin{table}
    \centering
    \begin{tabular}{|c c c c|}
      \hline
      Model & Precision & Recall & $F_{1} \downarrow$  \\
      \hline\hline
      Random Forest & 0.940 & 0.916 & 0.927 \\
      \hline
      \textbf{Random Forest w/HB} & \textbf{0.934} & \textbf{0.935} & \textbf{0.935} \\
      \hline
    \end{tabular}
    \caption{\textit{Precision}, \textit{Recall} and $F_{1}$ metrics of random forests with and without hyperparameters optimization (using the default ones in \texttt{scikit-learn}). For \texttt{Hyperband} $R=216$ and $\eta=3$ are used.}
    \label{table:benchmark}
  \end{table}

  \begin{figure}
    \centering
    \begin{tikzpicture}[scale=1.85]
      \draw[thick] (0, 0) -- (2, 0);
      \draw[thick] (0, 0) -- (0, 2);
      \draw[thick] (2, 2) -- (2, 0);
      \draw[thick] (2, 2) -- (0, 2);
      \draw[thick] (0, 1) -- (2, 1);
      \draw[thick] (1, 0) -- (1, 2);
      \node[rotate=0] (h1) at (1, 2.5) {\textbf{Actual}};
        \node[rotate=0] (h11) at (0.5, 2.25) {Positive};
        \node[rotate=0] (h12) at (1.5, 2.25) {Negative};
      \node[rotate=90] (h2) at (-0.5, 1) {\textbf{Predicted}};
        \node[rotate=90] (h21) at (-0.25, 1.5) {Positive};
        \node[rotate=90] (h22) at (-0.25, 0.5) {Negative};
      \coordinate[label={\textbf{412}}] (TP) at (0.5, 1.33);
      \coordinate[label={ 28}] (FN) at (0.5, 0.33);
      \coordinate[label={ 29}] (FP) at (1.5, 1.33);
      \coordinate[label={\textbf{703}}] (TN) at (1.5, 0.33);
    \end{tikzpicture}
    \caption{Confusion matrix over the test set.}
    \label{figure:confusion}
  \end{figure}

  In random forests the features importance are calculated by averaging the features importance in each decision tree.
  For a decision tree the features importance are calculated during the fit and are computed as the normalized total reduction of the criterion brought by each feature.

  The most important feature is PRF $\Delta\theta_{SQ}$ (\texttt{Pixel Response Function}), that basically is the angular offset on the plane of the sky between the centroid of the star and the calculated centroid of the star during the passage of the object of interest.
  It's very important because a lot of false positives are due to the transit of another star in binary systems.
  So, if this value is big it's most likely that the object of interest is not an exoplanet.
  For some reason, the planetary radius represents another discriminant feature to determinate if an hypothetical exoplanet is real or a false positive.
  It is possible that objects that have a very small radius can be attributed to measurements noise.

  In conclusion, we can say that most of the false positive occurrences are attributed to binary star systems and measurements noise.

  \begin{figure}
    \centering
    \begin{tikzpicture}
      \begin{axis}[
        ybar,
        ylabel=Importance,
        xlabel=Feature \#,
        symbolic x coords={21, 5, 3, 6, 1},
        bar width=16pt,
        ymin=0.0, ymax=0.35,
        ytick={0.0, 0.05, 0.10, 0.15, 0.20, 0.25, 0.30, 0.35},
        yticklabel style={/pgf/number format/fixed, /pgf/number format/fixed zerofill}
      ]
        \addplot[style={fill=white}] coordinates {
          (21, 0.296)
          (5, 0.24)
          (3, 0.069)
          (6, 0.052)
          (1, 0.050)
        };
      \end{axis}
    \end{tikzpicture}
    \caption{Five most important features importance.}
    \label{figure:importances}
  \end{figure}

\clearpage
\appendix
\appendixpage

\section{Exoplanet Selected Features}
  The documentation of the following exoplanet selected features can be found at \texttt{https://exoplanetarchive.ipac.caltech.edu}.
  However most of them are auto-explicative.
  The features used in this work are regarding the observed star, the processed features of the object of interest and various spectrometric metrics.
  \begin{multicols}{2}
    \label{appendix:features}
    \begin{enumerate}
      \item Orbital Period [$days$]
      \item Transit Duration [$hrs$]
      \item Transit Depth [$ppm$]
      \item Fitted Stellar Density [$g/cm^{3}$]
      \item Planetary Radius [$Earth\:radii$]
      \item Inclination [$deg$]
      \item Equilibrium Temperature [$K$]
      \item Insolation Flux [$Earth\:flux$]
      \item Stellar Effective Temperature [$K$]
      \item Stellar Surface Gravity [$\log_{10}(cm/s^{2})$]
      \item Stellar Metallicity [$dex$]
      \item Stellar Radius [$Solar\:radii$]
      \item Stellar Mass [$Solar\:mass$]
      \item RA [$decimal\:degrees$]
      \item Dec [$decimal\:degrees$]
      \item Kepler-band [$mag$]
      \item FW $\Delta\alpha$ [$sec$]
      \item FW $\Delta\delta$ [$arcsec$]
      \item PRF $\Delta\alpha_{SQ}$ [$arcsec$]
      \item PRF $\Delta\delta_{SQ}$ [$arcsec$]
      \item PRF $\Delta\theta_{SQ}$ [$arcsec$]
    \end{enumerate}
  \end{multicols}

\section{Decision tree split criterions}
  \label{appendix:splitcriterions}
  \subsection{Gini impurity}
    Citing \texttt{wikipedia.org}, Gini impurity is a measure of how often a randomly chosen element from the set would be incorrectly labeled if it was randomly labeled according to the distribution of labels in the subset.
    To compute Gini impurity for a set of items with $J$ classes, suppose $i\in\{1,2,...,J\}$, and let $p_{i}$ be the faction of items labeled with class $i$ in the set.
    \[{I} _{G}(p)=\sum _{i=1}^{J}p_{i}\sum _{k\neq i}p_{k}=\sum _{i=1}^{J}p_{i}(1-p_{i})=1-\sum _{i=1}^{J}{p_{i}}^{2}\]
  \subsection{Information gain}
    Information gain is based on the concept of entropy and information content from information theory.
    For each node of the tree, the information value represents the expected amount of information that would be needed to specify whether a new instance should be classified yes or no, given the example reached that node.
    Given $T$ the set of training examples and $X$ the selected feature.
    \[IG(T, X)=\mathrm{H}(T)-\mathrm{H}(T|X)=\mathrm{H}(T)-\sum_{x}{\Pr(X=x)\mathrm{H}(T|X=x)}\]

\bibliography{biblio.bib}
\bibliographystyle{unsrt}

\end{document}


\subsection{Carrier trapping and Shockley-Read-Hall recombination}

\begin{figure}
To describe charge becoming trapping into trap states and recombination associated with those states the model uses Shockley-Read-Hall (SRH) theory. A 0D depiction of this SRH recombination and trapping is shown in figure \ref{fig:dos_structure}, the free electron and hole carrier distributions are labeled as n free and p free respectively. The trapped carrier populations are denoted with n trap and p trap , they are depicted with filled red and blue boxes. SRH theory describes the rates at which electrons and holes become captured and escape from the carrier traps. If one considers a single electron trap, the change in population of this trap can be described by four carrier capture and escape rates as depicted in figure \ref{fig:dos_structure}. The rate rec describes the rate at which electrons become captured into the electron trap, $r_{ee}$ is the rate which electrons can escape from the trap back to the free electron population, $r_{hc}$ is the rate at which free holes get trapped and $r_{he}$ is the rate at which holes escape back to the free hole population. Recombination is described by holes becoming captured into electron space slice through our 1D traps. Analogous processes are also defined for the hole traps.

\centering
\includegraphics[width=40mm]{./images/dos_structure.jpg}
\caption{Trap filling in both energy and position space as the solar cell is taken from a negative bias
Carrier trapping, de-trapping, and recombination}
\label{fig:dos_structure}
\end{figure}

\begin{table}
\begin{center}
  \begin{tabular}{lll}
  \hline
  Mechanism & Symbol & Description  \\
  \hline
Electron capture rate & $r_{ec}$ & $n v_{th} \sigma_{n} N_{t}(1-f)$ \\
Electron escape rate & $r_{ee}$ & $e_{n} N_{t} f$ \\
Hole capture rate & $r_{hc}$ & $p v_{th} \sigma_{p} N_{t} f$ \\
Hole escape rate & $r_{he}$ & $e_{p} N_{t} (1-f)$\\
  \hline
\end{tabular}
\end{center}
\caption{Shockley-Read-Hall trap capture and emission rates, where $f$ is the fermi-Dirac occupation function and $N_{t}$ is the trap density of a single carrier trap.}
\label{tab:rates}
\end{table}

For each trap level the carrier balance \ref{eq:srhrate} is solved, giving each trap level an independent quasi-Fermi level. Each point in position space can be allocated between 10 and 160 independent trap states.  The rates of each process $r_{ec}$, $r_{ee}$, $r_{hc}$, and $r_{he}$ are give in table \ref{tab:rates}.

\begin{equation}
\label{eq:srhrate}
\frac{\delta n_t}{\partial t}=r_{ec}-r_{ee}-r_{hc}+r_{he}
\end{equation}

The escape probabilities are given by:

\begin{equation}
\label{eq:taile}
e_n=v_{th}\sigma_{n} N_{c} exp \left ( \frac{E_t-E_c}{kT}\right )
\end{equation}

and

\begin{equation}
\label{eq:taile}
e_p=v_{th}\sigma_{p} N_{v} exp \left ( \frac{E_v-E_t}{kT}\right )
\end{equation}

 where $\sigma_{n,p}$ are the trap cross sections, $v_{th}$ is the thermal emission velocity of the carriers, and $N_{c,v}$ are the effective density of states for free electrons or holes.  The distribution of trapped states (DoS) is defined between the mobility edges as

\begin{equation}
\label{eq:taile}
\rho^{e/h}(E)=N^{e/h}exp(E/E_{u}^{e/h})
\end{equation}

where , $N_{e/h}$ is the density of trap states at the LUMO or HOMO band edge
in states/eV and where $E_{U}^{e/h}$ is slope energy of the density of states.

The value of $N_{t}$ for any given trap level is calculated by averaging the DoS function over the energy ($\Delta E$ ) which a trap occupies:

\begin{equation}
\label{eq:taile}
N_{t}(E)=\frac{\int^{E+\Delta E/2}_{E-\Delta E/2} \rho^{e}{E} dE}{\Delta E}
\end{equation}

The occupation function is given by the equation,
\begin{equation}
f(E_{t},F_{t})=\frac{1}{e^{\frac{E_{t}-F_{t}}{kT}}+1}
\end{equation}
Where, $E_{t}$ is the trap level, and $F_{t}$ is the Fermi-Level of the trap.
The carrier escape rates for electrons and holes are given by

\subsubsection{Free-to-free carrier recombination}
A free-carrier-to-free-carrier recombination pathway is also included in the model. However, most organic solar cells have a great deal of trap states and an ideality factor greater than 1.0 suggesting that free to free recombination is not the dominant mechanism.  It is also worth noting that since I included this recombination pathway in the model I have not found it useful to reproduce experimental results.

Free-to-free recombination is described using equation \ref{equ:freetofree}

\begin{equation}
R_{free}=k_{r}(n_{f}p_{f}-n_{0}p_{0})
\label{equ:freetofree}
\end{equation}


\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{amsmath, amssymb}
% \usepackage{proof}
\usepackage{syntax}
\usepackage{listings}
\usepackage{xcolor}
\usepackage{hyperref}

\title{Pixy Semantics}
\author{Reed Mullanix, Finn Hackett}
\date{February 2018}

\begin{document}

\maketitle

\section{Introduction}
The semantics of Pixy can be divided up into 3 portions: The term language, how that term language is evaluated, and the type system.
\section{Term Language}
The term language of Pixy is (roughly) as follows:
\begin{grammar}
<expr>  ::= <literal>
    \alt <var>
    \alt "nil"
    \alt "?" <expr>
    \alt "if" <expr> "then" <expr> "else" <expr>
    \alt <expr> "fby" <expr>
    \alt <expr> "where" (<var> "=" <expr>)*
    \alt "fun" <var> "=>" <expr>
    \alt <expr> <expr>
\end{grammar}

{\color{red} NOTE:} This is incomplete! We need to standardize on the term language.

\section{Evaluation}
The evaluation rules for Pixy are quite different from other languages.
To begin with, each expression can be seen as a taking a State and producing a value and a new State. This state is then fed back into the expression to produce a new State and value, and so on. However, some expressions pose some problems. For example, when evaluating \lstinline{if ... then ... else} expression, we should only really evaluate one of the branches, but doing so may skip important stateful evaluation inside of the untaken branch. To reconcile this, we present a model of evaluation which we call "Choked Evaluation". Whenever we are presented with a branching construct, we still evaluate the branches, with the caveat that all variables \textit{and} literals evaluate to \lstinline{nil} on the branch that is not taken.

Another point we need to make is that evaluation is only valid on \textbf{closed} expressions, or expressions that have no free variables.

{\color{red} NOTE:} Insert full evaluation semantics here.

{\color{red} NOTE:} We need to spec out when exactly evaluation terminates for a given step.

\section{Type Theory}
Typically, type systems follow this general form:
\begin{itemize}
    \item The user declares the construction and elimination rules for a type.
    \item The user then uses theses construction rules to create programs.
\end{itemize}
We prefer to take a different approach, which has been strongly influenced
by systems such as NuPRL. Generally speaking, our type system works as follows:
\begin{itemize}
    \item The user writes a program.
    \item The user then creates a proof that the program inhabits some type.
\end{itemize}

That of course raises the question: When does a program inhabit a type? To answer that, we must first answer what exactly a type is in Pixy. We define a type as having 2 components:
\begin{enumerate}
    \item A collection of canonical inhabitants.
    \item An equivalence relation over those inhabitants.
\end{enumerate}

For example, the canonical inhabitants of the type \lstinline{Nat} are $0,1,2,3...$ and the equivalence relationship is just the equivalence relationship of natural numbers. When we say that $a \in A$, what we are really saying is that $a = a$ under the equality relationship imposed by $A$. This point may seem slightly pedantic, but it has large implications. This can be extended to separate elements, so we could also propose that $a = b \in A$, or that 2 terms $a$ and $b$ are equivalent under the equality relation of $A$. Note that the canonical inhabitants aren't the only members of a type. Any term that evaluates to a canonical inhabitant is also a member of the type. On top of that, if we have 2 terms $t, t'$ and they evaluate to $a, a'$ respectively, and $a = a' \in A$, then $t, t' \in A$ as well!

Continuing in the spirit of NuPRL, what exactly is $a \in A$? Well, if we use the logic of Propositions-as-Types, $a \in A$ should really just be a type! We shall denote this type as $Eq\ a\ b\ A$. We shall also include all of the standard portions of Martin-Löf Type Theory.

Time is represented by using a the type $Next : \star \rightarrow \star$. This type corresponds with the $\circ$ operator in Linear Temporal logic. With this primitive type, we can begin to define the operators of Linear Temporal Logic using inductive and coinductive types.
\begin{align*}
    &\circ A \text{ corresponds to } Next(A) \\
    &\square A \text{ corresponds to } \nu \sigma . A \times Next(\sigma) \\
    &\lozenge A \text{ corresponds to } \mu \sigma . A + Next(\sigma) \\
    &A \triangleright B \text{ corresponds to } \mu \sigma . A \times Next(B + \sigma) \\
\end{align*}

There is an alternitave encoding by using temporally-indexed types, which makes quantification over time easier. This was not chosen, as the inductive/coinductive defintions make inductive/coinductive reasoning easier, which makes the kind of proofs we wish to do easier (For example, proving that 2 streams are in sync).

% {\color{red} NOTE:} This section is incomplete, as we have multiple ways of preceding. I have listed out the possible options.

% \begin{enumerate}
%     \item Use a temporally indexed dependent type. This allows us to encode
%     certain properties such as "$\forall$ Times t, ..." and "$\exists$ Time t, ..." easily.
%     \item Use a co-inductive stream type. This would allow us to more easily
%     prove relationships between 2 streams.
% \end{enumerate}

\section{Relating Programs to Types}
Note again that we do not derive the types of programs from the bottom-up, as is the norm. Rather, we prove that programs inhabit types from the top-down, using a proof refinement system.

To begin, a proof is a tree of \textbf{Judgments}, which consists of a number of \textbf{hypotheses} of the form $x:A$ followed by a \textbf{Goal}, which is of the form $term:T$. To proceed with the proof, we need to use refinement rules, which are ways of decomposing sub-goals. For example, say we had some term \lstinline{fun x => x}, and we wanted to prove that this term is a member of $Bool \rightarrow Bool$. An example proof would be as follows:
\begin{verbatim}
    H >> (fun x => x) in Bool -> Bool by intro-function.
        x:Bool, H >> x in Bool by hypothesis x.
        H >> Bool in U by bool-intro-universe.
\end{verbatim}
Note that we use 3 rules here, \lstinline{intro-function}, \lstinline{hypothesis}, and \lstinline{intro-universe}. These correspond to the standard type inference rules, but there is a catch: We cannot infer the types. This is because a term can inhabit many potential types. For example, we could also prove that \lstinline{fun x => x} inhabits the type $\Pi_{A:U}.A \rightarrow A$:
\begin{verbatim}
    H >> (fun x => x) in (A:U) -> A -> A by intro-function-pi
        A:U, x:A, H >> x in A by hypothesis x.
\end{verbatim}

{\color{red} NOTE:} The above rule needs some thinking about. As such, I have decided to not include it in the rule section yet.

{\color{red} NOTE:} Write some examples that show how to use the rules to prove nil-safety.

\section{Rules}
\subsection{Bool}
\begin{verbatim}
    H >> true in Bool by intro-true.
    H >> false in Bool by intro-false
    H >> Bool in U1 by bool-intro-universe.
\end{verbatim}
\subsection{Nil}
\begin{verbatim}
    H >> nil in Nil by intro-nil.
    -- TODO: Write the choking type rules.
    -- Needs clarification
\end{verbatim}
\subsection{Functions}
\begin{verbatim}
    H >> (fun x => b) in (x:A) -> B by intro-function.
        x:A, H >> b in B.
        H >> A in Ui.
        H >> B(x) in Ui. -- We may have to be careful about universe levels?
    H >> (x:A) -> B in Ui by function-intro-universe.
        H >> A in Ui.
        H >> B in Ui.
\end{verbatim}
\subsection{Universes}
\begin{verbatim}
    H >> Ui in Uj by universe-cumulative.
        -- Note, i < j.
\end{verbatim}

\section{Bibliography}
{\color{red} NOTE:} This Section needs to be properly formatted, right now it is just a dumping ground for things that have influenced this work.\\
\url{http://www.nuprl.org/book/} \\
\url{http://www.nuprl.org/documents/Constable/naive.pdf} \\
\url{http://ect.bell-labs.com/who/ajeffrey/papers/plpv12.pdf} \\

\end{document}


%---------------------------------------------------------------------
%  Writeup of the improved search of Run II data for single top quark
%  production at DZero.
%  Started: Oct 2006
%  Authors: The Single Top Working Group
%---------------------------------------------------------------------
%

\appendix
\section*{Appendix 3 --- Normalization of Non-Top Backgrounds to Data}
\label{appendix-matrixmethod}

We use the matrix method~\cite{mm_explained} to estimate how many
events in the preseleceted sample contain a fake lepton (originating
from QCD-multijet production) and how many events have a real isolated
lepton (like the ones originating from $W$+jets or $\ttbar$). Two
data samples are defined: the \emph{tight} sample, which is the signal
sample after all selection cuts have been applied; and the
\emph{loose} sample, where the same selection has been applied but
requiring only loose lepton isolation, thus omitting the tight
requirements in the lepton isolation.

The tight data sample, with $N_{{\rm tight}}$ events, is a subset of
the loose data sample with $N_{\rm loose}$ events. The loose sample
contains $N_{\rm loose}^{{\rm real-}\ell}$ events with a real lepton
(signal-like events, mostly $W$+jets and $\ttbar$) and $N_{\rm
loose}^{{\rm fake-}\ell}$ fake lepton events, which is the number of
QCD events in the loose sample. We measure the efficiency for a real
isolated lepton to pass the tight lepton isolation selection and call
it $\varepsilon_{{\rm real-}\ell}$; and the efficiency for a fake
lepton to pass the tight lepton isolation criteria, $\varepsilon_{{\rm
fake-}\ell}$.

With these definitions, the matrix method is applied by using these
two equations:
\begin{eqnarray*}
N_{\rm loose}   & = & N_{\rm loose}^{{\rm fake-}\ell}
                      + N_{\rm loose}^{{\rm real-}\ell} \\
N_{{\rm tight}} & = & N_{{\rm tight}}^{{\rm fake-}\ell}
                      + N_{{\rm tight}}^{{\rm real-}\ell}
                      = \varepsilon_{{\rm fake-}\ell} \;
                        N_{\rm loose}^{{\rm fake-}\ell}
                      + \varepsilon_{{\rm real-}\ell} \;
                        N_{\rm loose}^{{\rm real-}\ell}
\end{eqnarray*}
\noindent to solve for $N_{\rm loose}^{{\rm fake-}\ell}$ and $N_{\rm
loose}^{{\rm real-}\ell}$ so that the QCD and the $W$-like
contributions in the tight sample $N_{\rm tight}^{{\rm fake-}\ell}$
and $N_{\rm tight}^{{\rm real-}\ell}$ can be determined.

The results of the matrix method, which we apply separately in each
jet multiplicity bin, are shown in Table~\ref{mm-numbers}. The
pretagged orthogonal sample is scaled to $N_{\rm tight}^{{\rm
fake-}\ell}$ and the $W$+jets MC samples
($Wb\bar{b}$+$Wc\bar{c}$+$Wjj$) are scaled to $N_{\rm tight}^{{\rm
real-}\ell}$, after subtracting the expected $\ttbar$ events in each
bin of the tight sample.

\vspace{0.2in}
\begin{table}[!h!tbp]
\begin{center}
\begin{minipage}{6.5in}
\begin{ruledtabular}
\begin{tabular}{l||ccccc|ccccc}
\multicolumn{11}{c}{\hspace{1in}\underline{Normalization of $W$+Jets and Multijets to Data}}\vspace{0.1in} \\
& \multicolumn{5}{c|}{Electron Channel} & \multicolumn{5}{c}{Muon Channel}    \\
                               & 1 jet  & 2 jets & 3 jets & 4 jets & 5+ jets
                               & 1 jet  & 2 jets & 3 jets & 4 jets & 5+ jets \\
\hline
$N_{{\rm loose}}$              & 38,935 & 15,213 &  7,118 &  2,191 &   654  & 18,714 &  7,092 &  3,054 &   878  &   221 \\
$N_{{\rm tight}}$              & 27,370 &  8,220 &  3,075 &    874 &   223  & 17,816 &  6,432 &  2,590 &   727  &   173 \\
$N_{\rm tight}^{{\rm fake-}e}$ &  1,691 &  1,433 &    860 &    256 &    86  &    498 &    329 &    223 &    56  &    10 \\
$N_{\rm tight}^{{\rm real-}e}$ & 25,679 &  6,787 &  2,215 &    618 &   137  & 17,319 &  6,105 &  2,369 &   669  &   162 \\
$\varepsilon_{{\rm real-}e}$   &   0.87 &   0.87 &   0.87 &   0.87 &  0.87  &   0.99 &   0.99 &   0.99 &  0.96  &  0.88 \\
$\varepsilon_{{\rm fake-}e}$   &   0.18 &   0.19 &   0.19 &   0.17 &  0.17  &   0.41 &   0.36 &   0.34 &  0.31  &  0.25
\end{tabular}
\end{ruledtabular}
\vspace{-0.1in}
\caption[mmnumbers]{Matrix method yields in the electron and muon channels:
the loose and tight selected events and the expected contribution from
multijet and $W$-like events.}
\label{mm-numbers}
\end{minipage}
\end{center}
\end{table}

% from the old muon-only table:
%$N_{\rm tight}^{{\rm real-}\mu}$($W$+jets) & 17,313 &  6,064 &  2,276 &   563  &   116 \\
%$N_{\rm tight}^{{\rm real-}\mu}$($\ttbar$) &      6 &     41 &     93 &   106  &    46 \\

\begin{figure}[!h!tbp]
\begin{center}
\includegraphics[width=0.32\textwidth]{figures/electron/cc_EqOneJet_PreTag_WTransverseMass.eps
\includegraphics[width=0.32\textwidth]{figures/electron/cc_EqTwoJet_PreTag_WTransverseMass.eps
\includegraphics[width=0.32\textwidth]{figures/electron/cc_EqThreeJet_PreTag_WTransverseMass.eps
\includegraphics[width=0.32\textwidth]{figures/muon/mu_EqOneJet_PreTag_WTransverseMass.eps
\includegraphics[width=0.32\textwidth]{figures/muon/mu_EqTwoJet_PreTag_WTransverseMass.eps
\includegraphics[width=0.32\textwidth]{figures/muon/mu_EqThreeJet_PreTag_WTransverseMass.eps
\end{center}
\vspace{-0.1in}
\caption[mu_2jet]{The leading jet $\pt$, second leading jet $\pt$, the muon
$\pt$, the $\met$ and the $\Delta R$(jet1,jet2) after selection but before tagging (first column), with 1
$b$-tag (center column) and with 2 $b$-tags (right column).}
\label{fig:mu_2jet}
\end{figure}


\subsection{Controller Design using Dead-Time and PT1 element}

Now that $T_u$, $T_g$, $K_{p,crit}$ and $\tau_{crit}$ have been determined, we
use the formulas  in tables \ref{tab:ziegler_nichols}, \ref{tab:CHR_rejection}
and \ref{tab:CHR_tracking} to create various  P,  PI  and PID controllers. The
resulting    closed   loop   step   functions   are   plotted    in    figures
\ref{fig:Tt_PT1_P},     \ref{fig:Tt_PT1_PI}      and     \ref{fig:Tt_PT1_PID}.

The  P-controller   obtained   from   the   Ziegler-Nichols  method  is  quite
unbelievable. The motor requires about 30  seconds  to reach its end speed, so
why is  it  able  to  reach  its  target  value  after  just three seconds? By
simulating what happens on the output of the controller (the voltage that gets
fed   into    the    motor),    we   see   what's   going   on   (see   figure
\ref{fig:Tt_PT1_P_voltage}).

The  voltage  levels  being  used  to  control the motor  grossly  exceed  the
capabilities of the driver and the maximum rating of the motor. One would have
to  decrease the $K_p$ parameter in this controller until the voltage  reaches
acceptable levels again in order to make this controller feasible.  The  issue
then,  though,  is  that  the  target  value  doesn't  get  reached  any more.

\begin{figure}[h]
    \centering
    \includegraphics[width=\imagewidth]{images/Tt_PT1_P}
    \caption{Various P controllers}
    \label{fig:Tt_PT1_P}
\end{figure}

\begin{figure}[h]
    \centering
    \includegraphics[width=\imagewidth]{images/Tt_PT1_PI}
    \caption{Various PI controllers}
    \label{fig:Tt_PT1_PI}
\end{figure}

\begin{figure}[h]
    \centering
    \includegraphics[width=\imagewidth]{images/Tt_PT1_PID}
    \caption{Various PID controllers}
    \label{fig:Tt_PT1_PID}
\end{figure}

\begin{figure}[t]
    \centering
    \includegraphics[width=\imagewidth]{images/Tt_PT1_P_voltages}
    \caption{Simulated voltage on the input of the motor}
    \label{fig:Tt_PT1_P_voltage}
\end{figure}

\clearpage


\documentclass[12pt]{article}
%\usepackage{a4wide}
\usepackage[margin=2.5cm,a4paper]{geometry}
%\usepackage[latin1]{inputenc}
\usepackage{amsmath}
\usepackage{amsthm}
\usepackage{paralist}
\usepackage{bbm}
\usepackage{amssymb}
\usepackage{pdfpages}
\usepackage[T1]{fontenc}

\usepackage[pdftex,bookmarks,colorlinks,breaklinks,pdfpagelabels]{hyperref}

\usepackage{epic,eepic,latexsym,verbatim}
%\usepackage{youngtab}
%\usepackage{bbm}
\newcommand{\ket}[1]{|#1\rangle}
\newcommand{\bra}[1]{\langle #1|}
\newcommand{\bracket}[2]{\langle #1|#2\rangle}
\newcommand{\ketbra}[1]{|#1\rangle\langle #1|}
\newcommand{\average}[1]{\langle #1\rangle}
\newcommand{\minus}{\!-\!}

\newcommand{\om}{\omega}
%\newcommand{\th}{\theta}
\newcommand{\rn}{\mathbb{R}^n}
\newcommand{\R}{\mathbb{R}}
\setlength{\parindent}{0pt}
\newtheorem{theorem}{Theorem}
\newcommand{\question}[2]{{\bf #1} {\em #2}\\}
\newcommand{\problem}[3]{{\bf #1} {\em #2}\\[5\lineskip]
{#3}}

\newcounter{dummy}
\setcounter{dummy}{0}
\newtheorem{p}{Exercise}[dummy]
\renewcommand\thep{\arabic{p}}
\setcounter{secnumdepth}{0}
\overfullrule=5pt

\author{Joseph M.\ Renes}
\title{Solutions to Baez and Muniain's \\\emph{Gauge Fields, Knots and Gravity}}
\begin{document}
\renewcommand{\abstractname}{\vspace{-\baselineskip}}

\hypersetup{pageanchor=true}

\maketitle

\begin{abstract}
Here are my solutions to some of the problems in \emph{Gauge Fields, Knots and Gravity}.
At some point I also wrote up a few notes on the Lie derivative in the style of the book.\\

version 1.0 --- Importing original writeup of solutions and Lie derivative.
\end{abstract}

\pdfbookmark[1]{\contentsname}{Contents}
\tableofcontents

\newpage

\section{Part I}
\input{src/I1}

\input{src/I2}

\input{src/I3}

\input{src/I4}

\input{src/I5}

\input{src/I6}

\section{Part II}
\setcounter{dummy}{1}
\setcounter{p}{0}

\input{src/II1}

\setcounter{p}{54}

\input{src/II2}

\input{src/II3}

%\section{The Lie derivative}

%\pdfbookmark[1]{The Lie derivative}{lie}
\includepdf[pages=-,addtotoc={1,section,1,The Lie derivative,lie}
]{src/lie.pdf}

\includepdf[pages=-]{src/expmap.pdf}

\end{document}


\chapter*{Introduction}

In math, people prefer to use ``$ \log_e $'' or just ``$ \log $'' rather than ``$ \ln $'', which is the opposite to many scientific fields. Due to its simplicity (especially when typing), I will also stick to the usage of ``$ \ln $'' which is identical to ``$ \log_e $''. But to avoid any confusion, I will always specify the base 10 (``$ \lg $'').

At many places in physics and astronomy, people use $ \equiv $ as ``is defined as'' or any kind of ``trivial by the definition'' cases. But I will use $ := $ for the definition and use $ \equiv $ to mean ``equality holds trivially by the definition''. For example, under 1-D linear homogeneous isotropic matter, linear electric polarization is written as $ P \equiv \epsilon_0 \chi E $ or $ \chi := P/\epsilon_0 E $. The latter is the \textit{definition} of electric susceptability, while the former is \textit{trivially true by the definition}.

For distributions, $ \sim $ means ``follows'', i.e., $ X \sim \mathcal{N}(\mu, \sigma^2) $ means ``the random variable $ X $ follows the Gaussian (normal) distribution with mean $ \mu $ and standard deviation $ \sigma $ (thus variance $ \sigma^2 $).'' When this is an approximation, the symbol with dot ``$ \simdot $'' is used, such as $ X \simdot \mathcal{N}(\mu, \sigma^2) $. In this case, it is interpreted as ``approximately follows.''

In this note, I used some math-like notations like theorem (Thm). However, the theorem here is not necessarily the same as that of mathematics. I tried to put some important assumptions and summary notes to theorem.

The references are
\begin{itemize}
\item ``Walpole'' is Walpole et al. 2013, ``Essentials of Probabilities and Statistics''.
\item ``Sivia'' is Sivia and Skilling 2006, ``Data Analysis A Bayesian Tutorial'', 2/e.
\end{itemize}


\documentclass[10pt,journal,compsoc]{IEEEtran}

\usepackage[pdftex]{graphicx}
\usepackage{cite}
\usepackage{subcaption}
\hyphenation{op-tical net-works semi-conduc-tor}
\graphicspath{ {../images/} }

\begin{document}

\title{Robotics Software Engineer Nanodegree: Deep RL Arm Manipulation}
\author{Manuel Huertas L\'opez}

\markboth{Deep RL Arm Manipulation project, Robotic Nanodegree, Udacity}%
{}

\IEEEtitleabstractindextext{%

\begin{abstract}

Reinforcement learning is an area of machine learning. It instructs an agent to take actions in an environment in order to maximize a cumulative reward. In this project a DQN algorithm will be used to guide a robotic arm to touch a cylindrical object. The idea behind the algorithm is to find an action value function; this function will give the cumulative expected reward for every state and action pair values. Once this function is obtained the action for every state will be the one that maximizes the reward. DQN is a variation of the algorithm where this function is replaced with a Deep Neural Network.

\end{abstract}

\begin{IEEEkeywords}
Robot, Udacity, Deep Reinforcemente Learning.
\end{IEEEkeywords}}

\maketitle
\IEEEdisplaynontitleabstractindextext
\IEEEpeerreviewmaketitle
\section{Introduction}

In this project a Robotic Arm with three degree of freedom will be taught in order to touch a cylindrical shape in the scene. Rather than using the forward and backward kinematic to calculate the position of the arm with respect to the joint angles or to move the arm to a goal position sending demands to the joints, the robot will try some random movements and it will receive a positive or negative reward. A video camera will be recording constatinualsy the robot movement and this information is used to train a neural network, the network will try to reproduce the sequence of movement that maximise the cumulative reward. As the final goal the robot will find the combination of joint movement, without a model, to approximate and finally touch the object in the scene.

Every Episodic finalized when the robot: touch the ground, touch the cylinder or the maximum number of iteration is reached. The final reward will be positive just in case the robot touch the cylinder, and there are some intermediate positive reward bases on smooth movement and when the robot is closer to the cylinder. The problem can be defined as a Markov Decision Process \cite{udacity} where:
\begin{itemize}
\item a finite set of states: the image from the video camera and all the pixel value combinations.
\item a finite set of actions: the movement every joint.
\item a finite set of rewards: the positive and negative reward depending on the robot behaviour.
\item the one-step dynamics of the environment: given the current state and the action, the expected new state and new reward. The neural network will obtain this during thetrainingg process.
\item a discount rate between [0,1]: the discount rate.
\end{itemize}

\label{sec:introduction}

\section{Background / Formulation}

Our goal is to find a function that for every state, the image in this case, tell use which action to take, the movement of the joint. This problem is not easy since there are a big number of states, pixel combination of the image, and sequences of movements. In this project an algorithm based in DQN have been used. In DQN a Deep Neural Network will build a function where the input will be the pixel of the image and the output the expected cumulative reward for every possible action. Selecting the action with the highest cumulative reward will give us the desired action. In addition, a Long Short Term Memory network have been used to use the information of the sequence of multiple images, instead of a single frame.

\subsection{Deep Q Network}

The Q-Learning algoritm is a model-free algoritm. The values for each state-action pair are estimated based on observation of the environment \cite{deepreinforcementlearning}. The Q-Learning algoritm estimate the state action funcition using the following equation:

\begin{align}
Q(s_{t},a_{t})=(1-\alpha) \cdot Q(s_{t},a_{t})+\alpha \cdot (r_{t}+\gamma max_a\cdot Q(s_{t+1},a))
\end{align}

The meaning of this parameter are:

\begin{itemize}
\item s: state value at time t, from the observation.
\item a: action value a time t.
\item alpha: learning rate, the way in which new values replaces old ones.
\item r: reward at time t.
\item gamma: discount factor.
\end{itemize}

In addition, the greedy policy is used to select between a current best action, exploit, or to select a random action, explore. If during the learning process we always select the action with currently the best reward we are never going to explore new solutions whose can give a better discounted future reward. On the other one, we need to select the current best action in order to use the knowledge we are acquiring iteratively. One way to accomplish this is to explore more at the beginning of the learning process, and gradually reduce the exploration favouring the exploitation of the current knowledge. For doing this the following parameters are defined:

\begin{itemize}
\item EPS\_START: starging gready value.
\item EPS\_END: ending gready value.
\item EPS\_DECAY: gready decay rate
\end{itemize}

Finally, the state action function is replaced with a deep neural network. The input of the network is the image from the video camera, and the output the reward or every action. The loss function, for the gradient descent algorithm, is defined as:

\begin{align}
Loss=1/2 \cdot [r_{t} + max_{a_{t+1}}\cdot Q(s_{t+a},a_{t+1};\theta_{t-1}}) - Q(s,a;\theta}]^2
\end{align}

Two extra techniques are included to make the neural network converge to the final solution. Experience replay, where a memory of experience is used to store previous values; mini-batches of these values are used as input of the neural network during learning. This technique helps to break the temporal dependencies of the learning process. Since the target of the loss function and the predictions have parameters in common, the process of learning can not converge to the correct solution, fixed Q targets is used to update the Q function after several iterations, in this way the small changes in the Q function approximation are not affecting the learning process.

The following extra parameters are defined.

\begin{itemize}
\item  INPUT\_WIDTH: Image from the camera, width
\item  INPUT\_HEIGHT: Image from the camera, height
\item  OPTIMIZER: Optimizer for the gradient descent algorithm
\item  LEARNING\_RATE: Neural Network learning rate.
\item  REPLAY\_MEMORY: Size of the memory for the experience replay technique.
\item  BATCH\_SIZE: Batch size for the experience replay technique.
\end{itemize}

\subsection{Long Short Term Memory}

LSTM is a special Recurrent Neural Network, capable of learning long-term dependencies. The Long short-term memory adds memory to the DQN to remember the sequence of movements of the joint to find and touch the object in the scene \cite{lstmnetwork}.

\section{Robotic Arm}

In a reinforcement learning problem, a temporary series of interactions are produced until the episodic ends: S_{0},A_{0},R_{1},S_{1},A_{1},R_{2},...,R_{T},S_{T}

In this project, an episode ends when: the robot's arm touches the ground, a maximum of iterations is reached, or the robot touches the object in the scene; the states are the image, captured by the video camera, of the scene; the actions can be to move the joints at random using position or speed control; finally, the reward strategy is defined as a negative number when the robot finishes the episodic without touching the object, the positive number when it touches the object and an incremental or decremental reward when the arm approaches the target or not.

Different reward strategies were tested, the final choose solution was: a big number positive number when the robot touches the object; a negative number when the robot touch the ground or the maximum iterations are reached, to this negative number a 1\% of the distance to the goal is added, in order to decrease the negative reward depending on the closest to the goal; a smooth reward when the robot is searching the object in function of the smoothest of the movement and the distance to the goal.

\subsection{Rewards}

The positive reward when the object is touched is 1000000.0. The negative base reward is -1.0. When the episodice ends without success to the negative base reward is added 1\% the distance to the goal. Finally when the Robot is searching a soomoth average goal delta is added to the distance to the goal as follow:

AverageGoalDelta  = ( AverageGoalDelta * 0.2 ) + (DistanceDelta * (1-0.2));

Reward Searching  = AverageGoalDelta + 0.01*exp(-pow(DistanceGoal,2));

The same rewards were used for both objectives.

\subsection{Hiper Paramters}

Finally the hiper-parameters selecting were:

\begin{itemize}
\item  VELOCITY\_CONTROL[false]: The position control was used.
\item  NUMBER\_ACTIONS[6]: The number of actions was calculated as number of DOF x 2.
\item  ALLOW\_RANDOM[true]: The learnning process allows random decisions to explore new solutions.
\item  GAMMA[0.9]: discount factor, selected was 0.9.
\item  EPS\_START[0.9]: Starting greedy value.
\item  EPS\_END[0.01] Ending greedy value.
\item  EPS\_DECAY[200]: Greedy decay rate
\item  INPUT\_WIDTH[64]: Image from the camera width
\item  INPUT\_HEIGHT[64]: Image from the camera height
\item  OPTIMIZER[RMSprop]: Optimizer for the gradient descent algorithm
\item  LEARNING\_RATE[0.01]: Neural Network learning rate.
\item  REPLAY\_MEMORY[10000]: Size of the memory for the experience replay technique.
\item  BATCH\_SIZE[64]: Batch size for the experience replay technique.
\item  USE\_LSTM[true]: Uses Long Short Memory.
\item  LSTM\_SIZE[512]: Sizes of the LSTM network.
\end{itemize}

The same Hiper Parameters were used for both objectives.

\section{Results}

In order to find the reward and hiper-parameters a trial and error strategy was used. At the beginning velocity control was used, but the desired accuracy of 80\% of success touching the object with the gripper base was not achieve, for that reason the position control was selected. The reward strategy consists in give a high positive number when the robot touches the object, this big number was selected because the robot could accumulate small rewards during the searching process and it was important to stimulate the robot not to keep searching but to move toward the goal. During the searching process a small amount of the distance to goal was added to the reward to indicate the robot's arm that the action should move the arm in the object direction. Also, a factor depending on of the delta movement, was used to increase, when the movement was directing the robot to the goal, or decrease when the movement was in the other direction. This searching strategy was used to indicate the robot's arm to keep moving and to do it smoothly  to the goal direction. Finally, when the robot ends the episodic without success, a negative reward is given. To this negative reward a factor of the distance to the goal is added to indicate the robot how close was to the solution, even having failed.

\begin{figure}[h]
\centering
\begin{subfigure}[b]{0.4\textwidth}
\includegraphics[scale=0.25]{Learnning_Collision_ARM}
\caption{Robot's Arm touching the object}
\end{subfigure}
\begin{subfigure}[b]{0.4\textwidth}
\includegraphics[scale=0.27]{Learnning_Collision_Gripper}
\caption{Robot's Arm touching the object with the base gripper}
\end{subfigure}
\caption{Robot's Arm}
\end{figure}

\section{Discussion}

The project consists in completing eight tasks:

\begin{enumerate}
\item  Subscribe to camera and collision topics: This was easily achieved following the udacity instructions.
\item  Create the DQN Agent: Udacity provided the class, it was just necessary to create an instance.
\item  Velocity or position based control of arm joints: position control was selected because it was easier to define the rewards.
\item Reward for robot gripper hitting the ground: using a bounding box for the gripper it was detected when it was hitting the ground; a negative reward plus a 1\% to the distance to the goal was used.
\item Issue an interim reward based on the distance to the object: an incremental, reward positive or negative, was used, depending on the robot movement to the goal or not.
\item Issue a reward based on collision between the arm and the object: a very high number was used to stimulate the robot to move to the goal.
\item Tuning the hyper parameters: a trail an error approach was used to find the final parameters.
\item  Issue a reward based on collision between the arm’s gripper base and the object: the same reward and hyper parameters were used for the two objectives.
\end{itemize}

\section{Conclusion / Future work}

In order to improve the project a deeper understating about how the reward affects the learning process will be needed. A way to do this can be to visualize the Q-Function using a 2D plot where the x-axis can be the actions, y-axis the state, and the coloured value of the graph the reward. The rewards were selected using some logic about how the human will learn to solve the problem. Different strategies will be needed to be explored in order to find a solution that converges quicklier to the final solution.

% Bibliography
\begin{thebibliography}{9}
\bibliographystyle{ieeetr}

\bibitem{udacity}
Robotics Software Engineer Nanodegree program
\textit{https://eu.udacity.com}

\bibitem{deepreinforcementlearning}
\textit{Human-level control through deep reinforcement learning}.

\bibitem{lstmnetwork}
Understanding LSTM Networks
\textit{http://colah.github.io/posts/2015-08-Understanding-LSTMs}

\end{thebibliography}

\end{document}


% Created 2022-05-03 Tue 11:38
% Intended LaTeX compiler: pdflatex
\documentclass[t]{beamer}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{graphicx}
\usepackage{longtable}
\usepackage{wrapfig}
\usepackage{rotating}
\usepackage[normalem]{ulem}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{capt-of}
\usepackage{hyperref}
\mode<beamer>{\usetheme{Amsterdam}}
\mode<beamer>{\usecolortheme{rose}}
\usepackage{fontspec}
\usepackage{polyglossia}
\setmainlanguage[babelshorthands=true]{german}
\usepackage{hyperref}
\usepackage{color}
\usepackage{xcolor}
\usepackage[misc]{ifsym}
\definecolor{darkblue}{rgb}{0,0,.5}
\definecolor{darkgreen}{rgb}{0,.5,0}
\definecolor{islamicgreen}{rgb}{0.0, 0.56, 0.0}
\definecolor{darkred}{rgb}{0.5,0,0}
\definecolor{mintedbg}{rgb}{0.95,0.95,0.95}
\definecolor{arsenic}{rgb}{0.23, 0.27, 0.29}
\definecolor{prussianblue}{rgb}{0.0, 0.19, 0.33}
\definecolor{coolblack}{rgb}{0.0, 0.18, 0.39}
\hypersetup{colorlinks=true, breaklinks=true, anchorcolor=blue,linkcolor=white, citecolor=islamicgreen, filecolor=darkred,  urlcolor=darkblue}
\usepackage{booktabs}
\usepackage{pgf}
\usepackage{minted}
\RequirePackage{fancyvrb}
\DefineVerbatimEnvironment{verbatim}{Verbatim}{fontsize=\scriptsize}
\usetheme{default}
\author{Göran Kirchner\thanks{e\_kirchnerg@doz.hwr-berlin.de}}
\date{2021-05-03}
\title{Funktionale Programmierung in F\# (5)}
\subtitle{Parser Combinators}
\hypersetup{
 pdfauthor={Göran Kirchner},
 pdftitle={Funktionale Programmierung in F\# (5)},
 pdfkeywords={},
 pdfsubject={},
 pdfcreator={Emacs 27.2 (Org mode 9.5.2)},
 pdflang={English}}
\begin{document}

\maketitle

\section{Ziel }
\label{sec:org0ee434b}
\begin{frame}[label={sec:orgc9d6c29}]{Programm}
\begin{itemize}
\item Hausaufgaben
\item Test
\item Parser (Kombinatoren)
\end{itemize}
\end{frame}

\section{Hausaufgaben }
\label{sec:org62f30a3}
\begin{frame}[label={sec:orgfc210c3},fragile]{Accumulate}
 \begin{minted}[bgcolor=mintedbg,frame=none,framesep=0pt,mathescape=true,fontsize=\scriptsize,breaklines=true,linenos=false,numbersep=5pt,gobble=0]{fsharp}
let rec accumulateR func input acc =
    match input with
    | [] -> acc |> List.rev
    | head::tail -> accumulateR func tail (func head :: acc)
let accumulate func input = accumulateR func input []
let test1 = accumulate (fun x -> x * x) [1; 2; 3]
let test2 = accumulate (fun (x:string) -> x.ToUpper()) ["hello"; "world"]
test2
\end{minted}

\begin{verbatim}
val accumulateR: func: ('a -> 'b) -> input: 'a list -> acc: 'b list -> 'b list
val accumulate: func: ('a -> 'b) -> input: 'a list -> 'b list
val test1: int list = [1; 4; 9]
val test2: string list = ["HELLO"; "WORLD"]
val it: string list = ["HELLO"; "WORLD"]
\end{verbatim}
\end{frame}

\begin{frame}[label={sec:orgd3a40f9},fragile]{Space Age}
 \begin{minted}[bgcolor=mintedbg,frame=none,framesep=0pt,mathescape=true,fontsize=\scriptsize,breaklines=true,linenos=false,numbersep=5pt,gobble=0]{fsharp}
type Planet =
    | Mercury
    | Venus
    | Earth
    | Mars
    | Jupiter
    | Saturn
    | Uranus
    | Neptune
let orbitalPeriodRelativeToEarthOn planet =
    match planet with
    | Mercury -> 0.2408467
    | Venus -> 0.61519726
    | Earth -> 1.0
    | Mars -> 1.8808158
    | Jupiter -> 11.862615
    | Saturn -> 29.447498
    | Uranus -> 84.016846
    | Neptune -> 164.79132
\end{minted}

\begin{verbatim}
type Planet =
  | Mercury
  | Venus
  | Earth
  | Mars
  | Jupiter
  | Saturn
  | Uranus
  | Neptune
val orbitalPeriodRelativeToEarthOn: planet: Planet -> float
\end{verbatim}
\end{frame}

\begin{frame}[label={sec:org155f1d4},fragile]{Space Age (II)}
 \begin{minted}[bgcolor=mintedbg,frame=none,framesep=0pt,mathescape=true,fontsize=\scriptsize,breaklines=true,linenos=false,numbersep=5pt,gobble=0]{fsharp}
open System
[<Literal>]
let SecondsInOneEarthYear = 31557600.0
let secondsInAYearOn planet =
    SecondsInOneEarthYear * orbitalPeriodRelativeToEarthOn planet
let round (number : float) = Math.Round(number, 2)
let age (planet: Planet) (seconds: int64): float =
    float seconds / (secondsInAYearOn planet)
    |> round
let test1 = age Earth 1000000000L
\end{minted}

\begin{verbatim}
[<Literal>]
val SecondsInOneEarthYear: float = 31557600
val secondsInAYearOn: planet: Planet -> float
val round: number: float -> float
val age: planet: Planet -> seconds: int64 -> float
val test1: float = 31.69
\end{verbatim}
\end{frame}

\begin{frame}[label={sec:org48530c4},fragile]{Zusammenfassung}
 \begin{itemize}
\item nutze \href{https://exercism.io}{exercism.io}!
\item Formatierung (dotnet fantomas)
\item Vermeide \texttt{mutable}!!
\item nur wichtiges verdient einen Namen
\item Vertraue der \alert{Pipe} (\texttt{>>}, \texttt{|>}, \ldots{})!!
\item If-Then-Else mit Boolean ist unnötig
\item Parametrisiere!
\item If-Then-Else vermeiden \ldots{} besser \texttt{match}!
\item Be lazy! (vermeide \texttt{for}-loops)
\item \href{https://fsharpforfunandprofit.com/troubleshooting-fsharp/}{Troubleshooting F\#}
\item \href{https://docs.microsoft.com/de-de/dotnet/fsharp/style-guide/}{F\#-Styleguide}
\end{itemize}
\end{frame}

\section{Test }
\label{sec:org3061079}
\begin{frame}[label={sec:orgd6d78e6}]{Test}
\begin{itemize}
\item 90 Minuten
\item Ergebnis per \href{mailto://e\_kirchnerg@doz.hwr-berlin.de}{E-Mail} an \href{mailto://e\_kirchnerg@doz.hwr-berlin.de}{e\_kirchnerg@doz.hwr-berlin.de}.
\end{itemize}

\(\leadsto\) \href{../src/5/test.md}{Test}
\end{frame}

\section{Parser }
\label{sec:org0caa37e}
\begin{frame}[label={sec:orga6ffe1c},fragile]{Parser 1 (hard-coded character)}
 \begin{minted}[bgcolor=mintedbg,frame=none,framesep=0pt,mathescape=true,fontsize=\scriptsize,breaklines=true,linenos=false,numbersep=5pt,gobble=0]{fsharp}
open System
let A_Parser str =
    if String.IsNullOrEmpty(str) then
        (false,"")
    else if str.[0] = 'A' then
        let remaining = str.[1..]
        (true,remaining)
    else
        (false,str)
let inputABC = "ABCD";;
let inputZBC = "ZBCD";;
let test11 = A_Parser inputABC
let test12 = A_Parser inputZBC
\end{minted}

\begin{verbatim}
val test11: bool * string = (true, "BCD")
val test12: bool * string = (false, "ZBCD")
\end{verbatim}
\end{frame}

\begin{frame}[label={sec:orgc4ae52c},fragile]{Parser 2 (match a specified character)}
 \begin{minted}[bgcolor=mintedbg,frame=none,framesep=0pt,mathescape=true,fontsize=\scriptsize,breaklines=true,linenos=false,numbersep=5pt,gobble=0]{fsharp}
let pchar (charToMatch,str) =
    if String.IsNullOrEmpty(str) then
        let msg = "No more input"
        (msg,"")
    else
        let first = str.[0]
        if first = charToMatch then
            let remaining = str.[1..]
            let msg = sprintf "Found %c" charToMatch
            (msg,remaining)
        else
            let msg = sprintf "Expecting '%c'. Got '%c'" charToMatch first
            (msg,str)
\end{minted}

\begin{verbatim}
val pchar: charToMatch: char * str: string -> string * string
\end{verbatim}
\end{frame}

\begin{frame}[label={sec:org82f0e21},fragile]{Parser 2 (2)}
 \begin{minted}[bgcolor=mintedbg,frame=none,framesep=0pt,mathescape=true,fontsize=\scriptsize,breaklines=true,linenos=false,numbersep=5pt,gobble=0]{fsharp}
let inputABC = "ABCD";;
let inputZBC = "ZBCD";;
let test21 = pchar('A',inputABC)
let test22 = pchar('A',inputZBC)
\end{minted}

\begin{verbatim}
val test21: string * string = ("Found A", "BCD")
val test22: string * string = ("Expecting 'A'. Got 'Z'", "ZBCD")
\end{verbatim}
\end{frame}

\begin{frame}[label={sec:orgfd2993e},fragile]{Parser 3 (return a Result)}
 \begin{minted}[bgcolor=mintedbg,frame=none,framesep=0pt,mathescape=true,fontsize=\scriptsize,breaklines=true,linenos=false,numbersep=5pt,gobble=0]{fsharp}
let pchar (charToMatch, s) =
    if String.IsNullOrEmpty(s) then
        Error "No more input"
    else
        let first = s.[0]
        if first = charToMatch then
            let remaining = s.[1..]
            Ok (charToMatch, remaining)
        else
            let msg = sprintf "Expecting '%c'. Got '%c'" charToMatch first
            Error msg
\end{minted}

\begin{verbatim}
val pchar: charToMatch: char * s: string -> Result<(char * string),string>
\end{verbatim}
\end{frame}

\begin{frame}[label={sec:org4f16965},fragile]{Parser 3 (2)}
 \begin{minted}[bgcolor=mintedbg,frame=none,framesep=0pt,mathescape=true,fontsize=\scriptsize,breaklines=true,linenos=false,numbersep=5pt,gobble=0]{fsharp}
let test31 = pchar('A',inputABC)
let test32 = pchar('A',inputZBC)
let test33 = pchar('Z',inputZBC)
\end{minted}

\begin{verbatim}
val test31: Result<(char * string),string> = Ok ('A', "BCD")
val test32: Result<(char * string),string> = Error "Expecting 'A'. Got 'Z'"
val test33: Result<(char * string),string> = Ok ('Z', "BCD")
\end{verbatim}
\end{frame}

\begin{frame}[label={sec:orgf25d48e},fragile]{Parser 4 (use currying)}
 \begin{minted}[bgcolor=mintedbg,frame=none,framesep=0pt,mathescape=true,fontsize=\scriptsize,breaklines=true,linenos=false,numbersep=5pt,gobble=0]{fsharp}
let pchar charToMatch str =
    if String.IsNullOrEmpty(str) then
        Error "No more input"
    else
        let first = str.[0]
        if first = charToMatch then
            let remaining = str.[1..]
            Ok (charToMatch,remaining)
        else
            let msg = sprintf "Expecting '%c'. Got '%c'" charToMatch first
            Error msg
\end{minted}

\begin{verbatim}
val pchar: charToMatch: char -> str: string -> Result<(char * string),string>
\end{verbatim}
\end{frame}

\begin{frame}[label={sec:org5144ddb},fragile]{Parser 4 (2)}
 \begin{minted}[bgcolor=mintedbg,frame=none,framesep=0pt,mathescape=true,fontsize=\scriptsize,breaklines=true,linenos=false,numbersep=5pt,gobble=0]{fsharp}
let parseA = pchar 'A'
let inputABC = "ABC"
let inputZBC = "ZBC"
let test41 = parseA inputABC
let test42 = parseA inputZBC
let parseZ = pchar 'Z'
let test43 = parseZ inputZBC
\end{minted}

\begin{verbatim}
val parseA: (string -> Result<(char * string),string>)
val inputABC: string = "ABC"
val inputZBC: string = "ZBC"
val test41: Result<(char * string),string> = Ok ('A', "BC")
val test42: Result<(char * string),string> = Error "Expecting 'A'. Got 'Z'"
val parseZ: (string -> Result<(char * string),string>)
val test43: Result<(char * string),string> = Ok ('Z', "BC")
\end{verbatim}
\end{frame}

\begin{frame}[label={sec:orga7e56d5},fragile]{Parser 5 (type to wrap the parser function)}
 \begin{minted}[bgcolor=mintedbg,frame=none,framesep=0pt,mathescape=true,fontsize=\scriptsize,breaklines=true,linenos=false,numbersep=5pt,gobble=0]{fsharp}
type Parser<'T> =
    | Parser of (string -> Result<'T , string>)
let pchar charToMatch =
    let innerFn str =
        if String.IsNullOrEmpty(str) then
            Error "No more input"
        else
            let first = str.[0]
            if first = charToMatch then
                let remaining = str.[1..]
                Ok (charToMatch, remaining)
            else
                let msg = sprintf "Expecting '%c'. Got '%c'" charToMatch first
                Error msg
    Parser innerFn
\end{minted}

\begin{verbatim}
type Parser<'T> = | Parser of (string -> Result<'T,string>)
val pchar: charToMatch: char -> Parser<char * string>
\end{verbatim}
\end{frame}

\begin{frame}[label={sec:orgb7ce182},fragile]{Parser 5 (2)}
 \begin{minted}[bgcolor=mintedbg,frame=none,framesep=0pt,mathescape=true,fontsize=\scriptsize,breaklines=true,linenos=false,numbersep=5pt,gobble=0]{fsharp}
let parseA = pchar 'A'
let inputABC = "ABC"
parseA inputABC
\end{minted}

\begin{verbatim}
parseA inputABC;;
  ^^^^^^

...: error FS0003: This value is not a function and cannot be applied.
\end{verbatim}
\end{frame}

\begin{frame}[label={sec:org732898b},fragile]{Parser 5 (3)}
 \begin{minted}[bgcolor=mintedbg,frame=none,framesep=0pt,mathescape=true,fontsize=\scriptsize,breaklines=true,linenos=false,numbersep=5pt,gobble=0]{fsharp}
let run parser input =
    let (Parser innerFn) = parser
    innerFn input
let parseA = pchar 'A'
let inputABC = "ABC"
let test1 = run parseA inputABC
let inputZBC = "ZBC"
let test2 = run parseA inputZBC
\end{minted}

\begin{verbatim}
val run: parser: Parser<'a> -> input: string -> Result<'a,string>
val parseA: Parser<char * string> = Parser <fun:pchar@74-6>
val inputABC: string = "ABC"
val test1: Result<(char * string),string> = Ok ('A', "BC")
val inputZBC: string = "ZBC"
val test2: Result<(char * string),string> = Error "Expecting 'A'. Got 'Z'"
\end{verbatim}
\end{frame}

\section{Parser Kombinatoren }
\label{sec:org3348783}
\begin{frame}[label={sec:org7b54569}]{Understanding Parser Combinators}
\(\leadsto\) \href{./5 Understanding parser combinators.pdf}{Understanding parser combinators} (Scott Wlashin)
\end{frame}

\begin{frame}[label={sec:orgb2e078e}]{FParsec Tutorial}
\begin{itemize}
\item \href{http://www.quanttec.com/fparsec/tutorial.html\#}{FParsec Tutorial}
\item \href{http://www.quanttec.com/fparsec/users-guide/}{User’s Guide}
\item \href{http://www.quanttec.com/fparsec/about/fparsec-vs-alternatives.html}{FParsec vs alternatives}
\end{itemize}
\end{frame}

\begin{frame}[label={sec:org84ac197},fragile]{Using FParsec (1)}
 \begin{minted}[bgcolor=mintedbg,frame=none,framesep=0pt,mathescape=true,fontsize=\scriptsize,breaklines=true,linenos=false,numbersep=5pt,gobble=0]{fsharp}
#r "../src/5/02-fparsec/lib/FParsecCS.dll";;
#r "../src/5/02-fparsec/lib/FParsec.dll";;
open FParsec
let test p str =
    match run p str with
    | Success(result, _, _)   -> printfn "Success: %A" result
    | Failure(errorMsg, _, _) -> printfn "Failure: %s" errorMsg;;
test pfloat "1.25"
test pfloat "1.25E 2"
\end{minted}

\begin{verbatim}
Success: 1.25
Failure: Error in Ln: 1 Col: 6
1.25E 2
     ^
Expecting: decimal digit

val it: unit = ()
\end{verbatim}
\end{frame}

\begin{frame}[label={sec:org1359eae},fragile]{Using FParsec (2)}
 \begin{minted}[bgcolor=mintedbg,frame=none,framesep=0pt,mathescape=true,fontsize=\scriptsize,breaklines=true,linenos=false,numbersep=5pt,gobble=0]{fsharp}
let str s = pstring s
let floatBetweenBrackets:Parser<float, unit>  = str "[" >>. pfloat .>> str "]";;

test floatBetweenBrackets "[1.0]"
test floatBetweenBrackets "[]"
test floatBetweenBrackets "[1.0]"
\end{minted}

\begin{verbatim}
Success: 1.0
Failure: Error in Ln: 1 Col: 2
[]
 ^
Expecting: floating-point number

Success: 1.0
val it: unit = ()
\end{verbatim}
\end{frame}

\begin{frame}[label={sec:org9ef280b},fragile]{Using FParsec (3)}
 \begin{minted}[bgcolor=mintedbg,frame=none,framesep=0pt,mathescape=true,fontsize=\scriptsize,breaklines=true,linenos=false,numbersep=5pt,gobble=0]{fsharp}
let betweenStrings s1 s2 p = str s1 >>. p .>> str s2;;
let floatBetweenBrackets_:Parser<float, unit> = pfloat |> betweenStrings "[" "]";;
let floatBetweenDoubleBrackets_:Parser<float, unit> = pfloat |> betweenStrings "[[" "]]";;
test floatBetweenBrackets_ "[1.0]"
test floatBetweenDoubleBrackets_ "[[1.0]]"
let between_ pBegin pEnd p  = pBegin >>. p .>> pEnd;;
let betweenStrings_ s1 s2 p = p |> between_ (str s1) (str s2);;
test (many floatBetweenBrackets) ""
test (many floatBetweenBrackets) "[1.0]"
test (many floatBetweenBrackets) "[2][3][4]"
test (many floatBetweenBrackets) "[1][2.0E]"
\end{minted}

\begin{verbatim}
Success: []
Success: [1.0]
Success: [2.0; 3.0; 4.0]
Failure: Error in Ln: 1 Col: 9
[1][2.0E]
        ^
Expecting: decimal digit

val it: unit = ()
\end{verbatim}
\end{frame}

\section{Ende }
\label{sec:orgd52857a}
\begin{frame}[label={sec:org00f27f4}]{Zusammenfassung (Kurs)}
\begin{itemize}
\item Wichtige Werkzeuge (git, dotnet, code)
\item Elementare Syntax
\item Funktionen, Pattern Matching, Discriminated Unions (DU)
\item Tuple, Record, List, Array, Seq
\item funktionale Operationen auf Listen (Tail-Rekursion)
\item funktionaler Umgang mit fehlenden Daten (Option)
\item funktionaler Umgang mit Fehlern (Result)
\item funktionales Design (statt Patterns: Funktionen \& Verkettung)
\item funktionales Refactoring
\item funktionales Domain Modeling (DDD)
\item eigenschaftsbasiertes Testen (Property Based Testing) (cool!!)
\item funktionale Parser (Kombinatoren) (noch cooler!!)
\end{itemize}
\(\leadsto\) \alert{\alert{Was ist Funktionale Programmierung?}}
\end{frame}

\begin{frame}[label={sec:org36aba8b}]{Links}
\begin{itemize}
\item \href{https://fsharp.org/}{fsharp.org}
\item \href{https://docs.microsoft.com/de-de/dotnet/fsharp/}{docs.microsoft.com/../dotnet/fsharp}
\item \href{https://sergeytihon.com/}{F\# weekly}
\item \href{https://fsharpforfunandprofit.com/}{fsharpforfunandprofit.com}
\item \href{https://github.com/fsprojects/awesome-fsharp}{github.com/../awesome-fsharp}
\end{itemize}
\end{frame}

\begin{frame}[label={sec:org5528b6b}]{Ende}
\begin{itemize}
\item Wie geht es weiter?
\item \href{https://exercism.io}{Exercism!}
\item Buchtipps
\begin{itemize}
\item \href{https://pragprog.com/book/swdddf/domain-modeling-made-functional}{Domain Modeling Made Functional} (F\#)
\item \href{https://www.apress.com/gp/book/9781484239995}{Stylish F\#} (F\#)
\item \href{https://www.cambridge.org/core/books/pearls-of-functional-algorithm-design/B0CF0AC5A205AF9491298684113B088F\#}{Perls of Functional Algorithm Design} (Haskell)
\item \href{https://www.cs.ox.ac.uk/publications/books/functional/}{Thinking Functional with Haskell} (Haskell)
\item \href{http://www.paulgraham.com/onlisp.html}{On Lisp} (LISP)
\item \href{http://www.iqool.de/FPMP.html}{Funktionale Programmierung und Metaprogrammierung} (LISP)
\item \href{https://github.com/norvig/paip-lisp}{Paradigms of Artificial Intelligence Programming} (LISP)
\item \href{https://adv-r.hadley.nz/}{Advanced R} (R)
\end{itemize}
\item Sprachen: \href{https://www.r-project.org/}{R}, \href{https://www.haskell.org/}{Haskell}, \href{https://clojure.org/}{Clojure}, \href{https://lisp-lang.org/}{Common Lisp}, \href{https://elixir-lang.org/}{Elixir}, \href{https://code.kx.com/q/}{q}

\item \alert{\alert{Have FUN!}}
\end{itemize}
\end{frame}
\end{document}