\section{Other concepts} \label{sec:other_NN_concepts}
This section will briefly describe some other concepts that are relevant to this document but not warrant a thorough explanation.

\subsection{Overfitting} \label{sub:overfitting}
One fundamental principle, not only for neural networks but many other machine learning techniques, is the idea of generalization. An algorithm that learned by training on some given data should be able to produce good results not only on this seen data, but also on similar and never before seen data. In the case of supervising learning problems for example, all the data is already labeled, there is no need to create a complex algorithm that can find these labels since the answer is already know. When teaching machines to think, the goal is not to make them memorize all the inputs and outputs, it is instead to make them learn how to abstract the data, to capture the fundamental patterns, and to interpret the input in a high level (e.g. seeing shapes, and objects instead of just pixels).

Since in the majority of cases the data available for training is only a fraction of the possible values expected for the problem, a machine learning algorithm must be able to use this relatively small amount of data to project how all the input space is distributed. This is as much a task for the algorithm as is for the dataset itself, the training data should contain enough samples to offer an accurate representation of how the input space is truly distributed. Ideally this data should be diverse enough to represent well the possible inputs (e.g. when training to differentiate between cats and dogs, the data should contain photos of both cats and dogs of multiple colors, in various angles, and with different poses) and it also should contain as much samples as possible\footnote{
    Although having too much representation for one region of the input space in relation with the others can introduce bias in the model. For example, \cite{networkBias2018} found that some gender classification systems had a significant higher error rates for dark skinned women (as high as 30\% higher than for white males) and that some facial analysis datasets underrepresented minorities in the samples. A similar problem happened to amazon, where an AI recruiting system was discriminating women for jobs since most of the resumes that it was trained on came from men \cite{amazonBias2018}. One proposed approach to combat this problem is to train another network to learn how the data is distributed and sample more from underrepresented data \cite{debiasingVAE2019}.
}.

\begin{figure} [hbt]
    \centering
    \caption{Visual representation of a learning algorithm fit to noisy data}
    \includegraphics[width=0.8\textwidth]{chapters/NeuralNets/figures/fitting.pdf}
    \fonte{From the author (2021)}
    \label{fig:fitting}
\end{figure}
When the dataset is insufficient or the learning algorithm is not capable enough, the final model will only partially represent the data, but will not be able to capture all the details of the representation; this is called \textit{Underfitting}, the model was not able to fully represent the structure of the problem. In the other hand, the algorithm can also go to the opposite extreme and learn the dataset too well, this means that it will pick up particularities of the training data that don't generalize to the whole input distribution; this is called \textit{Overfitting} and it is facilitated by small datasets that don't represent much variety (noise in a sample has more influence in the signal to noise ratio of the whole dataset), or by networks that have too many degrees of freedom. \autoref{fig:fitting} visually represents the different fits for a 1-dimensional case.

Overfitting is however so common that it is almost natural to expect its appearance, it is one of the most prevalent problems in machine learning and there are many existing techniques to avoid it. The first step is to have a good quality dataset, data augmentation methods can be used to easily multiply the number of samples without much downsides. In the algorithmic side, next subsection will discuss regularization and how it can combat overfitting among other things.

\subsection{Regularization} \label{sub:regularization}
According to \cite[p. 117]{deepLearningBook2016}: ``\textit{Regularization is any modiﬁcation we make to a learning algorithm that is intended to reduce its generalization error but not its training error}''. This is a general definition that encompasses many different techniques for regularization, but for this document it will suffice to describe only dropout and the $\ell_1$ and $\ell_2$ norms.

\subsubsection{L1 and L2 norms}
Since one of the reasons overfitting happens is because the models are given a large number of free parameters to adjust to the dataset (worth mentioning again the 175 billion parameters present in GPT-3 \cite{gpt3_2020}), then a common way of reducing overfitting is to limit the choice of parameters by adding a penalization term $\Omega(\bm{\theta})$ to the loss function as shown in \autoref{eq:regularization_penalty} \cite[p. 226]{deepLearningBook2016}.
\begin{equation} \label{eq:regularization_penalty}
    \tilde{J}(\bm{X}, \bm{\theta}) = J(\bm{X}, \bm{\theta}) + \lambda\Omega(\bm{\theta})
\end{equation}

The \gls{lambda} term is a positive hyperparameter that determines how much the penalty is relevant to the overall loss. For $\ell_1$ and $\ell_2$ norms, $\Omega(\bm{\theta})$ will be the respective norm of the $\bm{\theta}$ parameters. The $\ell_2$ norm is the standard distance in euclidean space, so for this norm the penalty term is given by \autoref{eq:l2_norm}.
\begin{equation} \label{eq:l2_norm}
    \ell_2(\bm{\theta}) = \|\bm{\theta}\|_2 = \sqrt{\theta_0^2 + \theta_1^2 + \dots + \theta_n^2}
\end{equation}

For the $\ell_1$ norm, the value calculated is what is called the Manhattan distance or taxicab distance \cite[p. 102]{dataDrivenScience2019} and is calculated as shown is \autoref{eq:l1_norm}
\begin{equation} \label{eq:l1_norm}
    \ell_1(\bm{\theta}) = \|\bm{\theta}\|_1 =  |\theta_0| + |\theta_1| + \dots + |\theta_n|
\end{equation}

Both norms will penalize large parameter values and the modified loss function in \autoref{eq:regularization_penalty} will make gradient descent favor solutions that use smaller parameters. The \gls{lambda} hyperparameter then determines how much to favor these simpler solutions.

A simple explanation on why smaller values would reduce overfitting can be given by considering the case of fitting a polynomial to some data. Consider the overfitting case seen before in \autoref{fig:fitting}, fitting a polynomial to reduce the square error on that noisy data results in a reasonable good approximation around the data points, but the curve suddenly changes directions in the extremes because of the terms with the highest powers in the polynomial. To make the curve fit the noisy data, a least squares approach will usually produce very high coefficients to balance the higher powers, this reduces the error but will almost never produce the true behaviour behind the data. Penalizing high coefficients with regularization is a way to force the solution to assume simpler values.

This is in no way a rigorous argument for regularization, but it can be used to have an intuition. The goal of this section is just to introduce the concept, regularization is an active area of research and a lot of support for it is based on empirical evidence, there is no complete theory to explain how and why it works so well \cite[chap. 3]{NN&DL2015}.

One last thing to mention is the fact that $\ell_1$ norm promotes sparsity in the parameters, this means that this approach will usually find solutions where multiple parameters will equal zero, larger values of \gls{lambda} will promote more sparse models.

\subsubsection{Dropout}
Since the local minimum reached by a training process will depend on the starting values for the parameters, one common way to increase the accuracy of a model is create an ensemble of different networks trained in the same dataset but with different starting points and/or architectures \cite[Chapter 6]{NN&DL2015}. The model output can then be a combination of the outputs of the ensemble, the combination could be majority voting, an average between outputs, or any other sensible function. This gives better results because it is not expected that randomly initialized networks would make the same mistakes in all situations \cite[p. 253]{deepLearningBook2016}.

Dropout is a technique that can be interpreted as a way to approximate an exponentially large ensemble of networks while using a single network and without needing to train more models \cite{dropout2012}. The idea behind it is that during training, when feedforwarding the input through the network, some units should have a probability $p$ of being dropped from the calculation, that is, their activations are set to zero.

The idea of removing units during calculation may sound counter-intuitive, but like the $\ell$ norms it is a way of restricting the learning process so that the network can't build very complex connections. By removing random units in each iteration, the network must be able to learn to represent the data even if a great number of units are removed, it no longer can expect that a combination of units will be present when evaluating the result and so it cannot depend on complex inter-correlations between units \cite{dropout2012}.

By dropping different units, dropout effectively trains the ensemble of all sub-networks of the base network. Typical dropout rates are 20\% for the input layer and 50\% for hidden layers \cite[p. 255, 257]{deepLearningBook2016}.

\subsection{Optimizers} \label{sub:optimizers}
Gradient descent has the theoretical basis for convergence, but it has big problems for use in practice. The first problem is that the whole training set must be averaged when calculating the loss $J(\bm{\theta})$, even for small datasets this is quite restrictive, for larger datasets containing hundreds of thousands or millions of samples this can make each iteration take an immense amount of time. This type of approach is commonly called Batch Gradient Descent.

Another problem is that, even if calculating the gradient was very fast, the simple parameter update (recall \autoref{eq:gradient_descent}) can become very slow, specially later in training. The point in parameter space can reach a region in the loss surface where the curvature is minimal, making the gradients very small; the updates could also jump around a valley in the loss surface, never reaching the local minimum \cite{momentumWorks2017}.

For calculating the gradients faster there is the approach called \glsreset{SGD}\gls{SGD}. There are several different methods for better traversing the loss surface, these are called optimizers. For the purposes of this document only the Momentum, RMSProp, and \gls{Adam} optimizers will be briefly explained.

\subsubsection{Stochastic Gradient Descent}
Instead of calculating the gradient $\nabla_{\bm{\theta}} J$ for all inputs in the dataset, \gls{SGD} instead approximates this value by calculating it for a single sample on the input. This makes the updates much faster, but will cause the updates to fluctuate heavily because of the gross approximation; this however can be beneficial since it can make the updates jump to potentially better local minima and it has been shown that, for small learning rates, \gls{SGD} will also converge \cite{optimizers2016}.

Another way to approximate the gradient is by calculating it only for a mini-batch of the dataset, that is, a subset of $m$ samples of the data. This is called Mini-batch Gradient Descent but is also more commonly referred to as \gls{SGD} as well. Using mini-batches instead of a single sample gives more stable updates while still being very fast, the values for $m$ usually fall between $50$ and $256$ \cite{optimizers2016}, but lower values are also common.

Another advantage of the stochastic approach is that the noise in the gradient approximation has an added regularization effect \cite[p. 5]{practical_gradient_recomendations2012}.

\subsubsection{Momentum, RMSProp, and Adam}
\glsreset{Adam}
The simple parameter update (\autoref{eq:gradient_descent}) for Gradient Descent has some problems that make it difficult to converge in practice. Some of the problems mentioned by \textcite{optimizers2016} are: that it is hard to choose a good value for the learning rate; only using a single learning rate can be insufficient; the parameters can get stuck in difficult regions of the loss surface, especially saddle points.

Between the many different approaches to combat this, one of the most popular that is also recommended when training \acp{GAN} \cite[p. 20, 27]{nipsGAN2017} is the \gls{Adam} optimizer. This optimizer is essentially a combination of Momentum and RMSProp, two other very popular optimizers.

Momentum, as the name suggests, is a way to give some inertia to the gradient updates, accelerating in the consistent directions while dampening oscillations. This technique has a strong theoretical basis and can give a quadratic speedup on many functions \cite{momentumWorks2017}. Momentum changes the parameter updates by adding a velocity term controlled by a new hyperparameter $\beta$ as shown in the following equations.
\begin{gather} \label{eq:momentum}
    \bm{m}_{i} = \beta \bm{m}_{i-1} + \nabla_{\bm{\theta}} J(\bm{\theta}_{i-1}) \\
    %
    \bm{\theta}_{i} = \bm{\theta}_{i-1} - \eta \bm{m}_{i}
\end{gather}

The RMSProp algorithm also tries to reinforce movement to the most relevant directions, it does this by keeping a exponentially moving average $v$ of the gradient accumulation \cite[303-304]{deepLearningBook2016}. It also introduces another $\beta$ hyperparameter and changes the updates as follows.
\begin{gather}
    \bm{v}_{i} = \beta \bm{v}_{i-1} + (1 - \beta)\nabla_{\bm{\theta}} J(\bm{\theta}_{i-1})^2 \\
    %
    \bm{\theta}_{i} = \bm{\theta}_{i-1} - \frac{\eta}{\sqrt{\bm{v}_{i} + \epsilon}} \nabla_{\bm{\theta}} J(\bm{\theta}_{i-1})
\end{gather}

Recall that although these equations are represented in vector form, all operation are performed element-wise. Finally for the \gls{Adam} algorithm, it can be seen as a combination of the two previous algorithms. It introduces two new hyperparameters $\beta_1$ and $\beta_2$ and the parameter updates work as follows \cite{adam2017}.
\begin{gather}
    \bm{m}_i = \beta_1 \bm{m}_{i-1} + (1 - \beta_1) \nabla_{\bm{\theta}} J(\bm{\theta}_{i-1}) \\[5pt]
    \bm{v}_i = \beta_2 \bm{v}_{i-1} + (1 - \beta_2) \nabla_{\bm{\theta}} J(\bm{\theta}_{i-1})^2 \\[5pt]
    \hat{\bm{m}_i} = \frac{\bm{m}_i}{1 - \beta_1^i} \\[5pt]
    \hat{\bm{v}_i} = \frac{\bm{v}_i}{1 - \beta_2^i} \\[5pt]
    \bm{\theta}_i = \bm{\theta}_{i-1} - \eta \frac{\hat{\bm{m}_i}}{\sqrt{\hat{\bm{v}_i} + \epsilon}}
\end{gather}

\textcite{adam2017} mentions that good default values for the hyperparameters are $\alpha = 0.001$, $\beta_1 = 0.9$, $\beta_2 = 0.999$ and $\epsilon = 10^{-8}$. The Tensorflow library \cite{tensorflow2015} follows this default, only changing $\epsilon$ to a order of magnitude higher, that is, $\epsilon = 10^{-7}$.

\subsection{Batch Normalization}
One important aspect in training neural networks is the distribution of the inputs for a layer. As mentioned in \autoref{subsub:sigmoid} about sigmoid, \textcite{efficientBackprop2012} showed that inputs that are not zero centered will have some bias effect on parameter updates, to avoid this the layer inputs should all be shifted to have a mean of zero; the authors also suggest to normalize all inputs in order to have the same covariance and, if possible, to have their values be uncorrelated. This should significantly speed up the learning process since the network will not have to adapt to a different distribution for every input.

Normalizing inputs is then an easy and effective way for better learning, therefore being a very commonly used technique that will also be used for the experiments proposed in this document.

However \textcite{batchnorm2015} note that for any layer in the network, the normalization idea for its inputs also applies; that is, any hidden layer feeding its output to the next layer can be considered as an input layer for a sub-network consisting of all the subsequent layers. Therefore, the same advantages for normalized inputs would be beneficial by normalizing hidden layer outputs.

Looking from the training perspective, each training iteration updates all parameters at the same time, but the gradient change in a layer gives the best change considering that all other layers remain constant \cite[p.313-314]{deepLearningBook2016}. In reality, changing the parameters of the previous layers will affect the distribution of inputs for the current layer, this constant change to the inputs of a layer resulting from changes to previous layers is what \textcite{batchnorm2015} call \textit{internal covariate shift}.

To address these issues \textcite{batchnorm2015} proposed a technique known as \gls{BN}, this method can be applied to any hidden layer and it consists of normalizing the layer inputs by the mean and variance of all the inputs in the current mini-batch. So, for a mini-batch $\mathcal{B}$ of size $m$ and inputs $(\bm{x}_1, \dots, \bm{x}_m)$, the normalized inputs $\hat{\bm{x}}_i$ are calculated using the mean $\bm{\mu}_{\mathcal{B}}$ and variance $\bm{\sigma}_{\mathcal{B}}$ of the batch. Equations \ref{eq:batch_mean}, \ref{eq:batch_variance}, and \ref{eq:batch_normalized_input} show how the mean, variance, and normalized inputs are calculated.
\begin{gather}
    \bm{\mu}_{\mathcal{B}} = \frac{1}{m}
    \sum_{i=1}^{m}{\bm{x}_i} \label{eq:batch_mean} \\[5pt]
    %
    \bm{\sigma_}{\mathcal{B}}^2 = \frac{1}{m}
    \sum_{i=1}^{m}{ \left( \bm{x}_i - \bm{\mu}_{\mathcal{B}} \right)^2 } \label{eq:batch_variance} \\[5pt]
    %
    \hat{\bm{x}}_i = \frac{ \bm{x}_i - \bm{\mu}_{\mathcal{B}} }
    { \sqrt{\bm{\sigma}_{\mathcal{B}}^2 + \epsilon} } \label{eq:batch_normalized_input}
\end{gather}

Note that these operations are applied element-wise for each component of the vectors, and again $\epsilon$ is used for numerical stability.

These operations result in the vectors $\hat{\bm{x}}_i$ having a mean of $0$ and variance of $1$ for each of their dimensions. However, by limiting the distribution to only these values also limits the representation power of the network \cite{batchnorm2015}, so the last step is to scale each one of the normalized activations by two new learnable parameters, $\gamma$ and $\beta$, unique for each activation, to obtain the transformed input $\bm{y}_i$ that can have any mean and variance. \autoref{eq:batch_transformed_input} shows how to scale the dimension $k$ of the normalized input $i$.
\begin{equation} \label{eq:batch_transformed_input}
    y_i^{(k)} = \gamma^{(k)} x_i^{(k)} + \beta^{(k)}
\end{equation}

This method has proven to be very powerful, with the original paper being able to reach the same accuracy as the, at the time state of the art image classification model, in 14 times less training iterations \cite{batchnorm2015}. The authors also cite a regularization effect of \gls{BN} that can reduce the need for other techniques like dropout.

When the authors originally proposed this method, they recommended applying the batch normalization operation directly to the weighted inputs of the network and only after apply the activation to the normalized values. However, since then \textcite{CaffeNetBench2017} empirically showed that applying \gls{BN} after the activation produces better results.

\subsection{Vanishing Gradients} \label{sub:vanishing_gradients}
Recall that for Equations \ref{eq:delta_last_layer} and \ref{eq:delta_hidden_layer} the $\delta$ term used to calculate the gradients, was directly dependent on the derivative of the activation function with relation to the weighted inputs $f'(\bm{z}^{(l)})$. Also note that the gradients are backpropagated through the network, which means that the $\delta$ values in a hidden layer are calculated from the $\delta$ in the next layer.

Consider now the case for a sigmoid or \gls{tanh} activation function which is applied to an weighted input far from zero, the activation in this case will be very close to the maximum or minimum value that the function can produce. When a unit outputs this kind of value it is said that it saturated, any slight deviation in the weighted input will barely have any difference in the activation value, in other words the derivative of the activation is close to zero.

Combining the points made in the last paragraphs it is possible to recognize a problem, when a unit saturates its gradient will be very small since the derivative of the activation is close to zero. Even worse than that is the fact that this small gradient will be backpropagated through the network, reducing the gradients of all previous layers. When many units saturate, this effect can compound, making the gradients for deeper layers very small and greatly reducing the training speed; this is the problem known as the \textit{Vanishing Gradients} problem.

One alternate case is when the gradients are high and compound to make the gradients in deeper layers become very high, making the steps too large and not converging to any local minima. This is called the \textit{Exploding Gradients} problem.

Both situations are undesirable and are specially worrying when trying to train very deep neural networks. Ideally the gradients should all be close to $1$ to make training consistent, but for saturating activation functions like sigmoid, there will usually be vanishing gradients \cite{NN&DL2015}. That is one of the main reasons to use non-saturating activation functions like \gls{ReLU} in the hidden layers of the network. Other common ways to combat this problem is to carefully initialize the network parameters and use small learning rates \cite{batchnorm2015}, these authors also suggest that \gls{BN} can be used on saturating units since it can reduce the chance that the input of the units will fall to the saturating regions.


% !TEX root = atlas_iros_16.tex
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{\large Modeling and Identification}
\label{sec:advance}

In this section we shortly review the system modeling and enhanced impedance controller approach (\ref{sec:advance_controller}), modifications to the momentum-based disturbance observer to include wrist wrench measurements (\ref{sec:advance_observer}), compensation of modeling errors and collision handling (\ref{sec:friction_comp_and_collDet}), and an identification scheme that iteratively estimates friction characteristics and rigid body parameters (\ref{sec:advance_ident}).

\subsection{System Model and Joint Impedance Controller}
\label{sec:advance_controller}
This work focuses on the 7R type serial chain arms of the Boston Dynamics humanoid robot Atlas, see Fig.~\ref{fig:StickCollisionPhoto}.
The arms employ a hybrid actuation concept where the first four joints (shz, shx, ely, elx) are hydraulic, and the wrist joints (wry, wrx, wry2) are driven by electromechanical gear drives.
We assume the standard fixed-base rigid joint arm model
\begin{equation}
\bm{M(\bm{q})}\ddot{\bm{q}}+\bm{C}(\bm{q},\dot{\bm{q}})\dot{\bm{q}}+\bm{g}(\bm{q})=\bm{\tau}_\mathrm{m}-\bm{\tau}_\mathrm{f}+\bm{\tau}_\mathrm{ext}
\label{eqn:invdyn}
\end{equation}
with generalized joint position $\bm{q} \in \mathbb{R}^{n_\mathrm{j}}$ ($n_\mathrm{j}$ being the number of joints), positive definite and symmetric inertia matrix $\bm{M}(\bm{q})$, centrifugal and Coriolis matrix $\bm{C}(\bm{q}, \dot{\bm{q}})$, gravity torque vector $\bm{g}(\bm{q})$, actuator torques $\bm{\tau}_\mathrm{m}$, friction torques $\bm{\tau}_\mathrm{f}$ and external torques $\bm{\tau}_\mathrm{ext}$.
The friction torque $\bm{\tau}_\mathrm{f}$ is composed of the motor side friction $\bm{\tau}_{\mathrm{f},\theta}$ and the link side friction $\bm{\tau}_{\mathrm{f},q}$.
Both hydraulic and electromechanic actuators are considered as ideal torque sources, generating a motor torque $\bm{\tau}_\mathrm{m}$, allowing a separation of the actuator dynamics from the rigid body model.

For soft-robotics control of the system we chose the joint impedance control torque $\bm{\tau}_\mathrm{d}$ to be
\begin{equation}
\begin{gathered}
\bm{\tau}_\mathrm{d}=
\bm{K}(\bm{q}_\mathrm{d}-\bm{q})+\bm{D}(\dot{\bm{q}}_\mathrm{d}-\dot{\bm{q}})+\hat{\bm{g}}(\bm{q}) % tau_I
\\
+\hat{\bm{C}}(\bm{q},\dot{\bm{q}})\dot{\bm{q}} % tau_c
+\hat{\bm{M}}(\bm{q}_\mathrm{d})\ddot{\bm{q}}_\mathrm{d} % tau_ff
+\kappa_{\mathrm{f}} \hat{\bm{\tau}}_\mathrm{f}(\dot{\bm{q}})
-\kappa_{\varepsilon} \hat{\bm{\tau}}_\mathrm{\varepsilon}(\bm{q}, \dot{\bm{q}},\bm{\tau}_\mathrm{m})\;,
\label{eqn:controller}
\end{gathered}
\end{equation}
where $\bm{q}_\mathrm{d}, \dot{\bm{q}}_\mathrm{d}, \ddot{\bm{q}}_\mathrm{d}$ are the desired position, velocity, and acceleration, respectively. The matrices $\bm{K}=\mathrm{diag}\{k_i\}$ and $\bm{D}_\xi=\mathrm{diag}\{d_{\xi,i}\}$ denote diagonal positive definite stiffness and modal damping and $\bm{D}$ the resulting positive definite damping matrix.
$\hat{\bm{g}}$ and $\hat{\bm{C}}$ are the gravity and centrifugal/Coriolis estimates.
The inertial feedforward term makes use of the estimated inertia matrix $\hat{\bm{M}}$ as a function of the desired position $\bm{q}_\mathrm{d}$.
The compensation terms $\hat{\bm{\tau}}_\mathrm{f}(\dot{\bm{q}})$ (viscous and Coulomb friction) and $\hat{\bm{\tau}}_\mathrm{\varepsilon}(\bm{q}, \dot{\bm{q}},\bm{\tau}_\mathrm{m})$ (estimated disturbance from Sec.~\ref{sec:advance_observer}) are activated via the scalars $\kappa_{\mathrm{f}}\in \{0,1\}$ and $\kappa_{\varepsilon}\in \{0,1\}$, respectively.

Combining (\ref{eqn:invdyn}) and (\ref{eqn:controller}) leads to the closed-loop dynamics
\begin{equation}
\begin{gathered}
\bm{M(\bm{q})}\ddot{\bm{q}}-\hat{\bm{M}}(\bm{q}_\mathrm{d})\ddot{\bm{q}}_\mathrm{d}
+\bm{D}(\dot{\bm{q}}-\dot{\bm{q}}_\mathrm{d})
+\bm{K}(\bm{q}-\bm{q}_\mathrm{d})
 \\
=\bm{\tau}_\mathrm{ext}
+ \bm{\delta}
-\kappa_{\varepsilon} \hat{\bm{\tau}}_\mathrm{\varepsilon},
\label{eqn:closedloop_general}
\end{gathered}
\end{equation}
where $\bm{\delta}$ denotes lumped dynamics and friction modeling errors and errors caused by sensor drift, offsets and time delays.
We assume these effects to be additive.

\subsection{Disturbance Observer}
\label{sec:advance_observer}
%
\begin{figure}
\centering
\input{./figures/MechErsatzBSB/rigid_flexible_joint_struktur.pdf_tex}
\caption{Relevant torques acting along the mechanical structure for the rigid joint model (a) with link inertia $\bm{M}$ and the flexible joint model (b) with motor inertia $\bm{B}$ and motor position $\bm{\theta}$.}
\label{fig:rigid_flexible_joint_structure}
\SkipBeforeText
\end{figure}
%
Before introducing our observer design, let us shortly summarize the underlying problem of collision detection with typical hydraulic robots.
Figure~\ref{fig:rigid_flexible_joint_structure} emphasizes the friction torques that are relevant for rigid joint models in comparison to the flexible joint case.
The latter represents e.g. electromechanically actuated robots with elastic joints and torque sensing \cite{Ott2008}.
In the former case, the total friction $\bm{\tau}_\mathrm{f}$ and external torques $\bm{\tau}_\mathrm{ext}$ act on a single body that represents both motor and link inertia and sum up to a total disturbance torque.
Thus, except under certain modeling assumptions they cannot be separated with standard proprioceptive sensing and according observer techniques.
In the latter case, motor and link-side dynamics are coupled via the joint stiffness $\bm{K}_{\mathrm{J}}$.
Typically, this originates either from rather elastic gears such as Harmonic Drives in combination with joint torque sensors (rather high inherent stiffness $\bm{K}_{\mathrm{J}}$), or from deliberately placed spring elements as e.g. in the Series Elastic Actuation (SEA \cite{PrattWil1995}) case (rather low inherent stiffness $\bm{K}_{\mathrm{J}}$).
Since the link side friction $\bm{\tau}_{\mathrm{f},q}$ is usually low (it is mainly caused by low friction link-side bearings), it may be neglected and the link-side observer essentially estimates the true external joint torques \cite{Haddadin2014}.
Therefore, it is possible to set up two observer schemes, one for the actuator side estimating $\bm{\tau}_{\mathrm{f},\theta}$ \cite{LeTienAlbDeHir2008}, and one for the link side estimating $\bm{\tau}_\mathrm{ext}$.
Note that one can obviously set up an elastic joint model for the hydraulic case as well.
However, this would only have a similar implication if an additional joint torque sensor for decoupling would be inserted after link friction.

To be able to distinguish between internal and external effects, we extend the momentum-based disturbance observer from \cite{DeLucaMat2003,DeLucaMat2004,Haddadin2014} to include measurements of external wrenches $\medmuskip=1.3mu
\thinmuskip=1.3mu
\thickmuskip=1.3mu
\bm{\mathcal{F}}_\mathrm{ext,EE}=\begin{pmatrix}\bm{f}_{\mathrm{ext,EE}} & \bm{m}_{\mathrm{ext,EE}}\end{pmatrix}^\mathrm{T}$ that act on the end-effector, i.e. ``after'' the sensor \cite{OttHenLee2013}.
The extended residual for end-effector contacts is then defined as
%
\begin{equation}
\medmuskip=0.5mu
\thinmuskip=0.5mu
\thickmuskip=0.5mu
\hat{\bm{\tau}}_\mathrm{\varepsilon}=\bm{K}_\mathrm{o}
\left(
\hat{\bm{M}}(\bm{q})\dot{\bm{q}}
-\int\limits_0^t
 [\bm{\tau}_\mathrm{m}
 -\bm{\gamma}(\bm{q},\dot{\bm{q}})
 +\hat{\bm{\tau}}_\mathrm{\varepsilon}
 -\bm{\alpha}(\bm{q},\dot{\bm{q}})
 ]\mathrm{d}\tilde{t}
 \right),
\label{eqn:observer}
\end{equation}
%
where $\bm{K}_\mathrm{o}=\mathrm{diag}\{k_{\mathrm{o},i}\}>\bm{0}$ is the observer gain matrix and
\begin{equation}
\medmuskip=0.05mu
\thinmuskip=0.05mu
\thickmuskip=0.05mu
\bm{\gamma}(\bm{q},\dot{\bm{q}}):=
\hat{\bm{g}}(\bm{q})
+\hat{\bm{C}}(\bm{q},\dot{\bm{q}})\dot{\bm{q}}
-\dot{\hat{\bm{M}}}(\bm{q})\dot{\bm{q}}
=
\hat{\bm{g}}(\bm{q})-\hat{\bm{C}}^\mathrm{T}(\bm{q},\dot{\bm{q}})\dot{\bm{q}}\;.
\label{eqn:gamma}
\end{equation}
Equality (\ref{eqn:gamma}) follows directly from the skew-symmetry of $\dot{\hat{\bm{M}}}(\bm{q})-2\hat{\bm{C}}(\bm{q},\dot{\bm{q}})$ \cite{DeLucaAlbHadHir2006}.
The vector $\bm{\alpha}(\bm{q},\dot{\bm{q}})$ is defined as
\begin{equation}
\begin{split}
\bm{\alpha}(\bm{q},\dot{\bm{q}}) :=&
\kappa_{\mathrm{f}}\hat{\bm{\tau}}_\mathrm{f}(\dot{\bm{q}}) -\kappa_{\mathrm{ext}}\bm{\tau}_{\mathrm{ext,EE}} \\
=&
\kappa_{\mathrm{f}}\hat{\bm{\tau}}_\mathrm{f}(\dot{\bm{q}})
-\kappa_{\mathrm{ext}}\bm{J}^{\mathrm{T}}(\bm{q}) \bm{\mathcal{F}}_\mathrm{ext,EE} \; .
\end{split}
\label{eqn:observer_settings}
\end{equation}
The contact wrench $\bm{\mathcal{F}}_\mathrm{ext,EE}$ is typically measured with a load-compensated force/torque sensor in the robot wrist.
The resulting external joint torques $\bm{\tau}_{\mathrm{ext,EE}}$ are obtained by the well known mapping via the end-effector Jacobian $\bm{J}(\bm{q})$.
Its feedback is activated via $\kappa_{\mathrm{ext}} \in \{0,1\}$.
Note that the components of $\bm{\mathcal{F}}_\mathrm{ext,EE}$ that are in the kernel of $\bm{J}^{\mathrm{T}}(\bm{q})$ are absorbed by the structure of the robot and are not reflected in $\bm{\tau}_{\mathrm{ext,EE}}$.

To distinguish directly measurable joint torques originating from external wrenches at the end-effector $\bm{\tau}_{\mathrm{ext,EE}}$ from joint torques caused by external wrenches at the structure $\bm{\tau}_{\mathrm{ext,links}}$ that cannot be measured by the wrist sensor, we define the total external torque vector $\bm{\tau}_{\mathrm{ext}}$ to be
\begin{equation}
\bm{\tau}_{\mathrm{ext}} := \bm{\tau}_{\mathrm{ext,EE}} + \bm{\tau}_{\mathrm{ext,links}}.
\label{eqn:extjointtorqueseparata}
\end{equation}
The true disturbance joint torque ${\bm{\tau}}_\mathrm{\varepsilon}$ for rigid joint models consists of the joint torques from (\ref{eqn:extjointtorqueseparata}) plus the error term $\bm{\delta}$ in (\ref{eqn:closedloop_general}).
%
As derived in \cite{Haddadin2014} the observed disturbance torque, also for this extended form, converges element-wise with first order dynamics (presented in frequency domain)
\begin{equation}
\hat{\tau}_{\mathrm{\varepsilon},i}=
\frac{k_{\mathrm{o},i}}{s+k_{\mathrm{o},i}}
\left(\tau_{\mathrm{ext,EE},i}(1-\kappa_{\mathrm{ext}})
+ \tau_{\mathrm{ext,link},i}
+ \delta_{i}
\right)
\label{eqn:disttorqueconvergence}
\end{equation}
where $1/k_{\mathrm{o},i}$ is the time constant.
%
\begin{figure}
\centering
\includegraphics{./figures/ObserverFlowChart/observer_scheme}
\caption{Flowchart of the proposed observer structure.}
\label{fig:observer_flowchart}
\SkipBeforeText
\end{figure}
%
Figure~\ref{fig:observer_flowchart} depicts an overview of the overall observer structure.

In the next section, we outline how this extended disturbance observer is used to compensate for model errors and detect collisions simultaneously.

\subsection{Compensation of Model Errors and Collision Detection}
\label{sec:friction_comp_and_collDet}

Assuming $\bm{\tau}_\mathrm{ext}$ to have slower dynamics than the observer (sufficiently large observer gain $\bm{K}_{\mathrm{o}}$).
One can approximate the disturbance torque as
\mbox{$\hat{\bm{\tau}}_\mathrm{\varepsilon}\approx
\left(\bm{\tau}_\mathrm{ext,EE}(1-\kappa_{\mathrm{ext}})
+ \bm{\tau}_\mathrm{ext,link}
+ \bm{\delta}
\right)$}.
Therefore, one obtains from equation (\ref{eqn:closedloop_general})
\begin{equation}
\begin{split}
\bm{M(\bm{q})}\ddot{\bm{q}}-\hat{\bm{M}}(\bm{q}_\mathrm{d})\ddot{\bm{q}}_\mathrm{d}
+\bm{D}(\dot{\bm{q}}-\dot{\bm{q}}_\mathrm{d})
+\bm{K}(\bm{q}-\bm{q}_\mathrm{d})\\
= \bm{\tau}_\mathrm{ext}+\bm{\delta}
-\kappa_{\varepsilon}\left(
  \bm{\tau}_\mathrm{ext,EE}(1-\kappa_{\mathrm{ext}})
+ \bm{\tau}_\mathrm{ext,link}
+ \bm{\delta}
\right)\;.
\end{split}
\label{eqn:closed_loop_converged}
\end{equation}
The trajectory tracking performance can thus be improved significantly by using the disturbance compensation with $\kappa_{\varepsilon}=1$ in (\ref{eqn:controller}) and $\kappa_\mathrm{ext}=0$ in (\ref{eqn:observer_settings}), as this would eliminate model inaccuracies $\bm{\delta}$.
The obvious drawback would be the loss of compliance w.r.t. external torques, since equation (\ref{eqn:closed_loop_converged}) becomes
\begin{equation}
\bm{M(\bm{q})}\ddot{\bm{q}}-\hat{\bm{M}}(\bm{q}_\mathrm{d})\ddot{\bm{q}}_\mathrm{d}
+\bm{D}(\dot{\bm{q}}-\dot{\bm{q}}_\mathrm{d})
+\bm{K}(\bm{q}-\bm{q}_\mathrm{d})
= \bm{0}.
\label{eqn:compLoss}
\end{equation}
Thus, the system no longer reacts to external forces in case of precise disturbance estimates.
This unwanted increase in stiffness could be avoided for interaction with the end-effector by exploiting wrist wrench sensing.
Setting $\kappa_{\varepsilon}=1$ in (\ref{eqn:controller}) and $\kappa_{\mathrm{ext}}=1$ in (\ref{eqn:observer_settings}), the closed loop behavior (\ref{eqn:closedloop_general}) becomes
%
\begin{equation}
\medmuskip=3mu
\thinmuskip=3mu
\thickmuskip=3mu
\bm{M(\bm{q})}\ddot{\bm{q}}-\hat{\bm{M}}(\bm{q}_\mathrm{d})\ddot{\bm{q}}_\mathrm{d}
+\bm{D}(\dot{\bm{q}}-\dot{\bm{q}}_\mathrm{d})
+\bm{K}(\bm{q}-\bm{q}_\mathrm{d})=
\bm{\tau}_\mathrm{ext,EE}\;.
\label{eqn:regainComp}
\end{equation}
This scheme has similarities to the one proposed in \cite{Oh1999} and will be termed disturbance compensation (DC) with external forces compliance (EFC) from now on. A qualitative comparison for the different settings of $\kappa_{\varepsilon}$ and $\kappa_\mathrm{ext}$ is shown in the Table of Fig.~\ref{fig:ReactionScheme}.
%

In summary, it is possible to
%
\begin{enumerate}
\item detect end-effector contacts with the inertia compensated wrench sensor,
\item detect contacts along the entire robot structure beyond a tolerance band with the extended observer,
\item render end-effector compliance while having stiff behavior for contacts along the robot structure.
\end{enumerate}

The missing compliance for link collisions can still be encountered by switching to compliance control as soon as a link collision is detected.

Using the above observer, we implemented a simple collision detection scheme that is based on a constant disturbance joint torque threshold ${\bm{\zeta}}$.
%
\begin{equation}
\text{CollDet}=
\begin{cases}
    1,& \text{if } \hat{\bm{\tau}}_{\varepsilon} > {\bm{\zeta}} \text{ (component wise)}\\
    0,& \text{otherwise}
\end{cases}
\label{eqn:CollDet}
\end{equation}
For (\ref{eqn:CollDet}) to work properly, one has to solve the trade-off between robustness and convergence speed of the observer.
The first order observer dynamics in (\ref{eqn:disttorqueconvergence}) with time constant $1/k_{\mathrm{o},i}$ makes the collision detection robust against sensor noise and peaks, as long as $\bm{K}_{\mathrm{o}}$ is not chosen too large.
However, large $\bm{K}_{\mathrm{o}}$ leads to faster convergence of $\hat{\bm{\tau}}_\mathrm{\varepsilon}$ to $\bm{\tau}_\mathrm{ext}$.

\begin{figure}
\centering
\includegraphics[scale=0.999]{./figures/ReactionScheme/reaction_scheme}
\SkipBeforePicture
\begin{center}\footnotesize
\begin{tabular}{|l|c|c|c|}
\hline
Property & $\kappa_{\varepsilon}=0$ & $\medmuskip=0mu
\thinmuskip=0mu \thickmuskip=0mu \kappa_{\varepsilon}=1, \kappa_{\mathrm{ext}}=0$ & $\medmuskip=0mu \thinmuskip=0mu \thickmuskip=0mu \kappa_{\varepsilon}=\kappa_{\mathrm{ext}}=1$ \\
\hline
Increased accuracy      & \no & \ok & \ok \\
\hline
End-effector compliance  & \ok & \no & \ok \\
\hline
Link compliance          & \ok & \no & \no \\
\hline
Closed-loop behavior & (\ref{eqn:closedloop_general}) & (\ref{eqn:compLoss}) & (\ref{eqn:regainComp}) \\
\hline
\end{tabular}
\end{center}
\SkipBeforePicture
\caption{Context sensitive control and reaction scheme and comparison between different setups of controller (\ref{eqn:controller}).
Control mode \mbox{$\kappa_\mathrm{ext}=1$} is used for tasks which need end-effector compliance, \mbox{$\kappa_\mathrm{ext}=0$} for tasks which do not. When a collision is detected, the controller switches into compliant mode ($\kappa_\varepsilon=0$).}
\label{fig:ReactionScheme}
\SkipBeforeText
\end{figure}

For collision reaction, in this work, we switch to gravity compensation mode \cite{Haddadin2014}:
\begin{equation}
\bm{\tau}_\mathrm{d}=
\begin{cases}
    \hat{\bm{g}}(\bm{q}),& \text{if CollDet}=1 \\
    \bm{\tau}_\mathrm{d} \text{ from (\ref{eqn:controller})},& \text{otherwise}\;.
\end{cases}
\label{eqn:CollReact}
\end{equation}
%
Another possibility would be to implement the scheme depicted in Fig.~\ref{fig:ReactionScheme}.
It uses the three control modes in the table in a context sensitive manner.
For example, for grasping objects end-effector compliance and high position accuracy is needed, therefore \mbox{$\kappa_\varepsilon=1$} and \mbox{$\kappa_\mathrm{ext}=1$} is used. For moving objects of unknown weight, end-effector compliance is not wanted and therefore \mbox{$\kappa_\mathrm{ext}=1$} is used.
Finally, if the collision threshold is exceeded, the robot switches into full compliant mode (\mbox{$\kappa_\varepsilon=0$}) to avoid damage.

Next, we outline our system identification and friction modeling approach.


%#!platex DeepLearningTurbidite.tex

\section{notation}
The symbols L, M and T denote dimensions of length, mass and time respectively. The symbol [1] denotes that the value is dimensionless.

\begin{description}
\item[$C_T$] Total layer-averaged sediment concentration [$\mathrm{1}$]
\item[$C_i$] Layer-averaged sediment concentration of the $i$th grain-size class [$\mathrm{1}$]
\item[$C_f$] Bed friction coefficient [$\mathrm{1}$]
\item[$e_{si}$] Sediment entrainment coefficient [$\mathrm{1}$]
\item[$F_i$] Volumetric fraction of the $i$th grain-size class in the active layer [$\mathrm{1}$]
\item[$L_a$] Thickness of the active layer [$\mathrm{L}$]
\item[$R$] Submerged specific density of sediment particles ($=1 - \rho_s / \rho_f$) [$\mathrm{1}$]
\item[$S$] Bed slope [1]
\item[$U$] Layer-averaged velocity of turbidity currents [$\mathrm{LT^{-1}}$]
 \item[$g$] Acceleration of gravity [$\mathrm{LT^{-2}}$]
 \item[$h$] Flow depth of turbidity current [$\mathrm{L}$]
 \item[$l_0$] Initial length of suspended sediment cloud [$\mathrm{L}$]
  \item[$r_{0i}$] Ratio of near-bed sediment concentration of the $i$th grain-size class to layer-averaged concentration [$1$]
 \item[$t$] Time [$\mathrm{T}$]
 \item[$w_{si}$] Settling velocity of sediment of the $i$th grain-size class [$\mathrm{LT^{-1}}$]
 \item[$x$] Bed-attached streamwise coordinate [$\mathrm{L}$]
 \item[$H_0$] Initial height of suspended sediment cloud [$\mathrm{L}$]
 \item[$J$] Number of model input parameters [1]
 \item[$K$] Number of test datasets [1]
 \item[$N$] Number of grain size classes [1]
  \item[$S_l$] Basin slope [1]
 \item[$\eta_T$] Thickness of the turbidite [$\mathrm{L}$]
 \item[$\eta_i$] Volume per unit area of sediment of the $i$th grain-size class [$\mathrm{L}$]
 \item[$\lambda_p$] Porosity of the turbidite [$\mathrm{1}$]
\item[$\rho_s$] Density of sediment particles [$\mathrm{ML^{-3}}$]
\item[$\rho_f$] Density of the water [$\mathrm{ML^{-3}}$]

\end{description}


\documentclass[a4paper]{article} %,twoside
\usepackage[english]{babel}
%\usepackage[final,ulem=normalem]{changes}

%%
%% Packages to be used.
%%
\usepackage[utf8]{inputenc}
\usepackage{amsmath,amsfonts,amsthm,amssymb,amscd}
\usepackage{booktabs}
\usepackage{multirow}
%\usepackage{zibtitlepage}
\usepackage{alltt}
\usepackage{algorithm}
\usepackage{algorithmic}
\renewcommand{\algorithmicrequire}{\textbf{Input:}}
\renewcommand{\algorithmicensure}{\textbf{Output:}}

%%
%% PDF / DVI output
%%
\usepackage{color}
\usepackage[pdftex]{graphicx}
\usepackage[pdftex]{thumbpdf}
\usepackage{epstopdf}
\graphicspath{{figures/}}

\usepackage{tikz}
\usetikzlibrary{arrows,calc,decorations.markings,intersections,positioning,shapes.arrows}

\usepackage{caption}
\usepackage{subcaption}

\usepackage{cite}
\usepackage[colorlinks=true]{hyperref}

\usepackage{url}
\usepackage{todonotes}

\usepackage{listings}
\lstset{language=Fortran,
	basicstyle=\small\sffamily,
	columns=fullflexible,
	captionpos=b}

\lstdefinelanguage{tt}{
		basicstyle=\ttfamily}

%Remove later, used only for getting todos in the margin
%\setlength{\marginparwidth}{3cm}

%%
%% Special sets.
%%
\newcommand{\N}{\mathbb{N}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\One}{\boldsymbol{1}}

%%
%% Mathematical convenience.
%%
\DeclareMathOperator{\ld}{log_2}
\DeclareMathOperator{\convex}{co}
\newcommand{\half}{\frac{1}{2}}
\newcommand{\Ol}[1]{\mathcal{O}(#1)}
\newcommand{\dual}[1]{#1^\star}
\newcommand{\norm}[1]{\left\|#1\right\|}
\newcommand{\const}{\mathrm{const}\,}
\renewcommand{\d}{\mathrm{d}}

\usepackage{pgfplots}
\usepackage{pgfplotstable}
\pgfplotsset{compat=newest}
\usepackage{booktabs}

\begin{document}

\title{Optimal Control of the Initial Condition for the KS Equation With LibPFASST}
\author{Sebastian Götschel}
\maketitle

\section{Problem setting}

We are considering the 1D Kuramoto-Sivashinsky equation
\begin{equation} \label{eq:KS}
y_t = -y y_x - y_{xx} - y_{xxxx}, \quad x \in \Omega := [0,32\pi]
\end{equation}
with periodic boundary conditions on some time interval $t \in [0,T]$. Given some measured data (in the optimal control setting called \emph{desired state}) $y_d(t,x)$, the aim is to identify an initial condition $y(0,x) = u(x)$ such that the solution $y(t,x;u)$ minimizes the deviation from the desired state. We will often refer to $u$ as the \emph{control}. Put into formula, we want to minimize the \emph{objective functional}
\begin{equation}\label{eq:obj}
J(y,u) := \frac{1}{2}\int_{0}^{T} \int_\Omega (y-y_d)^2 \mathrm{d}x\ \mathrm{d}t + R(u),
\end{equation}
where $R$ is some generic regularization term, e.g., $R(u) = \frac{\lambda}{2} \norm{u}^2$.

\paragraph{Gradient-based optimization.} For the gradient-based methods considered here, the optimization iteration proceeds as
\begin{align}
u_{k+1} &= u_k + \alpha_k d_k \label{eq:update_u} \\
d_{k+1} &= -\nabla j(u_{k+1})+ \beta_k d_k, \label{eq:update_d}
\end{align}
where $k$ is the iteration number, $\nabla j(u_{k})$ the \emph{reduced gradient}, and $\alpha_k$ some step length for the control update. The parameter $\beta_k$ is used to update the descent direction $d_k$ and determines the type of method (steepest descent for $\beta_k = 0$, various choices for nonlinear conjugate gradient methods).

\paragraph{Computation of the reduced gradient.} Adjoint gradient computation consists of three steps:
\begin{enumerate}
	\item Solve the PDE \eqref{eq:KS} for a given control $u$
	\item Solve the adjoint PDE
	\begin{equation}\label{eq:adjoint}
		-p_t = p p_x - p_{xx} - p_{xxxx} + (y-y_d), \quad x \in \Omega := [0,32\pi]
	\end{equation}
	with periodic boundary boundary conditions and $p(T,x) = 0$.
	\item Set $\nabla j(u) =  p(0,x) + R'(u)$.
\end{enumerate}
The source term in eq.~\ref{eq:adjoint} comes from differentiating the objective functional \ref{eq:obj} with respect to the state $y$. If, instead of or in addition to, the difference $y-y_d$ in $[0,T] \times \Omega$ we track some desired final state $y_d^T(x)-y(T,x)$ the terminal condition in the adjoint gets modified to $p(T,x) =  y_d^T(x)-y(T,x)$ to account for this deviation.

For details on PDE-constrained optimization see, e.g., \cite{HinzeEtAl2009,Troeltzsch2010}.

\section{Implementation}

This tutorial example consists of six source files
\begin{itemize}
	\item \lstinline+hooks.f90+ - output residuals during sweeps/PFASST iterations
	\item \lstinline+level.f90+ - interpolation/restriction of solutions
	\item \lstinline+main.f90+ - main program, contains main optimization loop
	\item \lstinline+pf_optimization_1d.f90+ - routines for evaluating objective, gradient, steps size selection
	\item \lstinline+probin.f90+ - define problem parameters
	\item \lstinline+sweeper.f90+ - defines problem equations and optimization quantities
\end{itemize}
as well as
\begin{itemize}
	\item \lstinline+Makefile+
	\item \lstinline+imex.nml+ - specify input parameters for test runs
	\item \lstinline+plot.py+ - helper file to produce figures from npy output.
\end{itemize}

In the following we provide a walkthrough through the main parts of the implementation. For all details you will need to go through the actual source code, though.

\subsection{Encapsulation, sweeper, and controller for optimal control}

The LibPFASST library comes with a special data structure \lstinline|pf_ndarray_oc_t| (file \lstinline|pf_ndarray_oc_encap.f90|) to store solutions as well as a IMEX sweeper \lstinline|pf_imexQ_oc_t| (file \lstinline|pf_imexQ_oc_sweeper.f90|) capable of forward- and backward-in-time IMEX-SDC sweeps.

As the optimal control problem requires state and adjoint solution, PFASST's \lstinline|pf_ndarray_t| is extended to contain two variables stored in \lstinline|yflatarray| and \lstinline|pflatarray|. It provides all the functionality of \lstinline|pf_ndarray_t|, with the functions taking an optional \lstinline|flag| argument specifying on which component to operate on (\lstinline|flag==1| operates on \lstinline|yflatarry|, \lstinline|flag==2|  on \lstinline|pflatarry|, and \lstinline|flag==0| on both).

Similarly, the optimal control sweeper provides the same functionality as the standard IMEX sweeper \lstinline|pf_imex_sweeper_t|, again with a \lstinline|flag| argument for the function specifying the component and thus forward or backward sweeps (or both).

Finally, for running PFASST routines like \lstinline|pf_predictor_oc|, \lstinline|pf_pfasst_block_oc| or \lstinline|pf_vcycle_oc| are provided in \lstinline|pf_parallel_oc.f90|. These routines are similar to the standard PFASST functions in \lstinline|pf_parallel.f90|, but adapted to the optimal control setting (e.g., sweeping backwards on the adjoint).

In most cases, the user should not need to edit these library files to solve application problems.

\subsection{The optimization loop}

Besides setting up PFASST and allocating data structures for various quantities used during the optimization, the main part of solving the optimal control problems happens in the optimization loop in \lstinline|main.f90|, see Listing~\ref{lst:optloop}.

\begin{lstlisting}[frame=tb,caption={Most important parts of the optimization loop from \lstinline|main.f90|},label=lst:optloop]
do k=1,max_opt_iter
	if (k .eq. 1) then
		call evaluate_objective(...)
	 else
		objective=objectiveNew ! from step size selection/linesearch
	end if

	call mpi_allreduce(objective, globObj, 1, MPI_REAL8, MPI_SUM, pf%comm%comm, ierror)
	if (globObj < tol_obj) exit ! leave loop

	if (k .eq. 1) then   ! in later iterations, this is done in linesearch
		call evaluate_gradient(...)
	end if

	 ! compute gradient norms
	globL2NormGradSq = sum(gradient**2)*Lx/dble(nvars(pf%nlevels))
	globLinftyNormGrad = maxval(abs(gradient))

	if (sqrt(globL2NormGradSq) < tol_grad) exit

	! determine new search direction, here: steepest descent
	beta = 0.0_pfdp
	searchDir = -gradient + beta*prevSearchDir

	! line search for step size selection
	call armijo_step()

	if (stepTooSmall) exit
end do
\end{lstlisting}
%
Algorithmic parameters like \lstinline|max_opt_iter| or \lstinline|tol_grad| are defined in \lstinline|probin.f90| and can be set via command line parameters or an \lstinline|*.nml| file. The main work happens in the functions \lstinline|evaluate_objective| and \lstinline|evaluate_gradient| from \lstinline|pf_optimization_1d.f90|, where state and adjoint equations are solved, the objective functional gets evaluated and the gradient assembled. Only in the first optimization iteration these are called directly; in the following iterations they get called during linesearch (e.g., \lstinline|armijo_step|).

Solving the state equation by PFASST is done in \lstinline|evaluate_objective|, and is shown in excerpts in Listing~\ref{lst:evalobj}.
\begin{lstlisting}[frame=tb,caption={\lstinline|evalute_objective| from \lstinline|pf_optimization_1d.f90|},label=lst:evalobj]
subroutine evaluate_objective(...)
	! ...
	! set initial condition
	q1%yflatarray = ctrl
	call pf%levels(pf%nlevels)%q0%copy(q1, 1)
	do step = 1, nsteps ! this iterates over all steps this processor computes
		thisstep = (step-1)*pf%comm%nproc + pf%rank
		! this is the 'true' time step we are computing
		! i.e., this processors time interval in this step
		! is [thisstep*dt, (thisstep+1)*dt]

		! use PFASST to solve on this block of time steps, i.e., every processors computes
		! their time step in parallel, exchanging data to transmit
		! initial conditions on the time intervals
		pf%state%pfblock = step
		call pf_pfasst_block_oc(pf, dt, step*pf%comm%nproc, .true., 1, step=thisstep)

		! evaluate objective functional for all quadrature nodes
		do m = 1, nnodes
			call objective_function(...)
		end do
		! integrate objective functional
		objective = ...

		! copy/broadcase qend to q0 for next step
		if( step < nsteps ) then
			call pf%levels(pf%nlevels)%qend%pack(...)
			call pf_broadcast(...)
			call pf%levels(pf%nlevels)%q0%unpack(...)
		end if
	end do

	! L2(Omega) control norm
	L2NormCtrlSq = sum(ctrl**2)*Lx/dble(nx)
    objective = 0.5*objective + 0.5*alpha*L2NormCtrlSq
end subroutine evaluate_objective
\end{lstlisting}
The main difference to the other Tutorial examples, where a single PDE is solved only once, is, that we cannot use the usual PFASST controller, but have assign blocks of time steps to processors by hand as well as record the solution values in order to solve the adjoint equation backward in time. There, the processors need to solve the specified time steps for which they have the state equation's solution, thus avoiding costly communication as well as memory issues, as every processor only has to store the values it actually needs.

The \lstinline|evaluate_gradient| is similar, and consists of solving the adjoint and constructing the gradient. As the gradient requires the value of the adjoint at $t=0$, the first processor broadcasts this solution to the other processors, see Listing~\ref{lst:evalgrad}. This broadcast could be left out when it is ensured that only the first processor updates the control (i.e., the initial condition); this modification is left as an exercise for the reader.

\begin{lstlisting}[frame=tb,caption={Excerpt of \lstinline|evalute_gradient| from \lstinline|pf_optimization_1d.f90|},label=lst:evalgrad]
if(pf%rank == 0) then
	! gradient is adjoint at t0 (plus regularization term derivative = alpha*ctrl)
	! pack away on rank 0
	call pf%levels(pf%nlevels)%q0%pack(pf%levels(pf%nlevels)%send, 2)
end if
! broadcast
call pf_broadcast(pf, pf%levels(pf%nlevels)%send, pf%levels(pf%nlevels)%mpibuflen, 0)
! everybody sets gradient (actually this could be skipped, only rank 0 needs initial condition)
gradient = pf%levels(pf%nlevels)%send
gradient = gradient + alpha * ctrl
\end{lstlisting}

We do not discuss the line search functions \lstinline|armijo_step| and \lstinline|strong_wolfe_step| here. They are rather straightforward implementations of the standard Armijo- and strong Wolfe conditions (see your favorite textbook on nonlinear optimization, e.g., \cite{NocedalWright2006}).

\subsection{Specifying the equations}

During sweeps, the sweeper calls \lstinline|feval| and \lstinline|fcomp| routines, which are defined in \lstinline|sweeper.f90|. Here the equations are specified. We use an IMEX approach, treating $-y_{xx}-y_{xxxx}$ implicitly.

\begin{lstlisting}[frame=tb,caption={Excerpt of \lstinline|feval| from \lstinline|sweeper.f90|},label=lst:feval]
select case (flags) ! are we solving for state or adjoint?
case(1) ! State
	!  Grab the arrays from the encap
	yvec => get_array1d_oc(y,1) ! ( ,1) means get state component
	fvec => get_array1d_oc(f,1)

	! explicit piece: -y y_x
	tmp = -0.5_pfdp*yvec*yvec
	call fft%conv(tmp, this%ddx, fvec)

case(2) ! Adjoint
	!  Grab the arrays from the encap
	yvec => get_array1d_oc(y,1) ! state sol required for rhs (y-y_desired)
	fvec => get_array1d_oc(f,2)
	pvec => get_array1d_oc(y,2)

	! explicit piece: +y y_x
	tmp = 0.5_pfdp*pvec*pvec
	call fft%conv(tmp, this%ddx, fvec)

	! source term from distributed tracking objective
	fvec = fvec + (yvec-this%ydesired(mystep, idx, :))
end select
\end{lstlisting}

Listing \ref{lst:feval} shows an excerpt of the \lstinline|feval| function, specifying the explicit part of the equation. \lstinline|flags| specifies which equation is to be solved, \lstinline|flags==1| is the state equation, \lstinline|flags==2| specifies the adjoint. In the latter, the source term from the distributed tracking type objective enters. The implicit piece is also evaluated using FFT, \lstinline|call fft%conv(yvec,-this%lap-this%lap*this%lap,fvec)|; this does not differ between state and adjoint.

\begin{lstlisting}[frame=tb,caption={Excerpt of \lstinline|fcomp| from \lstinline|sweeper.f90|},label=lst:fcomp]
if (piece == 2) then
	! Apply the inverse operator with the FFT convolution
	call fft%conv(rhsvec,1.0_pfdp/(1.0_pfdp - dtq*(-this%lap*this%lap-this%lap)),yvec)
	fvec = (yvec - rhsvec) / dtq
end if
\end{lstlisting}

The \lstinline|fcomp| function in Listing \ref{lst:fcomp} is similar to the other tutorial examples and has no modifications due to the optimization problem.

\subsection{Helper and convenience functions}

In \lstinline|sweeper.f90| some helper functions specific to the optimization problem can be found. These are, e.g., \lstinline|initialize_ocp| to allocate memory for the desired state, or \lstinline|set_ydesired| to fill the variable with a desired state coming from measurements or artificially generated data. More importantly, functionality to evaluate the objective functional is provided (\lstinline|objective_function(...)|), as well as functions to restrict the desired state (and the computed fine state solution) to coarser levels as required for solving the adjoint.

\section{Numerical example}

Here we show two numerical results, without giving much detail. Please consider \lstinline|imex.nml| for input parameters.

\subsection{Forward solve}

We use the example from \cite{KassamTrefethen2005} and solve the KS equation for $x\in[0,32\pi]$ on the time interval $[0,60]$ using the initial condition $y(0,x) = \cos(x/16)(1+\sin(x/16))$. The solution using 600 time steps and two levels with 3/5 Lobatto nodes and 512/1024 spatial degrees of freedom is shown in Fig.~\ref{fig:KS60}.

\begin{figure}
\includegraphics[scale=0.7]{figures/solKS_T60_nstep600_nx1024.png}
\caption{Solution of the KS equation}
\label{fig:KS60}
\end{figure}

\subsection{Identifying the initial condition}

Due to the complicated dynamics, identifying the initial condition is demonstrated on the rather short time interval $[0,1]$, with 100 time steps. The other discretization parameters are as above. We start the optimization with an initial guess $u_0 = 0.75 u_\text{exact}$. The target state is shown in Fig.~\ref{fig:yd}. Fig.~\ref{fig:optprogress} shows the progress of the steepest descent method, while Fig.~\ref{fig:ctrl} shows the computed control (i.e., identified initial condition) as well as the difference of the identified and true initial condition.

\begin{figure}
	\includegraphics[scale=0.7]{figures/solKS_T1_nstep100_nx1024_yd.png}
	\caption{Target state (KS solution with correct initial condition).}
	\label{fig:yd}
\end{figure}

\begin{figure}
	\includegraphics[width=0.5\textwidth]{figures/gradientKS_T1_nstep100_nx1024_u0_075uexact_progress.pdf}
	\includegraphics[width=0.5\textwidth]{figures/objKS_T1_nstep100_nx1024_u0_075uexact_progress.pdf}
	\caption{Optimization progress. Left: norm reduced gradient. Right: Target mismatch.}
	\label{fig:optprogress}
\end{figure}

\begin{figure}
	\includegraphics[width=0.5\textwidth]{figures/ctrlKS_T1_nstep100_nx1024_u0_075uexact_progress.png}
	\includegraphics[width=0.5\textwidth]{figures/ctrlKS_T1_nstep100_nx1024_u0_075uexact_diff_final.png}
	\caption{Left: Initial and final control. Right: difference of final to exact control.}
		\label{fig:ctrl}
\end{figure}

\bibliographystyle{plain}
\bibliography{references}

\end{document}


\section{Evolution of Trust}
  \subimport{thesis/definitions/}{turns.tex}
  \subimport{common/}{turnexample.tex}
  We use $prev\left(j\right)$ and $next\left(j\right)$ to denote the previous and next turn respectively played by
  $Player(j)$. A formal definition can be found in the Appendix.
  \subimport{common/definitions/}{damage.tex}
  \subimport{common/definitions/}{history.tex}
  \noindent Knowledge of the initial graph $\mathcal{G}_0$, all players' initial capital and the history amount to full
  comprehension of the evolution of the game. Building on the example of Fig.~\ref{fig:utxo}, we can see the resulting graph
  when $D$ plays
  \begin{equation}
  \label{turnexample}
    Turn_1 = \{Steal\left(1, A\right), Add\left(4, C\right), Add\left(-1, B\right)\} \enspace.
  \end{equation}
  \subimport{common/figures/}{turnexample.tikz}

  \noindent In the form presented here, Trust Is Risk is controlled by an environment that chooses a player, receives the turn
  that this player wishes to play and, if this turn is valid, executes it. These steps are repeated indefinitely. We assume
  players are chosen in a way that, after her turn, a player will eventually play again later.
  \subimport{thesis/algorithms/}{trustisriskgame.tex}

  \noindent \texttt{strategy[}$A$\texttt{]()} provides player $A$ with full knowledge of the game, except for the capitals of
  other players. This assumption may not be realistic because out of band knowledge and traffic analysis methods can be used
  to infer the capital of other players.

  \texttt{executeTurn()} checks the validity of \texttt{Turn} and substitutes it with an empty turn if invalid.
  Subsequently, it creates the new graph $\mathcal{G}_j$ and updates the history accordingly. For the routine code,
  see the Appendix.


%!TEX root = ../Thesis.tex
\chapter{Introduction}\label{cha:introduction}
%
Here's a citation: \cite{Wacker2016}. Two pictures are shown, Figures \ref{fig:morten} and \ref{fig:eps}. There's also a table: Table \ref{tab:a-table}.

\section{A Section}\label{sec:a-section}

\subsection{A Subsection}\label{ssec:a-subsection}

\subsubsection{A Subsubsection}\label{sssec:a-subsubsection}

\lipsum[2]

\begin{figure}[htbp]
  \centering
  \includegraphics[width=.3\textwidth]{img/morten}
  \caption{Some dude}
  \label{fig:morten}
\end{figure}

\begin{figure}[htbp]
  \centering
  \includegraphics[width=.7\textwidth]{img/time-constant}
  \caption{An eps image}
  \label{fig:eps}
\end{figure}

\begin{table}[htbp]
  \centering
  \caption{A table}
  \begin{tabular}{cc}
    \toprule
    Something & Something else \\
    \midrule
    A & a \\
    B & b \\
    \bottomrule
  \end{tabular}
  \label{tab:a-table}
\end{table}

\begin{equation*}
  \begin{aligned}
      & \underset{\*x, \*u}{\text{min}}
      & & \int_{t_k}^{t_{k+T}} ( \*{e}(\*{x})^{\top} \*{Q}_p \*{e}(\*{x}) +
      \dot{\*{e}}(\*{x})^{\top} \*{Q}_d \dot{\*{e}}(\*{x}) +
      \ddot{\*{q}}^{\top} \*{R}_{\ddot{\*{q}}} \ddot{\*{q}} +
      \*{u}^{\top} \*{R}_{\*{u}} \*{u} ) \,dt\\
      & \text{s.t.}
      & & \dot{\*x} = \*f(\*x,\*u), \\
      &&& \bar{\*x}_{lb} \leq \*x \leq \bar{\*x}_{ub}, \\
      &&& \bar{\*u}_{lb} \leq \*u \leq \bar{\*u}_{ub}, \\
      &&& \bar{\ddot{\*q}}_{lb} \leq \ddot{\*q} \leq \bar{\ddot{\*q}}_{ub}, \\
      &&& \*x(t_k) = \*x_0. \\
      \end{aligned}
\end{equation*}

\begin{equation*}
  f(x,u) = \begin{bmatrix} \dot{q} \\ M(q)^{-1} (\tau - C(q, \dot{q}) - G(q)) \end{bmatrix}
\end{equation*}

\begin{equation*}
  p(\*y | \*X, \boldsymbol{\theta}) = \dots
\end{equation*}


\chapter{Motivation}
\label{cha:motivation}
In this chapter, we describe the motivations behind adding copatterns to
Idris. We find that copatterns can be a means of recovering from an unfortunate
loss of subject reduction. However, copatterns might not be the universal tool
for coinductive reasoning. Additionally, we motivate the addition of an
inference system for guarded recursion.

\section{Recovering Subject Reduction}
\label{sec:recov-subj-reduct}
%#########
% Hvorfor mistes subject reduction ved dependent pattern matching på codata?
% Mister Idris subject reduction?
% Modeksempel (helst implementeret i Idris)
% Hvordan afhjælper copatterns problemet?

% Ny viden: Copatterns kan være et værktøj til at undgå at man mister subject reduction
%#########
Subject reduction is a property of a type system. Also called type preservation,
a type system has the property of subject reduction if evaluation of a
well-typed term does not cause its type to change\,\citep[Section~8.3.3]{Pierce:2002:TPL:509043}.
In some dependently typed programming languages with explicit coinductive types,
e.g. Coq and Idris, values of such types are given as (potentially) infinite
trees of constructors. Accordingly, they can be analysed with (dependent)
pattern matching. However, this leads to a loss of subject reduction. The
problem was first identified by Gim\'{e}nez\,\citep{Gimenez96uncalcul} for the
Calculus of (Co)Inductive constructions, but was later (in 2008) shown by Oury
to persist in both Coq and
Agda\,\citep{OuryCounterexampleCoq,OuryCounterexampleAgda}.

By implementing Oury's counterexample
(Figure~\ref{fig:ourys_counterexample_idris}), we have found that the problem
arises in Idris as well. Oury's counterexample unfolds as follows: Given a
coinductive type \texttt{A} with a single constructor \texttt{A} $\to$ \texttt{A}, which we may call
\texttt{In}, and an inhabitant \texttt{b} of \texttt{A} (of which there exists only one,
namely the fixed point of \texttt{In}), we can show that \texttt{b} is definitionally
equal to its own unfolding by exploiting dependent pattern matching. When
\texttt{b} is given as argument to \texttt{forceEq} in
Figure~\ref{fig:ourys_counterexample_idris}, \texttt{b} is reduced to \texttt{In
  (Delay (In (Delay a)))} by dependent pattern matching, such that \texttt{forceEq b}
reduces to \texttt{Refl}. These reductions ensure that the implementation of
\texttt{p} is well-typed. The problem arises when we attempt to replace the
right-hand side of \texttt{p} with \texttt{Refl}, which is the definition of
\texttt{forceEq b}. In this case, the Idris type checker rejects the program,
complaining that \texttt{b} cannot be unified with \texttt{In b}. Thus, the
evaluation of \texttt{forceEq b} changes the type of \texttt{Refl}, and subject
reduction is lost.

\begin{figure}
\begin{lstlisting}[mathescape]
codata A : Type where
  In : A $\to$ A

b : A
b = In b

force : A $\to$ A
force (In (Delay (In (Delay a)))) = In (Delay (In (Delay a)))

forceEq : (x : A) $\to$ x = force x
forceEq (In (Delay (In (Delay a)))) = Refl

p : b = In (In b)
p = forceEq b
\end{lstlisting}
  \caption{Oury's counterexample implemented in Idris, leading to a loss of
    subject reduction.}
\label{fig:ourys_counterexample_idris}
\end{figure}

Dependent pattern matching on coinductive data is problematic, but we cannot
simply disallow it, since we then lose the ability to write any interesting
programs involving coinductive data. This ability can be regained by introducing
copatterns. Instead of analysing coinductive data with dependent pattern
matching, we can choose to synthesize it using copatterns. In
Figure~\ref{fig:ourys_counterexample_copatterns}, we have translated Oury's
counterexample to an Idris implementation with copatterns, where we have
refrained from using dependent pattern matching. Using copatterns, we can still
define both \texttt{b} and \texttt{force}, but we cannot provide an
implementation for \texttt{p}, since the reduction behaviour from dependent
pattern matching is (virtually) unavailable. Hence, we have both regained
expressiveness and avoided this particular loss of subject reduction.

\begin{figure}[h]
\begin{lstlisting}[mathescape]
corecord A : Type where
  out : A $\to$ A

b : A
&out b = b

force : A $\to$ A
&out force a = out a
&out &out force a = out (out a)

forceEq : (x : A) $\to$ x = force x
forceEq a = $\uwave{\text{Refl}}$

p : b = out (out b)
p = $\uwave{\text{forceEq b}}$
\end{lstlisting}
  \caption{By using copatterns to synthesize coinductive data, we can both
    implement functions on coinductive data and preserve subject
    reduction. The underlined right-hand sides indicate type errors detected by
    the Idris type checker.}
\label{fig:ourys_counterexample_copatterns}
\end{figure}

In Agda, pattern matching on coinductive data is currently
disallowed\,\citep{Danielsson09}, providing the user with copatterns instead. By
implementing copatterns in Idris, we provide a stepping stone in the same
direction. However, this solution may seem to avoid the problem, rather than
solving it. As pointed out by McBride\,\citep{McBride:2009}, the underlying
problem is that we can have intensional equalities between coinductive values
which are merely bisimilar. To remedy the situation, he proposes that one could
have an observational propositional equality, which takes into account both
functional extensionality and bisimilarity for coinductive values. Also,
Altenkirch et al.\,\citep{Altenkirch:2007} proposes the adoption of an
observational type theory, which is essentially an intensional type theory where
one can have propsitional equalities up to observation, as opposed to
construction. But until these ideas are incorporated into a practical system,
avoiding the problem seems superior to ignoring it.

\section{The Use Case for Copatterns}
\label{sec:motivation_copatterns}
%############
% Pattern matching on data with top-level product structure vs. copatterns
% Mixed induction-coinduction, coinductive resumption monad
% Coinduction (definition og operationel intuition) (måske bisimilarity)

% Ny viden: Hvornår bør/kan jeg bruge copatterns?
%############
% Pattern matching has become a ubiquitous tool for analysing data in functional
% programming languages. In combination with inductive data types, pattern
% matching has evolved into such a widely applicable technique that many users may

% be willing to disregard its drawbacks. Consider the following implementation of
% a function which interleaves a filtered version of \texttt{xs} with \texttt{ys}:
% \begin{lstlisting}[mathescape]
% interleaveFilter : (a $\to$ Bool) $\to$ Stream a $\to$ Stream a $\to$ Stream a
% interleaveFilter p (x :: xs) (y :: ys) =
% if p (head xs) then case xs of
%                      xx :: xxs
%    case (filter p xs) of
%      px :: _ => (px :: y :: (interleaveFilter p xs ys)
% \end{lstlisting}
% \begin{lstlisting}[mathescape]
% interleaveFilter : (a $\to$ Bool) $\to$ Stream a $\to$ Stream a $\to$ Stream a
% head       (interleaveFilter p xs ys)  = head (filter p xs)
% head (tail (interleaveFilter p xs ys)) = head ys
% tail (tail (interleaveFilter p xs ys)) =
%                                interleaveFilter p (tail xs) (tail ys)

% mapNth : (a -> b) -> Nat -> Nat -> Stream a -> Stream b
% mapNth f (S n') m s = case s of
%                        x :: xs => x :: mapNth f n' xs
% mapNth f Z m s = case s of
%                   x :: xs => f x :: mapNth f m m xs

% \end{lstlisting}

%\subsection{Copatterns are Useful for Record Types}
% Recursive types!
\label{sec:prod-vs.-copr}
If we must disallow pattern matching on coinductive data to preserve subject
reduction, how much expressiveness can we recover by using copatterns? At
present, not as much as one might have hoped. Copatterns are well-suited for
working with data that has a top-level product structure. In the general sense
of recursive types, this means that copatterns can be readily used for types
which has a product-of-sums structure, i.e.
${\mu X.\,A\times B\times\cdots\times X}$ or
${\nu X.\,A\times B\times\cdots\times X}$, since a well-defined projection
exists for each $A,\,B,\,\cdots,\,X$. In Idris, this amounts exactly to the
types which can be defined as (co)inductive record types. Consider the
deterministic finite automaton defined in Figure~\ref{fig:state_machine}, which
accepts strings of binary numbers that contain an even number of zeros (a
graphical representation is given for reference in
Figure~\ref{fig:state_machine_graphical}). Inspired by
Jacobs\,\citep{JacobsCoalgebra}, we define a state as a type
${\mu S. (A\,\to\,S)\times Bool}$, where $A\,\to\,S$ is the transition function
(defined for each state) and $Bool$ indicates whether the state is a final
(i.e. accepting) state. Consequently, we can define each state (\texttt{SA} and
\texttt{SB} in Figure~\ref{fig:state_machine}) quite elegantly using copatterns.

\begin{figure}[h]
\begin{lstlisting}[mathescape]
data Binary = Zero | One

record State a where
  transitions : a $\to$ State a
  isFinal     : Bool

mutual
  SA : State Binary
  &transitions SA = ts
   where
     ts : Binary $\to$ State Binary
     ts Zero = SB
     ts One  = SA
  &isFinal     SA = True

  SB : State Binary
  &transitions SB = ts
   where
     ts : Binary $\to$ State Binary
     ts Zero = SA
     ts One  = SB
  &isFinal     SB = False

isAccepted : List Binary $\to$ Bool
isAccepted = isAccepted$'$ SA
  where
   isAccepted$'$ : State Binary $\to$ List Binary $\to$ Bool
   isAccepted$'$ s []        = isFinal s
   isAccepted$'$ s (b :: bs) = isAccepted$'$ ((transitions s) b) bs
\end{lstlisting}
  \caption{A finite state machine implemented using copatterns, identifying
    binary strings which contain an even amount of zeros.}
\label{fig:state_machine}
\end{figure}

\begin{figure}[h]
\centering
\includegraphics{figures/dfa}
\caption{A graphical representation of the automation defined in
  Figure~\ref{fig:state_machine}.}
\label{fig:state_machine_graphical}
\end{figure}

Unfortunately, copatterns are not very well-suited for types that have a top-level
sum-of-products structure, i.e. ${\mu X.\,A + B + \cdots + X}$ and
${\nu X.\,A + B + \cdots + X}$, because values of such types can take on
different forms in different contexts. In other words, the observable behaviour
depends on a further analysis, and the projections must therefore make
such an analysis possible. Consider the possibly infinite list encoded in
Figure~\ref{fig:colist}. Here, it is not possible to have a projection that
accesses the first element of a \texttt{CoList} directly, since it may be
empty. Instead, we can access an unfolding of the list, which may give either
answer.

\begin{figure}[h]
\begin{lstlisting}[mathescape]
corecord CoList a where
  unfold : Either (a, CoList a) ()

map : (a $\to$ b) $\to$ CoList a $\to$ CoList b
&unfold map f xs = case (unfold xs) of
                      Left (x, xs) => Left (f x, map f xs)
                      Right () => Right ()
\end{lstlisting}
  \caption{A possibly infinite list type and a \texttt{map} function operating on it.}
  \label{fig:colist}
\end{figure}

The inherent uncertainty or ambiguity in types with sums-of-product structure
cannot be elegantly handled with copatterns. We can try to eliminate the need
for further analysis by tagging all values with an indicator of its state, as
shown with the modified CoList in Figure~\ref{fig:dependent_colist}. However, as
soon as the value of the tag is given indirectly, e.g. by reference to other values,
the type checker may become unable to reduce the tag to a canonical form, thereby
defeating its purpose.

\begin{figure}[h]
\begin{lstlisting}[mathescape]
corecord CoList a where
  isNil : Bool
  elem  : if isNil then () else (a, CoList a)

nil : CoList a
&isNil nil = True
&elem  nil = ()

cons : a $\to$ CoList a $\to$ CoList a
&isNil cons x xs = False
&elem  cons x xs = (x, xs)

toggle : Nat $\to$ Nat $\to$ CoList Nat
&isNil toggle n m = False
&elem toggle n m = (n, (cons m (toggle m n)))
\end{lstlisting}
\caption{A version of CoList defined with a boolean tag indicating whether
  additional elements of the list are available.}
\label{fig:dependent_colist}
\end{figure}

Since many useful techniques involve coinductive types which do not have a
straightforward product-of-sums structure, e.g. coinductive
resumptions\,\citep{Pirog2014273} and mixed
induction-coinduction\,\citep{Danielsson09mixinginduction}, \texttt{co}patterns
are therefore not necessarily a valuable tool for defining all kinds of
\texttt{co}inductive data. Instead, they provide elegant definitions when your data can be
clearly defined by its external properties, rather than by its internal
structure. Hence, we find copatterns at this stage to be universally \emph{applicable}, but not
universally \emph{useful}.

\section{Less Restrictive Productivity Checking}
\label{sec:less-restr-prod}
% Hvorfor vil vi inferere guarded recursion?
%###########
% Dette afsnit har i princippet intet at gøre med copatterns!
% Referer til baggrund om syntactic guardedness
% Hvad kan guarded recursion som syntactic guardedness?
% Hvorfor er guarded recursion frygteligt at skrive / ikke brugervenligt?
% Bedre end syntactic guardedness + Bruger vil ikke skrive det => Inferer det

% Ny viden: Hvorfor vil vi gerne inferere guarded recursion?
%###########
Because the current productivity checker in Idris uses the syntactic
guardedness principle to prove productivity, users writing programs on
coinductive types must place all recursive calls directly under a call to a
coinductive constructor. This limitation means that we cannot have simple
definitions such as \texttt{toggle} from Figure~\ref{fig:dependent_colist}
proven total by the compiler. Using the technique of guarded recursion presented
in Section~\ref{sec:guarded-recursion} we are able to prove that toggle is
total, as shown in Figure~\ref{fig:toggle_guarded_recursion}. The proof is quite
involved, as it also requires us to define guarded recursive versions of
\texttt{cons} and \texttt{CoList} on which the original definition of
\texttt{toggle} depends. The coinductive \texttt{CoList} must be amended such
that the recursive reference in the \texttt{elem} projection is not be
immediately available.

\begin{figure}[h]
\begin{lstlisting}[mathescape]
corecord $_g$CoList a where
  isNil : Bool
  elem  : if isNil then () else (a, $\laterkappa$(CoList a))
  constructor CoCons

cons : $\forall\kappa.$ a $\to$ $_g$CoList a $\to$ $_g$CoList a
cons = $\Lambda\kappa$. fix$^\kappa$($\lambda$rec.$\lambda$x.$\lambda$xs. CoCons$\ $False (x, (Next xs)))

toggle$'$ : $\laterkappa$(Nat $\to$ Nat $\to$ Stream) $\to$ Nat $\to$ Nat $\to$ $_g$CoList Nat
toggle$'$ rec n m = CoCons False (n, (cons m ((rec $\tensor$ m) $\tensor$ n)))

toggle : $\forall\kappa.$ Nat $\to$ Nat $\to$ $_g$CoList Nat
toggle = $\Lambda\kappa.$ fix$^\kappa$($\lambda$rec.$\lambda$n.$\lambda$m. toggle$'$ rec n m)
\end{lstlisting}
  \caption{An implementation of \texttt{toggle} from
    Figure~\ref{fig:dependent_colist} using guarded recursion, given in
    Idris-like syntax.}
\label{fig:toggle_guarded_recursion}
\end{figure}

Although guarded recursion widens the range of programs we can prove total as
compared to syntactic guardedness, requiring the user to write guarded recursive
programs is probably not feasible due to their complexity. Even disregarding
this complexity, building such programs quickly becomes an onerous
task. Consequently, we would like to have a system which can automatically build
guarded recursive versions of at least some user-written programs, lifting the burden of
building productivity proofs by guarded recursion from the user and onto the
compiler. The structure and implementation of such a system will be the subject
of Chapter~\ref{cha:infer-guard-recurs}.

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "../copatterns-thesis"
%%% End:


\section{Feature Evaluation}

Successful classification largely depends on the predictive value of the features. We, therefore, evaluate for the features whether they will be useful in the discrimination between the formal and informal classes. The evaluation requires the satellite image to be divided into the formal and informal classes. This division is represented using a ground truth, which is a mask covering the satellite image the location and shape of the slums. The mask is used to divide each block of pixels into the two classes: either formal or informal. Each block contains a vector of features, characterizing that particular block of pixels. All blocks of the same class will be grouped, allowing for the analysis of the distribution values of the two classes. If the distribution of the values of a particular feature varies substantially between the two classes, it should be a suitable candidate for classification.

The ground truth mask is constructed using a vector file containing the boundaries of informal areas. The boundary file is rasterized and applied on top of the satellite image, creating a mask of the location of the informal areas. Because the feature calculation is performed on blocks instead of pixels, we transform the pixel based mask into a block-based mask where every pixel represents a block of pixels in the original image. In the mask, the pixels with the value zero represent blocks in the image designated as formal while ones represent blocks of slums.

We visualize the predictive value of the features using a boxplot and a kernel density estimation plot. When a feature is distinctive, the distribution of the values in the two classes will be different, thus creating a visual difference in both plots. We attempted to use KL divergence as an objective measurement as an alternative for the visually inclined box plot and kernel density estimation. However, we were not able to extract reasonable KL divergence values from our data as for many outcomes would indicate a divergence of infinite value. We will therefore visually evaluate the differences between the formal and informal value distributions in the two types of plots.

The features will be calculated for all combinations of image sections, color bands, block sizes, and scales, resulting in large amounts of data of which only a selection is shown in the following sections. For HoG, LSR and RID the features are extracted from the three sections presented in Figure
\ref{fig:sections} for all three color bands. We analyze the impact of the block size and scale parameters by using the block sizes 20, 40, and 60 in combination with the scales 50, 100, and150. The individual
impact of each of the two parameters is evaluated using a comparison between the baseline and the increased parameter, for example, a comparison between the results obtained from a block size of 20 compared to a block size of 60, with all other parameters constant. The evaluation is on the assumption that all parameters are independent.

Because every combination of image section, block size and scale will have three different results due to the RGB color bands, we refrain from visualizing all bands and pick the first band, which is red. The results seem to vary slightly between the red, green, and blue band, although it is not significant. Furthermore, we will select the most expressive feature for each extraction method to reduce the complexity of the visualization.

\subsection{Histogram of Oriented Gradients}

\begin{figure}
	\centering
	\begin{tabular}{cc}
		\subfloat{\includegraphics[width=7cm]{images/HoG/inc_bk/section_1_boxplot_hog_BK20_SC100_F2}}&
		\subfloat{\includegraphics[width=7cm]{images/HoG/inc_bk/section_1_boxplot_hog_BK60_SC100_F2}}\\
		\subfloat{\includegraphics[width=7cm]{images/HoG/inc_bk/section_1_kde_hog_BK20_SC100_F2}}&
		\subfloat{\includegraphics[width=7cm]{images/HoG/inc_bk/section_1_kde_hog_BK60_SC100_F2}}
	\end{tabular}
	\caption{The effect of increased block size on the HoG features. From left to
		right: a block size of 20 and 60 respectively.}
	\label{hog_inc_bk}
\end{figure}

For the Histogram of Oriented Gradients, the third feature is selected for visualization in this section, which is the variance. Figure \ref{hog_inc_bk} shows the effect of increased block size on the distribution of values in both classes. This example uses a constant scale of 100 pixels and a variable block size of 20 and 60 pixels. It seems that the block size, in this case, does not influence both distributions significantly. As a result, an increase in the size of a block within the tested range should have little to no influence on the predictive value of the feature.

\begin{figure}
	\centering
	\begin{tabular}{cc}
		\subfloat{\includegraphics[width=7cm]{images/HoG/inc_sc/section_1_boxplot_hog_BK20_SC50_F2}}&
		\subfloat{\includegraphics[width=7cm]{images/HoG/inc_sc/section_1_boxplot_hog_BK20_SC150_F2}}\\
		\subfloat{\includegraphics[width=7cm]{images/HoG/inc_sc/section_1_kde_hog_BK20_SC50_F2}}&
		\subfloat{\includegraphics[width=7cm]{images/HoG/inc_sc/section_1_kde_hog_BK20_SC150_F2}}\\
	\end{tabular}
	\caption{The effect of increased scale on the HoG features. From left to
		right: a scale of 50 and 150 respectively}
	\label{hog_inc_sc}
\end{figure}

Figure \ref{hog_inc_sc} illustrates the effect of increased scale on a constant block size. It seems to indicate that an increased scale leads to increased separation of the two distribution, thus increasing the predictive value of the feature. Scales larger than 150 do not seem to improve the performance; in contrast, the performance seems to decrease with scales larger than 150. In the paper of Graesser \textit{et al.} combined multiple scales to improve the detection of informal settlements. We will experiment with multiple scales and scale combinations in the results section.

\subsection{Line Support Region}

The Line Support Region feature visualized is the line mean entropy, which appeared to be the most expressive of the features. Furthermore, we showed the results from the first section because it produced the most distinct results from the three sections. The individual performances of the different sections will be evaluated in the results section. Similar to the analysis of HoG, the increase of the block size does not seem to influence the distribution of the two classes. The observed difference is as negligible as the comparison of the HoG in Figure \ref{hog_inc_bk}

\begin{figure}
	\centering
	\begin{tabular}{cc}
		\subfloat{\includegraphics[width=7cm]{images/LSR/inc_sc/section_1_boxplot_lsr_BK20_SC50_F1}}&
		\subfloat{\includegraphics[width=7cm]{images/LSR/inc_sc/section_1_boxplot_lsr_BK20_SC150_F1}}\\
		\subfloat{\includegraphics[width=7cm]{images/LSR/inc_sc/section_1_kde_lsr_BK20_SC50_F1}}&
		\subfloat{\includegraphics[width=7cm]{images/LSR/inc_sc/section_1_kde_lsr_BK20_SC150_F1}}
	\end{tabular}
	\caption{The effect of increased scale on the second SLR feature. From left to
		right: a scale of 50 and 150 respectively.}
	\label{lsr_inc_sc}
\end{figure}

The distributions for different scales of the LSR feature is visualized in Figure \ref{lsr_inc_sc}. Similar to the features from the Histogram of Oriented Gradients, it appears that an increase in scale seems to increase the difference between the distributions of the formal and informal classes, even though this is not as clear in the other two sections, it seems to indicate that increased scale might improve performance

\subsection{Road Intersection Density}

\begin{figure}
	\centering
	\begin{tabular}{ccc}
		\subfloat{\includegraphics[height=3.8cm]{images/RID/1}}&
		\subfloat{\includegraphics[height=3.8cm]{images/RID/2}}&
		\subfloat{\includegraphics[height=3.8cm]{images/RID/3}}\\
		\subfloat{\includegraphics[height=3.8cm]{images/RID/4}}&
		\subfloat{\includegraphics[height=3.8cm]{images/RID/5}}&
		\subfloat{\includegraphics[height=3.8cm]{images/RID/6}}\\
	\end{tabular}
	\caption{The distribution of values for the Road Intersection Density feature. Top: Scale 150; Bottom: Scale 50}
	\label{rid}
\end{figure}

The Road Intersection Density method generates a single feature produced by the Getis and Ord local G function. We have evaluated different block sizes and scales for RID although, in this case, the meaning of the scale and block size is slightly different than with HoG and LSR. In the calculation of the G function, we rasterize the locations of the intersection onto a grid, where the size of the blocks in the grid is the block size. The blocks in the grid contain a counter of the number of intersections that fell within that block. The G function is calculated over this grid where the scale is the neighborhood around a block, in the same manner as the HoG and LSR feature although the shape is a circle instead of a square.

Figure \ref{rid} shows the differences in distribution for the RID feature for scales 150 and 50. The low scale seems to create artifacts and does not have a distinctly different distribution. The higher scale, on the other hand, seems to have a difference, although not quite significant. Using RID together with the other two sections results in performance that is similar to the bottom right image in the figure. In the evaluation of RID, we have only used the parameters that worked well with the first section and applied this to the second and third section; this is likely cause for the low performance in the other sections. We would have to use three different sets of parameters to increase performance for the other two sections. It is clear that this approach does not seems scalable to large images where there is a significant variance in the types of roads.

\subsection{Conclusion}

With the right parameters, there is a clear distinction between the two classes for both the HoG and LSR features. RID, in contrast, does not seem to have a clear distinction, likely caused by a lack of specific parameters for section 2 and 3. For HoG and LSR, it seems that an increase in scale improves the distinctness of the two classes, although the increased scale is computationally expensive. On the three sections, the time to compute features at a high scale is measured in hours rather than minutes. Using features with high scales on large images is, therefore, discouraged. Besides, increasing the scale seems to give diminishing returns while taking significantly more time to compute. We use 150 as the maximum scale in the classification experiments to reduce the time needed to compute the features.
For HoG and LSR, the block size did not seem to improve or decrease performance. As a result, we would get the same performance with large block sizes as small block sizes using less computation. For now, the block size remains 20 pixels as the paper by Graesser \textit{et al.} describes. We will test our hypothesis on the effect of block size and scale during the classification experiments in the results section.


\documentclass[main.tex]{subfiles}
\begin{document}

\marginpar{Wednesday\\ 2020-12-2, \\ compiled \\ \today}

We will consider another kind of baryogenesis mechanism, connected to the reheating phase.

The baryon asymmetry generated by the out-of-equilibrium decays of \(X\) particles is of the order of
%
\begin{align}
B \approx \frac{\epsilon}{g_*}
\,,
\end{align}
%
where \(g_* \approx \num{e2}\) for the SM, or even \(g_* \approx \num{e3}\) BTSM.

A possible approach is to identify the \(X\) particle as the inflaton \(\varphi \).

We will use the usual notations: \(n_X\), \(n_B\), and \(\epsilon \) for the amount of \(CP\) violation.
The Boltzmann equation reads
%
\begin{align}
\dot{n}_X + 3H n_X = - \Gamma _X n_X
\,,
\end{align}
%
which is equivalent to
%
\begin{align}
\dot{\rho}_\varphi + 3 H \rho _\varphi = - \Gamma _\varphi \rho _\varphi
\,,
\end{align}
%
with the identification \(\rho _\varphi = m_X n_X\), where \(m_X\) is the effective of the inflaton field before the start of oscillations.

The right-hand side should read \(- \Gamma _X (n_X - n_X^{\text{eq}})\), but we approximated it; the BE for baryon asymmetry on the other hand is
%
\begin{align}
\dot{n}_B + 3 H n_B = +\epsilon \Gamma _X n_X
\,,
\end{align}
%
and we also need to write the Friedmann equation
%
\begin{align}
H^2 = \frac{8 \pi G}{3} \qty(\rho _\varphi +\rho _R)
\,.
\end{align}

The number density of \(X\) decays like
%
\begin{align}
n_X = n_{X, i} \qty(\frac{a_i}{a})^3 e^{-\Gamma _\varphi (t - t_i)}
\,,
\end{align}
%
where \(t_i\) is the ``oscillation time''.

As usual, we solve the BE for baryon asymmetry by writing the LHS as
%
\begin{align}
\frac{1}{a^3} \dv[]{}{t} (n_B a^3)
\,,
\end{align}
%
therefore
%
\begin{align}
n_B a^3 = \epsilon \Gamma _X \int_{t_i}^{t} n_{X, i} \qty(\frac{a_i}{a})^3 a^3e^{- \Gamma _\varphi (t - t_i)} \dd{t}
\,.
\end{align}

Doing the integral we get,
%
\begin{align}
n_B a^3 = \epsilon n_{X, i} a_i^3 \qty(1 - e^{- \Gamma _\varphi (t - t_i)})
\,.
\end{align}

At late times, we will get
%
\begin{align}
n_B a_f^3 = \epsilon n_{X, i} a_i^3
\,.
\end{align}

This has a very simple physical meaning: the left-hand side counts the number of net (signed) baryons in a comoving volume, while the right-hand side is \(\epsilon \) multiplied by the number of \(X\) particles in a comoving volume.

The final \(B\) is given by
%
\begin{align}
B_f = \eval{\frac{n_B}{s}}_{t_f} = \frac{n_B a_f^3}{s a_f^3} = \frac{\epsilon n_{X, i} a_i^3}{S_f}
\,,
\end{align}
%
but what is \(S_f\), the total entropy at a late time?
There are two ways to compute it: one is to solve the thermodynamic differential equations, and the alternative which we will adopt.

We write it as
%
\begin{align}
S_f =s_f a_f^3
\,,
\end{align}
%
and we know that in general for radiation domination
%
\begin{align}
s = \frac{2 \pi^2}{45} g_* T^3
\,,
\end{align}
%
and also
%
\begin{align}
\rho _R = \frac{\pi^2}{30} g_* T^{4} \implies T = g_*^{-1/4} \rho _R^{1/4} \qty(\frac{30}{\pi^2})^{1/4}
\,,
\end{align}
%
therefore
%
\begin{align}
S = s a^3 = g_*^{1/4} \rho _R^{3/4} a^3
\,.
\end{align}

Using this estimate, we can write
%
\begin{align}
B_f = \frac{\epsilon n_{X, i} a_i^3}{g_{*f}^{1/4} \rho _{R, f}^{3/4} a_f^3}
\,.
\end{align}

We can build a hierarchy of timescales, and consider a time \(t\) such that: \(t_i = t _{\text{oscill}} < t \leq t_\varphi = \Gamma _\varphi^{-1}\).
In this phase, \(\varphi \) dominates.
Then, with the assumption that there is complete conversion of the energy between \(\varphi \) and radiation, we can write
%
\begin{align}
\rho_{R, f}^{3/4} = \rho _{X, i}^{3/4} \qty(\frac{a_i}{a_f})^{9/4}
\,,
\end{align}
%
so, since \(a \propto t^{2/3}\) (matter domination):
%
\begin{align}
B_f &= \frac{\epsilon n_{X, i} a_i^{3/4}}{g_{*f}^{1/4} \rho _{X, i}^{3/4} a_f^{3/4}}  \\
&= \frac{\epsilon n_{X, i}}{g_*^{1/4} \rho _{X, i}^{3/4}} t_i^{1/2} \Gamma _\varphi^{1/2}
\,,
\end{align}
%
therefore in Friedmann's equation we have
%
\begin{align}
H_i^2 = \frac{8 \pi }{3} \frac{\rho_{X,i}}{m_P^2 } \approx \frac{1}{t_i^2}
\,,
\end{align}
%
therefore \(t_i \approx m_P \rho _{X, i}^{-1/2}\).
Plugging this in and using \(\rho _{X, i} = m_X n_{X, i}\), we get
%
\begin{align}
B_f &\approx \frac{\epsilon g_*^{-1/4} m_P^{1/2} \Gamma _\varphi^{1/2}}{m_X}  \\
&\approx \epsilon \frac{T_{RH}}{m_X}
\,,
\end{align}
%
where we recognize the approximate expression for the reheating temperature \eqref{eq:reheating-temperature}.

This expression should be compared with \(B_f = \epsilon / g_*\), the alternative. Which is larger?
For \(k \ll 1\), the mass of the bosons violating baryon number must be quite large: \(m_X \gtrsim \SI{e10}{GeV}\).
In order for these to be relativistic, the temperature must then also be very large: \(T \gtrsim m_X \sim \SI{e10}{GeV}\), which constrains the reheating temperature.

In this alternative scenario, with the inflaton violating baryon number conservation, we are allowed to violate this constraint.

Are \(B\), \(C\) and \(CP\) violated in the SM? Yes, in some nonperturbative electroweak processes, but not enough.

\section{Dark matter production}

We will start with the ``freeze-out'' mechanism.
From the latest Planck data, we know that at a \SI{68}{\percent} CL:
%
\begin{align}
\Omega_{DM} h^2 &= \num{.120(1)}   \\
h &= \num{.674(5)}
\,,
\end{align}
%
therefore \(\Omega_{DM} \approx \SI{26.4}{\percent}\).
Dark Matter as measured here must be a non-relativistic pressureless fluid.

Consider a massive particle \(\psi \), with mass \(m_\psi \).
Let us define
%
\begin{align}
y = n_\psi / s
\,.
\end{align}

As long as \(\Gamma _\psi < H\), the abundance of \(\psi \) ``freezes out'' when \(z = m_\psi / T \) reaches 1.

The Boltzmann equation for a process \(\psi \overline{\psi} \leftrightarrow X \overline{X}\) where \(X\) is in equilibrium with the plasma reads
%
\begin{align}
\dot{n}_\psi + 3 H n_\psi = - \expval{\sigma \abs{v}} \qty[n_\psi^2 - (n_\psi^{\text{eq}})^2]
\,.
\end{align}

Then,
%
\begin{align}
\dot{y} = - \expval{\sigma \abs{v}} s \qty[y^2 - y^2 _{\text{eq}}]
\,,
\end{align}
%
therefore, because
%
\begin{align}
\dv{z}{t} = zH
\,,
\end{align}
%
we can express the same with derivatives with respect to \(z\), denoted with primes:
%
\begin{align}
y' &= - \frac{\expval{\sigma \abs{v}} s}{z H} \qty[y^2 - y^2 _{\text{eq}}]  \\
&= - \frac{z\expval{\sigma \abs{v}} s}{H(z=1)} \qty[y^2 - y^2 _{\text{eq}}]
\,,
\end{align}
%
since \(z^2 H = H(z=1)\), which corresponds to the moment at which \(m_\psi = T\).

%
\begin{align} \label{eq:y-eq-freezeout}
y _{\text{eq}} = \frac{n_\psi^{\text{eq}}}{s} = \begin{cases}
    \num{.278} \frac{g _{\text{eff}}}{g_{*s}} & z \ll 1  \\
    \num{.145} \frac{g_\psi }{g_{*s}} z^{3/2} e^{-z} & z \gg 1
\,.
\end{cases}
\end{align}

In the \(z \gg 1\) limit inverse processes are suppressed.

The effective number of degrees of freedom is given by
%
\begin{align}
g _{\text{eff}} = \begin{cases}
    g_\psi & \text{boson} \\
    \frac{3}{4} g_\psi & \text{fermion}
\,.
\end{cases}
\end{align}

We can write the derivative of \(y\) with respect to \(z\) as
%
\begin{align}
y' &= - y^2 _{\text{eq}} \frac{\expval{\sigma \abs{v}} s}{zH} \qty[ \qty(\frac{y}{y _{\text{eq}}})^2 - 1] \\
\frac{z}{y _{\text{eq}}} y' &= - \underbrace{n^\psi  _{\text{eq}} \frac{\expval{\sigma \abs{v}} s}{H}}_{\Gamma_A / H} \qty[ \qty(\frac{y}{y _{\text{eq}}})^2 - 1]
\,.
\end{align}

If \(\Gamma _A / H <1\), then the right-hand side becomes small, so the left-hand side (which is basically \(\Delta y / y\) in a comoving volume) must also be small, meaning that \(y\) is roughly constant (frozen out): we have produced a relic abundance of \(\psi \) particles.

\subsection{Hot Dark Matter relics}

These particles are called ``hot'' since they \textbf{decouple while still being relativistic}. This is what we would expect for neutrinos, for example.

This particle decouples in the upper plateau of the curve \(y(z)\), so we expect little dependence for it on \(z_f\).

Here we will have
%
\begin{align}
y_\infty &= y(z \to \infty ) \approx y _{\text{eq}} (z_f)  \\
&\approx \frac{\zeta (3) g _{\text{eff}}T_f^3 / \pi^2}{(\pi^2 / 45) g_{*s } (z_f) T_f^3} \approx \num{.278} \frac{g _{\text{eff}}}{g_{*s} (z_f)}
\marginnote{Used \eqref{eq:y-eq-freezeout}.}
\,.
\end{align}

Now,
%
\begin{align}
\rho_{\psi 0 } = m_\psi n_{\psi 0 } = m_\psi y_\infty s_0
\,,
\end{align}
%
where \(s_0 \approx 1.8 g_{*s} (t_0 ) n_{\gamma 0 }\), where \(n_{\gamma 0} \approx \SI{422}{cm^{-3}}\).

Assuming that there is one relativistic neutrino species (this is conservative, we know there to be at least 2), we can write
%
\begin{align}
g_{*s} (t_0) \approx \num{2.63}
\,,
\end{align}
%
therefore \(s_0  \approx \num{2e3}\).
This means that
%
\begin{align}
\rho_{0, \text{crit}} = \SI{1.054e4}{eV cm^{-3}} h^2
\,,
\end{align}
%
and
%
\begin{align}
\Omega_{0 \psi } h^2 \approx \num{5.56e-2} \times \frac{m_\psi }{\SI{}{eV}} \frac{g _{\text{eff}}}{g_{*s} (z_f)}
\,.
\end{align}

In our case, \(g _{\text{eff}} = (3/4) \times 2\) since we have one neutrino species (with two polarizations), while \(g_{*s}(z_f) = \num{10.75}\), accounting for three neutrinos, electrons, positrons and photons.

From Planck data, including lensing of the CMB and BAO, we know that at a \SI{95}{\percent} CL:
%
\begin{align}
\sum _{\nu } m_\nu < \SI{.12}{eV}
\,.
\end{align}

They will surely be relativistic when they decouple at \(T_f \sim \SI{}{MeV}\).
This tells us that, if neutrinos are \(\psi \),
%
\begin{align}
\Omega_{0 \nu  \overline{\nu}} h^2 = \frac{\qty(\sum _{\nu } m_\nu  / \SI{}{eV})}{128} \lesssim \num{e-3}
\,.
\end{align}

Thus, we learn that neutrinos cannot constitute the majority of DM.
Despite this, they have a characteristic signature effect on structure formation.

The free-stream scale is given by
%
\begin{align}
\lambda_{\text{fs}}(t) = a(t) \int_0^{t} \frac{v_\nu }{a(\widetilde{t})} \dd{\widetilde{t}}
\,.
\end{align}

Below this scale (\(\lambda < \lambda _{\text{fs}}\)), neutrinos move too fast and damp the power spectrum of matter.

\end{document}