\chapter{Literature Study}
\label{cha:litreview}
This chapter contains a review of some of the most important techniques necessary to build an end-to-end speech recognition system. After looking at possible input feature representations, the most important machine learning tools will be explored. Finally sequence-to-sequence labeling methods such as connectionist temporal classification (CTC) and the listen attend and spell (LAS) architecture are covered.

\section{Preprocessing and feature extraction}
\subsubsection{Filter-Bank features}
\begin{figure}
\centering
\includegraphics[width=1.0\linewidth]{/png/timitInput}
\caption{Frequency Bank input computed from a sentence contained in the \textit{TIMIT} dataset. Time is shown on x and Frequency on the y-Axis.}
\label{fig:timitInput}
\end{figure}
Filter-banks are collections of filters. These filters can be spread out over audible frequencies\footnote{Approximately 16 to 16000 Hz.}. Filter-bank output is commonly used as input for speech analysis \cite{Huang2001}\cite{Chan2015}. The number of filter-banks depends on the required resolution, 32 is a common choice \cite{Juang1987}. The energy within the part of the signal spectrum described by all individual filters is measured. Figure~\ref{fig:timitInput} shows the resulting energy measurements using 23 filters, for a sentence recording contained in the \textit{TIMIT} data set.
The general argument for filter banks in speech recognition is that the cochlea, in the human ear, resembles a filter bank \cite[page 30]{Huang2001}. Humans do not perceive frequency linearly. Experimental evidence suggests, that our perception of is scaled according to the Mel-Scale \cite[page 34]{Huang2001}:
\begin{equation}
B(f) = 1125 ln(1 + f / 700).
\end{equation}
A normalized plot of this function is shown in figure~\ref{fig:mel} on the left.
According to the Mel-scale, humans are able to distinguish more lower frequencies than higher frequencies. In the plot the first four thousand Hertz occupy roughly eighty percent of the scale. The band from four thousand to eight thousand Hertz is left with only about twenty percent of the scale, even tough half of the considered frequencies are in this band.
\begin{figure}
\centering
\includestandalone[width=6cm]{tikz/melScale}
\includestandalone[width=6cm]{tikz/melBank}
\caption{The Mel-scale (blue) with Mel-Frequency Cepstrum Coefficients (red)  is shown on the left. Filterbanks with Mel-spaced filters are depicted on the right.}
\label{fig:mel}
\end{figure}
Mel spaced filter-banks are an attempt to include the human perception in speech recognition. The filter functions are defined by \cite[page 317]{Huang2001}:
\begin{align}
H_m &= 0 									   &\text{if}\;\; & k < f[m-1], \\
H_m &= \frac{k      - f[m-1] }{f[m] - f[m-1]}  &\text{if}\;\; & f[m-1] \leq k \leq f[m], \\
H_m &= \frac{f[m+1] - k      }{f[m + 1] - f[m]}&\text{if}\;\; & f[m] \leq k \leq f[m+1], \\
H_m &= 0									   &\text{if}\;\; & k > f[m+1].
\end{align}
In the equations above $H_m$ denotes the magnitude of filter $m$ with a total of $M$ filters. The frequency is denoted by $k$, the vector $f$ contains $M+2$ linearly spaced filter border values. These are the red stars on the left of  figure~\ref{fig:mel}. The right plot shows the triangular filter banks. These banks are spaced according to the same values. Roughly speaking using mel-filter banks means using a high filter resolution where human hearing is good and a low resolution where it is bad. \\
In addition to perceptually motivated instantaneous features, derivative information can be appended to each feature vector. These so called delta features add information
regarding the speed of change taking into account $2*N$ feature vectors. $N$ denotes the number of frames back and forward in time. The derivative information can be computed using \cite{Woodland2006}. The mel feature vectors are denoted by $\mathbf{f}$:
\begin{equation}
\mathbf{d}_t = \frac{\sum\limits_{n=1}^{N} n (\mathbf{f}_{t+n}- \mathbf{f}_{t-n})}{2 \sum\limits_{n=1}^{N} n^2}.
\end{equation}
The formula above is a central difference, because the value at time t is not taken into account, in contrast to forward or backward differences, where the value at t is part of the difference. A common choice for $N$ is two. In this case the formula above simplifies to:
\begin{equation}
\mathbf{d}_t = \frac{2\mathbf{f}_{t+2} + \mathbf{f}_{t+1} - \mathbf{f}_{t-1} - 2\mathbf{f}_{t-2}}{10}.
\end{equation}
In this case two future frames are used to compute the derivative at time $t$, which implies, that at computation time a full recording is available. Should that not be the case a backward difference scheme should be considered instead. To further augment the features second derivative or acceleration information can be added as well. These can be computed by simply applying the central difference to the deltas one more time. It is important to note that augmenting the features with derivative information significantly increases their dimension. If 40 mel-filters are used then the first and second derivatives will contain 40 entries each, therefore increasing the input dimension from $40$ to $120$.

Before using the features it is best practice to standardize the components of the input vectors. It is desirable to use features with an overall mean of zero and a standard deviation of 1 over the entire training set \cite[page 30]{Graves2008}. The feature normalization process starts by computing the mean,
\begin{equation}
\mathbf{m}_i = \frac{1}{\|S\|} \sum\limits_{\mathbf{x} \in S} \mathbf{x}_i,
\end{equation}
and standard deviation,
\begin{equation}
\sigma_i = \sqrt{\frac{1}{|S| - 1}\sum\limits_{x \in S}(\mathbf{m}_i - \mathbf{x}_i)^2}.
\end{equation}
Of a data set $S$ consisting of input $\mathbf{x}$ and target $\mathbf{t}$ pairs. Standardized input vectors can then be computed using:
\begin{equation}
\hat{\mathbf{x}}_i = \frac{\mathbf{x}_i - \mathbf{m}_i}{\sigma_i} \text{  for all } \mathbf{x}_i \in S.
\end{equation}
Input standardization is beneficial to network performance, because it places the input values in ranges, which standard activation functions such as sigmoids or hyperbolic tangents can handle better \cite[page 30]{Graves2008}.

Mel-Frequency banks are considered high level feature inputs. When the recognition system is found to work with these, features on a lower level or even raw data could be used as input. The idea behind doing less preprocessing, is that the network might be able to come up with something better on its own.

\section{Neural Networks}
Artificial neural networks where originally designed to model biological neurons. Even tough it is now known that artificial neural networks have little in common with their counterparts in biology, the models are still popular due to their useful pattern recognition properties \cite[page 13]{Graves2008}. All neural networks consist of elementary functions with weighted and biased inputs, the operations necessary to evaluate them form a computational graph. In a nutshell the idea is to choose the weights and biases such that the network output corresponds to patterns seen in the input data. The process of finding good weights is called training. Training in turn is done using gradient descent.

\subsection{Gradient descent}
The optimization process of neural networks works with a training data set $\{\{\mathbf{x}_1,\mathbf{t}_1\}$, $\dots ,$ $\{\mathbf{x}_p,\mathbf{t}_p\} \}$ \cite[page 156]{Rojas1996}. The elements of this set are the input- and output-patters $\mathbf{x}$ and $\mathbf{t}$ respectively. For each input vector the network output $\mathbf{o}$ is computed. Ideally the network output should be the same as the desired target $\mathbf{t}$ one for all data pairs. The difference between target and current outputs could be measured by the cost function \cite[page 156]{Rojas1996}:
\begin{align}
E = \frac{1}{2}\sum\limits_{i=1}^{p} \| \mathbf{o}_i - \mathbf{t}_i \|^2.
\end{align}
However generally the more complex cross-entropy is used as a cost function, because it leads to faster convergence and better results in practice \cite{Golik2013}. For classification tasks, cross entropy is defined as \cite[page 245]{Bishop1995}:
\begin{equation}
E = - \sum\limits_{i=1}^{p} {\mathbf{t}_i \ln(\mathbf{o}_i)} + (1 - \mathbf{t}_i)\ln(1 - \mathbf{o}_i).
\label{eq:corssEntropy}
\end{equation}
In order to understand why the cross entropy is a measure of output error, the two scalar cases $t=0$ and $t=1$ are considered. For $t = 0$ the cross entropy simplifies to $-\ln(1 - o)$. Looking at figure~\ref{fig:logPlot}, this expression turns out to be a good cost function, because for values close to the desired output $o = 0$, $-\ln(1 - o)$ will be close to zero. If the output moves away from the desired value zero, the cost grows asymptotically. If $t = 1$ is considered the cost function simplifies to $-\ln(o)$ which will be zero for the desired output and once more displays asymptotic growth for undesired values far away from one. If the target probability is somewhere between these two extreme cases the cost will be the sum of the two cases considered earlier. Cross entropy is statistically motivated, the output vectors $\mathbf{o}$ describe probability distributions over the possible labels, their elements must therefore be $\in (0,1)$ and sum op to one.
\begin{figure}
\centering
\includestandalone[width=6cm]{tikz/logPlot}
\caption{Plot of $y = -\ln(x)$ for $x \in (0,1)$.}
\label{fig:logPlot}
\end{figure}

During the training process a local minimum of the error function $E$ is sought. At this minimum the difference between the network output $\mathbf{o}$ and the target values $\mathbf{t}$ is small. This value might not be the smallest possible value, termination can happen at the global or a local minimum.
After the training process has completed the network is expected to identify similarities to data seen during the training process and produce a similar output.
In order to reach a local minimum, the gradient of the error function is needed. The key idea of gradient descent is to follow the negative gradient until a local minimum is reached.
Neural networks can be considered as large composite functions, which are made up of elementary operations. The evaluation of the network can be written as a graph. Computations are done at each node and information travels trough the network along directed edges from node to node. In order to create a computational graph for the training process each of the output units of the network under consideration are connected to a new node which computes $\frac{1}{2}(o_{ij} - t_{ij})^2$\cite[page 157]{Rojas1996} or the cross-entropy term of one data target pair. These new nodes in turn are connected to one more node, which sums up all error values and produces $E_i$. The process described above must be repeated for all training data pairs. One final node is added, which sums up all values $E_i$. Its output gives the value for the error function $E$ which is now in the form of a large graph.\\
Reverse mode algorithmic differentiation or back-propagation is an algorithm to compute the gradient of a graph consisting of basic elementary operations. As an example its operation is now illustrated using the function \cite[page 69]{Diehl2013}:
\begin{align}
f(w_1,w_2,w_3) = \sin(w_1 w_2) + \exp(w_1 w_2 w_3).
\label{eq:backFun}
\end{align}
This function written in terms of five elementary operations as \cite[page 70]{Diehl2013}:
\begin{align}
w_4 = w_1 w_2, \\
w_5 = sin(w_4),\\
w_6 = w_4 w_3, \\
w_7 = exp(w_6), \\
w_8 = w_5 + w_7, \\
o = w_8.
\end{align}
Computing the gradient means computing the partial derivatives of $f$ with respect to all inputs. In this case this means finding:
\begin{align}
\frac{\partial f(w_1,w_2,w_3)}{\partial w_1} = w_2 (\cos(w_1 w_2) + w_3 \exp(w_1 w_2 w_3)), \\
\frac{\partial f(w_1,w_2,w_3)}{\partial w_2} = w_1 (\cos(w_1 w_2) + w_3 \exp(w_1 w_2 w_3)), \\
\frac{\partial f(w_1,w_2,w_3)}{\partial w_3} = w_1  w_2 \exp(w_1 w_2 w_3).
\end{align}
Above the derivatives have been found by hand using the chain rule. Now these will be computed using back-propagation. Figure~\ref{fig:networkGraph} shows a graphical representation of equation~\ref{eq:backFun}. The partial derivatives needed for the backwards sweep can be found on the edges. \\
\begin{figure}
\centering
\includestandalone[width=12cm]{tikz/gd_exampleNet}
\caption{Example function network with partial derivatives.}
\label{fig:networkGraph}
\includestandalone[width=12cm]{tikz/gd_reverseSweep}
\caption{Reverse sweep.}
\label{fig:reverseSweep}
\end{figure}
The gradient is computed using a forward and backward sweep. During the forward sweep the inputs are fed into the network and the functions at each node are evaluated layer by layer, until the output at the last node is known. In figure~\ref{fig:networkGraph} this means computing $w_4$ to $w_8$. \\
After the forward sweep the gradient is found by going back trough the network from the output to each input node. Using a seed value of $1$ at the output node the lower unit values are computed by multiplying the associated partial derivative found on each edge. If a node has more then one incoming value, their sum is computed. The process is illustrated in figure~\ref{fig:reverseSweep}. At the roots of the tree the partial derivatives of the output with respect to each input can be found. Together these root values make up the gradient.
To be able to perform the first forward sweep the network weights are initialized at random. The training data pairs are known and can be added as constants to the graph.
If the weights of the network are stored in a weight vector $\mathbf{w}$ the value of the gradient after the $n$th update may be written as \cite[page 27]{Graves2008}:
\begin{equation}
\triangle \mathbf{w}(n) = -\alpha \frac{\partial f}{\partial \mathbf{w}(n)}.
\end{equation}
Where $\alpha \in [0,1]$ denotes the learning rate. Unfortunately simple gradient descent tends to get stuck in local optima. In order to increase the chance of gradient descent to escape from such a local minimum a momentum term can be added to the formulation \cite[page 267]{Bishop1995}\cite[page 27]{Graves2008}:
\begin{equation}
\triangle \mathbf{w}(n) = m\triangle \mathbf{w}(n-1) -\alpha \frac{\partial f}{\partial \mathbf{w}(n)}.
\end{equation}
Above $m \in [0,1]$ denotes the momentum. In order to understand the effect of the momentum term consider a weight space region of very low curvature, where it can be assumed, that the gradient stays constant. Using the momentum formula above yields \cite[page 267]{Bishop1995}:
\begin{align}
\triangle \mathbf{w}(n) &= -\alpha \frac{\partial f}{\partial \mathbf{w}} (1 + m + m^2 + \dots) \\
						&= -\frac{\alpha}{ 1 - m}\frac{\partial f}{\partial \mathbf{w}}.
\end{align}
Using the fact that the sum of the geometric series is known and $|m| < 1$ must hold. The momentum term therefore changes the effective learning rate from $\alpha$ to $\frac{\alpha}{ 1 - m}$ in regions of low curvature. Elsewhere the gradient is oscillatory, which leads to cancellation of successive contribution of the momentum term \cite[page 267]{Bishop1995}.

\subsection{Regularization}
The optimization process works on the training set, but later the trained network should be able to recognize patterns seen in the training data set in other inputs and produce similar outputs. In order to achieve this goal it is important not to memorize training data samples. Another problem with complex networks is that they are able to track noise in the training data.
Regularization techniques aim to counteract sample memorization and noise tracking in order to ensure that progress on the training data carries over to new data sets or in other words regularization aims to improve generalization.

\subsubsection{Early stopping}
The idea of early stopping is to observe the training and validation loss during training. As long as both are falling simultaneously, we assume that
the network will also generalize well to other data. This is a reasonable assumption, because the validation data set plays no part in computing the weight updates. When the training set loss continues fall, but the validation loss rises, the network is overfitting \cite[page 31]{Graves2012}. The training process should be stopped early in theses cases.
A test set should be used to guard against parameter choices that work well only on the training and validation set. Ideally the performance on the test set should only be evaluated, once the training process has finished.

Care must be taken, when dividing the entire data set into training, validation and test subsets. All three sets must be good representations of the overall data set. Differences for example regarding the noise level should be avoided. Furthermore the sizes of all sets must be chosen carefully. Using too few samples for validation and testing, makes it very hard to guard against over-fitting. If the training set becomes to small, overall performance might suffer. In practice a good trade off must be found experimentally.

When data collection is expensive cross validation should be used. Cross validation divides the merged training and validation data sets into $k$ subsets. During each iteration one of the $k$ subsets is used for validation and the remaining $k-1$ sets are used for training. Cross-validation has the advantage that each subset is used for training and validation, thus making more efficient use of the total data set. The test set is very important when working with cross validation, because the validation data is part of the training process. It should be sufficiently large to detect over-fitting.

\subsubsection{Input Noise}
Training with input noise means that fixed variance, zero mean Gaussian noise is added to the inputs during training \cite[page 32]{Graves2012}. The targets remain unchanged. The idea is to reuse the training data set more efficiently. When the artifical noise is similar to the noise already present in the data set, reusing the noisy samples is better than reusing the original samples. Input noise decreases the reliability of the inputs during training, it covers small variations caused by noise in the data, which the network could remember without it during training, if the noise was not present. Furthermore the artificial noise will help the network to deal with similar noise in the future, making the trained system more robust.
A problem with input noise is that it is difficult to determine how large the variance should be, one option is to manually tune the parameter using the validation performance as a metric \cite[page 32]{Graves2012}.

\subsubsection{Weight noise}
When using weight noise zero mean fixed variance Gaussian noise is added to the weights during training. The idea is to reduce the precision, with which the network weights can be described \cite[page 32]{Graves2012}. The reduced precision makes it impossible for the network to model small variations in the training samples, which are likely to be noise. Instead it will only be able to model larger changes and generalize better.
When using input noise the following algorithm should be used \cite[page 33]{Graves2012}:
\begin{lstlisting}[language=python]
while stopping condition is False:
    Shuffle training data pairs
    for each training sample pair:
        Add Gaussian noise to weights
        compute the gradient
        restore original weights
        update the original weights
\end{lstlisting}
However weight noise should be used with care, because it can lead to very slow convergence \cite[page 32]{Graves2012}.

\subsubsection{Regularized loss minimization}
Regularized loss minimization includes a model complexity penalty term in the cost function. Instead of minimizing the loss $E$ an augmented loss \cite[p. 338]{Bishop1995}\cite[p. 171]{Ben-david2014}:
\begin{equation}
\hat{E} = E + v\Omega.
\end{equation}
is used. $\Omega$ stands for a penalty term, which should grow, when the network is likely to over-fit the training data. $v \in (0,1)$ denotes the weight parameter, which determines how much emphasis the optimization algorithm should place on regularization. A straightforward choice for $\Omega$ is the $l_2$ norm of the model weights:
\begin{equation}
\Omega_{l2} = \|\mathbf{w}\|^2 = (\sqrt{ \sum w^2})^2.
\end{equation}
The weight norm serves as a measure for the model complexity. The idea of choosing the simplest possible model which can explain the data, often referred to as Occam's razor is expressed in the modified cost function $\hat{E}$, the result of the optimization process will be a compromise between fit to the training data and model complexity.

\subsubsection{Dropout}
\begin{figure}
\includegraphics[width=1.0\linewidth]{/png/dropout}
\caption{\enquote{Dropout Neural Net Model. Left: A standard neural net with 2 hidden layers. Right: An example of a thinned net produced by applying dropout to the network on the left. Crossed units have been dropped.} \cite{Srivastava2014}, image and caption taken from the same source.}
\label{fig:Dropout}
\end{figure}
Like input and weight noise, dropout is part of the group of regularization techniques, that artificially corrupt training data to improve prediction performance \cite{Wager2013}. When applying dropout between two layers, some units and their connections are zeroed out at random during training. The process is illustrated in figure~\ref{fig:Dropout}.
One interpretation relates dropout networks to committee networks. Committees consist of several independently trained networks, each of which makes predictions on its own. The final prediction is then produced by using the mean output of the committee members or by feeding all outputs into another neural net, which decides on the final output.
When training large deep networks however the committee approach becomes infeasible due to its large memory requirements. Dropout networks are related to network committees \cite{Srivastava2014}, because when some units are dropped the remaining units will be forced to learn to work together without the dropped neurons. Just like committee networks, dropout networks contain multiple mechanisms to recognize the same pattern, and thus gain some of the prediction robustness that committee networks are known to have.
Another motivation for dropout comes from the theory of sexual selection \cite{Srivastava2014}. The theory assumes that the superiority of sexual reproduction is caused by the fact that it removes strong interconnections between genes. After all a specific gene might not be there a generation from now. To function over many generations, genes must therefore develop redundant ways that work with many other different genes.
Following this interpretation dropout forces network units to do the same thing, thereby improving performance.

\subsection{Deep Neural Networks}
Deep learning deals with the design of computational nets, consisting of multiple layers, which learn representations of data. Deep networks are often used when abstract
patterns must be found. Simple networks are often unable to cope with the problem complexity, and fail to recognize the underlying structure. Human speech features such complex patterns, therefore deep networks must be considered. Increasing the network depth leads to many additional weights, for which a suitable value must be found.
In addition to the added memory load caused by the extra weights, the optimization process must look trough large data sets until an optimum is reached.
\subsubsection{Stochastic gradient descent}
When training deep  networks on very large data sets, working with the full data set to compute the current gradient becomes very inefficient. It is common practice to use stochastic gradient descent instead of the classic algorithm. The idea is to select small subsets of the full data set. These so called mini-batches are then used to compute the outputs and errors, as well as the gradients for these smaller examples. Instead of working with the gradient of the union of these mini-batches the average gradient is computed and used to update the network weights. This approach dramatically reduces the memory requirements of the training process, which makes it possible to work with more sophisticated network designs.

\subsection{Recurrent Neural Networks}
\begin{figure}
\centering
\includestandalone[width=7cm]{tikz/recurrentCell}
\caption{Visualization of a single recurrent cell.}
\label{fig:singleRecCell}
\includestandalone[width=9cm]{tikz/recNet}
\caption{Rolled (left) and unrolled (right) recurrent neural net with two units.}
\label{fig:unrolledNet}
\end{figure}
Good speech processing tools take context into account. To spell the letters which make up a word, one must know what the previous letter was, in order to make the right decision.
Feed-forward neural nets do not possess memory. These networks make decisions, starting from zero every time. In oder to fix this a cell state variable can be introduced. A simple recurrent cell where the current state is set to the previous output is shown in figure~\ref{fig:singleRecCell}. Another way to depict the same architecture is shown in Figure~\ref{fig:unrolledNet}, here the cell state is not labeled explicitly, the single cell on the left is just a simplified version of figure~\ref{fig:singleRecCell}. If the network is unrolled in time the flow of the state becomes apparent, which is done in figure~\ref{fig:unrolledNet} on the right.
The unrolled form shows a direct dependency of the output at time $t$ on the previous output at $t-1$, which in turn depends on the previous outputs. This causes $y_t$ and $y_{t-1}$ to change together. In other words: The introduction of recurrent connections leads to correlation of the two outputs.

\subsubsection{The exploding and vanishing gradient problem}
Even tough past information is available in theory, RNNs have only limited access to contextual information in practice \cite[page 1]{Graves2008}. Due to problems with gradient descent on correlated data, the back-propagated derivative can sometimes become weaker and weaker until it ultimately vanishes \cite{Hochreiter1998}. Another problem is that sometimes classical recurrent neural nets produce a gradient that blows up \cite{Pascanu2012}. The exploding gradients can be fixed by clipping, but vanishing gradients require more sophisticated treatment \cite{Bengio1993}.

\subsubsection{Long short-term memory}
Initially researchers tried to solve the vanishing gradient problem by making changes to the training algorithm using simulated annealing, time delays or compression \cite[page 32]{Graves2008}. However a good solution to the problem turned out to be changing the RNN cell architecture. Long short-term memory (LSTM) cells as proposed in \cite{Hochreiter1995} are more complex network units. LSTMs are differentiable versions of computer memory chips \cite[page 37]{Graves2012}. Memory chips generally have read, write, and erase ports which can be set in order to allow the chip's state to be read, modified or emptied. In the LSTM case the input, output, and forget gates have the same function \cite[page 33]{Graves2008}. Throughout the literature these gates are generally denoted as $\mathbf{i}, \mathbf{f}$ and $\mathbf{o}$. The content of the memory or state is written as $\mathbf{c}$, but will referred to as $\mathbf{s}$ in this thesis in order to avoid confusion with the LAS context vector. The output is denoted by $\mathbf{h}$, while a $t$ subscript denotes the time step.
LSTM uses the differentiable equation system \cite[page 5]{Graves2013}\footnote{ Various versions of LSTM cells exist. This one is commonly referred to as the \textquotedblleft peephole\textquotedblright $ \: $ variant. }:
\begin{align}
\mathbf{i_t} &= \sigma (\mathbf{W}_{ix} \mathbf{x}_t + \mathbf{W}_{ih} \mathbf{h}_{t-1} + \mathbf{W}_{is} \mathbf{s}_{t-1} +\mathbf{b}_i), \\
\mathbf{f}_t &= \sigma (\mathbf{W}_{fx} \mathbf{x}_t + \mathbf{W}_{fh} \mathbf{h}_{t-1} + \mathbf{W}_{fs} \mathbf{s}_{t-1} +\mathbf{b}_f), \\
\mathbf{s}_t &= \mathbf{f}_t \mathbf{s}_{t-1} + \mathbf{i}_t \tanh( \mathbf{W}_{sx} \mathbf{x}_t + \mathbf{W}_{sh} \mathbf{h}_{t-1} + \mathbf{b}_c ), \\
\mathbf{o}_t &= \sigma (\mathbf{W}_{ox} \mathbf{x}_t + \mathbf{W}_{oh} \mathbf{h}_{t-1} + \mathbf{W}_{os} \mathbf{s}_t + \mathbf{b}_o ), \\
\mathbf{h}_t &= \mathbf{o}_t \tanh(\mathbf{s}_t). \\
\end{align}
From the definition of the matrix product follows that,
\begin{equation}
\mathbf{A}\mathbf{x}_1 + \mathbf{B}\mathbf{x}_2
=
\begin{bmatrix} \mathbf{A} & \mathbf{B} \end{bmatrix} \cdot
\begin{bmatrix} \mathbf{x}_1 \\ \mathbf{x}_2 \end{bmatrix}.
\end{equation}
Which this relation in mind the equations above can be rewritten, by creating column wise concatenated weight matrices for every neuron gate $W_i$, $W_f$, $W_o$, as well as for the state $W_s$. These matrices can then be multiplied by a row wise concatenated vector $[\mathbf{x}_t \; \mathbf{h}_{t-1} \; \mathbf{s}_t]^T$, which leads to the slightly simplified system of equations below:
\begin{align}
\mathbf{i}_t &= \sigma (\mathbf{W}_i [\mathbf{x}_t \; \mathbf{h}_{t-1} \; \mathbf{s}_{t-1}]^T + \mathbf{b}_i), \\
\mathbf{f}_t &= \sigma (\mathbf{W}_f [\mathbf{x}_t \; \mathbf{h}_{t-1} \; \mathbf{s}_{t-1}]^T + \mathbf{b}_f), \\
\mathbf{s}_t &= \mathbf{f}_t \mathbf{s}_{t-1} + \mathbf{i}_t \tanh( \mathbf{W}_s [\mathbf{x}_t \; \mathbf{h}_{t-1}]^T + \mathbf{b}_s ), \\
\mathbf{o}_t &= \sigma (\mathbf{W}_o [\mathbf{x}_t \; \mathbf{h}_{t-1} \; \mathbf{s}_t]^T + \mathbf{b}_o ), \\
\mathbf{h}_t &= \mathbf{o}_t \tanh(\mathbf{s}_t).
\end{align}
\begin{figure}
\includestandalone[width=\textwidth]{tikz/lstm}
\caption{Visualization of the LSTM architecture.}
\label{fig:lstm}
\end{figure}
This system of equations is visualized in figure~\ref{fig:lstm}. Just like diagram~\ref{fig:singleRecCell} this depiction is read from bottom to top. The most important part is the line from $\mathbf{s}_{t-1}$ to $\mathbf{s}_{t}$ \cite{Colah2015}. It records operations on the cell state $\mathbf{s}_t$. The cell state contains information from the past which helps the block make decisions regarding the current output $\mathbf{h}_t$. The sigmoid functions $\sigma(\cdot)$ are applied element wise on the input vectors and produce outputs between zero and one. In the case of the forget gate output $\mathbf{f}_t$ these values $\in (0,1)$ serve as a measure of how much of the past state the cell would like to remember. One means keep this variable and zero throw it away \cite{Colah2015}.
The following task is to determine what should be added to the memory. This information can be found in the input gate result $\mathbf{i}_t$. $\mathbf{i}_t$ is multiplied element wise with the candidate values $\mathbf{\bar{s}}_t$. These are computed by a hyperbolic tangent neuron.  The $\tanh(\cdot)$ function makes sure all vector elements are between $-1$ and $1$. The neuron computing the candidate state values $\mathbf{\bar{s}}_t$ looks at input data and the past outputs. Both are labeled $\mathbf{w}$ in figure~\ref{fig:lstm}, $\mathbf{w}$ contains all information that could possibly be included in the new state. Finally the weighted candidate values are added to what was previously stored. This operation leads to the updated memory state $\mathbf{s}_t$.
Last but not least the new output value has to be computed, which will be a filtered version of the cell state. How much, if anything, of each state variable will be send outside is decided by the output gate. It's output $\mathbf{o}_t$ is multiplied with a rescaled version of the cell state. The rescaling is done using another hyperbolic tangent, which again sets all values between minus one and one. The product of this rescaled state and the weights found in $\mathbf{o}_t$ then yields the new output $\mathbf{h}_t$.

At this point it is interesting to note that the graphs considered in this thesis will almost exclusively consist
out the exponential, logarithmic, sigmoid and hyperbolic tangent functions. These functions are intimately related. It is well known that the exponential and logarithmic functions are connected by:
\begin{equation}
\exp(\ln(x)) = x \;\;\; \text{if} \; x > 0.
\end{equation}
The sigmoid and hyperbolic tangent functions in turn consist out of exponentials, and are related trough \cite[page 15]{Graves2008}:
\begin{equation}
\tanh(x) = 2\sigma(2x) - 1.
\end{equation}
The equation above can be proven by using the definitions of the sigmoidal $\sigma = \frac{1}{1 + \exp(-x)}$, and hyperbolic tangent $\tanh = \frac{\exp(2x) - 1}{\exp(2x) + 1}$:
\begin{align}
\tanh(x) &= \frac{\exp(2x) - 1}{\exp(2x) + 1}, \\
&= \frac{2\exp(2x) - (\exp(2x) + 1)}{\exp(2x) + 1}, \\
&= \frac{2 \exp(2x)}{\exp(2x) + 1} - \frac{\exp(2x) + 1}{\exp(2x) + 1}, \\
&= \frac{2 \exp(2x)}{\exp(2x) + 1} - 1, \\
&= \frac{2}{\frac{\exp(2x) + 1}{\exp(2x)}} - 1, \\
&= \frac{2}{1 + \frac{1}{\exp(2x)}} - 1, \\
&= 2\frac{1}{1 + \exp(-2x)} - 1 = 2\sigma(2x) - 1.
\end{align}
These two non-linear functions are thus very similar, in the LSTM-cell their different function value ranges decide how they are used. Sigmoids determine how much of something should be used, stored, or deleted, because it's output values range between zero and one, while computations done on the cell state use the hyperbolic tangent, due to it's wider range of function values.

\subsubsection{Bidirectional Long Short Term Memory}
With the advent of LSTMs deep recurrent networks became feasible in speech recognition \cite{Graves2013b}. RNNs are always deep in time, because their hidden state depends on past inputs. To enable abstraction their structure must also be deep in space. A bidirectional LSTM layer is shown in figure~\ref{fig:blstm}. It is important to note, that several LSTM layers are often followed by a single linear layer, as is the case in \cite{Graves2013b}:
\begin{align}
\overrightarrow{\mathbf{h}}_t &= \text{LSTM}(\mathbf{W}_{\overrightarrow{\mathbf{h}}_t} [\mathbf{x}_t \; \mathbf{h}_{t-1}]^T + \mathbf{b}_{\overrightarrow{\mathbf{h}}_t}),
\\
\overleftarrow{\mathbf{h}}_t &= \text{LSTM}(\mathbf{W}_{\overleftarrow{\mathbf{h}}_t} [\mathbf{x}_t \; \mathbf{h}_{t+1}]^T + \mathbf{b}_{\overleftarrow{\mathbf{h}}_t}),
\\
\mathbf{y}_t &= \mathbf{W}_{y} [\overrightarrow{\mathbf{h}}_t \; \overleftarrow{\mathbf{h}}_t]^T + \mathbf{b}_y.
\end{align}
\begin{figure}
\centering
\includestandalone[height=7 cm]{tikz/blstm2}
\caption{A bidirectional Long short term memory layer, according to \cite{Graves2013b} }
\label{fig:blstm}
\end{figure}
If stacked on top of each other, these bidirectional LSTM layers form a deep recurrent network. Defining $\mathbf{h}^0 = \mathbf{x}$, $\mathbf{h}^N = \mathbf{y}$ looking at time from $t = 1$ to $T$ and taking $N$ layers leads to:
\begin{align}
\overrightarrow{\mathbf{h}}_t^n &= \text{LSTM}(\mathbf{W}_{\overrightarrow{\mathbf{h}}_t}^n [\mathbf{h}_t^{n-1} \; \mathbf{h}_{t-1}^n]^T + \mathbf{b}_{\overrightarrow{\mathbf{h}}_t}^n),
\\
\overleftarrow{\mathbf{h}}_t^n &= \text{LSTM}(\mathbf{W}_{\overleftarrow{\mathbf{h}}_t}^n [\mathbf{h}_t^{n-1} \; \mathbf{h}_{t+1}^n]^T + \mathbf{b}_{\overleftarrow{\mathbf{h}}_t}^n),
\\
\mathbf{h}_t^N &= \mathbf{W}_{y}^N [\overrightarrow{\mathbf{h}}_t \; \overleftarrow{\mathbf{h}}_t]^T + \mathbf{b}_y^N.
\end{align}
In this setting each LSTM cell has access to information from before or after it. For this to work the speech sequence, which is analyzed has to be recorded completely. In this case future information is available and should be used for recognition purposes.

\section{Connectionist temporal classification}
A bidirectional LSTM layer establishes access to past and future input information, but the alignment of the speech inputs and it's transcription remains unknown. Connectionist temporal classification is an output layer, which in tandem with the BLSTM layer enables the network to work around this problem, while at the same time allowing the gradient to propagate trough it. The resulting network can be trained end-to-end.

CTC consists of a softmax layer with one output per alphabet element plus one for blank our no label. The idea is to allow the network to output a label at any input time step, and post process these outputs to get the overall sequence of labels correct, while ignoring the timing of each label.
More formally $L$ denotes the used alphabet and $L'$ = $L \cup \oslash $ the alphabet with the empty label. The network output vectors are denoted by $\mathbf{y}^t$ and will have length $|L'|$. Given the training set S and the input sequence $\mathbf{x}$ the probability for observing a sequence $\pi$ consisting of elements from $L'$ with length T is given by \cite[page 56]{Graves2008}:
\begin{equation}
p(\pi| \mathbf{x}, S) = \Pi_{t=1}^T y^t_{\pi_t}.
\end{equation}
In a first step all a mapping $\mathcal{B}$ is introduced, which simplifies the allowed paths trough the label probabilities, by removing duplicates. For example $\mathcal{B}(x \oslash xy \oslash) = \mathcal{B}(\oslash x x \oslash xyy) = xxy$. In other words a new label is only produced if the output label changed in the path. The probability of a labeling $l$ can be computed by summing up the probabilities of all paths, which map to the same labeling \cite[page 57]{Graves2008}:
\begin{equation}
p(\mathbf{l|\mathbf{x}}) = \sum\limits_{\pi \in \mathcal{B}^{-1}(\mathbf{l})}p(\pi|\mathbf{x}).
\label{eq:CTCPaths}
\end{equation}
Generally speaking this collapsing process of different paths is what makes it possible to use CTC with unaligned data.

Equation~\ref{eq:CTCPaths} must be evaluated efficiently, which can be done by using a forward-backward algorithm. Such algorithms compute forward and backward pass variables $\alpha_t(s)$ and $\beta_t(s)$. The two variables break the probability of correctly predicting the sequence into a prefix $\alpha$ and suffix $\beta$ part.
During training their product at any time step $t$ and sequence position $s$ yields the probability sum of all label paths found in the network output $\mathbf{y}$ and mapped onto $\mathbf{t}$ by $\mathcal{B}$.  The probability of predicting the targets ($\mathbf{l} = \mathbf{t}$) given the inputs is then given by:
\begin{equation}
p(\mathbf{t}|\mathbf{x}) = \sum\limits_{s=1}^{| \mathbf{t}' |} \alpha_t(s) \beta_t(s).
\end{equation}
Which is running tough the $| \mathbf{t}' |$ elements of the target sequence with added blanks at the start, end and between all labels. The sum checks all possible alignments, which could follow from the mapped outputs. Adding up the negative logarithm of the probabilities above leads to the CTC cost function. Decoding can be done using simlpe greedy decoding or a more complex beam search variant called prefix search. More details can be found in  \cite[chapter 7]{Graves2008}, \cite[chapter 7]{Graves2012} and \cite{Graves2006}.

CTC takes into account only the features and (depending on how the beam search is done) ignores the label it previously produced. Additionally the number of network outputs is coupled to the input frames. This shortcoming led to the development of so called transducers. These models decouple the decoding time from the input time and use past labels in an attempt to improve performance.

\section{Listen, Attend and Spell \cite{Chan2015}}
The Listen Attend and Spell architecture is a deep neural network, designed to jointly learn to align and transcribe speech data. In contrast to CTC it belongs to the transducer family of speech recognition algorithms. All transducers have in common that they consist of an encoder and a decoder, with decoupled time steps. The idea also appeared in the automatic translation literature \cite{Bahdanau2015}. Working on the source language text, an annotator or encoder produces high level annotations. These annotations are then given to the translator or decoder, which using its own output time step works with the annotations to compute a context, which it then uses to produce a translation.
In the speech application the encoder is referred to as the listener. The decoder as the speller. Combined these two form the las-network. The listener is a pyramidal recurrent neural net. It accepts filter bank spectra $\mathbf{x}_n$ as inputs and produces compressed high level output features $\mathbf{h}_m$. Compression reduces the computational load during feature processing later.
The speller in turn accepts the features as input and outputs distributions over character sequences $\mathbf{y}_p$. Due to the uncoupling of the input and decoding time step, it requires a state, as well as an attention mechanism. The state provides a memory of what happened in the decoder in the past. The attention function determines, which listener-features are relevant at a given decoding time step. Combining attention and state information makes it possible to label the input data. An overview of the las-achrcitecture is given in figure~\ref*{fig:las}.

\subsection{The Listener}
\label{subsec:Listener}
The listener, consists of Long Short Term Memory blocks. These blocks are arranged in layers. The inputs are first fed into a recurrent bidirectional layer (BLSTM). This choice gives the system access to future data, therefore only fully recorded data can be analyzed. This system is restricted to applications, which do not require transcriptions in real time or where its acceptable to wait with decoding until a sentence has been recorded completely. When going up in figure~\ref*{fig:las}, pyramidal layers (pBLSTM) follow the initial BLSTM layer. The pyramidal structure concatenates the hidden values computed previously, such that their time dimension is halved:
\begin{equation}
\mathbf{h}_{t}^n = \text{BLSTM}(\mathbf{h}_{t-1}^{n}, [\mathbf{h}_{2t}^{n-1}, \mathbf{h}_{2t+1}^{n-1}]).
\end{equation}
Technically instead of two, three or more previously computed feature vectors could be concatenated, which increases the compression factor per pyramidal layer.
This operation reduces the length $U$ of the high level features $\mathbf{H}$. Without this compression the following attend and spell operation has a hard time extracting the relevant information, because a longer time span has to be considered to decode a single character. Additionally the compression reduces the problem complexity, which speeds up the training process.

\subsection{Attend and spell}
\label{subsec:AttendAndSpell}
The speller takes the features and produces a distribution over Latin character sequences as output. The computation of this output involves the context vector $\mathbf{c}_i$, the decoder state $\mathbf{s}_i$, the features $\mathbf{H}$ and the previous output $\mathbf{y}_i$. The index $i$ denotes decoding time, $i-1$ is used to refer to results from the last time step. The last decoding step $I$, at which the system terminates is a learned quantity. While the last input step $U$, depends on the input features, lower case $u$ denotes the input step.\\
During operation the attend and spell functions keep track of previous output labels and previously important features, which are contained in the context. This information is stored in the state $\mathbf{s}_i$. To function the network must determine, which part of the computed features in $\mathbf{H}$ are relevant at any given decoding time step $i$. The context vector $\mathbf{c}_i$ contains a linear combination of relevant features, weighted according to their importance. The AttentionContext function determines these weights based on the state. Finally the speller function finds a probability over possible labels using the relevant features in the context and the state. The computing steps are therefore \cite[page 4]{Chan2015}:
\begin{align}
 \mathbf{s}_i &= \text{RNN}(\mathbf{s}_{i-1}, \mathbf{y}_{i-1}, \mathbf{c}_{i-1}), \\
 \mathbf{c}_i &= \text{AttentionContext}(\mathbf{s}_i,\mathbf{H}), \\
  P(\mathbf{y}_i|\mathbf{x}, \mathbf{y}_{<i}) &= \text{CharacterDistribution}(\mathbf{s}_i,\textbf{c}_i).
\end{align}
The state follows from a recurrent multilayer LSTM. In contrast to the listener the speller is causal, meaning that it makes decisions based only on information computed during previous decoding steps. LSTMs are necessary here, because past states must be remembered. The attention mechanism, called AttentionContext above, computes a new context vector once every time step.
This computation starts with the determination of the scalar energy $e_{i,u}$, which will be used as weight for its corresponding feature vector  $\mathbf{h}_u$. The computation starts with two feedforward neural networks or multilayer perceptrons (MLP), $\phi$ and $\psi$ \cite[page 5]{Chan2015}:
\begin{align}
e_{i,u} = \phi(\mathbf{s}_i)^T \psi(\mathbf{h_u}), \\
\alpha_{i,u} = \frac{ \exp(e_{i,u})}{ \sum\limits_{u} \exp(e_{i,u})}, \\
\label{eq:alphas}
\mathbf{c}_i = \sum\limits_{u} \alpha_{i,u} \mathbf{h}_u.
\end{align}
$\alpha$ is produced by running $\mathbf{e}$ trough a softmax function, which scales $\mathbf{e}$ such that all elements are within $(0,1)$ and add up to one. These scaled weights, can then be used to form the context vector $\mathbf{c}_i$. When the training process converges the $\alpha_i$s typically follow a distribution with sharp edges \cite[page 5]{Chan2015}. Thus it is justified to think of the alphas as a sliding window. This window contains only the currently relevant parts of the condensed input data set.

\begin{figure}
\includestandalone[width=0.9\textwidth]{tikz/lasArcBottomUp}
\caption{The LAS architecture \cite[page 3]{Chan2015}. BLSTM blocks are shown in red. LSTM blocks in blue and attention nets in green.}
\label{fig:las}
\end{figure}

\subsection{Training}
For end-to-end speech recognition all networks must be trained jointly. The objective is to maximize the logarithmic probability:
\begin{equation}
\max\limits_\theta \sum\limits_{i} \log P(y_i | \mathbf{x}, y_{<i};\theta).
\end{equation}
Here $y_i$ denotes the current output distribution, $x$ the input, $\theta$ the various network parameters and finally $y_{<i}$ the ground truth, which is the known true desired output. In practice the objective is minimized by working with a cross entropy loss function.
Using the known output during training creates a situation, where the past outputs are always right. In practice however the situation will be different, as the network is going to make mistakes. As it is desired to create a robust model it is necessary to sometimes include the character distribution generated by the networks being trained.
Which leads to the objective \cite[page 5]{Chan2015}:
\begin{align}
\hat{\mathbf{y}}_{i} = \text{CharacterDistribution}(s_i,\textbf{c}_i), \\
\max_{\theta} \sum\limits_{i} \log P(y_i|\mathbf{x}, \hat{y}_{<i};\theta).
\end{align}
The novelty in comparison to the previous expression is that $\hat{y}_{<i}$ is sometimes taken from the past network outputs instead of the ground truth. An idea which Chan et al. found in \cite{Bengio2015}.

\subsection{Decoding with beam search}
\label{sec:beamsearch}
In order to generate a readable text, it is necessary to choose characters from the generated character distributions. One way to do this is to simply pick the most likely letter from each distribution. This approach is called greedy decoding. It ignores the possibility of generating better results by also considering less likely options. It is reasonable to expect better results, when considering more than just the most likely label, because the attend and spell decoder takes past labels into account. Consequently a broader search trough the most likely options should be performed. Unfortunately memory limitations make it impossible to search trough all possible combinations. Therefore only the $n$ most likely options are explored and the rest is ignored. This approach is referred to as beam search. Taking into account the most likely options for each label produces a tree of possible transcriptions. The different routes along this tree are called hypotheses. A score for each hypothesis can be computed, by multiplication of the probability values the las-network assigned to each branch along its path. In order to account for different hypotheses lengths the total probability must be divided by the hypothesis length. In beam search only the most likely hypotheses are kept, the beam width determines their number. Using only las-probabilities is the equivalent of taking only acoustic data and their labels into account. In general text data is far more abundantly available then speech data. To take advantage of these large text corpora a language model trained on these can be used to make a more informed decision when choosing a beam-hypothesis. A selection can then be made according to \cite[page 6]{Chan2015}:
\begin{align}
s(\mathbf{y}|\mathbf{x}) = \frac{\log P(\mathbf{y}|\mathbf{x})}{ |\mathbf{y}|_c} + \lambda \log P_{LM}(\mathbf{y}).
\end{align}
Here $P_{LM}$ denotes the weight the language model assigns to each hypothesis. And $\lambda$ is a weight factor, which determines the language model importance. The formula above describes beam selection using a language model to re-score the attend and spell probabilities with a language model.

\subsection{Levenshtein distance}
\begin{table}
\centering
\begin{tabular}{ |l|c|c|c|c|r| } \hline
             & $\epsilon$ & h & u & i & s \\ \hline
$\epsilon$   & 0          & 1 & 2 & 3 & 4 \\ \hline
         h   & 1          & 0 & 1 & 2 & 3 \\ \hline
         o   & 2          & 1 & 1 & 2 & 3 \\ \hline
         u   & 3          & 2 & 1 & 2 & 3 \\ \hline
         s   & 4          & 3 & 2 & 2 & 2 \\ \hline
         e   & 5          & 4 & 3 & 3 & \textbf{3} \\ \hline
\end{tabular}
\caption{An example of the edit distance matrix for the dutch word huis and
its english translation house. The edit distance can be found in the cell
in the lower right of the table.}
\label{tab:editDistExample}
\end{table}
A metric is required to measure the quality of label sequences. The Levenshtein or edit distance is such a metric. It can be thought of as the minimum number insertions, deletions or substitutions required to transform one sequence into another.
An example computation for the two words huis and house is shown in table~\ref{tab:editDistExample}. The table cells count replacements, deletions or insertions. At position $i,j$ in the table the first sequence is taken into account until element $i$, likewise the second sequence is considered up to position $j$. In order to compute the entries in the table the first row and column of the distance matrix $D$ are initialized to:
\begin{equation}
D[0,j] = j \text{ and } D[i,0] = 0.
\end{equation}
These entries are known beforehand, because if one index is kept at zero,
no elements of the corresponding sequence are considered and $i$ or $j$ insertions will always be necessary.
If $i \neq 0$ and $j \neq 0$ the following update rule is used \cite[slide 10]{Langmead2016}:
\begin{align}
D[i,j] = \min\left\{
  \begin{array}{@{}ll@{}}
    D[i-1, j] + 1, & \text{deletion} \\
    D[i,j-1] + 1, & \text{insertion} \\
    D[i-1,j-1] + \delta(x[i-1], y[j-1]) & \text{match or replacement},
  \end{array}\right. \\
\text{with } \delta(a,b) = 0 \text{ if } a=b \text{ else } 0.
\end{align}
Based in these rules the table~\ref{tab:editDistExample}, has been computed. Intuitively the distance three makes sense, because to go from huis to house, an o has to be inserted, the i replaced with an s and finally and an extra e has to be added, which amounts to three edits.

\section{Tensor-flow}
In this section is devoted to the toolbox, which was used to implement the Listen Attend and spell, architecture. According to the Tensor-flow authors \cite{Agarwal2015}:

\enquote{TensorFlow is an interface for expressing machine learning algorithms, and an implementation for executing such algorithms}.

In tensorflow a directed graph is constructed based on the input operations which define a computational model. The graph represents how data flows trough the model. Each node of the graph has zero or more inputs and zero or more outputs. The nodes represent operations. Tensors flow along the edges of the graph.
Graphs are executed in sessions. When running a graph the desired output variables or operations must be specified. Tensorflow then checks if all inputs required to compute the requested output values or perform the desired operation are present. Afterwards it evaluates only the operations including the requested node to find the desired output value or complete the requested operation.
Generally tensors do not outlive a single execution of the graph. Variables however are persistent. The model parameters must be stored in variables. If an operation computing gradient values was evaluated previously, these values can then be applied to weight variables to improve the model parameters over multiple graph executions.


\documentclass[a4paper]{book}
\usepackage[margin=1cm]{geometry}
\usepackage{rail}

\begin{document}
%\begin{rail}
%    decl : 'def' identifier '=' ( expression + ';' )
%    | 'type' identifier '=' type
%    ;
%\end{rail}
%\begin{rail}
%    number
%    : '0' (('b'|'B') (('0'|'1') +) | (('0-7') +) | ('x'|'X') ('0-9,a-f,A-F' +));
%\end{rail}

\chapter{Basic Tokens}
\section{Numbers}
\begin{rail}
    binary : '0' ('b'|'B') (('0'|'1') +);
    octal : '0' (('0-7') +);
    hex : '0' ('x'|'X') ('0-9,a-f,A-F' +);
    genericint : '1-9' ('0-9' +);
    signedint : genericint ('i'|'I');
    unsignedint : genericint ('u'|'U');
    specifiedint : signedint genericint;
    specifieduint : unsignedint genericint;
    double : '0-9' '.' '0-9';
    float : double ('f'|'F');
    expdouble : (double | int) ('e'|'E') (()|'+'|'-') genericint;
    expfloat : expdouble ('f'|'F');
    integer : binary | octal | hex | genericint | signedint | unsignedint | specifiedint | specifieduint;
    decimal : double | float | expdouble | expfloat;
    number : integer | decimal;
\end{rail}
\section{Nouns}
\begin{rail}
    nouninitial : 'a-z'|'A-Z';
    noun : nouninitial ((()|nouninitial|'0-9') +);
\end{rail}
%\begin{tikzpicture}
%
%    \node (none) at (0,8) {none};
%
%    \node (noun) at (4,8) {noun};
%    \node (zero) at (4,6) {number (zero)};
%    \node (number) at (4,0) {number};
%
%    \node (binnumi) at (8,9) {binary*};
%    \node (binnum) at (12,9) {binary};
%    \node (octnum) at (8,7) {octal};
%    \node (hexnumi) at (8,5) {hexidecimal*};
%    \node (hexnum) at (12,5) {hexidecimal};
%    \node (double) at (8,3) {double};
%    \node (float) at (12,1) {float};
%
%    \draw[->] (none.east) -- node[below] {0} (zero.west);
%    \draw[->] (none.east) -- node[below] {1-9} (number.west);
%
%    \draw[->] (zero.east) -- node[below] {b,B} (binnumi.west);
%    \draw[->] (zero.east) -- node[below] {0-7} (octnum.west);
%    \draw[->] (zero.east) -- node[below] {x,X} (hexnumi.west);
%    \draw[->] (zero.east) -- node[below] {.} (double.west);
%    \draw[->] (number.east) -- node[below] {.} (double.west);
%    \draw[->] (binnumi.east) -- node[above] {0,1} (binnum.west);
%    \draw[->] (hexnumi.east) -- node[above] {0-9,a-f,A-F} (hexnum.west);
%    \draw (binnum) to[out=45,in=135] node[above] {0,1} (binnum);
%    \draw (octnum) to[out=45,in=135] node[above] {0-7} (octnum);
%    \draw (hexnum) to[out=45,in=135] node[above] {0-9,a-f,A-F} (hexnum);
%    \draw (number) to[out=45,in=135] node[above] {0-9} (number);
%    \draw (double) to[out=45,in=135] node[above] {0-9} (double);
%    \draw[->] (double.east) -- node[below] {f,F} (float.west);
%
%\end{tikzpicture}
\end{document}


\section{Introduction}

\begin{frame}
\frametitle{Introduction}
\begin{itemize}
\item Linear mixed models are frequently used to analyze repeated measures data.
\vfill\item Usual assumption: both random effect and error term follow normal
distributions.
\vfill\item Some proposals have been made in the literature for relaxing the assumption of normality. \pause
\vfill\item Frequently, classes of LMMs consider that the error terms are conditionally independent.
\vfill\item However, in longitudinal studies, repeated measures are collected over time and hence the error term tends to be serially correlated.
\end{itemize}
\end{frame}

\begin{frame}{Motivation: sleepstudy data}
\protect\hypertarget{motivation-sleepstudy-data}{}

\begin{itemize}
\tightlist
\item
  The average reaction time per day for subjects was evaluated by
  Gregory et al.~(2003) in a sleep deprivation study.
\vfill\item
  On day 0 the subjects had their normal amount of sleep and starting
  that night they were restricted to 3 hours of sleep per night for 9
  days, and the reaction time basead on a series of tests was measured
  on each day for each subject.
\vfill\item
  The data are available at the R package \emph{lme4}.
\end{itemize}

\end{frame}

\begin{frame}{Motivation: sleepstudy data}
\protect\hypertarget{motivation-sleepstudy-data-1}{}

\begin{center}\includegraphics[width=0.85\linewidth]{codes_files/figure-beamer/data-1} \end{center}

\end{frame}

\begin{frame}[fragile]{Fitting a LMM to the sleepstudy dataset}
\protect\hypertarget{fitting-a-lmm-to-the-sleepstudy-dataset}{}

\vspace{-.2cm}
\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{library}\NormalTok{(nlme)}
\NormalTok{fitlme <-}\StringTok{ }\KeywordTok{lme}\NormalTok{(Reaction}\OperatorTok{~}\NormalTok{Days,}\DataTypeTok{data=}\NormalTok{sleepstudy,}
            \DataTypeTok{random=}\OperatorTok{~}\NormalTok{Days}\OperatorTok{|}\NormalTok{Subject)}
\end{Highlighting}
\end{Shaded}

%Plotting the random effects estimates obtained from the LMM:
\begin{center}\includegraphics[width=0.7\linewidth]{codes_files/figure-beamer/fit1plot-1} \end{center}

\end{frame}

\section{Model formulation}
\begin{frame}\frametitle{Scale mixture of skew-normal (SMSN) distributions}
The $\textrm{SN}_p(\bmu,\bSigma,\mathbf{\blambda})$ distribution (Azzalini and Valle, 1996) can be defined from:
\begin{equation*}
    f(\mathbf{y})=2\phi_p(\mathbf{y};\bmu, \bSigma)
\Phi(A),\,\,\,\,\,\mathbf{y}\in \mathbb{R}^p,
\end{equation*}
where $A = \blambda^{\top}
\bSigma^{-1/2}\left(\mathbf{y}-\bmu\right)$.
\pause \vspace{.4cm}

The $\textrm{SMSN}_p(\bmu,\bSigma,\mathbf{\blambda};H)$ class of distributions can then be defined through the following pdf:
\begin{equation*}
f(\mathbf{y})=2\int_{0}^{\infty}\phi_p(\mathbf{y};\bmu,\kappa(u) \bSigma)\
\Phi(\kappa(u)^{-1/2} A)d H(u;\bnu),
\end{equation*}
$\mathbf{y}\in \mathbb{R}^p$, for some positive weight function $\kappa(u)$.
\end{frame}
%------------------------------------------------
\begin{frame}\frametitle{SMSN - special cases}
\begin{itemize}\setlength{\itemsep}{\fill}
    \item When $\blambda=0$, we get the $\textrm{SMN}_p(\bmu,\bSigma ;H)$;
    \item When  $\kappa(u)=u^{-1}$, we get the skew-normal/independent (SNI) class of distributions:
    \begin{itemize}\setlength{\itemsep}{.3cm}
        \item[-] By taking $U\sim \textrm{Gamma}(\nu/2,\nu/2)$, the $\ST_p(\bmu,\bSigma,\blambda,\nu)$ can be derived;
        \item[-] By taking $U\sim \textrm{Beta}(\nu,1)$, the $\SSL_p(\bmu,\bSigma,\blambda,\nu)$ can be derived;
        \item[-] By taking $U$ as a discrete random variable with probability function given by  $h(u|\bnu)=\nu_1 \mathbb{I}_{\{\nu_2\}}(u)+(1-\nu_1) \mathbb{I}_{\{1\}}(u)$, the $\SCN_p(\bmu,\bSigma,\blambda,\nu,\rho)$ can be derived, where $\nu_1,\nu_2\in (0,1)$.
    \end{itemize}
\end{itemize}
\end{frame}
%------------------------------------------------
\begin{frame}\frametitle{SMSN linear mixed model (SMSN-LMM)}
In general, a normal linear mixed effects model is defined as
\begin{equation}
\textbf{Y}_i=\textbf{X}_i\bbeta+\textbf{Z}_i\textbf{b}_i+\bepsilon_i,\,\,\,\,\ii\,,
\label{modeleq}
\end{equation}
where
\begin{itemize}
    \item $\textbf{X}_i$ of dimension $n_i\times l$ is the design matrix corresponding to the fixed effects,
    \item $\bbeta$ of dimension $l\times 1$ is a vector of population-averaged regression coefficients called fixed effects, \item $\textbf{Z}_i$ of dimension $n_i\times q$  is the design matrix corresponding to the $q\times 1$ random effects vector $\textbf{b}_i$, and
    \item $\bepsilon_i$ of dimension $n_i\times 1$ is the vector of random errors.
\end{itemize}
\end{frame}
%------------------------------------------------
\begin{frame}\frametitle{SMSN-LMM}
Usual assumptions:
\begin{itemize}
    \item $\textbf{b}_i\iid N_q(\mathbf{0},\textbf{D})$ $\perp$ $\bepsilon_i\ind N_{n_i}(\mathbf{0},\bSigma_i)$,
    \item The $q \times q$  random effects covariance matrix $\textbf{D}$ may be unstructured or structured,
    \item the $n_i \times n_i$ error covariance matrix $\bSigma_i$ is commonly written as $\sigma_e^2 \R_i$, where $\R_i$ can be a known matrix or a structured matrix depending on a vector of parameter, say $\bphi$.
\end{itemize}
\end{frame}
%------------------------------------------------
\begin{frame}\frametitle{SMSN-LMM}
Likewise, the SMSN-LMM can be defined by considering
\begin{equation}\label{modSnmis}
    \left( \begin{array}{c}
         \textbf{b}_i \\
         \bepsilon_i
    \end{array}\right) \ind \textrm{SMSN}_{q+n_i}
    \left( \left(\begin{array}{c}
         c\bDelta \\
         \mathbf{0}
    \end{array} \right), \left(\begin{array}{cc}
         \mathbf{D} & \mathbf{0} \\
         \mathbf{0} & \bSigma_i
    \end{array} \right), \left(\begin{array}{c}
         \blambda \\
         \mathbf{0}
    \end{array} \right); H \right),
\end{equation}
$\ii$, where
\begin{itemize}
    \item $c= c(\bnu)=-\sqrt{\frac{2}{\pi}}k_1$, $k_1=\E\{U^{-1/2}\}$, $\bDelta = \textbf{D}^{1/2} \bdelta$,
    \item  $\textbf{D} = \textbf{D}(\balpha)$ depends on unknown and reduced parameter vector $\balpha$, and
    \item $\bSigma_i = \sigma_e^2 \R_i$, with $\R_i = \R_i(\bphi)$, $\bphi=(\phi_1,\hdots,\phi_p)^\top$, being a structured matrix.
\end{itemize}
\end{frame}

%------------------------------------------------
\begin{frame}\frametitle{Within-subject dependence structures}
\begin{enumerate}
    \item Conditional independence (CI): $\textbf{R}_i = \mathbf{I}_{n_i}$.
    \item Autoregressive dependence of order $p$ (AR($p$)): \begin{equation*}
\mathbf{R}_{i}=\mathbf{R}_{i}(\bphi)=\frac{1}{1-\phi_1\rho_1-\ldots -\phi_p\rho_p}[\rho_{|r-s|}],
\end{equation*}
where $\rho_1,\hdots,\rho_p$ are the theoretical autocorrelations of the process and functions of $\bphi = (\phi_1,\ldots,\phi_p)^\top$, and they satisfy the Yule-Walker equations.
\item Damped exponential correlation (DEC):
\begin{equation*}
    \textbf{R}_i = \textbf{R}_i(\phi_1,\phi_2,\textbf{t}_i) = \left[ \phi_1^{|t_{ij}-t_{ik}|^{\phi_2}}\right], \,\,\,\, 0<\phi_1<1,\,\, \phi_2>0.
\end{equation*}
\end{enumerate}
\end{frame}
%------------------------------------------------
\begin{frame}\frametitle{Important remark}
The SMSN-LMM can be written hierarchically as follows:
\vspace{-.2cm}\begin{eqnarray*}
    \Y_i|\bi,U_i=u_i &\ind& \textrm{N}_{n_i}\left(\textbf{X}_i\bbeta+\textbf{Z}_i\textbf{b}_i,
    u_i^{-1}\sigma_e^2\textbf{R}_i\right),\\
    \bi|T_i=t_i,U_i=u_i &\ind& \textrm{N}_{q}\left(\bDelta t_i,
    u_i^{-1}\bGamma\right),\\
    T_i|U_i=u_i &\ind& \textrm{TN}\left(c,u_i^{-1}, (c,\infty)\right), \,\textrm{ and }\\
    U_i &\ind&  \textrm{H}(\cdot;\bnu),
    \end{eqnarray*}

\vspace{-.2cm}which are all independent, and TN$(\mu,\tau,(a,b))$ denotes the univariate normal distribution (N$(\mu,\tau)$) truncated on the interval $(a,b)$.

This representation is useful for the implementation of an EM-type algorithm, for details see Schumacher et al. (2020).
\end{frame}
%------------------------------------------------

\begin{frame}{Tools for model evaluation}
\protect\hypertarget{tools-for-model-evaluation}{}
\begin{itemize}\sepvfill
    \item Likelihood ratio (LR) test;
    \item Mahalanobis distance (known distribution);
    \item Healy-type plot;
    \item Empirical autocorrelation function (ACF) for standardized marginal residuals, which at lag $l$ can be defined as
\vspace{-.2cm}\begin{equation*}
    \widehat{\rho}(l) = \frac{\sum_{i=1}^n \sum_{\{ (j,k) |t_k-t_j=l\}} r_{it_j} r_{it_k} / N(l)}{
    \sum_{i=1}^n \sum_{j=1}^{n_i} r_{it_j}^2/N(0) },
\end{equation*}

\vspace{-.4cm}
where $\textbf{r}_i = \widehat{\bUpsilon}_i^{-1/2}\left(\y_i - \textbf{X}_i\widehat{\bbeta} \right)$ is the standardized marginal residual vector for subject $i$, with $\bUpsilon_i=\text{Var}(\Y_i)$, and $N(\cdot)$ is the number of pairs used in the respective numerator summation.
\end{itemize}

\end{frame}

\section{The skewlmm package}
\begin{frame}[fragile]{The \texttt{R} package \emph{skewlmm}}
\protect\hypertarget{the-package}{}

\begin{itemize}\sepvfill
\item
  The package \emph{skewlmm} implemets an EM-type algorithm in
  \texttt{R} using S3 class, containing methods for estimating and
  predicting the SM(S)N-LMM.
\item
  It has an user-friendly interface with generic \texttt{R} functions
  \texttt{print}, \texttt{summary}, \texttt{plot}, \texttt{fitted},
  \texttt{residuals} and \texttt{predict} implemented.
\item
  The main functions in the package are \texttt{smsn.lmm()} and
  \texttt{smn.lmm()}, which fit a SMSN-LMM and a SMN-LMM, respectively.
\end{itemize}

\end{frame}

\begin{frame}[fragile]{The \texttt{R} package \emph{skewlmm}}
\protect\hypertarget{the-package-1}{}

The basic syntax of these functions is as follows:

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{smsn.lmm}\NormalTok{(data, formFixed, groupVar, formRandom, }
\NormalTok{         depStruct, distr, ...)}
\KeywordTok{smn.lmm}\NormalTok{(data, formFixed, groupVar, formRandom, }
\NormalTok{        depStruct, distr, ...)}
\end{Highlighting}
\end{Shaded}

where

\begin{itemize}
\item
  \texttt{data}: A data frame containing all the variables to be used in
  the model.
\item
  \texttt{formFixed}: A two-sided linear formula object describing the
  fixed effects part of the model.
\item
  \texttt{groupVar}: A character containing the name of the variable
  which represents the subjects or groups in data.
\end{itemize}

\end{frame}

\begin{frame}[fragile]

\begin{itemize}
\item
  \texttt{formRandom}: A one-sided linear formula object describing the
  random effects part of the model.
\item
  \texttt{depStruct}: A character indicating which dependence structure
  should be used.
\item
  \texttt{distr}: A character indicating which distribution should be
  used.\pause
\end{itemize}

Some other useful arguments:

\begin{itemize}
\item
  \texttt{timeVar}: A character containing the name of the variable
  which represents the time in data.
\item
  \texttt{pAR}: The order of the autoregressive process that should be
  used (if depStruct=``ARp'').
\item
  \texttt{initialValues}: A named list containing initial parameter
  values.
\end{itemize}

\end{frame}

\begin{frame}[fragile]

The functions return an object of the class \texttt{SMSN} and
\texttt{SMN}, respectively, containing a list of elements, and the
following methods/ functions are available to these classes:

\begin{multicols}{2}
\begin{itemize}
\item
  \texttt{print}
\item
  \texttt{summary}
\item
  \texttt{fitted}
\item
  \texttt{plot}
\item
  \texttt{predict}
\item
  \texttt{residuals}
\item
  \texttt{ranef}
\item
  \texttt{acfresid}
\item
  \texttt{healy.plot}
\item
  \texttt{lr.test}
\item
  \texttt{mahalDist}
\end{itemize}
\end{multicols}

\end{frame}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{frame}[fragile]{Example: sleepstudy data}
\protect\hypertarget{example-sleepstudy-data}{}

\scriptsize

\begin{Shaded}
\begin{Highlighting}[]
\CommentTok{#fitting a (CI)-SMN-LMM, default is distr='norm'}
\NormalTok{fit0<-}\KeywordTok{smn.lmm}\NormalTok{(}\DataTypeTok{data=}\NormalTok{sleepstudy,}\DataTypeTok{formFixed =}\NormalTok{ Reaction}\OperatorTok{~}\NormalTok{Days,}
              \DataTypeTok{formRandom =} \OperatorTok{~}\NormalTok{Days,}\DataTypeTok{groupVar =} \StringTok{"Subject"}\NormalTok{,}\DataTypeTok{quiet =}\NormalTok{ T)}
\end{Highlighting}
\end{Shaded}

\begin{Shaded}
\begin{Highlighting}[]
\CommentTok{#fitting a (CI)-SMSN-LMM, default is distr='sn'}
\NormalTok{fitskew0<-}\KeywordTok{smsn.lmm}\NormalTok{(}\DataTypeTok{data=}\NormalTok{sleepstudy,}\DataTypeTok{formFixed =}\NormalTok{ Reaction}\OperatorTok{~}\NormalTok{Days,}
              \DataTypeTok{formRandom =} \OperatorTok{~}\NormalTok{Days,}\DataTypeTok{groupVar =} \StringTok{"Subject"}\NormalTok{,}\DataTypeTok{quiet =}\NormalTok{ T)}
\end{Highlighting}
\end{Shaded}

\end{frame}

\begin{frame}[fragile]
\tiny

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{fit0}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## Linear mixed models with distribution norm and dependency structure CI
## Call:
## smn.lmm(data = sleepstudy, formFixed = Reaction ~ Days, groupVar = "Subject",
##     formRandom = ~Days, quiet = T)
##
## Fixed:Reaction ~ Days
## Random:~Days
##   Estimated variance (D):
##             (Intercept)     Days
## (Intercept)   560.68002 11.91827
## Days           11.91827 32.52525
##
## Estimated parameters:
##      (Intercept)    Days   sigma2  Dsqrt1 Dsqrt2 Dsqrt3
##         251.4051 10.4673 655.4400 23.6752 0.4059 5.6886
## s.e.      7.2257  1.5541  31.3784 10.0856 1.7514 1.4270
##
## Model selection criteria:
##   logLik     AIC      BIC
##  -875.97 1763.94 1783.098
##
## Number of observations: 180
## Number of groups: 18
\end{verbatim}

\end{frame}

\begin{frame}[fragile]

\tiny

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{fitskew0}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## Linear mixed models with distribution sn and dependency structure CI
## Call:
## smsn.lmm(data = sleepstudy, formFixed = Reaction ~ Days, groupVar = "Subject",
##     formRandom = ~Days, quiet = T)
##
## Fixed:Reaction ~ Days
## Random:~Days
##   Estimated variance (D):
##             (Intercept)     Days
## (Intercept)  1432.65701 35.22718
## Days           35.22718 33.76855
##
## Estimated parameters:
##      (Intercept)    Days   sigma2  Dsqrt1 Dsqrt2 Dsqrt3 lambda1 lambda2
##         251.4073 10.4657 652.6598 37.8418 0.8080 5.7546 -4.2917 -0.2477
## s.e.     11.6328  2.1610  32.7878 28.9642 5.2063 1.6311      NA      NA
##
## Model selection criteria:
##    logLik      AIC      BIC
##  -875.354 1766.709 1792.253
##
## Number of observations: 180
## Number of groups: 18
\end{verbatim}

\end{frame}

\begin{frame}[fragile]{Changing the distribution}
\protect\hypertarget{changing-the-distribution}{}

\scriptsize

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{fit1<-}\KeywordTok{smn.lmm}\NormalTok{(}\DataTypeTok{data=}\NormalTok{sleepstudy,}\DataTypeTok{formFixed =}\NormalTok{ Reaction}\OperatorTok{~}\NormalTok{Days,}\DataTypeTok{distr =} \StringTok{'t'}\NormalTok{,}
              \DataTypeTok{formRandom =} \OperatorTok{~}\NormalTok{Days,}\DataTypeTok{groupVar =} \StringTok{"Subject"}\NormalTok{,}\DataTypeTok{quiet =}\NormalTok{ T)}
\end{Highlighting}
\end{Shaded}

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{fitskew1<-}\KeywordTok{smsn.lmm}\NormalTok{(}\DataTypeTok{data=}\NormalTok{sleepstudy,}\DataTypeTok{formFixed =}\NormalTok{ Reaction}\OperatorTok{~}\NormalTok{Days,}\DataTypeTok{distr =} \StringTok{'st'}\NormalTok{,}
                  \DataTypeTok{formRandom =} \OperatorTok{~}\NormalTok{Days,}\DataTypeTok{groupVar =} \StringTok{"Subject"}\NormalTok{,}\DataTypeTok{quiet =}\NormalTok{ T)}
\end{Highlighting}
\end{Shaded}

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{fit2<-}\KeywordTok{smn.lmm}\NormalTok{(}\DataTypeTok{data=}\NormalTok{sleepstudy,}\DataTypeTok{formFixed =}\NormalTok{ Reaction}\OperatorTok{~}\NormalTok{Days,}\DataTypeTok{distr =} \StringTok{'sl'}\NormalTok{,}
              \DataTypeTok{formRandom =} \OperatorTok{~}\NormalTok{Days,}\DataTypeTok{groupVar =} \StringTok{"Subject"}\NormalTok{,}\DataTypeTok{quiet =}\NormalTok{ T)}
\end{Highlighting}
\end{Shaded}

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{fitskew2<-}\KeywordTok{smsn.lmm}\NormalTok{(}\DataTypeTok{data=}\NormalTok{sleepstudy,}\DataTypeTok{formFixed =}\NormalTok{ Reaction}\OperatorTok{~}\NormalTok{Days,}\DataTypeTok{distr =} \StringTok{'ssl'}\NormalTok{,}
                  \DataTypeTok{formRandom =} \OperatorTok{~}\NormalTok{Days,}\DataTypeTok{groupVar =} \StringTok{"Subject"}\NormalTok{,}\DataTypeTok{quiet =}\NormalTok{ T)}
\end{Highlighting}
\end{Shaded}

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{fit3<-}\KeywordTok{smn.lmm}\NormalTok{(}\DataTypeTok{data=}\NormalTok{sleepstudy,}\DataTypeTok{formFixed =}\NormalTok{ Reaction}\OperatorTok{~}\NormalTok{Days,}\DataTypeTok{distr =} \StringTok{'cn'}\NormalTok{,}
              \DataTypeTok{formRandom =} \OperatorTok{~}\NormalTok{Days,}\DataTypeTok{groupVar =} \StringTok{"Subject"}\NormalTok{,}\DataTypeTok{quiet =}\NormalTok{ T)}
\end{Highlighting}
\end{Shaded}

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{fitskew3<-}\KeywordTok{smsn.lmm}\NormalTok{(}\DataTypeTok{data=}\NormalTok{sleepstudy,}\DataTypeTok{formFixed =}\NormalTok{ Reaction}\OperatorTok{~}\NormalTok{Days,}\DataTypeTok{distr =} \StringTok{'scn'}\NormalTok{,}
                  \DataTypeTok{formRandom =} \OperatorTok{~}\NormalTok{Days,}\DataTypeTok{groupVar =} \StringTok{"Subject"}\NormalTok{,}\DataTypeTok{quiet =}\NormalTok{ T)}
\end{Highlighting}
\end{Shaded}

\end{frame}

\begin{frame}{Comparing the fitted models}
\protect\hypertarget{comparing-the-fitted-models}{}

\begin{longtable}[]{@{}crr@{}}
\toprule
distr & AIC & BIC\tabularnewline
\midrule
\endhead
norm & 1763.9 & 1783.1\tabularnewline
sn & 1766.7 & 1792.3\tabularnewline
t & 1737.5 & 1759.8\tabularnewline
st & 1739.8 & 1768.6\tabularnewline
sl & 1736.2 & \bf 1758.5\tabularnewline
ssl & 1738.4 & 1767.2\tabularnewline
cn & \bf 1733.7 & 1759.3\tabularnewline
scn & 1735.9 & 1767.8\tabularnewline
\bottomrule
\end{longtable}

\end{frame}

\begin{frame}[fragile]{Assessing the goodness of fit using a Healy-type
plot}
\protect\hypertarget{assessing-the-goodness-of-fit-using-a-healy-type-plot}{}

\scriptsize

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{grid.arrange}\NormalTok{(}\KeywordTok{healy.plot}\NormalTok{(fit0),}\KeywordTok{healy.plot}\NormalTok{(fit1),}
             \KeywordTok{healy.plot}\NormalTok{(fit2),}\KeywordTok{healy.plot}\NormalTok{(fit3))}
\end{Highlighting}
\end{Shaded}

\begin{center}\includegraphics[width=0.75\linewidth]{codes_files/figure-beamer/healy1-1} \end{center}

\end{frame}

\begin{frame}[fragile]{LR test for \(H_0: \lambda=0\) in the CN-LMM}
\protect\hypertarget{lr-test-for-h_0-lambda0-in-the-cn-lmm}{}

\small

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{lr.test}\NormalTok{(fitskew3,fit3)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## Model selection criteria:
##            logLik      AIC      BIC
## fitskew3 -857.926 1735.851 1767.781
## fit3     -858.861 1733.722 1759.266
##
##     Likelihood-ratio Test
##
## chi-square statistics =  1.871248
## df =  2
## p-value =  0.392341
##
## The null hypothesis that both models represent the
## data equally well is not rejected at level  0.05
\end{verbatim}

\end{frame}

\begin{frame}[fragile]{Computing the ACF of the residuals from CN-LMM}
\protect\hypertarget{computing-the-acf-of-the-residuals-from-cn-lmm}{}

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{acfresid}\NormalTok{(fit3)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##    lag         ACF n.used
## 1    0  1.00000000    180
## 2    1  0.19793878    162
## 3    2 -0.07329748    144
## 4    3 -0.21972223    126
## 5    4 -0.06519806    108
## 6    5 -0.13723727     90
## 7    6 -0.27485055     72
## 8    7 -0.08778128     54
## 9    8  0.19912222     36
## 10   9  0.58707921     18
\end{verbatim}

\end{frame}

\begin{frame}[fragile]{Plotting the ACF (CI-CN-LMM)}
\protect\hypertarget{plotting-the-acf-ci-cn-lmm}{}

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{plot}\NormalTok{(}\KeywordTok{acfresid}\NormalTok{(fit3,}\DataTypeTok{calcCI =}\NormalTok{ T,}\DataTypeTok{maxLag =} \DecValTok{6}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

\begin{center}\includegraphics[width=0.7\linewidth]{codes_files/figure-beamer/fit6-1} \end{center}

\end{frame}

\begin{frame}[fragile]{Fitting an AR(p)-SMN-LMM}
\protect\hypertarget{fitting-an-arp-smn-lmm}{}

\scriptsize

\begin{Shaded}
\begin{Highlighting}[]
\CommentTok{#sl}
\NormalTok{fit2ar1<-}\KeywordTok{smn.lmm}\NormalTok{(}\DataTypeTok{data=}\NormalTok{sleepstudy,}\DataTypeTok{formFixed =}\NormalTok{ Reaction}\OperatorTok{~}\NormalTok{Days,}
                 \DataTypeTok{formRandom =} \OperatorTok{~}\NormalTok{Days,}\DataTypeTok{groupVar =} \StringTok{"Subject"}\NormalTok{,}
                 \DataTypeTok{distr=}\StringTok{"sl"}\NormalTok{,}\DataTypeTok{depStruct =} \StringTok{"ARp"}\NormalTok{,}\DataTypeTok{pAR=}\DecValTok{1}\NormalTok{,}\DataTypeTok{quiet=}\NormalTok{T)}
\end{Highlighting}
\end{Shaded}

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{fit2ar2<-}\KeywordTok{smn.lmm}\NormalTok{(}\DataTypeTok{data=}\NormalTok{sleepstudy,}\DataTypeTok{formFixed =}\NormalTok{ Reaction}\OperatorTok{~}\NormalTok{Days,}
                 \DataTypeTok{formRandom =} \OperatorTok{~}\NormalTok{Days,}\DataTypeTok{groupVar =} \StringTok{"Subject"}\NormalTok{,}
                 \DataTypeTok{distr=}\StringTok{"sl"}\NormalTok{,}\DataTypeTok{depStruct =} \StringTok{"ARp"}\NormalTok{,}\DataTypeTok{pAR=}\DecValTok{2}\NormalTok{,}\DataTypeTok{quiet=}\NormalTok{T)}
\end{Highlighting}
\end{Shaded}

\begin{Shaded}
\begin{Highlighting}[]
\CommentTok{#cn}
\NormalTok{fit3ar1<-}\KeywordTok{smn.lmm}\NormalTok{(}\DataTypeTok{data=}\NormalTok{sleepstudy,}\DataTypeTok{formFixed =}\NormalTok{ Reaction}\OperatorTok{~}\NormalTok{Days,}
                 \DataTypeTok{formRandom =} \OperatorTok{~}\NormalTok{Days,}\DataTypeTok{groupVar =} \StringTok{"Subject"}\NormalTok{,}
                 \DataTypeTok{distr=}\StringTok{"cn"}\NormalTok{,}\DataTypeTok{depStruct =} \StringTok{"ARp"}\NormalTok{,}\DataTypeTok{pAR=}\DecValTok{1}\NormalTok{,}\DataTypeTok{quiet=}\NormalTok{T)}
\end{Highlighting}
\end{Shaded}

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{fit3ar2<-}\KeywordTok{smn.lmm}\NormalTok{(}\DataTypeTok{data=}\NormalTok{sleepstudy,}\DataTypeTok{formFixed =}\NormalTok{ Reaction}\OperatorTok{~}\NormalTok{Days,}
                 \DataTypeTok{formRandom =} \OperatorTok{~}\NormalTok{Days,}\DataTypeTok{groupVar =} \StringTok{"Subject"}\NormalTok{,}
                 \DataTypeTok{distr=}\StringTok{"cn"}\NormalTok{,}\DataTypeTok{depStruct =} \StringTok{"ARp"}\NormalTok{,}\DataTypeTok{pAR=}\DecValTok{2}\NormalTok{,}\DataTypeTok{quiet=}\NormalTok{T)}
\end{Highlighting}
\end{Shaded}

\end{frame}

\begin{frame}[fragile]{Fitting a DEC-SMN-LMM}
\protect\hypertarget{fitting-a-dec-smn-lmm}{}

\scriptsize

\begin{Shaded}
\begin{Highlighting}[]
\CommentTok{#sl}
\NormalTok{fit2dec<-}\KeywordTok{smn.lmm}\NormalTok{(}\DataTypeTok{data=}\NormalTok{sleepstudy,}\DataTypeTok{formFixed =}\NormalTok{ Reaction}\OperatorTok{~}\NormalTok{Days,}
                 \DataTypeTok{formRandom =} \OperatorTok{~}\NormalTok{Days,}\DataTypeTok{groupVar =} \StringTok{"Subject"}\NormalTok{,}
                 \DataTypeTok{distr=}\StringTok{"sl"}\NormalTok{,}\DataTypeTok{depStruct =} \StringTok{"DEC"}\NormalTok{,}\DataTypeTok{quiet=}\NormalTok{T,}
                 \DataTypeTok{timeVar =} \StringTok{"Days"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{Shaded}
\begin{Highlighting}[]
\CommentTok{#cn}
\NormalTok{fit3dec<-}\KeywordTok{smn.lmm}\NormalTok{(}\DataTypeTok{data=}\NormalTok{sleepstudy,}\DataTypeTok{formFixed =}\NormalTok{ Reaction}\OperatorTok{~}\NormalTok{Days,}
                 \DataTypeTok{formRandom =} \OperatorTok{~}\NormalTok{Days,}\DataTypeTok{groupVar =} \StringTok{"Subject"}\NormalTok{,}
                 \DataTypeTok{distr=}\StringTok{"cn"}\NormalTok{,}\DataTypeTok{depStruct =} \StringTok{"DEC"}\NormalTok{,}\DataTypeTok{quiet=}\NormalTok{T,}
                 \DataTypeTok{timeVar =} \StringTok{"Days"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\normalsize
Since the data are equally spaced and sorted by time, the use of
\texttt{timeVar} in here is optional (the function will use the position
if \texttt{timeVar} is not provided).

\end{frame}

\begin{frame}{Comparing the fitted models}
\protect\hypertarget{comparing-the-fitted-models-1}{}

\begin{longtable}[]{@{}llrr@{}}
\toprule
distr & depStruct & AIC & BIC\tabularnewline
\midrule
\endhead
sl & CI & 1736.2 & 1758.5\tabularnewline
sl & AR1 & 1716.9 & \bf 1742.5\tabularnewline
sl & AR2 & 1717.3 & 1746.0\tabularnewline
sl & DEC & 1718.2 & 1746.9\tabularnewline
cn & CI & 1733.7 & 1759.3\tabularnewline
cn & AR1 & 1715.5 & 1744.3\tabularnewline
cn & AR2 & \bf 1714.9 & 1746.9\tabularnewline
cn & DEC & 1716.3 & 1748.2\tabularnewline
\bottomrule
\end{longtable}

\end{frame}

\begin{frame}[fragile]{Healy plot for the AR(\(p\))-SMN-LMM}
\protect\hypertarget{healy-plot-for-the-arp-smn-lmm}{}

\scriptsize

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{grid.arrange}\NormalTok{(}\KeywordTok{healy.plot}\NormalTok{(fit2ar1),}\KeywordTok{healy.plot}\NormalTok{(fit3ar1),}
            \KeywordTok{healy.plot}\NormalTok{(fit3ar2),}\DataTypeTok{ncol=}\DecValTok{2}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{center}\includegraphics[width=0.75\linewidth]{codes_files/figure-beamer/healy2-1} \end{center}

\end{frame}

\begin{frame}[fragile]{LR test for \(H_0: \phi_2=0\) in the CN-LMM}
\protect\hypertarget{lr-test-for-h_0-phi_20-in-the-cn-lmm}{}

\scriptsize

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{lr.test}\NormalTok{(fit3ar1,fit3ar2)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##
## Model selection criteria:
##           logLik      AIC      BIC
## fit3ar1 -848.760 1715.520 1744.257
## fit3ar2 -847.463 1714.925 1746.855
##
##     Likelihood-ratio Test
##
## chi-square statistics =  2.594949
## df =  1
## p-value =  0.1072049
##
## The null hypothesis that both models represent the
## data equally well is not rejected at level  0.05
\end{verbatim}

\end{frame}

\begin{frame}[fragile]{ACF plot for AR(1)-SMN-LMM}
\protect\hypertarget{acf-plot-for-ar1-smn-lmm}{}

\scriptsize

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{grid.arrange}\NormalTok{(}\KeywordTok{plot}\NormalTok{(}\KeywordTok{acfresid}\NormalTok{(fit2ar1,}\DataTypeTok{calcCI =}\NormalTok{ T,}\DataTypeTok{maxLag =} \DecValTok{6}\NormalTok{))}\OperatorTok{+}
\StringTok{               }\KeywordTok{ggtitle}\NormalTok{(}\StringTok{"AR(1)-SL-LMM"}\NormalTok{),}
             \KeywordTok{plot}\NormalTok{(}\KeywordTok{acfresid}\NormalTok{(fit3ar1,}\DataTypeTok{calcCI =}\NormalTok{ T,}\DataTypeTok{maxLag =} \DecValTok{6}\NormalTok{))}\OperatorTok{+}
\StringTok{               }\KeywordTok{ggtitle}\NormalTok{(}\StringTok{"AR(1)-CN-LMM"}\NormalTok{), }\DataTypeTok{ncol=}\DecValTok{2}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{center}\includegraphics[width=0.8\linewidth]{codes_files/figure-beamer/acf1-1} \end{center}

\end{frame}

% \begin{frame}[fragile]{Fitted models}
% \protect\hypertarget{fitted-models}{}

% \scriptsize

% \begin{Shaded}
% \begin{Highlighting}[]
% \CommentTok{#sl}
% \KeywordTok{cbind}\NormalTok{(fit2ar1}\OperatorTok{$}\NormalTok{theta,fit2ar1}\OperatorTok{$}\NormalTok{std.error) }\OperatorTok{%>%}\StringTok{ }
% \StringTok{  }\NormalTok{knitr}\OperatorTok{::}\KeywordTok{kable}\NormalTok{(}\DataTypeTok{digits =} \DecValTok{2}\NormalTok{,}\DataTypeTok{col.names =} \KeywordTok{c}\NormalTok{(}\StringTok{'estimate'}\NormalTok{,}\StringTok{'std.error'}\NormalTok{))}
% \end{Highlighting}
% \end{Shaded}

% \begin{longtable}[]{@{}lrr@{}}
% \toprule
% & estimate & std.error\tabularnewline
% \midrule
% \endhead
% (Intercept) & 251.55 & 7.56\tabularnewline
% Days & 9.81 & 1.66\tabularnewline
% sigma2 & 266.24 & 63.43\tabularnewline
% phiAR1 & 0.57 & 0.14\tabularnewline
% Dsqrt1 & 14.21 & 10.15\tabularnewline
% Dsqrt2 & 2.58 & 4.16\tabularnewline
% Dsqrt3 & 2.17 & 6.45\tabularnewline
% nu1 & 1.52 & NA\tabularnewline
% \bottomrule
% \end{longtable}

% \end{frame}

% \begin{frame}[fragile]

% \scriptsize

% \begin{Shaded}
% \begin{Highlighting}[]
% \CommentTok{#cn}
% \KeywordTok{cbind}\NormalTok{(fit3ar1}\OperatorTok{$}\NormalTok{theta,fit3ar1}\OperatorTok{$}\NormalTok{std.error) }\OperatorTok{%>%}\StringTok{ }
% \StringTok{  }\NormalTok{knitr}\OperatorTok{::}\KeywordTok{kable}\NormalTok{(}\DataTypeTok{digits =} \DecValTok{2}\NormalTok{,}\DataTypeTok{col.names =} \KeywordTok{c}\NormalTok{(}\StringTok{'estimate'}\NormalTok{,}\StringTok{'std.error'}\NormalTok{))}
% \end{Highlighting}
% \end{Shaded}

% \begin{longtable}[]{@{}lrr@{}}
% \toprule
% & estimate & std.error\tabularnewline
% \midrule
% \endhead
% (Intercept) & 252.55 & 7.02\tabularnewline
% Days & 9.99 & 1.57\tabularnewline
% sigma2 & 417.84 & 78.77\tabularnewline
% phiAR1 & 0.56 & 0.14\tabularnewline
% Dsqrt1 & 16.90 & 11.82\tabularnewline
% Dsqrt2 & 2.82 & 4.76\tabularnewline
% Dsqrt3 & 2.89 & 6.31\tabularnewline
% nu1 & 0.12 & NA\tabularnewline
% nu2 & 0.16 & NA\tabularnewline
% \bottomrule
% \end{longtable}

% \end{frame}

\begin{frame}[fragile]{Fitted models}
\protect\hypertarget{fitted-models}{}

\scriptsize

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{cbind}\NormalTok{(}\NormalTok{fit2ar1}\OperatorTok{$}\NormalTok{theta}\NormalTok{,}\NormalTok{fit2ar1}\OperatorTok{$}\NormalTok{std.error}\NormalTok{)}
\KeywordTok{cbind}\NormalTok{(}\NormalTok{fit3ar1}\OperatorTok{$}\NormalTok{theta,fit3ar1}\OperatorTok{$}\NormalTok{std.error) }
\end{Highlighting}
\end{Shaded}

\begin{longtable}[]{@{}lrrrr@{}}
\toprule
& \multicolumn{2}{c}{{SL}} &\multicolumn{2}{c}{{CN}}\tabularnewline
& estimate & std.error & estimate & std.error\tabularnewline
\midrule
\endhead
(Intercept) & 251.55 & 7.56 & 252.55 & 7.02\tabularnewline
Days & 9.81 & 1.66 & 9.99 & 1.57\tabularnewline
sigma2 & 266.24 & 63.43 & 417.84 & 78.77\tabularnewline
phiAR1 & 0.57 & 0.14 & 0.56 & 0.14\tabularnewline
Dsqrt1 & 14.21 & 10.15 & 16.90 & 11.82\tabularnewline
Dsqrt2 & 2.58 & 4.16 & 2.82 & 4.76\tabularnewline
Dsqrt3 & 2.17 & 6.45 & 2.89 & 6.31\tabularnewline
nu1 & 1.52 & - & 0.12 & -\tabularnewline
nu2& - & - & 0.16 & -\tabularnewline
\bottomrule
\end{longtable}

\end{frame}

\begin{frame}[fragile]{Mahalanobis distance for AR(1)-SMN-LMM}
\protect\hypertarget{mahalanobis-distance-for-ar1-smn-lmm}{}

\scriptsize

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{grid.arrange}\NormalTok{(}\KeywordTok{plot}\NormalTok{(}\KeywordTok{mahalDist}\NormalTok{(fit2ar1),}\DataTypeTok{fitobject =}\NormalTok{ fit2ar1,}\DataTypeTok{nlabels =} \DecValTok{0}\NormalTok{)}\OperatorTok{+}
\StringTok{               }\KeywordTok{ggtitle}\NormalTok{(}\StringTok{'AR(1)-SL-LMM'}\NormalTok{),}
             \KeywordTok{plot}\NormalTok{(}\KeywordTok{mahalDist}\NormalTok{(fit3ar1),}\DataTypeTok{fitobject =}\NormalTok{ fit3ar1,}\DataTypeTok{nlabels =} \DecValTok{0}\NormalTok{)}\OperatorTok{+}
\StringTok{               }\KeywordTok{ggtitle}\NormalTok{(}\StringTok{'AR(1)-CN-LMM'}\NormalTok{),} \DataTypeTok{ncol=}\DecValTok{2}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{center}\includegraphics[width=0.8\linewidth]{codes_files/figure-beamer/mahal1-1} \end{center}

\end{frame}

\begin{frame}[fragile]{Mahalanobis distance versus \(\hat{u}\) for
AR(1)-SMN-LMM}
\protect\hypertarget{mahalanobis-distance-versus-hatu-for-ar1-smn-lmm}{}

\scriptsize

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{grid.arrange}\NormalTok{(}\KeywordTok{qplot}\NormalTok{(}\KeywordTok{mahalDist}\NormalTok{(fit2ar1),fit2ar1}\OperatorTok{$}\NormalTok{uhat)}\OperatorTok{+}\KeywordTok{theme_minimal}\NormalTok{()}\OperatorTok{+}
\StringTok{               }\KeywordTok{ylab}\NormalTok{(}\StringTok{"uhat"}\NormalTok{)}\OperatorTok{+}\KeywordTok{xlab}\NormalTok{(}\StringTok{"Mahalanobis distance"}\NormalTok{)}\OperatorTok{+}
\StringTok{               }\KeywordTok{ggtitle}\NormalTok{(}\StringTok{'AR(1)-SL-LMM'}\NormalTok{),}
             \KeywordTok{qplot}\NormalTok{(}\KeywordTok{mahalDist}\NormalTok{(fit3ar1),fit3ar1}\OperatorTok{$}\NormalTok{uhat)}\OperatorTok{+}\KeywordTok{theme_minimal}\NormalTok{()}\OperatorTok{+}
\StringTok{               }\KeywordTok{ylab}\NormalTok{(}\StringTok{"uhat"}\NormalTok{)}\OperatorTok{+}\KeywordTok{xlab}\NormalTok{(}\StringTok{"Mahalanobis distance"}\NormalTok{)}\OperatorTok{+}
\StringTok{               }\KeywordTok{ggtitle}\NormalTok{(}\StringTok{'AR(1)-CN-LMM'}\NormalTok{),} \DataTypeTok{ncol=}\DecValTok{2}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{center}\includegraphics[width=0.8\linewidth]{codes_files/figure-beamer/mahal2-1} \end{center}

\end{frame}

\begin{frame}[fragile]{Plotting fitted models}
\protect\hypertarget{plot-fitted-models}{}

\scriptsize

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{grid.arrange}\NormalTok{(}\KeywordTok{plot}\NormalTok{(fit2ar1,}\DataTypeTok{type =} \StringTok{"normalized"}\NormalTok{)}\OperatorTok{+}\KeywordTok{ggtitle}\NormalTok{(}\StringTok{'AR(1)-SL-LMM'}\NormalTok{),}
             \KeywordTok{plot}\NormalTok{(fit3ar1,}\DataTypeTok{type =} \StringTok{"normalized"}\NormalTok{)}\OperatorTok{+}\KeywordTok{ggtitle}\NormalTok{(}\StringTok{'AR(1)-CN-LMM'}\NormalTok{),}
             \DataTypeTok{ncol=}\DecValTok{2}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{center}\includegraphics[width=0.85\linewidth]{codes_files/figure-beamer/extra1-1} \end{center}

\end{frame}

\begin{frame}[fragile]{Prediction of future measurements}
\protect\hypertarget{prediction-of-future-measurements}{}

\scriptsize

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{tail}\NormalTok{(sleepstudy,}\DataTypeTok{n=}\DecValTok{3}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##     Reaction Days Subject
## 178 343.2199    7     372
## 179 369.1417    8     372
## 180 364.1236    9     372
\end{verbatim}

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{predData <-}\StringTok{ }\KeywordTok{data.frame}\NormalTok{(}\DataTypeTok{Reaction=}\OtherTok{NA}\NormalTok{,}\DataTypeTok{Days=}\DecValTok{10}\OperatorTok{:}\DecValTok{11}\NormalTok{,}\DataTypeTok{Subject=}\StringTok{"372"}\NormalTok{)}
\KeywordTok{predict}\NormalTok{(fit2ar1,}\DataTypeTok{newData =}\NormalTok{ predData)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##   groupVar Days    ypred
## 1      372   10 377.0998
## 2      372   11 389.5745
\end{verbatim}

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{predict}\NormalTok{(fit3ar1,}\DataTypeTok{newData =}\NormalTok{ predData)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##   groupVar Days    ypred
## 1      372   10 377.1056
## 2      372   11 389.5576
\end{verbatim}

\end{frame}

\section{Concluding remarks}
\begin{frame}{Concluding remarks}
    Some additional features are currently under development:
    \begin{itemize}
        \item Estimation of some important special cases, such as when $\bD$ is diagonal or in blocks;
        \item Use of parallel optimization to improve performance;
        \item Use of a method for acceleration of the convergence rate of the EM-type algorithm used in the estimation procedure.
    \end{itemize}
\end{frame}

\begin{frame}{Main references}
\protect\hypertarget{references}{}
\vspace{-.2cm}
\tiny{
\begin{thebibliography}{99} % Beamer does not support BibTeX so references must be inserted manually as below
\bibitem{azzalini} Azzalini, A. \& A. D. Valle (1996).
\newblock The multivariate skew-normal distribution.
\newblock Biometrika 83(4), 715–726.

\bibitem{belenky} Belenky, G., Wesensten, N. J., Thorne, D. R., Thomas, M, L., Sing, H. C., Redmond, D. P., Russo, M. B. \& Balkin, T. J. (2003).
\newblock Patterns of performance degradation and restoration during sleep restriction and subsequent recovery: a sleep
  dose-response study.
\newblock Journal of Sleep Research 12, 1--12.

\bibitem{box} Box, G. E., G. M. Jenkins, G. C. Reinsel \& G. M. Ljung (2015).
\newblock Time series analysis: forecasting and control.
\newblock John Wiley \& Sons.

\bibitem{dempster} Dempster, A., Laird, N. \& Rubin, D. (1977).
\newblock Maximum likelihood from incomplete data via the EM algorithm.
\newblock Journal of the Royal Statistical Society, Series B 39, 1–38.

\bibitem{lachos} Lachos, V. H., P. Ghosh \& R. B. Arellano-Valle (2010).
\newblock Likelihood based inference for skew–normal independent linear mixed models.
\newblock Statistica Sinica 20, 303–322.

\bibitem{munoz} Muñoz, A., V. Carey, J. P. Schouten, M. Segal \& B. Rosner (1992).
\newblock A parametric family of correlation structures for the analysis of longitudinal data.
\newblock Biometrics 48, 733–742.

\bibitem{schumacher} Schumacher, F. L., Matos, L. A., \& Lachos, V. H. (2020).
\newblock Scale mixture of skew-normal linear mixed models with within-subject serial dependence.
\newblock arXiv preprint arXiv:2002.01040.

\end{thebibliography}
}
\end{frame}

\begin{frame}
%\vspace{-.5cm}
\begin{columns}
\column{.45\textwidth}
\begin{tabular}{cc}
         \large Preprint&  \large GitHub \\
         \href{https://arxiv.org/abs/2002.01040}{\includegraphics[ height=2.2cm, keepaspectratio=true]{Figuras/paper-arxiv.png}}
         & \href{https://github.com/fernandalschumacher}{\includegraphics[ height=2.2cm, keepaspectratio=true]{Figuras/github.png}}
\end{tabular}
\column{.49\textwidth}
\vspace{-.4cm}\center{\Large Acknowledgments}
\\%\includegraphics[ height=1.7cm, keepaspectratio=true]{imecc} \hspace{.5cm}
\includegraphics[ height=2.2cm, keepaspectratio=true]{Figuras/cnpq_capes.jpg}
\end{columns}

\vfill

\fboxsep=0pt
\noindent%
\begin{minipage}[t]{0.38\linewidth}
\begin{flushright}
\vspace{.01cm}
\includegraphics[ height=1cm, keepaspectratio=true]{Figuras/contact.jpg}
\end{flushright}
\end{minipage}%
\hfill%
\begin{minipage}[t]{0.6\linewidth}
\begin{flushright}
%\color{paynesgrey}
Fernanda Lang Schumacher\\
PhD student, IMECC -- UNICAMP\\
\color{black}
\nolinkurl{fernandalschumacher@gmail.com}
%\url{https://www.linkedin.com/in/fernanda-lang-schumacher-18123775}
\end{flushright}
\end{minipage}
%\center{\huge Thank you!}
\end{frame}


\clearpage
\phantomsection
\addcontentsline{toc}{subsection}{SRL}
\label{insn:srl}
\subsection*{SRL: Shift Right Logical}

\subsubsection*{Format}

\textrm{SRL \%rd, \%r1, \%r2}

\begin{center}
\begin{bytefield}[endianness=big,bitformatting=\scriptsize]{32}
\bitheader{0,7,8,15,16,23,24,31} \\
\bitbox{8}{0x05}
\bitbox{8}{r1}
\bitbox{8}{r2}
\bitbox{8}{rd}
\end{bytefield}
\end{center}

\subsubsection*{Description}

This instruction shifts the value found in regiseter \register{r1} right by
the number of bits found in register \register{r2}, placing the results in
register \register{rd}. This instruction only operates on \textbf{unsigned}
integers.

\subsubsection*{Pseudocode}

\begin{verbatim}
%rd = %r1 >> %r2
\end{verbatim}

\subsubsection*{Constraints}

\registerop{r1}, \registerop{r2}, and \registerop{rd} must be less than
\nregs{}.

\medskip
\noindent
\registerop{rd} cannot be \register{r0}.

\subsubsection*{Failure modes}

This instruction has no run-time failure modes beyond its constraints.