\documentclass[10pt,conference,compsocconf]{IEEEtran}

\usepackage{amsmath}    % math
\usepackage{graphicx}   % For figure environment
\usepackage{todonotes}
\usepackage{subcaption}

\usepackage{hyperref}
\begin{document}
\title{Mini-project II}

\author{
  Ada Pozo P\'{e}rez, Luc\'{i}a Montero Sanchis, Milica Novakovic\\
  \textit{Deep Learning 2018, EPFL Lausanne, Switzerland}
}

\maketitle

\begin{abstract}
  This report describes the implementation of a simple deep learning framework. Several different optimizers, activation functions, layers and loss functions are implemented, including fully connected, dropout, \textit{ReLU}, \textit{Tanh}, sequential and \textit{MSE} loss among others. This framework is used for testing different architectures on a toy dataset, using different optimizers, in order to determine the effects that different activation functions and optimizers have in the training and accuracy.
\end{abstract}

\section{Introduction}
	In this project we implement a deep learning framework using only \textit{Pytorch} tensors and \textit{numpy}.

	The framework is built with two base classes: \textit{Module} and \textit{Optimizer}. The following containers are implemented as subclasses for them:

	\begin{itemize}
	    \item Layers: \textit{Linear}, \textit{Sequential} and \textit{Dropout}.
	    \item Activation functions: \textit{ReLU}, \textit{Leaky ReLU}, \textit{Sigmoid} and \textit{Tanh}.
	    \item Loss functions: \textit{Mean Squared Error (MSE)} and \textit{Cross-entropy loss}.
	    \item Optimizers: \textit{Stochastic Gradient Descent} and \textit{Adam}.
	\end{itemize}

    To test our framework we generate a toy dataset and run several experiments on it, comparing the performance of different combinations of modules, activations, optimizers and loss functions.

    The rest of this report is organized as follows: Section \ref{sec:implementation} describes how the different containers were implemented while Section \ref{sec:experiments} shows the results of the experiment to test the framework. Conclusions are drawn in Section \ref{sec:summary}.

\section{Implementation} % (fold)
\label{sec:implementation}
    We have defined two generic classes, \textit{Module} and \textit{Optimizer}, that are used as parent classes for the implementations. The optimizers are based on the structure of \textit{Optimizer}, whereas the remaining implementations are based on \textit{Module}.

    \textit{Optimizer} has two methods:
    \begin{itemize}
        \item \textit{step}: implements the update of the parameters.
        \item \textit{adaptive\_lr}: returns a generator that decays the learning rate by a factor with every optimization step.
    \end{itemize}

    In \textit{Module} we can find the methods:
    \begin{itemize}
        \item \textit{forward} and  \textit{backward}: implement the forward and backward passes of backpropagation, respectively.
        \item \textit{update}: updates the values of the parameters and resets the computed gradients to zero.
        \item \textit{params}: returns the values and the last computed gradients of the parameters of the layer.
    \end{itemize}

  \subsection{Layers} % (fold)
  \label{sub:modules}
    % Not sure about having ``modules'' as a title here
    \subsubsection{Linear} % (fold)
    \label{ssub:linear}
        The fully connected layer is implemented in the \textit{Linear} module. It is created specifying the number of input and output units. It is also possible to specify the default learning rate, which will be used to update the parameters. Nevertheless, this learning rate is only used when no learning rate is specified for the \textit{update} function.

        Unless a specific standard deviation (std) is specified during initialization, the weights are initialized with the Xavier \cite{Glorot10understandingthe} initialization adapted for ReLU \cite{DBLP:journals/corr/HeZR015} -- following a zero-mean Gaussian distribution with standard deviation given by $\text{std} = \sqrt{2/n}$,
        where $n$ is the number of input units.
        If a value for parameter \textit{std\_w} is specified, then the weights are initialized according to the zero-mean Gaussian distribution whose std value is \textit{std\_w}. For instance, when using $\tanh$ or sigmoid activation functions we recommend to specify a value for the parameter equal to $\text{std} = 1\sqrt{n}$.

        In addition to this, in the initialization it is also specified whether there is a bias -- in which case the bias values would be initialized following a Gaussian distribution with zero mean and standard deviation given by the parameter \textit{std\_b}. If the value of parameter \textit{std\_b} is not specified, they are initialized to 0. If there is no bias, in the backpropagation this parameter is ignored.

        During the forward pass, the input to the layer is saved for the subsequent gradient computation. The output of the layer $s^{(l)}$ is computed with the equation \ref{eq:forward}, where $w^{(l)}$ and $b^{(l)}$ are the values of the weights and the bias respectively and $x^{(l-1)}$ is the input to the layer. The bias term is considered if it was specified to do so, otherwise it is ignored.
        \begin{equation}
      	    \label{eq:forward}
            s^{(l)} = w^{(l)}x^{(l-1)} + b^{(l)}
        \end{equation}

        The backward pass function receives as an argument the gradient of the loss with respect to the output of the layer ($\partial l/\partial s^{(l)}$) and returns the gradient of the loss with respect to the input of the layer, which is computed as given by equation \ref{eq:backward_x}. The gradient of the loss with respect to the weights and with respect to the bias are computed with equations \ref{eq:backward_w} and \ref{eq:backward_s}, respectively:
         \begin{equation}
      	    \label{eq:backward_x}
            \frac{\partial l}{\partial x^{(l-1)}} =\frac{\partial l}{\partial s^{(l)}}  w^{(l)}
         \end{equation}
         \begin{equation}
      	    \label{eq:backward_w}
            \frac{\partial l}{\partial w^{(l)}} =\left(\frac{\partial l}{\partial s^{(l)}}\right)^T x^{(l-1)}
         \end{equation}
         \begin{equation}
      	    \label{eq:backward_s}
            \frac{\partial l}{\partial b^{(l)}} =\frac{\partial l}{\partial s^{(l)}}
         \end{equation}

        The update of the parameters is performed when the method \textit{update} is called. If a learning rate is passed as a parameter it is used for the update, otherwise the one specified upon initialization is used. As will be explained in more detail in the optimizers section \ref{sub:optimizers}, the update can be done directly on the derivatives of the bias and weights (for instance, when using SGD), or in the case when the parameter \textit{values} is specified it will use these values instead (if Adam is used, the optimizer computes the parameter \textit{values} that should be used). The equations used for the updates are therefore given by \ref{eq:update_w} and \ref{eq:update_b}, where $\gamma$ is the learning rate:
        \begin{equation}
            \label{eq:update_w}
            w^{(l)}\leftarrow
            \begin{cases}
              w^{(l)} -  \gamma \cdot \partial l/\partial w^{(l)}, & \text{if SGD is used} \\
              w^{(l)} -  \gamma \cdot \text{value}_w, & \text{if Adam is used}
            \end{cases}
        \end{equation}

        \begin{equation}
            \label{eq:update_b}
            b^{(l)}\leftarrow
            \begin{cases}
              b^{(l)} -  \gamma \cdot \partial l/\partial w^{(l)}, & \text{if SGD is used} \\
              b^{(l)} -  \gamma \cdot \text{value}_b, & \text{if Adam is used}
            \end{cases}
        \end{equation}

        When using \textit{Adam}, $\text{value}_w$ and $\text{value}_b$ are computed as given by equation \ref{eq:valueadam}.

        Lastly, the \textit{Linear} module's method \textit{params} returns a list of tuples containing a tuple with the weights and the gradient and, if bias is activated, a second tuple with the values of the bias and the gradient.
    % subsubsection linear (end)

  	\subsubsection{Sequential} % (fold)
  	\label{ssub:sequential}
        This class implements a container for a list of layers. Its parameters are the list of layers and a variable that indicates whether the model is in training or testing mode. In the forward pass, it simply calls the \textit{forward} method of each layer, propagating the output through the layers. In the backward pass, it propagates the gradient calling the corresponding method of each layer iterating through them in reverse order.

        Likewise, the \textit{params} method calls the \textit{params} method of each layer and combines their outputs, returning a list of tuples containing the values and gradient for each parameter.

        The \textit{update} method iterates over all the layers calling each layer's corresponding method. If a learning rate $\gamma$ is specified, then it is used to update all parameters. It is also possible to specify the \textit{values} parameter that should be used for the update, instead of using the gradient computed during the forward pass. As previously mentioned, this is useful for implementing the Adam optimizer and will be explained later.
   	% subsubsection sequential (end)

  	\subsubsection{Dropout} % (fold)
  	\label{ssub:dropout}
        We implement a dropout layer by defining a class \textit{Dropout}. This layer is created taking as parameters the probability of dropping a hidden unit ($p_{drop}$) and whether the model is training. If it is not training, the forward and backward passes simply return their respective inputs.

        The forward pass is implemented in the \textit{forward} method. It creates a mask of hidden units to drop and saves it for the backward pass. The mask is used to shut down the corresponding units of the input data. Lastly, the result is normalized by $1-p_{drop}$. The \textit{backward} method carries out the backward pass, using the mask saved in the forward pass to shut down the same units of the gradient of the next layer. The result is again normalized by $1-p_{drop}$.
  	% subsubsection dropout (end)
  % subsection modules (end)

  \subsection{Activation functions} % (fold)
  \label{sub:activation_functions}
    The following activation functions are implemented in our framework:
  	\subsubsection{ReLU} % (fold)
  	\label{ssub:relu}
        In the forward pass this activation layer applies the non-linear function $f(x) = max(0,x)$ to the input \cite{Nair:2010:RLU:3104322.3104425}. The backward pass is computed with the derivative:
      	\begin{equation*}
            f'(x)=
            \begin{cases}
              1, & \text{if}\ x>0 \\
              0, & \text{otherwise}
            \end{cases}
        \end{equation*}
  	% subsubsection relu (end)

  	\subsubsection{Leaky ReLU} % (fold)
  	\label{ssub:leakyrelu}
        We implement a parametric Leaky ReLU, which takes takes a parameter $\alpha$ and computes the forward pass as:
      	\begin{equation}
            f(x)=
            \begin{cases}
              x, & \text{if}\ x>0 \\
              \alpha\cdot x, & \text{otherwise}
            \end{cases}
        \end{equation}

        In the backward pass, the gradient is given by:
      	\begin{equation}
            f'(x)=
            \begin{cases}
              1, & \text{if}\ x>0 \\
               \alpha, & \text{otherwise}
            \end{cases}
        \end{equation}

  	% subsubsection leakyrelu (end)
  	\subsubsection{Tanh} % (fold)
  	\label{ssub:tanh}
        We implement the hyperbolic tangent activation function in the class \textit{Tanh}. This class computes in the \textit{forward} method the $\tanh$ of the input, which is saved for the next step. In the backward pass step, we compute the gradient as $1-\tanh^2{x}$
  	% subsubsection tanh (end)
  	\subsubsection{Sigmoid} % (fold)
  	\label{ssub:sigmoid}
        We implement the forward pass of this function using the $\tanh$ function to ensure that there is no overflow, according to the formula $\sigma(x) = 0.5 + 0.5\cdot\tanh{(x/2)}$. We store the obtained value and then compute the derivative as $\sigma(x) \cdot (1-\sigma(x))$.
  	% subsubsection sigmoid (end)
  % subsection activation_functions (end)
  \subsection{Loss functions} % (fold)
  \label{sub:loss_functions}

  Two loss functions are implemented in this framework: the Mean Squared Error (MSE) and the Cross Entropy Loss.
  	\subsubsection{Mean Squared Error (MSE)} % (fold)
  	\label{ssub:mean_squared_error}
        In the forward pass, this module computes the difference between the output of the network $f(x,w)$ and the ground truth $y$ and saves it for the backward pass. The loss returned is computed as:
      	\begin{equation}
      	Loss = \sum_{i=1}^{N}(y_i - f(x_i,w))^2
      	\end{equation}

      	The backward pass returns the gradient of the loss, which is computed using the stored difference between the output of the network and the ground truth:
      	\begin{equation}
      	    \nabla_{Loss} = 2 (y - f(x,w))
      	\end{equation}
  	% subsubsection mean_squared_error (end)
  	\subsubsection{Cross-entropy loss} % (fold)
  	\label{ssub:cross_entropy_loss}
  	This module computes the loss of the network in the forward pass as:
  	\begin{equation}
  	    Loss = -\sum_{i=1}^{N} y_i\log (f(x_i,w)) + (1 - y_i)\log(1 - f(x_i,w))
  	\end{equation}

  	In the backward pass the gradient of the loss is computed with the following equation:
  	\begin{equation}
  	    \nabla_{Loss} =  y - f(x,w)
  	\end{equation}
  	% subsubsection cross_entropy_loss (end)
  % subsection loss_functions (end)
  \subsection{Optimizers} % (fold)
  \label{sub:optimizers}
  As explained in the beginning of Section \ref{sec:implementation}, the optimizers are defined using a class with a \textit{step} method that updates the parameters and a method \textit{adaptive\_lr} that implements a generator that yields decaying learning rates. The decay is controlled by parameters $\kappa$ and $\eta$ according to:
    \begin{equation}
    \label{eq:lr}
  	    \gamma^{(t)} =  \eta\cdot t^ {-\kappa}
  	\end{equation}
  	where $t$ is incremented every time the generator is called.
  	\subsubsection{Stochastic Gradient Descent (SGD)} % (fold)
  	\label{sub:stochastic_gradient_descent_}
        The parameter update using Stochastic Gradient Descent calls the \textit{update} method of the model (explained in Sections \ref{ssub:linear} and \ref{ssub:sequential}). This update can be done either with a fixed learning rate if $\kappa$ and $\eta$ are not passed as parameters when creating the optimizer, or with the described decaying learning rate if they are passed. In the second case, each time \textit{step} is called the next learning rate is generated and passed to the \textit{update} method of the model.
  	% subsection stochastic_gradient_descent_ (end)
  	\subsubsection{Adam} % (fold)
  	\label{ssub:adam}
        This class is implemented as described in \cite{kingma2014adam}.  Given the parameters of the optimizer settings $\beta_1$, $\beta_2$ and $\epsilon$, at each timestep $t$ we first calculate the gradient with respect to the stochastic objective $g_t$ and then the first and second moment vectors are computed according to equations \ref{eq:fmv} and \ref{eq:smv} respectively:
  		\begin{equation} \label{eq:fmv}
            m_t=\beta_1\cdot m_{t-1}+(1-\beta_1)\cdot g_t
        \end{equation}
  		\begin{equation}
  		    \label{eq:smv}
            v_t=\beta_2\cdot v_{t-1}+(1-\beta_2)\cdot g_t^2
        \end{equation}

        With the previously obtained results we compute the \textit{value} argument that is used to update the parameter, as given by:
        \begin{equation}
  		    \label{eq:valueadam}
            \text{value} = \frac{m_t}{1 - \beta_1^{t}}\cdot \frac{1}{\epsilon+\sqrt{v_t/(1-\beta_2^t)}}
        \end{equation}
        Then, each parameter $\theta$ is updated as $\theta_{t}=\theta_{t-1}-lr\cdot\text{value}$.\\
        It is worth mentioning that this optimizer has a \textit{restart} method that resets the value of $t$ and the moments to $0$ (which are also the values with which the optimizer is initialized).

        The default values for the parameters are the ones used in   \cite{kingma2014adam}, that is, a learning rate of $10^{-3}$, and $\beta_1 = 0.9$ and $\beta_2 = 0.999$.
  	% subsubsection adam (end)
  % subsection optimizers (end)
% section implementation (end)

\section{Experiments} % (fold)

    \label{sec:experiments}
    \begin{figure}[t]
        \centering
        \includegraphics[width=0.45\textwidth]{figures/architecture_p2.png}
        \caption{Architecture of neural network}
        \label{fig:archi}
    \end{figure}

    \subsection{Datasets} % (fold)
    \label{sub:dataset}
    The train and test datasets are each composed of 1000 randomly generated samples. Each sample consists on two values chosen from a uniform distribution between 0 and 1, corresponding to a point in the 2D space. The samples that fall inside the disk of radius $1/\sqrt{2\pi}$ centered in $(0.5, 0.5)$ are labelled as 1's, whereas the rest are labelled as 0's.

    \subsection{Architecture} % (fold)
    \label{sub:architecture}

    The architecture of the system in the experiments can be found in Figure \ref{fig:archi}. It has 2 input and 2 output units and 3 fully connected layers with 25 units each. The fully connected layers are followed by a dropout layer, whose drop probability is set on the experiments. The two first hidden layers use the same activation, while the last one may use a different one. These activation functions vary depending on the experiment.

    \subsection{Results} % (fold)
    \label{sub:results}
        \begin{figure*}[t]
            \centering
            \begin{subfigure}{0.45\linewidth}
                \centering
                \includegraphics[width=\linewidth]{figures/res_circle_tr.png}
                \caption{Training set - 98.4\% accuracy.}
                \label{fig:rescircletr}
            \end{subfigure}%
            \begin{subfigure}{0.45\linewidth}
            \centering
            \includegraphics[width=\linewidth]{figures/res_circle_te.png}
            \caption{Test set - 97.5\% accuracy.}
            \label{fig:rescirclete}
            \end{subfigure}
            \caption{Classification results. Misclassified samples are shown in red, whereas correctly classified ones are shown in different gray levels depending on the class.}
            \label{fig:rescircle}
        \end{figure*}

        \begin{figure*}[t]
        \centering
        \begin{subfigure}{0.45\linewidth}
            \centering
            \includegraphics[width=\textwidth]{figures/adamsgdloss.png}
            \caption{Loss.}
            \label{fig:adamsgdloss}
             \end{subfigure}%
            \begin{subfigure}{0.45\linewidth}
            \centering
            \includegraphics[width=\textwidth]{figures/adamsgdaccuracy.png}
            \caption{Accuracy.}
            \label{fig:adamsgdaccuracy}
             \end{subfigure}
             \caption{Performance per epoch using SGD and Adam.}
            \label{fig:perf_optim}

        \end{figure*}
        To test our framework, we evaluate the performance with three different experiments:

        \begin{enumerate}
            \item General performance: this experiment aims to compare the performance on the train and test sets, checking whether there is any overfitting and if dropout is needed.
            \item Comparison of the optimizers: this experiment compares the performance in terms of speed and accuracy when using \textit{Adam} or \textit{SGD} as optimizers.
            \item Comparison of the activation functions: similarly to the previous experiment, this one compares the performance with different activation functions (\textit{Tanh}, \textit{ReLU} and \textit{Leaky ReLU}) for the first two layers.
        \end{enumerate}

        All the experiments use a a batch size of 50 samples and train the model for 50 epochs.

        \textbf{General performance.} For the first experiment, the model uses \textit{ReLU} as the activation function for the first two fully connected layers and  \textit{Sigmoid}  for the last one. No dropout is used - i.e. it has  0 drop probability. The optimizer used is \textit{Adam} with a learning rate of $10^{-3}$, and $\beta_1 = 0.9$ and $\beta_2 = 0.999$.  The classification results obtained are shown in Figures \ref{fig:rescircletr} and \ref{fig:rescirclete}, for training and test respectively, with the misclassified samples in red.

        The accuracy obtained for the test set is 97.5\%, very similar to the 98.4\% obtained for the training set. Hence, the model does not overfit, as could be expected since both sets are sampled from the same distribution. Therefore, we do not use dropout in any of the following experiments carried out.  It is also worth noting that all misclassified points are located in the border area.

        \textbf{Comparison of the optimizers.} In the second experiment we use the same architecture as in the previous one, but we train the model both with \textit{Adam} and \textit{SGD}. Adam uses the same parameters as before, while for \textit{SGD} we use the adaptive learning rate given by equation \ref{eq:lr}, with $\kappa=0.6$ and $\eta=0.035$. In Figure \ref{fig:perf_optim} can be found the loss and accuracy for the optimizers per epoch. From these figures one can observe that during the first 20 epochs, the training with SGD is faster, with a steeper slope than the one found using \textit{Adam}. However, this changes in the following epochs and  by epoch 50 both accuracy and loss are slightly better when using \textit{Adam}. Similarly to the previous experiment, there is no overfitting.

        \textbf{Comparison of the activation functions.}
        Lastly, we compare the results obtained using different activation functions (\textit{Tanh}, \textit{ReLU} and \textit{Leaky ReLU}) for the first two hidden layers using \textit{Adam} as optimizer. The results obtained can be observed in Figure \ref{fig:activ1comp}. In this plot we have omitted the test accuracy curves for simplicity since they were very close to their corresponding training accuracies. The accuracies obtained after 50 epochs are quite similar and do not seem to be strongly affected by the activation functions chosen for the first two activation layers. However, it is worth mentioning that \textit{Tanh} learns the fastest at first, while \textit{Leaky ReLU} seems to be the slowest.
        \begin{figure}[htp!]
            \centering
            \includegraphics[width=0.45\textwidth]{figures/activ1comp.png}
            \caption{Adam's accuracy with different activation functions.}
            \label{fig:activ1comp}
        \end{figure}

\section{Conclusions} % (fold)
\label{sec:summary}
	In this project we have implemented and used a simple deep learning framework for classifying the points that fall inside and outside a disk. The performance obtained with the chosen architecture was higher than 95\% in both train and test sets after 50 epochs. Since both sets were sampled from the same distribution, there was no overfitting in the train data. The misclassified points are located in the border area, which was to be expected because a small variation in these points results in a change of their label -- whereas small changes in points that are not in the border would not modify the labels. It should also be noted that by using a well-chosen learning rate -- in this case a decaying one -- \textit{SGD} achieved a similar result to the one obtained with \textit{Adam} width default parameters.
% section summary (end)

\bibliographystyle{IEEEtran}
\bibliography{literature}

\end{document}


\documentclass[jou]{apa6}

\usepackage[american]{babel}
\usepackage{hyperref}
\usepackage{amsthm}
\usepackage{thmtools}

\usepackage{csquotes}
\usepackage[style=apa,sortcites=true,sorting=nyt,backend=biber]{biblatex}
\DeclareLanguageMapping{american}{american-apa}
\addbibresource{bibliography.bib}

\title{2019-12-14 Class Summary: Chinese Remainder Theorem}

\author{Kalvis Aps\={\i}tis, {\small \tt kalvis.apsitis}{\small \tt @gmail.com}}
\affiliation{Riga Business School (RBS), University of Latvia (LU)}

\leftheader{NMS Selection Training in Number Theory: 2019-12-14 Class Summary}

\declaretheoremstyle[headfont=\normalfont\bfseries,notefont=\mdseries\bfseries,bodyfont = \normalfont,headpunct={:}]{normalhead}
\declaretheorem[name={Example}, style=normalhead,numberwithin=section]{problem}

\setcounter{section}{2}

\abstract{This document lists the key results from the December 14, 2019 class in
Number Theory. This training for competition math is aimed at 16\textendash{}18 year
olds (typically, Grades 10\textendash{}12).
}

\keywords{Chinese remainder theorem, Bezout's identity, Modular arithmetic.}

\begin{document}
\maketitle

\section{Examples}

These are {\bf not} the homework problems; it is just a supplementary study material
with examples taken from the lecture; most of them were analyzed during the lecture.
In case you have forgotten something, hints for these examples are given at the end of this document.
Full notes and solutions in Latvian: \url{http://linen-tracer-682.appspot.com/numtheory-tales/tale-numtheory-jun03-crt/content.html#/section}

\begin{problem}
Find integers $x,y$ such that $18x + 42y = 6$.
(Here we have chosen $6 = \mathit{gcd}(18,42)$.)
\end{problem}

\begin{problem}
Prove that the sequence $1,11,111,\ldots$ contains
an infinite subsequence such that any two members of that subsequence
are mutually prime.
\end{problem}

{\bf Blankinship Algorithm:} Blankinship Algorithm can be
used to find solutions for the
Bezout's identity: the integers $x,y$ such that $ax+by=d$.
It is explained here: \url{http://mathworld.wolfram.com/BlankinshipAlgorithm.html}.
It applies Gaussian row operations to a $2 \times 3$ matrix: you
should know how to subtract one row from another.

{\bf Inverse Congruence Class:} For a congruence class $a$
that is mutually prime with modulo $m$, denote by $a^{-1}$
a congruence class such that $a^{-1}\cdot a \equiv 1$ modulo $m$.
(In other words: Given a number $a < m$, find some number $b$
such that $a \cdot b$ gives remainder $1$ when divided by $m$.)

\begin{problem}
Find inverses $1^{-1}$, $3^{-1}$, $5^{-1}$, $7^{-1}$, $9^{-1}$, $11^{-1}$, $13^{-1}$, $15^{-1}$
(all modulo $16$).
\end{problem}

{\bf Chinese Remainder Theorem:}
For multiple mutually prime modulos $m_1,m_2,\ldots,m_k$ one can find $x$ that is
congruent to any numbers $a_1,a_2,\ldots,a_k$ with respect to those modulos.

\begin{problem}
Find a natural number $x$ that is a solution to this system of congruences:
$$\left\{ \begin{array}{l}
x \equiv 1\;(\mathit{mod}\,3)\\
x \equiv 2\;(\mathit{mod}\,5)\\
x \equiv 3\;(\mathit{mod}\,7)
\end{array} \right.$$
\end{problem}

\begin{problem}
Assume that you want to find number $x$ that satisfies both congruences:
$$\left\{ \begin{array}{l}
x \equiv 4\;(\mathit{mod}\,5)\\
x \equiv 6\;(\mathit{mod}\,11)
\end{array} \right.$$
You can solve this "graphically" - build a $5 \times 11$ table representing
all possible pairs of remainders, when you divide numbers by $5$ and by $11$.
Fill in this table by choosing $x=1,2,3,\ldots$ until you find the necessary
combination of remainders $(4;6)$.
\end{problem}

\begin{problem}
Find the smallest positive integer $n$, such that
numbers $\sqrt[5]{5n}$, $\sqrt[6]{6n}$, $\sqrt[7]{7n}$ are all positive integers.\\
(From {\em Vilniaus universiteto Matematikos ir informatikos fakulteto olimpiadas} -
a Lithuanian olympiad for high school students by Vilnius university; 2016,
Grade 10, P3.)
\end{problem}

\begin{problem}
Prove that for each positive integer n, there are pairwise relatively prime integers
$k_0, k_1, \ldots, k_n$ , all strictly greater than $1$,
such that $k_0 k_1 \ldots k_n - 1$ is the product of
two consecutive integers.
\end{problem}

\begin{problem}
Prove that for every positive integer $n$, there exist integers $a$ and $b$ such that $4a^2 + 9b^2 - 1$ is divisible by $n$.\\
({\em Math Prize for Girls Olympiad, 2010, P2}).
\end{problem}

\begin{problem}
Are there infinitely many Fibonacci numbers that give the following remainders when divided by $1001$:\\
{\bf (a)} remainder $0$; {\bf (b)} remainder $900$; {\bf (c)} remainder $1000$.
\end{problem}

\begin{problem}
Prove or disprove the following hypotheses.\\
{\bf (a)} For all $k \geq  2,$ each sequence of $k$ consecutive positive integers contains a number that is not divisible by any prime number less than $k$.\\
{\bf (a)} For all $k\geq 2,$ each sequence of $k$ consecutive positive integers contains a number that is relatively prime to all other members of the sequence.\\
({\em Baltic Way, 2016, P2}).
\end{problem}

\newpage

\section{Hints for Some Examples}

{\bf Hint 2.1:} You can find this by trial and error for small numbers.
Or you can run Euclidean Algorithm to find the GCD (greatest common divisor)
of numbers $18$ and $42$. It will tell you, how many times you should add
or subtract $18$ and $42$ to get number $6$. (Blaninship's method is essentially
the same thing.)

{\bf Hint 2.2:} If you build the following sequence of mutual primes:
$2,3,7,43,\ldots$ (every next number equals the product of all the previous ones plus $1$),
then the corresponding numbers $11$, $111$, $1111111$, and so on will give remainder $1$
every time you divide them one by another.

{\bf Hint 2.3:} In order to find, say, $9^{-1}$ (modulo $16$), you can
try out all the odd remainders ($1,3,\ldots,13,15$). Or you can solve the
Bezout's identity $9x - 16y = 1$. Blankinship's algorithm again.

{\bf Hint 2.4:} One can build such a number step by step -
first write all the numbers congruent to $1$ (modulo $3$):
$1,4,7,\ldots$ until you find one that gives remainder $2$ when
divided by $5$, and so on. (Since $3,5,7$ are mutually prime,
Chinese remainder theorem promises that you will succeed.)

{\bf Hint 2.5:} Solution is shown in the table: \url{https://bit.ly/2MCzMmf}.
Try to locate numbers
$0,1,2,3,\ldots$ in this table and see the sequence how they fill up the table.
Similar ideas are used by problems that ask you to "Measure exactly $4$ liters
of water, given two jugs with volumes $5L$ and $11L$ respectively".

{\bf Hint 2.6:} Search for $n$ in the form $n = 2^a3^b5^c7^d$.
Then write the necessary conditions (as modular congruences) for all
the unknown powers $a,b,c,d$.

{\bf Hint 2.7:} Look at the polynomial $F(t) = t^2 + t + 1$ (it is
a product of two consecutive numbers $t$ and $t+1$ plus $1$).\\
Note that all the remainders it gives, when divided by $2$, by $3$, etc.
are periodic. And if $F(t)$ sometimes is divisible by a prime $p_1$ and
sometimes by a prime $p_2$, then eventually $F(t)$ (for some special
arguments $t$) will be divisible by them both: $p_1 \cdot p_2$.\\
Now, all you need to show that there are infinitely many primes that
sometimes divide the values of $F(t)$. At this point remember
the proof that there are infinitely many primes. Assume that this is not true -
i.e. $F(t)$ is divisible by only finitely many primes. Then plug into
$F(t)=t^2 + t + 1$ the number
$t = p_1\cdot{}p_2\ldots{}\cdot{}p_k+1$ their product plus $1$.

{\bf Hint 2.8:} Use Chinese remainder theorem to avoid looking at {\em all} possible
$n$. Just look at the prime powers $p^k$ (and all the remaining $n$ can be
obtained by combining the solutions for $p^k$ in a certain way).\\
Next, consider two separate cases: $n = 2^k$ (you can now pick $b$
so that it is inverse of $3$ modulo $2^k$ - so that the term $9b^2 - 1$
is congruent to $0$). On the other hand, if $n=p^k$ for some other $p \neq 2$,
then pick $a$ equal to inverse of $2$ modulo $p^k$ for similar reasons.

{\bf Hint 2.9:} The remainders of Fibonacci numbers when divided by any fixed $d$
are periodic (because the pairs of neighboring remainders eventually
start to repeat). What is more interesting: All the remainders of Fibonacci sequence
are "clean periodic" (not just "eventually periodic") - every remainder belongs to
the period. If $F_0 = 0$ (divisible by any $d$), then it means that infinitely
often $F_n$ will be divisible by that $d$.\\
{\bf (a)} is simple - since $F_0$ is divisible by $1001$, then the remainder $0$
is clearly in the period (modulo $1001$).\\
For {\bf (b)} and {\bf (c)} you need to factorize $1001$ as a product of three prime
factors and search for the combinations of remainders (as per Chinese remainder
theorem).

{\bf Hint 2.10:} Statement {\bf (a)} is clearly false. Just start from $2$ and you
will find a sequence, where every member is divisible by some small prime.\\
For {\bf (b)} you need to express some segment of $k$ subsequent numbers as an overlap
of several arithmetic progressions with prime differences $d < k$
(so that every progression contains at least two members among these $k$ subsequent
numbers and all the $k$ numbers are covered at least by one sequence).\\
This is doable when $k=17$. See \url{https://bit.ly/2Q2XASA}. Finally - use
Chinese remainder theorem to find an actual value $N$ such that the numbers
from $N$ to $N+16$ (inclusive) give the remainders you need.

\end{document}


%!TEX root = ../thesis.tex

\chapter{Adaptivity}
\label{adp_sec:main}
\section{Introduction}
\input{adaptivity/intro.tex}

\section{Error indicator}
\label{adap_sec:error_indicator}
\input{adaptivity/error_indicator.tex}

\section{Machine learning}
\input{adaptivity/machine_learning.tex}

\section{Merging triangle}
\label{adap_merge_triangle}
\input{adaptivity/merge_tri.tex}

\section{Matrix representation of NURBS Curves}
\label{adap_sec_mrep2d}
\input{adaptivity/mrep.tex}

\section{Numerical examples}
\input{adaptivity/ex_short_cantilever.tex}
\input{adaptivity/ex_chole_adap.tex}
\input{adaptivity/ex_bracket_adap.tex}

\section{Conclusions}
\paragraph{}
In this chapter, the machine learning algorithm is adopted to develop an extensible and flexible error indicator.
Any other error estimators can be added to the existing framework and their effects can be detected based on the performance indicators in machine learning.
A MLP trained error estimator that concludes expressions related to the eigenvalues of the SBFEM formulation and some key geometric properties of the scaled boundary finite element gives a higher convergence rate compared to the uniform refinement.
In order to improve the learning effectiveness of the MLP, regularization methods including bagging and dropout are utilized.
Due to the lack of the eigenvalue error indicator in the first order triangular element, method that eliminates these situation is developed.
A matrix representation of the NURBS curves is presented to achieve a higher efficiency and stability in calculating the intersections between edge of the element and the NURBS curve.
Stress analysis is conducted on 2D linear elasticity problems.


\documentclass[main.tex]{subfiles}
\begin{document}

\section{Electron scattering}

\subsection{Compton scattering onto an electron at rest}

\marginpar{Sunday\\ 2020-8-23, \\ compiled \\ \today}

We need to account for the fact that light has a quantum nature, which is not addressed in the classical treatment of scattering.
Also, now we will account for the momentum of the photon.

We start of with an electron at rest, and a photon with energy \(h \nu \) and momentum \(h \nu / c\) impinging on it.
After the scattering, the photon will have energy \(h \nu '\) and momentum \(h \nu ' / c\), while the electron will have momentum \(m v \gamma \).
Let us call the angle between the direction of the incoming photon and the direction of the outgoing one \(\theta \).

In terms of four-vectors, we can express the momenta of the photon before and after as
%
\begin{align}
k^{\mu } = \frac{\epsilon}{c} \left[\begin{array}{c}
1 \\
\vec{\Omega}
\end{array}\right]
\qquad \text{and} \qquad
k^{\prime \mu } = \frac{\epsilon'}{c} \left[\begin{array}{c}
1 \\
\vec{\Omega}'
\end{array}\right]
\,,
\end{align}
%
where \(\vec{\Omega}\) and \(\vec{\Omega}'\) are unit vectors defining the propagation directions, such that \(\vec{\Omega} \cdot \vec{\Omega}' = \cos \theta \).
On the other hand, the momenta of the electron will be
%
\begin{align}
p^{\mu }  = \left[\begin{array}{c}
mc \\
\vec{0}
\end{array}\right]
\qquad \text{and} \qquad
p^{\prime \mu } = \gamma \left[\begin{array}{c}
mc \\
m \vec{v}
\end{array}\right]
\,.
\end{align}

Since the particles are unchanged after the scattering, both the incoming and outgoing momenta must satisfy \(p^{\mu } p_{\mu } = - m^2 c^2\) and \(k^{\mu } k_{\mu } = 0\) (in any frame: they are Lorentz scalars).
Because of momentum conservation, we can also impose the four equations
%
\begin{align}
p^{\mu } + k^{\mu } = p^{\prime \mu } + k^{\prime \mu }
\,.
\end{align}

Solving the system of equations yields
%
\begin{align}
\epsilon' = \frac{\epsilon }{1 + \frac{\epsilon }{mc^2} \qty(1 - \vec{\Omega} \cdot \vec{\Omega}')}
\,,
\end{align}
%
implying that we must have \(\epsilon' \leq \epsilon \): the photon will lose energy in the scattering.
The calculation which yields the differential cross section of the scattering is quite complicated and requires the full machinery of QED; here we just give the result:
%
\begin{align}
\frac{ \dd{\sigma }  }{ \dd{\vec{\Omega} }'}
= \frac{r_0^2}{2 } \qty(\frac{\epsilon '}{\epsilon })^2
\qty[ \frac{\epsilon}{\epsilon '} + \frac{\epsilon'}{\epsilon } - (1 - \vec{\Omega} \cdot \vec{\Omega}')^2]
\,,
\end{align}
%
where \(\epsilon \) is the energy divided by \(m c^2\) and \(r_0 \) is the classical electron radius.
\todo[inline]{Missing square for the \(\sin \theta \) term in the KN cross section! see \cite[eq.\ 7.4]{rybickiRadiativeProcessesAstrophysics1979}.}

If we substitute in our formula for the energy, using \(\xi = \cos \theta = \vec{\Omega} \cdot \vec{\Omega}'\), we find
%
\begin{align}
\frac{ \dd{\sigma }}{ \dd{\Omega }' \dd{\epsilon }'}
= \frac{r_0^2}{2} \frac{1 + \xi^2}{\qty(1 + \epsilon (1 - \xi ))^2}
\qty[1 + \frac{\epsilon^2 (1 - \xi )^2}{(1 + \xi )^2 (1 + \epsilon (1 - \xi ))}]
\delta \qty(\epsilon ' - \frac{\epsilon }{1 + \epsilon (1 - \xi )})
\,.
\end{align}

We have inserted a \(\dd{\epsilon '}\) differential for the outgoing photon energy, we are considering a density in one more variable but it is singular there, whence the \(\delta \) function, which describes the distribution in \(\epsilon '\) space --- concentrated at the only value allowed by 4-momentum conservation.
Note that this makes sense dimensionally, as the dimension of a \(\delta \) function is the inverse of the dimension of its argument.

In the low energy limit, \(h \nu \ll m_e c^2 \) or \(\epsilon \to 0\), we find
%
\begin{align}
\frac{ \dd{\sigma }}{ \dd{\Omega }' \dd{\epsilon }'}
=
\frac{r_0^2}{2} (1 + \xi^2) \delta (\epsilon ' - \epsilon )
\,,
\end{align}
%
which is the Thomson cross section.

Let us now define the \textbf{Compton scattering kernel} \(\sigma \): it is the differential cross section times the electron density,
%
\begin{align}
\sigma (\epsilon \to \epsilon ', \xi ) = n_e \frac{ \dd{\sigma }}{ \dd{\Omega }' \dd{\epsilon }'}
\,.
\end{align}

We can integrate this in order to find the total cross section presented by the electrons to photons of an energy \(\epsilon \):
%
\begin{align}
\sigma (\epsilon ) &= \int \dd{\Omega '} \dd{\epsilon '} \sigma (\epsilon \to \epsilon ', \xi ) \\
&= \frac{3}{4} n_e \sigma_{T} \qty[ \qty(\frac{1 + \epsilon }{\epsilon^3})
\qty(\frac{2 \epsilon (1 + \epsilon )}{1 + 2 \epsilon }- \log \qty(1 + 2 \epsilon ))
+
\frac{1}{2 \epsilon } \log \qty(1 + 2 \epsilon )
-
\frac{1 + 3 \epsilon }{(1 + 2 \epsilon )^2}
]
\,.
\end{align}

How does this differ from the Thomson cross section? For \(\log \epsilon \lesssim -1 \) we have \(\sigma \approx \sigma_T\), while as \(\epsilon \) increases the cross section goes to zero.

\begin{figure}[ht]
\centering
\includegraphics[width=\textwidth]{figures/compton-sigma.pdf}
\caption{Compton cross section as a function of \(\epsilon \), photon energy by \(m_e c^2\). }
\label{fig:compton-sigma}
\end{figure}

In the low energy limit we have the expansion
%
\begin{align}
\sigma (\epsilon )\approx \sigma_T \qty(1 -2 \epsilon + \frac{26}{5} \epsilon^2)
\,.
\end{align}

The introduction of these nonconservative aspects complicates the radiative transfer equation for scattering.
The absorption term is
%
\begin{align}
- \alpha _\nu ^{(s)} I _\nu =
- I (\epsilon , \Omega ) n_e \int \dd{\Omega }' \dd{\epsilon }' \frac{ \dd{\sigma }}{ \dd{\Omega }' \dd{\epsilon }' } =     - I (\epsilon , \Omega  ) \sigma (\epsilon )
\,,
\end{align}
%
while for the emission term the intensity must go inside the integral, so we have
%
\begin{align}
j_\nu^{(s)} =
n_e \int \dd{\Omega }' \dd{\epsilon '} I(\epsilon ', \Omega ') \frac{ \dd{\sigma }}{ \dd{\Omega }' \dd{\epsilon }' } = \int \dd{\Omega }' \dd{\epsilon }'
I(\epsilon ', \Omega ')
 \sigma (\epsilon \to \epsilon ', \xi )
\,,
\end{align}
%
which cannot be expressed in terms of the integrated kernel \(\sigma (\epsilon )\).

For Thomson scattering the absorption term is similar, with \(n_e \sigma_T\) instead of \(\sigma (\epsilon )\).
The emission term, on the other hand, can now be evaluated to yield
%
\begin{align}
\int \dd{\Omega }' \dd{\epsilon }' \sigma (\epsilon \to \epsilon ', \xi )
= \frac{r_0^2}{2} n_e \int \dd{\Omega '} \dd{\epsilon '} I(\epsilon ', \Omega ') (1 + \xi^2) \delta (\epsilon ' - \epsilon ) \sim n_e \sigma_T J(\epsilon )
\,,
\end{align}
%
as long as we neglect the angular dependence of the intensity.

We are always restricting ourselves to electrons which are initially at rest. For them Thomson scattering is a good approximation; photons more energetic than a few tens of \SI{}{keV} (hard X-rays) hardly scatter, since the Klein-Nishina cross section drops at high energies.

So, by using the Thomson limit we do not get it wrong by much.

The main conclusions we can draw from this is that we expect no spectral modifications (since Thomson scattering is conservative) but we do expect angular redistribution (since Thomson scattering is essentially isotropic).

\subsection{Scattering in plasmas}

Electrons are not at rest in any realistic astrophysical setting. In order to have free electrons we need a plasma, which is made of (at least partially) ionized gas.
So, we need a way to describe the fractional ionization.
Let us consider only hydrogen for simplicity: we define the collisional ionization fraction \(x\) (ionized atoms divided by total atoms), which can be expressed in terms of the temperature as
%
\begin{align}
x = \frac{F}{1 + F}
\qquad \text{where} \qquad
F = 2T \exp(-\frac{\SI{1.58e5}{K}}{T})
\,.
\end{align}

The way \(x\) looks as a function of temperature is a kind of sigmoid: it is close to zero, then at \(T\) around \SI{e4}{K} it quickly rises, and becomes close to 1 at a few times \SI{e4}{K}. At temperatures larger than \SI{e5}{K} the plasma is basically fully ionized (which makes sense: the first ionization energy of hydrogen is \(\SI{13.6}{eV} \approx \SI{1.6e5}{K}\)).

At these temperatures, the electrons will be moving quite a lot because of thermal motion.

\subsection{Inverse Compton scattering}

Scattering onto moving electrons is often called \textbf{inverse Compton scattering}, since in this case the photon might gain energy instead of losing it.
We start off with a photon with momentum \(k^{\mu } = (\epsilon /c) (1, \vec{\Omega})\) and an electron with momentum \(p^{\mu } = \gamma (mc, m \vec{v})\); the outgoing electron and momentum momenta, energies and velocities will be denoted with a prime.

We know how to deal with scattering off a stationary electron, and stationarity is relative: if we boost to the electron's rest frame we can apply the results from regular Compton scattering, and then we will need to boost back to our frame.

We shall denote quantities calculated in the ERF (electron rest frame) with a pedix \(e\): for example the energy of the photon in the ERF before the scattering will be \(\epsilon_e\).
This means that in the ERF we will have the equality
%
\begin{align}
\epsilon_e'
= \frac{\epsilon_e}{1 + \frac{\epsilon_e}{mc^2} (1 - \cos \Theta )}
\approx \epsilon_e \qty(1 - \frac{\epsilon_e}{mc^2} (1 - \cos \Theta ))
\approx \epsilon_e
\qquad \text{if } \epsilon_e \ll m c^2
\,.
\end{align}

We are saying that in the rest frame the scattering will basically be conservative, this is for sure an approximation but, because of what we discussed earlier about the Klein-Nishina cross section dropping off at high energies, not a large one.

The Lorentz transform from the LAB frame to the ERF for the energy of the photon is:
%
\begin{align}
\epsilon_e = \gamma \epsilon \qty(1 - \beta \cos \theta )
\,,
\end{align}
%
while the transform back from  the ERF to the LAB frame is
%
\begin{align}
\epsilon' = \gamma \epsilon_e' \qty(1 + \beta \cos \theta _e )
\,.
\end{align}

Note that the quantities \(\beta \) and \(\gamma \) refer to the velocity of the electron: since we are approximating the scattering as conservative in the ERF calculating them before or after the scattering is the same (the direction of the electron can change: this is accounted for by letting \(\theta\) and \(\theta '\) be different).
So, we can just insert these two multiplicative factors one after another to find the total change in energy of the photon:
%
\begin{align}
\epsilon ' &\approx \gamma^2 \epsilon \qty(1 - \beta \cos \theta ) \qty(1 + \beta \cos \theta'_e )
\,.
\end{align}

\todo[inline]{Inaccuracy in the slides: the first formula does not really make sense, since the approximation of the nonconservativeness factor being equal to one has been made in part of it (\(\beta \) and \(\gamma \) being the same before and after), so it does not make sense to write it out.}

Let us neglect the angular terms for now: generally they are of order unity. Instead, the main factor in the formula is \(\gamma^2>1\): this means that in general the energy of the scattered photon is of the order \(\epsilon ' = \gamma^2 \epsilon > \epsilon  \).
The boost of the energy of the photon depends on how relativistic the electron is; if the electron is very relativistic the boost in energy for the photon can be quite large.

In order to make predictions about the effect of this for a population of electrons and photons, we can Lorentz transform the Compton Scattering Kernel: the final result of the manipulation is
%
\begin{align}
\sigma (\epsilon \to \epsilon ', \xi ) = \frac{D}{D'} \sigma_e \qty(\epsilon _e \to \epsilon _e', \xi _e)
\,,
\end{align}
%
where \(D = 1 - \vec{\Omega} \cdot \vec{v} / c = 1 - \beta \cos \theta \) and \(D' = 1 - \vec{\Omega}' \cdot \vec{v}' / c \approx 1 + \beta \cos \theta '_e\) are the factors in the Lorentz transforms, which allow us to write \(\epsilon _e = \gamma D \epsilon \) and \(\epsilon _e' = \gamma D' \epsilon '\).

The transformation law for \(\xi = \vec{\Omega} \cdot \vec{\Omega}\) is
%
\begin{align}
1 - \xi = \frac{1 - \xi _e}{\gamma^2 D D'}
\,,
\end{align}
%
while the electron energy density transforms as \(n = \gamma n_e\).

This allows us to write down an explicit expression for \(\sigma (\epsilon \to \epsilon ', \vec{\Omega}, \vec{\Omega}')\) in the lab frame for a population of single-speed electrons:
%
\begin{align}
\sigma (\epsilon \to \epsilon ', \vec{\Omega}, \vec{\Omega}', v)
=
\frac{n r_0^2}{2 \epsilon \nu \gamma }
\qty[1 + \qty(1 + \frac{1 - \xi }{\gamma^2 D D'})^2
+ \frac{\epsilon \epsilon ' (1 - \xi )^2}{\gamma^2 D D'}]
\delta \qty(\xi -1 + \frac{\gamma D}{\epsilon'} - \frac{\gamma D'}{\epsilon })
\,,
\end{align}
%
but we must also consider the fact that the electrons are distributed with different velocities: if their distribution is isotropic, so that \(\dd{n} = n f(v) \dd[3]{v}\) then we can integrate across all velocity space, getting an expression like
%
\begin{align}
\sigma (\epsilon \to \epsilon ', \xi )
= \int \dd{v} \sin \theta_v \dd{\theta _v} \dd{\phi _v}
 v^2 n f(v)
\sigma (\epsilon \to \epsilon ', \vec{\Omega}, \vec{\Omega}', v)
\,.
\end{align}

\end{document}


\chapter{Related work}\label{relatedwork}
State-space dimensionality reduction has been a topic of interest in RL for several years. In their 2015 paper, Curran et al. \cite{mario} used PCA to reduce the dimensionality of the state-space in a Super Mario environment. They found that with the right number of principal components, an agent using PCA was able to converge to a better policy and needed less episodes than an agent using the full observation. Curran et al. corrobarated these results in more environments in later research in 2016 \cite{pca_curran}. Similar results were found by Shah et al. in 2011 in the context of neuroscience \cite{pca_neural}. Bitzer et al. also found similar results in 2010. \cite{pca_bitzer}. Reducing the dimensionality using PCA led to better policies and quicker convergences; this was interpreted as ``[...] due to the higher dimensionality, learning in the full 4D state space requires far more exploration to cover the same proportion of space" \cite{pca_bitzer}.

Unlike the Starcraft II environment used in our research having image/grid based observations, the states and observations used in these papers are comprised of non-spatial related variables, i.e. linear states; to, for instance, denote the presence of one or more enemies within one grid-cell distance of Mario's position, Curran et al. use one variable with $2^8$ possible values ($0$-$255$). Hence we are using PCA in a different setting, with an observation format more commonly used in modern RL. Furthermore they use the Q-learning algorithm (mentioned in section \ref{pl-dqn}) which is not a state-of-the-art learning algorithm anymore and handles larger state-spaces badly due to using a lookup-table instead of a neural network for the Q-function.

Research has also been done with regards to using autoencoders for state-space-dimensionality reduction in RL. Lange and Riedmiller in 2010 \cite{AE_2010} used visual data as observations and found that the agent was able to find an optimal policy using lower dimensional data from the autoencoder. They did not compare the performance of this agent with any other agent. Furthermore, their system only has $31$ possible states, which is very limited; in contrast, our Starcaft II environment has a total of $731.187$ possible states.

In 2016, Finn et al. successfully showed the possibility of training an RL agent on a latent representation given by an autoencoder on visual information \cite{ae_visual}. The same year, Van Hoof et al. \cite{AE_2016} used an autoencoder to project noisy sensor data unto a lower dimensional space. They found that the agent using the autoencoder was far better able to find a good policy than the agent using full observations. This was explained as the agent being too sensitive to noisy data, hence being unable to train even decently. Our research in contrast explores an environment where the baseline agent is able to train to a good policy.

Ha and Schmidhuber use a variational autoencoder (VAE) \cite{vae} in their 2018 paper to train an agent in image based environments \cite{rl_vae}. Through this method, their agent outperformed baseline agents. They did find that the VAE produced mixed results for the latent representation; in an environment where details are important, they were lost in the latent representation. Differently from our research, they do not compare different reduction methods. Furthermore they do not use deep reinforcement learning, but train the agent using \textit{evolution strategies} \cite{es}. Lee et al. also train an agent on a variational autoencoder and found similar results \cite{rl_vaetwo}.

Khan et al. used an autoencoder to project visual data to a latent space and were able to train a self-driving car in the CARLA environment \cite{rl_carla}\cite{carla}. However, no comparison to a baseline agent (or other reduction methods) was given.

In 2019, Prakash et al. \cite{AE_2019} also used an autoencoder on visual data to project the observation data onto a lower dimensional space. They found that an agent using the autoencoder far outperformed the baseline agent. The baseline agent did not find a decent policy, though the authors claim that with enough episodes it would have.

Another paper in 2019 by Gelada et al. \cite{deepmdp} compared using a DeepMDP with using an autoencoder for state-space dimensionality reduction. They found that in simpler environments a DeepMDP can find better representations on lower dimensionality space than an autoencoder. Simultaneously though, they also find that in more complex environments, like Atari games, DeepMDPs can have trouble finding a good representation and that its loss can be difficult to optimize. They also find that a DeepMDP agent generally outperforms a baseline agent.

%project_matrix_2020
%dimred_overview_2008


\section{Checking Property Preservation}
\label{sec:lts-transformation:proppres}

In this section, we show how property preservation of transformations can be checked by generating networks from rule systems and comparing the LTSs of these networks.
Figure~\ref{fig:lts-transformation:check-generation} gives an overview of the approach.

\begin{figure}[hbt]
\centering
\includegraphics[scale=0.6]{lts-transformation/figs/check-generation}
\caption{Checking property preservation by comparing LTSs}
\label{fig:lts-transformation:check-generation}
\end{figure}

First, networks are generated based on the left and right-hand sides of transformation rules.
Then, the network LTSs corresponding to these networks are generated, while applying maximal hiding regarding the property at hand.
Finally, property preservation is checked by comparing the network LTSs generated from left-hand sides of transformation rules with the network LTSs of the corresponding right-hand sides.
If all pairs of LTSs are divergence-sensitive branching bisimilar, the rule system preserves the property at hand.

More formally, a terminating, confluent rule system $\Sigma$ is $\Sf$-preserving for a property~$\Sf \in \dsbrLmu$ iff $\models_{\smodel} \Sf \iff \models_{T_{\Sigma} (\smodel)} \Sf$ for all networks~$\smodel$.
Thus, if $\Sigma$ is $\Sf$-preserving and $\models_{\smodel} \Sf$, we can conclude that~$\models_{T_{\Sigma} (\smodel)} \Sf$ without rechecking property~$\Sf$ for network~$T_{\Sigma} (\smodel)$.
Since $\dsbrLmu$ is compatible with maximal hiding and DSBB, as discussed in Section~\ref{subsec:lts-transformation:lmu}, $\Sigma$ is $\Sf$-preserving if $\maxabstr(\smodel) \dsbbis \maxabstr(T_{\Sigma} (\smodel))$.
In this section, we discuss under which conditions a rule system implies the bisimilarity of $\maxabstr(\smodel)$ and $\maxabstr(T_{\Sigma} (\smodel))$.
The most important condition roughly boils down to checking whether, after some appropriate rewriting, the left and right patterns of the transformation rules are divergence-sensitive branching bisimilar after maximal hiding.
If this is the case, then applying the rules does not result in a network with structurally different behavior.

Without loss of generality, for each network~$\smodel = \networktuple$ and rule system~$\Sigma = \rulesystemtuple$, we assume that each $r \in R$ has exactly one match to some $\Pi[i]$ and that each $\Pi[i]$ is matched on by exactly one $r$.
This is expressed by indexing the~$r \in R$ such that rule~$r_i$ is matched on~$\Pi[i]$.
If~$R$ contains only one rule, we omit its index.
Since~$\Sigma$ is confluent, the results of this section can be lifted to the more general case where rules may have an arbitrary number of matches.
With this assumption, it can also safely be assumed that all the $\actions{i}$ are disjoint.
If this is not the case, some renaming of actions and a corresponding modification of the synchronization rules can resolve this.

\subsection{Extended Transformation Rules}
To show that a rule system~$\Sigma$ preserves a property~$\Sf$ for every network~\smodel, we show that a divergence-sensitive branching bisimulation between the states of~$\maxabstr(\smodel)$ and~$\maxabstr(T_{\Sigma} (\smodel))$ can be constructed based on the divergence-sensitive branching bisimulations between networks constructed from the transformation rules.
To construct these networks from the transformation rules, we extend the transformation rules with self-loops on the glue-states.
The matches of the glue-states of a given transformation rule may be part of transitions that are not present in the patterns of this rule.
The transformation rules are extended to make explicit that such transitions may exist.
Each self-loop is labeled with an action uniquely related to the corresponding state.

\begin{definition}
\label{def:lts-transformation:extendedtransformationrule}
Given an LTS transformation rule $r = \transruletuple{r}$, the corresponding extended transformation rule is defined as~$r_\kappa = \langle \Llts^r_\kappa, \Rlts^r_\kappa \rangle$, where
\begin{itemize}
\item
$\Llts^r_\kappa = \langle \states{\Llts^r}, \actions{\Llts^r} \cup \{ \kappa_s \mid s \in \states{\Llts^r} \cap \states{\Rlts^r} \}, \transitions{\Llts^r} \cup \{ \langle s, \kappa_s, s \rangle \mid s \in \states{\Llts^r} \cap \states{\Rlts^r} \}, \initialstates{\Llts^r} \rangle$;
\item
$\Rlts^r_\kappa = \langle \states{\Rlts^r}, \actions{\Rlts^r} \cup \{ \kappa_s \mid s \in \states{\Llts^r} \cap \states{\Rlts^r} \}, \transitions{\Rlts^r} \cup \{ \langle s, \kappa_s, s \rangle \mid s \in \states{\Llts^r} \cap \states{\Rlts^r} \}, \initialstates{\Rlts^r} \rangle$.
\end{itemize}
\end{definition}

\noindent
We assume that the $\kappa$-actions are not originally in $\actions{\Llts^r}$.
Without these loops, a DSBB check of patterns could consider two deadlock states to be bisimilar, while they are actually different glue-states that are possibly matched on states with different outgoing transitions not present in the patterns.
For example, without the dashed $\kappa$-loops, states~$\it{ii}$ and~$\it{iii}$ in Figure~\ref{fig:lts-transformation:kappa} would be related if~$a,b \in h_{\actions{}}(\Sf)$.
With the $\kappa$-loops, however, they are not related, as indicated by the cross.
Thus, the extra transitions ensure that $\Llts^r_\kappa \dsbbis \Rlts^r_\kappa$ iff there exists a divergence-sensitive branching bisimulation that relates all the glue-states to themselves.
A glue-state $s$ in $\Llts^r_\kappa$ (or $\Rlts^r_\kappa$) with self-loop $s\ \smash{\xrightarrow{\kappa_s}}\ s$ must at least be related to itself in $\Rlts^r_\kappa$ (or $\Llts^r_\kappa$) since it is the only state where a $\kappa_s$-transition is enabled.

\begin{figure}[hbt]
\centering
\includegraphics[scale=0.2]{lts-transformation/figs/kappa-loops}
\caption{An extended transformation rule}
\label{fig:lts-transformation:kappa}
\end{figure}

\subsection{Synchronizing Behavior}
If a rule system consists of multiple transformation rules, then multiple LTS transformations can be applied in a single transformation step.
To check $\Sf$-preservation of such rule systems, we need to take possible synchronization between different rule patterns into~account.

\begin{figure}[hbt]
\centering
\includegraphics[scale=0.2]{lts-transformation/figs/comparison-network}
\caption{A network and the corresponding network LTS}
\label{fig:lts-transformation:comparison-network}
\end{figure}

In Figure~\ref{fig:lts-transformation:comparison-network}, a network involving synchronization and the corresponding network LTS are shown.
The network LTS is obtained after hiding the actions in~$h$, shown in the middle of the figure, which is the hiding set for some property~$\Sf$.
The rule system shown on the left of Figure~\ref{fig:lts-transformation:preserving-rules} can be applied to the network of Figure~\ref{fig:lts-transformation:comparison-network}.
Individually, the rules seem to fundamentally change the behavior of the process LTSs, as shown in the middle of Figure~\ref{fig:lts-transformation:preserving-rules}.
However, since the rule system also adds the new synchronization rules~$\langle\langle c1, e1 \rangle, c1e1\rangle$ and~$\langle\langle c2, \bullet \rangle, c2\rangle$, and the actions~$\it{c1e1}$ and~$\it{c2}$ can be hidden,
the final network LTS, as shown on the right of the figure, is bisimilar to the one before transformation.
To incorporate such possible dependencies between rule patterns, we developed a $\Sf$-preservation check involving networks of rule patterns.

\begin{figure}[hbt]
\centering
\includegraphics[scale=0.2]{lts-transformation/figs/preserving-rules}
\caption{A rule system and an example of its application}
\label{fig:lts-transformation:preserving-rules}
\end{figure}

In general, when considering transformation rules that affect synchronizing actions and thus involve multiple process LTSs, it cannot be determined whether a given rule system~$\Sigma = \rulesystemtuple$ involving such rules is $\Sf$-preserving by just analyzing the $\Llts^\ri$ and $\Rlts^\ri$ of all $\ri \in R$.
However, this can be done if $\Sigma$ has a number of properties regarding synchronizing behavior of a network $\smodel$, which we together call synchronization uniformity.

Before we can give a definition of synchronization uniformity, we need a number of auxiliary definitions.
The set of actions involved in synchronization vector~$\vectornot{t}$ is ${\actions{}}(\vectornot{t}) = \{ a \mid \exists i \in 1..n. \vectornot{t}[i] = a \wedge a \neq \bullet \}$,
and the set of actions involved in synchronization rules in~\synchrules with multiple processes is $\syncactions (\synchrules) = \{a \mid \exists \langle \vectornot{t},a' \rangle \in \synchrules. a \in {\actions{}}(\vectornot{t}) \wedge |{\it Ac}(\vectornot{t})|>1 \}$.
The set of indices of process LTSs that can potentially synchronize with behavior in $\Llts^\ri$ according to a synchronization rule in \synchrules is $\inv(\Llts^\ri, \synchrules) = \bigcup \{ {\it Ac}(\vectornot{t}) \mid \syncruletuple \in \synchrules \wedge \vectornot{t}[i] \in \actions{\Llts^\ri} \}$.
This definition states that $j$ is in $\inv(\Llts^\ri, \synchrules)$ iff there exists a synchronization rule $\syncruletuple$ in $\synchrules$ such that both $i \neq \bullet$ and $j \neq \bullet$, i.e.\ both $i$ and $j$ are active for that rule, and the behavior in $\Pi[i]$ is matched on by transformation rule $\ri = \transruletuple{r_i}$.
The set of actions of process $j$ on which the actions in~$\Llts^\ri$ depend according to the synchronization rules in~\synchrules is $\dep^{\Llts^\ri}(j, \synchrules) = \{ \vectornot{t}[j] \mid \syncruletuple \in \synchrules \wedge \vectornot{t}[i] \in \actions{\Llts^\ri} \wedge \vectornot{t}[j] \neq \bullet \}$.
In other words, the set of all actions $\bigcup_{t \in F} \{\vectornot{t}[j] \} \setminus \{\bullet\}$ constitutes $\dep^{\Llts^\ri}(j, \synchrules)$, where~$F$ represents the set of all synchronization rules applicable on $\Llts^\ri$.
Sets $\inv(\Rlts^\ri, \synchrules)$ and $\dep^{\Rlts^\ri}(j, \synchrules)$ are defined similarly.
Now, synchronization uniformity can be defined.

\begin{definition}
\label{def:lts-transformation:uniformity}
We say that rule system~$\Sigma = \rulesystemtuple$ is synchronization uniform w.r.t.\ network~$\smodel = \networktuple$ iff the following holds.
\begin{enumerate}
\item
$\forall a \in \actions{s} (\synchrules). (\exists \ri \in R. a \in \actions{\Llts^\ri}) \implies \forall s_1\xrightarrow{a}_{i} s_2 . s_1, s_2 \in m_\ri(\states{\Llts^\ri})$;
\item
$\forall \ri \in R, j \in \inv(\Llts^\ri, \synchrules). \dep^{\Llts^\ri}(j, \synchrules) \subseteq \actions{\Llts^\rj}$;
\item
$\forall \langle\vectornot{t},a\rangle \in \hat\synchrules, i \in 1..n. \vectornot{t}[i] = \bullet \vee \vectornot{t}[i] \in \actions{\Rlts^\ri}$.
\end{enumerate}
\end{definition}

The first condition states that if a transformation rule is applicable to a synchronizing transition, then it is applicable to all synchronizing transitions with the same label in~\smodel.
If this is not guaranteed, it becomes very hard to reason about the model after transformation because it is difficult to determine a priori exactly which transitions in different process LTSs will be able to synchronize in the network.
Therefore, predicting the effect of rewriting, for example, $a$-transitions in some places while keeping other $a$-transitions the same is as difficult.
Checking this condition requires inspecting the process LTSs of a network, unless we impose an additional restriction on rule systems.
If we require that all left-hand patterns of rules that modify synchronizing transitions consist of a single transition, the first condition holds, regardless of the structure of the process LTSs of~\smodel.
The second condition states that all actions that can synchronize with $\Llts^\ri$ are also transformed by $\Sigma$.
If this does not hold, it becomes hard to analyze the synchronizing behavior as appearing in transformation patterns.
In such cases, some of the behavior that is relevant for this analysis is not present in any of the patterns, which makes analysis based only on the rule system impossible.
Finally, the third condition states that each new synchronization rule~$\langle\vectornot{t},a\rangle \in \hat\synchrules$ involves actions from the $\Rlts^\ri$ of the corresponding rule~$\ri$ only.
It is crucial to rule out the possibility of transforming merely by introducing synchronization rules, because this also prevents analysis solely based on rule systems.
For example, if we define a new synchronization rule involving existing actions~$a$ and~$b$, and these actions were previously not allowed to synchronize, then we clearly change the model without actually transforming anything.

In the remainder of this chapter, we only consider rule systems that are synchronization uniform regarding a given model.
This may seem a big assumption, but in practice, one tends to transform synchronizing behavior in a uniform way.
Usually, synchronizing actions, say~$a$ and~$b$, represent communication.
If one wants to transform this behavior, it is natural to do this consistently in all places where~$a$ and~$b$ occur, and to transform the behavior of both communicating parties to keep them compatible with each other.

\subsection{Networks of transformation rules}

From a rule system, networks of transformation rules can be constructed.

\begin{definition}
\label{def:lts-transformation:rule-network}
For a model~$\smodel = \networktuple$, rule system~$\Sigma = \rulesystemtuple$, and a rule~$\ri \in R$, the vector~$\xi^\ri$ of transformation rules relevant for the behavior in $\Llts^\ri$ is defined as follows, for all $j \in 1..n$.

\[
\xi^\ri[j] =
\left\{
\begin{array}{ll}
\ast \mapsto \ast & {\rm if} ~ j \not\in \inv(\Llts^\ri, \synchrules) \\
r_{j,\kappa}	  & {\rm if} ~ j \in \inv(\Llts^\ri, \synchrules),
\end{array}
\right.
\]
\end{definition}

\noindent
where $\ast$ is a dummy state.
For a given vector $\xi^\ri$, $\xi_{\Llts}^\ri$ is the vector of left patterns of the extended transformation rules in~$\xi^\ri$, and $\xi_{\Rlts}^\ri$ is the vector of right patterns.
The networks~$\Xi_\Llts^\ri = (\xi_\Llts^\ri, \synchrules)$ and $\Xi_\Rlts^\ri = (\xi_\Rlts^\ri, \synchrules \cup \hat\synchrules)$ allow comparing synchronizing behavior in rule patterns, before and after transformation according to $\Sigma$, in particular involving~$\ri$.

\begin{figure}[hbt]
\centering
\includegraphics[scale=0.2]{lts-transformation/figs/non-preserving-rules}
\caption{A non-preserving transformation and an example of its application}
\label{fig:lts-transformation:non-preserving-trafo}
\end{figure}

On the left of Figure~\ref{fig:lts-transformation:non-preserving-trafo}, another example of a rule system is shown.
In general, this rule system is not $\Sf$-preserving.
Applying this rule system to the network in Figure~\ref{fig:lts-transformation:comparison-network} results in the network shown in the middle of Figure~\ref{fig:lts-transformation:non-preserving-trafo}.
The corresponding network LTS shown on the right of this figure is obtained after hiding the actions in~$h$.
The networks of the left and right patterns of the two transformation rules in this figure are shown in Figure~\ref{fig:lts-transformation:subsets-ltss:middle}.
Actions in~$h$ are hidden.
The dotted lines in this figure illustrate that a divergence-sensitive branching bisimulation exists for these two networks.

%%%\begin{figure}[hbt]
%%%\centering
%%%\includegraphics[scale=0.2]{lts-transformation/figs/subsets-ltss}
%%%\caption{Network LTSs of networks of transformation rules}
%%%\label{fig:lts-transformation:subsets-ltss}
%%%\end{figure}

\begin{figure}[hbt]
  \hfill
  \begin{subfigure}[b]{102pt}
    \centering
    \includegraphics[scale=0.2]{lts-transformation/figs/subsets-ltss-left}
    \caption{}
    \label{fig:lts-transformation:subsets-ltss:left}
  \end{subfigure}
  \hfill
  \begin{subfigure}[b]{106pt}
    \centering
    \includegraphics[scale=0.2]{lts-transformation/figs/subsets-ltss-middle}
    \caption{}
    \label{fig:lts-transformation:subsets-ltss:middle}
  \end{subfigure}
  \hfill
  \begin{subfigure}[b]{102pt}
    \centering
    \includegraphics[scale=0.2]{lts-transformation/figs/subsets-ltss-right}
    \caption{}
    \label{fig:lts-transformation:subsets-ltss:right}
  \end{subfigure}
  \hfill
  \caption{Network LTSs of networks of transformation rules}
\end{figure}

Even though a divergence-sensitive branching bisimulation exists between the networks of the left and right patterns of the transformation rules in Figure~\ref{fig:lts-transformation:non-preserving-trafo}, the rule system is not \Sf-preserving.
This clearly shows that it is not sufficient to only take successful synchronization in account.
Instead, we also need to consider situations in which some parties are able to perform a synchronizing action whereas at least one other party involved in the synchronization is not.
For example, the state labeled (1 3) in Figure~\ref{fig:lts-transformation:comparison-network} has a $\tau$-loop that cannot be simulated by the network LTS in Figure~\ref{fig:lts-transformation:non-preserving-trafo}.
This $\tau$-loop is the result of hiding the $b$-loop of the leftmost process of Figure~\ref{fig:lts-transformation:comparison-network}.
This process can perform action~$b$ independently.
After transformation, however, a $\tau$-cycle can only result from interaction between the transformed process LTSs shown in the middle of Figure~\ref{fig:lts-transformation:non-preserving-trafo}.
In situations where both processes are able to interact successfully, an infinite number of $\tau$-actions can be performed, as shown in Figure~\ref{fig:lts-transformation:subsets-ltss:middle}.
However, if the required synchronization between processes is impossible, as is the case in the state labeled (1 3) in Figure~\ref{fig:lts-transformation:non-preserving-trafo}, only one $\tau$-action can be performed.

To be able to consider such scenarios, we define a projection operator on networks of transformation rules.

\begin{definition}
\label{def:lts-transformation:projection}
For each vector of transformation rules~$\xi^\ri$ and~$j \in 1..n$, the projection operator $/I$ ($I \subseteq 1..n$) is defined as follows.
\[
\xi^\ri/I[j] = \left\{
\begin{array}{ll}
\xi^\ri[j]                    & {\rm if} ~ j \in I \\
\ast \mapsto \ast & {\rm otherwise}
\end{array}
\right.
\]
\end{definition}

\noindent
This operator can similarly be applied on the vectors of patterns~$\xi_{\Llts}^\ri$ and~$\xi_{\Rlts}^\ri$, and we say that $\Xi_\Llts^\ri/I = (\xi_\Llts^\ri/I, \synchrules)$ and $\Xi_\Rlts^\ri/I = (\xi_\Rlts^\ri/I, \synchrules \cup \hat\synchrules)$, for a given model~$\smodel = \networktuple$ and rule system~$\Sigma = \rulesystemtuple$ such that $\ri \in R$.

Figure~\ref{fig:lts-transformation:subsets-ltss:left} shows the left and right patterns of the rule network that contains only the topmost transformation rule of Figure~\ref{fig:lts-transformation:non-preserving-trafo}.
The left pattern of this rule network is constructed from the left pattern of this transformation rule, using the synchronization rules and the hiding set of Figure~\ref{fig:lts-transformation:comparison-network}.
The right pattern of this rule network is constructed from the right pattern of this transformation rule, using the synchronization rules and the hiding set of Figure~\ref{fig:lts-transformation:non-preserving-trafo}.
Because these patterns are not divergence-sensitive branching bisimilar, as indicated by the cross, this rule system is not \Sf-preserving.
The remaining patterns are shown in Figure~\ref{fig:lts-transformation:subsets-ltss:right}.

\subsection{Constructing a Bisimulation Relation}
The following theorem formalizes our $\Sf$-preservation check.

\begin{theorem}
\label{theo:lts-transformation:checkpreservation}
Let $\smodel = \networktuple$ be a model and $\Sf \in \dsbrLmu$ a temporal property such that $\models_{\smodel} \Sf$.
Then $\Sigma = \rulesystemtuple$ is $\Sf$-preserving if the following holds, for all $\ri \in R$, $I \subseteq \inv(\Llts^\ri, \synchrules)$.
\begin{equation}
\maxabstr(\Xi_{\Llts}^\ri/I ) \dsbbis \maxabstr(\Xi_{\Rlts}^\ri/I )
\label{eq:lts-transformation:maintheo}
\end{equation}
\end{theorem}

Proving this theorem for a rule system~$\Sigma$ and a property~\Sf entails constructing a relation between~$\maxabstr(\smodel)$ and~$\maxabstr(T_{\Sigma} (\smodel))$ based on the divergence-sensitive branching bisimulations in equation~\ref{eq:lts-transformation:maintheo}, for some model~\smodel, and proving that this relation is a divergence-sensitive branching bisimulation too.
To describe the construction of this relation, we need a number of auxiliary definitions.

First, a state in the network LTS of network~\smodel is a vector $\vectornot{s} = \langle \vectornot{s}[1],\ldots, \vectornot{s}[n] \rangle$.
An arbitrary $\vectornot{s} \in \states{\smodel}$ can have up to $n$ elements that are matched on by some transformation rule.
For an LTS~$\graph$ and a transformation rule~$r = \transruletuple{r}$,
matching $m_r$ is extended to sets of states such that $m_r(\states{\Llts^r}) \subseteq \states{\graph}$ refers to the set of states to which the states in $\states{\Llts^r}$ are matched,
and $\hat m_r(\states{\Llts^r}) = \{ s \in m_r(\states{\Llts^r}) \mid m_r^{-1}(s) \not\in \states{\Rlts^r} \}$ refers to the states in~\states{\graph} that relate to non-glue-states in $\Llts^r$.
By definition, the latter states are those that are removed from~\graph by rule~$r$.
We denote the set of indices of elements in $\vectornot{s}$ matched on by the corresponding rule with $M(\vectornot{s}) = \{ i \mid \vectornot{s}[i] \in m_\ri(\states{\Llts^\ri}) \}$.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%In a similar way, $\hat M(\vectornot{s}) = \{ i \mid \vectornot{s}[i] \in \hat m_\ri(\states{\Llts^\ri}) \}$.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
As indicated above, we assume that if $\vectornot{s}[i]$ is matched on, then it is matched on by rule $r_i$.
Furthermore, for a rule system~$\Sigma = \rulesystemtuple$ and a network~\smodel, a simulation relation exists between the states in~\smodel and those in the rule networks~$\Xi_\Llts^\ri/I$ as well as between the states in~$T_{\Sigma} (\smodel)$ and those in~$\Xi_\Rlts^\ri/I$.
This can be formalized as follows.
For a state vector $\vectornot{s}^*$ in a rule network~$\Xi_\Llts^{\ri}/I$ and a state vector $\vectornot{s}$ in an LTS network~$\smodel$, we say that $\vectornot{s}$ simulates $\vectornot{s}^*$, denoted $\vectornot{s}^* \vdash \vectornot{s}$, iff $\forall i \in 1..n. \vectornot{s}^*[i] \neq \ast \implies \vectornot{s}[i] = m_\ri(\vectornot{s}^*[i])$.
In other words, besides the $\ast$-states, all process states in $\vectornot{s}^*$ are matched on the corresponding states in $\vectornot{s}$.
Now, first of all, if $\vectornot{s}^* \vdash \vectornot{s}$, and $\vectornot{s}^*\xrightarrow{a} \vectornot{s}^*'$, then also $\vectornot{s}\xrightarrow{a} \vectornot{s}'$ and $\vectornot{s}^*' \vdash \vectornot{s}'$.
Second of all, in cases that $\vectornot{s}\xrightarrow{a} \vectornot{s}'$ and some synchronization vector~$\vectornot{t}$ enables transition~$\vectornot{s}\xrightarrow{a} \vectornot{s}'$, with ${\it Ac}(\vectornot{t}) \subseteq I$, ${\it Ac}(\vectornot{t}) \subseteq M(\vectornot{s})$, and ${\it Ac}(\vectornot{t}) \subseteq M(\vectornot{s}')$, then the involved behavior of every active $\Pi[i]$ ($i \in {\it Ac}(\vectornot{t})$) is matched on by $\ri$.
The definition for~$\Xi_\Rlts^{\ri}/I$ and~$T_{\Sigma} (\smodel)$ is similar.
These simulation relations are preserved after maximal hiding.

\begin{figure}[hbt]
\centering
\includegraphics[scale=0.2]{lts-transformation/figs/relations}
\caption{Constructing a divergence-sensitive branching bisimulation}
\label{fig:lts-transformation:relations}
\end{figure}

A relation between two networks representing a model before and after transformation can be constructed by combining the bisimulations between pairs of rule networks with the simulation relations between model networks and rule networks.
In Figure~\ref{fig:lts-transformation:relations}, the dashed lines connect some of the states in the LTSs of the rule networks with the states in the LTSs of the models that simulate them.
The dotted lines in the figure denote the divergence-sensitive branching bisimulations between pairs of rule networks.
By combining these relations, states in the LTS on the left of the figure are related to states in the LTS on the right.
We refer to the relation formed by combining these relations as~$D$.
To increase the readability of the figure, not all states that are related according to~$D$ are connected.
For states~$\vectornot{s}$ of~$\smodel$ and~$\vectornot{p}$ of~$T_{\Sigma} (\smodel)$, we define $\vectornot{s}\ D'\ \vectornot{p}$ iff $\forall i \not \in M(\vectornot{s}) . \vectornot{s}[i] = \vectornot{p}[i]$.
In words, $D'$ relates all state vectors with exactly the same elements apart from those matched on by a transformation rule.
In a sense, $D'$ is a strong bisimulation for the behavior not subjected to transformation.
As mentioned above, not all states that are related according to~$D$ are connected in the figure.
Only those states of the LTS on the left of the figure are connected to states of the LTS on the right that are also related according to~$D'$.
Thus, the dashed and dotted lines together illustrate the relation~$(D \cap D')$ by connecting some of the states that are related according to this relation.
Relation~$D'' = \{ (\vectornot{s}, \vectornot{s}) \mid M(\vectornot{s}) = \emptyset \}$ relates all states that are not subjected to transformation.
The grey lines in Figure~\ref{fig:lts-transformation:relations} connect the states that are related according to~$D''$.
Given these relations, a relation~$C = (D \cap D') \cup D''$ can be constructed, which relates~$\maxabstr(\smodel)$ and~$\maxabstr(T_{\Sigma} (\smodel))$.
By considering the cases of Definition~\ref{def:lts-transformation:dsbbsim}, it can be shown that this relation is a divergence-sensitive branching bisimulation~\cite{EngelenWijsPropPres2012}.