\chapter{Introduction} \label{sec:intro}
Positioning assets, such as facilities and equipment, within a pre-defined region, such as a plot of land or a building, in a fashion that is tailoured towards a criteria of optimality for a specific problem is one endeavour that has multiple applications in different fields, primarily due to the benefits it provides. Finding the best possible asset positioning can result in improved operations efficiency, better productivity \cite{El-Baz2004}, and even decreases in expenses \cite{Sahin2011}. As a matter of fact, due to the benefits of asset positioning, \$300 billion dollars have been spent each year on just determining suboptimal locations of buildings and facilities in the United States alone \cite{Asl2015a}. This is further proof of the importance of asset positioning. One entertaining example of said application is showcased by Barriga et al. (2014). In their paper, the authors developed a genetic algorithm that optimized placement of buildings in a StarCraft match. The algorithm produced building placements that allowed the defending player's base to better survive base assaults from the opposing player \cite{Barriga2014}. Developing an open-plan office layout is another application of asset positioning. Chen et al. (2020) also developed a genetic algorithm that generates an open-office layout where the space utilization is maximized as possible \cite{Chen2020}. This task of arranging assets within a given space according to some criteria has a formal term, which is the "facility layout problem", often abbreviated as "FLP". We will be discussing facility layout problems in more detail in this chapter.

FLP is a field that has been researched as early as 1957 (with Koopsman T.C., and Beckman, M. being the first to model the problem) \cite{Kusiak1987}, and there is still active research around it to this day. This research paper is one of the testaments to that. In this research, we will be solving the classical facility layout problem using a recent optimization algorithm called the Grey Wolf Optimization (GWO) algorithm. The specific type of FLP that we will solve is called the unequal area static FLP. The categorization will be discussed later in this paper. The proposed algorithm will then be compared to a genetic algorithm using experimental data used in other related papers.

\section{Facility Layout Problem}
The problem of arranging a set of facilities and/or machines in a pre-determined area, or a set of possible locations (such as in the work of Farmakis, P., and Chassiakos, A. \cite{Farmakis2018}) is called the facility layout problem (FLP). The facilities and/or machines are arranged in such a way that the resulting layout is in line with some criteria or objectives and under certain constraints. These constraints, which must not be violated, include shape, size, orientation, pick-up/drop-off points \cite{Hosseini-Nasab2018}, and usable area \cite{Fernando2015}. Facilities and/or machines must also not overlap. Solutions that satisfy the aforementioned conditions are called feasible solutions \cite{Meller1996}.

Generally, the facility layout problem is considered to be an \textbf{NP-Hard} problem \cite{Drira2007}. Hosseini-Nasab, H., Fereidouni, S., and Fatemi, S. have noted in their systematic review of FLP that most researches dealing with the facility layout problem model their problems either as a quadratic assignment problem (QAP) or a mixed integer programming problem \cite{Hosseini-Nasab2018}. According to Drira, A., Pierreval, H., and Hajri-Gabouj, S., the former is sometimes used in discrete FLP formulations, while the latter is often used in continuous formulations \cite{Drira2007}. Discrete and continuous FLP formulations will be discussed later. \textbf{Quadratic assignment problems} deal with placing $n$ facilities in $n$ locations in such a way that minimizes the assignment cost. The assignment cost is the sum of all facility pairs's flow rate between each other multiplied by their flow rate \cite{QAPDefinition}. This assignment cost is commonly seen in many FLP researches, as we will discuss later. QAP is also known to be an NP-Hard problem \cite{Garey1979}. It should be noted though that \textit{some} instances of QAP are easy to solve \cite{Feizollahi2015}. The other modeling framework, \textbf{mixed integer programming}, can solve problems with both discrete decisions and continuous variables. An example of such problem is the assignment problem \cite{Richards2005}, which the FLP can be classified under. In this formulation, a set of integer and real-valued integers are being optimized based on an objective function that is being minimized or maximized, while satisfying constraints which are linear equations or inequalities \cite{Wolsey2008}. Mixed integer programming, when in the context of optimization, is also known to be NP-Hard \cite{Richards2005}. These two formulations being known to be generally NP-Hard proves that FLP is indeed generally NP-Hard.

The fact that FLP is an NP-Hard problem has resulted in many research works that utilize heuristics (such as simulated annealing and genetic algorithms). Note that there are also works that utilize exact methods, which seek to find the \textit{optimal} solution for a problem. However, the NP-Hard nature of the FLP prevents them from finding the solution in large problems within reasonable time \cite{Asl2015}.

\subsection{The Basic Mathematical Model}
Each problem instances of the facility layout problem naturally will have their own mathematical models tailor-fit for their problem instance. Nevertheless, based on our observations and from readings, most of those models are derivatives of or use (such as in \cite{Garcia-Hernandez2013}, \cite{Lin2019}, and \cite{Navarro2016}) what will be calling a basic minimization function, which is defined as:

$$
\text{min} F = \sum_{i=1}^{n}\sum_{j=1}^{n}c_{ij}f_{ij}d_{ij}
$$

where $N$ is the number of facilities, $c_{ij}$ is the cost of handling materials between locations $i$ and $j$, $f_{ij}$ is the flow rate between $i$ and $j$, and $d_{ij}$ is the distance between the centroids of $i$ and $j$. The distance function may differ from work to work. For example, Liu, J., et. al. uses the Manhattan distance in their work \cite{Liu2018}, while in the work of Ripon, K. S. N., et. al., Euclidean distance was used \cite{Ripon2013}. In works that derive from this formula, such as in \cite{Farmakis2018}, \cite{Solimanpur2008}, and \cite{Peng2018}, it was observed that $d_{ij}$, or a similar variable or expression, is commonly present in the work's objective function while $c_{ij}$ and $f_{ij}$ \textit{may} be present and/or the work uses more or fewer variables.

Drira, A., Pierreval, H., and Hajri-Gabouj, S. note the same observation but showcase a slightly differing formula in their 2007 survey of facility layout problems. Unlike the basic minimization formula above, their formula has $f_{ij}$ and $c_{ij}$ combined. They also note that the function above is typically used in continuous formulations of the FLP. The discrete formulation uses a similar function, but ensures that a facility is only in one location, a location only contains one facility, and makes sure that only pairs of locations that contain facilities contribute to the fitness value of a solution. (The descriptions and the differences of the discrete and continuous formulations are discussed in the next section.) Additionally, they mention that the function is also subject to the following constraints: (1) facilities must obviously not overlap with one another, and (2) the total area used by the facilities must be equal to or less than the allotted area \cite{Drira2007}. These constraints have been observed to be generally in many FLP works.

\subsection{Discrete vs Continuous Formulations}
Solving instances of the facility layout problem requires determining the form of the solution. The form is highly dependent on the problem being solved. Some problems may require a solution that assigns assets to pre-existing locations, while others may require more flexibility. Facility layout problems may be categorized based on the characteristics of these solutions, or formally known as formulations: discrete, and continuous.

In a discrete formulation, the region where the facilities will be laid out are divided into equal rectangular blocks of the same shape and size, or have pre-determined possible facility locations \cite{Drira2007}. Each facility will be given a number of blocks, or be assigned to one facility location, respectively. This formulation, however, does not suit well when the facilities require exact positions and it cannot model facility attributes such as orientation. In problems that have such requirements, a continuous formulation is more appropriate \cite{Hosseini-Nasab2018}. Facilities in a continuous formulation are usually located by either their centroid coordinates, half length, and half width, or by their bottom-left coordinates, length, and width \cite{Drira2007}. This allows for the formulation's flexibility compared to its discrete counterpart. However, this does provide challenges towards ensuring that no two facilities overlap with one another. Discrete formulations do not need to consider this problem due to their inherent characteristics.

\subsection{Static vs Dynamic Facility Layout Problems}
Another categorization for facility layout problems is based on whether the layouts will change over time. There are situations where a regular change of layout over some periods of time is necessitated. The layout of facilities in a construction is one example. As the construction of a building moves to from phase to another, the layout of facilities within the construction site change to better fit the needs of the current phase of construction \cite{Farmakis2018}. A similar need is the motivation behind changing layouts in manufactories. Product demand variations, and even a change in product design can incline a factory's management to reorganize facilities in the building to be more efficient in response to the changes \cite{Pourhassan2017}. There are two categories for the aformentioned criteria. These are: (1) static, and (2) dynamic. We will refer to these categories as \textbf{"period-based layout categories"} in this paper.

The survey of Hoisseini-Nasab et al. (2018) showed that the most common period-based categorization in literature is the static facility layout problem \cite{Hosseini-Nasab2018}. This is likely due to the fact that static facility layout problems are easier to solve than dynamic facility layout problems. Though, it is also possible that many problems just happen to not require consideration of variable changes over time. The \textbf{static facility layout problem}, abbreviated as SFLP, is a type of FLP where variables to be considered such as material handling costs do not change for a considerable amount of time \cite{Perez-Gosende2020}. For this type of problems, only a single layout is generated since no changes are made in the considered variables over time.

However, some industries will find SFLPs inadequate for their needs. There are companies that require adaptability to changes to, for example, product demands. For cases like this, the other category, dynamic facility layout problems are more appropriate \cite{DerakhshanAsl2017}. In the dynamic facility layout problem, abbreviated to DFLP, the variables to be considered change over time, unlike in SFLP. The cost of rearranging facilities are also considered in the problem \cite{Hosseini2016}. The solutions for DFLPs are also divided into time periods, where each period has a different layout. This period may equate to years, seasons, months, or weeks \cite{DerakhshanAsl2017}. DFLPs can also be viewed as extensions of SFLP, since each layout in a period can be viewed as a solution to an SFLP with that period's variables into consideration but with rearrangement costs considered. While most research today is focused on SFLPs, Hosseini-Nasab et al. (2018) recommends that research should deal with DFLPs more these days due to rapid scientific developments, and product changes \cite{Hosseini-Nasab2018}.

\subsection{Other FLP Classifications}
Facility layout problems can also be categorized based on different characteristics. FLPs can be divided by the area of their facilities. The facilities may have the same areas, referred to as equal areas, or have different areas, referred this time to as unequal areas \cite{DerakhshanAsl2017}. They can also be divided based on the possible arrangements of facilities. Some problems may have facilities located only in a single pre-defined row (single-row), or they may be placed anywhere in the region (open field) \cite{Drira2007}. There are multiple classifications for FLP and discussing them in this chapter would take long and dislocate the focus of this paper. Due to that, we would like to refer the reader to the papers of Drira et al. (2007) \cite{Drira2007} and Hosseini-Nasab et al. (2018) \cite{Hosseini-Nasab2018} for more information on FLP classifications.

\section{The Grey Wolf Optimization Algorithm}
In this paper, we will be using the Grey Wolf Optimization algorithm to solve the unequal-area static facility layout problem. As such, we will be introducing the algorithm here for us to gain a better understanding of the algorithm.

\begin{figure}[h!]
	\centering
	\includegraphics{./images/chap01-introduction/wolves.jpg}
	\caption{The Grey Wolf Optimization algorithm was inspired from the behaviour of grey wolves. Pictured are white wolves, different from grey wolves, but why pass up the opportunity to add a meme in a research paper? Profeshonal.}
	\label{wolves-meme}
\end{figure}

The Grey Wolf Optimization algorithm, abbreviated as GWO, was first conceived by Mirjalili  et al. (2014) in 2014. The optimization algorithm is inspired from the hunting and social behaviour of grey wolves. There is a hierarchy in packs of wolves. Each category in the hierarchy have specific responsibilities. There are four categories: alpha ($\alpha$), beta ($\beta$), delta ($\delta$), and omega ($\omega$). Alpha wolves are responsible for making major decisions for the pack. Every wolf must follow the alpha. However, sometimes the alpha follows other wolves. The second in line is the beta, which ensures the discipline of the pack and advises the alpha. They also command other wolves and reinforces the alpha's commands. The lowest in the hierarchy are the omegas. They must follow the orders of the other wolves, and are the last to eat. Despite their low status, they are still crucial in the pack as their absence causes the pack to face internal fighting and problems. If a wolf is not an alpha, beta, nor omega, they are considered to a delta, the third category in the hierarchy. Deltas may act as scouts, sentinels, elders, hunters, or caretakers. They are also at a category higher than the omegas \cite{Mirjalili2014} \cite{Gupta2018}. As an interesting side note, the inspiration for Grey Wolf Optimization initially came from The Grey, a movie where survivors of a plane crash must survive, but a pack of grey wolves surround them \cite{TheGreyPlotSummary2011}.

\subsection{Mathematical Model}
The mathematical model assumes the existence of a "pack of wolves". The number of wolves in this pack can be determined by the researcher. Each wolf of this a solution to the problem. We will be delving into the model more in this section, discussing about the model of leadership hierarchy, encircling, and hunting behaviour of grey wolves. The prey being hunted in this scenario is the best solution for a given problem \cite{Mirjalili2014}.

\subsubsection{Leadership Hierarchy}
Solutions are assigned to a certain hierarchy in the mathematical model of GWO. The fittest solution is considered the alpha ($\alpha$), while the second and third fittest are considered to be the beta  (\beta) and delta ($\delta$) solutions. The rest of the solutions are referred to as the omega solutions. The leading wolves guide the omegas towards the prey throughout the search process \cite{Gupta2018}.

\subsubsection{Encircling the Prey}
Prey encirclement, which is one of the first steps when grey wolves hunt for their prey, can be modeled with the following:

\begin{align*}
	\vec{X}(t + 1) &= \vec{X_{p}}(t)
					  - \vec{A} \cdot \vec{D}                     \\
	\vec{D}        &= \left | \vec{C} \cdot \vec{X_{p}}(t)
					  - \vec{X}(t) \right | \\
	\vec{A}        &= 2 \cdot \vec{a} \cdot \vec{r_{1}}
					  - \vec{a}                \\
	\vec{C}        &= 2 \cdot \vec{r_{2}}
\end{align*}

where $\vec{X}(t)$ and $\vec{X}(t + 1)$ are the positions of the wolf at the iteration $t$ and $t + 1$ respectively, $\vec{X_{p}}(t)$ represents the location of the prey at the iteration $t$, $\vec{D}$ is the difference vector, $\vec{A}$ and $\vec{C}$ are coefficient vectors, and $\vec{r_{1}}$ and $\vec{r_{2}}$ are uniformly random vectors with the range $\left[ 0, 1 \right]$. $\vec{a}$ is vector that linearly decreases from $2$ to $0$ over the course of iterations \cite{Mirjalili2014}. The original paper on GWO does not specify but Gupta, S. and Deep, K. provided the following equation to specify the decrease of $\vec{a}$ from $2$ to $0$ \cite{Gupta2018}:

$$
\vec{a} = 2 - 2 \cdot \left( \frac{t}{\text{maximum number of iterations}} \right)
$$

Note that the multiplication of vectors in the equations above is a component-wise multiplication, and not a dot product \cite{Mirjalili2020-MathModel}.

\subsubsection{Hunting}
We typically do not know the position of the prey in an abstract search space. As such, it is presumed that the $\alpha$, $\beta$, and $\delta$ solutions have the best idea so far of the position of the prey \cite{Mirjalili2014}. Each wolf updates their positions based on the following equations.

\begin{align}
	\vec{D}_{\alpha} &= \left | \left ( \vec{C}_{1} \cdot \vec{X}_{\alpha} \right ) - \vec{X} \right | \\
	\vec{D}_{\beta} &= \left | \left ( \vec{C}_{2} \cdot \vec{X}_{\beta} \right ) - \vec{X} \right | \\
	\vec{D}_{\delta} &= \left | \left ( \vec{C}_{3} \cdot \vec{X}_{\delta} \right ) - \vec{X} \right | \\
	\vec{X_{1}^{'}} &= \vec{X_{\alpha}}(t) - \vec{A_{\alpha}} \cdot \vec{D_{\alpha}} \\
	\vec{X_{2}^{'}} &= \vec{X_{\beta}}(t) - \vec{A_{\beta}} \cdot \vec{D_{\beta}} \\
	\vec{X_{3}^{'}} &= \vec{X_{\delta}}(t) - \vec{A_{\delta}} \cdot \vec{D_{\delta}} \\
	\vec{X}(t + 1)  &= \frac{\vec{X_{1}^{'}} + \vec{X_{2}^{'}} + \vec{X_{3}^{'}}}{3}
\end{align}

where $\vec{X_{\alpha}}$, $\vec{X_{\beta}}$, and $\vec{X_{\delta}}$ represent the $\alpha$, $\beta$, and $\delta$ solutions \cite{Gupta2018}.

\subsubsection{Exploration and Exploitation}
The exploration phase of metaheuristics is modeled by the search phase, while exploitation is modeled by the attack phase. When $\left| \vec{A} \right| < 1$, or $\vec{C} < 1$, GWO is undergoing exploitation of the search space. Exploitation can be viewed as the wolves approaching towards the prey. On the other hand, when $\left| \vec{A} \right| > 1$, or $\vec{C} > 1$, the algorithm is in the search phase, where the wolves can be viewed as searching for the prey. In the search process, as the number of iterations $t$ reach the maximun possible number, the algorithm tends to focus more on exploitation than exploration. $\vec{A}$ and $\vec{a}$ eventually approach $0$, leaving $\vec{C}$ the sole vector to eventually influence the search exploration. At this point, the algorithm will intensify towards exploitation.


\chapter{Basics on classification}\label{ch:2}

\begin{remark}{Outline}
In this chapter, we introduce classification models from a machine learning perspective.
First, in Section~\ref{sec:2:classification}, we give a self contained introduction to
classification, including the most-common algorithms, and the main applications of
classification models. Then, in Section~\ref{sec:2:measures}, we present the different evaluation
measures that are normally used for analyzing the performance of classification methods.
\end{remark}

\section{Introduction}
\label{sec:2:classification}

In machine learning, classification refers to the attempt of identifying to which of a set of
classes a new example belongs, based on learning from examples whose class membership is known.
A classification task begins with a training set in which the class of a set of examples is known.
For example, a classification model that predicts credit card fraud is developed by analyzing
many observed credit transactions over a period of time. The class in this case is a variable which
indicates for each example whether or not the transaction was or not a fraud. Also, the features,
are the transaction attributes like place, amount and time of the transaction.

Then, during the training process, a classification algorithm finds the patterns and relationships
between the values of the features and the values of the target class. Different algorithms use
different methods and techniques to estimate these relationships. Afterwards, these relationships
are summarized in a model that is able to make predictions on new sets of data.

In general, there are two types of classification models: binary and multi-class. In binary
classification problems, the objective is to classify examples between two classes, usually
referred to as the negative and positive classes. On the other hand, multi-class problems are not
bound to two classes but instead aim to classify examples among a number of classes. In this work we
focus on binary classification problems.

Binary classification algorithms are widely used across a variety of domains. For example in the
medical field, models have been used for making predictions about tumors, probability
of a disease, probability of selecting the right drug for a particular patient, and estimating the
probability of relapsing, among others \citep{Herland2014}. In the financial sector, classification
models have been successfully applied for fraud detection, credit scoring, portfolio management and
algorithmic trading. Also, in marketing, several models are being currently used for churn modeling,
customer targeting, behavior prediction and direct marketing \citep{Baesens2014}. Additionally,
classification algorithms are used in many other emerging applications such as terrorism
prevention, malware detection, computer security, energy consumption prediction, spam
classification, and others \citep{Kriegel2007}.

Formally, a binary classification algorithm deals with the problem	of predicting the class $y_i$
of a set $\mathcal{S}$ of examples or instances $i$, given their $k$ features \mbox{$\mathbf{x}_i
\in \mathbb{R}^k$}. The objective is to construct a function $f(\cal{S})$ that makes a prediction
$c_i$ of the class of each example $i$ from $\cal{S}$ using its feature vector $\mathbf{x}_i$,
where $\vert \mathcal{S} \vert= N$. Moreover, some algorithms allow to not only estimate the
prediction, but also its confidence, in the form of the probability $\hat p_i$ of belonging to the
positive class, i.e. $c_i = 1$. The way for finding from $\hat p_i$ to $c_i$ is simply by defining
a probability threshold $t$, and applying the following formula
\begin{equation}\label{eq_pred}
  c_i =
  \begin{cases}
    \phantom{-}0 \phantom{-} \mbox{if} \phantom{-} \hat p_i \le t\\
    \phantom{-}1 \phantom{-}\mbox{otherwise,}
  \end{cases}
\end{equation}
Usually $t=\frac{1}{2}$ \citep{Hastie2009}. However, if $t \ne \frac{1}{2}$, the
function that generates the predicted class labels $\mathbf{c}=[c_i]$ is denoted as
$f^t$.

\begin{figure}
	\centering
	\input{figures/src/ch2_fig_classification_process}
  \caption{Classification process}
  \label{fig:2:1}
\end{figure}

In \figurename{ \ref{fig:2:1}}, the process of training and prediction in a classification
algorithm are summarized. First, during the training phase, using a training set
$\mathcal{S}_{train}$, an algorithm is trained to predict $\mathbf{y}$, where
$\mathbf{y}=[y_i]$. Then the algorithm is used to estimate the classes
$\mathbf{c}$ of a set of testing examples $\mathcal{S}_{test}$.

\begin{figure}[!t]
\centering
\includegraphics{ch2_fig1b}
\caption{Example of a classification algorithm. Using a set of examples from two classes, a
	classification algorithm is learned in order to separate between the positives and the negatives.
}
\label{fig:2:2}
\end{figure}

There exists several algorithms that can be used for classification tasks. In general a
classification algorithm is learned with the objective of finding patterns that separate between
the different classes \citep{Hastie2009}. In order to clarify this intuition, in \figurename{
\ref{fig:2:2}} an example of a classification algorithm is shown. Let us consider a set of
examples, where the red points represent the positive examples and the blue ones the negative
examples. The objective of a classifier is to find the best way to separate between the positive and
negative examples. Then, the output of a classifier learned using the set of training examples is
shown as the dashed black line. It is observed that this classifier is able to separate almost all
the examples using a linear classifier. However, not all examples are correctly classified. In
particular, there are four negative examples that were predicted as positive, and five positive
examples that were predicted as negative. In the next section, we present the standard methods for
evaluating the performance of a classification algorithm.

\section{Traditional evaluation measures}
\label{sec:2:measures}

When evaluating the performance of a classification algorithm, the first thing to do is to check
the number of examples that were misclassified, since the true class of the examples
is known. Therefore, evaluating the error of a model is as simple as counting the number of times
an example is misclassified divided by the number of examples
\begin{equation}\label{eqn:ch2:error}
Err(f({\cal S})) = 1 -\frac{1}{N}  \sum_{i=1}^N \mathbf{1}_{y_i}(c_i),
\end{equation}
where $\mathbf{1}_q(z)$ is an indicator function that is calculated as:
\begin{equation}
   \mathbf{1}_q(z) =
  \begin{cases}
    \phantom{-}1 \phantom{-} \mbox{if} \phantom{-} z = q\\
    \phantom{-}0 \phantom{-} \mbox{if} \phantom{-} z \neq q.
  \end{cases}
\end{equation}
Moreover the accuracy is defined as the percentage of times the algorithm
made the correct prediction
\begin{equation}\label{eqn:2:accuracy}
Acc(f({\cal S})) = 1- Err(f({\cal S})).
\end{equation}

However, just knowing these statistics is not enough to make decisions, as in many applications
it is important to know where the errors are coming from. In particular, the misclassified examples
may belong only to one class, which may give interesting insights about the problem. A way to
observe the different errors is by looking at the confusion matrix, as shown in
\mbox{\tablename{~\ref{tab:2:1}}}. Afterwards, using the cost matrix several statistics are
extracted. In particular:
  \begin{flalign}
    &Recall = \frac{TP}{TP+FN} &\\
    &Precision = \frac{TP}{TP+FP}& \\
    &F_1Score = 2\frac{Precision \cdot Recall}{Precision + Recall}&
  \end{flalign}

	\begin{table}[!t]
		\centering
		\footnotesize
    \begin{tabular}{c|c|c}
      \multicolumn{3}{c}{}\\
			\multicolumn{1}{c|}{}  & Actual Positive& Actual Negative \\
			\multicolumn{1}{c|}{} & $y_i=1$& $y_i=0$ \\
			\hline
			Predicted Positive 		& \multirow{ 2}{*}{True Positive ($TP$)} & \multirow{
			2}{*}{False Positive ($FP$)} \\
			$c_i=1$ & &\\
			\hline
			Predicted Negative  	& \multirow{ 2}{*}{False Negative ($FN$)} & \multirow{
			2}{*}{True Negative ($TN$)} \\
			$c_i=0$ & &\\
		\end{tabular}
		\caption{Classification confusion matrix}
		\label{tab:2:1}
  \end{table}

 \newpage
  As an illustrative example, the different statistics are calculated for the example presented
  in Section~\ref{sec:2:classification}. First, the confusion matrix is calculated as
  follows:
  \begin{center}
    \footnotesize
  \begin{tabular}{c|c|c}
    \multicolumn{1}{c|}{}  & Actual Positive& Actual Negative \\
    \multicolumn{1}{c|}{} & $y_i=1$& $y_i=0$ \\
    \hline
    Predicted Positive    & \multirow{ 2}{*}{36} & \multirow{
    2}{*}{4} \\
    $c_i=1$ & &\\
    \hline
    Predicted Negative    & \multirow{ 2}{*}{5} & \multirow{
    2}{*}{68} \\
    $c_i=0$ & &\\
  \end{tabular}
  \end{center}
  Then using the confusion matrix, the different statistics are calculated as: Error =
11.11\%, Recall = 87.8\%, Precision = 90\% and $F_1Score$ = 88.8\%.

There are, however, several instances that are misclassified, that is because the simple linear
classifier that was used in this example may not be good enough to separate between the positive
and negative classes. In order to make a comparison, using the same example, a new
algorithm is learned. This time the algorithm made the correct prediction more often as shown in
\figurename{ \ref{fig:2:3}}. Afterwards, the confusion matrix is calculated as follows:

\begin{figure}[t!]
  \centering
  \includegraphics{ch2_fig2}
  \caption{Example of a classification algorithm. Using a set of examples from two classes, a
  classification algorithm is learned in order to separate between the positives and the negatives.
}
  \label{fig:2:3}
\end{figure}

\begin{center}
		\footnotesize
    \begin{tabular}{c|c|c}
			\multicolumn{1}{c|}{}  & Actual Positive& Actual Negative \\
			\multicolumn{1}{c|}{} & $y_i=1$& $y_i=0$ \\
			\hline
			Predicted Positive 		& \multirow{ 2}{*}{37} & \multirow{
			2}{*}{2} \\
			$c_i=1$ & &\\
			\hline
			Predicted Negative  	& \multirow{ 2}{*}{4} & \multirow{
			2}{*}{70} \\
			$c_i=0$ & &\\
		\end{tabular}
\end{center}
Then the different statistics are calculated as: Error = 5.3\%, Recall = 90.2\%, Precision =
94.9\% and $F_1Score$ = 92.5\%.
It is observed that in this case the FP are reduced more than the FN, this leads to a higher
increase in precision  than in recall. There is not a single rule regarding which one is more
important than the other, it depends on the application. For example in applications with a high
false negative cost such as failing to identify a tumor in a medical exam, the recall should be the
priority, even if that implies having a significant number of false positives. On the other hand,
In applications such as spam detection, predicting a normal email as spam it may have a large
impact on the customer, therefore, in this example is better to allow some false negatives and
focus on the false positives.

It is not always straightforward  to define the right tradeoff between false positives
and false negatives. The best approximation to solve that, is to focus on the actual costs incurred
by the different decisions. This is usually solved using cost-sensitive classification methods.

\subsection{Brier score}
\label{sec:2:brier}

Traditional evaluation measures of binary classification problems, such as Accuracy and
$F_1Score$, provide a way to analyze the performance of a model. However, when using the classifier
output as a basis for decision making, there is a need of a measure that takes into account not
only the misclassification of a classifier predicted class $c$, but also the quality of the
estimated probabilities $\mathbf{\hat p}$ \citep{cohen2004}. The most appropriate  is the Brier
score \citep{brier1950}. The Brier score belongs to the class of so-called proper scores which are
used in evaluating the subjective probability assessment of the prediction \citep{DeGroot1983}. The
Brier score is the average squared difference between the estimated probability and the true class
label. It is defined as:
\begin{equation}
  BS(f(\mathcal{S})) = \frac{1}{N} \sum_{i=1}^{N} (\hat p_i - y_i)^2.
\end{equation}

The main justification of this score is based on decision theoretic considerations, in the sense
that, a forecaster should pay a price proportional to the confidence with which it asserts its
decision.

\subsection*{}


\renewcommand*\chappic{img/intro.pdf}
\renewcommand*\chapquote{}
\renewcommand*\chapquotesrc{}
%
\chapter{Introduction}
\label{ch:intro}
\section{Overview}
\label{sec:intro-overview}
%
Hash functions are used as cryptographic primitives in many applications and protocols.
They take an arbitrary input message and provide a hash value. Input message and hash value
are considered as byte strings in a particular encoding.
The hash value is of fixed length and satisfies several properties which make it useful
in a variety of applications.

In this thesis, we consider the hash algorithms MD4 and SHA-256.
Our goal is to find hash collisions using differential cryptanalysis.
We define differences between two messages and determine actual bits
such that the two messages result in the same hash value. \\
This whole equation system will be modelled as a satisfiability problem.
A SAT solver reports satisfiability if and only if the particular
differences can be resolved and an actual hash collision is found.
We introduce a bit condition notation which allows us to visualize
such differential states. Verification is done by several SAT solvers
and we compare their runtime. Because the Boolean functions modelled in
a CNF have a major influence on the runtime, we investigate several
approaches and compare them.

Based on experience with these kind of problems with previous heuristic search tools
we aim to apply best practices to a satisfiability setting.
We will discuss, which SAT techniques lead to best performance characteristics
for our MD4 and SHA-256 testcases.

\section{Thesis Outline}
\label{sec:intro-outline}
%
This thesis is organized as follows:

\begin{description}
\item[In Chapter~\ref{ch:intro},] we briefly introduce basic subjects of this
  thesis. We explain our high-level goal involving hash functions and SAT solvers.

\item[In Chapter~\ref{ch:hash},] we introduce the MD4 and SHA-256 hash functions.
  Certain design decisions imply certain properties which can be used in differential
  cryptanalysis. We discuss those decisions in this chapter after a formal definition
  of the function itself. Beginning with this chapter we develop a theoretical notion
  of our tools.

\item[In Chapter~\ref{ch:dc},] we discuss approaches of differential cryptanalysis.
  We start with work done by Wang, et al. and followingly introduce differential
  notation to simplify representation of differential states. This way we can easily dump
  hash collisions.

\item[In Chapter~\ref{ch:sat},] we discuss SAT solving techniques. We discuss how
  the problem needs to be encoded and give a brief overview over used SAT solvers.
  This includes a customized lingeling version by Armin Biere for our purposes.

\item[In Chapter~\ref{ch:features},] we define SAT features which help us to
  classify SAT problems. This is a small subproject we did to look at properties
  of resulting DIMACS CNF files.

\item[In Chapter~\ref{ch:enc},] we discuss how we represent a problem (i.e. the hash
  function and a differential characteristic) as SAT~problem. This ultimatively
  allows us to solve the problem using a SAT solver.

\item[In Chapter~\ref{ch:results},] we present the result of our work.
  Runtimes are the main part of this chapter, but also results of Chapter~\ref{ch:features}
  are presented.

\item[In Chapter~\ref{ch:summary},] we conclude and discuss future work based on our results.
\end{description}


\documentclass[letterpaper,11pt]{article}
\usepackage{ltcadiz}

%%side margin is 1.0in + oddside, so we're creating a 1 inch margin
\oddsidemargin 0.0in
\topmargin 0.0in
\textwidth 6.5in

\usepackage{fullpage}

\title{Z Examples Found on the Internet}

\begin{document}

\maketitle

\section{Introduction}

\subsection{Z section heading}

    \begin{zsection}
    \SECTION gearsystem \parents standard\_toolkit
    \end{zsection}

\subsection{Free types paragraph}

    \begin{zed}
    gearing ::= Single | Hub \ldata \nat \rdata
        | Derailleurs \ldata \nat \cross \nat \rdata
    \end{zed}

\subsection{Horizontal definition paragraph}

    \begin{zed}
    Wellgeared == [ gears : gearing | gears \in \ran Derailleurs ]
    \end{zed}

\subsection{Generic axiomatic description paragraph}

    \begin{gendef}[X,Y]
    First : X \cross Y \fun X \\
    Second : X \cross Y \fun Y
    \where
    \forall x : X; y : Y @ \\
    \t1     First~(x,y) = x \land \\
    \t1     Second~(x,y) = y
    \end{gendef}

\subsection{Schema definition paragraph}

    \begin{schema}{Gear}
    Wellgeared \\
    ingear : \nat \cross \nat
    \where
    \exists r, s : \nat | gears = Derailleurs (r,s) @\\
    \t1 First~ingear < r \land Second~ingear < s
    \end{schema}

\subsection{Generic schema definition paragraph}

    \begin{schema}{Bicycle}[G]
    gears:\power G
    \end{schema}

\subsection{Axiomatic description paragraph}

    \begin{axdef}
    numgears : gearing \fun \nat
    \where
    \forall g : gearing @ \\
    \t1 (g = Single \implies numgears~g = 1) \land \\
    \t1 (\exists x : \nat @ g = Hub~x \implies numgears~g = x) \land \\
    \t1 (\exists r, s :\nat @ g = Derailleurs~(r, s) \implies numgears~g = r*s)
    \end{axdef}

\subsection{Conjecture paragraph}

    \begin{theorem}{Lemma1}
    \thrm numgears (Derailleurs (3,7)) = 21
    \end{theorem}

\end{document}


% This LaTeX was auto-generated from an M-file by MATLAB.
% To make changes, update the M-file and republish this document.

\subsection*{conT.m}

\begin{par}
\textbf{Summary:} Test derivatives of controller functions. It is assumed that the controller function computes the mean and the variance of the control signal for a Gaussian distributed input $x\sim\mathcal N(m,s)$
\end{par} \vspace{1em}
\begin{verbatim}function [dd dy dh] = conT(deriv, policy, m, s, delta)\end{verbatim}
\begin{par}
\textbf{Input arguments:}
\end{par} \vspace{1em}
\begin{verbatim}deriv    desired derivative. options:
     (i)    'dMdm' - derivative of the mean of the predicted control
             wrt the mean of the input distribution
     (ii)   'dMds' - derivative of the mean of the predicted control
             wrt the variance of the input distribution
     (iii)  'dMdp' - derivative of the mean of the predicted control
             wrt the controller parameters
     (iv)   'dSdm' - derivative of the variance of the predicted control
             wrt the mean of the input distribution
     (v)    'dSds' - derivative of the variance of the predicted control
             wrt the variance of the input distribution
     (vi)   'dSdp' - derivative of the variance of the predicted control
              wrt the controller parameters
     (vii)  'dCdm' - derivative of inv(s)*(covariance of the input and the
             predicted control) wrt the mean of the input distribution
     (viii) 'dCds' - derivative of inv(s)*(covariance of the input and the
             predicted control) wrt the variance of the input distribution
     (ix)   'dCdp' - derivative of inv(s)*(covariance of the input and the
             predicted control) wrt the controller parameters
policy   policy structure
  .fcn   function handle to policy
  .\ensuremath{<}\ensuremath{>}    other fields that are passed on to the policy
m        mean of the input distribution
s        covariance of the input distribution
delta    (optional) finite difference parameter. Default: 1e-4\end{verbatim}
\begin{par}
\textbf{Output arguments:}
\end{par} \vspace{1em}
\begin{verbatim}dd         relative error of analytical vs. finite difference gradient
dy         analytical gradient
dh         finite difference gradient\end{verbatim}
\begin{par}
Copyright (C) 2008-2013 by Marc Deisenroth, Andrew McHutchon, Joe Hall, and Carl Edward Rasmussen.
\end{par} \vspace{1em}
\begin{par}
Last modified: 2013-05-30
\end{par} \vspace{1em}

\begin{lstlisting}
function [dd dy dh] = conT(deriv, policy, m, s, delta)
\end{lstlisting}

\subsection*{Code}

\begin{lstlisting}
if nargin < 5; delta = 1e-4; end % default value

% if no input arguments, create random policy parameters and check derivatives
if nargin == 0
  D = 2;
  d = 3;
  policy.w = randn(D,d);
  policy.b = randn(D,1);
  m = randn(d,1);
  s = randn(d); s = s*s';
  policy.maxU = '2 3';
  policy.fcn = @(policy, m, s)conCat(@conLin, @gSat, policy, m, s);
end

D = length(policy.maxU);
d = length(m);

switch deriv
  case 'dMdm'
      [dd dy dh] = checkgrad(@conT0, m, delta, policy, s);

  case 'dSdm'
      [dd dy dh] = checkgrad(@conT1, m, delta, policy, s);

  case 'dCdm'
      [dd dy dh] = checkgrad(@conT2, m, delta, policy, s);

  case 'dMds'
      [dd dy dh] = checkgrad(@conT3, s(tril(ones(d))==1), delta, policy, m);

  case 'dSds'
      [dd dy dh] = checkgrad(@conT4, s(tril(ones(d))==1), delta, policy, m);

  case 'dCds'
      [dd dy dh] = checkgrad(@conT5, s(tril(ones(d))==1), delta, policy, m);

  case 'dMdp'
      [dd dy dh] = checkgrad(@conT6, policy.p, delta, policy, m, s);

  case 'dSdp'
      [dd dy dh] = checkgrad(@conT7, policy.p, delta, policy, m, s);

  case 'dCdp'
      [dd dy dh] = checkgrad(@conT8, policy.p, delta, policy, m, s);

end
\end{lstlisting}

\begin{lstlisting}
function [f, df] = conT0(m, policy, s)                                  % dMdm
if nargout < 2
    M = policy.fcn(policy, m, s);
else
    [M, S, C, dMdm] = policy.fcn(policy, m, s);
    df = dMdm;
end
f = M;

function [f, df] = conT1(m, policy, s)                                  % dSdm
if nargout < 2
    [M, S] = policy.fcn(policy, m, s);
else
    [M, S, C, dMdm, dSdm] = policy.fcn(policy, m, s);
     df = dSdm;
end
f = S;

function [f, df] = conT2(m, policy, s)                                  % dCdm
if nargout < 2
    [M, S, C] = policy.fcn(policy, m, s);
else
    [M, S, C, dMdm, dSdm, dCdm] = policy.fcn(policy, m, s);
    df = dCdm;
end
f = C;

function [f, df] = conT3(s, policy, m)                                  % dMds
d = length(m);
v(tril(ones(d))==1) = s; s = reshape(v,d,d); s = s+s'-diag(diag(s));
if nargout < 2
    M = policy.fcn(policy, m, s);
else
    [M, S, C, dMdm, dSdm, dCdm, dMds] = policy.fcn(policy, m, s);
    dd = length(M); dMds = reshape(dMds,dd,d,d); df = zeros(dd,d*(d+1)/2);
    for i=1:dd;
        dMdsi(:,:) = dMds(i,:,:); dMdsi = dMdsi + dMdsi'-diag(diag(dMdsi));
        df(i,:) = dMdsi(tril(ones(d))==1);
    end
end
f = M;

function [f, df] = conT4(s, policy, m)                                  % dSds
d = length(m);
v(tril(ones(d))==1) = s; s = reshape(v,d,d); s = s+s'-diag(diag(s));
if nargout < 2
    [M, S] = policy.fcn(policy, m, s);
else
    [M, S, C, dMdm, dSdm, dCdm, dMds, dSds] = policy.fcn(policy, m, s);
    dd = length(M); dSds = reshape(dSds,dd,dd,d,d); df = zeros(dd,dd,d*(d+1)/2);
    for i=1:dd; for j=1:dd
        dSdsi(:,:) = dSds(i,j,:,:); dSdsi = dSdsi+dSdsi'-diag(diag(dSdsi));
        df(i,j,:) = dSdsi(tril(ones(d))==1);
    end; end
end
f = S;

function [f, df] = conT5(s, policy, m)                                  % dCds
d = length(m);
v(tril(ones(d))==1) = s; s = reshape(v,d,d); s = s+s'-diag(diag(s));
if nargout < 2
    [M, S, C] = policy.fcn(policy, m, s);
else
    [M, S, C, dMdm, dSdm, dCdm, dMds, dSds, dCds] = policy.fcn(policy, m, s);
    dd = length(M); dCds = reshape(dCds,d,dd,d,d); df = zeros(d,dd,d*(d+1)/2);
    for i=1:d; for j=1:dd
        dCdsi = squeeze(dCds(i,j,:,:)); dCdsi = dCdsi+dCdsi'-diag(diag(dCdsi));
        df(i,j,:) = dCdsi(tril(ones(d))==1);
    end; end
end
f = C;

function [f, df] = conT6(p, policy, m, s)                               % dMdp
policy.p = p;
if nargout < 2
    M = policy.fcn(policy, m, s);
else
    [M, S, C, dMdm, dSdm, dCdm, dMds, dSds, dCds, dMdp] = policy.fcn(policy, m, s);
    df = dMdp;
end
f = M;

function [f, df] = conT7(p, policy, m, s)
policy.p = p;
if nargout < 2
    [M, S] = policy.fcn(policy, m, s);
else
    [M, S, C, dMdm, dSdm, dCdm, dMds, dSds, dCds, dMdp, dSdp] = policy.fcn(policy, m, s);
    df = dSdp;
end
f = S;

function [f, df] = conT8(p, policy, m, s)
policy.p = p;
if nargout < 2
    [M, S, C] = policy.fcn(policy, m, s);
else
    [M, S, C, dMdm, dSdm, dCdm, dMds, dSds, dCds, dMdp, dSdp, dCdp] = ...
                                                      policy.fcn(policy, m, s);
    df = dCdp;
end
f = C;
\end{lstlisting}


\chapter{Discussion}\label{c4}

As mentioned earlier, kNN compares the test sample to k neighbouring training samples to determine the class. This means all training samples have to be stored in memory \cite{Gutie} which in turn could slow down the computer, causing the classifier to take a longer time to produce results. Being a non-parametric technique \cite{16N} unlike DT and RF, kNN is prone to the curse of dimensionality \cite{Gutie}, which happens when the dimensions or number of features increases \cite{Distr}. This might explain the lower performance metric scores in comparison. There are big differences between the optimal k values for some turbines as shown in \autoref{f3} which could be due to the data for each turbine having different distributions and characteristics.

Overall, using a single multiclass-multilabel classifier with imbalanced training data produced better scores, which could be due to presence of correlations between the different turbine categories used as labels that are generalised better using this approach \cite{110}.

As the labels `gearbox' and `electrical system' are in the top three out of 14 labels used causing longest downtimes, they should have more samples classed as `faulty' and `X hours before fault' compared to other categories, which therefore should result in better performance as the classifier would have learned the characteristics of different samples belonging to the same classes. This was not the case, however, looking at the confusion matrices in \autoref{a4}. The classifier tends to misclassify `faulty' and `X hours before fault' classes as `normal' at a higher rate than the accuracy for these classes. This is still the case after removing the `curtailment' class, which saw an improvement in the accuracy of the `faulty' class. The accuracies of `X hours before fault' classes could potentially be increased by reducing the 6-hour intervals used and the maximum of 48 hours before a fault. The analysis should be repeated by reducing 48 hours to a smaller timescale, such as 12 hours, or by combining all samples that fall under this with `faulty' points to make the classification binary. Although this is likely to improve the performance, the classifier will not be able to give an indication of the timescale before a potential fault, therefore making it more difficult to decide on the appropriate action to be taken to avoid catastrophic failure in the turbine.

Based on the feature importance in \autoref{t6}, active power SCADA fields were least influential in predicting faults for both labels. The labelled power curves for `electrical system' in \autoref{f5} below shows no clear relationship between fault points and the power curve shape. There are also many overlapping `normal', `faulty' and `X hours before fault' points even after filtration of curtailment and anomalies, which could explain why this feature was less important.

\begin{figure}
  \centering
  \begin{subfigure}[t]{.5\textwidth}
    \centering
    \includegraphics[height=5cm]{../images/f5a}
    \caption{\label{f5a}}
  \end{subfigure}%
  \begin{subfigure}[t]{.5\textwidth}
    \centering
    \includegraphics[height=5cm]{../images/f5b}
    \caption{\label{f5b}}
  \end{subfigure}
  \begin{subfigure}[t]{.5\textwidth}
    \centering
    \includegraphics[height=5cm]{../images/f5c}
    \caption{\label{f5c}}
  \end{subfigure}
  \caption{\label{f5}Labelled power curve for turbine 1 with turbine category 10 (`electrical system') through the two stages of filtering out anomalous and curtailment points labelled as `normal'. The original power curve is shown in \ref{f5a}. The first stage involves a filter based on a pitch angle threshold, which produces \ref{f5b}. The second stage involves several additional filters to produce the final power curve \ref{f5c}.}
\end{figure}

The reactive power and generator speed played a bigger role in the classification for `electrical system', which makes sense considering the reactive power is produced as a result of impedance in the current due to electromagnetic fields produced by generators and transformers \cite{React}. It is likely that the features used in classification for this label are unsuitable. A fault in the electrical system would be reflected in voltages, currents, frequencies \cite{Overb} and temperature of power switchboards and cables. Electrical system faults could also be caused by environmental conditions such as lightning strikes and contact of wires with wildlife \cite{Overb}. If there are such conditions recorded as environmental downtime categories, these should be accounted for when analysing faults in the electrical system.

The `gearbox' label was also found to perform poorly for these classes, despite having a higher mean F1 score than `electrical faults', which could be due to feature selection as well. Statistics from the National Renewable Energy Laboratory's gearbox failure database indicate that most faults are caused by bearings, gears and other components including filtration and lubrication systems \cite{Stati15}. These are mostly due to wear, fatigue and cracks \cite{Sheng11} and may be detected with higher accuracy if the features include quantities such as torque, oil pressure and gearbox temperature.

The SCADA data provided for this project only had 17 SCADA fields, of which 10 are used as features. This did not include voltages, currents, frequencies, torques or temperature readings, but the SCADA system for the turbine model used does measure these parameters. When a more complete SCADA data is available, the evaluation should be done by increasing the number of features to include these fields. The role of environmental conditions on failures could explain why wind speed and direction were very influential in detecting faults in the two labels analysed, although further in-depth analysis is required to verify this.

Balancing the training dataset improved the classification accuracy of `X hours before fault' classes slightly. An overall improved model may be developed by oversampling only these classes for training, but there is a trade-off between this and the training time and computational resources required.

The dataset could have incorrect readings in the SCADA fields caused by broken or unresponsive sensors which are not detected as unusual when the downtime data is used in labelling. This was why a power threshold before cut-in speed was applied to remove redundant data points, by visually inspecting the power curve of the turbine as seen in \autoref{f1}. The dataset should be manually inspected for all other features using curves such as pitch versus power and power versus rotor speed, to see if there are any other incorrect values previously undetected which may affect the classifier's accuracy in detecting faults. The rows of data corresponding to these values should then be excluded from the training data.

\section{Future work}

In addition to possible areas for future work discussed above, the following were identified.

When more data is available, the analysis should be repeated using historic datasets spanning the life of the turbine. Historic data would have recorded the different states a turbine has experienced over its life, and therefore when a classifier is trained on this, it could detect future turbine states easier. However, this will mean the training data will be bigger, which in turn causes longer training time and more computing resources to be used. Another area of work is to test the performance of a classifier using different lengths of datasets for training while keeping the testing set and hyperparameter settings constant. This will allow for the most appropriate length of dataset for training to be determined based on the resources available to produce satisfactory results in terms of training time and classification accuracy.

After a classifier has been trained and used in practice, its performance over time should be monitored. If the performance is found to diminish over time or after a major component replacement, the classifier should be retrained using recent data. As the classification makes a distinction between the different faults, the ability to alert relevant maintenance professionals for a specific fault automatically is possible.

Instead of using turbine categories in the downtime data, which is supervised, for labelling, the alarm logs, which are unsupervised, can instead be used to compare the results. The number of alarm logs for the turbines used, however, is 480, compared to 23 turbine categories. This will mean the number of labels will be much higher, which will cause longer computational time. A solution to reduce this is to group similar alarms into one class. Another approach would be to use a single label with each alarm as a separate class, but there is likely to be overlap between classes when fault prediction is also included. If actual failure records are available, a comparison can be made between the predicted classes, actual classes, and actual failures that have occurred and their costs.

Further optimisation of hyperparameters is possible, such as finding the optimal number of estimators for RF. Each optimisation takes time and is limited by the specifications of the computer used, which is why this was not carried out in this project. Detailed analysis done on the results using RF for two labels above should be repeated for each label and classifier used for fair comparisons to be made.

The methodology could also be tested for wind turbines of different models in different sites. Provided these turbines have similar SCADA data with downtime records, only slight modifications to the codes, such as the data source, field names, and number of features, would be required in order to be used on other turbine models.

In industry, a cost function analysis needs to be done prior to implementing this fault detection method. It is defined as the cost of a false alarm (false positive) or failing to detect a fault in advance (false negative). False positives and false negatives both incur charges. The first is due to transporting labour and equipment to site, which could be expensive especially for sites in harsh environments, such as offshore wind farms. The second would cause unscheduled downtime, the loss of revenue due to no power generation and replacement of turbine components due to irreversible damage. This cost should then be compared to the cost of alternatively using a condition monitoring system, and the overall cost of running the wind farm or wind turbine. The analysis should give an indication on which performance metric is more important; if the cost of false negatives is more, attention should be paid to the recall score, while the precision is more important if false positives cost more \cite{deRu15}.