%% ----------------------------------------------------------------
%% Chapter 5: A machine-learning model for SANS
%% ----------------------------------------------------------------
%!TeX root = subfile
\label{chapter:zanspy}
\documentclass[../main.tex]{subfiles}
% ----------------------------------------------------------------
\begin{document}

The modelling of the SANS closure terms is critical for the prediction of 3-D spanwise-averaged dynamics in 2-D systems.
An standard EVM, such as the Smagorinsky model, has failed to correctly predict the SANS closure terms because of if intrinsic dissipative and linear behaviour.
While other physical models allowing energy backscatter (energy transfer from small to large scales) such as the dynamic Smagorinsky model or a transport equation model could be explored in the future, in this chapter a data-driven model is proposed.
The limited knowledge on the spanwise stresses dynamics motivates this choice.
In particular, a supervised machine-learning (ML) model based on a deep convolutional neural network has been designed for this purpose.
A-priori results show up to 92\% correlation between target and predicted closure terms; more than an order of magnitude better than the EVM correlation.
Additionally, the ML model performance is assessed for a different flow configuration (Reynolds regime and body shape) to the training case and high correlation values are still captured.
The new SANS equations and ML closure model are also used for a-posteriori prediction.
Even though evidence of known stability issues with long-time ML predictions for dynamical systems is found for the present data-driven model, the closed SANS simulations are still capable of predicting wake metrics and induced forces with errors from 1-10\%.
This results in approximately an order of magnitude improvement over standard 2-D simulations while reducing the computational cost of 3-D simulations by 99.5\%.

\section{Introduction and literature review}

In recent years, machine learning, i.e. algorithms that reveal information by processing raw data inputs, have made an impact across numerous disciplines: from image recognition, to improved web searches, content filtering, natural language processing, among others \citep{LeCun2015}.
The fluid mechanics community has also adopted ML techniques both in experimental and computational work, specifically with the use of deep-learning methods \citep{Kutz2017}, where multiple layers of trainable parameters are stacked sequentially.
As data-driven models learn from observations, the vast amounts of data generated through field measurements and high-fidelity simulations has accelerated the application of ML models to fluid flow problems \citep{Brunton2020}.

Supervised learning, or model optimisation based on labelled examples, has yielded the exploration of novel flow classification, control, and regression techniques.
Within classification and uncertainty quantification, detection of low RANS accuracy regions in the flow has been accomplished with the use of different ML strategies such as support vector machines and random trees \citep{Ling2015}.
Deep artificial neural networks (ANNs) have been employed for uncertainty quantification in RANS \citep{Geneva2019}, pointwise classification and blending of appropriate turbulence closures in LES \citep{Maulik2019}, and discovery of active flow control strategies such as drag reduction in turbulent channel flow \citep{Lee1997}.

Different control strategies have been developed using reinforcement learning which, in contrast to supervised learning, does not rely in labelled examples and the algorithm objective is to maximise a long-term reward signal based on self-proposed actions \citep{SuttonBarto2018}.
Reinforcement learning has been used for (e.g.) active wake flow control \citep{Rabault2019a,Rabault2019b} and efficient control of collective swimmers \citep{Verma2018}.
Other unsupervised learning models such as generative adversarial networks \citep{Goodfellow2014} have been used for unsteady flow field generation \citep{Lee2019}.

The use of ML models for regression problems in fluid flows has also bloomed in the last few years. ANNs have been used for the reconstruction of turbulent spatio-temporal sampled data \citep{Deng2019}, physics-based discovery of governing equations parameters such as Reynolds number \citep{Raissi2019}, quantitative prediction of flow fields from scalar concentration snapshots \citep{Raissi2020}, and RANS modelling \citep{Duraisamy2015,Ling2016,Lapeyre2019}, among others.
Convolutional neural networks (CNNs) have also become a popular tool to solve regression problems.
CNNs have been used for turbulent flow field super-resolution \citep{Fukami2019,Liu2020}, modelling of surface pressure distribution from sparse wake flow data \citep{Ye2020}, nonlinear mode decomposition \citep{Murata2019}, and prediction of turbulent fields \citep{Lapeyre2019,Kim2020b,Kim2020a}, among others.

Specifically within turbulence closures, CNNs have been applied in RANS \citep{Weatheritt2016} and LES \citep{Gamahara2017,Beck2019} frameworks, establishing its utility for revealing hidden correlations between system variables for which physical laws are, a-priori, unknown \citep{Brenner2019}.
In particular, CNNs have been developed to efficiently learn spatial structures in data arrays by exploiting the translational invariance inherent in convolutions \citep{LeCun1995}.
This makes them a very attractive tool for modelling novel and spatially-complex fields such as the new SANS closures.

With respect to the strip-based SANS framework, the general methodology of the data-driven ML model is based on the idea that training a SANS closure on a single local spanwise segment (see \fref{fig:strips}) automatically generalises to all the spanwise-averaged strips because of the system homogeneity in that direction.
Hence, the database required to train the model can be generated in a single 3-D simulation with a span equivalent to the local spanwise-averaging length.
For example, a structure of aspect ratio 100 can be reduced to 100 strips, or 2-D simulations, resolving the (1-diameter) SANS equations.
The required SANS closure model can be previously trained using data of a 3-D high-fidelity simulation of a 1-diameter-long spanwise-periodic cylinder, noting that the 3-D simulation is the most computationally expensive part of the method as detailed in \aref{chapter:appendixE}.
In this chapter, the ML SANS closure model is assessed for a single spanwise-averaged strip of 1 diameter.

\section{Machine-learning model} \label{sec:ML_model}

Data-driven closures are particularly helpful when physical information of the system to be modelled is not known a-priori, and this is certainly the case for the novel spanwise stresses developed in this work.
Hence, in this section we present a ML model for the spanwise fluctuations thus providing closure to the SANS equations.

A CNN is proposed to map resolved quantities, $\mathrm{X}_n=\left\lbrace U,V,P\right\rbrace$, to closure terms, $\mathrm{Y}_n$.
The CNN architecture is depicted in \fref{fig:cnn2}.
The closure terms of the a-priori analysis are the SSR tensor components (similarly to \sref{sec:evm}), $\mathrm{Y}_n=\left\lbrace \avg{u\p u\p},\avg{u\p v\p},\avg{v\p v\p}\right\rbrace$.
On the other hand, the perfect closure is targeted as the output of the a-posteriori analysis, $\mathrm{Y}_n=\lbrace \mathcal{S}^R_x,\mathcal{S}^R_y\rbrace$, in order to improve the model stability as will be discussed in \sref{sec:ML_a-post}.

The ML model architecture is based on a multiple-input multiple-output CNN (MIMO-CNN).
The need of a high-dimensional tunable function that can reveal hidden correlations in spatial data motivates the CNN choice \citep{LeCun1989}.
The same model architecture is used for both a-priori and a-posteriori analysis, where only the number of outputs changes from three to two, respectively.
Each input branch is composed of an encoding part and a decoding part.
The number of filters doubles at every convolutional layer in the encoding part, reaching a maximum of 64 filters, and halves in the decoding part.
Each convolutional layer consists of a 2-D convolution operation with a $5\times5$ filter (convolution kernel), a batch normalisation operation, and a rectified linear unit (ReLU) activation function.
The total number of trainable parameters is 944,739.
The batch normalisation helps to speed up the neural network learning rate and improves its stability with respect to the random weights initialisation \citep{Ioffe2015}.
The ReLU function allows higher learning rates and handles vanishing-gradient problems better than other standard activation functions such as the Sigmoid or the $\tanh$ functions \citep{Nair2010}.
All input branches are then concatenated into a single branch, for which another encoding-decoding branch follows.
The last layer consists of a wake mask function and a Gaussian filter.

The wake mask helps providing a clean output in the regions of the domain where the closure terms are negligible.
The wake mask is based in the vorticity field for which the condition $|\Omega_z|>\epsilon$ is applied.
Hence, a field of 1s (and 0s) is generated when the condition is met (or not).
The wake mask for the a-priori stage considers all the points within the wake limit.
The wake limit is activated using the $\epsilon=1\cdot10^{-3}$ threshold (\fref{fig:wake1}).
This helps the model focusing on the non-trivial region during optimisation while still providing some areas within the wake width where the closure term is not active.
The wake mask for the a-posteriori stage applies the criterion in a pointwise manner, hence points within the wake region might not be activated, particularity in the far wake region (\fref{fig:wake2}).
Also, the criterion is more strict by setting $\epsilon=3.5\cdot10^{-3}$.
This is enforced to improve the stability of the a-posteriori analysis so longer temporal dynamics can be evaluated.
Finally, a Gaussian filter of $\sigma=0.5$ (resulting in a $5\times5$ kernel matching the feature maps size) is applied to the output of the wake detection layer.

\begin{figure}[t]
\centering
\includegraphics[width=0.95\linewidth]{chapter5/cnn/cnn}\vspace{-0.1cm}
\caption{CNN architecture.
An encoding-decoding branch is considered for each input field.
The input branches are then concatenated $(\frown)$ and another encoding-decoding branch follows.
The closure terms are separately extracted on the last layer.
Arrows legend: Dark blue: Conv2D $5\times5$ $+$ batch normalisation $+$ activation function.
Yellow: MaxPooling $2\times2$.
Magenta: UpSampling $2\times2$.
Green: Conv2D $1\times1$ $+$ wake mask $+$ Gaussian filter.
Block thickness indicates the number of filters, which double/halve at each convolutional operation in the encoding/decoding parts.}
\label{fig:cnn2}
\end{figure}

\begin{figure}[!ht]
\centering
\begin{subfigure}{0.7\linewidth}
 	\includegraphics[width=\linewidth]{chapter5/wake/vort}
 	\caption{$\Omega_z$}
\end{subfigure}
\begin{subfigure}{0.7\linewidth}
    \includegraphics[width=\linewidth]{chapter5/wake/SRx}
	\caption{$\mathcal{S}^R_x$}
\end{subfigure}
\begin{subfigure}{0.7\linewidth}
	\includegraphics[width=\linewidth]{chapter5/wake/wake1}
	\caption{Wake mask used in the a-priori analysis.}\label{fig:wake1}
\end{subfigure}
\begin{subfigure}{0.7\linewidth}
	\includegraphics[width=\linewidth]{chapter5/wake/wake2}
	\caption{Wake mask used in the a-posteriori analysis.}\label{fig:wake2}
\end{subfigure}
\caption{A wake mask from the vorticity field is generated to focus the ML model outputs on the non-trivial wake region alone. The aim is to match the region where the target output $(\mathcal{S}^R_x)$ is activated. In the a-posteriori analysis, the far-wake region is not fully activated in order to aid the closure stability.}
\end{figure}

A detailed hyper-parametric study of the ML model has been conducted for the circular cylinder case (similarly to \sref{sec:SANS_circular_cylinder}) in \cite{Font2020a}, and is also included in \aref{chapter:appendixD}.
In short, the present model architecture was compared against two different MIMO-CNNs, and this architecture offered the best accuracy among the tested models.
Also, it was found that the larger the CNN (in terms of trainable parameters), the faster high correlation coefficients were obtained for a similar number of training epochs (where epoch refers to an optimisation iteration on the training dataset).
The number of trainable parameters was modified by changing the number of filters per layer.
The sum of the squared error as a loss function provided the best performance when compared to the sum of the absolute error.
The use of primitive variables (velocity and pressure) did not yield significant differences when compared to the velocity gradient tensor components set.
Both these input sets showed a better performance than a third input set containing the vorticity field and the distance function to the cylinder.

Unsteady turbulent flow past a circular cylinder with a 1 diameter span $(L_z=1)$ at $Re=10^4$ (as considered in \sref{sec:SANS_circular_cylinder}) has been used to generate a dataset containing a total of 7000 samples of $1216\times540$ size during $\Delta t^*=1650$ and using a sampling rate of $\delta t^*=0.25$.
The 1 diameter span is sufficiently long to sustain 3-D turbulence on a meaningful wake region and it is also a sensible choice of the minimum structural mode wavelength that would need to be resolved in a SANS-based strip-theory framework.
The input fields are normalised with the standard score, i.e. $\hat{x} = (x-\overline{x})/\sigma_x$.
As in \cite{Beck2019, Kim2020a}, a single $Re$ regime is used in the dataset generation, which is deemed sufficient because of the turbulent nature and unsteadiness of the flow.
The constant energy transfer rate across different scales in the inertial subrange of turbulence implies that closures trained on one $Re$ should generalise reasonably well to unseen $Re$ as long as the mean flow characteristics are similar.
Such study is conducted in \sref{sec:generalisation}.

The dataset is split with 6000 samples for optimisation (training), 500 samples for validation during training, and 500 for evaluation (testing) purposes.
The dataset split is chosen as often encountered in the machine learning literature, probably adopted from power law (or Pareto) distributions \citep{Newman2005}.
To ensure the closure's ability to generalise and avoid overfitting, the time windows for testing, optimisation and validation do not overlap, i.e. the closure is assessed on a continuous window of previously unseen flow.
A mini-batch stochastic gradient descent method based on the Adam optimiser \citep{Kingma2014} is used to update the network weights during training.
The sum of the squared error function is used for optimisation
\begin{equation}
\mathrm{Loss}=\sum^m_i\sum^n_j \pars{\mathrm{Y}_{i,j}-\mathrm{Y}_{i,j}^{\mathrm{ML}}}^2,
\end{equation}
where $m$ is the size (number of samples) of the mini-batch, $n$ is the output index, and the superscript $(\cdot)^{\mathrm{ML}}$ denotes the ML model prediction.
When the validation error does not decrease for 5 epochs or a total of 60 epochs is reached, training is stopped (a.k.a. early-stopping).

\begin{figure}[!t]
\setlength{\columnsep}{-1pt}
\begin{multicols}{2}
\centering
\includegraphics[width=1.02\linewidth]{chapter5/a-priori/history_ab}
\caption{Training of the $\mathrm{Y}_n=\big\lbrace\langle u' u'\rangle,\langle u' v'\rangle,\langle v' v'\rangle\big\rbrace$ output set.
The vertical red dashed line indicates the early stop.
Grey: combined training loss.
Black: combined validation loss.
Blue: $\langle u' u'\rangle$ correlation coefficient.
Purple: $\langle u' v'\rangle$ correlation coefficient.
Cyan: $\langle v' v'\rangle$ correlation coefficient.}\label{fig:ab_history}

\includegraphics[width=1.02\linewidth]{chapter5/a-priori/history_SR}
\caption{Training of the $\mathrm{Y}_n=\big\lbrace \mathcal{S}^R_x, \mathcal{S}^R_y \big\rbrace$ output set.
Grey: combined training loss.
Black: combined validation loss.
Blue: $\mathcal{S}^R_x$ correlation coefficient.
Purple: $\mathcal{S}^R_y$ correlation coefficient.}
\label{fig:SR_history}
\end{multicols}
\end{figure}

The ML model training history of the SSR tensor components is plotted in \fref{fig:ab_history}, where a rapid convergence cap be appreciated.
The early-stopping criterion is triggered on the 46th epoch since it provides the lowest validation error not improved for the next 5 epochs.
On the other hand, the training history of the perfect closure components is plotted in \fref{fig:SR_history}.
In this case, the early-stopping criterion is not triggered and optimisation is performed during 60 epochs (the maximum defined).

\section{Results}

\subsection{A-priori analysis}

To allow a comparison with the EVM, the anisotropic part of the SSR tensor is recovered from the ML model predictions of the full SSR tensor.
The fact that the ML model yields the full SSR tensor already provides an advantage with respect to the EVM since no further modifications of the governing equations are required, nor an additional transport equation for the SSR kinetic energy.

The predicted anisotropic SSR tensor components $(\boldsymbol\tau^{r,\,\mathrm{ML}}_{11}, \boldsymbol\tau^{r,\,\mathrm{ML}}_{12}, \boldsymbol\tau^{r,\,\mathrm{ML}}_{22})$ are depicted in \fref{fig:ML_a-priori}, and the predicted components of the full SSR tensor $(\avg{u\p u\p}^{\mathrm{ML}}, \avg{v\p v\p}^{\mathrm{ML}})$ are depicted in \fref{fig:ML_ab_a-priori}.
It can be appreciated that large-scale structures are overall correctly captured.
On the other hand, small-scale structures, such as those encountered in near-wake region, are more difficult to predict.
Similar issues have been found for other CNN-based data-driven models \citep{Lee2019}.
A Gaussian filtering operation together with a wake detection feature aids the CNN to provide less noisy predictions focused on the wake region alone.
This helps mitigating error propagation in an a-posteriori framework (see next section), however, it can have an impact on the small-scale structures as well.

A significant improvement is qualitatively observed when comparing the predictions against the EVM, where the latter fails to capture almost all flow structures except for the largest ones (\fref{fig:EVM_a-priori}).
Furthermore, correlation values above $90\%$ have been achieved for all the anisotropic SSR tensor components (\tref{tab:ML_a-priori}), as well as the full SSR tensor components (\tref{tab:ML_ab_a-priori}).
It can be concluded that the ML model provides a significantly better prediction of the closure terms compared to the EVM, which is critical for SANS to correctly model turbulence wake dynamics.

\begin{figure}[!ht]
\centering
\begin{subfigure}[t]{0.6\linewidth}
    \includegraphics[width=\linewidth]{chapter5/a-priori/tau_r_11_lowres}
    \caption{$\boldsymbol\tau^r_{11}$}
\end{subfigure}
\begin{subfigure}[t]{0.6\linewidth}
    \includegraphics[width=\linewidth]{chapter5/a-priori/tau_r_11_ML_lowres}
    \caption{$\boldsymbol\tau^{r,\,\mathrm{ML}}_{11}$}
\end{subfigure}
\begin{subfigure}[t]{0.6\linewidth}
    \includegraphics[width=\linewidth]{chapter5/a-priori/tau_r_12_lowres}
    \caption{$\boldsymbol\tau^r_{12}$}
\end{subfigure}
\begin{subfigure}[t]{0.6\linewidth}
    \includegraphics[width=\linewidth]{chapter5/a-priori/tau_r_12_ML_lowres}
    \caption{$\boldsymbol\tau^{r,\,\mathrm{ML}}_{12}$}
\end{subfigure}
\begin{subfigure}[t]{0.6\linewidth}
    \includegraphics[width=\linewidth]{chapter5/a-priori/tau_r_22_lowres}
    \caption{$\boldsymbol\tau^r_{22}$}
\end{subfigure}
\begin{subfigure}[t]{0.6\linewidth}
    \includegraphics[width=\linewidth]{chapter5/a-priori/tau_r_22_ML_lowres}
    \caption{$\boldsymbol\tau^{r,\,\mathrm{ML}}_{22}$}
\end{subfigure}
\caption{ML model prediction of the anisotropic SSR tensor components compared to reference data.}
\label{fig:ML_a-priori}
\end{figure}

\begin{table}[t]
\centering
\caption{Correlation coefficients between components of the anisotropic SSR tensor and predictions provided by the EVM and the ML model.}
\begin{tabular}{lrrr}
\toprule
$\mathcal{CC}$ & $\boldsymbol\tau^r_{11}$ & $\boldsymbol\tau^r_{12}$ & $\boldsymbol\tau^r_{22}$ \\
\midrule
EVM &  0.06& 0.06 & 0.14\\
ML &  0.90   & 0.92 & 0.92\\
\bottomrule \label{tab:ML_a-priori}
\end{tabular}
\end{table}

\begin{table}[t]
\centering
\caption{Correlation coefficients between target SSR tensor components and ML model predictions.}
\begin{tabular}{lrrr}
\toprule
$\mathcal{CC}$ & $\avg{u\p u\p}$ & $\avg{u\p v\p}$ &$\avg{v\p v\p}$  \\
\midrule
ML & 0.95 & 0.92 & 0.96\\
\bottomrule \label{tab:ML_ab_a-priori}
\end{tabular}

{\footnotesize The correlation coefficients are calculated for the 500 snapshots of the test dataset and the average values are provided. \par}
\end{table}

\begin{figure}[!ht]
\centering
\begin{subfigure}[t]{0.6\linewidth}
    \includegraphics[width=\linewidth]{chapter5/a-priori/tau_R_11_lowres}
    \caption{$\avg{u\p u\p}$}
\end{subfigure}
\begin{subfigure}[t]{0.6\linewidth}
    \includegraphics[width=\linewidth]{chapter5/a-priori/tau_R_11_ML_lowres}
    \caption{$\avg{u\p u\p}^{\mathrm{ML}}$}
\end{subfigure}
\begin{subfigure}[t]{0.6\linewidth}
    \includegraphics[width=\linewidth]{chapter5/a-priori/tau_R_22_lowres}
    \caption{$\avg{v\p v\p}$}
\end{subfigure}
\begin{subfigure}[t]{0.6\linewidth}
    \includegraphics[width=\linewidth]{chapter5/a-priori/tau_R_22_ML_lowres}
    \caption{$\avg{v\p v\p}^{\mathrm{ML}}$}
\end{subfigure}
\caption{ML model prediction of the full SSR tensor components compared to reference data.}
\label{fig:ML_ab_a-priori}
\end{figure}

\clearpage

\subsection{Generalisation across flow regimes and geometries}\label{sec:generalisation}

So far, it has been demonstrated that the trained ML model can correctly predict the SANS closure terms of previously unseen snapshots of the cylinder case, i.e. the model has correctly generalised in time.
Here, the generalisation of the ML model to other flow configurations such as body shapes and Reynolds regimes is assessed.
This analysis is important because, first, it provides information on whether training the ML model in a fixed set-up can still be useful for the prediction of the SANS stresses in a different case.
Second, it is a good test to check that the ML model has not over-fitted the training data even with the use of the early-stopping criterion.

Four different cases are studied:
Flow past a circular cylinder at $Re=1000$.
Flow past a circular cylinder at $Re=3900$.
Flow past an ellipse with axis $(x,y)=(2D,D)$ (ellipse 1) at $Re=10^4$.
Flow past an ellipse with axis $(x,y)=(0.5D,D)$ (ellipse 2) at $Re=10^4$.
Similarly to the circular cylinder case at $Re=10^4$, a test dataset of 500 snapshots has been generated during $\Delta t^*=125$ time units after reaching a statistically stationary state of the wake for all cases.
Both output target sets, i.e. the full SSR tensor components and the perfect closure components, are analysed.

\tref{tab:ML_generalisation} quantifies the accuracy of the ML model in terms of mean correlation coefficient to target data.
Note that correlation values are provided for two wake regions, and these can be visualized in \fref{fig:cc_regions}:
one involving the non-trivial wake region (same region as used in previous results, noted in green dashed lines) and one discarding the region surrounding the body (noted in blue dashed lines), i.e. $x\in[0.75D_x,12D_y]$ where $D_x$ and $D_y$ are the streamwise and crossflow body lengths, respectively.
With respect to the model generalisation in different Reynolds regimes, its prediction ability on the $Re=3900$ case is similar to the training case ($Re=10^4$) since the correlation values provided in both cases are very close.
For the $Re=1000$ case, similar results are also obtained to the training case, specially when the near-body region is cropped out of the correlation analysis.
Qualitatively, the ML model prediction of the spanwise stresses is displayed in \fref{fig:ML_generalisation_R1} and \fref{fig:ML_generalisation_R2} for the $Re=1000$ and $Re=3900$ cases, respectively.
Again, a similar performance to the training case is observed;
mid- and large-scale structures are correctly captured whereas small-scale structures are not reconstructed as in the target data.
For the $Re=1000$ case, it can also be observed how the ML model struggles to predict the spanwise stresses in the region near to the body.

\begin{figure}[t]
\centering
\includegraphics[width=0.75\linewidth]{chapter5/generalisation/cc_regions}
\caption{Regions used for the ML model analysis.
Black: Input and output region of the ML model.
Green dashed: Region used to compute the correlation coefficient between ML predictions and target data (full wake).
Blue dotted: Region used to compute the correlation coefficient between ML predictions and target data omitting the near-body domain (downstream wake).}
\label{fig:cc_regions}
\end{figure}

Regarding the ellipse cases at $Re=10^4$, high correlation values are observed when the near-body region is cropped out of the correlation analysis as displayed \tref{tab:ML_generalisation}.
On the other hand, the ML model fails to correctly predict the spanwise stresses in the shear-layer region.
This behaviour can be appreciated in \fref{fig:ML_generalisation_E1} and \fref{fig:ML_generalisation_E2}, where patches of high-intensity spanwise stresses arise at the vicinity of the body.
To a lesser extent, such discrepancies can also be observed for the $Re=1000$ and $Re=3900$ circular cylinder cases.
It indicates that training the ML model in a single flow configuration generalises well for lower Reynolds regimes and body shape across the wake except for the the shear-layer region, which performs better when it resembles the training data.
It is important to stress that the shear layer information contained in the training database is limited, so the spatial patterns learnt by the CNN related to this region cannot be recombined for previously unseen cases.
This limitation has also been reported in \citet{Lee2019}, where flow past a circular cylinder at $Re=300$ and $Re=500$ is used as training data and the model is then tested at $Re=150,400,3900$.
Similarly, their results show a better agreement for the test cases that resemble the flow within the training Reynolds regime.
Another issue arising in our generalisation results is the over- or under-prediction of the spanwise stresses intensity (not reflected in the correlation coefficients), noting that the ML model predictions and the reference data are plotted using the same scale.
This can be observed in \fref{fig:ML_generalisation_E1_uv}.
The current input data normalization method, based in each snapshot standard score, is a factor that might result in this difference.

Overall, the ML model has shown an excellent performance on the generalisation across lower $Re$ and different body shapes providing correlation values similar to the training case.
Additionally, this indicates that the training data is not over-fitted.
Two limitations on the generalisation have been observed: the prediction of the spanwise stresses in the shear-layer region and the over- or under-prediction of the spanwise stresses intensity in some cases.

\clearpage

\begin{table}[t]
\centering
\caption{Correlation coefficients between target data and ML model predictions for the different generalisation cases. FW and DW refers to the correlation coefficient for the green-dashed region and blue-dotted region of \fref{fig:cc_regions}, respectively.}
        \begin{tabular}{ccccccc}
            \toprule
            \multicolumn{1}{c}{Case}& $\mathcal{CC}$ & $\avg{u\p u\p}$ & $\avg{u\p v\p}$ &$\avg{v\p v\p}$ & $\mathcal{S}^R_x$ & $\mathcal{S}^R_y$ \\
            \midrule
            \multirow{2}{*}{Cylinder, $Re=1000$} & \multicolumn{1}{c}{FW} & \multicolumn{1}{c}{0.86} & \multicolumn{1}{c}{0.90} & \multicolumn{1}{c}{0.92} & \multicolumn{1}{c}{0.86} & \multicolumn{1}{c}{0.91} \\
                                & \multicolumn{1}{c}{DW} & \multicolumn{1}{c}{0.94} & \multicolumn{1}{c}{0.94} & \multicolumn{1}{c}{0.96} & \multicolumn{1}{c}{0.93} & \multicolumn{1}{c}{0.95} \\
            \midrule
            \multirow{2}{*}{Cylinder, $Re=3900$} & \multicolumn{1}{c}{FW} & \multicolumn{1}{c}{0.94} & \multicolumn{1}{c}{0.91} & \multicolumn{1}{c}{0.95} & \multicolumn{1}{c}{0.89} & \multicolumn{1}{c}{0.92} \\
                                & \multicolumn{1}{c}{DW} & \multicolumn{1}{c}{0.96} & \multicolumn{1}{c}{0.93} & \multicolumn{1}{c}{0.96} & \multicolumn{1}{c}{0.91} & \multicolumn{1}{c}{0.93} \\
            \midrule
            \multirow{2}{*}{Ellipse 1, $Re=10^4$} & \multicolumn{1}{c}{FW} & \multicolumn{1}{c}{0.61} & \multicolumn{1}{c}{0.60} & \multicolumn{1}{c}{0.79} & \multicolumn{1}{c}{0.67} & \multicolumn{1}{c}{0.69} \\
                                & \multicolumn{1}{c}{DW} & \multicolumn{1}{c}{0.93} & \multicolumn{1}{c}{0.90} & \multicolumn{1}{c}{0.95} & \multicolumn{1}{c}{0.86} & \multicolumn{1}{c}{0.91} \\
            \midrule
            \multirow{2}{*}{Ellipse 2, $Re=10^4$} & \multicolumn{1}{c}{FW} & \multicolumn{1}{c}{0.91} & \multicolumn{1}{c}{0.21} & \multicolumn{1}{c}{0.92} & \multicolumn{1}{c}{0.07} & \multicolumn{1}{c}{0.06} \\
                                & \multicolumn{1}{c}{DW} & \multicolumn{1}{c}{0.94} & \multicolumn{1}{c}{0.91} & \multicolumn{1}{c}{0.94} & \multicolumn{1}{c}{0.89} & 0.89 \\
            \bottomrule
        \end{tabular}\label{tab:ML_generalisation}

\vspace{0.5cm}
{\footnotesize The correlation coefficients are calculated for the 500 snapshots of the test dataset and the average values are provided. \par}
\end{table}

\begin{figure*}[!ht]
\begin{multicols}{2}
\begin{subfigure}{\linewidth}
    \includegraphics[width=\linewidth]{chapter5/generalisation/Re1000/tau_R_11_lowres}
    \caption{$\avg{u\p u\p}$}
\end{subfigure}
\begin{subfigure}{\linewidth}
    \includegraphics[width=\linewidth]{chapter5/generalisation/Re1000/tau_r_12_lowres}
    \caption{$\avg{u\p v\p}$}
\end{subfigure}
\begin{subfigure}{\linewidth}
    \includegraphics[width=\linewidth]{chapter5/generalisation/Re1000/SRx_lowres}
    \caption{$\mathcal{S}^R_x$}
\end{subfigure}
\begin{subfigure}{\linewidth}
    \includegraphics[width=\linewidth]{chapter5/generalisation/Re1000/tau_R_11_ML_lowres}
    \caption{$\avg{u\p u\p}^{\mathrm{ML}}$}
\end{subfigure}
\begin{subfigure}{\linewidth}
    \includegraphics[width=\linewidth]{chapter5/generalisation/Re1000/tau_r_12_ML_lowres}
    \caption{$\avg{u\p v\p}^{\mathrm{ML}}$}
\end{subfigure}
\begin{subfigure}{\linewidth}
    \includegraphics[width=\linewidth]{chapter5/generalisation/Re1000/SRx_ML_lowres}
    \caption{$\mathcal{S}_x^{R,\,\mathrm{ML}}$}
\end{subfigure}
\end{multicols}
\caption{Cylinder, $Re=1000$ case: ML model predictions of components of the SSR tensor and the perfect closure compared to reference data.}
\label{fig:ML_generalisation_R1}
\end{figure*}

\begin{figure*}[!ht]
\begin{multicols}{2}
\begin{subfigure}{\linewidth}
    \includegraphics[width=\linewidth]{chapter5/generalisation/Re3900/tau_R_11_lowres}
    \caption{$\avg{u\p u\p}$}
\end{subfigure}
\begin{subfigure}{\linewidth}
    \includegraphics[width=\linewidth]{chapter5/generalisation/Re3900/tau_r_12_lowres}
    \caption{$\avg{u\p v\p}$}
\end{subfigure}
\begin{subfigure}{\linewidth}
    \includegraphics[width=\linewidth]{chapter5/generalisation/Re3900/SRx_lowres}
    \caption{$\mathcal{S}^R_x$}
\end{subfigure}
\begin{subfigure}{\linewidth}
    \includegraphics[width=\linewidth]{chapter5/generalisation/Re3900/tau_R_11_ML_lowres}
    \caption{$\avg{u\p u\p}^{\mathrm{ML}}$}
\end{subfigure}
\begin{subfigure}{\linewidth}
    \includegraphics[width=\linewidth]{chapter5/generalisation/Re3900/tau_r_12_ML_lowres}
    \caption{$\avg{u\p v\p}^{\mathrm{ML}}$}
\end{subfigure}
\begin{subfigure}{\linewidth}
    \includegraphics[width=\linewidth]{chapter5/generalisation/Re3900/SRx_ML_lowres}
    \caption{$\mathcal{S}_x^{R,\,\mathrm{ML}}$}
\end{subfigure}
\end{multicols}
\caption{Cylinder, $Re=3900$ case: ML model predictions of components of the SSR tensor and the perfect closure compared to reference data.}
\label{fig:ML_generalisation_R2}
\end{figure*}

\begin{figure*}[!ht]
\begin{multicols}{2}
\begin{subfigure}{\linewidth}
    \includegraphics[width=\linewidth]{chapter5/generalisation/ellipse_1/tau_R_11_lowres}
    \caption{$\avg{u\p u\p}$}
\end{subfigure}
\begin{subfigure}{\linewidth}
    \includegraphics[width=\linewidth]{chapter5/generalisation/ellipse_1/tau_r_12_lowres}
    \caption{$\avg{u\p v\p}$}
\end{subfigure}
\begin{subfigure}{\linewidth}
    \includegraphics[width=\linewidth]{chapter5/generalisation/ellipse_1/SRx_lowres}
    \caption{$\mathcal{S}^R_x$}
\end{subfigure}
\begin{subfigure}{\linewidth}
    \includegraphics[width=\linewidth]{chapter5/generalisation/ellipse_1/tau_R_11_ML_lowres}
    \caption{$\avg{u\p u\p}^{\mathrm{ML}}$}
\end{subfigure}
\begin{subfigure}{\linewidth}
    \includegraphics[width=\linewidth]{chapter5/generalisation/ellipse_1/tau_r_12_ML_lowres}
    \caption{$\avg{u\p v\p}^{\mathrm{ML}}$} \label{fig:ML_generalisation_E1_uv}
\end{subfigure}
\begin{subfigure}{\linewidth}
    \includegraphics[width=\linewidth]{chapter5/generalisation/ellipse_1/SRx_ML_lowres}
    \caption{$\mathcal{S}_x^{R,\,\mathrm{ML}}$}
\end{subfigure}
\end{multicols}
\caption{Ellipse 1, $Re=10^4$ case: ML model predictions of components of the SSR tensor and the perfect closure compared to reference data.}
\label{fig:ML_generalisation_E1}
\end{figure*}

\begin{figure*}[!ht]
\begin{multicols}{2}
\begin{subfigure}{\linewidth}
    \includegraphics[width=\linewidth]{chapter5/generalisation/ellipse_2/tau_R_11_lowres}
    \caption{$\avg{u\p u\p}$}
\end{subfigure}
\begin{subfigure}{\linewidth}
    \includegraphics[width=\linewidth]{chapter5/generalisation/ellipse_2/tau_r_12_lowres}
    \caption{$\avg{u\p v\p}$}
\end{subfigure}
\begin{subfigure}{\linewidth}
    \includegraphics[width=\linewidth]{chapter5/generalisation/ellipse_2/SRx_lowres}
    \caption{$\mathcal{S}^R_x$}
\end{subfigure}
\begin{subfigure}{\linewidth}
    \includegraphics[width=\linewidth]{chapter5/generalisation/ellipse_2/tau_R_11_ML_lowres}
    \caption{$\avg{u\p u\p}^{\mathrm{ML}}$}
\end{subfigure}
\begin{subfigure}{\linewidth}
    \includegraphics[width=\linewidth]{chapter5/generalisation/ellipse_2/tau_r_12_ML_lowres}
    \caption{$\avg{u\p v\p}^{\mathrm{ML}}$}
\end{subfigure}
\begin{subfigure}{\linewidth}
    \includegraphics[width=\linewidth]{chapter5/generalisation/ellipse_2/SRx_ML_lowres}
    \caption{$\mathcal{S}_x^{R,\,\mathrm{ML}}$}
\end{subfigure}
\end{multicols}
\caption{Ellipse 2, $Re=10^4$ case: ML model predictions of components of the SSR tensor and the perfect closure compared to reference data.}
\label{fig:ML_generalisation_E2}
\end{figure*}

\newpage

\newpage

\subsection{A-posteriori analysis} \label{sec:ML_a-post}

The use of the ML model in the a-posteriori framework is assessed next, i.e. the trained closure is embedded directly into the simulation and no knowledge of the correct flow is assumed.
As before, the goal is to recover the unsteady spanwise-averaged solution of the original 3-D flow using only a 2-D system.
Starting from a spanwise-averaged initial condition, the ML model is incorporated into a 2-D solver providing closure to the SANS equations at every time step.

The most important aspect of the a-posteriori analysis revolves around the system stability.
Ideally, a stable closure would be provided at every time step even when noisy inputs are given to the ML model.
For the system to be stable, the spanwise-average attractor should dictate the temporal evolution of the flow and prevent the 2-D flow dynamics dominating as well as prevent the growth of non-physical effects.

For our test case, it has been found that directly incorporating the raw output of the SSR tensor ML model prediction $(\boldsymbol\tau_{ij}^{R,\,\mathrm{ML}})$ into the fluid solver makes it to rapidly diverge from the physical solution as the flow evolves in time.
The noise in the predicted fields is amplified by the divergence operator, which in turn significantly contaminates the resolved quantities at every time step in a feedback loop.
To mitigate this issue, the ML model has been trained to predict the perfect closure $(\mathcal{S}^{R})$ instead of the SSR tensor.
Hence, the new target outputs of the CNN are $\mathrm{Y}_n=\lbrace \mathcal{S}^R_x,\mathcal{S}^R_y\rbrace$.
Issues related to data-driven closure stability are discussed in detail in \cite{Cruz2019} under a RANS formulation.
It is suggested that learning the divergence of the residual tensor instead of its components helps reduce the error accumulation on the resolved quantities.

\begin{table}[t]
\centering
\caption{Correlation coefficients between the perfect closure and the ML model predictions.}
\begin{tabular}{lcc}
\toprule
$\mathcal{CC}$ & $\mathcal{S}^R_x$ & $\mathcal{S}^R_y$ \\
\midrule
ML & 0.89 & 0.90\\
\bottomrule \label{tab:ML_SR_a-priori}
\end{tabular}

{\footnotesize The correlation coefficients are calculated for the 500 snapshots of the test dataset and the average values are provided. \par}
\end{table}
\begin{table}[t]
\centering
\caption{Comparison of different metrics recorded during $\Delta t^*=10$ time units for the 3-D, SANS, and 2-D systems.}
\begin{tabular}{lrrr}
\toprule
 &$\left\langle 3\text{-}\mathrm{D} \right\rangle$ & SANS & $2\text{-}\mathrm{D}$\\
\midrule
$\overline{C}_D$ & 1.17 & 1.14 (-2.6\%)& 1.35 (+15.4\%)\\
$\overline{C}_L$ & 0.46 & 0.51 (+10.9\%) & 0.91 (+97.8\%)\\
$\overline{E}$ & 374.34 & 374.84 (+0.1\%) & 376.44 (+0.6\%) \\
$\overline{Z}$ & 0.0150 & 0.0151 (+0.6\%) & 0.0189 (26.0\%) \\
Cost [h] & 80.4 & 0.43 (-99.5\%) & 0.21 (-99.7\%) \\
\bottomrule
\label{tab:ML_a-posteriori}
\end{tabular}

{\footnotesize The overline indicates a temporal average except for the lift coefficient, for which the r.m.s. value is provided.
Energy $(E)$ and enstrophy $(Z)$ of the 3-D system are calculated using the streamwise and crossflow velocity components alone.
The relative error to the 3-D case is expressed in parenthesis.
The computational cost is expressed in hours and normalised by number of processes and clock speed.
The same constant time step is used in all systems.}
\end{table}

Before inserting the ML model prediction of the perfect closure into the 2-D system, an a-priori analysis of the trained model is performed (the training history is displayed in \fref{fig:SR_history}).
High correlation values are found for both perfect closure components, as presented in \tref{tab:ML_SR_a-priori}.
Qualitatively, the ML model prediction of both components is displayed in \fref{fig:ML_SR_a-priori}.
Again, mid- and large-scale structures are correctly captured while small-scale structures present a weaker correlation.

Learning the perfect closure improves the system stability and delays the transition to 2-D dynamics.
Starting from a spanwise-averaged snapshot of the 3-D flow, a-posteriori statistics are recorded as the flow evolves over $\Delta t^*=10$ time units, and these are summarised in \tref{tab:ML_a-posteriori}.
As shown in \fref{fig:E_Z_a-posteriori}, the global enstrophy deviates from the spanwise-averaged reference data obtained in the 3-D system as a result of the ML model error accumulation.
This is also reflected in the kinetic energy evolution, which increases with respect to the reference data.
On the other hand, both metrics still provide a notable improvement compared to the pure 2-D prediction.
The forces induced to the cylinder (\fref{fig:forces_a-posteriori}) also appear to be in better agreement with the reference data when including perfect closure ML predictions.
It becomes clear from \tref{tab:ML_a-posteriori} that the SANS equations yield results closer to a 3-D system while offering the same computational cost order of 2-D simulations.
The computational cost overhead between the SANS and 2-D systems is related to the data transfer between the fluid solver and the ML model, plus the ML model prediction time.

\begin{figure}[t]
\centering
\begin{subfigure}[t]{0.6\linewidth}
    \includegraphics[width=\linewidth]{chapter5/a-posteriori/SR/SRx_lowres}
    \caption{$\mathcal{S}^R_x$}
\end{subfigure}
\begin{subfigure}[t]{0.6\linewidth}
    \includegraphics[width=\linewidth]{chapter5/a-posteriori/SR/SRx_ML_lowres}
    \caption{$\mathcal{S}^{R,\,\mathrm{ML}}_x$}
\end{subfigure}
\begin{subfigure}[t]{0.6\linewidth}
    \includegraphics[width=\linewidth]{chapter5/a-posteriori/SR/SRy_lowres}
    \caption{$\mathcal{S}^R_y$}
\end{subfigure}
\begin{subfigure}[t]{0.6\linewidth}
    \includegraphics[width=\linewidth]{chapter5/a-posteriori/SR/SRy_ML_lowres}
    \caption{$\mathcal{S}^{R,\,\mathrm{ML}}_y$}
\end{subfigure}
\caption{ML model prediction of the perfect closure components.}
\label{fig:ML_SR_a-priori}
\end{figure}

\begin{figure}[t]
\centering
\includegraphics[width=0.50\linewidth]{chapter5/a-posteriori/E}
\includegraphics[width=0.48\linewidth]{chapter5/a-posteriori/Z}
\caption{Kinetic energy (left) and enstrophy (right) evolution during $\Delta t^*=10$ for the 3-D spanwise-averaged (reference data), SANS, and 2-D systems.
Simulations are started from a 3-D spanwise-averaged snapshot at $t^*_0$, corresponding to the scaling energy $(E_0)$ and enstrophy $(Z_0)$.
Energy and enstrophy are computed as detailed in \fref{fig:perfect_t-g} caption.}
\label{fig:E_Z_a-posteriori}
\end{figure}
\begin{figure}[t]
\centering
\includegraphics[width=0.48\linewidth]{chapter5/a-posteriori/forces}
\caption{Lift (bottom) and drag (top) forces induced to the cylinder (computed as described in \fref{fig:perfect_cc_forces}).
The lines legend is equivalent to \fref{fig:E_Z_a-posteriori}.}
\label{fig:forces_a-posteriori}
\end{figure}

Additionally, the instantaneous spanwise vorticity is depicted in \fref{fig:ML_a-posteriori}.
It is worth noting that the shear layer roll-up region of the SANS snapshot is not as coherent as the 2-D system, where structures have already reorganised into large scales.
In particular, the bottom detaching vortex of the SANS simulation is not as energised as the pure 2-D simulation one, and small-scale structures are still found in the SANS snapshot.
The close-wake region dictates the forces induced to the cylinder, explaining the significant difference in lift and drag forces shown in \tref{tab:ML_a-posteriori}.
In general, vortices found in the SANS simulation do not appear as coherent as in the 2-D case, where the inverse energy cascade promoted via vortex-merging and vortex-thinning events rapidly two-dimensionalises the wake structures.
A closer look on the SANS small-scale structures also shows that these are not correlated to the reference data.
This can also be appreciated at the top shear layer roll-up, where the error starts to propagate to the far field.
The ML closure error is responsible for such phenomenon, and ultimately yields the divergence of the system dynamics.

\begin{figure}[t]
\centering
\begin{subfigure}[t]{0.7\linewidth}
 	\includegraphics[width=\linewidth]{chapter5/a-posteriori/3-D_t=2651_lowres}
 	\caption{$\left\langle 3\text{-}\mathrm{D} \right\rangle$}
\end{subfigure}
\begin{subfigure}[t]{0.7\linewidth}
	\includegraphics[width=\linewidth]{chapter5/a-posteriori/SANS_t=2651_lowres}
	\caption{SANS}
\end{subfigure}
\begin{subfigure}[t]{0.7\linewidth}
	\includegraphics[width=\linewidth]{chapter5/a-posteriori/2-D_t=2651_lowres}
	\caption{$2\text{-}\mathrm{D}$}
\end{subfigure}
\caption{Spanwise vorticity snapshots for: (a) Spanwise-averaged 3-D flow (reference data).
(b) SANS with $\mathcal{S}^{R,\,\mathrm{ML}}$.
(c) 2-D simulation (no-model).
The SANS and the 2-D systems are started from a spanwise-averaged snapshot of the 3-D simulation at $t^*_0$.
The snapshots above correspond to $\Delta t^*=1$.}
\label{fig:ML_a-posteriori}
\end{figure}

\section{Discussion and conclusion}

This chapter introduces a ML model for the SANS equations, enabling 3-D turbulent flows with a homogeneous flow direction to be modelled using a 2-D system of equations with an observed speed up of 99.5\%.
The SSR tensor is critical to the success of this description, driving the 2-D flow dynamics to match the 3-D statistics and flow forces.
While EVMs are completely inadequate to model the SSR tensor, we develop a data-driven closure based on a deep-learning network with tested correlation statistics above 90\%.
The trained ML model also performs well in different Reynolds regimes and body geometries to the training case, despite some discrepancies in the near-body region.

The success of ML models for turbulent flows in an a-posteriori set-up is still limited, and the error propagation into the resolved quantities remains an open issue \citep{Wang2017, Beck2019}.
Stability has been previously achieved by projecting the ML model output into a known stable closure such as EVMs for time-averaged (RANS) or filtered (LES) systems \citep{Maulik2018, Beck2019, Cruz2019}.
Also, highly-resolved simulations help to mitigate the closure weight resulting in more stable computations \citep{Gamahara2017}.
Unfortunately, these solutions cannot be implemented in SANS because, as previously exposed, the EVMs hypothesis is intrinsically different from the nature of the SSR tensor.

Employing recurrent neural networks (RNN) for the prediction of SANS closure terms is a research direction worth exploring in the future.
RNNs incorporate the system temporal dynamics so they can offer a more stable a-posteriori solution targeting statistically stationary metrics \citep{Vlachas2018, Kim2020b}.
In this sense, the use of long-short term neural networks (a type of RNN) similarly to \cite{Vlachas2018}, together with CNNs could help overcome the issues exposed in the SANS closure.
Additionally, a physics-informed loss function can also improve the model stability as shown in \cite{Lee2019}.
Finally, while the current model is initialised from a 3-D averaged simulation, a more stable closure should be capable of developing 3-D turbulence dynamics even when initialised with a pure 2-D flow snapshot.

Overall, the new SANS flow description combined with advances in data-driven modelling has the potential to vastly improve the speed and accuracy of turbulence simulations for a prevalent class of engineering and environmental applications.
% ----------------------------------------------------------------
\end{document}


Face detection and recognition is not a new subject to computer science. Through time there were many attempts to provide satisfactory solutions and approximations. Techniques used for face detection spans from a regular well-known image recognition via edge detection, to more advanced, precise, though computationally expensive use of neural networks. This chapter will provide a basic overview of the mainstream approaches, listing the most current state-of-the-art methods and means to the problem.

Each section in this chapter represents a family of solutions. Providing a full description, explanation, and understanding of their respective problem, would span a book of its own, hence we aim to cover only a high level walk-though. We may also revisit certain aspects in more detail later in the text, as well.

\section{Convolutional neural networks}

There are many kinds of neural networks, one of which is called Convolutional Network\,\cite[p.~330]{deeplearningbook}, or CNN (Convolutional Neural Network). CNN specialises in processing data of known, grid-like structure. That includes for example time-series data, which can be represented as one dimensional grid of samples taken in regular intervals during a period of time. It also includes image data, represented as 2D grid of pixels. As the name suggests, CNNs employs a mathematical operation called \textbf{convolution}. In practical application, that stands for a substitution of a matrix multiplication by this linear mathematical operation - convolution. And this is done in at least one of the network's layers.

This section aims to explain what convolution is, what are the motivations behind its usage in neural networks. Later we'll describe a \textbf{pooling} operation, which is used in almost all CNNs as well.

\subsection{Convolution}
\label{ss:conv}

Convolution as a mathematical operation generally symbolise an operation on two functions of real-valued argument. Let's start with an example, paraphrased from Hinton's book\,\cite{deeplearningbook}, of such functions and demonstrate a motivation behind convolution:

Assume we have a vehicle and a laser parking sensor mounted on its front. This sensor is used to measures a distance to some object, let's say a docking station for the vehicle. And we want to park the vehicle at some precise proximity to the object. The sensor provides a single output $p(t)$, which reads as a position of the vehicle at certain time. Both $p$ and $t$ are real-valued, which means that the sensor can provide a different output value at any instance in time.

We also have to count in that our sensor is not always fully precise and reliable, so the measurements provided may be noisy. To obtain a more relevant data\,--\,a less noisy estimate of the position against the object, we can base our measurement on an average of several data samples. Since the vehicle is moving, we have to assume that more recent outputs should provide more relevant reading. So, we can weight our average and give the more recent data points more importance. Let's define this weight function as $w(a)$, where the argument $a$ symbolize age of a measurement.

Calculating the weighted average at every moment, we can get a smoother estimation of the vehicle's position as a new function:

\begin{equation}
    s(t) = \int p(a) w(t - a) da
\end{equation}

The function $s$ is formally known as \textbf{convolution}. An asterisk symbol is usually used to denote this operation:

\begin{equation}
    s(t) = (p * w)(t)
\end{equation}

To complete the definition, we have to note that $w$ has to be a valid probability density function, which in our example has to meet a criteria $a < 0: w(a) = 0$. That limits our function to weight only past samples, since we can's assume that our sensor can look into the future, that would be silly. That is a limitation to our use case only. In general, convolution is defined for all functions where the integral above is defined.

Convolutional neural networks use a bit different terminology, than we used in the example. The function providing data we want to consume ($p$ from the example) is simply called \textbf{input}. The other parameter to the convolution, in our example that was the weight function, is called a \textbf{kernel}. The result of convolution ($s$) of input ($p$) over a kernel ($w$) is simply called \textbf{output} or \textbf{feature map}.

Our expectation that the sensor from the example above, would provide a measurement continuously, in any instant of time, is not realistic. Time is usually discretized in digital world, therefore it is meaningful to assume the sensor is providing measurements at regular intervals. So instead of integrating over time continuum, we can define\,\cite{deeplearningbook} a discrete convolution as:

\begin{equation}
    s(t) = \sum_{a=-\infty}^{\infty} p(a) w(t - a) = (p * w)(t)
\end{equation}

Our example was quite simple and straightforward, usually in artificial intelligence and deep-learning the input is common to be multidimensional array (tensor) of data. Also the kernel happens to be a tensor of parameters which values are often obtained by some learning algorithm.

Finally, when processing multidimensional data, convolution is used over multiple axis at the same time. That's important for our application, because pictures, we are about to process, have more than one dimension. So in case we have an input image $I$\,--\,a two-dimensional bitmap, our convolution should use a two-dimensional kernel $K$ as well\,\cite{deeplearningbook}:

\begin{equation}
    S(x,y) = (I * K)(x, y) = \sum_i\sum_j I(m, n) K(x - i)(y - j)
\end{equation}

\begin{figure}[ht]
    \includegraphics[width=\textwidth]{obrazky-figures/convolution.pdf}
    \caption{Convolution operation with 2D input data and 2D kernel.}\label{fig:convolution}
\end{figure}

\subsection{Use of convolution in neural networks}

In machine learning, we leverage multiple important properties of the convolution operation:

\begin{itemize}
    \item Sparse interaction
    \item Sharing of parameters
    \item Equivariant representation
    \item Input of variable size
\end{itemize}

\subsubsection{Sparse interaction}

As you can notice in Figure\,\ref{fig:convolution}, kernel can be of different size than input. When the input data is for example an image of millions of pixels, the size of kernel allows convolution to select only specific subset of inputs for each output. In traditional neural networks, a matrix multiplication is used instead of convolution. That means every output unit interacts with every input unit. In convolutional networks, the kernel size limits that just to certain subset of input units. It is called \textbf{sparse interaction}\,\cite[p.~335]{deeplearningbook} or \textbf{sparse weights} and it is accomplished by restricting kernel to a smaller size than the input. That results in smaller memory footprint of the model, since it requires fewer parameters to be stored. while at the same time it improves statistical efficiency. It has also impact on performance, since computing the output requires fewer operations compared to matrix multiplication. A graphical demonstration can be seen on Figure\,\ref{fig:sparse_b}.

\begin{figure}[ht]
    \centering
    \includegraphics[width=.6\textwidth]{obrazky-figures/sparse_b.pdf}
    \caption{\textit{Sparse interaction viewed from below}: The highlighted units demonstrates propagation of one input unit from current layer $s_3$ to the next layer. On the top you can see all the output $n$ units, which are affected by this particular input. On the bottom picture, you can see how the same situation is represented in traditional matrix multiplication.}\label{fig:sparse_b}
\end{figure}

Deep convolutional networks allow indirect interaction between units which would be out of reach for given kernel size. This property is called \textbf{receptive field}\,\cite[p.~337]{deeplearningbook} of a unit and can be seen when we look at the network from perspective of the output layer (see figure\,\ref{fig:sparse_a}).

\begin{figure}[ht]
    \centering
    \includegraphics[width=.6\textwidth]{obrazky-figures/sparse_a.pdf}
    \caption{\textit{Sparse interaction viewed from above}: Highlighted portion of the image shows all units affecting the current layer ($s$ units). The amount of units is smaller on the top picture. That represents sparse interaction. On the bottom is the same situation, when the current layer is formed by matrix multiplication in a traditional network.}\label{fig:sparse_a}
\end{figure}

However, this view limits us just to direct influence on a unit. Receptive field lists also indirect influence, hence when multiple convolutional layers are used by the network the field grows. This can be seen on figure\,\ref{fig:receptive}. The effect can be enhanced when network contains additional features like \textit{strided convolution}\,\ref{ss:strided} or \textit{pooling}\,\ref{ss:pooling}. This means, despite the direct influence is very sparse, the final impact through indirect influence can make the units deeper in the network connected to most of the image on input.

\begin{figure}[ht]
    \centering
    \includegraphics[width=.6\textwidth]{obrazky-figures/receptive.pdf}
    \caption{Receptive field of unit $s_3$.}\label{fig:receptive}
\end{figure}

\subsubsection{Parameter sharing}

A feature of convolution referring to a reuse of parameters in more than one computation. In matrix multiplication, each weight element is calculated and used only once when computing the output. The weight is multiplied by one element of the input and then never used again. In contrast, convolution keeps it is kernel the same an uses its elements for every output calculation. This brings an advantage of learning just one set of weights for the whole input, rather than computing and remembering a set of weights for each output unit. Parameter sharing has no impact on forward propagation but it does further improve memory efficiency of a stored model.

\subsubsection{Equivariant representation}

A function is equivariant to another when $f(g(x)) = g(f(x))$. Convolution is naturally equivariant for example to translation\,\cite[p.~339]{deeplearningbook}. Imagine we have an input image and we shift the image some pixels to any direction of choice. When convolution is performed, the same kernel is applied to any set of pixels, therefore the set of features, our network layer aims to collect, can be found in the shifted image as well as in the original. It would just appear shifted in the output feature map. Here we can benefit from parameter sharing in use cases like edge detection. We're interested in the same feature, no matter where it appears on the image. In other use cases, like for example face detection, we might not be interested in the full parameter sharing. Imagine we have a kernel which is trained to detect a mouth. In order to work properly, we should restrict this kernel to look for the feature only in bottom portion of the picture, because detecting a mouth on forehead would not result in proper outputs. We'll cover more about multi-kernel convolution layers later.

\subsubsection{Variable size of the input}

Convolution can process data samples of different sizes. When the use case requires the network to be robust enough to properly process for example images, where each of the sample has different dimensions, this is a problem in matrix multiplication. The network can't apply the fixed size weight matrix on an input of different size. On the other hand, convolution is easy to perform, the situation is really similar to input of a fixed size, just the kernel is applied different amount of times.

\subsection{Pooling}
\label{ss:pooling}

In neural networks, convolution layer does not mean solely a convolution operation is applied\,\cite{handbook_face}. Typically such layer consists of multiple stages:

\begin{enumerate}
    \item \textbf{Convolution stage}: Multiple parallel convolutions are computed, which produces a set of linear activations.
    \item \textbf{Detector stage}: Each of the linear activations from previous stage is run through a nonlinear activation function.
    \item \textbf{Pooling stage}: Further modification of the layer.
\end{enumerate}

The last, pooling stage, provides better understanding of the convolution output. Instead of returning a set of features, it reduces redundancy of neighbouring outputs and provides a summary statistics. This helps the network to be invariant to small transition of the input. This means that if we translate the input of small amount of pixels, the output stays the same. This makes such network more robust. In a situations like a face detection, we don't need to know exact pixel coordinates of a feature, of an eye for example. We just need to define a region, where we are looking for such feature.

There are many pooling operations, let's list some of the most popular ones like \textbf{max pooling}\,\cite{maxpooling} (reports the maximum output in rectangular neighbourhood), average of rectangular neighbourhood, $L^2$ norm of the rectangular neighbourhood, and weighted average of distance from the central pixel.

Invariance to transition is produced by pooling over spatial regions, however pooling layer can learn an invariance to a transformation of other kinds as well. That happens if we pool over outputs of other separately parametrized convolutions. As an example you can see an invariance to slant in cursive on figure\,\ref{fig:pooling}. This principle is accented mainly in max-out networks\,\cite{maxout}.

\begin{figure}[ht]
    \centering
    \includegraphics[width=.8\textwidth]{obrazky-figures/pooling.pdf}
    \caption{\textit{Pooling response with learned invariance to slant}: Here we have 3 filters, where all of them had a task to learn a handwritten number 3. Each of them resulted in learning of different slant of the number. When a number 3 is given as the input, one of the filters will match it and cause a high activation in corresponding detector unit. Due to use of pooling, the max pooling unit has a large activation as well, no matter which filter matched the number.}\label{fig:pooling}
\end{figure}

Since pooling can summarise a response of layer over whole neighbourhood of input units, it is not necessary to have the same amount of pooling units as the detectors. We can leverage pooling to provide down-sampling as can be seen in Figure\,\ref{fig:downsample}. That further improves performance of the network since it lowers the amount of inputs for the next layer.

\textbf{Pooling with down-sampling} is an essential step when dealing with input of variable size. Let's say we want to use the convolution to detect a face on images with different resolutions. We have learned our detectors to register mouth in the bottom half of the image and another two sets of detectors to locate eyes, each in one of the top quadrants. We can use convolution layer with down-sampling pooling to provide the required classification. We expect to be provided by 3 activations on the output, and each of the detector has assigned its portion of the input. It does not matter, if the portion contains one amount of pixels or much more.

\begin{figure}[ht]
    \centering
    \includegraphics[width=.6\textwidth]{obrazky-figures/downsampling.pdf}
    \caption{Pooling with down-sampling.}\label{fig:downsample}
\end{figure}

In deep learning this is often the case. We don't refer to convolution as a simple single operation as described in the beginning of this chapter. Such convolution layer with a single kernel would be capable of extracting only a single feature, although in many spatial locations. Usually many convolutions are applied and performed in parallel. That can provide many different kernels for different features, which are interesting for the use case. As a result, the network can locate many kinds of features at many locations.

\subsection{Strided convolution}
\label{ss:strided}

Also when we have stumbled upon the down-sampling in pooling, this step can be sometimes omitted and simplified even more. Although the result is similar to the pooling with down-sampling, the logic in based on different assumptions. In pooling operation, we leverage all the information retrieved by the convolutions and simplify the output.

A strided convolution is rather avoiding some of the convolutions at all. That further lowers the computational costs, hence at a risk of not extracting all the features in such detail. In this case we sample pixels in every direction with a step of $s$. The step $s$ is called a \textbf{stride}. It is also possible to define a separate stride for each step direction.

\begin{figure}
    \centering
    \includegraphics[width=.6\textwidth]{obrazky-figures/conv_down.pdf}
    \caption{Convolution and pooling with down-sampling.}\label{fig:conv_down}
\end{figure}

\begin{figure}
    \centering
    \includegraphics[width=.6\textwidth]{obrazky-figures/strided.pdf}
    \caption{Strided convolution.}\label{fig:strided}
\end{figure}

As it can be clearly seen on the images\,\ref{fig:conv_down} and\,\ref{fig:strided} the two step down-sampling is computationally more expensive than the strided optimisation, while it can provide similar results.

\subsection{Zero padding}

Another feature which is essential to CNNs implementation is padding with zeros. This maintains the network's ability to preserve the width of its input if needed. The input tensor is padded with zeros on the ends, so a convolution operation does not shrink the size of input vector by a fraction of kernel. Without padding we are forced to either keep kernels small, or let the networks shrink in spatial extent. Both are extreme limitations of network's power. Padding allows to control independently both: the size of kernel and resolution of the output.

\subsection{Other convolution layer types}

Sometimes our desire is to rather use locally connected layers. This resembles a discrete convolution with small kernel, but without parameter sharing. Therefore this layer type is sometimes called \textbf{unshared convolution}\,\cite{locally_connected}. Unshared type of convolutional layer is useful when we aim to detect features, which are local and there's no assumption that the same feature should occur across all the input.

Another type available is a \textbf{tiled convolution}\,\cite{tiled_conv} layer. A layer type meant to offer a compromise between locally connected layers and convolutional layers. Instead of learning a set of weights for every spatial location, this layer type provides tiling. That means a single set of weights is learned and it is later applied in rotation, providing different set of weighs for neighbouring locations. This makes the outcome similar to locally connected layer, while keeping the benefit of lower cost of convolutional layer, since requirements to store parameters would grow by factor of kernel set size, instead of size of a whole feature map. A comparison overview of different convolution layer types can be seen on figure\,\ref{fig:compare_layers}.

\begin{figure}
    \centering
    \includegraphics[width=.6\textwidth]{obrazky-figures/locally_connected.pdf} \\
    Locally connected \\
    \bigskip
    \includegraphics[width=.6\textwidth]{obrazky-figures/convolution_tiled.pdf} \\
    Tiled convolution \\
    \bigskip
    \includegraphics[width=.6\textwidth]{obrazky-figures/convolution_std.pdf} \\
    Standard convolution \\
    \bigskip
    \includegraphics[width=.6\textwidth]{obrazky-figures/fully_connected.pdf} \\
    Fully connected \\
    \bigskip
    \caption{Comparison of convolutional layers.}
    \label{fig:compare_layers}
\end{figure}

\section{Capsule neural networks}

Convolutional neural networks are a state-of-the-art of current deep learning. They are hugely popular and they can provide great results and solve problems, which were unimaginable before. Despite all their power they embody fundamental drawbacks and limitations\,\cite{capsule_compare}. This and availability of grater computational power lead to creation of Capsule neural networks (CapsNET)\,\cite{capsule}.

\subsection{Rationale}

CNNs operate with features and their recognition. Deeper convolution layer detect simple features, for example edges or colour gradients. Layers higher in the network are designed to detect specific combinations of such features and creates more complex ones. And finally, on the very top of the network, a dense layer takes the very high level features and provides a prediction of classification. And since many times the convolution is made invariant to different transformations, this can lead to impossible results. Mere presence of an object provides indication of feature presence. Relation between detected features is not considered at all. When simple features are composed to a more complex ones, translational or rotational relationship does not play any role.

We have already tackled the way CNN is using to deal with this problem. Pooling and more convolutional layers of smaller kernels are applied to reduce spatial size of information lost during each convolution. This aims to increase the field of view for convolutional layers higher in the network, therefore allowing them to locate features in larger portion of the input image. Pooling made CNNs surprisingly effective and one of the top performing architectures, though still enhancing loss of information.

Prof. Geoffrey Hinton, one of fathers of deep learning and praised founder of many principles and author of algorithms, is also the author of capsule neural networks. Hinton wrote\,\footnote{\url{https://www.reddit.com/r/MachineLearning/comments/2lmo0l/ama_geoffrey_hinton/clyj4jv/}}:
\begin{quotation}
    The pooling operation used in convolutional neural networks is a big mistake and the fact that it works so well is a disaster.
\end{quotation}

To demonstrate this drawback imagine a CNN face detector. Deeper layers of network detects parts of facial features. The higher layers combine these parts into complex features like an \textit{eye}, \textit{nose} or \textit{mouth}. Our multi-kernel convolution would allow us to define regions, where we expect such features, though we can't define the relation between them. The network can simply recognise an image as a valid face, despite for example the \textit{eye} is rotated to the opposite direction than a \textit{mouth}. An extreme demonstration of such problem can be seen on figure\,\ref{fig:capsule_face}.

\begin{figure}[ht]
    \centering
    \includegraphics[height=0.3\textwidth]{obrazky-figures/face_normal.pdf}
    \hspace{10em}
    \includegraphics[height=0.3\textwidth]{obrazky-figures/face_disorted.pdf}
    \caption{Both of these images can be evaluated to be a face. Spatial location, relation and pose between simpler features are not considered by this type of network. Both images appear similar to CNN. \textit{Original artwork provided by Freepik\,\cite{freepik}}.}
    \label{fig:capsule_face}
\end{figure}

\subsection{Inverse graphics}

Hinton, inspired by computer graphics, tried to explain and reconstruct human brain's visual cognitive functions. The brain itself in fact does the opposite process to \textit{rendering} we know from computer graphics. Hinton calls it \textbf{inverse graphics}. A visual information is decomposed into a hierarchical representation of objects, which are matched against known, learned patterns. This relationship matrix is stored in our brains. One key factor is that object representation is not dependent on view angle.

So how do we model this hierarchical relationship inside a neural network? Here we can learn from solutions already discovered in another field\,--\,in computer graphics. 3D modelling uses something called a \textbf{pose}. This represents relation between 3D objects and provides a rotation and translation transformation matrix. In neural networks we represent that as a 4D \textit{pose matrix}\,\cite{capsule}. This results in combination of information about object relations with internal representation of object data. Hence it becomes easy for a model to recognise that it just sees a different view of something it already saw before.

\begin{figure}[ht]
    \centering
    \includegraphics[width=.8\textwidth]{obrazky-figures/duck.pdf}
    \caption{CNN struggles to recognise this is the same object. Human brain, on the other hand, immediately understands that objects on the picture are identical, despite being viewed from multiple different angles. \textit{Original artwork provided by Freepik\,\cite{freepik}}.}
    \label{fig:duck}
\end{figure}

Let's discuss a situation on the Figure\,\ref{fig:duck}. A human looking at this picture can easily recognise that it is a rubber duck viewed from different angles. Our internal representation of a rubber duck is independent on the viewing angle. It may be the first time that these particular pictures are shown to you, although you simply know their meaning. Since there's no internal representation of 3D space in CNNs, it really struggles. On the other hand, for CapsNET, this problem is easy, since the 3D relations are explicitly modelled. Experiments shown that usage of capsule neural networks can further reduce the error rate\,\cite{capsule_compare}.

And this is not the only benefit of usage of capsules. It also significantly lowers the amount of data required to train such network to achieve comparable performance to a CNN. And this make sense, since capsule theory is much closer to the way how brain does work. If a brain tries to learn to distinguish a horse from a cow, it only needs to be presented with few images, at most couple of dozens. CNN would require to be presented by many thousands of image samples, to achieve comparable performance. In this particular aspect we can compare CNNs to a brute-force approach to deep learning\,\cite{capsnet_comparative_perf}.

\subsection{Understanding capsules}

Let's step back and introduce capsules in the CapsNet networks. What does a term \textit{capsule} actually mean? What is the mathematical representation and what key principles are used? How we can represent it as a neural network architecture? How does a network using capsules look like from the engineering aspect? And how should such network look like in our particular use case?

There are many ways to implement capsules and in this example we will follow Hinton\,\cite{capsule} and use a simple, 3 layer shallow capsule network with dynamic routing. As shown later, such shallow network and straightforward implementation works well when used together with dynamic routing and can achieve results comparable to much deeper CNN networks.

Desired representation of a capsule layer output in CapsNet network represents a probabilistic likelihood of the entity mapped by one particular capsule being present in the current input. To facilitate this we use a non-linear activation called \textbf{squash} and is shown as Equation\,\ref{eq:squash}. It ensures that the length of a vector is converted into a probability score of feature presence in current input kernel for each capsule. The aim is to resize vector based on their size. Shorter vectors get shrunk to minimal lengths while long vector are shrunk to size slightly below 1. This non-linearity can be later leveraged in discriminative learning process.

\begin{equation}
    v_j = \frac{\norm{s_j}^2}{1 + \norm{s_j}^2} \frac{s_j}{\norm{s_j}}
    \label{eq:squash}
\end{equation}

In this equation a squash activation is computed for a capsule $j$ where $v_j$ would represent squashed output and $s_j$ is the total input.

In any later layer, except the first layer of capsules, the capsule can understand the $s_j$ input as a weighted sum mapping over all prediction vectors $\hat{u_j|i}$ decided by the capsule layer below. This prediction vectors are a product of $u_i$ output of previous capsule layer and its $W_{ij}$ weights matrix:

\begin{equation}
    s_j = \sum_{i}c_{ij}\hat{u}_{j|i},\quad \hat{u}_{j|i} = W_{ij}u_i
    \label{eq:capsule}
\end{equation}

Here we are introducing $c_{ij}$ coupling coefficients which are determined by the dynamic routing process in multiple iterations. The $c_{ij}$ coefficients are computed for each capsule $i$ in relation to every capsule $j$ in the layer above. For each capsule $i$ the sum of these coefficients is equal to 1 and represents a special soft-max for routing. This activation uses initial logits $b_{ij}$ that are log prior probabilities\,\cite{capsule_compare} of the likelihood a capsule $i$ is coupled to capsule $j$. In general, routing provides each capsule $j$ a mean to determine which capsules $i$ are interesting enough and aligned in the same fashion for it to base its prediction upon. This coupling is determined independently on current input.

\begin{equation}
    c_{ij} = \frac{exp(b_{ij})}{\sum_{k}exp(b_{ik})}
    \label{eq:softmax}
\end{equation}

Later on we will define two distinct layer types: \textit{prediction capsules} and \textit{primary capsules}. These are required to communicate and rate the activations based on accuracy of the prediction. The algorithm used is called \textbf{dynamic routing by agreement} and is described in greater detail by Hinton in publication \textit{Dynamic routing between capsules}\,\cite{capsule}. It utilizes the already described coupling coefficients and log prior. The algorithm goes as this:

\begin{algorithm}[H]
    \caption{Dynamic routing by agreement.}\label{alg:routing}
    \begin{algorithmic}[1]
        \Procedure{ROUTING}{$\hat{u}_{j|i}, r, l$}
        \ForAll{capsule $i$ in layer $j$ and capsule $j$ in layer $(l + 1)$} $b_{ij}\gets 0$ \EndFor
        \For{$r$ iterations}
            \ForAll{capsule $i$ in layer $l$} $c_{i}\gets softmax(b_i)$ \EndFor
            \ForAll{capsule $j$ in layer $(l + 1)$} $s_{j}\gets \sum_ic_{ij}\hat{u}_{j|i}$ \EndFor
            \ForAll{capsule $j$ in layer $(l + 1)$} $v_{j}\gets squash(s_j)$ \EndFor
            \ForAll{capsule $i$ in layer $l$ and capsule $j$ in layer $(l + 1)$} $b_{ij}\gets b_{ij} + \hat{u}_{j|i} v_j$ \EndFor
        \EndFor
        \State \textbf{return} $v_j$
    \EndProcedure
    \end{algorithmic}
\end{algorithm}

The aim is simple, yet harder to achieve. All and every capsule in \textit{primary capsule} and \textit{prediction capsule} layers are made to agree on the location of the features in space and their orientation. At first all the routing logits $b_{ij}$ are initialized to zero. That means each input capsule's output is sent to capsules in the next layer with equal probability $c_{ij}$. In time, the logits are learned and since they are independent on the current image\footnote{Each image is expected to include one and just one face, not more.}, they can be trained at the same time as all the other weights. Their only dependency is the type and location of the capsules involved. Its training process involves iteratively adjusting and refining the logits based on measurement of agreement between the prediction $\hat{u}_{j|i}$ made by capsule $i$ in the underlying layer and current output $v_j$ of each capsule $j$ in the layer above.

\begin{equation}
    a_{ij} = v_j \cdot \hat{u}_{j|i}
    \label{eq:agreement}
\end{equation}

The measuring of agreement is simplified to a scalar product as per equation\,\ref{eq:agreement}. This rate is used to as an addend to the log prior $b_{ij}$ and used to compute new value of the coupling coefficient $c_{ij}$.

\subsection{Architecture using capsules}

Before we move on and utilize this principles, we need to step back and understand the capsule networks as whole. Generally speaking a Capsule network (CapsNET) consists of two parts where each has a distinct use: \textbf{Encoder} and \textbf{Decoder}:

A decoder network is a fully connected (dense) or convolutional network, which purpose is simply to reconstruct an image based on prediction. This is used to provide proper feature adaptive learning experience in order to maximize classification potential of the encoder, in other words its a regularization method for capsule networks to reach valid conclusions. Based on recognized features and selected activation in the encoder, decoder attempts to recreate the input object. In simple use cases like MNIST classification a 3 layer dense network can be used. This was demonstrated by Hinton and can be seen on figure\,\ref{fig:decoder}. In more feature rich environment with bigger resolution required on input a fully connected layer doesn't provide enough interpolation and resizing capabilities so a convolutional network with resize (enlarge) has to be used. Such network can be of up to 10 layers total. In later chapter we will provide examples of such decoder networks.

\begin{figure}[ht]
    \centering
    \includegraphics[height=11em]{obrazky-figures/reconsArch.png}
    \caption{Decoder architecture as per Hinton\,\cite{capsule}.}
    \label{fig:decoder}
\end{figure}

Encoder is simply the gro of capsule network. This is the truly capsulized network. There are many configurations available, though with growing complexity\,--\,meaning adding capsule layers, the computational complexity grows exponentially. This is due to routing and amount of connections required between each capsule. In this thesis, we will be using the Hinton's hinted layering with a slight twist to facilitate more detailed and granular feature recognition across much wider space.

As said the 3 layer encoder consist of two convolutional layers (one traditional, one capsule-based) and one capsule-based fully connected layer. The first layer
\textit{encoder\_conv1} is a traditional 2D convolution. Each pixel intensity of the input image gets converted in this layer to the activation in a detector of local features. Detected local features are used as input for next layer, primary capsules.

\begin{figure}[ht]
    \centering
    \includegraphics[width=\textwidth]{obrazky-figures/capsulearch.png}
    \caption{Encoder architecture as per Hinton\,\cite{capsule}.}
    \label{fig:encoder}
\end{figure}

Naturally each of these parts of the network can be assigned their own loss function. A different balance\,--\,loss weight enforced over each of these functions can greatly improve the overall loss of the network. For \textit{decoder} a loss function would be straightforward to guess. Since we're dealing with images, 2D data comparison, for example a simple \textit{mean squared error} is a suitable metric.

\begin{equation}
    MSE = \frac{1}{n} \sum_{i=0}^{n} (Y_{i} - \hat{Y}_{i})^2
\end{equation}

On the other hand, the encoder part required more sophisticated mean to compute its inconsistency between the predicted value and its ground truth. Therefore a \textit{margin loss} is used for each label in which we're classifying the images. For each of them and individual loss is calculated and the total loss is said to be a simple sum of these partial losses:

\begin{equation}
    L_k = T_k max(0, m^+ - \norm{v_k})^2 + \lambda (1 - T_k) max(0, \norm{v_k} - m^-)^2
    \label{eq:margin_loss}
\end{equation}

where, $T_k = 1$ is a hot one encoded truth of the capsule belonging to that particular label. Then $m^+ = 0.9$ and $m^- = 0.1$ are the loss boundaries and we use the down-weighting factor of $\lambda = 0.5$ to stop shrinking in lengths of activation vectors when initial learning happens\,\cite{capsule}.

\subsection{Primary capsules}

This composite layer encapsulates a convolution with vectorization and \textit{squashing}. Therefore this layer is also called \textit{convolutional capsule layer}. It provides knowledge about greater context of each local feature, and makes more abstract detection about overall shape and orientation of such feature\,\cite{capsule_compare}. Key factor in this layer is a non-linear activation called \textit{squashing}. We've already covered the squashing function by equation\,\ref{eq:squash}. Hinton, describes this layer\,\cite{capsule} as:

\begin{quotation}
    In convolutional capsule layers each unit in a capsule is a convolutional unit. Therefore, each capsule will output a grid of vectors rather than a single output vector.
\end{quotation}

The quoted statement stipulates boundaries by which this layer is different to a standard convolution. Since we're not interested in a scalar feature identity, and require more complex result, a whole vector of attributes that holds much broader context of the feature location. This brings couple of limitations of the convolution itself though. For example a strided convolution or pooling would mean disaster for the spatial collocation of a feature. Therefore we eliminate our use of these advanced convolutional optimizations and we are obliged to use a convolution in its most simple and pure form.

\subsection{Prediction capsules}

Next layer is understood to be a fully connected capsule layer, which purpose is to rate the features located by the previous \textit{primary capsule} layer and bundle the smallest amount of unique activation per label. As it may sound confusing, we need to elaborate further and step by step. We already know, that the output of previous layer in each of its capsules is a local grid of vectors which are unique to each its member as well as for each capsule. Key word is "grid". It signifies the output is three dimensional. However traditionally a feature is understood as a 2D location. Our primary capsules holds also another interesting factor to this. We can now understand and recognize a feature in relation to others. And that's precisely what is done in this layer. The amount of prediction capsules corresponds to the amount of target labels. Therefore each capsule is mapped to one particular label, in our case an identity, and is expected to locate the features which are unique to this particular identity as well as all the common features which define a face. However this is not an easy task, and a simple weighted multiplication over a kernel would not provide enough insight into the complicated data. Therefore an algorithm called \textit{dynamic routing}\,\cite{capsule_compare} is used. This routing algorithm has been already discussed and can be seen as Algorithm\,\ref{alg:routing}.

The main purpose is to recognise that particular features which holds the same or similar context as the others in each particular label. The context can be of many different types, from simple distance of multiple features in 3D space, over a consistent angle hold between two features, to for instance a simple contextualized location relative to other features.