% ------------------------------------------------------------------
\chapter{Computational blocks}\label{s:blocks}
% ------------------------------------------------------------------

This chapters describes the individual computational blocks supported by \matconvnet. The interface of a CNN computational block !<block>! is designed after the discussion in \autoref{s:fundamentals}. The block is implemented as a MATLAB function !y = vl_nn<block>(x,w)! that takes as input MATLAB arrays !x! and !w! representing the input data and parameters and returns an array !y! as output. In general, !x! and !y! are 4D real arrays packing $N$ maps or images, as discussed above, whereas !w! may have an arbitrary shape.

The function implementing each block is capable of working in the backward direction as well, in order to compute derivatives. This is done by passing a third optional argument !dzdy! representing the derivative of the output of the network with respect to $\by$; in this case, the function returns the derivatives ![dzdx,dzdw] = vl_nn<block>(x,w,dzdy)! with respect to the input data and parameters. The arrays !dzdx!, !dzdy! and !dzdw! have the same dimensions of !x!, !y! and !w! respectively (see \autoref{s:back}).

Different functions may use a slightly different syntax, as needed: many functions can take additional optional arguments, specified as a property-value pairs; some do not have parameters  !w! (e.g. a rectified linear unit); others can take multiple inputs and parameters, in which case there may be more than one !x!, !w!, !dzdx!, !dzdy! or !dzdw!. See the rest of the chapter and MATLAB inline help for details on the syntax.\footnote{Other parts of the library will wrap these functions into objects with a perfectly uniform interface; however, the low-level functions aim at providing a straightforward and obvious interface even if this means differing slightly from block to block.}

The rest of the chapter describes the blocks implemented in \matconvnet, with a particular focus on their analytical definition. Refer instead to MATLAB inline help for further details on the syntax.

% ------------------------------------------------------------------
\section{Convolution}\label{s:convolution}
% ------------------------------------------------------------------

\begin{figure}[t]
	\centering
	\includegraphics[width=0.7\textwidth]{figures/svg/conv}
	\caption{\textbf{Convolution.} The figure illustrates the process of filtering a 1D signal $\bx$ by a filter $f$ to obtain a signal $\by$. The filter has $H'=4$ elements and is applied with a stride of $S_h =2$ samples. The purple areas represented padding $P_-=2$ and $P_+=3$ which is zero-filled. Filters are applied in a sliding-window manner across the input signal. The samples of $\bx$ involved in the calculation of a sample of $\by$ are shown with arrow. Note that the rightmost sample of $\bx$  is never processed by any filter application due to the sampling step. While in this case the sample is in the padded region, this can happen also without padding.}\label{f:conv}
\end{figure}

The convolutional block is implemented by the function !vl_nnconv!. !y=vl_nnconv(x,f,b)! computes the convolution of the input map $\bx$ with a bank of $K$ multi-dimensional filters $\bff$ and biases $b$. Here
\[
 \bx\in\real^{H \times W \times D}, \quad
 \bff\in\real^{H' \times W' \times D \times D''}, \quad
 \by\in\real^{H'' \times W'' \times D''}.
\]
The process of convolving a signal is illustrated in \autoref{f:conv} for a 1D slice. Formally, the output is given by
\[
y_{i''j''d''}
=
b_{d''}
+
\sum_{i'=1}^{H'}
\sum_{j'=1}^{W'}
\sum_{d'=1}^D
f_{i'j'd} \times x_{i''+i'-1,j''+j'-1,d',d''}.
\]
The call !vl_nnconv(x,f,[])! does not use the biases. Note that the function works with arbitrarily sized inputs and filters (as opposed to, for example, square images). See \autoref{s:impl-convolution} for technical details.

\paragraph{Padding and stride.} !vl_nnconv! allows to specify  top-bottom-left-right paddings $(P_h^-,P_h^+,P_w^-,P_w^+)$ of the input array and subsampling strides $(S_h,S_w)$ of the output array:
\[
y_{i''j''d''}
=
b_{d''}
+
\sum_{i'=1}^{H'}
\sum_{j'=1}^{W'}
\sum_{d'=1}^D
f_{i'j'd} \times x_{S_h (i''-1)+i'-P_h^-, S_w(j''-1)+j' - P_w^-,d',d''}.
\]
In this expression, the array $\bx$ is implicitly extended with zeros as needed.

\paragraph{Output size.} !vl_nnconv! computes only the ``valid'' part of the convolution; i.e. it requires each application of a filter to be fully contained in the input support.  The size of the output is computed in \autoref{s:receptive-simple-filters} and is given by:
\[
  H'' = 1 + \left\lfloor \frac{H - H' + P_h^- + P_h^+}{S_h} \right\rfloor.
\]
Note that the padded input must be at least as large as the filters: $H +P_h^- + P_h^+ \geq H'$, otherwise an error is thrown.

\paragraph{Receptive field size and geometric transformations.} Very often it is useful to relate geometrically the indexes of the various array to the input data (usually images) in term of coordinate transformations and size of the receptive field (i.e. of the image region that affects an output). This is derived in \autoref{s:receptive-simple-filters}.

\paragraph{Fully connected layers.} In other libraries, a \emph{fully connected blocks or layers} are linear functions where each output dimension depends on all the input dimensions. \matconvnet does not distinguishes between fully connected layers and convolutional blocks. Instead, the former is a special case of the latter obtained when the output map $\by$ has dimensions $W''=H''=1$. Internally, !vl_nnconv! handle this case more efficiently when possible.

\paragraph{Filter groups.} For additional flexibility, !vl_nnconv! allows to group channels of the input array $\bx$ and apply to each group different subsets of filters. To use this feature, specify as input a bank  of $D''$ filters $\bff\in\real^{H'\times W'\times D'\times D''}$ such that $D'$ divides the number of input dimensions $D$. These are treated as $g=D/D'$ filter groups; the first group is applied to dimensions $d=1,\dots,D'$ of the input $\bx$; the second group to dimensions $d=D'+1,\dots,2D'$ and so on. Note that the output is still an array $\by\in\real^{H''\times W''\times D''}$.

An application of grouping is implementing the Krizhevsky and Hinton network~\cite{krizhevsky12imagenet} which uses two such streams. Another application is sum pooling; in the latter case, one can specify $D$ groups of $D'=1$ dimensional filters identical filters of value 1 (however, this is considerably slower than calling the dedicated pooling function as given in \autoref{s:pooling}).

% ------------------------------------------------------------------
\section{Convolution transpose (deconvolution)}\label{s:convt}
% ------------------------------------------------------------------

\begin{figure}[t]
	\centering
	\includegraphics[width=0.7\textwidth]{figures/svg/convt}
	\caption{\textbf{Convolution transpose.} The figure illustrates the process of filtering a 1D signal $x$ by a filter $f$ to obtain a signal $y$. The filter is applied in a sliding-window, in a pattern that is the transpose of \autoref{f:conv}. The filter has $H'=4$ samples in total, although each filter application uses two of them (blue squares) in a circulant manner. The purple areas represented crops $C_-=2$ and $C_+=3$ which are discarded.  The samples of $x$ involved in the calculation of a sample of $y$ are shown with arrow. Note that, differently from \autoref{f:conv}, there is not any samples to the right of $\by$ which is not involved in a convolution operation. This is because the width $H''$ of the output $\by$, which given $H'$ can be determined up to $U_h$ samples, is selected to be the smallest possible.}\label{f:convt}
\end{figure}

The \emph{convolution transpose} block (sometimes referred to as ``deconvolution'') is the transpose of the convolution block described in \autoref{s:convolution}. In \matconvnet, convolution transpose is  implemented by the function !vl_nnconvt!.

In order to understand convolution transpose, let:
\[
 \bx\in\real^{H \times W \times D}, \quad
 \bff\in\real^{H' \times W' \times D \times D''}, \quad
 \by\in\real^{H'' \times W'' \times D''}, \quad
\]
be the input tensor, filters, and output tensors. Imagine operating in the reverse direction by using the filter bank $\bff$ to convolve the output $\by$ to obtain the input $\bx$, using the definitions given in~\autoref{s:convolution} for the convolution operator; since convolution is linear, it can be expressed as a matrix $M$ such that  $\vv \bx = M \vv\by$; convolution transpose computes instead $\vv \by = M^\top \vv \bx$. This process is illustrated for a 1D slice in \autoref{f:convt}.

There are two important applications of convolution transpose The first one are the so called \emph{deconvolutional networks}~\cite{zeiler14visualizing} and other networks such as convolutional decoders that use the transpose of a convolution. The second one is implementing data interpolation. In fact, as the convolution block supports input padding and output downsampling, the convolution transpose block supports input upsampling and output cropping.

Convolution transpose can be expressed in closed form in the following rather unwieldy expression (derived in \autoref{s:impl-convolution-transpose}):
\begin{multline}\label{e:convt}
y_{i''j''d''} =
 \sum_{d'=1}^{D}
 \sum_{i'=0}^{q(H',S_h)}
 \sum_{j'=0}^{q(W',S_w)}
f_{
1+ S_hi' + m(i''+ P_h^-, S_h),\ %
1+ S_wj' + m(j''+ P_w^-, S_w),\ %
d'',
d'
}
\times \\
x_{
1 - i' + q(i''+P_h^-,S_h),\ %
1 - j' + q(j''+P_w^-,S_w),\ %
d'
}
\end{multline}
where
\[
m(k,S) = (k - 1) \bmod S,
\qquad
q(k,n) = \left\lfloor \frac{k-1}{S} \right\rfloor,
\]
$(S_h,S_w)$ are the vertical and horizontal \emph{input upsampling factors},  $(P_h^-,P_h^+,P_h^-,P_h^+)$ the \emph{output crops}, and $\bx$ and $\bff$ are zero-padded as needed in the calculation. Note also that filter $k$ is stored as a slice $\bff_{:,:,k,:}$ of the 4D tensor $\bff$.

The height of the output array $\by$ is given by
\[
  H'' = S_h (H - 1) + H' -P^-_h - P^+_h.
\]
A similar formula holds true for the width. These formulas are derived in \autoref{s:receptive-convolution-transpose} along with expression for the receptive field of the operator.

We now illustrate the action of convolution transpose in an example (see also \autoref{f:convt}).  Consider a 1D slice in the vertical direction, assume that the crop parameters are zero, and that $S_h>1$. Consider the output sample $y_{i''}$ where the index $i''$ is chose such that $S_h$ divides $i''-1$; according to~\eqref{e:convt}, this sample is obtained as a weighted summation of $x_{i'' / S_h},x_{i''/S_h-1},...$ (note that the order is reversed). The weights are the filter elements $f_1$, $f_{S_h}$,$f_{2S_h},\dots$ subsampled with a step of $S_h$. Now consider computing the element $y_{i''+1}$; due to the rounding in the quotient operation $q(i'',S_h)$, this output sampled is obtained as a weighted combination of the same elements of the input $x$ that were used to compute $y_{i''}$; however, the filter weights are now shifted by one place to the right: $f_2$, $f_{S_h+1}$,$f_{2S_h+1}$, $\dots$. The same is true for $i''+2, i'' + 3,\dots$ until we hit $i'' + S_h$. Here the cycle restarts after shifting $\bx$ to the right by one place. Effectively, convolution transpose works as an \emph{interpolating filter}.

% ------------------------------------------------------------------
\section{Spatial pooling}\label{s:pooling}
% ------------------------------------------------------------------

!vl_nnpool! implements max and sum pooling. The \emph{max pooling} operator computes the maximum response of each feature channel in a $H' \times W'$ patch
\[
y_{i''j''d} = \max_{1\leq i' \leq H', 1 \leq j' \leq W'} x_{i''+i-1',j''+j'-1,d}.
\]
resulting in an output of size $\by\in\real^{H''\times W'' \times D}$, similar to the convolution operator of \autoref{s:convolution}. Sum-pooling computes the average of the values instead:
\[
y_{i''j''d} = \frac{1}{W'H'}
\sum_{1\leq i' \leq H', 1 \leq j' \leq W'} x_{i''+i'-1,j''+j'-1,d}.
\]
Detailed calculation of the derivatives are provided in \autoref{s:impl-pooling}.

\paragraph{Padding and stride.} Similar to the convolution operator of \autoref{s:convolution}, !vl_nnpool! supports padding the input; however, the effect is different from padding in the convolutional block as pooling regions straddling the image boundaries are cropped. For max pooling, this is equivalent to extending the input data with $-\infty$; for sum pooling, this is similar to padding with zeros, but the normalization factor at the boundaries is smaller to account for the smaller integration area.

% ------------------------------------------------------------------
\section{Activation functions}\label{s:activation}
% ------------------------------------------------------------------

\matconvnet supports the following activation functions:
%
\begin{itemize}
\item \emph{ReLU.} !vl_nnrelu! computes the \emph{Rectified Linear Unit} (ReLU):
\[
 y_{ijd} = \max\{0, x_{ijd}\}.
\]

\item \emph{Sigmoid.} !vl_nnsigmoid! computes the \emph{sigmoid}:
\[
 y_{ijd} = \sigma(x_{ijd}) = \frac{1}{1+e^{-x_{ijd}}}.
\]
\end{itemize}
%
See \autoref{s:impl-activation} for implementation details.

% ------------------------------------------------------------------
\section{Normalization}\label{s:normalization}
% ------------------------------------------------------------------

% ------------------------------------------------------------------
\subsection{Local response normalization (LRN)}\label{s:ccnormalization}
% ------------------------------------------------------------------

!vl_nnnormalize! implements a cross-channel normalization operator. Normalization applied independently at each spatial location and groups of channels to get:
\[
 y_{ijk} = x_{ijk} \left( \kappa + \alpha \sum_{t\in G(k)} x_{ijt}^2 \right)^{-\beta},
\]
where, for each output channel $k$, $G(k) \subset \{1, 2, \dots, D\}$ is a corresponding subset of input channels. Note that input $\bx$ and output $\by$ have the same dimensions. Note also that the operator is applied across feature channels in a convolutional manner at all spatial locations.

See \autoref{s:impl-ccnormalization} for implementation details.

% ------------------------------------------------------------------
\subsection{Batch normalization}\label{s:bnorm}
% ------------------------------------------------------------------

!vl_nnbnorm! implements batch normalization~\cite{ioffe2015}. Batch normalization is somewhat different from other neural network blocks in that it performs computation across images/feature maps in a batch (whereas most blocks process different images/feature maps individually). !y = vl_nnbnorm(x, w, b)! normalizes each channel of the feature map $\mathbf{x}$ averaging over spatial locations and batch instances. Let $T$ the batch size; then
\[
\mathbf{x}, \mathbf{y} \in \mathbb{R}^{H \times W \times K \times T},
\qquad\mathbf{w} \in \mathbb{R}^{K},
\qquad\mathbf{b} \in \mathbb{R}^{K}.
\]
Note that in this case the input and output arrays are explicitly treated as 4D tensors in order to work with a batch of feature maps. The tensors  $\mathbf{w}$ and $\mathbf{b}$ define component-wise multiplicative and additive constants. The output feature map is given by
\[
y_{ijkt} = w_k \frac{x_{ijkt} - \mu_{k}}{\sqrt{\sigma_k^2 + \epsilon}} + b_k,
\quad
\mu_{k} = \frac{1}{HWT}\sum_{i=1}^H \sum_{j=1}^W \sum_{t=1}^{T} x_{ijkt},
\quad
\sigma^2_{k} = \frac{1}{HWT}\sum_{i=1}^H \sum_{j=1}^W \sum_{t=1}^{T} (x_{ijkt} - \mu_{k})^2.
\]
See \autoref{s:impl-bnorm} for implementation details.

% ------------------------------------------------------------------
\subsection{Spatial normalization}\label{s:spnorm}
% ------------------------------------------------------------------

!vl_nnspnorm! implements spatial normalization. Spatial normalization operator acts on different feature channels independently and rescales each input feature by the energy of the features in a local neighborhood . First, the energy of the features is evaluated in a neighbourhood $W'\times H'$
\[
n_{i''j''d}^2 = \frac{1}{W'H'}
\sum_{1\leq i' \leq H', 1 \leq j' \leq W'} x^2_{
i''+i'-1-\lfloor \frac{H'-1}{2}\rfloor,
j''+j'-1-\lfloor \frac{W'-1}{2}\rfloor,
d}.
\]
In practice, the factor $1/W'H'$ is adjusted at the boundaries to account for the fact that neighbors must be cropped. Then this is used to normalize the input:
\[
y_{i''j''d} = \frac{1}{(1 + \alpha n_{i''j''d}^2)^\beta} x_{i''j''d}.
\]
See \autoref{s:spnorm} for implementation details.

% ------------------------------------------------------------------
\subsection{Softmax}\label{s:softmax}
% ------------------------------------------------------------------

!vl_nnsoftmax! computes the softmax operator:
\[
 y_{ijk} = \frac{e^{x_{ijk}}}{\sum_{t=1}^D e^{x_{ijt}}}.
\]
Note that the operator is applied across feature channels and in a convolutional manner at all spatial locations. Softmax can be seen as the combination of an activation function (exponential) and a normalization operator. See \autoref{s:impl-softmax} for implementation details.

% ------------------------------------------------------------------
\section{Losses and comparisons}\label{s:losses}
% ------------------------------------------------------------------

% ------------------------------------------------------------------
\subsection{Log-loss}\label{s:loss}
% ------------------------------------------------------------------

!vl_logloss! computes the \emph{logarithmic loss}
\[
 y = \ell(\bx,c) = - \sum_{ij} \log x_{ijc}
\]
where $c \in \{1,2,\dots,D\}$ is the ground-truth class. Note that the operator is applied across input channels in a convolutional manner, summing the loss computed at each spatial location into a single scalar.  See \autoref{s:impl-loss} for implementation details.

% ------------------------------------------------------------------
\subsection{Softmax log-loss}\label{s:sfloss}
% ------------------------------------------------------------------

!vl_softmaxloss! combines the softmax layer and the log-loss into one step for improved numerical stability. It computes
\[
y = - \sum_{ij} \left(
x_{ijc} - \log \sum_{d=1}^D e^{x_{ijd}}
\right)
\]
where $c$ is the ground-truth class. See \autoref{s:impl-sfloss} for implementation details.

% ------------------------------------------------------------------
\subsection{$p$-distance}\label{s:pdistance}
% ------------------------------------------------------------------

The !vl_nnpdist! function computes the $p$-th power $p$-distance between the vectors in the input data $\bx$ and a target $\bar\bx$:
\[
  y_{ij} = \left(\sum_d |x_{ijd} - \bar x_{ijd}|^p\right)^\frac{1}{p}
\]
Note that this operator is applied convolutionally, i.e. at each spatial location $ij$ one extracts and compares vectors $x_{ij:}$. By specifying the option !'noRoot', true! it is possible to compute a variant omitting the root:
\[
  y_{ij} = \sum_d |x_{ijd} - \bar x_{ijd}|^p, \qquad p > 0.
\]
See \autoref{s:impl-pdistance} for implementation details.

%% ------------------------------------------------------------------
%\subsection{Product}\label{s:product}
%% ------------------------------------------------------------------
%
%\[
% y_{ijd} = x^{(1)}_{ijd} x^{(2)}_{ijd}
%\]
%
%\paragraph{Implementation details.}
%\[
% \frac{dz}{dx^{(1)}_{ijd}}
%  =
% \sum_{i''j''d''}
%  \frac{dz}{dy_{i''j''d''}}
%  \frac{dy_{i''j''d''}}{dx^{(1)}_{ijd}}
%  =
%  \frac{dz}{dy_{ijd''}}
%  x^{(2)}_{ijd},
%  \qquad
%  \frac{dz}{dx^{(2)}_{ijd}}
%   =
%  \frac{dz}{dy_{ijd}}
%  x^{(1)}_{ijd}.
%\]
%
%
%% ------------------------------------------------------------------
%\subsection{Split}\label{s:split}
%% ------------------------------------------------------------------
%
%\[
% y_{ijd}^{(1)} = x_{ijd}, \qquad y_{ijd}^{(2)} = x_{ijd}
%\]
%
%\[
% \frac{dz}{dx_{ijd}} =
%\sum_{i''j''d''}
% \frac{dz}{dy_{i''j''d''}^{(1)}}
%  \frac{dy_{i''j''d''}^{(1)}}{dx_{ijd}}
% +
%  \frac{dz}{dy_{i''j''d''}^{(2)}}
%  \frac{dy_{i''j''d''}^{(2)}}{dx_{ijd}}
%\]


Some general instructions on using the template follow.
It is a good idea to keep this file around for future reference.

\subsection{Structure}
\label{sec:instructions:structure}
\lipsum[5] % Dummy text

\paragraph{Paragraph Description} \lipsum[7] % Dummy text

\paragraph{Different Paragraph Description} \lipsum[8] % Dummy text

\subsection{Citations and References}
\label{sec:instructions:references}
A statement requiring citation \cite{Figueredo:2009dg}.
You can also add references to \autoref{sec:introduction} and \autoref{sec:instructions:structure}.
It is also possible to use \sectionautorefname~\vref{sec:instructions:structure},
or \sectionautorefname~\vref{sec:summary}.

\subsubsection{Subsubsection}

\lipsum[12] % Dummy text

\begin{description}
\item[Word] Definition
\item[Concept] Explanation
\item[Idea] Text
\end{description}

\lipsum[12] % Dummy text

\begin{itemize}[noitemsep] % [noitemsep] removes whitespace between the items for a compact look
\item First item in a list
\item Second item in a list
\item Third item in a list
\end{itemize}

\subsubsection{Table}

\lipsum[13] % Dummy text

\begin{table}[hbt]
\caption{Table of Grades}
\centering
\begin{tabular}{llr}
\toprule
\multicolumn{2}{c}{Name} \\
\cmidrule(r){1-2}
First name & Last Name & Grade \\
\midrule
John & Doe & $7.5$ \\
Richard & Miles & $2$ \\
\bottomrule
\end{tabular}
\label{tab:label}
\end{table}

Reference to \autoref{tab:label}, or using Table~\vref{tab:label}.

\subsection{Figures}
\label{sec:instructions:figures}

Reference to \autoref{fig:gallery}.

\begin{figure}[tb]
\centering
\includegraphics[width=0.5\columnwidth]{GalleriaStampe}
% The text in the square bracket is the caption for the list of figures while the text in the curly brackets is the figure caption
\caption[An example of a floating figure]{
  An example of a floating figure (a reproduction from the
  \emph{Gallery of prints}, M.~Escher,\index{Escher, M.~C.}
  from \url{http://www.mcescher.com/}).
}
\label{fig:gallery}
\end{figure}

Reference the figure composed of multiple subfigures as \autoref{fig:esempio}.
For sub-figures, reference as \autoref{fig:ipsum}.
You can also reference as \figureautorefname~\vref{fig:ipsum},
which will also add the page number where appropriate.

\lipsum[5-8] % Dummy text

\begin{figure}[tb]
\centering
\subfloat[A city market.]{\includegraphics[width=.45\columnwidth]{Lorem}} \quad
\subfloat[Forest landscape.]{\includegraphics[width=.45\columnwidth]{Ipsum}\label{fig:ipsum}} \\
\subfloat[Mountain landscape.]{\includegraphics[width=.45\columnwidth]{Dolor}} \quad
\subfloat[A tile decoration.]{\includegraphics[width=.45\columnwidth]{Sit}}
% The text in the square bracket is the caption for the list of figures while the text in the curly brackets is the figure caption
\caption[A number of pictures.]{A number of pictures with no common theme.}
\label{fig:esempio}
\end{figure}

\subsection{Comments and to-do's}
There are many ways to add to-do items.
You can use \TODO{XX}{number here} to put a note with a side-note comment.

To add a reminder to add a reference to a future section use
\todoref{Section on benefits}.

The \texttt{todonotes} package has many ways of adding to-do's.
For example, you can use \todo{Simple to-do}, or a fancy one \todo[fancyline]{Fancy!}.
You can even add \todo[inline]{an inline} todo.

\begin{comment}
Everything inside this section will be ignore.
The comment package can also support generic comment environment, and conditional comments.
\end{comment}

You can also include end notes. \endnote{example endnote}
But, this may interfere with footnotes.\footnote{a footnote}

\subsection{Math}

Some mathematics in the text: $\cos\pi=-1$ and $\alpha$.

\begin{equation}
\cos^3 \theta =\frac{1}{4}\cos\theta+\frac{3}{4}\cos 3\theta
\label{eq:refname2}
\end{equation}

\lipsum[5] % Dummy text

\begin{definition}[Gauss]
To a mathematician it is obvious that
$\int_{-\infty}^{+\infty}
e^{-x^2}\,dx=\sqrt{\pi}$.
\end{definition}

\begin{theorem}[Pythagoras]
The square of the hypotenuse (the side opposite the right angle) is equal to the sum of the squares of the other two sides.
\end{theorem}

\begin{proof}
We have that $\log(1)^2 = 2\log(1)$.
But we also have that $\log(-1)^2=\log(1)=0$.
Then $2\log(-1)=0$, from which the proof.
\end{proof}

\subsection{Units}
Avoid writing units directly.
You can use \SI{60}{\km} and \SI{3.6}{\giga\Hz}.
You can do \si{\km\per\second}.
For binary units, see \SI{100}{\mebi\byte} and \SI[prefixes-as-symbols=false]{30}{\kibi\bit}.

You can use ranges such as \numrange{2}{3} and lists \numlist{1;2;3;4}.

\subsection{Lists}

\lipsum[5] % Dummy text

\begin{enumerate}[noitemsep] % [noitemsep] removes whitespace between the items for a compact look
\item First item in a list
\item Second item in a list
\item Third item in a list
\end{enumerate}

\subsection{Abbreviations}
You should use \eg, \etc, \adhoc, \circa, \ala, and \apriori from the foreign package.

\subsection{Algorithms}
A small easy algorithm:

\begin{algorithm}[H]
\SetAlgoLined
\KwData{this text}
\KwResult{how to write algorithm with \LaTeX2e }

initialization\;
\While{not at end of this document}{
  read current\;
  \eIf{understand}{
    go to next section\;
	current section becomes this one\;
  }{
    go back to the beginning of current section\;
  }
}
\caption{How to write algorithms}
\end{algorithm}

And a more involved algorithm:

\input{example_algorithm_1}

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "{{cookiecutter.file_name}}"
%%% End:


% --------------------------------
% studies, performance

% ==============
%
% --------------
\section{Parallel Strong Scaling and Comparison with OpenCMISS Iron}\label{sec:parallel_strong_scaling_opencmiss}

After the performance of different optimization types has been evaluated for a scenario with a single number of MPI processes in the last section, we now conduct a strong scaling study with the optimization type \code{fast-vc-apx-e}, which was found to be the most performant, and compare the runtimes to the reference software OpenCMISS Iron.

\subsection{Evaluation of Runtimes}\label{sec:strong_scaling_runtimes_opencmiss_opendihu}
For a fair comparison, we take care to exactly compute the same scenario with both software packages. The simulated scenario uses the same model as in the previous section: fiber based electrophysiology with the monodomain model given by \cref{eq:monodomain} on every muscle fiber, including the 1D electric conduction along the fibers and the 0D subcellular model of Shorten et al. \cite{Shorten2007}. The fibers are coupled with the bidomain equation in \cref{eq:bidomain1}, which is solved on the 3D domain to yield the EMG signals. No fat layer is considered in this study, as this feature is not available in our OpenCMISS implementation.

The simulated scenario contains 81 fibers with \num{1480} elements each, a coarse 3D mesh with \num{775} nodes and \num{6718591} degrees of freedom in total.
We use our improved OpenCMISS setup, which is discussed in \cref{sec:opencmiss_numeric_improvements} and employs the second order numerical timestepping schemes and the improved linear solvers:
The monodomain model is solved using a Strang operator splitting with Crank-Nicolson and Heun's methods. A conjugate-gradient solver is used for the linear system of the bidomain equation.

We use timestep widths of $\dt_\text{0D}=\SI{1e-4}{\milli\second}$, $\dt_\text{1D}=\dt_\text{splitting}=\SI{5e-4}{\milli\second}$, $\dt_\text{3D}=\SI{1e-1}{\milli\second}$ and a simulation end time of $t_\text{end}=\SI{2}{\milli\second}$.
During this time, the resulting values are written to output files after every $\SI{0.1}{\milli\second}$.
The fibers are assigned to 10 MUs that are activated in a ramp every $\SI{0.2}{\milli\second}$ from $t=\SI{0}{\milli\second}$ to $t=\SI{1.8}{\milli\second}$.

Within the strong scaling study, the same scenario is computed with different numbers of processes, ranging from one to 18 in this case. We use the cubes partitioning strategies presented in \cref{sec:opencmiss_parallel_partitioning} in both OpenCMISS and OpenDiHu.
The study is executed on the same Intel Core i9-10980XE processor as the studies in the previous section.
We measure the total user time of the simulation program, which includes the runtimes for initialization, computation of system matrices and the duration of file output. However, the majority of the runtime in this scenario is spent in the numerical solvers.

In OpenDiHu, we use the setup corresponding to the \code{fast-vc-apx-e} scenario in the last section with enabled approximation of log and exp functions. We measure the runtime of two variants. The first variant computes all subcellular models and performs the same work as the OpenCMISS Iron implementation. In the second variant, the adaptive computation described in \cref{sec:adaptive_computation_for_fiber_based} is enabled, which only computes fibers that have been activated and the subcellular models that are not in equilibrium. For the chosen ramp activation pattern of the MUs, the second variant computes approximately only half of the subcellular model instances.

% weak scaling runtime
\begin{figure}
  \centering%
  \begin{subfigure}[t]{0.49\textwidth}%
    \centering%
    \includegraphics[width=\textwidth]{images/results/studies/0_strong_scaling_runtime.pdf}%
    \caption{Runtime of the simulation programs for OpenCMISS (blue) and two variants of OpenDiHu (orange and green), see the description in the text for details.
    %For OpenDiHu, two variants are shown: the normal computation of all fibers (orange) and the optimization that computes only subcellular model instances that are not in equilibrium (green). In addition, the problem is solved on the GPU (red).
    }%
    \label{fig:0_strong_scaling_runtime}%
  \end{subfigure}
  \quad
  \begin{subfigure}[t]{0.47\textwidth}%
    \centering%
    \includegraphics[width=\textwidth]{images/results/studies/0_strong_scaling_memory.pdf}%
    \caption{Memory consumption per process at the end of the program.}%
    \label{fig:0_strong_scaling_memory}%
  \end{subfigure}
  \caption{Electrophysiology Solver in OpenDiHu: Strong scaling study of fiber based electrophysiology and comparison between the implementations of OpenDiHu and OpenCMISS Iron. The same scenario is solved with both software packages and for increasing numbers of processes from one to 18.}%
  \label{fig:0_strong_scaling}%
\end{figure}%

\Cref{fig:0_strong_scaling_runtime} shows the resulting runtimes in this study. It can be seen that the runtime decreases monotonically for higher numbers of processes for all three tested simulations. The OpenDiHu implementation exhibits lower runtimes for all numbers of processes. The reduction in runtime between OpenCMISS Iron and the first OpenDiHu variant is given by a factor of approximately 100 with a maximum factor of 186 for 4 processes. In addition, the second OpenDiHu variant approximately halves the runtimes as expected, because only half of the subcellular models are computed.

In summary, the improvements to the EMG simulation software, which are described in this work, include the numerical improvements in \cref{sec:opencmiss_numeric_improvements} with a speedup of 2.5, the software improvements with a speedup of over 100 and a measured maximum of 186, and the algorithmic improvement of adaptive 0D model computations, whose speedup factor is scenario dependent. In the present study, the two latter factors, i.e., the speedup between the improved OpenCMISS Iron software and the OpenDiHu scenario with adaptive computation, give a combined maximum speedup of 363 for the measurement with two processes.

Moreover, the computation of this study was also carried out with the \code{gpu} optimization type in OpenDiHu, using the same GPU as in the last section. One process was started on the CPU, which offloaded the computational work of the 0D and 1D problems to the GPU. However, the GPU memory was not sufficient for the computation of all 81 fibers. Therefore, we only compute one fiber, but keep the rest of the simulation scenario equal to the other measurements. The req square in \cref{fig:0_strong_scaling_runtime} shows the measured runtime multiplied by the factor 81 for compensation. As in the studies of the previous section, the computation on the GPU has higher runtimes than the computation on the CPU.

As the memory consumption was a limiting factor for parallelism in OpenCMISS Iron as shown in \cref{sec:opencmiss_memory}, we also measure the memory consumption per process at the end of the simulation in both software frameworks. \Cref{fig:0_strong_scaling_memory} shows the result for OpenCMISS and OpenDiHu. The two variants of OpenDiHu have the same memory consumption characteristics, as the only difference between the variants is that the computation of certain subcellular models is switched on or off.

It can be seen that the increased parallelism leads to a reduction of the used memory per process in both pieces of software. OpenDiHu approaches a saturation value of $\SI{200}{\mebi\byte}$ for eight and more processes. For OpenCMISS, the memory consumption is higher, but reduces more quickly also for higher numbers of processes. However, the relation between the two curves increases from a value of $\SI{6.168}{\gibi\byte} : \SI{1.078}{\gibi\byte} = 5.7$ for one process to $\SI{2.054}{\gibi\byte} : \SI{0.202}{\gibi\byte} = 10.2$ for 18 processes.

As a result, this study shows a large memory efficiency improvement in OpenDiHu compared with the OpenCMISS Iron software. For OpenCMISS, the memory scaling in this parallel strong scaling scenario is not as bad as in the parallel weak scaling considered in \cref{fig:opencmiss_memory} in \cref{sec:opencmiss_memory}. However, the total memory for all processes still increases to approximately $18 \cdot \SI{2.054}{\gibi\byte} \approx \SI{37}{\gibi\byte}$, which is higher than the main memory capacity of the used processor.

% roofline model
\begin{figure}[H]
  \centering%
  \includegraphics[width=0.8\textwidth]{images/results/studies/0_roofline.pdf}%
  \caption{Electrophysiology model in OpenDiHu: Roofline model of the strong scaling study. The blue, green and orange data points correspond to the runs of the OpenCMISS and OpenDiHu variants, as given in \cref{fig:0_strong_scaling_runtime}.}%
  \label{fig:0_roofline}%
\end{figure}%

\subsection{Roofline Model}\label{sec:roofline_model}
To further investigate the computational behavior, we also present the performance measurements of the solvers in a roofline model.
\Cref{fig:0_roofline} shows the resulting diagram with the data points of all CPU runs in the strong scaling study of \cref{fig:0_strong_scaling_runtime}. The $x$ axis shows the computational intensity of the simulation, which is measured in double-precision floating-point operations (FLOP) per byte of data that are transferred between the CPU and the main memory and caches. The $y$ axis measures the performance in GFLOP per second. The highest possible performance is given by the peak performance of the processor, which is $P_\text{max} = \SI{715}{\giga\flop\per\second}$ in this case. Furthermore, the performance is limited by the amount of payload data that can be transferred to the CPU over the memory bus. The memory bandwidths of the L1, L2 and L3 caches and the main memory (DRAM) correspond to the shown diagonals in \cref{fig:0_roofline} and form the \say{roofline} of the model.

We measured the memory bandwidths of the Caches and the peak performance using the Empirical Roofline Tool \cite{ert}. The main memory bandwidth was retrieved from the processors' documentation.

To locate the simulation runs of the study in the roofline model, we used hardware counters to count floating point instructions and memory access operations.
For the runs with OpenCMISS Iron and OpenDiHu, we started the hardware counters \SI{90}{\second}, respectively \SI{15}{\second} after the beginning of the simulation, such that the initialization phase was not included in the measured data. The counters were kept active for 15, 30 and 60 seconds, depending on the expected runtimes of the different runs. The counted numbers of events were then divided by the acquisition time to yield the required rates of memory bandwidth and floating-point performance.

\Cref{fig:0_roofline} shows the measured points in the roofline model corresponding to the curves in \cref{fig:0_strong_scaling_runtime}. All data points are located at the right-hand side of the memory bandwidth limits, which indicates that the simulation is compute bound.
The highest computational intensity and performance are both achieved by the OpenDiHu variant given in orange color, which computes all fibers and subcellular models regardless of their activation state. The values for the adaptive variant given in green color are lower, as fewer computations are performed and a higher portion of the runtime and compute power is spent on determining which subcellular model has to be computed. The two metrics are lowest for the OpenCMISS runs given in blue color.

The roofline diagram shows the data points for all parallel runs and the number of processes is noted in the plot. The run with 18 processes is the most meaningful, as this means that the whole processor is employed. The largest performance for the OpenDiHu run in orange color has a value of \SI{180.157}{\giga\flop\per\second} which corresponds to \SI{25.2}{\percent} of the peak performance and is a very good value. The rated \SI{100}{\percent} of peak performance for processors are practically unreachable. For example, the peak performance assumes only fused multiply add operations and requires a power management that maximizes the employment of the boost clock frequency in the processor. These conditions are not fulfilled in our computations of realistic models and scenarios.
The performance values of the runs with 18 processes for the green and blue data points are \SI{13.9}{\percent} and \SI{6.0}{\percent}, respectively.

Furthermore, the measurements with lower process counts can also be assessed with a scaled down peak performance according to the fraction of used cores. However, this assessment is slightly off, as, e.g., the CPU can use a higher clock frequency, if only the heat dissipation of one active core has to be compensated. The performance for the OpenDiHu run with one process given by the orange point is \SI{11.966}{\giga\flop\per\second}, which corresponds to \SI{30.1}{\percent} of the fractional peak performance of $\SI{715}{\giga\flop\per\second}/18 = \SI{39.7}{\giga\flop\per\second}$.

% Crank-Nicolson
% scenario_name: vc-1-ws,  n_subdomains: 1 1 1,  n_ranks: 1,  end_time: 2.0
% dt_0D:           1e-04, diffusion_solver_type:      cg
% dt_1D:           5e-04, potential_flow_solver_type: gmres, approx. exp.: True
% dt_splitting:    5e-04, emg_solver_type:            cg, emg_initial_guess_nonzero: False
% dt_3D:           1e-01, paraview_output: True, optimization_type: vc, enable_weak_scaling: True
% output_timestep: 1e-01  stimulation_frequency: 1.0 1/ms = 1000.0 Hz
% fiber_file:              /data/scratch/maierbn/opendihu/examples/electrophysiology/input/left_biceps_brachii_9x9fibers.bin
% cellml_file:             /data/scratch/maierbn/opendihu/examples/electrophysiology/input/new_slow_TK_2014_12_08.cellml
% fiber_distribution_file: /data/scratch/maierbn/opendihu/examples/electrophysiology/input/MU_fibre_distribution_10MUs.txt
% firing_times_file:       /data/scratch/maierbn/opendihu/examples/electrophysiology/input/MU_firing_times_immediately.txt
% ********************************************************************************
% prefactor: sigma_eff/(Am*Cm) = 0.03079310344827586 = 8.93 / (500.0*0.58)
% diffusion solver type: cg
% n fibers:              81 (9 x 9)
% n points per fiber:    1481
% 1 rank, partitioning: x1 x y1 x z1
% 9 x 9 = 81 fibers, per partition: 9 x 9 = 81
% per fiber: 1D mesh    nodes global: 1481, local: 1481
%   sampling 3D mesh with stride 2 x 2 x 50
%     linear 3D mesh    nodes global: 5 x 5 x 31 = 775, local: 5 x 5 x 31 = 775
%     linear 3D mesh elements global: 4 x 4 x 30 = 480, local: 4 x 4 x 30 = 480
% number of degrees of freedom:
%                     1D fiber:       1481  (per process: 1481)
%             0D-1D monodomain:      82936  (per process: 82936)
%  all fibers 0D-1D monodomain:    6717816  (per process: 6717816)
%                  3D bidomain:        775  (per process: 775)
%                        total:    6718591  (per process: 6718591)
%

% weak scaling data
% opencmiss
% 1: duration 82046.0 s, memory consumption per process (high watermark): 6.168 GB
% 2: duration 52877.0 s, memory consumption per process (high watermark): 5.257 GB
% 4: duration 33802.0 s, memory consumption per process (high watermark): 4.639 GB
% 8: duration 17230.0 s, memory consumption per process (high watermark): 4.027 GB
% 12: duration 8840.0 s, memory consumption per process (high watermark): 2.829 GB
% 18: duration 5122.0 s, memory consumption per process (high watermark): 2.054 GB
%
% opendihu (all fibers)
% 1: duration: 586.3s, 584.7s, memory consumption per process (high watermark): 1.078 GB, speedup to opencmiss: 139.93621121932085
% 2: duration: 303.9s, 298.9s, memory consumption per process (high watermark): 0.783 GB, speedup to opencmiss: 174.00046069301393
% 4: duration: 181.6s, 175.5s, memory consumption per process (high watermark): 0.447 GB, speedup to opencmiss: 186.09081024539964
% 8: duration: 111.9s, 104.8s, memory consumption per process (high watermark): 0.278 GB, speedup to opencmiss: 153.99740805291145
% 12: duration: 87.5s, 79.9s, memory consumption per process (high watermark): 0.249 GB, speedup to opencmiss: 100.99490645975152
% 18: duration: 58.3s, 49.7s, memory consumption per process (high watermark): 0.202 GB, speedup to opencmiss: 87.8207692747328
%
% opendihu (only active parts)
% 1: duration: 270.1s, 268.4s, memory consumption per process (high watermark): 1.079 GB, speedup to opencmiss: 303.70534888025173
% 2: duration: 145.3s, 140.3s, memory consumption per process (high watermark): 0.783 GB, speedup to opencmiss: 363.96613436123346
% 4: duration: 96.5s, 90.3s, memory consumption per process (high watermark): 0.449 GB, speedup to opencmiss: 350.15279432330243
% 8: duration: 64.6s, 57.7s, memory consumption per process (high watermark): 0.278 GB, speedup to opencmiss: 266.7389116804706
% 12: duration: 49.4s, 41.7s, memory consumption per process (high watermark): 0.249 GB, speedup to opencmiss: 178.76944336776825
% 18: duration: 42.0s, 33.7s, memory consumption per process (high watermark): 0.202 GB, speedup to opencmiss: 122.07995127183165
%
% roofline data:
% opencmiss
% 1 rank(s), performance: 3.332 GFLOP/s, mem bandwidth: 3.117 GB/s, intensity: 64.143 FLOP/B
% 2 rank(s), performance: 5.087 GFLOP/s, mem bandwidth: 5.793 GB/s, intensity: 52.702 FLOP/B
% 4 rank(s), performance: 7.583 GFLOP/s, mem bandwidth: 9.507 GB/s, intensity: 47.871 FLOP/B
% 8 rank(s), performance: 8.616 GFLOP/s, mem bandwidth: 16.026 GB/s, intensity: 32.276 FLOP/B
% 12 rank(s), performance: 19.012 GFLOP/s, mem bandwidth: 25.037 GB/s, intensity: 45.634 FLOP/B
% 18 rank(s), performance: 42.582 GFLOP/s, mem bandwidth: 42.175 GB/s, intensity: 60.725 FLOP/B
%
% opendihu (all fibers)
% 1 rank(s), performance: 11.966 GFLOP/s, mem bandwidth: 1.550 GB/s, intensity: 231.674 FLOP/B
% 2 rank(s), performance: 25.273 GFLOP/s, mem bandwidth: 2.839 GB/s, intensity: 267.084 FLOP/B
% 4 rank(s), performance: 45.371 GFLOP/s, mem bandwidth: 5.396 GB/s, intensity: 252.419 FLOP/B
% 8 rank(s), performance: 81.855 GFLOP/s, mem bandwidth: 9.770 GB/s, intensity: 251.630 FLOP/B
% 12 rank(s), performance: 107.793 GFLOP/s, mem bandwidth: 13.531 GB/s, intensity: 239.385 FLOP/B
% 18 rank(s), performance: 180.157 GFLOP/s, mem bandwidth: 19.612 GB/s, intensity: 276.235 FLOP/B
%
% opendihu (only active parts)
% 1 rank(s), performance: 6.376 GFLOP/s, mem bandwidth: 1.679 GB/s, intensity: 56.978 FLOP/B
% 2 rank(s), performance: 18.736 GFLOP/s, mem bandwidth: 2.512 GB/s, intensity: 111.957 FLOP/B
% 4 rank(s), performance: 29.185 GFLOP/s, mem bandwidth: 5.178 GB/s, intensity: 84.649 FLOP/B
% 8 rank(s), performance: 46.207 GFLOP/s, mem bandwidth: 9.046 GB/s, intensity: 76.793 FLOP/B
% 12 rank(s), performance: 84.096 GFLOP/s, mem bandwidth: 12.867 GB/s, intensity: 98.369 FLOP/B
% 18 rank(s), performance: 99.256 GFLOP/s, mem bandwidth: 17.020 GB/s, intensity: 87.893 FLOP/B

\begin{reproduce_no_break}
  The scripts to run the studies in this scenario and to create the plots are available in the repository at \href{https://github.com/dihu-stuttgart/performance}{github.com/dihu-stuttgart/performance}
  in the directory \code{opendihu/20_fibers_emg_avx_opencmiss}:
  \begin{lstlisting}[columns=fullflexible,breaklines=true,postbreak=\mbox{\textcolor{gray}{$\hookrightarrow$}\space}]
    ./0_run.sh
  \end{lstlisting}
  The directory also contains a script that performs all steps to install OpenCMISS, if needed.

  Note, the studies in the previous and the current sections were carried out on the computer with hostname \code{pcsgs05} in the institute network at the time of writing.
\end{reproduce_no_break}

%-----
\section{Performance Measurements on the GPU}\label{sec:performance_gpu}

In the previous two sections, measurements were made on a GPU, which produced worse results than the CPU code. The used GPU was an NVIDIA GeForce RTX 3080, a high-end consumer graphics card, which mainly targets graphics rendering performance using single-precision operations. The ratio between double-precision and single-precision performance is 1:64. However, single-precision calculations were found to not yield a stable subcellular model solver, as the precision is too low.
% quadro: https://www.techpowerup.com/gpu-specs/quadro-gp100.c2994
% RTX: https://www.techpowerup.com/gpu-specs/geforce-rtx-3080.c3621

In this section, we conduct two studies, where the first study uses the same GPU hardware as before. The second study is executed on an NVIDIA Quadro GP100 GPU, which has a double-precision to single-precision performance ratio of 1:2.
The rated double-precision performance of the Quadro card is \SI{5.168}{\tera\flops}, which is ten times higher than the value of \SI{465.1}{\giga\flops} for the GeForce card.

The CPU hardware connected with the Quadro card contains a dual-socket CPU with two 12-core Intel Xeon Silver 4116 processors with \SI{2.1}{\giga\hertz} base frequency and \SI{3}{\giga\hertz} maximum turbo frequency, yielding a total core count of 24, and being equipped with main memory of \SI{188}{\gibi\byte}.

Computational hardware can be compared by its average thermal design power dissipation (TDP). For the studies in the previous two sections, the TDP values for the used CPU and GPU were \SI{165}{\watt} and \SI{320}{\watt}. For the second study in the current section, the values for CPU and GPU are $2\cdot \SI{85}{\watt} = \SI{170}{\watt}$ and \SI{235}{\watt}. This indicates that the employed hardware is in a comparable electrical power range. However, the GPU is more specialized for our double-precision needs.

\subsection{Strong Scaling with the GPU for the Hodgkin-Huxley Model}\label{sec:strong_scaling_gpu_hodgkin_huxley}

While the studies in the last two sections only used one process on the CPU, which managed the offloaded computations on the GPU,
we now additionally consider parallelism on the CPU.

The first study compares the strong scaling with and without GPU acceleration. The runs with GPU acceleration partition the computational domain as usual to multiple subdomains, which are handled by dedicated processes on the CPU. The solution of the 0D and 1D models is performed on the GPU, and every process independently transfers its part of the computational work to the same GPU. The 3D model is fully solved using MPI parallelism on the CPU. We compare this setup with a pure CPU based strong scaling study.

The scenario solves the fiber based electrophysiology model without fat layer with 169 fiber meshes of 1480 elements each and a 3D mesh with 1984 elements. The subcellular model of Hodgkin and Huxley \cite{Hodgkin1952} is used. The computation uses the same numerical parameters as in the first study in \cref{sec:evaluation_of_code_gen}, a 3D solver timestep of $\dt_\text{3D}=\SI{4e-01}{\ms}$ and a simulation end time of $\SI{10}{\ms}$. Moreover, the setup equals the settings of the \code{fast-vc} and \code{fast-gpu} scenarios in \cref{sec:evaluation_of_code_gen}.

\begin{figure}%
  \centering%
  \begin{subfigure}[t]{0.45\textwidth}%
    \centering%
    \includegraphics[height=52mm]{images/results/studies/16_hodgkin_huxley_gpu.pdf}%
    \caption{Study, where every process on the CPU offloads the 0D and 1D model computations to the GPU.}%
    \label{fig:16_hodgkin_huxley_gpu}%
  \end{subfigure}
  \,
  \begin{subfigure}[t]{0.53\textwidth}%
    \centering%
    \includegraphics[height=52mm]{images/results/studies/16_hodgkin_huxley_cpu.pdf}%
    \caption{CPU-only study.}%
    \label{fig:16_hodgkin_huxley_cpu}%
  \end{subfigure}
  \caption{Electrophysiology model in OpenDiHu: Strong scaling study with and without GPU usage. A scenario with 169 fibers and the subcellular model of Hodgkin and Huxley is simulated. The vertical bars indicate the standard deviation of the runtimes in the set of measurements, which consists of multiple runs and the values of all processors in every run.}%
  \label{fig:16_hodgkin_huxley_cpu_gpu}%
\end{figure}%

\Cref{fig:16_hodgkin_huxley_cpu_gpu} presents the results for the two studies with and without GPU usage. \Cref{fig:16_hodgkin_huxley_cpu} shows the runtimes of different parts in the simulation of the CPU-only strong scaling study. It can be seen that the 0D computations account for most of the runtime, followed by the 1D computations. The 0D computations involve the solution of the Hodgkin-Huxley subcellular model. The 1D computations consists of solving 1D problems in serial using the Thomas algorithm. The 3D solver time is negligible as the 3D problem is only solved every \num{13333} timesteps. The 3D solution is only required right before the VTK file output step for the EMG values. This step occurs every $\SI{0.4}{\ms}$, which corresponds to an EMG sampling frequency of \SI{2.5}{\kilo\hertz}.
The runtimes for initialization and the file output itself are also very low compared with the runtimes of the computations.

\Cref{fig:16_hodgkin_huxley_cpu} shows that the total runtime decreases with higher process counts in this strong scaling study. The parallel efficiency $E_p = T_1/(T_p\,p)$ reaches $E_p=\SI{65.2}{\percent}$ for $p=18$ processes. We observe that the 0D and the 1D solver and the VTK file output have good strong scaling properties, whereas the initialization and the solution of the 3D model contain serial code portions that prohibit optimal scaling.

\Cref{fig:16_hodgkin_huxley_gpu} shows the analog study, where the 0D and 1D computations are offloaded to the GPU. The runtimes of these individual model parts are not explicitly measured, only the total runtime is known.

The plot shows an increasing total runtime for higher CPU parallelism.
The runtimes for initialization, file output and the 3D solver are equal to the CPU-only study.
The increasing total runtime shows that the GPU is better at solving the complete 0D and 1D problems
given by one CPU process than at the same computation, but split to several parts and provided by different MPI processes.
The benefit of using multiple CPU processes to interface the GPU in this study is, thus, only that the VTK output functionality gets parallelized. However, this effect is negligible.

An absolute comparison between the runtimes in \cref{fig:16_hodgkin_huxley_gpu} and \cref{fig:16_hodgkin_huxley_cpu} also reveals that the scenarios for one to 18 processes using the GPU have 4.8 to 181 times longer total runtimes. In this study, the memory transfer between the CPU and the GPU has a low influence on the total runtime, as this transfer only happens before and after the 3D model is solved. The measured runtimes, therefore, correspond to the computation on the GPU.

%0/18 : This is opendihu 1.2, built Jan 16 2021, C++ 201402, GCC 10.2.0, current time: 2021/1/17 12:37:12, hostname: pcsgs02, n ranks: 18
%0/18 : Open MPI v3.1.6, package: Open MPI maierbn@sgscl1 Distribution, ident: 3.1.6, repo rev: v3.1.6, Mar 18, 2020
%0/18 : File "/data/scratch/maierbn/opendihu/examples/electrophysiology/fibers/fibers_emg/settings_fibers_emg.py" loaded.
%0/18 : ---------------------------------------- begin python output ----------------------------------------
%Loading variables from "/data/scratch/maierbn/performance/opendihu/16_hodgkin_huxley_gpu/hodgkin_huxley_gpu.py".
%scenario_name: gpu-18,  n_subdomains: 3 2 3,  n_ranks: 18,  end_time: 10.0
%dt_0D:           3e-05, diffusion_solver_type:      cg
%dt_1D:           1e-05, potential_flow_solver_type: gmres
%dt_splitting:    3e-05, emg_solver_type:            cg, emg_initial_guess_nonzero: False
%dt_3D:           4e-01, paraview_output: True, optimization_type: gpu, use_single_precision: False
%output_timestep: 4e-01  stimulation_frequency: 0.1 1/ms = 100.0 Hz
%fiber_file:              /data/scratch/maierbn/opendihu/examples/electrophysiology/input/left_biceps_brachii_13x13fibers.bin
%cellml_file:             /data/scratch/maierbn/opendihu/examples/electrophysiology/input/hodgkin_huxley_1952.c
%fiber_distribution_file: /data/scratch/maierbn/opendihu/examples/electrophysiology/input/MU_fibre_distribution_10MUs.txt
%firing_times_file:       /data/scratch/maierbn/opendihu/examples/electrophysiology/input/MU_firing_times_always.txt
%********************************************************************************
%prefactor: sigma_eff/(Am*Cm) = 0.03079310344827586 = 8.93 / (500.0*0.58)
%diffusion solver type: cg
%n fibers:              169 (13 x 13)
%n points per fiber:    1481
%18 ranks, partitioning: x3 x y2 x z3
%13 x 13 = 169 fibers, per partition: 4 x 6 = 24
%per fiber: 1D mesh    nodes global: 1481, local: 494
  %sampling 3D mesh with stride 2 x 2 x 50
    %linear 3D mesh    nodes global: 8 x 8 x 31 = 1984, local: 3 x 4 x 10 = 120
    %linear 3D mesh elements global: 7 x 7 x 30 = 1470, local: 3 x 4 x 10 = 120
%number of degrees of freedom:
                    %1D fiber:       1481  (per process: 494)
            %0D-1D monodomain:       5924  (per process: 1976)
 %all fibers 0D-1D monodomain:    1001156  (per process: 47424)
                 %3D bidomain:       1984  (per process: 120)
                       %total:    1003140  (per process: 47544)
%Python config parsed in 0.1s.

\subsection{Evaluation of Hybrid CPU and GPU Computation for the Shorten Model}\label{sec:evaluation_hybrid}

Whereas previously, the subcellular model of Hodgkin and Huxley was solved on the GPU, we now switch to the more compute intense model of Shorten et al. \cite{Shorten2007}. This model has higher memory demands, such that it is not possible to solve it with OpenMP 4.5 for a muscle fiber mesh with 1481 nodes on the GeForce RTX 3080 GPU. As noted before, we use the NVIDIA Quadro GP100 GPU for the next study.
This GPU is also not capable of solving the whole set of 1481 models instances per fiber for 169 fibers at the same time. For one instance of the subcellular model, 57 state and rate variables each, and 71 intermediate variables have to be stored, along with other data, such as element lengths for every element.

Thus, we follow a hybrid approach. We parallelize the scenario to 27 processes on the CPU. Only one process offloads its subdomain to the GPU. In this way, both the CPU and the GPU take part in the computation and the available hardware capabilities are fully exploited.

\begin{figure}
  \centering%
  \includegraphics[width=0.9\textwidth]{images/results/studies/gpu_parallelization.png}%
  \caption{Electrophysiology solver in OpenDiHu: Partitioning of the 169 fibers to 27 processes used in the runtime study with hybrid CPU-GPU usage. The image also shows the resulting EMG signals $\phi_e$ on the muscle surface.}%
  \label{fig:gpu_parallelization}%
\end{figure}

The scenario and the numerical parameters are the same as described for the study with the Shorten model in \cref{sec:evaluation_of_code_gen}. 49 muscle fibers are used and parallelized to $3\times 3\times 3=27$ subdomains. \Cref{fig:gpu_parallelization} visualizes the partitioning of the fibers by different colors and the EMG values $\phi_e$ on the muscle surface at the simulation end time of $t_\text{end}=\SI{10}{\ms}$.

\Cref{fig:17_shorten_gpu} visualizes the runtimes of two runs. The first bar only employs the CPU and provides the reference for the measured runtimes. The second bar corresponds to the hybrid run, where one process employs the GPU. The 0D and 1D solver runtimes in the second bar are averaged over the CPU computations. The total runtime involves both the CPU and the GPU computations.
It can be seen that the total runtime given by the total bar heights is higher for the hybrid run. In the hybrid runs, the CPU processes have to wait at the synchronization point in the solution of the 3D problem until the GPU process has completed the 0D/1D computations.

\begin{figure}
  \centering%
  \includegraphics[width=0.6\textwidth]{images/results/studies/17_shorten_gpu.pdf}%
  \caption{Electrophysiology model in OpenDiHu: Runtime study for a CPU-only computation and a hybrid parallelization that employs both the CPU and the GPU. For the hybrid approach, one of the 27 partitions was computed on the GPU. The compute-intense subcellular model of Shorten et al. is simulated.}%
  \label{fig:17_shorten_gpu}%
\end{figure}

\subsection{Conclusion}

Several scenarios with computations of the 0D subcellular and 1D electric conduction models on the GPU have been evaluated.
\Cref{sec:evaluation_of_code_gen} compared the runtime for the Hodgkin-Huxley subcellular model on 625 fibers on the GPU with implementations on the CPU.
In \cref{sec:strong_scaling_runtimes_opencmiss_opendihu}, the computation on the GPU with the Shorten subcellular model was measured for one fiber.
\Cref{sec:strong_scaling_gpu_hodgkin_huxley} conducted a strong scaling study with the Hodgkin-Huxley model on 169 fibers and \cref{sec:evaluation_hybrid} evaluated a hybrid approach, where the Shorten model on different fibers was computed on the GPU and the CPU at the same time. This last study used a GPU with higher double-precision performance than the previous studies.

In all of these studies, the GPU computations could not compete with their CPU counterparts. The GPU implementation of the models relied on target-specific CUDA code, which was automatically generated by the OpenMP 4.5 pragmas produced by the code generator in OpenDiHu. The CPU computations used highly optimized CPU code with explicit vector instructions, an approach that is close to optimal as shown by the roofline model in \cref{sec:roofline_model}. Thus, the comparison considers different levels of optimization. We cannot conclude in general that the GPU is less suited to solve the fiber based electrophysiology models than the CPU. However, the GPU support in OpenMP is not competitive with our optimized CPU implementation.

The required GPU support of OpenMP in the GNU compiler is functional to the extent needed in our studies only since GCC version 11, which, at the time of writing, is still experimental and not yet released. Further performance gains can be expected in the future as compiler development advances. One problem is also the high memory requirement for the subcellular models, which only allows a certain number of subcellular model instances to be computed on the given hardware. It is also not clear, whether the high memory consumption is also an artifact of the compiler and will reduce with later compiler versions.

Despite the lower performance, it was shown that OpenDiHu can be used to solve the monodomain equation \cref{eq:monodomain} with different subcellular models on the GPU. Switching between the CPU and GPU variants can be accomplished by only changing the \code{optimizationType} parameter between \code{`vc`} and \code{`gpu`}. Hybrid strategies, where some processes use \code{`vc`} and others \code{`gpu`}, have been demonstrated.

In future work, the performance issue of the GPU computations can be addressed by using different technologies to access the compute power of GPUs.
Examples are to directly use the CUDA programming language or the C++ based abstraction layer for acceleration hardware SYCL \cite{sycl}. OpenDiHu already provides a reference implementation for such improvements by the code generator, which outputs model specific code with OpenMP pragmas for GPU offloading. This code implements proper, economical data transfer between the devices and contains hints how to distribute the workload on the GPU by the respective pragma placements. It could be used as a starting point to integrate further \say{optimization types} in the code generator.

%-----

% GPU slower in all studies
% comparison with highly optimized CPU code, near optimal
% GPU OpenMP not yet optimal, requires experimental GCC 11
% problem is also memory requiremnet
% GPU can be easily switched, hybrid is possible in opendihu
% future work: SYCL, CUDA extended code generator

\begin{reproduce_no_break}
  The scripts to run the studies and to create the plots for \cref{fig:16_hodgkin_huxley_cpu_gpu,fig:17_shorten_gpu} are available in the repository at \href{https://github.com/dihu-stuttgart/performance}{github.com/dihu-stuttgart/performance}
  in the directories \code{opendihu/16_hodgkin_huxley_gpu} and  \code{opendihu/17_shorten_gpu}.
\end{reproduce_no_break}

% ------------
%
% f===========


\chapter{A Vision Workbench Cookbook}\label{ch:cookbook}

This chapter provides simple bite-sized examples of how to use the
Vision Workbench to perform a range of common tasks.

\vfill\eject
\section{Removing Camera Lens Distortion}\label{sec:cookbook.distortion}
All digital camera systems introduce some amount of distortion in the
image.  In some high-precision cameras this distortion may be very small,
while in other such as those with fisheye lenses it may be very large.
In either case removing camera lens distortion is the first step in many
image processing algorithms.  The idea is to transform the image so that
it looks as if it were taken with a perfect pinhole camera.  The resulting
image is usually called either ``linearized'', ``undistorted'', or ``rectified''.

First you must create a camera model object, such as a \verb#PinholeModel# or
\verb#CAHVORModel#, containing a reasonably accurate model of your camera.
(Computing one if you don't have one already is another topic altogether.)
Then you can simply compute the linearized image like this.
\begin{verbatim}
  result = linearize_camera_transform( image, camera_model );
\end{verbatim}
This function is declared in the header \verb#<vw/Camera/CameraTransform.h>#
and located in the \verb#vw::camera# namespace.  By default it uses
the usual default edge extension and interpolation modes, which may not
be best depending on your needs.  Often you will get better linearization
results by overriding those defaults using the optional arguments.
\begin{verbatim}
  result = linearize_camera_transform( image, camera_model,
                                ConstantEdgeExtension(), BicubicInterpolation() );
\end{verbatim}

In many cases you would also like to know the linearized camera model that the
linearized image corresponds to.  You can compute it directly using the
\verb#linearize_camera# function.
\begin{verbatim}
  linearized_model = linearize_camera( camera_model, image.cols(), image.rows() );
\end{verbatim}
The linearization function needs to know the image dimensions so that
it can select the linearized model that best fits the distorted model
over the given range.  If you do use that function to compute the
linearized model in advance, or if you want to specify an alternate
linearized camera model of your own, then you should use the more general
camera transformation function.
\begin{verbatim}
  result = camera_transform( image, camera_model, linearized_model );
\end{verbatim}
This function allows you to transform images between any two concentric
camera models; undistorting an image is just a special case.  You can again
override the default edge extension and interpolation settings if you wish.

Finally, if you require an even greater degree of control you can
achieve it by manually creating a \verb#CameraTransform# object and
applying it to the source image directly using the regular
\verb#transform# function.  This is in fact exactly what happens inside
the convenience functions described above.  Here is a simple example,
assuming the CAHVOR camera model.
\begin{verbatim}
  CAHVORModel cahvor( "my_camera_model.cahvor" );
  CAHVModel cahv = linearize_camera( cahvor, image.cols(), image.rows() );
  CameraTransform<CAHVORModel,CAHVModel> ctx( cahvor, cahv );
  result = transform( image, ctx );
\end{verbatim}
You might choose to do it this way if you want to compose the camera
transform with another transform in a single step for efficiency,
for example.