\section{Closed-loop \tuner for asynchronous training}
\label{sec:async_app}
In Section~\ref{sec:async_tuner}, we briefly discuss the closed-loop momentum control mechanism in \asynctuner. In this section, after presenting more preliminaries on asynchrony, we show with details on the mechanism:
it measures the dynamics on a running system and controls momentum with a negative feedback loop.
\paragraph{Preliminaries}
Asynchrony is a popular parallelization technique \citep{recht2011hogwild} that avoids synchronization barriers.
When training on $M$ asynchronous workers, staleness (the number of model updates between a worker's read and write operations) is on average $\tau=M-1$,
i.e., the gradient in the SGD update is delayed by $\tau$ iterations as $\nabla f_{S_{t - \tau}}(x_{t - \tau} )$.
Asynchrony yields faster steps, but can
increase the number of iterations to achieve the same solution,
a tradeoff between hardware and statistical
efficiency~\citep{DBLP:journals/pvldb/ZhangR14}.
\citet{mitliagkas2016asynchrony} interpret asynchrony as added momentum dynamics.
Experiments in \citet{hadjis2016omnivore} support this finding, and demonstrate that reducing algorithmic momentum can compensate for asynchrony-induced momentum
and significantly reduce the number of iterations for convergence.
Motivated by that result, we use the model
in~\eqref{equ:exp_async_update_app}, where the total momentum, $\mu_T$, includes both asynchrony-induced and algorithmic  momentum, $\mu$, in~\eqref{eqn:momentum_gd}.
\begin{equation}
	\mathbb{E}[ x_{t+1} - x_t ]
	= \mu_T \mathbb{E}[x_t - x_{t-1}] - \alpha \mathbb{E}\nabla f(x_{t})
\label{equ:exp_async_update_app}
\end{equation}
We will use this expression to design an estimator for the value of total momentum, $\hat{\mu}_T$.
This estimator is a basic building block of \asynctuner, that {\em removes the need to manually compensate for the effects of asynchrony}.

\paragraph{Measuring the momentum dynamics}
\Asynctuner estimates total momentum $\mu_{T}$ on a running system and uses a negative feedback loop to adjust algorithmic momentum accordingly.
Equation~\eqref{equ:exp_async_update} gives an estimate of $\hat{\mu}_T$ on a system with staleness $\tau$, based on \eqref{equ:exp_async_update}.
\begin{align}
\hat{\mu}_T
					= \mathop{\mathsf{median}}\left(
							\frac{x_{t - \tau} - x_{t - \tau-1} + \alpha \nabla_{S_{t-\tau -1}} f(x_{t - \tau - 1} )}
							{x_{t - \tau-1} - x_{t - \tau-2}}
					\right)
\label{eqn:momentum_measurement}
\end{align}
We use $\tau$-stale model values to match the staleness of the gradient,  and perform all operations in an elementwise fashion.
This way we get a total momentum measurement from each variable;
the median combines them into a more robust estimate.

\paragraph{Closing the asynchrony loop}
Given a reliable measurement of $\mu_{T}$,
we can use it to adjust the value of algorithmic momentum so that the total momentum matches the \emph{target momentum} as decided by \tuner in Algorithm~\ref{alg:basic-algo}.
\Asynctuner in Algorithm~\ref{alg:async-algo} %(in Appendix~\ref{sec:async_yf})
uses a simple negative feedback loop to achieve the adjustment.
%Figure~\ref{fig:we-can-measure} demonstrates that under asynchrony the measured total momentum is strictly higher than the algorithmic momentum (middle plot), as expected from theory;
%closing the feedback loop (right plot) leads to total momentum matching the target momentum.
%Closing the loop, as we will see, improves performance significantly.
%Note for asynchronous-parallel training, as the estimates and parameter tuning is unstable in the beginning when there are only a small number of iterations, we use initial learning $\frac{1}{\tau + 1}$ instead of $1.0$ to prevent overflow in the beginning.

%\begin{algorithm}[H]
%	\caption{\Asynctuner}
%	\begin{algorithmic}[1]
%%	\State Input: $\mu\gets0$, $\alpha \gets \frac{1}{\tau + 1}$, $\gamma\gets0.01, \tau$ (staleness)
%	\State Input: $\mu\gets0$, $\alpha \gets 0.0001$, $\gamma\gets0.01, \tau$ (staleness)
%	\For { $t\gets1$ to $T$}
%	\State $x_t\!\gets\!x_{t - 1} + \mu (x_{t - 1} - x_{t - 2} ) - \alpha \nabla_{S_t} f(x_{t - \tau - 1} )$
%	\State $\mu^*,\alpha \gets \Call{\tuner}{\nabla_{S_t} f(x_{t - \tau - 1} ), \beta}$ %(get momentum from the dynamic range)
%	\State $\hat{\mu_T}
%					\gets \mathop{\mathsf{median}}\left(
%							\frac{x_{t - \tau} - x_{t - \tau-1} + \alpha \nabla_{S_{t-\tau-1}} f(x_{t - \tau - 1} )}
%							{x_{t - \tau-1} - x_{t - \tau-2}}
%					\right)$ \Comment{Measuring total momentum}
%	\State $\mu \leftarrow \mu + \gamma \cdot (\mu^* - \hat{\mu_T})$ \Comment{Closing the loop}
%	\EndFor
%\end{algorithmic}
%\label{alg:async-algo}
%\end{algorithm}

%In Section~\ref{sec:async_tuner}, we briefly discuss the mechanism of our designed \Asynctuner in asynchronous-parallel setting. In this appendix, we expand the details in total momentum estimator, $\hat{\mu_T}$, and present the full \Asynctuner in Algorithm~\ref{alg:async-algo} with extensive discussion.
%\paragraph{Measuring the momentum dynamics}
%Remember, we use the formula in~\eqref{equ:exp_async_update_app} to model the momentum dynamics in asynchronous-parallel systems
%\Asynctuner estimates total momentum $\mu_{T}$ on a running system and uses a negative feedback loop to adjust algorithmic momentum accordingly.
%\begin{equation}
%	\mathbb{E}[ x_{t+1} - x_t ]
%	= \mu_T \mathbb{E}[x_t - x_{t-1}] - \alpha \mathbb{E}\nabla f(x_{t})
%\label{equ:exp_async_update_app}
%\end{equation}
%Equation~\eqref{eqn:momentum_measurement_app} gives an estimate of $\hat{\mu_T}$ on a system with staleness $\tau$, based on \eqref{equ:exp_async_update_app}.
%\begin{align}
%\hat{\mu_T}
%					= \mathop{\mathsf{median}}\left(
%							\frac{x_{t - \tau} - x_{t - \tau-1} + \alpha \nabla_{S_{t-\tau -1}} f(x_{t - \tau - 1} )}
%							{x_{t - \tau-1} - x_{t - \tau-2}}
%					\right)
%\label{eqn:momentum_measurement_app}
%\end{align}
%We use $\tau$-stale model values to match the staleness of the gradient,  and perform all operations in an elementwise fashion.
%This way we get a total momentum measurement from each variable;
%the median combines them into a more robust estimate.
%
%%\label{subsec:closed_loop_YF}
%%\begin{figure}
%%\centering
%%\includegraphics[width=0.95\linewidth]{experiment_results/resnet/mom_dynamic_3_annotated.pdf}
%%	\caption{
%%	Momentum dynamics on CIFAR100 ResNet.
%%	Running \tuner, total momentum is equal to algorithmic momentum in a synchronous setting (left). Total momentum is greater than algorithmic momentum on 16 asynchronous workers, due to asynchrony-induced momentum (middle).
%%	Using the momentum feedback mechanism of \asynctuner, lowers algorithmic momentum and brings total momentum to match the target value on 16 asynchronous workers (right).
%%	Red dots are individual total momentum estimates, $\hat{\mu}_T$, at each iteration.
%%The solid red line is a running average of those estimates.
%%	}
%%	\label{fig:we-can-measure}
%%\end{figure}
%
%\paragraph{Closing the asynchrony loop}
%Given a reliable measurement of $\mu_{T}$,
%we can use it to adjust the value of algorithmic momentum so that the total momentum matches the \emph{target momentum} as decided by \tuner in Algorithm~\ref{alg:basic-algo}.
%\Asynctuner in Algorithm~\ref{alg:async-algo} %(in Appendix~\ref{sec:async_yf})
%uses a simple negative feedback loop to achieve the adjustment.
%%Figure~\ref{fig:we-can-measure} demonstrates that under asynchrony the measured total momentum is strictly higher than the algorithmic momentum (middle plot), as expected from theory;
%%closing the feedback loop (right plot) leads to total momentum matching the target momentum.
%%Closing the loop, as we will see, improves performance significantly.
%%%Note for asynchronous-parallel training, as the estimates and parameter tuning is unstable in the beginning when there are only a small number of iterations, we use initial learning $\frac{1}{\tau + 1}$ instead of $1.0$ to prevent overflow in the beginning.
%%
%%%\begin{algorithm}[H]
%%%	\caption{\Asynctuner}
%%%	\begin{algorithmic}[1]
%%%%	\State Input: $\mu\gets0$, $\alpha \gets \frac{1}{\tau + 1}$, $\gamma\gets0.01, \tau$ (staleness)
%%%	\State Input: $\mu\gets0$, $\alpha \gets 0.0001$, $\gamma\gets0.01, \tau$ (staleness)
%%%	\For { $t\gets1$ to $T$}
%%%	\State $x_t\!\gets\!x_{t - 1} + \mu (x_{t - 1} - x_{t - 2} ) - \alpha \nabla_{S_t} f(x_{t - \tau - 1} )$
%%%	\State $\mu^*,\alpha \gets \Call{\tuner}{\nabla_{S_t} f(x_{t - \tau - 1} ), \beta}$ %(get momentum from the dynamic range)
%%%	\State $\hat{\mu_T}
%%%					\gets \mathop{\mathsf{median}}\left(
%%%							\frac{x_{t - \tau} - x_{t - \tau-1} + \alpha \nabla_{S_{t-\tau-1}} f(x_{t - \tau - 1} )}
%%%							{x_{t - \tau-1} - x_{t - \tau-2}}
%%%					\right)$ \Comment{Measuring total momentum}
%%%	\State $\mu \leftarrow \mu + \gamma \cdot (\mu^* - \hat{\mu_T})$ \Comment{Closing the loop}
%%%	\EndFor
%%%\end{algorithmic}
%%%\label{alg:async-algo}
%%%\end{algorithm}
%%
%
%
%

\begin{algorithm}[h]
	\caption{\Asynctuner}
	\begin{algorithmic}[1]
%	\State Input: $\mu\gets0$, $\alpha \gets \frac{1}{\tau + 1}$, $\gamma\gets0.01, \tau$ (staleness)
	\State Input: $\mu\gets0$, $\alpha \gets 0.0001$, $\gamma\gets0.01, \tau$ (staleness)
	\For { $t\gets1$ to $T$}
	\State $x_t\!\gets\!x_{t - 1} + \mu (x_{t - 1} - x_{t - 2} ) - \alpha \nabla_{S_t} f(x_{t - \tau - 1} )$
	\State $\mu^*,\alpha \gets \Call{\tuner}{\nabla_{S_t} f(x_{t - \tau - 1} ), \beta}$ %(get momentum from the dynamic range)
	\State $\hat{\mu_T}
					\gets \mathop{\mathsf{median}}\left(
							\frac{x_{t - \tau} - x_{t - \tau-1} + \alpha \nabla_{S_{t-\tau-1}} f(x_{t - \tau - 1} )}
							{x_{t - \tau-1} - x_{t - \tau-2}}
					\right)$ \Comment{Measuring total momentum}
	\State $\mu \leftarrow \mu + \gamma \cdot (\mu^* - \hat{\mu_T})$ \Comment{Closing the loop}
	\EndFor
\end{algorithmic}
\label{alg:async-algo}
\end{algorithm}


\chapter{BUILDING A TIN WOODSMAN}

\begin{figure}
\centering \includegraphics{wm-prims.ps}
\caption{WoodsMan Primitives.}
\label{wm-prims}
\end{figure}

The purpose of this tutorial is to demonstrate how to build a model using
a few basic primitives.  The model to be constructed is a tin woodsman.
The four primitives used in the construction of the tin woodsman
are an ARB8, a cylinder, an ellipsoid, and a torus.
These four primitives will be duplicated several times,
and each copy will be modified using solid editing,
to obtain the required shapes.
The finished version of this database can be found in
the BRL-CAD Package file ``db/woodsman.g''.

\section{Create Primitives}

\noindent{\tt
\$ {\em mged woodsman.g} \\
BRL-CAD Release 3.0 Graphics Editor (MGED) Compilation 82 \\
    Thu Sep 22 08:08:39 EDT 1988 \\
    mikel@video.br:/cad/.mged.4d2 \\
 \\
woodsman.g: No such file or directory \\
Create new database (y|n)[n]? {\em y} \\
attach (nu|tek|tek4109|ps|plot|sgi)[nu]? {\em sgi} \\
ATTACHING sgi (SGI 4d) \\
Untitled MGED Database (units=mm) \\
mged> {\em size 20} \\
mged> {\em title A Tin Woodsman} \\
mged> {\em in solid8 rpp} \\
Enter XMIN, XMAX, YMIN, YMAX, ZMIN, ZMAX: {\em -1 1  -1 1  -1 1} \\
mged> {\em in torus tor} \\
Enter X, Y, Z of vertex: {\em 0 0 0} \\
Enter X, Y, Z of normal vector: {\em 0 1 0} \\
Enter radius 1: {\em 1} \\
Enter radius 2: {\em 0.2} \\
mged> {\em in ellipsoid ellg} \\
Enter X, Y, Z of vertex: {\em 0 0 0} \\
Enter X, Y, Z of vector A: {\em 1 0 0} \\
Enter X, Y, Z of vector B: {\em 0 0.5 0} \\
Enter X, Y, Z of vector C: {\em 0 0 0.5} \\
mged> {\em in cylinder rcc} \\
Enter X, Y, Z of vertex: {\em 0 0 0} \\
Enter X, Y, Z of height (H) vector: {\em 2 0 0} \\
Enter radius: {\em 1} \\
mged>
}

At this point, the screen should look like Figure \ref{wm-prims}.

\section{Copy Primitives and Set Up for Edit}

Although eight copies of the ellipsoid and two copies of the cylinder
shall be used in the final solid, fewer copies are made initially since there
is replication in the editing of these primitives.

\noindent{\tt
mged> {\em cp ellipsoid e.2} \\
mged> {\em cp ellipsoid e.6} \\
mged> {\em cp cylinder c.1} \\
mged> {\em cp solid8 s.1} \\
mged> {\em cp torus t.1} \\
mged> {\em Z} \\
mged> {\em e e.* c.1 t.1 s.1} \\
vectorized in 0 sec \\
mged>  \\
}

\begin{figure}
\centering \includegraphics{wm-hat1.ps}
\caption{Funnel Bowl Cylinder After Rotation.}
\label{wm-hat1}
\end{figure}

\begin{figure}
\centering \includegraphics{wm-hat2.ps}
\caption{Funnel Bowl Cylinder After End Scaling.}
\label{wm-hat2}
\end{figure}

\begin{figure}
\centering \includegraphics{wm-hat3.ps}
\caption{Funnel Bowl Cylinder After Moving.}
\label{wm-hat3}
\end{figure}

\begin{figure}
\centering \includegraphics{wm-tube.ps}
\caption{Funnel Tube Scaled and Positioned.}
\label{wm-tube}
\end{figure}

\section{Create Funnel Hat}

The solid ``cylinder'' has a height vector (``H'') which is 2mm
long.
This will be used to good advantage, to make the Tin Woodsman's
funnel hat, with the bowl of the funnel being 2mm high,
and the tube of the funnel being 2mm long.
The tube of the funnel will point straight up the +Y axis.

\noindent{\tt
mged> {\em sed c.1} \\
mged> {\em Select the ``Rotate'' entry in the solid edit menu} \\
mged> {\em p 0 45 90} \\
mged>
}

This places the cylinder so that the lines BD and AC are at the outer
ends of the cylinder.  See Figure \ref{wm-hat1}.

Next, the cylinder is shaped to look like the top of a funnel.
The vectors c and d are scaled.

\noindent{\tt
{\em Select the ``edit menu'' entry in the solid edit menu} \\
{\em Select the ``scale c'' entry in the TGC menu} \\
mged> {\em p .1} \\
{\em Select the ``scale d'' entry in the TGC menu} \\
mged> {\em p .1} \\
mged> \\
}

Figure \ref{wm-hat2} is the new shape of the cone.
Note how the on-screen display records the new lengths of the ``c''
and ``d'' vectors.
This cone must be moved to the planned locations for the top of the head.

\noindent{\tt
{\em Select the ``Translate'' entry in the solid edit menu} \\
mged> {\em p 0 2.2 0} \\
{\em Select the ``ACCEPT Edit'' entry in the button menu} \\
mged> \\
}

The bottom of the hat is now properly shaped and positioned.  The
new version of the solid ``c.1'' has been saved in the model database.
The editor returns to the viewing state.  See Figure \ref{wm-hat3}.

A copy of the saved ``c.1'' cone is made.  A byproduct of the {\em cp}
command is to display the new solid, as if the {\em cp c.1 c.2} command
had been immediately followed by an {\em e c.2} command.
This new solid will be edited to make the neck of the funnel,
which is the top of the hat.
The ``c.2'' copy of the cone must be
scaled down to become a tube and the tube must be placed on top of the
cone ``c.1''.

\noindent{\tt
mged> {\em cp c.1 c.2} \\
mged> {\em sed c.2} \\
{\em Select ``scale A,B'' in the TGC menu} \\
mged> {\em p 0.1} \\
{\em Select ``Translate'' in the Solid Edit menu} \\
mged> {\em p 0 4.2 0} \\
{\em Select ``ACCEPT Edit'' in the Button menu} \\
mged>
}

Figure \ref{wm-tube} is the new shape of the funnel tube.
The woodsman's hat is comprised of solids c.1 and c.2.

\begin{figure}
\centering \includegraphics{wm-head.ps}
\caption{Head Sphere.}
\label{wm-head}
\end{figure}

\section{Building the Head}

The head of our Tin Woodsman is perfectly spherical, and will be located
at coordinates (0, 2, 0).
While it would be possible to duplicate the ellipsoid solid created above,
and modify it to produce the desired sphere, since all the parameters
of the head sphere are known, it is more economical simply to use
the {\em in} command to construct it directly.
Figure \ref{wm-head} shows the results of this operation.

\noindent{\tt
mged> {\em in e.1} \\
Enter solid type: {\em sph} \\
Enter X, Y, Z of vertex: {\em 0 2 0} \\
Enter radius: {\em 1} \\
mged>
}

\begin{figure}
\centering \includegraphics{wm-collar.ps}
\caption{The Woodsman's Collar.}
\label{wm-collar}
\end{figure}

\section{Building the Collar}

The torus (primitive t.1) is used to build a collar between the head and
the body.
The ring of the collar is scaled to 0.1 of its original size,
and repositioned at the base of the head.
The results of this step are shown in Figure \ref{wm-collar}.

\noindent{\tt
mged> {\em sed t.1} \\
{\em Select ``scale radius 2'' in the TORUS menu} \\
mged> {\em p 0.1} \\
{\em Select ``Translate'' in the Solid Edit menu} \\
mged> {\em p 0 1 0} \\
{\em Select ``ACCEPT Edit'' in the Button menu} \\
mged>
}

\begin{figure}
\centering \includegraphics{wm-body.ps}
\caption{The Woodsman's Body.}
\label{wm-body}
\end{figure}

\section{Building the Body}

The ARB8 (primitive s.1) is used to build the body.
The original height of s.1 is only 2mm;  the required
length of the body is 3mm, so the extrusion command is
used to adjust the position of the lower (-Y) face of the solid.
The result is shown in Figure \ref{wm-body}.

\noindent{\tt
mged> {\em sed s.1} \\
mged> {\em extrude 2367 3} \\
{\em Select ``ACCEPT Edit'' in the Button menu} \\
mged>
}

\begin{figure}
\centering \includegraphics{wm-arm1.ps}
\caption{An Upper Arm Prototype.}
\label{wm-arm1}
\end{figure}

\begin{figure}
\centering \includegraphics{wm-arm2.ps}
\caption{The Woodsman's Arms.}
\label{wm-arm2}
\end{figure}

\section{Building the Arms}

The ellipsoid primitive (e.2) is used to build the upper and lower parts
of the left and right arms.
The original solid ``e.2'' is oriented with the major axis of the ellipse
oriented along the X axis.
The arms need to have the major axis oriented along the Y axis,
so first the solid is rotated.
A more graceful arm is obtained by decreasing the length of
the B and C vectors, and the resulting upper arm solid can be
seen in Figure \ref{wm-arm1}.

\noindent{\tt
mged> {\em sed e.2} \\
{\em Select ``Rotate'' in the Solid Edit menu} \\
mged> {\em p 0 45 90} \\
{\em Select ``edit menu'' in the Solid Edit menu} \\
{\em Select ``scale B'' in the ELLIPSOID menu} \\
mged> {\em p 0.25} \\
{\em Select ``scale C'' in the ELLIPSOID menu} \\
mged> {\em p 0.25} \\
mged>
}

This e.2 solid will now be moved into final position as the upper left arm.
Then it will be duplicated three times
to make the rest of the arm parts.  Finally, each new arm
part will be translated into the proper position,
as seen in Figure \ref{wm-arm2}.

\noindent{\tt
{\em Select ``Translate'' in the Solid Edit menu} \\
mged> {\em p -1.3 0 0} \\
{\em Select ``ACCEPT Edit'' in the Button menu -- This is the upper left arm} \\
mged> {\em cp e.2 e.3} \\
mged> {\em cp e.2 e.4} \\
mged> {\em cp e.2 e.5} \\
mged> {\em sed e.3} \\
{\em Select ``Translate'' in the Solid Edit menu} \\
mged> {\em p -1.3 -2 0} \\
{\em Select ``ACCEPT Edit'' in the Button menu -- This is the lower left arm} \\
mged> {\em sed e.4} \\
{\em Select ``Translate'' in the Solid Edit menu} \\
mged> {\em p 1.3 0 0} \\
{\em Select ``ACCEPT Edit'' in the Button menu -- This is the upper right arm} \\
mged> {\em sed e.5} \\
{\em Select ``Translate'' in the Solid Edit menu} \\
mged> {\em p 1.3 -2 0} \\
{\em Select ``ACCEPT Edit'' in the Button menu -- This is the lower right arm} \\
mged>
}

\begin{figure}
\centering \includegraphics{wm-leg1.ps}
\caption{The First Leg.}
\label{wm-leg1}
\end{figure}

\begin{figure}
\centering \includegraphics{wm-final1.ps}
\caption{The Tin Woodsman.}
\label{wm-final1}
\end{figure}

\section{Building the Legs}

The ellipsoid primitive (e.6) is used as a prototype
to build the upper and lower parts of both legs from.
The primitive e.6 is scaled, rotated, and translated
into position as the upper left leg, as seen in Figure \ref{wm-leg1}.
Then, copies are made and translated to the remaining positions,
just like the arms were.

\noindent{\tt
mged> {\em sed e.6} \\
{\em Select ``Rotate'' in the Solid Edit menu} \\
mged> {\em p 0 45 90} \\
{\em Select ``Translate'' in the Solid Edit menu} \\
mged> {\em p -0.5 -3 0} \\
{\em Select ``ACCEPT Edit'' in the Button menu -- This is the upper left leg} \\
mged> {\em cp e.6 e.7} \\
mged> {\em cp e.6 e.8} \\
mged> {\em cp e.6 e.9} \\
mged> {\em sed e.7} \\
{\em Select ``Translate'' in the Solid Edit menu} \\
mged> {\em p -0.5 -5 0} \\
{\em Select ``ACCEPT Edit'' in the Button menu -- This is the lower left leg} \\
mged> {\em sed e.8} \\
{\em Select ``Translate'' in the Solid Edit menu} \\
mged> {\em p 0.5 -3 0} \\
{\em Select ``ACCEPT Edit'' in the Button menu -- This is the upper right leg} \\
mged> {\em sed e.9} \\
{\em Select ``Translate'' in the Solid Edit menu} \\
mged> {\em p 0.5 -5 0} \\
{\em Select ``ACCEPT Edit'' in the Button menu -- This is the lower right leg} \\
}

Figure \ref{wm-final1} is the view on the screen, the Tin Woodsman.
Take a moment to use the rotation knobs to view the model from various
angles.

\section{Building Regions}

So far, this example has concentrated on describing the basic shapes
involved in making the Tin Woodsman, without concern for establishing
a proper hierarchical structure.  To illustrate this point,
the various solids will be grouped by purpose and composition.
First, a region will be constructed to contain the torso,
and the color of ``cadet blue'' will be assigned:

\noindent{\tt
mged> {\em r torso.r u s.1} \\
Defaulting item number to 1001 \\
Creating region id=1000, air=0, los=100, GIFTmaterial=1 \\
mged> {\em mater torso.r} \\
Material = \\
Material?  (CR to skip) {\em plastic} \\
Param = \\
Parameter string? (CR to skip) {\em [RETURN]} \\
Color = (No color specified) \\
Color R G B (0..255)? (CR to skip) {\em 95 159 159} \\
Inherit = 0:  lower nodes (towards leaves) override \\
Inheritance (0|1)? (CR to skip) {\em [RETURN]} \\
mged>
}

Second, a region will be constructed to contain the collar,
which will be colored red:

\noindent{\tt
mged> {\em r collar.r u t.1} \\
Defaulting item number to 1003 \\
Creating region id=1002, air=0, los=100, GIFTmaterial=1 \\
mged> {\em mater collar.r} \\
Material = \\
Material?  (CR to skip) {\em plastic} \\
Param = \\
Parameter string? (CR to skip)  {\em [RETURN]} \\
Color = (No color specified) \\
Color R G B (0..255)? (CR to skip) {\em 255 127 0} \\
Inherit = 0:  lower nodes (towards leaves) override \\
Inheritance (0|1)? (CR to skip)  {\em [RETURN]} \\
mged>
}

Third, a region will be constructed to contain all the limbs,
and a flesh color will be assigned.
Even though none of the limbs touch each other, note how they
are combined with the UNION operation, to create a single
object of uniform composition and color.

\noindent{\tt
mged> {\em r limbs.r u e.2 u e.3 u e.4 u e.5 u e.6 u e.7 u e.8 u e.9} \\
Defaulting item number to 1001 \\
Creating region id=1000, air=0, los=100, GIFTmaterial=1 \\
mged> {\em mater limbs.r} \\
Material = \\
Material?  (CR to skip) {\em plastic} \\
Param = \\
Parameter string? (CR to skip)  {\em [RETURN]} \\
Color = 0 0 0 \\
Color R G B (0..255)? (CR to skip) {\em 255 200 160} \\
Inherit = 0:  lower nodes (towards leaves) override \\
Inheritance (0|1)? (CR to skip)  {\em [RETURN]} \\
mged>
}

Next, the funnel needs to be placed in a region.
For the sake of simplicity, the funnel will be solid, rather
than having a hollow center.
Note that the interior of the funnel overlaps with the top
of the Woodsman's head.
The funnel can be made ``form fitting'' by subtracting out
the overlap zone:

\noindent{\tt
mged> {\em r funnel.r u c.1 - e.1 u c.2 - e.1} \\
Defaulting item number to 1004 \\
Creating region id=1003, air=0, los=100, GIFTmaterial=1 \\
mged> {\em mater funnel.r} \\
Material = \\
Material?  (CR to skip) {\em plastic} \\
Param = \\
Parameter string? (CR to skip) {\em sh=100} \\
Color = (No color specified) \\
Color R G B (0..255)? (CR to skip) {\em 35 107 142} \\
Inherit = 0:  lower nodes (towards leaves) override \\
Inheritance (0|1)? (CR to skip)
mged> {\em l funnel.r} \\
funnel.r (len 4) REGION id=1003  (air=0, los=100, GIFTmater=1) -- \\
Material 'plastic' 'sh=100' \\
Color 35 107 142 \\
\ \ u c.1 \\
\ \ - e.1 \\
\ \ u c.2 \\
\ \ - e.1 \\
mged>
}

\begin{figure}
\centering \includegraphics{wm-hat-E.ps}
\caption{Evaluation of Funnel Hat Region.}
\label{wm-hat-E}
\end{figure}

Note how the boolean expression was written.
The concept that we need to express here is
the combination of all the funnel parts, minus the
portion of the head that overlaps with the inside of the funnel.
The natural way to write this is
\begin{center}
(c.1 union c.2) - e.1
\end{center}
but note that there are no grouping operations permitted in the {\em r}
command.
Furthermore, for historic reasons, union operations bind more loosely than
intersection and subtraction, i.e., there are implied groups
between union operations.  Thus, the expression above needs to be
rewritten as the formula:
\begin{center}
(c.1 - e.1) union (c.2 - e.1)
\end{center}
which with the binding precedence can be expressed as:
\begin{center}
c.1 - e.1 union c.2 - e.1
\end{center}
which is what was entered in the sequence above.
To see the effect that this command had on the shape of ``funnel.r'',
run these commands, the effect of which is shown in Figure \ref{wm-hat-E}:

\noindent{\tt
mged> {\em Z } \\
mged> {\em E funnel.r } \\
vectorized in 1 sec \\
mged>
}

These regions should be grouped together into a group,
for convenience in referencing.  This can be done with these commands:

\noindent{\tt
mged> {\em g man.g collar.r funnel.r limbs.r torso.r} \\
mged> {\em Z} \\
mged> {\em e man.g} \\
vectorized in 1 sec \\
mged>
}

The grouping {\em g} command combined the regions, the Zap command {\em Z}
cleared the screen, and the edit {\em e man.g} command drew the whole
object.
As an exercise, run
the database structure printing command {\em tree man.g}
to obtain a simple depiction of the tree structure that has been created.
For the final step of this example, the model will be ray-traced.
Run the command:

\noindent{\tt
mged> {\em rt -s128} \\
rt -s50 -M -s128 woodsman.g man.g  \\
db title:  A Tin Woodsman \\
Buffering single scanlines \\
initial dynamic memory use=35152.\\
Interpreting command stream in old format\\
GETTREE: 0.01 CPU secs in 1 elapsed secs (1\%)\\
                                              \\
...................Frame     0...................\\
PREP: 0.01 CPU secs in 0.01 elapsed secs (100\%)\\
shooting at 13 solids in 4 regions              \\
model X(-2,2), Y(-6,7), Z(-2,2)\\
Beam radius=0.078125 mm, divergence=0 mm/1mm\\
                                             \\
SHOT: 3.73 CPU secs in 6 elapsed secs (62.1667\%)\\
Additional dynamic memory used=29728. bytes\\
3515 solid/ray intersections: 1005 hits + 2510 miss\\
pruned 28.6\%:  13647 model RPP, 8197 dups, 10740 RPP\\
Frame     0:    16384 pixels in       3.73 sec =    4392.49 pixels/sec\\
Frame     0:    16384 rays   in       3.73 sec =    4392.49 rays/sec (RTFM)\\
\\
Press RETURN to reattach\\
{\em [RETURN]} \\
mged>
}


%% LyX 2.0.3 created this file.  For more info, see http://www.lyx.org/.
%% Do not edit unless you really know what you are doing.
\documentclass[twoside,english]{paper}
\usepackage{lmodern}
\renewcommand{\ttdefault}{lmodern}
\usepackage[T1]{fontenc}
\usepackage[latin9]{inputenc}
\usepackage[a4paper]{geometry}
\geometry{verbose,tmargin=3cm,bmargin=2.5cm,lmargin=2cm,rmargin=2cm}
\usepackage{color}
\usepackage{babel}
\usepackage{float}
\usepackage{bm}
\usepackage{amsthm}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{graphicx}
\usepackage{esint}
\usepackage[unicode=true,pdfusetitle,
 bookmarks=true,bookmarksnumbered=false,bookmarksopen=false,
 breaklinks=false,pdfborder={0 0 0},backref=false,colorlinks=false]
 {hyperref}
\usepackage{breakurl}
\usepackage{mathrsfs}

\makeatletter

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% LyX specific LaTeX commands.
%% Because html converters don't know tabularnewline
\providecommand{\tabularnewline}{\\}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Textclass specific LaTeX commands.
\numberwithin{equation}{section}
\numberwithin{figure}{section}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% User specified LaTeX commands.
\usepackage{babel}

\@ifundefined{showcaptionsetup}{}{%
 \PassOptionsToPackage{caption=false}{subfig}}
\usepackage{subfig}
\makeatother

\begin{document}

\title{Computation of the charged current structure functions}

\maketitle

\section{The structure of the observables}

The structure in terms of PDFs of the charged current (CC) structure
functions is complicated by the mixing between down- and up-type
quarks provided by the CKM matrix. As a first step, we write the
$\mathcal{O}(\alpha_s)$ contribution to $F=F_2,F_L$ (we will consider
$F_3$ later) in a convenient way as:
\begin{equation}\label{compactNu}
F^{\nu} = \sum_{U=u,c,t}\sum_{D=d,s,b}|V_{UD}|^2\left[C_\pm\left(D +\overline{U}\right) +2 C_gg\right]
\end{equation}
and:
\begin{equation}\label{compactNub}
F^{\overline{\nu}} = \sum_{U=u,c,t}\sum_{D=d,s,b}|V_{UD}|^2\left[C_\pm\left(\overline{D} +U\right) +2 C_gg\right]
\end{equation}
where we have omitted the convolution symbol and an overall factor
$2x$. At this order we don't have to worry about whether $C_+$ or
$C_-$ has to be used because they coincide. However, in the following
it will appear naturally which one has be used and where. One can
combine the expressions above conveniently as:
\begin{equation}
F^{\pm} \equiv \frac{F^{\nu} \pm F^{\overline{\nu}}}2=\frac12
\sum_{U=u,c,t}\sum_{D=d,s,b}|V_{UD}|^2\left[C_\pm\left(D^\pm
    \pm U^\pm\right) + P^{\pm}4C_gg\right]
\end{equation}
where we have used the usual definition $q^{\pm} = q\pm\overline{q}$
and defined the projector:
\begin{equation}
P^\pm=\frac{1\pm1}2\,.
\end{equation}
It should be noted that the subscript $\pm$ to the quark coefficient
function $C_\pm$ because is now associated to each of $F^\pm$.

Now we need to express these observables in terms of PDFs in the
evolution basis. The starting point is the relation:
\begin{equation}\label{TranformationBella}
q_i^\pm = \sum_{j=1}^6M_{ij}d^\pm_j\,,
\end{equation}
where $d^\pm_j$ belong to the QCD evolution basis, that is:
$d^+_1=\Sigma$, $d^+_2=-T_3$, $d^+_3=T_8$, $d^+_4=T_{15}$, $d^+_5=T_{24}$, and
$d^+_6=T_{35}$ and $d^-_1=V$, $d^-_2=-V_3$, $d^-_3=V_8$, $d^-_4=V_{15}$,
$d^-_5=V_{24}$, and $d^-_6=V_{35}$.  Note that here we are using the more
``natural'' ordering for the distibutions $q_i=\{d,u,s,c,b,t\}$ rather
than that where $u$ comes before $d$; this is the reason of the minus
sign in front of $T_3$ and $V_3$. The trasformation matrix $M_{ij}$
can be written as:
\begin{equation}\label{TransDef}
\begin{array}{l}
\displaystyle M_{ij}=\theta_{ji}\frac{1-\delta_{ij}j}{j(j-1)}\quad j\geq 2\,,\\
\\
\displaystyle M_{i1} = \frac{1}{6}\,,
\end{array}
\end{equation}
with $\theta_{ji}=1$ for $j\geq i$ and zero otherwise. In addition,
one can show that $M_{ij}$ is such that:
\begin{equation}\label{eq:properties}
\sum_{j=1}^6M_{ij} = 0\,,\quad\mbox{and}\quad \sum_{i=1}^6M_{ij} = \delta_{1j}\,.
\end{equation}

Using eq.~(\ref{TranformationBella}) we can make the following
identifications:
\begin{equation}
D^{\pm} = q_{2j-1}^\pm\quad\mbox{and}\quad U^{\pm} =
q_{2j}^\pm\,,\quad j=1,2,3\,,
\end{equation}
so that we can write:
\begin{equation}
F^\pm=
\frac12\sum_{i=1}^3\sum_{j=1}^3|V_{2i,(2j-1)}|^2\left[C_\pm\left(q_{2j-1}^\pm
    \pm q_{2i}^\pm\right) + 4P^{\pm} C_g g\right]\,.
\end{equation}
Using the definition of $M_{ij}$ in eq.~(\ref{TransDef}), we can
rewrite $F^{\pm}$ in terms of PDFs in the evolution basis as:
\begin{equation}\label{eq:decompF2L}
F^\pm=
\sum_{i=1}^3\sum_{j=1}^3|V_{2i,(2j-1)}|^2 F_{ij}^\pm\,,
\end{equation}
with:
\begin{equation}\label{F2Ldef}
F_{ij}^\pm=
C_g 2P^\pm g
+
C_\pm^{\rm S} P^\pm \frac16 d_1^\pm
+ C_\pm\sum_{k=2}^6\frac{\theta_{k,2j-1}(1-\delta_{2j-1,k}k)\pm \theta_{k,2i}(1-\delta_{2i,k}k) }{2k(k-1)}d_k^\pm\,.
\end{equation}

Eq.~(\ref{F2Ldef}) is valid only for $F_2$ and $F_3$. In order to
obtain a similar equation also for $F_3$, one needs to change sign to
the antiquark distributions, $i.e.$
$\overline{q}_i\rightarrow - \overline{q}_i$. In the QCD evolution
basis, this has the consequence of exchanging the $T$-like
distributions with the $V$-like ones, that is to say
$d_k^+\leftrightarrow d_k^-$. It is the easy to see that:
\begin{equation}
F_3^\pm=
\sum_{i=1}^3\sum_{j=1}^3|V_{2i,(2j-1)}|^2 F_{3,ij}^\pm\,,
\end{equation}
with:
\begin{equation}\label{F3def}
F_{3,ij}^\pm=
C_g 2P^\pm g
+
C_\pm^{\rm S} P^\mp \frac16 d_1^\pm
+ C_\pm\sum_{k=2}^6\frac{\theta_{k,2j-1}(1-\delta_{2j-1,k}k)\mp \theta_{k,2i}(1-\delta_{2i,k}k) }{2k(k-1)}d_k^\pm\,.
\end{equation}

It is now useful to consider the inclusive structure functions and
exploit the unitarity of the CKM matrix elements $V_{UD}$:
\begin{equation}
\sum_{i=1}^3|V_{2i,(2j-1)}|^2 = \sum_{j=1}^3|V_{2i,(2j-1)}|^2 = 1\quad\Rightarrow\quad \sum_{i=1}^3\sum_{j=1}^3|V_{2i,(2j-1)}|^2 = 3\,.
\end{equation}
Summing over $i$ and $j$ in eq.~(\ref{eq:decompF2L}) and using
eq.~(\ref{F2Ldef}), one obtains:
\begin{equation}
F^\pm=
C_g 6P^\pm g
+
C_\pm^{\rm S} P^\pm \frac12 d_1^\pm
+ \frac12 C_\pm\sum_{k=2}^6 d_k^\pm\sum_{l=1}^6(\pm 1)^{l+1}M_{lk}\,.
\end{equation}
Considering separately $F^+$ and $F^-$ and using
eq.~(\ref{eq:properties}), one finds:
\begin{equation}
F^+= C_g 6g + C_+^{\rm S} \frac12 d_1^+
\end{equation}
and:
\begin{equation}
F^-= \frac12 C_-\sum_{k=2}^6\left[\frac{P^+}{k-1}-\frac{P^-}{k}\right]d_k^-\,,
\end{equation}
with the even/odd projectors defined as:
\begin{equation}
P_k^{\pm} = \frac{1\pm(-1)^k}{2}\,.
\end{equation}

It should be pointed out that such simple expressions (independent of
the CMK matrix elements) is achievable only if it is possible to
factorize the non-singlet coefficient functions as implicitly done in
eqs.~(\ref{F2Ldef}) and~(\ref{F3def}). In fact, this is possible only
in the ZM case in which the coefficient functions of each PDF
combination is the same.

For $F_3$ we find:
\begin{equation}
F_3^+= C_g 6g + C_-^{\rm S} \frac12 d_1^-
\end{equation}
and:
\begin{equation}
F_3^-= \frac12 C_+\sum_{k=2}^6\left[\frac{P_k^+}{k-1}-\frac{P_k^-}{k}\right]d_k^+\,.
\end{equation}

\end{document}


\chapter{\label{chap:deep-anns}Deep neural networks}
\xdef\xvec{10mm}
\xdef\yvec{15mm}

\emph{Deep learning} refers to a set of machine learning methods
that have recently been (re)popularized.
One of the important aspects of the deep learning is
the use of deeper neural networks with more than one hidden layers.
They have been successfully applied to many machine learning methods,
and they are also the dominant approach used in
the natural language processing.
Deep ANNs are not just fully-connected feed-forward networks
with multiple hidden layers as the one presented
in Figure~\ref{fig:deep-ff-network}.
The typical architectures used in the field involve
\emph{sparse} connectivity and \emph{weight sharing}.
This lecture will introduce two common architectures,
\emph{recurrent networks} (RNNs) and \emph{convolutional networks} (CNNs).
\begin{marginfigure}
  \centering
%      \tikzset{external/export next=false}
      \tikzsetnextfilename{deep-network}
			\begin{tikzpicture}[shorten >=1pt,->,x=10mm, y=13mm,
													blue!40!black]
				\tikzset{neuron/.style={draw,%
																circle,%
																fill=black!20,%
																inner sep=0,%
                                minimum size=8}
        };

        \foreach \x in {1, ..., 4} {
          \node (n-0-\x) at (\x, 0) {};
					\foreach \y in {1, ..., 4} {
						\node[neuron] (n-\y-\x) at (\x, \y) {};
					}
        }
        \foreach \x in {1, ..., 4} {
					\foreach \y in {1, ..., 4} {
            \pgfmathparse{int(\y-1)}
            \xdef\yy{\pgfmathresult};
            \foreach \xx in {1, ..., 4} {
              \draw (n-\yy-\xx) -- (n-\y-\x);
            }
					}
        }

        \node[yshift=-2mm] at (n-0-1) {$x_{1}$};
        \node[yshift=-2mm] at (n-0-4) {$x_{m}$};
        \node at (2.5, 0) {\ldots};
        \node[neuron] (o-1) at (2, 5) {};
        \node[neuron] (o-2) at (3, 5) {};

        \foreach \x in {1, ..., 4} {
					\foreach \y in {1, ..., 2} {
              \draw (n-4-\x) -- (o-\y);
          }
        }
      \end{tikzpicture}
  \caption{A deep feed-forward (fully-connected) network.}
  \label{fig:deep-ff-network}
\end{marginfigure}

Earlier, we noted that an ANN with a single hidden layer is
a universal function approximator.
That is, it can approximate any computable function
with arbitrary precision.
Then, a natural question to ask is `why should one use more than one layer?'
The first reason is related to the proof that ANNs with a single hidden layer
are universal approximator.
The proof is very general,
and there is no way to tell how many units one needs in
the single hidden layer.
The second reason is to do with the fact that
certain problems seem to suit well to ANN architectures
with multiple layer.
These involve problems where layers, or hierarchies of features are useful.
A common example from image processing is,
for example,
recognizing objects that are composed of simple shapes,
which are combination of even simpler lines or curves (e.g., edges in the image)
which in turn are combinations of smaller parts,
and so on.
However,
the depth does not have to be only be for a hierarchy of features.
As we will see soon,
it one can also represent time as depth in a deep network.

Although there has been many interesting recent developments,
many of the ideas are developed during 1980's and 1990's.
The most important reason for the present success and the renewed interest
is probably the developments in computing hardware.
Particularly, availability of vector processors,
such as graphical processing units (GPUs) in personal computers,
that perform linear algebra operations efficiently
made training large neural networks feasible.
The increased availability of labeled and unlabeled data is another reason.
At present,
the deep networks are the default or dominant method in many fields,
including NLP.
In this lecture, we will introduce two architectures that are commonly used
in NLP, namely RNNs and CNNs,
and discuss some of the common practices and issues that arise
while training deep networks.

\section{Recurrent neural networks}

\begin{marginfigure}
  \centering
%  \tikzset{external/export next=false}
  \tikzsetnextfilename{rnn-example}
    \begin{tikzpicture}[blue!50!black,x=\xvec,y=\yvec]
      \tikzset{neuron/.style={draw,%
                              circle,%
                              inner sep=1pt,%
                              minimum size=6mm,%
                              fill=black!20%
        }
      };
      \foreach \x in {1, ..., 4} {
        \node (i-\x) at (\x, 0) {$x_{\x}$};
        \node[neuron] (h-\x) at (\x, 1) {$h_{\x}$};
      }
      \node[neuron] (o) at (2.5, 2) {$y$};
      \foreach \x in {1, ..., 4} {
        \foreach \y in {1, ..., 4} {
          \draw[->] (i-\x) -- (h-\y);
        }
        \draw[->] (h-\x) -- (o);
      }
      \draw[very thick] ([xshift=-2mm,yshift=-2mm]h-1.south west)
        rectangle ([xshift=2mm,yshift=2mm]h-4.north east);
      \draw[very thick,->,>=stealth]
        ([yshift=1mm,xshift=-2mm]h-1.north) arc (45:335:0.5);
    \end{tikzpicture}
  \caption{A schematic representation of a recurrent network.
    The thick recurrent link on the hidden layer indicates
    connections from each hidden unit to every hidden unit
    (including itself).
  }
  \label{fig:rnn-example}
\end{marginfigure}
Recurrent neural networks (RNNs) are sequence learning models.
Unlike feed-forward networks which only has a forward flow of information
during prediction,
RNNs include (time-delayed) loops.
Figure~\ref{fig:rnn-example} presents a typical RNN.
Without the thick recurrent link presented, the RNN is simply
a feed forward network.
What makes RNNs special is the backwards loop over the hidden units.
This makes an RNN to use the information in the previous hidden states
as well as the current input.
Hence, although the RNNs process a single input item (e.g., word)
at a time, they have a memory,
they may make use of the information from the past observations
(e.g., earlier words).

\begin{marginfigure}
  \centering
%      \tikzset{external/export next=false}
      \tikzsetnextfilename{rnn-elman-network}
      \begin{tikzpicture}[blue!50!black,x=8mm]
        \tikzset{layer/.style={draw,%
                               inner sep=1pt,%
                               minimum width=20mm,%
                               minimum height=7mm%
          }
        };

        \node[layer,draw=none] (inp) at (0,0) {Input};
        \node[layer,fill=black!20,left=8mm of inp] (c) {Context units};
        \node[layer,fill=black!20,above=1cm of inp] (h) {Hidden units};
        \node[layer,fill=black!20,above=1cm of h] (o) {Output units};
        \draw[very thick,->,shorten >=1pt,>=stealth] (inp) -- (h);
        \draw[very thick,->,shorten >=1pt,>=stealth] (h) -- (o);
        \draw[very thick,->,shorten >=1pt,>=stealth] (c) -- (h);
        \draw[red,yshift=1mm,inner sep=0pt,minimum width=9mm,minimum height=5mm,very thick,->,>=stealth] (h.west) arc (90:158:2.2)
          node[above,sloped,midway,black] {copy};
      \end{tikzpicture}
  \caption{Another schematic representation of a recurrent network,
    which used describing simple recurrent networks
    (SRNs, also known as Elman networks).
    The link with label `copy' does not have any associated weights.
  }
  \label{fig:srn}
\end{marginfigure}
Another way to look at a recurrent network,
often used for introducing \emph{simple recurrent networks}
(SRNs) is to assume that we have a set of `context' units
which are the copies of the hidden units from the past time step.
This is shown in Figure~\ref{fig:srn},
where the special link labeled `copy' does not have any learned weights,
but the other links,
including the one from the context unit to the hidden units,
have weights that are learned.
As a result,
the hidden units can combine the information
from the past hidden representation and the current input.
This representation should make it clear the forward operation of
an RNN.

In an SRN, like the one presented in Figure~\ref{fig:srn},
it is also possible to apply the standard backpropagation (BP) algorithm,
since the weights that are learned are feed-forward.
However, applying standard BP
means that error is not backpropagated more than one time step.
In modern recurrent networks,
a modified version of the BP algorithm,
often called \emph{backpropagation through time} (BPTT), is used.
To understand the BPTT,
it is useful to unfold, or unroll the network.
An unrolled recurrent network is presented in Figure~\ref{fig:rnn-unrolled}.
\begin{figure}
  \centering
%      \tikzset{external/export next=false}
    \tikzsetnextfilename{rnn-unrolled}
    \begin{tikzpicture}
      \tikzset{layer/.style={draw,%
                             inner sep=1pt,%
                             minimum width=12mm,%
                             minimum height=8mm,%
                             fill=black!20%
        }
      };
      \tikzset{ilayer/.style={layer,fill=white,rounded corners}
      };

      \node[] (x0) {$\vect{x}^{(0)}$};
      \node[right=of x0] (x1) {$\vect{x}^{(1)}$};
      \node[right=of x1] (x2) {\dots};
      \node[right=of x2] (x3) {$\vect{x}^{(t-1)}$};
      \node[right=of x3] (x4) {$\vect{x}^{(t)}$};
      \node[layer,above of=x0,yshift=5mm] (h0) {$h^{(0)}$};
      \node[layer,above of=x1,yshift=5mm] (h1) {$h^{(1)}$};
      \node[above of=x2,yshift=5mm] (h2) {\ldots};
      \node[layer,above of=x3,yshift=5mm] (h3) {$h^{(t-1)}$};
      \node[layer,above of=x4,yshift=5mm] (h4) {$h^{(t)}$};
      \node[layer,above=10mm of h0] (y0) {$y^{(0)}$};
      \node[layer,above=10mm of h1] (y1) {$y^{(1)}$};
      \node[above=10mm of h2] (y2) {\ldots};
      \node[layer,above=10mm of h3] (y3) {$y^{(t-1)}$};
      \node[layer,above=10mm of h4] (y4) {$y^{(t)}$};
      \foreach \x in {0, ..., 4}{
        \ifthenelse{\x = 2}{}{%
          \draw[very thick,->,shorten >=1pt,>=stealth,orange]
            (x\x) -- (h\x);
          \draw[very thick,->,shorten >=1pt,>=stealth,blue]
            (h\x) -- (y\x);
        }
        \ifthenelse{\x = 0}{}{%
          \pgfmathparse{int(int(\x)-1)};
          \xdef\prevx{\pgfmathresult};
          \draw[very thick,->,shorten >=1pt,>=stealth]
              (h\prevx) -- (h\x);
        }
      }
    \end{tikzpicture}
  \caption{An unrolled RNN.
    The superscripts indicate the time steps.
    Note that the weights represented with the links with the same color
    are \emph{shared}.
  }
  \label{fig:rnn-unrolled}
\end{figure}

The representation in in Figure~\ref{fig:rnn-unrolled} is describes the
same type of network described in Figure~\ref{fig:rnn-example}.
The difference is, in Figure~\ref{fig:rnn-unrolled},
we represent each time step separately.
This representation also turns the network into a deep feed-forward network.
And application of the BP algorithm is also straightforward.
The error made at any time step is reflected the input and
hidden layer representations before this time step.
It is also important to realize in Figure~\ref{fig:rnn-unrolled} is that,
although the network is deep, most parameters are shared.

Although BPTT gives us a way to apply BP to recurrent networks,
the (time)depth in RNNs lead to an problem called \emph{unstable gradients}.
To appreciate the problem,
remember that updates applied to weights in each layer is
calculated using the chain rule of derivatives.
As a result,
the update applied to the weights in earlier stages of a deep network
will be composed of a large number long chain of (matrix) multiplications.
Multiplying a series of (positive) numbers less than \num{1}
will cause the error signal to be very small,
slowing down learning, maybe to the extent that nothing is learned.
Similarly, multiplying a series of (positive) numbers greater than \num{1}
will cause the error signal to be too large,
causing instabilities due to large weight updates.
The former case is called \emph{vanishing gradients},
and the latter is called \emph{exploding gradients} in the literature.

To solve the exploding gradients,
often a simple technique called \emph{gradient clipping} is used.
Gradient clipping simply means truncating gradients
larger than a particular value to a fixed threshold.
The solution of the vanishing gradients is more involved.

\subsection{Gated recurrent networks}
To solve the vanishing gradient problem,
and allow an RNN to learn through a longer time distance,
a type of RNNs that are called \emph{gated RNNs} are used.
We will not go into the details of the gated recurrent networks in this class.
However, we briefly mention two variants that are popular in the field.

\begin{marginfigure}
  \centering
%    \tikzset{external/export next=false}
    \tikzsetnextfilename{lstm-cell}
    \begin{tikzpicture}[y=6mm,x=6mm,thick,font=\scriptsize]
      \draw[rounded corners] (0,0) rectangle (6,5);
      \node[draw,inner sep=2pt] (n1) at (1,2) {$\sigma_{f}$};
      \node[draw,inner sep=2pt] (n2) at (2,2) {$\sigma_{i}$};
      \node[font=\tiny,fill=gray!30,draw,inner sep=2pt] (n3) at (3,2) {tanh};
      \node[draw,inner sep=2pt] (n4) at (4,2) {$\sigma_{o}$};
      \node[draw,circle,inner sep=0pt] (f) at (3,3) {$\times$};
      \node[draw,circle,inner sep=0pt] (o) at (5,2) {$\times$};
      \node[draw,circle,inner sep=0pt] (c1) at (1,4) {$\times$};
      \node[draw,circle,inner sep=0pt] (c2) at (3,4) {$+$};
      \node[font=\tiny,draw,ellipse,minimum width=2.5em,inner sep=1pt] (co) at (5,3) {tanh};
      \draw[->] (-0.2,4) -- (c1);
      \draw[->] (c1)   -- (c2);
      \draw[->] (c2)   -- (6.2,4);

      \draw[->,rounded corners] (-0.2,1) -- (4,1) -- (n4.south);
      \draw[->] (1,-0.2) -- (n1);
      \draw[->] (2,1) -- (n2);
      \draw[->] (3,1) -- (n3);

      \draw[->] (n1) -- (c1);
      \draw[->,rounded corners] (n2) -- (2,3) -- (f);
      \draw[->] (n3) -- (f);
      \draw[->] (f) -- (c2);

      \draw[->] (n4) -- (o);
      \draw[->] (5,4) -- (co);
      \draw[->] (co) -- (o);
      \draw[->,rounded corners] (o) -- (5,1) -- (6.2, 1);

      \node[anchor=east] at (-0.2,4) {$c^\text{(t-1)}$};
      \node[anchor=east] at (-0.2,1) {$h^\text{(t-1)}$};
      \node[anchor=west] at (6.2,4) {$c^\text{(t)}$};
      \node[anchor=west] at (6.2,1) {$h^\text{(t)}$};

      \node[anchor=north] at (1,-0.2) {$x^{(t)}$};
    \end{tikzpicture}
  \caption{A schematic representation an LSTM cell.
    The drawing similar to the ones from
    a \href{https://colah.github.io/posts/2015-08-Understanding-LSTMs/}{blog post by Chris Olah}.
  }
  \label{fig:lstm}
\end{marginfigure}
The \emph{long-short-term memory} (LSTM) cell,
which is presented in Figure~\ref{fig:lstm},
controls the information kept, added or removed in the hidden representation
through a number of `gates'.
The LSTM keeps two vectors of hidden representations,
the one called the `hidden state' (`\vect{h}' in the figure)
and the other one is called the `cell state' (`\vect{c}' in the figure).
The idea is the that the cell state is the keeps the memory,
while a combination of the cell state, previous hidden state,
and the current input is passed to the output layer.
Both the cell state and the hidden layer is also passed to the next
time step after a number of operations.
The unshaded square blocks in the figure are called gates.
They are simply ANN layers with sigmoid activation function.
The circles (and the ellipse) in the figure represent
element-wise vector operations.
The forget gate ($\sigma_{f}$)
controls what is removed (or kept) from the cell state
and they control what is removed, and added to cell state
based on the previous hidden state and the current input.
The input gate ($\sigma_{i}$) controls what is added to the cell state.
And the output gate ($\sigma_{o}$) controls the hidden unit output.

The LSTM and its variants has been used successfully
in many sequence learning tasks.
A somewhat simpler variant,
called simply \emph{gated recurrent unit} (GRU),
has also become quite popular,
and likely to be found in many standard neural network tools and libraries.
The gated RNNs are complex models,
the success of one variant or the other differs in different applications.
However, in for most uses, gated RNNs yield better results than simple RNNs.

\subsection{Different uses of RNNs}

RNNs have been used in a number of different linguistic problems.
The architecture is flexible, and can be extended in many ways.
Here, we briefly go through some of the common variations.

A very common practice is to use \emph{bidirectional} RNNs.%
A bidirectional RNN is composed of two RNNs,
one run forward as we discussed above,
and another one run backwards through the sequence.
The hidden representations from both RNNs are then combined
and fed to the later layers in the network architecture.
Unless the application requires online sequential processing,
bidirectional networks are possible,
and often perform better than unidirectional variants.
A bidirectional RNN is shown in Figure~\ref{fig:bidirectional-rnn}.
\begin{marginfigure}
  \centering
%    \tikzset{external/export next=false}
    \tikzsetnextfilename{rnn-bidirectional}
    \begin{tikzpicture}[x=5mm]
      \tikzset{layer/.style={draw,%
                             inner sep=1pt,%
                             minimum width=6mm,%
                             minimum height=4mm,%
                             fill=black!20%
        }
      };
      \tikzset{ilayer/.style={layer,fill=white,rounded corners}
      };

      \node[font=\scriptsize] (x0) {$\vect{x}^\text{(t-1)}$};
      \node[font=\scriptsize,right=6mm of x0] (x1) {$\vect{x}^\text{(t)}$};
      \node[font=\scriptsize,right=6mm of x1] (x2) {$\vect{x}^\text{(t+1)}$};

      \node[layer,above of=x0,yshift=5mm] (h0) {};
      \node[layer,above of=x1,yshift=5mm] (h1) {};
      \node[layer,above of=x2,yshift=5mm] (h2) {};

      \node[layer,above of=h0,yshift=5mm] (hh0) {};
      \node[layer,above of=h1,yshift=5mm] (hh1) {};
      \node[layer,above of=h2,yshift=5mm] (hh2) {};

      \node[font=\scriptsize,layer,above=8mm of hh0] (y0) {$y^\text{(t-1)}$};
      \node[font=\scriptsize,layer,above=8mm of hh1] (y1) {$y^\text{(t)}$};
      \node[font=\scriptsize,layer,above=8mm of hh2] (y2) {$y^\text{(t-1)}$};

      \foreach \x in {0, 1, 2} {
        \draw[->,>=stealth] (x\x) -- (h\x);
        \draw[->,>=stealth] (hh\x) -- (y\x);
      }

      \draw[->,>=stealth] ([xshift=-5mm]h0.west) -- (h0);
      \draw[->,>=stealth] (h0) -- (h1);
      \draw[->,>=stealth] (h1) -- (h2);
      \draw[->,>=stealth] (h2) -- ([xshift=5mm]h2.east);

      \draw[->,>=stealth] (hh0) -- ([xshift=-5mm]hh0.west);
      \draw[->,>=stealth] (hh2) -- (hh1);
      \draw[->,>=stealth] (hh1) -- (hh0);
      \draw[->,>=stealth] ([xshift=5mm]hh2.east) -- (hh2);

      \draw[->] (x0) to[bend left=30] (hh0);
      \draw[->] (x1) to[bend left=30] (hh1);
      \draw[->] (x2) to[bend left=30] (hh2);
      \draw[->] (h0) to[bend right=35] (y0);
      \draw[->] (h1) to[bend right=35] (y1);
      \draw[->] (h2) to[bend right=35] (y2);
%      \draw[->,>=stealth] (x0.north west) arc (210:135:1.9);
%      \draw[->,>=stealth] (x1.north west) arc (210:135:1.9);
%      \draw[->,>=stealth] (x2.north west) arc (210:135:1.9);
%      \draw[->,>=stealth] ([yshift=-2mm]h0.north east)
%        arc[start angle=-30, end angle=45, radius=2.2];
%      \draw[->,>=stealth] ([yshift=-2mm]h1.north east)
%        arc[start angle=-30, end angle=45, radius=2.2];
%      \draw[->,>=stealth] ([yshift=-2mm]h2.north east)
%        arc[start angle=-30, end angle=45, radius=2.2];

%      \node at ([xshift=-3cm]h0.west) {Forward states};
%      \node at ([xshift=-1cm]h0.west) {\ldots};
%      \node at ([xshift=1cm]h2.east) {\ldots};
%      \node at ([xshift=-3cm]hh0.west) {Backward states};
%      \node at ([xshift=-1cm]hh0.west) {\ldots};
%      \node at ([xshift=1cm]hh2.east) {\ldots};

    \end{tikzpicture}
  \caption{A bidirectional RNN.}
  \label{fig:bidirectional-rnn}
\end{marginfigure}

RNNs can be used for a typical sequence model such as hidden Markov models,
In this case, the we use output of the RNN at each time step
to predict a label as shown in Figure~\ref{fig:rnn-unrolled}.
Such a network learns a one-to-one mapping
between equal-length inputs and the outputs.
The output layer is typically a classification
(e.g., using softmax activation).
This type of networks have many applications in NLP
typical examples including POS tagging, and named entity recognition (NER).

Another use of RNNs is depicted in Figure~\ref{fig:rnn-many-to-one}.
In this case the intermediate representations build by the RNN
is not used for any prediction.
The network builds a representation $h^{(t)}$ for the whole sequence,
and this representation is used for assigning a label to the sequence.
This RNN configuration is used frequently for sequence classification tasks,
e.g., text classification tasks like spam detection.%
\sidenote[][-3\baselineskip]{A variation of this architecture,
  where all the intermediate representations are combined somehow
  for a single final prediction is also common.}
\begin{figure}
  \centering
%    \tikzset{external/export next=false}
    \tikzsetnextfilename{rnn-many-to-one}
  \begin{tikzpicture}
    \tikzset{layer/.style={draw,%
                           inner sep=1pt,%
                           minimum width=12mm,%
                           minimum height=8mm,%
                           fill=black!20%
      }
    };
    \tikzset{ilayer/.style={layer,fill=white,rounded corners}
    };

    \node[] (x0) {$\vect{x}^{(0)}$};
    \node[right=of x0] (x1) {$\vect{x}^{(1)}$};
    \node[right=of x1] (x2) {\dots};
    \node[right=of x2] (x3) {$\vect{x}^{(t-1)}$};
    \node[right=of x3] (x4) {$\vect{x}^{(t)}$};
    \node[layer,above of=x0,yshift=5mm] (h0) {$h^{(0)}$};
    \node[layer,above of=x1,yshift=5mm] (h1) {$h^{(1)}$};
    \node[above of=x2,yshift=5mm] (h2) {\ldots};
    \node[layer,above of=x3,yshift=5mm] (h3) {$h^{(t-1)}$};
    \node[layer,above of=x4,yshift=5mm] (h4) {$h^{(t)}$};

    \node[layer,above=10mm of h4] (y4) {$y^{(t)}$};
    \foreach \x in {0, ..., 4}{
      \ifthenelse{\x = 2}{}{%
        \draw[very thick,->,shorten >=1pt,>=stealth]
          (x\x) -- (h\x);
      }
      \ifthenelse{\x = 0}{}{%
        \pgfmathparse{int(int(\x)-1)};
        \xdef\prevx{\pgfmathresult};
        \draw[very thick,->,shorten >=1pt,>=stealth]
            (h\prevx) -- (h\x);
      }
    }
    \draw[very thick,->,shorten >=1pt,>=stealth] (h4) -- (y4);
  \end{tikzpicture}
  \caption{An RNN for sequence classification.
    Only the final representation built by the RNN is used for prediction.}
  \label{fig:rnn-many-to-one}
\end{figure}

On final standard variant that is interesting for NLP applications
called a \emph{sequence-to-sequence} (or seq2seq) network.
In fact, this is an encoder--decoder architecture,
where both encoder and the decoder are recurrent networks.
In this setup, shown in Figure~\ref{fig:rnn-seq2seq},
the encoder RNN builds a representation for the complete input sequence,
which typically is terminated by a special end-of-sequence symbol.
The decoder's hidden layer is initialized using this representation,
and it is expected to produce the output sequence,
followed by the end-of-sequence symbol.
A very common variation in many applications is
to provide the previous output as input to the encoder
(shown with the gray curved arrows in the figure).
Note that we can train the network gold-standard output sequence.
However, during prediction time, the model has to rely on its own output.
\begin{figure*}
  \centering
%    \tikzset{external/export next=false}
    \tikzsetnextfilename{rnn-seq2seq}
  \begin{tikzpicture}[
      minimum width=12mm,
      minimum height=8mm,
    ]
    \tikzset{layer/.style={draw,%
                           inner sep=1pt,%
                           minimum width=10mm,%
                           minimum height=8mm,%
                           fill=black!20%
      }
    };
    %    \tikzset{ilayer/.style={layer,fill=white,rounded corners}};

    \node[] (x0) {$\vect{x}^{(0)}$};
    \node[right=of x0] (x1) {$\vect{x}^{(1)}$};
    \node[right=of x1] (x2) {\dots};
    \node[right=of x2] (x3) {$\langle\text{eos}\rangle$};
    \node[right=of x3] (x4) {};
    \node[right=of x4] (x5) {};
    \node[right=of x5] (x6) {};
    \node[right=of x6] (x7) {};

    \node[layer,above=of x0] (h0) {};
    \node[layer,above=of x1] (h1) {};
    \node[above=of x2] (h2) {\ldots};
    \node[layer,above=of x3] (h3) {};
    \node[layer,above=of x4] (h4) {};
    \node[above=of x5] (h5) {\ldots};
    \node[layer,above=of x6] (h6) {};
    \node[layer,above=of x7] (h7) {};

    \node[layer,above=of h3.base] (y3) {$y^{(1)}$};
    \node[layer,above=of h4.base] (y4) {$y^{(2)}$};
    \node[above=of h5.base] (y5) {\ldots};
    \node[layer,above=of h6.base] (y6) {$y^{(t)}$};
    \node[layer,above=of h7.base] (y7) {$\langle\text{eos}\rangle$};
    \foreach \x in {0, ..., 7}{
      \ifthenelse{\x = 2 \OR \x > 3}{}{%
        \draw[thick,->,shorten >=1pt,>=stealth]
          (x\x) -- (h\x);
      }
      \ifthenelse{\x = 0}{}{%
        \pgfmathparse{int(int(\x)-1)};
        \xdef\prevx{\pgfmathresult};
        \draw[thick,->,shorten >=1pt,>=stealth]
            (h\prevx) -- (h\x);
      }
    }
    \draw[gray,thick,->] (y3.north)
      .. controls ++(45:3) and ++(45:-3) ..
      (h4.south);
    \draw[gray,thick,->] (y6.north)
      .. controls ++(45:3) and ++(45:-3) ..
      (h7.south);
    \draw[thick,->,shorten >=1pt,>=stealth] (h4) -- (y4);
    \draw[thick,->,shorten >=1pt,>=stealth] (h3) -- (y3);
  \end{tikzpicture}
  \caption{A sequence-to-sequence model.}
  \label{fig:rnn-seq2seq}
\end{figure*}

Sequence-to-sequence networks similar to the one in Figure~\ref{fig:rnn-seq2seq}
are capable of transforming a sequence
to another sequence with a different length.
They are used in many applications,
probably most popular application being machine translation.
The modern seq2seq models are generally more complex than
the one described above.
A very popular extension to such models,
called an \emph{attention mechanism}
to provide the intermediate representations built by the encoder
to the decoder time steps,
often passing through another network component that learns
what parts of the input is more important for the present prediction task.
It is also common to use deeper, stacked,
RNN layers for both encoder and the decoder,
and bidirectional RNNs for the encoder part of the network.%
\sidenote{Can we use a bidirectional layer for the decoder?}

\section{Convolutional networks}

Convolutional neural networks (CNNs) are another type of popular ANN architecture.
They have become particularly popular in image processing tasks,
but they also made their way into language processing.

Convolution is an operation, a filter, applied to a signal,
which transforms it based on the neighbouring values at any point.
It has its roots in in signal processing,
where it is typically applied to a continuous signal.
However, for our purposes it is a filter that transforms each discrete unit
based on its neighbors.

\begin{marginfigure}
  \centering
%    \tikzset{external/export next=false}
    \tikzsetnextfilename{convolution-images}
    \begin{tikzpicture}[blue!50!black,x=2.8mm,y=2.8mm]
%      \draw[gray!50,very thin] (0,0) grid[step=1] (20,8);
      \draw[step=1] (0,0) grid[thin] (6,6);
      \draw[step=1] (8,3) grid[thin] (11,6);
      \draw[step=1] (13,1) grid[thin] (17,5);

      \xdef\x{2}
      \xdef\y{2}
      \draw[red,very thick,fill=red!40,opacity=0.2]
        ($(\x,\y) + (-1, -1) $)  rectangle ($(\x,\y) + (2, 2)$);
      \draw[red,very thick,fill=red!40]
        (\x,\y)  rectangle ($(\x,\y) + (1, 1)$);
      \draw[blue,very thick,fill=blue!40]
        ($(\x, \y) + (12,0)$) rectangle ($(\x, \y) + (13,1)$);
      \draw[red] ($ (\x, \y) + (2,-1)$) -- (8,3);
      \draw[red] ($ (\x, \y) + (2,2)$) -- (8,6);
      \draw[blue] (11,3) -- ($ (\x, \y) + (12,0) $);
      \draw[blue] (11,6) -- ($ (\x, \y) + (12,1) $);

      \node[anchor=south,font=\footnotesize] at (3, 6)
        {Input ($\textcolor{red}{\vect{X}}$)};
      \node[anchor=south,font=\footnotesize] at (9.5, 6)
        {Filter ($\vect{W}$)};
      \node[anchor=south,font=\footnotesize] at (15, 6)
        {Output ($\textcolor{blue}{\vect{Y}}$)};
      \node[anchor=south,font=\footnotesize] at (9.5, 0)
        {$\textcolor{blue}{y_{i}} = \sum\limits_{i} w_{i} \textcolor{red}{x_{i}}$};
    \end{tikzpicture}
    \caption{A demonstration of convolution in image processing.
      Every pixel in the image is passed through a filter,
      where the transformed value of the pixel is a weighted
      combination of its original value and its neighbors.
    }\label{fig:image-convolution}
\end{marginfigure}
In image processing a filter is typically a square matrix,
that slides over the complete image to transform every pixel,
as demonstrated in Figure~\ref{fig:image-convolution}.
Note that the convolution is not well-defined on the pixels
at the edges of the image.
In practice, one `pads' the images
(with values appropriate for the filter applied)
to obtain a transformed image with the same size as the original image.
Otherwise, the result would be a smaller image (as in Figure~\ref{fig:image-convolution}).

\begin{marginfigure}
  \centering
  \begin{tcolorbox}[center upper,center lower,]
      Blurring
      \[
        \frac{1}{16} \begin{bmatrix}
          1 & 2 & 1 \\
          2 & 4 & 2 \\
          1 & 2 & 1 \\
        \end{bmatrix}
      \]
      \tcblower
     Edge detection
      \[
        \begin{bmatrix}
          -1 & -1 & -1 \\
          -1 &  8 & -1 \\
          -1 & -1 & -1 \\
        \end{bmatrix}
      \]
  \end{tcolorbox}
  \caption{Two example filters (convolutions) used in image processing:
  blurring (top) and edge detection (bottom).}
  \label{fig:image-covolution-examples}
\end{marginfigure}
In standard image processing software many of the operations on images are
done through convolution operations.
Figure~\ref{fig:image-covolution-examples} shows two common filters used
by image processing software.
The first one, blurring, replaces a pixel
with a weighted average of its neighborhood,
making all pixels similar to their neighbors
and removing the details from the image.
The second filter shown, edge detection,
is probably more useful for machine learning.
The filter replaces pixels with similar intensities with its neighbors
with numbers close to 0,
while the pixels that are different from their neighbors
are assigned to larger intensity values.

The fixed filters demonstrated above are useful (and used)
in image processing software.
However, for machine learning we want to \emph{learn} these filters.
In a typical CNN application, we learn many such filters,
trained on the task we are interested in.%
\sidenote{Note that the values in the filter matrices above are
  the parameters that we want to learn.}
The hope is that each filter learns some useful aspect of the data.
For example, edges with different slants from given pixels.
It is also very common to stack the convolutional layers.
In a nutshell,
the idea with multiple layers of convolution is to
learn a hierarchy of filters.
Continuing with the examples with edges with different orientations,
another layer build on edges may learn useful (geometric) shapes,
and yet another layer may recognize object composed of these shapes,
and so on.
Figure~\ref{fig:convolution-layers} first two stages
of this hypothetical scenario.
If our aim is, for example, predicting whether an image contains people,
convolutions over the shapes
shown in the lower part of Figure~\ref{fig:convolution-layers}
are likely to  be useful.
\begin{marginfigure}
  \centering
  \begin{tcolorbox}[center upper,center lower,]
          \tikzset{external/export next=false}
%          \tikzsetnextfilename{cnn-image-example-features1}%
          \tikz[x=5mm,y=5mm,thick]%
            {\draw (0,0) -- (0,1)
                   (1,1) -- (2,0)
                   (3,0) -- (4,1)
                   (5, 0.5) -- (6, 0.5);}
    \tcblower
          \tikzset{external/export next=false}
%          \tikzsetnextfilename{cnn-image-example-features2}%
            \tikz[x=5mm,y=5mm,thick]%
            {\draw (0,0) rectangle (1,1);
             \draw (2,0.5) -- (3,0) -- (4,0.5) -- (3, 1) -- cycle;
             \draw (5.3,0) -- (5.4,1)
                   (5.8, 1) -- (6.0, 0)
                   (5.5, 0) -- (5.6, 0.6) -- (5.7, 0);}
  \end{tcolorbox}
  \caption{Results of possible applications of layers of convolutions.
    First layer of convolution may learn different filters for
    edges with different orientations (top),
    while another layer built on it may learn geometric shapes built with them
    (bottom).
    Yet another layer may be used to detect objects,
    like houses, windows, people,
    based on these shapes (not shown in the figure).
  }
  \label{fig:convolution-layers}
\end{marginfigure}

We discussed convolutions in the context of 2D objects (images),
since they are most commonly used in this area.
However, they can easily be extended to 3D objects
(e.g., for processing videos),
or applicable to 1D sequences (for speech and language processing).

\begin{marginfigure}
  \centering
  \tikzset{external/export next=false}
    \begin{tikzpicture}[blue!50!black,
                        x=5mm, y=5mm,
                        minimum size=5mm,
                        node distance=6mm,
                        inner sep=0pt]
      \tikzset{nedge/.style={>=stealth,->,thick}}
      \tikzset{elabel/.style={midway,
                              sloped,
                              above,
                              yshift=-1ex,
                              font=\footnotesize}}
      \node (x1) {$x_{1}$};
      \node[above=15mm of x1] (h1) {};
      \foreach \x in {2, ..., 5} {%
        \pgfmathparse{int(\x)-1};
        \xdef\prevx{\pgfmathresult};
        \node[right=of x\prevx] (x\x) {$x_{\x}$};
        \ifthenelse{\x = 5}{}{%
          \node[right=of h\prevx,circle,fill=black!25] (h\x) {$h_{\x}$};
        }
      };
      \draw[nedge,blue] (x1) -- (h2)
        node[elabel,xshift=-3mm] {$w_{\text{-}1}$};
      \draw[nedge] (x2) -- (h2)
        node[elabel] {$w_{0}$};
      \draw[nedge,blue] (x2) -- (h3)
        node[elabel,xshift=-3mm] {$w_{1}$};
      \draw[nedge,red] (x3) -- (h2)
        node[elabel,xshift=-3mm] {$w_{1}$};
      \draw[nedge] (x3) -- (h3)
        node[elabel] {$w_{0}$};
      \draw[nedge,blue] (x3) -- (h4)
        node[elabel,xshift=-3mm] {$w_{\text{-}1}$};
      \draw[nedge,red] (x4) -- (h3)
        node[elabel,xshift=-3mm] {$w_{1}$};
      \draw[nedge] (x4) -- (h4) node[elabel] {$w_{0}$};
      \draw[nedge,red] (x5) -- (h4)
        node[elabel,xshift=-3mm] {$w_{1}$};
    \end{tikzpicture}
  \caption{Demonstration of  1D convolution.
    The weights indicated with the same colors are
    the same regardless of their position in the sequence.
  }
  \label{fig:covolution-1d}
\end{marginfigure}
Now we look at the convolutions more closely,
but assuming that we work on a single-dimensional sequence.
The units in the sequence, for our purposes,
can be (representations of) words, characters, phonemes,
or other linguistic objects.
Figure~\ref{fig:covolution-1d} shows convolutions applied to such a sequence.
If we were running this convolutional network on words,
the convolution would learn something (useful) about word trigrams.
For example, the final aim is sentiment classification,
this convolution would result in higher values at the hidden representation
if for trigrams that are associated with negative or positive sentiments.
There are two aspects of the CNNs that set them apart
from typical neural networks such as MLP.
First, the weights are \emph{shared},
the same filter is run through the entire sequence
without modifying the weights during prediction.
And second,
the input layer and the hidden layers are not fully connected.
As well as being suitable for picking certain features,
these aspects reduce the number of parameters learned,
and complexity of the network.
In practice, many such filters
(possibly with different input window size)
used in combination with multiple layers of convolutions,
and finally with a fully connected prediction layer,
e.g., a sigmoid or softmax classifier.

\begin{marginfigure}
  \centering
    \tikzset{external/export next=false}
%    \tikzsetnextfilename{cnn-pooling}
    \begin{tikzpicture}[blue!50!black,
                        x=5mm, y=5mm,
                        minimum size=5mm,
                        node distance=4mm,
                        inner sep=0pt]
      \tikzset{nedge/.style={>=stealth,->,thick}}
      \tikzset{elabel/.style={midway,
                              sloped,above,
                              yshift=-1ex,
                              font=\footnotesize}}
      \node (x1) {$x_{1}$};
      \node[above=1cm of x1,circle,fill=black!25] (h1) {$h_{1}$};
      \node[above=1cm of h1] (hh1) {};
      \foreach \x in {2, ..., 5} {%
        \pgfmathparse{int(int(\x)-1)};
        \xdef\prevx{\pgfmathresult};
        \node[right=of x\prevx] (x\x) {$x_{\x}$};
        \node[right=of h\prevx,circle,fill=black!25] (h\x) {$h_{\x}$};
        \ifthenelse{\x = 5}{}{%
          \node[right=of hh\prevx,circle,fill=black!25]
            (hh\x) {$h_{\prevx}^{'}$};
        }
      };
      \draw[nedge] (x1) -- (h1);
      \draw[nedge,blue] (x1) -- (h2);
      \draw[nedge,red] (x2) -- (h1);
      \draw[nedge] (x2) -- (h2);
      \draw[nedge,blue] (x2) -- (h3);
      \draw[nedge,red] (x3) -- (h2);
      \draw[nedge] (x3) -- (h3);
      \draw[nedge,blue] (x3) -- (h4);
      \draw[nedge,red] (x4) -- (h3);
      \draw[nedge] (x4) -- (h4);
      \draw[nedge,blue] (x4) -- (h5);
      \draw[nedge,red] (x5) -- (h4);
      \draw[nedge] (x5) -- (h5);

      \draw[nedge] (h1) -- (hh2);
      \draw[nedge] (h2) -- (hh2);
      \draw[nedge] (h3) -- (hh2);
      \draw[nedge] (h2) -- (hh3);
      \draw[nedge] (h3) -- (hh3);
      \draw[nedge] (h4) -- (hh3);
      \draw[nedge] (h3) -- (hh4);
      \draw[nedge] (h4) -- (hh4);
      \draw[nedge] (h5) -- (hh4);

      \node[left=of h1,rotate=90,font=\footnotesize] {Convolution};
      \node[left=of hh1,rotate=90,font=\footnotesize] {Pooling};

    \end{tikzpicture}
  \caption{Demonstration of  pooling.
  }
  \label{fig:cnn-pooling}
\end{marginfigure}
Returning to the example of sentiment classification,
a CNN layer like the one in Figure~\ref{fig:covolution-1d} will
discover a trigram with high-sentiment content wherever it occurs
in the sequence.
However downstream classification layer need to still consider
hidden layer activations as separate features.
In many problems,
we do not want this location sensitivity.
For example, a phrase like \emph{not worth seeing} in a movie review
is an indication of a negative sentiment wherever it appears.
To make the features learned by convolutions
\emph{location invariant}, a concept called \emph{pooling} is applied.
Pooling simply calculates a statistics,
most commonly `maximum', over a range of its inputs.
When it is applied to convolutions as in Figure~\ref{fig:cnn-pooling},
the new features are relatively location invariant.
Another aspect of the pooling you to note is that,
it is a fixed operation,
there are no weights to learned in the pooling layer.

If we apply the `max pooling',
the value of $h_{1}'$ in the figure is
the maximum of $h_{1}$, $h_{2}$ and $h_{3}$,
which means if the convolution detects any interesting trigrams
from $x_{1}$ to $x_{4}$, it $h_{1}'$ will indicate it.
Hence, to some extent, a classifier that uses
the output of the network shown in Figure~\ref{fig:cnn-pooling}
will be insensitive to the location of the feature detected by the convolution.
However, Figure~\ref{fig:cnn-pooling} still retains some location sensitivity,
which may be useful for some applications.
For example, for a face recognition network,
it is likely important to detect eyes above a nose.
In problems where location is not useful at all
(which is the case in many text classification examples)
one can pool over the complete convolution output,
passing a single feature to the classifier from this filter.%
Remember that we typically use many convolutions,
hence, in this case,
the classifier will be given a single feature from each convolution.

\begin{marginfigure}
  \centering
    \tikzset{external/export next=false}
%    \tikzsetnextfilename{cnn-pooling}
    \begin{tikzpicture}[blue!50!black,
                        x=5mm, y=5mm,
                        minimum size=5mm,
                        node distance=4mm,
                        inner sep=0pt]
      \tikzset{nedge/.style={>=stealth,->,thick}}
      \tikzset{elabel/.style={midway,
                              sloped,above,
                              yshift=-1ex,
                              font=\footnotesize}}
      \node (x1) {$x_{1}$};
      \node[above=1cm of x1,circle,fill=black!25] (h1) {$h_{1}$};
      \foreach \x in {2, ..., 5} {%
        \pgfmathparse{int(int(\x)-1)};
        \xdef\prevx{\pgfmathresult};
        \node[right=of x\prevx] (x\x) {$x_{\x}$};
        \node[right=of h\prevx,circle,fill=black!25] (h\x) {$h_{\x}$};
      };

      \node[above=1cm of h2,circle,fill=black!25] (hh1) {$h_{1}^{'}$};
      \node[above=1cm of h4,circle,fill=black!25] (hh2) {$h_{2}^{'}$};

      \draw[nedge] (x1) -- (h1);
      \draw[nedge,blue] (x1) -- (h2);
      \draw[nedge,red] (x2) -- (h1);
      \draw[nedge] (x2) -- (h2);
      \draw[nedge,blue] (x2) -- (h3);
      \draw[nedge,red] (x3) -- (h2);
      \draw[nedge] (x3) -- (h3);
      \draw[nedge,blue] (x3) -- (h4);
      \draw[nedge,red] (x4) -- (h3);
      \draw[nedge] (x4) -- (h4);
      \draw[nedge,blue] (x4) -- (h5);
      \draw[nedge,red] (x5) -- (h4);
      \draw[nedge] (x5) -- (h5);

      \draw[nedge] (h1) -- (hh1);
      \draw[nedge] (h2) -- (hh1);
      \draw[nedge] (h3) -- (hh1);
      \draw[nedge] (h3) -- (hh2);
      \draw[nedge] (h4) -- (hh2);
      \draw[nedge] (h5) -- (hh2);

    \end{tikzpicture}
  \caption{The same network presented in Figure~\ref{fig:cnn-pooling},
    but the pooling layer has a stride of \num{3}.
  }
  \label{fig:cnn-stride}
\end{marginfigure}
For both convolutions and pooling,
the examples we looked at so far cover the whole sequence
by shifting the filter one unit at a time.
It is common to define a larger \emph{stride},
that shifts the convolution or pooling over more
than one unit at a time.
Figure~\ref{fig:cnn-stride} repeats the network shown
in Figure~\ref{fig:cnn-pooling} with a stride of \num{3} on the pooling layer.
With this configuration,
each output of the pooling layer covers exactly half of the convolutions.
However, note that due to hierarchical nature of the network,
they are affected by larger spans of input.

Most CNNs used in practice are deep,
resulting in diminishing numbers of features
when successive convolution or pooling layers are
stacked without \emph{padding} as shown in Figure~\ref{fig:cnn-no-padding}.
This is sometimes called \emph{valid} padding,
meaning that each convolution is calculated on real data.
However, it is a common practice to pad the input,
typically with \num{0}s,
so that the output of the network stays stable.
Figure~\ref{fig:cnn-padding} shows an example of padding applied
to the same network.
Here, each layer is padded from both sides with a single `imaginary' input.
For an image, this would mean padding the matrix from all sides.
With a stride of \num{1},
padding only one unit from all sides as in Figure~\ref{fig:cnn-padding}
results in an output the same dimensions as the input.
Hence, it is often called \emph{same} padding.
With a larger stride, however, this would result in a reduction
in the output dimension more than expected from the stride.
In such cases it is an option to pad as many values as necessary
to make sure that the output is of expected dimension,
which is sometimes called \emph{full} padding.

\begin{marginfigure}
  \centering
      \tikzset{external/export next=false}
%      \tikzsetnextfilename{cnn-padding}
      \begin{tikzpicture}[x=4mm,y=8mm,blue!50!black]
        \tikzset{nedge/.style={>=stealth,->}}
        \tikzset{neuron/.style={draw,%
                                circle,%
                                fill=black!20,%
                                inner sep=0,%
                                minimum size=8}
        }
          \node[inner sep=0, minimum size=8] at (-1, 0) {};
          \foreach \y in {0, ..., 4} {
            \pgfmathparse{int(\y+1)};
            \xdef\xstart{\pgfmathresult};
            \pgfmathparse{int(9-\y)};
            \xdef\xend{\pgfmathresult};
            \foreach \x in {\xstart, ..., \xend} {
              \node[neuron] (n-\x-\y) at (\x, \y) {};
              \ifthenelse{\y = 0}{}{%
                \pgfmathparse{int(\y-1)};
                \xdef\prevy{\pgfmathresult};
                \pgfmathparse{int(\x-1)};
                \xdef\prevx{\pgfmathresult};
                \pgfmathparse{int(\x+1)};
                \xdef\nextx{\pgfmathresult};
                \draw[nedge] (n-\prevx-\prevy) -- (n-\x-\y);
                \draw[nedge] (n-\x-\prevy) -- (n-\x-\y);
                \draw[nedge] (n-\nextx-\prevy) -- (n-\x-\y);
              }
            }
          }
          \node[inner sep=0, minimum size=8] at (10, 0) {};
      \end{tikzpicture}
  \caption{A deep CNN network without padding.}
  \label{fig:cnn-no-padding}
\end{marginfigure}
\begin{marginfigure}
  \centering
      \tikzset{external/export next=false}
%      \tikzsetnextfilename{cnn-padding}
      \begin{tikzpicture}[x=4mm,y=8mm,blue!50!black]
        \tikzset{nedge/.style={>=stealth,->}}
        \tikzset{neuron/.style={draw,%
                                circle,%
                                fill=black!20,%
                                inner sep=0,%
                                minimum size=8}
        }
          \foreach \y in {0, ..., 4} {
            \ifthenelse{\y = 4}{}{%
              \node[neuron,fill=white] (n--1-\y) at (-1, \y) {};
              \node[neuron,fill=white] (n-10-\y) at (10, \y) {};
            }
            \foreach \x in {0, ..., 9} {
              \node[neuron] (n-\x-\y) at (\x, \y) {};
              \ifthenelse{\y = 0}{}{%
                \pgfmathparse{int(\y-1)};
                \xdef\prevy{\pgfmathresult};
                \pgfmathparse{int(\x-1)};
                \xdef\prevx{\pgfmathresult};
                \pgfmathparse{int(\x+1)};
                \xdef\nextx{\pgfmathresult};
                \draw[nedge] (n-\prevx-\prevy) -- (n-\x-\y);
                \draw[nedge] (n-\x-\prevy) -- (n-\x-\y);
                \draw[nedge] (n-\nextx-\prevy) -- (n-\x-\y);
              }
            }
          }
      \end{tikzpicture}
  \caption{The same network in Figure~\ref{fig:cnn-no-padding} with padding.
  }
  \label{fig:cnn-padding}
\end{marginfigure}

In NLP, most common use of CNNs is text classification.
Given a sequence of texts,
we typically define multiple convolutions, or filters.
Each convolution, after training,
will detect an `n-gram' feature with the width of the convolution.
Although in theory a larger convolution width should learn
features within its window that are based on a (discontinuous) sub-sequence,
examples of convolutions with different sizes are sometime used.
Figure~\ref{fig:cnn-sentiment} demonstrates a possibly way to use
CNNs for text classification.
The first layer in the network is typically an \emph{embedding} layer,
a dense (as opposed to sparse one-hot) representation of the words
(we will cover embeddings later in this class).
The example uses three convolutions,
first two with width \num{2},
and the last one with width \num{3}.
The first convolution in the example, finds something interesting
in input bigram \emph{not really},
while the second one is more sensitive to bigram \emph{really worth}.
The network does a max pooling over the whole sequence,
and then uses the resulting representations as an input to
a classifier with a single hidden layer.
The final layer, a single (likely sigmoid) unit is used for binary classification.

\begin{figure}
  \tikzset{external/export next=false}
%  \tikzsetnextfilename{cnn-in-nlp-example}
  \begin{tikzpicture}[x=1cm, y=1cm,blue!50!black,thick, node distance=2mm]
    \tikzset{inp/.style={minimum width=2cm,minimum height=6mm}}

    \node[inp] (not) at (0,0) {not};
    \node[inp,right=of not] (really) {really};
    \node[inp,right=of really] (worth) {worth};
    \node[inp,right=of worth] (seeing) {seeing};

    \foreach \n in {not, really, worth, seeing} {
      \node[above=of \n,inner sep=0pt] (V-\n)
        {\tikz{\draw[thick] (0,0) grid[step=5mm] (1.5,0.5);}};
    }

    \node[inner sep=0pt] (c-1) at (1,2.5)
      {\tikz{\draw[thick,inner sep=0pt] (0,0) grid[step=5mm] (2.0,0.5);}};
    \node[right=of c-1,inner sep=0pt] (c-2)
      {\tikz{\draw[thick,inner sep=0pt] (0,0) grid[step=5mm] (2.0,0.5);}};
    \node[right=of c-2,inner sep=0pt] (c-3)
      {\tikz{\draw[thick,inner sep=0pt] (0,0) grid[step=5mm] (2.0,0.5);}};

    \draw[red] (V-not.north west) -- (c-1.south west);
    \draw[red] (V-really.north east) -- ($(c-1.south west) + (0.5, 0) $);

    \draw[red] (V-really.north west) -- ($(c-1.south west) + (0.5, 0) $);
    \draw[red] (V-worth.north east) -- ($(c-1.south west) + (1.0, 0) $);

    \draw[blue] (V-not.north west) -- (c-2.south west);
    \draw[blue] (V-really.north east) -- ($(c-2.south west) + (0.5, 0) $);

    \draw[very thick,red, fill=red!80] (c-1.south west)
      rectangle ($ (c-1.south west) + (0.5, 0.5) $);

    \draw[very thick,red,fill=red!30] ($ (c-1.south west) + (0.5, 0) $)
      rectangle ($ (c-1.south west) + (1, 0.5) $);

    \draw[very thick,blue, fill=blue!30] (c-2.south west)
      rectangle ($ (c-2.south west) + (0.5, 0.5) $);

    \draw[blue] (V-really.north west) -- ($(c-2.south west) + (0.5, 0) $);
    \draw[blue] (V-worth.north east) -- ($(c-2.south west) + (1, 0) $);

    \draw[very thick,blue, fill=blue!80] ($(c-2.south west) + (0.5, 0) $)
      rectangle ($ (c-2.south west) + (1, 0.5) $);

    \draw[orange] (V-not.north west) -- (c-3.south west);
    \draw[orange] (V-worth.north east) -- ($(c-3.south west) + (0.5, 0) $);
    \draw[very thick,orange, fill=orange!30] (c-3.south west)
      rectangle ($ (c-3.south west) + (0.5, 0.5) $);

    \node[draw, minimum height=5mm, minimum width=5mm, inner sep=0pt,%
          above=8mm of c-1, fill=red!80] (p-1) {};
    \node[draw, minimum height=5mm, minimum width=5mm, inner sep=0pt,%
          above=8mm of c-2, fill=blue!80] (p-2) {};
    \node[draw, minimum height=5mm, minimum width=5mm, inner sep=0pt,%
          above=8mm of c-3, fill=orange!30] (p-3) {};

    \draw (c-1.north west) -- (p-1.south west);
    \draw (c-1.north east) -- (p-1.south east);
    \draw (c-2.north west) -- (p-2.south west);
    \draw (c-2.north east) -- (p-2.south east);
    \draw (c-3.north west) -- (p-3.south west);
    \draw (c-3.north east) -- (p-3.south east);

    \node[draw,circle, minimum size=5mm, inner sep=0pt,%
          above=8mm of p-1] (h-1) {};
    \node[draw,circle, minimum size=5mm, inner sep=0pt,%
          above=8mm of p-2] (h-2) {};
    \node[draw,circle, minimum size=5mm, inner sep=0pt,%
          above=8mm of p-3] (h-3) {};

    \node[draw,circle, minimum size=5mm, inner sep=0pt,%
          above=8mm of h-2] (o) {};

    \draw[->] (p-1) -- (h-1);
    \draw[->] (p-1) -- (h-2);
    \draw[->] (p-1) -- (h-3);
    \draw[->] (p-2) -- (h-1);
    \draw[->] (p-2) -- (h-2);
    \draw[->] (p-2) -- (h-3);
    \draw[->] (p-3) -- (h-1);
    \draw[->] (p-3) -- (h-2);
    \draw[->] (p-3) -- (h-3);

    \draw[->] (h-1) -- (o);
    \draw[->] (h-2) -- (o);
    \draw[->] (h-3) -- (o);

    \node[blue,anchor=east,font=\footnotesize] at (-1,0) {Input};
    \node[blue,anchor=east,font=\footnotesize] at (-1,0.8) {Word vectors};
    \node[blue,anchor=east,font=\footnotesize] at (-1,1.5) {Convolution};
    \node[blue,anchor=east,font=\footnotesize] at (-1,2.5) {Feature maps};
    \node[blue,anchor=east,font=\footnotesize] at (-1,3.2) {Pooling};
    \node[blue,anchor=east,font=\footnotesize] at (-1,3.9) {Features};
    \node[blue,anchor=east,font=\footnotesize] at (-1,5.2) {Classifier};
  \end{tikzpicture}
  \caption{A demonstration of CNNs used for text classification.}
  \label{fig:cnn-sentiment}
\end{figure}

In real-life examples,
especially in image processing systems,
CNNs are typically deeper and also more structured.
This may make their training difficult despite the sparse connectivity
and shared weights.
For very large systems, it is also a common practice to pre-train components
of the network piece-by-piece.
In the NLP applications
like the one demonstrated in Figure~\ref{fig:cnn-sentiment},
the embeddings are typically trained separately,
and most of the time training continues on the task as well.


\section{Generate synthetic data}
\label{S:SYNTHETIC}
The creation of synthetic data is possible using OpenBDLM.
The analysis of synthetic data is useful for validation, test, and debugging purposes because the true value of the hidden states and model parameters are known.
OpenBDLM uses the transition model  of the state-space modelling approach (see Section~\ref{SS:LGSSM}) to create realistic synthetic data.
There are two ways for creating synthetic data using OpenBDLM:

\begin{itemize}
\item From the interactive tool
\item From an existing project.
\end{itemize}

\subsection{Generate synthetic data using the interactive tool}
The creation of the synthetic data from the interactive tool (option \colorbox{light-gray}{\lstinline[basicstyle = \mlttfamily \small, backgroundcolor = \color{light-gray}]!0!} from the starting menu) enables the creation of synthetic data from scratch.
OpenBDLM requests the user to provide the number of time series, and to define the time vector (starting time, end time, timestep).
In the next step, the user has to define the time-series dependence (if applicable), to provide the number of model class, and to define a set of block components for each time series, as well as model constrains between model classes (if applicable).
Default values for initial hidden states mean values and model parameters are automatically assigned for each block component.
In the case of two model classes, the synthetic baseline will switch between the first and the second model class according to the transition probability values (see Section~\ref{SS:THSKF}).
The amplitude of each synthetic anomaly (i.e. change of the local trend) is sampled randomly in a normal distribution of zero mean and standard deviation $\sigma_{w}^{12}$ as defined in the switching process noise transition matrix.
Alternately, the user may choose to create \emph{custom anomalies}.
In such a case, the beginning (in sample index), duration (in number of samples) and amplitude (in change of the local trend) of each anomaly is user specified.
The information about custom anomaly are stored in the field \lstinline[basicstyle = \mlttfamily \small, backgroundcolor = \color{light-gray}]!custom_anomalies! of the structure variable \lstinline[basicstyle = \mlttfamily \small, backgroundcolor = \color{light-gray}]!misc!:
\begin{itemize}
\item \lstinline[basicstyle = \mlttfamily \small, backgroundcolor = \color{light-gray}]!misc.custom_anomalies.start_custom_anomalies!: this field stores a $1\times \mathtt{A}$ vector of integers, where $\mathtt{A}$ is the total number of synthetic anomaly. Each value indicates the sample index of the anomaly start.
\item \lstinline[basicstyle = \mlttfamily \small, backgroundcolor = \color{light-gray}]!misc.custom_anomalies.duration_custom_anomalies!: this field stores a $1\times \mathtt{A}$ vector of integers, where $\mathtt{A}$ is the total number of synthetic anomaly. Each value indicates the anomaly duration in number of samples.
\item \lstinline[basicstyle = \mlttfamily \small, backgroundcolor = \color{light-gray}]!misc.custom_anomalies.amplitude_custom_anomalies!: this field stores a $1\times \mathtt{A}$ vector of real number, where $\mathtt{A}$ is the total number of synthetic anomaly. Each value indicates the amplitude of the anomaly in change of the local trend.
\end{itemize}
The synthetic data are saved in \lstinline[basicstyle = \mlttfamily \small, backgroundcolor = \color{light-gray}]!DATA_*.mat! and \lstinline[basicstyle = \mlttfamily \small, backgroundcolor = \color{light-gray}]!*.csv! data files, and a \lstinline[basicstyle = \mlttfamily \small, backgroundcolor = \color{light-gray}]!PROJ_*.mat! project file is created that stores the information about the model (structure variable \lstinline[basicstyle = \mlttfamily \small, backgroundcolor = \color{light-gray}]!model!), and the true hidden states (see structure variable \lstinline[basicstyle = \mlttfamily \small, backgroundcolor = \color{light-gray}]!estimation.ref!).

\begin{table}[h]
     \caption{Default value of model parameters and initial hidden states $\bm{\mu}_{0}$ and $\mathbf{\Sigma}_{0}$ for synthetic data generation.}
     \centering
     \begin{tabular}{r|lp{3.1cm}p{4cm}}\toprule
        & $\bm{\theta}$ & $\bm{\mu}_{0}$ & diag$(\mathbf{\Sigma}_{0})$ \\\cmidrule(lr){1-4}
    $\mathtt{LL}$   &  $\sigma_{w}^{\mathtt{LL}}=0$ &$[10]$ & $[0.1^{2}]$ \\
    $\mathtt{LT}$    & $\sigma_{w}^{\mathtt{LT}}=10^{-7}$ &  $[10, -0.1\times10^{-2}]$ & $[0.1^{2}, 0.1^{2}]$ \\
     $\mathtt{LA}$   & $\sigma_{w}^{\mathtt{LA}}=10^{-8}$  &  $[10, -0.1\times10^{-2} , -0.1\times10^{-5}]$ & $[0.1^{2}, 0.1^{2}, 0.1^{2}]$ \\
     $\mathtt{P}$  &  $p=[365.24, 1, 182.62] $, $\sigma_{w}^{\mathtt{P}}=0$  &$[10, 10]$ & $[0.2^{2}, 0.2^{2}]$  \\
     $\mathtt{KR}$  & $p=[365.24]$, $\ell=0.5$, $\sigma_{w,0}^{\mathtt{KR}}=\sigma_{w,1}^{\mathtt{KR}}=0$ &  $[$$-0.97$, $1.65$, $1.73$, $-1.91$, $0.23$, $0.37$, $-2.89$, $-0.22$, $0.73$, $-1.83$$]$ & $[$$0.01^{2}$, $0.01^{2}$, $0.01^{2}$, $0.01^{2}$, $0.01^{2}$, $0.01^{2}$, $0.01^{2}$, $0.01^{2}$, $0.01^{2}$, $0.01^{2}$$]$  \\
         $\mathtt{AR}$   &  $\phi^{\mathtt{AR}}=0.75$, $\sigma_{w}^{\mathtt{AR}}=1$  &$[0]$ & $[0.1^{2}]$ \\\bottomrule
     \end{tabular}
\label{table:defaultsynthetic}
\end{table}

\subsection{Generate synthetic data from an existing project}

Once a project is loaded, it is possible to create synthetic data from it (option \colorbox{light-gray}{\lstinline[basicstyle = \mlttfamily \small, backgroundcolor = \color{light-gray}]!16!} from the main menu (see Listing~\ref{LST:OpenBDLMMainMenu}).
The synthetic data time vector will be the same as the time vector in memory, and missing data will be replicated.
The model used to create the synthetic data will be the same as the model of the current project, including current initial hidden states as well as model parameters values.
The creation of synthetic data in this way is particularly useful to closely mimic real dataset.
The synthetic data are saved in \lstinline[basicstyle = \mlttfamily \small, backgroundcolor = \color{light-gray}]!DATA_new_*.mat! and \lstinline[basicstyle = \mlttfamily \small, backgroundcolor = \color{light-gray}]!*.csv! data files, and a \lstinline[basicstyle = \mlttfamily \small, backgroundcolor = \color{light-gray}]!PROJ_new_*.mat! new project file is created that stores the information about the model (structure variable \lstinline[basicstyle = \mlttfamily \small, backgroundcolor = \color{light-gray}]!model!), and the true hidden states (see structure variable \lstinline[basicstyle = \mlttfamily \small, backgroundcolor = \color{light-gray}]!estimation.ref!).

\subsection{Synthetic data generation functions}

The synthetic data creation workflow is presented Figure~\ref{FIG:SyntheticDataCreationWorkflow}.
The OpenBLDM functions used for synthetic data creation are:

\begin{description}[style=unboxed]\setlength\itemsep{0em}
\item[Pilot function for synthetic data creation] \leavevmode
  \begin{lstlisting}[ basicstyle = \mlttfamily \small, breaklines=true]
[data,model,estimation,misc]=piloteSimulateData(data,model,estimation,misc)
  \end{lstlisting}

\item[Creates synthetic data] \leavevmode
  \begin{lstlisting}[ basicstyle = \mlttfamily \small, breaklines=true]
[data,model,estimation,misc]=SimulateData(data,model,misc,varargin)
  \end{lstlisting}

\item[Create synthetic data from transition probabilities] \leavevmode
  \begin{lstlisting}[ basicstyle = \mlttfamily \small, breaklines=true]
[data, model, estimation, misc]=simulateDataFromTransitionProbabilities(data,model,misc)
  \end{lstlisting}

\item[Create synthetic data from custom anomalies (for two model classes only)] \leavevmode
  \begin{lstlisting}[ basicstyle = \mlttfamily \small, breaklines=true]
[data,model,estimation,misc]=simulateDataFromCustomAnomalies(data,model,misc)
  \end{lstlisting}

  \item[Models configuration for synthetic data (for synthetic data creation from interactive tool only)] \leavevmode
  \begin{lstlisting}[ basicstyle = \mlttfamily \small, breaklines=true]
[data,model,estimation,misc]=configureModelForDataSimulation(data,model,estimation,misc)
 \end{lstlisting}

 \item[Requests user inputs to define the number of synthetic time series to create (for synthetic data creation from interactive tool only)] \leavevmode
  \begin{lstlisting}[ basicstyle = \mlttfamily \small, breaklines=true]
  [data,misc]=defineDataLabels(data,misc)
 \end{lstlisting}

\item[Requests user inputs to define synthetic data time vector (for synthetic data creation from interactive tool only)] \leavevmode
  \begin{lstlisting}[ basicstyle = \mlttfamily \small, breaklines=true]
[data,misc]=defineTimestamps(data,misc)
 \end{lstlisting}

\end{description}

\begin{figure}[h]
  \centering
  \captionsetup{justification=centering}
\scalebox{0.7}{
\begin{tikzpicture}

\node[paralightgray](inputSDC){\begin{tabular}{c}  \lstinline[ basicstyle = \mlttfamily \small]!data! \\ \lstinline[ basicstyle = \mlttfamily \small]!model.param_properties! \\ \lstinline[ basicstyle = \mlttfamily \small]!model.initX, initV, initS! \end{tabular}};
\node[eslightgray](piloteSDC)[below of = inputSDC, yshift = -1cm]{\phantom{} piloteSimulateData.m \phantom{}};
\node[eslightgray](SDC)[below of = piloteSDC, yshift = -1cm]{\phantom{} SimulateData.m \phantom{}};
\node[testlightgray](testCustom)[below of = SDC, yshift = -1.5cm]{\begin{tabular}{c}  custom  \\ anomalies ?  \end{tabular}};
\node[eslightgray](SDCtransition)[below of = testCustom, yshift = -1.75cm, xshift = -3cm]{\begin{tabular}{c} SimulateData \\ FromTransitionProbabilities.m \end{tabular}};
\node[eslightgray](SDCcustom)[below of = testCustom , yshift = -1.75cm, xshift = 3cm]{\begin{tabular}{c} SimulateData \\ FromCustomAnomalies.m \end{tabular}};
\node[paralightgray](outputSDC)[below of = inputSDC, yshift = -10cm]{\lstinline[ basicstyle = \mlttfamily \small]!estimation.ref!};
%
\path[->, draw, thick] (inputSDC)edge(piloteSDC);
\path[->, draw, thick] (piloteSDC)edge(SDC);
\path[->, draw, thick] (SDC)edge(testCustom);
\path[->, draw, thick] (testCustom.east) -| (2cm,-6.5cm) -| node[pos=0.25, above]{yes} (SDCcustom);
\path[->, draw, thick] (testCustom.west) -| (-2cm,-6.5cm) -| node[pos=0.25, above]{no} (SDCtransition);
\path[->, draw, thick] (SDCtransition.south) |- (0cm,-10cm) -|  (outputSDC.north);
\path[->, draw, thick] (SDCcustom.south) |- (0cm,-10cm) -|  (outputSDC.north);
\end{tikzpicture} }
\caption{Synthetic data creation workflow} \label{FIG:SyntheticDataCreationWorkflow}
\end{figure}