% !TEX root = thesis.tex
\documentclass[thesis]{subfiles}

\begin{document}
\chapter[Co-adaption in Deep Neural Networks]{Co-adaption in\texorpdfstring{\\}{ }Deep Neural Networks}\label{pairablation}
As discussed in \cref{optimizationlink}, there is a link between structural priors and a failure in our current optimization methods. Dropout\index{dropout} in particular seems like it may have a link to structural priors given that it is claimed to be randomly sampling `thinner' network architectures composed of subsets of the neurons from the model. It can also be argued, as we will demonstrate, that dropout is an optimization trick rather than a form of regularization. Here we will present work that, while not being substantial enough for publication, may provide an interesting insight into the problem of `co-adaption' as discussed by \citet{Hinton2012} and the mechanism of dropout.

\section{The Limitations of First Order Optimization}
With the increasing number of practical applications of deep learning, the optimization of deep neural networks is of critical importance, with generalization, accuracy and training time all direct consequences. Due to practical considerations of training large state-of-the-art models with limited computational resources, network optimization is restricted to first order methods in practice -- typically stochastic gradient descent with momentum. With breakthroughs in initialization~\citep{glorot2010understanding,He2015b} and maintenance of numerical precision during training~\citep{Ioffe2015} alleviating the `vanishing gradient' problem, such methods have surpassed human accuracy on large scale image recognition datasets, amongst other breakthrough results. This success has overshadowed any weakness of the current methods of training deep networks.

There has been much evidence to suggest that the optimization of deep networks remains a concern however. \citet{NIPS2014_5484} showed that shallow networks could be regressed from deep networks, and claim that the success of deep networks could be explained by our inability to properly train shallow networks from scratch. More recently, \citet{He2015,He2016} in particular have shown that the optimization of very deep networks can expose fundamental optimization issues where training loss \emph{increases} for deeper networks, whereas even the trivial solution to maintain training loss --- the identity mapping --- is not discovered by the optimization. They suggest a work-around for this problem is to utilize residual layers, incorporating the identity explicitly. Why the optimization fails so spectacularly without identity connections remains unexplained however.

\citet{martens2010deep} suggests that `pathological curvature (see \cref{pathological}) is a possible explanation for the difficulty of training deep networks. For some networks, the error surface can have a complex curvature  and the solution is to use a second order optimization, proposing a more efficient method, `Hessian-free' optimization.

In this \namecref{pairablation} we will demonstrate that some of the contemporary issues in training deep networks may be explained as being caused by this pathological curvature in the high dimensional error surfaces, and our use of first-order optimization methods. We will show that in particular dropout, which has empirically been shown to improve generalization in deep networks can also be explained in this light, and why this improvement diminishes in the presence of batch normalization\index{batch normalization}, as observed by \citet{Ioffe2015}.
% - can't learn identity, because this would have many neurons with close distance! like in \citet{martens2010deep}, this results in bad curvature

\section{Co-adpation of Hidden Units in Deep Networks}
\citet{dropout,dropoutjmlr} proposed dropout as a regularization method for neural networks. In randomly dropping out hidden units --- zeroing out a random subset on each layer --- it was claimed that complex co-adaptions of these hidden units on training data, which do not generalize to the test set, are prevented. In practice dropout has seen remarkable success in improving the generalization of large neural networks, especially in the context of large fully-connected layers. Several follow-up methods have similarly suggested alternative methods of preventing this co-adaption~\citep{Cogswell2016}.

Although empirically dropout works well, the claim that hidden units learn to co-adapt has not itself been well demonstrated, and seemingly straightforward methods of doing have serious drawbacks. Showing the covariance/correlation between pairs of hidden units doesn't give the full story, since covariance shows the linear relationship between the units, but in a deep network this relationship is likely to be highly non-linear. Mutual information cannot be used either since most modern networks need to use unbounded activation functions, such as \glspl{relu}, in order to train effectively.

A crude but effective method of analyzing the importance of hidden units in neural networks is ablation -- zeroing out certain units in a trained network of interest, and observing the effect on the training and test loss. This method can be extended to empirically evaluate the importance of pairs of hidden units in trained neural networks. For each pair of hidden units, zero out the parameters of both units, and observe the effect on the loss calculated over the training/test set: pairwise ablation. This is however very expensive since each pair of hidden units must be evaluated over the entire dataset.

We will focus on layer-wise filter co-dependence, and thus only needed to evaluate the pairwise ablation within each layer. In addition for large datasets, such as \gls{ilsvrc}2012~\citep{ILSVRC2015}, a random subset of the dataset was evaluated.

\paragraph{ResNet-50}
\Cref{fig:resnet50ablation_conv1_top5} shows the results of pairwise ablation on a \gls{resnet}\index{ResNet}-50 \citet{He2015} network trained on \gls{ilsvrc}2012, on both the training set (\cref{fig:resnet50ablation_conv1_top5_train}) and validation set (\cref{fig:resnet50ablation_conv1_top5_test}).

\begin{figure}[tbp]
\begin{subfigure}[b]{0.33\textheight}
\centering
\begin{tikzpicture}
\begin{axis}[
    height=0.27\textheight,
    axis equal image,
%    axis lines=none,
    enlargelimits=false,
    %colorbar,
    %colorbar style={
    %    yticklabel={\pgfmathprintnumber\tick\,\%},
    %    yticklabel style={font=\footnotesize}
    %},
%    colormap name={Paired-12},
    %colormap name={RdYlBu-9},
    baseline,
    scale only axis,
    xmax = 64,
    xmin = 0,
    ymax = 64,
    ymin = 0,
]
%\addplot[surf,
%    view={0}{90},
\addplot [
    matrix plot*,
    point meta=explicit,
    point meta min=-1.6E-02,
    point meta max=1.0E-02,
] file [
    x index=0,
    y index=1,
    meta index=2,
]{ablationdata/resnet50/ablation_train_top5_conv1_square.dat};
\end{axis}
\end{tikzpicture}
\caption{Top-5 training acc.\ diff.\ for filter pairs}
\label{fig:resnet50ablation_conv1_top5_train_hist}
\end{subfigure}
\begin{subfigure}[b]{0.33\textheight}
\centering
\begin{tikzpicture}
\begin{axis}[
    height=0.27\textheight,
    axis equal image,
%    axis lines=none,
    enlargelimits=false,
    colorbar,
    colorbar style={
        yticklabel={\pgfmathprintnumber\tick\,\%},
        yticklabel style={font=\footnotesize\sffamily\sansmath}
    },
    %colormap name={RdYlBu-9},
    baseline,
    scale only axis,
    xmax = 64,
    xmin = 0,
    ymax = 64,
    ymin = 0,
]
%\addplot[surf,
%    view={0}{90},
\addplot [
    matrix plot*,
    point meta=explicit,
    point meta min=-1.6E-02,
    point meta max=1.0E-02,
] file [
    x index=0,
    y index=1,
    meta index=2,
]{ablationdata/resnet50/ablation_test_top5_conv1_square.dat};
\end{axis}
\end{tikzpicture}
\caption{Top-5 validation acc.\ diff.\ for filter pairs}
\label{fig:resnet50ablation_conv1_top5_test_hist}
\end{subfigure}\\
\begin{subfigure}[c]{0.47\textwidth}
\centering
\includegraphics[height=0.23\textheight]{resnet50histtrain}
\caption{Histogram of change in top-5 train.\ acc.}
\label{fig:resnet50ablation_conv1_top5_train}
\end{subfigure}
\hfill
\begin{subfigure}[c]{0.47\textwidth}
\centering
\includegraphics[height=0.24\textheight]{resnet50histtest}
\caption{Histogram of change in top-5 val.\ acc.}
\label{fig:resnet50ablation_conv1_top5_test}
\end{subfigure}

%\input{resnet50fig.tex}
\caption[Pairwise filter ablations in ResNet 50]{Histogram of the change in top-5 accuracy for all pairwise filter ablations of a 2500 randomly sampled images from the \glsfmttext{ilsvrc} training/validation set in \texttt{conv1} of \glsfmttext{resnet}\index{ResNet}-50.}
\label{fig:resnet50ablation_conv1_top5}
\end{figure}

While, as expected, most pairwise ablations result in a decrease in accuracy, a small but significant number of pairwise ablations result in an \emph{increase} in accuracy (and decrease in loss). For the validation set this seems to provide clear evidence of hidden units co-adapting, and hence overfitting to the training data. Surprisingly however, this effect is also evident when evaluating on the training set, and by definition cannot be explained by overfitting. We observed similar effects on all other layers of the network.

\paragraph{\Gls{mnist}} This effect is not limited to large state-of-the-art deep networks, or indeed even deep networks. Surprisingly this effect is reproducible with a minimal single hidden-layer \gls{mnist} network. With a fully-connected network with one hidden layer, \gls{relu} activation functions, and a variety of optimization methods, we find that this co-adaption is still present above a minimal number of hidden units.
\begin{figure}[tbp]
\begin{subfigure}[b]{0.95\linewidth}
\centering
\begin{tikzpicture}
% \begin{axis}[
%     axis x line=bottom,
%     axis y line=left,
%     \setplotcyclecat{5},
%     scale only axis,
%     width=0.8\linewidth,
%     height=0.16\textheight,
%     ylabel={$\max\left(-\Delta L\right)$},
%     xmin = 10,
%     xmax = 300,
%     ymode=log,
% %    xmode=log,
%     legend columns=-1,
%     legend style={at={(0.5,1.1)},anchor=south,text depth=.5ex},
% ]
% \addplot +[mark=none] table [x index=0, y expr=-\thisrowno{1}, col sep=comma]{ablationdata/mnist/chainer-mnist-ablate-stats_train_loss.log};
% \addplot +[mark=none] table [x index=0, y expr=-\thisrowno{1}, col sep=comma]{ablationdata/mnist/chainer-mnist-ablate-momentum-stats_train_loss.log};
% \addplot +[mark=none] table [x index=0, y expr=-\thisrowno{1}, col sep=comma]{ablationdata/mnist/chainer-mnist-ablate-wd-stats_train_loss.log};
% \addplot +[mark=none] table [x index=0, y expr=-\thisrowno{1}, col sep=comma]{ablationdata/mnist/chainer-mnist-ablate-dropout-stats_train_loss.log};
% \addplot +[mark=none] table [x index=0, y expr=-\thisrowno{1}, col sep=comma]{ablationdata/mnist/hessianfree-ablate-mnist-stats_train_loss.log};
% \legend{\gls{sgd}, momentum, weight decay, dropout, HF}
% \end{axis}
\begin{axis}[
    axis y line=left,
    axis x line=bottom,
    % axis x line=none,
    \setplotcyclecat{5},
    scale only axis,
    width=0.8\linewidth,
    height=0.14\textheight,
    xlabel={Hidden Units},
    ylabel={$\max\left(\Delta L\right)$},
    xmin = 10,
    xmax = 300,
    ymode=log,
%    xmode=log,,
     legend columns=-1,
     legend style={at={(0.5,1.1)},anchor=south,text depth=.5ex},
]
\addplot +[mark=none] table [x index=0, y expr=\thisrowno{2}, col sep=comma]{ablationdata/mnist/chainer-mnist-ablate-stats_train_loss.log};
\addplot +[mark=none] table [x index=0, y expr=\thisrowno{2}, col sep=comma]{ablationdata/mnist/chainer-mnist-ablate-momentum-stats_train_loss.log};
\addplot +[mark=none] table [x index=0, y expr=\thisrowno{2}, col sep=comma]{ablationdata/mnist/chainer-mnist-ablate-wd-stats_train_loss.log};
\addplot +[mark=none] table [x index=0, y expr=\thisrowno{2}, col sep=comma]{ablationdata/mnist/chainer-mnist-ablate-dropout-stats_train_loss.log};
%\addplot +[mark=none] table [x index=0, y expr=\thisrowno{2}, col sep=comma]{ablationdata/mnist/hessianfree-ablate-mnist-stats_train_loss.log};
\legend{\gls{sgd}, momentum, weight decay, dropout, HF}
\end{axis}
\end{tikzpicture}
\caption{Training loss change: maximum increase}
%Maximum decrease (solid/left axis), and maximum increase (dotted/right axis).}
\label{fig:mnist_ablation_train_loss}
\end{subfigure}
\begin{subfigure}[b]{0.95\linewidth}
\centering
\begin{tikzpicture}
% \begin{axis}[
%     axis y line=left,
%     axis x line=bottom,
%     \setplotcyclecat{5},
%     scale only axis,
%     width=0.8\linewidth,
%     height=0.16\textheight,
%     xlabel={Hidden Units},
%     ylabel={$\max\left(\Delta \textrm{Acc.}\right)$},
%     xmin = 10,
%     xmax = 300,
%     ymode=log,
% %    xmode=log,
% ]
% \addplot +[mark=none] table [x index=0, y expr=\thisrowno{2}, col sep=comma]{ablationdata/mnist/chainer-mnist-ablate-stats_train_acc.log};
% \addplot +[mark=none] table [x index=0, y expr=\thisrowno{2}, col sep=comma]{ablationdata/mnist/chainer-mnist-ablate-momentum-stats_train_acc.log};
% \addplot +[mark=none] table [x index=0, y expr=\thisrowno{2}, col sep=comma]{ablationdata/mnist/chainer-mnist-ablate-wd-stats_train_acc.log};
% \addplot +[mark=none] table [x index=0, y expr=\thisrowno{2}, col sep=comma]{ablationdata/mnist/chainer-mnist-ablate-dropout-stats_train_acc.log};
% \addplot +[mark=none] table [x index=0, y expr=\thisrowno{2}, col sep=comma]{ablationdata/mnist/hessianfree-ablate-mnist-stats_train_acc.log};
% \end{axis}
\begin{axis}[
    axis y line=left,
    axis x line=bottom,
    % axis x line=none,
    \setplotcyclecat{5},
    scale only axis,
    width=0.8\linewidth,
    height=0.14\textheight,
    xlabel={Hidden Units},
    ylabel={$\max\left(-\Delta \textrm{Acc.}\right)$},
    xmin = 10,
    xmax = 300,
    ymode=log,
%    xmode=log,
]
\addplot +[mark=none] table [x index=0, y expr=-\thisrowno{1}, col sep=comma]{ablationdata/mnist/chainer-mnist-ablate-stats_train_acc.log};
\addplot +[mark=none] table [x index=0, y expr=-\thisrowno{1}, col sep=comma]{ablationdata/mnist/chainer-mnist-ablate-momentum-stats_train_acc.log};
\addplot +[mark=none] table [x index=0, y expr=-\thisrowno{1}, col sep=comma]{ablationdata/mnist/chainer-mnist-ablate-wd-stats_train_acc.log};
\addplot +[mark=none] table [x index=0, y expr=-\thisrowno{1}, col sep=comma]{ablationdata/mnist/chainer-mnist-ablate-dropout-stats_train_acc.log};
%\addplot +[mark=none] table [x index=0, y expr=-\thisrowno{1}, col sep=comma]{ablationdata/mnist/hessianfree-ablate-mnist-stats_train_acc.log};
\end{axis}
\end{tikzpicture}
\caption{Training accuracy: maximum decrease}
%Maximum increase (solid/left axis), and maximum decrease (dotted/right axis).}
\label{fig:mnist_ablation_train_acc}
\end{subfigure}
\begin{subfigure}[b]{0.95\linewidth}
\centering
\begin{tikzpicture}
% \begin{axis}[
%     axis y line=left,
%     axis x line=bottom,
%     \setplotcyclecat{5},
%     scale only axis,
%     width=0.8\linewidth,
%     height=0.16\textheight,
%     xlabel={Hidden Units},
%     ylabel={$\max\left(-\Delta L\right)$},
%     xmin = 10,
%     xmax = 300,
%     ymode=log,
% %    xmode=log,
% ]
% \addplot +[mark=none] table [x index=0, y expr=-\thisrowno{1}, col sep=comma]{ablationdata/mnist/chainer-mnist-ablate-stats_test_loss.log};
% \addplot +[mark=none] table [x index=0, y expr=-\thisrowno{1}, col sep=comma]{ablationdata/mnist/chainer-mnist-ablate-momentum-stats_test_loss.log};
% \addplot +[mark=none] table [x index=0, y expr=-\thisrowno{1}, col sep=comma]{ablationdata/mnist/chainer-mnist-ablate-wd-stats_test_loss.log};
% \addplot +[mark=none] table [x index=0, y expr=-\thisrowno{1}, col sep=comma]{ablationdata/mnist/chainer-mnist-ablate-dropout-stats_test_loss.log};
% \addplot +[mark=none] table [x index=0, y expr=-\thisrowno{1}, col sep=comma]{ablationdata/mnist/hessianfree-ablate-mnist-stats_test_loss.log};
% \end{axis}
\begin{axis}[
    axis y line=left,
    axis x line=bottom,
    % axis x line=none,
    \setplotcyclecat{5},
    scale only axis,
    width=0.8\linewidth,
    height=0.14\textheight,
    xlabel={Hidden Units},
    ylabel={$\max\left(\Delta L\right)$},
    xmin = 10,
    xmax = 300,
    ymode=log,
%    xmode=log,
]
\addplot +[mark=none] table [x index=0, y expr=\thisrowno{2}, col sep=comma]{ablationdata/mnist/chainer-mnist-ablate-stats_test_loss.log};
\addplot +[mark=none] table [x index=0, y expr=\thisrowno{2}, col sep=comma]{ablationdata/mnist/chainer-mnist-ablate-momentum-stats_test_loss.log};
\addplot +[mark=none] table [x index=0, y expr=\thisrowno{2}, col sep=comma]{ablationdata/mnist/chainer-mnist-ablate-wd-stats_test_loss.log};
\addplot +[mark=none] table [x index=0, y expr=\thisrowno{2}, col sep=comma]{ablationdata/mnist/chainer-mnist-ablate-dropout-stats_test_loss.log};
%\addplot +[mark=none] table [x index=0, y expr=\thisrowno{2}, col sep=comma]{ablationdata/mnist/hessianfree-ablate-mnist-stats_test_loss.log};
\end{axis}
\end{tikzpicture}
\caption{Test loss: maximum increase}
%Maximum decrease (solid/left axis), and maximum increase (dotted/right axis).}
\label{fig:mnist_ablation_test_loss}
\end{subfigure}
\begin{subfigure}[b]{0.95\linewidth}
\centering
\begin{tikzpicture}
% \begin{axis}[
%     axis x line=bottom,
%     axis y line=left,
%     \setplotcyclecat{5},
%     scale only axis,
%     width=0.8\linewidth,
%     height=0.16\textheight,
%     xlabel={Hidden Units},
%     ylabel={$\max\left(\Delta \textrm{Acc.}\right)$},
%     xmin = 10,
%     xmax = 300,
%     ymode=log,
% %    xmode=log,
% ]
% \addplot +[mark=none] table [x index=0, y expr=\thisrowno{2}, col sep=comma]{ablationdata/mnist/chainer-mnist-ablate-stats_test_acc.log};
% \addplot +[mark=none] table [x index=0, y expr=\thisrowno{2}, col sep=comma]{ablationdata/mnist/chainer-mnist-ablate-momentum-stats_test_acc.log};
% \addplot +[mark=none] table [x index=0, y expr=\thisrowno{2}, col sep=comma]{ablationdata/mnist/chainer-mnist-ablate-wd-stats_test_acc.log};
% \addplot +[mark=none] table [x index=0, y expr=\thisrowno{2}, col sep=comma]{ablationdata/mnist/chainer-mnist-ablate-dropout-stats_test_acc.log};
% \addplot +[mark=none] table [x index=0, y expr=\thisrowno{2}, col sep=comma]{ablationdata/mnist/hessianfree-ablate-mnist-stats_test_acc.log};
% \end{axis}
\begin{axis}[
    axis y line=left,
    axis x line=bottom,
    % axis x line=none,
    \setplotcyclecat{5},
    scale only axis,
    width=0.8\linewidth,
    height=0.14\textheight,
    xlabel={Hidden Units},
    ylabel={$\max\left(-\Delta \textrm{Acc.}\right)$},
    xmin = 10,
    xmax = 300,
    ymode=log,
%    xmode=log,
]
\addplot +[mark=none] table [x index=0, y expr=-\thisrowno{1}, col sep=comma]{ablationdata/mnist/chainer-mnist-ablate-stats_test_acc.log};
\addplot +[mark=none] table [x index=0, y expr=-\thisrowno{1}, col sep=comma]{ablationdata/mnist/chainer-mnist-ablate-momentum-stats_test_acc.log};
\addplot +[mark=none] table [x index=0, y expr=-\thisrowno{1}, col sep=comma]{ablationdata/mnist/chainer-mnist-ablate-wd-stats_test_acc.log};
\addplot +[mark=none] table [x index=0, y expr=-\thisrowno{1}, col sep=comma]{ablationdata/mnist/chainer-mnist-ablate-dropout-stats_test_acc.log};
%\addplot +[mark=none] table [x index=0, y expr=-\thisrowno{1}, col sep=comma]{ablationdata/mnist/hessianfree-ablate-mnist-stats_test_acc.log};
\end{axis}
\end{tikzpicture}
\caption{Test accuracy: maximum decrease}
%Maximum increase (solid/left axis), and maximum decrease (dotted/right axis).}
\label{fig:mnist_ablation_test_acc}
\end{subfigure}
\caption[Pairwise filter ablation for MNIST]{Maximum increase and decrease in training/test loss/accuracy for a single-layer hidden MNIST classification network under pairwise ablation of the hidden units.
%Solid lines show the maximum decrease in loss or increase in accuracy for all pairs of neurons.
%Dotted lines
The plots show the maximum increase in loss or decrease in accuracy. Both are measures of the level of neural co-adpation in the trained networks.}\label{fig:mnist_ablation}
\end{figure}
\begin{figure}[tbp]
\begin{subfigure}[b]{0.95\linewidth}
\centering
\begin{tikzpicture}
\begin{axis}[
    ybar,
    bar width=0.1,
    axis x line=bottom,
    axis y line=left,
    \setplotcyclecat{5},
    scale only axis,
    width=0.8\linewidth,
    height=0.3\textheight,
    xlabel={Hidden Units},
    ylabel={\% of pairwise units where $\Delta L < 0$},
    yticklabel=\pgfmathprintnumber{\tick}\,\%,
    xmin = 10,
    xmax = 100,
%    ymode=log,
%    xmode=log,
    legend columns=-1,
    legend style={at={(0.5,1.1)},anchor=south,text depth=.5ex},
]
\addplot +[mark=none] table [x index=0, y expr=100*\thisrowno{5} / (\thisrowno{0}*\thisrowno{0}), col sep=comma]{ablationdata/mnist/chainer-mnist-ablate-stats_train_loss.log};
\addplot +[mark=none] table [x index=0, y expr=100*\thisrowno{5} / (\thisrowno{0}*\thisrowno{0}), col sep=comma]{ablationdata/mnist/chainer-mnist-ablate-momentum-stats_train_loss.log};
\addplot +[mark=none] table [x index=0, y expr=100*\thisrowno{5} / (\thisrowno{0}*\thisrowno{0}), col sep=comma]{ablationdata/mnist/chainer-mnist-ablate-wd-stats_train_loss.log};
\addplot +[mark=none] table [x index=0, y expr=100*\thisrowno{5} / (\thisrowno{0}*\thisrowno{0}), col sep=comma]{ablationdata/mnist/chainer-mnist-ablate-dropout-stats_train_loss.log};
%\addplot +[mark=none] table [x index=0, y expr=100*\thisrowno{5} / (\thisrowno{0}*\thisrowno{0}), col sep=comma]{ablationdata/mnist/hessianfree-ablate-mnist-stats_train_loss.log};
\legend{\gls{sgd}, momentum, weight decay\index{weight decay}, dropout, HF}
\end{axis}
\end{tikzpicture}
\caption{Training Loss: Percentage of pairs of hidden units that exhibit adverse co-adaption.}
\label{fig:mnist_ablation_stats_train_loss_lt}
\end{subfigure}
\begin{subfigure}[b]{0.95\linewidth}
\centering
\begin{tikzpicture}
\begin{axis}[
    ybar,
    bar width=0.1,
    axis x line=bottom,
    axis y line=left,
    \setplotcyclecat{5},
    scale only axis,
    width=0.8\linewidth,
    height=0.3\textheight,
    xlabel={Hidden Units},
    ylabel={\% of pairwise units where $\Delta L = 0$},
    yticklabel=\pgfmathprintnumber{\tick}\,\%,
    xmin = 10,
    xmax = 100,
%    ymode=log,
%    xmode=log,
]
\addplot +[mark=none] table [x index=0, y expr=100*\thisrowno{4} / (\thisrowno{0}*\thisrowno{0}), col sep=comma]{ablationdata/mnist/chainer-mnist-ablate-stats_train_loss.log};
\addplot +[mark=none] table [x index=0, y expr=100*\thisrowno{4} / (\thisrowno{0}*\thisrowno{0}), col sep=comma]{ablationdata/mnist/chainer-mnist-ablate-momentum-stats_train_loss.log};
\addplot +[mark=none] table [x index=0, y expr=100*\thisrowno{4} / (\thisrowno{0}*\thisrowno{0}), col sep=comma]{ablationdata/mnist/chainer-mnist-ablate-wd-stats_train_loss.log};
\addplot +[mark=none] table [x index=0, y expr=100*\thisrowno{4} / (\thisrowno{0}*\thisrowno{0}), col sep=comma]{ablationdata/mnist/chainer-mnist-ablate-dropout-stats_train_loss.log};
%\addplot +[mark=none] table [x index=0, y expr=100*\thisrowno{4} / (\thisrowno{0}*\thisrowno{0}), col sep=comma]{ablationdata/mnist/hessianfree-ablate-mnist-stats_train_loss.log};
\end{axis}
\end{tikzpicture}
\caption{Training Loss: Percentage of pairs of hidden units that are independent.}
\label{fig:mnist_ablation_stats_train_loss_eq}
\end{subfigure}
% \begin{subfigure}[b]{0.95\linewidth}
% \centering
% \begin{tikzpicture}
% \begin{axis}[
%     ybar,
%     bar width=0.1,
%     axis x line=bottom,
%     axis y line=left,
%     \setplotcyclecat{5},
%     scale only axis,
%     width=0.8\linewidth,
%     height=0.16\textheight,
%     xlabel={Hidden Units},
%     ylabel={\% of pairwise units where $\Delta L > 0$},
%     yticklabel=\pgfmathprintnumber{\tick}\,\%,
%     xmin = 10,
%     xmax = 100,
% %    ymode=log,
% %    xmode=log,
% ]
% \addplot +[mark=none] table [x index=0, y expr=100*\thisrowno{6} / (\thisrowno{0}*\thisrowno{0}), col sep=comma]{ablationdata/mnist/chainer-mnist-ablate-stats_train_loss.log};
% \addplot +[mark=none] table [x index=0, y expr=100*\thisrowno{6} / (\thisrowno{0}*\thisrowno{0}), col sep=comma]{ablationdata/mnist/chainer-mnist-ablate-momentum-stats_train_loss.log};
% \addplot +[mark=none] table [x index=0, y expr=100*\thisrowno{6} / (\thisrowno{0}*\thisrowno{0}), col sep=comma]{ablationdata/mnist/chainer-mnist-ablate-wd-stats_train_loss.log};
% \addplot +[mark=none] table [x index=0, y expr=100*\thisrowno{6} / (\thisrowno{0}*\thisrowno{0}), col sep=comma]{ablationdata/mnist/chainer-mnist-ablate-dropout-stats_train_loss.log};
% \addplot +[mark=none] table [x index=0, y expr=100*\thisrowno{6} / (\thisrowno{0}*\thisrowno{0}), col sep=comma]{ablationdata/mnist/hessianfree-ablate-mnist-stats_train_loss.log};
% \end{axis}
% \end{tikzpicture}
% \caption{Training Loss: Percentage of pairs of hidden units that are codependent.}
% \label{fig:mnist_ablation_stats_train_loss_gt}
% \end{subfigure}
\caption[Pairwise filter ablation counts for MNIST]{Number of pairs of hidden units in a single-layer hidden MNIST classification network which under ablation, are adversely dependent ($\Delta L < 0$) or independent ($\Delta L = 0$) for the training set.}
\label{fig:mnist_ablation_stats}
\end{figure}
\Cref{fig:mnist_ablation} shows the minimum and maximum increase in training loss/accuracy and test loss/accuracy for the MNIST network with different numbers of hidden units, when pairwise filters are ablated, as evaluated on the entire MNIST training/test sets. The effect of training with weight decay\index{weight decay} dropout and momentum are also compared with vanilla \gls{sgd}. Weight decay\index{weight decay} and dropout are considered to be regularization methods, while momentum is an optimization trick that is intended to help avoid some issues of using a first order optimization --- it can help speed up learning in the presence of some types of pathological curvature that would otherwise lead to slow optimization or a poor local minima with vanilla \gls{sgd}.

In \cref{fig:mnist_ablation} momentum clearly helps avoid co-adaption as compared to vanilla \gls{sgd}, where co-adaption is significant. Weight decay\index{weight decay} on the other hand, does not seem to have a helpful effect on co-adaption, giving almost identical results to vanilla \gls{sgd}. As a form of regularization, this might be expected, as it should help generalization, not training fit.

Taken by itself, this observation suggests that co-adaption may be a symptom of an optimization problem, and thus an optimization trick like momentum helps while regularization does not. On the other hand, dropout seems to reduce the number of co-adapted units significantly, and is even effective at reducing co-adaption at \emph{training time}. If dropout is a regularization method, then this seems to conflict with our findings with weight decay\index{weight decay} and momentum.

\section{Dropout as an Optimization Trick}\label{dropoutasopttrick}
Dropout can also be thought of as a orthogonal projection of the error surface onto a random lower-dimensional subspace, in which the curvature of the error surface may no longer exhibit pathological issues, and optimization may be easier. For example, if in a subset of the dimensions, a deep valley exists (as illustrated in \cref{fig:pathological}), and these dimensions are dropped-out, first-order optimization will be substantially easier. Random projection is a well established method for dimensionality reduction of high dimensional spaces~\citep{kaski1998dimensionality,fodor2002survey}.
If a layer has $N$ nodes, and a width matrix $W$, and input vector $x$, then dropout on the layer of $K/N$ nodes may be defined as the transformation:
\begin{equation}
    \textrm{dropout}(\mathbf{W}) = \mathbf{D}_{i} \mathbf{W},
\end{equation}
where $D$ is a diagonal binary matrix, with rank $K$, defining an orthogonal projection onto a $K$-dimensional subspace.

To demonstrate that it is this projection, rather than the zeroing out of neurons itself, that is responsible for performance improvements with dropout, we can instead perform a random projection in a different orthogonal co-ordinate basis, which does not dropout (zero) any neurons. To do this we can first rotate the parameters with random rotation matrix into a non-axis aligned co-ordinate basis, and perform dropout (orthogonal projection) in the rotated space, and then rotate back into the original co-ordinate basis:
\begin{equation}
    \textrm{dropproject}(\mathbf{W}_l) = \mathbf{R}^{-1}\textrm{dropout}(\mathbf{R} \mathbf{W}_l),
\end{equation}
for a random rotation matrix $\mathbf{R}\in \textrm{SO}_N$. This ``dropproject'' method avoids zeroing out any of the units, while still performing an equivalent projection as that in dropout.

\begin{figure}[tbp]
\begin{subfigure}[b]{0.95\linewidth}
\centering
\begin{tikzpicture}
\begin{axis}[
    thick,
    \setplotcyclecat{5},
    scale only axis,
    axis x line=bottom,
    axis y line=left,
    %axis line style={thick},
    width=0.8\linewidth,
    height=0.14\textheight,
    xlabel={Epochs},
    ylabel={Log Loss},
    ymode=log,
    legend columns=-1,
    legend style={at={(0.5,1.1)},anchor=south,text depth=.5ex},
]
\addplot +[mark=none] table [x=epoch, y expr=\thisrow{main/loss}, col sep=comma]{ablationdata/dropproject/train_standard.csv};
\addplot +[mark=none] table [x=epoch, y expr=\thisrow{main/loss}, col sep=comma]{ablationdata/dropproject/train_dropout.csv};
\addplot +[mark=none] table [x=epoch, y expr=\thisrow{main/loss}, col sep=comma]{ablationdata/dropproject/train_dropproject.csv};
\addplot +[mark=none] table [x=epoch, y expr=\thisrow{main/loss}, col sep=comma]{ablationdata/dropproject/train_dropproject10.csv};
\addplot +[mark=none] table [x=epoch, y expr=\thisrow{main/loss}, col sep=comma]{ablationdata/dropproject/train_dropprojectrandom.csv};
\legend{
standard,
dropout,
dropproject,
DP-Random 10,
DP-Random}
\end{axis}
\end{tikzpicture}
\caption{CIFAR Training Log Loss}
\label{fig:cifar_dropproject_train_loss}
\end{subfigure}
\begin{subfigure}[b]{0.95\linewidth}
\begin{tikzpicture}
\begin{axis}[
    thick,
    \setplotcyclecat{5},
    scale only axis,
    axis x line=bottom,
    axis y line=left,
    %axis line style={thick},
    width=0.8\linewidth,
    height=0.14\textheight,
    xlabel={Epochs},
    ylabel={Log Loss},
    ymode=log,
%    ymax=1,
    ymin=0,
]
\addplot +[mark=none] table [x=epoch, y expr=\thisrow{validation/main/loss}, col sep=comma]{ablationdata/dropproject/train_standard.csv};
\addplot +[mark=none] table [x=epoch, y expr=\thisrow{validation/main/loss}, col sep=comma]{ablationdata/dropproject/train_dropout.csv};
\addplot +[mark=none] table [x=epoch, y expr=\thisrow{validation/main/loss}, col sep=comma]{ablationdata/dropproject/train_dropproject.csv};
\addplot +[mark=none] table [x=epoch, y expr=\thisrow{validation/main/loss}, col sep=comma]{ablationdata/dropproject/train_dropproject10.csv};
\addplot +[mark=none] table [x=epoch, y expr=\thisrow{validation/main/loss}, col sep=comma]{ablationdata/dropproject/train_dropprojectrandom.csv};
\end{axis}
\end{tikzpicture}
\caption{CIFAR Test Log Loss}
\label{fig:cifar_dropproject_test_loss}
\end{subfigure}
\begin{subfigure}[b]{0.95\linewidth}
\begin{tikzpicture}
\begin{axis}[
    thick,
    \setplotcyclecat{5},
    scale only axis,
    axis x line=bottom,
    axis y line=left,
    %axis line style={thick},
    width=0.8\linewidth,
    height=0.14\textheight,
    xlabel={Epochs},
    ylabel={Training Error},
%    ymax=1,
%    ymin=0,
]
\addplot +[mark=none] table [x=epoch, y expr=1-\thisrow{main/accuracy}, col sep=comma]{ablationdata/dropproject/train_standard.csv};
\addplot +[mark=none] table [x=epoch, y expr=1-\thisrow{main/accuracy}, col sep=comma]{ablationdata/dropproject/train_dropout.csv};
\addplot +[mark=none] table [x=epoch, y expr=1-\thisrow{main/accuracy}, col sep=comma]{ablationdata/dropproject/train_dropproject.csv};
\addplot +[mark=none] table [x=epoch, y expr=1-\thisrow{main/accuracy}, col sep=comma]{ablationdata/dropproject/train_dropproject10.csv};
\addplot +[mark=none] table [x=epoch, y expr=1-\thisrow{main/accuracy}, col sep=comma]{ablationdata/dropproject/train_dropprojectrandom.csv};
\end{axis}
\end{tikzpicture}
\caption{\Glsfmttext{cifar10} Training Error}
\label{fig:cifar_dropproject_train_acc}
\end{subfigure}
\begin{subfigure}[b]{0.95\linewidth}
\begin{tikzpicture}
\begin{axis}[
    thick,
    \setplotcyclecat{5},
    scale only axis,
    axis x line=bottom,
    axis y line=left,
    %axis line style={thick},
    width=0.8\linewidth,
    height=0.14\textheight,
    xlabel={Epochs},
    ylabel={Test Error},
%    ymin = 0,
]
\addplot +[mark=none] table [x=epoch, y expr=1-\thisrow{validation/main/accuracy}, col sep=comma]{ablationdata/dropproject/train_standard.csv};
\addplot +[mark=none] table [x=epoch, y expr=1-\thisrow{validation/main/accuracy}, col sep=comma]{ablationdata/dropproject/train_dropout.csv};
\addplot +[mark=none] table [x=epoch, y expr=1-\thisrow{validation/main/accuracy}, col sep=comma]{ablationdata/dropproject/train_dropproject.csv};
\addplot +[mark=none] table [x=epoch, y expr=1-\thisrow{validation/main/accuracy}, col sep=comma]{ablationdata/dropproject/train_dropproject10.csv};
\addplot +[mark=none] table [x=epoch, y expr=1-\thisrow{validation/main/accuracy}, col sep=comma]{ablationdata/dropproject/train_dropprojectrandom.csv};
\end{axis}
\end{tikzpicture}
\caption{\Glsfmttext{cifar10} Test Error}
\label{fig:cifar_dropproject_test_acc}
\end{subfigure}
\caption[Dropout \vs dropproject for \glsfmttext{vgg}/\glsfmttext{cifar10}]{Training and test curves for a \glsfmttext{vgg} network on \glsfmttext{cifar10} comparing dropout to dropproject.}
\label{fig:cifar_dropproject}
\end{figure}

\Cref{fig:cifar_dropproject} compares the effect of dropout, and various forms of dropproject, on a large VGG model trained on the \gls{cifar10}~\citep{CIFAR10} dataset. Both dropout and dropproject are only applied to the two large fully-connected layers of the VGG network. With \textbf{dropproject} (DP), one random rotation matrix is generated and used for the duration of training. For \textbf{DP-Random10}, 10 random rotation matrices are generated and a single rotation matrix is randomly chosen from these for each mini-batch during training. Finally, \textbf{DP-Random} generates a random rotation matrix for each mini-batch.

Both methods have close to identical effect on training loss/error, and are drastically different than the plots of the standard network without dropout/dropproject. At test time, both methods also achieve comparable minimum error, but in the loss curve it can be seen that dropproject appears to start overfitting earlier than dropout. While the projection (and not regularization) appears to be responsible for the increased generalization and speed of training, the zeroing out of units for dropout also has a small regularization effect not seen in dropproject. None of the variants of dropproject seem to be different, indicating that the random projection itself rather than the random rotation into a different co-ordinate basis is important for the effect.

\paragraph{Dropout and Batch Normalization}
It has been observed empirically by \citet{Ioffe2015} and others that when used with batch normalization\index{batch normalization}, dropout is not as effective. In light of our understanding of dropout as being an optimization method for error surfaces with highly complex curvature, we can explain this. As explained by~\citep{martens2010deep}, an important property of second order optimization methods is `scale invariance' --- robustness to any linear rescaling of the model parameters. For example, if we are in an elliptically shaped local minima of the error surface, ideally we would want different a higher learning rate for parameters in the direction of the major axis, as compared to those in the direction of the minor axis, as RMSprop~\citep{rmsprop} attempts. When using batch normalization\index{batch normalization}, the layer-wise error surface is whitened, reducing the importance of scale-invariance in optimization, and any related optimization tricks.
% \mynote{Questions to answer:
% \begin{itemize}
% \item Do residual connections alleviate neural co-adaption?
% \item Learning identity mapping is difficult due to bad curvature, exact situation cited as example by \citet{martens2010deep}.
% \item Is learning deep networks easier than learning wide networks because of neural co-adaption?
% \end{itemize}}

\end{document}


\section{Semantic simplification}
\label{sec-comp}

The subsections \ref{sec-ss-intro}, \ref{sec-ss-model} and \ref{sec-ss-repr}
of this section have been taken from \cite{rww3}.

\subsection{Introduction}
\label{sec-ss-intro}

{\GF} is intended to express a variety of methods of human reasoning.
Though the word "reasoning" usually connotes a logical deductive process of
using facts and assertions to obtain conclusions, much of human intelligence
relies more upon observation than upon deduction.
We look at a book. The book is seen to be "green", as an immediate observation,
not as a deduction involving, say, analysis of wavelengths of light and
sensory receptors  in the eye. Similarly, humans cross streets without
conscious analysis of the traffic flow, add numbers without resorting to basic
set theory, and play chess without considering each move in terms of the
geometry of the board.

Any system which hopes to express a variety of reasoning processes, therefore
needs a method of doing purely computational tasks.
In {\GF}, the {\bf semantic interpretation mechanism}, which provides this
ability, consists of two parts:
\begin{itemize}
\item {\GF}'s {\bf semantic attachment mechanism} permits the user to define a
      ``correspondence'' between the various constants (function symbols,
      predicate constants, individual constants) of the language and
      corresponding objects of the programming language {\HG}.
\item facts about the {\HG} structure can be used directly in the proof
      via the {\bf semantic simplification mechanism}, eliminating the
      necessity of a possibly complicated deduction.
\end{itemize}
For example, obvious attachments to the function symbol $+$ and to the
individual constants $17$, $34$, $51$ would allow to conclude $17+34=51$ in
one step, instead of computing $34$ successors of $17$.
In order to explain this more clearly we first give an informal account of the
technical details.

\subsection{``Intended'' and ``computational'' models}
\label{sec-ss-model}

The declarations made by a {\GF} user specify a first order language
$L=\langle P,F,C\rangle$, where $P$ is the list of {\predconst}s, $F$ the list
of {\funconst}s, and $C$ the list of {\indconst}s (see section \ref{sec-decl}).

A model for such a language is a structure $M=\langle D,P',F',C'\rangle$ where
$D$ is a set and $P'$,$F'$ and $C'$ are lists of predicates over $D$, functions
over $D$, and individuals of $D$ such that the arities of the symbols in $P$ and
$F$ match the arities of the predicates and functions at the correspondent
positions in $P'$ and $F'$.
The idea here is that the language $L$ is used for making statements about
structures such as $M$ (what we call {\bf ``intended'' or ``standard'' model}).
In particular, when the user writes down a theory in {\GF}, he generally has
in mind some particular model for his language, and the axioms of his theory
are intended to express the properties of this particular model.

The fact that {\GF} is really a {\HG} program running in a LISP
environment, inspires the following idea: some parts of a model for a {\GF}
language can often be expressed computationally in the sense that the elements
of $D$ can be represented by s-expressions, and the predicates and functions
on $D$ can be represented by {\HG} functions and predicates.
It should then be possible to use the computational representation to aid
{\GF} deductions concerning the model.
For example, suppose the theory we are interested in, is first order number
theory, and the model that we have in mind is the set of natural numbers
together with the operations of successor, addition and multiplication.
The numerals have natural representations as {\HG} numbers, and the
functions in question have {\tt PLUS1}, {\tt PLUS}, {\tt TIMES} as their {\HG}
counterparts.
As mentioned above it should then be possible to use the computational
representation to provide swift deductions of such statements as $25+37=52$.

The semantic attachment mechanism in {\GF} allows the user to set up these
computational representations of his subject matter, and the semantic
interpretation mechanism allows to use these representations to aid deduction
in {\GF}.

With the above overview in mind, let us proceed to the details.

Given a language $L=\langle P,F,C\rangle$ and a model
$M=\langle D,P',F',C'\rangle$, we define an interpretation function $I$.
For each {\term} $t$ of $L$ in which no free variable occurs, $I(t)$ is the
individual in $D$ which $t$ denotes.
In particular we define the interpretation of an {\indconst} $c$ to be the
individual $c'$ in $D$, and where $f$ is a {\funconst}, and the interpretation
of {\term} $t_1,\ldots,t_n$ are defined, we inductively define the
interpretation of the {\term}
$f(t_1,\ldots,t_n)$ to be $f'(I(t_1),\ldots,I(t_n))$.
We may extend the interpretation function to formulas (again without free
variables) over $L$ by defining $I(w)$ to be the object {\tt TRUE} exactly
when the formula $w$ is true of the model (for a technical definition
see \cite{kleene2}).

When $f'$ is the function in a model corresponding to the {\funconst} $f$ in
$L$, we will also say that $f'$ is the interpretation of $f$, and similarly
for predconsts.

Now we define a {\bf computational model} to be an object
$K=\langle D',P'',F'',C''\rangle$, where it is understood that $D'$ is a set
of s-expressions, and $P''$, $F''$ and $C''$ are lists of {\HG} predicates,
functions and s-expressions respectively, with the appropriate restrictions on
arities.

From the extensional point of view, a computational model is for a language
just like a set-theoretic model for a language, except that we do not require
that the functions and predicates concerned be total; that is functions and
predicates may be undefined (non-terminating) for some elements
of $D'$.

We define an {\bf attachment map} $att$ from terms and formulas of $L$ into
$K$ in a manner exactly analogous to the definition of $I$ given above.

We have one last map to worry about, the map {\bf $rep$} which gives, for each
object in the domain $D'$ of the computational model $K$, the object it
represents in the domain $D$ of the model $M$.

Now we may define precisely the meaning of attachments made in the {\GF}
system: the attachment of an {\indconst} $c$ to an s-expression $c''$
signifies that $c$ and $c''$ represent the same object in the model, that is
to say, $I(c)=rep(c'')$.
Similarly, the attachment of a {\funconst} $f$ to a {\HG} function $f''$
signifies that the result of applying $f''$ to an s-expression $c''$ which
represents an individual $c'$ in the model, is a s-expression which represents
the individual $f'(c')$ in the model.
The analogous statements hold for attachments to {\predconst}s.

The above conditions are equivalent to the statement that the
diagram in figure \ref{fig-ss} commutes.

\begin{figure}[htb]
\begin{center}
\setlength{\unitlength}{0.0125in}%
\begin{picture}(288,260)(92,540)
\thicklines
\put(340,580){\oval(80,80)}
\put(160,580){\oval(80,80)}
\put(160,760){\oval(80,80)}
\put(200,580){\vector( 1, 0){100}}
\put(194,726){\vector( 1,-1){112}}
\put(160,720){\vector( 0,-1){100}}
\put(340,559){\makebox(0,0)[b]{\raisebox{0pt}[0pt][0pt]{\twlrm model}}}
\put(340,577){\makebox(0,0)[b]{\raisebox{0pt}[0pt][0pt]{\twlrm of intended}}}
\put(340,595){\makebox(0,0)[b]{\raisebox{0pt}[0pt][0pt]{\twlrm Domain }}}
\put(160,562){\makebox(0,0)[b]{\raisebox{0pt}[0pt][0pt]{\twlrm sexpr}}}
\put(160,580){\makebox(0,0)[b]{\raisebox{0pt}[0pt][0pt]{\twlrm {\HG}}}}
\put(160,752){\makebox(0,0)[b]{\raisebox{0pt}[0pt][0pt]{\twlrm Terms}}}
\put(160,770){\makebox(0,0)[b]{\raisebox{0pt}[0pt][0pt]{\twlrm {\GF}}}}
\put(250,560){\makebox(0,0)[b]{\raisebox{0pt}[0pt][0pt]{\twlrm Representation}}}
\put(280,660){\makebox(0,0)[b]{\raisebox{0pt}[0pt][0pt]{\twlrm I}}}
\put(120,660){\makebox(0,0)[b]{\raisebox{0pt}[0pt][0pt]{\twlrm attachment}}}
\end{picture}
\end{center}
\label{fig-ss}
\caption{intended model - computational model mappings}
\end{figure}

\subsection{Multiple representation functions}
\label{sec-ss-repr}

The semantic attachment mechanism allows several representation of the model
by {\HG} s-expressions to be in force at the same time.
We will seek to motivate this aspect of the semantic attachment mechanism
by means of an example: consider a theory of chess with includes a general
theory of lists as a subtheory (this subtheory would be applied in arguments
about lists of pieces, lists of game positions and so on).
The intended model of such a theory includes at least two kinds of objects:
chess positions and lists.
Lists and positions form disjoint domains in the model, though it may be
possible to build lists of chess position.
If we are going to build a computational representation of this model, we will
need to represent positions and lists by s-expressions in such a way that no
s-expression represents both a list and a position.
The natural representation of a chess position as an s-expression is as a
list of eight lists, each of which is a list of eight piece names (one of
which is "empty" or some such), and the natural representation of lists as
s-expressions is the direct representation as {\HG} lists.
This representation scheme cannot be used, since it will not be possible to
decide whether a given list of eight lists of eight piece names represents a
chess board or a list of list of pieces.
That is to say, the map $rep$ will not be well defined.
It is of course not hard to solve this problem by the use of some slightly
fancier coding, but a general solution to the problem of disambiguating
computational representations is available.
Suppose that the intended model of a {\GF} theory $T$ includes the disjoint
domains $D_1,\ldots,D_n$, and suppose further that we have a different coding
function for each of these domains.
That is we have $n$ different {\bf representation functions} $rep_i$ which map
the domain of s-expressions into domain of the model, with the property that
the range of $rep_i$ is a subset of $D_i$.
Then it is possible that a single s-expression codes two different objects
$d_i$, $d_j$ in the model, but as long as we know what coding function $rep_i$
to apply, there is no ambiguity.

Then the definition of the $att$ map may be extended to take account of the
possibility of multiple representations in the following way: the domain of
the $att$ map will still consist of the set of {\GF} terms and formulas, but
its range will now lie in the set of pairs of the form $\langle$ representation
function, s-expression $\rangle$.

The soundness condition for the $att$ map is now that, when
$att(t)=\langle rep, c'' \rangle$, we have $rep(c'')=I(t)$.
In order to specify this new more complicated $att$ map, the user of the {\GF}
system must give representation information concerning his attachments.

Specifically, each representation function must be given a name and when the
attachment to an {\indconst} is given, the name of the associate
representation function must be given as well.
Similarly, when the attachment $f''$ to a {\funconst} $f$ is specified, the
(names of the) representations of its arguments and of the value it returns
must be given, and when the attachment to a {\predconst} is specified, the
representations of its arguments must also be specified.

The significance of specifying that the representations of the arguments and
value of the attachment $f''$ to a {\funconst} $f$ are
$R_1,\ldots,R_n$ and $R_{n+1}$ respectively, is that
$R_{n+1}(f''(c''_1,\ldots,c''_n))=f'(R_1(c''_1),\ldots,R_n(c''_n))$
where $f'$ is the interpretation of $f$, whenever $c''_1$,..,$c''_n$ are
s-expressions in the domains of $R_1$,..,$R_n$.
The same holds for attachments to {\predconst}, mutatis mutandis.
Given the attachments with representation information for individual symbols,
the map $att$ on the domain of terms and formulas is defined inductively in
the obvious way: if $f$ is attached to $f''$ and the declared representation
of the arguments of $f''$ are $R_1,\ldots,R_n$ and terms $t_1$,..,$t_n$ have
attachments with representations $R_1$,..,$R_n$ then
$att(f(t_1,\ldots,t_n))=f''(att(t_1),\ldots,att(t_n))$.
Under this definition the diagram above commutes for each individual
representation function.

Note that if the representation of the attachment of any term $t_i$ does not
match that of its place in the argument list, then
$f''(att(t_1),\ldots,att(t_n))$ cannot be expected to represent the
interpretation of $f(t_1,\ldots,t_n)$.
The reason for this is that the correctness of a computation which purports to
represent a mathematical function depends on the representation of the
arguments of the function as data objects.
For example, no one would expect a floating point multiplication algorithm to
behave correctly if its arguments were encoded as integers rather than
floating point numbers.

Finally, note that the attachment map, as well as the s-expressions which
represent functions, may be partial.
The user is never required to provide an attachment for any {\GF} symbol, nor is
any attachment to a {\funconst} or {\predconst} required to be complete.

The semantic simplification mechanism will use whatever information is
available and if there will be insufficient information, it will return this
fact to the user.


%
% description: Mathematical notation used
%

\chapter{Notation}
\label{app:notation}

\section*{General math notation}

\begin{tabularx}{\textwidth}{*{2}{l}}
Symbol           & Meaning \\
\hline
$\infty$         & Infinity \\
\end{tabularx}


\documentclass[main.tex]{subfiles}
\begin{document}

\section*{Fri Jan 10 2020}

Why does the photon have two degrees of freedom?

We know that we can express
%
\begin{align}
\vec{B} = \nabla \times \vec{A} \qquad \text{and} \qquad
\vec{E} = - \vec{\nabla} \phi - \pdv{}{t} \vec{A}
\,,
\end{align}
%
and if we define the 4-vector \(A^{\mu } = (\phi , \vec{A})\) then this potential is invariant under the gauge transformation \(A_{\mu } \rightarrow A_{\mu } + \partial_{\mu } \xi\) for any function \(\xi \).

This is the defining property of electromagnetism! If we have a potential \(A_{\mu }\) we can define the covariant derivative \(D_{\mu } = \partial_{\mu } + ie A_{\mu }\): then the Lagrangian is written as
%
\begin{align}
\mathcal{L} = D_{\mu } \phi  D^{\mu } \phi^{*} - m^2 \phi^2 - F_{\mu \nu } F^{\mu \nu }
\,,
\end{align}
%
and this field theory has \(U(1)\) symmetry, \(\phi \rightarrow e^{i \xi } \phi \), which corresponds to \(A_{\mu } \rightarrow A_{\mu } + \partial_{\mu } \xi \)! This is actually the defining property of electromagnetism in the QED formulation.

Similarly, GR follows from the invariance under a certain kind od gauge transformation: in this case, the gauge transformations are \emph{diffeomorphisms} \(x \rightarrow x' (x)\).

In electromagnetism, we have the 4 component for the potential, but we can fix the gauge by setting \(\partial_{\mu }  A^{\mu } = 0\) and the residual gauge by setting \(A^{0} = 0\): then we are left with 2 degrees of freedom.

This is completely analogous to the way we found the graviton to have 2 degrees of freedom, that is, two polarizations.

Now we discuss how we actually detected the presence of gravitons.

\subsection{Interferometric GW detection}

GWs are detected interferometrically since interferometers are our most accurate way to measure distances.

The setup is a Michelson-Morley interferometer.
Call \(\Delta L\) the difference in the two paths the light takes, if \(\Delta L = n \lambda \) with \(n \in \mathbb{N} \) and \(\lambda \) begin the wavelength of light, then we have perfectly constructive interference.
If instead \(\Delta L = (n + 1/ 2) \lambda \) we have perfectly destructive interference.

In actual GW experiments we will have \(\Delta L \ll \lambda \), so we will not go from in-phase to counter-phase, but there will be a slight decrease of power, by which we will be able to detect the GW.

We take the mirrors to be suspended and free to move in the horizontal direction. So, for our purposes (to linear order in the displacement), the mirrors are free particles which move along geodesics.

So we have two mirrors at \(\vec{x}_{1} \) and \(\vec{x}_{2}\): let us say that the \(x\) axis is along their distance.

Then, their distance will be given by
%
\begin{align}
d = \int_{x_1 }^{x_2 } \dd{x} \sqrt{g_{11} }
\,.
\end{align}

In principle, the GW can change \(d\) by changing \(x_1 \) and \(x_2 \) and \emph{also} by changing the metric element \(\sqrt{g_{11} }\) in the space between them.

\begin{claim}
The effect of the change of the positions \(x_1 \) and \(x_2 \) is at most \(\mathcal{O}(h^2)\).
\end{claim}

\begin{proof}
First, let us assume that there is no GW and we have just one mirror in a fixed position in Minkowski spacetime.
Then, the initial 4-velocity of the mirror (before the arrival of the GW) is \(u^{\alpha }_{\tau =0} = [1, \vec{0}]\).

Then, the GW arrives and the mirror will follow the geodesic equation:
%
\begin{align}
\dv{ u^{\alpha }  }{\tau } + \Gamma^{\alpha }_{\beta \gamma } u^{\beta } u^{\gamma } = 0
\,.
\end{align}

We want to consider this at first order in \(h\).
The Christoffel sybols at order \(h^{0}\) are zero since in Minkowski spacetime there is no curvature.

So, if we want a nonzero first-order Christoffel term, we need to consider the zeroth order in the 4-velocity.

Then, we are left with \(u^{\alpha } = u^{\alpha }_{\tau =0}\), or
%
\begin{align}
\dv{ u^{i }}{\tau } = - \Gamma^{i }_{00}
\,,
\end{align}
%
since the \(\Gamma^{0}_{00}\) is zero. Now we can compute
%
\begin{align}
\Gamma^{i}_{00} = \frac{1}{2} \eta^{i \lambda } \qty(h_{\lambda 0, 0} + h_{\lambda 0,0} - h_{00 , \lambda }) = 0
\,,
\end{align}
%
because in our gauge \(h_{0 \mu } = 0\).
\end{proof}

A note: this is only true in our gauge. In other gauges, the result is the same but there is also a first-order contribution to the change in the distance between the mirrors from the change in their coordinate positions.

We set the mirrors at \(\vec{x}_{1}\) and \(\vec{x}_{2} = \vec{x}_{1} + T \hat{L}_{12}\), where \(\hat{L}_{12}\) is a unit vector while \(T\) is the (unperturbed) travel time (or distance, since \(c=1\)) between the two mirrors.

The worldline of the laser light is given by \(t = t_1 + \lambda \) and \(\vec{x} = \vec{x}_{1} \hat{L}_{12} \lambda \), for \(\lambda \in [0,T]\).

The photons move with \(\dd{s^2} =0\), which means
%
\begin{subequations}
\begin{align}
0 &= - \dd{t^2} + \qty(\delta_{ij} + h_{ij}) \dd{x^{i}} \dd{x^{j}}  \\
&= - \dd{t^2} + \qty(\delta_{ij} + h_{ij}) \hat{L}_{12}^{i} \hat{L}_{12}^{j} \dd{\lambda^2}
\,,
\end{align}
\end{subequations}
%
but \(\hat{L}_{12}^{i} \hat{L}_{12}^{j} \delta_{ij} = 1\) since it is a unit vector, so we get
%
\begin{align}
0 &= - \dd{t^2} + \qty(1 + \hat{L}_{12}^{i} \hat{L}_{12}^{j} h_{ij})  \dd{\lambda^2}
\,,
\end{align}
%
so we get
%
\begin{subequations}
\begin{align}
\dd{t} &= \sqrt{1 + \hat{L}_{12}^{i} \hat{L}_{12}^{j} h_{ij}} \dd{\lambda }  \\
& \approx \qty(1 + \frac{1}{2} \hat{L}_{12}^{i} \hat{L}_{12}^{j} h_{ij}) \dd{\lambda }
\,.
\end{align}
\end{subequations}

So, we need to evaluate this along the trajectory: we find
%
\begin{align}
T_{12} = \int_{0}^{T} \dd{\lambda } \qty(1 + \frac{1}{2} \hat{L}_{12}^{i} \hat{L}_{12}^{j} \sum _{r = +, \times } h_{r} e_{ij, r} \cos(k (t_1 + \lambda t) - \vec{k} \cdot \qty(\vec{x}_{1} + \hat{L}_{12} \lambda )))
\,,
\end{align}
%
which just amounts to plugging in our expressions for \(t\) and \(\vec{x}\) into the formula for \(h_{ij} (t, \vec{x})\).

We set the phase to zero for brevity, a more general consideration will include it.

The first term is the unperturbed travel time  \(T\). Then, we can bring the constants outside the integral:
%
\begin{align}
T_{12} = T + \frac{1}{2} \hat{L}_{12}^{i} \hat{L}_{12}^{j} \sum _{r = +, \times } h_{r} e_{ij, r}  \int_{0}^{T} \dd{\lambda }  \cos(k (t_1 + \lambda ) - \vec{k} \cdot \qty(\vec{x}_{1} + \hat{L}_{12} \lambda ))
\,.
\end{align}

Now, we can make an approximation which is applicable to ground-based interferometers: the \emph{short-arm} approximation. The arguments of the cosine depend on \(\lambda \), but we can assume \(k \lambda  < kT \ll 1\). This means that the travel time of the laser is much smaller than the period of the GW. Then, the cosine is approximately \(\lambda \)-independent and can be brought outside of the integral. So we find
%
\begin{align}
T_{12} = T + \frac{1}{2} \hat{L}_{12}^{i} \hat{L}_{12}^{j} \sum _{r = +, \times } h_{r} e_{ij, r} \cos(k t_1 - \vec{k} \cdot \vec{x}_{1}) T
\,.
\end{align}

Let us justify the approximation: \(k \sim \SI{100}{Hz}\), while \(L \sim \SI{4}{km}\). So, \(kT \approx 2 \pi \times \SI{4}{km} \times \SI{100}{km} / c \approx \num{8e-3}\), which is small.

We have three points in the interferometer: \(\vec{x}_{1}\) is the beamsplitter, \(\vec{x}_{2}\) and \(\vec{x}_{3}\) are the ends of the arms. We need to compute the difference between the times \(\vec{x}_{1} \rightarrow \vec{x}_{2} \rightarrow \vec{x}_{1}\) and \(\vec{x}_{1} \rightarrow \vec{x}_{3} \rightarrow \vec{x}_{1}\), repeated a few hundred times (the number of bounces of the beam is called the \emph{finesse factor}). Let us just compute \(\Delta T = T_{12} - T_{13} \) for a single ``bounce'', to see what the effect looks like. We actually compute \(\Delta T / T\), in order to see what fraction of difference of travel time we are looking at. It will be given by
%
\begin{align}
\frac{\Delta T}{T} = \cos(kt - \vec{k} \vec{x}) \sum _{r = + , \times } h_{r} e_{ij, r} \frac{\hat{L}_{12}^{i}\hat{L}_{12}^{j} - \hat{L}_{13}^{i}\hat{L}_{13}^{j}}{2}
\,.
\end{align}

Writing it this way is convenient, since we do not need to consider a single geometry: there are \SI{90}{\degree} interferometers such as LIGO, and \SI{60}{\degree} ones such as LISA, or the Einstein telescope.

Let us say that
%
\begin{align}
\hat{L}_{12} = [\cos \alpha , \sin \alpha , 0]^{\top} \qquad \text{and} \qquad \hat{L}_{13} = [\cos(\alpha + \pi /2), \sin(\alpha + \pi /2) , 0]^{\top}
\,.
\end{align}

We need to compute the following products:
%
\begin{subequations}
\begin{align}
\hat{L}_{12}^{\top} e_{+} \hat{L}_{12} &= \cos^2\alpha - \sin^2\alpha = \cos(2 \alpha )
\\
\hat{L}_{12}^{\top} e_{ \times } \hat{L}_{12} &= 2 \sin \alpha \cos \alpha = \sin(2 \alpha )
\,,
\end{align}
\end{subequations}
%
and similarly for \(\hat{L}_{13} \), where we can substitute \(\alpha \rightarrow \alpha + \pi /2\). We get:
%
\begin{subequations}
\begin{align}
\frac{\Delta T}{T} &= \frac{1}{2} \cos(kt - \vec{k} \cdot \vec{x})
h_{+}\qty(  \qty(\cos(2 \alpha ) - \cos(2 \alpha  + \pi ))) +
h_{ \times } \qty(  \qty(\sin(2 \alpha ) - \sin(2 \alpha  + \pi )))  \\
&= \cos(kt - \vec{k} \cdot \vec{x}) \qty(
h_{+}\cos(2 \alpha ) +
h_{\times} \sin(2 \alpha )) \marginnote{Simplified factors of 2}
\,,
\end{align}
\end{subequations}
%
where we used the fact that \(\cos(x + \pi ) = - \cos(x)\) and similarly for the sine.

So, this is a \(\mathcal{O}(h)\) effect, with \(h \sim \num{e-21}\), which means \(\Delta L  \sim \SI{4}{km} \times \num{e-21} \sim \SI{4e-18}{m}\).

The fact that we have many photons in the laser helps, and the fact that they bounce several times also does.

\end{document}


\documentclass[12pt]{beamer}

\input{inc.tex}

\usepackage{mathrsfs}
\usetikzlibrary{positioning,calc}
\usepackage{tikz-cd}
\usepackage{ebproof}
\newcommand\M{\text{M}(X\uplus X^{-1})}
\newcommand\G{\mathscr{G}(X)}
\renewcommand\F{\mathscr{F}(X)}
\newcommand\V{\mathbb{V}}
\newcommand\E{\mathbb{E}}
\newcommand\D{\mathbb{D}}
\newcommand\bet{\rightarrow_\beta}
\newcommand\beq{=_\beta}
\newcommand\Rel{\text{Rel}}
\newcommand\Set{\text{Set}}
\newcommand\Oper{\text{Oper}}
\newcommand\Inv{\text{Inv}}
\newcommand\Pol{\text{Pol}}
\renewcommand\P{\mathscr{P}}
\newcommand\ar{\text{ar}}
\newcommand\arf{\text{ar}}
\newcommand\csp{\text{CSP}}
\renewcommand\C{\mathscr{C}}
\newcommand\fset{\text{FinSet}}
\newcommand\ffun{\text{FinFun}}
\newcommand\im{\text{Im }}
\newcommand\sem[1]{\llbracket {#1} \rrbracket}
\newcommand\ext{\text{Ext}}
\newcommand\fns{\mathscr{F}}
\newcommand\brk[1]{[ {#1} ]}
\newcommand\psh{\text{PSh}}
\newcommand\sh{\text{Sh}}
\newcommand\shc{\text{Sh}_\text{can}}
\newcommand{\tproof}[1]{{\scantokens{\begin{prooftree}#1\end{prooftree}}}}
\newcommand{\cf}{\mathbb{F}}
\newcommand{\tsigma}{\widetilde{\Sigma}}
\newcommand{\tgamma}{\widetilde{\Gamma}}
\newcommand{\tlambda}{\widetilde{\Lambda}}
\newcommand{\colim}{\text{colim}}
\newcommand{\fcsp}{\text{FCSP}}
\newcommand{\op}{\text{op}}
\newcommand{\tf}{\widetilde{F}}
\newcommand{\tg}{\widetilde{G}}
\newcommand{\tH}{\widetilde{H}}
\newcommand{\ml}[1]{\langle{#1}\rangle}

%  ____                                    ____             __ _
% | __ )  ___  __ _ _ __ ___   ___ _ __   / ___|___  _ __  / _(_) __ _
% |  _ \ / _ \/ _` | '_ ` _ \ / _ \ '__| | |   / _ \| '_ \| |_| |/ _` |
% | |_) |  __/ (_| | | | | | |  __/ |    | |__| (_) | | | |  _| | (_| |
% |____/ \___|\__,_|_| |_| |_|\___|_|     \____\___/|_| |_|_| |_|\__, |
%                                                                |___/
\usepackage{etoolbox}
\makeatletter
\preto{\@verbatim}{\topsep=0pt \partopsep=0pt }
\makeatother

% Le logo
\logo{\includegraphics[scale=0.07]{logo.jpg}}

% Au début de chaque section
\AtBeginSection[]{
	\begin{frame}
		\frametitle{\textbf{\insertsectionhead}}
        \framesubtitle{Section \thesection}
		\tableofcontents[currentsection,hideothersubsections]
	\end{frame}
}

% Au début de chaque sous-section
\AtBeginSubsection[]{
	\begin{frame}
        \frametitle{\textbf{Section} \thesection-\thesubsection}
        \begin{block}{Subsection \thesubsection}
			\Large \textbf{\insertsubsectionhead}
		\end{block}
	\end{frame}
}

%Put a nice theme
\usetheme{Berkeley}
\beamertemplatenavigationsymbolsempty
\makeatletter
\g@addto@macro\scriptsize{%
\setlength\abovedisplayskip{1pt}
\setlength\belowdisplayskip{1pt}
\setlength\abovedisplayshortskip{1pt}
\setlength\belowdisplayshortskip{1pt}
}
\g@addto@macro\normalsize{%
\setlength\abovedisplayskip{1pt}
\setlength\belowdisplayskip{1pt}
\setlength\abovedisplayshortskip{1pt}
\setlength\belowdisplayshortskip{1pt}
}
\makeatother
\setbeamertemplate{itemize item}[square]
\setbeamertemplate{itemize subitem}[triangle]

%  ____                                        _
% |  _ \  ___   ___ _   _ _ __ ___   ___ _ __ | |_
% | | | |/ _ \ / __| | | | '_ ` _ \ / _ \ '_ \| __|
% | |_| | (_) | (__| |_| | | | | | |  __/ | | | |_
% |____/ \___/ \___|\__,_|_| |_| |_|\___|_| |_|\__|
%
\author{Antonin Reitz \and Luc Chabassier}
\title[Hex Mesh]
{Hexahedral Mesh Structure Visualization and Evaluation\\
    {\normalsize Paper presentation}}
\institute{\begin{center}MPRI
\end{center}}

\begin{document}
\addtobeamertemplate{block begin}{\setlength\abovedisplayskip{0pt}}

\begin{frame}
    \maketitle
\end{frame}

\begin{frame}
    \tableofcontents
\end{frame}

\section{Introduction}

\begin{frame}[fragile]
  \frametitle{Introduction}
  \includegraphics[height=0.5\textheight]{./images/vis-3d-ex1-sheets.png}
\end{frame}
% state of the art
% improve methods (circular base components case)
% + first norm to evaluate such methods

\section{Definitions}

\begin{frame}[fragile]
  \frametitle{Hexahedral mesh}

  \begin{center}\begin{tikzpicture}[scale=0.8]
    \draw[color=gray] (0,0) -- (4,1) -- (4,7);
    \draw[color=gray] (4,1) -- (10,1);
    \draw[color=gray] (0,3) -- (4,4);
    \draw[color=gray] (2,0.5) -- (2,6.5);
    \draw[color=gray] (2,0.5) -- (8,0.5);
    \draw[color=gray] (3,0) -- (7,1);
    \draw[color=gray] (7,1) -- (7,7);
    \draw[color=gray] (4,4) -- (10,4);
    \draw[color=gray,dashed] (5,0.5) -- (5,6.5);
    \draw[color=gray,dashed] (2,3.5) -- (8,3.5);
    \draw[color=gray,dashed] (3,3) -- (7,4);

    \draw (0,0) -- (6,0) -- (6,6) -- (0,6) -- cycle;
    \draw (6,6) -- (10,7) -- (4,7) -- (0,6);
    \draw (6,0) -- (10,1) -- (10,7);
    \draw (3,0) -- (3,6);
    \draw (0,3) -- (6,3);
    \draw (8,0.5) -- (8,6.5);
    \draw (6,3) -- (10,4);
    \draw (3,6) -- (7,7);
    \draw (2,6.5) -- (8,6.5);
  \end{tikzpicture}\end{center}
\end{frame}

\begin{frame}[fragile]
  \frametitle{Basic vocabulary}

  \begin{center}\begin{tikzpicture}[scale=0.8]
    \draw[color=gray] (0,0) -- (4,1) -- (4,7);
    \draw[color=gray] (4,1) -- (10,1);
    \draw[color=gray] (0,3) -- (4,4);
    \draw[color=gray] (2,0.5) -- (2,6.5);
    \draw[color=gray] (2,0.5) -- (8,0.5);
    \draw[color=gray] (3,0) -- (7,1);
    \draw[color=gray] (7,1) -- (7,7);
    \draw[color=gray] (4,4) -- (10,4);
    \draw[color=gray,dashed] (5,0.5) -- (5,6.5);
    \draw[color=gray,dashed] (2,3.5) -- (8,3.5);
    \draw[color=gray,dashed] (3,3) -- (7,4);

    \draw (0,0) -- (6,0) -- (6,6) -- (0,6) -- cycle;
    \draw (6,6) -- (10,7) -- (4,7) -- (0,6);
    \draw (6,0) -- (10,1) -- (10,7);
    \draw (3,0) -- (3,6);
    \draw (0,3) -- (6,3);
    \draw (8,0.5) -- (8,6.5);
    \draw (6,3) -- (10,4);
    \draw (3,6) -- (7,7);
    \draw (2,6.5) -- (8,6.5);

    \fill[color=blue,fill=blue] (10,7) circle (0.1);
    \node[anchor=south east,color=blue] at (10,7) {Vertex};
    \draw[thick,color=green] (2,6.5) -- (4,7);
    \node[anchor=south east,color=green] at (3,6.75) {Edge};
    \fill[color=cyan,fill=cyan,opacity=0.5] (0,6) -- (3,6) -- (3,3) -- (0,3) -- cycle;
    \node[anchor=north,color=cyan] at (1.5,3) {Face};
    \fill[color=red,fill=red,opacity=0.5] (3,0) -- (6,0) -- (8,0.5) -- (8,3.5) -- (5,3.5) -- (3,3) -- cycle;
    \node[anchor=north,color=red] at (4.5,0) {Cell};
  \end{tikzpicture}\end{center}
\end{frame}

\begin{frame}[fragile]
  \frametitle{Singularities}

  \begin{center}\begin{tikzpicture}[scale=0.8]
    \draw[thick,color=blue,opacity=0.5] (0,0) -- (4,1) -- (4,7);
    \draw[thick,color=blue,opacity=0.5] (4,1) -- (10,1);
    \draw[color=gray] (0,3) -- (4,4);
    \draw[color=gray] (2,0.5) -- (2,6.5);
    \draw[color=gray] (2,0.5) -- (8,0.5);
    \draw[color=gray] (3,0) -- (7,1);
    \draw[color=gray] (7,1) -- (7,7);
    \draw[color=gray] (4,4) -- (10,4);
    \draw[color=gray,dashed] (5,0.5) -- (5,6.5);
    \draw[color=gray,dashed] (2,3.5) -- (8,3.5);
    \draw[color=gray,dashed] (3,3) -- (7,4);

    \draw[thick,color=blue] (0,0) -- (6,0) -- (6,6) -- (0,6) -- cycle;
    \draw[thick,color=blue] (6,6) -- (10,7) -- (4,7) -- (0,6);
    \draw[thick,color=blue] (6,0) -- (10,1) -- (10,7);
    \draw (3,0) -- (3,6);
    \draw (0,3) -- (6,3);
    \draw (8,0.5) -- (8,6.5);
    \draw (6,3) -- (10,4);
    \draw (3,6) -- (7,7);
    \draw (2,6.5) -- (8,6.5);

    \fill[color=green,fill=green,opacity=0.8] (0,0) circle (0.1);
    \fill[color=green,fill=green,opacity=0.8] (6,0) circle (0.1);
    \fill[color=green,fill=green,opacity=0.8] (0,6) circle (0.1);
    \fill[color=green,fill=green,opacity=0.8] (6,6) circle (0.1);
    \fill[color=green,fill=green,opacity=0.5] (4,1) circle (0.1);
    \fill[color=green,fill=green,opacity=0.8] (10,1) circle (0.1);
    \fill[color=green,fill=green,opacity=0.8] (4,7) circle (0.1);
    \fill[color=green,fill=green,opacity=0.8] (10,7) circle (0.1);
  \end{tikzpicture}\end{center}
\end{frame}

\begin{frame}[fragile]
  \frametitle{A more complicated example}

  \begin{center}\begin{tikzpicture}[scale=0.4,%
      x={(1cm,-0.05cm)},%
      y={(0.1cm,0.5cm)},%
      z={(0cm,5.5cm)}]
      % % Bottom level
      \coordinate (root) at (0,0,0);
      \draw ($ (-8,4,0) + (root) $) -- ($ (0,0,0) + (root) $) -- ($ (8,4,0) + (root) $);
      \draw[gray] ($ (8,4,0) + (root) $) -- ($ (4,10,0) + (root) $) -- ($ (-4,10,0) + (root) $) -- ($ (-8,4,0) + (root) $);
      \draw[gray,dashed] ($ (0,5.0,0) + (root) $) -- ($ (4,2,0) + (root) $);
      \draw[gray,dashed] ($ (0,5.0,0) + (root) $) -- ($ (6,7.0,0) + (root) $);
      \draw[gray,dashed] ($ (0,5.0,0) + (root) $) -- ($ (0,10,0) + (root) $);
      \draw[gray,dashed] ($ (0,5.0,0) + (root) $) -- ($ (-6,7.0,0) + (root) $);
      \draw[gray,dashed] ($ (0,5.0,0) + (root) $) -- ($ (-4,2,0) + (root) $);
      \draw[gray,dashed] ($ (6,3.0,0) + (root) $) -- ($ (3,6,0) + (root) $) -- ($ (2,10,0) + (root) $);
      \draw[gray,dashed] ($ (5,8.5,0) + (root) $) -- ($ (0,7.5,0) + (root) $) -- ($ (-5,8.5,0) + (root) $);
      \draw[gray,dashed] ($ (-2,10,0) + (root) $) -- ($ (-3,6,0) + (root) $) -- ($ (-6,3.0,0) + (root) $);
      \draw[gray,dashed] ($ (-7,5.5,0) + (root) $) -- ($ (-2,3.5,0) + (root) $) -- ($ (2,1.0,0) + (root) $);
      \draw[gray,dashed] ($ (-2,1.0,0) + (root) $) -- ($ (2,3.5,0) + (root) $) -- ($ (7,5.5,0) + (root) $);

      % % Middle level
      \coordinate (root) at (0,0,1);
      \draw ($ (-8,4,0) + (root) $) -- ($ (0,0,0) + (root) $) -- ($ (8,4,0) + (root) $);
      \draw[gray] ($ (8,4,0) + (root) $) -- ($ (4,10,0) + (root) $) -- ($ (-4,10,0) + (root) $) -- ($ (-8,4,0) + (root) $);
      \draw[gray,dashed] ($ (0,5.0,0) + (root) $) -- ($ (4,2,0) + (root) $);
      \draw[gray,dashed] ($ (0,5.0,0) + (root) $) -- ($ (6,7.0,0) + (root) $);
      \draw[gray,dashed] ($ (0,5.0,0) + (root) $) -- ($ (0,10,0) + (root) $);
      \draw[gray,dashed] ($ (0,5.0,0) + (root) $) -- ($ (-6,7.0,0) + (root) $);
      \draw[gray,dashed] ($ (0,5.0,0) + (root) $) -- ($ (-4,2,0) + (root) $);
      \draw[gray,dashed] ($ (6,3.0,0) + (root) $) -- ($ (3,6,0) + (root) $) -- ($ (2,10,0) + (root) $);
      \draw[gray,dashed] ($ (5,8.5,0) + (root) $) -- ($ (0,7.5,0) + (root) $) -- ($ (-5,8.5,0) + (root) $);
      \draw[gray,dashed] ($ (-2,10,0) + (root) $) -- ($ (-3,6,0) + (root) $) -- ($ (-6,3.0,0) + (root) $);
      \draw[gray,dashed] ($ (-7,5.5,0) + (root) $) -- ($ (-2,3.5,0) + (root) $) -- ($ (2,1.0,0) + (root) $);
      \draw[gray,dashed] ($ (-2,1.0,0) + (root) $) -- ($ (2,3.5,0) + (root) $) -- ($ (7,5.5,0) + (root) $);

      % % Vertical lines
      \coordinate (root) at (0,0,2);
      \draw[gray] (-7,5.5,0) -- ($ (-7,5.5,0) + (root) $);
      \draw[gray,dashed] (-4.5,4.5,0) -- ($ (-4.5,4.5,0) + (root) $);
      \draw[gray,dashed] (-2,3.5,0) -- ($ (-2,3.5,0) + (root) $);
      \draw[gray,dashed] (0,2.25,0) -- ($ (0,2.25,0) + (root) $);
      \draw[gray,dashed] (2,3.5,0) -- ($ (2,3.5,0) + (root) $);
      \draw[gray,dashed] (4.5,4.5,0) -- ($ (4.5,4.5,0) + (root) $);
      \draw[gray] (7,5.5,0) -- ($ (7,5.5,0) + (root) $);
      \draw[gray] (-6,7.0,0) -- ($ (-6,7.0,0) + (root) $);
      \draw[gray,dashed] (-3,6,0) -- ($ (-3,6,0) + (root) $);
      \draw[gray,dashed] (0,5.0,0) -- ($ (0,5.0,0) + (root) $);
      \draw[gray,dashed] (3,6,0) -- ($ (3,6,0) + (root) $);
      \draw[gray] (6,7.0,0) -- ($ (6,7.0,0) + (root) $);
      \draw[gray] (-5,8.5,0) -- ($ (-5,8.5,0) + (root) $);
      \draw[gray,dashed] (-2.5,8,0) -- ($ (-2.5,8,0) + (root) $);
      \draw[gray,dashed] (0,7.5,0) -- ($ (0,7.5,0) + (root) $);
      \draw[gray,dashed] (2.5,8,0) -- ($ (2.5,8,0) + (root) $);
      \draw[gray] (5,8.5,0) -- ($ (5,8.5,0) + (root) $);
      \draw[gray] (-4,10,0) -- ($ (-4,10,0) + (root) $);
      \draw[gray,dashed] (-2,10,0) -- ($ (-2,10,0) + (root) $);
      \draw[gray,dashed] (0,10,0) -- ($ (0,10,0) + (root) $);
      \draw[gray,dashed] (2,10,0) -- ($ (2,10,0) + (root) $);
      \draw[gray] (4,10,0) -- ($ (4,10,0) + (root) $);
      \draw (-8,4,0) -- ($ (-8,4,0) + (root) $);
      \draw (-6,3.0,0) -- ($ (-6,3.0,0) + (root) $);
      \draw (-4,2,0) -- ($ (-4,2,0) + (root) $);
      \draw (-2,1.0,0) -- ($ (-2,1.0,0) + (root) $);
      \draw (0,0,0) -- ($ (0,0,0) + (root) $);
      \draw (2,1.0,0) -- ($ (2,1.0,0) + (root) $);
      \draw (4,2,0) -- ($ (4,2,0) + (root) $);
      \draw (6,3.0,0) -- ($ (6,3.0,0) + (root) $);
      \draw (8,4,0) -- ($ (8,4,0) + (root) $);

      % Top level
      \draw ($ (-8,4,0) + (root) $) -- ($ (0,0,0) + (root) $) -- ($ (8,4,0) + (root) $);
      \draw ($ (8,4,0) + (root) $) -- ($ (4,10,0) + (root) $) -- ($ (-4,10,0) + (root) $) -- ($ (-8,4,0) + (root) $);
      \draw ($ (0,5.0,0) + (root) $) -- ($ (4,2,0) + (root) $);
      \draw ($ (0,5.0,0) + (root) $) -- ($ (6,7.0,0) + (root) $);
      \draw ($ (0,5.0,0) + (root) $) -- ($ (0,10,0) + (root) $);
      \draw ($ (0,5.0,0) + (root) $) -- ($ (-6,7.0,0) + (root) $);
      \draw ($ (0,5.0,0) + (root) $) -- ($ (-4,2,0) + (root) $);
      \draw ($ (6,3.0,0) + (root) $) -- ($ (3,6,0) + (root) $) -- ($ (2,10,0) + (root) $);
      \draw ($ (5,8.5,0) + (root) $) -- ($ (0,7.5,0) + (root) $) -- ($ (-5,8.5,0) + (root) $);
      \draw ($ (-2,10,0) + (root) $) -- ($ (-3,6,0) + (root) $) -- ($ (-6,3.0,0) + (root) $);
      \draw ($ (-7,5.5,0) + (root) $) -- ($ (-2,3.5,0) + (root) $) -- ($ (2,1.0,0) + (root) $);
      \draw ($ (-2,1.0,0) + (root) $) -- ($ (2,3.5,0) + (root) $) -- ($ (7,5.5,0) + (root) $);
  \end{tikzpicture}\end{center}
\end{frame}

\begin{frame}[fragile]
  \frametitle{A more complicated example : singularities}

  \begin{center}\begin{tikzpicture}[scale=0.4,%
      x={(1cm,-0.05cm)},%
      y={(0.1cm,0.5cm)},%
      z={(0cm,5.5cm)}]
      % % Bottom level
      \coordinate (root) at (0,0,0);
      \draw[thick,blue] ($ (-8,4,0) + (root) $) -- ($ (0,0,0) + (root) $) -- ($ (8,4,0) + (root) $);
      \draw[thick,blue,opacity=0.5] ($ (8,4,0) + (root) $) -- ($ (4,10,0) + (root) $) -- ($ (-4,10,0) + (root) $) -- ($ (-8,4,0) + (root) $);
      \draw[gray,dashed] ($ (0,5.0,0) + (root) $) -- ($ (4,2,0) + (root) $);
      \draw[gray,dashed] ($ (0,5.0,0) + (root) $) -- ($ (6,7.0,0) + (root) $);
      \draw[gray,dashed] ($ (0,5.0,0) + (root) $) -- ($ (0,10,0) + (root) $);
      \draw[gray,dashed] ($ (0,5.0,0) + (root) $) -- ($ (-6,7.0,0) + (root) $);
      \draw[gray,dashed] ($ (0,5.0,0) + (root) $) -- ($ (-4,2,0) + (root) $);
      \draw[gray,dashed] ($ (6,3.0,0) + (root) $) -- ($ (3,6,0) + (root) $) -- ($ (2,10,0) + (root) $);
      \draw[gray,dashed] ($ (5,8.5,0) + (root) $) -- ($ (0,7.5,0) + (root) $) -- ($ (-5,8.5,0) + (root) $);
      \draw[gray,dashed] ($ (-2,10,0) + (root) $) -- ($ (-3,6,0) + (root) $) -- ($ (-6,3.0,0) + (root) $);
      \draw[gray,dashed] ($ (-7,5.5,0) + (root) $) -- ($ (-2,3.5,0) + (root) $) -- ($ (2,1.0,0) + (root) $);
      \draw[gray,dashed] ($ (-2,1.0,0) + (root) $) -- ($ (2,3.5,0) + (root) $) -- ($ (7,5.5,0) + (root) $);

      % % Middle level
      \coordinate (root) at (0,0,1);
      \draw ($ (-8,4,0) + (root) $) -- ($ (0,0,0) + (root) $) -- ($ (8,4,0) + (root) $);
      \draw[gray] ($ (8,4,0) + (root) $) -- ($ (4,10,0) + (root) $) -- ($ (-4,10,0) + (root) $) -- ($ (-8,4,0) + (root) $);
      \draw[gray,dashed] ($ (0,5.0,0) + (root) $) -- ($ (4,2,0) + (root) $);
      \draw[gray,dashed] ($ (0,5.0,0) + (root) $) -- ($ (6,7.0,0) + (root) $);
      \draw[gray,dashed] ($ (0,5.0,0) + (root) $) -- ($ (0,10,0) + (root) $);
      \draw[gray,dashed] ($ (0,5.0,0) + (root) $) -- ($ (-6,7.0,0) + (root) $);
      \draw[gray,dashed] ($ (0,5.0,0) + (root) $) -- ($ (-4,2,0) + (root) $);
      \draw[gray,dashed] ($ (6,3.0,0) + (root) $) -- ($ (3,6,0) + (root) $) -- ($ (2,10,0) + (root) $);
      \draw[gray,dashed] ($ (5,8.5,0) + (root) $) -- ($ (0,7.5,0) + (root) $) -- ($ (-5,8.5,0) + (root) $);
      \draw[gray,dashed] ($ (-2,10,0) + (root) $) -- ($ (-3,6,0) + (root) $) -- ($ (-6,3.0,0) + (root) $);
      \draw[gray,dashed] ($ (-7,5.5,0) + (root) $) -- ($ (-2,3.5,0) + (root) $) -- ($ (2,1.0,0) + (root) $);
      \draw[gray,dashed] ($ (-2,1.0,0) + (root) $) -- ($ (2,3.5,0) + (root) $) -- ($ (7,5.5,0) + (root) $);

      % % Vertical lines
      \coordinate (root) at (0,0,2);
      \draw[gray] (-7,5.5,0) -- ($ (-7,5.5,0) + (root) $);
      \draw[gray,dashed] (-4.5,4.5,0) -- ($ (-4.5,4.5,0) + (root) $);
      \draw[gray,dashed] (-2,3.5,0) -- ($ (-2,3.5,0) + (root) $);
      \draw[gray,dashed] (0,2.25,0) -- ($ (0,2.25,0) + (root) $);
      \draw[gray,dashed] (2,3.5,0) -- ($ (2,3.5,0) + (root) $);
      \draw[gray,dashed] (4.5,4.5,0) -- ($ (4.5,4.5,0) + (root) $);
      \draw[gray] (7,5.5,0) -- ($ (7,5.5,0) + (root) $);
      \draw[gray] (-6,7.0,0) -- ($ (-6,7.0,0) + (root) $);
      \draw[gray,dashed] (-3,6,0) -- ($ (-3,6,0) + (root) $);
      \draw[thick,blue,opacity=0.5] (0,5.0,0) -- ($ (0,5.0,0) + (root) $);
      \draw[gray,dashed] (3,6,0) -- ($ (3,6,0) + (root) $);
      \draw[gray] (6,7.0,0) -- ($ (6,7.0,0) + (root) $);
      \draw[gray] (-5,8.5,0) -- ($ (-5,8.5,0) + (root) $);
      \draw[gray,dashed] (-2.5,8,0) -- ($ (-2.5,8,0) + (root) $);
      \draw[gray,dashed] (0,7.5,0) -- ($ (0,7.5,0) + (root) $);
      \draw[gray,dashed] (2.5,8,0) -- ($ (2.5,8,0) + (root) $);
      \draw[gray] (5,8.5,0) -- ($ (5,8.5,0) + (root) $);
      \draw[thick,blue,opacity=0.5] (-4,10,0) -- ($ (-4,10,0) + (root) $);
      \draw[gray,dashed] (-2,10,0) -- ($ (-2,10,0) + (root) $);
      \draw[gray,dashed] (0,10,0) -- ($ (0,10,0) + (root) $);
      \draw[gray,dashed] (2,10,0) -- ($ (2,10,0) + (root) $);
      \draw[thick,blue,opacity=0.5] (4,10,0) -- ($ (4,10,0) + (root) $);
      \draw[thick,blue] (-8,4,0) -- ($ (-8,4,0) + (root) $);
      \draw (-6,3.0,0) -- ($ (-6,3.0,0) + (root) $);
      \draw (-4,2,0) -- ($ (-4,2,0) + (root) $);
      \draw (-2,1.0,0) -- ($ (-2,1.0,0) + (root) $);
      \draw[thick,blue] (0,0,0) -- ($ (0,0,0) + (root) $);
      \draw (2,1.0,0) -- ($ (2,1.0,0) + (root) $);
      \draw (4,2,0) -- ($ (4,2,0) + (root) $);
      \draw (6,3.0,0) -- ($ (6,3.0,0) + (root) $);
      \draw[thick,blue] (8,4,0) -- ($ (8,4,0) + (root) $);

      % Top level
      \draw[thick,blue] ($ (-8,4,0) + (root) $) -- ($ (0,0,0) + (root) $) -- ($ (8,4,0) + (root) $);
      \draw[thick,blue] ($ (8,4,0) + (root) $) -- ($ (4,10,0) + (root) $) -- ($ (-4,10,0) + (root) $) -- ($ (-8,4,0) + (root) $);
      \draw ($ (0,5.0,0) + (root) $) -- ($ (4,2,0) + (root) $);
      \draw ($ (0,5.0,0) + (root) $) -- ($ (6,7.0,0) + (root) $);
      \draw ($ (0,5.0,0) + (root) $) -- ($ (0,10,0) + (root) $);
      \draw ($ (0,5.0,0) + (root) $) -- ($ (-6,7.0,0) + (root) $);
      \draw ($ (0,5.0,0) + (root) $) -- ($ (-4,2,0) + (root) $);
      \draw ($ (6,3.0,0) + (root) $) -- ($ (3,6,0) + (root) $) -- ($ (2,10,0) + (root) $);
      \draw ($ (5,8.5,0) + (root) $) -- ($ (0,7.5,0) + (root) $) -- ($ (-5,8.5,0) + (root) $);
      \draw ($ (-2,10,0) + (root) $) -- ($ (-3,6,0) + (root) $) -- ($ (-6,3.0,0) + (root) $);
      \draw ($ (-7,5.5,0) + (root) $) -- ($ (-2,3.5,0) + (root) $) -- ($ (2,1.0,0) + (root) $);
      \draw ($ (-2,1.0,0) + (root) $) -- ($ (2,3.5,0) + (root) $) -- ($ (7,5.5,0) + (root) $);

      % Singular vertices
      \fill[color=green,fill=green,opacity=0.8] (0,0,0) ellipse (0.2 and 0.4);
      \fill[color=green,fill=green,opacity=0.8] (-8,4,0) ellipse (0.2 and 0.4);
      \fill[color=green,fill=green,opacity=0.8] (8,4,0) ellipse (0.2 and 0.4);
      \fill[color=green,fill=green,opacity=0.5] (0,5,0) ellipse (0.2 and 0.4);
      \fill[color=green,fill=green,opacity=0.5] (-4,10,0) ellipse (0.2 and 0.4);
      \fill[color=green,fill=green,opacity=0.5] (4,10,0) ellipse (0.2 and 0.4);
      \fill[color=green,fill=green,opacity=0.8] (0,0,2) ellipse (0.2 and 0.4);
      \fill[color=green,fill=green,opacity=0.8] (-8,4,2) ellipse (0.2 and 0.4);
      \fill[color=green,fill=green,opacity=0.8] (8,4,2) ellipse (0.2 and 0.4);
      \fill[color=green,fill=green,opacity=0.5] (0,5,2) ellipse (0.2 and 0.4);
      \fill[color=green,fill=green,opacity=0.5] (-4,10,2) ellipse (0.2 and 0.4);
      \fill[color=green,fill=green,opacity=0.5] (4,10,2) ellipse (0.2 and 0.4);
  \end{tikzpicture}\end{center}
\end{frame}

\begin{frame}[fragile]
  \frametitle{Separation surface}

  \begin{center}\begin{tikzpicture}[scale=0.4,%
      x={(1cm,-0.05cm)},%
      y={(0.1cm,0.5cm)},%
      z={(0cm,5.5cm)}]
      % % Bottom level
      \coordinate (root) at (0,0,0);
      \draw[thick,blue] ($ (-8,4,0) + (root) $) -- ($ (0,0,0) + (root) $) -- ($ (8,4,0) + (root) $);
      \draw[thick,blue,opacity=0.5] ($ (8,4,0) + (root) $) -- ($ (4,10,0) + (root) $) -- ($ (-4,10,0) + (root) $) -- ($ (-8,4,0) + (root) $);
      \draw[gray,dashed] ($ (0,5.0,0) + (root) $) -- ($ (4,2,0) + (root) $);
      \draw[gray,dashed] ($ (0,5.0,0) + (root) $) -- ($ (6,7.0,0) + (root) $);
      \draw[gray,dashed] ($ (0,5.0,0) + (root) $) -- ($ (0,10,0) + (root) $);
      \draw[gray,dashed] ($ (0,5.0,0) + (root) $) -- ($ (-6,7.0,0) + (root) $);
      \draw[gray,dashed] ($ (0,5.0,0) + (root) $) -- ($ (-4,2,0) + (root) $);
      \draw[gray,dashed] ($ (6,3.0,0) + (root) $) -- ($ (3,6,0) + (root) $) -- ($ (2,10,0) + (root) $);
      \draw[gray,dashed] ($ (5,8.5,0) + (root) $) -- ($ (0,7.5,0) + (root) $) -- ($ (-5,8.5,0) + (root) $);
      \draw[gray,dashed] ($ (-2,10,0) + (root) $) -- ($ (-3,6,0) + (root) $) -- ($ (-6,3.0,0) + (root) $);
      \draw[gray,dashed] ($ (-7,5.5,0) + (root) $) -- ($ (-2,3.5,0) + (root) $) -- ($ (2,1.0,0) + (root) $);
      \draw[gray,dashed] ($ (-2,1.0,0) + (root) $) -- ($ (2,3.5,0) + (root) $) -- ($ (7,5.5,0) + (root) $);

      % % Middle level
      \coordinate (root) at (0,0,1);
      \draw ($ (-8,4,0) + (root) $) -- ($ (0,0,0) + (root) $) -- ($ (8,4,0) + (root) $);
      \draw[gray] ($ (8,4,0) + (root) $) -- ($ (4,10,0) + (root) $) -- ($ (-4,10,0) + (root) $) -- ($ (-8,4,0) + (root) $);
      \draw[gray,dashed] ($ (0,5.0,0) + (root) $) -- ($ (4,2,0) + (root) $);
      \draw[gray,dashed] ($ (0,5.0,0) + (root) $) -- ($ (6,7.0,0) + (root) $);
      \draw[gray,dashed] ($ (0,5.0,0) + (root) $) -- ($ (0,10,0) + (root) $);
      \draw[gray,dashed] ($ (0,5.0,0) + (root) $) -- ($ (-6,7.0,0) + (root) $);
      \draw[gray,dashed] ($ (0,5.0,0) + (root) $) -- ($ (-4,2,0) + (root) $);
      \draw[gray,dashed] ($ (6,3.0,0) + (root) $) -- ($ (3,6,0) + (root) $) -- ($ (2,10,0) + (root) $);
      \draw[gray,dashed] ($ (5,8.5,0) + (root) $) -- ($ (0,7.5,0) + (root) $) -- ($ (-5,8.5,0) + (root) $);
      \draw[gray,dashed] ($ (-2,10,0) + (root) $) -- ($ (-3,6,0) + (root) $) -- ($ (-6,3.0,0) + (root) $);
      \draw[gray,dashed] ($ (-7,5.5,0) + (root) $) -- ($ (-2,3.5,0) + (root) $) -- ($ (2,1.0,0) + (root) $);
      \draw[gray,dashed] ($ (-2,1.0,0) + (root) $) -- ($ (2,3.5,0) + (root) $) -- ($ (7,5.5,0) + (root) $);

      % % Vertical lines
      \coordinate (root) at (0,0,2);
      \draw[gray] (-7,5.5,0) -- ($ (-7,5.5,0) + (root) $);
      \draw[gray,dashed] (-4.5,4.5,0) -- ($ (-4.5,4.5,0) + (root) $);
      \draw[gray,dashed] (-2,3.5,0) -- ($ (-2,3.5,0) + (root) $);
      \draw[gray,dashed] (0,2.25,0) -- ($ (0,2.25,0) + (root) $);
      \draw[gray,dashed] (2,3.5,0) -- ($ (2,3.5,0) + (root) $);
      \draw[gray,dashed] (4.5,4.5,0) -- ($ (4.5,4.5,0) + (root) $);
      \draw[gray] (7,5.5,0) -- ($ (7,5.5,0) + (root) $);
      \draw[gray] (-6,7.0,0) -- ($ (-6,7.0,0) + (root) $);
      \draw[gray,dashed] (-3,6,0) -- ($ (-3,6,0) + (root) $);
      \draw[thick,blue,opacity=0.5] (0,5.0,0) -- ($ (0,5.0,0) + (root) $);
      \draw[gray,dashed] (3,6,0) -- ($ (3,6,0) + (root) $);
      \draw[gray] (6,7.0,0) -- ($ (6,7.0,0) + (root) $);
      \draw[gray] (-5,8.5,0) -- ($ (-5,8.5,0) + (root) $);
      \draw[gray,dashed] (-2.5,8,0) -- ($ (-2.5,8,0) + (root) $);
      \draw[gray,dashed] (0,7.5,0) -- ($ (0,7.5,0) + (root) $);
      \draw[gray,dashed] (2.5,8,0) -- ($ (2.5,8,0) + (root) $);
      \draw[gray] (5,8.5,0) -- ($ (5,8.5,0) + (root) $);
      \draw[thick,blue,opacity=0.5] (-4,10,0) -- ($ (-4,10,0) + (root) $);
      \draw[gray,dashed] (-2,10,0) -- ($ (-2,10,0) + (root) $);
      \draw[gray,dashed] (0,10,0) -- ($ (0,10,0) + (root) $);
      \draw[gray,dashed] (2,10,0) -- ($ (2,10,0) + (root) $);
      \draw[thick,blue,opacity=0.5] (4,10,0) -- ($ (4,10,0) + (root) $);
      \draw[thick,blue] (-8,4,0) -- ($ (-8,4,0) + (root) $);
      \draw (-6,3.0,0) -- ($ (-6,3.0,0) + (root) $);
      \draw (-4,2,0) -- ($ (-4,2,0) + (root) $);
      \draw (-2,1.0,0) -- ($ (-2,1.0,0) + (root) $);
      \draw[thick,blue] (0,0,0) -- ($ (0,0,0) + (root) $);
      \draw (2,1.0,0) -- ($ (2,1.0,0) + (root) $);
      \draw (4,2,0) -- ($ (4,2,0) + (root) $);
      \draw (6,3.0,0) -- ($ (6,3.0,0) + (root) $);
      \draw[thick,blue] (8,4,0) -- ($ (8,4,0) + (root) $);

      % Top level
      \draw[thick,blue] ($ (-8,4,0) + (root) $) -- ($ (0,0,0) + (root) $) -- ($ (8,4,0) + (root) $);
      \draw[thick,blue] ($ (8,4,0) + (root) $) -- ($ (4,10,0) + (root) $) -- ($ (-4,10,0) + (root) $) -- ($ (-8,4,0) + (root) $);
      \draw ($ (0,5.0,0) + (root) $) -- ($ (4,2,0) + (root) $);
      \draw ($ (0,5.0,0) + (root) $) -- ($ (6,7.0,0) + (root) $);
      \draw ($ (0,5.0,0) + (root) $) -- ($ (0,10,0) + (root) $);
      \draw ($ (0,5.0,0) + (root) $) -- ($ (-6,7.0,0) + (root) $);
      \draw ($ (0,5.0,0) + (root) $) -- ($ (-4,2,0) + (root) $);
      \draw ($ (6,3.0,0) + (root) $) -- ($ (3,6,0) + (root) $) -- ($ (2,10,0) + (root) $);
      \draw ($ (5,8.5,0) + (root) $) -- ($ (0,7.5,0) + (root) $) -- ($ (-5,8.5,0) + (root) $);
      \draw ($ (-2,10,0) + (root) $) -- ($ (-3,6,0) + (root) $) -- ($ (-6,3.0,0) + (root) $);
      \draw ($ (-7,5.5,0) + (root) $) -- ($ (-2,3.5,0) + (root) $) -- ($ (2,1.0,0) + (root) $);
      \draw ($ (-2,1.0,0) + (root) $) -- ($ (2,3.5,0) + (root) $) -- ($ (7,5.5,0) + (root) $);

      % Separation surface 1
      \fill[color=red,fill=red,opacity=0.8] (0,5,1) -- (3,6,1) -- (3,6,2) -- (0,5,2) -- cycle;
      \fill[color=red,fill=red,opacity=0.5] (3,6,1) -- (6,7,1) -- (6,7,2) -- (3,6,2) -- cycle;
      \fill[color=red,fill=red,opacity=0.5] (0,5,1) -- (3,6,1) -- (3,6,0) -- (0,5,0) -- cycle;
      \fill[color=red,fill=red,opacity=0.2] (3,6,1) -- (6,7,1) -- (6,7,0) -- (3,6,0) -- cycle;

      % Singular vertices
      \fill[color=green,fill=green,opacity=0.8] (0,0,0) ellipse (0.2 and 0.4);
      \fill[color=green,fill=green,opacity=0.8] (-8,4,0) ellipse (0.2 and 0.4);
      \fill[color=green,fill=green,opacity=0.8] (8,4,0) ellipse (0.2 and 0.4);
      \fill[color=green,fill=green,opacity=0.5] (0,5,0) ellipse (0.2 and 0.4);
      \fill[color=green,fill=green,opacity=0.5] (-4,10,0) ellipse (0.2 and 0.4);
      \fill[color=green,fill=green,opacity=0.5] (4,10,0) ellipse (0.2 and 0.4);
      \fill[color=green,fill=green,opacity=0.8] (0,0,2) ellipse (0.2 and 0.4);
      \fill[color=green,fill=green,opacity=0.8] (-8,4,2) ellipse (0.2 and 0.4);
      \fill[color=green,fill=green,opacity=0.8] (8,4,2) ellipse (0.2 and 0.4);
      \fill[color=green,fill=green,opacity=0.5] (0,5,2) ellipse (0.2 and 0.4);
      \fill[color=green,fill=green,opacity=0.5] (-4,10,2) ellipse (0.2 and 0.4);
      \fill[color=green,fill=green,opacity=0.5] (4,10,2) ellipse (0.2 and 0.4);
  \end{tikzpicture}\end{center}
\end{frame}

\begin{frame}[fragile]
  \frametitle{Base complex}

  \begin{center}\begin{tikzpicture}[scale=0.4,%
      x={(1cm,-0.05cm)},%
      y={(0.1cm,0.5cm)},%
      z={(0cm,5.5cm)}]
      % % Bottom level
      \coordinate (root) at (0,0,0);
      \draw ($ (-8,4,0) + (root) $) -- ($ (0,0,0) + (root) $) -- ($ (8,4,0) + (root) $);
      \draw[gray] ($ (8,4,0) + (root) $) -- ($ (4,10,0) + (root) $) -- ($ (-4,10,0) + (root) $) -- ($ (-8,4,0) + (root) $);
      \draw[gray,dashed] ($ (0,5.0,0) + (root) $) -- ($ (4,2,0) + (root) $);
      \draw[gray,dashed] ($ (0,5.0,0) + (root) $) -- ($ (6,7.0,0) + (root) $);
      \draw[gray,dashed] ($ (0,5.0,0) + (root) $) -- ($ (0,10,0) + (root) $);
      \draw[gray,dashed] ($ (0,5.0,0) + (root) $) -- ($ (-6,7.0,0) + (root) $);
      \draw[gray,dashed] ($ (0,5.0,0) + (root) $) -- ($ (-4,2,0) + (root) $);
      \draw[gray,dashed] ($ (6,3.0,0) + (root) $) -- ($ (3,6,0) + (root) $) -- ($ (2,10,0) + (root) $);
      \draw[gray,dashed] ($ (5,8.5,0) + (root) $) -- ($ (0,7.5,0) + (root) $) -- ($ (-5,8.5,0) + (root) $);
      \draw[gray,dashed] ($ (-2,10,0) + (root) $) -- ($ (-3,6,0) + (root) $) -- ($ (-6,3.0,0) + (root) $);
      \draw[gray,dashed] ($ (-7,5.5,0) + (root) $) -- ($ (-2,3.5,0) + (root) $) -- ($ (2,1.0,0) + (root) $);
      \draw[gray,dashed] ($ (-2,1.0,0) + (root) $) -- ($ (2,3.5,0) + (root) $) -- ($ (7,5.5,0) + (root) $);

      % % Middle level
      \coordinate (root) at (0,0,1);
      \draw ($ (-8,4,0) + (root) $) -- ($ (0,0,0) + (root) $) -- ($ (8,4,0) + (root) $);
      \draw[gray] ($ (8,4,0) + (root) $) -- ($ (4,10,0) + (root) $) -- ($ (-4,10,0) + (root) $) -- ($ (-8,4,0) + (root) $);
      \draw[gray,dashed] ($ (0,5.0,0) + (root) $) -- ($ (4,2,0) + (root) $);
      \draw[gray,dashed] ($ (0,5.0,0) + (root) $) -- ($ (6,7.0,0) + (root) $);
      \draw[gray,dashed] ($ (0,5.0,0) + (root) $) -- ($ (0,10,0) + (root) $);
      \draw[gray,dashed] ($ (0,5.0,0) + (root) $) -- ($ (-6,7.0,0) + (root) $);
      \draw[gray,dashed] ($ (0,5.0,0) + (root) $) -- ($ (-4,2,0) + (root) $);
      \draw[gray,dashed] ($ (6,3.0,0) + (root) $) -- ($ (3,6,0) + (root) $) -- ($ (2,10,0) + (root) $);
      \draw[gray,dashed] ($ (5,8.5,0) + (root) $) -- ($ (0,7.5,0) + (root) $) -- ($ (-5,8.5,0) + (root) $);
      \draw[gray,dashed] ($ (-2,10,0) + (root) $) -- ($ (-3,6,0) + (root) $) -- ($ (-6,3.0,0) + (root) $);
      \draw[gray,dashed] ($ (-7,5.5,0) + (root) $) -- ($ (-2,3.5,0) + (root) $) -- ($ (2,1.0,0) + (root) $);
      \draw[gray,dashed] ($ (-2,1.0,0) + (root) $) -- ($ (2,3.5,0) + (root) $) -- ($ (7,5.5,0) + (root) $);

      % % Vertical lines
      \coordinate (root) at (0,0,2);
      \draw[gray] (-7,5.5,0) -- ($ (-7,5.5,0) + (root) $);
      \draw[gray,dashed] (-4.5,4.5,0) -- ($ (-4.5,4.5,0) + (root) $);
      \draw[gray,dashed] (-2,3.5,0) -- ($ (-2,3.5,0) + (root) $);
      \draw[gray,dashed] (0,2.25,0) -- ($ (0,2.25,0) + (root) $);
      \draw[gray,dashed] (2,3.5,0) -- ($ (2,3.5,0) + (root) $);
      \draw[gray,dashed] (4.5,4.5,0) -- ($ (4.5,4.5,0) + (root) $);
      \draw[gray] (7,5.5,0) -- ($ (7,5.5,0) + (root) $);
      \draw[gray] (-6,7.0,0) -- ($ (-6,7.0,0) + (root) $);
      \draw[gray,dashed] (-3,6,0) -- ($ (-3,6,0) + (root) $);
      \draw[gray,dashed] (0,5.0,0) -- ($ (0,5.0,0) + (root) $);
      \draw[gray,dashed] (3,6,0) -- ($ (3,6,0) + (root) $);
      \draw[gray] (6,7.0,0) -- ($ (6,7.0,0) + (root) $);
      \draw[gray] (-5,8.5,0) -- ($ (-5,8.5,0) + (root) $);
      \draw[gray,dashed] (-2.5,8,0) -- ($ (-2.5,8,0) + (root) $);
      \draw[gray,dashed] (0,7.5,0) -- ($ (0,7.5,0) + (root) $);
      \draw[gray,dashed] (2.5,8,0) -- ($ (2.5,8,0) + (root) $);
      \draw[gray] (5,8.5,0) -- ($ (5,8.5,0) + (root) $);
      \draw[gray] (-4,10,0) -- ($ (-4,10,0) + (root) $);
      \draw[gray,dashed] (-2,10,0) -- ($ (-2,10,0) + (root) $);
      \draw[gray,dashed] (0,10,0) -- ($ (0,10,0) + (root) $);
      \draw[gray,dashed] (2,10,0) -- ($ (2,10,0) + (root) $);
      \draw[gray] (4,10,0) -- ($ (4,10,0) + (root) $);
      \draw (-8,4,0) -- ($ (-8,4,0) + (root) $);
      \draw (-6,3.0,0) -- ($ (-6,3.0,0) + (root) $);
      \draw (-4,2,0) -- ($ (-4,2,0) + (root) $);
      \draw (-2,1.0,0) -- ($ (-2,1.0,0) + (root) $);
      \draw (0,0,0) -- ($ (0,0,0) + (root) $);
      \draw (2,1.0,0) -- ($ (2,1.0,0) + (root) $);
      \draw (4,2,0) -- ($ (4,2,0) + (root) $);
      \draw (6,3.0,0) -- ($ (6,3.0,0) + (root) $);
      \draw (8,4,0) -- ($ (8,4,0) + (root) $);

      % Top level
      \draw ($ (-8,4,0) + (root) $) -- ($ (0,0,0) + (root) $) -- ($ (8,4,0) + (root) $);
      \draw ($ (8,4,0) + (root) $) -- ($ (4,10,0) + (root) $) -- ($ (-4,10,0) + (root) $) -- ($ (-8,4,0) + (root) $);
      \draw ($ (0,5.0,0) + (root) $) -- ($ (4,2,0) + (root) $);
      \draw ($ (0,5.0,0) + (root) $) -- ($ (6,7.0,0) + (root) $);
      \draw ($ (0,5.0,0) + (root) $) -- ($ (0,10,0) + (root) $);
      \draw ($ (0,5.0,0) + (root) $) -- ($ (-6,7.0,0) + (root) $);
      \draw ($ (0,5.0,0) + (root) $) -- ($ (-4,2,0) + (root) $);
      \draw ($ (6,3.0,0) + (root) $) -- ($ (3,6,0) + (root) $) -- ($ (2,10,0) + (root) $);
      \draw ($ (5,8.5,0) + (root) $) -- ($ (0,7.5,0) + (root) $) -- ($ (-5,8.5,0) + (root) $);
      \draw ($ (-2,10,0) + (root) $) -- ($ (-3,6,0) + (root) $) -- ($ (-6,3.0,0) + (root) $);
      \draw ($ (-7,5.5,0) + (root) $) -- ($ (-2,3.5,0) + (root) $) -- ($ (2,1.0,0) + (root) $);
      \draw ($ (-2,1.0,0) + (root) $) -- ($ (2,3.5,0) + (root) $) -- ($ (7,5.5,0) + (root) $);

      % Components
      \fill[color=red,fill=purple,opacity=0.4] (-4,10,2) -- (-6,7,2) -- (-6,7,0) -- (0,5,0) -- (0,10,0) -- (0,10,2) -- cycle;
      \fill[color=blue,fill=blue,opacity=0.4] (0,10,2) -- (4,10,2) -- (6,7,2) -- (6,7,0) -- (0,5,0) -- (0,5,2) -- cycle;
      \fill[color=cyan,fill=cyan,opacity=0.4] (-6,7,2) -- (0,5,2) -- (0,5,0) -- (-4,2,0) -- (-8,4,0) -- (-8,4,2) -- cycle;
      \fill[color=yellow,fill=yellow,opacity=0.4] (6,7,2) -- (0,5,2) -- (0,5,0) -- (4,2,0) -- (8,4,0) -- (8,4,2) -- cycle;
      \fill[color=green,fill=green,opacity=0.4] (0,0,0) -- (4,2,0) -- (4,2,2) -- (0,5,2) -- (-4,2,2) -- (-4,2,0) -- cycle;
  \end{tikzpicture}\end{center}
\end{frame}

\begin{frame}
  \frametitle{Circular base component}

  \begin{center}
    \includegraphics[height=0.6\textheight]{./images/circular-base-x4.png}
    \includegraphics[height=0.6\textheight]{./images/circular-base-x1.png}
  \end{center}
\end{frame}

\begin{frame}[fragile]
  \frametitle{Real example 1}
  \begin{columns}
    \begin{column}{0.5\textwidth}
      \includegraphics[width=\textwidth]{./images/ex1-singularities.png}
    \end{column}
    \begin{column}{0.5\textwidth}
      \includegraphics[width=\textwidth]{./images/ex1-base-complex.png}
    \end{column}
  \end{columns}
\end{frame}

\begin{frame}[fragile]
  \frametitle{Real example 2}
  \begin{columns}
    \begin{column}{0.5\textwidth}
      \includegraphics[width=\textwidth]{./images/ex2-singularities.png}
    \end{column}
    \begin{column}{0.5\textwidth}
      \includegraphics[width=\textwidth]{./images/ex2-base-complex.png}
    \end{column}
  \end{columns}
\end{frame}

\begin{frame}[fragile]
  \frametitle{Sheets}
  \begin{center}
    \includegraphics[height=0.8\textheight]{./images/ex2-sheet1.png}
  \end{center}
\end{frame}

\begin{frame}[fragile]
  \frametitle{Dual}
  \begin{center}
    \includegraphics[height=0.6\textwidth]{./images/ex2-sheet1.png}
    \includegraphics[height=0.6\textwidth]{./images/dual.png}
  \end{center}
\end{frame}

\section{Selection}

\begin{frame}
  \frametitle{Sheets selection}
  \begin{columns}
    \begin{column}{0.5\textwidth}
      \includegraphics[width=0.9\textwidth]{./images/vis-2d-ex2-allsheets.png}
      \begin{center}All sheets\end{center}
    \end{column}
    \begin{column}{0.5\textwidth}
      \includegraphics[width=0.9\textwidth]{./images/vis-2d-ex2-sheets.png}
      \begin{center}Extracted main sheets\end{center}
    \end{column}
  \end{columns}
\end{frame}

\begin{frame}
  \frametitle{Sheets interactions}
  \begin{columns}
    \begin{column}{0.3\textwidth}
      \includegraphics[width=\textwidth]{./images/sheet-intersect.png}
      \begin{center}Intersecting\end{center}
    \end{column}
    \begin{column}{0.3\textwidth}
      \includegraphics[width=\textwidth]{./images/sheet-adjacent.png}
      \begin{center}Adjacent\end{center}
    \end{column}
    \begin{column}{0.3\textwidth}
      \includegraphics[width=\textwidth]{./images/sheet-hybrid.png}
      \begin{center}Hybrid\end{center}
    \end{column}
  \end{columns}
\end{frame}

\begin{frame}
  \frametitle{Sheet complexity}
  \begin{center}
    \includegraphics[height=1.1\textheight]{./images/self-intersect.png}
  \end{center}
\end{frame}

\begin{frame}
  \frametitle{Sheet covering selection}

  \begin{itemize}
    \item Cover all base complexes
    \item Minimize the intersection of sheets
    \item Maximize individual sheet complexity
  \end{itemize}

  \begin{block}{Remark}
    This is a special case of the set cover problem.
  \end{block}
\end{frame}

\begin{frame}
  \frametitle{Sheet interaction matrix}
  \begin{columns}
    \begin{column}{0.5\textwidth}
      \begin{center}
        \includegraphics[height=\textwidth]{./images/interaction.png}
      \end{center}
    \end{column}
    \begin{column}{0.5\textwidth}
      \includegraphics[width=\textwidth]{./images/matrix.png}
    \end{column}
  \end{columns}
  \begin{block}{Complexity measure}\begin{equation*}
      \sqrt{\text{trace}(M^TM)}
  \end{equation*}\end{block}
\end{frame}

\section{Visualisation}

% \subsection{3D visualization}

\begin{frame}[fragile]
  \frametitle{3D visualization: first example}
  \includegraphics[height=0.5\textheight]{./images/vis-3d-ex1-sheets.png}
  \includegraphics[height=0.5\textheight]{./images/vis-3d-ex1-curves.png}
\end{frame}

\begin{frame}[fragile]
  \frametitle{3D visualization: second example}
  \includegraphics[height=0.5\textheight]{./images/vis-3d-ex2-sheets.png}
  \includegraphics[height=0.5\textheight]{./images/vis-3d-ex2-curves.png}
\end{frame}

% \subsection{2D matrix view}

\begin{frame}[fragile]
  \frametitle{2D matrix view: let's take an example}
  \begin{center}
  \includegraphics[height=0.9\textheight]{./images/vis-2d-ex2-sheets-w-hints.png}
  \includegraphics[height=0.5\textheight]{./images/vis-2d-ex2-matrix.png}
  \end{center}
\end{frame}

\section{Conclusion}

\begin{frame}
  \frametitle{Complete analysis}
  \includegraphics[width=\textwidth]{./images/complete.png}
\end{frame}

\begin{frame}
  \frametitle{Hybrid representation}
  \includegraphics[width=\textwidth]{./images/hybrid.png}
\end{frame}

\begin{frame}
  \frametitle{Remarks}
  \begin{itemize}
    \item[$+$] self-contained (definitions included)
    \item[$+$] cool and useful pictures
    \item[$+$] available source code
    \item[$-$] no code documentation
    \item[$-$] some details of the defined norm are arbitrary
  \end{itemize}
\end{frame}

\section*{}

\begin{frame}
  \frametitle{Thank you! Any question?}
  \begin{center}
    Thank you! Any question?
  \end{center}
\end{frame}

\end{document}