\section{Methods for monocular reconstruction of articulated subjects}

    Having discussed methods for modelling articulated subjects, this section will discuss approaches for reconstructing the 3D shape and pose of a subject from a monocular image or video. It is important to note that this task is challenging and fundamentally ill-posed. In common with other challenging 3D reconstruction tasks, input images will typically exhibit variation in camera view, lighting and environmental occlusion. However, 3D reconstruction pipelines for articulated subjects must also deal with variation due to body shape, body pose, clothing and self-occlusion (body parts obscuring other parts). In addition, the challenge of reconstructing 3D models from 2D images is also inherently ambiguous. As explained by Toshev and Szegedy~\cite{toshev2014deeppose}, even if 2D structure can be determined (for example, using 2D keypoint prediction), the subsequent `lifting' step to recover 3D remains ill-posed, as the space of consistent 3D poses for given 2D landmark locations is infinite. It is for this reason that the history of monocular 3D reconstruction makes extensive use of 3D morphable models (or other geometric, temporal, structural priors), as they provide necessary optimization contraints.

    The theme of this section is therefore to discuss how 3D morphable models (3DMMs) can be incoporated into 3D reconstruction pipelines. In general, algorithms take as input an image or video and predict a set of 3D model parameters $\alpha$ (often factored into shape $\shape$ and pose $\pose$). Once determined, the output parameters are then supplied to the morphable model's generator function $g: \alpha \mapsto \R{3}$ (e.g. $\SMPL: (\shape, \pose) \mapsto \R{3}$ or $\SMAL: (\shape, \pose) \mapsto \R{3}$) to produce vertex positions $V \subseteq \R{3}$ for the model $M = (V, T)$ with fixed triangulation $T$. For completeness and comparison, this section will make some mention of the small class of 3D reconstruction methods for articulated subjets which operate without an explicit 3D morphable model. Although currently a developing area, current work in this category typically requires either paired 3D training data, employ alternative (and arguably more restrictive) shape priors (e.g. symmetry constraints) or produce results of significantly lower fidelity.

    Methods which align a parametric 3D model to monocular input date back as far as 1963, in a seminal paper by Roberts~\cite{xxx}. Roberts presents a method which optimizes parameters for viewpoint and cuboidal shape primitives to reconstruct a 2D line image. Model-based methods have also been applied to understand object structure, starting with fitting of geometric primitives~\cite{xxx} and later with Active Shape Models~\cite{xxx} which learn deformation priors from a provided training set. Perhaps due to the numerous commercial applications, the majority of recent work in 3D shape and pose recovery focuses particuarly on \emph{humans} as a special case. The first example of such an approach is the seminal work of Blanz and Vetter~\cite{blanz-vetter} who built the first 3D morphable face model by aligning 3D scans and optimized the parameters to provide a fit to a single image. Since then, the research community has collected a multitude of open source human datasets which provide strong supervisory signals for training deep neural networks. These include include accurate 3D deformable template models~\cite{loper15smpl} generated from real human scans, 3D motion capture datasets~\cite{ionescu2013human3,vonmarcard2018recovering} and large 2D datasets~\cite{lin2014microsoft,johnson2010clustered,andriluka14cvpr} which provide keypoint and silhouette annotations. The combination of these publically available datasets and their incoporation into deep learning pipelines have led to impressive reconstruction results when tested on in-the-wild human images and videos. Unfortunately, the diversity among animal subjects and the practical challenges associated with data capture have resulted in few datasets being made available. Despite appearing superficially similar to human tracking, these factors result in specific challenges to animal tracking which must be carefully handled. Perhaps for this reason, the body of related literature for animal tracking is considerably sparser. The remainder of this section will focus on methods for 3D pose estimation, followed by 3D shape and pose reconstruction of human and animal bodies. Further discussion on techniques for body part (e.g. face, hands) reconstruction are deferred to the following survey papers~\cite{xxx, xxx}.

\subsection{3D Pose Estimation}

% The reconstruction of an arbitrary configuration of 3D points from a single monocular RGB image has three characteristics that affect its performance: (i) it is a severely ill-posed problem because similar image projections can be derived from different 3D poses; (ii) it is an ill-conditioned problem since minor errors in the loca- tions of the 2D body joints can have large consequences in the 3D space; and (iii) it suffers from high dimensionality ( Agarwal and Triggs, 2006 ).

    Techniques for 3D pose estimation output a set of 3D keypoint locations which can be combined to form a skeletal outline. Apart from basic limb measurements, no other shape detail (e.g. surface definition, object density etc.) is obtained. However, it should be noted that this output form is often perfectly satisfactory depending on the intended application. In particular, this family of techniques have found numerous applications in controllerless gaming (e.g. Microsoft Kinect~\cite{kinectpaper}), motion capture (e.g. for digital character generation~\cite{xxx}), gait analysis (e.g. identifying lameness in cattle~\cite{xxx}) and many more.

    The general approach is to recover a 3D skeleton such that the 3D joints project to known or estimated 2D joints subject to anatomical priors. Early approaches in this category fit human stick figures with various constraints, including assumptions of fixed limb lengths~\cite{xxx}, length ratios~\cite{xxx} or that limb lengths are isometric across individuals and vary only in global scaling~\cite{xxx}. More advanced techniques built statistical models of shape variation using anthropometric tables or learnt them from motion capture data~\cite{barron2001estimating}.

    A broad category of appraoches for this are methods for \emph{non-rigid structure from motion}~\cite{xxx}. The general formulation is to express a 3D skeleton $S \in \RR{3}{P}$ on $P$ points as a linear combination of basis shapes $S_{1}, \dots, S_{k}$ where $S_{i} \in \RR{3}{P}$. Precisely:

    \begin{equation}
        S = \sum_{i=1}^K l_{i} \cdot S_{i} \quad S, S_{i} \in \RR{3}{P} \quad l_{i} \in \R{}
    \end{equation}

    Assuming scaled orthographic projection, the following expression represents the projection of $P$ points of $S$ into 2D image coordinates $(u_{i}, v_{i})$:

    \begin{equation}
        \begin{bmatrix}
            u_{1} & u_{2} & \dots & u_{P} \\
            v_{1} & v_{2} & \dots & v_{P} \\
        \end{bmatrix}
        = R \cdot ( \sum_{i=1}^{K} l_{i} \cdot S_{i} ) + T
    \end{equation}

    or equivalently:

    \begin{equation}
        \begin{bmatrix}
            u_{1} & u_{2} & \dots & u_{P} \\
            v_{1} & v_{2} & \dots & v_{P} \\
        \end{bmatrix}
        =
        \begin{bmatrix}
            l_{1}R & \dots & l_{K}R
        \end{bmatrix}
        \cdot
        \begin{bmatrix}
            S_{1} \\
            S_{2} \\
            \dots \\
            S_{K} \\
        \end{bmatrix}
    \end{equation}

    \def\PT#1#2#3{#1_{#2}^{(#3)}}
    \def\T#1#2{#1^{(#2)}}

    This can then be extended to handle multiple views of the subject taken over a monocular video sequence. Let $(\PT{u}{i}{t}, \PT{v}{i}{t})$ denote the tracked 2D point at timestep $t$. This gives rise to the following system, taken over $N$ timesteps:

    \begin{equation}
        \underbrace{
        \begin{bmatrix}
            \PT{u}{i}{1} & \dots & \PT{u}{P}{1} \\
            \PT{v}{i}{1} & \dots & \PT{v}{P}{1} \\
            \PT{u}{i}{2} & \dots & \PT{u}{P}{2} \\
            \PT{v}{i}{2} & \dots & \PT{v}{P}{2} \\
            & \dots & \\
            \PT{u}{i}{N} & \dots & \PT{u}{P}{N} \\
            \PT{v}{i}{N} & \dots & \PT{v}{P}{N} \\
        \end{bmatrix}
        }_{W}
        =
        \underbrace{
        \begin{bmatrix}
            \PT{l}{1}{1}\T{R}{1} & \dots & \PT{l}{K}{1}\T{R}{1} \\
            \PT{l}{1}{2}\T{R}{2} & \dots & \PT{l}{K}{2}\T{R}{2} \\
            & \dots & \\
            \PT{l}{1}{N}\T{R}{N} & \dots & \PT{l}{K}{2}\T{R}{N} \\
        \end{bmatrix}
        }_{Q}
        \cdot
        \underbrace{
        \begin{bmatrix}
            S_{1} \\
            S_{2} \\
            \dots \\
            S_{K}
        \end{bmatrix}
        }_{B}
    \end{equation}

    % https://arxiv.org/pdf/1705.03098.pdf

    This shows the tracking matrix $W$ can be factored into 2 matrices: $Q$ which contains the camera pose $\T{R}{t}$ and configuration weights $\PT{l}{1}{t}, \dots, \PT{l}{K}{t}$ per frame ${t}$. $B$ encodes the $K$ basis shapes $S_{i}$. This system can be factored with singular value decomposition to yield the shape basis $S_{i}$, per-frame camera rotations $R$ and per-frame configuration weights $l$. A number of techniques follow this formulation~\cite{xxx, xxx, xxx}, but start with a shape basis learnt from available motion capture datasets (e.g. CMU~\cite{xxx}).

    More recent approaches were designed to be fully automatic. Shotton et al.~\cite{kinectpaper} designed a commerically-available system for 3D human skeletal tracking which required a depth sensor. A generative 3D body model was used to synthesize a large training dataset of depth images with corresponding body part labels. Density estimators for each body part are then used in combination to localize body joints with a calculated confidence value. Taylor et al.~\cite{taylor2012vitruvian} predict dense correpsondences between image pixels (again, with depth so in $\R{3}$) and a representative 3D human body model, again by training on synthetic depth images. \Cref{chap:cgas} of this thesis demonstrates a technique for predicting keypoints by training on synthetic \emph{silhouette} data, rendered from an animal deformable body model, which overcomes the need for depth imagery at test time.

    Automatic monocular approaches often take advantage of 2D keypoint or body part detectors when reasoning about 3D skeletons. Simo-Serra et al.~\cite{xxx, xxx} form a probabilistic model that models both 3D pose and 2D keypoints jointly, overcoming noise among 2D body parts. Other approaches~\cite{xxx, xxx} employ a two-stage pipeline; they begin by localizing 2D joint positions on an input image before running a subsequent optimization step that `lifts` these to a 3D pose. Tangential work~\cite{xxx} takes uses detected 2D joints to perform a nearest neighbour search in a 3D mocap dataset. The most recent two-stage pipelines rely on deep convolutional networks to predict keypoints. Examples of such systems include DeepPose~\cite{toshev2014deeppose}, an approach which employs a CNN to reason jointly about 2D landmark detection and 3D pose estimation from single RGB images. Pishchulin et al.~\cite{pishchulin2016deepcut} later introduced DeepCut which extends DeepPose to the multi-person case.

    State-of-the-art techniques now operate as a direct regression to a 3D pose. Most often, paired 3D training data (such as is available from datasets such as Human3.6M~\cite{xxx}) is required which is generally expensive to obtain, particularly for animal categories. One branch of approaches~\cite{tekin2016direct} predicts body configuration in terms of angles. Other approaches include Pavlakos et al.~\cite{xxx}, who use a 2D joint predictor~\cite{xxx} followed by a deep architecture to regress 3D heatmaps. Moreno-Noguer~\cite{xxx} learn a pairwise distance matrix from 2D-to-3D space in order to allow unlikely 3D predictions to be ruled out with a suitable prior. These techniques were designed under the assumption that neural networks would struggle to learn a `lifting' function from 2D to 3D pose. This assumption was corrected by Martinez et al.~\cite{xxx} who demonstrate the effectiveness of a simple architecture at regressing accurate 3D keypoints from 2D predictions. This technique was later interpreted probabilistically by The technique was interpreted probabilistically by Li et al.~\cite{xxx}, who handled ambiguity in the 2D-to-3D lifting problem with a mixture density network. Related work that predicts a depth segmentation (so not strictly 3D keypoints) is SURREAL~\cite{xxx} who train their network with data generated synthetically with a 3D human body model.

% \subsection{Toolkit}

\subsection{Model-based human shape and pose}

    % only discuss dense methods -- discuss methods that do/do not explicitly model shape

    This section will discuss methods for reconstructing a full 3D \emph{dense} human from a monocular image or video sequence. Early work in this category fit shape primitives combined into a kinematic tree to silhouettes extracted from the input~\cite{xxx, xxx, xxx}. The introduction of the 3D deformable human body model known as SCAPE~\cite{xxx} enabled various fitting approaches. Sigal et al~\cite{xxx} compute shape features from manually extracted silhouettes and use a mixture of experts formulation for predicting SCAPE model parameters. Later, Guan et al.~\cite{xxx} fit the SCAPE model to provided keypoints, extracted silhouettes, edges and shading cues. They also define an interpenetration term that penalizes self-intersecting body parts, although this does not lead to easy optimization. Hasler et al.~\cite{xxx}, Zhou et al.~\cite{xxx} and Chen et al.~\cite{xxx} present a similar approach, although show optimization only to input keypoints and manually or semi-manually (e.g. GraphCut~\cite{xxx}) extracted silhouettes.

    A significant advance was made by the introduction of SMPLify~\cite{xxx}, the first fully-automatic method for monocular 3D human pose and shape reconstruction. Many of the concepts presented by SMPLify are used throughout this thesis, making it worthy of study.

    %     \begin{equation}
    %         E_{\text{data}}(\theta,U) =\sum_{i=1}^{n}s_{i} \cdot d(x_{i}, M(u_{i}; \theta))
    %     \end{equation}
    %     where $M(u_{i}, \theta)$ is the position of vertex $u_{i}$ on the vitruvian manifold mesh after having been displaced by an LBS deformation with respect to the pose~$\theta$.

    %     The sheer quantity of correspondences greatly constrain their optimizer which works well, even on challenging input images. Much of this report focuses on how this paper can be extended to work for animal subjects, incorporating deep learning correspondence prediction and working from monocular RGB input data.

    \subsubsection{Fitting a 3D model to 2D keypoints}

    \def\J#1{J_\mathrm{#1}}

    SMPLify works by fitting the SMPL~\cite{loper15smpl} model to a set of 2D image locations predicted by DeepCut~\cite{xxx}, a deep convolutional neural network. For an input image $I$, DeepCut predicts a set of image keypoint locations $\J{est} \in \RR{23}{2}$ which correspond to locations on the 3D SMPL mesh $\J{\SMPL}$. Precisely $\J{\SMPL} = R_{\pose}(J(\shape))$ where $J(\shape)$ computes 3D skeleton positions from SMPL shape parameters $\shape$, and $R_{\pose}$ is the global rigid transformation effected by SMPL pose parameters $\pose$. A model fitting approach is then used to align the SMPL model to the predicted keypoint positions. This is achieved through optimizing the SMPL parameters $(\shape, \pose)$, global translation $\trans$ and camera parameters $K$, subject to priors over pose, shape and limb interpenetration priors.

    The key energy term used in the optimization (and indeed throughout this thesis) is given by $\E{J}(\shape, \pose; K, \J{est})$ and measures the weighted 2D distance between estimated keypoints $\J{est}$ and the corresponding SMPL joints $J_{\SMPL}$.

    % E_{J}(\shape, \pose, \trans; K, \J{est}) = \sum_{joint, i} w_{i} \rho(\Pi_{K}(\J{\SMPL, i} - \J{est, i}))

    \begin{equation}
        E_{J}(\shape, \pose, \trans; K, \J{est}) = \sum_{\mathrm{joint}, i} w_{i} \rho(\Pi_{K}(\J{\SMPL, i} - \J{est, i}))
    \end{equation}

    The weighted 2D distance is implemented using the Geman-McClure~\cite{xxx} penalty function $\rho$ which helps deal with noisy DeepCut estimates. SMPLify implements $\Pi_{K}$ perspective camera model with known (or roughly initialized) focal length although others opt for orthographic projection. The following definition provides a quick primer for this:

    \begin{definition}[Primer on camera geometry]

        Perspective projection is a function which maps a 3D structure to blah blah.

        \begin{equation}
            2X = Y
        \end{equation}

        Orthographic projection is the following:

        \begin{equation}
            3X = Z
        \end{equation}

    \end{definition}

    The full energy formulation is then given as:

    \begin{equation}
        E(\shape, \pose) = E_{J}(\shape, \pose; K, J_{\text{est}}) + \lambda_{\pose}E_{\pose}(\pose) + \lambda_{\alpha}E_{\alpha}(\pose) + \lambda_{\text{sp}}E_{\text{sp}}(\pose; \shape) + \lambda_{\shape}E_{\shape}(\shape)
    \end{equation}

    where the following energy terms are employed, balanced according to the $\lambda$ scalar weights:

    \begin{itemize}
        \item $E_{\pose}(\pose)$ is referred to as a \textit{pose prior} which favours more likely poses by assigning large punishment to those that deviate from known poses collected from a large dataset.
        \item $E_{\shape}(\shape)$ is referred to as a \textit{shape prior} which favours more likely pose-invariant shape configurations by assigning large punishment to those that deviate from known shapes collected from a large dataset.
        \item $E_{\alpha}(\pose)$ is a \textit{joint limit} prior which ensures particular joints remain within acceptable angle limits. For example, a knee joint in a human model should be prohibited from bending more than 5 degrees upwards.
        \item $E_{sp}(\pose; \shape)$ is an \textit{interpenetration} term, which can only be defined in such shape modelling approaches. Using both shape and pose from the model, it is possible to determine if any limbs are self-intersecting, or intersect other parts of the body and assign appropriate penalty.
    \end{itemize}

    % TODO: Variations on SMPLify
    SMPLify has recently undergone subsequent variations, including Huang et al.~\lazycite{https://arxiv.org/abs/1707.07548}{Huang Multiview} who fit SMPL to multi-view images and Pavlakos et al.~\lazycite{https://arxiv.org/abs/1904.05866}{SMPL-X fitting} who extend SMPL with hand and facial expression parameters and follow a similar fitting procedure. \Cref{chap:cgas} of this thesis will introduce a \emph{self-supervised} version of SMPLify that uses synthetic data for training, thereby overcoming the need for a large 2D dataset with manually-labelled keypoints.

    An example result can be seen in Figure \ref{fig:smplify}:

    \begin{figure}[H] % Example image
        \center{\includegraphics[width=0.95\linewidth]{fitting_smpl}}
        \caption{SMPLify: Fitting the SMPL model to the Leeds Sports Dataset.}
        \label{fig:smplify}
    \end{figure}

    \subsubsection{Direct regression}
    The most recent, and state-of-the-art approaches employ deep learning techniques to solve the entire optimization problem by directly regressing shape and pose parameters of the template model. An early approach by Tan et al.~\cite{tan17indirect} used deep neural networks to learn an encoding $f: \RR{H}{W} \mapsto (\pose, \shape, \trans, K)$ of input images to SMPL pose and shape, translation and camera parameters. Their method makes use of a \emph{silhouette renderer} $R : (V, T) \mapsto \{0,1\}^{H \times W}$ (learnt using synthetic data) capable of producing a binary silhouette from a predicted SMPL mesh. In this way, $R$ allows the network's SMPL predictions to be supervised to ensure generated silhouettes match ground-truth annotations.
    Improvements were realised by incoporating an abudance of available human data into the training pipeline. Apart from 3D morphable models (e.g. SMPL~\cite{loper15smpl}), methods use paired 3D motion capture data (e.g. Human3.6M~\cite{ionescu2013human3,IonescuSminchisescu11}, 3DPW~\cite{vonmarcard2018recovering}) to help relate 2D appearance and the underlying 3D structure, unpaired 3D motion capture data (e.g. CMU~\lazycite{CMU}{CMU}) for learning detailed priors over the distribution of human shapes and poses, and large 2D keypoint datasets (e.g. MSCOCO~\lazycite{COCO}{COCO}, LSP~\lazycite{LSP}{LSP}, MPI~\lazycite{MPI}{MPI}) which help promote generalization to `in-the-wild' scenarios. A notable work in this category is Human Mesh Recovery (HMR) of Kanazawa et al.~\cite{kanazawa18end-to-end}, although multiple concurrent works exist\lazycite{https://proceedings.neurips.cc/paper/2017/file/ab452534c5ce28c4fbb0e102d4a4fb2e-Paper.pdf}{Hsiao-Yu Fish Tung}, \lazycite{https://www.seas.upenn.edu/~pavlakos/projects/humanshape/}{pavlakos}, \lazycite{http://virtualhumans.mpi-inf.mpg.de/papers/omran2018NBF/omran2018NBF.pdf}{neural body fitting} and \lazycite{DenseRaC}{DenseRaC}. \Cref{chap:3dmulti} of this thesis will explore a method for modelling uncertainty in 3D reconstruction, by exploring extensions to the aforementioned architectures.

    % Show the basic structure of these networks in a picture

    BodyNet \lazycite{https://www.di.ens.fr/willow/research/bodynet/}{bodynet} follow a similar pipeline with the inclusion of a texture prediction module supervised by 2D body part segementations \lazycite{body part dataset}{dataset}. Silhouette data has also been shown to assist in accurate reconstruction of clothes, hair and other appearance detail~\cite{alldieck2019learning} \lazycite{https://arxiv.org/pdf/1901.00049.pdf}{SiCloPe} \lazycite{https://arxiv.org/abs/2004.04572}{ARCH}. Methods typically represent 3D clothes as `freeform' vertex deformations; in other words, deformations beyond the standard SMPL blend shapes which represent unclothed bodies. This presents significantly more degrees of freedom in the optimization, which must be controlled; either by large training datasets with limited variation or mesh-based deformation priors such as ARAP.

    While the dominant paradigm in human reconstruction is now end-to-end deep learning methods, SPIN~\cite{kolotouros19learning} show impressive improvement by incorporating an energy minimization process within their training loop to further minimize a 2D reprojection loss subject to fixed pose \& shape priors. This idea inspired the work presented in \Cref{chap:wldo} of this thesis, in which a 3D dog shape prior is learnt during the training loop via expectation maximization.

    % https://openaccess.thecvf.com/content_ICCV_2019/papers/Xu_DenseRaC_Joint_3D_Pose_and_Shape_Estimation_by_Dense_Render-and-Compare_ICCV_2019_paper.pdf

    \subsubsection{Implicit representations}\label{ss:implicit}

    There have also been a few recent works that reconstruct 3D humans without an explicit template prior. The general idea is to interpret a neural network as an implicit representation of the 3D surface. In particular, the network $f$ is trained to map sampled 3D locations $(x,y,z) \in \R{3}$ to an occupancy value $\{0,1\}$ (and optionally a texture value $(r,g,b) \in \R{3}$). This results in a memory efficient representation of the 3D surface as the space used to embed the surface does not need to be explicitly stored. Saito et al.~\cite{pifuSHNMKL19,saito2020pifuhd} present an example of such an architecture. Their network maps 3D locations with ResNet~\lazycite{Resnet}{Resnet} features sampled at coordinates projected with a known weak-perpective camera to occupancy and texture values. Li et al.\lazycite{https://project-splinter.github.io/monoport/}{MonoPort} later sped up reconstruction and rendering to allow real-time inference. Neural Radiance Fields (NeRF) capture high resolution details of static scenes by mapping 3D coordinates $(x,y,z)$ and viewing directions $(\theta, \phi)$ to volume density and view-dependent emitted radiances. The work was adapted to articulated structures~\lazycite{ANerf}{ANerf} by learning new skin for the SMPL model, using SPIN~\cite{kolotouros19learning} as a basic 3D skeleton predictor. A current downside of these techniques is the need to train a separate model per scene/subject and the slow inference time. However, these works do demonstrate high quality results and are a worthy direction for future research.

    % TODO - add these!

    % https://zju3dv.github.io/animatable_nerf/

    % https://pablopalafox.github.io/npms/palafox2021npms.pdf - decent related work about implicit representations for 3D shapes & deformable shapes. This paper learns different MLPs, one for shape and one for pose from existing datasets e.g. AMASS [30], DeformingThings4D [26], CAPE [30], MANO [45]

    % Multi-view Neural Human Rendering - https://openaccess.thecvf.com/content_CVPR_2020/papers/Wu_Multi-View_Neural_Human_Rendering_CVPR_2020_paper.pdf
    % https://arxiv.org/pdf/2104.03953.pdf - SNARF: Differentiable Forward Skinning for Animating Non-Rigid Neural Implicit Shapes
    % https://arxiv.org/abs/2102.06199 A-NERF
    % https://arxiv.org/abs/2011.14143 i3DMM - human heads
    % https://arxiv.org/abs/2007.11432 Combining Implicit Function Learning and Parametric Models for 3D Human Reconstruction
    % https://www.sciencedirect.com/science/article/abs/pii/S0097849321000777: DIMNet: Dense implicit function network for 3D human body reconstruction
    % https://pablopalafox.github.io/npms/?fbclid=IwAR0EPN8SuJ1j8LZyiccF4EgNA67nvrmmk7_kOf6P7Gk_fN1zgX32Z2txZmg - NPMs:

    % TODO: talk about how they are slow, but methods have been developed to speed them up.

    % Show the basic structure of these networks in a picture

\subsection{Model-based animals}

    Having discussed reconstruction methods for other articulated subjects, this section will move on work related to the main topic of this thesis: 3D reconstruction of animal subjects. As summarized above, there are multiple challenges associated with animals which are not present with humans. Firstly, the general class of animal subjects is significantly more diverse in appearance and structure than the human category. This holds (although to a lesser extent) with various important animal subcategorizations, such as the class of medium-to-large quadrupeds, dog breeds, or birds. This diversity leads to challenges when designing 3D morphable models, since extreme deformations must allowed while penalizing even subtle adapations deemed unnatural. Also of concern are the significant variations in animal motion patterns (causing complex self-occlusion), body textures including fur (which vary even within breeds) and difficult environmental conditions. An interesting advantageous aspect to reconstructing animals is that they are less frequently clothed; a common nusiance factor for human reconstruction. These factors combined with practical challenges associated with capturing and annotating images has also led to a lack of open-source datasets which depict animal subjects. Of particular concern to this thesis is the lack of 3D training data captured from real animal subjects (typically using motion capture or static scans). Equivalent resources made available for humans have been used as a basis for building 3D morphable models (e.g. SMPL~\cite{loper15smpl}, FAUST~\lazycite{FAUST}{FAUST}), for learning priors over human shape and pose~\lazycite{learning priors}{learning priors}, and to provide per-image 3D supervisation when training deep neural networks~\cite{kanazawa18end-to-end}. Thankfully, online animal imagery which could serve as the basis for future datasets is plentiful, as are datasets with 2D silhouette data~\cite{lin2014microsoft,everingham2010pascal,DAVIS2017-2nd}. A formal comparison is given in \Cref{tab:animal-dsets}.  While animals are often featured in computer vision literature, there are still relatively few works that focus on accurate 3D animal reconstruction. However, the creative approaches used in these methods makes a formal review worthwhile. A summary of recent approaches is tabulated in \Cref{tab:literature}.

    \input{Chapter3/Tables/animal-dsets-tbl.tex}

    \input{Chapter3/Tables/lit-review-tbl.tex}

    % Table~\ref{tab:literature} summarizes previous work on animal reconstruction.
    % It is interesting to note that while several papers demonstrate reconstruction across species, which {\em prima facie} is a richer class than just dogs, the test-time requirements (e.g. manually-clicked keypoints/silhouette segmentations, input image quality etc.) are considerably higher for those systems.
    % Thus we claim that the achievement of reconstructing a full range of dog breeds,
    % with variable fur length, varying shape and pose of ears, and with considerable occlusion, is a significant contribution.

    % \subsubsection{Animals intro}

    % Cashman and Fitzgibbon~\cite{cashman2013shape} obtained one of the first 3D morphable animal models, but their work was limited to small classes of objects (e.g. dolphins, pigeons), and did not incorporate a skeleton.  Their work also showed the use of the 2D silhouette for fitting, which is key to our method.
    % Reinert {\em et al.} \cite{reinert2016animated} meanwhile construct 3D meshes by fitting generalized cylinders to hand-drawn skeletons.
    % Combined skeletal and morphable models were used by Khamis {\em et al.}~\cite{hand-shape} for modelling the human hand, and Loper {\em et al.}~\cite{loper15smpl} in the SMPL model which has been extensively used for human tracking.

    % The SMPL model was extended to animals by Zuffi {\em et al.}~\cite{zuffi2017menagerie}, where the lack of motion capture data for animal subjects is cleverly overcome by building the model from $41$ 3D scans of toy figurines from five quadruped families in arbitrary poses. Their paper demonstrates single-frame fits of their model to real-world animal data, showing that despite the model being built from ``artists' impressions'' it remains an accurate model of real animals. This is borne out further by our work.  Their paper did however depend on per-frame human annotated keypoint labels, which would be costly and challenging to obtain for large video sequences. This work was recently extended~\cite{zuffi_lions} with a refinement step that optimizes over model vertex positions. This can be considered independent to the initial SMAL model fit and would be trivial to add to our method.

    \subsubsection{Learning animal shape from 2D image collections}
    % Early work in reconstructing animals includes Chen et al.~\lazycite{chensharks}{chensharks} who learn a simple shape space by registering 11 3D shark models.

    Early work in animals focuses on learning 3D animal shape spaces by registering an input 3D model to image collections. Examples include Chen et al.\lazycite{chensharks}{chensharks} who learn a shark model and Cashman et al.~\cite{cashman2013shape} who recover a parameterized, morphable 3D model from unrelated 2D images depicting examples of the target class. Their method requires user-supplied 2D object outlines and point constraints for each image, and a single rigid mesh for the entire object class. The authors demonstrate recovering an 8-parameter morphable dolphin model from 32 images sourced from the Internet. To reduce required user activity, it is reasonable to assume that given sufficient labelled training data, it would be simple to manipulate a convolutional network architecture able to perform foreground / background segmentation and identify key points (say, joints) for the desired object class. The system achieves impressive results when optimizing over both pose and shape parameters across a range of object classes, but suffers from an overly smooth shape representation which causes trouble for strongly articulated classes such as polar bears. Related is the work of Ntouskos et al.\lazycite{ntouskos}{ntouskos} who use an optimization scheme to combine geometric primitives fit to segmented animal parts.

    \begin{figure}[t] % Example image
        \center{\includegraphics[width=0.7\linewidth]{dolphins}}
        \caption{8-parameter dolphin model with annotated contour (left) and contour generators (middle and right).}
        \label{fig:cashman_fitzgibbon}
    \end{figure}

    \subsubsection{Reconstructing animals with unknown skeleton}

    The approach was later extended to articulated classes in work by Stebbing et al.~\cite{arap_stebbing}, who optimize a 3D template model to animal video sequences. In this work, rather than defining an internal skeleton, the template model is rigged with virtual markers, which assign each mesh vertex $v_i$ to one of $M$ groups that share a set of basis rotations $B_{m}$. User interaction is again reuqired to segement the animal from the background and to provide mesh-to-image keypoint correspondences.

    \begin{figure}[t]
        \centering
        \begin{subfigure}{0.5\textwidth}
        \centering
            \includegraphics[height=0.5\linewidth]{arapsfm/arap_annotated_template}
            \caption{Template mesh with joint movement constraints.}
        \end{subfigure}%
        \begin{subfigure}{0.5\textwidth}
        \centering
            \includegraphics[height=0.5\linewidth]{arapsfm/arap_point_tracks}
            \caption{Example of user supplied point tracks.}
        \end{subfigure}%
        \caption{User input required for the deformable mesh animation algorithm, reprinted from~\cite{arap_stebbing}.}
        \label{fig:arap_user}
    \end{figure}

    Through reasonably accurate pose fitting and by allowing some pose-invariant shape deformation, this work produces smooth meshes which match the input video. Moreover, experimentation demonstrates that ARAP is a useful prior for reconstructing articulated, non-rigid motion in instances that an internal skeleton is a priori unknown. However, the shape attributes for the reconstructed model are not particularly accurate, which results in frequent errors appearing at internal occluding contours. In addition, the large non-convex optimization algorithm is an expensive operation, taking around 1 minute per video frame on a standard Linux workstation. Results are shown on 11 sequences.

    Results showing this work fitting a crude dog template mesh to a sample video obtained from YouTube are shown previously in Figure \ref{fig:intro_arap_output}. Figure \ref{fig:arap_output} shows another example, which operates on a template impala mesh. Similar is the technique of Favreau et al.~\lazycite{favreau}{favreau}, who fitting geometric primitives to a 2D video sequence with manually sketched body part labels. % TODO

    \begin{figure}[t]
        \center{\includegraphics[width=0.95\linewidth]{arapsfm/arapsfm_output.png}}
        \caption{Example of an dog template being fit to input video sequence, reprinted from~\cite{arap_stebbing}}
        \label{fig:arap_output}
    \end{figure}

    \subsubsection{Fitting a morphable animal mesh to images}
    % Draw out the silhouette term as important here
    % Talk about subsequent SMALR + SMALST in the same section
    Unfortunately, none of the techniques above are suitable for reconstructing an animal category with significant shape diversity. Zuffi et al.~\cite{zuffi2017menagerie} made a significant contribution by releasing SMAL, a deformable 3D quadruped model (analagous to SMPL~\cite{loper15smpl} for human reconstruction) build from $41$ scans of artist-designed toy figurines. The authors also released shape and pose priors generated from artist data. The authors also discuss a small modification to the SMPLify~\lazycite{smplify}{smplify} approach in order to fit the SMAL model to RGB animal input images. However, an example result showing the result of the optimizer fitting the SMAL mesh to an RGB image of a fox can be seen in Figure \ref{fig:smalify}. The whole optimization process takes around 1 minute per frame.

    \begin{figure}[t] % Example image
        \center{\includegraphics[width=0.95\linewidth]{fitting_smal}}
        \caption{Fitting SMAL to a hand segmented animal, reprinted from~\cite{zuffi2017menagerie}.}
        \label{fig:smalify}
    \end{figure}

    The terms of the optimization largely mirror the work of SMALify, although the interpenetration term is ommited and joints are provided manually rather than by a CNN. Importantly, the optimization incoporates an additional \emph{silhouette} term which aligns the 3D model shape to a binary silhouette image extracted by a user. \Cref{chap:cgas} of this thesis shows silhouette terms are of particular importance when reconstructing accurate animal shape. The following definition describes the process for differentiably rendering a 3D model to form a silhouette image which can then be compared to the provided 2D silhouette:

    \begin{definition}[Differentiable Rendering]
        The process of generating a 2D image from a 3D polygon mesh is known as rendering and can be achieved through a process known as raytracing. Raytracing is a rendering technique able to generate photorealistic 2D images from the scene. It can be considered the opposite process by which the human eye perceives the world, as this method involves lines being cast outwards, beginning at a point known as the \emph{camera origin}. Figure \ref{fig:raycasting} shows a typical set up, in which rays are cast from the camera origin through each pixel on the image plane. The colour for the pixel is obtained by following the ray through the scene until a light source or non-reflective surface is reached, taking into account any reflections or non-opaque scene items. Due to the considerable comptuation required, the operation is often parallelized and assigned to the GPU. However, the technique is typically considered unsuitable for real-time rendering of complex scenes (due to complex ray paths) or when high resolution images (many rays required) are needed. However, for this work, scenes are typically made up of a single non-reflective, solid mesh surface and contain no complex elements (e.g.\ shadows, non-constant lighting.

        \begin{figure}[H] % Example image
            \center{\includegraphics[width=0.5\linewidth]{ray_trace}}
            \caption{Diagram showing raycast rendering.~\cite{rendering}.}
            \label{fig:raycasting}
        \end{figure}

        It is also worth noting that the standard method for raycasting is not differentiable, causing problems for differentiable optimizers (including neural networks). However, alternative rendering methods~\cite{loper2014opendr} are available for these purposes.
    \end{definition}

    The SMAL authors~\cite{zuffi2017menagerie} demonstrate their system by fitting their deformable 3D model to quadruped species. Subsequent work includes SMALR~\cite{zuffi_lions}, who fit to multiple images of an animal and incoporate similar freeform deformation term used to model clothing in 3D human pipelines. They show their technique, which takes advantage of multi-view constraints afforded by fitting to video sequences, allows high-quality reconstructions of more exotic animal categories with details not covered in the original SMAL 3D model training set.

    A common property of techniques discussed so far is the reliance on manual annotations (commonly 2D silhouettes and keypoint correspondences) at test time. 3D-Safari~\cite{Zuffi19Safari} overcome this by training a deep network on synthetic data (generated by applying SMALR~\cite{zuffi_lions} on a large input dataset) to recover detailed zebra shapes in the wild. \Cref{chap:cgas} of this thesis demonstrates a technique which overcomes the need for a large input dataset of joint annotations by training on synthetic data generated from the graphics model alone.

    A further drawback of the approaches is their reliance on a test-time energy-based optimization procedure, which is susceptible to failure with poor quality keypoint/silhouette predictions and increases the computational burden. \Cref{chap:wldo} of this method presents an automatic reconstruction method that overcomes the need for additional energy-based refinement, and is trained purely from single in-the-wild images.

    Tangential to these approaches is the work of Kulkarni et al.\lazycite{Canonical Surface Mapping}{Canonical Surface Mapping} who reconstruct pose (although not shape) parameters for an rigged mesh using a deep network, learning from geometry cycle consistency terms rather than annotated keypoints. However, the shape is fixed to the input mesh and pose is also often confused, particularly determining the difference between the legs. The results appear of significantly poorer quality than the results in \Cref{chap:cgas} and \Cref{chap:wldo} of this thesis. Also of consideration is the end-to-end network of Kanazawa et al.~\cite{kanazawa2018birds} and subsequent paper which overcomes the need for keypoints by capturing uncertainty in camera viewpoints. In addition, their network does not require an input 3D morphable model, although they do initialize a sphere model to a mean bird. In addition, the bird category exhibits more limited articulation than our dog category.

    % TODO: LASR: Learning Articulated Shape Reconstruction from a Monocular Video: https://arxiv.org/pdf/2105.02976.pdf, https://lasr-google.github.io/ mesh based

    % TODO:
    %Finally, there are techniques which do not produce a mesh~\cite{Agudo_2018_CVPR,novotny19c3dpo} or rely heavily on input 2D keypoints or video at test-time~\cite{vicente_3dv,Probst2018_ECCVa}.
    % Unsupervised Learning of Probably Symmetric Deformable 3D Objects from Images in the Wild
    % Canonical Surface Mapping via Geometric Cycle Consistency https://arxiv.org/abs/1907.10043

% \subsubsection{Literature review tables}

% The closest work in terms of scale is the category-specific mesh reconstruction of Kanazawa et al.~\cite{kanazawa2018birds}, where 2850 images of birds were reconstructed.  However doing so for the complex pose and shape variations of dogs required the advances described in this paper.

% Caves near Tubingham - first identified 3D models created by people.
% Then cave paintings

% Birth of photography, trying to understand animal behaviour
% Muybridge, 1878 in photography
% Marey 1882 in France

% Marr and Nisihara, 1978 -- modelling animals with cylinders
% Weizmann horse dataset, ECCV 2002
% MSRC dataset, ICCV 2005 - labelling animal parts, included cows
% ImageNet 2009 - animals lumped in
%  Caltech-UCSD Bird dataset - fine grained classification. Realised that animals were a special class of entity in the world, and have interesting features
% Crall et al. HotSpotter 2013 - identifying individual zebra by their patterns
% Cashman & Fitzgibbon 2013 - foundation of 3D animals, for are dolphins. Also giraffes apparently?
% SMAL model - zuffi et al
% Mathis - deeplabcut keypoint detection. avenues for scientific exploration
% Gravin et al. DeepPoseKit. Looking at collective behaviour of large swarms of animals.

% 3D cheetahs in the wild (7500 frames + 2D pose, 20K 3D frames)
% Horse keypoint detection. does imagenet performance correlate to pose estimation robustness. Horse10, Horse-C? -- mathis, biasi et al WACV 2021.
% AnimalPose dataset.
% iRodents dataset (on iNaturalist?)
% SuperAnimal pretraining? Ye et al. CV4Animals.
% MacaquePose

% end-to-end bottom-up multi-agent pose estimation


\documentclass{llncs}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{comment}
\usepackage{hyperref}
\usepackage{longtable}
\usepackage{stmaryrd}
\newcommand{\interp}[1]{\llbracket #1 \rrbracket}
\newcommand{\maps}{\colon}
\newcommand{\FinSet}{\mathrm{FinSet}}
\newcommand{\Set}{\mathrm{Set}}
\newcommand{\Cat}{\mathrm{Cat}}
\newcommand{\Calc}{\mathrm{Calc}}
\newcommand{\Mon}{\mathrm{Mon}}
\newcommand{\BoolAlg}{\mathrm{BoolAlg}}
\renewcommand{\Form}{\mathrm{Form}}
\newcommand{\leftu}{\mathrm{left}}
\newcommand{\rightu}{\mathrm{right}}
\newcommand{\send}{\mathrm{send}}
\newcommand{\recv}{\mathrm{recv}}
\newcommand{\comm}{\mathrm{comm}}
\renewcommand{\quote}[1]{``#1"}
\newcommand{\deref}[1]{\mathrm{eval}(#1)}
\newcommand{\op}{\mathrm{op}}
\newcommand{\NN}{\mathbb{N}}
\newcommand{\lpquote}{\ulcorner}
\newcommand{\rpquote}{\urcorner}
\newcommand{\quotep}[1]{\lpquote #1 \rpquote}
\makeatletter
\gdef\tshortstack{\@ifnextchar[\@tshortstack{\@tshortstack[c]}}
\gdef\@tshortstack[#1]{%
  \leavevmode
  \vtop\bgroup
    \baselineskip-\p@\lineskip 3\p@
    \let\mb@l\hss\let\mb@r\hss
    \expandafter\let\csname mb@#1\endcsname\relax
    \let\\\@stackcr
    \@ishortstack}
\makeatother

\title{Modal Logics via a distributive law}
\author{
Michael Stay\inst{1}\\
\and
L.G. Meredith\inst{2}\\
}
\institute{
  {Pyrofex Corp.}\\
  \email{\fontsize{8}{8}\selectfont stay@pyrofex.net}\\
  \and
  {Synereo, Ltd}\\
  \email{\fontsize{8}{8}\selectfont greg@synereo.com}
}
\begin{document}
\maketitle
\begin{abstract}
\noindent

\end{abstract}
\section{Introduction}

\subsection{The reflective higher-order $\pi$-calculus}

Lawvere theories are limited in that they only talk about products of sorts.  A lambda theory is a generalization of a Lawvere theory that has the ability to talk about function sorts like $A \Rightarrow B$ and sums like $A+B$ or $1 + A + A^2 + \cdots,$ more commonly denoted with the Kleene star $A^*.$  One can think of lambda theories as having access to a ``library'' that takes care of the details of bound variables and substitution for us so we do not have to implement all that machinery.  Much of the work on nominal logics has been about exactly this factorization \cite{GabbayMJ:picfm}.  A lambda theory Th(Calc) is a bicartesian closed 2-category equipped with an identity-on-objects functor from $(\FinSet/\Sigma)^{\op}$ to Th(Calc).  Models of Th(Calc) are functors to Cat that preserve all the structure.

The $\pi$-calculus was invented in the early 1990s by Robin Milner as a model of networks of processes with a dynamically changing topology; two processes initially unaware of each other can be introduced by a third process.  The reflective higher order $\pi$-calculus uses quoted processes as names; the term constructors for quote and eval replace the more traditional nu and replicate constructors.  We also add a ``comm'' term to restrict the contexts in which reduction can occur \cite{DBLP:journals/corr/StayM15}.

Here is a presentation of the multisorted lambda theory Th(RHOpi) for the reflective higher-order $\pi$-calculus:
\begin{center}
  \begin{longtable}{|p{0.3\linewidth}|p{0.7\linewidth}|}
    \hline
    Sorts:
    \begin{itemize}
      \item $N$ for names
      \item $P$ for processes
    \end{itemize}\bigskip
    Term constructors:
    \begin{itemize}
      \item $\mathsf{!}\maps N \times P^* \to P$
      \item \raggedright $\forall k \in \NN \; \mathsf{for}\maps N^{k} \times ((N^*)^k \Rightarrow P) \to P$
      \item $|\maps P^2 \to P$
      \item $0\maps 1 \to P$
      \item $\comm\maps 1 \to P$
      \item $\quotep{-}\maps P \to N$
      \item $\deref{-}\maps N \to P$
    \end{itemize}
    &
    Rewrites:
    \begin{itemize}
      \item $\alpha\maps (p_1 | p_2) | p_3 \Rightarrow p_1 | (p_2 | p_3)$
      \item $\beta\maps p_1 | p_2 \Rightarrow p_2 | p_1$
      \item $\iota\maps 0 | p \Rightarrow p$
      \item \raggedright $\chi\maps \mathsf{for}( x^1; \ldots; x^k, q) \;|\; \Pi_{i=1}^{k} x^1\mathsf{!}(p^i_1, \ldots, p^i_{n^i})\;|\; \comm \Rightarrow q([\quotep{p^1_1}, \ldots, \quotep{p^1_{n^1}}], \ldots, [\quotep{p^k_1}, \ldots, \quotep{p^k_{n^k}}]) \;|\; \comm$
      \item $\epsilon\maps \deref{\quotep{p}} \Rightarrow p$
    \end{itemize}
    Equations:
    \begin{itemize}
      \item \tshortstack[l]{$\alpha = P^3, \beta = P^2, \iota = P$ ($|$ and 0 form a \\ commutative monoid)}
      \item \tshortstack[l]{$\epsilon = P$ (evaluating a quoted process is the \\ same as the process itself)}
    \end{itemize}\\
    \hline
  \end{longtable}
\end{center}

For readability and consistency with common practice (see Scala
for-notation) we introduce syntactic sugar for the input guarded
process forms, writing it as

\[\interp{\mathsf{for}( \vec{y^1} \leftarrow x^1;\ldots ; \vec{y^k} \leftarrow x^k)\{ p \}} = \mathsf{for}( x^1, \ldots, x^k, \lambda \vec{y^1} \ldots \vec{y^k}.p)\]

The simplest RHOpi processes are 0, the ``do nothing'' process; and comm, a ``catalyst'' process that enables communication on a channel.  The only rewrite that is not an identity is $\chi,$ the communication event.  The $\chi$ rewrite is neither confluent nor deterministic.  For example, we can model contention for resources with the term
\[ \recv(x, P)\;|\;\recv(x, Q)\;|\;\send(x,R)\;|\;\comm \]
which has two rewrites out of it, one where the continuation $P$ is invoked on the name $\quotep{R}$ and the other where the continuation $Q$ is invoked on it.  We can model message arrival order nondeterminism with the term
\[ \send(x, P)\;|\;\send(x, Q)\;|\;\recv(x,R)\;|\;\comm \]
which has two similar rewrites out of it, one where the continuation $P$ is invoked on the name $\quote{R}$ and one where $P$ is invoked on the name $\quote{Q}$.

In the theory above, comm is preserved by the rewrites; one can think of each comm instance as representing a processor.  An alternative would be to consume comm in the $\chi$ rewrite; then comm would track clock ticks; an application of a consumable comm is the formal verification of billing code for tracking compute resources.

Replication of processes, and therefore general recursion, can be encoded \cite{DBLP:journals/entcs/MeredithR05} via
\[D(x) = \recv(x, y\mapsto \send(x, \deref{y}) | \deref{y})\]
\[!P = \send(x, D(x) | P) | D(x).\]

%% P    ::= { [P [| P]]}
%%       | for( Ptrn <- X [; Ptrn <- X] )P
%%       | X!( P )
%%       | *X
%%       | V
%% V    ::= Arry | Grnd
%% Grnd ::= Bool | Num | Str | ...
%% Ptrn ::= X | _ | ...
%% X    ::= @P | [Char]+

%% for( y1 <- x1 ){ for( y2 <- x2 )P }
%% for(

%% \begin{mathpar}
%%   \inferrule* [lab=process] {} {P \bc \{ [P [| P]] \}}
%%   \and
%%   \inferrule* [lab=summation] {} {{M,N} \bm \mathsf{for}( Ptrn
%%     \leftarrow X [] }
%%   \and
%%   \inferrule* [lab=agent] {} {{A} \bc (\vec{x})P \;| \; \clift{\vec{P}}}
%%   \and
%%   \inferrule* [lab=name] {} {{x,y} \bc \quotep{P}}
%% \end{mathpar}

\[ \interp{u \langle C\rangle v} = \{ t \;|\; \exists \rho\maps C[t, u] \Rightarrow v \}; \]

Caires' \cite{Caires} operator $\triangleright$ for rely-guarantee properties, the adjunct to $|,$ is an instance of our generic modal operator, where $u \triangleright v = u \langle - | - \rangle v:$
\[ \interp{u \triangleright v} = \{ t \;|\; \exists \rho\maps (t\;|\;u) \Rightarrow v\} \]

\[\interp{ \langle \quotep{\phi} \rangle \psi } = \{ t | \exists x\in\interp{\quotep{\phi}} \rho : x!(\top) | t \Rightarrow \psi \}\]

Then we have that $\langle x \rangle \psi$ is merely restricting $\quotep{\phi}$ to the singleton namespace consisting of $x$.

\[ \langle \rangle \psi = \interp{ 0 \langle -|- \rangle \psi } \]

\subsection{Three useful kinds of formulae}
\subsubsection{Responsiveness}
Always responsive to outside, but may contain internal deadlocks:
\[ \mbox{resp}(\quotep{\phi}) = \exists x \in \quotep{\phi} . \langle x\rangle(\top).\mbox{resp}(\quotep{\phi}) \]

Attempt to say ``no deadlocks anywhere'':

\[\begin{array}{rl}
  \mbox{Responsive}( \quotep{\phi} ) = & \\
  & \langle \quotep{\phi} \rangle \mbox{Responsive}(\quotep{\phi}) \\
  & \lor\; \mbox{outputOnly}\; | \;\mbox{Responsive}(\quotep{\phi}) \\
  & \lor\; \exists x. x?\top.\top \;|\; x!\top \;|\; \top \implies \langle\rangle \mbox{Responsive}(\quotep{\phi})\\
  & \lor\; \mbox{Responsive}(\quotep{\phi}) \; | \;\mbox{Responsive}(\quotep{\phi}) \\
\end{array}\]
where
\[ \mbox{outputOnly} = 0 \lor ( \quotep{\top} ! \top \;|\; \mbox{outputOnly} ) \]

Note that there can be infinite prefixes and liveness is undecidable.  E.g. for every exchange in the last line check some finitely refutable conjecture like Riemann's hypothesis or Goldbach conjecture and return to the first line only if a counterexample is found.

It is easy to see that a process that satisfies $\mbox{Responsive}$ will have a shape that
looks like

\[\Pi\mathsf{for}( v \leftarrow u )P \;|\; \Pi x!(Q) \;|\; \Pi I\]

where $u, x \in \quotep{\phi}$, but $\neg u \bot x$, $P \models \mbox{Responsive}(\quotep{\phi})$ and $\Pi I$ is communicates internally before becoming
$\mbox{Responsive}$. Building on this observation we can adapt the
formula to be parametric in a formula $\psi$, and in this way enforce
a property on the components that eventually communicate with the environment.

\[\begin{array}{rl}
\mbox{ResponsiveP}( \quotep{\phi}, \psi ) = \psi \land \mbox{RecResponsiveP}( \quotep{\phi}, \psi ) \\
\end{array}\]

\[\begin{array}{rl}
\mbox{RecResponsiveP}( \quotep{\phi}, \psi ) = & \\
  & \langle \quotep{\phi} \rangle \mbox{ResponsiveP}( \quotep{\phi}, \psi ) \\
  & \lor\; \mbox{outputOnly}\; | \;\mbox{ResponsiveP}( \quotep{\phi}, \psi ) \\
  & \lor\; \exists x. x?\top.\top \;|\; x!\top \;|\; \top \implies \langle\rangle \mbox{ResponsiveP}( \quotep{\phi}, \psi )\\
  & \lor\; \mbox{ResponsiveP}( \quotep{\phi}, \psi ) \; | \;\mbox{ResponsiveP}( \quotep{\phi}, \psi ) \\
\end{array}\]

\subsubsection{Containment}
\[ \mu X. \langle \quotep{\phi} \rangle ((X \lor 0)\;|\;\neg\langle\quotep{\neg \phi}\rangle\top)\;|\;\neg\langle\quotep{\neg \phi}\rangle\top \]
\subsubsection{Fuel}
Revisit liveness: prove there's enough fuel to get back to first line.

Computational complexity.

\bibliographystyle{amsplain}
\bibliography{ladl}
\end{document}


\section{Smol Zooid}

\subsection{Processes}

\begin{frame}
    \frametitle{Goals}
    \begin{sticky}%
    \vspace{-.7cm}
    \begin{enumerate}
    \item Certifying \textbf{individual} processes of a \textbf{distributed system}
    \item Extracting runnable code
    \item Avoiding complex formalisations of binders, whenever possible
    \end{enumerate}
    \end{sticky}
\end{frame}

\pgfdeclarelayer{background}
\pgfsetlayers{background,main}
\begin{frame}
    \frametitle{Overview}

  \centering
  \begin{tikzpicture}[commutative diagrams/every diagram]
    \begin{pgfonlayer}{background}
      \fill[rounded corners, yellow!20!white] (-0.8,0.5) rectangle (7.5,-3.5);
    \end{pgfonlayer}

    \node(G0)at (0,0) {$\G$};
    \node(G1)at (3,0) {$\coG$};
    \node(G2)at (6,0) {\small{global trace}};
    \node(L0)at (0,-1.5) {$\lT$};
    \node(L1)at (3,-1.5) {$\colT$};
    \node(L2)at (6,-1.5) {\small{local trace}};
    \node(P0)at (0,-3) {$\proc$};
    \node(P2)at (6,-3) {\small{process trace}};
    \node(OC)at (0,-5) {\small{OCaml code}};
    \node(ZZ)at (3.5,-5) {\bf \dslName};

     \path[commutative diagrams/.cd,every arrow,font=\scriptsize]
     (G0) edge node[above] {$\Re$} (G1)
     (G1) edge node[above] {LTS} (G2)
     (L0) edge node[above] {$\Re$} (L1)
     (L1) edge node[above] {LTS} (L2)
     (G0) edge node[right] {$\upharpoonright$} (L0)
     (G1) edge node[right] {$\upharpoonright^{\textsf{c}}$} (L1)
     (G2) edge[<->] node[right] {$=$} (L2)
     (L0) edge node[right] {$\ofLt$} (P0)
     (P0) edge node[above, xshift=-.5cm] {LTS} (P2)
     (P2) edge node[right] {erase} (L2)
     (P0) edge node[left] {extraction} (OC)
     (P0) edge[<->] node[right,xshift=.1cm] {DSL layer} (ZZ)
     ;
%      \path[dashed, red, commutative diagrams/.cd, font=\scriptsize]
%      (L0) edge[bend left=40] node[below left,yshift=.4cm, xshift=-.2cm]
%        {Well typed!} (ZZ)
%      (P2) edge[] node[below right] {Certified semantics!} (ZZ)
%      (OC) edge[] node[above] {Extractable!} (ZZ)
%      ;
  \end{tikzpicture}

\end{frame}

\begin{frame}
    \frametitle{Smol Zooid}
    \begin{greenbox}{}
    \begin{itemize}
    \item We combine \textbf{shallow/deep embeddings} of binders
    \begin{itemize}
    \item We use DeBruijn indices for the deeply embedded binders
    \end{itemize}
    \item SZooid constructs are \textbf{well-typed by construction}
    \item We leverage \textbf{Coq code extraction} mechanism
    \item For simplicity, SZooid does not cover \emph{choices}
    \end{itemize}
    \end{greenbox}
\end{frame}

\begin{frame}[fragile]
    \frametitle{Core Processes}
    In: \url{http://github.com/emtst/gentleAdventure}
\vspace{.5cm}
    \begin{minted}{coq}
Inductive proc :=
| Inact | Rec (e : proc) | Jump (X : nat)
| Send (p : participant) {T : type}
     (x : interp_type T) (k : proc)
| Recv (p : participant) {T : type}
    (k : interp_type T -> proc)

| ReadIO {T : type} (k : interp_type T -> proc)
| WriteIO {T : type} (x : interp_type T) (k : proc).
    \end{minted}
\end{frame}

\begin{frame}[fragile]
    \frametitle{Payload Types}
    We need to define a type for payload types:
    \begin{itemize}
    \item We need a decidable equality on payload types
    \item We need a decidable equality on payload values
    \end{itemize}
\vspace{1cm}
    \begin{minted}{coq}
Inductive type := Nat | Bool | ...
Definition interp_type : type -> Type := ...
    \end{minted}
\end{frame}

\subsection{Traces}

\begin{frame}[fragile]
    \frametitle{Semantics: events}
    The semantics is an LTS:
    \begin{itemize}
    \item the labels are the \textbf{communication events}
    \item it is parameterised by a \textbf{payload interpretation function}
    \item traces are obtained as the greatest fixpoint of the LTS step
    \end{itemize}
\vspace{.5cm}
    \begin{minted}{coq}
Inductive action := a_send | a_recv.
Record event interp_payload :=
  { action_type  : action;
    subj         : participant;
    party        : participant;
    payload_type : type;
    payload      : interp_payload payload_type
  }.
    \end{minted}
\end{frame}

\begin{frame}[fragile]
    \frametitle{Semantics: Recursion Variables and I/O}
    \verb|p_unroll : proc -> proc|
\vspace{.2cm}

\verb|p_unroll| exposes the first communication action in a process:
\begin{itemize}
\item ``runs'' any I/O action
\item unfolds recursion
\end{itemize}
\vspace{1cm}
    \begin{minted}{coq}
Definition p_unroll : proc -> proc := ...
    \end{minted}
\end{frame}

\begin{frame}[fragile]
    \frametitle{Semantics: step}
The step of the LTS is defined as a \textbf{function}:
\vspace{.5cm}
    \begin{minted}{coq}
Definition step' e E :=
  match e with
  | Send p T x k =>
    if (action_type E == a_send) && (party E == p) &&
       (eq_payload (payload E) x)
    then Some k else None
  | Recv p T k => ...  | _ => None
  end.
Definition step e := step' (p_unroll e).
    \end{minted}
\end{frame}

\subsection{Local Types}

\begin{frame}[fragile]
    \frametitle{Local Types}
    We introduce a typing discipline that associates processes with \textbf{local types}, that
characterise their communication behaviour:

    \begin{minted}{coq}
Inductive lty :=
  | l_end
  | l_jump (X : nat)
  | l_rec (k : lty)
  | l_send (p : participant) (T : type) (l : lty)
  | l_recv (p : participant) (T : type) (l : lty).
    \end{minted}
\end{frame}

\begin{frame}[fragile]
    \frametitle{Type System}
    \begin{minted}{coq}
Inductive of_lty : proc -> lty -> Prop :=
| lt_Send    p T k L x :
    of_lty k L -> of_lty (@Send p T x k) (l_send p T L)
| lt_ReadIO  T k L :
    (forall x, of_lty (k x) L) -> of_lty (@ReadIO T k) L
| ...
.
    \end{minted}
\end{frame}

\begin{frame}[fragile]
    \frametitle{Smol Zooid: Smart Constructors (I)}
    \begin{sticky}
\vspace{-.5cm}
    \begin{itemize}
        \item It would be tedious to type up both a local type and a process
        \item Users would need to provide a proof that processes are well-typed
    \end{itemize}
    \end{sticky}
    \begin{greenbox}{}
        We define \textbf{SZooid} (Smol Zooid), to write
well-typed processes by construction, avoiding repetition.
    \end{greenbox}
\end{frame}

\begin{frame}[fragile]
    \frametitle{Smol Zooid: Smart Constructors (and II)}
\begin{minted}{coq}
Definition SZooid L := { p | of_lty p L}.

Definition z_Send  p T x L (k : SZooid L)
  : SZooid (l_send p T L)
  := exist _ _ (lt_Send p x (proj2_sig k)).
...
\end{minted}
\end{frame}

\begin{frame}[fragile]
    \frametitle{Inferring Local Types}
    SZooid constructs fully determine their types from their inputs, so we can ask Coq to \emph{infer} local types
associated with SZooid terms:
\vspace{1cm}
\begin{minted}{coq}
Definition AZooid := { L & SZooid L }.
\end{minted}
\end{frame}

\subsection{Subject Reduction}

\begin{frame}[fragile]
\frametitle{Subject Reduction}
\begin{minted}{coq}
Theorem preservation (e : proc) (L : lty)
    (H : of_lty e L) (E : rt_event) :
  forall e',
    step e E = Some e' ->
      exists L', lstep L (ev_erase E) = Some L' /\
                 of_lty e' L'.
\end{minted}
\end{frame}

\subsection{Extraction}

\begin{frame}[fragile]
\frametitle{Extraction}
\begin{itemize}
\item We convert \verb|proc| to function calls in an \textbf{ambient monad}
\item We extract the monadic code to OCaml
\item The ambient monad needs to be implemented in OCaml
\item Processes are extracted using Higher-Order modules, so it is straightforward to change the underlying
transport
\vspace{.2cm}
\item \textbf{Remark:} SZooid does not provide an implementation of the ambient monad, but Zooid does, using TCP/IP sockets:
\url{https://github.com/emtst/zooid-cmpst}
\end{itemize}
\end{frame}

\begin{frame}[fragile]
\frametitle{Extraction Module}
\begin{minted}{coq}
Module ProcExtraction (MP : ProcessMonad).
  Fixpoint extract_proc (d : nat) (p : proc) : MP.t unit
    := match p with
       | Send p T x k
         => MP.bind (MP.send T p x)
                    (fun=> extract_proc d k)
       ...
End ProcExtraction.
\end{minted}
\end{frame}

\begin{frame}[fragile]
\frametitle{Example Extraction}
\begin{minted}{coq}
Module  ALICE (MP : ProcessMonad) : PROCESS(MP).
  Module PE := ProcExtraction(MP).
  Definition proc :=
    Eval compute in PE.extract_proc 0 alice.
End ALICE.

Extraction ALICE.
\end{minted}
\end{frame}

\begin{frame}[fragile]
\frametitle{Summary}
\begin{sticky}
\begin{itemize}
\item We have seen how to encode a small calculus of Multiparty Processes, with a basic type system
\item \textbf{Next: how do we relate traces of individual processes to a larger system?}
\end{itemize}
\end{sticky}
\end{frame}


\section{Related Work} \label{sec:related-work}
We summarize three closely related papers and point out similarities and differences with our
work.
The first paper describes another distribute \textsc{WCOJ} on Timely Dataflow.
The second, compares multiple worst-case optimal join algorithm parallelization
approaches to determine the best given a specific use-case.
Third we describe a general graph pattern mining system build on Spark which also
employs work-stealing.

\subsection{\textsc{WCOJ} on Timely Data Flow}\label{subsec:wcoj-timely-data-flow}

Mc Sherry et al. published a distributed worst-case optimal join based on Timely Data Flow in 2018~\cite{ammar2018distributed,naiad}.
In their paper they introduce three algorithms: \textit{BigJoin}, \textit{Delta-BigJoin} and \textit{BigJoin-S}.
They implement only the first two algorithms.

\textit{BigJoin-S} is only described not implemented but comes with stronger theoretical guarantees.
Namely, it is worst-case optimal in computation and communication with respect to the output size of the query
given by the AGM bound.
\textit{BigJoin-S} can guarantee work balance.
Moreover, it achieves optimality and work-balance while using low amounts of memory on all workers;
the memory usage per worker is in $\mathcal{O} (\frac{IN}{w})$ with $IN$ size of the input relationships and $w$ the number of workers.

The other two join algorithm are only worst-case optimal in computation and communication costs but
do not guarantee work-balance nor do they give the same low memory guarantees.
Although, in praxis, they achieve both on many real-world datasets.

\textit{Delta-BigJoin} is an incremental algorithm which computes the new instances of the subgraph given a batch of new edges.
Hence, it operates in a different setting than our work.
We assume static graphs while they operate on graphs with the ability to find new instances caused by insertions.

\textit{BigJoin} is closest to our work.
It has been implemented and is a worst-case optimal join for static graphs.

In the following paragraphs, we describe \textit{BigJoin}, analyse why it is not likely to be a good fit for Spark,
discuss and compare the index structures used in their work to represent the input relationships and compare the
guarantees are given by them and us.

\subsubsection{The \textit{BigJoin} algorithm}
\textit{BigJoin} encodes a \textit{Generic Join} (see~\cref{subsec:worst-case-optimal-join-algorithm}) into multiple
timely dataflow operators.

In short, Timely Dataflow operators are distributed over multiple workers and each of them takes a stream of input data, operates
on it and sends it to the next operator which can be processed on a different worker.
Examples for operators are \textit{map} functions, \textit{filters} \textit{count} or \textit{min}.
It is important to note that sending the output to an operator on a different worker is a fast, streaming operation, as opposed to,
Spark's shuffles which are synchronous and slow because they involve disk writes and reads.

For the \textit{BigJoin} the authors require each worker to hold an index for each input relationship which maps prefixes
of the global variable order to the possible bindings for the next variable.
In use-case of graph pattern matching, this means that each worker holds an index into the forward and backward adjacency lists.

Their algorithm runs in multiple rounds;
one per variable in the join query.
In each round, they bind one variable.
So each round takes the prefixes as input and fixes one more binding.

A single round starts with all prefixes from the former round distributed among all workers arbitrarily.
Then they find join relations that offers the smallest set of extensions for each prefix.
This is done in steps with one step per relationship.
In each step, the prefix is sent to a worker by the hash of the attributes bound in the relationship of that step.
When the relationship offers less possible values for the new binding then the current minimum, i.e. the size of its matching adjacency
the list is smaller, we remember it as the new minimum for the given prefix.

Next, they hash the values of the prefix which are defined in the relationship with the least extensions and use these hashes to
distribute the tuples over all workers.
Then, each worker produces all possible extensions for each prefix.

Finally, each round ends with filtering out all extensions that are not in the intersection of extensions offered by each relationship.
This again takes one filtering step per relationship in the join.

This is a simple instance of the \textit{Generic Join} implemented in data flow operators.

The algorithm described so far can build a high amount of possible extensions in each round.
This keeps it from keeping worst-case optimal guarantees for memory usage.
The authors fix this problem by batching the prefixes:
they allow only a certain number of prefixes in the system at all times.
They defer building new prefixes until the current batch of prefixes has been completed.
This is natively supported by Timely Dataflow.

\subsubsection{Applicability to Spark and comparison to GraphWCOJ}
\textit{BigJoin} is not suitable for Spark.
This has multiple reasons.

Most importantly, it uses too many shuffle rounds.
Each round and each step in a round requires communication and therefore a shuffle.
In total, the algorithm uses $2R \times V$ rounds for a query with $R$ relations and $V$ variables.
As pointed out before this is no big problem in Timely Dataflow because shuffles are fast and asynchronous.
However, in Spark, this is not the case.

We would like to point out that binary join plans can solve the same queries in $R - 1$ shuffle rounds and
that our solution does not require any communication rounds.

Second, Spark does not support batching queries naturally as Timely Dataflow.
Building support for batching into Spark would be an engineering effort.
Additionally, it would be hard to define a good user interface over a batched query in Spark.

\textsc{GraphWCOJ} does not require batching because it only processes at most as many prefixes as workers in the system in parallel.
Therefore, we do not have the problem of memory pressure to remember prefixes.
This is because the \textsc{LFTJ} algorithm is a non-materialized representation of the join.
When the Leapfrog Triejoin is executed, it changes its state such that the state always represents the non-materialized
part of the join;
the state is encoded in the positions of the \textit{TrieIterators}.
In other words, the \textsc{LFTJ} performs a depth-first search of all possible bindings while the \textit{BigJoin}
performs a batched breadth-first search.

\subsubsection{Indices used by \textit{BigJoin} and GraphWCOJ}
The index structures used in their and our solutions are the same; one forward and one backward index over the whole graph on each
worker.
It is possible to distribute the index of \textit{BigJoin} such that each worker holds only a part
of the index.
This is because each worker needs to hold only the possible extensions for the prefixes that map to it for each relationship.
We analyse this in the next paragraph.

The prefixes are mapped to workers by the hash of the attributes already bound.
For graph pattern matching this is one or zero attributes;
the edge relationship has two attributes and one is a new, yet unbound variable in the prefix.

We can reach a distribution of the indices such that each worker holds $\frac{I}{w}$ with $I$ the size of the indices.
For that, we choose the same hash function for each variable such that always the same values match to the same worker.
However, this solution is likely to lead to high skew and work imbalance because if a value is a heavy hitter the
worker needs to process it for each binding over and over.

It is better to use different hash functions per variable.
In this case, we can estimate the percentage of the whole index hold by each worker by the binomial distribution.
This distribution models the probability that out of $N$ independent trials $k$ succeed with the likelihood of $p$ for a single trial
to succeed.
We model the event of a key from the index being assigned to a worker as trial.
The likelihood is $\frac{1}{w}$.
We have as many trials as variables in the join: $N = V$.
We are interested in the case that the tuple is not assigned by any of the variables, so $k = 0$.
Then we have the likelihood that a tuple is not assigned given by $\mathcal{B} (V, 0, \frac{1}{w})$;
so the fraction of the indices assigned to each worker is $1 - \mathcal{B} (V, 0, \frac{1}{w})$.

We plot this function for different numbers of workers and variables in~\cref{fig:big-join-indices}.
The split of the indices held by each worker decreases drastically with the numbers of workers in
the system.
Hence, this partitioning scheme scales relatively well.

\begin{figure}
    \centering
    \includesvg[width=0.5\textwidth]{svg/big-join-indices}
    \caption{Expected split of the indexes hold on each worker for different numbers of workers used
    and variables in the query.
    }
    \label{fig:big-join-indices}
\end{figure}

\subsubsection{Theoretical guarantees}
\textit{BigJoin} guarantees computational and communication worst-case optimality.
However, the communication optimality does not take into account how the indices are generated on each worker.
If they are sent to each worker, this would be not worst-case optimal.
The extensions of \textit{BigJoin-S} additionally give work-balance and low memory usage in $\mathcal{O} (\frac{IN}{w})$.

GraphWCOJ guarantees computational worst-case optimality which it inherits from \textsc{LFTJ}.
Worst-case optimal communication is given by the fact that we do not communicate.
This is if we do not take the distribution of the indices into account which is in line with the analysis of the discussed paper.

If we take the distribution of indices into account, our algorithm is not worst-case optimal.
During our setup, we broadcast the indices used.
This is not optimal for a single query;
Shares would be optimal.
However, it amortizes quickly over multiple queries, while Shares converges to broadcasting for big queries.

We can not guarantee work-balance.
However, we get close to it by using work-stealing.
With work-stealing, we are optimal within the size of a single task.

GraphWCOJ's memory footprint does not depend on the size of the input nor of the output of the join.
Its memory usage is given by the size of the Java objects used which depend on the query.
However, this size should be neglectful small for all but embedded use-cases.

\subsubsection{Conclusion}
We conclude that our approach is the better fit for Spark because it requires less shuffling and no batching.
GraphWCOJ gives nearly the same theoretical guarantees as \textit{BigJoin}.
While they can distribute their indices we cannot;
we rely on the fact that each worker holds the complete index.

Finally, we would like to point out that~\cite{ammar2018distributed} does not publish any number on the amount
of network traffic caused by their algorithm.
Given that it sends many prefixes via the network this could be a bottleneck in many deployments, i.e. in cheaper instances
in the Amazon cloud.
An analysis of the network traffic would be beneficial for a better understanding of the advantages and disadvantages
of their approach.

% General comparision?
% rather not
% but if so, state tracked, number of prefixes in the system at any time,
% our system is monolith, theirs is built of simple operators

% Experiments
% Scaling of BigJoin not given, cannot be compared
% Single threaded on twitter graph (big one) (need to double check which) and LJ, LJ needs 6.5s
% BigJoin 8 workers 16 cores each, takes 3.4 s to find all triangles in LJ
% BigJoin 10 machines 16 cores each 4-clique, house, 5-clique. they do not report dataset, maybe i can find the dataset int
% seed paper
% no experiment regarding communication costs

% Implemenation: https://github.com/frankmcsherry/dataflow-join

\subsection{Survey and experimental analysis of distributed subgraph matching}
On the 28th July 2019, L. Lai et al. published a survey with experiments for multiple
distributed graph pattern matching algoritms~\cite{longbin}\footnote{The survey was published on arXiv 5 months after we started our
thesis in
February.}.
Here, we focus on four of the strategies they tested: \textit{BigJoin} (see \cref{subsec:wcoj-timely-data-flow}),
Shares, fully replicated graph and binary joins.
All of their algorithms are implemented in Timely Dataflow;
so far they are not open-source.
They ran the all algorithms on 9 different queries over 8 datasets mostly on a cluster of 10 machines
and 3 workers per machine.
Below we first summarize the most important design decisions for each algorithm, then
highlight their most interesting findings and finally compare their results with ours.

\textit{BigJoin} is implemented as described above but uses a \textsc{CSR} data structure,
triangle indexing and a specific form of compression as optimization.

The Shares algorithm is configured as described in~\cref{subsubsec:shares} and uses
\textit{DualSim} as the local algorithm.
\textit{DualSim} is a specialized subgraph matching algorithm.
The authors show that it beats the worst-case optimal join used in \textit{EmptyHeaded}~\cite{emptyheaded}
which is a form of the \textit{Generic Join} (see \cref{subsec:worst-case-optimal-join-algorithm}).

The survey also covers our strategy of fully replicating the graph on all machines.
They choose \textit{DualSim} as a local algorithm and a round-robin partitioning on the
second join variable.

Finally, they implement the binary joins with hash joins and use a sophisticated query optimizer
to devise the best join order.

The most important finding of this work is that fully replicating the graph on all machines
is the best option if the graph fits into memory even in Timely Dataflow with its deeply
optimized and asynchronous communication routines.
They establish that fully replicating the graph is nearly always the fastest strategy,
has the lowest memory footprint\footnote{Shares replicates too much data,
\textit{BigJoin} needs to hold many prefixes in memory and binary joins incur intermediary results.}, no
further
communication costs and scales better
than
all other strategies up to 60 workers.

In line with our argument against Shares, they find that this strategy is nearly always
beaten by most other strategies.
They establish that it takes longer than the other strategies on nearly all queries and datasets.
Furthermore, it shows the weakest ability to scale.

They report that \textit{BigJoin} or binary joins are the best option if fully
replicating the graph is out of the question.
Binary joins can be used for star and clique joins if it is possible to index all triangles in
the graph and keep this index in memory.
Otherwise, \textit{BigJoin} is preferable in most cases.

Finally, they study the communication costs of the binary joins, Shares and \textit{BigJoin}.
They find that graph pattern matching is computation bound problem when 10 GB switches are used
for networking but communication costs dominate in 1 GB switched networks.
They draw their conclusions from experiments run with 10 GB network infrastructure.

Interestingly, Shares incurs fewer communication costs than \textit{BigJoin}.

Their paper differs from our thesis in multiple ways.

We implement our system in Spark which has wide-spread usage in industry and a surrounding eco-system
of graph pattern matching systems (see \cref{subsec:graphs-on-spark}).

We give a comparison between a column store, binary search based Leapfrog Triejoin and
our \textsc{CSR} based GraphWCOJ.
They do not report on the benefits of \textsc{CSR} in the context of \textsc{WCOJ}.

Their implementation of the fully replicated strategy differs from ours in two important factors.

First, they use a different local algorithm called Dual Sim~\cite{dualsim}.

Second, they use a different partitioning scheme.
Their scheme replicates work of finding bindings for the first and second variable in a query and does not actively counter skew.
The skew-resilience of their scheme is based on the fact that it partitions the work on the second
binding.
Hence, it distributes skew of the first binding equally.
However, as we see with our work-stealing approach this does not guarantee skew freeness for
bigger queries (see \cref{subsec:scaling-graphWCOJ}).

Their scheme could be applied to our system.
It is simpler than work-stealing but less resilient to skew.

However, they come to the same conclusions, namely that fully replicating is the preferred strategy
when the graph fits into main memory and that Shares is not a good strategy for graph pattern matching.

\subsection{Fractal: a graph pattern mining system on Spark} \label{subsec:fractal}
Fractal is a general-purpose graph pattern mining system built on top of Spark published at SIGMOD'19~\cite{fractal}.
We first describe the relevant aspects of their system.
Then we compare it to our approach.

Graph pattern mining includes the problem of graph pattern matching (as defined in~\cref{subsec:graph-pattern-matching}).
Additionally, it includes problems such as frequent subgraph mining or keyword-based subgraph search.

To support all these problems in a single system, the authors describe their own programming interface made
off initialization operators, workflow operators and output operators.
Each workflow is described as a sequence of these operators.
All workflows are based around extending a subgraph starting from a single edge, vertex or a user-described
pattern.
This makes for three initialization operators one to start from a vertex, edge or pattern each.

The workflow operators process the subgraphs induced by the initialization operators.
They can expand the subgraph, e.g. if the subgraph is vertex induced, one expand step adds all neighbouring
vertices to the subgraph.

Another workflow operator is to filter the subgraph instances.
Then it is possible to aggregate subgraphs by computing a key, a value and possibly a reduction.
Finally, these workflow steps can be looped to be repeated multiple times.

To execute the workflow the user can use one out of two output operators: \textit{subgraphs} and
\textit{aggregation} to list all matching subgraphs or aggregate all matching subgraphs respectively.

For example, the workflow $vertexInduced().filter(sg => fully connected).subgraphs()$ enumerates
all cliques in a graph.

Fractal maps these workflows to Spark by splitting them into \textit{fractal steps} on
synchronization points, e.g. an aggregation which results is required in the next step.
Each step maps to a Spark job which is scheduled by the Spark scheduler.
Fractal schedules the \textit{fractal steps} in the correct order and waits for them to complete
before starting the next one.

A typical problem of graph pattern mining is the high amount of memory needed to keep partial matches;
the state of the algorithm.
Fractal counters this problem by enlisting subgraphs with a depth-first strategy.
Furthermore, it starts computing all subgraphs from scratch for each step, instead of keeping them in
memory in between the steps.
They only keep the results of the aggregations to be used by the next step.

Another problem in graph pattern mining is work-balance because some parts of the graph are
more work-intensive than others.
Fractal tackles this problem with work-stealing.
They use a hierarchical work-stealing approach.
First, each thread tries to steal work from another thread within the same Spark executor.
Only if this is not possible, they request work from another machine.

The local work-stealing is implemented by sharing the same subgraph enumerators;
an iterator-like data structure that saves the state of the subgraph matching algorithm in a prefix
match.
The subgraph enumerator offers a thread-safe method to generate the next prefix.
Hence, a thread can steal work simply by using the enumerator of another thread.

The second hierarchy of work-stealing is between multiple Spark workers.
The authors support that by using Akka to implement a simple message passing interface between
all Spark workers.

Their experiments show near-linear scaling for the described work-stealing strategy up to 280
execution threads over 10 machines.

Fractal is similar to our system in some aspects.
They also solve graph pattern matching on Spark, inspired our approach to work-stealing and
choose a depth-first subgraph enumeration approach.
We discuss these similarities below and outline the differences.

Like us, Fractal solves the problem of graph pattern matching.
However, they offer the ability to solve multiple other common subgraph related problems as well.
The biggest difference is that they directly support aggregation over subgraphs, which, for example,
allows them to solve frequent subgraph mining.

They build an independent system on top of Spark's infrastructure which comes with their own
imperative query language.
We integrate a single algorithm deeply into Spark's query optimizer.
Therefore, our contribution can be easily integrated into other graph systems building on Spark, e.g.
G-Core~\cite{gcore} or CAPS~\cite{caps}.
These systems would offer a declarative interface to our worst-case optimal joins.

The work-stealing approach of Fractal inspired our solution.
We also use a shared object to steal work within a single Spark executor.
Anyhow, our approach is simpler and less fine-grained.
They allow stealing work at every level of the depth-first enumeration of all subgraphs.
We only share work on the first level.
This makes their solution strictly more fine-grained and likely to perform better on big
queries.
We discuss this issue in our future work section~\ref{subsubsec:finer-grained-work-stealing}.

Fractal is built for Spark in cluster mode.
Hence, they allow processes to steal work from different workers.
GraphWCOJ is currently limited to a single worker.
However, their message-passing based solution is directly applicable to our system if
we extend to multiple workers.
Again, we talk about this in more depth in future work (\ref{subsubsec:cluster-mode}).
In short, we could use the same message-passing implementation but instead of stealing
from a subgraph enumerator, the work would be taken from the queue on each worker.

Both systems enlist the subgraphs in a depth-first like fashion.
In both systems, this is highly beneficial to memory usage;
the problem of breadth-first algorithms has been discussed for \textit{BigJoin} before (\cref{subsec:wcoj-timely-data-flow}).

To conclude, Fractal is a complete system with its own query interface.
This makes it more powerful than our system but also less integrated into Spark.
Hence, it forces the user to adapt to their imperative language and hinders
integration with declarative graph query languages.
We used a similar work-stealing algorithm in our work.
Both systems recognize and demonstrate the advantages of a depth-first approach.

%\subsection{Adaptive Query Exectution}

% Code generation
%A novel development in Spark is the ability to generate code to execute queries on the fly, called WholeStage code-generation, based on a technique used in the Hyper database~\cite{hyper,jira-whole-stage,1m-rows-laptop}.
%Compiled queries have been shown to be multiple magnitudes faster than interpreted queries traditionally used in most database systems.
%Interpreted queries are most commonly implemented using the Volcano model~\cite{volcano}.
%This model provides a simple and composable interface for algebraic operators; basically, every operator would provide an iterator interface.
%This interface would be used by a query by calling next on the root operator, who in turn calls next on each of its children and so on until the next calls reach the scanning operators at the bottom of the query execution tree.
%These would provide a single tuple which would then be "pulled" upwards through the query tree and processed by all operators.
%When it reaches the root operator, the result is delivered to the user.
%This happens for every tuple; hence, the approach can be described as tuple-at-a-time.
%Although, this interface is simple yet powerful due to its composability, it is also quite computation intensive mainly due to the high number of calls to the next function, which is often a virtual function call.
%This high number of virtual function call is not only CPU intensive but also makes bad use of CPU registers (they are spilled on every function call) and hinders compiler optimizations.
%Compiled queries avoid these costs by generating code specific to each query consisting mainly out of multiple, tight for-loops following each other.
%This speeds up processing by keeping data in the CPU registers as long as possible and avoiding materilization and function calls.
%Furthermore, it allows compiler optimizations, such as loop unrolling or ~\cite{hyper}
%We are not aware of any published efforts to speed up worst-case optimal joins via code generation.

%We aim to combine the research on worst-case optimal join algorithm and Spark's extensible optimizer Catalyst to speed up graph processing for all Spark users.
%In particular, this work will be based on either of the two distributed versions of worst-case optimal join algorithms mentioned above.
%We hope to further their work by evaluating which approach (shuffle + local join or timely data flow) works best on a MapReduce based processing engine as well as proving that worst-case optimal join algorithms
%can improve performance on a complex, optimized, existing platform that has not been built with them in mind originally, albeit their high additional cost (e.g. for sorting and need for special data structures).


\section{Feature extraction}
\label{feature-extraction}

This Sections presents the original dataset structure and the
steps followed to create training and test sets from it.

Note that the models mentioned in this section, for the first and
extended datasets are
three layers \emph{multilayer perceptrons},
a feed forward neural network where each layer
is densely connected  to the following,
with a reasonable number of neurons, trained
for 100 epochs with default parameters for the \emph{stochastic gradient
descent optimizer}.~\cite{mlp}\cite{sgd}\\
A \emph{convolutional neural network}, a network with
a series of convolutional and pooling layers, followed by
some densely connected ones, is applied to the image dataset, trained for 15 epochs
with default parameters for the Adam optimizer.~\cite{cnn} \\
For all datasets, the accuracy mentioned is computed with a \emph{cross-validation}
approach.~\cite{cross}\\

A deeper discussion about the models structure as well as the validation
techniques used in the project can be found at Section \vref{model-definition}.

\subsection{Dataset structure}
\label{dataset-structure}

The UrbanSound8k dataset contains ten folds of audio samples, each one about
four seconds long. The samples are divided in ten classes.\\
From the total of ten folds, the number one, two, three, four and six
are taken as a training set, the others are each one a test set.
For this reason the following count about class numerosity considers
only the five training folds.

\begin{center}
    \begin{tabular}{ |l|c| }
        \hline
        Class name & Number of samples \\
        \hline
        air conditioner & 500 \\
        car horn & 208 \\
        children playing & 500 \\
        dog bark & 500 \\
        drilling & 500 \\
        engine idling & 517 \\
        gun shot & 190 \\
        jackhammer & 548 \\
        siren & 536 \\
        street music & 500 \\
        \hline
    \end{tabular}
\end{center}

The table shows a clear class imbalance. In particular, the classes
\emph{car horn} and \emph{gun shot} are not as numerous as the others.
This can lead to poor performances on the these two categories, it is
therefore taken
into consideration with training.

The following table shows the number of samples in the training set and
the various test sets.

\begin{center}
    \begin{tabular}{ |l|c| }
        \hline
        Dataset & Number of samples \\
        \hline
        Training set & 4499 \\
        Test set 5 & 936 \\
        Test set 7 & 838 \\
        Test set 8 & 806 \\
        Test set 9 & 816 \\
        Test set 10 & 837 \\
        \hline
    \end{tabular}
\end{center}
All the operations on the datasets are performed with \emph{Pandas} library.~\cite{pandas}

\subsection{First dataset}
\label{first}
Extracting features from audio files is not straightforward, nonetheless there
are a collection of audio characteristics that are commonly used in audio machine learning
applications.~\cite{features}

To extract information from audio files \emph{Librosa} is used.~\cite{librosa}
The library provides many methods to choose from,
to keep it simple, for the first try with this dataset, the extracted features
are these three ones:
\begin{enumerate}
    \item \emph{Mel-frequency cepstral coefficients}: the first 20 coefficients
    are extracted;
    \item \emph{Chromagram}: the first 12 chromas are considered;
    \item \emph{Root-mean-square}.
\end{enumerate}
Method parameters that are not specified in the above list, are left on their default values.
Each feature consists of an array of arrays containing measurements.
A series of functions are applied to each sub-array and results
are concatenated in a final feature vector.
The functions applied are \emph{minimum}, \emph{maximum}, \emph{mean}
and \emph{median} from the \emph{Numpy} library.~\cite{numpy}

This approach results in 132 components feature vectors.

\paragraph{Parallelizing feature extraction}
Extracting the three features listed above is really intensive
but the task is easily parallelizable, in fact, each file is independent
from one another.

For this purpose \emph{Dask} is used to speed up the computation and
extract features from audio files in a multi-processing fashion.~\cite{dask}
The main idea is to build an execution plan, where each audio file
is managed in parallel by a collection of workers.
Improvement is great as the time to process a single fold
is cutted into a third.

\paragraph{Feature scaling}
After testing some neural networks on the first dataset results
are not promising. One of the reasons is the big difference in
ranges among feature vector components, for instance,
some audio characteristics are in the order of thousands while others
range between zero and one.

To mitigate this effect a \emph{StandardScaler} with default parameters from \emph{scikit learn} is applied.~\cite{scaler}
The result is a dataset where each feature has more or less a distribution
centered in zero with unit variance.
This leads to an improvement on the results using the same
model as before.

\subsection{Extended dataset}
\label{extended-dataset}

Results using the three features named in the previous Subsection
are promising but not enough, thus, to improve accuracy on the training set,
new audio characteristics are extracted, namely:
\begin{enumerate}
    \item \emph{Zero-crossing rate};
    \item \emph{Roll-off frequency};
    \item \emph{Spectral flux onset strength}.
\end{enumerate}
As before methods parameters are left on default, but
this time \emph{standard deviation} is added to
the previous functions \emph{minimum}, \emph{maximum}, \emph{mean}
and \emph{median}. Each one is applied to the feature sub-vectors and
results are concatenated, leading to a total of 180 features for each audio file.
Scaling yields to promising results on the first dataset,
so the same approach is applied to the extended one.

After testing a network on the new training set
we can see a better accuracy.

\paragraph{Feature selection}
Adding new features can lead to better results
in the end but they all need to be useful to the model.
For this reason the extended dataset is subject of some experiments
with feature selection, in particular \emph{PCA} algorithm from scikit
learn is applied.~\cite{pca}

The main idea is to select a reduced number of features from the total,
loosing as little information as possible. This approach often leads to better results,
as useless features are discarded.

The method used to select features offers the possibility to specify how much
variance to preserve in the reduced dataset, this means that the number of
components is not given explicitly, indeed we try to preserve 99 percent of the original variance.

This technique resulted in 102 features, unfortunately, a model applied to this new dataset
failed to reach the performances obtained by the previous one, nonetheless, performances
improved with respect to the scaled dataset.

\subsection{Image dataset}
A completely different approach is followed this time to represent audio files
as images, to later apply a convolutional neural network.

\paragraph{Audio as an image}
Representing an audio as an image is not straightforward but can be done after
some preprocessing.
The main idea is to use audio features and consider them in a two dimensional
space, where the value of a single cell can be viewed as a pixel.
Note that some features are well suited for this kind of representation, for instance,
some of the previously extracted ones are given in output as a two dimensional vector.

Usually, an image has three channels, red, green and blue, but
image classification on grayscale images, with one single channel
is also relevant. This means that one can extract multiple features and view them
as different channels, or choose only one to have the equivalent of a grayscale image.

\paragraph{Short time Furier Transform}
For this try with image classification the Short-time Furier transform
is extracted from the audio files, and then used as a single channel image.
In particular, we extract 128 intervals with the \emph{stft} function from Librosa, and, to accomodate
the different lenghts of audio in the dataset, pad the result to 256 wide vectors.
The result is an $128 \times 256$ image for each sample. Figure \ref{img}
shows an example of an image obtained for each class.

\begin{figure}
    \includegraphics[width=\textwidth]{images/class_images.png}
    \caption{Short time Furier transform on samples coming from each class.
    The scale on the Y axis is logarithmic, the X axis represent time and
    the black portion on the edges is due to the applied padding.}
    \label{img}
\end{figure}

We can see some difference between the classes, for instance, the \emph{dog barking}
shows short repeated sounds while \emph{air conditioner}
is constant. The hope is that those subtle differences are enough lo learn and classify.

\paragraph{Scaling pixel values}
Scaling techniques  are usually applied even to images to prevent high disparities
among pixel value ranges, for this reason a Standard scaler with default parameters is
once again applied to the training set to mitigate this effect.
We can not apply it directly to the images, as they are two dimensional, but we can
flatten the pixel arrays, apply scaling and reshape them in the original $128 \times 256$ format.

Testing a convolutional network
on the image dataset gives results that are comparable to the ones obtained on the initial scaled training
set, discussed at Subsection \vref{first}.
\newpage