\chapter{Theory}

In the following sections, examples of a figure, an equation, a table, a chemical structure, a list, a listing and a to-do note are shown.
These examples are not particularly well-made\dots%
\footnote{Here is another footnote to illustrate cross-chapter numbering.}

\section{Figure}
\begin{figure}[H]
    \centering
    \includegraphics[width=0.45\linewidth, trim=3cm 11cm 3cm 11cm]{chapters/theory/X.pdf}
    \includegraphics[width=0.45\linewidth, trim=3cm 11cm 3cm 11cm]{chapters/theory/Y.pdf}
    \caption{Surface and contour plots showing the two dimensional function $z(x,y)=\sin(x+y)\cos(2x)$.}
\end{figure}

\section{Equation}
\begin{equation}
    f(t)=\left\{%
    \begin{array}{ll}
        1,\qquad & t< 1 \\
        t^2 & t\geq 1
    \end{array}\right.
\end{equation}

\section{Table}
\begin{table}[H]
    \centering
    \caption[This (instead of the potentially long caption) appears in the list of tables.]{Values of $f(t)$ for $t=0,1,\dots 5$.}
    \begin{tabular}{lllllll}
        \toprule
        $t$ & 0 & 1 & 2 & 3 & 4 & 5 \\ \midrule
        $f(t)$ & 1 & 1 & 4 & 9 & 16 & 25 \\ \bottomrule
    \end{tabular}
\end{table}

\section{Chemical structure}
\begin{center}
    \chemfig{X*5(-E-T-A-L-)}
\end{center}

\section{List}
\begin{enumerate}
    \item The first item
    \begin{enumerate}
        \item Nested item 1
        \item Nested item 2
    \end{enumerate}
    \item The second item
    \item The third item
    \item \dots
\end{enumerate}

\section{Source code listing}
An example of a source code listing.
The output can be drastically improved with the optional settings.
\lstset{language=Matlab}
\begin{lstlisting}[frame=single]
% Generate x- and y-nodes
x = linspace(0, 1);
y = linspace(0, 1);

% Calculate z = f(x, y)
for i = 1:length(x)
    for j = 1:length(y)
        z(i, j) = x(i) + 2*y(j);
    end
end
\end{lstlisting}

\section{To-do note}
The \texttt{todo} package enables to-do notes to be added in the page margin. This can be a very convenient way of making notes in the document during the process of writing. All notes can be hidden by using the option \emph{disable} when loading the package in the settings. \todo{Example of a to-do note.}


\documentclass[10pt,conference,compsocconf,a4paper]{IEEEtran}

\usepackage{graphicx}
\usepackage{amsmath, amssymb}
\usepackage{commath}
\usepackage{booktabs}
\usepackage{siunitx}
\usepackage[labelformat=simple, labelfont=normalfont]{subcaption}  % Side-by-side figures
\usepackage[labelfont=sc]{caption}  % Using captionof outside of figure environment
\usepackage[colorlinks, bookmarks=false, citecolor=black, linkcolor=black, urlcolor=blue]{hyperref}  % Cite colors + autoref
\usepackage{xurl}  % hypen breaks in urls

\newcommand{\G}{\mathcal{G}}
\renewcommand{\vec}[1]{\boldsymbol{#1}}
\newcommand{\unit}[1]{\ [\si{#1}]}
\newcommand{\nunit}[1]{\ \si{#1}}
\newcommand{\avg}[1]{\langle{#1}\rangle}

% braces around equation number in referencing
\makeatletter
\let\oldtheequation\theequation
\renewcommand\tagform@[1]{\maketag@@@{\ignorespaces#1\unskip\@@italiccorr}}
\renewcommand\theequation{(\oldtheequation)}
\makeatother

% Prevent latex from streching out paragraph spacings
\raggedbottom

% braces around subfig number
\renewcommand\thesubfigure{\,(\alph{subfigure})}

% Smallcaps short autoref
\newcommand*{\shortautoref}[1]{%
	\begingroup
	\def\equationautorefname{\textsc{Eq.}}%
	\def\tableautorefname{\textsc{Tab.}}%
	\def\figureautorefname{\textsc{Fig.}}%
	\autoref{#1}%
	\endgroup
}

% Itemize spacing
\let\olditemize=\itemize
\let\endolditemize=\enditemize
\renewenvironment{itemize}{\olditemize \itemsep0em}{\endolditemize}

% subfigure spacing
\captionsetup[subfigure]{aboveskip=1pt}

\begin{document}
\title{CS-433 -- Project 2\\Convolutional graph neural networks for tracking yeast cells}

\author{
	Greta Brenna, Fanny Massard, Niels Vadot\\
	\textit{EPFL, Switzerland}
}

\maketitle

\begin{abstract}
	We present a graph-based machine learning method that is able to track yeast cells throughout a movie, after having segmented the frames (here, this is done with YeaZ \cite{dietler_convolutional_2020}). The method achieves better accuracy than the Hungarian algorithm implemented in YeaZ, thanks to the context-awareness of the graph structure. The method relies on translating the tracking problem to a binary node classification problem, and should generalize to any tracking problem where segmentations are available. The source code, as well as explicative Python notebooks, are available at \cite{vadot_cs-433_2021}.
\end{abstract}

\section{Context}

	The tracking problem emerges when addressing a more specific problem, that of lineage determination of budding yeast : for each new bud, the parent cell needs to be determined. This is made difficult in densely packed environments, but budneck markers can be used to manually determine the parent of each bud. However, it might not always be convenient to use such markers, and the question arises if lineage determination is possible with only a phase contrast image. Buds are most visible when they grow, so a machine learning algorithm might be able to pick up on spatio-temporal correlations in the segmentations to determine the parent. However, feeding a machine learning algorithm coherent temporal information requires a consistent tracking of the cells, which is the main problem this report addresses.

	The classical approach to tracking is to extract some features from the segmentation geometry (YeaZ uses cell surface area and center of mass), and minimize the pairwise euclidean distance in the feature space using the Hungarian algorithm. The problem with this method is that it starts to break down for large colonies, since in that regime many cells have similar geometry and entire sections of the colony might drift away. Then, it would be interesting to look at pairwise relationships between cells and their nearest neighbors to track the cells, similarly to how puzzle pieces fall into place given the right context.

	A graph-based approach arises, which can naturally encode cells as nodes with certain features, and nearest-neighbor relationships are edge features. The recent development of graph neural networks (GNN) \cite{zhou_graph_2021} and growing open-source codebase \cite{paszke_pytorch_2019, fey_fast_2019} makes it possible to easily implement machine learning methods that are able to learn on graphs.

\section{Prior work}

	The problem of tracking arises in other domains, such as reconstruction of particle trajectory in detectors \cite{dezoort_charged_2021}. In this approach, timestamped detection events in cylindrical coordinates are represented as nodes, and edge features are constructed by considering geometrical differences of the two detection events it connects. Features are then encoded with an MLP, and an edge-classification task is performed in order to attribute multiple detection events to the same particle. In our context, cells are not moving in a highly correlated manner like a particle through a detector, but the idea of encoding geometrical features can be repurposed.

	Tracking nodes, as they evolve in time-dependent graphs, can be reformulated as the problem of graph matching. \cite{kipf_semi-supervised_2017} considers the problem of finding a subgraph in a larger graph. It does this by applying graph convolutions, embedding the nodes in a high-dimensional space, and minimizing the total cost of pairwise euclidean distance to match nodes from the two graphs. We try to reformulate the tracking problem as a graph matching problem, and use a similar idea by maximizing a score to assign nodes between two graphs.

	% Other papers \cite{cordonnier_extrapolating_2019} propose a graph neural network-based approach to predict trajectories, however they assume highly correlated motion or decision agents responsible for the motion, which is not applicable here. Furthermore, the task here is to reconstruct trajectories, not to predict them.

\section{Data preprocessing} \label{sec:data_preprocessing}

	\subsection{Feature extraction and cell graph generation}

		The full data preprocessing pipeline is schematized in \shortautoref{fig:pipeline}. The data used here consists of 5 colonies of budding yeast, with phase contrast images taken at 5 minute intervals, for a total of 180 images (15 hours of growth time) per colony. The first 4 colonies (in the data, named \texttt{colony00[0123]}) are used for generating training (and validation) datasets, and the last colony (named \texttt{colony007}) is kept as a final testing set. On average, the colonies started with 2 cells and had 110 cells at the end.

		The movies were segmented and tracked semi-automatically using YeaZ \cite{dietler_convolutional_2020}. From the segmentation masks, simple geometrical features were extracted from the cells : surface area $A$, radius $r$ and eccentricity $e$. To do this, an ellipse was fitted to the cell by using PCA analysis on the cell mask coordinates, which can then be used to obtain the semi-major and minor axes $a$ and $b$ respectively. Then we compute $e=\sqrt{1-(\frac{a}{b})^2}$, and the radius of the circle with the same area as the ellipse $r=\sqrt{ab}$. \shortautoref{fig:hist_x} shows the distribution of the node features. The distributions are strongly peaked around their mean, respectively $\avg{A} \approx 21 \pm 14 \nunit{\micro \meter \squared}$, $\avg{r} \approx 2.5 \pm 0.8 \nunit{\micro \meter}$ and $\avg{e} \approx 0.52 \pm 0.14$. Notice the distribution for $r$ has a few outliers corresponding to very large cells, and these outliers are even more visible for $A$, explained by the scaling law $A \sim r^2$. It however doesn't make sense to remove these outliers, since that would have the same effect as removing the cell from the colony, impacting its direct neighbors.

		Nearest neighbor features were extracted by considering the contour of each cell mask, and placing an edge between cells if the minimum distance between their contours is below $1.3 \nunit{\micro \meter}$ (this threshold is arbitrary, and can be changed as needed when calling \texttt{bread\_nn.graph.build\_cellgraph}). Extracted features are $\rho$, the center of mass (CM) to CM distance between the neighboring cells, $\theta$, the angle of the CM to CM vector with respect to the horizontal axis, and $\ell$, the minimal distance between the contours. \shortautoref{fig:hist_e} shows the distribution of these features. $\rho$ shows a distribution peaked around $\avg{\rho} \approx 5.4 \pm 1.2$, which makes sense, as on average cell has a radius of $r \approx 2.5 \nunit{\micro \meter}$, meaning two average cells would be separated by approximately $2r \approx 5.0 \nunit{\micro \meter}$. $\theta$ shows a near-uniform distribution, with mean and standard deviation $\avg{\theta} \approx 0 \pm 1.8$. Again, this is intuitive, since we don't expect cells to be preferentially aligned in one direction. Note that we allow $\theta$ to take values in $[-\pi, \pi]$ instead of $[0, \pi]$, because edges are not stored bidirectionally (to save memory), so the sign of $\theta$ encodes directionality of the edge. $\ell$ is strongly peaked around $\avg{\ell} \approx 0.34 \pm 0.22 \nunit{\micro \meter}$. Physically, $\ell$ is not that representative since membranes are supposed to be in contact, and the mean being slightly offset is an artifact from the segmentation process. Nevertheless, $\ell$ can be a good distance metric between cells, since it is independent of their sizes.

		In total, 720 cell graphs we constructed, for a total of 25510 nodes and 48018 edges (note : edges are considered bidirectional, but are stored as one-directional to save memory). The Python notebook used for this process can be found under \texttt{experiment\_gnn/build\_cellgraphs.ipynb}.

	\subsection{Assignment graph generation}

		We formulate the tracking problem as a graph matching problem. Suppose we have generated two graphs $\G^{(1)}$ and $\G^{(2)}$ storing extracted features from two segmentations, and we want to track cells from segmentation (1) to segmentation (2). For this, we construct an assignment graph $\G^{(a;x,e)}$. Let $\vec{v_i^{(1)}}, \vec{e_{i,j}^{(1)}}$ (resp. $\vec{v_a^{(2)}}, \vec{e_{a,b}^{(1)}}$) encode the nodes and edges of $\G^{(1)}$ (resp. $\G^{(2)}$). Nodes $\vec{v_{ia}^{(a)}}$ of $\G^{(a;x,e)}$ are obtained by concatenating features from $\vec{v_i^{(1)}}$ and $\vec{v_a^{(2)}}$. Edges $\vec{e_{ia,jb}^{(a)}}$ of $\G^{(a)}$ link nodes $\vec{v_{ia}^{(a)}}$ and $\vec{v_{jb}^{(a)}}$ if $\vec{e_{i,j}^{(1)}}$ and $\vec{e_{a,b}^{(2)}}$ exist in $\G^{(1)}$ and $\G^{(2)}$ respectively, in which case edge features are similarly obtained by concatenation.

		Finally, we similarly build the corresponding target graph $\G^{(a;y)}$, where the node features are simply 2-dimensional one-hot encoder vectors, indicating whether a node $\vec{v_{ia}^{(a)}}$ corresponds to a correct tracking. Note that at this point, there is no trace left of the cell ids that were used to differentiate cells in the segmentations.

		Concretely, the assignment graph dataset was obtained by considering all pairs of segmentations separated by at most 20 frames (1h40), constructing the assignment graphs, and repeating this for each colony. The Python notebook used for this process can be found under \texttt{experiment\_gnn/build\_assgraphs.ipynb}.

		\begin{figure*}
			\centering
			\includegraphics[width=\linewidth]{figures/pipeline.pdf}
			% TODO : elliptical fits line colors
			\caption{Data preprocessing pipeline. A movie of a budding yeast colony is taken (in phase contrast, as is the case here), then YeaZ \cite{dietler_convolutional_2020} is used to segment individual frames, and semi-automatically track cells. Manual corrections to segmentation and tracking are applied as required. From the segmentations, geometric features are extracted, as well as features describing the neighborhood of each cell. This is stored in a graph structure $\G^{(1)}$ and $\G^{(2)}$, from which an assignment graph $\G^{(a;x,e)}$ is built. A ground truth assignment graph $\G^{(a;y)}$ is also built. Both assignment graphs are then used by the GNN for training.}
			\label{fig:pipeline}
		\end{figure*}

\section{Methods}

	\subsection{GNN structure}

		The GNN structure is summarized in \shortautoref{fig:nn}. Node and edge features of $\G^{(a;x,e)}$ are embedded by two MLPs with ReLU activation. The resulting embedded graph is then convolved by multiple DeepGCN layers \cite{li_deepergcn_2020}. One DeepGCN layer consists of a batch normalization layer (LayerNorm), an activation function (ReLU), one dropout layer, one GENConv layer, then finally a residual connection layer. In particular, the GENConv layer constructs messages $\vec{x_i}'$ from node and edge features $\vec{x_i}, \vec{e_{i,j}}$ as
		$$
			\vec{x_i}' = \textrm{MLP}(\vec{x_i}' + \textrm{AGG}(\{\textrm{ReLU}(\vec{x_i} + \vec{e_{i,j}}) + \epsilon \;|\; i \in \mathcal{N}(j)\})),
		$$
		where $\mathcal{N}(j)$ denotes indices of the nodes connected to node $j$, $\textrm{AGG}$ is a permutation-invariant aggregation function (in this case, a softmax), and $\epsilon$ is a small learnable constant. The $\textrm{MLP}$ has 2 layers, and preserves the number of channels of the input.

		The final linear layer maps the embedded node channels to two channels, and the resulting graph $\G^{(a;\hat y)}$ is then passed to CrossEntropyLoss along with the ground truth $\G^{(a;y)}$ for backpropagation.

	\subsection{Making predictions} \label{sec:methods:pred}

		Let $n_1$ and $n_2$ be the number of cells in the first and second images respectively. We define an assignment matrix $A$ with components $A_{i,j} = 1$ if cell $i$ in the first image is the same as cell $j$ in the second image, else $A_{i,j} = 0$. For the example in \shortautoref{fig:pipeline}, the assignment matrix is
		$$
			A = \begin{pmatrix}
				1 & 0 & 0 & 0 & 0 & 0 & 0 \\
				0 & 1 & 0 & 0 & 0 & 0 & 0 \\
				0 & 0 & 1 & 0 & 0 & 0 & 0 \\
				0 & 0 & 0 & 1 & 0 & 0 & 0 \\
				0 & 0 & 0 & 0 & 1 & 0 & 0
			\end{pmatrix}.
		$$

		From the predicted graph $\G^{(a;\hat y)}$, we can construct an assignment matrix by reshaping the node features into a matrix $Z$ of shape ($n_1$, $n_2$, $2$), where the last dimension stores weights for the classification decision. There are multiple ways of transforming $Z$ into $A$, but we describe only the method that empirically worked the best (other methods are presented and discussed in \texttt{experiment\_gnn/pipeline.ipynb}).

		By taking differences along the last axis of $Z$, we obtain a score matrix $S$ of shape ($n_1$, $n_2$), where each entry's magnitude can be interpreted as the confidence of the GNN of the assignment. The goal is then to find $A^\star$ as such to maximize the assignment score, mathematically
		$$
			A^\star = \arg \max_{A \in \{0, 1\}^{n_1 \times n_2}} \sum_{i,j} S_{i,j} A_{i,j}.
		$$
		This linear sum assignment (linsum) is easily solved by using \texttt{scipy.optimize.linear\_sum\_assignment} from the Scipy package \cite{2020SciPy-NMeth}.

		\begin{figure*}
			\centering
			\includegraphics[width=\linewidth]{figures/nn.pdf}
			\caption{The GNN structure. For a given input graph, the node and edge features are encoded by MLPs. The resulting graph is then convolved multiple times following DeepGCN \cite{li_deepergcn_2020}. A final linear layer maps the encoded node features down to two, and CrossEntropyLoss is computed for backpropagation. For evaluation, the node features of $\G^{(a;\hat y)}$ can be reduced and reshaped to build an assignment matrix.}
			\label{fig:nn}
		\end{figure*}

\section{Results}

	This section discusses performance on the models trained with the constructed graphs, as described in \shortautoref{sec:data_preprocessing}. All models are trained with the Adam optimizer with varible exponentially decaying learning rate, adjusted manually depending on the batch size used. Unless specified otherwise, all layers use dropout (if it applies) during training with probability parameter $0.1$. The random seed is fixed at the start of each training session. Training and validation datasets are shuffled at each epoch, and make up $80\%$ and $20\%$ of the total dataset respectively.

	For evaluation, models predicting assignment matrix $\hat A$ are compared to the ground truth $A$ using the accuracy metric defined as
	$$
		\textrm{acc}(\hat A, A) = \frac{1}{n_1 n_2} \sum_{i,j} \delta_{A_{i,j}, \hat{A}_{i,j}}.
	$$

	\subsection{MLP baseline}

		As a baseline for the GNN, we train a MLP on the node features $\vec{X}$ of $G^{(a;x,e)}$ to evaluate if the graph structure (that is, addition of edge features) really adds valuable information for learning.

		For this, we need to equilibrate the labels $\vec{y}$, because asymptotically the number of nodes in the assignment graph grows as $n_1 n_2$, but the number of assignments where $y=1$ grows as $n_1$. Therefore, labels $y=1$ are underrepresented in large graphs, and equilibration is done by removing samples where $y=0$ from the training batch, until both classes are equally represented.

		Furthermore, batch normalization is applied before the MLP in order to improve training, as described in \cite{ioffe_batch_2015}.

		A hyperparameter scan was performed on the number of MLP layers and hidden channels, and found that the best performing MLP obtained an accuracy of approximately $0.88$. More layers and hidden channels were found to marginally improve the accuracy, before it started to drop due to overfitting.

	\subsection{GNN performance}

		We perform a scan on the following hyperparameters : $N_\textrm{enc}$ number of layers in both encoder MLPs, $N_\textrm{conv}$ number of DeepGCN layers, and $N_\textrm{dim}$ the number of hidden channels. Other hyperparameters, such as dropout, learning rate, exponential learning rate decay and training batch size were found to not significatively improve final performance, but can decrease training time if correctly chosen (here they were set to $0.1$, $\sim 10^{-4}$, $\sim 0.98$, $\sim 32$ respectively). For the study, we chose $N_\textrm{enc} \in \{1,2,3,4,5\}$, $N_\textrm{conv} \in \{1,2,3,4,5,8,11\}$ and $N_\textrm{dim} \in \{30, 60, 90, 120\}$. \shortautoref{fig:hparams} plots the results of the hyperparameter scan in parallel coordinates.

		Models with less than $0.99$ accuracy are obtained with few parameters overall, $N_\textrm{enc} \in \{1,2,3\}$, $N_\textrm{conv} \in \{1,2,3,4\}$ and $N_\textrm{dim} \in \{30, 60\}$. For better accuracy, it was found most effective to increase the number of encoder layers $N_\textrm{enc} \in \{2,3\}$, which systematically drove up the accuracy up to $0.999$. Following this, increasing $N_\textrm{enc}$ only marginally improved accuracy, but allowing up to 5 convolution layers gave models obtaining accuracy of up to $0.9999$. In this regime, the graph neural network is able to capture most of the complexity of the problem, but still struggles with edge cases (for instance, a cell moving a large amount). Further increasing the number of parameters (via $N_\textrm{conv}$ or $N_\textrm{dim}$, they seem to compensate each other) results in even better accuracy, where edge cases are now correctly handled. The best models obtained an accuracy of $0.99997$, and utilized $N_\textrm{enc} \geq 3$, $N_\textrm{conv} \geq 8$ and $N_\textrm{dim} \geq 80$.

		Intuitively, the performance gain of increasing $N_\textrm{enc}$ eventually caps off, because each cell graph node or edge is decribed with only 3 features, and the MLP can only transform information it is given. Increasing $N_\textrm{conv}$ intuitively can be seen as increasing the ``communication distance'' between cells, meaning each node has a ``better view'' of the whole graph structure. This of course comes with the cost of requiring more dimensions to store the information in, explaining the need to increase $N_\textrm{dim}$, and how both compensate each other. It is not useful to be able to ``see'' further in the graph ($N_\textrm{conv}$ large) if you cannot ``store'' the information ($N_\textrm{dim}$ small), and vice-versa.

		We now compare the performance of the best GNN model ($N_\textrm{enc}=5$, $N_\textrm{conv}=11$, $N_\textrm{dim}=120$) against the built-in tracking system of YeaZ, using the test dataset. From the 180 frames in the dataset, 885 pairs are tested, corresponding to frames separated by 5 to 25 minutes. Numerical results are presented in \shortautoref{tab:test_acc}. The linear sum assignment method works the best, and performs better than YeaZ, especially for pairs of frames separated by a long time difference, and in situations where cells move around a lot (specific examples are shown in \shortautoref{fig:mistracks}). The dataset's frames were separated by 5 minutes here, however one should expect approximately 15 minutes of frame separation in real applications, in which case this method has been shown to be performant.

		% Out of the 180 interframes of the testing set, YeaZ's tracker made at least one mistake on 3 of them, while our model made none. This seems plausible, as this test evaluates performance on only one frame difference (5 minutes), but the GNN has been trained on frames separated by at most  Accuracies for different methods of generating assignment matrice from the model's output are reported in \shortautoref{tab:test_acc}. This of course is not enough testing data to definitely conclude

		% Note that experimentally, better tracking performance has been observed when using values larger than this threshold, around $2 \nunit{\micro \meter}$. This of course depends on the exact colony morphology

		\begin{table}
			\centering
			% \begin{tabular}{l||rrrrr|r}
			% 	\toprule
			% 	{} &       naive &     fw &    bw &  bw+corr &      linsum &        yeaz \\
			% 	\midrule
			% 	mean  &    0.98698 &    0.99834 &    0.98689 &       0.99534 &    \textbf{0.99991} &    0.99420 \\
			% 	std   &    0.04319 &    0.01482 &    0.04011 &       0.01767 &    0.00064 &    0.05908 \\
			% 	min   &    0.77777 &    0.77777 &    0.75000 &       0.77777 &    0.99259 &    0.33333 \\
			% 	25\%   &    0.99725 &    1.00000 &    0.99359 &       1.00000 &    1.00000 &    1.00000 \\
			% 	50\%   &    1.00000 &    1.00000 &    1.00000 &       1.00000 &    1.00000 &    1.00000 \\
			% 	75\%   &    1.00000 &    1.00000 &    1.00000 &       1.00000 &    1.00000 &    1.00000 \\
			% 	max   &    1.00000 &    1.00000 &    1.00000 &       1.00000 &    1.00000 &    1.00000 \\
			% 	\bottomrule
			% \end{tabular}
			\begin{tabular}{lrrrrrrrr}
			\toprule
			{} &             mean &       std &      25\% &  50\% &  75\% \\
			\midrule
			naive         &  0.986984 &  0.043199 &  0.997253 &  1.0 &  1.0 \\
			fw            &  0.998341 &  0.014823 &  1.000000 &  1.0 &  1.0 \\
			bw            &  0.986895 &  0.040118 &  0.993590 &  1.0 &  1.0 \\
			bw+corr       &  0.995342 &  0.017675 &  1.000000 &  1.0 &  1.0 \\
			linsum        &  \textbf{0.999913} &  0.000645 &  1.000000 &  1.0 &  1.0 \\
			\midrule
			yeaz          &  0.994201 &  0.059083 &  1.000000 &  1.0 &  1.0 \\
			\bottomrule
			\end{tabular}
			\caption{Test accuracy of different methods of generating the assignment matrices from the GNN's output. The testing set (\texttt{colony007}) was split into pairs of 885 interframes, separated in real time by 5 to 25 minutes. ``naive'' refers to classifying each node simply based on the weights for each class. ``fw'' refers to the forward method, which forces each cell in the first frame to match one in the second frame. ``bw'' refers to the backwards method, which forces each cell in the second frame to match one in the first frame. ``bw+corr'' is a corrected version of the backward method, which accounts for buds. ``linsum'' uses the linear sum optimization method, as presented in \shortautoref{sec:methods:pred}. ``yeaz'' is the baseline tracking method. See \texttt{experiment\_gnn/pipeline.ipynb} for more information.}
			\label{tab:test_acc}
		\end{table}

\section{Conclusion}

	We have shown that this graph neural network model is able to learn the structure of a yeast cell colony, and is able to reliably track cells from frame to frame. It has been shown that the model exceeds YeaZ's integrated tracker performance, using a relatively small training dataset of only 4 colonies growing over 15 hours and imaged every 5 minutes. If implemented in YeaZ's GUI, tracking tasks will become easier and significantly less time-consuming, especially for large colonies.

	One of the limitations of this model is the need to have segmented the images and verified the segmentations. We however argue that thanks to advancements in this domain \cite{dietler_convolutional_2020,stringer_cellpose_2020}, this has become a trivial task. Nevertheless, it is a good practice to verify the segmentations manually, and correct as needed.

	The model presented in this report essentially only uses two graphs as an input, and could benefit from having a ``longer history'' of the colony in order to make better tracking predictions. Multiple papers \cite{singer_node_2019,jin_recurrent_2020,rossi_temporal_2020} address the problem of temporal graphs, and it might be worth investigating to further the research. In summary, using temporal graph networks are a promising approach to constructing lineages from only phase contrast (or bright field) microscopy movies. This would likely have to be coupled to an attention system, for which literature and implementations for graph neural networks already exist \cite{velickovic_graph_2018,brody_how_2021}.

	% Rome wasn't built in a day, and graph neural networks aren't mastered in a month.

\newpage
\bibliographystyle{IEEEtran}
\bibliography{literature}

\newpage
\appendix

	\begin{figure}[h!]
		\centering
		\includegraphics[width=\linewidth]{figures/mistracks.pdf}
		\caption{Segmentations from \texttt{colony007} highlighting moving cells. Out of the 179 interframes, YeaZ makes 3 tracking mistakes. On frame 115, cells 9 and 14 move together towards cell 5, and YeaZ tracks cell 9 to cell 16 and cell 14 to cell 9. On frame 163, cell 22 moves around the top of the colony, and YeaZ tracks cell 10 to 22, cell 10 to 29, cell 29 to cell 25 and cell 25 to 34. On frame 171, cells 22 and 34 move together around the top while swapping places, and YeaZ tracks cell 22 to 34 and vice-versa.}
		\label{fig:mistracks}
	\end{figure}

	\begin{figure}
		\centering
		\includegraphics[width=\linewidth]{figures/hist_x.pdf}
		\captionof{figure}{Distribution of node features.}
		\label{fig:hist_x}
	\end{figure}

	\begin{figure}
		\centering
		\includegraphics[width=\linewidth]{figures/hist_e.pdf}
		\captionof{figure}{Distribution of edge features.}
		\label{fig:hist_e}
	\end{figure}

	\begin{figure}
		\centering
		\includegraphics[width=\linewidth]{figures/hparams.pdf}
		\caption{Parallel coordinates plot of the hyperparameter scan. The best performing model uses $N_\textrm{enc}=5$, $N_\textrm{conv}=11$, $N_\textrm{dim}=120$, but a very similar performance can be achieved for less parameters, $N_\textrm{enc} \geq 3$, $N_\textrm{conv} \geq 8$ and $N_\textrm{dim} \geq 80$.}
		\label{fig:hparams}
	\end{figure}

\end{document}


\chapter{Introduction}
\section{first section}
This is the introduction Figure \ref{fig:my_label}.

Now reference an eqn about linear algebra: \eqref{eq:Linear Algebra}.
\begin{figure}
    \centering
    \includegraphics[width=\linewidth]{figures/StabilityMargin.pdf}
    \caption{Caption}
    \label{fig:my_label}
\end{figure}
\begin{equation} \label{eq:Linear Algebra}
\underline{A}\vec{x}=\underline{B}
\end{equation}
\lstinputlisting[language=Matlab]{code/Add.m}
% Matlab Figure%%%%%%%%%%%%%%%%%%%%%%
\begin{figure}

	\tikzset{every picture/.style={scale=.8}}%
	\centering
	\import{figures/}{test1.tex}
	\caption{Free body diagram of a beam section in planar bending.}
	\label{fig:test1}
\end{figure}
% Inkscape Figure%%%%%%%%%%%%%%%%%%%%%
\begin{figure}
	\centering
	\def\svgwidth{400pt}
	\import{figures/}{BeamElement.pdf_tex}
	\caption{Beam Element with nodal displacements.}
	\label{fig:BeamElem}
\end{figure}


\documentclass[main.tex]{subfiles}
\begin{document}

\section{Gauge conditions}

\marginpar{Tuesday\\ 2021-6-22, \\ compiled \\ \today}

So far we have only discussed maximal slicing, \(k = - \nabla_a n^a = 0\).

Now let us see Schwarzschild foliations: in \((t, R)\) Schwarzschild coordinates we can consider \(t= \const\) hypersurfaces.

For \(R > 2M\) time \(t\) is a temporal coordinate. Here, these constant-\(t\) hypersurfaces satisfy \(k_{ab} = 0\), so we also have \(k=0\).

If we draw a Kruskal \(U\), \(V\) diagram the horizon \(R = 2M\) is a diagonal line \(U=V\), while the singularity is a hyperbola.

This slicing is therefore \emph{maximal}, and \emph{not horizon penetrating}.

If we use the isotropic radius \(r\) as a coordinate, the lapse reads
%
\begin{align}
\alpha = \qty(1 - \frac{M}{2r}) \qty(1 + \frac{M}{2r})^{-1}
\,.
\end{align}

\(\alpha \) is antisymetric, and negative in the III Kruskal region.

In the region \(R < 2M\), the foliation is given by \(R = \const\) because there \(R\) is a timelike coordinate.

Now, \(\alpha = (2M / R - 1)^{-1}\), while the metric determinant  is
%
\begin{align}
\gamma = R^{4} \sin^2 \theta \qty( \frac{2M}{R } - 1)
\,,
\end{align}
%
and we can use
%
\begin{align}
k = - \frac{1}{\alpha } \mathscr{L}_m \log \sqrt{ \gamma }
= \frac{3M - 2R}{R^2 \qty( 2M / R - 1)^{1/2}}
\,,
\end{align}
%
therefore the \(R = 3M/2\) slice is maximal.

It can be proven that this is a limit slice of a certain maximal foliation of stuff which looks like hyperbolas in a Kruskal diagram, which is maximal and symmetric with respect to \(R = 2M\).

These correspond to solutions of the maximal equation the lapse:
%
\begin{align}
D_i D^{i}  \alpha - \alpha \qty()\dots = 0
\,
\end{align}
%
with boundary conditions such that \(\alpha \to 1\) at \(\iota_0 \), and \(\alpha (2M) = 0\) (case 1, a Dirichlet boundary condition) or \(\partial_{R} \alpha (3M/2) = 0\) (case 2, a Neumann boundary condition).

Different inner BCs determine different foliations and different properties.
Both of the foliations we discussed are singularity-avoiding.

In Eddington-Finkelstein (Kerr-Schild) coordinates, we can set
%
\begin{align}
t_{KS} = t + 2M \log \qty( \frac{r}{2M} - 1)
\,,
\end{align}
%
which yields a non-singularity avoiding foliation.
How do we construct, in general, maximal foliations of Schwarzschild?

We consider the transformation
%
\begin{align}
t \to \widetilde{t} = t + h( R)
\,,
\end{align}
%
where \(h(R)\) is a \emph{height function}, which (because of \(0 = k\)) satisfies the equation
%
\begin{align}
h'(R) = \frac{c^2}{A^2(R) \qty[ A(R) R^{4} + c^2s]}
\,,
\end{align}
%
where \(A(R) = (1- 2M /R)\) and \(c\) is a constant of integration.

The metric is given in terms of
%
\begin{align}
\alpha = f(R) &= 1 + \frac{2M}{R} + \frac{c^2}{R^{4}}
\,,
\end{align}
%
while the shift is
%
\begin{align}
\beta^{r} = \frac{c}{R^2} \sqrt{f(R)}
\,,
\end{align}
%
and
%
\begin{align}
\gamma_{ij} \dd{x^{i}} \dd{x^{j}} = f^{-1} (R) \dd{R^2} + R^2 \dd{\Omega }
\,,
\end{align}
%
which yields a family of foliations for different choices of \(c\): \(c=0\) is standard Schwarszchild spacetime, but other choices are possible, and \(c = (3/4) \sqrt{3} M^2\).

The main property of maximal slicing, which is quite general, is that the lapse \(\alpha \) goes to zero in the regions of highest curvature.

This is called ``lapse freezing'' or ``lapse collapse'', and it is a generic property indicating the fact that the gauge is singularity-avoiding.

This is defined by \(\square x^{\mu }=  0\) (Harmonic  gauge), so for \(\mu = 0\) we have \(\square t = 0\).

This means
%
\begin{align}
0 = \square t = \sqrt{-g} \partial_{\mu } \qty(\sqrt{-g} g^{\mu \nu } \partial_{\nu } t) = \sqrt{-g} \partial_{\mu } \qty(\sqrt{-g} g^{\mu 0})
\,.
\end{align}
%

This means that
%
\begin{align}
0 = \partial_{t} \qty(\alpha \sqrt{\gamma } g^{00}) + \partial_{i} \qty(\alpha \sqrt{\gamma } g^{0i})
\,,
\end{align}
%
but \(g^{00} = \alpha^{-2}\) and \(g^{0i}= \beta^{i} / \alpha^2 \).

The equation then reads
%
\begin{align}
0 = \partial_{t} \alpha - \beta^{i} \partial_{i} \alpha  - \alpha
\underbrace{\qty[ \frac{1}{\sqrt{\gamma }} \partial_{t} \sqrt{\gamma } - \frac{1}{\sqrt{\gamma }} \partial_{i} \qty(\sqrt{\gamma } \beta^{i})]}_{- \alpha k}
\,,
\end{align}
%
so the equation for \(\alpha \) reads \(\mathscr{L}_m \alpha = - \alpha^2 k\).

If we set \(\beta^{i} = 0\), we get \(\alpha = C(x^{i}) \sqrt{\gamma }\).
In the case of Schwarzschild, we have \(\partial_{t} \alpha = 0\), \(\beta^{i} = 0\) and \(k = 0\).

Constant-\(t\) slices of Schwarzschild are harmonic slices.

The Bona-Masso family, also known as \(1 + \log\) slicing, is determined by
%
\begin{align}
\mathscr{L}_m \alpha = - \alpha^2 f(\alpha ) k
\,.
\end{align}

We can fall back to geodesic slicing with \(f(\alpha ) = 0\), or harmonic slicing with \(f(\alpha ) = 1\), or we can set \(f(\alpha )= 2/\alpha \): this is \(1+ \log\) slicing.

If we set \(\beta^{i} = 0\), we have \(\partial_{t} \alpha = \partial_{t}  \log \gamma \), a solution to which is  \(\alpha = 1 + \log \gamma \).

For Schwarzschild, this is the Height function method:
%
\begin{align}
\alpha^2 = 1 - \frac{2M}{R} + \frac{c^2}{R^{4} } e^{\alpha }
\,,
\end{align}
%
which is very close to maximal slicing; however this is now an implicit equation with this new exponential term.

These are called ``trumpet slices'': in an embedding diagram they look like a wormhole, however they end at \(R = 3M/2\).

This \(1 + \log\) slicing is what is typically used today to evolve astrophysical black holes.

What about spatial gauge and the choice of shift \(\beta^{i}\)?

One item in our wishlist was ``minimal distortion'': we can quantify it through a distortion tensor,
%
\begin{align}
Q_{ij} = \partial_{t} \gamma_{ij} = - 2 \alpha k_{ij} + \mathscr{L}_\beta \gamma_{ij}
\,.
\end{align}

From this tensor we can consider
%
\begin{align}
\Sigma_{ij} = Q_{ij} - \frac{1}{3} Q \gamma_{ij} = \dots = - 2 \alpha A_{ij} + (L \beta )_{ij} = \psi^{4} \partial_{t} \widetilde{\gamma}_{ij}
\,.
\end{align}

We can use this to define a functional:
%
\begin{align}
I[\beta^{i}] &= \int_{\Sigma _t} \Sigma_{ij} \Sigma^{ij} \sqrt{\gamma } \dd[3]{x} =  \\
&= \int_{\Sigma _t} \qty[ 4 \alpha^2 A_{ij}A^{ij} - 4 \alpha A_{ij} (L \beta )^{ij} + (L \beta )_{ij} (L \beta )^{ij} ] \sqrt{\gamma } \dd[3]{x}
\,,
\end{align}
%
which we can extremize: we set \(0 = \delta I [\beta^{i}]\), which yields
%
\begin{align}
\delta I [\beta^{i}] &= \int_{\Sigma _t} 2 \delta \qty[ \Sigma_{ij} (L \beta )^{ij}] \sqrt{\gamma } \dd[3]{x}  \\
&= 2 \int_{\Sigma _t} \Sigma_{ij} \qty(D^{i} \delta \beta^{j} + D^{j} \delta \beta^{i} - \underbrace{\frac{2}{3} D_k \delta \beta^{k} \gamma^{ij}}_{\Sigma = 0}) \sqrt{\gamma } \dd[3]{x}  \\
&= 4 \int_{\Sigma _t} \Sigma_{ij} D^{i} \delta \beta^{j} \sqrt{\gamma } \dd[3]{x}  \\
&= 4 \int_{\Sigma _t} \qty[ D^{i} \qty(\Sigma_{ij} \delta \beta^{j}) - D^{i} \Sigma_{ij} \delta \beta^{j} ] \sqrt{\gamma } \dd[3]{x}  \\
&= \underbrace{4 \int_{\partial \Sigma _t} \Sigma_{ij} \delta \beta^{j} \sqrt{\gamma } \dd[3]{x} }_{= 0, \eval{\delta \beta^{j}}_{ \partial} = 0} - 4
\int_{\Sigma _t} D^{i} \Sigma_{ij} \delta \beta^{j} \sqrt{\gamma } \dd[3]{x} = 0
\,,
\end{align}
%
therefore our condition, by the usual lemma of funcitonal variational calculus, is \(D^{i} \Sigma_{ij} = 0\).
In terms of \(\beta^{i}\), this defines what is knowns as \emph{minimal distortion shift}:
%
\begin{align}
\triangle_L \beta^{i} = 2 D_j \qty(\alpha A^{ij}) - 16 \pi \alpha P^{i } + \frac{4}{3} \alpha D^{i } k + 2 A^{ij} D_j \alpha
\,.
\end{align}

This gauge, as written, is not really used, however it is the starting point for other ones which better achieve the distortion minimization.

An observation: both \(Q_{ij} \) and \(\Sigma_{ij}\) are 0 if \(\partial_{t}\) is a Killing vector, so in that case minimal distortion is satisfied automatically for stationary spacetimes in adapted coordinates.

How do we use this then?
We can use approximate minimal distortion equations:
%
\begin{align}
0 = D^{i} \Sigma_{ij} = D^{i} \qty(\psi^{4} \partial_{t} \widetilde{\gamma}_{ij}) \approx \widetilde{D}^{i} \qty(\partial_{t} \widetilde{\gamma}_{ij})
\,.
\end{align}
%
this elliptic equation for \(\beta^{i}\) is easier to implement numerically.

This is called \(\Gamma \) freezing: we can write the equation as
%
\begin{align}
0 = D_j \dot{\widetilde{\gamma}} = \partial_{t} D_j  \widetilde{\gamma}_{ij} = \dots = - \partial_{t} \widetilde{\Gamma}^{i}
\,,
\end{align}
%
(see equation 9.32 in the notes), where
%
\begin{align}
\widetilde{\Gamma}^{i} = - D_j \widetilde{\Gamma}^{ij} = \qty(\widetilde{\Gamma}^{i}_{jk} - F^{i}_{jk}) \widetilde{\gamma}^{jk}
\,.
\end{align}

We get an elliptic equation for \(\beta^{i}\), which is written as
%
\begin{align}
\widetilde{\gamma}^{jk} D_j D_k \beta^{i} + \dots = 0
\,,
\end{align}
%
which is nice because it is written in terms of partial derivatives.

Alcubierre and colleagues have proposed a \textbf{parabolic \(\Gamma \) driver}: the idea is to write \emph{evolution} equations for \(\beta^{i}\), such that the solutions ``asymptote'' to the ``equilibrium'' solution of the \(\Gamma \)-freezing.

This equation will look like
%
\begin{align}
\partial_{t} \beta^{i} = k \partial_{t} \widetilde{\Gamma}^{i} \approx k \qty( \widetilde{\gamma}^{jk} D_j D_k \beta^{i} + \dots)
\,,
\end{align}
%
a parabolic equation.

For \(t \to \infty \) this asymptotes to the \(\Gamma \)-freezing solution, however parabolic PDEs are known to be \emph{stiff}, which severely constrains the number of timesteps which can be taken.

Therefore, people have also considered \emph{hyperbolic drivers}:
%
\begin{align}
\partial_{tt} \beta^{i} &= k \partial_{t} \widetilde{\Gamma}^{i} - \qty(\eta \partial_{t} \log k) \partial_{t} \beta^{i}   \\
&= k \qty(\widetilde{\gamma}^{jk} \partial_{j} \partial_{k} \beta^{i} + \dots ) - \qty(\eta - \partial_{t} \log k) \partial_{t} \beta^{i}
\,,
\end{align}
%
and we can discard some term to get a damped wave equation:
%
\begin{align}
\partial_{tt} \beta^{i} = k \widetilde{\gamma}^{ij} \partial_{j} \partial_{k} \beta^{i} - \eta \partial_{t} \beta^{i}
\,.
\end{align}

This equation transports things and damps them.

A simpler first-order version is written as an advection-like equation:
%
\begin{align}
\partial_{t} \beta^{i} = \mu _s \widetilde{\Gamma}^{i} - \eta \beta^{i} + \beta^{j} \partial_{j} \beta^{i}
\,,
\end{align}
%
which has speed \(\mu _s\) (which can be chosen freely) and a damping term \(\eta > 0\), which can be also chosen.

These equations should be compared to the hyperbolic \(\Gamma \)-driver with the harmonic shift equation.

In summary:
we have seen, for \(\alpha \):
\begin{enumerate}
    \item geodesic gauge \(\alpha = 1\), \(\beta^{i} = 0\);
    \item maximal slicing \(k = 0\);
    \item harmonic slicing \(\square t = 0\);
    \item the Bona-Masso family, which includes harmonic slicing as well as \(1+ \log\) slicing;
\end{enumerate}

and for spatial slicing
\begin{enumerate}
    \item minimal distortion;
    \item \(\Gamma \)-drivers.
\end{enumerate}

In geodesic gauge, the simulation crashes at \(t = \pi \) for \(M = 1\): this is because that is the point at which the observer falls into the singularity.

With \(1 + \log\) slicing as well as a \(\Gamma \) driver we reach a sort of stationary configuration and the simulation does not crash.

\end{document}


\documentclass{article}
\usepackage{amssymb}
\usepackage{amsmath}
\usepackage{url}
\usepackage{comment}
\usepackage[dvipdfm]{graphicx}
\usepackage[dvipdfm,truedimen,margin=25mm, nohead]{geometry}
%\usepackage{color}
%\usepackage[bottom]{footmisc}

%\newtheorem{theorem}{Theorem}[section]
%\newtheorem{conjecture}[theorem]{Conjecture}
%\newtheorem{corollary}[theorem]{Corollary}
%\newtheorem{proposition}[theorem]{Proposition}
%\newtheorem{lemma}[theorem]{Lemma}
%\newdef{definition}[theorem]{Definition}
%\newdef{remark}[theorem]{Remark}

\def\F2{{\mathbb F}_2}
%\def\wt{{\rm wt}}
%\def\wo{{\rm wt}_o}
%\def\wf{{\rm wt}_f}
%\def\UL{{\rm ul}}
%\def\bx{{{\mathbf x}}}
%\def\by{{{\mathbf y}}}
%\def\bz{{{\mathbf z}}}
%\def\bw{{{\mathbf w}}}
%\def\bu{{{\mathbf u}}}

%\def\im{{\mathrm{Im}}}
%\def\ker{{\mathrm{Ker}}}
%\def\id{{\mathrm{Id}}}
%\def\tr{{\mathrm{tr}}}

\title{Technical Report: Dynamic Creator of 64-bit Mersenne Twister}
\author{Mutsuo Saito \and Makoto Matsumoto}
\date{2019-8-3}

\begin{document}

\maketitle

\begin{abstract}
  On this report, we describe a dynamic creation program for 64-bit
  Mersenne Twister.
\end{abstract}

%\keywords{
%Mersenne Twister; Dynamic Creation; Pseudorandom Number; Parallel Computing
%}

%\category{G.3}{Mathematics of Computing}{PROBABILITY AND STATISTICS}[Random number generation]

\subsection*{Improvement}

Dynamic Creation of Pserudorandom Number Generators (DC)~\cite{DC} was
proposed by Matsumoto and Nishimura and a program for 32-bit Mersenne
Twister (MT)~\cite{MT} written in C is published on web page
\url{http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/DC/dc.html} and
Git Hub \url{https://github.com/MersenneTwister-Lab/dcmt}.  64-bit
Mersenne Twister (MT64)~\cite{MT64} was proposed by Nishimura,
but　dynamic creation program for MT64 has not been supported.

There was a difficulty in calculating tempering parameter of MT64.
Two tempering parameter sizes of MT are 25 and 17 bits, on the other
hand, those of MT64 are 47 and 27 bits.  Increased sizes of tempering
parameters made dynamic creation very slow and impractical. To solve
this difficulty, now, we adopt ``Partial Bit Pattern'' algorithm,
which is the algorithm used in Mersenne Twister of Graphic Processors
(MTGP)~\cite{MTGP}.  The algorithm separates tempering parameters some
bit blocks and selects a bit pattern which gives the best $k(v)$ in
the blocks.　(Note: we talk about the algorithm of searching tempering
paramaeters. The tempering algorithm itself of MT64 is not changed.)

\subsection*{Results of the Dynamic Creation}

We made C++ program
dcmt64(\url{https://github.com/MersenneTwister-Lab/dcmt64}).  We
executed the program on Super Computer of The Institute of Statistical
Mathematics(ISM) in Japan.

\begin{center}
  Execution Environment:\\
  \medskip
\begin{tabular}{rl} \hline
  CPU & Intel Xeon Gold 6154 (18 core, 3.0GHz)\\
  Memory & 384GB \\
  OS & Red Hat Linux Enterprise Server 7\\
  Compiler & g++ \\
  \hline
\end{tabular}

\bigskip
Execution Results:\\
\medskip
\begin{tabular}{ll|ll} \hline
  used time & 10h & used process & 36 \\ \hline
  specified mexp & 19937 & parameters found & 1161 \\
  \hline
\end{tabular}
\end{center}

Dimension defect(DD)\cite[\S 1.2]{MT} is a simple evaluation index of
$F2$-linear random number generators. The less DD is the better.
The minimum DD of found parameters is 5023 and the maximum one
is 25258. 95 percent parameters have DD less than 7000.

\bibliographystyle{plain}
\bibliography{sfmt-kanren}
\end{document}


%!TEX root=main.tex

\section{Related Work} \label{sec:related_work}

\subsection{Uncertainty in Neural Networks}
Neural networks have achieved state-of-the-art results on estimating human intent~\cite{Vemula2017, Alahi2016}. However, one fundamental difference in human and neural network reasoning is uncertainty. Neural networks normally predict a mean prediction value and can fail over-confidently on novel data. Humans, in comparison, would reveal their uncertainty saying \textit{I have never seen this behavior before, hence I do not know how to predict the behavior}. To give the neural networks a similar notion of uncertainty and \textit{know what they don't know}, we have explored the recent field uncertainty-aware neural networks.

Bayesian Neural Networks reason about predictive uncertainty by keeping track of a distribution for each network's parameter~\cite{MacKay1992, Neal1996}. However, they are computationally intractable due to the expensive calculation of the Bayesian posterior. Even approximate methods, such as Markov Chain Monte Carlo or variational methods, come with extensive computational cost~\cite{Louizos2016, Graves2011, Springenberg_2016}. Other works, proposes MC-Dropout~\cite{Gal2015}: the activation of Dropout~\cite{Dropout2014} during test as approximate Bayesian inference in deep Gaussian processes. However, uncertainty estimates with MC-Dropout have shown to be overconfident on novel data~\cite{Osband2016, Lakshmi2016}.

Alternative works proposes the Bootstrapping as an approximation of model uncertainty in neural networks~\cite{Osband2016, Lakshmi2016}. Intuitively, an ensemble of randomly initialized models is trained on overlapping samples of a training dataset and during test the sample variance of predictions indicates the ensemble uncertainty. The predictions of each model will be similar for data points that have often occurred in the training set and differ for data points that have only occurred in one sample of the training set or not occurred at all.

\subsection{Intuitive physics understanding}
This work was inspired by \cite{Lerer, Bramley2017} who explored how to simulate our intuition about physics, and did so using deep feed-forward models to learn intuitive physics.

\begin{figure}[]
	\begin{subfigure}[]{1\linewidth}
		\centering
		\includegraphics[width=0.95\linewidth]{figures/sim_no_coll.png}
		\caption{Example trajectory with no collision.}
		\label{fig:sim_no_coll}
	\end{subfigure}
	\\
	\begin{subfigure}[]{1\linewidth}
		\centering
		\includegraphics[width=0.95\linewidth]{figures/sim_coll.png}
		\caption{Example trajectory with collision.}
		\label{fig:sim_coll}
	\end{subfigure}
    \caption{Pedestrian Simulation. In the beginning of each episode, the pedestrian simulation randomly spawns two pedestrians (dark orange and blue circle) at time $t=0$ at a random $x-y$ position. The simulation uses a pedestrian dynamics model to propagate the positions over time. The  bird's eye perspective of the pedestrian positions is displayed. At each time step the updated pedestrian position is plotted whose color becomes lighter over time. In ~\cref{fig:sim_no_coll} two pedestrians avoid each other and in~\cref{fig:sim_coll} two pedestrians collide.}
    \label{fig:ped_sim}
\end{figure}


\subsection{Vectors}
\label{Vectors}
A vector is essentially a map \(f(i)\) such that given \(i, 0 \leq i < n\), it
returns a value of a given type. The main types that \Q\ supports are four variants
of integers (1, 2, 4, and 8 byte) and two variations of floating point (single
and double precision). In addition, it supports bit vectors and constant length
strings. There is limited support for variable length strings, which are used
primarily as dictionaries.

Note that \Q\ has 6 types of numbers, in contrast with Lua which uses a
single type {\tt number}, internally a double precision floating point.
This is because data bandwidth plays a significant role in determining
performance, as illustrated by Nvidia's
introduction of half-precision floating point  \cite{nvidia2017}. The user is
encouraged (but not required) to use the smallest type that supports the actual
dynamic range required.

When a vector is created, we need to specify (i) its type (ii) whether it has
null values (iii) how it will be populuated. We can either (a) ``push'' data to
it, much like writing to a file in append mode or (b) we can provide a generator
function, which generates a chunk at a time when invoked.
%% TODO Have not introduced chunk as yet

Vectors are evaluated lazily. Hence, a statement like
{\tt x = Q.const(\{len = 10, qtype = I4, val = 0\})} does not actually create
ten
4-byte integers with value 0 as one might suspect. Data is populated only when
{\tt eval()} is explicity invoked on the vector or the data is
implicitly required by some other operator e.g. {\tt Q.print\_csv(x)}

Vectors are processed in chunks. Consider an expression  like \(\sum (a + b\times
c)\), written in \Q\ as {\tt d = Q.sum(Q.add(Q.mul(b,c), a))}.
When {\tt d} is eval'd, computation alternates between the {\tt mul, add, sum}
operators
processing chunks of data at a time until there are
no more.
The chunk size, \(n_C\), is chosen large enough that it is amenable to
vectorization and parallelization and small enough that its memory consumption
is low.

Vectors are not mutable (with few exceptions)
and must be produced sequentially. In other words, the \(i^{th}\) element must
be produced before the \({i+1}^{th}\). Vectors
operate in ``chunks'' of a fixed size. Let us say that the chunk size is 64K and
that we have produced 65K elements. In that case, the current chunk would have
only 1K elements. Whether one can get access to an element in the previous chunk
depends on whether the vector has been ``memo-ized''. The default behavior, with
a concomitant performance hit, is to memo-ize. However, when the programmer is
aware that the vector will be consumed in a streaming fashion, they set memo
to false.

Memo-izing is done by appending previous chunks in binary format to a file.
Subsequent reads of this vector are done by mmap-ing the file. Not all
algorithms are readily transformed into streaming operations e.g. sort. There
are a few cases where we support modifying a vector after it has been fully
materialized by opening it in write mode and mmap-ing it.

Mmap-ing gives us the illusion of a linear address space. This is useful to
incoporate algorithms and libraries that have not been written with streaming in
mind e.g., \Q\ borrows heavily from LAPACK when needed.

\Q's run time is an alternate approach to ``stream fusion''
\cite{Mainland2017}. In
that paper, the authors identify this as a technique that allows a compiler to
``cope with boxed numeric types, handle lazy evaluation, and
eliminate intermediate data structures''.


\documentclass{article}
%--------------------------------------------------------
\usepackage{hyperref}
\usepackage{caption}
\usepackage{subcaption}

\usepackage{graphicx}

\usepackage[left=2.9cm, right=2.9cm, top=3cm, bottom=2cm]{geometry}

\usepackage{fancyhdr}
\pagestyle{fancy}
\fancyhf{}
\rhead{31.01.2019}
\lhead{Pilz GmbH \& Co. KG}
\cfoot{\thepage}
\renewcommand{\headrulewidth}{0pt}
\renewcommand{\footrulewidth}{0pt}

\begin{document}
\title{Motion Blending}
\author{Pilz GmbH \& Co. KG}
\date{31.01.2019}

\maketitle

\begin{abstract}
\noindent This article describes the techniques used to blend the position and orientation of two robot trajectories within the {\tt pilz\_industrial\_motion\_planner}\cite{pilztrajectorygeneration} ROS package.
\end{abstract}

\newpage

\section{Position blending}

\subsection{Introduction}

Suppose we have two trajectories $x_1(t)$, $t\in[0,T_1]$ and $x_2(t)$, $t\in[0,T_2]$ and we want to make a transition from $x_1(t)$ to $x_2(t)$. The transition window technique assumes that the transition happens in a predefined time window. During this transition window, the resulted trajectory (blending trajectory) is given by:
\begin{equation}
	x_b(t) = x_1(t) + \alpha(s(t))(x_2(t)-x_1(t)), t \in [t_0, t_0 + T]
\end{equation}
in which $x_b(t)$ is the transition trajectory. $t_0$ is the start time of transition window and $T$ represents the transition time. $\alpha(s)$ is the blend function and $s$ is normalized time parameter:
\begin{equation}
s = \frac{t - t_0}{T}
\label{eq:1}
\end{equation}
which changes from 0 to 1 during the transition window. Following polynomial is selected as $\alpha(s)$ so that the boundary conditions at the start and end point of the transition window are fulfilled \cite{lloyd1993trajectory}:
\begin{equation}
\alpha(s) = 6s^5 - 15s^4 + 10s^3.
\label{eq:2}
\end{equation}

\subsection{Application for blending robot trajectory}
We want to move the robot from $p_1$ to $p_2$, then from $p_2$ to $p_3$. $p_2$ is a blending way-point which means that it does not need to be reached exactly. We want the robot moves alongside $p_2$ without stop. The whole process is described below with an one dimensional example.

\begin{enumerate}
	\item Generate motion trajectories $x_1(t)$ from $p_1$ to $p_2$ and $x_2(t)$ from $p_2$ to $p_3$. $x_1(t)$ and $x_2(t)$ both start and stop with zero velocity/acceleration. Both trajectories start with time zero. As a simple example we generated two one-dimensional linear trajectories in Figure.\ref{ori_traj}. For robot motion without blending, the two trajectories are executed one after the other, which means $x_2(t)$ needs to be timely shifted by the duration of $x_1(t)$.
	\begin{figure}[ht]%
	\includegraphics[width=0.9\columnwidth]{figure/original_trajectories.eps}%
	\caption{One-dimensional linear trajectory}%
	\label{ori_traj}
	\end{figure}

	\item According to the blending radius $r$, the points $p_{b1}$ on $x_1(t)$ and $p_{b2}$ on $x_2(t)$ which intersects with the blending sphere are computed. We also compute the durations $d_1$ for moving from $p_{b1}$ to $p_2$ on $x_1$ and $d_2$ for moving from $p_2$ to $p_{b2}$ on $x_2$. The transition window should start earliest from the time of $p_{b1}$ on $x_1(t)$ is reached, and ends latest at the time of $p_{b2}$ on $x_2(t)$ is reached. In the example, we take $r=3$ and $p_{b1} = p_{b2} = 5$ (see Figure.\ref{ori_traj}).

	\item Timely shift the $x_2(t)$ and select the transition window time $T$ according to the above rules. In order to avoid stop on the blending trajectory, the time shift $T_s$ of $x_2(t)$ should be smaller than the duration of $x_1(t)$. We now have the second trajectory as $x_2(t-T_s)$ for blending. In the example the duration of $x_1(t)$ is $6s$. Figure.\ref{blend_case_1} shows a blending case that we shift the $x_2(t)$ with $6s$, which is almost the same as motion without blending. Figure.\ref{blend_case_2} shows a blending case that we shift the $x_2(t)$ with $3.5s$ and the blending starts at 3.5s, ends at 5.5s. Figure.\ref{blend_case_3} shows a blending case that we shift the $x_2(t)$ with $4.5s$ and the blending starts at 3.5s, ends at 6.5s.
\end{enumerate}

In the actual implementation, we make the following choice given the durations $d_1$ and $d_2$ inside the blending sphere:
\begin{itemize}
\item If $d_1\leq d_2$ we shift exactly to the time when $p_{b1}$ is reached on $x_1$ (when the blending sphere is entered),
\item if $d_1>d_2$ we compute $T_s$ such that $T_s+d_2=T_1$. This choice minimizes increases in acceleration/deceleration on the resulting blend trajectory.
\end{itemize}

\section{Blending the orientation}
To blend the orientation, the method described in \cite{dantam2014orientation} is used. The equations (18)-(20) in \cite{dantam2014orientation} are used to calculate the orientation along the blend trajectory. In our application, due to the fact that the orientation change along the original (not blended) trajectories has smooth acceleration and deceleration phases, (18) and (19) from paper \cite{dantam2014orientation} do not need to be calculated.\newline
For the sake of clarity, it is important to note that our functions for $u_{ij}(t)$ and $u_{jk}(t)$ are different. However, we account for this difference by not explicitly calculating (18) and (19) and using the given samples of the original (not blended) trajectories instead. Furthermore,  (\ref{eq:2}) is used for $u_{j}(t)$, in other words, $u_{j}(t) = \alpha(s(t))$.

\begin{figure}[ht]
	\vspace*{-5mm}
	\centering
	\begin{subfigure}[ht]{0.8\textwidth}%
		\centering
		\includegraphics[width=0.8\columnwidth]{figure/blend_case_1.eps}%
		\caption{Motion blend case 1: $T_s = 6s$, blending starts at 3.5s, ends at 8s. The resulted blending trajectory comes to a stop in the middle.}%
		\label{blend_case_1}%
	\end{subfigure}

	\begin{subfigure}[ht]{0.8\textwidth}%
		\centering
		\includegraphics[width=0.8\columnwidth]{figure/blend_case_2.eps}%
		\caption{Motion blend case 2: $T_s = 3.5s$, blending starts at 3.5s, ends at 5.5s. The resulted blending trajectory smoothly transits from first trajectory to the second trajectory. The velocity profile has no jumps.}%
		\label{blend_case_2}%
	\end{subfigure}

	\begin{subfigure}[ht]{0.8\textwidth}%
		\centering
		\includegraphics[width=0.8\columnwidth]{figure/blend_case_3.eps}%
		\caption{Motion blend case 3: $T_s = 4.5s$, blending starts at 3.5s, ends at 6.5s. The resulted blending trajectory smoothly transits from first trajectory to the second trajectory. The velocity profile has no jumps.}%
		\label{blend_case_3}%
	\end{subfigure}

\end{figure}

\begin{thebibliography}{9}
\bibitem {lloyd1993trajectory}Lloyd, John and Hayward, Vincent \textit{Trajectory generation for sensor-driven and time-varying tasks}, The International journal of robotics research, 1993
\bibitem {dantam2014orientation}Dantam, Neil and Stilman, Mike \textit{Spherical Parabolic Blends for Robot Workspace Trajectories}, International Conference on Intelligent Robots and Systems (IROS), 2014
\bibitem{pilztrajectorygeneration}\url{https://github.com/ros-planning/moveit/tree/master/moveit_planners/pilz_industrial_motion_planner}
\end{thebibliography}
\end{document}


\documentclass[11pt]{scrartcl}
\usepackage[sexy]{evan}
\usepackage{placeins}
\usepackage{ amssymb }
\usepackage{verbatim}

\begin{document}
\newcommand{\Lagr}{\mathcal{L}}
\newcommand{\Ztran}{\mathcal{Z}}

\title{Control Theory: Theory and Applications for Robotics}
\subtitle{Including Tricks Learned on the Job}
\author{Maverick Zhang}
\date{\today}
\maketitle

\newpage
\tableofcontents
\newpage

\section{Introduction}

This is a paper that I hope will introduce new members to the UC Berkeley and M.I.T. Robotics group to the necessary mathematics and ideas in order to help us out with controlling the robots. A lot of fields intersect at robotics, the biggest of which are mathematics, physics, electrical engineering, mechanical engineering, and computer science. Whenever somebody writes a reference document of some kind, they are always going to be agonizing over how much is too much or too little. The more broad the content, the more easily a person completely unfamiliar to the current field will be able to learn it. The more strict the content, the less. Unfortunately, the tradeoff is that it takes far more effort to cover.

	My current plans for this introductory paper will be getting somebody who understands at least AP Calculus BC, and AP Physics C up to the level of a somebody who can intuitively make choices in robotics control theory. As such, I will be making sure understanding (even if it is not rigorous) is put first before rigorous theory. Eventually, sections that are completely rigorous will be written to shore up gaps in understanding.

	I have to say, while control theory may seem a difficult mathematical creature, the best way to learn it is to play with it yourself. I will be adding notes that will challenge you to understand a concept through experiment, which is the way this theory should be learned.

	I recommend you read this book in passes. The first pass is the skim. You read quickly trying to understand the ideas behind control theory and robotics. The second pass you should certainly pay closer attention, probably reading proofs in depth and taking as long as it needs to completely explain the proof without this as a reference. Any further passes should be done to ensure you have complete understanding and shoring up any weak points.

\section{A Fixer Upper: All The Math And Physics You Need Refreshed From High School}

Okay, time to level with you, your Calc BC or Physics C teacher might've been great, but they had a strict timeline to stick to and didn't let you explore some ideas behind these beautiful subjects. I hope that in this section we can get a little more acquainted with these two fields and also remember them a bit better.

\subsection{Calculus, But Better}

Calculus is a humongous subject. Most of you guys who finished calculus probably thought, ``Wow, that was hard, now it's time to forget all about it," or maybe, ``When am I ever going to use this subject?" Both of these responses are perfectly fine, but I hope that I can at least answer the second question.

	I think we all agree that word problems in calculus suuuuuuck. It's like, I know you're just disguising a math problem with words to prevent me from answering faster, and that nothing in real life ever uses calculus this way. Well, now let's try to come up with plausible ways to use it. In addition, I want to recast some ideas in calculus in a different perspective.

\subsubsection{Derivatives}

We can think of calculus as the mathematics behind approximation. Others might say change, but it is more fundamentally the former. As you should hopefully remember, the main two ingredients in calculus are derivatives and integrals, but I think the third ingredient is *drumroll* \ldots Taylor Series!

	I can hear you groaning already. Good thing we won't be using them as annoyingly this time around. The great thing about Taylor series, however much you hate them, is that they are a perfect illustration of this reinterpretation of calculus. The Taylor seriees shows you perfectly how calculus approximates, so let's recast derivatives in terms of these Taylor Series.

	The formula for Taylor series is

\[
\sum_{n=0}^{\infty}\frac{f^{(n)}(c)(x-c)^n}{n!}
\]
which is a basically creates an infinite series made of simple terms power functions that will approximate (most) functions rather well around a point $c.$ For some functions like $\sin(x)$ or $e^x$, this approximation will eventually approximate the function over all of $\mathbb{R}$, while with other functions like $1/(1-x)$ it will only work from $(-1,1)$. However, for our purposes, we won't be concerning ourselves with the crazy functions that don't have Taylor series (like $e^{-1/x^2}$ around $0$). Instead, we will only remark that Taylor series approximate a function really well around a certain point.

	The existence of Taylor series is intuitive enough, but just because we have an idea doesn't mean it's practical. So with Taylor series, what can we do to understand it better?

	Let's try to understand what happens when we forget about a huge portion of the series. We chop it off after $n=0$. Alright, so what does our Taylor series do for us? We approximate a function $f$ near $c$ to find that the best approximation is

\[
f(x) \approx f(c) \quad \text{when $x$ is near $c.$}
\]

%%%Add graph of this%%%

	That's pretty unexciting. I think everybody understands that if you want to approximate a function near a point, you can just use that point. What happens when we chop off the Taylor series at $n=1$? We will find something known as the \vocab{first order approximation} to the function:

\[
f(x) \approx f(c) + f'(c)(x-c) \quad \text{when $x$ is near $c.$}
\]

	I think we all agree that this is more exciting. This function is linear in $x$. It is known as the tangent line approximation because it is the line that best kisses the graph at this point. If we examine a graph, it is quite clear that this approximation will let you venture a little bit further than the previous one when approximating. It is from this expression that we can also see why the derivative is so important as the tool of approximation, it is the slope of the tangent line. However, this also shows us

%%%Add Tangent Line%%%

\end{document}


\XtoCBlock{Exp}
\label{block:Exp}
\begin{figure}[H]\includegraphics{Exp}\end{figure}

\begin{XtoCtabular}{Inports}
In & Input u\tabularnewline
\hline
\end{XtoCtabular}

\begin{XtoCtabular}{Outports}
Out & Result of exp(u)\tabularnewline
\hline
\end{XtoCtabular}

\subsubsection*{Description:}
Computation of the exponential of the input.

% include optional documentation file
\InputIfFileExists{\XcHomePath/Library/Math/Doc/Exp_Info.tex}{\vspace{1ex}}{}

\subsubsection*{Implementations:}
\begin{tabular}{l l}
\textbf{FiP8} & 8 Bit Fixed Point Implementation\tabularnewline
\textbf{FiP16} & 16 Bit Fixed Point Implementation\tabularnewline
\textbf{FiP32} & 32 Bit Fixed Point Implementation\tabularnewline
\end{tabular}

\XtoCImplementation{FiP8}
\index{Block ID!4848}
\nopagebreak[0]
% Implementation details
\begin{tabular}{l l}
\textbf{Name} & FiP8 \tabularnewline
\textbf{ID} & 4848 \tabularnewline
\textbf{Revision} & 0.1 \tabularnewline
\textbf{C filename} & Exp\_FiP8.c \tabularnewline
\textbf{H filename} & Exp\_FiP8.h \tabularnewline
\end{tabular}
\vspace{1ex}

8 Bit Fixed Point Implementation

% Implementation data structure
\XtoCDataStruct{Data Structure:}
\begin{lstlisting}
typedef struct {
     uint16        ID;
     int8          *In;
     int8          Out;
} EXP_FIP8;
\end{lstlisting}

\ifdefined \AddTestReports
\InputIfFileExists{\XcHomePath/Library/Math/Doc/Test_Exp_FiP8.tex}{}{}
\fi
\XtoCImplementation{FiP16}
\index{Block ID!4849}
\nopagebreak[0]
% Implementation details
\begin{tabular}{l l}
\textbf{Name} & FiP16 \tabularnewline
\textbf{ID} & 4849 \tabularnewline
\textbf{Revision} & 0.1 \tabularnewline
\textbf{C filename} & Exp\_FiP16.c \tabularnewline
\textbf{H filename} & Exp\_FiP16.h \tabularnewline
\end{tabular}
\vspace{1ex}

16 Bit Fixed Point Implementation

% Implementation data structure
\XtoCDataStruct{Data Structure:}
\begin{lstlisting}
typedef struct {
     uint16        ID;
     int16         *In;
     int16         Out;
} EXP_FIP16;
\end{lstlisting}

\ifdefined \AddTestReports
\InputIfFileExists{\XcHomePath/Library/Math/Doc/Test_Exp_FiP16.tex}{}{}
\fi
\XtoCImplementation{FiP32}
\index{Block ID!4850}
\nopagebreak[0]
% Implementation details
\begin{tabular}{l l}
\textbf{Name} & FiP32 \tabularnewline
\textbf{ID} & 4850 \tabularnewline
\textbf{Revision} & 0.1 \tabularnewline
\textbf{C filename} & Exp\_FiP32.c \tabularnewline
\textbf{H filename} & Exp\_FiP32.h \tabularnewline
\end{tabular}
\vspace{1ex}

32 Bit Fixed Point Implementation

% Implementation data structure
\XtoCDataStruct{Data Structure:}
\begin{lstlisting}
typedef struct {
     uint16        ID;
     int32         *In;
     int32         Out;
} EXP_FIP32;
\end{lstlisting}

\ifdefined \AddTestReports
\InputIfFileExists{\XcHomePath/Library/Math/Doc/Test_Exp_FiP32.tex}{}{}
\fi