\section{Exploring Errors}

%Introduction
\begin{frame}{Exploring Errors}
\begin{itemize}
    % \item After building and understanding all three of our models, we began to explore the errors associated with them
    \item Our definition of error:

    \begin{align*}
        \mbox{error} = \frac{||v_{observed} - v_{predicted}||}{||v_{observed}||}
    \end{align*}
    \item While this is not the only metric we used (we also used error square and angular velocity error), this was our primary one
    % \item In the following slides, we will present some of the experiments we conducted to better understand and reduce predictive error from the models

\end{itemize}

\end{frame}

%Experiment 1: Errors and Positions
%\subsection{Experiment 1: Errors and Positions}
\begin{frame}{Experiment 1: Errors and Position}
    \begin{itemize}
        \item Question: Could measurement errors in position and velocity explain the difference between our $v_{predicted}$ and the $v_{observed}$
        \item To this end, we began to iteratively change the measured x and y COM and contact positions of the ellipse by a percentage
    \end{itemize}

    \begin{figure}[!htb]
    \minipage{0.32\textwidth}
      \includegraphics[width=\linewidth]{figures/changeInXPos.jpg}
      \caption{Changing the contact x position in Wang}
      \label{fig:awesome_image1}
    \endminipage\hfill
    \minipage{0.32\textwidth}%
      \includegraphics[width=\linewidth]{figures/errorVsOptMethod.png}
      \caption{Changing the contact x position in IRB}
      \label{fig:awesome_image3}
    \endminipage\hfill
    \minipage{0.32\textwidth}
      \includegraphics[width=\linewidth]{figures/trial900.jpg}
      \caption{Changing the angle of the ellipse in IRB}
      \label{fig:awesome_image2}
    \endminipage
    \end{figure}

\end{frame}

\begin{frame}{Experiment 1: Errors and Position}
Findings: across all three models, changing by a percentage the positions of the ellipse had a big impact on the accuracy of the results \\
\vspace{\baselineskip}
In general, this is what we saw:
    \begin{itemize}
        \item The measured position already gave a small error (few cases)
        \item Changing the measured position by a little bit gave a large drop in error (most cases)
        \item Changing the measured position by a lot to give a noticeable drop in error (few cases)
    \end{itemize}
\vspace{\baselineskip}
Conclusion: the data we were provided is not completely accurate!

\end{frame}

%Experiment 2: Errors and Moments
%\subsection{Experiment 2: Errors and Moments}
\begin{frame}{Experiment 2: Errors and Impact Angle}
At first, we were able to find a relationship between the error and the impact angle.

    \begin{itemize}
%        \item The pre-impact angle is related to both the moment arm and the geometry of the ellipse, which in turn directly affects the torque resulting from the impulse pair
        \item After running our models, we saw a relation between impact angles and the normalized errors (AP and Classic IRB)
        %\item For IRB with torque, the average width (the error offset) was also related to the impact angle.
    \end{itemize}
\vspace{0.5\baselineskip}
Which means that Some impact angles prevented some of the models from accurately predicting the post-impact states.

    \vspace{0.5\baselineskip}
     We will be exploring HOW that link came to be in later slides, but we are still unsure of WHY exactly the models systematically mispredict at certain angles.
\end{frame}

\begin{frame}{Experiment 2: Errors and Impact Angle}

 \begin{figure}
 \centering
        \includegraphics[scale=0.12]{figures/IRBWidthAngle.jpg}
        \caption{IRB With Torque}
        \label{fig:IRBAngle}
\end{figure}
\vspace{-1\baselineskip}

\begin{itemize}
    \item Width can be interpreted as Error
    \item We can see that peaks here occur around -110, -60, 60, 110

\end{itemize}

\end{frame}

\begin{frame}{Experiment 2: Errors and Impact Angle}
\begin{itemize}
    \item Does this trend extend to other models we have looked at?
\end{itemize}

\begin{figure}
    \centering
    \quad
    \begin{subfigure}[b]{0.45\linewidth}
        \includegraphics[scale=0.11]{figures/APAngleVsError.jpg}
        \caption{AP Poisson}
        \label{fig:AP_angle}
    \end{subfigure}
    \quad
    \begin{subfigure}[b]{0.45\linewidth}
        \includegraphics[scale=0.11]{figures/CIRBAngleVsError.jpg}
        \caption{Classical IRB}
        \label{fig:WangAngle}
    \end{subfigure}
\end{figure}
\vspace{-1\baselineskip}

\begin{itemize}
    \item The answer is, kind of!
\end{itemize}

\end{frame}

% \begin{frame}{The Link}

% Overall, this is the link that we have observed between all of our variables
% \vspace{0.5\baselineskip}

% %NEEDS INFO CHECK TO MAKE SURE THIS IS CORRECT

% \begin{itemize}
%     \item  Experiment 3 ($\Delta \dot \theta$ and Impact Angles) - At specific angles, the change in angular velocity is greater than at others. $\Rightarrow$
%     \item Experiment 4 ($\Delta \dot \theta$ and Torque) - There is a smaller change in angular velocity at certain angles because at those angles the resultant torque is smaller. $\Rightarrow$
%     \item Experiment 5 ($\Delta \dot \theta$ and Errors) - The smaller the change in angular velocity, the greater the prediction error. $\Rightarrow$
%     \item  Experiment 6 (Moments and Errors) - The smaller the Torque, the greater the prediction error.
% \end{itemize}

% \vspace{0.25\baselineskip}
% %\begin{itemize}
%         %\item In summary: pre-impact angle is related to both the moment arm and the geometry of the ellipse, which in turn directly affects the torque resulting from the impulse pair
%         In summary: At specific angles, there are smaller resultant torques, and at these smaller resultant torques, the prediction error of the IRB model is greater.
%     %\end{itemize}

% \end{frame}

%Experiment 3: Impact angles and Change in angular velocity
\begin{frame}{Experiment 3: $\Delta \dot \theta$ and Impact Angles}

\begin{itemize}
    \item We expected the ellipse to have a greater change in angular velocity if it didn't fall right on its center of mass.
    \item The greater the moment arm, the greater the change in angular velocity.
\end{itemize}
\begin{figure}
    \centering
    \includegraphics[scale=0.12]{figures/impact angle vs change in angular velocity.jpg}
    \caption{All moments vs Error}
    \label{fig:MomentsError}
\end{figure}
\item At around -110, 60, 110 the change in angular velocity is close to 0.
\end{frame}

%Experiment 4: Change in angular velocity and Torque
\begin{frame}{Experiment 4: $\Delta \dot \theta$ and Torque}
%this kind of ties in to the previous slide/it's already been said that greater change in angular velocity leads is related to greater moment/torque so maybe just delete this one?

\begin{figure}
    \centering
    \includegraphics[scale=0.21]{figures/AngularVvsMoment.jpg}
    \caption{Classic IRB: Change in Angular Velocity vs Total Moment}
    \label{fig:MomentsError}
\end{figure}

\end{frame}

%Experiment 5: Link between angular velocity and error
\begin{frame}{Experiment 5: $\Delta \dot \theta$ and Errors}

\begin{figure}
    \centering
    \includegraphics[scale=0.17]{figures/changeInOmegaEllipse.jpg}
    \caption{Classic IRB: Change in Angular Velocity vs Error}
    \label{fig:AngularVError}
\end{figure}
\end{frame}

%The important part about this slide is the idea of overcompensation. The width isn't just helping adjust for changes in angular velocity that are too large for the IRB to predict and need more torque.... It is allowing for the IRB to change the impulses and create large torques/changes in omega which the width can then cancel out potentially

%Experiment 6: Link between Moments and Error [editttttttt]
\begin{frame}{Experiment 6: Moments and Errors}

% Since a greater angular velocity change led to a smaller error, and the angular velocity change is directly related to the net torque, As a sanity check, we graphed the net torque vs the normalized error hoping to see the same trend.

%\vspace{0.25\baselineskip} [PRESENTATION NOTES]
%The difference between these next 2 plots is the magnitude of the normalized error. IRB torque has a minor error magnitude compared to the classical model, because the compression width applies a moment to "Correct" the angular velocity.

\begin{figure}
    \centering
    \quad
    \begin{subfigure}[b]{0.4\linewidth}
        \includegraphics[scale=0.105]{figures/IRB Classic Net torque vs error.jpg}
        \caption{Net Torque vs Error}
        \label{fig:AP_angle}
    \end{subfigure}
    \quad
    \begin{subfigure}[b]{0.5\linewidth}
        \includegraphics[scale=0.13]{figures/MomentProp2.jpg}
        \caption{IRB Moment vs Torque}
        \label{fig:MomentIRB}
    \end{subfigure}
\end{figure}

\end{frame}

\begin{frame}{The Link}

\tikzstyle{startstop} = [rectangle, rounded corners, minimum width=2.5cm, minimum height=1.75cm, text centered, text width = 4cm, draw=black]
\tikzstyle{arrow} = [thick,->,>=stealth]
\begin{tikzpicture}[node distance=2cm]

\node (exp2) [startstop, xshift = 3.5cm, yshift = 11cm] {\textbf{Experiment 2:} \\ Error is maximized at specific angles};
\node (exp3) [startstop, xshift = 7.25cm, yshift = 8.5 cm] {\textbf{Experiment 3:} \\ $\Delta \dot \theta$ is min at these angles};
\node (exp4) [startstop, xshift = 6.35cm, yshift = 6cm] {\textbf{Experiment 4:} \\  $\Delta \dot \theta$ decreases as the res. torque decreases};
\node (exp5) [startstop, xshift = 0.65cm, yshift = 6cm] {\textbf{Experiment 5:} \\ $\Delta \dot \theta$ decreases as prediction error increases};
\node (exp6) [startstop,xshift = 0.25cm, yshift = 8.5cm] {\textbf{Experiment 6:} \\ Therefore as torque decreases, error increases};

\draw [arrow] (exp3) -- (exp4);
\draw [arrow] (exp4) -- (exp5);
\draw [arrow] (exp5) -- (exp6);
\draw [arrow] (exp6) -- (exp2);
\draw [arrow] (exp2) -- (exp3);

\end{tikzpicture}
\end{frame}


\documentclass{article}
\usepackage{graphicx}
\usepackage{amssymb}
\graphicspath{{figures/}}
\DeclareGraphicsRule{*}{mps}{*}{}

\usepackage{color}
\newcommand{\red}[1]{{\color{red} #1}}

\begin{document}
\section{Increasing numbers}
The graphs are generated by a sequence of numbers $\alpha_i\in\mathbb{Z}$ appearing in random (not sorted) order from left to right.
\[\{\alpha_1,...,\alpha_n\}\]
We set $s=\alpha_1$ and $t=\alpha_n$. Odd numbers are red.
Two numbers $\alpha_i,\alpha_j$ are connected if:
\begin{itemize}
\item $i<j$
\item $\alpha_i<\alpha_j$
\end{itemize}
The edges are directed from the smaller to the larger number.
Some examples:
\begin{figure}[h]
  \centering
\includegraphics{increase_n8_1.0}
  \caption{increase_n8_1, n=8}
\end{figure}
\begin{figure}[h]
  \centering
\includegraphics{increase_n8_2.0}
  \caption{increase_n8_2, n=8}
\end{figure}
\begin{figure}[h]
  \centering
  \includegraphics{increase_n8_3.0}
  \caption{increase_n8_3, n=8}
\end{figure}

\end{document}


\chapter{Resource Locations}
\label{1}

\abstract{This chapter defines resources and their locations in the file system.}

\section{What is a resource?}
\label{sec:1}
Use the template

\begin{equation}
a \times b = c\;,
\end{equation}

\begin{eqnarray}
a \times b = c \nonumber\\
\vec{a} \cdot \vec{b}=\vec{c}
\label{eq:01}
\end{eqnarray}

%puts image at the bottom of the page
\begin{figure}[b]
\sidecaption
\includegraphics[scale=.20]{figure.png}
\caption{If the width of the}
\label{fig:1}
\end{figure}

\paragraph{Paragraph Heading}
Instead of simply

%puts image at the top of the page
\begin{figure}[t]
\sidecaption[t]
\includegraphics[scale=.20]{figure.png}
\caption{Please write your figure caption here}
\label{fig:2}       % Give a unique label
\end{figure}

\runinhead{Run-in Heading Boldface Version} Use the \LaTeX\ automatism for all your cross-references and citations as has already been described in Sect.~\ref{sec:2}.

\begin{table}
\caption{Please write your table caption here}
\label{tab:1}
    \begin{tabular}{p{2cm}p{2.4cm}p{2cm}p{4.9cm}}
    \hline\noalign{\smallskip}
    Classes & Subclass & Length & Action Mechanism  \\
    \noalign{\smallskip}\svhline\noalign{\smallskip}
    Translation & mRNA$^a$  & 22 (19--25) & Translation repression, mRNA cleavage\\
    Translation & mRNA cleavage & 21 & mRNA cleavage\\
    Translation & mRNA  & 21--22 & mRNA cleavage\\
    Translation & mRNA  & 24--26 & Histone and DNA Modification\\
    \noalign{\smallskip}\hline\noalign{\smallskip}
    \end{tabular}
$^a$ Table foot note (with superscript)
\end{table}

\begin{description}[Type 1]
\item[Type 1]{That addresses central themes pertainng to migration, health, and disease. In Sect.~\ref{sec:1}, Wilson discusses the role of human migration in infectious disease distributions and patterns.}
\item[Type 2]{That addresses central themes pertainng to migration, health, and disease. In Sect.~\ref{subsec:2}, Wilson discusses the role of human migration in infectious disease distributions and patterns.}
\end{description}

\begin{theorem}
Theorem text goes here.
\end{theorem}

\begin{definition}
Definition text goes here.
\end{definition}

\begin{proof}
Proof text goes here.
\qed
\end{proof}


\begin{appendix}

\chapter{Algorithm for balancing lookup tables train set}\label{Chapter:results}

Since the dataset for the lookup tables is small -- even if length 2 compositions was created using tables \textit{t1} to \textit{t8}, there could only be 64 compositions which would bijectively map 8 \lq  3 bit inputs to 3 bit outputs\rq{} thereby leading to a total of 512 samples only -- we try to ensure that the compositions present in the training data are uniformly distributed while simultaneously keeping the distribution of the output strings in the training data uniform as well. This ensures that the model doesn't get biased towards any particular composition or an output string. As has been explained in section \ref{lt:splits}, the heldout inputs set is created by randomly taking out 2 inputs for each of the 28 compositions in training. Therefore the resultant training set should have the following properties:
\begin{itemize}
	\item The total number of data points  $= 28*6 = 168$.
	\item There are 8 outputs. The total number of compositions that lead to a particular output $=168/8 = 21$.
\end{itemize}

.
\begin{algorithm}
	\caption{Create training with uniform distribution of both compositions and outputs}
	\begin{algorithmic}
		\STATE We start with 28 compositions with 8 inputs each.
		\STATE Create an empty dataframe of dimension (21,8) with the 8 output strings as it's columns.
		\STATE Create a dictionary Y with compositions as it's keys and values=0.
		\FOR {i in range [0, 21)}
		\FOR {output in dataframe column}
		\STATE Sample composition
		\IF{composition \textbf{not in} row[i] AND composition \textbf{not in} output column AND Y[composition] $<$ 21 }
		\STATE dataframe[i][output] = composition
		\STATE Y[composition] += 1
		\ELSE
		\STATE continue
		\ENDIF
		\ENDFOR
		\ENDFOR
	\end{algorithmic}
\end{algorithm}

\end{appendix}


%
% Complete documentation on the extended LaTeX markup used for Insight
% documentation is available in ``Documenting Insight'', which is part
% of the standard documentation for Insight.  It may be found online
% at:
%
%     http://www.itk.org/

\documentclass{InsightArticle}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
%  hyperref should be the last package to be loaded.
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\usepackage[dvips,
bookmarks,
bookmarksopen,
backref,
colorlinks,linkcolor={blue},citecolor={blue},urlcolor={blue},
]{hyperref}
% to be able to use options in graphics
\usepackage{graphicx}
% for pseudo code
\usepackage{listings}
% subfigures
\usepackage{subfigure}

%  This is a template for Papers to the Insight Journal.
%  It is comparable to a technical report format.

% The title should be descriptive enough for people to be able to find
% the relevant document.
\title{Parallel algorithms for erosion and dilation of label images.}

\newcommand{\IJhandlerIDnumber}{3399}

% Increment the release number whenever significant changes are made.
% The author and/or editor can define 'significant' however they like.
\release{0.00}

% At minimum, give your name and an email address.  You can include a
% snail-mail address if you like.
\author{Richard Beare{$^1$} {\small and} Paul Jackway{$^2$}}
\authoraddress{Richard.Beare@monash.edu\\Department of Medicine\\Monash University\\Melbourne\\Australia{$^1$}\\CSIRO Mathematics Informatics and Statistics\\Dutton Park\\Queensland\\Australia{$^2$}}

\begin{document}

\IJhandlefooter{\IJhandlerIDnumber}

\maketitle

\ifhtml
\chapter*{Front Matter\label{front}}
\fi

\begin{abstract}
\noindent
It is sometimes useful to be able to apply binary morphological
operations, such as erosions and dilations, to labelled images in a
fashion that preserves the labels. This article introduces a
specialised class implementing parallel methods described in
\cite{beare2011parallel} that provide very fast dilations by circles
and spheres of arbitary size. Comparisons with other implementations
using currently available building blocks are also made.
\end{abstract}
\IJhandlenote{\IJhandlerIDnumber}
\tableofcontents

\section{Introduction}
The link between Euclidean distance transforms and binary
morphological operations is well known - erosions and dilations by
circles and spheres can be performed by thresholding the distance
transform. Efficient and readily parallelizable distance transform
algorithms can, in turn, be based on erosions and dilations by
parabolic structuring elements. The classes outlined in this article
extend the contact point algorithm used for parabolic erosions and
dilations to facilitate operations on label images. A more complete
discussion is available in \cite{beare2011parallel}. The classes
introduced here are able to separate touching labels during erosion
and split touching labels at the midpoint during dilation.

\section{The classes}
The {\em itk::LabelSetDilateImageFilter} and {\em
  itk::LabelSetErodeImageFilter} implement dilation and erosion of
label images. They share a common parent class. The methods controlling class behaviour are:
\begin{itemize}
\item {\em UseImageSpacing}: Defines whether the radius refers to voxels or world dimensions. Default is false, meaning radius is in voxels.
\item {\em Set/GetRadius}: Set the size of the dilation or erosion. There are versions to set the size in all directions to be the same, corresponding to a circular or spherical structuring element, and independently, corresponding to an ellipsoid structuring element (with axes parallel to image axes).
\end{itemize}

Examples of use are available with the package.

Please note that these are specialised classes which cannot use
arbitrary structuring elements.

The code from this contribution is also available at
\url{https://github.com/richardbeare/LabelErodeDilate}.

\subsection{Notes for label erosion}
The class provided for label erosion {\bf will separate} touching
labels. If this is not the desired behaviour, then implement label
erosion via binary erosion and masking.

\section{Alternative approaches to label dilation}
As advised on the ITK mailing list, label dilation can be implemented
via distance transforms and watershed transforms. This algorith is
illustrated in SimpleITK python code below (courtesy of Bradely
Lowekamp):

\lstset{language=Python}
\begin{lstlisting}
def MultilabelDilation(img, radius=1,kernel=sitk.BinaryDilateImageFilter.Ball):
    distImg = sitk.SignedMaurerDistanceMap(img != 0,
                                           insideIsPositive=False,
                                           squaredDistance=False,
                                           useImageSpacing=False)
    dilatImg = sitk.BinaryDilate(img!=0, radius, kernel)
    wsImg = sitk.MorphologicalWatershedFromMarkers(distImg, img)
    return dilatImg*wsImg
\end{lstlisting}

There are a couple of altnernatives to this algorithm implemented in
C++ and provided in {\em multilabelDilation.h}. The first version is a
variant of the code above, which avoids using the binary dilate
operation and thresholds the distance transform instead. This version
is called {\em multilabelDilation}. The second version uses the {\em
DanielssonDistanceMapImageFilter} to produce both a distance map and a
Voronoi tesselation. The distance map is thresholded to produce a
dilation which is then used to mask the Voronoi tesselation. This
version is called {\em multilabelDilationDanielsson}. The Danielsson
filter is slower than the Maurer filter, but this approach avoids the
watershed transform step as the information provided by the Voronoi
map is removes the need for the watershed. Comparisons of performance
are below.

\subsection{Problems with the watershed approach}
It turns out that the distance transform followed by watershed
transform approach to label dilation only works reliably for smaller
dilations, say 10. For larger dilations there is a chance of regions
leaking across borders and becoming incorrect. Leaks of this kind
result from the nature of propagation in the border zone combined with
tied distance values. The test image {\em vCenters10.mha} demonstates
this issue.

\subsection{Dilations via distance maps versus binary dilation}
There are subtle differences between the effective structuring element
produced when thresholding distance maps versus those provided by the
{\em BinaryBallStructuringElement}. The latter is a Bresenham circle
or sphere, which means that any voxel which is partly inside the
specified radius is included in the structuring elements. Distance
maps, on the other hand, typically compute distances to voxel
centres. Thus any voxel whos centre is closer than the specified
radius is included, resulting in a slightly smaller structuring
element.

In addition, the parabolic dilation used in the LabelDilate tool
treats voxels with centres exactly the dilation radius from the seed
as ``outside''. This results from the optimized use of the contact
point algorithm to avoid computation of a complete distance
transform. There may be some fudge factors that can be added to avoid
this behaviour, which will be investigated in the future.

\subsection{Other differences}
There are minor differences in results obtained from the Danielsson
approach and the new approach discussed here. Some of those
differences are along boundaries. My investigations suggest that the
Danielsson approach is wrong in these cases. Explicit calculation of
the distance between the locations of seeds and differences in
labelling indicate that the points are incorrectly labelled by the
Danielsson approach. The {\em reportNonZero} tool and {\em check.R}
functions were used to investigate these problems.

\section{Performance}
The specialised version was developed due to ease of parallel
implementation and performance results show that it is indeed much
faster than versions that can be built using existing tools. It also
scales moderately well with increased execution threads. The execution
times and speedups for a 12 core Intel(R) Xeon(R) CPU X5650 @ 2.67GHz
on a $182 \times 218 \times 182$ brain atlas are shown in Figures
\ref{fig:exec}. None of the methods have a
significant dependence on dilation size, as the structuring element is
not explicit. Speedup at 8 cores for the Maurer method is 1.86
versus 4.71 for the parabolic method. The lack of scalability of the
Maurer method is likely to be largely caused by the watershed step,
which is not parallel. The Danielsson approach is much slower, despite
avoiding the need for an explicit watershed transform. There is some
redundancy in the Maurer approach, as a signed distance transform is
computed by not used. In the single threaded the specialised version
is 7 times faster than the Mauer approach and 180 times faster than
the Danielsson-based method.

\begin{figure}[htbp]
\centering
\includegraphics[scale=0.35]{exectimes}
\includegraphics[scale=0.35]{speedups}
\caption{Execution times and speedups for all methods.\label{fig:exec}}
\end{figure}

\section{Sample results}
Label images occur in many situations. This is an example of a brain
atlas in which different labels represent different anatomical
regions. Examples of 2D processing (operations applied to a single
slice of the atlas) are shown in Figure \ref{fig:2d}. Examples of a 3D
processing are shown in Figures \ref{fig:3dorig} to \ref{fig:3ddil}.

\begin{figure}[htbp]
\centering
\includegraphics[scale=0.75]{axial_color}
\includegraphics[scale=0.75]{laberode2d_color}
\includegraphics[scale=0.75]{labdilate2d_color}
\caption{Original, eroded and dilated atlases - a single slice with a 2d circular structuring element.\label{fig:2d}}
\end{figure}

\begin{figure}[htbp]
\centering
\includegraphics[scale=0.15]{atlas_orig}
\caption{Axial, sagittal and coronal slices through a labelled brain atlas.\label{fig:3dorig}}
\end{figure}

\begin{figure}[htbp]
\centering
\includegraphics[scale=0.15]{atlas_erode_color}
\caption{Atlas in Figure \ref{fig:3dorig} after applying 3d labelled erosion. \label{fig:3dero}}
\end{figure}

\begin{figure}[htbp]
\centering
\includegraphics[scale=0.15]{atlas_dilate_color}
\caption{Atlas in Figure \ref{fig:3dorig} after applying 3d labelled dilation. \label{fig:3ddil}}
\end{figure}

\section{Conclusion}
This article provides two classes for erosions and dilations of label
images using algorithms developed in a previously published
work. These methods use parallel, scan-line base algorithms that offer
very fast operations.
\bibliographystyle{plain}
\bibliography{local,InsightJournal}
\nocite{ITKSoftwareGuide}

\end{document}


\subsubsection{\stid{3.06} PETSc-TAO} \label{subsubsect:petsc}
\paragraph{Overview}

Algebraic solvers (generally nonlinear solvers that use sparse linear solvers) and integrators form the
core computation of many numerical simulations. No scalable ``black box'' sparse solvers or integrators
work for all applications, nor are there single implementations that work well for all problem sizes.
Hence, algebraic solver and integrator packages provide a wide variety of algorithms and implementations
that can be customized for the application and range of problem sizes. PETSc/TAO~\cite{petsc:homepage,petsc-man}
is a widely used numerical library for the scalable solution of linear, nonlinear, and variational systems,
for integration of ODE/DAE systems and computation of their adjoints, and for numerical optimization.
This project focuses on three topics: (1) partially matrix-free scalable solvers to efficiently use
many-core and GPU-based systems; (2) reduced synchronization algorithms that can scale to larger
concurrency than solvers with synchronization points; and (3) performance and data structure
optimizations for all the core data structures to better utilize many-core and GPU-based
systems as well as provide scalability to the exascale systems.

The availability of systems with over 100 times the processing power of today's machines compels the utilization
of these systems not just for a single ``forward solve'' (as discussed above), but rather within a tight loop
of optimization, sensitivity analysis (SA), and uncertain quantification (UQ). This requires the implementation
of a new scalable library for managing a dynamic hierarchical collection of running scalable simulations, where
the simulations directly feed results into the optimization, SA, and UQ solvers.  This library, which we call
libEnsemble, directs the multiple concurrent ``function evaluations'' through the tight coupling and
feedback. This work consist of two parts: (1) the development of libEnsemble; and (2) the development
of application-relevant algorithms to utilize libEnsemble.

\paragraph{Key Challenges}

A key challenge for scaling the PETSc/TAO numerical libraries to Exascale systems is that traditional
``sparse-matrix-based'' techniques for linear, nonlinear, and ODE solvers, as well as optimization
algorithms, are memory-bandwidth limited.  Another difficulty is that any synchronizations
required across all compute units---for example, an inner product or a norm---can
dramatically affect the scaling of the solvers.  Another challenge is the need to
support the variety of accelerators that will be available on the exascale systems
and the programming models that application teams use for performance
portability.

Running an ensemble of simulations requires a coordination layer that handles load balancing and
allows the collection of running simulations to grow and shrink based on feedback. Thus, our
libEnsemble library must be able to dynamically start simulations with different parameters,
resume simulations to obtain more accurate results, prune running simulations that the solvers
determine can no longer provide useful information, monitor the progress of the simulations,
and stop failed or hung simulations, and collect data from the individual simulations both
while they are running and at the end.

\paragraph{Solution Strategy}

To address the scalability of the numerical libraries, we implemented new solvers and data
structures including: pipeline Krylov methods that delay the use of the results of inner
products and norms, allowing overlapping of the reductions and other computation; partially
matrix-free solvers using high-order methods that have high floating-point-to-memory-access
ratios and good potential to use many-core and GPU-based systems; and in-node optimizations
of sparse matrix-matrix products needed by algebraic multigrid to better utilize many-core
systems.

Our strategy for coordinating ensemble computations has been to develop libEnsemble
to satisfy our needs.  This library should not be confused with workflow-based
scripting systems; rather it is a library that, through the tight coupling and
feedback, directs the multiple concurrent ``function evaluations'' needed by
optimization, SA, and UQ solvers.

\paragraph{Recent Progress}

In the past year, we have released PETSc/TAO 3.14 (available at \url{http://www.mcs.anl.gov/petsc}),
which features enhanced GPU support.  The library now supports CUDA-11 and HIP, along with CUDA-aware
MPI, which allows direct communication of data between Summit GPUs, bypassing the previously needed
step of first copying the data to the CPU memory. This enhancement reduces the latency of the
communication and improves bandwidth.  An experimental Kokkos backend for some matrix and
vector operations using KokkosKernels was also provided, as one step in the refactoring
process to support the variety of accelerators needed for exascale systems and the
programming models for performance portability wanted by applications.

\begin{figure}
\centering
\includegraphics[trim = 0in .2in 1.7in .2in, clip, width=0.9\textwidth]{projects/2.3.3-MathLibs/2.3.3.06-PETSc-TAO/petsc_arch}
\caption{The improved PETSc/TAO architecture enables users to utilize a variety of programming
models for GPUs independently of PETSc's internal programming model.}
\label{fig:petsc-tao-fig}
\end{figure}

We have also release libEnsemble 0.7.1 (available at \url{https://github.com/Libensemble/libensemble}).
This release includes new generator functions and examples, changes to become xSDK compatible, and
improved testing across available platforms.

\paragraph{Next Steps}

Our next efforts are:
\begin{enumerate}
  \item \textbf{Performance and application assessment}:
  We will provide updated performance reports of PETSc/TAO on the architectures available to us.
  We will work with our applications to assess the usage of our software technologies and our
  progress toward reaching our impact goals.
  We will add a libEnsemble guide for function writer users to the documentation and survey
  the libEnsemble user community.
  \item \textbf{PETSc/TAO release with full-functionality on available hardware}:
  We will release a version of PETSc/TAO that fully supports the hardware and software on the
  architectures available to us.
  We will begin testing important kernels using different backends and prepare more methods to utilize accelerators.
  \item \textbf{libEnsemble release with enhanced capabilities}:
  We will release a version of libEnsemble that implements a method for Bayesian calibration.
  We will connect libEnsemble to continuous integration tools and demonstrate capabilities.
  \item \textbf{PETSc/TAO release focused on performance on available hardware}:
  We will release a version of PETSc/TAO with performance improvements on the architectures available to us.
  We will continue testing important kernels using different backends and optimize more methods to
  utilize accelerators.
\end{enumerate}


\section{Understanding ``Out Of Thin Air'' using Temporal Logic}
\label{sec:logic}

A significant challenge for a software memory model is to relax order enough
to allow efficient implementation without admitting anomalous
behaviors---called \emph{out of thin air} (\oota) in the literature
\cite{vacuous,DBLP:conf/esop/BattyMNPS15,BoehmOOTA}.  The most famous example
is \ref{OOTA3} from \textsection\ref{sec:intro}.  Here we inline
initialization in order to fit the format of our proof rules:
\begin{align}
  \label{OOTA3} \tag{\textsc{oota1}}
  \PW{y}{0}\SEMI
  \PW{y}{x}
  \!\PAR\!
  \PW{x}{0}\SEMI
  \PR{y}{r}\SEMI \PW{x}{r}
  &&
  %\nonumber
  \smash{\hbox{\begin{tikzinline}[node distance=1.2em]
  \event{rx}{\DR{x}{1}}{}
  \event{wy}{\DW{y}{1}}{right=of rx}
  \po{rx}{wy}
  \event{y0}{\DW{y}{0}}{left=of rx}
  \event{x0}{\DW{x}{0}}{right=2em of wy}
  \event{ry}{\DR{y}{1}}{right=2em of x0}
  \event{wx}{\DW{x}{1}}{right=of ry}
  \po{ry}{wx}
  \rf[out=10,in=170]{wy}{ry}
  \rf[out=170,in=10]{wx}{rx}
  \wk[out=-15,in=-165]{y0}{wy}
  \wk[out=-15,in=-165]{x0}{wx}
    \end{tikzinline}}}
\end{align}
Although Java does not allow \oota{} behaviors of \ref{OOTA3},
\citet{DBLP:journals/toplas/Lochbihler13} showed that it does allow \oota\
behaviors of \ref{OOTA1}, also from \textsection\ref{sec:intro}.  In
\cite{DBLP:conf/lics/JeffreyR16}, we described a logic that rules out
\ref{OOTA3} but not \ref{OOTA1} or its variant \ref{OOTA4}.  In this section,
we provide a more accurate test of \oota{} behaviors by enhancing our
previous logic with temporal features.

On first read, we suggest that readers skip to the examples and the
discussion that follows, coming back to the definitions as necessary.
Example~\ref{ex:thin} discusses the canonical \oota{} example \ref{OOTA3};
the analysis is trivial and well-known \cite{DBLP:conf/lics/JeffreyR16,
  DBLP:conf/popl/KangHLVD17}.  Example~\ref{ex:lochb} is more interesting.
There, we discuss \ref{OOTA4}, which is a variant of
\citeauthor{DBLP:journals/toplas/Lochbihler13}'s \ref{OOTA1}.
% In this case, the violation is a subtler
% temporal property.  We develop a logic sufficient to prove that our semantics
% disallows \oota\ on \eqref{lochbihler}.

The logic given here is not meant to be definitive; in
\textsection\ref{sec:limits}, we discuss \oota{} examples that appear to
require non-trivial extensions
\cite{DBLP:conf/esop/SvendsenPDLV18,DBLP:journals/pacmpl/ChakrabortyV19}.

\noparagraph{Definitions}
We adapt past linear temporal logic (\pLTL)
\cite{Lichtenstein:1985:GP:648065.747612} to pomsets by dropping the previous
instant operator and adopting strict versions of the temporal operators.
The atoms of our logic are write and read events.
% \begin{displaymath}
%   \afo \QUAD::=\QUAD
%   \DR{\aLoc}{\aVal}
%   \mid
%   \DW\aLoc\aVal
%   \afo \wedge\bfo
%   \mid \lnot \afo
%   \once\afo
%   \mid \always\afo
% \end{displaymath}
%\begin{definition} %[Satisfaction]
Given a pomset $\aPS$ and event $\aEv$, define:\nofootnote{Let $\FALSE$, $\lor$,
  $\Rightarrow$ and $\once$ as usual;
  for example,
  $\once\afo = \lnot(\always\lnot\afo)$.}
\begin{displaymath}
  \renewcommand{\arraycolsep}{.2ex}
    \begin{array}{lrll}
      \aPS,\aEv &\models& \DW{\aLoc}{\aVal} &\text{ if } \labelingAct(\aEv) = \DW{\aLoc}{\aVal} \text{ and } \TRUE \text{ implies } \labelingForm(\aEv) \\
      \aPS,\aEv &\models& \DR{\aLoc}{\aVal} &\text{ if } \labelingAct(\aEv) = \DR{\aLoc}{\aVal} \text{ and } \TRUE \text{ implies } \labelingForm(\aEv) \\
      \aPS,\aEv &\models& \afo\land\bfo &\text{ if } \aPS,\aEv \models  \afo \text{ and } \aPS,\aEv \models  \bfo \\
      \aPS,\aEv &\models& \TRUE\\
      \aPS,\aEv &\models& \lnot\afo &\text{ if } \aPS,\aEv \not\models \afo \\
      \aPS,\aEv &\models& \always\afo &\text{ if } \forall \bEv \lt \aEv.\; \aPS,\bEv \models \afo\\
      \aPS,\aEv &\models& \once\afo &\text{ if } \exists \bEv \lt \aEv.\;  \aPS,\bEv \models \afo
    \end{array}
  \end{displaymath}

  Define $\FALSE$, $\lor$, and $\Rightarrow$ as usual.

  % \begin{definition}
  Let $\aPS \models \afo$ if
  $\aPS,\aEv \models\afo$, for all $\aEv \in \Event$.

  Let $\aPSS\models \afo$
  if $\aPS \models\afo$, for all $\aPS \in \aPSS$.

Let
  \begin{math}
    \afo, \aPSS \models \bfo  \text{ if } \{ \aPS \mid \aPS \models \afo \} \parallel \aPSS \models \bfo.
  \end{math}
%\end{definition}

  Let $\afo$ be \emph{downclosed} when
  $\{ \aPS \mid \aPS \models \afo \}$ is.

% Thus, $\aPS\models \afo \land \always\afo$ whenever $\aPS \models
  % \afo$. This fact relies on the use of universal quantification in the definition.

% We define other connectives as standard:
% $\once\afo = \lnot(\always\lnot\afo)$,
% %$\FALSE = \lnot(\TRUE)$
% $\afo\lor\bfo = \lnot(\lnot(\afo)\land\lnot(\bfo))$, and
% $\afo\Rightarrow\bfo = \lnot(\afo) \lor\ \bfo$.
% \begin{displaymath}
% \begin{array}{lrl}
% \once\afo &=& \lnot(\always\lnot\afo) \\
% \FALSE &=& \lnot(\TRUE) \\
% \afo\lor\bfo &=& \lnot(\lnot(\afo)\land\lnot(\bfo)) \\
% \afo\Rightarrow\bfo &=& \lnot(\afo) \lor\ \bfo
% \end{array}
% \end{displaymath}
%Let $$ be defined as $$.
%In addition, let $\FALSE$, $\lor$ and $$ be defined in the
%standard way.
% $\afo\lor\bfo$ for $\lnot(\lnot \afo \land \lnot \bfo)$,
% and $\afo \Rightarrow \bfo$ for $\lnot \afo \lor \bfo$.
  The past operators do not include the current instant, and so
  do \emph{not} satisfy
  $(\always\afo\Rightarrow\once\afo)$. The order-minimal elements always validate
    $\always\afo$ and invalidate
    $\once\afo$.
  However, we can prove the following:
% \begin{align*}
%   \frac{\aPS \models \afo \Rightarrow\once{\afo}}{\aPS \models \lnot \afo}\text{(Coinduction)}
%   &&
%   \frac{\aPS \models \always\afo \Rightarrow\afo}{\aPS \models \afo}\text{(Induction)}
% \end{align*}
% \begin{lemma}
% Given an pomset $\aPS$.
\begin{align*}
  \tag{Induction}
  \aPS \models& (\always\afo \Rightarrow\afo) \Rightarrow\afo
  \\[-1ex]
  \tag{Coinduction}
  \aPS \models& (\afo \Rightarrow\once{\afo}) \Rightarrow\lnot \afo
  \\[-1ex]
  \tag{Weakening}
  \aPS \models& (\afo \Rightarrow\once{\bfo}) \Rightarrow (\once\afo \Rightarrow\once{\bfo})
\end{align*}
% \end{lemma}
% \begin{proof}
% We prove that any node in a pomset satisfies these formulas.
%The proof for both rules proceeds by induction on the length of the maximal path from a root to a node.
%\end{proof}

% \begin{description}
% \item[Coinduction.]
%   \begin{math}
%     (\afo \Rightarrow\once{\afo}) \Rightarrow\lnot \afo
%   \end{math}
% \item[Induction.]
%   \begin{math}
%     (\always\afo \Rightarrow\afo) \Rightarrow\afo
%   \end{math}
% \end{description}

%We now present two proof rules for programs.

%\paragraph*{Proof rules for programs}
We present two additional proof rules.
The first provides a logical view of \emph{$\aLoc$-closure} (Def.~\ref{def:rf}):
%The soundness proof is straightforward.
% \begin{math}
%   \closed(\aLoc) = (\DR{\aLoc}{\aVal} \Rightarrow \once \DW{\aLoc}{\aVal}).
% \end{math}
% Although this definition does not mention intervening writes, it is
% sufficient for our example.
\begin{displaymath}
  %\tag{Closing $\aLoc$}
  \frac{
    \afo \text{ is independent of } \aLoc
    \qquad
    %\aPS \models \closed(\aLoc) \Rightarrow \afo
    \aPS \models (\DR{\aLoc}{\aVal} \Rightarrow \once \DW{\aLoc}{\aVal}) \Rightarrow \afo
  }{
    \nu \aLoc \DOT \aPS \models \afo
  }
\end{displaymath}
%It is straightforward to establish that this rule is sound.
% Although it does
% not mention intervening writes, the rule is sufficient for our examples.

The second rule describes concurrent composition, in the style of~\citet{Abadi:1993:CS:151646.151649}.  To simplify the presentation, we
consider the special case with a single invariant.
% We view the
% composition result as capturing key aspects of no-ThinAirRead, as will become
% clearer in the examples below.
% In order to state the theorem, we generalize the satisfaction relation to
% include environment assumptions.

\begin{proposition}%[Composition]
  Let $\afo$ be downclosed.  Let $\aPSS_1, \aPSS_2$ be
  augmentation\hyp{}closed. %\footnote{$\aPS'$ is an augmentation of $\aPS$ if
 %   $\Event'=\Event$, $\aEv\le\bEv$ implies $\aEv\le'\bEv$, $\aEv\gtN\bEv$
 %   implies $\aEv\gtN'\bEv$, and
 %   % $\labeling'(\aEv)=\labeling(\aEv)$
 %   if $\labeling(\aEv) = (\bForm \mid \bAct)$ then
 %   $\labeling'(\aEv) = (\bForm' \mid \bAct)$ where $\bForm'$ implies
 %   $\bForm$.}
  Then:
  \begin{displaymath}
    %\tag{Composition}
    \frac{
      \afo, \aPSS_1 \models\afo
      \qquad
      \afo, \aPSS_2 \models\afo
    }{\aPSS_1 \parallel \aPSS_2 \models \afo}
  \end{displaymath}
\end{proposition}
\begin{proof}[Proof sketch]
  We will show that all downsets in the downset closures of
  $\aPSS_1 \parallel \aPSS_2$ satisfy the required property.  Proof proceeds
  by induction on downsets of $\aPS \in \aPSS_1 \parallel \aPSS_2$.
  %
  The case for empty downset  follows from assumption that  $\afo$ is downset closed.
  %
  For the inductive case, consider %$\aPS$ in the downset closure of $\aPSS_1 \parallel \aPSS_2$, i.e.
  $\aPS \in \aPS_1 \parallel \aPS_2$ where
  $\aPS_i \in \aPSS_i$.  Since $\aPSS_1$ and $\aPSS_2$ are augmentation
  closed, we can assume that the restriction of $\aPS$ to the events of
  $\aPS_i$ coincides with $\aPS_i$, for $i=1,2$.
  %
  Consider a downset $\aPS'$ derived by removing a maximal element $\aEv$ from
  $\aPS$.  Suppose $\aEv$ comes from $\aPS_1$ (the other case is
  symmetric). Since $\aPS_2$ is a downset of $\aPS'$ and $\aPS' \models \afo$
  by induction hypothesis, we deduce that $\aPS_2 \models \afo$.
  % Thus, $\aPS_2 \in \mods{(\afo)}$.
  Since $\aPS_1 \in \aPSS_1$, by assumption $\afo, \aPSS_1 \models\afo$ we
  deduce that $\aPS \models \afo$.
\end{proof}

% The logic is defined with respect to downclosed sets, but $\sem{\aCmd}$
% includes only completed pomsets.  For reasoning in the logic, we downclose
% the semantics, considering pomsets that may not be completed.  Let
% $\semdown{\aCmd}=\{\aPS'\mid\aPS'$ is a downset of some
% $\aPS \in \sem{\aCmd}\}$.

\begin{example}
\label{ex:thin}
\noparagraph{Basic Examples}
With all variables initialized to $0$, we show that \ref{OOTA3}
satisfies
\begin{math}
  \lnot\DW{x}{1}.
\end{math}

We start with the invariant:
\begin{displaymath}
  [\DW{x}{1}\Rightarrow\once\DR{y}{1}]
  \land
  [\DW{y}{1}\Rightarrow\once\DR{x}{1}]
\end{displaymath}
This invariant holds for each thread; thus, it holds for the
aggregate program by composition.  Closing $y$ yields
\begin{math}
  \DR{y}{1} \Rightarrow \once\DW{y}{1}.
\end{math}
Weakening the right conjunct: % yields
\begin{math}
  \once\DW{y}{1}\Rightarrow\once\DR{x}{1}.
\end{math}
Chaining these together: %yields
\begin{math}
  \DR{y}{1} \Rightarrow \once\DR{x}{1}.
\end{math}
Weakening:  %yields
\begin{math}
  \once\DR{y}{1} \allowbreak\Rightarrow \once\DR{x}{1}.
\end{math}
Chaining into the left conjunct:  %yields
\begin{math}
  \DW{x}{1} \Rightarrow \once\DR{x}{1}.
\end{math}
Closing $x$,
% \begin{math}
%   \DR{x}{1} \Rightarrow \once\DW{x}{1}.
% \end{math}
weakening,
% \begin{math}
%   \once\DR{x}{1} \Rightarrow \once\DW{x}{1}.
% \end{math}
then chaining: %, yields
\begin{math}
  \DW{x}{1} \Rightarrow \once\DW{x}{1}.
\end{math}
By coinduction,
\begin{math}
  \lnot\DW{x}{1}.
\end{math}
%as required.
\end{example}

% The same reasoning can be applied to the control flow variant of \ref{OOTA3}
% \cite[CYC]{DBLP:conf/popl/VafeiadisBCMN15}:
% \begin{math}
%   %\tag{\textsc{cyc}}\label{CYC}
%   \IF{x}\THEN \PW{y}{1} \FI \!\PAR\! \IF{y}\THEN \PW{x}{1} \FI.
%   % &&
%   % %\nonumber
%   % \hbox{\begin{tikzinline}[node distance=1.5em]
%   % \event{rx}{\DR{x}{1}}{}
%   % \event{wy}{\DW{y}{1}}{right=of rx}
%   % \po{rx}{wy}
%   % \event{ry}{\DR{y}{1}}{right=2em of wy}
%   % \event{wx}{\DW{x}{1}}{right=of ry}
%   % \po{ry}{wx}
%   % \rf{wy}{ry}
%   % \rf[out=170,in=10]{wx}{rx}
%   %   \end{tikzinline}}
% \end{math}
% The program is data-race-free. Thus, allowing an execution that writes $1$
% would violate \drfsc{}.

\begin{example}
  \label{ex:lochb}
  \noparagraph{Lochbihler's Example} %The essential temporal property of
  % \ref{OOTA1} is: \emph{A write of $1$ to $y$ must be preceded by a read of
  %   $1$ from $x$, and if $1$ is written to $z$ then a write of $1$ to $x$
  %   must be preceded by a read of $1$ from $y$.}
  % \emph{allocation at type $\classC$ is preceded by reading $0$ for
  % $b$}.
  Because our language lacks object creation, we cannot consider
  \citeauthor{DBLP:journals/toplas/Lochbihler13}'s example (\ref{OOTA1}) directly.  Instead we study \ref{OOTA4}, which has the same
  temporal structure.
  The essential temporal property of
  \ref{OOTA4} is: \emph{A write of $1$ to $y$ must be preceded by a read of
    $1$ from $x$, and if $1$ is written to $z$ then a write of $1$ to $x$
    must be preceded by a read of $1$ from $y$.}
  % : \emph{writing $1$ for $x$ is preceded by reading $0$ for
  % $b$}.
  We show an attempted execution that violates this invariant, eliding
  initialization:
% A more general principle, in the spirit of~\citet{Abadi:1993:CS:151646.151649} can be proved.  We chose the simple case of temporal invariants to illustrate the idea in a simple form.  Even this simple version has interesting consequences.
\begin{gather}
  %\tag{\textsc{oota4}}
  \label{OOTA4}\tag{\textsc{oota4}}
  %   Z=1;
  % ||
  %   a=X; // 1
  %   Y=a;
  % ||
  %   b=Z; // 0
  %   if(b){
  %     X=1
  %   } else {
  %     c=Y; // 1
  %     X=c;
  %     W=c;
  %   }
  %     \VAR  \PW{x}{0}\SEMI \VAR  \PW{y}{0}\SEMI \VAR  \PW{z}{0}\SEMI
  \begin{gathered}
    %\PW{y}{0}\SEMI
    \PW{y}{x}
  \PAR
  % \PW{x}{0}\SEMI
  % \PW{z}{0}\SEMI
  \PR{y}{r} \SEMI \IF{b}\THEN  \PW{x}{r} \SEMI \PW{z}{r} \ELSE \PW{x}{1} \FI
  \PAR
  %\PW{b}{0}\SEMI
    \PW{b}{1}
  \\[-1ex]
% \hbox{\begin{tikzinline}[node distance=1.25em]
%   \event{wy0}{\DW{y}{0}}{}
%   \event{rx}{\DR{x}{1}}{right=.5em of wy0}
%   \event{wy}{\DW{y}{1}}{right=of rx}
%   \po{rx}{wy}
%   \event{wx0}{\DW{x}{0}}{right=2em of wy}
%   \event{wz0}{\DW{z}{0}}{right=.5em of wx0}
%   \event{ry}{\DR{y}{1}}{right=of wz0}
%   \event{wx}{\DW{x}{1}}{right=of ry}
%   \event{wz}{\DW{z}{1}}{right=of wx}
%   \po{ry}{wx}
%   \event{rb}{\DR{b}{1}}{right=of wz}
%   \event{wb0}{\DW{b}{0}}{right=2em of rb}
%   \event{wb1}{\DW{b}{1}}{right=.8em of wb0}
%   \wk{wb0}{wb1}
%   \rf[out=15,in=165]{wy}{ry}
%   \rf[out=-170,in=-10]{wx}{rx}
%   \po[out=25,in=25]{ry}{wz}
%   \wk[out=25,in=155]{wy0}{wy}
%   \wk[out=19,in=161]{wx0}{wx}
%   \wk[out=19,in=161]{wz0}{wz}
%   \rf[out=-170,in=-17]{wb1}{rb}
%   \po{rb}{wz}
% \end{tikzinline}}
\hbox{\begin{tikzinline}[node distance=1.5em]
      \event{rx}{\DR{x}{1}}{}
      \event{wy}{\DW{y}{1}}{right=of rx}
      \po{rx}{wy}
      \event{ry}{\DR{y}{1}}{right=3em of wy}
      \event{wx}{\DW{x}{1}}{right=of ry}
      \event{wz}{\DW{z}{1}}{right=of wx}
      \event{rb}{\DR{b}{1}}{right=of wz}
      \event{wb1}{\DW{b}{1}}{right=3em of rb}
      \po{ry}{wx}
      \rf{wb1}{rb}
      \rf{wy}{ry}
      \rf[out=-170,in=-10]{wx}{rx}
      \po{rb}{wz}
      \po[out=15,in=165]{ry}{wz}
\end{tikzinline}}
  \end{gathered}
\end{gather}
As we discussed in \textsection\ref{sec:pop} there is a dependency from
$(\DR{y}{1})$ to $(\DW{x}{1})$; thus, the outcome is disallowed.  This
outcome is also disallowed by our event structures model
\citep[\textsection9]{DBLP:journals/lmcs/JeffreyR19}, although the logic
given in that paper is insufficient to establish this fact.  The outcome is
\emph{allowed} by \citet{Manson:2005:JMM:1047659.1040336}, \citet{DBLP:conf/esop/JagadeesanPR10},
\citet{DBLP:conf/popl/KangHLVD17}, and
\citet{DBLP:journals/pacmpl/ChakrabortyV19}.
% , demonstrating a lack of
% compositionality in these models.
% \begin{tikzdisplay}[node distance=1.5em]
%   \event{wy0}{\DW{y}{0}}{}
%   \event{rx}{\DR{x}{1}}{right=4.5em of wy0}
%   \event{wy}{\DW{y}{1}}{right=of rx}
%   \po{rx}{wy}
%   \wk[bend left]{wy0}{wy}
%   \event{wx0}{\DW{x}{0}}{below=of wy0}
%   \event{rz}{\DR{z}{0}}{right=of wx0}
%   \event{ry}{\DR{y}{1}}{right=of rz}
%   \event{wx}{\DW{x}{1}}{right=of ry}
%   \event{ry1}{\DR{y}{1}}{right=of wx}
%   \event{wa}{\DW{a}{1}}{right=of ry1}
%   \rf{wy}{ry1}
%   \po{ry}{wx}
%   \wk[bend right]{wx0}{wx}
%   \rf{wy}{ry}
%   \rf{wx}{rx}
%   \event{wz0}{\DW{z}{0}}{below=of wx0}
%   \event{wz1}{\DW{z}{1}}{right=of wz0}
%   \rf{wz0}{rz}
%   \wk{wz0}{wz1}
%   \po{ry1}{wa}
%   \po[bend right]{rz}{wa}
% \end{tikzdisplay}

To establish that this outcome is disallowed here, we prove
\begin{math}
  \lnot\DW{z}{1},
\end{math}
starting with invariant:
% which holds for each of the three threads, and thus, by composition, for the
% aggregate program:
\begin{align*}
  [\once\DW{y}{1} \Rightarrow \once\DR{x}{1}]
  \land
  [\notonce\DW{z}{1} \Rightarrow (\once\DR{y}{1} \land \always(\DW{x}{1} \Rightarrow \once\DR{y}{1}))]
\end{align*}
Closing $y$ and chaining into the left conjunct:
% \begin{math}
%   \once\DR{y}{1} \Rightarrow \once\DW{y}{1}. % \Rightarrow \once\DR{x}{1}
% \end{math}
% Chaining this implication on the left:
\begin{math}
  \once\DR{y}{1} \Rightarrow \once\DR{x}{1}.
\end{math}
% We can weaken this to:
% \begin{math}
%   \once\DR{y}{1} \Rightarrow \once\DR{x}{1}. % \Rightarrow \once\DR{x}{1}
% \end{math}
Chaining into the right conjunct:
\begin{displaymath}
  \notonce\DW{z}{1} \Rightarrow (\once\DR{x}{1} \land \always(\DW{x}{1} \Rightarrow \once\DR{x}{1}))
\end{displaymath}
Closing $x$:
% \begin{math}
%   \once\DR{x}{1} \Rightarrow \once\DW{x}{1}.
% \end{math}
%  Weakening and chaining again:
%we can replace $\once\DR{x}{1}$ with $\once\DW{x}{1}$:
\begin{math}
  \notonce\DW{z}{1} \Rightarrow (\once\DW{x}{1} \land \always(\DW{x}{1} \Rightarrow \once\DW{x}{1}).
\end{math}
Applying coinduction to the right conjunct:
\begin{displaymath}
  \notonce\DW{z}{1} \Rightarrow (\once\DW{x}{1} \land \always(\lnot \DW{x}{1}))
\end{displaymath}
Simplifying:
\begin{math}
  \notonce\DW{z}{1} \Rightarrow \FALSE,
\end{math}
as required.
\end{example}

\begin{comment}
  \color{red} Need to sort this out.
  Alan proposes:
\begin{verbatim}
     (W y 2) => <>(R x 1)
     (W y 1) => <>(R x 0)
     (W x 1) => <>(R y 1)
   <>(W x 1) => not(<>(W x 2))  --- which should be???  <>(W x 0) => not(<>(W x 1))
\end{verbatim}

2020/09/30: This seems to go bad because of initialization...
The formula
\begin{verbatim}
<>Wx0 => not(<>Wx1)
\end{verbatim}
does not hold for
\begin{verbatim}
x=0; x=y
\end{verbatim}

2020/09/10:  I am worried about the compositionality of this predicate:
\begin{verbatim}
I think
   <>(W x 0 => not(<>(W x 1)))
holds for
   x=0; r=y
and
   x=1
but not
   x=0; r=y || x=1
as shown by the execution
   Wx1 < Wx0 < Ry0
\end{verbatim}

It is impossible to fulfill $(\DR{y}{1})$ in the following
\cite[RNG]{DBLP:conf/esop/SvendsenPDLV18}:
\begin{align*}
  \taglabel{OOTA5}
    ( \PW{y}{x{+}1}
    \PAR
    \PW{x}{y} ) && \hbox{\begin{tikzinline}[node distance=1.5em]
        \event{rx}{\DR{x}{1}}{}
        \event{wy}{\DW{y}{2}}{right=of rx}
        \po{rx}{wy}
        \event{ry}{\DR{y}{1}}{right=3em of wy}
        \event{wx}{\DW{x}{1}}{right=of ry}
        \po{ry}{wx}
        \rf[out=170,in=10]{wx}{rx}
      \end{tikzinline}}
\end{align*}
The proof proceeds as before, starting with the following invariant:
\begin{gather*}
  [\DW{y}{2} \Rightarrow \once\DR{x}{1}] \land
  [\once\DW{x}{1} \Rightarrow \once\DR{y}{1}] \land
  [\once\DW{y}{1} \Rightarrow \once\DR{x}{0}] \land
  [\once\DW{x}{0} \Rightarrow \lnot(\once\DW{x}{1})]
\end{gather*}
\begin{verbatim}
  Wy2 => <>Rx1  /\  <>Wx1 => <>Ry1  /\  <>Wy1 => <>Rx0
close x and y
  Wy2 => <>Wx1  /\  <>Wx1 => <>Wy1  /\  <>Wy1 => <>Wx0
chain
  Wy1 => <>Wx0
chain with <>Wx0 => not(<>Wx1)
\end{verbatim}
\end{comment}

% Many examples are superficially similar, but in fact have fewer dependencies.
% A referee for a previous version of this paper expected that the following example is
% ``the same'':
% \begin{gather*}
%   \tag{OOTA?}\label{OOTA?}
%     \PW{y}{x}
%   \PAR
%     \IF{y}\THEN \PR{y}{r}\SEMI \PW{x}{r}\SEMI a\GETS r \ELSE \PW{x}{1} \FI
%   \\
%   \hbox{\begin{tikzinline}[node distance=1.5em]
%   \event{rx}{\DR{x}{1}}{}
%   \event{wy}{\DW{y}{1}}{right=of rx}
%   \po{rx}{wy}
%   \event{ry}{\DR{y}{1}}{right=2em of wy}
%   \event{wx}{\DW{x}{1}}{right=of ry}
%   \event{wa}{\DW{a}{1}}{right=of wx}
%   \rf[out=-15,in=-165]{wy}{ry}
%   \rf[out=170,in=10]{wx}{rx}
%   \po[out=-15,in=-165]{ry}{wa}
%     \end{tikzinline}}
% \end{gather*}
% In this execution, $\DW{x}{1}$ is independent of $\DR{y}{1}$, thus there is no
% \oota{} behavior.

Many examples are superficially similar, but in fact have fewer dependencies,
such as \eqref{OOTA?} from \textsection\ref{sec:intro}.

\noparagraph{RFUB: Register assignment From an Unexecuted Branch}
\citeauthor{BoehmOOTA}'s [\citeyear{BoehmOOTA}] \ref{RFUB} example presents
another potential form of \oota{} behavior.
% , in the context of compiler
% optimization.
Our analysis shows that there is no \oota{} behavior in
\ref{RFUB}, only a false dependency:
%\citet{BoehmOOTA} \labeltext{considers}{page:rfub} the following programs:
\begin{gather*}
  \tag{\textsc{rfub}}\label{RFUB}
  \sem{\PR{y}{r}\SEMI \PW{x}{r}}
  \not\supseteq
  \sem{\PR{y}{r}\SEMI \IF{r \NOTEQ 1} \THEN \PW{z}{1}\SEMI \LET{r}{1}\FI \SEMI \PW{x}{r}}
\end{gather*}
The left command is half of \ref{OOTA3}. %, from \textsection\ref{sec:logic}.
The right command is dubbed \rfub{}, for \emph{Register assignment From an
  Unexecuted Branch}.  \citeauthor{BoehmOOTA} observes that in the context
$\PW{x}{y} \PAR \hole{}$, these programs have different behaviors.  Yet the
\oota{} example on the left never writes $1$.  Why should the unexecuted
branch change that?  Because of the conditional, the write to $x$ in
\ref{RFUB} is independent of the read from $y$.  It useful to considering the
Hoare logic formulas satisfied by the two threads above: we have
$\hoare{\TRUE}{\ref{RFUB}}{x=1}$ for the right thread of \ref{RFUB}, but not
$\hoare{\TRUE}{\ref{OOTA3}}{x=1}$ for the right thread of \ref{OOTA3}.  The
change in the thread from \ref{OOTA3} to \ref{RFUB} is not a valid refinement
under Hoare logic; thus, it is expected that \ref{RFUB} may have additional
behaviors.

Understanding \oota{} behavior is notoriously difficult, even for the
greatest minds in the field!  % We believe that \emph{logic} is the only tool
% that can cut the horrible knot that semanticists have tied themselves in.
% Preconditions provide a \emph{natural} solution to working out these
% dependencies.
This example shows the wisdom of using existing tools, such as preconditions
and Hoare logic, to model new problems, such as relaxed memory.
% We don't
% need to abandon established ideas; we only need to adapt them!
% On page \pageref{page:rfub}, we discuss \citeauthor{BoehmOOTA}'s
% [\citeyear{BoehmOOTA}] \ref{RFUB} example, which presents another potential
% form of \oota{} behavior, in the context of compiler optimization.  Our
% analysis shows that there is no \oota{} behavior in \ref{RFUB}, instead
% \citeauthor{BoehmOOTA}'s analysis has a false dependency.

% Understanding \oota{} behavior is notoriously difficult, even for the
% greatest minds in the field!  We believe that \emph{logic} is the only tool
% that can cut the horrible knot that semanticists have tied themselves in.
% Preconditions provide a \emph{natural} solution to working out these
% dependencies.

% \endinput

% \paragraph{Load buffering and thin air.}
% The program
% \begin{math}
%   %\PW{x}{0}\SEMI \PW{y}{0}\SEMI
%   (\PW{y}{x} \PAR \bReg\GETS y\SEMI \PW{x}{1})
% \end{math}
% has top level executions that result in the final outcome $x = y = 1$, such as:
% \begin{tikzdisplay}[node distance=1.5em]
%   % \event{wx0}{\DW{x}{0}}{}
%   % \event{wy0}{\DW{y}{0}}{below=wx0}
%   \event{rx}{\DR{x}{1}}{}
%   \event{wy}{\DW{y}{1}}{right=of rx}
%   \po{rx}{wy}
%   \event{ry}{\DR{y}{1}}{right=3em of wy}
%   \event{wx}{\DW{x}{1}}{right=of ry}
%   \rf{wy}{ry}
%   \rf[out=170,in=10]{wx}{rx}
%   %\po{rx}{wy}
% \end{tikzdisplay}
% In \textsection\ref{sec:logic} we provide machinery to prove that this
% outcome is impossible if there is order from read to write in both
% threads.  This order can be achieved by replacing the second thread
% \begin{math}
%   (\bReg\GETS y\SEMI \PW{x}{1})
% \end{math}
% with
% \begin{math}
%   (\bReg\GETS y^\mRA\SEMI \PW{x}{1})
% \end{math}
% or
% \begin{math}
%   (\IF{y}\THEN \PW{x}{1}\FI)
% \end{math}
% or
% \begin{math}
%   (\PW{x}{y}).
% \end{math}

% A more interesting example is the following variant of \eqref{types}:
% \begin{displaymath}
%   %\label{OOTA4}
%   % \PW{x}{0}\SEMI
%   %\PW{y}{0}\SEMI
%   (
%     \PW{y}{x}
%   \PAR
%     \IF{z}\THEN \PW{x}{1} \ELSE \PW{x}{y}\SEMI a\GETS y \FI
%   \PAR
%     \PW{z}{0}\SEMI \PW{z}{1}
%   )
% \end{displaymath}
% This program is allowed to write $1$ to $a$ under many speculative
% memory models
% \cite{Manson:2005:JMM:1047659.1040336,DBLP:conf/esop/JagadeesanPR10,DBLP:conf/popl/KangHLVD17},
% even though the read of $1$ from $y$ in the else branch of the second
% thread arises out of thin air.   \citet{DBLP:journals/toplas/Lochbihler13}
% argues that such executions compromise type safety unless object allocation
% partitions memory by type.
% In our model, the attempted execution is:
% \begin{tikzdisplay}[node distance=1.5em]
%   \event{rx}{\DR{x}{1}}{}
%   \event{wy}{\DW{y}{1}}{below=of rx}
%   \po{rx}{wy}
%   \event{ry}{\DR{y}{1}}{right=of rx}
%   \event{wx}{\DW{x}{1}}{below=of ry}
%   \po{ry}{wx}
%   \rf{wy}{ry}
%   \rf{wx}{rx}
%   \event{rz}{\DR{z}{0}}{right=of ry}
%   \event{wz0}{\DW{z}{0}}{right=of rz}
%   \rf{wz0}{rz}
%   \event{wz1}{\DW{z}{1}}{right=of wz0}
%   \wk{wz0}{wz1}
%   \event{ry1}{\DR{y}{1}}{below=of rz}
%   \rf[bend right]{wy}{ry1}
%   \event{wa}{\DW{a}{1}}{right=of ry1}
%   \po{ry1}{wa}
%   \po{rz}{wa}
% \end{tikzdisplay}
% This is forbidden by the evident cycle.

% \begin{verbatim}

% y=x+1; a=y || x=y
% prove a!=2

% Wyv_1 /\ Wyv_2 => v_1 == v_2 (and maybe v_1==0 \/ v_2==0)
% Wx1 => <>-1 Ry1
% Wy1 => <>-1 Rx1
% \end{verbatim}

% Local Variables:
% mode: latex
% TeX-master: "paper"
% End: