\subsubsection{\stid{3.13} CLOVER Sub-project heFFTe}\label{subsubsect:fftecp}

\paragraph{Overview}

The {\it Highly Efficient FFTs for Exascale} ({\bf heFFTe}) project provides sustainable
high-performance multidimensional Fast Fourier Transforms (FFTs) for Exascale
platforms~\cite{thasd19,heffte-iccs20}.
HeFFTe leverages established but {\it ad hoc}
software tools that have traditionally been part of application
codes, but not extracted as independent, supported libraries.
%
The main objective of the heFFTe project is to:
%\begin{itemize}
%\item
1)~
      Collect existing FFT capabilities from ECP
      application teams;
%\item
2)~
      Assess gaps, extend, and make available various FFT
      capabilities as a sustainable math library;
%\item
3)~
      Explore opportunities to build multidimensional FFTs
      while leveraging on-node concurrency from
      batched FFT formulations;
%\item
4)~
      Focus on capabilities for Exascale platforms.
%\end{itemize}

FFTs are used in many applications including molecular dynamics,
spectrum estimation, fast convolution and correlation, signal
modulation and many wireless multimedia applications. The
distributed 3D FFT is one of the most important routines used
in molecular dynamics (MD) computations, and its performance can
affect MD scalability. The performance of the first
principles calculations strongly depends on the performance of the
FFT solver that performs many FFTs of size $\approx 10^7$ points in
a calculation that we call batched FFT. Moreover, Poisson PDE-type
equations arising from many engineering areas, such as plasma
simulation and density fields, need to solve FFTs of size larger than $10^9$.
%
More than a dozen ECP applications use FFT in their codes.
ECP applications that require FFT-based solvers suffer from the lack of
fast and scalable 3D FFT routines for distributed-heterogeneous parallel
systems as the ones projected for the upcoming exascale computing systems.
To address these needs, heFFTe functionalities are first delivered
to CoPA projects using LAMMPS (molecular dynamics) and HACC (Hardware Accelerated
Cosmology Code).

The heFFTe software stack is illustrated in the left-hand side of Figure~\ref{fig:fft-ecp-pipeline},
while the main components of the heFFTe framework are illustrated in the right-hand side of
Figure~\ref{fig:fft-ecp-pipeline}.
% The first and last step address the need
% for a flexible FFT API to take application-specific input and output (bricks/pencils),
% including arbitrary initial decompositions.
% Currently, heFFTe provides efficient
% GPU support for all communication primitives and features in FFTMPI and SWFFT.

\begin{figure}[htb]
    \centering
    \includegraphics[width=0.42\textwidth]{projects/2.3.3-MathLibs/2.3.3.13-CLOVER/heffte}~~
    \raisebox{.4\height}{\includegraphics[width=0.56\textwidth]{projects/2.3.3-MathLibs/2.3.3.13-CLOVER/ffttransormations}}
    \caption{\label{fig:fft-ecp-pipeline}
    {\bf Left}: the heFFTe software stack. {\bf Right}: 3D FFT computational pipeline in heFFTe with:~
      1) Flexible API for application-specific input and output,
         including bricks/pencils/etc.;~
      2) Efficient packing/unpacking and MPI communication
         routines;~
      3) Efficient 1D/2D/3D FFTs on the node.}
\end{figure}

\paragraph{Key  Challenges}
\begin{enumerate}
\item
\textbf{Communication costs:}
%Today's machines have very complex memory hierarchies and thus data movement,
%data layout translation, and communication should be the main focus of any
%distributed FFT library that aims to improve the performance of any ECP
%application that relies on FFT.
Communication costs are main bottleneck
on current systems; this includes low node bandwidth (relative to
high compute capabilities) and sub-optimal accelerator-aware MPI
communications that can lead to some stong scalability issues~\cite{heffte-pact21}.

\item
\textbf{Application specifics:}
ECP applications that require FFT-based solvers suffer from the lack of fast
and scalable FFTs for distributed-heterogeneous parallel systems
as the ones projected for the upcoming exascale computing systems. Also, ECP
applications need different application-specific versions of FFTs,
and dictate parallelism and data distributions (where is the data, how is
distributed, what is the parallelism, etc.). This requires application
knowledge and API designs with a suitable modular high-performance
implementation that is flexible and easy to use and integrate in ECP applications.

\item
\textbf{Performance portability:}
Performance portability across different architectures is always a challenge.
This is further exacerbated due to the many application and
hardware-specific FFT versions needed.
\end{enumerate}

\paragraph{Solution Strategy}

\begin{enumerate}
\item
\textbf{Communications and GPU optimizations:}
FFTs are communication bound and a main focus in heFFTe is on algorithmic
design to minimize communication and efficient GPU
implementations~\cite{sc19,eurompi19,heffte-pact21,hpec21}.
Other strategies include the use of mixed-precision calculations~\cite{Haidar2018,tcfft18}
and data compression for reduced communications (including lossy, e.g., using ZFP
compression)~\cite{Anztetal2020}.
\item
\textbf{Evolving design:}
heFFTe is designed to support the fftMPI and SWFFT functionalities,
which are already integrated in ECP applications. Thus, heFFTe benefits
directly these applications and provides integrated solutions.
More functionalities and application-specific optimizations will be added
through heFFTe backends to support various ECP applications.
\item
\textbf{Autotuning:}
Performance portability will be addressed through use of standards (like 1D FFTs
from vendors), portable linear algebra (LA) using MAGMA~\cite{Tomov_2010_pcsa},
and parameterized versions that will be tuned across architectures. We have extensive
expertise and well proven track record in the development and use of autotuning techniques
for important LA kernels~\cite{Nath2010,Kurzak2012gemmfermi}.
\end{enumerate}

\paragraph{Recent Progress}
The heFFTe team completed two main milestones involving software releases adding
numerous stability, performance, and scalability enhancements, as well as new
functionalities~\cite{heffte-pact21}.
HeFFTe 2.1 was released in April 2021, and heFFTe 2.2 was released in October
2021. HeFFTe 2.1 added support for multidimensional FFTs and optimizations for real data.
This included the development of R2C and C2R FFTs and their integration in heFFTe and
specific optimizations in ECP applications. Support and optimizations was extended for AMD
GPUs, dependence on MAGMA was added, as well as spack installation, and integration of heFFTe in xSDK.
HeFFTe was also integrated in CoPA projects and ExaAM/Meumapps with new application-specific
optimizations, tuning, and added Intel GPU support.
HeFFTe 2.2 concentrated on adding support and optimization of the HIP and Intel GPU backends to
heFFTe. HeFFTe's functional and performance portability on these architectures and
multicore CPUs was established~\cite{hpec21}, as well as ease of integration in the ECP applications
using heFFTe.
Multidimensional FFTs for
fast discrete convolution, cosine (DCT), and sine (DST) transforms were also added with support
for Nvidia, AMD,
and Intel GPUs. HeFFTe has demonstrate very good strong scalability and performance that is
close to 90\% of the roofline peak on the newly added in heFFTe v2.2 R2C, C2R, convolution,
DCS and DST transformations~\cite{heffte-pact21,heffte-iccs20} (see Figure~\ref{fig:fft-ecp-progress}).
An FFT benchmark for a number of FFT libraries, including a design study and evaluation of FFT
codes used in the ECP applications, was also developed~\cite{fftbenchmark}.

\begin{figure}[htb]
   \centering
%   \includegraphics[width=0.55\textwidth]{projects/2.3.3-MathLibs/2.3.3.13-CLOVER/heFFTeAcceleration}~~~
   \includegraphics[width=0.32\textwidth]{projects/2.3.3-MathLibs/2.3.3.13-CLOVER/heFFTeStrongScalability}
   \includegraphics[width=0.65\textwidth]{projects/2.3.3-MathLibs/2.3.3.13-CLOVER/heffte_conv_dst}
    \caption{\label{fig:fft-ecp-progress}
    %{\bf Left}: heFFTe acceleration of $1024^3$ FFT on 4 Summit nodes.
    %            Note: nodal computations are accelerated $43\times$.
    {\bf Left}: heFFTe strong scalability on $1024^3$
                FFT on up to 256 nodes ($\times 6$ V100 GPUs;
                double complex arithmetic; starting and ending with bricks;
                performance assumes $5 N^3 log_2 N^3$ flops).
    {\bf Middle}: Convolution of $512^3$ multidimensional arrays in double complex arithmetic.
    {\bf Right}: Performance of the Discrete Sine Transformation (DST) in heFFTe 2.2.
    }
\end{figure}

\paragraph{Next Steps}
Next steps of work are adding multidimensional batched FFTs and optimizations.
This will include support for AMD, Nvidia, and Intel GPUs.
Further integration and use will be added to CoPA applications and the ExaAM project.
Autotuning framework that hides backend selection and other parameters from users
will be added, as well as improved GPU-aware MPI Alltoallv routines, mixed-precision,
and approximate FFTs. Existing FFT libraries will be evaluated through the FFT
benchmark~\cite{fftbenchmark}.


\documentclass{article}
\usepackage{amsmath}
\usepackage{ulem}
\usepackage{asciimth}

\def\startTrace{\message{Starting trace...$}%
    \tracingmacros=2\tracingassigns=2\tracingcommands=2%
}
\def\endTrace{\message{Ending trace...$}\tracingmacros=0\tracingassigns=0\tracingcommands=0}

\def\textbs{\char92\relax} % backslash

% OK, note this doesn't really test spaces well.
\begingroup
\catcode32=13%
\gdef\changeSpaces{%
\catcode32=13%
\catcode126=12
\let =\ %
}
\endgroup
%HACK
\def\changeSpacesBack{\catcode32=10\catcode126=13}

\def\testRow{\hline\changeSpaces\testRowA}
\def\testRowA#1{%
    \changeSpacesBack
    \exampleRow{\sanitizeVerbose#1\end}{`#1`}%
}

\def\exampleRow#1#2{%
    {\tt\char18#1\char18}%
    %& #2\tabularnewline%
    & \mathcell{#2}\tabularnewline
}

% This is a little bit of a hack, but it was the easiest way I could
% figure out to get the math looking pretty inside of a table cell.
\def\mathcell#1{{
    \fboxrule=0pt
    \fbox{$\displaystyle #1$}%
    }%
}

% We can't just set the catcodes of ^ and _ because we want them as letters
% in the \tt but as super/subscripts in the `...`.
\def\sanitizeVerbose{\futurelet\nextTok\sanitizeVerboseA}
\def\sanitizeVerboseA{%
    \ifx\nextTok\bgroup\let\next=\sanitizeGroupAndRepeat
    \else\let\next=\sanitizeVerboseB
    \fi\next
}
\def\sanitizeGroupAndRepeat#1{%
    \{\sanitizeVerbose#1\end\}\sanitizeVerbose
}
\def\sanitizeVerboseB#1{%
    \ifx#1\end\let\next=\relax
    \else\ifx#1^\def\next{\char94\sanitizeVerbose}%
    \else\ifx#1_\def\next{\char95\sanitizeVerbose}%
    % If it's a macro, unpack the name.
    \else\ifcat\noexpand#1\def\def\next{\string#1\sanitizeVerbose}%
    \else\def\next{#1\sanitizeVerbose}%
    \fi\fi\fi\fi
    \next
}

\def\rawExampleRow{\hline\changeSpaces\readFirstPart}
\def\readFirstPart#1{\changeSpacesBack\exampleRow{#1}}

\newenvironment{examples}[1]
    {\renewcommand{\arraystretch}{1.5}
    \renewcommand{\tabcolsep}{0.2cm}
    \noindent\begin{tabular}{|p{6cm}|p{4cm}|}
        \multicolumn{2}{c}{\textbf{#1}}\\}
    {\hline\end{tabular}}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\title{The {\tt asciimth} package}
\author{Judah Jacobson}
\begin{document}
\maketitle

\section{Introduction}
The {\tt asciimth} package provides a more readable way to typeset
mathematics in \LaTeX{}.
We provide a few environments to replace those provided by \LaTeX{} and {\tt amsmath}:
\begin{itemize}
\item The {\tt asciimth} and {\tt asciimth*} environments act like {\tt equation} and {\tt
equation*}, except that the math inside uses the asciimth syntax.
\item The {\tt alignA} and {\tt alignA*} environments act like {\tt align} and {\tt align*}:
formatting is done by \verb|&| and \verb|\\|, but the math inside is written using {\tt
asciimth}'s syntax.
\end{itemize}
For example, we can use the following code to typeset a formula for the root of a cubic polynomial (via Wikipedia):
\begin{verbatim}
When `x^3+a x^2+b x+c`, one of the roots is
\begin{asciimth*}
-1/3( a +root3( (m + sqrt(m^2-4k^3) ) / 2 )
        +root3( (m - sqrt(m^2-4k^3) ) / 2 ))
\end{asciimth*}
where:
\begin{alignA*}
m &= 2a^3-9a,\\
k &= a^2-3b.
\end{alignA*}
\end{verbatim}

It renders as:

When `x^3+a x^2+b x+c`, one of the roots is
\begin{asciimth*}
-1/3( a +root3( (m + sqrt(m^2-4k^3) ) / 2 )
        +root3( (m - sqrt(m^2-4k^3) ) / 2 ))
\end{asciimth*}
where:
\begin{alignA*}
m &= 2a^3-9a,\\
k &= a^2-3b.
\end{alignA*}

Additionally,
any text surrounded by
left-quotes (\char18\ldots\char18) will be parsed by the {\tt asciimth}
engine.  For example, typing {\tt\char18 (x-1)\char94(2x)/(4-3)\char18} in
the middle of a paragraph produces `(x-1)^(2x)/(4-3)`.

That notation can be mixed with standard \TeX{} math commands by
inserting a left-quoted expression in the middle of an existing expression or
equation.  For example, \verb|$\frac{1}{2}-`3/4`$| is typeset as
$\frac{1}{2}-`3/4`$.  It can also be embedded into display math
(\verb|$$| or \verb|\[|) or a math environment such as {\tt multiline} or
\texttt{align}.

For compatibility with the common use of left-quotes, the empty expression
{\tt\char18\char18} is typeset as the begin-quotes symbol (``).
However, if you really need them, we provide the macros {\tt \textbs makeQuoteOther}
and {\tt\textbs makeQuoteActive} which turn that special behavior off and on, respectively.

This document displays a range of example equations as typeset by {\tt
asciimth}.  It serves both as documentation and as a comprehensive
test of the package.

\begin{section}{Tables}

\begin{examples}{Simple commands}
\testRow{9+alpha}
\testRow{gamma>2 implies gamma*gamma>4}
\testRow{x ge 0 implies x x ge 0}
\testRow{t}
\testRow{1-a}
\testRow{1 234}
\testRow{a b - a\,b}
\testRow{ (9)}
\testRow{9+(4-(alpha))}
\testRow{(-)}
\testRow{[2,3)}
\testRow{"Im"(f) " such that " f>0}
\rawExampleRow{ }{` `}
\rawExampleRow{}{``}
\end{examples}

\begin{examples}{Fractions and parentheses}
\testRow{9/2}
\testRow{(1//2]/(7//4)}
\testRow{(9)/(7) + ((8))/(((15)))}
\testRow{ 24/3 + ((alpha+2)/2 * 5)/7}
\testRow{2/3/3}
\testRow{-5/zeta}
\testRow{z/((q*(2/beta to gamma)))}
\testRow{([2/3])/7}
\testRow{Phi = 1+1/(1+1/(1+cdots))}
\testRow{left< 2/3 right|}
\testRow{(7 middle| 2/3 right.}
\end{examples}

\begin{examples}{Fonts}
\testRow{hat(a) + tilde(b) + bar(c) + widehat(a b c)}
\testRow{cal(T) [bf(x)] in bb(R)^n}
\end{examples}

\begin{examples}{Exponents and subscripts}
\testRow{5^14}
\testRow{5_14}
\testRow{(9^-74)}
\testRow{ 125^-74}
\testRow{1/125^74}
\testRow{(1/125)^74}
\testRow{alpha^(2+3)/5}
\testRow{x^n y_1^-t alpha^-(beta-z)}
\testRow{(q r s)_(123)^-[456/z]}
\testRow{a_1^2}
\testRow{(x)_-i + q^r_(s)}
\testRow{(a+b)_(k_1)^(x+2)}
\testRow{sum_(n=1)^infty 1/n^2}
\testRow{int_pi^infty 1/(sin x)^2\,d x}
\testRow{lim_substack(x to 0\\ y to 1) x^2(y-1)^2}
\end{examples}

\begin{examples}{Square roots}
\testRow{sqrt 2}
\testRow{sqrt(2+4/5)}
\testRow{2/sqrt(3alpha)}
\testRow{sqrt sqrt 3}
\testRow{(2+3)/sqrt sqrt sqrt 3}
\testRow{root n (x^2+1)}
\testRow{(-b pm sqrt(b^2-4a c))/(2a)}
\testRow{sqrt x^3}
\end{examples}

\begin{examples}{Symbols}
\testRow{a_1...a_n}
\testRow{a_1,...,a_n}
\testRow{f:bb(R)^2->bb(R)}
\testRow{>= <= => != -= ~ ~= ~~ ~-}
\end{examples}

\begin{examples}{Matrices}
\testRow{matrix[y]}
\testRow{matrix(a,1,c)}
\testRow{matrix(3a;b^2;c-5d)}
\testRow{matrix[2, sqrt(x^3), x^3/(7-x); (d),e,f]}
\testRow{matrix[x;y; z right.}
\testRow{matrix[A_(1,1), ... ,A_(1,n); vdots,ddots,vdots; A_(m,1), ... ,A_(m,n)]}
\end{examples}

\begin{examples}{Embedded LaTeX environments}
\rawExampleRow
{x/env\{pmatrix\}(2 \& 3 \textbs\textbs 4 \& 5)}
{`x / env{pmatrix}(2&3\\4&5)`}
\rawExampleRow
{env\{cases\}(sqrt(x) \& " if " x >= 0\textbs\textbs "undefined" \& " otherwise.")}
{`env{cases}(sqrt(x) & " if " x >= 0\\ 0 & " otherwise.")`}
\end{examples}

\par
\par
The {\tt raw} command lets you embed raw TeX commands inside of an asciimth environment.
Unlike for other commands, curly braces must be used for {\tt raw}.

As a shortcut, you can also type \texttt{label\{equationName\}} inside of an {\tt asciimth} or {\tt alignA} environment.  It will have the same effect as typing\\ {\tt raw\{\char92label\{equationName\}\}}

\begin{examples}{Braces and raw commands}
\testRow{2^(3+y) + raw{\frac{5}{6x}}}
\testRow{2^raw{\frac78}}
\rawExampleRow
{\relax2 + raw\{\textbs{}begin\{pmatrix\}2\textbs\textbs3
\textbs{}end\{pmatrix\}\}}
{`2 / raw{\begin{pmatrix}2\\3\end{pmatrix}}`}
\testRow{t*{2+alpha}^2}
\testRow{t*{(2+alpha)}^2}
\testRow{ {2+x}/{gamma^2beta}}
\testRow{ \{2/(x+y)\}^3}
\end{examples}

\end{section}

\begin{section}{Layout tests}

These examples cover a few issues which are caused by the fact that we always
use {\tt\string\left} and {\tt\string\right} delimiters:
\begin{itemize}
\item Using {\tt\string\left(\ldots\string\right)} causes different
spacing than
{\tt(\ldots)}.  That issue has been resolved.
\item The parentheses around the inner `sum` in the last example
are too large; they unnecessarily grow
to cover the subscript `j`.
\end{itemize}

\begin{examples}{Parenthesis tests}
\testRow{f(x)+g(x/y)}
\testRow{(x+y)(x-y)}
\testRow{f(x^2)+f(2^(2/y))}
\testRow{(x^2)^2}
\testRow{sum(x_i)+sum(x/y).}
\testRow{sum_i(sum_j(2*x))}
\end{examples}

\end{section}

\end{document}


\documentclass[10pt,a4paper]{article}
\usepackage{todonotes}
\usepackage{amsmath}
\usepackage{booktabs}                    % Support for better table layout
\usepackage{tabularx}

\title{Formal description of the generated ILP}

\begin{document}

\section{Problem description}

Given the following:

\begin{itemize}
	\item A set of properties $P = P_{software} \cup P_{hardware}$
	\item A set of software components $C$
	\item For each software component $c \in C$
	\begin{itemize}
		\item a set of provided properties $P(c) \subseteq P_{software}$
		\item a set of implementations $I(c)$
	\end{itemize}
	\item For each implementation $i \in I(c)$ of component $c$
	\begin{itemize}
		\item a set of required software components $Req(i) \subseteq C \setminus i$
		\item a set of modes $M(i)$
	\end{itemize}
	\item A set of resources (containers) $R$
	\item For each resource $r \in R$
	\begin{itemize}
		\item a set of provided properties $P(r) \subseteq P_{hardware}$
	\end{itemize}
\end{itemize}

The total number of modes is therefore
\begin{equation}
	\sum_{c}^{C} \sum_{i}^{I(c)} |M(i)|
\end{equation}

Given a mode $m$ of implementation $i$ and component $c$, there is a contract restricting the deployment of $m$, containing any number of the following items

\begin{itemize}
	\item Constraints $CS(c_{req},m) = (p, \circ, val)$ for a required software component $c_{req} \in Req(i)$
	\begin{equation}
	p \circ val, \text{where } p \in P(c_{req}), \circ \in \{<, \leq, >, \geq \}
	\end{equation}
	\item Constraints $CR(m) = (p, \circ, val)$ for a resource
	\begin{equation}
	p \circ val, \text{where } p \in P_{hardware}, \circ \in \{<, \leq, >, \geq \}
	\end{equation}
	\item Provisions $Prov_m = (p, \circ, val)$
	\begin{equation}
	p \overset{!}{\circ} val, \text{where } p \in P(c), \circ \in \{<, \leq, >, \geq, = \}
	\end{equation}
\end{itemize}

For each element $E$ of a contract, there are functions to ``access'' their sub-elements:
\begin{equation}
	E = (p, \circ, val) \Leftrightarrow p(E) = p, sense(E) = \circ, val(E) = val
\end{equation}

The optimization problem is to find a mapping from one mode of each software component\footnote{which is needed to serve the request} to one resource such that all constraints of the contracts hold.

\section{ILP description}

There are two kinds of used variables, $x_{m,r}$ and $y_{i}$, in the ILP. To describe the constraints, some auxiliary functions are needed:
\begin{itemize}
	\item $x_{m,r}$ \dots Deployment of mode $m$ on resource $r$ (binary variable)
	\item $y_{i}$ \dots Deployment of implementation $i$ (binary variable)
	\item $reqModes(p)$ \dots Every mode having a constraint on the property~$p$
	\item $reqModes(p,c) = reqModes(p) \cap \left( \bigcup\limits_i^{I(c)} M(i) \right)$ \dots Every mode of component~$c$ having a constraint on the property $p$
	\item $Req(c)$ \dots Every component required by implementations of component $c$
	\item
	$
	val(p,x) \in
	\begin{cases}
	P_{Software} \times M & \text{value of property }p\text{ required in mode }x \\
	P_{Hardware} \times R & \text{value of property }p\text{ provided by resource }x \\
	\end{cases}
	$
	\item $req_{all}(c) = \bigcup_i^{I(c)} Req(i) $
	\item $req_{map}(c) = \bigcup_c^{req_{all}(c)} \langle c, I^{-1}(c)  \rangle $
\end{itemize}

\subsection{The constraints}

Implementation $i$ is deployed $\Leftrightarrow y_i \equiv 1$, also ensures, that only one mode is deployed per component
\begin{equation}
	\forall c \in C, \forall i \in I(c): y_i = \sum_{r}^{R} \sum_{m}^{M(i)} x_{m,r} \label{eq:impl-shortcut}
\end{equation}
Deploy required components:
\begin{equation}
	\forall c \in C, \langle c_{req}, I' \rangle = req_{map}(c):
	\sum_{i_{req}}^{I(c_{req})} y_{i_{req}} \geq \sum_{i}^{I'} y_i
	\label{eq:req-comp}
\end{equation}
Software property requirement clauses:
\begin{multline}
	\forall c \in C, \forall p \in P_{software}: \\
	\sum_{r_1}^{R} \sum_{c_{req}}^{Req(c)} \sum_{i_{req}}^{I(c_{req})} \sum_m^{M(i_{req})} x_{m_{req},r_1} * val (p, m_{req}) \label{eq:sw-req} \circ \sum_{r_2}^{R} \sum_{m}^{reqModes(p,c)} x_{m,r_2}
\end{multline}
Hardware property requirement clauses:
\begin{equation}
	\forall r \in R, \forall p \in P(r): \sum_{m}^{reqModes(p)} x_{m,r} * val(p,m) \circ val(p,r) \label{eq:hw-req}
\end{equation}

The constraint sense $\circ$ in equations \eqref{eq:sw-req} and \eqref{eq:hw-req} are connected to the contract constraints and are read from there. Actually one constraint is created for each unique pair of property and constraint sense. However, putting that into a formula would unnecessarily complicate it.

\subsection{The objective function}

Let $p_{goal} \in P$ be the property to optimize for.

\begin{equation}
	objective = \sum_{r}^{R} \sum_{m}^{reqModes(p_{goal})} x_{m,r} * val(p_{goal},m)
\end{equation}

\section{Metrics for the generated ILP}

Number of variables $n$:
\begin{equation}
n = |R| * \sum_{c}^{C} \sum_{i}^{I(c)} 1 + |M(i)|
\end{equation}
%
Number of constraints $m$ (at most one required component, some hardware- and $S$~software clauses per mode on average):
\begin{equation}
m = |C| + %
S * \sum_{c}^{C} \sum_{i}^{I(c)} 1 + |M(i)| + %
\sum_r^{R} * P(r)
\end{equation}

\subsection{Sample values}

Assuming $S = 2$ and $\overline{P(r)} = 2$.

\begin{table}[th]
	\aboverulesep0.0mm
	\belowrulesep0.5mm
	\centering
%	\scriptsize
	\newcolumntype{R}{>{\raggedleft\arraybackslash}X}
	\begin{tabularx}{.9\linewidth}{ r R R R | R R }
		$|C|$ & $\overline{|I|}$ & $\overline{|M|}$ & $|R|$ & $n$ & $m$ \\
		\midrule
		10           & 10           & 2           & 100           &  20000 &  810 \\
		\textbf{100} & 10           & 2           & 100           & 200000 & 6300 \\
		10           & \textbf{100} & 2           & 100           & 200000 & 6210 \\
		10           & 10           & \textbf{20} & 100           & 200000 & 4410 \\
		10           & 10           & 2           & \textbf{1000} & 200000 & 2610 \\[1mm]
	\end{tabularx}
	\caption{Number of variables $n$ and constraints $m$ for some example values}
	\label{tab:saved-energy}
\end{table}

\end{document}


\section{Introduction}
\label{sec:intro}

One fundamental task in \emph{Bayesian networks} (BNs) \cite{pear88} is inference.
Given that some variables have been observed, the task is to compute posterior probabilities of other variables.
\cite{koll09} introduce readers to exact inference in discrete BNs with the \emph{Variable Elimination} (VE) \cite{zhan94} algorithm.
The VE algorithm uses its own terminology such as elimination orderings to modify the \emph{conditional probability tables} (CPTs) of the BN to answer queries.

Another fundamental task in BNs is modeling, by which we mean testing which conditional independence relations hold in a given BN.
More specifically, we want to know whether two sets $X$ and $Z$ of variables are conditionally independent given a third set $Y$ of variables.
\cite{pearl86,pear88} introduced the \emph{directed separation} (d-separation) algorithm for this task.
d-Separation uses its own specialized terminology such as closed convergent valves in the \emph{directed acyclic graph} (DAG) to determine whether or not an independent holds.

In this paper, we aim to establish computation that is common to both inference and modeling.
We organize the common computation as an algorithm, called \emph{Simple Propagation} (SP).
SP takes the factorization of the BN CPTs and two sets $X$ and $Y$ of variables in the BN.
SP modifies the factorization by removing all variables relevant to $X$ and $Y$.
The output of SP can now be used for both inference and modeling.
Surprisingly, one salient feature of SP is that it performs the bulk of the work leaving only a few steps to be executed for inference and for modeling.
Another advantage of SP is that it brings unified terminology.
Thereby, the work here provides a deeper understanding of BNs.

This paper is organized as follows.
In Section \ref{sec:back}, background is given.
Unifying inference and modeling is done in Section \ref{sec:new}
Section \ref{sec:adv} draws advantages of the new method.
Conclusions are shown in Section \ref{sec:conc}.

B - The Platform/Paper/sections/introduction.tex


As explained in previous sections, the Ginan repository consist on two main components. The \textit{precise orbit deterination} (POD) component estimates precise satellite position and orbital parameters, while the \textit{parameter estimation algorithm} (PEA) monitor systematic biases asociated with GNSS signals. The Ginan software follows uses the \textit{precise point positioning}(PPP) philosophy for processing of GNSS signals. PPP was originally developed as a GNSS based positioning method for calculating location of autonomous receivers with high levels of accuracy and precision. PPP aims to calculate the end user position by rigorously modelling and/or estimating error sources in GNSS measurements.
The systematic of errors in GNSS signals can be summarised as:
\begin{itemize}
	\item Satellite state estimation errors: position, clock offset, hardware biases, antenna effects
	\item Receiver state estimation errors: position, clock offset, hardware biases, antenna effects
	\item Atmospheric effects: ionospheric propagation delay, tropospheric propagation delay
	\item Other (modellable) enviromental effects: Relativistic corrections, phase windup
\end{itemize}

The various components of the Ginan software package are designed to model or estimate these errors as parameters. A diagram illustrating of the way the Ginan components interact to estimate these parameters can be found in figure \ref{fig:PEAnPOD}. In the example illustrated by the figure:
\begin{enumerate}
	\item  The POD in orbit fitting mode is used to calculate an a-priori position and the linearization partials of orbit parameters
	\item  The PEA, in network mode, estimate orbital parameters from orbit partials
	\item  The POD, use the orbital parameters to estimate and predict precise satellite positions
	\item  The PEA, in network mode, is used to estimate wide-area parameters: satellite clock offsets, satellite hardware bias and atmospheric delays
	\item  The PEA, in end-user mode, is used to callculate local parameters like receiver position, receiver clock offset and local atmospheric delays
\end{enumerate}

Other parameters, such as antenna, phase windup and relativistic effects are calculated from predefined models.

A description of each components and its use is presented below.\\

\chapter{Using the POD Module}
GINAN applications use YAML format to define configuration files. After installing the dependencies and compiling building the POD application, the POD processing can be started by typing the command.
\begin{lstlisting}
$ ./pod -y <path_to_config_file>
\end{lstlisting}
Details on the configuration parameters included in YAML files can be found in chapter \ref{ch:pod_yaml_configuration}. Configuration files corresponding to the examples in this section can be found in the \textit{ginan/examples} directory.\\

The POD module has two main modes of operation, the orbit fitting mode and the orbit integration/prediction mode. In orbit fitting mode, precise orbit parameters are calculated from, potentially inaccurate, satellite position pseudo-observations. In orbit integration mode, precise satellite positions are estimated/predicted from precise orbit parameters.\\

\section{Using the POD for orbit fitting}
The orbit fitting mode can be selected by setting the \textit{ pod\_mode\_fit} to true and \textit{ic\_input\_format : sp3} to true. In this mode, the POD will take satellite position pseudo-measurments from a SP3 formatted file and estimate the orbit state of each satellite contained in the SP3 file. The SP3 file containing a priory satellite position needs to be specified as the \textit{pseudobs\_orbit\_filename} parameter.
The orbit state in POD is represented by a set of parameters consisting of
\begin{itemize}
	\item Satellite position (in ITRF or TCRF) at the first epoch in the SP3 file
	\item Satellite velocity (in ITRF or TCRF) at the first epoch in the SP3 file
	\item Up to 9 parameters describing the Solar Radiation Pressure over the fitting time
\end{itemize}
These initial conditions, and the models described in chapter \ref{ch:observation_modelling} will allow for the precise determination of satellite positions over the fitting arch (set by the \textit{orbit\_arc\_determination} parameter).\\

The main outputs from this mode of operation are the a-posteriori satellite position in SP3 format, and the orbit partials of satellite positions with respect to the initial conditions.
The ouput SP3 file which can be found on \textit{output\_directory/gagWWWWD.sp3} where \textit{WWWW} is the GPS week and /textit{D} is the GPS day of the first epoc on the SP3 files.
The orbit partials are written in Ginan's proprietary Initial Conditions File (ICF) format, and can be found in  \textit{output\_directory/gagWWWWD\_orbit\_partials.out}.
Configuration files, \textit{ex21\_pod\_fit\_gps} and \textit{ex21\_pod\_fit\_gnss},  for this mode of operation are included in Ginans \textit{examples} folder.\\

\section{using the POD for orbit integration/prediction}
The orbit fitting mode can be selected by setting the \textit{ pod\_mode\_ic\_int} to true and \textit{ic\_input\_format : icf} to true.
 In this mode, the POD will take the initial conditions contained in the ICF formatted files and propagaes the satellite positions forward over the time period specified by the sum of \textit{orbit\_arc\_determination} and \textit{orbit\_arc\_prediction} parameters.
 It also propagates the satellite position backwards by a number of hours specified by the \textit{orbit\_arc\_backwards} parameter.
 The ICF file containing the satellites initial condition and radiation pressure parametes needs to be specified as the \textit{ic\_input\_format : ic\_filename} parameter.\\

It is to note that the orbit fitting mode will also use the orbit integration operation after estimating the initial conditions from pseudo-observations.
Although the  mode \textit{ pod\_mode\_fit} will only integrated for a number of hours specified  by \textit{orbit\_arc\_determination}.
Selecting the \textit{pod\_mode\_predict} will propagate the initial conditions a number of hours specified by the sum of \textit{orbit\_arc\_determination} and \textit{orbit\_arc\_prediction}
The integrated/predicted satellite position will be output to a SP3 formatted file located in \textit{output\_directory/gagWWWWD.sp3}.\\

The configuration file to perform orbit integration/prediction from SP3 files is \textit{ex23\_pod\_prd\_gps}.
The configuration file to perform orbit integration/prediction from ICF files is \textit{ex24\_pod\_ic\_gps}. Both located in Ginans \textit{examples} folder.\\


\documentclass[12pt, letterpaper]{article}
\include{eu}
\pagestyle{plain}
\begin{document}
\section*{\textsl{Einstein's Universe} Final Exam}
\setcounter{problem}{1}

\begin{problem}
  (From Problem Set 1)
  What, approximately, is the cargo volume of an armored truck?
\end{problem}

\begin{problem}
  (From Problem Set 1)
  What is the mass of one mole of water?
\end{problem}

\begin{problem}
  (From Term Exam 1)
  Classical mechanics, or Newtonian mechanics, is only valid in certain
circumstances. When do the laws of classical mechanics, like $\vec{F} =
m\,\vec{a}$ for example, become wrong or break down? There are many answers
to this problem; I will take anything correct.
\end{problem}

\begin{problem}
  (From Term Exam 1)
  Which of the following physical quantities are vectors?
\\
\textsl{(a)}~energy,
\textsl{(b)}~mass,
\textsl{(c)}~force,
\textsl{(d)}~momentum,
\textsl{(e)}~acceleration.
\end{problem}

\begin{problem}
  (From Problem Set 2)
  What is the mass $M$ you found for a piano string?
\end{problem}

\begin{problem}
  (From Problem Set 2)
  What, approximately, is the volume of an ice molecule?
\end{problem}

\begin{problem}
  (From Term Exam 2)
  This wave on a string is moving to the left. The string is moving only
up and down. State, for each of the three points A, B, and C, which way those
bits of string are moving.\\
\includegraphics{wavepulse.png}
\end{problem}

\begin{problem}
  (From Term Exam 2)
  A $50\,\g$ mass will have roughly what weight? Give your answer in force units.
\end{problem}

\begin{problem}
  (From Problem Set 3)
  What is the Lorentz factor $\gamma$ for something moving relative
  to you at speed $v = 0.99\,c$?
\end{problem}

\begin{problem}
  (From Problem Set 3)
  What is the distance from us to the Galactic Center?
\end{problem}

\begin{problem}
  (From Term Exam 3)
  Give the lengths of the three sides of the triangle that you drew for the
  moving-light-clock problem (Problem Set 3, problem 3).
\end{problem}

\begin{problem}
  (From Term Exam 3)
  Muons live for a couple of milliseconds. Naively, therefore, they can't travel more than few hundred meters,
even traveling near the speed of light. And yet, they often are observed to travel many kilometers.
How is this possible?
\end{problem}

\begin{problem}
  (From Problem Set 4)
  Which produces more energy per unit mass?
  Burning fossil fuels,
  nuclear fission with uranium fuel, or
  nuclear fusion with hydrogen fuel?
\end{problem}

\begin{problem}
  (From Problem Set 4)
  An elevator in New York City is accelerating downwards
  at acceleration $g$ (the gravitational acceleration).
  What is the magnitude of the normal force on a box of mass $M$ inside this elevator?
\end{problem}

\begin{problem}
  (From Term Exam 4)
  In normal space, the straight line (or geodesic) is the path of \emph{shortest
total distance} between two points. In spacetime, the geodesic is the path
of what?
\end{problem}

\begin{problem}
  (From Term Exam 4)
  If you are in a rocket that is moving in the $x$-direction with
respect to the Earth at $0.75\,c$ and, inside the rocket, you are
moving at $0.75\,c$ in the $x$ direction with respect to the rocket,
how fast are you moving with respect to the Earth? No need to
calculate. All I want to know is: Are you moving closer to $0.75\,c$,
$0.95\,c$, or $1.5\,c$?
\end{problem}

\begin{problem}
  (From Problem Set 5)
  What, roughly, is the insolation on the surface of Pluto?
\end{problem}

\begin{problem}
  (From Problem Set 5)
  What is the recession speed (relative to us) of a galaxy that has a Doppler
  factor of 10?
\end{problem}

\begin{problem}
  (From Term Exam 5)
  If a galaxy is moving away at 5 percent of the speed of light, the K
line (which is at a rest-frame or natural wavelength of around $\lambda =
3900$\,\AA), will be shifted to the red. What, roughly, will be the change
$\Delta\lambda$ in the wavelength of the line?
\end{problem}

\begin{problem}
  (From Term Exam 5)
  What was the policy introduced at the University in Berlin in 1933 that caused Einstein's job to be terminated?
\end{problem}

\begin{problem}
  (From Problem Set 6)
  What is the mass of the black hole at the center of the Milky Way?
\end{problem}

\begin{problem}
  (From Problem Set 6)
  How much energy was radiated away in gravitational radiation in event GW150914?
  You can give your answer in energy units, or mass units (since, after all, there
  is mass--energy equivalence).
\end{problem}

\begin{problem}
  State one thing that you learned in this class
  that you expect to remember for many years.
\end{problem}

\begin{problem}
  Einstein brought two changes to physics: relativity (special and general) and
  quantization of energy and mass. Which do you think is more important, and why?
\end{problem}

\end{document}


\chapter{dataReset}

\section{Introduction}

A linearized process may have parameters that are initialized, changed and used, and subsequently ignored until they are used again.
In the interval from the last change or use of a parameter to its subsequent reuse, a parameter could have different values.
These values contribute to the size of the state space of the process \emph{without} adding any new behavior!

It may be advantageous to detect from which moment the value of a parameter is no longer used and set it to a default value instead.
This is called `resetting'.
Analysis of an LPE can yield information on when parameters can be safely reset.

\section{Algorithm}

The algorithm is based on earlier work \cite{van2009state}.

As a preparative step, several determinations are made about each parameter $p$ of the LPE per summand $s$:
\begin{itemize}

\item Is the value of $p$ potentially altered by $s$?
If so, $p$ is marked as `changed'.

\item Does $p$ occur in the guard of $s$?
If so, $p$ is marked as `directly used'.

\item Is $p$ marked as `directly used' in $s$?
Or does $p$ occur in the assignment by $s$ to a parameter that is marked as `changed'?
In either case, $p$ is marked as `used'.

\item Must the value of $p$ have a specific (unique) value $v$ in order for $s$ to be enabled?
If so, $p$ is marked as `having a source' for $s$, with the source being $v$.

\item Does the value of $p$ have a specific (unique) value $v$ immediately after $s$ has been applied?
If so, $p$ is marked as `having a destination' for $s$, with the source being $v$.

\item Does $p$ have both a source and a destination for $s$?
Then $p$ is called a \emph{ruling parameter}.

\end{itemize}

\subsection{Control flow graphs}

A parameter of the LPE is a \emph{control flow parameter} of the LPE if for all summands it is either a ruling parameter or not marked as `changed' (or both).
A parameter that is not a control flow parameter is defined as a \emph{data parameter}.

\vspace{1mm}

For each control flow parameter $f$ of the LPE, a control flow graph is constructed.
There is a state in the control flow graph for each source and destination of $f$ (across all summands).
Two states $s_1$ and $s_2$ are connected by an edge $(s_1, i, s_2)$ if there is a summand $i$ where the source of $f$ is represented by one of the states and where the destination of $f$ is represented by the other state.
The direction of such an edge is from source state to destination state.

\subsection{Belongs-to function}

Next, we define the \emph{belongs-to} function.
The belongs-to function maps each data parameter $d$ to some set of control flow parameters $b(d)$ as follows

\begin{align*}
b(d) = F \cap \bigcap\limits_{s \in S}^{} \text{ruling}(s)
\end{align*}

where

\begin{itemize}
\item $F$ is the set of all control flow parameters;
\item $S$ is the set of all summands in which $d$ is marked as `changed' or `used';
\item $\text{ruling}(s)$ is the set of all parameters that rule summand $s$.
\end{itemize}

\clearpage
\subsection{Relevance relation}

The relevance relation $R$ bundles the information that has been gathered so far.
The relation $R$ relates a data parameter $d$, a control flow parameter $f$, and a value $v$ if $d$ may be used after a state in which $f$ has value $v$.
This is denoted $R(d, f, v)$.
Intuitively, this is a situation in which $d$ should \emph{not} be reset.

The computation of $R$ is a fixpoint algorithm: modifications are applied iteratively until $R$ no longer changes.
The initial value of $R$ is set to

\begin{align*}
R_0 = \bigcup\limits_{\substack{i \in S \\ d_k \in \text{directlyUsed}(i)}}^{} \;\{\; (d_k, d_j, \text{source}(i, d_j)) \;|\; d_j \in b(d_k) \;\}
\end{align*}

where

\begin{itemize}
\item $S$ is the set of all summands;
\item $\text{directlyUsed}(i)$ is a function that gives the set of all parameters marked as `directly used' in a summand $i$;
\item $\text{source}(i, f)$ is a function that gives the source value of a control flow parameter $f$ for a summand $i$.
\end{itemize}

Each iteration can be split into two steps.
The first step checks for control flow graphs in which a data parameter $d$ has already been marked as `relevant' whether this implies that $d$ is also relevant in preceding states (of the same control flow graph).
If so, the appropriate triples are added to $R$:

\begin{align*}
R_{n}{'} = R_{n-1} \cup \bigcup\limits_{\substack{i \in S}}^{} \;\left\{\; (d_k, d_j, s) \;\middle|\; \substack{(d_l, d_j, t) \in R_{n-1} \\ d_j \in b(d_k) \\ d_k \in \text{vars}(v_i(d_l)) \\ (s, i, t) \in E_{d_j}} \;\right\}
\end{align*}

where

\begin{itemize}
\item $E_{f}$ is the set of edges that are part of the control flow graph of control flow parameter $f$.
\end{itemize}

The second step is similar, but data parameters that are found to be relevant are added as triples to $R$ in relation to \emph{another} control flow parameter:

\begin{align*}
R_{n} = R_{n}{'} \cup \bigcup\limits_{\substack{i \in S}}^{} \;\left\{\; (d_k, d_j, \text{source}(i, d_j)) \;\middle|\; \substack{(d_l, d_p, t) \in R_{n}{'} \\ d_j \in b(d_k),\; d_j \notin b(d_l) \\ d_k \in \text{vars}(v_i(d_l)) \\ (r, i, t) \in E_{d_p}} \;\right\}
\end{align*}

\clearpage
\subsection{Summand modification}

Finally, each summand of an LPE can be modified by making use of the belongs-to function $b$ and the relevance relation $R$.
Intuitively, we check whether the value of a parameter is `relevant' after a specific summand $i$ has been applied.
If so, the parameter should not be reset.
Otherwise, the parameter can be reset, for example to its initialization value:

\begin{align*}
v_{i}{'}(d_k) = \begin{cases}
v_{i}(d_k) & \text{if } \bigwedge\limits_{\substack{d_j \in \text{ruling}(i) \\ d_j \in b(d_k)}}^{} R(d_k, d_j, \text{dest}(i, d_j)) \\
v_{0}(d_k) & \text{otherwise}
\end{cases}
\end{align*}

where

\begin{itemize}
\item $\text{ruling}(s)$ is the set of all parameters that rule summand $s$;
\item $\text{source}(i, f)$ is a function that gives the destination value of a control flow parameter $f$ for a summand $i$.
\end{itemize}


\section{Background and Preliminaries}
\label{sec:bg}

This section introduces our running example, necessary background of ML system internals, as well as common types of redundancy.

\subsection{Running Example}

Example~\ref{ex:1} shows a user-level example ML pipeline---written in SystemDS' DML scripting language with R-like syntax \cite{BoehmADGIKLPR20}---which we use as a running example throughout this paper.

\begin{example} [GridSearch LM] \label{ex:1} We read a feature matrix \mat{X} and labels \mat{y}, and extract 10 random subsets of 15 features. For each feature set, we tune the linear regression (lm) hyper-parameters regularization, intercept, and tolerance via grid search and print the loss.
\begin{lstlisting}
 1: X = read('data/X.csv'); # 1M x 100
 2: y = read('data/y.csv'); # 1M x 1
 3: for( i in 1:10) {
 4:   s = sample(15, ncol(X));
 5:   [loss, B] = gridSearch('lm', 'l2norm',
        list(X[,s],y), list('reg','icpt','tol'),...);
 6:   print("Feature set ["+toString(s)+"]: "+loss);
 7: }
\end{lstlisting}
High-level primitives like \texttt{gridSearch} and \texttt{lm} are themselves script-based built-in functions and imported accordingly. Below functions show their key characteristics in simplified form:
\begin{lstlisting}
01: gridSearch = function(...) return(...) {
02:   HP = ... # materialize hyper-parameter tuples
03:   parfor( i in 1:nrow(HP) ) { # parallel for
04:     largs = ... # setup list hyper-parameters
05:     rB[i,] = t(eval(train, largs));
06:     rL[i,] = eval(score, list(X,y,t(rB[i,])));
07: } }
08: lm = function(...) return(...) {
09:   if (ncol(X) <= 1024)  # select closed-form
10:     B = lmDS(X, y, icpt, reg, verbose);
11:   else                  # select iterative
12:     B = lmCG(X, y, icpt, reg, tol, maxi, verbose);
13: }
14: lmDS = function(...) return(...) {
15:   if (icpt > 0) {
16:     X = cbind(X, matrix(1,nrow(X),1));
17:     if (icpt == 2)
18:       X = scaleAndShift(X); # mu=0,sd=1
19:   } ...
20:   A = t(X) %*% X + diag(matrix(reg,ncol(X),1);
21:   b = t(X) %*% y;
22:   beta = solve(A, b);
23: }
24: lmCG = function(...) return(...) {
25:   if (icpt > 0) {
26:     X = cbind(X, matrix(1,nrow(X),1));
27:     if (icpt == 2)
28:       X = scaleAndShift(X); # mu=0,sd=1
29:   } ...
30:   while (i<maxi & norm_r2>norm_r2_tgt) {
31:     q = t(X) %*% (X %*% ssX_p); ...
32:     p = -r + (norm_r2 / old_norm_r2) * p;
33: } }
\end{lstlisting}
The \texttt{gridSearch} function enumerates and materializes all hyper-parameter combinations $\mat{HP}$ of the passed parameters and value ranges, and invokes training (\texttt{lm}) and scoring (\texttt{l2norm}) functions to find the best model and loss. The \texttt{lm} function in turn dispatches---based on the number of features---either to a closed-form method with $\mathcal{O}(m\cdot n^2 + n^3)$ complexity (\texttt{lmDS}); or an iterative conjugate-gradient method with $\mathcal{O}(m \cdot n)$ per iteration (\texttt{lmCG}), which performs better for many features as it requires $\leq n$ iterations until convergence.
\vspace{-0.1cm}
\end{example}

\subsection{ML Systems Background}
\label{sec:mlsys}

There is a variety of existing ML systems. Relevant for understanding this paper, are especially the underlying techniques for program and DAG compilation, and operator scheduling~\cite{2019Boehm}. Here, we focus primarily on lazy evaluation and program compilation.

\textbf{Program/DAG Compilation:} We distinguish three types of compilation in contemporary ML systems: (1) interpretation or eager execution, (2) lazy expression or DAG compilation, and (3) program compilation. First, interpretation as used in R, PyTorch \cite{PaszkeGMLBCKLGA19}, or Python libraries like NumPy \cite{WaltCV11} or Scikit-learn \cite{PedregosaVGMTGBPWDVPCBPD11} execute operations as-is and the host language (e.g., Python) handles the scoping of variables. Second, systems like TensorFlow \cite{AbadiBCCDDDGIIK16}, OptiML \cite{SujeethLBRCWAOO11}, and Mahout Samsara \cite{MahoutSamsara} performing lazy expression evaluation that lazily collects a DAG of operations, which is optimized and executed on demand. Some of these systems---like TensorFlow or OptiML---additionally provide control flow primitives, integrated in the data flow graph. Here, the host language still interprets the control flow, and thus, unrolls operations into a larger DAG. However, recent work like AutoGraph \cite{abs-1810-08061} automatically compiles TensorFlow control flow primitives. Only bound output variables leave the scope of expression evaluation. Third, program compilation in systems like Julia \cite{BezansonEKS17}, SystemML \cite{BoehmDEEMPRRSST16}, SystemDS~\cite{BoehmADGIKLPR20}, and Cumulon \cite{HuangB013} compiles a script into a hierarchy of program blocks, where every last-level block contains DAGs of operations. Accordingly, control flow and variable scoping is handled by the ML system itself. Despite the large optimization scope of lazy expression evaluation and program compilation, unnecessary redundancy cannot be fully eliminated via code motion and common subexpression elimination (CSE) because the conditional control flow is often unknown.

\begin{figure}[!t]
	\centering
	\includegraphics[scale=0.32]{figures/background}
	\vspace{-0.25cm}
	\caption{\label{fig:background}Operator Scheduling and Runtime Plans.}
\end{figure}

\textbf{Operator Scheduling:} Given a DAG of operations of an expression or program block, operator scheduling then determines an execution order of the individual operations, subject to the explicit data dependencies (i.e., edges) of the data flow graph. The two predominant approaches are sequential and parallel instruction streams. First, a sequential instruction stream linearizes the DAG---in depth- or breadth-first order---into a sequence of instructions that is executed one-at-a-time. For example, Figure~\ref{fig:background} shows a plan of runtime instructions in SystemDS for lines 21-23 of Example 1. A symbol table holds references to live variables and their metadata. Instructions are executed sequentially, read their inputs from a variable map (a.k.a. symbol table), and put their outputs back. Such a serial execution model---as used in PyTorch \cite{PaszkeGMLBCKLGA19} and SystemML \cite{BoehmDEEMPRRSST16,BoehmBERRSTT14}---is simple and allows bounding the memory requirements. Second, parallel instruction streams---as used in TensorFlow \cite{AbadiBCCDDDGIIK16}---leverage inter-operator parallelism: when all inputs of an operation are available, this operation is enqueued for parallel execution. This  execution model offers a high degree of parallelism (for many small operations) but makes memory requirements less predictable.

\subsection{Sources of Redundancy}
\label{sec:redundancy}

We can now return to our running example and discuss common sources of fine-grained redundancy.

\begin{example} [GridSearch LM Redundancy]
The user script from Example~\ref{ex:1} with a $1\text{M} \times 100$ feature matrix $\mat{X}$ and three hyper-parameters (\texttt{reg}, \texttt{icpt}, \texttt{tol} with 6, 3, and 5 values) exhibits multiple sources of redundancy.
%
First, since $\mat{X}$ has 100 features, all calls to \texttt{lm} are dispatched to \texttt{lmDS} and thus, one of the hyper-parameters (\texttt{tol}) is irrelevant and we train five times more models than necessary.
%
Second, evaluating different $\lambda$ parameters (\texttt{reg}) for \texttt{lmDS} exhibits fine-grained operational redundancy. The core operations $\mat{X}^{\top}\mat{X}$ and $\mat{X}^{\top}\mat{y}$ are independent of \texttt{reg} and thus, should be executed only once for different $\lambda$.
%
Third, both \texttt{lmDS} and \texttt{lmCG} have the same pre-processing block, and for 2/3 of \texttt{icpt} values, we perform the same \texttt{cbind} operation, which is expensive because it creates an intermediate larger than $\mat{X}$.
%
Fourth, appending a column of ones does not require re-executing $\mat{X}^{\top}\mat{X}$ and $\mat{X}^{\top}\mat{y}$. Instead we can reuse these intermediates and augment them with $\text{colSums}(\mat{X})$, $\text{sum}(\mat{y})$ and $\text{nrow}(\mat{X})$. Similarly, the random feature sets will exhibit overlapping features whose results can be reused.
%
%Overall, eliminating all redundancy allows us to reduce the number of floating point operations from XXX to XXX.
\end{example}

\textbf{Types of Redundancy:} Existing work performs reuse for coarse-grained sub-tasks in ML pipelines \cite{SparksVKFR17, ZhangKR14, VartakTMZ18, XinMMLSP18, ShangZBKECBUK19, DerakhshanMARM20}. Generalizing upon the previous example, we further extend this to common types of \emph{fine-grained} redundancy:
\begin{itemize}
\item \emph{Full Function or Block Redundancy:} At all levels of the program hierarchy, there is potential for full reuse of the outputs of program blocks. This reuse is a form of function memoization \cite{CrankshawWZFGS17}, which requires deterministic operations. %on the taken control path
\item \emph{Full Operation Redundancy:} Last-level operations can be reused for equivalent inputs, given that all non-determinism (e.g., a system-generated seed) is exposed from these operations and cast to a basic input as well.
\item \emph{Partial Operation Redundancy:} Operation inputs with overlapping rows or columns further allow reuse by extraction from---or augmentation of---previously computed results.
\end{itemize}
Together, these different types of redundancy motivate a design with (1) fine-grained lineage tracing, (2) multi-level, lineage-based reuse, and (3) exploitation of both full and partial reuse.

\textbf{Applicability in ML Systems:} Fine-grained lineage tracing and reuse is applicable in ML systems with eager execution, lazy  evaluation, and program compilation. In contrast, multi-level tracing, deduplication, and reuse require access to control structures and thus, are limited to systems with program compilation scope.


% This is LLNCS.DOC the documentation file of
% the LaTeX2e class from Springer-Verlag
% for Lbecture Notes in Computer Science, version 2.4
\documentclass{article} % For LaTeX2e
\usepackage{nips15submit_e,times}
\usepackage{hyperref}
\usepackage{url}
\usepackage[dvips]{graphicx}
\usepackage{xcolor}
%\usepackage{url}
\usepackage{colortbl}
\usepackage{multirow}
\usepackage{stmaryrd}

%\usepackage{amssymb}
%\newtheorem{definition}{Definition} % [section]
%\newtheorem{example}{Example} % [section]
\newcommand{\pivot}[1]{\mathbin{\, {#1} \,}}
\newcommand{\Pivot}[1]{\mathbin{\; {#1} \;}}
\newcommand{\Var}[0]{\mbox{\texttt{Var}}}
\newcommand{\tuple}[1]{\(\langle #1\rangle\)}
\def\withmath#1{\relax\ifmmode#1\else{$ #1 $}\fi}
\def\defeq{\withmath{\stackrel{d}{=}}}
\let\from=\leftarrow

\newcommand{\keywords}[1]{\par\addvspace\baselineskip
\noindent\keywordname\enspace\ignorespaces#1}
\input{commands.tex}

\begin{document}
%\bibliographystyle{acmtrans}

\long\def\comment#1{}
\def\mtimes{}
\def\LL#1{\llbracket #1 \rrbracket}
\title{Notes on differentiable logic \\
{\small Logic through vector calculus} \\
{\small *** DRAFT v0.06 ***}}

\author{Vijay Saraswat \\
IBM T.J. Watson Research Center\\
1101 Kitchawan Road\\
Yorktown Heights, NY 10598 \\
\texttt{vijay@saraswat.org} \\
\And
Radha Jagadeesan \\
De Paul University \\
243 S. Wabash Avenue \\
Chicago, IL 60604 \\
\texttt{rjagadeesan@cs.depaul.edu}
}

\newcommand{\fix}{\marginpar{FIX}}
\newcommand{\new}{\marginpar{NEW}}

\nipsfinalcopy % Uncomment for camera-ready version

\maketitle

\begin{abstract}
First-order logic has crisp notions of individuation and predication: individuals are taken  to be fully formed objects from some underlying set $U$ which typically has no internal structure, and $k$-ary predicates are taken to be $k$-ary maps from $U$ to $B=\{0,1\}$ that specify whether a particular tuple is in the relation or not.

Recent years have seen some remarkable successes in the application of machine learning techniques to a variety of applications in speech, vision and natural language understanding. Typically, these successes are based on the use of continuous (non-convex) optimization techniques (such as gradient descent). The need to account for logical structure has led to the development of several {\em ad hoc} techniques based on embedding individuals into $n$-dimensional vectors (``distributional representations'', \cite{unified-arch-nlp,mikolov-word-vec,baroni2014don}), and treating predicates as tensors \cite{coecke-2010,grefenstette-2013,socher-ntn-2013,bishan-iclr15,rocktaschel-2015,order-embedding-kiros,TransIE}, without a proper understanding of how these techniques relate to logic.

%% TODO: 1-Lipschitz vs continuous.
In these notes we give an account of a precise embedding of first-order logic (over finitary structures) into vector spaces over $\Re$, via the ``Fourier expansion'' of Boolean functions as multilinear polynomials. This immediately permits a continuous embedding of predicates into $[0,1]^n \rightarrow [0,1]$ that enables techniques such as gradient descent to be used directly on such formulas.

%\keywords{machine learning, concurrent constraint programming}
\end{abstract}

\def\Or{\vee}
\def\And{\wedge}
\def\Arrow{\rightarrow}
\def\Xor{\;\mbox{xor}\;}
\def\Ind{\;\mbox{Ind}}

\section{Introduction}
The {\em supervised machine learning} approach is applicable in settings where the programmer is concerned with specifying a function that must work accurately even in the presence of significant amounts of noise in the input. For instance, suppose the programmer must write code that labels an arbitrary input image with a tag (``dog with hat'') that best characterizes the image. The space of variations in the input is generally so vast and potentially so ill-understood mathematically that it may simply not be possible for the user to programmatically specify all the logic for the function.

Instead we desire an approach whereby (portions of the) logic may be {\em learnt} by automatic techniques, given a (potentially very) large collection of observations (input/output pairs) of the given function.

Concretely, we may describe the problem as follows. Instead of providing a fully realized, executable function $f$, the programmer specifies a space $\cal F$ of possible functions, obtained by varying the parameters $\theta$ of some parametric function $f_{\theta}$.  Also available is the set of {\em observations} $(\bar{x}_i, \bar{y}_i)$ (for $i\in I$) specifying pairs of input values with their associated output values, and a {\em loss function} $L$ which can take two values $\bar{y},\bar{y}'$ and return a real value  $L(\bar{y},\bar{y}')$ which measures how far one value is from the other. The machine learning problem is now to determine a specific value $\hat{\theta}$ for the parameters which minimizes $$\Sigma_{i \in I} L(f_\theta(\bar{x}_i), \bar{y}_i)$$

Many algorithmic techniques are available, under various conditions, to address this problem. For instance, in situations where $f_{\theta}$ represents a differentiable function of its parameters $\theta$, one may use stochastic gradient descent to find a minima. Under other conditions, the Expectation Maximization algorithm may be used.

A very powerful instance of this general picture is obtained with (feed forward) {\em deep neural networks} (see, e.g., \cite{deep-learning-nature-2015}).  In this setting, the parametric function $f_{\theta}$ is given by $k$ layers of compute elements (called ``neurons''), an element in layer $i$ connected to some (possibly all) elements in the preceding layer. Each neuron is associated with a set of weights $W$ (its parameters) used to linearly sum its inputs. (There is also usually a bias value.) Some non-linear (but piece-wise differentiable) thresholding function (e.g. sigmoid, $\lambda t.\,(1/(1+e^{-t}))$) is used to determine the output of this element. One can think of the entire network as specified by a $k$-nested functional term, involving summations and pointwise applications of the thresholding function. It has been shown that any arbitrary non-linear (real-valued) function can be approximated arbitrarily closely with sufficiently many parameters, and sufficient training input.

What has made the machine learning problem of great interest in recent years is the availability of large amounts of labeled training data, and vast amounts of (CPU, GPU) computation that can be brought to bear to train the given function. A better understanding of how training algorithms such as asynchronous stochastic gradient networks  \cite{distbelief} can be made to work for deep networks has brought about a startling increase in accuracy in many applications, and a significant increase in the range of applications amenable to these techniques.

\subsection{Differentiable programming languages}
A significant drawback of deep neural networks is their {\em   opacity}. In general, it is not possible to give a meaningful answer to the question of {\em why} the learnt program does what it does. At hand is simply the value of the (potentially billions of) parameters, poor material from which to construct causally coherent explanations accessible to a human observer.

We are interested in developing the foundations of a powerful framework for machine learning and reasoning which gives the programmer flexibility in expressing significant amount of prior knowledge in the struture of the space $\cal F$. We wish to use the full power of a general-purpose programming framework, allied with machine learning. Conceptually, the programmer may write programs in a high-level language while leaving ``holes'' (also called {\em   sketches}, \cite{solar-lezama:asplos06}) to be filled in by an appropriate automated assistant, such as a supervised machine learning algorithm, or a constraint solver. For instance, a program may leave the definition of a procedure {\em q?} unspecified, merely leaving in place an invocation of the procedure on particular arguments (e.g.{} {\em q?(X,3,true)}). The task of the solver is to determine the definition of the procedure so that the overall program exhibits the desired observations (approximately).

While considerable progress has been made in the last decade on the use of constraint solvers for sketching (see, e.g., \cite{armando}), the use of machine learning for sketching is an unexplored area. In particular, it calls for developing a ``differentiable'' programming language in which programs are embedded in a continuous space, may have user-specified parameters, and are end-to-end differentiable with respect to these parameters.

Of particular interest to us are programming languages based on logic -- such as definite clause logic programming \cite{Kowalski76} and concurrent constraint programming \cite{CCP}, including probabilistic versions thereof \cite{muggleton:srl07,Gupta97b,cussens}. Such frameworks carry dual advantages -- they permit the direct expression of knowledge about the real world (via first-order formalizations), and they offer the full power of programming languages for algorithmic tasks.

The focus of this note is to establish the foundations for a continuous embedding of logic. Future version of this note / subsequent papers will develop machine learning applications and implementations.

\subsection{Related work}

%% Also describe recent general end-to-end differentiable approaches. Neural Turing Machines.

%% Work on open IE and matrix completion as a way of learning new predicates.

%% Work by Bishan Yang, and Rajarshi Das and others on doing inference in knowledge graph through neural network techniques.

%% Work on Neural Theorem Proving

%% Work by Grefenstette and Cock on using tensors for first order formulas.
\section{Basic Idea}
We develop a continuous embedding for first-order logic, building on the ideas of \cite{boolean-function}. Let us quickly sketch the construction.

Let $B=\{0,1\}$, $I=[0,1]$, $U=B^n$ (the space of $n$-vectors over $B$) and $V=I^n$. The universe of individuals is taken to be $U$, predicates are thought of as {\em multilinear polynomials} from $U^k$ to $B$, and logical functions are taken as maps from $U^k$ to $U$. The polynomial representation (also called the {\em Fourier expansion}) provides the basis for an embedding of  predicates into the space of continuous functions $V^k \rightarrow I$ and an embedding of logical functions into continuous functions $V\rightarrow V$.

Note that the set of all Boolean functions in $n$ variables forms a vector space over $\Re$. The elementary monomials $\Pi_{i\in A} x_i$ (with $A \subseteq \{1,\ldots, n\}$) form a basis. (The identity $x\mtimes x=x$ can be used to reduce all monomials to this form.)

We now proceed step by step.
\subsection{Propositional logic}\label{section:prop-logic}
Let the underlying base domain be $B$, with $0$ interpreted as false and $1$ as true. Define $\bar{a}$ as $1-a$. Define:

\begin{align*}
  \LL{\neg a} &= \bar{\LL{a}}\\
  \LL{a \And b} &= \LL{a}\mtimes \LL{b}
\end{align*}

The definition of the other logical operators follows:
\begin{align*}
  \LL{a\Or b} &= \overline{\bar{a}\bar{b}} = a+b(1-a)=a(1-b)+b \\
  \LL{a \Arrow b} &= \overline{\LL{a}\mtimes\bar{\LL{b}}}\\
  \LL{a \Xor b} &= \LL{a} + \LL{b} - 2\mtimes\LL{a}\mtimes\LL{b}=(\LL{a}-\LL{b})^2\\
  \LL{(a = b)} &=\LL{a}\mtimes \LL{b} + \bar{\LL{a}}\bar{\LL{b}} =\overline{(\LL{a}-\LL{b})^2}
\end{align*}

% TODO: Look into using max for Or and min for And. How is the completion of these different
% from the definitions above?

The indicator function $\Ind_a(x)$ which for $a,x \in B$ returns
$1$ if $a$ equals $x$ and $0$ otherwise, is just $\_=\_$.

\subsection{Representing first-order logic}

\subsubsection{Representing the domain of discourse}
For now we shall consider only finite domains (the finiteness assumption of the underlying domain is standard practice in machine learning).

We shall now make an assumption fundamental to this vector space approach to logic: individuals can be associated with vectors over \(B^n\). That is, we assume that the space of individuals carries ``internal'' structure: an individual is, intrinsically, a linear combination of \(n\) unit vectors in some space, and the predicates of interest to us respect this linear structure. Note that the ``classical'' domain of discourse for logic -- consisting of a set of \(n\) individuals with no ``internal'' connection to each other -- can be accommodated in the current setting by considering a space with \(n\) dimensions, and associating each individual with a unit vector along a given dimension. Thus the individuals are ``linearly independent'' of each other (orthogonal). This use of ``one-hot'' representations comes at a price -- $n$ bits are used to represent a set of size \(n\), rather than a set of size \(2^n\). %\footnote{Of course, a ``sparse'' representation of the vector could be used, each vector being represented by the index of the unique element that is \(1\) -- this is a representation of size \(\log_2(n)\).}

The justification for such an assumption  comes from the ``associational'' theories of Robert Firth (``you shall know the meaning of words by the company they keep''), \cite{firth-57}. In natural language processing, such a ``vector representation'' of words (including words such as {\sc London} and {\sc Paris}) has met with some success.  Various mechanisms have been devised (e.g.{} \cite{word-to-vec}) using associational techniques to associate dense vectors with objects. Determining the dimensionality of this vector space now becomes a critical task in this approach to knowledge representation. Empirically, \(n=300\) and \(n=600\) have been used; likely these numbers are low for general knowledge representation tasks.

Note that the linear representation automatically comes with a ``built-in'' similarity relation $\sim$ (the cosine distance between the vectors).

%We shall assume an external mechanism has been used to identify the individuals in the domain of interest to us (e.g.{} the characters in some novel), based on empirically available data. Further, this may have been done using ``distributional'' techniques, so that the individuals are in fact represented as (dense) vectors in some high dimensional space $U$ \cite{word-to-vec}. This linear representation automatically comes with a ``built-in'' similarity relation $\sim$ (the cosine distance between the vectors). The value of $n$ will depend on the amount of data available and the number of individuals; $n=300$ and $n=600$ have been used to create vectors for words in various corpora. Determining the right value of \(n\) is a critical task in this approach to knowledge representation.

We can extend the technique of indicator functions discussed above to $n$-vectors. By $\Ind_a(x)$ (for $a,x$ $n$-vectors over \(B\)) we will mean \(\Pi_{i   \in 1\ldots n} \Ind_{a_i}(x_i)\).  Therefore, any function \(f:B^n \rightarrow \Re\) can be represented as a multilinear polynomial (its ``Fourier expansion'') \cite{boolean-function}:
\begin{equation}\label{eqn:fourier-expansion}
\LL{f} \defeq \Sigma_{a \in \{0,1\}^n} f(a)\Ind_{a}(x)
\end{equation}

Alternatively, we can use a different basis, the monomials \cite{PB-optimization}. Every
function $f:\{0,1\}^n \rightarrow \Re$ can be uniquely represented as a multilinear polynomial:
$$ c_0 + \Sigma_{k \in 1\ldots m} (c_k\; \Pi_{i \in A_k} x_i)$$
\noindent for real coefficients $c_0, c_1, \ldots c_m$ and non-empty subsets $A_1, \ldots, A_m$ of $N = \{1, \ldots, n\}$.

\begin{example}
Let us work out $\mbox{and2}: \{0,1\}^2 \rightarrow \Re$ (a unary function that returns 1 iff both components are 1):

\begin{align*}
\LL{and2(x)}  &= 1\mtimes \Ind_{(1,1)}(x) + 0\mtimes\Ind_{(1,0)}(x) + 0\mtimes\Ind_{(0,1)}(x) + 0\mtimes\Ind_{(0,0)}(x)\\
&= \Ind_{(1,1)}(x) \\
&= (1 \mtimes x_1+0\mtimes(1-x_1))\mtimes(1 \mtimes x_2+0\mtimes(1-x_2))\\
  &=x_1\mtimes x_2
\end{align*}
\end{example}

\subsubsection{Representing first-order predicates}
We can use the technique of indicator functions to represent a $k$-ary predicate $p$ as a multilinear polynomial over \(kn\) variables.
\begin{equation}
\LL{p(x_1,\ldots, x_k)} \defeq \Sigma_{a_1,\ldots,a_k \in U} p(a_1, \ldots, a_k) \mtimes \Pi_{i \in 1\ldots k} \Ind_{a_i}(x_i)
\end{equation}

%% TODO: Somewhere we need to introduce the point that p is represented by n^k numbers only.
%% We have to find a way to use the linear expansion here to reduce the size of these formulas
%%

Importantly, the above representation is linear in \(p(a_1, \ldots, a_k)\). If the extension of the predicate is only partially known, each value for which it is not known can be replaced by a variable without affecting multilinearity. In this sense, the representation supports a form of ``higher-orderedness'' as well.

Note that this technique can be used to represent the equality predicate: Here $p(a,b)=1$ if $a=b$, and $0$ otherwise.

A $k$-ary function $f:U^k \rightarrow U$ is represented by $n$ polynomials, one for each dimension.

The Boolean operations are defined as in Section~\ref{section:prop-logic}. Since the domain is finite, existential quantification is represented by a finite disjunction:
\begin{equation}
  \LL{\exists X \, \phi}\defeq \LL{\bigvee_{a \in U} \phi[a/X]}
\end{equation}
\noindent Universal quantification is represented by a finite conjunction:
\begin{equation}
  \LL{\forall X \, \phi} \defeq \LL{\bigwedge_{a \in U} \phi[a/X]}
\end{equation}

Thus, any first-order formula (with equality) in a logical vocabulary \({\cal L}\) can be represented as multilinear polynomial over \(B\). Importantly, formulas with free variables are representable as well.

\begin{example}
  We can represent the formula \(\forall X, Y. P(X,Y) \leftarrow \exists Z. r(X,Z) \wedge P(Z, Y) \) (for \(r\) a binary predicate whose extension is known and \(P\) a binary predicate whose extension is not known) as follows. We introduce \(|U|^2\) variables \(P_{a,b}\) for \(a,b\in U\):
  \[
  \begin{array}{lll}
    \LL{P(Z,Y)} &=& \Sigma_{a,b \in U} P_{a,b} \mtimes \LL{Z=a}\mtimes \LL{Y=B}\\
    \LL{r(X,Z)} &=& \Sigma_{a,b \in U} r(a,b) \mtimes \LL{X=a}\mtimes \LL{Z=B}\\
    \LL{\exists Z\ r(X,Z)\wedge P(Z,Y)} &=&
  \end{array}
  \]
\end{example}

\subsection{Tensor representation}
A multilinear function $U^k \rightarrow \Re$ can be represented directly as a (covariant) tensor $T_k(U)$ \cite[Chapter 8]{lee-book-2000}. The tensor maps a tuple \tuple{u_1,\ldots, u_k} to $1$ if the tuple is in the relation and to $0$ otherwise. Function application is implemented through tensor contraction. Note that in practice such a tensor is implemented extensionally merely as a \(k\)-dimensional array with \(n\) elements in each dimension (thus requiring \(n^k\) numbers).

For more details, please see Appendix~\ref{sec:tensor} where a relatively self-contained background is provided for tensors, and examples worked out.

\subsection{Continuous embedding}
We now take the all important step of passing to the continuous version of the above representation.
The continuous representation of a first-order formula $p$, $p^c$ is given by the above polynomial, with
variables ranging over $[0,1]$. It is easy to see that the result will also be in $[0,1]$. (The result is the weighted sum of $p(a_1, \ldots,
  a_k)$, with the weight supplied by the distance between $(x_1,
  \ldots, x_k)$ and $(a_1, \ldots, a_k)$ using $\Ind_{a_i}(x_i)$. The
  weights sum to $1$, as can be shown by an inductive argument,
  leveraging the fact that $\Ind_{0}(x) + \Ind_{1}(x)=(1x+0(1-x))+(0x+1(1-x))=1$.)

Similarly for $f^c$.

The intuition is that $[0,1]^n$ represents a continuous smearing of the space $\{0,1\}^n$ of ``ideal'' individuals. The polynomial representation of a relation faithfully represents the behavior of the original (discrete) relation on the ideal points, and represents the behavior on intermediate (``smeared'') points as a linear combination of the behavior on ideal points.
% TODO: Should be able to formulate using probabilities and expectations.

The rules for differentiation of these polynomials wrt a variable are standard.

\begin{example}
Assume the underlying domain is expressible as 2-dimensional vectors. Let {\em bill} be represented by \((1, 0)\), {\em hilary} by \((0,1)\), and the extension of the predicate {\em love/2} be given by
\begin{lstlisting}
love(bill,hillary).
love(hilary,bill).
love(hilary,hilary).
\end{lstlisting}
The representation of {\em love(x,y)} is:
\begin{align*}
&\Ind_{(1,0)}(x)\Ind_{(0,1)}(y) + \Ind_{(0,1)}(x)\Ind_{(1,0)}(y) + \Ind_{(0,1)}(x)\Ind_{(0,1)}(y)  \\
&= x_1\bar{x_2}\bar{y_1}y_2 + \bar{x_1}x_2y_1\bar{y_2} + \bar{x_1}x_2\bar{y_1}y_2
\end{align*}
For \(((1,0),(0,1))\) the polynomial evaluates to \(1\); for \(((1,0),(1,0))\) it evaluates to \(0\).

For someone \((0.9,0.2)\) more like {\em bill} than {\em hilary}, the degree to which they love
{\em bill} is given by:
\(((0.9)(0.8)(0)(0) + (0.1)(0.2)(1)(1) + (0.1)(0.2)(0)(0)=0.02\).
%$((0.9)(0.8)(1)(1) + (0.1)(0.2)(0)(0) + (0.1)(0.2)(1)(1)=(0.72)+(0.02)=0.74$.

For someone (\((0.5,0.5)\)) with characteristics of both {\em bill} and {\em hilary} we get the degree to which they love {\em hilary}  is given by
%$(0.5)(0.5)(0.5)(0.5)+(0.5)(0.5)(0.5)(0.5)+(0.5)(0.5)(0.5)(0.5)=0.1875$.
\((0.5)(0.5)(1)(1)+(0.5)(0.5)(0)(0)+(0.5)(0.5)(1)(1)=0.5$, and the degree to which they love {\em bill} is given by $(0.5)(0.5)(0)(0) + (0.5)(0.5)(1)(1) + (0.5)(0.5)(0)(0)=0.25\).

\end{example}

\subsection{Expressiveness}
% TODO: motivate and write up more extensively.
% Define 1-Lipschitz. Motivate why they are interesting in this context.
We note that by adding max and step functions we can represent any $1$-Lipschitz function in the following sense.
%This monomials as discussed above, together with max and step functions

Consider the metric \(d(X,Y) = max \{\lvert x_i - y_i \rvert \mid i\in 1\ldots n \}\) over \(I^n\). For a
rational number \(r\) s.t. \(0 \leq r \leq 1\) and \(Q\) a vector of rational numbers, let \(step_{r,Q}:I^n \rightarrow \Re\) be defined as \(max(0, r-d(X,Q))\). This is the ``step'' function that is \(r\) at \(X\), then drops off to zero at the maximum possible rate.

\begin{definition}
  Let \(\cal F\) be the functions generated thus: any multilinear function is in \(\cal F\); if \(f,g \in {\cal F}\), then so is \(max(f,g)\); \(step_{r,Q} \in {\cal F}\) for rational \(r \in I\) s.t. \(0 \leq r \leq 1\) and vector \(Q\) of rationals.
\end{definition}

\begin{theorem}
  For any \(1\)-Lipschitz function \(g:I^n \rightarrow \Re\) and \(\epsilon\) there exists an \(f \in {\cal F}\) s.t. for all \(X\), \(\lvert g(X)-f(X) \rvert \lt \epsilon\).
\end{theorem}
In proof, consider the set \(\{ step_{r,Q} \alt f(Q) \ge r \}\). Closing this set under finite \(max\) yields a increasing sequence of functions that converges uniformly to \(f\).

(Note: this proof does not depend on monomials.)

\section{Applications to Machine Learning}

We outline some directions that open up as a result of our approach.

\subsection{Learning arbitrary definite clauses}
The representation of formulas as polynomials makes it possible to use symbolic approaches to answer some logical questions. Here we discuss the {\em rule extraction} task of \cite[Algorithm 1]{bishan-iclr15}.

The basic approach in the paper is as follows. Entities are represented by $n$-vectors (as we discuss). A binary relation $p$ is represented as a matrix $\LL{p}$, with a predication $p(x,y)$ evaluated as $\LL{x}^t \LL{p}\LL{y}$.\footnote{\cite{bishan-iclr15} also discusses a form where predicates are represented as vectors; however this does not yield good results.} Their approach learns {\em closed path} definite clauses; these are clauses of the form:
$$ p_1(X_1, X_2), p_2(X_2,X_3), \ldots, p_n(X_n, X_{n+1}) \rightarrow p(X_1, X_{n+1})$$
Here $p_i$ are known binary relations (trained matrices are available for them); this set is closed under inversion, since the inverse of a relation can be obtain by matrix operations. The basic idea is to find ``high probability paths'' in the knowledge graph (entities are types of nodes, predicates are binary edges), and compute the matrix representing the relation between the starting vertex $a$ and the final vertex $b$ by composing the matrices of the relations along the path. This is then compared with the existing relation labeling the edge $(a,b)$ using the $L_2$ norm; the rule is accepted if the norm is below a certain threshold. An example of a learned rule is:
\begin{lstlisting}
 nationality(A,C) :- bornInCity(A,B), cityOfCountry(B,C).
\end{lstlisting}

The main drawback of \cite{bishan-iclr15} is that they have an {\em ad hoc} rule for conjunction which permits them to only represent conjunctions of ``adjacent'' relations via matrix multiply. They do not have general compositional rules which permit the computation of conjunction of arbitrary formulas. For instance their technique cannot be used to learn clauses such as:
\begin{lstlisting}
 dualCitizen(A, C, C1) :-
   bornIn(A,B), cityOfCountry(B, C1), C != C1,
   residentOfCountry(C),
   naturalizedInCountry(C).
\end{lstlisting}
With the techniques of this paper we can proceed as follows. The key point is that formulas are polynomials that can be manipulated algebraically. Assume we have identified a first-order formula $\phi(\bar{X})$ as the candidate for the body of a clause with head (an atomic formula) $\psi(\bar{Y})$. Typically $\bar{Y}\subseteq \bar{X}$, consider for the moment that $\bar{X}=\bar{Y}$ (for the more general case we will need an order-reducing tensor operation, corresponding to projection or existential quantification). We need to determine whether the clause $\phi(\bar{X})\rightarrow \psi(\bar{Y})$ is the case. Operating symbolically on the corresponding polynomials, we need to determine whether $1-\phi(1-\psi) \approx 1$, i.e.{} $\phi \approx \phi\psi$. This can be done by tensor operations.

In essence, we can use the theory presented in this paper to address the problem of rule learning in a more general way.

\subsection{CCP programs with sketches}

\section{Work to do}

How does one deal with infinite domains?

{\em How does one deal with infinite domains? Does one need to? Do the
usual machine learning algorithms make sense over infinite domains?
The representations are all in terms of finite vectors/ matrices /
tensors, and individuals are represented with $n$-vectors, so in any
case we are basically considering only an equivalence class of the
domain with $2^n$ classes.}

{\em Track down whether there is any work in psuodo-Boolean
  optimization related to optimization in the presence of first-order
  constraints.
}

{\em Are there general techniques from polynomial theory to help us do
  logic, e.g. using Grobner basis, as in \cite{Brickenstein:2009:PFG:1550968.1551286}.}

{\em Go back to the work on constrained clustering, adding these kinds
of representations of first-order constraints. But first have to find
a tractable way of dealing with quantifiers. Look to hyper-doctrinal paper as a general
way to introduce quantifiers.}

{\em Applications in constraint programming?}

\paragraph{Acknowledgements.} Luc de Raedt, Angelika Kimmig, Tom Mitchell, William Cohen, Bishan Yang, Michael Witbrock, Piero Molino, \ldots

\bibliographystyle{alpha}
\bibliography{master}

\appendix
\newpage

\section{Clausal form and logic programming}
\subsubsection{Clausal representation}
The general representation of a clause with literals $a_i,b_j$ is:
\begin{equation}
\LL{a_1,\ldots, a_m \Arrow b_1, \ldots, b_n} = 1-(\Pi_{i \in 1\ldots m} \LL{a_i}) \mtimes (\Pi_{j\in 1\ldots n} \bar{b}_j)
\end{equation}
\noindent (Think: $a_1,\ldots, a_m \Arrow b_1, \ldots, b_n$ is the same as $\neg ((a_1 \wedge \ldots a_m)
\wedge (\neg b_1) \wedge \ldots \wedge (\neg b_n))$.)

Definite clauses, negative clauses, unit clauses, and empty clauses are the special cases with
$n=1$, $n=0$, $(m=0,n=1)$ and $(m=0,n=0)$:
  \begin{align*}
    \LL{a_1,\ldots, a_m \Arrow b} &= 1-{(\Pi_{i \in 1\ldots m} a_i) \mtimes \bar{b}}\\
    \LL{a_1,\ldots, a_m \Arrow } &= 1-{\Pi_{i \in 1\ldots m} a_i}\\
    \LL{ \Arrow b} &= b\\
    \LL{ \Arrow } &= 1 - 1 = 0
  \end{align*}

Now we can see how unit resolution works.  Given $a=1$, unit resolution reduces $1-a\mtimes\phi$ to $1-\phi$; given $a=0$, $1- (1-a)\mtimes \phi$ reduces to $1-\phi$.

More generally, one can think of definite clause programming thus. We are given a theory $\Sigma=\{(1-\phi_1),\ldots, (1-\phi_n)\}$, where each formula $1-\phi_i$ represents a definite clause.
Let us say that a {\em valuation} is a function from the set of variables to $\{0,1\}$. A {\em solution} of $\Sigma$ assigns $1$ to each formula $1-\phi_i$ in $\Sigma$, i.e.{} assigns zero to each $\phi_i$.

Now we are given a query $q=a_1 \And \ldots \And a_k$ and asked to establish $\Sigma \vdash q$. That is, every solution for $\Sigma$ must assign $1$ to each $a_j, j\in 1\ldots k$. Backward chaining consists of finding some clause $b_1,\ldots, b_m\Arrow a_j$, for some $j\in 1\ldots k$ and replacing the query $q$ with $a_1\And \ldots \And a_{j-1} \And b_1 \And \ldots b_m \And \ldots \And a_k$. (Note that $m$ could be zero, thus decreasing the size of the resulting query.) This process is repeated until the size of the final query is $0$.

Why is this sound? Because every solution $v$ that assigns $0$ to $b_1\mtimes \ldots \mtimes b_j\mtimes (1-a_j)$ and $1$ to
$a_1\mtimes \ldots \mtimes a_{j_1}\mtimes b_1 \mtimes \ldots \mtimes b_m \mtimes a_{j+1}\mtimes \ldots \mtimes a_k$ can only do so by
assigning $1$ to $b_1, \ldots, b_m$ {\em and} $1$ to $a_j$, and $1$ to $a_1,\ldots, a_{j-1},a_{j+1},\ldots a_k$. Therefore it assigns $1$ to $a_1 \mtimes \ldots \mtimes a_k$.

\paragraph{Posiform.}
%% TODO: Have not introduced pseudo-Boolean yet. Why is this important to say. What does ``many ways'' mean?
Let $\bar{x}$ stand for $(1-x)$. Every pseudo-Boolean function can also be represented in many ways through ``clausal decompositions'', called {\em posiforms}:
\begin{equation}
  f(x_1,\ldots, x_n)=b_0 + \Sigma_{k \in 1\ldots m} b_k (\Pi_{i\in A_k} x_i \Pi_{j\in B_k}(\bar{x_j}))
\end{equation}
\noindent where $b_k\geq 0$ for $k \in 1\ldots m$.

\input{appendix-tensor-3}
\end{document}