\documentclass{sigplanconf}
\toappear{}

\usepackage{amssymb}
\usepackage{amsmath}
\usepackage{amsthm}
\usepackage{stmaryrd}
\usepackage{xcolor}
\usepackage{listings}
\usepackage{lstcoq}
\usepackage{comment}
\usepackage{xspace}
\usepackage{flushend}

\lstset{ %
  numberbychapter=false, %
  language=coq, %
%%  frame=lines, %
  frameshape={yyy}{n}{n}{yyy}, %
  framexleftmargin=-3pt,
  framexrightmargin=-3pt,
  numberstyle=\tiny, %
  basicstyle=\footnotesize, %
  captionpos=b,
  numbersep=5pt,
  xleftmargin=5pt,
  xrightmargin=5pt}

%\usepackage[bookmarks=true,colorlinks=true, citecolor=cyan]{hyperref} %%linkcolor=MidnightBlue,

\newcommand{\FOR}{{\tt for}\relax\ifmmode\ \else\xspace\fi}
\newcommand{\FORALL}{{\tt forall}\relax\ifmmode\ \else\xspace\fi}
\newcommand{\EXISTS}{{\tt exists}\relax\ifmmode\ \else\xspace\fi}
\newcommand{\WHERE}{{\tt where}\relax\ifmmode\ \else\xspace\fi}
\newcommand{\IN}{ \ {\tt in} \ }
\newcommand{\RETURN}{{\tt return}\relax\ifmmode\ \else\xspace\fi}
\newcommand{\DO}{{\tt do}}
\newcommand{\IF}{{\tt if} \ }
\newcommand{\THEN}{{\tt then} \ }
\newcommand{\ELSE}{{\tt else} \ }
\newcommand{\ZERO}{{\tt zero}}
\newcommand{\FALSE}{{\tt false}}
\newcommand{\BIND}{{\tt bind}}
\newcommand{\UNION}{{\tt union}}
\newcommand{\MAP}{{\tt map}}
\newcommand{\CONS}{{\tt Cons}}
\newcommand{\NIL}{{\tt Nil}}

\newcommand{\greg}[1]{\textcolor{blue}{GREG: #1}}
\newcommand{\ltac}[0]{\ensuremath{\mathcal{L}_{\mathrm{tac}}}}
\newcommand{\relation}[1]{\ensuremath{\mathit{#1}}\xspace}

\begin{document}

\title{Using Dependent Types and Tactics to Enable Semantic Optimization of Language-Integrated Queries}
%\titlenote{To appear in DBPL 2015}}

\authorinfo{Gregory Malecha}{University of California, San Diego, USA}{\sf gmalecha@eng.ucsd.edu}

\authorinfo{Ryan Wisnesky}{Massachusetts Institute of Technology, USA}{\sf wisnesky@math.mit.edu}

\date{\today}

\maketitle
%\vspace*{-.3in}
\begin{abstract}
Semantic optimization --- the use of data integrity constraints to optimize relational queries --- has been well studied but, owing to limitations in how SQL handles constraints, has not often been applied by mainstream RDBMSs. In a language-integrated query setting, however, the query provider is free to rewrite queries before they are executed on an RDBMS.  We show, using Coq as our ambient language, how to use dependent types to represent a well known class of constraints --- embedded, implicational dependencies --- and how Coq tactics can be used to implement a particular kind of semantic optimization: tableaux minimization, which minimizes the number of joins required by a query.
\end{abstract}

\category{H.2.3}{Database programming languages}{}

\keywords Chase, Coq, LINQ, semantic optimization

\section{Introduction}

{\it Semantic optimization}~\cite{foundations,Deutsch:2006:QRC:1121995.1122010,Popa99anequational} is the
use of data integrity constraints such as keys, functional dependencies, inclusions, and join decompositions to optimize relational queries. For example~\cite{foundations}, consider the following contrived query over a relation (set of records) \relation{Movies}
with fields ${\sf title}$, ${\sf director}$, and ${\sf actor}$:
\begin{eqnarray*}
& & \FOR (m_1 \IN \relation{Movies}) \ (m_2 \IN \relation{Movies}) \\
 & & \WHERE m_1.{\sf title} = m_2.{\sf title} \\
 & & \RETURN (m_1.{\sf director}, m_2.{\sf actor})
\end{eqnarray*}
This query returns (a set of) tuples $(d,a)$ where $a$ acted in a movie directed by $d$.  A na\"ive implementation of this query will require a join.  However, when \relation{Movies} satisfies the functional dependency ${\sf title} \to {\sf director}$ (meaning that
if $({\sf director}: d, {\sf title}: t, {\sf actor}: a)$ and $({\sf director}: d^\prime, {\sf title}: t^\prime, {\sf actor}: a^\prime)$ are \relation{Movies} records such that $t = t^\prime$, then $d = d^\prime$), this query is equivalent to:
\begin{eqnarray*}
& & \FOR (m \IN \relation{Movies}) \\
 & & \RETURN (m.{\sf director}, m.{\sf actor})
 \end{eqnarray*}
which can be evaluated without a join.  (Note that if \relation{Movies} did not satisfy the functional dependency, the equivalence would not necessarily hold.)

Of course, knowing that the functional dependency holds, a programmer might simply write the optimized query to begin with.
However, constraints are not always known at compile time, such as when relations are indexed dynamically.
In addition, not all queries are written by programmers.
For example, information-integration systems such as Clio~\cite{haas:clio} automatically generate large numbers of un-optimized queries. %that should be optimized.
%In cases such as these, semantic optimization must be performed automatically to achieve the significant, potentially order-of-magnitude speed-ups enabled by semantic optimization are well-documented in the literature~\cite{Cheng:1999:ITS:645925.671357}.
%\greg{This last sentence does not seem to fit with this paragraph}

Although certain RDBMS's such as DB2 can perform limited amounts of semantic optimization~\cite{Cheng:1999:ITS:645925.671357}, RDBMS's are fundamentally limited by the expressiveness of SQL as a constraint specification language.
For example, SQL includes keys and foreign keys but constraints such as the functional dependency above are not directly expressible in SQL. (Technically, functional dependencies can be encoded as {\tt CHECK} constraints, but even {\tt CHECK} constraints cannot capture multi-table constraints such as join decompositions.)
In relational database theory, a fragment of first-order logic, {\it embedded, implicational dependencies} (EDs) are used to capture almost all constraints used in practice, including keys, foreign keys, inclusions, functional dependencies, and join decompositions.
A large body of literature has been developed to facilitate reasoning about queries in the presence of EDs~\cite{Popa99anequational}.

\paragraph{Contributions and Outline} In this paper we demonstrate that dependently-typed language-integrated query systems (LINQs~\cite{monad}) that compile to SQL can expose data integrity constraints, in the guise of EDs, as first-class objects to their users. This feature allows them to apply sophisticated semantic optimization techniques before translating user queries into SQL.  In particular, we show, using Coq~\cite{coq:coq} as our ambient language, how to use dependent equality types to represent EDs, and how to use Coq tactics to implement a particular kind of semantic optimization: tableaux minimization, which minimizes the number of joins required by a query.  This paper is divided into two parts: the first part is a tutorial on tableaux minimization, and the second part is a Coq rendering of the first part. The Coq development is available at {\sf github.com/gmalecha/semantic-query}.

\paragraph{Related Work} Most theoretical work on language-integrated query systems is done in a simply-typed setting~\cite{tannen:1992:NEQ:645500.655920}.  In practice, however, sophisticated type systems are often used to to facilitate embedding the query sublanguage into a general purpose programming language.  For example, labeled row types~\cite{mpj:jones1996a} can be used to embed DBMS records into a programming language, and the Opaleye library for Haskell uses the Arrow type-class to statically enforce the well-formedness of its SQL output~\cite{opaleye}.  Rarer still are dependently-typed embedded query languages.
Although Coq has been used to prove the correctness of certain database-related languages, data structures, and algorithms~\cite{Malecha:2010:TVR:1706299.1706329,coqdb}, none of this work is concerned with using Coq directly as an embedded query language as we are doing in this paper (i.e., these works use deep embeddings of query languages, while we use a shallow embedding).

\section{Queries}
%For ease of exposition, in the first part of this paper we will assume we are working in a strongly-normalizing typed $\lambda$-calculus with first-class records, such as~\cite{mpj:jones1996a}.

In this paper we will focus on relational {\it conjunctive queries}~\cite{foundations}, and for the first part of this paper the specifics of our query language will not matter.   We will write $(l_1: e_1, \ldots, l_N: e_N)$ to indicate a record with unique labels $l_1, \ldots l_N$ formed from expressions $e_1, \ldots, e_N$, where an expression has the form $v.l$ for a variable $v$ and label $l$.  We will abbreviate (potentially 0-length) vectors of variables $x_1,...,x_N$ as $\overrightarrow{x}$.  We will write $P(\overrightarrow{x})$ to indicate a conjunction of equalities over expressions over variables $\overrightarrow{x}$.  Assumed base relations (often called {\it roots}) will be written in capital letters, such as $\overrightarrow{X}$.  A {\it tableau} has the form:
\begin{eqnarray*}
 & & \FOR \overrightarrow{(x \IN X)} \\
 & & \WHERE P(\overrightarrow{x})
\end{eqnarray*}
The $\overrightarrow{(x \IN X)}$ are called {\it generators}.  A (conjunctive) {\it query} is a pair of a  tableau and a record (``return clause'') $R(\overrightarrow{x})$:
\begin{eqnarray*}
 & & \FOR \overrightarrow{(x \IN X)} \\
& & \WHERE  P(\overrightarrow{x}) \\
 & & \RETURN R(\overrightarrow{x})
\end{eqnarray*}

\paragraph{Extensions}
We will only consider relational conjunctive queries in this paper, but many extensions to conjunctive queries have been studied in the literature~\cite{foundations}.  Three extensions are particularly important, because many results about semantic optimization, including tableaux minimization, hold for these extensions~\cite{Popa99anequational}:
\begin{itemize}
\item  It is possible to allow generators to be dependent, thereby allowing, for example, nested relations~\cite{Popa99anequational}:
\begin{normalsize}
$$ \FOR (g \IN \relation{Groups}) \ (p \IN g) \ \ldots $$
\end{normalsize}
\item It is possible to interpret queries in arbitrary {\it monads with zeroes}, for example, the list monad or the bag monad.  However, the  optimization procedure described in this paper is only sound for monads that are both commutative and idempotent~\cite{Popa99anequational}:
$$
\FOR (x \IN X)(y \IN Y)  \cong \FOR (y \IN Y) (x \IN X)
$$
$$
\FOR (x \IN X) \cong \FOR (x \IN X)(x \IN X)
$$
Such monads arise, for example, as power monads on topoi~\cite{BW}.
\item It is also possible to interpret queries in {\it monad algebras}~\cite{755736}.  For example, it is possible to write a query to find the largest element of a set:
\begin{normalsize}
$$ \FOR (x \IN \relation{SetOfInts}) \ {\tt max} \ x $$
\end{normalsize}

\end{itemize}

\section{Embedded Dependencies}
\label{sec:eds}

An {\it embedded dependency (ED)}~\cite{foundations} is a pair of tableaux, where one tableau is universally quantified, and the other tableau is existentially quantified:
\begin{eqnarray*}
C & := & \FORALL \overrightarrow{(x \IN X)} \\
 & & \WHERE P(\overrightarrow{x}) \\
 & & \EXISTS \overrightarrow{(y \IN Y)} \\
 & & \WHERE B(\overrightarrow{x}, \overrightarrow{y})
\end{eqnarray*}

\paragraph{Example}
The functional dependency from our example from the introduction is written (the \EXISTS clause is empty):
\begin{eqnarray*}
& & \FORALL (x \IN \relation{Movies}) \ (y \IN \relation{Movies}) \\
& & \WHERE x.{\sf title} = y.{\sf title}, \\
& & \EXISTS \\
& & \WHERE x.{\sf director} = y.{\sf director}
\end{eqnarray*}
An ED $C$ gives rise to two conjunctive queries, the {\it front} and {\it back} of $C$.  We write $\mathcal{L}
(\overrightarrow{x})$ to indicate a record capturing the variables $\overrightarrow{x}
$; e.g., $({\sf x_1}: x_1, \ldots ,{\sf x_N}: x_N)$.  %The front of an ED is:
\begin{eqnarray*}
front(C)& := & \FOR \overrightarrow{(x \IN X)} \\
& & \WHERE P(\overrightarrow{x}) \\
& & \RETURN \mathcal{L}(\overrightarrow{x})  \\
& & \\
%\end{eqnarray*}
%\end{normalsize}
%and the back is
%\begin{normalsize}
%\begin{eqnarray*}
back(C) & := & \FOR \overrightarrow{(x \IN X)} \ \overrightarrow{(y \IN Y)} \\
& & \WHERE P(\overrightarrow{x}) \wedge B(\overrightarrow{x}, \overrightarrow{y}) \\
& & \RETURN \mathcal{L}(\overrightarrow{x})
\end{eqnarray*}
It is easy to establish~\cite{Popa99anequational} that
\[
\forall I, \quad I \models C \quad \textnormal{iff} \quad front(C)(I) = back(C)(I)
\]
In the above, $I \models C$ should be read ``constraint $C$ holds on instance $I$.''
In the second half of this paper, we will use a dependent equality type corresponding to the above equation as a type of proofs that an ED holds in a particular instance.

\textbf{Notation}  When two queries $Q_1$ and $Q_2$ give the same result on every instance, we write $Q_1 \cong Q_2$.  When $Q_1$ and $Q_2$ give the same result on every instance satisfying some set of EDs $C$, we write $C \vdash Q_1 \cong Q_2$.

\section{Homomorphisms}

A {\it homomorphism} $h : Q_1 \to Q_2$ between queries:
%% \begin{eqnarray*}
%% Q_1 & := & \FOR \overrightarrow{(v_1 \IN V_1)} \\
%%           & & \WHERE P_1(\overrightarrow{v_1}) \\
%%           & & \RETURN R_1(\overrightarrow{v_1}) \\
%% \to_h & & \\
%% Q_2 & := & \FOR \overrightarrow{(v_2 \IN V_2)} \\
%%           & & \WHERE P_2(\overrightarrow{v_2}) \\
%%           & & \RETURN R_2(\overrightarrow{v_2})
%% \end{eqnarray*}
\begin{eqnarray*}
\begin{array}{l}
\FOR \overrightarrow{(v_1 \IN V_1)} \\
\WHERE P_1(\overrightarrow{v_1}) \\
\RETURN R_1(\overrightarrow{v_1})
\end{array} & \to_h &
\begin{array}{l}
\FOR \overrightarrow{(v_2 \IN V_2)} \\
\WHERE P_2(\overrightarrow{v_2}) \\
\RETURN R_2(\overrightarrow{v_2})
\end{array}
\end{eqnarray*}
is a substitution mapping the $\FOR$-bound variables of $Q_1$ (namely, $
\overrightarrow{v_1}$) to the $\FOR$-bound variables of $Q_2$ (namely, $
\overrightarrow{v_2}$) that preserves the structure of $Q_1$ in the sense that:
\begin{itemize}
\item
 $(h(v_{1_i}) \IN V_{1_i})$ $ \in$ $\overrightarrow{(v_2 \IN V_2)}$ -- the image of each generator in $Q_1$ is found in the generators of $Q_2$.

\item $P_2(\overrightarrow{v_2})$ $\vdash$ $P_1(h(\overrightarrow{v_1}))$ -- the image of the where clause of $Q_1$ is entailed by the where clause of $Q_2$.

\item $P_2$ $\vdash$ $R_1(h(\overrightarrow{v_1})) = R_2(\overrightarrow{v_2})$ -- the image of the return clause of $Q_1$ is equal to the return clause of $Q_2$, under $P_2$.
\end{itemize}
A homomorphism of tableaux is defined the same way, except that the condition about $\RETURN$ clauses is dropped.

{\bf Notation} We write $Q_1 \leftrightarrow Q_2$ to mean that there exists homomorphisms $Q_1 \to Q_2$ and $Q_2 \to Q_1$ and we say that $Q_1$ and $Q_2$ are {\it homomorphically equivalent}.  The existence of a homomorphism $Q_1 \to Q_2$ implies that for every $I$, $Q_2(I) \subseteq Q_1(I)$, and vice versa~\cite{foundations}.  Hence, by bi-directional subset containment, $Q_1 \cong Q_2$ iff $Q_1 \leftrightarrow Q_2$.

\paragraph{Example} Consider our \relation{Movies} query ($Q_1$) and its semantically optimized counter-part ($Q_2$):
%When queries are {\it path-conjunctive}---that is, when $P_1$, $P_2$ are conjunctions of equalities between paths of the form $v.l_1,.\ldots l_n, $ and  $R_1$ and $R_2$ are records built from paths, as we are assuming in this paper, finding homomorphisms is NP-hard.  Moreover, in this case there are practical, sound heuristics~\cite{Deutsch:2006:QRC:1121995.1122010} based on pruning the search space of substitutions to remove candidates that are ``obviously wrong'' based on a partial variable assignment.
\begin{eqnarray*}
Q_1 & := & \FOR (m_1 \IN \relation{Movies}) \ (m_2 \IN \relation{Movies}) \\
 & & \WHERE m_1.{\sf title} = m_2.{\sf title} \\
 & & \RETURN (m_1.{\sf director}, m_2.{\sf actor}) \\
%% \end{eqnarray*}
%% and the semantically optimized query:
%% \begin{eqnarray*}
Q_2 & := & \FOR (m \IN \relation{Movies}) \\
 & & \RETURN (m.{\sf director}, m.{\sf actor})
\end{eqnarray*}

It is easy to see that for every instance $I$ that satisfies the embedded dependency above, $Q_2(I) \subseteq Q_1(I)$.
To check that there is a homomorphism $h : Q_1 \to Q_2$; namely, the substitution $m_1 \mapsto m, m_2 \mapsto m$,
 we first apply $h$ to $Q_1$ (which will ``shadow'' the name $m$):

%\greg{This query seems strange? Are we shadowing the name $m$? Or is something else going on?}
\begin{eqnarray*}
h(Q_1) & := & \FOR (m \IN \relation{Movies}) \ (m \IN \relation{Movies}) \\
 & & \WHERE m.{\sf title} = m.{\sf title} \\
 & & \RETURN (m.{\sf director}, m.{\sf actor})
\end{eqnarray*}
We see that each generator $(m \IN \relation{Movies})$ in $h(Q_1)$ appears in $Q_2$.
Next, we see that the \WHERE clause of $h(Q_1)$ is a tautology and hence is entailed by the (empty) \WHERE clause of $Q_2$.
Finally, we see that the two \RETURN clauses are equal, and conclude that $m_1 \mapsto m, m_2 \mapsto m$ is a homomorphism.

Note that there is no homomorphism $Q_2 \to Q_1$, and so $Q_1 \ncong Q_2$.
There are only two candidates: $m \mapsto m_1$ and $m \mapsto m_2$.
Neither works since the image of $Q_2$'s \RETURN clause under either substitution (i.e. either $\RETURN (m_1.{\sf director}, m_1.{\sf actor})$ or $\RETURN (m_2.{\sf director}, m_2.{\sf actor})$) is equivalent to $Q_1$'s \RETURN clause ($\RETURN (m_1.{\sf director}, m_2.{\sf actor})$) under the equality in $Q_1$ ($m_1.{\sf title} = m_2.{\sf title}$).

%  Indeed, consider the instance:
%\begin{normalsize}
%\begin{eqnarray*}
%{\sf title} & {\sf director} & {\sf actor} \\
%T & D_1 & A \\
%T & D_2 & B
%\end{eqnarray*}
%\end{normalsize}
%$Q_1$ and $Q_2$ evaluate to, respectively
%
%\parbox{3in}{
%\begin{normalsize}
%\begin{eqnarray*}
% & {\sf director} & {\sf actor} \\
%& D_1 & A \\
%& D_1 & B \\
%& D_2 & A \\
%& D_2 & B
%\end{eqnarray*}
%\end{normalsize}
%} \ \ \ \ \parbox{3in}{
%\begin{normalsize}
%\begin{eqnarray*}
% & {\sf director} & {\sf actor} \\
%& D_1 & A \\
%& D_2 & B
%\end{eqnarray*}
%\end{normalsize}
%}
%
%Of course, if we had chosen an instance $I$ that satisfied the functional dependency {\sf Title} $\to$ {\sf Director}, then $Q_1(I)$ and $Q_2(I)$ would have evaluated to the same result.

\section{The Chase}
\label{sec:chase}

The chase is a confluent rewriting procedure that rewrites queries using EDs~\cite{foundations}.   Let

 %We now describe the chase, and in the next section we show how to use it to optimize queries.
\[
\begin{array}{ccc} %% \parbox{1.5in}{
\begin{array}[t]{rcl}
 C & := & \FORALL \overrightarrow{(x \IN X)} \\
 & & \WHERE P(\overrightarrow{x}) \\
 & & \EXISTS \overrightarrow{(y \IN Y)} \\
 & & \WHERE B(\overrightarrow{x}, \overrightarrow{y})
\end{array} & &
%% }
%% \parbox{1.5in}{
\begin{array}[t]{rcl}
Q & := & \FOR \overrightarrow{(v \IN V)} \\
 & & \WHERE  O(\overrightarrow{v}) \\
 & & \RETURN R(\overrightarrow{v})
\end{array}
\end{array}
\]
and suppose there exists a (tableau) homomorphism $h : front(C) \to Q$.
A \emph{chase step} rewrites $Q$ into $step(C,Q)$ by adding the image of the back of $C$:
\begin{eqnarray*}
step(C,Q) & := & \FOR \overrightarrow{(v \IN V)} \ \overrightarrow{(y \IN Y)} \\
 & & \WHERE  O(\overrightarrow{v}) \wedge B(\overrightarrow{h(x)}, \overrightarrow{y})
\\
 & & \RETURN R(\overrightarrow{v})
\end{eqnarray*}
A chase step is semantics-preserving on instances that satisfy the constraints~\cite{Popa99anequational}, i.e.
\[
C \vdash Q \cong step(C,Q)
\]

The {\it chase} algorithm itself simply repeats the chase step until it finds a fixed point (up to homomorphic equivalence).
That is:
\[\begin{array}{l}
chase(C,Q) \equiv Q' \quad \textrm{iff} \\
\quad Q \rightsquigarrow step(C, Q) \rightsquigarrow step(C, step(C, Q)) \rightsquigarrow \ldots \rightsquigarrow Q'
\end{array}
\]
%The termination condition is to not take a chase step when there is a homomorphism extending $h$ from $chase(Q, C)$ to $Q$.
Termination of the chase is undecidable, but if it terminates the final result is unique (up to homomorphic equivalence)~\cite{Deutsch:2006:QRC:1121995.1122010}.  Provided certain fairness conditions are met~\cite{Deutsch:2006:QRC:1121995.1122010}, the chase extends easily to sets of EDs by choosing a particular ED to chase with at each step.

A key theorem about the chase is that it reduces the question of query equivalence under constraints to homomorphic equivalence.
This means that, if $\vec{C}$ is a set of EDs and $Q_1$ and $Q_2$ are queries:
\[
\vec{C} \vdash Q_1 \cong Q_2 \ \ \ \  \textnormal{iff} \ \ \ \ chase(\vec{C},Q_1) \leftrightarrow chase(\vec{C}, Q_2)
\]
\paragraph{Example}
As we showed in the previous example, there is a homomorphism $x \mapsto m_1, y \mapsto m_2$ from the front of our constraint
\begin{normalsize}
\begin{eqnarray*}
C& := & \FORALL (x \IN \relation{Movies}) \ (y \IN \relation{Movies}) \\
& & \WHERE x.{\sf title} = y.{\sf title}, \\
& & \EXISTS \\
& & \WHERE x.{\sf director} = y.{\sf director}
\end{eqnarray*}
\end{normalsize}
to our original query:

\begin{eqnarray*}
Q_1 & := & \FOR (m_1 \IN \relation{Movies}) \ (m_2 \IN \relation{Movies}) \\
 & & \WHERE m_1.{\sf title} = m_2.{\sf title} \\
 & & \RETURN (m_1.{\sf director}, m_2.{\sf actor})
\end{eqnarray*}
Hence, we can take a chase step:
\begin{eqnarray*}
step(C, Q_1) & := & \FOR (m_1 \IN \relation{Movies}) \ (m_2 \IN \relation{Movies}) \\
 & & \WHERE m_1.{\sf title} = m_2.{\sf title} \wedge \\
 & & \ \ \ \ \ \ \ \ \ \ \ \  m_1.{\sf director} = m_2.{\sf
director} \\
 & & \RETURN (m_1.{\sf director}, m_2.{\sf actor})
\end{eqnarray*}
which adds the \WHERE clause of the back of the constraint to the query's \WHERE clause.
At this point we stop chasing, since $step(C, step(C, Q_1))$ is homomorphically equivalent (even syntactically equivalent) to $step(C, Q_1)$.
By the soundness of the chase we have established that $C \vdash Q_1 \cong chase(C, Q_1)$.
%In general, it is not enough to check for the syntactic equality of $chase(Q, C)$ and $Q$ to stop the chase, as queries can be equivalent without being syntactically equal.  Hence, we must use homomorphisms to detect termination.

\section{Tableaux Minimization}
\label{sec:minimize}

We now demonstrate how to minimize queries in the presence of EDs, using a technique known as ``tableaux minimization using chase and back-chase''~\cite{Deutsch:2006:QRC:1121995.1122010}.
Suppose we are given a query $Q$ and set of EDs $C$.
We first chase $Q$ with $C$ to obtain $U$, a so-called {\it universal plan}.
We then search for sub-queries of $U$ (which are intuitively obtained by removing generators from $U$), chasing each in turn with $C$ to check for equivalence with $U$.
There will always be a unique minimal query (up to homomorphic equivalence)~\cite{Deutsch:2006:QRC:1121995.1122010}.

\paragraph{Example - Movies}
Start with our query and constraint from the introduction:
\begin{eqnarray*}
Q_1 & := & \FOR (m_1 \IN \relation{Movies}) \ (m_2 \IN \relation{Movies}) \\
 & & \WHERE m_1.{\sf title} = m_2.{\sf title} \\
 & & \RETURN (m_1.{\sf director}, m_2.{\sf actor}) \\
% \end{eqnarray*}
 %\begin{eqnarray*}
 & & \\
C & := & \FORALL (x \IN \relation{Movies}) \ (y \IN \relation{Movies}) \\
& & \WHERE x.{\sf title} = y.{\sf title} \\
& & \EXISTS \\
& & \WHERE x.{\sf director} = y.{\sf director}
\end{eqnarray*}
The universal plan, i.e., $chase(C,Q_1)$, is:
\begin{eqnarray*}
U & := & \FOR (m_1 \IN \relation{Movies}) \ (m_2 \IN \relation{Movies}) \\
 & & \WHERE m_1.{\sf title} = m_2.{\sf title} \wedge \\
 & & \qquad\quad m_1.{\sf director} = m_2.{\sf
director} \\
 & & \RETURN (m_1.{\sf director}, m_2.{\sf actor})
\end{eqnarray*}

We proceed with tableau minimization by searching for sub-queries of $U$.
Removing the generator $(m_1 \IN \relation{Movies})$ and replacing $m_1$ with $m_2$ in the body of $Q$ gives a smaller query:
\begin{eqnarray*}
Q_2 & := & \FOR (m_2 \IN \relation{Movies}) \\
 & & \RETURN (m_2.{\sf director}, m_2.{\sf actor})
\end{eqnarray*}
To justify this, we need to check that $C \vdash Q_1 \cong Q_2$, which we can reduce to checking $U = chase(C,Q_1) \leftrightarrow chase(C, Q_2)$.
We find that $chase(C, Q_2) \cong Q_2$, so we will actually check that  $U \leftrightarrow Q_2$.
The identity substitution is a homomorphism $Q_2 \to U$: the important part to notice is the \RETURN clause, wherein $(m_2.{\sf director},$ $m_2.{\sf actor})$ is equal to $(m_1.{\sf director},$ $m_2.{\sf actor})$ precisely because of the equality $m_1.{\sf director}$ $=$ $m_2.{\sf director}$, which appears in $U$ but not in $Q_1$.
There is also a homomorphism $U \to Q_2$, namely, $m_2 \mapsto m, m_1 \mapsto m$.  We thus conclude that $C \vdash U \cong Q_2 \cong Q_1$.

\paragraph{Example - Indexing}
As we remarked in the introduction, a programmer might be able to optimize our \relation{Movies} query directly, without using the chase at all.  But sometimes constraints are not available to the programmer, such as when indices are generated dynamically.  Consider the following query, which returns the names of all \relation{People} between 16 and 18 years old:
\begin{eqnarray*}
Q_1 & := & \FOR (p \IN \relation{People}) \\
 & & \WHERE p.{\sf age} > 16 \wedge p.{\sf age} < 18 \\
 & & \RETURN p.{\sf name}
\end{eqnarray*}
Technically, this query is not a purely conjunctive query because the \WHERE\ clause involves the less-than predicate $<$.
However, the machinery of tableaux minimization can still be used.
%% , and one of the advantages of our Coq development is that users are free to write arbitrary Coq expressions in where clauses, and Coq tactics can be used to reason about such where clauses.

Depending on the underlying access patterns, or the whims of a database administrator, an RDBMS might transparently index \relation{People} by creating another relation \relation{Children}, such that the following two constraints hold:
\begin{eqnarray*}
C_1 & := & \FORALL (p \IN \relation{People}) \\
 & & \WHERE p.{\sf age} < 21 \\
 & & \EXISTS (c \IN \relation{Children}) \\
 & & \WHERE p.{\sf name} = c.{\sf name} \wedge p.{\sf age} = c.{\sf age}  \\
 & & \\
  C_2 & := & \FORALL (c \IN \relation{Children}) \\
  & & \WHERE \\
 & & \EXISTS (p \IN \relation{People}) \\
 & & \WHERE p.{\sf name} = c.{\sf name} \wedge p.{\sf age} = c.{\sf age}
\end{eqnarray*}

In order to use this new index, queries written against \relation{People} must be rewritten to use \relation{Children}.
Tableaux minimization provides an automated mechanism to do so.

Let $C = \{ C_1, C_2\}$.  First, we find the universal plan $U = chase(C, Q_1)$.  We begin by chase stepping $Q$ with $C_1$.  The identity substitution is a homomorphism $front(C_1) \to Q_1$, because $p.{\sf age} < 21$ is entailed by $p.{\sf age} > 16 \wedge p.{\sf age} < 18$; thus we chase step to:
\begin{normalsize}
\begin{eqnarray*}
U & := & \FOR (p \IN Person) \ (c \IN \relation{Children}) \\
 & & \WHERE p.{\sf age} > 16 \wedge p.{\sf age} < 18 \wedge \\
 & & \ \ \ \ \ \ \ \ \ \ \ \ p.{\sf name} = c.{\sf name} \wedge p.{\sf age} = c.{\sf age} \\
 & & \RETURN p.{\sf name}
\end{eqnarray*}
\end{normalsize}
and we find that $U \cong step(C_1, U)$, so no further chase steps using $C_1$ are possible.  Now we chase step $U$ using $C_2$, and we find that $U \cong step(C_2, U)$, so no further chase steps with $C_2$ are possible.  Hence we have computed the universal plan $U = chase(C,Q_1)$.

Next, we minimize the universal plan by removing the \relation{Person} generator (note that to do so we must replace each occurrence of $p$ with some other well-typed variable, in this case $c$):
\begin{eqnarray*}
Q_2 & := & \FOR (c \IN \relation{Children}) \\
 & & \WHERE c.{\sf age} > 16 \wedge c.{\sf age} < 18 \\
  & & \RETURN c.{\sf name}
\end{eqnarray*}
We now ``back-chase'' $Q_2$ with $C$.  We can take no chase steps with $C_1$, because there is no substitution $h$ that makes $(h(p) \IN Person)$ equal to $(c \IN \relation{Children})$.  We can chase step with $C_2$ using the identity substitution to obtain:

\begin{eqnarray*}
Q_2' & := & \FOR (c \IN \relation{Children}) \ (p \IN Person) \\
 & & \WHERE c.{\sf age} > 16 \wedge c.{\sf age} < 18 \wedge \\
 & & \ \ \ \ \ \ \ \ \ \ \ \   p.{\sf name} = c.{\sf name} \wedge p.{\sf age} = c.{\sf age}\\
  & & \RETURN c.{\sf name}
\end{eqnarray*}

At this point, no further steps with $C_1$ or $C_2$ are possible.
Hence we have computed $Q_2' = chase(C, Q_2)$.
Recall that our goal is to check that $C \vdash Q_1 \cong Q_2$, which we do by checking $U = chase(C, Q_1) \leftrightarrow chase(C, Q_2) = Q_2'$; i.e., by checking $U \leftrightarrow Q_2'$.
It's easy to prove that $U$ and $Q_2'$ are homomorphically equivalent under the substitution $p \mapsto c, c \mapsto p$, which concludes the optimization.

%
%We check that $C \vdash Q^\prime \cong U$ by checking $chase(C, Q') \leftrightarrow U = chase(C,Q)$.  We start by finding $chase(C, Q')$, which turns out to be $Q'$ because no chase steps can be taken: there is no substitution $h$ that makes $(h(p) \IN Person)$ equal to $(c \IN \relation{Children})$.  By the same reasoning there is no homomorphism $U \to Q^\prime$, and hence $C$ $\nvdash$ $Q^\prime \cong U$.  Indeed, there may be extra tuples in \relation{Children} that do not appear in \relation{People}.
%
%Fortunately, if our index was built correctly we know that an additional constraint holds:
%\begin{normalsize}
%\begin{eqnarray*}
%\end{eqnarray*}
%\end{normalsize}
%As such, we may chase $Q^\prime$ with $C^\prime$ (using the identity substitution) to obtain the equivalent (under $C'$):
%\begin{normalsize}
%\begin{eqnarray*}
%Q^{\prime\prime} & := & \FOR (c \IN Children) \ (p \IN Person) \\
% & & \WHERE c.{\sf age} > 16 \wedge c.{\sf age} < 18 \wedge \\
% & & \ \ \ \ \ \ \ \ \ \ \ \   p.{\sf name} = c.{\sf name} \wedge p.{\sf age} = c.{\sf age}\\
%  & & \RETURN c.{\sf name}
%\end{eqnarray*}
%\end{normalsize}
%Now we can see that the identity substitution is a homomorphism $Q^{\prime\prime} \leftrightarrow U$ (owing to the fact that $p.{\sf name} = c.{\sf name}$ and $p.{\sf age} = c.{\sf age}$), and since $chase(C',U) \cong U$, we know that $C, C' \vdash Q^{\prime\prime} \cong U$.  We established earlier that $C \vdash U \cong Q$ and that $C' \vdash Q'' \cong Q'$.  These facts allow us to conclude that $C, C' \vdash U \cong Q \cong Q' \cong Q''$.
%\vpsace{-.1in}
\section{Coq Development - Overview}

In the rest of this paper we demonstrate how to shallowly embed relational conjunctive queries into Coq and how to use dependent types and tactics to implement tableaux minimization as described in the first half of this paper.  We will continue to use the running movies example from the first half of the paper.  That query is expressed in Coq as:
\begin{coq}
Definition Movie : Type := (string $\times$ string $\times$ string).
Definition Movies : set Movie := ...

Definition title x := fst x. (* x.$\textcolor{dkgreen}{\textsf{title}}$ *)
Definition director x := fst (snd x). (* x.$\textcolor{dkgreen}{\textsf{director}}$ *)
Definition actor x := snd (snd x). (* x.$\textcolor{dkgreen}{\textsf{actor}}$ *)

Definition q : set (string $\times$ string) :=
  m1 <- Movies ; m2 <- Movies ;
  guard (m1.$\textsf{title}$ = m2.$\textsf{title}$) ;
  return (m1.$\textsf{director}$, m2.$\textsf{actor}$).
\end{coq}
Here, we define a relation \coqe{Movies} where each tuple has type \coqe{Movie}.
We also declare simple functions to access individual fields of the \coqe{Movie} type.
For consistency with the first half of the paper, we will continue to use dot notation.
The next definition defines the query, named \coqe{q}.
Our Coq query syntax is inspired by Haskell's syntax for monadic computations, and we use Coq's extensible parsing mechanism to define (optional) custom syntax for parsing query expressions.
Intuitively, \coqe{<-} means \FOR, \coqe{guard} means \WHERE, and \coqe{return} means \RETURN.
To the right of the colon is the query's type, \coqe{set (string $\times$ string)}, which represents a set that contains pairs of strings.

%While this flexible syntax may make it easier to structure large queries, there are other benefits to it that we will see in Section~\ref{sec:low-level}.
Given the above query and a representation of the functional dependency from the introduction (\coqe{title_director_ed}) which we will describe shortly, we can ask Coq to automatically construct the minimized query with the following `proof script':
\begin{coq}
Definition optimized_query:
{q$_{opt}$ : M (string $\times$ string) | title_director_ed -> q$_{opt}$ $\cong$ q}.
optimize solver.
Defined.
\end{coq}
%Again the first line declares the name of a term.
The type of \coqe{optimized_query} says that \coqe{optimized_query} is a pair of a query, \coqe{q$_{opt}$}, and a proof that \coqe{q$_{opt}$} is equivalent to \coqe{q} on instances satisfying \coqe{title_director_ed}.
%% Here $\vdash$ is an infix operation that defines when two queries are equal under some constraint.
The actual Coq term corresponding to \coqe{optimized_query} is constructed by the \coqe{optimize} tactic.  We can see the result of the optimization by asking Coq to print the first component of the pair: \\

\begin{coq}
Eval compute in (proj1_sig optimized_query).
(* = x <- Movies ; return (x.$\textcolor{dkgreen}{\textsf{director}}$, x.$\textcolor{dkgreen}{\textsf{actor}}$)
 *   : set (string $\textcolor{dkgreen}{\times}$ string)   *)
\end{coq}
%and we see that the minimized query computed by Coq is indeed correct.

%Here, ``...'' elide the reduction strategy which tells Coq which symbols to keep abstract during reduction and the \coqe{proj1_sig} simply forgets the proof component.

%Now, the the right of the colon we have a more interesting type, noteably a dependent one.
%This syntax represents a dependent pair of a query (\coqe{q$_{opt}$}) of type \coqe{M (string * string)} and a proof of the proposition to the right of the vertical bar.
%Namely, this proof carries around a witness that under the assumption that the \coqe{title_director_ed} is provable about the database, the meaning of \coqe{q$_{opt}$} is the same as the meaning of \coqe{q}.
%Since we with Coq to fill in this value for us, we end the definition with a period and use the ``tactic'' \coqe{optimize} to discharge fill in the appropriate value.
%The keyword \coqe{Defined} signals the end of the definition since \coqe{optimize} completely fills in the term.

%% \paragraph{Querying in Coq.} All of the queries described in this paper are executable within Coq, using an implementation of sets as lists.  Even on small examples (5 rows), the optimized query above performs 2x faster than the non-optimized version.
%% While a production database would likely peform considerably faster, and on considerably larger relations, the semantic optimization that we achieve in this example enables us to reduce the query's asymptotic running time \emph{in a fully verified way}.

\subsection{Queries and Constraints in Coq}

\begin{figure}[t]
\label{fig:chaseable-functor}
\begin{coq}
Class DataModel (M : Type -> Type) : Type :=
{ Mret  : forall {T}, T -> M T
; Mbind : forall {T U}, M T -> (T -> M U) -> M U
; Mzero : forall {T}, M T
; Mimpl : forall {T}, M T -> M T -> Prop
; Mprod : forall {T U}, M T -> M U -> M (T * U) :=
     fun _ _ m1 m2 => Mbind m1 (fun x => Mbind m2 (fun y => Mret (x,y)))
; Mguard : forall {T}, bool -> M T -> M T :=
     fun _ P m => if p then m else Mzero
 (* plus many axioms *)
}.

(* Queries *)
Definition query {S T: Type}
  (P : M S) (C : S -> bool) (E : S -> T) : M T :=
  Mbind P (fun x => Mguard (C x) (Mret (E x))).

(* Embedded Dependencies *)
Definition embedded_dependency {S S': Type}
  (F : M S) (Gf : S -> bool) (B : M S') (Gb : S -> S' -> bool)
:= Meq (query F Gf (fun x => x))
       (query (Mprod F B)
              (fun ab => Gf (fst ab) && Gb (fst ab) (snd ab))
              (fun x => fst x)).
\end{coq}
%; Mimpl : forall {T}, M T -> M T -> Prop

%%   (** theorems **)
%% ; Reflexive_Mimpl : forall {A}, Reflexive (@Mimpl A)
%% ; Transitive_Mimpl : forall {A}, Transitive (@Mimpl A)

%% ; Proper_Mbind_impl : forall {A B},
%%     Proper (Mimpl ==> (pointwise_relation _ Mimpl) ==> Mimpl) (@Mbind A B)
%% ; Proper_Mret_impl : forall {A},
%%     Proper (eq ==> Mimpl) (@Mret A)

%% ; Mbind_assoc : forall {A B C} (c1 : M A) (c2 : A -> M B) (c3 : B -> M C),
%%     Meq (Mbind (Mbind c1 c2) c3)
%%         (Mbind c1 (fun x => Mbind (c2 x) c3))
%% ; Mbind_Mret : forall {A B} (x : A) (c : A -> M B),
%%     Meq (Mbind (Mret x) c) (c x)
%% ; Mret_Mbind : forall {A} (c : M A),
%%     Meq (Mbind c Mret) c
%% ; Mbind_Mzero : forall {A B : Type} (x : A -> M B), Meq (Mbind Mzero x) Mzero
%% ; Mbind_ignore : forall {T U} (x : M T) (y : M U),
%%               Mimpl (Mbind x (fun _ => y)) y
%% ; Mimpl_Mzero : forall {T} (c : M T), Mimpl Mzero c

%% ; Mbind_perm : forall {T U V} (m1 : M T) (m2 : M U) (f : T -> U -> M V),
%%     Meq (Mbind m1 (fun x => Mbind m2 (f x)))
%%         (Mbind m2 (fun y => Mbind m1 (fun x => f x y)))
%% ; Mbind_dup : forall {T U} (m : M T) (f : T * T -> M U),
%%     Mimpl (Mbind m (fun x => f (x,x)))
%%           (Mbind m (fun x => Mbind m (fun y => f (x,y))))

%% ; chaseable : forall (S S' T U : Type)
%%     (P : M S) (C : S -> bool) (E : S -> T)
%%     (F : M S') (Gf : S' -> bool) (B : M U) (Gb : S' -> U -> bool) :
%%     (edc : embedded_dependency F Gf B Gb)
%%     (h : S -> S'),
%%     Mimpl (Mmap h P) F ->
%%     forall x, C x = true -> Gf (h x) = true ->
%%     Meq (query P C E)
%%         (query (Mprod P B)
%%                (fun ab => C (fst ab) && Gb (h (fst ab)) (snd ab))
%%                (fun ab => E (fst ab))).

\caption{The \coqe{DataModel} and the definitions of {\tt for}-{\tt where}-{\tt return} queries and embedded dependencies.}
\label{fig:chaseable-functor}
\end{figure}

%% \noindent\greg{Begin}
%% Our Coq development is parametric in the implementation of the underlying type of sets.  For example, we can choose \coqe{set x := list x} and implement sets as lists, or choose \coqe{set x := x -> Prop} and implement sets as Coq ``ensembles''.
%% \noindent\greg{end}

We begin by defining a type of sets.
 Coq's rich type system gives us many possible choices, including sets as lists, \coqe{set x := list x}, and sets as predicates, \coqe{set x := x -> Prop}.
In fact, we need not use sets at all: any commutative, idempotent monad with zero defines a type of collections for which the chase is sound~\cite{Popa99anequational}.
To capture this generality, we use Coq's type class mechanism to express ``chaseable monads'' (see Figure~\ref{fig:chaseable-functor}) which we will refer to using the type constructor \coqe{M}.
The operations, but not the required axioms, of our monadic interface are shown in Figure~\ref{fig:chaseable-functor}.
Intuitively, in a set monad these operations are:
%% These functors come equipped with five operations for building instances and the corresponding theorems for reasoning about them.
%% Clients will then be able to apply our automation to \emph{any} structure satisfying these laws.
%% %% This generality is why much of our Coq code refers to \coqe{M} (for monad) rather than \coqe{set}; .  We have found in practice, however, that it is often much easier to use the concrete definition of each \coqe{M} when proving theorems than to reason axiomatically about arbitrary \coqe{M}.
%% Our Coq library is parametric in an underlying type of sets, and hence we need an interface to interact with various set implementations.  Our interface is based on monads~\cite{monad}, which are a useful interface for many kinds of collections, including sets~\cite{monad}.
\begin{itemize}
\item \coqe{Mret v} is the singleton set containing only \coqe{v}.
\item \coqe{Mbind m k} is the function that unions all sets \coqe{k x} for every \coqe{x} in the set \coqe{m}.  We will write \coqe{x <- m ; k} for \coqe{Mbind m (fun x => k)} where \coqe{x} occurs free in \coqe{k}.
\item \coqe{Mzero} is the empty set.
\item \coqe{Mimpl m$_1$ m$_2$} states that \coqe{m$_1$} is a subset of \coqe{m$_2$}. We extend subset to equivalence (written \coqe{Meq m$_1$ m$_2$}) using mutual containment.
\end{itemize}
Using these as primitives, we define two additional operations.
\begin{itemize}
\item \coqe{Mprod} is the Cartesian product of two sets.  (In an arbitrary monad we require a {\it strength}~\cite{BW}.)
\item \coqe{Mguard P m} is defined as \coqe{Mzero}, if \coqe{P} is false and \coqe{m} otherwise.
\end{itemize}
On top of the data model we define {\tt for}-{\tt where}-{\tt return} queries (Figure~\ref{fig:chaseable-functor}).
In the definition, \coqe{P} represents the \FOR clause, \coqe{C} represents the \WHERE clause, and \coqe{E} represents the \RETURN clause.
Note that when using \coqe{query} \coqe{S} and \coqe{T} will be omitted since they can inferred from the other arguments (e.g., $S$ is determined by $P$).
Concretely, the movies query from the introduction is represented as:
\begin{coq}
query (Mprod Movies Movies)
      (fun ab => (fst ab).$\textsf{title}$ ?= (snd ab).$\textsf{title}$)
      (fun ab => ((fst ab).$\textsf{director}$, (snd ab).$\textsf{actor}$))
\end{coq}
%% Note that the $S$ and $T$ arguments are omitted since they can be inferred from the other arguments (e.g., $S$ is determined by $P$).
$P$ constructs the product of the \relation{Movies} relation with itself.
The \WHERE clause is a function that accepts a pair where the first element (\coqe{fst ab}) comes from the first copy of \relation{Movies} and the second (\coqe{snd ab}) comes from the second copy of \relation{Movies}; the \WHERE clause checks equality using Coq's boolean-valued equality operator which we denote with \coqe{?=}.  The \RETURN clause returns a pair of strings constructed from the pair of \relation{Movies}.

%% Using these operations we define the \coqe{Mimpl} relation, \coqe{Mimpl m$_1$ m$_2$}, which states that \coqe{m$_1$} is a subset of \coqe{m$_2$}.  The \coqe{Meq} (also written $\cong$) relation is then defined in the standard way, i.e. \coqe{Meq m$_1$ m$_2$} if and only if \coqe{Mimpl m$_1$ m$_2$} and \coqe{Mimpl m$_2$ m$_1$}.

We define embedded dependencies (Figure~\ref{fig:chaseable-functor}) using the $front = back$ definition from Section~\ref{sec:eds}.
Here, \coqe{F} is the \FOR clause of the front, \coqe{Gf} is the \WHERE clause of the front, \coqe{B} is the \EXISTS clause of the back, and \coqe{Gb} is the \WHERE clause of the back.
In this representation, the ED for the movies example is the following:
\begin{coq}
Definition title_implies_director : Prop :=
  embedded_dependency
    (Mprod Movies Movies)
    (fun ab => (fst ab).$\textsf{title}$ ?= (snd ab).$\textsf{title}$)
    (Mret tt)
    (fun ab _ => (fst ab).$\textsf{director}$ ?= (snd ab).$\textsf{director}$).
\end{coq}
Here, the empty \EXISTS clause is represented by a singleton set carrying the unit value \coqe{tt}.
%% The main difference is the use of the unit type (with a single element \coqe{tt}) in the \FOR clause of the back to signify the lack of generators.

%% \begin{coq}
%% \end{coq}

%% \greg{it would be good to give a concrete example here, so readers can get an idea of what they should be thinking about.}

%We use the techniques described in later sections to normalize raw Coq code into this particular form.

\paragraph{Query Normalization}
In the first half of the paper, all of our queries were normalized into a single \FOR clause, a single \WHERE clause, and a single \RETURN clause.  But in a language-integrated query system, we can relax this requirement.  For example, we could introduce an additional guard condition after binding \coqe{m1}.
\begin{coq}
Definition q_LOR : set (string $\times$ string) :=
  m1 <- Movies ;
  guard (m1.$\textsf{title}$ ?= ``Lord of the Rings'') ;
  m2 <- Movies ;
  guard (m1.$\textsf{title}$ ?= m2.$\textsf{title}$ ) ;
  return (m1.$\textsf{director}$, m2.$\textsf{actor}$).
\end{coq}
As is well-known~\cite{monad}, monadic computations such as relational conjunctive queries can always be normalized into the flat form used in the first half of this paper, and our optimizer performs this normalization (in a fully verified way) during optimization.

\subsection{Background: Tactic-Based Programming}
\label{sec:tactic-based}

The core of the Coq Proof Assistant is a pure and dependently-typed functional programming language, called Gallina.  In addition to Gallina, Coq also includes a Turing-complete, untyped ``tactic'' language, called \ltac{}, which can be used to construct Gallina terms in a semi-imperative style.
\ltac{} is a meta-language for Gallina in much the same way that macros are a meta-language for C++.
\ltac{} tactics generate Gallina terms that are checked by the Coq kernel in the same way that hand-written terms are checked.

{\bf Remark.}
Our query optimizer is fully automated: to use it, a user simply invokes a tactic called {\tt optimize}.
By the completeness theorem for the chase~\cite{Popa99anequational}, {\tt optimize} will find the minimal equivalent query or diverge when none exists.  The details of our Coq implementation that we now discuss describe the design decisions we made when building our optimizer are not necessary for clients of our optimizer to understand.
However, many of the techniques are general-purpose and can be re-used in other developments which use tactics for similar purposes.

\paragraph{The \ltac{} Programming Model}
In this paper, we use \ltac{} to optimize queries in a similar way to Fiat's use of \ltac{} to optimize programs~\cite{DBLP:conf/popl/DelawarePGC15}.
This use of \ltac{} is somewhat unorthodox; the standard use of \ltac{} is to prove theorems via ``proof scripts'' that express how proofs should be built.
Indeed, the operational semantics of \ltac{} are tailored to fit this standard use of \ltac{}, rather than to fit our use of \ltac{} as a query optimizer.
For example, every \ltac{} tactic runs against a particular ``proof obligation'' (a.k.a goal), such as \coqe{x + y = y + x}.
In general, these goals are fully determined and it is simply the tactic's job to find a suitable proof.
In order to use \ltac{} to optimize queries we will run our tactics on partially specified goals which specify a constraint on a ``unification variable'' that the tactic will seek to fill in while constructing a proof.
The values of these unification variables will be the optimized queries.

We will demonstrate our use of \ltac{} on a simple goal, where we want to instantiate the unification variable \coqe{?n} with an optimized version of the query on the right:
\begin{coq}
Meq ?n (x <- Movie ; y <- Movie ; ret x)
\end{coq}
We think of the form of this goal as a ``calling convention'' of sorts for the optimization.
Here, \coqe{?n} is the unification variable that we are trying to fill in and the constraint is that the query we pick must be equivalent to the query `\coqe{x <- Movie ; y <- Movie ; ret x}'.
We could simply pick \coqe{?n} by reflexivity to be the full query, but that would require the database execution engine to do an unnecessary join.
Instead, we will optimize the query on the right by appealing to various transformations that are provably correct.

\paragraph{Optimization via Rewriting}
One simple way to perform optimization is through rewriting.
For example, the following lemma expresses that queries are idempotent (note that the second generator is not used in the rest of the query):
\begin{coq}
Lemma Mbind_dedup : forall {T U} (m : M T) (k : T -> M U),
  Meq (Mbind m k) (Mbind m (fun x => Mbind m (fun _ => k x))).
Proof. ... Qed.
\end{coq}
Because \coqe{Meq} is an equivalence relation, we can use \ltac's setoid rewriter to rewrite the goal using \coqe{Mbind_dedup}.
Running \coqe{setoid_rewrite <- Mbind_dedup} on the goal above results in the new goal:
\begin{coq}
Meq ?n (x <- Movie ; Mret x)
\end{coq}
Note that this does \emph{not} solve the unification variable \coqe{?n} or the goal; only the right-hand side of the \coqe{Meq} is changed yielding a new goal to operate on.
Because no more optimization is possible, we solve the goal by appealing to the reflexivity of \coqe{Meq} which implicitly instantiates \coqe{?n} with `\coqe{x <- Movie ; Mret x}'.
%% we can pick \coqe{?n} to be \coqe{x <- Movie ; ret x} which we witness by the reflexivity of \coqe{Meq}.
%% Doing this both picks a , and solves the goal.
Note that the optimized query no longer requires the extraneous join.

\paragraph{Optimization via Applying Lemmas}
Rewriting is a useful reasoning technique because the rewriting machinery is often able to automatically construct the proofs necessary to justify the manipulation deep within the term.
However, because the above reasoning is occurring at the top level of the goal, we could also use \ltac's more primitive \coqe{eapply} tactic with \coqe{Mbind_dedup} to yield a proof in a single step.
While both rewriting and applying lemmas are useful for manipulation, we find that most interesting optimization is better phrased using application because it is more predicable.
In large goals, rewriting could occur anywhere within the term, whereas lemma application will only apply at the top level.

In building up larger automation procedures using \coqe{eapply}, we use lemmas expressed so that their conclusion matches the current goal and their premises express new constraints that we will pass to other tactics.
Consider the following (slightly contrived) example:
\begin{coq}
Meq ?n (Mprod a b)
\end{coq}
If we wish to optimize \coqe{a} and \coqe{b} independently then we can \coqe{eapply} the following lemma:
\begin{coq}
Lemma opt_plus : forall {T U} (m1 m1' : M T) (m2 m2' : M U),
  Meq m1' m1 ->
  Meq m2' m2 ->
  Meq (Mprod m1' m2') (Mprod m1 m2).
\end{coq}
When we \coqe{eapply} this lemma to the above goal, we expect \coqe{m1} and \coqe{m2} to be completely determined while \coqe{m1'} and \coqe{m2'} to be new unification variables.
%Because the tactic will introduce new unification variables, we use the \coqe{eapply} variant of the \coqe{apply} tactic.
Concretely, when we run \coqe{eapply opt_plus} on the above goal, we get the following two sub-goals:
\begin{center}
\begin{tabular}{ccc}
\lstinline!Meq ?m1' m1! & \qquad & \lstinline!Meq ?m2' m2! \\
\end{tabular}
\end{center}
while at the same time instantiating \coqe{?n} with \coqe{Mprod ?m1' ?m2'}.
As our automation continues to fill in \coqe{?m1'} and \coqe{?m2'} (for example, during further optimization), \coqe{?n} will be automatically updated to reflect these changes.

%% The discussion in this section illustrates the large design space that must be explored when building the query optimizer itself.

%% The ``standard'' approach to developing an algorithm such as the chase in Coq would be to program it in Gallina, Coq's pure functional programming language.
%% By the Curry-Howard correspondence~\cite{} this programming language is also a logic where propositions are types and programs are proofs.
%% A more complete description of the correspondence can be found in a variety of sources~\cite{}.
%% We will return to this approach in more detail in Section~\ref{sec:??}, in this work we develop our optimization in a different way.

%% In addition to Gallina, Coq also comes with another programming language, \ltac.
%% Unlike Gallina, which is strongly-typed and pure, \ltac\ is untyped and partial.
%% \ltac\ is a proof scripting language that is used to construct proof terms in an imperative style.
%% In this section we will discuss the key aspects of \ltac\ using a simple example for implementing addition.
%% \ltac\ is centered around manipulating Gallina terms.

%% The most common use of \ltac\ is to construct proofs of Gallina propositions.
%% For example, proving that \coqe{x + y = y + x} can be done by using the Gallina theorem that proves the commutativity of addition, i.e.
%% \begin{coq}
%% Theorem plus_comm : forall (n m : nat), n + m = m + n.
%% Proof. ... Qed.
%% \end{coq}
%% In \ltac{} we can apply this theorem using the \coqe{apply} tactic.
%% \begin{coq}
%% (* x, y : nat
%%  * ==============
%%  * x + y = y + x
%%  *)
%% apply plus_comm. (* => goal solved *)
%% \end{coq}
%% The implementation of \coqe{apply} unifies \coqe{x + y = y + x} with \coqe{?n + ?m = ?m + ?n} where the question mark variables (e.g. \coqe{?n}) are flexible unification variables that the unification can pick values for.
%% To solve this problem, the unification algorithm picks \coqe{?n $\mapsto$ x} and \coqe{?m $\mapsto$ y}.

%% In addition to \coqe{apply}, \ltac{} also provides \coqe{rewrite} to perform rewriting by user-defined relations.
%% Rewriting can be considerably more flexible than direct function application but works in much the same way.
%% Generalizing the goal above to \coqe{(x + y) * 3 = (y + x) * 3} makes \coqe{plus_comm} no longer immediately apply, but since equality is a transitive relation we can use \ltac{} to rewrite in the conclusion and then solve the goal by the reflexivity of equality.
%% \begin{coq}
%% (* x, y : nat
%%  * ==============
%%  * (x + y) * 3 = (y + x) * 3
%%  *)
%% rewrite plus_comm.
%% (* x, y : nat
%%  * ==============
%%  * (y + x) * 3 = (y + x) * 3
%%  *)
%% reflexivity. (* => goal solved *)
%% \end{coq}

%% In addition to solving concrete goals such as the ones above, \ltac{} is also able to manipulate unification variables directly.
%% Take the goal \coqe{?x = 3} for example.
%% The \coqe{reflexivity} tactic will solve this goal by instantiating \coqe{?x} with the value 3.
%% These unification variables are commonly used when proving existential quantifiers.
%% For example, in the following goal we use \coqe{eexists} to introduce a new unification varible for \coqe{x} and then solve the resulting equation with \coqe{reflexivity}.
%% \begin{coq}
%% (* ==================
%%  * exists x : nat , x = 3
%%  *)
%% $\texttt{\textcolor{dkblue}{eexists}}$.
%% (* ==================
%%  * ?x = 3
%%  *)
%% reflexivity. (* => goal solved *)
%% \end{coq}

%% \greg{this is a very unfocused section}

\begin{comment}
\subsection{Normalization}
\label{sec:normalization}

The first step of our optimization pipeline normalizes queries into the {\tt for}...{\tt where}...{\tt return} structure presented in Section~\ref{sec:queries}.
In Coq, we can define this structure as a function that takes the three pieces of the query and stitches them together into an instance.
The definition is the following:
\begin{coq}
Definition query {S T: Type}
  (P : M S) (C : S -> bool) (E : S -> T) : M T :=
  Mbind P (fun x => Mguard (C x) (Mret (E x))).
\end{coq}
Here, \coqe{P} represents the \FOR clause that generates the tableau,
\coqe{C} represents the conditional \WHERE clause, and \coqe{E} represents the \RETURN clause.
In our representation, we use \coqe{Mprod} to construct a relational cross-product of two ``binds.''
Thus, the normalized form of our movies query is the following:
\begin{coq}
query (Mprod Movies Movies)
      (fun x => (fst x).(title) = (snd x).(title))
      (fun x => ((fst x).(director), (snd x).(actor)))
\end{coq}

%% In addition to generating this term, in order to fit into our fully-verified pipeline we must also generate a proof that guarantees that the transformation yields an equivalent instance.
%% Within the shallow encoding we can perform this normalization by incrementally applying proven theorems that witness the soundness of individual transformations.

%% The process begins by asking Coq to generate a unification variable representing the final answer and using it to witness the answer.
The phase starts on the goal \coqe{Meq ?1 q}.
The first theorem that we apply is an administrative theorem to massage the goal into the form expected by the rest of the normalization process.
%%  which we solve using a set of theorems crafted to specifically match on goals of this form.
\begin{coq}
Theorem prep_for_normal : forall {T} (q q' : M T),
  Meq q' (Mbind (query (Mret tt) (fun _ => true) (fun x => x))
                       (fun _ => q)) ->
  Meq q' q.
Proof. ... Qed.
\end{coq}
When applied to the initial goal, the first theorem (\coqe{prep_for_normal}), produces a new goal where left-hand side is unchanged and the right-hand side of the equivalence now binds the empty query and then returns \coqe{q}.

While not particularly insightful from a proof-theoretic point of view, this transformation lays the groundwork for the remaining theorems to move components of \coqe{q} into the query.
For example, \coqe{normal_pull_plus} moves a bind from the instance and inserts it into the query.
\begin{coq}
Lemma normal_pull_plus
: forall {T U V W : Type} (qb : M T) (qg : T -> bool) (qr : T -> U) x (y : _ -> _ -> M V),
  Meq q'
      (Mbind (query (Mprod qb x) (fun x => qg (fst x)) (fun x => (qr (fst x), snd x)))
             (fun val : U * W => y (fst val) (snd val))).
  Meq q'
      (Mbind (query qb qg qr)
             (fun val : U => Mbind x (y val))).
Proof. ... Qed.
\end{coq}
Concretely, applying \coqe{normal_pull_plus} to the goal produced by \coqe{prep_for_normal} produces a new goal that has shrunk the size of the query to normalize (by ``removing'' a bind) and placing it in the \FOR clause of the \coqe{query} definition.
\begin{coq}
Meq ?m (x <- query (Mprod (Mret tt) Movie) (fun _ => true) (fun x => x) ;
        y <- Movie ; guard ((snd x).(title) = y.(title)) ;
        Mret ((snd x).(title), y.(actor)))
\end{coq}

It is important to notice that applying this theorem will happily lift non-atomic binds into the \FOR clause.
For example, if applied directly to the goal \coqe{x <- (y <- Movies ; guard (y = 1) ; return y) ; return x}, \coqe{normal_pull_plus} will construct the following term that is not in the desired normal form.
\begin{coq}
x <- query (Mprod (Mret tt) (y <- Movies ; guard (y = 1)))
           (fun _ => true) Mret ;
return x
\end{coq}
To avoid this, we first perform a pre-processing phase that rewrites using a collection of lemmas which flatten nested binds and pushes guard expressions downwards toward the return.

Beyond lifting binds, the normalization process also includes lemmas for lifting \WHERE  and \RETURN clauses into the query.
The later of these is slightly different because lifting the final \RETURN clause into the query marks the end of normalization.
Therefore, unlike the other lemmas, \coqe{normal_pull_ret} does not have a premise.
\begin{coq}
Lemma normal_pull_ret
: forall {T U V : Type} (qb : M T) qg (qr : T -> U) (y : _ -> V),
  Meq (Mbind (query qb qg qr)
             (fun val : U => Mret (y val)))
      (query qb qg (fun x => y (qr x))).
Proof. ... Qed.
\end{coq}

%% To handle nested structures such as  we preface the normalization step by rewriting with lemmas that flatten the query and push binds downward.
%% For example \coqe{Mbind_assoc} (shown below) converts the nested query above into \coqe{y <- Movies ; x <- return y ; return x}.
%% \begin{coq}
%% Lemma Mbind_assoc
%% : forall (A B C : Type) (c1 : M A) (c2 : A -> M B)
%%          (c3 : B -> M C),
%%   Meq (Mbind (Mbind c1 c2) c3)
%%       (Mbind c1 (fun x : A => Mbind (c2 x) c3))
%% \end{coq}

The final result of normalization for the movies query is exactly the query presented in Section~\ref{sec:example}.
In our syntax:
\begin{coq}
query (Mprod Movies Movies)
      (fun x => (fst x).(title) = (snd x).(title))
      (fun x => ((fst x).(director), (snd x).(actor)))
\end{coq}
\end{comment}

\subsection{Implementing the Chase as a Tactic}
\label{sec:ltac-chase}

Tableaux minimization makes heavy use of the chase, which we have implemented as an \ltac{} tactic.
Implementing the chase as a tactic (as opposed to a Gallina function) has two critical advantages.
First, {\it Coq tactics can invoke other Coq tactics.}
As we saw in the indexing example in the first half of the paper, running the chase can involve reasoning over predicates such as $<$ that appear in \WHERE clauses.
%% If we were implementing the chase as a Gallina function (such as in~\cite{coqdb}), we would be forced to choose some predetermined set of predicates and implement, as part of the Gallina program implementing the chase, a reasoning procedure for terms involving these predicates.
By implementing the chase as a tactic, we can appeal to Coq's \coqe{omega} tactic for reasoning about $<$, for example.
Moreover, the particular predicates and associated tactics need not be determined in advance; users are free to write arbitrary Gallina code in \WHERE clauses, and to build custom tactics for reasoning about this code.
The second reason is that we are not obligated to justify termination of \ltac{} scripts.
The chase may never terminate, but all Gallina functions must terminate, so any Gallina (but not \ltac{}) implementation of the chase must be explicitly bounded by some number of chase steps.
(If desired, our \ltac{} version of the chase can also be given an explicit bound).

Our chase tactic applies to goals that are contingent on EDs.
These EDs are expressed using the following goal structure:
\begin{coq}
title_director ->
Meq ?q
    (m1 <- Movies ; m2 <- Movies ;
     guard (m1.$\textsf{title}$ = m2.$\textsf{title}$) ;
     return (m1.$\textsf{director}$, m2.$\textsf{actor}$))
\end{coq}

Recall from the first half of the paper that running the chase required several steps: choosing an ED, finding a candidate substitution, checking that a candidate is in fact a homomorphism, taking a chase step, and then checking for convergence to a fixed point.
In the rest of this section we describe how to implement each of these in \ltac{}.
In the process we distill reusable patterns for building optimization procedures in \ltac.

%% Chasing a query requires solving 4 problems in order:
%% \begin{enumerate}
%% \item the tactic must find an ED to chase with (Section~\ref{sec:traverse-ed}).
%% \item the tactic must find a homomorphism from the front of the ED to the binders of the query (Section~\ref{sec:morphism-find}).
%% \item the tactic must check that the \WHERE clause of the ED implies the \WHERE clause of the query under the mapping that our tactic found in the previous step (Section~\ref{sec:side-condition}).
%% \item the tactic must ensure that chasing will introduce new information (Section~\ref{sec:progress}); i.e., the tactic must terminate when a fixed point is reached.
%% \end{enumerate}

%% We will now examine each of these steps in turn.

\subsubsection{Finding an Embedded Dependency}
\label{sec:traverse-ed}

In the movies example, there is only one embedded dependency to chase with, but in more complex examples such as the indexing example there are multiple EDs.
To find an appropriate ED, we can exhaustively consider all of the user-supplied EDs.
Demonstrating this exhaustive enumeration serves as a useful primer for the more complicated next step of finding a homomorphism.
Indeed, this section describes a \emph{general-purpose design pattern} for doing exhaustive enumeration using \ltac{}.

When there are multiple candidate EDs (e.g., \coqe{A}, \coqe{B}, and \coqe{C}) that we wish to use to optimize the query \coqe{q}, the goal posed to the optimizer tactic has the following form:
\begin{coq}
A /\ B /\ C -> Meq ?q q
\end{coq}

Two lemmas allow us to exhaustively consider these EDs.
\begin{coq}
Lemma ed_pick_left : forall {A B C}, (A -> C) -> (A /\ B -> C).
Proof. tauto. Qed.
Lemma ed_pick_right : forall {A B C}, (B -> C) -> (A /\ B -> C).
Proof. tauto. Qed.
\end{coq}
Both of these lemmas are trivial tautologies that can be proven by the \coqe{tauto} tactic; however, we have found that separating the task of crafting each lemma from its proof dramatically improves our ability to evolve our optimizer.
%% Therefore, all of the lemmas that we show in this section have trivial proofs that often appeal to other lemmas which are phrased to simplify the proving process.}.
\coqe{ed_pick_left} focuses on the left-hand-side of a conjunction while \coqe{ed_pick_right} focuses on the right-hand-side.
When only a single ED remains, we can process it, in this case chasing with it, which we will discuss in more detail in the next section.

We use these lemmas in the following backtracking search tactic which uses \ltac's \coqe{+} combinator\footnote{The + combinator is new in Coq 8.5. Before Coq 8.5, performing this search modularly required writing tactics in continuation-passing style.} in the following recursive tactic.
\begin{coq}
Ltac ed_search :=
  lazymatch goal with
  | $\vdash$ _ /\ _ -> _ =>
    (  simple eapply ed_pick_left
     + simple eapply ed_pick_right) ; ed_search
  | $\vdash$ _ -> _ => idtac
  end.
\end{coq}
Here, `\coqe{lazymatch goal with}' performs syntactic matching of the goal against the candidate patterns selecting the first that matches.
The first branch chooses either the left- or right-side using the above lemmas and then continues the search by recursively calling \coqe{ed_search}.
The second branch is the default case which finishes the search when the premise does not contain a conjunction using \coqe{idtac} which is a no-op.
It is important to note that the semantics of \coqe{+} causes deep backtracking.
For example, \coqe{(a + b) ; c} is equivalent to \coqe{(a; c) + (b; c)}.
This behavior allows us to chain \coqe{ed_search} with the chase (described in the next section) and backtrack to consider a new ED if the chase step fails to make progress with the first ED.

%% The previous sections havee shown how to chase a single embedded dependency, chasing a collection of EDs is not much more difficult.
%% Embedded dependencies are communicated via premises on the entailment, so all that remains is to non-deterministically select the appropriate ED.
%% The approach here is the same as when we non-deterministically found the morphism; we simply perform an exhaustive backtracking search trying to chase each one until we succeed.
%% For convenience we require that multiple EDs are combined with $\wedge$.

\subsubsection{Running the Chase Step}
\label{sec:chase-step}

Once we have isolated a single ED to chase, we need to apply a theorem witnessing the soundness of the chase.
This soundness theorem is phrased to match the current goal.
%% For the chase, this prepping step is expressed by the soundness of the The core of this prepping step is to apply the following theorem which expresses the soundness of the chase.
\begin{lstlisting}[numbers=left]
Theorem chase_sound {S S' T U}
  (P : M S) (C : S -> bool) (E : S -> T)
  (F : M S') (Gf : S' -> bool) (B : M U) (Gb : S' -> U -> bool)
: forall (h : S -> S'),
    Mimpl (Mmap h P) F ->
    (forall x, C x = true -> Gf (h x) = true) ->
    embedded_dependency F Gf B Gb ->
    Meq (query P C E)
        (query (Mprod P B)
               (fun ab : S $\times$ U => C (fst ab) &&
                                  Gb (h (fst ab)) (snd ab))
               (fun ab => E (fst ab))).
\end{lstlisting}
In this theorem, lines 8-12 will unify with the goal.
Line 8 and the first argument to \coqe{Meq} extract the values of the goal and the second-argument to \coqe{Meq} instantiates the unification variable with the optimized query.
Line 4 represents the substitution ($h$).
Line 5 expresses the requirement that $h$ maps variables in the query ($P$) to the variables in the front of the ED ($F$).
Line 6 represents the requirement on the \WHERE clause, namely that the \WHERE clause of the query ($C$) implies the \WHERE clause of the front of the ED ($Gf$).
%% In this theorem, all of the variables in the first three lines are known once we know the ED and the query that we are chasing.  The next three lines (starting with the colon) express the tableau homomorphism from the front of the ED to the query.
In the next two sections we describe the representation of these pieces in more detail and explain how we compute this homomorphism.

\paragraph{Finding a Homomorphism}
In the definition of a chase step in Section~\ref{sec:chase} a homomorphism is a map from variables bound in the front of the embedded dependency to the variables bound in the \FOR clause of the query.
%% In our movies query example this corresponds to a mapping that assigns values in $\{x,y\}$ to values in $\{m_1,m_2\}$.
Since our queries are Gallina terms, explicitly referencing binders by name can be quite difficult, and is not very extensible (i.e., not invariant under $\alpha$-renaming).
Instead, we encode our mapping of binders as a function between the types being bound.
%Thinking of the environment as a tuple, where each variable corresponds to a different component this shift in view is analagous to determining a function from $(\tau_0 \times .. \times \tau_i)$ to $(\sigma_0 \times .. \times \sigma_j)$, rather than a function from the projections, i.e. from $(\tau_0\times..\times\tau_i \rightarrow \tau_0) \times .. \times (\tau_0\times..\times\tau_i\rightarrow \tau_i)$ to $(\sigma_0\times..\times\sigma_j \rightarrow \sigma_0) \times .. \times (\sigma_0\times..\times\sigma_i\rightarrow \sigma_i)$.
In this example, the type of the \FOR clause of the query is \coqe{Movie $\times$ Movie} and the type of the \FORALL clause is also \coqe{Movie $\times$ Movie}, so we are looking for a function \coqe{h : Movie $\times$ Movie -> Movie $\times$ Movie}.
Looking at the query and the ED, there are four choices:
\[\begin{array}{lcl}
\footnotesize\texttt{h x = (fst x, fst x)} & & \footnotesize\texttt{h x = (snd x, fst x)} \\
\footnotesize\texttt{h x = (fst x, snd x)} & & \footnotesize\texttt{h x = (snd x, snd x)} \\
\end{array}
\]
%Where \coqe{fst} extracts the first element of the pair and \coqe{snd} extracts the second.

We are going to construct these functions incrementally using theorems that represent individual steps of reasoning.
The search is much the same as the search for isolating a particular ED, but with two main differences.
First, we must now find a binder for each binder in the front of the ED, i.e. we must perform a new search for each binder.
Second, while doing this, \emph{we must explicitly construct the substitution \coqe{h}} so that we can use it when checking the remainder of the homomorphism (line 6 in \coqe{chase_sound}).
When searching for the homomorphism, the goal will have the following form, where \coqe{P} is the \FOR clause of the query and \coqe{F} is the \FORALL clause in the ED:
\begin{coq}
Mimpl (Mmap ?h P) F
\end{coq}
In the above %% \coqe{Mimpl}, is defined as subset containment (or rather, the analog of subset containment for arbitrary monads), and
\coqe{Mmap} expresses the application of the substitution \coqe{?h}, i.e. \coqe{Mmap f m = x <- m ; Mret (f x)}.
Note that here we are proving an inclusion (\coqe{Mimpl}) rather than an equality.
This is essential because fields not used to construct $F$ from $P$ could be empty which would make the left-hand side empty while the right-hand side would be non-empty.

When solving this goal, the first task is to break \coqe{F} down into atomic units that correspond to the binders.
The \coqe{pick_split} lemma applies when \coqe{F} is formed from a \coqe{Mprod}:
\begin{coq}
Lemma pick_split
: forall {T U U' : Type} (m : M T) (u : M U) (u' : M U') f g,
  Mimpl (Mmap f m) u ->
  Mimpl (Mmap g m) u' ->
  Mimpl (Mmap (fun x => (f x, g x)) m) (Mprod u u').
Proof. ... Qed.
\end{coq}
This lemma states that we can find a morphism from \coqe{m} to \coqe{Mprod u u'} if we can find a morphism from \coqe{m} to \coqe{u} and from \coqe{m} to \coqe{u'}.
Note that in addition to breaking down the morphism by decomposing it into \coqe{f} and \coqe{g}, the left-hand side of the implication in the conclusion also shows how to use \coqe{f} and \coqe{g} to construct the final homomorphism.

\begin{figure}
\begin{coq}
Lemma pick_left
: forall {T' U' V} (f' : U' -> V) (x : M V) (y : M T') (k' : M U'),
  Mimpl (Mmap f' k') x ->
  Mimpl (Mmap (fun x => f' (fst x)) (Mprod k' y)) x.
Proof. ... Qed.
Lemma pick_right
: forall {T' U' V} (f' : U' -> V) (x : M V) (y : M T') (k' : M U'),
  Mimpl (Mmap f' k') x ->
  Mimpl (Mmap (fun x => f' (snd x)) (Mprod y k')) x.
Proof. ... Qed.
Lemma pick_here
: forall {T} (x : M T), Mimpl (Mmap (fun x => x) x) x.
Proof. ... Qed.
\end{coq}
\caption{Manipulation lemmas for picking generators.}
\label{fig:manipulation-lemmas}
\end{figure}

Repeatedly applying \coqe{pick_split} will eventually break the \FORALL clause down into atomic elements that we can match up with the query.
This matching is essentially the same as the ED search procedure described in Section~\ref{sec:traverse-ed} except that, as above, we must record the way to reconstruct the \coqe{h} function.
The three lemmas in Figure~\ref{fig:manipulation-lemmas} express (and justify) these manipulations.
\coqe{pick_left} decides to use only the left-hand side of the \coqe{Mprod k' y} to determine \coqe{x}, \coqe{pick_right} is analogous for the right-hand side.
Finally, \coqe{pick_here} applies when the value being searched for is exactly the value being bound in which case it can pick the value directly.

%% We combine these proofs into into a search using \ltac's backtracking \coqe{+} operator.
%% The core of the procedure is the following:
%% \begin{coq}
%% Ltac find_bind_morphism :=
%%   lazymatch goal with
%%   | |- Mimpl (Mmap _ _) (Mprod _ _) =>
%%       (eapply pick_split)
%%   | |- Mimpl _ _  =>
%%       (simple eapply pick_here)
%%     + (simple eapply pick_left)
%%     + (simple eapply pick_right)
%%   end ; find_bind_morphism.
%% \end{coq}
%% Applied to our simple example this essentially amounts to:
%% \begin{coq}
%% eapply pick_split ;
%%   (simple eapply pick_here + simple eapply pick_left + simple eapply pick_right) ;
%%   (simple eapply pick_here + simple eapply pick_left + simple eapply pick_right)
%% \end{coq}
%% The first application splits the morphism into two parts, and the semi-colon runs the remainder of the tactic on the two independent goals.
%% The plus operator effectively allows choices to be made independently and allows backtracking to encode the non-deterministic choice.
%% In this case, along the first goal we \coqe{pick_left} and then \coqe{pick_here} and along the second we \coqe{pick_right} and then \coqe{pick_here}.

The search completes by solving the goal and, by side-effect, instantiating \coqe{?h} with a function representing the substitution.
%% with each of the four candidate substitutions written above, and the next step is to attempt to solve the side condition (i.e., that a candidate is in fact a homomorphism).

\paragraph{Proving the Side-Conditions}
With a candidate substitution in hand, the next step is to discharge the side condition which ensures that the \WHERE clause of the embedded dependency implies the \WHERE clause of the query.
In our movies example, this amounts to the following: %%  (we will write, for example, \coqe{z.$\textsf{title}$} instead of \coqe{title z} in an effort to make things more readable):
\begin{coq}
forall x : Movie $\times$ Movie, (fst x).$\textsf{title}$ ?= (snd x).$\textsf{title}$ = true
     -> (fst (h x)).$\textsf{title}$ ?= (snd (h x)).$\textsf{title}$
\end{coq}

Once we get to this step, \coqe{h} is exactly one of the substitutions constructed by the previous step.
When we plug in a particular substitution, i.e. `\coqe{h x = (fst x, snd x)}' (recall that we are enumerating all of the potential homomorphisms), and simplify, we are left to solve the following goal: %(here, \coqe{=} is the function that computes whether the two arguments are equal, while the naked \coqe{=} is the equality relation itself):
\begin{coq}
forall x : Movie $\times$ Movie, (fst x).$\textsf{title}$ ?= (snd x).$\textsf{title}$ = true
     -> (fst x).$\textsf{title}$ ?= (snd x).$\textsf{title}$ = true
\end{coq}

While this goal is true simply by equational reasoning, in general these side conditions can require more complex reasoning.
For example, in the indexing example from Section~\ref{sec:minimize} we must prove the following implication which relies on arithmetic reasoning:
\begin{coq}
forall p, p.age > 16 && p.age < 18 = true -> p.age < 21 = true
\end{coq}
%% Here, equational reasoning alone is insufficient to solve the goal.
We can use Coq's \coqe{omega} tactic to discharge arithmetic goals such as the one above.
In order to facilitate this sort of domain-specific reasoning, we parameterize the chase tactic by a tactic to discharge side-conditions.
%% This step is parametrized by the solver the should be used to solve the goal allowing the user to inject his or her own automation to reason about domain-specific predicates such as arithmetic or case-insensitive string matching.
%% In addition, we can develop our own theorems and \ltac{} for automating different domains, for example the length of strings, the case of characters, or even complex arithmetic on floating point numbers.

\paragraph{Ensuring Progress}
After the side-condition is checked, the final step is to ensure that this chase step makes progress by adding \emph{new} structure to the query.
Semantically, we express this by ensuring that the new query is not homomorphically equivalent to the old one.
Checking homomorphic equivalence between queries is essentially the same as what we have done up to this point except that we must also show that the morphism preserves the \RETURN clause of the query under the assumptions in the \WHERE clause.
%% This check simply requires that we have a tactic that can find an isomorphism between queries, which is essentially the same as what we have done up until this point, except that we also need to prove that the \RETURN clauses of the two queries are equal under the assumptions in the \WHERE clause.
The \ltac{} to check this condition is straightforward given the machinery that we developed up to this point.
The entire \ltac{} is (essentially) the following:
\begin{coq}
Ltac prove_query_morphism solver :=
  eapply check_query_morphism_apply ;
    [ find_bind_morphism
    | simpl ; solve [ solver ]
    | simpl ; solve [ solver ] ]).

Ltac prove_query_homomorphic_equal solver :=
  split; prove_query_morphism solver.
\end{coq}
Here chaining the \coqe{eapply} with \coqe{; [ a | b | c ]} allows us to specify different tactics to run on each of the individual goals produced by \coqe{eapply check_query_morphism_apply}.
Note again that these tactics are parameterized by the underlying solver (\coqe{solver}) that they will use to discharge the side-conditions.

\subsubsection{Computing the Fixed-Point}
With the ability to iterate over EDs (Section~\ref{sec:traverse-ed}) and to compute a chase step (Section~\ref{sec:chase-step}), it is simple to implement the entire chase.
%% The key detail to be aware of when constructing the full chase lies in delimiting the backtracking that the algorithm requires.
%% When we fail to chase with a particular ED, it is no longer necessary to backtrack into the application of that ED.
%% To ensure that we do not backtrack between two different chase steps, we use \ltac's \coqe{once} and \coqe{first} tacticals.
The tactic is the following:
\begin{coq}
Ltac chase solver :=
  repeat first
    [ eapply transitive_refine_conditional ;
      [ solve [ ed_search ; chase_step solver ]
      | ]
    | reflexivity ].
\end{coq}
where \coqe{transitive_refine_conditional} expresses transitivity under implication:
\begin{coq}
Lemma transitive_refine_conditional
: forall {T} (a b c : M T) (P : Prop),
   (P -> Meq b c) -> (P -> Meq a b) -> (P -> Meq a c).
Proof. ... Qed.
\end{coq}
In the \coqe{chase} tactic, \coqe{repeat} computes the transitive closure of the chase step by running the rest of the tactic until the goal is solved or the tactic fails.
\coqe{first [ a | b ]} runs \coqe{a} and, if \coqe{a} fails, runs \coqe{b}.
The first branch uses the transitivity lemma above to break the goal into two sub-goals.
For the first sub-goal, we run a single step of the chase by first finding an embedded dependency (\coqe{ed_search}) and then chasing it using \coqe{solver} to solve the side conditions (\coqe{chase_step solver}).
The \coqe{solve} ensures that the tactic completely solves the goal which guarantees that we completed the chase step.
If the goal is solved, the second goal is interpreted by the rest of the tactic due to the \coqe{repeat}.
If the chase step fails, then the second branch of the \coqe{first} runs solving the goal using \coqe{reflexivity} which picks the input query to be the output query.

%% It should be noted that using \coqe{repeat} to implement this is not the same as implementing this tactic recursively.
%% In the implementation above using \coqe{repeat} there search will never backtrack between multiple iterations.
%% While this particular tactic would never backtrack when it is invoked correctly, just using the
%% This implementation is not the same same as a recursive implementation  different then making a recursive call to \coqe{chase solver} in the second branch because the \coqe{repeat} does not backtrack between iterations, while the recursive implementation would.

%% The \coqe{transitive_refine_conditional} theorem applies transitivity speculating progress while leaving the door open to further optimization.
%% The first of the two goals (which corresponds to the chase step) is solved using the tactic on the second line.
%% Here, \coqe{solve} requires that the inner tactic completely solves the goal, and \coqe{once} prevents backtracking triggered by later pieces of the search.
%% If progress can not be made in the chase, then \coqe{solve} fails and \coqe{first} solves the goal using \coqe{reflexive_refine_conditional} which picks the current query as the result.

%Note that by developing this procedure in \ltac{} there is no way to reason about whether it terminates.

\subsection{Query Minimization}
\label{sec:minimization}

The final step in optimization is to remove redundant generators (binds) from the query.
By now, we have already discussed most of the techniques.
The high-level approach is to use incremental lemmas to iterate through the binders and attempt to drop each one by expressing a side-condition that expresses that the information in that binder can be reconstructed from the other binders.

\begin{figure}
\begin{coq}
Lemma minimize_drop
: forall {T T' V : Type} (qb : M T) (qb' : M T') qg (qr : _ -> V) f (qb'' : M T') qg'',
   Find f
-> Meq (query (Mprod qb qb') qg qr)
       (query qb' (fun y => qg (f y,y)) (fun y => qr (f y,y)))
-> Meq (query qb' (fun y => qg (f y,y)) (fun y => qr (f y,y)))
       (query qb'' (fun y => qg'' (f y,y)) (fun y => qr (f y,y)))
-> Meq (query (Mprod qb qb') qg qr)
       (query (Mmap (fun y => (f y, y)) qb'') qg'' qr).
Proof. ... Qed.

Lemma minimize_keep
: forall {T T' V : Type} (qb : M T) (qb' : M T') qg (qr : _ -> V) (qb'' : M T') qg'',
  (forall x : T,
   Meq (query qb' (fun y => qg (x,y)) (fun y => qr (x,y)))
       (query qb'' (fun y => qg'' (x,y)) (fun y => qr (x,y)))) ->
  Meq (query (Mprod qb qb') qg qr)
      (query (Mprod qb qb'') qg'' qr).
Proof. ... Qed.
\end{coq}
\caption{Minimization lemmas.}
\label{fig:minimize-lemmas}
\end{figure}

There are two core lemmas that express minimization transformations (shown in Figure~\ref{fig:minimize-lemmas}).
The first, \coqe{minimize_drop}, states that we can drop the first (left) binder if we can find a way to compute it from the right binder, i.e. a morphism from \coqe{Mmap f qb'} to \coqe{qb}.
The first premise of this theorem, \coqe{Find f}, is a dummy premise; it is trivially true.
We include it in order to simplify constructing \coqe{f} using tactics.
For example, we can easily write lemmas analogous to \coqe{pick_left}, \coqe{pick_right}, and \coqe{pick_here} looking only at the type of \coqe{f}.
The second premise, ensures that this choice of \coqe{f} respects the equivalence of the query.
To solve it, we ``back chase'' the left-hand side of the equivalence and try to determine if it is homomorphically equivalent to the right-hand side.
The second lemma, \coqe{minimize_keep}, is the fallback case. %% , which the automation will use if it can not drop the generator.
If the tactic can not find a way to re-construct the removed binder from the other binders of the query, then it cannot remove that binder.
However, it can (and should) still optimize the rest of the query.
To represent the rest of the query, we universally quantify over the values that could come from the relation and recursively optimize the rest of the query.
%% (represented by quantifying over any value from the relation and ensuring that the rest of the query is still valid).

\paragraph{Post-processing}
The primary purpose of the {\tt optimize} tactic is to minimize the number of binds in a query, but we have also added some query simplification steps to be performed after minimization.  In the movies example, the result of minimization is:
\begin{coq}
query (Mmap (fun x => (x,x)) Movie)
      (fun xy => (fst xy).$\textsf{title}$ = (snd xy).$\textsf{title}$)
      (fun xy => ((fst xy).$\textsf{director}$, (snd xy).$\textsf{actor}$))
\end{coq}
After simplification, the duplication of the bound variable is removed and the \WHERE clause is eliminated since it is testing the equality of \coqe{x.$\textsf{title}$} with itself.  The final query is the following:
\begin{coq}
query Movie (fun _ => true) (fun x => (x.$\textsf{director}$, x.$\textsf{actor}$))
\end{coq}

%% \subsection{Nested Queries}

%% One of the main benefits to using a shallow embedding of queries is that shallowly-embedded queries are naturally nested.  For example, it is simple to write a Gallina expression that flattens a nested set using union:
%% \begin{coq}
%% Definition q_nested (M: set (set string)) : set string :=
%%   m <- M;
%%   m' <- m;
%%   return m'.
%% \end{coq}
%% Tableaux minimization extends to nested relations~\cite{popa99equational}, and our \ltac{} query optimizer can be incrementally extended to handle nested relations by generalizing the lemmas from the previous sections. %\ltac{} is the ability to easily extend the syntax of queries and the effect of the optimizer simply by proving relatively simple lemmas such as the ones in the previous sections.

% the above \coqe{normal_pull_plus} is not sufficient to handle nested queries where \coqe{x} could depend on the values that were previously bound.
%Normally, supporting this type of term manipulation would be quite painful since we would need to track the number of binders that we are under or generate fresh names.
%Instead, we can foist all of that complexity back onto Coq by phrasing the appropriate lemma and using Coq's higher-order unification to solve the problem for us.
%Using this technique, the generalization of \coqe{normal_pull_plus} to handle nested relations would be the following:
%\begin{coq}
%Lemma normal_pull_dplus_ret_id
%: forall {T U V W : Type} (qb : M T) qg x (y : _ -> _ -> M V),
%  Meq (Mbind (query qb qg (fun x => x))
%             (fun val : T => Mbind (x val) (y val)))
%      (Mbind (query (Mdplus qb x) (fun x => qg (fst x)) (fun x => (fst x, snd x)))
%             (fun val : T * W => y (fst val) (snd val))).
%Proof. ... Qed.
%\end{coq}

%% \subsection{Non-Semantic Optimization}
%% \greg{not finished but could be interesting and relatively simple}

\section{Discussion of Tactic-Based Optimization}
\label{sec:discussion}

%In this paper we presented our implementation of a verifying query optimizer as an \ltac{} tactic, an approach that is similar to the work of Delaware et.al.~\cite{DBLP:conf/popl/DelawarePGC15}.
Tactic-based development has several trade-offs compared to a more traditional approach that would implement a query optimizer as a Gallina function (e.g.,~\cite{coqdb}).

\paragraph{Benefits}
The primary benefit of implementing the optimizer as a tactic is the ability to extend the optimizer with a minimal amount of work.
Much of this benefit is due to the flexibility of working indirectly on Gallina's underlying terms.
For example, we can extend the chase algorithm with support for nested relations by proving new lemmas that show how to locally manipulate terms.
A non-tactic implementation would need to adjust its concrete term representations to support the more sophisticated structure of nested queries and then update all of its algorithms (and proofs) to work on the new nested representation.
Similarly, we explicitly support arbitrary Coq computation within \WHERE clauses.
While the examples in this paper use only equality ($=$) and less-than ($<$) in \WHERE clauses, there is nothing preventing us from reasoning about more complex operations (e.g., case-insensitive string comparison).

Another benefit of the tactic-based approach is that we are able to re-use a considerable amount of Coq's underlying infrastructure.
Features such as higher-order unification, existing automation libraries, and \ltac's backtracking search mechanism are all useful when building a query optimizer.
We have found, in particular, that the new backtracking proof search facilities, namely \coqe{+} and dependent goals, introduced in Coq 8.5, are extremely useful when building tactics such as these.
For example, without this backtracking feature, tactics that wish to backtrack must be written in continuation passing style, and even then it can be quite difficult to maintain all the necessary information.
%%  tactics can not backtrack between different goals which initially forced us to more meticulously implement tactics in continuation passing style.

\paragraph{Drawbacks}
There are also drawbacks to using tactics to implement a procedure as sophisticated as semantic optimization.
First, tactics are completely untyped which makes it cumbersome to track down errors, which are often due to simple typos.
While simple types would help track some information, throughout the course of development we found that one of the most cumbersome tasks was keeping track of simple properties about goals; for example, whether the unification variable to be constructed was on the left or the right of an \coqe{Meq}.
Similarly, many lemmas had to be duplicated to handle extra bits of context; for example, we had to write separate lemmas for chaining together refinements in the presence and absence of embedded dependencies.
Several authors have proposed more richly typed tactic-based programming languages, notably Mtac~\cite{ziliani2013mtac} and VeriML~\cite{stampoulis2010veriml}, and while neither of these are as mature or rich as \ltac, it would be interesting to explore whether their features would be useful in our development process.

Another problem inherent to \ltac{} is speed.
Figure~\ref{fig:performance} shows the time it takes to optimize the queries presented in this paper using a Intel Core i5-4460  CPU at 3.20GHz on Coq 8.5 beta 2.  The {\tt normalize} task (not discussed) is the time it takes to normalize Coq terms, using the monad laws, into the flat form \coqe{query P C R}.
The {\tt chase} and {\tt minimize} phase are the phases described in Sections~\ref{sec:ltac-chase} and~\ref{sec:minimization}.
The final {\tt simplify} phase performs the rudimentary non-semantic optimization, discussed briefly at the end of Section~\ref{sec:minimization}.
The Total row is the total of all of the phases run from beginning to end with no other timing or intermediate results.
Overall, the optimizer is somewhat slow, especially on the index example.
In this case, a non-negligible fraction of the overall time in the index example can be attributed to solving numeric side conditions using the \coqe{omega} tactic.

\begin{figure}
\centering
\begin{tabular}{l  r  r}
               & \multicolumn{2}{ c}{Time (s)} \\
\textbf{Phase} & \textbf{Movie} & \textbf{Index} \\\hline
Normalize      & 0.64  & 0.48 \\
Chase          & 0.89  & 3.9 \\
Minimize       & 1.14  & 34.14 \\
Simplify       & 0.09  & 0.23 \\\hline
%
%Normalize      & 0.71  & 0.55 \\
%Chase          & 1.17  & 5.16 \\
%Minimize       & 1.53  & 45.60 \\
%Simplify       & 0.11  & 0.27 \\\hline
\textbf{Total} & 3.50  & 38.8 \\
\end{tabular}

\caption{Performance of the optimizer.}
\label{fig:performance}
\end{figure}

There are a few caveats to keep in mind when interpreting these performance results:
\begin{itemize}

\item Finding a homomorphism between two (conjunctive) tableaux is an NP-hard problem~\cite{Deutsch:2006:QRC:1121995.1122010}.  Specialized heuristic algorithms exist to solve this problem quickly, but our implementation uses a na\"ive brute-force search.

\item Coq's \ltac{} implementation is interpreted and single-threaded.

\item Optimization can be done offline when speed is not crucial.

%% \item A non-negligible fraction of the overall time in the index example can also be attributed to solving numeric side conditions using the \coqe{omega} tactic.

%% \item All reduction of Gallina terms is done using Coq's interpreter (using {\tt vm\_compute}).  Reduction can often be done more quickly by compilation to OCaml (using {\tt native\_compute}).

\end{itemize}

One way to make our query optimizer faster is to implement pieces of it directly as Coq programs (similar to~\cite{coqdb}).
Indeed, our initial implementation was a Gallina function rather than a tactic and it ran nearly instantaneously on the movies query.
However, implementing the optimizer entirely as a Gallina program suffers from the drawbacks discussed above.
In particular, scaling it to the index example would have required implementing (and proving sound) a Gallina function to reason about numbers and $<$.
While in theory such a procedure would only need to be implemented once, composing Gallina functions such as these has been a notoriously difficult problem which has only recently been addressed~\cite{malecha2014mirror-shard}.
%% Hence, we believe there is a balance to be struck, where the pieces of the optimizer that are invoked often should be implemented as Coq programs, and the pieces of the optimizer than need to be extensible by users should be implemented as Coq tactics.
Finding the exact balance between the two approaches is a promising direction for future work.

% we quickly found that that approach (similar to~\cite{coqdb}) required an inordinate amount of work to extend: it required building a Coq program to solve
%
%of the benefits of working in a rich logic such as Coq is the ability to leverage dependent types to offset some of the burden of building proofs.
%For example, rather than writing tactics to compute the chase and then use Coq's logic to prove the procedure sound similar to the work of Benzaken~\cite{coqdb}.
%This can yield a substantial improvement in performance.
%For example, optimizing the Movie query from beginning to end is practically instantaneous using Coq's \coqe{vm_compute} reduction mechanism~\cite{gregoire2002vmcompute}.
%
%The price we pay for this speed is the need to reimplement some of Coq's internal features such as unification.
%For example, while we were able to quickly write a basic tautology solver capable of discharging the side conditions needed for the movies example, it would havce been quite a bit more work to implement a procedure capable of reasoning about numbers necessary for the indexing example.
%Fundamentally, however, there is nothing that prevents us from performing this kind of reasoning within the logic, and a variety of work~\cite{malecha2015thesis,besson2007micromega,braibant2011aac,lescuyer2009sat,lescuyer2011these} has shown it to be both an efficient and powerful mechanism.

%% There has been some work in marrying the beneficial parts of tactic-based and standard programming within a proof assistant.
%% \textsc{MirrorCore} defines a deep embedding of a simply typed language within Coq and builds a tactic language, \textsc{Rtac}, on top of it~\cite{malecha2015thesis}.
%% Emperical results show that this can yield orders of magnitude performance improvement on some problems and provides more flexibility to directly manipulate terms than the tactic-base approach that we use in this work.

\paragraph{Coq as a DBPL}

Although our focus in this paper is on the semantic optimization of queries shallowly embedded in Coq, it is worth reflecting more broadly on the use of Coq as a host language for embedded queries.  The biggest drawback of Coq in this context is its purity.  Establishing a connection to a database is invariably an effectful operation, and in Coq such operations must be defined as axioms and segregated behind a monadic interface~\cite{Malecha:2010:TVR:1706299.1706329}.  Coq can extract such effectful code into OCaml or Haskell where it can be executed, but the Coq kernel itself cannot execute effectful code.

On the positive side, Coq inherits all of the usual advantages of a strongly-typed functional programming language for hosting a query language~\cite{monad}.  Moreover, Coq's dependent types can be used to express computations that cannot be (safely) expressed in languages with weaker type systems.  For example, it is possible to define a Coq function $f$ that can only be called on an input instance $I$ that is known to satisfy some constraint $C$ using the following pseudo-code:
\begin{coq}
Definition f (C: ED) I (pf: holds I C) := ...
\end{coq}
Indeed, the constraint $C$ need not even be known at compile time.

\section{Conclusion}
In this paper we have described a verifying semantic query optimizer for Coq based on the chase~\cite{Deutsch:2006:QRC:1121995.1122010}.  Our approach leverages programming with tactics to manipulate raw Coq terms and simultaneously produce an optimized query and a proof that the query has the same meaning as the input query.  Implementing the optimizer as a tactic has many advantages, including, first and foremost, essentially limitless extensibility: because Coq tactics can invoke other tactics, users are free to plug-in their own proof automation to be used during the optimization process.  For example, one of our examples makes use of Coq's \coqe{omega} tactic for reasoning about natural numbers and the less than relation $<$.   As far as we are aware, no other work on formalizing the relational model in Coq (e.g., ~\cite{coqdb}) implements the chase as a tactic.

However, implementing query optimization as a tactic does pose certain challenges, most of which arise because \ltac{} was designed with proof scripting in mind, rather than query optimization.
To work in this environment we construct terms incrementally using unification variables and express properties about them using goals.
These goals form a sort of ``calling convention'' for tactics and we use lemmas to perform incremental reasoning either through rewriting or direct application.
We leverage backtracking search to explore multiple potential optimization paths.
In phrasing problems dealing with binders we express syntactic manipulation of binders extensionally as functions that operate on environments.
%% For example, queries need to be manipulated indirectly through applying and rewriting using lemmas; ; and traditional first-order phrasings of syntax in terms of binders with concrete variable names needs to be translated to manipulations of environments.
We hope that our solutions to these issues in this paper will allow others to create their own query (and program) optimizers as Coq tactics.

While not currently competitive with more traditional programming languages in terms of speed, \ltac{} is continually improving and similar systems feature prominently in the development of large-scale verified software systems such as Idris~\cite{brady2013idris} and Agda~\cite{agda}.  Moreover, recent work~\cite{malecha2015thesis,devriese2013tsmp,vanderwalt2013engineering-reflection-agda} has shown how tactic-based programming can be made first-class (i.e., reified into the language) by using dependent types.
%These systems have all begun to develop their own customizable and extensible automation libraries that will enable them to be used in the same ways that we currently use scripting languages.
This next wave of tactic-based languages and libraries are already demonstrating substantial performance improvements and it is likely that tactic-based optimization will soon be a viable design pattern for language-integrated query systems.

%% Todo: need to hammer home why writing the chase as a tactic is hard, and why doing it that was is valuable.  This is a good place to compare to MirrorCore~\cite{malecha2014thesis}, rather than the related work.

{\bf Acknowledgements}  The authors would like to thank Lucian Popa for answering many questions about semantic optimization.  This work was supported by ONR grant N000141310260 and AFOSR grant FA9550-14-1-0031.

\bibliographystyle{plain}
\bibliography{thesisbib}

\end{document}

%%  LocalWords:  RDBMSs RDBMS Coq decompositions implicational EDs
%%  LocalWords:  LINQs SQL homomorphisms homomorphically homomorphism
%%  LocalWords:  homomorphic eapply Mret ret Gallina Mimpl Mmap snd
%%  LocalWords:  Qed fst morphism Ltac simpl parameterized Gallina's
%%  LocalWords:  reflexivity