\section{Conclusion}
Our implementation of \ProjectName{} and experiments show that imputation of missing values can be successfully integrated into the relational algebra and
existing cost-based plan optimization frameworks. We implement imputation actions, such as dropping or imputing values with a machine
learning technique, as operators in the algebra and use a simple, yet effective, cost model to consider trade-offs in
imputation quality and runtime. We show that different values for the trade-off parameter can yield substantially
different plans, allowing the user to express their preferences for performance on a per-query basis.
Our experiments with the CDC NHANES, freeCodeCamp and ACS datasets
show that \ProjectName{} can be used successfully with real-world data, running standard SQL
queries over tuples with missing values.
We craft a series of realistic queries, meant to resemble the kind of questions an analyst might ask during
the data exploration phase. The plans selected for each query execute an order-of-magnitude faster than
 the standard approach of imputing on the entire base tables and
then formulating queries. Furthermore, the difference in query results between the two approaches is
shown to be small in all queries considered (\lowsmapealphazero{}-\highsmapealphaone{}\% error, \Cref{sec:results}). Data analysts need not commit to a specific imputation strategy and can instead
vary this across queries.

We discuss the long history of dealing with missing data both in the statistics and database communities.
In contrast to existing work in statistics, our emphasis is not on the specific algorithm
used to impute but rather on the placement of imputation steps in the execution of a query. In contrast to existing database work,
we incorporate imputation into a cost-based optimizer and hide any details
regarding missing values inside the system, allowing users to use traditional SQL as if the database were complete.
While prior work has incorporated operations such as prediction into their databases, this has been domain-specific
in some cases, and in others, operations have not been integrated into the planner. The novelty
of our contribution lies in the formalization of missing value imputation for query planning, resulting in performance
gains over traditional approaches. This approach acknowledges that different users performing different queries on the same
dataset will likely have varying imputation needs and that execution plans should appropriately reflect that variation.

\subsection{Future Work}
\ProjectName{} opens up multiple avenues for further work.

\begin{itemize}
\item \textbf{Imputation confidence:} Obtaining confidence measures for query results
produced when missing values are imputed with a statistical model is
another possible improvement to the system. These measures could be obtained in general
using resampling methods like cross-validation and bootstrapping\cite{kohavi1995study}.
In contrast to standard confidence calculations, one added complexity is that
imputation takes place at various points in the plan, so the composition of results
through standard query operations needs to be taken into account when computing confidence
measures.
In addition, multiple imputation could be used within query execution and confidence measures like variances could
similarly propagate through the query plan.

\item \textbf{Imputation operators:} We explored a subset of possible imputation operators.
In particular, we explored two variants of the \textit{Drop} and \textit{Impute} operations. The system
could be extended to consider a broader family of operators. As the search space grows,
there will likely be additional steps needed in order to avoid sub-optimal plans.

\item \textbf{Adaptive query planning:} Currently, the imputation is local to a given query, meaning
no information is shared across queries. An intriguing direction would be to take advantage
of imputation and execution data generated by repeated queries. Note that this goes beyond
simply caching imputation results, and instead could entail operations such as extending
intermediate results with prior imputation values to improve accuracy, or pruning out
query plans which have been shown to produce low-confidence imputations.
\end{itemize}

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "main"
%%% End:


\section{Using CGCA library on HECToR}

The basic assumption of the CGCA model
is that a 3D coarray grid is used,
\texttt{[lcob(1):ucob(1),lcob(2):ucob(2),lcob(3):ucob(3)]},
where \texttt{lcob(3)} and \texttt{ucob(3)} are
1D arrays of size 3 storing the lower and the upper
{\em cobounds} of the coarray.

Each node on Hector phase 3 has 32 cores.
Since one is charged per node, it makes
perfect sense to use 1 image/core,
i.e. 1GB max for core, probably lower in
practice due to OS overheads, etc.

Previous timing analysis shows that max
speed is achived if communication is
minimised.
This partly means that decreasing the
boundary (halo) areas between images
is a good idea.
This, in turn, means that the ideal
layout of images (or image grid) is
such that images are arranged
as \texttt{[$n,n,n$]}, where $n=\textrm{\em ncores}^{1/3}$,
and {\em ncores} is the total number of cores used.

Previous work \cite{phillips2012} showed that
about $10^5$ cells are required to represent
a grain for the model to be scale independent.
This value is used here.
Table \ref{tab:hector1}  gives some idea of run times
and costs.

\begin{table}[h]
\centering
\begin{tabular}{rrrrrr}
\hline
\\
cores	&coarray	&grains	&walltime,	&Charge,	&grains \\
	&		&	&m:s		&kAU		&per kAU (GA) \\
\hline
\\
512	&(200,200,200)[8,8,8]	&40,960	&5:36	&0.2(check!)	&$2.0 \times 10^5$ (check!) \\
512	&(100,100,100)[8,8,8]	&5120	&1:16	&0.05(check!)	&$1.0 \times 10^5$(check!) \\
4096	&(10,10,10)[16,16,16]	&40	&2:15	&0.66		&60 \\
8192	&(10,10,10)[16,16,32]	&81	&5:47	&3.04		&27 \\
\hline
\end{tabular}
\caption{Coarray layout timings, for test \texttt{AAO},
to help plan Hector jobs.
Entries are sorted by the last column, which is a
relative cost of the run, i.e. how many grains can
be analysed per kAU.}
\label{tab:hector1}
\end{table}

There must be a balance between the computation
time and the communication time.
Clearly, at least for test \texttt{AAO},
which is solidification and volume calculation,
arrays of $10^3$ are too small, leading to
relatively very short computation times
and very long communication times.
Adding more cores leads to poorer efficiency,
i.e. lower GA.
Using arrays of $200^3$ increases efficiency by 5 orders
of magnitude, from small arrays on 8192 cores
to larger arrays on 512 cores.

This data is very important,
and must be taken into account when designing future models.


    \filetitle{ffrf}{Filter frequency response function of transition variables to measurement variables}{model/ffrf}

	\paragraph{Syntax}\label{syntax}

\begin{verbatim}
[F,List] = ffrf(M,Freq,...)
\end{verbatim}

\paragraph{Input arguments}\label{input-arguments}

\begin{itemize}
\item
  \texttt{M} {[} model {]} - Model object for which the frequency
  response function will be computed.
\item
  \texttt{Freq} {[} numeric {]} - Vector of frequencies for which the
  response function will be computed.
\end{itemize}

\paragraph{Output arguments}\label{output-arguments}

\begin{itemize}
\item
  \texttt{F} {[} namedmat \textbar{} numeric {]} - Array with frequency
  responses of transition variables (in rows) to measurement variables
  (in columns).
\item
  \texttt{List} {[} cell {]} - List of transition variables in rows of
  the \texttt{F} matrix, and list of measurement variables in columns of
  the \texttt{F} matrix.
\end{itemize}

\paragraph{Options}\label{options}

\begin{itemize}
\item
  \texttt{'include='} {[} char \textbar{} cellstr \textbar{}
  \emph{\texttt{Inf}} {]} - Include the effect of the listed measurement
  variables only; \texttt{Inf} means all measurement variables.
\item
  \texttt{'exclude='} {[} char \textbar{} cellstr \textbar{}
  \emph{empty} {]} - Remove the effect of the listed measurement
  variables.
\item
  \texttt{'maxIter='} {[} numeric \textbar{} \emph{500} {]} - Maximum
  number of iteration when computing the steady-state Kalman filter.
\item
  \texttt{'matrixFmt='} {[} \emph{\texttt{'namedmat'}} \textbar{}
  \texttt{'plain'} {]} - Return matrix \texttt{F} as either a
  \href{namedmat/Contents}{\texttt{namedmat}} object (i.e.~matrix with
  named rows and columns) or a plain numeric array.
\item
  \texttt{'select='} {[} \emph{\texttt{@all}} \textbar{} char \textbar{}
  cellstr {]} - Return FFRF for selected variables only; \texttt{@all}
  means all variables.
\item
  \texttt{'tolerance='} {[} numeric \textbar{} \emph{\texttt{1e-7}} {]}
  - Convergence tolerance when computing the steady-state Kalman filter.
\end{itemize}

\paragraph{Description}\label{description}

\paragraph{Example}\label{example}


\documentclass[11pt]{article}
\usepackage[margin=1in]{geometry}
\usepackage{csquotes}
\usepackage{listings}
\lstset{basicstyle=\ttfamily\footnotesize,breaklines=true}
\usepackage{hyperref}

\title{Homework 2: Validity}
\author{
  Shishido, Juan\\
  \texttt{juanshishido}
}

\begin{document}
\maketitle

\section{Implementation}

In this part of the assignment, I evaluate the following hypothesis:

\begin{displayquote}
Movies that contain John Goodman have a significantly different number of box
office hits (either higher or lower) than those that do not.
\end{displayquote}

To do this, I implement a permutation test in Python. The null hypothesis, in
this case, is that there is no difference in the proportion of box office hits
between movies with and without John Goodman. The significance level needed in
order to reject the null hypothesis is $\alpha = 0.01$. The effect size and
direction are also discussed.

To create the data set with which the hypothesis would be evaluated, I merged
the \lstinline{movie.feature.txt} and \lstinline{movie.box_office.txt} files.
The former contained data on 8,304 unique movies. With the latter, I identified
59 movies featuring John Goodman. Of those, 24 (40.7\%) were labeled as box
office hits. Of the 8,245 films not featuring John Goodman, only 2,054 (24.9\%)
were labeled as box office hits.

A permutation test estimates the probability of seeing a test statistic as
extreme as the one observed. This is derived by randomly shuffling the labels,
calculating the test statistic for each of these permutations, and counting the
number of times the permutation-based test statistics were more extreme than
the actual. The resulting proportion is the $\hat{p}$-value.

In my implementation, the labels were randomly shuffled 10,000 times. The
resulting $\hat{p}$-value was 0.0068. Based on this, we reject the null
hypothesis that there is no difference in the proportion of box office hits
between movies with and without John Goodman. The "John Goodman effect" is
positive---that is, movies featuring John Goodman tend to be box office hits
more often than movies without him. Using the information mentioned above, we
can say that the effect size is 15.8 percentage points (40.7\% - 24.9\%).

The Python code used for this analysis can be found on GitHub
\href{https://github.com/juanshishido/info290-dds/blob/master/assignments/assignment02/code/permutation.py}{[link]}.

\end{document}