\chapter{Cross Validation}
\section{Motivation}

Generally people train a neural network with the training set and obtain the test  accuracy after each epoch. After the training is finished, the highest ever or the last test  accuracy is reported in the paper. However, in our opinion, this may be not only abusing and polluting the test set, but also not generalizable.

A test set, as its name tells, should be concealed and be tested only after all the model structures are set up and the training algorithm is fixed. In this way, the test set is never touched by the reseachers and serves as independent and fair comparison target.

However, this is not the case in the field of deep learning. What people are doing is  first choosing a model and training algorithm, then evaluate the accuracy with the test set. If the test accuracy is not satisfactory, they turn to  choose a new combination of model structures and training algorithm in order to achieve a higher test accuracy. By doing so, the test set is evaluated many times or maybe thousands of times.  The test set is thus touched by the reseachers  and doesn't serves as independent and fair comparison any more. Instead, we say the original test set becomes part of the validation set. Without a proper framework or mechanism, such scenarios will be ongoing and  independent and fair comparison target will be missing. Therefore we want to propose a universal framework to evaluate the performance of the neural networks given a dataset.

\section{Universal framework}
The framework takes as input the proposed neural network, training algorithm and dateset.

Suppose we are given a specific dataset such as CIFAR-10 with 50000 training data and 10000 validation data.
In each round, we randomly choose 6000 balanced data from CIFAR-10 as the test set, namely 600 data for each label. The remaining data are used as the training set. Then we evaluate the test accuracy after finishing training the neural network.
After ten rounds, we take the average of the test accuracies as the metric reported.

The algorithm is as below.
%\begin{breakablealgorithm}%[!htb]
\begin{algorithm}%[!htb]
	\caption{Overlapping CV training algorithm}
\hspace*{\algorithmicindent} \textbf{Input} Dataset $D$, neural network $N$
	\begin{algorithmic}
	\For{$i = 1:10$}
		\State 1. Randomly choose 10\% balanced data from $D$ and denote it as $V$, the validation set. The remaining data are  denoted as $T$, the training set.
		\State 2. Train $N$ with $T$ with the stopping criterion $S$
		\State 3. When the training is finished, evaluate $N$ with $V$ and denote the validation accuracy as $A_i$
	\EndFor
	\end{algorithmic}
\hspace*{\algorithmicindent} \textbf{Output} $A(D,N)=\sum_i A_i /10$
\end{algorithm}

The  standard  cross validation algorithm is as below.
\begin{algorithm}%[!htb]
	\caption{Standard CV training algorithm}
\hspace*{\algorithmicindent} \textbf{Input} Dataset $D$, neural network $N$\\
\hspace*{\algorithmicindent} \textbf{Data split}  Randomly divide the dataset $D$ into 10 folds with numbers 1,2,..,10
	\begin{algorithmic}
	\For{$i = 1:10$}
		\State 1. Choose the $i$-th fold and denote it as $V$, the validation set. The remaining data are  denoted as $T$, the training set.
		\State 2. Train $N$ with $T$ with the stopping criterion $S$
		\State 3. When the training is finished, evaluate $N$ with $V$ and denote the validation accuracy as $A_i$
	\EndFor
	\end{algorithmic}
\hspace*{\algorithmicindent} \textbf{Output} $A(D,N)=\sum_i A_i /10$
\end{algorithm}

The  leave-one-fold  cross validation algorithm is as below.
\begin{algorithm}%[!htb]
	\caption{Leave-one-fold CV training algorithm}
\hspace*{\algorithmicindent} \textbf{Input} Dataset $D$, neural network $N$\\
\hspace*{\algorithmicindent} \textbf{Data split}  Randomly divide the dataset $D$ into 10 folds with numbers 1,2,..,10. Fix the $10$-th fold as the test set and denote it as $T_2$.
	\begin{algorithmic}
	\For{$i = 1:9$}
		\State 1. Choose the $i$-th fold and denote it as $V$, the validation set. The remaining data are  denoted as $T$, the training set.
		\State 2. Train $N$ with $T$ with the stopping criterion $S$
		\State 3. When the training is finished, evaluate $N$ with $T_2$ and denote the test accuracy as $A_i$
	\EndFor
	\end{algorithmic}
\hspace*{\algorithmicindent} \textbf{Output} $A(D,N)=\sum_i A_i /9$
\end{algorithm}

Things to discuss:

1. If we randomly choose 10\% data and do it 10 times, this is technically not the cross validation.

Cross validation is when we randomly divide the data into 10 folds of equal sizes. Then in each round, we take one fold  sequentially as the validation set, and use the remaining data as the training set.

2. Does the stopping criterion $S$ depend on $D$ and $N$? We want to use the automative learning rate scheduling algorithm, but it doesn't have a stopping criterion. Moreover, based on the result on DenseNet, SASA by Lian Zhang depends on the initial parameter tuning.

Therefore we will take different dataset splitting, learning rate scheduling, best validation accuracy vs last epoch with various models on popular dataset for comparison.

\section{ResNet-18 on CIFAR-10}
In this section, we will use test network ResNet-18 on CIFAR-10. Since the traditional training include the validation set to choose the model, therefore we split the original 54000 training data as training set with 48000 data and validation set with 6000 data. There are also two ways to choose the model, namely choosing the model after the last epoch or choosing the model with the highest validation accuracy.

\subsection{Fixed epoch and manual decreasing learning rate}
We choose to use fixed epoch training, namely 120 epochs, with changing the learning rate manually, i.e., decrease by 10 for every 30 epochs. The initial learning rate is 0.1, the momentum is 0.9 and the weight-dacay is 0.0001. The batch size is 128.

The time cost is around 17000 seconds.

If we split the sets randomly for 10 times, the results are listed in the table.
\begin{table}[!htbp]
	\centering
	\caption{Accuracy of different sets for model of last epoch and the one with best validation accuracy }
	\label{table:cv1}
	\begin{tabular}{|c|c|c|c|c|c|c|c|c|}
		\hline
		% after \\: \hline or \cline{col1-col2} \cline{col3-col4} ...
%		Iteration &  \multicolumn{2}{{|c|}}{SGD}  &    \multicolumn{2}{{|c|}}{l1-prox}    \\\cline{2-5}
%
fold	&	train accu(best	&	train accu(last	&	val accu(best	&	val accu(last	&	test accu(best 	&	test accu(last	\\
	&	 val model)	&	 epoch model)	&	val model)	&	 epoch model)	&	val model)	&	 epoch model)	\\\hline
1	&	99.92	&	99.91	&	93.13	&	92.48	&	93.40	&	93.20	\\\hline
2	&	99.92	&	99.90	&	93.05	&	92.77	&	92.52	&	92.62	\\\hline
3	&	99.93	&	99.89	&	93.17	&	92.58	&	93.35	&	93.12	\\\hline
4	&	99.91	&	99.90	&	92.97	&	92.40	&	92.33	&	92.42	\\\hline
5	&	99.91	&	99.91	&	93.30	&	93.05	&	92.63	&	92.70	\\\hline
6	&	99.91	&	99.90	&	93.50	&	93.47	&	93.12	&	92.77	\\\hline
7	&	99.92	&	99.88	&	93.32	&	93.00	&	93.47	&	93.12	\\\hline
8	&	99.92	&	99.88	&	93.35	&	92.80	&	93.40	&	93.23	\\\hline
9	&	99.90	&	99.87	&	93.78	&	93.45	&	92.53	&	92.17	\\\hline
10	&	99.92	&	99.90	&	93.53	&	93.20	&	92.05	&	92.02	\\\hline
avg	&	99.92	&	99.89	&	93.31	&	92.92	&	92.88	&	92.74	\\\hline
	\end{tabular}
\end{table}

If we split the sets into 10 folds and do the standard cross-validation, the results are listed in the table.
\begin{table}[!htbp]
	\centering
	\caption{Accuracy of different sets for model of last epoch and the one with best validation accuracy }
	\label{table:cv2}
	\begin{tabular}{|c|c|c|c|c|c|c|c|c|}
		\hline
		% after \\: \hline or \cline{col1-col2} \cline{col3-col4} ...
%		Iteration &  \multicolumn{2}{{|c|}}{SGD}  &    \multicolumn{2}{{|c|}}{l1-prox}    \\\cline{2-5}
%
fold	&	train accu(best	&	train accu(last	&	val accu(best	&	val accu(last	&	test accu(best 	&	test accu(last	\\
	&	 val model)	&	 epoch model)	&	val model)	&	 epoch model)	&	val model)	&	 epoch model)	\\\hline
1	&	99.93	&	99.92	&	94.00	&	93.52	&	93.07	&	93.43	\\\hline
2	&	99.94	&	99.88	&	93.83	&	93.53	&	93.12	&	93.07	\\\hline
3	&	99.91	&	99.90	&	93.23	&	92.92	&	93.42	&	93.32	\\\hline
4	&	99.92	&	99.90	&	93.40	&	93.05	&	93.45	&	93.22	\\\hline
5	&	99.91	&	99.89	&	93.42	&	92.93	&	92.65	&	92.67	\\\hline
6	&	99.94	&	99.88	&	93.33	&	92.82	&	93.22	&	92.87	\\\hline
7	&	99.92	&	99.90	&	93.55	&	93.17	&	93.17	&	93.40	\\\hline
8	&	99.88	&	99.85	&	92.72	&	92.35	&	92.97	&	92.35	\\\hline
9	&	99.90	&	99.86	&	93.22	&	92.65	&	92.40	&	92.27	\\\hline
10	&	99.90	&	99.88	&	93.55	&	92.82	&	92.73	&	92.97	\\\hline
avg	&	99.92	&	99.89	&	93.43	&	92.98	&	93.02	&	92.96	\\\hline
	\end{tabular}
\end{table}

If we split the sets into 10 folds, fix the first fold as the test set, take one fold as the validation set and the remaining as the training set, the results are listed in the table.
\begin{table}[!htbp]
	\centering
	\caption{Accuracy of different sets for model of last epoch and the one with best validation accuracy }
	\label{table:cv1}
	\begin{tabular}{|c|c|c|c|c|c|c|c|c|}
		\hline
		% after \\: \hline or \cline{col1-col2} \cline{col3-col4} ...
%		Iteration &  \multicolumn{2}{{|c|}}{SGD}  &    \multicolumn{2}{{|c|}}{l1-prox}    \\\cline{2-5}
%
fold	&	train accu(best	&	train accu(last	&	val accu(best	&	val accu(last	&	test accu(best 	&	test accu(last	\\
	&	 val model)	&	 epoch model)	&	val model)	&	 epoch model)	&	val model)	&	 epoch model)	\\\hline
1	&	99.93	&	99.93	&	93.58	&	93.13	&	92.55	&	92.65	\\\hline
2	&	99.94	&	99.89	&	93.57	&	92.98	&	92.93	&	92.78	\\\hline
3	&	99.93	&	99.89	&	93.48	&	93.03	&	92.85	&	93.20	\\\hline
4	&	99.95	&	99.92	&	93.28	&	92.95	&	93.03	&	93.18	\\\hline
5	&	99.92	&	99.90	&	93.12	&	92.75	&	92.92	&	92.50	\\\hline
6	&	99.94	&	99.93	&	93.52	&	92.93	&	92.92	&	93.43	\\\hline
7	&	99.93	&	99.92	&	93.40	&	93.10	&	93.00	&	93.15	\\\hline
8	&	99.93	&	99.90	&	92.77	&	92.18	&	92.58	&	93.02	\\\hline
9	&	99.93	&	99.90	&	92.80	&	92.22	&	92.52	&	92.37	\\\hline
avg	&	99.93	&	99.91	&	93.28	&	92.81	&	92.81	&	92.92	\\\hline
	\end{tabular}
\end{table}

\newpage
\section{VGG-16 on CIFAR-10}
In this section, we will use test network VGG-16 on CIFAR-10. Since the traditional training include the validation set to choose the model, therefore we split the original 54000 training data as training set with 48000 data and validation set with 6000 data. There are also two ways to choose the model, namely choosing the model after the last epoch or choosing the model with the highest validation accuracy.

\subsection{Fixed epoch and manual decreasing learning rate}
We choose to use fixed epoch training, namely 160 epochs.
The initial learning rate is set to 0.1, and is divided by 10 at 50\% and 75\% of the total number of training epochs.
The momentum is 0.9 and the weight-dacay is 0.0001. The batch size is 128.

The time cost is around 22000 seconds.

If we split the sets randomly for 10 times, the results are listed in the table.
\begin{table}[!htbp]
	\centering
	\caption{Accuracy of different sets for model of  last epoch and the one with best validation accuracy }
	\label{table:cv3}
	\begin{tabular}{|c|c|c|c|c|c|c|c|c|}
		\hline
		% after \\: \hline or \cline{col1-col2} \cline{col3-col4} ...
%		Iteration &  \multicolumn{2}{{|c|}}{SGD}  &    \multicolumn{2}{{|c|}}{l1-prox}    \\\cline{2-5}
%
fold	&	train accu(best	&	train accu(last	&	val accu(best	&	val accu(last	&	test accu(best 	&	test accu(last	\\
	&	 val model)	&	 epoch model)	&	val model)	&	 epoch model)	&	val model)	&	 epoch model)	\\\hline
1	&	99.97	&	99.97	&	93.12	&	92.77	&	93.28	&	92.65	\\\hline
2	&	99.97	&	99.96	&	92.85	&	92.65	&	92.92	&	93.18	\\\hline
3	&	99.99	&	99.96	&	93.12	&	92.48	&	92.78	&	92.90	\\\hline
4	&	99.97	&	99.95	&	92.98	&	92.72	&	92.07	&	92.40	\\\hline
5	&	99.97	&	99.95	&	93.38	&	93.08	&	92.37	&	92.40	\\\hline
6	&	99.97	&	99.95	&	93.10	&	92.80	&	92.35	&	92.57	\\\hline
7	&	99.98	&	99.97	&	93.05	&	92.65	&	92.90	&	93.20	\\\hline
8	&	99.97	&	99.97	&	93.03	&	92.72	&	93.07	&	92.82	\\\hline
9	&	99.97	&	99.96	&	93.50	&	93.07	&	92.37	&	92.60	\\\hline
10	&	99.98	&	99.96	&	93.43	&	93.18	&	92.20	&	92.57	\\\hline
avg	&	99.97	&	99.96	&	93.16	&	92.81	&	92.63	&	92.73	\\\hline
	\end{tabular}
\end{table}

If we split the sets into 10 folds and do the standard cross-validation, the results are listed in the table.
\begin{table}[!htbp]
	\centering
	\caption{Accuracy of different sets for model of  last epoch and the one with best validation accuracy }
	\label{table:cv4}
	\begin{tabular}{|c|c|c|c|c|c|c|c|c|}
		\hline
		% after \\: \hline or \cline{col1-col2} \cline{col3-col4} ...
%		Iteration &  \multicolumn{2}{{|c|}}{SGD}  &    \multicolumn{2}{{|c|}}{l1-prox}    \\\cline{2-5}
%
fold	&	train accu(best	&	train accu(last	&	val accu(best	&	val accu(last	&	test accu(best 	&	test accu(last	\\
	&	 val model)	&	 epoch model)	&	val model)	&	 epoch model)	&	val model)	&	 epoch model)	\\\hline
1	&	99.98	&	99.96	&	93.45	&	92.92	&	92.68	&	92.92	\\\hline
2	&	99.97	&	99.95	&	93.47	&	93.25	&	92.75	&	92.98	\\\hline
3	&	99.97	&	99.97	&	93.08	&	92.55	&	92.65	&	93.38	\\\hline
4	&	99.97	&	99.96	&	93.27	&	92.97	&	92.67	&	92.80	\\\hline
5	&	99.97	&	99.95	&	92.92	&	92.42	&	92.53	&	92.40	\\\hline
6	&	99.98	&	99.98	&	92.87	&	92.42	&	92.87	&	93.08	\\\hline
7	&	99.97	&	99.96	&	93.28	&	93.13	&	93.07	&	93.22	\\\hline
8	&	99.98	&	99.96	&	93.10	&	92.72	&	92.90	&	93.15	\\\hline
9	&	99.97	&	99.96	&	93.38	&	93.02	&	92.48	&	92.83	\\\hline
10	&	99.98	&	99.97	&	93.22	&	92.92	&	92.57	&	92.52	\\\hline
avg	&	99.97	&	99.96	&	93.20	&	92.83	&	92.72	&	92.93	\\\hline	\end{tabular}
\end{table}

If we split the sets into 10 folds, fix the first fold as the test set, take one fold as the validation set and the remaining as the training set, the results are listed in the table.
\begin{table}[!htbp]
	\centering
	\caption{Accuracy of different sets for model of last epoch and the one with best validation accuracy }
	\label{table:cv1}
	\begin{tabular}{|c|c|c|c|c|c|c|c|c|}
		\hline
		% after \\: \hline or \cline{col1-col2} \cline{col3-col4} ...
%		Iteration &  \multicolumn{2}{{|c|}}{SGD}  &    \multicolumn{2}{{|c|}}{l1-prox}    \\\cline{2-5}
%
fold	&	train accu(best	&	train accu(last	&	val accu(best	&	val accu(last	&	test accu(best 	&	test accu(last	\\
	&	 val model)	&	 epoch model)	&	val model)	&	 epoch model)	&	val model)	&	 epoch model)	\\\hline
1	&	99.98	&	99.98	&	93.40	&	93.40	&	93.13	&	93.10	\\\hline
2	&	99.98	&	99.97	&	93.37	&	93.23	&	93.00	&	93.05	\\\hline
3	&	99.98	&	99.97	&	93.47	&	92.85	&	92.67	&	93.25	\\\hline
4	&	99.98	&	99.96	&	93.02	&	92.45	&	93.20	&	92.73	\\\hline
5	&	99.98	&	99.95	&	93.57	&	93.18	&	92.70	&	93.00	\\\hline
6	&	99.98	&	99.97	&	93.43	&	92.77	&	92.80	&	92.65	\\\hline
7	&	99.97	&	99.96	&	93.18	&	92.78	&	92.95	&	92.55	\\\hline
8	&	99.98	&	99.97	&	92.72	&	92.18	&	93.17	&	92.90	\\\hline
9	&	99.97	&	99.96	&	93.47	&	92.95	&	92.72	&	92.63	\\\hline
avg	&	99.98	&	99.97	&	93.29	&	92.87	&	92.93	&	92.87	\\\hline
	\end{tabular}
\end{table}

\newpage
\section{VGG-19 on CIFAR-10}
In this section, we will use test network VGG-16 on CIFAR-10. Since the traditional training include the validation set to choose the model, therefore we split the original 54000 training data as training set with 48000 data and validation set with 6000 data. There are also two ways to choose the model, namely choosing the model after the last epoch or choosing the model with the highest validation accuracy.

\subsection{Fixed epoch and manual decreasing learning rate}
We choose to use fixed epoch training, namely 160 epochs.
The initial learning rate is set to 0.1, and is divided by 10 at 50\% and 75\% of the total number of training epochs.
The momentum is 0.9 and the weight-dacay is 0.0001. The batch size is 128.

The time cost is around 27000 seconds.

If we split the sets randomly for 10 times, the results are listed in the table.
\begin{table}[!htbp]
	\centering
	\caption{Accuracy of different sets for model of  last epoch and the one with best validation accuracy }
	\label{table:cv3}
	\begin{tabular}{|c|c|c|c|c|c|c|c|c|}
		\hline
		% after \\: \hline or \cline{col1-col2} \cline{col3-col4} ...
%		Iteration &  \multicolumn{2}{{|c|}}{SGD}  &    \multicolumn{2}{{|c|}}{l1-prox}    \\\cline{2-5}
%
fold	&	train accu(best	&	train accu(last	&	val accu(best	&	val accu(last	&	test accu(best 	&	test accu(last	\\
	&	 val model)	&	 epoch model)	&	val model)	&	 epoch model)	&	val model)	&	 epoch model)	\\\hline
1	&	99.98	&	99.97	&	93.15	&	92.65	&	92.82	&	92.85	\\\hline
2	&	99.98	&	99.97	&	93.38	&	92.78	&	92.35	&	92.85	\\\hline
3	&	99.98	&	99.97	&	92.73	&	92.33	&	92.27	&	92.05	\\\hline
4	&	99.97	&	99.96	&	92.73	&	92.40	&	92.22	&	92.82	\\\hline
5	&	99.98	&	99.96	&	93.18	&	92.85	&	92.13	&	92.13	\\\hline
6	&	99.99	&	99.99	&	92.82	&	92.45	&	92.30	&	91.90	\\\hline
7	&	99.97	&	99.97	&	92.75	&	92.23	&	92.68	&	92.52	\\\hline
8	&	99.98	&	99.96	&	92.70	&	92.62	&	93.02	&	92.92	\\\hline
9	&	99.98	&	99.97	&	93.45	&	93.15	&	92.12	&	92.13	\\\hline
10	&	99.98	&	99.97	&	93.18	&	92.83	&	91.93	&	91.82	\\\hline
avg	&	99.98	&	99.97	&	93.01	&	92.63	&	92.38	&	92.40	\\\hline
	\end{tabular}
\end{table}

If we split the sets into 10 folds and do the standard cross-validation, the results are listed in the table.

\begin{table}[!htbp]
	\centering
	\caption{Accuracy of different sets for model of  last epoch and the one with best validation accuracy }
	\label{table:cv3}
	\begin{tabular}{|c|c|c|c|c|c|c|c|c|}
		\hline
		% after \\: \hline or \cline{col1-col2} \cline{col3-col4} ...
%		Iteration &  \multicolumn{2}{{|c|}}{SGD}  &    \multicolumn{2}{{|c|}}{l1-prox}    \\\cline{2-5}
%
fold	&	train accu(best	&	train accu(last	&	val accu(best	&	val accu(last	&	test accu(best 	&	test accu(last	\\
	&	 val model)	&	 epoch model)	&	val model)	&	 epoch model)	&	val model)	&	 epoch model)	\\\hline
1	&	99.98	&	99.98	&	93.38	&	92.75	&	92.70	&	92.78	\\\hline
2	&	99.97	&	99.97	&	93.43	&	93.07	&	93.05	&	92.87	\\\hline
3	&	99.98	&	99.96	&	93.27	&	93.08	&	93.03	&	92.70	\\\hline
4	&	99.99	&	99.99	&	92.75	&	92.27	&	92.82	&	92.43	\\\hline
5	&	99.98	&	99.98	&	92.93	&	92.53	&	91.67	&	92.08	\\\hline
6	&	99.98	&	99.97	&	92.92	&	92.50	&	93.25	&	93.05	\\\hline
7	&	99.98	&	99.97	&	93.00	&	92.65	&	92.55	&	92.30	\\\hline
8	&	99.98	&	99.96	&	92.70	&	92.42	&	92.82	&	92.72	\\\hline
9	&	99.98	&	99.98	&	93.22	&	93.05	&	92.48	&	92.70	\\\hline
10	&	99.97	&	99.95	&	93.13	&	92.42	&	92.80	&	93.00	\\\hline
avg	&	99.98	&	99.97	&	93.07	&	92.67	&	92.72	&	92.66	\\\hline
	\end{tabular}
\end{table}

If we split the sets into 10 folds, fix the first fold as the test set, take one fold as the validation set and the remaining as the training set, the results are listed in the table.
\begin{table}[!htbp]
	\centering
	\caption{Accuracy of different sets for model of last epoch and the one with best validation accuracy }
	\label{table:cv1}
	\begin{tabular}{|c|c|c|c|c|c|c|c|c|}
		\hline
		% after \\: \hline or \cline{col1-col2} \cline{col3-col4} ...
%		Iteration &  \multicolumn{2}{{|c|}}{SGD}  &    \multicolumn{2}{{|c|}}{l1-prox}    \\\cline{2-5}
%
fold	&	train accu(best	&	train accu(last	&	val accu(best	&	val accu(last	&	test accu(best 	&	test accu(last	\\
	&	 val model)	&	 epoch model)	&	val model)	&	 epoch model)	&	val model)	&	 epoch model)	\\\hline
1	&	99.98	&	99.98	&	93.37	&	92.87	&	92.42	&	92.88	\\\hline
2	&	99.99	&	99.98	&	92.97	&	92.47	&	93.27	&	92.95	\\\hline
3	&	99.98	&	99.97	&	93.27	&	92.75	&	92.60	&	93.02	\\\hline
4	&	99.98	&	99.97	&	93.05	&	92.55	&	92.88	&	92.65	\\\hline
5	&	99.99	&	99.97	&	93.63	&	92.85	&	92.70	&	92.87	\\\hline
6	&	99.98	&	99.95	&	92.87	&	92.40	&	91.85	&	92.82	\\\hline
7	&	99.98	&	99.97	&	93.32	&	92.77	&	92.93	&	93.07	\\\hline
8	&	99.98	&	99.98	&	92.70	&	92.70	&	92.93	&	93.23	\\\hline
9	&	99.99	&	99.99	&	93.07	&	92.82	&	92.70	&	92.17	\\\hline
avg	&	99.98	&	99.97	&	93.14	&	92.69	&	92.70	&	92.85	\\\hline
	\end{tabular}
\end{table}

\newpage
\section{Discussion}
We find the test accuracy of the random splitting cross validation is consistently lower than that of the other two methods. This may happen due to the existence of the outliers.

\subsection{Outliers}
There can be some problems about the data labels.

\begin{enumerate}
\item Images with wrong labels: the label is wrong or there is a better label than given.
\item Multi-label images: There should be multiple labels to a image since it contains multiple objects.
\item Ontological issues: the class maillot appears twice, the existence of is-a relationships like bathtub is a tub, misnomers like projectile and missile, and unanticipated issues caused by words with multiple definitions like corn and ear.
\end{enumerate}

\subsection{Distribution}
For simplicity, we assume there are $N_0$ outliers out of 60000 over all images in CIFAR-10 dataset. Suppose we do a $k$-fold cross validation for the overlapping and standard one, we want to compared the total number of outliers in all the test folds, $X$.

As for the standard $k$-fold cross validation, since there are no overlapping among the test folds, each outlier will appear in a test fold once and only once. Therefore the total number of outliers in all the test folds is $N_0$.

As for the overlapping $k$-fold cross validation, things are little bit different. We state the procedure here to better understand the underlying distribution.
\begin{enumerate}
\item We choose $\frac{60000}{k}$ balanced data from the dataset as the test set. Here we assume the distribution of outliers in different classes is uniform. Then this is a drawing of $N=\frac{60000}{k}$ data without replacement. For each drawing, the probability of obtaining an  outlier is a constant $p=\frac{N_0}{60000}$. Therefore the number of outliers in this test fold follows a binomial distribution $B_1(N,p)$. The mean is $\mu=N*p=\frac{N_0}{k}$ and the variance is $$\sigma=N*p*(1-p)=\frac{N_0*(60000-N_0)}{60000*k}.$$
\item Then we repeat the first step for k times. Since the setup is the same, we have the same binomial distributions $B_i(N,p), i=1,2,3,..,k$. By the property of binomials, $$X \sim \sum_{i=1}^{k}B_i(N,p)=B(N*k,p)=B(60000,\frac{N_0}{60000}).$$ The mean is $\mu=N*k*p=N_0$ and the variance is $$\sigma=N*k*p*(1-p)=\frac{N_0*(60000-N_0)}{60000}.$$
\end{enumerate}

From the analysis, we can see the mean is also  $N_0$, but there is a large variance $\frac{N_0*(60000-N_0)}{60000}$. It means in each experiment of the overlapping $k$-fold cross validation, the total number of outliers in all the test folds can be quite different. We may have encountered a bad case where $X$ is larger than $N_0$ such that the test accuracy is consistently lower than that of the other two methods. It contradicts to our original goal to use statistical methods to reduce randomness and improve the repeatability and substantivity. Therefore we would suggest to use the second and third method for better evaluation.

\section{Conclusion}
Based on our numerical result, we can conclude the following things as for the framework and for the traditional training algorithms.

\begin{enumerate}
\item People use the test set multiple times. The best they can do is to run the same setup of the experiment multiple times to ensure the robustness and repeatability. With our cross-validation framework, we can fairly compare the performance of the proposed model and training algorithms.
\item As for the first two kinds of data splitting, the regular cross validation taking validation set and test set sequentially achieved better accuracy for all the experiments. We also prefer this way since we don't need to consider the problem of repeated data in different folds for the cross validation taking random validation set and test set.
\item As for the last two kinds of data splitting, the regular cross validation taking validation set and test set sequentially achieved comparable  accuracy with the leave-one-fold cross validation. It suggests that when the test data is already given, there is not much difference about the data splitting and model selection.
\item In the overlapping $k$-fold cross validation, the total number of outliers in all the test folds follows a binomial distribution and can be quite different. Therefore we don't want to use this method for evaluation.
\item The average test accuracy is comparable as for taking the model with best validation accuracy or the final model after training. This coincides with our observation that these two ways don't make a difference. It also explains why both ways exist when researchers are reporting their results.
\end{enumerate}


Language models have to  cope with out-of-vocabulary words, that is internally represented
with the word class  {\tt \_unk\_}.  In order  to
compare perplexity of LMs having  different vocabulary size it is better
to define  a conventional dictionary  size, or dictionary  upper bound
size,  trough the  parameter  ({\tt -dub}).  In  the  following example,  we
compare the perplexity of the full vocabulary LM against the perplexity of the
LM estimated over the more frequent 10K-words. In our comparison, we assume a dictionary
upper bound of one million words.

\begin{verbatim}
$>tlm -tr=train.10k.www -n=3 -lm=wb -te=test -dub=1000000
  n=49984 LP=342160.8721 PP=939.5565162 OVVRate=0.07666453265

$>tlm -tr=train.www -n=3 -lm=wb -te=test -dub=1000000
  n=49984 LP=336276.7842 PP=835.2144716 OVVRate=0.05007602433
\end{verbatim}

\noindent
The  large  difference  in  perplexity  between the two LMs is   explained  by  the
significantly higher  OOV rate of the 10K-word LM.

\noindent
N-gram LMs generally apply frequency smoothing techniques, and combine
smoothed frequencies according to  two main schemes: interpolation and
back-off.  The  toolkit assumes interpolation
as default.  The back-off  scheme is computationally more costly but
often provides better performance. It  can be activated with the option
{\tt -bo=yes}, e.g.:

\begin{verbatim}
$>tlm -tr=train.10k.www -n=3 -lm=wb -te=test -dub=1000000 -bo=yes
  n=49984 LP=337278.3227 PP=852.1186066 OVVRate=0.07666453265
\end{verbatim}

\noindent
This toolkit implements several frequency smoothing methods, which are
specified  by  the  parameter  {\tt -lm}.  Three  methods  are  particularly
recommended:
\begin{itemize}
\item [a)] {\bf Modified shift-beta}, also known as  ``improved kneser-ney smoothing''.
This smoothing scheme gives top performance when training data is not
very sparse but it is more time and memory consuming during the estimation phase:

\begin{verbatim}
$>tlm -tr=train.www -n=3 -lm=msb -te=test -dub=1000000 -bo=yes
  n=49984 LP=321877.3411 PP=626.1609806 OVVRate=0.05007602433
\end{verbatim}

\item [b)] {\bf Witten Bell smoothing}. This is an excellent smoothing
   method which works well in every data condition and is much less time and memory consuming:

\begin{verbatim}
$> tlm -tr=train.www -n=3 -lm=wb -te=test -dub=1000000  -bo=yes
  n=49984 LP=331577.2279 PP=760.2652095 OVVRate=0.05007602433
\end{verbatim}

\item [c)] {\bf Shift-beta smoothing}. This smoothing method is a simpler and cheaper version
of the Modified shift-beta method and works sometimes better than Witten-Bell method:

\begin{verbatim}
$> tlm -tr=train.www -n=3 -lm=sb -te=test -dub=1000000  -bo=yes
  n=49984 LP=334724.5032 PP=809.6750442 OVVRate=0.05007602433
\end{verbatim}

\noindent
Moreover, the non linear smoothing parameter $\beta$ can be specified with the option {\tt -beta}:
\begin{verbatim}
$> tlm -tr=train.www -n=3 -lm=sb -beta=0.001 -te=test -dub=1000000
       -bo=yes
  n=49984 LP=449339.8282 PP=8019.836058 OVVRate=0.05007602433
\end{verbatim}
This could be helpful in case we need to use language models with very limited frequency smoothing.

\end{itemize}
\subsection*{Limited Vocabulary}
\noindent
Using an  n-gram table  with a fixed  or limited  dictionary  will cause
some performance  degradation, as LM smoothing  statistics result
slightly distorted. A  valid alternative is to estimate  the LM on the
full dictionary of the training corpus and to use a limited dictionary
just when  saving the  LM on a  file.  This  can be achieved  with the
option {\tt -d} (or {\tt -dictionary}):
\begin{verbatim}
$> tlm -tr=train.www -n=3 -lm=msb -bo=y -te=test -o=train.lm -d=top10k
\end{verbatim}


\chapter{Fixed Parameters}
\label{appendix:fixed_parameters}

\begin{itemize}
\item \textbf{Tournament size}: 4
\item \textbf{Population size}: 100
\item \textbf{Number of Generations}: 250
\item \textbf{Maximum Program Tree Depth}: 17
\item \textbf{Function set}: $\{ +, -, *, /, \log, \exp, \sin, \cos, -x \}$
\item \textbf{Terminal set}: Only the features in the data set.
\end{itemize}


\section{Kinky string decay simulation.}

\hspace{1.0em}
For kinky string decay simulation we have assumed the two steps process:
\begin{enumerate}
\item split gluon $g\rightarrow q_1 \bar{q}_1$ and create two longitudinal
strings;
\item decay longitudunal strings $q\bar{q}_1 \rightarrow h$ and
 $\bar{q}q_1 \rightarrow h$ into hadrons $h$.
\end{enumerate}
The production of $q_1\bar{q}_1$ is considered similar (the same sampling of
quark flavors and the same $p_t$-distribution for the quarks) as a production
of $q\bar{q}$-pairs during the longitudinal string decay.
We took the $g\rightarrow q_1\bar{q}_1$ splitting function \cite{APKW80}:
\begin{equation}
\label{KSD1}f^{q}_{g}(z)=z^2+(1-z)^2,
\end{equation}
where $z=\frac{E_q+p^z_q}{E_g+p^z_g}$, derived by Altarelli and
Parisi \cite{AP77}.


\subsubsection{\stid{3.06} PETSc-TAO} \label{subsubsect:petsc}
\paragraph{Overview}

Algebraic solvers (generally nonlinear solvers that use sparse linear solvers via Newton's method) and ODE/DAE
integrators form the core computation of many numerical simulations. No scalable ``black box'' sparse solvers
or integrators work for all applications, nor single implementations that work well for all scales of
problem size. Hence, algebraic solver packages provide a wide variety of algorithms and implementations
that can be customized for the application and range of problem sizes at hand. PETSc~\cite{petsc:homepage,petsc-man}
is a widely used software library for the scalable solution of linear, nonlinear, and ODE/DAE systems and
computation of adjoints (sometimes called sensitivities) of ODE systems. We focus on three topics: (1) partially
matrix-free scalable solvers efficiently use many-core and GPU-based systems; (2) reduced synchronization
algorithms that can scale to larger concurrency than solvers with synchronization points; and (3) performance
and data structure optimizations for all the core data structures to better utilize many-core and GPU-based
systems as well as provide scalability to the Exascale.

The availability of systems with over 100 times the processing power of today's machines compels the utilization
of these systems not just for a single ``forward solve'' simulation (as discussed above) but rather within a
tight loop of optimization, sensitivity analysis (SA), and uncertain quantification (UQ). This requires the
implementation of a new, scalable library for managing a dynamic hierarchical collection of running scalable
simulations, where the simulations directly feed results into the optimization, SA, and UQ solvers.  This library,
which we call libEnsemble, directs the multiple concurrent ``function evaluations'' through the tight coupling
and feedback described above. This work consist of two parts: (1) the development of libEnsemble and (2) the
development of algorithms and software to utilize libEnsemble.

\paragraph{Key Challenges}

A key challenge for for scaling the PETSc/TAO numerical libraries to Exascale systems is that
traditional ``sparse-matrix-based'' techniques for linear, nonlinear, and ODE solvers, as well
as optimization algorithms, are memory-bandwidth limited.  Another difficulty is that any
synchronizations required across all compute units--for example, an inner product or a
norm--can dramatically affect the scaling of the solvers.

Running an ensemble of simulation requires a coordination layer that handles load balancing and
allows the collection of running simulations to grow and shrink based on feedback. Thus, this
library must be able to dynamically start simulations with different parameters, resume
simulations to obtain more accurate results, prune running simulations that the solvers
determine can no longer provide useful information, monitor the progress of the simulations,
and stop failed or hung simulations, and collect data from the individual simulations both
while they are running and at the end.

\paragraph{Solution Strategy}

To address the scalability of the numerical libraries, we are developing new solvers and data
structures including pipeline Krylov methods that delay the use of the results of inner products
and norms, allowing overlapping of the reductions and other computation; partially matrix-free
solvers using high-order methods that have high floating-point-to-memory-access ratios and
good potential to use many-core and GPU-based systems; and in-node optimizations of sparse
matrix-matrix products needed by algebraic multigrid to better utilize many-core systems
using a thread neutral ``bypass MPI'' approach, which implements default interprocessor
communication using MPI but bypasses the use of MPI in performance-critical regions
for higher performance and thereby maintains MPI portability.

Our strategy for coordinating ensemble computations has been to develop libEnsemble
to satisfy our needs.  This library should not be confused with workflow-based
scripting systems; rather it is a library that, through the tight coupling and
feedback described above, directs the multiple concurrent ``function evaluations''
needed by optimization, SA, and UQ solvers.

\paragraph{Recent Progress}

In the past year, we have released PETSc/TAO 3.12 (available at \url{http://www.mcs.anl.gov/petsc})
that features enhanced GPU support.  Perhaps the most important is the support for CUDA-aware
MPI, which allows direct communication of data between Summit GPUs, bypassing the previously
needed step of first copying the data to the CPU memory. This enhancement reduces the latency
of the communication and improves bandwidth. For example, on Summit for sparse matrix-vector
products, this led to a speedup of 33\% on one node and a speedup of 13\% on on four nodes
for the same size problem. Another important addition is the ability to efficiently read
in large meshes on thousands of nodes along with support for collect use of HDF5 calls.
With these additions, the algebraic multigrid solver GAMG is up to 12x faster at scale
on all of Summit when using the GPUs compared to using just the CPUs.

\begin{figure}
\centering
\includegraphics[width=0.5\textwidth]{projects/2.3.3-MathLibs/2.3.3.06-PETSc-TAO/petsc_profile}
\caption{Profile information on the effect of vector size on vector operations compared with
memory throughput (one MPI rank per GPU) for PETSc/TAO 3.12. Note the log scale.}
\label{fig:petsc-tao-fig}
\end{figure}

We have also release libEnsemble 0.5.2 (available at \url{https://github.com/Libensemble/libensemble}).
Notably, this release includes several changes in progressing toward xSDK compliance had have added
support for testing on MacOS to support more applications.  We also have improved I/O, logging,
profiling, as well as resource detection on Summit.

\paragraph{Next Steps}

Our next efforts are:
\begin{enumerate}
  \item \textbf{Enhanced application integration and performance optimization and benchmarking}:
  Refresh the AMReX/PETSc interface and ensure that the PETSc linear solvers can be used with AMReX.
  Consolidate the two MPI communication modules (VecScatter and PetscSF) to share the same code base
  and implement two important communication optimizations in the unified code base: internode node-message
  aggregation optimization, which is critical for scalability of GAMG, and intranode shared memory
  optimization.
  Ensure that libEnsemble satisfies eight of the xSDK requirements.
  Continue to produce benchmark results on Summit.
  Engage with ECP applications and co-design centers to identify new capabilities that can result in
  FY2021-2023 application integration activities.
  \item \textbf{Harmonization, performance optimization, and software release}:
  Harmonize the quasi-Newton and line search modules between SNES and TAO to improve maintainability.
  Implement GPU optimizations of the quasi-Newton methods.
  Ensure that libEnsemble satisfies all but one of the xSDK requirements.
  Continue to produce benchmark results on Summit.
  Continue to engage with ECP applications and co-design centers.
  Release a new version of PETSc/TAO.
  \item \textbf{Enhanced application support and software release}:
  Implement and optimize the ability to use libCEED under PETSc.
  Continue to produce benchmark results on Summit.
  Continue to engage with ECP applications and co-design centers.
  Release a new version of libEnsemble that is compatible with the xSDK community policy requirements and included in the xSDK.
  \item \textbf{Benchmarking, performance optimization and software release}:
  Demonstrate new libEnsemble functionality by having libEnsemble serve as the driver/outer process that
  invokes another xSDK tool.
  Complete benchmarking one or more of the PETSc based applications on Summit at scale including GAMG support
  on the GPUs and identify performance bottlenecks.
  Incorporate new desired capabilities resulting from our engagement with ECP applications and co-design centers into our FY2021 plans.
  Release a new version of PETSc/TAO.
\end{enumerate}


\documentclass[main.tex]{subfiles}
\begin{document}

% \section*{Thu Dec 19 2019}

% The lessons on gravitational waves will be on the 8th and 9th of January, from 14:30 to 16:30, in rooms LUF2 and P2B respectively.

\subsection{The Chandrasekhar limit in more detail}

We want to derive the Chandrasekhar limit in a more precise manner.

% The number density of electrions is given by
% %
% \begin{align}
% n_e = Y_{e} \frac{\rho_c}{m_H}
% \,,
% \end{align}
% %
% in the nonrelativistic case the pressure is given by
% %
% \begin{align}
% P_c = k _{\text{NR}} n_e^{5/3} = k _{\text{NR}} \qty(\frac{Y_e \rho_{c}}{m_H})^{5/3}
% \,,
% \end{align}
%
% but we can also derive it by
% %
% \begin{align}
% P_c = k _{\text{NR}} n_e^{5/3} = k _{\text{NR}} \qty(\frac{Y_e \rho_{c}}{m_H})^{5/3}
% \,,
% \end{align}
% %
% and equating these we find:
%
% \begin{align}
%  k _{\text{NR}} \qty(\frac{Y_e \rho_{c}}{m_H})^{5/3}
%  = k _{\text{NR}} n_e^{5/3} = k _{\text{NR}} \qty(\frac{Y_e \rho_{c}}{m_H})^{5/3}
% \,,
% \end{align}
% %
% which implies
% %
% \begin{align}
% \rho_{c} = \frac{3.1}{Y_e^{5}} \qty(\frac{M}{M_{*}})^2 \frac{m_H}{(h / m_e c^2)^3}
% \,,
% \end{align}
%
% where \(\alpha_{G} = G m_H^2 / (\hbar c) \approx \SI{5.9e-30}{}\), while
% %
% \begin{align}
% m_{*} = \alpha_{G}^{-3 /2 } m_H = 1.85 M_{\odot}
% \,,
% \end{align}
% %
% while for the ultrarelativistic case we get
% %
% \begin{align}
% P_C = k _{\text{UR}} n_e^{4/3} = k _{\text{UR}} \qty(\frac{Y_e \rho_{c} }{m_H})^{4/3}
% \,,
% \end{align}
%
% so in this particular case the density \(\rho_{c}\) simplifies from the equations: we get a critical mass
% %
% \begin{align}
% M _{\text{CHANDRA}} = \qty(\frac{36}{\pi })^{1/2}
% \qty(\frac{Y_e}{m_H})^2
% \qty(\frac{k _{\text{UR}}}{G})^{3/2} \approx
% 2.3 Y_e^2 m_{*} \approx 4.3 Y_e^2 M_{\odot}
% \,,
% \end{align}
%
% and we assume that we are in the fully degenerate case: in the integration over momenta of the phase space distribution we insert a cutoff at the Fermi energy.

% We find:
Instead of approximating the gas as either ultrarelativistic or nonrelativistic, we can use the correct expression for the particle energy in the integral for the momentum:
%
\begin{align}
P = \frac{4 \pi }{3 h^3} g_{*} \int_{0}^{p_F} \dd{p} p^2 \frac{p^2 c^2}{\epsilon_{p}}
\,,
\end{align}
%
with \(\epsilon_{p} = \qty(p^2c^2 + m^2c^{4})^{1/2}\).

We change variables to the dimensionless \(x = p / (m_e c)\) and substitute \(g_* = 2\), since electrons have spin \(1/2\):
%
\begin{align}
P = \frac{8 \pi }{3 h^3} m_e^4 c^{5} \int_{0}^{x_F} \frac{x^{4}}{(1+x^2)^{1/2}} \dd{x}
\,.
\end{align}

The variable \(x_F\) is given by:
\begin{align}
x_F = \frac{p_F}{m_ec} =
\qty(\frac{3 n_e}{8 \pi })^{1/3} \frac{h}{m_e c} =
\qty(\frac{3 Y_e \rho_{c}}{8 \pi m_H})^{1/3} \frac{h}{m_e c}
\,,
\end{align}
%
and, since the electrons are fully degenerate, \(x_F \gg 1 \) corresponds to the ultrarelativistic case while \(x_F \ll 1\) corresponds to the nonrelativistic case.

Now, as \(x_F \to \infty \) the integral is asymptotically
%
\begin{align}
\int_0^{x_F} \frac{x^{4}}{\sqrt{1 + x^2}} \dd{x} \sim \frac{x_F^{4}}{4}
\,,
\end{align}
%
so we define
%
\begin{align}
I(x_F) &= \frac{4}{x_F^{4}}\int_0^{x_F} \frac{x^{4}}{(1 + x^2)^{1/2}} \dd{x} \\
&= \frac{3}{2 x^{4}} \qty( x (1+x^2)^{1/2} \qty(\frac{2x^2}{3} -1) + \log \qty(x + (1+x^2)^{1/2}))
\,,
\end{align}
%
which approaches 1 as \(x_F \to \infty \).
Then, we can write the pressure as
%
\begin{align}
P &= \frac{8 \pi }{3 h^3} m_e^{4} c^{5} \frac{x_F^{4}}{4} I(x_F)  \\
&= k_{UR} n_e^{4/3} I(x_F)
\,.
\end{align}

We can see that this manipulation works by explicitly doing the calculation, but we can also just observe that the limiting case \(x_F \to \infty \) must reduce to the ultrarelativistic approximation, so the prefactor must be the same.

% By comparison with the ultrare
% so we get \(P = k _{\text{UR}} n_e^{4/3}I(x_F)\), where we incorporated the integral in the term \(I(x_F)\): this is given by
%

% so if \(x_F \gg 1\) we have \(I(x_F) \sim 1\), the ultrarelativistic case,
In the nonrelativistic case, \(x_F \ll 1\), we have \(I(x_F) \sim 4 x_F / 5\), and since \(x_F \sim n_e^{1/3}\) this yields \(P \sim n_e^{5/3}\) as expected.
This expression \emph{interpolates} between the two limits.

Then, we can apply the same reasoning as before: we compare the pressure of the Fermi gas with the prediction of the Clayton model to find
%
\begin{align}
k _{\text{UR}} \qty(\frac{Y_e \rho_{c}}{m_H})^{4/3} I(x_F)
\approx \qty(\frac{\pi }{36})^{1/3} G M^{4/3} \rho_{c}^{4/3}
\,,
\end{align}
%
so we can extract the mass:
%
\begin{align}
M = I(x_F)^{3/2} M _{\text{Ch}}
\,,
\end{align}
%
where \(M _{\text{Ch}} \approx \SI{1.4}{M_{\odot}}\) is the Chandrasekhar mass we defined earlier.
% and the important thing is that \(x_F \propto n_e^{1/3} \propto \rho_{c}^{1/3}\).

% [Graph: on the \(x\) axis \(M/ M _{\text{CHANDRA}}\), on the \(y \) axis \(\rho_{c}\).]
\begin{figure}[ht]
\centering
\includegraphics[width=\textwidth]{figures/chandrasekhar_limit.pdf}
\caption{A plot of \(M / M _{\text{Ch}}\) against \(x_F \propto n_e^{1/3} \propto \rho _c^{1/3}\).}
\label{fig:chandrasekhar}
\end{figure}

% \todo[inline]{Quick and dirty Mathematica plot, to update (and flip the axes maybe).}
% If we increase the mass, the star is not able to support itself by the pressure due to being a gas of degenerate electrons.

We can see that as we increase the mass approaching \(M _{\text{Ch}}\) the central density diverges: this is not physically possible, of course, so as it gets higher some usually-prohibited process takes over; typically for white dwarfs this is electron capture, by which electrons and protons combine into neutrons, forming a neutron star.

We have not used any particular characteristics of electrons beyond their being fermions, so this line of reasoning may be used to also bound the mass of a neutron star, since neutrons are fermions as well.
The issue, as we will see, is that general-relativistic corrections become important in that case, since neutron stars have a much higher density.

% This gives us
% %
% \begin{align}
% M _{\text{CHANDRA }} = 3.1 Y_e^2 m_{*}
% = 5.8 Y_e^2 M_{\odot} = 1.4 M_{\odot}
% \,.
% \end{align}

\subsection{White dwarf characteristics}

% It can be shown that the mean density is around
Now that we have a model for the equation of state at the core of a fully degenerate object like a white dwarf, we can try to extract some of its characteristics: it is reasonable (from more complete studies of the object) to estimate the mean density as 1/6 of the central one,
%
\begin{align}
\expval{\rho } = \frac{1}{6} \rho_{c}
= \frac{\num{.51}}{Y_e^2} \qty(\frac{M}{m_{*}})^2
\frac{m_H}{( h / m_e c)^3}
\,,
\end{align}
%
so we can estimate the radius as
%
\begin{align}
R = \qty(\frac{3 M }{4 \pi \expval{\rho }})^{1/3}
\approx \num{.77} Y_e^{5/3} \qty(\frac{M}{m_{*}})^{1/3} \underbrace{\alpha_{G}^{-1/2} \frac{h}{m_e c} }_{\ell_{WD}}
\,,
\end{align}
%
% and the object at the end is
which is of the same order of magnitude as the characteristic length
%
\begin{align}
\ell_{WD} = \alpha_{G}^{-1/2} \frac{h}{m_e c} \approx \SI{3e7}{m} \approx \num{0.04} R_{\odot}
\,,
\end{align}
%
so, taking \(Y_e = 0.5\) we can express the radius as
%
\begin{align}
R = \frac{R_{\odot}}{74} \qty(\frac{M_{\odot}}{M})^{1/3}
\,.
\end{align}

% The luminosity is given by
Using the radius we can estimate the luminosity of the thermal radiation emitted by these bodies:
%
\begin{align}
L = 4 \pi R^2 \sigma T_E^{4}
= \frac{1}{74^2} \qty(\frac{M_{\odot}}{M})^{4/3}
\qty(\frac{T_E }{\SI{6000}{K}}) L_{\odot}
\,,
\end{align}
%
so if we take a typical effective temperature of around \SI{e4}{K} (recall that white dwarfs are in the blue part of the HR diagram), \(M = \num{.4} M_{\odot}\) we get \(L \approx \num{3e-3} L_{\odot}\): they are very dim.

\subsection{Neutron stars}

The first thing to consider when discussing neutron stars is the fact that neutrons, as we discussed, are usually unstable, with lifetimes on the order of \SI{10}{min}. How can a neutron star be stable then?
% We deal with degenerate stars: the Pauli exclusion principle plays a critical role, and the process
The process through which neutrons decay is:
%
\begin{align}
\ce{n} \rightarrow \ce{p} + \ce{e-} + \overline{\nu}_{e}
\,,
\end{align}
%
and the crucial fact is that neutron stars are composed of a degenerate neutron gas \emph{as well as} a degenerate \emph{ultrarelativistic} electron gas: the Fermi temperature is much higher than \(m_e c^2/ k_B\) \cite[eq.\ 1]{yakovlevNeutrinoEmissionNeutron}.
% When a neutron decays
While the electron gas is ultrarelativistic the neutron gas is not; the energy released by neutron decay is of the order of \SI{800}{keV}, so the momentum an emitted electron would have would be well within the Fermi sphere, which is already full!

Thus, neutron decay is inhibited; on the other hand, electron capture, which looks like
%
\begin{align}
\ce{e-} + \ce{p} \rightarrow \ce{n} + \nu_{e}
\,,
\end{align}
%
is favoured, and it can increase the number of neutrons.

We can look at the Saha formula to get numerical estimates for the equilibrium between these processes.
The chemical potential of neutrinos can be neglected, therefore we find
%
\begin{align}
\mu_{\ce{n}} = \mu_{\ce{p}} + \mu_{\ce{e}}
\,,
\end{align}
%
and, since as we saw earlier the chemical potential of a Fermi gas is its Fermi energy, the same equation holds for their Fermi energies:
% This means that a certain point we will have only neutrons.
% We need to exploit the Fermi exclusion principle. Why does the equation
%
\begin{align}
\epsilon_{F, \ce{n}} = \epsilon_{F, \ce{p}} + \epsilon_{F, \ce{e}}
\,.
\end{align}
%
% favour neutrons?

How does this translate into the number densities of the three constituents? We have the constraint that the number density of protons must equal the number density of electrons in order to ensure local neutrality, while there is no constraint on the ratio between neutrons and protons.
% , but the Fermi energy depends on the number density of these.
% typical numbers then become

We will not get into the calculation, but due to the slight mass imbalance \(m_n > m_p\) we have a large difference in the number densities: typically,
%
\begin{align}
n_{\ce{p}} = n_{\ce{e}} = \frac{n_{\ce{n}}}{200}
\,.
\end{align}

% We will have
Since the overwhelming majority of the particles in the neutron star are neutrons, we can make our calculations with the approximation that the number of neutrons per baryon is \(Y_n \approx 1\):
%
\begin{align}
n_{\ce{n}} = Y_n \frac{\rho_{c}}{m_{\ce{n}}} \approx \frac{\rho_{c}}{m_{\ce{n}}}
\,.
\end{align}

Typical values for these densities are \(\rho_c \approx \SI{2e17}{kg/m^3}\) and \(n_{\ce{n}} \approx \SI{e44}{m^{-3}}\).

% For a neutron star we have
With a similar reasoning to the one we applied to white dwarfs we can calculate
%
\begin{align}
\rho_{c}^{\text{NS}} \approx 3.1 \qty(\frac{M}{M_{*}})^2 \frac{m_n}{\qty(h / m_n c)^3}
\,,
\end{align}
%
% while for a white dwarf we have
which, due to the fact that \(m_{n} \gg m_e\), is much larger than the corresponding result for white dwarfs
%
\begin{align}
\rho_{c}^{WD} \approx \frac{3.1}{Y_e^{5}} \qty(\frac{M}{M_{*}})^2 \frac{m_H}{(h / m_e c^2)^{3}}
\,.
\end{align}
% \todo[inline]{why is the  \(c\), right?}

In both cases, we used the characteristic mass \(M_{*} = \alpha_{G}^{-3 /5 } m_n \approx 1.85 M_{\odot}\).

As we did before, knowing the central density we can estimate the average one, which then allows us to calculate the radius:
% and for the radius we get
%
\begin{align}
R = \num{.77} \qty(\frac{M_{*}}{M})^{1/3} \alpha_{G}^{-1/2}  \frac{h}{m_n c}
\,,
\end{align}
%
where the characteristic length is given by
%
\begin{align}
L_n = \alpha_{G}^{-1/2} \frac{h}{m_n c} \approx \SI{17}{km} \approx \frac{1}{1200} L_e
\,,
\end{align}
%
1200 times smaller than the corresponding length scale for white dwarfs (denoted with an \(e\) for ``electron degeneracy'').
% \(L_n\) is the characteristic length scale for a NS, \(L_e\) is the characteristic length for a white dwarf.

Finally, we can compute a maximum mass: \(M _{\text{max}}^{\text{NS}} = 3.1 M_{*} = 5.8 M_{\odot}\).
% This is the mass of a star \emph{remnant}, which encompasses mass from the core only: the initial star will be much larger.

% Can this NS become a BH? we need to compute
We have neglected general-relativistic effects, but would they be relevant? The quantity we need to compute is the ratio of the Schwarzschild radius to the actual radius of the NS:
%
\begin{align}
\frac{R_{\text{Schw}}}{R} =
\frac{2GM}{Rc^2} \approx \num{.4} \qty(\frac{M}{M_{*}})^{4/3}
\,,
\end{align}
%
which is large, of order 1! We have not computed a minimum mass for a neutron star, but typically their mass is of the order of the Chandrasekhar mass, \(M^{NS} \approx 1.4 M_{\odot}\) because of how they form in supernovae (so, \(M / M_{*} \sim 1\)).
% qualitatively we can say that if it is too small then the degenerate electron gas may not be relativistic enough to stop neutron collapse.
The neutron star might not be small enough to actually collapse into a black hole, but surely general relativity must be considered when describing its dynamics.
% so if the mass is large enough we can reach the critical value of \(GM / Rc^2 = 2\).

Neutron stars were first detected as very regular radio pulses: \emph{pulsars}. These are due to the very strong (\(\sim \SI{e8}{T}\)) magnetic fields accelerating particles in beams aligned with the magnetic poles of the NS; these are not aligned with the rotation axis of the NS, so they constantly change the direction of emission, and the Earth can happen to be in this cone.

In order to estimate how fast these pulses can be (in a classical and rough way), let us assume that the NS is rotating barely below a speed which would disintegrate it, so that its binding energy equals its rotational energy: neglecting the order-1 numerical factors we have
% We have
%
\begin{align}
\frac{GM^2}{R} \sim M R^2 \omega^2 _{\text{max}}
\,,
\end{align}
%
% the maximum angular velocity which can be supported gravitationally: it comes out to be
we can then compute the minimum period using the expression we have for the radius in terms of \(M\) and \(M_{*}\):
%
\begin{align}
\tau _{\text{min}} = \frac{2 \pi }{\omega _{\text{max}}}
\approx 2 \pi \qty(\frac{R^3}{GM})^{1/2}
\approx
11 \qty(\frac{M_{*}}{M}) \alpha_{G}^{-1/2} \frac{h}{m_n c^2}
\approx \num{.6} \frac{M_{*}}{M} \SI{}{ms}
\,.
\end{align}

The signals produced by pulsars are of this order of magnitude --- we have observed ``millisecond pulsars'', so NSs do indeed rotate close to these extremely high rates.

Neutron stars can also produce gravitational waves in their rotation, as long as they have a slight asymmetry (a ``mountain'', although their typical sizes are of the order of centimeters); these would have frequencies in the Hz range. We cannot detect these with ground-based detectors, but we might be able to do so with space-based ones.
% This gives us a bound for gravitational waves of astrophysical origin.
% \todo[inline]{no comment on the GR corrections to this formula: however I'd expect them to be significant}

\subsection{Relativistic corrections to the equation of state}

% We move on to \emph{the GR issue}.
Black holes are objects which are so dense that their radius is smaller than the Schwarzschild radius:
%
\begin{align}
R < \frac{2GM}{c^2} = R _{\text{Sch}}
\,.
\end{align}

General Relativity predicts that when this is the case an \textbf{event horizon} form, a surface which forms a causal boundary: as is almost cliché, \emph{not even light can escape}.

Let us see how the classical description of a star fails for objects with relativistic masses.
The equation of hydrostatic balance, derived under classical assumptions, reads:
%
\begin{align}
\dv{P}{r} = - \frac{Gm \rho }{r^2}
\,.
\end{align}
%
% while the GR equations for this are the TOV equation:

If we seek a relativistic analogue under similar assumption (spherical symmetry and equilibrium) we find the \textbf{Tolman-Oppenheimer-Volkov} equation:
%
\begin{align}
\dv{P}{r} = - \frac{Gm  \rho }{r^2} \qty(1 + \frac{P}{\rho c^2}) \qty(1 + \frac{4 \pi r^3 P}{m c^2}) \qty(1 - \frac{2Gm}{rc^2})^{-1}
\,.
\end{align}
%

This equation is exact under the assumptions we mentioned.
In the classical limit (\(2Gm/ c^2 \ll r\) and \(P \ll \rho c^2\)) this reduces to the classical hydrostatic balance equation.
The first correction is reminiscent of the second Friedmann equation: \emph{the pressure itself contributes to the inertia of the system}.

Let us see how their predictions differ assuming constant density, \(\rho \equiv \rho_0 \). In the Newtonian case the mass below a radius \(r\) is given by
% If we have constant density, in the Newtonian case we get
%
\begin{align}
m(r) = \frac{4 \pi }{3} \rho_0 r^3
\,,
\end{align}
%
% so than we can integrate and get
using which we can integrate the hydrostatic balance equation (from the surface, where the pressure vanishes) to find:
%
\begin{align}
P(r)
= \int_{R}^{r_0 } \underbrace{\qty(- \frac{Gm \rho }{r^2})}_{\dv*{P}{r}} \dd{r}
= \frac{2 \pi G}{3}  \rho_0^2 \qty(R^2 - r_0^2)
\,.
\end{align}
%
% where we inserted the boundary condition \(P(R) = 0\).

Then, the central pressure is given by
%
\begin{align}
P_c^{\text{classical}} = \frac{2 \pi }{3} G \rho_0^2 R^2 = \qty(\frac{\pi }{6})^{1/3} G M^{2/3} \rho_c^{4/3}
\,.
\end{align}

Keeping the constant-density assumption, this can be done analytically in the relativistic case as well! We get
%
\begin{align}
P(r) &= \rho_0 c^2 \qty(\frac{(1 - 2GM r^2 / R^3 c^2)^{1/2} - (1 - 2GM / Rc^2)^{1/2}}{3 \qty(1 - 2GM / Rc^2)^{1/2} - \qty(1 - 2GM r^2 / R^3c^2)^{1/2}})  \\
P_c^{\text{relativistic}} &= \rho_0 c^2 \frac{1 - \sqrt{1 - \frac{2GM}{Rc^2}}}{3 \sqrt{1 - \frac{2GM}{Rc^2}} - 1}
\,.
\end{align}

%
% \todo[inline]{Check exponents}
% so we get
% %
% \begin{align}
% P_c = \frac{2 \pi }{3} G \rho_0^2R^2 = \qty(\frac{\pi }{6})^{1/3} GM^{2/3} \rho_{c}^{4/3}
% \,,
% \end{align}
% %
% so we can look at what happens when we consider \(r=0\): we get
% %
% \begin{align}
% P_c = \rho_0 c^2 \qty(\frac{1 - \sqrt{1 - 2GM / Rc^2}}{3 \sqrt{1 - 2GM / Rc^2} - 1})
% \,,
% \end{align}
% %

% so we can see that the central pressure is finite as long as
In the classical model we had a finite central pressure for each value of the mass and density; now, instead, in order for the pressure to not diverge we must require
%
\begin{align}
% \frac{2GM}{Rc^2} < \frac{8}{9}
R > \frac{9}{8} \frac{2GM}{c^2}
\,.
\end{align}
%
% which is not \(1/2\) since we have made some approximations.

This is known as the \textbf{Buchdahl bound}; the radius of a non-black-hole (with constant density) cannot be arbitrarily close to the Schwarzschild radius, it must be at least \SI{12.5}{\percent} larger in order for the pressure to not diverge at the center.\footnote{
More stringent bounds can also be derived --- in \textcite[fig.\ 2]{lattimerNeutronStarObservations2007} a plot is shown of possible equations of state of neutron stars in a mass versus radius plane.
The bound we derived is denoted as \(P < \infty \), and we can also see a ``causality'' bound, which is related to the speed of sound in an ultrarelativistic medium.
Realistic equations of state can reach approximately \(R \gtrsim 3GM /c^2\).}

% \todo[inline]{A mass bound for neutron stars follows, both in Pacciani's notes and in the , but I don't follow the logic.}
This also yields a mass limit for neutron stars. Let us estimate the constant density \(\rho_0\) by assuming that each neutron takes up a sphere of radius its Compton wavelength: \(r_n \approx h / m_n c\), so
%
\begin{align}
\rho_0 \approx \frac{m_n}{ \frac{4 \pi }{3} r_n^3}
\approx \frac{3 m_n^4 c^3}{4 \pi h^3}
\,.
\end{align}

A more accurate estimate would be given by \(r_n \approx \num{.7} h / m_n c\).

so that, using the fact that \(\rho_0 = M / ( 4 \pi R^3 / 3)\), we can start manipulating the Buchdahl bound:
%
\begin{align}
M &< \frac{4 c^2}{9GR} = \frac{4 c^2}{9 G} \qty(\frac{4 \pi \rho_0 }{3M})^{1/3}  \\
M^{2/3} &< \frac{4 c^2}{9G} \qty(\frac{4 \pi }{3} \frac{3 m_n^4 c^3}{4 \pi h^3})^{1/3} \\
M  &< \qty(\frac{8 \pi}{9})^{3/2} M_{*}
\,.
\end{align}

This yields a bound on the order of \(M \lesssim 5 M_{\odot}\).
% Then we have a bound
% %
% \begin{align}

% \,,
% \end{align}
% %
% with \(f \sim 1\).
% This means than the objects becomes contained inside its Schwarzschild radius, \(R = 2GM/c^2\).

% Tomorrow we will speak of galaxy formation.

\end{document}


% This file is part of the OWL project.
% Copyright 2014 the authors.

% ## notes:
% - Use ``star'' not ``source'' for the target of the OWL.

% ## to-do:
% - Should I include detrendings like PDC and median filtering?
% - Needs affiliations etc.
% - Finish zeroth draft.
% - Send for comments to Lupton, Carey, Wright, Brandner, Loredo
% - Who is in charge of the PanSTARRS variability field?

\documentclass[12pt, letterpaper, preprint]{aastex}

\usepackage{color,hyperref}
\definecolor{linkcolor}{rgb}{0,0,0.5}
\hypersetup{colorlinks=true,linkcolor=linkcolor,citecolor=linkcolor,
            filecolor=linkcolor,urlcolor=linkcolor}

\newcommand{\documentname}{\textsl{Article}}
\newcommand{\project}[1]{\textsl{#1}}
\newcommand{\foreign}[1]{\textsl{#1}}
\newcommand{\etal}{\foreign{et\,al.}}
\newcommand{\transpose}[1]{{#1}^{\!\mathsf T}}
\newcommand{\inverse}[1]{{#1}^{-1}}
\newcommand{\E}[1]{\mathrm{E}[{#1}]}
\newcommand{\Var}[1]{\mathrm{Var}[{#1}]}
\newcommand{\Covar}[1]{\mathrm{Covar}[{#1}]}
\input{vc}

% affils
\newcommand{\ccpp}{1}
\newcommand{\mpia}{2}
\newcommand{\contact}{3}
\newcommand{\courant}{4}

\begin{document}\sloppy\sloppypar

\title{
  Optimized ``soft'' aperture photometry for time-domain astronomy
}
\author{
  David W. Hogg\altaffilmark{\ccpp,\mpia,\contact},
  Dan Foreman-Mackey\altaffilmark{\mpia},
  Jonathan Goodman\altaffilmark{\courant}
}
\altaffiltext{\ccpp}{%
  Center for Cosmology and Particle Physics,
  Department of Physics, New York University}
\altaffiltext{\mpia}{%
  Max-Planck-Institut f\"ur Astronomie}
\altaffiltext{\contact}{%
  \texttt{david.hogg@nyu.edu}}
\altaffiltext{\courant}{%
  Courant Institute for Mathematical Sciences,
  New York University}

\begin{abstract}
Precise stellar photometry campaigns by \project{Kepler}, \project{Spitzer}, and \project{PanSTARRS}
  (among many others)
  create precise lightcurves by fixing the star in detector coordinates,
  and estimating brightness with a fixed linear combination of background-subtracted pixel values.
Under any complete set of assumptions about the noise in the detector,
  there is a particular choice of pixel weights for use in the brightness estimate
  that maximizes signal-to-noise.
When precision requirements are extremely high,
  the noise model must necessarily include pointing and point-spread-function variations,
  which introduce highly correlated noise at the pixel level.
Here we generalize previous results on optimized photometric estimators
  to include arbitrarily correlated pixels.
Amusingly, the approach we advocate does not in the end
  require any advance knowledge of any noise source;
  once there are enough epochs observed,
  the pixel-level data are sufficient to build an empirical data-driven correlated noise model.
The optimized pixel weights can be found analytically;
  when they are used to perform photometry, they work extremely well.
We demonstrate enormous improvements to the \project{Kepler} photometry of isolated stars.
The Optimized-Weight Linear (OWL) photometry produces photometric measurements so uniform that
  they outperform or obviate much more heuristic (and much more scientifically damaging)
  ``detrending'' schemes in many real cases.
In particular, OWL photometry minimizes instrument-induced variance
  but preserves all intrinsic stellar variability.
All the OWL code is available under an open-source license.
\end{abstract}

\keywords{
  instrumentation: photometers
  ---
  methods: data analysis
  ---
  methods: statistical
  ---
  techniques: photometric
}

\section{Rules of the photometry game}

Precise photometry is the bread-and-butter of the exoplanet community.
The unprecedented precision of the \project{Kepler} photometry has led to the discovery of
dozens of Earth-radius planets (HOGG CITES).
\project{Kepler}-level (part in $10^5$) precise photometry is, no doubt,
  going to be of great importance in many other areas as well.

Imagine you are trying to do very sensitive relative photometry on a variable source.
The most precise methodologies at present involve fixing the position of the source on the detector
  (cite examples)
  to reduce dependency of the results on the flat-field or flat-fielding errors.
They also often involve defocusing the telescope
  (cite examples),
  to illuminate the pixels more uniformly
  (and hence reduce intra-pixel sensitivity issues),
  avoid saturation,
  and make the observations less atmosphere-dependent.
There are also strategies related to calibration or the choices of comparison stars
  (cite examples).
Once all these choices are made,
  the choice of methodology for actually photometering the source remains.
Most projects choose something akin to aperture photometry
  (cite examples),
  in which the pixels within an aperture are co-added with unit weights,
  and pixels outside the aperture are not used at all.
Here we ask only about this last choice:
What is the best way to measure the brightness of a nearly constant source,
  given a set of precise images of the source taken over time?

FOR THE ABOVE CHECK TWITTER FOR REFS AND THANK TWEEPS.

We are going to answer this question using optimization, of course;
  we are going to find the ``best'' method.
We will assume that the investigator obeyed the usual rules,
  so he or she fixed the position of the star on the detector,
  possibly defocused the telescope,
  and avoided saturation.
However, we are also going to assume---%
  as is \emph{always the case}---%
  that there are some residual,
  unaccounted-for positional offsets (jitter) and point-spread-function changes.
These offsets and changes could be random or systematic over time,
  but we will assume that they are unknown \foreign{a priori}.
That is, the telescope house-keeping data are not good enough to track them at the level we care about.
We are trying to measure \emph{extremely} precise photometry,
  at the level of $10^{-4}$ or $10^{-5}$ or better.

The conditions we have assumed might seem strange,
  but they are generic for the most precise photometry systems currently operating.
\project{Kepler}, for instance, has all of these problems,
  as does [insert projects here, like Wright's].

It is possible to think of this problem as the problem of finding the best strategy in a single-player game.
\emph{The Rules} of this game are as follows:
\begin{itemize}
\item
  There is copious multi-epoch single-band imaging,
  with many epochs (many images),
  and a known position in that imaging of a star of great interest (the ``object'' star).
  There is also perhaps some ``pixel mask'' indicating what pixels are permitted for use
  in the vicinity of that star.
\item
  There may or may not be a list of ``comparison'' stars
  that are used to calibrate or ratio the photometry of the object star.
  If there aren't, then it can be assumed that the imaging is very well calibrated.
\item
  The imaging is background- and foreground-subtracted,
  such that the mean level of the smooth foregrounds and backgrounds in the imaging is zero.
  The background level might vary or the subtraction of it might not be perfect,
  but we can assume that it is has a mean of zero across pixels and epochs.
\item
  It is expected that the star is fairly stationary in device coordinates (pixel location).
  However, in each image it has been offset slightly by some amount from its fiducial position.
  There are no reliable meta-data to help you understand those small offsets.
  Any comparison stars are also subject to small offsets;
  the comparison-star offsets are not necessarily the same as the object-star offsets,
  because there is camera rotation, variable optical distortion, proper motion, and parallax.
\item
  It is expected that the point-spread function (PSF) is fairly constant.
  However, again, in each image it has a small difference from the fiducial point-spread function.
  There are no reliable meta-data to help with this,
  and the comparison star PSFs will also vary but not in precisely the same way,
  though their variations might be non-trivially correlated.
\item
  There are no reliable meta-data about read noise or gains,
  so there is no precise noise estimate for any pixel at any epoch.
\item
  When you photometer a star,
  the only operation you are permitted is linear weighted sums of pixels in the images.
  Furthermore, any weighted sums must be precisely identical from epoch to epoch.
  That is, the ``aperture''
  (which in fact may be some complicated set of pixel weights)
  is required to be identical from image to image.
\end{itemize}
These rules are generic;
  this is a common situation for precise, multi-epoch photometry.

The only rule to which we object is the last one:
In principle it should be better to do focal-plane modeling
  and perform photometry as a maximum-likelihood value
  or some statistics of a posterior PDF
  based on a parameterized likelihood function (for example, \citealt{hoggwhitepaper}).
However, there are many applications in which
  the physical information one would like for building such a model is not available.
We would parameterize and fit nonetheless,
  but most astronomical projects don't take this approach at present.
There are also situations in which the output of full likelihood fitting for the data
  shows correlations between the final photometry and nuisance parameters (such as PSF parameters).
In these cases, the investigator is left wondering if the residual photometric variance
  has contributions from nuisance-parameter variance or fit residuals or model mismatch.

There have been some previous attempts to optimize photometric estimators subject to \emph{The Rules},
  in generic (but limiting) situations.
When the pointing and point-spread function are both known precisely,
  the pixel noise variance is known for every pixel,
  and the pixel noise is independent (uncorrelated) pixel-to-pixel,
  there are simple analytic results (\citealt{naylor}).
These results reduce to the psf-as-matched-filter standard practice
  in the further simplification of sky-dominated noise.
There are no previous results (to our knowledge) that take account of
  the highly covariant noise created by variable pointing or optics.
This is the first primary novelty of this work.
The second primary novelty is the realization that the noise---%
  even the covariant part created by optics and pointing---%
  can be determined empirically from the data themselves,
  with no prior knowledge of the characteristics of the detector,
  telescope, or atmosphere.

\section{Well-calibrated data}

In the (possibly fantastical) situation in which the imaging data are all properly calibrated,
  we have no need for comparison stars.
We are going do the photometry using nothing but the pixels near the object star.

Before we say what ``properly calibrated'' means,
  it is necessary to specify what we mean by an ``image''.
Ordinarily it is our view that
  an image is a measurement of the smoothed intensity field
  or smoothed photon phase-space density.
It ought to have units of energy per time per area per solid angle
  or photons per time per area per solid angle.
The problem with this view of an image
  is that if we do aperture photometry---%
  that is, if we make linear weighted sums of image pixels---%
  to get a total stellar flux,
  the weights in the weighted sum must have units of \emph{solid angle}.
Small changes to the optics or atmosphere will change the astrometric mapping of the camera,
  changing the solid angles of the pixels.
Aperture photometry is the wrong thing to do in this formulation of an image;
  or it is an okay thing to do if the weights are permitted to vary with the astrometric distortion.
That would violate \emph{The Rules} above.

So the image we want for these purposes is not a measurement of the intensity field,
  but rather a kind of ``flux map''.
It ought to have units of energy per time per area
  or photons per time per area.
These values can be co-added with dimensionless weights to deliver a stellar flux.
The only difference between an intensity image and a flux map is a factor of
  the angular size of the pixels.
In many imagers pixels are similar in size so these differences aren't apparent.
However, at the levels of precision we care about here
  ($10^{-4}$ or $10^{-5}$ or better)
  these details matter.

What is meant by ``properly calibrated''?
The image (flux map) is properly calibrated when it is can be treated as a measurement of---%
  or is directly proportional to---%
  a smoothed map of energy or photon number per time per area.
It is ``smoothed'' because it is convolved (or correlated) with some finite
  and possibly spatially varying PSF
  (which we don't know).

Are any imaging data properly calibrated according to these criteria?
We don't know, but the \project{Kepler} Satellite may be.
The standard data products of the \project{Hubble Space Telescope}
  are given in intensity units,
  but the astrometric solution is known to high precision,
  so (with a tiny bit of work)
  these also can be properly calibrated for aperture photometry according to this definition.

We do not endorse the view that images should be presented as flux maps!
We are just sayin' that \emph{if} you need to obey \emph{The Rules},
  then you need to convert your intensity maps into flux maps before playing the game.

\section{Optimized photometry}

We have at each of $N$ epochs $n$ a set of $D$ pixels in a small patch
  that contains a star.
The set of measured, properly calibrated (see above),
  correctly background-subtracted (see above)
  pixel values at epoch $n$ form a $D$-dimensional column vector $y_n$.
According to \emph{The Rules}, all we are permitted to do at epoch $n$
  is produce a weighted sum
\begin{eqnarray}
q_n &\leftarrow& \transpose{w}\, y_n
\quad ,
\end{eqnarray}
  where $q_n$ is a scalar photometric measurement,
  $w$ is a $D$-dimensional column vector of weights,
  and (because of the transpose)
  the product is effectively an inner (scalar or dot) product.
\emph{The Rules} state that the weight vector $w$ must not depend on $n$ or anything else;
  the weights are the same at every epoch.

We can see $q_n$ as an estimator---a point estimate---of the stellar brightness.
If we knew the expectation value (mean) for the pixel data $y_n$ and also the
  variance of those data around the mean,
  we could predict the expectation and variance of the estimator.
The expectation value would be:
\begin{eqnarray}
\E{q_n} &=& \transpose{w}\, \mu_n
\\
\mu_n &\equiv& \E{y_n}
\\
\Var{q_n} &=& \transpose{w}\, C_n\, w
\\
C_n &\equiv& \Covar{y_n}
\quad ,
\end{eqnarray}
  where $\E{x}$ is the expectation value of $x$,
  $\Var{x}$ is the variance of (a scalar) $x$ around its expectation value,
  and $\Covar{x}$ is the $D\times D$ symmetric, positive definite covariance matrix
  of (a vector) $x$ around its expectation value.
In the simple case of independent noise from pixel-to-pixel,
  $\Covar{y_n}$ becomes diagonal,
  and the variance $\Var{q_n}$ becomes simply the sum of the diagonal variances,
  weighted by the squared weights.
The expected squared signal-to-noise ratio $\Sigma^2$ for $q_n$ is
\begin{eqnarray}
\Sigma^2 &=& \frac{[\transpose{w}\, \mu_n]^2}{\transpose{w}\, C_n\, w}
\quad .
\end{eqnarray}

It is almost always assumed that the pixel-to-pixel noise is independent (diagonal);
  why do we not assume this here?
The answer lies in \emph{The Rules}:
Variations of the pointing and point-spread function (and background),
  which are all unknown,
  correlate the pixels.
That is, they produce a kind of noise that is neither photon nor read noise in the detector.
They produce a jitter noise that leads to covariant variations of the individual pixel values.

We win the game if we find the weight vector $w$ that optimizes signal-to-noise.
It can't optimize the signal-to-noise at epoch $n$;
  it must optimize the mean or typical signal-to-noise across epochs.
That is, we want to find the weight vector $w$ that optimizes
\begin{eqnarray}
  \Sigma^2 &=& \frac{[\transpose{w}\, \mu]^2}{\transpose{w}\, C\, w}
  \quad ,
\end{eqnarray}
where now $\mu$ is the expectation value of the data vectors $y_n$ but now over \emph{all} epochs,
  and $C$ is the covariance matrix for the data but again over \emph{all} epochs.
The signal-to-noise-optimized weight vector is simply
\begin{eqnarray}
  w &\leftarrow& \inverse{C}\, \mu
  \quad ,
\end{eqnarray}
  where the inverse operation is the full matrix inverse.
This optimized-weight result is the natural generalization to correlated noise
  of the equivalent result (for independent pixel noise)
  obtained previously (\citealt{naylor}).

Previous work has called these weights ``optimal'' but they are only really ``optimized''.
For one, the word ``optimal'' always begs the question ``with respect to what?''
For another, we reserve the word ``optimal'' for estimators
  that have some glimmer of hope of saturating the Cram\'er--Rao bound (CITE).
The optimized estimators presented here will not do this in general
  (and nor will those of \citealt{naylor}),
  because they are not based on any reasonable likelihood function.

By assumption, we don't know the point-spread function,
  nor the noise model,
  nor the pointing or PSF jitter.
Therefore, we can't compute $\mu$ and $C$ from first principles; we have to estimate them.
Fortunately, we have $N$ epochs of data;
  we can compute a (suitably robustly estimated)
  \emph{empirical} mean $\hat{\mu}$ and \emph{empirical} covariance $\hat{C}$
  of the original data vectors $y_n$.
That is, we will do something like
\begin{eqnarray}
  \hat{\mu} &\leftarrow& \frac{1}{N}\,\sum_{n=1}^N y_n
  \\
  \hat{C} &\leftarrow& \frac{1}{N-1}\,\sum_{n=1}^N [y_n - \hat{\mu}]\,\transpose{[y_n - \hat{\mu}]}
  \\
  \hat{\Sigma}^2 &=& \frac{[\transpose{w}\, \hat{\mu}]^2}{\transpose{w}\, \hat{C}\, w}
  \\
  \hat{w} &\leftarrow& \inverse{\hat{C}}\, \hat{\mu}
  \quad ,
\end{eqnarray}
  where the operations are averages over the epochs,
  the vector product is the outer or tensor product,
  $\hat{\Sigma}^2$ is the \emph{empirical} signal-to-noise,
  and $\hat{w}$ are the weights that maximize it.
Technically we require $N \gg D$
  (far more epochs than there are pixels in the tiny patch).
It turns out that with small modifications
  (to make the means robust to outliers)
  this works well, at least for \project{Kepler} data (see below).
The Optimized-Weight Linear (OWL) photometry $q_n$ is then
\begin{eqnarray}
  q_n &\leftarrow& \transpose{\hat{w}}\, y_n
  \quad ,
\end{eqnarray}
  that is, one scalar $q_n$ per epoch $n$, obtained by a weighted sum of the $y_n$.
We have not just obeyed \emph{The Rules},
  we have come up with the best possible strategy in terms of empirical signal-to-noise.

There is one overall scale that is free;
  any scalar multiple of the weight vector $\hat{w}$ is identical in signal-to-noise
  to any other scalar multiple.
In practice it doesn't matter how we choose this unless either machine precision is an issue,
  or else some kind of absolute aspects of photometry are required.

\section{\project{Kepler} data}

For each star in the Kepler data (identified by a KIC number),
  there is a small patch of telemetered pixels,
  and some 4000 epochs per quarter and up to 17 quarters of data.
The patch of pixels has an associated mask,
  which indicates both which pixels are good data,
  and also which were summed to make the official pipeline
  Simple Aperture Photometry (SAP).
The weights used to generate the SAP photometry are constant within a quarter,
  and all snapped to either zero or unity.
\figurename~\ref{fig:pixels} shows time histories for the $D=19$ pixels relevant
  to KIC~3335426 from the fifth quarter in which it was observed.
Also shown are the three pixels that got unit weight in the SAP photometry for this star.

The \project{Kepler} data are extremely good, but there are some outliers,
  presumably from cosmic rays.
We want to compute empirical means and covariances that are not overly drawn by these outliers.
We compute the empirical statistics with an iterated sigma-clipping.
In each iteration, we censor (remove from the sum)
  any datum $y_n$ that is more than a ``4 sigma'' outlier
  given the empirical mean $\hat{\mu}$ and covariance $\hat{C}$ computed in the previous iteration.
To determine the 4-sigma criterion, we use the Gaussian approximation to the chi-squared distribution
  with the appropriate number of degrees of freedom.
We iterate this sigma-clipping through five iterations,
  which in practice seems to lead to convergence.
Importantly, we sigma-clip at the epoch level, not the pixel level.

\figurename~\ref{fig:images} shows the robustly estimated empirical mean image $\hat{\mu}$,
  the diagonal elements of the empirical covariance matrix $\hat{C}$,
  the top eigenvector of the covariance matrix,
  and the OWL weights, along with the weights used by the \project{Kepler} pipeline,
  and a few data examples.
We resolved the overall scale degeneracy for the weights (see above)
  by requiring that the OWL and SAP photometry produce identical median flux.
The optimized weight vector shown in \figurename~\ref{fig:images} might seem counterintuitive;
  it puts substantial weight off the peak of the mean intensity.
The fundamental reason is that the weight vector is trying to get as orthogonal as possible
  to the high-variance eigenvectors of the variance tensor.
In the end, the highest-weight pixels do not all deliver significant signal;
  in \figurename~\ref{fig:images} we also show
  the pixel-wise product of the OWL weights with the empirical mean values,
  which gives a sense of where the bulk of the photometric signal is coming from.

\figurename~\ref{fig:results} shows the \project{Kepler} SAP flux for KIC~3335426,
  along with the OWL and some variants thereof (described below).
The OWL photometry very effectively  detrends the data,
  even though it involves \emph{no explicit detrending}.
By ``detrending'' we mean filtering or fitting-and-subtracting
  the SAP photometry (as in the \project{Kepler} PDC or median-filtered photometry).
Detrendings remove intrinsic stellar variability and create artificial epoch-to-epoch correlations
  in the photometric noise;
  they are the bane of exoplaneteers.
That is, the OWL is just a soft-aperture aperture photometry, identical from epoch to epoch
  that obviates detrending.
It is simply the weighting of the pixels that optimizes the empirical output signal-to-noise.

In detail, this first shot at OWL photometry is a tiny bit noisier
  in an epoch-to-epoch sense
  than the SAP photometry,
  despite being far better in its overall trend.
This is coming from the fact that the OWL weights are trying to suppress the dominant
  sources of variance,
  which are entering the data on long time-scales.
The focus on the biggest variance components limits what the OWL can do.
This opens up the next area of discussion:
How do we estimate the covariance matrix $\hat{C}$ so that it is dominated by the sources of variance
  that we most want to suppress?

\section{Estimation of the covariance tensor}

HOGG:
Something about what is really happening (trying to zero out dominant eigenvectors).

HOGG:
Something about the lack of prior knowledge:
\begin{itemize}
\item
We aren't assuming \emph{anything} about the sources of variance.
\item
We aren't doing \emph{anything} about intrinsic variability of the source.
\item
Some intrinsic variability we do care about, some we don't.
\item
We aren't using the time-ordering of the data at all;
  different effects happen on different time scales.
\end{itemize}
All these issues turn into decisions or knobs that we can turn in variations.
The variations will fall into two categories:
We can change how we estimate the covariance matrix $C$,
  or we can change how we constrain or regularize the weights $w$.

HOGG:
On the covariance estimation, here are some basic ideas:
\begin{itemize}
\item
Use local differences of pixel values instead of pixel values.
\item
Mix the dominant eigenvectors from the full set of pixel values
  with the small-scale noise inferred from the local differences.
\item
Perform some kind of factor analysis that splits the variance into
  ``diagonal'' and ``covariant'' parts.
\item
Use covariances across pixel patches from different stars
  to identify extrinsic (rather than intrinsic)
  sources of variances.
\end{itemize}

HOGG:
\begin{eqnarray}
\hat{C}_S & \leftarrow & \frac{1}{N-1}\,\sum_{n=1}^N [y_n - \hat{\mu}]_\perp\,\transpose{[y_n - \hat{\mu}]}_\perp
\\
x_\perp & \equiv & x - \sum_{k=1}^K u_k\,\transpose{u}_k\,x
\\
\delta_{kk'} &=& \transpose{u}_k\,u_{k'}
\quad ,
\end{eqnarray}
where $S$ is a subspace of the full $D$-dimensional pixel space,
  the $\perp$ subscripts indicate that the vectors have been projected into the subspace $S$,
  the $u_k$ vectors are orthonormal unit vectors that span the complement of subspace $S$,
  and $\delta_{kk'}$ is the integer (index) delta function.

%% HOGG:
%% In yet another variation, we estimated $\hat{\mu}$ and $\hat{C}$ using not the
%%   straight-up data, but the \emph{differences} between each data point and
%%   its time-ordered neighbor.
%% Again with the sigma clipping.
%% This leads to the DOWL, DOPW, and DTSA, shown in \figurename~\ref{fig:dresults}.

\section{Regularization of the weight vector}

It is also the case, in the above, that
  we aren't putting in any prior knowledge about what the $w$ image ought to look like
  in sane cases.
On the weight constraints and regularization, here are some basic ideas:
\begin{itemize}
\item
Require non-negativity.
\item
Require smoothness.
\item
Require similarity to the SAP aperture or some other sensible aperture.
\item
Require that the aperture be a circular top-hat or a smoothed version of that.
\end{itemize}

After running the OWL on multiple \project{Kepler} targets,
  we developed a few intuitions that led to variations.
In one, we constrain the weights to be positive;
  this protects against the optimizer finding locations in weight-space
  that perform subtractions of large numbers to reduce variance.
This problem arises most strongly when the star being photometerd is intrisically variable
  with a large amplitude:
In these cases, the largest variance can be from the star itself;
  the OWL tries to find weights that cancel out the target star!

The Optimized Positive-Weight (OPW) photometry works still at the optimum of
\begin{eqnarray}
\hat{\Sigma}^2 &=& \frac{[\transpose{w}\, \hat{\mu}]^2}{\transpose{w}\, \hat{C}\, w}
\quad ,
\end{eqnarray}
but now subject to the constraint that every element of $w$ must be non-negative.
\figurename~\ref{fig:images_opw} and \figurename~\ref{fig:results}
 show the results of the OPW for KIC~3335426,
 and \figurename~\ref{fig:results_variable} show the results of the OPW
 for a more variable star.
The positivity constraint seems salutory when the target star is somewhat variable.

In another variation, we looked at stars that saturate the \project{Kepler} detectors.
These stars show bleed trails along the CCD columns,
  although the \project{Kepler} gains have been set cleverly
  to ensure that precise photometry of these saturated stars is nonetheless possible with SAP photometry.
In the case of saturated stars, the OWL and OPW photometry do not work well
  because the saturation leads to some pixels being exceedingly constant,
  even if the source is intrinsically moving or varying.
That is, the saturated pixels do not show much empirical variance,
  and get overly large weight in the optimization.

For these saturated cases we devised a Tweaked Simple Aperture (TSA) photometry,
  which capitalizes on the clever apertures chosen by the \project{Kepler} team
  to capture saturated signal:
In the TSA, we lump together the SAP-coadded pixels into a single super-pixel.
We then perform the OWL methodology,
  but now with the list of pixels not being the full list of pixels,
  but rather the super-pixel plus the list of all the non-SAP pixels in the patch.
The TSA is a generalization of the OWL to non-square or non-uniform pixels.

We show the TSA results in \figurename~\ref{fig:results}.
It seems to be producing better epoch-to-epoch scatter;
  this might because it represents an even stronger regularization of the weight image $\hat{w}$.

HOGG:
In yet another variation, we fix the aperture to be a circular top-hat,
  and only permit its center and radius to be adjusted in the optimization...

NOTE TO JASON WRIGHT: WE CAN HELP YOU!

\section{Comparison stars}

HOGG:  GIVE THE SETUP

HOGG:  You can operate the OWL for all $K$ stars.
Then scale each star so it predicts as well as possible the object star.
Then take robust means or medians of the $K$ predictors and divide them out.

HOGG:  Or you can use all of the pixels in all the patches around all $K$ stars
  to build a predictive model for the pixels in the object star's patch.
This permits creation of a very stable data-driven calibration of the data.
We are pursuing this in a separate project;
  it is not even close to being ready for prime time.

\section{Discussion}

What happened?
We permitted the aperture to be soft.
We optimized the aperture weighting.
We obviated detrending forever.

In the results shown in \figurename~\ref{fig:results},
  the instrumental effects on the lightcurve were removed beautifully by the OWL,
  but in detail
  the adjacent epoch-to-epoch noise was inflated a tiny bit;
  why?
The OWL knows nothing about time ordering of the data;
  it makes no distinctions between slow and fast variations;
  it is just trying to maximize signal-to-noise.
Slow variations like those shown in \figurename~\ref{fig:results}
  are just another source of total variance.
If a user knows a lot about what timescales are relevant,
  the OWL can be improved enormously.
To demonstrate, we ran an experiment in which we filtered the \project{Kepler} pixel-level data
  for KIC~XXXX in quarter XXXX
  with a 48-hour median filter to remove the long-timescale variability.
We subtracted the median filter output from each pixel lightcurve and added back in the mean.
The pixel lightcurves are shown in \figurename~\ref{fig:pixels_filter}
  and the output of the OWL photometry is shown in \figurename~\ref{fig:results_filter}.
Because the pixels were partially ``detrended'' by this process,
  the OWL weights are not putting as much effort into removing the large-scale trends.

This kind of detrending is common in the exoplanet community;
  it has been valuable for finding exoplanets.
It brings many disadvantages;
  one is that any detrending method that happens prior to exoplanet search
  distorts the transit shapes and depths.
Another is that it removes stellar variability that might be of intrinsic interest.
Another is that it correlates data that are close in time;
  it creates non-trivial noise for lightcurve fitting.
The OWL photometry---in its standard form shown in \figurename~\ref{fig:results}---%
  produces independent photometry at every epoch
  and preserves transit shapes and all intrinsic stellar variability.

In the end, a forward-modeling approach ought to win.
It won't obey \emph{The Rules}, but it is the only approach that has a chance
  of saturating the Cram\'er--Rao bound.
The forward-modeling approach is not trivial, however,
  because the usual methods of image modeling,
  which make use of simple models for the flat-field, point-spread function,
  and the ``scene''
  have (in general) model mismatch issues that project residuals
  in the calibration parameters onto the photometry of greatest interest.
That is, a forward-modeling approach that obtains photometry at the same
  precision as the OWL would be expensive and complex.
Still, it is the \emph{Right Thing To Do (tm)} and thus the One True Path for the future.
One way to see this is to note that there is no obvious or simple generalization of the OWL
  to the case of multi-instrument or multi-band imaging
  (where pixel scale or bandpass vary from epoch to epoch).

The OWL, OPW, and TSA methods all depend very strongly on good background subtraction.
The reason is that it seeks to obtain high expectation value and low variance.
If there is a constant background that hasn't been subtracted,
  the weights will choose to add in a lot of constant background
  and possibly down-weight the pixels being hit by the variable object star.
The main requirement is that the background has been subtracted such
  that the mean background level in all pixels at all epochs is negligible.
The background subtraction can be noisy and variable,
  but it must be accurate in the mean.

On a related note, the image patch under consideration ought not be \emph{far}
  bigger than the region over which a normal would perform photometry.
For example, if the number of pixels in the object-star image patch becomes
  much larger than the number of epochs,
  the OWL will be able to find linear combinations of pixels that have
  more-or-less perfectly constant flux.
This over-fitting issue is not a problem in what's shown above,
  but could start to become a problem if large patches get used,
  and would be exacerbated in the presence of non-zero mean background subtraction residuals.
This deserves further study.

The biggest (in our view) limitation of the OWL is that it cannot be used
  to accurately measure the properties of stars
  that are close enough that their point-spread functions overlap,
  or stars in crowded fields.
The issue is that the optimization procedure has no knowledge or understanding
  of the PSF or even of the fact that it is a star being observed.
In our view, the only justifiable solution to this problem is forward modeling,
  in which there are parameterized beliefs about the PSF and the scene,
  and in which the output photometry comes from analysis of a likelihood function.
The only conceivable generalizations of the OWL that might be relevant would either
  \textsl{(a)}~involve taking the optimum subject to the constraint that the weights must be
  orthogonal to some estimate of the overlapping star point-spread functions, or else
  \textsl{(b)}~require modeling and subtraction of the confusing source or sources
  in the field.
These are interesting ideas,
  but probably no easier or more reliable than full forward modeling.
Although we consider this non-isolated problem to be the most severe limitation of what we are doing,
  we note that the tremendous discoveries and measurements of the \project{Kepler} mission
  have almost all been generated with straight-up aperture photometry,
  followed by \foreign{ex post facto} checks for blending.
There certainly is a great deal that can be done in the isolated-star limit.

All of the code used in this project is available
  from \url{http://github.com/davidwhogg/OWL/} (or forks thereof)
  under the MIT open-source software license.
This code (plus some dependencies) can be run
  to re-generate all of the figures and results in this \documentname;
  this version of the paper was generated with git hash
  \texttt{\githash} (\gitdate).

\acknowledgments
It is a pleasure to thank
  Ruth Angus (Oxford),
  Morgan Fouesneau (MPIA),
  Fengji Hou (NYU),
  Christian Knigge (Southampton),
  Molei Tao (NYU), and
  Jason Wright (PSU)
for valuable comments and suggestions.
This work was partially supported by the Moore--Sloan Data Science Environment at NYU,
  by NASA (grant NNX08AJ48G).
  and by the NSF (grant AST-0908357).
This research made use of the \project{NASA Astrophysics Data System}.

\newcommand{\arxiv}[1]{\href{http://arxiv.org/abs/#1}{arXiv:#1}}
\begin{thebibliography}{}\raggedright
\bibitem[Hogg \etal(2013)]{hoggwhitepaper}
Hogg, D.~W., Angus, R., Barclay, T., et al.\ 2013,
Maximizing Kepler science return per telemetered pixel: Detailed models of the focal plane in the two-wheel era,
\arxiv{1309.0653}
\bibitem[Naylor(1998)]{naylor}
Naylor, T.\ 1998,
An optimal extraction algorithm for imaging photometry,
\mnras, 296, 339
\end{thebibliography}

\clearpage
\begin{figure}
\includegraphics[width=\textwidth]{../py/kic_03335426_05_pixels.png}
\caption{
The ``lightcurves'' (flux \foreign{vs} time) for 19 pixels near...HOGG
\label{fig:pixels}}
\end{figure}

\clearpage
\begin{figure}
\includegraphics[width=\textwidth]{../py/kic_03335426_05_images_owl.png}
\caption{
The top row shows example image patches (data $y_n$) from the \project{Kepler} data on KIC XXXX in quarter XXXX.
The middle row shows the empirical mean image $\hat{\mu}$,
  the diagonal elements of the empirical covariance matrix $\hat{C}$,
  and its dominant eigenvector.
The bottom row shows the pixel weights used in the official \project{Kepler} SAP photometry,
  and the OWL weights $\hat{w}$.
The bottom-right panels shows the OWL weights times the empirical mean;
  this shows the mean contributions of the individual pixels to the final OWL photometry.
\label{fig:images}}
\end{figure}

\clearpage
\begin{figure}
\includegraphics[width=\textwidth]{../py/kic_03335426_05_images_opw.png}
\caption{
The same as the bottom row of \figurename~\ref{fig:images}
  but for the optimized positive-weight (OPW) photometry.
In this case, the positivity constraint does not make much difference.
\label{fig:images_opw}}
\end{figure}

\clearpage
\begin{figure}
\includegraphics[width=\textwidth]{../py/kic_03335426_05_photometry.png}
\caption{
The \project{Kepler} SAP flux,
  the Optimized-Weight Linear (OWL) photometry,
  the Optimized Positive-Weight (OPW) photometry,
  and the Tweaked Simple Aperture (TSA) photometry,
  all for quarter 5 of KIC~3335426.
\label{fig:results}}
\end{figure}

\clearpage
\begin{figure}
\includegraphics[width=\textwidth]{../py/kic_03335426_05_diff_photometry.png}
\caption{
The \project{Kepler} SAP flux,
  the Difference-Optimized-Weight Linear (DOWL) photometry,
  the Difference-Optimized Positive-Weight (DOPW) photometry,
  and the Difference-Tweaked Simple Aperture (DTSA) photometry,
  all for quarter 5 of KIC~3335426.
\label{fig:dresults}}
\end{figure}

\end{document}