\begin{figure}[H]
  \centering
  \includegraphics[width=0.7\linewidth]{./chapters/exp1/methodology1_3.png}
  \caption[Third portion of the flowchart from Figure \ref{fig:method1}]
          {Third portion of the flowchart from Figure \ref{fig:method1} being
           described in this section.}
\end{figure}

Since this work has yet to be benchmarked with simple algorithms, two
straightforward algorithms were chosen, \textit{k}-nearest neighbors and
decision trees, and were introduced in Section \ref{sec:algs}. Also covered in
that section is the mathematical framework of the \gls{MLL} calculation method.
The implementation details of these three approaches are covered here.

\subsection{Scikit-learn Algorithms}

The machine learning toolkit chosen for this work is scikit-learn
\cite{scikit}, a package in python.  Virtually all modern \gls{ML} toolkits
will have acceptably fast and reliable algorithms, but the use of python
provides a platform for seamless integration of all the tools in the workflow.
This section walks through the implementation of the two scikit algorithms.

The first step is splitting the training set into the features ($\vec{X}$, with
29 nuclide mass measurements), and a single set of labels ($\vec{y}$, e.g.,
burnup). After this, the set of features $\vec{X}$ is scaled so that there is a
zero mean and unit variance; this is done because the magnitude of the features
vary widely and it can be either helpful or required for many scikit-learn
algorithms. Next, the two algorithms are initialized.  The labels that require
regression analysis (burnup, enrichment, cooling time) are predicted using the
\texttt{KNeighborsRegressor} and \texttt{DecisionTreeRegressor} in
scikit-learn. The reactor type label uses classifiers:
\texttt{KNeighborsClassifier} and \texttt{DecisionTreeClassifier}.

Each algorithm has a list of hyperparameters that impact the model complexity
and prediction performance, as discussed in Section \ref{sec:complexity}. Since
this work is a demonstrative investigation, many of the default choices are
retained. However, some of these were tuned using hyperparameter optimization,
shown in Table \ref{tbl:exp1hypparam}.  Hyperparameter optimization involves
completing the training, predicting, and error evaluation process for a list of
parameters that are of a continuous nature.  This process is carried out with
\textit{k} for \textit{k}-nearest neighbors, choosing the value that provides
the best results. For decision trees, the maximum features was chosen to be 29
because the performance was highly variable otherwise, but this is not
necessarily ideal for the best performance. Instead, only the maximum depth was
tuned via optimization.  There are also other parameters governing algorithm
behavior that have a discrete set of choices.  The selections made for each of
these algorithms is summarized below (note this is just a subset of all
available hyperparameters \cite{scikit}):
\begin{enumerate}
  \item \texttt{KNeighborsClassifier} \& \texttt{KNeighborsRegressor}
    \begin{itemize}
      \item Number of nearest neighbors (\textit{k}) to include changes for
            each prediction category and is in Table \ref{tbl:exp1hypparam}.
      \item Distance metric is "Manhattan" (aka $L_1$ loss, or absolute
            differences): $d_i = \sum_{j=1}^{N_{feats}}
            |x_{j,train} - x_{j,test}|$.
      \item Sample weighting is with respect to distance and not uniform :
            $w_i = 1/d_i$.
    \end{itemize}
  \item \texttt{DecisionTreeClassifier} \& \texttt{DecisionTreeRegressor}
    \begin{itemize}
      \item Maximum number of features included is the length of the feature
            set: 29.
      \item Maximum depth of decision tree changes for each prediction category
            and is in Table \ref{tbl:exp1hypparam}.
      \item For both the classifier and regressor, the default splitting
            criterion (function for measuring the quality of a split) is kept.
            This is the Gini impurity for the former (see Equation
            \ref{eq:gini}), and mean squared error (aka $L_2$ loss, see
            Equation \ref{eq:mse}) for the latter.
      \item For the classifier, the \texttt{class\_weight} is "balanced",
            meaning that the weights are inversely proportional to the frequency
            of each class (\gls{PWR}, \gls{BWR}, \gls{PHWR}) in the training set.
    \end{itemize}
\end{enumerate}

\begin{table}[!hbt]
  \centering
  \begin{tabular}{@{}lcll@{}}
  \toprule
    \textbf{\begin{tabular}[c]{@{}l@{}}Prediction\\ Parameter\end{tabular}}
  & \textbf{\begin{tabular}[c]{@{}l@{}}\textit{k} \\ (N neighbors)\end{tabular}}
  & \textbf{\begin{tabular}[c]{@{}l@{}}Max \\ Depth\end{tabular}}
  & \textbf{\begin{tabular}[c]{@{}l@{}}Max \\ Features\end{tabular}} \\
  \toprule
  Reactor Type & 4                 & 56        & 29           \\
  Burnup       & 1                 & 77        & 29           \\
  Enrichment   & 2                 & 45        & 29           \\
  Cooling Time & 1                 & 73        & 29           \\ \bottomrule
  \end{tabular}
  \caption[Optimized algorithm hyperparameters for the 29 nuclide training set]
          {Optimized algorithm hyperparameters for the 29 nuclide mass training
           set.}
  \label{tbl:exp1hypparam}
\end{table}

After the algorithms are initialized, the training set iteratively undergoes
error injection according to the method in Section \ref{sec:inforeduc1}. Next,
the (information reduced) data set is preprocessed by scaling and normalization
because the nuclide concentrations vary by many orders of magnitude. After
this, the training data features have a zero mean and unit variance. The
initialized algorithm is then used to fit the training set and is then ready
for prediction of "unseen" test samples.

Introduced in Section \ref{sec:testerr}, these algorithms provide predictions
based on the \textit{k}-fold \gls{CV} approach (not to be confused with the
\textit{k} in \textit{k}-nearest neighbors). Tested were 5, 10, and 15 \gls{CV}
folds and they all performed similarly because the training set is large.  The
results shown in Table \ref{tbl:cvdiff} show the percent change in the accuracy
score or \gls{MAE} relative to when $k=5$.  This means that the accuracy score
is under $0.5\%$ higher for reactor type predictions for both
\textit{k}-nearest neighbors and decision trees changing \textit{k} folds from
5 to 10 or 15.  And the \gls{MAE}s are slightly lower (under $7\%$ difference)
when changing \textit{k} folds to 10 or 15.  Given the close performance, the
number of \gls{CV} folds was chosen to be 5 based on computational expediency.
The \texttt{KFold} cross-validator with observation shuffling (i.e., the data
set was shuffled prior to splitting) in the scikit package carried out this
task for the regression prediction cases.  For reactor type classification, the
\texttt{StratifiedKFold} cross-validator was used since there is an imbalance
of classes in the training database.

\begin{table}
  \centering
  \begin{tabular}{@{}lllll@{}}
    \toprule
      & \multicolumn{2}{l}{\textbf{\begin{tabular}[c]{@{}l@{}}\% Difference, \\
                                   \textit{k}=5 to \textit{k}=10\end{tabular}}}
      & \multicolumn{2}{l}{\textbf{\begin{tabular}[c]{@{}l@{}}\% Difference, \\
                                   \textit{k}=5 to \textit{k}=15\end{tabular}}} \\
        \toprule
                           & kNN    & DTree  & kNN    & DTree  \\ \midrule
    Reactor Type           & 0.4    & 0.2    & 0.4    & 0.4    \\
    Burnup                 & -4.0   & -2.1   & -5.2   & -2.6   \\
    Enrichment             & -5.1   & -6.1   & -6.6   & -6.7   \\
    Time Since Irradiation & -1.8   & -3.1   & -2.3   & -3.8   \\ \bottomrule
  \end{tabular}
  \caption[Comparison of different \acrshort{CV} implementation errors]
          {Comparison of prediction errors between different \textit{k}-fold
           \acrshort{CV} implementations.}
  \label{tbl:cvdiff}
\end{table}

Because the performance needs to be compared to the \gls{MLL} predictions, the
scikit method \texttt{cross\_val\_predict} was used, as it returns the
predictions of each \gls{CV} fold as it becomes the test set. Thus, the entire
training set becomes a test case at some point, and the predictions return
equal the number of entries in the training set.

The scripts written to run the scikit-learn algorithms were deployed using
\gls{UW}-Madison's \gls{CHTC} resources and the \gls{UW} campus grid.

\subsection{Maximum Log-Likelihood Calculations}

The \gls{MLL} calculation method is implemented in python using the SciPy
statistics toolkit and NumPy functionality \cite{scipy, numpy}. In this method,
one test sample is removed from the training set at a time, and the
log-likelihood calculation in Equation \ref{eq:loglike} is performed between
the test sample and all the rows of the training set.  The observation with the
largest log-likelihood is returned as the predicted labels. Just like with the
scikit-learn algorithm-based predictions, the entire training set becomes a
test sample at some point, so the predictions returned equal the number of
entries in the training set.

A key difference in how the \gls{MLL} method provides predictions is that it
returns all the labels together, so it is not doing any generalizing on the
continuous variable labels (burnup, enrichment, and cooling time). It is
analogous to \textit{k}-nearest neighbors where $k = 1$, because there is no
model being created, it just matches upon the call for prediction. Because of
this, the \gls{MLL} method performing well is highly dependent on the training
set being sufficiently large.

The scripts written to run the \gls{MLL} calculations were deployed using
\gls{UW}--Madison's \gls{CHTC} resources, the \gls{UW} campus grid, and the
\gls{OSG} \cite{osg07, osg09}.


\chapter{Cross Validation}
\section{Motivation}

Generally people train a neural network with the training set and obtain the test  accuracy after each epoch. After the training is finished, the highest ever or the last test  accuracy is reported in the paper. However, in our opinion, this may be not only abusing and polluting the test set, but also not generalizable.

A test set, as its name tells, should be concealed and be tested only after all the model structures are set up and the training algorithm is fixed. In this way, the test set is never touched by the reseachers and serves as independent and fair comparison target.

However, this is not the case in the field of deep learning. What people are doing is  first choosing a model and training algorithm, then evaluate the accuracy with the test set. If the test accuracy is not satisfactory, they turn to  choose a new combination of model structures and training algorithm in order to achieve a higher test accuracy. By doing so, the test set is evaluated many times or maybe thousands of times.  The test set is thus touched by the reseachers  and doesn't serves as independent and fair comparison any more. Instead, we say the original test set becomes part of the validation set. Without a proper framework or mechanism, such scenarios will be ongoing and  independent and fair comparison target will be missing. Therefore we want to propose a universal framework to evaluate the performance of the neural networks given a dataset.

\section{Universal framework}
The framework takes as input the proposed neural network, training algorithm and dateset.

Suppose we are given a specific dataset such as CIFAR-10 with 50000 training data and 10000 validation data.
In each round, we randomly choose 6000 balanced data from CIFAR-10 as the test set, namely 600 data for each label. The remaining data are used as the training set. Then we evaluate the test accuracy after finishing training the neural network.
After ten rounds, we take the average of the test accuracies as the metric reported.

The algorithm is as below.
%\begin{breakablealgorithm}%[!htb]
\begin{algorithm}%[!htb]
	\caption{Overlapping CV training algorithm}
\hspace*{\algorithmicindent} \textbf{Input} Dataset $D$, neural network $N$
	\begin{algorithmic}
	\For{$i = 1:10$}
		\State 1. Randomly choose 10\% balanced data from $D$ and denote it as $V$, the validation set. The remaining data are  denoted as $T$, the training set.
		\State 2. Train $N$ with $T$ with the stopping criterion $S$
		\State 3. When the training is finished, evaluate $N$ with $V$ and denote the validation accuracy as $A_i$
	\EndFor
	\end{algorithmic}
\hspace*{\algorithmicindent} \textbf{Output} $A(D,N)=\sum_i A_i /10$
\end{algorithm}

The  standard  cross validation algorithm is as below.
\begin{algorithm}%[!htb]
	\caption{Standard CV training algorithm}
\hspace*{\algorithmicindent} \textbf{Input} Dataset $D$, neural network $N$\\
\hspace*{\algorithmicindent} \textbf{Data split}  Randomly divide the dataset $D$ into 10 folds with numbers 1,2,..,10
	\begin{algorithmic}
	\For{$i = 1:10$}
		\State 1. Choose the $i$-th fold and denote it as $V$, the validation set. The remaining data are  denoted as $T$, the training set.
		\State 2. Train $N$ with $T$ with the stopping criterion $S$
		\State 3. When the training is finished, evaluate $N$ with $V$ and denote the validation accuracy as $A_i$
	\EndFor
	\end{algorithmic}
\hspace*{\algorithmicindent} \textbf{Output} $A(D,N)=\sum_i A_i /10$
\end{algorithm}

The  leave-one-fold  cross validation algorithm is as below.
\begin{algorithm}%[!htb]
	\caption{Leave-one-fold CV training algorithm}
\hspace*{\algorithmicindent} \textbf{Input} Dataset $D$, neural network $N$\\
\hspace*{\algorithmicindent} \textbf{Data split}  Randomly divide the dataset $D$ into 10 folds with numbers 1,2,..,10. Fix the $10$-th fold as the test set and denote it as $T_2$.
	\begin{algorithmic}
	\For{$i = 1:9$}
		\State 1. Choose the $i$-th fold and denote it as $V$, the validation set. The remaining data are  denoted as $T$, the training set.
		\State 2. Train $N$ with $T$ with the stopping criterion $S$
		\State 3. When the training is finished, evaluate $N$ with $T_2$ and denote the test accuracy as $A_i$
	\EndFor
	\end{algorithmic}
\hspace*{\algorithmicindent} \textbf{Output} $A(D,N)=\sum_i A_i /9$
\end{algorithm}

Things to discuss:

1. If we randomly choose 10\% data and do it 10 times, this is technically not the cross validation.

Cross validation is when we randomly divide the data into 10 folds of equal sizes. Then in each round, we take one fold  sequentially as the validation set, and use the remaining data as the training set.

2. Does the stopping criterion $S$ depend on $D$ and $N$? We want to use the automative learning rate scheduling algorithm, but it doesn't have a stopping criterion. Moreover, based on the result on DenseNet, SASA by Lian Zhang depends on the initial parameter tuning.

Therefore we will take different dataset splitting, learning rate scheduling, best validation accuracy vs last epoch with various models on popular dataset for comparison.

\section{ResNet-18 on CIFAR-10}
In this section, we will use test network ResNet-18 on CIFAR-10. Since the traditional training include the validation set to choose the model, therefore we split the original 54000 training data as training set with 48000 data and validation set with 6000 data. There are also two ways to choose the model, namely choosing the model after the last epoch or choosing the model with the highest validation accuracy.

\subsection{Fixed epoch and manual decreasing learning rate}
We choose to use fixed epoch training, namely 120 epochs, with changing the learning rate manually, i.e., decrease by 10 for every 30 epochs. The initial learning rate is 0.1, the momentum is 0.9 and the weight-dacay is 0.0001. The batch size is 128.

The time cost is around 17000 seconds.

If we split the sets randomly for 10 times, the results are listed in the table.
\begin{table}[!htbp]
	\centering
	\caption{Accuracy of different sets for model of last epoch and the one with best validation accuracy }
	\label{table:cv1}
	\begin{tabular}{|c|c|c|c|c|c|c|c|c|}
		\hline
		% after \\: \hline or \cline{col1-col2} \cline{col3-col4} ...
%		Iteration &  \multicolumn{2}{{|c|}}{SGD}  &    \multicolumn{2}{{|c|}}{l1-prox}    \\\cline{2-5}
%
fold	&	train accu(best	&	train accu(last	&	val accu(best	&	val accu(last	&	test accu(best 	&	test accu(last	\\
	&	 val model)	&	 epoch model)	&	val model)	&	 epoch model)	&	val model)	&	 epoch model)	\\\hline
1	&	99.92	&	99.91	&	93.13	&	92.48	&	93.40	&	93.20	\\\hline
2	&	99.92	&	99.90	&	93.05	&	92.77	&	92.52	&	92.62	\\\hline
3	&	99.93	&	99.89	&	93.17	&	92.58	&	93.35	&	93.12	\\\hline
4	&	99.91	&	99.90	&	92.97	&	92.40	&	92.33	&	92.42	\\\hline
5	&	99.91	&	99.91	&	93.30	&	93.05	&	92.63	&	92.70	\\\hline
6	&	99.91	&	99.90	&	93.50	&	93.47	&	93.12	&	92.77	\\\hline
7	&	99.92	&	99.88	&	93.32	&	93.00	&	93.47	&	93.12	\\\hline
8	&	99.92	&	99.88	&	93.35	&	92.80	&	93.40	&	93.23	\\\hline
9	&	99.90	&	99.87	&	93.78	&	93.45	&	92.53	&	92.17	\\\hline
10	&	99.92	&	99.90	&	93.53	&	93.20	&	92.05	&	92.02	\\\hline
avg	&	99.92	&	99.89	&	93.31	&	92.92	&	92.88	&	92.74	\\\hline
	\end{tabular}
\end{table}

If we split the sets into 10 folds and do the standard cross-validation, the results are listed in the table.
\begin{table}[!htbp]
	\centering
	\caption{Accuracy of different sets for model of last epoch and the one with best validation accuracy }
	\label{table:cv2}
	\begin{tabular}{|c|c|c|c|c|c|c|c|c|}
		\hline
		% after \\: \hline or \cline{col1-col2} \cline{col3-col4} ...
%		Iteration &  \multicolumn{2}{{|c|}}{SGD}  &    \multicolumn{2}{{|c|}}{l1-prox}    \\\cline{2-5}
%
fold	&	train accu(best	&	train accu(last	&	val accu(best	&	val accu(last	&	test accu(best 	&	test accu(last	\\
	&	 val model)	&	 epoch model)	&	val model)	&	 epoch model)	&	val model)	&	 epoch model)	\\\hline
1	&	99.93	&	99.92	&	94.00	&	93.52	&	93.07	&	93.43	\\\hline
2	&	99.94	&	99.88	&	93.83	&	93.53	&	93.12	&	93.07	\\\hline
3	&	99.91	&	99.90	&	93.23	&	92.92	&	93.42	&	93.32	\\\hline
4	&	99.92	&	99.90	&	93.40	&	93.05	&	93.45	&	93.22	\\\hline
5	&	99.91	&	99.89	&	93.42	&	92.93	&	92.65	&	92.67	\\\hline
6	&	99.94	&	99.88	&	93.33	&	92.82	&	93.22	&	92.87	\\\hline
7	&	99.92	&	99.90	&	93.55	&	93.17	&	93.17	&	93.40	\\\hline
8	&	99.88	&	99.85	&	92.72	&	92.35	&	92.97	&	92.35	\\\hline
9	&	99.90	&	99.86	&	93.22	&	92.65	&	92.40	&	92.27	\\\hline
10	&	99.90	&	99.88	&	93.55	&	92.82	&	92.73	&	92.97	\\\hline
avg	&	99.92	&	99.89	&	93.43	&	92.98	&	93.02	&	92.96	\\\hline
	\end{tabular}
\end{table}

If we split the sets into 10 folds, fix the first fold as the test set, take one fold as the validation set and the remaining as the training set, the results are listed in the table.
\begin{table}[!htbp]
	\centering
	\caption{Accuracy of different sets for model of last epoch and the one with best validation accuracy }
	\label{table:cv1}
	\begin{tabular}{|c|c|c|c|c|c|c|c|c|}
		\hline
		% after \\: \hline or \cline{col1-col2} \cline{col3-col4} ...
%		Iteration &  \multicolumn{2}{{|c|}}{SGD}  &    \multicolumn{2}{{|c|}}{l1-prox}    \\\cline{2-5}
%
fold	&	train accu(best	&	train accu(last	&	val accu(best	&	val accu(last	&	test accu(best 	&	test accu(last	\\
	&	 val model)	&	 epoch model)	&	val model)	&	 epoch model)	&	val model)	&	 epoch model)	\\\hline
1	&	99.93	&	99.93	&	93.58	&	93.13	&	92.55	&	92.65	\\\hline
2	&	99.94	&	99.89	&	93.57	&	92.98	&	92.93	&	92.78	\\\hline
3	&	99.93	&	99.89	&	93.48	&	93.03	&	92.85	&	93.20	\\\hline
4	&	99.95	&	99.92	&	93.28	&	92.95	&	93.03	&	93.18	\\\hline
5	&	99.92	&	99.90	&	93.12	&	92.75	&	92.92	&	92.50	\\\hline
6	&	99.94	&	99.93	&	93.52	&	92.93	&	92.92	&	93.43	\\\hline
7	&	99.93	&	99.92	&	93.40	&	93.10	&	93.00	&	93.15	\\\hline
8	&	99.93	&	99.90	&	92.77	&	92.18	&	92.58	&	93.02	\\\hline
9	&	99.93	&	99.90	&	92.80	&	92.22	&	92.52	&	92.37	\\\hline
avg	&	99.93	&	99.91	&	93.28	&	92.81	&	92.81	&	92.92	\\\hline
	\end{tabular}
\end{table}

\newpage
\section{VGG-16 on CIFAR-10}
In this section, we will use test network VGG-16 on CIFAR-10. Since the traditional training include the validation set to choose the model, therefore we split the original 54000 training data as training set with 48000 data and validation set with 6000 data. There are also two ways to choose the model, namely choosing the model after the last epoch or choosing the model with the highest validation accuracy.

\subsection{Fixed epoch and manual decreasing learning rate}
We choose to use fixed epoch training, namely 160 epochs.
The initial learning rate is set to 0.1, and is divided by 10 at 50\% and 75\% of the total number of training epochs.
The momentum is 0.9 and the weight-dacay is 0.0001. The batch size is 128.

The time cost is around 22000 seconds.

If we split the sets randomly for 10 times, the results are listed in the table.
\begin{table}[!htbp]
	\centering
	\caption{Accuracy of different sets for model of  last epoch and the one with best validation accuracy }
	\label{table:cv3}
	\begin{tabular}{|c|c|c|c|c|c|c|c|c|}
		\hline
		% after \\: \hline or \cline{col1-col2} \cline{col3-col4} ...
%		Iteration &  \multicolumn{2}{{|c|}}{SGD}  &    \multicolumn{2}{{|c|}}{l1-prox}    \\\cline{2-5}
%
fold	&	train accu(best	&	train accu(last	&	val accu(best	&	val accu(last	&	test accu(best 	&	test accu(last	\\
	&	 val model)	&	 epoch model)	&	val model)	&	 epoch model)	&	val model)	&	 epoch model)	\\\hline
1	&	99.97	&	99.97	&	93.12	&	92.77	&	93.28	&	92.65	\\\hline
2	&	99.97	&	99.96	&	92.85	&	92.65	&	92.92	&	93.18	\\\hline
3	&	99.99	&	99.96	&	93.12	&	92.48	&	92.78	&	92.90	\\\hline
4	&	99.97	&	99.95	&	92.98	&	92.72	&	92.07	&	92.40	\\\hline
5	&	99.97	&	99.95	&	93.38	&	93.08	&	92.37	&	92.40	\\\hline
6	&	99.97	&	99.95	&	93.10	&	92.80	&	92.35	&	92.57	\\\hline
7	&	99.98	&	99.97	&	93.05	&	92.65	&	92.90	&	93.20	\\\hline
8	&	99.97	&	99.97	&	93.03	&	92.72	&	93.07	&	92.82	\\\hline
9	&	99.97	&	99.96	&	93.50	&	93.07	&	92.37	&	92.60	\\\hline
10	&	99.98	&	99.96	&	93.43	&	93.18	&	92.20	&	92.57	\\\hline
avg	&	99.97	&	99.96	&	93.16	&	92.81	&	92.63	&	92.73	\\\hline
	\end{tabular}
\end{table}

If we split the sets into 10 folds and do the standard cross-validation, the results are listed in the table.
\begin{table}[!htbp]
	\centering
	\caption{Accuracy of different sets for model of  last epoch and the one with best validation accuracy }
	\label{table:cv4}
	\begin{tabular}{|c|c|c|c|c|c|c|c|c|}
		\hline
		% after \\: \hline or \cline{col1-col2} \cline{col3-col4} ...
%		Iteration &  \multicolumn{2}{{|c|}}{SGD}  &    \multicolumn{2}{{|c|}}{l1-prox}    \\\cline{2-5}
%
fold	&	train accu(best	&	train accu(last	&	val accu(best	&	val accu(last	&	test accu(best 	&	test accu(last	\\
	&	 val model)	&	 epoch model)	&	val model)	&	 epoch model)	&	val model)	&	 epoch model)	\\\hline
1	&	99.98	&	99.96	&	93.45	&	92.92	&	92.68	&	92.92	\\\hline
2	&	99.97	&	99.95	&	93.47	&	93.25	&	92.75	&	92.98	\\\hline
3	&	99.97	&	99.97	&	93.08	&	92.55	&	92.65	&	93.38	\\\hline
4	&	99.97	&	99.96	&	93.27	&	92.97	&	92.67	&	92.80	\\\hline
5	&	99.97	&	99.95	&	92.92	&	92.42	&	92.53	&	92.40	\\\hline
6	&	99.98	&	99.98	&	92.87	&	92.42	&	92.87	&	93.08	\\\hline
7	&	99.97	&	99.96	&	93.28	&	93.13	&	93.07	&	93.22	\\\hline
8	&	99.98	&	99.96	&	93.10	&	92.72	&	92.90	&	93.15	\\\hline
9	&	99.97	&	99.96	&	93.38	&	93.02	&	92.48	&	92.83	\\\hline
10	&	99.98	&	99.97	&	93.22	&	92.92	&	92.57	&	92.52	\\\hline
avg	&	99.97	&	99.96	&	93.20	&	92.83	&	92.72	&	92.93	\\\hline	\end{tabular}
\end{table}

If we split the sets into 10 folds, fix the first fold as the test set, take one fold as the validation set and the remaining as the training set, the results are listed in the table.
\begin{table}[!htbp]
	\centering
	\caption{Accuracy of different sets for model of last epoch and the one with best validation accuracy }
	\label{table:cv1}
	\begin{tabular}{|c|c|c|c|c|c|c|c|c|}
		\hline
		% after \\: \hline or \cline{col1-col2} \cline{col3-col4} ...
%		Iteration &  \multicolumn{2}{{|c|}}{SGD}  &    \multicolumn{2}{{|c|}}{l1-prox}    \\\cline{2-5}
%
fold	&	train accu(best	&	train accu(last	&	val accu(best	&	val accu(last	&	test accu(best 	&	test accu(last	\\
	&	 val model)	&	 epoch model)	&	val model)	&	 epoch model)	&	val model)	&	 epoch model)	\\\hline
1	&	99.98	&	99.98	&	93.40	&	93.40	&	93.13	&	93.10	\\\hline
2	&	99.98	&	99.97	&	93.37	&	93.23	&	93.00	&	93.05	\\\hline
3	&	99.98	&	99.97	&	93.47	&	92.85	&	92.67	&	93.25	\\\hline
4	&	99.98	&	99.96	&	93.02	&	92.45	&	93.20	&	92.73	\\\hline
5	&	99.98	&	99.95	&	93.57	&	93.18	&	92.70	&	93.00	\\\hline
6	&	99.98	&	99.97	&	93.43	&	92.77	&	92.80	&	92.65	\\\hline
7	&	99.97	&	99.96	&	93.18	&	92.78	&	92.95	&	92.55	\\\hline
8	&	99.98	&	99.97	&	92.72	&	92.18	&	93.17	&	92.90	\\\hline
9	&	99.97	&	99.96	&	93.47	&	92.95	&	92.72	&	92.63	\\\hline
avg	&	99.98	&	99.97	&	93.29	&	92.87	&	92.93	&	92.87	\\\hline
	\end{tabular}
\end{table}

\newpage
\section{VGG-19 on CIFAR-10}
In this section, we will use test network VGG-16 on CIFAR-10. Since the traditional training include the validation set to choose the model, therefore we split the original 54000 training data as training set with 48000 data and validation set with 6000 data. There are also two ways to choose the model, namely choosing the model after the last epoch or choosing the model with the highest validation accuracy.

\subsection{Fixed epoch and manual decreasing learning rate}
We choose to use fixed epoch training, namely 160 epochs.
The initial learning rate is set to 0.1, and is divided by 10 at 50\% and 75\% of the total number of training epochs.
The momentum is 0.9 and the weight-dacay is 0.0001. The batch size is 128.

The time cost is around 27000 seconds.

If we split the sets randomly for 10 times, the results are listed in the table.
\begin{table}[!htbp]
	\centering
	\caption{Accuracy of different sets for model of  last epoch and the one with best validation accuracy }
	\label{table:cv3}
	\begin{tabular}{|c|c|c|c|c|c|c|c|c|}
		\hline
		% after \\: \hline or \cline{col1-col2} \cline{col3-col4} ...
%		Iteration &  \multicolumn{2}{{|c|}}{SGD}  &    \multicolumn{2}{{|c|}}{l1-prox}    \\\cline{2-5}
%
fold	&	train accu(best	&	train accu(last	&	val accu(best	&	val accu(last	&	test accu(best 	&	test accu(last	\\
	&	 val model)	&	 epoch model)	&	val model)	&	 epoch model)	&	val model)	&	 epoch model)	\\\hline
1	&	99.98	&	99.97	&	93.15	&	92.65	&	92.82	&	92.85	\\\hline
2	&	99.98	&	99.97	&	93.38	&	92.78	&	92.35	&	92.85	\\\hline
3	&	99.98	&	99.97	&	92.73	&	92.33	&	92.27	&	92.05	\\\hline
4	&	99.97	&	99.96	&	92.73	&	92.40	&	92.22	&	92.82	\\\hline
5	&	99.98	&	99.96	&	93.18	&	92.85	&	92.13	&	92.13	\\\hline
6	&	99.99	&	99.99	&	92.82	&	92.45	&	92.30	&	91.90	\\\hline
7	&	99.97	&	99.97	&	92.75	&	92.23	&	92.68	&	92.52	\\\hline
8	&	99.98	&	99.96	&	92.70	&	92.62	&	93.02	&	92.92	\\\hline
9	&	99.98	&	99.97	&	93.45	&	93.15	&	92.12	&	92.13	\\\hline
10	&	99.98	&	99.97	&	93.18	&	92.83	&	91.93	&	91.82	\\\hline
avg	&	99.98	&	99.97	&	93.01	&	92.63	&	92.38	&	92.40	\\\hline
	\end{tabular}
\end{table}

If we split the sets into 10 folds and do the standard cross-validation, the results are listed in the table.

\begin{table}[!htbp]
	\centering
	\caption{Accuracy of different sets for model of  last epoch and the one with best validation accuracy }
	\label{table:cv3}
	\begin{tabular}{|c|c|c|c|c|c|c|c|c|}
		\hline
		% after \\: \hline or \cline{col1-col2} \cline{col3-col4} ...
%		Iteration &  \multicolumn{2}{{|c|}}{SGD}  &    \multicolumn{2}{{|c|}}{l1-prox}    \\\cline{2-5}
%
fold	&	train accu(best	&	train accu(last	&	val accu(best	&	val accu(last	&	test accu(best 	&	test accu(last	\\
	&	 val model)	&	 epoch model)	&	val model)	&	 epoch model)	&	val model)	&	 epoch model)	\\\hline
1	&	99.98	&	99.98	&	93.38	&	92.75	&	92.70	&	92.78	\\\hline
2	&	99.97	&	99.97	&	93.43	&	93.07	&	93.05	&	92.87	\\\hline
3	&	99.98	&	99.96	&	93.27	&	93.08	&	93.03	&	92.70	\\\hline
4	&	99.99	&	99.99	&	92.75	&	92.27	&	92.82	&	92.43	\\\hline
5	&	99.98	&	99.98	&	92.93	&	92.53	&	91.67	&	92.08	\\\hline
6	&	99.98	&	99.97	&	92.92	&	92.50	&	93.25	&	93.05	\\\hline
7	&	99.98	&	99.97	&	93.00	&	92.65	&	92.55	&	92.30	\\\hline
8	&	99.98	&	99.96	&	92.70	&	92.42	&	92.82	&	92.72	\\\hline
9	&	99.98	&	99.98	&	93.22	&	93.05	&	92.48	&	92.70	\\\hline
10	&	99.97	&	99.95	&	93.13	&	92.42	&	92.80	&	93.00	\\\hline
avg	&	99.98	&	99.97	&	93.07	&	92.67	&	92.72	&	92.66	\\\hline
	\end{tabular}
\end{table}

If we split the sets into 10 folds, fix the first fold as the test set, take one fold as the validation set and the remaining as the training set, the results are listed in the table.
\begin{table}[!htbp]
	\centering
	\caption{Accuracy of different sets for model of last epoch and the one with best validation accuracy }
	\label{table:cv1}
	\begin{tabular}{|c|c|c|c|c|c|c|c|c|}
		\hline
		% after \\: \hline or \cline{col1-col2} \cline{col3-col4} ...
%		Iteration &  \multicolumn{2}{{|c|}}{SGD}  &    \multicolumn{2}{{|c|}}{l1-prox}    \\\cline{2-5}
%
fold	&	train accu(best	&	train accu(last	&	val accu(best	&	val accu(last	&	test accu(best 	&	test accu(last	\\
	&	 val model)	&	 epoch model)	&	val model)	&	 epoch model)	&	val model)	&	 epoch model)	\\\hline
1	&	99.98	&	99.98	&	93.37	&	92.87	&	92.42	&	92.88	\\\hline
2	&	99.99	&	99.98	&	92.97	&	92.47	&	93.27	&	92.95	\\\hline
3	&	99.98	&	99.97	&	93.27	&	92.75	&	92.60	&	93.02	\\\hline
4	&	99.98	&	99.97	&	93.05	&	92.55	&	92.88	&	92.65	\\\hline
5	&	99.99	&	99.97	&	93.63	&	92.85	&	92.70	&	92.87	\\\hline
6	&	99.98	&	99.95	&	92.87	&	92.40	&	91.85	&	92.82	\\\hline
7	&	99.98	&	99.97	&	93.32	&	92.77	&	92.93	&	93.07	\\\hline
8	&	99.98	&	99.98	&	92.70	&	92.70	&	92.93	&	93.23	\\\hline
9	&	99.99	&	99.99	&	93.07	&	92.82	&	92.70	&	92.17	\\\hline
avg	&	99.98	&	99.97	&	93.14	&	92.69	&	92.70	&	92.85	\\\hline
	\end{tabular}
\end{table}

\newpage
\section{Discussion}
We find the test accuracy of the random splitting cross validation is consistently lower than that of the other two methods. This may happen due to the existence of the outliers.

\subsection{Outliers}
There can be some problems about the data labels.

\begin{enumerate}
\item Images with wrong labels: the label is wrong or there is a better label than given.
\item Multi-label images: There should be multiple labels to a image since it contains multiple objects.
\item Ontological issues: the class maillot appears twice, the existence of is-a relationships like bathtub is a tub, misnomers like projectile and missile, and unanticipated issues caused by words with multiple definitions like corn and ear.
\end{enumerate}

\subsection{Distribution}
For simplicity, we assume there are $N_0$ outliers out of 60000 over all images in CIFAR-10 dataset. Suppose we do a $k$-fold cross validation for the overlapping and standard one, we want to compared the total number of outliers in all the test folds, $X$.

As for the standard $k$-fold cross validation, since there are no overlapping among the test folds, each outlier will appear in a test fold once and only once. Therefore the total number of outliers in all the test folds is $N_0$.

As for the overlapping $k$-fold cross validation, things are little bit different. We state the procedure here to better understand the underlying distribution.
\begin{enumerate}
\item We choose $\frac{60000}{k}$ balanced data from the dataset as the test set. Here we assume the distribution of outliers in different classes is uniform. Then this is a drawing of $N=\frac{60000}{k}$ data without replacement. For each drawing, the probability of obtaining an  outlier is a constant $p=\frac{N_0}{60000}$. Therefore the number of outliers in this test fold follows a binomial distribution $B_1(N,p)$. The mean is $\mu=N*p=\frac{N_0}{k}$ and the variance is $$\sigma=N*p*(1-p)=\frac{N_0*(60000-N_0)}{60000*k}.$$
\item Then we repeat the first step for k times. Since the setup is the same, we have the same binomial distributions $B_i(N,p), i=1,2,3,..,k$. By the property of binomials, $$X \sim \sum_{i=1}^{k}B_i(N,p)=B(N*k,p)=B(60000,\frac{N_0}{60000}).$$ The mean is $\mu=N*k*p=N_0$ and the variance is $$\sigma=N*k*p*(1-p)=\frac{N_0*(60000-N_0)}{60000}.$$
\end{enumerate}

From the analysis, we can see the mean is also  $N_0$, but there is a large variance $\frac{N_0*(60000-N_0)}{60000}$. It means in each experiment of the overlapping $k$-fold cross validation, the total number of outliers in all the test folds can be quite different. We may have encountered a bad case where $X$ is larger than $N_0$ such that the test accuracy is consistently lower than that of the other two methods. It contradicts to our original goal to use statistical methods to reduce randomness and improve the repeatability and substantivity. Therefore we would suggest to use the second and third method for better evaluation.

\section{Conclusion}
Based on our numerical result, we can conclude the following things as for the framework and for the traditional training algorithms.

\begin{enumerate}
\item People use the test set multiple times. The best they can do is to run the same setup of the experiment multiple times to ensure the robustness and repeatability. With our cross-validation framework, we can fairly compare the performance of the proposed model and training algorithms.
\item As for the first two kinds of data splitting, the regular cross validation taking validation set and test set sequentially achieved better accuracy for all the experiments. We also prefer this way since we don't need to consider the problem of repeated data in different folds for the cross validation taking random validation set and test set.
\item As for the last two kinds of data splitting, the regular cross validation taking validation set and test set sequentially achieved comparable  accuracy with the leave-one-fold cross validation. It suggests that when the test data is already given, there is not much difference about the data splitting and model selection.
\item In the overlapping $k$-fold cross validation, the total number of outliers in all the test folds follows a binomial distribution and can be quite different. Therefore we don't want to use this method for evaluation.
\item The average test accuracy is comparable as for taking the model with best validation accuracy or the final model after training. This coincides with our observation that these two ways don't make a difference. It also explains why both ways exist when researchers are reporting their results.
\end{enumerate}


\chapter{Fixed Parameters}
\label{appendix:fixed_parameters}

\begin{itemize}
\item \textbf{Tournament size}: 4
\item \textbf{Population size}: 100
\item \textbf{Number of Generations}: 250
\item \textbf{Maximum Program Tree Depth}: 17
\item \textbf{Function set}: $\{ +, -, *, /, \log, \exp, \sin, \cos, -x \}$
\item \textbf{Terminal set}: Only the features in the data set.
\end{itemize}


\section{Kinky string decay simulation.}

\hspace{1.0em}
For kinky string decay simulation we have assumed the two steps process:
\begin{enumerate}
\item split gluon $g\rightarrow q_1 \bar{q}_1$ and create two longitudinal
strings;
\item decay longitudunal strings $q\bar{q}_1 \rightarrow h$ and
 $\bar{q}q_1 \rightarrow h$ into hadrons $h$.
\end{enumerate}
The production of $q_1\bar{q}_1$ is considered similar (the same sampling of
quark flavors and the same $p_t$-distribution for the quarks) as a production
of $q\bar{q}$-pairs during the longitudinal string decay.
We took the $g\rightarrow q_1\bar{q}_1$ splitting function \cite{APKW80}:
\begin{equation}
\label{KSD1}f^{q}_{g}(z)=z^2+(1-z)^2,
\end{equation}
where $z=\frac{E_q+p^z_q}{E_g+p^z_g}$, derived by Altarelli and
Parisi \cite{AP77}.


\subsubsection{\stid{3.06} PETSc-TAO} \label{subsubsect:petsc}
\paragraph{Overview}

Algebraic solvers (generally nonlinear solvers that use sparse linear solvers via Newton's method) and ODE/DAE
integrators form the core computation of many numerical simulations. No scalable ``black box'' sparse solvers
or integrators work for all applications, nor single implementations that work well for all scales of
problem size. Hence, algebraic solver packages provide a wide variety of algorithms and implementations
that can be customized for the application and range of problem sizes at hand. PETSc~\cite{petsc:homepage,petsc-man}
is a widely used software library for the scalable solution of linear, nonlinear, and ODE/DAE systems and
computation of adjoints (sometimes called sensitivities) of ODE systems. We focus on three topics: (1) partially
matrix-free scalable solvers efficiently use many-core and GPU-based systems; (2) reduced synchronization
algorithms that can scale to larger concurrency than solvers with synchronization points; and (3) performance
and data structure optimizations for all the core data structures to better utilize many-core and GPU-based
systems as well as provide scalability to the Exascale.

The availability of systems with over 100 times the processing power of today's machines compels the utilization
of these systems not just for a single ``forward solve'' simulation (as discussed above) but rather within a
tight loop of optimization, sensitivity analysis (SA), and uncertain quantification (UQ). This requires the
implementation of a new, scalable library for managing a dynamic hierarchical collection of running scalable
simulations, where the simulations directly feed results into the optimization, SA, and UQ solvers.  This library,
which we call libEnsemble, directs the multiple concurrent ``function evaluations'' through the tight coupling
and feedback described above. This work consist of two parts: (1) the development of libEnsemble and (2) the
development of algorithms and software to utilize libEnsemble.

\paragraph{Key Challenges}

A key challenge for for scaling the PETSc/TAO numerical libraries to Exascale systems is that
traditional ``sparse-matrix-based'' techniques for linear, nonlinear, and ODE solvers, as well
as optimization algorithms, are memory-bandwidth limited.  Another difficulty is that any
synchronizations required across all compute units--for example, an inner product or a
norm--can dramatically affect the scaling of the solvers.

Running an ensemble of simulation requires a coordination layer that handles load balancing and
allows the collection of running simulations to grow and shrink based on feedback. Thus, this
library must be able to dynamically start simulations with different parameters, resume
simulations to obtain more accurate results, prune running simulations that the solvers
determine can no longer provide useful information, monitor the progress of the simulations,
and stop failed or hung simulations, and collect data from the individual simulations both
while they are running and at the end.

\paragraph{Solution Strategy}

To address the scalability of the numerical libraries, we are developing new solvers and data
structures including pipeline Krylov methods that delay the use of the results of inner products
and norms, allowing overlapping of the reductions and other computation; partially matrix-free
solvers using high-order methods that have high floating-point-to-memory-access ratios and
good potential to use many-core and GPU-based systems; and in-node optimizations of sparse
matrix-matrix products needed by algebraic multigrid to better utilize many-core systems
using a thread neutral ``bypass MPI'' approach, which implements default interprocessor
communication using MPI but bypasses the use of MPI in performance-critical regions
for higher performance and thereby maintains MPI portability.

Our strategy for coordinating ensemble computations has been to develop libEnsemble
to satisfy our needs.  This library should not be confused with workflow-based
scripting systems; rather it is a library that, through the tight coupling and
feedback described above, directs the multiple concurrent ``function evaluations''
needed by optimization, SA, and UQ solvers.

\paragraph{Recent Progress}

In the past year, we have released PETSc/TAO 3.12 (available at \url{http://www.mcs.anl.gov/petsc})
that features enhanced GPU support.  Perhaps the most important is the support for CUDA-aware
MPI, which allows direct communication of data between Summit GPUs, bypassing the previously
needed step of first copying the data to the CPU memory. This enhancement reduces the latency
of the communication and improves bandwidth. For example, on Summit for sparse matrix-vector
products, this led to a speedup of 33\% on one node and a speedup of 13\% on on four nodes
for the same size problem. Another important addition is the ability to efficiently read
in large meshes on thousands of nodes along with support for collect use of HDF5 calls.
With these additions, the algebraic multigrid solver GAMG is up to 12x faster at scale
on all of Summit when using the GPUs compared to using just the CPUs.

\begin{figure}
\centering
\includegraphics[width=0.5\textwidth]{projects/2.3.3-MathLibs/2.3.3.06-PETSc-TAO/petsc_profile}
\caption{Profile information on the effect of vector size on vector operations compared with
memory throughput (one MPI rank per GPU) for PETSc/TAO 3.12. Note the log scale.}
\label{fig:petsc-tao-fig}
\end{figure}

We have also release libEnsemble 0.5.2 (available at \url{https://github.com/Libensemble/libensemble}).
Notably, this release includes several changes in progressing toward xSDK compliance had have added
support for testing on MacOS to support more applications.  We also have improved I/O, logging,
profiling, as well as resource detection on Summit.

\paragraph{Next Steps}

Our next efforts are:
\begin{enumerate}
  \item \textbf{Enhanced application integration and performance optimization and benchmarking}:
  Refresh the AMReX/PETSc interface and ensure that the PETSc linear solvers can be used with AMReX.
  Consolidate the two MPI communication modules (VecScatter and PetscSF) to share the same code base
  and implement two important communication optimizations in the unified code base: internode node-message
  aggregation optimization, which is critical for scalability of GAMG, and intranode shared memory
  optimization.
  Ensure that libEnsemble satisfies eight of the xSDK requirements.
  Continue to produce benchmark results on Summit.
  Engage with ECP applications and co-design centers to identify new capabilities that can result in
  FY2021-2023 application integration activities.
  \item \textbf{Harmonization, performance optimization, and software release}:
  Harmonize the quasi-Newton and line search modules between SNES and TAO to improve maintainability.
  Implement GPU optimizations of the quasi-Newton methods.
  Ensure that libEnsemble satisfies all but one of the xSDK requirements.
  Continue to produce benchmark results on Summit.
  Continue to engage with ECP applications and co-design centers.
  Release a new version of PETSc/TAO.
  \item \textbf{Enhanced application support and software release}:
  Implement and optimize the ability to use libCEED under PETSc.
  Continue to produce benchmark results on Summit.
  Continue to engage with ECP applications and co-design centers.
  Release a new version of libEnsemble that is compatible with the xSDK community policy requirements and included in the xSDK.
  \item \textbf{Benchmarking, performance optimization and software release}:
  Demonstrate new libEnsemble functionality by having libEnsemble serve as the driver/outer process that
  invokes another xSDK tool.
  Complete benchmarking one or more of the PETSc based applications on Summit at scale including GAMG support
  on the GPUs and identify performance bottlenecks.
  Incorporate new desired capabilities resulting from our engagement with ECP applications and co-design centers into our FY2021 plans.
  Release a new version of PETSc/TAO.
\end{enumerate}


\documentclass[main.tex]{subfiles}
\begin{document}

% \section*{Thu Dec 19 2019}

% The lessons on gravitational waves will be on the 8th and 9th of January, from 14:30 to 16:30, in rooms LUF2 and P2B respectively.

\subsection{The Chandrasekhar limit in more detail}

We want to derive the Chandrasekhar limit in a more precise manner.

% The number density of electrions is given by
% %
% \begin{align}
% n_e = Y_{e} \frac{\rho_c}{m_H}
% \,,
% \end{align}
% %
% in the nonrelativistic case the pressure is given by
% %
% \begin{align}
% P_c = k _{\text{NR}} n_e^{5/3} = k _{\text{NR}} \qty(\frac{Y_e \rho_{c}}{m_H})^{5/3}
% \,,
% \end{align}
%
% but we can also derive it by
% %
% \begin{align}
% P_c = k _{\text{NR}} n_e^{5/3} = k _{\text{NR}} \qty(\frac{Y_e \rho_{c}}{m_H})^{5/3}
% \,,
% \end{align}
% %
% and equating these we find:
%
% \begin{align}
%  k _{\text{NR}} \qty(\frac{Y_e \rho_{c}}{m_H})^{5/3}
%  = k _{\text{NR}} n_e^{5/3} = k _{\text{NR}} \qty(\frac{Y_e \rho_{c}}{m_H})^{5/3}
% \,,
% \end{align}
% %
% which implies
% %
% \begin{align}
% \rho_{c} = \frac{3.1}{Y_e^{5}} \qty(\frac{M}{M_{*}})^2 \frac{m_H}{(h / m_e c^2)^3}
% \,,
% \end{align}
%
% where \(\alpha_{G} = G m_H^2 / (\hbar c) \approx \SI{5.9e-30}{}\), while
% %
% \begin{align}
% m_{*} = \alpha_{G}^{-3 /2 } m_H = 1.85 M_{\odot}
% \,,
% \end{align}
% %
% while for the ultrarelativistic case we get
% %
% \begin{align}
% P_C = k _{\text{UR}} n_e^{4/3} = k _{\text{UR}} \qty(\frac{Y_e \rho_{c} }{m_H})^{4/3}
% \,,
% \end{align}
%
% so in this particular case the density \(\rho_{c}\) simplifies from the equations: we get a critical mass
% %
% \begin{align}
% M _{\text{CHANDRA}} = \qty(\frac{36}{\pi })^{1/2}
% \qty(\frac{Y_e}{m_H})^2
% \qty(\frac{k _{\text{UR}}}{G})^{3/2} \approx
% 2.3 Y_e^2 m_{*} \approx 4.3 Y_e^2 M_{\odot}
% \,,
% \end{align}
%
% and we assume that we are in the fully degenerate case: in the integration over momenta of the phase space distribution we insert a cutoff at the Fermi energy.

% We find:
Instead of approximating the gas as either ultrarelativistic or nonrelativistic, we can use the correct expression for the particle energy in the integral for the momentum:
%
\begin{align}
P = \frac{4 \pi }{3 h^3} g_{*} \int_{0}^{p_F} \dd{p} p^2 \frac{p^2 c^2}{\epsilon_{p}}
\,,
\end{align}
%
with \(\epsilon_{p} = \qty(p^2c^2 + m^2c^{4})^{1/2}\).

We change variables to the dimensionless \(x = p / (m_e c)\) and substitute \(g_* = 2\), since electrons have spin \(1/2\):
%
\begin{align}
P = \frac{8 \pi }{3 h^3} m_e^4 c^{5} \int_{0}^{x_F} \frac{x^{4}}{(1+x^2)^{1/2}} \dd{x}
\,.
\end{align}

The variable \(x_F\) is given by:
\begin{align}
x_F = \frac{p_F}{m_ec} =
\qty(\frac{3 n_e}{8 \pi })^{1/3} \frac{h}{m_e c} =
\qty(\frac{3 Y_e \rho_{c}}{8 \pi m_H})^{1/3} \frac{h}{m_e c}
\,,
\end{align}
%
and, since the electrons are fully degenerate, \(x_F \gg 1 \) corresponds to the ultrarelativistic case while \(x_F \ll 1\) corresponds to the nonrelativistic case.

Now, as \(x_F \to \infty \) the integral is asymptotically
%
\begin{align}
\int_0^{x_F} \frac{x^{4}}{\sqrt{1 + x^2}} \dd{x} \sim \frac{x_F^{4}}{4}
\,,
\end{align}
%
so we define
%
\begin{align}
I(x_F) &= \frac{4}{x_F^{4}}\int_0^{x_F} \frac{x^{4}}{(1 + x^2)^{1/2}} \dd{x} \\
&= \frac{3}{2 x^{4}} \qty( x (1+x^2)^{1/2} \qty(\frac{2x^2}{3} -1) + \log \qty(x + (1+x^2)^{1/2}))
\,,
\end{align}
%
which approaches 1 as \(x_F \to \infty \).
Then, we can write the pressure as
%
\begin{align}
P &= \frac{8 \pi }{3 h^3} m_e^{4} c^{5} \frac{x_F^{4}}{4} I(x_F)  \\
&= k_{UR} n_e^{4/3} I(x_F)
\,.
\end{align}

We can see that this manipulation works by explicitly doing the calculation, but we can also just observe that the limiting case \(x_F \to \infty \) must reduce to the ultrarelativistic approximation, so the prefactor must be the same.

% By comparison with the ultrare
% so we get \(P = k _{\text{UR}} n_e^{4/3}I(x_F)\), where we incorporated the integral in the term \(I(x_F)\): this is given by
%

% so if \(x_F \gg 1\) we have \(I(x_F) \sim 1\), the ultrarelativistic case,
In the nonrelativistic case, \(x_F \ll 1\), we have \(I(x_F) \sim 4 x_F / 5\), and since \(x_F \sim n_e^{1/3}\) this yields \(P \sim n_e^{5/3}\) as expected.
This expression \emph{interpolates} between the two limits.

Then, we can apply the same reasoning as before: we compare the pressure of the Fermi gas with the prediction of the Clayton model to find
%
\begin{align}
k _{\text{UR}} \qty(\frac{Y_e \rho_{c}}{m_H})^{4/3} I(x_F)
\approx \qty(\frac{\pi }{36})^{1/3} G M^{4/3} \rho_{c}^{4/3}
\,,
\end{align}
%
so we can extract the mass:
%
\begin{align}
M = I(x_F)^{3/2} M _{\text{Ch}}
\,,
\end{align}
%
where \(M _{\text{Ch}} \approx \SI{1.4}{M_{\odot}}\) is the Chandrasekhar mass we defined earlier.
% and the important thing is that \(x_F \propto n_e^{1/3} \propto \rho_{c}^{1/3}\).

% [Graph: on the \(x\) axis \(M/ M _{\text{CHANDRA}}\), on the \(y \) axis \(\rho_{c}\).]
\begin{figure}[ht]
\centering
\includegraphics[width=\textwidth]{figures/chandrasekhar_limit.pdf}
\caption{A plot of \(M / M _{\text{Ch}}\) against \(x_F \propto n_e^{1/3} \propto \rho _c^{1/3}\).}
\label{fig:chandrasekhar}
\end{figure}

% \todo[inline]{Quick and dirty Mathematica plot, to update (and flip the axes maybe).}
% If we increase the mass, the star is not able to support itself by the pressure due to being a gas of degenerate electrons.

We can see that as we increase the mass approaching \(M _{\text{Ch}}\) the central density diverges: this is not physically possible, of course, so as it gets higher some usually-prohibited process takes over; typically for white dwarfs this is electron capture, by which electrons and protons combine into neutrons, forming a neutron star.

We have not used any particular characteristics of electrons beyond their being fermions, so this line of reasoning may be used to also bound the mass of a neutron star, since neutrons are fermions as well.
The issue, as we will see, is that general-relativistic corrections become important in that case, since neutron stars have a much higher density.

% This gives us
% %
% \begin{align}
% M _{\text{CHANDRA }} = 3.1 Y_e^2 m_{*}
% = 5.8 Y_e^2 M_{\odot} = 1.4 M_{\odot}
% \,.
% \end{align}

\subsection{White dwarf characteristics}

% It can be shown that the mean density is around
Now that we have a model for the equation of state at the core of a fully degenerate object like a white dwarf, we can try to extract some of its characteristics: it is reasonable (from more complete studies of the object) to estimate the mean density as 1/6 of the central one,
%
\begin{align}
\expval{\rho } = \frac{1}{6} \rho_{c}
= \frac{\num{.51}}{Y_e^2} \qty(\frac{M}{m_{*}})^2
\frac{m_H}{( h / m_e c)^3}
\,,
\end{align}
%
so we can estimate the radius as
%
\begin{align}
R = \qty(\frac{3 M }{4 \pi \expval{\rho }})^{1/3}
\approx \num{.77} Y_e^{5/3} \qty(\frac{M}{m_{*}})^{1/3} \underbrace{\alpha_{G}^{-1/2} \frac{h}{m_e c} }_{\ell_{WD}}
\,,
\end{align}
%
% and the object at the end is
which is of the same order of magnitude as the characteristic length
%
\begin{align}
\ell_{WD} = \alpha_{G}^{-1/2} \frac{h}{m_e c} \approx \SI{3e7}{m} \approx \num{0.04} R_{\odot}
\,,
\end{align}
%
so, taking \(Y_e = 0.5\) we can express the radius as
%
\begin{align}
R = \frac{R_{\odot}}{74} \qty(\frac{M_{\odot}}{M})^{1/3}
\,.
\end{align}

% The luminosity is given by
Using the radius we can estimate the luminosity of the thermal radiation emitted by these bodies:
%
\begin{align}
L = 4 \pi R^2 \sigma T_E^{4}
= \frac{1}{74^2} \qty(\frac{M_{\odot}}{M})^{4/3}
\qty(\frac{T_E }{\SI{6000}{K}}) L_{\odot}
\,,
\end{align}
%
so if we take a typical effective temperature of around \SI{e4}{K} (recall that white dwarfs are in the blue part of the HR diagram), \(M = \num{.4} M_{\odot}\) we get \(L \approx \num{3e-3} L_{\odot}\): they are very dim.

\subsection{Neutron stars}

The first thing to consider when discussing neutron stars is the fact that neutrons, as we discussed, are usually unstable, with lifetimes on the order of \SI{10}{min}. How can a neutron star be stable then?
% We deal with degenerate stars: the Pauli exclusion principle plays a critical role, and the process
The process through which neutrons decay is:
%
\begin{align}
\ce{n} \rightarrow \ce{p} + \ce{e-} + \overline{\nu}_{e}
\,,
\end{align}
%
and the crucial fact is that neutron stars are composed of a degenerate neutron gas \emph{as well as} a degenerate \emph{ultrarelativistic} electron gas: the Fermi temperature is much higher than \(m_e c^2/ k_B\) \cite[eq.\ 1]{yakovlevNeutrinoEmissionNeutron}.
% When a neutron decays
While the electron gas is ultrarelativistic the neutron gas is not; the energy released by neutron decay is of the order of \SI{800}{keV}, so the momentum an emitted electron would have would be well within the Fermi sphere, which is already full!

Thus, neutron decay is inhibited; on the other hand, electron capture, which looks like
%
\begin{align}
\ce{e-} + \ce{p} \rightarrow \ce{n} + \nu_{e}
\,,
\end{align}
%
is favoured, and it can increase the number of neutrons.

We can look at the Saha formula to get numerical estimates for the equilibrium between these processes.
The chemical potential of neutrinos can be neglected, therefore we find
%
\begin{align}
\mu_{\ce{n}} = \mu_{\ce{p}} + \mu_{\ce{e}}
\,,
\end{align}
%
and, since as we saw earlier the chemical potential of a Fermi gas is its Fermi energy, the same equation holds for their Fermi energies:
% This means that a certain point we will have only neutrons.
% We need to exploit the Fermi exclusion principle. Why does the equation
%
\begin{align}
\epsilon_{F, \ce{n}} = \epsilon_{F, \ce{p}} + \epsilon_{F, \ce{e}}
\,.
\end{align}
%
% favour neutrons?

How does this translate into the number densities of the three constituents? We have the constraint that the number density of protons must equal the number density of electrons in order to ensure local neutrality, while there is no constraint on the ratio between neutrons and protons.
% , but the Fermi energy depends on the number density of these.
% typical numbers then become

We will not get into the calculation, but due to the slight mass imbalance \(m_n > m_p\) we have a large difference in the number densities: typically,
%
\begin{align}
n_{\ce{p}} = n_{\ce{e}} = \frac{n_{\ce{n}}}{200}
\,.
\end{align}

% We will have
Since the overwhelming majority of the particles in the neutron star are neutrons, we can make our calculations with the approximation that the number of neutrons per baryon is \(Y_n \approx 1\):
%
\begin{align}
n_{\ce{n}} = Y_n \frac{\rho_{c}}{m_{\ce{n}}} \approx \frac{\rho_{c}}{m_{\ce{n}}}
\,.
\end{align}

Typical values for these densities are \(\rho_c \approx \SI{2e17}{kg/m^3}\) and \(n_{\ce{n}} \approx \SI{e44}{m^{-3}}\).

% For a neutron star we have
With a similar reasoning to the one we applied to white dwarfs we can calculate
%
\begin{align}
\rho_{c}^{\text{NS}} \approx 3.1 \qty(\frac{M}{M_{*}})^2 \frac{m_n}{\qty(h / m_n c)^3}
\,,
\end{align}
%
% while for a white dwarf we have
which, due to the fact that \(m_{n} \gg m_e\), is much larger than the corresponding result for white dwarfs
%
\begin{align}
\rho_{c}^{WD} \approx \frac{3.1}{Y_e^{5}} \qty(\frac{M}{M_{*}})^2 \frac{m_H}{(h / m_e c^2)^{3}}
\,.
\end{align}
% \todo[inline]{why is the  \(c\), right?}

In both cases, we used the characteristic mass \(M_{*} = \alpha_{G}^{-3 /5 } m_n \approx 1.85 M_{\odot}\).

As we did before, knowing the central density we can estimate the average one, which then allows us to calculate the radius:
% and for the radius we get
%
\begin{align}
R = \num{.77} \qty(\frac{M_{*}}{M})^{1/3} \alpha_{G}^{-1/2}  \frac{h}{m_n c}
\,,
\end{align}
%
where the characteristic length is given by
%
\begin{align}
L_n = \alpha_{G}^{-1/2} \frac{h}{m_n c} \approx \SI{17}{km} \approx \frac{1}{1200} L_e
\,,
\end{align}
%
1200 times smaller than the corresponding length scale for white dwarfs (denoted with an \(e\) for ``electron degeneracy'').
% \(L_n\) is the characteristic length scale for a NS, \(L_e\) is the characteristic length for a white dwarf.

Finally, we can compute a maximum mass: \(M _{\text{max}}^{\text{NS}} = 3.1 M_{*} = 5.8 M_{\odot}\).
% This is the mass of a star \emph{remnant}, which encompasses mass from the core only: the initial star will be much larger.

% Can this NS become a BH? we need to compute
We have neglected general-relativistic effects, but would they be relevant? The quantity we need to compute is the ratio of the Schwarzschild radius to the actual radius of the NS:
%
\begin{align}
\frac{R_{\text{Schw}}}{R} =
\frac{2GM}{Rc^2} \approx \num{.4} \qty(\frac{M}{M_{*}})^{4/3}
\,,
\end{align}
%
which is large, of order 1! We have not computed a minimum mass for a neutron star, but typically their mass is of the order of the Chandrasekhar mass, \(M^{NS} \approx 1.4 M_{\odot}\) because of how they form in supernovae (so, \(M / M_{*} \sim 1\)).
% qualitatively we can say that if it is too small then the degenerate electron gas may not be relativistic enough to stop neutron collapse.
The neutron star might not be small enough to actually collapse into a black hole, but surely general relativity must be considered when describing its dynamics.
% so if the mass is large enough we can reach the critical value of \(GM / Rc^2 = 2\).

Neutron stars were first detected as very regular radio pulses: \emph{pulsars}. These are due to the very strong (\(\sim \SI{e8}{T}\)) magnetic fields accelerating particles in beams aligned with the magnetic poles of the NS; these are not aligned with the rotation axis of the NS, so they constantly change the direction of emission, and the Earth can happen to be in this cone.

In order to estimate how fast these pulses can be (in a classical and rough way), let us assume that the NS is rotating barely below a speed which would disintegrate it, so that its binding energy equals its rotational energy: neglecting the order-1 numerical factors we have
% We have
%
\begin{align}
\frac{GM^2}{R} \sim M R^2 \omega^2 _{\text{max}}
\,,
\end{align}
%
% the maximum angular velocity which can be supported gravitationally: it comes out to be
we can then compute the minimum period using the expression we have for the radius in terms of \(M\) and \(M_{*}\):
%
\begin{align}
\tau _{\text{min}} = \frac{2 \pi }{\omega _{\text{max}}}
\approx 2 \pi \qty(\frac{R^3}{GM})^{1/2}
\approx
11 \qty(\frac{M_{*}}{M}) \alpha_{G}^{-1/2} \frac{h}{m_n c^2}
\approx \num{.6} \frac{M_{*}}{M} \SI{}{ms}
\,.
\end{align}

The signals produced by pulsars are of this order of magnitude --- we have observed ``millisecond pulsars'', so NSs do indeed rotate close to these extremely high rates.

Neutron stars can also produce gravitational waves in their rotation, as long as they have a slight asymmetry (a ``mountain'', although their typical sizes are of the order of centimeters); these would have frequencies in the Hz range. We cannot detect these with ground-based detectors, but we might be able to do so with space-based ones.
% This gives us a bound for gravitational waves of astrophysical origin.
% \todo[inline]{no comment on the GR corrections to this formula: however I'd expect them to be significant}

\subsection{Relativistic corrections to the equation of state}

% We move on to \emph{the GR issue}.
Black holes are objects which are so dense that their radius is smaller than the Schwarzschild radius:
%
\begin{align}
R < \frac{2GM}{c^2} = R _{\text{Sch}}
\,.
\end{align}

General Relativity predicts that when this is the case an \textbf{event horizon} form, a surface which forms a causal boundary: as is almost cliché, \emph{not even light can escape}.

Let us see how the classical description of a star fails for objects with relativistic masses.
The equation of hydrostatic balance, derived under classical assumptions, reads:
%
\begin{align}
\dv{P}{r} = - \frac{Gm \rho }{r^2}
\,.
\end{align}
%
% while the GR equations for this are the TOV equation:

If we seek a relativistic analogue under similar assumption (spherical symmetry and equilibrium) we find the \textbf{Tolman-Oppenheimer-Volkov} equation:
%
\begin{align}
\dv{P}{r} = - \frac{Gm  \rho }{r^2} \qty(1 + \frac{P}{\rho c^2}) \qty(1 + \frac{4 \pi r^3 P}{m c^2}) \qty(1 - \frac{2Gm}{rc^2})^{-1}
\,.
\end{align}
%

This equation is exact under the assumptions we mentioned.
In the classical limit (\(2Gm/ c^2 \ll r\) and \(P \ll \rho c^2\)) this reduces to the classical hydrostatic balance equation.
The first correction is reminiscent of the second Friedmann equation: \emph{the pressure itself contributes to the inertia of the system}.

Let us see how their predictions differ assuming constant density, \(\rho \equiv \rho_0 \). In the Newtonian case the mass below a radius \(r\) is given by
% If we have constant density, in the Newtonian case we get
%
\begin{align}
m(r) = \frac{4 \pi }{3} \rho_0 r^3
\,,
\end{align}
%
% so than we can integrate and get
using which we can integrate the hydrostatic balance equation (from the surface, where the pressure vanishes) to find:
%
\begin{align}
P(r)
= \int_{R}^{r_0 } \underbrace{\qty(- \frac{Gm \rho }{r^2})}_{\dv*{P}{r}} \dd{r}
= \frac{2 \pi G}{3}  \rho_0^2 \qty(R^2 - r_0^2)
\,.
\end{align}
%
% where we inserted the boundary condition \(P(R) = 0\).

Then, the central pressure is given by
%
\begin{align}
P_c^{\text{classical}} = \frac{2 \pi }{3} G \rho_0^2 R^2 = \qty(\frac{\pi }{6})^{1/3} G M^{2/3} \rho_c^{4/3}
\,.
\end{align}

Keeping the constant-density assumption, this can be done analytically in the relativistic case as well! We get
%
\begin{align}
P(r) &= \rho_0 c^2 \qty(\frac{(1 - 2GM r^2 / R^3 c^2)^{1/2} - (1 - 2GM / Rc^2)^{1/2}}{3 \qty(1 - 2GM / Rc^2)^{1/2} - \qty(1 - 2GM r^2 / R^3c^2)^{1/2}})  \\
P_c^{\text{relativistic}} &= \rho_0 c^2 \frac{1 - \sqrt{1 - \frac{2GM}{Rc^2}}}{3 \sqrt{1 - \frac{2GM}{Rc^2}} - 1}
\,.
\end{align}

%
% \todo[inline]{Check exponents}
% so we get
% %
% \begin{align}
% P_c = \frac{2 \pi }{3} G \rho_0^2R^2 = \qty(\frac{\pi }{6})^{1/3} GM^{2/3} \rho_{c}^{4/3}
% \,,
% \end{align}
% %
% so we can look at what happens when we consider \(r=0\): we get
% %
% \begin{align}
% P_c = \rho_0 c^2 \qty(\frac{1 - \sqrt{1 - 2GM / Rc^2}}{3 \sqrt{1 - 2GM / Rc^2} - 1})
% \,,
% \end{align}
% %

% so we can see that the central pressure is finite as long as
In the classical model we had a finite central pressure for each value of the mass and density; now, instead, in order for the pressure to not diverge we must require
%
\begin{align}
% \frac{2GM}{Rc^2} < \frac{8}{9}
R > \frac{9}{8} \frac{2GM}{c^2}
\,.
\end{align}
%
% which is not \(1/2\) since we have made some approximations.

This is known as the \textbf{Buchdahl bound}; the radius of a non-black-hole (with constant density) cannot be arbitrarily close to the Schwarzschild radius, it must be at least \SI{12.5}{\percent} larger in order for the pressure to not diverge at the center.\footnote{
More stringent bounds can also be derived --- in \textcite[fig.\ 2]{lattimerNeutronStarObservations2007} a plot is shown of possible equations of state of neutron stars in a mass versus radius plane.
The bound we derived is denoted as \(P < \infty \), and we can also see a ``causality'' bound, which is related to the speed of sound in an ultrarelativistic medium.
Realistic equations of state can reach approximately \(R \gtrsim 3GM /c^2\).}

% \todo[inline]{A mass bound for neutron stars follows, both in Pacciani's notes and in the , but I don't follow the logic.}
This also yields a mass limit for neutron stars. Let us estimate the constant density \(\rho_0\) by assuming that each neutron takes up a sphere of radius its Compton wavelength: \(r_n \approx h / m_n c\), so
%
\begin{align}
\rho_0 \approx \frac{m_n}{ \frac{4 \pi }{3} r_n^3}
\approx \frac{3 m_n^4 c^3}{4 \pi h^3}
\,.
\end{align}

A more accurate estimate would be given by \(r_n \approx \num{.7} h / m_n c\).

so that, using the fact that \(\rho_0 = M / ( 4 \pi R^3 / 3)\), we can start manipulating the Buchdahl bound:
%
\begin{align}
M &< \frac{4 c^2}{9GR} = \frac{4 c^2}{9 G} \qty(\frac{4 \pi \rho_0 }{3M})^{1/3}  \\
M^{2/3} &< \frac{4 c^2}{9G} \qty(\frac{4 \pi }{3} \frac{3 m_n^4 c^3}{4 \pi h^3})^{1/3} \\
M  &< \qty(\frac{8 \pi}{9})^{3/2} M_{*}
\,.
\end{align}

This yields a bound on the order of \(M \lesssim 5 M_{\odot}\).
% Then we have a bound
% %
% \begin{align}

% \,,
% \end{align}
% %
% with \(f \sim 1\).
% This means than the objects becomes contained inside its Schwarzschild radius, \(R = 2GM/c^2\).

% Tomorrow we will speak of galaxy formation.

\end{document}