% design and analysis
%
% DESIGN AND ANALYSIS: Objective of the study, data source, statistical model/tools/methodology,
% validity of the assumptions if any, results of the study (graphs, tables will go here),
% results discussion, (interpretations/consclusions/inferences)
%
%----------------------------------------------------------------------------------------
%	PACKAGES AND OTHER DOCUMENT CONFIGURATIONS
%----------------------------------------------------------------------------------------
% none

\section{Design and Analysis}
To best model the dichotomous response variable, Y\textunderscore HighGradeCancer, in the Prostate Cancer case study, I will employ a multiple logistic regression model, where 1 indicates high grade cancer and 0 indicates not high grade cancer. \par
In statistics, if \(\pi = f(x)\) is a probability then \(\frac{\pi}{1-\pi}\) is the corresponding \textit{odds}, and the \textbf{logit} of the probability is the logarithm of the odds:
\begin{equation}
	logit(\pi) = log(\frac{\pi}{1-\pi})
\end{equation}

Now, simple logistic regression means assuming that \(\pi(x)\) is related to \(\beta_0 + \beta_1x\) (the \textit{logit response function}) by the logit function. By equating \(logit(\pi)\) to the logit response function, we understand that the logarithm of the odds is a linear function of the predictor. In particular, the slope parameter \(\beta_1\) is the change in the log odds associated with a one-unit increase in \textit{x}. This implies that the odds itself changes by the multiplicative factor \(e^{\beta_1}\) when \textit{x} increases by 1 unit.

\begin{equation}
	log(\frac{\pi}{1-\pi}) = \beta_0 + \beta_1x
\end{equation}

From here, straightforward algebra will then show the Simple Linear Regression Model:

\begin{equation}
	E[Y] = \pi(x) = \frac{e^{\beta_0 + \beta_1x}}{1+e^{\beta_0 + \beta_1x}}
\end{equation}

Next, this simple logistic regression model is easily extended to more than one predictor variable by inclusion of the following two vectors, in matrix notation:

\[
	\boldsymbol{\beta} =
	\begin{bmatrix}
		\beta_0 \\ \beta_1 \\ \vdots \\ \beta_{p-1}
	\end{bmatrix} \quad
	\textbf{X} =
	\begin{bmatrix}
		1 \\ X_1 \\ X_2 \\ \vdots \\ X_{p-1}
	\end{bmatrix}
\]

With this notation, the simple logistic response function (Eqn. 3) extends to the multiple logistic response function as follows:

\begin{equation}
	E[Y] = \pi(\textbf{X}) = \frac{exp(\textbf{X}'\boldsymbol{\beta})}{1+exp(\textbf{X}'\boldsymbol{\beta})}
\end{equation}

Fitting the logistic regression to the sample data requires that the parameters \(\beta_0\), \(\beta_1\),\(\cdots\), \(\beta_{p-1}\) be estimated. This will be done using the maximum likelihood technique provided within the statistical packages of both \textbf{R} and \textit{Python}.

\subsection{Data Transformations and Standardization}
Variable transformation is an important technique to create robust models using logistic regression, and the appropriate transformations on continuous variables are necessary to optimize the model predictiveness. Because the predictors are linear in the log of the odds, it is often helpful to transform the continuous variables to create a more linear relationship. \par
The raw data collected contained several predictors with high skewness values. A few concerning features were determined to be PSA Level (skewness = 4.39), Cancer Volume (skewness = 2.18), and Weight (skewness = 7.46). As a prepossessing step to reduce skewness, I elected to transform these continuous predictor variables using the log-transformation, and standardize \textit{all} the data on top of that. The standardization step was used to normalize the data, and did not affect any underlying distributions among the predictor variables. \par
The finalized data skewness is summarized directly below in Figure 1.

\begin{figure}[H]
	\centering
	\includegraphics[scale=0.9]{final_skewness}
	\caption{Finalized Skewness Values of Transformed Predictor Variables.}
\end{figure}

Additionally, I've included the histogram of PSA Level vs. Cancer Volume in Figure 2 - a helpful visual for the two predictors which carried the most significance through much of my analysis, as we soon shall see. Notice how the distributions exhibit no notable skewness, are quite symmetrical, and are centered on zero.

\begin{figure}[H]
	\centering
	\includegraphics{psalevel_cancervol_skewness}
	\caption{Finalized PSALevel vs. CancerVol Histogram.}
\end{figure}

\subsection{Model Selection}
The data of 97 individual men in the Prostate Cancer sample was split at 80\% for train (model-building) and test (validation) sets. The training set is a random 76 observations and was used for fitting the model, and the remaining 21 cases were saved to serve as a validation data set. Figure 3 in columns 1-8 contains the variables, and shows a portion of the finalized and processed training data. \par

\begin{figure}[H]
	\centering
	\includegraphics[scale=0.7]{train_data_python}
	\caption{Portion of Processed Model-Building Data Set - \textit{Python} Dataframe.}
\end{figure}

\pagebreak
\subsubsection{Best Subsets Procedure}
The procedure outlined here will help identify a group of subset models that give the best values of a specified criterion. This technique has been developed by time-saving algorithms which can find the most promising models, without having to evaluate all \(2^{p-1}\) candidates. The use of the best subset procedure is based on the \textit{AIC\textsubscript{p}} criteria, where promising models will yield a relatively small value. \par
The minimized \textit{AIC\textsubscript{p}} stepwise output given by \textbf{R} is provided in Figure 4 below.

\begin{figure}[H]
	\centering
	\includegraphics[scale=0.9]{best_subset}
	\caption{Full Linear Model  \textit{AIC\textsubscript{p}} Best Subset Results - \textbf{R} Output.}
\end{figure}

In this procedure, I instructed \textbf{R} to iterate "backwards" through all 7 predictor variables and it was determined \textit{AIC\textsubscript{p}} was minimized for \(p=3\). In particular, the results reveal that the best two-predictor model for this criteria is based on \textit{PSA Level} and \textit{Cancer Volume}. The \textit{AIC\textsubscript{p}} was minimized to 50.63, with a Null Deviance equal to 72.61 and Residual Deviance equal to 44.63.

\subsubsection{Model Fitting}
A first-order multiple logistic regression model with two predictor variables was considered to be reasonable by \S4.2.1:

\begin{equation}
\pi(\textbf{X}) = \frac{exp(\textbf{X}'\boldsymbol{\beta})}{1+exp(\textbf{X}'\boldsymbol{\beta})} = [1+exp(-\textbf{X}'\boldsymbol{\beta})]^{-1}
\end{equation}

where:

\begin{equation}
\textbf{X}'\boldsymbol{\beta} = \beta_0+\beta_1X_1+\beta_2X_2
\end{equation}

This model was then fit by the method of maximum likelihood to the data from the 76 random training cases. Results are summarized in the Figure 5 \textit{Python} output below. Provided in the output are the estimated coefficients, their standard errors, \textit{z}-scores, \textit{p}-values, and the accompanying 95\% confidence intervals. \par

\begin{figure}[H]
	\centering
	\includegraphics[scale=0.9]{model_fit_python}
	\caption{Maximum Likelihood Estimates of Logistic Regression Function - \textit{Python} Output.}
\end{figure}

Thus, the estimated logistic response function is:

\begin{equation}
\hat{\pi}=[ 1+ exp(-2.6867 + 1.0577X_1 + 1.5502X_2)]^{-1}
\end{equation}

\textbf{Note}: Although the PSALevel predictor is not of 5\% significance (\textit{p}-value=0.0879), I did find it necessary to maintain it within the model. When removed, the Residual Deviance score of 44.628 from Figure 4 rose to a value of 48.123. Therefore, I've deemed it significant, and a valuable and impactful variable to achieve high model accuracy, and have not removed it from this subset of predictors. \par
With the estimated logistic regression equation now developed, it is left to consider second-order options and make adjustments if required, analyze the residuals and influential observations, test goodness of fit, apply a prediction rule for new observations, and finally apply the final model to the validation data and evaluate the results.

\subsubsection{Geometric Interpretation}
When fitting a standard multiple logistic regression model with two predictors, the estimated regression shape is an S-shaped surface in three-dimensional space. Figure 6 displays a three-dimensional plot of the estimated logistic response function that depicts the relationship between the diagnosis of high grade prostate cancer (\textit{Y}, the binary outcome) and two continuous predictors, PSA Level (\textit{X}\textsubscript{1}) and Cancer Volume (\textit{X}\textsubscript{2}). \par
This surface increases in an approximately linear fashion with increasing values of PSA Level and Cancer Volume, but levels off and is nearly horizontal for very small and large values of these predictors.

\begin{figure}[H]
	\centering
	\includegraphics[scale=0.6]{3d_plot}
	\caption{Three-Dimensional Fitted Logistic Response Surface.}
\end{figure}

\subsubsection{Second-Order Predictors}
Occasionally, the first-order logistic model may not provide a sufficient fit to the data, and the inclusion of higher-order predictors may be considered. I'll conclude my model development stage by attempting to fit the Prostate Cancer data to a \textit{polynomial logistic} regression model of the second order, and analyze the results. \par
For simplicity, a 2\textsuperscript{nd}-order polynomial model in \textit{two} predictors has a logit response function as:

\begin{equation}
logit(\pi) = \beta_0 + \beta_1x_1 + \beta_2x_2 + \beta_{11}x_1^2 + \beta_{22}x_2^2 + \beta_{12}x_1x_2
\end{equation}

\noindent and can be extended to more predictors by the inclusion of additional variables, their coefficients, and accompanying cross terms. Please recall, the Prostate Cancer data set considers 7 predictors. \par
In many situations the true regression function has one or more peaks or valleys, and in such cases a polynomial function can provide a satisfactory approximation. However, a polynomial fit was not successful here, as indicated by non-significant \textit{p}-values across all predictors, at 5\% significance (Figure 7).

\begin{figure}[H]
	\centering
	\includegraphics[scale=0.9]{poly_output}
	\caption{Logistic Regression Fit for Second-Order Model - \textbf{R} Output.}
\end{figure}

Additionally, my preliminary scatter plot analysis did not indicate any reason to believe a polynomial fit would be suitable in this study. For example, PSALevel was a major focus of this study and I've provided the scatter plot below in Figure 8. Additional scatter plots are provided in the Appendix, \S7.1.

\begin{figure}[H]
	\centering
	\includegraphics{psalevel_scatterplot}
	\caption{PSALevel vs Y\_HighGradeCancer Scatterplot - Train Data.}
\end{figure}

Without evidence to be concerned of successfully fitting a model with second-order predictors, I will move forward with my analysis of the previously developed multiple logistic linear regression model.

\subsection{Analysis of Residuals}
In this section I will discuss the analysis of residuals and the identification of any influential observations for logistic regression. Due to the nature of logistic regression, and the fact that non-constant variance is always present in this setting, I will focus only on the detection of model inadequacy.

\pagebreak
\subsubsection{Logistic Regression Residuals}
If the logistic regression model is correct, then \(E[Y_i]=\pi_i\) and it follows that:

\begin{equation}
E[Y_i-\hat{\pi}_i]=E[e_i]=0
\end{equation}

This suggests that if the model is correct, a lowess smooth of the plot of residuals against the linear predictor \(\hat{\pi}^{'}_i\) should result in approximately a horizontal line with zero intercept. Any significant departure from this suggests that the model may be inadequate. Shown in Figure 9 are the Pearson residuals plotted against the linear predictor, with the lowess smooth superimposed.

\begin{figure}[H]
	\centering
	\includegraphics[scale=0.70]{residual_plot}
	\caption{Pearson Residual Plot with Lowess Smooth.}
\end{figure}

Looking at the plot, the lowess smooth adequately approximates a line having zero slope and zero intercept, and I conclude that no significant model inadequacy is apparent.

\subsubsection{Influential Observations}
To aid in the identification of influential observations, I will use the \textbf{Cook's Distance} statistic, \(D_i\), which measures the standardized change in the linear predictor \(\hat{\pi}_i\) when the \textit{i}th case is deleted. Cook's distances are listed in the \textbf{R} Appendix \S7.2 for a portion of the Prostate Cancer testing data. \par
The plot of distances in Figure 10 identifies observation 90 as being the most outlying in the \textit{X} space, and therefore potentially influential - observations 37 and 91 also read relatively high values. Observation 90 was temporarily deleted and the logistic regression fit was obtained. The results were not particularly different from those obtained from the full test set, and the observation was retained. \textbf{Note}: I additionally and temporarily removed observations 37 and 91 and obtained a fit to the updated final model. The results were not particularly different, and those records were also retained. Thus, no changes to the model are yet necessary.

\begin{figure}[H]
	\centering
	\includegraphics[scale=0.55]{cooks_distance}
	\caption{Index Plot of Cook's Distances.}
\end{figure}

\subsection{Goodness Of Fit Evaluation}
The appropriateness of the fitted logistic regression model needs to be examined before it is accepted for use. In particular, we need to examine whether the estimated response function for the data is monotonic and sigmoidal in shape, as are logistic response functions. Here I will employ the Hosmer-Lemeshow test, which is useful for unreplicated data sets, as is the Prostate Cancer data. The test can detect major departures from a logistic response function, and the alternatives of interest are as follows:

\begin{align}
\begin{split}
	H_0: E[Y]=  [1+exp(-\textbf{X}'\boldsymbol{\beta})]^{-1} \\
	H_1: E[Y] \neq  [1+exp(-\textbf{X}'\boldsymbol{\beta})]^{-1}
\end{split}
\end{align}

\subsubsection{Hosmer-Lemeshow}
The Hosmer-Lemeshow Goodness of Fit procedure consists of grouping that data into classes with similar fitted values \(\hat{\pi}_i\), with approximately the same number of cases in each class. Once the groups are formed, the Hosmer-Lemeshow goodness of fit statistic is calculated by using the Pearson chi-square test statistic of observed and expected frequencies. The test statistic is known to be well approximated by the chi-square distribution with \(c-2\) degrees of freedom.

\begin{equation}
	\chi^2 = \sum_{j=1}^{c} \sum_{k=0}^{1} \frac{(O_{jk}-E_{jk})^2}{E_{jk}}
\end{equation}

The output from \textbf{R} using 5 groups is shown in Figure 11 below.

\begin{figure}[H]
	\centering
	\includegraphics{gof_detail}
	% \caption{insert caption here}
	\includegraphics{gof_results}
	\caption{Hosmer-Lemshow Goodness of Fit Test for Logistic Regression Function.}
\end{figure}

Large values of the test statistic X\textsuperscript{2} indicate that the logistic response function is not appropriate. The decision rule for testing the alternatives (Eqn. 10) when controlling the level of significance at \(\alpha\) therefore is:

\begin{align}
\begin{split}
	\textrm{If X\textsuperscript{2}} \leq \chi^2(1-\alpha; c-p)\textrm{, conclude } H_0 \\
	\textrm{If X\textsuperscript{2}} > \chi^2(1-\alpha; c-p)\textrm{, conclude } H_1
\end{split}
\end{align}

Thus, for \(\alpha=0.05\) and \(c-2=5-2=3\), we require \(\chi^2(0.95; 3)=7.81\). Since \(X^2=0.838\leq7.81\), we conclude \textit{H}\textsubscript{0}, that the logistic response function is appropriate. The \textit{p}-value of the test is 0.8403.

\subsection{Development of ROC Curve}
Multiple logistic regression is often employed for making predictions for new observations.
The \textit{receiver operating characteristic} (ROC) \textit{curve} plots \(P(\hat{Y}=1 | Y=1)\) as a function of \(1-P(\hat{Y}=0 | Y=0)\) and is an effective way to graphically display prediction rule information, and possible cutoff points. \par
The "True Positive" \textit{y}-axis on an ROC curve is also known as \textit{sensitivity}, and the "False Positive" \textit{x}-axis is 1-\textit{specificity}. Figure 12 below exhibits the ROC curve for my model (Eqn. 7) for all possible cut points between 0 and 1.

\begin{figure}[H]
	\centering
	\includegraphics[scale=0.45]{roc}
	\caption{ROC Curve.}
\end{figure}

\subsubsection{Prediction Rule}
In the training data set (which represented a random 80\% of the 97 provided observations), there were 14 men who were observed as high grade cancer patients; hence the estimated proportion of persons who had high grade cancer is \(14/76=0.184\). This proportion can be used as the starting point in the search for the best cutoff in the prediction rule. \par
Thus, if \(\hat{\pi}_h\) represents a newly fitted observation, my first prediction rule investigated is:

\begin{equation}
	\textrm{Predict 1 if } \hat{\pi}_h \geq 0.184\textrm{; predict 0 if } \hat{\pi}_h < 0.184
\end{equation}

The Confusion Matrix of Table 1 below provides a summary of the number of correct and incorrect classifications based on the initial prediction rule (Eqn. 13). Of the 62 men without high grade cancer, 13 would be incorrectly predicted to have high grade cancer, or an error rate of 21.0\%. Furthermore, of the 14 persons with high grade cancer, 1 would be incorrectly predicted to not have high grade cancer, or 7.1\%. Altogether, \(13+1=14\) of the 76 predictions would be incorrect, so that the prediction error rate for the rule is \(14/76=0.184\) or 18.4\%. Coincidentally, the model exactly matches our training set proportions with the current prediction rule. \par

\begin{table}[H]
	\centering
	\begin{tabular}{ |c||c|c||c|  }
 	\hline
 	\multicolumn{4}{|c|}{Prediction Rule Eqn. 13} \\
 	\hline\hline
 	True Classification&\(\hat{Y}=0\)&\(\hat{Y}=1\)&Total\\
 	\hline
 	\(Y=0\)&49&13&62\\
 	\(Y=1\)&1&13&14\\
 	\hline\hline
 	Total&50&26&76\\
 	\hline
	\end{tabular}
 	\caption{Classification based on Logistic Response Function Eqn. 7 and Prediction Rule Eqn. 13.}
\end{table}

\pagebreak
With this baseline understood, it is straightforward to choose a stronger cutoff point in utilizing the ROC curve of Figure 12. As detailed above, the false-positive rate is not ideal at 21.0\% - there are too many cases where a man may opt for additional screening and treatment, even invasive actions, because he believes he has high grade prostate cancer. It will be wise to now reference the ROC curve to better choose a prediction cutoff, while also not significantly disturbing the false-negative accuracy for the worse. \par
Looking at Figure 12, a step occurs at 0.20 and I use this value for my new cutoff candidate. Thus, my updated prediction rule is stated as follows:

\begin{equation}
	\textrm{Predict 1 if } \hat{\pi}_h \geq 0.20\textrm{; predict 0 if } \hat{\pi}_h < 0.20
\end{equation}

and the effects of this change can be summarized by the Confusion Matrix in Table 2 below.

\begin{table}[H]
	\centering
	\begin{tabular}{ |c||c|c||c| }
 	\hline
 	\multicolumn{4}{|c|}{Prediction Rule Eqn. 14} \\
 	\hline\hline
 	True Classification&\(\hat{Y}=0\)&\(\hat{Y}=1\)&Total\\
 	\hline
 	\(Y=0\)&52&10&62\\
 	\(Y=1\)&2&12&14\\
 	\hline\hline
 	Total&54&226&76\\
 	\hline
	\end{tabular}
 	\caption{Classification based on Logistic Response Function Eqn. 7 and Prediction Rule Eqn. 14.}
\end{table}

Here, of the 62 men without high grade cancer, 10 are incorrectly predicted, or an error rate of 16.1\%. Continuing, of the 14 men with high grade cancer, 2 would be incorrectly predicted, or an error rate of 14.3\%. Altogether, updated prediction rule (Eqn. 14) now provides a total error rate of \(12/76=0.158\) or 15.8\%. Thus, the model accuracy has now increased with a significantly better false-positive rate, which is intended to reduce unnecessary financial stress across the healthcare economy.

\subsection{Model: Strengths and Weaknesses}
\subsubsection{Strengths}
-The two predictors which build the final logistic model are PSA Level and Cancer Volume, and they both are adequately correlated with the dependent (outcome) variable Y\_HighGradeCancer - their correlation values are 0.489 and 0.493, respectively. In fact, they are more correlated to the dependent variable than any other predictors of the data set. A consumable heat-map version of a correlation matrix is provided by Figure 13 below, with a color legend given in the upper-left corner.

\begin{figure}[H]
	\centering
	\includegraphics[scale=0.7]{corr_map}
	\caption{Correlation Heatmap - Train Data}
\end{figure}

-The final model has an excellent accuracy score of 84.2\% against training data. If this model also performs well in the validation step, then deploying such a model in use could possibly help non high grade cancer men be categorized as such, and thus not pursue unnecessary invasive testing and not inflate costs within the healthcare system. Also, by properly identifying those men who are high grade cancer patients, treatment and a plan can be devised sooner, as well as doing so by only use of PSA Level and Cancer Volume information, and not invasive testing. \\

\subsubsection{Weaknesses}
-An initial concern while building this model occurs at the level of the provided raw data, namely the existence of multicollinearity. Figure 14 below is the Correlation Matrix of the two final predictors which built the final logistic model: PSA Level and Cancer Volume.

\begin{figure}[H]
	\centering
	\includegraphics[scale=1.0]{corr_matrix}
	\caption{PSALevel vs. CancerVol Correlation Matrix - Train Data}
\end{figure}

\pagebreak
As shown, PSA Level and Cancer Volume have a mild correlation value of 0.738 in the full data set. One primary danger in designing models with multicollinearity is that small changes to the input data can lead to large changes in the model, which can further lead to over-fitting. Therefore, this logistic model may be considered mildly "noisy", sensitive, and not particularly robust. \\

-The Goodness of Fit Evaluation of \S4.4 deserves some concern regarding the Pearson chi-square test. As described previously, the Hosmer-Lemeshow procedure was utilized to determine a goodness of fit, and the test statistic is known to be well approximated by the chi-square distribution with \(c-2\) degrees of freedom (Eqn. 11). However, in view of the \textbf{R} output (Figure 11) with 5 groupings, the expected values (\textit{y\textsubscript{1}}) returned were: 0, 0, 1, 4, 9. Because many values are less than 5, and two of the expected values equal 0, the conditions for a chi-square test may be voided, and it may not be an appropriate test procedure here. At the very lest, the results of the Hosmer-Lemeshow test should be accepted carefully. \\

-The final model may produce high false-negative rates. In view of Figure 15-A we see that the first prediction cutoff of 0.184 produced 1 count of false-negatives (\(7.1\%\) error rate), and an overall model accuracy of \(81.6\%\). After ROC analysis the final prediction cutoff I've employed is 0.20. By Figure 15-B this rule produces 2 counts of false-negatives (\(14.3\%\) error rate), and an overall model accuracy of 84.2\%. Thus the overall model accuracy has improved 2.6 percentage points by correctly predicting more non high grade cancer cases, but has concurrently doubled the false-negative rate. Because correctly identifying high grade cancer patients may be considered most important, this arrangement may be a downfall of the final model.

\begin{figure}[H]
\centering
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=.8\linewidth]{confusion_matrix_PR1}
  \caption{Confusion Matrix - 0.184 Cutoff Rule.}
  \label{fig:sub1}
\end{subfigure}%
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=.8\linewidth]{confusion_matrix_PR2}
  \caption{Confusion Matrix - 0.20 Cutoff Rule}
  \label{fig:sub2}
\end{subfigure}
\caption{Classification Based on Logistic Response Function (Eqn. 7) and Prediction Rules (Eqn. 13) and (Eqn. 14).}
\label{fig:test}
\end{figure}


\chapter{Latent linear models}

\section{Factor analysis}
One problem with mixture models is that they only use a single latent variable to generate the observations. In particular, each observation can only come from one of $K$ prototypes. One can think of a mixture model as using $K$ hidden binary variables, representing a one-hot encoding of the cluster identity. But because these variables are mutually exclusive, the model is still limited in its representational power.

An alternative is to use a vector of real-valued latent variables,$\vec{z}_i \in \mathbb{R}^L$. The simplest prior to use is a Gaussian (we will consider other choices later):
\begin{equation}\label{eqn:FA-prior}
p(\vec{z}_i)=\mathcal{N}(\vec{z}_i|\vec{\mu}_0,\vec{\Sigma}_0)
\end{equation}
If the observations are also continuous, so $\vec{x}_i \in \mathbb{R}^D$, we may use a Gaussian for the likelihood. Just as in linear regression, we will assume the mean is a linear function of the (hidden) inputs, thus yielding
\begin{equation}\label{eqn:FA-class-conditional-density}
p(\vec{x}_i|\vec{z}_i,\vec{\theta})=\mathcal{N}(\vec{x}_i|\vec{W}\vec{z}_i+\vec{\mu},\vec{\Psi})
\end{equation}
where \vec{W} is a $D \times L$ matrix, known as the \textbf{factor loading matrix}, and $\vec{\Psi}$ is a $D \times D$ covariance matrix. We take $\vec{\Psi}$ to be diagonal, since the whole point of the model is to “force” $\vec{z}_i$ to explain the correlation, rather than “baking it in” to the observation’s covariance. This overall model is called \textbf{factor analysis} or \textbf{FA}. The special case in which $\vec{\Psi}=\sigma^2\vec{I}$ is called \textbf{probabilistic principal components analysis} or \textbf{PPCA}. The reason for this name will become apparent later.

\subsection{FA is a low rank parameterization of an MVN}
FA can be thought of as a way of specifying a joint density model on $\vec{x}$ using a small number of parameters. To see this, note that from Equation \ref{eqn:Linear-Gaussian-system-normalizer}, the induced marginal distribution $p(\vec{x}_i|\vec{\theta})$ is a Gaussian:
\begin{align}
p(\vec{x}_i|\vec{\theta}) & = \int \mathcal{N}(\vec{x}_i|\vec{W}\vec{z}_i+\vec{\mu},\vec{\Psi})\mathcal{N}(\vec{z}_i|\vec{\mu}_0,\vec{\Sigma}_0)\mathrm{d}\vec{z}_i \nonumber \\
 & = \mathcal{N}(\vec{x}_i|\vec{W}\vec{\mu}_0+\vec{\mu},\vec{\Psi}+\vec{W}\vec{\Sigma}_0\vec{W})
\end{align}
From this, we see that we can set $\vec{\mu}_0=0$ without loss of generality, since we can always absorb $\vec{W}\vec{\mu}_0$ into $\vec{\mu}$. Similarly, we can set $\vec{\Sigma}_0=\vec{I}$ without loss of generality, because we can always “emulate” a correlated prior by using defining a new weight matrix, $\tilde{\vec{W}}=\vec{W}\vec{\Sigma}_0^{-\frac{1}{2}}$. So we can rewrite Equation \ref{eqn:FA-prior} and \ref{eqn:FA-class-conditional-density} as:
\begin{align}
p(\vec{z}_i)&=\mathcal{N}(\vec{z}_i|\vec{0},\vec{I}) \\
p(\vec{x}_i|\vec{z}_i,\vec{\theta})&=\mathcal{N}(\vec{x}_i|\vec{W}\vec{z}_i+\vec{\mu},\vec{\Psi})
\end{align}

We thus see that FA approximates the covariance matrix of the visible vector using a low-rank decomposition:
\begin{equation}\label{eqn:FA-prior}
\vec{C} \triangleq \mathrm{cov}[\vec{x}]=\vec{W}\vec{W}^T+\vec{\Psi}
\end{equation}
This only uses $O(LD)$ parameters, which allows a flexible compromise between a full covariance Gaussian, with $O(D^2)$ parameters, and a diagonal covariance, with $O(D)$ parameters. Note that if we did not restrict $\vec{\Psi}$ to be diagonal, we could trivially set $\vec{\Psi}$ to a full covariance matrix; then we could set $\vec{W}=0$, in which case the latent factors would not be required.

\subsection{Inference of the latent factors}
\begin{align}
p(\vec{z}_i|\vec{x}_i,\vec{\theta}) & = \mathcal{N}(\vec{z}_i|\vec{\mu}_i,\vec{\Sigma}_i) \\
\vec{\Sigma}_i & \triangleq (\vec{\Sigma}_0^{-1}+\vec{W}^T\vec{\Psi}^{-1}\vec{W})^{-1} \\
               & =(\vec{I}+\vec{W}^T\vec{\Psi}^{-1}\vec{W})^{-1} \\
\vec{\mu}_i & \triangleq \vec{\Sigma}_i[\vec{W}^T\vec{\Psi}^{-1}(\vec{x}_i-\vec{\mu})+\vec{\Sigma}_0^{-1}\vec{\mu}_0] \\
            & =\vec{\Sigma}_i\vec{W}^T\vec{\Psi}^{-1}(\vec{x}_i-\vec{\mu})
\end{align}
Note that in the FA model, $\vec{\Sigma}_i$ is actually independent of $i$, so we can denote it by $\vec{\Sigma}$. Computing this matrix takes $O(L^3+L^2D)$ time, and computing each $\vec{\mu}_i=\mathbb{E}[\vec{z}_i|\vec{x}_i,\vec{\theta}]$ takes $O(L^2+LD)$ time. The $\vec{\mu}_i$ are sometimes called the \textbf{latent scores}, or \textbf{latent factors}.

\subsection{Unidentifiability}
Just like with mixture models, FA is also unidentifiable. To see this, suppose $\vec{R}$ is an arbitrary orthogonal rotation matrix, satisfying $\vec{R}\vec{R}^T=\vec{I}$. Let us define $\tilde{\vec{W}}=\vec{W}\vec{R}$, then the likelihood function of this modified matrix is the same as for the unmodified matrix, since $\vec{W}\vec{R}\vec{R}^T\vec{W}^T+\vec{\Psi}=\vec{W}\vec{W}^T+\vec{\Psi}$. Geometrically, multiplying $\vec{W}$ by an orthogonal matrix is like rotating $\vec{z}$ before generating $\vec{x}$.

To ensure a unique solution, we need to remove $L(L-1)/2$ degrees of freedom, since that is the number of orthonormal matrices of size $L \times L$.\footnote{To see this, note that there are $L-1$ free parameters in $\vec{R}$ in the first column (since the column vector must be normalized to unit length), there are $L-2$ free parameters in the second column (which must be orthogonal to the first), and so on.} In total, the FA model has $D+LD-L(L-1)/2$ free parameters (excluding the mean), where the first term arises from $\vec{\Psi}$. Obviously we require this to be less than or equal to $D(D+1)/2$, which is the number of parameters in an unconstrained (but symmetric) covariance matrix. This gives us an upper bound on $L$, as follows:
\begin{equation}
L_{\mathrm{max}}=\lfloor D+0.5(1-\sqrt{1+8D}) \rfloor
\end{equation}

For example, $D=6$ implies $L \leq 3$. But we usually never choose this upper bound, since it would result in overfitting (see discussion in Section \ref{sec:FA-Choosing-L} on how to choose $L$).

Unfortunately, even if we set $L < L_{\mathrm{max}}$, we still cannot uniquely identify the parameters, since the rotational ambiguity still exists. Non-identifiability does not affect the predictive performance of the model. However, it does affect the loading matrix, and hence the interpretation of the latent factors. Since factor analysis is often used to uncover structure in the data, this problem needs to be addressed. Here are some commonly used solutions:
\begin{itemize}
\item{\textbf{Forcing $\vec{W}$ to be orthonormal} Perhaps the cleanest solution to the identifiability problem is to force $\vec{W}$ to be orthonormal, and to order the columns by decreasing variance of the corresponding latent factors. This is the approach adopted by PCA, which we will discuss in Section \ref{sec:PCA}. The result is not necessarily more interpretable, but at least it is unique.}
\item{\textbf{Forcing $\vec{W}$ to be lower triangular} One way to achieve identifiability, which is popular in the Bayesian community (e.g., (Lopes and West 2004)), is to ensure that the first visible feature is only generated by the first latent factor, the second visible feature is only generated by the first two latent factors, and so on. For example, if $L=3$ and $D=4$, the correspond factor loading matrix is given by
\begin{equation*}
\vec{W}=\left(\begin{array}{ccc}
w_{11} & 0 & 0 \\
w_{21} & w_{22} & 0 \\
w_{31} & w_{32} & w_{33} \\
w_{41} & w_{32} & w_{43}
\end{array}\right)
\end{equation*}
We also require that $w_{jj} >0$ for $j =1:L$. The total number of parameters in this constrained matrix is $D+DL-L(L-1)/2$, which is equal to the number of uniquely identifiable parameters. The disadvantage of this method is that the first $L$ visible variables, known as the \textbf{founder variables}, affect the interpretation of the latent factors, and so must be chosen carefully.}
\item{\textbf{Sparsity promoting priors on the weights} Instead of pre-specifying which entries in $\vec{W}$ are zero, we can encourage the entries to be zero, using $\ell_1$ regularization (Zou et al. 2006), ARD (Bishop 1999; Archambeau and Bach 2008), or spike-and-slab priors (Rattray et al. 2009). This is called sparse factor analysis. This does not necessarily ensure a unique MAP estimate, but it does encourage interpretable solutions. See Section 13.8 TODO.}
\item{\textbf{Choosing an informative rotation matrix} There are a variety of heuristic methods that try to find rotation matrices $\vec{R}$ which can be used to modify $\vec{W}$(and hence the latent factors) so as to try to increase the interpretability, typically by encouraging them to be (approximately) sparse. One popular method is known as \textbf{varimax}(Kaiser 1958).}
\item{\textbf{Use of non-Gaussian priors for the latent factors} In Section \ref{sec:ICA}, we will dicuss how replacing $p(\vec{z}_i)$ with a non-Gaussian distribution can enable us to sometimes uniquely identify $\vec{W}$ as well as the latent factors. This technique is known as ICA.}
\end{itemize}

\subsection{Mixtures of factor analysers}
The FA model assumes that the data lives on a low dimensional linear manifold. In reality, most
data is better modeled by some form of low dimensional \emph{curved} manifold. We can approximate a curved manifold by a piecewise linear manifold. This suggests the following model: let the $k$'th linear subspace of dimensionality $L_k$ be represented by $\vec{W}_k$, for $k=1:K$. Suppose we have a latent indicator $qi \in \{1,\cdots,K\}$ specifying which subspace we should use to generate the data. We then sample $\vec{z}_i$ from a Gaussian prior and pass it through the $\vec{W}_k$ matrix (where $k=q_i$), and add noise. More precisely, the model is as follows:
\begin{align}
p(q_i|\vec{\theta}) & =\mathrm{Cat}(q_i\vec{\pi}) \\
p(\vec{z}_i|\vec{\theta}) & =\mathcal{N}(\vec{z}_i|\vec{0},\vec{I}) \\
p(\vec{x}_i|q_i=k,\vec{z}_i,\vec{\theta}) & =\mathcal{N}(\vec{x}_i|\vec{W}\vec{z}_i+\vec{\mu}_k,\vec{\Psi})
\end{align}
This is called a \textbf{mixture of factor analysers}(MFA) (Hinton et al. 1997).

Another way to think about this model is as a low-rank version of a mixture of Gaussians. In particular, this model needs $O(KLD)$ parameters instead of the $O(KD^2)$ parameters needed for a mixture of full covariance Gaussians. This can reduce overfitting. In fact, MFA is a good generic density model for high-dimensional real-valued data.

\subsection{EM for factor analysis models}
Below we state the results without proof. The derivation can be found in (Ghahramani and Hinton 1996a). To obtain the results for a single factor analyser, just set $r_{ic} =1$ and $c=1$ in the equations below. In Section \ref{sec:EM-for-PCA} we will see a further simplification of these equations that arises when fitting a PPCA model, where the results will turn out to have a particularly simple and elegant interpretation.

In the E-step, we compute the posterior responsibility of cluster $k$ for data point $i$ using
\begin{equation}
r_{ik} \triangleq p(q_i=k|\vec{x}_i,\vec{\theta}) \propto \pi_k\mathcal{N}(\vec{x}_i|\vec{\mu}_k,\vec{W}_k\vec{W}_k^T\vec{\Psi}_k)
\end{equation}

The conditional posterior for $\vec{z}_i$ is given by
\begin{align}
p(\vec{z}_i|\vec{x}_i,q_i=k,\vec{\theta}) & = \mathcal{N}(\vec{z}_i|\vec{\mu}_{ik},\vec{\Sigma}_{ik}) \\
\vec{\Sigma}_{ik} & \triangleq (\vec{I}+\vec{W}_k^T\vec{\Psi}_k^{-1}\vec{W})_k^{-1} \\
\vec{\mu}_{ik} & \triangleq \vec{\Sigma}_{ik}\vec{W}_k^T\vec{\Psi}_k^{-1}(\vec{x}_i-\vec{\mu}_k)
\end{align}

In the M step, it is easiest to estimate $\vec{\mu}_k$ and $\vec{W}_k$ at the same time, by defining $\tilde{\vec{W}}_k=(\vec{W}_k,\vec{\mu}_k)$, $\tilde{\vec{z}}=(\vec{z},1)$, also, define
\begin{align}
\tilde{\vec{W}}_k & =(\vec{W}_k,\vec{\mu}_k) \\
\tilde{\vec{z}} & =(\vec{z},1) \\
\vec{b}_{ik} & \triangleq \mathbb{E}[\tilde{\vec{z}}|\vec{x}_i,q_i=k]=\mathbb{E}[(\vec{\mu}_{ik};1)] \\
\vec{C}_{ik} & \triangleq \mathbb{E}[\tilde{\vec{z}}\tilde{\vec{z}}^T|\vec{x}_i,q_i=k] \\
    & =\left(\begin{array}{cc}
\mathbb{E}[\vec{z}\vec{z}^T|\vec{x}_i,q_i=k] & \mathbb{E}[\vec{z}|\vec{x}_i,q_i=k] \\
\mathbb{E}[\vec{z}|\vec{x}_i,q_i=k]^T & 1
\end{array}\right)
\end{align}
Then the M step is as follows:
\begin{align}
\hat{\pi}_k & = \frac{1}{N}\sum\limits_{i=1}^N r_{ik} \\
\hat{\tilde{\vec{W}}}_k & = \left(\sum\limits_{i=1}^N r_{ik}\vec{x}_i\vec{b}_{ik}^T\right)\left(\sum\limits_{i=1}^N r_{ik}\vec{x}_i\vec{C}_{ik}^T\right)^{-1} \label{eqn:FA-EM-W} \\
\hat{\vec{\Psi}} & = \frac{1}{N}\mathrm{diag}\left[\sum\limits_{i=1}^N r_{ik}(\vec{x}_i-\hat{\tilde{\vec{W}}}_{ik}\vec{b}_{ik})\vec{x}_i^T\right]
\end{align}

Note that these updates are for “vanilla” EM. A much faster version of this algorithm, based on ECM, is described in (Zhao and Yu 2008).

\subsection{Fitting FA models with missing data}
\label{sec:Fitting-FA-models-with-missing-data}
In many applications, such as collaborative filtering, we have missing data. One virtue of the EM approach to fitting an FA/PPCA model is that it is easy to extend to this case. However, overfitting can be a problem if there is a lot of missing data. Consequently it is important to perform MAP estimation or to use Bayesian inference. See e.g., (Ilin and Raiko 2010) for details.

\section{Principal components analysis (PCA)}
\label{sec:PCA}
Consider the FA model where we constrain $\vec{\Psi}=\sigma^2\vec{I}$, and $\vec{W}$ to be orthonormal. It can be shown (Tipping and Bishop 1999) that, as $\sigma^2 \rightarrow 0$, this model reduces to classical (nonprobabilistic) \textbf{principal components analysis}(PCA), also known as the Karhunen Loeve transform. The version where $\sigma^2 > 0$ is known as \textbf{probabilistic PCA}(\textbf{PPCA}) (Tipping and Bishop 1999), orsensible PCA(Roweis 1997).

\subsection{Classical PCA}

\subsubsection{Statement of the theorem}
The synthesis viewof classical PCA is summarized in the forllowing theorem.

\begin{theorem}
Suppose we want to find an orthogonal set of $L$ linear basis vectors $\vec{w}_j \in \mathbb{R}^D$, and the corresponding scores $\vec{z}_i \in \mathbb{R}^L$, such that we minimize the average \textbf{reconstruction error}
\begin{equation}
J(\vec{W},\vec{Z})=\frac{1}{N}\sum\limits_{i=1}^N \lVert\vec{x}_i-\hat{\vec{x}}_i\rVert^2
\end{equation}
where $\hat{\vec{x}}_i=\vec{W}\vec{z}_i$, subject to the constraint that $\vec{W}$ is orthonormal. Equivalently, we can write this objective as follows
\begin{equation}
J(\vec{W},\vec{Z})=\frac{1}{N} \lVert\vec{X}-\vec{W}\vec{Z}^T\rVert^2
\end{equation}
where $\vec{Z}$ is an $N \times L$ matrix with the $\vec{z}_i$ in its rows, and $\lVert A\rVert_F$ is the \textbf{Frobenius norm} of matrix $\vec{A}$, defined by
\begin{equation}
\lVert A\rVert_F \triangleq \sqrt{\sum\limits_{i=1}^M \sum\limits_{j=1}^N a_{ij}^2}=\sqrt{\mathrm{tr}(\vec{A}^T\vec{A})}
\end{equation}

The optimal solution is obtained by setting $\hat{\vec{W}}=\vec{V}_L$, where $\vec{V}_L$ contains the $L$ eigenvectors with largest eigenvalues of the empirical covariance matrix, $\hat{\vec{\Sigma}}=\frac{1}{N}\sum_{i=1}^N \vec{x}_i\vec{x}_i^T$. (We assume the $\vec{x}_i$ have zero mean, for notational simplicity.) Furthermore, the optimal low-dimensional encoding of the data is given by $\hat{\vec{z}}_i=\vec{W}^T\vec{x}_i$, which is an orthogonal projection of the data onto the column space spanned by the eigenvectors.
\end{theorem}

An example of this is shown in Figure \ref{fig:PCA-PPCA}(a) for $D=2$ and $L=1$. The diagonal line is the vector $\vec{w}_1$; this is called the first principal component or principal direction. The data points $\vec{x}_i \in \mathbb{R}^2$ are orthogonally projected onto this line to get $\vec{z}_i \in \mathbb{R}$. This is the best 1-dimensional approximation to the data. (We will discuss Figure \ref{fig:PCA-PPCA}(b) later.)

\begin{figure}[hbtp]
\centering
\subfloat[]{\includegraphics[scale=.60]{PCA-PPCA-a.png}} \\
\subfloat[]{\includegraphics[scale=.60]{PCA-PPCA-b.png}}
\caption{An illustration of PCA and PPCA where $D=2$ and $L=1$. Circles are the original data points, crosses are the reconstructions. The red star is the data mean. (a) PCA. The points are orthogonally projected onto the line. (b) PPCA. The projection is no longer orthogonal: the reconstructions are shrunk towards the data mean (red star).}
\label{fig:PCA-PPCA}
\end{figure}

The principal directions are the ones along which the data shows maximal variance. This means that PCA can be “misled” by directions in which the variance is high merely because of the measurement scale. It is therefore standard practice to standardize the data first, or equivalently, to work with correlation matrices instead of covariance matrices.

\subsubsection{Proof *}
See Section 12.2.2 of MLAPP.

\subsection{Singular value decomposition (SVD)}
We have defined the solution to PCA in terms of eigenvectors of the covariance matrix. However, there is another way to obtain the solution, based on the \textbf{singular value decomposition}, or \textbf{SVD}. This basically generalizes the notion of eigenvectors from square matrices to any kind of matrix.

\begin{theorem}(\textbf{SVD}).
Any matrix can be decomposed as follows
\begin{equation}\label{eqn:SVD}
\underbrace{\vec{X}}_{N \times D}=\underbrace{\vec{U}}_{N \times N}\underbrace{\vec{\Sigma}}_{N \times D}\underbrace{\vec{V}^T}_{D \times D}
\end{equation}
where $\vec{U}$ is an $N \times N$ matrix whose columns are orthornormal(so $\vec{U}^T\vec{U}=\vec{I}$), $\vec{V}$ is $D \times D$ matrix whose rows and columns are orthonormal (so $\vec{V}^T\vec{V}=\vec{V}\vec{V}^T=\vec{I}_D$), and $\vec{\Sigma}$ is a $N \times D$ matrix containing the $r=\min(N,D)$ singular values $\sigma_i \geq 0$ on the main diagonal, with 0s filling the rest of the matrix.
\end{theorem}

This shows how to decompose the matrix $X$ into the product of three matrices: $\vec{V}$ describes an orthonormal basis in the domain, and $\vec{U}$ describes an orthonormal basis in the co-domain, and $\vec{\Sigma}$ describes how much the vectors in $\vec{V}$ are stretched to give the vectors in $\vec{U}$.

Since there are at most $D$ singular values (assuming $N>D$), the last $N−D$ columns of $\vec{U}$ are irrelevant, since they will be multiplied by 0. The \textbf{economy sized SVD}, or \textbf{thin SVD}, avoids computing these unnecessary elements. Let us denote this decomposition by $\hat{\vec{U}}\hat{\vec{\Sigma}}\hat{\vec{V}}^T$.If $N>D$, we have
\begin{equation}
\underbrace{\vec{X}}_{N \times D}=\underbrace{\hat{\vec{U}}}_{N \times D}\underbrace{\hat{\vec{\Sigma}}}_{D \times D}\underbrace{\hat{\vec{V}}^T}_{D \times D}
\end{equation}
as in Figure \ref{fig:SVD}(a). If $N<D$, we have
\begin{equation}
\underbrace{\vec{X}}_{N \times D}=\underbrace{\hat{\vec{U}}}_{N \times N}\underbrace{\hat{\vec{\Sigma}}}_{N \times N}\underbrace{\hat{\vec{V}}^T}_{N \times D}
\end{equation}
Computing the economy-sized SVD takes $O(ND\min(N,D))$ time (Golub and van Loan 1996, p254).

\begin{figure}[hbtp]
\centering
\subfloat[]{\includegraphics[scale=.50]{SVD-a.png}} \\
\subfloat[]{\includegraphics[scale=.50]{SVD-b.png}}
\caption{(a) SVD decomposition of non-square matrices $\vec{X}=\vec{U}\vec{\Sigma}\vec{V}^T$. The shaded parts of $\vec{\Sigma}$, and all the off-diagonal terms, are zero. The shaded entries in $\vec{U}$ and $\vec{\Sigma}$ are not computed in the economy-sized version, since they are not needed. (b) Truncated SVD approximation of rank $L$.}
\label{fig:SVD}
\end{figure}

The connection between eigenvectors and singular vectors is the following:
\begin{align}
\vec{U}&=\mathrm{evec}(\vec{X}\vec{X}^T) \\
\vec{V}&=\mathrm{evec}(\vec{X}^T\vec{X}) \\
\vec{\Sigma}^2&=\mathrm{eval}(\vec{X}\vec{X}^T)=\mathrm{eval}(\vec{X}^T\vec{X})
\end{align}
For the proof please read Section 12.2.3 of MLAPP.

Since the eigenvectors are unaffected by linear scaling of a matrix, we see that the right singular vectors of $\vec{X}$ are equal to the eigenvectors of the empirical covariance $\hat{\vec{\Sigma}}$. Furthermore, the eigenvalues of $\hat{\vec{\Sigma}}$ are a scaled version of the squared singular values.

However, the connection between PCA and SVD goes deeper. From Equation \ref{eqn:SVD}, we can represent a rank $r$ matrix as follows:
\begin{equation*}
\vec{X}=\sigma_1\left(\begin{array}{c} | \\ \vec{u}_1 \\ | \end{array}\right)\left(\begin{array}{ccc} - & \vec{v}_1 & - \end{array}\right)+\cdots+\sigma_r\left(\begin{array}{c} | \\ \vec{u}_r \\ | \end{array}\right)\left(\begin{array}{ccc} - & \vec{v}_r^T & - \end{array}\right)
\end{equation*}
If the singular values die off quickly, we can produce a rank $L$ approximation to the matrix as follows:
\begin{align}
\vec{X} & \approx \sigma_1\left(\begin{array}{c} | \\ \vec{u}_1 \\ | \end{array}\right)\left(\begin{array}{ccc} - & \vec{v}_1 & - \end{array}\right)+\cdots+\sigma_r\left(\begin{array}{c} | \\ \vec{u}_L \\ | \end{array}\right)\left(\begin{array}{ccc} - & \vec{v}_L^T & - \end{array}\right) \nonumber \\
 & = \vec{U}_{:,1:L}\vec{\Sigma}_{1:L,1:L}\vec{V}_{:,1:L}^T
\end{align}
This is called a \textbf{truncated SVD} (see Figure \ref{fig:SVD}(b)).

One can show that the error in this approximation is given by
\begin{equation}
\lVert \vec{X}-\vec{X}_L \rVert_F \approx \sigma_L
\end{equation}
Furthermore, one can show that the SVD offers the best rank $L$ approximation to a matrix (best in the sense of minimizing the above Frobenius norm).

Let us connect this back to PCA. Let $\vec{X}=\vec{U}\vec{\Sigma}\vec{V}^T$ be a truncated SVD of $\vec{X}$. We know that $\hat{\vec{W}}=\vec{V}$, and that $\hat{\vec{Z}}=\vec{X}\hat{\vec{W}}$, so
\begin{equation}
\hat{\vec{Z}}=\vec{U}\vec{\Sigma}\vec{V}^T\vec{V}=\vec{U}\vec{\Sigma}
\end{equation}
Furthermore, the optimal reconstruction is given by $\hat{\vec{X}}=\vec{Z}\hat{\vec{W}}$,so we find
\begin{equation}
\hat{\vec{X}}=\vec{U}\vec{\Sigma}\vec{V}^T
\end{equation}
This is precisely the same as a truncated SVD approximation! This is another illustration of the fact that PCA is the best low rank approximation to the data.

\subsection{Probabilistic PCA}
\begin{theorem}((\textbf{Tipping and Bishop 1999})).
Consider a factor analysis model in which $\vec{\Psi}=\sigma^2\vec{I}$ and $\vec{W}$ is orthogonal. The observed data log likelihood is given by
\begin{align}
\log p(\vec{X}|\vec{W},\sigma^2\vec{I}) & =-\frac{N}{2}\ln|\vec{C}|-\frac{1}{2}\sum\limits_{i=1}^N \vec{x}_i^T\vec{C}^{-1}\vec{x}_i \nonumber \\
   & =-\frac{N}{2}\ln|\vec{C}|+\mathrm{tr}(\vec{C}^{-1}\vec{\Sigma})
\end{align}
where $\vec{C}=\vec{W}\vec{W}^T+\sigma^2\vec{I}$ and $\vec{\Sigma}=\frac{1}{N}\sum_{i=1}^N \vec{x}_i\vec{x}_i^T=\frac{1}{N}\vec{X}\vec{X}^T$. (We are assuming centred data, for notational simplicity.) The maxima of the log-likelihood are given by
\begin{equation}
\hat{\vec{W}}=\vec{V}(\vec{\Lambda}-\sigma^2\vec{I})^{\frac{1}{2}}\vec{R}
\end{equation}
where $\vec{R}$ is an arbitrary $L \times L$ orthogonal matrix, $\vec{V}$ is the $D \times L$ matrix whose columns are the first $L$ eigenvectors of $\vec{\Sigma}$, and $\vec{\Lambda}$ is the corresponding diagonal matrix of eigenvalues. Without loss of generality, we can set $\vec{R}=\vec{I}$. Furthermore, the MLE of the noise variance is given by
\begin{equation}
\hat{\sigma}^2=\frac{1}{D-L}\sum\limits_{j=L+1}^D \lambda_j
\end{equation}
which is the average variance associated with the discarded dimensions.
\end{theorem}

Thus, as $\sigma^2 \rightarrow 0$, we have $\hat{\vec{W}} \rightarrow \vec{V}$, as in classical PCA. What about $\hat{\vec{Z}}$? It is easy to see that the posterior over the latent factors is given by
\begin{align}
p(\vec{z}_i|\vec{x}_i,\hat{\vec{\theta}}) & = \mathcal{N}(\vec{z}_i|\hat{\vec{F}}^{-1}\hat{\vec{W}}^T\vec{x}_i,\sigma^2\hat{\vec{F}}^{-1}) \label{eqn:PPCA-posterior}\\
\hat{\vec{F}} & \triangleq \hat{\vec{W}}^T\hat{\vec{W}}+\sigma^2\vec{I}
\end{align}
(Do not confuse $\vec{F} = \vec{W}^T\vec{W}+\sigma^2\vec{I}$ with $\vec{C}=\vec{W}\vec{W}^T+\sigma^2\vec{I}$.) Hence, as $\sigma^2 \rightarrow 0$, we find $\hat{\vec{W}} \rightarrow \vec{V}$, $\hat{\vec{F}} \rightarrow \vec{I}$ and $\vec{z}_i \rightarrow \vec{V}^T\vec{x}_i$. Thus the posterior mean is obtained by an orthogonal projection of the data onto the column space of $\vec{V}$, as in classical PCA.

Note, however, that if $\sigma^2 \rightarrow 0$, the posterior mean is not an orthogonal projection, since it is shrunk somewhat towards the prior mean, as illustrated in Figure \ref{fig:PCA-PPCA}(b). This sounds like an undesirable property, but it means that the reconstructions will be closer to the overall data mean, $\hat{\vec{\mu}}=\bar{\vec{x}}$.

\subsection{EM algorithm for PCA}
\label{sec:EM-for-PCA}
Although the usual way to fit a PCA model uses eigenvector methods, or the SVD, we can also use EM, which will turn out to have some advantages that we discuss below. EM for PCA relies on the probabilistic formulation of PCA. However the algorithm continues to work in the zero noise limit, $\sigma^2=0$, as shown by (Roweis 1997).

Let $\tilde{\vec{Z}}$ be a $L \times N$ matrix storing the posterior means (low-dimensional representations) along its columns. Similarly, let $\tilde{\vec{X}}=\vec{X}^T$ store the original data along its columns. From Equation \ref{eqn:PPCA-posterior}, when $\sigma^2=0$, we have
\begin{equation}
\tilde{\vec{Z}}=(\vec{W}^T\vec{W})^{-1}\vec{W}^T\tilde{\vec{X}}
\end{equation}
This constitutes the E step. Notice that this is just an orthogonal projection of the data.

From Equation \ref{eqn:FA-EM-W}, the M step is given by
\begin{equation}
\hat{\vec{W}}=\left(\sum\limits_{i=1}^N \vec{x}_i\mathbb{E}[\vec{z}_i]^T\right)\left(\sum\limits_{i=1}^N \mathbb{E}[\vec{z}_i]\mathbb{E}[\vec{z}_i]^T\right)^{-1}
\end{equation}
where we exploited the fact that $\vec{\Sigma}=\mathrm{cov}[\vec{z}_i|\vec{x}_i,\vec{\theta}]=\vec{0}$ when $\sigma^2=0$.

(Tipping and Bishop 1999) showed that the only stable fixed point of the EM algorithm is the globally optimal solution. That is, the EM algorithm converges to a solution where $\vec{W}$ spans the same linear subspace as that defined by the first $L$ eigenvectors. However, if we want $\vec{W}$ to be orthogonal, and to contain the eigenvectors in descending order of eigenvalue, we have to orthogonalize the resulting matrix (which can be done quite cheaply). Alternatively, we can modify EM to give the principal basis directly (Ahn and Oh 2003).

This algorithm has a simple physical analogy in the case $D=2$ and $L=1$(Roweis 1997). Consider some points in $\mathbb{R}^2$ attached by springs to a rigid rod, whose orientation is defined by a vector $\vec{w}$. Let $\vec{z}_i$ be the location where the $i$'th spring attaches to the rod. See Figure 12.11 of MLAPP for an illustration.

Apart from this pleasing intuitive interpretation, EM for PCA has the following advantages over eigenvector methods:
\begin{itemize}
\item{EM can be faster. In particular, assuming $N,D \gg L$, the dominant cost of EM is the projection operation in the E step, so the overall time is $O(TLND)$, where $T$ is the number of iterations. This is much faster than the $O(\min(ND^2,DN^2))$ time required by straightforward eigenvector methods, although more sophisticated eigenvector methods, such as the Lanczos algorithm, have running times comparable to EM.}
\item{EM can be implemented in an online fashion, i.e., we can update our estimate of $\vec{W}$ as the data streams in.}
\item{EM can handle missing data in a simple way (see Section \ref{sec:Fitting-FA-models-with-missing-data}).}
\item{EM can be extended to handle mixtures of PPCA/ FA models.}
\item{EM can be modified to variational EM or to variational Bayes EM to fit more complex models.}
\end{itemize}

\section{Choosing the number of latent dimensions}
\label{sec:FA-Choosing-L}
In Section \ref{sec:Model-selection-for-LVM}, we discussed how to choose the number of components $K$ in a mixture model. In this section, we discuss how to choose the number of latent dimensions $L$ in a FA/PCA model.

\subsection{Model selection for FA/PPCA}
TODO

\subsection{Model selection for PCA}
TODO

\section{PCA for categorical data}
In this section, we consider extending the factor analysis model to the case where the observed data is categorical rather than real-valued. That is, the data has the form $y_{ij} \in \{1,...,C\}$, where $j=1:R$ is the number of observed response variables. We assume each $y_{ij}$ is generated from a latent variable $\vec{z}_i \in \mathbb{R}^L$, with a Gaussian prior, which is passed through the softmax function as follows:
\begin{align}
p(\vec{z}_i) & = \mathcal{N}(\vec{z}_i|\vec{0},\vec{I}) \\
p(\vec{y}_i|\vec{z}_i,\vec{\theta}) & = \prod\limits_{j=1}^R \mathrm{Cat}(y_{ir}|\mathcal{S}(\vec{W}_r^T\vec{z}_i+\vec{w}_{0r}))
\end{align}
where $\vec{W}_r \in \mathbb{R}^L$ is the factor loading matrix for response $j$, and $\vec{W}_{0r} \in \mathbb{R}^M$ is the offset term for response $r$, and $\vec{\theta}=(\vec{W}_r,\vec{W}_{0r})_{r=1}^R$. (We need an explicit offset term, since clamping one element of $\vec{z}_i$ to 1 can cause problems when computing the posterior covariance.) As in factor analysis, we have defined the prior mean to be $\vec{\mu}_0=\vec{0}$ and the prior covariance $\vec{V}_0=\vec{I}$, since we can capture non-zero mean by changing $\vec{w}_{0j}$ and non-identity covariance by changing $\vec{W}_r$. We will call this categorical PCA. See Chapter 27 TODO for a discussion of related models.

In (Khan et al. 2010), we show that this model outperforms finite mixture models on the task of imputing missing entries in design matrices consisting of real and categorical data. This is useful for analysing social science survey data, which often has missing data and variables of mixed type.

\section{PCA for paired and multi-view data}

\subsection{Supervised PCA (latent factor regression)}

\subsection{Discriminative supervised PCA}

\subsection{Canonical correlation analysis}

\section{Independent Component Analysis (ICA)}
\label{sec:ICA}
Let $\vec{x}_t \in \mathbb{R}^D$ be the observed signal at the sensors at “time” $t$, and $\vec{z}_t \in \mathbb{R}^L$ be the vector of source signals. We assume that
\begin{equation}
\vec{x}_t=\vec{W}\vec{z}_t+\vec{\epsilon}_t
\end{equation}
where $\vec{W}$ is an $D \times L$ matrix, and $\vec{\epsilon}_t \sim \mathcal{N}(\vec{0},\vec{\Psi})$. In this section, we treat each time point as an independent observation, i.e., we do not model temporal correlation (so we could replace the $t$ index with $i$, but we stick with t to be consistent with much of the ICA literature). The goal is to infer the source signals, $p(\vec{z}_t|\vec{x}_t,\vec{\theta})$. In this context, $\vec{W}$ is called the \textbf{mixing matrix}. If $L=D$ (number of sources = number of sensors), it will be a square matrix. Often we will assume the noise level, $|\vec{\Psi}|$, is zero, for simplicity.

So far, the model is identical to factor analysis. However, we will use a different prior for $p(\vec{z}_t)$. In PCA, we assume each source is independent, and has a Gaussian distribution. We will now relax this Gaussian assumption and let the source distributions be any \emph{non-Gaussian} distribution
\begin{equation}
p(\vec{z}_t) =\prod\limits_{j=1}^L p_j(z_{tj})
\end{equation}
Without loss of generality, we can constrain the variance of the source distributions to be \vec{1}, because any other variance can be modelled by scaling the rows of $\vec{W}$ appropriately. The resulting model is known as \textbf{independent component analysis} or \textbf{ICA}.

The reason the Gaussian distribution is disallowed as a source prior in ICA is that it does not permit unique recovery of the sources. This is because the PCA likelihood is invariant to any orthogonal transformation of the sources $\vec{z}_t$ and mixing matrix $\vec{W}$. PCA can recover the best linear subspace in which the signals lie, but cannot uniquely recover the signals themselves.

ICA requires that $\vec{W}$ is square and hence invertible. In the non-square case (e.g., where we have more sources than sensors), we cannot uniquely recover the true signal, but we can compute the posterior $p(\vec{z}_t|\vec{x}_t,\hat{\vec{W}})$, which represents our beliefs about the source. In both cases, we need to estimate Was well as the source distributions $p_j$. We discuss how to do this below.

\subsection{Maximum likelihood estimation}
In this section, we discuss ways to estimate square mixing matrices $\vec{W}$ for the noise-free ICA model. As usual, we will assume that the observations have been centered; hence we can also assume $\vec{z}$ is zero-mean. In addition, we assume the observations have been whitened, which can be done with PCA.

If the data is centered and whitened, we have $\mathbb{E}[\vec{x}\vec{x}^T]=\vec{I}$. But in the noise free case, we also have
\begin{equation}
\mathrm{cov}[\vec{x}]=\mathbb{E}[\vec{x}\vec{x}^T]=\vec{W}\mathbb{E}[\vec{z}\vec{z}^T]\vec{W}^T
\end{equation}
Hence we see that $\vec{W}$ must be orthogonal. This reduces the number of parameters we have to estimate from $D^2$ to $D(D-1)/2$. It will also simplify the math and the algorithms.

Let $\vec{V}=\vec{W}^{-1}$; these are often called the recognition weights, as opposed to $\vec{W}$, which are the generative weights.

Since $\vec{x}=\vec{W}\vec{z}$, we have, from Equation \ref{eqn:Multivariate-transformation},
\begin{align}
p_x(\vec{W}\vec{z}_t) & =p_z(\vec{z}_t)|\mathrm{det}(\vec{W}^{-1})| \nonumber \\
   & = p_z(\vec{V}\vec{x}_t)|\mathrm{det}(\vec{V})|
\end{align}
Hence we can write the log-likelihood, assuming $T$ iid samples, as follows:
\begin{equation*}
\frac{1}{T} \log p(\mathcal{D}|\vec{V})=\log|\mathrm{det}(\vec{V})|+\frac{1}{T}\sum\limits_{j=1}^L \sum\limits_{t=1}^T \log p_j(\vec{v}_j^T\vec{x}_t)
\end{equation*}
where $\vec{v}_j$ is the $j$'th row of $\vec{V}$. Since we are constraining $\vec{V}$ to be orthogonal, the first term is a constant, so we can drop it. We can also replace the average over the data with an expectation operator to get the following objective
\begin{equation}
\mathrm{NLL}(\vec{V})=\sum\limits_{j=1}^L \mathbb{E}[G_j(\vec{z}_j)]
\end{equation}
where $\vec{z}_j=\vec{v}_j^T\vec{x}$ and $G_j(\vec{z}) \triangleq -\log p_j(\vec{z})$. We want to minimize this subject to the constraint that the rows of $\vec{V}$ are orthogonal. We also want them to be unit norm, since this ensures that the variance of the factors is unity (since, with whitened data, $\mathbb{E}[\vec{v}_j^T\vec{x}]=\lVert \vec{v}_j \rVert^2$, which is necessary to fix the scale of the weights. In otherwords, $\vec{V}$ should be an orthonormal matrix.

It is straightforward to derive a gradient descent algorithm to fit this model; however, it is rather slow. One can also derive a faster algorithm that follows the natural gradient; see e.g., (MacKay 2003, ch 34) for details. A popular alternative is to use an approximate Newton method, which we discuss in Section \ref{sec:FastICA}. Another approach is to use EM, which we discuss in Section \ref{sec:ICA-EM}.

\subsection{The FastICA algorithm}
\label{sec:FastICA}

\subsection{Using EM}
\label{sec:ICA-EM}

\subsection{Other estimation principles *}


%---------------------------Aspect Frobenius-----------------------------
\section{Aspect Frobenius\label{s:tri-aspect-Frobenius}}

The aspect Frobenius is the sum of the edge lengths squared divided by the area
and normalized so that a unit equilateral triangle has a value of $1$.
\[
  q = \frac{{\normvec{{L_0}}}^{2} +
            {\normvec{{L_1}}}^{2} +
            {\normvec{{L_2}}}^{2}}{4A\sqrt{3}}
\]

Note that in earlier versions of \verd{}, this metric was
called the triangle aspect ratio.

\trimetrictable{aspect Frobenius}%
{$1$}%                                                Dimension
{$[1,1.3]$}%                                          Acceptable range
{$[1,DBL\_MAX]$}%                                     Normal range
{$[1,DBL\_MAX]$}%                                     Full range
{$1$}%                                                Unit equilateral triangle value
{\cite{pebay:03}}%                                    Reference(s)
{v\_tri\_aspect\_frobenius}%                            Verdict function name


\section{Serre classes}\label{section-serre-classes}
\begin{definition}\label{serre-class}
    A class $\cC$ of abelian groups is a \emph{Serre class} if:
    \begin{enumerate}
	\setcounter{enumi}{0}
	\item $0\in \cC$.
	\item if I have a short exact sequence $0\to A\to B\to C\to 0$, then
	    $A\& C\in \cC$ if and only if $B\in\cC$.
    \end{enumerate}
\end{definition}
Some consequences of this definition: a Serre class is closed under
isomorphisms (easy). A Serre class is closed under subobjects and quotients,
because there is a short exact equence
$$0\to A\hookrightarrow B \to B/A\to 0.$$
Consider an exact sequence $A\to B\to C$ (not necessarily a \emph{short} exact
sequence). If $A,C\in \cC$, then $B\in \cC$ because we have a short exact
sequence:
$$
\xymatrix{
    & & & \coker i\ar[d]\ar[r] & 0\\
    & A\ar[r]^i\ar[d] & B\ar[r]^p\ar[ur] & C & 0\\
    0\ar[r] & \ker p\ar[ur] & & &
}
$$
Some examples are in order.
\begin{example}
    \begin{enumerate}
	\item $\cC = \{0\}$, and $\cC$ the class of all abelian groups.
	\item Let $\cC$ be the class of all torsion abelian groups. We need to
	    check that $\cC$ satisfies the second condition of Definition
	    \ref{}. Consider a short exact sequence
	    $$0\to A\xrightarrow{i} B\xrightarrow{p} C\to 0.$$
	    We need to show that $B$ is torsion if $A$ and $C$ are torsion. To
	    see this, let $b\in B$. Then $p(b)$ is killed by some integer $n$,
	    so there exists $a\in A$ such that $i(a) = nb$. SInce $A$ is
	    torsion, it follows that $b$ is torsion, too.
	\item Let $\cP$ be a set of primes. Define:
	    $$
	    \cC_{\cP} = \{ A : \text{if }p\not\in\cP,\text{ then }p:A\xrightarrow{\simeq} A\text{, i.e., }A \text{ is a } \Z[1/p]\text{-module}\}
	    $$
	    Let $\Z_{(\cP)} = \Z[1/p: p\not\in\cP]\subseteq\QQ$.

	    For instance, if $\cP$ is the set of all primes, then $\cC_\cP$ is
	    the Serre class of all abelian groups. If $\cP$ is the set of all
	    primes other than $\ell$, then $\cC_\cP$ is the Serre class
	    consisting of all $\Z[1/\ell]$-modules. If $\cP = \{\ell\}$, then
	    $\cC_{\{\ell\}} =: \cC_\ell$ is the Serre class of all
	    $\Z_{(\ell)}$-modules. If $\cP = \emptyset$, then $\cC_\emptyset$
	    is all rational vector spaces.
	\item If $\cC$ and $\cC^\prime$ are Serre classes, then so is $\cC\cap
	    \cC^\prime$. For instance, $\cC_\text{tors} \cap \cC_\text{fg}$ is
	    the Serre class $\cC_\text{finite}$. Likewise, $\cC_p \cap
	    \cC_\text{tors}$ is the Serre class of all $p$-torsion abelian
	    groups.
    \end{enumerate}
\end{example}
Here are some straightforward consequences of the definition:
\begin{enumerate}
    \item If $C_\bullet$ is a chain complex, and $C_n\in \cC$, then
	$H_n(C_\bullet)\in\cC$.
    \item Suppose $F_\ast A$ is a filtration on an abelian group. If
	$A\in\cC$, then $\gr_nA\in\cC$ for all $n$. If $F_\ast A$ is finite
	and $\gr_n A\in\cC$ for all $n$, then $A\in\cC$.
    \item Suppose we have a spectral sequence $\{E_r\}$. If $E^2_{s,t}\in \cC$,
	then $E^r_{s,t}\in \cC$ for $r\geq 2$. It follows that if $\{E^r\}$ is
	a right half-plane spectral sequence, then $E^{s+1}_{s,t}\fib
	E^{s+2}_{s,t}\fib\cdots\fib E^\infty_{s,t}\in\cC$.

        Thus, if the spectral sequence comes from a filtered complex (which is
	bounded below, such that for all $n$ there exists an $s$ such that $F_s
	H_n(C) = H_n(C)$, i.e., the homology of the filtration stabilizes),
	then $E^\infty_{s,t} = \gr_s H_{s+t}(C)$. This means that if the
	$E^2_{s,t}\in\cC$ for all $s+t = n$, then $H_n(C)\in\cC$.
\end{enumerate}
To apply this to the Serre spectral sequence, we need an additional axiom for
Definition \ref{serre-class}:
\begin{enumerate}
	\setcounter{enumi}{1}
    \item if $A,B\in\cC$, then so are $A\otimes B$ and $\Tor_1(A,B)$.
\end{enumerate}
All of the examples given above satisfy this additional axiom.
\begin{terminology}
    $f:A\to B$ is said to be a $\cC$-epimorphism if $\coker f\in \cC$, a
    $\cC$-monomorphism if $\ker f\in\cC$, and a $\cC$-isomorphism if it is a
    $\cC$-epimorphism and a $\cC$-monomorphism.
\end{terminology}
\begin{prop}
    Let $\pi:E\to B$ be a fibration and $B$ path connected, such that the fiber
    $F = \pi^{-1}(\ast)$ is path connected. Suppose $\pi_1(B)$ acts trivially
    on $H_\ast(F)$.

    Let $\cC$ be a Serre class satisfiying Axiom 2. Let $s\geq 3$, and assume
    that $H_n(E)\in\cC$ where $1\leq n<s-1$ and $H_t(B)\in \cC$ for $1\leq
    t<s$. Then $H_t(F)\in\cC$ for $1\leq t<s-1$.
\end{prop}
\begin{proof}
    We will do the case $s=3$, for starters.
    We're gonna want to relate the low-dimension homology of these groups.
    What can I say?
    We know that $H_0(E) = \Z$ since it's connected.
    I have $H_1(E)\to H_1(B)$, via $\pi$.
    This is one of the edge homomorphisms, and thus it surjects (no possibility for a differential coming in).
    I now have a map $H_1(F)\to H_1(E)$.
    But I have a possible $d^2:H_2(B)\to H_1(F)$, which is a transgression that gives:
    $$
    H_2(B)\xrightarrow{\partial} H_1(F)\to H_1(E)\to H_1(B)\to 0
    $$

    Let me take a step back and say something general.
    You might be interested in knowing when something in $H_n(F)$ maps to zero in $H_n(E)$.
    I.e., what's the kernel of $H_n(F)\to H_n(E)$.
    The sseq gives an obstruction to being an isomorphism.
    The only way that something can be killed by $H_n(F)\to H_n(E)$ is described by:
    $$
    \ker(H_n(F)\to H_n(E)) = \bigcup\left(\img \text{ of }d^r\text{ hitting }E^r_{0,n}\right)
    $$
    You can also say what the cokernel is:
    it's whatever's left in $E^\infty_{s,t}$ with $s+t = n$.
    These obstruct $H_n(F)\to H_n(E)$ from being surjective.

    In the same way, I can do this for the base.
    If I have a class in $H_n(E)$, that maps to $H_n(B)$, the question is: what's the image?
    Well, the only obstruction is the possibility is that the element in $H_n(B)$ supports a nonzero differential.
    Thus:
    $$
    \img(H_n(E)\xrightarrow{\pi_\ast} H_n(B)) = \bigcap\left(\ker(d^r:E^r_{r,0}\to\cdots)\right)
    $$
    Again, you can think of the sseq as giving obstructions.
    And also, the obstruction to that map being a monomorphism that might occur in lower filtration along the same total degree line.

    Back to our argument.
    We had the low-dimensional exact sequence:
    $$
    H_2(B)\xrightarrow{\partial} H_1(F)\to H_1(E)\to H_1(B)\to 0
    $$
    Here $p=3$, so we have $H_2(B)\in\cC$ and $H_1(E)\in\cC$.
    Thus $H_1(F)\in\cC$.
    That's the only thing to check when $p=3$.

    Let's do one more case of this induction.
    What does this say?
    Now I'll do $p=4$.
    We're interested in knowing if $E^2_{0,3}\in\cC$.
    There are now two possible differentials!
    I have $H_2(F) = E^2_{0,2}\fib E^3_{0,2}$.
    This quotient comes from $d^2:E^2_{2,1}\to E^2_{0,2}$.
    Now, $d^3:E^3_{3,0}\to E^3_{0,2}$ which gives a surjection $E^3_{0,2}\fib E^4_{0,2}\simeq E^\infty_{0,2}\hookrightarrow H_2(E)$.
    Now, our assumptions were that $E^2_{2,1},E^3_{3,0},H_2(E)\in\cC$.
    Thus $E^3_{0,2}\in\cC$ and so $E^2_{0,2} = H_2(F)\in\cC$.
    Ta-da!
\end{proof}
We're close to doing actual calculations, but I have to talk about the multiplicative structure on the Serre sseq first.


\subsection{Stratified sampling and analysis}
Stratified sampling  \cite{bickel1984asymptotic} gives a more refined version of the Monte Carlo method.

\begin{lemma} \label{lem:stratified}
 For any
  nonoverlaping decomposition $G=G_1\cup G_2\cup \cdots \cup G_M$ and
  positive integer $n$, let $n_i=\lceil \lambda(G_i)n\rceil$ be the smallest integer larger than $\lambda(G_i)n$ and $\displaystyle N=\sum_{i=1}^M n_i$.  Let $\theta_{i,j}\in G_i (1\le j\le n_i)$ and
\begin{equation}
g_n=\sum_{i=1}^M \lambda(G_i)g_{n_i}^i \quad \mbox{with}\quad g_{n_i}^i={1 \over n_i}\sum_{j=1}^{n_i} g(\theta_{i,j}).
\end{equation}
It holds that
\begin{equation}
\mathbb{E}_N(\mathbb{E}_G g -g_N)^2=\sum_{i=1}^M {\lambda^2(G_i) \over n_i} \mathbb{E}_{G_i}\big (g -\mathbb{E}_{G_i}g\big )^2
\le {1 \over n} \max_{1\le i\le M}\sup_{\theta, \theta'\in G_i}\big |g(\theta) - g(\theta')\big |^2.
\end{equation}
\end{lemma}

\begin{proof}%[Proof of Lemma \ref{lem:stratified}]
It follows from definition that
\begin{equation}
g(x, \theta) = \sum_{i=1}^M \lambda(G_i) g(x, \theta),\qquad \mathbb{E}_G g =\sum_{i=1}^M\lambda(G_i) \int_{G_i} g\lambda_i(\theta)d\theta = \sum_{i=1}^M \lambda(G_i) \mathbb{E}_{G_i}g.
\end{equation}
Thus, the difference $g - \mathbb{E}_G g$ is a linear combination of $g -\mathbb{E}_{G_i}g$ on each $G_i$ as follows
\begin{equation}
g - \mathbb{E}_G g = \sum_{i=1}^M \lambda(G_i) \big (g -\mathbb{E}_{G_i}g\big ).
\end{equation}
It follows from
\begin{equation}
\mathbb{E}_G g-g_n=\sum_{i=1}^M \lambda(G_i) (\mathbb{E}_{G_i}g -  g^i_{ n_i})
\end{equation}
and \eqref{En} that
\begin{equation}
\mathbb{E}_n(\mathbb{E}_G g -g_n)^2 =\sum_{i,j=1}^M \lambda(G_i) \lambda(G_j) \mathbb{E}_n \big ((\mathbb{E}_{G_i}g -  g^i_{ n_i})(\mathbb{E}_{G_j}g -  g^j_{ n_j})\big )
=\sum_{i,j=1}^M \lambda(G_i) \lambda(G_j) I_{ij}
\end{equation}
with
$$
I_{ij} = \mathbb{E}_n \big ((\mathbb{E}_{G_i}g -  g^i_{ n_i})(\mathbb{E}_{G_j}g -  g^j_{ n_j})\big ).
$$
\iffalse
If $i\neq j$,
\begin{equation}\label{eq:Iij}
\begin{split}
I_{ij}=&\int_{G\times \cdots\times G} (\mathbb{E}_{G_i}g -  g^i_{ n_i})(\mathbb{E}_{G_j}g -  g^j_{ n_j})\lambda(\theta_{i,n_1})\cdots\lambda(\theta_{i,n_i})\lambda(\theta_{j,1})\cdots \lambda(\theta_{j,n_j})d\theta_{i,1}\cdots d\theta_{i,n_i}d\theta_{j,1}\cdots d\theta_{j,n_j}
\\
=&\int_{G\times \cdots\times G} (\mathbb{E}_{G_i}g -  g^i_{ n_i})\lambda(\theta_{i,n_1})\cdots\lambda(\theta_{i,n_i})d\theta_{i,1}\cdots d\theta_{i,n_i}
\\
&+
\int_{G\times \cdots\times G} (\mathbb{E}_{G_i}g -  g^j_{ n_j})\lambda(\theta_{j,1})\cdots \lambda(\theta_{j,n_j})d\theta_{j,1}\cdots d\theta_{j,n_j}
\\
=&\lambda^{n_i}(G_i)\int_{G_i\times \cdots\times G_i} (\mathbb{E}_{G_i}g -  g^i_{ n_i})\lambda_i(\theta_{i,n_1})\cdots\lambda_i(\theta_{i,n_i})d\theta_{i,1}\cdots d\theta_{i,n_i}
\\
&+
\lambda^{n_j}(G_j)\int_{G\times \cdots\times G} (\mathbb{E}_{G_i}g -  g^j_{ n_j})\lambda_j(\theta_{j,1})\cdots \lambda_j(\theta_{j,n_j})d\theta_{j,1}\cdots d\theta_{j,n_j}
\\
=&0.
\end{split}
\end{equation}
\fi
By Lemma \ref{MC},
\begin{equation}
I_{ij} =  \mathbb{E}_n \big ((\mathbb{E}_{G_i}g -  g^i_{ n_i})^2\big )\delta_{ij}= {1\over n_i} \mathbb{E} \big ((\mathbb{E}_{G_i}g -  g)^2\big )\delta_{ij}.
\end{equation}
Thus,
\begin{equation}
\mathbb{E}_n(\mathbb{E}_G g -g_n)^2 =\sum_{i=1}^M {\lambda^2(G_i) \over n_i} \mathbb{E}_{G_i}\big (g -\mathbb{E}_{G_i}g\big )^2,
\end{equation}
which completes the proof.
\end{proof}

Lemma \ref{MC} and Lemma \ref{lem:stratified} represent two simple identities and subsequent inequalities that can be verified by a direct calculation. Actually Lemma \ref{MC} is a special case of Lemma  \ref{lem:stratified} with $M=1$. Lemma \ref{MC} and Lemma \ref{lem:stratified} are the basis of Monte-Carlo sampling and  stratified sampling in statistics.

\begin{lemma}\label{lem:stratifiedapprox}
\textup{[Stratified Sampling]}
	For $u(x)$ in \eqref{uint}
	with positive $\rho(\theta)\in L^1(G)$, given any positive integers $n$ and $M\le n$, for  any nonoverlaping decomposition $G=G_1\cup G_2\cup \cdots \cup G_M$, there exists$\{\theta_i^\ast\}_{i=1}^N$ with $n\le N \le 2n$ such that
	\begin{equation}
	\|u - u_N\|_{L^2(\Omega)} \leq N^{-1/2}\|\rho\|_{L^1(G)}\max_{1\le j\le M}\sup_{\theta_{j},\theta_{j}'\in G_j} \| g(x,\theta_j) - g(x,\theta_j')\|_{L^2(\Omega)}
	\end{equation}
	where
	$$
	u_N(x)= {2\|\rho\|_{L^1(G)}\over N}\sum_{i=1}^N\beta_i g(x,\theta_i^\ast)
	$$
	and $\beta_i\in [0,1]$.
	\end{lemma}

\begin{proof}
Let $n_j=\lceil \lambda(G_j)n\rceil$ and $\theta_{i,j} \in G_j(1\leq i\leq n_j)$. Define $\displaystyle N=\sum_{j=1}^M n_j$ and
$$
u_N(x)=\|\rho\|_{L^1(G)}\sum_{i=1}^M \lambda(G_i)g_{n_i}^i \quad \mbox{with}\quad g_{n_i}^i={1 \over n_i}\sum_{j=1}^{n_i} g(\theta_{i,j}).
$$
Since
$
\displaystyle u(x)=\|\rho\|_{L^1(G)}\sum_{i=1}^M \lambda(G_i)\mathbb{E}_{G_i} g,
$
by Lemma \ref{MC},
\begin{equation}
\begin{split}
\mathbb{E}_N\|u- u_N \|_{L^2(\Omega)}^2=& \|\rho\|_{L^1(G)}^2
 \sum_{j=1}^{M}{\lambda^2(G_j) \over n_j}\mathbb{E}_{G_j}\|\mathbb{E}_{G_j} g -  g\|^2_{L^2(\Omega)}
\\
\le &\|\rho\|_{L^1(G)}^2 \sum_{j=1}^{M} {\lambda^2(G_j)\over n_j}\sup_{\theta_{j},\theta_{j}'\in G_j} \| g(x,\theta_j) - g(x,\theta_j')\|^2_{L^2(\Omega)}.
\end{split}
\end{equation}
Since ${\lambda(G_j)\over n_j}\le {1\over n}$ and $\displaystyle \sum_{j=1}^M \lambda(G_j)=1$,
\begin{equation}
\mathbb{E}_N\|u - u_N \|_{L^2(\Omega)}^2\leq n^{-1}\|\rho\|_{L^1(G)}^2\max_{1\le j\le M}\sup_{\theta_{j},\theta_{j}'\in G_j} \| g(x,\theta_j) - g(x,\theta_j')\|^2_{L^2(\Omega)}.
\end{equation}
There exist $\{\theta_{i,j}^\ast\}$ such that $\theta_{i,j}^\ast\in G_i$ and
\begin{equation}
\| u-u_N \|_{L^2(\Omega)}^2\leq n^{-1} \|\rho\|_{L^1(G)}^2\max_{1\le j\le M}\sup_{\theta_{j},\theta_{j}'\in G_j} \| g(x,\theta_j) - g(x,\theta_j')\|^2_{L^2(\Omega)}.
\end{equation}
Note that $n\le N\le n+M\le 2n$,
$$
u_N(x) =  {2\|\rho\|_{L^1(G)}\over N}\sum_{j=1}^{M}\frac{N\lambda(G_j)}{2n_j}\sum_{i=1}^{n_j} g(x,\theta_{i,j}^\ast)
 =  {2\|\rho\|_{L^1(G)}\over N}\sum_{j=1}^{M}\beta_{i,j}\sum_{i=1}^{n_j} g(x,\theta_{i,j}^\ast)
$$
with
\begin{equation}
\beta_{i,j}= \frac{N\lambda(G_j)}{2n_j}\le \frac{2\lambda(G_j)n}{2\lambda(G_j)n}\le 1,
\end{equation}
which completes the proof.
\end{proof}


\subsection{Cross-Validation Criterion}
There are several popular information criteria for model selection: for example, Akaike information criterion (\textbf{AIC}), corrected AIC (\textbf{AIC\textsuperscript{c}}), Schwarz Bayesian information criterion (\textbf{SIC}), Hannan-Quinn (\textbf{HQ}) and Mallows' C\textsubscript{p} (\textbf{C\textsubscript{p}}). Most criteria have two components in their formulas: the first part measures model fit while the second penalizes overfitting. The quantity measuring in-sample fit are the same for most criteria, but they differ in the degree of penalization. For instance, AIC penalizes each additional parameter by $2$ while SIC penalizes overfitting by the logarithm of sample size, so SIC tends to select a more parsimonious model than AIC if the sample size is large.

For the forecasting analysis, what we care about is the test error rate assessing the model predictive ability, not the training error rate produced in the model estimation stage, so selecting a information criterion which gives a good estimate of the expected test error rate is crucial. Cross-validation is such a criterion. Specifically, we focus on the use of leave-one-out cross-validation for this paper, though other CV variants, such as K--fold cross-validation, may be considered. Cross-validation is computationally simple for one-step ahead predictive model selection and is shown robust to conditional heteroscedasticity in the econometrics and statistics literature. For forecast combination, researchers have applied CV to the quadratic programming based model averaging analysis, but its setting does not include structural break.

The sample leave-one-out cross-validation criterion can be computed by the following procedure:
\begin{equation} \label{cv}
	CV_T(k) = \frac{1}{T}\sum_{t=1}^{T}\tilde{e}_{t}(k)^{2}
\end{equation}
where $\tilde{e}_{t}(k) = y_t - \tilde{\beta}_{-t}(k)'x_t(k)$ are the residuals from the regression with the $t^{\mathrm{th}}$ observation dropped and $\tilde{\beta}_{-t}(k) = (\sum_{i\not= t}x_i(k) x_i(k)')^{-1}(\sum_{i\not= t}x_i(k) y_i)$ is the associated vector of parameter estimates. Intuitively, this procedure is trying to estimate the expected test error rate based on the model training data. Though equation~\ref{cv} implies that we need to run regression $T$ times for given sample size $T$, luckily, for linear models, we can calculate sample CV value by running regression only once. Formally, the leave-one-out cross validation residuals can be computed from the full sample least squares residuals, $\tilde{e_{t}} = \frac{\hat{e_{t}}}{1 - h_{t}}$, where $h_{t} = x_{t}'(X_{t}'X_{t})^{-1}x_{t}$ is the leverage associated with observation $t$, $\hat{e_{t}}$ is the full sample least squares residual and $\tilde{e_{t}}$ is the cross-validation residual. So we can rewrite equation~\ref{cv} as
\begin{equation}
    CV_T(k) = \frac{1}{T}\sum_{t=1}^{T}\left( \frac{\hat{e_{t}}(k)}{1 - h_{t}} \right)^{2}
\end{equation}
In the next section we are going to show how model averaging weights are derived from the cross-validation criterion.


\subsection{Alpha-beta pruning}

\subsubsection{Pruning}

Alpha-beta pruning

We can use pruning to reduce the number of nodes we search.

Alpha-beta pruning is a method for pruning. For each node we maintain two values:

\begin{itemize}
\item Alpha. The lower bound on the maximum value of the children.
\item Beta. The upper bound on the minimum value of the children.
\end{itemize}

Consider a player node with a score of \(2\), which has a neighbour. This neighbour has a

We can know for sure we don't need to explore some paths. for example consider the mini player. there is a node we know the minimax value of a node is 2, and there is a neighbour with one child with a value of 3, then

like DFS, but keep track of:

\begin{itemize}
\item alpha (largest value for max across viewed children)
\item initialise to  \(+/-\infty \)
\item initialise to \(\infty \)
\end{itemize}

Propagate \(\alpha \) and \(\beta \) down during search. prune where \(\alpha \ge \beta \)

Update \(\alpha \) and \(\beta \) by propagating upwards.

Only update alpha on max nodes, beta on min node

Ordering matters. if it's worst, then no pruning. want an ordering with lots of pruning
	p can:
	p + do shallow nodes
	p + order node so best are done first
	p + domain knowledge heuristics (chess: capture, threaten, forward, backwards)

In practice can get time down to \(O(b^\frac{m}{2})\).

Use heuristic. deep blue uses \(6000\)


\section{Integrating Factor Method}
All 1st order linear differential equations have the form
\begin{equation*}
	a_1(x)\dd{y}{x} + a_0(x)y = b_1(x)
\end{equation*}
which can be rewritten as
\begin{equation*}
	\dd{y}{x} + a(x)y = b(x)
\end{equation*}

\noindent
This equation isn't always separable, and we can't just integrate both sides unless $a_1(x)y$ = 0.\\

\noindent
If $a_0(x) = a_1'(x)$, then we could rewrite the equation and solve by doing the product rule in reverse.
\begin{equation*}
	\left(a_1(x)y\right)' = b_1(x) \implies y = \frac{\int{b_1(x) \mathrm{d}x}}{a_1(x)}
\end{equation*}

\noindent
It's possible to rearrange into this form by multiplying the equation by some function. Specifically, what were looking for is a function $\mu(x)$ such that
\begin{equation*}
	\mu(x)\dd{y}{x} + \mu(x)a(x)y = \mu(x)b(x) \text{ and } \mu'(x) = \mu(x)a(x)
\end{equation*}
This equation involving $\mu(x)$ is one that we know how to solve because it's separable.\footnote{Although we are taking an indefinite integral to find $\mu(x)$, we do not have a $+ C$ term.}
\begin{equation*}
	\mu'(x) = \mu(x)a(x) \implies \mu(x) = e^{\int{a(x) \mathrm{d}x}}
\end{equation*}
Substituting the solution for $\mu(x)$ back,
\begin{equation*}
	e^{\int{a(x) \mathrm{d}x}}\dd{y}{x} + e^{\int{a(x) \mathrm{d}x}}a(x)y = e^{\int{a(x) \mathrm{d}x}}b(x) \implies e^{\int{a(x) \mathrm{d}x}}\dd{y}{x} + \mu'(x)y = e^{\int{a(x) \mathrm{d}x}}b(x)
\end{equation*}
Applying the product rule in reverse,
\begin{equation*}
	y = \frac{\int{\mu(x)} b(x) \mathrm{d}x}{\mu(x)} \text{, } \mu(x) = e^{\int{a(x) \mathrm{d}x}}
\end{equation*}

%\noindent
%This method gives us a way to solve all first-order linear ODE's.

\ifodd\includeFirstOrderLinearODEExamples\input{./1stOrderLinearODE/integratingFactor/integratingFactor_example.tex}\fi


\documentclass[12pt]{article}

\usepackage{hyperref}
\usepackage{graphicx}

\title{Analysis of Common Sorting Algorithms}
\author{Utkrisht Dhankar}

\begin{document}
	\maketitle
	\newpage

	\section{Abstract}
	There are many sorting algorithms available today, and sorting is one of the most used operations in modern computing. Therefore, it is worthwhile to examine common sorting algorithms, and see their strengths and weeknesses. Here, I'll be analysing 6 common sorting algorithms: Bubble Sort, Insertion Sort, Selection Sort, Rank Sort, Merge Sort, and Quick Sort.

	\section{Theoretical Analysis}
		\subsection{Bubble Sort}
		Bubble sort is theoretically the slowest algorithm in the list of algorithms analysed here. The basic idea is to go through the list, swapping elements every time two adjacent out of order elements are seen. This algorithm is always $ \mathcal{O} (n^2) $, and the constants associated are generally high. It tends to be very slow as in each iteration there are very few swap operations but many comparisions. The pseudocode for Bubble Sort is as follows\footnote{from Bubble Sort on Wikipedia \url{https://en.wikipedia.org/wiki/Bubble_sort}}.

			\begin{center}
\begin{verbatim}
procedure bubbleSort( A : list of sortable items )
   n = length(A)
   repeat
     swapped = false
     for i = 1 to n-1 inclusive do
       /* if this pair is out of order */
       if A[i-1] > A[i] then
         /* swap them and remember something changed */
         swap( A[i-1], A[i] )
         swapped = true
       end if
     end for
   until not swapped
end procedure
\end{verbatim}
			\end{center}

		\subsection{Insertion Sort}
		Insertion sort is also theoretically quite a slow algorithm, but it has some advantages, especially when sorting arrays that are almost sorted already. The idea with Insertion Sort is to keep a list of sorted elements in place at the start of the array being sorted, and insert the first unsorted element into the sorted array in its correct place. In such cases, Insertion Sort's running time is close to or even equal to $ \mathcal{O} (n) $ as it does not modify the array on each iteration if it finds an already sorted element in place, and so it is quite useful in such cases. It's worst case time complexity is $ \mathcal{O} (n^2) $, just like Bubble Sort, but the constants associated are lower. The pseudocode is as follows \footnote{Cormen, Thomas H.; Leiserson, Charles E.; Rivest, Ronald L.; Stein, Clifford (2001) [1990]. \textit{Introduction to Algorithms (2nd ed.)}. MIT Press and McGraw-Hill. Section 2.1: Insertion sort, pp. 15 to 21. }:

		\begin{verbatim}
for i = 1 to length(A)
    x = A[i]
    j = i - 1
    while j >= 0 and A[j] > x
        A[j+1] = A[j]
        j = j - 1
    end while
    A[j+1] = x[3]
 end for
		\end{verbatim}

		\subsection{Selection Sort}
		Selection Sort is a similar sorting algorithm to Insertion Sort, in that we maintain a list of sorted elements, but this time, we select the smallest element from the unsorted list and append it to the sorted list each iteration. This does not have the advantage that Insertion Sort has when it comes to close to sorted arrays however, as finding the minimum element in the unsorted array always takes a lot of time. It's running time is always $ \mathcal{O} (n^2) $. The pseudocode for Selection Sort is as follows \footnote{Modified from \url{http://freefeast.info/general-it-articles/selection-sort-pseudo-code-of-selection-sort-selection-sort-in-data-structure/}} :

		\begin{verbatim}
SELECTION-SORT(A, n)
    for j = 1 to n-1
         smallest = j
       	 for i = j + 1 to n
              if A[ i ] < A[ smallest ]
                  smallest = i
          	  Exchange A[ j ] and A[ smallest ]
		\end{verbatim}

		\subsection{Rank Sort}
		Rank Sort maintains a seperate list of the rank of the elements, where the rank is the number of elements smaller than or equal to an element. This creates some ambiguity when two numbers are the same, but this can be avoided by creating a loop that only looks forward for elements, and not backwards. Rank Sort is always $ \mathcal{O} (n^2) $, and also requires $ \mathcal{O} (n) $ extra space to store the auxiliary list. The Pseudocode for Rank Sort is as follows:

		\begin{verbatim}
RANK-SORT(A, n)
    ranks = zeros(n)
    for (i = 1 to n - 1)
        for (j = i to n - 1)
            if (A[i] > A[j])
                rank[i]++
            else
                rank[j]++

    copy = A
    for (i = 1 to n-1)
        A[rank[i]] = copy[i]
		\end{verbatim}

		\subsection{Merge Sort}
		Merge Sort is the first of the algorithms I'll be analysing here which uses the divide and conquer methodology to sort the array. The idea is to divide the array to be sorted into two halves, sort the left and right halves recursively, and then merge the two sorted arrays using a standard merging procedure. This algorithm always takes $ \mathcal{O} (n log(n)) $ time, but it requires some extra space to store the copied subarrays. It is in general much faster than the $ \mathcal{O} (n^2) $ algorithms. The pseudocode for this algorithm is as follows \footnote{Modified from \url{https://en.wikipedia.org/wiki/Merge_sort}}

		\begin{verbatim}
function merge_sort(list m)
    if length of m ≤ 1 then
        return m

    var left := empty list
    var right := empty list
    for each x with index i in m do
        if i < (length of m)/2 then
            add x to left
        else
            add x to right

    left := merge_sort(left)
    right := merge_sort(right)

    return merge(left, right)
		\end{verbatim}

		\subsection{Quick Sort}
		Quick Sort is a very fast sorting algorithm that uses the divide and conquer methodology to achieve $ \mathcal{O} (n log(n)) $ average case running time. The reason for it being quick is that the constants associated with it are much smaller than those associated with Merge Sort. This is because the merge operation, a heavy operation that slows Merge Sort down, is trivial in Quick Sort. The idea in Quick Sort is to select a pivot element, put all elements larger than it to its right, and smaller than it to its left, and then recursively sort the two subarrays. The pseudocode for Quick Sort is as follows \footnote{Cormen, Thomas H.; Leiserson, Charles E.; Rivest, Ronald L.; Stein, Clifford (2009) [1990]. "Quicksort". \textit{Introduction to Algorithms (3rd ed.)}. MIT Press and McGraw-Hill. pp. 170 to 190} :

		\begin{verbatim}
algorithm quicksort(A, lo, hi) is
    if lo < hi then
        p := partition(A, lo, hi)
        quicksort(A, lo, p – 1)
        quicksort(A, p + 1, hi)

algorithm partition(A, lo, hi) is
    pivot := A[hi]
    i := lo - 1
    for j := lo to hi - 1 do
        if A[j] ≤ pivot then
            i := i + 1
            swap A[i] with A[j]
    swap A[i+1] with A[hi]
    return i + 1
		\end{verbatim}

	\section{Observations}
	I implemented all the six algorithms and ran them on datasets ranging as high as 5 million integers, with random, ascending, and descending order elements. Here are the results, plotted in Google Sheets \footnote{For the raw data, see \url{https://docs.google.com/spreadsheets/d/1SNPT1LTcow5kVDeKIXEFPN6cPsf4cfh08tgTT1_FfjA/edit?usp=sharing}}.

	\begin{center}
		\includegraphics[width=120mm]{"Random n squared".png}

		\textbf{Figure 1.} The $ \mathcal{O} (n^2) $ on a random input

		\includegraphics[width=120mm]{"Random n log n".png}

		\textbf{Figure 2.} The $ \mathcal{O} (n log (n)) $ on a random input

		\includegraphics[width=120mm]{"Asc n squared".png}

		\textbf{Figure 3.} The $ \mathcal{O} (n^2) $ on a ascending input

		\includegraphics[width=120mm]{"Asc n log n".png}

		\textbf{Figure 4.} The $ \mathcal{O} (n log (n)) $ on a ascending input

		\includegraphics[width=120mm]{"Desc n squared".png}

		\textbf{Figure 5.} The $ \mathcal{O} (n^2) $ on a descending input

		\includegraphics[width=120mm]{"Desc n log n".png}

		\textbf{Figure 6.} The $ \mathcal{O} (n log (n)) $ on a descending input
	\end{center}

	Some things of note: (a) The graphs have been plotted on $log$ vs $log$ scale, as any other method would have resulted in very hard to read graphs. (b) the results which took longer than 100 seconds have been removed. However, as in the things tend to form straight lines, it is easy to project forward, and (c) Google Sheets does not allow labels for numbers smaller in magnitude than 0.01, so some of the labels say 0 when they should say 0.001, 0.0001 etc.

	When running the random input on the $ \mathcal{O} (n^2) $ algorithms, it is clear that Insertion sort is the quickest. Slightly slower than it on random input is Selection Sort. After a substantial gap is Rank Sort, followed by a somewhat smaller gap to Bubble Sort, the slowest of the four algorithms. The reasons for this were analysed in the \textit{Theoretical Analysis} section.

	Of the $ \mathcal{O} (n log(n)) $ algorithms, on random input merge sort was clearly the quickest by a fairly large margin. However, both of these algorithms are much faster than the $ \mathcal{O} (n^2) $ algorithms on random input, as they even managed to completely sort 5000000 integers very quickly.

	When sorting ascending input (already sorted), Insertion Sort's advantage over Selection Sort is clear. It acts like an $ \mathcal{O} (n) $ algorithm rather than an $ \mathcal{O} (n^2) $ algorithm. Interestingly, Bubble Sort was the quickest of the $ \mathcal{O} (n^2) $ algorithms, as it's comparisons are very simple and it doesn't do any other operations.

	In the $ \mathcal{O} (n log(n)) $ algorithms, Merge Sort acts like an $ \mathcal{O} (n log(n)) $ algorithm while Quick Sort is much slower, and acts like an $ \mathcal{O} (n^2) $ algorithm. The reason for Quick Sort's slowness is that every time it divides the array into sub-arrays where one sub-array only has 1 element, and the rest go into the other sub-array.

	On Descending Inputs, in the $ \mathcal{O} (n^2) $ algorithms, Selection Sort is clearly the quickest. The reason for this is that in Insertion sort, on Descending input, it has to swap many elements to get the element being inserted into it's correct position, where in Selection Sort this swap operation is simple, and only the comparisons take long. Rank Sort is interestingly enough similarly quick as Insertion Sort, and Bubble Sort is the slowest.

	In the $ \mathcal{O} (n log (n)) $ algorithms, Merge Sort is again much quicker than Quick Sort, for the same reasons as in the case of the ascending (already sorted) inputs.

	\section{Conclusions}

	Many of these sorting algorithms have practical applications. Quick Sort is the best general purpose sorting algorithm for random elements, as it is quickest by far. Therefore, many languages bundle Quick Sort as their default sorting algorithm. Merge Sort quite useful in cases where given arrays have a high likelihood of being both nearly sorted or being in random order as on average it is quicker when this is the case, because of Quick Sort's slowness in the already sorted cases. Both of these algorithms are also highly parallelisable, as they divide their tasks into sub tasks, which makes for a straightforward multithreading application.

	Insertion Sort is very useful for simple, quick sorting of nearly sorted arrays, and is also useful when previous order needs to be maintained between equal elements (it is a stable sort, where the $ \mathcal{O} (n log(n)) $ algorithms are unstable. Rank Sort is useful when the ranking data is needed after sorting. Bubble Sort can be useful when sorting almost sorted data exclusively, but can take up a lot of CPU time if a random array is given, and thus is usually avoided.

	\section*{Miscellaneous References}
		\begin{enumerate}
			\item All time and space complexities taken from class notes on given algorithms.
			\item All execution of the algorithms was done on a MacBook Pro (Retina, 15-inch, Mid 2014).
			\item Complete implementations of the algorithms and the data structures involved can be found in the attached source files.
			\item The input data used was generated by a script. Both the data and the script are available at \url{https://github.com/sahilarora535/cs202-assignment02}.
		\end{enumerate}

\end{document}


\subsection{SVD as a geometrical interpretation}

And what was the advantage of the factorization then? In simple terms,
decomposing the function behind matrix $A$, as a sequence of three
simpler (easier to understand) transformations. Here the geometric
interpretation helps to complete the picture, as orthogonal (change of
basis) matrices do represent rigid transformation in space, that is,
they do not alter the lengths of vectors (hence, preserve
shapes). Strictly speaking, orthogonal matrices can be decomposed as a
rotation and a reflection; but for geometric intuition, is often
desirable to think in the rotation part only. \\

On the other hand, diagonal matrices are the simplest transformation
possible, they do not change the basis but just expand or contract the
coordinates along the axis given by the basis. Again, if we consider
the generic case of diagonal matrices, a negative element $D_{ii}$
would additional provoke a reflection in the axis $i$; but since the
SVD decomposition produces only positive elements on the diagonal, we
ignore this case and just think in terms of contractions or
expansions along the axes. \\

Armed with this geometrical insight, we can enhance our understanding
of the action of $A$ through the SVD decomposition, by associating to
the simpler operations the corresponding geometrical transformations.
The geometric visualization usually requires a couple of
simplifications: first of all, the dimensions of domain and codomain
must be reduced; as it is easier to visualize things in \R{2} or
\R{3}, than in an arbitrary \R{n}. Given two dimensions fit well in a
screen, let us pick $\R{n} = \R{m} = \R{2}$. \\

Secondly, we need to focus our attention in an specific set of
points (as visualizing the effect of a linear transformation against
``all'' vectors in space, even for \R{2}, is a quite abstract and
complex task). The usual procedure is to pick the vectors in the
unitary sphere in \R{2} (which contains in particular the columns of
$V$, as they are unit orthogonal vectors). \\

Let us proceed now: let the matrix $A$ be of
dimensions $2  \times 2$, the matrices $V$ and $U$ be formed
by unit column vectors $\{\vec{v_1},\vec{v_2}\}$ and $\{\vec{u_1},\vec{u_2}\}$,
respectively; and let matrix $\Sigma$ be $diag(\sigma_1,\sigma_2)$
such that $\sigma_1 > \sigma_2 > 0$. We will additionally assume that
$\sigma_1 > 1$ and $\sigma_2 < 1$, in order to allow them represent an
expansion and contraction, respectively. The previously described steps of
the SVD factorization, can be now augmented with the corresponding
geometrical transformations: \\

\begin{enumerate}
\item Start with unit sphere in \R{2}, with the unit vectors \vec{v_1}
  and \vec{v_2} living inside of it.
\item Action of \trans{V}: Rotate the space such that \vec{v_1} and
  \vec{v_2} become the new orthogonal basis (this transformation
  leaves the shape of the sphere intact).
\item Action of $\Sigma$: Once rotated, the unit sphere is expanded
  in the direction of \vec{v_1} (per $\sigma_1$), and contracted in the
  direction of \vec{v_2} (per $\sigma_2$).
\item Action of $U$: Once reshaped, the resulting ellipse is taken
  from the basis $\{u_1,u_2\}$ back to the canonical basis (this
  transformation changes the orientation of the ellipse, given that
  is not a symmetric figure; but still it preserves it shape).
\end{enumerate}
\hfill

These steps can be summarized in the following figure \footnote{Which
  was taken from a \href{http://math.stackexchange.com/questions/243811/visualization-of-singular-value-decomposition-of-a-symmetric-matrix}{MathStackExchange phorum post}, we
  could not locate back the original source though.}: \\

\begin{figure}[h]
  \centering
  \includegraphics[width=15cm]{svd-geo-diag}
  \caption{Geometrical interpretation of $A = U \Sigma \trans{V}$,
    over the unit sphere.}
  \label{fig:svd-geo-diag}
\end{figure}
\hfill