In the previous chapter, the grey values were modelled using a compound Poisson distribution to reflect the science of x-rays. This model has a linear relationship between the variance and the expectation. As a result, it would be possible to predict the variance of the grey value, given the grey value. This opens up new ways to predict the uncertainty of each pixel in an x-ray projection.

In this chapter, generalised linear models \citep{nelder1972generalized, nelder1972generalized_2, mccullagh1984generalized} with different link functions and polynomial features were selected using forward stepwise selection. These selected models were compared using cross-validation to find the best model and verified using residual analysis.

\section{Generalised Linear Models}

The sample variance-mean data were obtained from the replicate projections. Let $x_{i,j}$ be the grey value of the $i$th pixel, ignoring any spatial information, from the $j$th replicate projection for $i=1,2,3,\ldots,N$ and $j=1,2,3,\ldots,m$. The sample mean and sample variance grey value for the $i$th pixel are
\begin{equation}
    x_{i}=\frac{1}{m}\sum_{j=1}^m x_{i,j}
\end{equation}
and
\begin{equation}
    y_{i} =
    \frac{1}{m-1}
    \sum_{j=1}^m
        \left(
            x_{i,j} - {x}_{i}
        \right)^2
\end{equation}
respectively. The symbol $y$ should not be confused with the latent variable in the previous chapter. Only pixels in the region of interest (ROI) were considered here, that is, pixels which represent the test sample. The ROI was created by manually segmenting the test sample from the projection.

The aim is to model and predict the grey value variance of a pixel given its grey value by using a model fitted onto the sample variance-mean data. The variance-mean data is denoted as $(x_1,y_1),(x_2,y_2),\ldots,(x_N,y_N)$ where $N$ is the area of the ROI or the size of the dataset. Let $Y(x)$ be a random variable and the sample variance given a grey value mean $x$. It was assumed that the standard error from estimating the mean was negligible so that the uncertainty is captured by the random variable $Y$. It was assumed that for a given pixel, the grey values are Normal and i.i.d. Let $\sigma^2(x)$ be the variance given a grey value $x$, then it can be shown that
\begin{equation}
\dfrac{(m-1)Y(x)}{\sigma^2(x)}\sim\chi^2_{m-1}
\end{equation}
which results in
\begin{equation}
Y(x)\sim\gammaDist\left(\alpha,\dfrac{\alpha}{\sigma^2(x)}\right)
\end{equation}
where $\alpha=(m-1)/2$ is the shape parameter. The expectation and variance are
\begin{equation}
\expectation\left[Y(x)\right]=\sigma^2(x)
\end{equation}
and
\begin{equation}
\variance\left[Y(x)\right]=\left(\dfrac{\sigma^2(x)}{\alpha}\right)^2
\end{equation}
respectively.

This framework allows the use of generalised linear models (GLM) \citep{nelder1972generalized,nelder1972generalized_2, mccullagh1984generalized}. In the gamma distribution case, a GLM can be used to model
\begin{equation}
Y(x)\sim\gammaDist\left(\alpha,\dfrac{\alpha}{g^{-1}(\eta(x))}\right)
\end{equation}
where $g(y)$ is the link function and $\eta(x)$ is a linear function called the systematic component. It should be noted that
\begin{equation}
  \expectation\left[Y(x)\right]=g^{-1}(\eta(x))
\end{equation}
which shows how the link function and systematic component work together. Examples of link functions are the identity link
\begin{equation}
g(y)=g^{-1}(y)=y
\end{equation}
and, for the gamma distribution case, the canonical link
\begin{equation}
g(y)=g^{-1}(y)=1/y \ .
\end{equation}

An example of a systematic component are polynomial features $\eta(x)=\sum_{r=0}^{p}\beta_r x^{r}$ so that when used with the identity link, for example, $\expectation\left[Y(x)\right] = \sum_{r=0}^{p}\beta_r x^{r}$. Iterative reweighted least squares \citep{friedman2001elements} can be used to estimate the parameters $\beta_0, \beta_1, \ldots, \beta_{p}$ given data for the model to fit onto.

Once the parameters have been estimated, prediction of the variance given a grey value $x$ is done by using $\widehat{y}(x) = g^{-1}(\widehat{\eta}(x))$ where $\widehat{\eta}(x)$ is the systematic component using the estimated parameters.

\section{Model Selection}

This section describes how forward stepwise selection \citep{efroymson1960multiple, friedman2001elements} was used to select which polynomial features to use in the systematic component.

In summary, forward stepwise selection fits a basic model to the data initially. A feature is added to make the model more complicated at each step to improve the fit onto the data. This is continued until the model cannot be improved subject to overfitting. The Akaike information criterion (AIC) \citep{akaike1974new, friedman2001elements} and the Bayesian information criterion (BIC) \citep{schwarz1978estimating, friedman2001elements} are criteria which can be used to assess the fit of the model at each step without overfitting to the data too much.

The AIC and BIC are given as
\begin{equation}
\AIC = 2k-2\ln L
\end{equation}
and
\begin{equation}
\BIC = k\ln N - 2\ln L
\end{equation}
respectively where $k$ is the number of parameters in the systematic component and $\ln L$ is the log-likelihood of the GLM. The model with the lowest AIC or BIC is preferred. GLM aims to maximise the log-likelihood but the additional terms in the criteria penalise models with too many terms. The log-likelihood is given as
\begin{equation}
  \ln L = \sum_{i=1}^N \left[
    \alpha\ln\alpha
    -\ln\Gamma(\alpha)
    -\alpha\ln \widehat{y}_i
    +(\alpha-1)\ln y_i
    -\frac{\alpha y_i}{\widehat{y}_i}
  \right]
  \ .
\end{equation}
where $\widehat{y}_i=\widehat{y}(x_i)$. $\alpha=(m-1)/2$ was assumed to be known so does not need to be estimated.

The procedure is as follows. A criterion and a link function were chosen beforehand. In the initial step, a GLM with systematic component $\eta(x)=\beta_0$ was fitted and the criterion was recorded. In the next step, a polynomial feature with one order higher was added to the systematic component $\eta(x)=\beta_0+\beta_1 x$, fitted and the criterion recorded. Also, a polynomial feature with one order lower was added $\beta(x)=\beta_{-1}x^{-1}+\beta_0$ and fitted separately with the criterion recorded. The model which decreased the criterion the most was accepted. Adding higher and lower order polynomials to the systematic component was repeated, for example, after accepting $\eta(x)=\beta_0+\beta_1 x$, the following systematic components $\eta(x)=\beta_0+\beta_1 x+\beta_2x^2$ and $\eta(x)=\beta_{-1}x^{-1}+\beta_0+\beta_1 x$ were fitted and assessed. The fitted parameters may change when adding more terms. This is repeated until the criterion cannot be decreased and the procedure is left with the final model.

Forward stepwise selection was conducted on the datasets \texttt{AbsNoFilter} and \texttt{AbsFilter}. The procedure was repeated 10 times by using a random permutation, with replacement, of the replicate projections to obtain a different sample variance-mean data which introduced some variation to the data \citep{efron1979bootstrap}. The procedure was also repeated using various shading corrections to investigate the effects of shading correction on the variance-mean relationship.

\begin{sidewaystable}
\centering
\begin{tabular}{ll|ccc}
\multicolumn{2}{c|}{Identity Link}& null & bw & linear \\ \hline
\multirow{3}{*}{AIC} & order     & \inputNumber{../figures/varmean/GlmSelectAicAbsNoFilterDeg30null_identityorder.txt}     & \inputNumber{../figures/varmean/GlmSelectAicAbsNoFilterDeg30bw_identityorder.txt}     & \inputNumber{../figures/varmean/GlmSelectAicAbsNoFilterDeg30linear_identityorder.txt}     \\
                     & votes     & \inputNumber{../figures/varmean/GlmSelectAicAbsNoFilterDeg30null_identityvote.txt}      & \inputNumber{../figures/varmean/GlmSelectAicAbsNoFilterDeg30bw_identityvote.txt}      & \inputNumber{../figures/varmean/GlmSelectAicAbsNoFilterDeg30linear_identityvote.txt}      \\
                     & criterion & \inputNumber{../figures/varmean/GlmSelectAicAbsNoFilterDeg30null_identitycriterion.txt} & \inputNumber{../figures/varmean/GlmSelectAicAbsNoFilterDeg30bw_identitycriterion.txt} & \inputNumber{../figures/varmean/GlmSelectAicAbsNoFilterDeg30linear_identitycriterion.txt} \\ \hline
\multirow{3}{*}{BIC} & order     & \inputNumber{../figures/varmean/GlmSelectBicAbsNoFilterDeg30null_identityorder.txt}     & \inputNumber{../figures/varmean/GlmSelectBicAbsNoFilterDeg30bw_identityorder.txt}     & \inputNumber{../figures/varmean/GlmSelectBicAbsNoFilterDeg30linear_identityorder.txt}     \\
                     & votes     & \inputNumber{../figures/varmean/GlmSelectBicAbsNoFilterDeg30null_identityvote.txt}      & \inputNumber{../figures/varmean/GlmSelectBicAbsNoFilterDeg30bw_identityvote.txt}       & \inputNumber{../figures/varmean/GlmSelectBicAbsNoFilterDeg30linear_identityvote.txt}      \\
                     & criterion & \inputNumber{../figures/varmean/GlmSelectBicAbsNoFilterDeg30null_identitycriterion.txt} & \inputNumber{../figures/varmean/GlmSelectBicAbsNoFilterDeg30bw_identitycriterion.txt} & \inputNumber{../figures/varmean/GlmSelectBicAbsNoFilterDeg30linear_identitycriterion.txt} \\
\\
\multicolumn{2}{c|}{Canonical Link}& null & bw & linear \\ \hline
\multirow{3}{*}{AIC} & order      & \inputNumber{../figures/varmean/GlmSelectAicAbsNoFilterDeg30null_reciprocalorder.txt}     & \inputNumber{../figures/varmean/GlmSelectAicAbsNoFilterDeg30bw_reciprocalorder.txt}     & \inputNumber{../figures/varmean/GlmSelectAicAbsNoFilterDeg30linear_reciprocalorder.txt}     \\
                     & votes      & \inputNumber{../figures/varmean/GlmSelectAicAbsNoFilterDeg30null_reciprocalvote.txt}      & \inputNumber{../figures/varmean/GlmSelectAicAbsNoFilterDeg30bw_reciprocalvote.txt}      & \inputNumber{../figures/varmean/GlmSelectAicAbsNoFilterDeg30linear_reciprocalvote.txt}      \\
                     & criterion  & \inputNumber{../figures/varmean/GlmSelectAicAbsNoFilterDeg30null_reciprocalcriterion.txt} & \inputNumber{../figures/varmean/GlmSelectAicAbsNoFilterDeg30bw_reciprocalcriterion.txt} & \inputNumber{../figures/varmean/GlmSelectAicAbsNoFilterDeg30linear_reciprocalcriterion.txt} \\ \hline
\multirow{3}{*}{BIC} & order      & \inputNumber{../figures/varmean/GlmSelectBicAbsNoFilterDeg30null_reciprocalorder.txt}     & \inputNumber{../figures/varmean/GlmSelectBicAbsNoFilterDeg30bw_reciprocalorder.txt}     & \inputNumber{../figures/varmean/GlmSelectBicAbsNoFilterDeg30linear_reciprocalorder.txt}     \\
                     & votes      & \inputNumber{../figures/varmean/GlmSelectBicAbsNoFilterDeg30null_reciprocalvote.txt}      & \inputNumber{../figures/varmean/GlmSelectBicAbsNoFilterDeg30bw_reciprocalvote.txt}      & \inputNumber{../figures/varmean/GlmSelectBicAbsNoFilterDeg30linear_reciprocalvote.txt}      \\
                     & criterion  & \inputNumber{../figures/varmean/GlmSelectBicAbsNoFilterDeg30null_reciprocalcriterion.txt} & \inputNumber{../figures/varmean/GlmSelectBicAbsNoFilterDeg30bw_reciprocalcriterion.txt} & \inputNumber{../figures/varmean/GlmSelectBicAbsNoFilterDeg30linear_reciprocalcriterion.txt}
\end{tabular}
\caption{Forward stepwise selection was used to find suitable polynomial features when fitting a gamma GLM onto the sample variance-mean data from the projections in \texttt{AbsNoFilter} at \ang{30}. The columns of the table represent different shading corrections. Forward stepwise selection was repeated 10 times by bootstrapping the replicate projections. `Order' shows the most commonly selected polynomial orders and `votes' shows the number of times it was selected out of the 10 repeats. The error bars are the standard deviation from the 10 repeats.}
\label{table:meanVar_glmselect_absnofilterdeg30}
\end{sidewaystable}

\begin{sidewaystable}
\centering
\begin{tabular}{ll|ccc}
\multicolumn{2}{c|}{Identity Link}& null & bw & linear \\ \hline
\multirow{3}{*}{AIC} & order     & \inputNumber{../figures/varmean/GlmSelectAicAbsNoFilterDeg120null_identityorder.txt}     & \inputNumber{../figures/varmean/GlmSelectAicAbsNoFilterDeg120bw_identityorder.txt}     & \inputNumber{../figures/varmean/GlmSelectAicAbsNoFilterDeg120linear_identityorder.txt}     \\
                     & votes     & \inputNumber{../figures/varmean/GlmSelectAicAbsNoFilterDeg120null_identityvote.txt}      & \inputNumber{../figures/varmean/GlmSelectAicAbsNoFilterDeg120bw_identityvote.txt}      & \inputNumber{../figures/varmean/GlmSelectAicAbsNoFilterDeg120linear_identityvote.txt}      \\
                     & criterion & \inputNumber{../figures/varmean/GlmSelectAicAbsNoFilterDeg120null_identitycriterion.txt} & \inputNumber{../figures/varmean/GlmSelectAicAbsNoFilterDeg120bw_identitycriterion.txt} & \inputNumber{../figures/varmean/GlmSelectAicAbsNoFilterDeg120linear_identitycriterion.txt} \\ \hline
\multirow{3}{*}{BIC} & order     & \inputNumber{../figures/varmean/GlmSelectBicAbsNoFilterDeg120null_identityorder.txt}     & \inputNumber{../figures/varmean/GlmSelectBicAbsNoFilterDeg120bw_identityorder.txt}     & \inputNumber{../figures/varmean/GlmSelectBicAbsNoFilterDeg120linear_identityorder.txt}     \\
                     & votes     & \inputNumber{../figures/varmean/GlmSelectBicAbsNoFilterDeg120null_identityvote.txt}      & \inputNumber{../figures/varmean/GlmSelectBicAbsNoFilterDeg120bw_identityvote.txt}       & \inputNumber{../figures/varmean/GlmSelectBicAbsNoFilterDeg120linear_identityvote.txt}      \\
                     & criterion & \inputNumber{../figures/varmean/GlmSelectBicAbsNoFilterDeg120null_identitycriterion.txt} & \inputNumber{../figures/varmean/GlmSelectBicAbsNoFilterDeg120bw_identitycriterion.txt} & \inputNumber{../figures/varmean/GlmSelectBicAbsNoFilterDeg120linear_identitycriterion.txt} \\
\\
\multicolumn{2}{c|}{Canonical Link}& null & bw & linear \\ \hline
\multirow{3}{*}{AIC} & order      & \inputNumber{../figures/varmean/GlmSelectAicAbsNoFilterDeg120null_reciprocalorder.txt}     & \inputNumber{../figures/varmean/GlmSelectAicAbsNoFilterDeg120bw_reciprocalorder.txt}     & \inputNumber{../figures/varmean/GlmSelectAicAbsNoFilterDeg120linear_reciprocalorder.txt}     \\
                     & votes      & \inputNumber{../figures/varmean/GlmSelectAicAbsNoFilterDeg120null_reciprocalvote.txt}      & \inputNumber{../figures/varmean/GlmSelectAicAbsNoFilterDeg120bw_reciprocalvote.txt}      & \inputNumber{../figures/varmean/GlmSelectAicAbsNoFilterDeg120linear_reciprocalvote.txt}      \\
                     & criterion  & \inputNumber{../figures/varmean/GlmSelectAicAbsNoFilterDeg120null_reciprocalcriterion.txt} & \inputNumber{../figures/varmean/GlmSelectAicAbsNoFilterDeg120bw_reciprocalcriterion.txt} & \inputNumber{../figures/varmean/GlmSelectAicAbsNoFilterDeg120linear_reciprocalcriterion.txt} \\ \hline
\multirow{3}{*}{BIC} & order      & \inputNumber{../figures/varmean/GlmSelectBicAbsNoFilterDeg120null_reciprocalorder.txt}     & \inputNumber{../figures/varmean/GlmSelectBicAbsNoFilterDeg120bw_reciprocalorder.txt}     & \inputNumber{../figures/varmean/GlmSelectBicAbsNoFilterDeg120linear_reciprocalorder.txt}     \\
                     & votes      & \inputNumber{../figures/varmean/GlmSelectBicAbsNoFilterDeg120null_reciprocalvote.txt}      & \inputNumber{../figures/varmean/GlmSelectBicAbsNoFilterDeg120bw_reciprocalvote.txt}      & \inputNumber{../figures/varmean/GlmSelectBicAbsNoFilterDeg120linear_reciprocalvote.txt}      \\
                     & criterion  & \inputNumber{../figures/varmean/GlmSelectBicAbsNoFilterDeg120null_reciprocalcriterion.txt} & \inputNumber{../figures/varmean/GlmSelectBicAbsNoFilterDeg120bw_reciprocalcriterion.txt} & \inputNumber{../figures/varmean/GlmSelectBicAbsNoFilterDeg120linear_reciprocalcriterion.txt}
\end{tabular}
\caption{Forward stepwise selection was used to find suitable polynomial features when fitting a gamma GLM onto the sample variance-mean data from the projections in \texttt{AbsNoFilter} at \ang{120}. The columns of the table represent different shading corrections. Forward stepwise selection was repeated 10 times by bootstrapping the replicate projections. `Order' shows the most commonly selected polynomial orders and `votes' shows the number of times it was selected out of the 10 repeats. The error bars are the standard deviation from the 10 repeats.}
\label{table:meanVar_glmselect_absnofilterdeg120}
\end{sidewaystable}

\begin{sidewaystable}
\centering
\begin{tabular}{ll|ccc}
\multicolumn{2}{c|}{Identity Link}& null & bw & linear \\ \hline
\multirow{3}{*}{AIC} & order     & \inputNumber{../figures/varmean/GlmSelectAicAbsFilterDeg30null_identityorder.txt}     & \inputNumber{../figures/varmean/GlmSelectAicAbsFilterDeg30bw_identityorder.txt}     & \inputNumber{../figures/varmean/GlmSelectAicAbsFilterDeg30linear_identityorder.txt}     \\
                     & votes     & \inputNumber{../figures/varmean/GlmSelectAicAbsFilterDeg30null_identityvote.txt}      & \inputNumber{../figures/varmean/GlmSelectAicAbsFilterDeg30bw_identityvote.txt}      & \inputNumber{../figures/varmean/GlmSelectAicAbsFilterDeg30linear_identityvote.txt}      \\
                     & criterion & \inputNumber{../figures/varmean/GlmSelectAicAbsFilterDeg30null_identitycriterion.txt} & \inputNumber{../figures/varmean/GlmSelectAicAbsFilterDeg30bw_identitycriterion.txt} & \inputNumber{../figures/varmean/GlmSelectAicAbsFilterDeg30linear_identitycriterion.txt} \\ \hline
\multirow{3}{*}{BIC} & order     & \inputNumber{../figures/varmean/GlmSelectBicAbsFilterDeg30null_identityorder.txt}     & \inputNumber{../figures/varmean/GlmSelectBicAbsFilterDeg30bw_identityorder.txt}     & \inputNumber{../figures/varmean/GlmSelectBicAbsFilterDeg30linear_identityorder.txt}     \\
                     & votes     & \inputNumber{../figures/varmean/GlmSelectBicAbsFilterDeg30null_identityvote.txt}      & \inputNumber{../figures/varmean/GlmSelectBicAbsFilterDeg30bw_identityvote.txt}       & \inputNumber{../figures/varmean/GlmSelectBicAbsFilterDeg30linear_identityvote.txt}      \\
                     & criterion & \inputNumber{../figures/varmean/GlmSelectBicAbsFilterDeg30null_identitycriterion.txt} & \inputNumber{../figures/varmean/GlmSelectBicAbsFilterDeg30bw_identitycriterion.txt} & \inputNumber{../figures/varmean/GlmSelectBicAbsFilterDeg30linear_identitycriterion.txt} \\
\\
\multicolumn{2}{c|}{Canonical Link}& null & bw & linear \\ \hline
\multirow{3}{*}{AIC} & order      & \inputNumber{../figures/varmean/GlmSelectAicAbsFilterDeg30null_reciprocalorder.txt}     & \inputNumber{../figures/varmean/GlmSelectAicAbsFilterDeg30bw_reciprocalorder.txt}     & \inputNumber{../figures/varmean/GlmSelectAicAbsFilterDeg30linear_reciprocalorder.txt}     \\
                     & votes      & \inputNumber{../figures/varmean/GlmSelectAicAbsFilterDeg30null_reciprocalvote.txt}      & \inputNumber{../figures/varmean/GlmSelectAicAbsFilterDeg30bw_reciprocalvote.txt}      & \inputNumber{../figures/varmean/GlmSelectAicAbsFilterDeg30linear_reciprocalvote.txt}      \\
                     & criterion  & \inputNumber{../figures/varmean/GlmSelectAicAbsFilterDeg30null_reciprocalcriterion.txt} & \inputNumber{../figures/varmean/GlmSelectAicAbsFilterDeg30bw_reciprocalcriterion.txt} & \inputNumber{../figures/varmean/GlmSelectAicAbsFilterDeg30linear_reciprocalcriterion.txt} \\ \hline
\multirow{3}{*}{BIC} & order      & \inputNumber{../figures/varmean/GlmSelectBicAbsFilterDeg30null_reciprocalorder.txt}     & \inputNumber{../figures/varmean/GlmSelectBicAbsFilterDeg30bw_reciprocalorder.txt}     & \inputNumber{../figures/varmean/GlmSelectBicAbsFilterDeg30linear_reciprocalorder.txt}     \\
                     & votes      & \inputNumber{../figures/varmean/GlmSelectBicAbsFilterDeg30null_reciprocalvote.txt}      & \inputNumber{../figures/varmean/GlmSelectBicAbsFilterDeg30bw_reciprocalvote.txt}      & \inputNumber{../figures/varmean/GlmSelectBicAbsFilterDeg30linear_reciprocalvote.txt}      \\
                     & criterion  & \inputNumber{../figures/varmean/GlmSelectBicAbsFilterDeg30null_reciprocalcriterion.txt} & \inputNumber{../figures/varmean/GlmSelectBicAbsFilterDeg30bw_reciprocalcriterion.txt} & \inputNumber{../figures/varmean/GlmSelectBicAbsFilterDeg30linear_reciprocalcriterion.txt}
\end{tabular}
\caption{Forward stepwise selection was used to find suitable polynomial features when fitting a gamma GLM onto the sample variance-mean data from the projections in \texttt{AbsFilter} at \ang{30}. The columns of the table represent different shading corrections. Forward stepwise selection was repeated 10 times by bootstrapping the replicate projections. `Order' shows the most commonly selected polynomial orders and `votes' shows the number of times it was selected out of the 10 repeats. The error bars are the standard deviation from the 10 repeats.}
\label{table:meanVar_glmselect_absfilterdeg30}
\end{sidewaystable}

\begin{sidewaystable}
\centering
\begin{tabular}{ll|ccc}
\multicolumn{2}{c|}{Identity Link}& null & bw & linear \\ \hline
\multirow{3}{*}{AIC} & order     & \inputNumber{../figures/varmean/GlmSelectAicAbsFilterDeg120null_identityorder.txt}     & \inputNumber{../figures/varmean/GlmSelectAicAbsFilterDeg120bw_identityorder.txt}     & \inputNumber{../figures/varmean/GlmSelectAicAbsFilterDeg120linear_identityorder.txt}     \\
                     & votes     & \inputNumber{../figures/varmean/GlmSelectAicAbsFilterDeg120null_identityvote.txt}      & \inputNumber{../figures/varmean/GlmSelectAicAbsFilterDeg120bw_identityvote.txt}      & \inputNumber{../figures/varmean/GlmSelectAicAbsFilterDeg120linear_identityvote.txt}      \\
                     & criterion & \inputNumber{../figures/varmean/GlmSelectAicAbsFilterDeg120null_identitycriterion.txt} & \inputNumber{../figures/varmean/GlmSelectAicAbsFilterDeg120bw_identitycriterion.txt} & \inputNumber{../figures/varmean/GlmSelectAicAbsFilterDeg120linear_identitycriterion.txt} \\ \hline
\multirow{3}{*}{BIC} & order     & \inputNumber{../figures/varmean/GlmSelectBicAbsFilterDeg120null_identityorder.txt}     & \inputNumber{../figures/varmean/GlmSelectBicAbsFilterDeg120bw_identityorder.txt}     & \inputNumber{../figures/varmean/GlmSelectBicAbsFilterDeg120linear_identityorder.txt}     \\
                     & votes     & \inputNumber{../figures/varmean/GlmSelectBicAbsFilterDeg120null_identityvote.txt}      & \inputNumber{../figures/varmean/GlmSelectBicAbsFilterDeg120bw_identityvote.txt}       & \inputNumber{../figures/varmean/GlmSelectBicAbsFilterDeg120linear_identityvote.txt}      \\
                     & criterion & \inputNumber{../figures/varmean/GlmSelectBicAbsFilterDeg120null_identitycriterion.txt} & \inputNumber{../figures/varmean/GlmSelectBicAbsFilterDeg120bw_identitycriterion.txt} & \inputNumber{../figures/varmean/GlmSelectBicAbsFilterDeg120linear_identitycriterion.txt} \\
\\
\multicolumn{2}{c|}{Canonical Link}& null & bw & linear \\ \hline
\multirow{3}{*}{AIC} & order      & \inputNumber{../figures/varmean/GlmSelectAicAbsFilterDeg120null_reciprocalorder.txt}     & \inputNumber{../figures/varmean/GlmSelectAicAbsFilterDeg120bw_reciprocalorder.txt}     & \inputNumber{../figures/varmean/GlmSelectAicAbsFilterDeg120linear_reciprocalorder.txt}     \\
                     & votes      & \inputNumber{../figures/varmean/GlmSelectAicAbsFilterDeg120null_reciprocalvote.txt}      & \inputNumber{../figures/varmean/GlmSelectAicAbsFilterDeg120bw_reciprocalvote.txt}      & \inputNumber{../figures/varmean/GlmSelectAicAbsFilterDeg120linear_reciprocalvote.txt}      \\
                     & criterion  & \inputNumber{../figures/varmean/GlmSelectAicAbsFilterDeg120null_reciprocalcriterion.txt} & \inputNumber{../figures/varmean/GlmSelectAicAbsFilterDeg120bw_reciprocalcriterion.txt} & \inputNumber{../figures/varmean/GlmSelectAicAbsFilterDeg120linear_reciprocalcriterion.txt} \\ \hline
\multirow{3}{*}{BIC} & order      & \inputNumber{../figures/varmean/GlmSelectBicAbsFilterDeg120null_reciprocalorder.txt}     & \inputNumber{../figures/varmean/GlmSelectBicAbsFilterDeg120bw_reciprocalorder.txt}     & \inputNumber{../figures/varmean/GlmSelectBicAbsFilterDeg120linear_reciprocalorder.txt}     \\
                     & votes      & \inputNumber{../figures/varmean/GlmSelectBicAbsFilterDeg120null_reciprocalvote.txt}      & \inputNumber{../figures/varmean/GlmSelectBicAbsFilterDeg120bw_reciprocalvote.txt}      & \inputNumber{../figures/varmean/GlmSelectBicAbsFilterDeg120linear_reciprocalvote.txt}      \\
                     & criterion  & \inputNumber{../figures/varmean/GlmSelectBicAbsFilterDeg120null_reciprocalcriterion.txt} & \inputNumber{../figures/varmean/GlmSelectBicAbsFilterDeg120bw_reciprocalcriterion.txt} & \inputNumber{../figures/varmean/GlmSelectBicAbsFilterDeg120linear_reciprocalcriterion.txt}
\end{tabular}
\caption{Forward stepwise selection was used to find suitable polynomial features when fitting a gamma GLM onto the sample variance-mean data from the projections in \texttt{AbsFilter} at \ang{120}. The columns of the table represent different shading corrections. Forward stepwise selection was repeated 10 times by bootstrapping the replicate projections. `Order' shows the most commonly selected polynomial orders and `votes' shows the number of times it was selected out of the 10 repeats. The error bars are the standard deviation from the 10 repeats.}
\label{table:meanVar_glmselect_absfilterdeg120}
\end{sidewaystable}

The results are shown in Tables \ref{table:meanVar_glmselect_absnofilterdeg30} and \ref{table:meanVar_glmselect_absnofilterdeg120} for \texttt{AbsNoFilter} and Tables \ref{table:meanVar_glmselect_absfilterdeg30} and \ref{table:meanVar_glmselect_absfilterdeg120} for \texttt{AbsFilter}. The models selected are quite simple and all have two features. Different shading corrections or different criteria had no effect on the selected model. The method was quite robust to the variation introduced to the dataset when repeating the experiment because all 10 repeats consistently selected the same model.

There was some variation to the selected models between datasets. For example, when using the identity link, \texttt{AbsNoFilter} preferred $\eta(x)=\beta_0+\beta_1 x$ whereas \texttt{AbsFilter} preferred $\eta(x)=\beta_{-1}x^{-1}+\beta_0$. Using the canonical link, both datasets selected $\eta(x)=\beta_{-1}x^{-1}+\beta_0$ which correspond to $\widehat{y}(x)=\left(\beta_{-1}x^{-1}+\beta_0\right)^{-1}$.

Figures \ref{fig:meanVar_varMeanExample_AbsNoFilter} and \ref{fig:meanVar_varMeanExample_AbsFilter} shows the GLM fits for the datasets \texttt{AbsNoFilter} and \texttt{AbsFilter} respectively. The prediction intervals were obtained using the distribution $\widehat{Y}(x)\sim\gammaDist\left(\alpha, \dfrac{\alpha}{g^{-1}\left(\widehat{\eta}(x)\right)}\right)$. The fits all looked reasonable except for \texttt{AbsNoFilter} at \ang{120} where the fit did not capture the inflection. The forward stepwise selection may not picked this up because there were a lot of low grey values, causing a leverage towards these low grey values. As a result, the inflection did not stand out to the method.

\begin{figure}
  \centering
    \centerline{
    \begin{subfigure}[b]{\subSize}
        \includegraphics[width=\textwidth]{../figures/varmean/varMeanExample_AbsNoFilterDeg30_identity000010.eps}
        \caption{${y}(x)=\beta_0+\beta_1 x$ at \ang{30}}
    \end{subfigure}
    \begin{subfigure}[b]{\subSize}
        \includegraphics[width=\textwidth]{../figures/varmean/varMeanExample_AbsNoFilterDeg30_reciprocal000100.eps}
        \caption{${y}(x)=(\beta_{-1}x^{-1}\beta_0)^{-1}$ at \ang{30}}
    \end{subfigure}
    }
    \centerline{
    \begin{subfigure}[b]{\subSize}
        \includegraphics[width=\textwidth]{../figures/varmean/varMeanExample_AbsNoFilterDeg120_identity000010.eps}
        \caption{${y}(x)=\beta_0+\beta_1 x$ at \ang{120}}
    \end{subfigure}
    \begin{subfigure}[b]{\subSize}
        \includegraphics[width=\textwidth]{../figures/varmean/varMeanExample_AbsNoFilterDeg120_reciprocal000100.eps}
        \caption{${y}(x)=(\beta_{-1}x^{-1}\beta_0)^{-1}$ at \ang{120}}
    \end{subfigure}
    }
    \caption{Log frequency density histogram of the sample variance-mean data from \texttt{AbsNoFilter} with linear shading correction. The solid red line shows the GLM fit along with the 68\% prediction interval as dashed lines. The colour scales are in units of $\log\SI{}{\adu^{-3}}$.}
    \label{fig:meanVar_varMeanExample_AbsNoFilter}
\end{figure}

\begin{figure}
  \centering
    \centerline{
    \begin{subfigure}[b]{\subSize}
        \includegraphics[width=\textwidth]{../figures/varmean/varMeanExample_AbsFilterDeg30_identity000100.eps}
        \caption{${y}(x)=\beta_{-1}x^{-1}+\beta_0$ at \ang{30}}
    \end{subfigure}
    \begin{subfigure}[b]{\subSize}
        \includegraphics[width=\textwidth]{../figures/varmean/varMeanExample_AbsFilterDeg30_reciprocal000100.eps}
        \caption{${y}(x)=(\beta_{-1}x^{-1}+\beta_0)^{-1}$ at \ang{30}}
    \end{subfigure}
    }
    \centerline{
    \begin{subfigure}[b]{\subSize}
        \includegraphics[width=\textwidth]{../figures/varmean/varMeanExample_AbsFilterDeg120_identity000100.eps}
        \caption{${y}(x)=\beta_{-1}x^{-1}+\beta_0$ at \ang{120}}
    \end{subfigure}
    \begin{subfigure}[b]{\subSize}
        \includegraphics[width=\textwidth]{../figures/varmean/varMeanExample_AbsFilterDeg120_reciprocal000100.eps}
        \caption{${y}(x)=(\beta_{-1}x^{-1}+\beta_0)^{-1}$ at \ang{120}}
    \end{subfigure}
    }
    \caption{Log frequency density histogram of the sample variance-mean data from \texttt{AbsFilter} with linear shading correction. The solid red line shows the GLM fit along with the 68\% prediction interval as dashed lines. The colour scales are in units of $\log\SI{}{\adu^{-3}}$.}
    \label{fig:meanVar_varMeanExample_AbsFilter}
\end{figure}

\section{Cross-Validation}

Instead of assessing the model fit using the AIC and BIC, the performance on predicting the variance given a grey value was assessed using cross-validation \citep{allen1974relationship, stone1974cross, stone1977asymptotic, friedman2001elements}. Cross-validation assesses the model to make predictions on data it has not seen before. This was done by randomly splitting the $N$ data points into two disjoint sets, the training set and the test set $\tau$. The model was fitted onto the training set. Afterwards, the fitted model predicts the variances given the grey values in the test set which were then compared to the actual variances.

It was chosen that the training and test set are of the same size, a 50:50 spilt. The sizes of the sets can be different, for example, a 75:25 spilt. A 50:50 spilt was chosen because a large training set is an overkill for a high data and low dimensional problem. Here, a model with a few parameters is fitted onto millions of pixels.

The mean scaled deviance was used to assess the performance of variance prediction of the test set. The deviance is defined to be
\begin{equation}
  D = 2\left(
    \ln L_\mathrm{s} - \ln L_\tau
  \right)
\end{equation}
where
\begin{equation}
\ln L_\tau = \sum_{i\in\tau} \left[
    \alpha\ln\alpha
    -\ln\Gamma(\alpha)
    -\alpha\ln \widehat{y}_i
    +(\alpha-1)\ln y_i
    -\frac{\alpha y_i}{\widehat{y}_i}
  \right]
\end{equation}
is the log-likelihood of the test set given the fitted model and $\ln L_\mathrm{s}$ is the saturated log-likelihood. This is obtained by replacing all $\widehat{y}_i$ with $y_i$ in $\ln L_\tau$ so that
\begin{equation}
  \ln L_\mathrm{s} = \sum_{i\in\tau}\left[
    \alpha\ln\alpha
    -\ln\Gamma(\alpha)
    -\ln y_i
    -\alpha
  \right]
  \ .
\end{equation}
Following from this, the deviance is
\begin{equation}
  D = 2\alpha
  \sum_{i\in\tau}\left[
      \dfrac{
          y_i-\widehat{y}_i
      }
      {
          \widehat{y}_i
      }
      - \ln\left(\dfrac{y_i}{\widehat{y}_i}\right)
  \right]
  \ .
\end{equation}
The mean scaled deviance is obtained by removing the factor of $\alpha$ and dividing by $N_\tau$ to get
\begin{equation}
    D_\mathrm{s} = \dfrac{2}{N_\tau}
    \sum_{i\in\tau}\left[
        \dfrac{
            y_i-\widehat{y}_i
        }
        {
            \widehat{y}_i
        }
        - \ln\left(\dfrac{y_i}{\widehat{y}_i}\right)
    \right]
    \ .
\end{equation}

\begin{figure}
  \centering
  \includegraphics[width=\subSize]{../figures/varmean/devianceGraph.eps}
  \caption{The scaled deviance loss function}
  \label{fig:meanVar_deviance}
\end{figure}

The mean scaled deviance is a loss function which increases as $y_i/\widehat{y}_i$ deviates from one, this is shown in Figure \ref{fig:meanVar_deviance} and it should be noted that the $x$-axis is in log scale. Another way to show this is by letting $r_i = y_i/\widehat{y}_i$ and $d_i = 2\left[r_i-1-\ln r_i\right]$ be an element from the sum in the deviance. For $r_i\approx 1$, $\ln(r_i)\approx(r_i-1)-(r_i-1)^2/2$ which implies that $d_i\approx (r_i-1)^2$ with minimum at one. For $r_i$ deviate greatly from one, the loss function is asymmetric. For example a ratio of $r_i=10^1$ has a greater penalty than $r_i=10^{-1}$. This means that in extreme cases, overestimates are penalised less than underestimates relative to $y_i$.

Assuming the model is correct, it is given that $D\sim\chi_{N_\tau-k}^2$ which implies that for large $N_\tau$,
\begin{equation}
\expectation\left[D_\mathrm{s}\right] = \dfrac{1}{\alpha}
\end{equation}
and
\begin{equation}
\variance\left[D_\mathrm{s}\right] = \dfrac{2}{\alpha^2 N_\tau} \ .
\end{equation}
Because $\alpha=(m-1)/2$, this shows that the number of replicated projections used to obtain the sample variance-mean data has an influence on the mean scaled deviance. This result can be used to estimate $\alpha$ if it is unknown.

Cross validation was performed on the datasets \texttt{AbsNoFilter} and \texttt{AbsFilter} with various shading corrections. The models selected from forward stepwise selection in the previous section were assessed. They are ${y}(x)=\beta_0+\beta_1 x$ and ${y}(x)=\beta_{-1}x^{-1}+\beta_0$ using the identity link and ${y}(x)=(\beta_{-1}x^{-1}+\beta_0)^{-1}$ using the canonical link. The analysis was repeated 100 times by using a random permutation with replacement of the replicate projections to obtain a different sample variance-mean data which introduced some variation to the data \citep{efron1979bootstrap}.

\begin{figure}
  \centering
  \centerline{
  \begin{subfigure}[b]{\subSize}
      \includegraphics[width=\textwidth]{../figures/varmean/VarMeanCvAbsNoFilterDeg30_testmeanscaleddeviance.eps}
      \caption{\texttt{AbsNoFilter} at \ang{30}}
  \end{subfigure}
  \begin{subfigure}[b]{\subSize}
      \includegraphics[width=\textwidth]{../figures/varmean/VarMeanCvAbsNoFilterDeg120_testmeanscaleddeviance.eps}
      \caption{\texttt{AbsNoFilter} at \ang{120}}
  \end{subfigure}
  }
    \centerline{
  \begin{subfigure}[b]{\subSize}
      \includegraphics[width=\textwidth]{../figures/varmean/VarMeanCvAbsFilterDeg30_testmeanscaleddeviance.eps}
      \caption{\texttt{AbsFilter} at \ang{30}}
  \end{subfigure}
  \begin{subfigure}[b]{\subSize}
      \includegraphics[width=\textwidth]{../figures/varmean/VarMeanCvAbsFilterDeg120_testmeanscaleddeviance.eps}
      \caption{\texttt{AbsFilter} at \ang{120}}
  \end{subfigure}
  }
  \caption{Test mean scaled deviance from predicting variances in a test set using a GLM fitted onto a training set. The different colours represent different shading corrections. The boxplots represent the 100 repeats of the analysis by bootstrapping the replicate projections.}
  \label{fig:meanVar_varMeanCv}
\end{figure}

The results from the cross-validation is shown in Figure \ref{fig:meanVar_varMeanCv}. The performances of the three candidate models were very similar. One exception is the model ${y}(x)=\beta_{-1}x^{-1}+\beta_0$ fitted onto the \texttt{AbsNoFilter} dataset because the mean scaled deviance was significantly larger. This is expected from a model not favoured in the forward stepwise selection in the previous section.

Shading correction did not have a significant effect on the analysis.

From these results, it is recommended that the relationship ${y}(x)=\beta_0+\beta_1 x$ should be used for its simple form, similar performance to other candidate models and connections to the compound Poisson.

\section{Residual Analysis}

Residual analysis was conducted to study for anything overlooked. The model ${y}(x)=\beta_0+\beta_1 x$ was fitted onto the entire sample variance-mean data. A residual plot plots $r_i=y_i-\widehat{y}(x_i)$ for all $x_i$ in the dataset which was done using a 2D histogram. Due to the gamma GLM, higher grey values typically have bigger residuals in magnitude. A prediction residual interval is included in the histogram to aid judging the residuals. This interval was acquired by obtaining the prediction interval and subtracting it from the fit.

\begin{figure}
  \centering
  \centerline{
  \begin{subfigure}[b]{\subSize}
      \includegraphics[width=\textwidth]{../figures/varmean/varMeanResidualAbsNoFilterDeg30_vsgreyvalue.eps}
      \caption{\texttt{AbsNoFilter} at \ang{30}}
  \end{subfigure}
  \begin{subfigure}[b]{\subSize}
      \includegraphics[width=\textwidth]{../figures/varmean/varMeanResidualAbsNoFilterDeg120_vsgreyvalue.eps}
      \caption{\texttt{AbsNoFilter} at \ang{120}}
  \end{subfigure}
  }
  \centerline{
  \begin{subfigure}[b]{\subSize}
      \includegraphics[width=\textwidth]{../figures/varmean/varMeanResidualAbsFilterDeg30_vsgreyvalue.eps}
      \caption{\texttt{AbsFilter} at \ang{30}}
  \end{subfigure}
  \begin{subfigure}[b]{\subSize}
      \includegraphics[width=\textwidth]{../figures/varmean/varMeanResidualAbsFilterDeg120_vsgreyvalue.eps}
      \caption{\texttt{AbsFilter} at \ang{120}}
  \end{subfigure}
  }
  \caption{Log frequency density histogram of the residuals given the mean grey value. The residuals are from fitting a gamma GLM of the form ${y}(x)=\beta_0+\beta_1 x$ onto the sample variance-mean data obtained from the replicate projections. The dashed red line shows the 68\% prediction residual interval. The colour scales are in units of $\log\SI{}{\adu^{-3}}$.}
  \label{fig:meanVar_residualGreyvalue}
\end{figure}

The residual plots are shown in Figure \ref{fig:meanVar_residualGreyvalue}. The residuals are all sensible and increased in magnitude with the grey value which is captured by the prediction residual interval. The inflection in \texttt{AbsNoFilter} at \ang{120}, as discussed previously, can be seen more clearly here.

\begin{figure}
  \centering
  \centerline{
  \begin{subfigure}[b]{\subSize}
      \includegraphics[width=\textwidth]{../figures/varmean/varMeanResidualAbsNoFilterDeg30_spatial.eps}
      \caption{\texttt{AbsNoFilter} at \ang{30}}
  \end{subfigure}
  \begin{subfigure}[b]{\subSize}
      \includegraphics[width=\textwidth]{../figures/varmean/varMeanResidualAbsNoFilterDeg120_spatial.eps}
      \caption{\texttt{AbsNoFilter} at \ang{120}}
  \end{subfigure}
  }
  \centerline{
  \begin{subfigure}[b]{\subSize}
      \includegraphics[width=\textwidth]{../figures/varmean/varMeanResidualAbsFilterDeg30_spatial.eps}
      \caption{\texttt{AbsFilter} at \ang{30}}
  \end{subfigure}
  \begin{subfigure}[b]{\subSize}
      \includegraphics[width=\textwidth]{../figures/varmean/varMeanResidualAbsFilterDeg120_spatial.eps}
      \caption{\texttt{AbsFilter} at \ang{120}}
  \end{subfigure}
  }
  \caption{Absolute value residuals from fitting a gamma GLM of the form ${y}(x)=\beta_0+\beta_1 x$ onto the sample variance-mean data obtained from the replicate projections. Highlighted in a dashed red box is an example of an inflection. The colour scales are in units of \SI{}{\adu^2}.}
  \label{fig:meanVar_spatialResidual}
\end{figure}

Because no spatial information was used in modelling the variance, the residuals were plotted on top of the projections to look for any spatial structure, this is shown in Figure \ref{fig:meanVar_spatialResidual}. Because of the geometry of the test sample, lower grey values, thus smaller residuals, were found in the middle of the projection. Similarly, higher grey values, thus bigger residuals, were found on the edges of the test sample. Considering that, the residuals do not show any striking spatial structure. Again, the inflection in \texttt{AbsNoFilter} at \ang{120} can be seen more clearly here.

\section{Conclusion}

Various parametric models were investigated. A gamma GLM with identity and canonical link functions were looked at with different polynomial features. In terms of cross-validation, most of the models looked at have similar prediction performance when predicting the variance given a grey value. Thus, the relationship $\widehat{y}(x)=\beta_0+\beta_1 x$ using the identity link is attractive for its simplicity and links with the compound Poisson model.

It was found that shading correction had little effect on the variance-mean relationship.

As discussed in the compound Poisson chapter, a linear relationship can be derived by assuming no beam hardening, then it should be expected that a linear relationship is observed in the dataset \texttt{AbsFilter}. In the forward stepwise selection, \texttt{AbsNoFilter} consistently selected the linear model, given the identity link function. \texttt{AbsFilter} did not select the linear model but in terms of cross-validation, the linear model was just as good as the other models.

It was unusual to see an inflection in the sample variance-mean data in \texttt{AbsNoFilter} at \ang{120} which was not captured by the GLM. A good explanation was not found but perhaps the absence of an x-ray filter may have contributed to this. As a result, the \texttt{AbsNoFilter} dataset was discarded in favour of the \texttt{AbsFilter} dataset from the next chapter.

There exist model selection methods such as lasso and elastic net \citep{tibshirani1996regression, zou2005regularization, friedman2010regularization} but they are catered for high dimensional problems involving hundreds of parameters. In this problem, only a few parameters were needed to find a good model thus forward stepwise selection was sufficient. More flexible models such as non-parametric and machine learning methods may be used, however, they are slow and are unnecessary in a low dimensional problem with a large number of data points.

With the ability to predict the variance given a grey value of a pixel, the uncertainty can be quantified. In the next chapter, the projection is compared to a simulated projection in the face of that uncertainty.


\section{Abstract}

Finding real roots of polynomials is a fundamental problem in scientific
computing. This project aims to implement a real-root isolation program based on
Budan’s Theorem and Continued Fraction. Both methods are developed from
Descartes’ rule of signs and work only on square-free polynomials. Therefore,
square-free decomposition is the very first step to isolate the roots of
polynomials. This project applies Yun’s algorithm to make the polynomials have
no repeated roots. And in order to handle errors that come from floating point
computation, interval arithmetic is introduced to replace exact numbers. Besides
that, many methods are used to optimize the program and running time of these
two algorithms are compared. This report also discusses how errors propagated
through computation and how it affects the success of the program.


\paragraph{Definition} A clique of a graph $G$ is a connected subgraph of $G$ which is
isomorphic to a complete graph. For simplicity, we shall denote cliques by the
complete graph $K_t$ to which the clique is isomorphic where $t$ is a positive integer.
\paragraph{Lemma} Assume a bipartite graph has a clique $K_t$. Then $t \leq 2$.
\paragraph{Proof of Lemma} By contradiction: we prove that if a graph has a clique
$K_3$ then it is not bipartite. Further, for $n \geq 3$, $K_n$ has a clique $K_3$, which can
be seen by considering the subgraph generated by any three vertices in $K_t$ and
realising that this subgraph is indeed $K_3$.
\paragraph{} Let $G$ be bipartite and assume it has a clique $K_3$. Label the vertices in $K_3$
as $u,v,w$. A bipartite graph is a graph which has two disjoint subsets $V_1$ and
$V_2$ contained in $V\left(G\right)$ such that $V_1 \cup V_2 = V\left(G\right)$ and $G\left[V_1\right]$ and $G\left[V_2\right]$ are
both discrete, or empty. Choose such $V_1$ and $V_2$ and w.l.o.g. let $u \in V_1$. Then, since $u$
cannot be adjacent to any vertex in $V_1$, we have $v,w \in V_2$, which in turn
implies that $G\left[V_2\right]$ has the edge $vw$ and so is not empty. This contradiction
tells us that $G$ cannot have a $K_3$ clique, which proves the lemma.
$\square$
\paragraph{Answer}For a bipartite graph $G$ with order greater than 5, we have that at
least one of $V_1$, $V_2$ is of order greater than or equal to 3, which implies that $\bar{G}$
has a $K_3$ clique as every pair of non-adjacent vertices in $G$ are adjacent in $\bar{G}$ (and vice
versa) i.e. $\bar{G}\left[V_1\right] \cong K_{n\left(V_1\right)}$ and $\bar{G}\left[V_2\right] \cong K_{n\left(V_2\right)}$.


\section{Splines}
In many situations, breakpoints in the regression function do not make sense.
Would forcing the piecewise polynomials to be continuous suffice? What
about continuous first derivatives?

We start by consider the subspaces of the piecewise polynomial
space. We will denote it with ${\cal
  PP}_k(\bt)$ with $\bt = (t_1,\dots,t_m)'$ the break-points or
interior knots. Different break points define different spaces.

We can put constrains on the behavior of the functions
$g$ at the break points. (We can construct tests to see if these
constrains are suggested by the data but, will not go into this here)

Here is  a trick for forcing the constrains and keeping the linear
model set-up. We can write any function $g \in {\cal PP}_k(\bt)$ in
{\it the truncated basis power}:
\begin{eqnarray*}
g(x) &=& \theta_{0,1} + \theta_{0,2} x + \dots + \theta_{0,k} x^{k-1}
+\\
&\,&  \theta_{1,1}(x-t_1)^0_+ + \theta_{1,2} (x-t_1)^1_+ + \dots +
\theta_{1,k} (x-t_1)^{k-1}_+ +\\
&& \vdots\\
&& \theta_{m,1}(x-t_m)^0_+ + \theta_{m,2} (x-t_m)^1_+ + \dots +
\theta_{m,k} (x-t_m)^{k-1}_+
\end{eqnarray*}
where $(\cdot)_+ = \max(\cdot,0)$. Written in this way the
coefficients $\theta_{1,1},\dots,\theta_{1,k}$ record the jumps in the
different derivative from the first piece to the second.

Notice that the constrains reduce the number of parameters. This is in
agreement with the fact that we are forcing more smoothness.

Now we can force constrains, such as continuity, by putting constrains
like $\theta_{1,1}=0$ etc...

We will concentrate on the cubic splines which are continuous
and have continuous first and second derivatives. In this case we can
write:
\begin{eqnarray*}
g(x) &=& \theta_{0,1} + \theta_{0,2} x + \dots + \theta_{0,4} x^3
+ \theta_{1,k} (x-t_1)^{3}+ \dots + \theta_{m,k} (x-t_m)^{3}
\end{eqnarray*}
How many ``parameters'' in this space?

Note: It is always possible to have less restrictions at knots where we
believe the behavior is ``less smooth'', e.g for the Sr ratios, we
may have ``unsmoothness'' around KTB.

We can write this as a linear space. This setting is not
computationally convenient. In S-Plus there is a function {\tt bs()}
that makes a basis that is convenient for computations.

There is asymptotic theory that goes along with all this but we will
not go into the details. We will just notice that
\[
\E[ f(x) - g(x) ] = O(h_l^{2k} + 1/n_l)
\]
where $h_l$ is the size of the interval where $x$ is in and $n_l$ is
the number of points in it. What does this say?

\subsection{Splines in terms of Spaces and sub-spaces}
The$p$-dimensional
spaces described in Section 4.1 were defined through basis function
$B_j(\bx), j=1,\dots,p$. So
in general we defined for a given range $I \subset {\mathbb R}^k$
\[
{\cal G} =\{ g: g(\bx) = \sum_{j=1}^p \theta_j \beta_j(\bx), \bx \in I,
(\theta_1,\dots,\theta_p) \in {\mathbb R}^p \}
\]

In the previous section we concentrated on $\bx \in \mathbb R$.

In practice we have design points $x_1,\dots,x_n$ and a vector of
responses $\by = (y_1,\dots,y_n)$. We can think of $\by$ as an element
in the $n$-dimensional vector space ${\mathbb R}^n$. In fact we can go
a step further and define a Hilbert space with the usual inner product
definition that gives us the norm
\[
||\by|| = \sum_{i=1}^n y_i^2
\]
Now we can think of least squares estimation as the projection of the
data $\by$ to the sub-space $\bG \subset {\mathbb R}^n$ defined by $\cal
G$ in the following way
\[
\bG = \{ \g \in {\mathbb R}^n: \g = [g(x_1),\dots,g(x_n)]', g \in
{\cal G} \}
\]
Because this space is spanned by the vectors
$[B_1(x_1),\dots,B_p(x_n) ]$ the projection of $\by$ onto $\bG$ is
\[
\bB(\bB'\bB)^{-}\bB'\by
\]
as learned in 751. Here $[\bB]_{ij} = B_j(x_i)$.


\subsection{Sample Variance} \label{sample_variance}
The sample variance based on \cite{chan1983algorithms}. Given is a time series
$Q = (q_1, q_2, \dots, q_i, \dots, q_l)$ with length $l > 0$ over the domain set $\mathbb{U}$ and a distance measure
function $d$ with $d: \mathbb{U} \times \mathbb{U} \to \mathbb{R}$. The variance of a time series $Q$ can be
calculated by the following formula.
\begin{equation}
    VAR(Q) = \frac{1}{l}\sum \limits_{i=1}^{l} d(q_i, \bar{q})^2
\end{equation}
where $\bar{q}$ is the mean of the time series $Q$ as mentioned in \ref{time_series_normalization}. The sample variance
is also a promising candidate as time series measure for a measure based filter.


\subsection{Basic Facts}

\begin{itemize}
    \item If square matrix $A$ is invertible (or inverse exists), then $\boxed{A^{-1}A=AA^{-1}=I}$.
    \item Can test invertibility of matrix using elimination, i.e. the $n\times n$ matrix $A$ must have $n$ nonzero pivots.
    \item If $\mathrm{det}(A)\neq 0$, then $A$ is invertible.
\end{itemize}

\subsection{Computing inverses}

Can compute inverses with Gauss-Jordan, eliminating $[A\:I]$ to $[I\:A^{-1}]$.
If a matrix is invertible, then solution to $A\tb{x}=\tb{b}$ is $\tb{x}=A^{-1}\tb{b}$.\\

A $2\times 2$ matrix is only invertible if $ad-bc\neq 0$:

\[\boxed{
    A^{-1}=
    \frac{1}{ad-bc}
    \begin{bmatrix}
        d&-b\\
        -c&a
    \end{bmatrix}
.}\]

Matrix inversion occurs in reverse order:

\[\boxed{(ABC)^{-1}=C^{-1}B^{-1}A^{-1}}.\]


If you haven't downloaded and unzipped \href{https://libaoj.in/courses/2021f/MATH3341/zip/Math.3341.zip}{\texttt{Math.3341.zip}}. Download and unzip it under \verb|H:| (H Drive if you are working on the Remote Lab). Change the current working directory by typing \verb|cd H:\Math.3341\Math.3341.Lab.10| in the Command Window, and type \verb|edit lab_10_script| in the Command Window to edit \verb|lab_10_script.m|.
%
\section{Define Meshgrid and Evaluate Function at Meshgrid}
\begin{enumerate}[(a)]
    \item Define an anonymous function \verb|f| using the following formula:
        $$
        f(x, y) = \pi^2 [\sin(\pi x) + 4 \sin(2\pi x) + \sin(\pi y) + 4 \sin(2 \pi y)].
        $$
    \item Define both \verb|x| and \verb|y| starting from -1 to 1 with 30 points using \verb|linspace|.
    \item Create meshgrid by \verb|[X, Y] = meshgrid(x, y);|
    \item Evaluate $f(x, y)$ at the mesh grid by \verb|Z = f(X, Y)|.
\end{enumerate}
%---------------------------------------------
\section{Mesh Plots}
%---------------------------------------------

\begin{enumerate}[(a)]
    \item Run the script \verb|lab_10_script.m|.
    \item Mimick the first subplot to create subplot 2, use \verb|meshc(Z)| instead of \verb|mesh(Z)|, change the colormap to \verb|winter| and also change the title to be \verb|winter| as well.
    \item Repeat the above step to create subplot 3 with \verb|mesh(x, y, Z)| and colormap to be \verb|pink|.
    \item Repeat the above step to create subplot 4 with \verb|mesh(Z)| and colormap to be \verb|prism|. Add \verb|hidden off| to the line below the last line.
\end{enumerate}

%---------------------------------------------
\section{Surf Plots}
%---------------------------------------------
Repeat Part 2 to create 4 subplots.
\begin{enumerate}[(a)]
    \item For subplot 1, change \verb|mesh(Z)| to \verb|surf(Z)|.
    \item For subplot 2, change \verb|meshc(Z)| to \verb|surfc(Z)|.
    \item For subplot 3, change \verb|mesh(x, y, Z)| to \verb|surf(x, y, Z)|.
    \item For subplot 4, change \verb|mesh(Z)| to \verb|surf(Z)|.
\end{enumerate}

At last, run the script \verb|lab_10_script.m|, it might take some time to save the figures. You will upload the script file \verb|lab_10_script.m|, and two figure files \verb|lab_10_figure_1.pdf|, \verb|lab_10_figure_2.pdf| to Overleaf, and in \verb|body.tex| change the caption for each figuer. Recompile, and submit the generated .pdf file on WyoCourses.


\section{Maximum likelihood estimation}

Let $\left\{ E,\left(\mathbf{P}_{\theta }\right)_{\theta \in \Theta }\right\}$  be a statistical model associated with a sample of i.i.d. random variables $X_1, X_2, \dots , X_ n$. Assume that there exists $\theta ^* \in \Theta$ such that $X_ i \sim \mathbf{P}_{\theta ^*}$.

The \textbf{likelihood} of the model is the product of the $n$ samples of the pdf/pmf:

\begin{align*}
L_n{(X_1, X_2, \dots , X_ n, \theta)} = \\
{\begin{cases}
    \displaystyle \prod_{i=1}^np_\theta(x_i) & \text{if }E\text{ is discrete} \\
    \displaystyle \prod_{i=1}^nf_\theta(x_i) & \text{if }E\text{ is continous}
\end{cases}}
\end{align*}
The maximum likelihood estimator is the (unique) $\theta$ that minimizes $\widehat{\text {KL}}\left(\mathbf{P}_{\theta ^*}, \mathbf{P}_{\theta }\right)$ over the parameter space. (The minimizer of the KL divergence is unique due to it being strictly convex in the space of distributions once  is fixed.)
\begin{align*}
\widehat{\theta }_ n^{MLE}& = \displaystyle \text {argmin}_{\theta \in \Theta }\widehat{\text {KL}}_ n\left(\mathbf{P}_{\theta ^*}, \mathbf{P}_{\theta }\right)\\
&= \displaystyle \text {argmax}_{\theta \in \Theta } \sum _{i=1}^{n} \ln p_{\theta }(X_ i)\\
&= \displaystyle \text {argmax}_{\theta \in \Theta } \ln \left(\prod _{i=1}^{n} p_{\theta }(X_ i)\right)
\end{align*}

Since taking derivatives of products is hard but easy for sums and $exp()$ is very common in pdfs we usually take the log of the likelihood function before maximizing it.
\begin{align*}
\ell((X_1, X_2, \dots , X_ n, \theta)) &= ln(L_n{(X_1, X_2, \dots , X_ n, \theta)})\\
 &= \sum_{i=1}^n  ln(L_i(X_i,\theta)
\end{align*}

Cookbook: set up the likelihood function, take log of likelihood function. Take the partial derivative of the loglikelihood function wrt. the parameter(s). Set the partial derivative(s) to zero and solve for the parameter.

If an indicator function on the pdf/pmf does not depend on the parameter, it can be ignored. If it depends on the parameter it can't be ignored because there is an discontinuity in the loglikelihood function. The maximum/minimum of the $X_i$ is then the maximum likelihood estimator.

\subsection{Fisher Information}

The Fisher information is the covariance matrix of the gradient of the loglikelihood function. It is equal to the negative expectation of the Hessian of the loglikelihood function and captures the negative of the expected curvature of the loglikelihood function.\\

Let $\theta \in \Theta \subset \mathbb {R}^ d$ and let $\left(E,\left\{ \mathbf{P}_\theta \right\} _{\theta \in \Theta }\right)$ be a statistical model. Let $f_{\theta }(\mathbf x)$ be the pdf of the distribution $\mathbf{P}_\theta$. Then, the Fisher information of the statistical model is.\\

$\mathcal{I}(\theta) = Cov(\nabla \ell(\theta)) = \\ = \mathbb{E}[\nabla \ell(\theta))\nabla\ell(\theta)^T] -  \mathbb{E}[\nabla \ell(\theta)]\mathbb{E}[\nabla \ell(\theta)] =\\ = -\mathbb{E}[\mathbb{H}\ell(\theta)]$\\

Where $\ell (\theta ) = \ln f_\theta (\mathbf X)$.If $\nabla\ell(\theta) \in \mathbb{R}^d$ it is a $d \times d$ matrix. The definition when the distribution has a pmf $p_\theta (\mathbf x)$ is also the same, with the expectation taken with respect to the pmf.\\

Let $(\mathbb {R}, \{ \mathbf{P}_\theta \} _{\theta \in \mathbb {R}})$ denote a continuous statistical model. Let $f_\theta (x)$ denote the pdf (probability density function) of the continuous distribution $\mathbf{P}_\theta$. Assume that $f_\theta (x)$ is twice-differentiable as a function of the parameter $\theta$.\\

Formula for the calculation of Fisher Information of $X$:\\

$\mathcal{I}(\theta )= \int _{-\infty }^\infty \frac{\left(\frac{\partial f_\theta (x)}{\partial \theta }\right)^2}{f_\theta (x)} \,  dx$ \\

Models with one parameter (ie. Bernulli):\\

$\mathcal{I}(\theta ) = \textsf{Var}(\ell '(\theta ))$\\

$\mathcal{I}(\theta ) = - \mathbf{E}(\ell ''(\theta ))$\\

Models with multiple parameters (ie. Gaussians):\\

$\mathcal{I}(\theta ) = -\mathbb E\left[\mathbf{H}\ell (\theta )\right]$\\

Cookbook:\\

Better to use 2nd derivative.\\

\begin{itemize}
  \item Find loglikelihood
  \item Take second derivative (=Hessian if multivariate)
  \item Massage second derivative or Hessian (isolate functions of $X_i$ to use with $- \mathbf{E}(\ell ''(\theta ))$ or $-\mathbb E\left[\mathbf{H}\ell (\theta )\right]$.
  \item Find the expectation of the functions of $X_i$ and subsitute them back into the Hessian or the second derivative. Be extra careful to subsitute the right power back. $\mathbb{E}[X_i] \neq \mathbb{E}[X_i^2]$.
  \item Don't forget the minus sign!
\end{itemize}

\subsection{Asymptotic normality of the maximum likelihood estimator}

Under certain conditions the MLE is asymptotically normal and consistent. This applies even if the MLE is not the sample average.

Let the true parameter $\theta^{*} \in \Theta$. Necessary assumptions:

\begin{itemize}
\item The parameter is identifiable
\item For all $\theta \in \Theta$, the support $\mathbb{P}_{\theta}$ does not depend on $\theta$ (e.g. like in $Unif(0,\theta)$);
\item $\theta^{*}$ is not on the boundary of $\Theta$;
\item Fisher information $\mathcal{I}(\theta)$  is invertible in the neighborhood of $\theta^{*}$
\item A few more technical conditions
\end{itemize}

The asymptotic variance of the MLE is the inverse of the fisher information.

$\sqrt(n)(\widehat{\theta }_ n^{\text {MLE}} - \theta^*) \xrightarrow[n \rightarrow \infty]{(d)} N_d(0,\mathcal{I}(\theta^* )^{-1})$\\


\lab{The Shooting Method for Boundary Value Problems}{The Shooting Method for Boundary Value Problems}
\label{lab:Shooting}

Consider a boundary value problem of the form
\begin{align}
	\label{shooting_bvp}
	\begin{split}
y'' &= f(x,y,y'), \quad a \leq x \leq b, \\
y(a) &= \alpha, \quad y(b) = \beta.
\end{split}
\end{align}
One natural way to approach this problem is to study the initial value problem (IVP) associated with this differential equation:
\begin{align}
	\label{shooting_ivp}
	\begin{split}
y'' &= f(x,y,y'), \quad a \leq x \leq b, \\
y(a) &= \alpha, \quad y'(a) = t.
	\end{split}
\end{align}

\begin{warn}
Here, do not mistake $t$ as an independent time variable.
It is a parameter for the boundary condition.
The prime $'$ notation represents the derivative with respect to $x$, which is an important distinction later in this section.
\end{warn}
The goal is to determine an  appropriate value $t$ for the initial slope, so that the solution of the IVP is also a solution of the boundary value problem.

Let $y(x,t)$ be the solution of \eqref{shooting_ivp}.
We can rewrite the initial value conditions as
\begin{align}
y(a, t) = \alpha, \quad y'(a, t) = t
\end{align}
We wish to find a value of $t$ so that $y(b,t) - \beta = 0$.
Applying Newton's method to the function $h(t) = y(b,t) - \beta$, we obtain the iterative method
\begin{align*}
	t_{n+1} &= t_n - \frac{ h(t_n)}{h'(t_n) }, \\
	&= t_n - \frac{ y(b,t_n) - \beta}{\frac{d}{dt} \left.y(b,t)\right|_{t_n} },\quad n = 0,1,\ldots .
\end{align*}

We recall that Newton's method requires a good initial guess $t_0$; a plausible initial guess would be the average rate of change of the solution across the entire interval, so that $t_0 =  (\beta - \alpha)/(b-a)$.
If this initial guess is not sufficient, the initial guess may be refined by looking at the solution $y(x,t_0)$ of the initial value problem.

This method requires us to evaluate or approximate the function $\frac{d}{dt} \left.y(b,t)\right|_{t_n}$.
This term may be approximated with a finite difference, giving us the iterative method
\[t_{n+1} = t_n - \frac{ (y(b,t_n) - \beta)(t_n - t_{n-1})}{y(b,t_n) - y(b,t_{n-1}) }, \,\, n = 1, 2,\hdots\]
This variation of the shooting algorithm is called the secant method, and requires two initial values instead of one.
Notice that finding $y(b,t_n)$ requires solving the initial value problem using RK4 or some other method.

For example, consider the boundary value problem
\begin{equation}
\begin{split}
\label{bvp1}
y'' &= -4y -9\sin(x), \,\, x \in [0,3\pi/4],\\
y(0) &= 1, \\
y(3 \pi/4) &= -\frac{1+3\sqrt{2}}{2}.
\end{split}
\end{equation}
The following code implements the secant method to solve \eqref{bvp1}.
Notice that \li{odeint} is the solver used for the initial value problems.

\lstinputlisting[style=FromFile]{secant_method.py}

% The following code employs the secant method to find the initial slope $y'(0) = t$ of a solution of \ref{bvp1}.
% The numerical solution may then be computed and plotted using \li{dopri5} and \li{matplotlib}.
% \ref{bvp1} has an exact solution $y(x) = \cos(2x) + (1/2)\sin(2x) -3\sin(x)$.

% Recall that the necessary dependencies are \li{numpy} and the \li{ode} class from \newline \li{scipy.integrate}.
% The first step to finding the initial slope $y'(0) = t$ is to define the ODE as a first order system of differential equations, and to define the necessary parameters:
% \begin{lstlisting}
% def ode_f(x,y): return numpy.array([y[1] , -4.*y[0] - 9.*numpy.sin(x)])
%
% a, b = 0., 3*numpy.pi/4.
% alpha, beta =  1., -(1.+3*numpy.sqrt(2))/2.
%
% \end{lstlisting}
% To calculate $y(b,t_0)$ we use the \li{ode} class:
% \begin{lstlisting}
% reltol, abstol = 1e-9,1e-8
% example = ode(ode_f).set_integrator('dopri5',atol=abstol,rtol=reltol)
% example.set_initial_value(np.array([alpha,t0]),a)
% y0 = example.integrate(b)[0]
% \end{lstlisting}

% \begin{problem} Use the secant method to solve the bvp
% \begin{equation*}
% \begin{split}
% y'' &= -4y -9\sin(x), \quad x \in [0,3\pi/4],\\
% y(0) &= 1, \\
% y(3\pi/4) &=-\frac{1+3\sqrt{2}}{2}.
% \end{split}
% \end{equation*}
% When you code the secant method, be sure to include a maximum number of iterations, as well as an appropriate stopping criteria.
% For example, if $|y(b,t_n)-\beta|<10^{-8},$ you have likely found a solution.
% \end{problem}

\begin{problem}
Appropriately defined initial value problems will usually have a unique solution.
Boundary value problems are not so straightforward; they may have no solution or they may have several.
You may have to determine which solution is physically interesting.
The following bvp has at least two solutions.
Using the secant method, find both numerical solutions and their initial slopes.
(Their plots are given in Figure \ref{prob:shooting1}.)
What initial values $t_0, t_1$ did you use to find them?
\begin{equation*}
\begin{split}
y'' &= -4y -9\sin(x), \,\, x \in [0,\pi],\\
y(0) &= 1, \\
y(\pi) &=1.
\end{split}
\end{equation*}

\begin{figure}[H]
\includegraphics[width=\textwidth]{Fig1.pdf}
\caption{Two solutions of $y'' = -4y -9\sin(x),$ both satisfying the boundary conditions $y(0) = y(\pi) = 1.$}
\label{prob:shooting1}
\end{figure}
\end{problem}

Let us consider how to solve for $\frac{d}{dt} y(b,t)$.
We will assume that the function $y(x,t)$ can be differentiated with respect to $x$ and $t$ in any order, and let  $z(x,t) = \frac{d}{dt} y(x,t).$
Using the chain rule, we obtain
\begin{eqnarray*}
z'' = \frac{d}{dt} y''(x,t) &=& \frac{\partial f}{\partial y} (x,y(x,t),y'(x,t)) \cdot \frac{dy}{dt}(x,t) ,\\
&+& \frac{\partial f}{\partial y'} (x,y(x,t),y'(x,t)) \cdot \frac{dy'}{dt}(x,t),
\end{eqnarray*}
Using the initial conditions associated with $y(x,t)$ and noting that $z(x, t) = \frac{d}{dt}y(x, t)$ and $z'(x, t) = \frac{d}{dt}y'(x, t)$, we obtain the following initial value problem for $z(x,t)$:
\begin{eqnarray*}
z'' &=& \frac{\partial f}{\partial y} (x,y,y') z + \frac{\partial f}{\partial y'} (x,y,y') z'
,\,\,a \leq x \leq b, \\
 z(a, t) &=& 0,\ z'(a, t) = 1.
\end{eqnarray*}

To use Newton's method, the (coupled) IVPs for $y$ and $z$ must be solved simultaneously.
The iterative method then becomes
\[t_{n+1} = t_n - \frac{ y(b,t_n) - \beta}{z(b,t_n)}, \,\, n = 0,1,\hdots\]

\begin{problem}
Use Newton's method to solve the BVP
\begin{equation*}
\begin{split}
y'' &= 3 + \frac{2y}{x^2}, \,\, x \in [1,e],\\
y(1) &= 6, \\
y(e) &= e^2 + 6/e.
\end{split}
\end{equation*}
Plot your solution.
(Compare with Figure \ref{prob:shooting2}.)
What is an appropriate initial guess?

Hint: Update the ode() function from the previous problem to solve for $y$, $y'$, $z$, $z'$ simultaneously.
This can be done by first rewriting the equations for $y"$ and $z"$ as a system of first order differential equations.

\begin{figure}[H]
\includegraphics[width=\textwidth]{Fig2.pdf}
\caption{The solution of  $y'' = 3 + 2y/x^2,$ satisfying the boundary conditions $y(1) = 6$, $ y(e) =  e^2 + 6/e$.}
\label{prob:shooting2}
\end{figure}
\end{problem}

\section*{The Cannon Problem}

Consider the problem of aiming a projectile at a given target.
Here we will construct a differential equation that describes the path of the projectile and takes into account air resistance.
We will then use the shooting method to determine the angle at which the projectile should be launched.

Let the coordinates of the projectile be given by $\vec{r}(t) = \langle x(t), y(t) \rangle$.
If $\theta(t)$ represents the angle of the velocity vector from the positive $x$-axis and $v(t)$ represents the speed of the projectile ($ |\vec{v}(t) |$), then we have
\begin{align*}
\dot{x} &= v\cos{\theta},\\
\dot{y} &= v\sin{\theta}.
%x' &= v\cos{\theta},\\
%y' &= v\sin{\theta}.
\end{align*} Note that each of $x,y,\theta$, and $v$ are functions of $t$, so the dot %prime
denotes $\dfrac{d}{dt}$.
The tangent vector to the path traced by the projectile is the unit vector in the direction of the projectile's velocity, so $\vec{T}(t) = \langle \cos{\theta}, \sin{\theta} \rangle$.
The unit normal vector $\vec{N} (t)$ is given by $\vec{N} (t)= \langle -\sin{\theta}, \cos{\theta} \rangle$.
Thus the relationship between basis vectors $\vec{i}, \vec{j}$, and $\vec{T(t)}, \vec{N}(t)$ is given by
\[\left[\begin{array}{cc}\cos{\theta} & \sin{\theta} \\-\sin{\theta} & \cos{\theta}\end{array}\right] \left[\begin{array}{c}\vec{i} \\\vec{j}\end{array}\right] = \left[\begin{array}{c}\vec{T(t)} \\\vec{N(t)}\end{array}\right]\]
Let $F_g$ represent the force on the projectile due to gravity, and $F_d$ represent the force on the projectile due to air resistance. (We assume the air is still.)
From Newton's law we have
\begin{align*}
m \dot{\vec{v}} &= F_g + F_d.
%m \vec{v}\ ' &= F_g + F_d.
\end{align*}

The drag equation from fluid dynamics says that the force on the projectile due to air resistance is $k v^2 = (1/2)\rho c_D A v^2$, where $\rho$ is the mass density of air (about $1.225$ $\text{kg}/\text{m}^3$), $v$ is the speed of the projectile, and $A$ is its cross-sectional area.
The drag coefficient $c_D$ is a dimensionless quantity that changes with respect to the shape of the object.
(If we assume our projectile is spherical with a diameter of $.2$ m, then its drag coefficient $c_D \approx 0.47$, its cross-sectional area is $\pi/100$ $ \text{m}^2$, and we obtain $k \approx 0.009$.)

Thus the total force on the shell is
\begin{align}
m \dot{\vec{v}} &= -mg \vec{j} - kv^2 \vec{T},\nonumber \\
%m \vec{v}\ ' &= -mg \vec{j} - kv^2 \vec{T},\nonumber \\
&= -mg( \sin{\theta} \vec{T} + \cos{\theta} \vec{N} ) - kv^2 \vec{T},\nonumber\\
&= (-mg \sin{\theta} - k v^2 ) \vec{T} - mg \cos{\theta} \vec{N}.\label{eqn:TForce1}
\end{align}
From the identity
$\vec{v} = \langle \dot{x}, \dot{y} \rangle = \langle v \cos{\theta}, v \sin{\theta} \rangle$
%$\vec{v} = \langle x', y' \rangle = \langle v \cos{\theta}, v \sin{\theta} \rangle$
we have
\begin{align}
m \dot{\vec{v}} = {} & m\langle \dot{v} \cos{\theta} - v\sin{\theta} \cdot \dot{\theta} ,\dot{v}\sin{\theta} + v\cos{\theta} \cdot \dot{\theta} \rangle \nonumber \\
= {} & m(\dot{v}\cos{\theta} - v\sin{\theta} \cdot \dot{\theta})(\cos{\theta} \vec{T} - \sin{\theta}\vec{N}) \nonumber \\
& + m(\dot{v} \sin{\theta} + v\cos{\theta} \cdot \dot{\theta})( \sin{\theta} \vec{T} + \cos{\theta} \vec{N}) ,  \nonumber \\
= {} & m(\vec{T} \cdot \dot{v} + \vec{N} \cdot v \cdot \dot{\theta}) .\label{eqn:TForce2}
%
%m \vec{v}' = {} & m\langle v' \cos{\theta} - v\sin{\theta} \cdot \theta' ,v'\sin{\theta} + v\cos{\theta} \cdot \theta' \rangle \nonumber \\
%= {} & m(v'\cos{\theta} - v\sin{\theta} \cdot \theta')(\cos{\theta} \vec{T} - \sin{\theta}\vec{N}) \nonumber \\
%& + m(v' \sin{\theta} + v\cos{\theta} \cdot \theta')( \sin{\theta} \vec{T} + \cos{\theta} \vec{N}) ,  \nonumber \\
%= {} & m(\vec{T} \cdot v' + \vec{N} \cdot v \cdot \theta') .\label{eqn:TForce2}
\end{align}
From equations \eqref{eqn:TForce1} and \eqref{eqn:TForce2} we have
\begin{align*}
%m v' &= -mg\sin{\theta} - k v^2,\\
m \dot{v} &= -mg\sin{\theta} - k v^2,\\
mv\dot{\theta} &= -mg \cos{\theta}.
%mv\theta' &= -mg \cos{\theta}.
\end{align*}
Thus we have the coupled system of differential equations
\begin{align}
\dot{x} &= v\cos{\theta}, \nonumber \\
\dot{y} &= v\sin{\theta},\nonumber \\
\dot{v} &= -g\sin{\theta} -  k v^2/m,\nonumber \\
\dot{\theta} &= -g \cos{\theta}/v. \nonumber
%x' &= v\cos{\theta}, \nonumber \\
%y' &= v\sin{\theta},\nonumber \\
%v' &= -g\sin{\theta} -  k v^2/m,\nonumber \\
%\theta' &= -g \cos{\theta}/v. \nonumber
\end{align}

The independent variable $t$ used above is unessential to our problem.
If we assume that $t$ is an smooth invertible function of $x$ ($t = t(x)$), then we obtain
\begin{align*}
\frac{dy}{dx} &= \frac{dy}{dt}\frac{dt}{dx} ,\\
&= \frac{dy}{dt} \frac{1}{v\cos{\theta}}, \\
&= \frac{v \sin{\theta}}{v\cos{\theta}} = \tan{\theta}.
\end{align*}
We find $\frac{dv}{dx}$ and $\frac{d\theta}{dx}$ in a similar manner.
Thus our system of differential equations becomes
\begin{align}
	\begin{split}
\frac{dy}{dx} &= \tan {\theta} ,\\
\frac{dv}{dx} &= -\frac{g \sin{\theta} + \mu v^2}{v \cos{\theta}},\\
\frac{d\theta}{dx} &= -\frac{g}{v^2}, \label{eqn:cannon_DEs}
	\end{split}
\end{align}
where $\mu = k/m.$
In the next problem we will assume that the projectile has a mass of about $60$ kg, so that $\mu \approx .0003$.

\begin{problem}
Suppose a projectile is fired from a cannon with velocity $45\text{ m/s}^2$.
At what angle $\theta(0)$ should it be fired to land at a distance of $195\text{ m}$?

There should be two initial angles $\theta(0)$ that produce a solution for this bvp.
Use the secant method to numerically compute and then plot both trajectories.
\begin{align}
	\label{eqn:cannon_shooting}
	\begin{split}
\frac{dy}{dx} &= \tan {\theta} ,\\
\frac{dv}{dx} &= -\frac{g \sin{\theta} + \mu v^2}{v \cos{\theta}},\\
\frac{d\theta}{dx} &= -\frac{g}{v^2},\\
y(0)&= y(195) = 0,\\
v(0) &= 45 \text{ m/s}^2
	\end{split}
\end{align}
($g = 9.8067\text{ m/s}^2$.)
Find both solutions for this boundary value problem when $\mu = .0003$.
Compare with the solutions when $\mu = 0.$
Their graphs are given in Figure \ref{fig:shooting_cannon_comparison2}.

Hint: This is a system of three first order differential equations, and so our secant method requires a slight modification.
Keeping in mind that the unknown initial condition is $\theta(0)$, not $y'(0)$, define an appropriate function $h(t)$.
\end{problem}

\begin{figure}
\includegraphics[width=\textwidth]{Cannon_with_AirResistance.pdf}
\caption{Two solutions of the system of equations \eqref{eqn:cannon_DEs}, both with initial conditions  $y(0) = 0 \text{ m}$, $ v(0) = 45 \text{ m/s}$, and $\theta(0)=\pi/3$.
The black curve is the trajectory of a projectile immune to air resistance ($\mu = 0$).
The red curve describes the trajectory of a more realistic projectile ($\mu = .0003$).}
\label{fig:shooting_cannon_comparison1}
\end{figure}

\begin{figure}
\includegraphics[width=\textwidth]{Cannon_Shooting.pdf}
\caption{Two solutions of the boundary value problem \eqref{eqn:cannon_shooting} when the air resistance is described by the parameter $\mu = .0003$.
Also both solutions when air resistance is not described in the model ($\mu = 0$).}
\label{fig:shooting_cannon_comparison2}
\end{figure}


\lab{Algorithms}{Complex Numbers}{Complex Numbers}
\label{Lab:complex_intro}

\objective{Learn to perform basic computation and visualization with complex numbers.}

\begin{comment}
\section*{Arithmetic with complex numbers}
Computationally, complex numbers are really just a pair of floating point numbers.
The first is understood to represent the real part of the complex number.
The second is understood to represent the imaginary part.
In Python, a complex number $a + b i$ can be defined using either of the following methods
\begin{lstlisting}
complex(a, b)
a + 1.0j * b
\end{lstlisting}
Python lets us define purely imaginary numbers using the syntax shown there with \li{j}.
As another example, $2 i$ would be \li{2.0j}.

Python also includes the built in \li{cmath} library.
This library provides basic math functions for complex numbers.
For examle, the functions \li{polar} and \li{rect} can be used to convert to and from polar coordinates (discussed below).
It also includes basic functions like \li{sin}, \li{cos}, etc. with their domains extended to the complex numbers.

There are some computational advantages and disadvantages to using complex numbers.
Addition or subtraction between two complex numbers now requires two floating point operations (one for each part of the number), so it takes longer.
Multiplication and division are more costly.
Mutliplication is easily seen to take 4 floating point multiplies and two floating point adds instead of just a single floating point multiply.
The fact that there two numbers are stored also increases the amount of floating point error that can be created in each computation.

Most of the basic functions in NumPy support complex numbers.
Fore example, \li{numpy.absolute} will take the absolute value of a complex number without any trouble.
You can also acces the real and imaginary parts of a complex array via the \li{real} and \li{imag} attributes.
For example,
\begin{lstlisting}
import numpy as np
from numpy.random import rand
Z = rand(10) + 1.0j * rand(10)
# print real part
print Z.real
# print imaginary part
print Z.imag
\end{lstlisting}

The complex conjugate of a complex number or array can be obtained using the \li{conjugate} method.
For example, if \li{A} is a complex number or complex array, its conjugate (or elementwise conjugate) is obtained via
\begin{lstlisting}
A.conjugate()
\end{lstlisting}
\end{comment}

\section*{Polar Representation of Complex Numbers}

One of the most important results in Complex Analysis is Euler's Formula.
It states that:

\[e^{i\theta}=cos(\theta)+i sin(\theta)\]

One way to derive this important result is to consider the taylor series expansion of each of the functions involved.

From this formula, we can see that $e^x$ maps the imaginary axis onto the unit circle on the complex plane.
From our knowledge of the sine and cosine functios, we can also know that this mapping has a period of $2\pi$.
From here, notice that we may represent any number on the complex plane in the form $r e^{i\theta}$ for $r\geq 0$ and $0 \leq \theta \leq 2\pi$.
This is what is known as the polar representation of a complex number.
Conceptually, it is \emph{identical} to the polar representation of points in the standard cartesian plane.
The primary difference is that, here we define a map from $\mathbb{C}$ to the polar coordinates instead of from $\mathbb{R}^{2}$ to the polar coordinates.
The number $\theta$ is known as the argument, or phase, of a complex number.
The number $r$ is known as the modulus, magnitude, or absolute value of a complex number.
In the complex plane we define $|z|$ as the modulus of $z$ and $arg(z)$ as the argument of $z$.
To get the argument and the modulus for an array of complex numbers you can use the functions \li{np.angle} and \li{np.absolute} respectively.

\section*{Visualization of Complex Functions}
Functions that map the complex plane to itself cannot be visualized  the same ways we usually visualize functions that map $\mathbb{R}$ to itself.
Since $\mathbb{C}$ is isomorphic to $\mathbb{R}^2$, visualizing a mapping from the complex numbers to themselves is the same as vizualizing a mapping from $\mathbb{R}^2 \to \mathbb{R}^2$.
This gives a hint at how we can go about visualizing complex valued functions.
A complex valued plot can be thought of as two separate maps from $\mathbb{R}^2 \to \mathbb{R}$.
The $arg(z)$ maps the function to a color.  The $|z|$ maps the number to the hue. The picture is what the identity map maps to Figure \ref{fig:identity}

\begin{figure}
\includegraphics[width=\textwidth]{Identity.png}
\caption{Color plot of the identity map.}
\label{fig:identity}
\end{figure}

The colorize function below takes in an array of complex values and outputs the RBG colors for each point.

\begin{lstlisting}
import numpy as np
import matplotlib.pyplot as plt
from colorsys import hls_to_rgb

def colorize(z):
    zy=np.flipud(z)
    r = np.abs(zy)
    arg = np.angle(zy)

    h = (arg + np.pi)  / (2 * np.pi) + 0.5
    l = 1.0 - 1.0/(1.0 + r**0.3)
    s = 0.8

    c = np.vectorize(hls_to_rgb) (h,l,s) # --> tuple
    c = np.array(c)  # -->  array of (3,n,m) shape, but need (n,m,3)
    c = c.swapaxes(0,2)
    c = c.swapaxes(0,1)
    return c
\end{lstlisting}

The following code uses the colorize function to plot  $\frac{x^2 - 1}{x}$ see \ref{fig:example} for what the output looks like.

\begin{lstlisting}
def f(z):
	return (z**2-1)/z
xbounds=(-.5, 1.5)
ybounds=(-1, 1)
res=401
x = np.linspace(xbounds[0],xbounds[1], res)
y = np.linspace(ybounds[0],ybounds[1], res)
X,Y = np.meshgrid(x,y)
Z=f(X+Y*1j)
Zc=colorize(Z)
plt.imshow(Zc,extent=(xbounds[0],xbounds[1],ybounds[0],ybounds[1]))
plt.show()
\end{lstlisting}

\begin{figure}
\includegraphics[width=\textwidth]{example.png}
\caption{Colorplot of the  function $\frac{x^2 - 1}{x}$.}
\label{fig:example}
\end{figure}

\begin{problem}
Write a function called \li{plot_complex} following the code above. Have it take in a function, xbounds, ybounds and a resolution.
\end{problem}

From these plots you can see the poles and zeros. The zeros are the black dots and the poles are the white dots on the plots. the order of the pole or zero can be read from the number of times the full set of colors show up around the pole or zero. As you can see from Figure \ref{fig:funcplot} $x^2-1$ has two zeros of order one at $1$ and $-1$. $x^4-\frac{1}{x^4}$ has eight zeros of order one and a pole of order four.
For essential singularities the colors circle outward like in \ref{fig:e}, the plot of $e^\frac{1}{z}$.

\begin{figure}
\begin{subfigure}{.49\textwidth}
\includegraphics[width=\textwidth]{function.png}
\end{subfigure}
\begin{subfigure}{.49\textwidth}
\includegraphics[width=\textwidth]{function2.png}
\end{subfigure}
\caption{Colorplot of the  function $x^2 - 1$ and $x^4-\frac{1}{x^4}$.}
\label{fig:funcplot}
\end{figure}

\begin{figure}
\includegraphics[width=\textwidth]{function1.png}
\caption{Color plot of the $e^\frac{1}{z}$.}
\label{fig:e}
\end{figure}

\begin{problem}

Plot the following functions and look at the plots. Write a function that prints out an estimate of the roots and/or poles and their order.
\begin{itemize}
\item Plot $e^z$ from $x \in [-8,8]$ and $y \in [-8,8]$, \item $z^2-2z^7+2z^6-4z^5+2z^4-2z^3-5z^2+4z-4$ from $x \in [-2.5,2.5]$ and $y \in [-2.5,2.5]$
\item Plot $\frac{16z^4+32z^3+32z^2+16z+4}{16z^4-16z^3+5z^2}$ from $x \in [-1,1]$ and $y \in [-1,1]$,
\item Plot $\sin{\frac{1}{z}}$ from $x \in [-.8,.8]$ and $y \in [-.8,.8]$,
\end{itemize}

\end{problem}

\section*{Multi-Valued Functions}

Another important topic in Complex Analysis is the study of multiple valued functions.
These functions arise as we consider the inverses of functions that are not strictly one to one on the complex plane.
A classic example is $\sqrt{x}$, which may take two values for every nonzero point of the complex plane.

In the Real numbers we worked with functions like this by simply restricting their output on a certain domain.
We can do a similar thing in the Complex plane.
Loosely speaking, such a restriction is called a branch.
Computationally we restrict the output to a single portion of the actual possible values of the multifunction.
We call inverse functions that have multiple values like this ``multi-valued functions" or ``multifuctions."
Numpy automatically restricts the output of multi-valued functions. So to get other cuts you have to modify the function to get the cut you want, like multiplying the output of the $\sqrt{x}$ by $-1$.
Figure \ref{fig:sqrt} shows the two cuts surfaces for $\sqrt{z}$ in the complex plane.

\begin{figure}
\begin{subfigure}{.49\textwidth}
\includegraphics[width=\textwidth]{possqrt.png}
\end{subfigure}
\begin{subfigure}{.49\textwidth}
\includegraphics[width=\textwidth]{negsqrt.png}
\end{subfigure}
\caption{The positive and negative cut respectiviely of the function $\sqrt{z}$.}
\label{fig:sqrt}
\end{figure}

These are some very basic examples.
Another simple example is $\ln\left(z\right)$ which has a single value for the real part and infinitely many possible values for its imaginary part.
This is because for any complex $z\neq 0$, we have $e^z=e^{z+2n\pi}$ where $n$ is any integer.

\begin{problem}
Plot three cuts of $\ln\left(z\right)$ and$\arctan{x}$ on $x \in [-2,2]$ and $y \in [-2,2]$
\end{problem}

All three of these functions can be made analytic at almost any point, except at their singularities, but that depends on how we cut the domain to give it a single value.
When Integrating such functions, be careful about integrating across such cuts in the domain.

\begin{comment}
\begin{problem}
Write two functions, both accepting accept a natural number $n$.
Have one function plot the riemann surface for the real part $f(z)=\sqrt[n]{z}$ and the other plot the imaginary part.

Hint: Convert $z$ in $f(z)=\sqrt[n]{z}$ to polar form as $z=re^{\theta + 2k\pi}$
If you then plug this into $\sqrt[n]{z}$ the function takes the form
\[f(z)=\sqrt[n]{r} e^{i \frac{\theta + 2 \pi k}{n}}\]
Notice that here $f(z)$ has distinct values for $k = 0, 1, \dots, n-1$ a total of $n$ different values). Each value of $k$ corresponds to a different branch, which you can plot as $n$ separate surfaces.

If you use just one surface to plot each branch you will get erroneous vertical lines from jump discontinuities. Split each branch into two surfaces to get rid of these lines. You can investigate where the discontinuities occur by first plotting it as just one surface. The discontinuities happen at the same place for all $n$;
\end{problem}
\end{comment}

\section*{Contour Integrals in the Complex Plane}

From multivariable calculus, you may recall that an integral may be taken along a path.
This is very similar to what we can be done in the complex plane.
Consider the function $f(z)$ on the complex plane.
Let $z=x+iy$.
Let $u$ and $v$ be the real and imaginary parts of $f$ respectively.
We integrate $f$ along some contour $C$ in the complex plane, beginning at $z=a$ and ending at $z=b$.
This integral may be written
\[\int_c f(z)dz\]
Parameterizing $z$, we have
\[\int_a^b f\left( c\left(t\right)\right) c'\left(t\right) dt\]
Expanding into real and imaginary parts (where $c\left(t\right) = x\left(t\right) + i y\left(t\right)$), we have
\[\int_a^b \left(u \left(c \left(t\right)\right) x'\left(t\right)-v\left(c \left(t\right)\right) y'\left(t\right)\right) dt + i \int_a^b\left(v \left(c \left(t\right)\right)x'\left(t\right)+u\left(c \left(t\right)\right) y'\left(t\right)\right) dt\]
We have now written this complex integral as the sum of two real valued integrals in $\mathbb{R}$.
Note that this implies that $\int_C f(z) dz$ may depend on the contour we choose and not just on the endpoints $a$ and $b$.

\begin{problem}
Write a function which takes a complex function $f(z)$, a contour parameterization $c(t)$ of a contour $c$, and the integration bounds on $t$ and returns the integral of $f$ along the contour $c$.
Use the numerical integration function \li{sympy.mpmath.quad} and the numerical derivative function \li{sympy.mpmath.diff} included in mpmath (which is, in turn, included as a submodule of sympy).
These functions already work for complex numbers.
To do something similar with the integration routines in SciPy, we would have to separate the function into real and imaginary parts, as is shown above.

Using the function you just defined, integrate the following functions along the following contours
\begin{itemize}
\item $\bar{z}$ counterclockwise along the unit ball starting and ending at $1$
\item $\bar{z}$ along a straight line from $0$ to $1+i$
\item $\bar{z}$ along the real axis from $0$ to $1$, then along the line from $1$ to $1+i$
\item $\bar{z}$ along the unit ball centered at $i$ from $0$ to $1+i$
\item $e^z$ counterclockwise along the unit ball starting and ending at $1$
\item $e^z$ along a straight line from $0$ to $1+i$
\item $e^z$ along the real axis from $0$ to $1$, then along the line from $1$ to $1+i$
\item $e^z$ along the unit ball centered at $i$ from $0$ to $1+i$
\end{itemize}
\end{problem}

Notice that, for a holomorphic function on a simply connected domain, the integrals from one point to another are not path dependent for any contours that lie within the domain.
An immediate consequence of the theorem is that for a complex function $f$, holomorphic on a simply connected domain $D$, and a contour $C$ lying entirely within $D$ which begins and ends at some point $a\in D$,
\[\int_C f(z)dz=0\]

The quadrature algorithms used in many of the integration algorithms work along a straight line between the integration bounds in the complex plane, so for holomorphic functions we should be able to use the integration function we wrote earlier.
For example, integrating $e^z$ from $-1-i$ to $1+i$ can be done numerically like this:
\begin{lstlisting}
from sympy import mpmath as mp
mp.quad(lambda z: mp.exp(z), (complex(-1, -1), complex(1, 1)))
\end{lstlisting}

\section*{The Cauchy Integral Formula}

Another major theorem in complex analysis is called Cauchy's Integral Formula (not to be confused with Cauchy's Integral Theorem).
It states that for a domain $D$ in the complex plane, containing some contour $C$ and the interior of $C$, for any $z_0$ in the interior of $C$,
\[f(z_0)=\frac{1}{2\pi i} \int_C \frac{f(z)}{z-z_0} dz\]

With more work, this theorem can be used to show that any function $f$ holomorphic on some domain $D$ is also infinitely differentiable on that domain.
In fact, the $n$th derivative of $f$ is given by the formula
\[f^{(n)}(z_0) = \frac{n!}{2\pi i} \int_C \frac{f(z)}{(z-z_0)^{n+1}} dz\]
This result is also important because it allows us to relate the value of $f$ on the inside of a contour to the value of $f$ on the contour itself.
In other words, the values of $f$ inside the contour depend only on the values of $f$ along the contour itself.
A related theorem (the Morera theorem) states that if some function $f$ is continuous on a domain $D$ and for every contour beginning and ending at the same point, the formula $\int_C f(z) dz = 0$ holds, then $f$ is holomorphic on $D$.

\begin{problem}
Using Cauchy's Integral Formula, write a python function which returns a callable function which evaluates a complex function $f$ along the interior of a contour $C$.
It should accept a callable function for the paramaterization of $C$, a callable function for the values of $f$ along $C$, and the bounds on the parameter used.
Assume in your function that $C$ begins and ends at the same point and that $f$ also begins and ends at the same value (so that $f$ is continous along $C$)
Try it out on simple functions like $e^x$ with complex values and compare what you get with what calling the functions normally gives you.
\end{problem}

Notice that in Cauchy's Integral Formula, we are integrating along a contour that begins and ends at the same point.
The function is also holomorphic at every point except $z_0$. At $z_0$ the integrand is undefined and has a singularity.
This integral around a singularity has some useful properties.
We will discuss these properties later on.


 \chapter{Continuous Latent Variables}
 \label{chap:Continuous Latent Variables}
 \section{Principal Component Analysis}
 \subsection{Introduction}
 Principal Component Analysis is widely used for applications such as dimensionality reduction,lossy data compression,feature extraction,and data visualization.Also known as the Karhunen-Loeve transform.There are two definitions giving rise to the same algorithm.PCA can be defined as the orthogonal projection of the data onto a lower dimensional linear space,known as the principal subspace,such that the variance of the projected data  is maximized.Equivalently,it can be defined as the linear projection the minimizes the average projection cost,
defined as  the linear projection that minimizes the average projection cost,defined as the mean squared distance between the data points and their projections.
\subsection{Maximum variance formulation}
Consider a data set of observations $\{x_n\}$ where $n = 1,...,N$,and $x_n$ is a Euclidean variable with dimensionality D.
Our goal is to project the data onto a space having dimensionality $M < D$ while maximizing the variance of the projected
data.We define the direction of this space using a D-dimensional unit vector $\mathbf{u_1^T}\mathbf{u_1} = 1$.Each
data point $\mathbf{x_n}$ is then projected onto a scalar value $\mathbf{u_1^T}\mathbf{x_n}$.The mean of the projected
data is $\mathbf{u_1^T}\bar{\mathbf{x}}$ where the $\bar{\mathbf{x}}$ is the sample set mean given by
\begin{align}
\bar{\mathbf{x}} = \frac{1}{N}\sum_{n=1}^{N}{\mathbf{x_n}}
\end{align}
and the variance of the projected data is given by
\begin{align}
\frac{1}{N}\sum_{n=1}^{N}\{\mathbf{u_1^T}\mathbf{x_n} - \mathbf{u_1^T}\bar{\mathbf{x}}\}^2
&= \frac{1}{N}\sum_{n=1}^{N}{\{\mathbf{u_1^T}(\mathbf{x_n} - \bar{\mathbf{x}})\}^2} \\
&= \frac{1}{N}\sum_{n=1}^{N}{\{\mathbf{u_1^T(\mathbf{x_n - \bar{\mathbf{x}}})(\mathbf{x_n -\bar{x}})^T\mathbf{u_1^T} }  \}} \\
&= \mathbf{u_1^T}\mathbf{S}\mathbf{u_1}
\end{align}
where $\mathbf{S}$ is the data covariance matrix defined by
\begin{align}
\mathbf{S} = \frac{1}{N}\sum_{n=1}^{N}(\mathbf{x_n}-\bar{\mathbf{x}})(\mathbf{x_n}-\mathbf{\bar{x}})^T
\end{align}
We now maximize the projected variance $\mathbf{u_1^T}\mathbf{S}\mathbf{u_1}$ with respect to $\mathbf{u_1}$,which is a
constrained maximization to prevent $\parallel\mathbf{u_1}\parallel\rightarrow \infty$ .The appropriate constraint
comes from the normalization condition $\mathbf{u_1^T}\mathbf{u_1}=1$.To enforce this constraint,we introduce a
Lagrange multiplier that we shall denote by $\lambda_1$,and then make an unconstrained maximization of
\begin{equation}
\mathbf{u_1^T}\mathbf{S}\mathbf{u_1} + \lambda_1(1-\mathbf{u_1^T}\mathbf{u_1})
\end{equation}
By setting the derivative with respect to $\mathbf{u_1}$ equal to zero,we see that this quantity will have a stationary
point when
\begin{equation}
\mathbf{S}\mathbf{u_1} = \lambda_1\mathbf{u_1}
\end{equation}
which says that $\mathbf{u_1}$ must be an eigenvector of $\mathbf{S}$.If we left-multiply by $\mathbf{u_1^T}$ and make use
of $\mathbf{u_1^T}{u_1} = 1$,we see that the variance is given by
\begin{equation}
\mathbf{u_1^TSu_1} = \lambda_1
\end{equation}
and so the variance will be a maximum when we set $\mathbf{u_1}$ equal to the eigenvector having the largest
eigenvalue $\lambda_1$.This eigenvector is known as the first principal component.

\subsection{Minimum-error formulation}

\subsection{Applications of PCA}
\subsection{PCA for high-dimensional data}

\section{Probabilistic PCA}

\section{Kernel PCA}

\section{Nonlinear Latent Variable Models}