\chapter{Latent linear models}

\section{Factor analysis}
One problem with mixture models is that they only use a single latent variable to generate the observations. In particular, each observation can only come from one of $K$ prototypes. One can think of a mixture model as using $K$ hidden binary variables, representing a one-hot encoding of the cluster identity. But because these variables are mutually exclusive, the model is still limited in its representational power.

An alternative is to use a vector of real-valued latent variables,$\vec{z}_i \in \mathbb{R}^L$. The simplest prior to use is a Gaussian (we will consider other choices later):
\begin{equation}\label{eqn:FA-prior}
p(\vec{z}_i)=\mathcal{N}(\vec{z}_i|\vec{\mu}_0,\vec{\Sigma}_0)
\end{equation}
If the observations are also continuous, so $\vec{x}_i \in \mathbb{R}^D$, we may use a Gaussian for the likelihood. Just as in linear regression, we will assume the mean is a linear function of the (hidden) inputs, thus yielding
\begin{equation}\label{eqn:FA-class-conditional-density}
p(\vec{x}_i|\vec{z}_i,\vec{\theta})=\mathcal{N}(\vec{x}_i|\vec{W}\vec{z}_i+\vec{\mu},\vec{\Psi})
\end{equation}
where \vec{W} is a $D \times L$ matrix, known as the \textbf{factor loading matrix}, and $\vec{\Psi}$ is a $D \times D$ covariance matrix. We take $\vec{\Psi}$ to be diagonal, since the whole point of the model is to “force” $\vec{z}_i$ to explain the correlation, rather than “baking it in” to the observation’s covariance. This overall model is called \textbf{factor analysis} or \textbf{FA}. The special case in which $\vec{\Psi}=\sigma^2\vec{I}$ is called \textbf{probabilistic principal components analysis} or \textbf{PPCA}. The reason for this name will become apparent later.

\subsection{FA is a low rank parameterization of an MVN}
FA can be thought of as a way of specifying a joint density model on $\vec{x}$ using a small number of parameters. To see this, note that from Equation \ref{eqn:Linear-Gaussian-system-normalizer}, the induced marginal distribution $p(\vec{x}_i|\vec{\theta})$ is a Gaussian:
\begin{align}
p(\vec{x}_i|\vec{\theta}) & = \int \mathcal{N}(\vec{x}_i|\vec{W}\vec{z}_i+\vec{\mu},\vec{\Psi})\mathcal{N}(\vec{z}_i|\vec{\mu}_0,\vec{\Sigma}_0)\mathrm{d}\vec{z}_i \nonumber \\
 & = \mathcal{N}(\vec{x}_i|\vec{W}\vec{\mu}_0+\vec{\mu},\vec{\Psi}+\vec{W}\vec{\Sigma}_0\vec{W})
\end{align}
From this, we see that we can set $\vec{\mu}_0=0$ without loss of generality, since we can always absorb $\vec{W}\vec{\mu}_0$ into $\vec{\mu}$. Similarly, we can set $\vec{\Sigma}_0=\vec{I}$ without loss of generality, because we can always “emulate” a correlated prior by using defining a new weight matrix, $\tilde{\vec{W}}=\vec{W}\vec{\Sigma}_0^{-\frac{1}{2}}$. So we can rewrite Equation \ref{eqn:FA-prior} and \ref{eqn:FA-class-conditional-density} as:
\begin{align}
p(\vec{z}_i)&=\mathcal{N}(\vec{z}_i|\vec{0},\vec{I}) \\
p(\vec{x}_i|\vec{z}_i,\vec{\theta})&=\mathcal{N}(\vec{x}_i|\vec{W}\vec{z}_i+\vec{\mu},\vec{\Psi})
\end{align}

We thus see that FA approximates the covariance matrix of the visible vector using a low-rank decomposition:
\begin{equation}\label{eqn:FA-prior}
\vec{C} \triangleq \mathrm{cov}[\vec{x}]=\vec{W}\vec{W}^T+\vec{\Psi}
\end{equation}
This only uses $O(LD)$ parameters, which allows a flexible compromise between a full covariance Gaussian, with $O(D^2)$ parameters, and a diagonal covariance, with $O(D)$ parameters. Note that if we did not restrict $\vec{\Psi}$ to be diagonal, we could trivially set $\vec{\Psi}$ to a full covariance matrix; then we could set $\vec{W}=0$, in which case the latent factors would not be required.

\subsection{Inference of the latent factors}
\begin{align}
p(\vec{z}_i|\vec{x}_i,\vec{\theta}) & = \mathcal{N}(\vec{z}_i|\vec{\mu}_i,\vec{\Sigma}_i) \\
\vec{\Sigma}_i & \triangleq (\vec{\Sigma}_0^{-1}+\vec{W}^T\vec{\Psi}^{-1}\vec{W})^{-1} \\
               & =(\vec{I}+\vec{W}^T\vec{\Psi}^{-1}\vec{W})^{-1} \\
\vec{\mu}_i & \triangleq \vec{\Sigma}_i[\vec{W}^T\vec{\Psi}^{-1}(\vec{x}_i-\vec{\mu})+\vec{\Sigma}_0^{-1}\vec{\mu}_0] \\
            & =\vec{\Sigma}_i\vec{W}^T\vec{\Psi}^{-1}(\vec{x}_i-\vec{\mu})
\end{align}
Note that in the FA model, $\vec{\Sigma}_i$ is actually independent of $i$, so we can denote it by $\vec{\Sigma}$. Computing this matrix takes $O(L^3+L^2D)$ time, and computing each $\vec{\mu}_i=\mathbb{E}[\vec{z}_i|\vec{x}_i,\vec{\theta}]$ takes $O(L^2+LD)$ time. The $\vec{\mu}_i$ are sometimes called the \textbf{latent scores}, or \textbf{latent factors}.

\subsection{Unidentifiability}
Just like with mixture models, FA is also unidentifiable. To see this, suppose $\vec{R}$ is an arbitrary orthogonal rotation matrix, satisfying $\vec{R}\vec{R}^T=\vec{I}$. Let us define $\tilde{\vec{W}}=\vec{W}\vec{R}$, then the likelihood function of this modified matrix is the same as for the unmodified matrix, since $\vec{W}\vec{R}\vec{R}^T\vec{W}^T+\vec{\Psi}=\vec{W}\vec{W}^T+\vec{\Psi}$. Geometrically, multiplying $\vec{W}$ by an orthogonal matrix is like rotating $\vec{z}$ before generating $\vec{x}$.

To ensure a unique solution, we need to remove $L(L-1)/2$ degrees of freedom, since that is the number of orthonormal matrices of size $L \times L$.\footnote{To see this, note that there are $L-1$ free parameters in $\vec{R}$ in the first column (since the column vector must be normalized to unit length), there are $L-2$ free parameters in the second column (which must be orthogonal to the first), and so on.} In total, the FA model has $D+LD-L(L-1)/2$ free parameters (excluding the mean), where the first term arises from $\vec{\Psi}$. Obviously we require this to be less than or equal to $D(D+1)/2$, which is the number of parameters in an unconstrained (but symmetric) covariance matrix. This gives us an upper bound on $L$, as follows:
\begin{equation}
L_{\mathrm{max}}=\lfloor D+0.5(1-\sqrt{1+8D}) \rfloor
\end{equation}

For example, $D=6$ implies $L \leq 3$. But we usually never choose this upper bound, since it would result in overfitting (see discussion in Section \ref{sec:FA-Choosing-L} on how to choose $L$).

Unfortunately, even if we set $L < L_{\mathrm{max}}$, we still cannot uniquely identify the parameters, since the rotational ambiguity still exists. Non-identifiability does not affect the predictive performance of the model. However, it does affect the loading matrix, and hence the interpretation of the latent factors. Since factor analysis is often used to uncover structure in the data, this problem needs to be addressed. Here are some commonly used solutions:
\begin{itemize}
\item{\textbf{Forcing $\vec{W}$ to be orthonormal} Perhaps the cleanest solution to the identifiability problem is to force $\vec{W}$ to be orthonormal, and to order the columns by decreasing variance of the corresponding latent factors. This is the approach adopted by PCA, which we will discuss in Section \ref{sec:PCA}. The result is not necessarily more interpretable, but at least it is unique.}
\item{\textbf{Forcing $\vec{W}$ to be lower triangular} One way to achieve identifiability, which is popular in the Bayesian community (e.g., (Lopes and West 2004)), is to ensure that the first visible feature is only generated by the first latent factor, the second visible feature is only generated by the first two latent factors, and so on. For example, if $L=3$ and $D=4$, the correspond factor loading matrix is given by
\begin{equation*}
\vec{W}=\left(\begin{array}{ccc}
w_{11} & 0 & 0 \\
w_{21} & w_{22} & 0 \\
w_{31} & w_{32} & w_{33} \\
w_{41} & w_{32} & w_{43}
\end{array}\right)
\end{equation*}
We also require that $w_{jj} >0$ for $j =1:L$. The total number of parameters in this constrained matrix is $D+DL-L(L-1)/2$, which is equal to the number of uniquely identifiable parameters. The disadvantage of this method is that the first $L$ visible variables, known as the \textbf{founder variables}, affect the interpretation of the latent factors, and so must be chosen carefully.}
\item{\textbf{Sparsity promoting priors on the weights} Instead of pre-specifying which entries in $\vec{W}$ are zero, we can encourage the entries to be zero, using $\ell_1$ regularization (Zou et al. 2006), ARD (Bishop 1999; Archambeau and Bach 2008), or spike-and-slab priors (Rattray et al. 2009). This is called sparse factor analysis. This does not necessarily ensure a unique MAP estimate, but it does encourage interpretable solutions. See Section 13.8 TODO.}
\item{\textbf{Choosing an informative rotation matrix} There are a variety of heuristic methods that try to find rotation matrices $\vec{R}$ which can be used to modify $\vec{W}$(and hence the latent factors) so as to try to increase the interpretability, typically by encouraging them to be (approximately) sparse. One popular method is known as \textbf{varimax}(Kaiser 1958).}
\item{\textbf{Use of non-Gaussian priors for the latent factors} In Section \ref{sec:ICA}, we will dicuss how replacing $p(\vec{z}_i)$ with a non-Gaussian distribution can enable us to sometimes uniquely identify $\vec{W}$ as well as the latent factors. This technique is known as ICA.}
\end{itemize}

\subsection{Mixtures of factor analysers}
The FA model assumes that the data lives on a low dimensional linear manifold. In reality, most
data is better modeled by some form of low dimensional \emph{curved} manifold. We can approximate a curved manifold by a piecewise linear manifold. This suggests the following model: let the $k$'th linear subspace of dimensionality $L_k$ be represented by $\vec{W}_k$, for $k=1:K$. Suppose we have a latent indicator $qi \in \{1,\cdots,K\}$ specifying which subspace we should use to generate the data. We then sample $\vec{z}_i$ from a Gaussian prior and pass it through the $\vec{W}_k$ matrix (where $k=q_i$), and add noise. More precisely, the model is as follows:
\begin{align}
p(q_i|\vec{\theta}) & =\mathrm{Cat}(q_i\vec{\pi}) \\
p(\vec{z}_i|\vec{\theta}) & =\mathcal{N}(\vec{z}_i|\vec{0},\vec{I}) \\
p(\vec{x}_i|q_i=k,\vec{z}_i,\vec{\theta}) & =\mathcal{N}(\vec{x}_i|\vec{W}\vec{z}_i+\vec{\mu}_k,\vec{\Psi})
\end{align}
This is called a \textbf{mixture of factor analysers}(MFA) (Hinton et al. 1997).

Another way to think about this model is as a low-rank version of a mixture of Gaussians. In particular, this model needs $O(KLD)$ parameters instead of the $O(KD^2)$ parameters needed for a mixture of full covariance Gaussians. This can reduce overfitting. In fact, MFA is a good generic density model for high-dimensional real-valued data.

\subsection{EM for factor analysis models}
Below we state the results without proof. The derivation can be found in (Ghahramani and Hinton 1996a). To obtain the results for a single factor analyser, just set $r_{ic} =1$ and $c=1$ in the equations below. In Section \ref{sec:EM-for-PCA} we will see a further simplification of these equations that arises when fitting a PPCA model, where the results will turn out to have a particularly simple and elegant interpretation.

In the E-step, we compute the posterior responsibility of cluster $k$ for data point $i$ using
\begin{equation}
r_{ik} \triangleq p(q_i=k|\vec{x}_i,\vec{\theta}) \propto \pi_k\mathcal{N}(\vec{x}_i|\vec{\mu}_k,\vec{W}_k\vec{W}_k^T\vec{\Psi}_k)
\end{equation}

The conditional posterior for $\vec{z}_i$ is given by
\begin{align}
p(\vec{z}_i|\vec{x}_i,q_i=k,\vec{\theta}) & = \mathcal{N}(\vec{z}_i|\vec{\mu}_{ik},\vec{\Sigma}_{ik}) \\
\vec{\Sigma}_{ik} & \triangleq (\vec{I}+\vec{W}_k^T\vec{\Psi}_k^{-1}\vec{W})_k^{-1} \\
\vec{\mu}_{ik} & \triangleq \vec{\Sigma}_{ik}\vec{W}_k^T\vec{\Psi}_k^{-1}(\vec{x}_i-\vec{\mu}_k)
\end{align}

In the M step, it is easiest to estimate $\vec{\mu}_k$ and $\vec{W}_k$ at the same time, by defining $\tilde{\vec{W}}_k=(\vec{W}_k,\vec{\mu}_k)$, $\tilde{\vec{z}}=(\vec{z},1)$, also, define
\begin{align}
\tilde{\vec{W}}_k & =(\vec{W}_k,\vec{\mu}_k) \\
\tilde{\vec{z}} & =(\vec{z},1) \\
\vec{b}_{ik} & \triangleq \mathbb{E}[\tilde{\vec{z}}|\vec{x}_i,q_i=k]=\mathbb{E}[(\vec{\mu}_{ik};1)] \\
\vec{C}_{ik} & \triangleq \mathbb{E}[\tilde{\vec{z}}\tilde{\vec{z}}^T|\vec{x}_i,q_i=k] \\
    & =\left(\begin{array}{cc}
\mathbb{E}[\vec{z}\vec{z}^T|\vec{x}_i,q_i=k] & \mathbb{E}[\vec{z}|\vec{x}_i,q_i=k] \\
\mathbb{E}[\vec{z}|\vec{x}_i,q_i=k]^T & 1
\end{array}\right)
\end{align}
Then the M step is as follows:
\begin{align}
\hat{\pi}_k & = \frac{1}{N}\sum\limits_{i=1}^N r_{ik} \\
\hat{\tilde{\vec{W}}}_k & = \left(\sum\limits_{i=1}^N r_{ik}\vec{x}_i\vec{b}_{ik}^T\right)\left(\sum\limits_{i=1}^N r_{ik}\vec{x}_i\vec{C}_{ik}^T\right)^{-1} \label{eqn:FA-EM-W} \\
\hat{\vec{\Psi}} & = \frac{1}{N}\mathrm{diag}\left[\sum\limits_{i=1}^N r_{ik}(\vec{x}_i-\hat{\tilde{\vec{W}}}_{ik}\vec{b}_{ik})\vec{x}_i^T\right]
\end{align}

Note that these updates are for “vanilla” EM. A much faster version of this algorithm, based on ECM, is described in (Zhao and Yu 2008).

\subsection{Fitting FA models with missing data}
\label{sec:Fitting-FA-models-with-missing-data}
In many applications, such as collaborative filtering, we have missing data. One virtue of the EM approach to fitting an FA/PPCA model is that it is easy to extend to this case. However, overfitting can be a problem if there is a lot of missing data. Consequently it is important to perform MAP estimation or to use Bayesian inference. See e.g., (Ilin and Raiko 2010) for details.

\section{Principal components analysis (PCA)}
\label{sec:PCA}
Consider the FA model where we constrain $\vec{\Psi}=\sigma^2\vec{I}$, and $\vec{W}$ to be orthonormal. It can be shown (Tipping and Bishop 1999) that, as $\sigma^2 \rightarrow 0$, this model reduces to classical (nonprobabilistic) \textbf{principal components analysis}(PCA), also known as the Karhunen Loeve transform. The version where $\sigma^2 > 0$ is known as \textbf{probabilistic PCA}(\textbf{PPCA}) (Tipping and Bishop 1999), orsensible PCA(Roweis 1997).

\subsection{Classical PCA}

\subsubsection{Statement of the theorem}
The synthesis viewof classical PCA is summarized in the forllowing theorem.

\begin{theorem}
Suppose we want to find an orthogonal set of $L$ linear basis vectors $\vec{w}_j \in \mathbb{R}^D$, and the corresponding scores $\vec{z}_i \in \mathbb{R}^L$, such that we minimize the average \textbf{reconstruction error}
\begin{equation}
J(\vec{W},\vec{Z})=\frac{1}{N}\sum\limits_{i=1}^N \lVert\vec{x}_i-\hat{\vec{x}}_i\rVert^2
\end{equation}
where $\hat{\vec{x}}_i=\vec{W}\vec{z}_i$, subject to the constraint that $\vec{W}$ is orthonormal. Equivalently, we can write this objective as follows
\begin{equation}
J(\vec{W},\vec{Z})=\frac{1}{N} \lVert\vec{X}-\vec{W}\vec{Z}^T\rVert^2
\end{equation}
where $\vec{Z}$ is an $N \times L$ matrix with the $\vec{z}_i$ in its rows, and $\lVert A\rVert_F$ is the \textbf{Frobenius norm} of matrix $\vec{A}$, defined by
\begin{equation}
\lVert A\rVert_F \triangleq \sqrt{\sum\limits_{i=1}^M \sum\limits_{j=1}^N a_{ij}^2}=\sqrt{\mathrm{tr}(\vec{A}^T\vec{A})}
\end{equation}

The optimal solution is obtained by setting $\hat{\vec{W}}=\vec{V}_L$, where $\vec{V}_L$ contains the $L$ eigenvectors with largest eigenvalues of the empirical covariance matrix, $\hat{\vec{\Sigma}}=\frac{1}{N}\sum_{i=1}^N \vec{x}_i\vec{x}_i^T$. (We assume the $\vec{x}_i$ have zero mean, for notational simplicity.) Furthermore, the optimal low-dimensional encoding of the data is given by $\hat{\vec{z}}_i=\vec{W}^T\vec{x}_i$, which is an orthogonal projection of the data onto the column space spanned by the eigenvectors.
\end{theorem}

An example of this is shown in Figure \ref{fig:PCA-PPCA}(a) for $D=2$ and $L=1$. The diagonal line is the vector $\vec{w}_1$; this is called the first principal component or principal direction. The data points $\vec{x}_i \in \mathbb{R}^2$ are orthogonally projected onto this line to get $\vec{z}_i \in \mathbb{R}$. This is the best 1-dimensional approximation to the data. (We will discuss Figure \ref{fig:PCA-PPCA}(b) later.)

\begin{figure}[hbtp]
\centering
\subfloat[]{\includegraphics[scale=.60]{PCA-PPCA-a.png}} \\
\subfloat[]{\includegraphics[scale=.60]{PCA-PPCA-b.png}}
\caption{An illustration of PCA and PPCA where $D=2$ and $L=1$. Circles are the original data points, crosses are the reconstructions. The red star is the data mean. (a) PCA. The points are orthogonally projected onto the line. (b) PPCA. The projection is no longer orthogonal: the reconstructions are shrunk towards the data mean (red star).}
\label{fig:PCA-PPCA}
\end{figure}

The principal directions are the ones along which the data shows maximal variance. This means that PCA can be “misled” by directions in which the variance is high merely because of the measurement scale. It is therefore standard practice to standardize the data first, or equivalently, to work with correlation matrices instead of covariance matrices.

\subsubsection{Proof *}
See Section 12.2.2 of MLAPP.

\subsection{Singular value decomposition (SVD)}
We have defined the solution to PCA in terms of eigenvectors of the covariance matrix. However, there is another way to obtain the solution, based on the \textbf{singular value decomposition}, or \textbf{SVD}. This basically generalizes the notion of eigenvectors from square matrices to any kind of matrix.

\begin{theorem}(\textbf{SVD}).
Any matrix can be decomposed as follows
\begin{equation}\label{eqn:SVD}
\underbrace{\vec{X}}_{N \times D}=\underbrace{\vec{U}}_{N \times N}\underbrace{\vec{\Sigma}}_{N \times D}\underbrace{\vec{V}^T}_{D \times D}
\end{equation}
where $\vec{U}$ is an $N \times N$ matrix whose columns are orthornormal(so $\vec{U}^T\vec{U}=\vec{I}$), $\vec{V}$ is $D \times D$ matrix whose rows and columns are orthonormal (so $\vec{V}^T\vec{V}=\vec{V}\vec{V}^T=\vec{I}_D$), and $\vec{\Sigma}$ is a $N \times D$ matrix containing the $r=\min(N,D)$ singular values $\sigma_i \geq 0$ on the main diagonal, with 0s filling the rest of the matrix.
\end{theorem}

This shows how to decompose the matrix $X$ into the product of three matrices: $\vec{V}$ describes an orthonormal basis in the domain, and $\vec{U}$ describes an orthonormal basis in the co-domain, and $\vec{\Sigma}$ describes how much the vectors in $\vec{V}$ are stretched to give the vectors in $\vec{U}$.

Since there are at most $D$ singular values (assuming $N>D$), the last $N−D$ columns of $\vec{U}$ are irrelevant, since they will be multiplied by 0. The \textbf{economy sized SVD}, or \textbf{thin SVD}, avoids computing these unnecessary elements. Let us denote this decomposition by $\hat{\vec{U}}\hat{\vec{\Sigma}}\hat{\vec{V}}^T$.If $N>D$, we have
\begin{equation}
\underbrace{\vec{X}}_{N \times D}=\underbrace{\hat{\vec{U}}}_{N \times D}\underbrace{\hat{\vec{\Sigma}}}_{D \times D}\underbrace{\hat{\vec{V}}^T}_{D \times D}
\end{equation}
as in Figure \ref{fig:SVD}(a). If $N<D$, we have
\begin{equation}
\underbrace{\vec{X}}_{N \times D}=\underbrace{\hat{\vec{U}}}_{N \times N}\underbrace{\hat{\vec{\Sigma}}}_{N \times N}\underbrace{\hat{\vec{V}}^T}_{N \times D}
\end{equation}
Computing the economy-sized SVD takes $O(ND\min(N,D))$ time (Golub and van Loan 1996, p254).

\begin{figure}[hbtp]
\centering
\subfloat[]{\includegraphics[scale=.50]{SVD-a.png}} \\
\subfloat[]{\includegraphics[scale=.50]{SVD-b.png}}
\caption{(a) SVD decomposition of non-square matrices $\vec{X}=\vec{U}\vec{\Sigma}\vec{V}^T$. The shaded parts of $\vec{\Sigma}$, and all the off-diagonal terms, are zero. The shaded entries in $\vec{U}$ and $\vec{\Sigma}$ are not computed in the economy-sized version, since they are not needed. (b) Truncated SVD approximation of rank $L$.}
\label{fig:SVD}
\end{figure}

The connection between eigenvectors and singular vectors is the following:
\begin{align}
\vec{U}&=\mathrm{evec}(\vec{X}\vec{X}^T) \\
\vec{V}&=\mathrm{evec}(\vec{X}^T\vec{X}) \\
\vec{\Sigma}^2&=\mathrm{eval}(\vec{X}\vec{X}^T)=\mathrm{eval}(\vec{X}^T\vec{X})
\end{align}
For the proof please read Section 12.2.3 of MLAPP.

Since the eigenvectors are unaffected by linear scaling of a matrix, we see that the right singular vectors of $\vec{X}$ are equal to the eigenvectors of the empirical covariance $\hat{\vec{\Sigma}}$. Furthermore, the eigenvalues of $\hat{\vec{\Sigma}}$ are a scaled version of the squared singular values.

However, the connection between PCA and SVD goes deeper. From Equation \ref{eqn:SVD}, we can represent a rank $r$ matrix as follows:
\begin{equation*}
\vec{X}=\sigma_1\left(\begin{array}{c} | \\ \vec{u}_1 \\ | \end{array}\right)\left(\begin{array}{ccc} - & \vec{v}_1 & - \end{array}\right)+\cdots+\sigma_r\left(\begin{array}{c} | \\ \vec{u}_r \\ | \end{array}\right)\left(\begin{array}{ccc} - & \vec{v}_r^T & - \end{array}\right)
\end{equation*}
If the singular values die off quickly, we can produce a rank $L$ approximation to the matrix as follows:
\begin{align}
\vec{X} & \approx \sigma_1\left(\begin{array}{c} | \\ \vec{u}_1 \\ | \end{array}\right)\left(\begin{array}{ccc} - & \vec{v}_1 & - \end{array}\right)+\cdots+\sigma_r\left(\begin{array}{c} | \\ \vec{u}_L \\ | \end{array}\right)\left(\begin{array}{ccc} - & \vec{v}_L^T & - \end{array}\right) \nonumber \\
 & = \vec{U}_{:,1:L}\vec{\Sigma}_{1:L,1:L}\vec{V}_{:,1:L}^T
\end{align}
This is called a \textbf{truncated SVD} (see Figure \ref{fig:SVD}(b)).

One can show that the error in this approximation is given by
\begin{equation}
\lVert \vec{X}-\vec{X}_L \rVert_F \approx \sigma_L
\end{equation}
Furthermore, one can show that the SVD offers the best rank $L$ approximation to a matrix (best in the sense of minimizing the above Frobenius norm).

Let us connect this back to PCA. Let $\vec{X}=\vec{U}\vec{\Sigma}\vec{V}^T$ be a truncated SVD of $\vec{X}$. We know that $\hat{\vec{W}}=\vec{V}$, and that $\hat{\vec{Z}}=\vec{X}\hat{\vec{W}}$, so
\begin{equation}
\hat{\vec{Z}}=\vec{U}\vec{\Sigma}\vec{V}^T\vec{V}=\vec{U}\vec{\Sigma}
\end{equation}
Furthermore, the optimal reconstruction is given by $\hat{\vec{X}}=\vec{Z}\hat{\vec{W}}$,so we find
\begin{equation}
\hat{\vec{X}}=\vec{U}\vec{\Sigma}\vec{V}^T
\end{equation}
This is precisely the same as a truncated SVD approximation! This is another illustration of the fact that PCA is the best low rank approximation to the data.

\subsection{Probabilistic PCA}
\begin{theorem}((\textbf{Tipping and Bishop 1999})).
Consider a factor analysis model in which $\vec{\Psi}=\sigma^2\vec{I}$ and $\vec{W}$ is orthogonal. The observed data log likelihood is given by
\begin{align}
\log p(\vec{X}|\vec{W},\sigma^2\vec{I}) & =-\frac{N}{2}\ln|\vec{C}|-\frac{1}{2}\sum\limits_{i=1}^N \vec{x}_i^T\vec{C}^{-1}\vec{x}_i \nonumber \\
   & =-\frac{N}{2}\ln|\vec{C}|+\mathrm{tr}(\vec{C}^{-1}\vec{\Sigma})
\end{align}
where $\vec{C}=\vec{W}\vec{W}^T+\sigma^2\vec{I}$ and $\vec{\Sigma}=\frac{1}{N}\sum_{i=1}^N \vec{x}_i\vec{x}_i^T=\frac{1}{N}\vec{X}\vec{X}^T$. (We are assuming centred data, for notational simplicity.) The maxima of the log-likelihood are given by
\begin{equation}
\hat{\vec{W}}=\vec{V}(\vec{\Lambda}-\sigma^2\vec{I})^{\frac{1}{2}}\vec{R}
\end{equation}
where $\vec{R}$ is an arbitrary $L \times L$ orthogonal matrix, $\vec{V}$ is the $D \times L$ matrix whose columns are the first $L$ eigenvectors of $\vec{\Sigma}$, and $\vec{\Lambda}$ is the corresponding diagonal matrix of eigenvalues. Without loss of generality, we can set $\vec{R}=\vec{I}$. Furthermore, the MLE of the noise variance is given by
\begin{equation}
\hat{\sigma}^2=\frac{1}{D-L}\sum\limits_{j=L+1}^D \lambda_j
\end{equation}
which is the average variance associated with the discarded dimensions.
\end{theorem}

Thus, as $\sigma^2 \rightarrow 0$, we have $\hat{\vec{W}} \rightarrow \vec{V}$, as in classical PCA. What about $\hat{\vec{Z}}$? It is easy to see that the posterior over the latent factors is given by
\begin{align}
p(\vec{z}_i|\vec{x}_i,\hat{\vec{\theta}}) & = \mathcal{N}(\vec{z}_i|\hat{\vec{F}}^{-1}\hat{\vec{W}}^T\vec{x}_i,\sigma^2\hat{\vec{F}}^{-1}) \label{eqn:PPCA-posterior}\\
\hat{\vec{F}} & \triangleq \hat{\vec{W}}^T\hat{\vec{W}}+\sigma^2\vec{I}
\end{align}
(Do not confuse $\vec{F} = \vec{W}^T\vec{W}+\sigma^2\vec{I}$ with $\vec{C}=\vec{W}\vec{W}^T+\sigma^2\vec{I}$.) Hence, as $\sigma^2 \rightarrow 0$, we find $\hat{\vec{W}} \rightarrow \vec{V}$, $\hat{\vec{F}} \rightarrow \vec{I}$ and $\vec{z}_i \rightarrow \vec{V}^T\vec{x}_i$. Thus the posterior mean is obtained by an orthogonal projection of the data onto the column space of $\vec{V}$, as in classical PCA.

Note, however, that if $\sigma^2 \rightarrow 0$, the posterior mean is not an orthogonal projection, since it is shrunk somewhat towards the prior mean, as illustrated in Figure \ref{fig:PCA-PPCA}(b). This sounds like an undesirable property, but it means that the reconstructions will be closer to the overall data mean, $\hat{\vec{\mu}}=\bar{\vec{x}}$.

\subsection{EM algorithm for PCA}
\label{sec:EM-for-PCA}
Although the usual way to fit a PCA model uses eigenvector methods, or the SVD, we can also use EM, which will turn out to have some advantages that we discuss below. EM for PCA relies on the probabilistic formulation of PCA. However the algorithm continues to work in the zero noise limit, $\sigma^2=0$, as shown by (Roweis 1997).

Let $\tilde{\vec{Z}}$ be a $L \times N$ matrix storing the posterior means (low-dimensional representations) along its columns. Similarly, let $\tilde{\vec{X}}=\vec{X}^T$ store the original data along its columns. From Equation \ref{eqn:PPCA-posterior}, when $\sigma^2=0$, we have
\begin{equation}
\tilde{\vec{Z}}=(\vec{W}^T\vec{W})^{-1}\vec{W}^T\tilde{\vec{X}}
\end{equation}
This constitutes the E step. Notice that this is just an orthogonal projection of the data.

From Equation \ref{eqn:FA-EM-W}, the M step is given by
\begin{equation}
\hat{\vec{W}}=\left(\sum\limits_{i=1}^N \vec{x}_i\mathbb{E}[\vec{z}_i]^T\right)\left(\sum\limits_{i=1}^N \mathbb{E}[\vec{z}_i]\mathbb{E}[\vec{z}_i]^T\right)^{-1}
\end{equation}
where we exploited the fact that $\vec{\Sigma}=\mathrm{cov}[\vec{z}_i|\vec{x}_i,\vec{\theta}]=\vec{0}$ when $\sigma^2=0$.

(Tipping and Bishop 1999) showed that the only stable fixed point of the EM algorithm is the globally optimal solution. That is, the EM algorithm converges to a solution where $\vec{W}$ spans the same linear subspace as that defined by the first $L$ eigenvectors. However, if we want $\vec{W}$ to be orthogonal, and to contain the eigenvectors in descending order of eigenvalue, we have to orthogonalize the resulting matrix (which can be done quite cheaply). Alternatively, we can modify EM to give the principal basis directly (Ahn and Oh 2003).

This algorithm has a simple physical analogy in the case $D=2$ and $L=1$(Roweis 1997). Consider some points in $\mathbb{R}^2$ attached by springs to a rigid rod, whose orientation is defined by a vector $\vec{w}$. Let $\vec{z}_i$ be the location where the $i$'th spring attaches to the rod. See Figure 12.11 of MLAPP for an illustration.

Apart from this pleasing intuitive interpretation, EM for PCA has the following advantages over eigenvector methods:
\begin{itemize}
\item{EM can be faster. In particular, assuming $N,D \gg L$, the dominant cost of EM is the projection operation in the E step, so the overall time is $O(TLND)$, where $T$ is the number of iterations. This is much faster than the $O(\min(ND^2,DN^2))$ time required by straightforward eigenvector methods, although more sophisticated eigenvector methods, such as the Lanczos algorithm, have running times comparable to EM.}
\item{EM can be implemented in an online fashion, i.e., we can update our estimate of $\vec{W}$ as the data streams in.}
\item{EM can handle missing data in a simple way (see Section \ref{sec:Fitting-FA-models-with-missing-data}).}
\item{EM can be extended to handle mixtures of PPCA/ FA models.}
\item{EM can be modified to variational EM or to variational Bayes EM to fit more complex models.}
\end{itemize}

\section{Choosing the number of latent dimensions}
\label{sec:FA-Choosing-L}
In Section \ref{sec:Model-selection-for-LVM}, we discussed how to choose the number of components $K$ in a mixture model. In this section, we discuss how to choose the number of latent dimensions $L$ in a FA/PCA model.

\subsection{Model selection for FA/PPCA}
TODO

\subsection{Model selection for PCA}
TODO

\section{PCA for categorical data}
In this section, we consider extending the factor analysis model to the case where the observed data is categorical rather than real-valued. That is, the data has the form $y_{ij} \in \{1,...,C\}$, where $j=1:R$ is the number of observed response variables. We assume each $y_{ij}$ is generated from a latent variable $\vec{z}_i \in \mathbb{R}^L$, with a Gaussian prior, which is passed through the softmax function as follows:
\begin{align}
p(\vec{z}_i) & = \mathcal{N}(\vec{z}_i|\vec{0},\vec{I}) \\
p(\vec{y}_i|\vec{z}_i,\vec{\theta}) & = \prod\limits_{j=1}^R \mathrm{Cat}(y_{ir}|\mathcal{S}(\vec{W}_r^T\vec{z}_i+\vec{w}_{0r}))
\end{align}
where $\vec{W}_r \in \mathbb{R}^L$ is the factor loading matrix for response $j$, and $\vec{W}_{0r} \in \mathbb{R}^M$ is the offset term for response $r$, and $\vec{\theta}=(\vec{W}_r,\vec{W}_{0r})_{r=1}^R$. (We need an explicit offset term, since clamping one element of $\vec{z}_i$ to 1 can cause problems when computing the posterior covariance.) As in factor analysis, we have defined the prior mean to be $\vec{\mu}_0=\vec{0}$ and the prior covariance $\vec{V}_0=\vec{I}$, since we can capture non-zero mean by changing $\vec{w}_{0j}$ and non-identity covariance by changing $\vec{W}_r$. We will call this categorical PCA. See Chapter 27 TODO for a discussion of related models.

In (Khan et al. 2010), we show that this model outperforms finite mixture models on the task of imputing missing entries in design matrices consisting of real and categorical data. This is useful for analysing social science survey data, which often has missing data and variables of mixed type.

\section{PCA for paired and multi-view data}

\subsection{Supervised PCA (latent factor regression)}

\subsection{Discriminative supervised PCA}

\subsection{Canonical correlation analysis}

\section{Independent Component Analysis (ICA)}
\label{sec:ICA}
Let $\vec{x}_t \in \mathbb{R}^D$ be the observed signal at the sensors at “time” $t$, and $\vec{z}_t \in \mathbb{R}^L$ be the vector of source signals. We assume that
\begin{equation}
\vec{x}_t=\vec{W}\vec{z}_t+\vec{\epsilon}_t
\end{equation}
where $\vec{W}$ is an $D \times L$ matrix, and $\vec{\epsilon}_t \sim \mathcal{N}(\vec{0},\vec{\Psi})$. In this section, we treat each time point as an independent observation, i.e., we do not model temporal correlation (so we could replace the $t$ index with $i$, but we stick with t to be consistent with much of the ICA literature). The goal is to infer the source signals, $p(\vec{z}_t|\vec{x}_t,\vec{\theta})$. In this context, $\vec{W}$ is called the \textbf{mixing matrix}. If $L=D$ (number of sources = number of sensors), it will be a square matrix. Often we will assume the noise level, $|\vec{\Psi}|$, is zero, for simplicity.

So far, the model is identical to factor analysis. However, we will use a different prior for $p(\vec{z}_t)$. In PCA, we assume each source is independent, and has a Gaussian distribution. We will now relax this Gaussian assumption and let the source distributions be any \emph{non-Gaussian} distribution
\begin{equation}
p(\vec{z}_t) =\prod\limits_{j=1}^L p_j(z_{tj})
\end{equation}
Without loss of generality, we can constrain the variance of the source distributions to be \vec{1}, because any other variance can be modelled by scaling the rows of $\vec{W}$ appropriately. The resulting model is known as \textbf{independent component analysis} or \textbf{ICA}.

The reason the Gaussian distribution is disallowed as a source prior in ICA is that it does not permit unique recovery of the sources. This is because the PCA likelihood is invariant to any orthogonal transformation of the sources $\vec{z}_t$ and mixing matrix $\vec{W}$. PCA can recover the best linear subspace in which the signals lie, but cannot uniquely recover the signals themselves.

ICA requires that $\vec{W}$ is square and hence invertible. In the non-square case (e.g., where we have more sources than sensors), we cannot uniquely recover the true signal, but we can compute the posterior $p(\vec{z}_t|\vec{x}_t,\hat{\vec{W}})$, which represents our beliefs about the source. In both cases, we need to estimate Was well as the source distributions $p_j$. We discuss how to do this below.

\subsection{Maximum likelihood estimation}
In this section, we discuss ways to estimate square mixing matrices $\vec{W}$ for the noise-free ICA model. As usual, we will assume that the observations have been centered; hence we can also assume $\vec{z}$ is zero-mean. In addition, we assume the observations have been whitened, which can be done with PCA.

If the data is centered and whitened, we have $\mathbb{E}[\vec{x}\vec{x}^T]=\vec{I}$. But in the noise free case, we also have
\begin{equation}
\mathrm{cov}[\vec{x}]=\mathbb{E}[\vec{x}\vec{x}^T]=\vec{W}\mathbb{E}[\vec{z}\vec{z}^T]\vec{W}^T
\end{equation}
Hence we see that $\vec{W}$ must be orthogonal. This reduces the number of parameters we have to estimate from $D^2$ to $D(D-1)/2$. It will also simplify the math and the algorithms.

Let $\vec{V}=\vec{W}^{-1}$; these are often called the recognition weights, as opposed to $\vec{W}$, which are the generative weights.

Since $\vec{x}=\vec{W}\vec{z}$, we have, from Equation \ref{eqn:Multivariate-transformation},
\begin{align}
p_x(\vec{W}\vec{z}_t) & =p_z(\vec{z}_t)|\mathrm{det}(\vec{W}^{-1})| \nonumber \\
   & = p_z(\vec{V}\vec{x}_t)|\mathrm{det}(\vec{V})|
\end{align}
Hence we can write the log-likelihood, assuming $T$ iid samples, as follows:
\begin{equation*}
\frac{1}{T} \log p(\mathcal{D}|\vec{V})=\log|\mathrm{det}(\vec{V})|+\frac{1}{T}\sum\limits_{j=1}^L \sum\limits_{t=1}^T \log p_j(\vec{v}_j^T\vec{x}_t)
\end{equation*}
where $\vec{v}_j$ is the $j$'th row of $\vec{V}$. Since we are constraining $\vec{V}$ to be orthogonal, the first term is a constant, so we can drop it. We can also replace the average over the data with an expectation operator to get the following objective
\begin{equation}
\mathrm{NLL}(\vec{V})=\sum\limits_{j=1}^L \mathbb{E}[G_j(\vec{z}_j)]
\end{equation}
where $\vec{z}_j=\vec{v}_j^T\vec{x}$ and $G_j(\vec{z}) \triangleq -\log p_j(\vec{z})$. We want to minimize this subject to the constraint that the rows of $\vec{V}$ are orthogonal. We also want them to be unit norm, since this ensures that the variance of the factors is unity (since, with whitened data, $\mathbb{E}[\vec{v}_j^T\vec{x}]=\lVert \vec{v}_j \rVert^2$, which is necessary to fix the scale of the weights. In otherwords, $\vec{V}$ should be an orthonormal matrix.

It is straightforward to derive a gradient descent algorithm to fit this model; however, it is rather slow. One can also derive a faster algorithm that follows the natural gradient; see e.g., (MacKay 2003, ch 34) for details. A popular alternative is to use an approximate Newton method, which we discuss in Section \ref{sec:FastICA}. Another approach is to use EM, which we discuss in Section \ref{sec:ICA-EM}.

\subsection{The FastICA algorithm}
\label{sec:FastICA}

\subsection{Using EM}
\label{sec:ICA-EM}

\subsection{Other estimation principles *}


%---------------------------Aspect Frobenius-----------------------------
\section{Aspect Frobenius\label{s:tri-aspect-Frobenius}}

The aspect Frobenius is the sum of the edge lengths squared divided by the area
and normalized so that a unit equilateral triangle has a value of $1$.
\[
  q = \frac{{\normvec{{L_0}}}^{2} +
            {\normvec{{L_1}}}^{2} +
            {\normvec{{L_2}}}^{2}}{4A\sqrt{3}}
\]

Note that in earlier versions of \verd{}, this metric was
called the triangle aspect ratio.

\trimetrictable{aspect Frobenius}%
{$1$}%                                                Dimension
{$[1,1.3]$}%                                          Acceptable range
{$[1,DBL\_MAX]$}%                                     Normal range
{$[1,DBL\_MAX]$}%                                     Full range
{$1$}%                                                Unit equilateral triangle value
{\cite{pebay:03}}%                                    Reference(s)
{v\_tri\_aspect\_frobenius}%                            Verdict function name


\section{Serre classes}\label{section-serre-classes}
\begin{definition}\label{serre-class}
    A class $\cC$ of abelian groups is a \emph{Serre class} if:
    \begin{enumerate}
	\setcounter{enumi}{0}
	\item $0\in \cC$.
	\item if I have a short exact sequence $0\to A\to B\to C\to 0$, then
	    $A\& C\in \cC$ if and only if $B\in\cC$.
    \end{enumerate}
\end{definition}
Some consequences of this definition: a Serre class is closed under
isomorphisms (easy). A Serre class is closed under subobjects and quotients,
because there is a short exact equence
$$0\to A\hookrightarrow B \to B/A\to 0.$$
Consider an exact sequence $A\to B\to C$ (not necessarily a \emph{short} exact
sequence). If $A,C\in \cC$, then $B\in \cC$ because we have a short exact
sequence:
$$
\xymatrix{
    & & & \coker i\ar[d]\ar[r] & 0\\
    & A\ar[r]^i\ar[d] & B\ar[r]^p\ar[ur] & C & 0\\
    0\ar[r] & \ker p\ar[ur] & & &
}
$$
Some examples are in order.
\begin{example}
    \begin{enumerate}
	\item $\cC = \{0\}$, and $\cC$ the class of all abelian groups.
	\item Let $\cC$ be the class of all torsion abelian groups. We need to
	    check that $\cC$ satisfies the second condition of Definition
	    \ref{}. Consider a short exact sequence
	    $$0\to A\xrightarrow{i} B\xrightarrow{p} C\to 0.$$
	    We need to show that $B$ is torsion if $A$ and $C$ are torsion. To
	    see this, let $b\in B$. Then $p(b)$ is killed by some integer $n$,
	    so there exists $a\in A$ such that $i(a) = nb$. SInce $A$ is
	    torsion, it follows that $b$ is torsion, too.
	\item Let $\cP$ be a set of primes. Define:
	    $$
	    \cC_{\cP} = \{ A : \text{if }p\not\in\cP,\text{ then }p:A\xrightarrow{\simeq} A\text{, i.e., }A \text{ is a } \Z[1/p]\text{-module}\}
	    $$
	    Let $\Z_{(\cP)} = \Z[1/p: p\not\in\cP]\subseteq\QQ$.

	    For instance, if $\cP$ is the set of all primes, then $\cC_\cP$ is
	    the Serre class of all abelian groups. If $\cP$ is the set of all
	    primes other than $\ell$, then $\cC_\cP$ is the Serre class
	    consisting of all $\Z[1/\ell]$-modules. If $\cP = \{\ell\}$, then
	    $\cC_{\{\ell\}} =: \cC_\ell$ is the Serre class of all
	    $\Z_{(\ell)}$-modules. If $\cP = \emptyset$, then $\cC_\emptyset$
	    is all rational vector spaces.
	\item If $\cC$ and $\cC^\prime$ are Serre classes, then so is $\cC\cap
	    \cC^\prime$. For instance, $\cC_\text{tors} \cap \cC_\text{fg}$ is
	    the Serre class $\cC_\text{finite}$. Likewise, $\cC_p \cap
	    \cC_\text{tors}$ is the Serre class of all $p$-torsion abelian
	    groups.
    \end{enumerate}
\end{example}
Here are some straightforward consequences of the definition:
\begin{enumerate}
    \item If $C_\bullet$ is a chain complex, and $C_n\in \cC$, then
	$H_n(C_\bullet)\in\cC$.
    \item Suppose $F_\ast A$ is a filtration on an abelian group. If
	$A\in\cC$, then $\gr_nA\in\cC$ for all $n$. If $F_\ast A$ is finite
	and $\gr_n A\in\cC$ for all $n$, then $A\in\cC$.
    \item Suppose we have a spectral sequence $\{E_r\}$. If $E^2_{s,t}\in \cC$,
	then $E^r_{s,t}\in \cC$ for $r\geq 2$. It follows that if $\{E^r\}$ is
	a right half-plane spectral sequence, then $E^{s+1}_{s,t}\fib
	E^{s+2}_{s,t}\fib\cdots\fib E^\infty_{s,t}\in\cC$.

        Thus, if the spectral sequence comes from a filtered complex (which is
	bounded below, such that for all $n$ there exists an $s$ such that $F_s
	H_n(C) = H_n(C)$, i.e., the homology of the filtration stabilizes),
	then $E^\infty_{s,t} = \gr_s H_{s+t}(C)$. This means that if the
	$E^2_{s,t}\in\cC$ for all $s+t = n$, then $H_n(C)\in\cC$.
\end{enumerate}
To apply this to the Serre spectral sequence, we need an additional axiom for
Definition \ref{serre-class}:
\begin{enumerate}
	\setcounter{enumi}{1}
    \item if $A,B\in\cC$, then so are $A\otimes B$ and $\Tor_1(A,B)$.
\end{enumerate}
All of the examples given above satisfy this additional axiom.
\begin{terminology}
    $f:A\to B$ is said to be a $\cC$-epimorphism if $\coker f\in \cC$, a
    $\cC$-monomorphism if $\ker f\in\cC$, and a $\cC$-isomorphism if it is a
    $\cC$-epimorphism and a $\cC$-monomorphism.
\end{terminology}
\begin{prop}
    Let $\pi:E\to B$ be a fibration and $B$ path connected, such that the fiber
    $F = \pi^{-1}(\ast)$ is path connected. Suppose $\pi_1(B)$ acts trivially
    on $H_\ast(F)$.

    Let $\cC$ be a Serre class satisfiying Axiom 2. Let $s\geq 3$, and assume
    that $H_n(E)\in\cC$ where $1\leq n<s-1$ and $H_t(B)\in \cC$ for $1\leq
    t<s$. Then $H_t(F)\in\cC$ for $1\leq t<s-1$.
\end{prop}
\begin{proof}
    We will do the case $s=3$, for starters.
    We're gonna want to relate the low-dimension homology of these groups.
    What can I say?
    We know that $H_0(E) = \Z$ since it's connected.
    I have $H_1(E)\to H_1(B)$, via $\pi$.
    This is one of the edge homomorphisms, and thus it surjects (no possibility for a differential coming in).
    I now have a map $H_1(F)\to H_1(E)$.
    But I have a possible $d^2:H_2(B)\to H_1(F)$, which is a transgression that gives:
    $$
    H_2(B)\xrightarrow{\partial} H_1(F)\to H_1(E)\to H_1(B)\to 0
    $$

    Let me take a step back and say something general.
    You might be interested in knowing when something in $H_n(F)$ maps to zero in $H_n(E)$.
    I.e., what's the kernel of $H_n(F)\to H_n(E)$.
    The sseq gives an obstruction to being an isomorphism.
    The only way that something can be killed by $H_n(F)\to H_n(E)$ is described by:
    $$
    \ker(H_n(F)\to H_n(E)) = \bigcup\left(\img \text{ of }d^r\text{ hitting }E^r_{0,n}\right)
    $$
    You can also say what the cokernel is:
    it's whatever's left in $E^\infty_{s,t}$ with $s+t = n$.
    These obstruct $H_n(F)\to H_n(E)$ from being surjective.

    In the same way, I can do this for the base.
    If I have a class in $H_n(E)$, that maps to $H_n(B)$, the question is: what's the image?
    Well, the only obstruction is the possibility is that the element in $H_n(B)$ supports a nonzero differential.
    Thus:
    $$
    \img(H_n(E)\xrightarrow{\pi_\ast} H_n(B)) = \bigcap\left(\ker(d^r:E^r_{r,0}\to\cdots)\right)
    $$
    Again, you can think of the sseq as giving obstructions.
    And also, the obstruction to that map being a monomorphism that might occur in lower filtration along the same total degree line.

    Back to our argument.
    We had the low-dimensional exact sequence:
    $$
    H_2(B)\xrightarrow{\partial} H_1(F)\to H_1(E)\to H_1(B)\to 0
    $$
    Here $p=3$, so we have $H_2(B)\in\cC$ and $H_1(E)\in\cC$.
    Thus $H_1(F)\in\cC$.
    That's the only thing to check when $p=3$.

    Let's do one more case of this induction.
    What does this say?
    Now I'll do $p=4$.
    We're interested in knowing if $E^2_{0,3}\in\cC$.
    There are now two possible differentials!
    I have $H_2(F) = E^2_{0,2}\fib E^3_{0,2}$.
    This quotient comes from $d^2:E^2_{2,1}\to E^2_{0,2}$.
    Now, $d^3:E^3_{3,0}\to E^3_{0,2}$ which gives a surjection $E^3_{0,2}\fib E^4_{0,2}\simeq E^\infty_{0,2}\hookrightarrow H_2(E)$.
    Now, our assumptions were that $E^2_{2,1},E^3_{3,0},H_2(E)\in\cC$.
    Thus $E^3_{0,2}\in\cC$ and so $E^2_{0,2} = H_2(F)\in\cC$.
    Ta-da!
\end{proof}
We're close to doing actual calculations, but I have to talk about the multiplicative structure on the Serre sseq first.


\subsection{Stratified sampling and analysis}
Stratified sampling  \cite{bickel1984asymptotic} gives a more refined version of the Monte Carlo method.

\begin{lemma} \label{lem:stratified}
 For any
  nonoverlaping decomposition $G=G_1\cup G_2\cup \cdots \cup G_M$ and
  positive integer $n$, let $n_i=\lceil \lambda(G_i)n\rceil$ be the smallest integer larger than $\lambda(G_i)n$ and $\displaystyle N=\sum_{i=1}^M n_i$.  Let $\theta_{i,j}\in G_i (1\le j\le n_i)$ and
\begin{equation}
g_n=\sum_{i=1}^M \lambda(G_i)g_{n_i}^i \quad \mbox{with}\quad g_{n_i}^i={1 \over n_i}\sum_{j=1}^{n_i} g(\theta_{i,j}).
\end{equation}
It holds that
\begin{equation}
\mathbb{E}_N(\mathbb{E}_G g -g_N)^2=\sum_{i=1}^M {\lambda^2(G_i) \over n_i} \mathbb{E}_{G_i}\big (g -\mathbb{E}_{G_i}g\big )^2
\le {1 \over n} \max_{1\le i\le M}\sup_{\theta, \theta'\in G_i}\big |g(\theta) - g(\theta')\big |^2.
\end{equation}
\end{lemma}

\begin{proof}%[Proof of Lemma \ref{lem:stratified}]
It follows from definition that
\begin{equation}
g(x, \theta) = \sum_{i=1}^M \lambda(G_i) g(x, \theta),\qquad \mathbb{E}_G g =\sum_{i=1}^M\lambda(G_i) \int_{G_i} g\lambda_i(\theta)d\theta = \sum_{i=1}^M \lambda(G_i) \mathbb{E}_{G_i}g.
\end{equation}
Thus, the difference $g - \mathbb{E}_G g$ is a linear combination of $g -\mathbb{E}_{G_i}g$ on each $G_i$ as follows
\begin{equation}
g - \mathbb{E}_G g = \sum_{i=1}^M \lambda(G_i) \big (g -\mathbb{E}_{G_i}g\big ).
\end{equation}
It follows from
\begin{equation}
\mathbb{E}_G g-g_n=\sum_{i=1}^M \lambda(G_i) (\mathbb{E}_{G_i}g -  g^i_{ n_i})
\end{equation}
and \eqref{En} that
\begin{equation}
\mathbb{E}_n(\mathbb{E}_G g -g_n)^2 =\sum_{i,j=1}^M \lambda(G_i) \lambda(G_j) \mathbb{E}_n \big ((\mathbb{E}_{G_i}g -  g^i_{ n_i})(\mathbb{E}_{G_j}g -  g^j_{ n_j})\big )
=\sum_{i,j=1}^M \lambda(G_i) \lambda(G_j) I_{ij}
\end{equation}
with
$$
I_{ij} = \mathbb{E}_n \big ((\mathbb{E}_{G_i}g -  g^i_{ n_i})(\mathbb{E}_{G_j}g -  g^j_{ n_j})\big ).
$$
\iffalse
If $i\neq j$,
\begin{equation}\label{eq:Iij}
\begin{split}
I_{ij}=&\int_{G\times \cdots\times G} (\mathbb{E}_{G_i}g -  g^i_{ n_i})(\mathbb{E}_{G_j}g -  g^j_{ n_j})\lambda(\theta_{i,n_1})\cdots\lambda(\theta_{i,n_i})\lambda(\theta_{j,1})\cdots \lambda(\theta_{j,n_j})d\theta_{i,1}\cdots d\theta_{i,n_i}d\theta_{j,1}\cdots d\theta_{j,n_j}
\\
=&\int_{G\times \cdots\times G} (\mathbb{E}_{G_i}g -  g^i_{ n_i})\lambda(\theta_{i,n_1})\cdots\lambda(\theta_{i,n_i})d\theta_{i,1}\cdots d\theta_{i,n_i}
\\
&+
\int_{G\times \cdots\times G} (\mathbb{E}_{G_i}g -  g^j_{ n_j})\lambda(\theta_{j,1})\cdots \lambda(\theta_{j,n_j})d\theta_{j,1}\cdots d\theta_{j,n_j}
\\
=&\lambda^{n_i}(G_i)\int_{G_i\times \cdots\times G_i} (\mathbb{E}_{G_i}g -  g^i_{ n_i})\lambda_i(\theta_{i,n_1})\cdots\lambda_i(\theta_{i,n_i})d\theta_{i,1}\cdots d\theta_{i,n_i}
\\
&+
\lambda^{n_j}(G_j)\int_{G\times \cdots\times G} (\mathbb{E}_{G_i}g -  g^j_{ n_j})\lambda_j(\theta_{j,1})\cdots \lambda_j(\theta_{j,n_j})d\theta_{j,1}\cdots d\theta_{j,n_j}
\\
=&0.
\end{split}
\end{equation}
\fi
By Lemma \ref{MC},
\begin{equation}
I_{ij} =  \mathbb{E}_n \big ((\mathbb{E}_{G_i}g -  g^i_{ n_i})^2\big )\delta_{ij}= {1\over n_i} \mathbb{E} \big ((\mathbb{E}_{G_i}g -  g)^2\big )\delta_{ij}.
\end{equation}
Thus,
\begin{equation}
\mathbb{E}_n(\mathbb{E}_G g -g_n)^2 =\sum_{i=1}^M {\lambda^2(G_i) \over n_i} \mathbb{E}_{G_i}\big (g -\mathbb{E}_{G_i}g\big )^2,
\end{equation}
which completes the proof.
\end{proof}

Lemma \ref{MC} and Lemma \ref{lem:stratified} represent two simple identities and subsequent inequalities that can be verified by a direct calculation. Actually Lemma \ref{MC} is a special case of Lemma  \ref{lem:stratified} with $M=1$. Lemma \ref{MC} and Lemma \ref{lem:stratified} are the basis of Monte-Carlo sampling and  stratified sampling in statistics.

\begin{lemma}\label{lem:stratifiedapprox}
\textup{[Stratified Sampling]}
	For $u(x)$ in \eqref{uint}
	with positive $\rho(\theta)\in L^1(G)$, given any positive integers $n$ and $M\le n$, for  any nonoverlaping decomposition $G=G_1\cup G_2\cup \cdots \cup G_M$, there exists$\{\theta_i^\ast\}_{i=1}^N$ with $n\le N \le 2n$ such that
	\begin{equation}
	\|u - u_N\|_{L^2(\Omega)} \leq N^{-1/2}\|\rho\|_{L^1(G)}\max_{1\le j\le M}\sup_{\theta_{j},\theta_{j}'\in G_j} \| g(x,\theta_j) - g(x,\theta_j')\|_{L^2(\Omega)}
	\end{equation}
	where
	$$
	u_N(x)= {2\|\rho\|_{L^1(G)}\over N}\sum_{i=1}^N\beta_i g(x,\theta_i^\ast)
	$$
	and $\beta_i\in [0,1]$.
	\end{lemma}

\begin{proof}
Let $n_j=\lceil \lambda(G_j)n\rceil$ and $\theta_{i,j} \in G_j(1\leq i\leq n_j)$. Define $\displaystyle N=\sum_{j=1}^M n_j$ and
$$
u_N(x)=\|\rho\|_{L^1(G)}\sum_{i=1}^M \lambda(G_i)g_{n_i}^i \quad \mbox{with}\quad g_{n_i}^i={1 \over n_i}\sum_{j=1}^{n_i} g(\theta_{i,j}).
$$
Since
$
\displaystyle u(x)=\|\rho\|_{L^1(G)}\sum_{i=1}^M \lambda(G_i)\mathbb{E}_{G_i} g,
$
by Lemma \ref{MC},
\begin{equation}
\begin{split}
\mathbb{E}_N\|u- u_N \|_{L^2(\Omega)}^2=& \|\rho\|_{L^1(G)}^2
 \sum_{j=1}^{M}{\lambda^2(G_j) \over n_j}\mathbb{E}_{G_j}\|\mathbb{E}_{G_j} g -  g\|^2_{L^2(\Omega)}
\\
\le &\|\rho\|_{L^1(G)}^2 \sum_{j=1}^{M} {\lambda^2(G_j)\over n_j}\sup_{\theta_{j},\theta_{j}'\in G_j} \| g(x,\theta_j) - g(x,\theta_j')\|^2_{L^2(\Omega)}.
\end{split}
\end{equation}
Since ${\lambda(G_j)\over n_j}\le {1\over n}$ and $\displaystyle \sum_{j=1}^M \lambda(G_j)=1$,
\begin{equation}
\mathbb{E}_N\|u - u_N \|_{L^2(\Omega)}^2\leq n^{-1}\|\rho\|_{L^1(G)}^2\max_{1\le j\le M}\sup_{\theta_{j},\theta_{j}'\in G_j} \| g(x,\theta_j) - g(x,\theta_j')\|^2_{L^2(\Omega)}.
\end{equation}
There exist $\{\theta_{i,j}^\ast\}$ such that $\theta_{i,j}^\ast\in G_i$ and
\begin{equation}
\| u-u_N \|_{L^2(\Omega)}^2\leq n^{-1} \|\rho\|_{L^1(G)}^2\max_{1\le j\le M}\sup_{\theta_{j},\theta_{j}'\in G_j} \| g(x,\theta_j) - g(x,\theta_j')\|^2_{L^2(\Omega)}.
\end{equation}
Note that $n\le N\le n+M\le 2n$,
$$
u_N(x) =  {2\|\rho\|_{L^1(G)}\over N}\sum_{j=1}^{M}\frac{N\lambda(G_j)}{2n_j}\sum_{i=1}^{n_j} g(x,\theta_{i,j}^\ast)
 =  {2\|\rho\|_{L^1(G)}\over N}\sum_{j=1}^{M}\beta_{i,j}\sum_{i=1}^{n_j} g(x,\theta_{i,j}^\ast)
$$
with
\begin{equation}
\beta_{i,j}= \frac{N\lambda(G_j)}{2n_j}\le \frac{2\lambda(G_j)n}{2\lambda(G_j)n}\le 1,
\end{equation}
which completes the proof.
\end{proof}


\subsection{Cross-Validation Criterion}
There are several popular information criteria for model selection: for example, Akaike information criterion (\textbf{AIC}), corrected AIC (\textbf{AIC\textsuperscript{c}}), Schwarz Bayesian information criterion (\textbf{SIC}), Hannan-Quinn (\textbf{HQ}) and Mallows' C\textsubscript{p} (\textbf{C\textsubscript{p}}). Most criteria have two components in their formulas: the first part measures model fit while the second penalizes overfitting. The quantity measuring in-sample fit are the same for most criteria, but they differ in the degree of penalization. For instance, AIC penalizes each additional parameter by $2$ while SIC penalizes overfitting by the logarithm of sample size, so SIC tends to select a more parsimonious model than AIC if the sample size is large.

For the forecasting analysis, what we care about is the test error rate assessing the model predictive ability, not the training error rate produced in the model estimation stage, so selecting a information criterion which gives a good estimate of the expected test error rate is crucial. Cross-validation is such a criterion. Specifically, we focus on the use of leave-one-out cross-validation for this paper, though other CV variants, such as K--fold cross-validation, may be considered. Cross-validation is computationally simple for one-step ahead predictive model selection and is shown robust to conditional heteroscedasticity in the econometrics and statistics literature. For forecast combination, researchers have applied CV to the quadratic programming based model averaging analysis, but its setting does not include structural break.

The sample leave-one-out cross-validation criterion can be computed by the following procedure:
\begin{equation} \label{cv}
	CV_T(k) = \frac{1}{T}\sum_{t=1}^{T}\tilde{e}_{t}(k)^{2}
\end{equation}
where $\tilde{e}_{t}(k) = y_t - \tilde{\beta}_{-t}(k)'x_t(k)$ are the residuals from the regression with the $t^{\mathrm{th}}$ observation dropped and $\tilde{\beta}_{-t}(k) = (\sum_{i\not= t}x_i(k) x_i(k)')^{-1}(\sum_{i\not= t}x_i(k) y_i)$ is the associated vector of parameter estimates. Intuitively, this procedure is trying to estimate the expected test error rate based on the model training data. Though equation~\ref{cv} implies that we need to run regression $T$ times for given sample size $T$, luckily, for linear models, we can calculate sample CV value by running regression only once. Formally, the leave-one-out cross validation residuals can be computed from the full sample least squares residuals, $\tilde{e_{t}} = \frac{\hat{e_{t}}}{1 - h_{t}}$, where $h_{t} = x_{t}'(X_{t}'X_{t})^{-1}x_{t}$ is the leverage associated with observation $t$, $\hat{e_{t}}$ is the full sample least squares residual and $\tilde{e_{t}}$ is the cross-validation residual. So we can rewrite equation~\ref{cv} as
\begin{equation}
    CV_T(k) = \frac{1}{T}\sum_{t=1}^{T}\left( \frac{\hat{e_{t}}(k)}{1 - h_{t}} \right)^{2}
\end{equation}
In the next section we are going to show how model averaging weights are derived from the cross-validation criterion.


\section{Integrating Factor Method}
All 1st order linear differential equations have the form
\begin{equation*}
	a_1(x)\dd{y}{x} + a_0(x)y = b_1(x)
\end{equation*}
which can be rewritten as
\begin{equation*}
	\dd{y}{x} + a(x)y = b(x)
\end{equation*}

\noindent
This equation isn't always separable, and we can't just integrate both sides unless $a_1(x)y$ = 0.\\

\noindent
If $a_0(x) = a_1'(x)$, then we could rewrite the equation and solve by doing the product rule in reverse.
\begin{equation*}
	\left(a_1(x)y\right)' = b_1(x) \implies y = \frac{\int{b_1(x) \mathrm{d}x}}{a_1(x)}
\end{equation*}

\noindent
It's possible to rearrange into this form by multiplying the equation by some function. Specifically, what were looking for is a function $\mu(x)$ such that
\begin{equation*}
	\mu(x)\dd{y}{x} + \mu(x)a(x)y = \mu(x)b(x) \text{ and } \mu'(x) = \mu(x)a(x)
\end{equation*}
This equation involving $\mu(x)$ is one that we know how to solve because it's separable.\footnote{Although we are taking an indefinite integral to find $\mu(x)$, we do not have a $+ C$ term.}
\begin{equation*}
	\mu'(x) = \mu(x)a(x) \implies \mu(x) = e^{\int{a(x) \mathrm{d}x}}
\end{equation*}
Substituting the solution for $\mu(x)$ back,
\begin{equation*}
	e^{\int{a(x) \mathrm{d}x}}\dd{y}{x} + e^{\int{a(x) \mathrm{d}x}}a(x)y = e^{\int{a(x) \mathrm{d}x}}b(x) \implies e^{\int{a(x) \mathrm{d}x}}\dd{y}{x} + \mu'(x)y = e^{\int{a(x) \mathrm{d}x}}b(x)
\end{equation*}
Applying the product rule in reverse,
\begin{equation*}
	y = \frac{\int{\mu(x)} b(x) \mathrm{d}x}{\mu(x)} \text{, } \mu(x) = e^{\int{a(x) \mathrm{d}x}}
\end{equation*}

%\noindent
%This method gives us a way to solve all first-order linear ODE's.

\ifodd\includeFirstOrderLinearODEExamples\input{./1stOrderLinearODE/integratingFactor/integratingFactor_example.tex}\fi


\subsection{SVD as a geometrical interpretation}

And what was the advantage of the factorization then? In simple terms,
decomposing the function behind matrix $A$, as a sequence of three
simpler (easier to understand) transformations. Here the geometric
interpretation helps to complete the picture, as orthogonal (change of
basis) matrices do represent rigid transformation in space, that is,
they do not alter the lengths of vectors (hence, preserve
shapes). Strictly speaking, orthogonal matrices can be decomposed as a
rotation and a reflection; but for geometric intuition, is often
desirable to think in the rotation part only. \\

On the other hand, diagonal matrices are the simplest transformation
possible, they do not change the basis but just expand or contract the
coordinates along the axis given by the basis. Again, if we consider
the generic case of diagonal matrices, a negative element $D_{ii}$
would additional provoke a reflection in the axis $i$; but since the
SVD decomposition produces only positive elements on the diagonal, we
ignore this case and just think in terms of contractions or
expansions along the axes. \\

Armed with this geometrical insight, we can enhance our understanding
of the action of $A$ through the SVD decomposition, by associating to
the simpler operations the corresponding geometrical transformations.
The geometric visualization usually requires a couple of
simplifications: first of all, the dimensions of domain and codomain
must be reduced; as it is easier to visualize things in \R{2} or
\R{3}, than in an arbitrary \R{n}. Given two dimensions fit well in a
screen, let us pick $\R{n} = \R{m} = \R{2}$. \\

Secondly, we need to focus our attention in an specific set of
points (as visualizing the effect of a linear transformation against
``all'' vectors in space, even for \R{2}, is a quite abstract and
complex task). The usual procedure is to pick the vectors in the
unitary sphere in \R{2} (which contains in particular the columns of
$V$, as they are unit orthogonal vectors). \\

Let us proceed now: let the matrix $A$ be of
dimensions $2  \times 2$, the matrices $V$ and $U$ be formed
by unit column vectors $\{\vec{v_1},\vec{v_2}\}$ and $\{\vec{u_1},\vec{u_2}\}$,
respectively; and let matrix $\Sigma$ be $diag(\sigma_1,\sigma_2)$
such that $\sigma_1 > \sigma_2 > 0$. We will additionally assume that
$\sigma_1 > 1$ and $\sigma_2 < 1$, in order to allow them represent an
expansion and contraction, respectively. The previously described steps of
the SVD factorization, can be now augmented with the corresponding
geometrical transformations: \\

\begin{enumerate}
\item Start with unit sphere in \R{2}, with the unit vectors \vec{v_1}
  and \vec{v_2} living inside of it.
\item Action of \trans{V}: Rotate the space such that \vec{v_1} and
  \vec{v_2} become the new orthogonal basis (this transformation
  leaves the shape of the sphere intact).
\item Action of $\Sigma$: Once rotated, the unit sphere is expanded
  in the direction of \vec{v_1} (per $\sigma_1$), and contracted in the
  direction of \vec{v_2} (per $\sigma_2$).
\item Action of $U$: Once reshaped, the resulting ellipse is taken
  from the basis $\{u_1,u_2\}$ back to the canonical basis (this
  transformation changes the orientation of the ellipse, given that
  is not a symmetric figure; but still it preserves it shape).
\end{enumerate}
\hfill

These steps can be summarized in the following figure \footnote{Which
  was taken from a \href{http://math.stackexchange.com/questions/243811/visualization-of-singular-value-decomposition-of-a-symmetric-matrix}{MathStackExchange phorum post}, we
  could not locate back the original source though.}: \\

\begin{figure}[h]
  \centering
  \includegraphics[width=15cm]{svd-geo-diag}
  \caption{Geometrical interpretation of $A = U \Sigma \trans{V}$,
    over the unit sphere.}
  \label{fig:svd-geo-diag}
\end{figure}
\hfill


\subsection{From Langevin dynamics to Fokker-Planck equation}

Now that we built a description for the probability distribution over the
ensemble of trajectories that a Markovian stochastic process can follow we are
left with the task of how to define the Fokker-Plank equation coefficients from
the stochastic dynamics defined by the Langevin equation. Specifically in its
more general form we derived \eref{eq_fokker_planck} to depend on three things:
the probability distribution of allele frequencies $P(f, t)$, and the two jump
moments $a^{(1)}(f, t)$ and $a^{(2)}(f, t)$, where these moments are given by
\eref{eq_jump_mom}. A lot of the population genetics literature use a
non-mathematical argument to directly assign the values of these coefficients
given the either deterministic or stochastic nature of the terms. For example,
given the dynamics that we defined in \secref{seq_determ_mut} for the
deterministic form of mutation and selection (\eref{eq_sel_mut}) most texts
would directly assign these dynamics to be the first jump moment
$a^{(1)}(f,t)$. There is nothing intrinsically wrong with this approach, since
this is actually the correct answer, but it is not necessarily obvious that
this should be the case especially since \eref{eq_fokker_planck} requires us to
take a derivative with respect to $x$ of this coefficient multiplied by the
allele frequency distribution.

We will take a more formal approach and find how the terms in the Langevin
dynamics that we defined in \secref{sec_langevin_intro} connect to the
coefficients of the Fokker-Planck equation. To do so first we introduce a
quantity defined as
\begin{equation}
    \mathcal{A}^{(m)}(x; \tau, t) \equiv \int_{-\infty}^\infty dx\;
    (x - x')^m P(x, t + \tau \mid x', t), \; \text{for } m \geq 1.
    \label{eq_compute_jump_mom}
\end{equation}
This is nothing else than the average of $[x(t + \tau) - x(t)]^m$ with a sharp
initial condition $x(t) = x'$ for a Markov process that obeys the transition
probability distribution $P(x, t + \tau \mid x', t)$. Given that our Langevin
dynamics are Markov processes, the ensemble of trajectories must obey a
continuous master equation with such transition probability. Therefore
\eref{eq_compute_jump_mom} can be thought as the average displacement between
two time points $t$ and $t + \tau$ for our Langevin dynamics. In other words
$\mathcal{A}^{(m)}(x; \tau, t)$ can be expressed as
\begin{equation}
    \mathcal{A}^{(m)}(x; \tau, t) \equiv
    \ee{\left[x(t + \tau) - x(t)\right]^m \mid x(t) = x'}.
    \label{eq_average_A}
\end{equation}
Just as we did to derive the master equation, for small $\tau$ we can rewrite
the conditional distribution $P(x, t + \tau \mid x', t)$ using
\eref{eq_transition_short_time}. Using this we can rewrite
\eref{eq_compute_jump_mom} as
\begin{equation}
    \mathcal{A}^{(m)}(x; \tau, t) = \int_{-\infty}^\infty dx\; (x - x')^m
    \underbrace{
    \left[ \delta(x - x') \left( 1 - a^{(0)}(x', t) \tau \right) +
    \phi_t(x \mid x')\tau \right]
    }_{P(x, t + \tau \mid x', t)}.
\end{equation}
Distributing the integrals results in
\begin{equation}
    \begin{aligned}
    \mathcal{A}^{(m)}(x; \tau, t) &=
    \int_{-\infty}^\infty dx\; (x - x')^m \delta(x - x') \\
    &- \int_{-\infty}^\infty dx\; (x - x')^m \delta(x - x') a^{(0)}(x', t)
    \tau \\
    &+ \int_{-\infty}^\infty dx\; (x - x')^m \phi_t(x \mid x') \tau.
    \end{aligned}
\end{equation}
For the first two terms we have that the $\delta$-function is one only when $x
= x'$ while the term $(x - x')^m$ is zero for this particular case. Therefore
both of these terms cancel and we are only left with
\begin{equation}
    \mathcal{A}^{(m)}(x; \tau, t) = \int_{-\infty}^\infty dx\; (x - x')^m
    \phi_t(x \mid x') \tau.
\end{equation}
This is non other than the definition of the jump moments times $\tau$ given by
\eref{eq_jump_mom} since we defined $r \equiv x - x'$ and $\phi_t(x'; r) \equiv
\phi_t(x \mid x')$. Now we can see the connection; we started defining
$\mathcal{A}^{(m)}(x; \tau, t)$ to be the average displacement defined by our
Langevin dynamics over a small time interval $\tau$, and we found that this is
connected to the jump moments that define the coefficients of the Fokker-Planck
equation. So we have that
\begin{equation}
    \mathcal{A}^{(m)}(x; \tau, t) = \tau \cdot a^{(m)}(x, t).
\end{equation}
Given this result we can compute the jump moment by taking the derivative of
$\mathcal{A}^{(m)}$ with respect to $\tau$ and evaluating this derivative at
$\tau = 0$. This is
\begin{equation}
    a^{(m)}(x, t) = \left. {\partial \over \partial \tau}
    \mathcal{A}^{(m)}(x; \tau, t) \right\vert_{\tau = 0}.
    \label{eq_A_tau_deriv}
\end{equation}
The reason we evaluate $\tau = 0$ is for convenience. In principle we could
evaluate the derivative for any value of $\tau$ since $a^{(m)}$ does not depend
on $\tau$, but zero is the one that simplifies things the most. Using the
definition of derivatives we then write \eref{eq_A_tau_deriv} as
\begin{equation}
    a^{(m)}(x, t) = \lim_{\Dt \rightarrow 0}
    \left.{\mathcal{A}^{(m)}(x, \tau + \Dt, t) -
    \mathcal{A}^{(m)}(x, \tau, t) \over \Dt} \right\vert_{\tau = 0}.
\end{equation}
If we now substitute the equivalence of $\mathcal{A}^{(m)}$ with the average
displacement on our Langevin dynamics, i.e. \eref{eq_average_A} we obtain
\footnotesize
\begin{equation}
    a^{(m)}(x, t) = \lim_{\Dt \rightarrow 0} {1 \over \Dt}
    \left[
    \ee{\left[x(t + \tau + \Dt) - x(t)\right]^m \mid x(t) = x'} -
    \ee{\left[x(t + \tau) - x(t)\right]^m \mid x(t) = x'}
    \right]_{\tau = 0}.
    \label{eq_deriv_average}
\end{equation}
\normalsize
Now we use the condition $\tau = 0$ that we chose for simplification since the
second term in \eref{eq_deriv_average} cancels. We are then left with a simple
expression of the form
\begin{equation}
    a^{(m)}(x, t) = \lim_{\Dt \rightarrow 0}
    {\ee{\left[x(t + \Dt) - x(t)\right]^m \mid x(t) = x'} \over \Dt}.
\end{equation}
This result shows that given that our Langevin dynamics are a continuous Markov
process, we can compute the displacement dynamics for a small time interval,
elevate it to the $m\tth$ power, and then average it, to then compute the $m$
coefficient for the Fokker-Planck equation. Let's do this first for the first
coefficient $a^{(m)}(x, t)$.

\subsubsection{Directional coefficient of the Fokker-Planck equation}

For the first coefficient of the Fokker-Planck equation that defines the
directional terms of the dynamics we have
\begin{equation}
    a^{(1)}(x, t) = \lim_{\Dt \rightarrow 0}
    {\ee{x(t + \Dt) - x(t) \mid x(t) = x'} \over \Dt}.
\end{equation}
Recall that we defined the Langevin equations as a stochastic differential
equation given in \eref{eq_langevin}. Since $x(t)$ is the solution to this
differential equation we can use the fundamental theorem of calculus to rewrite
the average displacement as
\begin{equation}
    \ee{x(t + \Dt) - x(t) \mid x(t) = x'} =
    \ee{\left. \int_t^{t + \Dt} dt'\; {dx \over dt'} \right\vert x(t) = x'}.
\end{equation}
We then substitute \eref{eq_langevin} on the right-hand side to obtain
\begin{equation}
    \ee{x(t + \Dt) - x(t) \mid x(t) = x'} =
    \ee{\left.\int_t^{t + \Dt} dt'\; A(x, t') + B(x, t') \xi(t')\right\vert
    x(t) = x'}.
\end{equation}
If we distribute the integral and use the linearity of expected values this
results in
\begin{equation}
    \ee{x(t + \Dt) - x(t) \mid x(t) = x'} =
    \int_t^{t + \Dt} dt\; A(x, t') +
    \int_t^{t + \Dt} dt\; B(x, t') \ee{ \left.\xi(t') \right\vert x(t) = x'},
\end{equation}
where the only thing we needed to average over was the noise term $\xi(t)$. For
the Langevin equation we defined the average of this noise term to be zero
regardless of the initial position; that implies that the average displacement
is then simply given by
\begin{equation}
    \ee{x(t + \Dt) - x(t) \mid x(t) = x'} =
    \int_t^{t + \Dt} A(x, t').
\end{equation}
As a consequence the directional coefficient of the Langevin equation is then
of the form
\begin{equation}
    a^{(1)}(x, t) = \lim_{\Dt \rightarrow 0} {1 \over \Dt}
    \int_t^{t + \Dt} A(x, t').
\end{equation}
Since we are taking the limit when $\Dt \rightarrow 0$ we can approximate the
integral simply as
\begin{equation}
    a^{(1)}(x, t) \approx \lim_{\Dt \rightarrow 0}{1 \over \Dt} [A(x, t) \Dt] =
    A(x, t).
    \label{eq_approx_int_directional}
\end{equation}
We therefore mathematically justify the empirical argument commonly found in
the literature that the directional term associated with the Langevin equation
can be directly associated with the directional term of the Fokker-Planck
equation. In other words we showed that
\begin{equation}
    a^{(1)}(x, t) = A(x, t).
\end{equation}

\subsubsection{Diffusive coefficient of the Fokker-Planck equation}

To compute the second coefficient of the Fokker-Planck equation associated with
the diffusive random displacements we can follow the exact same recipe. First
we know that this coefficient is given by
\begin{equation}
    a^{(2)}(x, t) = \lim_{\Dt \rightarrow 0}
    {\ee{\left[x(t + \Dt) - x(t)\right]^2 \mid x(t) = x'} \over \Dt}.
\end{equation}
Again using the fundamental theorem of calculus gives us
\begin{equation}
    a^{(2)}(x, t) = \lim_{\Dt \rightarrow 0}
    {\ee{ \left. \left[
    \int_t^{t + \Dt} dt'\; {dx \over dt'}
    \right]^2 \right\vert x(t) = x'} \over \Dt}.
    \label{eq_diffusive_fp_lim}
\end{equation}
Substituting \eref{eq_langevin} on the expected value on right-hand side gives
\begin{equation}
    \ee{ \left. \left[
    \int_t^{t + \Dt} dt'\; {dx \over dt'}
    \right]^2 \right\vert x(t) = x'} =
    \ee{ \left. \left[
    \int_t^{t + \Dt} dt'\; A(x, t') +
    \int_t^{t + \Dt} dt'\; B(x, t') \xi(t').
    \right]^2 \right\vert x(t) = x'},
\end{equation}
where we already distributed the integral. We now expand the binomial on the
right-hand side and use the linearity of expected values to obtain
\begin{equation}
    \begin{aligned}
    \ee{ \left. \left[
    \int_t^{t + \Dt} dt'\; A(x, t') +
    \int_t^{t + \Dt} dt'\; B(x, t') \xi(t')
    \right]^2 \right\vert x(t) = x'} =
    \left(
    \int_t^{t + \Dt} dt'\; A(x, t')
    \right)^2\\
    + 2
    \int_t^{t + \Dt} dt'\; A(x, t')
    \int_t^{t + \Dt} dt'\; B(x, t') \ee{ \left.\xi(t')
    \right\vert x(t) = x'}\\
    \left(
    \int_t^{t + \Dt} dt'\; B(x, t') \ee{\left.\xi(t')
    \right\vert x(t) = x'}
    \right)^2,
    \end{aligned}
    \label{eq_int_binom}
\end{equation}
where again we used the fact that the only variable we can average over is the
white noise term $\xi(t)$. We can see that the second term on the right-hand
side cancels since the average of our stochastic variable is equal to zero. For
the two terms left we can rewrite the squared integrals as double integrals.
Let's take a look at each of these terms one at the time. First the integral
involving the directional term takes the form
\begin{equation}
    \left(
        \int_t^{t + \Dt} dt'\; A(x, t')
    \right)^2 =
    \int_t^{t + \Dt} dt'\; \int_t^{t + \Dt} dt''\;
    A(x, t') A(x, t'').
\end{equation}
Just as we did for \eref{eq_approx_int_directional} we can use the fact that we
are working at the limit where $\Dt \rightarrow 0$ to approximate the integral
as
\begin{equation}
    \int_t^{t + \Dt} dt'\; \int_t^{t + \Dt} dt''\;
    A(x, t') A(x, t'') \approx \left[ A(x, t)  \Dt\right]^2.
    \label{eq_directional_int_approx}
\end{equation}
For the integral on \eref{eq_int_binom} involving the diffusive term we again
rewrite it as a double integral of the form
\begin{equation}
    \left(
    \int_t^{t + \Dt} dt'\; B(x, t') \ee{\left.\xi(t')
    \right\vert x(t) = x'}
    \right)^2 =
    \int_t^{t + \Dt} dt'\; \int_t^{t + \Dt} dt''\;
    B(x, t') B(x, t'') \ee{\left. \xi(t') \xi(t'') \right\vert x(t) = x}.
\end{equation}
In \secref{sec_langevin_intro} where we introduced the Langevin equation we
defined the second moment of our noise term as given in \eref{eq_noise_delta}.
Using this we evaluate one of the integrals first obtaining
\begin{equation}
    \int_t^{t + \Dt} dt'\; \int_t^{t + \Dt} dt''\;
    B(x, t') B(x, t'') \ee{\left. \xi(t') \xi(t'') \right\vert x(t) = x} =
    2D \int_t^{t + \Dt} dt'\; B(x, t')^2.
\end{equation}
Finally we can again approximate the integral as
\begin{equation}
    2D \int_t^{t + \Dt} dt'\; B(x, t')^2 \approx 2D B(x, t)^2 \Dt.
    \label{eq_diffusive_int_approx}
\end{equation}
Putting back \eref{eq_directional_int_approx} and
\eref{eq_diffusive_int_approx} into our computation of the second coefficient
of the Fokker-Planck equation in \eref{eq_diffusive_fp_lim} results in
\begin{equation}
    a^{(2)}(x, t) \approx \lim_{\Dt \rightarrow 0} {1 \over \Dt}
    \left[ A(x, t)^2\Dt^2 + 2D B(x, t)^2\Dt \right].
\end{equation}
Once we simplify the corresponding $\Dt$ terms and take the limit, the
directional term goes to zero and we are left with a Fokker-Planck diffusive
coefficient of the form
\begin{equation}
    a^{(2)}(x, t) = 2D B(x, t)^2.
\end{equation}


\section{Exercises in Transformations of Random Variables}\label{S:xsTransformationsOfRVs}
\begin{ExerciseList}
%transformation of RVs
\Exercise
Let $X$ be the outcome of a fair die roll with probability mass function given by
\[
f_X(x) =
\begin{cases}
\frac{1}{6} & \text{ if } x \in \{1,2,3,4,5,6\}\\
0 & \text{ otherwise} \, .
\end{cases}
\]
If $Y = (X-3)^2$ then find the probability mass function of $Y$,  $f_Y (y)$.
\Answer
Using Equation~\eqref{E:PMFOfgOfX}, we can tabulate as follows:

\begin{tabular}{|c|c|c|c|c|}
\hline
$y$ & $0$ & $1$ & $4$ & $9$\\\hline
$f_Y(y)$ & $f_X(3)=\frac{1}{6}$ & $f_X(2)+f_X(4)=\frac{2}{6}$ & $f_X(1)+f_X(5)=\frac{2}{6}$ & $f_X(6)=\frac{1}{6}$\\\hline
\end{tabular}

\Exercise
Given a natural number $n$ as a parameter, i.e., given a parameter $n \in \{1,2,3,\ldots\}$, let $X$ be a discrete uniform random variable on the finite set
$$\mathbb{X}=\{-n,-n+1,\ldots,-1,0,1,\ldots,n-1,n\}$$
i.e.~the probability mass function of $X$ is:
\[
f_X(x;n) =
\begin{cases}
\frac{1}{2n+1} & \text{ if } x \in \mathbb{X} \\
0 & \text{ otherwise}\, .
\end{cases}
\]
Find the probability mass function $f_Y(y;n)$ for $Y=|X|$, the absolute value of $X$.
\Answer
The probability mass function $f_Y(y;n)$ for $Y=|X|$, the absolute value of $X$, comes from applying the formula:
\[
f_Y(y;n) = \sum_{x \in \{ x: g(x)=y\}} f_X(x;n) \enspace ,
\]
as follows:
\[
f_Y(y) =
\begin{cases}
\sum_{x \in \{ x: |x|=0\}} f_X(x;n) = f_X(0;n) = \frac{1}{2n+1} & \text{ if } y =0 \\
 \sum_{x \in \{ x: |x|=y\}} f_X(x;n) = \left( f_X(y;n)+f_X(-y;n) \right) = \frac{2}{2n+1} & \text{ if } y \in \{1,2,\ldots,n\} \\
0 & \text{otherwise}
\end{cases}
\]

\Exercise
If $X$ is a $\geometric(\theta)$ random variable and $Y=\left(\frac{1}{2}\right)^X$ then find an expression for $f_Y(y)$.
\Answer
We are given that $Y=2^{-X}$. Define the function
$$g: \{1,2,3,\ldots\} \to \{2^{-1},2^{-2},2^{-3},\ldots\}$$
by $y=g(x)=2^{-x}$.  Then $g$ is one-to-one and onto and so by Equation~\eqref{E:PMFOfgOfX},
\[
f_Y(y) = \sum_{x \in g^{[-1]}(y)} f_X(x) = \sum_{x \in g^{-1}(y)} f_X(x) = \sum_{x \in \{ -\log_2(y) \} } f_X(x) = f_X(-\log_2(y))\, .
\]
Note that the second equality above is emphasizing that the inverse image $g{[-1]}(y)$ is indeed the inverse function $g^{-1}(y)$ for this $g:\{1,2,3,\ldots\} \to \{2^{-1},2^{-2},2^{-3},\ldots\}$.  Therefore,
\[
f_Y(y) =
\begin{cases}
f_X(-\log_2(y)) = \theta (1-\theta)^{-\log_2(y)-1} & \text{ if } y \in \{2^{-1},2^{-2},2^{-3},\ldots\}\\
0 & \text{ otherwise} \, .
\end{cases}
\]

\Exercise
If $X$ is a $\poisson(\lambda)$ random variable find the probability mass function,  $f_Y (y)$, of
\[
Y\,=\,\frac{1}{(X+1)^{2}} \enspace .
\]
\Answer
Since $X$ is a $\poisson(\lambda)$ random variable (by suppressing the `$;\lambda$' in the argument to $f_X(\cdot)$ for notational ease), we get
\[ f_X(x) \,=\, \P(X=x)\,=\,
\begin{cases}
\frac{\lambda^x \, e^{-\lambda} }{ x!} & \text{ for }  x= 0, 1, 2, \dots \\
0 & \text{ otherwise} \,.
\end{cases}
\]

If $Y=(X+1)^{-2}=1/(X+1)^2$ then  \[ \{\ldots,2,1,0\} \ni x \xmapsto{(x+1)^{-2}} y \,\in\, \left\{1, \frac{1}{4}, \frac{1}{9}, \dots \right\} \]
and since $y=g(x)=(x+1)^{-2}$ {\scriptsize as it maps or associates each $y \in \left\{1, \frac{1}{4}, \frac{1}{9}, \dots \right\}$ to exactly one $x \in \{0,1,2,\ldots\}$ given by $g^{-1}(y)=y^{-1/2}-1 = \frac{1}{\sqrt{y}}-1= x$, its inverse function, this is because $g$ is {\em injective} or {\em one-to-one} as explained here if you want to recall quickly \url{https://en.wikipedia.org/wiki/Injective_function} again}, so we get:
\[
f_Y(y)\;=\; \P(Y=y)\;=\; \sum_{\{ x: g(x)=y\}}f_X(x)  \;=\; f_X\left( \frac{1}{\sqrt y}-1\right) \;=\; \frac{\lambda^{(\frac{1}{\sqrt y} -1)} \, e^{-\lambda}}{(\frac{1}{\sqrt y} -1)!}
\]
for $y = 1, \frac{1}{4}, \frac{1}{9}, \dots $, and $0$ otherwise.\\[4pt]
{\scriptsize CAUTION: This is a discrete RV and so don't just blindly apply the change of variable formula that only applies to continuous RV with a monotone and one-to-one function
$g$ with inverse $g^{-1}$; it's just that in this discrete RV setting the inverse image also happens to satisfy these properties. But Poisson is discrete and `change of variable formula'' is hence inapplicable.}

\Exercise
If $X$ is a continuous random variable with probability density function
\[
f_X(x)\;=\;\begin{cases} x e^{- x} &  x\geqslant 0\\ 0 & x< 0\end{cases},
\]
find the probability density function of $Y\,=\, e^X$.
\Answer
Since  $y=g(x) = e^x$ is a monotone increasing function for $x \geqslant 0$,  we can apply the change of variable formula.

Now $x= g^{-1}(y) = \log_e(y)$ is a monotone increasing function for $y$ in $[1,\infty)$
\[\left|  \frac{d}{dy}\left(g^{-1}(y)\right)\right|\;=\; \left|  \frac{d}{dy}\left(\log_e(y)\right)\right|\;=\; \frac{1}{y}\,.\]

Therefore
\[
  f_Y(y) \;=\; f_X\left( \log_e(y)\right) \times  \left| \frac{1}{y} \right|\;=\;  \log_e(y) \,e^{-\log_e(y)} \times  \frac{1}{y} \;=\; \log_e(y) \, \frac{1}{y^2}
\]
since $e^{-\log_e(y)} = e^{\log_e(y^{-1})}  = y^{-1}$.

So the  probability density
  function  of $Y$ is given by
\[f_Y(y)\;=\;\begin{cases} \displaystyle  \frac{\log_e(y)}{y^2} &   \text{ if } x \geqslant 1 \\ 0 &
  \text{otherwise}
\end{cases} \,.
\]

\Exercise
If $X$, the received power at an antenna is an $\exponential(\lambda)$ random variable then find the probability density function of the amplitude $Y\,=\, \sqrt X$.
\Answer
Since $y=g(x) = \sqrt{x}$ is a monotone increasing function for $x \geqslant 0$,  we can apply the change of variable formula.

Now $x= g^{-1}(y) = y^2$ is a monotone increasing function for $y\geqslant 0$ so on this interval
\[
\left|  \frac{d}{dy}\left(g^{-1}(y)\right)\right|\;=\; \left|  \frac{d}{dy}\left( y^2\right)\right|\;=\; 2y\,.\]
Therefore
\[
f_Y(y)
\;=\; f_X\left(  y^2\right) \times  \left| 2y \right|
\;=\;  \lambda e ^{-\lambda y^2} \times  2y  \;=\;   2 \lambda y e ^{-\lambda y^2}\,.
\]
So the  probability density function  of $Y$ is given by
\[
f_Y(y)\;=\;\begin{cases} \displaystyle  2 \lambda y e ^{-\lambda y^2}   &    y\geqslant 0 \\ 0 & y <0
\end{cases} \,.
\]

\Exercise
If $X$ is a $\uniform(a,b)$ random variable where $0 < a < b$, find  the probability density function, $f_Y (y)$, of \[ Y\,=\, \log_e(X) \, .\]
\Answer
First note that $y=g(x) = \log_e(x)$ is a monotone increasing function over  $a \leqslant x \leqslant b$, so we can apply the change of variable formula.

 $x= g^{-1}(y) = e^y$ is a monotone increasing function over $\log_e(a) \leqslant   \log_e(x) \leqslant \log_e(b)$, that is, over $\log_e(a) \leqslant   y \leqslant   \log_e(b)$.

For $\log_e(a) \leqslant   y \leqslant   \log_e(b)$,

\[\left|  \frac{d}{dy}\left(g^{-1}(y)\right)\right|\;=\; \left|  \frac{d}{dy}\left(e^y)\right)\right|\;=\; e^y\,.\]

Therefore
\[
  f_Y(y) \;=\; f_X\left(g^{-1}(y)\right) \times  \left|  \frac{d}{dy} \left(g^{-1}(y)\right)\right|\;=\; \frac{1}{b-a} \times e^y\,.
\]
So the  probability density
  function  of $Y$ is given by
\[f_Y(y)\;=\;\begin{cases} \displaystyle  \frac{e^y}{b-a}  &   \log_e(a)\leqslant x \leqslant \log_e(b)\\ 0 &
  \text{otherwise}
\end{cases} \,.
\]

\end{ExerciseList}


\subsubsection{Second Order Variation of Parameters}
\noindent
We'll modify our second order equation of have a 1 as the coefficient of the $y''$ term by dividing to get an equation of the form
\begin{equation*}
	y'' + py' + qy = g(x).
\end{equation*}
Just like for undetermined coefficients, we'll find homogeneous and particular solutions $y = y_h + y_p$.
Since the equation is second-order, the solution to the homogeneous equation will yield two fundamental solutions $y_1$ and $y_2$ where $y_h = C_1y_1 + C_2y_2$.\\

\noindent
So, we can write $y$ as
\begin{equation*}
	y(x) = A(x)y_1 + B(x)y_2,
\end{equation*}
where
\begin{equation*}
	\begin{cases}
		A'y_1  + B'y_2  = 0 \\
		A'y_1' + B'y_2' = g(x)
	\end{cases}.
\end{equation*}
We will then solve this system to solve for $A'$ and $B'$ and integrate.

\begin{example}
	Find the general solution to the following equation
	\begin{equation*}
		y'' + y = \csc{x}
	\end{equation*}
	given that $y_h = C_1\cos{x} + C_2\sin{x}$.
\end{example}
\noindent
$y_h$ gives us our two fundamental solutions
\begin{equation*}
	\begin{cases}
		y_1 = \cos{x} \\
		y_2 = \sin{x}
	\end{cases}.
\end{equation*}
So, our system of equations is
\begin{equation*}
	\begin{cases}
		A'\cos{x} + B'\sin{x} = 0 \\
		-A'\sin{x} + B'\cos{x} = \csc{x}
	\end{cases} \to \begin{cases}
		A'\cos{x} + B'\sin{x} = 0 \\
		-A'\cos{x} + B'\frac{\cos^{2}{x}}{\sin{x}} = \frac{\cos{x}}{\sin^{2}{x}}
	\end{cases}.
\end{equation*}
So,
\begin{equation*}
	B'\frac{1}{\sin{x}} = \frac{\cos{x}}{\sin^{2}{x}} \implies B' = \frac{\cos{x}}{\sin{x}} = \cot{x} \implies B = \ln{\abs{\sin{x}}} + C_2,
\end{equation*}
and
\begin{equation*}
	A'\cos{x} + \cos{x} = 0 \implies A' = -1 \implies A = -x + C_1.
\end{equation*}
So, our general solution is
\begin{equation*}
	y = \left(C_1 - x\right)\cos{x} + \left(\ln{\abs{\sin{x}}} + C_2\right)\sin{x} = C_1\cos{x} + C_2\sin{x} - x\cos{x} + \sin{x}\ln{\abs{\sin{x}}}.
\end{equation*}
Note how $y_h$ and $y_p$ appear together.


\subsection{Symbols}

As we saw earlier, symbols are defined using an equals sign.

\begin{Verbatim}[formatcom=\color{blue}]
N = 212^17
\end{Verbatim}

\noindent
No result is printed when a symbol is defined.
To see the value of a symbol, just evaluate it.

\begin{Verbatim}[formatcom=\color{blue}]
N
\end{Verbatim}

\noindent
$\displaystyle N=3529471145760275132301897342055866171392$

\bigskip
\noindent
Symbols can have more that one letter.
Everything after the first letter is displayed as a subscript.

\begin{Verbatim}[formatcom=\color{blue}]
NA = 6.02214 10^23
NA
\end{Verbatim}

\noindent
$\displaystyle N_A=6.02214\times10^{23}$

\bigskip
\noindent
A symbol can be the name of a Greek letter.

\begin{Verbatim}[formatcom=\color{blue}]
xi = 1/2
xi
\end{Verbatim}

\noindent
$\displaystyle \xi=\tfrac{1}{2}$

\bigskip
\noindent
Greek letters can appear in subscripts.

\begin{Verbatim}[formatcom=\color{blue}]
Amu = 2.0
Amu
\end{Verbatim}

\noindent
$\displaystyle A_\mu=2.0$

\bigskip
\noindent
The following example shows how
Eigenmath scans the entire symbol to find Greek letters.

\begin{Verbatim}[formatcom=\color{blue}]
alphamunu = 1
alphamunu
\end{Verbatim}

\noindent
$\displaystyle \alpha_{\mu\nu}=1$

\bigskip
\noindent
When a symbolic chain is defined,
Eigenmath follows the chain as far as possible.
The following example sets $A=B$ followed by $B=C$.
Then when $A$ is evaluated, the result is $C$.

\begin{Verbatim}[formatcom=\color{blue}]
A = B
B = C
A
\end{Verbatim}

\noindent
$\displaystyle A=C$

\bigskip
\noindent
Although $A=C$ is printed,
inside the program the binding of $A$ is still $B$, as can be seen with
the $binding$ function.

\begin{Verbatim}[formatcom=\color{blue}]
binding(A)
\end{Verbatim}

\noindent
$\displaystyle B$

\bigskip
\noindent
The {\it quote} function returns its argument unevaluated
and can be used to clear a symbol.
The following example clears $A$ so that its evaluation goes back to
being $A$ instead of $C$.

\begin{Verbatim}[formatcom=\color{blue}]
A = quote(A)
A
\end{Verbatim}

\noindent
$\displaystyle A$

\subsection{User defined functions}

Most of the functions commonly used in math and physics are included in Eigenmath.
See the Reference section at the end of the manual for a complete list.
There is also a facility for the user to define additional functions.

\bigskip
\noindent
User functions are defined using the syntax {\it function-name} ( {\it arg-list} ) = {\it expr}
where {\it arg-list} is a comma separated list of zero to nine symbols that receive arguments.
Unlike symbol definitions, {\it expr} is not evaluated when {\it function-name} is defined.
Instead, {\it expr} is evaluated when {\it function-name} is used in a subsequent computation.

\bigskip
\noindent
The following example defines a sinc function and evaluates it at $\pi/2$.

\begin{Verbatim}[formatcom=\color{blue}]
f(x) = sin(x)/x
f(pi/2)
\end{Verbatim}

\noindent
$\displaystyle \frac{2}{\pi}$

\bigskip
\noindent
After a user function is defined, {\it expr} can be recalled using the {\it binding} function.

\begin{Verbatim}[formatcom=\color{blue}]
binding(f)
\end{Verbatim}

\noindent
$\displaystyle \frac{\sin(x)}{x}$

\bigskip
\noindent
If local symbols are needed in a function, they can be appended to {\it arg-list}.
(The caller does not have to supply all the arguments.)
The following example uses Rodrigues's formula to
compute an associated Legendre function of $\cos\theta$.
\begin{equation*}
P_n^m(x)=\frac{1}{2^n\,n!}(1-x^2)^{m/2}\frac{d^{n+m}}{dx^{n+m}}(x^2-1)^n
\end{equation*}
Function $P$ below first computes $P_n^m(x)$ for local variable
$x$ and then uses {\it eval} to replace $x$ with $f$.
In this case, $f=\cos\theta$.

\begin{Verbatim}[formatcom=\color{blue}]
x = 123 -- global x in use, need local x in P
P(f,n,m,x) = eval(1/(2^n n!) (1 - x^2)^(m/2) d((x^2 - 1)^n,x,n + m),x,f)
P(cos(theta),2,0) -- arguments f, n, m, but not x
\end{Verbatim}

\noindent
$\displaystyle \tfrac{3}{2} \cos(\theta)^2-\tfrac{1}{2}$

\bigskip
\noindent
The scope of function arguments is limited to the function definition.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% CS624: Analysis of Algorithms
% Copyright 2015 Pejman Ghorbanzade <pejman@ghorbanzade.com>
% Creative Commons Attribution-ShareAlike 4.0 International License
% More info: https://github.com/ghorbanzade/beacon
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section*{Question 1}

The operation \textsc{Heap-Delete}$(A,i)$ deletes the item in node $i$ from heap $A$.
Give an implementation of \textsc{Heap-Delete} that runs in $\mathcal{O}(\log n)$ time for an $n$-element max-heap.
You are as well expected to show that your proposed algorithm run in $\mathcal{O}(\log n)$ time.

\subsection*{Solution}

To remove node $i$ from array $A$ where $A$ is a \textit{Heap} data structure with $n$ elements, we first replace $A[i]$ with $A[n]$; last node in height $H$ where $H$ is height of the tree.
This is done to maintain \textit{completeness} of the binary tree.
Obviously this step has a constant runtime cost which is negligible.
As it is likely that such modification violates the \textit{heap} structure of the tree, we should \textsc{Heapify} the modified binary tree to maintain the heap structure after deletion.

Now the following two cases might occur:

\begin{enumerate}[label=(\alph*)]
\item $A[n] < A[\lceil \frac{i}{2} \rceil ] $\\
If node $n$ is within the subtree with root at node $i$, we can rest assured that the value of node $n$ is less than the value of parent of node $i$, in which case \textsc{Heapify} can only be called for the subtree whose root is node $\lceil \frac{i}{2} \rceil$, to move node $i$ one step lower.

\item $A[n] > A[\lceil \frac{i}{2} \rceil ] $
If there is no subtree that includes nodes $i$ and $n$, the value of node $n$ may be higher than the value of parent of node $i$, in which case each call of \textsc{Heapify} is likely to move node $i$ one step higher.
\end{enumerate}

In either case, the number of calls to \textsc{Heapify} is surely less than or equal to the the height $H$ of the tree.
Since $ 2^{H} - 1 < n $, we can derive $ h < \log {n+1} $ and thus the runtime of proposed \textsc{Heap-Delete} algorithm is $\mathcal{O}(\log n)$.


\section{Divergence Theorem}
\begin{theorem}
	Let $V$ be a compact solid, and let $S$ be its boundary surface. For any differentiable vector field $\vec{F}(x,y,z)$,
	\begin{equation*}
		\oint\limits_{S}{\oint{\vec{F} \cdot \mathrm{d}\vec{s}}} = \iiint\limits_{V}{\nabla \cdot \vec{F}\mathrm{d}V}
	\end{equation*}
	where $\mathrm{d}\vec{s} = \hat{n}\mathrm{d}s = (\vec{r_u}\times\vec{r_v})\mathrm{d}u\mathrm{d}v$ and $\mathrm{d}V$ is the volume differential.
\end{theorem}

[INSERT IMAGE]

\noindent
This says that the flux through a closed surface is equal to the sum of the divergence inside that surface.\\
Intuitively, divergence describes how much a vector field is going in or out at a point, so summing it up inside some solid would tell us the amount the vector field is going in or out on the solid’s boundary, which is flux.\\

\noindent
For example, let’s find the flux through the unit sphere centered at the origin from the vector field $\vec{F}(x,y,z) = \langle x, y, z^2 \rangle$.
\begin{equation*}
	S = \left\{(\rho, \theta, \phi) \mid \rho=1, 0 \leq \theta \leq 2\pi, 0 \leq \phi \leq \pi \right\}
\end{equation*}
\begin{equation*}
	V = \left\{(\rho, \theta, \phi) \mid 0 \leq \rho \leq 1, 0 \leq\ theta \leq 2\pi, 0 \leq \phi \leq \pi \right\}
\end{equation*}
\begin{align*}
	\text{Flux} &= \oint\limits_{S}{\oint{\vec{F} \cdot \mathrm{d}\vec{s}}}	\\
	&= \int_{0}^{1}{\int_{0}^{\pi}{\int_{0}^{2\pi}{\nabla \cdot \langle x, y, z^2 \rangle\rho^2\sin{\phi}\mathrm{d}\theta}\mathrm{d}\phi}\mathrm{d}\rho} \\
	&= \int_{0}^{1}{\int_{0}^{\pi}{\int_{0}^{2\pi}{(2 + 2z)\rho^2\sin{\phi}\mathrm{d}\theta}\mathrm{d}\phi}\mathrm{d}\rho} \\
	&= \int_{0}^{1}{\int_{0}^{\pi}{\int_{0}^{2\pi}{(2 + 2\rho\sin{\phi})\rho^2\sin{\phi}\mathrm{d}}\mathrm{d}}\mathrm{d}} \\
	&= 2\pi\int_{0}^{1}{\int_{0}^{\pi}{2\rho^2\sin{\phi} + 2\rho^3\sin{\phi}\cos{\phi}\mathrm{d}\phi}\mathrm{d}\rho} \\
	&= 4\pi\int_{0}^{\pi}{\frac{1}{3}\sin{\phi} + \frac{1}{4}\sin{\phi}\cos{\phi}\mathrm{d}\phi} \\
	&= \frac{8\pi}{3}
\end{align*}

\input{./vectorAnalysis/gausssLaws}


\lab{Application}{Newton's Method, Julia Sets and Basins of Attraction}{Newton's Method}
\label{lab:NewtonsMethod}
\objective{Understand Newton's Method. Understand definition of basin of attraction.  Basic understanding of Julia Sets.}

One important technique in technical computing is Newton's method.
The goal of Newton's method is to find $\overline{x}$ such that $f\left(\overline{x}\right) = 0$.
This method is especially important in optimization, where our goal is to find minima and maxima of functions.
Newton's method is an iterative method.
In one dimension, it is defined as follows:
\[
x_{n+1} = x_n - \frac{f(x_n)}{f'(x_n)}
\]

Essentially Newton's method approximates a function by its tangent line, and then uses the zero of the tangent line as the next guess for $x_n$.

\begin{figure}[h]
\centering
\includegraphics[width=\textwidth]{newton_iters}
\caption{An illustration of how one iteration of Newton's method works}
\end{figure}

Newton's method is powerful because of the speed of convergence.
In many cases Newton's method converges to the actual root quadratically, meaning that the error term is squared at every iteration.
This fast convergence makes it a very powerful algorithm.

Newton's method does suffer from the flaw that its convergence is dependent upon an initial guess.
If the initial guess is not sufficiently close the convergence can be much slower, or may never occur.
There are even certain pathological functions for which newton's method will never converge.

\begin{problem}
\label{prob:newton_arr}
Write a Newton's method function that runs whether or not the user inputs a derivative function.
In python this can be done by defining the derivative function as a keyword argument that defaults to \li{None}.
Also accept a maximum of how many iterations it will run and a tolerance used for the stopping condition.
Return a tuple with the computed value and boolean value telling whether it converged or not.

Compare the performance of Newton's method when you input the derivative and when you don't.
How well does each converge?
Which runs faster?
Try the following functions:

\begin{itemize}
\item $cos(x)$
\item $x^2sin(\frac{1}{x})$
\item $(\frac{sin(x)}{x})-x$
\item $x^2-1$
\item $x^3 - x$
\end{itemize}

For the last two, look at the results of the iteration on equispaced points between -2 and 2.
Which points converge to which roots?

There are also examples of when Newton's Method does not converge.
Test your newton's method function on $x^{1/3}$.
Have it test at eqispaced starting points on a given interval and return the corresponding values after a given number of iterations.
What do you see?
\end{problem}

\begin{comment}
\begin{problem}
Extend your Newton's method even further so that it will work on systems of equations.
Suppose that $F: \mathbb{R}^n \rightarrow \mathbb{R}^n $.
The relevant equation is
\[
x_{i+1} = x_i - J^{-1}F(x_i)
\]
Note that you should not calculate the inverse Jacobian.
\li{scipy.linalg.solve(A,b)} gives you the solution $x$ to the equation $Ax=b$.
Use this fact to calculate $J^{-1}F$ from $J$ and $F$.
You should be able to make this function work whether or not the user inputs a Jacobian.
This also means that you will have to use your own \li{jacobian} function.
\end{problem}
\end{comment}

\section*{Basins of Attraction and Julia Sets}
Thus far we have only noted the fact that Newton's method converges under good circumstances.
Another topic of interest is determining which starting values converge to which roots.
An plot showing which points converge to which roots for the function $x^2 + x - 2$ is shown in Figure \ref{Fig:basins1}.

\begin{figure}
\begin{center}
\includegraphics[scale=0.5]{basins1}
\caption{The plot of $f(x) = x^2 + x - 2$ along with 20 seed values for Newton's Method.
The green values all converge to the root at -2, and the black values will converge to the root at 1.}
\label{Fig:basins1}
\end{center}
\end{figure}

It turns out that for $f(x) = x^2 + x - 2$, any seed value will converge to one root or the other.
We call the set of points that converge to a single value through an iterative process a basin of attraction.
We can see in Figure \ref{Fig:basins1} a set of seed values that are color coded to indicate which root they converge to with Newton's method.

We now extend these ideas to complex functions.
There is an entire field of mathematics devoted to the study of such functions, but here we only examine some basic properties as they pertain to the generation of Julia sets.
Like other functions we have thus far been exposed to, a complex function maps an input to a unique output.
However, in this case both the input and the output come from the space of complex numbers.

Given a sufficiently ``nice'' complex function, we can apply Newton's method in a similar way to how we applied it in the real case.
However due to the nature of the space we are mapping from and to, we no longer have access to the intuitive visual representations of functions that we saw in the real case.
We can, however, graph the basins of attraction for Newton's method on the complex plane.
For example, let:
\[
f(z) = z^2 - 1
\]
Derivatives for nice complex functions behave much the same way as in the real case:
\[
f'(z) = 2z
\]
It is straightforward to verify that $f$ has two roots at 1 and -1.

In order to better visualize basins of attraction in the complex plane we will use the function li{pcolormesh} from the pyplot library in matplotlib.
The following is an example of how to use this function.
It allows us to color different portions of a plane according to the value of a function at each point.

\begin{lstlisting}
import scipy as sp
from matplotlib import pyplot as plt
n=401
x=sp.linspace(-2,2,n)
y=sp.linspace(-2,2,n)
X,Y=sp.meshgrid(x,y)
def func(x):
    return sp.real(x**3-2*x**2-x+2)
C=func(X+complex(0,1)*Y)
plt.pcolormesh(X,Y,C)
plt.show()
\end{lstlisting}

\begin{problem}
Modify your code from Problem \ref{prob:newton_arr} to graph the basins of attraction for a polynomial in the complex plane.
Examine the basins of attraction for the following complex functions on the following domains.
\begin{itemize}

\item $z^3 - 2 z^2 - 2 z + 2$ on the set of complex numbers $a + b i$ such that $a \in \left[-\frac{1}{2}, 0\right]$ and $b \in \left[-\frac{1}{4}, \frac{1}{4}\right]$.

\item $3 z^3 - 2 z^2 - 2 z + 2$ on the set of complex numbers $a + b i$ such that $a \in \left[-1, 1\right]$ and $b \in \left[-1, 1\right]$.

\item $z^4 + 3 z^3 - 2 z^2 - 2 z + 2$ on the set of complex numbers $a + b i$ such that $a \in \left[-1, 1\right]$ and $b \in \left[-1, 1\right]$.

\item $z^3 - 1$ on the set of complex numbers $a + b i$ such that $a \in \left[-1, 1\right]$ and $b \in \left[-1, 1\right]$.

\end{itemize}
Be aware that Newton's method may not converge for every point in these intervals, so you will have to manually set the pixels that result in a value of \li{nan} to something else.
One good approach is to set the roots of the polynomial to numbers $0, 1, \dots, k$ and then set the values of \li{nan} to $k+1$.

Make sure that when plotting you start with low resolution.
Even if you generate a plot using an array of $100$ by $100$ points, you will be running Newton's method on your test function $10,000$ times, so be careful.
\end{problem}

Another well-studied fractal in the complex plane is the Mandelbrot set.
It is defined as the points $c \in \mathbb{C}$ for which the sequence
\[z_n = z_{n-1}^2 + c\]
is bounded.

\begin{problem}
Generate another grid of complex numbers on $[-1.5,.5]\times[-i,i]$.
Consider the recurrence relation $x_n = x_{n-1}^2 + c$.
Run this iteration 30 times with a resolution of $200 \times 200$ and display the plot.
\end{problem}