\lab{The SVD and Image Compression}{The SVD and Image Compression}
\label{lab:SVD}
\objective{The Singular Value Decomposition (SVD) is an incredibly useful matrix factorization that is widely used in both theoretical and applied mathematics.
The SVD is structured in a way that makes it easy to construct low-rank approximations of matrices, and it is therefore the basis of several data compression algorithms.
In this lab we learn to compute the SVD and use it to implement a simple image compression routine.
}

\begin{comment} % Vague motivation, most of which is in the book.
The \emph{Singular Value Decomposition} or \emph{SVD} is a matrix decomposition that is widely used in both theoretical and applied mathematics.
Originally discovered by theoretical mathematicians, it is a canonical way to decompose a matrix.
Its practical use became apparent later on when Erhard Schmidt showed that the SVD could be a computational tool for providing low-rank matrix approximations.
Modern developments continue to confirm the importance of the SVD in both computational and theoretical applications.

The theoretical use of the \emph{Singular Value Decomposition} or \emph{SVD} has long been appreciated.
In fact, the idea of a canonical way of decomposing a matrix was so alluring that the SVD was independently discovered by at least four people through use of both  integral equations and systems of linear equations.

However, it wasn't until Erhard Schmidt showed how the SVD could be a computational tool for providing low-rank approximations that the practical applications became apparent.
Since Schmidt's work, further developments have confirmed the importance of the SVD in both computational and theoretical applications.
\end{comment}

\begin{comment} % Historical background, super overkill.
The \emph{Singular Value Decomposition} is so important that it was discovered multiple times by different people in different ways.
The foundation work in systems of linear equations was laid by Gauss and Cauchy in the 1820's and later by Jacobi in 1846 with his work with the LU decomposition; however, the real work with the SVD started with Eugenio Beltrami.
In a paper he published in 1873 he was the first to work with the SVD; he was limited to real, square, nonsingular matrices with distinct singular values.
A year later, Camille Jordan published his own independent work that was more rigorous and avoided some of the pitfalls of Beltrami's research.
Together, Beltrami and Jordan are considered the co-discoverers of the singular value decomposition.

Later, James Joseph Sylvester independently presented an iterative algorithm and a rule for carrying out the reduction in 1889.
This rule was essentially Beltrami's work.
However Sylvester sent the note detailing his rule to the same journal Jordan had published in, showing not just his ignorance of Jordan's and Beltrami's works but his perception of the importance of the SVD.

The concurrent and independent derivation of the SVD shows its importance as a theoretical tool, but it wasn't until Erhard Schmidt discovered a computational use for the SVD that the practical applications became apparent.
Schmidt used integral equations rather than linear equations to derive the SVD, but his most important contribution was showing how the SVD could be a computational tool to obtain optimal, low-rank approximations.

Since Schmidt's work, further developments have confirmed the importance of the SVD in both practical and theoretical applications.
\end{comment}

The SVD of a matrix $A$ is a factorization $A = U \Sigma V\hrm$ where $U$ and $V$ have orthonormal columns and $\Sigma$ is diagonal.
The diagonal entries of $\Sigma$ are called the \emph{singular values} of $A$ and are the square roots of the eigenvalues of $A\hrm A$.
Since $A\hrm A$ is always positive semidefinite, its eigenvalues are all real and nonnegative, so the singular values are also real and nonnegative.
The singular values $\sigma_i$ are usually sorted in decreasing order so that $ \Sigma = \mbox{diag}(\sigma_1,\sigma_2,\ldots,\sigma_n)$ with $\sigma_1 \geq \sigma_2 \geq \ldots \geq \sigma_n \geq 0$.
The columns $\mathbf{u}_i$ of $U$, the columns $\mathbf{v}_i$ of $V$, and the singular values of $A$ satisfy $A\mathbf{v}_i = \sigma_i \mathbf{u}_i$.

Every $m\times n$ matrix $A$ of rank $r$ has an SVD with exactly $r$ nonzero singular values.
Like the QR decomposition, the SVD has two main forms.
\begin{itemize}
    \item \textbf{Full SVD}: Denoted $A = U\Sigma V\hrm$.
    $U$ is $m\times m$, $V$ is $n\times n$, and $\Sigma$ is $m\times n$.
    The first $r$ columns of $U$ span $\mathscr{R}(A)$, and the remaining $n -r$ columns span $\mathscr{N}(A\hrm)$.
    Likewise, the first $r$ columns of $V$ span $\mathscr{R}(A\hrm)$, and the last $m - r$ columns span $\mathscr{N}(A)$.
    \item \textbf{Compact (Reduced) SVD}: Denoted $A = U_1\Sigma_1 V\hrm_1$.
    $U_1$ is $m\times r$ (the first $r$ columns of $U$), $V_1$ is $n \times r$ (the first $r$ columns of $V$), and $\Sigma_1$ is $r\times r$ (the first $r\times r$ block of $\Sigma$).
    This smaller version of the SVD has all of the information needed to construct $A$ and nothing more.
    The zero singular values and the correpsonding columns of $U$ and $V$ are neglected.
\end{itemize}
%
\begin{align*} % Full and compace SVD.
\begin{array}{cccc}
\textcolor{red}{U_1\ (m \times r)} & \textcolor{blue}{\Sigma_1\ (r \times r)} & \textcolor{green}{V\hrm_1\ (r \times n)} \\ \\
\left[\begin{array}{ccccccc}
\arrayrulecolor{red}
\cline{2-4}
& \lvl{}     &        & \rvl{}     &          &        &      \\
& \lvl{}     &        & \rvl{}     &          &        &      \\
& \lvl{}     &        & \rvl{}     &          &        &      \\
& \lvl{\u_1} & \cdots & \rvl{\u_r} & \u_{r+1} & \cdots & \u_m \\
& \lvl{}     &        & \rvl{}     &          &        &      \\
& \lvl{}     &        & \rvl{}     &          &        &      \\
& \lvl{}     &        & \rvl{}     &          &        &      \\
\cline{2-4}
\end{array}\right]
&
\left[\begin{array}{cccccccc}
\arrayrulecolor{blue}
\cline{2-4}
& \lvl{\sigma_{1}}  &   & \rvl{}         &   &        &   \\
& \lvl{}       & \ddots & \rvl{}         &   &        &   \\
& \lvl{}       &        & \rvl{\sigma_{r}} & &        &   \\
\cline{2-4}
&              &        &                & 0 &        &   \\
&              &        &                &   & \ddots &   \\
&              &        &                &   &        & 0 \\
\end{array}\right]
&
\left[\begin{array}{ccccccc}
\arrayrulecolor{green}
\cline{2-6}
& \lvl{} & & \v\hrm_1 & & \rvl{} & \\
& \lvl{} & & \vdots   & & \rvl{} & \\
& \lvl{} & & \v\hrm_r & & \rvl{} & \\
\cline{2-6}
& & & \v\hrm_{r+1} & & & \\
& & & \vdots       & & & \\
& & & \v\hrm_n     & & & \\
\end{array}\right]
\\ \\
U\ (m \times m) & \Sigma\ (m \times n) & V\hrm (n \times n) \\
\end{array}
\end{align*}

Finally, the SVD yields an \emph{outer product expansion} of $A$ in terms of the singular values and the columns of $U$ and $V$,
\begin{equation}
\label{eq:svd-outer-product}
A = \sum_{i=1}^r \sigma_i\u_i\v\hrm_i.
\end{equation}
Note that only terms from the compact SVD are needed for this expansion.

\subsection*{Computing the Compact SVD} % -------------------------------------

It is difficult to compute the SVD from scratch because it is an eigenvalue-based decomposition.
However, given an eigenvalue solver such as \li{scipy.linalg.eig()}, the algorithm becomes much simpler.
First, obtain the eigenvalues and eigenvectors of $A\hrm A$, and use these to compute $\Sigma$.
Since $A\hrm A$ is normal, it has an orthonormal eigenbasis, so set the columns of $V$ to be the eigenvectors of $A\hrm A$.
Then, since $A\v_i = \sigma_i \u_i$, construct $U$ by setting its columns to be $\u_i = \frac{1}{\sigma_i}A\v_i$.

The key is to sort the singular values and the corresponding eigenvectors in the same manner.
In addition, it is computationally inefficient to keep track of the entire matrix $\Sigma$ since it is a matrix of mostly zeros, so we need only store the singular values as a vector $\boldsymbol{\sigma}$.
The entire procedure for computing the compact SVD is given below.
% For the compact SVD, keep all of the nonzero singular values.
% For the truncated SVD, keep only the largest $k$.

\begin{algorithm} % Compact SVD.
\begin{algorithmic}[1]
\Procedure{compact\_SVD}{$A$}
\State $\boldsymbol{\lambda}, V \gets$ eig$(A\hrm A)$
    \Comment{Calculate the eigenvalues and eigenvectors of $A\hrm A$.}
\State $\boldsymbol{\sigma} \gets \sqrt{\boldsymbol{\lambda}}$
    \Comment{Calculate the singular values of $A$.}
\State $\boldsymbol{\sigma} \gets$ sort$(\boldsymbol{\sigma})$
    \Comment{Sort the singular values \textbf{from greatest to least}.}
    \label{step:sort-singular-values}
\State $V \gets$ sort($V$)
    \Comment{Sort the eigenvectors \textbf{the same way} as in the previous step.}
\State $r \gets $ count($\boldsymbol{\sigma} \ne 0)$
    \Comment{Count the number of nonzero singular values (the rank of $A$).}
    \label{step:nonzero-singular-values}
\State $\boldsymbol{\sigma}_1 \gets \boldsymbol{\sigma}_{:r}$
    \Comment{Keep only the positive singular values.}
\State $V_1 \gets V_{:,:r}$
    \Comment{Keep only the corresponding eigenvectors.}
\State $U_1 \gets AV_1 / \boldsymbol{\sigma}_1$
    \Comment{Construct $U$ with array broadcasting.}
    \label{step:SVD-construct-U}
\State \pseudoli{return} $U_1, \boldsymbol{\sigma}_1, V\hrm_1$
\EndProcedure
\caption{}
\label{alg:compact-svd}
\end{algorithmic}
\end{algorithm}

\begin{problem} % Compute the compact SVD.
Write a function that accepts a matrix $A$ and a small error tolerance \li{tol}.
Use Algorithm \ref{alg:compact-svd} to compute the compact SVD of $A$.
In step \ref{step:nonzero-singular-values}, compute $r$ by counting the number of singular values that are greater than \li{tol}.

Consider the following tips for implementing the algorithm.
\begin{itemize}
    \item The Hermitian $A\hrm$ can be computed with \li{A.conj().T}.
    \item In step \ref{step:sort-singular-values}, the way that $\boldsymbol{\sigma}$ is sorted needs to be stored so that the columns of $V$ can be sorted the same way.
    Consider using \li{np.argsort()} and fancy indexing to do this, but remember that by default it sorts from least to greatest (not greatest to least).
    \item Step \ref{step:SVD-construct-U} can be done by looping over the columns of $V$, but it can be done more easily and efficiently with array broadcasting.
\end{itemize}

Test your function by calculating the compact SVD for random matrices.
Verify that $U$ and $V$ are orthonormal, that $U\Sigma V\hrm = A$, and that the number of nonzero singular values is the rank of $A$.
You may also want to compre your results to SciPy's SVD algorithm.
%
\begin{lstlisting}
>>> import numpy as np
>>> from scipy import linalg as la

# Generate a random matrix and get its compact SVD via SciPy.
>>> A = np.random.random((10,5))
>>> U,s,Vh = la.svd(A, full_matrices=False)
>>> print(U.shape, s.shape, Vh.shape)
(10, 5) (5,) (5, 5)

# Verify that U is orthonormal, U Sigma Vh = A, and the rank is correct.
>>> np.allclose(U.T @ U, np.identity(5))
<<True>>
>>> np.allclose(U @ np.diag(s) @ Vh, A)
<<True>>
>>> np.linalg.matrix_rank(A) == len(s)
<<True>>
\end{lstlisting}
\label{prob:calculate-compact-svd}
\end{problem}

\subsection*{Visualizing the SVD} % -------------------------------------------

An $m\times n$ matrix $A$ defines a linear transformation that sends points from $\mathbb{R}^n$ to $\mathbb{R}^m$.
The SVD decomposes a matrix into two rotations and a scaling, so that any linear transformation can be easily described geometrically.
Specifically, $V\hrm$ represents a rotation, $\Sigma$ a rescaling along the principal axes, and $U$ another rotation.

\begin{figure}
\centering
\begin{subfigure}{.35\textwidth}
  \centering
  \includegraphics[width=\textwidth]{figures/unit_circle.pdf}
  \caption{$S$}
\end{subfigure}
\quad
\begin{subfigure}{.35\textwidth}
  \centering
  \includegraphics[width=\textwidth]{figures/vcircle.pdf}
  \caption{$V\hrm S$}
\end{subfigure}
\\
\begin{subfigure}{.35\textwidth}
  \centering
  \includegraphics[width=\textwidth]{figures/svcircle.pdf}
  \caption{$\Sigma V\hrm S$}
\end{subfigure}
\quad
\begin{subfigure}{.35\textwidth}
  \centering
  \includegraphics[width=\textwidth]{figures/full_transformation.pdf}
  \caption{$U \Sigma V\hrm S$}
\end{subfigure}
\caption{Each step in transforming the unit circle and two unit vectors using the matrix $A$.}
\label{fig:svd-visualization}
\end{figure}

\begin{problem} % Visualize the SVD.
Write a function that accepts a $2 \times 2$ matrix $A$.
Generate a $2 \times 200$ matrix $S$ representing a set of 200 points on the unit circle, with $x$-coordinates on the top row and $y$-coordinates on the bottom row (recall the equation for the unit circle in polar coordinates:
$x = \cos(\theta)$, $y = \sin(\theta)$, $\theta\in[0,2\pi]$).
Also define the matrix
\[
E =
\left[\begin{array}{c|c|c}
\arrayrulecolor{gray}
\e_1 & \0 & \e_2
\end{array}\right]
=
\left[\begin{array}{ccc} 1 & 0 & 0 \\ 0 & 0 & 1 \end{array}\right],
\]
so that plotting the first row of $S$ against the second row of $S$ displays the unit circle, and plotting the first row of $E$ against its second row displays the standard basis vectors in $\mathbb{R}^2$.

Compute the full SVD $A = U\Sigma V\hrm$ using \li{scipy.linalg.svd()}.
Plot four subplots to demonstrate each step of the transformation, plotting $S$ and $E$, $V\hrm S$ and $V\hrm E$, $\Sigma V\hrm S$ and $\Sigma V\hrm E$, then $U\Sigma V\hrm S$ and $U \Sigma V\hrm E$.

For the matrix \[A =  \left[\begin{array}{cc}3 & 1\\1 & 3\end{array}\right],\]
your function should produce Figure \ref{fig:svd-visualization}.
\\
(Hint: Use \li{plt.axis("equal")} to fix the aspect ratio so that the circles don't appear elliptical.)
\end{problem}

\begin{comment}
\subsection*{Image Data Compression}
In this lab, we explore how the SVD can be used to compress image data.
Recall that an image is simply a matrix where each position is the color value for the pixel in that position.
The SVD lets us choose how much information to keep, and what information is most important.
Larger eigenvalues correspond to columns of $U$ and $V$ that contain more information, while smaller eigenvalues correspond to less important columns.
This idea is used in many areas of applied mathematics including signal processing, statistics, semantic indexing (search engines), and control theory.
\end{comment}

\section*{Using the SVD for Data Compression} % ===============================

\subsection*{Low-Rank Matrix Approximations} % --------------------------------

If $A$ is a $m\times n$ matrix of rank $r < \min\{m,n\}$, then the compact SVD offers a way to store $A$ with less memory.
Instead of storing all $mn$ values of $A$, storing the matrices $U_1$, $\Sigma_1$ and $V_1$ only requires saving a total of $mr+r+nr$ values.
For example, if $A$ is $100 \times 200$ and has rank $20$, then $A$ has $20,000$ values, but its compact SVD only has total $6,020$ entries, a significant decrease.

The \emph{truncated SVD} is an approximation to the compact SVD that allows even greater efficiency at the cost of a little accuracy.
Instead of keeping all of the nonzero singular values, the truncated SVD only keeps the first $s < r$ singular values, plus the corresponding columns of $U$ and $V$.
In this case, (\ref{eq:svd-outer-product}) becomes
\begin{equation*}
A_s = \sum_{i=1}^s \sigma_i\u_i\v\hrm_i.
\end{equation*}

More precisely, the truncated SVD of $A$ is $A_s = \widehat{U}\widehat{\Sigma} \widehat{V}\hrm$, where
$\widehat{U}$ is $m\times s$, $\widehat{V}$ is $n \times s$, and $\widehat{\Sigma}$ is $s\times s$.
The resulting matrix $A_s$ has rank $s$ and is only an approximation to $A$, since $r - s$ nonzero singular values are neglected.

\begin{align*} % Compact and truncated SVD.
\begin{array}{cccc}
\textcolor{red}{\widehat{U}\ (m \times s)} & \textcolor{blue}{\widehat{\Sigma}\ (s \times s)} & \textcolor{green}{\widehat{V}\hrm\ (s \times n)} \\
\left[\begin{array}{ccccccc}
\arrayrulecolor{red}
\cline{2-4}
& \lvl{}     &        & \rvl{}     &          &        &      \\
& \lvl{}     &        & \rvl{}     &          &        &      \\
& \lvl{}     &        & \rvl{}     &          &        &      \\
& \lvl{\u_1} & \cdots & \rvl{\u_s} & \u_{s+1} & \cdots & \u_r \\
& \lvl{}     &        & \rvl{}     &          &        &      \\
& \lvl{}     &        & \rvl{}     &          &        &      \\
& \lvl{}     &        & \rvl{}     &          &        &      \\
\cline{2-4}
\end{array}\right]
&
\left[\begin{array}{cccccccc}
\arrayrulecolor{blue}
\cline{2-4}
& \lvl{\sigma_{1}}  &   & \rvl{}         &   &        &   \\
& \lvl{}       & \ddots & \rvl{}         &   &        &   \\
& \lvl{}       &        & \rvl{\sigma_{s}} & &        &   \\
\cline{2-4}
&              &        &                & \sigma_{s+1} &        &   \\
&              &        &                &   & \ddots &   \\
&              &        &                &   &        & \sigma_r \\
\end{array}\right]
&
\left[\begin{array}{ccccccc}
\arrayrulecolor{green}
\cline{2-6}
& \lvl{} & & \v\hrm_1 & & \rvl{} & \\
& \lvl{} & & \vdots   & & \rvl{} & \\
& \lvl{} & & \v\hrm_s & & \rvl{} & \\
\cline{2-6}
& & & \v\hrm_{s+1} & & & \\
& & & \vdots       & & & \\
& & & \v\hrm_r     & & & \\
\end{array}\right]
\\
U_1\ (m \times r) & \Sigma_1\ (r \times r) & V\hrm_1 (r \times n) \\
\end{array}
\end{align*}

The beauty of the SVD is that it makes it easy to select the information that is most important.
Larger singular values correspond to columns of $U$ and $V$ that contain more information, so dropping the smallest singular values retains as much information as possible.
In fact, given a matrix $A$, its rank-$s$ truncated SVD approximation $A_s$ is the \emph{best rank $s$ approximation} of $A$ with respect to both the induced 2-norm and the Frobenius norm.
This result is called the \emph{Schmidt, Mirsky, Eckhart-Young theorem}, a very significant concept that appears in signal processing, statistics, machine learning, semantic indexing (search engines), and control theory.

\begin{comment}
We can also calculate $A_s$ by finding the full SVD, and setting all singular values after the $k$th to zero.
Thus the modified $\Sigma$ would be
\begin{equation*}
\Sigma_{s} = \mbox{diag}(\sigma_1,\sigma_2,\ldots,\sigma_s,0,\ldots,0).
\end{equation*}
Multiplying this matrix with the original $U$ and $V\hrm$ will give the same $A_s$ that was found by computing the truncated SVD directly.
\end{comment}

\begin{problem} % Lowest-rank approximation.
Write a function that accepts a matrix $A$ and a positive integer $s$.
\begin{enumerate}
\item Use your function from Problem \ref{prob:calculate-compact-svd} or \li{scipy.linalg.svd()} to compute the compact SVD of $A$, then form the truncated SVD by stripping off the appropriate columns and entries from $U_1$, $\Sigma_1$, and $V_1$.
Return the best rank $s$ approximation $A_s$ of $A$ (with respect to the induced 2-norm and Frobenius norm).
\item Also return the number of entries required to store the truncated form $\widehat{U}\widehat{\Sigma} \widehat{V}\hrm$ (where $\widehat{\Sigma}$ is stored as a one-dimensional array, not the full diagonal matrix).
The number of entries stored in NumPy array can be accessed by its \li{size} attribute.
\begin{lstlisting}
>>> A = np.random.random((20, 20))
>>> A.size
400
\end{lstlisting}
\item If $s$ is greater than the number of nonzero singular values of $A$ (meaning $s > $ rank$(A)$), raise a \li{ValueError}.
\end{enumerate}
Use \li{np.linalg.matrix_rank()} to verify the rank of your approximation.
% , and verify that the number of entries required is equal to $ms + s + ns$.
\label{prob:svd_approx}
\end{problem}

\subsection*{Error of Low-Rank Approximations} % ------------------------------

Another result of the Schmidt, Mirsky, Eckhart-Young theorem is that the exact 2-norm error of the best rank-$s$ approximation $A_s$ for the matrix $A$ is the $(s+1)$th singular value of $A$:
\begin{equation}
\label{eq:svd-approximation-error}
\|A - A_s\|_2 = \sigma_{s+1}.
\end{equation}
%
This offers a way to approximate $A$ within a desired error tolerance $\epsilon$:
choose $s$ such that $\sigma_{s+1}$ is the largest singular value that is less than $\epsilon$, then compute $A_s$.
This $A_s$ throws away as much information as possible without violating the property $\|A - A_s\|_2 < \epsilon$.

\begin{problem} % Lowest Rank Approximation
Write a function that accepts a matrix $A$ and an error tolerance $\epsilon$.
\begin{enumerate}
\item Compute the compact SVD of $A$, then use (\ref{eq:svd-approximation-error}) to compute the lowest rank approximation $A_s$ of $A$ with 2-norm error less than $\epsilon$.
Avoid calculating the SVD more than once.
\\ (Hint: \li{np.argmax()}, \li{np.where()}, and/or fancy indexing may be useful.)
\item As in the previous problem, also return the number of entries needed to store the resulting approximation $A_s$ via the truncated SVD.
\item If $\epsilon$ is less than or equal to the smallest singular value of $A$, raise a \li{ValueError}; in this case, $A$ cannot be approximated within the tolerance by a matrix of lesser rank.
\end{enumerate}
This function should be close to identical to the function from Problem \ref{prob:svd_approx}, but with the extra step of identifying the appropriate $s$.
Construct test cases to validate that $\| A - A_s \|_2 < \epsilon$.
\end{problem}

\subsection*{Image Compression} % ---------------------------------------------

Images are stored on a computer as matrices of pixel values.
Sending an image over the internet or a text message can be expensive, but computing and sending a low-rank SVD approximation of the image can considerably reduce the amount of data sent while retaining a high level of image detail.
Successive levels of detail can be sent after the inital low-rank approximation by sending additional singular values and the corresponding columns of V and U.

% TODO: redo the numbers in this paragraph for the full hubble image.
Examining the singular values of an image gives us an idea of how low-rank the approximation can be.
Figure \ref{fig:hubble} shows the image in \texttt{hubble\_gray.jpg} and a log plot of its singular values.
The plot in \ref{fig:hubble-log-svals} is typical for a photograph---the singular values start out large but drop off rapidly.
In this rank $1041$ image, $913$ of the singular values are $100$ or more times smaller than the largest singular value.
By discarding these relatively small singular values, we can retain all but the finest image details, while storing only a rank $128$ image.
This is a \textbf{huge} reduction in data size.

\begin{figure}[H] % Hubble image + plot of singular values.
\captionsetup[subfigure]{justification=centering}
\centering
\begin{subfigure}{.49\textwidth}
    \centering
    \includegraphics[height=5cm]{figures/hubble_gray.jpg}
    \caption{NGC 3603 (Hubble Space Telescope).}
    \label{fig:hubble-original-gray}
\end{subfigure}
%
\begin{subfigure}{.49\textwidth}
    \centering
    \includegraphics[width=\linewidth]{figures/hubble_svals.pdf}
    \caption{Singular values on a log scale.}
    \label{fig:hubble-log-svals}
\end{subfigure}
\caption{}
\label{fig:hubble}
\end{figure}

Figure \ref{fig:hubble-svd-rank-approximations} shows several low-rank approximations of the image in Figure \ref{fig:hubble-original-gray}.
Even at a low rank the image is recognizable.
By rank $120$, the approximation differs very little from the original.

\begin{figure}[H]
\centering
\begin{subfigure}{.32\textwidth}
    \centering
    \includegraphics[width=\textwidth]{figures/rank2.pdf}
    \caption{Rank 2}
\end{subfigure}
%
\begin{subfigure}{.32\textwidth}
    \centering
    \includegraphics[width=\textwidth]{figures/rank20.pdf}
    \caption{Rank 20}
    \label{fig:hubble-rank20-approximation}
\end{subfigure}
%
\begin{subfigure}{.32\textwidth}
    \centering
    \includegraphics[width=\textwidth]{figures/rank120.pdf}
    \caption{Rank 120}
\end{subfigure}
\caption{}
\label{fig:hubble-svd-rank-approximations}
\end{figure}

Grayscale images are stored on a computer as 2-dimensional arrays, while color images are stored as 3-dimensional arrays---one layer each for red, green, and blue arrays.
To read and display images, use \li{imageio.imread()} and \li{plt.imshow()}.
Images are read in as integer arrays with entries between 0 and 255 (\li{dtype=np.uint8}), but \li{plt.imshow()} works better if the image is an array of floats in the interval $[0,1]$.
Scale the image properly by dividing the array by $255$.

\begin{lstlisting}
>>> from imageio import imread
>>> from matplotlib import pyplot as plt

# Send the RGB values to the interval (0,1).
>>> image_gray = imread("hubble_gray.jpg") / 255.
>>> image_gray.shape            # Grayscale images are 2-d arrays.
(1158, 1041)
>>> image_color = imread("hubble.jpg") / 255.
>>> image_color.shape           # Color images are 3-d arrays.
(1158, 1041, 3)

# The final axis has 3 layers for red, green, and blue values.
>>> red_layer = image_color[:,:,0]
>>> red_layer.shape
(1158, 1041)

# Display a gray image.
>>> plt.imshow(red_layer, cmap="gray")
>>> plt.axis("off")             # Turn off axis ticks and labels.
>>> plt.show()

# Display a color image.
>>> plt.imshow(image_color)     # cmap=None by default.
>>> plt.axis("off")
>>> plt.show()
\end{lstlisting}

\begin{problem} % Image compression.
Write a function that accepts the name of an image file and an integer $s$.
Use your function from Problem \ref{prob:svd_approx}, to compute the best rank-$s$ approximation of the image.
Plot the original image and the approximation in separate subplots.
In the figure title, report the difference in number of entries required to store the original image and the approximation (use \li{plt.suptitle()}).

Your function should be able to handle both grayscale and color images.
Read the image in and check its dimensions to see if it is color or not.
Grayscale images can be approximated directly since they are represented by 2-dimensional arrays.
For color images, let $R$, $G$, and $B$ be the matrices for the red, green, and blue layers of the image, respectively.
Calculate the low-rank approximations $R_s$, $G_s$, and $B_s$ separately, then put them together in a new 3-dimensional array of the same shape as the original image.
\\ (Hint: \li{np.dstack()} may be useful for putting the color layers back together.)

Finally, it is possible for the low-rank approximations to have values slightly outside the valid range of RGB values.
Set any values outside of the interval $[0,1]$ to the closer of the two boundary values.
\\ (Hint: fancy indexing and/or \li{np.clip()} may be useful here.)

To check, compressing \texttt{hubble\_gray.jpg} with a rank $20$ approximation should appear similar to Figure \ref{fig:hubble-rank20-approximation} and save $1,161,478$ matrix entries.
\end{problem}

% NOTE: we used to use A.nbytes instead of A.size to measure the reduction,
% but when images (dtype=np.uint8) are converted to float arrays
% (dtype=np.float64), the resulting arrays take much more memory to store, so =
% reporting the number of bytes saved often ends up being bigger than the
% original picture.

\newpage

\section*{Additional Material} % ==============================================

\subsection*{More on Computing the SVD} % -------------------------------------

For an $m\times n$ matrix $A$ of rank $r < \min\{m,n\}$, the compact SVD of $A$ neglects last $m-r$ columns of $U$ and the last $n-r$ columns of $V$.
The remaining columns of each matrix can be calculated by using Gram-Schmidt orthonormalization.
If $m < r < n$ or $n < r < m$, only one of $U_1$ and $V_1$ will need to be filled in to construct the full $U$ or $V$.
Computing these extra columns is one way to obtain a basis for $\mathscr{N}(A\hrm)$ or $\mathscr{N}(A)$.

Algorithm \ref{alg:compact-svd} begins with the assumption that we have a way to compute the eigenvalues and eigenvectors of $A\hrm A$.
Computing eigenvalues is a notoriously difficult problem, and computing the SVD from scratch without an eigenvalue solver is much more difficult than the routine described by Algorithm \ref{alg:compact-svd}.
The procedure involves two phases:
\begin{enumerate}
    \item Factor $A$ into $A = U_a B V\hrm_a$ where $B$ is bidiagonal (only nonzero on the diagonal and the first superdiagonal) and $U_a$ and $V_a$ are orthonormal.
    This is usually done via \emph{Golub-Kahan Bidiagonalization}, which uses Householder reflections, or \emph{Lawson-Hanson-Chan bidiagonalization}, which relies on the QR decomposition.
    \item Factor $B$ into $B = U_b \Sigma V\hrm_b$ by the QR algorithm or a divide-and-conquer algorithm.
    Then the SVD of $A$ is given by $A = (U_aU_b)\Sigma (V_aV_b)\hrm$.
\end{enumerate}
For more details, see Lecture 31 of \cite{Trefethen1997} or Section 5.4 of \emph{Applied Numerical Linear Algebra} by James W. Demmel.

\subsection*{Animating Images with Matplotlib} % ------------------------------

Matplotlib can be used to animate images that change over time.
For instance, we can show how the low-rank approximations of an image change as the rank $s$ increases, showing how the image is recovered as more ranks are added.
Try using the following code to create such an animation.

\begin{lstlisting}
from matplotlib import pyplot as plt
from matplotlib.animation import FuncAnimation

def animate_images(images):
    """Animate a sequence of images. The input is a list where each
    entry is an array that will be one frame of the animation.
    """
    fig = plt.figure()
    plt.axis("off")
    im = plt.imshow(images[0], animated=True)

    def update(index):
        plt.title("Rank {} Approximation".<<format>>(index))
        im.set_array(images[index])
        return im,              # Note the comma!

    a = FuncAnimation(fig, update, frames=len(images), blit=True)
    plt.show()
\end{lstlisting}
See \url{https://matplotlib.org/examples/animation/dynamic_image.html} for another example.


\section{Data-analysis tips}\label{lab-1-data-analysis-tips}

In lab one you will be collecting measurements on several dependent
variables, in each of two manipulated conditions (the independent
variable). For each dependent variable you will want to determine
whether the manipulation had an effect. That is, did the independent
variable cause a change in the dependent variable. We know that
differences can sometimes be observed by chance alone, so we want to
conduct an inferential statistical test to determine the probability
that our observed difference could have been produced by chance alone.
To do this we will be conducting several t-tests. This is a short primer
on the process. You can conduct t-tests in the software of your choice,
or by hand using a calculator (or in excel). Here, we will use the free
and open-source statistical package called R, to illustrate the process.

Let's imagine we have two groups of 10 subjects each. Group A receives
condition 1 of the independent variable, and Group B recieves condition
2 of the independent variable. We then measure some behavior for all of
the subject in all of the groups. To make this more concrete, let's say
10 subjects drink coffee, and the the 10 subjects drink tea. Then we
present all of the subjects with a piece of art and ask them rate how
beautiful they think it is on a scale from 1 to 7.

When we collect all the data we should have 20 total ratings, one for
each subject in each group.

For example, if you put the data in a table it might look something like
the following. Note, the grey text box shows the R code used to simulate
the data. For, each group, we sample 10 numbers from a normal
distribution with a mean of 4, and a standard deviation of .5. Then we
put the numbers in a table.

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{coffee<-}\KeywordTok{round}\NormalTok{(}\KeywordTok{rnorm}\NormalTok{(}\DecValTok{10}\NormalTok{,}\DecValTok{4}\NormalTok{,.}\DecValTok{5}\NormalTok{))}
\NormalTok{tea<-}\KeywordTok{round}\NormalTok{(}\KeywordTok{rnorm}\NormalTok{(}\DecValTok{10}\NormalTok{,}\DecValTok{4}\NormalTok{,.}\DecValTok{5}\NormalTok{))}
\NormalTok{all_data<-}\KeywordTok{data.frame}\NormalTok{(coffee,tea)}
\KeywordTok{kable}\NormalTok{(all_data,}\DataTypeTok{format=}\StringTok{"latex"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{tabular}{r|r}
\hline
coffee & tea\\
\hline
4 & 5\\
\hline
3 & 4\\
\hline
3 & 4\\
\hline
3 & 3\\
\hline
4 & 3\\
\hline
4 & 5\\
\hline
4 & 4\\
\hline
4 & 4\\
\hline
4 & 4\\
\hline
3 & 4\\
\hline
\end{tabular}

We can do some quick descriptive statistics, for example, we might want
to know the means of the beauty ratings for the coffee and tea groups.

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{mean}\NormalTok{(coffee)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 3.6
\end{verbatim}

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{mean}\NormalTok{(tea)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 4
\end{verbatim}

The means aren't very different, and of course we should expect they
should be similar. After all, we sampled these means from the exact same
distribution. So, we should expect that on average, the means should
both be close to 4. However, they won't necessarilly be exactly 4,
because of variability introduced by random sampling.

\section{The t-test}\label{the-t-test}

What we want to do next is conduct an independent samples t-test. We
want to determine whether any possible difference between the coffee and
tea groups could have been produced by chance alone. We can conduct a
t-test in R very easily using the t.test function.

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{t.test}\NormalTok{(coffee,tea,}\DataTypeTok{var.equal=}\OtherTok{TRUE}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##
##  Two Sample t-test
##
## data:  coffee and tea
## t = -1.5, df = 18, p-value = 0.151
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##  -0.9602459  0.1602459
## sample estimates:
## mean of x mean of y
##       3.6       4.0
\end{verbatim}

R gives us back the t values, the degrees of freedom (df), and the
associated p-value. The p-value tells us the likelihood that our
difference, or a difference greater than the one we observed could have
been produced by chance.

\section{One more time}\label{one-more-time}

Let's try this whole process again, but this time we will simulate data
with an actual difference between the groups. For example, let's say we
want to simulate the idea that drinking coffee makes people think the
art is less beautiful by at least 2 points, and then reconduct the
t-test with the new simulated data. We will sample numbers from a normal
distribution with mean 3 for the coffee group, and mean 5 for the tea
group (for an average expected difference of 2).

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{coffee<-}\KeywordTok{round}\NormalTok{(}\KeywordTok{rnorm}\NormalTok{(}\DecValTok{10}\NormalTok{,}\DecValTok{3}\NormalTok{,.}\DecValTok{5}\NormalTok{))}
\NormalTok{tea<-}\KeywordTok{round}\NormalTok{(}\KeywordTok{rnorm}\NormalTok{(}\DecValTok{10}\NormalTok{,}\DecValTok{5}\NormalTok{,.}\DecValTok{5}\NormalTok{))}
\NormalTok{all_data<-}\KeywordTok{data.frame}\NormalTok{(coffee,tea)}
\KeywordTok{kable}\NormalTok{(all_data,}\DataTypeTok{format=}\StringTok{"latex"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{tabular}{r|r}
\hline
coffee & tea\\
\hline
3 & 5\\
\hline
3 & 5\\
\hline
3 & 4\\
\hline
3 & 6\\
\hline
2 & 5\\
\hline
4 & 5\\
\hline
3 & 6\\
\hline
3 & 5\\
\hline
3 & 5\\
\hline
3 & 5\\
\hline
\end{tabular}

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{mean}\NormalTok{(coffee)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 3
\end{verbatim}

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{mean}\NormalTok{(tea)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 5.1
\end{verbatim}

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{t.test}\NormalTok{(coffee,tea,}\DataTypeTok{var.equal=}\OtherTok{TRUE}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##
##  Two Sample t-test
##
## data:  coffee and tea
## t = -9, df = 18, p-value = 4.404e-08
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##  -2.590215 -1.609785
## sample estimates:
## mean of x mean of y
##       3.0       5.1
\end{verbatim}

\section{Writing up the results of a
t-test}\label{writing-up-the-results-of-a-t-test}

We've now conducted two different t-tests, and received different
results on each them. You will likely find different results for all of
the t-tests that you conduct for the lab experiment. However, you will
use the basic sentence structure to report all of the results. When you
report the results of your experiment along with statistical tests there
are two important features to include, the pattern of the results, and
the inferential statistic. In this situation, we would simply report the
means and the t-test information. Here are is an with made-up numbers.

The coffee group gave a lower mean beauty rating (M = 3.4) than the tea
group (M = 5.6), and the difference was significant, t (18) = 5.4, p
\textless{} .001.

So, just in one sentence we tell the reader what the means were in both
conditions, as whether the result was significant. APA style recommends
reporting exact p-values when they are greater than .001 (for example p
= .047). If the p-value is less than .001, then you just need to report
p \textless{} .001.


\label{sec:registration}

This section discusses functions that are minimized
to register (align) one mesh to another,
to register a mesh to a data set,
or to register a data set to a mesh.
The general approach is to minimize
a measure of distance between the {\it docking} mesh and its {\it target},
over some set of registration transforms,
for example, the rigid transformations of $\Reals^3$.

We most often compute a registration transform, $\Tr$,
to minimize a distance between the transformed mesh,
$\Tr\M$ and its target, $\Ta$:
\begin{equation}
\min_{\Tr} d(\M(\Tr),\Ta)
\end{equation}
The target, $\Ta$, may be another mesh, a set of data points,
or some other geometric object.
The transformed mesh
$\M(\Tr) = \M(\Tr\v) = \M(\Tr(\v_0 \ldots \v_{n-1}))$,
where $\v = (\v_0 \ldots  \v_{n-1})$
are the positions of the vertices of $\M$.

We are also sometimes interested in registering a data set
$\{\x_i\}$ to a mesh.
In this case, we minimize
\begin{equation}
\min_{\Tr} d(\M,\Tr\{\x_i\})
\end{equation}

Both are special cases of choosing a transform on $\Reals^{3n}$
by minimizing:
\begin{equation}
\min_{\Tr} f( \Tr \p),
\end{equation}
where
$\p = (\p_0 \ldots  \p_{n-1}) \in \Reals^{3n}$,
$\Tr : \Reals^{3n} \mapsto \Reals^{3n}$,
and
$f : \Reals^{3n} \mapsto \Re$.

\subsection{Distance measures}
\label{sec:Distance-measures}

\subsubsection{Vertex distance}
\label{sec:Vertex-distance}

Suppose 2 meshes have corresponding vertices,
that is,
for each vertex $\v_{0i}$ in the docking mesh $\M_0$,
there is a corresponding vertex $\v_{1i}$ in the target mesh $\M_1$,
and vice versa.
Then an obvious measure of distance
is the sum of distances between the corresponding points:
\begin{equation}
d_{\V}(\M_0,\M_1) = \sum_{i=0}^{n-1} \| \p_{0i} - \p_{1i} \|^2
\end{equation}
The gradient of $d_v$ with respect to the positions
of the $i$th vertex of $\M_0$ is:
\begin{equation}
\Gc{\p_{0i}}{d_v}{q} = 2 \left[ \q_{0i} - \p_{1i} \right]
\end{equation}

\subsubsection{Projection distance}
\label{sec:Projection-distance}

In many problems, the vertex correspondence used in
\autoref{sec:Vertex-distance} is not appropriate.
Or we may want to register a mesh to a target data set.
In such cases, we can use the projection distance:
\begin{equation}
d_{\Pr}(\M,\p_1) = \sum_{i=0}^{n-1} \| \Pr_{\M}(\p_{1i}) - \p_{1i} \|^2,
\end{equation}
where the target $\p_1 = (\p_{10}, \ldots \p_{1(n-1)})$
is either a set of data points,
or the set of positions of the vertices of a target mesh.
$\Pr_{\M}(\p)$ is the projection of the point $\p \in \Reals^3$
on the mesh $\M$.
This is the same as the general data fitting distance, and
derivatives with respect to the positions of the vertices
of $\M$ are given in \autoref{sec:data-fitting}.

\subsection{Transforms}
\label{sec:Transforms}

In this section, I describe common families
of transforms, $\{\Tr\}$, over which to minimize:
\begin{equation}
\min_{\Tr} f( \Tr \p),
\end{equation}
To do the minimization, we need to compute
the derivative:
\begin{equation}
\De{\Tr}{g(\Tr)}{\Tr_0}
= \De{\Tr}{f(\Tr\p)}{\Tr_0}
= \De{\q}{f(\q)}{\q=\Tr_0\p}
\circ
\De{\Tr}{\Tr\p}{\Tr_0}.
\end{equation}
o, equivalently, the gradient:
\begin{equation}
\Gc{\Tr}{f( \M(\Tr\p) )}{\Tr_0}
 =
\De{\Tr}{\Tr\p}{\Tr_0}^\dagger
\Gc{\q}{f(\M(\q)))}{\Tr_0\p}
\end{equation}

I assume in the following
that $\Df{\p}{f}$ (or $\Gb{\p}{f}$) is known,
so the main task is computing $\Df{\Tr}{\Tr\p}$.

As mentioned in above,
in registration problems
the function $f$ is usually a distance between
two meshes, or between a mesh and a data set,
with either the location of one mesh's vertices
or the data points allowed to vary.
Derivatives and gradients of distance functions,
with respect to vertex positions,
are given in sections \ref{sec:Distance-measures}
and \ref{sec:data-fitting}.

\subsubsection{Direct sum transforms}
\label{sec:Direct-sum-transforms}

In registration problems,we usually want to
apply the same transform to each vertex or data point.
Let $\Tr : \Reals^3 \mapsto \Reals^3$ be an element
of some family of transformations on $\Reals^3$.
A simple direct sum transform applies the same 3-dimensional
transform to each vertex point,
so the full mesh transform is
$\Tr_{3n} = \bigoplus^n \Tr$,
and $\Tr_{3n} (\bigoplus_{i=0}^{n-1} \p_i) = \bigoplus_{i=0}^{n-1} \Tr \p_i$
by transforming the locations of each of the vertices
of the mesh.

In general, suppose $\Tr_i :
{\mathcal D}_i \mapsto {\mathcal C}_i; i = 0 \ldots k-1$
are $k$ maps.
The direct sum of the $\Tr_i$ is:
\begin{equation}
\label{eq:diagonal-blocks}
\Tr =
\left( \bigoplus_{i=0}^{k-1} \Tr_i \right) :
\left( \bigoplus_{i=0}^{k-1}{\mathcal D}_i \right)
\mapsto
\left( \bigoplus_{i=0}^{k-1}{\mathcal C}_i \right)
\end{equation}
$\Tr$ is a transform whose
$k$ 'diagonal blocks' are the $\Tr_i$.

Using this notation,
$\Df{\p}{f(\p)} = \bigoplus_i \Df{\p_i}{f(\p)}$,
$\Df{\Tr}{\Tr\p} = \bigoplus_i \Df{\Tr}{\Tr\p_i}$,
and:
\begin{equation}
\label{eq:total-registration-transform-derivative}
\De{\Tr}{f( \Tr \p )}{\Tr_0}
 =
\sum_i
\De{\q_i}{f(\q)}{\q=\Tr_0\p}
\circ
\De{\Tr}{\Tr\p_i}{\Tr_0}
\end{equation}
For the gradient, the equivalent is:
\begin{equation}
\label{eq:registration-gradient-sum}
\Gc{\Tr}{f( \Tr \p )}{\Tr_0}
 =
\sum_i
\left( \De{\Tr}{\Tr\p_i}{\Tr_0} \right)^{\dagger} \;
\left( \Gc{\q_i}{f(\q)}{\q=\Tr_0\p} \right)
\end{equation}

The derivative with respect to vertex or data points,
$\Df{\p_i}{f(\p)}$,
is assumed to be known.
For example, the derivatives of functions
related to data fitting
are discussed in chapter \ref{sec:data-fitting}.

In this chapter, I focus on computing
$\De{\Tr}{\Tr\p}{\Tr_0}$
for $\Tr : \Reals^3 \mapsto \Reals^3$
and $\p \in \Reals^3$.

\subsubsection{General linear registration}
\label{sec:General-linear-registration}

Suppose $\Tr = \L$ a linear transform on $\Reals^3$.
We can represent $\L$ as a vector in $\Reals^9$:
\begin{equation}
\label{eq:L-vector}
\l = \left(\L_{00},\L_{01},\L_{02},
       \L_{10},\L_{11},\L_{12},
       \L_{20},\L_{21},\L_{22}\right),
\end{equation}
where $\L_{ij}$ is the $ij$-th element of a
matrix representation of $\L$.
We can identify a vector $\p \in \Reals^3$
with a linear transform $\Tr_{\p} : \Reals^9 \mapsto \Reals^3$
by defining $\Tr_{\p}\l = \L\p$.
In the coordinate system defined by equation \ref{eq:L-vector},
the matrix for $\Tr_{\p}$ is:
\begin{equation}
\label{eq:Tp-matrix}
\Tr_{\p} =
\left(
\begin{array}{lllllllll}
\p_0 & \p_1 & \p_2 &  0   &  0   &  0   &  0   &  0   &  0 \\
 0   &  0   &  0   & \p_0 & \p_1 & \p_2 &  0   &  0   &  0 \\
 0   &  0   &  0   &  0   &  0   &  0   & \p_0 & \p_1 & \p_2 \\
\end{array}
\right)
\end{equation}

Because linear transforms are their own derivatives
(see section\ref{sec:Derivatives-of-linear-functions}),
it follows that the derivative with respect to the
unconstrained set of linear registration transforms is:
\begin{equation}
\label{eq:linear-transform-derivative}
\Df{\L}{\left( \L \, \p \right)}
 \; = \;
\Df{\L}{\left( \Tr_{\p} \, \L \right)}
 \; = \;
\Tr_{\p}
\end{equation}

\subsubsection{Scaled rotations}
\label{sec:Scaled-rotations}

A scaled rotation
is a linear transform $\Sc = s \R$,
where $s \in \Re$ and $\R : \Reals^n \mapsto \Reals^n$
is a rotation.

The quaternions $\Qs$ are a convenient representation
for the scaled rotations on $\Reals^3$.
(See Faugeras~\cite[sec.~5.5.2]{Faugeras1993}.)

A quaternion is a 4-tuple:
\begin{equation}
\q = (w, x, y, z) = (w, \v),
\end{equation}
where $w, x, y, z \in \Re$ and $\v \in \Reals^3$.
The set of quaternions has several operations:
\begin{itemize}
\item Quaternion conjugation $\dagger$:
\begin{equation}
\q^\dagger = (w, \v)^\dagger = (w, - \v)
\end{equation}
\item The quaternion product $\diamond$:
\begin{equation}
\q_0 \diamond \q_1 = (w_0w_1 - \v_0 \bullet \v_1, w_0 \v_1 + w_1 \v_0 + \v_0 \times \v_1)
\end{equation}
\item Quaternion norm $\| \|_{\Qs}$:
\begin{equation}
\| \q \|_{\Qs}^2
= \q^\dagger \bullet \q
= \q \bullet \q^\dagger
= w^2 + \|\v\|^2
= w^2 + x^2 + y^2 + z^2
\end{equation}
\end{itemize}

The quaternion product can be extended to $\Reals^3$
by identifying $\p \in \Reals^3$ with
the quaternion $(0,\p)$.
This allows us to define a linear transform
on $\Reals^3$ for any quaternion:
\begin{equation}
\Sc(\q) \p = \q \diamond \p \diamond \q^\dagger
\end{equation}

It turns out that transforms $\Sc(\q)$ so defined are scaled rotations.
The scale is the squared norm of the quaternion $\| \q \|_{\Qs}^2$.
The rotation is about the axis of $\v$
by $\cos^{-1}(w / \| \q \|_{\Qs})$.
(Note that $\q$ and $-\q$ correspond to the same scaled rotation.)

$\Sc(\q)$ can be written as a matrix:
\begin{equation}
\label{eq:quaternion-matrix}
\Sc(\q) =
\left(
\begin{array}{ccc}
w^2 + x^2 - y^2 - z^2 & 2(xy - wz)            & 2(xz + wy)           \\
2(xy + wz)            & w^2 - x^2 + y^2 - z^2 & 2(yz - wx)           \\
2(xz - wy)            & 2(yz + wx)            & w^2 - x^2 - y^2 +z^2
\end{array}
\right)
\end{equation}

Notice that the adjoint (transpose) of $\Sc(\q)$
is the linear transform corresponding to the conjugate quaternion:
$\Sc^{\dagger}(\q) =  \Sc(\q^{\dagger})$.

If we consider $\Sc(\q)$ to be a 9-dimensional vector,
as in equation \ref{eq:L-vector},
then the derivative can be expressed as the matrix:
\begin{equation}
\label{eq:quaternion-derivative-matrix}
\Df{q}{\Sc(\q)}
 = 2 \left(
\begin{array}{rrrr}
  w &  x & -y & -z \\
  z &  y &  x &  w \\
 -y &  z & -w &  x \\
 -z &  y &  x & -w \\
  w & -x &  y & -z \\
  x &  w &  z &  y \\
  y &  z &  w &  x \\
 -z & -w &  z &  y \\
  w & -x & -y &  z \\
\end{array}
\right)
\end{equation}

It's also useful to express the derivative $\Df{q}{\Sc(\q)}$
as a set of partial derivative matrices,
which are computed
by differentiating the elements of matrix in equation
\ref{eq:quaternion-matrix}:
\begin{eqnarray}
\label{eq:quaternion-matrix-partial-derivatives}
\Df{w}{\Sc(\q)}
& = &
2 \left(
\begin{array}{rrr}
 w & -z &  y \\
 z &  w & -x \\
-y &  x &  w
\end{array}
\right)
\\
\nonumber
\\
\Df{x}{\Sc(\q)}
& = &
2 \left(
\begin{array}{rrr}
 x &  y &  z \\
 y & -x & -w \\
 z &  w & -x
\end{array}
\right)
\nonumber
\\
\nonumber
\\
\Df{y}{\Sc(\q)}
& = &
2 \left(
\begin{array}{rrr}
-y &  x &  w \\
 x &  y &  z \\
-w &  z & -y
\end{array}
\right)
\nonumber
\\
\nonumber
\\
\Df{z}{\Sc(\q)}
& = &
2 \left(
\begin{array}{rrr}
-z & -w &  x \\
 w & -z &  y \\
 x &  y &  z
\end{array}
\right)
\nonumber
\end{eqnarray}

Note that
$\Df{w}{\left( \Sc(\q) \; \p \right)}
 = \left( \Df{w}{\Sc(\q)}\right) \; \p$
(and similarly for the partials with respect to $x,y,$ and $z$).

We can write the total derivative in terms of the partials as:
\begin{eqnarray}
\Df{\q}{\left( \Sc(\q) \; \p \right)}
& = &
\left( \Df{\w}{\Sc(\q)} \; \p \right) \otimes \e_w
\\
& + &
\left( \Df{\x}{\Sc(\q)} \; \p \right) \otimes \e_x
\nonumber
\\
& + &
\left( \Df{\y}{\Sc(\q)} \; \p \right) \otimes \e_y
\nonumber
\\
& + &
\left( \Df{\z}{\Sc(\q)} \; \p \right) \otimes \e_z
\nonumber
\end{eqnarray}

In computing the gradients of registration penalties,
we sum expressions like
$\Df{\q}{\left( \Sc(\q)\p_i \right)}^{\dagger} \;
\Gf{\p_i}{f}$
(see equation \ref{eq:registration-gradient-sum}).
In terms of the partial derivative matrices,
this is:
\begin{equation}
\Df{\q}{\left( \Sc(\q)\p_i \right)}^{\dagger} \;
\Gf{\p_i}{f}
=
\left(
\begin{array}{c}
\left( \Df{\w}{\Sc(\q)} \; \p \right) \bullet \Gf{\p_i}{f} \\
\left( \Df{\x}{\Sc(\q)} \; \p \right) \bullet \Gf{\p_i}{f} \\
\left( \Df{\y}{\Sc(\q)} \; \p \right) \bullet \Gf{\p_i}{f} \\
\left( \Df{\z}{\Sc(\q)} \; \p \right) \bullet \Gf{\p_i}{f} \\
\end{array}
\right)
\end{equation}

We sometimes encounter the inverse transform, $\Sc^{-1}(\q)$,
which is the same as the linear transform,
$\Sc(\q^{-1})$,
corresponding to the inverse quaternion
(in the sense of the quaternion product):
$\q^{-1} = (w, x, y, z)^{-1}
         = {1 \over {\| q \|_{\Qs}^2}} (w, -x, -y, -z)$.

To compute derivatives of expressions
involving an inverse quaternion,
we can use the derivative of $\q^{-1}$
with respect to $\q$:
\begin{equation}
\Df{\q}{\q^{-1}}
=
{1 \over {\| \q \|_{\Qs}^4}}
\left(
\begin{array}{cccc}
-w^2+x^2+y^2+z^2 & -2wx              & -2wy              & -2wz \\
 2wx             & -w^2+x^2-y^2-z^2  &  2xy              &  2xz \\
 2wy             &   2xy             & -w^2-x^2+y^2-z^2  &  2yz \\
 2wz             &   2xz             &  2yz              &  -w^2-x^2-y^2+z^2 \\
\end{array}
\right)
\end{equation}

\paragraph{Rotations}
\label{sec:Rotations}

A common representation for the rotations on $\Reals^3$
is the set of {\it unit quaternions},
that is, the quaternions satisfying $\| \q \|_{\Qs} = 1$.
However, optimization under a nonlinear constraint
like $\| \q \|_{\Qs} = 1$ is relatively difficult,
so I instead use a redundant representation by general quaternion:
\begin{equation}
\R(\q) = {{\Sc(\q)} \over {\| \q \|_{\Qs}^2}}
\end{equation}
To avoid numerical instability, it's usually enough
to add a small penalty for $\| \q \|_{\Qs}$ far from $1$.

The partial derivative matrices of $\R(\q)$ are used in the
same way as,
and can be expressed in terms of,
the partials of $\Sc(\q)$:
\begin{equation}
\Df{v}{\R(\q)}
\; = \;
\Df{v}{\left(
{{\Sc(\q)} \over {\| \q \|_{\Qs}^2}}
\right)}
\; = \;
{{\Df{v}{\Sc(\q)}} \over {\| \q \|_{\Qs}^2}}
-
{{2v \Sc(\q)} \over {\| \q \|_{\Qs}^4}}
\end{equation}
where $v$ is any of $w$, $x$, $y$, or $z$.

\subsubsection{Shift registration}
\label{sec:Shift-registration}

A simple shift, or translation, adds a constant vector
to its argument: $\Tr \p = \p + \t,$
for some $\t \in \Reals^3$.
The derivative with respect to
an unconstrained translation vector
is simply
$\Df{\t}{(\p + \t)} = \I_3,$
where $\I_3$ is the identity on $\Reals^3$.

\subsubsection{Affine registration}
\label{sec:affine-registration}

An {\it affine transformation,} $\A : \Reals^m \mapsto \Reals^n$,
is a linear transformation plus a translation:
$\A \p = \L \p + \t$,
where $\L : \Reals^m \mapsto \Reals^n$ is a linear transform,
the {\it linear part} of the affine tranform,
and $\t \in \Reals^n$ is $\A$'s {\it translation}.

Note that, if $\A = (\L,\t)$, then its inverse is
$\A^{-1} = (\L^{-1}, - \L^{-1}\t)$.

(It is sometimes useful to use a redundant representation:
$\A \p = \L (\p + \t_0) + \t_1$,
where $\t_0 \in \Reals^m$ and $\t_1 \in \Reals^n$,
but, for simplicity, I'll stick to the minimal one-translation
representation in this discussion.)

In the context of mesh registration,
where $\A : \Reals^3 \mapsto \Reals^3$,
we can view $\L \in \Reals^9$ and
$\A = (\L, \t) \in \left(\Reals^9 \oplus \Reals^3 \right)= \Reals^{12}$,
and we can express derivatives with respect to $\A$
in terms of the independent partial derivatives
with respect to $\L$ and $\t$.

It follows from the results in the preceding sections,
that the derivative with respect to the
unconstrained set of affine registration transforms is:
\begin{equation}
\label{eq:affine-transform-derivative}
\Df{(\L,\t)}{\left( \A \, \p \right)}
 \; = \;
\Df{(\L,\t)}{\left( \L \p + \t \right)}
 \; = \;
\Tr_{\p} \oplus \I_3,
\end{equation}
where $\oplus$ indicates,
as in equation \ref{eq:diagonal-blocks},
that the derivative is formed from the 2
'diagonal blocks'.

\subsubsection{Euclidean registration}
\label{sec:euclidean-registration}

A {\it euclidean transform} is an affine transform
$\Eu : \Reals^n \mapsto \Reals^n$,
$\Eu \p = \Sc \p + \t $,
whose linear part is a scaled rotation:
$\Sc = s \R$,
where $s \in \Re$ and $\R : \Reals^n \mapsto \Reals^n$
is a rotation.

Euclidean transforms are easy to invert.

If $\Eu = (\Sc,\t)$, then its inverse is
$\Eu^{-1} = (\Sc^{-1}, - \Sc^{-1}\t)$.
The inverse of a rotation, $\R$, is its adjoint
(tranpose) $\R^{-1} = \R^{\dagger}$.
The inverse of a scaled rotation $\Sc = s \R$
is therefore
$\Sc^{-1} = (s \R)^{-1}
         = {1 \over s} \R^{\dagger}
         = {1 \over {s^2}} \Sc^{\dagger}$.
We can then write the inverse of a euclidean transform as
$\Eu^{-1} = {1 \over {s^2}}(\Sc^{\dagger}, - \Sc^{\dagger}\t)$

Using quaternions, we can represent euclidean transforms with
7 dimensional points:
$\Eu = (w_{\q}, x_{\q}, y_{\q}, z_{\q}, x_{\t}, y_{\t}, z_{\t})$,
where $\q = (w_{\q}, x_{\q}, y_{\q}, z_{\q})$ is a quaternion corresponding
to $\S$, the linear part of $\Eu$,
and $\t = (x_{\t}, y_{\t}, z_{\t})$ is the translation part.

In the 7-dimensional representation, the inverse is:
\begin{equation}
\Eu^{-1} =
{1 \over {\| \q \|_{\Qs}^4}}
\left(
\begin{array}{rcrcr}
& &  w_{\q} & & \\
& & -x_{\q} & & \\
& & -y_{\q} & & \\
& & -z_{\q} & & \\
(-w_{\q}^2 - x_{\q}^2 + y_{\q}^2 + z_{\q}^2)
x_{\t}
&
-
&
2 (w_{\q}z_{\q} + x_{\q}y_{\q})
y_{\t}
&
+
&
2 (w_{\q}y_{\q} - x_{\q}z_{\q})
z_{\t}
\\
 2 (w_{\q}z_{\q} - x_{\q}y_{\q})
x_{\t}
&
+
&
(-w_{\q}^2 + x_{\q}^2 - y_{\q}^2 + z_{\q}^2)
y_{\t}
&
-
&
2 (w_{\q}x_{\q} + y_{\q}z_{\q})
z_{\t}
\\
- 2 (w_{\q}y_{\q} + x_{\q}z_{\q})
x_{\t}
&
+
&
2 (w_{\q}x_{\q} - y_{\q}z_{\q})
y_{\t}
&
+
&
(-w_{\q}^2 + x_{\q}^2 + y_{\q}^2 - z_{\q}^2)
z_{\t}
\end{array}
\right)
\end{equation}

In computing the gradients of registration penalties,
we sum expressions like
$\Df{\q, \t}{\left( \Eu(\q,\t)\p_i \right)}^{\dagger} \;
\Gf{\p_i}{f}$
(see equation \ref{eq:registration-gradient-sum}).
Using the results in sections
\ref{sec:Scaled-rotations}
and
\ref{sec:Shift-registration},
it's not hard to see that,
in terms of the partial derivative matrices,
this is:
\begin{equation}
\label{eq:euclidean-transform-gradient}
\left[
\Df{\q,\t}{\; \Eu(\q,\t)\p_i}
\right]^{\dagger} \;
\Gf{\p_i}{f}
=
\left(
\begin{array}{c}
\left( \Df{\w_{\q}}{\Sc(\q)} \; \p_i \right) \bullet \Gf{\p_i}{f} \\
\left( \Df{\x_{\q}}{\Sc(\q)} \; \p_i \right) \bullet \Gf{\p_i}{f} \\
\left( \Df{\y_{\q}}{\Sc(\q)} \; \p_i \right) \bullet \Gf{\p_i}{f} \\
\left( \Df{\z_{\q}}{\Sc(\q)} \; \p_i \right) \bullet \Gf{\p_i}{f} \\
\end{array}
\right)
\oplus
\Gf{\p_i}{f}
\end{equation}

\subsubsection{Rigid registration}
\label{sec:rigid-registration}

A {\it rigid transform} is an affine transform
$\G \p  = \R \p + \t $,
whose linear part is a rotation, $\R$.
Using the representation, $\R(\q)$,
using {\em non-unit} quaternions,
presented in \autoref{sec:Rotations},
we get the same results for the gradient terms
as in equation \ref{eq:euclidean-transform-gradient},
with the partials of $\Sc(\q)$ replaced by the
partials of $\R(\q)$, that is:
\begin{equation}
\label{eq:rigid-transform-gradient}
\left[
\Df{\q,\t}{\; \G(\q,\t)\p_i}
\right]^{\dagger} \;
\Gf{\p_i}{f}
=
\left(
\begin{array}{c}
\left( \Df{\w_{\q}}{\R(\q)} \; \p_i \right) \bullet \Gf{\p_i}{f} \\
\left( \Df{\x_{\q}}{\R(\q)} \; \p_i \right) \bullet \Gf{\p_i}{f} \\
\left( \Df{\y_{\q}}{\R(\q)} \; \p_i \right) \bullet \Gf{\p_i}{f} \\
\left( \Df{\z_{\q}}{\R(\q)} \; \p_i \right) \bullet \Gf{\p_i}{f} \\
\end{array}
\right)
\oplus
\Gf{\p_i}{f}
\end{equation}


\section{CONTINUED\_FRACTION Operator}
\index{approximation}\index{rational number}

The operator CONTINUED\_FRACTION approximates the real number
( \nameref{rational} number, \nameref{rounded} number)
into a continued fraction. CONTINUE_FRACTION has one or
two arguments, the number to be converted and an optional
precision:
\begin{verbatim}
    continued\_fraction(<num>)
 or
    continued\_fraction(<num>,<size>)
\end{verbatim}
The result is a list of two elements: the
first one is the rational value of the approximation, the second one
is the list of terms of the continued fraction which represents the
same value according to the definition \verb&t0 +1/(t1 + 1/(t2 + ...))&.
Precision: the second optional parameter \meta{size} is an upper bound
for the absolute value of the result denominator. If omitted, the
approximation is performed up to the current system precision.

{\tt Examples:}
\begin{verbatim}
continued_fraction pi;

                  ->

  1146408
{---------,{3,7,15,1,292,1,1,1,2,1}}
  364913

continued_fraction(pi,100);

                  ->

  22
{----,{3,7}}
  7

\end{verbatim}


\section{Introduction}

Fourier series (FS) gives a new idea to expand a continuous and periodic signal with a series of trigonometric functions, which glanced at frequency analysis at the first time. Based on the idea of FS, other transformations were proposed to extend the limit on FS: (1) Fourier transform (FT) for continuous but non-periodic signals; (2) discrete time Fourier transform (DTFT) for sampled signals; (3) discrete Fourier transform (DFT) for completely discrete analysis. These transformations make up Fourier transform family under Dirichlet conditions. Besides, Laplace transform extends the frequency into complex domain and Z transform simplifies DTFT practically \cite{SignalSystem}.

Analysis in frequency domain provides more options to process signals or evaluate linear systems. For example, a spectrum given by fast Fourier transform (FFT, the fast algorithm for DFT) shows the frequency distribution of the signal; the location of poles and zeros of a system demonstrate the stability and dynamic properties.

In this paper, attentions are paid to digital signal processing. Power spectrum density is firstly introduced as a tool for frequency analysis and a new algorithm is delivered. Then the design and properties of some common used filters are discussed. Finally, an example is given to show processing procedure and the results.

\section{Power Spectrum Density}

DFT is used to analysis a certain signal. However, there are many kinds of stochastic signals whose statistical properties are the most concerned. As for a stationary and random signal, power spectrum density (PSD) is taken to estimate its properties in frequency domain \cite{SignalDetect}.

PSD is defined as the Fourier transform of the autocorrelation function of the given signal $x(t)$
\begin{equation}
    S_x(\omega) = \mathscr{F}\left(R_{xx}(\tau)\right) = \int_{-\infty}^{+\infty} R_{xx}(\tau) \iexp{-\mathrm{j} 2 \uppi \omega \tau} \diff \tau
\end{equation}
where the autocorrelation function $R_{xx}(\tau)$ is
\begin{equation}
    R_{xx}(\tau) = \lim_{T\to \infty} \frac{1}{2T}\int_{-T}^{T} x(t) x(t-\tau)  \diff t
\end{equation}
Thus, the PSD is computed by
\begin{align}
    S_x(\omega) &= \int_{-\infty}^{+\infty} \left(\lim_{T\to \infty} \frac{1}{2T}\int_{-T}^{T} x(t) x(t-\tau)  \diff t t\right) \iexp{-\mathrm{j} 2 \uppi \omega} \diff \tau \notag \\
    &= \lim_{T\to \infty} \frac{1}{2T} \int_{-T}^{T} x(t)\iexp{-\mathrm{j} 2 \uppi \omega t} \left( \int_{-\infty}^{+\infty} x(t-\tau) \iexp{-\mathrm{j} 2 \uppi \omega \left(\tau-t\right)} \right) \diff t \notag \\
    &= \lim_{T\to \infty} \frac{1}{2T} \int_{-T}^{T} x(t)\iexp{-\mathrm{j} 2 \uppi \omega t} X^*(\jomega) \diff t \notag \\
    &= \lim_{T\to \infty} \frac{1}{2T} X(\jomega) X^*(\jomega) \notag \\
    &= \lim_{T\to \infty} \frac{1}{2T} \left| X(\jomega) \right|^2
        \label{eq:psd}
\end{align}
Eq~\eqref{eq:psd} is also known as Wiener–Khinchin theorem, which demonstrate the relationship between PSD and general frequency spectrum.

\subsection{Classical Power Spectrum Density Estimation}

Analog signals are analyzed based on the sampled data. Two important effects should be considered in sampling: (1) sampling should satisfy Nyquist sampling theorem, where the sampling frequency should be at least twice the maximum frequency of the origin signal. In practice, an analog anti-aliasing filter is usually used before the signal is sampled. (2) Finite sampling is equivalent to applying a window function to the infinite signal, which leads to frequency leakage effect. Because of these two effects, the PSD computed from the sampled data is just the estimation of the PSD of origin analog signal. That's why we usually say `spectrum estimation' instead of `spectrum calculation'.

The most direct method to estimate PSD is to discretize Eq~\eqref{eq:psd} using DFT and cancel the limit as below.
\begin{equation}
    S_x(\omega) \approx \frac{1}{T} \left| X(k) \right|^2 = \frac{1}{Nf_s} \left| X(k) \right|^2
\end{equation}
where $T$ is the total time of data, $N$ is the total number of points, $f_s$ is the sampling frequency. $X(k)$ is the DFT of $x(n)$, given by
\begin{equation}
    X(k) = \sum_{n=0}^{N-1} x(n) \iexp{-\mathrm{j} 2 \uppi \frac{n}{N}k} \label{eq:DFT}
\end{equation}
In the case where a window function $w(n)$ is applied to the sampled data, the PSD should be normalized as below. The $2$ in the numerator converts the two-sided PSD to one-sided PSD, whose square root is the most used in practice.
\begin{equation}
    S_x(\omega) = \frac{2}{\sum_{n=0}^{N-1} w^2(n) f_s} \left| X(k) \right|^2 \label{eq:periodogram}
\end{equation}

PSD can be estimated by combining Eq~\eqref{eq:DFT} and Eq~\eqref{eq:periodogram}. This direct method is called `periodogram'. Both MATLAB and Python provide built-in functions to calculate the periodogram. They are named \verb|periodogram| and \verb|scipy.signal.periodogram|.

An example for PSD estimation is shown in Fig~\ref{fig:plotPSD}. The result of periodogram looks very noisy, especially in high frequency band. To improve the accuracy, Welch proposed a method by dividing the data into overlapped segments and computing the average of the periodogram of each segment.

\begin{figure}[!htb]
    \centering
    \includegraphics[width=0.5\textwidth]{Welch.pdf}
    \caption{The idea of Welch's method to estimate PSD.}
    \label{fig:Welch}
\end{figure}

The idea of Welch's method is shown in Fig~\ref{fig:Welch} and the relevant functions are \verb|pwelch| in MATLAB and \verb|scipy.signal.welch| in Python.

\subsection{Logarithmic Frequency Axis Power Spectral Density}

As shown in Fig~\ref{fig:plotPSD}, by averaging PSD using Welch's method, the curve is less noisy. However, because the data is divided into segments, the length decreases, which decreases the frequency resolution. To further improve Welch's method, a new method named Logarithmic frequency axis Power Spectral Density (LPSD) was proposed \cite{LPSD}.

Considering that most signals occupy a wideband in frequency domain, the PSD is usually plotted with a logarithmic axis. Thus, the basic idea of the LPSD is to construct logarithmic frequency axis rather than linear axis. As a consequence, the frequency resolution changes with frequency, which allows dividing data into different segments. That is to say, the LPSD algorithm succeeds the idea of Welch's method and divides data according to the frequency point at which the PSD is estimated. The detailed procedure of the algorithm is discussed hereafter.

A logarithmic axis is made up of geometric progression, the length of which is assumed to be $J_{\rm des}$ and can be set manually, and it's not necessary equal to the length of data, $N$. The common ratio $q$ and ideal frequency resolution can be calculated as below.
\begin{gather}
    \lg f_{\rm max} - \lg f_{\rm min} = q\left(J_{\rm des} -1 \right) \quad \Rightarrow \quad q = \left(\frac{f_{\rm max}}{f_{\rm min}}\right)^{\frac{1}{J_{\rm des} -1}} = \left(\frac{N}{2}\right)^{\frac{1}{J_{\rm des} -1}} \\
    r_0(j) = f(j+1) - f(j) = \left(q-1\right) f(j)
    \label{eq:r0}
\end{gather}
Note that the maximum frequency is given by Nyquist frequency $f_{\rm max} = \frac{f_s}{2}$, and the minimum frequency is limited by the total sampling time $f_{\rm min} = \frac{1}{T} = \frac{f_s}{N}$. The index is denoted as $j$, which shows that the resolutions differ between frequencies.

However, the ideal frequency resolution $r_0(j)$ could be less than $f_{\rm min}$, which is not meaningful. So the resolution should be adjusted. To do so, we assume the desired number of segments to be $K_{\rm des}$, and the overlap ratio is $\xi$. So the desired resolution can be computed according to Fig~\ref{fig:Welch}.
\begin{gather}
    \left(K_{\rm des}-1\right) \left(1-\xi\right) L_{\rm des} + L_{\rm des} = N \\
    r_{\rm des} = \frac{f_s}{L_{\rm des}} = \frac{f_s}{N}\left(\left(K_{\rm des}-1\right) \left(1-\xi\right)+1\right)
\end{gather}
The ideal frequency resolution should be adjusted as below.
\begin{equation}
    r'(j) = \begin{cases}
        r_0(j) & r_0(j) \ge r_{\rm des} \\
        \sqrt{r_0(j)r_{\rm des}} & r_0(j) < r_{\rm des} \text{ and } \sqrt{r_0(j)r_{\rm des}} > f_{\rm min}  \\
        f_{\rm min} & \text{ else }
       \end{cases}  \label{eq:rp}
\end{equation}
Besides, the frequency resolution should consider an integer length of data and be further modified.
\begin{equation}
    L(j) = \left\lfloor \frac{f_s}{r'(j)} \right\rfloor \quad \Rightarrow \quad r(j) = \frac{f_s}{L_(j)} \label{eq:r}
\end{equation}
So far, the logarithmic frequencies can be computed by iterations given by Eq~\eqref{eq:r0}, Eq~\eqref{eq:rp} and Eq~\eqref{eq:r}.

Similar to the Welch's method, $D$ and $L$ in Fig~\ref{fig:Welch} differ with index $j$. For $j$-th frequency point,
\begin{equation}
    D(j) = \left\lfloor \left( 1-\xi \right) L(j) \right\rfloor
\end{equation}
and the number of segments is computed below.
\begin{equation}
    K(j) = \left\lfloor \frac{N-L(j)}{D(j)+1} \right\rfloor
\end{equation}

Average is usually deducted in PSD estimation. The average of $j$-th frequency point and $k$-th segment is given below.
\begin{equation}
    a(j,k) = \frac{1}{L(j)} \sum_{l=0}^{L(j)-1} x(D(j)(k-1)+l)
\end{equation}
Deducting the average and applying window function $w(j,l)$, the segment to be transformed is gained.
\begin{equation}
    G(j,k,l) = \big( x(D(j)(k-1)+l) - a(j,k) \big) w(j,l)
\end{equation}
The PSD of $k$-th segment at frequency $f(j)$ is calculated by single point DFT as below.
\begin{equation}
    A(j,k) = \sum_{l=0}^{L(j)-1} G(j,k,l) \iexp{-\mathrm{j} 2 \uppi \frac{m(j)}{L(j)} l}
\end{equation}
Note that the frequency index $m(j)=\frac{f(j)}{r(j)}$ is not necessary to be an integer.

Finally, the PSD at $j$-th frequency point is given by the average of $K(j)$ segments according to Wiener–Khinchin theorem.
\begin{equation}
    P(j) = \frac{C_{\rm PSD}(j)}{K(j)} \sum_{k=1}^{K(j)} \left| A(j,k) \right|^2
\end{equation}
The coefficient $C_{\rm PSD}(j)$ is used to normalize the PSD, which is given below.
\begin{equation}
    C_{\rm PSD}(j) = \frac{2}{f_s\sum_{l=0}^{L(j)-1} w^2(j,l)}
\end{equation}

The complete procedure of the LPSD algorithm has been discussed. The function is realized in both Python and MATLAB code, which can be found in the appendix. A comparison is given in Fig~\ref{fig:plotPSD}. The signal is obtained by filtering a white noise with elliptic filter so that the PSD should be equal to the gain of the transfer function theoretically.

\begin{figure}[!htb]
    \centering
    \includegraphics[width=0.6\textwidth]{plotPSD.pdf}
    \caption{Comparison of three methods to estimate PSD.}
    \label{fig:plotPSD}
\end{figure}

It's obvious that both the periodogram and the LPSD algorithms can calculate PSD at minimum frequency limited by the sampling time. But the result of periodogram is the most noisy. Welch's method improves the accuracy but sacrifices the minimum frequency. The LPSD provides the best estimated PSD, the result of which is very close to the theoretical value. However, the cost of the LPSD is to spend more time for calculation because it uses single point DFT and cannot benefit from FFT.

\section{Filter Design}

Properties of a given signal in frequency domain can be obtained from its PSD. To change these properties, a filter with certain frequency response should be applied to the signal.

\subsection{Common Used Filters}

Since filter design has been a very proven technique, the theoretical analysis won't go far but some useful functions in MATLAB and Python will be introduced. Dynamic of an analog filter can be described in a differential function or a transfer function with Laplace transform as below.
\begin{gather}
    y^{(n)} + a_1 y^{(n-1)} + \cdots + a_n y = b_0 u^{(m)} + b_1 u^{(m-1)} + \cdots + b_m u \quad (m < n) \\
    H(s) = \frac{Y(s)}{U(s)} = \frac{b_0 s^{m} + b_1 s^{m-1} + \cdots + b_n}{s^{n} + a_1 s^{n-1} + \cdots + a_n} \quad (m < n)
\end{gather}
If there are poles in the transfer function, the filter would have infinite impulse response in time domain, for which we call it IIR filter for short.

To design a filter with built-in functions, properties of the filter shown in Fig~\ref{fig:filterdesign} should be specified. Taking low-pass filters as example, the frequency of passband is denoted by $f_{\rm pass}$ and $R_{\rm pass}$ is the allowed ripple. Stopband frequency is $f_{\rm stop}$ and the attenuation is $R_{\rm stop}$.

\begin{figure}[!htb]
    \centering
    \includegraphics[width=0.5\textwidth]{FilterDesign.pdf}
    \caption{Filter properties specification.}
    \label{fig:filterdesign}
\end{figure}

Some common used filters are listed in Tab~\ref{tab:filterdesign} with MATLAB built-in functions for filter design \cite{matlabdoc}. In these functions, \verb|Wp| and \verb|Ws| are respectively the angular frequency of passband and stopband ($\omega_p = 2 \uppi f_{\rm pass}$ for example). Passband ripple \verb|Rp| and stopband attenuation \verb|Rs| are described in decibels. With the option \verb|'s'|, the function designs an analog filter. If a digital filter is desired, functions should be called without the option \verb|'s'| and \verb|Wp| and \verb|Ws| should be the digital frequency which is normalized by the sampling frequency.

For Python users, there are functions with the same names in \verb|scipy.signal| package \cite{scipydoc}. If the package is imported by \verb|import scipy.signal as signal|, a prefix \verb|signal.| should be attached to the functions to be called. Besides, the programming rules also has to be noticed. For example, the command \verb|[b,a]=butter(n,Wn,'s')| in MATLAB is equivalent to the command \verb|b,a=signal.butter(n,Wn,analog=True)| in Python.

\begin{table}[!htb]
    \centering
    \caption{MATLAB built-in functions for IIR filter design}
    \label{tab:filterdesign}
    \begin{tabular}{ll}
        \toprule
        Name & Built-in Functions  \\
        \midrule
        Butterworth & \verb|[n,Wn] = buttord(Wp,Ws,Rp,Rs,'s');|        \\
                    & \verb|[b,a] = butter(n,Wn,'s');|                 \\
        Chebyshev Type I & \verb|[n,Wn] = cheb1ord(Wp,Ws,Rp,Rs,'s');|  \\
                    & \verb|[b,a] = cheby1(n,Rp,Wn,'s');|              \\
        Chebyshev Type II & \verb|[n,Wn] = cheb2ord(Wp,Ws,Rp,Rs,'s');| \\
                    & \verb|[b,a] = cheby2(n,Rs,Wn,'s');|              \\
        Elliptic Filter & \verb|[n,Wn] = ellipord(Wp,Ws,Rp,Rs,'s');|   \\
                    & \verb|[b,a] = ellip(n,Rp,Rs,Wn,'s');|            \\
        \bottomrule
    \end{tabular}
\end{table}

Comparison of four common used filters are shown in Fig~\ref{fig:filters} with options \verb|Wp=1|,\verb|Ws=3|, \verb|Rp=1|, \verb|Rp=40|. For the same target, elliptic filter has the least order but there are ripples in both passband and stopband. Butterworth filter has flat frequency response with passband, but the order is the largest, which means it takes more resources for calculation. Chebyshev filters have a compromise with order and ripple, where type I has ripples in passband and is flat in stopband, while type II is on the contrary.

\begin{figure}[!htb]
    \centering
    \includegraphics[width=0.6\textwidth]{filter.pdf}
    \caption{Comparison of four common used IIR filters.}
    \label{fig:filters}
\end{figure}

As has been mentioned, functions mentioned above can also be used to design digital filters. Besides, some methods such as `impulse invariance' and `bilinear' can be used to discretize an analog filter. Take an analog Butterworth low-pass filter for example, the discretized filter response is shown in Fig~\ref{fig:filter-c2d}. Note that the discretization could lead to aliasing effects, which may change the behaviors of the filter.

\begin{figure}[!htb]
    \centering
    \includegraphics[width=0.6\textwidth]{filter-c2d.pdf}
    \caption{Digital filter design by discretization.}
    \label{fig:filter-c2d}
\end{figure}

Digital systems have a unique property which allows all-zero system to exist. These systems are described in difference equation or transfer function with Z transform as below.
\begin{gather}
    y(n) = b_0 u(n) + b_1 u(n-1) + \cdots + b_m u(n-m)
     \sum_{k=0}^{m} b_k u(n-k) \\
    H(z) = \frac{Y(z)}{U(z)} = b_0 + b_1 z^{-1} + \cdots + b_m z^{-m} = \sum_{k=0}^{m} b_k z^{-m}
\end{gather}
The impulse response in time domain of this all-zero system is finite, thus this kind of filter is call FIR filter for short. Without poles, FIR filter is always stable.

Two methods are generally used to design a FIR filter. The first one is base on window function, whose idea is to apply a certain window function to the response of an IIR filter. MATLAB provides \verb|fir1| function to realize this design. For Python, the function is \verb|scipy.signal.firwin|.

Another method for FIR filter design is based on the frequency sample, where only filter order, frequencies and target responses are necessary. The function is named \verb|fir2| in MATLAB and \verb|scipy.signal.firwin2| in Python.

Applications with graphical user interface (GUI) are also available for filter design. They are \verb|filterBuilder| or \verb|filterDesigner| (or \verb|fdatool| in old version) in MATLAB and \verb|pyfdax| in Python \cite{pyfda}.

FIR filter is easy to realize in digital device while IIR filter has less order. Filter type should be selected according to the requirements and equipment constraints before it is designed. Design results should be checked, including stability, frequency response. Besides, for real-time applications, the filter must be causal.

\subsection{Zero-Phase Filtering}

The dynamic of the filter always leads to delay. To decrease the delay, an all-pass filter can be employed to adjust the phase response of the filter. As for sampled data, a technology name zero-phase filtering can be used. For digital filter with transfer function $H(z)$, there are four steps to realize zero-phase filtering \cite{mExamples}.
\begin{enumerate}[\indent Step (1)]
    \setlength{\itemsep}{0pt}
    \item Filter: $X(z) \rightarrow H(z)X(z)$ ;
    \item Reverse: $H(z)X(z) \rightarrow H(z^{-1})X(z^{-1})$ ;
    \item Filter: $H(z^{-1})X(z^{-1}) \rightarrow H(z)H(z^{-1})X(z^{-1})$ ;
    \item Reverse: $H(z)H(z^{-1})X(z^{-1}) \rightarrow H(z^{-1})H(z)X(z)$.
\end{enumerate}
The transfer function of the complete process $G(z)$ can be calculated as below.
\begin{equation}
    G(z) = H(z^{-1})H(z) = \left| H(\omega) \right| \iexp{\mathrm{j} \varphi (\omega)} \left| H(\omega) \right| \iexp{-\mathrm{j} \varphi (\omega)}  = \left| H(\omega) \right|^2
\end{equation}
There is no imaginary part in the transfer function so that no delay is introduced. From a practical point of view, when the data is reversed and filtered again, the same delay is introduced but in an opposite direction. Thus, the delay from the first filtering is compensated by the second filtering.

The function \verb|filter| in MATLAB or \verb|scipy.signal.lfilter| in Python can be used to filter data with specified filter. To apply zero-phase filtering, the function to be called is \verb|filtfilit| in MATLAB or \verb|scipy.signal.filtfilt| in Python.

\begin{figure}[!htb]
    \centering
    \includegraphics[width=0.6\textwidth]{ZeroPhaseFiltering.pdf}
    \caption{Comparison of normal filtering and zero-phase filtering.}
    \label{fig:zerofiltering}
\end{figure}

A comparison of normal filtering and zero-phase filtering is shown in Fig~\ref{fig:zerofiltering}. The delay of normal filtering is obvious. Attentions should also be paid to the transient performances. There is transient dynamic in the beginning due to the error of initial states. However, there is also transient dynamic in the end with zero-phase filtering, which is caused by the filtering of the reversed data at the third step. Unfortunately, zero-phase filtering is not causal, which means it cannot be used in real-time applications.

\section{Results}

Due to nonlinear properties, the AC current or voltage from the \SI{50}{Hz} power supply is polluted by some harmonic frequencies. To ensure the filter suppress the noise adequately, a simulation has been done whose input signal is listed in Tab~\ref{tab:input}.

\begin{table}[!htb]
    \centering
    \caption{Input signal for filter validation}
    \label{tab:input}
    \begin{tabular}{llllll}
        \toprule
        Frequency (\si{Hz}) & $50$ & $100$ & $150$ & $250$ & White Noise \\
        Amplitude (\si{A}) & $1.0$ & $0.3$ & $0.2$ & $0.1$ & $0.05$ (RMS) \\
        Phase (\si{deg}) & $0$ & $60$ & $-60$ & $-90$ & --- \\
        \bottomrule
    \end{tabular}
\end{table}

The length of data is $N=1000$ with sampling frequency $f_s=\SI{1}{kHz}$. The target is to filter out components with frequency above \SI{150}{Hz}. A Chebyshev Type II digital filter is chosen because IIR filter has less order and Chebyshev Type II provides flat frequency within passband. By Specifying $f_{\rm pass} = \SI{110}{Hz}$, $f_{\rm stop} = \SI{130}{Hz}$, $R_{\rm pass} = \SI{1}{dB}$ and $R_{\rm stop} = \SI{60}{dB}$, the designed filter is shown in Fig~\ref{fig:prjFilter}.

\begin{figure}[!htb]
    \centering
    \includegraphics[width=0.6\textwidth]{prjFilter.pdf}
    \caption{Filter frequency response.}
    \label{fig:prjFilter}
\end{figure}

\begin{figure}[!htb]
    \centering
    \subcaptionbox{Signals in time domain.}{\includegraphics[height=4.9cm]{prjSignal.pdf}}
    \hfill
    \subcaptionbox{Signals in frequency domain.}{\includegraphics[height=4.9cm]{prjPSD.pdf}}
    \caption{Comparison of original and filtered signal.}
    \label{fig:prjSignals}
\end{figure}

Original signal is compared with filtered signal in Fig~\ref{fig:prjSignals} in both time and frequency domain. The zero-phase filtering technology was employed so that no delay is introduced. Because the high-frequency disturbances are removed, the filtered curve seems more smooth. For the PSD, it's obvious that the filter suppress high-frequency noise by about three orders of magnitude as stopband attenuation is set to \SI{60}{dB}.

\section{Conclusions}

In this paper, the LPSD algorithm is employed to estimate PSD, which takes advantage of minimum frequency range from the periodogram and succeeds the idea of averaging from Welch's method. Unable to embed FFT, the LPSD takes more time to calculate but provides more precise estimation with logarithmic frequency axis. Filters are designed with built-in functions and zero-phase filtering is used to filter data, which can compensate the delay caused by normal filtering. Zero-phase filtering is not causal, thus it cannot be used in real-time applications. A simulation was designed to validate the filter to suppress high-frequency disturbances from nonlinear effects of the \SI{50}{Hz} power supply. The results confirmed the properties of the filter and the PSD showed the disturbances in high-frequency band was suppressed by three orders of magnitude.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% CS624: Analysis of Algorithms
% Copyright 2015 Pejman Ghorbanzade <pejman@ghorbanzade.com>
% Creative Commons Attribution-ShareAlike 4.0 International License
% More info: https://github.com/ghorbanzade/beacon
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section*{Question 5}

\begin{enumerate}
\item Prove that a vertex in a rooted tree can have at most one parent.
\item Prove that every vertex other than the root has exactly one parent.
\end{enumerate}

\subsection*{Solution}

\begin{enumerate}
\item Proof is given by contradiction.
Assume node $v_i$ is a vertex in a rooted tree with more than one parents.
In this case, there will be two nodes $v_{r_1}$ and $v_{r_2}$ for which $v_i$ is a child.
Then, either one of $v_{r_1}$ and $v_{r_2}$ or none of them is the root of the tree.

In case none of them is the root, there is another node $u$ which is root and there are two paths $u \rightarrow \cdots \rightarrow v_{r_1} \rightarrow v_i$ and $u \rightarrow \cdots \rightarrow v_{r_2} \rightarrow v_i$.
This means there is a simple loop of the form $u \rightarrow v_{r_1} \rightarrow \cdots \rightarrow v_i \rightarrow v_{r_2} \rightarrow \cdots \rightarrow u$ which contradicts the definition of the tree.

On the other hand, if one of $v_{r_1}$ and $v_{r_2}$ is the root, there is a path from the root to $v_i$ and $v_i$ to its other parent which means $v_i$ is the parent of the other node which contradicts the previous assumption that $v_i$ has two roots.

Therefore the initial assumption is false and proof is complete.

\item The proof is again given by contradiction.
Assume that in the tree with root $u$, there is a node $v_i$ with more than two parents.
In this case, $v_i$ is child of at least two nodes $v_{r_1}$ and $v_{r_2}$.
As there are distinct paths $u \rightarrow \cdots \rightarrow v_{r_1}$ and $u \rightarrow \cdots \rightarrow v_{r_2}$, there will be a simple loop of the form $u \rightarrow \cdots \rightarrow v_{r_1} \rightarrow v_i \rightarrow v_{r_2} \rightarrow \cdots \rightarrow u$ which contradicts the definition of the tree.
Thus the initial assumption is false and proof is complete.
\end{enumerate}


\section{Open loop analysis}
\subsection{Longitudinal motion}
Reducing the state space system such that it only contains the states $\left \{ V_t, \alpha, \theta, q \right \}$ and input $\left \{ \delta_{el} \right \}$ results in the following matrices for the state space in Equation~\ref{eq:ssac}. $I_4$ is a 4x4 identity matrix and $0_{4,1}$ is a 4x1 zero matrix.

\begin{equation}
    \label{eq:sslon}
    \begin{aligned}
        A_{lon}&=\begin{bmatrix}
            -0.08894   &  -10.7   &   -32.17   & -3.977 \\
            -0.0005042 & -0.05167 & -3.989e-14 &   0.9792 \\
                    0 &        0 &          0 &        1 \\
            1.228e-18 &  -0.6256 &          0 &  -0.2485
        \end{bmatrix} &
        B_{lon}&=\begin{bmatrix}
            -0.07104 \\
            -0.0002308 \\
                    0 \\
            -0.01541
        \end{bmatrix} \\\\
        C_{lon}&=I_4 &
        D_{lon}&=0_{4,1}
    \end{aligned}
\end{equation}

\begin{align}
    \dot{x} &= A_{lon} \cdot x + B_{lon} \cdot u_{el} \nonumber\\
    y &= C_{lon} \cdot x + D_{lon} \cdot u_{el} \label{eq:ssac}
\end{align}

Analyzing the eigenvalues of the $A_{lon}$ matrix yields the following parameters for the inherent flight motions.

\begin{table}[h!]
    \centering
    \begin{tabular}{ r | c c c c c }
                     & Poles                            & $\zeta$        & $\omega_n$     & $P$           & $T_{1/2}$     \\ \hline \hline
        Short period & $\e{-1.53}{-1} + \e{7.62}{-1}i$ & $\e{1.97}{-1}$ & $\e{7.77}{-1}$ & $8.09$        & $4.43$        \\
                     & $\e{-1.53}{-1} - \e{7.62}{-1}i$ & $\e{1.97}{-1}$ & $\e{7.77}{-1}$ & $8.09$        & $4.43$        \\ \hline
        Phugoid      & $\e{-4.17}{-2} + \e{1.23}{-1}i$ & $\e{3.22}{-1}$ & $\e{1.30}{-1}$ & $\e{4.82}{1}$ & $\e{1.56}{1}$ \\
                     & $\e{-4.17}{-2} - \e{1.23}{-1}i$ & $\e{3.22}{-1}$ & $\e{1.30}{-1}$ & $\e{4.82}{1}$ & $\e{1.56}{1}$
    \end{tabular}
    \caption{Longitudinal eigenmotions poles, damping rations, natural frequencies, periods and time to half amplitude.}
\end{table}

The poles are the eigenvalues of $A_{lon}$ matrix, $\zeta$ the damping, $\omega_n$ the natural frequency, $P$ the oscillation period and $T_{1/2}$ the time to damp to half amplitude. These parameters are calculated from the poles $\lambda =\xi \pm \eta i$ using the following equations.

\begin{align}
    \omega_n&=\left | \lambda\right | \\
    \zeta&=-\frac{\xi}{\omega_n} \\
    P&=\frac{2\pi}{\omega_n} \\
    T_{1/2}&=P \cdot \frac{\ln{2}}{2 \pi} \cdot \frac{\sqrt{1-\zeta^2}}{\zeta}
\end{align}

Simulations are performed to get the time response for each motion in order to check whether the calculated parameters are correct. In order to do this check visually two elements are added to the figures. The first one are the \emph{period markers}. These are a set of vertical lines separated by the calculated period $P$ and aligned with the first peak.

The second element are one or two exponential decay functions in the form shown below.

\begin{equation}
    f(t) = C_1\left(\frac{1}{2}\right)^{\frac{t}{T_{1/2}}} + C_2
\end{equation}

Where $t$ is the time,and $C_1$ and $C_2$ are constants used to adjust the initial value. The result of this function reduces by half every $T_{1/2}$, thus it can be used to check whether the amplitude of the oscillations is reducing at the same rate. Figure~\ref{fig:ol_sp} shows the results of these simulations.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.6\textwidth]{figures/ol_sp}
    \caption{Pitch-rate $q$ response during a short period oscillation, induced by a impulse elevator input.}
    \label{fig:ol_sp}
\end{figure}

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.6\textwidth]{figures/ol_ph}
    \caption{Pitch angle $\theta$ response during a phugoid induced by an initial pitch angle of $\theta_0=0.1\ [rad]$}
    \label{fig:ol_ph}
\end{figure}

\subsection{Lateral motion}
Reducing the state space system such that it only contains the states $\left \{ \beta, \phi, p, r \right \}$ and inputs $\left \{ \delta_{a}, \delta_{r} \right \}$ results in the following matrices for the state space in Equation~\ref{eq:ssaclat}. $I_4$ is a 4x4 identity matrix and $0_{4,2}$ is a 4x2 zero matrix.

\begin{align*}
    A_{lat}&=\begin{bmatrix}
        -0.04905 & 0.08741 &  0.5803 & -0.8129 \\
               0 &       0 &       1 &  0.7105 \\
          -2.547 &       0 & -0.2594 &   0.151 \\
           -0.59 &       0 & 0.02607 & -0.1313
    \end{bmatrix} &
    B_{lat}&=\begin{bmatrix}
        4.181e-05 & 0.0001227 \\
                0 &         0 \\
          -0.0444 &  -0.00523 \\
         0.002184 & -0.004975
    \end{bmatrix} \\\\
    C_{lat}&=I_4 &
    D_{lat}&=0_{4,2}
\end{align*}

\begin{align}
    \dot{x} &= A_{lat} \cdot x + B_{lat} \cdot u_{el} \nonumber\\
    y &= C_{lat} \cdot x + D_{lat} \cdot u_{el} \label{eq:ssaclat}
\end{align}

The poles for this system are,
\begin{table}[h!]
    \centering
    \begin{tabular}{ c }
        Poles \\ \hline \hline
        $\e{-1.68}{-1}$ \\
        $\e{-1.68}{-1}$ \\
        $\e{-5.18}{-2}$ \\
        $\e{-5.18}{-2}$ \\
    \end{tabular}
    \caption{Lateral eigenmotion poles}
\end{table}

For the lateral motion it is expected to find two complex poles for the dutch roll and two real poles, one for the aperiodic roll and one for the spiral motion. However under the given flight conditions the linearization code fails and generates systems with four complex poles.

In order to continue with the assignment the flight conditions for the Accelerometer Position Analysis are used.

\begin{equation*}
    h_{apa} = 15000ft \qquad  V_{apa}=500ft/s
\end{equation*}

Linearising under these flight conditions results in the following,

\begin{align*}
    A_{lat}&=\begin{bmatrix}
        -0.2022 & 0.06414 & 0.07827 & -0.9919 \\
              0 &       0 &       1 &  0.0781 \\
         -22.92 &       0 &  -2.254 &  0.5408 \\
          6.005 &       0 & -0.0404 & -0.3146
    \end{bmatrix} &
    B_{lat}&=\begin{bmatrix}
        0.0001724 & 0.0005058 \\
                0 &         0 \\
          -0.4623 &   0.05686 \\
         -0.02437 &  -0.04687
    \end{bmatrix} \\\\
    C_{lat}&=I_4 &
    D_{lat}&=0_{4,2}
\end{align*}

\begin{table}[h!]
    \centering
    \begin{tabular}{ r | c c c c c c }
                       & Poles                   & $\zeta$        & $\omega_n$     & $P$    & $T_{1/2}$     & $\tau$         \\ \hline \hline
        Dutch roll     & $\e{-3.20}{-1} + 2.74i$ & $\e{1.16}{-1}$ & $\e{7.77}{-1}$ & $2.28$ & $4.43$        &                \\
                       & $\e{-3.20}{-1} - 2.74i$ & $\e{1.16}{-1}$ & $\e{7.77}{-1}$ & $2.28$ & $2.14$        &                \\ \hline
        Aperiodic roll & $2.12$                  &                &                &        & $\e{1.56}{1}$ & $\e{4.72}{-1}$ \\ \hline
        Spiral         & $\e{-1.13}{-2}$         &                &                &        & $\e{1.56}{1}$ & $\e{8.88}{1}$
    \end{tabular}
    \caption{Lateral eigenmotions poles, damping rations, natural frequencies, periods, time to half amplitude and time constants.}
\end{table}

For the aperiodic eigenmotions, there is a new parameter $\tau$, the time constant and the time to damp to half amplitude $T_{1/2}$ is calculated differently.

\begin{align}
    \tau &= -\frac{1}{\lambda} \\
    T_{1/2} &= \ln{2}\ \tau
\end{align}

The results of these simulations are shown in the following figures. Despite being an aperiodic motions, the aperiodic roll and spiral results both contain oscillations. This is due to a less dominant dutch roll. To show this, Figure~\ref{fig:ol_apdsf} has the period markers with the period for the dutch roll added to it.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.6\textwidth]{figures/ol_dr}
    \caption{Yaw rate $r$ response during a dutch roll induced by an impulse rudder input.}
    \label{fig:ol_dr}
\end{figure}

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.6\textwidth]{figures/ol_ap}
    \caption{Roll rate $p$ response during an aperiodic roll induced by an initial roll rate of $r_0=0.1\
    [rad\ s^{-1}]$}
    \label{fig:ol_apdsf}
\end{figure}

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.6\textwidth]{figures/ol_si}
    \caption{Bank angle $\phi$ response during a spiral motion induced by an initial bank angle of $\phi_0=0.1\ [rad]$}
    \label{fig:ol_si}
\end{figure}

\clearpage