\documentclass[]{article}

% Define page
\usepackage[top=1in, bottom=1in, left=1in, right=1in]{geometry}

% Include packages for text modification
\usepackage{amsmath,amsfonts,amssymb}
\usepackage{bm}

% Include packages for citing
\usepackage[noadjust]{cite}

% Include packages for graphics
\usepackage{graphicx,psfrag}
\usepackage{subfigure}

% Include packages for tables
\usepackage{array,multirow}
\usepackage{ctable}

% Define new commands
\newcommand{\etal}{\emph{et al.}}
\newcommand{\vT}{\bm{T}}
\newcommand{\vmu}{\bm{\mu}}
\newcommand{\vTmu}{\bm{T_{\mu}}}
\newcommand{\vTx}[1][]{\bm{T}_{#1}(\bm{x})}
\newcommand{\vTy}[1][]{\bm{T}_{#1}(\bm{y})}
\newcommand{\vx}[1][]{\bm{x}_{#1}}
\newcommand{\vxt}[1][]{\bm{\widetilde x}_{#1}}
\newcommand{\vy}[1][]{\bm{y}_{#1}}
\newcommand{\D}[2]{\frac{\partial #1}{\partial #2}}
\newcommand{\Dd}[3]{\frac{\partial^2 #1}{\partial #2 \partial #3}}
\newcommand{\elastix}{\texttt{elastix}}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{document}

\title{Spatial derivatives and penalty terms in ITK and \elastix}

\author{Marius Staring and Stefan Klein}
\date{}
\maketitle

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% main text

\section{Introduction}

Image registration is the process of aligning images, and can be
defined as an optimisation problem:
\begin{align}
\hat \vmu &= \arg \min_{\vmu} \mathcal{C}(I_F, I_M;
\vmu),\label{eq:reg}
\end{align}
with $I_F$ and $I_M$ the $d$-dimensional fixed and moving image,
respectively, and $\vmu$ the vector of parameters of size $N$ that
model the transformation $\vT$. The cost function $\mathcal{C}$
consists of a similarity measure $\mathcal{S}(I_F, I_M; \vmu)$ that
defines the quality of alignment. Examples are the mean square
difference, normalised correlation, and mutual information measure.
In order to regularise the transformation $\vT_{\vmu}$ often a
penalty term $\mathcal{P}(\vmu)$ is added to the cost function, so
it becomes:
\begin{align}
\mathcal{C} &= \alpha_1 \mathcal{S} + \alpha_2 \mathcal{P},
\end{align}
where $\alpha_1, \alpha_2$ user-defined constants that weigh
similarity against regularity.

Penalty term are often based on the first or second order spatial
derivatives of the transformation. For example the bending energy of
the transformation, which is arguably the most common penalty term,
is defined in 2D as:
\begin{align}
\mathcal{P}_{\mathrm{BE}}(\vmu) &= \frac{1}{P} \sum_{\vxt[i]}
\left\| \frac{\partial^2 \vT}{\partial \vx \partial \vx^T}(\vxt[i])
\right\|_F^2 \\
&= \frac{1}{P} \sum_{\vxt[i]} \sum_{j = 1}^2 \left( \frac{\partial^2
T_j}{\partial x_1^2}(\vxt[i]) \right)^2  + 2 \left( \frac{\partial^2
T_j}{\partial x_1 \partial x_2}(\vxt[i]) \right)^2 + \left(
\frac{\partial^2 T_j}{\partial x_2^2}(\vxt[i]) \right)^2,
\end{align}
where $P$ is the number of points $\vxt[i]$, and the tilde denotes
the difference between a variable and a given point over which a
term is evaluated.

The optimisation problem (\ref{eq:reg}) is frequently solved using
an iterative gradient descent routine:
\begin{align}
\vmu_{k+1} &= \vmu_k - a_k \frac{\partial \mathcal{C}}{\partial
\vmu} = \vmu_k - a_k \left( \alpha_1 \frac{\partial
\mathcal{S}}{\partial \vmu} + \alpha_2 \frac{\partial
\mathcal{P}}{\partial \vmu} \right),\label{eq:opt}
\end{align}
with $a_k$ a user-defined declining function that defines the step
size.

The derivative of the similarity measure usually involves
computation of the spatial derivative of the moving image:
$\D{I_M}{\vx}$, and the derivative of the transformation to its
parameters: $\D{\vT}{\vmu}$. In the ITK the last derivative is
implemented using $\texttt{transform->GetJacobian()}$, i.e. the
derivative to the transformation parameters $\vmu$ is referred to as
`Jacobian'.

Penalty terms usually consist of the first and second order
\emph{spatial} derivatives of the transformation, i.e.
$\D{\vT}{\vx}$ and $\Dd{\vT}{\vx}{\vx^T}$. We will refer to these
derivatives as the `SpatialJacobian' and the `SpatialHessian' to
clearly distinguish between these derivatives and the `Jacobian'. In
order to apply the gradient descent optimisation routine
(\ref{eq:opt}), we additionally need the derivatives $\D{}{\vmu}
\D{\vT}{\vx}$ and $\D{}{\vmu} \Dd{\vT}{\vx}{\vx^T}$. These we call
the `JacobianOfSpatialJacobian' and `JacobianOfSpatialHessian',
respectively. See Table \ref{tab:notation} for details.

\begin{table}[h]
\centering
\begin{tabular}{llll}
\toprule \toprule
Name & definition & matrix size & written out in 2D \\
\midrule Transformation & $\vT = \vT_{\vmu}(\widetilde \vx)$ & $d
\times 1$ & $\begin{bmatrix} T_1(\vx) \\ T_2(\vx) \end{bmatrix}$ \\[3ex]
Jacobian & $\frac{\partial \vT_{\vmu}}{\partial \vmu}(\widetilde
\vx)$ & $d \times N$ & $\begin{bmatrix}
  \frac{\partial T_1}{\partial \mu_1}(\widetilde \vx) & \cdots & \frac{\partial T_1}{\partial \mu_N}(\widetilde \vx) \\
  \frac{\partial T_2}{\partial \mu_1}(\widetilde \vx) & \cdots & \frac{\partial T_2}{\partial \mu_N}(\widetilde \vx) \\
\end{bmatrix}$ \\[3ex]
SpatialJacobian & $\frac{\partial \vT_{\vmu}}{\partial
\vx}(\widetilde \vx)$ & $d \times d$ & $\begin{bmatrix}
  \frac{\partial T_1}{\partial x_1}(\widetilde \vx) \frac{\partial T_1}{\partial x_2}(\widetilde \vx) \\
  \frac{\partial T_2}{\partial x_1}(\widetilde \vx) \frac{\partial T_2}{\partial x_2}(\widetilde \vx) \\
\end{bmatrix}$ \\[3ex]
JacobianOfSpatialJacobian & $\frac{\partial}{\partial \vmu}
\frac{\partial \vT_{\vmu}}{\partial \vx}(\widetilde \vx)$ & $d
\times d \times N$ & $\begin{bmatrix}
  \frac{\partial}{\partial \mu_1} \frac{\partial \vT_{\vmu}}{\partial
\vx}(\widetilde \vx) & \cdots & \frac{\partial}{\partial \mu_N}
\frac{\partial \vT_{\vmu}}{\partial \vx}(\widetilde \vx)
\end{bmatrix}$ \\[3ex]
SpatialHessian & $\frac{\partial^2 \vT_{\vmu}}{\partial \vx
\partial \vx^T}$ & $d \times d \times d$ & $\left\{ \begin{bmatrix}
  \frac{\partial^2 T_1}{\partial x_1 \partial x_1}(\widetilde \vx) \frac{\partial^2 T_1}{\partial x_2 \partial x_1}(\widetilde \vx) \\
  \frac{\partial^2 T_1}{\partial x_2 \partial x_1}(\widetilde \vx) \frac{\partial^2 T_1}{\partial x_2 \partial x_2}(\widetilde \vx) \\
\end{bmatrix}, \begin{bmatrix}
  \frac{\partial^2 T_2}{\partial x_1 \partial x_1}(\widetilde \vx) \frac{\partial^2 T_2}{\partial x_2 \partial x_1}(\widetilde \vx) \\
  \frac{\partial^2 T_2}{\partial x_2 \partial x_1}(\widetilde \vx) \frac{\partial^2 T_2}{\partial x_2 \partial x_2}(\widetilde \vx) \\
\end{bmatrix} \right\}$ \\[3ex]
JacobianOfSpatialHessian & $\frac{\partial}{\partial \vmu}
\frac{\partial^2 \vT_{\vmu}}{\partial \vx \partial \vx^T}$ & $d
\times d \times d \times N$ & $\begin{bmatrix}
  \frac{\partial}{\partial \mu_1} \frac{\partial^2 \vT_{\vmu}}{\partial \vx
\partial \vx^T} & \cdots & \frac{\partial}{\partial \mu_N} \frac{\partial^2 \vT_{\vmu}}{\partial \vx
\partial \vx^T}
\end{bmatrix}$ \\
\bottomrule \bottomrule
\end{tabular}
\caption{Naming conventions and definitions for the transformation
and its derivatives used in this paper.}\label{tab:notation}
\end{table}

\section{Support for penalty terms in the ITK}

The derivative of the penalty term is not supported in the ITK. We
propose to add the following functions in the
\texttt{itk::Transform} class:
\begin{quote}
\begin{verbatim}
virtual void GetSpatialJacobian(
  const InputPointType &,
  SpatialJacobianType & ) const;

virtual void GetSpatialHessian(
  const InputPointType &,
  SpatialHessianType & ) const;

virtual void GetJacobianOfSpatialJacobian(
  const InputPointType &,
  JacobianOfSpatialJacobianType &,
  NonZeroJacobianIndicesType & ) const;

virtual void GetJacobianOfSpatialHessian(
  const InputPointType &,
  JacobianOfSpatialHessianType &,
  NonZeroJacobianIndicesType & ) const;
\end{verbatim}
\end{quote}
and additionally a function to implement a sparse version of the
Jacobian:
\begin{quote}
\begin{verbatim}
virtual void GetJacobian(
    const InputPointType &,
    JacobianType &,
    NonZeroJacobianIndicesType & ) const;
\end{verbatim}
\end{quote}

The ITK structures that were used to store the data are given in
Table \ref{tab:datastructures}. The Jacobian is of size $d \times
N$, and since the number of transformation parameters is flexible
for some transformations, the data structure used for storing the
Jacobian is an \texttt{itk::Array2D} object, which inherits from the
\texttt{vnl\_matrix}. This was already chosen previously in the ITK.
The SpatialJacobian is of fixed size $d \times d$, and therefore we
choose to use the \texttt{itk::Matrix} to store the SpatialJacobian,
which inherits from the \texttt{vnl\_matrix\_fixed}. For derivatives
to $\vmu$ we choose to use the \texttt{std::vector}. The
SpatialHessian gives us some problems, since we really need a 3D
matrix, but currently no such thing exists in the ITK. Therefore, we
opt for an \texttt{itk::FixedArray} of \texttt{itk::Matrix}.

\begin{table}[h]
\centering
\begin{tabular}{ll}
\toprule \toprule
Name & ITK structure \\
\midrule Transformation &  \\
Jacobian & \texttt{Array2D = vnl\_matrix} \\
SpatialJacobian & \texttt{Matrix = vnl\_matrix\_fixed} \\
JacobianOfSpatialJacobian & \texttt{std::vector< Matrix >} \\
SpatialHessian & \texttt{FixedArray< Matrix >}\footnotemark \\
JacobianOfSpatialHessian & \texttt{std::vector< FixedArray< Matrix > >} \\
NonZeroJacobianIndices & \texttt{std::vector< unsigned long >} \\
\bottomrule \bottomrule
\end{tabular}
\caption{The ITK structures that store the
data.}\label{tab:datastructures}
\end{table}
\footnotetext{by lack of a good 3D matrix structure}

From the function definitions above, notice that we chose to pass
the variables by reference, and as function arguments. The
\texttt{GetJacobian} in the \texttt{itk::Transform} is defined as:
\begin{quote}
\begin{verbatim}
virtual const JacobianType & GetJacobian( const InputPointType & ) const;
\end{verbatim}
\end{quote}
which returns a reference to the member variable
\texttt{m\_Jacobian}. It is possible, however, that this member is
only valid for the input point, namely for transformations with a
derivative dependent of the spatial position. Therefore, we think
that it should not be possible to access this parameter at a later
time, when the input point has possibly changed.

Also notice the \texttt{NonZeroJacobianIndicesType} in the function
definitions. These are meant for the support of sparse Jacobians,
JacobianOfSpatialJacobians, etc.

\section{Affine transformation}

For the affine transformation, the derivatives evaluate to the
following in 2D:
\begin{align}
\vTmu(\vxt) &= A \vx + \bm{t} = \begin{bmatrix} a_{11} & a_{12}
\\ a_{21} & a_{22} \end{bmatrix} \begin{bmatrix} \widetilde x_1 \\
\widetilde x_2 \end{bmatrix} + \begin{bmatrix} t_1 \\ t_2
\end{bmatrix} = \begin{bmatrix} \mu_0 & \mu_1
\\ \mu_2 & \mu_3 \end{bmatrix} \begin{bmatrix} \widetilde x_1 \\
\widetilde x_2 \end{bmatrix} + \begin{bmatrix} \mu_4 \\ \mu_5
\end{bmatrix}, \\
\D{\vT}{\vmu}(\vxt) &= \begin{bmatrix} \widetilde x_1 & \widetilde
x_2 & 0 & 0 & 1 & 0 \\ 0 & 0 & \widetilde x_1 & \widetilde x_2 & 0 &
1 \end{bmatrix}, \\
\D{\vT}{\vx}(\vxt) &= \begin{bmatrix} \mu_0 & \mu_1 \\ \mu_2 & \mu_3 \end{bmatrix}, \\
\Dd{\vT}{\vx}{\vx^T}(\vxt) &= O_{d \times d \times d}, \\
\D{}{\vmu} \D{\vT}{\vx}(\vxt) &= \left\{ \begin{bmatrix} 1 & 0 \\
0 & 0 \end{bmatrix}, \begin{bmatrix} 0 & 1 \\ 0 & 0
\end{bmatrix}, \begin{bmatrix} 0 & 0 \\ 1 & 0
\end{bmatrix}, \begin{bmatrix} 0 & 0 \\ 0 & 1
\end{bmatrix} \right\}, \\
\D{}{\vmu} \Dd{\vT}{\vx}{\vx^T}(\vxt) &= O_{d \times d \times d
\times N},
\end{align}
where $O_s$ is a zero matrix of size $s$.

%centre of rotation!!!

The \texttt{GetJacobianOfSpatialJacobian()} returns
\texttt{nonZeroJacobianIndices} = $[ 0, 1, 2, 3 ]$.

%For the SpatialHessian and the JacobianOfSpatialHessian we choose to
%not return matrices filled with zeros, but instead zero sized
%matrices. The implementation of the penalty term is assumed to check
%for the size. In case of an affine transform the size is zero, and
%the penalty term can simply return zero. This is a performance
%benefit compared to walking over the zero matrix, and adding and
%multiplying everything, which in the end also gives zero.

These derivatives are implemented in the
\texttt{itk::AdvancedMatrixOffsetTransformBase} class.

\section{B-spline transformation}

A transformation parameterised by third order B-splines can be
written in 2D as follows:
\begin{align}
\vTmu(\vxt) &= \begin{bmatrix} T_1(\vxt;\vmu) \\ T_2(\vxt;\vmu)
\end{bmatrix} = \begin{bmatrix} \widetilde{x}_1 \\ \widetilde{x}_2 \end{bmatrix}
 + \begin{bmatrix} \sum_{i} \mu_i \beta^3 \left(
\frac{ \widetilde{x}_1 - x_{1}^i}{\sigma_1} \right)
\beta^3 \left( \frac{\widetilde{x}_2 - x_{2}^i}{\sigma_2}\right) \\
 \sum_{i} \mu_{i+16} \beta^3 \left( \frac{ \widetilde{x}_1 - x_{1}^i}
 {\sigma_1} \right) \beta^3 \left( \frac{\widetilde{x}_2 - x_{2}^i}{\sigma_2} \right)
\end{bmatrix},
\end{align}
with $\vx^i$ the control points within the support of the B-spline
basis functions $\beta^3(\cdot)$, and $\sigma$ the B-spline grid
spacing. Here, we labelled the $x$-direction of the control points
with $\mu_0, \ldots, \mu_{15}$ and the $y$-direction with $\mu_{16},
\ldots, \mu_{31}$. (Since the support of a cubic B-spline is $4^d =
16$.)

For short notation, define:
\begin{align}
b_{33}^{i} &= \beta^3\left( (\widetilde{x}_1 - x_{1}^i) / \sigma_1
\right) \cdot \beta^3 \left( (\widetilde{x}_2 - x_{2}^i) / \sigma_2
\right), \\
b_{23}^{i} &= \left[ \beta^2 \left( (\widetilde{x}_1 -
x_{1}^i)/\sigma_1 + \tfrac{1}{2} \right) - \beta^2 \left(
(\widetilde{x}_1 - x_{1}^i)/\sigma_1 - \tfrac{1}{2} \right) \right]
\cdot \beta^3 \left( (\widetilde{x}_2 - x_{2}^i)/\sigma_2 \right) /
\sigma_1, \\
b_{32}^{i} &= \beta^3 \left( (\widetilde{x}_1 - x_{1}^i)/\sigma_1
\right) \cdot \left[ \beta^2 \left( (\widetilde{x}_2 -
x_{2}^i)/\sigma_2 + \tfrac{1}{2} \right) - \beta^2 \left( (
\widetilde{x}_2 - x_{2}^i)/\sigma_2 - \tfrac{1}{2} \right) \right] /
\sigma_2, \\
\begin{split}
b_{22}^{i} &= \left[ \beta^2 \left( ( \widetilde{x}_1 -
x_{1}^i)/\sigma_1 + \tfrac{1}{2} \right) - \beta^2 \left( (
\widetilde{x}_1 - x_{1}^i)/\sigma_1 - \tfrac{1}{2} \right) \right] \\
& \qquad \qquad \cdot \left[ \beta^2 \left( ( \widetilde{x}_2 -
x_{2}^i)/\sigma_2 + \tfrac{1}{2} \right) - \beta^2 \left( (
\widetilde{x}_2 - x_{2}^i)/\sigma_2 - \tfrac{1}{2} \right) \right] /
(\sigma_1 \sigma_2),
\end{split} \\
b_{13}^{i} &= \left[ \beta^1 \left( (\widetilde{x}_1 -
x_{1}^i)/\sigma_1 + 1 \right) - 2 \beta^1 \left( ( \widetilde{x}_1 -
x_{1}^i)/\sigma_1 \right) + \beta^1 \left( (\widetilde{x}_1 -
x_{1}^i)/\sigma_1 - 1 \right) \right] \cdot \beta^3 \left(
(\widetilde{x}_2 - x_{2}^i)/\sigma_2 \right) / \sigma_1^2, \\
b_{31}^{i} &= \beta^3 \left( (\widetilde{x}_1 - x_{1}^i)/\sigma_1
\right) \cdot \left[ \beta^1 \left( (\widetilde{x}_2 -
x_{2}^i)/\sigma_2 + 1 \right) - 2 \beta^1 \left( ( \widetilde{x}_2 -
x_{2}^i)/\sigma_2 \right) + \beta^1 \left( (\widetilde{x}_2 -
x_{2}^i)/\sigma_2 - 1 \right) \right] / \sigma_2^2.
\end{align}
From these equations we derive:
\begin{align}
\D{\vT}{\vmu}(\vxt) &= \begin{bmatrix} b_{33}^{0} & \cdots &
b_{33}^{15} & 0 & \cdots & 0 \\ 0 & \cdots & 0 & b_{33}^{0} &
\cdots & b_{33}^{15} \end{bmatrix}, \\
\D{\vT}{\vx}(\vxt) &= \begin{bmatrix} 1 + \sum_i \mu_i b_{23}^i &
\sum_i \mu_i b_{32}^i \\ \sum_i \mu_{i+16} b_{23}^{i} & 1 + \sum_i
\mu_{i+16} b_{32}^{i} \end{bmatrix}, \\
\Dd{\vT}{\vx}{\vx^T}(\vxt) &= \left\{ \begin{bmatrix} \sum_i \mu_i
b_{13}^i & \sum_i \mu_i b_{22}^i \\ \sum_i \mu_{i} b_{22}^{i} &
\sum_i \mu_{i} b_{31}^{i} \end{bmatrix}, \begin{bmatrix} \sum_i
\mu_{i+16} b_{13}^{i} & \sum_i \mu_{i+16} b_{22}^{i} \\ \sum_i
\mu_{i+16} b_{22}^{i} & \sum_i \mu_{i+16} b_{31}^{i}
\end{bmatrix} \right\} \\
\D{}{\vmu} \D{\vT}{\vx}(\vxt) &= \left\{ \begin{bmatrix} b_{23}^{0}
& b_{32}^{0} \\ 0 & 0 \end{bmatrix}, \cdots, \begin{bmatrix}
b_{23}^{15} & b_{32}^{15} \\ 0 & 0 \end{bmatrix}, \begin{bmatrix} 0
& 0 \\ b_{23}^{0} & b_{32}^{0} \end{bmatrix}, \cdots,
\begin{bmatrix} 0 & 0 \\ b_{23}^{15} & b_{32}^{15} \end{bmatrix} \right\}
\\
\D{}{\vmu} \Dd{\vT}{\vx}{\vx^T}(\vxt) &= \left\{ \left\{
\begin{bmatrix} b_{13}^0 & b_{22}^0 \\ b_{22}^0 & b_{31}^0 \end{bmatrix},
O_{d \times d} \right\}, \cdots, \left\{ \begin{bmatrix} b_{13}^{15}
& b_{22}^{15} \\ b_{22}^{15} & b_{31}^{15} \end{bmatrix}, O_{d
\times d} \right\}, \right. \\
& \qquad \left. \left\{ O_{d \times d},
\begin{bmatrix} b_{13}^{0} & b_{22}^{0} \\ b_{22}^{0} &
b_{31}^{0} \end{bmatrix} \right\}, \cdots, \left\{ O_{d \times d},
\begin{bmatrix} b_{13}^{15} & b_{22}^{15} \\ b_{22}^{15} &
b_{31}^{15} \end{bmatrix} \right\} \right\}.
\end{align}

These derivatives are implemented in the
\texttt{itk::AdvancedBSplineDeformableTransform} class.

\section{Combining transformations}

\texttt{elastix} supports combining multiple transformations by
addition or composition. Adding transformations is done via:
\begin{align}
\vTx &= \vTx[0] + \vTx[1] - \vx,
\end{align}
where $\vTx[0]$ is the initial transformation and $\vTx[1]$ the
current transformation. Only the current transformation is optimised
during the registration (as a choice). Composition of
transformations is defined via:
\begin{align}
\vTx &= \vT_1(\vTx[0]).
\end{align}

For these combined transformations we need to derive the relations
for the derivatives. Define $\bm{y} = \vT_0(\vxt)$.

\begin{align}
\intertext{Jacobian:}
\D{\vTmu}{\vmu}(\vxt) &= \D{}{\vmu} \left( \vT_0(\vxt) + \vT_1(\vxt)
- \vxt \right) = \D{}{\vmu} \vT_1(\vxt), \\
\D{\vTmu}{\vmu}(\vxt) &= \D{}{\vmu} \left( \vT_1(\vT_0(\vxt))
\right) = \D{}{\vmu} \vT_1(\bm{y}).
\intertext{SpatialJacobian:}
\D{\vTmu}{\vx}(\vxt) &= \D{}{\vx} \left( \vT_0(\vxt) + \vT_1(\vxt) -
\vxt \right) = \D{}{\vx} \vT_0(\vxt) + \D{}{\vx} \vT_1(\vxt) - \bm{I}, \\
%\D{\vTmu}{\vx}(\vxt) &= \D{}{\vx} \vT_1(\vT_0(\vxt)) = \D{}{\vx}
%\vT_1(\bm{y}) \cdot \D{}{\vx} \vT_0(\vxt). \\
\D{T_{\vmu,k}}{x_i}(\vxt) &= \left( \D{T_{1,k}}{\vx}(\vy) \right)^T
\D{\vT_0}{x_i}(\vxt) =  \left(\D{\vT_0}{x_i}(\vxt) \right)^T
\D{T_{1,k}}{\vx}(\vy)
\intertext{JacobianOfSpatialJacobian:}
\D{}{\vmu} \D{\vTmu}{\vx}(\vxt) &= \D{}{\vmu} \D{}{\vx} \left(
\vT_0(\vxt) + \vT_1(\vxt) - \vxt \right) = \D{}{\vmu} \D{}{\vx}
\vT_1(\vxt), \\
\D{}{\vmu} \D{\vTmu}{\vx}(\vxt) &= \D{}{\vmu} \D{}{\vx} \left(
\vT_1(\vT_0(\vxt)) \right) = \D{}{\vmu} \D{}{\vx} \vT_1(\bm{y}) \cdot
\D{}{\vx} \vT_0(\vxt).
\intertext{SpatialHessian:}
\Dd{\vTmu}{x_i}{x_j} &= \Dd{}{x_i}{x_j} \left( \vT_0(\vxt) +
\vT_1(\vxt) \right) = \Dd{}{x_i}{x_j} \vT_0(\vxt) +
\Dd{}{x_i}{x_j} \vT_1(\vxt), \\
%\Dd{\vTmu}{x_i}{x_j} &= \Dd{}{x_i}{x_j} \left( \vT_1(\vT_0(\vxt))
%\right) = \D{}{x_i} \vT_0(\vxt) \Dd{}{x_i}{x_j} \vT_1(\bm{y})
%\D{}{x_j} \vT_0(\vxt) + \Dd{}{x_i}{x_j} \vT_0(\vxt) \D{}{x_j}
%\vT_1(\bm{y}). \\
\Dd{T_{\vmu,k}}{x_i}{x_j}(\vxt) &= \left( \D{T_{1,k}}{\vx}(\vy)
\right)^T \Dd{\vT_0}{x_i}{x_j}(\vxt) + \left(\D{\vT_0}{x_i}(\vxt)
\right)^T \Dd{T_{1,k}}{\vx}{\vx^T}(\bm{y}) \D{\vT_0}{x_j}(\vxt)
\intertext{JacobianOfSpatialHessian:}
\D{}{\vmu} \Dd{\vTmu}{x_i}{x_j} &= \D{}{\vmu} \Dd{}{x_i}{x_j} \left(
\vT_0(\vxt) + \vT_1(\vxt) - \vxt \right) = \D{}{\vmu}
\Dd{}{x_i}{x_j} \vT_1(\vxt), \\
%\D{}{\vmu} \Dd{\vTmu}{\vx}{\vx^T} &= \D{}{\vmu} \Dd{}{x_i}{x_j}
%\left( \vT_1(\vT_0(\vxt)) \right) \\
% &= \D{}{x_i} \vT_0(\vxt) \D{}{\vmu} \Dd{}{x_i}{x_j} \vT_1(\bm{y})
%\D{}{x_j} \vT_0(\vxt) + \Dd{}{x_i}{x_j} \vT_0(\vxt) \D{}{\vmu}
%\D{}{x_j} \vT_1(\bm{y}). \\
\D{}{\vmu} \Dd{T_{mu,k}}{x_i}{x_j}(\vxt) &=
\Dd{T_{1,k}}{\vmu}{\vx^T}(\vy) \Dd{\vT_0}{x_i}{x_j}(\vxt) +
\left(\D{\vT_0}{x_i}(\vxt) \right)^T \left(\D{}{\vmu}
\Dd{T_{1,k}}{\vx}{\vx^T}(\bm{y})\right) \D{\vT_0}{x_j}(\vxt)
\end{align}

These derivatives are implemented in the
\texttt{itk::AdvancedCombinationTransform} class.

\section{Bending energy penalty term}

The bending energy was defined as:
\begin{align}
\mathcal{P}_{\mathrm{BE}}(\vmu) &= \frac{1}{P} \sum_{\vxt[i]}
\sum_{k,l,m = 1}^2 \left( \Dd{T_k}{x_l}{x_m}(\vxt[i]) \right)^2.
\end{align}

We constructed an \texttt{itk::TransformBendingEnergyPenaltyTerm}
which inherits from the \texttt{itk::ImageToImage\-Metric}. The
\texttt{GetValue()}-method is implemented like:
\begin{quote}
\begin{verbatim}
SpatialHessianType spatialHessian;
for all samples
  this->GetTransform()->GetSpatialHessian( sample, spatialHessian );
  for all k, l, m
    measure += spatialHessian[ k ][ l ][ m ]^2;
  end
end
measure /= this->m_NumberOfPixelsCounted;
\end{verbatim}
\end{quote}

For the \texttt{GetValueAndDerivative()} we have:
\begin{align}
\D{}{\vmu} \mathcal{P}_{\mathrm{BE}}(\vmu) &=  \frac{1}{P}
\sum_{\vxt[i]} \sum_{k,l,m = 1}^2 2 \Dd{T_k}{x_l}{x_m} (\vxt[i])
\D{}{\vmu} \Dd{T_k}{x_l}{x_m}(\vxt[i]),
\end{align}
which is implemented like:
\begin{quote}
\begin{verbatim}
SpatialHessianType spatialHessian;
JacobianOfSpatialHessianType jacobianOfSpatialHessian;
for all samples
  this->GetTransform()->GetSpatialHessian( sample, spatialHessian );
  this->GetTransform()->GetJacobianOfSpatialHessian( sample,
    jacobianOfSpatialHessian, nonZeroJacobianIndices );
  for all nonZeroJacobianIndices, k, l, m
    derivative[ nonZeroJacobianIndices[ mu ] ] += 2.0
      * spatialHessian[ k ][ i ][ j ]
      * jacobianOfSpatialHessian[ mu ][ k ][ i ][ j ];
  end
end
derivative /= this->m_NumberOfPixelsCounted;
\end{verbatim}
\end{quote}

This penalty term is implemented in the
\texttt{itk::TransformBendingEnergyPenaltyTerm} class.

%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Discussion and Conclusion}

This document describes the use and implementation of spatial
derivatives of coordinate transformations. These spatial derivatives
were exploited by the bending energy penalty term. Their usage is,
however, not limited to that penalty term, and many more penalty
terms can be implemented using the new functionality.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%\bibliographystyle{plainnat}
%\bibliography{mi}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\end{document}


\chapter{Multi-Rotor Development}
\label{chap:MR}
\section{Range \& Endurance}
For a multirotor, range can be estimated by how far it can be controlled by the controller before losing its connection. The range of a multirotor is dependent on certain factors like weight, motor power, weather conditions, etc. Based on the application requirement, they can be customized for longer ranges and better performances. In this project, the forward flight momentum theory was used to estimate the range and endurance of the aircraft for a given set of forward speeds. A velocity range of 0-20 m/s was chosen to perform this calculation and the estimated values of range, endurance, and forward speeds of the multi-rotor drone were calculated using the following steps as follows: \\\

\subsection{Forward Flight Momentum Theory}

The forward flight expression can be written as:

\begin{equation}
	\label{eq:range_multirotor}
	v = \frac{T}{2}
\end{equation}

At a given forward speed, V we can then solve for $ α_D $, v, T, $ P_{ind} $, $ P_{tot} $ as follows: \\\

\begin{enumerate}
	\item Calculate the quadrotor drag $ D = \frac{\rho C_D V^2}{2} $ at $ α_D = 0 $ Assume $ α_D $ does not affect the drag, otherwise need to iterate to find solution
	\item Solve $ α_D = tan^-1(\frac{D}{W}) $
	\item Square both sides of \ref{eq:range_multirotor} replace $ T_2 $ by $ W_2 $ + $ D_2 $ and re-arrange to get, $ v_4 + (2 V sin(α_D)) v_3 + V 2 v_2 −(W2 + D2)/(2 ρ A)2 = 0 $
	\item Positive real root of equation gives v, then $ P_{ind} = T_v; P_{tot} = T (v + V sin(α_D)) $
	\item Notice that this total power does not include profile drag, swirl, or additional
	losses due to non-uniform induced velocity.
	\item Solve for a range of speeds and plot results versus V

\end{enumerate}

The maximum range of the quadcopter is calculated using the formula: \\

Max. Range = $ (\frac{E_b * m_e * esc_e }{Minimum Total Power/Velocity}) $ \\

Where, $ E_b $  = Energy of the battery;  $ M_e $ = Motor efficiency;  $ Esc_e $ = ESC efficiency \\

The maximum endurance of the quadcopter is calculated using the formula: \\

Max. Endurance = $ (E_b * m_e * esc_e) / (Minimum Total Power) $ \\
Where, $ E_b $ = Energy of the battery; $ M_e $ = Motor efficiency; $ Esc_e $ = ESC efficiency \\

Based on the above-mentioned procedures, the results of the multi-rotor drone (quadcopter) are as follows: \\
\begin{itemize}
	\item Max. Range = 34 kms
	\item Max. Endurance = 68.1 minutes
	\item Forward Speed for range = 9.5 m/s
	\item Forward Speed for endurance = 7 m/s
\end{itemize}

\section{Quadrotor Dynamics Model}

The development of dynamics model of quadrotor was done by using the following state space matrices given for each control channel.

\textbf{Given State-Space Matrices} : \\
\begin{figure}
	\centering
	\includegraphics[width=0.8\textwidth]{Images/ss_1}
	\includegraphics[width=0.8\textwidth]{Images/ss_2}
	\caption{State-Space Matrices}
	\label{fig:ss}
\end{figure}

The output from position controller was fed into the dynamics model. The dynamics model consists of each control channel mentioned above put in a separate state-space block in Simulink. For roll control the input was the desired roll angle and the output from the state-space block was the actual roll angle. This actual roll angle was then fed into the roll to v control channel as input, to get the actual velocity v about y-axis.

For pitch control the input was desired pitch angle and the output was the actual pitch angle. This actual pitch angle was then fed into the pitch to u control channel which gives the actual velocity about x-axis.

Yaw control channel used desired yaw rate, also called as desired heading rate, for input while the output of this control channel is actual yaw angle. The last control channel used for the dynamic model of this quadrotor is altitude control, for this the input was desired vertical velocity w while the output was orientation along z- axis, which is also called altitude.

The Simulink design of Dynamic Model is shown in the figure below:
\begin{figure}
	\centering
	\includegraphics[width=0.8\textwidth]{Images/position_control}
	\caption{Simulink Design Dynamic Model}
	\label{fig:pc}
\end{figure}

\section{Development of Position/Orientation control system:}

Position/Orientation Control System contains two types of state input, one is the target x, y, and z coordinate input and the other is actual x, y and z coordinates of the quadrotor. A reference yaw command is also given into the position controller to ensure that the quadrotor maintains the commanded yaw while operating for the target position.

The target x coordinate is given into a summation block while the actual x orientation of the quadrotor is given to the summation block as feedback. This command is fed into the PD controller, which gives corresponding derivative $\dot{x}$ (i.e., speed along x axis). This derived velocity u is given to the summation block whereas the actual velocity $ u_a $ is given as feedback in the same block. A PD controller is used again which finally gives the derivative $\ddot{x}$ .

Similarly, target y position and actual y position of the quadrotor is also given into the summation block and then into the PD controller. The output from this PD controller is further given into a summation block where the actual velocity v is given as feedback. This output is then given into the PD controller to get acceleration along y-axis.

Actual heading angle psi, along with $\ddot{x}$ and $\ddot{y}$ is used to calculate the desired pitch angle and desired roll angle. The following equations [reference no?] are used for this: \\

\left[\begin{array}{c}
	\phi_{d} \\
	\theta_{d}
\end{array}\right]=\left[\begin{array}{cc}
	-\sin \psi & -\cos \psi \\
	\cos \psi & -\sin \psi
\end{array}\right]^{-1} \frac{m}{U_{1}}\left[\begin{array}{l}
	\ddot{x}_{d} \\
	\ddot{y}_{d}
\end{array}\right] \\

Since the derivation of these equations in (reference no) uses small angle approximation, we must ensure that the desired angles $\theta_{d}$ and $\phi_{d}$ are within the limit of –20° and 20°. Therefore, a saturation block is placed at the output of MATLAB function block.

We also need the desired vertical velocity w and the desired heading rate $\dot{psi$ to be given into the dynamics model of the quadrotor. For this, desired z coordinate and actual z coordinate are given into a summation block and PD controller is used to obtain the desired w velocity. Similarly, actual yaw angle and desired yaw angle along with PD controller are used to obtain heading rate $\dot{psi}$.

\section{Development of Position Estimation:}

For estimating the position of the quadrotor, the output states from the dynamic model i.e., psi, psi, theta, h, u and v, are used. The dynamic model provides information about the actual value of states of the quadrotor. To acquire data regarding the actual x and y coordinated of the quadrotor we integrate the u and v velocities, respectively. The Simulink model of position estimation subsystem is shown in \ref{fig:pe}.

\begin{figure}
	\centering
	\includegraphics[width=0.8\textwidth]{Images/pe}
	\caption{Position Estimation}
	\label{fig:pe}
\end{figure}

 \section{Development of linear simulation model:}

 For the development of the linear simulation model, a reference subsystem was created which was used for giving the desired coordinate information to the position controller. The desired pitch and roll angles $ \theta_{d} $ and $\phi_{d}$ along with the heading rate $\dot{psi}$ and desired vertical velocity wd goes into a summation block followed by a PID controller. To this summation block feedback is given of actual orientation of the respective states.

 Further, these values are given into the dynamics model subsystem which is also explained in the previous section. The actual values obtained from the dynamic model are used in the position estimation subsystem to get information about Euler angles, position, and velocities of the quadrotor.

 The reference position subsystem developed in for this project is an If-Else condition block which is used with Action block to activate a particular command for an If-Else condition. The reference command subsystem is shown in the \ref{fig:rp}.
 \begin{figure}
 	\centering
 	\includegraphics[width=0.8\textwidth]{Images/rp}
 	\caption{Reference Position}
 	\label{fig:rp}
 \end{figure}

The complete linear simulation system that was developed in Simulink is shown in the \ref{fig:cm}

PID Tuning methods used for this project were Inbuilt PID tuners provided by Simulink. For tuning PD controllers, the following steps were taken:
\begin{enumerate}
	\item Proportional gain Kp was increased until steady oscillations were obtained
	\item Derivate gain Kd was increased until the oscillations were critically damped
\end{enumerate} \\

\begin{figure}
	\centering
	\includegraphics[width=1.1\textwidth]{Images/multirotor}
	\caption{Complete Multirotor Dynamic System}
	\label{fig:cm}
\end{figure}

Total simulation time was taken to be 1500 seconds for the following four maneuvers: \\
\begin{enumerate}
	\item Take off and hover at 2 meters above origin
	\item Fly to the first target (x = 5 m, y = 6 m, h = 4 m) and hover
	\item Fly to the second target (x = -5 m, y = -6 m, h = 4 m) and hover
	\item Return to 2 meters above origin and land
\end{enumerate}

\section{Result}

The quadcopter was successfully able to complete all the given maneuvers for this project. The simulation result, which shows the trajectory of x, y and z coordinates of the quadcopter for 1500 seconds is shown in .

\begin{figure}
	\centering
	\includegraphics[width=1.1\textwidth]{Images/multirotor_sol}
	\caption{Quadrotor Simulation Solution}
	\label{fig:m_sol}
\end{figure}\\

In the figure the quadcopter is initially at x=0, y=0 and z=0 position, in the first 300 seconds it takes off and hover at x=0 y=0 and z=2 m. Next, the quadcopter reaches to x=5, y=6 and z=4.For the third maneuver it goes from its current position to the target position, x=-5, y=-6 and z=4 and then for the last maneuver is goes from current position to target position of x=0, y=0 and z= 2 and then back to the origin.


\chapter{Multi-scale Analysis}
\section{Introduction}

In this chapter, the tools for multi-scale and multi-resoltuion
processing (analysis, synthesis and fusion) will be presented. Most of
the algorithms are based on pyramidal approaches. These approaches
were first used for image compression and they are based on the fact
that, once an image has been low-pass filtered it does not have
details beyond the cut-off frequency of the low-pass
filter any more. Therefore, the image can be subsampled -- decimated -- without
any loss of information.

A pyramidal decomposition is thus performed applying the following 3
steps in an iterative way:
\begin{enumerate}
  \item Low pas filter the image $I_{n}$ in order to produce $F(I_n)$;
  \item Compute the difference $D_n = I_n - F(I_n)$ which corresponds
  to the details at level $n$;
  \item Subsample $F_(I_n)$ in order to obtain $I_{n+1}$.
\end{enumerate}

The result is a series of decrasing resolution images $I_k$ and a
series of decreasing resolution details $D_k$.

\section{Morphological Pyramid}\label{secMorphoPyr}

If the smoothing filter used in the pyramidal analysis is a
morphological filter, one cannot safely subsample the filtered image
without loss of information. However, by keeping the details possibly
lost in the down-sampling operation, such a decomposition can be used.

The Morphological Pyramid is an approach to such a
decomposition. Its computation process is an iterative analysis
involving smoothing by the morphological filter, computing the
details lost in the smoothing, down-sampling the current image, and
computing the details lost in the down-sampling.

\input{MorphologicalPyramidAnalysisFilterExample.tex}

\input{MorphologicalPyramidSynthesisFilterExample.tex}

\subsection{Morphological Pyramid Exploitation}
One of the possible uses of the morphological pyramid is the
segmentation of objects -- regions -- of a particular scale.

\input{MorphologicalPyramidSegmenterExample.tex}

This same approach can be applied to all the levels of the
morphological pyramid analysis.

\input{MorphologicalPyramidSegmentationExample.tex}


%\pagestyle{plain}

\chapter{Event Detection}
In this chapter, we will introduce the mechanism of event detection in this project.
We first apply some noise reduction or audio enhancement technique on the training data for our events.
Then we extract MFCC features from the training clips.
After getting features, a Gaussian Mixture Model is built on those features.

\section{Audio Enhancement}
Because the audios downloaded from SSEs are uploaded by other users.
Those clips are recorded by various types of devices and in different environments.
Hence we need to apply a noise reduction step to enhance the audio clips.

Among algorithms for speech enhancement, spectral subtraction was one of the first algorithms proposed.
Although it was originally proposed for speech enhancement, we use it in our project for noise is pretty much the same in speech or in more general audios.
The basic principles is that we assume the clean signal add some additive noise becomes the audio we get.
Then if we subtract the noise spectrum from the audio, we may get signal close to the original clean one.

Mathematically speaking, assume the input signal is $ y(t) = x(t) + n(t)$, where the signal $x(t)$ is the signal of interest and $n(t)$ is the additive noise.
The audio enhancement process takes on the following steps:
\begin{enumerate}
\item The input signal $y(t)$ is resampled to 16kHz and multiple channels are averaged to one channel.
\item Using short time Fourier transform (STFT), we obtain $Y_i(k) = X_i(k) + N_i(k)$, where $i$ is the frame indices and $k$ is the frequency bin indices of the spectrum.
\item The Minimum Statistics (MS) estimator \cite{martin2001noise} is used to the noise power spectral density from $y(t)$.
\item Spectral subtraction is used to remove the estimated noice spectrum from the input spectrum $Y_i(k)$.
Then we could get an estimated clean signal spectrum: $\hat{X_i}(k)$.
\item Applying inverse short time Fourier transform (ISTFT) on $\hat{X_i}(k)$ to get the estimated clean signal $\hat{x}(t)$.
\end{enumerate}

A speech processing toolbox for Matlab called VoiceBox is used for spectral subtraction\footnote{\url{http://www.ee.ic.ac.uk/hp/staff/dmb/voicebox/voicebox.html}}.
% noise subtraction example here

\section{Feature Extraction}
Since the audio data is downloaded from a sound website where clips are crowdsourced, the clips are in various format.
We converted all audio clips to WAV format and averaged the channels into one if it has multiple channels.
Moreover, clips are all downsampled to 16khz sample rate for feature extraction.

For the audio data, mel-frequency cepstrum coefficients (MFCCs) features are a widely-used feature.
It was brought up by Davis and Mermelstein in the 1980s.
MFCC features extract the spectral envelope from spectrums of audio frames.
Basically, the process of extracting MFCCs start by applying Fast Fourier transform (FFT) on framed audio.
FFT transform audio data from Amplitude-Time domain (frame) into Amplitude-Frequency domain (spectrum).
As each audio frame is transformed into a spectrum, the spectral envelope is extracted by Inverse FFT (IFFT).

Actually, one more filtering is performed before applying IFFT on spectrums.
Because MFCC features are based on human perceptual experiment, and human's ear is like a natural filter, where low-frequency area has more filter and high-frequency area has less. So a nonlinear function is applied to original spectrum to transform the frequency axis into the mel scale.
A popular formula to convert $f$ herts into $m$ mel is:
\begin{equation}
	m = 2595 \times \log_{10}(1+\frac{1}{700})
\end{equation}

% melscale.eps on 5566
\begin{figure}[htb]
\centering
\input{figure/eventdetect/melscale}
\caption{Mel scale versus hertz scale}
\label{fig:melscale}
\end{figure}

Figure \ref{fig:melscale} shows the non-linear transformation of hertz scale to mel scale.
We could see that the slope of the curve are decreasing as the pitch goes up in hertz scale.
Because of this feature of suppressing the higher frequency bands, MFCC enables a focus of the more useful range of bands in lower scale \cite{davis1980comparison}.

% Paraphrase !!!

\section{Model Selection}
From the previous feature extraction process, we could a matrix representation for an audio clip.
The column number stands for the dimension of the features and each row corresponds to one observation.
From these data, we need to build a model that can capture the overall feature distribution and also are convenient to use for testing data.

For this goal, Gaussian Mixture Model (GMM) is used.
First, a multivariate gaussian distribution has the following probability density:

% multivariate gaussian
\begin{equation}
 \mathcal{N}(\mathbf{x}| \mathbf{\mu}, \Sigma) =
\frac{1}{\sqrt{(2\pi)^D|\Sigma|}}e^{-\frac{1}{2}(\mathbf{x}-\mathbf{\mu})^T \Sigma^{-1} (\mathbf{x}-\mathbf{\mu})}
\end{equation}

In this equation, $\mathbf{\mu}$ and $\mathbf{\Sigma}$ are all vectors in $D$ dimensions, for discribing high dimensional data.
A gaussian mixture model (or density) is a weighted sum of N gaussian densities.
Typically, these gaussian densities have the same dimension, say $D$, and each gaussian are called a component.
Put many gaussian distributions together, we could get the density function for GMM:

% GMM density function
\begin{equation}
P(\mathbf{x}|\mathbf{\pi},\mathbf{\mu},\Sigma) = \sum_{k = 1}^{M} \pi_k
\mathcal{N}(\mathbf{x}|\mathbf{\mu}_k, \Sigma_k),
\end{equation}

The intuition of using GMM to model the audio events is that the individual component densities of a multi-model density may model some underlysing set of acoustic classes \cite{reynolds1995robust}.
Like in speech, a words may consist of some vowels and consonants.
An audio event is naturally more complicated than speech, and may therefore contain more characteristics.
Therefore, we use different gaussian densities with different $\mathbf{\mu}$ and $\mathbf\sigma$, where the $\mathbf\mu$ may capture the overall value for one class and the $\mathbf\sigma$ shows the variation of that class.
When these densities are added together, forming a GMM, they can have a good depiction of some unusual distributions.

%Before training GMMs for event training data, we first apply K-means algorithm on the data.
%K-means is also an unsupervised clustering method.
%It uses a Expectation-Maximization step to update the cluster centroid.
%Because K-means has a faster speed for iterating than GMMs, so we first run K-means and use its centroid result as initials for further iterating of GMMs.
%This part use the gaussian toolbox provided by Matlab\footnote{\url{http://cn.mathworks.com/help/stats/gmdistribution-class.html}} .

\section{Training Data Selection}
As previously mentioned, many of the clips in those Sound Search Engines are crowdsourced.
Hence, the quality of those clips are not guaranteed to be suitable as our event training data.
We need to select out those data which are similar, and discard those outliers.
Because outliers are very likely to be a noise, thus adding no knowledge for our event models, perhaps even bring bad effects.

Therefore, in our project, we added a step for comparing the similarity of the audio clips, and cluster similar clips together, while leaving others clips out.
Yet signal data of the audio clips are in different volume, and different duration, it is hard to directly compare their similarity.
So we resort to the distance between their corresponding GMMs.
Because the trained GMM are in the same dimension for different audio clips, and GMM represent the overall features, it is reasonable to cluster similar GMMs together.

In order to measure the distance between GMMs, we use Kullback-Leibler (KL) divergence.
KL divergence is a measure of the difference between two probability distributions.
More precisely, KL divergence of $Q$ from $P$, denoted by $KL(P||Q)$, is a measure of the information loss when we use $Q$ to approximate $P$.

For continuous random variables $\mathbf{x}$ and two distributions $P$ and $Q$, the KL divergence of $Q$ from $P$ is defined as:
\begin{equation}
KL(P||Q) = \int_{-\infty}^{+\infty}\ln(\frac{p(\mathbf{x})}{q(\mathbf{x})})p(\mathbf{x})\mathrm{d}x,
\label{eq:kl}
\end{equation}
where $p(\mathbf{x})$ and $q(\mathbf{x})$ are the density functions of $P$ and $Q$.
The divergence satisifies three properties:
\begin{enumerate}
\item{Self similarity: $KL(P||P) = 0$}.
\item{Self identification: $KL(P||Q) = 0$, only if $P = Q$}.
\item{Non-Negativity: $KL(P||Q) >= 0$ for any $P, Q$}.
\end{enumerate}

Because of these properties, KL divergence is used in many aspects of speech recognition \cite{olsen2003efficient}.
But we need to take notice that KL divergence is not a distance because it is not symmetric: $KL(P||Q) \neq KL(Q||P)$.
For two gaussian distributions, the KL divergence has a closed formed expression.
But for two gaussian mixture models, there is no such closed form expression.
To tackle this issue, we use an approximation method to the KL divergence proposed in \cite{hershey2007approximating}
It uses ``variational approximation'' for calculating KL divergence.
We use the \textit{gaussmixk} function in VoiceBox toolbox for implementing the approximated KL divergence\footnote{\url{http://www.ee.ic.ac.uk/hp/staff/dmb/voicebox/doc/voicebox/gaussmixk.html}}.

\section{Summary}
The method we propose for audible event detection are introduced in this chapter.
We first apply a noise reduction technique on the downloaded audio clips.
This audio enhancement process uses Minimum Statistics estimator to estimate the noise power spectral density.
Then this estimated noise are subtracted from the input signal to get a more cleaned signal for event models trainging.
Feature extraction are also reviewed in this chapter.
The features we use in this task are MFCC features.
They are notable for good capture of audio features because of the non-linear mel scale could suppress higher frequency bands and put more focus on the lower part.
From the extracted features, we build Gaussian Mixture Models (GMMs) on them.
The reason that GMMs are chosen against other models is that they comprised of multiple gaussian distributions, which are called components.
These components could be used to model the underlying sound generating process and help us model more complicated sound events.
In the end, we touch on the problem of removing some low-quality sound clips.
This removal process is carried out by clustering audio clips.
We use an approximation for Kullback-Leibler (KL) divergence to represent the distance between different GMMs, and thus, can also be viewed as the distance between audio clips.
Clip features which have similar GMMs are clustered together, and we stop the clustering process when the divergence is large enough.
Through this way, we could remove some noisy clips for their feature data is too far away from others.


\subsection{Testing for stationarity with Dickey-Fuller (DF) and Augmented Dicky-Fuller (ADF)}

\subsubsection{Stationarity}

\subsubsection{Unit roots}

\subsubsection{Integration order}

\subsubsection{Dickey-Fuller}

The Dickey-Fuller test tests if there is a unit root.

The AR(\(1\)) model is:

\(y_t=\alpha + \beta y_{t-1}+\epsilon_t\)

We can rewrite this as:

\(\Delta y_t=\alpha + (\beta -1)y_{t-1}+\epsilon_t\)

We test if \(\beta -1)=0\).

If the coefficient on the last term is \(1\) we have a random walk, and the process is non-stationary.

If the last term is \(<1\) then we have a stationary process.

\subsubsection{Variation: Removing the drift}

If our model has no intercept it is:

\(y_t=\beta y_{t-1}+\epsilon_t\)

\(\Delta y_t=(\beta -1)y_{t-1}+\epsilon_t\)

\subsubsection{Variation: Adding a deterministic trend}

If our model has a time trend it is:

\(y_t=\alpha \beta y_{t-1}+\gamma t + \epsilon_t\)

\(\Delta y_t=\alpha + (\beta -1)y_{t-1}+\gamma t+\epsilon_t\)

\subsubsection{Augmented Dickey-Fuller}

We include more lagged variables.

\(y_t=\alpha + \beta t + \sum_i^p \theta_i y_{t-i}+\epsilon_t\)

If no unit root, can do normal OLS?


% !TEX root =  manual.tex
\section{PCA Analysis}\label{sec:pca}

The emulator is dependent on the principal component analysis (PCA). PCA also provides insight of its own. The principal components show which linear combinations of observables are likely to be well constrained, and which combinations are uninteresting. Further, one can also estimate which parameters are driving specific principal components, at least within a linear picture.

PCA is performed by the following command,

\commandline{madai\_pca\_decompose stat1}

When running, \path{madai_pca_decompose} will read the parameters file \path{stat1/settings.dat}. The PCA analysis uses the following parameters:

\begin{quote}
{\tt MODEL\_OUTPUT\_DIR "./model\_output"}\\
{\tt EXPERIMENTAL\_RESULTS\_FILE "./experimental\_results.dat"}\\
$\vdots$
\end{quote}
The PCA analysis uses the experimental information only in calculating the error for comparing a model value to the experimental value, each of which has its own error.

\path{madai_pca_decompose} produces a file \path{stat1/PCADecomposition.dat}. This file holds the eigenvalues of $\Xi$ (the strength of the principal components) and the eigenvectors of $\Xi$ (which define the rotation of $\tilde{y}$ into the principal components, $z$. If $M$ experimental values are written down, the PCA analysis will determine and record the information recorded to rotate the vector $y_1\cdots y_M$ to $z_1\cdots z_M$. The emulator will in turn use only those components $z_1\cdots z_{M_Z}$ that correspond to the largest eigenvalues, $\lambda_i$, of the matrix $\Xi$. The number of components retained, $M_Z$, can be chosen during the emulator tuning or initialization.

The \path{madai_pca_decompose} file has the following format:
\begin{quote}
{\tt OUTPUT\_MEANS}\\
$M~~\leftarrow$ Number of observables.\\
$y_1$~~~$\leftarrow$ Means for observables averaged over the $N$ model runs.\\
$y_2$~~~$\swarrow$\\
$\vdots$\\
$y_M$\\
{\tt OUTPUT\_UNCERTAINTY\_SCALES}\\
$M~~\leftarrow$ Number of observables.\\
$\sigma_1$~~~$\leftarrow$ Uncertainties that incorporate both experimental and model uncertainties.\\
$\sigma_2$~~~$\leftarrow$ If different model runs have different $\sigma_i$, model error is averaged over runs.\\
$\sigma_3$~~~$\swarrow$\\
$\vdots$\\
$\sigma_M$\\
{\tt OUTPUT\_PCA\_EIGENVALUES}\\
$M$~~$\leftarrow$ Number of observables\\
$\lambda^{(M)}$~$\leftarrow$ Smallest eigenvalue of $\Xi_{ab}$ in Eq. \eqref{eq:Xidef}\\
$\lambda^{(M-1)}$~$\swarrow$\\
$\vdots$\\
$\lambda^{(1)}$~~~~$\leftarrow$ Largest eigenvalue\\
{\tt OUTPUT\_PCA\_EIGENVECTORS}\\
$M~~~~M$\\
\vspace*{-10pt}
\[
\hspace*{-130pt}
\begin{array}{cccc}
~~~~\xi_1^{\rm(M)}&\xi_1^{\rm(M-1)}&\cdots&\xi_1^{(1)}\\
~~~~\xi_2^{\rm(M)}&\xi_2^{\rm(M-1)}&\cdots&\xi_2^{(1)}\\
~~~~\vdots&\vdots&\ddots&\vdots\\
~~~~\xi_M^{\rm(M)}&\xi_M^{\rm(2)}&\cdots&\xi_M^{(1)}\\
\end{array} ~~~\leftarrow{\rm These~are~the~normalized~eigenvectors}
\]
{\tt END\_OF\_FILE}
\end{quote}

Before building an emulator, the experimental observables and the corresponding model values are rotated into a new basis. The rotation can be stated as,
\[
z_a=U_{ab}y_b,~~y_a=U^{-1}_{ab}z_b.
\]
As described in Eq. (\ref{eq:pcarotation}), the rotation matrix is comprised of eigenvectors of the matrix $\Xi_{ab}=\langle \tilde{y}_a\tilde{y}_b\rangle$, where $\tilde{y}_a\equiv (y_a-\langle y_a\rangle)/\sigma_a$, with $\langle y_a\rangle$ denoting $y_a({\bf x})$ averaged over model values calculated from the parameters taken from the prior distribution.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
% Author: David Oniani
%
%  _         _____   __  __
% | |    __ |_   _|__\ \/ /
% | |   / _` || |/ _ \\  /
% | |__| (_| || |  __//  \
% |_____\__,_||_|\___/_/\_\
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Document Definition
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\documentclass{article}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Packages and Related Settings
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

% Global, document-wide settings
\usepackage[english]{babel}
\usepackage[margin=1in]{geometry}
\usepackage[utf8]{inputenc}

% Other packages
\usepackage{caption}
\usepackage{hyperref}
\usepackage{mathtools}
\usepackage{minted}
\usepackage{sectsty}
\usepackage{xcolor}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Setup
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

% Black-blue color
\colorlet{bb}{black!50!blue}

% Orange color
\definecolor{orange}{HTML}{f0360e}

% PDF information and nice-looking urls
\hypersetup{%
    pdfauthor={David Oniani},
    pdftitle={Bresenham's Circle Drawing Algorithm},
    pdfsubject={bresenham, circle, geometry},
    pdfkeywords={bresenham, circle, geometry},
    pdflang={English},
    colorlinks=true,
    linkcolor={bb},
    citecolor={bb},
    urlcolor={bb}
}

% Change section color
\sectionfont{\color{orange}}

% Remove indentations from paragraphs
\setlength{\parindent}{0pt}

% Setting style for the minted package
\setminted{
    fontsize=\small,
    frame=lines,
    linenos,
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Author(s), Title, and Date
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

% Author(s)
\author{David Oniani\\
        \href{mailto:onianidavid@gmail.com}{onianidavid@gmail.com}}

% Title
\title{\textcolor{orange}{\textit{Bresenham's Circle Drawing Algorithm}}}

% Date
\date{\today}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Beginning of Document
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{document}
\maketitle

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Derivation
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Derivation}

Let the radius of the circle centered at \((0, 0)\) be given and equal \(r\). Any point \((x, y)\)
on the circle satisfies the following:
\begin{equation}
    x^2 + y^2 = r^2
\end{equation}

Consider the upper half of the first quadrant. The next point could then be placed either to the
east \((x + 1, y)\) or southeast \((x + 1, y - 1)\). The errors for both scenarios are:
\begin{align}
    &e(x + 1, y) = {(x + 1)}^2 + y^2 - r^2\\
    &e(x + 1, y - 1) = {(x + 1)}^2 + {(y - 1)}^2 - r^2
\end{align}

The decision parameter \(d\) representing the total error is defined as follows:
\begin{equation}
    d = e(x + 1, y) + e(x + 1, y - 1)
      = 2{(x + 1)}^2 + y^2 + {(y - 1)}^2 - 2r^2
\end{equation}

Since the \(x\) coordinate always increases, the next point is some \((x + 1, y_n)\) and we get:
\begin{equation}
    d_n = e(x + 2, y_n) + e(x + 2, y_n - 1)
        = 2{(x + 2)}^2 + {y_n}^2 + {(y_n - 1)}^2 - 2r^2
\end{equation}

The difference is equal to the following:
\begin{equation}
    d_n - d = 2(2x + 3) + \big({y_n}^2 - y^2\big) + \big({(y_n - 1)}^2 - {(y - 1)}^2\big)
\end{equation}

Finally, we have:
\begin{align}
    &d_n - d = 2(2x + 3) &&\implies d_n
             = d + 4x + 6 &&\text{ if } d \leq 0\ (y_n = y)\\
    &d_n - d = 2(2x + 3) - 4y + 4 &&\implies d_n
             = d + 4(x - y) + 10 &&\text{ if } d > 0\ (y_n = y - 1)\\
    &d_0 = 2{(0 + 1)}^2 + r^2 + {(r - 1)}^2 - 2r^2 &&\implies d_0
         = 3 - 2r &&\text{ at initial point } (0, r)
\end{align}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Implementation
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\clearpage
\section{Implementation}

\begin{itemize}
    \item Time Complexity: \(O(r)\) where \(r\) is the radius of the circle
    \item Space Complexity: \(O(1)\)
\end{itemize}

\begin{figure}[H]
    \inputminted{python}{bresenham.py}
    \caption*{A Python Implementation of Bresenham's Circle Drawing Algorithm}
\end{figure}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% End of Document
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\end{document}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Written By Michael Brodskiy
% Class: Analytic Geometry & Calculus III (Math-292)
% Professor: V. Cherkassky
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\documentclass[12pt]{article}
\usepackage{alphalph}
\usepackage[utf8]{inputenc}
\usepackage[russian,english]{babel}
\usepackage{titling}
\usepackage{amsmath}
\usepackage{graphicx}
\usepackage{enumitem}
\usepackage{amssymb}
\usepackage[super]{nth}
\usepackage{everysel}
\usepackage{ragged2e}
\usepackage{geometry}
\usepackage{fancyhdr}
\geometry{top=1.0in,bottom=1.0in,left=1.0in,right=1.0in}
\newcommand{\subtitle}[1]{%
  \posttitle{%
    \par\end{center}
    \begin{center}\large#1\end{center}
    \vskip0.5em}%

}
\usepackage{hyperref}
\hypersetup{
colorlinks=true,
linkcolor=blue,
filecolor=magenta,
urlcolor=blue,
citecolor=blue,
}

\urlstyle{same}

\title{Lecture XIV Notes}
\date{\today}
\author{Michael Brodskiy\\ \small Professor: V. Cherkassky}

% Mathematical Operations:

% Sum: $$\sum_{n=a}^{b} f(x) $$
% Integral: $$\int_{lower}^{upper} f(x) dx$$
% Limit: $$\lim_{x\to\infty} f(x)$$

\begin{document}

\maketitle

\section{Multiple Integrals $-$ 15.1}

In calculus I, integrals were defined as the area, $A$, under a curve, where:

$$A=\lim_{x\to\infty}\sum_{i=1}^n f(x_i^*) \Delta x$$

This formula was simplified to what is known as the integral:

$$\int_a^b f(x)\,dx$$

The volume of a multivariable function, then, is given by:

$$V \approx \sum_{i=1}^m \sum_{j=1}^n f(x_{ij}^*, y_{ij}^*) \Delta A$$
$$V=\lim_{m,n\to\infty}\sum_{i=1}^m \sum_{j=1}^n f(x_{ij}^*, y_{ij}^*) \Delta A $$
$$V=\iint_R f(x,y)\,dA$$

\subsection{Double Integral Properties}

Double integrals have many of the same properties as single integrals:

$$1. \iint_R [f(x,y)+g(x,y)]\,dA=\iint_R f(x,y)\,dA + \iint_R g(x,y)\,dA$$

If $c$ is a constant:

$$\iint_R cf(x,y)\,dA=c\iint_R f(x,y)\,dA$$

If $f(x,y)\geq g(x,y)$ for all $(x,y)$, then:

$$\iint_R f(x,y)\,dA \geq \iint_R g(x,y)\,dA$$

If $g(x)$ is a function of $x$ and $h(y)$ is a function of $y$:

$$\iint_R g(x)h(y) dA=\int_a^b g(x)\,dx\int_c^d h(y)\,dy$$

\subsection{Iterated Integrals}

The double integral, $\iint_R f(x,y)\,dA$, may be broken up in order to be calculated with ease:

$$\iint_R f(x,y) dA=\int_a^b \int_c^d f(x,y)\,dy\,dx$$

This may be solved using the method of partial integrals, where, much like partial derivatives, only one of the variables is treated as a variable at once. As a result of this, just like with partial derivatives, the order that the operation is done in yields the same result no matter what:

$$\int_a^b \int_c^d f(x,y)\,dy\,dx=\int_c^d\int_a^b f(x,y)\,dx\,dy$$

\subsection{Fubini's Theorem}

If $f(x,y)$ is continuous on $R=\{(x,y)|a\leq x\leq b, c\leq y\leq d\}$

$$\iint_R f(x,y) dA=\int_a^b \int_c^d f(x,y)\,dy\,dx=\int_c^d\int_a^b f(x,y)\,dx\,dy$$

\end{document}


\documentclass[12pt]{cdblatex}
\usepackage{exercises}
\usepackage{fancyhdr}
\usepackage{footer}

\begin{document}

% --------------------------------------------------------------------------------------------
\section*{Exercise 2.2 Covariant derivative of $v_{ab}$}

\begin{cadabra}
   {a,b,c,d,e,f,g,h,i,j,k,l,m,n,o,p,q,r,s,t,u#}::Indices(position=independent).

   \nabla{#}::Derivative.
   \partial{#}::PartialDerivative.

   # template for covariant derivative of a vector

   derivU := \nabla_{a}{A?^{b}} -> \partial_{a}{A?^{b}} + \Gamma^{b}_{c a} A?^{c}.
   derivD := \nabla_{a}{A?_{b}} -> \partial_{a}{A?_{b}} - \Gamma^{c}_{b a} A?_{c}.

   vab := v_{a b} -> A_{a} B_{b}.
   iab := A_{a} B_{b} -> v_{a b}.

   pab := \partial_{a}{A_{b}} B_{c} -> \partial_{a}{A_{b} B_{c}} - A_{b} \partial_{a}{B_{c}}.

   # create an object

   Dvab := \nabla_{a}{v_{b c}}.     # cdb (ex-0202.101,Dvab)

   # apply the rule, then simplify

   substitute     (Dvab,vab)        # cdb (ex-0202.102,Dvab)
   product_rule   (Dvab)            # cdb (ex-0202.103,Dvab)
   substitute     (Dvab,derivD)     # cdb (ex-0202.104,Dvab)
   substitute     (Dvab,derivU)     # cdb (ex-0202.105,Dvab)
   distribute     (Dvab)            # cdb (ex-0202.106,Dvab)
   substitute     (Dvab,pab)        # cdb (ex-0202.107,Dvab)
   canonicalise   (Dvab)            # cdb (ex-0202.108,Dvab)
   substitute     (Dvab,iab)        # cdb (ex-0202.109,Dvab)
   sort_product   (Dvab)            # cdb (ex-0202.110,Dvab)
\end{cadabra}

\begin{align}
   \cdb{ex-0202.101} &= \Cdb{ex-0202.102}\\
                     &= \Cdb{ex-0202.103}\\
                     &= \Cdb{ex-0202.104}\\
                     &= \Cdb{ex-0202.105}\\
                     &= \Cdb{ex-0202.106}\\
                     &= \Cdb{ex-0202.107}\\
                     &= \Cdb{ex-0202.108}\\
                     &= \Cdb{ex-0202.109}\\
                     &= \Cdb{ex-0202.110}
\end{align}

\end{document}


\section{Lines}

\begin{outcome}
  \begin{enumerate}
  \item Find the vector, parametric, and symmetric equations of a line.
  \item Determine whether a point is on a given line.
  \item Determine whether two lines intersect.
  \item Find the angle between two lines.
  \item Find the projection of a point onto a line.
  \end{enumerate}
\end{outcome}

We can use the concept of vectors and points to find equations for
lines in $\R^n$. Consider a straight line $L$ that passes through a
point $P$ in the direction given by a non-zero vector $\vect{d}$.
\begin{center}
  \begin{tikzpicture}[rotate=10]
    % Note: I deliberately made the red a bit lighter and the blue a bit
    % darker, so that it will also look okay in black-and-white.
    \draw[red!80](-4,0) -- node [above=3pt, pos=0.875] {$L$} (9,0);
    \draw[->,thick,blue!80!black](0,0) -- node[above left] {$\vect{d}$} (2,0);
    \fill (0,0) circle [radius=2.2pt] node [above=3pt] {$P$};
    \fill (5,0) circle [radius=2.2pt] node [above=3pt] {$Q$};
  \end{tikzpicture}
\end{center}
The line $L$ is infinitely long in both directions, although the
picture only shows a finite part of it. To find an equation for this
line, first suppose that $Q$ is an arbitrary point on $L$. Then the
vector $\longvect{PQ}$ is parallel to $\vect{d}$. In other words,
there exists some real number $t$ such that
\begin{equation*}
  \longvect{PQ} = t\,\vect{d}.
\end{equation*}
If $\vect{p}$ is the position vector of $P$ and $\vect{q}$ is the
position vector of $Q$, we can write
\begin{equation*}
  \longvect{PQ} = \vect{q}-\vect{p}.
\end{equation*}
Putting together the last two equations, we get $\vect{q}-\vect{p} =
t\,\vect{d}$, which we can write as
\begin{equation*}
  \vect{q} = \vect{p} + t\,\vect{d}.
\end{equation*}
This is called the \textbf{vector equation}%
\index{vector equation!of a line}%
\index{line!vector equation} of the line $L$. The vector $\vect{d}$ is
called the \textbf{direction vector}%
\index{direction vector}%
\index{vector!direction vector}, and $t$ is called a
\textbf{parameter}%
\index{parameter}. The parameter $t$ can be any real number; each time
we plug in a different number for $t$, we get a different point $Q$ on
the line. The following picture shows the effect of the parameter:
\begin{center}
  \begin{tikzpicture}[rotate=10]
    % Note: I deliberately made the red a bit lighter and the blue a bit
    % darker, so that it will also look okay in black-and-white.
    \draw[red!80](-4,0) -- node [above=3pt, pos=0.92] {$L$} (10,0);
    \draw[->,thick,blue!80!black](0,0) -- node[above=3pt] {$\vect{d}$} (2,0);
    \draw[->,thick,blue!80!black](3,-2) -- node[below=3pt] {$\vect{p}$} (0,0);
    \fill (0,0) circle [radius=2.2pt] node [above=3pt] {$P$};
    \fill (-2,0) circle [radius=2.2pt] node [below=9pt] {$t=-1$};
    \fill (0,0) circle [radius=2.2pt] node [below=10pt] {$t=0$};
    \fill (2,0) circle [radius=2.2pt] node [below=10pt] {$t=1$};
    \fill (4,0) circle [radius=2.2pt] node [below=10pt] {$t=2$};
    \fill (6,0) circle [radius=2.2pt] node [below=10pt] {$t=3$};
    \fill (8,0) circle [radius=2.2pt] node [below=10pt] {$t=3$};
    \fill (3,-2) circle [radius=2.2pt] node [right=3pt] {$0$};
  \end{tikzpicture}
\end{center}
The following definition summarizes the above.

\begin{definition}{Vector equation of a line}{vector-equation-of-line}
  Let $\vect{p}$ be a vector and $\vect{d}$ a non-zero vector. Then
  \begin{equation*}
    \vect{q} = \vect{p} + t\,\vect{d}
  \end{equation*}
  is the \textbf{vector equation}%
  \index{vector equation!of a line}%
  \index{line!vector equation} of a straight line $L$. Specifically,
  as the parameter $t$ ranges over the real numbers, $\vect{q}$ ranges
  over the position vectors of all the points $Q$ on the line $L$.
  The vector $\vect{d}$ is called the \textbf{direction vector}%
  \index{direction vector}%
  \index{vector!direction vector} of the line.
\end{definition}

\begin{example}{A line from a point and a direction vector}{line-point-and-direction-vector}
  Find a vector equation for the line which contains the point
  $P = (2,0,3)$ and has direction vector
  $\vect{d} = \mat{1,2,1}^T$.
\end{example}

\begin{solution}
  The position vector of the point $P$ is
  $\vect{p}=\mat{2,0,3}^T$. The equation of the line is
  $\vect{q}= \vect{p} + t\,\vect{d}$, which we can write as
  \begin{equation*}
    \begin{mymatrix}{c} x \\ y \\ z \end{mymatrix}
    = \begin{mymatrix}{c} 2 \\ 0 \\ 3 \end{mymatrix}
    + t \begin{mymatrix}{c} 1 \\ 2 \\ 1 \end{mymatrix}.
  \end{equation*}
\end{solution}

\begin{example}{A line from two points}{line-from-two-points}
  Find a vector equation for the line through the points
  $P = (1,2,0,1)$ and $R = (2,-4,6,3)$.
\end{example}

\begin{solution}
  We can use $P$ as the base point; its position vector is
  $\vect{p}=\mat{1,2,0,1}^T$. We can use
  $\vect{d}=\longvect{PR}=\mat{1,-6,6,2}$ as the direction vector. Then
  a vector equation of the line is
  $\vect{q} = \vect{p} + t\,\vect{d}$,
  which we can also write as
  \begin{equation*}
    \begin{mymatrix}{c} x \\ y \\ z \\ w \end{mymatrix}
    = \begin{mymatrix}{c} 1 \\ 2 \\ 0 \\ 1\end{mymatrix}
    + t \begin{mymatrix}{r} 1 \\ -6 \\ 6 \\ 2\end{mymatrix}.
  \end{equation*}
\end{solution}

When we write a vector equation in the form
\begin{equation*}
  \begin{mymatrix}{c} x_1 \\ x_2 \\ \vdots \\ x_n \end{mymatrix}
  = \begin{mymatrix}{c} p_1 \\ p_2 \\ \vdots \\ p_n \end{mymatrix}
  + t \begin{mymatrix}{r} d_1 \\ d_2 \\ \vdots \\ d_n \end{mymatrix},
\end{equation*}
it is also called the \textbf{component form}%
\index{vector equation!of a line!component form}%
\index{component form!line}%
\index{line!component form} of the vector equation.

Notice that the vector equation of a line is not unique. In fact,
there are infinitely many vector equations for the same line. For
example, we can replace the parameter $t$ with another parameter, say
$3s$ or $1-r$.

\begin{example}{Change of parameter}{change-of-parameter}
  Consider the vector equation from
  Example~\ref{exa:line-point-and-direction-vector},
  \begin{equation*}
    \begin{mymatrix}{c} x \\ y \\ z \end{mymatrix}
    = \begin{mymatrix}{c} 2 \\ 0 \\ 3 \end{mymatrix}
    + t \begin{mymatrix}{c} 1 \\ 2 \\ 1 \end{mymatrix}.
  \end{equation*}
  Find two other equations for the same line, by changing the
  parameter%
  \index{line!change of parameter}%
  \index{change of parameters!line} to $3s$ and to $1-r$.
\end{example}

\begin{solution}
  If we let $t=3s$, we get
  \begin{equation*}
    \begin{array}{rcl}
      \begin{mymatrix}{c} x \\ y \\ z \end{mymatrix}
      &=& \begin{mymatrix}{c} 2 \\ 0 \\ 3 \end{mymatrix}
      + 3s \begin{mymatrix}{c} 1 \\ 2 \\ 1 \end{mymatrix}\\\\[-1ex]
      &=& \begin{mymatrix}{c} 2 \\ 0 \\ 3 \end{mymatrix}
      + s \begin{mymatrix}{c} 3 \\ 6 \\ 3 \end{mymatrix}.
    \end{array}
  \end{equation*}
  If we let $t=1-r$, we get
  \begin{equation*}
    \begin{array}{rcl}
      \begin{mymatrix}{c} x \\ y \\ z \end{mymatrix}
      &=& \begin{mymatrix}{c} 2 \\ 0 \\ 3 \end{mymatrix}
      + (1-r) \begin{mymatrix}{c} 1 \\ 2 \\ 1 \end{mymatrix}\\\\[-1ex]
      &=& \begin{mymatrix}{c} 2 \\ 0 \\ 3 \end{mymatrix}
      + \begin{mymatrix}{c} 1 \\ 2 \\ 1 \end{mymatrix}
      - r \begin{mymatrix}{c} 1 \\ 2 \\ 1 \end{mymatrix}\\\\[-1ex]
      &=& \begin{mymatrix}{c} 3 \\ 2 \\ 4 \end{mymatrix}
      + r \begin{mymatrix}{c} -1 \\ -2 \\ -1 \end{mymatrix}.
    \end{array}
  \end{equation*}
\end{solution}

\begin{definition}{Parametric equations of a line}{parametric-equations}
  A line with vector equation
  \begin{equation*}
    \begin{mymatrix}{c} x_1 \\ x_2 \\ \vdots \\ x_n \end{mymatrix}
    = \begin{mymatrix}{c} p_1 \\ p_2 \\ \vdots \\ p_n \end{mymatrix}
    + t \begin{mymatrix}{r} d_1 \\ d_2 \\ \vdots \\ d_n \end{mymatrix}
  \end{equation*}
  can also be written as a set of $n$ scalar equations:
  \begin{equation*}
    \begin{array}{c@{~}c@{~}c}
      x_1 &=& p_1 + t\,d_1, \\
      x_2 &=& p_2 + t\,d_2, \\
          &\vdots&             \\
      x_n &=& p_n + t\,d_n,
    \end{array}
  \end{equation*}
  When written in this form, they are called the \textbf{parametric
    equations}%
  \index{parametric equations!of a line}%
  \index{line!parametric equations} of the line.
\end{definition}

\begin{example}{Parametric equations}{parametric-equation}
  Find parametric equations for the line through the points
  $P = (1,2,0,1)$ and $R = (2,-4,6,3)$.
\end{example}

\begin{solution}
  This is a same line as in Example~\ref{exa:line-from-two-points}. We
  can easily convert the vector equation
  \begin{equation*}
    \begin{mymatrix}{c} x \\ y \\ z \\ w \end{mymatrix}
    = \begin{mymatrix}{c} 1 \\ 2 \\ 0 \\ 1\end{mymatrix}
    + t \begin{mymatrix}{r} 1 \\ -6 \\ 6 \\ 2\end{mymatrix}
  \end{equation*}
  to a set of parametric equations:
  \begin{equation*}
    \begin{array}{c@{~}c@{~}l}
      x &=& 1 + t, \\
      y &=& 2 - 6t, \\
      z &=& 6t, \\
      w &=& 1 + 2t.
    \end{array}
  \end{equation*}
\end{solution}

\begin{example}{Determine whether a point is on a line}{point-on-line}
  Determine whether the point $P=(5,8,4)$ is on the line $L$ given by
  the vector equation
  \begin{equation*}
    \begin{mymatrix}{c} x \\ y \\ z \end{mymatrix}
    = \begin{mymatrix}{c} 1 \\ 2 \\ 1 \end{mymatrix}
    + t \begin{mymatrix}{c} 2 \\ 3 \\ 1 \end{mymatrix}.
  \end{equation*}
\end{example}

\begin{solution}
  The point $P$ is on the line $L$ if and only if there exists some
  $t\in\R$ such that
  \begin{equation*}
    \begin{mymatrix}{c} 1 \\ 2 \\ 1 \end{mymatrix}
    + t \begin{mymatrix}{c} 2 \\ 3 \\ 1 \end{mymatrix}
    = \begin{mymatrix}{c} 5 \\ 8 \\ 4 \end{mymatrix}.
  \end{equation*}
  Subtracting $\mat{1,2,1}^T$ from both sides of the equation, this is
  equivalent to
  \begin{equation*}
    t \begin{mymatrix}{c} 2 \\ 3 \\ 1 \end{mymatrix}
    = \begin{mymatrix}{c} 4 \\ 6 \\ 3 \end{mymatrix}.
  \end{equation*}
  We can write this as a set of parametric equations:
  \begin{equation*}
    \begin{array}{c@{~}c@{~}l}
      2t &=& 4, \\
      3t &=& 6, \\
      t &=& 3.
    \end{array}
  \end{equation*}
  This is a system of three linear equations in one variable, and we
  quickly see that it is inconsistent. Therefore, the point $P$ does
  not lie on the line $L$.
\end{solution}

\begin{example}{Determine whether two lines intersect}{lines-intersect}
  Determine whether the lines
  \begin{equation*}
    \begin{mymatrix}{c} x \\ y \\ z \end{mymatrix}
    = \begin{mymatrix}{c} 3 \\ 1 \\ 0 \end{mymatrix}
    + t \begin{mymatrix}{c} 2 \\ 0 \\ 1 \end{mymatrix}
    \quad\mbox{and}\quad
    \begin{mymatrix}{c} x \\ y \\ z \end{mymatrix}
    = \begin{mymatrix}{c} 1 \\ -1 \\ 5 \end{mymatrix}
    + s \begin{mymatrix}{c} 2 \\ 1 \\ -2 \end{mymatrix}
  \end{equation*}
  intersect. If yes, find the point of intersection.%
  \index{intersection!of two lines}
\end{example}

\begin{solution}
  The two lines intersect if and only if there exist $t,s\in\R$ such
  that
  \begin{equation*}
    \begin{mymatrix}{c} 3 \\ 1 \\ 0 \end{mymatrix}
    + t \begin{mymatrix}{c} 2 \\ 0 \\ 1 \end{mymatrix}
    = \begin{mymatrix}{c} 1 \\ -1 \\ 5 \end{mymatrix}
    + s \begin{mymatrix}{c} 2 \\ 1 \\ -2 \end{mymatrix}.
  \end{equation*}
  Bringing $s$ to the left-hand side, and subtracting $\mat{3,1,0}^T$
  from both sides of the equation, this is equivalent to
  \begin{equation*}
    t \begin{mymatrix}{c} 2 \\ 0 \\ 1 \end{mymatrix}
    - s \begin{mymatrix}{c} 2 \\ 1 \\ -2 \end{mymatrix}
    = \begin{mymatrix}{c} -2 \\ -2 \\ 5 \end{mymatrix}.
  \end{equation*}
  If we write this vector equation as a set of three parametric
  equations, it is a system of 3 linear equations in 2 variables. The
  augmented matrix of the system is
  \begin{equation*}
    \begin{mymatrix}{cc|c}
      2 & -2 & -2 \\
      0 & -1 & -2 \\
      1 & 2 & 5   \\
    \end{mymatrix}.
  \end{equation*}
  This system has {\rref}
  \begin{equation*}
    \begin{mymatrix}{cc|c}
      1 & 0 & 1 \\
      0 & 1 & 2 \\
      0 & 0 & 0 \\
    \end{mymatrix},
  \end{equation*}
  and has the unique solution $t=1$ and $s=2$. Therefore, the lines
  intersect. (Other possible cases are: If the system is inconsistent,
  the lines do not intersect. If the system has more than one
  solution, the lines are identical). We find the point of
  intersection by plugging the parameter $t=1$ into the equation of
  the first line (or equivalently, but plugging $s=2$ into the
  equation of the second line - doing it both ways is a good way to
  double-check your answer). Therefore, the point of intersection is
  \begin{equation*}
    \begin{mymatrix}{c} 3 \\ 1 \\ 0 \end{mymatrix}
    + 1 \begin{mymatrix}{c} 2 \\ 0 \\ 1 \end{mymatrix}
    = \begin{mymatrix}{c} 5 \\ 1 \\ 1 \end{mymatrix}.
  \end{equation*}
\end{solution}

There is one other form for a line which is useful, which is the
\textbf{symmetric form}.  Consider the line given by
\begin{equation*}
  \begin{array}{c@{~}c@{~}l}
    x &=& 1 + 2t, \\
    y &=& 1 - t, \\
    z &=& 3 + 2t. \\
  \end{array}
\end{equation*}
We can solve each equation for $t$:
\begin{equation*}
  \begin{array}{c@{~}c@{~}l}
    t &=& \frac{x-1}{2}, \\\\[-2ex]
    t &=& \frac{y-1}{-1}, \\\\[-2ex]
    t &=& \frac{z-3}{2}.
  \end{array}
\end{equation*}
Finally, we can eliminate $t$ from the equations by setting all three
equations equal to one another:
\begin{equation*}
  \frac{x-1}{2} = \frac{y-1}{-1} = \frac{z-3}{2}.
\end{equation*}
The latter is really a system of 2 equations in 3 variables.  This is
the \textbf{symmetric form}%
\index{line!symmetric form equation}%
\index{symmetric form} of the equation of the line.  In the following
example, we look at how to convert the equation of a line from
symmetric form to parametric form.

\begin{example}{Change symmetric form to parametric form}{symmetric-to-parametric}
  Consider the line whose equations are given in \textbf{symmetric form} as
  \begin{equation*}
    \frac{x-2}{3}=\frac{y-1}{2}=\frac{z+3}{1}.
  \end{equation*}
  Find parametric and vector equations for this line.
\end{example}

\begin{solution}
  We set all three quantities equal to $t$:
  \begin{equation*}
    t=\frac{x-2}{3}, \quad
    t=\frac{y-1}{2}, \quad
    t=\frac{z+3}{1}.
  \end{equation*}
  Solving these equations for $x,y,z$ yields
  \begin{equation*}
    \begin{array}{c}
      x = 2 + 3t, \\
      y = 1 + 2t, \\
      z = -3 + t.
    \end{array}
  \end{equation*}
  These are the parametric equations for the line. The vector equation
  is
  \begin{equation*}
    \begin{mymatrix}{c}
      x \\
      y \\
      z
    \end{mymatrix} =
    \begin{mymatrix}{r}
      2 \\
      1 \\
      -3
    \end{mymatrix}
    +
    t
    \begin{mymatrix}{r}
      3 \\
      2 \\
      1
    \end{mymatrix}.
  \end{equation*}
\end{solution}

We can use the dot product to find the angle between two intersecting
lines. This is simply the smallest angle between (any of) their
direction vectors. The only subtlety here is that if $\vect{u}$ is a
direction vector for a line, then so is $-\vect{u}$, and thus we will
find pairs of complementary angles. We will take the smaller of the
two angles.
\begin{center} \begin{tikzpicture}
    \filldraw[fill=green!20,draw=green!50!black] (0,0) -- (-25:8mm)
    arc (-25:25:8mm) -- cycle;
    \filldraw[fill=yellow!20,draw=yellow!50!black] (0,0) -- (25:8mm)
    arc (25:155:8mm) -- cycle;
    \node at (0:5mm) {$\theta$};
    \node at (90:4.2mm) {$\pi-\theta$};
    \draw[red!80](25:-2.5) -- (25:2.5);
    \draw[red!80](-25:-2.5) -- (-25:2.5);
    \draw[->,thick,red!80!black](0,0) -- node[above,pos=0.6]{$\vect{v}$} (25:2);
    \draw[->,thick,blue!80!black](0,0) -- node[below,pos=0.3]{$\vect{u}$} (-25:1.5);
    \draw[->,thick,blue!80!black](0,0) -- node[below,pos=0.6]{$-\vect{u}$} (155:1.5);
  \end{tikzpicture}
\end{center}

\begin{example}{Find the angle between two lines}{angle-between-two-lines}
  Find the angle between the two lines%
  \index{line!angle between}%
  \index{angle!between two lines}
  \begin{equation*}
    L_1:  \;
    \begin{mymatrix}{r} x \\ y \\ z \end{mymatrix}
    = \begin{mymatrix}{r} 1 \\ 2 \\ 0 \end{mymatrix}
    + t \begin{mymatrix}{r} -1 \\ 1 \\ 2 \end{mymatrix}
  \end{equation*}
  and
  \begin{equation*}
    L_2: \;
    \begin{mymatrix}{r} x \\ y \\ z \end{mymatrix}
    = \begin{mymatrix}{r} 0 \\ 3 \\ 2 \end{mymatrix}
    + s \begin{mymatrix}{r} 2 \\ 1 \\ -1 \end{mymatrix}.
  \end{equation*}
\end{example}

\begin{solution}
  The direction vectors are
  \begin{equation*}
    \vect{u}=\begin{mymatrix}{r} -1 \\ 1 \\ 2 \end{mymatrix}
    \quad\mbox{and}\quad
    \vect{v}=\begin{mymatrix}{r} 2 \\ 1 \\ -1 \end{mymatrix}.
  \end{equation*}
  The answer is given by
  \begin{equation*}
    \cos\theta =
    \frac{\vect{u}\dotprod\vect{v}}{\norm{\vect{u}}\norm{\vect{v}}} = -\frac{1}{2},
  \end{equation*}
  which gives $\theta=\frac{2\pi}{3}$.  Now the angles between any two
  direction vectors for these lines will either be $\frac{2 \pi}{3}$ or
  its complement $\phi = \pi - \frac{2 \pi}{3} = \frac{\pi}{3}$. We
  choose the smaller angle, and therefore conclude that the angle
  between the two lines is $\frac{\pi}{3}$.
\end{solution}

Finally, we will show how to use projections to find the shortest
distance from a point to a line.

\begin{example}{Shortest distance from a point to a line}{shortest-point-line}
  Let $L$ be the line which goes through the point $P = (0,4,-2)$ with
  direction vector
  $\vect{d} = \begin{mysmallmatrix}{r} 2 \\ 1 \\
    2 \end{mysmallmatrix}$, and let $Q=(1,3,5)$. Find the shortest
  distance%
  \index{distance!point to line} from $Q$ to the line $L$, and find
  the point $R$ on $L$ that is closest to $Q$.%
  \index{projection!point to line}
  \begin{center}
    \begin{tikzpicture}[rotate=-20]
      \draw[red!80](-1.5,0) -- node[above, pos=0.9]{$L$} (4.5,0);
      \draw[->,thick,blue!80!black](0,0) -- node[above right, pos=0.4]{$\vect{d}$} (2,0);
      \draw[dashed] (3,1.5) -- (3,0);
      \draw[fill](0,0) circle [radius=2.25pt] node[below=1ex]{$P$};
      \draw[fill](3,1.5) circle [radius=2.25pt] node[right=0.5ex]{$Q$};
      \draw[fill](3,0) circle [radius=2.25pt] node[below=1ex]{$R$};
    \end{tikzpicture}
  \end{center}
\end{example}

\begin{solution}
  In order to determine the shortest distance from $Q$ to $L$, we will
  first find the vector $\longvect{PQ}$ and then find the projection
  of this vector onto $L$.  The vector $\longvect{PQ}$ is given by
  \begin{equation*}
    \longvect{PQ}=
    \begin{mymatrix}{r} 1 \\ 3 \\ 5 \end{mymatrix}
    - \begin{mymatrix}{r} 0 \\ 4 \\ -2 \end{mymatrix}
    = \begin{mymatrix}{r} 1 \\ -1 \\ 7 \end{mymatrix}.
  \end{equation*}
  Then, if $R$ is the point on $L$ closest to $Q$, it follows that
  \begin{equation*}
    \longvect{PR}
    ~=~ \proj_{\vect{d}}\longvect{PQ} \\
    ~=~ \frac{\vect{d}\dotprod\longvect{PQ}}{\norm{\vect{d}}^2}\,\vect{d} \\
    ~=~ \frac{15}{9} \begin{mymatrix}{r} 2 \\ 1 \\ 2 \end{mymatrix} \\
    ~=~ \frac{5}{3} \begin{mymatrix}{r} 2 \\ 1 \\ 2 \end{mymatrix}.
  \end{equation*}
  Now, the distance from $Q$ to $L$ is given by
  \begin{equation*}
    \norm{\longvect{RQ}} = \norm{\longvect{PQ} - \longvect{PR}}
    = \sqrt{26}.
  \end{equation*}
  The point $R$ is found by adding the vector $\longvect{PR}$ to the
  position vector $\longvect{0P}$ for $P$ as follows
  \begin{equation*}
    \begin{mymatrix}{r} 0 \\ 4 \\ -2 \end{mymatrix}
    + \frac{5}{3} \begin{mymatrix}{r} 2 \\ 1 \\ 2 \end{mymatrix}
    ~=~
    \begin{mymatrix}{c}
      10/3 \\
      17/3 \\
      4/3
    \end{mymatrix}.
  \end{equation*}
  Therefore, $R = (\frac{10}{3}, \frac{17}{3}, \frac{4}{3})$.
\end{solution}


\chapter{Probability Models}
Write something here.
\newpage
\section{Multivariable Gaussian Distributions}
\subsection{Product of Experts}
\[ p(x) = \frac{1}{Z}\prod_{m=1}^M p_m(x) \]
Is (informally) equivalent to the intersection of Gaussian distributions.
\begin{remark}
    Product of Experts is beyond the scope of this course.
\end{remark}
\subsection{Mixture of Experts}
Weighted sum of multiple distributions.
\[ p(x) = \sum_{m=1}^M c_m p_m(x) \]
\begin{remark}
    Weighted sum of Gaussians is not necessarily a Gaussian. So mixture of Gaussians can be used to ``fit'' different distributions.
\end{remark}

\section{Gaussian Mixture Model}
\subsection{Sampling from a GMM}
\begin{enumerate}
    \item Sample $c_m$ from $c_1,c_2,\dots,c_M$
    \item Sample data from $\mathcal{N}(x|\mu_m,\Sigma_m)$
\end{enumerate}

\section{Some Information Theory}
\textbf{Information} contains \textbf{uncertainty}. The more uncertain something is, the more information is needed to eliminate its uncertainty.

\begin{definition}
    The information in discrete variable is
    \[ I(x) = -\log_2 P(x) \]
\end{definition}
\begin{definition}[Entropy]
    The
    \[ H = \mathbb{E}[-\log_2 P(x)] = -\sum_{x\in X}P(x)\log_2 P(x) \]
    \[ H = \mathbb{E}[I(x) = -\int_{-\infty}^{\infty}p(x)\ln(p(x))\mathrm{d}x] \]
\end{definition}
\begin{remark}
    \begin{itemize}
        \item Entropy is a function of distribution $p(x)$.
        \item Is the expectation of information of random variables.
    \end{itemize}
\end{remark}

\subsection{Entropy of common distriutions}

\subsection{Conditional Entropy}
\begin{definition}
    \[ H[y|x] = \sum_{x'}P(x)H(y|x=x') = -\sum_{x}P(x)\sum_yP(y|x)\ln P(TooFast) \]
\end{definition}


\documentclass{article}
\usepackage[utf8]{inputenc}

\usepackage{fullpage}
\usepackage{amsmath}
\usepackage{amsfonts}

\title{Numerical Methods for Solving Navier-Stokes Equations}
\author{Nikolas Tselepidis}
\date{}

\begin{document}

\maketitle

\section{Finite Pr Equations}

\paragraph{Conservation of momentum (dimensionless)}

\begin{equation}
\frac{1}{Pr} \left( \frac{\partial \vec{v}}{\partial t} + \vec{v} \cdot \nabla \vec{v} \right) =
    - \nabla P + \nabla^2 \vec{v} + Ra T \hat{y}
\end{equation}

Taking the curl of equation (1), and substituting the vorticity $\vec{\omega} = \nabla \times \vec{v}$, it follows that:

\begin{equation}
    \frac{1}{Pr} \left( \frac{\partial \omega}{\partial t} + v_x \frac{\partial \omega}{\partial x} + v_y \frac{\partial \omega}{\partial y} \right) =
     \nabla^2 \omega - Ra \frac{\partial T}{\partial x}
\end{equation}

In 2D only one component of vorticity is needed, i.e. the one perpedicular to the 2D plane:  $\nabla^2 \psi = \omega_z$.

\paragraph{Conservation of mass (continuity)}
\begin{equation}
    \frac{\partial T}{\partial t} + \vec{v} \cdot \nabla T = \kappa \nabla^2 T
\end{equation}

\paragraph{Conservation of energy}
\begin{equation}
    \nabla \cdot \vec{v} = 0
\end{equation}

\section{Solve}

\begin{equation}
    \nabla^2 \psi = \omega
\end{equation}

\begin{equation}
    \left( v_x, v_y \right) = \left( \frac{\partial \psi}{\partial y}, -\frac{\partial \psi}{\partial x} \right)
\end{equation}

\begin{equation}
    \frac{\partial T}{\partial t} = -v_x \frac{\partial T}{\partial x} -v_y \frac{\partial T}{\partial y} + \kappa \nabla^2 T
\end{equation}

\begin{equation}
    \frac{\partial \omega}{\partial t} = -v_x \frac{\partial \omega}{\partial x} -v_y \frac{\partial \omega}{\partial y} + Pr \nabla^2 \omega - Ra Pr \frac{\partial T}{\partial x}
\end{equation}

\end{document}


\documentclass[12pt]{cdblatex}
\usepackage{examples}
\usepackage{fancyhdr}
\usepackage{footer}

\begin{document}

% ============================================================================================
\bgroup
\CdbSetup{action=hide}
\begin{cadabra}
   import cdblib
   checkpoint_file = 'tests/semantic/output/example-11.json'
   cdblib.create (checkpoint_file)
   checkpoint = []
\end{cadabra}
\egroup

\clearpage

% ============================================================================================
\section*{Example 11 The RNC connection.}

\begin{cadabra}
   {a,b,c,d,e,f,g,h,i,j,k,l,m,n,o,p,q,r,s,u#}::Indices(position=independent).

   D{#}::PartialDerivative.
   \nabla{#}::Derivative.

   g_{a b}::Metric.
   g^{a b}::InverseMetric.
   g^{a b}::Weight(label=gnum,value=1).

   \delta{#}::KroneckerDelta.

   R_{a b c d}::RiemannTensor.
   R_{a b c d}::Depends(\nabla{#}).

   x^{a}::Depends(D{#}).
   x^{a}::Weight(label=xnum,value=1).

   Dx := D_{a}{x^{b}} -> \delta^{b}_{a}.   # cdb (ex-11.000,Dx)

   gab := g_{a b} -> g_{a b}
                  -  (1/3)   x^{c} x^{d} R_{a c b d}
                  -  (1/6)   x^{c} x^{d} x^{e} \nabla_{c}{R_{a d b e}}
                  +  (1/180) x^{c} x^{d} x^{e} x^{f} ( 8 g^{g h} R_{a c d g} R_{b e f h}
                                                      -9 \nabla_{c d}{R_{a e b f}} ).      # cdb (ex-11.001,gab)

   iab := g^{a b} -> g^{a b}
                  + (1/3)  x^{c} x^{d} g^{a e} g^{b f} R_{c e d f}
                  + (1/6)  x^{c} x^{d} x^{e} g^{a f} g^{b g} \nabla_{c}{R_{d f e g}}
                  + (1/60) x^{c} x^{d} x^{e} x^{f} g^{a g} g^{b h}
                                         ( 4 g^{i j} R_{c g d i} R_{e h f j}
                                          +3 \nabla_{c d}{R_{e g f h}} ).                  # cdb(ex-11.002,iab)

   distribute (gab)
   distribute (iab)

   ChrSym := \Gamma^{a}_{b c} -> 1/2 g^{a d} (  D_{b}{g_{d c}}
                                              + D_{c}{g_{b d}}
                                              - D_{d}{g_{b c}} ).   # cdb (ex-11.003,ChrSym)

   Gamma := \Gamma^{a}_{b c}.              # cdb (ex-11.100,Gamma)

   substitute     (Gamma,ChrSym)           # cdb (ex-11.101,Gamma)
   substitute     (Gamma,gab)              # cdb (ex-11.102,Gamma)
   substitute     (Gamma,iab)              # cdb (ex-11.103,Gamma)
   distribute     (Gamma)                  # cdb (ex-11.104,Gamma)
   unwrap         (Gamma)                  # cdb (ex-11.105,Gamma)
   product_rule   (Gamma)                  # cdb (ex-11.106,Gamma)
   distribute     (Gamma)                  # cdb (ex-11.107,Gamma)
   substitute     (Gamma,Dx)               # cdb (ex-11.108,Gamma)
   eliminate_kronecker (Gamma)             # cdb (ex-11.109,Gamma)

   def truncate (obj,n):

       ans = Ex("0")  # create a Cadabra object with value zero

       for i in range (0,n+1):
          foo := @(obj).
          bah  = Ex("xnum = " + str(i))
          distribute  (foo)
          keep_weight (foo, bah)
          ans = ans + foo

       return ans

   checkpoint.append (Gamma)

   # sort_product   (Gamma)  # 52.3 sec, 49 Mbyte
   # rename_dummies (Gamma)  # 58.6 sec, 51 Mbyte
   # canonicalise   (Gamma)  # killed after 20 mins and over 500 Mbyte

   Gamma = truncate (Gamma,3)   # cdb (ex-11.110,Gamma)  # allow up to 3rd order in x^a

   sort_product   (Gamma)
   rename_dummies (Gamma)
   canonicalise   (Gamma)

   checkpoint.append (Gamma)

   # ==========================================================================
   # the remaining code is just for pretty printing

   def product_sort (obj):
       substitute (obj,$ g^{a b}                   -> A001^{a b}                $)
       substitute (obj,$ x^{a}                     -> A002^{a}                  $)
       substitute (obj,$ z^{a}                     -> A003^{a}                  $)
       substitute (obj,$ R_{a b c d}               -> A004_{a b c d}            $)
       substitute (obj,$ \nabla_{e}{R_{a b c d}}   -> A005_{a b c d e}          $)
       substitute (obj,$ \nabla_{e f}{R_{a b c d}} -> A006_{a b c d e f}        $)
       sort_sum       (obj)
       sort_product   (obj)
       rename_dummies (obj)
       substitute (obj,$ A001^{a b}                -> g^{a b}                   $)
       substitute (obj,$ A002^{a}                  -> x^{a}                     $)
       substitute (obj,$ A003^{a}                  -> z^{a}                     $)
       substitute (obj,$ A004_{a b c d}            -> R_{a b c d}               $)
       substitute (obj,$ A005_{a b c d e}          -> \nabla_{e}{R_{a b c d}}   $)
       substitute (obj,$ A006_{a b c d e f}        -> \nabla_{e f}{R_{a b c d}} $)

   def get_xterm (obj,n):

       foo := @(obj).
       bah  = Ex("xnum = " + str(n))
       distribute  (foo)
       keep_weight (foo, bah)

       return foo

   def get_gterm (obj,n):

       foo := @(obj).
       bah  = Ex("gnum = " + str(n))
       distribute  (foo)
       keep_weight (foo, bah)

       return foo

   def reformat (obj,scale):

      foo  = Ex(str(scale))
      bah := @(foo) @(obj).

      distribute     (bah)
      product_sort   (bah)
      rename_dummies (bah)
      canonicalise   (bah)
      factor_out     (bah,$x^{a?},g^{b? c?}$)
      ans := @(bah) / @(foo).

      return ans

   gam1  = get_xterm (Gamma, 1)                          # cdb (ex-11.200,gam1)
   gam2  = get_xterm (Gamma, 2)                          # cdb (ex-11.201,gam2)
   gam3  = get_xterm (Gamma, 3)                          # cdb (ex-11.202,gam3)

   gam31 = get_gterm (gam3,  1)                          # cdb (ex-11.210,gam31)
   gam32 = get_gterm (gam3,  2)                          # cdb (ex-11.211,gam31)

   gam1  = reformat (gam1,   3)                          # cdb (ex-11.220,gam1)
   gam2  = reformat (gam2,  12)                          # cdb (ex-11.221,gam2)

   gam31 = reformat (gam31, 40)                          # cdb (ex-11.222,gam31)
   gam32 = reformat (gam32, 45)                          # cdb (ex-11.223,gam32)

   Gamma  := @(gam1) + @(gam2) + @(gam31) + @(gam32).    # cdb (ex-11.230,Gamma)
   Scaled := 360 @(Gamma).                               # cdb (ex-11.231,Scaled)

   checkpoint.append (Gamma)

\end{cadabra}

\clearpage

\begin{dgroup*}
   \Dmath*{\cdb{ex-11.100}(x) = \Cdb*[\hskip 3.0cm]{ex-11.230}}
\end{dgroup*}

\begin{dgroup*}
   \Dmath*{360 \cdb{ex-11.100}(x) = \Cdb*[\hskip 3.0cm]{ex-11.231}}
\end{dgroup*}

\clearpage

% ============================================================================================
% export to json format

% --------------------------------------------------------------------------------------------
\subsection*{Save $\Gamma^{a}{}_{bc}$ for later use in Example 12.}

\begin{cadabra}
   jsonfile = 'example-11.json'
   cdblib.create (jsonfile)
   cdblib.put ('Gamma',Gamma,jsonfile)
\end{cadabra}

\bgroup
\CdbSetup{action=hide}
\begin{cadabra}
   for i in range( len(checkpoint) ):
      cdblib.put ('check{:03d}'.format(i),checkpoint[i],checkpoint_file)
\end{cadabra}
\egroup

\end{document}


The chapter presents two ways of doing hypothesis tests on whole
models: (1) permutation tests where the connection is severed between
the explanatory and response variables, (2) tests such as \ANOVA\
where the sampling distribution is calculated from first principles.
In practice, first-principle tests are used most of the time.
Still, the permutation test is useful for developing intuition about
hypothesis testing --- our main purpose here ---
and for those not-so-rare occasions where the assumptions
behind the first-principle tests are dubious.

\subsection{The Permutation Test}

The idea of a permutation test is to enforce the null hypothesis that
there is no connection between the response variables and the
explanatory variables.  An effective way to do this is to randomize
the response variable in a way that is consistent with sampling
variability.  When constructing confidence intervals, the
\function{resampling} function was used.  Re-sampling will typically
repeat some cases and omit others.  Here, the \function{shuffle}
function will be used instead, to scramble the order of one or more
variables while leaving the others in their original state.

To illustrate, consider a model for exploring whether \VN{sex} and
\VN{mother}'s height are related to the height of the child: \datasetGalton
\begin{Schunk}
\begin{Sinput}
> galton = fetchData("galton.csv")
> mod = lm(height ~ sex + mother, data=galton)
> coefficients(mod)
\end{Sinput}
\begin{Soutput}
(Intercept)        sexM      mother
     41.450       5.177       0.353
\end{Soutput}
\end{Schunk}
The coefficients indicate that typical males are taller than typical
females by about 5 inches and that for each inch taller the mother is,
a child will typically be taller by 0.35 inches.
A reasonable test statistic to summarize the whole model is $R^2$:
\begin{Schunk}
\begin{Sinput}
> rsquared(mod)
\end{Sinput}
\begin{Soutput}
[1] 0.562
\end{Soutput}
\end{Schunk}

For confidence intervals, re-sampling was applied to the entire data
frame.  This selects random cases, but each selected case is an
authentic one that matches exactly the original values for that case.
The point of re-sampling is to get an idea of the variability
introduced by random sampling of authentic cases.
\begin{Schunk}
\begin{Sinput}
> do(5) * lm(height ~ sex + mother, data=resample(galton))
\end{Sinput}
\begin{Soutput}
  Intercept sexM mother sigma r-squared
1      42.3 5.37  0.337  2.29     0.593
2      42.1 4.93  0.344  2.34     0.550
3      43.5 5.23  0.321  2.36     0.568
4      37.9 5.26  0.409  2.34     0.579
5      39.9 5.14  0.378  2.29     0.570
\end{Soutput}
\end{Schunk}
The \indicatorVar{sex}{M} coefficients are tightly grouped near 5
inches, the \VN{mother} coefficients are around 0.3 to 0.4.

\index{P}{resample@\texttt{resample}*}
\index{P}{shuffle@\texttt{shuffle}*}
\index{P}{Hypothesis Testing!resample@\texttt{resample}*}
\index{P}{Hypothesis Testing!shuffle@\texttt{shuffle}*}

\index{P}{Resampling!resample@\texttt{resample}*}
\index{P}{Resampling!shuffle@\texttt{shuffle}*}
\index{C}{permutation test!computing}

In order to carry out a permutation test, do not randomize the
whole data frame.  Instead, shuffle just the response variable:
\begin{Schunk}
\begin{Sinput}
> do(5) * lm(shuffle(height) ~ sex + mother, data=galton)
\end{Sinput}
\begin{Soutput}
  Intercept     sexM  mother sigma r-squared
1      63.7 -0.30663  0.0507  3.58   0.00300
2      72.3  0.00676 -0.0859  3.58   0.00306
3      70.8  0.00563 -0.0629  3.58   0.00164
4      70.2  0.02282 -0.0533  3.58   0.00119
5      68.3 -0.55719 -0.0192  3.58   0.00613
\end{Soutput}
\end{Schunk}
Now the \indicatorVar{sex}{M} and \VN{mother}
coefficients are close to zero, as would be expected when
there is no relationship between the response variable and the
explanatory variables.

In constructing the sampling distribution under the null hypothesis,
you should do hundreds of trials of fitting the model to the scrambled
data, calculating the test statistic ($R^2$ here) for each trial.  Note that each
trial itself involves all of the cases in your sample, but those cases
have been changed so that the shuffled variable almost certainly
takes on a different value in every case than in the original data.
\begin{Schunk}
\begin{Sinput}
> nulltrials = do(500) *  rsquared(
     lm(shuffle(height) ~ sex + mother, data=galton))
\end{Sinput}
\end{Schunk}
Notice that the \function{rsquared} operator has been used to calculate
the test statistic $R^2$ from the model.  The output of \function{do}
is a data frame:
\begin{Schunk}
\begin{Sinput}
> nulltrials
\end{Sinput}
\end{Schunk}
\begin{Schunk}
\begin{Soutput}
       var
1 0.000920
2 0.001854
3 0.000533
... for 500 cases altogether ...
\end{Soutput}
\end{Schunk}
Naturally, all of the $R^2$ values for the trials are close to zero.
After all,  there is no
relation between the response variable (after randomization with
\function{shuffle}) and the explanatory variables.

The p-value can be calculated directly from the trials, by comparison
to the observed value in the actual data: $R^2$ was 0.5618.
\begin{Schunk}
\begin{Sinput}
> count(var > 0.5618, nulltrials)
\end{Sinput}
\begin{Soutput}
count.FALSE
        500
\end{Soutput}
\end{Schunk}
None of the 500 trials were greater than the value of the test statistic, 0.5618.
It wouldn't be fair to claim that $p=0$, since we only did 500 trials,
but it is reasonable to say that the permutation test shows the
p-value is $p \leq 1/500$.

\subsection{First-Principle Tests}

\index{C}{regression report!hypothesis testing}
\index{C}{hypothesis testing!in regression report}
On modern computers, the permutation test is entirely practical.  But
a few decades ago, it was not.  Great creativity was applied to
finding test statistics where the sampling distribution could be
estimated without extensive calculation.  One of these is the F
statistic.  This is still very useful today and is a standard part of
the regression report in many statistical packages.

\index{P}{summary@\texttt{summary}!for linear models}
\index{P}{Hypothesis Testing!summary@\texttt{summary}!for linear models}

Here is the regression report from the
\model{\VN{height}}{\VN{sex}+\VN{mother}}:
\begin{Schunk}
\begin{Sinput}
> mod = lm( height ~ sex + mother, data=galton)
> summary(mod)
\end{Sinput}
\begin{Soutput}
...
            Estimate Std. Error t value Pr(>|t|)
(Intercept)  41.4495     2.2095    18.8   <2e-16 ***
sexM          5.1767     0.1587    32.6   <2e-16 ***
mother        0.3531     0.0344    10.3   <2e-16 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 2.37 on 895 degrees of freedom
Multiple R-squared: 0.562,	Adjusted R-squared: 0.561
F-statistic:  574 on 2 and 895 DF,  p-value: <2e-16
\end{Soutput}
\end{Schunk}
The last line of the report
shows an F statistic of 574 based on an $R^2$ of 0.562 and
translates this to a p-value that is practically zero: \code{<2e-16}.

By way of showing that the regression report is rooted in the same
approach shown in the chapter, you can confirm the calculations.
There are $m=3$ coefficients and $n=898$ cases,
producing $n-m=895$ degrees of freedom in the denominator and $m-1=2$
degrees of freedom in the numerator. The calculation of the F
statistic from $R^2$ and the degrees of freedom follows the formula
given in the chapter.
$$ F = \frac{R^2}{m-1} \Biggm/ \frac{1-R^2}{n-m} $$
Plugging the values into the formula
\begin{Schunk}
\begin{Sinput}
> (0.562/2) / ((1-.562)/895)
\end{Sinput}
\begin{Soutput}
[1] 574
\end{Soutput}
\end{Schunk}
F is the test statistic.  To convert it to a p-value, you need to
calculate how extreme the value of F$=574.2$ is with reference to the
F distribution with 895 and 2 degrees of freedom.
\begin{Schunk}
\begin{Sinput}
> 1 - pf( 574.2, 2, 895)
\end{Sinput}
\begin{Soutput}
[1] 0
\end{Soutput}
\end{Schunk}
\index{P}{pf@\texttt{pf}}
\index{P}{Hypothesis Testing!pf@\texttt{pf}}
\index{C}{p value!from F statistic}
The calculation of p-values from F always follows this form.  In the
context of the F distribution, ``extreme'' always means ``bigger
than.''  So, calculate the area under the F distribution to the right
of the observed value.