\section{Expression types}\label{sec:dsdl_expression_types}

Expression types are a special category of data types whose instances can only exist and be operated upon
at the time of DSDL definition processing.
As such, expression types cannot be used to define attributes,
and their instances cannot be exchanged between nodes.

Expression types are used to represent values of constant expressions which are evaluated
when a DSDL definition is processed.
Results of such expressions can be used to define various constant properties,
such as array length boundaries or values of constant attributes.

Expression types are specified in this section.
Each expression type has a formal DSDL name for completeness;
even if such types can't be used to define attributes,
a well-defined formal name allows DSDL processing tools to emit well-formed
and understandable diagnostic messages.

\subsection{Rational number}\label{sec:dsdl_rational}

At the time of DSDL definition processing, integer and real numbers are represented internally as rational numbers
where the range of numerator and denominator is unlimited\footnote{%
Technically, the range may only be limited by the memory resources available to the DSDL processing tool.}.
DSDL processing tools are not permitted to introduce any implicit rational number transformations that
may result in a loss of information.

The DSDL name of the rational number type is ``\verb|rational|''.

Rational numbers are assumed to be stored in a normalized form, where the denominator is positive
and the greatest common divisor of the numerator and the denominator is one.

A rational number can be used in a context where an integer value is expected only if its denominator equals one.

Implicit conversions between boolean-valued entities and rational numbers are not allowed.

\begin{UAVCANSimpleTable}{Operators defined on instances of rational numbers}{|l l X X|}
    Op & Type & Constraints & Description
    \label{table:dsdl_operators_rational} \\

    \texttt{\textbf{+}}     & $(\texttt{rational}) \rightarrow \texttt{rational}$ & &
    No effect. \\
    \texttt{\textbf{-}}     & $(\texttt{rational}) \rightarrow \texttt{rational}$ & &
    Negation. \\

    \texttt{\textbf{**}}    & $(\texttt{rational}, \texttt{rational}) \rightarrow \texttt{rational}$ &
    Power denominator equals one &
    Exact exponentiation. \\

    \texttt{\textbf{**}}    & $(\texttt{rational}, \texttt{rational}) \rightarrow \texttt{rational}$ &
    Power denominator greater than one &
    Exponentiation with imp\-lem\-en\-ta\-ti\-on-de\-fin\-ed accuracy. \\

    \texttt{\textbf{*}}  & $(\texttt{rational}, \texttt{rational}) \rightarrow \texttt{rational}$ & &
    Exact multiplication. \\

    \texttt{\textbf{/}}  & $(\texttt{rational}, \texttt{rational}) \rightarrow \texttt{rational}$ &
    Non-zero divisor &
    Exact division. \\

    \texttt{\textbf{\%}} & $(\texttt{rational}, \texttt{rational}) \rightarrow \texttt{rational}$ &
    Non-zero divisor &
    Exact modulo. \\

    \texttt{\textbf{+}}  & $(\texttt{rational}, \texttt{rational}) \rightarrow \texttt{rational}$ & &
    Exact addition. \\

    \texttt{\textbf{-}}  & $(\texttt{rational}, \texttt{rational}) \rightarrow \texttt{rational}$ & &
    Exact subtraction. \\

    \texttt{\textbf{|}}  & $(\texttt{rational}, \texttt{rational}) \rightarrow \texttt{rational}$ &
    Denominators equal one &
    Bitwise or. \\

    \texttt{\textbf{\textasciicircum{}}} & $(\texttt{rational}, \texttt{rational}) \rightarrow \texttt{rational}$ &
    Denominators equal one &
    Bitwise xor. \\

    \texttt{\textbf{\&}} & $(\texttt{rational}, \texttt{rational}) \rightarrow \texttt{rational}$ &
    Denominators equal one &
    Bitwise and. \\

    \texttt{\textbf{!=}} & $(\texttt{rational}, \texttt{rational}) \rightarrow \texttt{bool}$ & & Exact inequality. \\
    \texttt{\textbf{==}} & $(\texttt{rational}, \texttt{rational}) \rightarrow \texttt{bool}$ & & Exact equality. \\
    \texttt{\textbf{<=}} & $(\texttt{rational}, \texttt{rational}) \rightarrow \texttt{bool}$ & & Less or equal. \\
    \texttt{\textbf{>=}} & $(\texttt{rational}, \texttt{rational}) \rightarrow \texttt{bool}$ & & Greater or equal. \\
    \texttt{\textbf{<}}  & $(\texttt{rational}, \texttt{rational}) \rightarrow \texttt{bool}$ & & Strictly less. \\
    \texttt{\textbf{>}}  & $(\texttt{rational}, \texttt{rational}) \rightarrow \texttt{bool}$ & & Strictly greater. \\

\end{UAVCANSimpleTable}

\subsection{Unicode string}\label{sec:dsdl_string}

This type contains a sequence of Unicode characters.
It is used to represent string literals internally.

The DSDL name of the Unicode string type is ``\verb|string|''.

A Unicode string containing one symbol whose code point is within $[0, 127]$
(i.e., an ASCII character) is implicitly convertible into a \verb|uint8|-typed constant attribute value,
where the value of the constant is to be equal the code point of the symbol.

\begin{UAVCANSimpleTable}{Operators defined on instances of Unicode strings}{|l l X|}
    Op & Type & Description
    \label{table:dsdl_operators_string} \\

    \texttt{\textbf{+}}  & $(\texttt{string}, \texttt{string}) \rightarrow \texttt{string}$ &
    Concatenation. \\

    \texttt{\textbf{!=}} & $(\texttt{string}, \texttt{string}) \rightarrow \texttt{bool}$ &
    Inequality of Unicode NFC normalized forms.
    NFC stands for \emph{Normalization Form Canonical Composition} --
    one of standard Unicode normalization forms where characters are recomposed by canonical equivalence. \\

    \texttt{\textbf{==}} & $(\texttt{string}, \texttt{string}) \rightarrow \texttt{bool}$ &
    Equality of Unicode NFC normalized forms. \\

\end{UAVCANSimpleTable}

The set of operations and conversions defined for Unicode strings is to be extended in future versions of
the specification.

\subsection{Set}\label{sec:dsdl_set}

A set type represents an unordered collection of unique objects.
All objects must be of the same type.
Uniqueness of elements is determined by application of the equality operator ``\verb|==|''.

The DSDL name of the set type is ``\verb|set|''.

A set can be constructed from a set literal, in which case such set must contain at least one element.

The attributes and operators defined on set instances are listed in the tables~\ref{table:dsdl_set_attributes}
and~\ref{table:dsdl_set_operators}, where $E$ represents the set element type.

\begin{UAVCANSimpleTable}{Attributes defined on instances of sets}{|l l X X|}
    Name & Type & Constraints & Description
    \label{table:dsdl_set_attributes} \\

    \texttt{min} & $E$ &
    Operator ``\texttt{<}'' is defined \mbox{$(E, E) \rightarrow \texttt{bool}$} &
    Smallest element in the set determined by sequential application of the operator ``\texttt{<}''. \\

    \texttt{max} & $E$ &
    Operator ``\texttt{>}'' is defined \mbox{$(E, E) \rightarrow \texttt{bool}$} &
    Greatest element in the set determined by sequential application of the operator ``\texttt{>}''. \\

    \texttt{count} & \texttt{rational} & &
    Cardinality. \\

\end{UAVCANSimpleTable}

\newcommand\SetElementwiseOperator[1]{%
    \texttt{\textbf{#1}} & $(\texttt{set}_\texttt{<E>}, E) \rightarrow \texttt{set}_\texttt{<R>}$ & $E$ is not a set &
    Elementwise $(E, E) \rightarrow R$.\\

    \texttt{\textbf{#1}} & $(E, \texttt{set}_\texttt{<E>}) \rightarrow \texttt{set}_\texttt{<R>}$ & $E$ is not a set &
    Elementwise $(E, E) \rightarrow R$.\\
}

\begin{UAVCANSimpleTable}{Operators defined on instances of sets}{|l l l X|}%
    \label{table:dsdl_set_operators}%
    Op & Type & Constraints & Description \\

    \texttt{\textbf{==}} & $(\texttt{set}_\texttt{<E>}, \texttt{set}_\texttt{<E>}) \rightarrow \texttt{bool}$ & &
    Left equals right. \\

    \texttt{\textbf{!=}} & $(\texttt{set}_\texttt{<E>}, \texttt{set}_\texttt{<E>}) \rightarrow \texttt{bool}$ & &
    Left does not equal right. \\

    \texttt{\textbf{<=}} & $(\texttt{set}_\texttt{<E>}, \texttt{set}_\texttt{<E>}) \rightarrow \texttt{bool}$ & &
    Left is a subset of right. \\

    \texttt{\textbf{>=}} & $(\texttt{set}_\texttt{<E>}, \texttt{set}_\texttt{<E>}) \rightarrow \texttt{bool}$ & &
    Left is a superset of right. \\

    \texttt{\textbf{<}}  & $(\texttt{set}_\texttt{<E>}, \texttt{set}_\texttt{<E>}) \rightarrow \texttt{bool}$ & &
    Left is a proper subset of right. \\

    \texttt{\textbf{>}}  & $(\texttt{set}_\texttt{<E>}, \texttt{set}_\texttt{<E>}) \rightarrow \texttt{bool}$ & &
    Left is a proper superset of right. \\

    \texttt{\textbf{|}} &
    $(\texttt{set}_\texttt{<E>}, \texttt{set}_\texttt{<E>}) \rightarrow \texttt{set}_\texttt{<E>}$ & &
    Union. \\

    \texttt{\textbf{\textasciicircum{}}} &
    $(\texttt{set}_\texttt{<E>}, \texttt{set}_\texttt{<E>}) \rightarrow \texttt{set}_\texttt{<E>}$ & &
    Disjunctive union. \\

    \texttt{\textbf{\&}} &
    $(\texttt{set}_\texttt{<E>}, \texttt{set}_\texttt{<E>}) \rightarrow \texttt{set}_\texttt{<E>}$ & &
    Intersection. \\

    \SetElementwiseOperator{**}
    \SetElementwiseOperator{*}
    \SetElementwiseOperator{/}
    \SetElementwiseOperator{\%}
    \SetElementwiseOperator{+}
    \SetElementwiseOperator{-}

\end{UAVCANSimpleTable}

\subsection{Serializable metatype}\label{sec:dsdl_metaserializable}

Serializable types (which are reviewed in section \ref{sec:dsdl_serializable_types})
are instances of the serializable metatype.
This metatype is convenient for expression of various relations and attributes defined on serializable types.

The DSDL name of the serializable metatype is ``\verb|metaserializable|''.

Available attributes are defined on a per-instance basis.


\section{Elimination of nondeterminism}

\subsection{Elimination of $\epsilon$-moves}
For each $p \xrightarrow{\epsilon} q$:
\begin{itemize}
    \item For each $q \xrightarrow{a} r$ where $a$ can be anything (including $\epsilon$) add an arc $p \xrightarrow{a} r$
    \item If $q$ was final, $p$ becomes final
    \item Remove $p \xrightarrow{\epsilon} q$
\end{itemize}

If there is a cycle of $\epsilon$-moves all the involved nodes can be merged.

\subsection{Berry-Sethi Method}
\begin{itemize}
    \item Number non-$\epsilon$ arcs and add $\dashv$ on exiting final states
    \item Compute $Ini$ and $Fin$ with the same rules as r.e.
    \item Apply Berry-Sethi construction
\end{itemize}
The automaton obtained is deterministic but can be non-minimal.


\subsection{The singular value decomposition}

We begin this section with an important definition.

\begin{definition}{Singular values}{singular-values}
Let $A$ be an $m\times n$-matrix. The singular values of $A$ are the square roots of the positive
eigenvalues of $A^TA$\index{singular values}.
\end{definition}

Singular Value Decomposition (SVD) can be thought of as
a generalization of orthogonal diagonalization of a symmetric matrix
to an arbitrary $m\times n$-matrix. This decomposition is the focus of this section.

The following is a useful result that will help when computing the SVD of matrices.

\begin{proposition}{}{same-non-zero-eigenvalues}
Let $A$ be an $m \times n$-matrix. Then $A^TA$ and $AA^T$ have the same \bf{non-zero} eigenvalues.
\end{proposition}

\begin{proof}
Suppose $A$ is an $m\times n$-matrix, and suppose that  $\lambda$ is a non-zero eigenvalue of $A^TA$.
Then there exists a non-zero vector $X\in \R^n$ such that

\begin{equation}\label{non-zero}
(A^TA)X=\lambda X.
\end{equation}

Multiplying both sides of this equation by $A$ yields:
\begin{eqnarray*}
A(A^TA)X & = & A\lambda X\\
(AA^T)(AX) & = & \lambda (AX).
\end{eqnarray*}
Since $\lambda\neq 0$ and $X\neq 0_n$, $\lambda X\neq 0_n$,
and thus by equation~\eqref{non-zero},
$(A^TA)X\neq 0_m$; thus $A^T(AX)\neq 0_m$,
implying that $AX\neq 0_m$.

Therefore $AX$ is an eigenvector of $AA^T$ corresponding to eigenvalue
$\lambda$.  An analogous argument can be used to show that every
non-zero eigenvalue of $AA^T$ is an eigenvalue of $A^TA$, thus
completing the proof.
\end{proof}

Given an $m\times n$-matrix $A$, we will see how to express $A$ as a product
\[ A=U\Sigma V^T\]
where
\begin{itemize}
\item $U$ is an $m\times m$ orthogonal matrix whose columns are
eigenvectors of $AA^T$.
\item $V$ is an $n\times n$ orthogonal matrix whose columns are
eigenvectors of $A^TA$.
\item $\Sigma$ is an $m\times n$-matrix whose only non-zero values
lie on its main diagonal, and are the singular values of $A$.
\end{itemize}

How can we find such a decomposition? We are aiming to decompose $A$ in the following form:

\begin{equation*}
A=U\begin{mymatrix}{cc}
\sigma & 0 \\
0 & 0
\end{mymatrix} V^T
\end{equation*}

where $\sigma $ is of the form
\[
\sigma =\begin{mymatrix}{ccc}
\sigma _{1} &  & 0 \\
& \ddots &  \\
0 &  & \sigma _{k}
\end{mymatrix}
\]

Thus $A^T=V\begin{mymatrix}{cc}
\sigma & 0 \\
0 & 0
\end{mymatrix} U^T$ and it follows that
\begin{equation*}
A^TA=V\begin{mymatrix}{cc}
\sigma & 0 \\
0 & 0
\end{mymatrix} U^TU\begin{mymatrix}{cc}
\sigma & 0 \\
0 & 0
\end{mymatrix} V^T=V\begin{mymatrix}{cc}
\sigma ^{2} & 0 \\
0 & 0
\end{mymatrix} V^T
\end{equation*}
and so $A^TAV=V\begin{mymatrix}{cc}
\sigma ^{2} & 0 \\
0 & 0
\end{mymatrix}$. Similarly, $AA^TU=U\begin{mymatrix}{cc}
\sigma ^{2} & 0 \\
0 & 0
\end{mymatrix}$. Therefore, you would find an orthonormal basis of eigenvectors
for $AA^T$ make them the columns of a matrix such that the
corresponding eigenvalues are decreasing. This gives $U$. You could then do
the same for $A^TA$ to get $V$.

We formalize this discussion in the following theorem.

\begin{theorem}{Singular value decomposition}{singular-value-decomposition}
Let $A$ be an $m\times n$-matrix. Then there exist
orthogonal matrices $U$ and $V$ of the appropriate size such that $A= U \Sigma V^T$ where $\Sigma$ is of the form
\[
\Sigma =
\begin{mymatrix}{cc}
\sigma & 0 \\
0 & 0
\end{mymatrix}
\]
and $\sigma $ is of the form
\[
\sigma =\begin{mymatrix}{ccc}
\sigma _{1} &  & 0 \\
& \ddots &  \\
0 &  & \sigma _{k}
\end{mymatrix}
\]
for the $\sigma _{i}$ the singular values of $A$.
\end{theorem}

\begin{proof}
There exists an orthonormal basis, $\set{\vect{v}_{i}} _{i=1}^{n}$ such that $
A^TA\vect{v}_{i}=\sigma _{i}^{2}\vect{v}_{i}$ where $\sigma
_{i}^{2}>0$ for $i=1,\ldots,k,(\sigma _{i}>0) $ and equals zero
if $i>k$. Thus for $i>k$, $A\vect{v}_{i}=\vect{0}$ because
\begin{equation*}
 A\vect{v}_{i}\dotprod A\vect{v}_{i} = A^TA\vect{v}_{i} \dotprod \vect{v}_{i}  = \vect{0} \dotprod \vect{v}_{i} =0.
\end{equation*}
For $i=1,\ldots,k$, define $\vect{u}_{i}\in \R^{m}$ by
\begin{equation*}
\vect{u}_{i}= \sigma _{i}^{-1}A\vect{v}_{i}.
\end{equation*}

Thus $A\vect{v}_{i}=\sigma _{i}\vect{u}_{i}$. Now
\begin{eqnarray*}
\vect{u}_{i} \dotprod \vect{u}_{j} &=&  \sigma _{i}^{-1}A
\vect{v}_{i} \dotprod \sigma _{j}^{-1}A\vect{v}_{j}  = \sigma_{i}^{-1}\vect{v}_{i} \dotprod \sigma _{j}^{-1}A^TA\vect{v}_{j} \\
&=& \sigma _{i}^{-1}\vect{v}_{i} \dotprod \sigma _{j}^{-1}\sigma _{j}^{2} \vect{v}_{j} =
\frac{\sigma _{j}}{\sigma _{i}}(\vect{v}_{i} \dotprod \vect{v}_{j})
=\delta _{ij}.
\end{eqnarray*}
Thus $\set{\vect{u}_{i}} _{i=1}^{k}$ is an orthonormal set of
vectors in $\R^{m}$. Also,
\begin{equation*}
AA^T\vect{u}_{i}=AA^T\sigma _{i}^{-1}A\vect{v}_{i}=\sigma
_{i}^{-1}AA^TA\vect{v}_{i}=\sigma _{i}^{-1}A\sigma _{i}^{2}\vect{v}
_{i}=\sigma _{i}^{2}\vect{u}_{i}.
\end{equation*}
Now extend $\set{\vect{u}_{i}} _{i=1}^{k}$ to an orthonormal
basis for all of $\R^{m}$, $\set{\vect{u}_{i}} _{i=1}^{m}$
and let
\begin{equation*}
U= \begin{mymatrix}{ccc}
\vect{u}_{1} & \cdots & \vect{u}_{m}
\end{mymatrix}
\end{equation*}
while $V= \mat{\vect{v}_{1},\ldots,\vect{v}_{n}}$. Thus $U$
is the matrix which has the $\vect{u}_{i}$ as columns and $V$ is defined
as the matrix which has the $\vect{v}_{i}$ as columns. Then
\begin{equation*}
U^TAV=\begin{mymatrix}{c}
\vect{u}_{1}^T \\
\vdots \\
\vect{u}_{k}^T \\
\vdots \\
\vect{u}_{m}^T
\end{mymatrix} A\mat{\vect{v}_{1},\ldots,\vect{v}_{n}}
\end{equation*}
\begin{equation*}
=\begin{mymatrix}{c}
\vect{u}_{1}^T \\
\vdots \\
\vect{u}_{k}^T \\
\vdots \\
\vect{u}_{m}^T
\end{mymatrix} \begin{mymatrix}{cccccc}
\sigma _{1}\vect{u}_{1} & \cdots & \sigma _{k}\vect{u}_{k} & \vect{0}
& \cdots & \vect{0}
\end{mymatrix} =\begin{mymatrix}{cc}
\sigma & 0 \\
0 & 0
\end{mymatrix}
\end{equation*}
where $\sigma $ is given in the statement of the theorem.
\end{proof}

The singular value decomposition has as an immediate corollary which is given in the following interesting result.

\begin{corollary}{Rank and singular values}{rank-singular-values}
Let $A$ be an $m\times n$-matrix. Then the rank of $A$ and $A^T$equals
the number of singular values.
\end{corollary}

%%\begin{proof}
%%Since $V$ and $U$ are unitary, it follows that
%%\begin{eqnarray*}
%%\rank(A) &=&\rank(U^TAV) =
%%\rank\mat{
%%\begin{array}{cc}
%%\sigma & 0 \\
%%0 & 0
%%\end{array}
%%} \\
%%&=&\text{number of singular values.}
%%\end{eqnarray*}
%%Also since $U,V$ are unitary,
%%\begin{equation*}
%%\rank(A^T) =\rank(V^TA^{\ast
%%}U) =\rank((U^TAV)^T)
%%\end{equation*}
%%\begin{equation*}
%%=\rank\paren{\mat{
%%\begin{array}{cc}
%%\sigma & 0 \\
%%0 & 0
%%\end{array}
%%} ^T} =\text{number of singular values.}
%%\end{equation*}
%%\end{proof}
%%
%%\medskip

Let's compute the Singular Value Decomposition of a simple matrix.

\begin{example}{Singular value decomposition}{SVD}
Let
$A=\begin{mymatrix}{rrr} 1 & -1 & 3 \\ 3 & 1 & 1 \end{mymatrix}$.
Find the Singular Value Decomposition (SVD) of $A$.
\end{example}

\begin{solution}
To begin, we compute $AA^T$ and $A^TA$.
\[ AA^T = \begin{mymatrix}{rrr} 1 & -1 & 3 \\ 3 & 1 & 1 \end{mymatrix}
\begin{mymatrix}{rr} 1 & 3 \\ -1 & 1 \\ 3 & 1  \end{mymatrix}
= \begin{mymatrix}{rr} 11 & 5 \\ 5 & 11  \end{mymatrix}.\]

\[ A^TA = \begin{mymatrix}{rr} 1 & 3 \\ -1 & 1 \\ 3 & 1  \end{mymatrix}
\begin{mymatrix}{rrr} 1 & -1 & 3 \\ 3 & 1 & 1 \end{mymatrix}
= \begin{mymatrix}{rrr} 10 & 2 & 6 \\ 2 & 2 & -2\\
6 & -2 & 10 \end{mymatrix}.\]

Since $AA^T$ is $2\times 2$ while $A^T A$ is $3\times 3$, and $AA^T$
and $A^TA$ have the same {\em non-zero} eigenvalues (by Proposition~\ref{prop:same-non-zero-eigenvalues}), we compute the characteristic polynomial  $c_{AA^T}(x)$ (because it's
easier to compute than $c_{A^TA}(x)$).

\begin{eqnarray*}
c_{AA^T}(x)& = &\det(xI-AA^T)= \begin{absmatrix}{cc}
x-11 & -5 \\ -5 & x-11 \end{absmatrix}\\
& = &(x-11)^2 - 25 \\
& = & x^2-22x+121-25\\
& = & x^2-22x+96\\
& = & (x-16)(x-6)
\end{eqnarray*}

Therefore, the eigenvalues of $AA^T$ are $\lambda_1=16$ and $\lambda_2=6$.

The eigenvalues of $A^TA$ are $\lambda_1=16$, $\lambda_2=6$, and
$\lambda_3=0$, and the singular values of $A$ are $\sigma_1=\sqrt{16}=4$ and
$\sigma_2=\sqrt{6}$.
By convention, we list the eigenvalues (and corresponding singular values)
in non-increasing order (i.e., from largest to smallest).

{\bf To find the matrix $V$}:

To construct the matrix $V$ we need to find eigenvectors for $A^TA$.
Since the eigenvalues of $AA^T$ are distinct, the corresponding
eigenvectors are orthogonal, and we need only normalize them.

$\lambda_1=16$: solve $(16I-A^TA)Y= 0$.

\[ \begin{mymatrix}{rrr|r}
6 & -2 & -6 & 0 \\ -2 & 14 & 2 & 0 \\ -6 & 2 & 6 & 0
\end{mymatrix}
\rightarrow
\begin{mymatrix}{rrr|r}
1 & 0 & -1 & 0 \\ 0 & 1 & 0 & 0 \\ 0 & 0 & 0 & 0
\end{mymatrix},
\mbox{ so }
Y=\begin{mymatrix}{r} t \\ 0 \\ t \end{mymatrix}
=t\begin{mymatrix}{r} 1 \\ 0 \\ 1 \end{mymatrix},
t\in \R. \]

$\lambda_2=6$: solve $(6I-A^TA)Y= 0$.

\[ \begin{mymatrix}{rrr|r}
-4 & -2 & -6 & 0 \\ -2 & 4 & 2 & 0 \\ -6 & 2 & -4 & 0
\end{mymatrix}
\rightarrow
\begin{mymatrix}{rrr|r}
1 & 0 & 1 & 0 \\ 0 & 1 & 1 & 0 \\ 0 & 0 & 0 & 0
\end{mymatrix},
\mbox{ so }
Y=\begin{mymatrix}{r} -s \\ -s \\ s \end{mymatrix}
=s\begin{mymatrix}{r} -1 \\ -1 \\ 1 \end{mymatrix},
s\in \R. \]

$\lambda_3=0$: solve $(-A^TA)Y= 0$.
\[ \begin{mymatrix}{rrr|r}
-10 & -2 & -6 & 0 \\ -2 & -2 & 2 & 0 \\ -6 & 2 & -10 & 0
\end{mymatrix}
\rightarrow
\begin{mymatrix}{rrr|r}
1 & 0 & 1 & 0 \\ 0 & 1 & -2 & 0 \\ 0 & 0 & 0 & 0
\end{mymatrix},
\mbox{ so }
Y=\begin{mymatrix}{r} -r \\ 2r \\ r \end{mymatrix}
=r\begin{mymatrix}{r} -1 \\ 2 \\ 1 \end{mymatrix},
r\in \R. \]

Let
\[ V_1=\frac{1}{\sqrt{2}}\begin{mymatrix}{r} 1\\ 0\\ 1 \end{mymatrix},
V_2=\frac{1}{\sqrt{3}}\begin{mymatrix}{r} -1\\ -1\\ 1 \end{mymatrix},
V_3=\frac{1}{\sqrt{6}}\begin{mymatrix}{r} -1\\ 2\\ 1 \end{mymatrix}.\]

Then
\[ V=\frac{1}{\sqrt{6}}\begin{mymatrix}{rrr}
\sqrt 3 & -\sqrt 2 & -1  \\
0 & -\sqrt 2 & 2 \\
\sqrt 3 & \sqrt 2 & 1 \end{mymatrix}.\]

Also,
\[ \Sigma = \begin{mymatrix}{rrr} 4 & 0 & 0 \\
0 & \sqrt 6 & 0 \end{mymatrix},\]
and we use $A$, $V^T$, and $\Sigma$ to find $U$.

Since $V$ is orthogonal and $A=U\Sigma V^T$, it follows that $AV=U\Sigma$.
Let $V=\begin{mymatrix}{ccc} V_1 & V_2 & V_3 \end{mymatrix}$, and
let $U=\begin{mymatrix}{cc} U_1 & U_2 \end{mymatrix}$, where
$U_1$ and $U_2$ are the two columns of $U$.

Then we have
\begin{eqnarray*}
A\begin{mymatrix}{ccc} V_1 & V_2 & V_3 \end{mymatrix}
&=& \begin{mymatrix}{cc} U_1 & U_2 \end{mymatrix}\Sigma\\
\begin{mymatrix}{ccc} AV_1 & AV_2 & AV_3 \end{mymatrix}
&=& \begin{mymatrix}{ccc} \sigma_1U_1 + 0U_2 &
0U_1 + \sigma_2 U_2 & 0 U_1 + 0 U_2 \end{mymatrix} \\
&=& \begin{mymatrix}{ccc} \sigma_1U_1 & \sigma_2 U_2 &
0 \end{mymatrix}
\end{eqnarray*}
which implies that $AV_1=\sigma_1U_1 = 4U_1$ and
$AV_2=\sigma_2U_2 = \sqrt 6 U_2$.

Thus,
\[ U_1 = \frac{1}{4}AV_1
= \frac{1}{4}
\begin{mymatrix}{rrr} 1 & -1 & 3 \\ 3 & 1 & 1 \end{mymatrix}
\frac{1}{\sqrt{2}}\begin{mymatrix}{r} 1\\ 0\\ 1 \end{mymatrix}
= \frac{1}{4\sqrt 2}\begin{mymatrix}{r} 4\\ 4 \end{mymatrix}
= \frac{1}{\sqrt 2}\begin{mymatrix}{r} 1\\ 1 \end{mymatrix},\]
and
\[ U_2 = \frac{1}{\sqrt 6}AV_2
= \frac{1}{\sqrt 6}
\begin{mymatrix}{rrr} 1 & -1 & 3 \\ 3 & 1 & 1 \end{mymatrix}
\frac{1}{\sqrt{3}}\begin{mymatrix}{r} -1\\ -1\\ 1 \end{mymatrix}
=\frac{1}{3\sqrt 2}\begin{mymatrix}{r} 3\\ -3 \end{mymatrix}
=\frac{1}{\sqrt 2}\begin{mymatrix}{r} 1\\ -1 \end{mymatrix}.
\]
Therefore,
\[ U=\frac{1}{\sqrt{2}}\begin{mymatrix}{rr} 1 & 1 \\
1 & -1 \end{mymatrix},\]
and
\begin{eqnarray*}
A & = & \begin{mymatrix}{rrr} 1 & -1 & 3 \\ 3 & 1 & 1 \end{mymatrix}\\
& = & \paren{\frac{1}{\sqrt{2}}\begin{mymatrix}{rr} 1 & 1 \\
1 & -1 \end{mymatrix}}
\begin{mymatrix}{rrr} 4 & 0 & 0 \\
0 & \sqrt 6 & 0 \end{mymatrix}
\paren{\frac{1}{\sqrt{6}}\begin{mymatrix}{rrr}
\sqrt 3 & 0 & \sqrt 3  \\
-\sqrt 2 & -\sqrt 2 & \sqrt2 \\
-1 & 2 & 1 \end{mymatrix}}.
\end{eqnarray*}
\end{solution}

Here is another example.

\begin{example}{Finding the SVD}{SVD2}
Find an SVD for
$A=\begin{mymatrix}{r} -1 \\ 2\\ 2 \end{mymatrix}$.
\end{example}

\begin{solution}
Since $A$ is $3\times 1$, $A^T A$ is a $1\times 1$-matrix
whose eigenvalues are easier to find than the eigenvalues of
the $3\times 3$-matrix $AA^T$.

\[ A^TA=\begin{mymatrix}{ccc} -1 & 2 & 2 \end{mymatrix}
\begin{mymatrix}{r} -1 \\ 2 \\ 2 \end{mymatrix}
=\begin{mymatrix}{r} 9 \end{mymatrix}.\]

Thus $A^TA$ has eigenvalue $\lambda_1=9$, and
the eigenvalues of $AA^T$ are $\lambda_1=9$, $\lambda_2=0$, and
$\lambda_3=0$.
Furthermore, $A$ has only one singular value, $\sigma_1=3$.

{\bf To find the matrix $V$}:
To do so we find an eigenvector for $A^TA$ and
normalize it.
In this case, finding a unit eigenvector is trivial:
$V_1=\begin{mymatrix}{r} 1 \end{mymatrix}$, and
\[ V=\begin{mymatrix}{r} 1 \end{mymatrix}.\]

Also,
$\Sigma =\begin{mymatrix}{r} 3 \\ 0\\ 0 \end{mymatrix}$,
and we use $A$, $V^T$, and $\Sigma$ to find $U$.

Now $AV=U\Sigma$, with
$V=\begin{mymatrix}{r} V_1 \end{mymatrix}$,
and $U=\begin{mymatrix}{rrr} U_1 & U_2 & U_3 \end{mymatrix}$,
where $U_1$, $U_2$, and $U_3$ are the columns of $U$.
Thus
\begin{eqnarray*}
A\begin{mymatrix}{r} V_1 \end{mymatrix}
&=& \begin{mymatrix}{rrr} U_1 & U_2 & U_3 \end{mymatrix}\Sigma\\
\begin{mymatrix}{r} AV_1 \end{mymatrix}
&=& \begin{mymatrix}{r} \sigma_1 U_1+0U_2+0U_3 \end{mymatrix}\\
&=& \begin{mymatrix}{r} \sigma_1 U_1 \end{mymatrix}
\end{eqnarray*}
This gives us $AV_1=\sigma_1 U_1= 3U_1$, so

\[ U_1 = \frac{1}{3}AV_1
= \frac{1}{3}
\begin{mymatrix}{r} -1 \\ 2 \\ 2 \end{mymatrix}
\begin{mymatrix}{r} 1 \end{mymatrix}
= \frac{1}{3}
\begin{mymatrix}{r} -1 \\ 2 \\ 2 \end{mymatrix}.\]

The vectors $U_2$ and $U_3$ are eigenvectors of $AA^T$ corresponding
to the eigenvalue $\lambda_2=\lambda_3=0$.
Instead of solving the system $(0I-AA^T)X= 0$ and then using the
Gram-Schmidt process on the resulting set of
two basic eigenvectors, the following approach may be used.

Find vectors $U_2$ and $U_3$ by first extending $\set{U_1}$ to a basis of
$\R^3$, then using the Gram-Schmidt algorithm to orthogonalize the basis,
and finally normalizing the vectors.

Starting with $\set{3U_1}$ instead of $\set{U_1}$ makes the
arithmetic a bit easier.
It is easy to verify that

\[ \set{\begin{mymatrix}{r} -1 \\ 2 \\ 2 \end{mymatrix},
\begin{mymatrix}{r} 1 \\ 0 \\ 0 \end{mymatrix},
\begin{mymatrix}{r} 0 \\ 1 \\ 0 \end{mymatrix}}\]
is a basis of $\R^3$.  Set

\[ E_1 = \begin{mymatrix}{r} -1 \\ 2 \\ 2 \end{mymatrix},
X_2 = \begin{mymatrix}{r} 1 \\ 0 \\ 0 \end{mymatrix},
X_3 =\begin{mymatrix}{r} 0 \\ 1 \\ 0 \end{mymatrix},\]

and apply the Gram-Schmidt algorithm to
$\set{E_1, X_2, X_3}$.

This gives us

\[ E_2 = \begin{mymatrix}{r} 4 \\ 1 \\ 1 \end{mymatrix}
\mbox{ and }
E_3 = \begin{mymatrix}{r} 0 \\ 1 \\ -1 \end{mymatrix}.\]

Therefore,
\[ U_2 = \frac{1}{\sqrt{18}}
 \begin{mymatrix}{r} 4 \\ 1 \\ 1 \end{mymatrix},
U_3 = \frac{1}{\sqrt 2}
\begin{mymatrix}{r} 0 \\ 1 \\ -1 \end{mymatrix},\]
and

\[ U = \begin{mymatrix}{rrr} -\frac{1}{3} & \frac{4}{\sqrt{18}} & 0 \\
\frac{2}{3} & \frac{1}{\sqrt{18}} & \frac{1}{\sqrt 2} \\
\frac{2}{3} & \frac{1}{\sqrt{18}} & -\frac{1}{\sqrt 2} \end{mymatrix}.\]

Finally,

\[ A =
\begin{mymatrix}{r} -1 \\ 2 \\ 2 \end{mymatrix}
=
\begin{mymatrix}{rrr} -\frac{1}{3} & \frac{4}{\sqrt{18}} & 0 \\
\frac{2}{3} & \frac{1}{\sqrt{18}} & \frac{1}{\sqrt 2} \\
\frac{2}{3} & \frac{1}{\sqrt{18}} & -\frac{1}{\sqrt 2} \end{mymatrix}
\begin{mymatrix}{r} 3 \\ 0 \\ 0 \end{mymatrix}
\begin{mymatrix}{r} 1 \end{mymatrix}.\]

\end{solution}

Consider another example.

\begin{example}{Find the SVD}{SVD3}
Find a singular value decomposition for the matrix
\begin{equation*}
A= \begin{mymatrix}{ccc}
\frac{2}{5}\sqrt{2}\sqrt{5} & \frac{4}{5}\sqrt{2}\sqrt{5} & 0 \\
\frac{2}{5}\sqrt{2}\sqrt{5} & \frac{4}{5}\sqrt{2}\sqrt{5} & 0
\end{mymatrix}
\end{equation*}
\end{example}

First consider $A^TA$
\begin{equation*}
\begin{mymatrix}{ccc}
\frac{16}{5} & \frac{32}{5} & 0 \\
\frac{32}{5} & \frac{64}{5} & 0 \\
0 & 0 & 0
\end{mymatrix}
\end{equation*}
What are some eigenvalues and eigenvectors? Some computing shows these are

\begin{equation*}
\set{\begin{mymatrix}{c}
0 \\
0 \\
1
\end{mymatrix} ,\begin{mymatrix}{c}
-\frac{2}{5}\sqrt{5} \\
\frac{1}{5}\sqrt{5} \\
0
\end{mymatrix} } \leftrightarrow 0,\set{\begin{mymatrix}{c}
\frac{1}{5}\sqrt{5} \\
\frac{2}{5}\sqrt{5} \\
0
\end{mymatrix} } \leftrightarrow 16
\end{equation*}
Thus the matrix $V$ is given by
\begin{equation*}
V=\begin{mymatrix}{ccc}
\frac{1}{5}\sqrt{5} & -\frac{2}{5}\sqrt{5} & 0 \\
\frac{2}{5}\sqrt{5} & \frac{1}{5}\sqrt{5} & 0 \\
0 & 0 & 1
\end{mymatrix}
\end{equation*}
Next consider $AA^T$
\begin{equation*}
\begin{mymatrix}{cc}
8 & 8 \\
8 & 8
\end{mymatrix}
\end{equation*}
Eigenvectors and eigenvalues are

\begin{equation*}
\set{\begin{mymatrix}{c}
-\frac{1}{2}\sqrt{2} \\
\frac{1}{2}\sqrt{2}
\end{mymatrix} } \leftrightarrow 0,\set{\begin{mymatrix}{c}
\frac{1}{2}\sqrt{2} \\
\frac{1}{2}\sqrt{2}
\end{mymatrix} } \leftrightarrow 16
\end{equation*}
Thus you can let $U$ be given by
\begin{equation*}
U=\begin{mymatrix}{cc}
\frac{1}{2}\sqrt{2} & -\frac{1}{2}\sqrt{2} \\
\frac{1}{2}\sqrt{2} & \frac{1}{2}\sqrt{2}
\end{mymatrix}
\end{equation*}
Let us check this. $U^TAV=$
\begin{equation*}
\begin{mymatrix}{cc}
\frac{1}{2}\sqrt{2} & \frac{1}{2}\sqrt{2} \\
-\frac{1}{2}\sqrt{2} & \frac{1}{2}\sqrt{2}
\end{mymatrix} \begin{mymatrix}{ccc}
\frac{2}{5}\sqrt{2}\sqrt{5} & \frac{4}{5}\sqrt{2}\sqrt{5} & 0 \\
\frac{2}{5}\sqrt{2}\sqrt{5} & \frac{4}{5}\sqrt{2}\sqrt{5} & 0
\end{mymatrix} \begin{mymatrix}{ccc}
\frac{1}{5}\sqrt{5} & -\frac{2}{5}\sqrt{5} & 0 \\
\frac{2}{5}\sqrt{5} & \frac{1}{5}\sqrt{5} & 0 \\
0 & 0 & 1
\end{mymatrix}
\end{equation*}
\begin{equation*}
=\begin{mymatrix}{ccc}
4 & 0 & 0 \\
0 & 0 & 0
\end{mymatrix}
\end{equation*}

This illustrates that if you have a good way to find the eigenvectors and
eigenvalues for a Hermitian matrix which has non-negative eigenvalues, then
you also have a good way to find the singular value decomposition of an
arbitrary matrix.


\section{}
The network presented in \ref{sec:sgd} uses a sigmoid function as the activation function
of neurons. Using the same structure and initial weights given in \ref{sec:weights} we replace
the sigmoid function with a bipolar sigmoid function given in equation \ref{eq:bipolar_sigmoid}

\begin{equation}
\label{eq:bipolar_sigmoid}
 f(x)= \frac{2}{1 + e^{-x}} - 1
\end{equation}

\begin{equation}
\label{eq:bipolar_sigmoid_derivative}
 f'(x)= \frac{1}{2} (1 - f(x)^2)
\end{equation}

The input and output to the network are too converted into their bipolar representation. The
plot of training cost vs iterations is shown in figure \ref{fig:cost_bipolar}. Comparing
figures \ref{fig:cost_sgd} and \ref{fig:cost_bipolar} we can see that the network using bipolar
sigmoid with bipolar XOR representation converged in fewer number of iterations then original
network.

\section{}
In order to evaluate the impact of Nguyen-Widrow approach of weight initialization on convergence
a neural network similar to the one described in \ref{sec:sgd} was constructed. Then the
sigmoid function was replaced with hyperbolic tangent function and bipolar representation of XOR
gate was used as the training data. The training cost with respect to the iterations is given in figure
\ref{fig:cost_tanh}. \newline

Above experiment was repeated after initializing the weights and biases as per the Nguyen-Widrow
approach. The plot of the training cost with respect iterations is shown in figure \ref{fig:cost_tanh_nw}.
Comparing figure \ref{fig:cost_tanh_nw} with figure \ref{fig:cost_tanh} it can seen that using Nguyen-Widrow approach
to initialize the weights smoothens the decrease in cost per iteration. Figure \ref{fig:tanh_nw_first_2500_compare} shows
the training cost for the first 2500 iteration of both cases.

\section{}
Initially a neural network with 3 units in first hidden layer and 2 units in second hidden layer was constructed.
The weights of the network is randomly initialized in the range $[-0.5, 0.5]$. As for the activation function,
hyperbolic tangent function was used and SGD with momentum coefficient($\mu$)$=0.5$ was used to optimize the parameters.
The plot for training cost with respect to iteration is given in figure \ref{fig:cost_two_hidden_layer}


\chapter{Short-rate models}
\label{chap:shortrate}

\added{This chapter introduces basics of short-rate modeling and short-rate models iwth affine term-structures.}

\section{Introduction to short-rate models}

\subsection{Term-structure equation}

\added{The derivation of the term-structure equation follows closely} \textcite[pp. 319--324]{bjork2004arbitrage}.

In this chapter we assume that the short rate process follows
\begin{align}
\label{short-ratedynamicsundersomemeasure}
\dx r(t) = \mu(t,r(t)) \dx t + \sigma(t,r(t)) \dx W(t) ,
\end{align}
where $W(t)$ is a Brownian motion under some measure $\Pm^*$ (this may be also physical measure) and $\mu$ and $\sigma$ are well-behaved functions. If $V(t,r(t))$ is a smooth function, then It\'{o}s lemma says that
\begin{align}
\dx V = \left( \frac{\partial V}{\partial t} + \mu \frac{\partial V}{\partial x} +\frac{1}{2} \sigma^2 \frac{\partial^2 V}{\partial x^2} \right) \dx t + \sigma \frac{\partial V}{\partial x} \dx W
\end{align}
Here we have dropped arguments for functions. Unlike in Black-Scholes model, we may not trade short rate $r(t)$ directly and it may not be used in hedging. But we may consider how to hedge a derivative using another contingent claim.

We consider two contingent claims with value processes $V_1(t, r(t))$ and $V_2(t,r(t))$. It\'{o}'s lemma gives us that
\begin{align}
\dx V_i = M_i \dx t + N_i \dx W,
\end{align}
where
\begin{align}
M_i &= \frac{\partial V_i}{\partial t} + \mu \frac{\partial V_i}{\partial r} + \frac{1}{2} \sigma^2 \frac{\partial^2 V_i}{\partial r^2}, \\
N_i &= \sigma \frac{\partial V_i}{\partial r} .
\end{align}
If $\Pi = V_1 + \delta V_2$ is a portfolio, then
\begin{align}
\dx \Pi = \left( M_1 + \delta M_2 \right) \dx t + \left( N_1 + \delta N_2 \right) \dx W .
\end{align}
By choosing $\delta = - \frac{N_1}{N_2}$, the Brownian motion disappears. In order to maintain absence of arbitrage, we have that
\begin{align}
\dx \Pi &= r \Pi \dx t \\
\left( M_1 - \frac{N_1}{N_2} M_2 \right) \dx t &= r \left( V_1 - \frac{N_1}{N_2} V_2 \right) \dx t .
\end{align}
Thus
\begin{align}
\frac{M_1 - r V_1}{N_1 } = \frac{M_2 - r V_2}{N_2}
\end{align}
Now $M_i, N_i$ and $V_i$ are functions of $t$ and $r(t)$. The left-hand side is independent of the portfolio $V_2$ and the right-hand size is independent of portfolio $V_1$. Hence there exist a function
\begin{align}
\lambda(t,r(t)) = \frac{M(t,r(t)) - r(t) V(t,r(t))}{N(t,r(t))}
\end{align}
called the market-price of risk, where $V$ is any interest-rate derivative with dynamics
\begin{align}
\dx V = M \dx t + N \dx W .
\end{align}
By combining this with earlier results, we get that the price process $V$ must satisfy partial differential equation
\begin{align}
\label{termstructureequation}
0 &= M - rV - \lambda N \\
&= \frac{\partial V}{\partial t} + \mu \frac{\partial V}{\partial r} + \frac{1}{2} \sigma^2 \frac{\partial^2 V}{\partial r^2} - rV - \lambda \sigma \frac{\partial V}{\partial r} \\
&= \frac{\partial V}{\partial t} + \left( \mu - \lambda \sigma \right) \frac{\partial V}{\partial r} + \frac{1}{2} \sigma^2 \frac{\partial^2 V}{\partial r^2} - rV
\end{align}
with the boundary condition given by the value $V(T, r(T))$.

\subsection{Fundamental models}

Suppose that $\Pm^* = \Pf$ is the physical measure. Models that are defined under the physical measure, are often called as fundamental models. If we denote
	\begin{align}
		\theta(t,r(t)) = \mu(t,r(t)) - \lambda(t,r(t)) \sigma(t,r(t))
	\end{align}
and assume that
	\begin{align}
		\lambda(t,r(t)) = \frac{\theta(t,r(t)) - \mu(t,r(t))}{\sigma(t,r(t))}
	\end{align}
may be used as a Girsanov kernel, then we get a new measure $\Pm_{\theta}$ under which
	\begin{align}
		r(t) = \theta(t,r(t)) \dx t + \sigma(t,r(t)) \dx W_{\theta} (t)
	\end{align}
and $W_{\theta} (t)$ is a Brownian motion. Now Feynman-Kac theorem implies that
	\begin{align}
		V(t) = \E_{\Pm_{\theta}} \left( \e^{-\int_t^T r(s) \dx s} V(T) \ | \ \F_t \right) .
	\end{align}
Unlike in the Black-Scholes model, the market price of risk $\lambda$ is not uniquely determined endogenously within the model. Here, the market price of risk is endogenously determined. The equivalent martingale measure (if it exists), it is not unique and prices depend on the choice of the function $\lambda$. However, the $\lambda$ is uniquely determined by the price of any interest rate derivative. Since the dynamic of Equation \ref{short-ratedynamicsundersomemeasure} are under the physical measure, we could use econometric time-series to estimate model parameters and market price of risk. This approach is problematic since neither the short-rate nor the market price of risk are not directly observable. According to \cite{chapmanlongpearson1999usingproxies}, using proxies to estimate parameters of short-rate models for single-factor affine models does not cause economically significant problems, but for more complex models, proxies cause significant errors. Also, for estimation, we usually have to assume a functional form for the market price of risk, which may be misspecified. Fundamental models have no guarantee that they will fit the observed term or volatility structures. If these obstacles are overcome, then the model may be used to price any instrument and forecast interest rates.

\subsection{Preference-free models}

Another approach is to assume that the Equation \ref{short-ratedynamicsundersomemeasure} holds under the risk-free measure $\Pm_0$. This means choosing $\lambda = 0$ in the Equation \ref{termstructureequation}, which will now read
\begin{align}
\label{termstructureequation2}
0 &= \frac{\partial V}{\partial t} + \mu \frac{\partial V}{\partial r} + \frac{1}{2} \sigma^2 \frac{\partial^2 V}{\partial r^2} - rV .
\end{align}
Therefore Feynman-Kac theorem implies that
	\begin{align}
		V(t) = \E_{\Pm_0} \left( \e^{-\int_t^T r(s) \dx s} V(T) \ | \ \F_t \right) .
	\end{align}
for all assets. Thus all discounted asset price process, where the num\'{e}raire is the bank account, are martingales under $\Pm_0$.

Under this methodology, we may use calibrate the model parameters using observed market prices. We may take liquid instruments and then use their prices to calibrate the model. We may not use historical data in calibration, since the assumed process is under the risk-free measure. The physical measure will be different from the risk-free measure, unless we explicitly make the strong assumption that the market price of risk will be zero. However, as the volatility term does not change under measure changes, diffusion term may be estimated with data that is collected under the risk-free measure. Models under this methodology may or may not be guaranteed to fit the observed term and volatility structures. They could be useful for pricing. They may not be used to forecast prices or interest rates without further assumptions.

\subsubsection{Preference-free +-models}

Preference-free models which have constant parameters $k, \theta, \sigma$ are notated with single plus sign. They do not probably fit the observed term and volatility structures. Most of the early fundamental models can be also interpreted as preference-free $+$-models.

\subsubsection{Preference-free ++-models}

Preference-free models which have constant parameters $k, \sigma$ but time-varying $\theta(t)$ are notated with double plus sign. They can be made to fit  the observed term structure but they probably do not match the volatility surface. \textcite{hull1990pricing} is often the prototypical example of a $++$-model.

When matched to term-structure and calibrated with cap or swaption prices, they can be useful in pricing of exotic interest-rate options.

\subsubsection{Preference-free +++-models}

Preference-free models which have time-varying $\theta(t), \sigma(t)$ (and sometimes $k$) are notated with triple plus sign. They can be made to fit both the observed term structure and some of the volatility surface. Variants of Hull-White models are sometimes $+++$-models.

Triple plus models can be useful in pricing but they could be prone to over-fitting.

\section{One-factor short-rate models}

One-factor short-rate model is model that has one underlying state variable that drives the evolution of the interest rates. This factor is often the short-rate itself. Often used form for one-factor short-rate process is
\begin{align}
\dx r (t) = k( \theta (t) - r (t) ) \dx t + \sigma (t) r (t)^{\gamma} \dx W (t)
\end{align}
The following table gives a quick overview of some of the models of this form.
\begin{center}
	\begin{tabular}{|l|l|l|l|l|}
		\hline
		Model & $k(t)$ & $\theta (t)$ & $\sigma (t)$ & $\gamma$ \\
		\hline
		\hline
		\cite{vasicek1977equilibrium} & $k$ & $\theta$ & $\sigma$ & $-$ \\
		\hline
		\cite{dothan1978term} & $-k(t)$ & $-$ & $\sigma$ & $-$ \\
		\hline
		\cite{coxingersollross1985theory} & $k$ & $\theta$ & $\sigma$ & $\frac{1}{2}$ \\
		\hline
		\cite{hull1990pricing} & $k$ & $\theta (t)$ & $\sigma (t)$ & $-$ \\
		\hline
	\end{tabular}
\end{center}

\subsection{Affine one-factor term-structures models}

\added{The derivation of the following theorem is based on } \textcite[pp. 329--331]{bjork2004arbitrage}.

A one-factor short-rate model has affine term-structure if the zero-coupon bond price is
\begin{align}
\label{affinitytermstructureequation}
\Bond(t,T) = \e^{A(t,T) - B(t,T)r(t)}
\end{align}
for all $0 \leq t \leq T$, where $A(t,T)$ and $B(t,T)$ deterministic and smooth functions. Since $\Bond(T,T) = 1$, we have that
\begin{align}
A(T,T) = B(T,T) = 0.
\end{align}
Under the assumption of affinity the instantaneous forward rate is
\begin{align}
f(t,T) &= - \frac{\partial \log \Bond(t,T)}{\partial T} \\
&= \frac{\partial (B(t,T)r(t) - A(t,T))}{\partial T} \\
&= \frac{\partial B(t,T)}{\partial T}r(t) - \frac{\partial A(t,T)}{\partial T}
\end{align}
for all $0 \leq t \leq T$.

If we denote that $p(t,T) = F(t,T,r)$ to make clear that $\Bond(t,T)$ is also a function of $r$, then we have that
\begin{align}
\frac{\partial F}{\partial t} &= \left( \frac{\partial A}{\partial t} - \frac{\partial B}{\partial t} r \right) F \\
\frac{\partial F}{\partial r} &= - B F \\
\frac{\partial^2 F}{\partial r^2} &= B^2 F
\end{align}
as the derivative of $r$ with respect to $t$ vanishes. Since the price of a zero-coupon price must satisfy \ref{fundamentaltermstructure_equationinaffine}, we have that
\begin{align}
\frac{\partial F}{\partial t} + \mu^* \frac{\partial F}{\partial r} + \frac{1}{2} \sigma^2 \frac{\partial^2 F}{\partial r^2} - r F &= 0 \\
F(T,T,r) = 1.
\end{align}
By combining these, we have that
\begin{align}
0 &= \left( \frac{\partial A}{\partial t} - \frac{\partial B}{\partial t} r \right) F - \mu^* B F + \frac{1}{2} \sigma^2 B^2 F - r F, \\
0 &= \frac{\partial A}{\partial t} - \frac{\partial B}{\partial t} r - \mu^* B + \frac{1}{2} \sigma^2 B^2 - r \\
&= \frac{\partial A}{\partial t} - ( 1 + \frac{\partial B}{\partial t} ) r - \mu^* B + \frac{1}{2} \sigma^2 B^2
\end{align}

If we suppose that$\mu^*(t)$ and $\sigma^2(t)$ are affine in short-rate $r(t)$, meaning that
\begin{align}
\mu^*(t) &= a(t)r(t) + b(t) \\
\sigma^2(t) &= c(t)r(t) + d(t),
\end{align}
where $a,b,c$ and $d$ are deterministic functions. Thus
\begin{align}
0 	&= \frac{\partial A}{\partial t} - ( 1 + \frac{\partial B}{\partial t} ) r - \mu^* B + \frac{1}{2} \sigma^2 B^2 \\
&= \frac{\partial A}{\partial t} - ( 1 + \frac{\partial B}{\partial t} ) r - \left( ar+b \right) B + \frac{1}{2} \left( cr+d \right) B^2 \\
&= \left( \frac{\partial A}{\partial t} - bB + \frac{1}{2} d B^2 \right) +
\left( \frac{1}{2} cB^2 - \frac{\partial B}{\partial t} -aB - 1 \right) r
\end{align}
and this must hold for all $t, T$ and $r(t)$. Thus the coefficients must equal zero and we have that
\begin{align}
\frac{\partial A}{\partial t} - bB + \frac{1}{2} d B^2 &= 0, \\
A(T,T) &= 0, \\
\frac{1}{2} cB^2 - \frac{\partial B}{\partial t} -aB - 1 &= 0, \\
B(T,T) &= 0 .
\end{align}

Thus we have proved the following.

\begin{thm}
	\label{affineconfition}
	If
	\begin{align}
	\mu^*(t) &= a(t) r + b(t) \\
	\sigma^2(t) &= c(t) r + d(t),
	\end{align}
	then the short-rate model has an affine term-structure model and the following equations holds:
	\begin{align}
	&\begin{cases}
	\frac{\partial B}{\partial t} = \frac{1}{2} c B^2  -a B - 1 \\
	B(T,T) = 0
	\end{cases} \\
	&\begin{cases}
	\frac{\partial A}{\partial t} - bB + \frac{1}{2} d B^2 &= 0 \\
	A(T,T) = 0 .
	\end{cases}
	\end{align}
\end{thm}

Equations of this type are Riccati Equations and they are easy to solved efficiently.

\subsection{Va\v{s}\'{i}\v{c}ek--model}

The material from this section is from \textcite[pp. 58--62]{brigo2007interest}.

\textcite{vasicek1977equilibrium} showed that under certain economic assumptions, the short-rate process is a Ornstein-Uhlenbeck process. This particular form was earlier suggest by \textcite{merton1971optimum}. Under the Va\v{s}\'{i}\v{c}ek--model, the short-rate process $r(t)$ is given by
\begin{align}
\dx r(t) = k ( \theta - r(t) ) \dx t + \sigma \dx W(t) ,
\end{align}
where $k, \theta, \sigma > 0$ and $r(0) = r_0$ are constants. Thus
	\begin{align}
		\mu(t,r(t)) &= k ( \theta - r(t) ) \\
		\sigma(r,t(t)) &= \sigma .
	\end{align}
Equivalent parametrization is
\begin{align}
\dx r(t) = ( \theta^* - k r(t) ) \dx t + \sigma \dx W(t) ,
\end{align}
where $\theta^* = \theta k$. We could also write
	\begin{align}
		r(t) &= \theta + Y(t) \\
		\dx Y(t) &= - k Y(t) \dx t + \sigma \dx W(t).
	\end{align}
Now $r(t) = g(t,Y(t))$, where $g(t,y) = m + y$ actually and It\^{o}'s lemma gives
	\begin{align}
		\dx r(t, Y(t)) &= - k Y(t) \dx t + \sigma \dx W(t) \\
			&= k ( \theta - r(t) ) \dx t + \sigma \dx W(t) .
	\end{align}

We consider a process $X(t) = \int_0^t \e^{k s} \dx W(s)$ so that $\dx X(t) = \e^{k t} \dx W(t)$. Now we define the function
\begin{align}
g(x,t) &= r_0 \e^{-k t} + \theta (1 - \e^{-k t}) + \sigma \e^{-k t} x \\
	&= \theta + \e^{-k t} \left( r_0 - \theta + \sigma x \right)
\end{align}
and
\begin{align}
\frac{ \partial g(x,t)}{ \partial x} &= \sigma \e^{-k t}, \\
\frac{ \partial^2 g(x,t)}{ \partial x^2} &= 0, \\
\frac{ \partial g(x,t)}{ \partial t} &= - k \e^{- k t} (r_0 - \theta +\sigma x ) \\
&= k (\theta -g(x,t) ).
\end{align}
Since the drift term for $X$ is zero and diffusion factor is $\e^{k t}$, It\^{o}'s lemma for the process $X(t)$ yields
\begin{align}
\dx g (X(t),t) &= \frac{ \partial g(X(t),t) }{ \partial t} \dx t + \e^{k t} \frac{ \partial g(X(t),t)}{ \partial x} \dx W \\
&= k (\theta -g(X(t),t) ) \dx t + \sigma W(t) .
\end{align}
As $X(0)=0$, we have that $g(X(0),0) = r_0$. Thus
\begin{align}
\label{vasiceksolution}
r(t) = g(X(t),t) = r_0 \e^{-k t} + \theta (1 - \e^{-k t}) + \sigma \e^{-k t} \int_0^t \e^{k s} \dx W(s) .
\end{align}
By Theorem \ref{gaussiancalculation}, the expected value of the integral in the equation \ref{vasiceksolution} is zero and it has variance of
\begin{align}
\int_0^t \e^{2 k s} \dx s = \frac{1}{2 k} (\e^{2 k t} -1).
\end{align}
Hence
\begin{align}
r(t) \sim N \left( r_0 \e^{-k t} + \theta (1 - \e^{-k t}), \frac{\sigma^2}{2k } (1-\e^{2k t}) \right) .
\end{align}
Since $r(t)$ is normally distributed in the Va\v{s}\'{i}\v{c}ek--model, there is a positive change that short-rate will be negative in a given time frame. If $t \rightarrow \infty$, then
\begin{align}
\E (r(t)) &\rightarrow \theta , \\
\Var ( r(t) ) &\rightarrow \frac{\sigma^2}{2k} .
\end{align}
We see that the parameter $\theta$ can be seen as the long-term mean and the short-rate has a tendency to move toward it. The parameter $k$ signifies the speed of this mean-reversion while $\sigma$ controls the volatility.

One of the features of the Va\v{s}\'{i}\v{c}ek--model is that there is a non-zero probability for negative rates. Earlier this was seen as a major drawback of the model.

\subsubsection{Bond pricing in the Va\v{s}\'{i}\v{c}ek--model}

If we assume that the short-rate process $r(t)$ is given by
\begin{align}
\dx r(t) = k ( \theta - r(t) ) \dx t + \sigma \dx W(t)^*
\end{align}
under the risk neutral measure. Now $\mu(t) = k \theta - k r(t)$ and $\sigma(t) = \sigma$ are affine in $r(t)$. By Theorem \ref{affineconfition}, Va\v{s}\'{i}\v{c}ek-model has affine term-structure and
\begin{align}
\label{vasicekaffine}
\begin{cases}
\frac{\partial B}{\partial t} = k B - 1 \\
\frac{\partial A}{\partial t} = k \theta B - \frac{1}{2} \sigma^2 B^2 \\
A(T,T) = B(T,T) = 0,
\end{cases}
\end{align}
Now
\begin{align}
B(t,T) = \frac{1}{k} \left( 1 - \e^{-k (T-t)} \right)
\end{align}
satisfies \ref{vasicekaffine} and therefore we might solve $A(t,T)$ by calculation the integral
\begin{align}
A(t,T) = A(t,T) - A(T,T) = - \int_t^T A(s,T) \dx s.
\end{align}
We note that $B^2 = \frac{B}{k} \left( \frac{\partial B}{\partial t} + 1 \right)$ and hence
\begin{align}
k \theta B - \frac{1}{2} \sigma^2 B^2 &= k \theta B - \frac{\sigma^2}{2k} B(1+\frac{\partial B}{\partial t}) \\
&= \frac{k^2 \theta - \frac{1}{2}\sigma^2}{k} B  - \frac{\sigma^2}{2k} \frac{\partial B}{\partial t} B \\
&= \frac{k^2 \theta - \frac{1}{2}\sigma^2}{k^2} (\frac{\partial B}{\partial t} + 1) - \frac{\sigma^2}{2k} B\frac{\partial B}{\partial t} .
\end{align}
Now the conditions in Equation \ref{vasicekaffine} will be satisfied by
\begin{align}
A(t,T) = \frac{k^2 \theta - \frac{1}{2}\sigma^2}{k^2} (B(t,T) - (T - t)) - \frac{\sigma^2}{4 k} B^2(t,T) .
\end{align}
Therefore Va\v{s}\'{i}\v{c}ek-model has term-structure defined by
\begin{align}
P(t,T) = \e^{A(t,T) - B(t,T)r(t)},
\end{align}
where
\begin{align}
B(t,T) &= \frac{1}{ k} \left( 1 - \e^{-k (T-t)} \right), \\
A(t,T) &= \frac{k^2 \theta - \frac{1}{2}\sigma^2}{k^2} (B(t,T) - (T - t)) - \frac{\sigma^2}{4k} B^2(t,T).
\end{align}

\subsubsection{Option pricing in the Va\v{s}\'{i}\v{c}ek--model}

Since the short-rate follows a Gaussian distribution, the price of a option on a zero-coupon bond can be calculated explicitly. We shall not do that. A European call option with maturity $S$ on a $T$-bond and exercise price $K$ has a price
	\begin{align}
		\ZBC(t, S, T, K) = \Bond(t,T) N(d_1) - K \Bond(t,S) N(d_2),
	\end{align}
at the time $t$, where
	\begin{align}
		d_1 &= \frac{ \log \frac{\Bond(t,T)}{K \Bond(t,S)} + \frac{V}{2} }{ \sqrt{V} } \\
		d_2 &= \frac{ \log \frac{\Bond(t,T)}{K \Bond(t,S)} - \frac{V}{2} }{ \sqrt{V} } \\
		V &= \sigma^2 \left( \frac{1 - \e^{-2k(T-S)}}{k} \right)^2 \frac{1 - \e^{-2k(S-t)}}{2k} .
	\end{align}
A European put option with maturity $S$ on a $T$-bond and exercise price $K$ has a price
\begin{align}
\ZBP(t, S, T, K) = K \Bond(t,S) N( - d_2) - \Bond(t,T) N(-d_1)
\end{align}
at the time $t$

\subsection{Cox-Ingersol-Ross--model (CIR)}
\label{sec:cir}

The material from this section is mainly from \textcite[pp. 64--68]{brigo2007interest}.

\textcite{coxingersollross1985theory} introduced the Cox-Ingersol-Ross--model (CIR) where the short-rate process $r_t$ is given by
\begin{align}
\dx r(t) = k( \theta - r(t) ) \dx t + \sigma \sqrt{r(t)} \dx W(t) ,
\end{align}
where $r_0, k, \theta$ and $\sigma$ are positive constants. Another widely used parametrization is
\begin{align}
\dx r(t) = ( \alpha - k r(t) ) \dx t + \sigma \sqrt{r(t)} \dx W(t) ,
\end{align}
where $\alpha = k \theta$. Like Va\v{s}\'{i}\v{c}ek--model, CIR features reversion toward the mean $\theta$ with $k$ as the strength of the reversion. But it also has non-constant volatility as the diffusion term is $\sigma \sqrt{r(t)}$. CIR model also has affine term-structure and therefore the bond prices can be efficiently solved. Unlike Va\v{s}\'{i}\v{c}ek--model, CIR--model can be specified so that the short-rate will be always positive.

\subsubsection{Bond pricing in the CIR--model}

By solving the Riccati equation, which we shall not do, we get that the bond price in CIR model is
	\begin{align}
		\Bond(t,T) = \e^{A(t,T) - B(t,T)r(t)} ,
	\end{align}
where
	\begin{align}
		A(t,T) &= \frac{2k\theta}{\sigma^2} \log \left( \frac{2 \beta \e^{ \frac{(\beta + k)(T-t)}{2} } }{(\beta + k)(\e^{\beta(T-t)} - 1) + 2\beta}  \right) \\
		B(t,T) &= \frac{ 2(\e^{\beta(T-t)} - 1) }{ (\beta + k)(\e^{\beta(T-t)} - 1) + 2\beta } \\
		\beta &= \sqrt{k^2+ 2\sigma^2}
	\end{align}

\subsubsection{Option pricing in the CIR--model}

A European call option with maturity $S$ on a $T$-bond and exercise price $K$ has a price
\begin{align}
\ZBC(t, S, T, K) = \Bond(t,T) \chi_1^2 - K \Bond(t,S) \chi_2^2,
\end{align}
at the time $t$, where
\begin{align}
	\chi_1^2 &= \chi^2 \left( v_1, \frac{4k\theta}{\sigma^2}, \frac{2\beta_3^2 r(t) \e^{\beta (S-t)} }{\beta_2 + \beta_3 + B(S,T)} \right) \\
	\chi_2^2 &= \chi^2 \left( v_2, \frac{4k\theta}{\sigma^2}, \frac{2\beta_3^2 r(t) \e^{\beta (S-t)} }{\beta_2 + \beta_3} \right) \\
	v_1 &= 2( \beta_2 + \beta_3 + B(S,T) ) \frac{A(S,T)-\log(K)}{B(S,T)} \\
	v_2 &= 2( \beta_2 + \beta_3 ) \frac{A(S,T)-\log(K)}{B(S,T)} \\
	\beta_2 &= \frac{k+\beta}{\sigma^2}\\
	\beta_3 &= \frac{2\beta}{\sigma^2 (\e^{\beta(S-t)} - 1) }
\end{align}
and $\chi^2(v,a,b)$ is the cumulative non-central chi-squared distribution with $A$ degrees of freedom and non-centrality parameter $b$.

\iffalse

\subsubsection{Glasserman algorithm}

Glasserman gives the following algorithm to generate a sampling of a path in a CIR-model on time grid $0 = t_0 < t_1 < \ldots < t_n$. We denote $d = \frac{k \theta}{\sigma^2}$.

\begin{description}
	\item[Case $d>1$]: For $i=0,1, \ldots, n$:
	\begin{enumerate}
		\item Set
		\begin{align}
		c &= \sigma^2 \frac{ 1 - \exp^{ - k ( t_{i+1} - t_i ) } }{4 k}, \\
		\lambda &= r(t_i) \frac{ \exp^{ - k ( t_{i+1} - t_i ) } }{ c }.
		\end{align}
		\item Generate
		\begin{align}
		Z &\sim N(0,1), \\
		X &\sim \chi_{d-1}^2.
		\end{align}
		\item Set
		\begin{align}
		r(t_{i+1}) = \left( (Z + \sqrt{\lambda})^2 + X \right) .
		\end{align}
	\end{enumerate}
	\item[Case $d \leq 1$]: For $i=0,1, \ldots, n$:
	\begin{enumerate}
		\item Set
		\begin{align}
		c &= \sigma^2 \frac{ 1 - \exp^{ - k ( t_{i+1} - t_i ) } }{4 k}, \\
		\lambda &= r(t_i) \frac{ \exp^{ - k ( t_{i+1} - t_i ) } }{ c }.
		\end{align}
		\item Generate
		\begin{align}
		Z &\sim \text{Poisson} ( \frac{\lambda}{2} ), \\
		X &\sim \chi_{d+2Z}^2.
		\end{align}
		\item Set
		\begin{align}
		r(t_{i+1}) = cX .
		\end{align}
	\end{enumerate}
\end{description}

\fi

\section{Multi-factor short-rate models}

Multi-factor short-rate models have more than one state variables that drive the evolution of the short-rate. \textcite{litterman1991common} demonstrated that while the majority of the yield curve movements can be explained by a single factor, it can not explain it all. Usually it is considered that at least 3 factors are needed.

One problem with one-factor affine one-factor term-structures models is that rates of different maturities are perfectly correlated. By Equation \ref{affinitytermstructureequation},
\begin{align}
\Bond(t,T) = \e^{A(t,T) - B(t,T)r(t)}
\end{align}
and the continuously compounded rate satisfies $\e^{R(t,T)(T-t)} \Bond(t,T) = 1$, we see that
\begin{align}
	R(t,T) = \frac{B(t,T)}{T-t} r(t) - \frac{A(t,T)}{T-t} .
\end{align}
This implies that rates are perfectly correlated. Thus multiple factors are needed to induce realistic correlations among the rates of different maturities.

In this section we shall follow the presentation based on \textcite[pp. 425--435]{nawalkabeliaevasoto2007dynamic}.

\subsection{Simple $A(M,N)$--models}
\label{subsec-AMN-interestrate}

We now define a class of models $A(M,N)$ with $N-M$ correlated Gaussian processes and $M$ uncorrelated square-root processes. The correlated gaussian processes are
\begin{align}
\dx Y_i(t) &= - k_i Y_i(t) \dx t + \nu_i \dx W_i (t),
\end{align}
where $W_i$ is a Wiener process and
\begin{align}
\dx W_i (t) \dx W_j (t) &= \rho_{ij} \dx t
\end{align}
for all $i,j = 1,2, \ldots, N-M$. Here $-1 < \rho_{ij} = \rho_{ji} < 1$ and $\rho_{ii} = 1$. The $M$ square-root processes are
\begin{align}
\dx X_m(t) = \alpha_m ( \theta_m - X_m(t) ) \dx t + \sigma_m \sqrt{ X_m(t) } \dx Z_m (t)
\end{align}
where $Z_m$ are independent Wiener process and
\begin{align}
\dx W_i (t) \dx Z_m (t) &= 0
\end{align}
for all $i = 1,2, \ldots, N-M$ and $m = 1,2, \ldots, M$. The short-rate is defined by
\begin{align}
r(t) = \delta + \sum_{m=1}^{M} X_m(t) + \sum_{i=1}^{N-M} Y_i(t),
\end{align}
where $\delta$ is a constant. Thus
\begin{align}
\dx r(t) &= \left( \sum_{m=1}^{M} \alpha_m ( \theta_m - X_m(t) ) - \sum_{i=1}^{N-M} k_i Y_i(t) \right) \dx t \\
&+ \sum_{m=1}^{M} \sigma_m \sqrt{ X_m(t) } \dx Z_m (t) + \sum_{i=1}^{N-M} \nu_i \dx W_i (t) .
\end{align}

If we define
\begin{align}
H(t,T) &= \int_t^T \delta \dx x = (T-t) \delta, \\
\beta_m &= \sqrt{\alpha_m^2 + 2 \sigma_m^2} , \\
C_i(x) &= \frac{1 - \e^{-k_i x}}{k_i}, \\
B_m(x) &= \frac{ 2 ( \e^{\beta_m x} - 1 ) }{ (\beta_m + \alpha_m)( \e^{\beta_m x} - 1) + 2 \beta_m }, \\
A(x) &= \sum_{m=1}^M \frac{2 \alpha_m \theta_m }{\sigma_m^2} \log \left( \frac{ 2 \beta_m \e^{ \frac{1}{2} (\beta_m + \alpha_m) x } }{ (\beta_m + \alpha_m)( \e^{\beta_m x -1} ) + 2 \beta_m } \right) \\
&+ \frac{1}{2} \sum_{i=1}^{N-M} \sum_{j=1}^{N-M} \frac{\nu_i \nu_j \rho_{ij}}{k_i k_j} \left( x - C_i(x) - C_j(x) + \frac{1-\e^{(k_i+k_j)x}}{k_i + k_j} \right)
\end{align}
for all $i = 1,2, \ldots, N-M$ and $m=1,2, \ldots, M$, then the price of a zero coupon bond is given by
\begin{align}
\Bond (t,T) = \exp{ \left( A(\tau) - \sum_{m=1}^M B_m(\tau) X_m(t) - \sum_{i=1}^{N-M} C_i(\tau) Y_i(t) - H(t,T) \right) },
\end{align}
where $\tau = T-t$. This model also has a semi-explicit formula for options on zero-coupon bonds. The method to calculate this will introduced in Section \ref{chap:fourier}.

We note that  Va\v{s}\'{i}\v{c}ek--model is $A(0,1)$ and CIR--model is $A(1,1)$ in this notation.

These $A(M,N)$--models can be made into $A(M,N)++$ models by using the following dynamic extension.

\section{Dynamic extension to match the given term-structure}
\label{sec:dynamicextension}

This section follows the paper by \textcite{brigomercurio2001deterministic}.

Let $(\Omega^x, \Pm^x, \F^x)$ be a probability space. We first assume the process $(x_{\alpha}(t))$ follows
	\begin{align}
		\dx x_{\alpha}(t) &= \mu(x_{\alpha}(t); \alpha) \dx t + \sigma(x_{\alpha}(t)); \alpha) \dx W_x(t) , \\
		x_{\alpha}(0) &= x_0
	\end{align}
under the measure $\Pm^x$, where $\alpha$ is a parameter vector. Let $\F_t^x$ be the member of a filtration generated by $x_{\alpha}$ up to time $t$. Suppose that the process $(x_{\alpha}(t))$ is the short-rate process under the risk-free measure $\Pm_x$ and the price of a zero-coupon bond is
	\begin{align}
		\Bond^x(t,T) &= \E_{\Pm_x} \left( \e^{ - \int_t^T x_{\alpha}(s) \ \dx s} \ | \ \F_t^{x_{\alpha}} \right) ,
	\end{align}
which is a function of $(t,T,x_{\alpha}, \alpha)$. \textcite{brigomercurio2001deterministic} calls this as a reference model. It is not guaranteed that the implied zero-curve structure by this model will match the observed market data.

Let $\varphi(t; \alpha, x_0) = \varphi(t; \alpha^*)$ be a deterministic real-valued function that it is at least integrable under any closed interval. Suppose that the short-rate follows
	\begin{align}
		r(t) = x(t) + \varphi(t; \alpha^*)
	\end{align}
and $(x(t))$ follows that same process under the risk-free measure $\Pm_0$ as $(x_{\alpha}(t))$ does under the measure $\Pm^x$. This model is the shifted model. This implies that
	\begin{align}
		\Bond(t,T) &= \E_{\Pm_0} \left( \e^{ - \int_t^T r(s) \ \dx s} \ | \ \F_t \right) \\
			&= \E_{\Pm_0} \left( \e^{ - \int_t^T (x(s) + \varphi(s; \alpha^*)) \ \dx s} \ | \ \F_t \right) \\
			&=  \E_{\Pm_0} \left( \e^{ - \int_t^T x(s)  \ \dx s} \ | \ \F_t \right) \e^{ - \int_t^T \varphi(s; \alpha^*) \ \dx s} \\
			&=  \E_{\Pm^x} \left( \e^{ - \int_t^T x_{\alpha}(s)  \ \dx s} \ | \ \F_t^{x_{\alpha}} \right) \e^{ - \int_t^T \varphi(s; \alpha^*) \ \dx s} \\
			&= \Bond^x(t,T) \e^{ - \int_t^T \varphi(s; \alpha^*) \ \dx s} .
	\end{align}
If we have a method to calculate the bond price under the reference model, then bond prices under the shifted can be calculated by a making a deterministic discounting based on the function $\varphi(t, \alpha^*)$.

In order to shorten the notations, we denote
	\begin{align}
		I(t,T,f) = \e^{ \int_t^T f(s) \ \dx s } ,
	\end{align}
where $f$ is a real valued function. If there is also a function
\begin{align}
\ZBC^x (t,S,T,K) = \E_{\Pm_x} \left( \e^{ - \int_t^S x_{\alpha}(s) \ \dx s} \left( \Bond^x(t,T) - K \right)^+ \ | \ \F_t^{x_{\alpha}} \right)
\end{align}
for the price at time $t$ of a call option maturing at $S$ for a $T$-bond under the reference model. Now the price under the shifted model is
	\begin{align}
	&\ZBC(t,S,T,K) \\ = &\E_{\Pm_0} \left( \e^{ - \int_t^S r(s) \ \dx s} \left( \Bond(S,T) - K \right)^+ \ | \ \F_t \right) \\
		= &\e^{ - \int_t^S \varphi(s; \alpha^*) \ \dx s} \E_{\Pm_0} \left( \e^{ - \int_t^S x(s) \ \dx s} \left( \Bond(S,T) - K \right)^+ \ | \ \F_t \right) \\
		= &I(t,S, -\varphi) \E_{\Pm_0} \left( I(t,S,-x) \left( \Bond(S,T) - K \right)^+ \ | \ \F_t \right)
	\end{align}
As
	\begin{align}
		&\E_{\Pm_0} \left( I(t,S,-x) \left( \Bond(S,T) - K \right)^+ \ | \ \F_t \right) \\
		= &\E_{\Pm_0} \left( I(t,S,-x) \left( \Bond^x(S,T) I(S,T,-\varphi) - K \right)^+ \ | \ \F_t \right) \\
		= &\E_{\Pm_0} \left( I(t,S,-x) \left( \Bond^x(S,T) - I(S,T,\varphi) K \right)^+ \ | \ \F_t \right) I(S,T,-\varphi) \\
		= &\ZBC^x(t,S,T,K\e^{ \int_S^T \varphi(s; \alpha^*) \ \dx s}) \e^{ - \int_S^T \varphi(s; \alpha^*) \ \dx s} .
	\end{align}
Here we have used again the equivalence on processes $x(t)$ and $x_{\alpha(t)}$. Hence
	\begin{align}
		&\ZBC(t,S,T,K) \\ = &\e^{ - \int_t^S \varphi(s; \alpha^*) \ \dx s} \ZBC^x(t,S,T,K\e^{ \int_S^T \varphi(s; \alpha^*) \ \dx s}) \e^{ - \int_S^T \varphi(s; \alpha) \ \dx s} \\
		= & \ZBC^x(t,S,T,K\e^{ \int_S^T \varphi(s; \alpha^*) \ \dx s}) \e^{ - \int_t^T \varphi(s; \alpha^*) \ \dx s}
	\end{align}
We see that the options prices for the shifted model can be computed easily, if the options prices can be calculated efficiently under the reference model. But not pure discounting is not enough, as we also have to shift probabilities by shifting the target strike.

Thus the following holds.

\begin{thm}
\label{bondandoptionpricesinextension}
	Under the earlier assumptions, the $T$-bond has a price
	\begin{align}
	\Bond(t,T) =  \Bond^x(t,T) \e^{ - \int_t^T \varphi(s; \alpha^*) \ \dx s}
	\end{align}
	and a call option with maturity $S$ on this bond has price
	\begin{align}
	\ZBC(t,S,T,K) =  \ZBC^x(t,S,T,K\e^{ \int_S^T \varphi(s; \alpha^*) \ \dx s}) \e^{ - \int_t^S \varphi(s; \alpha^*) \ \dx s}
	\end{align}
	at the time $t$.
\end{thm}

This model can achieve a perfect fit to the initial interest-rate term structure.

\begin{thm}
\label{extensionequivalentandfactor}
	Under the earlier assumptions, the following are equivalent:
\begin{itemize}
	\item The model
		\begin{align}
			r(t) = x(t) + \varphi(t; \alpha^*)
		\end{align}
	has a perfect fit to the given interest-rate term structure,
	\item for all $t \geq 0$
		\begin{align}
			\e^{-\int_t^T \varphi(s; \alpha^*) \dx s} = \frac{\Bond^M(0,T) \Bond^x(0,t)}{\Bond^M(0,t) \Bond^x(0,T)},
		\end{align}
	\item for all $t \geq 0$
	\begin{align}
		\varphi( t, \alpha^*) = \Forwardrate^M(0,t) - \Forwardrate^x(0,t)
	\end{align}
\end{itemize}
where $\Bond^M$ are observed market prices of zero-coupon bonds, $\Forwardrate^M(0,t)$ is the market implied forward-rate and $\Forwardrate^x(0,t)$ is the forward-rate implied by the reference model.
\end{thm}

\begin{proof}
We note that the perfect fit to market rates is equivalent to
	\begin{align}
		\label{perfectfitequationindynamicextension}
		\Bond^M(0,t) = \Bond(0,t) = \e^{ - \int_0^t \varphi(s; \alpha^*) \ \dx s} \Bond^x(0,t) .
	\end{align}
That is equivalent to
	\begin{align}
\e^{ - \int_t^T \varphi(s; \alpha^*) \ \dx s} &= \e^{ - \int_0^T \varphi(s; \alpha^*) \ \dx s} \e^{ \int_0^t \varphi(s; \alpha^*) \ \dx s} \\
	&= \frac{\Bond^M(0,T) }{ \Bond^x(0,T)} \frac{ \Bond^x(0,t)}{\Bond^M(0,t) }
\end{align}
Equation \ref{perfectfitequationindynamicextension} is also equivalent to
	\begin{align}
		\log \Bond^M(0,t) = \log \Bond^x(0,t) - \int_0^t \varphi(s; \alpha^*) \ \dx s
	\end{align}
for all $t \geq 0$ and this implies that the $-f^M(0,t) = - f^x(0,t) -  \varphi(t; \alpha^*)$.
\end{proof}

Theorem \ref{extensionequivalentandfactor} guarantees that no matter how the term-structure is shaped, we can fit it exactly with a suitable function. If we want to price bonds and options on bonds in the shifted model, then the calculation of the whole function $\varphi$ is unnecessary as we need only the values
		\begin{align}
\e^{-\int_t^T \varphi(s; \alpha^*) \dx s} = \frac{\Bond^M(0,T) \Bond^x(0,t)}{\Bond^M(0,t) \Bond^x(0,T)} .
\end{align}
Using the market values is preferable to estimating the forward curve, as the curve fitting may cause errors.

We note that if $\varphi(t) = \varphi(t; \alpha^*)$ is differentiable, then
	\begin{align}
		\dx r(t) &= \dx x(t) + \frac{\partial}{\partial t} \varphi(t; \alpha, x_0) \dx t \\
			&= \mu(x_{\alpha}(t); \alpha) \dx t + \sigma(x_{\alpha}(t)); \alpha) \dx W_x(t) + \frac{\partial}{\partial t} \varphi(t) \dx t \\
			&= \left( \mu(r(t) - \varphi(t); \alpha) + \frac{\partial}{\partial t} \varphi(t) \right) \dx t + \sigma(r(t) - \varphi(t)); \alpha) \dx W_x(t) .
	\end{align}

We also note that if $t=0$, then
	\begin{align}
		\Bond(0,T) &= \Bond^M(0,T) \\ &=  \Bond^x(0,T) \e^{ - \int_0^T \varphi(s; \alpha^*) \ \dx s}
	\end{align}
and
	\begin{align}
		\ZBC(0,S,T,K) &=  \ZBC^x(0,S,T,K\e^{ \int_S^T \varphi(s; \alpha^*) \ \dx s}) \e^{ - \int_0^S \varphi(s; \alpha^*) \ \dx s}
	\end{align}

\subsection{Va\v{s}\'{i}\v{c}ek++--model}
\label{sex:vasicek++}

By dynamically extending Va\v{s}\'{i}\v{c}ek--model, we get a model that is equivalent to a variant of Hull-White--model (\cite{hull1990pricing}, \cite{brigomercurio2001deterministic}). \added{This section follows } \textcite[pp. 100--102]{brigo2007interest}.

Suppose that we bootstrap the prices of $T_i$-bonds from the market for some $i=1,2, \ldots, n$. Let them be $\Bond^M(0,T_1), \Bond^M(0,T_2), \ldots , \Bond^M(0,T_n)$.

We assume that the evolution of $x(t)$ is given by
\begin{align}
\dx x(t) = k ( \theta - x(t) ) \dx t + \sigma \dx W(t) ,
\end{align}
where $k, \theta, \sigma > 0$ and $x(0) = x_0$ are constants and $W(t)$ is a brownian motion under the risk-free measure. We make the explicit assumption that $\theta = 0$, hence
\begin{align}
\dx x(t) = - k x(t) ) \dx t + \sigma \dx W(t) .
\end{align}
This assumption does not actually restrict the model at all. It only changes the center of the distribution, which does not matter as dynamic shift will be shifted similarly.

Now we know that
\begin{align}
\Bond^x(t,T) = \e^{- B(t,T)x(t)} ,
\end{align}
where
\begin{align}
B(t,T) &= \frac{ 2(\e^{\beta(T-t)} - 1) }{ (\beta + k)(\e^{\beta(T-t)} - 1) + 2\beta }, \\
\beta &= \sqrt{k^2+ 2\sigma^2}	.
\end{align}
We now define the short-rate by
	\begin{align}
		r(t) = x(t) + \varphi(t),
	\end{align}
where the function $\varphi$ is taken as in the Theorem \ref{extensionequivalentandfactor}. Now
	\begin{align}
		\dx r(t) &= \left( -kx(t) + \frac{\partial}{\partial t} \varphi(t) \right) \dx t + \sigma \dx W(t) \\
			&= \left( k( \varphi(t) -r(t) ) + \frac{\partial}{\partial t} \varphi(t) \right) \dx t + \sigma \dx W(t) .
	\end{align}

By Theorems \ref{bondandoptionpricesinextension} and \ref{extensionequivalentandfactor}, we have that
	\begin{align}
		\Bond(t,T) &= \Bond^x(t,T) \e^{ - \int_t^T \varphi(s; \alpha^*) \ \dx s} \\
			&= \Bond^x(t,T) \frac{\Bond^M(0,T) \Bond^x(0,t)}{\Bond^M(0,t \Bond^x(0,T)} \\
			&= \frac{\Bond^M(0,T) }{\Bond^M(0,t)} \e^{ -B(t,T)x(t) -B(0,t)x(0) + B(0,T)x(0) }
	\end{align}
if $t = T_i$ and $T = T_j$ for some $i < j$. If $t=0$, then $\Bond(0,T) = \Bond^M(0,T)$, as was expected.

A European call option with maturity $S$ on a $T$-bond and exercise price $K$ has a price
\begin{align}
\ZBC^x(t, S, T, K) = \Bond^x(t,T) N(d_1) - K \Bond^x(t,S) N(d_2),
\end{align}
at the time $t$ under the dynamics of $x(t)$, where
\begin{align}
d_1 &= \frac{ \log \frac{\Bond^x(t,T)}{K \Bond^x(t,S)} + \frac{V}{2} }{ \sqrt{V} } \\
d_2 &= \frac{ \log \frac{\Bond^x(t,T)}{K \Bond^x(t,S)} - \frac{V}{2} }{ \sqrt{V} } \\
V &= \sigma^2 \left( \frac{1 - \e^{-2k(T-S)}}{k} \right)^2 \frac{1 - \e^{-2k(S-t)}}{2k} .
\end{align}
Now, by Theorems \ref{bondandoptionpricesinextension} and \ref{extensionequivalentandfactor}, we have that
	\begin{align}
	\ZBC(t,S,T,K) &=  \ZBC^x(t,S,T,K\e^{ \int_S^T \varphi(s; \alpha^*) \ \dx s}) \e^{ - \int_t^S \varphi(s; \alpha^*) \ \dx s} \\
		&= \ZBC^x(t,S,T,K/I(S,T)) I(t,S),
	\end{align}
where
	\begin{align}
		I(t,S) &= \frac{\Bond^M(0,S) \Bond^x(0,t)}{\Bond^M(0,t) \Bond^x(0,S)} \\
				&= \frac{\Bond^M(0,S) }{\Bond^M(0,t)} \e^{ -B(0,t)x(0) + B(0,S)x(0) } \end{align}
and
	\begin{align}
		I(S,T) &= \frac{\Bond^M(0,T) \Bond^x(0,S)}{\Bond^M(0,S) \Bond^x(0,T)} \\
			&= \frac{\Bond^M(0,T) }{\Bond^M(0,S)} \e^{ -B(0,S)x(0) + B(0,T)x(0) } .
	\end{align}

\subsection{CIR++--model}

Extended CIR--model, or CIR++--model, can be constructed by combining the formulas in section \ref{sec:cir} with the results from section \ref{sec:dynamicextension} as was done in section \ref{sex:vasicek++}. As this is trivial, we shall writing the formulas again.

\subsection{G2++--model}

$A(2,2)++$--model is achieved by extending $A(2,2)$--model as above. Thus model is equivalent to model by \textcite{hull1994numerical} and it is called as  $G2++$--model by \textcite{brigo2007interest}.

\textcite[pp. 153--156, 172--173]{brigo2007interest} contains the proof for the following.

\begin{thm}
A European call option with maturity $S$ on a $T$-bond and exercise price $K$ has a price
\begin{align}
\ZBC^x(t, S, T, K) &= \Bond(t, T) \Phi \left( K^* + \frac{1}{2} S(t,S,T) \right) \\
&- \Bond(t, S) K \Phi \left( K^* - \frac{1}{2} S(t,S,T) \right)
\end{align}
where
\begin{align}
K^* & = \frac{\log \frac{\Bond(t,T)}{K\Bond(t,S)}}{S(t,S,T)} \\
S(t,S,T)^2 &= \sum_{i=1}^2 \frac{\nu_i^2}{2 k_i^3} \left( 1 - \e^{ -k_i(T-S) } \right)^2 \left( 1 - \e^{-2k_i(S-t)} \right) \\
&+ 2 \rho \frac{\nu_1 \nu_2}{ k_1 k_2 (k_1 + k_2) }  \left( 1 - \e^{ -k_1(T-S) } \right)  \left( 1 - \e^{ -k_2(T-S) } \right)  \left( 1 - \e^{ -(k_1 + k_2)(S-t) } \right)
\end{align}
\end{thm}


\subsection{Horizontal Asymptotes}\label{subsec:HorizontalAsymptotes}
The line $y=L$ is a \dfont{horizontal asymptote} of $f(x)$ if either
$$\lim_{x\to\infty}f(x)=L\qquad\mbox{or}\qquad\lim_{x\to-\infty}f(x)=L.$$

\begin{example}{Horizontal Asymptotes}{HorizontalAsymptotes}
Find the horizontal asymptotes of $\ds f(x)=\frac{|x|}{x}$.
\end{example}

\begin{solution}
We must compute two infinite limits.
First,
$$\lim_{x\to\infty}\frac{|x|}{x}.$$
Notice that for $x$ arbitrarily large that $x>0$, so that $|x|=x$.
In particular, for $x$ in the interval $(0,\infty)$ we have
$$\lim_{x\to\infty}\frac{|x|}{x}=\lim_{x\to\infty}\frac{x}{x}=1.$$
Second, we must compute
$$\lim_{x\to-\infty}\frac{|x|}{x}.$$
Notice that for $x$ arbitrarily large negative that $x<0$, so that $|x|=-x$.
In particular, for $x$ in the interval $(-\infty,0)$ we have
$$\lim_{x\to -\infty}\frac{|x|}{x}=\lim_{x\to -\infty}\frac{-x}{x}=-1.$$
Therefore there are two horizontal asymptotes, namely, $y=1$ and $y=-1$.
\end{solution}


\section{Pi Digits Computation}
\paragraph{Goals and algorithms.}
The goal of these program implementations were to compute the number pi using
Riemann sums. In other words combining the area under the curve split into small
rectangular intervals. For
parallelization, the strategy used was to evenly distribute the
number interval areas to be calculated among the threads created by the
program, and then join the results of each thread at the end to form the final
answer for pi. To allow for a substantial enough runtime to measure, the
programs were run using an absurdly high number of intervals.

\paragraph{Pi Implemented in C++.}
The C++ implementation used the POSIX pthread library, and creates $n$
threads, as the user specifies, to compute the area under the curve. In order
to isolate the performance of the thread creation, deletion, and parallelization
of pthreads, the timer used for measuring the program's runtime is started after
all initialization of variables occurs, and just before creation of the child
threads.

Once all
threads are created, the main thread waits idle while trying to join all child
threads. As each child thread terminates, the final sum of pi is added to. Once
all threads are finished, the final result is calculated and the timer stops.

\paragraph{Pi Implemented in Rust.}
The Rust implementation, like the C++ implementation, takes $n$ threads, as the
user specifies, to compute the area under the curve. Another similarity the Rust
version shares with its C++ counterpart is that it's timer begins after the all
initialization of variables, and just before creation of the child threads.

Once each thread finishes its work, it sends it's solution to a channel located
on the main thread where it is added to a global sum. Once all threads are
finished, the final sum is calculated and the timer stops.

\paragraph{User Experience writing pi in C++.}
Overall this program was trivial to implement in C++, as the problem is not very
complex, and room for memory errors to appear is narrow. There were no recorded
memory errors encountered during the writing of the C++ program, and no major
usability faults regarding pthreads, with one exception: Passing arguments to
a child thread is certainly type unsafe, and memory unsafe, and could be
a potential culprit for many runtime errors if not carefully managed. This is
due to C++ pthreads requiring a single {\cf void *} for all arguments, which
guarantees almost nothing about the data being passed into the thread, and
cannot allow for the compiler to check for safety. In the case of Pi, this was
not a major issue. Only a single integer value was to be passed into each
thread upon its creation, and so no risk for accessing out-of-bounds memory was
present.

\paragraph{User Experience writing pi in Rust.}
Writing this program felt much simpler and more inutitve than C++ with pthreads.
Since channels make asynchronous sums very trivial, the hard part was pretty
much already solved from the begining. The implementation looks nearly identical
to what a single threaded pi solver implementation would look like, except for
the addition of a thread spawn occuring in the main loop and a channel being
used to collect the sums. Even an individual with little knowledge of the
language could easily figure out whats going on, a feature that isn't shared
with the C++ implementation.


\documentclass[12pt]{scrartcl}

\input{preamble}

\makeatletter
\title{Hack 12.0}\let\Title\@title
\subtitle{Computer Science I\\
Recursion \& Memoization\\
{\small
\vskip1cm
Department of Computer Science \& Engineering \\
University of Nebraska--Lincoln}
\vskip-3cm}
%\author{Dr.\ Chris Bourke}
\date{~}
\makeatother

\begin{document}

\maketitle

\hrule

\input{instructions.tex}

Correctness: proportionally for test cases; it \textbf{must run on the grader} to
receive any points.

\section*{Problem Statement}

A binomial coefficient, ``$n$ choose $k$'' is a number that corresponds
to the number of ways to \emph{choose} $k$ items from a set of $n$ distinct
items.  You may be familiar with some the notations, $C(n,k)$ or $C_n^k$
or ${}_{n}C_k $, but most commonly this is written as
  $${n \choose k}$$
and read as ``$n$ choose $k$''.  There is an easy to compute formula involving
factorials:
  $${n \choose k} = \frac{n!}{(n-k)!k!}$$
For example, if we have $n = 4$ items, say $\{a, b, c, d\}$ and want to choose
$k=2$ of them, then there are
  $${4 \choose 2} = \frac{4!}{(4-2)!2!} = 6$$
ways of doing this.  The six ways are:
  $$\{a, b\}, \{a, c\}, \{a, d\}, \{b, c\}, \{b, d\}, \{c, d\}$$
There are a lot of other interpretations and applications for binomial
coefficients, but this hack will focus on computing their value using
a different formula, Pascal's Rule\footnote{Which can be used to generate
Pascal's Triangle, \url{https://en.wikipedia.org/wiki/Pascals_triangle}}:
  $${n \choose k} = {n-1 \choose k} + {n-1 \choose k-1}$$
which is a recursive formula.  The base cases for Pascal's Rule are when
$k = 0$ and $n = k$.  In both cases, the value is 1.  When $k = 0$, we are
not choosing any elements and so there is only one way of doing that (i.e.\
choose nothing).  When $n = k$ we are choosing every element, again there
is only one way of doing that.

\subsection*{Writing a Naive Recursion}

Implement and test the following function \emph{using a recursive}
solution:

\mintinline{c}{long choose(int n, int k);}

which takes $n$ and $k$ and computes ${n\choose k}$ using Pascal's Rule.
Note that the return type is a \mintinline{c}{long}\footnote{For those
using Windows, you may need to instead use a \mintinline{c}{long long}
data type to get a 64-bit integer.} which is a 64-bit
integer allowing you to compute values up to
  $$2^{63}-1 = 9,223,372,036,854,775,807$$
(a little over 9 quintillion).  Write a \mintinline{c}{main} function
that takes $n$ and $k$ as command line arguments and outputs the result
to the standard output so you can easily test it.

\subsection*{Benchmarking}

Run your program on values of $n, k$ in Table \ref{table:easyValues}
and time (roughly) how long it takes your program to execute.  You
can check your solutions with an online tool such as
\url{https://www.wolframalpha.com/}.

\begin{table}[ht]
\centering
\begin{tabular}{c|c}
$n$ & $k$ \\
\hline\hline
4 & 2 \\
10 & 5 \\
32 & 16 \\  %5 seconds
34 & 17 \\  %15 seconds
36 & 18 \\ %60 seconds
\end{tabular}
\caption{Test Values}
\label{table:easyValues}
\end{table}

Now formulate an estimate of how long your program would take to
execute with larger values.  You can make a \emph{rough} estimate
how many function calls are made using the binomial value itself.
That is, to compute ${n \choose k}$ using Pascal's Rule would make
\emph{about} ${n \choose k}$ function calls.

Use the running time of your program from the test values to
estimate how long your program would run for the values in
Table \ref{table:hardValues}.

\begin{table}[ht]
\centering
\begin{tabular}{c|r}
${n \choose k}$ &  value \\
\hline\hline
${54 \choose 27}$ & =     1,946,939,425,648,112 \\
${56 \choose 28}$ & =     7,648,690,600,760,440 \\
${58 \choose 29}$ & =    30,067,266,499,541,040 \\
${60 \choose 30}$ & =   118,264,581,564,861,424 \\
${62 \choose 31}$ & =   465,428,353,255,261,088 \\
${64 \choose 32}$ & = 1,832,624,140,942,590,534 \\
${66 \choose 33}$ & = 7,219,428,434,016,265,740 \\
\end{tabular}
\caption{Larger Values}
\label{table:hardValues}
\end{table}

\subsection*{Improving Performance with Memoization}

You'll now improve your program's performance using memoization
to avoid unnecessary repeated recursive calls.

\begin{enumerate}
  \item Write code (either in the \mintinline{c}{main} function or
  using another ``entry point'' function) to create a memoization table
  containing \mintinline{c}{long} values of dimension $(n+1) \times (k+1)$
  \item Initialize the values in the table to $-1$ as a flag value
  to indicate that the value in the table has not yet been set.
  \item Using your previous recursive implementation as a guide, write
  a new recursive function that also takes the table as a parameter.
  When the function needs to compute ${n \choose k}$ it checks the table
  first: if the value has already been computed (is not $-1$) then it
  returns that value.  Otherwise, it performs the recursive computation.
  Before returning the value, however, it should store it (\emph{cache}
  it) in the table so that subsequent computations avoid the recursion.
  \item Modify your \mintinline{c}{main} function to use this more
  efficient version and re-test it with the values above.  Compare the
  time it took using memoization versus the naive recursion.
  \item Rerun your program with the values in Tables \ref{table:easyValues}
  and \ref{table:hardValues} to verify they work and note the difference
  in running time.
\end{enumerate}

\section*{Instructions}

\begin{itemize}

  \item Place all of your function definitions in a source file named
  \mintinline{text}{binomial.c} and hand it in with your header file,
  \mintinline{text}{binomial.h}.  Place your \mintinline{c}{main}
  function in a file named \mintinline{text}{binomialDemo.c}

  \item You are encouraged to collaborate any number of students
  before, during, and after your scheduled hack session.

  \item Include the name(s) of everyone who worked together on
  this activity in your source file's header.

  \item Turn in all of your files via webhandin, making sure that
  it runs and executes correctly in the webgrader.  Each individual
  student will need to hand in their own copy and will receive
  their own individual grade.
\end{itemize}

\end{document}


\lab{Hash Tables}{Hash Tables}
\label{lab:HashTables}
\objective{Empower students with basic knowledge of the fundamental, canonical data structures in order to understand performance and runtime characteristics.}

\section*{Hash Tables}
A \emph{hash table} is a very simple data structure that trades memory space for speed.
The key advantage of a hash table is this speed; most of the operations of a hash table execute very efficiently, independent of its size.
As such, hash tables have very fast lookup times; they form the underlying data structure for Python's set and dictionary types.

One of the key components of a hash table is a \emph{hash function}.
A hash function takes a piece data as an input and outputs a natural number which corresponds to the index for that piece of data.
Since the hash function must be executed to perform any operation on the hash table (e.g. insertion or lookup), it is imperative that the hash function execute quickly.
Thus, the heart of a hash table is a good hash function.

We begin defining our \li{HashTable} class, then, by allocating space within a one-dimensional array (in this case, a Python list) and by taking both the array size and the hash function as arguments.
\begin{lstlisting}
class HashTable(object):
    def __init__(self, capacity):
        # Efficiently extend a list to the designated hash table size
        self.hashtable = [None] * capacity
        # Keeps track of the number of elements within the hash table
        self.size = 0
        # Indicates the size of the hash table
        self.capacity = capacity

    def load_factor(self):
        # The load factor tells how saturated a hash table is
        return float(self.size)/self.capacity
\end{lstlisting}

A good hash function will execute quickly and distribute items uniformly throughout the table.
In the hash tables in this lab, we will use Python's built-in \li{hash} function.
This code demonstrates the principle steps of inserting into a hash function.
Instead of using our \li{HashTable} class, however, we simply use a similar construct to show how the steps might be implemented in a specific instance:
\begin{lstlisting}
>>> capacity = 5
>>> table = [None] * capacity
>>> hash('three')
8354070563160704301
>>> hash('three') % capacity
1
>>> table[1] = 'one'
>>> print table
[None, 'three', None, None, None]
>>> table[hash('four') % capacity] = 'four'
>>> table[hash('five') % capacity] = 'five'
>>> print table
['four', 'three', 'five', None, None]

# We can locate 'five' quickly
# Notice that is faster than the list's builtin methods
>>> %timeit table.index('five')
1000000 loops, best of 3: 190 ns per loops
>>> %timeit table[hash('five') % capacity]
10000000 loops, best of 3: 111 ns per loop
>>> table[hash('two') % capacity] = 'two'
# Note that our load factor is 4/5.
>>> print table
['four', 'three', 'five', 'two', None]

# When our load factor passes a certain threshold, we need to make the hash table bigger
# Resizing the hash table means rehashing everything in it
>>> def resize(table, new_cap):
        new_table = [None] * new_cap
        for i in table:
            new_table[hash(i) % new_cap] = i
        return new_table
>>> table = resize(table, 7)
>>> print table
[None, None, None, 'two', None, None, 'four']
\end{lstlisting}

The ideal hash function maps unique inputs to unique outputs that are uniformly distributed over the hash space.
However, it is extremely difficult to create an ideal hash function; most hash functions will experience hash collisions.
Hash collisions are when two unique inputs are mapped to the same index in the hash table (in our example above, resizing the hash table resulted in hash collisions that lost half the strings we stored in the table).
Fortunately, there are ways to handle hash collisions.

\paragraph{Probing.}
One method that we can use to resolve hash collisions is probing.
A short example will be succintly illustrate this method.
\begin{lstlisting}
>>> capacity = 5
>>> table = [None] * capacity
# Now we add the words 'one' and 'two'
# Notice what happens when we add 'two'
>>> table[hash('one') % capacity] = 'one'
>>> print table
[None, None, None, 'one', None]
>>> table[hash('two') % capacity]
'one'

# We can resolve the collision by looking for the next available index
# Starting at index 3, we look at index 3+1 % capacity, 3+2 % capacity, 3+3 % capacity, ..., until we find and empty index
# Seeing that index 4 is None, we store 'two' there
>>> table[4] = 'two'
>>> print table
[None, None, None, 'one', 'two']
\end{lstlisting}
This is called linear probing.
There are other variations of this idea which are more optimal.

\paragraph{Chaining.}
Another method for resolving hash collisions is chaining.
This method slightly alters the structure of our hash table, so that at each index, a list is stored.
When an item is mapped to an index, it is appeneded to the list at that index.
\begin{lstlisting}
>>> capacity = 5
>>> table = [list() for i in xrange(capacity)]

# Now we add the words 'one' and 'two'
# Notice what happens when we add 'two'
>>> table[hash('one') % capacity].append('one')
>>> print table
[[], [], [], ['one'], []]
>>> table[hash('two') % capacity].append('two')
>>> print table
[[], [], [], ['one', 'two'], []]
\end{lstlisting}

\begin{problem}
Using the concepts illustrated in the examples above, and an insert method to your \li{HashTable} class.
Don't forget to update your size for the number of elements within the hashtable!
Use chaining to handle hash collisions (namely, implement your insert method via chaining).

Then, create a method for your hash table to resize.
Implement this method such that, when your hash table exceeds a load factor of $.8$, it will resize the hash table so that the load factor is below $.3$.
Don't forget to rehash the elements already within your table!

Initialize a hash table of size 4 and insert the following felines: `lion', `tiger', `cheetah', `cougar', `colocolo', `cat', `clouded leopard', and `jaguar'.
Print your hash table.
\label{prob:hash_table}
\end{problem}

\begin{info}
Usually exisiting implementations of data structures are better to use.
The implementations of linked lists, trees, and hash tables in this lab were designed to familiarize you with the data structure's mechanics through example and practice.
In real situations, you will likely use Python's deques, sets, or dictionaries.
Python doesn't have a built-in implementation of trees, but there are several Python libraries that have highly optimized implementations of trees.

Performance tuning for data structures can be tricky even for professionals.
For this reason, it is better to avoid writing your implementation when one already exists.
\end{info}


\chapter{Radial Basis Function}

\section{Least Square Estimator}
Linear Regression Equations:
\begin{equation*}
\begin{split}
y &= \sum_{i=1}^{n} w_i f_i(\mathbf{x}) \\
&= w_1 f_1(\mathbf{x}) + w_2 f_2(\mathbf{x}) + \ldots + w_n f_n(\mathbf{x})  \\
\mathbf{y} &= \mathbf{Aw}
\end{split}
\end{equation*}
The squared error:
\begin{equation*}
\begin{split}
E &=\| \mathbf{y -d} \|^{2} \\
&= \| \mathbf{d - Aw} \|^{2} \\
&= \mathbf{(d-Aw)^{T}(d-Aw)} \\
&= \mathbf{d^{T}d - d^{T}Aw - (Aw)^{T}d + (Aw)^{T}(Aw)} \\
&= \mathbf{d^{T}d - 2w^{T}A^{T}d + w^{T}A^{T}Aw}
\end{split}
\end{equation*}
For least-square estimator:
\begin{equation*}
\begin{split}
\frac{\partial E}{\partial \mathbf{w}} &= 0 \\
2\mathbf{A^{T}d} - 2\mathbf{A^{T}Aw} &= 0 \\
\mathbf{A^{T}Aw} &= \mathbf{A^{T}d} \\
\mathbf{w} &= \mathbf{(A^{T}A)^{-1}A^{T}d}
\end{split}
\end{equation*}
If A is non-singular, $\mathbf{(A^{T}A)^{-1}A^{T}}$ is pseudoinverse of A. \\
If A is singular, pseudoinverse is same as inverse of matrix. \\
Singular matrix: Square matrix that is not invertible.

\section{Cover's Separability Theorem}
A complex pattern-classification problem cast in a high-dimensional hidden space non-linearly is more likely to be linearly separable than in a low-dimensional input space.

\section{Basis Function Network}
\begin{figure}[h]
\centering
\includegraphics[width=10cm]{chapter5_1}
\end{figure}
$$f(\mathbf{x}) = y = \sum_{i=1}^{N} w_i \phi_i(\mathbf{x})$$
The radial basis functions technique consists of choosing a basis function of the following form:
$$f(\mathbf{x}) = \sum_{i=1}^{N} w_i \phi(\| \mathbf{\mathbf{x-\mu_{i}}}\|)$$

\section{Typical Radial Basis Functions}
Piecewise linear approximation:
$$\phi(\| \mathbf{x} - \pmb{\mu_{i}}\|) = \| \mathbf{x} - \pmb{\mu_{i}} \|$$
Cubic approximation:
$$\phi(\| \mathbf{x} - \pmb{\mu_{i}}\|) = \| \mathbf{x} - \pmb{\mu_{i}} \|^{3}$$
Gaussian function:
$$\phi(\| \mathbf{x} - \pmb{\mu_{i}}\|) = exp(-\frac{\| \mathbf{x} - \pmb{\mu_{i}} \|^{2}}{2 \sigma^{2}})$$
Thin plate splines:
$$\phi(\| \mathbf{x} - \pmb{\mu_{i}}\|) = \| \mathbf{x} - \pmb{\mu_{i}} \|^{2} log(\| \mathbf{x} - \pmb{\mu_{i}} \|)$$
Multiquadratic function:
$$\phi(\| \mathbf{x} - \pmb{\mu_{i}}\|) = (\| \mathbf{x} - \pmb{\mu_{i}} \|^{2} + \sigma^{2})^{1/2}$$
Inverse multiquadratic function:
$$\phi(\| \mathbf{x} - \pmb{\mu_{i}}\|) = (\| \mathbf{x} - \pmb{\mu_{i}} \|^{2} + \sigma^{2})^{-1/2}$$

\section{RBF as Interpolation Networks}
In the interpolation approach,RBF technique consists of choosing a function such that:
\begin{equation}
f(\mathbf{x}) = y = \sum_{i=1}^{L} w_i \phi(\| \mathbf{x - x_{i}} \|^{2})
\label{interpolation}
\end{equation}
And the interpolation condition is:
\begin{equation}
f(\mathbf{x}) = \mathbf{d}
\label{interpolation_condition}
\end{equation}
Let:
\begin{equation}
\phi_{li} = \phi(\| \mathbf{x_l - x_i} \|)
\label{temp4}
\end{equation}
Substituting \ref{interpolation_condition} and \ref{temp4} to \ref{interpolation}
\begin{equation}
\begin{split}
\sum_{i=1}^{L} w_i \phi(\| \mathbf{x_l - x_{i}} \|^{2}) &= d_l \\
\sum_{i=1}^{L} w_i \phi_{li} &= d_l
\end{split}
\end{equation}
Rewrite the equation in matrix form:
$$
\begin{bmatrix}
\phi_{11}  & \cdots & \phi_{1N} \\
\vdots     & \ddots & \vdots \\
\phi_{N1}  & \cdots & \phi_{NN}
\end{bmatrix}
\begin{bmatrix}
w_1 \\
\vdots \\
w_N
\end{bmatrix} =
\begin{bmatrix}
d_1 \\
\vdots \\
d_N
\end{bmatrix}
$$
$$\pmb{\phi w = d}$$
Assuming \emph{Gram Matrix} $\phi$ is non-singular:
$$\pmb{w = \phi^{-1} d}$$
\textit{Note: The number of hidden nodes is equal to the number of data points}

\section{RBF as Approximation Networks}
In a generalized Gaussian RBF network, each hidden node represents a cluster of patterns.
\begin{equation}
\begin{split}
\phi(\| \mathbf{x} - \pmb{\mu_{i}}\|) &= exp\Big(-\frac{\| \mathbf{x} - \pmb{\mu_{i}} \|^{2}}{2 \sigma^{2}} \Big) \\
&= exp\Big(- \frac{1}{2} (\mathbf{x} - \pmb{\mu_i})^{T} \mathbf{C_i^{-1}} (\mathbf{x} - \pmb{\mu_i}) \Big)
\end{split}
\end{equation}
The centroid $\pmb{\mu}$ and covariance matrix $\mathbf{C}$:
$$\pmb{\mu_{i}} = \frac{1}{L} \sum_{j=1}^{L} \mathbf{x_j}$$
$$\mathbf{C_i} = \frac{1}{L-1} \sum_{j=1}^{L} (\mathbf{x_j} - \pmb{\mu_i})(\mathbf{x_j} - \pmb{\mu_i})^{T}$$
RBF use k-means clustering algorithm to learn centroids of hidden layer neurons.


\begin{enumerate}

\item If $\underline{s}$ is a maximum length LFSR sequence of an LFSR of length
$L$, then it has period $2^L-1$ (as covered in lectures). This means that the
LFSR goes through $2^L-1$ distinct states before repeating a state.
Specifically, these states are the members of $\mathbb{F}_2^L \setminus
{(0,\dots,0)}$. Also, the first bits of these states are the bits of the
sequence in a single period.

To check whether $\underline{s}$ is balanced, we just need to check whether the
first bits of the members of $\mathbb{F}_2^L \setminus {(0,\dots,0)}$ (the
states) are balanced. Clearly, there are $2^{L-1}$ states that start with 1 and
$2^{L-1}-1$ states that start with 0 (since including $(0,\dots,0)$ as a state
implies that it is the only state). Thus, the number of 0s and 1s differ by one
and therefore the sequence is balanced: given a large $L$, a randomly chosen bit
will be 0 or 1 each with probability approximately $1/2$.

\item In a maximum length LFSR, the state of the LFSR register runs through all
of the members of $\mathbb{F}_2^L \setminus {(0,\dots,0)}$ in a single period
(\cite{slides}, Slide 36). A run of length $i$ occurs at a time $t$ with state
$s_t$ if the preceding state $s_{t-1}$ (at time $t-1$) starts with a known
substring of $i+2$ bits (eg. (0, 1, \dots, 1, 0) - 1 repeated $i$ times - or (1,
0, \dots, 0, 1) (0 repeated $i$ times)). There are $2^{L-i-2}$ such states that
precede the start of a run of $i$ 0s or 1s. Accounting for both runs of ones and
of zeros, the total number of states that precede the start of any run is $2
\times 2^{L-i-2} = 2^{L-i-1}$.

Any state that precedes the start of a run either starts with 01 or 10. This
shows that approximately half of the states precede the start of a run: the half
that begins with 01 or 10 rather than 11 or 00. If the total number of states is
$tot_{states} = 2^L-1$ (excluding (0,\dots,0) ), then the number of states that
precede the start of a run is $tot_{runs} = 2^{L-1}$ - half of the bit strings
of length L.

This shows that the number of runs of length $k$ for $1 \le k \le L-1$ is

\[2^{L-1-k} = 2^{-k} \times 2^{L-1}\]

which means that $2^{-k}$ of all runs are of length k.

This breaks down when you consider runs of length L. However, there can only be
one run of length L: the run of L 1s. The run of L 0s is the 0 state, and is
never reached in a maximum-length LFSR whose initial state is non-zero, as only
non-zero states are found in such an LFSR (\cite{slides}, Slide 37).

\item We will prove the result with linear algebra.

\paragraph{Lemma:} Let $k$ be an integer with $1 \le k < 2^L-1$ and let $T$
be a linear transformation $\mathbb{F}^L_2 \to \mathbb{F}^L_2$ which
generates an m-sequence $(s_i)_{i=1}^\infty$. Then $\text{ker}(T^k+\mathbb{I})=\{0\}$.

\paragraph{Proof of Lemma:} Define $V$ as $\text{im}(T^k+\mathbb{I})$ and let
$c_T$ be the characteristic (and, in this case, minimal) polynomial of $T$.
Suppose this were not the case, that
$\text{dim ker}(T^k+\mathbb{I})>0$. Then $\text{dim}\ V<L$ by Rank-Nullity Theorem.
But $V$ is $T$ invariant, since, for $v \in \mathbb{F}_2^L$,
$T(T^k+\mathbb{I})v=(T^k+\mathbb{I})Tv$,
so let $P$ be the characteristic polynomial of $T$ restricted to $V$.
If $v$ is a vector in $\mathbb{F}^L_2$, then note $P(T)(T^k+\mathbb{I})v=0$,
by Cayley-Hamilton theorem on $V$. So $P(T)(T^k+\mathbb{I})=0$, and hence
$c_T(x) | P(x)(x^k+1)$ ($c_T$ is the minimal polynomial of the sequence
since it is irreducible). But $c_T$ is irreducible since $T$ generates an
m-sequence, and $\deg P<L=\deg c_T$, so $c_T(x) | x^k+1$, which implies
$k\ge 2^L-1$ (\cite{slides}, Slide 37), a contradiction. This concludes
the proof of the lemma.

\paragraph{Main Proof:} The $2^L-1$ $L$-vectors starting at each position
in an m-sequence consist of every non-zero $L$-vector. Since
$\text{ker}(T^k+\mathbb{I})=\{0\}$, these vectors are permuted in this map.
Specifically, there are $2^{L-1}-1$ vectors starting with $0$ and $2^{L-1}$ vectors
starting with $1$ in the image of this map. However, $s_x$ and $s_{x+k}$
disagree iff $s_{x+k}\oplus s_x=1$, or iff $T^k{\bf s_x}+{\bf s_x}$ begins with
$1$, where ${\bf s_x}$ is the $L$-vector starting with $s_x$ in the m-sequence.
However, the probability of $T^k{\bf s_x}+{\bf s_x}=(T^k+\mathbb{I}){\bf s_x}$
starting with $1$ is approximately $1/2$, since $T^k+\mathbb{I}$ maps about half the non-zero
$L$-vectors to vectors starting with $0$ and about half to vectors starting with
$1$. So the m-sequence
is approximately uncorrelated with itself shifted $k$ places whenever $k$ is not
a multiple of $2^L-1$.

\end{enumerate}