\chapter{Appendix I: Quaternions}

There exists a correspondence between the orientation of a $ 3D $ object represented by a $ 3 \times 3 $ orthonormal matrix and unit quaternions in the group $ SO(3) $ \citep{Murray}. Quaternions give a global parametrization of $ SO(3) $, by using four numbers instead of three to represent a rotation. A quaternion is a vector quantity of the form,

  \[Q=q_{0} + q_{1}i + q_{2}j + q_{3}k  \ \ \ \ \ \ \ \  qi \in \R, i = 0, . . . , 3,\]

where, $ q_{0} $ is the scalar component of $ Q $ and $ \vec{q}= (q_{1}, q_{2}, q_{3}) $ is the vector component. A convenient notation is $ Q = (q_{0}, \vec{q}) $ with $ q_{0} \in \R $, $ \vec{q} \in \R^{3} $.
The set of quaternions Q is a 4-dimensional vector space over the reals and the unit quaternion form a group with respect to quaternion multiplication is a rotation group. Multiplication is distributive and associative, but not commutative.

For rotation through an angle $ \theta $ about the axis $ n $, the Euler parameters are defined as \citep{Hanson},
\begin{align*}
e_{0}&=\cos(\theta / 2) \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ e_{2}=\cos \beta \cdot\sin (\theta / 2)\\
e_{1}&=\cos \alpha \cdot\sin (\theta / 2) \ \ \ \ \ \ \ \ \ e_{3}=\cos \gamma \cdot\sin (\theta / 2)
\end{align*}
where, $ \alpha, \beta $ and $ \gamma $ are the elementary rotations about any given random axis.

Given a rotation matrix $ R=exp\left(\hat{\omega}\theta \right) $, we define the associated unit quaternion as
$ Q = \cos (\theta / 2) $,  $ \omega\sin (\theta / 2) $
where, $ \omega \in \R^{3} $ represents the unit axis of rotation and $ \theta \in \R^{3} $ represents the angle of rotation.
Given a unit quaternion $ Q = \left(q_{0}, \vec{q} \right) $, we can extract the corresponding rotation by setting,

\begin{center}
\ \ \ \ \ $\theta = 2\cos ^{-1} q_{0}, \ \ \ \ \ \ \
\omega = \left\lbrace
\begin{array}{ll}
\frac{\vec{q}}{\sin (\theta / 2)} & \mbox{if $ \theta \neq 0,$}\\
0 & \mbox{otherwise,}
\end{array}
\right.\
$
\end{center}

and  \[R = exp\left(\hat{\omega}\theta\right).\]

It is observed that the components of the quaternion are Euler parameters. The Euler parameters have a geometrical meaning and thus the same holds for the components of the corresponding unit quaternion.

The Frenet 3D coordinate frame can be expressed in the form of quaternions. Assuming that the columns of the equation given below are the vectors $ [t_{i}, n_{i}, b_{i}] $ respectively, $ [q'(t)] $ can be written in the form,
\begin{eqnarray}
\left[
\begin{array}{c}
q_{o}'\\
q_{1}'\\
q_{2}'\\
q_{3}'
\end{array}
\right]=
\dfrac{v}{2}
\left[
\begin{array}{cccc}
0 & -\tau & 0 & -\kappa \\
\tau & 0 & \kappa & 0 \\
0 & -\kappa & 0 & \tau \\
\kappa & 0 & -\tau & 0
\end{array}
\right]
\cdot
\left[
\begin{array}{c}
q_{o}\\
q_{1}\\
q_{2}\\
q_{3}
\end{array}
\right]
\label{13}
\end{eqnarray}
where $ v = \parallel \dot{c} \parallel $ and,
\begin{align*}
q_{o}= \cos (\theta / 2) = \dfrac{1}{\sqrt{2}} & \sqrt{\cos \theta +1} = \dfrac{1}{2} \sqrt{Trace (R) +1} \\
q_{1}=& \dfrac{R_{32}-R_{23}}{4q_{0}} \\
q_{2}=& \dfrac{R_{13}-R_{31}}{4q_{0}} \\
q_{3}=& \dfrac{R_{21}-R_{12}}{4q_{0}}
\end{align*}

Key properties of equation \ref{13} are \citep{Hanson},
\begin{enumerate}
\item The matrix on the right hand side is antisymmetric, so that $ q(t) \cdot q'(t)=0 $ is preserved and all unit quaternions remain unit quaternions as they are.
\item Nine coupled equations with six orthonormality constraints are reduced to four coupled equations with a single constraint of unit length.
\end{enumerate}

From the above equation \ref{13}, the $ curvature $,
\begin{equation}
\kappa = - \left[\dfrac{2(q_{3}q'_{0}+q_{1}q'_{2})}{v(q_{1}^{2}+q_{3}^{2})} \right]
\label{14}
\end{equation}
and the $ torsion $,
\begin{equation}
\tau = \dfrac{2q'_{2}+v\kappa q_{1}}{vq_{3}}
\label{15}
\end{equation}
can be extracted by the quaternion frame. The Frenet equations \ref{14} and \ref{15} may be integrated to generate a unique moving continuous frame along the protein backbone, for nonvanishing $ \kappa (t) $ with its space curve.

\section*{Motivation for using Quaternions}

\begin{enumerate}
\item It can be observed that the description of Euler angles and exponential co-ordinates suffers from the problem of $ \textit{singularity} $ in certain situations \citep{Hanson}. The extraction of parameters from any given rotational transformation matrix produces a singular case as the maps being many to one. While with the unit quaternion, extraction of unit quaternion from given rotational transformation being unique, there is no such case in which the inverse produces singularity. In quaternion space, the scalars, vectors, and quaternions are unified. Besides this, spatial vector can be expressed in quaternion space provides us with elegant properties for manipulating equations.

\item The quaternion frame summarizes nine matrix elements with six orthonormality constraints necessary to reduce the actual number of parameters of the frame to the three Euler angles. This forms a $ 3D $ orientation frame in terms of four quaternion frame variables with the single constraint of unit length that provides $ \textit{less computaional complexity} $.

\item The Frenet frame is periodic but not globally defined. As soon as the inflection points \citep{shuangwei} occur in the plane it is observed that the normal components of the Frenet frame switches sign instantly, while the quaternion frame has no such abrupt changes. The Frenet frame becomes undefined when the curvature $ \left(\kappa\right) $, vanishes along a straight line segment or at inflection point. It provides no such prescription to define a smooth transition from the frame coming and leaving the straight segment. The quaternion frame is comparatively $ \textit{smooth} $ throughout.
\end{enumerate}

% ------------------------------------------------------------------------

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "../thesis"
%%% End:


\chapter{Probability}

\section{Probability space}

\index{probability measure}%
\index{probability!measure}%
\index{measure!probability}%
A \emph{probability measure} \(\Pr : 2^\Omega \to \Real\) on a set \(\Omega\) is a measure where \(\Pr(\Omega) = 1\).

\index{probability space}%
\index{probability!space}%
\index{space!probability}%
A \emph{probability space} is a measure space \((\Omega,\Pr)\) where
\(\Omega\) is a set called the
\index{sample space}%
\index{sample!space}%
\index{space!sample}%
\emph{sample space}
and \(\Pr : 2^\Omega \to \Real\) is a probability measure.

\index{event}%
An \emph{event} is a subset of a sample space.
Relationship between two events \(A\) and \(B\):
They are
\index{events!equiprobable}%
\index{equiprobable events}%
\emph{equiprobable} iff \(\Pr(A) = \Pr(B)\),
\index{mutually exclusive events}%
\index{events!mutually exclusive}%
\emph{mutually exclusive} iff \(A \cap B = \emptyset\),
\index{independent events}%
\index{events!independent}%
\emph{independent} iff \(\Pr(A \cap B) = \Pr(A) \cdot \Pr(B)\).

\paragraph{Example}
A fair six-faced dice roll can be modeled with
\(\Omega = \{1,2,3,4,5,6\}\) and \(\Pr(\{1\}) = \Pr(\{2\}) = \Pr(\{3\}) = \Pr(\{4\}) = \Pr(\{5\}) = \Pr(\{6\}) = 1/6\).
The probability of rolling an odd face is \(\Pr(\{1,3,5\}) = \Pr(\{1\}) + \Pr(\{3\}) + \Pr(\{5\}) = 3/6\).

\section{Density}

\index{density of a measure}%
The \emph{density of the measure \(m : 2^U \to \Real\)}
is the \(f : U \to \Real\) where \(\forall A \subset U : \int_A f = m(A)\).

\index{probability density function}%
A \emph{probability density function} (pdf) is a function \(f : \Real \to \Real\) where \(\int_\Real f = 1\).

Let \((\Omega,\Pr)\) be a probability space.
The density of \(\Pr\) is a pdf.
The density of that space is the density of its measure.

\section{Distribution}

\index{distribution function}%
\index{cumulative distribution function}%
The \emph{distribution function} of \(\Pr:2^\Real\to\Real\) is \(F(a) = \Pr(\{ x~|~x \le a\})\),
which is also called the \emph{cumulative distribution function} of \(\Pr\).

Relationship between density and distribution:
Iff \(\Pr : 2^\Real \to \Real\),
\(f\) is the density of \(P\), and
\(F\) is the distribution of \(P\),
then \(F(a) = \int_{(-\infty,a]} f\).
We say that \(f\) is the pdf of \(F\).

\section{Conditional probability}

\index{conditional probability}%
\index{probability!conditional}%
The \emph{conditional probability of \(R\) given \(C\)}
is \(\Pr(R|C) = \Pr(R \cap C) / \Pr(C)\).

\(C\)-conditionalization maps \((\Omega,\Pr)\) to \((C,Q)\)
where \(Q(R) = \Pr(R|C)\).

Let \(f\) be the density of \(\Pr\).
Recall that \(\Pr(A) = \int_A f\).
Therefore \(\Pr(R|C) = \Pr(R \cap C)/\Pr(C) = \int_{R \cap C} f / \int_C f\).

\index{conditional density}%
\index{density!conditional}%
The conditional density \(f(r|C)\) satisfies \(\int_R (r \to f(r|C)) = \Pr(R|C) = Q(R)\).
\(\int_{R \cap C} f / \int_C f\).
The right side of the vertical bar is a set.

\index{conditional distribution}%
\index{distribution!conditional}%
The conditional distribution
\(F(r \le a|C) = \int_{(-\infty,a]} (r \to f(r|C))\).

\section{Random variable}

\index{random variable}%
\index{random variable!distribution of}%
\index{distribution!of random variable}%
A \emph{random variable} \(X : \Omega \to V\)
is a measurable function.
\index{real random variable}%
\index{random variable!real}%
Iff \(V = \Real\), then \(X\) is a \emph{real random variable}.
\index{probability space!of random variable}%
Such \(X\) maps
\((\Omega,\Pr)\) to \((V,Q)\)
where \(Q(A) = \Pr(\{\omega~|~X(\omega)\in A\}) = \Pr(X\in A)\).
Note that \(\Pr(X \in A)\) is an abuse of notation.

The \emph{distribution of \(X\)} is the distribution of \(Q\),
that is \(F(a) = \Pr(X \le a)\).

\(X \sim D\) means that \emph{the distribution of \(X\) is \(D\)}.

\index{probability mass function}%
A \emph{probability mass function} (pmf) \(f : \Omega \to \Real\)
is a function where \(\forall A \subseteq \Omega : \sum_{x \in A} f(x) = \Pr(A)\).
\index{probability density function}%
A \emph{probability density function} (pdf) \(f : \Omega \to \Real\)
is a function where \(\forall A \subseteq \Omega : \int_A f = \Pr(A)\).
A statement about pdfs can usually be translated into
a statement about pmfs by replacing integral with summation.

If \(A \subseteq \Real\), then a pmf \(f : A \to \Real\)
can be turned into a pdf \(g : \Real \to \Real\)
where
\(g(x) = \sum_{c \in A} f(c) \cdot \delta(x-c)\)
where
\index{Dirac delta function}%
\index{function!Dirac delta}%
\(\delta\) is the \emph{Dirac delta function}
where
\( \int_X \delta = [0 \in X] \)
where \([F]\) is 0 iff \(F\) is false and 1 iff \(F\) is true.

\paragraph{Iid}
Two random variables \(X\) and \(Y\) are
\index{random variables!independent}%
\index{independent random variables}%
\emph{independent} iff \(\forall A, B : \Pr(X \in A \wedge Y \in B) = \Pr(X \in A) \cdot \Pr(Y \in B)\),
\index{random variables!identically distributed}%
\index{identically-distributed random variables}%
\emph{identically distributed} iff
\(\forall A : \Pr(X \in A) = \Pr(Y \in A)\),
\index{random variables!independent and identically-distributed}%
\index{independent and identically-distributed random variables}%
\index{random variables!iid}%
\index{iid!random variables}%
\emph{iid} iff they are independent and identically-distributed.

\paragraph{Example}
If \(\Omega\) is the set of all people on Earth,
then \(X(\omega)\) might be the age of the person \(\omega\) in years.
If \(\Omega\) is the set of all cookies produced by a machine,
then \(X(\omega)\) might be the sugar amount in the cookie \(\omega\) in grams.
If \(\Omega\) is the set of all tasks,
then \(X(\omega)\) might be the time required to finish the task \(\omega\) in seconds.

\section{Joint and marginal}

\index{joint probability space}%
\index{probability space!joint}%
\index{probability!joint}%
Let there be \(n\) probability spaces \(S_k = (\Omega_k,\Pr_k)\).
The \emph{joint probability space} of \(S_1,\ldots,S_n\) is
\(S = (\Omega,\Pr)\)
where \(\Omega = \prod_{k=1}^n \Omega_k\).

Define the set \(W_k(A) = \{ \omega ~|~ \omega \in \Omega, ~ \omega_k \in A \}\).
Relationship: \(\Pr_k(A) = \Pr(W_k(A))\).
We say that \(\Pr_k\) is obtained by \emph{retaining} the \(k\)th component of \(S\).

Let \(N = \{1,\ldots,n\}\) be the set of the indexes of \(S\)
and let \(K \subseteq N\) be the set of the indexes we want to retain.
The result of
\index{retain}%
\emph{retaining} \(K\) in \(S\)
is \((\Omega,\Pr_K)\) where
\(\Pr_K(A) = \Pr\left(\bigcap_{k \in K} W_k(A_k)\right)\)
where \(A_k = \{ a_k ~|~ a \in A \} \).
\index{marginalize out}%
To \emph{marginalize out} \(K\) from \(S\) is to retain \(N-K\) in \(S\).
Note the relationship among ``retain'', ``marginalize out'', and ``all but'':
To retain \(x\) is to marginalize out all but \(x\),
and to marginalize out \(x\) is to retain all but \(x\).

\index{marginal probability space}%
\index{probability space!marginal}%
\index{probability!marginal}%
Marginal probability space?
Example?

Many random variables:
The notation \(\Pr(X_1\in A_1 \wedge \ldots \wedge X_n\in A_n)\) means
\(\Pr(\{(\omega_1,\ldots,\omega_n) ~|~ X_1(\omega_1) \in A_1, ~ \ldots, ~ X_n(\omega_n) \in A_n\})\)
where \(\Omega = \prod_{k=1}^n \Omega_k\)
and \(\forall k (X_k : \Omega_k \to V_k)\).
Such \((\Omega,\Pr)\) is a \emph{joint probability space}.

\section{Central tendency}

\index{average!weighted}%
\index{weighted average}%
The \emph{weighted average} of \(f:\Real\to\Real\) is
\(w(f) = \int_\Real (x \to f(x) \cdot x) / \int_\Real f\)
where \(\int_\Real f\) is the
\index{normalizing constant}%
\index{constant!normalizing}%
\emph{normalizing constant},
which is 1 iff \(f\) is a pdf.

\index{mean!of a pdf}%
The \emph{mean} of a pdf \(f\) is \(\mu(f) = w(f)\),
\index{mean!of a distribution}%
of a distribution \(D\) whose pdf is \(f\) is \(\mu(D) = \mu(f)\),
\index{mean!of a probability measure}%
of a probability measure \(\Pr\) whose distribution is \(D\) is
\(\mu(\Pr) = \mu(D)\),
\index{mean!of a probability space}%
of a probability space \(S = (\Real,\Pr)\) is \(\mu(S) = \mu(\Pr)\).

\index{expectation}%
The \emph{expectation} of a random variable \(X \sim D\) is \(\Expect{X} = \mu(D)\).
\index{linearity of expectation}%
\index{expectation!linearity of}%
Expectation is linear:
\(\Expect{aX} = a \Expect{X}\) and
\(\Expect{X+Y} = \Expect{X} + \Expect{Y}\).

\index{moment}%
The \emph{\(k\)th moment of \(X\)} is \(m_k(X) = \Expect{X^k}\)
where superscript means power and not composition.
Expectation is first moment.

\section{Dispersion tendency}

\index{central moment}%
The \emph{\(k\)th central moment of \(X\)} is \(\mu_k(X) = \Expect{(X - \Expect{X})^k}\).

\index{variance}%
\emph{Variance} is the second central moment.
The variance of \(X\) is \(\sigma_X^2 = \mu_2(X)\).

\index{standard deviation}%
\emph{Standard deviation} is square root of variance.

\index{skewness}%
\emph{Skewness} is the third central moment.

\section{Mixed moments}

Covariance is \(Cov(X,Y) = \Expect{(X-\Expect{X})(Y-\Expect{Y})}\).
Correlation is \(Cor(X,Y) = Cov(X,Y) / (\sigma_X \sigma_Y)\).

\section{Bernoulli trial}

\index{Bernoulli trial}%
A \emph{Bernoulli trial}
is a probability space \((\Omega,\Pr)\) where \(\Omega = \{0,1\}\),
\(\Pr(\{0\}) = 1 - p\), and \(\Pr(\{1\}) = p\).

\section{Bernoulli distribution}

\index{distribution!Bernoulli}%
\index{Bernoulli distribution}%
The pmf of \(\Bernoulli(p)\) (the \emph{Bernoulli distribution with parameter \(p\)})
is \(f(0) = 1 - p\) and \(f(1) = p\),
which can also be written \(f(k) = (1-p)^{1-k} p^k\) where \(k \in \{0,1\}\).

\section{Binomial distribution}

Bernoulli distribution is a special case of
\emph{binomial distribution} \(\Binomial(n,k,p)\) whose pmf is
\(f(k) = \binom{n}{k} (1-p)^k p^{n-k}\) where \(k : \Nat, k \le n\) with \(n=1\).
\(\Binomial(n,k,p)\) is the distribution of \(n\) iid instances of a Bernoulli trial.

The \emph{binomial formula} is \(\binom{n}{k} = \frac{n!}{k!(n-k)!}\).

\(\binom{n}{k}\) is the number of all \(k\)-sized subsets of an \(n\)-sized set.
The cardinality of \(\{ S ~|~ S \subseteq A, |S| = k \}\) is \(\binom{n}{k}\)
where \(|A| = n\).

The binomial formula is involved in the expansion \((x+y)^n = \sum_{k=0}^n \binom{n}{k} x^k y^{n-k}\).

\section{Multivariate random variable}

\index{random variable!multivariate}%
\index{multivariate random variable}%
\index{random vector}%
\index{random!vector}%
\index{vector!random}%
A \emph{multivariate random variable} or \emph{random vector}
is a vector of random variables,
or a vector-valued random variable,
or a random variable whose sample space is a vector space.

Define \(\{ X < a \}\) as \(\{ x ~|~ x \in u, ~ X~x < a \}\).
If \(f\) is a bijection and \(Y = f~X\),
then \(P~\{f~X < a\} = P~\{X < f^{-1}~a\}\).

Algebra on random variables:
If \(k : a\) and \(X : RV~a\) then \(k X : RV~a\).
If \(X : RV~a\) and \(Y : RV~a\) then \(X+Y : RV~a\).

\index{population}%
\index{population!to sample a}%
\index{population!sample space of}%
\index{population!probability space}%
\index{sample (noun)}%
\index{sample (verb)!a population}%
\index{observation}%
\index{unit!statistical}%
A \emph{population} is a probability space.
Let \(p\) be a population and \(s\) be its sample space.
A \emph{sample} of \(p\) is a subset of \(s\).
To \emph{sample} \(p\) is to pick a subset of \(s\).
An \emph{observation} is a member of \(s\).
A \emph{unit} is an observation.

\section{What are these rules for?}

\index{chain rule}%
\index{probability!chain rule}%
\index{conditional probability!chain rule}%
The \emph{chain rule} is \(\Pr(a \cap b) = \Pr(b|a) \cdot \Pr(a) = \Pr(a|b) \cdot \Pr(b)\).

\index{law of total probability}%
\index{probability!law of total probability}%
\index{conditional probability!law of total probability}%
The \emph{law of total probability} is \(\Pr(b) = \sum_{k=1}^n \Pr(b|a_k) \cdot \Pr(a_k) \)
where \(a_1,\ldots,a_n\) is a partitioning of \(a\).

\index{Bayes rule}%
The \emph{Bayes rule} is \(\Pr(a|b) = \Pr(b|a) \cdot \Pr(a)\)
where \(a_1,\ldots,a_n\) is a partitioning of \(a\).

\section{Materials looking for place to belong}

\index{Gaussian distribution}%
\index{normal distribution}%
\index{distribution!Gaussian}%
\index{distribution!normal}%
\index{Gaussian probability density function}%
\index{normal probability density function}%
\index{probability density function!Gaussian}%
\index{probability density function!normal}%
The \emph{Gaussian pdf} or the \emph{normal pdf}
with mean \(\mu\) and variance \(\sigma^2\) is
\(\frac{1}{\sqrt{2\sigma^2\pi}}\exp\left(-\frac{(x-\mu)^2}{2\sigma^2}\right)\).

\index{Poisson distribution}%
\index{Chi-square distribution}%
\index{Cauchy distribution}%
\index{Beta distribution}%
The \emph{Poisson distribution}...
The \emph{Chi-square distribution}...
The \emph{Cauchy distribution}...
The \emph{Beta distribution}...

\section{Random process}

The type of a \emph{random process} is \(\Nat \to \Omega \to R\).

\emph{Random process} is also called \emph{stochastic process}.

\section{Wiener process}


\chapter{Strong Induction}
\label{chapter:strong-induction}
Sometimes $P(k)$ is not enough to prove $P(k + 1)$ and we need all the
statements $P(1)$, \dots, $P(k)$. In this case we may use the following
induction  principle.

\begin{theorem}[The Strong Induction Principle]
\label{theorem:strong-induction}
    Let $P(n)$ be some statement about positive integer $n$.
    Hence, $P(n)$ is true for every integer $n > n_0$ iff
    \begin{description}
        \item [(the base case)] $P(n_0 + 1)$ is true and
        \item [(the induction step)] If $P(n_0 + 1)$, \dots, $P(n_0 + k)$ are true,
            then $P(n_0 + k + 1)$ is also true for all positive integers $k$.
  \end{description}
\end{theorem}

Before we prove this theorem let us present some applications of this
principle.

The Fibonacci numbers are defined as follows:
$f_0 = 0$, $f_1 = 1$, and $f_k = f_{k - 1} + f_{k - 2}$ for $k \ge 2$ (note
that they are also defined using strong induction since we use not only
$f_{k - 1}$ to define $f_k$).\footnote{%
  Fibonacci numbers are named after Italian mathematician Leonardo of Pisa,
  later known as Fibonacci. In his 1202 book ``Liber Abaci'', Fibonacci
  introduced the sequence to Western European mathematics. However, the sequence
  had been described as early as 200 BC in work by Indian mathematician Pingala
  on enumerating possible patterns of Sanskrit poetry formed from syllables of
  two lengths.

  Fibonacci numbers appear unexpectedly often in mathematics, so much so that
  there is an entire journal dedicated to their study, the ``Fibonacci
  Quarterly''. Applications of Fibonacci numbers include computer algorithms
  such as the Fibonacci search technique and the Fibonacci heap data structure,
  and graphs called Fibonacci cubes used for interconnecting parallel and
  distributed systems.
}

\begin{theorem}[The Binet formula]
    The Fibonacci numbers are given by the following formula
    \[
        f_n = \frac{\alpha^n - \beta^n}{\sqrt{5}},
    \]
    where $\alpha = \frac{1 + \sqrt{5}}{2}$ and
    $\beta = \frac{1 - \sqrt{5}}{2}$.
\end{theorem}
\begin{proof}
    We use the strong induction principle to prove this statement with $n_0 = -1$.
    Let us first prove the base case,
    $\frac{(\alpha^0 - \beta^0)}{\sqrt{5}} = 0 = f_0$.
    We also need to prove the induction step.
    \begin{itemize}
        \item If $k = 1$, then $\frac{(\alpha^1 - \beta^1)}{\sqrt{5}} = 1 = f_1$.
        \item Otherwise, by the induction hypothesis,
            $f_k = \frac{\alpha^k - \beta^k}{\sqrt{5}}$ and
            $f_{k - 1} = \frac{\alpha^{k - 1} - \beta^{k - 1}}{\sqrt{5}}$.
            By the definition of the Fibonacci numbers $f_{k + 1} = f_k + f_{k - 1}$.
            Hence,
            \[
                f_{k + 1} = \frac{\alpha^k - \beta^k}{\sqrt{5}} +
                \frac{\alpha^{k - 1} - \beta^{k - 1}}{\sqrt{5}}.
            \]
            Note that it is enough to show that
            \begin{equation}
            \label{equation:binet}
                \frac{\alpha^{k + 1} - \beta^{k + 1}}{\sqrt{5}} =
                \frac{\alpha^k - \beta^k}{\sqrt{5}} +
                \frac{\alpha^{k - 1} - \beta^{k - 1}}{\sqrt{5}}.
            \end{equation}
            Note that it is the same as
            \[
                \frac{\alpha^{k + 1} - \alpha^k - \alpha^{k - 1}}{\sqrt{5}} =
                \frac{\beta^{k + 1} - \beta^k - \beta^{k - 1}}{\sqrt{5}}.
            \]
            Additionally, note that $\alpha$ and $\beta$ are roots of the equation
            $x^2 - x - 1 = 0$. Hence,
            $\alpha^{k + 1} - \alpha^k - \alpha^{k - 1} = \alpha^{k - 1}(\alpha^2 -
            \alpha - 1) = 0$ and
            $\beta^{k + 1} - \beta^k - \beta^{k - 1} =
            \beta^{k - 1}(\beta^2 - \beta - 1) = 0$. Which implies
            equality~(\ref{equation:binet}).
    \end{itemize}
\end{proof}

\begin{template}
  \textbf{Template for proving a statement using strong induction.} \\

  We use induction by $n$. Base case for $n = n_0 + 1$: \emph{present some
  argument that proves the statement with $n$ replaced by $1$}.

  Now we need to prove the induction step from $n_0 + 1$, \dots, $n_0 + k - 1$
  to $k$. Let us assume now that the statement is true for any $n$ from $n_0 +
  1$ to $n_0 + k - 1$ for some $k$. \emph{Present some argument of the statement
  with $n$ replaced by $n_0 + k$ assuming the statement is true if we replace
  $n$ by $n_0 + \ell$ for $1 \le \ell < k - 1$}. Hence,
  the statement is true for all $n$ by the induction principle.
\end{template}

Another example of an application of the strong induction is the proof that
any number can be written in digital numeral systems with any base.
\begin{theorem}
  Let $b > 1$ be an integer. Then there is a unique representation
  $(c_0, \dots, c_\ell)_b$ of any positive number $n$ in the base-$b$ digital
  numeral system.
  In other words, for any positive integer $n$, there are unique
  $0 \le c_0, \dots, c_\ell < b$ such that $n = \sum_{i = 0}^\ell b^i c_i$.
\end{theorem}
\begin{proof}
  We prove the statement using strong induction by $n$. The base case for
  $n < b$ is clear (we can choose $\ell = 0$ and $c_0 = n$).
  Let us now prove the induction step. Assume the statement is true for all
  $k < n$. Let $n$ divided by $b$ be equal to $q$ with the remainder $c_0$.
  Note that $(n - c_0) / b < n$ is a positive integer. Hence,
  by the induction hypothesis, there are
  $0 \le c_1, \dots, c_\ell < b$
  such that $(n - c_0) / b = \sum_{i = 1}^\ell b^{i - 1} c_i$. Hence,
  $n = \sum_{i = 0}^\ell b^i c_i$.
\end{proof}
\nomenclature[U]{$(c_0, \dots, c_\ell)_b$}{%
  denotes the number $n = \sum_{i = 0}^\ell b^i c_i$; i.e., $c_0, \dots, c_\ell$
  are the digits of $n$ in the base-$b$ digital representation}

Now we are ready to prove the strong induction principle.
\begin{proof}[Proof of Theorem~\ref{theorem:strong-induction}]
  It is easy to see that if $P(n)$ is true for all $n > n_0$, then the base
  case and the induction steps are true. Let us prove that if the base case and
  the induction step are true, then $P(n)$ is true for all $n > n_0$.

  Let $Q(k)$ be the statement that $P(n_0 + 1)$, \dots, $P(n_0 + k)$ are true.
  Note that $Q(1)$ is true by the base case for $P$. Additionally, note that if
  $Q(k)$ is true, then $Q(k + 1)$ is also true, by the induction step for $P$.
  Hence, by the induction principle, $Q(k)$ is true for all positive integers
  $k$. Which implies that $P(n_0 + k)$ is true for all positive integers $k$.
\end{proof}

\section{Analysis of Recursive Algorithms}
\label{section:strong-induction-recursive}
To illustrate the power of recursive definitions and strong induction, consider
following game: Alice has chosen a number from $1$ to $1000$. Bob wants to
guess the number so he is asking Alice ``yes'' or ``no'' questions.
How many questions does Bob need to ask to determine the number in the
worst-case scenario?

The following simple algorithm allows Bob to learn the number using $10$
questions.

\begin{enumerate}
  \item Bob start with two numbers $\ell = 0$ and $u = 1000$.
  \item \label{algorithm-step:binary-search-result-guess-number}
    If there is only one integer $x$ such that $\ell < x \le u$, Bob says that
    Alice's number is $x$ and terminates the algorithm.
  \item \label{algorithm-step:binary-search-decision-guess-number}
    Bob asks whether the Alice's number is at most $(\ell + u) / 2$. If the
    answer is yes, then Bob replaces $u$ by $(\ell + u) / 2$; otherwise Bob
    replaces $\ell$ by $(\ell + u) / 2$.
    Bob goes to step~\ref{algorithm-step:binary-search-decision-guess-number}.
\end{enumerate}

We need to prove that Bob's algorithm is correct and that it makes at most $10$
questions. We prove a bit stronger statement. If on
step~\ref{algorithm-step:binary-search-decision-guess-number} $u - \ell < n$ and
Alice's number is between $\ell$ and $u$ ($u$ and $\ell$ are some reals), then
the algorithm returns Alice's number using at most $\log_2 n$ questions
($\log_2 1000 \le 10$). We are going to use strong induction by $n$.

The base case is clear since if $u - \ell < 1$ there is at most one integer
between $u$ and $\ell$ and by the assumption there is Alice's number between
$\ell$ and $u$. Hence, Bob is going to guess Alice's number correctly without asking
any questions.

Assume that the statement is true for all $m < n$. Note that
$\ell < (\ell + u) / 2 < u$ since $n \ge 2$. Therefore Alice's number is either
between $\ell$ and $(\ell + u) / 2$ or between $(\ell + u) / 2$ and $u$. Hence,
we go to step~\ref{algorithm-step:binary-search-decision-guess-number} with new
$\ell'$ and $u'$ such that $u' - \ell' < n / 2$ and Alice's number is between
$\ell'$ and $u'$. As a result, by the induction hypothesis, Bob is going to
guess Alice's number correctly using at most $1 + \log_2 (n / 2) = \log_2 n$
questions.

\begin{chapterendexercises}
    \exercise Let $a_0 = 2$, $a_1 = 5$, and $a_n = 5a_{n - 1} - 6 a_{n - 2}$
        for all integers $n \ge 2$. Show that $a_n = 3^n + 2^n$ for all integers
        $n \ge 0$.
        \begin{solution}
            We prove this using induction by $n$. The base case for $n \le 1$ is clear
            since $3^0 + 2^0 = 2$ and $3^1 + 2^1 = 5$.

            Let us prove the induction step. Assume that $a_n = 3^n + 2^n$ and
            $a_{n - 1} = 3^{n - 1} + 2^{n - 1}$, we need to prove that
            $a_{n + 1} = 3^{n + 1} + 2^{n + 1}$. Note that
            \begin{multline*}
                a_{n + 1} = 5a_n - 6 a_{n - 1} =
                5 \cdot 3^n + 5 \cdot 2^n - 6 \cdot 3^{n - 1} -
                6 \cdot 2^{n - 1} = \\
                3^{n - 1} \cdot 9 + 2^{n - 1} 4 = 3^{n + 1} + 2^{n + 1}.
            \end{multline*}
        \end{solution}
    \exercise Let $f_0 = 1$, $f_1 = 1$, and $f_{n + 2} = f_{n + 1} + f_n$ for
        all integers $n \ge 0$. Show that
        $f_n \ge \left(\frac{3}{2}\right)^{n - 2}$.
    \exercise Show that $f_{n + m} = f_{n - 1} f_{m - 1} + f_n f_m$.
    \exercise Give a nonadaptive algorithm for Bob that allows him to guess the
      number using $10$ queries. In other words, write $10$ questions such that
      answers to these questions allow Bob to guess the number.
    \exercise Show that \Cref{algorithm:binary-search} makes at most
      $6 + 2\log_2(n)$ comparisons.
      \begin{algorithm}
        \begin{algorithmic}[1]
          \Function{BinarySearch}{$e$, $a_1$, \dots, $a_n$}
            \If{$n \le 5$}
              \For{$i$ from $1$ to $n$}
                \If{$a_i = e$}
                  \State\Return{i}
                 \EndIf
              \EndFor
            \Else
              \State{$\ell \gets \floor{\frac{n}{2}}$}
              \If{$a_\ell \le e$}
                \State\Call{BinarySearch}{$e$, $a_1$, \dots, $a_\ell$}
              \Else
                \State\Call{BinarySearch}{$e$, $a_{\ell + 1}$, \dots, $a_n$}
              \EndIf
            \EndIf
          \EndFunction
        \end{algorithmic}
        \caption{The binary search algorithm that finds an element $e$ in the sorted
          list $a_1$, \dots, $a_n$.}
        \label{algorithm:binary-search}
      \end{algorithm}
\end{chapterendexercises}


\section{Additional comments}
\paragraph{\textbf{Remarks}}
\begin{enumerate}

	\item $k$ disjoint sets $A_1,A_2,\cdots,A_k$ represent $k$ classes. We use $A$ to denote
	\[
	A = A_1\cup A_2\cup \cdots \cup A_k.
	\]
	Naturally we can define the standard decision mapping $\bm{\chi}:A \rightarrow \{e_1,e_2,\cdots,e_k\}\subset \mathbb{R}^k$  as
	\begin{equation}
	\bm{\chi}(x) = e_i,\ x\in A_i,
	\end{equation}
	which is just like a characteristic function.

	\item To solve a classification problem, our goal is to find the standard decision mapping. But usually it's unpractical, because we seem to be better at dealing with continuous functions with continuous outputs. A natural idea is to reach the goal in two steps. We can use a composition of the following two mappings
	$$\bm{f}:A \rightarrow \mathbb{R}^k;\ x\mapsto y.$$
	$$\bm{\pi}: \mathbb{R}^k \rightarrow \{e_1,e_2,\cdots,e_k\};\ \ y\mapsto e_i,i = \argmax_j~y_j$$
	to obtain the standard decision mapping. Notice that the mapping $\bm{\pi}$ is fixed, so we only need to find a proper mapping $\bm{f}$.

	\item Assume that we want to find a proper $\bm{f}$ in a mapping sapce $\mathscr{H}$. The result of a classification problem depends on which $\mathscr{H}$ to choose and how to find the optimal mapping $\bm{f}$ in $\mathscr{H}$.\\

	\item Given a mapping space $\mathscr{H}$, if there exists an $\mathbf{f}\in \mathscr{H}$ such that
	\begin{equation}
	\bm{\chi} = \bm{\pi} \circ \bm{f}.
	\end{equation}
	we can say that $A_1,A_2,\cdots,A_k$ are $\mathscr{H}$-separable. Setting $\mathscr{H}$ to be the affine mapping space, $i.e.$ $\mathscr{H} = \{\bm{f}: \bm{f}(x) = Wx+b\}$, we obtain the definition of linearly separable.\\

	\item A linear model  is a method to find the optimal $W$ and $b$ when we take $\mathscr{H} = \{\bm{f}: \bm{f}(x) = Wx+b\}$.\\

	\item Assume that $\bm{\phi}$ is a feature mapping from $A$ to a feature space $\mathcal{H}$. A natural choice of $\mathscr{H}$ is
	\begin{equation}
	\mathscr{H} = \{\bm{f}: \bm{f}(x) = W\bm{\phi}(x)+b\}
	\end{equation}
	%If we set $\mathcal{H}$ to be a Reproduced Kernel Hilbert Space $w.r.t.$ a kernel function $\bm{k}$ and use SVM to find the optimal $W$ and $b$, we can obtain the methods of kernel SVM.
\end{enumerate}

\subsection{General approach for separation }
A collection of sets $A_i\subset \mathbb{R}^n$, $1\le i\le k$, are said to be separable if there exists a continuous function $h:\mathbb{R}^n\to\mathbb{R}^k$ such that $h(x)=e_i$.

A general approach:
\begin{itemize}
	\item Find a ``nonlinear" function $h_0:\mathbb{R}^n\to\mathbb{R}^m$ such that
\begin{enumerate}
	\item $\tilde{A}_i=h_0(A_i)\subset\mathbb{R}^m$ are linearly separable
	\item $m\ll n$
\end{enumerate}
\item Use a ``linear" classifier $l_0:\mathbb{R}^m\to\mathbb{R}^k$, then $h=l_0\circ h_0:\mathbb{R}^n\to\mathbb{R}^k$ is the classifier.
\end{itemize}

\begin{remark}
	\begin{enumerate}
		\item $h_0$: dimension reduction;
		\item Feature extraction;
		\item $h_0$: deep neural network, special function class;
		\item $x^{l+1}=\sigma(Wx^l+b)$, for CNN, $W,b$ are ``sparse";
		\item ``linear classifier" is fully connected layer.
	\end{enumerate}
	\end{remark}

\subsection{Feature map}

\begin{remark}
The "linear" here means that $wx+b$ is linear with respect to $x$. This condition is not crucial. Actually, we can replace $x$ by $\phi: \mathbb{R}^n\rightarrow \mathbb{R}^m$. To be specific, let  $\tilde x=w\phi(x)+b$ and $\tilde A_i=\phi (A_i)\in \mathbb{R}^m$. The function $\phi$ here is a feature map,
common choices include
\begin{enumerate}
\item $\phi(x)=x$;
\item finite elements, wavelets;
\item deep neural networks.
\end{enumerate}
\end{remark}

\input{6DL/LinearSets}


%================================
\section{More on Separation Axioms}
%================================

%--------------------------------
\begin{definition}
	[saperated sets]
	Let $(X, \mathcal T)$ be a topological space, and let $A, B \in \mathcal P(X)$.

	\begin{enumerate}[(i)]
		\item $A$ and $B$ are said to be \textit{separated} iff each is disjoint from other's closure.
		\item $A$ and $B$ are said to be \textit{separated by neighbourhoods} iff there are neighbourhoods $N_A$ of $A$ and $N_B$ of $B$ such that $N_A$ and $N_B$ are disjoint.
		\item $A$ and $B$ are said to be \textit{separated by closed neighbourhoods} iff there are closed neighbourhoods $\overline N_A$ of $A$ and $\overline N_B$ of $B$ such that $\overline N_A$ and $\overline N_B$ are disjoint.
		\item $A$ and $B$ are said to be \textit{separated by a continuous function} iff there is a continuous function $f: X \to \mathbb R$, such that $f[A] = \{0\}$ and $f[B] = \{1\}$.
		\item $A$ and $B$ are said to be \textit{precisely separated by a continuous function} iff there is a continuous function $f: X \to \mathbb R$, such that $f^{-1}[\{0\}] = A$ and $f^{-1}[\{1\}] = B$
	\end{enumerate}
\end{definition}
%--------------------------------

%--------------------------------
\begin{definition}
	[$T_{2 \nicefrac{1}{2}}$ spaces]
	\label{def: T_2.5 spaces}
	A topological space $(X, \mathcal T)$ is said to be $T_{2 \nicefrac{1}{2}}$ or \textit{Urysohn} iff two distinct points in $X$ are separated by closed neighbourhoods.
\end{definition}
%--------------------------------

%--------------------------------
\begin{example}
	[$T_2$ but not $T_{2 \nicefrac{1}{2}}$]
	\footnote{
		See \href{https://planetmath.org/hausdorffspacenotcompletelyhausdorff}{MathPlanet}.
	}
	(Remained as a problem)
	% todo: Remained as a problem!
\end{example}
%--------------------------------

%--------------------------------
\begin{definition}
	[$T_3$ spaces]
	\label{def: T_3 spaces}
	A topological space $(X, \mathcal T)$ is said to be $T_3$ or \textit{regular} iff it is $T_0$ and given any point $x \in (X, \mathcal T)$ and closed set $V \subseteq X$ with $x \notin V$ are separated by neighbourhoods.
\end{definition}
%--------------------------------

%--------------------------------
\begin{definition}
	[$T_{3\nicefrac{1}{2}}$ spaces]
	\label{def: T_3.5 spaces}
	A topological space $(X, \mathcal T)$ is said to be $T_{3 \nicefrac{1}{2}}$, or \textit{Tychonoff} or, \textit{completely $T_3$}, or \textit{completely regular}, iff it is $T_0$ and given any point $x$ and closed set $V \subseteq X$ with $x \notin V$, they are separated by a continuous function.
\end{definition}
%--------------------------------

%--------------------------------
\begin{definition}
	[$T_4$ spaces]
	\label{def: T_4 spaces}
	A topological space $(X, \mathcal T)$ is said to be $T_4$ or \textit{normal} iff it is Hausdorff and any tow disjoint closed subsets of $X$ are separated by neighbourhoods.
\end{definition}
%--------------------------------

%--------------------------------
\begin{proposition}
	[Urysohn's lemma]
	\label{prop: urysohn's lemma}
	A topological space is normal iff any two disjoint closed sets are separated by a continuous function.
\end{proposition}
%--------------------------------

%--------------------------------
\begin{definition}
	[$T_5$ spaces]
	\label{def: T_5 spaces}
	A topological space $(X, \mathcal T)$ is said to be $T_5$ or \textit{completely $T_4$} iff it is $T_1$ any two separated sets are separated by neighbourhoods.
\end{definition}
%--------------------------------

%--------------------------------
\begin{proposition}
	Every subspace of a $T_5$ space is normal.
\end{proposition}
%--------------------------------

%--------------------------------
\begin{definition}
	[$T_6$ spaces]
	\label{def: T_6 spaces}
	A topological space $(X, \mathcal T)$ is said to be $T_6$, or \textit{perfectly $T_4$} or \textit{perfectly normal} iff it is $T_1$ and any two disjoint closed sets are precisely separated by a continuous function.
\end{definition}
%--------------------------------

%--------------------------------
\begin{proposition}
	[Tietze extension theorem]
	\label{prop: Tietze extension theorem}
	Let $(X, \mathcal T)$ be normal topological space, and let $f: A \to (\mathbb R, \mathcal T')$ be a continuous map where $A$ is a closed subset of $X$ and $\mathcal T'$ is the standard topology (induced by Euclidean metric). Then there exists a continuous map
	$$
	F: (X, \mathcal T) \to (\mathbb R, \mathcal T'),
	$$
	such that
	$$
	\forall x \in A: f(x) = g(x).
	$$
\end{proposition}
%--------------------------------