\chapter{Appendix I: Quaternions}

There exists a correspondence between the orientation of a $ 3D $ object represented by a $ 3 \times 3 $ orthonormal matrix and unit quaternions in the group $ SO(3) $ \citep{Murray}. Quaternions give a global parametrization of $ SO(3) $, by using four numbers instead of three to represent a rotation. A quaternion is a vector quantity of the form,

  \[Q=q_{0} + q_{1}i + q_{2}j + q_{3}k  \ \ \ \ \ \ \ \  qi \in \R, i = 0, . . . , 3,\]

where, $ q_{0} $ is the scalar component of $ Q $ and $ \vec{q}= (q_{1}, q_{2}, q_{3}) $ is the vector component. A convenient notation is $ Q = (q_{0}, \vec{q}) $ with $ q_{0} \in \R $, $ \vec{q} \in \R^{3} $.
The set of quaternions Q is a 4-dimensional vector space over the reals and the unit quaternion form a group with respect to quaternion multiplication is a rotation group. Multiplication is distributive and associative, but not commutative.

For rotation through an angle $ \theta $ about the axis $ n $, the Euler parameters are defined as \citep{Hanson},
\begin{align*}
e_{0}&=\cos(\theta / 2) \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ e_{2}=\cos \beta \cdot\sin (\theta / 2)\\
e_{1}&=\cos \alpha \cdot\sin (\theta / 2) \ \ \ \ \ \ \ \ \ e_{3}=\cos \gamma \cdot\sin (\theta / 2)
\end{align*}
where, $ \alpha, \beta $ and $ \gamma $ are the elementary rotations about any given random axis.

Given a rotation matrix $ R=exp\left(\hat{\omega}\theta \right) $, we define the associated unit quaternion as
$ Q = \cos (\theta / 2) $,  $ \omega\sin (\theta / 2) $
where, $ \omega \in \R^{3} $ represents the unit axis of rotation and $ \theta \in \R^{3} $ represents the angle of rotation.
Given a unit quaternion $ Q = \left(q_{0}, \vec{q} \right) $, we can extract the corresponding rotation by setting,

\begin{center}
\ \ \ \ \ $\theta = 2\cos ^{-1} q_{0}, \ \ \ \ \ \ \
\omega = \left\lbrace
\begin{array}{ll}
\frac{\vec{q}}{\sin (\theta / 2)} & \mbox{if $ \theta \neq 0,$}\\
0 & \mbox{otherwise,}
\end{array}
\right.\
$
\end{center}

and  \[R = exp\left(\hat{\omega}\theta\right).\]

It is observed that the components of the quaternion are Euler parameters. The Euler parameters have a geometrical meaning and thus the same holds for the components of the corresponding unit quaternion.

The Frenet 3D coordinate frame can be expressed in the form of quaternions. Assuming that the columns of the equation given below are the vectors $ [t_{i}, n_{i}, b_{i}] $ respectively, $ [q'(t)] $ can be written in the form,
\begin{eqnarray}
\left[
\begin{array}{c}
q_{o}'\\
q_{1}'\\
q_{2}'\\
q_{3}'
\end{array}
\right]=
\dfrac{v}{2}
\left[
\begin{array}{cccc}
0 & -\tau & 0 & -\kappa \\
\tau & 0 & \kappa & 0 \\
0 & -\kappa & 0 & \tau \\
\kappa & 0 & -\tau & 0
\end{array}
\right]
\cdot
\left[
\begin{array}{c}
q_{o}\\
q_{1}\\
q_{2}\\
q_{3}
\end{array}
\right]
\label{13}
\end{eqnarray}
where $ v = \parallel \dot{c} \parallel $ and,
\begin{align*}
q_{o}= \cos (\theta / 2) = \dfrac{1}{\sqrt{2}} & \sqrt{\cos \theta +1} = \dfrac{1}{2} \sqrt{Trace (R) +1} \\
q_{1}=& \dfrac{R_{32}-R_{23}}{4q_{0}} \\
q_{2}=& \dfrac{R_{13}-R_{31}}{4q_{0}} \\
q_{3}=& \dfrac{R_{21}-R_{12}}{4q_{0}}
\end{align*}

Key properties of equation \ref{13} are \citep{Hanson},
\begin{enumerate}
\item The matrix on the right hand side is antisymmetric, so that $ q(t) \cdot q'(t)=0 $ is preserved and all unit quaternions remain unit quaternions as they are.
\item Nine coupled equations with six orthonormality constraints are reduced to four coupled equations with a single constraint of unit length.
\end{enumerate}

From the above equation \ref{13}, the $ curvature $,
\begin{equation}
\kappa = - \left[\dfrac{2(q_{3}q'_{0}+q_{1}q'_{2})}{v(q_{1}^{2}+q_{3}^{2})} \right]
\label{14}
\end{equation}
and the $ torsion $,
\begin{equation}
\tau = \dfrac{2q'_{2}+v\kappa q_{1}}{vq_{3}}
\label{15}
\end{equation}
can be extracted by the quaternion frame. The Frenet equations \ref{14} and \ref{15} may be integrated to generate a unique moving continuous frame along the protein backbone, for nonvanishing $ \kappa (t) $ with its space curve.

\section*{Motivation for using Quaternions}

\begin{enumerate}
\item It can be observed that the description of Euler angles and exponential co-ordinates suffers from the problem of $ \textit{singularity} $ in certain situations \citep{Hanson}. The extraction of parameters from any given rotational transformation matrix produces a singular case as the maps being many to one. While with the unit quaternion, extraction of unit quaternion from given rotational transformation being unique, there is no such case in which the inverse produces singularity. In quaternion space, the scalars, vectors, and quaternions are unified. Besides this, spatial vector can be expressed in quaternion space provides us with elegant properties for manipulating equations.

\item The quaternion frame summarizes nine matrix elements with six orthonormality constraints necessary to reduce the actual number of parameters of the frame to the three Euler angles. This forms a $ 3D $ orientation frame in terms of four quaternion frame variables with the single constraint of unit length that provides $ \textit{less computaional complexity} $.

\item The Frenet frame is periodic but not globally defined. As soon as the inflection points \citep{shuangwei} occur in the plane it is observed that the normal components of the Frenet frame switches sign instantly, while the quaternion frame has no such abrupt changes. The Frenet frame becomes undefined when the curvature $ \left(\kappa\right) $, vanishes along a straight line segment or at inflection point. It provides no such prescription to define a smooth transition from the frame coming and leaving the straight segment. The quaternion frame is comparatively $ \textit{smooth} $ throughout.
\end{enumerate}

% ------------------------------------------------------------------------

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "../thesis"
%%% End:


\chapter{Probability}

\section{Probability space}

\index{probability measure}%
\index{probability!measure}%
\index{measure!probability}%
A \emph{probability measure} \(\Pr : 2^\Omega \to \Real\) on a set \(\Omega\) is a measure where \(\Pr(\Omega) = 1\).

\index{probability space}%
\index{probability!space}%
\index{space!probability}%
A \emph{probability space} is a measure space \((\Omega,\Pr)\) where
\(\Omega\) is a set called the
\index{sample space}%
\index{sample!space}%
\index{space!sample}%
\emph{sample space}
and \(\Pr : 2^\Omega \to \Real\) is a probability measure.

\index{event}%
An \emph{event} is a subset of a sample space.
Relationship between two events \(A\) and \(B\):
They are
\index{events!equiprobable}%
\index{equiprobable events}%
\emph{equiprobable} iff \(\Pr(A) = \Pr(B)\),
\index{mutually exclusive events}%
\index{events!mutually exclusive}%
\emph{mutually exclusive} iff \(A \cap B = \emptyset\),
\index{independent events}%
\index{events!independent}%
\emph{independent} iff \(\Pr(A \cap B) = \Pr(A) \cdot \Pr(B)\).

\paragraph{Example}
A fair six-faced dice roll can be modeled with
\(\Omega = \{1,2,3,4,5,6\}\) and \(\Pr(\{1\}) = \Pr(\{2\}) = \Pr(\{3\}) = \Pr(\{4\}) = \Pr(\{5\}) = \Pr(\{6\}) = 1/6\).
The probability of rolling an odd face is \(\Pr(\{1,3,5\}) = \Pr(\{1\}) + \Pr(\{3\}) + \Pr(\{5\}) = 3/6\).

\section{Density}

\index{density of a measure}%
The \emph{density of the measure \(m : 2^U \to \Real\)}
is the \(f : U \to \Real\) where \(\forall A \subset U : \int_A f = m(A)\).

\index{probability density function}%
A \emph{probability density function} (pdf) is a function \(f : \Real \to \Real\) where \(\int_\Real f = 1\).

Let \((\Omega,\Pr)\) be a probability space.
The density of \(\Pr\) is a pdf.
The density of that space is the density of its measure.

\section{Distribution}

\index{distribution function}%
\index{cumulative distribution function}%
The \emph{distribution function} of \(\Pr:2^\Real\to\Real\) is \(F(a) = \Pr(\{ x~|~x \le a\})\),
which is also called the \emph{cumulative distribution function} of \(\Pr\).

Relationship between density and distribution:
Iff \(\Pr : 2^\Real \to \Real\),
\(f\) is the density of \(P\), and
\(F\) is the distribution of \(P\),
then \(F(a) = \int_{(-\infty,a]} f\).
We say that \(f\) is the pdf of \(F\).

\section{Conditional probability}

\index{conditional probability}%
\index{probability!conditional}%
The \emph{conditional probability of \(R\) given \(C\)}
is \(\Pr(R|C) = \Pr(R \cap C) / \Pr(C)\).

\(C\)-conditionalization maps \((\Omega,\Pr)\) to \((C,Q)\)
where \(Q(R) = \Pr(R|C)\).

Let \(f\) be the density of \(\Pr\).
Recall that \(\Pr(A) = \int_A f\).
Therefore \(\Pr(R|C) = \Pr(R \cap C)/\Pr(C) = \int_{R \cap C} f / \int_C f\).

\index{conditional density}%
\index{density!conditional}%
The conditional density \(f(r|C)\) satisfies \(\int_R (r \to f(r|C)) = \Pr(R|C) = Q(R)\).
\(\int_{R \cap C} f / \int_C f\).
The right side of the vertical bar is a set.

\index{conditional distribution}%
\index{distribution!conditional}%
The conditional distribution
\(F(r \le a|C) = \int_{(-\infty,a]} (r \to f(r|C))\).

\section{Random variable}

\index{random variable}%
\index{random variable!distribution of}%
\index{distribution!of random variable}%
A \emph{random variable} \(X : \Omega \to V\)
is a measurable function.
\index{real random variable}%
\index{random variable!real}%
Iff \(V = \Real\), then \(X\) is a \emph{real random variable}.
\index{probability space!of random variable}%
Such \(X\) maps
\((\Omega,\Pr)\) to \((V,Q)\)
where \(Q(A) = \Pr(\{\omega~|~X(\omega)\in A\}) = \Pr(X\in A)\).
Note that \(\Pr(X \in A)\) is an abuse of notation.

The \emph{distribution of \(X\)} is the distribution of \(Q\),
that is \(F(a) = \Pr(X \le a)\).

\(X \sim D\) means that \emph{the distribution of \(X\) is \(D\)}.

\index{probability mass function}%
A \emph{probability mass function} (pmf) \(f : \Omega \to \Real\)
is a function where \(\forall A \subseteq \Omega : \sum_{x \in A} f(x) = \Pr(A)\).
\index{probability density function}%
A \emph{probability density function} (pdf) \(f : \Omega \to \Real\)
is a function where \(\forall A \subseteq \Omega : \int_A f = \Pr(A)\).
A statement about pdfs can usually be translated into
a statement about pmfs by replacing integral with summation.

If \(A \subseteq \Real\), then a pmf \(f : A \to \Real\)
can be turned into a pdf \(g : \Real \to \Real\)
where
\(g(x) = \sum_{c \in A} f(c) \cdot \delta(x-c)\)
where
\index{Dirac delta function}%
\index{function!Dirac delta}%
\(\delta\) is the \emph{Dirac delta function}
where
\( \int_X \delta = [0 \in X] \)
where \([F]\) is 0 iff \(F\) is false and 1 iff \(F\) is true.

\paragraph{Iid}
Two random variables \(X\) and \(Y\) are
\index{random variables!independent}%
\index{independent random variables}%
\emph{independent} iff \(\forall A, B : \Pr(X \in A \wedge Y \in B) = \Pr(X \in A) \cdot \Pr(Y \in B)\),
\index{random variables!identically distributed}%
\index{identically-distributed random variables}%
\emph{identically distributed} iff
\(\forall A : \Pr(X \in A) = \Pr(Y \in A)\),
\index{random variables!independent and identically-distributed}%
\index{independent and identically-distributed random variables}%
\index{random variables!iid}%
\index{iid!random variables}%
\emph{iid} iff they are independent and identically-distributed.

\paragraph{Example}
If \(\Omega\) is the set of all people on Earth,
then \(X(\omega)\) might be the age of the person \(\omega\) in years.
If \(\Omega\) is the set of all cookies produced by a machine,
then \(X(\omega)\) might be the sugar amount in the cookie \(\omega\) in grams.
If \(\Omega\) is the set of all tasks,
then \(X(\omega)\) might be the time required to finish the task \(\omega\) in seconds.

\section{Joint and marginal}

\index{joint probability space}%
\index{probability space!joint}%
\index{probability!joint}%
Let there be \(n\) probability spaces \(S_k = (\Omega_k,\Pr_k)\).
The \emph{joint probability space} of \(S_1,\ldots,S_n\) is
\(S = (\Omega,\Pr)\)
where \(\Omega = \prod_{k=1}^n \Omega_k\).

Define the set \(W_k(A) = \{ \omega ~|~ \omega \in \Omega, ~ \omega_k \in A \}\).
Relationship: \(\Pr_k(A) = \Pr(W_k(A))\).
We say that \(\Pr_k\) is obtained by \emph{retaining} the \(k\)th component of \(S\).

Let \(N = \{1,\ldots,n\}\) be the set of the indexes of \(S\)
and let \(K \subseteq N\) be the set of the indexes we want to retain.
The result of
\index{retain}%
\emph{retaining} \(K\) in \(S\)
is \((\Omega,\Pr_K)\) where
\(\Pr_K(A) = \Pr\left(\bigcap_{k \in K} W_k(A_k)\right)\)
where \(A_k = \{ a_k ~|~ a \in A \} \).
\index{marginalize out}%
To \emph{marginalize out} \(K\) from \(S\) is to retain \(N-K\) in \(S\).
Note the relationship among ``retain'', ``marginalize out'', and ``all but'':
To retain \(x\) is to marginalize out all but \(x\),
and to marginalize out \(x\) is to retain all but \(x\).

\index{marginal probability space}%
\index{probability space!marginal}%
\index{probability!marginal}%
Marginal probability space?
Example?

Many random variables:
The notation \(\Pr(X_1\in A_1 \wedge \ldots \wedge X_n\in A_n)\) means
\(\Pr(\{(\omega_1,\ldots,\omega_n) ~|~ X_1(\omega_1) \in A_1, ~ \ldots, ~ X_n(\omega_n) \in A_n\})\)
where \(\Omega = \prod_{k=1}^n \Omega_k\)
and \(\forall k (X_k : \Omega_k \to V_k)\).
Such \((\Omega,\Pr)\) is a \emph{joint probability space}.

\section{Central tendency}

\index{average!weighted}%
\index{weighted average}%
The \emph{weighted average} of \(f:\Real\to\Real\) is
\(w(f) = \int_\Real (x \to f(x) \cdot x) / \int_\Real f\)
where \(\int_\Real f\) is the
\index{normalizing constant}%
\index{constant!normalizing}%
\emph{normalizing constant},
which is 1 iff \(f\) is a pdf.

\index{mean!of a pdf}%
The \emph{mean} of a pdf \(f\) is \(\mu(f) = w(f)\),
\index{mean!of a distribution}%
of a distribution \(D\) whose pdf is \(f\) is \(\mu(D) = \mu(f)\),
\index{mean!of a probability measure}%
of a probability measure \(\Pr\) whose distribution is \(D\) is
\(\mu(\Pr) = \mu(D)\),
\index{mean!of a probability space}%
of a probability space \(S = (\Real,\Pr)\) is \(\mu(S) = \mu(\Pr)\).

\index{expectation}%
The \emph{expectation} of a random variable \(X \sim D\) is \(\Expect{X} = \mu(D)\).
\index{linearity of expectation}%
\index{expectation!linearity of}%
Expectation is linear:
\(\Expect{aX} = a \Expect{X}\) and
\(\Expect{X+Y} = \Expect{X} + \Expect{Y}\).

\index{moment}%
The \emph{\(k\)th moment of \(X\)} is \(m_k(X) = \Expect{X^k}\)
where superscript means power and not composition.
Expectation is first moment.

\section{Dispersion tendency}

\index{central moment}%
The \emph{\(k\)th central moment of \(X\)} is \(\mu_k(X) = \Expect{(X - \Expect{X})^k}\).

\index{variance}%
\emph{Variance} is the second central moment.
The variance of \(X\) is \(\sigma_X^2 = \mu_2(X)\).

\index{standard deviation}%
\emph{Standard deviation} is square root of variance.

\index{skewness}%
\emph{Skewness} is the third central moment.

\section{Mixed moments}

Covariance is \(Cov(X,Y) = \Expect{(X-\Expect{X})(Y-\Expect{Y})}\).
Correlation is \(Cor(X,Y) = Cov(X,Y) / (\sigma_X \sigma_Y)\).

\section{Bernoulli trial}

\index{Bernoulli trial}%
A \emph{Bernoulli trial}
is a probability space \((\Omega,\Pr)\) where \(\Omega = \{0,1\}\),
\(\Pr(\{0\}) = 1 - p\), and \(\Pr(\{1\}) = p\).

\section{Bernoulli distribution}

\index{distribution!Bernoulli}%
\index{Bernoulli distribution}%
The pmf of \(\Bernoulli(p)\) (the \emph{Bernoulli distribution with parameter \(p\)})
is \(f(0) = 1 - p\) and \(f(1) = p\),
which can also be written \(f(k) = (1-p)^{1-k} p^k\) where \(k \in \{0,1\}\).

\section{Binomial distribution}

Bernoulli distribution is a special case of
\emph{binomial distribution} \(\Binomial(n,k,p)\) whose pmf is
\(f(k) = \binom{n}{k} (1-p)^k p^{n-k}\) where \(k : \Nat, k \le n\) with \(n=1\).
\(\Binomial(n,k,p)\) is the distribution of \(n\) iid instances of a Bernoulli trial.

The \emph{binomial formula} is \(\binom{n}{k} = \frac{n!}{k!(n-k)!}\).

\(\binom{n}{k}\) is the number of all \(k\)-sized subsets of an \(n\)-sized set.
The cardinality of \(\{ S ~|~ S \subseteq A, |S| = k \}\) is \(\binom{n}{k}\)
where \(|A| = n\).

The binomial formula is involved in the expansion \((x+y)^n = \sum_{k=0}^n \binom{n}{k} x^k y^{n-k}\).

\section{Multivariate random variable}

\index{random variable!multivariate}%
\index{multivariate random variable}%
\index{random vector}%
\index{random!vector}%
\index{vector!random}%
A \emph{multivariate random variable} or \emph{random vector}
is a vector of random variables,
or a vector-valued random variable,
or a random variable whose sample space is a vector space.

Define \(\{ X < a \}\) as \(\{ x ~|~ x \in u, ~ X~x < a \}\).
If \(f\) is a bijection and \(Y = f~X\),
then \(P~\{f~X < a\} = P~\{X < f^{-1}~a\}\).

Algebra on random variables:
If \(k : a\) and \(X : RV~a\) then \(k X : RV~a\).
If \(X : RV~a\) and \(Y : RV~a\) then \(X+Y : RV~a\).

\index{population}%
\index{population!to sample a}%
\index{population!sample space of}%
\index{population!probability space}%
\index{sample (noun)}%
\index{sample (verb)!a population}%
\index{observation}%
\index{unit!statistical}%
A \emph{population} is a probability space.
Let \(p\) be a population and \(s\) be its sample space.
A \emph{sample} of \(p\) is a subset of \(s\).
To \emph{sample} \(p\) is to pick a subset of \(s\).
An \emph{observation} is a member of \(s\).
A \emph{unit} is an observation.

\section{What are these rules for?}

\index{chain rule}%
\index{probability!chain rule}%
\index{conditional probability!chain rule}%
The \emph{chain rule} is \(\Pr(a \cap b) = \Pr(b|a) \cdot \Pr(a) = \Pr(a|b) \cdot \Pr(b)\).

\index{law of total probability}%
\index{probability!law of total probability}%
\index{conditional probability!law of total probability}%
The \emph{law of total probability} is \(\Pr(b) = \sum_{k=1}^n \Pr(b|a_k) \cdot \Pr(a_k) \)
where \(a_1,\ldots,a_n\) is a partitioning of \(a\).

\index{Bayes rule}%
The \emph{Bayes rule} is \(\Pr(a|b) = \Pr(b|a) \cdot \Pr(a)\)
where \(a_1,\ldots,a_n\) is a partitioning of \(a\).

\section{Materials looking for place to belong}

\index{Gaussian distribution}%
\index{normal distribution}%
\index{distribution!Gaussian}%
\index{distribution!normal}%
\index{Gaussian probability density function}%
\index{normal probability density function}%
\index{probability density function!Gaussian}%
\index{probability density function!normal}%
The \emph{Gaussian pdf} or the \emph{normal pdf}
with mean \(\mu\) and variance \(\sigma^2\) is
\(\frac{1}{\sqrt{2\sigma^2\pi}}\exp\left(-\frac{(x-\mu)^2}{2\sigma^2}\right)\).

\index{Poisson distribution}%
\index{Chi-square distribution}%
\index{Cauchy distribution}%
\index{Beta distribution}%
The \emph{Poisson distribution}...
The \emph{Chi-square distribution}...
The \emph{Cauchy distribution}...
The \emph{Beta distribution}...

\section{Random process}

The type of a \emph{random process} is \(\Nat \to \Omega \to R\).

\emph{Random process} is also called \emph{stochastic process}.

\section{Wiener process}


\chapter{Strong Induction}
\label{chapter:strong-induction}
Sometimes $P(k)$ is not enough to prove $P(k + 1)$ and we need all the
statements $P(1)$, \dots, $P(k)$. In this case we may use the following
induction  principle.

\begin{theorem}[The Strong Induction Principle]
\label{theorem:strong-induction}
    Let $P(n)$ be some statement about positive integer $n$.
    Hence, $P(n)$ is true for every integer $n > n_0$ iff
    \begin{description}
        \item [(the base case)] $P(n_0 + 1)$ is true and
        \item [(the induction step)] If $P(n_0 + 1)$, \dots, $P(n_0 + k)$ are true,
            then $P(n_0 + k + 1)$ is also true for all positive integers $k$.
  \end{description}
\end{theorem}

Before we prove this theorem let us present some applications of this
principle.

The Fibonacci numbers are defined as follows:
$f_0 = 0$, $f_1 = 1$, and $f_k = f_{k - 1} + f_{k - 2}$ for $k \ge 2$ (note
that they are also defined using strong induction since we use not only
$f_{k - 1}$ to define $f_k$).\footnote{%
  Fibonacci numbers are named after Italian mathematician Leonardo of Pisa,
  later known as Fibonacci. In his 1202 book ``Liber Abaci'', Fibonacci
  introduced the sequence to Western European mathematics. However, the sequence
  had been described as early as 200 BC in work by Indian mathematician Pingala
  on enumerating possible patterns of Sanskrit poetry formed from syllables of
  two lengths.

  Fibonacci numbers appear unexpectedly often in mathematics, so much so that
  there is an entire journal dedicated to their study, the ``Fibonacci
  Quarterly''. Applications of Fibonacci numbers include computer algorithms
  such as the Fibonacci search technique and the Fibonacci heap data structure,
  and graphs called Fibonacci cubes used for interconnecting parallel and
  distributed systems.
}

\begin{theorem}[The Binet formula]
    The Fibonacci numbers are given by the following formula
    \[
        f_n = \frac{\alpha^n - \beta^n}{\sqrt{5}},
    \]
    where $\alpha = \frac{1 + \sqrt{5}}{2}$ and
    $\beta = \frac{1 - \sqrt{5}}{2}$.
\end{theorem}
\begin{proof}
    We use the strong induction principle to prove this statement with $n_0 = -1$.
    Let us first prove the base case,
    $\frac{(\alpha^0 - \beta^0)}{\sqrt{5}} = 0 = f_0$.
    We also need to prove the induction step.
    \begin{itemize}
        \item If $k = 1$, then $\frac{(\alpha^1 - \beta^1)}{\sqrt{5}} = 1 = f_1$.
        \item Otherwise, by the induction hypothesis,
            $f_k = \frac{\alpha^k - \beta^k}{\sqrt{5}}$ and
            $f_{k - 1} = \frac{\alpha^{k - 1} - \beta^{k - 1}}{\sqrt{5}}$.
            By the definition of the Fibonacci numbers $f_{k + 1} = f_k + f_{k - 1}$.
            Hence,
            \[
                f_{k + 1} = \frac{\alpha^k - \beta^k}{\sqrt{5}} +
                \frac{\alpha^{k - 1} - \beta^{k - 1}}{\sqrt{5}}.
            \]
            Note that it is enough to show that
            \begin{equation}
            \label{equation:binet}
                \frac{\alpha^{k + 1} - \beta^{k + 1}}{\sqrt{5}} =
                \frac{\alpha^k - \beta^k}{\sqrt{5}} +
                \frac{\alpha^{k - 1} - \beta^{k - 1}}{\sqrt{5}}.
            \end{equation}
            Note that it is the same as
            \[
                \frac{\alpha^{k + 1} - \alpha^k - \alpha^{k - 1}}{\sqrt{5}} =
                \frac{\beta^{k + 1} - \beta^k - \beta^{k - 1}}{\sqrt{5}}.
            \]
            Additionally, note that $\alpha$ and $\beta$ are roots of the equation
            $x^2 - x - 1 = 0$. Hence,
            $\alpha^{k + 1} - \alpha^k - \alpha^{k - 1} = \alpha^{k - 1}(\alpha^2 -
            \alpha - 1) = 0$ and
            $\beta^{k + 1} - \beta^k - \beta^{k - 1} =
            \beta^{k - 1}(\beta^2 - \beta - 1) = 0$. Which implies
            equality~(\ref{equation:binet}).
    \end{itemize}
\end{proof}

\begin{template}
  \textbf{Template for proving a statement using strong induction.} \\

  We use induction by $n$. Base case for $n = n_0 + 1$: \emph{present some
  argument that proves the statement with $n$ replaced by $1$}.

  Now we need to prove the induction step from $n_0 + 1$, \dots, $n_0 + k - 1$
  to $k$. Let us assume now that the statement is true for any $n$ from $n_0 +
  1$ to $n_0 + k - 1$ for some $k$. \emph{Present some argument of the statement
  with $n$ replaced by $n_0 + k$ assuming the statement is true if we replace
  $n$ by $n_0 + \ell$ for $1 \le \ell < k - 1$}. Hence,
  the statement is true for all $n$ by the induction principle.
\end{template}

Another example of an application of the strong induction is the proof that
any number can be written in digital numeral systems with any base.
\begin{theorem}
  Let $b > 1$ be an integer. Then there is a unique representation
  $(c_0, \dots, c_\ell)_b$ of any positive number $n$ in the base-$b$ digital
  numeral system.
  In other words, for any positive integer $n$, there are unique
  $0 \le c_0, \dots, c_\ell < b$ such that $n = \sum_{i = 0}^\ell b^i c_i$.
\end{theorem}
\begin{proof}
  We prove the statement using strong induction by $n$. The base case for
  $n < b$ is clear (we can choose $\ell = 0$ and $c_0 = n$).
  Let us now prove the induction step. Assume the statement is true for all
  $k < n$. Let $n$ divided by $b$ be equal to $q$ with the remainder $c_0$.
  Note that $(n - c_0) / b < n$ is a positive integer. Hence,
  by the induction hypothesis, there are
  $0 \le c_1, \dots, c_\ell < b$
  such that $(n - c_0) / b = \sum_{i = 1}^\ell b^{i - 1} c_i$. Hence,
  $n = \sum_{i = 0}^\ell b^i c_i$.
\end{proof}
\nomenclature[U]{$(c_0, \dots, c_\ell)_b$}{%
  denotes the number $n = \sum_{i = 0}^\ell b^i c_i$; i.e., $c_0, \dots, c_\ell$
  are the digits of $n$ in the base-$b$ digital representation}

Now we are ready to prove the strong induction principle.
\begin{proof}[Proof of Theorem~\ref{theorem:strong-induction}]
  It is easy to see that if $P(n)$ is true for all $n > n_0$, then the base
  case and the induction steps are true. Let us prove that if the base case and
  the induction step are true, then $P(n)$ is true for all $n > n_0$.

  Let $Q(k)$ be the statement that $P(n_0 + 1)$, \dots, $P(n_0 + k)$ are true.
  Note that $Q(1)$ is true by the base case for $P$. Additionally, note that if
  $Q(k)$ is true, then $Q(k + 1)$ is also true, by the induction step for $P$.
  Hence, by the induction principle, $Q(k)$ is true for all positive integers
  $k$. Which implies that $P(n_0 + k)$ is true for all positive integers $k$.
\end{proof}

\section{Analysis of Recursive Algorithms}
\label{section:strong-induction-recursive}
To illustrate the power of recursive definitions and strong induction, consider
following game: Alice has chosen a number from $1$ to $1000$. Bob wants to
guess the number so he is asking Alice ``yes'' or ``no'' questions.
How many questions does Bob need to ask to determine the number in the
worst-case scenario?

The following simple algorithm allows Bob to learn the number using $10$
questions.

\begin{enumerate}
  \item Bob start with two numbers $\ell = 0$ and $u = 1000$.
  \item \label{algorithm-step:binary-search-result-guess-number}
    If there is only one integer $x$ such that $\ell < x \le u$, Bob says that
    Alice's number is $x$ and terminates the algorithm.
  \item \label{algorithm-step:binary-search-decision-guess-number}
    Bob asks whether the Alice's number is at most $(\ell + u) / 2$. If the
    answer is yes, then Bob replaces $u$ by $(\ell + u) / 2$; otherwise Bob
    replaces $\ell$ by $(\ell + u) / 2$.
    Bob goes to step~\ref{algorithm-step:binary-search-decision-guess-number}.
\end{enumerate}

We need to prove that Bob's algorithm is correct and that it makes at most $10$
questions. We prove a bit stronger statement. If on
step~\ref{algorithm-step:binary-search-decision-guess-number} $u - \ell < n$ and
Alice's number is between $\ell$ and $u$ ($u$ and $\ell$ are some reals), then
the algorithm returns Alice's number using at most $\log_2 n$ questions
($\log_2 1000 \le 10$). We are going to use strong induction by $n$.

The base case is clear since if $u - \ell < 1$ there is at most one integer
between $u$ and $\ell$ and by the assumption there is Alice's number between
$\ell$ and $u$. Hence, Bob is going to guess Alice's number correctly without asking
any questions.

Assume that the statement is true for all $m < n$. Note that
$\ell < (\ell + u) / 2 < u$ since $n \ge 2$. Therefore Alice's number is either
between $\ell$ and $(\ell + u) / 2$ or between $(\ell + u) / 2$ and $u$. Hence,
we go to step~\ref{algorithm-step:binary-search-decision-guess-number} with new
$\ell'$ and $u'$ such that $u' - \ell' < n / 2$ and Alice's number is between
$\ell'$ and $u'$. As a result, by the induction hypothesis, Bob is going to
guess Alice's number correctly using at most $1 + \log_2 (n / 2) = \log_2 n$
questions.

\begin{chapterendexercises}
    \exercise Let $a_0 = 2$, $a_1 = 5$, and $a_n = 5a_{n - 1} - 6 a_{n - 2}$
        for all integers $n \ge 2$. Show that $a_n = 3^n + 2^n$ for all integers
        $n \ge 0$.
        \begin{solution}
            We prove this using induction by $n$. The base case for $n \le 1$ is clear
            since $3^0 + 2^0 = 2$ and $3^1 + 2^1 = 5$.

            Let us prove the induction step. Assume that $a_n = 3^n + 2^n$ and
            $a_{n - 1} = 3^{n - 1} + 2^{n - 1}$, we need to prove that
            $a_{n + 1} = 3^{n + 1} + 2^{n + 1}$. Note that
            \begin{multline*}
                a_{n + 1} = 5a_n - 6 a_{n - 1} =
                5 \cdot 3^n + 5 \cdot 2^n - 6 \cdot 3^{n - 1} -
                6 \cdot 2^{n - 1} = \\
                3^{n - 1} \cdot 9 + 2^{n - 1} 4 = 3^{n + 1} + 2^{n + 1}.
            \end{multline*}
        \end{solution}
    \exercise Let $f_0 = 1$, $f_1 = 1$, and $f_{n + 2} = f_{n + 1} + f_n$ for
        all integers $n \ge 0$. Show that
        $f_n \ge \left(\frac{3}{2}\right)^{n - 2}$.
    \exercise Show that $f_{n + m} = f_{n - 1} f_{m - 1} + f_n f_m$.
    \exercise Give a nonadaptive algorithm for Bob that allows him to guess the
      number using $10$ queries. In other words, write $10$ questions such that
      answers to these questions allow Bob to guess the number.
    \exercise Show that \Cref{algorithm:binary-search} makes at most
      $6 + 2\log_2(n)$ comparisons.
      \begin{algorithm}
        \begin{algorithmic}[1]
          \Function{BinarySearch}{$e$, $a_1$, \dots, $a_n$}
            \If{$n \le 5$}
              \For{$i$ from $1$ to $n$}
                \If{$a_i = e$}
                  \State\Return{i}
                 \EndIf
              \EndFor
            \Else
              \State{$\ell \gets \floor{\frac{n}{2}}$}
              \If{$a_\ell \le e$}
                \State\Call{BinarySearch}{$e$, $a_1$, \dots, $a_\ell$}
              \Else
                \State\Call{BinarySearch}{$e$, $a_{\ell + 1}$, \dots, $a_n$}
              \EndIf
            \EndIf
          \EndFunction
        \end{algorithmic}
        \caption{The binary search algorithm that finds an element $e$ in the sorted
          list $a_1$, \dots, $a_n$.}
        \label{algorithm:binary-search}
      \end{algorithm}
\end{chapterendexercises}


\section{Additional comments}
\paragraph{\textbf{Remarks}}
\begin{enumerate}

	\item $k$ disjoint sets $A_1,A_2,\cdots,A_k$ represent $k$ classes. We use $A$ to denote
	\[
	A = A_1\cup A_2\cup \cdots \cup A_k.
	\]
	Naturally we can define the standard decision mapping $\bm{\chi}:A \rightarrow \{e_1,e_2,\cdots,e_k\}\subset \mathbb{R}^k$  as
	\begin{equation}
	\bm{\chi}(x) = e_i,\ x\in A_i,
	\end{equation}
	which is just like a characteristic function.

	\item To solve a classification problem, our goal is to find the standard decision mapping. But usually it's unpractical, because we seem to be better at dealing with continuous functions with continuous outputs. A natural idea is to reach the goal in two steps. We can use a composition of the following two mappings
	$$\bm{f}:A \rightarrow \mathbb{R}^k;\ x\mapsto y.$$
	$$\bm{\pi}: \mathbb{R}^k \rightarrow \{e_1,e_2,\cdots,e_k\};\ \ y\mapsto e_i,i = \argmax_j~y_j$$
	to obtain the standard decision mapping. Notice that the mapping $\bm{\pi}$ is fixed, so we only need to find a proper mapping $\bm{f}$.

	\item Assume that we want to find a proper $\bm{f}$ in a mapping sapce $\mathscr{H}$. The result of a classification problem depends on which $\mathscr{H}$ to choose and how to find the optimal mapping $\bm{f}$ in $\mathscr{H}$.\\

	\item Given a mapping space $\mathscr{H}$, if there exists an $\mathbf{f}\in \mathscr{H}$ such that
	\begin{equation}
	\bm{\chi} = \bm{\pi} \circ \bm{f}.
	\end{equation}
	we can say that $A_1,A_2,\cdots,A_k$ are $\mathscr{H}$-separable. Setting $\mathscr{H}$ to be the affine mapping space, $i.e.$ $\mathscr{H} = \{\bm{f}: \bm{f}(x) = Wx+b\}$, we obtain the definition of linearly separable.\\

	\item A linear model  is a method to find the optimal $W$ and $b$ when we take $\mathscr{H} = \{\bm{f}: \bm{f}(x) = Wx+b\}$.\\

	\item Assume that $\bm{\phi}$ is a feature mapping from $A$ to a feature space $\mathcal{H}$. A natural choice of $\mathscr{H}$ is
	\begin{equation}
	\mathscr{H} = \{\bm{f}: \bm{f}(x) = W\bm{\phi}(x)+b\}
	\end{equation}
	%If we set $\mathcal{H}$ to be a Reproduced Kernel Hilbert Space $w.r.t.$ a kernel function $\bm{k}$ and use SVM to find the optimal $W$ and $b$, we can obtain the methods of kernel SVM.
\end{enumerate}

\subsection{General approach for separation }
A collection of sets $A_i\subset \mathbb{R}^n$, $1\le i\le k$, are said to be separable if there exists a continuous function $h:\mathbb{R}^n\to\mathbb{R}^k$ such that $h(x)=e_i$.

A general approach:
\begin{itemize}
	\item Find a ``nonlinear" function $h_0:\mathbb{R}^n\to\mathbb{R}^m$ such that
\begin{enumerate}
	\item $\tilde{A}_i=h_0(A_i)\subset\mathbb{R}^m$ are linearly separable
	\item $m\ll n$
\end{enumerate}
\item Use a ``linear" classifier $l_0:\mathbb{R}^m\to\mathbb{R}^k$, then $h=l_0\circ h_0:\mathbb{R}^n\to\mathbb{R}^k$ is the classifier.
\end{itemize}

\begin{remark}
	\begin{enumerate}
		\item $h_0$: dimension reduction;
		\item Feature extraction;
		\item $h_0$: deep neural network, special function class;
		\item $x^{l+1}=\sigma(Wx^l+b)$, for CNN, $W,b$ are ``sparse";
		\item ``linear classifier" is fully connected layer.
	\end{enumerate}
	\end{remark}

\subsection{Feature map}

\begin{remark}
The "linear" here means that $wx+b$ is linear with respect to $x$. This condition is not crucial. Actually, we can replace $x$ by $\phi: \mathbb{R}^n\rightarrow \mathbb{R}^m$. To be specific, let  $\tilde x=w\phi(x)+b$ and $\tilde A_i=\phi (A_i)\in \mathbb{R}^m$. The function $\phi$ here is a feature map,
common choices include
\begin{enumerate}
\item $\phi(x)=x$;
\item finite elements, wavelets;
\item deep neural networks.
\end{enumerate}
\end{remark}

\input{6DL/LinearSets}


%================================
\section{More on Separation Axioms}
%================================

%--------------------------------
\begin{definition}
	[saperated sets]
	Let $(X, \mathcal T)$ be a topological space, and let $A, B \in \mathcal P(X)$.

	\begin{enumerate}[(i)]
		\item $A$ and $B$ are said to be \textit{separated} iff each is disjoint from other's closure.
		\item $A$ and $B$ are said to be \textit{separated by neighbourhoods} iff there are neighbourhoods $N_A$ of $A$ and $N_B$ of $B$ such that $N_A$ and $N_B$ are disjoint.
		\item $A$ and $B$ are said to be \textit{separated by closed neighbourhoods} iff there are closed neighbourhoods $\overline N_A$ of $A$ and $\overline N_B$ of $B$ such that $\overline N_A$ and $\overline N_B$ are disjoint.
		\item $A$ and $B$ are said to be \textit{separated by a continuous function} iff there is a continuous function $f: X \to \mathbb R$, such that $f[A] = \{0\}$ and $f[B] = \{1\}$.
		\item $A$ and $B$ are said to be \textit{precisely separated by a continuous function} iff there is a continuous function $f: X \to \mathbb R$, such that $f^{-1}[\{0\}] = A$ and $f^{-1}[\{1\}] = B$
	\end{enumerate}
\end{definition}
%--------------------------------

%--------------------------------
\begin{definition}
	[$T_{2 \nicefrac{1}{2}}$ spaces]
	\label{def: T_2.5 spaces}
	A topological space $(X, \mathcal T)$ is said to be $T_{2 \nicefrac{1}{2}}$ or \textit{Urysohn} iff two distinct points in $X$ are separated by closed neighbourhoods.
\end{definition}
%--------------------------------

%--------------------------------
\begin{example}
	[$T_2$ but not $T_{2 \nicefrac{1}{2}}$]
	\footnote{
		See \href{https://planetmath.org/hausdorffspacenotcompletelyhausdorff}{MathPlanet}.
	}
	(Remained as a problem)
	% todo: Remained as a problem!
\end{example}
%--------------------------------

%--------------------------------
\begin{definition}
	[$T_3$ spaces]
	\label{def: T_3 spaces}
	A topological space $(X, \mathcal T)$ is said to be $T_3$ or \textit{regular} iff it is $T_0$ and given any point $x \in (X, \mathcal T)$ and closed set $V \subseteq X$ with $x \notin V$ are separated by neighbourhoods.
\end{definition}
%--------------------------------

%--------------------------------
\begin{definition}
	[$T_{3\nicefrac{1}{2}}$ spaces]
	\label{def: T_3.5 spaces}
	A topological space $(X, \mathcal T)$ is said to be $T_{3 \nicefrac{1}{2}}$, or \textit{Tychonoff} or, \textit{completely $T_3$}, or \textit{completely regular}, iff it is $T_0$ and given any point $x$ and closed set $V \subseteq X$ with $x \notin V$, they are separated by a continuous function.
\end{definition}
%--------------------------------

%--------------------------------
\begin{definition}
	[$T_4$ spaces]
	\label{def: T_4 spaces}
	A topological space $(X, \mathcal T)$ is said to be $T_4$ or \textit{normal} iff it is Hausdorff and any tow disjoint closed subsets of $X$ are separated by neighbourhoods.
\end{definition}
%--------------------------------

%--------------------------------
\begin{proposition}
	[Urysohn's lemma]
	\label{prop: urysohn's lemma}
	A topological space is normal iff any two disjoint closed sets are separated by a continuous function.
\end{proposition}
%--------------------------------

%--------------------------------
\begin{definition}
	[$T_5$ spaces]
	\label{def: T_5 spaces}
	A topological space $(X, \mathcal T)$ is said to be $T_5$ or \textit{completely $T_4$} iff it is $T_1$ any two separated sets are separated by neighbourhoods.
\end{definition}
%--------------------------------

%--------------------------------
\begin{proposition}
	Every subspace of a $T_5$ space is normal.
\end{proposition}
%--------------------------------

%--------------------------------
\begin{definition}
	[$T_6$ spaces]
	\label{def: T_6 spaces}
	A topological space $(X, \mathcal T)$ is said to be $T_6$, or \textit{perfectly $T_4$} or \textit{perfectly normal} iff it is $T_1$ and any two disjoint closed sets are precisely separated by a continuous function.
\end{definition}
%--------------------------------

%--------------------------------
\begin{proposition}
	[Tietze extension theorem]
	\label{prop: Tietze extension theorem}
	Let $(X, \mathcal T)$ be normal topological space, and let $f: A \to (\mathbb R, \mathcal T')$ be a continuous map where $A$ is a closed subset of $X$ and $\mathcal T'$ is the standard topology (induced by Euclidean metric). Then there exists a continuous map
	$$
	F: (X, \mathcal T) \to (\mathbb R, \mathcal T'),
	$$
	such that
	$$
	\forall x \in A: f(x) = g(x).
	$$
\end{proposition}
%--------------------------------


\section{$\mathbf{A} = \mathbf{U\Sigma V}^{\dagger}$ Singular Value Decomposition}
\label{sec:svd}

The Singular Value Decomposition (SVD) exists for \underline{any} matrix $\mathbf{A}\in\mathbb{C}^{m\times n}$, and is a closely related alternative to the eigendecomposition (cf. section \ref{sec:diagonalization}) that works for non-square matrices. The contrast to diagonalization is that the eigenvectors and eigenvalues of $\mathbf{A}^{\dagger}\mathbf{A}$ and $\mathbf{A}\mathbf{A}^{\dagger}$ found, rather than to look for the eigensystem of $\mathbf{A}$ itself. The advantage is that $\mathbf{A}^{\dagger}\mathbf{A}$ and $\mathbf{A}\mathbf{A}^{\dagger}$ always have the convenient properties of being square, hermitian and positive semidefinite.

SVD comes up incessantly in the context of data analysis. In general, the decomposition has the form:

\begin{equation}
\mathbf{A} = \mathbf{U\Sigma V^{\dagger}}
\end{equation}

Where $\mathbf{U}$ and $\mathbf{V}$ are unitary, and $\Sigma$ is a diagonal matrix with real and positive entries $\sigma_i^2$ along the diagonal, so that $\sigma_1 \geq \sigma^2_2 \geq ...\geq \sigma^2_n$. $\mathbf{U}$ is the matrix of left singular vectors, which are the eigenvectors of $\mathbf{A A^{\dagger}}$. $\mathbf{V}$ is the matrix of right singular vectors, which are the eigenvectors of $\mathbf{A^{\dagger} A}$. The singular values are the square roots of the eigenvalues of $\mathbf{A^{\dagger}A}$ or, equivalently, $\mathbf{AA^{\dagger}}$. If the $\mathbf{A}$ happens to be square and symmetric ($\mathbf{A}=\mathbf{A^T}$, cf. section \ref{sec:hermitian}), then the singular values are simply the absolute values of the eigenvalues of $\mathbf{A}$. The right and left singular vectors of the matrix are orthonormal bases that respectively span the column and row space of $\mathbf{A}$. The number of singular values of $\mathbf{A}$ is the rank of $\mathbf{A}$.
\\
The singular values are always real and positive or zero, because $\mathbf{A^{\dagger} A}$ and $\mathbf{A A^{\dagger}}$ are  hermitian and positive semidefinite (cf. sections \ref{sec:hermitian}, \ref{sec:definite}). When the matrix $\mathbf{A}$ has purely real entries, then the left and right singular vectors are also real, which is not necessarily the case for complex $\mathbf{A}$.
\\

The SVD of a matrix is unique up to the sign columns in $\mathbf{U}$ and $\mathbf{V}$. That is, the SVD is valid under transforming $\mathbf{u}_i, \mathbf{v}_i \rightarrow -\mathbf{u}_i, -\mathbf{v}_i $ if $\mathbf{u}_i$ and $ \mathbf{v}_i$ are the $i$th vectors in $\mathbf{U}$ and $\mathbf{V}$. This has the consequence that the basis of singular vectors that is found using SVD does not remain consistently oriented in the presence or noise or the slow evolution of a system. This is addressed in greater length in section \ref{sec:datasvd} on performing SVD specifically on data matrices.

\subsection{Full and Economy SVDs}
While these properties are always true, unfortunately people use a range of conventions when it comes to the size of $\mathbf{U}$, $\mathbf{\Sigma}$ and $\mathbf{V}$. \\

The first convention is for $\mathbf{U}$ and $\mathbf{V}$ to be square, in which case they contain the full set of left and right singular vectors, and $\Sigma$ has dimension $m\times n$, with $0$ entries in rows $i>n$. This is known as the \textit{full SVD}.\\

\citeasnoun{friedman2001elements} uses the the convention where $\Sigma$ is square, i.e. $\mathbf{U}:\ m\times n$, $\mathbf{\Sigma}:\ n\times n$ and $\mathbf{V}:\ n\times n$. This means that $U$ does not contain the full set of $m$ left singular vectors. Note that, in this case, $\mathbf{U^{\dagger}}\mathbf{U} = \mathbf{I}$ but $\mathbf{U}\mathbf{U^{\dagger}} \neq \mathbf{I}$. \possessivecite{friedman2001elements} convention is of advantage in the context of data analysis, where the data matrix tends to be "tall and skinny" (i.e. $m>>n$), and only the first $n$ left singular vectors are relevant. Also, a letting $\Sigma$ be a square matrix significantly simplifies calculations. \possessivecite{friedman2001elements} is known as the \textit{economy} or \textit{compact} SVD.\\

The two different layouts are illustrated in Figure \ref{fig:full_vs_economy_svd}, which I brazenly copied from \citeasnoun{mathworkseigs}.

\begin{figure}
\centering
\includegraphics[scale=0.5]{full_vs_economy_svd.png}
\caption{Dimensions for Full vs. Economy SVDs}
\label{fig:full_vs_economy_svd}
\end{figure}

\subsection{Matrix Approximation}
The SVD enables the decomposition of the matrix into a sum:

\begin{equation}
\mathbf{A} = \sum_{i=1}^n \sigma_i (\mathbf{u}_i \otimes \mathbf{v}_i)
\end{equation}

Which may be truncated at some rank $r$, resulting in a rank-$r$ approximation to $\mathbf{A}$. This is known as the \textit{truncated singular value estimator} (TSVD):

\begin{equation}
\mathbf{\hat{A}} = \sum_{i=1}^r \sigma_i (\mathbf{u}_i \otimes \mathbf{v}_i)
\end{equation}

Where $\mathbf{u}_i$ is the $i$th left singular vector and $\mathbf{v}_i$ is the $i$th right singular vector. Following the Eckart-Young-Mirsky Theorem, taking the first $r \leq n$ terms of this series is the best rank-$r$ approximation to $\mathbf{A}$ under the Frobenius norm of the error, i.e.  $\underset{\mathrm{rank}(\mathbf{A}^{(r)})=r\leq n}{\argmin} ||\mathbf{A^r}-\mathbf{A}||_F$. The Frobenius norm essentially measures the elementwise mean squares error (MSE), cf. section \ref{sec:frobenius}).\\

In the context of data analysis, the reduced rank representation of a data matrix $\mathbf{A}$ acts as a filter in which the components that explain less of the variance in the dataset are removed, with the hope being that these correspond to noise. The intuition is that noise probably originates from a random process, so that components that correspond to noise have weak correlation with the entries of the dataset and therefore small singular values. On the other hand, the signal probably systematic, so that the components that correspond to the signal have larger singular value. Since TSVD minimizes the average mean squares error (AMSE) amounts to the rank-$r$ maximum likelihood estimator of the signal under the assumption of normally distributed noise with mean zero.

By virtue of being expressible in terms of fewer base vectors, a lower rank approximation is also a very effective compression method. For a more elaborate discussion of SVD in data analysis, see section \ref{sec:datasvd}. In a practical context, the question of selecting the appropriate rank of the approximation emerges. Section \ref{sec:truncation} covers a few approaches to this.

% Geometric interpretation
\subsection{Geometric Interpretation of SVD}
A common intuitive interpretation of $\mathbf{U}$, $\mathbf{\Sigma}$ and $\mathbf{V}$ is to see them as a decomposition of the action of the matrix $\mathbf{A}\in\mathbb{R}^{m\times n}$ into a rotation, a stretching and another rotation. Let $\{\mathbf{x} : |\mathbf{x}|=1 \}$ be the points on the surface of a unit sphere, then $\mathbf{Ax}$:

\begin{itemize}
\item $\mathbf{V}$ rotates the unit sphere, which has no effect. However, it expresses each $\mathbf{x}$ in terms of a different coordinate system (the right singular basis of $\mathbf{A}$).
\item $\mathbf{\Sigma}$ stretches the unit sphere into an ellipse with principal axes aligned to that coordinate system.
\item $\mathbf{U}$ rotates the ellipse.
\end{itemize}

In the above, "rotation" might include coordinate flips that are undone by $\mathbf{U}$ following $\mathbf{V}$. Alternatively, for a vector $\mathbf{v}\in\mathbf{R}^{n}$, the operation $\mathbf{Av}$:

\begin{itemize}
\item $\mathbf{V}$ is a change of basis matrix that expresses $\mathbf{x}$ in terms of a right eigenbasis of $\mathbf{A}$
\item $\mathbf{\Sigma}$ performs a stretching in that basis
\item $\mathbf{U}$ is a change of basis from the right eigenspace into the left eigenspace of $\mathbf{A}$.
\end{itemize}


\section*{Exercises}

\begin{ex} In the following, polar coordinates $(r,\theta) $ for a
point in the plane are given. Find the corresponding Cartesian coordinates.

\begin{enumerate}
\item $(2,\pi /4) $

\item $(-2,\pi /4) $

\item $(3,\pi /3) $

\item $(-3,\pi /3) $

\item $(2,5\pi /6) $

\item $(-2,11\pi /6) $

\item $(2,\pi /2) $

\item $(1,3\pi /2) $

\item $(-3,3\pi /4) $

\item $(3,5\pi /4) $

\item $(-2,\pi /6) $
\end{enumerate}
%\begin{sol}
%\end{sol}
\end{ex}

\begin{ex} Consider the following Cartesian coordinates $(x,y)$. Find polar coordinates corresponding to these points.

\begin{enumerate}
\item $(-1,1) $

\item $(\sqrt{3},-1) $

\item $(0,2) $

\item $(-5,0) $

\item $(-2\sqrt{3},2) $

\item $(2,-2) $

\item $(-1,\sqrt{3}) $

\item $(-1,-\sqrt{3}) $
\end{enumerate}
%\begin{sol}
%\end{sol}
\end{ex}

\begin{ex} The following relations are written in terms of Cartesian coordinates $(x, y)$. Rewrite them in terms of polar coordinates, $(r, \theta)$.

\begin{enumerate}
\item $y=x^{2}$

\item $y=2x+6$

\item $x^{2}+y^{2}=4$

\item $x^{2}-y^{2}=1$
\end{enumerate}
%\begin{sol}
%\end{sol}
\end{ex}

\begin{ex} Use a calculator or computer algebra system to graph the following
polar relations.

\begin{enumerate}
\item $r=1-\sin (2\theta) ,\theta \in \mat{0,2\pi} $

\item $r=\sin (4\theta) ,\theta \in \mat{0,2\pi} $

\item $r=\cos (3\theta) +\sin (2\theta) ,\theta
\in \mat{0,2\pi} $

\item $r=\theta ,\ \theta \in \mat{0,15} $
\end{enumerate}
%\begin{sol}
%\end{sol}
\end{ex}

\begin{ex} Graph the polar equation $r=1+\sin \theta $ for $\theta \in \mat{0,2\pi}$.
%\begin{sol}
%\end{sol}
\end{ex}

\begin{ex} Graph the polar equation $r=2+\sin \theta $ for $\theta \in \mat{0,2\pi}$.
%\begin{sol}
%\end{sol}
\end{ex}

\begin{ex} Graph the polar equation $r=1+2\sin \theta $ for $\theta \in \mat{0,2\pi}$.
%\begin{sol}
%\end{sol}
\end{ex}

\begin{ex} Graph the polar equation $r=2+\sin (2\theta) $ for $\theta \in \mat{0,2\pi
}$.
%\begin{sol}
%\end{sol}
\end{ex}

\begin{ex} Graph the polar equation $r=1+\sin (2\theta) $ for $\theta \in \mat{0,2\pi
}$.
%\begin{sol}
%\end{sol}
\end{ex}

\begin{ex} Graph the polar equation $r=1+\sin (3\theta) $ for $\theta \in \mat{0,2\pi
}$.
%\begin{sol}
%\end{sol}
\end{ex}

\begin{ex} Describe how to solve for $r$ and $\theta $ in terms of $x$ and $y$ in polar
coordinates.
%\begin{sol}
%\end{sol}
\end{ex}

\begin{ex} This problem deals with parabolas, ellipses, and
hyperbolas and their equations. Let $l,e>0$ and consider
\begin{equation*}
r=\frac{l}{1\pm e\cos \theta }
\end{equation*}
Show that if $e=0$, the graph of this equation gives a circle. Show that if $0<e<1$, the graph is an ellipse, if $e=1$
it is a parabola and if $e>1$, it is a hyperbola.
%\begin{sol}
%\end{sol}
\end{ex}


\section*{Exercises}

\begin{ex}
  Find the basic solutions of each of the following homogeneous
  systems of equations.
  \begin{equation*}
    (a)~
    \begin{array}{r@{~}c@{~}l}
      2x + 3y + 4z &=& 0 \\
      x - 2y + z &=& 0 \\
      4x - y + 6z &=& 0
    \end{array}
    \quad
    (b)~
    \begin{array}{r@{~}c@{~}l}
       x -  y +  z &=& 0 \\
      -x - 2y - 4z &=& 0 \\
      2x +  y + 5z &=& 0
    \end{array}
    \quad
    (c)~
    \begin{array}{r@{~}c@{~}l}
      x +  y -  z + 2w &=& 0 \\
      x + 3y +  z + 6w &=& 0 \\
      x + 2y      + 4w &=& 0
    \end{array}
  \end{equation*}
\end{ex}

\begin{ex}
  Which of the following homogeneous systems of linear equations have
  non-trivial solutions?
  \begin{enumerate}
  \item[(a)] 4 equations in 3 variables, rank 3.
  \item[(b)] 3 equations in 4 variables, rank 3.
  \item[(c)] 4 equations in 3 variables, rank 2.
  \item[(d)] 3 equations in 4 variables, rank 2.
  \end{enumerate}

  \begin{sol}
    The systems (b), (c), and (d) have non-trivial solutions, because
    the rank is less than the number of variables.
  \end{sol}
\end{ex}

\begin{ex}
  My system of equations has a solution $(x,y,z) = (1,2,4)$. The
  associated homogeneous system has basic solutions $(x,y,z) =
  (1,0,1)$ and $(x,y,z)=(0,1,-1)$. What is the general solution of my
  system of equations?

  \begin{sol}
    The general solution is $(x,y,z) = (1,2,4)+s(1,0,1)+t(0,1,-1)$, or
    equivalently, $(x,y,z) = (1+s, 2+t, 4+s-t)$, where $s$ and $t$ are
    parameters.
  \end{sol}
\end{ex}


% based on example 8 in pythontex_gallery
% https://github.com/gpoore/pythontex/

\documentclass[12pt]{mpllatex}
\usepackage{examples}

\begin{document}

\section*{Step-by-step integration}

This is another nice example drawn from the Pythontex gallery, see
\ \url{https://github.com/gpoore/pythontex}.

It shows the step-by-step computations of a simple triple integral.

\vspace{12pt}

\begin{maple}
   # Define limits of integration
   x_max := 2:   y_max := 3:   z_max := 4:
   x_min := 0:   y_min := 0:   z_min := 0:

   ans := int(f(x,y,z), [x=x_min..x_max, y=y_min..y_max, z=z_min..z_max]):            # mpl(lhs.01,ans)

   f := (x,y,z) -> x*y + y*sin(z) + cos(x+y):

   ans := ''int''(''int''(''int''(f(x,y,z), x=x_min..x_max), y=y_min..y_max), z=z_min..z_max):    # mpl(rhs.01,ans)
   ans := ''int''(''int''(int(f(x,y,z), x=x_min..x_max), y=y_min..y_max), z=z_min..z_max):        # mpl(rhs.02,ans)
   ans := ''int''(int(int(f(x,y,z), x=x_min..x_max), y=y_min..y_max), z=z_min..z_max):            # mpl(rhs.03,ans)
   ans := int(int(int(f(x,y,z), x=x_min..x_max), y=y_min..y_max), z=z_min..z_max):                # mpl(rhs.04,ans)

   # And now, a numerical approximation
   ans := evalf[15](ans):                                                             # mpl(rhs.05,ans)

\end{maple}

\begin{minipage}[t]{0.65\textwidth}
\begin{align*}
   \mpl{lhs.01} &= \mpl{rhs.01}\\
                &= \mpl{rhs.02}\\
                &= \mpl{rhs.03}\\
                &= \mpl{rhs.04}\\
                &\approx \mpl{rhs.05}
\end{align*}
\end{minipage}
\hskip 1cm
\lower16pt\hbox{%
\begin{minipage}[t]{0.35\textwidth}
\begin{latex}
   \begin{align*}
      \mpl{lhs.01} &= \mpl{rhs.01}\\
                   &= \mpl{rhs.02}\\
                   &= \mpl{rhs.03}\\
                   &= \mpl{rhs.04}\\
                   &\approx \mpl{rhs.05}
   \end{align*}
\end{latex}
\end{minipage}}

\end{document}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% CS624: Analysis of Algorithms
% Copyright 2015 Pejman Ghorbanzade <pejman@ghorbanzade.com>
% Creative Commons Attribution-ShareAlike 4.0 International License
% More info: https://github.com/ghorbanzade/beacon
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section*{Question 1}

\begin{enumerate}[label=(\alph*)]
\item Use the substitution method to show that $T(n) = 8T(n/2) + n^3 = \mathcal{O}(n^3\log n)$.
Assume $T(2)=d$ where $d$ is a constant.
Use induction and do not skip stages.

\item Prove or disprove $f(n) = \Theta(f(n/2))$.
If true, show why.
Otherwise give a counter-example.
\end{enumerate}

\subsection*{Solution}

\begin{enumerate}[label=(\alph*)]
\item Proof is given by induction on $n$.
Using assumption $T(2) = d$, $T(1) = \frac{T(2)-1}{8} = \frac{d-1}{8} = e$ where $e$ is a constant.
\begin{enumerate}[label=\arabic*.]
\item \textit{Base Case}\\
We take the base step as $n = 1$.
In this case, there exist constants $f > 0$, $n_0 = 1$ where $f = e(\frac{d}{8}+\frac{d}{1})$ such that $T(1) \leq f \times n^3 \log n$ for all $n \geq n_0$.
Therefore, $T(1) = \mathcal{O}(n^3 \log n)$.
\item \textit{Inductive Hypothesis}\\
We form inductive hypothesis as $T(k)=\mathcal{O}(n^3 \log n)$.
In other words, that there are constants $c$ and $n'_0$ such that $T(k) \leq ck^3 \log k$ for $k \geq n'_0$.
\item \textit{Induction Step}\\
Using inductive hypothesis we will show $T(2k) = O(n^3\log n)$; in other words, that there are constants $c^\prime$ and $n''_0$ such that $T(2k) \leq c' k^3 \log k$ for all $k \geq n''_0$.

\begin{equation}\label{eq11}
T(2k) = 8 T(\frac{2k}{2}) + (2k)^3
\end{equation}
By Inductive hypothesis, Equation \ref{eq11} can be written as shown in Equation \ref{eq12}.
\begin{equation}\label{eq12}
\begin{aligned}
T(2k) &\leq 8c \big( (2k)^3 \log (2k) \big) + (2k)^3\\
&\leq (8c + 1) (2k)^3 \log (2k)\\
&\leq 8(8c+1) (k)^3 (\log (k) + \log 2)\\
&\leq 16(8c+1) (k^3 \log k)
\end{aligned}
\end{equation}
Therefore, by taking $c' = 16(8c+1)$ and $n''_0 = n'_0$ we have shown Equation \ref{eq13} holds for all $k \geq n''_0$.
\begin{equation}\label{eq13}
T(2k) \leq c' \big( (2k)^3 \log 2k \big)
\end{equation}
Hence, we have shown $T(2k) = \mathcal{O}(n^3\log n)$ and the induction is complete.
\end{enumerate}
\item The statement is not valid.
To disprove, $f(n)=2^n$ can be given as a counter-example.
What statement suggests is there are constants $C$ and $n_0$ such that $2^n < C(2^{n/2})$; in other words, $2^{n/2} < C$ for all $n > n_0$.
This is obviously not true since there is no limit in our choice of $n$.
\end{enumerate}


\section{Eigenvalues and Eigenvectors}
For a linear map $T:V\to V$ where $V$ is a real or complex vector space, a vector $\underline{v}\neq\underline{0}$ is called an eigenvector of $T$ with eigenvalue $\lambda$ if $T\underline{v}=\lambda\underline{v}$.
Note that this happens if and only if
$$A\underline{v}=\lambda\underline{v}\iff (A-\lambda I)\underline{v}=\underline{0}$$
where $A$ is the matrix of $T$.
So $\lambda$ is an eigenvalue if and only if $\det(A-\lambda I)=0$ if and only if it is a root of the polynomial
$$\chi_A(t)=\det(A-tI)$$
which is a degree $n$ polynomial.
\begin{definition}
    For $A$ an $n\times n$ matrix, the characteristic polynomial $\chi_A(t)$ is defined as $\det(A-tI)$.
\end{definition}
This gives us a way to find the eigenvalues by looking at the characteristic polynomial and to find the eigenvectors by looking at the kernel of $A-\lambda I$.
\begin{example}
    1. Let $V=\mathbb C^3$ and
    $$A=\begin{pmatrix}
        2&i\\
        -i&2
    \end{pmatrix}$$
    So $\chi_A(t)=(2-t)^2-1$, so $\lambda$ is $1$ or $3$.
    For the eigenvalue $\lambda=1$, we find that the eigenvectors are spanned by $(1,i)$, and for $\lambda=3$, the eigenvectors are spanned by $(1,-i)$.
    \\
    2. Let $V=\mathbb R^2$, consider the shear
    $$A=
    \begin{pmatrix}
        1&1\\
        0&1
    \end{pmatrix}$$
    So $\chi_A(t)=(1-t)^2$, so $\lambda=1$.
    The eigenvectors are then spanned by $(1,0)$
    \\
    3. Let $V=\mathbb C^2$, and
    $$U=\begin{pmatrix}
        \cos\theta&-\sin\theta\\
        \sin\theta&\cos\theta
    \end{pmatrix}$$
    So $\chi_U(t)=t^2-2t\cos\theta+1$, so $\lambda=e^{\pm i\theta}$, so the eigenvectors are spanned by $(1,\pm i)$ respectively.\\
    4. Consider $V=\mathbb C^n$ and $A=\operatorname{diag}(\lambda_1,\lambda_2,\ldots,\lambda_n)$, then the eigenvalues are $\lambda_i$ and the corresponding eigenvectors are spanned by $\underline{e_i}$.
\end{example}
\begin{proposition}
    1. There exists at least one eigenvalue.
    In fact, there exists $n$ eigenvalues counting multiplicity.\\
    2. The trace of the matrix equals the sum of the eigenvalues (counting multiplicity as well).\\
    3. $\det A=\chi_A(0)$.\\
    4. If $A$ is real, then non-real eigenvalues occur in conjugates.
\end{proposition}
\begin{proof}
    Trivial.
\end{proof}
\begin{definition}
    For an eigenvalue $\lambda$ of a matrix $A$, define the eigenspace $E_\lambda$ is the set of all $\underline{v}$ with $A\underline{v}=\lambda\underline{v}$, so $E_\lambda=\ker(A-\lambda I)$.
    The geometrical multiplicity of $\lambda$ is $m_\lambda=\operatorname{null}E_\lambda$, and the algebraic multiplicity $M_\lambda$ is the multiplicity of $\lambda$ as a root of $\chi_A(t)$.
    So $\chi_A(t)=(t-\lambda)f(t)$ for $f(\lambda)\neq 0$.
\end{definition}
\begin{proposition}\label{alg_ge_geom}
    $M_\lambda\ge m_\lambda$.
\end{proposition}
\begin{proof}
    Postponed to later.
\end{proof}
Note that the strict inequality can happen in many non-trivial cases.
\begin{example}
    1. Take
    $$\begin{pmatrix}
        -2&2&-3\\
        2&1&-6\\
        -1&-2&0
    \end{pmatrix}$$
    Then $\chi_A(t)=(5-t)(t+3)^2$, so $5,-3$ are the eigenvalues.
    By calculation we find that $E_5=\operatorname{span}\{(1,2,-1)^\top\}$ and $E_{-3}=\operatorname{span}\{(-2,1,0)^\top,(3,0,1)^\top\}$, so $M_5=m_5=1,M_{-3}=m_{-3}=2$.\\
    2. Take
    $$A=\begin{pmatrix}
        -3&-1&1\\
        -1&-3&1\\
        -2&-2&0
    \end{pmatrix}$$
    Then $\chi_A(t)=-(t+2)^3$, so the only eigenvalue is $-2$, but $E_{-2}$ only has dimension $2$, therefore $M_{-2}=3\neq 2=m_{-2}$.\\
    3. Consider the reflection across some plane through the origin.
    It reflects every point on the normal without changing direction and fixes every point on the plane.
    So the only eigenvalues would be $\pm 1$ and $M_{-1}=m_{-1}=1,M_1=m_1=2$.\\
    4. Consider the generic rotation through some normal.
    So the only real eigenvalue would be $1$ and $M_1=m_1=1$ since the transformation changes every vector's direction except those on the normal, which it fixes.
\end{example}
\begin{proposition}
    Let $\underline{v_1},\underline{v_2},\ldots,\underline{v_r}$ be eigenvectors of a matrix $A$ with eigenvalues $\lambda_1,\lambda_2,\ldots,\lambda_r$.
    If the eigenvalues are distinct, then the eigenvectors are linearly independent.
\end{proposition}
\begin{proof}
    Suppose $\underline{w}=\sum_ia_i\underline{v_i}$, then $(A-\lambda I)\underline{w}=\sum_ia_i(\lambda_j-\lambda)\underline{v}$.
    Let $\underline{w}=\underline{0}$, and if we can find a linear combination where not all $a_i$'s is $0$, then we pick such a linear combination $\underline{0}=\sum_ia_i\underline{v_i}$ such that the number of nonzero $a_i$'s is the least.
    WLOG in this choice $a_1\neq 0$, then $\underline{0}=(A+\lambda I)\underline{0}=\sum_ia_i(\lambda_j-\lambda_1)\underline{v}$, which contradicts the minimality of the number of nonzero $a_i$ of our choice.
\end{proof}
\begin{proof}[Alternative proof]
    Given a linear relation $\underline{0}=\sum_ia_i\underline{v_i}$, then for any $j$, the consider
    $$\underline{0}=\prod_{i\neq j}(A-\lambda_iI)\sum_{i=1}^ra_i\underline{v_i}=a_j\underline{v_j}\prod_{i\neq j}(\lambda_j-\lambda_i)$$
    So $a_j=0$.
\end{proof}
\subsection{Diagonalizability}
\begin{proposition}
    For an $n\times n$ matrix $A$ acting on $V=\mathbb R^n$ or $\mathbb C^n$, the followings are equivalent:\\
    1. There exists a basis for $V$ consisting of eigenvectors of $A$.\\
    2. There exists an $n\times n$ invertible matrix $P$ such that $P^{-1}AP=D$ where is a diagonal matrix whose entries are the eigenvalues.
\end{proposition}
If either of these conditions hold, then we say $A$ is diagonalizable.
\begin{proof}
    For any matrix $P$, $AP$ has columns $A\underline{C_i}^{(P)}$, and $DP$ has columns
    $$\lambda_i\underline{C_i}^{(P)}$$
    so what it is saying is that every column of $P$ is an eigenvector of $A$.
    So if there is a basis of eigenvectors, we can just take $P$ to be the matrix whose columns are the basis eigenvectors.
    Conversely, given such a matrix $P$, its columns are a basis for $V$.
\end{proof}
\begin{example}
    1. Consider
    $$A=
    \begin{pmatrix}
        1&1\\
        0&1
    \end{pmatrix}$$
    So we only have one eigenvector, hence $A$ is not diagonalizable.\\
    2. Consider
    $$U=
    \begin{pmatrix}
        \cos\theta&-\sin\theta\\
        \sin\theta&\cos\theta
    \end{pmatrix}$$
    So it does not have real eigenvalues, but it has $2$ complex eigenvalues with linearly independent eigenvectors, thus it is diagonalizable over $\mathbb C^n$.
    Indeed, we can let
    $$P=
    \begin{pmatrix}
        1&1\\
        -i&i
    \end{pmatrix}$$
    Then
    $$P^{-1}UP=\begin{pmatrix}
        e^{i\theta}&0\\
        0&e^{-i\theta}
    \end{pmatrix}$$
\end{example}
\begin{proposition}
    If $A$ has $n$ distinct eigenvalues, then it is disgonalizable.
\end{proposition}
\begin{proof}
    Different eigenvalues give linearly independent eigenvectors.
\end{proof}
\begin{proposition}
    $A$ is diagonalizable if and only if $M_\lambda=m_\lambda$ for each eigenvalue $\lambda$ of $A$.
\end{proposition}
\begin{proof}
    The union of the basis vectors for the eigenspaces would be a basis for the entire space since we would have $\sum_{\lambda}M_\lambda=\sum_{\lambda}m_\lambda=n$.
\end{proof}
\begin{example}
    1. Again use
    $$A=\begin{pmatrix}
        -2&2&-3\\
        2&1&-6\\
        -1&-2&0
    \end{pmatrix}$$
    We already know from previous example that $\lambda=5,3$ and $M_5=m_5=1,M_{-3}=m_{-3}=2$, so $A$ is diagonalizable.
    Indeed, we have
    $$P^{-1}AP=\begin{pmatrix}
        5&&\\
        &-3&\\
        &&-3
    \end{pmatrix},P=\begin{pmatrix}
        1&-2&3\\
        2&1&0\\
        -1&0&1
    \end{pmatrix}$$
    2. Use the previous example
    $$A=\begin{pmatrix}
        -3&-1&1\\
        -1&-3&1\\
        -2&-2&0
    \end{pmatrix}$$
    Then $\lambda=-2$ and $M_{-2}=3>m_{-2}=2$, so $A$ is not diagonalizable.
    Indeed, if it is, then there is some invertible $P$ with $A=P(-2I)P^{-1}=-2I$ which is a contradiction.
\end{example}
\begin{definition}
    Two $n\times n$ matrices $A,B$ are similar if there is some invertible $n\times n$ matrix $P$ such that $A=PBP^{-1}$.
\end{definition}
This is trivially an equivalence relation.
\begin{proposition}
    If $A,B$ are similar, then $\operatorname{tr}B=\operatorname{tr}A$ and $\det B=\det A$.
    Also $\chi_B=\chi_A$.
\end{proposition}
In fact, if two matrices are similar, then we could get from one to the other by a change of basis.
The results here then are obvious.
\begin{proof}
    $$\operatorname{tr}(B)=\operatorname{tr}(P^{-1}AP)=\operatorname{tr}(APP^{-1})=\operatorname{tr}(A)$$
    $$\det(B)=\det(P^{-1}AP)=\det(P)^{-1}\det(A)\det(P)=\det(A)$$
    $$\chi_A(t)=\det(A-tI)=\det(P^{-1})\det(A-tI)\det(P)=\det(B-tI)=\chi_B(t)$$
    As desired.
\end{proof}
Note that if $A$ is duagonalizable, then so is $B$ since they are all in the same equivalence class containing some diagonal matrix.
\begin{proof}[Proof of Theorem \ref{alg_ge_geom}]
    Choose an eigenvalue $\lambda$ of an $n\times n$ matrix $A$ with $m_\lambda=r$ and a basis $\underline{v_1},\ldots,\underline{v_r}$ for $E_{\lambda}$.
    Extend this basis to the entire space by adding vectors $\underline{w_{r+1}},\ldots,\underline{w_n}$.
    Consider the matrix $P$ with columns $\underline{C_i}(P)=\underline{v_i}$ for $i=1,\ldots,r$ and $\underline{C_a}(P)=w_a$ for $a=r+1,\ldots,n$, then $AP=PB$ where $B$ is of the form
    $$B=\left(\begin{array}{@{}c|c@{}}
        \lambda I&*\\
        \hline
        0&\hat{B}
    \end{array}\right)$$
    Note that $P^{-1}AP=B$, so $\chi_A(t)=\chi_B(t)=(\lambda-t)^r\det(\hat{B}-tI)$, so $M_\lambda\ge m_\lambda$.
\end{proof}
\subsection{Diagonalisation of Hermitian and Symmetric Matrices}
Observe that if $A$ is Hermitian, then $(A\underline{v})^\dagger\underline{w}=\underline{v}^\dagger A\underline{w}$ for any complex vectors $\underline{v},\underline{w}$.
\begin{proposition}
    Hermitian matrices have real eigenvalues and orthogonal eigenvectors for distinct eigenvalues.
    In addition, if the matrix is actually real (hence symmetric), then for each eigenvalue stated above, we can choose real eigenvectors.
\end{proposition}
\begin{proof}
    Let $A$ be a Hemitian matrix and $\lambda$ an eigenvalue of it with eigenvector $\underline{v}$, then
    $$\bar{\lambda}\underline{v}^\dagger\underline{v}=(A\underline{v})^\dagger\underline{v}=\underline{v}^\dagger A\underline{v}=\lambda\underline{v}^\dagger\underline{v}$$
    So $\lambda=\bar\lambda$, thus $\lambda$ is real.\\
    If $\lambda,\mu$ are distinct eigenvalues with eigenvectors $\underline{v},\underline{w}$, then
    $$\mu\underline{v}^\dagger\underline{w}=\underline{v}^\dagger(A\underline{w})=(A\underline{v})^\dagger\underline{w}=\lambda\underline{v}^\dagger\underline{w}\implies (\lambda-\mu)\underline{v}^\dagger\underline{w}=0$$
    So $\underline{v}^\dagger\underline{w}=0$ since $\mu\neq\lambda$, thus $\underline{v}\perp\underline{w}$.\\
    For the last part, just choose the real or imaginary part (at least one of them is nonzero) of the eigenvectors.
\end{proof}
\begin{proposition}[Gram-Schmidt Orthogonalization]
    Given a linearly independent set of vectors $\underline{w_1},\underline{w_2},\ldots,\underline{w_k}$ in $\mathbb C^n$, we can construct a set $\underline{u_1},\underline{u_2},\ldots,\underline{u_k}$ of orthonormal vectors such that $\operatorname{span}(\{\underline{u_i}\})=\operatorname{span}(\{w_i\})$.
\end{proposition}
\begin{proof}
    We do it recursively by
    $$\underline{u_1}=\frac{1}{\|\underline{v_1}\|}\underline{v_1},\forall j>1,\underline{w_j}^{(1)}=\underline{w_j}-(\underline{u_1}^\dagger \underline{w_j})\underline{u_1}$$
    and
    $$\underline{u_r}=\frac{1}{\|\underline{w_r}^{(r-1)}\|}\underline{w_r}^{(r-1)},\forall j>r,\underline{w_j}^{(r)}=\underline{w_j}^{(r-1)}-(\underline{u_r}^\dagger \underline{w_j}^{(r-1)})\underline{u_r}$$
    Which, as one can easily verify, is valid.
\end{proof}
Using Gram-Schmidt, we can convert any basis for some eigenspace to an orthonormal basis $\mathscr B_\lambda$.
If $\lambda_i$ are distinct eigenvalues of some Hermitian basis, then by choosing this orthonormal basis for all eigenspaces, we have an orthonormal set of eigenvectors.
\begin{example}
    1. Take
    $$A=\begin{pmatrix}
        2&i\\
        -i&2
    \end{pmatrix},A^\dagger=A$$
    So the eigenvalues are $1,3$ with eigenvectors $2^{-1/2}(1,i)^\top,2^{-1/2}(1,-i)^\top$.
    Note that these two eigenvectors is an orthonormal basis for $\mathbb R^2$.
    So we can choose
    $$P=\frac{1}{\sqrt{2}}\begin{pmatrix}
        1&1\\
        i&-i
    \end{pmatrix}$$
    Which is unitary and $P^{-1}AP=\operatorname{diag}(1,3)$.\\
    2. Consider
    $$A=\begin{pmatrix}
        0&1&1\\
        1&0&1\\
        1&1&0
    \end{pmatrix}$$
    which is real and symmetric thus Hermitian and have eigenvalues $-1,2$ where the algebraic multiplicity of $-1$ is $2$.
    By calculation we find that the geometric multiplicity of $-1$ is also $2$.
    We can find an orthonormal basis for $E_{-1}$ by the Gram-Schmidt process.
    For example we can have
    $$E_{-1}=\operatorname{span}\{2^{-1/2}(1,-1,0)^\top,6^{-1/2}(1,1,-2)^\top\}$$
    Also $E_{2}=\operatorname{span}\{3^{-1/2}(1,1,1)^\top\}$, thus we have found an orthonormal eigenbasis for $\mathbb R^3$.
    We can also find the corresponding (unitary hence orthogonal) $P$ which diagonalizes $A$.
\end{example}
\begin{theorem}
    Any $n\times n$ Hermitian matrix is diagonalizable by an unitary matrix.
    Equivalently, there is an orthonormal eigenbasis.
    In particular, for real and symmetric case, we can choose such that the orthonormal basis is real.
    \label{hermitian_diag}
\end{theorem}
\begin{proof}
    We have already established all of the theorem but the part of the matrix being diagonalizable, so it suffices to show that, which is sadly not examinable, but here goes:\\
    Consider a Hermitian $A=\mathbb C^n\to\mathbb C^n$.
    If $V$ is any subspace of $\mathbb C^n$ that is invariant under $A$, then for any $\underline{v_1},\underline{v_2}\in V$ we have $\underline{v_1}^\dagger (A\underline{v_2})=(A\underline{v_1})^\dagger\underline{v_2}$.
    We shall prove by induction on $m=\dim V\le n$ that such a subspace has an orthonormal basis of eigenvectors of $A$, which shall establish the result by taking $m=n$.
    As the initial step is trivial, we will show the induction step.\\
    Given $V$ as above, since we are working in $\mathbb C$, the characteristic equation always has a root, so there is some $\underline{v}\in V$ such that $A\underline{v}=\lambda\underline{v}$ for some $\lambda$.
    Let $W$ be the subspace $\{\underline{w}\in V:\underline{v}^\dagger\underline{w}=0\}$, then $\dim W=m-1<m$.
    Now
    $$\underline{w}\in W\implies \underline{v}^\dagger (A\underline{w})=(A\underline{v})^\dagger\underline{w}=\lambda\underline{v}^\dagger\underline{w}=0\implies A\underline{w}\in W$$
    So $W$ is invariant under $A$, so by induction hypothesis, there is an orthonormal basis for $W$ of eigenvectors of $A$, say $\{\underline{u_1},\ldots,\underline{u_{m-1}}\}$, then we deem $\underline{u_m}=|\underline{v}|^{-1}\underline{v}$, then $\{\underline{u_1},\ldots,\underline{u_m}\}$ gives an orthonormal basis of $V$, and the proof is done.
\end{proof}
\subsection{Quadratic Forms}
\begin{definition}
    A quadratic form is a function $\mathcal F:\mathbb R^n\to\mathbb R$ given by $\mathcal{F}(\underline{x})=\underline{x}^\top A\underline{x}$ where $A$ is a real symmetric $n\times n$ matrix.
    So $\mathcal F(\underline{x})=x_iA_{ij}x_j$.
\end{definition}
So by Theorem \ref{hermitian_diag}, we can diagonalize it by an orthogonal matrix $P$.
Then by letting $\underline{x'}=P^\top\underline{x}$, we have $\mathcal F(\underline{x})=\underline{x'}^\top D\underline{x'}$ where $D=\operatorname{diag}(\lambda_1,\lambda_2,\ldots,\lambda_n)$, so we sau that $\mathcal F$ is diagonalized.
Note that $\underline{x}=x_i'\underline{u_i}$ where $\underline{u_i}$ is the orthonormal eigenbasis.
Thus we can interpret what we have done as not changing the vector, but instead changing the basis in which we are working under.
This basis (and the corresponding coordinate axes) is called the principle axes of the quadratic form.
\begin{example}
    1. In $\mathbb R^2$, take
    $$A=\begin{pmatrix}
        \alpha&\beta\\
        \beta&\alpha
    \end{pmatrix}$$
    So we can diagonalize it by $\lambda_1=\alpha+\beta$, $\lambda_2=\alpha-\beta$ and $\underline{u_1}=2^{-1/2}(1,1)^\top,\underline{u_2}=2^{-1/2}(1,-1)^\top$
    Hence $\underline{x'}=(2^{-1/2}(x_1+x_2),2^{-1/2}(x_1-x_2))^\top$.
    For example, if we take $\alpha=3/2,\beta=-1/2$, then the quadratic form defines an ellipse.
    If we let $\beta=3/2,\alpha=-1/2$, then the quadratic from would be a hyperbola.\\
    2. In $\mathbb R^3$, by a suitable choice of basis, $\mathcal F(\underline{x})=\lambda_1x_1^2+\lambda_2x_2^2+\lambda_3^2$.
    If $\lambda_i>0$, the graph is an ellipsoid (a sphere stretched towards the directions of the axes).
    Otherwise, for example, the matrix that we have introduced in a former example
    $$A=\begin{pmatrix}
        0&1&1\\
        1&0&1\\
        1&1&0
    \end{pmatrix}$$
    has eigenvalues $-1,2$ where the algebraic multiplicity of $-1$ is $2$.
    So on the principle axes $\mathcal F_A(\underline{x})=-x_1^2-x_2^2+2x_3^2$.
    Hence it gives a hyperboloid (obtained by rotating a hyperbola).
\end{example}
It has important application when we use the Hessians as the quadratic Form.
\subsection{Cayley-Hamilton Theorem}
\begin{theorem}[Cayley-Hamilton]
    $\chi_A(A)=0$.
\end{theorem}
\begin{remark}
    If $c_0I+c_1A+\cdots+c_nA^n$, then $A(c_1I+C_2A+\cdots+c_nA^{n-1})=-c_0I$.
    So if $c_0=\det A\neq 0$, we have
    $$A^{-1}=-\frac{1}{c_0}(c_1I+C_2A+\cdots+c_nA^{n-1})$$
\end{remark}
\begin{proof}
    The general $2\times 2$ case is immediate.
    The case for diagonalizable matrices is also obvious.
    The general case can be deal with a continuity argument or using modules.
    A proof by basic algebra is also presented in lecture but is deemed not elegant enough by the author, hence is omitted.
\end{proof}


%---------------------------Radius Ratio----------------------------------
\section{Radius Ratio\label{s:tet-radius-ratio}}

This metric is commonly known as the radius ratio since it is the
normalized ratio of the radius of the inscribed sphere to the radius
of the circumsphere. Note that it is equal to the tetrahedral aspect
$\beta$ for positively-oriented tetrahedra.

The radius ratio is the quotient of these two radii normalized by $\frac{1}{3}$ so
that an equilateral tetrahedron has quality of 1:
\begin{eqnarray*}
q & = & \frac{R}{3 r} \nonumber \\
  & = & \frac { \left|
   \normvec{L_3}^2 \left( \vec L_2 \times \vec L_0 \right) +
   \normvec{L_2}^2 \left( \vec L_3 \times \vec L_0 \right) +
   \normvec{L_0}^2 \left( \vec L_3 \times \vec L_2 \right)
   \right| A}{108 V^2}.
\end{eqnarray*}

Note that if $|V| < DBL\_MIN$, we set $q = DBL\_MAX$.

\tetmetrictable{radius ratio}%
{$1$}%                  Dimension
{$[1,3]$}%              Acceptable range
{$[1,DBL\_MAX]$}%       Normal range
{$[1,DBL\_MAX]$}%       Full range
{$1$}%                  Equilateral tet
{\cite{par:93}}%        Citation
{v\_tet\_radius\_ratio}% Verdict function name


\subsection{Representing wave functions}

Wave function are of the form:

\(\cos(ax + b)\)

\(\sin(ax + b)\)

We can use the following identities:

\begin{itemize}
\item \(\cos(x)=\sin(x+\dfrac{\tau }{8})\)
\item \(\sin(-x)=-\sin(x)\)
\item \(\sin(a+b)=\sin(a)\cos(b)+\sin(b)\cos(a)\)
\end{itemize}

So we can write any function as:

\subsubsection{Using \(e\)}


\lab{Algorithms}{QR decomposition}{QR decomposition}
\label{lab:QRdecomp}

\objective{Understand how the QR algorithm works and write your own implementation.}

The QR decomposition is used to represent any matrix as the multiple of an orthogonal matrix and an upper triangular matrix. This decomposition is useful in computing least squares and is part of a common method for finding eigenvalues.

\section*{Review of Gram Schmidt}

\vspace{5mm}
\begin{theorem}[Gram-Schmidt Orthogonalization Process] Let
$\{\x_i\}_{i=1}^n$ be a basis for the inner product space $V$. Let
\[
\q_1 = \frac{\x_1}{\norm{\x_1}},
\]
and define $\q_2,\q_3,\ldots,\q_n$ recursively by
\[
\q_{k+1} = \frac{\x_{k+1} - \p_k}{\norm{\x_{k+1} - \p_k}},
\]
where
\[
\p_k = \sum^k_{j=1} \langle \x_{k+1}, \q_j\rangle \q_j
\]
is a projection of $\x_{k+1}$ onto the subspace $\mbox{Span}(\q_1,\q_2,\ldots,\q_k)$.  Then the set $\{\q_i\}_{i=1}^n$ is an orthonormal basis for $V$.
\end{theorem}
\vspace{5mm}

For the above algorithm, let $r_{k k} = \|\x_k - \p_{k-1}\|$ and
$r_{j k} = \langle \x_k, \q_j\rangle$.  Then
\begin{align*}
r_{1 1} \q_1 &= \x_1\\
r_{k k} \q_k &= \x_k - r_{1 k} \q_1 - r_{2 k} \q_2 - r_{3 k} \q_3 -
\ldots - r_{k-1, k} \q_{k-1},\quad k=2,\ldots,n.
\end{align*}
This can be written as
\begin{align*}
\x_1 &= r_{1 1} \q_1\\
\x_2 &= r_{1 2} \q_1 + r_{2 2} \q_2\\
\vdots \:\: &= \quad \vdots\\
\x_n &= r_{1 n} \q_1 + r_{2 n} \q_2 + \ldots + r_{n n} \q_{n},
\end{align*}
or in matrix form as
\[
\begin{pmatrix}
\vdots & \vdots & & \vdots\\
\x_1 & \x_2 & \cdots & \x_n\\
\vdots & \vdots & & \vdots
\end{pmatrix}
=
\begin{pmatrix}
\vdots & \vdots & & \vdots\\
\q_1 & \q_2 & \cdots & \q_n\\
\vdots & \vdots & & \vdots
\end{pmatrix}
\begin{pmatrix}
r_{1 1} & r_{1 2} & \cdots & r_{1 n}\\
0 & r_{2 2} & \cdots & r_{2 n}\\
\vdots & \vdots & \ddots & \vdots\\
0 & 0 & \cdots & r_{n n}
\end{pmatrix}.
\]
Hence if our original basis $\{\x_i\}_{i=1}^n$ correspond to column
vectors of a matrix $A$, we can likewise write the resulting
orthonormal basis $\{\q_i\}_{i=1}^n$ as a matrix $Q$ of column
vectors.  Then we have that $A = Q R$, where $R$ is the above
nonsingular upper-triangular $n\times n$ matrix.  This is the QR
Decomposition and is summarized by the following theorem:
\vspace{5mm}
\begin{theorem}
Let $A$ be an $m\times n$ matrix of rank $n$.  Then $A$ can be
factored into a product $Q R$, where $Q$ is an $m\times n$ matrix
with orthonormal columns and $R$ is a nonsingular $n \times n$ upper
triangular matrix.
\end{theorem}

There are three mode options available in SciPy's implementation of QR Decomposition.  We will be using the ``economic" option.
\begin{lstlisting}[style=python]
: import scipy as sp
: from scipy import linalg as la
: A = sp.randn(4,3)
: Q, R = la.qr(A, mode='economic')
: sp.dot(Q, R) == A     there will be some False entries
: sp.dot(Q, R) - A
: sp.dot(Q.T, Q)
\end{lstlisting}

In order to interpret the results correctly, we need to understand that the computer has limited precision (especially with floating point numbers).  This is why \li{sp.dot(Q, R)} is not exactly equal to A.  But subtracting the two yields numbers that are essentially zero.  This shows that indeed the product of $Q$ and $R$ is $A$.  Note also that $Q^T Q = I$.  This implies that the column vectors of $Q$ are orthonormal (why?).

\section*{Solving Least Squares Problems}

For large or ill-conditioned problems, the QR decomposition provides
a nice method for computing least squares solutions of
over-determined matrices.  Consider the problem $A x = b$.  Recall
that the least squares solution is $\widehat x = (A^T A)^{-1}A^T b$.
Alternatively, we write the linear system as
\[
Q R x = b.
\]
We then multiply both sides by $Q^T$, yielding
\[
R x = Q^T b.
\]
Then $\widehat x = R^{-1} Q^T b$.

\section*{Computational Remark}

Numerically, the Gram Schmidt process can have problems due to
finite precision arithmetic. Specifically, due to rounding errors,
the resulting basis may not be orthonormal. To combat this, we
actually carry out a slightly revised algorithm called Modified Gram
Schmidt.  To do this, we compute $\q_1$ as before.  We then project
it out of each of the remaining original vectors
$\x_2,\x_3,\ldots,\x_n$ via
\[
\x_k := \x_k - \langle \x_{k}, \q_1\rangle \q_1,\quad k=2,\ldots,n.
\]
Then we compute $\q_2$ to be the unit vector of $\x_2$, that is,
\[
\q_2 = \frac{\x_2}{\|\x_2\|}.
\]
We repeat by projecting out $\q_2$ from the remaining vectors
$\x_3,\x_4,\ldots,\x_n$.

\begin{problem}
\label{prob:QR}
Write your own implementation of the QR decomposition.  It should accept as input a matrix $A$ and computes its QR decomposition, returning the matrices $Q$ and $R$. Be sure to use the numerically stable Modified Gram Schmidt algorithm.
\end{problem}


\part{Inequalities}

\chapter{Piecewise Functions}

\section{Piecewise Functions}

Piecewise functions typically feature one or more points at which the function changes from one form to another. To graph a piecewise function, simply graph each piece and then restrict it to its designated domain. Pay special attention when plotting the breaking point (closed circle includes the point, open circle excludes the point). \\

\begin{exercise}\nonumber
	Graph the piecewise function given by \\
	\begin{align}
		y = \begin{cases}
			1   & x \le 0 \\
			4^x & x > 0
		\end{cases}
	\end{align}

	\begin{figure}[H]
		\centering
		\begin{tikzpicture}[scale=0.3,yscale=0.4]
			\draw[->] (-7,0) -- (7,0) node[right] {$ x $};
			\draw[->] (0,-18) -- (0,18) node[above] {$ y $};
		\end{tikzpicture}
	\end{figure}
\end{exercise}

\begin{exercise}\nonumber
	Graph the piecewise function given by \\
	\begin{align}
		y = \begin{cases}
			{1 \over 2}x + 3 & x < -2         \\
			0                & -2 \le x \le 2 \\
			x^2 - 1          & x > 2
		\end{cases}
	\end{align}

	\begin{figure}[H]
		\centering
		\begin{tikzpicture}[scale=0.5,yscale=0.3]
			\draw[->] (-10,0) -- (6,0) node[right] {$ x $};
			\draw[->] (0,-5) -- (0,20) node[above] {$ y $};
		\end{tikzpicture}
	\end{figure}
\end{exercise}

\newpage

\chapter{Absolute Value Functions}

\section{Absolute Value Functions}

A very special and common piecewise function is the absolute value function. \\

\begin{itemize}
	\item
	      $ y = |x| = \begin{cases} x & x \ge 0 \\ -x & x < 0 \end{cases} $ \\

	\item
	      $ x \in \mathbb R $ \\

	\item
	      $ y \ge 0 $ \\
\end{itemize}

\begin{figure}[H]
	\centering
	\begin{tikzpicture}[scale=0.8]
		\draw[->] (-4,0) -- (4,0) node[right] {$ x $};
		\draw[->] (0,-2) -- (0,4) node[above] {$ y $};
		\draw[very thick,color=red,domain=-3:0] plot (\x,{-\x});
		\draw[very thick,color=red,domain=0:3] plot (\x,{\x}) node[right] {$ y = |x| $};
	\end{tikzpicture}
	\caption{absolute value function}
\end{figure}

\begin{theorem}[Absolute Values]
	\begin{align}
		 & |ab| = |a| \cdot |b|                           \\
		 & \left|{a \over b}\right| = {|a| \over |b|}     \\
		 & if\ |a| \le b,\ then\ -b \le a \le b           \\
		 & if\ |a| \ge b,\ then\ a \ge b \ or \ a \le -b  \\
		 & (Triangle \ Inequality)\ |a + b| \le |a| + |b|
	\end{align}
\end{theorem}

\chapter{Inequalities Notation}

\section{Inequalities Notation}

When solving equation we may get a single answer, or a number of answers that satisfy the equation. \\

Consider $ 3x - 5 = 1 $, only one value satisfies this equation. \\

But if we consider $ x^2 - 1 = 3 $, more than one value satisfies this equation. \\

Inequalities notation like $ 1 \le x < 3 $, where the symbols $ \le $ and $ \ge $ indicate inclusion of an endpoint, and $ < $ and $ > $ indicate exclusion of an endpoint. \\

A second notation is interval notation, for example, $ x \in [1, 3) $, where a square (or closed) bracket indicates inclusion of an endpoint, and a round (or open) bracket indicates exclusion of an endpoint. \\

The infinity symbol $ \infty $ is always accompanied by round brackets. \\

\begin{exercise}\nonumber
	Write each of the following in interval notation. \\

	(a) $ 2 \le x \le 7 $ \\
	\\
	\\

	(b) $ x < 9 $ \\
	\\
	\\

	(c) $ -3 > x > 0 $ \\
	\\
	\\
\end{exercise}

\begin{exercise}\nonumber
	Write each of the following using inequalities. \\

	(a) $ x \in [3, 6) $ \\
				\\
				\\

				(b) $ x \in (-2, 4) $ \\
				\\
				\\

				(c) $ x \in (-\infty, -1] $ \\
	\\
	\\
\end{exercise}

It is possible to have ranges of values that are disjoint. We use the union symbol $ \cup $ to include all of the values in any of the disjoint ranges. For example, $ [-1, 4) \cup [7, 10) $ meas $ -1 \le x < 4 $ or $ 7 \le x < 10 $. \\

\begin{exercise}\nonumber
	Express each of the following in interval notation. \\

	(a) $ -3 \le x < {1 \over 2} $ or $ 4 < x < 7 $ \\
	\\
	\\

	(b) $ 1 \le x < 5 $ or $ 3 \le x < 7 $ \\
	\\
	\\

	(c) $ x \in [-2, 6) \bigcup (0, 5) $ \\
	\\
	\\
\end{exercise}

Intersection symbol $ \cap $ allows only the values that are common between intervals. For example, $ [-1, 6) \cap (2, 7) $ means $ (2, 6) $. \\

\begin{exercise}\nonumber
	Express each of the following in interval notation. \\

	(a) $ -2 < x \le 6 $ and $ 0 < x < 7 $ \\
	\\
	\\

	(b) $ x \in [0, 5] \bigcap [3, 5] $ \\
	\\
	\\

	(c) $ -4 < x < 0 $ and $ 3 < x < 7 $ \\
	\\
	\\
\end{exercise}

\section{Solving Inequalities}

When solving inequalities, there are a few rules that we must follow: \\

\begin{enumerate}
	\item
	      When it comes to addition, subtraction, multiplication, and division, what you do to one side of the inequality, you must do to the other. \\

	\item
	      If you multiply or divide by a negative quantity, you must flip the inequality. \\

	\item
	      If both sides are positive or both sides are negative, then you can take the reciprocal of both sides, but you must flip the inequality. \\
\end{enumerate}

\begin{exercise}\nonumber
	Find all values of x that satisfy the following. \\

	(a)
	\begin{align}
		-6x + 7 & \ge 8x \\
		\\
		\\
	\end{align} \\

	(b)
	\begin{align}
		-{5 \over 2} < 4 - 2x \le 1 \\
		\\
		\\
		\\
	\end{align} \\

	(c)
	\begin{align}
		5x^3 + 27 & > -13 \\
		\\
		\\
		\\
	\end{align} \\

	(d)
	\begin{align}
		3x^2 + 2 & < -4 \\
		\\
		\\
		\\
	\end{align} \\

	(e)
	\begin{align}
		\sqrt{x-1} & > 4 \\
		\\
		\\
	\end{align} \\

	(f)
	\begin{align}
		log_2(3x) & \le -3 \\
		\\
		\\
		\\
	\end{align}
\end{exercise}

\chapter{The Case Method}

\section{The Case Method}

Consider the following example: $ {x - 3 \over x - 1} < 10 $ \\

You might be tempted to cross multiply, but be careful! The quantity $ x - 1$ is not always positive. If we multiply by $ x - 1$, the inequality needs to flip for some values of $ x $. How might we deal with this? \\

\begin{enumerate}
	\item
	      Separate into 2 cases \\
	      \begin{table}[H]
		      \centering
		      \begin{tabular}{|c|c|} \hline
			      \textbf{Case 1} & \textbf{Case 2}                \\ \hline
			      $ \begin{array}{ccc} x - 1 & > & 0 \\ x & > & 1 \end{array} $
			                      & $ \begin{array}{ccc} x - 1 & < & 0 \\ x & < & 1 \end{array} $ \\ \hline
		      \end{tabular}
	      \end{table}

	\item
	      Solve the original problem under each assumption \\
	      \begin{table}[H]
		      \centering
		      \begin{tabular}{|c|c|} \hline
			      \textbf{Case 1} & \textbf{Case 2}                \\ \hline
			      $ \begin{array}{ccc} {x-3 \over x-1} & < & 10 \\ x - 3 & < & 10(x-1) \\ -9x & < & -7 \\x & > & {7 \over 9} \end{array} $
			                      & $ \begin{array}{ccc} {x-3 \over x-1} & < & 10 \\ x - 3 & > & 10(x-1) \\ -9x & > & -7 \\ x & < & {7 \over 9} \end{array} $ \\ \hline
		      \end{tabular}
	      \end{table}

	\item
	      Find all common points between assumption and solution \\
	      \begin{table}[H]
		      \centering
		      \begin{tabular}{|c|c|} \hline
			      \textbf{Case 1} & \textbf{Case 2}                \\ \hline
			      $ \begin{array}{ccc} x > 1 \ and \ x > {7 \over 9} \\ x \in (1, \infty) \end{array} $
			                      & $ \begin{array}{ccc} x < 1 \ and \ x < {7 \over 9} \\ x \in \left(-\infty, {7 \over 9}\right) \end{array} $ \\ \hline
		      \end{tabular}
	      \end{table}

	\item
	      Consolidate the 2 cases by taking the union \\
	      $$
		      x \in (1, \infty) \cup \left(-\infty, {7 \over 9}\right)
	      $$
\end{enumerate}

\begin{exercise}\nonumber
	Find all values of x such that $ {7x - 2 \over 1 - 2x} \ge 4 $. \\

	\begin{enumerate}
		\item
		      Separate into 2 cases \\
		      \begin{table}[H]
			      \centering
			      \begin{tabular}{|c|c|} \hline
				      \textbf{\hspace{2cm}Case 1\hspace{2cm}} & \textbf{\hspace{2cm}Case 2\hspace{2cm}} \\ \hline
			      \end{tabular}
		      \end{table}
		      \vspace{2cm}

		\item
		      Solve the original problem under each assumption \\
		      \begin{table}[H]
			      \centering
			      \begin{tabular}{|c|c|} \hline
				      \textbf{\hspace{2cm}Case 1\hspace{2cm}} & \textbf{\hspace{2cm}Case 2\hspace{2cm}} \\ \hline
			      \end{tabular}
		      \end{table}
		      \vspace{3cm}

		\item
		      Find all common points between assumption and solution \\
		      \begin{table}[H]
			      \centering
			      \begin{tabular}{|c|c|} \hline
				      \textbf{\hspace{2cm}Case 1\hspace{2cm}} & \textbf{\hspace{2cm}Case 2\hspace{2cm}} \\ \hline
			      \end{tabular}
		      \end{table}
		      \vspace{3cm}

		\item
		      Consolidate the 2 cases by taking the union \\
		      \\
	\end{enumerate}
\end{exercise}

\chapter{The Number Line Method}

\section{The Number Line Method}

Another method for solving inequalities uses the following basic logic: \\

\begin{align}
	\nonumber
	(+)(+) = + &  & {(+) \over (+)} = + \\
	\nonumber
	(-)(-) = + &  & {(-) \over (-)} = + \\
	\nonumber
	(+)(-) = - &  & {(+) \over (-)} = - \\
	\nonumber
	(-)(+) = - &  & {(-) \over (+)} = -
\end{align}

By manipulating expressions into factors that are multiplied and/or divided on one side of the inequality (with a zero appearing on the other side), we can simply consider the combinations of positive and negative factors to draw conclusions. \\

\begin{exercise}\nonumber
	Find all values of x that satisfy $ 3x^2 - 13x > -10 $. \\

	\begin{align}
		\\
		\\
		\\
		\\
	\end{align}

	\begin{figure}[H]
		\centering
		\begin{tikzpicture}
			\draw[->] (-5.5,0)--(5.5,0);
			\foreach \x in {-5,-4,...,5}
			\draw (\x,0.2)--(\x,0);
			\draw[very thick] (-2,0)--(-2,1)--(-6,1) node[above] {$ (-)(-) $};
			\draw[very thick] (2,0)--(2,1)--(6,1) node[above] {$ (+)(+) $};
		\end{tikzpicture}
	\end{figure}

	\vspace{1cm}
\end{exercise}

\begin{exercise}\nonumber
	Find all values of x that satisfy $ x - 2 \ge {4 \over x+1} $. \\

	\begin{align}
		\\
		\\
		\\
		\\
	\end{align}

	\begin{figure}[H]
		\centering
		\begin{tikzpicture}
			\draw[->] (-5.5,0)--(5.5,0);
			\foreach \x in {-5,-4,...,5}
			\draw (\x,0.2)--(\x,0);
			\draw[very thick] (-3,0)--(-3,1)--(-1,1)--(-1,0) node[below] {$ {(-)(+)} \over (-) $};
			\draw[very thick] (2,0)--(2,1)--(4,1)--(4,0) node[below] {$ {(+)(+)} \over (+) $};
		\end{tikzpicture}
	\end{figure}

	\vspace{1cm}
\end{exercise}