\chapter{Sparse Kernel Machines}
\label{chap:Sparse Kernel Machines}
\section{Introduction}
One of significant limitations of kernel methods is that the kernel function $\mathcal{k}(\vec{x}_n,\vec{x}_m)$ must be evaluated for all possible pairs $\vec{x}_n$ and $\vec{x}_m$ of training points,computationally infeasible.Kernel-based algorithms that have \textbf{sparse} solutions predict for new inputs depend only on the kernel function evaluated at a subset of the training data points,such as \textbf{suport vector machine}(SVM).
\section{Maximum Margin Classifiers}
Two-class classification problem using linear models
\begin{align}\label{eqn:maximum margin classifier regpresentation}
y(x) = \vec{w}^T\phi(x)+b
\end{align}
where $\phi(x)$ denotes a fixed feature-space transformation,and $b$ is bias parameter.

For linear separable feature space,the parameters satisfies $y(\vec{x}_n)>0$ for points having $t_n=+1$ and $y(\vec{x}_n)<0$ for $t_n=-1$,so that $t_ny(\vec{x}_n)>0$ for all points.

\textbf{Margin} is the smallest distance between the decision boundary and any of the samples.The maximum margin solution can be motivated using \textbf{computational learning theory},also known as \textbf{statistical learning theory}.
\begin{figure*}

	\includegraphics{prml/Figure7.1a.jpg}
	\includegraphics{prml/Figure7.1b.jpg}
	\caption{margin}
\end{figure*}

The \textbf{funtional margin} $t_ny(\vec{x}) > 0$ for data points correctly classified.The maximize it
\begin{align}
\vec{w},b = \arg\max\limits_{\vec{w},b}\{\dfrac{1}{\parallel\vec{w}\parallel}\min_n
[t_n(\vec{w}^T\phi(\vec{x}_n)+b)] \}
\end{align}
We can rescale parameters to set
\begin{align}
t_n(\vec{w}^T\phi(\vec{x}_n)+b) &= 1
\end{align}
for point closet to the surface.Then all data points satisfies the constraint
\begin{align}\label{ineqn:margin constraint}
t_n(\vec{w}^T\phi(\vec{x}_n)+b) \geq 1,n=1,...,N.
\end{align}
This is the \textbf{canonical representation of the decision hyperplane}.The optimization problem is equivalent to
\begin{align}
\arg\min\limits_{\vec{w},b}\dfrac{1}{2}\parallel\vec{w}\parallel^2
\end{align}
subject to constraints \ref{ineqn:margin constraint},which is a \textbf{quadratic programming} problem.

Introducing Lagrange multipliers $a_n\geq 0$
\begin{align}
L(\vec{w},b,\vec{a}) = \dfrac{1}{2}\parallel\vec{w}\parallel^2-
\sum_{n=1}^{N}a_n\{ t_n(\vec{w}^T\phi(\vec{x}_n)+b)-1 \}
\end{align}
where $\vec{a} = (a_1,...,a_N)^T$.Setting the derivatives of $L$ with respect to $\vec{w}$ and $b$ equal to zero,we obtain conditions
\begin{align}
\vec{w}&=\sum_{n=1}^{N}a_n t_n\phi(\vec{x}_n) \\
0 &=\sum_{n=1}^{N}a_n t_n
\end{align}
Eliminating $\vec{w}$ and $b$ gives the \textbf{dual representation} of the maximum margin problem
\begin{align}
\hat{L}(\vec{a})=\sum_{n=1}^{N}a_n -\dfrac{1}{2}\sum_{n=1}^{N}\sum_{m=1}^{N}a_n a_m t_n t_m k(\vec{x}_n,\vec{x}_m)
\end{align}
with respect to $\vec{a}$ subject to the constraints
\begin{align}
a_n &\geq 0,n =1,...,N \\
\sum_{n=1}^{N}a_n t_n &= 0.
\end{align}
The kernel function is defined by $k(\vec{x},\vec{x}') = \phi(\vec{x})^T\phi(\vec{x}')$.

The solution to a quadratic programming problem in $M$ variables has computational complexity of $O(M^3)$.

$y(\vec{x})$ can be expressed by
\begin{align}
y(\vec{x}) = \sum_{n=1}^{N}a_n t_n k(\vec{x},\vec{x}_n) + b.
\end{align}
The $Karush-Kuhn-Tucker(KKT)$ conditions require following the properties hold
\begin{align}
a_n \geq 0 \\
t_n y(\vec{x}_n) -1 &\geq 0 \\
a_n\{t_n y(\vec{x}_n) -1 &= 0\}
\end{align}
Data points for which $a_n=0$ will disappear and remaining ones are called \textbf{suport vectors}.They lie on the maximum margin hyperplanes in feature space.

Having solved the quadratic programming problem,the threshold parameter $b$
\begin{align}
t_n(\sum_{m\in \mathcal{S}}{a_m t_m k(\vec{\vec{x}_n,\vec{x}_m})+b}) =1
\end{align}
where $\mathcal{S}$ denotes the set of indices of the support vectors.Multiply through by $t_n$,making use of $t_n2=1$,and then average these equations over all support vectors
\begin{align}
b=\dfrac{1}{N_{\mathcal{S}}}\sum_{n\in\mathcal{S}}(t_n-\sum_{m\in\mathcal{S}}a_m t_m \mathcal{k}(\vec{x}_n,\vec{x}_m))
\end{align}
where $N_{\mathcal{S}}$ is the total number of support vectors.
Express the maximum-margin classifier in terms of the minimization of an error function with a quadratic regularizer
\begin{align}
\sum_{n=1}^{N}E_{\infty}(y(\vec{x}_n)t_n-1) + \lambda\parallel\vec{w}\parallel^2
\end{align}
where $E_{\infty}(z)$ is zero if $z\geq 0$ and $\infty$ otherwise to ensure the margin constraint.

\subsection{Overlapping class distributions}
In practice,the class-conditional distributions may overlap,in which case exact separation of the training data can lead to poor generalization.Introduce \textbf{slack variables}, $\xi_n\geq 0$ where $n=1,...N$,one for each training data points.We allow points on the 'wrong side' but with a penalty that increases of the distance from the boundary.
\begin{align}
\xi_n = \mid t_n-y(\vec{x}_n)\mid
\end{align}
\begin{SCfigure*}
	\caption{slack variables $\epsilon_n\geq 0$.Data points with circles around them are support vectors}
	\includegraphics[]{prml/Figure7.3.jpg}
\end{SCfigure*}

Then the classification constraint are replaced by
\begin{align}
t_n y(\vec{x}_n) \geq 1-\xi_n,n=1,...,N
\end{align}
This is described as relaxing the hard margin constraint to give a \textbf{soft margin} and allows misclassification of training set data points.

Maximize the margin with softly penalized points on the wrong side of the margin boundary.
\begin{align}
C\sum_{n=1}^{N}\xi_n +\dfrac{1}{2}\parallel\vec{w}\parallel^2
\end{align}
where the parameter $C$ controls the trade-off between the slack variable penalty and the margin,minimizing errors and controlling model complexity.In the limit $C\longrightarrow \infty$,the model gets more complex,less data points are misclassified.

Minimization with constraint
\begin{align}
L(\vec{w},b\vec{a}) =\dfrac{1}{2}\parallel\vec{w}\parallel^2+C\sum_{n=1}^{N}\xi_n-\sum_{n=1}^{N}a_n\{t_n y(\vec{x}_n)-1+\xi_n \} -\sum_{n=1}^{N}\mu_n\xi_n
\end{align}
where $\{a_n\geq 0\}$ and $\{\mu_n \geq 0 \}$ are Lagrange multipliers.The KKT set of conditions are given by
\begin{align}
a_n &\geq 0 \\
t_n y(\vec{x}_n)-1+\xi_n &\geq 0 \\
a_n(t_n y(\vec{x}_n)-1+\xi_n) &= 0 \\
\mu_n &\geq 0 \\
\xi_n &\geq 0 \\
\mu_n\xi_n &=0
\end{align}
where $n=1,...,N$.

Optimize out $\vec{w},b$ and $\{\xi_n\}$
\begin{align}
\dfrac{\partial L}{\partial\vec{w}} =0 &\Rightarrow \vec{w}=\sum_{n=1}^{N}a_n t_n \\
\dfrac{\partial L}{\partial b}=0 &\Rightarrow \sum_{n=1}^{N}a_n t_n =0 \\
\dfrac{\partial L}{\partial \xi_n} =0 &\Rightarrow a_n = C-\mu_n
\end{align}
Eliminated,the dual Lagrangian is in the form
\begin{align}\label{eqn:SVM Lagrangian}
\hat{L}(\vec{a}) = \sum_{n=1}^{N}a_n -\dfrac{1}{2}\sum_{n=1}^{N}\sum_{m=1}^{N}a_n a_m t_n t_m \mathcal{k}(\vec{x}_n,\vec{x}_m)
\end{align}
which is identical to the separable case,with different constraints:
\begin{align}
0\leq a_n \leq C\\
\sum_{n=1}^{N}a_n t_n = 0
\end{align}
for $n=1,...,N$,where the former are known as \textbf{box constraints}.

A subset of data points having $a_n =0$ do not contribute to the predictive model.Support vectors have $a_n > 0$ and satisfy
\begin{align}
t_n y(\vec{x}_n) &= 1-\xi_n
\end{align}
if $a_n <C$,then implies that $\mu_n > 0$,which requires $\xi_n =0$ and hence such points lie on the margin.Points with $a_n=C$ can lie inside the margin and can either be correctly classified if $\xi_n \leq 1$ or misclassified if $\xi_n >1$.

To determine $b$,we note that support vectors for which $0\leq a_n \leq C$ have $\xi_n =0$ so that $t_n y(\vec{x}_n)=1$ and hence satisfy
\begin{align}
t_n(\sum_{m\in\mathcal{S}}a_m t_m \mathcal{k}(\vec{x}_n,\vec{x}_m)+b) =1
\end{align}
A numerically stable solution is obtained by averaging:
\begin{align}
	b=\dfrac{1}{N_{\mathcal{M}}}\sum_{n\in\mathcal{M}} (t_n - \sum_{m\in\mathcal{S}}a_m t_m \mathcal{k}(\vec{x}_n,\vec{x}_m))
\end{align}
where $\mathcal{M}$ denotes the set of indices of data points having $0\leq a_n \leq C$.

\subsection{Multiclass Support Vector Machine loss}
the score for the $j$-th class is the $j$-th element: $\vec{s}_j=f(\vec{x}_i,\vec{W})_j$. The Multiclass SVM loss for the $i$-th example is then formalized as follows:
\begin{align}
    L_i = \sum_{j\neq y_i} \max(0, \vec{s}_j - \vec{s}_{y_i} + \Delta)
\end{align}
With linear score function $f(\vec{x}_i; \vec{W}) =  \vec{W} \vec{x}_i$, rewrite the loss function in this equivalent form:
\begin{align}
    L_i = \sum_{j\neq y_i} \max(0, \vec{w}_j^T x_i - \vec{w}_{y_i}^T x_i + \Delta)
\end{align}
The threshold at zero $max(0,-)$ is called \textbf{hinge loss}.And there is a squared hinge loss (L2-SVM).
The gradient of loss function for a single point $i$:
\begin{align}
    \nabla_{w_{y_i}} L_i = - \left( \sum_{j\neq y_i} \mathbb{1}(w_j^Tx_i - w_{y_i}^Tx_i + \Delta > 0) \right) x_i
\end{align}
where $\mathbb{1}$ is the indicator function that is one if the condition inside is true or zero otherwise.
For the rows where $j \neq y_i$ the gradient is:
\begin{align}
    \nabla_{w_j} L_i = \mathbb{1}(w_j^Tx_i - w_{y_i}^Tx_i + \Delta > 0) x_i
\end{align}

\subsubsection{$\nu$-SVM}
Maximize
\begin{align}
\hat{L}(\vec{a})=-\dfrac{1}{2}\sum_{n=1}^{N}\sum_{m=1}^{N}a_n a_m t_n t_m\mathcal{k}(\vec{x}_n,\vec{x}_m)
\end{align}
subject to the constraints
\begin{align}
0\leq a_n \leq 1/N\\
\sum_{n=1}^{N}a_n t_n = 0 \\
\sum_{n=1}^{N}a_n \geq \nu
\end{align}
The parameter $\nu$,which replaces $C$,can be interpreted as both an upper bound on the fraction of \textbf{margin errors} and a lower bound on the fraction of support vectors.

\subsubsection{Optimization}
\begin{description}
	\item[\textbf{chunking}]Lagrangian is unchanged if we remove the rows and columns of the kernel matrix corresponding to Lagrange multipliers.Implemented using \textbf{protected conjugate gradients}.
	\item[\textbf{Decomposition methods}] solves a series of smaller quadratic programming problems but are designed so that each of these is of a fixed size.
	\item[\textbf{sequantial minimal optimization} or SMO] Takes the concept of chunking to the extreme limit and considers just two Lagrange multipliers at a time.Heuristics are given for choosing the pair of Lagrange multipliers to be considered at each step.
\end{description}

Support vector machines don't manage to avoid the curse of dimensionality because there are constraints amongst the feature values that restrict the effective dimensionality of feature space.

\subsubsection{Relation to Logistic Regression}
The objective function can be written
\begin{align}
\sum_{n=1}^{N}E_{SV}(y_n t_n) +\lambda\parallel\vec{w}\parallel^2
\end{align}
where $\lambda=(2C)^{-1}$,and $E_{SV}(\cdot)$ is the \textbf{hinge} error function defined by
\begin{align}
E_{SV}(y_n t_n)= [1-y_n t_n]_{+}
\end{align}
where $[\cdot]_+$ denotes the positive part.

For logistic regression,target variable $t\in \{-1,1\}$,so
\begin{align}
p(t|y)=\sigma(yt)
\end{align}
Construct an error function by taking the negative logarithm of likelihood function with a quadratic regularizer
\begin{align}
\sum_{n=1}^{N}E_{LR}(y_n t_n)+\lambda\parallel\vec{w}\parallel^2
\end{align}
where
\begin{align}
E_{LR}(yt)=\ln(1+\exp(-yt))
\end{align}
\begin{SCfigure*}
	\caption{Plot of the ‘hinge’ error function used in support vector machines, shown in blue, along with the error function for logistic regression, rescaled by a factor of 1/ ln(2) so that it passes through the point (0, 1), shown in red. Also shown are the misclassification error in black and the squared error in green.}
	\includegraphics{prml/Figure7.5.jpg}
\end{SCfigure*}

\subsubsection{Multiclass SVMs}
\begin{description}
	\item[\textbf{one-versus-the-rest}] approach:$K$ separate SVMs for each class.
	\item[\textbf{one-versus-one}].$K(K-1)/2$ different 2-class SVMs on possible pairs of classes,which can lead to ambiguities.
	\item[\textbf{single-class}] support vector machines,which solve an unsupervised learning problem related to probability density estimation.
\end{description}

\subsubsection{SVMs for regression}
In simple linear regression we minimize a regularized error function given by
\begin{align}
\dfrac{1}{2}\sum\limits_{n=1}^{N}\{y_n-t_n\}^2+\dfrac{\lambda}{2}\parallel\vec{w}\parallel^2.
\end{align}
To obtain \textbf{sparse solutions},the quadratic error function is replaced by an \textbf{$\epsilon$-insensitive error function}.For example
\begin{align}
E_{\epsilon}(y(\vec{x})-t) = \begin{cases}
0,&\text{if}\mid y(\vec{x})-t\mid < \epsilon;\\
y(\vec{x})-t\mid - \epsilon,&\text{otherwise}
\end{cases}
\end{align}
We therefore minimize
\begin{align}
C\sum\limits_{n=1}^{N}E_{\epsilon}(y(\vec{x}_n)-t_n)+\dfrac{1}{2}\mid\vec{w}\mid^2
\end{align}
where $y(\vec{x})$ is the prediction function.The (inverse) regularization parameter denoted $C$,appears in front of the error term.

Introducing two \textbf{slack variables} for each data point $\vec{x}_n$.The condition for target points to lie inside the $\epsilon$-tube is that $y_n -\epsilon \leq t_n \leq y_n+\epsilon$,and for those outside:
\begin{align}
&\begin{cases}
\xi_n &\geq 0\\
\hat{\xi_n} &\geq 0,n=1,...,N\\
\end{cases}\\
& y(\vec{x}_n)+\epsilon < t_n \leq y(\vec{x}_n)+\epsilon +\xi_n \\
& y(\vec{x}_n)-\epsilon > t_n \geq y(\vec{x}_n)-\epsilon -\hat{\xi_n}
\end{align}
\begin{SCfigure*}
	\caption{Illustration of SVM regression, showing the regression curve together with the $\epsilon$-insensitive tube. }
	\includegraphics[]{prml/Figure7.7.jpg}
\end{SCfigure*}

The error function for support vector regression can then be written as
\begin{align}
C\sum\limits_{n=1}^{N}(\xi_n+\hat{\xi_n})+\dfrac{1}{2}\parallel\vec{w}\parallel^2
\end{align}
which must be minimized subject to the constraints.Introducing Lagrange multipliers
\begin{align}\label{eqn:svm regression Lagrangian}
L =& C\sum\limits_{n=1}^{N}(\xi_n+\hat{\xi_n})+\dfrac{1}{2}\parallel\vec{w}\parallel^2
- \sum\limits_{n=1}^{N}(\mu_n\xi_n+\hat{\mu_n}\hat{\xi_n}) \\
&-\sum\limits_{n=1}^{N}a_n(\epsilon+\xi_n+y_n-t_n) -\sum\limits_{n=1}^{N}\hat{a_n}(\epsilon+\hat{\xi_n}+y_n-t_n)
\end{align}
Substitute for $y(\vec{x})$ using \ref{eqn:maximum margin classifier regpresentation} and set the derivatives of the Lagrangian with respect to $\vec{x},b,\xi_n,\hat{\xi_n}$ to zero,giving
\begin{align}
\dfrac{\partial L}{\partial\vec{w}} =0 &\Rightarrow \vec{w}=\sum_{n=1}^{N}(a_n-\hat{a_n})\phi(\vec{x}_n) \\
\dfrac{\partial L}{\partial b} =0 &\Rightarrow \sum_{n=1}^{N}(a_n-\hat{a_n})=0 \\
\dfrac{\partial L}{\partial\xi_n} =0 &\Rightarrow a_n+\mu_n=0 \\
\dfrac{\partial L}{\partial\hat{\xi_n}} =0 &\Rightarrow \hat{a_n}+\hat{\mu_n}=0 \\
\end{align}
Using these to eliminate the corresponding variables,we see the dual problem of maximizing
\begin{align}
\hat{L}(\vec{a},\hat{\vec{a}}) =&-\dfrac{1}{2}\sum_{n=1}^{N}\sum_{m=1}^{N}(a_n-\hat{a_n})(a_m-\hat{a_m})\mathcal{k}(\vec{x}_n,\vec{x}_m) \\
 &-\epsilon\sum_{n=1}^{N}(a_n+\hat{a_n})+\sum_{n=1}^{N}(a_n-\hat{a_n})t_n
\end{align}
with respect to $\{a_n\},\{\hat{a_n}\}	$.We have the \textbf{box constraints}
\begin{align}
0\leq a_n \leq C\\
0\leq \hat{a_n}\leq C
\end{align}
After substitution,the predictions can be made using
\begin{align}
y(\vec{x})=\sum_{n=1}^{N}(a_n-\hat{a_n})\mathcal{k}(\vec{x},\vec{x}_n)+b
\end{align}

The corresponding $karush-Kuhn-Tucker$(KKT) conditons for \ref{eqn:svm regression Lagrangian},which state that \textbf{at the solution the product of the dual variables and the constraints must vanish} are given by
\begin{align}
a_n(\epsilon+\xi_n+y_n-t_n) &= 0\\
\hat{a_n}(\epsilon+\hat{\xi_n}-y_n+t_n) &= 0\\
(C-a_n)\xi_n & = 0\\
(C-\hat{a_n})\hat{\xi_n} &=0
\end{align}

The support vectors are those data points that contribute to predictions,in other words those for which either $a_n \neq 0$ or $\hat{a_n} \neq 0$.These are points lying on the boundary of the $\epsilon$-tube or outside the tube.

The parameter $b$ satisfies
\begin{align}
\epsilon+y_n-t_n= 0
\end{align}
for points $0<a_n<C$.Solving for it
\begin{align}
b &= t_n-\epsilon-\vec{w}^T\phi(\vec{x}_n) \\
  &= t_n-\epsilon-\sum_{m=1}^{N}(a_m-\hat{a_m})\mathcal{k}(\vec{x}_n,\vec{x}_m)
\end{align}

An alternative formulation of SVM regression is $\nu$ SVM.
\begin{align}
\hat{L}(\vec{a},\hat{\vec{a}}) =&-\dfrac{1}{2}\sum_{n=1}^{N}\sum_{m=1}^{N}(a_n-\hat{a_n})(a_m-\hat{a_m})\mathcal{k}(\vec{x}_n,\vec{x}_m) \\
&-0\times\epsilon\sum_{n=1}^{N}(a_n+\hat{a_n})+\sum_{n=1}^{N}(a_n-\hat{a_n})t_n
\end{align}
subject to constraints
\begin{align}
0\leq a_n &\leq C/N\\
0\leq \hat{a_n} &\leq C/N \\
\sum_{n=1}^{N}(a_n-\hat{a_n}) &=0 \\
\sum_{n=1}^{N}(a_n+\hat{a_n}) &\leq \nu C \\
\end{align}
There are at most $\nu N$ data points falling outside the insensitive tube,which at least $\nu N$ data points are support vectors and so lie either on the tube or outside it.

\section{Relevance Vector Machines}
TODO


\subsection{Soft-margin SVC}

\subsubsection{Soft margin}

Soft margin

Data may not be linearly separable, so we introduce a hinge loss function

\(Max(0, 1-y_i(wx-b))\)

We then minimise

\(\lambda ||w||^2+ [\dfrac{1}{n}\sum_{i=1}^n \max (0, 1-y_i(wx_i-b)]\)

This introduces \(\lambda\) as a parameter.


\chapter{CT Frequency Response}

In this lecture we are going to focus on the frequency response and highlight it's importance in linear systems theory.

\section{Determining the frequency response (FR) of a CT system}

The frequency response of a CT LTI system can be thought of as arising in several equivalent ways. What follows is a common, but not exhaustive, list of ways the frequency response can be derived from other representations.

\subsection*{Using the Eigenvalues / Transfer Function}

Recall if we apply the Eigenfunction $e^{st}$ for the complex frequency $s \in \mathbb{C}$ as the input to a LTI system, the output is the Eigenfunction scaled by the Eigenvalue (transfer function) $H(s)$ for values of $s$ in the region of convergence, where
\[
H(s) = \int\limits_{-\infty}^{\infty} h(t) e^{st}\; dt \; .
\]
is the bilateral Laplace transform of the impulse response.

\begin{center}
  \includegraphics[scale=1]{graphics/18-ct-tf.pdf}
\end{center}

If a system is stable, then the region of convergence includes the imaginary axis $s = j\omega$. In that case, evaluating the Eigenvalues on the imaginary axis $s = j\omega$ gives the CT frequency response $H(j\omega)$. This converts from a function of a complex variable, $s$, to one of a real variable $\omega$.

\begin{example} Consider a system with Eigenvalues (transfer function)
  \[
  H(s) = \frac{2}{s+5}\mbox{ for } \Re{s} > -5
  \]
  Determine the frequency response of the system, if possible.\\

  Solution: We first need to check of the system is stable using the region-of-convergence. Since the real part of the region of convergence includes the imaginary axis ($\Re s = 0$), the system is stable. To find the frequency response we substitute $s = j\omega$ to give
  \[
  H(j\omega) = \frac{2}{j\omega+5}
  \]
  \\$\blacksquare$
\end{example}

\begin{example} Consider an apparently similar system with Eigenvalues
  \[
  H(s) = \frac{2}{s-5}\mbox{ for } \Re{s} > 5
  \]
  Determine the frequency response of the system, if possible.\\

  Solution: Again, we first need to check of the system is stable using the region-of-convergence. Since the real part of the region of convergence does not include the imaginary axis ($\Re s = 0$), the system is unstable. Thus, the frequency response does not exist.
  \\$\blacksquare$
\end{example}

\subsection*{Using the CTFT}

Another way we can view the frequency response is as the CT Fourier Transform of the impulse response. If the system is stable, then the impulse response is absolutely integrable, and the Fourier transform exists giving $H(j\omega) = \mathcal{F}\left\{h(t)\right\}$. This is connected to the transfer function by noting the bilateral Laplace transform and the Fourier Transform are identical under the substitution $s = j\omega$, which is allowed if the system is stable.

\begin{example} Suppose the impulse response of a CT LTI system is given by
  \[
  h(t) = \left(e^{-t}-e^{-6t}\right)u(t)
  \]
  Determine the frequency response of the system, if possible.\\

  Solution: If the system is stable, the Fourier transform of the impulse response exists. Since
  \[
  \int\limits_{0}^{\infty} \left| e^{-t}-e^{-6t} \right| \; dt < \int\limits_{0}^{\infty} e^{-t} \; dt < \infty
  \]
  the system is stable and the Fourier Transform exists, giving
  \[
H(j\omega) = \mathcal{F}\left\{ \left(e^{-t}-e^{-6t}\right)u(t) \right\} = \mathcal{F}\left\{ \left(e^{-t}u(t) \right\} - \mathcal{F}\left\{e^{-6t}\right)u(t) \right\} = \frac{1}{j\omega + 1} - \frac{1}{j\omega + 6} = \frac{7}{6-\omega^2 + j7\omega}
\]\\
$\blacksquare$
\end{example}

\subsection*{Directly from a LCCDE}

By the convolution theorem of the CTFT, the frequency response is the ratio of the output to input in the frequency domain, i.e.
\[
H(j\omega) = \frac{Y(j\omega)}{X(j\omega)}
\]
We can easily determine this ratio from the LCCDE representation of the system using the derivative property of the Fourier Transform. Recall this property states if $\mathcal{F}\{x(t)\} = X(j\omega)$ then
\[
\mathcal{F}\left\{\frac{d^n x}{dt^n}(t) \right\} = (j\omega)^n  X(j\omega) \; .
\]

If the system is stable (and thus the frequency response exists) then \textbf{all} roots of the characteriztic equation $Q(D)$ have real parts that are less than zero. If the system is stable we can take the Fourier transform of each term of the LCCDE using the derivative property, then algebrically solve for the ratio of output to input. Note this provides a signifigant savings in analysis effort since we do not have to first find the impulse response, then take it's Fourier transform to arrive at the frequency response (although that approach is still valid).

\begin{example} Consider a sytem decribed by the LCCDE
  \[
\frac{d^2y}{dt^2}(t) + 15\frac{dy}{dt}(t) + 50y(t) = 10x(t)
  \]
  Determine the frequency response of the system, if possible.

  Solution: We first need to check for stability. The characteristic equation is $Q(D) = D^2 + 15D + 50$ which has two real roots $-10$ and $-5$. Since both are less than zero, the system is stable. Next we take the Fourier transform of both sides and apply the derivative property
  \[
  (j\omega)^2Y(j\omega) + 15(j\omega) Y(j\omega) + 50Y(j\omega) = 10X(j\omega)
  \]
  and rearrange to get the frequency response
  \[
  H(j\omega) = \frac{Y(j\omega)}{X(j\omega)} = \frac{10}{(j\omega)^2 + 15(j\omega) + 50} = \frac{10}{50-\omega^2 + j15\omega}
  \]\\
  $\blacksquare$
\end{example}

\section{Magnitude-phase representation of the CTFR}

Note that any complex valued function can be expressed in polar form using the magnitude and phase. Specifically the input and output can be put into this form
\[
X(j\omega) = |X(j\omega)|e^{\angle X(j\omega)}
\]
\[
Y(j\omega) = |Y(j\omega)|e^{\angle Y(j\omega)}
\]

By the convolution theorem then
  \[
  H(j\omega) = \frac{Y(j\omega)}{X(j\omega)} = \frac{|Y(j\omega)|e^{\angle Y(j\omega)}}{X(j\omega) = |X(j\omega)|e^{\angle X(j\omega)}} = \frac{|Y(j\omega)|}{|X(j\omega)|}e^{\angle Y(j\omega) - \angle X(j\omega)} = |H(j\omega)|e^{\angle H(j\omega)}
  \]
  Thus we see that
  \[
  |H(j\omega)| = \frac{|Y(j\omega)|}{|X(j\omega)|}
  \]
  and
  \[
  \angle H(j\omega) = \angle Y(j\omega) - \angle X(j\omega)
  \]
  This is the magnitude and phase representation of the frequency response.

\section{CTFR acting on sinusoids}

The advantage of the magnitude and phase representation of the frequency response, is the ease with which we can find the output due to a sinusoidal input. If we apply a sinusoidal input $x(t) = A e^{j\omega t}$, the output is a the same sinusoid scaled by the frequency response $y(t) = H(j\omega) A e^{j\omega t}$.

\begin{center}
  \includegraphics[scale=1]{graphics/18-ct-fr.pdf}
\end{center}

Now using the magnitude and phase representation
\[
y(t) = H(j\omega) A e^{j\omega t} = |H(j\omega)|e^{\angle H(j\omega)} A e^{j\omega t} = A |H(j\omega)| e^{j\omega t + \angle H(j\omega)}
\]
Thus we can interpret the frequency response as telling us how the input sinsusoids are scaled in magnitude and phase shifted as they pass through the system.

By the linearity property this extends to real sinusoidal inputs since
\begin{align*}
  x(t) &\longrightarrow y(t)\\
  \sin(\omega t) &\longrightarrow \frac{1}{2j}|H(j\omega)| e^{j\omega t + \angle H(j\omega)} - \frac{1}{2j}|H(j\omega)| e^{-j\omega t + \angle H(j\omega)}\\
  \sin(\omega t) &\longrightarrow |H(j\omega)|\sin(\omega t + \angle H(j\omega))
\end{align*}
and
\begin{align*}
  x(t) &\longrightarrow y(t)\\
  \cos(\omega t) &\longrightarrow \frac{1}{2}|H(j\omega)| e^{j\omega t + \angle H(j\omega)} + \frac{1}{2}|H(j\omega)| e^{-j\omega t + \angle H(j\omega)}\\
  \cos(\omega t) &\longrightarrow |H(j\omega)|\cos(\omega t + \angle H(j\omega))
\end{align*}

Also by the linearity property this analysis extends to the CT Fourier representation of a signal (an infinite sum of sinusoids):
\[
x(t) = \frac{1}{2\pi}\int\limits_{-\infty}^{\infty} X(j \omega) \, e^{j \omega t}\; d\omega \;\longrightarrow\; y(t) = \frac{1}{2\pi}\int\limits_{-\infty}^{\infty} H(j \omega) X(j \omega) \, e^{j \omega t}\; d\omega = \frac{1}{2\pi}\int\limits_{-\infty}^{\infty} \left| H(j \omega)\right| X(j \omega) \, e^{j \omega t + \angle H(j \omega)}\; d\omega
\]

Thus we arrive at the reason for the name \textit{Frequency Response} -- it specifies the the response of a stable system to any linear combination of sinusoidal inputs, i.e. any signal with a Fourier Transform.

\subsection{Bode plots}

We can visualize the frequency response as a plot of the real and imaginary part, or, of the magnitude and phase. Since the magnitude and phase allow us to directly see the system behavior at a given frequency, those plots are much more useful.

Rather than simply plot the magnitude and phase as a function of $\omega$, it is common to change the abscissa (horizontal / $\omega$-axis) to be on a logarithmic scale and so only plot the positive frequency portion of the spectrum (recall if the signal is real, the frequency response is even, so no information is lost). This is because the frequency response for physically realizable systems changes slowly as a function of frequency. Plotting on a log-scale compresses this information horizontally so that we can see how a wide range of frequency content is scaled. When plotting the magnitude spectrum it is also common to make the ordinate (vertical / gain axis) to be in decibels (dB). This is because of Weber's law, which states the humans perceive a doubling in strength of stimulus, when it is actually a ten-fold increase. Thus the magnitude of the frequency response in dB is $20 \log_{10} |H(j\omega)|$. When the frequency response is plotted this particular way we get what is called a \textit{Bode plot} (after the engineer Hendrik Wade Bode, an important figure in the development of control theory).

You will likely encounter Bode plots at several points in your career, so it is important to understand them well enough to create them on your own using software and read them. Also data-sheets and other documentation for CT devices generally use a Bode plot rather than giving an explicit mathematical model of the frequency response. It is also instructive to learn how to plot them manually (which was the traditional way to do it) since it gives you insight that can help with reverse engineering a model, however we do not cover this in detail in this course. Note that we will plot the spectrum as a function of frequency in units of rad/s, but it is also common to see it plotted in units of Hz. Take care to read the horizontal axis label as mixing up the two is a common source of error.

\begin{example} Consider a frequency response given by
  \[
  H(j\omega) = \frac{20000}{(j\omega)^2 + 300(j\omega) + 20000}
  \]
  The following Matlab code shows you how to plot the spectrum as a Bode plot (with some extra code to make it look nicer). You should read the documentation for the \texttt{bode} command in Matlab. It is also easy to just compute the magnitude and phase yourself.

\begin{verbatim}
H = tf([20000],[1,300,20000]);
[mag,ph,w] = bode(H);

% Create a nice bode plot
hFig = figure();
hold on;

subplot(2,1,1);
hm = semilogx(w,20*log10(squeeze(mag)));
grid on;
hTitle  = title ('Frequency Response');
hYLabel1 = ylabel('Magnitude (dB)');
set(gca, 'FontSize', 14, 'YTick', -60:10:20, ...
    'Box', 'off', 'LineWidth', 2);

subplot(2,1,2);
hp = semilogx(w,squeeze(ph*(pi/180)));
grid on;
hYLabel2 = ylabel('Phase (radians)');
hXLabel = xlabel('Frequency (rad/s)');
set(gca, 'FontSize', 14, 'Box', 'off', 'LineWidth', 2);

set(hm, 'linewidth', 2);
set(hp, 'linewidth', 2);
set([hXLabel, hYLabel1, hYLabel2]  , ...
     'FontSize'   , 14          );
set( hTitle                    , ...
     'FontSize'   , 14          , ...
     'FontWeight' , 'bold'      );
\end{verbatim}
This gives the following plot
\begin{center}
\includegraphics[scale=0.5]{graphics/lecture20_1.png}
\end{center}

$\blacksquare$
\end{example}

To read a Bode plot to see the behavior of the system at a given frequency, one need only read the values off the plot and convert from dB to a unit-less gain. A common mistake is to not realize the horizontal axis is logarithmic.

\begin{example}
  Suppose you are given the Bode plot (only) from the previous example and are asked what the output of the system is when the input is $x(t) = \cos(2\pi 32 t)$, i.e. a sinusoid at 32 Hz.\\
  \textbf{Solution:} First we determine the frequency in rad/s, $\omega = 2\pi 32 \approx 200$ rad/s. We go to that frequency on the Bode plot and read off a value of about $-10$ dB for the magnitude and about $-1.9$ rad for the phase. To convert back from dB
  \[
  |H(200)| = 10^{\frac{-10}{20}} \approx 0.3
  \]
  so the output would be
  \[
  y(t) \approx 0.3\cos(2\pi 32 t - 1.9)
  \]
  $\blacksquare$
\end{example}

\section{CTFR of first and second order systems}

TODO


\gotosection{2}{2}
\subsection{Solving equations with row reduction}

\begin{exercise}{3}
\begin{enumerate}
\item By row reduction, the soultion for this system is $x = w-2y$, $w = 3z$, $v = 0$, and $y, w$ are free variables. Now confirming the solution without using row reduction.

Label the four equations as $A, B, C, D$. Substituting $x+2y$ with $a$, $A+B$ gives $a = w$. Substituting this into $C$ and $D$ gives $3z + 2v = 9w$ and $z + v = 3w$, and in order to make these two compatible, $v = 0$, $z = 3w$. Since $a = x+2y = w$, $x=w-2y$.

\item Only $y, w$ are free variables, so the family of solutions depend on two variables.
\end{enumerate}
\end{exercise}

\begin{exercise}{5}
\begin{enumerate}
\item If $a=0$, the system has a set of solutions $y=2, z=3$, and $x$ is a free variable.

If $a \neq 0$, the matrix can be row reduced as:
$[A|\Vect{b}] = \xmat{a&1&0&2\\0&a&1&3} \rightarrow
  \rops{R_1: 1/aR_1}
  \xmat{1&1/a&0&2/a\\0&a&1&3} \rightarrow
  \rops{R_2: 1/aR_2}
  \xmat{1&1/a&0&2/a\\0&1&1/a&3/a} \\ \rightarrow
  \rops{R_1: R_1 - 1/aR_2}
  \xmat{1&0&-1/a^2&(2a-3)/a^2\\0&1&1/a&3/a}$

Then, as long as $a \neq 0$, the result matrix is in echelon form and is consistent, so it has solutions. Therefore, for any value of $a$, the system has solutions.

\item There are three unknowns but only two equations, it's impossible for this system to have a unique solution.
\end{enumerate}
\end{exercise}

\begin{exercise}{7}
The matrix can be row reduced as:
$[A|\Vect{b}] = \xmat{1&1&2&1 \\ 1&-1&a&b \\ 2&0&-b&0} \rightarrow
  \rops{R_2: -R_1 + R_2 \\ R_2: -2R_1 + R_3} \\
    \xmat{1&1&2&1 \\ 0&-2&a-2&b-1 \\ 0&-2&-b-4&-2} \rightarrow
  \rops{R_2: -1/2R_2}
    \xmat{1&1&2&1 \\ 0&1&(2-a)/2&(1-b)/2 \\ 0&-2&-b-4&-2} \\ \rightarrow
  \rops{R_1: R_1 - R_2 \\ R_3: 2R_2 + R_3}
    \xmat{1&0 & (2+a)/2 & (1+b)/2 \\
          0&1 & (2-a)/2 & (1-b)/2 \\
          0&0 & 2+a+b   & b+1}$

\begin{enumerate}
\item If $2+a+b = 0$, there exists two possibilities: first, $b+1 \neq 0$, the linear system is inconsistent and thus has no solutions; second, $b + 1 = 0$, then $a=b=-1$, substitute $a,b$:
$$\xmat{1&0 & 1/2 & 0  \\
        0&1 & 3/2 & -1 \\
        0&0 & 0   & 0}$$

The result matrix is in echelon form. The third column has no pivotal, so the system has infinitely many solutions with one free variable:
\begin{align*}
  x &= 0 - 1/2s  \\
  y &= -1 - 3/2s \\
  z &= s
\end{align*}
If $2+a+b \neq 0$, the matrix can be further reduced into the identity matrix, and therefore has a unique solution:
$$x = \frac{1}{2} \cdot \frac{b(b+1)}{2+a+b} \quad
  y = \frac{1}{2} \cdot \frac{2a-3b-b^2}{2+a+b} \quad
  z = \frac{b+1}{2+a+b}$$
\end{enumerate}
\end{exercise}


%
% Chapter 4.4
%

\section*{4.4 Indefinite Integrals and Net Change Theorem}

$$ \int f(x)dx \quad \text{ means } \quad F'(x)=f(x) $$
\\
Notice that a definite integral is a number, while an indefinite integral is a function (or family of functions).

\subsection*{Formulae for Indefinite Integrals}

$$ \int cf(x)dx = c \int f(x)dx $$
$$ \int kf(x)dx = kx + C $$
$$ \int [f(x) +  g(x)]dx = \int f(x)dx + \int g(x)dx $$
$$ \int x^n dx = \frac{x^{n+1}}{n+1} + C \quad \text{ when } n \neq -1 $$
$$ \int \sin(x)dx = -\cos(x) + C $$
$$ \int \cos(x)dx = \sin(x) + C $$
$$ \int \sec^2(x)dx = \tan(x) + C $$
$$ \int \csc^2(x)dx = -\cot(x) + C $$
$$ \int \sec(x)\tan(x)dx = \sec(x) + C $$
$$ \int \csc(x)\cot(x)dx = -\csc(x) + C $$

\subsection*{Net Change Theorem}

The integral of a rate of change is the net change
$$ \int_a^b F'(x)dx = F(b) - F(a) $$


\chapter{Golden Ratio}

In mathematics and the arts, two quantities are in the golden ratio if their ratio is the same as the ratio of their sum to the larger of the two quantities, i.e.~their maximum. The figure on the right illustrates the geometric relationship. Expressed algebraically, for quantities $a$ and $b$ with $a > b$,
\begin{equation}
 \frac{a+b}{a} = \frac{a}{b} \ \stackrel{\text{def}}{=}\ \varphi,
\end{equation}
where the Greek letter $\varphi$ represents the golden ratio. Its value is:
\begin{equation}
\varphi = \frac{1+\sqrt{5}}{2} = 1.61803\,39887\ldots.
\end{equation}

\section{History}

Ancient Greek mathematicians first studied what we now call the golden ratio because of its frequent appearance in geometry. The division of a line into ``extreme and mean ratio'' (the golden section) is important in the geometry of regular pentagrams and pentagons. Euclid's Elements  provides the first known written definition of what is now called the golden ratio: ``A straight line is said to have been cut in extreme and mean ratio when, as the whole line is to the greater segment, so is the greater to the less.'' Euclid explains a construction for cutting (sectioning) a line ``in extreme and mean ratio'', i.e., the golden ratio. (See Figure~\ref{fig:line:golden}.) Throughout the Elements, several propositions (theorems in modern terminology) and their proofs employ the golden ratio.

\begin{figure}[hbt!]\centering
\includegraphics[width=.3\textwidth]{220px-Golden-ratio-line}
\caption{Line segments in the golden ratio}
\label{fig:line:golden}
\end{figure}

\begin{figure}[hbt!]\centering
\includegraphics[width=.3\textwidth]{SimilarGoldenRectangles}
\caption{Golden rectangles}
\end{figure}

\section{Calculation}
Two quantities $a$ and $b$ are said to be in the golden ratio $\varphi$ if:
\begin{equation}
 \frac{a+b}{a} = \frac{a}{b} = \varphi.
\end{equation}

One method for finding the value of $\varphi$ is to start with the left fraction. Through simplifying the fraction and substituting in $\frac{b}{a} = \frac{1}{\varphi}$,
\begin{equation}
\frac{a+b}{a} = 1 + \frac{b}{a} = 1 + \frac{1}{\varphi},
\end{equation}

By definition, it is shown that
\begin{equation}
 1 + \frac{1}{\varphi} = \varphi.
\end{equation}
Multiplying by $\varphi$ gives
\begin{equation*}
\varphi + 1 = \varphi^2
\end{equation*}
which can be rearranged to
\begin{equation*}
{\varphi}^2 - \varphi - 1 = 0.
\end{equation*}
Using the quadratic formula, two solutions are obtained:
\begin{equation*}
\varphi = \frac{1 + \sqrt{5}}{2} = 1.61803\,39887\dots
\end{equation*}
and
\begin{equation*}
\varphi = \frac{1 - \sqrt{5}}{2} = -0.6180\,339887\dots
\end{equation*}
Because $\varphi$ is the ratio between positive quantities $\varphi$ is necessarily positive:
\begin{equation*}
\varphi = \frac{1 + \sqrt{5}}{2} = 1.61803\,39887\dots .
\end{equation*}

Different representations of the golden ratio are given in Table~\ref{tab:goldenratio}.

\begin{table}[hbt!]\centering
\caption{Number representations of the golden ratio}
\label{tab:goldenratio}

\begin{tabular}{|l|l|}
\hline
Form & Representation\\\hline
Binary & 1.1001111000110111011\ldots\\\hline
Decimal & 1.6180339887498948482\ldots\\\hline
Hexadecimal	& 1.9E3779B97F4A7C15F39\ldots\\\hline
Continued fraction & $1 + \cfrac{1}{1 + \cfrac{1}{1 + \cfrac{1}{1 + \cfrac{1}{1 + \ddots}}}}$\\[6ex]\hline
Algebraic form & $\displaystyle\frac{1 + \sqrt{5}}{2}$\\[2ex]\hline
Infinite series & $\displaystyle\frac{13}{8}+\sum_{n=0}^{\infty}\frac{(-1)^{(n+1)}(2n+1)!}{(n+2)!\,n!\,4^{(2n+3)}}$\\[2ex]\hline
\end{tabular}
\end{table}


\pagebreak
\newpage

\pagebreak
\newpage

\section{Classifier}

Classifiers were developed for three cases.

\begin{itemize}
 \item Classifier with $\Sigma = \sigma^2 \cdot \mathbf{I}$
 \item Classifier with $\Sigma = \Sigma_i$
 \item Classifier with arbitrary $\Sigma$
\end{itemize}

\subsection{Case A}
For the first case, $\sigma$ was assigned as $\sigma = 0.25$.
Given this restriction, the multivariate PDF was obtained, as shown in Figure \ref{fig: case a PDF}

\begin{figure}[htb!]
\centering
 \includegraphics[width = 0.8\textwidth]{pdfCaseA}
 \caption{PDF for Case A}
 \label{fig: case a PDF}
\end{figure}

\begin{align*}
 g_i(x) &= - \frac{\norm{x-\mu_i}^2}{2\sigma^2} + \ln P(\omega_i)\\
 \norm{x-\mu_i}^2 &= (x-\mu_i)^T (x-\mu_i)\\
 g_i(x) &= w^T_i x + \omega_{i0}\\
 w_i &= \frac{1}{\sigma^2} \mu\\
 w_{i0} &= \frac{-1}{2 \sigma^2} \mu^T_i \mu_i + ļn P(\omega_i)
\end{align*}

\pagebreak
\newpage

Afterwards, the \emph{a posteriori} probability is obtained and the boundaries of the classifier are obtained.
The results are shown in Figure \ref{fig: posteriori case A}.

\begin{figure}[htb!]
\centering
 \includegraphics[width= 0.8\textwidth]{classifierSetosaCaseA}
 \caption{Case A - A posteriori probability}
 \label{fig: posteriori case A}
\end{figure}

\pagebreak
\newpage

The classifier was tested against a sample of 75 entries from the dataset. Each entry was assigned a colour based on the assigned variety:
yellow for Setosa, black for Versicolor and magenta for Virginica.
This is presented in Figure \ref{fig: classifier case A}.

\begin{figure}[htb!]
\centering
 \includegraphics[width= \textwidth]{testClassifierCaseA}
 \caption{Case A - Boundaries}
 \label{fig: classifier case A}
\end{figure}

\pagebreak
\newpage

\pagebreak
\newpage

\subsection{Case B}
For the first case, $\Sigma$ was assigned as

\begin{equation*}
 \begin{pmatrix}
  0.26 & 0.04 &  0.02 & 0.01 \\
  0.04 &  0.22 & 0.03 & 0.02 \\
  0.02 & 0.03 & 0.15 & 0.15\\
  0.01 & 0.02 &  0.15 & 0.31
 \end{pmatrix}
\end{equation*}

This new covariance matrix influenced the multivariate gaussian distribution of the system,
as seen in Figure \ref{fig: case b PDF}.

\begin{figure}[htb!]
 \includegraphics[width = \textwidth]{pdfCaseB}
 \caption{PDF for Case B}
 \label{fig: case b PDF}
\end{figure}

\pagebreak
\newpage

Afterwards, the \emph{a posteriori} probability is obtained and the boundaries of the classifier are obtained.
The results are shown in Figure \ref{fig: posteriori case B}.

\begin{figure}[htb!]
\centering
 \includegraphics[width= 0.8\textwidth]{classifierSetosaCaseB}
 \caption{Case B - A posteriori probability}
 \label{fig: posteriori case B}
\end{figure}

\begin{align*}
 g_i(x) &= x^T W_i x + w^T_i x + \omega_{i0}\\
 W_i &= -\frac{1}{2}\Sigma^{-1}_i\\
 w_i &= \Sigma^{-1}_i \mu_i\\
 \omega_{i0}&= -\frac{1}{2} \mu^T_i \Sigma^{-1}_i \mu_i - \frac{1}{2}\ln \det(\Sigma_i) + \ln P(\omega_i)
\end{align*}

\pagebreak
\newpage

The classifier was tested against a sample of 75 entries from the dataset. Each entry was assigned a colour based on the assigned variety:
yellow for Setosa, black for Versicolor and magenta for Virginica.
This is presented in Figure \ref{fig: classifier case B}.

\begin{figure}[htb!]
\centering
 \includegraphics[width= \textwidth]{testClassifierCaseB}
 \caption{Case B - Boundaries}
 \label{fig: classifier case B}
\end{figure}

\pagebreak
\newpage

\pagebreak
\newpage

\subsection{Case C}
For the third case, $\Sigma$ was left untouched, allowing the covariance matrix to
reflect the nature of the data available.
This is observed in Figure \ref{fig: case c PDF}.

\begin{figure}[htb!]
 \includegraphics[width = \textwidth]{pdfCaseC}
 \caption{PDF for Case C}
 \label{fig: case c PDF}
\end{figure}

\pagebreak
\newpage

Afterwards, the \emph{a posteriori} probability is obtained and the boundaries of the classifier are obtained.
The results are shown in Figure \ref{fig: posteriori case C}.

\begin{figure}[htb!]
\centering
 \includegraphics[width= 0.8\textwidth]{classifierSetosaCaseC}
 \caption{Case C - A posteriori probability}
 \label{fig: posteriori case C}
\end{figure}

\pagebreak
\newpage

\begin{align*}
 g_i(x) &= -\frac{1}{2} (x-\mu_i)^T \Sigma^{-1} (x-\mu) + \ln P(\omega_i)\\
 g_i(x) &= w^T_i x + \omega_{i0}\\
 w_i &= \Sigma^{-1} \mu_i\\
 w_{i0} &= \frac{-1}{2}\mu^T_i \Sigma^{-1}  \mu_i + ļn P(\omega_i)
\end{align*}

The classifier was tested against a sample of 75 entries from the dataset. Each entry was assigned a colour based on the assigned variety:
yellow for Setosa, black for Versicolor and magenta for Virginica.
This is presented in Figure \ref{fig: classifier case C}.

\begin{figure}[htb!]
\centering
 \includegraphics[width= \textwidth]{testClassifierCaseC}
 \caption{Case C - Boundaries}
 \label{fig: classifier case C}
\end{figure}

\pagebreak
\newpage


\documentclass[12pt]{article}

\begin{document}

\def\pair{\stackrel{\mbox{\tiny FT}}{\Longleftrightarrow}}
\def\ft{{\cal F}}
\def\ift{{\cal F}^{\mbox{\,-}1}}
\parindent 0pt

\section{Smoothing spectra via Fourier transforms}

Smoothing data with an even smoothing function is equivalent to
convolving the data with that function. This can be achieved via
Fourier transforms.

\subsection{Definition of terms}

\begin{itemize}
\item Let $\ft(g)$ denote the Fourier transform of a function $g$.
\item Let $\ift(G)$ denote the inverse Fourier transform of a function $G$.
\item Let $.$ denote multiplication, and $\otimes$ denote convolution.
\item Let the smoothing function be denoted by $s$.
\item Let the original arrays of data values and weights be denoted by $f$ and $w$, respectively.
\item Let the smoothed arrays of data values and weights be denoted by $\overline{f}$ and $\overline{w}$, respectively.
\end{itemize}

\subsection{Smoothing a discrete array using an unweighted convolution}

The Fourier convolution theorem says that multiplying the Fourier
transforms of two continuous functions by each other and then taking
the inverse transform of the result yields the convolution of the two
functions.

\begin{eqnarray}
     \overline{f} &=& f \otimes s \\
%
     \label{convolve}
     \overline{f} &=& \ift\left[\ft(f).\ft(s)\right]
\end{eqnarray}

For discretely sampled data one can either convolve the two functions
explicitly via

\begin{equation}
 \overline{f_{i}} = \frac{\sum_{u} f_{u}.s_{i-u}}
	                 {\sum_{u} s_{i-u}}
\end{equation}

or substitute a discrete Fourier transform for each of the Fourier
transforms shown in equation~\ref{convolve}. In practice, if the
width of the smoothing function encompasses a significant number of
elements of the function being smoothed then it will be faster to use the
latter approach, particularly if a Fast Fourier Transform can be used.

\subsection{Smoothing a discrete array using a weighted convolution}

If the data are weighted then one must take account of the weights
during convolution, and the convolution must be modified to

\begin{eqnarray}
 \overline{f_{i}} &=& \frac{\sum_{u} f_{u}.w_{u}.s_{i-u}} {\sum_{u} w_{u}.s_{i-u}}\\
     \overline{f} &=& \frac {f.w \otimes s} {w \otimes s}\\
     \overline{f} &=& \frac {\ift[\ft(f.w).\ft(s)]} {\ift[\ft(w).\ft(s)]}
\end{eqnarray}

The original weights of the unsmoothed data are also modified by smoothing.
If the weight of each $f_i$ is defined in terms of its variance
$\sigma_i^2$, as

\begin{equation}
  w_{i}  = \frac{1}{\sigma_{i}^2},
\end{equation}

Then the weight $\overline{w_{i}}$ that should be assigned to
$\overline{f_{i}}$, is given by:

\begin{eqnarray}
 \frac{1}{\overline{w_{i}}} &=& \sum_{j}\left(\frac{\partial\overline{f_{i}}}{\partial f_{j}}\right)^{2} . \frac{1}{w_{j}} \\
%
 \frac{\partial \overline{f_{i}}}{\partial f_{j}} &=& \frac {w_{j}.s_{i-j}}{\sum_{u} w_{u} . s_{i-u}} \\
%
 \frac{1}{\overline{w_{i}}} &=& \sum_{j}\left(\frac {w_{j}.s_{i-j}}{\sum_{u} w_{u} . s_{i-u}}\right)^{2} . \frac{1}{w_{j}} \\
%
 \label{variance}
 \frac{1}{\overline{w_{i}}} &=& \frac {\sum_{j} w_{j}.s_{i-j}^2}{(\sum_{u} w_{u} . s_{i-u})^{2}}
\end{eqnarray}

In terms of convolutions this can be re-expressed as

\begin{equation}
 \frac{1}{\overline{w_{i}}} = \frac {w \otimes s^{2} } {(w \otimes s)^2}.
\end{equation}

The smoothed data array and its weights can then be obtained via
Fourier transforms as:

\begin{eqnarray}
 \overline{f} =& \frac {f.w \otimes s} {w \otimes s} &= \frac{\ift\left[\ft(f.w).\ft(s)\right]} {\ift\left[\ft(w).\ft(s)\right]}\\
 \overline{w} =& \frac {(w \otimes s)^2}{w \otimes s^{2} } &= \frac{\left(\ift\left[\ft(w).\ft(s)\right]\right)^2}{\ift\left[\ft(w).\ft(s^{2})\right]}
\end{eqnarray}

Thus, to account for data weights when smoothing an array, one must
first determine $\ft(s)$ and $\ft(s^2)$ for the chosen smoothing
function.

\section{Smoothing functions}

\def\fwhm{\mbox{\tiny FWHM}}

Each smoothing function will be described in terms of its full-width
at half maximum value, denoted as $\fwhm$. For convenience the
dependent coordinate of each function will be defined such that it is
unity where the function falls to half its maximum value. In terms of
the spectrum channel--based coordinates, $i$ this requires the
following change of coordinates.

\begin{equation}
  \delta x = \frac{2 \delta i}{\fwhm}.
\end{equation}

In the Fourier plane this has the effect of modifying the delay
coordinate increment per channel $j$ from $1/n$ to $2.\fwhm/n$,
yielding

\begin{equation}
  \delta t = 2.\delta j.\fwhm / n.
\end{equation}

Note that in order that the total power in the smoothed data be
unchanged by smoothing. The area under the smoothing function and thus
the value at the origin of its Fourier transform, must be unity.

\subsection{Hanning}

\begin{eqnarray}
s_{x} &=& \frac{\sin(\pi x)} {2\pi x(1-x^2)}\\
\ft(s_{x}) &=& \left\{
              \begin{array}{rl} \frac{1}{2}[1 + \cos(2\pi t)] & |t| \le \frac{1}{2}\\
                                                            0 & |t|   > \frac{1}{2}\\
              \end{array}
        \right.\\
\ft(s_{x}^2) &=&  \left\{
              \begin{array}{rl} \frac{1}{4}[(1 - |t|)(1+\frac{1}{2}\cos(2\pi t)) + \frac{3}{4\pi}\sin(2\pi|t|)] & |t| \le 1\\
                                                            0 & |t|   > 1\\
              \end{array}
        \right.
\end{eqnarray}

\subsection{Gaussian}

\begin{eqnarray}
s_{x} &=& \sqrt{\ln(2)/\pi} \; e^{-\ln(2) x^{2}}\\
\ft(s_{x}) &=& e^{-\frac{\pi^2 t^2}{\ln(2)}}\\
\ft(s_{x}^2) &=& \sqrt{\ln(2)/2\pi} \; e^{-\frac{\pi^2 t^2}{2 \ln(2)}}
\end{eqnarray}

\subsection{Boxcar}

\begin{eqnarray}
s_{x} &=& \left\{
              \begin{array}{ll} \frac{1}{2} & |x| \le 1\\
                                          0 & |x|   > 1\\
              \end{array}
        \right.\\
\ft(s_{x}) &=& \frac{\sin(2\pi t)}{2\pi t}\\
\ft(s_{x}^2) &=& \frac{1}{2} \frac{\sin(2\pi t)}{2\pi t}
\end{eqnarray}

\subsection{Sinc}

\begin{eqnarray}
s_{x} &=& \frac{\beta}{\pi} \frac{\sin(\beta x)}{\beta x}\\
\beta &=& 1.8954942670340\\
\ft(s_{x}) &=& \left\{
              \begin{array}{ll} 1 & |t| \le \frac{\beta}{2\pi}\\
                                0 & |t|   > \frac{\beta}{2\pi}\\
              \end{array}
        \right.\\
\ft(s_{x}^2) &=& \left\{
                     \begin{array}{ll} \frac{\beta}{\pi} - t & |t| \le \frac{\beta}{\pi}\\
                                                           0 & |t|   > \frac{\beta}{\pi}\\
                     \end{array}
                  \right.
\end{eqnarray}

\end{document}


\subsection{Raising and lowering indices}

We showed that the inner product between two vectors with the same basis can be written as:

\(\langle v, w\rangle =\langle \sum_i e_iv^i, \sum_j f_jw^j\rangle \)

\(\langle v, w\rangle =v^i\overline {w^j}\langle e_i, e_j\rangle\)

Defining the metric as:

\(g_{ij}:=\langle e_i,e_j\rangle \)

\(\langle v, w\rangle =v^i\overline {w^j}g_{ij}\)

\subsubsection{Metric inverse}

We can use this to define the inverse of the metric.

\(g^{ij}:=(g_{ij})^{-1}\)

We can use this to raise and lower vectors.

\(v_i:=v^jg_{ij}\)

\subsubsection{Raising and lowering indices of tensors}

If we have tensor:

\(T_{ij}\)

We can define:

\(T_i^k=T_{ij}g^{jk}\)

\(T^{il}=T_{ij}g^{jk}g^{kl}\)

\subsubsection{Tensor contraction}

If we have:

\(T_{ij}x^j\)

We can contract it to:

\(T_{ij}x^j=v_i\)

Similarly we can have:

\(T^{ij}x_j=v^i\)


\section{K-Nearest Neighbors}
The $k$-nearest neighbors algorithm, commonly known as
$k$-NN, is a non-parametric approach where the response of a data point
is determined by the nature of its $k$ neighbors from the training set.
It can be used in both classification and regression settings.
The higher the parameter $k$, the higher the bias, and the lower the parameter
$k$, the higher the variance.
\subsection{Classification}
In $k$-NN classification, the output is a class membership. An object is
classified by a majority vote of its neighbors, with the object being assigned
to the class most common among its $k$ nearest neighbors ($k$ is a positive integer,
typically small). If $k = 1$, then the object is simply assigned to the class
of that single nearest neighbor.
\subsection{Regression}
In $k$-NN regression, the output is the property value for the object.
This value is the average of the values of its k nearest neighbors.
\begin{equation} y = \frac{1}{k} \sum_{x_i \in \mathcal{N}_k(x)} y_i \end{equation}
where $\mathcal{N}_k(x)$ is the $k$ nearest points around $x$.


%---------------------------Maximum Aspect Frobenius---------------------------
\section{Maximum Aspect Frobenius}

For hexahedra, there is not a unique definition of the aspect Frobenius.
Instead, we use the aspect Frobenius
defined for tetrahedra (see section~\S\ref{s:tet-aspect-Frobenius}),
but choose the reference $W$ element to be right isosceles at
the hexahedral corner. Consider the eight tetrahedra formed by edges
incident to the corner of a hexahedron.
Given a corner vertex $i$ and its three adjacent vertices $j$, $k$, and $\ell$ ordered
in a clockwise manner (so that $ijk\ell$ is a positively oriented tetrahedron),
denote the tetrahedral aspect frobenius of that corner as $F_{ijk\ell}$.
To obtain a single value for the metric, we take the maximum value of the eight unique tetrahedral aspects
\[
  q = \max\left(F_{0134}, F_{1205}, F_{2316}, F_{3027}, F_{4750}, F_{5461}, F_{6572}, F_{7643} \right).
\]

In the past, this metric was called the condition number and computed
in terms of the Jacobian matrices $A_i$ and
their determinants $\alpha_i$ as in \S\ref{s:hex}.
We provide that method of computation below for reference purposes.
First, define
\[
\kappa(A_i)
  = \left|A_i\right| \left|A_i^{-1}\right|
  = \frac {\left|A_i\right| \left|\mathrm{adj}(A_i)\right|}{\alpha_i}.
\]
There are 9 of these matrices and we evaluate the condition number at each and take a third of the maximum:
\[
q = \frac {1}{3} \max\left\{ \kappa(A_0), \kappa(A_1), \ldots, \kappa(A_8) \right\}
\]
The first 8 matrices represent the condition at the corners and the last represents the condition number
at the element's center.
Note that if $\alpha_i \leq DBL\_MIN$, for any $i$, then $q = DBL\_MAX$.

\hexmetrictable{maximum aspect frobenius}%
{$1$}%                                        Dimension
{$[1,3]$}%                                    Acceptable range
{$[1,DBL\_MAX]$}%                             Normal range
{$[1,DBL\_MAX]$}%                             Full range
{$1$}%                                        Cube
{\cite{knu:00}}%                              Citation
{v\_hex\_max\_aspect\_frobenius \textnormal{or} %
 v\_hex\_condition$^*$}%                      Verdict function name

\noindent\,$^*$ indicates a function that is deprecated and may be removed in future versions of \verd.


\section{Transformations}
In this section we will show how to implement three transformations of the plane, in increasing difficulty. We will see that they all correspond to linear transformations on complex numbers, that is, functions of the form $f(p) = a \ast p + b$ for $a,b,p \in \complex$, and deduce a way to compute a general transformation that combines all three.

\subsection{Translation}
To translate an object by a vector $\vv{v}$, we simply need to add $\vv{v}$ to every point in the object. The corresponding function is $f(p) = p + \vv{v}$ with $\vv{v} \in \complex$.

\centerFig{transfo0}

The implementation is self-explanatory:
\begin{lstlisting}
pt translate(pt v, pt p) {return p+v;}
\end{lstlisting}

\subsection{Scaling}
To scale an object by a certain ratio $\alpha$ around a center $c$, we need to shorten or lengthen the vector from $c$ to every point by a factor $\alpha$, while conserving the direction.
The corresponding function is $f(p) = c + \alpha(p-c)$ ($\alpha$ is a real here, so this is a scalar multiplication).
%This is also a linear transformation on complex numbers: we can write $f(x) = \alpha \ast x + (1-\alpha)p$.

\centerFig{transfo1}

Again, the implementation is just a translation of the expression into code:
\begin{lstlisting}
pt scale(pt c, double factor, pt p) {
    return c + (p-c)*factor;
}
\end{lstlisting}

\subsection{Rotation}\label{ss:rotation}
To rotate an object by a certain angle $\phi$ around center $c$, we need to rotate the vector from $c$ to every point by $\phi$. From our study of polar coordinates in \ref{polar-form} we know this is equivalent to multiplying by $\cis\phi$, so the corresponding function is $f(p) = c + \cis\phi\ast(p-c)$.

\centerFig{transfo2}

In particular, we will often use the (counter-clockwise) rotation centered on the origin. We use complex multiplication to figure out the formula:
\begin{align*}
(x+yi)\ast\cis\phi &= (x+yi)\ast(\cos\phi + i\sin\phi) \\
&= (x\cos\phi - y\sin\phi) + (x\sin\phi + y\cos\phi)i
\end{align*}
which gives the following implementation:
\begin{lstlisting}
pt rot(pt p, double a) {
    return {p.x*cos(a) - p.y*sin(a), p.x*sin(a) + p.y*cos(a)};
}
\end{lstlisting}
which if using \lstinline|complex| can be simplified to just
\begin{lstlisting}
pt rot(pt p, double a) {return p * polar(1.0, a);}
\end{lstlisting}

And among those, we will use the rotation by $90\degree$ quite often:
\begin{align*}
(x+yi)\ast\cis(90\degree) &= (x+yi)\ast(\cos(90\degree) + i\sin(90\degree)) \\
&= (x+yi)\ast i = -y+xi
\end{align*}
It works fine with integer coordinates, which is very useful:
\begin{lstlisting}
pt perp(pt p) {return {-p.y, p.x};}
\end{lstlisting}

\subsection{General linear transformation}
It is easy to check that all those transformations are of the form $f(p) = a \ast p + b$ as claimed in the beginning of this section. In fact, all transformations of this type can be obtained as combinations of translations, scalings and rotations.\footnote{Actually, if $a = 1$ it is just a translation, and if $a \neq 1$ it is the combination of a scaling and a rotation combination from a well-chosen center.}

Just like for real numbers, to determine a linear transformation such as this one, we only need to know the image of two points to know the complete function. Indeed, if we know $f(p) = a \ast p + b$ and $f(q) = a \ast q + b$, then we can find $a$ as $\frac{f(q)-f(p)}{q-p}$, and then $b$ as $f(p) - a \ast p$.

And thus if we want to know a new point $f(r)$ of that transformation, we can then compute it as:
\[f(r) = f(p) + (r-p) \ast \frac{f(q)-f(p)}{q-p}\]

%Note that the triangles $pqr$ and $f(p)f(q)f(r)$ formed this way are similar:
\centerFig{transfo3}

This is easy to implement using \lstinline|complex|:
\begin{lstlisting}
pt linearTransfo(pt p, pt q, pt r, pt fp, pt fq) {
    return fp + (r-p) * (fq-fp) / (q-p);
}
\end{lstlisting}

Otherwise, you can use the cryptic but surprisingly short solution from \cite{kactl} (see the next sections for \lstinline|dot()| and \lstinline|cross()|):
\begin{lstlisting}
pt linearTransfo(pt p, pt q, pt r, pt fp, pt fq) {
    pt pq = q-p, num{cross(pq, fq-fp), dot(pq, fq-fp)};
    return fp + pt{cross(r-p, num), dot(r-p, num)} / sq(pq);
}
\end{lstlisting}


%================================
\section{Topological Spaces}
%================================

%--------------------------------
\begin{definition}
	[topology]
	\label{def: topology}
	Let $X$ be a set, and let a family $\mathcal T \subseteq \mathcal P(X)$. $\mathcal T$ is called a \textit{topology} on $X$ iff it satisfies the \textit{open set axioms}:
	\begin{enumerate}[\bf O1.]
		\item $X \in \mathcal T$;
		\item $\mathcal T$ is closed under arbitrary union;
		\item $\mathcal T$ is closed under finite intersection.
	\end{enumerate}
\end{definition}
%--------------------------------

%--------------------------------
\begin{theorem}
	$\emptyset \in \mathcal T$.

	\begin{proof}
		By O2 in Definition \ref{def: topology}, for all $\mathcal A \subseteq \mathcal T$, $\bigcup \mathcal A \in \mathcal T$. Clearly, $\emptyset \in \mathcal T$, then we have
		$$
		\emptyset = \bigcup \emptyset \in \mathcal T.
		$$
	\end{proof}
\end{theorem}
%--------------------------------

%--------------------------------
\begin{definition}
	[topological spaces]
	\label{def: topological spaces}
	With the condition in Definition \ref{def: topology}, the pair $(X, \mathcal T)$ is called a \textit{topological space}. All subsets of $X$ in $\mathcal T$ are said to be \textit{open} in $(X, \mathcal T)$.
\end{definition}
%--------------------------------

%--------------------------------
\begin{definition}
	[closed sets]
	\label{def: closed sets}
	Let $(X, \mathcal T)$ be a topological space. A subset $V$ of $X$ is said to be \textit{closed} iff there is an open set $U$ in $X$ such that
	$$
	V = X \setminus V.
	$$
\end{definition}
%--------------------------------

%--------------------------------
\begin{corollary}
	\label{prop: dark side of topology}
	Let $(X, \mathcal T)$ be a topological space, and let $\mathcal C$ be the family of all closed sets in $X$. Then
	\begin{enumerate}[(i)]
		\item
		$\emptyset, X \in \mathcal C$;

		\item
		$\mathcal C$ is closed under arbitrary intersection;

		\item
		$\mathcal C$ is closed under finite union.
	\end{enumerate}

	\begin{proof}
		\
		\begin{enumerate}[(i)]
			\item
			$X \in \mathcal T$ implies $X \setminus X = \emptyset \in \mathcal C$; and $\emptyset \in \mathcal T$ implies $X \setminus \emptyset = X \in \mathcal C$;

			\item
			As $\mathcal T$ is closed under arbitrary union, then by Definition \ref{def: closed sets} and De Morgan's Law, $\mathcal C$ is closed under arbitrary intersection.

			\item
			As $\mathcal T$ is closed under finite intersection, then by Definition \ref{def: closed sets} and De Morgan's Law, $\mathcal C$ is closed under finite union.
		\end{enumerate}
	\end{proof}
\end{corollary}
%--------------------------------

%--------------------------------
\begin{definition}
	[finer and coarser topology]
	\label{def: finer and coarser topology} Let $X$ be any set, and let $\mathcal T, \mathcal T'$ be topologies on $X$. $\mathcal T$ is said to be \textit{finer} than $\mathcal T'$ iff $\mathcal T \supseteq \mathcal T'$; respectively, $\mathcal T$ is said to be \textit{coarser} than $\mathcal T'$ iff $\mathcal T \subseteq \mathcal T'$.
\end{definition}
%--------------------------------

%--------------------------------
\begin{definition}
	[neighbourhood]
	\label{def: neighbourhood}
	Given $(X, \mathcal T)$ as a topological space and a point $x \in X$, a subset $N \subseteq X$ is called a \textit{neighbourhood} iff it contains an open set $U$ containing $x$.
	% todo: find a place for neighbourhood.
\end{definition}
%--------------------------------

%--------------------------------
\begin{theorem}
	\label{prop: alt-def of open sets by neighbourhoods}
	Given $(X, \mathcal T)$ as a topological space and $U \subseteq X$, $U$ is open iff for all $x \in U$, there is a neighbourhood $N$ of $x$ such that $N \subseteq U$.

	\begin{proof}
		If $U$ is open, then $U$ itself is a neighbourhood of $x$ contained in $U$.

		Conversely, if for all $x \in U$, there is a neighbourhood $N_x$ of $x$ contained in $U$, then there is a open neighbourhood $U_x \ni x$ contained in $N_x$. Then we have
		$$
		U \supseteq \bigcup_{x \in U} U_x.
		$$
		Suppose $U$ is not open, then $U$ is a proper superset in the relation above. Then there exists $y \in U$ which is not in any $U_x$. This implies that such a $y$ does not have any neighbourhood $N_y$ in $U$, for such an $N_y$ must contains an open $U_y \ni y$. For if it does, then there must be a $U_x$ contains $y$. This is a contradiction. Thus,
		$$
		U = \bigcup_{x \in U} U_x
		$$
		is open.
	\end{proof}
\end{theorem}
%--------------------------------


\section{Formulary of Probability}
\label{sec:Formulary-Probability}

\subsection{Foundations}

\begin{description}

	\item [Failure Rate]
		\begin{equation}
		r(x) = \frac{f(x)}{\overline{F}(x)}
		\end{equation}

	\item [Coefficient of Variation]
		\begin{equation}
		C_{X}^{2} = \frac{\variance{X}}{\expected{X^{2}}}
		\end{equation}

\end{description}

\subsection{Exponential Distribution}

\begin{description}

	\item [Exponential p.d.f]
		\begin{equation}
		f(x) = \left\{\begin{matrix}
		\lambda e^{-\lambda x} & x \geq 0\\
		0 & x \leq 0
		\end{matrix}\right.
		\end{equation}

	\item [Exponential c.d.f]
		\begin{equation}
		\begin{aligned}
		F(x) =
		\left\{\begin{matrix}
		1 - e^{-\lambda x} & x \geq 0\\
		0 & x \leq 0
		\end{matrix}\right.
		\end{aligned}
		\end{equation}

	\item [Exponential Mean]
		\begin{equation}
		\expected{X} = \frac{1}{\lambda}
		\end{equation}

	\item [Exponential Variance]
		\begin{equation}
		\variance{X} = \frac{1}{\lambda^{2}}
		\end{equation}

	\item [Exponential Precedency]
		\begin{equation}
		\probability{X_{1} < X_{2}} = \frac{\lambda_{1}}{\lambda_{1} + \lambda_{2}}
		\end{equation}

	\item [Exponential Minimum]
		\begin{equation}
		\min(X_{1},X_{2}) \sim Exp(\lambda_{1} + \lambda_{2})
		\end{equation}

\end{description}

\subsection{Poisson Distribution}

\begin{description}

	\item [Poisson p.m.f]
		\begin{equation}
		f(k,t) = \frac{(\lambda t)^{k} e^{-\lambda t}}{k!}
		\end{equation}

	\item [Poisson c.d.f]
		\begin{equation}
		F(k,t) = \sum_{x=0}^{t} f(k,x)
		\end{equation}

	\item [Poisson Mean]
		\begin{equation}
		\expected{N(t)} = \lambda t
		\end{equation}

	\item [Poisson Variance]
		\begin{equation}
		\variance{N(t)} = \lambda t
		\end{equation}

	\item [Poisson Inter-arrivals]
		\begin{equation}
		\probability{N(s+t)-N(s) = n} \sim Poisson(\lambda t)
		\end{equation}

	\item [Poisson Merging]
		Given two independent Poisson process with rate $\lambda_{1}$ and $\lambda_{2}$, the merged process is a Poisson process with rate $(\lambda_{1} + \lambda_{2})$.

	\item [Poisson Splitting]
		Given a Poisson process with rate $\lambda$, whose events are partitioned in class-A with probability $p$ and class-B with probability $(1-p)$, the class-A process is a Poisson process with rate $p \lambda$ and the class-B process is a Poisson process with rate $(1-p) \lambda$, and these processes are independent.

	\item [Poisson Uniformity]
		If $k$ events of a Poisson process occur by time $t$, then the $k$ events are distributed independently and uniformly in $[0,t]$.

\end{description}

\subsection{Pareto Distribution}

\begin{description}

	\item [Pareto distribution]
		$X$ is Pareto distributed with order $\alpha$ (that is, $X \sim Pareto(\alpha)$) if
		\begin{equation}
		\overline{F}(x) = \probability{X > x} = x^{-\alpha} \quad \forall x \geq 1, 0 < \alpha < 2
		\end{equation}

	\item [Bounded Pareto distribution]
		$X$ is Bounded Pareto distributed with order $\alpha$ and bounds $k,p$ (that is, $X \sim BP(k,p,\alpha)$) if
		\begin{equation}
		f(x) = \probability{X \leq x} = \alpha x^{-\alpha -1} \cdot \frac{k^{\alpha}}{1-\Big(\frac{k}{p})^{\alpha}} \quad \forall k \leq x \geq p, 0 < \alpha < 2
		\end{equation}

\end{description}


\lab{Introduction to SymPy}{Introduction to SymPy}

\objective{Most implementations of numerical algorithms focus on crunching, relating, or visualizing numerical data.
However, it is sometimes convenient or necessary to represent parts of an algorithm symbolically.
The SymPy module provides a way to do symbolic mathematics in Python, including algebra, differentiation, integration, and more.
In this lab we introduce SymPy syntax and emphasize how to use symbolic algebra for numerical computing.}

\section*{Symbolic Variables and Expressions} % ===============================

Most variables in Python refer to a number, string, or data structure.
Doing computations on such variables results in more numbers, strings, or data structures.
A \emph{symbolic variable} is a variable that represents a mathematical symbol, such as $x$ or $\theta$, not a number or another kind of data.
Operating on symbolic variables results in an \emph{expression}, representative of an actual mathematical expression.
For example, if a symbolic variable \li{Y} refers to a mathematical variable $y$, the multiplication \li{3*Y} refers to the expression $3y$.
This is all done without assigning an actual numerical value to \li{Y}.

SymPy \cite{meurer2017sympy} is Python's library for doing symbolic algebra and calculus.
It is typically imported with \li{import sympy as sy}, and symbolic variables are usually defined using \li{sy.symbols()}.
% Expressions can also be derived from a string using \li{sy.sympify()} or the shortcut \li{sy.S()}.
% \li{sy.Symbol()} can also be used to define a single variable, but will not be used in this lab for consistency.

\begin{lstlisting}
>>> import sympy as sy
>>> x0 = sy.symbols('x0')                   # Define a single variable.

# Define multiple symbolic variables simultaneously.
>>> x2, x3 = sy.symbols('x2, x3')           # Separate symbols by commas,
>>> m, a = sy.symbols('mass acceleration')  # by spaces,
>>> x, y, z = sy.symbols('x:z')             # or by colons.
>>> x4, x5, x6 = sy.symbols('x4:7')

# Combine symbolic variables to form expressions.
>>> expr = x**2 + x*y + 3*x*y + 4*y**3
>>> force = m * a
>>> print(expr, force, sep='\n')
x**2 + 4*x*y + 4*y**3
acceleration*mass
\end{lstlisting}
%
% # Create an expression from a string.
% >>> sy.S("x0**2+x1**3+x2")
% x0**2 + x1**3 + x2

% Pre-defined variables, including greek letters, can be imported from \li{sympy.abc}.
% To see a list of these variables, run \li{dir(sy.abc)} after \li{sy.abc} has been imported.

SymPy has its own version for each of the standard mathematical functions like $\sin(x)$, $\log(x)$, and $\sqrt{x}$, and includes predefined variables for special numbers such as $\pi$.
The naming conventions for most functions match NumPy, but some of the built-in constants are named slightly differently.
\[
\begin{array}{cl}
\text{Functions}
&
\begin{tabular}{c|c|c|c|c|c}
    $\sin(x)$ & $\arcsin(x)$ & $\sinh(x)$ & $e^x$ & $\log(x)$ & $\sqrt{x}$
    \\ \hline
    \li{sy.sin()} & \li{sy.asin()} & \li{sy.sinh()} & \li{sy.exp()} & \li{sy.log()} & \li{sy.sqrt()}
\end{tabular}
\\ \\
\text{Constants}
&
\begin{tabular}{c|c|c|c}
    $\pi$ & $e$ & $i = \sqrt{-1}$ & $\infty$
    \\ \hline
    \li{sy.pi} & \li{sy.E} & \li{sy.I} & \li{sy.oo}
\end{tabular}
\end{array}
\]
Other trigonometric functions like $\cos(x)$ follow the same naming conventions.
For a complete list of SymPy functions, see \url{http://docs.sympy.org/latest/modules/functions/index.html}.

\begin{warn}
Always use SymPy functions and constants when creating expressions instead of using NumPy's functions and constants.
Later we will show how to make NumPy and SymPy cooperate.
\begin{lstlisting}
>>> import numpy as np

>>> x = sy.symbols('x')
>>> np.exp(x)                       # Try to use NumPy to represent e**x.
<<Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'Symbol' object has no attribute 'exp'>>

>>> sy.exp(x)                       # Use SymPy's version instead.
<<exp(x)>>
\end{lstlisting}
\end{warn}

\begin{info}
SymPy defines its own numeric types for integers, floats, and rational numbers.
For example, the \li{sy.Rational} class is similar to the standard library's \li{fractions.Fraction} class, and should be used to represent fractions in SymPy expressions.

\begin{lstlisting}
>>> x = sy.symbols('x')
>>> (2/3) * sy.sin(x)               # 2/3 returns a float, not a rational.
0.666666666666667*sin(x)

>>> sy.Rational(2, 3) * sy.sin(x)   # Keep 2/3 symbolic.
2*sin(x)/3
\end{lstlisting}
Always be aware of which numeric types are being used in an expression.
Using rationals and integers where possible is important for simplifying expressions.
% To keep $\frac{2}{3}$ symbolic, use \li{sy.Rational(2,3)*sy.sin(x)} or \li{sy.Integer(2)/3*sy.sin(x)}.
\end{info}

\begin{problem} % Write a simple expression.
Write a function that returns the expression $\frac{2}{5} e^{x^2-y}\cosh(x+y) + \frac{3}{7}\log(xy+1)$ symbolically.
Make sure that the fractions remain symbolic.
\end{problem}

\subsection*{Sums and Products} % ---------------------------------------------

Expressions that can be written as a sum or a product can be constructed with \li{sy.summation()} or \li{sy.product()}, respectively.
Each of these functions accepts an expression that represents one term of the sum or product, then a tuple indicating the indexing variable and which values it should take on.
For example, the following code constructs the sum and product given below.
\[
\sum_{i=1}^{4} x + iy
\qquad\qquad
\prod_{i=0}^{5} x + iy
\]

\begin{lstlisting}
>>> x, y, i = sy.symbols('x y i')

>>> sy.summation(x + i*y, (i, 1, 4))    # Sum over i=1,2,3,4.
<<4*x + 10*y>>

>>> sy.product(x + i*y, (i, 0, 5))      # Multiply over i=0,1,2,3,4,5.
<<x*(x + y)*(x + 2*y)*(x + 3*y)*(x + 4*y)*(x + 5*y)>>
\end{lstlisting}

\subsection*{Simplifying Expressions} % ---------------------------------------

The expressions for the summation and product in the previous example are automatically simplified.
More complicated expressions can be simplified with one or more of the following functions.

\begin{table}[H]
\centering
\begin{tabular}{r|l}
Function & Description \\ \hline
% \li{sy.apart()} & Partial fraction decomposition. \\
\li{sy.cancel()} & Cancel common factors in the numerator and denominator. \\
% \li{sy.collect()} & Factor out all rational powers of a specified variable. \\
\li{sy.expand()} & Expand a factored expression. \\
\li{sy.factor()} & Factor an expanded expression. \\
% \li{powsimp()} & Combine powers with similar bases and exponents. \\
\li{sy.radsimp()} & Rationalize the denominator of an expression. \\
% \li{sy.ratsimp()} & Return the simplest form of the addition of two rational expressions. \\
% \li{seperate()} & Rewrite a power of products to a product of powers. \\
\li{sy.simplify()} & Simplify an expression.\\
% \li{together()} & Combine rational functions.\\
\li{sy.trigsimp()} & Simplify only the trigonometric parts of the expression. \\
\end{tabular}
\end{table}

\begin{lstlisting}
>>> x = sy.symbols('x')
>>> expr = (x**2 + 2*x + 1) / ((x+1)*((sy.sin(x)/sy.cos(x))**2 + 1))
>>> print(expr)
(x**2 + 2*x + 1)/((x + 1)*(sin(x)**2/cos(x)**2 + 1))

>>> sy.simplify(expr)
(x + 1)*cos(x)**2
\end{lstlisting}

The generic \li{sy.simplify()} tries to simplify an expression in any possible way.
This is often computationally expensive; using more specific simplifiers when possible reduces the cost.

\begin{lstlisting}
>>> expr = sy.product(x + i*y, (i, 0, 3))
>>> print(expr)
x*(x + y)*(x + 2*y)*(x + 3*y)

>>> expr_long = sy.expand(expr)         # Expand the product terms.
>>> print(expr_long)
x**4 + 6*x**3*y + 11*x**2*y**2 + 6*x*y**3

>>> expr_long /= (x + 3*y)
>>> print(expr_long)
(x**4 + 6*x**3*y + 11*x**2*y**2 + 6*x*y**3)/(x + 3*y)

>>> expr_short = sy.cancel(expr_long)   # Cancel out the denominator.
x**3 + 3*x**2*y + 2*x*y**2

>>> sy.factor(expr_short)               # Factor the result.
x*(x + y)*(x + 2*y)

# Simplify the trigonometric parts of an expression.
>>> sy.trigsimp(2*sy.sin(x)*sy.cos(x))
<<sin(2*x)>>
\end{lstlisting}

See \url{http://docs.sympy.org/latest/tutorial/simplification.html} for more examples.

\begin{warn}
% Keep the following in mind when simplifying expressions.
\begin{enumerate}
\item Simplifications return new expressions; they do not modify existing expressions in place.

\item The \li{==} operator compares two expressions for exact structural equality, not algebraic equivalence.
Simplify or expand expressions before comparing them with \li{==}.

\item Expressions containing floats may not simplify as expected.
Always use integers and SymPy rationals in expressions when appropriate.
\end{enumerate}

\begin{lstlisting}
>>> expr = 2*sy.sin(x)*sy.cos(x)
>>> sy.trigsimp(expr)
sin(2*x)
>> print(expr)
2*sin(x)*cos(x)                 # The original expression is unchanged.

>>> 2*sy.sin(x)*sy.cos(x) == sy.sin(2*x)
<<False>>                           # The two expression structures differ.

>>> sy.factor(x**2.0 - 1)
x**2.0 - 1                      # Factorization fails due to the 2.0.
\end{lstlisting}
\end{warn}

\begin{problem}
Write a function that computes and simplifies the following expression.
\[\prod_{i=1}^{5} \sum_{j=i}^{5}j(\sin(x)+\cos(x))\]
\end{problem}

\section*{Evaluating Expressions} % ===========================================

Every SymPy expression has a \li{subs()} method that substitutes one variable for another.
The result is usually still a symbolic expression, even if a numerical value is used in the substitution.
The \li{evalf()} method actually evaluates the expression numerically after all symbolic variables have been assigned a value.
Both of these methods can accept a dictionary to reassign multiple symbols simultaneously.

\begin{lstlisting}
>>> x,y = sy.symbols('x y')
>>> expr = sy.expand((x + y)**3)
>>> print(expr)
x**3 + 3*x**2*y + 3*x*y**2 + y**3

# Replace the symbolic variable y with the expression 2x.
>>> expr.subs(y, 2*x)
27*x**3

# Replace x with pi and y with 1.
>>> new_expr = expr.subs({x:sy.pi, y:1})
>>> print(new_expr)
1 + 3*pi + 3*pi**2 + pi**3
>>> new_expr.evalf()                # Numerically evaluate the expression.
71.0398678443373

# Evaluate the expression by providing values for each variable.
>>> expr.evalf(subs={x:1, y:2})
27.0000000000000
\end{lstlisting}

These operations are good for evaluating an expression at a single point, but it is typically more useful to turn the expression into a reusable numerical function.
To this end, \li{sy.lambdify()} takes in a symbolic variable (or list of variables) and an expression, then returns a callable function that corresponds to the expression.

\begin{lstlisting}
# Turn the expression sin(x)^2 into a function with x as the variable.
>>> f = sy.lambdify(x, sy.sin(x)**2)
>>> print(f(0), f(np.pi/2), f(np.pi), sep='  ')
0.0  1.0  1.4997597826618576e-32

# Lambdify a function of several variables.
>>> f = sy.lambdify((x,y), sy.sin(x)**2 + sy.cos(y)**2)
>>> print(f(0,1), f(1,0), f(np.pi, np.pi), sep='  ')
0.2919265817264289  1.708073418273571  1.0
\end{lstlisting}

By default, \li{sy.lambdify()} uses the \li{math} module to convert an expression to a function.
For example, \li{sy.sin()} is converted to \li{math.sin()}.
By providing \li{"numpy"} as an additional argument, \li{sy.lambdify()} replaces symbolic functions with their NumPy equivalents instead, so \li{sy.sin()} is converted to \li{np.sin()}.
This allows the resulting function to act element-wise on NumPy arrays, not just on single data points.

\begin{lstlisting}
>>> f = sy.lambdify(x, 2*sy.sin(2*x), "numpy")
>>> f(np.linspace(0, 2*np.pi, 9))   # Evaluate f() at many points.
array([  0.00000000e+00,   2.00000000e+00,   2.44929360e-16,
        -2.00000000e+00,  -4.89858720e-16,   2.00000000e+00,
         7.34788079e-16,  -2.00000000e+00,  -9.79717439e-16])
\end{lstlisting}

\begin{info}
It is almost always computationally cheaper to lambdify a function than to use substitutions.
According to the SymPy documentation, using \li{sy.lambdify()} to do numerical evaluations ``takes on the order of hundreds of nanoseconds, roughly two orders of magnitude faster than the \li{subs()} method.''

\begin{lstlisting}
<g<In [1]:>g> import sympy as sy
<g<In [2]:>g> import numpy as np

# Define a symbol, an expression, and points to plug into the expression.
<g<In [3]:>g> x = sy.symbols('x')
<g<In [4]:>g> expr = sy.tanh(x)
<g<In [5]:>g> points = np.random.random(10000)

# Time using evalf() on each of the random points.
<g<In [6]:>g> <p<%time>p> _ = [expr.subs(x, pt).evalf() for pt in points]
<<CPU times: user 5.29 s, sys: 40.3 ms, total: 5.33 s
Wall time: 5.36 s>>

# Lambdify the expression and time using the resulting function.
<g<In [7]:>g> f = sy.lambdify(x, expr)
<g<In [8]:>g> <p<%time>p> _ = [f(pt) for pt in points]
<<CPU times: user 5.39 ms, sys: 648 micros, total: 6.04 ms
Wall time: 7.75 ms>>      # About 1000 times faster than evalf().

# Lambdify the expression with NumPy and repeat the experiment.
<g<In [9]:>g> f = sy.lambdify(x, expr, "numpy")
<g<In [10]:>g> <p<%time>p> _ = f(points)
<<CPU times: user 381 micros, sys: 63 micros, total: 444 micros
Wall time: 282 micros>>   # About 10 times faster than regular lambdify.
\end{lstlisting}
\end{info}

\begin{problem}
The Maclaurin series up to order $N$ for $e^x$ is defined as
\begin{equation}
e^x \approx \sum_{n=0}^N \frac{x^n}{n!}.
\label{eq:sympy-maclaurin-exponential}
\end{equation}
Write a function that accepts an integer $N$.
Define an expression for (\ref{eq:sympy-maclaurin-exponential}), then substitute in $-y^2$ for $x$ to get a truncated Maclaurin series of $e^{-y^2}$.
Lambdify the resulting expression and plot the series on the domain $y\in[-2,2]$.
Plot $e^{-y^2}$ over the same domain for comparison.
\\(Hint: use \li{sy.factorial()} to compute the factorial.)

Call your function with increasing values of $N$ to check that the series converges correctly.
\end{problem}

\section*{Solving Symbolic Equations} % =======================================

A SymPy expression by itself is not an equation.
However, \li{sy.solve()} equates an expression with zero and solves for a specified variable.
In this way, SymPy can be used to solve equations.

\begin{lstlisting}
>>> x,y = sy.symbols('x y')

# Solve x^2 - 2x + 1 = 0 for x.
>>> sy.solve(x**2 - 2*x + 1, x)
[1]                                 # The result is a list of solutions.

# Solve x^2 - 1 = 0 for x.
>>> sy.solve(x**2 - 1, x)
[-1, 1]                             # This equation has two solutions.

# Solutions can also be expressions involving other variables.
>>> sy.solve(x/(y-x) + (x-y)/y, x)
[y*(-sqrt(5) + 3)/2, y*(sqrt(5) + 3)/2]
\end{lstlisting}

\begin{problem}
The following equation represents a rose curve in cartesian coordinates:
\begin{equation}
0 = 1 - \frac{(x^2 + y^2)^{7/2} + 18x^5 y - 60x^3 y^3 + 18xy^5}{(x^2 + y^2)^3}.
\label{eq:prob-polar-parametric}
\end{equation}
The curve is not the image of a single function (such a function would fail the vertical line test), so the best way to plot it is to convert (\ref{eq:prob-polar-parametric}) to a pair of parametric equations that depend on the angle parameter $\theta$.

Construct an expression for the nonzero side of (\ref{eq:prob-polar-parametric}) and convert it to polar coordinates with the substitutions $x = r\cos(\theta)$ and $y = r\sin(\theta)$.
Simplify the result, then solve it for $r$.
There are two solutions due to the presence of an $r^2$ term; pick one and lambdify it to get a function $r(\theta)$.
Use this function to plot $x(\theta) = r(\theta)\cos(\theta)$ against $y(\theta) = r(\theta)\sin(\theta)$ for $\theta \in [0,2\pi]$.
\\(Hint: use \li{sy.Rational()} for the fractional exponent.)
\end{problem}

\subsection*{Linear Algebra} % ------------------------------------------------

Sympy can also solve systems of equations.
A system of linear equations $A\x = \b$ is solved in a slightly different way than in NumPy and SciPy: instead of defining the matrix $A$ and the vector $\b$ separately, define the augmented matrix $M = [A\mid\b]$ and call \li{sy.solve_linear_system()} on $M$.

SymPy matrices are defined with \li{sy.Matrix()}, with the same syntax as $2$-dimensional NumPy arrays.
For example, the following code solves the system given below.
\[
\begin{array}{rrrrrrr}
 x & + &   y & + &  z & = & 5 \\
2x & + &  4y & + & 3z & = & 2 \\
5x & + & 10y & + & 2z & = & 4
\end{array}
\]
\begin{lstlisting}
>>> x, y, z = sy.symbols('x y z')

# Define the augmented matrix M = [A|b].
>>> M = sy.Matrix([ [1,  1, 1, 5],
                    [2,  4, 3, 2],
                    [5, 10, 2, 4] ])

# Solve the system, providing symbolic variables to solve for.
>>> sy.solve_linear_system(M, x, y, z)
{x: 98/11, y: -45/11, z: 2/11}
\end{lstlisting}

SymPy matrices support the standard matrix operations of addition \li{+}, subtraction \li{-}, and multiplication \li{@}.
Additionally, SymPy matrices are equipped with many useful methods, some of which are listed below.
See \url{http://docs.sympy.org/latest/modules/matrices/matrices.html} for more methods and examples.

\begin{table}[H]
\centering
\begin{tabular}{r|l}
Method & Returns \\ \hline
% \li{applyfunc()} & Apply a function to every element of the matrix. \\
% \li{condition_number()} & The condition number. \\
\li{det()} & The determinant. \\
% \li{diagonalize()} & Diagonalize a matrix. \\
\li{eigenvals()} & The eigenvalues and their multiplicities. \\
\li{eigenvects()} & The eigenvectors and their corresponding eigenvalues. \\
\li{inv()} & The matrix inverse. \\
\li{is_nilpotent()} & \li{True} if the matrix is nilpotent. \\
% \li{multiply_elementwise()} & Multiply matrices elementwise. \\
\li{norm()} & The Frobenius, $\infty$, $1$, or $2$ norm. \\
\li{nullspace()} & The nullspace as a list of vectors. \\
\li{rref()} & The reduced row-echelon form. \\
\li{singular_values()} & The singular values. \\
\end{tabular}
\end{table}

\begin{warn}
The \li{*} operator performs matrix multiplication on SymPy matrices.
To perform element-wise multiplication, use the \li{multiply_elementwise()} method instead.
\end{warn}

\begin{problem}
Find the eigenvalues of the following matrix by solving for $\lambda$ in the characteristic equation $\det(A - \lambda I) = 0$.
\begin{align*}
A =
\left[\begin{array}{cccc}
x-y & x   & 0   \\
x   & x-y & x   \\
0   & x   & x-y \\
\end{array}\right]
\end{align*}
Also compute the eigenvectors by solving the linear system $A - \lambda I = \0$ for each eigenvalue $\lambda$.
Return a dictionary mapping the eigenvalues to their eigenvectors.
\\(Hint: the \li{nullspace()} method may be useful.)

Check that $A\v = \lambda\v$ for each eigenvalue-eigenvector pair $(\lambda, \v)$.
Compare your results to the \li{eigenvals()} and \li{eigenvects()} methods for SymPy matrices.
\end{problem}

\section*{Calculus} % =========================================================

SymPy is also equipped to perform standard calculus operations, including derivatives, integrals, and taking limits.
Like other elements of SymPy, calculus operations can be temporally expensive, but they give exact solutions whenever solutions exist.

\subsection*{Differentiation} % -----------------------------------------------

The command \li{sy.Derivative()} creates a closed form, unevaluated derivative of an expression.
This is like putting $\frac{d}{dx}$ in front of an expression without actually calculating the derivative symbolically.
The resulting expression has a \li{doit()} method that can be used to evaluate the actual derivative.
Equivalently, \li{sy.diff()} immediately takes the derivative of an expression.

Both \li{sy.Derivative()} and \li{sy.diff()} accept a single expression, then the variable or variables that the derivative is being taken with respect to.

\begin{lstlisting}
>>> x, y = sy.symbols('x y')
>>> f = sy.sin(y)*sy.cos(x)**2

# Make an expression for the derivative of f with respect to x.
>>> df = sy.Derivative(f, x)
>>> print(df)
Derivative(sin(y)*cos(x)**2, x)

>>> df.doit()                       # Perform the actual differentiation.
-2*sin(x)*sin(y)*cos(x)

# Alternatively, calculate the derivative of f in a single step.
>>> sy.diff(f, x)
-2*sin(x)*sin(y)*cos(x)

# Calculate the derivative with respect to x, then y, then x again.
>>> sy.diff(f, x, y, x)
2*(sin(x)**2 - cos(x)**2)*cos(y)    # Note this expression could be simplified.
\end{lstlisting}

\begin{problem} % Single-variable optimization with differentiation.
Let $f:\mathbb{R}\rightarrow\mathbb{R}$ be a smooth function.
A \emph{critical point} of $f$ is a number $x_0\in\mathbb{R}$ satisfying $f'(x_0) = 0$.
The second derivative test states that a critical point $x_0$ is a local minimum of $f$ if $f''(x_0) > 0$, or a local maximum of $f$ if $f''(x_0) < 0$ (if $f''(x_0) = 0$, the test is inconclusive).

Now consider the polynomial
\[
p(x) = 2x^6 - 51x^4 + 48x^3 + 312x^2 - 576x - 100.
\]
Use SymPy to find all critical points of $p$ and classify each as a local minimum or a local maximum.
Plot $p(x)$ over $x\in[-5,5]$ and mark each of the minima in one color and the maxima in another color.
Return the collections of the x-values corresponding to the local minima and local maxima as sets.
\end{problem}

The \emph{Jacobian matrix} of a multivariable function $f:\mathbb{R}^n \rightarrow \mathbb{R}^m$ at a point $\x_0 \in \mathbb{R}^n$ is the $m \times n$ matrix $J$ whose entries are given by
\begin{equation*}
J_{ij} = \frac{\partial f_i}{\partial x_j}(\x_0).
\end{equation*}
For example, the Jacobian for a function $f:\mathbb{R}^3 \rightarrow \mathbb{R}^2$ is defined by
\[
J = \left[\begin{array}{c|c|c}
\arrayrulecolor{gray}
\frac{\partial f}{\partial x_1}&\frac{\partial f}{\partial x_2}&\frac{\partial f}{\partial x_3}
\end{array}\right]
=
\left[\begin{array}{ccc}
\frac{\partial f_1}{\partial x_1}&\frac{\partial f_1}{\partial x_2}&\frac{\partial f_1}{\partial x_3}
\\ \\
\frac{\partial f_2}{\partial x_1}&\frac{\partial f_2}{\partial x_2}&\frac{\partial f_2}{\partial x_3}
\end{array}\right],
\qquad
\text{where}
\qquad
f(\x) =
\left[\begin{array}{c}
f_1(\x) \\ f_2(\x)
\end{array}\right],
\quad
\x = \left[\begin{array}{c}
x_1 \\ x_2 \\ x_3
\end{array}\right].
\]

To calculate the Jacobian matrix of a multivariate function with SymPy, define that function as a symbolic matrix (\li{sy.Matrix()}) and use its \li{jacobian()} method.
The method requires a list of variables that prescribes the ordering of the differentiation.

\begin{lstlisting}
# Create a matrix of symbolic variables.
>>> r, t = sy.symbols('r theta')
>>> f = sy.Matrix([r*sy.cos(t), r*sy.sin(t)])

# Find the Jacobian matrix of f with respect to r and theta.
>>> J = f.jacobian([r,t])
>>> J
Matrix([
[cos(theta), -r*sin(theta)],
[sin(theta),  r*cos(theta)]])

# Evaluate the Jacobian matrix at the point (1, pi/2).
>>> J.subs({r:1, t:sy.pi/2})
Matrix([
[0, -1],
[1,  0]])

# Calculate the (symbolic) determinant of the Jacobian matrix.
>>> sy.simplify(J.det())
r
\end{lstlisting}

\subsection*{Integration} % ---------------------------------------------------

The function \li{sy.Integral()} creates an unevaluated integral expression.
This is like putting an integral sign in front of an expression without actually evaluating the integral symbolically or numerically.
The resulting expression has a \li{doit()} method that can be used to evaluate the actual integral.
Equivalently, \li{sy.integrate()} immediately integrates an expression.

Both \li{sy.Derivative()} and \li{sy.diff()} accept a single expression, then a tuple or tuples containing the variable of integration and, optionally, the bounds of integration.

\begin{lstlisting}
# Calculate the indefinite integral of sec(x).
>>> sy.integrate(sy.sec(x), x)
-log(sin(x) - 1)/2 + log(sin(x) + 1)/2

# Integrate cos(x)^2 from 0 to pi/2.
>>> sy.integrate(sy.cos(x)**2, (x,0,sy.pi/2))
pi/4

# Compute the integral of (y^2)(x^2) dx dy with x from 0 to 2, y from -1 to 1.
>>> sy.integrate(y**2 * x**2, (x,0,2), (y,-1,1))
16/9
\end{lstlisting}

\begin{problem}
% Many unpleasant integrals can be made tractable by a \emph{change of variables}, such as using polar, cylindrical, or spherical coordinates instead of the usual Cartesian system.
Let $f:\mathbb{R}^3\rightarrow\mathbb{R}$ be a smooth function.
The volume integral of $f$ over the sphere $S$ of radius $r$ can written in spherical coordinates as
\begin{align*}
\underset{S}{\iiint} f(x,y,z) dV = \int_0^\pi\int_0^{2\pi}\int_0^r f(h_1(\rho, \theta, \phi), h_2(\rho, \theta, \phi), h_3(\rho, \theta, \phi)) |\det(J)| \:d\rho\:d\theta\:d\phi,
\end{align*}
where $J$ is the Jacobian of the function $h:\mathbb{R}^3\rightarrow\mathbb{R}^3$ given by
\[
h(\rho, \theta, \phi) =
\left[\begin{array}{c}
h_1(\rho, \theta, \phi) \\
h_2(\rho, \theta, \phi) \\
h_3(\rho, \theta, \phi)
\end{array}\right]
=
\left[\begin{array}{c}
\rho\sin(\phi)\cos(\theta) \\
\rho\sin(\phi)\sin(\theta) \\
\rho\cos(\phi)
\end{array}\right].
\]
Calculate the volume integral of $f(x,y,z) = (x^2 + y^2 + z^2)^2$ over the sphere of radius $r$.
Lambdify the resulting expression (with $r$ as the independent variable) and plot the integral value for $r \in [0, 3]$.
In addition, return the value of the integral when $r=2$.
\\ (Hint: simplify the integrand before computing the integral.
In this case, $|\det(J)| = -\det(J)$.)

To check your answer, when $r=3$, the value of the integral is $\frac{8748}{7}\pi$.
\end{problem}

\begin{warn}
SymPy isn't perfect.
It solves some integrals incorrectly, simplifies some expressions poorly, and is significantly slower than numerical computations.
However, it is generally very useful for simplifying parts of an algorithm, getting exact answers, and handling tedious algebra quickly.
\end{warn}

\newpage

\section*{Additional Material} % ==============================================

\subsection*{Pretty Printing} % -----------------------------------------------

SymPy expressions, especially complicated ones, can be hard to read.
Calling \li{sy.init_printing()} changes the way that certain expressions are displayed to be more readable; in a Jupyter Notebook, the rendering is done with \LaTeX, as displayed below.
Furthermore, the function \li{sy.latex()} converts an expression into actual \LaTeX \ code for use in other settings.

\begin{center}
\includegraphics[width=\textwidth]{figures/pretty_printing.png}
\end{center}

% SymPy also has a simplified plotting wrapper around Matplotlib.
% See \url{http://docs.sympy.org/latest/modules/plotting.html} for further details.
% \begin{lstlisting}
% # Plot sin(x)e^x where x ranges from -3 to 3.
% >>> expr = sy.sin(x) * sy.exp(x)
% >>> sy.plot(expr, (x, -3, 3))
% \end{lstlisting}
%
% \begin{figure}[H]
% \includegraphics[width=.7\textwidth]{figures/plotting_wrapper.pdf}
% \end{figure}

\subsection*{Limits} % --------------------------------------------------------

Limits can be expressed, similar to derivatives or integrals, with \li{sy.Limit()}.
Alternatively, \li{sy.limit()} (lowercase) evaluates a limit directly.

\begin{lstlisting}
# Define the limit of a^(1/x) as x approaches infinity.
>>> a, x = sy.symbols('a x')
>>> sy.Limit(a**(1/x), x, sy.oo)
<<Limit(a**(1/x), x, oo, dir='-')>>

# Use the doit() method or sy.limit() to evaluate a limit.
>>> sy.limit((1+x)**(1/x), x, 0)
E

# Evaluate a limit as x approaches 0 from the negative direction.
>>> sy.limit(1/x, x, 0, '-')
-oo
\end{lstlisting}

Use limits instead of the \li{subs()} method when the value to be substituted is $\infty$ or is a singularity.

\begin{lstlisting}
>>> expr = x / 2**x
>>> expr.subs(x, sy.oo)
nan
>>> sy.limit(expr, x, sy.oo)
0
\end{lstlisting}

Refer to \url{http://docs.sympy.org/latest/tutorial/calculus.html} for SymPy's official documentation on calculus operations.

\begin{comment} % Too much additional material already.
\subsection*{Series Expansions} % ---------------------------------------------

Series can be depicted using the summation function, but SymPy also has built-in functions to depict them.
Expressions involving series can be created using \li{series()} as a function or as a method.
The function takes in the expression, variable, the point the series is centered at, and the order of the series.
The point the series is centered at defaults to zero and the order to six.

\begin{lstlisting}
# Find the terms in the series for cos(x) centered around x = 0 of order 6.
>>> sy.series(sy.cos(x),x)
1 - x**2/2 + x**4/24 + O(x**6)

# Center the series at x = pi of order 4.
>>> sy.series(sy.cos(x),x,sy.pi, 4)
-1 + (x - pi)**2/2 + O((x - pi)**4, (x,pi))

# Use removeO() to remove the order from the output.
>>> sy.series(sy.exp(sy.cos(x)),x,1,2).removeO()
-(x - 1)*exp(cos(1))*sin(1) + exp(cos(1))
\end{lstlisting}

Power series can be represented in SymPy with the ``Formal Power Series'' class.
When an \li{fps} object is created, it returns sequences of coefficients and powers, followed by the independent and then the common terms.
Use the \li{truncate()} or \li{polynomial()} methods to visualize individual terms of the series.
These methods input the highest order to be visualized and default to six.

\begin{lstlisting}
>>> f = sy.fps(sy.cos(x),x)
>>> f.truncate()
1 - x**2/2 + x**4/24 + O(x**6)
>>> f.polynomial()
x**4/24 - x**2/2 + 1
\end{lstlisting}

Further series features in SymPy include the computation of residues, laurent series and fourier series.
\end{comment}

\subsection*{Numerical Integration} % -----------------------------------------

Many integrals cannot be solved analytically.
% Additionally, the definite integral of a function may be undefined.
% If SymPy is unable to compute an integral, it returns an integral object.
As an alternative to the \li{doit()} method, the \li{as_sum()} method approximates the integral with a summation.
This method accepts the number of terms to use and a string indicating which approximation rule to use (\li{"left"}, \li{"right"}, \li{"midpoint"}, or \li{"trapezoid"}).

\begin{lstlisting}
>>> x = sy.symbols('x')

# Try integrating e^(x^2) from 0 to pi.
>>> I = sy.Integral(sy.exp(x**2), (x,0,sy.pi))
>>> I.doit()
sqrt(pi)*erfi(pi)/2                 # The result is not very helpful.

# Instead, approximate the integral with a sum.
>>> I.as_sum(10, 'left').evalf()
1162.85031639195
\end{lstlisting}

% In addition, SymPy includes several integral transforms, such as the Laplace, Fourier, sine, and cosine transforms.
% It can also do several types of integrals, such as line integrals.
See \url{http://docs.sympy.org/latest/modules/integrals/integrals.html} for more documentation on integration with SymPy.

\subsection*{Differential Equations} % ----------------------------------------

SymPy can be used to solve both ordinary and partial differential equations.
The documentation for working with PDE functions is at \url{http://docs.sympy.org/dev/modules/solvers/pde.html}

The general form of a first-order differential equation is $\frac{dx}{dt} = f(x(t),t)$.
To represent the unknown function $x(t)$, use \li{sy.Function()}.
Just as \li{sy.solve()} is used to solve an expression for a given \textbf{variable}, \li{sy.dsolve()} solves an ODE for a particular \textbf{function}.
When there are multiple solutions, \li{sy.dsolve()} returns a list; when arbitrary constants are involved they are given as \li{C1}, \li{C2}, and so on.
Use \li{sy.checkodesol()} to check that a function is a solution to a differential equation.

\begin{lstlisting}
>>> t = sy.symbols('t')
>>> x = sy.Function('x')

# Solve the equation x''(t) - 2x'(t) + x(t) = sin(t).
>>> ode = x(t).diff(t, t) - 2*x(t).diff(t) + x(t) - sy.sin(t)
>>> sy.dsolve(ode, x(t))
Eq(x(t), (C1 + C2*t)*exp(t) + cos(t)/2) # C1 and C2 are arbitrary constants.
\end{lstlisting}

Since there are many types of ODEs, \li{sy.dsolve()} may also take a hint indicating what solving strategy to use.
See \li{sy.ode.allhints} for a list of possible hints, or use \li{sy.classify_ode()} to see the list of hints that may apply to a particular equation.

% Old problem.
\begin{comment}
\begin{problem}
Return the solution to the following differential equation.
\begin{equation*}
 y_{xxx} + 3y_{xx} + 3y_{x} + y = x^{5}e^x + x^{4}\sin\left(x\right) + x^{3}\cos\left(3x\right)
\end{equation*}

Check the result $y(x)$ by plugging it back into an expression where one side of the equation is subtracted to another.
Use the \li{doit()} and \li{simplify()} methods to ensure that the new expression simplifies to $0$.
\end{problem}
\end{comment}