%*******************************************************************************
%***********************************    Background   *****************************
%*******************************************************************************
%!TEX root = 0.main.tex

\setcounter{page}{1}

%********************************** %First Section  **************************************
\section {Introduction and general background}

\subsection{Introduction}

Neural Networks (NNs) are popular algorithms for regression and classification tasks. Taking as example an image classification problem, a neural network perform multiple combinations of linear and non-linear transformations of each image $I$ to assign it a label $C_I$ chosen in the set of all the possible labels $\mathcal C$. The first \textit{layer} of the neural network transforms the input image $I$ in a vector - called \textit{feature map} - $\mathbf f_1$ through a function $\phi_1$.  The output feature map of the first layer is used as input of the second layer that transforms it through a function $\phi_2$, and so on, until the original image $I$ is mapped into a label $C_I$ by the last, $n$-th layer of the neural network:
$$C_I = \phi_n \circ \phi_{n-1}\circ ... \phi_2\circ\phi_1 (I)$$
 With a large \textit{training set} of pre labeled images at its disposal, a NN is capable of learning the optimal transformations $\phi_i$ that let it map each input image to its correct label. Since the functions $\phi_i$ have many degrees of freedom - even millions - a neural network is able to learn very complex transformations. In the work of Cs\'aji \cite{NN}, NNs have been proved to be universal function approximators, meaning that with a sufficient number of parameters NNs are able to approximate any continuous function on a compact domain. This makes NNs the optimal tool for complex tasks such as image classification, image segmentation, speech recognition and natural language processing.

Convolutional Neural Networks (CNNs) are a subset of NNs whose layer structure has been specifically designed for image recognition and segmentation. For this purpose, they don't have all the degrees of freedom of a \textit{fully connected} neural network: each layer is constrained to learn only those transformations of the input that are \textit{equivariant} to translations of the input. This means that a translation of the input image will not result in a change of class. The layers $\phi_i$ of a CNN are \textit{convolutions} with some kernels $k_i$, that were learned during the training phase. Thanks to their design, training of CNNs is faster - thanks to the smaller number of parameters to be learned compared to a fully connected NN -, easier - since there's no need of artificially \textit{augmenting} the dataset with translated copies of the same image -, and leads to very accurate results \cite{SCNN}, \cite{Esteves}.

Spherical convolutional neural networks (SCNNs) are CNNs that have been designed to deal with spherical data, whose layer design makes them equivariant to \textit{rotations} of the input.  Examples of tasks where data is naturally represented on a sphere are (i) climate science, where data is sampled on the surface of the Earth, (ii) cosmology, where observations are naturally projected on a sphere centered around the observer (see Figure \ref{fig:cosmicradiation}), and (iii) virtual reality, where the images are represented on a sphere centered around the player. Being able to come up with rotation equivariant architectures brings with it all the advantages that traditional CNNs have brought for traditional (euclidean) image classification tasks: training is faster, easier and results are very accurate. Each layer of a SCNN performs a \textit{spherical convolution} of the input feature map with a kernel $k_i$ learned during the training phase. One of the main issues with traditional SCNNs is the computational complexity of computing at each layer the Spherical Harmonic Transform of the data to perform the convolution. To overcome this issue, Perraudin et al. \cite{DeepSphere} proposed a Graph Convolutional Neural Network (GCNN) that is almost equivariant to rotations, replacing the SHT with a more efficient Graph Convolution.
\begin{figure}
	\centering
	\caption{\label{fig:cosmicradiation} Cosmic microwave background map, the oldest electromagnetic radiation in the universe. Source: Wikipedia}
	\includegraphics[width=0.4\textwidth]{figs/literaturereview/WMAP.png}
\end{figure}

This work is organized as follows: in Chapter 1 we start by presenting fundamental concepts of spectral theory on the sphere and we present classical ways of building rotation equivariant neural networks through the use of the classical SHT.  We present then some basics of Spectral Graph Theory that lay the foundations of Graph Convolutional Neural Networks.  In Chapter 2 we present the general framework of how to discretize the Laplace-Beltrami operator on a general manifold, concluding with the special case of the Heat Kernel Graph Laplacian (HKGL) approximation, together with some convergence results. We continue in Chapter 3 by introducing DeepSphere \cite{DeepSphere}, a Graph Spherical Convolutional Neural Network (GCNN) that uses a graph Laplacian matrix $\mathbf L$ similar to the HKGL to perform graph convolutions that are almost equivariant to rotations. We study the spectral properties and the equivariance error of DeepSphere and we show a way to build a graph $G'$ such that the corresponding graph Laplacian matrix $\mathbf L'$ shows better spectral and equivariance properties. In Chapter 4 we show better graph constructions than the HKGL on non uniform sampling measures. To conclude, we show a different approach to perform rotation invariant convolutions that uses the Finite Element Method (FEM) approximation of the Laplace-Beltrami operator on the the sphere. Chapter 5 concludes this work by presenting some experimental results obtained by Gusset et al. \cite{Gusset} that implemented the graph $G'$ on a GCNN and compared its performances to DeepSphere on a well known dataset \cite{SHREC17} showing that the new graph $G'$ performs better in real applications. We finish by comparing the FEM and the graph approach, discussing the general problem of how to incorporate geometrical informations about the sphere in the graph.

\subsection{Fourier Transforms and Convolutions on the 2-Sphere}\label{sec:Fourier on the Sphere}
The goal of this section is to present to the reader some fundamental results of spectral theory on the sphere that we will need in this work. We present a brief review of Banach and Hilbert spaces, spherical harmonics, Fourier transform and convolution on $\mathbb S^2$. We refer to Sections 2 and 3 of the work of Driscoll and Healy \cite{Driscoll:1994:CFT:184069.184073} for a more detailed and effective review of spectral theory on the Sphere.

\paragraph{Banach and Hilbert spaces.}
A \textit{norm} $\norm\cdot:\ X\to\mathbb R$ on a vector space $X$ is a subadditive, positive definite function such that $\norm{x+y}\leq\norm x +\norm y,\ \forall x,y\in X$ (triangle inequality). A \textit{Cauchy sequence} $(x_n)\subset X$ is a sequence such that $\forall \epsilon>0\  \exists M>0: $ $\forall i,j>M$ $ \norm{x_i-x_j}<\epsilon$. A \textit{Banach space} $(X, \norm{\cdot})$ is a normed vector space on the scalar field $F$ that is \textit{complete}, meaning that $X$ is "big enough" such that for every Cauchy sequence $(x_n)\subset X$ there exist a $x\in X$ such that $x$ is the limit of $(x_n)$ in $X$ i.e. $\norm{x_n-x}\rightarrow 0$. A \textit{basis} of $(X, \norm\cdot)$ is a minimal set of linearly independent vectors $\mathcal B \subset X$ such that every element of $X$ can be written as linear combination of the elements of $\mathcal B$. A scalar product is a function $\langle\cdot,\cdot\rangle: X\times X \rightarrow \mathbb F$ that is linear in the first argument, positive definite and conjugate symmetric. Through a scalar product we can define the notion of angle $\theta$ between two elements $x, y \in X$ through the following formula:
$$
\cos \theta = \frac{\langle x, y\rangle}{\norm x \norm y}
$$
In particular we can define the notion of orthogonality: two elements  $x, y \in X$ are orthogonal if and only if $\langle x, y\rangle=0$. We are interested in those particular Banach spaces where we can define a notion of orthogonality between vectors. A Banach space $(X, \norm \cdot )$ is a \textit{Hilbert space} when the norm $ \norm \cdot $ can be induced by a \textit{scalar product}: $\norm \cdot = \sqrt{\langle\cdot,\cdot\rangle}$. We can now define an \textit{orthonormal} basis of $X$: a basis $\mathcal B \subset X$ such that $\forall x, y \in \mathcal B, \norm x = \norm y = 1 \text{and } \langle x, y\rangle = 0$. Given an orthonormal basis $\mathcal B = \{b_i\}_{i\in I}$ we can write each vector in its \textit{Fourier series}
\begin{equation}\label{eq:abstract fourier}
x = \sum_{i\in I} \langle x, b_i\rangle b_i
\end{equation}
If the set $I$ is countable the Hilbert space $(X, \norm\cdot)$ is called \textit{separable}. Having a countable orthonormal basis, and thus the possibility of representing each vector through its Fourier series enormously simplifies many problems.
\paragraph{Spherical Harmonics.}
 Given the usual parametrization $x = x(\theta, \phi), \theta\in[0,\pi], \phi\in[0,2\pi]$ of the sphere
\begin{align*}
\mathbb{S}^{2}&=\left\{\omega=\left(\omega_{1}, \omega_{2}, \omega_{3}\right) \in \mathbb{R}^{3} :\|x\|_{\mathbb{R}^{3}}=\left(\omega_{1}^{2}+\omega_{2}^{2}+\omega_{3}^{2}\right)^{1 / 2}=1\right\}\\
\omega_{1}&=\cos (\phi) \sin (\theta), \quad \omega_{2}=\sin (\phi) \sin (\theta), \quad \omega_{3}=\cos (\theta)
\end{align*}
the Hilbert space $L^2(\mathbb S^2)$ is defined as the space of square-integrable functions endowed with the scalar product $\langle f,g\rangle=\int_{\mathbb S^2}f(\omega)\overline g(\omega)d\omega$ where the measure $d\omega$ is the rotation-invariant measure such that
\begin{align}
\int_{\omega \in \mathbb S^{2}} f(\omega) d \omega&=\int_{\phi=0}^{2 \pi} \int_{\theta=0}^{\pi} f(\omega(\theta, \phi)) \sin \theta d \theta d \phi\\
\int_{\omega \in \mathbb S^{2}} f(g \omega) d \omega&=\int_{\omega \in \mathbb S^{2}} f(\omega) d \omega, \quad g \in S O(3)
\end{align}

For each rotation $g\in SO(3)$ we define a corresponding rotation operator $\Lambda(g)$ by
\begin{equation}\label{eq:rotation operator}
	\Lambda(g) f(\omega)=f\left(g^{-1} \omega\right)
\end{equation}
A space is invariant under the rotations $g$ in $SO(3)$ if all operators $\Lambda(g)$ take each function of the space back into the space. As very well written by Driscoll et al \cite{Driscoll:1994:CFT:184069.184073}:

\vspace{0.2cm}
\textit{Fourier analysis on the sphere amounts to the decomposition of the space of square integrable functions on \(\mathbb S^{2}\) in minimal subspaces $V_\ell$ invariant under all of the rotations in \(S O(3),\) thus simplifying the analysis of rotation-invariant operators.}
\vspace{0.2cm}

It's a well known fact \cite{Driscoll:1994:CFT:184069.184073} that the $\ell$-th invariant subspace $V_\ell\subset L^2(\mathbb S^2)$ is made of polynomials of $\mathbb R^3$ of degree $\ell$ restricted to $\mathbb S^2$, and has dimension $2\ell+1$. Its elements are called \textit{spherical harmonics} of degree $\ell$. These subspaces are orthogonal between them, and correspond to the eigenspaces of the Laplace-Beltrami operator $\Delta_{\mathbb S^2}$. For a thorough introduction to how to define the Laplace-Beltrami operator on a manifold and its properties, see \cite{rosenberg_1997}. The set of all the orthonormal basis $Y_\ell^m,\ -\ell\leq m\leq\ell$ of each subspace $V_\ell$ gives an orthonormal basis of $L^2(\mathbb S^2)$. The analytical expression of the spherical harmonic $Y_\ell^m(\theta, \phi)$ is actually known \cite{Driscoll:1994:CFT:184069.184073}:
\begin{equation}\label{eq:spherical harmonics}
	Y_\ell^m(\theta, \phi) = (-1)^{m} \sqrt{\frac{(2 \ell+1)(\ell-m) !}{4 \pi(\ell+m) !}} P_{\ell}^{m}(\cos \theta) e^{i m \phi}
\end{equation}
where $P_{\ell}^{m}$ are the \textit{Legendre functions} as defined in \cite{Driscoll:1994:CFT:184069.184073}.
\vspace{0.5cm}
\begin{remark}
	Saying that the space $V_\ell$ is invariant under rotations $SO(3)$ means that under any rotation $g\in SO(3)$, any spherical harmonic $Y_\ell^m\in V_\ell$ is transformed into a linear combination of the others spherical harmonics of the same degree $\ell$:
	$$
	\Lambda(g) Y_{\ell}^{m}(\omega)=\sum_{|k| \leq \ell} Y_{\ell}^{k}(\omega) \alpha_{k, m}^{(\ell)}(g).
	$$
\end{remark}
\vspace{0.5cm}

\paragraph{Fourier transform.}
We can now expand each function $f\in L^2(\mathbb S^2)$ in the orthonormal coordinate system given by the spherical harmonics
\begin{align}\label{eq:inverse spherical fourier transform}
	f(\omega) &= \sum_{\ell\in\mathbb N}\sum_{|m|\leq \ell}\hat f(\ell,m) Y_\ell^m(\omega)\\
	\hat f(\ell,m) &=\int_{\omega\in\mathbb S^2}f(\omega)Y_\ell^m(\omega)d\omega \label{eq:SHT}
\end{align}
where the coefficients $\hat f(\ell,m)$ are the \textit{Fourier coefficients} of $f$. The computation of $\hat f(\ell, m)$ is called Spherical Harmonic Transform (SHT). Thanks to equation (\ref{eq:spherical harmonics}) we can decompose the computation of the SHT (\ref{eq:SHT}) in the two directions $(\theta, \phi)$. One reason for which the most popular sampling schemes of the sphere have the pixels lie on isolatitude circles is that it is possible to use standard one-dimensional FFT algorithms to compute the longitudinal part of the transform, making the computation of the SHT $\mathcal O(n^{3/2})$, where $n$ is the number of pixels \cite{HEALPix}.

\paragraph{Convolutions.}
Convolution on the sphere is profoundly different than convolution on the Euclidean plane $\mathbb R^2$. Since translations are isomorphic to $\mathbb R^2$, the convolution $f*g(x)$ of two functions $f, g \in L^2(\mathbb R^2)$ is itself a function on the plane:
\begin{equation}\label{eq:plane convolution}
	 \int_{\mathbb R^2} f(y)g(x-y)dy = f*g(x):\quad \mathbb R^2 \to\mathbb R
\end{equation}
On the sphere things work differently: translations are replaced by rotations, but due to the fact that $SO(3)$ is not isomorphic to $\mathbb S^2$, if we define the convolution on $\mathbb S^2$ as follows:
\begin{equation} \label{eq:cohen convolution}
f* k(g) := \int_{\eta \in \mathbb S^2} \Lambda(g)k( \eta) f(\eta) d\eta=\int_{\eta \in \mathbb S^2} k(g^{-1} \eta) f(\eta) d\eta\\
\end{equation}
$f*k(g): SO(3)\to\mathbb R$ is not a function of the sphere anymore, but it is a function of the special rotation group $SO(3)$. In section \ref{sec:Chapter1:SCNN} we will explain how Cohen et al. \cite{SCNN} use in their work this definition of convolution on the sphere to construct a rotation equivariant NN. However, the definition of convolution that we will use in this work is the following, where the integral is performed not on the sphere but on the rotation group $SO(3)$:

\begin{equation}\label{eq:convolution}
 k * f(\omega)=\int_{g \in S O(3)} k(g \eta) f\left(g^{-1} \omega\right) d g
\end{equation}

where $dg$ is the measure on $SO(3)$ that can be written in terms of the three Euler angles $(\theta, \phi, \psi)$
$$dg=\sin\theta d\theta d\phi d\psi$$
In this way $k * f(\omega)$ is still a function defined on $\mathbb S^2$. However, integrating on $SO(3)$ means integrating on the third Euler angle $\psi$, that in practice means using definition (\ref{eq:cohen convolution}) with the use of \textit{radial} kernels $k$ only.
Using the convolution defined in equation (\ref{eq:convolution}), the following theorem \cite{Driscoll:1994:CFT:184069.184073} generalizes on the sphere a well known property of convolutions and Fourier transforms:
\vspace{0.5cm}
\begin{theorem}\label{theo:convolution}
	Given two functions $f, h$ in $L^2(\mathbb S^2)$, the Fourier transform of the convolution is a pointwise product of the transforms
$$
\hat{(f * h)}(\ell, m)=2 \pi \sqrt{\frac{4 \pi}{2 \ell+1}} \hat{f}(\ell, m) \hat{h}(\ell, 0).
$$
\end{theorem}
\vspace{0.5cm}

\subsection{Spherical Convolutional Neural Networks}\label{sec:Chapter1:SCNN}
Cohen et al. \cite{SCNN} proposed a NN where the first layer performs a convolution on the sphere as defined by equation (\ref{eq:cohen convolution}). The output feature map - a signal on $SO(3)$ - is processed by the deeper layers that perform other convolutions in $SO(3)$. All these convolutions are performed in the spectral domain as in theorem \ref{theo:convolution}, meaning that every signal has to be Fourier-transformed first, at each forward and backward step of the training phase of the NN. This approach, even with the use of Generalized FFT algorithms for $\mathbb S^2$ and $SO(3)$, remains both computationally expensive ($\mathcal O(n^{3/2})$) and memory expensive, due to the need of storing kernels defined on the much bigger space $SO(3)$. In section \ref{sec:Chapter5:Experimental validation} we report in table (\ref{tab:SHREC17_class}) the results of Gusset et al. \cite{Gusset}, that compared both the training and inference time of Cohen's SCNN, showing how slow this architecture is compared to other rotation equivariant NNs.
\subsection{Spectral Graph Theory} \label{sec:Chapter1: Spectral Graph Theory}
\paragraph{Graphs.}
For the purposes of this work, a \textit{weighted undirected graph} $G(V, E, \mathbf W)$ is defined by a vertex set $V$, an edge set $E$, where the edges are unordered pairs of vertices, and the matrix $\mathbf W$ whose entries $w_{ij}$ represent the weight of the edge $(v_i, v_j)$. $G$ is a \textit{simple} graph, if $w_{ij}$ assume only values in $\{0, 1\}$. Undirected graphs are common mathematical objects used to model simple, symmetric relationships between things. The edge $e_{ij} = (v_i, v_j) \in E$ is the mathematical object that translates the fact that the vertices $v_i, v_j$ are in a relationship, and the weight $w_{ij}$ measures how strong this relationship is. Common examples of graphs include friendship graphs, where people are the vertices and the edges represent friendship, or electric network graphs, where vertices represent electronic components and edges represent wires.
\paragraph{The graph Laplacian.}
If $\mathbf D$ is the diagonal matrix $\mathbf D_{ii} = \sum_j w_{ij}$, one can define \cite{Vandergheynst} the combinatorial graph Laplacian $\mathbf L$
\begin{equation}\label{eq:graph Laplacian}
		\mathbf L = \mathbf D-\mathbf W
\end{equation}
 and the symmetric normalized graph Laplacian $\mathbf L'$
\begin{equation}\label{eq:normalized graph Laplacian}
\mathbf L' =  \mathbf D^{-1/2}\mathbf L\mathbf D^{-1/2} = \mathbf I - \mathbf D^{-1/2}\mathbf W\mathbf D^{-1/2}
\end{equation}
In a simple friendship graph $G$, one can define a vector $\mathbf f$ such that each entry $ f_i$ is the age of the person associated with the vertex $v_i$, and could try to measure how much people tend to be friends with people of the same age. In other words, how smooth the signal $\mathbf f$ is on the graph $G$. A good measure for the smoothness of a signal on a graph is given by the \textit{Dirichlet energy} of the signal $\mathbf f$, i.e., the quadratic form associated with the normalized Laplace operator $\mathbf L'$:
\begin{equation}\label{eq:quadratic form}
	\mathbf f^\intercal \mathbf L' \mathbf f = \sum_{\left(v_{j}, v_{k}\right) \in {E}} \frac{\boldsymbol{W}_{j k}}{\sqrt{d_{j} {d}_{k}}}\left({f}_{j}-{f}_{k}\right)^{2}
\end{equation}
The reason why the Dirichlet energy is a good measure of the smoothness of $\mathbf f$ is easier to understand in the case of a simple graph, where it reduces to the sum
\begin{equation}\label{eq:simple dirichlet energy}
	\mathbf f^\intercal \mathbf L' \mathbf f = \sum_{\left(v_{j}, v_{k}\right) \in {E}} \left({f}_{j}-{f}_{k}\right)^{2}.
\end{equation}
 that will grow for each edge $(v_i, v_j)$ connecting people with very different age. Although the Dirichlet energy (\ref{eq:quadratic form}) works also for the combinatorial graph Laplacian $\mathbf L$, in practice it is preferred to use the symmetric normalized Laplacian when the degree distribution is wide. Another way of looking at equation (\ref{eq:simple dirichlet energy}) is as the following: the differences ${f}_j-{f}_k$ can be seen as the \textit{gradient} $\nabla \mathbf{f}$ that is a signal on the edges $(v_j, v_k)$ and equation (\ref{eq:simple dirichlet energy}) as the quadratic norm of such gradient $\norm {\nabla \mathbf{f}}^2$.
\paragraph{Graph Fourier transform}
Since the graph Laplacian is a symmetric matrix, we can write its eigen decomposition \cite{Strang}
$$
\mathbf L = \mathbf V\mathbf \Lambda\mathbf V^\intercal
$$
 where $\mathbf V$ is the orthonormal basis of $\mathbb R^n$ of eigenvectors, and $\mathbf \Lambda$ the real diagonal matrix of the eigenvalues $\mathbf \Lambda = \text{diag}(\lambda_i\in \mathbb R)$. Similarly to the continuous domain, where the Fourier transform of a signal $f$ is defined as the projection of $f$ on the orthonormal eigenbasis of the Laplace-Beltrami operator $\Delta$, on a graph we can define a graph Fourier transform $\mathcal F_G: \mathbb R^n\to\mathbb R^n$ of a discrete signal $\mathbf f\in\mathbb R^n$ as the projection of $\mathbf f$ on the eigenvectors of the graph Laplacian $\mathbf L$:
\begin{equation}\label{eq:graph fourier}
\mathcal F_G(\mathbf f) := \mathbf V^\intercal\mathbf f = \hat{\mathbf f}
\end{equation}
The inverse graph Fourier transform $\mathcal F^{-1}_G$ is thus
\begin{equation}\label{eq:graph fourier inverse}
\mathcal F^{-1}_G(\hat{\mathbf f}) := \mathbf V \hat{\mathbf f} = \mathbf V\mathbf V^\intercal\mathbf f = {\mathbf f}
\end{equation}
In the continuous case, the eigenvalues of the Laplace-Beltrami operator are associated with a notion of \textit{frequency} of the corresponding eigenfunction. In a graph we have a similar notion: define the Rayleigh quotient of a vector $\mathbf v \in \mathbb R^n$ to be
\begin{equation}\label{eq:Rayleigh}
	\mathcal R(\mathbf v) = \frac{\mathbf v^\intercal\mathbf L \mathbf v}{\mathbf v^\intercal\mathbf v}
\end{equation}
The well known \cite{Strang} Courant-Fischer characterization of eigenvalues and eigenvectors of symmetric matrices (\ref{eq:courantfisher}) can be interpreted in light of what we wrote about the interpretation of the Dirichlet energy (\ref{eq:quadratic form}) $\mathbf v^\intercal \mathbf L \mathbf v$ as a measure of the smoothness of $\mathbf v$. The eigenvalue $\lambda_i$ is the measure of smoothness of the eigenvector $\mathbf v_i$, that is the smoothest vector perpendicular to the lower-degree eigenvectors $\mathbf v_1, ... \mathbf v_{i-1}$.
\begin{equation}\label{eq:courantfisher}
	\begin{aligned}
	&\lambda_1 = \min_{\mathbf v\neq 0} \mathcal R(v)\\
	&\mathbf v_1 = \text{argmin}_{\norm {\mathbf v} = 1, \mathbf v\neq 0} \mathcal R(v)\\
	&\begin{cases}
	\lambda_i = \min_{\norm {\mathbf v} = 1, \mathbf v\perp \mathbf v_1, ..., \mathbf v_{i-1}} \mathcal R(v)\\
	\mathbf v_i = \text{arg min}_{\norm {\mathbf v} = 1, \mathbf v\perp \mathbf v_1, ..., \mathbf v_{i-1}}  \mathcal R(v)
	\end{cases}
	\end{aligned}
\end{equation}
\begin{remark}
	It is interesting to notice that the Dirichlet energy of a signal $\mathbf f$ on a graph $G$ could change drastically by changing the underlying topology of $G$. In figure \ref{fig:graph} we see two simple graphs $G=(V, E),\ G' =(V, E')$ with the same signal $\mathbf f$ represented as vertical red bars over the vertex set $V$. On $G$, the signal $\mathbf f$ varies smoothly across the graph since the edges $V$ connect only those vertices with similar values of $\mathbf f$. On $G'$, since we added edges between vertices with very different values of $\mathbf f$, we will have that the Dirichlet energy of $\mathbf f$ calculated on the graph $G'$ will be much higher than the one calculated on the graph $G$
\end{remark}
\begin{figure}
	\centering
	\includegraphics[width=0.8\textwidth]{figs/chapter1/graph.png}
	\caption{\label{fig:graph}Different graph topologies can drastically change the measure of smoothness of a signal $\mathbf f$, here represented as red vertical bars on the vertices.}
\end{figure}

\paragraph{Convolution and filtering on graphs}
 As the plane $\mathbb R^2$ is symmetric with respect to any translation, and the sphere $\mathbb S^2$ is symmetric with respect to any rotation, the respective definitions of convolution are equivariant respectively to these two symmetry groups. Since there are no such global symmetries in a general graph $G$, definitions (\ref{eq:plane convolution}), (\ref{eq:convolution}) can not be extended naturally on graphs. However, both on $\mathbb R^2$ and on $\mathbb S^2$, the convolution of a signal $f$ with a kernel $k$ can be performed in the spectral domain by multiplying the transformed signal $\hat f$ times the transformed kernel $\hat k$:
\begin{equation}\label{eq:convolution normal}
f*k = \mathcal F^{-1}(\hat f \cdot \hat k)
\end{equation}
In a similar way we can define a notion of convolution also in the graph spectral domain. We use the following definition \cite{Vandergheynst} of the convolution of a signal $\mathbf f$ times a kernel $\mathbf K$:
\begin{equation}\label{eq:graph convolution}
	\Omega_\mathbf K\mathbf f = \mathcal{F}^{-1}_G(\mathbf K \hat {\mathbf f})= \mathbf V\mathbf K  \mathbf V^\intercal {\mathbf f}
\end{equation}
where $\mathbf K$ is a \textit{diagonal} \textit{matrix} $K_{ii} = k_i$. Graphs convolutions are different from the ones we are used to define in Euclidean domains, since the graph kernels $\mathbf K$ are diagonal matrices that can not be thought as the Fourier transform of a corresponding kernel defined in the spatial (vertex) domain. The diagonal elements $k_{i}$ can be thought as functions of the corresponding eigenvalues
$$
k_{i}= k(\lambda_i)
$$
thus providing an intuitive frequency interpretation of the kernel $\mathbf K$. In this way the convolution can be seen as a \textit{filtering} operation: for example, a kernel $k(\lambda_i) = \exp (-\lambda_i)$ will be the kernel of a low-pass filter since it will cut the high frequencies.

\subsection{The Equivariance error for graph convolutions}
Take a sampling scheme $V=\{v_i\in\mathbb S^2, i=0, ..., n\}$ of the sphere, a weighted undirected graph $G(V, E, \mathbf W)$, a signal $f: \mathbb S^2\to\mathbb R$ and its sampled representation $\mathbf f:\ f_i=f(v_i)$. Suppose that there exists a sampling operator $T_V: L^2(\mathbb S^2) \supset F\to \mathbb R^n,\  T_V(f) = \mathbf f$ defined on a suitable subspace $F$ of $L^2(\mathbb S^2)$ such that it is invertible, i.e., we can unambiguously reconstruct the function $f\in F$ from its sampled values $\mathbf f$. The existence of such subspace depends on the sampling scheme $V$, and its characterization is a common problem in signal processing \cite{Driscoll:1994:CFT:184069.184073}. Recall the definition (\ref{eq:rotation operator}) of the rotation operator $\Lambda(g), g\in SO(3)$.

We now want to understand how to set the edges and the weights of $G$ such that
\begin{equation}\label{eq:equivariance}
T \Lambda(g) T^{-1} \Omega_k T f = \Omega_k T \Lambda(g) f
\end{equation}
i.e., the graph convolution $\Omega_k$ and any rotation $\Lambda(g)$ commute.

Verifying equation (\ref{eq:equivariance}) is really hard in practice, due to the fact that for almost all samplings schemes $V$ it is not known if there exists a space $F$ in which $T$ is invertible. A special case is the \textit{equiangular sampling} scheme described in section \ref{sec:Chapter3: Heat Kernel Graph Laplacian on the Equiangular Sampling} where the sampling theorem \ref{theo:equiangular sampling theorem} holds \cite{Driscoll:1994:CFT:184069.184073}. With all the other sampling schemes, there are no sampling theorems available, but there are implementations of discrete SHT to reconstruct a sampled signal $\mathbf f$, thus providing a way to approximate $T^{-1}$. Thanks to this we are able, for a given sampling, a given function $f$, a given rotation $g$, and a given kernel $k$, to compute the \textit{normalized equivariance error}
\begin{equation}\label{eq:equivariance error}
E_{G}(f, g) = \left(\frac{ \norm {T \Lambda(g) T^{-1} \Omega_k Tf - \Omega_k T \Lambda(g) f}_{L^2(\mathbb R^2)}}{\norm {Tf}_{L^2(\mathbb R^2)}}\right)^2
\end{equation}
where $T^{-1}$ is substituted with a discrete SHT in case $T$ is not invertible.
A measure of how equivariant a graph is with respect to rotations will then be given by the \textit{mean equivariance error}
\begin{equation}\label{eq:mean equivariance error}
\overline E_G = \mathbb E_{f, g}\ 	E_G(f, g)
\end{equation}
In practice the expected value is obtained by averaging over a finite number of random functions and random rotations. The mean equivariance error $\overline E_G$ gives us an indication of how close the graph $G$ is from being equivariant to rotations. Now we state an intuitive concept that explains how to construct rotation invariant graphs, i.e. graphs such that $\overline{E_G}$ is small.
\begin{snugshade*}
	The mean equivariance error $\overline{E_G}$ will be small if the scalar product $\mathbf f^\intercal \mathbf v_{i(\ell, m)}$ well approximates $\hat {f}(\ell,m)$ i.e., the $L^2$ scalar product of the continuous signal \\
	$\hat {f}(\ell,m)= \int_{\eta \in \mathbb S^2}f(\eta)Y_\ell^m(\eta)d\mu(\eta)$.

	\textit{If $V$ is an equal area sampling scheme}, i.e. the area around each pixel $v_i$ is the same, $\overline{E_G}$ will be small if the graph Laplacian $\mathbf L$ is such that its eigenvectors $\mathbf v_i$ well approximate the eigenfunctions of the Laplace-Beltrami operator $\Delta_{\mathbb S^2}$ evaluated in the points of the sampling scheme, i.e.,
	$$
	\mathbf v_{i(\ell, m)} \approxeq Y_\ell^m(x_i)
	$$
\end{snugshade*}

In this way we framed the problem of constructing a rotation invariant graph with the more general problem of coming up with a matrix $\mathbf L$ with some specific spectral properties. Graphs are only one of many ways of coming up with such matrix $\mathbf L$, and many other methods have been already studied in the literature. In the next Chapter we present a brief overview of some of these methods providing the general context in which graph filtering can be framed.


The dirichlet module contains routines for using Beta, Gamma, and
Dirichlet densities. In particular, it implements mixture Dirichlet
priors, which are useful in a variety of parameter estimation tasks.

The Dirichlet PDF is:

\begin{equation}
 P(\vec{p}) = \frac{\Gamma{\sum_i \alpha_i}}{\prod_i \Gamma(\alpha_i)} \prod_i p_i^{\alpha_i-1}
\end{equation}

\subsection{The dirichlet API}

The module implements one object, \ccode{ESL\_MIXDCHLET}, which holds
a mixture Dirichlet prior.

The API consists of the following functions:

\vspace{1em}
\begin{tabular}{ll}\hline
   \multicolumn{2}{c}{\textbf{mixture Dirichlet priors}}\\
\ccode{esl\_mixdchlet\_Create()}            & Allocates a new mixture Dirichlet.\\
\ccode{esl\_mixdchlet\_Destroy()}           & Free's a mixture Dirichlet.\\
\ccode{esl\_mixdchlet\_MPParameters()}      & Mean posterior probability parameter estimation.\\
   \multicolumn{2}{c}{\textbf{Dirichlet, Gamma density routines}}\\
\ccode{esl\_dirichlet\_LogProbData()}       & $\log P( \mbox{count vector} \mid \mbox{mixture Dirichlet}$.\\
\ccode{esl\_dirichlet\_LogProbProbs()}      & $\log P( \mbox{prob vector} \mid \mbox{mixture Dirichlet}$.\\
\ccode{esl\_dirichlet\_LogGamma()}          & $\log \Gamma(x)$.\\
   \multicolumn{2}{c}{\textbf{Sampling code}}\\
\ccode{esl\_dirichlet\_\{D,F\}Sample()}            & Sample a probability vector from a Dirichlet.\\
\ccode{esl\_dirichlet\_\{D,F\}SampleUniform()}     & Sample a probability vector uniformly.\\
\ccode{esl\_dirichlet\_SampleBeta()}        & Sample from Beta$(\theta_1, \theta_2$).\\
   \multicolumn{2}{c}{\textbf{Input from file}}\\
\ccode{esl\_mixdchlet\_Read()}              & Input of a mixture Dirichlet from a file.\\ \hline
\end{tabular}

\subsection{Example of using the dirichlet API}

An example that reads a mixture Dirichlet prior from a file, samples a
probability vector from it, samples a count vector from the
probability vector, then infers which mixture component generated the
data and uses mean posterior estimation to reestimate the probability
vector from the counts:

\input{cexcerpts/dirichlet_example}

\subsection{Accessing information in the ESL\_MIXDCHLET object}

You may need to access the information in the \ccode{ESL\_MIXDCHLET}
object. The data fields inside this structure are:

\input{cexcerpts/dirichlet_mixdchlet}


\documentclass[../Notes/main.tex]{subfiles}

\begin{document}
\section{Dynamic Programming}

\subsection{Knapsack}
Assuming \(w>0\), \(i\geq0\) and \(w\geq w_i\)
\begin{align*}
    DP[w][i] & = \max(V_i+DP[w-w_i][i-1],DP[w][i-1])
\end{align*}
Base cases:
\begin{align*}
    DP[w][i] & =\begin{cases}
        DP[w][i] & \text{if }w<w_i                \\
        0        & \text{if }i<0\text{ or }w\leq0
    \end{cases}
\end{align*}

\subsection{Matrix Chain Multiplication}
Given a monoid \(M\) with operation \(\cdot\), a cost function \(c:M^2\rightarrow \mathbb{R}_{>0}\), and a finite sequence of elements of \(M\), \(a_1,\dots,a_n\), we can calculate the minimum cost of operating the elements (i.e. where to put the parenthesis on \(a_1\cdot\ldots\cdot a_n\) such that the total cost is minimized (the cost function is used when every operation is done)) with the following DP:
\begin{equation*}
    DP[i][j]=\min_{i\leq k<j}(c(A_{i,k},A_{k+1,j})+DP[i][k]+DP[k+1][j])
\end{equation*}
Where \(1\leq i\leq j\leq n\) and \(A_{i,j}=a_i\cdot\ldots\cdot a_j\)
%\lstinputlisting[firstline=3]{Matrix_Chain_Multiplication/MCM.cpp}

\subsection{Longest Increasing Subsequence}
Given a sequence \(a_1,\ldots,a_n\), we can find the LIS using the following method, using an auxiliary array \(L\) of size \(n\) and a counter \(currL\), set \(L[0]=a_1\) and \(currL=1\) then, in the sequence order, for each element \(a_i\) search for the lower bound \(L[j]\) of \(a_i\) between \(L[0]\) and \(L[currL-1]\), if \(j=currL-1\) then set \(L[currL]=a_i\) and increase \(curr_l\) by one, in the other case set \(L[j]=a_i\).
%\lstinputlisting[firstline=3]{LIS/LIS.cpp}
\end{document}


The first full-blown logic program that we see is, quite fittingly\footnote{Boolean logic is an interpretation of logic itself, which fits snugly within our own implementation of an interpretation of logic. Recursion can be quite fun, as we will see in one of the last chapters.}, a program for the definition and evaluation of boolean expressions.

\paragraph{Symbols}
The basic symbols that we need represents the boolean values of \texttt{TRUE} and \texttt{FALSE}, and they both have type \texttt{Value}:

\begin{lstlisting}
Data [] "TRUE" [] Priority 10000 Type Value
Data [] "FALSE" [] Priority 10000 Type Value
\end{lstlisting}

Boolean expressions, all of type \texttt{Expr}, are:
\begin{inparaenum}[\itshape i\upshape)]
\item traditional negation (the unary operator \texttt{!} following the tradition of C-like languages);
\item disjunction of boolean expressions (commonly known as ``or'');
\item conjunction of boolean expressions (commonly known as ``and'').
\end{inparaenum}

\begin{lstlisting}
Data [] "!" [Expr] Priority 30 Type Expr
Data [Expr] "|" [Expr] Priority 10 Type Expr
Data [Expr] "&" [Expr] Priority 20 Type Expr
\end{lstlisting}

Of course we want to be able to use values as expressions, otherwise writing \texttt{TRUE \& TRUE} would not be allowed. For this reason we specify that anything that has type \texttt{Value} can also be used where a type \texttt{Expr} is expected:

\begin{lstlisting}
Value is Expr
\end{lstlisting}

In order to compute the \texttt{Value} of an \texttt{Expr}, we define the \texttt{eval} function.

\begin{lstlisting}
Func [] "eval" [Expr] Priority 1 Type Expr => Value
\end{lstlisting}

\paragraph{Rules}
The only rules that we can define involve the \texttt{eval} function, which is the only function that we have defined. The first two rules  trivially specify that when we reach the evaluation of \texttt{TRUE} or \texttt{FALSE}, then we do not need to further proceed:

\begin{lstlisting}
------------------ (G0)
eval TRUE => TRUE

-------------------- (G1)
eval FALSE => FALSE
\end{lstlisting}

If we reach the evaluation of the negation of some expression \texttt{a}, then we will evaluate \texttt{a}. If the evaluation of \texttt{a} returns \texttt{TRUE}, then the evaluation of the negation of \texttt{a} returns \texttt{FALSE}:

\begin{lstlisting}
eval a => TRUE
----------------- (NEG0)
eval !a => FALSE
\end{lstlisting}

Similarly, if we reach the evaluation of the negation of some expression \texttt{a}, and evaluation of \texttt{a} returns \texttt{FALSE}, then the evaluation of the negation of \texttt{a} returns \texttt{TRUE}:

\begin{lstlisting}
eval a => FALSE
---------------- (NEG1)
eval !a => TRUE
\end{lstlisting}

When evaluating the disjunction of two expressions \texttt{a} and \texttt{b}, we try to evaluate \texttt{a}:
\begin{inparaenum}[\itshape i\upshape)]
\item if \texttt{a} evaluates to \texttt{TRUE}, then there is no need to further evaluate \texttt{b} and we can directly return \texttt{TRUE};
\item if \texttt{a} evaluates to \texttt{FALSE}, then we evaluate \texttt{b} and return whatever result of its evaluation.
\end{inparaenum}

\begin{lstlisting}
eval a => TRUE
------------------- (OR0)
eval (a|b) => TRUE

eval a => FALSE
eval b => y
----------------- (OR1)
eval (a|b) => y
\end{lstlisting}

When evaluating the conjunction of two expressions \texttt{a} and \texttt{b}, we try to evaluate \texttt{a}:
\begin{inparaenum}[\itshape i\upshape)]
\item if \texttt{a} evaluates to \texttt{FALSE}, then there is no need to further evaluate \texttt{b} and we can directly return \texttt{FALSE};
\item if \texttt{a} evaluates to \texttt{TRUE}, then we evaluate \texttt{b} and return whatever result of its evaluation.
\end{inparaenum}

\begin{lstlisting}
eval a => FALSE
-------------------- (AND0)
eval (a&b) => FALSE

eval a => TRUE
eval b => y
---------------- (AND1)
eval (a&b) => y
\end{lstlisting}

\paragraph{Example run}
Consider now the evaluation of an expression such as \texttt{eval (FALSE | !(TRUE \& FALSE))}. We begin with

\begin{lstlisting}
------------------------------------
eval (FALSE | !(TRUE & FALSE)) => ?
\end{lstlisting}

This expression is an instance of \texttt{eval (a|b)}, therefore we investigate the first premise according to rule \texttt{OR0} (notice the question marks at the end of some lines, which mean that we are verifying that this is indeed the case):

\begin{lstlisting}
eval FALSE => TRUE?
---------------------------------------
eval (FALSE | !(TRUE & FALSE)) => TRUE?
\end{lstlisting}

Since \texttt{eval FALSE} does never return \texttt{TRUE}, this branch of execution is interrupted. We try the alternate rule \texttt{OR1} for \texttt{eval (a|b)}, therefore we investigate two premises:

\begin{lstlisting}
eval FALSE => FALSE?
eval !(TRUE & FALSE) => y
------------------------------------
eval (FALSE | !(TRUE & FALSE)) => y
\end{lstlisting}

The first premise, \texttt{eval FALSE => FALSE}, is trivially verified by the rule \texttt{G1}. We may thus delete it from the tree of derivation:

\begin{lstlisting}
eval !(TRUE & FALSE) => y
------------------------------------
eval (FALSE | !(TRUE & FALSE)) => y
\end{lstlisting}

The second premise is itself complex, so we need to find its result and then assign it to \texttt{y}. We begin with rule \texttt{NEG1}\footnote{Note that Meta-Casanova would actually first try \texttt{NEG0}, which fails, but for reasons of space we skip that lengthy and fruitless derivation.}, which expects the input to evaluate to \texttt{FALSE}. In this case, since we know from \texttt{NEG1} that \texttt{eval !(TRUE \& FALSE)} would return \texttt{TRUE}, we substitute \texttt{y} with \texttt{TRUE} speculatively:

\begin{lstlisting}
eval (TRUE & FALSE) => FALSE?
-----------------------------
eval !(TRUE & FALSE) => TRUE
---------------------------------------
eval (FALSE | !(TRUE & FALSE)) => TRUE
\end{lstlisting}

We now need to evaluate premise \texttt{eval (TRUE \& FALSE)}, and we do so with application of rule \texttt{AND1}\footnote{Again, \texttt{AND1} is chosen ad-hoc to go directly to the result.}. Rule \texttt{AND1} evaluates the second term and returns its result:

\begin{lstlisting}
eval TRUE => TRUE?
eval FALSE => y
------------------------------
eval (TRUE & FALSE) => y
y == FALSE?
-----------------------------
eval !(TRUE & FALSE) => TRUE
---------------------------------------
eval (FALSE | !(TRUE & FALSE)) => TRUE
\end{lstlisting}

Fortunately, \texttt{eval TRUE => TRUE} is trivially verified by rule \texttt{G0}; we can safely remove it from our tree of derivation:

\begin{lstlisting}
eval FALSE => y
------------------------------
eval (TRUE & FALSE) => y
y == FALSE?
-----------------------------
eval !(TRUE & FALSE) => TRUE
---------------------------------------
eval (FALSE | !(TRUE & FALSE)) => TRUE
\end{lstlisting}

\texttt{eval FALSE} returns \texttt{FALSE} as an immediate consequence of rule \texttt{G1}. Therefore, we can replace \texttt{y} with \texttt{FALSE}:

\begin{lstlisting}
eval FALSE => FALSE
------------------------------
eval (TRUE & FALSE) => FALSE
FALSE == FALSE?
-----------------------------
eval !(TRUE & FALSE) => TRUE
---------------------------------------
eval (FALSE | !(TRUE & FALSE)) => TRUE
\end{lstlisting}

The two upper premises are now completely evaluated, so we can safely remove them from the derivation tree:

\begin{lstlisting}
FALSE == FALSE?
-----------------------------
eval !(TRUE & FALSE) => TRUE
---------------------------------------
eval (FALSE | !(TRUE & FALSE)) => TRUE
\end{lstlisting}

We now need to verify that the result of the last evaluation of \texttt{AND1} is compatible with the evaluation of \texttt{NEG1}, that is \texttt{FALSE == FALSE}. This is trivially the case, and therefore we can discharg the last two premises:

\begin{lstlisting}
eval (FALSE | !(TRUE & FALSE)) => TRUE
\end{lstlisting}

Since we have no more premises, we can safely conclude that indeed the result of the evaluation of the original proposition \texttt{eval (FALSE | !(TRUE \& FALSE))} yields \texttt{TRUE}, which is also what we would expect from intuition.


\section{Directional Derivatives}\label{sec:DirectionalDerivatives}

We still have not answered one of our first questions about the
steepness of a surface: starting at a
point on a surface given by $f(x,y)$, and walking in a particular
direction, how steep is the surface? We are now ready to answer the
question.

We already know roughly what has to be done: as shown in
Figure~\ref{fig:parabolic bowl cross-section}, we extend a line in
the $x$-$y$ plane to a vertical plane, and we then compute the slope
of the curve that is the cross-section of the surface in that
plane. The major stumbling block is that what appears in this plane to
be the horizontal axis, namely the line in the $x$-$y$ plane, is not
an actual axis---we know nothing about the ``units'' along the
axis. Our goal is to make this line into a $t$ axis; then we need
formulas to write $x$ and $y$ in terms of this new variable $t$; then
we can write $z$ in terms of $t$ since we know $z$ in terms of $x$ and
$y$; and finally we can simply take the derivative.

So we need to somehow ``mark off'' units on the line, and we need a
convenient way to refer to the line in calculations. It turns out that
we can accomplish both by using the vector form of a line. Suppose
that $\vect{u}$ is a unit vector $\langle u_1,u_2\rangle$ in the
direction of interest. A vector equation for the line through
$(x_0,y_0)$ in this direction is $\vect{v}(t)=\langle
u_1t+x_0,u_2t+y_0\rangle$. The height of the surface above the point
$(u_1t+x_0,u_2t+y_0)$ is $g(t)=f(u_1t+x_0,u_2t+y_0)$. Because $\vect{u}$
is a unit vector, the value of $t$ is precisely the distance along the
line from $(x_0,y_0)$ to $(u_1t+x_0,u_2t+y_0)$; this means that the
line is effectively a $t$ axis, with origin at the point $(x_0,y_0)$,
so the slope we seek is
\begin{align*}
g'(0)&=\langle f_x(x_0,y_0),f_y(x_0,y_0)\rangle\cdot
\langle u_1,u_2\rangle	\\
&=\langle f_x,f_y\rangle\cdot\vect{u}	\\
&=\nabla f\cdot \vect{u}	\\
\end{align*}
Here we have used the chain rule and the derivatives
${d\over dt}(u_1t+x_0)=u_1$ and ${d\over dt}(u_2t+y_0)=u_2$.
The vector $\langle f_x,f_y\rangle$ is very useful, so it has its own
symbol, $\nabla f$, pronounced ``del f''\index{del ($\nabla$)};
it is also called the
\dfont{gradient} of $f$.\index{gradient}

\begin{example}{Slope}{SlopeEllipticParaboloid}
Find the slope of $z=x^2+y^2$ at $(1,2)$ in the direction of the
vector $\langle 3,4\rangle$.
\end{example}
\begin{solution}
We first compute the gradient at $(1,2)$:
$\nabla f=\langle 2x,2y\rangle$, which is $\langle 2,4\rangle$ at
$(1,2)$. A unit vector in the desired direction is $\langle
3/5,4/5\rangle$, and the desired slope is then
$\langle 2,4\rangle\cdot\langle 3/5,4/5\rangle=6/5+16/5=22/5$.
\end{solution}

\begin{example}{Tangent Vector}{TangentVectorEllipticParaboloid}
Find a tangent vector to $z=x^2+y^2$ at $(1,2)$ in the direction of the
vector $\langle 3,4\rangle$ and show that it is parallel to the
tangent plane at that point.
\end{example}
\begin{solution}
Since $\langle 3/5,4/5\rangle$ is a unit vector in the desired
direction, we can easily expand it to a tangent vector simply by
adding the third coordinate computed in the previous example: $\langle
3/5,4/5,22/5\rangle$.  To see that this vector is parallel to the
tangent plane, we can compute its dot product with a normal to the
plane. We know that a normal to the tangent plane is
$$\langle f_x(1,2),f_y(1,2),-1\rangle = \langle 2,4,-1\rangle,$$
and the dot product is $\langle 2,4,-1\rangle\cdot\langle
3/5,4/5,22/5\rangle=6/5+16/5-22/5=0$, so the two vectors are
perpendicular. (Note that the vector normal to the surface, namely
$\langle f_x,f_y,-1\rangle$, is simply the gradient with a $-1$ tacked
on as the third component.)
\end{solution}

The slope of a surface given by $z=f(x,y)$ in the direction of a
(two-dimensional) vector $\vect{u}$ is called the \dfont{directional
  derivative}\index{directional derivative} of $f$, written $D_{\vect{u}}f$.
The directional derivative immediately provides us with some
additional information. We know that
$$D_{\vect{u}}f=\nabla f\cdot \vect{u}=|\nabla f||\vect{u}|\cos\theta=
|\nabla f|\cos\theta$$
if $\vect{u}$ is a unit vector; $\theta$ is the angle between $\nabla f$
and $\vect{u}$. This tells us immediately that the largest value of
$D_{\vect{u}}f$ occurs when $\cos\theta=1$, namely, when $\theta=0$, so
$\nabla f$ is parallel to $\vect{u}$. In other words, the gradient
$\nabla f$ points in the direction of steepest ascent of the surface, and
$|\nabla f|$ is the slope in that direction. Likewise, the smallest value of
$D_{\vect{u}}f$ occurs when $\cos\theta=-1$, namely, when $\theta=\pi$, so
$\nabla f$ is anti-parallel to $\vect{u}$. In other words,
$-\nabla f$ points in the direction of steepest descent of the surface, and
$-|\nabla f|$ is the slope in that direction.

\begin{example}{Direction of Steepest Ascent and Descent}{AscentDescentEllipticParaboloid}
Investigate the direction of steepest ascent and descent for
$z=x^2+y^2$.
\end{example}
\begin{solution}
The gradient is $\langle 2x,2y\rangle=2\langle x,y\rangle$; this is a
vector parallel to the vector $\langle x,y\rangle$, so the direction
of steepest ascent is directly away from the origin, starting at the
point $(x,y)$. The direction of steepest descent is thus directly
toward the origin from $(x,y)$. Note that at $(0,0)$ the gradient
vector is $\langle 0,0\rangle$, which has no direction, and it is
clear from the plot of this surface that there is a minimum point at
the origin, and tangent vectors in all directions are parallel to the
$x$-$y$ plane.
\end{solution}

If $\nabla f$ is perpendicular to $\vect{u}$,
$D_{\vect{u}}f=|\nabla f|\cos(\pi/2)=0$, since $\cos(\pi/2)=0$.
This means that in either of the two directions perpendicular to
$\nabla f$, the slope of the surface is 0; this implies that a vector in
either of these directions is tangent to the level curve at that
point. Starting with $\nabla f=\langle f_x,f_y\rangle$, it is easy to
find a vector perpendicular to it: either $\langle f_y,-f_x\rangle$ or
$\langle -f_y,f_x\rangle$ will work.

If $f(x,y,z)$ is a function of three variables, all the calculations
proceed in essentially the same way. The rate at which $f$ changes in
a particular direction is $\nabla f\cdot\vect{u}$, where now
$\nabla f=\langle f_x,f_y,f_z\rangle$ and $\vect{u}=\langle
u_1,u_2,u_3\rangle$ is a unit vector. Again $\nabla f$ points in the
direction of maximum rate of increase, $-\nabla f$ points in the
direction of maximum rate of decrease, and any vector perpendicular to
$\nabla f$ is tangent to the level surface $f(x,y,z)=k$ at the point
in question. Of course there are no longer just two such vectors; the
vectors perpendicular to $\nabla f$ describe the tangent plane to the
level surface, or in other words $\nabla f$ is a normal to the tangent
plane.

\begin{example}{Gradient}{GradientTemperature}
Suppose the temperature at a point in space is given by
$T(x,y,z)=T_0/(1+x^2+y^2+z^2)$; at the origin the temperature in
Kelvin is
$T_0>0$, and it decreases in every direction from there. It might be,
for example, that there is a source of heat at the origin, and as we
get farther from the source, the temperature decreases.
The gradient is
\begin{align*}
\nabla T&=\langle {-2T_0x\over (1+x^2+y^2+z^2)^2}+
{-2T_0x\over (1+x^2+y^2+z^2)^2}+{-2T_0x\over (1+x^2+y^2+z^2)^2}\rangle	\\
&={-2T_0\over (1+x^2+y^2+z^2)^2}\langle x,y,z\rangle.	\\
\end{align*}
The gradient points directly at the origin from the point
$(x,y,z)$---by moving directly toward the heat source, we increase the
temperature as quickly as possible.
\end{example}

\begin{example}{Tangent Plane}{TangentPlaneParallel}
Find the points on the surface defined by $x^2+2y^2+3z^2=1$
where the tangent plane  is parallel to the plane defined by
$3x-y+3z=1$.
\end{example}
\begin{solution}
Two planes are parallel if their normals are parallel or
anti-parallel, so we want to find the points on the surface with
normal parallel or anti-parallel to $\langle 3,-1,3\rangle$.
Let $f=x^2+2y^2+3z^2$; the gradient of $f$ is normal to the level
surface at every point, so we are looking for a gradient parallel
or anti-parallel to
$\langle 3,-1,3\rangle$. The gradient is $\langle 2x,4y,6z\rangle$; if
it is parallel or anti-parallel to $\langle 3,-1,3\rangle$, then
$$\langle 2x,4y,6z\rangle=k\langle 3,-1,3\rangle$$
for some $k$. This means we need a solution to the equations
$$2x=3k\qquad 4y=-k\qquad 6z=3k$$
but this is three equations in four unknowns---we need another
equation. What we haven't used so far is that the points we seek are
on the surface $x^2+2y^2+3z^2=1$; this is the fourth equation. If we
solve the first three equations for $x$, $y$, and $z$ and substitute
into the fourth equation we get
\begin{align*}
1&=\left({3k\over2}\right)^2+2\left({-k\over4}\right)^2+3\left({3k\over6}\right)^2	\\
&=\left({9\over4}+{2\over16}+{3\over4}\right)k^2	\\
&={25\over8}k^2	\\
\end{align*}
so $\ds k=\pm{2\sqrt2\over 5}$. The desired points are
$\ds\left({3\sqrt2\over5},-{\sqrt2\over10},{\sqrt2\over 5}\right)$
and $\ds\left(-{3\sqrt2\over5},{\sqrt2\over10},-{\sqrt2\over
  5}\right)$.
\end{solution}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\Opensolutionfile{solutions}[ex]
\section*{Exercises for \ref{sec:DirectionalDerivatives}}

\begin{enumialphparenastyle}

\begin{ex}
Find $D_{\vect{u}} f$ for
$\ds f=x^2+xy+y^2$ in the direction of $\vect{u}=\langle 2,1\rangle$
at the point $(1,1)$.
\begin{sol}
$9\sqrt5/5$
\end{sol}
\end{ex}

\begin{ex}
Find $D_{\vect{u}} f$ for
$\ds f=\sin(xy)$ in the direction of $\vect{u}=\langle -1,1\rangle$
at the point $(3,1)$.
\begin{sol}
$\sqrt2\cos3$
\end{sol}
\end{ex}

\begin{ex}
Find $D_{\vect{u}} f$ for
$\ds f=e^x\cos(y)$ in the direction 30 degrees from the positive $x$ axis
at the point $(1,\pi/4)$.
\begin{sol}
$e\sqrt2(\sqrt3-1)/4$
\end{sol}
\end{ex}

\begin{ex}
The temperature of a thin plate in the $x$-$y$ plane is
$\ds T=x^2+y^2$. How fast does temperature change at the point $(1,5)$
moving in a direction 30 degrees from the positive $x$ axis?
\begin{sol}
$\sqrt3+5$
\end{sol}
\end{ex}

\begin{ex}
Suppose the density of a thin plate at $(x,y)$ is
$\ds 1/\sqrt{x^2+y^2+1}$. Find the rate of change of the density at
$(2,1)$ in a direction $\pi/3$ radians from the positive $x$ axis.
\begin{sol}
$-\sqrt6(2+\sqrt3)/72$
\end{sol}
\end{ex}

\begin{ex}
Suppose the electric potential at $(x,y)$ is
$\ds\ln\sqrt{x^2+y^2}$. Find the rate of change of the potential at
$(3,4)$ toward the origin and also in a direction at a right angle to
the direction toward the origin.
\begin{sol}
$-1/5$, $0$
\end{sol}
\end{ex}

\begin{ex}
A plane perpendicular to the $x$-$y$ plane contains the
point $(2,1,8)$ on the paraboloid $z=x^2+4y^2$. The cross-section of
the paraboloid created by this plane has slope 0 at this point. Find
an equation of the plane.
\begin{sol}
$4(x-2)+8(y-1)=0$
\end{sol}
\end{ex}

\begin{ex}
A plane perpendicular to the $x$-$y$ plane contains the
point $(3,2,2)$ on the paraboloid $36z=4x^2+9y^2$. The cross-section
of the paraboloid created by this plane has slope 0 at this point.
Find an equation of the plane.
\begin{sol}
$2(x-3)+3(y-2)=0$
\end{sol}
\end{ex}

\begin{ex}
Suppose the temperature at $(x,y,z)$ is given by
$\ds T=xy+\sin(yz)$. In what direction should you go from the point
$(1,1,1)$ to decrease the temperature as quickly as possible? What is
the rate of change of temperature in this direction?
\begin{sol}
$\langle -1,-1-\cos1,-\cos1\rangle$, $-\sqrt{2+2\cos1+2\cos^21}$
\end{sol}
\end{ex}

\begin{ex}
Suppose the temperature at $(x,y,z)$ is given by
$\ds T=xyz$. In what direction can you go from the point
$(1,1,1)$ to maintain the same temperature?
\begin{sol}
Any direction perpendicular to $\nabla T=\langle
1,1,1\rangle$,
for example, $\langle -1,1,0\rangle$
\end{sol}
\end{ex}

\begin{ex}
Find an equation for the plane tangent to
$\ds x^2-3y^2+z^2=7$ at $(1,1,3)$.
\begin{sol}
$2(x-1)-6(y-1)+6(z-3)=0$
\end{sol}
\end{ex}

\begin{ex}
Find an equation for the plane tangent to
$\ds xyz=6$ at $(1,2,3)$.
\begin{sol}
$6(x-1)+3(y-2)+2(z-3)=0$
\end{sol}
\end{ex}

\begin{ex}
Find an equation for the line normal to
$\ds x^2+2y^2+4z^2=26 $ at $(2,-3,-1)$.
\begin{sol}
$\langle 2+4t,-3-12t,-1-8t\rangle$
\end{sol}
\end{ex}

\begin{ex}
Find an equation for the line normal to
$\ds x^2+y^2+9z^2=56$ at $(4,2,-2)$.
\begin{sol}
$\langle 4+8t,2+4t,-2-36t\rangle$
\end{sol}
\end{ex}

\begin{ex}
Find an equation for the line normal to
$\ds x^2+5y^2-z^2=0$ at $(4,2,6)$.
\begin{sol}
$\langle 4+8t,2+20t,6-12t\rangle$
\end{sol}
\end{ex}

\begin{ex}
Find the directions in which the directional derivative of
$f(x,y)=x^2+\sin(xy)$ at the point $(1,0)$ has the value 1.
\begin{sol}
$\langle 0,1\rangle$, $\langle 4/5,-3/5\rangle$
\end{sol}
\end{ex}

\begin{ex}
Show that the curve
$\vect{r}(t) = \langle\ln(t),t\ln(t),t\rangle$
is tangent to the surface $xz^2-yz+\cos(xy) = 1$ at the point
$(0,0,1)$.
\end{ex}

\begin{ex}
A bug is crawling on the surface of a hot plate, the
temperature of which at the point $x$ units to the right of the lower
left corner and $y$ units up from the lower left corner is given by
$T(x,y)=100-x^2-3y^3$.

\begin{enumerate}
	\item If the bug is at the point $(2,1)$, in what direction should it
	  move to cool off the fastest?  How fast will the temperature drop in
	  this direction?
	\item If the bug is at the point $(1,3)$, in what direction should it
	  move in order to maintain its temperature?
\end{enumerate}
\begin{sol}
\begin{enumerate}
	\item	$\langle 4,9\rangle$
	\item	$\langle -81,2\rangle$ or $\langle 81,-2\rangle$
\end{enumerate}
\end{sol}
\end{ex}

\begin{ex}
The elevation on a portion of a hill is given by $f(x,y) =
100 -4x^2 - 2y$.  From the location above $(2,1)$, in which direction will
water run?
\begin{sol}
in the direction of $\langle 8,1\rangle$
\end{sol}
\end{ex}

\begin{ex}
Suppose that $g(x,y)=y-x^2$.  Find the gradient at the point
$(-1, 3)$.  Sketch the level curve to the graph of $g$ when
$g(x,y)=2$, and plot both the tangent line and the gradient vector at
the point $(-1,3)$. (Make your sketch large).  What do you notice,
geometrically?
\begin{sol}
$\ds \nabla g(-1,3)=\langle 2,1\rangle$
\end{sol}
\end{ex}

\begin{ex}
The gradient $\nabla f$ is a vector
valued function of two variables.  Prove the following gradient rules.
Assume $f(x,y)$ and $g(x,y)$ are differentiable functions.

\begin{enumerate}
	\item $\nabla(fg)=f\nabla(g)+g\nabla(f)$
	\item $\nabla(f/g)=(g\nabla f - f \nabla g)/g^2$
	\item $\nabla((f(x,y))^n)=nf(x,y)^{n-1}\nabla f$
\end{enumerate}
\end{ex}

\end{enumialphparenastyle}


\subsection{Dirac delta}

\subsubsection{Kronecker delta}

The function is: \(\delta_{ij} \)

If \(i=j\) this is \(1\). Otherwise it is \(0\).

We introduced this in linear algebra.

\subsubsection{Dirac delta}

The Dirac delta replaces the Kronecker delta for continuous functions.

That is, we want:

\begin{itemize}
\item \(\delta (x\ne 0)=0\)
\item \(\delta (0)=+\infty\)
\item \(\int_{-\infty }^{\infty}\delta (x)dx=1\)
\end{itemize}


\subsection{Basic Properties}
All of these properties should look familiar from properties of limits and continuity.
Let $f$ and $g$ be differentiable functions of $x$.
Let $c$ be some real constant.
\begin{align*}
	\textbf{Sum and Difference Rule: }& (f \pm g)^\prime = f^\prime \pm g^\prime \\
	\textbf{Constant Multiple Rule: }& (cf)^\prime = cf^\prime \\
	\textbf{Constant Rule: }& (c)^\prime = 0
\end{align*}
\footnotetext{This follows from using the sum and difference rule with the power rule.}


\section{CAPM}

For the market porfolio to be optional the RRR of all risky assets must be the same

$RRR_i = \frac{\bar{r}_i-r_F}{\sigma_{iM}/ \sigma_M} = SR_M = \frac{\bar{r}_M-r_F}{\sigma_M}$

then

$ \bar{r}_i-r_F = \frac{\sigma_{iM}}{\sigma_M^2} (\bar{r}_M-r_F)  = \beta_{iM} (\bar{r}_M-r_F) $

$\beta_{iM}$ is a measure of asset it's systematic risk: exposure to the market.
$\bar{r}_M-r_F$ gives the premium per unit of systematic risk.

\subsection*{Risk and return in CAPM}
We can decompose an asset’s return into three pieces:

$ \tilde{r}_i-r_F = \alpha_i + \beta_{iM} (\tilde{r}_M-r_F) + \tilde{\epsilon}_i$

$E[\tilde{\epsilon}_i]=0$, $Cov[\tilde{r}_M-r_F) , \tilde{\epsilon}_i]=0$

Three characteristics of an asset: Alpha, according to CAPM, alpha should be zero for all assets. Beta: measures an asset’s systematic risk. SD[$\tilde{\epsilon}_i$] measures non-systematic risk.

\subsection*{Leverage: equity beta vs asset betas}

The assets of the firm serve to pay all investors, and so:
$A=E+D$

$\beta_A = \frac{E}{E+D} \beta_E + \frac{D}{E+D} \beta_D$

\subsection*{R15Q1}
inputs: $E[r_M]=14\%$, $E[r_P]=16\%$, $r_f=6\%$, $\sigma_M=25\%$

$E[r_P] = r_F + \beta_P (E[r_M]-r_f) = 16\% = 6\% + \beta_P (14\% - 6\%) =1.25$

to make a portfolio to be located in the capital line, one must use the risk-free asset with weight $w$:
 $E[r_P]=w r_f + (1-w) E[r_M] = 6\%w+14\%(1-w) \implies w=\frac{E[r_P]-r_M}{r_F-r_M}=-0.25$

$VaR[r_P] = Var[w r_f + (1-w) E[r_M]] = (1-w)^2 Var[r_M] \implies \sigma_P = 31.25\% $

to find correlation use:
$\beta_P = \frac{Cov(r_P, r_M)}{Var(r_M)}=\frac{\rho_{P,M} \sigma_M \sigma_P }{\sigma_M^2} \implies \rho_{P,M}=\frac{\beta_P \sigma_M}{\sigma_P} = 1$

\subsection*{R15Q5}
Empirically estimating CAPM.  $r_i-r_f = \alpha + \beta_{MKT}^i (r_{MKT}-r_f) + \epsilon_i$  In Excel : \texttt{LINEST($r_i-r_f$, $r_{MKT}-r_f$,1,0)} this yields $\beta_{MKT}$ and $\alpha$ in that order.


\section{Examples of CW-complexes}
\subsection{Bringing you up-to-speed on CW-complexes}
\begin{definition}
    A \emph{relative CW-complex} is a pair $(X,A)$, together with a filtration
    $$A=X_{-1}\subseteq X_0\subseteq X_1\subseteq\cdots\subseteq X,$$
    such that for all $n$, the space $X_n$ sits in a pushout square:
    $$
    \xymatrix{
	\coprod_{\alpha\in \Sigma_n}S^{n-1}\ar[r]\ar[d]_{\text{attaching maps}} & \coprod_{\alpha\in \Sigma_n}D^n\ar[d]^{\text{characteristic maps}}\\
	X_{n-1}\ar[r] & X_n,
    }
    $$
    and $X=\varinjlim X_n$.
\end{definition}
If $A=\emptyset$, this is just the definition of a CW-complex.
In this case, $X$ is also compactly generated.
(This is one of the reasons for defining compactly generated spaces.)
Often, $X$ will be a CW-complex, and $A$ will be a subcomplex.
If $A$ is Hausdorff, then so is $X$.

If $X$ and $Y$ are both CW-complexes, define
$$(X\times^k Y)_n = \bigcup_{i+j = n}X_i\times Y_j;$$
this gives a CW-structure on the product $X\times^k Y$.
Any closed smooth manifold admits a CW-structure.

\begin{example}[Complex projective space]
    The complex projective $n$-space $\CP^n$ is a CW-complex, with skeleta $\CP^0\subseteq\CP^1\subseteq\cdots\subseteq \CP^n$.
    Indeed, any complex line through the origin meets the hemisphere defined by
    $\begin{pmatrix}z_0\\\vdots\\z_n\end{pmatrix}$ with $||z||=1$, $\Im(z_n) = 0$, and $\Re(z_n)\geq 0$.
	Such a line meets this hemisphere (which is just $D^{2n}$) at one point --- unless it's on the equator;
	this gives the desired pushout diagram:
    \begin{equation*}
	\xymatrix{
	    S^{2n-1}\ar[r]\ar[d] & D^{2n}\ar[d]\\
	    \CP^{n-1}\ar[r] & \CP^n.
	    }
    \end{equation*}
\end{example}
\begin{example}[Grassmannians]
    Let $V=\RR^n$ or $\cC^n$ or $\mathbf{H}^n$, for some fixed $n$.
    Define the Grassmannian $\Gra_k(\RR^n)$ to be the collection of $k$-dimensional subspaces of $V$.
    This is equivalent to specifying a $k\times n$ rank $k$ matrix.

    %just linalg here, nothing to see
    %The span of rows of $A$ is the row space $V_A$. This is the span of the rows of the reduced reduced echelon form of $A$. An entry is a \emph{pivot} if its leftmost nonzero is in its row. A column is \emph{pivotal} if it contains a pivot. Any matrix is reduced echelon if the $i$th pivotal column is $e_i$.

    For instance, $\Gra_2(\RR^4)$ is, as a set, the disjoint union of:
    \begin{equation*}
	\begin{pmatrix}& 1 & \\ & & 1\end{pmatrix},\begin{pmatrix}&1&\ast\\&&1\end{pmatrix},\begin{pmatrix}1&\ast&\ast\\&&1\end{pmatrix},\begin{pmatrix}&1&\ast\\&1&\ast\end{pmatrix},\begin{pmatrix}1&\ast&\ast\\&1&\ast\end{pmatrix},\begin{pmatrix}1&\ast&\ast\\1&\ast&\ast\end{pmatrix}.
    \end{equation*}
    Motivated by this, define:
    \begin{definition}
	The $j$-skeleton of $\Gra(V)$ is
	$$\mathrm{sk}_j\Gra_k(V) = \{A:\text{row echelon representation with at most $j$ free entries}\}.$$
    \end{definition}
    For a proof that this is indeed a CW-structure, see \cite[\S 6]{milnorstasheff}.
    %They don't know it in 18.06, but they're constructing a CW-structure for the Grassmannian.
\end{example}
The top-dimensional cell tells us that
$$\dim\Gra_k(\RR^n) = k(n-k).$$
The complex Grassmannian has cells in only even dimensions.
We know the homology of Grassmannians: Poincar\'e duality is visible if we count the number of cells.
(Consider, for instance, in $\Gra_2(\RR^4)$).


\section{Method of Steepest Descent}
This is a method of approximating complex line integrals.
Let us first consider the simpler case of a real integral.
We look for an approximation to the Gamma function
$\Gamma(x + 1)$
when $x$ is large,
starting from its definition as an integral
\begin{align}
    \Gamma(x + 1)
    &=
    \int_{0}^{\infty} t^x e^{-t}\, dt
\end{align}
We want to get an approximate integral in the limit that $x$ is very large.
OK so how do we begin?

The first thing is note that the integrand has a maximum when
$t=x$.
To see this, note the integrand is extreme when
\begin{align}
    \frac{d}{dt}\left(
        t^x e^{-t}
    \right) = 0
\end{align}
at $t=t_0$.
Solving $t_0=x$,
for large $x$,
the integrand is very highly peaked around $t=t_0$.

So to a good approximation,
the value of the integral is dominated by they region around the peak.
For large $x$,
the contribution of the integral is dominated by the region around the peak.
For a first pass,
we can just consider the region around the peak and discard everything else.
That's the approximation we make.
It's similar to what we do when we look at the method of steepest descent.

The first thing to do is to write the integrand in an exponential form,
like $e^{f(t)}$
and expand $f(t)$ in a Taylor series
about the maximum.

For example,
\begin{align}
    e^{f(t)} &=
    t^{x} e^{-t}
    = e^{x \log t - t}
\end{align}
Then
\begin{align}
    f(t) &= x\log t - t\\
    f'(t) &= \frac{x}{t} - 1\\
    f''(t) &= -\frac{x}{t^2}
\end{align}
Then,
expanding about $t=x$,
we have
\begin{align}
    f(t) &=
    (x \log x - x)
    - \frac{1}{2}
    \frac{1}{x} {(t - x)}^2 + \cdots
\end{align}
Now we approximate the integral by contribution from the region around the
maximum,
\begin{align}
    \Gamma(x + 1) &\approx
    \int_{0}^{\infty} dt\,
    \exp\left\{
    x\log x - x \frac{1}{2x}{(t - x)}^2
    \right\}\\
    &\approx
    e^{x\log x - x}\int_{-\infty}^{\infty}\exp\left\{
    -\frac{1}{2x}{(t - x)}^2
    \right\}
\end{align}
The error in changing the integral lower bound from 0 to $\infty$
is negligible
and it's a Gaussian integral so you get
\begin{align}
    \Gamma(x + 1) \approx
    e^{x \log x - x}\sqrt{2\pi x} = \sqrt{2\pi x} x^x e^{-x}
\end{align}
This is the first term in Stirling's formula.

The method of steepest descent is generally applied to integrals of the form
\begin{align}
    I(s) &=
    \int_C g(z) e^{s f(z)} dz
\end{align}
where $s$ is real, large and positive
and $f(z)$ and $g(z)$ are analytic functions.
This restriction on $s$ being real is not important because if it were
complex,
you can pull out the imaginary part and sign
and absorb it into $g$ and $f$.
Assume you've already done that.

For $f(z)=u& + iv$
we expect most of the contribution to the integral to come from the region
where $u$ is large.
But by Cauchy's theorem,
we have to freedom to move the contour,
so the idea is the move the contour to pass through where the value is
peaked.

The idea behind the method of steepest descent it to choose the contour $C$
so that it runs over a region where $u$ is large and highly peaked
so that $I(s)$ is dominated by the contribution from this region.

Now we come to the complication.
Remember, you had to prove that neither $u$ nor $v$
can have a maximum or a minimum in that region.
All you have are saddle points.
But actually this turns out not to matter.
The question is not what is the global maximum of this function.
All we have to do is choose a path such that it has a maximum
\emph{along} that path.
When going along a path,
you approach a point from only one direction.
So choose your path such that the path has a maximum along the path,
and the integral is dominated by the contribution from that peak.
On the real line,
there's only one path of integral,
either you have a peak or you don't.
In the complex plane,
there is no maximum,
but you have the flexibility to choose the path,
so choose the path so that along that path you have a maximum,
and the integral will be dominated by the region around that maximum
on the path.

So let's write that down.

However, because $f(z)$ is an analytic function,
neighbour $u$ nor $v$ can have a maximum or a minimum

Any extrema of $u(x, y)$ where
\begin{align}
    \frac{\partial u}{\partial x}
    &=
    \frac{\partial u}{\partial y} = 0
\end{align}
must be a saddle point,
where the surface looks like a saddle or a mountain pass.
At any such point,
we also have
\begin{align}
    \frac{\partial v}{\partial x}
    =
    \frac{\partial v}{\partial y} = 0
\end{align}
because of the Cauchy-Riemann conditions.

At the saddle point, we have
$f'(z) = 0$.
Then we can approximate $f(z)$ near the saddle point,
\begin{align}
    f(z) &=
    f(z_0)
    + \frac{1}{2} f''(z_0){(z - z_0)}^2.
\end{align}
Here $z=z_0$ is the saddle point.

\begin{align}
    f''(z_0) &= |f''(z_0)| e^{i\theta}\\
    z - z_0 &= |z - z_0| e^{i\alpha}
\end{align}
and so approximate
\begin{align}
    u(x, y) &\approx
    u(x_0, y_0) + \frac{1}{2}|f''(z_0)|
    |z - z_0|^2 \cos(\theta + 2\alpha)\\
    v(x, y) &\approx
    v(x_0, y_0) + \frac{1}{2}|f''(z_0)|
    |z - z_0|^2 \sin(\theta + 2\alpha)\\
\end{align}
So this $\alpha$ controls the direction that you're approaching the saddle
point.
You can choose $\alpha$ such that as you leave the saddle point,
$u$ is increasing,
or you can choose $\alpha$ such that as you leave the saddle point
$u$ is decreasing.
I think that's clear.

On the surface $u(x, y)$,
the part of paths of steepest decent from the saddle point
into the valleys are those for which $\cos(\theta + 2\alpha)=-1$,
so that
\begin{align}
    \alpha &= - \frac{\theta}{2} \pm \frac{\pi}{2}
\end{align}
We can pick a path that rises very fast and falls very fast.
Why do we approach along this direction?
We want the path of steepest descent.

Along these directions,
$\sin(\theta + 2\alpha)=0$
so $v(x, y)$ is constant.
Then the oscillating factor $e^{isv}$
in the integrand will not produce large cancellations
along that line.
This is pretty important.
This $e^{isv}$ is a phase factor,
and this phase can be positive or negative,
so it doesn't help if $u$ is huge and this phase factor is
oscillating like crazy.

It's very important that  there are no large oscillations in this phase as we
integrate along this saddle point.
Everything is working out just the way we want.
$e^{u}$ is large about that specific point,
and there are no large cancellations coming from that phase.

In contrast,
the paths for which
$\cos(\theta + 2\alpha)=1$
are the paths of steepest ascent.
Those are the directions along which $u$ is growing as opposed to falling
most quickly as you leave the saddle point.
Along those directions,
it's like you are going along a mountain ridge.
These paths are at right angles with respect to path of steepest descent.

We now deform the contour of integration so as to pass over the saddle point
along the path of steepest descent.
Since the saddle point is a maximum of $u(x, y)$
\emph{along this path},
for large $s$ we expect that the contribution to the integral will be dominated
by the region around the saddle point.

Writing $z = z_0 + l e^{i\alpha}$,
\begin{align}
    I(s) &= \int_{C'}
    g\left( z_0 + l e^{i\alpha} \right)
    \exp\left[
        s f(z_0 + l e^{i\alpha})
    \right]
    e^{i\alpha}\, dl
\end{align}
So now we approximate by Taylor expanding $f$ and $g$ around the point.
\begin{align}
    f\left( z_0 + l e^{i\alpha} \right) &\approx
    f(z_0) + \frac{1}{2}f''\left( z_0 \right) l^2 e^{2i\alpha}
    + \cdots\\
    g\left( z_0 + l e^{i\alpha} \right) &\approx
    g(z_0) + \frac{1}{2}g''\left( z_0 \right) l e^{i\alpha}
    + \cdots\\
\end{align}
Assuming $g(z)$ is slowly varying,
we can approximate
\begin{align}
    I(s) &\approx
    g\left( z_0 \right)
    e^{s f\left(z_0 \right)}
    e^{i\alpha}
    \int_{-\infty}^{\infty}
    e^{-\frac{1}{2}\left|f''\left( z_0 \right)\right| sl^2}
    \,dl
\end{align}
Now this is a Gaussian integral,
and we changed the limits to $(-\infty, \infty)$,
but it's not going to change the answer significantly.
And when the dust settles
\begin{align}
    I(s) &\approx
    \frac{%
        \sqrt{2\pi} g(z_0) e^{sf(z_0)} e^{i\alpha}
    }{%
        \sqrt{s |f''(z_0)|}
    }
\end{align}
This is the one big formula you're going to have to remember when you take the
exam,
because you're not going to be allowed to bring cheatsheets.

You have to remember $\alpha$ is related to $\theta$ in that way
and $\theta$ is the phase of $f''$.

Now there is one complication,
which applies in the real world,
but probably not your homework problems.

Recall that there are 2 possible values of the angle $\alpha$,
which differ by $\pi$.
Which value of $\alpha$ should we choose?
\begin{align}
    \alpha &= -\frac{\theta}{2} \pm \frac{\pi}{2}
\end{align}
That depends on the direction we cross the saddle point.
If the direction is reversed,
the value of the integral will flip sign.

Let me explain what I mean.
Let's say you have to integrate from $A$ to $B$.
Let's say the saddle point is somewhere in the middle.
You want to cross the saddle point along that direction from $A$ to $B$.
And so this would be the angle $\alpha$.

[picture]

But suppose $B$ is here and $A$ is here.

[picture]

Now you have to go in the opposite direction.
The angle $\alpha$ is this angle,
which is different from the original $\alpha$ by $\pi$.
Depending on where you're starting or ending,
you're going through the saddle point in a different direction,
in which case you get a different value of $\alpha$.
So it's either you can go this way or that way,
and it's just a minus sign difference.
You just reverse the direction you're integration.

So that ambiguity is hidden here:
$\alpha = -\frac{\theta}{2} \pm
\frac{\pi}{2}$.

Sometimes you can have weird contours.
In the real world,
you need to create the full ridge.
Make sure you only cross it once over the direction you chose.
Remember if you're signing a paper,
you're putting your reputation on the line,
and these subtleties do show up.
This is not going to be in the homework of course.

I have 2 examples in the notes,
but I don't have time to go other both of them.
Let's do the shorter example because we don't have a whole bunch of time.
Most of what I talk about is based on Matthews and Walker.

\begin{example}
    The Hankel function of the first kind,
    which is a solution of the Bessel equation,
    may be represented by the contour integral
    \begin{align}
        H_{\nu}^{(1)}(s) &=
        \frac{1}{\pi i}
        \int_{\epsilon}^{\infty e^{i\pi}}
        e^{\frac{s}{2}\left( z - \frac{1}{z} \right)}
        \frac{dz}{z^{\nu + 1}}
    \end{align}
    where the $(1)$ means first kind.
    Here $\epsilon\to 0$, and the contour is shown in the figure.
    There is a branch cut because of the $1/z^{\nu + 1}$,
    and you choose the branch cut to go along the negative real axis.
    The contour to integrate over looks like this:
    Start from $\epsilon$ then spiral anticlockwise around to negative
    $-\infty$ staying above the branch cut.
\end{example}
This is a solution to the Bessel equation,
because if you substitute it in,
the integrand doesn't satsify it,
but inside the integral,
you take a total derivative and these limits are cunningly chosen so that at the
limits the derivative piece once you do the integration vanishes.
We won't have to discuss that,
since this course doesn't cover special functions at all.
We won't have time to discuss,
but I can just outline.
The thing you're left with is a total derivative,
and since it's sitting on a derivative,
the endpoints you're left with are 0 at the end points.
There is an extended discussion Arfken.

Our problem is just to find an approximation to this integral in the region of
large $s$.

The integrand has a branch cut along the negative $x$-axis
and the contour is to be taken from $\epsilon$ to $-\infty$
above this branch cut.

We want to approximate this integral,
so we have a formula for the method of steepest descent.
The integral we began from is this integral
\begin{align}
    I(s) &=
    \int_C g(z) e^{s f(z)} dz
    \approx
    \frac{%
        \sqrt{2\pi} g(z_0) e^{sf(z_0)} e^{i\alpha}
    }{%
        \sqrt{s |f''(z_0)|}
    }
\end{align}
So we choose
\begin{align}
    g(z) &= \frac{1}{z^{\nu + 1}}\\
    f(z) &= \frac{1}{2}\left( z - \frac{1}{z} \right)
\end{align}
Take derivatives
\begin{align}
    f'(z) &= \frac{1}{2}\left( 1 + \frac{1}{z^2} \right)\\
    f''(z) &= -\frac{i}{z^3}
\end{align}
So there are saddle points at $z = \pm i$,
but we want $z_0 = +i$
because that's above the branch cut.

\begin{align}
    f(z_0) &= \frac{1}{2}\left( i - \frac{1}{i} \right) = 1\\
    f''(z_0) &= -\frac{1}{i^3} = -i
\end{align}
Then
\begin{align}
    f''(z_0) &= |f''(z_0)| e^{i\pi/2}
\end{align}
so
\begin{align}
    \theta = - \frac{\pi}{2}
\end{align}
so we get
\begin{align}
    \alpha &= - \frac{\theta}{2} \pm \frac{\pi}{2}\\
    &= \frac{3\pi}{4} \quad \text{or} -\frac{\pi}{4}
\end{align}
Because of the direction we're going,
the correct angle is
\begin{align}
    \alpha=\frac{3\pi}{4}.
\end{align}
Then,
\begin{align}
    I(s) &\approx
    \frac{%
        \sqrt{2\pi} g(z_0) e^{sf(z_0)} e^{i\alpha}
    }{%
        \sqrt{s |f''(z_0)|}
    }\\
    &=
    \sqrt{\frac{2}{\pi s}}
    e^{i\left(
        s - \nu \frac{\pi}{2} + \frac{\pi}{4}
    \right)}
\end{align}
So, any questions?

We're done for the mathematical methods part of the course.

Next lecture,
we're back to in person,
and we will begin the classical mechanics part of the course.

I'll see you next week.


\documentclass{notes}

  \title{Example Turing Machine}
  \author{ian.mcloughlin@gmit.ie}
  \date{\today}

\begin{document}

  \subsection*{Language}
  \begin{align*}
      A    &= \{ 0, 1 \} \\
      A^*  &= \{ \epsilon, 0, 1, 00, 01, 10, 11, 000, \ldots \} \\
      L    &= \{ \epsilon, 01, 0011, 000111, \ldots \} \\
      A^* \setminus L &= \{ 0, 1, 00, 10, \ldots \} \\[2mm]
    \mathbb{N}_0 &= \{ 0 , 1 , 2 , 3 , \ldots \} \\
      L &= \{ 0^i 1^i | i \in \mathbb{N}_0 \} \\
  \end{align*}

  \section*{Turing machine}
  \begin{center}
    \begin{tabular}{x{1cm}x{1cm}x{1cm}x{1cm}x{1cm}}
      \toprule
      State & Input & Write & Move & Next \\
      \midrule
      \(q_0\) & \bl & \bl & R & \(q_a\) \\
      \(q_0\) &   0 & \bl & R & \(q_1\) \\
      \(q_0\) &   1 &   1 & R & \(q_f\) \\
      \midrule
      \(q_1\) & \bl & \bl & L & \(q_2\) \\
      \(q_1\) &   0 &   0 & R & \(q_1\) \\
      \(q_1\) &   1 &   1 & R & \(q_1\) \\
      \midrule
      \(q_2\) & \bl & \bl & R & \(q_f\) \\
      \(q_2\) &   0 &   0 & R & \(q_f\) \\
      \(q_2\) &   1 & \bl & L & \(q_3\) \\
      \midrule
      \(q_3\) & \bl & \bl & R & \(q_0\) \\
      \(q_3\) &   0 &   0 & L & \(q_3\) \\
      \(q_3\) &   1 &   1 & L & \(q_3\) \\
      \bottomrule
    \end{tabular}
  \end{center}

  \section*{Example input}

  \begin{align*}
    &\           q_0 000111    \rightarrow  q_1 00111  \rightarrow 0 q_1 0111 \rightarrow 00 q_1 111 \\
    &\rightarrow 001 q_1 11    \rightarrow 0011 q_1 1  \rightarrow 00111 q_1  \rightarrow 0011 q_2 1 \\
    &\rightarrow 001 q_3 1     \rightarrow 00 q_3 11   \rightarrow 0 q_3 011  \rightarrow q_3 0011   \\
    &\rightarrow q_3 \blm 0011 \rightarrow q_0 0011    \rightarrow q_1 011    \rightarrow 0 q_1 11   \\
    &\rightarrow 01 q_1 1      \rightarrow 011 q_1     \rightarrow 01 q_2 1   \rightarrow 0 q_3 1    \\
    &\rightarrow q_3 01        \rightarrow q_3 \blm 01 \rightarrow q_0 01     \rightarrow q_1 1      \\
    &\rightarrow 1 q_1         \rightarrow q_2 1       \rightarrow q_3        \rightarrow q_1 \rightarrow q_a \\
  \end{align*}

  \subsection*{Steps}
  \begin{align*}
    &q_0 000111 \rightarrow \ldots 13 \textrm{ steps} \ldots \rightarrow q_0 0011 \\
    &\rightarrow \ldots 9 \textrm{ steps} \ldots \rightarrow q_0 01 \rightarrow \ldots 5 \textrm{ steps} \ldots\\
    & \rightarrow q_0 \rightarrow \ldots 1 \textrm{ step} \ldots \rightarrow q_a \qquad (28 \textrm{ total})
  \end{align*}

    \subsection*{Simulation}

    \begin{center}
    \begin{tabular}{crrrrrrrr}
      \toprule
      $n$   & 0 &  2 &  4 &  6 &  8 & 10 & 12 &  14 \\
      \midrule
      $f(n)$ & 1 &  6 & 15 & 28 & 45 & 66 & 91 & 120 \\
      \bottomrule
    \end{tabular}
    \end{center}

    \subsection*{Sequence}
    OEIS~\cite{oeisA000384} gives sequence formula:
    \[ a(i):\mathbb{N} \rightarrow \mathbb{N}_0 = i (2i - 1) \]

    So, \(a(1)=1\), \(a(2)=6\), \(a(3)=15\), and so on.
    We index as \(2\mathbb{N} = \{ 0,2,4,6,8,10,\ldots \} \). Transform:
    \[ h(n):2\mathbb{N}_0 \rightarrow \mathbb{N} = \frac{n}{2} + 1. \]
    So, \(h(0)=1\), \(h(2)=2\), \(h(4)=3\), and so on.

    \begin{align*}
    f(n):2\mathbb{N}_0 \rightarrow \mathbb{N}_0 &= a(h(n)) \\
         &= \left(\frac{n}{2} + 1\right) \left(2 \left(\frac{n}{2} + 1\right) - 1 \right) \\
         &= \left(\frac{n}{2} + 1\right) \left(n + 2 - 1\right) \\
         &= \frac{1}{2} \left(n + 2\right) \left(n + 1\right) \\
         &= \frac{1}{2} \left(n^2 + 3n + 2 \right) \\
    \end{align*}

    So, $f(n)$ is $O(n^2)$.

    \subsection*{Justification}
    Is \( f(n) \) the correct formula for the number of steps taken for an accepted input of length \(n\)?

    Each pass right and left across the \(j\) non-blank tape cells, the machine takes \(j+1\) steps right, followed by \(j\) steps left.
    \begin{center}
    \begin{tabular}{llrr}
    \toprule
        \textbf{Start} & \textbf{End} & \textbf{Right} & \textbf{Left} \\
        \midrule
        000111       & 0011         & 7 & 6 \\
        0011         & 01           & 5 & 4 \\
        01           & \(\epsilon\) & 3 & 2 \\
        \(\epsilon\) & \(q_a\)     & 1 & 0 \\
    \bottomrule
    \end{tabular}
    \end{center}

    \begin{align*}
       f(n) &= (n+1) + n + \ldots + 2 + 1 + 0 \\
            &= \left((n+1) + 0\right) + \left((n) + 1\right) + \ldots \\
            &= \left(\frac{n}{2} + 1\right)(n+1) \\
    \end{align*}

    \subsection*{Decider}
    Does the Turing Machine always halt and if so, does it reject in \(O(n^2)\)?
    Is \(L \in \mathbf{P}\)?

  \bibliography{bibliography}
\end{document}


\subsection{Numerical Approximation and Analysis}\label{sec:sample-algorithm}

The numerical approximation of the updated density can be achieved by following Algorithm~\ref{alg:updated_density}, which involves approximating a push-forward distribution using a finite set of samples drawn from the initial density and mapped forward by $\qoi$.
Once an approximation of the predicted density is formed, it can be evaluated alongside the initial and observed density at any other point so long as evaluation through $\qoi$ is available.

\begin{algorithm}[hbtp]
\DontPrintSemicolon
Draw $\nsamps$ samples from the initial density to construct the set $\set{\param^{(\iparam)}}_{\iparam=1}^{\nsamps} \subset \pspace$.
	\For{$\iparam = 1, \hdots, \nsamps$}{
	Compute $\qoi_\iparam = \qoi(\param^{(\iparam)})$.\\
	}
	Approximate $\predicted\Q$, the push-forward of $\initial$, by some method such as kernel density estimation.
  \For{$\iparam = 1, \hdots, \nsamps$}{
	Compute $\updated\lami = \initial\Qi \frac{\observed\Qi}{\predicted\Qi}$.\\
	}

 \caption{Numerical Approximation of the Inverse Density using the Sample-Based Approach}
 \label{alg:updated_density}
\end{algorithm}

TK -
picture of pushforward given different number of samples, overview of KDE


\chapter{Manifolds and differentiable structures}

\begin{problem}{1}
Show that every (differentiable) manifold possesses a countable (differentiable) atlas.
\end{problem}
\begin{proof}
Let $\mathcal{A} = \{(U_\alpha, h_\alpha) \mid \alpha \in A \}$ be an atlas for $M.$ Since $M$ is second countable, there exists a countable basis $B = \{V_n \mid n \in \N \}.$ For each $p \in M$ there exist $V_k \in B$ and $(U_\beta, h_\beta) \in \mathcal A$ such that $p \in V_k \subset U_\beta.$ Since $B$ is countable, there is a sequence $\{\alpha_{k}\}_{k}^\infty \subset A$ such that $\bigcup_k U_{\alpha_k} = M$. It follows that $\{(U_{\alpha_k}), h_{\alpha_l} \mid k \in \N \}$ is a countable atlas of $M.$
\end{proof}

\begin{problem}{2}
Show that the sphere $S^n$ possesses a differentiable atlas with precisely two charts. Also, one with only one chart?
\end{problem}
\begin{proof}
Let $p_1 = (0, \ldots, 0, 1) \in S^n$ and $p_2 = (0,\ldots,0, -1) \in S^n$. Then the sets $U_ 1 = S^n - {p_1}$ and $U_2 = S^n - \{p_2\}$ are open. Define $h_1: U_1 \rightarrow \R^n$ and $h_2: U_2 \rightarrow \R^n$ by
$$
h_1(x_1, \ldots, x_{n+1}) = \left(\frac{x_1}{1-x_{n+1}}, \ldots, \frac{x_n}{1-x_{n+1}}\right),
$$
$$
h_2(x_1, \ldots, x_{n+1}) = \left(\frac{x_1}{1+x_{n+1}}, \ldots, \frac{x_n}{1+x_{n+1}}\right).
$$

A direct argument shows that these functions are homeomorphisms from $U_1, U_2$ onto $\R^{n}.$ Their inverses, $h_1^{-1}, h_2^{-1}$, are defined in the following way:
$$
h_1^{-1}(x_1, \ldots, x_n) = \left( \frac{2x_1}{1 + r^2}, \ldots,  \frac{2 x_n}{1+r^2}, \frac{r^2-1}{1+r^2} \right),
$$
$$
h_2^{-1}(x_1, \ldots, x_n) = \left( \frac{2x_1}{1 - r^2}, \ldots,  \frac{2 x_n}{1-r^2}, \frac{r^2}{1-r^2} \right).
$$

with $r^2 = \sum_{k=1}^n x_k^2.$ Then $h_2 \circ h_1^{-1}: \R^n \rightarrow \R^n$ is defined
$$ (h_2 \circ h_1^{-1}) (x_1, \ldots, x_n) = h_2 \left( \frac{2x_1}{1 + r^2}, \ldots,  \frac{2 x_n}{1+r^2}, \frac{r^2-1}{1+r^2} \right) = \frac{1}{r^2}(x_1, \ldots, x_n).$$

Hence $h_2 \circ h_1^{-1}$ is differentiable. In a similar way, $h_1 \circ h_2^{-1}$ is differentiable. Therefore, the atlas $\{(U_1, h_1), (U_2, h_2)\}$ defines a differentiable structure.

\end{proof}

\begin{problem}{3}
Describe the chart transformation for the atlas $\R P^n$ in (1.5(c)), and show that it is differentiable.
\end{problem}

\begin{problem}{4}
Let $M$ be a differentiable manifold and $\tau: M \rightarrow M$ a fixed point free involution, that is, $\tau$ is a diffeomorphism with $\tau \circ \tau = Id_M$ and $\tau(x) \neq x$ for all $x.$

Show that the quotient space $M/\tau$, which is obtained from $M$ by identification of points corresponding to each other under $\tau$, is a topological manifold which possesses exactly one differentiable structure with respect to which the projection $M \rightarrow M/\tau$ is locally diffeomorphic.
\end{problem}
\begin{proof}
Let $p: M \rightarrow M/\tau$ be the canonical projection and let $\mathcal A$ be a maximal atlas of $M$. Since $M$ is Hausdorff, for each $x \in M$ there exist open sets $U, V$ such that $x\in U$, $\tau(x) \in V$ and $U \cap V = \varnothing.$ Since $\tau$ is continuous, $W = U \cap \tau^{-1}(V)$ is a non-empty open set. This open set satisfies $W \cap \tau(W) = \varnothing$; then the set $\mathcal B = \{(W, k) \in \mathcal A \mid W \cap \tau(W) \}$ is non-empty.

For each $(W, k)\in \mathcal B$, the function $p$ is injective in $W$. Let $x, y \in W$ with $p(x) = p(y)$, that is, $x,y$ belong to the same equivalence class. But $W \cap \tau(W) = \varnothing$, hence $x=y.$ Since by definition $p$ is surjective, we have shown that $p$ is bijective in $W.$

A direct argument shows that the set $\mathcal C = \{(p(W), k \circ p^{-1}) \mid (W, k) \in \mathcal B \}$ induces a differentiable structure in $M/\tau$. To show that $p$ is locally diffeomorphic with respect to these differentiable structures, let $x \in M$ and $(W, k)\in \mathcal B$ with $x\in W$. The composition $(k \circ p^{-1}) \circ p \circ k^{-1} = Id_{\R^n}$ is differentiable. The inverse $p^{-1}: p(W) \rightarrow W$ of $p$ is also differentiable, since the composition $k \circ p^{-1}\circ (k \circ p^{-1})^{-1} = k \circ p^{-1} \circ (p \circ k^{-1}) = Id_{\R^n}$ is differentiable.

UNIQUENESS
\end{proof}

\begin{problem}{5}
Show that $\R P^1 \cong S^1.$
\end{problem}
\begin{proof}

\end{proof}

\begin{problem}{6}
Provide the surface of a cube $\{x\in \R^{n+1} \mid \max \{\lvert x_i \rvert\ = 1 \}$ with the structure of a differentiable manifold.
\end{problem}

\begin{problem}{7}
Let $M$ be a differentiable manifold and $f: N \rightarrow M$ a homeomorphism. Prove that $N$ possesses exactly one structure as a differentiable manifold, so that $f$ is diffeomorphic.
\end{problem}

\begin{problem}{8}
Provide the complex projective sapce $\C P^n$ with the structure of a $2n-$dimensional differentiable manifold. This space is defined as follows: on the complex vector space $\C^{n+1},$ one has the equivalence relation $x \sim y$ if and only if there is a number $\lambda \in \C$, $\lambda \neq 0$, so that $\lambda x = y$. The quotient space $(\C - \{0\}) / \sim $ is defined to be $\C P^n$.
\end{problem}

\begin{problem}{9}
Prove that if $M$ is a non-empty, $n-$dimensional manifold and $k \leq n$, then tere is an embedding $\R^k \rightarrow M.$
\end{problem}

\begin{problem}{10}
Let $N$ be a compact, $M$ a connected manifold, both of dimension $n$ and non-empty. Let $f: N \rightarrow M$ be an embedding. Show that $f$ is a diffeomorphism.
\end{problem}

\begin{problem}{11}
Show that $S^n$ is a submanifold of $\R^{n+1}.$
\end{problem}

\begin{problem}{12}
Describe an embedding $S^1 \times S^1 \rightarrow \R^3$ by means of elementary functions.
\end{problem}

\begin{problem}{13}
Show that the composition of two embeddings is again an embedding and the Cartesian product $f_1 \times f_2: N_1 \times N_2 \rightarrow M_1 \times M_2$, of two embeddings $f_1, f_2$, is again an embedding.
\end{problem}

\begin{problem}{14}
Show that if the $n-$dimensional manifold $M$ is a product of spheres, then there exists an embedding $M\rightarrow \R^{n+1}.$
\end{problem}

\begin{problem}{15}
The points of $\C P^k$ are described by the homogeneous coordinates $x = [x_0, \ldots, x_k]:= \text{ class of} (x_0, \ldots, x_k)$ under $\sim$. Show that the mapping
$$
f: \C P^m \times \C P^n \rightarrow \C P^{mn + m + n}
$$
$$
(x, y) \mapsto [x_0y_0, x_0 y_1, \ldots, x_v y_\mu, \ldots, x_m y_n ]
$$
is an embedding. Show the same for the real projective spaces.
\end{problem}

\begin{problem}{16}
Let $M(m \times n)$ be the vector space of real $(m\times n)-$matrices, and $M_r(m \times n)$ the subset of matrices of rank $r$. Then $M_r(m \times n)$ is a submanifold of $M(m \times n)$ of codimension $(n-r)(m-r)$ for $r \leq \min \{m,n\}.$
\end{problem}

\begin{problem}{17}
The inclusion $\R^{n+1} \subset \R^{n+2}$ induces an embedding $\R P^n \subset \R P^n{n+1}$ and $\R P^{n+1} - \R P^{n} \cong \R^{n+1}.$
\end{problem}

\begin{problem}{18}
Let $\R^{n+1} = \{(x, a_0, \ldots, a_{n-1}) \mid x, a_i \in \R \}$. The set of points such that $x^n + a_{n-1}x^{n-1}+ \ldots + a_0 = 0$ is a submanifold of codimension $1$ of $\R^{n+1}$, and is diffeomorphic to $\R^n.$
\end{problem}

\begin{problem}{19}
The set $C^\infty(M)$ is an algebra under the natural addition and multiplication of functions. A differentiable mapping $f: M \rightarrow N$ defines an algebra homomorphism
$$
f^\ast: C^\infty(N) \rightarrow C^\infty(M), \ \ \phi \mapsto \phi \circ f
$$
with the functorial properties: $Id^\ast_M = Id;$ $(f \circ g)^\ast = g^\ast \circ f^\ast.$
\end{problem}

\begin{problem}{20}
Notation as in 19. For a point $p\in M$ let
$$
\mathrm{M}_p = \{\phi \in C^\infty(M) \mid \phi(p) = 0 \}.
$$
Show:
\begin{itemize}
\item[(a)] $\mathrm{M}_p$ is a maximal ideal of $C^\infty(M).$
\item[(b)] If $M$ is compact and $\mathrm{M} \in C^\infty(M)$ is a maximal ideal, then tehere exists some $p \in \mathrm{M}$ such that $\mathrm{M} = \mathrm{M}_p.$
\end{itemize}
\end{problem}