\input{../preamble}

\renewcommand{\thefootnote}{\fnsymbol{footnote}}

\begin{document}
\title{Euler Characteristic of the Sphere}
\author{Apurva Nakade}
\maketitle

A topological invariant assigns to a topological space, in our case a surface, an algebraic object such as a number, a polynomial or a vector space. If two surfaces have different topological invariants then they must be topologically inequivalent. (However the converse is not always true in that two inequivalent surfaces can have the same topological invariants.) The simplest non-trivial topological invariant is the Euler characteristic. The Euler characteristic assigns to each surface an integer and can be thought of as a way of quantifying shapes.

Euler characteristic for surfaces is computed using graphs. We begin with the simplest surface, the sphere $ S ^ 2 $.

\begin{example}
	For a polyhedron $P$ let $v,e,f$ denote the number of vertices, edges and faces respectively. When $P$ is a tetrahedron $v = 4, e=6, f=4$ and hence $v-e+f = 2$. When $P$ is a cube $v = 8, e=12, f=6$ and hence $v-e+f = 2$.
\end{example}

What does the cube and the tetrahedron have in common? They can all be continuously deformed to the 2 dimensional sphere $ S^2 $.

\section{Planar graphs}
We'll start by computing the Euler characteristic of a planar graph. For us a graph $ G $ is a pair of finite sets $(V,E) $ where the vertices, $ V $, are distinct points in the standard plane $ \R ^2 $ and the edges, $ E $, are segments connecting two vertices. We say that a graph is \textbf{connected} if each vertex is connected to every other vertex via a sequence of edges. We say that a graph is \textbf{planar} if no two edges intersect in a point outside the set of vertices.

A planar graph divides the plane $ \R ^ 2 $ into \textbf{faces}. Denote the set of faces by $ F $.\footnote{We won't include the unbounded face in $F$, for us all the faces are (possibly non-convex) polygons.} (It is possible for the set $ F $ to be empty.) The \textbf{Euler characteristic} of a planar graph $ G $ is defined to be
\begin{align*}
	\chi (G):= |V| - |E| + |F|
\end{align*}
where $ |S| $ denotes the size of the set $ S $.

\begin{thm}\label{thm:euler_characteristic_of_graphs}
	The Euler characteristic of a connected planar graph is 1.
\end{thm}

Exercise \ref{proof_of_euler's_thm} describes one proof of this theorem using induction on the number of faces. First we prove the theorem in the case when the graph has no faces.

A \emph{connected} graph $ G $ without any face, i.e. when $ F $ is an empty set  or equivalently $ |F| = 0 $, is called a \textbf{tree}. (A graph with no faces but which is not necessarily connected is called a forest!) A vertex in a tree with only one edge attached to it is called a \textbf{leaf}.

\begin{exercise}
	For a tree $ G $ what is the relationship between $ |V| $ and $ |E| $? What is $ \chi (G)$?
\end{exercise}

\begin{exercise}\label{proof_of_euler's_thm}
	The following is the proof of \eqref{thm:euler_characteristic_of_graphs}
	\begin{enumerate}
		\item For a connected planar graph $ G $ with at least 1 face, show that it is possible to delete an edge and obtain a graph $ G' $ such that $G' $ has exactly one less face than $ G $.
		\item What is the relationship between the Euler characteristic of $ G $ and $ G' $?
		\item Induct on $ |F| $ to complete the proof of Theorem \ref{thm:euler_characteristic_of_graphs}. (What is the base case for induction here?)
	\end{enumerate}
\end{exercise}

\begin{exercise}
	What is the Euler characteristic of a planar graph which is not necessarily connected?
\end{exercise}

\section{Euler characteristic of $ S ^ 2 $}

A \textbf{surface graph} on a sphere $ S ^ 2 $ is a \emph{connected planar} graph $ G = (V,E) $ such that $ V $ and $ E $ are now on $ S ^ 2 $ and all the faces are \textbf{polygons}. The tetrahedron, the cube, and the octahedron, provide examples of such surface graphs. A surface graph is called a \textbf{triangulation} if all the faces are triangles.

\begin{thm}\label{thm:euler_characteristic_of_sphere}
	The Euler characteristic of any surface graph $G$	on $ S ^ 2 $ is 2. Hence we can define the Euler characteristic of $ S^2 $ as $\chi(S^2):= \chi(G)$ and we have, $$ \chi(S ^ 2) = 2 $$
\end{thm}

\begin{exercise}
	Theorem \ref{thm:euler_characteristic_of_sphere} follows directly from Theorem \ref{thm:euler_characteristic_of_graphs} for planar graphs,
	\begin{enumerate}
		\item Explain how a surface graph on $ S ^ 2 $ gives rise to a planar graph on $ \R ^ 2 $.
		\item Draw the planar graphs for the cube, the tetrahedron and the octahedron.
		\item Show that the Euler characteristic of a graph on $ S ^ 2 $ is 2.
	\end{enumerate}
\end{exercise}

\section{Brussel Sprouts}
The game of \textbf{Brussel Sprouts} starts with 2 crosses. Each move involves joining two free ends with a curve not crossing any existing line and then putting a short stroke across the line to create two new free ends. The game ends when no such move is possible.
\begin{center}
	\begin{tabular}{c}
		{\includegraphics[width=5cm, height=5cm] {images/brussel_sprouts}} \\
		A game of Brussel Sprouts                                          \\\\
	\end{tabular}
\end{center}

\begin{exercise}
	Play a few games of Brussel Sprouts!
\end{exercise}

It turns out that every game of Brussel Sprouts always ends in the same number of steps! The following exercises describe a proof of it,

\begin{exercise}Let $G$ be the connected planar graph (vertices are the crosses) at the end of the game.
	\begin{enumerate}
		\item What happens to the number of free ends after each move? How many free ends are there in the end?
		\item Argue that at each stage of the game every face should have at least one open end on it's boundary. Further, argue that there cannot be two or more open ends on the boundary of a face at the end of the game. Hence every face of $G$ should have exactly 1 open end on it's boundary. The same is true for the \textit{unbounded face}.
		\item Conclude that $G$ has 7 faces.
	\end{enumerate}
\end{exercise}

\begin{exercise}
	Assume that the game ends in $ \mathbf{n} $ steps.
	\begin{enumerate}
		\item How many vertices and edges are added after each move? Argue that $ |E| = 2n $ and $ |V| = 2 + n $.
		\item Use Theorem \ref{thm:euler_characteristic_of_graphs} to find $ n $.
	\end{enumerate}
\end{exercise}
Can you generalize the above proof to $k$ crosses in the beginning? How about playing Brussel Sprouts on a Torus?

\iffalse\subsection{Platonic Solids}
A platonic solid is a convex polyhedron all of whose faces are regular polygons which are congruent to each other, i.e. all the edges have the same length and all the faces have the same number of sides. We'll assume that the word convex means that the platonic solid is topologically equivalent to $S^2$. We've seen 3 platonic solids earlier, the cube, the tetrahedron, and the octahedron. There are exactly two more,

\begin{center}\begin{tabular}{c c}
	%\includegraphics[width=3cm, height=3cm]{icosahedron} & %\includegraphics[width=3cm, height=3cm]{dodecahedron} \\
	icosahedron & dodecahedron \\\\
	\end{tabular}
\end{center}

We can use the Euler characteristic to prove that these 5 are the only ones possible.

Consider a platonic solid $ S $ and think of it as the graph $(V,E,F)$. Suppose all the faces of $ S $ are polygons with $ \mathbf{n} $ number of edges. Further suppose that $ \mathbf{p} $ edges of $S $ intersect at a single vertex.

\begin{exercise}
	Argue that both $ n $ and $ p $ should be at least 3. What are $ n$ and $ p $ for the five platonic solids?
\end{exercise}

\begin{exercise}\leavevmode
	\begin{enumerate}
		\item By counting the total number of edges cleverly, show that $ p |V| = 2 |E| $ and $ n |F| = 2|E| $.
		\item Use Theorem \ref{thm:euler_characteristic_of_sphere} to conclude
		      \begin{align}\label{eq:platonic_solid}
		      	\dfrac{1}{n} - \dfrac{1}{2} + \dfrac{1}{p} = \dfrac{1}{|E|}
		      \end{align}
		      and because $ |E| > 0 $ this implies $ \frac{1}{n} + \frac{1}{p} >  \frac{1}{2}$.
		\item Show that the above inequality cannot hold if $ n $ and $ p $ are both bigger than 4. Now we have that both $n$ and $p$ are at least 3 and one of them is at the most 4.
		\item Find (by trial and error) the possible values of $ n, p, |E| $ that satisfy (\ref{eq:platonic_solid}) and relate them to the 5 platonic solids.
	\end{enumerate}
\end{exercise}
\fi

\end{document}

02 Euler Characteristic/01 Euler's theorem for a plane.tex


\documentclass[bnt.tex]{subfile}

\begin{document}
	\section{Congruence}
		We may call congruence (also known as \textit{modular arithmetic}) the dual of divisibility. It was first introduced and highly
		used by \textit{Carl Fredrich Gauss}.
			\begin{definition}
				If two integers $a$ and $b$ leave the same remainder upon division by $n$, then $a$ and $b$ are said to be \textit{congruent modulo} $n$. In other words, $a$ leaves remainder $b$ (not necessarily minimum or absolute minimum) upon division by $n$.
			\end{definition}

			\begin{example}
				Since $14$ and $62$ leaves the same remainder $6$ upon division by $8$, we say that $14$ and $62$ are congruent modulo $8$. We denote it by $14\equiv62\pmod8$ and say $14$ is congruent to $62$ modulo $8$. Likewise, $11\equiv4\pmod7$. Note that these remainders can be negative. So, we can also take
					\begin{align*}
						11 & \equiv-1\pmod6
					\end{align*}
			\end{example}
		The set $\mathbb{Z}_n=\{0,1,2,\ldots,n-1\}$ (the set of integers modulo $n$) is called the \textit{complete set of residue class modulo} $n$. However, we mostly consider the set $Z_n-\{0\}$. This is called a complete set of residue class modulo $n$ because any integer gives a remainder upon division by $n$ which is an element of this set. Also, it is obvious that every integer gives a unique remainder upon division by $n$ which belongs to this set. This actually follows from $\#10$ of divisibility.
			\begin{definition}
				$P(x)$ is a polynomial a sum of some powers of $x$ (obviously finite). That is,
					\begin{align*}
						P(x) & = a_nx^n+\cdots+a_1x+a_0
					\end{align*}
				The highest power of a polynomial is called \textit{degree} which is $n$ in this case.
			\end{definition}
		The following proposition discusses some of the basics of modular arithmetic.
			\begin{proposition}
				We let $a,b,k,n$ be positive integers.
					\begin{enumerate}[(1)]
						\item $a\equiv b\pmod n\iff n|a-b$. This is straightforward from the definition.
						\item $a\equiv a\pmod n$ (reflexive property).
						\item If $a\equiv b\pmod n$ then $b\equiv a\pmod n$ (symmetric property) and vice versa.
						\item If $a\equiv b\pmod n$ and $b\equiv c\pmod n$ then $a\equiv c\pmod n$ (transitive property).
						\item If $a\equiv b\pmod n$ then $a+nk\equiv b\pmod n$ holds as well.
					\end{enumerate}
			\end{proposition}

			\begin{proposition}[Operations in congruence]
				Assume that $a,b,c,d$ are positive integers. Moreover $a\equiv b\pmod n$ and $c\equiv d\pmod n$. Then
					\begin{enumerate}[(1)]
						\item $a+b\equiv c+d\pmod n$.
						\item $a-b\equiv c-d\pmod n$.
						\item $ac\equiv bd\pmod n$.
					\end{enumerate}
			\end{proposition}
		Proving them is easy. From $a\equiv b\pmod n$, we can say $a=b+nk$ for some integer $k$. Similarly, $c=d+nl$ for some integer $l$. Now,
			\begin{align*}
				a+b & = c+d+n(k+l)\\
					& \equiv c+d\pmod n
			\end{align*}
		The ones described in the previous proposition or the other ones in this proposition can be proved in the same way. We have a corollary.
			\begin{align*}
				ac & \equiv bc\pmod n
			\end{align*}
		You can see, congruence is keeping up with equations so far. So you may want to conjecture that if $ac\equiv bc\pmod n$ then $a\equiv b\pmod n$. However, is that correct? Take some examples. See if all of them are consistent with this statement.
\end{document}


% Ref fra reliability prob
\chapter{Nonhomogenous Poisson process}
\section{Introduction}
In this chapter we introduce the nonhomogenous Poisson process (NHPP) and its sufficient statistics. A NHPP is when the rate function for a Poisson process is dependent on time. This process is first of all a counting process. The following definitions are from \cite{rausand2004system}. The definition of a counting process is as follows
\begin{defn}
A stochastic process $\{N(t), t \geq 0 \}$ is said to be a counting process if $N(t)$ satisfies:
\begin{enumerate}
\item $N(t)\geq 0$.
\item $N(t)$ is integer valued.
\item If $s<t$, then $N(s) \leq N(t)$.
\item For $s<t$, $[N(t) - N(s)]$ represents the number of failures that have occured in the interval $(s,t]$.
\end{enumerate}
\end{defn}
Furthermore the definition of a NHPP is as given below.
\begin{defn} % S 277
A counting process $\{ N(t), t \geq 0 \}$ is a nonhomogeneous (or nonstationary) Poisson process with rate function $\lambda (t)$ for $t \geq 0 $, if
\begin{enumerate}
\item $N(0) = 0$.
\item $\{N(t), t \geq 0\}$ has independent increments.
\item $Pr(N(t + \Delta t) - N(t) \geq 2) = o(\Delta t)$, which means that the system will not experience more than one failure at the same time.
\item $Pr(N(t + \Delta t) - N(t) = 1) = \lambda(t) \Delta t + o(\Delta t)$.
\end{enumerate}
\end{defn}
For this NHPP we are going to use a rate function which is a combination of power-law and log-linear. The rate function is as follows,
\begin{equation}
\lambda (t) = abt^{b-1}e^{ct},
\label{eq:rate}
\end{equation}
where $a$, $b$ and $c$ are parameters of the model. $N(t)$ is Poisson distributed with parameter
\begin{equation}
\Lambda(t) = \int_{0}^{t} \lambda(u) du.
\label{eq:largelambda}
\end{equation}
From the NHPP the number of failures are given by,
\begin{equation}
P(N(t) = n) = \frac{\Lambda(t)^n}{n!}e^{-\Lambda(t)} \quad n=0,1,2,... \quad 0\leq t \leq \tau.
\label{eq:NumNHPP}
\end{equation}
To generate data the desired number of data points can be picked. This means that the parameters of the NHPP must be chosen to fulfill the desired number of data points. From a NHPP we have that the expected number of events within a time $\tau$ is $\Lambda(\tau)$. Since we have assumed that time runs are between 0 and $\tau$, the expected number of events is
\begin{equation*}
E[events] = \Lambda(\tau).
\end{equation*}

\section{Parameters for rate function}
Since the rate function is a combination of log-linear and power-law, as seen in equation \ref{eq:rate}, the shape of the function can vary a lot. The parameters $b$ and $c$ are the one to take notice of. These determine the shape of the function. The parameter $a$ is just for scaling the expected number of events. By testing for different parameters we have chosen six shapes of the rate function to be used. The parameter $a$ is then chosen so that the expected number of events is around 30. This is our desired expected number of events. The resulting shapes are denoted as model 1 through 6. The different models can be viewed in figures \ref{fig:SmallLambdaA10B04C2}, \ref{fig:SmallLambdaA160B2CM3}, \ref{fig:SmallLambdaA20B2C1}, \ref{fig:SmallLambdaA30B07C0} \ref{fig:SmallLambdaA50B2CM1} and \ref{fig:SmallLambdaA6B3C2}. From these figues we see that model 2 and 4 are close to a gamma density \cite{stacy1962generalization}.
\includefigure[width=0.9\textwidth]{fig/SmallLambdaA10B04C2.png}{fig:SmallLambdaA10B04C2}{Plot of rate function for model 1 given $a = 10$, $b = 0.4$ and $c = 2$.}
\includefigure[width=0.9\textwidth]{fig/SmallLambdaA160B2CM3.png}{fig:SmallLambdaA160B2CM3}{Plot of rate function for model 2 given $a = 160$, $b = 2$ and $c = -3$.}
\includefigure[width=0.9\textwidth]{fig/SmallLambdaA20B2C1.png}{fig:SmallLambdaA20B2C1}{Plot of rate function for model 3 given $a = 20$, $b = 2$ and $c = 1$.}
\includefigure[width=0.9\textwidth]{fig/SmallLambdaA30B07C0.png}{fig:SmallLambdaA30B07C0}{Plot of rate function for model 4 given $a = 30$, $b = 0.7$ and $c = 0$.}
\includefigure[width=0.9\textwidth]{fig/SmallLambdaA50B2CM1.png}{fig:SmallLambdaA50B2CM1}{Plot of rate function for model 5 given $a = 50$, $b = 2$ and $c = -1$.}
\includefigure[width=0.9\textwidth]{fig/SmallLambdaA6B3C2.png}{fig:SmallLambdaA6B3C2}{Plot of rate function for model 6 given $a = 6$, $b = 3$ and $c = 2$.}

%


\subsection{Classical DNN}
First, we have a more comprehensive notation for classical DNN models.
\begin{equation}\label{eq:DNNdef_J}
\begin{aligned}
{\rm{DNN}_J} :=\{& f:f=
\theta^J \circ \sigma \circ \theta^{J-1} \cdots \sigma \circ \theta^0(x), \\
&\theta^\ell \in \mathbb{R}^{n^{\ell+1} \times (n^\ell+1)}, \quad n^0 = d, \quad n^{J+1} = 1, \quad n^\ell \in \mathbb{N}^+\}.
\end{aligned}
\end{equation}

Thus to say, we have the general two definition for DNN with
\begin{itemize}
\item  $\sigma \circ \theta$ type:
\begin{equation}\label{eq:sigma+theta}
\begin{aligned}
f^0 &= x, \\
f^{i+1} &= \sigma \circ \theta^{i}(f^i), \\
{\rm DNN}_J &= \{\theta^J(f^J)\}.
\end{aligned}
\end{equation}

\item $\theta \circ \sigma $ type:
\begin{equation}\label{eq:theta+sigma}
\begin{aligned}
f^0 &= \theta^0(x), \\
f^{i+1} &=  \theta^{i+1} \circ \sigma (f^i), \\
{\rm DNN}_J &= \{f^J\}.
\end{aligned}
\end{equation}
\end{itemize}

\subsection{DNN type ResNet}
For simplicity, we choose $\sigma \circ \theta$ type as example.

\paragraph{ResNet}
The ResNet can be written as
\begin{equation}\label{ori-ResNet-dnn}
\begin{cases}
f^0 &= x, \\
f^{i} &= \sigma \left( P^i f^{i-1} + \mathcal{F}^{ i} (f^{i-1}) \right), \quad i = 1:J ,\\
{\rm ResNet}_{J} &= \{  \theta^J f^{J} \}.
\end{cases}
\end{equation}
Here
\begin{equation}\label{eq:F-ResNet}
\mathcal{F}^{i} (f^{i-1}) = \xi^{i} \circ \sigma \circ \eta^{i} (f^{i-1}),
\end{equation}
means ResNet with skip connection distant 2. And $P^i$ is use to fit the dimension as
\begin{equation}\label{eq:P^i}
P^i: \mathbb{R}^{n_{i-1}} \mapsto \mathbb{R}^{n_i}.
\end{equation}

\paragraph{iResNet}
The iResNet can be written as:
\begin{equation}\label{ori-iResNet-dnn}
\begin{cases}
f^0 &= x, \\
f^{i} &=  P^i f^{i-1} + \mathcal{F}^{ i} (f^{i-1}) , \quad i = 1:J ,\\
{\rm iResNet}_{J} &= \{  \theta^J f^{J} \}.
\end{cases}
\end{equation}
Here
\begin{equation}\label{eq:F-iResNet}
\mathcal{F}^{i} (f^{i-1}) = \xi^{i} \circ \sigma \circ \eta^{i}  \circ \sigma (f^{i-1}),
\end{equation}
means iResNet with skip connection distant 2.
And $P^i$ is use to fit the dimension as in ResNet in \eqref{eq:P^i}.

The only difference between ResNet and iResNet can be viewed as
putting a $\sigma$ in different places.

And we also need to notice that ${\rm ResNet}_J$ or  ${\rm iResNet}_J$
are often called DNN with $2J$-th layers if the distance of skip connection
is $2$ as in \eqref{eq:F-ResNet} and \eqref{eq:F-iResNet}.

\subsection{DNN type MgNet}
Similar with ResNet, we can rewrite MgNet.

Here use $\theta \circ \sigma$ type as example.
\begin{equation}\label{ori-MgNetNet-dnn}
\begin{cases}
f^0 &= 0, \quad f^0 = \theta^0(x) \\
f^{i} &=  P^i f^{i-1} + \mathcal{F}^{ i} (f^{i-1}) , \quad i = 1:J ,\\
{\rm iResNet}_{J} &= \{  f^{J} \}.
\end{cases}
\end{equation}
Here
\begin{equation}\label{eq:F-MgNet}
\mathcal{F}^{i} (f^{i-1}) = \xi^{i} \left( f^{i-1} +  \sigma \circ \eta^{i} \circ \sigma(f^{i-1}) \right).
\end{equation}

\subsection{DNN type DenseNet}
In fact, DenseNet might be simple for definition in DNN case.

Here use $\sigma \circ \theta$ type as example.
\begin{equation}\label{ori-DenseNet-dnn}
\begin{cases}
f^0 &= x, \\
f^{i} &=   \sigma \circ \theta^{i}([f^{i-1}, f^{i-2}, \cdots, f^0]) , \quad i = 1:J ,\\
{\rm DenseNet}_{J} &= \{  \theta^J f^{J} \}.
\end{cases}
\end{equation}

Here $[f^{i-1}, f^{i-2}, \cdots, f^0]$ means a long vector by collecting all
outputs from $f^0$ to $f^{i-1}$, thus to say
$$
{\rm dim}([f^{i-1}, f^{i-2}, \cdots, f^0]) = \sum_{i=0}^{i-1} n_i.
$$

\section{A Universal DNN Model}

\subsection{Kailai's definition}
A DNN is defined as a tuple $M=(\mathcal{S}, \mathcal{O}, s_0, F, \delta)$
\begin{itemize}
	\item $\mathcal{S}$ is a non-empty set of states.
	\item $\mathcal{O}$ is a finite, non-empty set of parametrized operators.
	\item $s_0\in \mathcal{S}$ is the initial input.
	\item $F\subset \mathcal{S}$ is the set of final states~(outputs).
	\item $\delta: 2^{\mathcal{S}}\times \mathcal{O} \rightarrow \mathcal{S}$ is the mapping function.
\end{itemize}
and an acceptable ordered sequence $(\delta_1, \delta_2, \ldots, \delta_n)$, which maps $s_0$ to $\delta_n \circ \delta_{n-1} \circ \delta_1 (s_0) \in F$.

\subsection{Juncai's definition}
The idea is that, deep neural network comes from the composition of linear and
element-wise activation.
So, we define the basic component of our model as:
\begin{equation}
\mathcal L_{\sigma,1}(x) = Wx + b + \sigma(\tilde Wx + \tilde b),
\end{equation}
where
\begin{equation}
x \in \mathbb{R}^d, \quad W, ~ \tilde W \in \mathbb{R}^{n \times d} \quad \text{and} \quad b,~ \tilde b \in \mathbb{R}^n.
\end{equation}
Then we try to define an important operator in the universal DNN model,
known as $\mathcal L_{\sigma, \ell}(x^1, \cdots, x^k)$, by recursion of $\mathcal L_{\sigma,1}$.
For $x^i \in \mathbb{R}^{n_i}, i = 1:\ell$,  we have
\begin{equation}
\mathcal L_{\sigma, \ell}(x^1, \cdots, x^k) = \mathcal L_{\sigma,1}
\left([\mathcal L_{\sigma, \ell-1}(\hat x^1), \mathcal L_{\sigma, \ell-1}(\hat x^2), \cdots, \mathcal L_{\sigma, \ell-1}(\hat x^k)]\right),
\end{equation}
where
\begin{equation}
\mathcal L_{\sigma, \ell-1}(\hat x^k) = \mathcal L_{\sigma, \ell-1} (x^1, \cdots, x^{k-1}, x^{k+1}, \cdots, x^\ell),
\end{equation}
and
\begin{equation}
[\mathcal L_{\sigma, \ell-1}(\hat x^1), \mathcal L_{\sigma, \ell-1}(\hat x^2), \cdots, \mathcal L_{\sigma, \ell-1}(\hat x^k)],
\end{equation}
means to collect all the output of $\mathcal L_{\sigma, \ell-1}(\hat x^k)$ into one vector such
that it can be the input of $\mathcal L_{\sigma, 1}$.

Then we define the $J-$layer universal DNN model by recursion as:
\begin{equation}
\begin{cases}
f^{0} &= x,  \\
f^{\ell} &= \mathcal L_{\sigma, \ell}(f^0,\cdots, f^{\ell-1}), \quad \ell = 1:J, \\
f(x) &= W^J f^J + b^J.
\end{cases}
\end{equation}


\documentclass{article}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{eqnarray}
\usepackage[utf8]{inputenc}

\setlength{\parindent}{0pt}
\setlength{\parskip}{1em}

\newcommand{\K}{\mathbb{K}}
\newcommand{\N}{\mathbb{N}}
\newcommand{\Z}{\mathbb{Z}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\C}{\mathbb{C}}
\newcommand{\Q}{\mathbb{Q}}
\newcommand{\F}{\mathcal{F}}
\newcommand{\G}{\mathcal{G}}

\begin{document}

\section{Polynomials}

\textbf{Definition 1.}  Polynomial function $P \colon \mathbb{R} \to \mathbb{R}$ can be presented in the form of
$$P(x)=a_nx^n+a_{n-1}x^{n-1}+\dots+a_1x+a_0$$
where $a_0,\dots,a_n$ are real numbers called the polynomial coefficients. Largest $n$ for which $a_n\ne 0$ is called the degree of polynomial.

\textbf{Theorem 2 (Bezout's theorem).}
A polynomial $P(x)$ is divisible by the binomial $(x-a)$ if and only if $P(a)=0$.

\textbf{Theorem 3 (The fundamental theorem of algebra).}
Every non-constant polynomial has a complex root.

\textbf{Theorem 4 (The rational root theorem).} If $x = p/q$ is a rational zero
of a polynomial $P(x) = a_nx^n +\hdots +a_0$ with integer coefficients and $(p, q)=1$,
then $p | a_0$ and $q | a_n$.

\textbf{Theorem 5 (Vieta's formulae).} If the solutions polynomial of degree $n$ are $x_1,x_2,\dots,x_n$ and  $a_n=1$, then the following holds:
\begin{eqnarray*}
x_1+x_2+\ldots+x_n &=& -a_{n-1},\\
x_1x_2+x_1x_3+\ldots+x_{n-1}x_{n} &=& \hphantom{-} a_{n-2}, \\
x_1x_2x_3+x_1x_2x_4+\ldots+x_{n-2}x_{n-1}x_n &=& -a_{n-3},\\
\ldots \\
x_1x_2\ldots x_n &=& (-1)^n a_0.
\end{eqnarray*}

\begin{enumerate}

\item
Find the roots of polynomial $P(x)=x^5-x^4-13x^3+x^2+12x$.

\item % Talvine lahtine võistlus 2010, noorem rühm
Find the value of $x^5+2x^2-4x+2010$ given that $x^3+2x+2=0$.

\item
How many points do we need to uniquely define a polynomial of degree $n$?

\item
The roots for polynomial $P_2(x)=ax^2+bx+c$ are $x_1$ and $x_2$. Find the coefficients for third order polynomial which has roots $x_1^2$, $x_2^2$ and $x_1x_2$.

\item % PSS 24
In $x^3+px^2+qx+r$ one root is the sum of the two others. Find the relationship between $p$, $q$ and $r$.

\item % http://eqworld.ipmnet.ru/en/solutions/ae/ae0106.pdf
Find the roots of the polynomial $P(x)=ax^4+bx^3+cx^2+bx+a$

\item %PSS 31
Polynomial with integer coefficients $ax^3+bx^2+cx+d$ has $ad$ odd and $bc$ even. Show that at least one zero of the polynomial is irrational.

\item % PRoblem solving strategies 72
Polynomial of degree $n$ with non-negative coefficients and leading coefficient $1$ and constant term $1$ has $n$ real roots. Prove that $P(2)\geq 3^n$.

\end{enumerate}

\end{document}


\lab{Python}{Generators}{Generators}
\label{lab:Python_Generators}

Lists, tuples, sets, and dictionaries are examples of sequences in Python.
Often it is useful to visit all the elements of a sequence.  This process of visiting
each element of a sequence once is called \emph{iteration}.
Each of the Python types we have mentioned define their own iterators.
You use them every time you execute the statement \li{for <elem> in <list>}.
Python has a special type of object called a \emph{generator}.
Similar to an iterator, a generator returns a sequence of values.
The important difference is that a generator computes the next value in the sequence and returns it.
It never has to store all of the values in the sequence.
A good way to think of this is to remember: \emph{Iterators return their values while generators yield their values}.
In fact, Python uses the \li{yield} keyword to define a generator.
Let's illustrate the difference between iterating and generating by re-implementing Python's \li{range} function
as both an iterator and a generator.
\begin{lstlisting}
def range_iter(start, stop, step=1):
    i = start
    r = []
    while i < stop:
       r.append(i)
       i = i + step
    return r
\end{lstlisting}
\begin{lstlisting}
def range_gen(start, stop, step=1):
    i = start
    while i < stop:
        yield i
        i = i + step
\end{lstlisting}
The two functions look very similar.  But, you will soon see that they behave very differently.
The first function, when executed will immediately build a list, one element at a time until the done
at which point it returns the entire list.  On the author's computer, this function takes about 1.43ms
to build and return a list of 10000 elements.  The second function is only marginally better at
returning the 10000 elements in approximately 1.09ms.
This gap between the two functions grows much wider as the inputs are increased.
Where the real difference in the two functions lies is in the time the function takes to execute when
it is first called.  The first function requires 1.43ms to execute the first time.  The second function,
a mere .00000041ms!  The reason for this is the first function calculates its results and returns them all at once.  The second function only creates a \emph{generator} object.  This object has several methods.
The most important and most useful method is the \li{next()} method.  Every time this method is called,
the generator resumes from its previous state and computes the next value in the sequence. Each time \li{yield} statement is executed, the generator is effectively suspended until \li{next()} is called again.
When the generator has finished executing, a \li{StopIteration} exception is raised the generator terminates.
You can send values and throw exceptions to the generators via the \li{send()} and \li{throw()} methods respectively.
For more information on these methods we refer to PEP 342 (Python Enhancement Proposal 342).

When you are writing \li{for} loops, you should be sure to use a generator whenever it is reasonable to do so.
The \li{xrange()} function that is built into Python is a generator based version of \li{range()} and is generally faster when used in \li{for} loops.

When are generators helpful? If you encounter the situations below, consider trying to solve
your problem using generators.  Generators have proven to be very useful in these situations.
\begin{itemize}
\item \emph{Iterating through only part of a sequence.}
It is inefficient to create an entire sequence if we know that we will not need all of it.
Representing the sequence as a generator could possibly avoid excess memory use and excess computation.
\item \emph{Iterating through a sequence once.} Consider the statement
\li{sum([i for i in range(1000) if i\%2 == 0])}.
We are creating two sequences just to iterate through them once and never use them again.
We can make this more efficient using generators.
\li{sum(i for i in xrange(1000) if i\%2 == 0)}
Notice that we have used syntax similar to a list comprehension.
The line \li{(i for i in xrange(1000) if i\%2 == 0)} will define a generator object similar to the list made by \li{[i for i in xrange(1000) if i\%2 == 0]}.
The solution using generators will often execute faster, but it will almost always be more memory efficient.  Consider using generators for any function that reduces a sequence to a single value.  Examples of such functions are \li{min()}, \li{max()}, and \li{sum()}.
\item \emph{Calculating large sequences.}  The sequence must be stored somewhere in computer memory.
If the sequence is large, we could exhaust all available memory.
\item \emph{Calculations involving infinite sequences.}  Pre-computed sequences are necessarily finite.  We cannot create a list that stores all natural numbers, but we can create a generator that returns the next natural number every time it is called.
\end{itemize}

\section*{Combinations}
Combinations are subsets of a set.  The power set of a set, $S$, is the set of all combinations
of elements in $S$.  The cardinality of the the power set is $2^{\abs{S}}$.  Clearly, if our
set is of any appreciable size, the power set will be much larger.  Let's look at one method for
generating the power set of a set.

We can generate the power set of a set by representing each element in a set by one bit.
If a bit is 0, then we do not include the element in the current combination.  If a
bit is 1, the element is included in the combination.
Using this representation we can count through the power set of any finite set $A$, by counting from 0 to $2^{\abs{A}}-1$, looking at the binary representation of the number, and then including or excluding elements based on the value of each bit.

\begin{problem}
Write a function that will accept a list and return a list of all the combinations
of elements of that list.
\end{problem}

This is, however, an inefficient way to generate combinations.  Ideally, we would want
to generate the next combination by modifying the previous combination.  However, with
our current method of counting in binary, we are rebuilding the combination from scratch
each time.  For example it we have a combination represented by \texttt{01111}, the next
combination would be \texttt{10000}.  We added one element and removed four elements!
It can be useful to get our next combination by changing the previous combination as little as possible.
Fortunately, there is a really neat way of doing this--reflected binary codes or Gray codes.
A Gray code is a sequence of binary numbers where each new number is generated by
changing the previous code by exactly one bit.  Geometrically, we can think of it as
traversing a unit cube by moving only along the edges of the cube.
Grey codes are used frequently in error correction schemes.

A Gray code is constructed as follows.
\footnote{Brualdi, Richard Brualdi. \emph{Introductory Combinatorics}. New Jersey: Pearson, 2009}
The Gray code of order $n$ can be calculated using recursion in the following way:
\begin{enumerate}
\item The Gray code of order 1 is 0, 1.
\item Compute the Gray code of order $n-1$.  Write the binary numbers in a new list and then write them again in reverse order on the end of the same list (reflect them so that the last Gray code or order $n-1$ is first and the first Gray code of order $n-1$ is last).
\item Prepend a 0 to the first half of the list and prepend a 1 to the remaining half of the list.
\end{enumerate}
While this is a simple algorithm to describe, it is not very efficient.
To generate a Gray code of order $n$, we have to generate all previous Gray codes.
If we were to calculate the Gray codes up to order 6, we would calculate the Gray code of order 6 once, order 5 twice, order 4 three times, order 3 four times, and order 2 five times!
Fortunately, there exists another way to compute Gray codes of order $n$ much more efficiently.
The algorithm is given in Brualdi's book.
We first note that a Gray code of order $n$ is of length $2^n$ with each code of length $n$.
We also observe that the reflected gray codes always begin with $0\dots0$ and terminate with $10\dots0$.
\begin{enumerate}
\item Start with $0\dots0$ ($n$ zeros).  This is the current binary number.
\item Sum the digits of the current binary number.
\begin{enumerate}
\item If the sum of the digits is even, add 1 to the last digit mod 2.
This becomes our new current Gray and we go back to step 1.
\item If the sum of the digits is odd, find the rightmost non-zero digit and add 1 to the digit immediately to the left of it.
This becomes our new current number and we go back to step 1.
\item If the rightmost non-zero digit of the number is the first one you have reached the end of the Gray code.
\end{enumerate}
\end{enumerate}

\begin{problem}
\label{prob:brualdi_gray}
Implement the algorithm above for calculating the Gray code of order $n$.
Your implementation must function as a generator.
\end{problem}

\begin{problem}
\label{prob:changed_elem_gray}
Gray codes are an ordering of the power set such that any consecutive subsets differ by exactly one element.  Write a generator that will only return the change required to arrive at the next subset in the ordering.  Each time the generator is called,
it should return a single element and a boolean value that determines if that element should be
added or removed from the set to obtain the next Gray code.  This function should be able to accept any set with arbitrary elements.
\end{problem}

\section*{Itertools}
There is a very powerful library in the Python Standard Library that is built around the concept
of generators and iterators.  The functions in \li{itertools} are designed to be used as
building blocks in larger functions.  These functions are fast and memory efficient.
We encourage you to view the documentation for \li{itertools} on the Python website.  Itertools has a much more general and much faster solution for generating combinations and permutations of a set.
The generator that we have written as a solution to \ref{prob:changed_elem_gray} is especially useful when we are interested in the changed elements between successive subsets, or when we want to continuously update a single data structure using the elements of a Gray code.
The benefits of such an approach would be particularly apparent in cases where it is costly to add or remove items from the data sructure.
The documentation for \li{itertools} also includes various recipes for other useful generator functions.
Feel free to look through the documentation \url{http://docs.python.org/2/library/itertools.html}.

\begin{problem}
\label{prob:subblocks}
Write a generator function that will evenly split a 1-dimensional array into sub-blocks.
Your function should be capable of returning sub-blocks that could possibly overlap.
The function should accept as arguments: a 1-dimensional array, a block width, and an optional offset.
If the array cannot be evenly divided into sub-blocks, raise an error.
For example, if I want to divide an array\li{X} with 9 elements into subblocks of 3 elements with a step of 2 between each block, my generator would return the values \li{X[0:3]}, \li{X[2:5]}, \li{X[4:7]}, and \li{X[6:9]}.
It would raise an error if I were to ask for that sort of stepping on an array with 10 entries since the last item could not be included in any subblock.
\end{problem}

\begin{problem}
Expand your solution to Problem \ref{prob:subblocks} to work with 2-dimensional
arrays.  You may implement them as separate functions if you wish.
\end{problem}


\label{sec:data-fitting}

We can fit a triangular mesh, $\M$, to a set of data points, $\{\d_i; i=0 \ldots n-1\}$
by minimizing the sum of the $l_2$ distances from the points to the mesh:
\begin{equation}
f(\M) = \sum_{i=0}^{n-1} \| \d_i - \Pr_\M (\d_i) \|^2 ,
\end{equation}
where $\Pr_\M (\d_i)$ is the point on $\M$ closest to $\d_i$,
that is, the {\em projection} of $\d_i$ on $\M$.
Note that $f:\Reals^{3n} \mapsto \Re$,
where $n$ is the number of vertices in $\M$.

We compute $\Pr_\M (\d)$ by minimizing  $\| \d - \Pr_\s (\d) \|^2$
over all simplices $\s \in \M$.
We need only consider the faces of $\M$,
those edges not in any face,
and the vertices not in any edge,
because the closest point on a face must at least
as the closest point on any of its edges,
and similarly for vertices.
More generally, spatial binning of the simplices and data can greatly
reduce the number of simplices that need to be examined.

The following sections consider the projection of a single
data point $\d$ on a mesh $\M$,
and the gradient of the squared distance,
as a function of the vertex positions $\p(\v)$.

Unfortunately, derivatives of the distance function are not continuous.
Second derivative discontinuities occur
when a data point is on the boundary
of a 'watershed' region, the set of points
projecting on a vertex or the interior of an edge or face.
Gradient discontinuities are encountered
when a data point is equidistant from 2 distinct closest mesh points.

\subsection{Distance to vertex}
\label{sec:Distance-to-vertex}

Let $\p = \p(\v)$ be the position of a particular vertex $\v$,
and $\d$ the 3d data point.
It follows from equation \ref{eq:l2-gradient} that
\begin{equation}
\label{eq:vertex-distance-gradient}
\Gc{\p}{\| \p - \d \|^2}{\q} = 2 ( \q - \d ).
\end{equation}

The distance to the nearest vertex in a set of vertices $\V$ is:
$\min_{\v \in \V} \| \p(\v) - \d \|^2$.
If $\v^{\mathrm min}$ is the minimizing vertex,
and
$\p^{\mathrm min}$ its position,
then the partial gradient with respect
to the position of any other vertex is zero,
and the partial gradient with respect to $\p^{\mathrm min}$
is given in equation \ref{eq:vertex-distance-gradient}.
Note that the gradient is only defined and continuous
while $\d$ is within the interior of the
Voronoi regions surrounding the vertices.

\subsection{Distance to edge}
\label{sec:Distance-to-edge}

Let the edge $\e$ have end points $\p = (\p_0, \p_1) \in \Reals^6$.
We can write the projection of a data point $\d$ on $\e$ as:
\begin{equation}
\Pr_\p (\d) = b_0(\p) \p_0 + b_1(\p) \p_1
\end{equation}
where
\begin{eqnarray}
b_0(\p) & = &
\min\left(0,\max\left(1,
{{ (\d - \p_1) \bullet (\p_0 - \p_1) }
\over
{ \| \p_0 - \p_1 \|^2 }
}\right) \right) \\
b_1(\p) & = & 1 - b_0(\p)
\nonumber
\end{eqnarray}

\begin{eqnarray}
\label{eq:edge-distance-gradient-derivation}
\De{\p_0}{ \| \Pr_{\p} (\d) - \d \|^2 }{\q}
& = &
2 \left( \Pr_{\q} (\d) - \d \right)^\dagger
\De{\p_0}{\Pr_{\p} (\d) }{\q}
\\
& = &
2 \left( \Pr_{\q} (\d) - \d \right)^\dagger
\De{\p_0}{\left[ b_0(\p)\p_0 + b_1(\p)\p_1 \right]}{\q}
\nonumber \\
& = &
2 \left( \Pr_{\q} (\d) - \d \right)^\dagger
\De{\p_0}{\left[ b_0(\p)\p_0 + (1 - b_0(\p))\p_1 \right]}{\q}
\nonumber \\
& = &
2 \left( \Pr_{\q} (\d) - \d \right)^\dagger
\De{\p_0}{\left[ b_0(\p)(\p_0 - \p_1) \right]}{\q}
\nonumber \\
& = &
2 \left( \Pr_{\q} (\d) - \d \right)^\dagger
\left[ b_0(\q) \I + (\q_0 - \q_1) \otimes \Gc{\p_0}{b_0(\p)}{\q} \right]
\nonumber
\end{eqnarray}
Because $\left( \Pr_{\q} (\d) - \d \right)$ is orthogonal to
$\left( \q_0 - \q_1 \right)$, we get:
\begin{eqnarray}
\label{eq:edge-distance-gradient}
\Gc{\p_0}{ \| \Pr_{\p} (\d) - \d \|^2 }{\q}
& = & 2 b_0(\q) \left[ \Pr_{\q} (\d) - \d \right]
\\
\Gc{\p_1}{ \| \Pr_{\p} (\d) - \d|^2 }{\q}
& = & 2 b_1(\q) \left[ \Pr_\q (\d) - \d \right]
\nonumber
\end{eqnarray}

As in the vertex case,
the distance to the nearest edge in a set of edges $\E$ is:
\begin{equation}
\| \Pr_{\E} (\d) - \d|^2 = \min_{\e \in \E} \| \Pr_{\p(\e)}(\d) - \d \|^2
\end{equation}
If $\e^{\min}$ is the minimizing edge,
$\v_0^{\min}$ and $\v_1^{\min}$ its vertices,
and $\p_0^{\min}$ and $\p_1^{\min}$
the corresponding endpoints,
then the partial gradient with respect to
the position of any
other vertex is zero,
and the partial gradient with respect to $\p_0^{\min}$ and $\p_1^{\min}$
is given in equation \ref{eq:edge-distance-gradient}.

The total gradient is defined and continuous
when $\d$ is within the union of the watershed regions
of $\e^{\min}$ and its vertices.
It is also continuous where the watershed of one of the vertices
meets the watershed of any of the edges containing that vertex.
It is not if $\d$ lies on the boundary of the
watershed of $\e^{\min}$ and the watershed of an
edge with which it does not share a vertex.

\subsection{Distance to face}
\label{sec:Distance-to-face}

Let the face $\f$ have corner points $\p = (\p_0, \p_1, \p_2) \in \Reals^9$.
As in the edge case,
we can write the projection of a data point $\d$ on $\f$
in terms of the barycentric coordinates as:
\begin{equation}
\Pr_\p (\d) = b_0(\p) \p_0 + b_1(\p) \p_1 + b_2(\p) \p_2,
\end{equation}
and, by an argument simlar to that used in
equation \ref{eq:edge-distance-gradient-derivation},
we can show that
\begin{eqnarray}
\label{eq:face-distance-gradient}
\Gc{\p_0}{ \| \Pr_{\p} (\d) - \d \|^2 }{\q}
& = & 2 b_0(\q) \left[ \Pr_{\q} (\d) - \d \right]
\\
\Gc{\p_1}{ \| \Pr_{\p} (\d) - \d|^2 }{\q}
& = & 2 b_1(\q) \left[ \Pr_\q (\d) - \d \right]
\nonumber
\\
\Gc{\p_2}{ \| \Pr_{\p} (\d) - \d|^2 }{\q}
& = & 2 b_2(\q) \left[ \Pr_\q (\d) - \d \right]
\nonumber
\end{eqnarray}

Computing the barycentric coordinates for the projection
on a face (triangle) is slightly more complicated than
for an edge (line segment).

First center the problem by letting
$\v = \p - \p_0$,
$\v_1 = \p_1 - \p_0$, and $\v_2 = \p_2 - \p_0$.
Then compute the raw, unbounded barycentric coordinates
of the projection of $\p$ onto the plane
spanned by the triangle:
\begin{eqnarray}
r_0(\p) & = & 1 - r_1(\p) - r_2(\p)
\\
r_1(\p) & = & v \bullet {{\v_1 \perp \v_2} \over {\| \v_1 \perp \v_2 \|^2} }
\nonumber
\\
r_2(\p) & = & v \bullet {{\v_2 \perp \v_1} \over {\| \v_2 \perp \v_1 \|^2} }
\nonumber
\end{eqnarray}
To correctly bound the raw coordinates to numbers between 0 and 1,
we need to determine whether the projected point is in
the interior of the triangle, on one of the edges,
or on one of the vertices.

\begin{description}

\item[Vertex case:]
If any 2 of the $r_i$ are negative,
then $\p$ projects on the remaining vertex.
Set the 2 $b_i$ corresponding to the negative $r_i$
to 0 and the remaining $b_i$ to 1.

\item[Edge case:]
If any single 1 of the $r_i$ is negative,
then $\p$ projects on the opposite edge.
Set the $b_i$ corresponding to the negative $r_i$
to 0.
Go to \autoref{sec:Distance-to-edge} to see
how to compute the remaining barycentric coordinates
by projecting on the edge

\item[Interior case:]
If none of the $r_i$ is negative,
then $\p$ projects on the interior
and each $b_i = r_i$

\end{description}


\documentclass[main.tex]{subfiles}

\begin{document}

	\section{Band structure}\label{sec:band_structure}
	\subsection{Theory}
	In a crystal, the potential must necessarily be periodic in the unit cell, otherwise the discrete translational symmetry is broken. As such the following equation holds for any lattice vector $ \V{R} $:
	\begin{equation}\label{eq:band_period_V}
		V(\V{r}+\V{R}) = V(\V{r}).
	\end{equation}
	Due to this periodicity, the potential can also be expressed as a sum over reciprocal lattice vectors. To see this, we start with the potential operator:
	\begin{equation}
		\hat{V} = \infint V(\V{r}) \ket{\V{r}}\bra{\V{r}}\ud \V{r}.
	\end{equation}
	We insert two continuous identities for $ \V{k} $ and $ \V{k}' $, which is the equivalent of taking the Fourier transform of the potential:
	\begin{align}
		\hat{V} &= \infint V(\V{r}) \bb{ \infint \ket{\V{k}} \bra{\V{k}} \ud \V{k}  } \ket{\V{r}}\bra{\V{r}} \bb{\infint \ket{\V{k}'} \bra{\V{k}'} \ud \V{k}' } \ud \V{r} , \\
		&=\infint \infint \infint V(\V{r}) \ket{\V{k}} \braket{\V{k} | \V{r}} \braket{\V{r} | \V{k}'} \bra{\V{k}}  \ud\V{k}'   \ud\V{k}   \ud\V{r}.
	\end{align}
	The two middle brakets are $ \braket{\V{k}| \V{r}}= \sqrt{(2\pi)^3}\inverse e^{-i \V{k} \D \V{r}} $ and $ \braket{\V{r} | \V{k}'} = \sqrt{(2\pi)^3}\inverse e^{i \V{k}' \D \V{r}} $, where $ \sqrt{(2\pi)^3}\inverse $ is a normalization factor. This then becomes
	\begin{align}
		\hat{V} &= \frac{1}{(2\pi)^3}\infint \infint  \infint  V(\V{r}) e^{-i(\V{k}-\V{k}') \D \V{r}} \ket{\V{k}} \bra{\V{k}'} \ud\V{k}'   \ud\V{k}   \ud\V{r}.
	\end{align}
	Next we use the now familiar trick of letting $ \V{r} = \V{R} + \V{x} $:
	\begin{equation}
		\hat{V} = \frac{1}{(2\pi)^3}\infint \infint  \bb{\sum_{\V{R}} e^{i(\V{k}'-\V{k}) \D \V{R}}}  \int_{\substack{\text{unit-} \\\text{cell}}} V(\V{x})\  e^{-i (\V{k}-\V{k}') \D \V{x}} \ket{\V{k}} \bra{\V{k}'} \ud \V{x} \ud\V{k}'   \ud\V{k}.
	\end{equation}
	This sum, like before, equals 0 for $ \V{k}'-\V{k} \neq \V{G} $. However, when $ \V{k}'-\V{k} = \V{G} $ this sum becomes infinite (given the infinite amount of lattice points), with a prefactor of $ (2\pi)^3 / a^{-3} $ \cite{simon}. This then gives
	\begin{align}
		\hat{V} &= \frac{1}{a^3} \sum_{G} \infint \infint \delta(\V{k}'-\V{k}-\V{G}) \int_{\substack{\text{unit-} \\\text{cell}}} V(\V{x})\  e^{-i \V{G} \D \V{x}} \ket{\V{k}} \bra{\V{k}'} \ud \V{x} \ud\V{k}'   \ud\V{k}, \\
		&= \sum_{G} V_{\V{G}}  \infint \ket{\V{k}} \bra{\V{k}-\V{G}} \ud \V{k}, \quad V_{\V{G}} = \frac{1}{a^3} \int_{\substack{\text{unit-} \\\text{cell}}} V(\V{x}) e^{-i\V{G} \D \V{x}} \ud \V{x},
	\end{align}
	where $ V_{\V{G}} $ is the Fourier transform of the potential, in $ \V{G} $, over the unit cell, which is the same as the structure factor for scattering, except for a normalisation factor. Now, inserting two additional identities, but this time for $ \V{r} $ and $ \V{r}' $:
	\begin{align}
		\hat{V} &=\sum_{G}  V_{\V{G}} \infint \bb{\infint \ket{\V{r}} \bra{\V{r}} \ud \V{r}} \ket{\V{k}} \bra{\V{k}+ \V{G}} \bb{\infint \ket{\V{r}'} \bra{\V{r}'} \ud \V{r}'} \ud \V{k}, \\
		&=\sum_{G} V_{\V{G}} \infint  \infint \infint \sqrt{(2\pi)^3}\inverse e^{i \V{k} \D \V{r}} \sqrt{(2\pi)^3}\inverse e^{-i (\V{k}+\V{G}) \D \V{r}'} \ket{\V{r}} \bra{\V{r}'} \ud \V{k} \ud \V{r} \ud \V{r}', \\
		&=\frac{1}{(2\pi)^3} \sum_{G} V_{\V{G}} \infint \infint \infint  e^{-i \V{G} \D \V{r}'} e^{i \V{k} \D (\V{r}-\V{r}')} \ket{\V{r}} \bra{\V{r}'} \ud \V{k} \ud \V{r} \ud \V{r}'.
	\end{align}
	We can get rid of two of these integrals, if we use the fact that \cite{riley}
	\begin{equation} \label{eq:delta_funct}
		\delta(\V{r}-\V{r}') = \frac{1}{(2\pi)^3} \infint  e^{i \V{k} \D (\V{r}-\V{r}')} \ud \V{k},
	\end{equation}
	as we then get
	\begin{equation}
		\hat{V} = \sum_{\V{G}} V_{\V{G}} \infint e^{-i \V{G} \D \V{r}} \ket{\V{r}} \bra{\V{r}} \ud \V{r},
	\end{equation}
	which, if the prefactors and the sum is taken inside the integral, is the same form as we started with! Thus we get our desired result of
	\begin{equation}
		V(\V{r}) =  \sum_{\V{G}} V_{\V{G}} \,e^{i \V{G} \D \V{r}}, \quad V_{\V{G}} = \frac{1}{a^3} \int_{\substack{\text{unit-} \\\text{cell}}}  V(\V{x})\  e^{-i \V{G} \D \V{x}} \ud \V{x},
	\end{equation}
	where we have made the substitution $ \V{G} \to -\V{G} $ in the expression for $ V(\V{r}) $. This does not change the sum index as all terms are still included, and as we shall see below, we have $ V_{\V{G}}= V_{-\V{G}} $, so the above equality holds.

	This form allows us to write the Schrödinger equation in a form where the dispersion relation is easily calculated numerically. First we Fourier transform the equation:
	\begin{equation}
		\infint e^{-i \V{k} \D \V{r}} \bb{\frac{\V{p}^2}{2m} + V(\V{r})}\psi(\V{r}) \ud \V{r}= \infint e^{-i\V{k} \D \V{r}}\, E\, \psi(\V{r}) \ud \V{r} = E \,\tilde{\psi}(\V{k}).
	\end{equation}
	The kinetic energy term is just
	\begin{equation}
		\mathcal{F}\bb{\frac{\V{p}^2}{2m} \psi(\V{r})} = -\frac{\hbar^2}{2m}\infint e^{-i \V{k} \D \V{r}}\, \nabla^2\psi(\V{r}) \ud \V{r} = \frac{\hbar^2 \V{k}^2}{2m}\tilde{\psi(\V{k})},
	\end{equation}
	whilst the potential energy term is
	\begin{align}
		\mathcal{F}[V(\V{r}) \psi(\V{r})] &= \infint e^{-i \V{k} \D \V{r}}\, V(\V{r})\, \psi(\V{r}) \ud \V{r} = \infint e^{-i \V{k}\D \V{r}} \bb{\sum_{\V{G}} e^{i \V{G} \D \V{r}}\, V_{\V{G}} }\psi (\V{r}) \ud \V{r}, \\
		&= \sum_{\V{G}} V_{\V{G}} \infint e^{-i(\V{k}-\V{G})\D \V{r}}\, \psi(\V{r}) \ud \V{r},
	\end{align}
	where the integral is just the Fourier transform of the wave function, in $ \V{k}-\V{G} $:
	\begin{equation}
		\mathcal{F}[V(\V{r}) \psi(\V{r})] = \sum_{\V{G}} V_{\V{G}} \, \tilde{\psi}(\V{k}-\V{G}).
	\end{equation}
	With this expression, the whole equation becomes
	\begin{equation}
		\sum_{\V{G}} \bb{\frac{\hbar^2 \V{k}^2}{2m} \delta_{\V{G},0} + V_{\V{G}} } \tilde{\psi}(\V{k}-\V{G}) = E\, \tilde{\psi}(\V{k}).
	\end{equation}
	This equation gives the energy for a single value of $ \V{k} $, by relating the state $ \tilde{\psi}(\V{k}) $ to all other states with the same crystal momentum. If we then consider all the different equations for states with the same crystal momentum $ \tilde{\psi} (\V{k}-\V{G}) $, we can describe them all as a matrix equation, where the eigenvalues are the energies for the state with wave vector $ \V{k} $, in all of the different bands. The first (lowest) eigenvalue is thus the energy of $ \tilde{\psi} (\V{k}) $ in the lowest band.

	To calculate the dispersion relation for a given lattice and potential we then need to find the eigenvalue of the above matrix equation for a range of different $ \V{k} $. However, due to the unbounded nature of $ \V{G} $, the matrix and vector will both have an infinite amount of elements. For the purposes of the program we need to only allow some set of $ \V{G} $, making the matrix and vector finite dimensional.

	This can be thought of through the lens of perturbation theory. If we have a free particle (corresponding to no allowed value of $ \V{G} $), we just get a $ 1\times 1$ ``matrix'', whose eigenvalue trivially is the energy of a free particle. Adding the potential for $ \V{G} = 0 $ gives the first order perturbation, shifting the state by some constant energy (the matrix equation still only has one allowed state). Allowing the set of next smallest values for $ \V{G} $ would then constitute a second order perturbation, where the particle is allowed to scatter into these states. Further allowing a larger set of $ \V{G} $ will give a more accurate calculation of the dispersion relation for the particle, until finally it becomes exact when the full, infinite spectrum of values for $ \V{G} $ is included.

	\subsection{Implementation}
	In the above calculations we have assumed a three dimensional system. In the following however, we will restrict ourselves to two dimensions and a square lattice with lattice spacing $ a $. The reciprocal lattice vectors $ \V{G} $ can then be indexed with the coefficients $ m_1 $ and $ m_2 $, as $ \V{G} = \frac{2\pi}{a} (m_1 \U{x} + m_2 \U{y}) $. This also means that the sum over $ \V{G} $ can be expressed as a double sum over $ m_1 $ and $ m_2 $.

	To describe the matrix it is helpful to first describe the vector in the equation. Let us call this $ \ket{\psi} $. This consists of $ \tilde{\psi} (\V{k}+\V{G}) $ for all the allowed $ \V{G} $. Let us also call these $ \psi[m_1, m_2] $:
	\begin{equation}
		\ket{\psi} = \begin{pmatrix}
		\vdots \\ \tilde{\psi}(\V{k}-\V{G}_1) \\ \tilde{\psi}(\V{k}) \\ \tilde{\psi}(\V{k}+\V{G}_1) \\ \vdots
		\end{pmatrix} = \begin{pmatrix}
		\vdots \\ \psi\coef{0, -1}\\ \psi\coef{0, 0} \\ \psi\coef{0, 1} \\ \vdots
		\end{pmatrix},
	\end{equation}
	where $ \V{G}_1 = \frac{2\pi}{a} \U{y}$, for example. Correspondingly we write $ V_{\V{G}} $ as $ V\coef{m_1, m_2} $.

	The matrix in the above equation can be split into two different matrices: One for the kinetic energy $ T $, which is just a diagonal matrix, and one for the potential energy $ V $. The diagonal elements of the kinetic energy matrix are just the energy of the state, if no potential was present:

	\begin{equation}
		T = \frac{\hbar^2}{2m} \begin{pmatrix}
			\ddots	& 		 			&			& 					& \\
					& (\V{k}-\V{G}_1)^2	& 			& 					& \\
					& 	 				& \V{k}^2	& 					& \\
					&					&			& (\V{k}+\V{G}_1)^2	& \\
					&					&			&					& \ddots
		\end{pmatrix}.
	\end{equation}
	The potential energy matrix is a bit more complicated. It is best described with an example. Say we allow $ m_1, m_2 \in \{-1,0,1\} $. The sum still has an infinite amount of terms, but we only allow 9 of these in our matrix equation. These terms have pairs of coefficients for $ \psi $ that are in the ordered list
	\begin{equation}
		[m_1, m_2] \in \{ [-1, -1], [-1, 0], [-1, 1], [0, -1], [0, 0], [0, 1], [1,-1], [1, 0], [1,1]  \}.
	\end{equation}

	If we then calculate (part of) the row equation for $ m_1 = -1, m_2 = 0 $ (which is the second row of the potential matrix) we get
	\begin{align*}
		E \psi\coef{-1,0} = {} & \sum_{m'_1 =  -\infty}^{\infty}\sum_{m'_2 = -\infty}^{\infty} V\coef{m'_1, m'_2} \ \psi\coef{-1-m'_1, -m'_2}, \\
		= {} & \dots +  V\coef{-2, -1}\psi\coef{1,1} + V\coef{-2, 0} \psi\coef{1, 0} + V\coef{-2, 1}\psi\coef{1,-1} +  \dots  \\
		& + V\coef{-1, -1} \psi\coef{-2, 1} + V\coef{-1, 0} \psi\coef{-2, 0} + V\coef{-1, 1} \psi\coef{-2, -1} + \dots\\
		& + V\coef{0, -1} \psi\coef{-1, 1} + V\coef{0, 0} \psi\coef{-1, 0} + V\coef{0, 1}\psi\coef{-1, -1} + \dots\\
		&+ V\coef{1, -1} \psi \coef{0,1} + V\coef{1, 0} \psi \coef{0, 0} + V\coef{1,1} \psi \coef{0, -1} + \dots.
	\end{align*}
	Now, the 4th to 6th  shown terms contain ``disallowed'' states, i.e. they have coefficients for $ \psi $ outside of the allowed range. The other 9 shown terms are ``allowed'' states however.

	As for the ordering of the rows of the matrix: $ [-1, -1] $ is the first pair of coefficients in the ordered set of coefficients. Because of this $ V\coef{0, 1} \psi\coef{-1,-1}$ is the first term in the matrix equation, and $ V\coef{0, 1} $ is then the first element in the matrix row. The whole row is then:
	\begin{equation}
		\begin{pmatrix}
			V\coef{0, 1} &  V\coef{0, 0} & V\coef{0, -1} & V\coef{1, 1} & V\coef{1, 0} & V\coef{1,-1} & V\coef{-2, 1} & V\coef{-2, 0} & V\coef{-2, -1}
		\end{pmatrix}.
	\end{equation}
	The algorithm for constructing the potential matrix is as follows:
	\begin{enumerate}
		\item Choose a range of values for $ m_1 $ and $ m_2 $ and arrange them in an ordered set.
		\item For each pair of coefficients in this set, ($ [m_1, m_2] $, corresponding to some row), calculate $ [m_1-m_1', m_2-m_2'] $, where $ [m_1', m_2'] $ are all the pairs of coefficients from the same set, corresponding to the columns of the matrix.
		\item This new pair of coefficients, $ [m_1-m_1', m_2-m_2'] $, will be coefficients for the potential at the corresponding matrix element: $ V\coef{m_1-m_1', m_2-m_2'} $. (The corresponding matrix element is the one with row/column corresponding to the position of $ [m_1, m_2]/[m_1', m_2'] $ in the ordered set).
	\end{enumerate}
	With this algorithm in mind, we can see a couple of characteristics of the matrix:
	\begin{itemize}
		\item The diagonal elements are all $ V\coef{0,0} $, corresponding to the state not being scattered into any other state.
		\item The matrix is Hermitian, if $ V\coef{m_1, m_2} = V\coef{-m_1, -m_2}$, which is the same condition one gets when using perturbation theory to calculate the band structure in the Nearly Free Electron Model, see section \ref{sec:nearly_free}. This is all very fortunate since the potential matrix must necessarily be hermitian for the whole Hamiltonian to be hermitian, which it has to be, since it corresponds to an observable quantity, namely energy.
	\end{itemize}
	On the last characteristic: Since the potential is real and symmetric (rather, it is even about any lattice point $ \V{R} $, in both the $ x $ and $ y $-direction), it is guaranteed that $ V\coef{m_1, m_2} = V\coef{-m_1, -m_2}$. This is because the Fourier transform of a real and even function is also real and even \cite{riley}. As such, for any periodic potential (that is even in both $ x $ and $ y $ about lattice points) the potential matrix, and therefore also the whole Hamiltonian is hermitian!

	Specific potentials implemented in the program include a two dimensional Dirac Comb, and a harmonic potential:
	\begin{equation}
		V_{\text{dirac}}(\V{r}) = V_0 a^2 \sum_{\V{R}} \delta(\V{r}-\V{R}), \quad V_{\text{harmonic}}(\V{r}) = V_0 \bb{\cos\pp{\frac{2\pi}{a} x} + \cos\pp{\frac{2\pi}{a} y}},
	\end{equation}
	where the $ a^2 $ in the Dirac comb potential arises from the fact that the Dirac delta function carries units $ \e{m}^{-2} $ (in general it carries units that are inverse of its argument as can be seen from Eq. \eqref{eq:delta_funct} by dimensional analysis). Now, $ V\coef{m_1, m_2} $ for the Dirac Comb potential is easily calculated. We let the unit cell run from $ -a/2 $ to $ a/2 $ in both $ x $ and $ y $, to make sure the delta functions are well within the integration limits. Then it just becomes:
	\begin{equation}
		V\coef{m_1, m_2} = V_0 \frac{a^2}{a^2} \int_{-a/2}^{a/2} \int_{-a/2}^{a/2} e^{i 2\pi (m_1 x + m_2 y)/a} \delta(\V{r}) = V_0 \ud x \ud y,
	\end{equation}
	and the potential matrix is just a matrix full of ones, scaled by $ V_0 $. For the harmonic potential we let the unit cell run from $ 0 $ to $ a $ in both directions and get
	\begin{equation}
		V\coef{m_1, m_2} = \frac{V_0}{a^2} \int_{0}^{a} \int_0^a e^{i 2\pi(m_1x + m_2y)/a} \bb{\cos\pp{\frac{2\pi}{a} x} + \cos\pp{\frac{2\pi}{a} y}} \ud x \ud y.
	\end{equation}
	This can be split into two integrals, $ I_1 $ and $ I_2 $. The first is
	\begin{align}
		I_1 &=  \frac{V_0}{a^2} \int_{0}^{a} \int_0^a e^{i 2\pi(m_1x + m_2y)/a}\cos\pp{\frac{2\pi}{a} x} \ud x \ud y , \\
		&= \frac{V_0}{2a^2} \int_{0}^{a} e^{i 2\pi m_2 y / a} \ud y\ \int_{0}^{a} \ e^{i 2\pi m_1 x/a} \bb{e^{i 2\pi x/a} + e^{-i 2\pi x/a}} \ud x, \\
		&= \frac{V_0}{2a^2} \int_0^a e^{i 2\pi m_2 y / a} \ud y\  \int_{0}^{a} \ud x\ \bb{e^{i 2\pi(m_1+1) x/a} + e^{-i 2\pi (m_1-1)x/a}} \ud x.
	\end{align}
	Now if $ m_2 = 0 $, the integrand in the $ y $-integral is just 1, and the whole integral evaluates to $ a $. If $ m_2 \neq 0 $ this is not the case. However, the antiderivative evaluated at both ends is the same, and the whole integral is 0:
	\begin{equation}
		\int_{0}^{a} e^{i 2\pi m_2 y/a} \ud y = \frac{a}{i 2\pi m_2} \bb{e^{i2\pi m_2 y /a}}_0^a = 0.
	\end{equation}
	The same goes for the integrals in $ x $, but with $ m_1 \pm 1 $ instead of $ m_2 $. As such
	\begin{equation}
		I_1 = \frac{V_0 a^2}{2 a^2} \bb{\delta_{m_2,0} (\delta_{m_1, 1} + \delta_{m_1, -1})},
	\end{equation}
	with a similar result for $ I_2 $. $ V\coef{m_1, m_2} $ is then
	\begin{equation}
		V\coef{m_1, m_2} = \begin{cases}
			\frac{V_0}{2} 	& \text{if } [m_1, m_2] \in \{[0,1], [0,-1], [1,0], [-1,0]\}, \\
			0					& \text{else}.
		\end{cases}
	\end{equation}
	And the potential energy matrix is
	\begin{equation}
		V_{\text{harmonic}} = \frac{V_0}{2} \begin{pmatrix}
		0 & 1 & 0 & 1 & 0 & 0 & 0 & 0 & 0 \\
		1 & 0 & 1 & 0 & 1 & 0 & 0 & 0 & 0 \\
		0 & 1 & 0 & 0 & 0 & 1 & 0 & 0 & 0 \\
		1 & 0 & 0 & 0 & 1 & 0 & 1 & 0 & 0 \\
		0 & 1 & 0 & 1 & 0 & 1 & 0 & 1 & 0 \\
		0 & 0 & 1 & 0 & 1 & 0 & 0 & 0 & 1 \\
		0 & 0 & 0 & 1 & 0 & 0 & 0 & 1 & 0 \\
		0 & 0 & 0 & 0 & 1 & 0 & 1 & 0 & 1 \\
		0 & 0 & 0 & 0 & 0 & 1 & 0 & 1 & 0
		\end{pmatrix}.
	\end{equation}
	With this we have the full Hamiltonian matrix, and we can solve it for a range of $ \V{k} $ to get the dispersion relation for a particle in a square lattice.

	To calculate the dispersion relation, the program needs values for $ V_0 $, $ a $ and $ \hbar^2/2m $ along with the range of coefficients for $ \V{G} $. To make this more simple for the user, we define $ k_0 \equiv 2\pi/a $ (the same factor as in the scattering program) and $ E_0 \equiv \hbar^2 k_0^2/m $. These allow us to rewrite the equation in a dimensionless form:
	\begin{equation}\label{key}
		\sum_{\tilde{\V{G}}} \bb{\frac{\tilde{\V{k}}^2}{2} \delta_{\tilde{\V{G}},0} + \tilde{V}_{\tilde{\V{G}}}} \tilde{\psi}(\tilde{\V{k}} - \tilde{\V{G}}) = \tilde{E} \tilde{\psi}(\tilde{\V{k}}),
	\end{equation}
	with $ \tilde{\V{G}} \equiv \V{G}/k_0 = m_1 \U{x}+m_2 \U{y} $, $ \tilde{\V{k}} \equiv \V{k}/k_0 $ $ \tilde{V}_{\tilde{\V{G}}} \equiv V_{\V{G}} / E_0$ and $ \tilde{E} \equiv E/E_0 $. This also redefines the first Brillouin zone to be $ -1/2 \leq \tilde{k} \leq 1/2 $ in each direction.

	With this the Hamiltonian matrix is created. The program also takes a value for the number of points in the first Brillouin-zone in $ k $-space and creates linearly spaced points from $ -1/2 $ to $ 1/2 $.

	For each point in the Brillouin zone the program then creates the corresponding kinetic energy matrix, adds the potential energy matrix, and finds the eigenvalues for the resulting matrix. These values are stored in a multidimensional array (represented as a series of matrices, each with size $ n_k \times n_k $, where $ n_k $ is the number of points in the Brillouin zone, in each direction). When the energies for all coordinates are calculated, the lowest energies for each of the points (corresponding to the first matrix in the multidimensional array) will then be the dispersion of the particle in the first band.

	Next we aim to find the Fermi sea and surface, and not just the dispersion relation. For a crystal with monovalent atoms the electrons fill up half a band, corresponding to half the area of the first Brillouin zone (in two dimensions). To represent this we use the fact that we have discretised the Brillouin zone into $ n_k^2 $ points. The Fermi sea is then covered by the values of $ \V{k} $ which have an energy lower than the $ n_k^2/2 $'th lowest energy. So to plot only the Fermi sea for a 2D material, we order the energies in the band from lowest to highest, find the middle value, and only plot points that have an energy lower than, or equal to, this energy.

	\subsection{Examples}
	Say we just want to find the dispersion of a free particle. Then we just call the program with $ \tilde{V}_0 \equiv V_0/E_0 = 0 $. For good measure we can specify that the potential should be the Dirac potential and that we want the edges of the dispersion relation shown (to increase visibility on a printed page):
\begin{lstlisting}
Band_structure(V0=0,
			   potential='dirac',
			   edges=True)
\end{lstlisting}
	And if we want the high-potential limit of the harmonic potential (which happens to be $ \tilde{V}_0 = 1 $) we write
\begin{lstlisting}
Band_structure(V0=1,
			   edges=True)
\end{lstlisting}
	We do not need to specify that the potential should be harmonic, as this is the default argument. To be explicit we could add the argument \texttt{potential='harmonic'}.	These lines produce figures \ref{fig:band_structure_none} and \ref{fig:band_structure_strong} respectively.
	\begin{figure}[h]
		\centering
		\includegraphics[width=\linewidth]{figures/band_structure_none.pdf}
		\caption{The dispersion relation for a free particle in a square lattice of monovalent atoms, along with the Fermi surface.}
		\label{fig:band_structure_none}
	\end{figure}

	\begin{figure}[h]
		\centering
		\includegraphics[width=\linewidth]{figures/band_structure_strong.pdf}
		\caption{The dispersion relation of a particle in a strong harmonic potential, on a square lattice of monovalent atoms, along with the corresponding Fermi surface.}
		\label{fig:band_structure_strong}
	\end{figure}

\end{document}


\documentstyle[11pt,reduce]{article}
\title{{\tt FPS}\\
A Package for the\\
Automatic Calculation \\
of Formal Power Series}
\date{}
\author{Wolfram Koepf\\
	ZIB Berlin \\
	Email: {\tt  Koepf@ZIB.de}
\\
\\
	Present \REDUCE{} form by \\
	Winfried Neun \\
	ZIB Berlin \\
        Email: {\tt Neun@ZIB.de}}
\begin{document}
\maketitle

\section{Introduction}
This package can expand functions of certain type into
their corresponding Laurent-Puiseux series as a sum of terms of the form
\begin{displaymath}
\sum_{k=0}^{\infty} a_{k} (x-x_{0})^{m k/n + s}
\end{displaymath}
where $m$ is the `symmetry number', $s$ is the `shift number',
$n$ is the `Puiseux number',
and $x_0$ is the `point of development'. The following types are
supported:
\begin{itemize}
\item
{\bf functions of `rational type'}, which are either rational or have a
rational derivative of some order;
\item
{\bf functions of `hypergeometric type'} where $a(k+m)/a(k)$ is a rational
function for some integer $m$;
\item
{\bf functions of `explike type'} which satisfy a linear homogeneous
differential equation with constant coefficients.
\end{itemize}

The FPS package is an implementation of the method
presented in \cite{Koepf:92}. The implementations of this package
for {\sc Maple} (by D.\ Gruntz) and {\sc Mathematica} (by W.\ Koepf)
served as guidelines for this one.

Numerous examples can be found in \cite{Koepf:93a}--\cite{Koepf:93b},
most of which are contained in the test file {\tt fps.tst}. Many
more examples can be found in the extensive bibliography of Hansen \cite{Han}.

\section{\REDUCE{} operator {\tt FPS}}
The FPS Package must be loaded first by:
\begin{verbatim}
load FPS;
\end{verbatim}
{\tt FPS(f,x,x0)} tries to find a formal power
series expansion for {\tt f} with respect to the variable {\tt x}
at the point of development {\tt x0}.
It also works for formal Laurent (negative exponents) and Puiseux series
(fractional exponents). If the third
argument is omitted, then {\tt x0:=0} is assumed.

Examples: {\tt FPS(asin(x)\verb+^+2,x)} results in
\begin{verbatim}

         2*k  2*k             2  2
        x   *2   *factorial(k) *x
infsum(----------------------------,k,0,infinity)
        factorial(2*k + 1)*(k + 1)
\end{verbatim}
{\tt FPS(sin x,x,pi)} gives
\begin{verbatim}
                   2*k       k
        ( - pi + x)   *( - 1) *( - pi + x)
infsum(------------------------------------,k,0,infinity)
                factorial(2*k + 1)
\end{verbatim}
and {\tt FPS(sqrt(2-x\verb+^+2),x)} yields
\begin{verbatim}
            2*k
         - x   *sqrt(2)*factorial(2*k)
infsum(--------------------------------,k,0,infinity)
           k             2
          8 *factorial(k) *(2*k - 1)
\end{verbatim}
Note: The result contains one or more {\tt infsum} terms such that it does
not interfere with the {\REDUCE} operator {\tt sum}. In graphical oriented
REDUCE interfaces this operator results in the usual $\sum$ notation.

If possible, the output is given using factorials. In some cases, the
use of the Pochhammer symbol {\tt pochhammer(a,k)}$:=a(a+1)\cdots(a+k-1)$
is necessary.

The operator {\tt FPS} uses the operator {\tt SimpleDE} of the next section.

If an error message of type
\begin{verbatim}
Could not find the limit of:
\end{verbatim}
occurs, you can set the corresponding limit yourself and try a
recalculation. In the computation of {\tt FPS(atan(cot(x)),x,0)},
REDUCE is not able to find the value for the limit
{\tt limit(atan(cot(x)),x,0)} since the {\tt atan} function is multi-valued.
One can choose the branch of {\tt atan} such that this limit equals
$\pi/2$ so that we may set
\begin{verbatim}
let limit(atan(cot(~x)),x,0)=>pi/2;
\end{verbatim}
and a recalculation of {\tt FPS(atan(cot(x)),x,0)}
yields the output {\tt pi - 2*x} which is
the correct local series representation.

\section{\REDUCE{} operator {\tt SimpleDE}}

{\tt SimpleDE(f,x)} tries to find a homogeneous linear differential
equation with polynomial coefficients for $f$ with respect to $x$.
Make sure that $y$ is not a used variable.
The setting {\tt factor df;} is recommended to receive a nicer output form.

Examples: {\tt SimpleDE(asin(x)\verb+^+2,x)} then results in
\begin{verbatim}
            2
df(y,x,3)*(x  - 1) + 3*df(y,x,2)*x + df(y,x)
\end{verbatim}
{\tt SimpleDE(exp(x\verb+^+(1/3)),x)} gives
\begin{verbatim}
              2
27*df(y,x,3)*x  + 54*df(y,x,2)*x + 6*df(y,x) - y
\end{verbatim}
and {\tt SimpleDE(sqrt(2-x\verb+^+2),x)} yields
\begin{verbatim}
          2
df(y,x)*(x  - 2) - x*y
\end{verbatim}
The depth for the search of a differential equation for {\tt f} is
controlled by the variable {\tt fps\verb+_+search\verb+_+depth};
higher values for {\tt fps\verb+_+search\verb+_+depth}
will increase the chance to find the solution, but increases the
complexity as well. The default value for {\tt fps\verb+_+search\verb+_+depth}
is 5. For {\tt FPS(sin(x\verb+^+(1/3)),x)}, or
{\tt SimpleDE(sin(x\verb+^+(1/3)),x)} e.\ g., a setting
{\tt fps\verb+_+search\verb+_+depth:=6} is necessary.

The output of the FPS package can be influenced by the
switch {\tt tracefps}. Setting {\tt on tracefps} causes various
prints of intermediate results.

\section{Problems in the current version}
The handling of logarithmic singularities is not yet implemented.

The rational type implementation is not yet complete.

The support of special functions \cite{Koepf:94}
will be part of the next version.

\begin{thebibliography}{9}

\bibitem{Han}
E.\ R. Hansen, {\em A table of series and products.}
Prentice-Hall, Englewood Cliffs, NJ, 1975.

\bibitem{Koepf:92} Wolfram Koepf,
{\em Power Series in Computer Algebra},
J.\ Symbolic Computation 13 (1992)

\bibitem{Koepf:93a} Wolfram Koepf,
{\em Examples for the Algorithmic Calculation of Formal
Puiseux, Laurent and Power series},
SIGSAM Bulletin 27, 1993, 20-32.

\bibitem{Koepf:93b} Wolfram Koepf,
{\em Algorithmic development of power series.} In:
Artificial intelligence and symbolic mathematical computing,
ed.\ by J.\ Calmet and J.\ A.\ Campbell,
International Conference AISMC-1, Karlsruhe, Germany, August 1992, Proceedings,
Lecture Notes in Computer Science {\bf 737}, Springer-Verlag,
Berlin--Heidelberg, 1993, 195--213.

\bibitem{Koepf:94} Wolfram Koepf,
{\em Algorithmic work with orthogonal polynomials and special functions.}
Konrad-Zuse-Zentrum Berlin (ZIB), Preprint SC 94-5, 1994.

\end{thebibliography}

\end{document}


\subsection{Loss functions}
\label{sec:loss functions}
Here we are going to discuss some of the loss functions that have been tested in this field so far, and that are relevant to this article. We will look away from the content loss as this part has stayed the same for virtually all papers on the matter.
\newline\newline
Style loss was initially discovered to be separable from content loss in a CNN (namely VGG16) by Gatys et al \cite{Gatys:1}. They calculated what is known as the Gram matrix which represents the style of an image as correlation between different features. The style loss is the squared error between the Gram matrix of the features of the reference style image and the image for which we are minimizing the loss function. This function has been the basis for several other style transfer algorithms, and was for instance approximated with a neural net by Johnson et al \cite{Johnson:1} trained on a certain style.
\newline\newline
Another key finding was that the distribution of the features was a more complete representation of style. This led to Huang et al. \cite{Huang:1} testing out minimizing the Wasserstein distance between the features instead. This new representation of style produced results of higher artistic quality. Other techniques based the idea of the distance between two distributions as loss has also been tested out with MMD (Maximum mean discrepancy) as the distance measure in a paper by Li et al. \cite{Li:2} where they also combined it with the gram matrix. Chuan Li and Michael Wand used markov random fields to calculate an alternative to the Gram matrix entirely \cite{Li:3}. Other style loss functions that noteworthy but not too relevant to delve into here are histogram loss \cite{Risser:1} and coral loss \cite{sun:1}
\newline\newline
Another possible loss function component for video style transfer is the temporal loss. As temporal loss has only been mentioned in the Ruder et al. \cite{Ruder:1} paper we will just briefly go some that are described here. To go from picture style transfer to video style transfer we can introduce a temporal loss to avoid flickering, meaning that we should punish adjacent frames for being too different from each other.The easiest way to do this is to just look at the difference between the current frame and the previous one and use the difference of the generated frame and the previous frame as a part of the loss function. This can however create some issues in the resulting video, as there are some parts of each image that should change between frames, and other parts that should not. A better way is to add weights for this loss function, penalizing the model for having difference on only the frames that are supposed to be the same. This is known as short-term temporal loss. Long-term temporal loss is doing the same but looking further back in multiple previous frames for occluded areas that are disoccluded in the current frame. This loss function tries to make areas occluded by moving objects stay the same when disoccluded again. However, this requires a lot of additional computation.
\newline\newline
After applying style transfer on an image with Gatys et al \cite{Gatys:1}, we can end up with an image with many high frequency artifacts. To get rid of these variations in the output image, we want to penalize local variation. We can define this local variation loss as
\begin{equation}
    \sum{|y_{i+1}-y_i|^\beta}
\end{equation}
where $\beta\in\{1,2\}.$ Here $y$ are the pixel-values in one row of the output image \cite{Zhaoyou:1}. Now we can extend Equation (\ref{eq:total_loss}) with the total variation loss
\begin{equation}
    \mathcal{L}_\text{total}(\vec{p}, \vec{a}, \vec{x})=\alpha\mathcal{L}_\text{content}(\vec{p},\vec{x})+\beta\mathcal{L}_\text{style}(\vec{a},\vec{x})+\lambda\mathcal{L}_\text{TV}(\vec{x})
\end{equation}
where $\lambda$ is the weight for the total variation loss.
\newpage


\documentclass{article}

\usepackage[utf8]{inputenc}
\usepackage{natbib}
\usepackage{amssymb,amsmath}

\begin{document}

\title{Simple Epidemiological Models.}

\maketitle

\section{SIR model}

The susceptible-infected-recovered (SIR) model in a closed population was
proposed by~\cite{kermack1927contribution} as a special case of a more general
model, and forms the framework of many compartmental models.  Susceptible
individuals, $S$, are infected by infected individuals, $I$, at a per-capita
rate $\beta I$, and infected individuals recover at a per-capita rate $\gamma$
to become recovered individuals, $R$.

\begin{align}
\frac{dS(t)}{dt} &= -\beta S(t) I(t)\\
\frac{dI(t)}{dt} &= \beta S(t) I(t)- \gamma I(t)\\
\frac{dR(t)}{dt} &= \gamma I(t)
\end{align}

\section{SEIR model}

The susceptible-exposed-infected-recovered (SEIR) model extends the SIR model
to include an exposed but non-infectious class. The implementation in this
section considers proportions of susceptibles, exposed, infectious individuals
in an open population, with no additional mortality associated with infection
(such that the population size remains constant and $R$ is not modelled
explicitly).

\begin{align}
\frac{dS(t)}{dt} &= \mu-\beta S(t) I(t) - \mu S(t)\\
\frac{dE(t)}{dt} &= \beta S(t) I(t)- (\sigma + \mu) E(t)\\
\frac{dI(t)}{dt} &= \sigma E(t)- (\gamma + \mu) I(t)\\
\frac{dR(t)}{dt} &= \gamma I(t) = \mu R
\end{align}

\bibliographystyle{plain}
\bibliography{main}

\end{document}


\chapter{Signals \& Systems}

\section{What is a system?}

A system is defined as a grouping of elements to be
analysed together. They can be categorised as linear or
non-linear, depending on the equations used to
describe them.
Linear systems are considered to be idealised systems,
whilst non-linear systems are those representing real-world
conditions.

Systems may also be categorised based on the
\emph{order} of their differential equations. Some examples
of categorised systems are shown in table \ref{table:
examples of systems}.

\begin{table}[h!]
\centering
\begin{tabular}{c|cc}
 & Linear & Non-Linear\\
 \hline
1st order & RC circuit & Population growth\\
2nd order & Spring Mass Damper & Pendulum\\
3rd order & - & Chaotic Systems\\
... & ...  & ... \\
Nth order & Wave Equation & General Relativity\\
\hline
\end{tabular}
\label{table: examples of systems}
\caption{Examples of linear and non-linear systems.}
\end{table}

\section{Linear Systems}
A system is considered linear if they meet the following
two principles:

\begin{itemize}
 \item Proportionality. Given an input $t$, the system will
return an output $f(t)$, and if given an input $\alpha t$
the system will return an output $f(\alpha t) = \alpha
f(t)$.
 \item Superposition. Given inputs $t_1$ and $t_2$, the
system will return outputs $f(t_1)$ and $f(t_2)$
respectively. If given an input $t_1 + t_2$ the system will
return an output $f(t_1 + t_2) = f(t_1) + f(t_2)$.
\end{itemize}

\subsection{Exercise: Linear systems}
Given the following equations, determine if they belong to
linear systems or not.

\textbf{Case 1:} $f(s) = 3s$. Evaluating the function to
test proportionality, we obtain the following:

\begin{equation*}
 \begin{split}
  f(2s) & =3(2s)\\
  & = 6 s\\
  2 f(s) &= 2 (3s)\\
  &= 6s\\
  f(2s)&=2f(s)
 \end{split}
\end{equation*}

For the superposition principle, we have:
\begin{equation*}
 \begin{split}
  f(s_1) & =3s_1\\
  f(s_2) & =3s_2\\
  f(s_1+s_2) & =3(s_1+s_2)\\
   &= 3s_1 + 3s_2\\
  f(s_1 + s_2) & = f(s_1) + f(s_2)
 \end{split}
\end{equation*}

We conclude that case 1 is linear.

% \vspace*{1cm}

An equation can be tested to meet both principles
simultaneously. Let $s=\alpha(s_1+s_2)$ and $f(s)$ be the
output of the system. A system is linear if the following
condition is met:

\begin{equation}
  f(\alpha(s_1+s_2))  = \alpha f(s_1) + \alpha f(s_2)
  \label{eq: proportionality and superposition}
\end{equation}

Evaluating \ref{eq: proportionality and superposition} for
case 1 as an example:

\begin{equation*}
\begin{split}
 f(\alpha (s_1 + s_2)) & = 3 (\alpha (s_1 + s_2))\\
 & = 3 \alpha s_1 + 3 \alpha s_2\\
 f(\alpha s) & = 3 \alpha s \\
 f(\alpha (s_1 + s_2)) & = \alpha f(s_1) + \alpha f(s_2)
\end{split}
\end{equation*}

\textbf{Case 2:} $f(s) = 3s + 1$.

\begin{equation*}
 \begin{split}
  f(\alpha s) & = 3 \alpha  s +1\\
  \alpha f(s) & = 3 \alpha s + \alpha\\
  f(\alpha s) & \neq \alpha f(s)\\
  f(s_1) + f(s_2) & = 3 s_1 + 3 s_2 + 2\\
  f(s_1 + s_2) & = 3 (s_1+s_2) + 1\\
  f(s_1 +s_2) & \neq f(s_1) + f(s_2)
 \end{split}
\end{equation*}

Case 2 is non-linear.

% \vspace*{1cm}

\textbf{Case 3:} $f(s) = 0.5 \cos (0.1 s) $

\begin{equation*}
 \begin{split}
  f(\alpha s) & = 0.5 \cos (0.1 \alpha s)\\
  \alpha f(s) & = 0.5 \alpha \cos (0.1 s)\\
  f(\alpha s) & \neq \alpha f(s)\\
  f(s_1) + f(s_2) & = 0.5 \cos (0.1 s_1) + 0.5 \cos (0.1
s_2) \\
  f(s_1 + s_2) & = 0.5 \cos (0.1 s_1 + 0.1 s_2)\\
  f(s_1 + s_2) & \neq f(s_1) + f(s_2)
 \end{split}
\end{equation*}

Case 3 is non-linear.

% \vspace*{1cm}

\textbf{Case 4:} $f(s) = 1.2 \mathrm{e}^{0.1 s}$

\begin{equation*}
 \begin{split}
f(\alpha (s_1+s_2))  & = 1.2 \mathrm{e}^{0.1 \alpha (s_1 +
s_2)}\\
\alpha f(s_1) & = 1.2 \alpha \mathrm{e}^{0.1 s_1}\\
\alpha f(s_1) + \alpha f(s_2) & = 1.2 \alpha
(\mathrm{e}^{0.1 s_1} + \mathrm{e}^{0.1 s_2})\\
f(\alpha (s_1+s_2))  & \neq \alpha f(s_1) + \alpha f(s_2)
 \end{split}
\end{equation*}

Case 4 is non-linear.

% \vspace*{1cm}

\textbf{Case 5:} $f(s) = \int\limits_0^t s(t)dt$
\begin{equation*}
 \begin{split}
f(\alpha (s_1 + s_2))  & = \alpha \int\limits_0^t
\Bigl( s_1(t)+s_2(t) \Bigr)  dt\\
& = \alpha \int\limits_0^t
s_1(t)dt + \alpha \int\limits_0^t
s_2(t)dt\\
\alpha f(s_1) + \alpha f(s_2) & = \alpha \int\limits_0^t
s_1(t)dt + \alpha \int\limits_0^t
s_2(t)dt\\
f(\alpha (s_1+s_2))  & = \alpha f(s_1) + \alpha f(s_2)
 \end{split}
\end{equation*}

Case 5 is linear.

% \vspace*{1cm}

\textbf{Case 6:} $f(s) = \frac{ds(t)}{dt}$

\begin{equation*}
 \begin{split}
f(\alpha (s_1 + s_2))  & =  \alpha
\frac{d \Bigl (s_1(t)+s_2(t) \Bigr)}{dt}\\
& = \alpha \frac{d s_1(t)}{dt} + \alpha \frac{d
s_2(t)}{dt} \\
\alpha f(s_1) + \alpha f(s_2) & = \alpha \frac{d s_1(t)}{dt}
+ \alpha \frac{d s_2(t)}{dt} \\
f(\alpha (s_1+s_2))  & = \alpha f(s_1) + \alpha f(s_2)
 \end{split}
\end{equation*}

Case 6 is linear.

Note that for all equations $\alpha \neq 1$ because using
the identity is not valid for proving compliance.

\subsection{Order of a system}

The order of a system is dependent on the highest order of
derivatives in the equations that describe it. A system
with no derivative terms is a \emph{zero order} system, a
system with a differential equation of order 1 is a
\emph{first-order} system, and so on.

Revisiting the cases shown in Exercise 1, we can classify
the systems depending on their order as well. Table
\ref{table: exercise 1 order} contains the classification
of each case.

\begin{table}[t]
\centering
\begin{tabular}{c|cc}
Equation & Type & Order\\
\hline
$3s$ & Linear & 0\\
$3s + 1$ & Non-linear & 0\\
$0.5 \cos (0.1 s)$ & Non-Linear & 0\\
$1.2 \mathrm{e}^{0.1s}$ & Non-Linear & 0\\
$\int\limits_0^{t} s(t)dt$ & Linear & 0\\
$\frac{d s(t)}{dt}$ & Linear & 1\\
\hline
\end{tabular}
\label{table: exercise 1 order}
\caption{Classification of systems.}
\end{table}

\section{Example: Spring-Mass-Damper System}

The Spring-Mass-Damper System (abbreviated SMD) is the most
commonly used abstraction for systems. Depending on the
initial conditions of the systems, it may or may not be
linear. An example is presented below.

Given the forces interacting in the Free Body Diagram, the
system's equation is obtained as follows:

\begin{equation}
 \begin{split}
  \sum F & = 0\\
  F_s(t) + F_d(t) - F(t) & = 0\\
 \end{split}
 \label{eq: sum of forces - week01}
\end{equation}

Recall the equations for springs and dampers, substituting
them in \eqref{eq: sum of forces - week01}.

\begin{equation*}
 \begin{split}
  F_s & = b \dot{x}\\
  F_d & = k x(t)\\
  b \dot{x} + k x(t) & = F(t)
 \end{split}
\end{equation*}

Rewriting the equation into the form $\dot x + f(t) x(t) =
g(t)$.

\begin{equation}
 \dot{x} + \frac{k}{b} x(t) = \frac{1}{b} F(t)
 \label{eq: system equation}
\end{equation}

\subsection{Linear System Case}

Solving the system via Laplace Transforms to obtain an
equation for x(t). Consider $x(0) = 0$ as the initial
condition.

\begin{equation}
 \begin{split}
  \mathfrak{L} \Bigl \{ \dot{x} + \frac{k}{b} x(t)  =
\frac{1}{b} F(t) \Bigr \} & \xrightarrow{} s \mathcal{X} -
x(0) + \frac{k}{b} \mathcal{X} = \frac{1}{b} \mathcal{F}
 \end{split}
 \label{eq: general laplace transform}
\end{equation}

Evaluating $x(0)=0$ gives us the system equation in the
frequency domain.

\begin{equation}
 s \mathcal{X} + \frac{k}{b} \mathcal{X} = \frac{1}{b}
\mathcal{F}
\label{eq: laplace transform}
\end{equation}

Solve \ref{eq: laplace transform} for $\mathcal{X}$ and
then obtain the inverse Laplace transform of the equation.

\begin{equation*}
 \begin{split}
  \mathcal{X} (s + \frac{k}{b}) & = \frac{1}{b}
\mathcal{F}\\
\mathcal{X} & = \frac{1}{b} \frac{1}{s + \frac{k}{b}}
\mathcal{F}\\
 \end{split}
\end{equation*}

\begin{equation}
 \begin{split}
  \mathfrak{L}^{-1} \Bigl \{ \mathcal{X}  = \frac{1}{b}
\frac{1}{s + \frac{k}{b}}
\mathcal{F}   \Bigr\} & \xrightarrow{} x(F(t)) =
\frac{1}{b} \mathrm{e}^{-\frac{k}{b}t} F(t)
 \end{split}
 \label{eq: linear system equation}
\end{equation}

Testing the principles of proportionality and superposition
on \ref{eq: linear system equation} we have

\begin{equation*}
 \begin{split}
  x(\alpha (F_1 + F_2)) & = \frac{\alpha}{b}
\mathrm{e}^{-\frac{k}{b}t} (F_1 + F_2)\\
\alpha x(F_1) & = \frac{\alpha}{b}
\mathrm{e}^{-\frac{k}{b}t} F_1\\
\alpha x(F_1) + \alpha x(F_2) & = \frac{\alpha}{b}
\mathrm{e}^{-\frac{k}{b}t} (F_1 + F_2)\\
x(\alpha (F_1 + F_2)) & = \alpha x(F_1) + \alpha x(F_2)
 \end{split}
\end{equation*}

We confirm that the system is linear for an initial
condition of $x(0)=0$.