\chapter{Flavors of rings}
We continue our exploration of rings by considering
some nice-ness properties that rings or ideals can satisfy,
which will be valuable later on.
As before, number theory is interlaced as motivation.
I guess I can tell you at the outset what the completed table
is going to look like, so you know what to expect.

\begin{center}
	\begin{tabular}[h]{lll}
		Ring noun & Ideal adjective & Relation \\ \hline
		PID & principal &
			$R$ is a PID $\iff$ $R$ is an integral domain, \\
			&& \qquad and every $I$ is principal \\
		Noetherian ring & finitely generated &
			$R$ is Noetherian $\iff$ every $I$ is fin.\ gen. \\
		field & maximal & $R/I$ is a field $\iff$ $I$ is maximal \\
		integral domain & prime & $R/I$ is an integral domain
			$\iff$ $I$ is prime \\
	\end{tabular}
\end{center}

\section{Fields}
\prototype{$\QQ$ is a field, but $\ZZ$ is not.}

We already saw this definition last chapter:
a field $K$ is a nontrivial ring for which every nonzero element is a unit.

In particular, there are only two ideals in a field:
the ideal $(0)$, which is maximal, and the entire field $K$.

\section{Integral domains}
\prototype{$\ZZ$ is an integral domain.}

In practice, we are often not so lucky that we have a full-fledged field.
Now it would be nice if we could still conclude the zero product property:
if $ab = 0$ then either $a = 0$ or $b = 0$.
If our ring is a field, this is true: if $b \neq 0$,
then we can multiply by $b\inv$ to get $a = 0$.
But many other rings we consider like $\ZZ$ and $\ZZ[x]$ also have this property,
despite not having division.

Not all rings though: in $\Zc{15}$,
\[ 3 \cdot 5 \equiv 0 \pmod{15}. \]
If $a, b \neq 0$ but $ab=0$ then we say $a$ and $b$ are \vocab{zero divisors}
of the ring $R$.
So we give a name to such rings.
\begin{definition}
	A nontrivial ring with no zero divisors
	is called an \vocab{integral domain}.\footnote{Some
		authors abbreviate this to ``domain'', notably Artin.}
\end{definition}
\begin{ques}
	Show that a field is an integral domain.
\end{ques}
\begin{exercise}
	[Cancellation in integral domains]
	Suppose $ac = bc$ in an integral domain, and $c \neq 0$.
	Show that that $a = b$.
	(There is no $c\inv$ to multiply by,
	so you have to use the definition.)
\end{exercise}

\begin{example}
	[Examples of integral domains]
	Every field is an integral domain,
	so all the previous examples apply.
	In addition:
	\begin{enumerate}[(a)]
		\ii $\ZZ$ is an integral domain, but it is not a field.
		\ii $\RR[x]$ is not a field,
		since there is no polynomial $P(x)$ with $xP(x) = 1$.
		However, $\RR[x]$ is an integral domain,
		because if $P(x) Q(x) = 0$ then one of $P$ or $Q$ is zero.
		\ii $\ZZ[x]$ is also an example of an integral domain.
		In fact, $R[x]$ is an integral domain for any integral domain $R$ (why?).
		\ii $\Zc n$ is a field (hence integral domain)
		exactly when $n$ is prime.
		When $n$ is not prime, it is a ring but not an integral domain.
	\end{enumerate}
	The trivial ring $0$ is \emph{not} considered an integral domain.
\end{example}

At this point, we go ahead and say:
\begin{definition}
	An integral domain where all ideals are principal
	is called a \vocab{principal ideal domain (PID)}.
\end{definition}
The ring $\ZZ/6\ZZ$ is an example of a ring
which is a principal ideal ring, but not an integral domain.
As we alluded to earlier, we will never really use ``principal ideal ring''
in any real way: we typically will want to strengthen it to PID.

\section{Prime ideals}
\prototype{$(5)$ is a prime ideal of $\ZZ$.}

We know that every integer can be factored (up to sign)
as a unique product of primes; for example $15 = 3 \cdot 5$
and $-10 = -2 \cdot 5$.
You might remember the proof involves the so-called B\'ezout's lemma,
which essentially says that $(a,b) = (\gcd(a,b))$;
in other words we've carefully used the fact that $\ZZ$ is a PID.

It turns out that for general rings, the situation is not as nice
as factoring elements because most rings are not PID's.
The classic example of something going wrong is
\[ 6 = 2 \cdot 3 = \left( 1-\sqrt{-5} \right)\left( 1+\sqrt{-5} \right) \]
in $\ZZ[\sqrt{-5}]$.
Nonetheless, we can sidestep the issue
and talk about factoring \emph{ideals}:
somehow the example $10 = 2 \cdot 5$ should be $(10) = (2) \cdot (5)$,
which says ``every multiple of $10$ is the product of a
multiple of $2$ and a multiple of $5$''.
I'd have to tell you then how to multiply two ideals, which I do
in the chapter on unique factorization.

Let's at least figure out what primes are.
In $\ZZ$, we have that $p \neq 1$ is prime if whenever $p \mid xy$,
either $p \mid x$ or $p \mid y$.
We port over this definition to our world of ideals.
\begin{definition}
	\label{def:prime_ideal}
	A \emph{proper} ideal $I \subsetneq R$ is a \vocab{prime ideal}
	if whenever $xy \in I$, either $x \in I$ or $y \in I$.
\end{definition}
The condition that $I$ is proper is analogous to the
fact that we don't consider $1$ to be a prime number.

\begin{example}[Examples and non-examples of prime ideals]
	\listhack
	\begin{enumerate}[(a)]
		\ii The ideal $(7)$ of $\ZZ$ is prime.
		\ii The ideal $(8)$ of $\ZZ$ is not prime,
		since $2 \cdot 4 = 8$.

		\ii The ideal $(x)$ of $\ZZ[x]$ is prime.
		\ii The ideal $(x^2)$ of $\ZZ[x]$ is not prime,
		since $x \cdot x = x^2$.

		\ii The ideal $(3,x)$ of $\ZZ[x]$ is prime.
		This is actually easiest to see
		using \Cref{thm:prime_ideal_quotient} below.

		\ii The ideal $(5) = 5\ZZ + 5i\ZZ$ of $\ZZ[i]$
		is not prime, since the elements
		$3+i$ and $3-i$ have product $10 \in (5)$,
		yet neither is itself in $(5)$.
	\end{enumerate}
\end{example}
\begin{remark}
	\label{rem:unit_sign_issue}
	Ideals have the nice property that they get rid of ``sign issues''.
	For example, in $\ZZ$, do we consider $-3$ to be a prime?
	When phrased with ideals, this annoyance goes away: $(-3) = (3)$.
	More generally, for a ring $R$, talking about ideals
	lets us ignore multiplication by a unit.
	(Note that $-1$ is a unit in $\ZZ$.)
\end{remark}

\begin{exercise}
	What do you call a ring $R$ for which the zero ideal $(0)$ is prime?
\end{exercise}

We also have:
\begin{theorem}[Prime ideal $\iff$ quotient is integral domain]
	\label{thm:prime_ideal_quotient}
	An ideal $I$ is prime if and only if $R/I$ is an integral domain.
\end{theorem}
\begin{exercise}
	[Mandatory]
	Convince yourself the theorem is true;
	it is just definition chasing.
	(A possible start is to consider $R = \ZZ$ and $I = (15)$.)
\end{exercise}

I now must regrettably inform you that unique factorization is still
not true even with the notion of a ``prime'' ideal
(though again I haven't told you how to multiply two ideals yet).
But it will become true with some additional assumptions
that will arise in algebraic number theory
(relevant buzzword: Dedekind domain).

\section{Maximal ideals}
\prototype{The ideal $(x,5)$ is maximal in $\ZZ[x]$, by quotient-ing.}

Here's another flavor of an ideal.
\begin{definition}
	A proper ideal $I$ of a ring $R$ is \vocab{maximal} if
	it is not contained in any other proper ideal.
\end{definition}

\begin{example}
	[Examples of maximal ideals]
	\listhack
	\begin{enumerate}[(a)]
		\ii The ideal $I = (7)$ of $\ZZ$ is maximal, because
		if an ideal $J$ contains $7$
		and an element $n$ not in $I$
		it must contain $\gcd(7,n) = 1$, and hence $J = \ZZ$.
		\ii The ideal $(x)$ is \emph{not} maximal in $\ZZ[x]$,
		because it's contained in $(x,5)$ (among others).
		\ii On the other hand, $(x,5)$ is indeed maximal in $\ZZ[x]$.
		This is actually easiest to verify using
		\Cref{thm:max_ideal_quotient} below.
		\ii Also, $(x)$ is maximal in $\CC[x]$,
		again appealing to \Cref{thm:max_ideal_quotient} below.
	\end{enumerate}
\end{example}

\begin{exercise}
	What do you call a ring $R$ for which the zero ideal $(0)$ is maximal?
\end{exercise}

There's an analogous theorem to the one for prime ideals.
\begin{theorem}
	[$I$ maximal $\iff$ $R/I$ field]
	\label{thm:max_ideal_quotient}
	An ideal $I$ is maximal if and only if $R/I$ is a field.
\end{theorem}
\begin{proof}
	A ring is a field if and only if $(0)$ is the only maximal ideal.
	So this follows by \Cref{prob:inclusion_preserving}.
\end{proof}

\begin{corollary}
	[Maximal ideals are prime]
	If $I$ is a maximal ideal of a ring $R$, then $I$ is prime.
\end{corollary}
\begin{proof}
	If $I$ is maximal, then $R/I$ is a field,
	hence an integral domain, so $I$ is prime.
\end{proof}

In practice, because modding out by generated ideals is pretty convenient,
this is a very efficient way to check whether an ideal is maximal.
\begin{example}
	[Modding out in $\ZZ{[x]}$]
	\listhack
	\begin{enumerate}[(a)]
		\ii This instantly implies that $(x,5)$ is a maximal ideal
		in $\ZZ[x]$, because if we mod out by $x$ and $5$ in $\ZZ[x]$,
		we just get $\FF_5$, which is a field.
		\ii On the other hand, modding out by just $x$ gives $\ZZ$,
		which is an integral domain but not a field; that's why $(x)$ is
		prime but not maximal.
	\end{enumerate}
\end{example}

As we saw, any maximal ideal is prime.
But now note that $\ZZ$ has the special property that
all of its nonzero prime ideals are also maximal.
It's with this condition and a few other minor conditions
that you get a so-called \emph{Dedekind domain}
where prime factorization of ideals \emph{does} work.
More on that later.

\section{Field of fractions}
\prototype{$\Frac(\ZZ) = \QQ$.}
As long as we are here, we take the time to introduce a useful
construction that turns any integral domain into a field.

\begin{definition}
	Given an integral domain $R$,
	we define its \vocab{field of fractions} or \vocab{fraction field}
	$\Frac(R)$ as follows:
	it consists of elements $a / b$, where $a,b \in R$ and $b \neq 0$.
	We set $a / b \sim c / d$ if and only if $bc = ad$.
	Addition and multiplication is defined by
	\begin{align*}
		\frac ab + \frac cd &= \frac{ad+bc}{bd} \\
		\frac ab \cdot \frac cd &= \frac{ac}{bd}.
	\end{align*}
\end{definition}
In fact everything you know about $\QQ$ basically carries over by analogy.
You can prove if you want that this indeed a field, but
considering how comfortable we are that $\QQ$ is well-defined,
I wouldn't worry about it\dots

\begin{definition}
	Let $k$ be a field.
	We define $k(x) = \Frac(k[x])$
	(read ``$k$ of $x$''),
	and call it the \vocab{field of rational functions}.
\end{definition}

\begin{example}
	[Examples of fraction fields]
	\listhack
	\begin{enumerate}[(a)]
		\ii By \emph{definition}, $\Frac(\ZZ) = \QQ$.
		\ii The field $\RR(x)$ consists of rational functions in $x$:
		\[ \RR(x) = \left\{ \frac{f(x)}{g(x)} \mid f,g \in \RR[x] \right\}. \]
		For example, $\frac{2x}{x^2-3}$ might be a typical element.
	\end{enumerate}
\end{example}

\begin{example}
	[Gaussian rationals]
	\label{ex:gaussian_rationals}
	Just like we defined $\ZZ[i]$ by abusing notation,
	we can also write $\QQ(i) = \Frac(\ZZ[i])$.
	Officially, it should consist of
	\[ \QQ(i) = \left\{ \frac{f(i)}{g(i)} \mid g(i) \ne 0 \right\} \]
	for polynomials $f$ and $g$ with rational coefficients.
	But since $i^2=-1$ this just leads to
	\[ \QQ(i) = \left\{ \frac{a+bi}{c+di} \mid a,b,c,d \in \QQ,
		(c,d) \ne (0,0) \right\}. \]
	And since $\frac{1}{c+di} = \frac{c-di}{c^2+d^2}$ we end up with
	\[ \QQ(i) = \left\{ a+bi \mid a,b \in \QQ \right\}. \]
\end{example}

\section{Unique factorization domains (UFD's)}
\prototype{$\ZZ$ and polynomial rings in general.}

Here is one stray definition that will be important
for those with a number-theoretic inclination.
Over the positive integers, we have a fundamental theorem of arithmetic,
stating that every integer is uniquely the product of prime numbers.

We can even make an analogous statement in $\ZZ$ or $\ZZ[i]$,
if we allow representations like $6 = (-2)(-3)$ and so on.
The trick is that we only consider everything \emph{up to units};
so $6 = (-2)(-3) = 2 \cdot 3$ are considered the same.

The general definition goes as follows.
\begin{definition}
	A nonzero non-unit of an integral domain $R$ is
	\vocab{irreducible} if it cannot be written as the product of two non-units.

	An integral domain $R$ is a \vocab{unique factorization domain}
	if every nonzero non-unit of $R$ can be written
	as the product of irreducible elements,
	which is unique up to multiplication by units.
\end{definition}
\begin{ques}
	Verify that $\ZZ$ is a UFD.
\end{ques}

\begin{example}
	[Examples of UFD's]
	\listhack
	\begin{enumerate}[(a)]
		\ii Fields are a ``degenerate'' example of UFD's:
		every nonzero element is a unit,
		so there is nothing to check.

		\ii $\ZZ$ is a UFD.
		The irreducible elements are $p$ and $-p$,
		for example $5$ or $-17$.

		\ii $\QQ[x]$ is a UFD:
		polynomials with rational coefficients
		can be uniquely factored, up to scaling by constants
		(as the units of $\QQ[x]$ are just the rational numbers).

		\ii $\ZZ[x]$ is a UFD.

		\ii The Gaussian integers $\ZZ[i]$ turns out to be a UFD too
		(and this will be proved in the chapters on algebraic number theory).

		\ii $\ZZ[\sqrt{-5}]$ is the classic non-example of a UFD:
		one may write
		\[ 6 = 2 \cdot 3 = \left( 1-\sqrt{-5} \right)
			\left( 1+\sqrt{-5} \right) \]
		but each of $2$, $3$, $1 \pm \sqrt{-5}$ is irreducible.
		(It turns out the right way to fix this is
		by considering prime \emph{ideals} instead,
		and this is one big motivation for \Cref{part:algnt1}.)

		\ii Theorem we won't prove: every PID is a UFD.
		\ii Theorem we won't prove: if $R$ is a UFD,
		so is $R[x]$ (and hence by induction so is
		$R[x,y]$, $R[x,y,z]$, \dots).
	\end{enumerate}
\end{example}

\section{\problemhead}
Not olympiad problems, but again the spirit is very close
to what you might see in an olympiad.

\begin{problem}
	Consider the ring
	\[ \QQ[\sqrt2] = \left\{ a + b\sqrt2 \mid a,b \in \QQ \right\}. \]
	Is it a field?
	\begin{hint}
		Yes.
	\end{hint}
\end{problem}

\begin{problem}
	[Homomorphisms from fields are injective]
	\label{prob:field_hom}
	Let $K$ be a field and $R$ a ring.
	Prove that any homomorphism $\psi \colon K \to R$
	is injective.\footnote{Note that $\psi$
		cannot be the zero map for us,
		since we require $\psi(1_K) = 1_R$.
		You sometimes find different statements in the literature.}
	\begin{hint}
		The kernel is an ideal of $K$!
	\end{hint}
\end{problem}

\begin{sproblem}
	[Pre-image of prime ideals]
	\label{prob:prime_preimage}
	Suppose $\phi \colon R \to S$ is a ring homomorphism,
	and $I \subseteq S$ is a prime ideal.
	Prove that $\phi\pre(I)$ is prime as well.
	\begin{hint}
		This is just a definition chase.
	\end{hint}
	\begin{sol}
		Consider $ab \in \phi\pre(I)$,
		meaning $\phi(ab) = \phi(a) \phi(b) \in I$.
		Since $I$ is prime, either $\phi(a) \in I$ or $\phi(b) \in I$.
		In the former case we get $a \in \phi\pre(I)$ as needed;
		the latter case we get $b \in \phi\pre(I)$.
	\end{sol}
\end{sproblem}

\begin{sproblem}
	\gim
	Let $R$ be an integral domain with finitely many elements.
	Prove that $R$ is a field.
	\label{prob:finite_domain_field}
	\begin{hint}
		Fermat's little theorem type argument;
		cancellation holds in integral domains.
	\end{hint}
	\begin{sol}
		Let $x \in R$ with $x \neq 0$.
		Look at the powers $x$, $x^2$, \dots.
		By pigeonhole, eventually two of them coincide.
		So assume $x^m = x^n$ where $m < n$, or equivalently
		\[ 0  = x \cdot x \cdot \dots \cdot x
			\cdot \left( x^{n-m} - 1 \right). \]
		Since $x \ne 0$, we get $x^{n-m} - 1 = 0$,
		or $x^{n-m} = 1$.
		So $x^{n-m-1}$ is an inverse for $x$.

		This means every nonzero element has an inverse,
		ergo $R$ is a field.
	\end{sol}
\end{sproblem}

\begin{sproblem}
	[Krull's theorem]
	\label{prob:krull_max_ideal}
	Let $R$ be a ring and $J$ a proper ideal.
	\begin{enumerate}[(a)]
		\ii Prove that if $R$ is Noetherian,
		then $J$ is contained in a maximal ideal $I$.
		\ii Use Zorn's lemma (\Cref{ch:zorn})
		to prove the result even if $R$ isn't Noetherian.
	\end{enumerate}
	\begin{hint}
		Just keep on adding in elements to get an ascending chain.
	\end{hint}
	\begin{sol}
		For part (b), look at the poset of \emph{proper} ideals.
		Apply Zorn's lemma (again using a union trick to verify the condition;
		be sure to verify that the union is proper!).
		In part (a) we are given no ascending infinite chains,
		so no need to use Zorn's lemma.
	\end{sol}
\end{sproblem}

\begin{problem}
	[{$\Spec k[x]$}]
	Describe the prime ideals of $\CC[x]$ and $\RR[x]$.
	\begin{hint}
		Use the fact that both are PID's.
	\end{hint}
	\begin{sol}
		The ideal $(0)$ is of course prime in both.
		Also, both rings are PID's.

		For $\CC[x]$ we get a prime ideal $(x-z)$ for each $z \in \CC$.

		For $\RR[x]$ a prime ideal $(x-a)$ for each $a \in \RR$
		and a prime ideal $(x^2 - ax + b)$ for each quadratic
		with two conjugate real roots.
	\end{sol}
\end{problem}

\begin{problem}
	Prove that any nonzero prime ideal of $\ZZ[\sqrt 2]$ is also a maximal ideal.
	%(This is starred since the main idea of the solution will get used extensively in
	%the algebraic NT chapters, even though the problem itself won't.)
	\label{prob:dedekind_sample}
	\begin{hint}
		Show that the quotient $\ZZ[\sqrt2]/I$ has finitely many elements
		for any nonzero prime ideal $I$.
		Therefore, the quotient is an integral domain, it is also a field,
		and thus $I$ was a maximal ideal.
	\end{hint}
\end{problem}


\documentclass[a4paper,11pt]{article}
\usepackage{a4wide}
\usepackage{amsmath}
\usepackage{amssymb}
\begin{document}
\begin{center}
  {\LARGE\bf Supplemental note for Week 2 Part 3}
  \end{center}
\begin{flushright}
  {\large\bf ver. 20170418-01}\\
 \ \\
{\large\bf Ryoichi Yamamoto}\\
\end{flushright}
\section{Binomial distribution $\rightarrow$ Normal distribution}

Start from Eq.(C6) shown below.
\begin{eqnarray}
  P(n)&=&\frac{M!}{n!(M-n)!}p^n(1-p)^{M-n}\\
  \ln P(n)&=&\ln M! -\ln n! - \ln (M-n)!+n\ln p +(M-n)\ln(1-p)\\
  &=&M(\ln M -1) -n(\ln n -1) - (M-n)(\ln (M-n)-1)\\& & + n\ln p +(M-n)\ln (1-p)
\end{eqnarray}
Here we used Stirling's approximation  valid for large $n$.
\begin{equation}
\ln n!\simeq n(\ln n -1)
\end{equation}
Take the 1st derivative of the above equation in terms of $n$.
\begin{eqnarray}
  \frac{d \ln P}{dn}&=&-(\ln n +1) + (\ln(M-n)+1) + \ln p -ln(1-p)\\
  &=&-\ln n + \ln(M-n)+\ln p -\ln(1-p)\\
  &=&\ln \left[\frac{M-n}{n} \right] - \ln \left[\frac{1-p}{p} \right]
\end{eqnarray}
When $n$ and $M$ are both large, the peak in $P(n)$ and also in $\ln P(n)$ is very sharp around the mean value $n=\langle N\rangle$, and thus
\begin{eqnarray}
  \left.\frac{d \ln P}{dn}\right|_{n=\langle N\rangle} &=&\ln \left[\frac{M-\langle N\rangle}{\langle N\rangle} \right] - \ln \left[\frac{1-p}{p} \right] =0\\
  \therefore \frac{M-\langle N\rangle}{\langle N\rangle} &=& \frac{1-p}{p} \\
  \langle N\rangle&=&Mp.
\end{eqnarray}
\\
Consider a Taylor expansion of $\ln P(n)$ around the mean $\langle N\rangle$ by defining $n=\langle N\rangle+\delta n$.
\begin{equation}
  \ln P(n) = \ln (\langle N\rangle) + \left.\frac{d \ln P}{dn}\right|_{n=\langle N\rangle}\delta n
  + \frac{1}{2}\left.\frac{d^2 \ln P}{dn^2}\right|_{n=\langle N\rangle}\delta n^2 + \cdots
  \end{equation}
From Eq.(7),
\begin{eqnarray}
  \frac{d^2 \ln P}{dn^2}&=&-\frac{1}{n}-\frac{1}{M-n}\\
  \left.\frac{d^2 \ln P}{dn^2}\right|_{n=\langle N\rangle}&=&-\frac{1}{Mp}-\frac{1}{M-Mp}=-\frac{1}{Mp(1-p)}=-\sigma^{-2}.
\end{eqnarray}
Terminate Eq.(12) at the 2nd order in terms of  $\delta n$, and use Eqs.(9) and (14).
\begin{equation}
P(n)=const.\times\exp\left(-\frac{\delta n^2}{2\sigma^2}\right)
\end{equation}
Determine $const.$ so that that $\int_{-\infty}^{\infty}P(n)dn=1$, we finally obtain the following normal distribution function with $\langle N\rangle=Mp$ and $\sigma^2=Mp(1-p)$.
\begin{equation}
P(n)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp\left(-\frac{(n-\langle N\rangle)^2}{2\sigma^2}\right)
\end{equation}

\section{Binomial distribution $\rightarrow$ Poisson distribution}

We now consider the limit of $M\rightarrow\infty$ while $\langle N\rangle=Mp=a$ remains constant.
Notice that the following approximations hold.
\begin{eqnarray}
%p=\frac{a}{M-1}&\simeq&\frac{a}{M}\\
M-n&\simeq& M\\
  \frac{M!}{(M-n)!}&=&M(M-1)\cdots(M-n+1)\simeq M^n
\end{eqnarray}
Again start from Eq.(C6), we can derive Poisson distribution as shown below
with $\langle N\rangle=Mp=a$ and $\sigma^2=Mp(1-p)\simeq a$.
\begin{eqnarray}
  P(n)&=&\frac{M!}{n!(M-n)!}p^n(1-p)^{M-n}\\
  &\simeq&\frac{1}{n!}M^n\left(\frac{a}{M}\right)^n\left(1-\frac{a}{M}\right)^M\\
  &\simeq&\frac{a^n e^{-a}}{n!}
\end{eqnarray}
Here we used
\begin{equation}
\lim_{M\rightarrow \infty}\left(1-\frac{a}{M}\right)^M= e^{-a}.
\end{equation}
\end{document}


\lab{IVP Methods}{IVP Methods}
\label{lab:IVP}

\objective{Implement several basic numerical methods for initial value problems, and use them to study harmonic oscillators.}

\section*{Methods for initial value problems}
Consider the initial value problem
\begin{align*}
y' &= f(x,y),\,\, a \leq x \leq b, \\
y(a) &= y_0,
\end{align*}
where $f$ is a continuous function. A solution is a continuously differentiable function $y(x)$ that satisfies the equation $y' = f(x,y)$ on the interval $[a,b]$ and for which $y(a) = y_0$.

There are many initial value problems (IVPs) where it is impossible to find a closed form (analytic) expression for the solution.
For other IVPs there is a closed form expression for the solution, but it may be difficult to interpret.
In either case, we can usually make good use of various methods of numerical approximation to study the solution. (It is still important to know mathematically that a solution exists, even when we cannot find an explicit solution!)

Consider the initial value problem
\begin{align*}
y'(x) &= \sin y(x), \\
y(0) &= y_0.
\end{align*}
This IVP does have solution, which is given implicitly by
\[x = \ln \left|\frac{\cos y_0 + \cot y_0}{\csc y + \cot y} \right|.\]
In this case, to understand the general solution it helps to use an IVP solver to plot the integral curves for some initial values. It is easy to show that this differential equation has constant solutions $y_n(x) = n \pi, n \in \mathbb{N}$. Knowing this, and after plotting several integral curves (see Figure \ref{ivp:int_curves} ), it is obvious how solutions of this IVP will behave.

\begin{figure}
\centering
\includegraphics[width=\textwidth]{Fig2.pdf}
\caption{Several integral curves for the differential equation $y' =\sin y$, using the python solver \li{dopri5}. }
\label{ivp:int_curves}
\end{figure}

Numerical methods for solving intial value problems require us to approximate the solution on a set of grid points $a = x_0< x_1<\hdots< x_n = b$ in our interval.  For simplicity we will assume that each of the $n$ subintervals $[x_{i-1},x_i]$ has equal length $h = (b-a)/n$. $h$ is called the \textit{step size}. We then look for values $y_0,y_1, \hdots, y_n$ that approximate our solution ( so $y_i \approx y(x_i)$).

For a fixed value of $i$, $ 1 \leq i \leq n$, Taylor's theorem says that
\begin{align*}
y(x_{i+1}) &= y(x_{i}) + h y'(x_i) + \frac{h^2}{2} y''(\xi_i)\text{ for some }\xi_i \in [x_i,x_{i+1}].
\end{align*}
For small values of $h$ the quantity $\frac{h^2}{2} y''(\xi_i)$ will be negligible and so we will have
\begin{align*}
y(x_{i+1}) &\approx y(x_{i}) + h y'(x_i)  ,\\
&\approx y(x_{i}) + h f(x_i,y(x_i)).
\end{align*}
This approximation leads to a first order ($\mathcal{O}(h^1)$) method called Euler's method: Let $y_0 = y(a)$, and for $i = 0, 1, \hdots, n-1$, let $y_{i+1} = y_i +hf(x_i,y_i)$.
% \begin{enumerate}
% \item Let $y_0 = y(a)$.
% \item For $i = 0, 1, \hdots, n-1$, let $y_{i+1} = y_i +hf(x_i,y_i)$.
% \end{enumerate}

Taylor's theorem also says that
\begin{align*}
y(x_{i}) &= y(x_{i+1}) - h y'(x_{i+1}) + \frac{h^2}{2} y''(\xi_i) \text{ for some } \xi_i \in [x_i,x_{i+1}], \\
\end{align*}
so that for small $h$
\begin{align*}
y(x_{i+1}) &\approx  y(x_{i}) + h f(x_{i+1},y(x_{i+1})).
\end{align*}
This approximation leads to the backwards Euler method, another first order method: Let $y_0 = y(a)$ and for $i = 0, 1, \hdots, n-1$, solve  $y_{i} = y_{i+1}-hf(x_{i+1},y_{i+1})$ for $y_{i+1}$.

Note that for both the Euler and backwards Euler methods, only $y_i, f, $ and other points in the interval $[x_i, x_{i+1}]$ are needed to find $y_{i+1}$. Because of this these are called \textit{one-step methods}.

Euler's method is an explicit method. The backwards Euler method is an implicit method since an equation must be solved at each step to find $y_{i+1}$. Explicit and implicit methods usually have their own advantages and disadvantages. While implicit methods require an equation to be solved at each time step, they also often have bettern stability properties than the corresponding explicit methods.

\begin{figure}[ht]
\centering
\includegraphics[width=\textwidth]{Fig1.pdf}
\caption{The solution of $y' -y= -2x+4, y(0) = 0$, is $y(x) = -2+2x + 2e^x.$ This is a plot of the solution, alongside approximations with Euler's method for several stepsizes.}
\label{ivp:euler}
\end{figure}

\begin{figure}[ht]
\centering
\includegraphics[width=\textwidth]{Fig3.pdf}
\caption{The solution of $y' -y= -2x+4,$ $ y(0) = 0$, is $y(x) = -2+2x + 2e^x.$ This is a loglog plot of the relative error in numerically approximating $y(2)$, using stepsizes $h = 0.2,$ $0.1,$ $0.05,$ $0.025,$ and $0.0125$. The plot shows first, second, and fourth order convergence for the Euler method, Midpoint method, and the RK4 method, respectively.}
\label{ivp:relative_error}
\end{figure}

\begin{problem} The solution of the IVP
\begin{align*}
y' + y &= 2-2x,\,\, 0 \leq x \leq 2, \\
y(0) &= 0,
\end{align*}
is given by $y(x) = 4-2x -4e^{-x}$.
Use Euler's method to numerically approximate the solution with step sizes $h = 0.4, 0.2$, and $0.1.$ Plot your results.
\end{problem}

So how do we come up with numerical methods with higher order accuracy? Using Taylor's theorem (as we did for Euler's method) to create higher-order one-step methods would lead to numerically approximating derivatives of $f(t,y)$ - not necessarily desirable.

Let us look for a second order method of the form

$y_{i+1} = y_i + a f(x_i+b, y_i+c).$

By expanding $a f(x+b, y+c)$ with Taylor's theorem and matching constants in the equation
\begin{align*}
f(x,y) + \frac{h}{2}f'(x,y) &= f(x,y) + \frac{h}{2}\frac{\partial f}{\partial x}(x,y) + \frac{h}{2}\frac{\partial f}{\partial y}(x,y) \cdot f(x,y),
\end{align*}
we find that $a = h, b = h/2,$ and $c = h/2$. This method is called the Midpoint method. IVP solvers with this general form are called \textit{Runge-Kutta methods}. Another second order Runge-Kutta method is the modified Euler method:

$y_{i+1} = y_i + \frac{h}{2}[ f(x_i, y_i) + f(x_{i+1}, y_i+ hf(x_i, y_i))]$

There are many Runge-Kutta methods with varying orders of accuracy. In practice, the methods of order four, such as the following, are most commonly used:
\begin{enumerate}
\item Apply initial conditions.
\item $K_1 = f(x_i,y_i),$
\item $K_2 = f(x_i + \frac{h}{2}, y_i + \frac{h}{2} K_1),$
\item $K_3 = f(x_i + \frac{h}{2} , y_i + \frac{h}{2} K_2),$
\item $K_4 = f(x_{i+1} , y_i + h K_3),$
\item $y_{i+1} = y_i + \frac{h}{6}(K_1 + 2K_2 + 2K_3 + K_4)$
\end{enumerate}

\begin{comment}
\begin{problem}
Suppose a differential equation is given by
\[ y' = f(t).\]
If we were to solve this by just integrating, which integration approximation method would correspond to Euler's method, backward Euler's method, modified Euler's method, the Midpoint method, and the fourth order Runge-Kutta method (RK4)?
\end{problem}
\end{comment}

Notice that these methods are doing a type of quadrature where we are sampling the function at different points and then performing computation using the samples and some inherent weights.

\begin{problem} Consider the IVP given by
\begin{align*}
y' + y &= 2-2x,\,\, 0 \leq x \leq 2, \\
y(0) &= 0.
\end{align*}
Use Euler's method, the Midpoint method, and RK4 to approximate the value of the solution at $x = 2$, with a stepsize of $h = 0.2,$ $ 0.1,$ $0.05 $, $0.025,$ and $0.0125.$ Create a log-log plot of the relative error of each approximation using the \li{loglog} function in \li{matplotlib}.(see Figure \ref{ivp:relative_error}).
\end{problem}

% \begin{problem}
% Plot the solutions of
% \[ y' + y = 2-2x\,\, 0 \leq x \leq 2, \]
% with initial conditions $y(0) = 0, 2, 4, 6, $ and $8$. Use RK4 to compute the solutions.
% \end{problem}

\section*{Harmonic Oscillators and Resonance} Harmonic oscillators show up often in classical mechanics.
A few examples include the pendulum (with small
 displacement), spring-mass systems, and the flow of electric current through various types of circuits.
A harmonic oscillator can be described by an initial value problem of the form
\begin{align*}
	my'' + \gamma y' + ky &= f(t) ,\\
	y(0) &= y_0,\\
	y'(0) &= y'_0.
\end{align*}

We will describe the construction of this mathematical model in the context of a spring-mass system.

Suppose an object with mass $m$ is placed at the end of a horizontal spring.
The natural position of the object is called the \textit{equilibrium position} for the system.
If the object is displaced from its equilibrium position and given an initial velocity,
it will act like a harmonic oscillator.
The principal property of a harmonic oscillator $y(t)$ is that once $y$ leaves its equilibrium value $y = 0$, it experiences a restoring force $F_r = -ky.$
This force pushes $y$ back towards its equilibrium.
Hooke's law says that this holds true for a
spring-mass system if the displacement $y$ is small.

Often there is an additional damping force $F_d$, often due to some type of friction. This force is usually proportional to the $y'$, is always in the opposite direction of $y'$ (the \emph{velocity}. You can think of it as drag), and represents energy leaving the system.
Thus we have $F_d = -\gamma y', $ where $ \gamma \geq 0$ is constant.
We may also need to consider an additional external force $f(t)$, or a driving force, that is interacting with our spring-mass system.

By using Newton's law we obtain
\begin{align*}
ma &= F = F_r + F_d + f(t),\\
my'' &= -ky -\gamma y' + f(t).
\end{align*}

\section*{Simple harmonic oscillators}
A simple harmonic oscillator is a harmonic oscillator that is not damped ($\gamma =0$), and is free ($f(t) \equiv 0$) rather than forced ($f(t) \not = 0$). A simple harmonic oscillator can described by the IVP
\begin{align*}
my'' + ky &= 0,\\
y(0) &= y_0,\\
y'(0) &= y_0'.
\end{align*}
The solution of this IVP is $y = c_1\cos (\omega_0 t) + c_2 \sin (\omega_0 t)$ where $\omega_0 = \sqrt{k/m}$ is the natural frequency of the oscillator and $c_1$ and $c_2$ are determined by applying the initial conditions. This in turn can be written in the form
\[y = A\sin (\omega_0 t + \delta) .\]

To solve this IVP using the fourth order Runge Kutta method (RK4), we need to write this system in the form
\[z'(t) = f(t,z(t)) \]
We can do this by letting $z_1 = y, z_2 = y'$. Then we have \[     z'=
 \left[\begin{array}{c}z_1 \\z_2\end{array}\right]'  =  \left[\begin{array}{c}z_2 \\\frac{-k}{m}z_1\end{array}\right]= f(z).\]

\begin{problem} Use the RK4 method to solve for the simple harmonic oscillator
described by
\begin{align*}
my'' + ky &= 0,\,\, 0 \leq t \leq 20, \\
y(0) &= 2, \\
y'(0) &= -1,
\end{align*}
for $m = 1$ and $k =1$. Note that in your implementation of RK4, the constants $K_1, K_2, K_3,$ and $K_4$ become vectors with $n$ entries, where $n$ is the number of equations in the first-order system.

Plot your solution $y(t)$.  Compare this with the solution of the IVP if  $m = 3$ and $k =1$. Consider: Why does the difference in solutions make sense physically?
\end{problem}

\section*{Damped free harmonic oscillators} We now consider damped free harmonic oscillators. These systems are described by the differential equation
\[my''(t) +\gamma y'(t) + ky(t) = 0.\]
For fixed values of $m$ and $k$, it is interesting to study the effect of the damping coefficient $\gamma$.

The roots of the characteristic equation are \[r_1,r_2 = \frac{-\gamma \pm \sqrt{\gamma^2 -4km}}{2m} .\]
Note that the real parts of $r_1$ and $r_2$ are always negative, and so any solution $y(t)$ will decay over time due to a dissipation of the system energy. There are several cases to consider for the general solution of this equation:
\begin{enumerate}
\item If $\gamma^2 > 4km$, then the general solution is $y(t) = c_1 e^{r_1t} + c_2e^{r_2t}$. Here the system is said to be $\textit{overdamped}$. Notice from the general solution that there is no oscillation in this case.
\item If $\gamma^2 = 4km$, then the general solution is $y(t) = c_1 e^{\gamma t/2m} + c_2 te^{\gamma t/2m}$. Here the system is said to be $\textit{critically damped}$.
\item If $\gamma^2 < 4km$, then the general solution is
\begin{align*}
y(t) &= e^{-\gamma t/2m} \left[c_1\cos(\mu t) + c_2 \sin (\mu t)\right],\\
&= R e^{-\gamma t/2m}  \sin (\mu t + \delta),
\end{align*}
where $R$ and $\delta$ are fixed, and $\mu = \sqrt{4km-\gamma^2}/2m.$ This system does oscillate.
\end{enumerate}

\begin{problem}
Use RK4 method to solve for the damped free harmonic oscillator
\begin{align*}
y'' +\gamma y'+ y &= 0, \,\, 0 \leq x \leq 20,\\
y(0) &= 1, \\
y'(0) &= -1.
\end{align*}
For $\gamma = 1/2,$ and $\gamma = 1$, simultaneously plot the solutions $y(t)$ and find $y(20)$ accurate to four significant digits. (Check that the relative error is less than $5 \cdot 10^{-5}$.)  How many subintervals do you need?
\end{problem}

\section*{Forced harmonic oscillators without damping}
Let's look at the systems described by the differential equation
\begin{align}
my''(t)  + ky(t) &= F(t). \label{Forced_harm_osc}
\end{align}
In many instances the external force $F(t)$ is periodic, so let us assume that $F(t) = F_0 \cos(\omega t)$. If $\omega_0 = \sqrt{k/m} \not = \omega,$ then the  general solution of \ref{Forced_harm_osc} is given by
\[y(t) = c_1 \cos (\omega_0 t) + c_2\sin (\omega_0 t) + \frac{F_0}{m(\omega_0^2 - \omega^2)} \cos (\omega t).\]
If $\omega_0 = \omega$, then the general solution is
\[y(t) = c_1 \cos (\omega_0 t) + c_2\sin (\omega_0 t) + \frac{F_0}{2m\omega_0} t \sin (\omega_0 t).\]
This last solution contains a term that grows arbitrarily large as $t \to \infty$.
If we included damping then the solution would be bounded, but will still be large for small $\gamma$ and $\omega$ close to $\omega_0$.
Consider a physical spring-mass system.
Equation \ref{Forced_harm_osc} holds only for small oscillations (this is where Hooke's law is applicable).
For larger oscillations, this equation will not hold.
However, the fact that the equation predicts large oscillations suggests the spring-mass system could fall apart as a result of the external force. Mechanical resonance has
been known to cause failure of bridges, buildings, and airplanes.

\begin{problem}
Use the RK4 method to solve for the undamped forced harmonic oscillator
\begin{align*}
2y'' + \gamma y' + 2y &= 2 \cos (\omega x), \,\, 0 \leq x \leq 40,\\
y(0) &= 2, \\
y'(0) &= -1.
\end{align*}
For the following values of $\gamma$ and $\omega,$ plot the solution $y(t)$ and find $y(40)$ correct to four decimal places:
\begin{align*}
	(\gamma, \omega) &= (0.5, 1.5),\\
	 &= (0.1, 1.1), \\
	&= (0.0, 1.0).
\end{align*}
\end{problem}


\chapter{Centrality measures}
\section{Some famous centrality measures}
The centrality of a node is a common problem of various fields such as biology, computer science, sociology, psychology. There are some old and popular measures to compute the centrality:
\begin{itemize}
	\item Closeness
	\item Betweenness
\end{itemize}
Next we examine some algorithms for computing them
\section{Closeness}
\subsection{Definition}
In words, the closeness says how a node is distant from the other nodes.\\
The formula for calculate is:
\begin{align*}
c(v) = \dfrac{n-1}{\sum_{u \in V} d(v,u)}
\end{align*}
A variant of this definition for non strongly connected components is
\begin{align*}
c(v)  = \dfrac{r(v) -1}{f(v)} \dfrac{r(v)-1}{n-1} = \dfrac{(r(v) - 1)^2}{(n-1)f(v)}
\end{align*}
 where $ r(v) $ is the set of reachable nodes from v. If the graph is strongly connected we have the previous definition, in fact
 \begin{align*}
 	\dfrac{(n - 1)^2}{(n-1)f(v)} = \dfrac{n-1}{f(v)}
 \end{align*}
 \subsection{A simple approach}
 The initial idea for computing the closeness is doing a BFS from all nodes to obtain all the distances but this have time complexity $ O(nm) $ which is too much for real graph.
 \section*{The pruned breadth first search}
  A similar problem is find the top k nodes with lower closeness but for doing this we need before the pruned breadth first search.\\
  \subsection{The upper bound on closeness}
 The pruned breadth first search use a lower bound on the farness of the nodes from v, that is equivalent to an upper bound of the closeness.\\
 Let's define the following terms:
 \begin{itemize}
 	\item $ \Gamma_d(v) = $ set of nodes at level \textit{d} of the BFS tree starting from \textit{v}
 	\item $ \gamma_d(v) = |\Gamma_d(v)| $
 	\item $ f_d(v) = \sum_{i=1}^{d} i \cdot \gamma_i(v) $
 	\item $ n_d(v) = \sum_{i=1}^{d} \gamma_i(v) $
\end{itemize}
Now let's define a lower bound on closeness:
\begin{center}
	$ f(v) \geq f_d(v) + (d+1) \cdot  \gamma_{d+1}(v) + (d+2)(r(v) - n_{d+1}(v))$
\end{center}
Since $ n_{d+1}(v) = \gamma_{d+1}(v) + n_d(v) $ we have
\begin{align*}
	 f(v) &\geq f_d(v) + (d+1) \cdot \gamma_{d+1}(v) + (d+2)\left[r(v) - (\gamma_{d+1}(v) + n_d(v))\right] \\
	 &= f_d(v) + (d+1) \cdot \gamma_{d+1}(v) - (d+2) \cdot \gamma_{d+1}(v) + (d+2)(r(v) - n_d(v)) \\
	 &= f_d(v)   - \gamma_{d+1}(v) + (d+2)(r(v) - n_d(v))
\end{align*}
We must consider that at the level \textit{d} we don't know how much is $ \gamma_{d+1}(v) $ so we use an upper bound that is the sum of degree of nodes on level \textit{d}:
\begin{align*}
	\gamma_{d+1}(v) \leq \sum_{u \in \Gamma_d(v)} degree(u) := \tilde{\gamma}_{d+1}(v)
\end{align*}
So we use the latter approximation on lower bound of farness $ f(v) $
\begin{align*}
	 f(v) &\geq \tilde{f}_d(v, r(v)) :=  f_d(v)   - \tilde{\gamma}_{d+1}(v) + (d+2)(r(v) - n_d(v))
\end{align*}
that is equivalent to an upper bound on closeness:
\begin{align*}
	c(v) \leq \dfrac{\left[r(v) - 1 \right]^2}{(n-1) \cdot  \tilde{f}_d(v, r(v)) } := \tilde{c}_d(v)
\end{align*}
\subsubsection{How to prune the breadth first search}
The pruned breadth first search receive in input the following arguments:
\begin{itemize}
	\item \textit{v}: the starting node of pruned BFS
	\item $ x_k $: the k-th biggest closeness value found until now (0 if not are processed at least k vertices)
\end{itemize}
then at each level \textit{d} of \textit{BFS} tree we check if $ x_k >  \tilde{c}_d(v)  \geq c_d(v)  $, and, if it is true, it means that it is not in the \textit{k} lowest values of closeness in the graph, so we can stop the \textit{BFS}.
\subsection{Compute the upper bound on closeness}
Once we have defined the algorithm in the latter section, then it remains one thing: calculate $ r(v) $ for each node.\\
We have four different cases:
\begin{itemize}
	\item Graph indirected and strongly connected: $ r(v) \rightarrow n \,\,\forall v \in V $
	\item  Graph indirected but not strongly connected: A linear time scan graph is needed then set $ r(v) = |SCC| : v \in SCC \,\, \forall v \in V $ that mean set $ r(v) $ to the size of strongly connected component where it belongs
	\item Graph directed and strongly connected :  $ r(v) \rightarrow n \,\,\forall v \in V $
	\item Graph directed but not strongly connected : see below
\end{itemize}
\subsubsection{Graph directed but not strongly connected}
Suppose that $ r(v) $ is included in the range $ [\alpha(v) , \omega(v)] $ a lower bound for $ \dfrac{1}{c(v)} $ is obtained simply bi reverting the upper bound in the latter section:
\begin{align*}
 \dfrac{1}{c(v)} \geq \dfrac{(n-1) \cdot \tilde{f}_d(v, r(v))}{(r(v) -1)^2}
\end{align*}
Then if we define $ a = d+2  $ and $ b = \tilde{\gamma}_{d+1}(v) + a(n_d(v) -1) - f_d(v) $ we have that
\begin{align*}
	f(v) \geq \tilde{f}_d(v, r(v)) &=  f_d(v)   - \tilde{\gamma}_{d+1}(v) + (d+2)(r(v) - n_d(v)) \\
	&=  f_d(v)   - \tilde{\gamma}_{d+1}(v) + a \cdot (r(v) - n_d(v)) \\
	&=  f_d(v)   - \tilde{\gamma}_{d+1}(v) + a \cdot (r(v) - n_d(v) + 1 - 1) \\
	&=  f_d(v)   - \tilde{\gamma}_{d+1}(v) + a \cdot (r(v) - 1))  - a \cdot n_d(v) + a\\
	&=  f_d(v)   - \tilde{\gamma}_{d+1}(v) + a \cdot (r(v) - 1))  + a (-  n_d(v) + 1)\\
	&=  a \cdot (r(v) - 1))   - \tilde{\gamma}_{d+1}(v)   - a (  n_d(v) -1)  + f_d(v)\\
	&= a \cdot (r(v) - 1))   - b
\end{align*}

so we can write the lower bound as
\begin{align*}
\dfrac{1}{c(v)} \geq \dfrac{(n-1) \cdot \tilde{f}_d(v, r(v))}{(r(v) -1)^2} =(n-1)\dfrac{a \cdot (r(v) - 1))   - b}{(r(v) -1)^2}
\end{align*}
Now consider the functions in the form $ g(x) = \dfrac{ax - b}{x^2} $ and the derivative $ g'(x) = \dfrac{-ax + 2b}{x^3} $: we can easily see that we have a zero in the derivative with $ x = \dfrac{2b}{a} $ and it is a local maximum because x is positive in range $ 0 < x < \frac{2b}{a} $ and negative in $ x > \frac{2b}{a} $. Then this it implies that for $ x > 0 $ there aren't local minimum so for any interval $ [x_1, x_2] $ the local minimum in the interval is $ x_1  $ or $ x_2 $ that in formula is $ \forall x \in [x_1, x_2] \,\, g(x)  \geq \min(g(x_1), g(x_2))$
\begin{figure}[H]
	\includegraphics[width=0.7\linewidth]{img/local_maximum}
	\caption{Plot of $ g(x) = \dfrac{ax -b}{x^2} $ with $ a = b = 1 $}
	\label{fig:localmaximum}
\end{figure}
Returning to the lower bound of $ \frac{1}{c(v)} $ we can use the same reasoning with $ (n-1)\dfrac{a \cdot (r(v) - 1))   - b}{(r(v) -1)^2} $ so we can say that
\begin{align*}
\dfrac{1}{c(v)} \geq (n-1) \cdot \dfrac{\tilde{f}_d(v, r(v))}{(r(v) -1)^2} \geq (n-1) \cdot \min\left( \dfrac{\tilde{f}_d(v, \alpha(v))}{(\alpha(v) -1)^2}, \dfrac{\tilde{f}_d(v, \omega(v))}{(\omega(v) -1)^2} \right)
\end{align*}

\subsection{Compute $ \alpha(v) \text{ and }  \omega(v) $}


\section{Representations of Counterpoint and Harmony}
\label{sec:cpharmony}

Harmony refers to simultaneously sounding tones (chords) and harmonic progression
a series of chords. It is in essence the of dual of counterpoint. A
fundamental representation used in Music Tools is a grid of
\texttt{Point}s, where a \texttt{Point} represents either a pitch, a
hold of a previous pitch, or a rest.

\begin{alltt}
data Point : Set where
  tone : Pitch \(\rightarrow\) Point
  hold : Pitch \(\rightarrow\) Point
  rest : Point
\end{alltt}

Music is thus considered to be quantized to some minimal duration,
with the horizontal dimension representing time and the vertical
dimension representing pitch. A slice in the horizontal direction is a
\texttt{Melody}, represented as a vector of points of some length $n$. Similarly a slice
in the vertical direction is a \texttt{Chord}, again of some fixed length.

\begin{alltt}
data Melody (n : \(\mathbb{N}\)) : Set where
  melody : Vec Point n \(\rightarrow\) Melody n

data Chord (n : \(\mathbb{N}\)) : Set where
  chord : Vec Point n \(\rightarrow\) Chord n
\end{alltt}

One can think of the grid as several melodies in parallel of the same
duration. This is essentially counterpoint, and thus given that
name. Note the crucial use of dependent types here to ensure all
melodies have the same duration $d$. The counterpoint is specified to
have $v$ voices.

\begin{alltt}
data Counterpoint (v : \(\mathbb{N}\)) (d : \(\mathbb{N}\)): Set where
  cp : Vec (Melody d) v \(\rightarrow\) Counterpoint v d
\end{alltt}

Dually one can think of the grid as a series of chords, in another
word a harmonic progression, although this is named \texttt{Harmony}
in the library for conciseness. Again dependent types enforce every
chord has the same number of voices.

\begin{alltt}
data Harmony (v : \(\mathbb{N}\)) (d : \(\mathbb{N}\)): Set where
  harmony : Vec (Chord v) d \(\rightarrow\) Harmony v d
\end{alltt}

Converting between \texttt{Counterpoint} and \texttt{Harmony} is
simply matrix transposition. These are fundamental representations
that can be used directly for analysis, for example, and there are
functions to convert them to MIDI for sound generation.

There are many other ways to represent music, which have advantages in
certain contexts. For example in Section~\ref{sec:cp}
\texttt{PitchInterval} and \texttt{PitchInterval2} were used instead
of \texttt{Counterpoint} since they more explictly represent the
structure of the music in species counterpoint. It is straightforward
to convert between the representations, and worth using the most
appropriate type for a given situation.


\chapter{Graphs}

\pgfplotsset{width=1\columnwidth,compat=1.9}

\begin{tikzpicture}
\begin{axis}
\addplot[color=red]{exp(x)};
\end{axis}
\end{tikzpicture}

\begin{tikzpicture}
\begin{axis}
\addplot3[surf,]
{exp(-x^2-y^2)*x};
\end{axis}
\end{tikzpicture}

\hskip 10pt

\begin{tikzpicture}
\begin{axis}[
axis lines = left,
xlabel = $x$,
ylabel = {$f(x)$},
]
%Below the red parabola is defined
\addplot [
domain=-10:10,
samples=100,
color=red,
]
{x^2 - 2*x - 1};
\addlegendentry{$x^2 - 2x - 1$}
%Here the blue parabloa is defined
\addplot [
domain=-10:10,
samples=100,
color=blue,
]
{x^2 + 2*x + 1};
\addlegendentry{$x^2 + 2x + 1$}

\end{axis}
\end{tikzpicture}


\chapter{Feedforward Neural Network}

Starting from the late 2000's, neural networks have been experiencing an exciting resurgence in the big data era. Given recent improvements in processing speeds, large-scale deep neural networks (DNNs), a.k.a. deep learning, are able to deliver highly accurate results when tackling several problem domains, including computer vision, natural language processing, speech recognition, and other AI challenges. For instance, the systems using DNNs have reported breakthroughs in object recognition accuracy on the ImageNet dataset, and have even achieved human-level performance for face recognition. Motivated by these encouraging results, there has been a growing interest from both academia and industry (Google, Facebook, Amazon, etc.), both of whom have been devoted significant resources to investigate, improve and promote exploration into DNNs. DNNs are revolutionizing a number of traditional and emerging real-world applications, such as self-driving systems, automatic machine translations, drug discovery and toxicology.

In this chapter we will introduce the basics of deep feedforward neural networks, define related terminologies, and analyze how to train and use neural network model.
\section{Feedforward Networks}
A \textit{network model} is a model expressed through compositions of function, where the composition relations forms an directed acyclic graph. Figure \ref{fig1} gives an example of a network model. In this example, $f_1$ and $f_3$ are composited by $f_2$ and $f_4$ separately, which in turn is composited $f_5$. Thus the expression of the entire model is
\begin{align*}
\label{enetwork}
f(\bm{x}) = f_5(f_2(f_1(\bm{x})), f_4(f_3(\bm{x})))
\end{align*}
In a network model, the input vector is denoted as $\bm{x}\in\mathbb{R}^m$. If the information flows through the function that evaluates $\bm{x}$, through each of the subsequent function, and finally goes to output vector $\bm{y}\in\mathbb{R}^n$. Such network is called a \textit{feedforward network}, as there is no feedback connections in which outputs of the model are fed back into itself.

\begin{wrapfigure}{r}{0.3\textwidth}
\centering
\includegraphics[width=0.3\textwidth]{Fig1}
\caption{Feedforward Network}
\label{fig1}
\end{wrapfigure}

Functions may form one or more layers according to which values they take as inputs (cf. Figure \ref{fig2}). The functions that directly take input $\bm{x}$ form the \textit{input layer}, functions that take values from input-layer functions form the \textit{first hidden layer}, and so on, until the last layer (call \textit{output layer}) which returns the output of the entire model. The total number of layers is called \textit{the depth} of the network. The \textit{width} of a layer is defined to be the number of functions on the layer. The functions are sometimes referred to as units, neurons, or activation functions, depending on the context. Nowadays neural networks with more than 1000 layers were used for tasks in computer vision, which justifies the name "deep learning". The training data do not provide information on what values the intermediate layers should produce, thus these layers are referred to as \textit{hidden layers}.

The information usually flow into a function from multiple sources. This is typically achieved by a weighted summation of the input values, possibly added with a bias value. For a single unit with activation function $\sigma$, the relation between the input vector $\bm x$ and the output vector $\bm y$ is given in the form of
\begin{equation}
    \bm y = \bm \sigma(\bm{W^Tx}+\bm\theta).
\end{equation}
Here $\bm W$ is a matrix of parameters representing the weighted summation of the input vector, $\bm\theta$ is a vector representing the bias value, and $\bm\sigma$ is the function that applies the activation function $\sigma$ to each of its input coordinates. For the ease of notation, we use new notations
$\bm x' = (\bm x^T, 1)^T$ and $\bm W' = (\bm W, \bm\theta^T)^T$ and simplify the notation above as
\begin{equation}
 \bm y = \bm \sigma(\bm{(W')^Tx'}).
\end{equation}
From now on we will drop the primes and stick with the simplified notations. As an example, the overall expression of the model illustrated in Figure \ref{fig2} can be expressed as

\begin{align*}
\bm{y} = \bm{W_3^T\cdot\sigma(W_2^T\cdot\sigma
(W_1^Tx))}
\end{align*}

The weight matrix related to the input of $j$-th layer is referred to as the weight matrix \textit{of that layer}. Every layer except for the input layer has a corresponding weight matrix.

\begin{wrapfigure}{r}{0.5\textwidth}
\centering
\includegraphics[width=0.5\textwidth]{Fig2}
\caption{Feedforward Network}
\label{fig2}
\end{wrapfigure}
%{\bf complicated network}
%Not every network model of interest is a layered feedforward network, as the topology of an acyclic graph can be quite complicated. Note that since there is no loop in the graph, one can still find a suitable order to let the information flow from input to output. Also there are network models that do contain directed cycles. An important class is called \textit{recurrent neural network}. We will study recurrent neural network at a later stage of the seminar.

The goal of a feedforward network is to approximate a function $g(\bm{x})$. More precisely, the goal is to find an appropriate set of weights so that the function representing the network is close to the target function $g$ (under certain measure). The challenge is to find a network model and a training method that is capable to well approximate the target function with limited memory space and computational power. The purpose the the next two sections is to introduce the basic setting of linear network models and neural network models, analyze their training procedures, and discuss the challenges of applying these models.

\section{Linear Feedforward Neural Network}
One way to understand feedforward networks is to begin with linear models and consider how to overcome their limitations. Linear models, such as logistic regression and linear regression, are appealing because they may be fit efficiently and reliably, either in closed form or with convex optimization. On the other hand, linear models also have the obvious defect that the model capacity is limited to linear functions, so the model cannot understand the interaction between any two input variables.

{\bf Linear Regression Model}. Linear Regression is the simplest form of regression. We model our system with a linear combination of features to produce one output. That is,
\begin{equation}
  f(\bm x, \bm w) = \bm w^T\bm x
\end{equation}
Here $\bm x$ is the input vector, and $w_0$, $\bm w$ are parameters. Our task is then to find the values for the parameters that provide the best fit to our training data. One way to measure the fitness of the parameters is to calculate the least squares error (loss function) over our dataset $S = \{(\bm x^1, y^1), ..., (\bm x^N, y^N)\}$
\begin{equation}
  L(S, \bm w) = \frac{1}{2}\sum_{i=1}^N(f(\bm x^i, \bm w)-y^i)^2
\end{equation}

In order to find the best fit, we must minimize $L(S, \bm w)$. The approach using gradient descent takes the following form:
\begin{equation}
  \bm w \leftarrow \bm w-\eta\nabla_{\bm w}L(S, \bm w)
\end{equation}
The step size $\eta$ may be application-dependent. The gradient of the loss function explicitly is expressed as
\begin{equation}
  \nabla_{\bm w}L(S, \bm w) = \sum_i(\bm w^T\bm{x^i} - y^i)\bm x^i = (\bm{w^TX-y})\cdot\bm X
\end{equation}
Here $\bm X = (\bm x^1,...,\bm x^N)$ and $\bm y=(y^1,...,y^N)$.
Note that there exists a closed form solution for linear regression, thus this gradient descent method is rarely used in practice.

{\bf Logistic regression model}. Another example on linear feedforward network is logistic regression model. We model our system as
\begin{equation}
%   \tanh(\bm w^T \bm x + b), \textit{ where } \tanh(a) = \frac{1-\exp(a)}{1+\exp(a)}
\frac{1}{1+\exp(\bm w^T \bm x)}
\end{equation}
With training set $S = \{(\bm x^1, y^1), ..., (\bm x^N, y^N)\}$, the loss function is defined by the negative log-likelihood
\begin{equation}
 L(\bm w) = \sum_{i=1}^N\log(1+\exp(-y^i(\bm w^T\bm x^i)))
\end{equation}
The loss function is minimized by the gradient descent
\begin{equation}
  \bm w = \bm w-\eta\nabla_{\bm w}L(\bm w)
\end{equation}
with $\nabla_{\bm w}L(\bm w) = -\sum_{i=1}^N y^i\frac{\exp(-y^i(\bm w^T\bm x^i+b))}{1+\exp(-y^i(\bm w^T\bm x^i+b))}\bm x^i$.

The convexity of the loss function guarantees that the gradient descent algorithm will find the minimum value.

{\bf Summary}.The key elements of a network model can be summarized as follows:
\begin{enumerate}
    \item A function representing a network structure.
    \item A loss function.
    \item An optimization algorithm that minimizes the value of the loss function.
\end{enumerate}

{\bf Limitations of Linear Network Model}. One important application of network models is the classification problem. A linear network is only capable of separating the dataset with hyperplanes. We are going to show a classical result on the limited capability of linear network models on classifying $p$ points in general position.

\begin{wrapfigure}{r}{0.4\textwidth}
\centering
\includegraphics[width=0.4\textwidth]{nonlinearseparable}
\caption{A non-linear dichotomy}
\label{fig4}
\end{wrapfigure}

When $n$ points are partitioned into two classes, we call such classification a \textit{dichotomy}. Moreover, we use the term \textit{linearly separable dichotomies} to represent the dichotomies that are realizable by a linear classifier (the boundary between the classes is a hyperplane). Figure \ref{fig4} shows an example of a non-linear dichotomy, as no straight line can separate the two blue points from the two red ones. In general, suppose that we have $p$ points in general position in $\mathbb{R}^N$. There are $2^p$ ways to partition these $p$ points into two classes, i.e., the $p$ points can form $2^p$ different dichotomies. The following theorem shows that the proportion of linear separable dichotomies will converge to 0 as $p\rightarrow\infty$.

% Cover's function counting theorem (1966)
% Suppose we have $p$ points in $\mathbb{R}^N$. Consider all possible partitions of these points into two classes. We have  $2^p$ such partitions. How many of there partitions yield linearly separable classes, i.e., where the two classes can be perfectly separated by an $(N-1)$-dimensional hyperplane? The only thing we assume about the points is that they are in general position, which means that any subset of $N$ or fewer points is linearly independent.

%  A dichotomy is a partition of a whole (or a set) into two parts (subsets). In other words, this couple of parts must be jointly exhaustive: everything must belong to one part or the other, and mutually exclusive: nothing can belong simultaneously to both parts.
% Such a partition is also frequently called a bipartition.

% The two parts thus formed are complements. In logic, the partitions are opposites if there exists a proposition such that it holds over one and not the other.

% Treating continuous variables or multicategorical variables as binary variables is called dichotomization. The discretization error inherent in dichotomization is temporarily ignored for modeling purposes.

\begin{theorem}[Cover, 1965]
Let $C(p, N)$ be the number of dichotomies of $p$ points in $N$-dimensional space that can be separated by a hyperplane. Then
\begin{align*}
\lim_{p\rightarrow\infty}\frac{C(p,N)}{2^p}=0.
\end{align*}
\end{theorem}

\begin{proof}
Firstly, we will find an explicit expression for $C(p, N)$ through induction. Suppose there are $p$ points located on the plane in general position, and we are trying to add one more point. There are two possibilities:

1) There exists a separating hyperplane that passes through the new point;

2) There does not exist a separating hyperplane that passes through the new point.

In case 1), No matter which class the new point belongs to, one can always shift the hyperplane infinitesimally so that this point locates on the same side of the hyperplane with other members of the class. Thus in this case each separable partitions in $p$ points gives rise to two separable partitions in $(p+1)$ points.

Now consider case 2). For any separating hyperplane, the new point must be on the same side with the same class of the previous $p$ points, as otherwise one can always construct a separating hyperplane that passes through the new point. Thus in this case each separable partitions in $p$ points will give rise to exactly on separable partitions in $(p+1)$ points.

Moreover, the number of separable partitions in case 1) is equal to $C(p-1, N)$, as one degree of freedom is eliminated by requiring passing through the new point. Therefore we obtain the following recursive expression

\begin{align*}
C(p+1, N) = 2C(p,N-1) + [C(p,N)-C(p, N-1)] = C(p,N) + C(p, N-1).
\end{align*}

Moreover, we have $C(1, N)=0$ for all $N<1$ and $C(1, N)=2$ for all $N\ge 1$. By induction we can prove

\begin{align*}
C(p+1,N) = 2\sum_{i=0}^{N-1}\big(^p_i\big).
\end{align*}
Here by convention we have $\big(^p_i\big)=0$ for $p<i$.

Finally derive
\begin{align*}
&\lim_{p\rightarrow\infty}\frac{C(p,N)}{2^p}
=2\sum_{i=0}^{N-1}\lim_{p\rightarrow\infty}\frac{\big(^p_i\big)}{2^p}
=2\sum_{i=0}^{N-1}0
=0.
\end{align*}
\end{proof}

It is possible to extend linear models to represent nonlinear functions to x by applying the model to a transformed input $\phi(x)$ or with the "kernel" trick. These techniques are beyond the scope of this chapter.

\section{Feedforward Neural Network}{\color{red}LZ: merge with chapter 1}
Neural networks are a computational approach that is based on a large collection of neural units (artificial neurons), loosely mimicking the way a biological brain solves problems with large clusters of biological neurons connected by axons. Each neural unit is connected with many others, and links can be enforcing or inhibitory in their effect on the activation state of connected neural units.

Proposed in 1940$'$s, neural networks are one of the most representative models in the artificial intelligence field. Simply speaking, a neural network tries to make a system that process information as a network of human neurons. Starting from late 2000s, neural networks(NNs) have been experiencing their exciting resurgence in the emerging big data era. The technique with the use of large-size deep neural networks (DNNs), a.k.a. deep learning, have produced state-of-the-art accuracy results on several well-known datasets in different artificial intelligent tasks, such as computer vision, natural language processing, speech recognition, etc.

{\bf Sigmoid function}. The sigmoid function is defined by
\begin{equation}
\sigma(t) = \frac{1}{1+e^{-t}}.
\end{equation}
The sigmoid function is monotonically increasing over $\mathbb{R}$ and is infinitely-differentiable. It has limit 0 and 1 as $t$ approaches $+\infty$ and $-\infty$, respectively. The graph of $\sigma(t)$ is shown in Figure \ref{fsig}.

A useful property of $\sigma(t)$ is that its derivative can be expressed by the function itself.

\begin{lemma}
\label{lsig}
$\sigma'(t) = \sigma(t)(1-\sigma(t))$.
\end{lemma}

\begin{figure}\label{fsig}
\centering
\includegraphics[width=0.3\textwidth]{Sigmoid}
\caption{Sigmoid Function}
\end{figure}

{\bf Neural network model}. In Deep Learning, sigmoid function is used as the activation function to introduce nonlinearity in the model. Inspired by biological neurons, an activation function basically works as a switch that can set the neuron on "ON" or "OFF". Besides the sigmoid function, commonly used activation functions are:

\begin{enumerate}
\item
Binary step function: $f(t) = \mathbbm{1}_{t\ge 0}$.
\item
Hyperbolic tangent function: $f(t) = tanh(t)=\frac{2}{1+e^{2t}}-1$.
\item
Arctan function: $f(t) = \tan^{-1}(t)$.
\item
Rectified linear unit (ReLU) $f(t) = max\{0, t\}$.
\end{enumerate}

A \textit{neural network} is a network model with non-linear unites on its hidden layers.

% To extend the learning model from linear to non-linear, we need to define non-linear unit, for example

% sigmoid function: expression, limits on both direction, derivative,

% "Derivative in terms of the output y":
% \begin{align*}
% y =& \frac{1}{1+e^{-z}}\\
% \frac{dy}{dz}=&\frac{-1(-e^{-z})}{(1+e^{-z})^2}\\
% =&\big(\frac{1}{1+e^{-z}}\big)\big(\frac{e^{-z}}{1+e^{-z}}\big)\\
% =&\big(\frac{1}{1+e^{-z}}\big)\big(1 - \frac{1}{1+e^{-z}}\big)
% =y(1-y)
% \end{align*}
% %Definition of activation function. Commonly used non-linear activation function: sigmoid, tanh, ReLU, binary step:

% A activation function basically works as a switch that can set the neuron on "ON"(1) or "OFF"(0). Commonly used activation functions are:

% The first one is sigmoid function:$\sigma_1(v)=\frac{1}{1+e^{-v}}$, if $\sigma_1(v)>0.5$, we take it as an "ON", otherwise we take it as an "OFF".The second one is hyperbolic tangent:$\sigma_2(v)=tanh^{-1}(v)$, if $\sigma_2(v)>0$, we take it as an "ON", otherwise we take it as an "OFF". The third one is linear function:$\sigma_3(v)$, for linear activation function, we have the similar setting for linear function just as the previous two activation functions.

{\bf Basic neural network model.} The first thing to do when defining a machine learning model is to set the \textit{hyper-parameters}.  For a neural network, hyper-parameters include the depth of the network $d$, the width of each layer $n$, the learning rate for gradient descent $r$, and the initial values for weight matrices $(\bm{W_1}(0),...,\bm{W_d}(0))$ and biases $(\bm{\theta_1}(0),...,\bm{\theta_d}(0))$.

 Suppose we have $m$ labeled training examples
$$\{(\bm{x^1, y^1}),...,(\bm{x^m, y^m})\}.$$ Let $\sigma_k$ ($\bm{W_k}$, $\bm{\theta_k}$, resp.) be the activation function (weight matrix, bias vector, resp.) for the $k-th$ hidden layer, $k=1,2,...,d-1$. Let $\bm{W_d}$ and $\bm{\theta_d}$ be the weight matrix and bias vector for the output layer.

The neural network model can be defined recursively as follows:
\begin{equation}
\label{enn}
\aligned
G_d(\bm{x}; \bm{W_1},...,\bm{W_d},\bm{\theta_1},..,\bm{\theta_d}) =& \bm{W_d^T}\bm{a_d}+\bm{\theta_d}\\
\bm{a_k} =& G_{k-1}(\bm{x}; \bm{W_1},...,\bm{W_{k-1}},\bm{\theta_1},..,\bm{\theta_{k-1}})\\
G_{k}(\bm{x}; \bm{W_1},...,\bm{W_k},\bm{\theta_1},..,\bm{\theta_k})=&
\sigma(\bm{W_k^Ta_k}+\bm{\theta_k}), k=2,...,d\\
G_1(\bm{x};\bm{W_1},\bm{\theta_1})=&\sigma(\bm{W_1^Tx}+\bm{\theta_1}).
\endaligned
\end{equation}

Here we use the convention $\sigma(x_1,...,x_n) = (\sigma(x_1),...,\sigma(x_n))$. When no confusion will occur, we will abbreviate $G_k(\bm{x}; \bm{W_1},...,\bm{W_d},\bm{\theta_1},..,\bm{\theta_d})$ as $G_k(\bm{x}; \bm{W_d},\bm{\theta_d})$ or even $G_k(\bm{x})$.

{\bf Example: Representing XOR function}.
An \textit{XOR function} ("exclusive or") is an operation on two binary values $x_1$ and $x_2$. When exactly one of these binary values is equal to 1, the XOR function return 1. Otherwise, it returns 0. Mathematically the XOR function can be expressed as:
\begin{equation}
    f(x_1, x_2) = |x_1 - x_2|.
\end{equation}
Our model provides a function $y = f(x_1, x_2;\bm{W_1}, \bm{W_2})$ and our learning algorithm will adapt the parameters $\bm{W_1}, \bm{W_2}$ to make $f$ as similar as possible the the XOR function.
\begin{wrapfigure}{r}{0.5\textwidth}
\centering
\includegraphics[width=0.25\textwidth]{XOR}
\caption{Representing XOR Function}
\label{fig3}
\end{wrapfigure}
Figure \ref{fig3} defines the network model with one hidden layer that returns exactly the same value as an XOR function on every element of the domain $\{(0,0), (0,1), (1,0), (1,1)\}$. Here the activation functions on the hidden layer are "perceptrons" $\mathbbm{1}_{x\ge a}$, where $a$ is the number inside each neuron. The expression of the neural network is
\begin{equation}
  f(x_1,x_2) = \mathbbm{1}_{x\ge 1}(x_1) -2\mathbbm{1}_{x\ge 2}(x_1+x_2) + \mathbbm{1}_{x\ge 1}(x_2).
\end{equation}
Equivalently, if we use notations
\begin{equation}\label{exor}
    \bm x = (x_1, x_2)^T,
    \bm{W_1} = \left[
    \begin{array}{cc}
        1 & 0 \\
        1 & 1 \\
        0 & 1
    \end{array}
    \right],
    \bm{W_2} = \left[
    \begin{array}{c}
        1  \\
        -2 \\
        1
    \end{array}
    \right],
\end{equation}
then equation \eqref{exor} is equivalent to the expression obtained from the general definition following equation \eqref{enn}

% Marvin Minsky and Seymour Papert (1969) showed that it was impossible for a single layer perceptron network to learn an XOR function.
In this example, we simply specified the solution, then showed that it obtained zero error. In a real situation, there might be billions of model parameters and billions of training examples, so one cannot simply guess the solution as we did here. Instead, a gradient-based optimization algorithm can find parameters that produce very little error.

% {\bf Backpropagation}. Backpropagation is a common method of training artificial neural networks and used in conjunction with an optimization method such as gradient descent. The algorithm repeats a two phase cycle, propagation and weight update. When an input vector is presented to the network, it is propagated forward through the network, layer by layer, until it reaches the output layer. The output of the network is then compared to the desired output, using a loss function, and an error value is calculated for each of the neurons in the output layer. The error values are then propagated backwards, starting from the output, until each neuron has an associated error value which roughly represents its contribution to the original output.

% {\bf Loss function}. For backpropagation to work, two assumptions are made about the form of the error function. The first is that it can be written as an average $E = \frac{1}{n}\sum_xE_x$ over error functions $E_x$, for individual training examples, $x$. The reason for this assumption is that the backpropagation algorithm calculates the gradient of the error function for a single training example, which needs to be generalized to the overall error function. In practice, training examples are placed in batches, and the error is averaged at the end of the batch, which is then used to update the weights. The second assumption is that it can be written as a function of the outputs from the neural network.

% {\bf Example loss function}. Let $y$, $y'$ be vectors in $\mathbb R^n$. Select an error function $E(y, y')$ measuring the difference between two outputs. The standard choice is $E(y, y') = 1/2\|y-y'\|^2$. The error function over $n$ training examples can be written as an average:
% \begin{equation}
%   E=\frac{1}{2n}\sum_x\|y(x)-y'(x)\|^2
% \end{equation}
% In order to demonstrate the idea of linear feedforward network, we give two examples

% %Gradient computation:
% We use Back Propagation to compute the gradient of loss function $(L)$. Firstly, we have a training set by which we can train our model: Training set:$\{(x^i,y^i)|i=1,...,P\}$. Then we introduce the approximating function: $F(x)=\sigma(W^Tx+\theta)$. Namely, we use $\{(x^i, \hat{y}^i)|\hat{y}^i=F(x^i)\}$  to approximate the relationship between $x^i$ and $y^i$. After defining the approximating function, we want to compute and minimize the bias(or distance) between our approximation $\hat{y}^i$ and the real data $y^i$, so we introduce the loss function (L): L=$\frac{1}{2}\Sigma_{i=1}^{P}(y^i-\hat{y}^i)^2=\frac{1}{2}\Sigma_{i=1}^{P}(y^i-F(x^i))^2$. After this step, our work switch to minimize the loss function (L). Usually, we use gradient descent method. Thus we must compute the gradient of our loss function. Back propagation is one of the most popular method we use to compute the gradient of loss function. The essence of back propagation is just variable substitution, we set: $a=W^Tx+\theta, f=\sigma(a)$. Then the computation of gradient by back propagation is just using chain rule to compute the gradient of loss function:

% \	$\frac{\partial{L}}{\partial{f^i}}=y^i-f^i$

% \	$\frac{\partial{L}}{\partial{a}}=\frac{\partial{L}}{\partial{f^i}}\frac{\partial{f^i}}{\partial{a}}=(y^i-f^i)\sigma'(a)=(y^i-f^i)\frac{1}{cosh^2(a)}$

% %Example: approximate quadratic function:

% Implement a neural network with two layers of fully connected weights, and use it to approximate quadratic function $f(x) = x^2$.

% Firstly, we obtain the training data, that is, generate 100 values of inputs, find the corresponding function value. And then, we determine hyper-parameters, with two layers of weights, there should be one hidden layer. Decide how many neurons there should be which will depend on the amount of data.

% The third step is to initialize weights, biases, and the learning rate. After that, we come to feedforward process which is to compute the output from each input value. We use the approximating function F to give the approximating value with respect to each point in training data. Then we want to compute the gradient of our loss function in order to minimize it. We often use back propagation, the steps of back propagation is defined in the previous section. Then we repeat feedforward process and back propagation several times when loss function becomes small enough. Now the model is trained. Finally, we test our neural network by generating another set of inputs and the corresponding value of f. Compute the loss of these test values, if the results are also small, we can say that our model is well-trained.

% %Improving the performance of back-propagation:

% There are several tricks to improve the performance of back-propagation: The first one is to avoid local minima which are not the global minima of our models. For example, we can make our loss function to become a convex function which has only one minimum. The second one is to keep derivatives from going to zero, if the derivative is very small, our decent efficiency becomes very low. The third choice is to compensate for error attenuation for deep layers or reduce learning rate when weights oscillate. Next one, we prefer to use small random initial random weights and small initial learning rate to avoid "herd effect", if the learning rate is large, the loss function may grow rather than decrease after each iteration.

\section{Training a neural network model}
{\color{red}LZ: repeated material removed}
% The error of a neural network with respect to a sample $(\bm{x},\bm{y})$ is defined by a loss function
% \begin{equation}
% \label{eloss}
% L(\bm{x}, \bm{y}; \bm{W_1},...,\bm{W_d},\bm{\theta_1},..,\bm{\theta_d}) = \frac{1}{2}\|\bm{y} - G_d(\bm{x})\|^2.
% \end{equation}

% The goal of training a neural network model is to find values for parameters $(\bm{W_1},...,\bm{W_d},\bm{\theta_1},..,\bm{\theta_d})$ such that the total loss
% \begin{equation}
% \label{etotalloss}
% L_{total}(\bm{W_1},...,\bm{W_d},\bm{\theta_1},..,\bm{\theta_d}) = \sum_{i=1}^m L(\bm{x^i}, \bm{y^i}; \bm{W_1},...,\bm{W_d},\bm{\theta_1},..,\bm{\theta_d})
% \end{equation}
% is very small. The parameters are adjusted according to $L_{total}$ using the following process called \textit{back-propagation}:
% \begin{equation}
% \label{ebp}
% \aligned
% \bm{\delta^d} =& \frac{\partial}{\partial \bm{a_d}}L_{total}(\bm{W_1},...,\bm{W_d},\bm{\theta_1},..,\bm{\theta_d})\\
% =& \sum_{i=1}^m \frac{\partial}{\partial \bm{a_d}}L(\bm{x^i}, \bm{y^i}; \bm{W_1},...,\bm{W_d},\bm{\theta_1},..,\bm{\theta_d})\\
% =& \sum_{i=1}^m \big(\bm{y^i} - G_d(\bm{x_i})\big)\\
% \bm{\delta^k} =& \frac{\partial}{\partial \bm{a_k}}L_{total}(\bm{W_1},...,\bm{W_d},\bm{\theta_1},..,\bm{\theta_d})\\
% =& \frac{\partial L_{total}}{\partial \bm{a_{k+1}}}\frac{\partial \bm{a_{k+1}}}{\partial \bm{a_k}}\\
% =& \bm{\delta^{k+1}}\bm{W_k^T}\sigma'(\bm{W_k^Ta_k}+\bm{\theta_k}), k=1,...,d-1.
% \endaligned
% \end{equation}
% For sigmoid activation function, we can apply Lemma \ref{lsig} and obtain
% \begin{equation}
% \label{ebpsig}
% \bm{\delta^k} =\bm{\delta^{k+1}}\bm{W_k^T}\circ \sigma(\bm{W_k^Ta_k}+\bm{\theta_k}) \circ (\bm{1} - \sigma(\bm{W_k^Ta_k}+\bm{\theta_k})), k=1,...,d-1.
% \end{equation}

% Here $\circ$ means element-wise multiplication of two vectors. The computation of $\bm{\delta^k}$ depends on the knowledge of $\bm{\delta^{k+1}}$, and hence the name back-propagation. The gradient of the loss function with respect to each parameter can be computed as follows: for each $k=1,...,d$ we have
% \begin{equation}
% \label{ebpparam}
% \aligned
% \frac{\partial}{\partial \bm{W_k}}L_{total}(\bm{W_1},...,\bm{W_d},\bm{\theta_1},..,\bm{\theta_d}) =& \frac{\partial}{\partial G_k}L_{total}(\bm{W_1},...,\bm{W_d},\bm{\theta_1},..,\bm{\theta_d})\bm{a_k}=\bm{\delta^k}\bm{a_k}\\
% \frac{\partial}{\partial \bm{\theta_k}}L_{total}(\bm{W_1},...,\bm{W_d},\bm{\theta_1},..,\bm{\theta_d}) =& \frac{\partial}{\partial G_k}L_{total}(\bm{W_1},...,\bm{W_d},\bm{\theta_1},..,\bm{\theta_d}) = \bm{\delta^k}
% \endaligned
% \end{equation}
% Here $G_0 = \bm{x}$. To obtain the expression in \eqref{ebpparam} we use the following properties of matrix differentiation
% \begin{equation}
% \label{ediff}
% \aligned
% \frac{\partial }{\partial \bm{A}}L(\bm{Ax + b}) =& \frac{\partial L}{\partial \bm{Ax + b}}\bm{X}\\
% \frac{\partial }{\partial \bm{b}}L(\bm{Ax + b}) =& \frac{\partial L}{\partial \bm{Ax + b}}
% \endaligned
% \end{equation}
% (Add proof)

% With the gradients computed, we use the following update rule to adjust the parameters: for $k=1,...,d,$
% \begin{equation}
% \label{ebpup}
% \aligned
% \bm{W_k} =& \bm{W_k} - r\frac{\partial L_{total}}{\partial \bm{W_k}}\\
% \bm{\theta_k} =& \bm{\theta_k} - r\frac{\partial L_{total}}{\partial \bm{\theta_k}}
% \endaligned
% \end{equation}
{\bf Complexity} The arithmetic complexity of the back-propagation process is dominated by the matrix multiplication in \eqref{ebp}. In practise variations of this algorithms such as \textit{stochasitc gradient descent} (SGD) is used to avoid large-scale matrix multiplication.

%\section{A Back-Propagation Example}
%We have seen a perceptron network with one hidden layer that represents the XOR function. In that case we hand-picked values for each parameter, while in practice a gradient-based optimization algorithm need to be applied. Here let's look at an example of finding a sigmoid network that approximate the XOR function.
%
%{\bf XOR function}. Recall that the XOR ("exclusive or") function $f^*(x_1,x_2)$ takes two binary values $x_1$ and $x_2$ as input. Thus the domain of $f^*$ is $\{(0,0),(0,1),(1,0),(1,1)\}$. When exactly one of $(x_1,x_2)$ is equal to 1, the function returns 1. Otherwise, it returns 0.
%
%For the learning of XOR function, we are going to use a neural network with two input units, one hidden layer with three sigmoid units, and one output unit. Additionally, the hidden layer and the output layer will include a bias. The weights and biases are usually initialized with random values. Here for the ease of illustration we set the initial values as:
%
%\begin{equation}
%\bm{W_1} = \left[
%\begin{array}{ccc}
%0.1 & 0.2 & 0.3\\
%0.4 & 0.5 & 0.6
%\end{array}
%\right]
%,
%\bm{W_2} = \left[
%\begin{array}{c}
%0.1\\
%0\\
%-0.1
%\end{array}
%\right]
%,
%\bm{\theta_1}= \left[
%\begin{array}{c}
%0.1\\
%0.1\\
%0.1
%\end{array}
%\right]
%,
%\bm{\theta_2} = 0.1.
%\end{equation}
%
%Note that in practice, the initial values for parameters are usually determined from a Gaussian distribution (possibly with normalization). This neural network model can be expressed mathematically as
%
%\begin{equation}
%\bm{y} = \bm{W_2^T}\sigma(\bm{W_1^Tx}+\bm{\theta_1})+\bm{\theta_2}
%\end{equation}
%
%{\bf The forward pass.} For each input tuple from $\{(0,0),(0,1),(1,0),(1,1)\}$, we calculate the output value (by convention $\sigma((x_1, ...,x_n)^T)=(\sigma(x_1),... \sigma(x_n))^T)$)
%\begin{enumerate}
%\item
%input: $\bm{x} = (0,0)^T$.
%Hidden layer:
%\begin{equation}
%\aligned
%\bm{a} =& \bm{W_1^Tx}+\bm{\theta_1} = (0.1,0.1,0.1)^T\\
%\bm{b} =& \sigma(\bm{a}) = (0.525,0.525,0.525)^T
%\endaligned
%\end{equation}
%Output layer:
%\begin{equation}
%\bm{y} = \bm{W_2^Tb}+\bm{\theta_2} = 0.1
%\end{equation}
%
%\item
%input: $\bm{x} = (0,1)^T$.
%Hidden layer:
%\begin{equation}
%\aligned
%\bm{a} =& \bm{W_1^Tx}+\bm{\theta_1} = (0.5,0.6,0.7)^T\\
%\bm{b} =& \sigma(\bm{a}) = (0.622,0.646,0.668)^T
%\endaligned
%\end{equation}
%
%Output layer:
%\begin{equation}
%\bm{y} = \bm{W_2^Tb}+\bm{\theta_2} = 0.0954
%\end{equation}
%
%\item
%input: $\bm{x} = (1,0)^T$.
%Hidden layer:
%\begin{equation}
%\aligned
%\bm{a} =& \bm{W_1^Tx}+\bm{\theta_1} = (0.2,0.3,0.4)^T\\
%\bm{b} =& \sigma(\bm{a}) = (0.550,0.574,0.599)^T
%\endaligned
%\end{equation}
%
%Output layer:
%\begin{equation}
%\bm{y} = \bm{W_2^Tb}+\bm{\theta_2} = -0.0951
%\end{equation}
%\item
%input: $\bm{x} = (1, 1)^T$.
%Hidden layer:
%\begin{equation}
%\aligned
%\bm{a} =& \bm{W_1^Tx}+\bm{\theta_1} = (0.6,0.8,1.0)^T\\
%\bm{b} =& \sigma(\bm{a} = (0.646,0.690,0.731)^T
%\endaligned
%\end{equation}
%
%Output layer:
%\begin{equation}
%\bm{y} = \bm{W_2^Tb}+\bm{\theta_2} = 0.0914
%\end{equation}
%\end{enumerate}
%
%{\bf Calculating the total error.} The set of inputs and outputs for XOR function includes
%\begin{align*}
%    \bm x^1 =& (0,0)^T, y^1 = 0\\
%    \bm x^2 =& (0,1)^T, y^2 = 1\\
%    \bm x^3 =& (1,0)^T, y^3 = 1\\
%    \bm x^4 =& (1,1)^T, y^4 = 0
%\end{align*}
%The expression for the neural network model is
%\begin{equation}
%G(\bm x; \bm W_1, \bm W_2, \bm \theta_1, \bm \theta_2) = \bm W_2^T\sigma(\bm W_1^T+\bm\theta_1)+\bm\theta_2.
%\end{equation}
%We can now calculate the error for each output neuron using the squared error function and sum them to get the total error:
%\begin{equation}
%L(\bm x; \bm W_1, \bm W_2, \bm \theta_1, \bm \theta_2) = \frac{1}{2}\sum_i (\bm{y^i} - G(\bm x^i; \bm W_1, \bm W_2, \bm \theta_1, \bm \theta_2))^2= 0.828
%\end{equation}
%
%Here the sum is over all four possible inputs. Note that The $\frac{1}{2}$ is included so that exponent is canceled when we differentiate later on. The result is eventually multiplied by a learning rate anyway so it doesn't matter that we introduce a constant here.
%
%{\bf The Backward Pass}
%Our goal with back-propagation is to update each of the weights in the network so that they cause the actual output to be closer the target output, thereby minimizing the error for each output neuron and the network as a whole.
%
%\begin{enumerate}
%\item
%Loss function: $L = 1/2\sum_i(\bm{y^i} - G(\bm{x^i}))^2$
%\begin{equation}
%\frac{\partial L}{\partial \bm{y}} = \sum_i \bm{y^i} - G(\bm{x^i}) = 0.0915
%\end{equation}
%\item
%Output layer: $\bm{y} = \bm{W_2^Tb}+\bm{\theta_2}$, $\bm{b} = \sigma(\bm{a})$
%\begin{equation}
%\aligned
%\frac{\partial L}{\partial \bm{W_2}} =& \bm{b}\big(\frac{\partial L}{\partial \bm{y}}\big)^T = \left[
%\begin{array}{c}
%-0.949\\
%-0.988\\
%-1.027
%\end{array}
%\right]
%\\
%\frac{\partial L}{\partial \bm{b}} =& \bm{W_2}\big(\frac{\partial L}{\partial \bm{y}}\big)=\left[
%\begin{array}{c}
%0.00915\\
%0     \\
%0.00915
%\end{array}
%\right]\\
%\frac{\partial L}{\partial \bm{\theta_2}} =& \frac{\partial L}{\partial \bm{y}}=-1.618\\
%\frac{\partial L}{\partial \bm{a}} =& \frac{\partial L}{\partial \bm{b}}\circ \sigma'(\bm{a}) = \frac{\partial L}{\partial \bm{b}}\circ \bm{b} \circ (\bm{1}-\bm{b})
%=\left[
%\begin{array}{c}
%0.0032\\
% 0.0002      \\
%-0.0022
%\end{array}
%\right]
%\endaligned
%\end{equation}
%Here $\circ$ represents element-wise multiplication, and $\bm{1}$ represents the vector with all coordinate equal to 1.
%\item
%Hidden layer:
%\begin{equation}
%\aligned
%\frac{\partial L}{\partial \bm{W_1}} =& \bm{x}\big(\frac{\partial L}{\partial \bm{a}}\big)^T = \left[
%\begin{array}{ccc}
%-0.0203 & 0        &  0.0199\\
%-0.0191 & 0        &  0.0182
%\end{array}
%\right]\\
%%\frac{\partial L}{\partial \bm{x}} =& \bm{W_1}\big(\frac{\partial L}{\partial \bm{a}}\big)\\
%\frac{\partial L}{\partial \bm{\theta_1}} =& \frac{\partial L}{\partial \bm{a}}=\left[
%\begin{array}{c}
%0.00209\\
%0\\
%-0.00180
%\end{array}
%\right]
%\endaligned
%\end{equation}
%\item
%Update parameters: fix learning rate $r = 0.01$
%\begin{equation}
%\aligned
%\bm{W_1} =& \bm{W_1} - r\frac{\partial L}{\partial \bm{W_1}}\\
%\bm{W_2} =& \bm{W_2} - r\frac{\partial L}{\partial \bm{W_2}}\\
%\bm{\theta_1} =& \bm{\theta_1} - r\frac{\partial L}{\partial \bm{\theta_1}}\\
%\bm{\theta_2} =& \bm{\theta_2} - r\frac{\partial L}{\partial \bm{\theta_2}}
%\endaligned
%\end{equation}
%\end{enumerate}
%
%After 30000 iterations, the total loss will drop to $6.97\times 10^{-9}$. See Figure \ref{fbp} for how the loss drops as iteration goes on.
%\begin{figure}[H]
%\label{fbp}
%\centering
%\includegraphics[scale=0.5]{Backpropfig}
%\caption{Total Loss after Each Iteration}
%\end{figure}
%
%The parameters now take values:
%\begin{equation}
%\bm{W_1} = \left[
%\begin{array}{ccc}
%1.070 & 1.055 & 3.465\\
%1.052 & 1.091 & 3.524
%\end{array}
%\right]
%,
%\bm{W_2} = \left[
%\begin{array}{c}
%-2.355\\
%-2.349\\
%3.488
%\end{array}
%\right]
%,
%\bm{\theta_1}= \left[
%\begin{array}{c}
%-1.208\\
%-1.233\\
%-0.650
%\end{array}
%\right]
%,
%\bm{\theta_2} = -0.125.
%\end{equation}
%And for each input the neural network will return:
%\begin{equation}
%\aligned
%(0,0)\mapsto 0.00004\\
%(0,1)\mapsto 0.99994\\
%(1,0)\mapsto 0.99994\\
%(1,1)\mapsto 0.00008
%\endaligned
%\end{equation}


%!TEX program = xelatex
\documentclass[12pt]{article}
\usepackage{amsthm}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{mathtools}
\usepackage{array}

\newcommand{\N}{\ensuremath{\mathbb{N}}}
\newcommand{\R}{\ensuremath{\mathbb{R}}}

\begin{document}

\section*{Problem 10}

\begin{proof}
  \begin{align*}
    \lim_{x \to 1^-}\frac{g(x)-g(1)}{x-1} = \lim_{x \to 1^-} 3(x+1) = 6.
  \end{align*}
  If we want $g(x)$ to be differentiable at $x=1$, we need $\lim_{x \to 1^+}\frac{g(x)-g(1)}{x-1}=6$ as well. For any $x>1$,
  \begin{align*}
    \frac{g(x)-g(1)}{x-1} &= \frac{a+bx - 3}{x-1} = \frac{bx +(a-3)}{x-1}.
  \end{align*}
  If $b\not=0$, then we can factor out $b$ and obtain
  \begin{align*}
    \frac{g(x)-g(1)}{x-1} &= \frac{b(x +\frac{a-3}{b})}{x-1}.
  \end{align*}
  If either $b=0$, or $b\not=0$ but $\frac{a-3}{b} \not=-1$, then the difference quotient does not converge as $x \rightarrow 1$. Therefore, $\frac{a-3}{b}=-1 \implies   a+b=3$. In this case,
  \begin{align*}
    \lim_{x \to 1^+}\frac{g(x)-g(1)}{x-1} = \lim_{x \to 1^+} \frac{b(x-1)}{x-1} = b.
  \end{align*}
  Therefore, we need to let $b=6$ and $a=3-b=-3$ for this function to be differentiable at $x=1$.
\end{proof}

\section*{Problem 17}

\begin{proof}
  For any $x\not=0$, we have
  \begin{align*}
    \frac{f(ax)-f(bx)}{cx} &= \frac{1}{c} \cdot \frac{f(ax)-f(bx)}{x}\\
                           &= \frac{1}{c} \cdot \frac{f(ax)-f(0)+f(0)-f(bx)}{x}\\
                           &= \frac{1}{c} \left[ \frac{f(ax)-f(0)}{x} - \frac{f(bx)- f(0)}{x} \right]
  \end{align*}
  Therefore,
  \begin{align*}
   \lim_{x \to 0} \frac{f(ax)-f(bx)}{cx}
   &= \lim_{x \to 0} \frac{1}{c} \left[ \frac{f(ax)-f(0)}{x} - \frac{f(bx)- f(0)}{x} \right] \\
   &= \lim_{x \to 0} \frac{1}{c} \left[ \lim_{x \to 0}\frac{f(ax)-f(0)}{x} - \lim_{x \to 0}\frac{f(bx)- f(0)}{x} \right] \\
   &= \frac{1}{c} \left[ \lim_{x \to 0}\frac{f(ax)-f(0)}{x} - \lim_{x \to 0}\frac{f(bx)- f(0)}{x} \right].
  \end{align*}
  We will first prove that $\lim_{x \to 0}\frac{f(ax)-f(0)}{x} =af^\prime(0)$. There are two cases:
  \begin{enumerate}
    \item If $a=0$, then for any $x \not=0$,
      \begin{align*}
        \frac{f(ax)-f(0)}{x} = \frac{f(0)-f(0)}{x} = 0.
      \end{align*}
      So $\lim_{x \to 0}\frac{f(ax)-f(0)}{x} = \lim_{x \to 0} 0 =af^\prime(0) $.
    \item If $a\not=0$, then we can define $h(x)=ax$, which is continuous at $x=0$. Since $a\not=0$, if $x \not= 0$, $h(x)=ax \not = h(0)$. Then, by Exercise 6, we obtain
      \begin{align*}
        \lim_{x \to 0}\frac{f(h(x))-f(h(0))}{h(x)- h(0)} = f^\prime(h(0)),
      \end{align*}
      which is
      \begin{align*}
        \lim_{x \to 0}\frac{f(ax)-f(0)}{ax} = f^\prime(0).
      \end{align*}
      Therefore,
      \begin{align*}
        \lim_{x \to 0}\frac{f(ax)-f(0)}{x} &= \lim_{x \to 0}a \cdot \frac{f(ax)-f(0)}{ax}\\
                                           &= \lim_{x \to 0}a \lim_{x \to 0}\cdot \frac{f(ax)-f(0)}{ax}\\
                                           &= af^\prime(0).
      \end{align*}
  \end{enumerate}
  Similarly, we can obtain $\lim_{x \to 0}\frac{f(bx)-f(0)}{x} =bf^\prime(0)$. Hence,
  \begin{align*}
   \lim_{x \to 0} \frac{f(ax)-f(bx)}{cx}
   &= \frac{1}{c} \left[ \lim_{x \to 0}\frac{f(ax)-f(0)}{x} - \lim_{x \to 0}\frac{f(bx)- f(0)}{x} \right] \\
   &= \frac{1}{c} \left[ a f^\prime(0)  - b f^\prime(0) \right] \\
   &= \left[ \frac{a-b}{c} \right]  f^\prime(0)
  \end{align*}
\end{proof}

\section*{Problem 5}

\begin{proof}
  By contradiction; suppose $f^{-1}$ is differentiable at $f(x_0)$, then because $f$ is differentiable at $x_0$, by the Chain Rule,
  \begin{align*}
    \left[ f^{-1}(f(x_0)) \right]^\prime &= \left( f^{-1} \right)^\prime \left(f(x_0)\right) \cdot f^\prime(x_0)\\
                                         &= \left( f^{-1} \right)^\prime \left(f(x_0)\right) \cdot 0\\ &= 0.
  \end{align*}
  However, since $f^{-1} \left( f(x) \right) =x$ for any $x \in I$ and $(x)^\prime = 1 \not=0$. We have reached a contradiction. Therefore, $f^{-1}$ is not differentiable at $x_0$.
\end{proof}

\section*{Problem 8}

\begin{proof}
  Since the sequence $\{x_n\}$ is strictly increasing and bounded, by the Monotone Convergence Theorem, $\{x_n\}$ converges to some number $x_0 = \sup \{x_n \,|\, n \in \N\}$. Because $x_n$ is strictly increasing, and $x_0 \geq x_n$ for any $n \in \N$, we can prove that $x_n < x_0$ for any $n \in \N$.

  By contradiction; suppose there is some $k \in \N$ such that $x_k = x_0$, then $x_{k+1} > x_k = x_0$, which contradicts the fact that $x_0$ is the supremum of $\{x_n\}$. Therefore, for any $n \in \N$, $x_n < x_0$.

  Since $f:\R \rightarrow \R$ is differentiable, $f$ is also continuous, so the sequence $f(x_n)$ converges to $f(x_0)$ as well. Because $\{f(x_n)\}$ is monotone increasing and convergent, $f(x_n)$ is bounded above by the Monotone Convergence Theorem. We can prove that the upper bound is $f(x_0)$, so that $f(x_n) \leq f(x_0)$ for any $n \in \N$.

  By contradiction; suppose there is some $k \in \N$ such that $f(x_k) > f(x_0)$. Then, by assumption, for any natural number $m \geq k$, $f(x_m) \geq f(x_k)$. However, since $f(x_n)$ is convergent, for any $\varepsilon > 0$, there exists some $N\in \N$ such that if $n \geq N$
  \begin{align*}
    |f(x_n) - f(x_0)| < \varepsilon \implies f(x_n)< f(x_0) + \varepsilon.
  \end{align*}
  Set $\varepsilon = f(x_k) - f(x_0) > 0$, then for any natural number $m \geq k$,
  \begin{align*}
    f(x_m) \geq f(x_k) = f(x_0) + \varepsilon
  \end{align*}
  We have reached a contradiction because for this particular $\varepsilon$ there could not exist any $N \in \N$ such that $n \geq N \implies f(x_n)< f(x_0)+\varepsilon$.
  Therefore, $f(x_0) = \sup \{f(x_n) \,|\, x \in \N\}$.

  Hence, for any $x \not=x_0$, because $x_n < x_0$, the difference quotient
  \begin{align*}
    \frac{f(x_n)-f(x_0)}{x_n-x_0}
  \end{align*}
  is always defined. Also, since $f(x_n) \leq f(x_0)$,
  \begin{align*}
    \frac{f(x_n)-f(x_0)}{x_n-x_0} \geq 0
  \end{align*}
  for any $n \in \N$.

  Because $f$ is differentiable, the difference quotient converges to some nonnegative number, so
  \begin{align*}
    f^\prime(x_0) = \lim_{n \to \infty} \frac{f(x_n)-f(x_0)}{x_n-x_0} \geq 0.
  \end{align*}
\end{proof}

\end{document}


\lab{Python}{The Top-flight Trio}{The Top-flight Trio}
\label{Lab:PythonAdvanced}

\section*{NumPy + SciPy}
Numpy provides an extremely efficient, multidimensional array datatype.  Mastery of this array datatype will be invaluable in your journey through numerical computing with Python.  Without NumPy arrays, Python would be too slow to be an effective tool.  The secret is that NumPy arrays are actually thin wrappers around C arrays.  Arrays use zero-based indexing, just like the rest of Python.  Many of the operations on NumPy arrays are almost as fast as they would be in C.  SciPy is a more feature rich environment than NumPy, but uses the array datatype provided by NumPy.  In these labs, we will almost always exclusively use the SciPy library.

To make the SciPy library available for use, you must import it.  We prefer to import SciPy into its own namespace for cleaner, clearer code.
\begin{lstlisting}[style=python]
: import scipy as sp
\end{lstlisting}

The best way to conceptually think of NumPy arrays is to visualize nested arrays.  For a $3\times4$ matrix, think of an array that contains three arrays each of length four.  Let's practice with a three dimensional matrix.

\[
A = \begin{bmatrix}
\begin{bmatrix}
0 & 1 \\
2 & 3 \\
4 & 5 \\
6 & 7 \\
8 & 9
\end{bmatrix}
\begin{bmatrix}
10 & 11 \\
12 & 13 \\
14 & 15 \\
16 & 17 \\
18 & 19
\end{bmatrix}
\begin{bmatrix}
20 & 21 \\
22 & 23 \\
24 & 25 \\
26 & 27 \\
28 & 29
\end{bmatrix}
\end{bmatrix}
\]

NumPy tracks each dimension by an axis number, also starting at $0$.  A one dimensional array has only one axis, a two dimensional, two axis.  An $n$ dimensional array has $n$ axis.  Our three dimensional array has dimensions $(3,5,2)$.  If we wanted to retrieve the value $16$, we would need to first retrieve index $1$ in axis $0$.
\[
A[1] = \begin{bmatrix}
10 & 11 \\
12 & 13 \\
14 & 15 \\
16 & 17 \\
18 & 19
\end{bmatrix}
\]

Then we need to retrieve index $3$ in axis $1$.
\[
A[1,3] = \begin{bmatrix}
         16 & 17 \\
         \end{bmatrix}
\]

Then we retrieve index $0$ of axis $2$.
\[
A[1,3,0] = 16
\]

\subsection*{Manipulating Arrays}
Array operations are, by default, element wise.  This includes multiplying arrays.

\begin{lstlisting}[style=python]
: a = sp.array(range(9)).reshape(3,3)
#flip a left right and up down.  We make b an upside down and backwards copy of a
: b = sp.flipud(sp.fliplr(a))
: a + b
array([[8, 8, 8],
       [8, 8, 8],
       [8, 8, 8]])
: a * b #this is NOT matrix multiplication
array([[ 0,  7, 12],
       [15, 16, 15],
       [12,  7,  0]])
: sp.dot(a, b) #this is matrix multiplication
array([[ 9,  6,  3],
       [54, 42, 30],
       [99, 78, 57]])
\end{lstlisting}

Unlike Python lists, arrays are allocated upon creation and are not dynamic.  It is expensive to change the dimensions of an array after it has been allocated.  We will demonstrate this by adding columns to an array vs allocating the entire array and changing the elements.

\begin{lstlisting}[style=python]
def allocateAll(n):
    #allocate an nxn array
    A = sp.zeros((n,n))
    for a in xrange(n):
        for b in xrange(n):
            A[a, b] = sp.math.cos(a*b)
    return A

def appendOne(n):
    A = sp.array([])
    for a in xrange(n):
        for b in xrange(n):
	    A = sp.append(A, sp.math.cos(a*b))
    return A.reshape(n, n)
\end{lstlisting}

\begin{lstlisting}[style=python]
: import mytimer
: with mytimer.timer() as timer:
:     for x in xrange(1, 50):
:         timer.time(appendOne, x)
:         timer.time(allocateAll, x)
: #let's plot the results
: import matplotlib.pyplot as plt
: sizes = range(1,50)
: plt.plot(sizes, timer.results['appendOne'], 'r')
: plt.plot(sizes, timer.results['allocateAll'], 'b')
: plt.show()
\end{lstlisting}

\begin{center}
\includegraphics[width=\textwidth]{alloc_vs_resize.pdf}
\end{center}

Why the big difference?  When appending to an array, a new array has to be created with room for the new entry.  Then each element of the old array has to be copied to the new array ending with the new value being inserted at the end of the array.  It is best to allocate space for the final array and change the elements in the array as needed.

\section*{Saving and Loading Arrays}
SciPy allows you to save arrays for later use and load them back again.
\begin{lstlisting}[style=python]
: A = sp.rand(9,9)
: B = sp.random.randint(50, size=(10,10))
: sp.save("testA", A) #saves a single array to a file
: sp.savez("testAB", A = A, B = B) #save multiple arrays to a file
: a = sp.load("testA.npy")
: z = sp.load("testAB.npz")
: z.keys() #the arrays referenced by key
\end{lstlisting}

\begin{problem}
Define a function that will generate an array with the multiplication table for a number, $n$.  Make the function as short and concise as possible.
\end{problem}

\section*{Matplotlib}
Matplotlib is a 2D plotting library for Python that can produce publication quality plots and graphs.  This library provides the visualization component to our numerical computing toolbox.

We import Matplotlib and make it available under the \li{plt} namespace.
\begin{lstlisting}[style=python]
: import matploblib.pyplot as plt
: x = sp.linspace(0, 2*sp.pi)
: plt.plot(x, sp.cos(x))
: plt.show()
\end{lstlisting}

Plotting with Matplotlib works much the same way as plotting with MATLAB.  Many of the command share similar names and accept similar inputs.  For a better idea of what Matplotlib can do, please visit their website at \url{http://matploblib.sourceforge.net}

\section*{Practice}
Let us look at the efficiency of NumPy arrays.  We will do this by implementing the Sieve of Eratosthenes using both arrays and lists.

\begin{problem}
Implement the Sieve of Eratosthenes using Python lists.  Make the implementation
\end{problem}


\section{Deep Learning: Basics}
\subsection{Basics}
\subsection{Activation Functions}
\subsection{Loss Functions}
\subsection{Backpropagation}
\subsection{Regularization Methods}
\subsection{Optimization Algorithms}
\subsection{Convolutional Networks}
\subsection{Recurrent Networks}
Recurrent networks allow us to work with sequences, where $x_t$ is the input
at time $t$, $h_t$ is the hidden state at time $t$, and $h_{t-1}$ is the
previous hidden state. RNNs allow us to propagate information through the
hidden state $h$. Hidden states defualt to zero.
\subsubsection{Elman RNN}
The Elman RNN is the simplest layers, yet prone to both the vanishing and
exploding gradient problem.
\begin{equation}
  h_t = \tanh \left( W_{ih} x_t + b_{ih} + W_{hh} h_{t-1} + b_{hh} \right)
\end{equation}
Here, $W$ are weight matricies, and $b$ are bias vectors, one each for the
input and hidden vectors.
\subsubsection{Long Short-Term Memory}
LSTM RNNs mitigate the vanishing gradient problem. They involve a more complex
set of equations corresponding to gates that control the amount of infomation
to propagate through the sequence. Here, $i$ is the input gate, $f$ is the
forget gate, $g$ is the cell gate, and $o$ is the output gate. LSTMs have
two hidden inputs, hidden state $h_t$ and cell state $c_t$.
\begin{equation}
  \begin{split}
    i_t \quad =& \quad \sigma \left( W_{ii} x_t + b_{ii} + W_{hi} h_{t-1} + b_{hi} \right) \\
    f_t \quad =& \quad \sigma \left( W_{if} x_t + b_{if} + W_{hf} h_{t-1} + b_{hf} \right) \\
    g_t \quad =& \quad \tanh \left( W_{ig} x_t + b_{ig} + W_{hg} h_{t-1} + b_{hg} \right) \\
    o_t \quad =& \quad \sigma \left( W_{io} x_t + b_{io} + W_{ho} h_{t-1} + b_{ho} \right) \\
    c_t \quad =& \quad f_t \circ c_{t-1} + i_t \circ g_t \\
    h_t \quad =& \quad o_t \circ \tanh \left( c_t \right) \\
  \end{split}
\end{equation}
Here, $\sigma$ is the sigmoid function.
\subsubsection{Gated Recurrent Unit}
GRU RNNs are another rnn layer designed to mitigate the vanishing gradient
problem. Here, we have the $r$ reset gate, $z$ update gate, and $n$ is the
new gate. $\sigma$ is the sigmoid function.
\begin{equation}
  \begin{split}
    r_t \quad =& \quad \sigma \left( W_{ir} x_t + b_{ir} + W_{hr} h_{t-1} + b_{hr} \right) \\
    z_t \quad =& \quad \sigma \left( W_{iz} x_t + b_{iz} + W_{hz} h_{t-1} + b_{hz} \right) \\
    n_t \quad =& \quad \tanh \left( W_{in} x_t + b_{in} + r_t \circ \left( W_{hn} h_{t-1} + b_{hn} \right) \right) \\
    h_t \quad =& \quad \left(1-z_t\right) \circ n_t + z_t \circ h_{t-1} \\
  \end{split}
\end{equation}
\subsubsection{Bidirectional RNNs}
Bidirectional RNNs run two separate RNN layers on the forward and reverse
sequence, and then concatenates them into one output vector.
\begin{equation}
  \begin{split}
    \overrightarrow{h_f} \quad =& \quad \operatorname{RNN}\left(\overrightarrow{x}\right) \\
    \\
    \overleftarrow{h_r} \quad =& \quad \operatorname{RNN}\left(\overleftarrow{x}\right) \\
    \\
    h_o \quad =& \quad \left[\overrightarrow{h_f} \,;\, \overrightarrow{h_r} \right] \\
  \end{split}
\end{equation}
Where $\operatorname{RNN}$ can be any of the 3 previously mentioned layers.
\subsubsection{Vanishing/Exploding Gradient}
The vanishing and exploding gradient phenomena are often encountered in the
context of RNNs. The reason why they happen is that it is difficult to
capture long term dependencies because of multiplicative gradient that can
be exponentially decreasing/increasing with respect to the number of layers.
\subsubsection{Gradient Clipping}
Gradient clipping is a technique used to cope with the exploding gradient
problem sometimes encountered when performing backpropagation.
By capping the maximum value for the gradient, this phenomenon is
controlled in practice.
\begin{equation}
  \nabla \mathcal{L}_{clipped} = \min \left( \nabla \mathcal{L}, \, C  \right)
\end{equation}
For some max value $C$.