\section{Complex networks}

Graph theory is a well established area of mathematics. More recently, other areas from physics to economics, sociology to ecology have found that the structures and properties of graphs give a powerful and practical method of representing systems of interest to them. This new focus on what might have been called applied graph theory is what is now mostly know as network science. One of the interesting aspects of network science is that it deals with structures that are non-regular (not lattice-like) but also not purely random. These heterogeneous patterns of connections have implications for the study of complex systems (systems of entities with non-trivial sets of typically non-linear interactions) and hence networks often get referred to as complex networks. The Complex part of the name is really a bit redundant, but it sounds good and is commonly used, so we'll stick with it.

\subsection{An introduction to graph theory \& networks}

A network or \emph{graph} is a collection of \emph{vertices} (nodes) $V=\{v_i\}$ and \emph{edges} (links) $E=\{e_{ij}\}$ which we denote $G= G(V,E)$.

If we consider the edge $e_{ij}$, between vertices $v_i$ and $v_j$ to be distinct from $e_{ji}$ then we say that the edge is \emph{directed}. A graph is directed if all its edges are directed. A \emph{simple graph} is a network (possibly directed) where there is a unique (directed) edge between connected pairs of nodes. A \emph{multigraph} allows for the possibility of multiple edges edges between the same nodes. We will mostly only consider simple graphs, though much of what we'll present also holds for multigraphs, with a slight adjustment.

The \emph{density} of a graph is the number of edges present in the graph divided by the number of possible edges. For an undirected simple graph with $|V|=n$ nodes, the density is given by
$$
	\rho = \frac{|E|}{\frac12 n(n-1)},
$$
where $|E|$ is the number of edges and where the factor of $\frac12$ comes from the fact that the edges are undirected so $e_{ij}=e_{ji}$.

The \emph{degree} of a node in a network, often denoted $k$ or $k_i$ for the degree of node $i$ is the number of edges that connect to that node. In the case of a directed network we have both \emph{in-degree} and \emph{out-degree}, the number of incoming and outgoing links, respectively.

The \emph{component} of a graph to which a vertex belongs is the set of vertices that can be reached from it by paths running along the edges of the graph. A node in a directed graph has both an \emph{in-component} and an \emph{out-component} corresponding to the nodes that it can be reached from and the nodes that can be reached from it. A graph need not consist of only a single component, though some properties are ambiguously defined when it does not.

The \emph{geodesic path} between a pair of nodes is the shortest path through the network that connects them. The shortest path need not be unique and for a directed network the shortest path can differ depending on direction.
It is sometimes interesting to look at the \emph{average shortest path} for a graph. That is the average over all possible shortest paths in the graph, computed for each pair of nodes.

The notation $e_{ij}$ for the edges of a network suggests that we can represent a network as a matrix. If we let the rows and columns of the matrix represent the vertices of the network, then the non-zero entries of the matrix represent edges between pairs of vertices. We call such a network the \emph{adjacency matrix} of the graph. The values of the non-zero entries can be used to denote \emph{edge weights} of a \emph{weighted graph}.

Many real-world networks consist of nodes of different types, with connection only between nodes of certain types. The simplest example of this is a \emph{bipartite network} (also known as a \emph{two-mode network}): a network of two types of vertices $V=\{v_i\}$ and $U=\{u_j\}$ where the edges only connect nodes of type $v$ to nodes of type $u$. For example, a publication network where $V$ represents authors and $U$ their publications.  (More generally, the two node types are often referred to as agents and artifacts.) Edges link authors to their publications but not publications to publications, nor authors to authors. In order to obtain a co-authorship network from this publication network it would be necessary to calculate the \emph{unipartite} or \emph{one-mode projection} of the bipartite network.

\subsection{Real-world networks: some examples}

{\bf Social networks:}

A social network is a set of people or groups with some pattern of contacts or interactions between them. They include such diverse examples as friendship networks of dolphins and interactions between characters in works of fiction, through to more traditional networks such as friendships between individuals, either in real life, or on Facebook, employment networks, and business relationships between companies.

Some of these networks, like the business relationships are best viewed as undirected networks, while others, like real-life friendship networks are directed. Friendship networks lend their name to an interesting feature of networks: the so called \emph{friendship paradox} which states that your friends will typically have more friends than you do. This arises from the fact that like many other real-world networks, friendship networks have a heavy-tailed degree distribution so picking a random node and following an edge from it is likely to lead you to a node with higher degree than the starting node.

\emph{Contact networks} are one important example of social networks. These are networks of when people are in close enough proximity to allow for a disease to spread between them. (Close enough obviously depends on the type of disease.) Contact networks play an important role in modeling of disease contagion --- for example the recent Ebola outbreak in West Africa was successfully modeled using contact networks and this played an important role in managing the epidemic.

{\bf Information networks:}

These could equally be called knowledge networks. Examples include the publication networks mentioned earlier (bipartite, undirected), citation networks (directed) whether as citations of publications or of patents, and the World Wide Web (directed). (Note --- the WWW should not be confused with the internet; one is a collection of electronic documents that can link to one another, the other is a collection of computers and switches connected by cables.) Preference networks are another example of information networks that are often used in real-world applications. Examples include networks of people linked to their Netflix viewings, or their Amazon purchases. Identifying patterns in such networks and predicting likely new links is a highly active area of both academic and commercial interest.

{\bf Technological \& infrastructure networks:}

These are networks with physical links between nodes. Examples include the internet (undirected); electricity grids (partly directed) perhaps also including the computers that control the grid; road, rail, shipping, or air transport networks; and electronic circuits.

{\bf Biological networks:}

These networks span multiple scales, from networks of interactions between proteins and their metabolites, or genes and proteins (both directed) that are used amongst other things, in the study of disease through to networks of interactions between species. The latter includes food webs (directed) and mutualistic networks like pollination networks (bipartite, directed). Neural networks are another example that is an active area of study; the complete neural network (282 neurons) is known for the nematode C. Elegans. Studying bigger brains, however, requires bigger brains. Though not entirely biological, I'll include in this class a couple of examples of physical networks. These are networks linking free energy minima of glasses via the saddle points on the energy surface that they sit on and networks linking confirmations of polymers when they can transition between a pair of structures.

\subsection{Properties of networks}

{\bf Small world effect:}

One of the best know network properties is the \emph{small world effect}. This gets it name from a famous experiment in the 1960s by the sociologist Stanley Milgram. In the experiment, letters were passed from person to person until they reached a designated target person. Milgram found that it took on average only about six hops for the letters to reach their destination leading to the so called \emph{six degrees of separation} phenomenon. This small world effect is an example of how particular network structures can provide short paths through the network connecting most pairs of nodes.

The mean geodesic path length for an undirected network is given by
$$
	L =\frac{1}{\frac12 n(n-1)}\sum_{i\geq j} d_{ij}
$$
where $d_{ij}$ is the geodesic distance between $v_i$ and $v_j$. In the case of disconnected networks, one must make a choice about how to count the contribution of pairs of nodes that are not connected. The Facebook friendship network is one of the best examples of a small world network. Despite having around 1 billion nodes and 100 billion links (i.e. a mean degree of 100) the mean geodesic distance for the Facebook network is only around four.

{\bf Transitivity/clustering:}

In mathematics, a set with a relationship $\sim$ is transitive if $a \sim b,~b\sim c \implies a \sim c$. In the context of networks the relationship means "is connected to". If a network is entirely transitive, then it would consist of totally connected components. More interesting is to measure how close a network is to being transitive by looking at how much clustering it has.

The \emph{global clustering coefficient} for a network is defined as
$$
	C=\frac{3\times\text{\# of triangles}}{\text{\# number of connected triples}}
$$
where a connected triple is a vertex with connections to two other vertices.

It is sometime interesting to look at how the clustery-ness of a network varies over the network, perhaps to study how clustering depends on some node property. To look at this, we use the \emph{local clustering coefficient} for a vertex $v_i$:
$$
	C_i = \frac{\text{\# triangle connected to }v_i}{\text{\# triples centered on }v_i}.
$$
Clearly, $C = \frac{1}{N}\sum_{i=1}^NC_i$.

Now that we've defined the clustering coefficient, we can look a little closer at the small world property.
If we start with a completely regular network (figure \ref{fig1}a) where each node has the same degree $k$ and is connected to its $k$ nearest neighbours we will have a high clustering coefficient, but a relatively long mena geodesic path. If we start to introduce random re-wirings of the network, with probability $p$, by either switching an end of an existing edge (figure \ref{fig1}b) or by introducing new random links (figure \ref{fig1}c), then we find that the mean geodesic path length drops quickly, while the mean clustering coefficient remains close to its original value as the re-wiring probability $p$ increases --- see figure \ref{fig2}. This means that while the long-range connections rapidly reduce the \emph{global} mean path geodesic length, any change to the \emph{local} structure is barely perceptible.

\begin{figure}
	\begin{center}
		\includegraphics[width = 12cm]{regularNW.png}
	\end{center}
	\caption{}
	\label{fig1}
\end{figure}

\begin{figure}
	\begin{center}
		\includegraphics[width = 8cm]{smallWorld.png}
	\end{center}
	\caption{Mean geodesic path length (solid line) and mean clustering coefficient (dashed line) as a function of the rewiring probability $p$, and measured relative to their maximum value, i.e. with no re-wiring.}
	\label{fig2}
\end{figure}

{\bf Degree distribution:}

The degree $k$ of a vertex is the number of edges that connect to that vertex. We write $p_k$ for the fraction of nodes in a network that have degree $k$, this is also the probability that a randomly selected node has degree $k$. One of the interesting properties of real-world networks is the degree distribution --- the histogram of $p_k$ as a function of $k$. For a network with uniformly random connections between nodes, the degree distribution is Poisson. This is not the case for most real-world networks --- these tend to have degree distributions that are highly right-skewed. In particular, many real-world networks tend to have a \emph{heavy-tail} --- a probability distribution that has a non-negligible probability of getting values much greater than the mean. In fact, the mean (and higher order moments like the variance) may not even be well defined for all heavy-tailed distributions.  Of particular interest is the case where the degree distribution is a \emph{power-law} or \emph{scale-free}. That is
$$
 p_k \sim k^{-\gamma}.
$$

Such degree distributions show up surprisingly often in real world networks: the WWW, the internet, metabolic networks, telephone call networks, and sexual contact networks. Power law degree distributions also arise from one of the most popular models for generating networks --- we'll see more on this later.

An alternative (i.e. non-power-law) distribution that is still heavy-tailed is the \emph{exponential distribution}
$$
	p_k\sim\exp(-k/\kappa).
$$

Power-law and exponential distributions are easily identified in data: they give a straight line on log-log and semi-log axes, respectively.

The probability generating functions that we introduced at the start of the course give a useful way of calculating network properties from their degree distributions. For example, if $G_0(x) = \sum_{k=0}^\infty p_kx^k$ is the PGF for the degree distribution of a network, then the first moment of the PGF gives the average degree, $\langle k \rangle$, of the network, according to
$$
	G_{0}'(x) = \sum_{k}kp_{k}x^{k-1},
$$
and
$$
	G_{0}'(1) = \sum_{k}kp_{k} = \langle k \rangle.
$$

Another important property is the degree distribution of first neighbours. The reasoning is as follows: by selecting a link at random, and following it until reaching a node, we will find that the node has, let's say, degree $k$. The probability with which we will reach such a node is proportional to its degree ($\propto kp_{k}$). The normalized distribution is given by
$$
	\dfrac{\sum kp_{k}x^{k}}{\sum_{k}kp_{k}} = x \dfrac{\sum kp_{k}x^{k-1}}{\langle k \rangle} = \dfrac{x}{\langle k \rangle}G_{0}'(x).
$$

Then, by following each one of the $k$ edges of a randomly chosen node of degree $k$, we have the distribution of the remaining outgoing edges of the $k$ first neighbours of that node generated by the function
$$
	G_{1}(x) = \dfrac{\sum_{k}kp_{k}(k)x^{k-1}}{\langle k \rangle} = \dfrac{1}{\langle k \rangle}G_{0}'(x)
$$
Note that because we are following the edges of a randomly chosen node, the distribution of the remaining edges of the first neighbours do not count the edge by which we arrived at that neighbour. That is why we get rid of one power of $x$ in the equation for $G_{1}(x)$.

From that, it is not hard to find a new PGF for the number of second neighbours of nodes in a network.
$$\sum_k p_k (G_1(x))^k $$
where
$$G_1(x) = \frac{G_{0}'(x)}{G_{0}'(1)}$$
This, in turn, is close to the PGF for the degree distribution of the projection of a bipartite network, since the second neighbours of a bipartite network are the neighbours of its projection.

\subsection{Network growth models}
It is one thing to be able to produce a network, fully formed, with a particular set of properties. It is quite another to prescribe a process that is able to grow, or generate, a network with those same properties.

One popular generative network model is the \emph{preferential attachment plus growth model} (PAGP model), also known as the Barabasi and Albert or BA model.

As the name suggests, at each time step, the PAPG model adds a node with degree $m$. Each of the edges of the new node attaches itself to a node in the existing network with a probability that is proportionate to the degree of the existing nodes. The edges in this model are undirected and once and edge is attached to a node, there is no re-wiring. It isn't too hard to prove that the PAPG model generates graphs with a power law degree distribution with $p_k \sim k^{-3}$. The fact that the power law exponent is fixed is unfortunate, though generalisation of the PAPG model allow for different exponents. It is also not hard to see that the PAPG model only produces connected graphs. This limits its applicability to some real world networks. Even though the PAPG model manages to grow networks that reproduce some real-world features (the power-law degree distribution), it does so in a way that may not match up with the growth processes of those real-world networks. For example the highest degree nodes of the PAPG model are the oldest nodes --- this is not the case with networks such as the WWW to which the PAPG model might be applied.

\subsection{Exponential random graph models}
Given a particular empirical network one often wants to create models of networks with similar observable properties. These observables might be any of the properties mention so far (or many others) such as mean clustering or a particular degree distribution. One approach is to consider the empirical network as one example of a network from an ensemble of networks that have been generated by the same underlying model. Exponential random graph models (ERGMs) provide a way to study the properties of such models by employing methods that come directly from the statistical mechanics that we have seen so far.

Consider a set $\mathcal{G}$ of graphs and a collection of observables $\{X_i\},~~i=1,2,\ldots,r$ for which we have measured expected values $\langle X_i\rangle$. It will often turn out that we only have a single measurement of these values. (Here we'll assume that the graphs in $\mathcal{G}$ are simple graphs, but the same approach works for many other possibilities.) Given a specific graph $G\in\mathcal{G}$ we want $P(G)$ the probability of finding that graph to be commensurate with the expected values of each of the graph observables $\{X_i\}$. Just like for statistical  mechanics, this is an under-determined problem: the number of degrees of freedom for the definition of $P(G)$ is much smaller than the number of constraints from the observables. However, as we know from statistical mechanics, the best approach is to pick the solution that maximises the Gibbs entropy
$$
	S= -\sum_{G\in\mathcal{G}}P(G)\ln P(G),
$$
subject to the constraints
$$
	\sum_G P(G)X_i(G) = \langle X_i\rangle
$$
and the normalization condition
$$
	\sum_G P(G) =1.
$$

To do this, we introduce the Lagrange multipliers $\alpha$ and $\theta_i$ and solve
$$
	\frac{\partial}{\partial P(G)}\left[S + \alpha \left(1-\sum_G P(G)\right) + \sum_i \theta_i \left(\langle X_i \rangle - \sum_G P(G)X_i(G)\right) \right] = 0
$$
for all graphs $G$.

This gives
$$
	\ln P(G) +1 +\alpha +\sum_i\theta_iX_i(G) = 0.
$$

We can rearrange this to get
$$
	P(G) = \frac{\exp(-H(G))}{Z},
$$
where $H(G) = \sum_i\theta_iX_i(G)$ is known as the graph Hamiltonian and $Z = \exp(\alpha+1)$ is the partition function. The requirement that $Z$ be a normalising factor requires that $\exp(\alpha+1) = \exp(-H(G))$.

\subsection{Recommended reading}
There are several good books on complex networks, including \emph{Lectures on Complex Networks} by S.N. Dorogovtsev and \emph{The Structure of Complex Networks} by E. Estrada, but some of the best content is in the form of journal survey articles. My picks are the two by M.E.J. Newman: \emph{The Structure and Function of Complex Networks} and, joint with J. Park \emph{Statistical Mechanics of Networks}. I used both of these extensively in writing this section. Also good is \emph{Statistical Mechanics of Complex Networks} by R. Albert and A.-L. Barabasi. For a good survey of generative models for power law distributions, see the survey article by Mitzenmacher.


\lesson{3}{Sep 14 2021 Tue (16:54:18)}{Solving Radical Equations}{Unit 1}

\begin{definition}[Radical Equations]
    Here is what a \textbf{Radical Equation} looks like:

    \[ T = 2\pi\sqrt{\frac{L}{32}} .\]
\end{definition}

\begin{example}[Radical Equations 1]
    Let's solve $\sqrt{(5x + 4)} = 7$:

    \begin{align}
        \sqrt{(5x + 4)} &= 7 \\
                        &= (\sqrt{5x + 4})^2 = (7)^2 \\
                        &= 5x + 4 = 49 \\
                        &= 5x = 45 \\
                        &= \frac{5x}{5} = \frac{45}{5} \\
                        &= x = 9
    \end{align}

    Now, let's check our work:

    \begin{align}
        \sqrt{5x + 4} &= 7 \\
                      &= \sqrt{5(9) + 4} = 7 \\
                      &= \sqrt{45 + 4} = 7 \\
                      &= \sqrt{49} = 7 \\
                      &= 7 = 7
    \end{align}
\end{example}

Let's try another one:

\begin{example}[Radical Equations 2]
    Let's solve $\sqrt{x - 3} + 4 = 1$:

    \begin{align}
        \sqrt{x - 3} + 4 &= 1 \\
                         &= \sqrt{x - 3} = -3 \\
                         &= (\sqrt{x - 3})^2 = (-3)^2 \\
                         &= x - 3 = 9 \\
                         &= x = 12
    \end{align}

    Now, let's check our work:

    \begin{align}
        \sqrt{x - 3} + 4 &= 1 \\
                         &= \sqrt{12 - 3} + 4 = 1 \\
                         &= \sqrt{9} + 4 = 1 \\
                         &= 3 + 4 = 1 \\
                         &= 7 \neq 1
    \end{align}

    We call this situation extraneous solution.
\end{example}

\subsubsection*{Quick Review of Factoring}

Let's look at factoring by group.

First, you want the split the middle term into factors of $15$ that combine to equal $2$.

Then, just factor by $GCF$.

\begin{align}
    x^2 + 2x - 15 &= x^2 - 3x + 5x - 15 \\
                  &= (x^2 - 3x) + (5x - 15) \\
                  &= x(x - 3) + 5(x - 3) \\
                  &= (x + 5)(x - 3)
\end{align}

\newpage


\section{Projective Geometry}
\subsection{Coordinates}
\subsubsection{Crossproduct}
$$ \bm{\tilde{x}_1} \times \bm{\tilde{x}_2} = [\bm{\tilde{x}_1}]_\times \bm{\tilde{x}_2} =
\begin{bmatrix}
0&z&-y\\
-z&0&x\\
y&-x&0\\
\end{bmatrix} \bm{\tilde{x}_2} $$
\subsubsection{2D, 3D points}
\textit{For 2D a small letter is used, a capital letter for 3D, a bold letter stands for a vector, while a normal one is a single entry from a vector}\\

\textbf{Inhomogeneous coordinates} (marked with plain letter):
$$ \bm{x} = (x,y)^\top \qquad \bm{X} = (x,y,z)^\top$$

\textbf{Homogeneous coordinates} (with a tilde on top):
$$ \tilde{\bm{x}} = (x,y,w)^\top \qquad \tilde{\bm{X}} = (x,y,z,w)^\top$$

also called " augmented coordinates $\bar{x}$ " when $w = 1$.\\

Scale is not important for incidence relation.\\

\textbf{Ideal point / point at infinity}:
$$\tilde{\bm{x}} = (x,y,0)^\top \qquad \tilde{\bm{X}} = (x,y,z,0)^\top$$

\textbf{Intersection of two lines}:
$$ \tilde{\bm{x}} = \bm{\tilde{l}_1} \times \bm{\tilde{l}_2} $$

3D Point form three planes:
$$ \begin{bmatrix}
\pi^{(1)\top}\\
\pi^{(2)\top}\\
\pi^{(3)\top}\\
\end{bmatrix} \tilde{\bm{X}} = 0 $$

\subsubsection{2D lines}

$$ \bm{l} = (a,b,c) $$

The point x lies on the line l if and only if
$$ \bm{l}^\top\tilde{\bm{x}} = ax+by+c=0$$

Line at infinity:
$$\bm{l}_\infty = (0,0,1)^\top$$

Horizontal and vertical lines:

\includegraphics[width=0.8\columnwidth]{pictures/verticalhorizontal}

\textbf{Line joining two points}:
$$ \bm{l} = \bm{\tilde{x}_1} \times \bm{\tilde{x}_2} $$

\subsubsection{3D Planes}

$$ \pi = (\pi_1,\pi_2,\pi_3,\pi_4)^\top $$

The point $\bm{\tilde{X}}$ lies on the plane $\pi$ if and only if
$$ \pi^T\bm{\tilde{X}} = \pi_1x+\pi_2y+\pi_3z+\pi_4w = 0 $$

Planes from points
$$ \begin{bmatrix}
	\tilde{\bm{X}}^{(1)\top}\\
	\tilde{\bm{X}}^{(2)\top}\\
	\tilde{\bm{X}}^{(3)\top}\\
\end{bmatrix} \pi = 0 $$

\subsubsection{Conic}
Inhomogenious:
$$ ax^2 + bxy + cy^2 + dx + ey + f = 0$$

5 Degrees of freedom

\subsection{Transformation}
\subsubsection{2D Transformation}
\begin{itemize}
	\item Rotation+translation/Euclidean
	\item Scaled Rotation/Similarity/Metric - This transformation preserves angles between lines and planes( Orthogonal views of the planar at all times)
	\item Affine - Parallel lines and planes remain parallel under affine transformations (general camera motion relative to the planar shape, but always keeping sufficient distance from the planar shape)
	\item Projectivity/Collineation/Projective transformation/Homography (all cameras motions are allowed, also when moving close to the planar shape)
\end{itemize}

$$\begin{pmatrix}
x'\\
y'\\
w'\\
\end{pmatrix} = \begin{bmatrix}
h_{11}&h_{12}&h_{13}\\
h_{21}&h_{22}&h_{23}\\
h_{31}&h_{32}&h_{33}\\
\end{bmatrix} \begin{pmatrix}
x\\
y\\
w\\
\end{pmatrix} $$

Point transformation: $\bm{\tilde{x}'} = H \bm{\tilde{x}}$\\
Line transformation: $\bm{l'} = H^{-\top} \bm{l}$\\

\includegraphics[width=\columnwidth]{pictures/2Dtransformations}

\subsubsection{3D Transformationen}

$$\begin{pmatrix}
x'\\
y'\\
z'\\
w'\\
\end{pmatrix} = \begin{bmatrix}
h_{11}&h_{12}&h_{13}&h_{14}\\
h_{21}&h_{22}&h_{23}&h_{24}\\
h_{31}&h_{32}&h_{33}&h_{34}\\
h_{41}&h_{42}&h_{43}&h_{44}\\
\end{bmatrix} \begin{pmatrix}
x\\
y\\
z\\
w\\
\end{pmatrix} $$

Point transformation: $\bm{\tilde{X}'} = H \bm{\tilde{X}}$\\
Plane transformation: $\bm{\pi'} = H^{-\top} \bm{\pi}$\\

\includegraphics[width=\columnwidth]{pictures/3Dtransformation}


\section{Mixed Models and Splines}

Mixed models are defined by
\[
\by = \bX \bb + \bZ \bu + \beps
\]
where $\by$ is a vector of $n$ observable random variables, $\bb$
is vector of $p$ unknown parameters having fixed values (fixed
effects), $\bX$ and $\bZ$ are known matrices and $\bu$ and
$\beps$ are vector, of length $q$ and $n$, of unobservable variables
(random
effects) such that $\E(\bu)=0$ and $\E(\beps)=0$ and
\[
\var\left[ \begin{array}{c} \bu \\ \beps \end{array} \right]
=
\left[ \begin{array}{cc} G&0\\ 0&R \end{array} \right] \sigma^2
\].
Here $R$ and $G$ are known positive definite matrices and $\sigma^2$
is a positive constant.

Random effects are especially useful to introduce correlation to the
random part of the model.

Robinson (1991) describes how the best linear unbiased predictor of
$\by$ is a ``good thing''. Speed (1991) notes that after defining an
appropriate $\bX$, $\bZ$, and $G$ we have that natural smoothing are
BLUPs.

The smoothing parameter is included in the $G$ and one can view it a
``nuisance'' parameter. Robinson (1991) suggests REML estimation
as a way of ``estimating'' the smoothness parameter. Speed notes that
this is equivalent to Wahba's Generalized Maximum Likelihood estimate
of the smoothing parameter.

Furthermore, this idea permits us to model nested curves in a natural
way. See Brumback and Rice (1998).


\subsection{Draw}

$draw(f,x)$ draws a graph of function $f$ of $x$.
(The default second argument is $x$.)

{\color{blue}
\begin{verbatim}
draw(x^2)
\end{verbatim}
}

\begin{center}
\includegraphics[scale=0.5]{parabola1.png}
\end{center}

\noindent
The vectors $xrange$ and $yrange$ control the scale of the graph.

{\color{blue}
\begin{verbatim}
xrange = (-1,1)
yrange = (0,2)
draw(x^2)
\end{verbatim}
}

\begin{center}
\includegraphics[scale=0.5]{parabola2.png}
\end{center}

\noindent
Parametric drawing occurs when a function returns a vector.
The vector $trange$ controls the parametric range.
The default is $trange=(-\pi,\pi)$.
In the following example, $draw$ varies $theta$
over the default range $-\pi$ to $+\pi$.

{\color{blue}
\begin{verbatim}
xrange = (-10,10)
yrange = (-10,10)
f = 5 (cos(theta),sin(theta))
draw(f,theta)
\end{verbatim}
}

\begin{center}
\includegraphics[scale=0.5]{circle1.png}
\end{center}

\noindent
In the following example, $trange$ is reduced
to draw a quarter circle instead of a full circle.

{\color{blue}
\begin{verbatim}
trange = (0,pi/2)
f = 5 (cos(theta),sin(theta))
draw(f,theta)
\end{verbatim}
}

\begin{center}
\includegraphics[scale=0.5]{circle2.png}
\end{center}

\noindent
Lemniscate.

{\color{blue}
\begin{verbatim}
trange = (-pi,pi)
X = cos(t) / (1 + sin(t)^2)
Y = sin(t) cos(t) / (1 + sin(t)^2)
f = 5 (X,Y)
draw(f,t)
\end{verbatim}
}

\begin{center}
\includegraphics[scale=0.5]{lemniscate.png}
\end{center}

\noindent
Cardioid.

{\color{blue}
\begin{verbatim}
r = (1 + cos(t)) / 2
u = (cos(t),sin(t))
f = r u
xrange = (-1,1)
yrange = (-1,1)
trange = (0,2pi)
draw(f,t)
\end{verbatim}
}

\begin{center}
\includegraphics[scale=0.5]{cardioid.png}
\end{center}


\subsection*{c)}

For some datasets it is of interest to only calculate the upper/lower bound of the estimate. As for our case, one is not particularly interested in a significant wave-lengths lower bound, but rather the upper bound. Here we will provide a one-sided parametric bootstrapped 95\% confidence interval for the 100-year return value of the significant wave-height data provided. \\

If we want to get the return value of the 100-year return value, we use the relation derived in the introduction above. To recall it is denoted by
\[ F^{-1}(1-1/T;\mu,\beta), \quad T=3\cdot14\cdot100.\] \\

If we want to get the return value of the 100-year return value, we use the relation derived in the introduction above. To recall it is denoted by
\[ F^{-1}(1-1/T;\mu,\beta), \quad T=3\cdot14\cdot100.\]
To get the return value we therefore use the relation for the inverse again in assignment \textbf{a)} by inserting $u=1-1/T$ and our estimated $\mu$ and $\beta$. Giving the return value as
\[  \mu-\beta \ln \left \{\ln\left ( \frac{1}{u}\right ) \right \}=x,\quad x=16.5436\]

This is however, only the maximum-value using our estimates and if we use the same arguments as before. Meaning, we calculate the maximum-value using the relation, but we here use the upper bounds for our estimate. This returns the value $x=17.3984$. As a result, our one-sided 95\% confidence bound of the 100-year return value for the wave-height is
\[ x_b=(16.5436,17.3984)\quad m \]


\section{Segments}
In this section we will discuss how to compute intersections and distances involving line segments.

\subsection{Point on segment}\label{onsegment}
As an introduction, let's first see how to check if a point $P$ lies on segment $[AB]$.

For this we will first define a useful subroutine \lstinline|inDisk()| that checks if a point $P$ lies on the disk of diameter $[AB]$. We know that the points on a disk are those which form angles $\geq 90\degree$ with the endpoints of a diameter. This can easily be checked by using dot product: $\widehat{APB} \geq 90\degree$ is equivalent $\dotv{PA}{PB} \leq 0$ (with the exception of $P=A,B$ in which case angle $\widehat{APB}$ is undefined).

\centerFig{segment0}

\begin{lstlisting}
bool inDisk(pt a, pt b, pt p) {
    return dot(a-p, b-p) <= 0;
}
\end{lstlisting}

\begin{mathy}
In fact, we can notice that $\dotv{PA}{PB}$ is equal to the power of point $P$ with respect to the circle of diameter $[AB]$: if $O$ is the center of that circle and $r$ its radius, then $\dotv{PA}{PB} = |OP|^2 - r^2$. This makes it perfect for our purpose.
\end{mathy}

With this subroutine in hand, it is easy to check whether $P$ is on segment $[AB]$: this is the case if and only if $P$ is on line $AB$ and also on the disk whose diameter is $AB$ (and thus is in the part the line between $A$ and $B$).

\begin{center}
\includeFig{segment1}

intersection of line and disk $=$ segment
\end{center}

\begin{lstlisting}
bool onSegment(pt a, pt b, pt p) {
    return orient(a,b,p) == 0 && inDisk(a,b,p);
}
\end{lstlisting}

\subsection{Segment-segment intersection}\label{ss:seg-seg-inter}
Finding the precise intersection between two segments $[AB]$ and $[CD]$ is quite tricky: many configurations are possible and the intersection itself might be empty, a single point or a whole segment.

To simplify things, we will separate the problem in two distinct cases:
\begin{enumerate}
\item Segments $[AB]$ and $[CD]$ intersect \term{properly}, that is, their intersection is one single point which is not an endpoint of either segment. This is easy to test with \lstinline|orient()|.
\item In all other cases, the intersection, if it exists, is determined by the endpoints. If it is a single point, it must be one of $A,B,C,D$, and if it is a whole segment, it will necessarily start and end with points in $A,B,C,D$.
\end{enumerate}

Let's deal with the first case: there is a single proper intersection point $I$. To test this, it suffices to test that $A$ and $B$ are on either side of line $CD$, and that $C$ and $D$ are on either side of line $AB$. If the test is positive, we find $I$ as a weighted average of $A$ and $B$.\footnote{We can understand the formula as the center of gravity of point $A$ with weight $|o_B|$ and point $B$ with weight $|o_A|$, which gives us a point $I$ on $[AB]$ such that $|IA|/|IB| = o_A/o_B$.}

\begin{center}
\includeFig{segment2}

proper intersection
\end{center}
\begin{lstlisting}
bool properInter(pt a, pt b, pt c, pt d, pt &out) {
    double oa = orient(c,d,a),
           ob = orient(c,d,b),
           oc = orient(a,b,c),
           od = orient(a,b,d);
    // Proper intersection exists iff opposite signs
    if (oa*ob < 0 && oc*od < 0) {
        out = (a*ob - b*oa) / (ob-oa);
        return true;
    }
    return false;
}
\end{lstlisting}

Then to deal with the second case, we will test for every point among $A,B,C,D$ if it is on the other segment. If it is, we add it to a set $S$. Clearly, an endpoint cannot be in the middle of the intersection segment, so $S$ will always contain 0, 1 or 2 distinct points, describing an empty intersection, a single intersection point or an intersection segment.

\centerFig{segment3}

\begin{lstlisting}
// To create sets of points we need a comparison function
struct cmpX {
    bool operator()(pt a, pt b) {
        return make_pair(a.x, a.y) < make_pair(b.x, b.y);
    }
};

set<pt,cmpX> inters(pt a, pt b, pt c, pt d) {
    pt out;
    if (properInter(a,b,c,d,out)) return {out};
    set<pt,cmpX> s;
    if (onSegment(c,d,a)) s.insert(a);
    if (onSegment(c,d,b)) s.insert(b);
    if (onSegment(a,b,c)) s.insert(c);
    if (onSegment(a,b,d)) s.insert(d);
    return s;
}
\end{lstlisting}

\subsection{Segment-point distance}
To find the distance between segment $[AB]$ and point $P$, there are two cases: either the closest point to $P$ on $[AB]$ is strictly between $A$ and $B$, or it is one of the endpoints ($A$ or $B$). The first case happens when the orthogonal projection of $P$ onto $AB$ is between $A$ and $B$.

\centerFig{segment4}

To check this, we can use the \lstinline|cmpProj()| method in \lstinline|line|.%\footnote{If desired, we can also eliminate the need for \lstinline|line| by using the direction vector \lstinline|v = b-a| to perform the computations directly.}
\begin{lstlisting}
double segPoint(pt a, pt b, pt p) {
    if (a != b) {
        line l(a,b);
        if (l.cmpProj(a,p) && l.cmpProj(p,b)) // if closest to projection
            return l.dist(p);                 // output distance to line
    }
    return min(abs(p-a), abs(p-b)); // otherwise distance to A or B
}
\end{lstlisting}

\subsection{Segment-segment distance}
We can find the distance between two segments $[AB]$ and $[CD]$ based on the segment-point distance if we separate into the same two cases as for segment-segment intersection:
\begin{enumerate}
\item Segments $[AB]$ and $[CD]$ intersect properly, in which case the distance is of course 0.
\item In all other cases, the shortest distance between the segments is attained in at least one of the endpoints, so we only need to test the four endpoints and report the minimum.
\end{enumerate}

This can be readily implemented with the functions at our disposal.
\begin{lstlisting}
double segSeg(pt a, pt b, pt c, pt d) {
    pt dummy;
    if (properInter(a,b,c,d,dummy))
        return 0;
    return min({segPoint(a,b,c), segPoint(a,b,d),
                segPoint(c,d,a), segPoint(c,d,b)});
}
\end{lstlisting}

Some possible cases are illustrated below.
\centerFig{segment5}


\section{Formal analysis}
The problem we must solve is to tell whether:
\begin{equation}
  E_{th} \le \frac{1}{N} \sum_{n=1}^N x^2[n]
\end{equation}
Since $ x[n] \in [-1, 1) $ and it is represented on 16 bits, it can be
written as
\begin{equation}
  x[n] = z[n]\cdot LSB
\end{equation}
where $z[n] \in [-2^{15}, 2^{15} - 1]$ and $LSB = \frac{1}{2^{15}}$. Therefore we
can equivalently check that:
\begin{equation}
  \sum_{n=1}^N |z[n]|^2 \ge \frac{N \cdot E_{th}}{LSB^2} = E_{th}'
\end{equation}
Since the sampling rate is $f_s$ and the frame duration is $T_{frame}$,
we can easily compute the number of samples per frame:
$N = \frac{T_{frame}}{1/f_s} = 16\si{\milli\second} \cdot 16\si{\kilo\hertz} = 256$
and hence $E_{th}' = 13743895347$.

% TODO: move to optimization section
% \paragraph{Absolute value} of $z > 0$ is $Z$, which is also the representation
% of $z$ if it is represented using the 2 complement on 16 bits. We note that
% $Z(15) = 0$ because $z > 0$.

% If $z < 0$, $|z|$ is represented by $\bar{Z} + 1$, with $Z\in[2^{15}, 2^{16}-1]$.
% So $\bar{Z} + 1 \in [1, 2^{15}]$, which requires 16 bits to be represented.
% We could approximate the representation of $|z|$ with $\bar{Z}$ for negative $z$,
% so the representation would require only 15 bits, as in the case of $z > 0$.

% We could hence use only 15 bits to represent $|z|$ for all cases.

Finally, note that, since data comes in serially, the component just needs to
keep a partial sum of the squares of the samples.

\begin{equation}
  \begin{cases}
    S[n] = |z[n]|^2 + S[n - 1] \\
    S[0] = 0
  \end{cases}
\end{equation}


\subsection{Intervals}

For a totally ordered set we can define a subset as being all elements with a relationship to a number. For example:

\([a,b]=\{x:a\le x \land x\le b\}\)

This denotes a closed interval. Using the definition above we can also define an open interval:

\((a,b)=\{x:a< x \land x< b\}\)


\section{Quantiles of a Distribution}

Let $\alpha$ in $(0,1)$. The quantile of order $1 - \alpha$ of a random variable $X$ is the number $q_{\alpha}$ such that:\\
\begin{align*}
\displaystyle \mathbb{P}\left(X\leq q_{\alpha }\right)& = q_{\alpha } = 1-\alpha\\
\mathbb{P}(X \geq q_{\alpha })& = \alpha\\
F_X(q_{\alpha})& = 1 - \alpha\\
F^{-1}_{X}(1-\alpha)&= \alpha
\end{align*}
If the distribution is \textbf{standard normal} $X \sim N(0,1)$:\\
\begin{align*}
\mathbb{P}(|X| > q_{\alpha})& = \alpha\\
& = 2\mathbf{\Phi}(q_{\alpha/2})
\end{align*}
Use \textbf{standardization} if a gaussian has unknown mean and variance $X \sim N(\mu,\sigma^2)$ to get the quantiles by using Z-tables (standard normal tables).\\

\begin{align*}
\mathbf{P}\left(X\leq t\right)& = \displaystyle \mathbf{P}\left(Z\leq \frac{t-\mu}{\sigma}\right)\\
& = \mathbf{\Phi}\left(\frac{t-\mu}{\sigma}\right)\\
Z &= \frac{X-\mu}{\sigma} \sim N(0,1)\\
q_{\alpha }& = \frac{t-\mu}{\sigma}
\end{align*}


\section*{Exercises}

\begin{ex}
  Find the distance between the points $P=(0,1,3)$ and
  $Q=(2,-1,0)$ in $\R^3$.
  \begin{sol}
    $\sqrt{17}$.
  \end{sol}
\end{ex}

\begin{ex}
  Find the distance between the points $P=(1,3,-1,0)$ and
  $Q=(2,2,3,3)$ in $\R^4$.
  \begin{sol}
    $\sqrt{27}$.
  \end{sol}
\end{ex}

\begin{ex}
  Describe the points in $\R^3$ that are equally distant from the two
  points $Q=(1,1,1)$ and $R=(-1,-1,-1)$.
  \begin{sol}
    The form the plane $x+y+z=0$.
  \end{sol}
\end{ex}

\begin{ex}
  Describe the points in $\R^3$ that have distance $1$ from the origin.
  \begin{sol}
    They form a sphere of radius $1$, given by the equation
    $x^2+y^2+z^2=1$.
  \end{sol}
\end{ex}

\begin{ex}
  Find the length of each of the following vectors.
  \begin{equation*}
    \vect{u}=\begin{mymatrix}{c}-3\\2\end{mymatrix}, \quad
    \vect{v}=\begin{mymatrix}{c}1\\-2\\5\end{mymatrix}, \quad
    \vect{w}=\begin{mymatrix}{c}1\\4\\-2\\1\end{mymatrix}.
  \end{equation*}
  \begin{sol}
    $\norm{\vect{u}} = \sqrt{13}$, $\norm{\vect{v}} = \sqrt{30}$,
    $\norm{\vect{w}} = \sqrt{22}$.
  \end{sol}
\end{ex}

\begin{ex}
  Prove the properties of Proposition~\ref{prop:properties-length}.
\end{ex}

\begin{ex}
  Prove that for all vectors $\vect{u}\in\R^n$, we have
  $\norm{-\vect{u}} = \norm{\vect{u}}$.
  \begin{sol}
    This follows from the last property of
    Proposition~\ref{prop:properties-length}: $\norm{-\vect{u}} =
    \norm{(-1)\vect{u}} = \abs{-1}\norm{\vect{u}} = \norm{\vect{u}}$.
  \end{sol}
\end{ex}

\begin{ex}
  Which of the following are unit vectors?
  \begin{equation*}
    \vect{u}=\frac{1}{2}\begin{mymatrix}{c}1\\1\\1\end{mymatrix}, \quad
    \vect{v}=\frac{1}{3}\begin{mymatrix}{c}1\\1\\1\end{mymatrix}, \quad
    \vect{w}=\frac{1}{2}\begin{mymatrix}{c}1\\1\\1\\1\end{mymatrix}.
  \end{equation*}
  \begin{sol}
    Only $\vect{w}$ is a unit vector.
  \end{sol}
\end{ex}

\begin{ex}
  Normalize the following vectors.
  \begin{equation*}
    \vect{u}=\begin{mymatrix}{r}1\\2\end{mymatrix}, \quad
    \vect{v}=\begin{mymatrix}{r}-2\\3\\2\end{mymatrix}, \quad
    \vect{w}=\begin{mymatrix}{r}5\\-3\\1\\-1\end{mymatrix}.
  \end{equation*}
  \begin{sol}
    \begin{equation*}
      \frac{1}{\norm{\vect{u}}}\vect{u} = \frac{1}{\sqrt{5}}\begin{mymatrix}{r}1\\2\end{mymatrix},\quad
      \frac{1}{\norm{\vect{v}}}\vect{v} = \frac{1}{\sqrt{17}}\begin{mymatrix}{r}-2\\3\\2\end{mymatrix},\quad
      \frac{1}{\norm{\vect{w}}}\vect{w} = \frac{1}{6}\begin{mymatrix}{r}5\\-3\\1\\-1\end{mymatrix}.
    \end{equation*}
  \end{sol}
\end{ex}


\subsection{Integrals}

\subsubsection{Cosine and sine}

\(\arccos (\theta)\), \(\arcsin (\theta )\) and difficulty of inversing

In order to determine \(\tau \) we need inverse functions for \(\cos (\theta )\) or \(\sin (\theta )\).

These are the \(\arccos (\theta )\) and \(\arcsin (\theta )\) functions respectively.

However this is not easily calculated. Instead we look for another function.

\subsubsection{Calculating \(\arctan (\theta )\)}

So we want a function to inverse this. This is the \(\arctan (\theta )\) function.

If \(y=\tan (\theta )\), then:

\(\theta =\arctan (y)\)

We know the derivative for \(\tan (\theta )\) is:

\(\dfrac{\delta }{\delta \theta }\tan (\theta )=1+\tan^2(\theta )\)

\(\dfrac{\delta y}{\delta \theta }=1+y^2\)

So

\(\dfrac{\delta \theta }{\delta y}=\dfrac{1}{1+y^2}\)

\(\dfrac{\delta }{\delta y}\arctan (y)=\dfrac{1}{1+y^2}\)

So the value for \(\arctan (k)\) is:

\(\arctan (k)=\arctan (a)+\int_a^k\dfrac{\delta }{\delta y}\arctan (y) \delta y\)

\(\arctan (k)=\arctan (a)+\int_a^k\dfrac{1}{1+y^2} \delta y\)

What do we know about this function? We know it can map to multiple values of \(\theta \) because the underlying \(\sin (\theta )\) and \(\cos (\theta )\) functions also loop.

We know that one of the results for \(\arctan (0)\) is \(0\).


\chapter{Graphs}

A \emph{graph} is an ordered pair $G=(V,E)$ where $V$ is a set of
\emph{vertices} and $E$ is a set of \emph{edges}.  An edge is a pair
of vertices which are said to be \emph{adjacent}.  An edge is said to
be \emph{incident} on its component vertices.  Usually we consider
edges to be unordered, in which case the graph is undirected and an
edge $\{A,B\}$ connects $A$ to $B$ and $B$ to $A$.  For example:

\begin{align*}
V &= \{ A, B, C \} \\ E &= \{ (A, B), (B, C) \}
\end{align*}

Which can be represented more visually:

{
  \includegraphics[scale=0.2]{SimpleGraph}
  %\caption{Demonstrates a simple graph}
  \label{fig:SimpleGraph}
}

Note that the vertices are represented by labeled circles and the
edges are represented by lines connecting vertices to one another.

The number of edges connecting a vertex $v$ is called the degree of
$v$ or $deg(v)$.  In the above example:

\begin{align*}
deg(A) = deg(C) &= 1 \\
deg(B) &= 2
\end{align*}

Sometimes an edge is directional, meaning the pair of vertices in an
edge is ordered.  In other words, the edge $(A, B)$ connects $A$ to
$B$, but not $B$ to $A$.  We say such an edge is incoming on $B$ and
outgoing on $A$.  A graph whose edges are ordered pairs is called a
$directed graph$ or $digraph$.

This is represented visually by an edge with an arrow at one end,
indicating the direction:

{
  \includegraphics[scale=0.2]{DiGraph}
  %\caption{Demonstrates a directed graph}
  \label{fig:DiGraph}
}

In a digraph, the number of incoming edges of $v$ is the in-degree or
$deg^-(v)$.  Similarly, the number of outgoing edges is the
out-degree or $deg^+(v)$.

In the above example:
%
\begin{align*}
deg^+(A) = deg^+(B) &= 1 \\
deg^-(B) = deg^-(C) &= 1 \\
deg^+(C) = deg^-(A) &= 0
\end{align*}

A \emph{simple graph} is a graph which contains no edges from any
vertex $v$ to itself $ (v,v) $, called loops.  A simple graph also
contains no multi-edges which connect more than two vertices.

A \emph{path} is a sequence of edges from vertex $u$ to vertex $v$.

Two vertices are said to be \emph{connected} if there exists a path
between them.

A graph is said to be connected if for any two vertices, there exists
a path between them. An adjacent vertex is called a \emph{neighbour}.

A \emph{complete} graph is one in which every vertex is adjacent to
every other vertex.

A \emph{subgraph} is a graph consisting of a subset of the vertices
and edges in another graph.

A \emph{connected component} is a connected subgraph which does
not disconnect any adjacent vertices.  It should be easy to see that a
connected graph has exactly one connected component.

A \emph{cycle} in a graph is when there exists a path from a vertex
back to itself without crossing any edge more than once.

A graph is said to be \emph{acyclic} when it does not contain cycles.

\section{Representation}

\subsection{Adjacency List}

One way to represent a graph is for every vertex, store a list of
neighbours.  For example:

{
  \includegraphics[scale=0.2]{SimpleGraph}
  %\caption{Demonstrates a simple graph}
  \label{fig:SimpleGraph}
}

can be represented:

{
  \includegraphics[scale=0.7]{AdjacencyList}
  %\caption{Demonstrates an adjacency list for a simple graph}
  \label{fig:AdjacencyList}
}

And

{
  \includegraphics[scale=0.2]{DiGraph}
  %\caption{Demonstrates a directed graph}
  \label{fig:DiGraph}
}

can be represented:

{
  \includegraphics[scale=1.0]{AdjacencyListDigraph}
  %\caption{Demonstrates an adjacency list for a directed graph}
  \label{fig:AdjacencyListDigraph}
}

\subsection{Adjacency Matrix}

Another representation of a graph is as a square matrix with $n$ rows
and columns where the element at row $i$ and column $j$ is 1 if there
is an edge between $v_i$ and $v_j$ and 0 otherwise.  For example:

{
  \includegraphics[scale=0.2]{SimpleGraph}
  %\caption{Demonstrates a simple graph}
  \label{fig:SimpleGraph}
}

can be represented

\[
\left[
\begin{array}{cccc}
  & a & b & c \\
a & 0 & 1 & 0 \\
b & 1 & 0 & 1 \\
c & 0 & 1 & 0
\end{array}
\right]
\]

As for directed graphs, row $i$ column $j$ is 1 if there exists an
edge $(v_i,v_j)$.  For example:

{
  \includegraphics[scale=0.2]{DiGraph}
  %\caption{Demonstrates a directed graph}
  \label{fig:DiGraph}
}

can be represented

\[
\left[
\begin{array}{cccc}
  & a & b & c \\
a & 0 & 1 & 0 \\
b & 0 & 0 & 1 \\
c & 0 & 0 & 0
\end{array}
\right]
\]


\subsection{FTC for Line Integrals}
\noindent
We saw earlier that $\mathrm{d}z = \nabla f \cdot \langle \mathrm{d}x, \mathrm{d}y \rangle$. This can be written as $\mathrm{d}z = \nabla f \cdot \mathrm{d}\vec{r}$ where $\vec{r}(t)$ parameterizes a simple curve $C$ and $a \leq t \leq b$.
\begin{equation*}
	\int\limits_{C}{\nabla f \cdot \mathrm{d}\vec{r}} = \int_{a}^{b}{(\nabla f\circ\vec{r}) \cdot \vec{r^\prime}\mathrm{d}t} = f\circ\vec{r}\rvert_{a}^{b} = f(\vec{r}(b))-f(\vec{r}(a))
\end{equation*}
\begin{equation*}
	\int\limits_{C}{\nabla f\cdot\mathrm{d}\vec{r}}=f(\vec{r}(b))-f(\vec{r}(a))
\end{equation*}
the fundamental theorem of calculus for line integrals.\\

\noindent
Let's test by computing a line integral directly and using the FTC for line integrals. Let the path be the top-half semicircle connecting $(1,0)$ to $(-1,0)$ and let $f(x,y) = 12 - 3x - y$.
\begin{center}
	$\vec{r}(t)=\langle\cos{t},\sin{t}\rangle$, $0\leq t\leq\pi$
	\begin{tabular}{c|c}
		Directly & FTC \\ \hline
		$\vec{r^\prime}(t) = \langle-\sin{t}, \cos{t}\rangle$ & $\vec{r}(0) = \langle 1, 0 \rangle$, $\vec{r}(\pi) = \langle -1, 0 \rangle$ \\
		$\nabla f = \langle -3, -1 \rangle$ & $f(\vec{r}(0)) = 9$, $f(\vec{r}(\pi)) = 15$ \\
		$\int_{0}^{\pi}{3\sin{t} - \cos{t}\mathrm{d}t} = 3 - \cos{t} - \sin{t}\rvert_{0}^{\pi} = 6$ & $15 - 9 = 6$ \\
	\end{tabular}
\end{center}

\input{./lineSurfaceIntegrals/potentialFunctions}


\lab{Applications}{Image Compression (SVD)}{SVD}

\objective{Explore the SVD as a method of image compression}

In this lab, we are going to explore how the SVD can be used to compress image data.
Recall that the SVD is a decomposition of an $m \times n$ matrix $A$ of rank $r$ into the product $A = U \Sigma V^H$, where $U$ and $V$ are unitary matrices having dimensions $m \times m$ and $n \times n$, respectively, and $\Sigma$ is an $m \times n$ diagonal matrix.
\begin{equation*}
\Sigma = \mbox{diag}(\sigma_1,\sigma_2,\ldots,\sigma_r,0,\ldots,0)
\end{equation*}
where $\sigma_1 \geq \sigma_2 \geq \ldots \geq \sigma_r > 0$ are the singular values of $A$.
Upon closer inspection, we see that we can write
\begin{equation*}
U = \begin{pmatrix}U_1 & U_2\end{pmatrix}, \quad \Sigma =
\begin{pmatrix}\Sigma_r & 0\\0 & 0\end{pmatrix}, \quad V =
\begin{pmatrix}V_1 & V_2\end{pmatrix},
\end{equation*}
where $U_1$ and $V_1$ have dimensions $m\times r$ and $n\times r$ respectively and $\Sigma_r$ is the $r\times r$ diagonal matrix of (nonzero) singular values.
Multiplying this out yields the reduced form of the SVD
\begin{equation*}
A =
\begin{pmatrix}U_1 & U_2\end{pmatrix}
\begin{pmatrix}\Sigma_r & 0\\0 & 0\end{pmatrix}
\begin{pmatrix}V^H_1 \\ V^H_2\end{pmatrix} =
U_1 \Sigma_r V_1^H
\end{equation*}

\subsection*{Low rank data storage}
If the rank of a given matrix is significantly smaller than its dimensions, the reduced form of the SVD offers a way to store $A$ with less memory.
Without the SVD, an $m\times n$ matrix requires storing $m*n$ values.
By decomposing the original matrix into the SVD reduced form, $U_1$, $\Sigma_r$ and $V_1$ together require $(m*r)+r+(n*r)$ values.
Thus if $r$ is much smaller than both $m$ and $n$, we can obtain considerable efficiency.
For example, suppose $m=100$, $n=200$ and $r=20$.
Then the original matrix would require storing $20,000$ values whereas the reduces form of the SVD only requires storing $6020$ values.

\subsection*{Low rank approximation}
The reduced form of the SVD also provides a way to approximate a
matrix with another one of lower rank.
This idea is used in many areas of applied mathematics including signal processing, statistics, semantic indexing (search engines), and control theory.
If we are given a matrix $A$ of rank $r$, we can find an approximate matrix $\widehat A$ of rank $s<r$ by taking the SVD of $A$ and setting all of its singular values after $\sigma_s$ to zero, that is,
\begin{equation*}
\Sigma_{\widehat A} = \sigma_1, \sigma_2, \ldots, \sigma_s,\sigma_{s+1}=0,\ldots,\sigma_r=0
\end{equation*}
and then multiplying the matrix back together again.
The more singular values we keep, the closer our approximation is to $A$.
The number of singular values we decide to preserve depends on how close of an approximation we need and what our size requirements are for $U_1$, $\Sigma_{\widehat A}$, and $V_1$.
Try plotting the the singular values.
We have plotted the singular values to the image below.
Matrix rank is on the x-axis and the eigenvalues are the y-axis.
Note that SVD orders the singular values from greatest to least.
The greatest eigenvalues contribute most to the image while the smallest eigenvalues hardly contribute anything to the final approximation.
By looking at the graph we can have a rough idea of how many singular values we need to preserve to have a good approximation of $A$.
The matrix rank of the image below is $670$.
However, as the plot shows, we could easily approximate the image using only the first half of the singular values.

%\begin{figure}
%\includegraphics[]{hubble_red.png}
%\includegraphics[scale=.4]{hubble_svals.pdf}
%\end{figure}

\begin{figure}
\begin{minipage}[b]{.4\linewidth}
\centering
\includegraphics[width=\textwidth]{hubble_red}
\end{minipage}
\hspace{0.5cm}
\begin{minipage}[b]{0.5\linewidth}
\centering
\includegraphics[width=\textwidth]{hubble_svals}
\end{minipage}
\caption{An image and its SVD.}
\end{figure}

\begin{lstlisting}
: import numpy as np
: from scipy.linalg import svd, norm
: A = np.array([[1,1,3,4], [5,4,3,7], [9,10,10,12], [13,14,15,16], [17,18,19,20]])
: U,s,Vt = svd(A)
: S = np.diag(s)
: Ahat = U[:,0:3].dot(S[0:3,0:3]).dot(Vt[0:3,:])
: norm(A-Ahat)
\end{lstlisting}
Note that $\widehat A$ is ``close'' to the original matrix $A$, but that its rank is 3 instead of 4.

\subsection*{Application to Imaging}
Enter the following into IPython (note that any image you might have will work):
\begin{lstlisting}
: import matplotlib.pyplot as plt
: X = plt.imread('fingerprint.png')[:,:,0].astype(float)
: X.nbytes      #number of bytes needed to store X
: plt.imshow(X)
: plt.show()
\end{lstlisting}
Computing the SVD of your image is simple.
Remember to make the singular values a diagonal matrix before multiplying.
\begin{lstlisting}
: U,s,Vt = svd(X)
: S = sp.diag(s)
\end{lstlisting}
In the next code block, $n$ represents the desired rank of the output.
\begin{lstlisting}
: n = 50
: u1, s1, vt1 = U[:,0:n], S[0:n,0:n], Vt[0:n,:]
: Xhat = u1.dot(s1).dot(vt1)
: (u1.nbytes + np.diag(s1).nbytes + vt1.nbytes) - X.nbytes   #should be negative
: plt.imshow(Xhat)
: plt.show()
\end{lstlisting}

\begin{problem}
Sometimes there is not enough available bandwidth to transmit a full resolution photograph.
You aim to reduce the amount of data that needs to be transmitted from a remote location such that loss of image detail is minimal, but the amount of data that needs to be sent has reduced as much as possible.
In other words, find the minimum rank needed to accurately represent a variety of images.
Record your results and comment on them.
\end{problem}


\section{Sequences}\label{sec:sequences}
While the idea of a sequence of numbers, $a_1,a_2,a_3,\ldots$ is
straightforward, it is useful to think of a sequence as a function. We
have dealt with functions whose domains are the real
numbers, or a subset of the real numbers, like $f(x)=\sin x$. A
sequence can be regarded as a function with domain as the natural numbers
$\mathbb{N}=\{1,2,3,\ldots\}$ or the non-negative integers,
$\ds \mathbb{Z}^{\geq 0}=\{0,1,2,3,\ldots\}$. The range of the function is still
allowed to be the set of all real numbers; we say that a sequence is
a function $f\colon \mathbb{N}\to\mathbb{R}$. Sequences are commonly denoted in several different, but equally acceptable ways:
$$a_1,a_2,a_3,\ldots $$
$$\left\{a_n\right\}_{n=1}^\infty $$
$$\left\{f(n)\right\}_{n=1}^\infty $$

As with functions of the real numbers,
we will most often encounter sequences that can be expressed by a
formula. We have already seen the sequence $\ds a_i=f(i)=1-1/2^i$.
Some other simple examples are:
\begin{align*}
f(i)&={i\over i+1}	\\
f(n)&={1\over2^n}	\\
f(n)&=\sin(n\pi/6)	\\
f(i)&={(i-1)(i+2)\over2^i}	\\
\end{align*}
Frequently these formulas will make sense if thought of either as
functions with domain $\mathbb{R}$ or $\mathbb{N}$, though occasionally one will make
sense for integer values only.

The main question of interest when dealing with sequences is what happens to the terms as we go further and further down the list. In particular, as $i$ becomes extremely large, does $a_i$ get closer to one specific value? This is reminiscent of a question we asked in Chapter~\ref{chap:Limits}, when looking at limits of functions. In fact, the problems are closely related and we define the limit of a sequence in a way similar to Definition~\ref{def:LimitAtInfinity}.

\begin{definition}{Limit of a Sequence}{LimitofaSequence}
Suppose that $\ds\left\{a_n\right\}_{n=1}^\infty$ is a sequence.
We say that $\ds \lim_{n\to \infty}a_n=L$ if for every $\epsilon>0$
there is an $N > 0$ so that whenever $n>N$, $|a_n-L|<\epsilon$. If
$\ds \lim_{n\to\infty}a_n=L$ we say that the sequence \dfont{converges} to $L$, otherwise it \dfont{diverges}.
\end{definition}

Intuitively, $\lim_{n\to\infty}a_2=L$ means that the further we go in the sequence, the closer the terms get to $L$.

\begin{example}{Exponential Sequence}{ExpSequenceEx}
Show that $\{2^{1/n}\}_{n=1}^{\infty}$ converges to 1.
\end{example}
\begin{solution}
Suppose $\epsilon>0$. Then let $N=\frac{1}{\log_2(1+\epsilon)}$. Note that $N>0$. Now if $n>N$, then
\begin{align*}
\ds n&>\frac{1}{\log_2(1+\epsilon)}	\\
\log_2(1+\epsilon)&>\frac{1}{n}\\
1+\epsilon&>2^{1/n}	\\
\epsilon&>2^{1/n}-1	\\
\epsilon&>\left|2^{1/n}-1\right|
\end{align*}
\end{solution}

Note that, as in Chapter~\ref{chap:Limits}, we generally need to work ``backwards'' from the last line of the proof to determine how to choose $N$. Having done so, we write the actual proof as we have done here to show that this value of $N$ `works'.

If a sequence is defined by a formula $\{f(i)\}_{i=1}^{\infty}$, we can often expand the domain of the function $f$ to the set of all (or almost all) real numbers. For example, $f(i)=\frac{1}{i}$ is defined for all non-zero real numbers.

When this happens, we can sometimes find the limit of the sequence $\{f(i)\}_{i=1}^{\infty}$ more easily by finding the limit of the function $f(x)$, $x\in\mathbb{R}$, as $x$ approaches infinity.

\begin{theorem}{Limit of a Sequence}{LimSeqTheorem}
If $\lim_{x\to\infty}f(x)=L$, where $f:\mathbb{R}\to\mathbb{R}$, then $\{f(i)\}_{i=1}^{\infty}$ converges to $L$.
\end{theorem}
\begin{proof}
This follows immediately from Definitions~\ref{def:LimitAtInfinity} and \ref{def:LimitofaSequence}.
\end{proof}

Hereafter we will use the convention that $x$ refers to a real-valued variable and $i$ and $n$ are integer-valued.

\begin{example}{Sequence of $1/n$}{SequenceOneOverN}
Show that $\{\frac{1}{n}\}_{n=0}^{\infty}$ converges to 0.
\end{example}
\begin{solution}
Since $\lim_{x\to\infty}\frac{1}{x}=0$, $\lim_{n\to\infty}\frac{1}{n}=0$.
\end{solution}

Note that the converse of Theorem~\ref{thm:LimSeqTheorem} is not true.

Let $f(n)=\sin(n\pi)$. This is the sequence
\[
  \sin(0\pi), \sin(1\pi),\sin(2\pi),\sin(3\pi),\ldots=0,0,0,0,\ldots
\]
since $\sin(n\pi)=0$ when $n$ is an integer. Thus
$\ds \lim_{n\to\infty}f(n)=0$. But $\ds \lim_{x\to\infty}f(x)$, when $x$ is
real, does not exist: as $x$ gets bigger and bigger, the values
$\sin(x\pi)$ do not get closer and closer to a single value, but take
on all values between $-1$ and $1$ over and over. In general, whenever
you want to know $\ds \lim_{n\to\infty}f(n)$ you should first attempt to
compute $\ds \lim_{x\to\infty}f(x)$, since if the latter exists it is also
equal to the first limit. But if for some reason
$\ds \lim_{x\to\infty}f(x)$ does not exist, it may still be true that
$\ds \lim_{n\to\infty}f(n)$ exists, but you'll have to figure out
another way to compute it.

It is occasionally useful to think of the graph of a sequence. Since
the function is defined only for integer values, the graph is just a
sequence of points. In Figure~\ref{fig:GraphsOfSequences} we see the
graphs of two sequences and the graphs of the corresponding real
functions.

\begin{figure}[H]
\centerline{
%\texonly
\vbox{\beginpicture
\normalgraphs
%\ninepoint
\setcoordinatesystem units <0.5truecm,0.5truecm> point at -14 0
\setplotarea x from 0 to 10, y from 0 to 5
\axis left ticks length <2pt> numbered from 0 to 5 by 1 /
\axis bottom  ticks length <2pt> numbered from 0 to 10 by 5 /
\setquadratic
\plot 0.200 5.000 0.445 2.247 0.690 1.449 0.935 1.070 1.180 0.847
1.425 0.702 1.670 0.599 1.915 0.522 2.160 0.463 2.405 0.416
2.650 0.377 2.895 0.345 3.140 0.318 3.385 0.295 3.630 0.275
3.875 0.258 4.120 0.243 4.365 0.229 4.610 0.217 4.855 0.206
5.100 0.196 5.345 0.187 5.590 0.179 5.835 0.171 6.080 0.164
6.325 0.158 6.570 0.152 6.815 0.147 7.060 0.142 7.305 0.137
7.550 0.132 7.795 0.128 8.040 0.124 8.285 0.121 8.530 0.117
8.775 0.114 9.020 0.111 9.265 0.108 9.510 0.105 9.755 0.103
10.000 0.100 /
\put {$f(x)=1/x$} at 5 2.5
\setcoordinatesystem units <0.5truecm,0.5truecm> point at 0 0
\setplotarea x from 0 to 10, y from 0 to 5
\axis left ticks length <2pt> numbered from 0 to 5 by 1 /
\axis bottom  ticks length <2pt> numbered from 0 to 10 by 5 /
\multiput {$\bullet$} at 1 1 2 0.5 3 0.3333 4 0.25 5 0.2
6 0.1667 7 0.1429 8 0.125 9 0.1111 10 0.1 /
\put {$f(n)=1/n$} at 5 2.5
\setcoordinatesystem units <0.63truecm,1truecm> point at -11.1 3
\setplotarea x from 0 to 8, y from -1 to 1
\axis left ticks length <2pt> numbered from -1 to 1 by 1 /
\axis bottom shiftedto y=0 /
\setquadratic
\plot 0.000 0.000 0.080 0.249 0.160 0.482 0.240 0.685 0.320 0.844
0.400 0.951 0.480 0.998 0.560 0.982 0.640 0.905 0.720 0.771
0.800 0.588 0.880 0.368 0.960 0.125 1.040 -0.125 1.120 -0.368
1.200 -0.588 1.280 -0.771 1.360 -0.905 1.440 -0.982 1.520 -0.998
1.600 -0.951 1.680 -0.844 1.760 -0.685 1.840 -0.482 1.920 -0.249
2.000 0.000 2.080 0.249 2.160 0.482 2.240 0.685 2.320 0.844
2.400 0.951 2.480 0.998 2.560 0.982 2.640 0.905 2.720 0.771
2.800 0.588 2.880 0.368 2.960 0.125 3.040 -0.125 3.120 -0.368
3.200 -0.588 3.280 -0.771 3.360 -0.905 3.440 -0.982 3.520 -0.998
3.600 -0.951 3.680 -0.844 3.760 -0.685 3.840 -0.482 3.920 -0.249
4.000 0.000 4.080 0.249 4.160 0.482 4.240 0.685 4.320 0.844
4.400 0.951 4.480 0.998 4.560 0.982 4.640 0.905 4.720 0.771
4.800 0.588 4.880 0.368 4.960 0.125 5.040 -0.125 5.120 -0.368
5.200 -0.588 5.280 -0.771 5.360 -0.905 5.440 -0.982 5.520 -0.998
5.600 -0.951 5.680 -0.844 5.760 -0.685 5.840 -0.482 5.920 -0.249
6.000 0.000 6.080 0.249 6.160 0.482 6.240 0.685 6.320 0.844
6.400 0.951 6.480 0.998 6.560 0.982 6.640 0.905 6.720 0.771
6.800 0.588 6.880 0.368 6.960 0.125 7.040 -0.125 7.120 -0.368
7.200 -0.588 7.280 -0.771 7.360 -0.905 7.440 -0.982 7.520 -0.998
7.600 -0.951 7.680 -0.844 7.760 -0.685 7.840 -0.482 7.920 -0.249
8.000 0.000 /
\put {$f(x)=\sin(x\pi)$} at 4 1.3
\setcoordinatesystem units <0.63truecm,1truecm> point at 0 3
\setplotarea x from 0 to 8, y from -1 to 1
\axis left ticks length <2pt> numbered from -1 to 1 by 1 /
\axis bottom shiftedto y=0 ticks length <2pt> numbered from 1 to 8 by 1 /
\multiput {$\bullet$} at 0 0 1 0 2 0 3 0 4 0 5 0 6 0 7 0 8 0 /
\put {$f(n)=\sin(n\pi)$} at 4 0.5
\endpicture}}
%\endtexonly
\caption{Graphs of sequences and their corresponding real functions.}
\label{fig:GraphsOfSequences}
\end{figure}

Not surprisingly, the properties of limits of real functions translate
into properties of sequences quite easily.
Theorem~\ref{thm:PropertiesLimits} about limits becomes:

\begin{theorem}{Properties of Sequences}{SequenceProperties}
Suppose that $\ds\lim_{n\to\infty}a_n=L$ and
$\ds\lim_{n\to\infty}b_n=M$ and
$k$ is some constant. Then
\begin{align*}
&\lim_{n\to\infty} ka_n = k\lim_{n\to\infty}a_n=kL	\\
&\lim_{n\to\infty} (a_n+b_n) = \lim_{n\to\infty}a_n+\lim_{n\to\infty}b_n=L+M	\\
&\lim_{n\to\infty} (a_n-b_n) = \lim_{n\to\infty}a_n-\lim_{n\to\infty}b_n=L-M	\\
&\lim_{n\to\infty} (a_nb_n) = \lim_{n\to\infty}a_n\cdot\lim_{n\to\infty}b_n=LM	\\
&\lim_{n\to\infty} {a_n\over b_n} = {\lim_{n\to\infty}a_n\over
  \lim_{n\to\infty}b_n}={L\over M},\hbox{ if $M$ is not 0}	\\
\end{align*}
\end{theorem}

Likewise the Squeeze Theorem (\ref{thm:SqueezeTheorem}) becomes:

\begin{theorem}{Squeeze Theorem for Sequences}{SequenceSqueezeTheorem}
Suppose that $\ds a_n \leq b_n \leq c_n$ for all $n>N$, for some $N$.
If $\ds\lim_{n\to\infty}a_n=\ds\lim_{n\to\infty}c_n=L$,
then $\ds\lim_{n\to\infty}b_n=L$.
\end{theorem}

And a final useful fact:

\begin{theorem}{Absolute Value Sequence}{AbsValueSequence}
$\ds\lim_{n\to\infty}|a_n|=0$ if and only if
$\ds\lim_{n\to\infty}a_n=0$.
\end{theorem}

This says simply that the size of $\ds |a_n|$ gets close to zero if and
only if $\ds a_n$ gets close to zero.

\begin{example}{Convergence of a Rational Fraction}{ConvRationalFraction}
Determine whether $\ds\left\{{n\over n+1}\right\}_{n=0}^\infty$ converges or
diverges. If it converges, compute the limit.
\end{example}
\begin{solution}
Defining $f(x)=\frac{x}{x+1}$ we obtain
$$
\lim_{x\to\infty}{x\over x+1}=\lim_{x\to\infty}1-{1\over x+1}=1-0=1.
$$
Thus the sequence converges to 1.
\end{solution}

\begin{example}{Convergence of Ratio with Natural Logarithm}{ConvNatLog}
Determine whether $\ds\bigg\{{\ln n\over n}\bigg\}_{n=1}^\infty$ converges or
diverges. If it converges, compute the limit.
\end{example}
\begin{solution}
We compute
$$\lim_{x\to\infty}{\ln x\over x}=\lim_{x\to\infty}{1/x\over 1}=
0,$$
using L'H\^opital's Rule. Thus the sequence converges to 0.
\end{solution}

\begin{example}{Alternating Ones}{AltOnes}
Determine whether $\ds\{(-1)^n\}_{n=0}^\infty$ converges or
diverges. If it converges, compute the limit.
\end{example}
\begin{solution}
$f(x)=(-1)^x$ is undefined for irrational values of $x$ so $\lim_{x\to\infty}(-1)^x$ does not exist. However, the sequence has a very simple pattern:
$$1,-1,1,-1,1\ldots$$
and clearly diverges.
\end{solution}

\begin{example}{Convergence of Exponential}{ConvExp}
Determine whether $\ds\{(-1/2)^n\}_{n=0}^\infty$ converges or
diverges. If it converges, compute the limit.
\end{example}
\begin{solution}
We consider the sequence
$\ds\{|(-1/2)^n|\}_{n=0}^\infty=\{(1/2)^n\}_{n=0}^\infty$.
Then
$$
  \lim_{x\to\infty}\left({1\over2}\right)^x=\lim_{x\to\infty}{1\over2^x}=0,
$$
so by Theorem~\ref{thm:AbsValueSequence} the sequence converges to 0.
\end{solution}

\begin{example}{Using the Squeeze Theorem for Sequences}{UsingSqThmSeq}
Determine whether $\ds\{(\sin n)/\sqrt{n}\}_{n=1}^\infty$ converges or
diverges. If it converges, compute the limit.
\end{example}
\begin{solution}
Since $|\sin n|\le 1$, $\ds 0\le|\sin n/\sqrt{n}|\le
1/\sqrt{n}$ and we can use Theorem~\ref{thm:SequenceSqueezeTheorem}
with $\ds a_n=0$ and $\ds c_n=1/\sqrt{n}$. Since
$\ds\lim_{n\to\infty} a_n=\ds\lim_{n\to\infty} c_n=0$,
$\ds\lim_{n\to\infty}\sin n/\sqrt{n}=0$ and the sequence converges to 0.
\end{solution}

\begin{example}{Geometric Sequence}{GeometricSequenceEx}
Let $r$ be a fixed real number. Determine when $\{r^n\}_{n=0}^{\infty}$ coverges.
\end{example}
\begin{solution}
A particularly common and useful sequence is $\ds \{r^n\}_{n=0}^\infty$,
for various values of $r$. Some are quite easy to understand: If $r=1$
the sequence converges to 1 since every term is 1, and likewise if
$r=0$ the sequence converges to 0. If $r=-1$ this is
the sequence of Example~\ref{exa:AltOnes} and
diverges. If $r>1$ or $r<-1$ the terms $\ds r^n$ get large without limit,
so the sequence diverges. If $0<r<1$ then the sequence converges to
0. If $-1<r<0$ then $\ds |r^n|=|r|^n$ and $0<|r|<1$, so the sequence
$\ds \{|r|^n\}_{n=0}^\infty$ converges to 0, so also
$\ds\{r^n\}_{n=0}^\infty$ converges to 0.
converges. In summary, $\ds \{r^n\}$ converges precisely when
$-1<r\le1$ in which case
\[
\lim_{n\to\infty} r^n=\left\{
\begin{array}{l l}
0 & \quad \text{if $-1<r<1$}\\
1 & \quad \text{if $r=1$}
\end{array}\right.
\]
\end{solution}

Sequences of this form, or the more general form $\{kr^n\}_{n=0}^{\infty}$, are called \dfont{geometric sequences} or \dfont{geometric progressions}. They are encountered in a large variety of mathematical and real-world applications.

Sometimes we will not be able to determine the limit of a sequence,
but we still would like to know whether it converges. In some cases we
can determine this even without being able to compute the limit.

A sequence is called \dfont{increasing}
or sometimes \dfont{strictly increasing} if $\ds a_i<a_{i+1}$ for all
$i$. It is called \dfont{non-decreasing} or sometimes
(unfortunately) \dfont{increasing} if $\ds a_i\le a_{i+1}$ for all
$i$. Similarly a sequence is \dfont{decreasing} if $\ds a_i>a_{i+1}$ for all $i$
and \dfont{non-increasing} if
$\ds a_i\ge a_{i+1}$ for all $i$. If a sequence has any of these
properties it is called \dfont{monotonic}.

\begin{example}{}{}
The sequence
$$
  \left\{{2^i-1\over2^i}\right\}_{i=1}^\infty=
  {1\over2},{3\over4},{7\over8},{15\over16},\ldots,
$$
is increasing, and
$$
  \left\{{n+1\over n}\right\}_{i=1}^\infty=
  {2\over1},{3\over2},{4\over3},{5\over4},\ldots
$$
is decreasing.
\end{example}

A sequence is \dfont{bounded above}
if there is some number $N$ such that $\ds a_n\le N$ for every $n$,
and \dfont{bounded below} if there is
some number $N$ such that $\ds a_n\ge N$ for every $n$. If a sequence
is bounded above and bounded below it is \dfont{bounded}. If a sequence $\ds
\{a_n\}_{n=0}^\infty$ is increasing or non-decreasing it is bounded
below (by $\ds a_0$), and if it is decreasing or non-increasing it is
bounded above (by $\ds a_0$).  Finally, with all this new terminology
we can state an important theorem.

\begin{theorem}{Bounded Monotonic Sequence}{BoundedMonotonicSequence}
If a sequence is bounded and monotonic then it converges.
\end{theorem}

We will not prove this, but the proof appears in many calculus books. It
is not hard to believe: suppose that a sequence is increasing and
bounded, so each term is larger than the one before, yet never larger
than some fixed value $N$. The terms must then get closer and closer
to some value between $\ds a_0$ and $N$. It need not be $N$, since $N$ may
be a ``too-generous'' upper bound; the limit will be the
smallest number that is above all of the terms $\ds a_i$.

\begin{example}{}{}
Determine whether $\ds\left\{\frac{2^i-1}{2^i}\right\}_{i=1}^{\infty}$ converges.
\end{example}
\begin{solution}
For every $i\geq 1$ we have $0<(2^i-1)/2^i<1$, so the sequence is bounded, and we have already observed that it is necessary. Therefore, the sequence converges.
\end{solution}

We don't actually need to know that a sequence is monotonic to apply
this theorem---it is enough to know that the sequence is
``eventually'' monotonic, that is, that at some point it becomes
increasing or decreasing. For example, the sequence $10$, $9$, $8$,
$15$, $3$, $21$, $4$, $3/4$, $7/8$, $15/16$, $31/32,\ldots$ is not
increasing, because among the first few terms it is not. But starting
with the term $3/4$ it is increasing, so the theorem tells us that the
sequence $3/4, 7/8, 15/16, 31/32,\ldots$ converges.  Since convergence
depends only on what happens as $n$ gets large, adding a few
terms at the beginning can't turn a convergent sequence into a
divergent one.

\begin{example}{}{}
Show that $\ds\{n^{1/n}\}$ converges.
\end{example}
\begin{solution}
We first show that
this sequence is decreasing, that is, that $\ds n^{1/n}>
(n+1)^{1/(n+1)}$. Consider the real function $\ds f(x)=x^{1/x}$ when
$x\ge1$. We can compute the derivative, $\ds f'(x)=x^{1/x}(1-\ln x)/x^2$,
and note that when $x\ge 3$ this is negative. Since the function has
negative slope, $\ds n^{1/n}>
(n+1)^{1/(n+1)}$ when $n\ge 3$. Since all terms of the sequence are
positive, the sequence is decreasing and bounded when $n\ge3$, and so
the sequence converges. (As it happens, we can compute the limit in
this case, but we know it converges even without knowing the limit; see
Exercise~\ref{ex:ExpLimit}.)
\end{solution}

\begin{example}{}{}
Show that $\ds\{n!/n^n\}$ converges.
\end{example}
\begin{solution}
If we look at the ratio of successive terms we see that:
$$
  {a_{n+1}\over a_n} = {(n+1)!\over (n+1)^{n+1}}{n^n\over n!}=
  {(n+1)!\over n!}{n^n\over (n+1)^{n+1}}=
  {n+1\over n+1}\left({n\over n+1}\right)^n=
  \left({n\over n+1}\right)^n < 1.
$$
Therefore $a_{n+1}<a_n$, and so the sequence is decreasing. Since all terms are positive, it is also bounded, and so it must converge.
(Again it is possible to compute the limit; see
Exercise~\ref{ex:FactorialLimit}.)
\end{solution}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\Opensolutionfile{solutions}[ex]
\section*{Exercises for \ref{sec:sequences}}

\begin{enumialphparenastyle}

\begin{ex}\label{ex:ExpLimit}
Compute $\ds\lim_{x\to\infty} x^{1/x}$.
\begin{sol}
$1$
\end{sol}
\end{ex}

\begin{ex}\label{ex:FactorialLimit}
Use the squeeze theorem to show that
$\ds\lim_{n\to\infty} {n!\over n^n}=0$.
\end{ex}

\begin{ex}
Determine whether $\ds\{\sqrt{n+47}-\sqrt{n}\}_{n=0}^\infty$
converges or diverges. If it converges, compute the limit.
\begin{sol}
$0$
\end{sol}
\end{ex}

\begin{ex}
Determine whether
$\ds\left\{{n^2+1\over (n+1)^2}\right\}_{n=0}^\infty$
converges or diverges. If it converges, compute the limit.
\begin{sol}
$1$
\end{sol}
\end{ex}

\begin{ex}
Determine whether
$\ds\left\{{n+47\over\sqrt{n^2+3n}}\right\}_{n=1}^\infty$
converges or diverges. If it converges, compute the limit.
\begin{sol}
$1$
\end{sol}
\end{ex}

\begin{ex}
Determine whether
$\ds\left\{{2^n\over n!}\right\}_{n=0}^\infty$
converges or diverges.
\begin{sol}
$0$
\end{sol}
\end{ex}

\end{enumialphparenastyle}