\lab{The Finite Difference Method}{The Finite Difference Method}
\label{lab:finitedifference2}

A \textbf{finite difference} for a function $f(x)$ is an expression of the form $f(x+s)-f(x+t)$.
Finite differences can give a good approximation of derivatives.

Suppose we have a function $u(x)$, defined on an interval $[a,b]$.
Let $a = x_0, x_1, \ldots x_{n-1}, x_n=b$ be a grid of $n+1$ evenly spaced points, with $x_{i+1}-x_i = h$, where $h = (b-a)/n$.

You are used to seeing the derivative $u'(x)$, which can written in centered-difference form as:
\begin{align*}
u'(x)=\lim_{h\to\infty}\frac{u(x+h)-u(x-h)}{2h}.
 \end{align*}
Suppose we are interested in knowing the value of the derivative at the points $\{x_i\}$. Even if we don't have a formula for $u'(x)$, we can approximate it using finite differences.
We first write the Taylor polynomial expansion of $u(x+h)$ and $u(x-h)$ centered at $x$. This gives

\begin{align}
u(x+h)=u(x)+u'(x)h+\frac{1}{2}u''(x)h^2+\frac{1}{6}u'''(x)h^3+\mathcal{O}(h^4)\label{plus}\\
u(x-h)=u(x)-u'(x)h+\frac{1}{2}u''(x)h^2-\frac{1}{6}u'''(x)h^3+\mathcal{O}(h^4)\label{minus}
\end{align}
Subtracting \eqref{minus} from \eqref{plus} and rearranging gives
\begin{align*}
u'(x)=\frac{u(x+h)-u(x-h)}{2h}+\mathcal{O}(h^2).
\end{align*}

In terms of our grid points $\{x_i\}$, we have:
\begin{align*}
u'(x_i) \approx \frac{u(x_i+h)-u(x_i-h)}{2h}=\frac{u(x_{i+1})-u(x_{i-1})}{2h}.
\end{align*}
We won't worry about the derivative at the endpoints, $u'(x_0)$ and $u'(x_n)$.
This allows us to approximate the values $\{u'(x_i)\}$ as the solution to a system of equations:

\begin{align}
\frac{1}{2h}
\begin{array}{c}
\begin{bmatrix}
-1 &0 & 1 & \\
& -1 & 0 & 1  \\
& & \ddots & \ddots & \ddots & \\
& & & -1 & 0 & 1 \\
& & & & -1 & 0 & 1
\end{bmatrix}\\
{\scriptstyle\textcolor{red}{(n-1)\times(n+1)}}
\end{array}\cdot
\begin{array}{c}
\begin{bmatrix}
u(x_0) \\ u(x_1)\\ \vdots  \\ u(x_{n-1}) \\ u(x_n)
\end{bmatrix}\\
{\scriptstyle\textcolor{red}{(n+1)\times1}}
\end{array} \approx
\begin{array}{c}
\begin{bmatrix}
u'(x_1) \\ u'(x_2)\\ \vdots  \\ u'(x_{n-2}) \\ u'(x_{n-1})
\end{bmatrix}\\
{\scriptstyle\textcolor{red}{(n-1)\times1}}
\end{array}.
\label{finitedifference2:up}
\end{align}

\noindent This can be rewritten with a $(n-1)\times(n-1)$ tridiagonal matrix instead:
\begin{align}
\frac{1}{2h}
\begin{array}{c}
\begin{bmatrix}
0 & 1 & \\
-1 & 0 & 1  \\
& \ddots & \ddots & \ddots \\
& & -1 & 0 & 1 \\
& & & -1 & 0
\end{bmatrix}\\
{\scriptstyle\textcolor{red}{(n-1)\times(n-1)}}
\end{array}\cdot
\begin{array}{c}
\begin{bmatrix}
u(x_1) \\ u(x_2)\\ \vdots  \\ u(x_{n-2}) \\ u(x_{n-1})
\end{bmatrix}\\
{\scriptstyle\textcolor{red}{(n-1)\times1}}
\end{array} +
\begin{array}{c}
\begin{bmatrix}
-u(x_0)/(2h) \\ 0 \\ \vdots  \\ 0 \\ u(x_n)/(2h)
\end{bmatrix}\\
{\scriptstyle\textcolor{red}{(n-1)\times1}}
\end{array} \approx
\begin{array}{c}
\begin{bmatrix}
u'(x_1) \\ u'(x_2)\\ \vdots  \\ u'(x_{n-2}) \\ u'(x_{n-1})
\end{bmatrix}\\
{\scriptstyle\textcolor{red}{(n-1)\times1}}
\end{array}.
\label{finitedifference2:u}
\end{align}

\noindent Next, we will consider the approximation for $u''(x)$.
If we let
\[u'(x) \approx \frac{u(x+\frac{h}{2})-u(x-\frac{h}{2})}{h}\]
then
\[u''(x) \approx \frac{u'(x+\frac{h}{2})-u'(x-\frac{h}{2})}{h} \approx \frac{\frac{u((x+\frac{h}{2})+\frac{h}{2})-u((x+\frac{h}{2})-\frac{h}{2})}{h}-\frac{u((x-\frac{h}{2})+\frac{h}{2})-u((x-\frac{h}{2})-\frac{h}{2})}{h}}{h}\]
\[= \frac{u(x+h) - 2u(x) + u(x-h)}{h^2}\]
You can achieve the same result by again consider the Taylor polynomial expansion and adding \eqref{plus} and \eqref{minus} and rearranging.
Thus
\[u''(x_i) \approx \frac{u(x_i+h) - 2u(x_i) + u(x_i-h)}{h^2}= \frac{u(x_{i+1}) - 2u(x_i) + u(x_{i-1})}{h^2}\]
Again ignoring the second derivative at the endpoints, this can be written in matrix form as
\begin{align}
\frac{1}{h^2}
\begin{array}{c}
\begin{bmatrix}
1 & -2 & 1 & \\
& 1 & -2 & 1  \\
& &\ddots &\ddots & \ddots  \\
& & & 1 & -2 & 1 \\
& & & & 1 & -2 & 1
\end{bmatrix}\\
{\scriptstyle\textcolor{red}{(n-1)\times(n+1)}}
\end{array}\cdot
\begin{array}{c}
\begin{bmatrix}
u(x_0) \\ u(x_1)\\ \vdots  \\ u(x_{n-1}) \\ u(x_n)
\end{bmatrix}\\
{\scriptstyle\textcolor{red}{(n+1)\times1}}
\end{array}
 \approx
 \begin{array}{c}
\begin{bmatrix}
u''(x_1) \\ u''(x_2)\\ \vdots  \\ u''(x_{n-2}) \\ u''(x_{n-1})
\end{bmatrix}\\
{\scriptstyle\textcolor{red}{(n-1)\times1}}
\label{finitedifference2:uppp}
\end{array}.
\end{align}

\noindent This can also be written with a $(n-1)\times(n-1)$ tridiagonal matrix:
\begin{align}
\frac{1}{h^2}
\begin{array}{c}
\begin{bmatrix}
-2 & 1 & \\
1 & -2 & 1  \\
& \ddots & \ddots & \ddots \\
& & 1 & -2 & 1 \\
& & & 1 & -2
\end{bmatrix}\\
{\scriptstyle\textcolor{red}{(n-1)\times(n-1)}}
\end{array}\cdot
\begin{array}{c}
\begin{bmatrix}
u(x_1) \\ u(x_2)\\ \vdots  \\ u(x_{n-2}) \\ u(x_{n-1})
\end{bmatrix}\\
{\scriptstyle\textcolor{red}{(n-1)\times1}}
\end{array} +
\begin{array}{c}
\begin{bmatrix}
u(x_0)/h^2 \\ 0 \\ \vdots  \\ 0 \\ u(x_n)/h^2
\end{bmatrix}\\
{\scriptstyle\textcolor{red}{(n-1)\times1}}
 \end{array} =
 \begin{array}{c}
\begin{bmatrix}
u''(x_1) \\ u''(x_2)\\ \vdots  \\ u''(x_{n-2}) \\ u''(x_{n-1})
\end{bmatrix}\\
{\scriptstyle\textcolor{red}{(n-1)\times1}}
\label{finitedifference2:upp}
\end{array}
\end{align}

\begin{problem}
	Let $u(x) = \sin((x+\pi)^2-1)$. Use \eqref{finitedifference2:up} - \eqref{finitedifference2:upp} to approximate $\frac{1}{2}u'' - u'$ at the grid points where $a=0$, $b=1$, and $n=10$. Graph the result.
\end{problem}

The previous equations are not only useful for approximating derivatives, but they can be also used to solve differential equations. Suppose that instead of knowing the function $u(x)$, we know that $\frac{1}{2}u'' - u' = f$, where the function $f(x)$ is given. How do we solve for $u(x)$?

\section*{Finite Difference Methods}

Numerical methods for differential equations seek to approximate the exact solution $u(x)$ at some finite collection of points in the domain of the problem.
Instead of analytically solving the original differential equation, defined over an infinite-dimensional function space, they use a well-chosen finite system of algebraic equations to approximate the original problem.

Consider the following differential equation:
\begin{align}
	\begin{split}
	&{ } \epsilon u''(x)-u(x)'= f(x), \quad x \in (0,1), \\
	&{ } u(0) = \alpha, \quad u(1) = \beta.
	\end{split} \label{eqn:singular_perturbed_BVP}
\end{align}
Equation \eqref{eqn:singular_perturbed_BVP} can be written $D u = f,$ where $D = \epsilon \frac{d^2}{dx^2} - \frac{d}{dx}$ is a differential operator defined on the infinite-dimensional space of functions that are twice continuously differentiable on $[0,1]$ and satisfy $u(0) = \alpha$, $u(1) = \beta$.

We look for an approximate solution $\{U_i\}$, where \[U_i \approx u(x_i)\] on an evenly spaced grid of points, $a = x_0, x_1, \ldots, x_n = b$,.
Our finite difference method will replace the differential operator $D = \epsilon \frac{d^2}{dx^2} - \frac{d}{dx}$, (which is defined on an infinite-dimensional space), with finite difference operators (defined on a finite dimensional space).
To do this, we replace derivative terms in the differential equation with appropriate difference expressions.

Recalling that
\begin{align*}
\frac{d^2}{dx^2} u(x_i) &= \frac{u(x_{i+1})- 2u(x_i) + u(x_{i-1})}{h^2} + \mathcal{O}(h^2),\\
\frac{d}{dx} u(x_i) &= \frac{u(x_{i+1})-u(x_{i-1})}{2h} + \mathcal{O}(h^2).
\end{align*}
we define the finite difference operator $D_h$ by
\begin{align}
D_h U_i = \epsilon \frac{1}{h^2}\left(U_{i+1} -2U_i + U_{i-1}\right) - \frac{1}{2h} \left(U_{i+1}-U_{i-1} \right). \label{fd_operator}
\end{align}
Thus we discretize equation \eqref{eqn:singular_perturbed_BVP} using the equations
\begin{align*}
	\frac{\epsilon}{h^2} (U_{i+1}- 2U_i + U_{i-1}) - \frac{1}{2h} \left(U_{i+1}-U_{i-1} \right)  &= f(x_i), \quad i = 1, \ldots, n-1,
\end{align*}
along with boundary conditions $U_{0} = \alpha$, $U_{n} = \beta$.

This gives $n+1$ equations and $n+1$ unknowns, and can be written in matrix form as
\[ \frac{1}{h^2}
\begin{array}{c}
\begin{bmatrix}h^2 & 0 &0&\hdots &0 \\ (\epsilon + h/2) &-2\epsilon & (\epsilon - h/2) &\hdots &0\\ \vdots &  & \ddots & &\vdots \\
0 & \hdots & (\epsilon + h/2) &-2\epsilon & (\epsilon - h/2) \\ 0 & \hdots & & 0 & h^2
\end{bmatrix}\\
{\scriptstyle\textcolor{red}{(n+1)\times(n+1)}}
\end{array}\cdot
\begin{array}{c}
\begin{bmatrix}U_0\\U_1\\ \vdots \\U_{n-1} \\U_{n}\end{bmatrix}\\
{\scriptstyle\textcolor{red}{(n+1)\times1}}
\end{array}=
\begin{array}{c}
\begin{bmatrix}\alpha\\f(x_1)\\ \vdots \\ f(x_{n-1}) \\ \beta \end{bmatrix}\\
{\scriptstyle\textcolor{red}{(n+1)\times1}}
\end{array}.
\]
As before, we can remove two equations to modify the system to obtain an $(n-1)\times (n-1)$ tridiagonal system:
\begin{align}
    \label{eqn:practical_fin_dif}
    \begin{split}
\frac{1}{h^2}
\begin{array}{c}
\begin{bmatrix}-2\epsilon & (\epsilon - h/2) &0&\hdots &0 \\ (\epsilon + h/2) &-2\epsilon & (\epsilon - h/2) &\hdots &0\\ \vdots &  & \ddots & &\vdots \\
0 & \hdots & (\epsilon + h/2) &-2\epsilon & (\epsilon - h/2) \\ 0 & \hdots & & (\epsilon + h/2) &-2\epsilon
\end{bmatrix}\\
{\scriptstyle\textcolor{red}{(n-1)\times(n-1)}}
\end{array}
\cdot
\begin{array}{c}
\begin{bmatrix}U_1\\U_2\\ \vdots \\U_{n-2} \\U_{n-1}\end{bmatrix}\\
{\scriptstyle\textcolor{red}{(n-1)\times1}}
\end{array}\\
 =
 \begin{array}{c}
 \begin{bmatrix}f(x_1) -\alpha(\epsilon + h/2)/h^2 \\f(x_2)\\ \vdots \\ f(x_{n-2})\\ f(x_{n-1})-\beta(\epsilon - h/2)/h^2 \end{bmatrix}\\
 {\scriptstyle\textcolor{red}{(n-1)\times1}}
 \end{array}.
    \end{split}
\end{align}

\begin{problem}
Use equation \eqref{eqn:practical_fin_dif} to solve the singularly perturbed BVP \eqref{eqn:singular_perturbed_BVP} with $\epsilon = 1/10$, $f(x) = -1$, $\alpha=1$, and $\beta=3$ on a grid with $n=30$ subintervals. Graph the solution.
This BVP is called singularly perturbed because of the location of the parameter $\epsilon$.
For $\epsilon = 0$ the ODE has a drastically different character - it then becomes first order, and can no longer support two boundary conditions.
    \label{prob:finitedifference2:prob1}
\end{problem}

\begin{figure}[h]
\centering
\includegraphics[width=10cm]{figure2.pdf}
\caption{The solution to Problem \ref{prob:finitedifference2:prob1}.
The solution gets steeper near $x = 1$ as $\epsilon $ gets small.}
\end{figure}

\begin{comment}
\begin{lstlisting}
import numpy as np
from scipy.sparse import spdiags
from scipy.sparse.linalg import spsolve
import matplotlib.pyplot as plt

def bvp(func, epsilon, alpha, beta, N):
	a,b = 0., 1. 	# Interval for the BVP
	h = (b-a)/N 	# The length of each subinterval

	# Initialize and define the vector F on the right
	F = np.empty(N-1.)
	F[0] = func(a+1.*h)-alpha*(epsilon+h/2.)*h**(-2.)
	F[N-2] = func(a+(N-1)*h)-beta*(epsilon-h/2.)*h**(-2.)
	for j in xrange(1,N-2):
		F[j] = func(a + (j+1)*h)

	# Here we define the arrays that will go on the diagonals of A
	data = np.empty((3,N-1))
	data[0,:] = -2.*epsilon*np.ones((1,N-1)) # main diagonal
	data[1,:]  = (epsilon+h/2.)*np.ones((1,N-1))  	 # off-diagonals
	data[2,:] = (epsilon-h/2.)*np.ones((1,N-1))
	# Next we specify on which diagonals they will be placed, and create A
	diags = np.array([0,-1,1])
	A=h**(-2.)*spdiags(data,diags,N-1,N-1).asformat('csr')

	U = np.empty(N+1)
	U[1:-1] = spsolve(A,F)
	U[0], U[-1] = alpha, beta
	return np.linspace(a,b,N+1), U

x, y = bvp(lambda x:-1., epsilon=.05,alpha=1, beta=3, N=400)
plt.plot(x,y,'-k',linewidth=2.0)
plt.show()

\end{lstlisting}
\end{comment}

\section*{A heuristic test for convergence}
\begin{figure}[ht]
\centering
\includegraphics[width=12cm]{example_convergence.pdf}
\caption{Demonstration of second order convergence for the finite difference approximation \eqref{fd_operator} of the BVP given in \eqref{eqn:singular_perturbed_BVP} with $\epsilon = .5$. } \label{fig:finitedifference2}
\end{figure}

The finite differences used above are second order approximations of the first and second derivatives of a function.  It seems reasonable to expect that the numerical solution would converge at a rate of about $\mathcal{O}(h^2)$. How can we check that a numerical approximation is reasonable?

Suppose a finite difference method is $\mathcal{O}(h^p)$ accurate.
This means that the error $E(h) \approx Ch^p$ for some constant $C$ as $h \to 0$ (in other words, for $h>0$ small enough).

So compute the approximation $y_k$ for each stepsize $h_k$, $h_1 > h_2> \ldots>h_m$.
$y_m$ should be the most accurate approximation, and will be thought of as the true solution.
Then the error of the approximation for
stepsize $h_k, k < m,$ is
\begin{align*}
	E(h_k) &= \max( \abs{ y_k - y_m}) \approx C h_k^p ,\\
	\log(E(h_k)) &= \log(C) + p \log(h_k).
\end{align*}

Thus on a log-log plot of $E(h)$ vs. $h$, these values should be on a straight line with slope $p$ when $h$ is small enough to start getting convergence. We should note that demonstrating second-order convergence does NOT imply that the numerical approximation is converging to the correct solution.

\begin{problem}
Visualize the $\mathcal{O}(h^2)$ convergence of this finite difference method by producing a loglog plot similar to Figure \ref{fig:finitedifference2}, except in the case $\epsilon=.1$. Implement a function \li{singular_bvp} to compute the finite difference solution to \ref{eqn:singular_perturbed_BVP}. Using $n = 5 \times 2^0,5 \times 2^1, \dots ,5 \times 2^9$ subintervals, compute 10 approximate solutions.

To produce the plot, treat the approximation with $n = 5 \times 2^9$ subintervals as the "true solution", and measure the error for the other approximations against it. Note that, since the number of subintervals for each approximation is a multiple of 2, we can compute the $L_{\infty}$ error for the $n=5 \times 2^j$ approximation by using the \li{step} argument in the array slicing syntax:
\begin{lstlisting}
# best approximation
sol_best = singular_bvp(eps,alpha,beta,f,5*(2**9))

# approximation with 5*(2^j) intervals
sol_approx = singular_bvp(eps,alpha,beta,f,5*(2**j))

# approximation error
error = np.max(np.abs(sol_approx - sol_best[::2**(9-j)]))
\end{lstlisting}
\end{problem}

\begin{problem}
Extend your finite difference code to the case of a general second order linear BVP with boundary conditions:
\begin{align*}
	&{ } a_1(x)y'' +a_2(x)y'+ a_3(x) y = f(x), \quad x \in (a,b),\\
	&{ } y(a) = \alpha, \quad y(b) = \beta.
\end{align*}
Use your code to solve the boundary value problem
\begin{align*}
	\epsilon y'' - 4(\pi - x^2)y = \cos x, \\
	y(0) = 0, \quad y(\pi/2) = 1,
\end{align*}
for $\epsilon = 0.1$ on a grid with $n=30$ subintervals.
\label{prob:finitedifference2:prob3}
Be sure to modify the finite difference operator $D_h$ in \eqref{fd_operator} correctly.

\end{problem}

\begin{figure}[h]
\centering
\includegraphics[width=10cm]{figure3.pdf}
\caption{The solution to Problem \ref{prob:finitedifference2:prob3}.
% The solution gets steeper near $x = 1$ as $\epsilon $ gets small.
}
\end{figure}

The next few problems will help you test your finite difference code.

\begin{problem}
Numerically solve the boundary value problem
\begin{align*}
	\epsilon y'' + xy' = -\epsilon \pi^2 \cos(\pi x) - \pi x \sin(\pi x), \\
	y(-1) = -2, \quad y(1) = 0,
\end{align*}
for $\epsilon = 0.1, 0.01,$ and $0.001$. Use a grid with $n=150$ subintervals.
\label{prob:finitedifference2:prob4}
\end{problem}

\begin{problem}
Numerically solve the boundary value problem
\begin{align*}
	(\epsilon +x^2)y'' + 4xy' + 2y = 0, \\
	y(-1) = 1/(1+\epsilon), \quad y(1) = 1/(1+\epsilon),
\end{align*}
for $\epsilon = 0.05, 0.02$. Use a grid with $n=150$ subintervals.
\label{prob:finitedifference2:prob5}
\end{problem}

\begin{figure}[h]
\centering
\includegraphics[width=11cm]{figure4.pdf}
\caption{The solution to Problem \ref{prob:finitedifference2:prob4}.
}
\end{figure}

\begin{figure}
\centering
\includegraphics[width=11cm]{figure5.pdf}
\caption{The solution to Problem \ref{prob:finitedifference2:prob5}.
}
\end{figure}


\section*{Exercises}

\begin{ex}
  Which of the following vectors are redundant? If there are redundant
  vectors, write each of them as a linear combination of previous
  vectors.
  \begin{equation*}
    \vect{u}_1 = \begin{mymatrix}{r} 1 \\ 0 \\ 1 \end{mymatrix},\quad
    \vect{u}_2 = \begin{mymatrix}{r} 2 \\ 0 \\ 2 \end{mymatrix},\quad
    \vect{u}_3 = \begin{mymatrix}{r} 1 \\ 2 \\ 1 \end{mymatrix},\quad
    \vect{u}_4 = \begin{mymatrix}{r} 1 \\ 6 \\ 1 \end{mymatrix}.
  \end{equation*}
  \begin{sol}
    $\vect{u}_2$ and $\vect{u}_4$ are redundant. We have
    $\vect{u}_2=2\vect{u}_1$ and $\vect{u}_4=3\vect{u}_3-2\vect{u}_1$.
  \end{sol}
\end{ex}

\begin{ex}
  Which of the following vectors are redundant? If there are redundant
  vectors, write each of them as a linear combination of previous
  vectors.
  \begin{equation*}
    \vect{u}_1 = \begin{mymatrix}{r} -1 \\ -2 \\ 2 \\ 3 \end{mymatrix},\quad
    \vect{u}_2 = \begin{mymatrix}{r} -3 \\ -4 \\ 3 \\ 3 \end{mymatrix},\quad
    \vect{u}_3 = \begin{mymatrix}{r} 0 \\ -1 \\ 4 \\ 3 \end{mymatrix},\quad
    \vect{u}_4 = \begin{mymatrix}{r} 0 \\ 2 \\ -3 \\ -6 \end{mymatrix}.
  \end{equation*}
  % \begin{sol}
  % \end{sol}
\end{ex}

\begin{ex}
  Use the method of
  Theorem~\ref{thm:characterization-linear-independence} to determine
  whether the following vectors are linearly independent. If they are
  linearly dependent, find a non-trivial linear combination of the
  vectors that is equal to $\vect{0}$.
  \begin{equation*}
    \vect{u}_1 = \begin{mymatrix}{r} 2 \\ 3 \\ 2 \end{mymatrix},\quad
    \vect{u}_2 = \begin{mymatrix}{r} -1 \\ 0 \\ 2 \end{mymatrix},\quad
    \vect{u}_3 = \begin{mymatrix}{r} -3 \\ -4 \\ -2 \end{mymatrix},\quad
    \vect{u}_4 = \begin{mymatrix}{r} 5 \\ 6 \\ 2 \end{mymatrix}.
  \end{equation*}
  \begin{sol}
    The vectors are linearly dependent. We have
    $\vect{u}_2+3\vect{u}_3+2\vect{u}_4=\vect{0}$.
  \end{sol}
\end{ex}

\begin{ex}
  Use the method of
  Theorem~\ref{thm:characterization-linear-independence} to determine
  whether the following vectors are linearly independent. If they are
  linearly dependent, find a non-trivial linear combination of the
  vectors that is equal to $\vect{0}$.
  \begin{equation*}
    \vect{u}_1 = \begin{mymatrix}{r} 1 \\ -1 \\ 0 \\ 1 \end{mymatrix},\quad
    \vect{u}_2 = \begin{mymatrix}{r} 1 \\ 6 \\ 7 \\ 1 \end{mymatrix},\quad
    \vect{u}_3 = \begin{mymatrix}{r} 3 \\ 5 \\ 8 \\ 3 \end{mymatrix},\quad
    \vect{u}_4 = \begin{mymatrix}{r} 1 \\ 0 \\ 1 \\ 1 \end{mymatrix}.
  \end{equation*}
  % \begin{sol}
  % \end{sol}
\end{ex}

\begin{ex}
  Are the following vectors linearly independent? If not, write one of
  them as a linear combination of the others.
  \begin{equation*}
    \vect{u} = \begin{mymatrix}{r} 1 \\ 3 \\ 1 \end{mymatrix},\quad
    \vect{v} = \begin{mymatrix}{r} 1 \\ 4 \\ 2 \end{mymatrix},\quad
    \vect{w} = \begin{mymatrix}{r} 1 \\ 1 \\ -1 \end{mymatrix}.
  \end{equation*}
  \begin{sol}
    The vectors are linearly dependent. We have
    $\vect{w} = 3\vect{u}-2\vect{v}$.
  \end{sol}
\end{ex}

\begin{ex}
  Find a linearly independent set of vectors that has the same span as
  the given vectors.
  \begin{equation*}
    \vect{u}_1 = \begin{mymatrix}{r} 2 \\ 0 \\ 3 \end{mymatrix},\quad
    \vect{u}_2 = \begin{mymatrix}{r} 1 \\ 3 \\ 5 \end{mymatrix},\quad
    \vect{u}_3 = \begin{mymatrix}{r} 3 \\ 3 \\ 8 \end{mymatrix},\quad
    \vect{u}_4 = \begin{mymatrix}{r} 3 \\ -3 \\ 1 \end{mymatrix}.
  \end{equation*}
  \begin{sol}
    $
    \begin{mymatrix}{rrrr}
      2 & 1 & 3 & 3 \\
      0 & 3 & 3 & -3 \\
      3 & 5 & 8 & 1 \\
    \end{mymatrix}
    \roweq
    \begin{mymatrix}{rrrr}
      1 & 4 & 5 & -2 \\
      0 & 1 & 1 & -1 \\
      0 & 0 & 0 &  0 \\
    \end{mymatrix}
    $.
    Linearly independent subset: $\set{\vect{u}_1,\vect{u}_2}$.
  \end{sol}
\end{ex}

\begin{ex}
  Find a linearly independent set of vectors that has the same span as
  the given vectors.
  \begin{equation*}
    \vect{u}_1 = \begin{mymatrix}{r} 1 \\ 3 \\ 3 \\ 1 \end{mymatrix},\quad
    \vect{u}_2 = \begin{mymatrix}{r} 2 \\ 6 \\ 6 \\ 2 \end{mymatrix},\quad
    \vect{u}_3 = \begin{mymatrix}{r} 1 \\ 0 \\ -3 \\ 1 \end{mymatrix},\quad
    \vect{u}_4 = \begin{mymatrix}{r} 1 \\ 2 \\ 1 \\ 1 \end{mymatrix}.
  \end{equation*}
  % \begin{sol}
  % \end{sol}
\end{ex}

\begin{ex}
  Here are some vectors in $\R^4$.
  \begin{equation*}
    \vect{u}_1 = \begin{mymatrix}{r} 1 \\ 1 \\ -1 \\ 1 \end{mymatrix},\quad
    \vect{u}_2 = \begin{mymatrix}{r} 1 \\ 2 \\ -1 \\ 1 \end{mymatrix},\quad
    \vect{u}_3 = \begin{mymatrix}{r} 1 \\ -2 \\ -1 \\ 1 \end{mymatrix},\quad
    \vect{u}_4 = \begin{mymatrix}{r} 1 \\ 2 \\ 0 \\ 1 \end{mymatrix},\quad
    \vect{u}_5 = \begin{mymatrix}{r} 1 \\ -1 \\ -1 \\ 1 \end{mymatrix}.
  \end{equation*}
  Explain why these vectors cannot possibly be linearly
  independent. Then obtain a linearly independent subset of these
  vectors that has the same span as these vectors.
  % \begin{sol}
  % \end{sol}
\end{ex}

\begin{ex}
  Here are some vectors in $\R^4$.
  \begin{equation*}
    \vect{u}_1 = \begin{mymatrix}{r} 1 \\ -1 \\ -1 \\ 1 \end{mymatrix},\quad
    \vect{u}_2 = \begin{mymatrix}{r} -3 \\ 3 \\ 3 \\ -3 \end{mymatrix},\quad
    \vect{u}_3 = \begin{mymatrix}{r} 1 \\ 0 \\ -1 \\ 1 \end{mymatrix},\quad
    \vect{u}_4 = \begin{mymatrix}{r} 2 \\ -9 \\ -2 \\ 2 \end{mymatrix},\quad
    \vect{u}_5 = \begin{mymatrix}{r} 1 \\ 0 \\ 0 \\ 1 \end{mymatrix}.
  \end{equation*}
  Explain why these vectors cannot possibly be linearly
  independent. Then find a non-trivial linear combination of these
  vectors that equals $\vect{0}$.
  % \begin{sol}
  % \end{sol}
\end{ex}

\begin{ex}
  Here are some vectors.
  \begin{equation*}
    \vect{u}_1 = \begin{mymatrix}{r} 1 \\ 1 \\ -2 \end{mymatrix},\quad
    \vect{u}_2 = \begin{mymatrix}{r} 2 \\ 2 \\ -4 \end{mymatrix},\quad
    \vect{u}_3 = \begin{mymatrix}{r} 2 \\ 7 \\ -4 \end{mymatrix},\quad
    \vect{u}_4 = \begin{mymatrix}{r} 5 \\ 7 \\ -10 \end{mymatrix},\quad
    \vect{u}_5 = \begin{mymatrix}{r} 12 \\ 17 \\ -24 \end{mymatrix}.
  \end{equation*}
  Describe the span of these vectors as the span of as few vectors as
  possible.
  \begin{sol}
    \begin{equation*}
      \begin{mymatrix}{rrrrr}
        1 & 2 & 2 & 5 & 12 \\
        1 & 2 & 7 & 7 & 17 \\
        -2 & -4 & -4 & -10 & -24 \\
      \end{mymatrix}
      \roweq
      \begin{mymatrix}{rrrrr}
        1 & 1 & 2 & 5 & 12 \\
        0 & 0 & 5 & 2 & 5 \\
        0 & 0 & 0 & 0 & 0 \\
      \end{mymatrix}.
    \end{equation*}
    Linearly independent subset: $\set{\vect{u}_1,\vect{u}_3}$. Since
    the rank is $2$, this is the smallest possible.
  \end{sol}
\end{ex}

\begin{ex}
  Here are some vectors.
  \begin{equation*}
    \vect{u}_1 = \begin{mymatrix}{r} 1 \\ 2 \\ -2 \end{mymatrix},\quad
    \vect{u}_2 = \begin{mymatrix}{r} 1 \\ 3 \\ -2 \end{mymatrix},\quad
    \vect{u}_3 = \begin{mymatrix}{r} 1 \\ -2 \\ -2 \end{mymatrix},\quad
    \vect{u}_4 = \begin{mymatrix}{r} -1 \\ 0 \\ 2 \end{mymatrix},\quad
    \vect{u}_5 = \begin{mymatrix}{r} 1 \\ 3 \\ -1 \end{mymatrix}.
  \end{equation*}
  Describe the span of these vectors as the span of as few vectors as
  possible.
  % \begin{sol}
  % \end{sol}
\end{ex}

\begin{ex}
  In this exercise, we use scalars from the field $\Z_3$ of integers
  modulo $3$ instead of real numbers (see Section~\ref{sec:fields},
  ``Fields'').  Use the extended casting-out algorithm to determine
  which of the following vectors are redundant. If there are redundant
  vectors, write each of them as a linear combination of previous
  vectors.
  \begin{equation*}
    \vect{u}_1 = \begin{mymatrix}{r} 2 \\ 1 \\ 0 \end{mymatrix},\quad
    \vect{u}_2 = \begin{mymatrix}{r} 1 \\ 1 \\ 2 \end{mymatrix},\quad
    \vect{u}_3 = \begin{mymatrix}{r} 2 \\ 0 \\ 2 \end{mymatrix},\quad
    \vect{u}_4 = \begin{mymatrix}{r} 0 \\ 1 \\ 1 \end{mymatrix}.
  \end{equation*}
  \begin{sol}
    We write the vectors as the columns of a matrix and reduce to
    {\rref} over $\Z_3$:
    \begin{equation*}
      \begin{mymatrix}{rrrr}
        2 & 1 & 2 & 0 \\
        1 & 1 & 0 & 1 \\
        0 & 2 & 2 & 1 \\
      \end{mymatrix}
      \quad\roweq\ldots\roweq\quad
      \begin{mymatrix}{rrrr}
        1 & 0 & 2 & 2 \\
        0 & 1 & 1 & 2 \\
        0 & 0 & 0 & 0 \\
      \end{mymatrix}.
    \end{equation*}
    $\vect{u}_3$ and $\vect{u}_4$ are redundant. We have
    $\vect{u}_3=2\vect{u}_1+1\vect{u}_2$ and
    $\vect{u}_4=2\vect{u}_1+2\vect{u}_2$.
  \end{sol}
\end{ex}

\begin{ex}
  Let $\vect{u},\vect{v},\vect{w}$ be linearly independent vectors in
  $\R^n$. Are the vectors $\vect{u}+\vect{v}$, $2\vect{u}+\vect{w}$,
  and $\vect{w}-2\vect{v}$ linearly independent?
  \begin{sol}
    From
    $a(\vect{u}+\vect{v}) + b(2\vect{u}+\vect{w}) +
    c(\vect{w}-2\vect{v})=\vect{0}$ we get
    $(a+2b)\vect{u} + (a-2c)\vect{v} + (b+c)\vect{w}=\vect{0}$.  Since
    $\vect{u},\vect{v},\vect{w}$ are linearly independent, this last
    system has only the trivial solution, so $a+2b=0$, $a-2c=0$, and
    $b+c=0$. However, these three equations have non-trivial
    solutions, for example $(a,b,c)=(2,-1,1)$. So the vectors
    $\vect{u}+\vect{v}$, $2\vect{u}+\vect{w}$, and
    $\vect{w}-2\vect{v}$ are linearly dependent.
  \end{sol}
\end{ex}

\begin{ex}
  Let $\vect{u},\vect{v},\vect{w}$ be linearly independent vectors in
  $\R^n$. Are the vectors $\vect{u}+\vect{v}$, $\vect{u}+\vect{w}$,
  and $\vect{w}+\vect{v}$ linearly independent?
  \begin{sol}
    From
    $a(\vect{u}+\vect{v}) + b(\vect{u}+\vect{w}) +
    c(\vect{w}+\vect{v})=\vect{0}$ we get
    $(a+b)\vect{u} + (a+c)\vect{v} + (b+c)\vect{w}=\vect{0}$.  Since
    $\vect{u},\vect{v},\vect{w}$ are linearly independent, this last
    system has only the trivial solution, so $a+b=0$, $a+c=0$, and
    $b+c=0$. Solving, we find the unique solution
    $(a,b,c)=(0,0,0)$. So the vectors $\vect{u}+\vect{v}$,
    $\vect{u}+\vect{w}$, and $\vect{w}+\vect{v}$ are linearly
    independent.
  \end{sol}
\end{ex}

\begin{ex}
  Suppose $A$ is an $m\times n$-matrix and
  $\set{\vect{w}_1,\ldots,\vect{w}_k}$ is a linearly independent
  set of vectors in $\R^m$. Now suppose
  $A\vect{z}_i=\vect{w}_i$. Show
  $\set{\vect{z}_1,\ldots,\vect{z}_k}$ is also linearly
  independent.
  \begin{sol}
    Assume
    $a_1\vect{z}_1+\ldots+a_k\vect{z}_k=\vect{0}$. Multiplying
    both sides of the equation by $A$, we get
    \begin{equation*}
      a_1A\vect{z}_1+\ldots+a_kA\vect{z}_k=
      a_1\vect{w}_1+\ldots+a_k\vect{w}_k=\vect{0}.
    \end{equation*}
    Since the $\vect{w}_i$ are linearly independent, it follows that
    each $a_i=0$. Therefore the $\vect{z}_i$ are linearly
    independent as well.
  \end{sol}
\end{ex}


\chapter[Initial value problems]{Initial value problems for Ordinary Differential Equations}

\section{Introduction}

A system of first order differential equation (ODE) is a relationship
between an unknown (vectorial) function $\by(x)$ and its derivative
$\by'(x)$. The general system of first order differential equations
has the form
%
\begin{equation}
  \by'(x)=\boldsymbol{f}(x,\by(x)). \label{odedef}
\end{equation}
%
The solution of the differential equation is a function $\by(x)$ that
satisfies~(\ref{odedef}). Analytic techniques produce a family of
solutions and an initial condition of the form
%
\begin{equation}\label{odeic}
  \by(0)= \by_0
\end{equation}
%
can be used to determine a member of this family. The differential
equations~(\ref{odedef}) and the initial condition~(\ref{odeic})
specify an {\it initial value problem}.  We assume that all the
differential equations that we are going to analyse satisfy the
conditions of the theorem that guarantees the existence and uniqueness
of the solution of an initial value problem.

\noindent \textbf{Remark} - If the original formulation of the problem
is in the form of second or higher order differential equations, we
can always recast it in the form~(\ref{odedef}) by introducing
appropriate new variables.

\section{Numerical differentiation}

We can make use of Taylor's theorem to obtain numerically estimates of
the derivative of a function $f(x)$ at a point $x_0$.  Indicate with
$h$ a small step in the variable $x$.  By Taylor's theorem we have
that:
%
\begin{equation}
  f(x_0+h) = f(x_0) + h f'(x_0) + \frac{h^2}{2} f''(x_0) +
  \frac{h^3}{6} f'''(x_0) + \order{h^4} .
  \label{IVP.eq:1}
\end{equation}
%
Solving for $f'(x_0)$ we obtain the \textit{forward difference
  estimate of the derivative}:
%
\begin{equation}
  f'(x_0) = \frac{f(x_0+h)-f(x_0)}{h} + \order{h} .
  \label{IVP.eq:2}
\end{equation}

\noindent
\textbf{Remark 1} - In other words the slope of the tangent to $f(x)$ at
$x_0$ is approximated with that of the line through $[x_0,f(x_0)]$ and
$[x_0+h,f(x_0+h)]$.

\noindent
\textbf{Remark 2} - The symbol $\order{h}$ on the far right of
equation~(\ref{IVP.eq:2}) indicates that the error of this estimate
decreases linearly with the step size $h$.

Another estimate of the derivative can be obtained by taking a step
backward:
%
\begin{equation}
  f(x_0-h) = f(x_0) - h f'(x_0) + \frac{h^2}{2} f''(x_0) -
  \frac{h^3}{6} f'''(x_0) + \order{h^4} .
  \label{IVP.eq:3}
\end{equation}
%
Solving for $f'(x_0)$ we obtain the \textit{backward difference
  estimate of the derivative}:
%
\begin{equation}
  f'(x_0) = \frac{f(x_0)-f(x_0-h)}{h} + \order{h} .
  \label{IVP.eq:4}
\end{equation}

Both the forward and the backward estimate of the derivative are order
one methods, in the sense that the error in the approximation
decreases with the first power of the step size $h$.  It is possible
to obtain a higher order method by using a more symmetric formula:
subtracting~(\ref{IVP.eq:3}) from~(\ref{IVP.eq:1}) we obtain the
\textit{central difference estimate of the derivative}:
%
\begin{equation}
  f'(x_0) = \frac{f(x_0+h)-f(x_0-h)}{2 h} + \order{h^2} .
  \label{IVP.eq:5}
\end{equation}

\noindent
\textbf{Remark 1} - The geometrical interpretation of this estimate is
that the slope of the tangent to $f(x)$ at $x_0$ is well approximated
by the slope of the secant through $[x_0-h,f(x_0-h)]$ and
$[x_0+h,f(x_0+h)]$.

\noindent
\textbf{Remark 2} - An intuitive explanation of why the central
difference is more accurate than either the forward and the backward
difference is that it contains information on the function on
\underline{both} sides of the point where the derivative is to be
estimated.

\noindent
\textbf{Exercise} - Obtain an accurate estimate of $f''(x_0)$.

\section{Errors}

All procedures to solve numerically an initial value problem consist
of transforming the continuous differential equation~(\ref{odedef})
into a discrete iteration procedure that starting from the initial
condition~(\ref{odeic}) returns the values of the dependent variables
$\by(x)$ at points $x_n = x_0 + n h$, with $h$ a small number called
the \textit{discretisation step}.

In this discretisation and iteration procedure several types of errors
arise.  These are classified as follows:
%
\begin{enumerate}
  \itemsep 0pt
  %
\item Local truncation error
  %
\item Local roundoff error
  %
\item Global truncation error
  %
\item Global roundoff error
  %
\item Total error
  %
\end{enumerate}
%
The \textbf{roundoff error} is caused by the limited precision of
computers.  The \textbf{global roundoff error} is the accumulation of
the local roundoff errors.  The \textbf{total error} is the sum of the
global truncation error and the global roundoff error.

The \textbf{local truncation error} is the error made in one step when
we replace a continuous process (like a derivative) with a discrete
one (like the numerical estimate of the derivative using forward
differences).  The local truncation error is inherent in any
algorithm.  The accumulation of all the local errors in the process of
an iterative procedure (like those used to integrate a differential
equation) give rise to the \textbf{global truncation} error.  Again,
this error will be present even if all calculations are performed
using exact arithmetic.  If the local truncation errors are
$\order{h^{n+1}}$, where $h$ is the discretisation step used in evaluating
the derivatives, then the global truncation error must be $\order{h^n}$
because the number of steps necessary to reach an arbitrary point
$x_f$, having started at $x_0$, is $(x_f-x_0)/h$.  We can proceed more
formally to establish a better bound on the global truncation error
and, more importantly, to understand better its relation with the
original differential equation.  For simplicity we consider only a
single first order differential equation instead of a system
like~(\ref{odedef}).  Moreover, we assume that no roundoff error is
involved.

Consider the initial value problem
%
\begin{equation}
  y'=f(x,y), \qquad y(0)=s, \qquad 0\le x\le X>0.
  \label{IVP.eq:30}
\end{equation}
%
The difference $y(x_n)-y_n$ is the global truncation error. This is
not simply the sum of all local truncation errors that entered at the
previous points. The key point is to understand how two solutions
differ at any point if they are started with different initial
conditions as each step in the numerical solution must use as its
initial value the approximated ordinate computed at the preceding
step.

We assume that $f_y=\pdv{f}{y}(x,y)$ is continuous
and satisfies the condition $f_y\le \lambda$ for $0\le x\le X>0$. The
solution is $y=y(x,s)$. We would like to know how the solution depends
on $s$. Define
%
\begin{equation}
  u(x)=\pdv{y}{s}(x,s) \, .
 \label{IVP.eq:58}
\end{equation}
%
We can obtain a differential equation - the variational equation - for
$u$ by differentiating with respect to $s$ in the initial value
problem~(\ref{IVP.eq:30}) to get
%
\begin{equation}
  u'(x) = \pdv{u}{x} = \pdv{y'}{s} = \pdv{f}{s} = f_y(x,y) u, \qquad u(0)=1, \qquad 0 \le x \le X > 0.
  \label{IVP.eq:31}
\end{equation}
%
Note that if $f_y \le \lambda$ for $0 \le x \le X > 0$, then the solution
of the variational equation satisfies the inequality
%
\begin{equation}
  |u(x)| \le e^{\lambda x}, \qquad 0\le x\le X>0.
  \label{IVP.eq:33}
\end{equation}
%
{\it Proof}: we have
%
\begin{equation}
  u' / u = f_y = \lambda-\alpha(x), \qquad \alpha(x)\ge 0.
  \label{IVP.eq:32}
\end{equation}
%
Integrating this inequality, we obtain
%
\begin{equation}
  \log|u| = \lambda x - \mint{0}{x}{\alpha(\tau)}{\tau}, \implies
  |u(x)| = e^{\lambda x} - \mint{0}{x}{\alpha(\tau)}{\tau} \le e^{\lambda x}.
  \label{IVP.eq:34}
\end{equation}
%
The last inequality is justified because
%
\begin{equation}
  \mint{0}{x}{\alpha(\tau)}{\tau} \ge 0.
  \label{IVP.eq:35}
\end{equation}
%
Using this inequality, it is easy to show that if the initial value
problem is solved with two initial values $s$ and $s+\delta $, the
solutions differ at $x$ by at most $|\delta|e^{\lambda x}$, as
%
\begin{equation}
  |y(x,s) - y(x,s+\delta )| = |\delta|
  \left | \pdv{}{s} y(x,s+\theta\delta) \right| =
  |u(x)| |\delta| \le e^{\lambda x}|\delta|, \qquad 0<\theta<1.
 \label{IVP.eq:36}
\end{equation}

\noindent \textbf{Global Error Theorem}: if all local truncation
errors $\delta_1,\delta_2,...,\delta_n$ do not exceed $\delta$ in
magnitude, then the global truncation error does not exceed
%
\begin{equation}
 \delta \frac{1-e^{n\lambda h}}{1-e^{\lambda h}}  .
 \label{IVP.eq:37}
\end{equation}

\noindent
{\it Proof}: In computing $y_1$ there was an error $|\delta_1|$.  In
computing $y_2$ the global error is
%
\begin{equation}
 |\delta_1| e^{\lambda h} + |\delta_2|,
 \label{IVP.eq:38}
\end{equation}
where the first term in the right-hand side is the error in the
initial condition, and the second term is the new truncation error. In
computing $y_3$ the global error is
%
\begin{equation}
 (|\delta_1| e^{\lambda h} + |\delta_2| ) e^{\lambda h} + |\delta_3|,
 \label{IVP.eq:39}
\end{equation}
%
and so on.  Finally, if $|\delta_i|\le \delta$, $i=1,2,...,n$, we
obtain for the global truncation error
%
\begin{equation}
 |\delta_1| e^{n\lambda h} + |\delta_2| e^{(n-1)\lambda h} + \ldots +
 |\delta_n| \le \delta \frac{1-e^{n\lambda h}}{1-e^{\lambda h}}.
 \label{IVP.eq:40}
\end{equation}

As a consequence of this theorem, if all local truncation errors
$\delta_i=\order{h^{n+1}})$, then the global truncation error is $\order{h^n}$.
In fact the numerator of the fraction is either order one or of the
order of $\exp[\lambda(b-a)]$.  The denominator is of the order of
$h$, assuming that $\lambda h \ll 1$.  If the global truncation error
is $\order{h^n}$ the method is said to be of \textbf{order
  $\boldsymbol{n}$}.

\section{Euler's methods}

In what follows, we assume that equation~(\ref{odedef}) refers to a
single variable, $y(x)$.  This makes the notation lighter: however,
all the results obtained can be extended to systems of first ordinary
differential equations.

We assume that we want to solve the initial value problem
%
\begin{equation}
  y' = f(x,y) , \qquad y(x_0) = y_0 .
  \label{IVP.eq:6}
\end{equation}
%
We introduce a step $h$ and we first obtain an estimate of $y(x)$ at
$x_1 = x_0 + h$ using Taylor's theorem, exactly as we had done for the
forward difference estimate of the derivative.  We obtain
%
\begin{equation}
  y(x_1) \equiv y(x_0+h) = y(x_0) + y'(x_0) h + \order{h^2} =
  y(x_0) + h f(x_0,y(x_0)) + \order{h^2} .
  \label{IVP.eq:7}
\end{equation}
%
By analogy we obtain that the value $y_n$ of the function at the point
$x_n = x_0 + n h$ is given by
%
\begin{equation}
  y_{n+1} \equiv y(x_{n+1}) = y_{n} + h f(x_{n},y_{n}) + \order{h^2} .
  \label{IVP.eq:10}
\end{equation}
%
This iteration scheme to estimate the solution of the initial value
problem~(\ref{IVP.eq:6}) at the points $x_n$ is called \textit{Euler's
  method}.  This method is extremely simple, but very inaccurate and
potentially unstable.  It is reliable only if an extremely small step
$h$ is used.  Its geometrical interpretation is that we use the slope
of the function $y(x_n)$ at the beginning of the interval
$[x_n,x_{n+1}]$ to estimate the value of $y(x_{n+1})$.  This suggests
that we can obtain a better estimate using an ``average'' of the slope
over the same interval, i.e.\ if we could compute
%
\begin{equation}
  y_{n+1} = y_n + h \frac{f(x_n,y_n) + f(x_{n+1},y_{n+1})}{2} .
  \label{IVP.eq:8}
\end{equation}
%
However, we do not know $y_{n+1}$ and so we cannot use this relation
as it stands.  However, we can use equation~(\ref{IVP.eq:7}) to
estimate the value of $y(x)$ at $x_{n+1}$ and use this value in
equation~(\ref{IVP.eq:8}) to obtain a refined estimate.  This
procedure is called the Euler predictor-corrector method and can be
summarised as:
%
\begin{subequations}
  \label{IVP.eq:9}
  \begin{align}
    y^{(p)}_{n+1} &= y_n + h f(x_n,y_n) & \text{Predictor step} \\
    y_{n+1} &= y_n + \frac{h}{2}
    \left [f(x_n,y_n) + f(x_{n+1},y^{(p)}_{n+1}) \right ] &
    \text{Corrector step}.
  \end{align}
\end{subequations}
%
We can show that the predictor-corrector method is of order $\order{h^3}$
by comparing the Taylor expansion of~(\ref{IVP.eq:6}) with that
of~(\ref{IVP.eq:9}).  However, we will not do this here as it is
exactly the same procedure that is used to find the coefficients of
the second order Runge-Kutta method (shown below).

On the other hand, we can find the error for the modified Euler method by
writing the Taylor expansion of equation~(\ref{IVP.eq:6}):
%
\begin{equation}
  y_{n+1} = y_n + y'_n h+ \frac{1}{2} y''_n h^2 + \order{h^3}.
  \label{IVP.eq:11}
\end{equation}
%
Replacing the second derivative by the forward-difference
approximation, we obtain
%
\begin{equation}
  y_{n+1} = y_n + y'_n h + \frac{1}{2} h^2 \frac{y'_{n+1}-y'_n}{h} + \order{h^3},
  \label{IVP.eq:12}
\end{equation}
%
and hence,
%
\begin{equation}
  y_{n+1} = y_n + \frac{h}{2}(y'_{n+1}+y'_n) + \order{h^3}.
  \label{IVP.eq:13}
\end{equation}
%
This shows that the error of one step of the modified Euler method is
$\order{h^3}$. This is the \textit{local error}. There is an accumulation
of local errors from step to step, so that the error over the whole
range of application, the \textit{global error}, is $\order{h^2}$.

\section{Runge-Kutta Methods}

The Euler method is not very accurate. Much greater accuracy can be
obtained more efficiently using a group of methods named after two
German mathematicians, Runge and Kutta.  The idea behind these methods
is to match the Taylor expansion of $y(x)$ at $x=x_n$ up to the
highest possible and/or convenient order.

As an example, let us consider the derivation of the second order
method.  Here, the increment to $y$ is a weighted average of two
estimates which we call $k_1$ and $k_2$. Thus for the equation
%
\begin{equation}
  \dv{y}{x} = f(x,y),
  \label{IVP.eq:14}
\end{equation}
%
we have
%
\begin{align}
  y_{n+1} & = y_n+a k_1+bk_2, \label{kutta} \\
  k_1    & = h f(x_n,y_n), \label{IVP.eq:15} \\
  k_2    & = h f(x_n + \alpha h, y_n + \beta k_1).\label{IVP.eq:16}
\end{align}
%
We fix the four parameter $a$, $b$, $\alpha$ and $\beta$ so that
(\ref{kutta}) agrees as well as possible with the Taylor series
expansion of the differential equation~(\ref{IVP.eq:14})
%
\begin{align}
  y_{n+1} &= y_n + h y_n' + \frac{h^2}{2} y_n'' + \ldots \nonumber \\
  &= y_n + h f(x_n,y_n) + \frac{h^2}{2} \dv{}{x} f(x_n,
  y_n) + \dots, \nonumber \\
  &= y_n + h f_n + h^2 \left (\frac{1}{2} f_x +
    \frac{1}{2}f_y f_n \right ) + \dots
  \label{tay}
\end{align}
%
where $f_n \equiv f(x_n,y_n)$.  On the other hand, using (\ref{kutta})
we have
%
\begin{equation}
  y_{n+1} = y_n + a h f_n + b h f[x_n+\alpha h, y_n+\beta h f_n].
  \label{IVP.eq:18}
\end{equation}
%
Expand the right-hand side of~(\ref{IVP.eq:18}) in a Taylor series
in terms of $x_n,y_n$
%
\begin{equation}
  y_{n+1} = y_n + a h f_n + b h \left [ f_n + f_x(x_n,y_n) \alpha h +
    f_y(x_n,y_n) f(x_n,y_n) \beta h \right ],
  \label{IVP.eq:19}
\end{equation}
%
or, rearranging,
%
\begin{equation}
  y_{n+1} = y_n + h (a+b) f_n + h^2 \left [ f_x(x_n,y_n) \alpha b +
    f_y(x_n,y_n) f(x_n,y_n) \beta b \right ].
  \label{IVP.eq:20}
\end{equation}
%
This result is identical to the Taylor series expansion (\ref{tay}) if
%
\begin{equation}
  a+b=1, \quad \alpha b=\frac{1}{2}, \quad\beta b=\frac{1}{2}.
  \label{IVP.eq:21}
\end{equation}
%
Note that there are only three equations to be satisfied by the four
unknowns. We can therefore assign an arbitrary value to one of the
unknowns. For example, if we take $a=b=\frac{1}{2}$, and
$\alpha=\beta=1$, we obtain the Euler predictor-corrector method.

Fourth-order Runge-Kutta methods are the most widely used and are
derived in a similar way. The most commonly used set of values leads
to the algorithm
%
\begin{equation}
  y_{n+1} = y_n + \frac{1}{6}(k_1+2k_2+2k_3+k_4),
  \label{IVP.eq:22}
\end{equation}
%
with
%
\begin{subequations}
\label{IVP.eq:23}
  \begin{align}
    k_1 &= h f(x_n,y_n), &
    k_2 &= h f(x_n+\frac{1}{2}h,y_n+\frac{1}{2}k_1), \\
    k_3 &= h f(x_n+\frac{1}{2}h,y_n+\frac{1}{2}k_2), &
    k_4 &= hf(x_n+h,y_n+k_3).
  \end{align}
\end{subequations}
%
The local error term for the fourth-order Runge-Kutta methods is
$\order{h^5}$; the global error is $\order{h^4}$.

\section{Error in Runge-Kutta methods}

Whatever the method used one needs to check whether the results of the
integration are reliable, i.e.\ one would like an estimate of the local
truncation error.  Moreover, the knowledge of this quantity would
allow us to use a variable step in integrating the differential
equation: if the local error is much smaller than a predefined
threshold then the step size can be increased.  If it is larger the
step size should be decreased.

Call $y_e(x_0+h)$ the exact value of the solution $y(x)$ of the
initial value problem
%
\begin{equation}
  y'(x) = f(x,y) , \qquad y(x_0) = y_0 .
\end{equation}
%
and indicate with $y_h(x_0+h)$ the solution obtained using a fourth
order Runge-Kutta method with step $h$.  By construction we have that
%
\begin{equation}
  |y_e(x_0+h) - y_h(x_0+h)| = C h^5.
  \label{IVP.eq:41}
\end{equation}
%
Here $C$ is a number independent of $h$ but dependent on $x_0$ and on
the function $y_e$.  To estimate $C h^5$ and, hence, the local error
we assume that $C$ does not change as $x$ changes from $x_0$ to $x_0 +
h$ (a similar procedure is used in Richardson's extrapolation).  Let
$y_{h/2}(x_0+h)$ be the solution obtained using two steps of length
$h/2$ of the fourth order Runge-Kutta method.  By assumption we have
that
%
\begin{subequations}
\label{IVP.eq:42}
  \begin{align}
    y_e(x_0+h) &= y_{h}(x_0+h) + C h^5 , \\
    y_e(x_0+h) &= y_{h/2}(x_0+h) + 2 C (h/2)^5 .
  \end{align}
\end{subequations}
%
By subtraction we obtain from these two equations that
%
\begin{equation}
  \text{Local truncation error} = C h^5 =
  \frac{y_h - y_{h/2}}{1-2^{-4}} .
  \label{IVP.eq:43}
\end{equation}
%
Thus the local truncation error is approximately $y_h - y_{h/2}$.

This estimate of the local error is rather expensive if used in a
variable step algorithm, because it requires two integrations to be
run at the same time for a total of 12 function evaluations.

A second and more efficient method is to compare the result of the
fourth order with those of a fifth order Runge-Kutta method.  As we
have seen in the derivation of the Runge-Kutta method of order 2, a
number of parameters must be selected.  A similar selection process
occurs in establishing higher order Runge-Kutta methods.
Consequently, there is not just one Runge-Kutta method of each order,
but a family of methods.  As shown in the following table, the number
of required function evaluations increases more rapidly than the order
of the Runge-Kutta methods:

\begin{center}
  \begin{tabular}{l|llllllll}
    Number of function evaluations & 1 & 2 & 3 & 4 & 5 & 6 & 7 & 8 \\ \hline
    Maximum order of Runge-Kutta method & 1 & 2 & 3 & 4 & 4 & 5 & 6 & 6
  \end{tabular}
\end{center}

\noindent
This makes the higher-order Runge-Kutta methods less attractive than
the fourth order method, since they are more expensive to use.
However, Fehlberg (1969) devised a fourth order Runge-Kutta method
that required five function evaluation and a fifth order Runge-Kutta
method that required six function evaluations, five of which were the
same as those used for the fourth-order method.  Therefore, with only
six function evaluations we have a fourth-order Runge-Kutta method
with estimate of the local error.

\section{Multistep Methods}

\subsection{Introduction}

The modified Euler method and Runge-Kutta methods for solving initial
value problem are single-step methods since they do not use any
knowledge of prior values of $y(x)$ when the solution is being
advanced from $x$ and $x+h$. If $x_0,x_1,...,x_n$ are steps along the
$x-$axis, then $y_{n+1}$ depends only on $y_n$, and knowledge of
$y_{n-1},...,y_0$ is not used.

It is reasonable to expect that more accurate results may be obtained
if previous values of $y(x)$ were used when estimating $y_{n+1}$.
This idea is at the heart of multi-step methods.  This principle is
transformed into an algorithm by writing the solution of the initial
value problem,
%
\begin{equation}
  \dv{y}{x} = f(x,y), \quad y(x_0)=y_0,
  \label{IVP.eq:25}
\end{equation}
%
as
%
\begin{equation}
  y_{n+1} = y_n + \mint{x_n}{x_{n+1}}{f(x,y(x))}{x}.
  \label{IVP.eq:24}
\end{equation}
%
The integral on the right can be approximated by a numerical
quadrature formula that depends on $\{ x_{n-j}, y_{n-j} \}$ and the
result will be a formula for generating the approximate solution step
by step.

The general form of a multistep method to solve an initial value problem
%
\begin{equation}
  y' = f(x,y), \qquad y(x_0) = y_0 ,
  \label{IVP.eq:44}
\end{equation}
%
is
%
\begin{equation}
  a_k y_{n+1} + a_{k-1} y_{n} + \ldots  + a_0 y_{n+1-k} =
  h [ b_k f_{n+1} + b_{k-1} f_{n} + \ldots + b_0 f_{n+1-k}]
  \label{IVP.eq:45}
\end{equation}
%
Such an algorithm is called a \textit{$k$-step method}. The
coefficients $a_i$, $b_i$ are given. As before, $y_i$ denotes an
approximation to the solution at $x_i=x_0+ih$, and
$f_i=f(x_i,y_i)$. This formula is used to compute $y_{n+1}$, assuming
that $y_{n+1-k},y_{n-k},...,y_{n}$ are known. We assume that $a_k\ne
0$. If $b_k=0$, the method is said to be \textit{explicit}, and
$y_{n+1}$ can be computed directly. In the opposite case the method is
said to be \textit{implicit}.

\subsection{Adams-Bashforth formula}

An explicit multistep method of the type
%
\begin{equation}
  y_{n+1} = y_n + a f_n + b f_{n-1} + c f_{n-2} +...,\label{adams}
\end{equation}
%
where $f_i=f(x_i,y_i)$ belongs to the class of Adam-Bashforth methods.
The Adam-Bashforth formula of order 5, based on the equally spaced
points $x_i = x_0 + i h$, $i=0,1,\ldots,n$ is:
%
\begin{equation}
  y_{n+1} = y_n + \frac{h}{720} [ 1901 f_n - 2774 f_{n-1} +
  2616 f_{n-2} - 1274 f_{n-3} + 251 f_{n-4}].\label{pred}
\end{equation}
%
To obtain the coefficients that appear in this equation we start by
observing that we wish to approximate the integral
%
\begin{equation}
  \mint{x_n}{x_{n+1}}{f[x,y(x)]}{x} \approx h [A f_n + B f_{n-1} +
  C f_{n-2} + D f_{n-3} + E f_{n-4}].
  \label{IVP.eq:26}
\end{equation}
The coefficients $A,B,C,D,E$ are determined by requiring that this
equation is exact whenever the integrand is a polynomial of degree
less than 5. For simplicity we assume that $x_n=0$, $x_{n-1}=-1$,
$x_{n-2}=-2$, $x_{n-3}=-3$, $x_{n-4}=-4$, and $h=1$. We take as a
basis the following five polynomials:
%
\begin{equation}
  \begin{aligned}
    p_0(x)&=1, \\
    p_1(x)&=x, \\
    p_2(x)&=x(x+1), \\
    p_3(x)&=x(x+1)(x+2), \\
    p_4(x)&=x(x+1)(x+2)(x+3) \\
  \end{aligned}
  \label{IVP.eq:27}
\end{equation}
%
When these are substituted in the equation
%
\begin{equation}
  \mint{0}{1}{p_m(x)}{x} \approx A p_m(0)+B p_m(-1)+C p_m(-2)+D p_m(-3)+E p_m(-4)
  \label{IVP.eq:28}
\end{equation}
%
for $m=0,1,2,3,4$, we obtain the system for determination of
$A,B,C,D,E$
%
\begin{equation}
  \left\{
    \begin{aligned}
      A + B + C + D + E & = 1,\\
      -B - 2C - 3D - 4E & = 1/2,\\
      2C + 6D + 12E     & = 5/6,\\
      -6D - 24E         & = 9/4,\\
      24E                & = 251/30.
    \end{aligned} \right .
  \label{IVP.eq:29}
\end{equation}
%
The coefficients of the Adam-Bashforth formula are obtained by back
substitution.

\smallskip

\noindent \textbf{Remark 1} - A special procedure must be employed to
start the method since initially only $y_0 \equiv y(x_0)$ is known. A
Runge-Kutta method is ideal for obtaining $y_1$, $y_2$, $y_3$ and
$y_4$.

\smallskip

\noindent \textbf{Remark 2} - The method is order five, i.e.\ the local
error is $\order{h^6}$ and the global error is $\order{h^5}$.

\subsection{Adams-Moulton formula}

The Adam-Bashforth formulae are most often used in conjunction with
other formulae to enhance their precision.  One such scheme can be set
up by making use of the implicit version of equation~(\ref{adams}):
%
\begin{equation}
  y_{n+1} = y_n + a f_{n+1} + b f_n + c f_{n-1} + \ldots
\end{equation}
%
Following the same steps as in the derivation of the coefficients of
the Adams-Bashforth formula we obtain the \textit{Adams-Moulton
  formula} of order 5:
%
\begin{equation}
  y_{n+1} = y_n + \frac{h}{720} [ 251 f_{n+1} + 646 f_n - 264 f_{n-1} +
  106 f_{n-2} - 19 f_{n-3}].
 \label{corr}
\end{equation}
%
This cannot be used directly as $y_{n+1}$ occurs on both sides of the
equation. However, we can set up a predictor-corrector algorithm that
uses the Adam-Bashforth formula to predict a tentative value for
$y^{(p)}_{n+1}$, and then the Adams-Moulton formula to compute a
corrected value of $y_{n+1}$ using $y^{(p)}_{n+1}$ on the right hand
side of~(\ref{corr}).  In other words, in~(\ref{corr}) we evaluate
$f_{n+1}$ as $f_{n+1}(x_{n+1},y^{(p)}_{n+1})$ using the predicted
value $y^{(p)}_{n+1}$ obtained from the Adam-Bashforth formula.

The Adams-Moulton method is extremely efficient: only two function
evaluations are needed per step for the former method, whereas six are
needed for the Runge-Kutta-Fehlberg method.  All have similar error
terms.  On the other hand changing the step size with the multistep
methods is considerably more awkward than with single step methods.

\subsection{Order of multistep methods}

The \textit{order} of a multistep method is the number of terms in the
Taylor expansion of the solution that are correctly represented by the
method.  The accuracy of a numerical solution is largely determined by
the order of the algorithm used to integrate the initial value
problems.  To determine the order of a given multistep method we
introduce the linear functional:
%
\begin{equation}
  L(y) = \sum_{p=0}^k [a_p y(x_{n-k}+ p h) - h b_p y'(x_{n-k}+p h)].
  \label{IVP.eq:46}
\end{equation}
%
This is a direct representation of~(\ref{IVP.eq:45}) once we take into
account that $y'=f(x,y)$.  If the method~(\ref{IVP.eq:45}) were exact
then we should have $L(y)=0$.  The order of the lowest non-zero term
in the Taylor expansion of~(\ref{IVP.eq:46}) is the order of the
method.

\noindent
Using the Taylor series for $y(p h)$ and $y'(p h)$ with $p=0,1,...,k$
we obtain:
%
\begin{equation}
  y(x_0+p h) = \sum_{j=0}^{+\infty} \frac{(p h)^j}{j!} y^{(j)} (x_0),
  \qquad
  y'(x_0+p h) = \sum_{j=0}^{+\infty} \frac{(p h)^j}{j!} y^{(j+1)}(x_0),
\end{equation}
%
so that
%
\begin{equation}
  L(y) = d_0 y(x_0) + d_1 h y'(x_0) + d_2 h^2 y''(x_0) + \ldots \, ,
\end{equation}
%
where
%
\begin{equation}
  d_0 = \sum_{p=0}^k a_p, \quad
  d_1 = \sum_{p=0}^k (p a_p-b_p), \quad
  d_2 = \sum_{p=0}^k \left ( \frac{p^2}{2} a_p - p b_p \right ), \ldots
\end{equation}
%
and, in general,
%
\begin{equation}
  d_j = \sum_{p=0}^k \left( \frac{p^j}{j!} a_p -
    \frac{p^{j-1}}{(j-1)!} b_p \right) ,
  \qquad j \ge 2 .
\end{equation}
%
If $d_0=d_1=...=d_m=0$, then
%
\begin{equation}
  L(y) = d_{m+1} h^{m+1} y^{(m+1)}(x_0) + \order{h^{m+2}}
\end{equation}
%
represents the local truncation error and the method has order $m$.

\smallskip

\noindent \textbf{Remark} - For the Adam-Bashforth method
$d_0=\ldots=d_5 = 0$ and $d_6=95/288$.  Hence the method if of order
five and the local error is $\order{h^6}$.

\subsection[Convergence, stability and consistency]{Convergence, stability and consistency of multistep methods}

A method to solve numerically initial value problems is
\textit{convergent} if in the limit of infinitely small step size $h$
the numerical solution converges to the exact solution:
%
\begin{equation}
  \lim_{h \to 0} y(x;h) = y(x)
\end{equation}
%
where $y(x)$ is the exact solution of
%
\begin{equation}
  y' = f(x,y), \qquad y(x_0) = y_0
\end{equation}
%
and $y(h;x)$ is the numerical solution obtained using an integration
step $h$.

A method is called \textit{stable} if the numerical solutions are
bounded at all iteration steps over a finite interval.  It is called
\textit{consistent} if at lowest order it is a faithful representation
of the differential equation we wish to integrate.

Convergence, stability and consistency are related one to the other.
In fact, one can show that \textit{a multistep method is convergent if
  and only if it is stable and consistent}.

It is fairly straightforward to determine the stability and
consistency of a multistep method like~(\ref{IVP.eq:45}).  Construct
the two polynomials:
%
\begin{align}
  p(z) &= a_k z^k + a_{k-1} z^{k-1} + \ldots ... + a_0,
  \label{IVP.eq:47} \\
  q(z) &= b_k z^k + b_{k-1} z^{k-1} + \ldots + b_0. \label{IVP.eq:48}
\end{align}
%
The first polynomial is called the \textit{stability polynomial}.  One
can show that a multistep method is consistent if $p(1)=0$ and
$p'(1)=q(1)$.  The question of stability is more complex.

A multi-step method is to all intents and purposes a difference
equation that is (hopefully) based on the differential equation that
we wish to solve, in the sense that as the integration step tends to
zero the difference equation tends to the differential equation
(i.e.\ that the difference equation is \textit{consistent} with the
differential equation).  We expect that a consistent difference
equation has a solution that is close to the solution of the original
differential equation.  However, it is also possible that it has other
solutions and that these may grow as the number of integration steps
increases so that they swamp the ``right'' solution.  We can,
therefore, consider two cases of stability: a method may be stable in
the sense that it represents faithfully the solution of the
differential equation over a finite interval.  However, as the size of
the integration region increases the spurious solutions of the
difference equation grow and the numerical solution no longer has much
to do with the exact solution of the differential equation.  We can
also require a stronger stability: we can, in fact, require that the
spurious solutions tend to zero as the number of integration steps
increases.  In this case we can use the method to integrate a given
differential equation for as long as we wish (within the limits of the
growth of the global integration error).

Call $\{r_p\}$, $=0,1,\ldots,k$ the roots of $p(z)$.  The stability
polynomial satisfies the \textit{root condition} if
%
\begin{equation*}
  |r_p| \le 1, \qquad 0 \le p \le k ,
\end{equation*}
%
and all roots that satisfy $|r_j|=1$ are simple.  The stability
polynomial satisfies the \textit{strong root condition} if
%
\begin{equation*}
  r_0 = 1, \, |r_p| < 1, \qquad 1 \le p \le k.
\end{equation*}

The root conditions are related to the stability of the multi-step
method.

\begin{enumerate}
  %
\item If the stability polynomial satisfies the root condition, then
  the method is stable, in the sense that for $h$ sufficiently small
  it will deliver accurate results over a small interval.
  %
\item If the stability polynomial satisfies the strong root condition,
  then the method is \textit{relatively stable}, meaning that, for $h$
  sufficiently small, the spurious solutions of the difference
  equation go to zero.
  %
\item A method that is stable, but not relatively stable is called
  \textit{weakly stable} and may exhibit diverging solutions for long
  integrations.
  %
\end{enumerate}

\medskip

\section{Summary}

Single-step (multi-stage) and multi-step methods have advantages and
disadvantages.  The following table tries to summarise the main
features of these methods.

\smallskip

\begin{center}
  \begin{tabular}{l|c c}
    & \textbf{Multi-Stage} & \textbf{Multi-Step} \\ \hline
    Self-starting & Yes & No \\
    Easy for variable steps & Yes & No \\
    Computationally efficient & No & Yes \\
    Theory ``intuitive'' & No & Yes
  \end{tabular}
\end{center}

% \begin{center} \begin{tabular}{|p{35mm}|p{35mm}||p{35mm}|p{35mm}|} \hline
%  \multicolumn{2}{|c||}{\textbf{Single-Step}} &
%  \multicolumn{2}{|c|}{\textbf{Multi-Step}} \\ \hline
%  \multicolumn{1}{|c|}{\textbf{Pros}} &
%  \multicolumn{1}{|c||}{\textbf{Cons}} &
%  \multicolumn{1}{|c|}{\textbf{Pros}} &
%  \multicolumn{1}{|c|}{\textbf{Cons}} \\ \hline \hline
%  The theory is intuitive & Computationally intensive &
%  Computationally efficient & The theory is rather involved \\
%  Self-starting & & & Not self-starting \\
%  Easily adapted to variable step algorithms & & &
%  Not easily adapted to variable step algorithms \\ \hline
% \end{tabular} \end{center}

\smallskip

The following table, summarises the main numerical features of some of
the algorithms that we have described.

\smallskip

\begin{center} \begin{tabularx}{1.1\textwidth}{llccccXc} \hline
\textbf{Method} & \textbf{Type} & \parbox[b]{12mm}{\textbf{Local Error}} &
\parbox[b]{12mm}{\textbf{Global Error}} &
\parbox[b]{10mm}{\textbf{F.E. / Step\footnote{Function Evaluations per Step}}} &
\textbf{Stability} &
\parbox[b]{15mm}{\textbf{Ease of changing step size}} &
\parbox[b]{10mm}{\textbf{Recom-mended?}} \\*[2mm] \hline \\
Modified Euler & Single-step & $\order{h^3}$ & $\order{h^2}$ & $2$ &
 Good & Good & No \\*[3mm]
\parbox[t]{25mm}{Fourth-order Runge-Kutta} & Single-step & $\order{h^5}$ &
$\order{h^4}$ & $4$ & Good & Good & Yes \\*[7mm]
\parbox[t]{25mm}{Runge-Kutta-Fehlberg} & Single-step & $\order{h^6}$ &
$\order{h^5}$ & $6$ & Good & Good & Yes \\*[7mm]
Milne & Multistep & $\order{h^5}$ & $\order{h^4}$ & 2 & Poor & Poor & No \\*[4mm]
Adams-Moulton & Multistep & $\order{h^5}$ & $\order{h^4}$ & 2 & Good & Poor &
Yes \\*[3mm] \hline
\end{tabularx} \end{center}

\section*{Further reading}

Topics covered here are also covered in
\begin{itemize}
\item Chapter 10 of Linz \& Wang, \textit{Exploring Numerical Methods}
  (QA297 LIN),
\item Chapter 8 of Kincaid \& Cheney, \textit{Numerical Analysis}
  (QA297 KIN),
\item Chapter 12 of S{\"u}li \& Mayers, \textit{An Introduction to
    Numerical Analysis} (not in library),
\item Part I (especially chapters 1--3, but chapters 4 and 5 are also
  useful) of Iserles, \textit{A First Course in the Numerical Analysis
    of Differential Equations} (QA297 ISE).
\end{itemize}
Note that notation and implied motivation, particularly around the
multistep methods, can be inconsistent with the presentation here.


\subsection{Average Values}
\noindent
We can think of the average value of a function over some interval as the answer to the question: “If I flattened this function into a box over the interval, what would the height of the box be?”\\
For single-variable functions, the answer is $\bar{f} = \frac{1}{b-a}\int_{a}^{b}{f(x)\mathrm{d}x} = \frac{\int_{a}^{b}{f(x)\mathrm{d}x}}{\int_{a}^{b}{\mathrm{d}x}}$.\\
This idea of summing a function over a domain and the dividing by the size of that domain holds into multivariable as
\begin{equation*}
	\bar{f} = \frac{\int\limits_{D}{f(x,y)\mathrm{d}D}}{\int\limits_{D}{\mathrm{d}D}}
\end{equation*}

\input{./multipleIntegrals/meanValueTheorem}


\subsection{The generalized eigenproblem}
\label{sec:generalized-eigenproblem}

An arbitrarily scaled vector $\w_i$ is a \emph{generalized eigenvector} (GEVec) for the ordered matrix pair $(\Rss, \Rnn)$ when the following holds:
%
\begin{equation}
\label{eq:generalized-eigenproblem}
\Rss \w_i = \lambda_i \Rnn \w_i,
\end{equation}
%
for some scalar $\lambda_i$, which is called the \emph{generalized eigenvalue} (GEVal) corresponding to $\w_i$. The largest scalar $\lambda_1$ for which \cref{eq:generalized-eigenproblem} holds is the `first' GEVal, and as mentioned before, the corresponding GEVec $\w_1$ is the solution $\what$ to the optimization problem \cref{eq:argmax_R}.

Since the 1960's, numerically stable algorithms exist that solve the generalized eigenproblem \cref{eq:generalized-eigenproblem} \cite{Golub2013}. A specialized algorithm is applicable when the input matrices are symmetric -- as is the case for $\Rss$ and $\Rnn$. This algorithm (based on a Cholesky factorization and the classical QR-algorithm for ordinary eigenproblems) is implemented in the LAPACK software package (as \texttt{ssygv} and \texttt{dsygv}), and can be easily applied using e.g. the \texttt{eig} function from MATLAB, or the \texttt{eigh} function from SciPy's \texttt{linalg} module.


\lab{Applications}{Least-squares fitting I}{Least-squares fitting I}
\label{LeastSquaresCircle}

\objective{This section will introduce Least Squares and teach a more
advanced application of Least Squares: fitting a circle to data.}
\section*{Least Squares}

It is well known that the displacement of a spring is proportional
to the force acting upon it, that is, $F = k x$.  The proportionality
constant $k$ is called Hooke's spring constant.  Consider a laboratory
experiment where different loads are placed on a spring and the displacement
is measured and recorded in the table below:
\vspace{5mm}\\
\begin{center}
\begin{tabular}{|c|c|}
	\hline
x & F \\
(cm) & (dyne)\\
\hline
1.04  & 3.11 \\
2.03  &  6.01\\
2.95  &  9.07\\
3.92  &  11.99\\
5.06  &  15.02\\
6.00  &  17.91\\
7.07  &  21.12\\
\hline
\end{tabular}
\end{center}
\vspace{5mm}
To find the spring constant $k$, we simply need to solve the following linear system
\[
\begin{pmatrix}
1.04\\
2.03\\
2.95\\
3.92\\
5.06\\
6.00\\
7.07\\
\end{pmatrix}
\begin{pmatrix}k\end{pmatrix} =
\begin{pmatrix}
3.11 \\
6.01\\
9.07\\
11.99\\
15.02\\
17.91\\
21.12\\
\end{pmatrix}.
\]
However, there is no solution to this system because it is overdetermined.
Instead, we seek the ``best'' $k$ that fits the data.
Least squares allows us to find that ``best'' solution.
We can find the least squares solution by computing the following in Python:
\begin{lstlisting}
>>> import numpy as np
>>> from scipy import linalg as la
>>> A = np.vstack([1.04,2.03,2.95,3.92,5.06,6.00,7.07])
>>> b = np.vstack([3.11,6.01,9.07,11.99,15.02,17.91,21.12])
>>> k = np.dot(np.dot(la.inv(np.dot(A.T,A)),A.T),b)
>>> k
array([[ 2.99568294]])
\end{lstlisting}
Hence, we find the spring constant to be $k = 2.9957$.
Note that \li{scipy.linalg} provides a built-in function for solving
least-squares problems.
We plot the data against the best fit as follows:
\begin{figure}[h!]
\label{fig1}
\begin{center}
\includegraphics[width=\textwidth]{line_lstsq}
\caption{The graph of the spring data together with its linear fit}
\label{Fig:SpringFit}
\end{center}
\end{figure}

\begin{lstlisting}
>>> from matplotlib import pyplot as plt
>>> x0 = np.linspace(0,8,100)
>>> y0 = k[0]*x0
>>> plt.plot(A,b,'*',x0,y0)
>>> plt.show()
\end{lstlisting}
See Figure \ref{Fig:SpringFit} to see how well the line fits the data.

\section*{General Line Fitting}

Suppose that we wish to fit a general line, that is $y=m x+b$, to the data set
$\{(x_k,y_k)\}^n_{k=1}$.  Assume that the line does not cross through the origin,
as in the previous example.  Then we seek both a slope and a $y$-intercept.
In this case, we set up the following linear system $A x = b$, or more precisely
\[
\begin{pmatrix}
x_1 & 1\\
x_2 & 1\\
x_3 & 1\\
\vdots & \vdots\\
x_n & 1
\end{pmatrix}
\begin{pmatrix}
m\\
b
\end{pmatrix}=
\begin{pmatrix}
y_1\\
y_2\\
y_3\\
\vdots\\
y_n
\end{pmatrix}.
\]
Note that $A$ has rank $2$ as long as not all of the $x_k$ values are the same.
Hence, the least squares solution
is given by
$$
\widehat{x} = (A^HA)^{-1}A^Hb.
$$
In what sense does this solution give us the best fit line for the data? Recall that since $A$ is injective,
the matrix $A(A^HA)^{-1}A^H$ is an orthogonal projector onto the range of $A$, which means that
$A(A^HA)^{-1}A^Hb = A\widehat{x}$ is the closest vector (with respect to the 2-norm) to $b$ that lies in the
range of $A$. That is, $\widehat{x}$ minimizes the error between $Ax$ and $b$, where the error is given
by the distance between these vectors, $\|b-Ax\|_2$. Another way to say this is that $\widehat{x}$ gives the
values $m$ and $b$ for which the sum of the squares of the distances from each data point $y_k$ to the value
$y = mx_k + b$ is as small as possible.

\section*{Loading Data from .npz Files}
For Least Squares problems as well as in many other contexts, loading data is often a necessary step before
proceeding with further analysis. Here we briefly review another data format in Python and the commands used
to load the data.

A \li{.npz} file is a compressed binary file that contains an archive of NumPy data structures.
A given file may therefore contain several arrays, each array associated with a unique string that identifies it.
When you load a \li{.npz} file in Python, a dictionary-like object is returned, and you can access the data by
providing the appropriate key. Note that when you load a \li{.npz} file, you must also be sure to close it when
you are finished. This is taken care of automatically if you use the \li{with ... as} keywords.

As an example, suppose that we have a file named \li{grades.npz} that contains several arrays, each giving the
homework scores of a particular student in a particular class. Assuming that one of the arrays is associated with
the key \li{'Abe'}, we can load this array in the following way:

\begin{lstlisting}
>>> with np.load('grades.npz') as grades:
>>>     abe_grades = grades['Abe']
>>> abe_grades
array([ 10.,  10.,  10.,  10.,  10.,  10.,  10.,  10.,  10.,  10.])
\end{lstlisting}

You will need to apply this technique in the next problem.

\begin{problem}
Write a function \li{fitLine} that takes no arguments and executes the following.
Load the \texttt{linepts} array from \texttt{data.npz}.
This consists of two columns corresponding to the $x$ and $y$ values of a given data set.
Use least squares to find the slope and $y$-intercept that best fits the data.
Then plot the data points and the line on the same graph.
The function should not return anything.
\end{problem}

\section*{Fitting data to a circle}

Recall that the equation of a circle, with radius $r$ centered at $(c_1,c_2)$, is given by
\begin{equation}
\label{circle}
(x-c_1)^2 + (y-c_2)^2 = r^2.
\end{equation}
Suppose we are given a set of data points closely forming a circle $\{(x_i,y_i)\}^n_{i=1}$.
The ``best'' fit is found via least squares by expanding \eqref{circle} to get
\[
2 c_1 x + 2 c_2 y + c_3 = x^2 + y^2,
\]
where $c_3 = r^2 - c_1^2 - c_2^2$.  Then we can write the linear system $A x = b$ as
\[
\begin{pmatrix}
2 x_1 & 2 y_1 & 1\\
2 x_2 & 2 y_2 & 1\\
\vdots & \vdots & \vdots \\
2 x_n & 2 y_n & 1
\end{pmatrix}
\begin{pmatrix}
c_1\\
c_2\\
c_3
\end{pmatrix}=
\begin{pmatrix}
x_1^2 + y_1^2\\
x_2^2 + y_2^2\\
\vdots\\
x_n^2 + y_n^2
\end{pmatrix},
\]
where the matrix $A$ and the vector $b$ are obtained by the given data and the unknown
$x$ contains the information about the center and radius of the circle and is obtained
by finding the least squares solution.

\section*{Example}

In this section, we fit the following points to a circle:
\begin{align*}
&(134,76),(104,146),(34,176),(-36,146),\\
&(-66,76),(-36,5),(34,-24),(104,5),(134,76)
\end{align*}

We enter them into Python as a $9\times 2$ array:
\begin{lstlisting}
>>> P = np.array([[134,76],[104,146],[34,176],[-36,146],
                  [-66,76],[-36,5],[34,-24],[104,5],[134,76]])
\end{lstlisting}
We compute $A$ and $b$ by entering the following:
\begin{lstlisting}
>>> A = A =np.hstack((2*P, np.ones((9,1))))
>>> b = (P**2).sum(axis=1)
\end{lstlisting}
Hence, we get the least squares solution
\begin{lstlisting}
>>> x = np.dot(np.dot(la.inv(np.dot(A.T,A)),A.T),b)
\end{lstlisting}
Then we find $c_1$, $c_2$, and $r$ by:
\begin{lstlisting}
>>> from math import sqrt
>>> c1, c2, c3 = x
>>> r = sqrt(c1**2 + c2**2 + c3)
\end{lstlisting}
We plot this by executing
\begin{lstlisting}
>>> theta = np.linspace(0,2*np.pi,200)
>>> plt.plot(r*np.cos(theta)+c1,r*np.sin(theta)+c2,'-',P[:,0],P[:,1],'*')
>>> plt.show()
\end{lstlisting}

\begin{problem}
Write a function \li{fitCircle} that does the following.
Load the \texttt{circlepts} array from \texttt{data.npz}.
This consists of two columns corresponding to the $x$ and $y$ values of a given
data set.  Use least squares to find the center and radius of the circle that best
fits the data.  Then plot the data points and the circle on the same graph.
The function should return nothing.
\end{problem}

\begin{problem}
The general equation for an ellipse is:
\[
ax^2 + bx + cxy + dy + ey^2 = 1
\]

Write a function \li{fitEllipse} that uses least squares to fit data to an ellipse.
The function should take a $n\times 2$ array as input, where the first column gives the $x$-coordinates
and the second column gives the $y$-coordinates. Find the least squares solution for $a, b, c, d,$ and $e$,
and return the solution.
\end{problem}

In these Least Squares problems, we have found best fit lines and ellipses relative to the 2-norm.
It is possible to generalize the idea of best fit curves relative to other norms.
See Figure \ref{Fig:ellipse} for an illustration of this.

\begin{figure}[h]
\label{ellipsefit}
\centering
\includegraphics[width=\textwidth]{ellipsefit.pdf}
\caption{Fitting an ellipse using different norms.}
\label{Fig:ellipse}
\end{figure}


\lab{Nearest Neighbor Search}{Nearest Neighbor Search}
\label{lab:NNS}

\objective{Introduce the nearest neighbor search problem, K-D trees, and the curse of dimensionality.}%Teach about branch and bound and the curse of dimensionality using the nearest neighbor search problem.}

\section*{The Nearest Neighbor Search Problem}

The nearest neighbor search problem is an optimization problem that arises in many applications, including computer vision, pattern recognition, internet marketing, and data compression.
The premise is this: suppose that you move into a new city with several post offices.
Since you are a busy individual, to minimize travel time you would like to know which post office is closest to your home.
More generally, the problem is to find the point in a set of points (the post offices) that is nearest to some other point (your home).

% Problem 1: Euclidean Distance
\begin{problem}
To solve the nearest neighbor problem, we must define a metric.
That is, we need a way of determining the distance between two points.
For this lab, we will use standard euclidean distance.

% should we say 1 x n (like usual) or 1 x k (to reflect the k-d tree idea)
Write a function that accepts two vectors (as $1$ x $k$ numpy arrays) and returns the distance between them.
If the two vectors do not have the same dimensions, raise a ValueError.
Make sure the function can handle data in $\mathbb{R}^n$ for any $n\in\mathbb{N}$
\end{problem}

The naive approach to solving the post office problem is to travel from your house to one of the post offices, then from your house to a different post office, and so on until all post offices have been visited. Then choose the post office that it took the least amount of time to get to from your house. This exhaustive method is clearly inefficient, and only feasible if the data set is very small.

% Problem 2: Exhaustive search method
\begin{problem}
Write a function that solves the nearest neighbor search problem by exhaustively checking all of the distances between a point of interest and each point in a data set.

% what should the format of the data be?
The function should take in the set of data points (as a $m$ x $k$ numpy array) and a single target point (as a $1$ x $k$ numpy array).
Return the point in the set that is closest to the point of interest (the nearest neighbor) and the distance between those two points. As in the previous problem, your function should be able to take in data in $\mathbb{R}^n$ for any $n\in\mathbb{N}$.
\end{problem}

% Bad choice of variables k and n
The complexity of this algorithm is $O(kn)$, where $k$ is the number of dimensions and $n$ is the number of data points.

\section*{K-D Trees}

\begin{figure}
\caption{This 2-dimensional tree partitions $\mathbb{R}^2$ at each level. (TODO: Make a figure like the one on Wikipedia or reference Wikipedia)}
\label{fig:k-binary-search}
\end{figure}

Fortunately, there is a particular data structure that lends itself well to solving this problem quickly.
A \emph{k-d tree}, or \emph{k-dimensional tree}, is a specialized binary search tree.
Using a k-d tree to solve a nearest neighbor search problem speeds things up by travelling toward the target incrementally instead of exhaustively searching every option.
Just as a regular binary search tree partitions the number line at every step of descent, a k-d tree partitions $\mathbb{R}^k$ at every step of descent.

At each level of the tree, for some $i$ the nodes to the left of the parent nodes have a lower value in the $i^{th}$ dimension and the nodes to the right of the parent node have a greater value in the $i^{th}$ dimension.
The choice of dimension $i$ cycles through the possible choices as we descend deeper through the tree.
In the $3$-dimensional case the root node is divided in the $x$ dimension, the children in the $y$ dimension, the grandchildren in the $z$ dimension, the great-grandchildren in the $x$ dimension, and so on.
See figure \ref{fig:k-binary-search} for an example in $\mathbb{R}^2$.

Constructing an optimal k-d tree requires sorting the data before each insertion, so the complexity is $O(n\log^2{n})$. However, once the tree is built we can use it as needed with much lower Big-O rates.

In this lab we will construct our own k-d tree class by inheriting from the previous lab's \li{BST} class. First, we need to create specialized nodes.

% KDT Node class -- should probably become example code instead of a problem
\begin{problem}
Copy or import your \li{BSTNode} class from the previous lab.
Write a \li{KDTNode} class that inherits from \li{BSTNode}.
% Optional vvv
Modify the constructor so that a \li{KDTNode} can only hold a list or numpy list.
% Optional ^^^
Add an attribute called \li{axis}.
This attribute will track how deep the node is in the tree so we know which entry of \li{data} to use as the bisector.

Write the \li{__lt__} and \li{__gt__} magic methods so that the $<$ and $>$ operators compare the $i^{th}$ entry of the data, where $i$ is the \li{axis} attribute of the node on the \emph{right side} of the operator.
For example,

\begin{lstlisting}
>>> x = KDTNode([1,2])
>>> y = KDTNode([3,1])
>>> y.axis = 0			# Compare the '0th' entry of the data
>>> x < y				# True, since 1 < 3
True
>>> x > y
False

>>> y.axis = 1			# Compare the '1st' entry of the data
>>> x < y				# False, since 2 > 1
False
>>> x > y
True
\end{lstlisting}

Finally, write the \li{__sub__} magic method so that \li{x} - \li{y} yields the euclidean distance between the data in node \li{x} and the data in node \li{y}.

(Hint: use the function from problem 1)
\end{problem}

Now we must construct the actual k-d tree class.
The structure requires an attribute that houses the dimension of the tree (the \emph{k} of the k-d tree).

\begin{lstlisting}
def class KDT(BST)
	"""A k-dimensional tree object."""

	def __init__(self, k)
	"""Set the dimension attribute 'k'."""
		BST.__init__(self)
		self.k = k
\end{lstlisting}

The only major difference between a k-d tree and a binary search tree is how the data is compared at each depth level.
As long as we set the \li{axis} attribute correctly in each node as it is inserted into the tree, we should be able to do this fairly easily because of our work in the previous problem.
Though we don't need to use a \li{find} function in solving the nearest neighbor problem, we provide the k-d tree version of \li{find} as an instructive example.

\begin{lstlisting}
class KDT(BST):
	# ...

	def find(self, data):
		"""Return the node containing 'data'. Raise a ValueError if there is
		no such node in the tree or if the tree is empty."""

		def _step(current, target):
			"""Recursively approach the target node."""

			if current is None:				# Base case: target not found.
				return current
			if current.data == target.data:	# Base case: target found!
				return current
			if target < current:			# Recursively search to the left
				return _step(current.left, target)
			else:							# Recursively search to the right
				return _step(current.right, target)

		if self.root is None:				# Check for empty tree
			raise ValueError(str(data) + " is not in the tree.")

		# Create a new node so that the KDTNode comparison operators can be used
		n = KDTNode(data)
		found = _step(self.root, n)
		if found is None:					# Report that the data was not found
			raise ValueError(str(data) + " is not in the tree.")
		return found						# Return the node containing 'data'
\end{lstlisting}

At every comparison in the \li{_step} function, the data of \li{target} and \li{current} are compared based on the \li{axis} attribute of \li{current}, since \li{current} is the right-hand operand.
% A little repetitive here
This way if each existing node in the tree has the correct \li{axis}, the correct comparisons will be made as we descend.

% This next bit should probably be part of the next problem.
\begin{comment}
To solve the nearest neighbor search problem, we only need to create the k-d tree once. Then we can use it multiple times with different target points. To prevent the user from messing with the tree, disable the \li{remove} method:
\begin{lstlisting}
class KDT(BST):
	# ...

	def remove(self, *args):
		raise ValueError("remove() has been disabled for this class.")
\end{lstlisting}
\end{comment}

\begin{problem}
Finish the \li{KDT} class by overriding \li{insert} and disabling \li{remove}.

To insert a new node, first check to make sure the data is the proper format. Then find the correct insertion location using an approach similar to the \li{find} method. Remember to double-link the nodes. Set the \li{axis} of the new node appropriately. Do not allow for duplicates in the tree.

To solve the nearest neighbor search problem, we need only create the k-d tree once. Then we can use it multiple times with different target points. To prevent the user from altering the tree, disable the \li{remove} method. Raise a \li{ValueError} if the method is called, and allow it to receive any number of arguments.

\end{problem}

Now that we have a k-d tree, we can use it to solve the nearest neighbor problem.
First we must load the tree with data so that it can be searched.

% This can be done by writing the following short function, or by adding stuff to the constructor of the KDT class.
% I like the constructor idea, but I have only written it so far as a function.
% This could also possibly be absorbed into the next problem.
\begin{problem}
Write a function that creates a \li{KDT} instance and loads it with a set of data.
The data will be a $m$ x $k$ numpy array.
For an optimal k-d tree, the data needs to be inserted in a very particular order.
However, inserting at random still usually produces a good tree.
For this problem, insert the data in the order that it is given (iterate through the rows).
\end{problem}

To solve the nearest neighbor problem, we need to do a careful recursive search through the tree.
At each step, we need to keep track of a current search node, the target point, the current best point (our current candidate for the nearest neighbor), and the current minimum distance (the distance from the nearest neighbor to the target).
We start the algorithm on the root node.
For each recursive step, we first check if the euclidean distance between the current search node and the target point is less than the current minimum distance.
If so, we update the nearest neighbor and calculate the new minimum distance.
We then descend through the tree and search recursively, similar to the \li{find} function.

We aren't done, however; after descending we need to check that there truly is no other nearest neighbor.
Technically, we are checking to see if the hypersphere of radius `minimum distance' around the supposed nearest neighbor does not intersect with any other hyperplanes created by the tree.
In the code we accomplish this by adding the minimum distance to the $i^{th}$ entry of the target point's data, where $i$ is the \li{axis} of the supposed nearest neighbor.
If this sum is greater than the $i^{th}$ entry of the current search node's data, then we need to descend in the \emph{opposite} direction that we came.

We summarize the algorithm below.

\begin{algorithm}
\begin{algorithmic}[1]
%d = euclidean\_metric
\Procedure{KDTSearch}{current, target, neighbor, distance}
\State index = current.axis
\State d = euclidean\_distance
	\If {d(current,target) $<$ distance}
		\State neighbor = current
		\State distance = d(current,target)
	\EndIf

	\If {target.data[index] $<$ current.data[index]}
		\State neighbor, distance = KDTSearch(current.left, target,
			\State									neighbor, distance)
		\If {target.data[index] + distance $>=$ current.data[index]}
			\State neighbor, distance = KDTSearch(current.right, target,
				\State									neighbor, distance)
		\EndIf
	\Else
		\State neighbor, distance = KDTSearch(current.right, target,
			\State									neighbor, distance)
		\If {target.data[index] + distance $<=$ current.data[index]}
			\State neighbor, distance = KDTSearch(current.left, target,
				\State									neighbor, distance)
		\EndIf
	\EndIf
\EndProcedure
\end{algorithmic}
\caption{k-d tree nearest neighbor search}
\label{alg:kdneighborz}
\end{algorithm}

\begin{comment}
\begin{algorithm}
\begin{algorithmic}[1]
\Procedure{KDSearch}{search\_point,parent\_node,b\_point,b\_distance,i}
    \If { Distance(search\_point,parent\_node.point) $<$ b\_distance }
        \State b\_point = parent\_node.point
        \State b\_distance = Distance from search\_point to parent\_node.point
    \EndIf

    \If{search\_point[i] $<$ parent\_node.point[i]}
        \State b\_point, b\_distance =
            \State KDSearch(search\_point,parent\_node.left\_child,b\_point,b\_distance,i+1)
        \If { search\_point[i] + b\_distance $geq$ parent\_node.point}
            \State b\_point, b\_distance =
                \State KDSearch(search\_point,parent\_node.right\_child,b\_point,b\_distance,i+1)
        \EndIf
    \Else
        \State b\_point, b\_distance =
            \State KDSearch(search\_point,parent\_node.right\_child,b\_point,b\_distance,i+1)
        \If {search\_point[i] + b\_distance $leq$ parent\_node.point}
            \State b\_point, b\_distance =
                \State KDSearch(search\_point,parent\_node.left\_child,b\_point,b\_distance,i+1)
        \EndIf
    \EndIf
\EndProcedure
\end{algorithmic}
\caption{Nearest Neighbor}
\label{alg:nearestneighbor}
\end{algorithm}
\end{comment}

\begin{problem}
Use Algorithm \ref{alg:kdneighborz} to write a function that solves the nearest neighbor search problem by searching through a k-d tree.
The function should take in a k-d tree (already loaded with data) and a single point of interest.
Output the nearest neighbor in the tree and the distance from the nearest neighbor to the point of interest.
\end{problem}

\section*{Scipy.spatial.KDTree}

Scipy has a built-in k-d tree structure.
It functions the same way that the \li{KDT} class does, except its operations have been heavily optimized.
To create the tree, we simply give it data in the initializer.
To solve the nearest neighbor search problem, we `query' the tree and give it a target point.
\li{query} returns a tuple of the minimum distance and the index of the nearest neighbor in the data.

\begin{lstlisting}
>>> from scipy.spatial import KDTree
>>> from numpy.random import random

# Initialize the tree with data
>>> data = random((100,20))
>>> scipy_kdt =  KDTree(data)

# Query the tree and print the minimum distance
>>> min_distance, node_index = scipy_kdt.query(random(20))
>>> print(min_distance)
1.1769198641683098

# Print the nearest neighbor by indexing into the tree's data
>>> print scipy_kdt.data[node_index]
array([ 0.30411204,  0.72439179,  0.81654881,  0.37483482,  0.37419281,
        0.19786408,  0.28339901,  0.26042625,  0.95736655,  0.9898518 ,
        0.3523297 ,  0.91607715,  0.91631037,  0.87982453,  0.66185015,
        0.8003657 ,  0.73600775,  0.7150884 ,  0.1836229 ,  0.89556912])
\end{lstlisting}

\section*{The Curse of Dimensionality}

In almost all multidimensional optimization problems, as the number of dimensions increases, the amount of time it takes to solve the problem increases exponentially.
This is the so-called `Curse of Dimensionality'.
In Algorithm \ref{alg:kdneighborz}, as the number of dimensions increases the number of times that we travel down both branches increases. Eventually using a k-d tree actually becomes less efficient than the exhaustive search method.

\begin{problem}
Write a function that creates three distinct figures.
In each figure, plot the time that it takes to solve the nearest neighbor search problem by exhaustive search, then by a k-d tree search using \li{KDT}, then again using \li{scipy.spatial.KDTree}.

In the first figure, use $n$ x 4 data sets with $n$ varying from 10,000 to 100,000 by multiples of 10,000.
Generate the data with \li{numpy.random.random()}.
Time only the searching of the k-d trees, not the building of them.

In the second figure, repeat the process with $n$ x 20 data sets, $n$ varying again from 10,000 to 100,000 by multiples of 10,000.

In the third figure, repeat the process with 20,000 x $k$ data sets, where $k$ varies from 2 to 50.
Here we are increasing the dimension, not the number of data points, and we expect to suffer from the curse of dimensionality.

\begin{figure}[H]
\includegraphics[width=\textwidth]{fourDTime.pdf}
\caption{The first plot may look something like this.
The green line is the naive algorithm and blue line is the kd-tree search.
The kd-tree is the clear winner so far.}
\label{fig:fourDTime}
\end{figure}

\begin{figure}[H]
\includegraphics[width=\textwidth]{twentyDTime.pdf}
\caption{The second graph may look something like this.
The green line is the naive version and blue line is using a kd-tree.
Now the k-d tree search is slower than the naive algorithm.}
\label{fig:twentyDTime}
\end{figure}

\begin{figure}[H]
\includegraphics[width=\textwidth]{curseD.pdf}
\caption{The third graph may look something like this.
Around 15 dimensions the time spikes for the k-d tree search algorithm.
At that point using a kd-tree is even less effective than the naive algorithm.}
\end{figure}

\end{problem}

Certain algorithms suffer less due to the curse of dimensionality, but are not as good in lower dimensions.
Others, like the one we have studied in this lab, are good in lower dimensions but suffer heavily in higher dimensions.
Knowing how to pick your algorithm is an important skill in multidimensional data analysis.

\begin{comment} % END OF LAB. The rest of this needs to be moved elsewhere.

The complexity of this algorithm is $O(k\log(n))$ in optimal time.
Its worst case is $O(k*n^{1-\frac{1}{k}})$ where $k$ is the number of dimensions and $n$ is the number of points in the tree.
The reasons for this are discussed in the next section.

\section{Curse of Dimensionality}

As you increase the number of dimensions the number of times that you have to go down both branches increases.
You get to the point where you eliminate very few points by using a k-d tree.

\begin{problem}
Time both algorithms for the number of data points being $10,000-100,000$ every multiple of $10,000$ with $20$ dimensions.
Plot both times on the same plot. Now how do the two algorithms compare?
\end{problem}

\begin{problem}
Time the SciPy built in function for searching a k-d tree (do not time the building of the kd-tree) for the number of dimensions points being $2-50$ with $20,000$ data points.
Plot the time.
What do you notice?
\li{from scipy.spatial import KDTree} will import the built in k-d tree.
Create the tree by \li{tree = KDTree(data)} and search it by \li{tree.query(point)}.

\li{scipy.spatial} includes a \li{cKDTree} object which is implemented in C.
Is it any better?
\end{problem}

\section*{Classification}
A common problem is correctly classifying data.
Suppose that you have a ten marbles, five of which are blue with a green stripe and the other five red with a purple stripe.
If your friend gave you an eleventh marble that was blue with a brown stripe, which group would you put it in?
Probably you would include it with the other blue marbles.
What if the marble that your friend gave you was blue with a purple stripe?
This marble shares characteristics with both groups.
Where you end up grouping it will depend on which characteristics are most important to you.

This is the intuitive classfication problem.
If we have data that is already grouped into distinct sets, into which set do we put new data?
Classification has myriad and sundry applications.
In this lab we will use it in the context of optical character recognition.

\section*{Nearest Neighbor Classification}
We will now more formally describe the classification problem and explain the nearest neighbor classification algorithm.
Suppose we have a collection of vectors $\{x_1, ..., x_m\}$ in $\R^n$ with corresponding labels $\{l_1, ..., l_k\}$ describing to which group each datum belongs.
This collection of vectors and labels is called our training set.
Each entry of a vector is called a feature and $n$ is the size of our feature set.
For example, consider the following vectors in $\R^3$

\begin{center}
\begin{tabular}{cc}
$(2,0,0)$ & $1$ \\
$(3,0,0)$ & $1$ \\
$(0,3,0)$ & $2$ \\
$(0,2,0)$ & $2$ \\
$(1/10,2,0)$ & $2$ \\
$(0,0,4)$ & $3$ \\
$(0,0,7)$ & $3$ \\
\end{tabular}
\end{center}

If we also have a metric on our space, we may determine the distance between all of these points.
If we are given a new datum and we wish to decide which of the three groups to include it in, one option is to choose the group to which its closest neighbor belongs.
Let us use the euclidean metric to classify $(0,0,5)$ against our training set.
We see that the distance from $(0,0,5)$ and $(0,0,4)$ is only $1$, while the distance to the remaining points is at least $2$.
The label of $(0,0,4)$ is $3$, and so we assign $(0,0,5)$ the same label.

Now, what if we wished to classify $(5/2,5/2,0)$?
This presents a problem since this point is equidistant from the points in label $1$ and label $2$.
In such a case it is up to the programmer to decide how to break the tie.

One case we need to consider is when the data that we are using in our search are on different scales.
For example, suppose we wished to classify people applying for a loan at a bank as `risky' or `safe.'
Further suppose that we know their age, education level, current debt, and how many credit cards they have.
We could encode their education level as integers between $0$ and $4$.

Suppose that Bob is 30 years old, well educated, has a debt of \$5000, and 3 credit cards.
Let's say that James is 18 years old, barely out of high school, \$5000 dollars in debt, and owns 4 credit cards.
If we were to use the Euclidean metric $d$ to measure how close Bob is to James, we would get a distance of
\[
d((30,4,5000,3),(18,0,5000,4)) = 12^2 + 4^2 + 1^2 = 161.
\]

However, if Alice is 29 years old, well educated, has a debt of \$4900, and 3 credit cards, then her distance from Bob is
\[
d((30,4,5000,3),(29,4,4900,3)) = 1^2 + 100^2 = 100001.
\]

Is James closer to Bob than Alice?  Most Banks would say no.
In order to classify these individuals better, we need to measure distance differently.
There are two different ways to handle this.
The first solution is to scale the data before inputting it into our algorithm.
In our previous example we could divide age by 100, education by 4, debt by 1000, and credit cards by 10 and then preform a nearest neighbor search on Bob.
Then the distance from Bob to James is
\[
d\left( \left( \frac{30}{100},\frac{4}{4},\frac{5000}{1000},\frac{3}{10} \right),\left(\frac{18}{100},\frac{0}{4},\frac{5000}{1000},\frac{4}{10}\right)\right) = .12^2 + 1^2 + .1^2 = 1.0244.
\]
while the distace from Alice to Bob is
\[
d\left(\left(\frac{30}{100},\frac{4}{4},\frac{5000}{1000},\frac{3}{10}\right),\left(\frac{29}{100},\frac{4}{4},\frac{4900}{1000},\frac{3}{10}\right)\right) = .1^2 + .01^2 = 0.0101.
\]
So now Bob is closer to Alice.

\begin{problem}
Write a function that takes in data points and the a vector that is used to be the scale.
The function outputs the scaled data points.
So if we were using it for our baking example it would take in the array representing the data Alice, Bob and James (where each person is a row) and the vector $[100,4,1000,10]$ and output the scaled data of Alice, Bob and James.
\end{problem}

Another alternative is to change the metric we use to measure distance with.
In both the examples above we used the Euclidean metric to measure distance.
Suppose we wished to apply the Euclidean metric to data set of colors.
We could map each color to an integer, say red to 1, violet to 2, blue to 3, green to 4, yellow to 5, and orange to 6.
If we use Euclidean metric green is closer to red than orange is when we'd probably consider orange to be the same distance from red as violet is.
In this case the Euclidean metric is not satisfactory and so we might create a new metric to measure the distance between colors.

\section*{K-Nearest Neighbor Classification}
Often we can improve the accuracy of a classifier by looking for other points besides the nearest neighbor.
Instead, we may choose an arbitrary number $k$ and give the point to be classified the majority label in from it's $k$ nearest neighbors.

There are pitfalls to this approach.
Consider a point that's closest neighbor has the label $0$.
If we only considered the nearest neighbor, then we would be finished.
However, what if the next $10$ nearest neighbors all had the label $1$?
Do you think that we should still classify the point as $0$?
What if the nearest point has a distance of $0.1$ and the next $10$ points have a distance of at least $100$?
The answer to these questions depend on the kind of data we are working with and the metric that we choose.
One should ensure to consider how to treat situations like this when working on classification algorithms.

The sklearn library has a neighbors module with a class KNeighborsClassifier for solving the $k$ nearest neighbors problem
\begin{lstlisting}
from sklearn import neighbors
nbrs = neighbors.KNeighborsClassifier(n_neighbors=8, weights='distance', p=2)
\end{lstlisting}

The \li{neighbors.KNeighborsClassifier} sets up the knearestnieghbor algorithm.
\li{n_neighbors} is how many neighbors you would like to find and \li{weights} specifies how you would like to weight the neighbors you have to make the classification.
Weights can be \li{'uniform'}, where the majority classification of the $k$ nearest neighbors determines the new data point's classification, or \li{'distance'}, where the neighbors nearer to the point have more weight than those farther away.
The argument \li{p} correpsods to the distance metric.
For this lab we will use \li{p=2} which is the Euclidean distance.

\begin{lstlisting}
nbrs.fit(points, labels)
\end{lstlisting}
Points and labels are your training data.
The \li{fit} function makes \li{nrbs} create a data structure containing those points and labels ready to be queried.

\begin{lstlisting}
nbrs.predict(testpoints)
\end{lstlisting}

The function \li{predict} takes in points to classify and outputs their respective labels.
%More information about this package can be found at http://scikit-learn.org/stable/modules/neighbors.html.
\begin{problem}
Get the post office handwritten digit data set. Load them with
\begin{lstlisting}
labels, points, testlabels, testpoints = np.load('PostalData.npz').items()
\end{lstlisting}
This contains a training set and a test set.
When you load the data the first entry of each array will be a name.
So \li{points[1]} and \li{labels[1]} point to the actual points and labels you want to use.
Each point is a  image that is $28 \times 28$ matrix of pixels that has been flattened.
The corresponding label indicates which number was written.
Try classifying the testpoints with \li{n_neighbors} as 4 and then as 10 and with \li{weights} \li{'uniform'} and then \li{'distance'}. Then do the classfication with \li{n_neighbors} being 1.
For each one return a report indicating how your classifier performs in terms of misclassifications as a percentange (testlabels are the true labels that correspond to the testpoints).
Which combination gives the most correct classifications?
(You may wish to streamline this process by writting a function that takes in \li{n_neighbors} and \li{weights} as arguments calls the neighbors functions appropriately)

A similar classification process is used by the United States Postal Service to automatically determine the zip code to send a letter to.

\begin{figure}[H]
\includegraphics[width=.25\textwidth]{Example.png}
\caption{An example of the number 6 taken from the data set}
\end{figure}
\end{problem}

\end{comment}