\lab{Algorithms}{Introduction to Wavelets}{Intro to Wavelets}

\objective{This lab explains the basic ideas of Wavelet Analysis
using the Haar wavelet as a prototypical example then presents applications of the Discrete Wavelet Transform in
image denoising and compression.}

Recall that in the context of Fourier analysis, one seeks to represent a
function in the frequency domain, and this is accomplished via the Fourier
transform. The Fourier transform allows us to analyze and process functions
in many useful ways, as you have seen in previous labs. There are, however,
drawbacks to this approach. For example, although a function's Fourier
transform gives us complete information on its frequency spectrum, time
information is lost. We can know which frequencies are the
most prevalent, but not when they occur. This is due in part to the fact that
the sinusoidal function $f(x) = e^{2\pi ix}$ -- on which the Fourier transform
is based -- has infinite support. Its nature is essentially \emph{non-local},
and so the Fourier transform fails to provide local information in both the
time and frequency domains. This brings us to the following question: are
there types of transforms that avoid the shortcomings mentioned above? The
answer is an emphatic yes. Enter Wavelet analysis.

\subsection*{The Haar Wavelet}

As noted earlier, the Fourier transform is based on the complex exponential
function. Let us alter the situation and consider instead the following
function, known as the \emph{Haar wavelet}:
\begin{equation*}
\psi(x) =
 \begin{cases}
  1 & \text{if } 0 \leq x < \frac{1}{2} \\
  -1 & \text{if } \frac{1}{2} \leq x < 1 \\
  0 & \text{otherwise.}
 \end{cases}
\end{equation*}

% It might be nice to plot this function and include the image in the lab.

Along with this wavelet, we introduce the associated \emph{scaling function}:
\begin{equation*}
\phi(x) =
 \begin{cases}
 1 & \text{if } 0 \leq x < 1 \\
 0 & \text{otherwise.}
 \end{cases}
\end{equation*}

From the wavelet and scaling function, we can generate two countable families
of dyadic dilates and translates given by
\begin{equation*}
\psi_{m,k}(x) = \psi(2^mx - k)
\end{equation*}
\begin{equation*}
\phi_{m,k}(x) = \phi(2^mx - k),
\end{equation*}
where $m,k \in \mathbb{Z}$.

Let us focus for the moment on that second family of functions, $\{\phi_{m,k}\}$.
If we fix $m$ and let $k$ vary over the integers, we have a countable collection of
simple functions. The support of a typical function $\phi_{m,k}$ is the interval
$[k2^{-m}, (k+1)2^{-m}]$, and for any $m \in \mathbb{Z}$ we have
\begin{equation*}
\mathbb{R} = \displaystyle\biguplus_k\,[k2^{-m}, (k+1)2^{-m}],
\end{equation*}
where $\uplus$ denotes a union over disjoint sets. Thus, the supports can be viewed as
a discretization of the real line, and we can use this collection of simple functions
to approximate any $f \in L^2(\mathbb{R})$ in the following sense:
\begin{equation*}
f(x) \approx f_m(x) := \displaystyle\sum_{k \in \mathbb{Z}}\alpha_{m,k}\phi_{m,k}(x),
\end{equation*}
where
\begin{equation*}
\alpha_{m,k} := 2^m \displaystyle \int_{k2^{-m}}^{(k+1)2^{-m}}f(x) dx
\end{equation*}
($\alpha_{m,k}$ is simply the average value of $f$ on $[k2^{-m},(k+1)2^{-m}]$). As you
would probably expect, the point-wise error between $f$ and its approximation $f_m$
(called a \emph{frame}) goes to zero as $m \to \infty$.

These frames are not quite good enough, however. Each coefficient $\alpha_{m,k}$
certainly captures local information about $f$ -- namely its average value on
a certain interval -- but it fails to tell us anything about how $f$ changes
on that interval. We need more information than is provided by $f_m$ in order
to know about discontinuities or high-frequency oscillations of $f$. To this end,
we now consider the wavelet function $\psi$.
Notice that the Haar wavelet is oscillatory in nature, and is thus better suited
to capture local information on how a function changes at a given point. For
any given $m$, we define a function $d_m$, called a \emph{detail}, as follows:
\begin{equation*}
d_m(x) := \displaystyle\sum_{k \in \mathbb{Z}}\beta_{m,k}\psi_{m,k}(x),
\end{equation*}
where
\begin{equation*}
\beta_{m,k} := 2^m \displaystyle \int_{-\infty}^{\infty}f(x) \psi_{m,k}(x) dx.
\end{equation*}
Each coefficient $\beta_{m,k}$ gives information about how $f$ changes on the
the interval $[k2^{-m}, (k+1)2^{-m}]$, and larger coefficients correspond
to larger spikes of width $2^{-m}$. Thus, as $m$ increases, the
detail function $d_m$ gives information about the higher-frequency oscillations
of the function. The details and approximation frames interact in the following way:
\begin{equation*}
f_{m+1} = f_m + d_m.
\end{equation*}
As a result of this fortuitous relationship, one can prove the decomposition
\begin{equation*}
L^2(R) = V_0 \oplus W_0 \oplus W_1 \oplus \cdots,
\end{equation*}
where $V_j := \text{span}\{\phi_{j,k}\}_{k \in \mathbb{Z}}$ and
$W_j := \text{span}\{\psi_{j,k}\}_{k \in \mathbb{Z}}$. This fact justifies
our hope to approximate and analyze functions using wavelets.
\begin{figure}[t]
\minipage{0.32\textwidth}
    \includegraphics[width=\linewidth]{sinecurve}
    \caption{$f(x) = \sin (x)$}
\endminipage\hfill
\minipage{0.32\textwidth}
    \includegraphics[width=\linewidth]{discreteSineCurve.pdf}
    \caption{$f_4$}
\endminipage\hfill
\minipage{0.32\textwidth}
    \includegraphics[width=\linewidth]{sineCurveDetail}
    \caption{$d_4$}
\endminipage
\end{figure}
\begin{problem}
Calculate and plot the approximation frames for $f(x) = \sin(x)$ on the interval $[0,2\pi]$
for $m = 4, 6, 8$. Note that because we are working on a finite interval,
we only need to calculate certain coefficients $\alpha_{m,k}$. In
particular, we only need the coefficients for $k = 0$ up to the first integer
$n$ such that $(n+1)2^{-m} > 2 \pi$ (why?). Furthermore, to plot the frame,
all we need is an array containing the relevant coefficients. Then simply plot
the coefficients against \li{linspace} with appropriate arguments
and set \li{drawstyle='steps'} in the \li{plt.plot} function.
\end{problem}

\begin{problem}
Now calculate the details for $f(x) = \sin(x)$ on the same interval and for the
same $m$ values given above. Use previous results to compute the coefficients
for $f_5$, $f_7$, and $f_9$ and plot them.
\end{problem}

\subsection*{The Discrete Wavelet Transform}

What purpose do these details and approximation frames serve? According to the
properties discussed above, we can approximate $L^2$ functions as follows:
\begin{align*}
f \approx f_{J+1} &= f_J + d_J \\
&= f_{J-1} + d_{J-1} + d_J \\
& \ldots\\
&= f_{I} + d_{I} + d_{I+1} + \cdots + d_J,
\end{align*}
where $1 \leq I \leq J$. If $f$ has compact support (as in the case of a finite-time signal,
for example), only finitely many of the coefficients in the frame and the details are
nonzero, thus enabling us to represent $f$ to a reasonable degree of accuracy in a very
efficient manner. The calculation of these detail coefficients is called the \emph{discrete
wavelet transform}. In the context of signals processing, one can imagine calculating these
coefficients, transmitting them, and then reproducing the approximated signal on the
receiving end. Furthermore, the coefficients of the details reflect the local properties
of the original function $f$ at the particular level of detail and resolution! This means
that we can discard many of the coefficients if we are only interested in reproducing a certain
part of the signal, or in recovering the entire signal to only a limited resolution. We can
also study just those frequencies of the signal that fall within a certain range (called a
sub-band) by examining the detail coefficients at a particular level. These
properties make the discrete wavelet transform an attractive alternative to the Fourier
transform in many applications. See Figure \ref{fig:dwt1D} for an example of the discrete Wavelet transform.

\begin{figure}
\centering
\includegraphics[width = 0.5\textwidth]{dwt1D}
\caption{A level 4 wavelet decomposition of a signal. The top panel is the original signal,
the next panel down is the approximation frame, and the remaining panels are the detail coefficients.
Notice how the approximation frame resembles a smoothed version of the original signal, while the
details capture the high-frequency oscillations and noise.}
\label{fig:dwt1D}
\end{figure}

In practice, we are often interested in analyzing discrete signals with compact support (that is,
finite-time signals that we have sampled at a finite number of points). If wavelet analysis is
to be of any use, we first need an efficient way to calculate the discrete wavelet transform.
The process described in the first section, while intuitive and illustrative of the mathematical
principles
behind wavelet analysis, is not the best approach to calculating the wavelet coefficients. It
turns out that the discrete wavelet transform can be implemented as an iterated low-pass/high-pass
filter bank, one iteration of which is shown graphically in the figure. We present the
algorithm without getting into the details of why it works.
\begin{figure}[H]
\centering
\includegraphics[width = 0.5\textwidth]{dwt1}
\caption{The one-dimensional discrete wavelet transform.}
\end{figure}
The input, $A_j$, represents the level-$j$ approximation frame, and we initialize $A_0$ to
simply be the original signal. Lo and Hi are the low-pass and high-pass filters, respectively.
(By \emph{filter} we mean a vector that serves the purpose of extracting or suppressing a
particular feature of the signal. The Lo and Hi filters are obtained from the wavelet at hand;
for the Haar wavelet, Lo $= (\sqrt{2}^{-1}, \sqrt{2}^{-1})$ and Hi $= (-\sqrt{2}^{-1}, \sqrt{2}
^{-1})$.) The box means convolve the input with the filter, and the circle means downsample by
a factor of two, i.e. remove either the even or odd-indexed entries of the input. The outputs,
$A_{j+1}$ and $D_{j+1}$, are the level-$(j+1)$ approximation frame and detail coefficients,
respectively. Note that the length of the input array is twice that of the output arrays. The
detail coefficients $D_{j+1}$ are stored, and $A_{j+1}$ is then fed back into the loop. Continue
this process until the length of the output is less than the length of the filters, and
return all of the stored detail coefficients as well as the final approximation frame.
\begin{problem}
Write a function that calculates the discrete wavelet transform as described above.
The inputs should be three one-dimensional NumPy arrays (the signal, low-pass filter, and
high-pass filter). The output should be a list of one-dimensional NumPy arrays in the
following form: $[A_n, D_n, \ldots, D_1]$. (Note: for the convolution, you may use
the \li{fftconvolve} function from the \li{scipy.signal} package using the default
\li{mode = 'full'} parameter, but note that the output array is one entry too large, and so
you need to omit the first entry. For downsampling, only keep the even-indexed entries.)
\end{problem}
We also need to know how to reconstruct the signal from the detail coefficients and
approximation frame. Fortunately, the algorithm described above is entirely reversible,
albeit with slightly different filters:
$$\text{Lo} = (\sqrt{2}^{-1}, \sqrt{2}^{-1})$$ and
$$\text{Hi} = (\sqrt{2}^{-1}, -\sqrt{2}^{-1}).$$
Given $A_{j+1}$ and $D_{j+1}$, simply upsample both arrays (by inserting a zero after
each entry of the original array), convolve the results
with the Lo and Hi filters, respectively (this time omit the \emph{last} entry of the
result), and add the outputs to obtain $A_j$. Continue the
process until you recover $A_0$, the original signal.
\begin{problem}
Write a function that calculates the inverse wavelet transform as described above.
The inputs should be a list of arrays (of the same form as the output of your discrete
wavelet transform function), the low-pass filter, and the high-pass filter. The output
should be a single array, the recovered signal. In order to check your work, compute
the discrete wavelet transform of a random array of length 64, then compute the inverse
transform, and compare the original signal with the recovered signal. The difference
should be very small.
\end{problem}

\subsection*{The two-dimensional Discrete Wavelet Transform}
Our discussion so far has focused on one-dimensional
discrete signals, but it is not difficult to extend the same ideas into the
realm of two-dimensional arrays. As you know, a digital image can be represented
as a matrix of pixel values (for simplicity we will consider grayscale images of
size $2^n \times 2^n$). We can perform the wavelet decomposition of
an image in much that same way as with one-dimensional signals. We once again
calculate detail and approximation coefficients using an iterative filter
bank, but now we generate four arrays of coefficients at each iteration as opposed to
just two. In essence, we perform the one-dimensional wavelet transform first on each
row, and then on each column of the matrix. Notice the similarity with our approach
to extending the one-dimensional Fourier transform to two-dimensional images. See
Figure \ref{fig:dwt2D} of an example of the two-dimensional Wavelet transform applied
to an image.

\begin{figure}
    \includegraphics[width=0.8\textwidth]{dwt2D.pdf}
    \caption{The level 2 wavelet coefficients of the Lena image. The upper left quadrant
    is the approximation frame, and the other quadrants are the details. Notice how the
    details highlight the parts of the image with high-frequency textures and borders.}
    \label{fig:dwt2D}
\end{figure}

The algorithm goes as follows.
Given an input matrix of size $2^n \times 2^n$, first operate on the rows as you would
in the one-dimensional Wavelet transform (i.e. convolve each row with the filters, then
downsample).
We then have two matrices of size $2^n \times 2^{n-1}$,
since each row has been downsampled by a factor of 2. Then for each of these two
intermediate matrices, operate on each column, yielding a total of four matrices of
size $2^{n-1} \times 2^{n-1}$. Figure \ref{fig:2dwt}
gives a graphical depiction of one iteration of the algorithm.

\begin{figure}[t]
    \includegraphics[width=0.8\textwidth]{2dwt.jpg}
    \caption{The 2-dimensional discrete wavelet transform.}
    \label{fig:2dwt}
\end{figure}

We initialize $LL_0$ to be the
original image matrix, and we terminate once the length of the rows or columns
is less than the length of the filters. We end up with a list of wavelet
coefficients, starting with the final approximation frame $LL_n$ followed by
collections of detail coefficients $(LH_n,HL_n,HH_n)$, $(LH_{n-1},HL_{n-1},HH_{n-1})$,
$\ldots$, $(LH_1,HL_1,HH_1)$. Note that at each iteration we operate first on the
rows (convolve with the filter, then downsample), and the we operate on the columns
of the resulting matrices (\emph{not} the original matrix). The size of the output
matrices have been reduced by a factor of two in both dimensions. As with the
one-dimensional algorithm, to reconstruct the image from the coefficients, we simply
reverse the process by upsampling, convolving, and adding (first the columns, then
the rows). We provide sample code for one iteration of the transform and the inverse.

\begin{lstlisting}
import numpy as np
from scipy.signal import fftconvolve

# given the current approximation frame image, and the filters lo_d and hi_d
# initialize empty arrays
temp = np.zeros([image.shape[0], image.shape[1]/2])
LL = np.zeros([image.shape[0]/2, image.shape[1]/2])
LH = np.zeros([image.shape[0]/2, image.shape[1]/2])
HL = np.zeros([image.shape[0]/2, image.shape[1]/2])
HH = np.zeros([image.shape[0]/2, image.shape[1]/2])

# low-pass filtering along the rows
for i in xrange(image.shape[0]):
	temp[i] = fftconvolve(image[i], lo_d, mode='full')[1::2]

# low and hi-pass filtering along the columns
for i in xrange(image.shape[1]/2):
	LL[:,i] = fftconvolve(temp[:,i],lo_d,mode='full')[1::2]
    LH[:,i] = fftconvolve(temp[:,i],hi_d,mode='full')[1::2]

# hi-pass filtering along the rows
for i in xrange(image.shape[0]):
	temp[i] = fftconvolve(image[i], hi_d, mode='full')[1::2]

# low and hi-pass filtering along the columns
for i in xrange(image.shape[1]/2):
	HL[:,i] = fftconvolve(temp[:,i],lo_d,mode='full')[1::2]
    HH[:,i] = fftconvolve(temp[:,i],hi_d,mode='full')[1::2]
\end{lstlisting}
At this point, the variables \li{LL, LH, HL, HH} contain the current level of wavelet coefficients.
You would then store \li{(LH, HL, HH)} in a list, and feed \li{LL} back into the same
block of code (with \li{LL} replacing \li{image}) to obtain the next level of coefficients.

Now, given a current level of wavelet coefficients, here is the code to recover the previous
approximation frame, which is the crucial step in the inverse transform.
\begin{lstlisting}
# given current coefficients LL, LH, HL, HH
# initialize temporary arrays
n = LL.shape[0]
temp1 = np.zeros([2*n,n])
temp2 = np.zeros([2*n,n])
up1 = np.zeros(2*n)
up2 = np.zeros(2*n)

# upsample and filter the columns of the coefficient arrays
for i in xrange(n):
	up1[1::2] = HH[:,i]
	up2[1::2] = HL[:,i]
	temp1[:,i] = fftconvolve(up1, hi_r)[1:] + fftconvolve(up2, lo_r)[1:]
	up1[1::2] = LH[:,i]
	up2[1::2] = LL[:,i]
	temp2[:,i] = fftconvolve(up1, hi_r)[1:] + fftconvolve(up2, lo_r)[1:]

# upsample and filter the rows, then add results together
result = sp.zeros([2*n,2*n])
for i in xrange(2*n):
	up1[1::2] = temp1[i]
	up2[1::2] = temp2[i]
	result[i] = fftconvolve(up1, hi_r)[1:] + fftconvolve(up2, lo_r)[1:]
\end{lstlisting}

\begin{problem}
Build off of the sample code to fully implement the two-dimensional discrete
wavelet transform as described above.
As before, the input to your function should consist of
three arrays: the input image, the low-pass filter, and the high-pass filter.
You should return a list of the following form: $$[LL_n,(LH_n,HL_n,HH_n), \ldots
,(LH_1,HL_1,HH_1)].$$

The inverse wavelet transform function should take as input a list
of that same form, as well as the reconstruction low-pass and high-pass filters,
and should return the reconstructed image.
\end{problem}

These wavelet coefficients are very useful in a variety of image processing
tasks. They allow us to analyze and manipulate images in terms of both their
frequency and spatial properties, and at differing levels of resolution.
Furthermore, wavelet bases often have the remarkable ability to represent
images in a very \textit{sparse} manner -- that is, most of the image
information is captured by a small subset of the wavelet coefficients.
In the remainder of this lab, we will see how the discrete wavelet transform
plays a role in edge detection, noise removal, and compression.

\subsection*{More Wavelets}
Up to this point, the only wavelet that we have considered is the Haar wavelet,
which is the simplest and historically first example. Wavelet analysis is a broad
field, however, and there are myriad other wavelets that have been studied and
applied. Your implementation of the discrete wavelet transform is quite general,
and you will find that different types of signals or functions call for different
wavelets. We will not go into detail here, but be aware that there is a large
selection of wavelets out there.

\begin{figure}[H]
\minipage{0.49\textwidth}
    \includegraphics[width=\linewidth]{mexicanHat}
    \caption{The Mexican Hat Wavelet}
\endminipage\hfill
\minipage{0.49\textwidth}
    \includegraphics[width=\linewidth]{db5_3}
    \caption{The Cohen-Daubechies-Feauveau 5/3 Wavelet}
\endminipage
\end{figure}

\section*{The PyWavelets Module}
PyWavelets is a Python library for Wavelet Analysis. It provides convenient and
efficient methods to calculate the one and two-dimensional discrete Wavelet
transform, as well as much more. Assuming that the package has been installed on
your machine, type the following to get started:
\begin{lstlisting}
>>> import pywt
\end{lstlisting}
Performing the basic discrete Wavelet transform is very simple.
Below, we compute the one-dimensional transform for a sinusoidal signal.
\begin{lstlisting}
>>> import numpy as np
>>> f = np.sin(np.linspace(0,8*np.pi, 256))
>>> fw = pywt.wavedec(f, 'haar')
\end{lstlisting}
The variable \li{fw} is now a list of arrays, starting with the final approximation
frame, followed by the various levels of detail coefficients, just like the output
of the wavelet transform function that you coded in the previous lab.
Plot the level 2 detail and verify that it resembles a blocky sinusoid.
\begin{lstlisting}
>>> from matplotlib import pyplot as plt
>>> plt.plot(fw[-2], linestyle='steps')
>>> plt.show()
\end{lstlisting}
We can give alter the arguments to the \li{wavedec} function to use different
wavelets or obtain different levels of the wavelet transform. The second
positional argument, as you will notice, is a string that gives the name of the
wavelet to be used. We first used the Haar wavelet, with which you are already
familiar. PyWavelets supports a number of different Wavelets, however, which you can
list by executing the following code:
\begin{lstlisting}
>>> # list the available Wavelet families
>>> print pywt.families()
['haar', 'db', 'sym', 'coif', 'bior', 'rbio', 'dmey']
>>> # list the available wavelets in the coif family
>>> print pywt.wavelist('coif')
['coif1', 'coif2', 'coif3', 'coif4', 'coif5']
\end{lstlisting}
We can also include optional arguments \li{mode} and \li{level} when calling the
\li{wavedec} function. Using these arguments, you can adjust the mode for dealing
with border distortion and the level of the Wavelet decomposition, respectively.

\begin{figure}[t]
    \includegraphics[width=\linewidth]{dwt2.pdf}
    \caption{Level 1 Wavelet decomposition of the Lena image.
    The upper left is the approximation frame, and the remaining
    plots are the detail coefficients.}
    \label{fig:dwt2}
\end{figure}

Now we illustrate how to perform a two-dimensional Wavelet transform using
PyWavelets. We will work with the traditional Lena image, performing a
two level wavelet transform using the Daubechies 4 Wavelet.
\begin{lstlisting}
>>> import scipy.misc
>>> lena = scipy.misc.lena()
>>> lw = pywt.wavedec2(lena, 'db4', level=2)
\end{lstlisting}
The variable \li{lw} is a list of tuples of arrays. The first entry of the list is
simply the level 2 approximation frame. The second entry of the list is a tuple of
the level 2 detail coefficients $LH$, $HL$, and $HH$ (in that order). The remaining
entries of the list are tuples containing the lower level detail coefficients.
Thus, to plot the level 1 $HL$ detail coefficients, we can execute the following code:
\begin{lstlisting}
>>> HL1 = lw[-1][1]
>>> plt.imshow(np.abs(HL1), cmap=plt.cm.Greys_r, interpolation='none')
>>> plt.show()
\end{lstlisting}
The output of this code should be a plot resembling the lower left plot given in Figure
\ref{fig:dwt2}.

We have only introduced a couple of the basic tools available in PyWavelets. There
are of course many more functions and methods that facilitate a more comprehensive
Wavelet analysis. In the remainder of this lab, we will explore three particular
applications of Wavelet analysis in the realm of image processing and compression.

%\section*{Edge Detection}
%It is often useful to identify the edges of objects and figures
%represented in images. The edge information can be used to classify images
%and group them with other similar images (this is part of a field called
%\textit{computer vision}), to segment the image into component parts, to
%sharpen blurry images, to filter out unnecessary details of the image,
%and so forth. Of course, our human eyes are very adept at recognizing edges,
%but enabling a computer to do the same is much more difficult. An edge can
%be thought of as a discontinuity in the image or a region of high contrast
%in either color or brightness. We can therefore leverage the high-frequency
%detail coefficients of the wavelet transform to detect the edges. Execute the
%following code:
%\begin{lstlisting}
%>>> # calculate one level of wavelet coefficients
%>>> coeffs = pywt.wavedec2(lena,'haar', level=1)
%\end{lstlisting}
%
%Note that the approximation coefficients are very close to the original
%image, while the detail coefficients are much more sparse, and roughly
%capture the edges in the image. In particular, the upper right coefficients
%emphasize the vertical edges, the lower left coefficients emphasize the
%horizontal edges, and the lower right coefficients emphasize the diagonal
%edges.
%
%\begin{problem}
%Now zero out the approximation coefficients and use your inverse DWT
%function to recreate the image. Plot its absolute value. This image is
%a fairly good representation of the edges. If we add this to the original
%image, we can increase the contrast at the edges (that is, make the dark
%side darker, and the light side lighter). Do this, and plot the original
%image side-by-side with the sharpened image. What do you notice? There
%are many image-sharpening techniques, and those based on wavelets
%are more sophisticated than what we have done here, but this gives the
%basic idea.
%\end{problem}
%the above section needs work, or maybe should be taken out completely.

\section*{Noise Removal}
Noise in an image can be defined as unwanted visual artifacts that
obscure the true image. Images can acquire noise from a variety of
sources, including the camera, transmission, and image processing
algorithms. Noise can be completely random and incoherent (as in
Figure \ref{fig:incoherent}), or it can be coherent and display
visual patterns (Figure \ref{fig:coherent}). In this section, we will
focus on reducing a particular type of random noise in images, called
\textit{Gaussian white noise}.

\begin{figure}[t]
\minipage{0.49\textwidth}
    \includegraphics[width=\linewidth]{phantom_random.pdf}
    \caption{The Phantom image with incoherent noise}
    \label{fig:incoherent}
\endminipage\hfill
\minipage{0.49\textwidth}
    \includegraphics[width=\linewidth]{phantom_coherent.pdf}
    \caption{The Phantom image with coherent noise}
    \label{fig:coherent}
\endminipage
\end{figure}

An image that is distorted by Gaussian white noise is one in which
every pixel has been perturbed by a small amount, such that the
perturbations are normally distributed. We can easily add such noise
to an image using the \li{np.random.normal} function.

\begin{lstlisting}
>>> noisyLena = lena + np.random.normal(scale=20, size=lena.shape)
>>> plt.imshow(noisyLena, cmap=plt.cm.Greys_r)
>>> plt.show()
\end{lstlisting}

Given an image with Gaussian white noise, how do we go about reducing
the noise level? Our approach will be based on the idea of thresholding.
It turns out that images are often sparse in the wavelet basis,
particularly in the high-frequency details. The Gaussian noise, however,
is very high frequency, and thus its wavelet transform will be
concentrated in high-frequency wavelet coefficients (of magnitude
roughly proportional to the variance of the noise). We can therefore
reduce the noise while preserving the true image by shrinking the
detail coefficients via hard or soft thresholding.

Given a positive threshold value $\tau$, hard thresholding sets
every wavelet coefficient whose magnitude is less than $\tau$ to
zero, while leaving the remaining coefficients untouched. Soft
thresholding also zeros out all coefficients of magnitude less than
$\tau$, but in addition maps every other coefficient $\beta$ to
$\beta - \tau$ if $\beta > 0$ or $\beta + \tau$ if $\beta < 0$.

Implementing these simple thresholding algorithms in Python is
straight-forward, but PyWavelets already provides this functionality.
The following code gives an example.

\begin{lstlisting}
>>> A = np.arange(-4,5).reshape(3,3)
>>> A
array([[-4, -3, -2],
       [-1,  0,  1],
       [ 2,  3,  4]])
>>> pywt.thresholding.hard(A,1.5)
array([[-4, -3, -2],
       [ 0,  0,  0],
       [ 2,  3,  4]])
>>> pywt.thresholding.soft(A,1.5)
array([[-2.5, -1.5, -0.5],
       [ 0. ,  0. ,  0. ],
       [ 0.5,  1.5,  2.5]])
\end{lstlisting}

Once the coefficients have been thresholded, we take the inverse
wavelet transform to recover the denoised image. This can be done
by calling the \li{waverec2} function, providing the list of Wavelet
coefficients as well as the name of the desired Wavelet as arguments.
The threshold value is generally a function of the variance of the noise,
and in real situations, we do not know what this variance is. In fact,
noise variance estimation in images is a research area in its own
right, but this goes beyond the scope of this lab, and so we will
assume that we already have a decent estimate of the variance.

\begin{figure}[t]
    \includegraphics[width=\linewidth]{denoise.pdf}
    \caption{Noisy Lena (left), denoised using hard thresholding (center),
    and denoised using soft thresholding (right).}
    \label{fig:denoise}
\end{figure}

\begin{problem}
Write functions that implement the hard and soft thresholding
techniques. The inputs should be a list of wavelet coefficients
in the usual form, as well as the threshold value. The output
should be the thresholded wavelet coefficients (also in
the usual form). Remember that we only want to threshold the
detail coefficients, and not the approximation coefficients.
You should therefore leave the first entry of the input
coefficient list unchanged.
\end{problem}

\begin{problem}
Create a noisy version of the Lena image by adding Gaussian
white noise of mean 0 and standard deviation $\sigma = 20$ (i.e. \li{scale=20}).
Compute four levels of the wavelet coefficients using the Daubechies 4 Wavelet,
and input these into your
thresholding functions (with $\tau = 3\sigma$ for the hard threshold,
and $\tau = 3\sigma/2$ for the soft threshold). Reconstruct the
two denoised images, and then plot these together alongside the
noisy image. Your output should match Figure \ref{fig:denoise}.

What do you notice? How does lowering or raising the
threshold affect the reconstructed images? What happens if you use
a different Wavelet?
\end{problem}

\section*{Image Compression}
We now turn to the problem of image compression. Explicitly saving
the value of every pixel in an image can be very costly in both
storage and transmission, and numerous image compression techniques
have been developed over the years to deal with this problem.
Transform methods have long played an important role in these
techniques; the popular JPEG image compression standard is based on
the discrete cosine transform. Starting from the early 1990's, much
research has gone into compression methods using the discrete wavelet
transform, and to great success. The JPEG2000 compression standard
and the FBI Fingerprint Image database, along with other systems,
take the wavelet approach.

The general framework for compression is fairly straightforward. First,
the image to be compressed undergoes some form of preprocessing (this
can include subtracting out its mean, tiling the image, or perhaps
nothing at all). Next, the wavelet coefficients are computed using some
specially constructed wavelet (JPEG2000 uses the either the
Cohen-Daubechies-Feauveau 9/7 or 5/3 wavelet) and then \textit{quantized},
 a process that we will explain shortly. The quantized coefficients are
 then grouped in a particular way and passed through an entropy encoder
 (such as Huffman coding, run length coding, or arithmetic coding). This
 coding step comes from the realm of information theory, and we will not
 worry about it in this lab. What you have left is a compact stream of bits
 that can then be saved or transmitted much more efficiently than the
 original image. All of the above steps are invertible, allowing us to
 reconstruct the image from the bitstream.

 The step in this process that we will focus on is quantization. Put simply,
 quantization is a process whereby the coefficients are converted into
 integers. If the coefficients are floating-point numbers, then this
 process introduces some loss of precision, and we call this \textit{
 lossy compression}. In the situation where all the coefficients are already
 integers, it is possible to compress the image without any loss of precision,
 and this is called \textit{lossless compression}. Quantization can be
 performed in a variety of ways, and we will explore one particular method
 called a \emph{uniform null-zone quantizer}. Given a coefficient $x$, we assign
 to it an integer $q$ given by
\begin{equation*}
q =
 \begin{cases}
   \lceil x / \delta - t/2 \rceil, &  x \geq 0\\
   \lfloor x / \delta + t/2 \rfloor & x \leq 0
 \end{cases}
\end{equation*}
where $1 \leq t \leq 2$ and $\delta > 0$ are adjustable parameters.
The inverse process, called de-quantization, consists of recovering
the coefficient $y$ from the quantized value $q$ via the equation
\begin{equation*}
 y =
  \begin{cases}
   (q - 1/2 + t/2)\delta & q > 0\\
   (q + 1/2 - t/2)\delta & q < 0\\
   0,                    & q = 0
  \end{cases}
 \end{equation*}
What we are essentially doing is mapping all wavelet coefficients that
fall in the interval $[-\delta,\delta]$ to 0 and all wavelet coefficients
in the interval $[j\delta,(j+1)\delta]$ to $(2j+1)\delta/2$ for integers
$j \geq 1$ and $j \leq -2$. This greatly reduces the number of distinct
coefficient values (indeed, most of the coefficients are mapped to
zero), allowing us, at the cost of some precision, to store less
information. The larger we choose $\delta$, the more compression we
can achieve, albeit with a correspondingly larger loss in precision.
\begin{problem}
Write function \li{quantize} and \li{dequantize} based on the discussion above.
In both cases, the inputs should be a list of wavelet coefficients in
standard form, the $\delta$ parameter, and the $t$ parameter with default
value of 2. The functions should return the quantized list of wavelet
coefficients.

For the Lena image, calculate the complete set of wavelet coefficients and then
quantize and de-quantize the coefficients, and reconstruct the image.
Do this for a few different values of $\delta$ (with $t=2$), and observe how
the image is distorted. Try using different Wavelets. Keep in mind that there are
specially designed Wavelets used in image compression that are not available
in PyWavelets, so distortion will be greater than tolerable in real-life settings.
\end{problem}


\section*{Limits}

\vspace{-10pt}

\begin{python}
   from sympy import *
   a, n, x, dx = symbols('a n x dx')
   ans = limit(sin(4*x)/x,x,0)                  # py (ans.301,ans)
   ans = limit(2**x/x,x,oo)                     # py (ans.302,ans)
   ans = limit(((x+dx)**2 - x**2)/dx, dx,0)     # py (ans.303,ans)
   ans = limit((4*n + 1)/(3*n - 1),n,oo)        # py (ans.304,ans)
   ans = limit((1+(a/n))**n,n,oo)               # py (ans.305,ans)
\end{python}

\begin{align*}
   &\py*{ans.301}\\
   &\py*{ans.302}\\
   &\py*{ans.303}\\
   &\py*{ans.304}\\
   &\py*{ans.305}
\end{align*}


\section{Fourier Transforms}
\subsection{Introduction}
\begin{definition}
    The Fourier transform (FT) of a function $f(x)$ is
    $$\tilde{f}(k)=(\mathcal F(f))(k)=\int_{-\infty}^\infty f(x)e^{-ikx}\,\mathrm dx$$
    The inverse Fourier transform is
    $$f(x)=(\mathcal F^{-1}(\tilde{f}))(x)=\frac{1}{2\pi}\int_{-\infty}^\infty\tilde{f}(k)e^{ikx}\,\mathrm dk$$
\end{definition}
Beware that there are several conventioned for FT.
$\tilde{f}$ is said to be in the frequency domain.
\begin{theorem}[Fourier Inversion Theorem]
    $\mathcal F^{-1}\circ\mathcal F(f)=f$ given that all integrals involved are well-behaved (a sufficient condition is $f,\tilde{f}$ both being absolutely integrable).
\end{theorem}
\begin{example}
    Take $f(x)=(1/(\sigma\sqrt\pi))e^{-x^2/\sigma^2}$, then we can either find $\tilde{f}$ by completing square or observe that
    $$\tilde{f}(k)=\frac{1}{\sigma\sqrt{\pi}}\int_{-\infty}^\infty e^{-x^2/\sigma^2}e^{-ikx}\,\mathrm dx=\frac{1}{\sigma\sqrt{\pi}}\int_{-\infty}^\infty e^{-x^2/\sigma^2}\cos(kx)\,\mathrm dx$$
    Differentiate under the integral sign,
    \begin{align*}
        \frac{\mathrm d\tilde{f}}{\mathrm dk}&=-\frac{1}{\sigma\sqrt\pi}\int_{-\infty}^\infty xe^{-x^2/\sigma^2}\sin(kx)\,\mathrm dx\\
        &=-\frac{1}{\sigma\sqrt\pi}\int_{-\infty}^\infty\left( \frac{k\sigma^2}{2} \right)e^{-x^2/\sigma^2}\cos(kx)\,\mathrm dx\\
        &=-\frac{k\sigma^2}{2}\tilde{f}(k)
    \end{align*}
    Integrating this differential equation on $\tilde{f}$ gives $\tilde{f}(k)=Ce^{-k^2\sigma^2/4}$ for some constant $C$.
    Setting $k=0$ gives $C=1$, therefore $\tilde{f}(k)=e^{-k^2\sigma^2/4}$.
    One can show that $\mathcal F^{-1}\tilde{f}=f$.
\end{example}
\begin{example}
    The Fourier transform of $f(x)=e^{-a|x|}$ is $\tilde{f}(k)=2a/(a^2+k^2)$ either by direct integration or superposition.
\end{example}
\subsection{Relation with Fourier Series}
Recall that the Fourier series has the form
$$f(x)=\sum_{n=-\infty}^\infty c_ne^{ik_nx},k_n=n\Delta k,\Delta k=\frac{\pi}{L}$$
Now we know that
$$c_n=\frac{1}{2L}\int_{-L}^Lf(x)e^{-ik_nx}\,\mathrm dx=\frac{\Delta k}{2\pi}\int_{-L}^Lf(x)e^{-ik_nx}\,\mathrm dx$$
Imagine taking $L\to\infty,\Delta k\to 0$ then we get the expression of Fourier transform multiplied by an infinitesimal term.
To further justify the analogy, observe that we get
$$f(x)=\sum_{n=-\infty}^\infty\frac{\Delta k}{2\pi}e^{ik_nx}\int_{-L}^Lf(x')e^{-ik_nx'}\,\mathrm dx'$$
This looks like a Riemann sum, so let us take the limit $L\to\infty,\Delta k\to 0$ which gives
$$f(x)=\frac{1}{2\pi}\int_{-\infty}^\infty \left( \int_{-\infty}^\infty f(x')e^{-ikx'}\,\mathrm dx' \right)e^{ikx}\,\mathrm dk=\mathcal F^{-1}\circ\mathcal F(f)(x)$$
Note that when $f$ is discontinuous at $x$, then $\mathcal F^{-1}\circ\mathcal F(f)(x)=(f(x_-)+f(x_+))/2$ which is a similar behaviour to that of a Fourier series.
\subsection{Properties of Fourier Transform}
The operators $\mathcal F,\mathcal F^{-1}$ is both linear.
Also, the translation $h(x)=f(x-\lambda)$ gives $\tilde{h}(k)=e^{-i\lambda k}\tilde{f}(k)$.
Correspondingly, the frequency shift $h(x)=e^{i\lambda x}f(x)$ gives $\tilde{h}(k)=\tilde{f}(k-\lambda)$.
The scaling $h(x)=f(\lambda x)$ for $\lambda\neq 0$ gives $\tilde{h}(k)=\tilde{f}(k/\lambda)/|\lambda|$.\\
These are all trivial facts, here is a slightly more interesting one:
$h(x)=xf(x)$ gives $\tilde{h}(k)=i\tilde{f}^\prime(k)$.
Indeed,
$$\tilde{h}(k)=\int_{-\infty}^\infty xf(x)e^{-ikx}\,\mathrm dx=-\frac{1}{i}\frac{\mathrm d}{\mathrm dk}\int_{-\infty}^\infty f(x)e^{-ikx}\,\mathrm dx=i\tilde{f}^\prime(k)$$
What's more important is that if $f$ vanishes at $\pm\infty$ and $h(x)=f^\prime(x)$ gives $\tilde{h}(k)=ik\tilde{f}(k)$ via integration by parts
$$\tilde{h}(k)=\int_{-\infty}^\infty f^\prime(x)e^{-ikx}\,\mathrm dx=[f(x)e^{-ikx}]_{-\infty}^\infty-\int_{-\infty}^\infty (-ik)f(x)e^{-ikx}\,\mathrm dx=ik\tilde{f}(k)$$
So we can employ Fourier transform to turn a (nice enough) differential equation into an algebraic one.\\
We also have a sense of duality between $x$ and $k$ here.
Observe that we have
$$f(-x)=\frac{1}{2\pi}\int_{-\infty}^\infty\tilde{f}(k)e^{-ikx}\,\mathrm dk,f(-k)=\frac{1}{2\pi}\int_{-\infty}^\infty\tilde{f}(x)e^{-ikx}\,\mathrm dx$$
Therefore $g(x)=\tilde{f}(x)$ iff $\tilde{g}(k)=2\pi f(-k)$..
So $f(-x)=(2\pi)^{-1}\mathcal F^2(f)(x)$.
Iterating this gives $\mathcal F^4(f)(x)=4\pi^2f(x)$.
\begin{example}
    Consider the ``top hat'' function
    $$f(x)=\begin{cases}
        1\text{, if $|x|\le a$}\\
        0\text{, if $|x|>a$}
    \end{cases}$$
    We get
    $$\tilde{f}(k)=\int_{-\infty}^\infty f(x)e^{-ikx}\,\mathrm dx=\int_{-a}^a\cos(kx)\,\mathrm dx=\frac{2\sin(ka)}{k}$$
    Fourier inversion theorem then gives
    $$\frac{1}{\pi}\int_{-\infty}^\infty e^{ikx}\frac{\sin ka}{k}\,\mathrm dk=\begin{cases}
        1\text{, if $|x|<a$}\\
        0\text{, if $|x|>a$}
    \end{cases}$$
    Set $x=0$ and $k\to x$ gives
    $$\int_0^\infty\frac{\sin(ax)}{x}\,\mathrm dx=\frac{\pi}{2}\operatorname{sgn}(a)=\begin{cases}
        \pi/2\text{, if $a>0$}\\
        0\text{, if $a=0$}\\
        -\pi/2\text{, if $a<0$}
    \end{cases}$$
    which is pretty awesome.
\end{example}
\subsection{Convolution and Parseval's Theorem}
Recall that
\begin{definition}
    The convolution of $f$ and $g$ is
    $$(f\ast g)(x)=\int_{-\infty}^\infty f(y)g(x-y)\,\mathrm dy$$
\end{definition}
We want to multiply together functions in frequency domain, that is $\tilde{h}=\tilde{f}\tilde{g}$, and find its inverse Fourier transform.
\begin{align*}
    h(x)&=\frac{1}{2\pi}\int_{-\infty}^\infty\tilde{f}(k)\tilde{g}(k)e^{ikx}\,\mathrm dk\\
    &=\frac{1}{2\pi}\int_{-\infty}^\infty\left( \int_{-\infty}^\infty f(y)e^{-iky}\,\mathrm dy \right)\tilde{g}(k)e^{ikx}\,\mathrm dk\\
    &=\int_{-\infty}^\infty f(y)\left( \frac{1}{2\pi}\int_{-\infty}^\infty\tilde{g}(k)e^{ik(x-y)}\,\mathrm dk \right)\,\mathrm dy\\
    &=\int_{-\infty}^\infty f(y)g(x-y)\,\mathrm dy\\
    &=(f\ast g)(x)
\end{align*}
By duality, we also have
$$h(x)=f(x)g(x)\implies \tilde{h}(k)=\frac{1}{2\pi}\int_{-\infty}^\infty \tilde{f}(p)\tilde{g}(k-p)\,\mathrm dp=\frac{1}{2\pi}(\tilde{f}\ast\tilde{g})(k)$$
Now consider $h(x)=g^*(-x)$, then
\begin{align*}
    \tilde{h}(k)&=\int_{-\infty}^\infty g^\ast(-x)e^{-ikx}=\left( \int_{-\infty}^\infty g(-x)e^{ikx}\,\mathrm dx \right)^*\\
    &=\left( \int_{-\infty}^\infty g(y)e^{-iky}\,\mathrm dy \right)^*=\tilde{g}^*(k)
\end{align*}
By our study of convolution we have
$$\int_{-\infty}^\infty f(y)g^*(y-x)\,\mathrm dy=\frac{1}{2\pi}\int_{-\infty}^\infty\tilde{f}(k)\tilde{g}^*(k)e^{ikx}\,\mathrm dx$$
Set $x\to 0$ gives
$$\int_{-\infty}^\infty f(y)g^*(y)\,\mathrm dy=\frac{1}{2\pi}\int_{-\infty}^\infty\tilde{f}(k)\tilde{g}^*(k)\,\mathrm dk\implies\langle g,f\rangle=\frac{1}{2\pi}\langle \tilde{g},\tilde{f}\rangle$$
Setting $g=f$ then gives
\begin{theorem}[Parseval's Theorem]
    $$\int_{-\infty}^\infty|f(x)|^2\,\mathrm dx=\frac{1}{2\pi}\int_{-\infty}^\infty |\tilde{f}(k)|^2\,\mathrm dk$$
\end{theorem}
\subsection{Fourier Transform of Generalised Functions}
We want to apply $\mathcal F$ to generalised functions.
If one wants to be precise, one can view them as the limit of the sequence of the Fourier transforms of well-defined well-behaved functions that approaches the generalised function in question.
The details can be justified by Parseval's theorem, but the treatment is beyond the scope of this course.\\
Of course the main culprit of generalised functions is $\delta$.
Surprisingly, $\delta$ is found naturally in Fourier transform.
For a well-behaved $f$,
\begin{align*}
    f(x)&=\mathcal F^{-1}(\mathcal F(f))(x)=\frac{1}{2\pi}\int_{-\infty}^\infty\int_{-\infty}^\infty f(u)e^{-iku}e^{ikx}\,\mathrm du\mathrm dk\\
    &=\int_{-\infty}^\infty f(u)\left( \frac{1}{2\pi}\int_{-\infty}^\infty e^{ik(x-u)}\,\mathrm dk \right)\,\mathrm du
\end{align*}
So we might identify
\footnote{The number of things that would go wrong with this is overwhelming, but what the hell.}
$$\delta(x-u)=\frac{1}{2\pi}\int_{-\infty}^\infty e^{ik(x-u)}\,\mathrm dk$$
A direct calculation, on the other hand yields
$$\tilde\delta(k)=\int_{-\infty}^\infty \delta(x)e^{ikx}\,\mathrm dx=1$$
So dually, the Fourier transform of the constant $f(x)=1$ is $\tilde{f}(k)=2\pi\delta(x)$.
Also $f(x)=\delta(x-a)$ has $\tilde{f}(k)=e^{ika}$.
If we have exponentials then we naturally have trigonometrics, so here goes:
$f(x)=\cos(\omega x)$ has $\tilde{f}(k)=\pi(\delta(k+\omega)+\delta(k-\omega))$ and $f(x)=\sin(\omega x)$ has $\tilde{f}(k)=i\pi (\delta(k+\omega)-\delta(k-\omega))$.\\
Let's move on to something slightly better.
Consider the Heaviside function
$$H(x)=\begin{cases}
    1\text{, if $x>0$}\\
    0\text{, if $x<0$}\\
    1/2\text{, if $x=0$}
\end{cases}$$
Then $H(x)+H(-x)=1$ for any $x$.
Therefore $\tilde{H}(k)+\tilde{H}(-k)=2\pi\delta(k)$.
But $H^\prime(x)=\delta(x)$, hence $ik\tilde{H}(k)=1$.
So for these results to be consistent (note that $k\delta(k)=1$), we had to have $\tilde{H}(k)=\pi\delta(k)+(ik)^{-1}$.
The formula can look nicer if we shift a bit and consider instead $f(x)=\operatorname{sgn}(x)/2$ which has $\tilde{f}(k)=(ik)^{-1}$.
\subsection{Applications of Fourier Transforms}
The first very important application of Fourier transforms is in boundary value problems of ODEs.
Consider the problem $y^{\prime\prime}-y=f(x)$ with homogeneous boundary conditions $y\to 0$ as $x\to\pm\infty$.
Apply Fourier transform on both sides gives
$$(-k^2-1)\tilde{y}=\tilde{f}\implies \tilde{y}(k)=\tilde{f}(k)\tilde{g}(k),\tilde{g}(k)=-\frac{1}{1+k^2}$$
Therefore
\begin{align*}
    y(x)&=\int_{-\infty}^\infty f(u)g(x-u)\,\mathrm du\\
    &=-\frac{1}{2}\int_{-\infty}^\infty f(u)e^{-|x-u|}\,\mathrm du\\
    &=-\frac{1}{2}\int_{-\infty}^xf(u)e^{u-x}\,\mathrm du-\frac{1}{2}\int_x^\infty f(u)e^{x-u}\,\mathrm du
\end{align*}
which is in the form of the solution we would have obtained if we use Green's function instead.
Of course, we can solve this by inverse Fourier transform as well, which is quite easy once we introduce Fast Fourier Transform (FFT).
We will go through it later.\\
Another motivation of Fourier transform in signal processing.
Suppose we are given some sort of input signal $J(t)$ which is acted on by some linear differential operator $\mathcal L$ to yield output $O(t)$.
The Fourier transform
$$\tilde{J}(\omega)=\int_{-\infty}^\infty J(t)e^{i\omega t}\,\mathrm dt$$
is called the resolution.
In the frequency domain, the action of $\mathcal L$ in $J(t)$ is just multiplying $\tilde{J}(\omega)$ by a transfer function $\tilde{R}\omega$ to yield the output
$$O(t)=\frac{1}{2\pi}\int_{-\infty}^\infty\tilde{R}(\omega)\tilde{J}(\omega)e^{i\omega t}\,\mathrm d\omega$$
The inverse Fourier transform $R$ of $\tilde{R}$ is called the response function.
So $O$ is simply $J\ast R$.
For example, if there is no input $J(t)=0$ for $t<0$.
By causality, we expect $R(t)=0$ for $t<0$.
Therefore
$$O(t)=\int_0^tJ(u)R(t-u)\,\mathrm du$$
which is in the same form as the Green's function in an initial value problem.\\
We want to explore the general transfer functions for a class of ODEs.
Suppose the input/output relation is
$$\mathcal LO=\left( \sum_{i=0}^na_i\frac{\mathrm d^i}{\mathrm dx^i} \right)O=J$$
where $a_i$ are constants.
Taking Fourier transform gives
$$(a_0+a_1(i\omega)+\ldots,a_n(i\omega)^n)\tilde{O}(\omega)=\tilde{J}(\omega)$$
Therefore $\tilde{R}(\omega)=(a_0+a_1(i\omega)+\ldots,a_n(i\omega)^n)^{-1}$.
Factorise the polynomial to turn it into the form $\tilde{R}(\omega)=((i\omega-c_1)^{k_1}\cdots (i\omega-c_r)^{k_r})^{-1}$ where $i\neq j\implies c_i\neq c_j$.
Partial fraction gives
$$\tilde{R}(\omega)=\frac{1}{(i\omega-c_1)^{k_1}\cdots (i\omega-c_r)^{k_r}}=\sum_{j=1}^r\sum_{m=1}^{k_j}\frac{\Gamma_{jm}}{(i\omega-c_j)^m}$$
for constants $\Gamma_{jm}$.
But we know that
$$\mathcal F^{-1}\left( \frac{1}{(i\omega-a)^m} \right)=\begin{cases}
    t^{m-1}e^{at}/(m-1)!\text{, if $t>0$}\\
    0\text{, if $t<0$}
\end{cases}$$
Getting back to time domain, we obtain the response function
$$R(t)=\sum_{j=1}^r\sum_{m=1}^{k_j}\Gamma_{jm}\frac{t^{m-1}}{(m-1)!}e^{c_jt},t>0$$
\begin{example}
    Consider the damp oscillator $\mathcal Ly=y^{\prime\prime}+2py^\prime+(p^2+q^2)y=f(t)$ with damping $p>0$ with homogeneous initial conditions $y(0)=y^\prime(0)=0$.
    The Fourier transform of this is $(i\omega)^2\tilde{y}+2ip\omega\tilde{y}+(p^2+q^2)\tilde{y}=\tilde{f}$, so
    $$\tilde{y}=\frac{\tilde{f}}{-\omega^2+2ip\omega+p^2+q^2}=\tilde{R}\tilde{f},\tilde{R}=\frac{1}{-\omega^2+2ip\omega+p^2+q^2}$$
    So
    $$y(t)=\int_0^tR(t-\tau)f(\tau)\,\mathrm d\tau,R(t-\tau)=\frac{1}{2\pi}\int_{-\infty}^\infty\frac{\exp(i\omega(t-\tau))\,\mathrm d\omega}{p^2+q^2+2ip\omega-\omega^2}$$
    which, as one can verify, is analogous to the Green's function methods since $\mathcal LR(t-\tau)=\delta(t-\tau)$.
\end{example}
\subsection{Discrete Fourier Transform}
Suppose we sample a signal $h(t)$ at equal times $t_n=n\Delta$ with time-sampling $\Delta$ and values $h_n=h(n\Delta)$ with $n\in\mathbb Z$.
That is, the sampling frequency is $f_s=1/\Delta$ (and angular frequency $\omega_s=2\pi f_s=2\pi/\Delta$).
The Nyquist frequency $f_c=1/(2\Delta)$ is the highest frequency actually sampled at $\Delta$.
Suppose we have a (sinusoidal) signal with a given frequency $f$
$$g_f(t)=A\cos(2\pi ft+\phi)=\frac{A}{2}(e^{i\phi}e^{2\pi ift}+e^{-i\phi}e^{-2\pi ift})$$
What happens if we sample at $f=f_c$?
We have
$$g_{f_c}(t_n)=A\cos\left( 2\pi\frac{1}{2\Delta}n\Delta+\phi \right)=(A\cos\phi)\cos(\pi n)=A'\cos(2\pi f_ct_n)$$
So information about phase and amplitude are lost.
Even worse if we sample above $f>f_c$:
If we sample at $f=f_c+\delta f$ for some small $\delta f>0$, then
$$g_f(t_n)=A\cos(2\pi(f_c+\delta f)t_n+\phi)=A\cos(2\pi(f_c-\delta f)t_n-\phi)$$
So the effect is just aliasing a ``ghost signal'' to frequency $f_c-\delta f$ (or $-(f_c-\delta f)$), which is a contamination of the information.
\begin{definition}
    A signal $g(t)$ is bandwidth limited if it contains no frequency above some $\omega_{\max{}}=2\pi f_{\max{}}$, that is $\tilde{g}(\omega)=0$ for any $|\omega|>\omega_{\max{}}$.
\end{definition}
So for a bandwidth limited signal $g(t)$ would have
$$g(t)=\frac{1}{2\pi}\int_{-\infty}^\infty\tilde{g}(\omega)e^{i\omega t}\,\mathrm d\omega=\frac{1}{2\pi}\int_{-\omega_{\max{}}}^{\omega_{\max{}}}\tilde{g}(\omega)e^{i\omega t}\,\mathrm d\omega$$
\begin{theorem}[Sampling Theorem]
    Let $g$ be a bandwidth limited signal and $\Delta=1/(2f_{\max{}})$, then define
    $$g_n=g(t_n)=\frac{1}{2\pi}\int_{-\omega_{\max{}}}^{\omega_{\max{}}}\tilde{g}(\omega)e^{i\pi n\omega/\omega_{\max{}}}\,\mathrm d\omega$$
    which induces a Fourier series
    $$\tilde{g}_{\mathrm{per}}(\omega)=\frac{\pi}{\omega_{\max{}}}\sum_{n=-\infty}^\infty g_ne^{-i\pi n\omega/\omega_{\max{}}}$$
    Then, we have
    $$\tilde{g}(\omega)=\tilde{g}_{\mathrm{per}}(\omega)\tilde{h}(\omega),\tilde{h}(\omega)=\begin{cases}
        1\text{, if $|\omega|\le\omega_{\max{}}$}\\
        0\text{, otherwise}
    \end{cases}$$
    Inverting which gives
    \begin{align*}
        g(t)&=\frac{1}{2\omega_{\max{}}}\sum_{n=-\infty}^\infty g_n\int_{-\omega_{\max{}}}^{\omega_{\max{}}}\exp\left( i\omega\left( t-\frac{n\pi}{\omega_{\max{}}} \right) \right)\,\mathrm d\omega\\
        &=\sum_{n=-\infty}^\infty g_n\frac{\sin(\omega_{\max{}} t-\pi n)}{\omega_{\max{}} t-\pi n}
    \end{align*}
    So $g(t)$ can be exactly represented after sampling at discrete times $t_n$.
\end{theorem}
\begin{proof}
    Self-explanatory.
\end{proof}
Suppose we have a finite number $N$ of samples $h_m=h(t_m)$ where $t_m=m\Delta$ for $m=0,\ldots,N-1$.
We want to approximate the Fourier Transform for $N$ frequencies within $f_c=1/(2\Delta)$ with equally spaced frequencies with space $\Delta_f=1/(N\Delta)$ in the range $[-f_c,f_c]$.
So basically we are just looking for $f_n=n\Delta_f=n/(N\Delta)$ where $n=-N/2,\ldots,0,\ldots,N/2$.
Note that $f_c$ and $-f_c$ are aliased together, so the $-N/2$ and $N/2$ are basically the same.
Also $(m+N/2)\Delta_f=f_c+\delta f$ is aliased to $(-m+N/2)\Delta_f=-(f_c-\delta f)$, so we choose instead $f_n=n/(N\Delta)$ with $n=0,\ldots,N-1$.
\begin{definition}
    Observe that
    \begin{align*}
        \tilde{h}(f_n)&=\int_{-\infty}^\infty h(t)e^{-2\pi if_nt}\,\mathrm dt\approx\Delta\sum_{n=0}^{N-1}h_me^{-2\pi i f_nt_m}\\
        &=\Delta\sum_{m=0}^{N-1}h_me^{-2\pi imn/N}=\Delta\tilde{h}_d(f_n)
    \end{align*}
    Here $\tilde{h}_d(f_n)=\tilde{h}_n$ is the discrete Fourier transform (DFT).
\end{definition}
So the matrix $[\operatorname{DFT}]_{mn}=e^{-2\pi imn/N}$ defines the discrete Fourier transform for $h=\{h_m\}$ as we have $\tilde{h}_d=[\operatorname{DFT}]h$.
The inverse of this matrix is its adjoint, i.e. $[\operatorname{DFT}]^{-1}=N^{-1}[\operatorname{DFT}]^\dagger$ and it is built from the $N^{th}$ roots of unity.
\begin{example}
    If $N=4$ and $\omega=-i$, then
    $$[\operatorname{DFT}]=\begin{pmatrix}
        1&1&1&1\\
        1&-i&-1&i\\
        1&-1&1&-1\\
        1&i&-1&-i
    \end{pmatrix}$$
\end{example}
The inverse DFT is
\begin{align*}
    h_m&=h(t_m)\approx\frac{1}{2\pi}\int_{-\infty}^\infty\tilde{h}(\omega)e^{i\omega t_m}\,\mathrm d\omega=\int_{-\infty}^\infty\tilde{h}(f)e^{2\pi ift_m}\,\mathrm df\\
    &\approx\frac{1}{N\Delta}\sum_{n=0}^{N-1}\Delta\tilde{h}_d(f_n)e^{2\pi imn/N}\\
    &=\frac{1}{N}\sum_{n=0}^{N-1}\tilde{h}_ne^{2\pi imn/N}
\end{align*}
In this frame, we can establish analogues of Parseval's theorem
$$\sum_{m=0}^{N-1}|h_m|^2=\frac{1}{N}\sum_{n=0}^{N-1}|\tilde{h}_n|^2$$
and convolution theorem
$$c_k=\sum_{m=0}^{N-1}g_mh_{k-m}\iff \tilde{c}_k=\tilde{g}_k\tilde{h}_k$$
This looks complicated, but there is actually a very efficient algorithm to do it, known as fast Fourier transform (FFT), by exploiting the symmetries of the expression.


%!TEX root =  ../main.tex

\subsection{Reference Triangle Within}
Consider any reference triangle with legs $x$ and $y$ and $r$, and angles $\frac{\tau}{4}$
and $\theta$.  The other angle must be $\frac{\tau}{4} - \theta$.  What are $\sin\theta$ and
$\cos\theta$?  What are $\sin(\frac{\tau}{4}-\theta)$ and $\cos(\frac{\tau}{4}-\theta)$?

insert diagram

Now, at last, we are prepared to answer why half of the six trigonometric functions have the
prefix ``co-''.  It stands for complement!

six co-definitions

We have repeatedly seen that sine is the signed height of the reference triangle within the unit
circle, while cosine is the signed width.  Because these lengths are within the unit circle (a
circle of radius 1), they must be in a Pythagorean relationship.  This reference triangle within
the unit circle provides an excellent visual to accompany the identity, $\sin^2\theta+\cos^2\theta
= 1$.  But through simple algebra manipulation -- dividing by $\sin^2\theta$ or $\cos^2\theta$ ---
it is easy to turn this identity into $1 + \tan^2\theta = \sec^2\theta$, or the more obscure
$\csc^2\theta + 1 = \cot^2\theta$.  What do they represent?

\subsection{Reference Triangles Without}
We have previously defined tangent as sine over cosine, which means we can set up the proportion:

\begin{equation}
\frac{\sin\theta}{\cos\theta} = \frac{\tan\theta}{1}
\end{equation}

This is not only algebra, but can be represented with similar triangles.  ``Sine is to cosine as tangent is to
1'', is the height of one triangle compared to its width, keeping a common angle and still being right in
another triangle whose height is tangent and width is 1.  This is a right triangle extending outside
the unit circle, but whole bottom edge is a radius thereof.

insert diagram

Similarly, there are two more proportions we must examine:
\begin{equation}
\frac{\cos\theta}{\sin\theta} = \frac{\cot\theta}{1}
\end{equation}
This is a way of saying the width vs. height of one triangle is cosine vs. sine, and that a similar right
triangle (with angle in common) has width cotangent and height 1.

insert diagram

Exercise: show that this get the same length as 1 over tangent equals cotangent over 1.

There is also
\begin{equation}
\frac{1}{\cos\theta} = \frac{\sec\theta}{1}
\end{equation}

Geometrically this means, if we draw a triangle with hypotenuse 1 and adjacent of cosine, then there must exist a unique, similar triangle with hypotenuse secant and adjacent 1.

\subsection{Relative}
It should be apparent now that trigonometry is very different from algebra.  Every expression can be
written in an infinite number of ways.  Every term is equal to 1 times itself (the Identity of Multiplication),
but 1 equals $\sin^2\theta + \cos^2\theta$ or $\sec^2\theta-\tan^2\theta$.  We practice turning
trigonometry expressions into specific forms so that we will be fluent in simplify complicated
expressions in the real world.

The co-function identities (above) show us what to do with $90^\circ/\frac{\tau}{4}$s inside of
trig functions, but what about other quadrants and negatives?  There are always lots of options
for how you might think about such things, so we will consider two ways to think about
simplifying $\sin(-x)$ and $\cos(-x)$.

Graphically, you can see that $y=\sin(x)$ is an odd function.  On the unit circle, if we begin at
standard position, we see that turning a positive angle yields a certain height above the $x$-axis,
while turning that same angle \emph{down} yields the opposite height.  Hence, $\sin(-x)=-\sin(x)$.
$y=\cos(-x)$ is an even function, begin symmetric about the $y$-axis.  On the unit circle, the
horizontal displacement caused by any angle is the same as its negative angle.  Hence,
$\cos(-x)=\cos(x)$.


\documentclass[11pt]{article}

\usepackage{amsfonts}
\usepackage{fancyvrb}
\usepackage{graphicx}
\usepackage{caption}
\usepackage{subcaption}
\usepackage{url}

\setlength{\oddsidemargin}{0in}
\setlength{\evensidemargin}{0in}
\setlength{\textwidth}{6.5in}
\setlength{\topmargin}{0in}
\setlength{\headsep}{0.5in}
\setlength{\textheight}{8.5in}
\setcounter{page}{1}
%\pagestyle{empty}
%\hbadness=10000

\begin{document}
\huge
\noindent
{Discrete Optimization Assignment:}
\vspace{0.25cm}

\noindent
{\bf Traveling Salesman Problem}
\normalsize

\section{Problem Statement}

In this assignment you will design an algorithm to solve a fundamental problem faced by every traveling salesperson, aptly named {\em The Traveling Salesman Problem (TSP)}.  All traveling salespeople start from their home, travel to several cities to sell their goods, and complete the day by returning home.  To minimize their costs, traveling salespeople strive to visit all of the cities using the shortest total travel distance.  This amounts to finding a visitation order of all of the cities that  minimizes the sum of distances traveled when moving from one city to another.  Figure \ref{fig:tsp} illustrates a small TSP and a feasible solution to that problem.  The cities are labeled from $0..4$.

\begin{figure}[h]
        \centering
        \begin{subfigure}[b]{8.0cm}%{0.3\textwidth}
                \centering
                \includegraphics[width=6cm]{figures/tsp_1.pdf}
                \caption{The TSP input data.}
                \label{fig:vrp:input}
        \end{subfigure}%
        ~ %add desired spacing between images, e. g. ~, \quad, \qquad etc.
          %(or a blank line to force the subfigure onto a new line)
        \hfill
        \begin{subfigure}[b]{8.0cm}
                \centering
                \includegraphics[width=6cm]{figures/tsp_2.pdf}
                \caption{A solution to the TSP.}
                \label{fig:vrp:sol}
        \end{subfigure}
        \caption{A Traveling Salesman Example}\label{fig:tsp}
\end{figure}

\section{Assignment}

Write an algorithm to solve the traveling salesman problem (a.k.a. minimize the length of a hamiltonian cycle\footnote{See \url{http://en.wikipedia.org/wiki/Hamiltonian_path}} of a graph).  The problem is mathematically formulated in the following way:  Given a list of locations $N = 0 \ldots n-1$ and coordinates for each location $\langle x_i,y_i \rangle \; i \in N$.  Let $v_i \;\; i \in N$ be a variable denoting the visitation order (i.e. the value of $v_i$ is the $i$-th location to be visited) and let ${\it dist}(l_1,l_2)$ be the Euclidean distance between two locations.\footnote{${\it dist}(n,m) = \sqrt{(x_{n} - x_{m})^2 + (y_{n} - y_{m})^2}$}  Then the traveling salesman problem is formalized as the following optimization problem,
$$
\begin{array}{ll}
\mbox{minimize:} & \displaystyle \sum_{i \in 0 \ldots n-1} dist(v_i,v_{i+1}) + dist(v_n,v_{0}) \\
\mbox{subject to:} & \\
     & v_i\mbox{ are a permutation of } N
\end{array}
$$
In this variant of the traveling salesman problem, we assume the sales person travels by helicopter and can go directly in a straight line from one point to another.

\section{Data Format Specification}

The input consists of $|N| + 1$ lines.  The first line contains one number $|N|$.
It is followed by $|N|$ lines, each line represents a point $\langle x_i, y_i \rangle$ where $x_i, y_j \in \mathbb{R}$.

\vspace{0.2cm}
\noindent
Input Format
\vspace{-0.2cm}
\begin{Verbatim}[frame=single]
|N|
x_0 y_0
x_1 y_1
...
x_|N|-1 y_|N|-1
\end{Verbatim}
%
The output has two lines.  The first line contains two values $obj$ and $opt$.  $obj$ is the length of the hamiltonian cycle (i.e. the objective value) as a real number.  $opt$ should be $1$ if your algorithm proved optimality and $0$ otherwise.  The next line is a list of $n$ values in $N$, one for each of the $v_i$ variables.  This line encodes the solution.

\vspace{0.2cm}
\noindent
Output Format
\vspace{-0.2cm}
\begin{Verbatim}[frame=single]
obj opt
v_0 v_1 v_2 ... v_|N|-1
\end{Verbatim}
%
%It is essential that the value order in the solution output matches the value order of the input.  Otherwise the grader will misinterpret the output.

\paragraph{Examples} \mbox{}
%\vspace{0.1cm}
%\noindent
(based on Figure \ref{fig:tsp})

\vspace{0.2cm}
\noindent
Input Example
\vspace{-0.2cm}
\begin{Verbatim}[frame=single]
5
0 0
0 0.5
0 1
1 1
1 0
\end{Verbatim}

\vspace{0.2cm}
\noindent
Output Example
\vspace{-0.2cm}
\begin{Verbatim}[frame=single]
5.2 0
0 4 1 3 2
\end{Verbatim}
%
This output represents the following hamiltonian cycle, $\{0 \rightarrow 4, 4 \rightarrow 1, 1 \rightarrow 3, 3 \rightarrow 2, 2 \rightarrow 0\}$

\section{Instructions}

\input{instructions.tex}

%We use \texttt{stdout} for output.
%Output to other stream will be ignored (you may want to send runtime information to \texttt{stderr}). Your submission will be tested on a department linux machine. If your algorithm is a standalone program, please name it \texttt{nr},
%otherwise, please specify the compilation procedure,
%it is appreciated if you also provide a script that follows the above format to run the program.

\paragraph{Resources}
You will find several  traveling salesman problem instances in the \texttt{data} directory provided with the handout.

%An example output file, \texttt{blabla.out}, is also provided.

%\section{Remarks}

\input{handin.tex}

\input{grading.tex}

\input{collaboration.tex}

%\paragraph{Questions} Please contact the class GTA Carleton (cjc@cs.brown.edu).

\input{warnings.tex}

%\paragraph{Hint}
%The optimal value for  \texttt{data/gc\_1000\_5} is near $85$.

\input{techReqs.tex}

\end{document}


\lab{Differentiation}{Differentiation}
\label{lab:Derivatives}
\objective{Derivatives are central in many applications.
Depending on the application and on the available information, the derivative may be calculated symbolically, numerically, or with differentiation software.
In this lab we explore these three ways to take a derivative, discuss what settings they are each appropriate for, and demonstrate their strengths and weaknesses.
% This lab should be done after completing the Python Essentials lab on SymPy.
}

\section*{Symbolic Differentiation} % =========================================

The derivative of a known mathematical function can be calculated symbolically with SymPy.
This method is the most precise way to take a derivative, but it is computationally expensive and requires knowing the closed form formula of the function.
Use \li{sy.diff()} to take a symbolic derivative.

\begin{lstlisting}
>>> import sympy as sy

>>> x = sy.symbols('x')
>>> sy.diff(x**3 + x, x)     # Differentiate x^3 + x with respect to x.
<<3*x**2 + 1>>
\end{lstlisting}

\begin{problem}
Write a function that defines $f(x) = (\sin(x) + 1)^{\sin(\cos(x))}$ and takes its symbolic derivative with respect to $x$ using SymPy.
Lambdify the resulting function so that it can accept NumPy arrays and return the resulting function handle.
% returns the derivative of $e^{\sin\left(\cos\left(x\right)\right)}$ at $x=1$ as a float using SymPy.

To check your function, plot $f$ and its derivative $f'$ over the domain $[-\pi, \pi]$.
It may be helpful to move the bottom spine to $0$ so you can see where the derivative crosses the $x$-axis.
\begin{lstlisting}
>>> from matplotlib import pyplot as plt

>>> ax = plt.gca()
>>> ax.spines["bottom"].set_position("zero")
\end{lstlisting}
% >>> for label in ["right", "top"]:      # Turn other spines off.
% ...     ax.spines[label].set_visible(False)
% ...
% \end{lstlisting}
\label{prob:sympy-symbolic-diff}
\end{problem}

\section*{Numerical Differentiation} % ========================================

% Derivatives can be approximated numerically by formulas called \emph{finite difference quotients}.
% Although these formulas do not return exact answers like SymPy, they are computationally inexpensive.
% They are especially advantageous in situations where the original function may not be known or when the using derivative is very complex.

One definition for the derivative of a function $f:\mathbb{R}\rightarrow\mathbb{R}$ at a point $x_0$ is
\[
f'(x_0) = \lim_{h\rightarrow 0} \frac{f(x_0 + h)-f(x_0)}{h}.
\]
Since this definition relies on $h$ approaching $0$, choosing a small, fixed value for $h$ approximates $f'(x_0)$:
\begin{equation}
f'(x_0) \approx \frac{f(x_0+h) - f(x_0)}{h}.
\label{eq:first-order-forward-difference}
\end{equation}
This approximation is called the \emph{first order forward difference quotient}.
Using the points $x_0$ and $x_0-h$ in place of $x_0+h$ and $x_0$, respectively, results in the \emph{first order backward difference quotient},
\begin{equation}
f'(x_0) \approx \frac{f(x_0) - f(x_0-h)}{h}.
\label{eq:first-order-backward-difference}
\end{equation}

Forward difference quotients use values of $f$ at $x_0$ and points greater than $x_0$, while backward difference quotients use the values of $f$ at $x_0$ and points less than $x_0$.
A \emph{centered difference quotient} uses points on either side of $x_0$, and typically results in a better approximation than the one-sided quotients.
Combining (\ref{eq:first-order-forward-difference}) and (\ref{eq:first-order-backward-difference}) yields the \emph{second order centered difference quotient},
\[
f'(x_0) = \frac{1}{2}f'(x_0) + \frac{1}{2}f'(x_0) \approx
\frac{f(x_0+h) - f(x_0)}{2h} + \frac{f(x_0) - f(x_0-h)}{2h}
= \frac{f(x_0+h) - f(x_0-h)}{2h}.
\]

\begin{figure}[H] % Show which points each quotient uses.
\centering
    \begin{tikzpicture}
    % Define colors
    % Create asterisks and name each point
    \foreach \p/\x in {a/-5,b/-4,c/-3,d/-2,e/-1,f/0,g/1,h/2,i/3,j/4,k/5}{\node[text depth=.25ex,text height=3ex] (\p) at (\x,0) {\textbf{*}};}
    % Circles
    \draw[thick,blue!30!black,opacity=0.5] (a) circle (7pt);
    \draw[thick,blue!30!black,opacity=0.5] (f) circle (7pt);
    \draw[thick,blue!30!black,opacity=0.5] (k) circle (7pt);
    % Arrows and labels below
    \draw[->,>=stealth',thick,blue,shorten >=-.1cm,text=black] (-5,-1) node[text=black,below,font=\normalsize,opacity=1,align=center] {$f'(\bar{x})$\\Forward Difference} -- (a);
    \draw[->,>=stealth',thick,green,shorten >=-.1cm] (0,-1) node[text=black,below,font=\normalsize,opacity=1,align=center] {$f'(\hat{x})$\\Centered Difference} -- (f);
    \draw[->,>=stealth',thick,red,shorten >=-.1cm] (5,-1) node[text=black,below,font=\normalsize,opacity=1,align=center] {$f'(\tilde{x})$\\Backward Difference} -- (k);
    % Labels above
    \foreach \s/\t in {a/\bar{x},b/\bar{x}+h,e/\hat{x}-h,f/\hat{x},g/\hat{x}+h,j/\tilde{x}-h,k/\tilde{x}}{\node[anchor=south,yshift=.5cm,text depth=.25ex,text height=1.5ex] at (\s) {$\t$};}
    % Background rectangles
    \begin{pgfonlayer}{background}
    \draw[ultra thick,draw=blue,fill=blue,opacity=0.2] (-5.5,.5) rectangle +(2,-1);
    \draw[ultra thick,draw=green,fill=green,opacity=0.2] (-1.5,.5) rectangle +(3,-1);
    \draw[ultra thick,draw=red,fill=red,opacity=0.2] (5.5,.5) rectangle +(-2,-1);
    \end{pgfonlayer}
    \end{tikzpicture}
\caption{} % The different types of difference quotients require evaluating the function at different points near the point of interest.}
\label{fig:difference-quotient-grid}
\end{figure}

\begin{info}
The finite difference quotients in this section all approximate the first derivative of a function.
The terms \emph{first order} and \emph{second order} refers to how quickly the approximation converges on the actual value of $f'(x_0)$ as $h$ approaches $0$, not to how many derivatives are being taken.

There are finite difference quotients for approximating higher order derivatives, such as $f''$ or $f'''$.
For example, the centered difference quotient
\[
f''(x_0) \approx \frac{f(x_0-h) - 2f(x_0) + f(x_0+h)}{h^2}
\]
approximates the second derivative.
This particular quotient is important for finite difference methods that approximate numerical solutions to some partial differential equations.
\end{info}

While we do not derive them here, there are other finite difference quotients that use more points to approximate the derivative, some of which are listed below.
Using more points generally results in better convergence properties.

\begin{table}[H]
\centering
\begin{tabular}{|c|c|c|}
\hline
Type & Order & Formula \\
\hline \multirow{6}{*}{Forward} & & \\
    & 1 & \Large{$\frac{f(x_0+h) - f(x_0)}{h}$} \\
    & & \\
    \cline{2-3} & & \\
    & 2 & \Large{$\frac{-3f(x_0) + 4f(x_0+h) - f(x_0+2h)}{2h}$} \\
    & & \\
\hline \multirow{6}{*}{Backward} & & \\
    & 1 & \Large{$\frac{f(x_0) - f(x_0-h)}{h}$} \\
    & & \\
    \cline{2-3} & & \\
    & 2 & \Large{$\frac{3f(x_0) - 4f(x_0-h) + f(x_0-2h)}{2h}$} \\
    & & \\
\hline \multirow{6}{*}{Centered} & & \\
    & 2 & \Large{$\frac{f(x_0+h) - f(x_0-h)}{2h}$} \\
    & & \\
    \cline{2-3} & & \\
    & 4 & \Large{$\frac{f(x_0-2h) - 8f(x_0-h) + 8f(x_0+h) -f(x_0+2h)}{12h}$} \\
    & & \\
\hline
\end{tabular}
\caption{Common finite difference quotients for approximating $f'(x_0)$.}
\label{table:finite-difference-quotients}
\end{table}

\begin{problem} % Implement finite difference quotients.
Write a function for each of the finite difference quotients listed in Table \ref{table:finite-difference-quotients}.
Each function should accept a function handle $f$, an array of points \li{x}, and a float $h$; each should return an array of the difference quotients evaluated at each point in \li{x}.

To test your functions, approximate the derivative of $f(x) = (\sin(x) + 1)^{\sin(\cos(x))}$ at each point of a domain over $[-\pi,\pi]$.
Plot the results and compare them to the results of Problem \ref{prob:sympy-symbolic-diff}.
\label{prob:implement-finite-difference-quotients}
\end{problem}

\subsection*{Convergence of Finite Difference Quotients} % --------------------

Finite difference quotients are typically derived using Taylor's formula.
This method also shows how the accuracy of the approximation increases as $h\rightarrow 0$:
\begin{equation}
f(x_0+h) = f(x_0) + f'(x_0)h + R_2(h)
\quad\Longrightarrow\quad
\frac{f(x_0+h) - f(x_0)}{h} - f'(x_0) = \frac{R_2(h)}{h},
\label{eq:taylor-formula-forward-diff}
\end{equation}
where $R_2(h) = h^2 \int_0^1 (1-t) f''(x_0+th)\:dt$.
Thus the absolute error of the first order forward difference quotient is
\[
\left| \frac{R_2(h)}{h} \right|
= |h| \left|\int_0^1 (1 - t)f''(x_0+th)\:dt\right|
\leq |h| \int_0^1 |1 - t||f''(x_0+th)|\:dt.
\]
If $f''$ is continuous, then for any $\delta > 0$, setting $M = \sup_{x \in (x_0-\delta, x_0+\delta)} f''(x)$ guarantees that
\[
\left | \frac{R_2(h)}{h} \right | \leq |h|\int_0^1 M dt = M|h|\hspace{0.03in}{\in}\hspace{0.03in}O(h).
\]
whenever $|h| < \delta$.
That is, the error decreases at the same rate as $h$.
If $h$ gets twice as small, the error does as well.
This is what is meant by a \emph{first order} approximation.
In a \emph{second order} approximation, the absolute error is $O(h^2)$, meaning that if $h$ gets twice as small, the error gets four times smaller.

\begin{info} % O(h^2) notation
The notation $O(f(n))$ is commonly used to describe the temporal or spatial complexity of an algorithm.
In that context, a $O(n^2)$ algorithm is much worse than a $O(n)$ algorithm.
However, when referring to error, a $O(h^2)$ algorithm is \textbf{better} than a $O(h)$ algorithm because it means that the accuracy improves faster as $h$ decreases.
\end{info}

\begin{problem} % Convergence of difference quotients.
Write a function that accepts a point $x_0$ at which to compute the derivative of $f(x) = (\sin(x)+1)^{\sin(\cos(x))}$.
Use your function from Problem \ref{prob:sympy-symbolic-diff} to compute the exact value of $f'(x_0)$.
Then use each your functions from Problem \ref{prob:implement-finite-difference-quotients} to get an approximate derivative $\tilde{f}'(x_0)$ for $h=10^{-8},10^{-7},\ldots,10^{-1},1$.
Track the absolute error $|f'(x_0) - \tilde{f}'(x_0)|$ for each trial, then plot the absolute error against $h$ on a log-log scale (use \li{plt.loglog()}).

Instead of using \li{np.linspace()} to create an array of $h$ values, use \li{np.logspace()}.
This function generates logarithmically spaced values between two powers of $10$.
\begin{lstlisting}
>>> import numpy as np
>>> np.logspace(-3, 0, 4)           # Get 4 values from 1e-3 to 1e0.
array([ 0.001,  0.01 ,  0.1  ,  1.   ])
\end{lstlisting}

For $x_0 = 1$, your plot should resemble the following figure.
\begin{figure}[H]
    \centering
    \includegraphics[width=.7\textwidth]{figures/error_plot.pdf}
\end{figure}
\label{prob:difference-quotient-convergence}
\end{problem}

\begin{warn}
Mathematically, choosing smaller $h$ values results in tighter approximations of $f'(x_0)$.
However, Problem \ref{prob:difference-quotient-convergence} shows that when $h$ gets too small, the error stops decreasing.
This numerical error is due to the denominator in each finite difference quotient becoming very small.
The optimal value of $h$ is usually one that is small, but not too small.
\end{warn}

\begin{problem}
The radar stations $A$ and $B$, separated by the distance $a = 500$ m, track a plane $C$ by recording the angles $\alpha$ and $\beta$ at one-second intervals.
Your goal, back at air traffic control, is to determine the speed of the plane.%
\footnote{This problem is adapted from an exercise in \cite{kiusalaas2013numerical}.}
%
\begin{figure}[H]
    \includegraphics[width=.5\textwidth]{figures/plane_diagram.png}
\end{figure}
%
Let the position of the plane at time $t$ be given by $(x(t),y(t))$.
The speed at time $t$ is the magnitude of the velocity vector, $\|\frac{d}{dt}(x(t),y(t))\| = \sqrt{x'(t)^2 + y'(t)^2}$.
The closed forms of the functions $x(t)$ and $y(t)$ are unknown (and may not exist at all), but we can still use numerical methods to estimate $x'(t)$ and $y'(t)$.
For example, at $t=3$, the second order centered difference quotient for $x'(t)$ is
\[
x'(3) \approx \frac{x(3+h) - x(3-h)}{2h} = \frac{1}{2}(x(4) - x(2)).
\]
In this case $h=1$ since data comes in from the radar stations at $1$ second intervals.

Successive readings for $\alpha$ and $\beta$ at integer times $t=7,8,\ldots,14$ are stored in the file \texttt{plane.npy}.
Each row in the array represents a different reading; the columns are the observation time $t$, the angle $\alpha$ (in degrees), and the angle $\beta$ (also in degrees), in that order.
The Cartesian coordinates of the plane can be calculated from the angles $\alpha$ and $\beta$ as follows.
\begin{equation}
\label{eq:differentiation-plane-conversion}
x(\alpha, \beta) = a \frac{\tan(\beta)}{\tan(\beta)-\tan(\alpha)}
\qquad
y(\alpha, \beta) = a \frac{\tan(\beta)\tan(\alpha)}{\tan(\beta)-\tan(\alpha)}
\end{equation}
Load the data, convert $\alpha$ and $\beta$ to radians, then compute the coordinates $x(t)$ and $y(t)$ at each given $t$ using \ref{eq:differentiation-plane-conversion}.
Approximate $x'(t)$ and $y'(t)$ using a forward difference quotient for $t=7$, a backward difference quotient for $t=14$, and a centered difference quotient for $t=8,9,\ldots,13$ (see Figure \ref{fig:difference-quotient-grid}).
Return the values of the speed $\sqrt{x'(t)^2+y'(t)^2}$ at each $t$.
\\(Hint: \li{np.deg2rad()} will be helpful.)
\end{problem}

\subsection*{Numerical Differentiation in Higher Dimensions} % ================

Finite difference quotients can also be used to approximate derivatives in higher dimensions.
The \emph{Jacobian matrix} of a function $f:\mathbb{R}^n \rightarrow \mathbb{R}^m$ at a point $\x_0 \in \mathbb{R}^n$ is the $m \times n$ matrix $J$ whose entries are given by
\begin{equation*}
J_{ij} = \frac{\partial f_i}{\partial x_j}(\x_0).
\end{equation*}
For example, the Jacobian for a function $f:\mathbb{R}^3 \rightarrow \mathbb{R}^2$ is defined by
\[
J = \left[\begin{array}{c|c|c}
\arrayrulecolor{gray}
\frac{\partial f}{\partial x_1}&\frac{\partial f}{\partial x_2}&\frac{\partial f}{\partial x_3}
\end{array}\right]
=
\left[\begin{array}{ccc}
\frac{\partial f_1}{\partial x_1}&\frac{\partial f_1}{\partial x_2}&\frac{\partial f_1}{\partial x_3}
\\ \\
\frac{\partial f_2}{\partial x_1}&\frac{\partial f_2}{\partial x_2}&\frac{\partial f_2}{\partial x_3}
\end{array}\right],
\qquad
\text{where}
\qquad
f(\x) =
\left[\begin{array}{c}
f_1(\x) \\ f_2(\x)
\end{array}\right],
\quad
\x = \left[\begin{array}{c}
x_1 \\ x_2 \\ x_3
\end{array}\right].
\]

The difference quotients in this case resemble directional derivatives.
The first order forward difference quotient for approximating a partial derivative is
\[
\frac{\partial f}{\partial x_j}(\x_0) \approx \frac{f(\x_0 + h\e_j) - f(\x_0)}{h},
\]
where $\e_j$ is the $j$th standard basis vector.
The second order centered difference approximation is
\begin{equation}
\frac{\partial f}{\partial x_j}(\x_0) \approx \frac{f(\x_0 + h\e_j) - f(\x_0 - h\e_j)}{2h}.
\label{eq:centered-quotient-high-dimension}
\end{equation}

\begin{problem}
Write a function that accepts a function $f:\mathbb{R}^n\rightarrow\mathbb{R}^m$, a point $\x_0 \in \mathbb{R}^n$, and a float $h$.
Approximate the Jacobian matrix of $f$ at $\x$ using the second order centered difference quotient in (\ref{eq:centered-quotient-high-dimension}).
\\(Hint: the standard basis vector $\e_j$ is the $j$th column of the $n\times n$ identity matrix $I$.)

To test your function, define a simple function like $f(x,y) = [x^2, x^3 - y]\trp$ where the Jacobian is easy to find analytically, then check the results of your function against SymPy or your own scratch work.
\label{prob:jac_center}
\end{problem}

\begin{comment} % Not enough room. Needs updating if added back in.
\begin{problem}
Find the error between your Jacobian function and the analytically computed derivative on the square $[-1,1] \times [-1,1]$ using ten thousand grid points (100 per side).
You may apply your Jacobian function to the points one at a time using a double \li{for} loop.  Once you get the error matrix for a given point, calculate the Frobenius norm of this matrix (\li{la.norm()} defaults to the Frobenius norm).  This norm will be your total error for that point.
Return the maximum error of your Jacobian function over all points in the square.

Hint: The following code defines the function
$f(x,y) = \left[\begin{array}{c} x^2 \\ x+y \end{array}\right]$.

\begin{lstlisting}
# f accepts a length-2 NumPy array
>>> f = lambda x: np.array([x[0]**2, x[0]+x[1]])
\end{lstlisting}
\end{problem}
\end{comment}

% TODO: Introduce Google tangent as well as HIPS Autograd. Be brief.

\section*{Differentiation Software} % =========================================

Many machine learning algorithms and structures, especially neural networks, rely on the gradient of a cost or objective function.
To facilitate their research, several organizations have recently developed Python packages for numerical differentiation.
For example, the Harvard Intelligent Probabilistic Systems Group (HIPS) started developing \li{autograd} in 2014 (\url{https://github.com/HIPS/autograd}) and Google released \li{tangent} in 2017 (\url{https://github.com/google/tangent}).
These tools are incredibly robust: they can differentiate functions with NumPy routines,\footnote{See \url{https://github.com/HIPS/autograd/blob/master/docs/tutorial.md} for which features Autograd supports.} \li{if} statements, \li{while} loops, and even recursion.
% They are therefore beneficial for calculating derivatives of unconventional or very complex functions.
We conclude with a brief introduction to Autograd.\footnote{Autograd is not included in Anaconda; install it with \texttt{pip install autograd}.}

Autograd's \li{grad()} accepts a scalar-valued function and returns its gradient as a function that accepts the same parameters as the original.
% The following code computes the derivative of $e^{\sin\left(\cos\left(x\right)\right)}$ at $x=1$ using Autograd.
To support most of the NumPy features, Autograd comes with its own thinly-wrapped version of Numpy, \li{autograd.numpy}.
Import this version of NumPy as \li{anp} to avoid confusion.

\begin{lstlisting}
>>> from autograd import numpy as anp       # Use autograd's version of NumPy.
>>> from autograd import grad

>>> g = lambda x: anp.exp(anp.sin(anp.cos(x)))
>>> dg = grad(g)                            # dg() is a callable function.
>>> dg(1.)                                  # Use floats as input, not ints.
-1.2069777039799139
\end{lstlisting}

\begin{comment} % Not enough room, and not useful later.
Autograd can differentiate a function as many times as desired.

\begin{lstlisting}
>>> f = lambda x: anp.sin(x) + 3**anp.cos(x)

# Calculate the first derivative.
>>> df = grad(f)

# Calculate the second derivative and so forth.
>>> ddf = grad(df)
>>> dddf = grad(ddf)
>>> dddf(1.)
2.683445898750351
\end{lstlisting}
\end{comment}

Functions that \li{grad()} produces do not support array broadcasting, meaning they do not accept arrays as input.
Autograd's \li{elementwise_grad()} returns functions that can accept arrays, like using \li{"numpy"} as an argument in SymPy's \li{sy.lambdify()}.

\begin{lstlisting}
>>> from autograd import elementwise_grad

>>> pts = anp.array([1, 2, 3], dtype=anp.<<float>>)
>>> dg = elementwise_grad(g)        # Calculate g'(x) with array support.
>>> dg(pts)                         # Evaluate g'(x) at each of the points.
array([-1.2069777 , -0.55514144, -0.03356146])
\end{lstlisting}

% >>> g = lambda x: anp.exp(anp.sin(anp.cos(x)))
% >>> g(pts)                          # Evaluate g(x) at an array of points.
% array([ 1.67262669,  0.66748447,  0.43343135])

SymPy would have no trouble differentiating $g(x)$ in these examples.
However, Autograd can also differentiate Python functions that look nothing like traditional mathematical functions.
For example, the following code computes the Taylor series of $e^{x}$ with a loop.

\begin{lstlisting}
>>> from sympy import factorial

>>> def taylor_exp(x, tol=.0001):
...     """Compute the Taylor series of e^x with terms greater than tol."""
...     result, i, term = 0, 0, x
...     while anp.abs(term) > tol:
...         term = x**i / int(factorial(i))
...         result, i = result + term, i + 1
...     return result
...
>>> d_exp = grad(taylor_exp)
>>> print(d_exp(2., .1), d_exp(2., .0001))
7.26666666667 7.38899470899
\end{lstlisting}

\begin{problem} % Take the derivative of a recursive function.
The \emph{Chebyshev Polynomials} satisfy the recursive relation
\[
T_0(x) = 1,\qquad T_1(x) = x,\qquad T_n(x) = 2xT_{n-1}(x) - T_{n-2}(x).
\]
Write a function that accepts an array $x$ and an integer $n$ and recursively computes $T_n(x)$.
Use Autograd and your first function to create a function for $T_n'(x)$.
Use this last function to plot each $T_n'(x)$ over the domain $[-1, 1]$ for $n=0,1,2,3,4$.
\\(Hint: Use \li{anp.ones_like(x)} to handle the case when $n = 0$.)
\end{problem}

\begin{problem} % Compare differentiation methods.
Let $f(x) = (\sin(x) + 1)^{\sin(\cos(x))}$ as in Problems \ref{prob:sympy-symbolic-diff} and \ref{prob:difference-quotient-convergence}.
Write a function that accepts an integer $N$ and performs the following experiment $N$ times.
\begin{enumerate}
\item Choose a random value $x_0$.
\item Use your function from Problem \ref{prob:sympy-symbolic-diff} to calculate the ``exact'' value of $f'(x_0)$.
Time how long the entire process takes, including calling your function (each iteration).
\item Time how long it takes to get an approximation $\tilde{f}'(x_0)$ of $f'(x_0)$ using the fourth-order centered difference quotient from Problem \ref{prob:difference-quotient-convergence}.
Record the absolute error $|f'(x_0) - \tilde{f}'(x_0)|$ of the approximation.
\item Time how long it takes to get an approximation $\bar{f}'(x_0)$ of $f'(x_0)$ using Autograd (calling \li{grad()} every time).
Record the absolute error $|f'(x_0) - \bar{f}'(x_0)|$ of the approximation.
\end{enumerate}

Plot the computation times versus the absolute errors on a log-log plot with different colors for SymPy, the difference quotient, and Autograd.
For SymPy, assume an absolute error of \li{1e-18} (since only positive values can be shown on a log plot).

For $N=200$, your plot should resemble the following figure.
Note that SymPy has the least error but the most computation time, and that the difference quotient takes the least amount of time but has the most error.
Autograd might be considered a ``happy medium,'' a least for this problem.

\begin{figure}[H]
    \includegraphics[width=.7\textwidth]{figures/efficiency.pdf}
    \caption{Solution with $N = 200$.}
\end{figure}
\end{problem}

\newpage

\section*{Additional Material} % ==============================================

\subsection*{More Autograd} % -------------------------------------------------

For scalar-valued functions with multiple inputs, the parameter \li{argnum} specifies the variable that the derivative is computed with respect to.
Providing a list for \li{argnum} gives several outputs.

\begin{lstlisting}
>>> f = lambda x,y: 3*x*y + 2*y - x

# Take the derivative of f with respect to the first variable, x.
>>> dfdx = grad(f, argnum=0)            # Should be dfdx(x,y) = 3y - 1,
>>> dfdx(5., 1.)                        # so dfdx(5,1) = 3 - 1 = 2.
2.0

# Take the gradient with respect to the second variable, y.
>>> dfdy = grad(f, argnum=1)            # Should be dfdy(x,y) = 3x + 2,
>>> dfdy(5., 1.)                        # so dfdy(5,1) = 15 + 2 = 17.
17.0

# Get the full gradient.
>>> grad_f = grad(f, argnum=[0,1])
>>> anp.array(grad_f(5., 1.))
array([  2.,  17.])
\end{lstlisting}

Finally, Autograd's \li{jacobian()} can differentiate vector-valued functions.
% The following example shows how to find the Jacobian of $f(x,y) = \left[\begin{array}{c} x^2 \\ x+y \end{array}\right]$.

\begin{lstlisting}
>>> from autograd import jacobian

>>> f = lambda x: anp.array([x[0]**2, x[0]+x[1]])
>>> f_jac = jacobian(f)
>>> f_jac(anp.array([1., 1.]))
array([[ 2.,  0.],
       [ 1.,  1.]])
\end{lstlisting}

\subsection*{Google Tangent} % ------------------------------------------------

Google's \li{tangent} package is similar to Autograd, both in purpose and syntax.
However, Tangent differentiates code ahead of time, while Autograd waits until the last second to actually do any calculations.
Tangent also tends to be slightly faster than Autograd.

\begin{lstlisting}
>>> import tangent              # Install with 'pip install tangent'.

>>> def f(x):                   # Tangent does not support lambda functions,
...     return x**2 - x + 3
...
>>> df = tangent.grad(f)
>>> df(10)                      # ...but the functions do accept integers.
19.0
\end{lstlisting}

\begin{comment} % NOT FINISHED! ...and not really necessary.
\subsection*{Convergence of the Centered Difference Quotient} % ---------------

Recall from the proof of Taylor's theorem that $R_k = \frac{f^{(k)}(x_0)}{k!}h^k + R_{k+1}$.
Therefore,
\begin{align*}
\frac{R_2(-h) - R_2(h)}{2h} &= \frac{1}{2h}\left(\frac{f''(x_0)}{2!}h^2 + R_{3}(-h) - \frac{f''(x_0)}{2!}h^2 - R_{3}(h) \right)\\
&= \frac{1}{2h} ( R_3(-h)-R_3(h))\\
&= \frac{1}{2h}\left(  \left( \int_0^1 \frac{(1-t)^2}{2} f'''(x_0+th) dt \right) h^3  -  \left(\int_0^1 \frac{(1-t)^2}{2} f'''(x_0-th) dt \right) h^3  \right)\\
&= \left(  \int_0^1 \frac{(1-t)^2}{4}( f'''(x_0+th)-f'''(x_0-th)) \right)h^2\\
&{\in}\hspace{0.3in}O(h^2)
\end{align*}
once $h$ is restricted to some $\delta$-neighborhood of 0.
So this centered difference quotient is of the second order and its error is smaller than using the first order forward and backward difference quotients when $|h|<1$.
\end{comment}

% =============================================================================
% OLD IMAGE FILTER MATERIAL (might work well in Fourier 2 lab?) ===============
% =============================================================================

\begin{comment}
\section*{Image Filters} % ====================================================

Recall that a computer stores an image as a 2-D array of pixel values (i.e., a matrix of intensities).
An image filter is a function that transforms an image by operating on it locally.
That is, to compute the $ij$th pixel value in the new image, an image filter uses only the pixels in a small neighborhood around the $ij$th pixel in the original image.

In this lab, we will use a filter derived from the gradient of an image to find edges in an image.

\subsection*{Convolutions} % --------------------------------------------------

One example of an image filter is to \emph{convolve} an image with a filter matrix.
A filter matrix is a matrix whose height and width are relatively small odd numbers.
If the filter matrix is
\[
F = \begin{array}
f_{-1,-1}&f_{-1,0}&f_{-1,1}\\
f_{0,-1}&f_{0,0}&f_{0,1}\\
f_{1,-1}&f_{1,0}&f_{1,1}
\end{array},
\]
then the convolution of an image $A$ with $F$ is $A \ast F = (C_{ij})$ where
\begin{equation}\label{equ:convolve}
C_{ij} = \sum_{k=-1}^1 \sum_{\ell=-1}^1 f_{k\ell}A_{i+k,j+\ell}.
\end{equation}
Say $A$ is an $m \times n$ matrix. Here, we take $A_{ij}=0$ when $i \not \in \{1, \ldots m\}$ or $j \not \in \{1, \ldots, n\}$.
The value of $C_{ij}$ is a linear combination of the nearby pixel values, with coefficients given by $F$ (see Figure \ref{fig:convolution}).
In fact, $C_{ij}$ equals the Frobenius inner product of $F$ with the $3 \times 3$ submatrix of $A$ centered at $ij$.

\begin{figure}
\centering
\begin{tikzpicture}
\node[draw, minimum width=12cm, minimum height=
    7cm, ultra thick](outer_rec)[]{};
\node[draw, minimum width=10cm, minimum height=
    5cm, ultra thick, fill=black!10!](inner_rec)[]{};
\node[draw, minimum width=2.5cm, minimum height=2.5cm,
    ultra thick, fill=black!30!](square1)at(-4.75,2.25){\textbullet};
\node[draw, minimum width=2.5cm, minimum height=2.5cm,
    ultra thick,fill=black!30!](square2)at(-1.75,-.75){\textbullet};
\node[draw, minimum width=2.5cm, minimum height=2.5cm,
    ultra thick, fill=black!30!](square3)at(4.75,.75){\textbullet};
\node[draw, minimum width=2.5cm, minimum height=2.5cm,
    ultra thick, fill=black!30!](square4)at(3.25,-1.25){\textbullet};
\draw[step=.5, ultra thin, color=black!50!](-6,-3.5)grid(6,3.5);

\node[draw, minimum width=10cm, minimum height=
    5cm, ultra thick, ][]{};
\node[draw, minimum width=2.5cm, minimum height=2.5cm,
    ultra thick]at(-4.75,2.25){};
\node[draw, minimum width=2.5cm, minimum height=2.5cm,
    ultra thick]at(-1.75,-.75){};
\node[draw, minimum width=2.5cm, minimum height=2.5cm,
    ultra thick]at(4.75,.75){};
\node[draw, minimum width=2.5cm, minimum height=2.5cm,
    ultra thick]at(3.25,-1.25){};
\end{tikzpicture}
\caption{This diagram illustrates how to convolve an image with a filter.
The light grey rectangle represents the original image $A$, and the dark grey squares are the filter $F$.
The larger rectangle is the image padded with zeros; i.e., all pixel values in the outer white band are 0.
To compute the entry of the convolution matrix $C$ located at a black dot, take the inner product of $F$ with the submatrix of the padded image centered at the dot.}
\label{fig:convolution}
\end{figure}

\subsubsection*{Implementation in NumPy} % - - - - - - - - - - - - - - - - - -

Let us write a function that convolves an image with a filter.
You can test this function on the image \li{cameraman.jpg}, which appears in Figure \ref{fig:cameraman}.
The following code loads this image and plots it with matplotlib.

\begin{lstlisting}
>>> image = plt.imread('cameraman.jpg')
>>> plt.imshow(image, cmap = 'gray')
>>> plt.show()
\end{lstlisting}

Here is the function definition and some setup.

\begin{lstlisting}
1. def Filter(image, F):
2.     m, n = image.shape
3.     h, k = F.shape
\end{lstlisting}

To convolve \li{image} with the filter \li{F}, we must first \emph{pad} the array \li{image} with zeros around the edges.
This is because in \eqref{equ:convolve}, entries $A_{ij}$ are set to zero when $i$ or $j$ is out of bounds.
We do this by creating a larger array of zeros, and then making the interior part of the array equal to the original image (see Figure \ref{fig:convolution}).

For example, if the filter is a $3 \times 3$ matrix, then the following code will pad the matrix with the appropriate number of zeros.

\begin{lstlisting}
 # Create a larger matrix of zeros
image_pad = np.zeros((m+2, n+2))
# Make the interior of image_pad equal to the original image
image_pad[1:1+m, 1:1+n] = image
\end{lstlisting}

We want to do this in general in our function.  Note that the number of zeros we need to pad our array depends on the size of the filter \li{F}.

\begin{lstlisting}
5.    image_pad = # Create an array of zeros of the appropriate size
6.   # Make the interior of image_pad equal to image
\end{lstlisting}

Finally, we iterate through the image to compute each entry of the convolution matrix.

\begin{lstlisting}
7.    C = np.zeros(image.shape)
8.    for i in range(m):
9.        for j in range(n):
10.            C[i,j] = # Compute C[i, j]
\end{lstlisting}

\subsubsection*{Gaussian Blur} % - - - - - - - - - - - - - - - - - - - - - - -

A \emph{Gaussian blur} is an image filter that operates on an image by convolving with the matrix
\[
G = \frac{1}{159}\begin{array}
2&4&5&4&2\\
4&9&12&9&4\\
5&12&15&12&5\\
4&9&12&9&4\\
2&4&5&4&2
\end{array}.
\]

Blurring an image can remove ``noise'', or random variation that is the visual analog of static in a radio signal (and equally undesirable).

\begin{problem}\label{prob:filter}
\leavevmode
Finish writing the function \li{Filter} by filling in lines 5, 6, and 10.  Hint: Note in \ref{equ:convolve}, $C_{ij}$ was calculated by summing from -1 to 1.  This is only the case if the filter \li{F} is $3 \times 3$. A slight modification is needed in the general case.  Test your function on the image \li{cameraman.jpg} using the Gaussian Blur. The result is in Figure \ref{fig:cameraman_blur}.
\end{problem}

\begin{figure}
\centering
\begin{subfigure}[b]{.49\textwidth}
\centering
\includegraphics[width=\textwidth]{figures/cameraman.jpg}
\caption{Unfiltered image.}
\label{fig:cameraman}
\end{subfigure}
\begin{subfigure}[b]{.49\textwidth}
\centering
\includegraphics[width=\textwidth]{figures/cameramanBlur.pdf}
\caption{Image after Gaussian blur is applied.}
\label{fig:cameraman_blur}
\end{subfigure}
\begin{subfigure}[b]{.49\textwidth}
\centering
\includegraphics[width=\textwidth]{figures/edges.pdf}
\caption{Image after the Sobel filter is applied.}
\label{fig:cameraman_edges}
\end{subfigure}
\caption{Here is an example of a Gaussian blur and the Sobel filter applied to an image.
This photo, known as ``cameraman,'' is a standard test image in image processing.
A database of such images can be downloaded from \url{http://www.imageprocessingplace.com/root_files_V3/image_databases.htm}.}
\label{fig:cameraman1}
\end{figure}

\subsection*{Edge Detection} % ------------------------------------------------

Automatic detection of edges in an image can be used to segment or sharpen the image.
We will find edges with the Sobel filter, which computes the gradient of the image at each pixel.
The magnitude of the gradient tells us the rate of change of the pixel values, and so large magnitudes should
correspond to edges within the image.
The Sobel filter is not a convolution, although it does use convolutions.

We can think of an image as a function from a $2 \times 2$ grid of points to $\mathbb{R}$.
The image maps a pixel location to an intensity.
It does not make sense to define the derivative of this function as a limit because the domain is discrete---a step size $h$ cannot take on arbitrarily small values.
Instead, we \emph{define} the derivative to be the centered difference quotient of the previous section.
That is, we define the derivative in the $x$-direction at the $ij$th pixel to be
\[
\frac{1}{2}A_{i+1, j} - \frac{1}{2}A_{i-1, j}.
\]

We can use a convolution to create a matrix $A_x$ whose $ij$th entry is the derivative of $A$ at the $ij$th entry, in the $x$-direction.
In fact, $A_x = A \ast S$, where
\[
S = \frac{1}{8}
\left[\begin{array}{ccc}
-1 & 0 & 1\\
-2 & 0 & 2\\
-1 & 0 & 1
\end{array}\right].
\]

Note that this convolution takes a weighted average of the $x$-derivatives at $(i, j)$, $(i, j+1)$, and $(i, j-1)$.
The derivative at $(i, j)$ is weighted by 2.
Using a weighted average instead of just the derivative at $(i, j)$ makes the derivative less affected by noise.

Now we can define the Sobel filter.
A Sobel filter applied to an image $A$ results in an array $B = (B_{ij})$ of 0's and 1's, where the 1's trace out the edges in the image.
By definition,
\[
B_{ij} = \left\{
     \begin{array}{ll}
       1 & \text{if}\; \;\|\nabla A(ij)\|_2 > M \\
       0 & \text{otherwise}.
     \end{array}
   \right.
\]
Here, $\nabla A(ij) = ((A \ast S)_{ij}, (A\ast S^T)_{ij})$ is the gradient of $A$ at the $ij$th pixel.
The constant $M$ should be ``sufficiently large'' enough to pick out those pixels with the largest gradient (i.e., those pixels that are part of an edge).
A good choice for $M$ is 4 times the average value of $\|\nabla A(ij)\|_2$ over the whole image $A$.

When the Sobel filter is applied to \li{cameraman.jpg}, we get the image in Figure \ref{fig:cameraman_edges}.
Here, the 1's in $B$ were mapped to ``white'' and the 0's were mapped to ``black.''

\begin{problem}
Write a function that accepts an image as input and applies the Sobel filter to the image.  Test your function on \li{cameraman.jpg}.  Hint: If you want to find the average of a matrix \li{A}, use the function \li{A.mean()}.
\end{problem}
\end{comment}