\documentclass[11pt,a4paper]{article}
% \renewcommand{\baselinestretch}{2}

\input{psfig}
\textwidth 17truecm
\textheight 24.2truecm
\hoffset -1.5truecm
\voffset -1.5truecm

\title{A New Entropy Measure Based on the Wavelet Transform and Noise Modeling}
\author{ J.-L. Starck \\
CEA/DSM/DAPNIA, CE-SACLAY, F-91191 Gif sur Yvette Cedex \\ [12pt]
F. Murtagh \\
Faculty of Informatics, University of Ulster, Magee College, \\
Londonderry BT48 7JL, Northern Ireland \\ [12pt]
R. Gastaud \\
CEA/DSM/DAPNIA, CE-SACLAY, F-91191 Gif sur Yvette Cedex}

\begin{document}
\maketitle

\abstract{
We present in this paper a new way to measure the information
in a signal, based on noise modeling. We show that the use of such
an entropy-related measure leads to good results for signal restoration.}

\section{Introduction}
The term ``entropy'' term is due to Clausius (1865), and the concept of
entropy was introduced by Boltzmann into statistical mechanics,
in order to measure the number of microscopic ways that a given macroscopic
state can be realized. Shannon \cite{shannon48}
founded the mathematical theory of
communication when he suggested that the information gained in a
measurement depends on the number of possible outcomes out of
which one is realized. Shannon also suggested that
the entropy can be used for maximization of the bits transferred under
a quality constraint. Jaynes \cite{jaynes57} proposed to use
the entropy measure
for radio interometric image deconvolution,  in
order to select in a set of possible solutions that which contains the
minimum of information, or following his entropy definition, that
which has a maximum entropy. In principle, the solution verifying such
a condition should be the most reliable. A lot of work has been carried out
in the last 30 years on the use of entropy for the general problem
of data filtering and deconvolution
\cite{ables74,bontekoe94,burg67,frieden75,gull91,narrayan86,pantin96,skilling84_1,
weir92}.

Traditionally information and entropy are determined from events and the
probability of their occurrence.  Signal and noise are basic building-blocks
of signal and data analysis in the physical sciences.  Instead of the
probability of an event, in this work
 we are led to consider the probabilities of our
data being either signal or noise.

Observed data $Y$ in the physical sciences
are generally corrupted by noise, which is often additive and which
follows in many cases a Gaussian distribution, a Poisson distribution, or
a combination of both.
 Using Bayes' theorem to evaluate the probability of the
realization of the original signal $X$,
knowing the data $Y$, we have

\begin{eqnarray}
 \mathrm{Prob}(X|Y) = \frac{\mathrm{Prob}(Y|X).\mathrm{Prob}(X)}{\mathrm{Prob}(Y)}
\label{eqn_bayes}
\end{eqnarray}
$\mathrm{Prob}(Y|X)$ is the conditional probability of getting the data
$Y$ given an original signal $X$, i.e.\ it represents the distribution
of the noise. It is given, in the case of uncorrelated Gaussian
noise with variance $\sigma^2$, by:
\begin{eqnarray}
 \mathrm{Prob}(Y|X) = \mathrm{exp}(-\sum_{pixels} \frac{ (Y-X)^2}{2{\sigma}^2} )
\label{eqn_proba}
\end{eqnarray}
The denominator in  equation \ref{eqn_bayes} is independent of $X$ and
 is considered as a constant (stationary noise).
$\mathrm{Prob(X)}$ is the a priori distribution
of the solution $X$. In the absence of any information on the solution
$X$ except its positivity, a possible course of action
is to derive the probability
of $X$ from its entropy, which is defined from information theory.

The main idea of information theory \cite{shannon48} is to establish
a relation between the received information and the probability of
the observed event \cite{bijaoui84}. If we note ${\cal I}(E)$ the information
related to the event $E$, and $p$ the probability of this
event happening, then we consider that
\begin{eqnarray}
%\cal{I}(E) =    f(p)
{\cal I}(E) = f(p)
\end{eqnarray}

Then we assume the two following principles:
\begin{itemize}
\item The information is a decreasing function of the probability. This
implies that the more information we have, the  less will be the probability
associated with one event.
\item Additivity of the information. If we have two independent events
$E_1$ and $E_2$, the information ${\cal I}(E)$ associated with the happening
of both is equal
to the addition of the information of each of them.
\begin{eqnarray}
{\cal I}(E) = {\cal I}(E_1) + {\cal I}(E_2)
\end{eqnarray}
\end{itemize}

Since $E_1$ (of probability $p_1$) and $E_2$ (of probability $p_2$) are
independent, then the probability of both happening is equal to the
product of $p_1$ and $p_2$.  Hence
\begin{eqnarray}
f(p_1 p_2) = f(p_1) + f(p_2)
\end{eqnarray}

Then we can say that the information measure is
\begin{eqnarray}
{\cal I}(E) = k \ln(p)
\end{eqnarray}
where k is a constant. Information must be positive, and $k$
is generally fixed at $-1$.

Another interesting measure is the mean information which is denoted
\begin{eqnarray}
H = - \sum_i p_i \ln(p_i)
\end{eqnarray}
This quantity is called the entropy of the system and was established by
Shannon in 1948 \cite{shannon48}.

This measure has several properties:
\begin{itemize}
\item It is maximal when all events have the same probability
$p_i = 1/ N_e$ ($N_e$ being the number of events), and is equal to
$\ln(N_e)$. It is in this
configuration that the system is the most undefined.
\item It is minimal when one event is sure. In this case, the system is
perfectly known, and no information can be added.
\item The entropy is a positive, continuous, and symmetric function.
%\item The mean information, obtained in two steps, can be added.
\end{itemize}

Then if we know the entropy $H$ of the solution (the next section
describes different ways to calculate it),
we derive its probability by
\begin{eqnarray}
\mathrm{Prob}(X) = \mathrm{exp}(- \alpha H(X))
\label{info_prop}
\end{eqnarray}

Given the data, the most probable image is obtained by maximizing
$\mathrm{Prob}(X|Y)$. Taking the logarithm of equation \ref{eqn_bayes}, we
thus need to maximize
\begin{eqnarray}
 \ln (\mathrm{Prob}(X|Y))  = - \alpha  H(X) + \ln(\mathrm{Prob}(Y|X)) -
\ln(\mathrm{Prob}(Y))
\end{eqnarray}
The last term is a constant and can be omitted.
Then, in the case of Gaussian noise, the solution is found by minimizing
\begin{eqnarray}
J(X) = \sum_{pixels} \frac{{(Y-X)}^{2}}{2 {\sigma}^{2}} + {\alpha} H(X)
= \frac{{\chi}^2}{2} + {\alpha} H(X)
\label{eqn_j1}
\end{eqnarray}
which is a linear combination of two terms: the entropy of the signal,
and a quantity corresponding to ${\chi}^2$ in statistics measuring the
discrepancy between the data and the predictions of the model.
$\alpha$ is a parameter that can be viewed alternatively as
a Lagrangian parameter or a value fixing the relative weight between
the goodness-of-fit and the entropy H.

For the deconvolution problem, the object-data relation is given by the
convolution
\begin{eqnarray}
Y = P * X
\end{eqnarray}
where $P$ is the point spread function, and the solution is found (in the case
of Gaussian noise) by minimizing
\begin{eqnarray}
J(X) = \sum_{pixels} \frac{{(Y-P*X)}^{2}}{2 {\sigma}^{2}} + {\alpha} H(X)
\end{eqnarray}

The way the entropy is defined is fundamental, because from its definition
will depend the solution. The next section discusses the different approaches
which have been proposed in the past.

\section{The concept of entropy}
\label{sect_entr}

We wish to estimate an unknown probability density $p(x)$ of the data.
A direct approach would be to build up the histogram of values $X(i)$, using
a suitable interval $\Delta x$, counting up how many times $m_k$ each interval
$(x_k, x_k + \Delta x)$ occurs among the N occurrences. Then the probability
that a data value belongs to an interval $k$ is $p_k = \frac{m_k}{N}$, and
each data value has a probability $p_k$. The entropy is defined by
\begin{eqnarray}
H_s(X) = - \sum_{k=1}^{m} p_k \ln(p_k)
\end{eqnarray}
where $m$ is the number of intervals.
The entropy is minimum and equal to zero when the signal is flat, and
increases when we have some fluctuations. Using this entropy in
equation~\ref{eqn_j1} for restoration leads to a minimum entropy restoration
 method.

The trouble with this approach is that, because the number of occurrences is
finite, the estimate $p_k$ will be in error by an amount proportional
to $m_k^{-\frac{1}{2}}$~\cite{frieden}. The error becomes significant when
$m_k$ is small. Furthermore this kind of entropy definition is not
easy to use for signal restoration, because the gradient of
equation~\ref{eqn_j1} is not easy to compute. For these reasons, other
entropy functions are generally used. The main ones are:
\begin{itemize}
\item Burg \cite{burg67}:
\begin{eqnarray}
H_b(X) = -\sum_{pixels} \ln(X)
\end{eqnarray}
\item Frieden \cite{frieden75}:
\begin{eqnarray}
H_f(X) = -\sum_{pixels}X \ln(X)
\end{eqnarray}
\item Gull and Skilling \cite{gull91}:
\begin{eqnarray}
H_g(X) = \sum_{pixels} X - M - X \ln(X|M)
\end{eqnarray}
\end{itemize}
Each of these entropies can be used, and they correspond to different
probability distributions that one can associate with
an image \cite{narrayan86}.
(See \cite{frieden75,skilling84_1,skilling89} for descriptions).
The last definition of the entropy has the advantage of having a zero
 maximum when $X$ equals the model $M$, usually taken as a flat image.
All of these entropy measures are negative, and maximum when the image is flat.
They are negative because an offset term is omitted which has no importance
for the minimization of the functional. The fact that we consider that
a signal has maximum information value when it is flat is evidently
a curious way to measure information. The probability of $X$
must be defined by $\mathrm{Prob}(X) = \mathrm{exp}(\alpha H(X))$. The sign has
been inverted
(see equation~\ref{info_prop}), which is natural if we want the best
solution to be  the smoothest. These three entropies lead to the
maximum entropy restoration method, for which the
solution is found by minimizing
(for Gaussian noise)
\begin{eqnarray}
J(X) = \sum_{pixels} \frac{{(Y-X)}^{2}}{2 {\sigma}^{2}} - {\alpha} H(X)
\label{eqn_j3}
\end{eqnarray}

In 1986, Narayan and Nityanda \cite{narrayan86}
compared several entropy functions,
 and finally
concluded by saying that all were comparable if they have good
properties, i.e.\ they enforce positivity, and they have a negative
second derivative which discourages ripple. They showed also that
results varied strongly with the background level, and
that these entropy functions produced poor results
for negative structures, i.e.\ structures under the background level
(absorption area in an image, absorption band in a spectrum, etc.), and
compact structures in the signal.
The Gull and Skilling entropy gives rise to
the difficulty of estimating a model.
Furthermore it has been shown \cite{bontekoe94}
that the solution was dependent on this choice.

Many studies \cite{weir92,bontekoe94,pantin96}
have been carried out in order to improve the functional to be minimized.
But the question which should be raised is: what is a good entropy for
signal restoration?

Trying to answer this corresponds to asking what is the information
in the signal. The entropy should verify the following criteria:
\begin{enumerate}
\item The information in a flat signal is zero.
\item The amount of information in a signal is independent of the background.
\item The amount of information is dependent on the noise. A given
signal $Y$ ($Y = X + Noise$) doesn't furnish the  same information if
the noise is high or small.
\item The entropy must work in the same way for a pixel which
has a value $B + \epsilon$ ($B$ being the background), and
for a pixel which has a value $B - \epsilon$.
\item The amount of information is dependent on the correlation in the signal.
If a signal $S$  presents large features above the noise, it contains
a lot of information. By generating a new set of  data from $S$, by
randomly taking the pixel values in $S$, the large features will
evidently disappear, and this new signal will contain less information.
But the pixel values will be the same as in $S$.
\end{enumerate}

\begin{figure}[htb]
\centerline{
\hbox{
\psfig{figure=lenna256.ps,bbllx=1.8cm,bblly=12.9cm,bburx=14.5cm,bbury=25.5cm,width=8cm,height=8cm,clip=}
\psfig{figure=scrambled_lenna.ps,bbllx=1.8cm,bblly=12.9cm,bburx=14.5cm,bbury=25.5cm,width=8cm,height=8cm,clip=}
}}
\caption{Lena image (left) and the same data distributed differently (right).
These
two images have the same entropy, using any of the standard entropy methods.}
\label{fig_lenna}
\end{figure}

Figure~\ref{fig_lenna} shows the Lena image (left), and an image obtained
by distributing randomly the Lena image pixel values. For someone, who is
not involved in image processing, the second image contains less information
than the first one. For someone working on image transmission, it is clear
that the second image will require more bits for a lossless transmission,
and from this point of view, he will consider that the second one contains
more information. The standard entropy methods produce exactly
the same value for both images and, for such methods, both images contain
the same amount of information. For data restoration, all fluctuations
due to noise are not of interest, and do not contain relevant
information.  From this physical point of view, that is the reason why
the standard
definition of entropy seems badly adapted to information measurement
in signal restoration.

\section{Entropy from noise modeling}

In the case of signal restoration, the noise is the main problem. This
means that we should not consider the probability of appearance of
a pixel value in an image, but rather its probability of being due to the
signal (or to the noise).
If we consider a variable $x$ which follows a probability distribution
$p(x)$, we
can define the information in $x$ by $- \ln(p(x))$, and a signal $S$ can
be considered as a set of individual variables $x_k$ (pixels), each of which
follows the same probability distribution.
Then the information contained in the data
can be measured by $- \sum_{pixel} \ln(p(x))$. If $x$ follows a Gaussian
distribution with zero mean, we have
\begin{eqnarray}
H(X) = \sum_{pixel} \frac{x^2}{2 \sigma^2}
\end{eqnarray}
The energy gives a good measurement of information. But many of the required
criteria are not fulfilled by using such an entropy (correlation between
pixels, background-independent, etc.). It seems difficult to derive
a good probability distribution from the pixel values which fulfill the
entropy requirements.

This is not so for transformed data, especially when using
the wavelet transform.
This has  already been done, in fact,
for finding threshold levels in filtering
methods by means of wavelet coefficient thresholding \cite{starck95}. Thus we
must introduce the concept of multiresolution into our entropy.
We will now consider that the information contained in some dataset
is the sum of the information at different resolution levels $j$.
Choosing the ``\`a trous'' wavelet transform (see \cite{starck95}
for a description of this wavelet transform algorithm), a signal $S$ can
be represented by:
\begin{eqnarray}
S(k) = \sum_{j=1}^{l} w_j(k) + c_l(k)
\end{eqnarray}
where $k$ is the pixel index, $w_j$ are the wavelet coefficients of $S$, $j$ the
resolution
level, and $c_l$ the smoothed version of $S$. Due to the properties of
the wavelet transform, the set $w_j(x)$ for all $x$ has a zero mean. From
noise modeling, we can derive the probability distribution in the
wavelet space of a wavelet coefficient, assuming it is due to the noise.
The entropy becomes
\begin{eqnarray}
H(X) = - \sum_{j=1}^{l}  \sum_{k=1}^{N} \ln ( p(w_j(k)))
\end{eqnarray}
For Gaussian noise, we get
\begin{eqnarray}
H(X) =  \sum_{j=1}^{l}  \sum_{k=1}^{N} \frac{w_j(k)^2}{2 \sigma_j^2}
\end{eqnarray}
where $\sigma_j$ is the noise at scale $j$. We see that
the information is proportional
to the energy of the wavelet coefficients.
The higher a wavelet coefficient, then the lower will be the  probability, and the
higher will
be the information furnished by this wavelet coefficient. We can see
easily that this entropy fulfills all the requirements of
section~\ref{sect_entr}.
If we consider two signals $S_1$, $S_2$, derived from a third one $S_0$ by
adding noise:
\begin{eqnarray}
S_1 & = & S_0 + N_1(\sigma_1) \\ \nonumber
S_2 & = & S_0 + N_2(\sigma_2)
\end{eqnarray}
then we have:
\begin{eqnarray}
\mbox{if } \sigma_1 < \sigma_2 \mbox{ then } H(S_1) > H(S_2)
\end{eqnarray}
and a flat image has zero entropy.

Our entropy definition is completely dependent on the noise modeling.
If we consider a signal $S$, and we assume that the noise is Gaussian, with
a  standard deviation equal to $\sigma$, we won't measure the same
information compared to  the case when we consider that the noise has
another standard deviation
value, or if the noise follows another distribution.

\begin{figure}[htb]
\centerline{
\hbox{
\psfig{figure=fig_multi_memscale.ps,bbllx=2.5cm,bblly=13cm,bburx=19.5cm,bbury=25.5cm,width=14cm,height=7cm,clip=}
}}
\caption{Multiscale entropy of the Lena image (continuous curve),
and multiscale
entropy of the scrambled image (dashed curve).}
\label{fig_multi_memscale}
\end{figure}
Figure~\ref{fig_multi_memscale} shows the information measure at each
scale for both the Lena image and its scrambled version. The global
information is the addition of the information at each scale. We see
that for the scrambled image (dashed curve), the
information-versus-scale
curve is flat, while for the unscrambled Lena image,
it increases with the scale.

\section{Signal information and noise information}
\subsection{Definition}
In the previous section, we have seen how it was possible to measure
the information related to a wavelet coefficient. Since the data
is composed of an original signal and noise, our information measure
is corrupted by noise. Trying to decompose our information measure
into two components, one ($H_S$) corresponding to the non-corrupted part, and
another  ($H_N$) to the corrupted part, we have
\begin{eqnarray}
H(X) = H_S(X) + H_N(X)
\end{eqnarray}
We will define in the following $H_S$ as the signal information, and $H_N$
as the noise information. It must be clear that noise does not
contain any information, and what we call noise information is a quantity
which is measured as information by the multiscale entropy, and which is
probably not informative to us.

As described in the previous section, the information $h$ relative to
a wavelet coefficient $w_j$ is $- \ln(p(w_j))$. If the wavelet coefficient
is small, its value can be due to the noise,
and  $h$ should be assigned to $H_N$.
If the wavelet coefficient is high, compared to the noise standard
deviation, $h$ cannot be due to the noise, and $h$ should be assigned to $H_S$.
$h$ can be distributed as $H_N$ or $H_S$ based on  the probability $p_n(w_j)$
that the wavelet coefficient is due to noise, or the probability
$p_s(w_j)$ that it is due to
signal.  We have $p_s(w_j) = 1 - p_n(w_j)$. We consider
that  $h_n(w_j) = - p_n(w_j) \ln(p(w_j))$ is
the noise information, and
$h_s(w_j) = - p_s(w_j) \ln(p(w_j))$ is the signal information.
Hence  signal information and  noise information are defined by
\begin{eqnarray}
H_s(X) & = & \sum_{j=1}^{l}  \sum_{k=1}^{N} h_s(w_j(k)) = - \sum_{j=1}^{l}
\sum_{k=1}^{N} p_s(w_j(k)) \ln ( p(w_j(k))) \\ \nonumber
H_n(X) & = & \sum_{j=1}^{l}  \sum_{k=1}^{N} h_n(w_j(k)) = - \sum_{j=1}^{l}
\sum_{k=1}^{N} p_n(w_j(k)) \ln ( p(w_j(k)))
\end{eqnarray}
For the Gaussian noise case, we estimate $p_n(w_j)$ that a wavelet
coefficient is due to the noise by
\begin{eqnarray}
p_n(w_j) = \mathrm{Prob}(W > \mid w_j \mid)  & =  & \frac{2}{\sqrt{2 \pi}
\sigma_j} \int_{\mid w_j \mid}^{+\infty} \exp(-W^2/2\sigma^2_j) dW\\ \nonumber
 & = & \mbox{erfc}(\frac{\mid w_j \mid }{\sqrt{2}\sigma_j})
\end{eqnarray}
and
\begin{eqnarray}
H_s(X) &= & \sum_{j=1}^{l}  \sum_{k=1}^{N}  \frac{w_j^2}{2\sigma_j^2}
\mbox{erf}(\frac{\mid w_j \mid }{\sqrt{2}\sigma_j}) \\ \nonumber
H_n(X) &= & \sum_{j=1}^{l}  \sum_{k=1}^{N}  \frac{w_j^2}{2\sigma_j^2}
\mbox{erfc}(\frac{\mid w_j \mid }{\sqrt{2}\sigma_j})
\label{eq_entrop_result_1}
\end{eqnarray}

Note that $H_s(X) + H_n(X)$ is always equal to $H(X)$.
For Gaussian noise, the functional to minimize becomes
\begin{eqnarray}
J(X) = \sum_{pixels} \frac{{(Y-X)}^{2}}{2 {\sigma}^{2}} + {\alpha} (H_s(X)+H_n(X))
\end{eqnarray}
If we want to preserve features with high signal-to-noise ratio from the
regularization, we just omit $H_s(X)$ and we get
\begin{eqnarray}
J(X) = \sum_{pixels} \frac{{(Y-X)}^{2}}{2 {\sigma}^{2}} + {\alpha} H_n(X)
\end{eqnarray}
We seek a solution which minimizes the amount of information which could
be due to the noise.

By this measure, information relative to high wavelet coefficients is
completely assigned to the signal. This allows us also to exclude
wavelet coefficients with high signal-to-noise ratio (SNR)
from the regularization.
It leads to perfect fit of the solution with the data at scales and
 positions with high SNR. If we want to consider the information due
to noise, even for significant wavelet coefficients, the noise information
relative to a wavelet coefficient is
\begin{eqnarray}
h_n(w_j) =  \int_{0}^{\mid w_j \mid } p_n(u|w_j) (\frac{\partial H(x)}{\partial
x})_{x=u} du
\end{eqnarray}
which gives for Gaussian noise
\begin{eqnarray}
h_n(w_j) =  \frac{1}{\sigma_j^2} \int_{0}^{\mid w_j \mid} u \mbox{
erfc}(\frac{\mid w_j \mid -u}{\sqrt{2} \sigma_j}) du
\label{eqn_hn2}
\end{eqnarray}
and the noise and signal information in a signal are
\begin{eqnarray}
H_s(X) & = & \sum_{j=1}^{l}  \sum_{k=1}^{N}  \frac{1}{\sigma_j^2} \int_{0}^{\mid
w_j \mid} u \mbox{ erf}(\frac{\mid w_j \mid -u}{\sqrt{2} \sigma_j}) du \\
\nonumber
H_n(X) & = & \sum_{j=1}^{l}  \sum_{k=1}^{N}   \frac{1}{\sigma_j^2} \int_{0}^{\mid
w_j \mid} u \mbox{ erfc}(\frac{\mid w_j \mid -u}{\sqrt{2} \sigma_j}) du
\label{eq_entrop_result_2}
\end{eqnarray}

Equations 27 and 32 lead to two
different ways to regularize a signal. The first requires that we use
all the information which is furnished in high wavelet coefficients, and
leads to an exact preservation of the flux in a structure. If the signal
presents high discontinuities, artifacts can appear in the solution
due to the fact that the wavelet coefficients located at the discontinuities
are not noisy, but have been modified like noise. The second equation
doesn't have this drawback, but a part of the flux of a structure
(compatible with noise amplitude) can be lost in the restoration process.
It is however not as effective as in the standard maximum entropy methods.

\subsection{A new approach for signal restoration}
The new  definition of the information contained
in noisy data can easily lead to a new approach for restoration of images.

The problem of filtering or restoring data $D$ can be expressed by the
following:
We search for a solution $\tilde D$ such that the difference between
$D$ and $\tilde D$ minimizes the information due to the signal, and
such that  $\tilde D$ minimizes the information due to the noise.
\begin{eqnarray}
J(\tilde D) = H_s(D-\tilde D) + H_n(\tilde D)
\label{eqn_func1}
\end{eqnarray}
Furthermore, the smoothness of the solution can be controlled by adding
a parameter:
\begin{eqnarray}
J(\tilde D) = H_s(D-\tilde D) + \alpha H_n(\tilde D)
\label{eqn_func2}
\end{eqnarray}
Here, $\alpha$ is considered as a constant value, but we can easely
imagine to have a regularization parameter per scale, or even per wavelet
coefficient, depending on the Signal to Noise Ratio of the data. This
direction will be investigated in the future.

Three points must be noted:
\begin{itemize}
\item The positivity of the solution is not enforced.
\item There is no constraint on the flux.
\item The last scale of the wavelet transform is not taken into
account in this entropy.
\end{itemize}
The first two points can be easily resolved by introducing strict a priori
 constraints on the solution \cite{thiebault95}:
\begin{eqnarray}
J(Z) = H_s(D-{\cal C}(Z)) + \alpha H_s({\cal C}(Z))
\end{eqnarray}
And the real solution is evidently $\tilde D = {\cal C}(Z)$.
Positivity and total flux conservation impose
\begin{eqnarray}
{\cal C}(Z)(x) = \frac{\sum_x I(x)}{\sum_x Z(x)^2} Z(x)^2
\end{eqnarray}
Any other constraint can evidently be introduced into the function ${\cal C}$.

There is no constraint to be introduced to cater for the third point above,
but this should not be a problem
if the number of scales we use for the entropy is high enough. Indeed, in
this case, the last scale becomes flat, and flux normalization should
correctly fix this level.

\subsection{Example}
\begin{figure}[htb]
\centerline{
\vbox{
\psfig{figure=fig_lit6_mem1.ps,bbllx=2.5cm,bblly=13cm,bburx=19.5cm,bbury=25.5cm,width=16cm,height=9cm}
}}
\caption{Spectrum and filtered spectrum superimposed.}
\label{fig_lit6_mem1}
\end{figure}

\begin{figure}[htb]
\centerline{
\vbox{
\psfig{figure=fig_lit6_mem2.ps,bbllx=2.5cm,bblly=13cm,bburx=19.5cm,bbury=25.5cm,width=16cm,height=10cm}
}}
\caption{Difference (upper part)
between the real spectrum and its smoothed version.
Part (pixels 400 to 500) of the spectrum (continuous curve),
with the filtered spectrum
overplotted (dashed).}
\label{fig_lit6_mem2}
\end{figure}
Figure~\ref{fig_lit6_mem1} presents a spectrum and the result (overplotted)
after filtering using the multiscale entropy. The difference between the
spectrum and its smoothed version is plotted in
Figure~\ref{fig_lit6_mem2} (upper part).
As we can see, the residual contains only noise.
In order to better see the quality of the smoothing, we have plotted
only a part of the spectrum (see lower part of Figure~\ref{fig_lit6_mem2}),
and the filtered spectrum superimposed. The absorption lines are not
modified using our filtering technique.

\begin{figure}[h]
\centerline{
\hbox{
\psfig{figure=fig_lenna_g10.ps,bbllx=1.8cm,bblly=12.8cm,bburx=14.5cm,bbury=25.5cm,width=8cm,height=8cm,clip=}
\psfig{figure=fig_filmem_g2_lenna.ps,bbllx=1.8cm,bblly=12.8cm,bburx=14.5cm,bbury=25.5cm,width=8cm,height=8cm,clip=}
}}
\caption{Left, Lena + Gaussian noise, right, filtered image .}
\label{fig_filter_gauss_noise}
\end{figure}
Figure~\ref{fig_filter_gauss_noise} (left) shows the Lena image
(see Figure~\ref{fig_lenna}) to which Gaussian
noise of standard deviation 10 has been added.  Figure~\ref{fig_filter_gauss_noise} (right) shows
the result using equation 32 with a regularization parameter value of 2.

\section{Conclusion}

We have seen that information must be measured from the transformed
data, and not from the data itself. This approach has been used in fact
for several years in the domain of image compression. Indeed, modern
image compression methods consist firstly of  applying a transformation
(cosine transform for JPEG, wavelet transform, etc.) to the image, and
then coding the coefficients obtained. A good transform for
image compression is obviously an orthogonal transform because there
is no redundancy, and the number of pixels is the same as in the original
image. The exact number of bits necessary to code the coefficients is
given by the Shannon entropy. For signal restoration, the problem is
not to reduce the number of bits in the representation of the data, and we
prefer to use a non-orthogonal wavelet transform, which avoids
artifacts in reconstruction due to undersampling.

We could
have used the Shannon entropy to measure the information at a given scale,
and derive the bins of the histogram from the standard deviation of the noise,
but for several reasons we thought it better to directly introduce
noise probability into our information measure.
Firstly, we have seen that this leads, for Gaussian noise, to
a very physical relation between the information and the wavelet coefficients:
information is proportional to the energy of the wavelet coefficients
normalized by the standard deviation of the noise.
Secondly, it works even in the case
of images with few photons/events (the histograms in this case present a bias).
We have seen that the equations are easy to manipulate.
Finally, experiments have
confirmed that this approach gives good results.
We have seen also that our new information
measure leads naturally to a new method for signal restoration.
We are now experimenting with this method, and working on generalizations to
other classes of noise.

\newpage
\include{annexA}
\include{annexB}

\begin{thebibliography}{99}
\bibitem{ables74} J.G. Ables, {\it Astronomy and Astrophysics Supplement Series},
15, 383--393, 1974.

\bibitem{bijaoui84} A. Bijaoui, {\it Introduction au
Traitement Num\'erique des Images}, Masson, Paris, 1984.

 \bibitem{bijaoui94} A. Bijaoui, J.L. Starck and
 F. Murtagh, ``Restauration des
 images
 multi-\'echelles par l'algorithme \`a trous'', {\it Traitement du Signal},
 11, 229--243, 1994.

\bibitem{bontekoe94} Tj.R. Bontekoe, E. Koper and D.J.M. Kester,
``Pyramid maximum entropy images of IRAS survey data'', {\it Astronomy and
 Astrophysics}, 294, 1037--1053, 1994.

\bibitem{burg67} J.P. Burg, {\it
Annual Meeting International Society Exploratory Geophysics}, 1967.
Reprinted in {\it Modern Spectral Analysis}, 1978, D.G. Childers, ed.,
IEEE Press, New York, 34--41, 1978.

 \bibitem{chui92} C.H. Chui, {\it Wavelet Analysis and Its
 Applications}, Academic Press, New York, 1992.

 \bibitem{daubechies88} I. Daubechies,
 ``Orthogonal bases of compactly supported
 wavelets'', {\it Communications on Pure and Applied Mathematics},
 41, 909--996, 1988.

\bibitem {frieden75} B.R. Frieden, ``Image enhancement and restoration'',
 {\it Topics in Applied Physics}, 6,  Springer-Verlag Berlin, 177--249, 1975.

\bibitem{frieden} B.R Frieden, {\it Probability, Statistical Optics, and
Data Testing: a problem solving approach}, 2nd edition, Springer-Verlag Berlin,
1991.

\bibitem{jaynes57} E.T. Jaynes, {\it Phys. Rev.}, 106, 620--630, 1957.

 \bibitem{holschneider89} M. Holschneider, R. Kronland-Martinet, J. Morlet
 and Ph. Tchamitchian, ``A real-time algorithm for signal analysis with the
 help of the wavelet transform'', in
 {\it Wavelets: Time-Frequency Methods and Phase Space},
 Springer-Verlag,  Berlin, 286--297, 1989.

\bibitem{gull91} S.F. Gull, and J. Skilling, MEMSYS5 User's Manual, 1991.

\bibitem{narrayan86} R. Narayan, and R. Nityananda, ``Maximum
Entropy Image Restoration in Astronomy'', Ann. Rev. Astron. Astrophys.,
24, pp 127--170, 1986.

\bibitem{pantin96} E. Pantin and J.L. Starck, ``Deconvolution of
astronomical images using the multiresolution maximum entropy method'',
 {\it Astronomy and Astrophysics Supplement Series}, 118, 575--585, 1996.

 \bibitem {ruskai92} M.M. Ruskai, G. Beylkin, R. Coifman, I. Daubechies,
 S. Mallat, Y. Meyer and  L. Raphael, {\it Wavelets and Their Applications},
 Jones and Barlett, 1992.

\bibitem{shannon48} C.E. Shannon,
``A Mathematical Theory for Communication'', {\it Bell System Tech J.} 27
379--423, 1948.

\bibitem{shensa92} M.J. Shensa, ``Discrete wavelet transforms:
 wedding  the \`a trous and Mallat algorithms", {\it IEEE Transactions on
 Signal
 Processing}, 40, 2464--2482, 1992.

\bibitem{skilling84_1} J. Skilling and S.F. Gull, {\it Proceedings of
the American Mathematical Society}, SIAM, 14, 167, 1984.

\bibitem{skilling89} J. Skilling, ``Classic maximum entropy'',
in: Skilling. J, ed., {\it Maximum
Entropy and
Bayesian Methods}, Kluwer, Dordrecht, 45--52, 1989.

\bibitem{starck95}  J.L. Starck, F. Murtagh and  A. Bijaoui,
``Multiresolution support
 applied to image filtering and deconvolution'', {\it Graphical
Models and Image Processing}, 57, 420--431, 1995.

\bibitem{starck97} J.L. Starck, F. Murtagh  and A. Bijaoui,
{\it Image Processing and Data Analysis: The
Multiscale Approach},  Cambridge University Press, forthcoming, 1997.

\bibitem{weir92} N. Weir, ``A multi-channel method of maximum entropy image
restoration'', in: D.M. Worral, C. Biemesderfer and J. Barnes,
eds., {\it Astronomical Data Analysis Software and System 1}, Astronomical
Society of the Pacific, San Francisco, 186--190, 1992.

\bibitem{thiebault95} E. Thiebault and J.-M. Conan, ``Strict a priori
constraints for maximum-likelihood blind deconvolution'', {\it JOSA},
12, 485--492, 1995.

\end{thebibliography}

\end{document}
\subsection{Spectrum filtering}

\subsection{Image filtering}
\begin{figure}[h]
\centerline{
\hbox{
\psfig{figure=fig_simu4_bw.ps,bbllx=1.8cm,bblly=12.8cm,bburx=14.5cm,bbury=25.5cm,width=16cm,height=16cm,clip=}
}}
\caption{(a) Simulated image, (b) simulated image and
Gaussian noise, (c) filtered image, and (d)
residual image.}
\label{fig_filter_gauss_noise}
\end{figure}

A simulated image containing stars and galaxies is shown in Fig.\
\ref{fig_filter_gauss_noise} (top left). The simulated noisy
image, the filtered image and the residual image are respectively shown in
Fig.\ \ref{fig_filter_gauss_noise} top right, bottom left, and
bottom right. We can see that there is no structure in the residual
image. The filtering was carried out using the multiresolution support.


In this section we individually present the interpretation and rationale for our solutions.

\subsection{Question 1}
Question 1 is intended to: \textit{``Find the top 10 most frequent routes during the last 30 minutes''}.

This particular query is the same that was asked in the first delivery of this course and the execution was inspired in that same implementation with spark streaming.

At first we started by translating the coordinates to a 300x300 grid as requested in the question. After that we filter all the invalid coordinates of the new grid since all valid values should be between 0 and 300.

In order to get the top 10 most frequent areas we run a query which selects the most common coordinate combinations, of pickup and drop off, and their \textit{count()}, grouped by all the coordinates of a ride, orders them in descending order, of the \textit{count()}, and limit the top to 10 results.

\subsection{Question 2}

Question 2 is intended to: \textit{``Identify areas that are currently most profitable for taxi drivers"}.\par
For this, we start by dividing the map into a 600x600 grid. To identify the most profitable areas, we divided the profit made in that area by the number of empty taxis.
For each area we calculate the average profit in that area, where we consider the sum of fares plus the sum tips (using the aggregation function \textit{sum}) dividing by the number of trips (using the aggregation function \textit{count}), in each area in the last 15 minutes.\par
To calculate the number of empty taxis, we selected, for each taxi, its highest dropoff$_-$datetime, in the last 30 minutes, and for each area we count the taxis, using the aggregation function \textit{count}.\par
For the presentation of results, we sorted by average profit, descending, selecting only the first five results.

\subsection{Question 3}
Question 3 is intended to: \textit{``send an alert whenever the average idle time of taxis is greater than 10 minutes''}.

In our implementation we start by only selecting the medallion, to identify the taxi, and the pickup and drop off times for each event. In this selection we perform a parsing of the string value of the times to a timestamp using ``$time:timestampInMilliseconds(time_-string, format)$'', for example.

Next we use the ``arrow'' (-$>$) logical pattern in order to select the sequence of drop off and pickup times from taxis with the same medallion and use it to calculate the idle time between rides of the same taxi.

After that we perform a query over a time batch of 10 seconds so that we can see the results quickly and we select the computed average idle time, calculated with the function \textit{$avg()$} and we add a \textit{$having \ avg_-idle_-time > (10.0*60.0*1000.0)$} so that we only alert when the average idle time exceeds 10 minutes.

At last at the sink we provide an alert message which says that a 10 minute threshold was exceeded.

\subsection{Question 4}

Question 4 is intended to: \textit{``Detect congested areas"}.\par
For this query, as in query 2 we split the map into a 600x600 grid. To identify the most congested areas, we used a logical pattern, selecting the areas that had a sequence of four trips, all increasing in duration, accessing the values in \textit{trip$_-$time$_-$in$_-$secs}.\par
More details about this query in the Detailed Query Presentation section.

\subsection{Question 5}

Question 5 is intended to: \textit{``Select the most pleasant taxi drivers"}.\par
For this query, the solution was quite simple. To select the most pleasant taxi driver, we grouped by taxi driver, adding (using the aggregation function \textit{sum}) the tips received by the taxi driver on all trips that occurred in the time window. Then, sorting by the tip value in a descending way, just select the taxi driver that is at the top, using "\textit{limit 1}".


\documentclass[11pt]{article}

\usepackage{amsmath}
\usepackage{listings}
\usepackage{hyperref}

\setlength\parindent{0pt}

\begin{document}
    \title{HPCSE II - Exercise 4}
    \author{Anian Ruoss}
    \maketitle

    \section*{Task 1}
    \label{sec:Task1}

    As in Task 2 of Part 2 of Homework 3, we use the head2DSolver to model
    the temperature distribution on the steel sheets.
    For every torch we want to determine the optimal beam width, beam
    intensity and x- and y-coordinates meaning that we have 16 parameters in
    total for the 4 robotic torches.
    For all parameters we know the upper and the lower bounds given by:
    \begin{itemize}
        \item $x \in [0.0, 0.5]$ for torches 1 and 2
        \item $x \in [0.5, 1.0]$ for torches 3 and 4
        \item $y \in [0.0, 1.0]$ for all torches
        \item $\text{beam intensity} \in [0.4, 0.6]$ for all torches
        \item $\text{beam width} \in [0.04, 0.06]$ for all torches
    \end{itemize}
    Since we do not have any additional information about the parameter
    distributions, we model all parameters as being uniformly distributed
    within their respective bounds.
    We employ Korali's CMA-ES solver to maximize the posterior
    distribution of the parameters (even though we don't really have
    informative prior distributions, but the posterior was given in the
    template) to find the most likely parameter values and we display the
    results in Listing~\ref{lst:Task1}.

    \begin{lstlisting}[basicstyle=\tiny, frame=single, caption={Korali
    output when maximizing the posterior distribution of the parameters for
    the model with four robotic torches.}, label={lst:Task1}]
    [Korali] Starting CMAES. Parameters: 17, Seed: 0xFFFFFFFFFFFFFFFF
    ...
    [Korali] Finished - Reason: Object variable changes < 1.00e-06
    [Korali] Parameter 'Sigma' Value: 0.937023
    [Korali] Parameter 'torch_1_x' Value: 0.242895
    [Korali] Parameter 'torch_1_y' Value: 0.240862
    [Korali] Parameter 'torch_1_intensity' Value: 0.505957
    [Korali] Parameter 'torch_1_width' Value: 0.046153
    [Korali] Parameter 'torch_2_x' Value: 0.251464
    [Korali] Parameter 'torch_2_y' Value: 0.741347
    [Korali] Parameter 'torch_2_intensity' Value: 0.479662
    [Korali] Parameter 'torch_2_width' Value: 0.051501
    [Korali] Parameter 'torch_3_x' Value: 0.757994
    [Korali] Parameter 'torch_3_y' Value: 0.254470
    [Korali] Parameter 'torch_3_intensity' Value: 0.474448
    [Korali] Parameter 'torch_3_width' Value: 0.054619
    [Korali] Parameter 'torch_4_x' Value: 0.760434
    [Korali] Parameter 'torch_4_y' Value: 0.770577
    [Korali] Parameter 'torch_4_intensity' Value: 0.475872
    [Korali] Parameter 'torch_4_width' Value: 0.059926
    [Korali] Total Elapsed Time: 234.156253s
    \end{lstlisting}

    Korali performs the optimization in roughly 4 minutes.
    We know from the lecture that CMA-ES is embarrassingly parallel since we
    can compute and evaluate every sample independently, which can be
    achieved by parallelizing the two for-loops at lines 11 and 14.
    Typically the evaluation is a lot more costly than random
    number generation, which is why we should already observe a considerable
    speedup from only parallelizing the loop at line 14.

    \section*{Task 2}
    \label{sec:Task2}

    We run the single tasking engine and display its output in
    Listing~\ref{lst:Task2Single}.

    \begin{lstlisting}[basicstyle=\tiny, frame=single, caption={Output from
    executing the single tasking engine.}, label={lst:Task2Single}]
    Processing 240 Samples each with 2 Parameter(s)...
    Verification Passed
    Total Running Time: 29.717s
    \end{lstlisting}

    \subsection*{a)}
    \label{subsec:Task2a}

    Since all samples are well-known at the beginning, they can be distributed
    evenly among all ranks and gathered back to one rank once the evaluations
    are completed.
    We implement this divide-and-conquer strategy with UPC++ and MPI and
    display the results obtained from running the implementations with 24
    ranks on an Euler compute node in listings~\ref{lst:Task2aUPCXX}
    and~\ref{lst:Task2aMPI} respectively.

    \begin{lstlisting}[basicstyle=\tiny, frame=single, caption={Output from
    executing the UPC++ tasking engine with the divide-and-conquer strategy.},
    label={lst:Task2aUPCXX}]
    Verification Passed
    Total time:           1.37665
    Average time:         1.17792
    Load imbalance ratio: 0.144362
    \end{lstlisting}

    \begin{lstlisting}[basicstyle=\tiny, frame=single, caption={Output from
    executing the MPI tasking engine with the divide-and-conquer strategy.},
    label={lst:Task2aMPI}]
    Verification Passed
    Total time:           1.31242
    Average time:         1.17451
    Load imbalance ratio: 0.10508
    \end{lstlisting}

    We observe a speedup of $\approx 21.5$ for UPC++ and $\approx 22.5$ for
    MPI and thus we report efficiencies of $\approx 90\%$ for UPC++ and
    $\approx 94.5\%$ for MPI.\@
    Both implementations suffer from a relatively high load imbalance ratio
    ($\approx 0.145$ for UPC++ and $\approx 0.105$ for MPI) which results
    from the fluctuation in evaluation times and the static distribution
    schedule.
    The MPI approach is practically identical to that employed for the UPC++
    code but the collective operations in MPI allow for a much cleaner
    implementation (especially using \textit{MPI\_Scatter} since it eliminates
    the need to care about the global sample index of sample that is processed
    locally on a given rank).
    In general, MPI feels more natural since it requires very explicit
    communication.

    \subsection*{b)}
    \label{subsec:Task2b}

    To solve the load imbalance problem observed in Task 2a) we implement
    the producer-consumer strategy which takes advantage of the fact that the
    evaluation times differ and distributes workloads
    according to rank availability and not according to a fixed scheme.
    We display the results obtained from running the UPC++ and MPI
    implementations with 24 ranks on an Euler compute node in
    listings~\ref{lst:Task2bUPCXX} and~\ref{lst:Task2bMPI} respectively.

    \begin{lstlisting}[basicstyle=\tiny, frame=single, caption={Output from
    executing the UPC++ tasking engine with the producer-consumer strategy.},
    label={lst:Task2bUPCXX}]
    Processing 240 Samples each with 2 Parameter(s)...
    Verification Passed
    Total time:           1.32383
    Average time:         1.25486
    Load imbalance ratio: 0.0520984
    \end{lstlisting}

    \begin{lstlisting}[basicstyle=\tiny, frame=single, caption={Output from
    executing the MPI tasking engine with the producer-consumer strategy.},
    label={lst:Task2bMPI}]
    Verification Passed
    Total time:           1.32354
    Average time:         1.24246
    Load imbalance ratio: 0.0612577
    \end{lstlisting}

    We observe for both UPC++ and MPI that the load imbalance ratio drops
    significantly compared to the divide-and-conquer strategy, although more
    drastically for UPC++.
    For UPC++ we observe that the total time decreases slightly compared to
    the divide-and-conquer strategy (speedup: $\approx 22.5$, efficiency:
    $\approx 93.5\%$), whereas for the MPI implementation the total running
    time increases marginally (speedup: $\approx 22.5$, efficiency:
    $\approx 93.5\%$).
    We take this as evidence that the UPC++ implementation has successfully
    solved the load imbalance problem.
    Even though the load imbalance has also dropped for MPI, the
    producer-consumer strategy requires a lot more communication which explains
    the slightly higher running time.
    The MPI approach differs slightly from the UPC++ approach:
    \begin{itemize}
        \item UPC++ employs a queue of consumers which contain a future among
        other data.
        The producer iterates over the queue and checks whether a RPC has
        completed before distributing another sample to the idling rank.
        \item The MPI implementation does not require a queue as the
        producer just sends samples and listens for results until all
        samples have been evaluated.
        Unlike the UPC++ implementation, we need to explicitly tell every
        rank that the evaluation has completed once all samples have been
        processed.
    \end{itemize}
    In general one can say that UPC++ is more suited for the implementation
    of the producer-consumer problem.
    Nevertheless, the MPI approach feels cleaner as its communication is more
    explicit\footnote{From \emph{The Zen of Python, by Tim Peters}:
    ``Explicit is better than implicit.``.}.

    \section*{Task 3}
    \label{sec:Task3}

    We run the single tasking engine and display its output in
    Listing~\ref{lst:Task3Single}.

    \begin{lstlisting}[basicstyle=\tiny, frame=single, caption={Output from
    executing the single tasking engine.}, label={lst:Task3Single}]
    Processing 240 Samples (24 initially available), each with 2 Parameter(s)...
    Verification Passed
    Total Running Time: 29.458s
    \end{lstlisting}

    Since not all samples are available at the beginning of the generation
    it does not make sense to have more ranks than initially available
    samples and we enforce this constraint with an assert.
    Apart from that our approaches are similar to those from Task 2b) and we
    display the results obtained from running the UPC++ and MPI
    implementations with 24 ranks on an Euler compute node in
    listings~\ref{lst:Task3UPCXX} and~\ref{lst:Task3MPI} respectively.

    \begin{lstlisting}[basicstyle=\tiny, frame=single, caption={Output from
    executing the UPC++ tasking engine .},
    label={lst:Task3UPCXX}]
    Processing 240 Samples (24 initially available), each with 2 Parameter(s)...
    Verification Passed
    Total Running Time: 1.355s
    \end{lstlisting}

    \begin{lstlisting}[basicstyle=\tiny, frame=single, caption={Output from
    executing the MPI tasking engine .},
    label={lst:Task3MPI}]
    Verification Passed
    Total Running Time: 1.311s
    \end{lstlisting}

    We observe speedups of $\approx 21.5$ for UPC++ and $\approx 22.5$ for
    MPI and correspondingly efficiencies of $\approx 90.5\%$ for UPC++ and
    $\approx 93.5\%$ for MPI.\@
    For the UPC++ implementation we faced the challenge that
    \mbox{\textit{getSample()}} and \mbox{\textit{updateEvaluation()}}
    have to be called from the root rank and that
    \mbox{\textit{updateEvaluation()}} has to be called after the evaluation
    has completed on a consumer rank.
    This problem can be elegantly solved with the \textit{then()} method
    from UPC++.
    Our MPI implementation is basically equivalent to that of Task 2b) and
    thus we refer to Task 2b) for a discussion of the differences between
    the UPC++ and MPI approaches.
    Using the $then()$ function is definitely more elegant than sending data
    back and forth as is required for MPI.\@

    \section*{Task 4}
    \label{sec:Task4}

    To exploit full parallelism we increase the population size to 23 since
    we have 23 consumer ranks and 1 producer rank.
    We run Korali with the single conduit and display its output in
    Listing~\ref{lst:Task4Single}.

    \begin{lstlisting}[basicstyle=\tiny, frame=single, caption={Korali
    output for the single conduit when running the code from Task 1 with
    population size 23.}, label={lst:Task4Single}]
    [Korali] Starting CMAES. Parameters: 17, Seed: 0xFFFFFFFFFFFFFFFF
    ...
    [Korali] Finished - Reason: Object variable changes < 1.00e-06
    [Korali] Parameter 'Sigma' Value: 0.937782
    [Korali] Parameter 'torch_1_x' Value: 0.251375
    [Korali] Parameter 'torch_1_y' Value: 0.741905
    [Korali] Parameter 'torch_1_intensity' Value: 0.414076
    [Korali] Parameter 'torch_1_width' Value: 0.060000
    [Korali] Parameter 'torch_2_x' Value: 0.242771
    [Korali] Parameter 'torch_2_y' Value: 0.240310
    [Korali] Parameter 'torch_2_intensity' Value: 0.435900
    [Korali] Parameter 'torch_2_width' Value: 0.053473
    [Korali] Parameter 'torch_3_x' Value: 0.760406
    [Korali] Parameter 'torch_3_y' Value: 0.770098
    [Korali] Parameter 'torch_3_intensity' Value: 0.590264
    [Korali] Parameter 'torch_3_width' Value: 0.048075
    [Korali] Parameter 'torch_4_x' Value: 0.756812
    [Korali] Parameter 'torch_4_y' Value: 0.254229
    [Korali] Parameter 'torch_4_intensity' Value: 0.501190
    [Korali] Parameter 'torch_4_width' Value: 0.051603
    [Korali] Total Elapsed Time: 441.061222s
    \end{lstlisting}

    Our UPC++ and MPI approaches are very similar to those of tasks 2b) and
    3 and we display the results obtained from running the UPC++ and MPI
    implementations with 24 ranks on an Euler compute node in
    listings~\ref{lst:Task4UPCXX} and~\ref{lst:Task4MPI}.

    \begin{lstlisting}[basicstyle=\tiny, frame=single, caption={Korali
    output for the UPC++ conduit when running the code from Task 1 with
    population size 23.}, label={lst:Task4UPCXX}]
    [Korali] Starting CMAES. Parameters: 17, Seed: 0xFFFFFFFFFFFFFFFF
    ...
    [Korali] Finished - Reason: Object variable changes < 1.00e-06
    [Korali] Parameter 'Sigma' Value: 0.937782
    [Korali] Parameter 'torch_1_x' Value: 0.251375
    [Korali] Parameter 'torch_1_y' Value: 0.741905
    [Korali] Parameter 'torch_1_intensity' Value: 0.414076
    [Korali] Parameter 'torch_1_width' Value: 0.060000
    [Korali] Parameter 'torch_2_x' Value: 0.242771
    [Korali] Parameter 'torch_2_y' Value: 0.240310
    [Korali] Parameter 'torch_2_intensity' Value: 0.435900
    [Korali] Parameter 'torch_2_width' Value: 0.053473
    [Korali] Parameter 'torch_3_x' Value: 0.760406
    [Korali] Parameter 'torch_3_y' Value: 0.770098
    [Korali] Parameter 'torch_3_intensity' Value: 0.590264
    [Korali] Parameter 'torch_3_width' Value: 0.048075
    [Korali] Parameter 'torch_4_x' Value: 0.756812
    [Korali] Parameter 'torch_4_y' Value: 0.254229
    [Korali] Parameter 'torch_4_intensity' Value: 0.501190
    [Korali] Parameter 'torch_4_width' Value: 0.051603
    [Korali] Total Elapsed Time: 21.553523s
    \end{lstlisting}

    \begin{lstlisting}[basicstyle=\tiny, frame=single, caption={Korali
    output for the MPI conduit when running the code from Task 1 with
    population size 23.}, label={lst:Task4MPI}]
    [Korali] Starting CMAES. Parameters: 17, Seed: 0xFFFFFFFFFFFFFFFF
    ...
    [Korali] Finished - Reason: Object variable changes < 1.00e-06
    [Korali] Parameter 'Sigma' Value: 0.937782
    [Korali] Parameter 'torch_1_x' Value: 0.251375
    [Korali] Parameter 'torch_1_y' Value: 0.741905
    [Korali] Parameter 'torch_1_intensity' Value: 0.414076
    [Korali] Parameter 'torch_1_width' Value: 0.060000
    [Korali] Parameter 'torch_2_x' Value: 0.242771
    [Korali] Parameter 'torch_2_y' Value: 0.240310
    [Korali] Parameter 'torch_2_intensity' Value: 0.435900
    [Korali] Parameter 'torch_2_width' Value: 0.053473
    [Korali] Parameter 'torch_3_x' Value: 0.760406
    [Korali] Parameter 'torch_3_y' Value: 0.770098
    [Korali] Parameter 'torch_3_intensity' Value: 0.590264
    [Korali] Parameter 'torch_3_width' Value: 0.048075
    [Korali] Parameter 'torch_4_x' Value: 0.756812
    [Korali] Parameter 'torch_4_y' Value: 0.254229
    [Korali] Parameter 'torch_4_intensity' Value: 0.501190
    [Korali] Parameter 'torch_4_width' Value: 0.051603
    [Korali] Total Elapsed Time: 22.081863s
    \end{lstlisting}

    We observe that the results are identical for all three conduits and
    that UPC++ achieves a speedup of $\approx 20.5$ with efficiency
    $\approx 85.5\%$ whereas MPI achieves a speedup of $\approx 20$ with
    efficiency $\approx 83\%$.
    Both implementations thus exceed the requirement of being at least 10x
    faster while producing results similar to those of the single conduit,
    implying that the CEO of the company will give us a raise.
    As mentioned above, both approaches are similar to those of tasks 2b)
    and 3 and can thus be easily adapted to the current task.
    However, for UPC++ we faced the new challenge that the number of
    parameters is not given as a constant.
    Consequently, we cannot use the \textit{Consumer} struct from Task 3 as
    its sample array would have to be allocated dynamically, which in turn
    means that \textit{Consumer} is not trivially copyable anymore.
    With UPC++ we can solve this problem by simply using the local pointer of
    the global sampleArrayPointer to access the sample data from every
    consumer rank.
    The MPI implementation for this task is basically identical to that of
    tasks 2b) and 3.
    For this reason, we refer to Task 2b) for a discussion on the difference
    between the implementations.
    As for Task 3, it was relatively easy to adapt the implementations to
    the current task for both frameworks.
    The most difficult part of the MPI conduit implementation is the
    integration with Korali which is described below:

    \begin{itemize}
        \item load the required modules and set the environment variables
        \begin{lstlisting}[basicstyle=\footnotesize]
    module load new
    module load gcc/6.3.0
    module load intel/2018.1
    module load impi/2018.1.163

    export UPCXX_GASNET_CONDUIT=smp
    export UPCXX_THREADMODE=seq
    export UPCXX_CODEMODE=O3
    export KORALI_CONDUIT=single
        \end{lstlisting}
        \item replace single.cpp with mpi.cpp
        \item replace the Makefile in the conduits directory
        with this
        \href{https://github.com/anianruoss/HPCSE-II/blob/master/exercise04/task1/conduits/Makefile_MPI}{Makefile}
        \item replace the Makefile in the task1 directory with this
        \href{https://github.com/anianruoss/HPCSE-II/blob/master/exercise04/task1/Makefile_MPI}{Makefile}
    \end{itemize}

\end{document}


\documentclass[a4paper]{article}
\usepackage{amsmath}
\usepackage{amsthm}
\usepackage{amssymb}
\usepackage{amsfonts}
\usepackage{graphicx}
\usepackage[left=2cm,right=2cm,bottom=2.5cm,top=2.5cm]{geometry}
\usepackage{fancyhdr}
\usepackage{subcaption}
\usepackage{hyperref}
\usepackage{enumitem}
\usepackage{float}
\usepackage{breqn}
\usepackage{soul}
\setlength{\headheight}{14.5pt}

%Swap bullets for long dash in itemize
	\renewcommand\labelitemi{---}
	\renewcommand\qedsymbol{\(\blacksquare\)}

% Typesets derivatives nicely using roman d. Adapts size automatically. Stolen from https://tex.stackexchange.com/questions/135944/commath-and-ifinner/135985#135985
% \od{y}{x} is ordinary derivative, \pd{f}{x} is partial derivative. See commath documentation for others
	\usepackage{amsmath}
	\newcommand{\dif}{\mathop{}\!\mathrm{d}}
	\newcommand{\Dif}{\mathop{}\!\mathrm{D}}

	\makeatletter
	\newcommand{\spx}[1]{%
		\if\relax\detokenize{#1}\relax
		\expandafter\@gobble
		\else
		\expandafter\@firstofone
		\fi
		{^{#1}}%
	}
	\makeatother

	\newcommand\pd[3][]{\frac{\partial\spx{#1}#2}{\partial#3\spx{#1}}}
	\newcommand\tpd[3][]{\tfrac{\partial\spx{#1}#2}{\partial#3\spx{#1}}}
	\newcommand\dpd[3][]{\dfrac{\partial\spx{#1}#2}{\partial#3\spx{#1}}}

	\newcommand{\md}[6]{\frac{\partial\spx{#2}#1}{\partial#3\spx{#4}\partial#5\spx{#6}}}
	\newcommand{\tmd}[6]{\tfrac{\partial\spx{#2}#1}{\partial#3\spx{#4}\partial#5\spx{#6}}}
	\newcommand{\dmd}[6]{\dfrac{\partial\spx{#2}#1}{\partial#3\spx{#4}\partial#5\spx{#6}}}

	\newcommand{\od}[3][]{\frac{\dif\spx{#1}#2}{\dif#3\spx{#1}}}
	\newcommand{\tod}[3][]{\tfrac{\dif\spx{#1}#2}{\dif#3\spx{#1}}}
	\newcommand{\dod}[3][]{\dfrac{\dif\spx{#1}#2}{\dif#3\spx{#1}}}

	\newcommand{\genericdel}[4]{%
		\ifcase#3\relax
		\ifx#1.\else#1\fi#4\ifx#2.\else#2\fi\or
		\bigl#1#4\bigr#2\or
		\Bigl#1#4\Bigr#2\or
		\biggl#1#4\biggr#2\or
		\Biggl#1#4\Biggr#2\else
		\left#1#4\right#2\fi
	}
	\newcommand{\del}[2][-1]{\genericdel(){#1}{#2}}
	\newcommand{\set}[2][-1]{\genericdel\{\}{#1}{#2}}
	\let\cbr\set
	\newcommand{\sbr}[2][-1]{\genericdel[]{#1}{#2}}
	\let\intoo\del
	\let\intcc\sbr
	\newcommand{\intoc}[2][-1]{\genericdel(]{#1}{#2}}
	\newcommand{\intco}[2][-1]{\genericdel[){#1}{#2}}
	\newcommand{\eval}[2][-1]{\genericdel.|{#1}{#2}}
	\newcommand{\envert}[2][-1]{\genericdel||{#1}{#2}}
	\let\abs\envert
	\newcommand{\sVert}[1][0]{%
		\ifcase#1\relax
		\rvert\or\bigr|\or\Bigr|\or\biggr|\or\Biggr
		\fi
	}
	\newcommand{\enVert}[2][-1]{\genericdel\|\|{#1}{#2}}
	\let\norm\enVert
	\newcommand{\fullfunction}[5]{%
		\begin{array}{@{}r@{}r@{}c@{}l@{}}
			#1 \colon & #2 & {}\longrightarrow{} & #3 \\
			& #4 & {}\longmapsto{}     & #5
		\end{array}
	}

\title{Presentation Sketch Script}
\author{B. Han, T.M. Hodgson, M. Holden \& M. Puza}

% Define new commands for typesetting here
\newcommand{\R}{\mathbb{R}}
\renewcommand{\L}{\mathcal{L}}
\renewcommand{\P}{\mathbb{P}}
%\renewcommand{\epsilon}{\varepsilon}
\newcommand{\e}{\mathrm{e}}
\newcommand{\grad}{\nabla}
\newcommand{\E}{\mathbb{E}}

\begin{document}
\maketitle
\section{Introduction}
``Recent Advances in Langevin Monte Carlo'' \\
The aim of MCMC is to be able to accurately sample from an unknown probability distribution. When do we not know a distribution?
 \begin{itemize}
     \item Bayesian Statistics, only have some data and an incomplete idea of what the distribution looks like
     \item Molecular Dynamics
 \end{itemize}
 All MCMC methods aim to sample as accurately and efficiently as possible. We will focus on distributions of the form
 \[\pi(x) = \mathcal{Z}^{-1} \e^{-U(x)}\]
 This distribution goes by many names as it crops up in many areas:
 \begin{itemize}
     \item \st{Statisticians know it as a `log-linear model'} \textbf{REALLY? -- MH}
     \item Mathematicians often call it a `Gibbs measure'
     \item Physicists know it as the `Boltzmann distribution' or `canonical ensemble'
 \end{itemize}
 \st{In fact, the Hammersley-Clifford theorem states that any measure that satisfies a Markov property can be expressed as a Gibbs measure}. So although we only look at distributions of this form, it is not actually as much of a restriction as it initially appears. Borrowing the terminology from MD, we call \(U\) the potential function (\(\mathcal{Z}\) is a normalising constant). We know potential, not normaliser. Let's look pursue this molecular dynamics link to gain some intuition. How does this distribution arise?

 In MD, \(U\) describes the potential energy of a particle, for this reason we call \(U\) a potential well. (Picture of potential well). If we let a particle move in this well, this is what we see (animation?). The particle moves in the well, up and down each side.

 The equation of motion for a particle moving like this (Brownian dynamics) is the SDE
 \[\dif X_t = -\grad U(X_t) \dif t +\sqrt{2}\dif W_t\]
 the (overdamped) Langevin equation. A natural question to ask is where is this particle on average? It spends most of the time in the bottom of the well (can see that just by watching), sometimes makes it quite high up the sides. Shock horror, it is exactly given by \(\pi\)! This is what we call the invariant measure of the SDE (kind of).

\section{Algorithms}
 So back to statistics: the problem of statistical sampling has just been reduced to watching particles in wells! Bonzer! ... Except not quite. Just like in the ordinary differential equation case, one can't solve exactly in many cases, so we must discretise. Like in ODEs we use the Euler method!

 Applying this to the Langevin equation gives an iterative scheme to approximate the continuous diffusion (this is the MC from which MCMC gets its name).
 \[X_{n+1} = X_n - h\grad U(X_n)  + \sqrt{2h}Z_{n+1}\]
 Let's see what this looks like (trace plot of approximation? Or vis in well with a higher stepsize) This is only exact when the step size tends to zero, which we obviously can't do. This means that the invariant measure (the average positions of the particles) may not be the same as the continuous case. Or we may not even be able to find one! (Chain is transient/not ergodic)

 Example:

 Most extreme case: \(h=1, U(x)=x^2/2\). Obvious invariant measure is \(N(0,1)\), but chain (immediately) converges to \(N(0,2)\).

 So we have to be careful with stepsize. Also if the potential is superlinear (sides of the well are too steep), the iterative scheme diverges, even though the underlying diffusion is ergodic.

 Can we fix this? Yeeees, with Metropolis rejection (explain? Idea is to just reject some of the samples with some probability - show vis with rejects). Doing so means that whenever the underlying diffusion is exp ergodic?, the chain also will be (in the limit -- not actually useful - show vis.). Still runs in to problems when the potential is superlinear.

 So this is the direction that most research has taken since its initial proposal in 1996. We have been looking at a different method that avoids Metropolis rejection -- TAMING!

 \section{Taming the Beast}
 We have seen in the visualisation that \texttt{ULA} and \texttt{MALA} don't quite work. The issue is the gradient of the potential well. Taming swaps the gradient for a function that is `close' to the gradient, but preserves the invariant measure of the diffusion.
\[\dif X_t = -\grad U(X_t) \dif t +\sqrt{2}\dif W_t \Longrightarrow \dif X_t = -T(X_t)\dif t +\sqrt{2}\dif W_t \]
Different choices of \(T\) are available, we consider
\[T=\frac{\grad U }{1+h\grad U}\]
What does this do? It reduces the effect the gradient has on the iteration. Visualise.

Stiffness effects motivates coordinatewise? Show how taming deals with stiff problems in 2d. (Impacts ability in ill-conditioned problems).

One final modification we wish to mention is to use a different method of discretisation: Higher order (Stochastic Runge-Kutta) \texttt{tHOLA}, Leimkuhler-Matthews (Non-Markovian) \texttt{LM}. We don't have time to go into these methods here but ask at the end if you're interested in how they work.
\section{Horse Racing}
So we have given a heuristic (qualitative) motivation of how these algorithms are different. Can we quantify each methods effectiveness on different potentials/distributions?

The existing results on \texttt{tULA} only looked at first and second moments Brosse et al.. This is sufficient only for Gaussian distributions, which aren't even superlinear. We wish to get some stronger results in terms of metrics.

Shitty boxplots \(\longrightarrow\) Histograms? Wasserstein? KL? Computer time? Pick two or three plots that best illustrate errors, or use vis.

We also have some theoretical non-asymptotic error bounds in these metrics.

\section{Conclusion}
What algorithm is best? Probably problem dependent, depends what you want out of the alorithm. Taming will give you an answer, which may be inaccurate -- but at least it is an answer.
\subsection*{Next Steps}
There are many many methods we haven't considered: HMC, mMALA, SGLD. Discuss SGLD? So hot right now NeurIPS.
\subsubsection*{Future Work}
We'd like to test these methods on real data, or even just simulated data, when we don't know the gradient analytically. This is much more relevant for the end user than a theoretical bound on convergence. Especially wish to test on `big data' as many (all?) MCMC methods suffer the curse of dimensionality. Lower dimensions may not show off the benefits of taming, may only become relevant in high dimensions. Also for algorithms with very complex iteration steps, they may lend themselves to parallelisation  better than simpler algorithms.

Finally, we've made all the code used for testing available on github, as well as the visualisation.

\begin{center}Any questions? PLease say no\end{center}
\end{document}


\documentclass{article}
\usepackage[elliptic]{qed}

\begin{document}

\section*{Numerical fallback}

The Legendre relation is a relation among the elliptic integrals of the first
($\ellipk$) and second ($\ellipe$) kinds, given by
%
\begin{qed}
    \ellipe(k^2) \ellipk(1-k^2) + \ellipe(1 - k^2) \ellipk(k^2)
    - \ellipk(k^2) \ellipk(1 - k^2) = \frac{\pi}{2}
\end{qed}
%
Unfortunately, \textsf{sympy} doesn't know how to automatically simplify it,
so \QED cannot analytically determine its validity. Nevertheless,
when we typeset it, we still got a passing badge. That's because \QED falls
back to numerical evaluation automatically.

\end{document}


\documentclass[executivepaper]{extarticle}
\usepackage[margin=3.0cm]{geometry}
\usepackage{fancyhdr}
\usepackage{extsizes}
\usepackage{enumerate}
\usepackage{enumitem}
\usepackage{mathtools}

\begin{document}

\vspace*{-40mm}

\begin{center}

\textbf{Modern Class Notes}

\end{center}

\section*{1/12/16}

\vspace{-5.5mm}

\noindent \rule{2cm}{0.5pt}

A group is an object that tries to describe the symmetry of something \\

Going to study:

\begin{enumerate}

\item Permutation groups

\item Matrix groups

\item Cosets and Legrange's Theorem

\item And other constructs

\end{enumerate}

Review of some discrete math stuff: \\

\begin{enumerate}

\item A function $f: x \rightarrow y$ (where X and Y are sets) is an assignment of an element of Y for each element of X

\item Fact: let $f: x \rightarrow y$ be a function. Then:

\begin{enumerate}

\item $f$ is invertible $\iff f$ is one to one

\end{enumerate}

\end{enumerate}

Ex)Let x={1, 2, 3,...,n}\\

$\pi(1)=2$

$\pi(2)=3$

$\pi(n-1)=n$

$\pi(n)=1$\\

A permutation is a one-to-one and onto function from a finite set to itself \\

At least 3 ways of recording a permutation: \\

\begin{enumerate}

\item Two line notation

\item One line notation (works for permutations of order up to 9)

\item Cycle notation: start with a number and write down where it goes and then repeat this process for the rest of the numbers

\item

\end{enumerate}

\end{document}


% Created: Enze Chen, June 2017
% Last edited: Enze Chen, February 2018
%
% Chapter 4 of the MSE 142 coursereader. This chapter discusses quantum tunneling using the finite potential step/barrier. It derives the math behind boundary conditions and tunneling probability. Interference from multiple boundaries is also discussed. There is an emphasis on applications to really drive the point home that this phenomenon is cool and more ubiquitous than we might initially think!

% Uncomment the following three lines and last line to individually compile this chapter
%\documentclass[12pt, english]{book}
%\usepackage{142crstyle}
%\begin{document}

\chapter{Quantum Tunneling} \label{ch:tunnel}
%{ \doublespacing
Imagine you're tossing a tennis ball against the wall. Classical mechanics says that the ball will bounce back at you, or, should you throw it very hard, it might get stuck in the wall. But what if the ball somehow \emph{goes through} the wall and emerges on the other side, leaving both unscathed? This is the basis for \textbf{quantum tunneling}, a phenomenon where a particle passes through a potential barrier that it classically could not surmount. As bizarre as this may seem, quantum mechanics says there is a non-zero probability for tunneling to occur, and this phenomenon is actually leveraged for a variety of physical applications. Tunneling at the quantum scale occurs in physical systems ranging from photons\footnote{See D. D. Coon \href{http://aapt.scitation.org/doi/10.1119/1.1972893}{\emph{American Journal of Physics}} \textbf{34}, 240 (1966).} to water\footnote{See \href{https://physics.aps.org/articles/v9/43}{this focus article} from \emph{Physics} \textbf{9}, 43 about the work of A. I. Kolesnikov et al, \href{https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.116.167802}{\emph{Physical Review Letters}} \textbf{116}, 167802 (2016).} and we will spend this chapter exploring the nature of this truly quantum mechanical behavior.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Finite potential step}
To facilitate our analysis of this behavior, we need to first devise a simple model for a quantum mechanical system, which we have shown here in Figure~\ref{fig:tunnel-step}. On the left half of the figure in region I, where $x < 0$, we have a free particle ($V=0$) traveling to the right with energy $E$. At $x=0$, the particle encounters a \textbf{potential step} with some \emph{finite} potential $V=V_0$ that extends for all $x>0$ into region II.

\begin{figure}[!h]
	\centering
	\includegraphics[width=0.42\linewidth]{tunnel-step}
	\caption{A simple model to demonstrate quantum tunneling. A free particle with energy $E$ is traveling to the right and at $x=0$ is incident on a finite potential step with magnitude $V_0$.}
	\label{fig:tunnel-step}
\end{figure}

Note that we have not said anything yet about the relationship between $E$ and $V_0$ (the vertical position of the particle in Figure~\ref{fig:tunnel-step} was arbitrarily drawn). Since the finite potential step is time-independent, we will try to understand how quantum mechanical particles interact with such a potential by solving the time-independent \Sch\ equation. In region I, we only have the free particle, so we can write down the \Sch\ equation as
\begin{equation}
	-\frac{\hbar^2}{2m} \dv[2]{\Psi}{x} = E\Psi \label{eq:reg-1-se}
\end{equation}
In region II, we have
\begin{equation}
	-\frac{\hbar^2}{2m} \dv[2]{\Psi}{x} = (E-V_0)\Psi \label{eq:reg-2-se}
\end{equation}
where the energy of the particle is now with reference to the step's potential. We've seen these kinds of equations before and know that the general solution are sines and cosines, as was the case with the particle in a box. Here we write the general solutions in a slightly different, but equivalent way:
\begin{tcolorbox}[title = Traveling wave solutions] \vspace{-2ex}
	\begin{align}
		\Psi_{\text{I}} &= Ae^{ik_1x} + Be^{-ik_1x} \label{eq:reg-1-wf} \\
		\Psi_{\text{II}} &= Ce^{ik_2x} + De^{-ik_2x} \label{eq:reg-2-wf}
	\end{align}
\end{tcolorbox}
where $A$, $B$, $C$, and $D$ are unknown coefficients. Whoa! We know by Euler's formula that complex exponentials can be reformulated as sinusoidal functions, so these solutions at least seem plausible on the surface. To be sure we did our math right, let's plug Equation~\ref{eq:reg-1-wf} into Equation~\ref{eq:reg-1-se} to verify that it is a valid solution.
\begin{align*}
	-\frac{\hbar^2}{2m} \dv[2]{\Psi_{\text{I}}}{x} &= E\Psi_{\text{I}} \\
	-\frac{\hbar^2}{2m} \dv[2]{x}(Ae^{ik_1x} + Be^{-ik_1x}) &= E(Ae^{ik_1x} + Be^{-ik_1x}) \\
	-\frac{\hbar^2}{2m} \dv{x}(Aik_1e^{ik_1x} - Bik_1e^{-ik_1x}) &= E(Ae^{ik_1x} + Be^{-ik_1x}) \\
	\frac{\hbar^2}{2m} \left(Ak_1^2e^{ik_1x} + Bk_1^2e^{-ik_1x}\right) &= E(Ae^{ik_1x} + Be^{-ik_1x}) \\
	k_1^2\left(Ae^{ik_1x} + Be^{-ik_1x}\right) &= \frac{2mE}{\hbar^2}(Ae^{ik_1x} + Be^{-ik_1x})
\end{align*}

So we see that
\begin{equation}
	k_1 = \sqrt{\frac{2mE}{\hbar^2}} \label{eq:reg-1-k}
\end{equation}
which is the same expression we found previously and shows that Equation~\ref{eq:reg-1-wf} is a valid solution. In a similar vein, we see that the magnitude of the wave vector in region II is given by
\begin{equation}
	k_2 = \sqrt{\frac{2m(E-V_0)}{\hbar^2}} \label{eq:reg-2-k}
\end{equation}

Note that if $E > V_0$, then $k_2$ is real and we have sinusoidal behavior in region II. This makes sense because a particle with higher energy than the potential step should be able to surmount it and continue exhibiting wave-like behavior. However, if $E < V_0$, then $k_2$ becomes imaginary, and the wave-like solutions given in Equation~\ref{eq:reg-2-wf} turn into exponentially growing and decaying functions. Thus it seems like even though the particle does not have sufficient energy to surmount the potential step, there is still something happening in region II \emph{inside} the step. We will come back to this point shortly. \par

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Forward and backward waves}
Now, the general solutions given by Equation~\ref{eq:reg-1-wf} and~\ref{eq:reg-2-wf} are perfectly acceptable mathematically, but we argued in class that there is also a useful physical interpretation of these solutions. Recall that the de Broglie relation relates the wave vector $k$ to the momentum. Now we see that these wave function solutions not only resemble the plane wave solutions that we started with, but they also have a superposition of positive and negative wave vectors which determine the direction of propagation of the wave. We therefore argue that we can associate $A$ with the amplitude of the \textbf{incident wave} on the step (since the exponent has a positive $k_1$) and $B$ with the amplitude of the \textbf{reflected wave} (arising from a reflection at the interface). Similarly, we will associate $C$ with the amplitude of the \textbf{transmitted wave} through the step and $D$ with the amplitude of the wave incident on the step from the right. \par

In the example shown in Figure~\ref{fig:tunnel-step}, because we are only considering a wave incident on the step from the left, the coefficient $D$ must be zero. For the case when $E > V_0$, we still have a complex exponential function, so the wave function hasn't changed much. But for the case when $E < V_0$, the wave function in region II is only an exponentially damped function, which as we discussed in class is analogous to an evanescent wave in wave optics.\footnote{Indeed, evanescent wave coupling causes \href{https://en.wikipedia.org/wiki/Total_internal_reflection\#Frustrated_total_internal_reflection}{``frustrated'' total internal reflection}, which is a very similar behavior to quantum tunneling.} To see why this is the case, we can define a constant
\begin{equation}
	k' = \sqrt{\frac{2m(V_0-E)}{\hbar^2}} \label{eq:reg-2-kp}
\end{equation}
such that
\begin{equation*}
	k_2 = \sqrt{\frac{2m(E-V_0)}{\hbar^2}} = i\sqrt{\frac{2m(V_0-E)}{\hbar^2}} = ik'
\end{equation*}

This allows us to rewrite Equation~\ref{eq:reg-2-wf} as
\begin{equation}
	\Psi_{\text{II}} = Ce^{ik_2x} = Ce^{-k'x} \label{eq:reg-2-wfp}
\end{equation}

and the exponential decay function emerges. Our goal now is to extract the reflection coefficient $R$ and transmission coefficient $T$. We define these two quantities as follows:
\begin{equation*}
	\boxed{R = \abs{\frac{B}{A}}^2 \quad \text{and} \quad T = \frac{k_2}{k_1}\abs{\frac{C}{A}}^2}
\end{equation*}
% TODO: check with Aaron about this
Similar to the statistical interpretation of the wave function, we have taken the modulus squared to represent the probability of an event occurring. \par

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Continuity condition}
In order to relate the coefficients from the wave function in region I with those from the wave function in region II, we have to be precise and specify the behavior at the boundary of the two regions (i.e. at $x=0$). Let us consider what the conditions are for continuity of the wave function and its first derivative in a region where the potential is discontinuous (as it is at $x=0$). Starting from the time-independent \Sch\ equation, we integrate over an infinitesimal region $(-\epsilon, \epsilon)$ about $x=0$:
\begin{align*}
	-\frac{\hbar^2}{2m} \dv[2]{\Psi}{x} + V(x)\Psi &= E\Psi \\
	\dv[2]{\Psi}{x} &= \frac{2m}{\hbar^2}(V(x) - E)\Psi \\
	\int_{-\epsilon}^{\epsilon} \dv[2]{\Psi(x)}{x} \dd{x} &= \frac{2m}{\hbar^2} \int_{-\epsilon}^{\epsilon} (V(x)-E)\Psi(x) \dd{x}
\end{align*}

Now as $\epsilon$ approaches zero, the right hand side approaches zero if $V(x)$ and $\Psi(x)$ are not infinite (and there are good reasons why $\Psi$, from which one calculates the probability density function, cannot go to infinity).\footnote{Though $\Psi(x)$ cannot be infinite at the boundary, $V(x)$ still can. We'll save this for the next chapter when we discuss the Kronig-Penney model.} We therefore conclude, after performing the straightforward integral on the left hand side, that
\begin{tcolorbox}[title = Continuity of the wave function] \vspace{-2ex}
	\begin{equation}
		\dv{\Psi}{x} \bigg|_{\epsilon} - \dv{\Psi}{x} \bigg|_{-\epsilon} = 0 \label{eq:cont-dx}
	\end{equation}
\end{tcolorbox}

This means that across the discontinuity in the potential, the first derivative of the wave function (and by extension the wave function itself) must be continuous.\footnote{Continuity of the wave function is an important point, so if you need another explanation to internalize the ideas, \href{https://www.quora.com/Why-does-the-wave-function-have-to-be-continuous}{Quora} actually gives a pretty good answer.}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Tunneling probability}
Equation~\ref{eq:cont-dx} allows us to obtain the desired transmission and reflection coefficients as follows. At the boundary, we require that
\begin{align*}
	\Psi_{\text{I}}(0) &= \Psi_{\text{II}}(0) \\
	\Psi_{\text{I}}'(0) &= \Psi_{\text{II}}'(0)
\end{align*}
which directly translates into
\begin{align}
A + B &= C \label{eq:abc1} \\
Ak_1 - Bk_1 &= Ck_2 \label{eq:abc2}
\end{align}

We can solve the above system of equations in many ways, such as multiplying Equation~\ref{eq:abc1} by $k_1$ and adding the result to Equation~\ref{eq:abc2}. Doing so gives
\begin{equation}
	\frac{C}{A} = \frac{2k_1}{k_1+k_2} \label{eq:c-a}
\end{equation}
and a similar method gives
\begin{equation}
	\frac{B}{A} = \frac{k_1 - k_2}{k_1 + k_2} \label{eq:b-a}
\end{equation}

Finally, this gives
\begin{tcolorbox}[title = $T$ and $R$ for finite potential step] \vspace{-2ex}
	\begin{align}
		T = \frac{k_2}{k_1}\abs{\frac{C}{A}}^2 &= \frac{4k_1k_2}{(k_1+k_2)^2} \label{eq:t-step} \\
		R = \abs{\frac{B}{A}}^2 &= \frac{(k_1-k_2)^2}{(k_1+k_2)^2} \label{eq:r-step}
	\end{align}
\end{tcolorbox}
If you have taken a course in optics or acoustics these equations should look familiar---they are similar to the results for the transmission and reflection coefficients for an acoustic wave or light wave incident on a boundary. Now we're beginning to see how quantum mechanics predicts transmission with a non-zero probability. Note here that the way we have defined $T$ and $R$ also gives us the identity $T + R = 1$. \par

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Finite potential barrier} \label{sec:tunnel-barrier}
Having successfully managed the previous example, we're now ready to tackle the slightly more complicated case (and the one that we've been waiting for!) of quantum tunneling through a finite potential barrier of width $L$ and height $V_0$ (Figure~\ref{fig:tunnel-barrier}). We will carry out the following calculations in particular for the case $E < V_0$, which is the most interesting and surprising case.

\begin{figure}[!h]
	\centering
	\includegraphics[width=0.6\linewidth]{tunnel-barrier}
	\caption{In this example, a free particle with energy $E$ is traveling to the right and at $x=0$ is incident on a finite potential barrier with width $L$ and height $V_0$. The potential is otherwise zero in front of and behind the barrier.}
	\label{fig:tunnel-barrier}
\end{figure}

With three regions, we can now write down the solutions to the time-independent \Sch\ equation as
\begin{align}
	\Psi_{\text{I}} &= Ae^{ikx} + Be^{-ikx} \label{eq:barr-1} \\
	\Psi_{\text{II}} &= Ce^{k'x} + De^{-k'x} \label{eq:barr-2} \\
	\Psi_{\text{III}} &= Fe^{ikx} \label{eq:barr-3}
\end{align}

Here, we have defined $k = \sqrt{\frac{2m}{\hbar^2}(E)}$ and $k'=\sqrt{\frac{2m}{\hbar^2}(V_0-E)}$ such that $k'$ is a real number. As before, we note that Equation~\ref{eq:barr-2} models exponential growth and decay inside the barrier. We also argue that a particle incident only from the left means that the only term in region III is the transmitted wave with amplitude $F$ (Equation~\ref{eq:barr-3}). Now our goal is to extract the transmission coefficient $T = \abs{\frac{F}{A}}^2$.\footnote{Whereas previously we had an extra $k_2/k_1$ term, in this case the momentum of the wave in region I and region III are the same! It might seem weird, but tunneling doesn't cause the wave to actually lose energy.} In this case, the transmitted wave with amplitude $F$ encodes within it all the possible ways in which the wave can transmit through the barrier, including multiple reflections inside the barrier. All of these complicated possibilities are accounted for simply by enforcing the continuity equations at $x=0$ and $x=L$. This gives us the following set of equations:
\begin{align*}
	A + B &= C + D \\
	A - B &= (C - D)\frac{k'}{ik} \\
	Ce^{k'L} + De^{k'L} &= Fe^{ikL} \\
	Ce^{k'L} - De^{k'L} &= Fe^{ikL}\frac{ik}{k'}
\end{align*}

We will not go through the gory details here of solving this system of equations as it involves messy algebra and nothing more of substance, but feel free to consult Appendix~\ref{sec:tunnel-deriv} for all the steps. After combining these equations and eliminating variables appropriately, we obtain the following equation relating $F$ to $A$:
\begin{equation}
	4Aikk'e^{-ikL} = F \left[(k'+ik)^2e^{-k'L} - (k'-ik)^2e^{k'L}\right] \label{eq:tunnel-fa}
\end{equation}

In the limit where $k'L \gg 1$ (i.e. the barrier width is much larger than the effective wavelength of the particle inside the barrier), one can simplify Equation~\ref{eq:tunnel-fa} to obtain
\begin{equation}
	T = \abs{\frac{F}{A}}^2 = \frac{16k^2k^{\prime 2}e^{-2k'L}}{(k^{\prime 2}+k^2)^2} \label{eq:tunnel-prob}
\end{equation}

Since this expression for $T$ is dominated by the exponential term, it asymptotically approaches the very simple equation
\begin{tcolorbox}[title = Tunneling probability approximation when $E < V_0$] \vspace{-2ex}
	\begin{equation}
		T \approx e^{-2k'L} \label{eq:tunnel-approx}
	\end{equation}
\end{tcolorbox}

The above equation is a very useful approximate solution for the tunneling probability that is often applied in practice. For completeness, we will also give the exact solution for the tunneling probability below.
\begin{tcolorbox}[title = Tunneling probability exact solution when $E < V_0$] \vspace{-2ex}
	\begin{equation}
		T = \abs{\frac{F}{A}}^2 = \frac{1}{1 + \left(\frac{k^2+k^{\prime 2}}{2kk'}\right)^2 \sinh^2(k'L)} \label{eq:tunnel-prob-full}
	\end{equation}
\end{tcolorbox}

The exact solution has a maximum value of 1 when $k'L=0$ and rapidly decreases to 0 as $k'L$ increases. However, $T$ is still non-zero, which means there is a chance for the particle to tunnel through the barrier and emerge on the other side! This is shown schematically in Figure~\ref{fig:tunnel-wave}.\footnote{What we've plotted is the actual wave function amplitude. How would the probability density differ?} As discussed in class, for $L \approx \SI{1}{\nano\meter}$ and for typical electron-volt-scale potential barriers, the transmission probability is close to 1, indicating how important these kinds of effects become at the nanoscale. \par

\begin{figure}[!h]
	\centering
	\includegraphics[width=0.48\linewidth]{tunnel-wave}
	\caption{An illustration of the amplitude of the wave function in the three regions. It exponentially decays through the barrier but emerges from the right side with a lower amplitude and the same frequency (energy) as the wave function in the left region. A non-zero value of the wave function on the right indicates that the particle may exist past the barrier.}
	\label{fig:tunnel-wave}
\end{figure}

Furthermore, note how the wave function in region III differs from the wave function in region I. The amplitude is much smaller, which makes sense as there is a much lower probability that we will find the particle on the right side of the barrier. However, should tunneling occur, we can apply the conservation of energy to see that the particle has the same energy as before, and hence the frequency of the wave function is the same. It's easy to get probability and energy mixed up, but make sure you can draw the distinction.\footnote{An analogy to classical mechanics: A ball with a certain amount of kinetic energy that rolls up and over a hill will have the same amount of kinetic energy when it arrives back to ground level (assuming no friction).} \par

Though we did not derive it here, it is interesting to look at the transmission probability for the case when $E > V_0$, which is given by Equation~\ref{eq:tunnel-prob-res} and plotted in Figure~\ref{fig:tunnel-prob} as a function of energy.
\begin{equation}
	\boxed{T = \frac{1}{1 + \left(\frac{k^2-k_2^2}{2kk_2}\right)^2 \sin^2(k_2L)}} \label{eq:tunnel-prob-res}
\end{equation}

\begin{figure}[!h]
	\centering
	\includegraphics[width=0.35\linewidth]{tunnel-prob}
	\caption{Tunneling probability as a function of energy for the finite potential barrier when $E > V_0$. At characteristic energies, the tunneling probability reaches 1 due to resonance of the wave function in the region of the barrier.}
	\label{fig:tunnel-prob}
\end{figure}

In this case, the wave function displays sinusoidal behavior in region II instead of exponential behavior, and we have made the appropriate substitution as $k_2=ik'$. It may come as no surprise that if the energy of the particle was high enough, then the transmission probability approaches 1 (i.e. it's as if the barrier wasn't even there). However, as seen in Figure~\ref{fig:tunnel-prob}, total transmission is also achievable at lower values of $E$, which correspond to tunneling \textbf{resonance} when the wavelength of the particle coincides with the width of the barrier. This leads to a constructive interference effect analogous to what we observed in the double-slit experiment.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section[Application: Devices]{Application: Optical and electronic devices}
Once the theory behind quantum tunneling was formally developed, scientists suddenly realized just how ubiquitous the phenomenon was. It turns out quantum tunneling is necessary to explain the radioactive decay of nuclei, the fusion reactions inside stars, and even the mechanisms behind photosynthesis! Furthermore, scientists have leveraged quantum tunneling to create many new nanoscale devices, two of which will be discussed here.

\subsection{Quantum cascade laser}
The first device we will survey here, the \textbf{quantum cascade laser} (QCL), actually combines quantum tunneling with the quantum wells that we analyzed in the previous chapter. First demonstrated in 1994 using a mixture of GaInAs/AlInAs, QCLs are semi-conductor lasers comprised of a stack of quantum well heterostructures (Figure~\ref{fig:qcl}).

\begin{figure}[!h]
	\centering
	\includegraphics[width=0.55\linewidth]{qcl}
	\caption{Schematic of the quantum cascade laser, where periodic layers of varying thicknesses confine electrons into different energy levels. A population inversion causes electron transitions that lead to photon emission and subsequent tunneling of the electron to the next period in the structure. Reproduced from J. Faist et al. \href{http://science.sciencemag.org/content/264/5158/553}{\emph{Science}} \textbf{264}, 5158 (1994).}
	\label{fig:qcl}
\end{figure}

Instead of having a bulk structure, QCLs use a periodic series of layers of different materials and thicknesses to achieve a superlattice. This superlattice is capable of confining electrons into different discrete sub-bands across all of the quantum wells. When a \textbf{population inversion} is achieved, where more electrons are in an excited state than in the ground state, electrons will drop in energy and emit photons, which is how all lasers operate in general. What makes the QCL unique, in addition to its highly tunable layers (e.g. grown with molecular beam epitaxy), is that once an electron has transitioned to a lower energy level in one period of the structure, it can then tunnel to the next period in the heterostructure and undergo more transitions, thereby releasing more photons in a \emph{cascading} effect. The ability for electrons to tunnel through barriers and emit multiple photons leads to higher power output for QCLs than normal lasers, which could lead to advances in spectroscopy, sensing, and signaling.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Scanning tunneling microscope}
Our second nanoscale device, the \textbf{scanning tunneling microscope} (STM), is a more popular tool that you might have previously heard about. It was developed in 1981 by Gerd Binnig and Heinrich Rohrer at IBM, and a schematic detailing the principles of operation is shown in Figure~\ref{fig:stm}.

\begin{figure}[!h]
	\centering
	\includegraphics[width=0.5\linewidth]{stm}
	\caption{Schematic for how the scanning tunneling microscope operates. As the tip moves across the surface, the control unit (CU) applies an appropriate voltage $V_p$ to a piezoelectric arm $P_z$ and modulates its height such that the same tunneling current $J_T$ is maintained. As a result, the tip moves up and down to trace a material's topography, which is imaged at angstrom-scale resolution. Reproduced from G. Binnig et al. \href{https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.49.57}{\emph{Phys. Rev. Lett.}} \textbf{49}, 57 (1982).}
	\label{fig:stm}
\end{figure}

In order to detect small changes on a material's surface, the STM leverages \textbf{piezoelectricity} to control the three arms that guide a sharp tip that sits less than \SI{1}{\nano\meter} away from the surface.\footnote{For those unfamiliar with piezoelectricity, it is briefly explained by \href{http://hyperphysics.phy-astr.gsu.edu/hbase/Solids/piezo.html}{Rod Nave} at Georgia State University.} As $P_x$ and $P_y$ raster scan the tip across the surface, a control unit applies a small voltage ($\sim\SI{10}{\milli\volt}$) to cause electrons to tunnel between the tip and the surface. It measures the tunneling current $J_T$ that's produced and adjusts the height of the third arm $P_z$ to maintain the same current. The piezoelectric arms are so sensitive to changes in $J_T$ that the STM is capable of imaging the surface topography with a resolution of \SI{1}{\angstrom} (\SI{e-10}{\meter}) or less! Not only are scientists able to image \emph{individual atoms} with this technique, but they have been able to manipulate the atoms as well, forming unique nanostructures like the quantum corral in Figure~\ref{fig:corral} and even a stop-motion animated short film by the folks at IBM Research.\footnote{A video of ``A Boy and His Atom'' and more information can be found on the \href{http://www.research.ibm.com/articles/madewithatoms.shtml\#fbid=9xTkVKSpT3k}{IBM Research website}.}

\begin{figure}[!h]
	\centering
	\includegraphics[width=0.4\linewidth]{corral}
	\caption{A quantum corral constructed by arranging 48 Fe atoms in a ring on Cu(111) surface using a STM. Reproduced from M. F. Crommie et al. \href{http://science.sciencemag.org/content/262/5131/218}{\emph{Science}} \textbf{262}, 5131 (1993).}
	\label{fig:corral}
\end{figure}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%\subsection{Resonant tunneling diode}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Summary}
To recap, in this chapter we analyzed the peculiar phenomenon of quantum tunneling, whereby a particle incident on a finite potential barrier has a non-zero probability of penetrating the barrier and emerging on the other side. This holds true even if the particle carried less energy than the barrier's potential, which is prohibited by the laws of classical mechanics. In that case, we argued that the wave function of the particle exhibits exponential behavior inside the barrier and it must maintain continuity with the wave function on the outside. This allowed us to derive the transmission probability as a function of the particle's wave vector and the barrier width, and we typically use Equation~\ref{eq:tunnel-approx} to approximate this probability. Tunneling is a vital mechanism for many natural processes and scientists have leveraged its power to create novel nanoscale devices like the quantum cascade laser and scanning tunneling microscope. In the next chapter, we will solve the \Sch\ equation for yet another model, one that consists of periodic potential barriers that are infinitely tall and infinitely narrow.

%} % for doublespacing
%\end{document}