\chapter{Multiscale Entropy Theory}
\label{ch_entrop}
\index{entropy}
\section{Entropy and Image Restoration}

\subsection{Introduction}

The term ``entropy'' is due to Clausius (1865), and the concept of
entropy was introduced by Boltzmann into statistical mechanics
in order to measure the number of microscopic ways that a given macroscopic
state can be realized. Shannon (1948) \cite{ima:shannon48} founded the mathematical theory of
communication when he suggested that the information gained in a
measurement depends on the number of possible outcomes out of
which one is realized. Shannon also suggested that
the entropy can be used for maximization of the bit transfer rate under
a quality constraint. Jaynes (1957) \cite{entropy:jaynes57}
proposed to use the entropy measure
for radio interferometric image deconvolution,  in
order to select between a set of possible solutions that which contains the
minimum of information or, following his entropy definition, that
which has maximum entropy. In principle, the solution verifying such
a condition should be the most reliable. A great deal of work has been
carried out
in the last 30 years on the use of the entropy for the general problem
of data filtering and deconvolution
\cite{entropy:ables74,entropy:bontekoe94,entropy:burg67,entropy:frieden75,entropy:gull91,entropy:narrayan86,starck:pan96,entropy:skilling89,entropy:weir92,entropy:djafari94,entropy:djafari98}.

Traditionally information and entropy are determined from events and the
probability of their occurrence.  Signal and noise are basic building-blocks
of signal and data analysis in the physical sciences.  Instead of the
probability of an event, we are led to consider the probabilities of our
data being either signal or noise.

Observed data $Y$ in the physical sciences
are generally corrupted by noise, which is often additive and which
follows in many cases a Gaussian distribution, a Poisson distribution, or
a combination of both.  Other noise models may also be considered.
 Using Bayes' theorem to evaluate the probability distribution of the
realization of the original signal $X$,
knowing the data $Y$, we have

\begin{eqnarray}
 \mathrm{p}(X|Y) = \frac{\mathrm{p}(Y|X).\mathrm{p}(X)}{\mathrm{p}(Y)}
\label{eqn_bayes}
\end{eqnarray}
$\mathrm{p}(Y|X)$ is the conditional probability distribution of getting the data
$Y$ given an original signal $X$, i.e.\ it represents the distribution
of the noise. It is given, in the case of uncorrelated Gaussian
noise with variance $\sigma^2$, by:
\begin{eqnarray}
 \mathrm{p}(Y|X) = \mathrm{exp} \left\{
-\sum_{pixels} \frac{ (Y-X)^2}{2{\sigma}^2} \right\}
\label{eqn_proba}
\end{eqnarray}
The denominator in  equation \ref{eqn_bayes} is independent of $X$ and
 is considered as a constant (stationary noise).
$\mathrm{p}(X)$ is the a priori distribution
of the solution $X$. In the absence of any information on the solution
$X$ except its positivity, a possible course of action
is to derive the probability
of $X$ from its entropy, which is defined from information theory.

The main idea of information theory \cite{ima:shannon48} is to establish
a relation between the received information and the probability of
the observed event \cite{ima:bijaoui84}. If we note ${\cal I}(E)$ the information
related to the event $E$, and $p$ the probability of this
event happening, then we consider that
\begin{eqnarray}
%\cal{I}(E) =    f(p)
{\cal I}(E) = f(p)
\end{eqnarray}

Then we assume the two following principles:
\begin{itemize}
\item The information is a decreasing function of the probability. This
implies that the more information we have, the  less will be the probability
associated with one event.
\item Additivity of the information. If we have two independent events
$E_1$ and $E_2$, the information ${\cal I}(E)$ associated with the happening
of both is equal
to the addition of the information of each of them.
\begin{eqnarray}
{\cal I}(E) = {\cal I}(E_1) + {\cal I}(E_2)
\end{eqnarray}
\end{itemize}

Since $E_1$ (of probability $p_1$) and $E_2$ (of probability $p_2$) are
independent, then the probability of both happening is equal to the
product of $p_1$ and $p_2$.  Hence
\begin{eqnarray}
f(p_1 p_2) = f(p_1) + f(p_2)
\end{eqnarray}

Then we can say that the information measure is
\begin{eqnarray}
{\cal I}(E) = k \ln(p)
\end{eqnarray}
where k is a constant. Information must be positive, and $k$
is generally fixed at $-1$.

Another interesting measure is the mean information which is denoted
\begin{eqnarray}
H = - \sum_i p_i \ln(p_i)
\end{eqnarray}
This quantity is called the entropy of the system and was established by
Shannon in 1948 \cite{ima:shannon48}.

This measure has several properties:
\begin{itemize}
\item It is maximal when all events have the same probability
$p_i = 1/ N_e$ ($N_e$ being the number of events), and is equal to
$\ln(N_e)$. It is in this
configuration that the system is the most undefined.
\item It is minimal when one event is sure. In this case, the system is
perfectly known, and no information can be added.
\item The entropy is a positive, continuous, and symmetric function.
%\item The mean information, obtained in two steps, can be added.
\end{itemize}

If we know the entropy $H$ of the solution (the next section
describes different ways to calculate it),
we derive its probability by
\begin{eqnarray}
\mathrm{p}(X) = \mathrm{exp}(- \alpha H(X))
\label{info_prop}
\end{eqnarray}

Given the data, the most probable image is obtained by maximizing
$\mathrm{p}(X|Y)$. Taking the logarithm of equation \ref{eqn_bayes}, we
thus need to maximize
\begin{eqnarray}
 \ln (\mathrm{p}(X|Y))  = - \alpha  H(X) + \ln(\mathrm{p}(Y|X)) -
\ln(\mathrm{p}(Y))
\end{eqnarray}
The last term is a constant and can be omitted.
Then, in the case of Gaussian noise, the solution is found by minimizing
\begin{eqnarray}
J(X) = \sum_{pixels} \frac{{(Y-X)}^{2}}{2 {\sigma}^{2}} + {\alpha} H(X)
= \frac{{\chi}^2}{2} + {\alpha} H(X)
\label{eqn_j1}
\end{eqnarray}
which is a linear combination of two terms: the entropy of the signal,
and a quantity corresponding to ${\chi}^2$ in statistics measuring the
discrepancy between the data and the predictions of the model.
$\alpha$ is a parameter that can be viewed alternatively as
a Lagrangian parameter or a value fixing the relative weight between
the goodness-of-fit and the entropy H.

For the deconvolution problem, the object-data relation is given by the
convolution
\begin{eqnarray}
Y = P * X
\end{eqnarray}
where $P$ is the point spread function, and the solution is found (in the case
of Gaussian noise) by minimizing
\begin{eqnarray}
J(X) = \sum_{pixels} \frac{{(Y-P*X)}^{2}}{2 {\sigma}^{2}} + {\alpha} H(X)
\end{eqnarray}

The way the entropy is defined is fundamental, because from its definition
will depend the solution. The next section discusses the different approaches
which have been proposed in the past.

\clearpage
\newpage

\subsection{The concept of entropy}
\label{sect_entr}
We wish to estimate an unknown probability density $p(X)$ of the data.
 Shannon \cite{ima:shannon48}, in the framework of the information
 theory, has defined the entropy of an image $X$ by
\begin{eqnarray}
H_s(X) = - \sum_{k=1}^{N_b} p_k \log p_k
\end{eqnarray}
where  $X=\left\{X_1,.. X_N \right\}$ is an image
containing integer values, $N_b$ is number of possible values which can
take a given pixel $X_k$
(256 for a 8 bits image), and
 $p_k$ values are derived the histogram of $X$:
\begin{eqnarray}
p_k = {  \mbox{\#} X_j = k \over  N}
\end{eqnarray}
$\mbox{\#} X_j = k $ giving the number of pixels  $X_j = k$.

If the image contains floating values, it is possible to
to build up the histogram $L$ of values $L_i$, using
a suitable interval $\Delta$, counting up how many times $m_k$ each interval
$(L_k, L_k + \Delta)$ occurs among the N occurrences. Then the probability
that a data value belongs to an interval $k$ is $p_k = \frac{m_k}{N}$, and
each data value has a probability $p_k$.

The entropy is minimum and equal to zero when the signal is flat, and
increases when we have some fluctuations. Using this entropy in
equation~\ref{eqn_j1} leads to minimize:
\begin{eqnarray}
J(X) = \frac{{\chi}^2}{2} + {\alpha} H_s(X)
\label{eqn_j2}
\end{eqnarray}
It is a minimum entropy restoration method.

The trouble with this approach is that, because the number of occurrences is
finite, the estimate $p_k$ will be in error by an amount proportional
to $m_k^{-\frac{1}{2}}$~\cite{entropy:frieden91}. The error becomes significant when
$m_k$ is small. Furthermore this kind of entropy definition is not
easy to use for signal restoration, because the gradient of
equation~\ref{eqn_j2} is not easy to compute.
For these reasons, other
entropy functions are generally used. The main ones are:
\begin{itemize}
\item Burg \cite{entropy:burg67}:
\begin{eqnarray}
H_b(X) = -\sum_{k=1}^N \ln(X_k)
\end{eqnarray}
\item Frieden \cite{entropy:frieden75}:
\begin{eqnarray}
H_f(X) = -\sum_{k=1}^N X_k \ln(X_k)
\end{eqnarray}
\item Gull and Skilling \cite{entropy:gull91}:
\begin{eqnarray}
H_g(X) = \sum_{k=1}^N  X_k - M_k - X_k \ln({X_k \over M_k})
\end{eqnarray}
where $M$ is a given model, usually taken as a flat image
\end{itemize}
where $N$ is the number of pixels, and $k$ represents an index pixel.

Each of these entropies can be used, and they correspond to different
probability distributions that one can associate with
an image \cite{entropy:narrayan86}.
(See \cite{entropy:frieden75,entropy:skilling89} for descriptions).
The last definition of the entropy has the advantage of having a zero
 maximum when $X$ equals the model $M$.
All of these entropy measures are negative (if $X_k > 1$),
and maximum when the image is flat.
They are negative because an offset term is omitted which has no importance
for the minimization of the functional. The fact that we consider that
a signal has maximum information value when it is flat is evidently
a curious way to measure information. A consequence is that we must
now maximize the entropy if we want a smooth solution, and
the probability of $X$
must be redefined by:
\begin{eqnarray}
\mathrm{p}(X) = \mathrm{exp}(\alpha H(X))
\end{eqnarray}
The sign has been inverted
(see equation~\ref{info_prop}), which is natural if we want the best
solution to be the smoothest. These three entropies, above, lead to the
Maximum Entropy Method method (MEM), for which the
solution is found by minimizing
(for Gaussian noise)
\begin{eqnarray}
J(X) = \sum_{k=1}^N \frac{{(Y_k-X_k)}^{2}}{2 {\sigma}^{2}} - {\alpha} H(X)
\label{eqn_j3}
\end{eqnarray}

These different entropy functions  which have
been proposed for image restoration have the property of being maximal when
the image is flat, and of decreasing when we introduce some information.
So minimizing the information is equivalent to maximizing the entropy, and
this has led to the well known Maximum Entropy Method (MEM). For the Shannon
entropy (which is obtained from the histogram of the data),
this is the opposite. The entropy is null for a flat image, and increases
when the data contains some information. So, if the Shannon entropy were
used for restoration, this would lead to a Minimum Entropy Method.

In 1986, Narayan and Nityanda \cite{entropy:narrayan86}
compared several entropy functions,
 and finally
concluded by saying that all were comparable if they have good
properties, i.e.\ they enforce positivity, and they have a negative
second derivative which discourages ripple. They showed also that
results varied strongly with the background level, and
that these entropy functions produced poor results
for negative structures, i.e.\ structures under the background level
(absorption area in an image, absorption band in a spectrum, etc.), and
compact structures in the signal.
The Gull and Skilling entropy gives rise to
the difficulty of estimating a model.
Furthermore it has been shown \cite{entropy:bontekoe94}
that the solution is dependent on this choice.

The determination of the $\alpha$ parameter is also not an easy task and in
fact it is a very serious problem facing the maximum entropy method.
In the historic MAXENT algorithm of Skilling and Gull, the choice of $\alpha$
is such that it must satisfy the ad hoc constraint $\chi^2=N$ when
the deconvolution is achieved, $N$ being
 the number of degrees of freedom of the system i.e.\ the number of pixels
in image deconvolution problems.
But this choice systematically leads to an under-fitting of the data
 \cite{entropy:titterington85} which is clearly apparent for imaging problems
with little blurring. In reality, the $\chi^2$ statistic is expected to
vary in the range $N\pm\sqrt{2N}$ from one data realization to another.
 In the Quantified Maximum Entropy point of view \cite{entropy:skilling89}, the
optimum value of $\alpha$ is determined by including its probability
P($\alpha$) in Bayes' equation and then by maximizing the marginal
probability of having $\alpha$, knowing the data and the model $m$.
 In practice, a value of $\alpha$ which is too large gives a resulting image
which is too regularized
\index{regularization}
with a large loss of resolution.  A value which is too small
leads to a poorly regularized solution showing unacceptable artifacts.
Taking a flat model of the prior image softens the discontinuities
which may appear unacceptable for astronomical images often containing
stars and other point-like objects. Therefore the basic maximum entropy
method appears to be not very
appropriate for this kind of image which contains high and low spatial
frequencies
at the same time. Another point to be noted
is a ringing effect of the maximum entropy method algorithm,
producing artifacts around bright sources.

 To solve these problems while still using the maximum entropy concept, some
enhancements of the maximum entropy method have been proposed.
Noticing that neighboring pixels of reconstructed images with MAXENT
could have values differing a lot in expected flat regions \cite{entropy:charter89},
Gull and Skilling introduced the concepts of hidden image $S$ and intrinsic
 correlation function $C$ (Gaussian or cubic spline-like)
in the Preblur MAXENT algorithm.
\index{maximum entropy method}
\index{MEM}
\index{intrinsic correlation function}

 The ICF describes a minimum scale length of correlation in the desired
image $O$ which is achieved by assuming that
\begin{eqnarray}
 O=C*S
\end{eqnarray}
This corresponds to imposing a
minimum resolution on the solution $O$.
Since the hidden space image $S$ is not
spatially correlated, this can be regularized by the entropy
\begin{eqnarray}
H_g(h)=\sum_{k=1}^N  S_k - M_k - S_k \ln(\frac{S_k}{M_k})
\end{eqnarray}

Since in astronomical images many scale lengths are present, the
{\it Multi-channel Maximum Entropy Method}, developed by Weir
\index{maximum entropy method}
\index{MEM}
\cite{entropy:weir91,entropy:weir92}, uses a set of
ICFs having different scale lengths, each defining a channel. The
visible-space image is now formed by a weighted sum of
 the visible-space image
channels $O_j$:

\begin{eqnarray}
  O= \sum_{j=1}^{N_c} p_j O_j
\end{eqnarray}
where $N_c$ is the number of channels.
Like in Preblur MAXENT, each solution $O_j$ is supposed to be the result of the
convolution between a hidden image $S_j$ with a low-pass filter (ICF) $C_j$:
\begin{eqnarray}
O_j = C_j * S_j
\end{eqnarray}

But such a method has several drawbacks:
\begin{enumerate}
\item The solution depends on the width of the ICFs \cite{entropy:bontekoe94}.
\item There is no rigorous way to fix the weights $p_j$ \cite{entropy:bontekoe94}.
\item The computation time increases linearly with the number of pixels.
\item The solution obtained depends on the choice of the models $M_j$
($j = 1 \dots N_c$) which were chosen independently of the channel.
\end{enumerate}
\index{intrinsic correlation function}

In 1993, Bontekoe et al.\ \cite{entropy:bontekoe94} used a special application
of this method which
they called Pyramid Maximum Entropy on infrared image data.
\index{pyramid}
The pyramidal approach allows the user to have constant ICF width, and the
computation time is reduced. It is demonstrated \cite{entropy:bontekoe94} that
all weights can be fixed ($p_j = 1$ for each channel).

This method eliminates the first three drawbacks, and
gives better reconstruction of the sharp
and smooth structures. But in addition
to the two last drawbacks, a new one is added:
as the images $O_j$ have different sizes (due to the pyramidal approach),
\index{pyramid}
the solution $O$ is built by duplicating the pixels of
the subimages $O_j$ of each channel. This procedure is known to produce
artifacts due to the  appearance of high frequencies which are
incompatible with the
real spectrum of the true image $\hat{O}$.

However this problem can
be easily overcome by duplicating the pixels before convolving with the ICF,
or expanding the channels using linear
interpolation. Thus the introduction of the ``pyramid of resolution'' has
solved some problems and brought lots of improvements to the classic maximum
entropy method, but has also raised other questions. In order to derive the
model from a physical value, Pantin and Starck \cite{starck:pan96}
introduced the wavelet
transform, and defined entropy as follows:
\begin{eqnarray}
H(O) =  \frac{1}{\sigma_I^2}\sum_{j=1}^l \sum_{k=1}^{N_j} \sigma_j( w_{j,k}- M_{j,k}- |w_{j,k}|\ln{\frac{|w_{j,k}|}{M_{j,k}}})
\label{eqn_entr}
\end{eqnarray}
where $\sigma_I$ is the noise standard deviation in the data, $l$ is the number of scales, and
$N_j$ is the number of samples in the band $j$
($N_j = N$ for the \`a trous algorithm).
The multiscale entropy is the sum of the entropy at each scale.

The coefficients $w_{j,k}$ are wavelet coefficients, and we
take the absolute value of $w_{j,k}$ in this definition because the
values of $w_{j,k}$ can be positive or negative, and a negative signal
contains also some information in
the wavelet transform.

The advantage of such a definition of entropy is
 the fact we can use previous work concerning the wavelet transform and
image restoration \cite{starck:mur95_2,starck:sta94_1,starck:sta94_4}.
The noise behavior has already been studied in the wavelet transform
and we can estimate the standard deviation of the noise $\sigma_j$
at scale $j$. These estimates can be naturally introduced in our
models $m_j$
\begin{eqnarray}
M_{j,k} = k_{m} \sigma_j
\end{eqnarray}
 The model $M_j$ at scale $j$ represents
the value taken by a wavelet coefficient in the absence of any relevant
 signal and, in  practice, it must be a small value compared to
any significant signal value.
Following the Gull and Skilling procedure, we take $M_j$ as a fraction of the
noise because the value of
$\sigma_j$ can be considered as a sort of physical limit under which a signal
cannot be distinguished from the noise ($k_m = \frac{1}{100}$).

\subsection{Conclusion}
\label{sect_5pt}

As described above, many studies
have been carried out in order to improve the functional to be minimized.
But the question which should be raised is: what is a good entropy for
signal restoration?

Trying to answer, this corresponds to asking what is the information
in the signal. We first assume that a signal $X$ can be decomposed in
several components:
\begin{eqnarray}
 X = S + B + N
\end{eqnarray}
where $S$ is the signal of interest, $B$ lis the background, and $N$  the noise.

The entropy should verify the following criteria \cite{starck:sta98_2}:
{\bf
\begin{enumerate}
\item The information in a flat signal is zero ($S=0$, $N=0$ et $B=\mathrm{Cst}$).
\item The amount of information in a signal is independent of the background
($H(X)$ is independent of $B$).
\item The amount of information is dependent on the noise
($H(X)$ is dependent of $N$).
A given signal $X$ doesn't furnish the  same information if
the noise $N$ is high or small.
\item The entropy must work in the same way for a pixel which
has a value $B + \epsilon$, and
for a pixel which has a value $B - \epsilon$.
$H(X)$ must be a function of the absolute value of $S$ instead of $S$.
\item The amount of information is dependent on the correlation in the signal.
If the signal $S$  presents large features above the noise, it contains
a lot of information. By generating a new set of  data from $S$, by
randomly taking the pixel values in $S$, the large features will
evidently disappear, and this new signal will contain less information.
But the pixel values will be the same as in $S$.
\end{enumerate}
}
% \begin{figure}[htb]
% \centerline{
% \hbox{
% \psfig{figure=lenna256.ps,bbllx=1.8cm,bblly=12.9cm,bburx=14.5cm,bbury=25.5cm,width=8cm,height=8cm,clip=}
% \psfig{figure=scrambled_lenna.ps,bbllx=1.8cm,bblly=12.9cm,bburx=14.5cm,bbury=25.5cm,width=8cm,height=8cm,clip=}
% }}
% \caption{Lena image (left) and the same data distributed differently (right).
% These
% two images have the same entropy, using any of the standard entropy methods.}
% \label{fig_lenna}
% \end{figure}
\begin{figure}[h]
\centerline{
\vbox{
\hbox{
\psfig{figure=fig_saturn.ps,bbllx=1.7cm,bblly=12.9cm,bburx=11.2cm,bbury=25.6cm,width=9.cm,height=12.3cm,clip=}
\psfig{figure=fig_saturn_scramble.ps,bbllx=1.7cm,bblly=12.9cm,bburx=11.2cm,bbury=25.6cm,width=9.cm,height=12.3cm,clip=}
% \vspace{21cm}
}
}}
\caption{Saturn image (left) and the same data distributed differently (right).
These
two images have the same entropy, using any of the standard entropy
definitions.}
\label{fig_saturn}
\end{figure}

Fig.~\ref{fig_saturn} illustrates the last point perfectly.
The second image is obtained  by distributing randomly the Saturn image pixel
values, and the standard entropy definitions  produce the same information
measurement for both images. The concept of information becomes really
subjective, or at least it depends on the application domain. Indeed, for
someone who is
not involved in image processing, the second image contains
{\em less} information
than the first one. For someone working on image transmission, it is clear
that the second image will require more bits for lossless transmission,
and from this point of view, he/she will consider that the second
image contains
{\em more} information. Finally, for data restoration, all fluctuations
due to noise are not of interest, and do not contain relevant
information. From this physical point of view,
the standard  definition of entropy seems badly adapted to information
measurement in signal restoration.

% \clearpage
% \newpage


\documentclass[simplex.tex]{subfiles}
% NO NEED TO INPUT PREAMBLES HERE
% packages are inherited; you can compile this on its own

\begin{document}
\subsection{Law of Large Graphs}
We note that low-rank methods can often be more easily interpreted. Moreover, eigenmode is observed among the embedded latent positions, with respect to different lobes in particular. This suggests to use low-rank methods from another perspective.
For all the 70 different regions based on the Desikan atlas (35 for each hemisphere), each one is assigned to one of the 10 lobes (5 for each hemisphere), i.e. Frontal, Parietal, Occipital, Temporal, and other. And we do a permutation test as following.

70 vertices are connected spatially as in the Desikan atlas. Let the adjacency matrix of these 70 vertices to be $A$. $A_{ij} = 1$ means vertex $i$ and vertex $j$ are spatially connected. We say vertex $j$ is a neighbor of vertex $i$ if $A_{ij} = 1$. We define $l_i$ be the lobe i.d. for vertex $i$.

We define a uniform $1$-flip to be:
\begin{itemize}
\item Select a pair of adjacent vertices (vertex $i_1$ and vertex $j_1$) across the boundary of lobes uniformly, i.e. $A_{i_1 j_1} = 1$ and $l(i_1) \ne l(j_1)$;
\item Uniformly select another pair of adjacent vertices (vertex $i_2$ and vertex $j_2$ where $i_1 \ne i_2$ and $j_1 \ne j_2$) across the same boundary of lobes uniformly, i.e. $A_{i_2 j_2} = 1$ and $l(i_1) = l(i_2)$ and $l(j_1) = l(j_2)$;
\item Reassign vertex $j_1$ to lobe $l_{i_1}$ and reassign vertex $i_2$ to lobe $l_{j_2}$.
\end{itemize}

By the definition, after a uniform $1$-flip, the number of vertices in each lobe keeps the same, where only two vertices are changed to a different lobe.

We define a uniform $k$-flip to be:
\begin{itemize}
\item Sequentially run the uniform $1$-flip $k$ times.
\end{itemize}

Note that after a uniform $k$-flip, the number of vertices in each lobe still keeps the same.

Let $X = [X_1, \cdots, X_n]^{\top}$ be the latent positions, where $X_i$ is the latent position for vertex $i$ sampled from distribution $f$. Test statistic $T(X, l)$ is defined as:
\[
T(X, l) = \frac{\sum_{i \ne j, l(i) = l(j)} \|X_i - X_j \|_2}{\sum_{i \ne j, l(i) = l(j)} 1} -
\frac{\sum_{i \ne j, l(i) \ne l(j)} \|X_i - X_j \|_2}{\sum_{i \ne j, l(i) \ne l(j)} 1}
\]

$H_0$: Differences between latent positions within lobes are the same compared to across lobes, i.e. $E_f[T(X, l)] = 0$.

$H_A$: Differences between latent positions within lobes are smaller compared to across lobes, i.e. $E_f[T(X, l)] < 0$.

We ran 1000 simulations for each number of flips and plot the results for the permutation test as in Figure~\ref{fig:violin}. The x-axis represents the different number of flips, while the y-axis represents the measure according to the lobe assignment, i.e. within lobes distances minus the across lobe distances. The dashed line is the baseline for the measure based on the true lobe assignment without any flipping. As the number of flips increases, we can see a clear evidence that $H_0$ is rejected with respect to $H_A$. Thus the embedded latent positions based on the low-rank method reflect the eigenmode with respect to different lobes.

\begin{figure}
\begin{cframed}
\centering
\includegraphics[height=.8\linewidth]{../../figs/violinplot_new_flip_2norm_1_8.pdf}
\caption{\textbf{Violin plot of the permutation test.} We ran 1000 simulations for each number of flips. The x-axis represents the different number of flips, while the y-axis represents the measure according to the lobe assignment, i.e. within lobes distances minus the across lobe distances. The dashed line is the baseline for the measure based on the true lobe assignment without any flipping. As the number of flips increases, we can see a clear evidence that $H_0$ is rejected with respect to $H_A$. Thus the embedded latent positions based on the low-rank method reflect the eigenmode with respect to different lobes.}
\label{fig:violin}
\end{cframed}
\end{figure}

%
\clearpage
\end{document}


% This is part of the TFTB Reference Manual.
% Copyright (C) 1996 CNRS (France) and Rice University (US).
% See the file refguide.tex for copying conditions.

\markright{tfrrppag}
\section*{\hspace*{-1.6cm} tfrrppag}

\vspace*{-.4cm}
\hspace*{-1.6cm}\rule[0in]{16.5cm}{.02cm}
\vspace*{.2cm}

{\bf \large \sf Purpose}\\
\hspace*{1.5cm}
\begin{minipage}[t]{13.5cm}
Reassigned pseudo Page time-frequency distribution.
\end{minipage}
\vspace*{.5cm}

{\bf \large \sf Synopsis}\\
\hspace*{1.5cm}
\begin{minipage}[t]{13.5cm}
\begin{verbatim}
[tfr,rtfr,hat] = tfrrppag(x)
[tfr,rtfr,hat] = tfrrppag(x,t)
[tfr,rtfr,hat] = tfrrppag(x,t,N)
[tfr,rtfr,hat] = tfrrppag(x,t,N,h)
[tfr,rtfr,hat] = tfrrppag(x,t,N,h,trace)
\end{verbatim}
\end{minipage}
\vspace*{.5cm}

{\bf \large \sf Description}\\
\hspace*{1.5cm}
\begin{minipage}[t]{13.5cm}
        {\ty tfrrppag} computes the pseudo Page distribution and its
        reassigned version. The reassigned pseudo Page distribution is
        given by the following expressions\,:
\begin{eqnarray*}
\hspace*{-.5cm}PP_x^{(r)}(t',\nu';h)=\iint_{-\infty}^{+\infty}
PP_x(t,\nu;h)\ \delta(t'-\hat{t}(x;t,\nu))\
\delta(\nu'-\hat{\nu}(x;t,\nu))\ dt\ d\nu,
\end{eqnarray*}
where
\begin{eqnarray*}
\hat{t}(x;t,\nu)=t\ \ \mbox{ and }\ \
\hat{\nu}(x;t,\nu)=\nu+\Im\left\{\dfrac{F_x(t,\nu;\ens{D}_h)\ F_x^*(t,\nu;h)}
{2\pi|F_x(t,\nu;h)|^2}\right\}.
\end{eqnarray*}
$\ens{D}_h(t)=\frac{dh}{dt}(t)$ and $F_x(t,\nu;h)$ is the short-time
Fourier transform of $x(t)$ with a causal analysis window $h(t)$.\\

\hspace*{-.5cm}\begin{tabular*}{14cm}{p{1.5cm} p{8cm} c}
Name & Description & Default value\\
\hline
        {\ty x}     & analyzed signal ({\ty Nx=length(x)})\\
        {\ty t}     & time instant(s)          & {\ty (1:Nx)}\\
        {\ty N}     & number of frequency bins & {\ty Nx}\\
        {\ty h}     & frequency smoothing window, {\ty h(0)} being forced to {\ty 1}
                                         & {\ty window(odd(N/4))}\\
        {\ty trace} & if nonzero, the progression of the algorithm is shown
                                         & {\ty 0}\\
     \hline {\ty tfr, rtfr} & time-frequency representation and its reassigned
            version\\
        {\ty hat}   & complex matrix of the reassignment vectors\\

\hline
\end{tabular*}
\vspace*{.5cm}

When called without output arguments, {\ty tfrrpmh} runs {\ty tfrqview}.
\end{minipage}

%\newpage

{\bf \large \sf Example}
\begin{verbatim}
         sig=fmlin(128,.1,.4);
         h=window(65,'gauss');
         tfrrppag(sig,1:128,128,h,1);
\end{verbatim}
\vspace*{.5cm}

{\bf \large \sf See Also}\\
\hspace*{1.5cm}
\begin{minipage}[t]{13.5cm}
all the {\ty tfr*} functions.
\end{minipage}
\vspace*{.5cm}

{\bf \large \sf Reference}\\
\hspace*{1.5cm}
\begin{minipage}[t]{13.5cm}
[1] F. Auger, P. Flandrin ``Improving the Readability of Time-Frequency and
Time-Scale Representations by the Reassignment Method'' IEEE Transactions
on Signal Processing, Vol. 43, No. 5, pp. 1068-89, 1995.
\end{minipage}


\documentclass{article}
\usepackage{graphicx}
\usepackage{titletoc}
\usepackage{titlesec}
\usepackage{geometry}
\usepackage{fontspec, xunicode, xltxtra}
\usepackage{float}
\usepackage{cite}
\usepackage{amsmath}
\usepackage{listings}
\usepackage{titletoc}

\geometry{left=3cm,right=3cm,top=3cm,bottom=3cm}
\DeclareMathOperator*{\argmin}{argmin}
\DeclareMathOperator*{\argmax}{argmax}
\DeclareMathOperator*{\var}{var}
\DeclareMathOperator*{\expec}{E}

\begin{document}
\title{\textsf{Homework 1 for Pattern Recognition}}
\author{Fan JIN\quad (2015011506)}
\maketitle

\section*{Question 1}
{
    It is equivalent to prove $R^2 = r^2$ and to prove $$\sum_1^n (\bar{y} - \hat{y}_i)^2 \cdot \sum_1^n (\bar{x} - x_i)^2 = (\sum_1^n (\bar{x} - x_i)(\bar{y} - y_i))^2,$$

    With $\hat{y}_i = b x_i + \bar{y} - b \bar{x}$, we have the left-hand side
    $$\sum_1^n (\bar{y} - \hat{y}_i)^2 \cdot \sum_1^n (\bar{x} - x_i)^2 = b^2 \cdot (\sum_1^n (\bar{x} - x_i)^2)^2.$$

    Since the regression coefficient $$b = \frac{\sum_1^n (\bar{x} - x_i)(\bar{y} - y_i)}{\sum_1^n (\bar{x} - x_i)^2},$$ plug it in and we find the left-hand side equals the right-hand side. Thus, the equation above is proved.
}

\section*{Question 2}
{
    \subsection*{Case 1: $N=10$, $\sigma = 0.5$}
    {
        Linear model: $$\hat{y} = 5.9758 + 3.0841x ,\quad R^2 = 0.9963$$
        Quadratic model: $$\hat{y} = 5.9826 + 3.0843x - 0.0047x^2 ,\quad R^2 = 0.9963$$
        Cubic model: $$\hat{y} = 5.9864 + 2.9573x + 0.0045x^2 + 0.0356x^3 ,\quad R^2 = 0.9965$$

        \begin{figure}[H]
            \centering
            \includegraphics[width = 0.6\linewidth]{2-2-linear-10-0.5.png}
            \caption{Linear}
        \end{figure}

        \begin{figure}[H]
            \centering
            \includegraphics[width = 0.6\linewidth]{2-2-quadratic-10-0.5.png}
            \caption{Quadratic}
        \end{figure}

        \begin{figure}[H]
            \centering
            \includegraphics[width = 0.6\linewidth]{2-2-cubic-10-0.5.png}
            \caption{Cubic}
        \end{figure}

        Linear model (on testing set): $$\mathrm{RSS} = 7.4800, \quad \mathrm{RSS}/\mathrm{TSS} = 0.0018$$

        Quadratic model (on testing set): $$\mathrm{RSS} = 7.4650, \quad \mathrm{RSS}/\mathrm{TSS} = 0.0017$$

        Cubic model (on testing set): $$\mathrm{RSS} = 7.2292, \quad \mathrm{RSS}/\mathrm{TSS} = 0.0017$$
    }

    \subsection*{Case 2: $N=100$, $\sigma = 0.5$}
    {
        Linear model: $$\hat{y} = 6.0152 + 2.9775x ,\quad R^2 = 0.9939$$
        Quadratic model: $$\hat{y} = 6.0469 + 2.9708x - 0.0311x^2 ,\quad R^2 = 0.9941$$
        Cubic model: $$\hat{y} = 6.0504 + 2.9986x - 0.0367x^2 - 0.0109x^3 ,\quad R^2 = 0.9941$$

        \begin{figure}[H]
            \centering
            \includegraphics[width = 0.6\linewidth]{2-2-linear-100-0.5.png}
            \caption{Linear}
        \end{figure}

        \begin{figure}[H]
            \centering
            \includegraphics[width = 0.6\linewidth]{2-2-quadratic-100-0.5.png}
            \caption{Quadratic}
        \end{figure}

        \begin{figure}[H]
            \centering
            \includegraphics[width = 0.6\linewidth]{2-2-cubic-100-0.5.png}
            \caption{Cubic}
        \end{figure}

        Linear model (on testing set): $$\mathrm{RSS} = 6.6353, \quad \mathrm{RSS}/\mathrm{TSS} = 0.0014$$

        Quadratic model (on testing set): $$\mathrm{RSS} = 6.5180, \quad \mathrm{RSS}/\mathrm{TSS} = 0.0014$$

        Cubic model (on testing set): $$\mathrm{RSS} = 6.8288, \quad \mathrm{RSS}/\mathrm{TSS} = 0.0015$$
    }

    \subsection*{Case 3: $N=10$, $\sigma = 2$}
    {
        Linear model: $$\hat{y} = 5.6122 + 4.3460x ,\quad R^2 = 0.6779$$
        Quadratic model: $$\hat{y} = 5.7220 + 4.3487x - 0.0745x^2 ,\quad R^2 = 0.6783$$
        Cubic model: $$\hat{y} = 5.7827 + 2.3172x + 0.0726x^2 + 0.5700x^3 ,\quad R^2 = 0.6967$$

        \begin{figure}[H]
            \centering
            \includegraphics[width = 0.6\linewidth]{2-2-linear-10-2.0.png}
            \caption{Linear}
        \end{figure}

        \begin{figure}[H]
            \centering
            \includegraphics[width = 0.6\linewidth]{2-2-quadratic-10-2.0.png}
            \caption{Quadratic}
        \end{figure}

        \begin{figure}[H]
            \centering
            \includegraphics[width = 0.6\linewidth]{2-2-cubic-10-2.0.png}
            \caption{Cubic}
        \end{figure}

        Linear model (on testing set): $$\mathrm{RSS} = 1914, \quad \mathrm{RSS}/\mathrm{TSS} = 0.3002$$

        Quadratic model (on testing set): $$\mathrm{RSS} = 1911, \quad \mathrm{RSS}/\mathrm{TSS} = 0.2996$$

        Cubic model (on testing set): $$\mathrm{RSS} = 1850, \quad \mathrm{RSS}/\mathrm{TSS} = 0.2901$$
    }

    \subsection*{Case 4: $N=100$, $\sigma = 2$}
    {
        Linear model: $$\hat{y} = 6.2433 + 2.6398x ,\quad R^2 = 0.3337$$
        Quadratic model: $$\hat{y} = 6.7499 + 2.5323x - 0.4973x^2 ,\quad R^2 = 0.3527$$
        Cubic model: $$\hat{y} = 6.8062 + 2.9781x - 0.5878x^2 - 0.1743x^3 ,\quad R^2 = 0.3560$$

        \begin{figure}[H]
            \centering
            \includegraphics[width = 0.6\linewidth]{2-2-linear-100-2.0.png}
            \caption{Linear}
        \end{figure}

        \begin{figure}[H]
            \centering
            \includegraphics[width = 0.6\linewidth]{2-2-quadratic-100-2.0.png}
            \caption{Quadratic}
        \end{figure}

        \begin{figure}[H]
            \centering
            \includegraphics[width = 0.6\linewidth]{2-2-cubic-100-2.0.png}
            \caption{Cubic}
        \end{figure}

        Linear model (on testing set): $$\mathrm{RSS} = 1698, \quad \mathrm{RSS}/\mathrm{TSS} = 0.3049$$

        Quadratic model (on testing set): $$\mathrm{RSS} = 1668, \quad \mathrm{RSS}/\mathrm{TSS} = 0.2995$$

        Cubic model (on testing set): $$\mathrm{RSS} = 1748, \quad \mathrm{RSS}/\mathrm{TSS} = 0.3138$$
    }

    \subsection*{Conclusion}
    {
        \begin{itemize}
            \item (\emph{Complexity}) For $\sigma = 0.5$ (small), the data have good linearity, and therefore, the quadratic and the cubic terms are close to 0, which means the linear model is adequate to explain the data. For $\sigma = 2$ (large), however, the linear term has a small impact on the response, compared to the error term. In this circumstance, the quadratic and the cubic term is not neglegible.

            \item For the 4 linear models, we have the following two tables:

            \begin{table}[!hbp]
                \begin{tabular}{|c|c|c|}
                \hline
                . & $N=10$ & $N=100$ \\
                \hline
                $\sigma=0.5$ & 0.9963 & 0.9939 \\
                \hline
                $\sigma=2$ & 0.6779 & 0.3337 \\
                \hline
                \end{tabular}
                \caption{$R^2$ on training set}
            \end{table}

            \begin{table}[!hbp]
                \begin{tabular}{|c|c|c|}
                \hline
                . & $N=10$ & $N=100$ \\
                \hline
                $\sigma=0.5$ & 0.9982 & 0.9986 \\
                \hline
                $\sigma=2$ & 0.6998 & 0.6951 \\
                \hline
                \end{tabular}
                \caption{$R^2$ on testing set}
            \end{table}

            We see the the $R^2$ on training set decreases dramatically with $N$ when $\sigma$ is large.

            \item The training $R^2$ and the testing $R^2$ are different if they are based on different $N$s. Also, there is some stocasticity rooted in this experiment, which depends on the random seed.

        \end{itemize}
    }
}

\section*{Question 3}
{
    \subsection*{Without interactive terms}
    {
        $$\hat{y} = 726.0731 - 0.7537x_1 - 161.5401x_2 + 61.4084x_3, \quad R^2 = 0.2304$$.
        Plug in $x = (110, 3, 1)$, one has $$\hat{y} = 219.9584$$.
    }

    \subsection*{With interactive terms}
    {
        $$\hat{y} = 1929.531 - 4.7578x_1 - 924.4123x_2 + 3.6749x_3 + 2.5331x_1 x_2 - 137.5951x_2 x_3 - 0.2866x_3 x_1, \quad R^2 = 0.6050$$.
        Plug in $x = (110, 3, 1)$, one has $$\hat{y} = -607.9640$$.

        What is unique about the interactive terms is that they give a much higher $R^2$, which seems to better the explainability of the model. But the prediction value is negative, and makes an outlier. This condraction is caused by the collinearity of the raw data.
    }

    \subsection*{Alternative: Half interactive terms}
    {
        We propose an alternative method, in which we keep interactive terms except $x_1 x_2$. We find this trick works!

        $$\hat{y} = 669.9723 - 0.8563x_1 - 107.6020x_2 + 160.5363x_3 - 98.7687x_2 x_3 + 0.2065x_3 x_1, \quad R^2 = 0.2437$$.
        Plug in $x = (110, 3, 1)$, one has $$\hat{y} = 139.9174$$.

        In other words, we assume the the gene and the tumor size have no interactive effect. Under this assumption, we have a model with better $R^2$, and also give a reasonable prediction.
    }
}

\section*{Source Code}
{
    Please download the souece code from http://39.106.23.58/files/PR1\_2015011506.7z
}

\clearpage
\end{document}


\subsection{Conics}\label{sec:Conics}

In this section we review equations of parabolas, circles, ellipses and hyperbolas.
We will give the equations of various conics in \dfont{standard form} along with a sketch.
A useful mnemonic is the following.\\

\begin{formulabox}[Mnemonic]
In each conic formula presented, the terms $\,x-h\,$ and $\,y-k\,$ will always appear.
The point $(h,k)$ will alway represent either the centre or vertex of the particular conic.
\end{formulabox}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%\subsubsection{Parabolas, circles, ellipses and hyperbolas}

\bigskip\noindent
\dfont{Vertical Parabola:} The equation of a vertical parabola is:
$$y-k=a(x-h)^2$$
$$\includegraphics[width=85mm]{images/conics1}$$
\begin{multicols}{2}
\begin{itemize}
	\item $(h,k)$ is the \ifont{vertex} of the parabola.
	\item $a$ is the vertical \ifont{stretch factor}.
	\item If $a>0$, the parabola opens \ifont{upward}.
	\item If $a<0$, the parabola opens \ifont{downward}.
\end{itemize}
\end{multicols}

\bigskip\noindent
\dfont{Horizontal Parabola:} The equation of a horizontal parabola is:
$$x-h=a(y-k)^2$$
$$\includegraphics[width=85mm]{images/conics2}$$
\begin{multicols}{2}
\begin{itemize}
	\item $(h,k)$ is the \ifont{vertex} of the parabola.
	\item $a$ is the horizontal \ifont{stretch factor}.
	\item If $a>0$, the parabola opens \ifont{right}.
	\item If $a<0$, the parabola opens \ifont{left}.
\end{itemize}
\end{multicols}

\bigskip\noindent
\dfont{Circle:} The equation of a circle is:
$$(x-h)^2+(y-k)^2=r^2$$
$$\includegraphics[width=40mm]{images/conics3}$$
\begin{multicols}{2}
\begin{itemize}
	\item $(h,k)$ is the \ifont{centre} of the circle.
	\item  $r$ is the \ifont{radius} of the circle.
\end{itemize}
\end{multicols}

\bigskip\noindent
\dfont{Ellipse:} The equation of an ellipse is:
$$\frac{(x-h)^2}{a^2}+\frac{(y-k)^2}{b^2}=1$$
$$\includegraphics[width=40mm]{images/conics4}$$
\begin{itemize}
	\item $(h,k)$ is the \ifont{centre} of the ellipse.
	\item $a$ is the \ifont{horizontal distance} from the centre to the edge of the ellipse.
	\item $b$ is the \ifont{vertical distance} from the centre to the edge of the ellipse.
\end{itemize}

\bigskip\noindent
\dfont{Horizontal Hyperbola:} The equation of a horizontal hyperbola is:
$$\frac{(x-h)^2}{a^2}-\frac{(y-k)^2}{b^2}=1$$
%$$\includegraphics[width=40mm]{images/conics5}$$
\begin{multicols}{2}
\begin{itemize}
	\item $(h,k)$ is the \ifont{centre} of the hyperbola.
	\item $a,b$ are the \ifont{reference box} values. The box has a centre of $(h,k)$.
	\item $a$ is the \ifont{horizontal distance} from the centre to the edge of the box.
	\item $b$ is the \ifont{vertical  distance} from the centre to the edge of the box.
\end{itemize}
\end{multicols}
Given the equation of a horizontal hyperbola, one may sketch it by first placing a dot at
the point $(h,k)$. Then draw a box around $(h,k)$ with horizontal distance $a$ and vertical distance $b$ to the edge of the box. Then draw dotted lines (called the \dfont{asymptotes} of the hyperbola) through the corners of the box. Finally, sketch the hyperbola in a horizontal direction as illustrated below.
$$\includegraphics[width=150mm]{images/conics6}$$

\bigskip\noindent
\dfont{Vertical Hyperbola:} The equation of a vertical hyperbola is:
$$\frac{(x-h)^2}{a^2}-\frac{(y-k)^2}{b^2}=-1$$
%$$\includegraphics[width=40mm]{images/conics7}$$
\begin{multicols}{2}
\begin{itemize}
	\item $(h,k)$ is the \ifont{centre} of the hyperbola.
	\item $a,b$ are the \ifont{reference box} values. The box has a centre of $(h,k)$.
	\item $a$ is the \ifont{horizontal distance} from the centre to the edge of the box.
	\item $b$ is the \ifont{vertical  distance} from the centre to the edge of the box.
\end{itemize}
\end{multicols}
Given the equation of a vertical hyperbola, one may sketch it by following the same steps as with a horizontal hyperbola, but sketching the hyperbola going in a vertical direction.
$$\includegraphics[width=150mm]{images/conics8}$$

\begin{formulabox}[Determining the Type of Conic]
An equation of the form
$$Ax^2+Bxy+Cy^2+Dx+Ey+F=0$$
gives rise to a graph that can be generated by performing a conic section (parabolas, circles, ellipses, hyperbolas).
Note that the $Bxy$ term involves conic rotation. The $Dx$, $Ex$, and $F$ terms affect the vertex and centre.
For simplicity, we will omit the $Bxy$ term.
To determine the type of graph we focus our analysis on the values of $A$ and $C$.
\begin{itemize}\setlength{\itemsep}{0 in}
	\item If $A=C$, the graph is a \ifont{circle}.
	\item If $AC>0$ (and $A\neq C$), the graph is an \ifont{ellipse}.
	\item If $AC=0$, the graph is a \ifont{parabola}.
	\item If $AC<0$, the graph is a \ifont{hyperbola}.
\end{itemize}
\end{formulabox}

%\subsubsection{Completing the square}
%The technique of \dfont{completing the square} allows us to determine the center of a circle/ellipse or the vertex of a parabola.
%It is also an important technique used for other purposes (for example, to derive the quadratic formula you can start by completing the square in $ax^2+bx+c=0$, $a\neq 0$).
%
%The main idea behind completing the square is to turn:
%$$ ax^2 + bx + c$$
%into
%$$a(x - h)^2 + k.$$
%One way to complete the square is to use the following formula:
%$$ax^2+bx+c=a\left(x+\frac{b}{2a}\right)^2-\frac{b^2}{4a^2}+c.$$
%But this formula is a bit complicated, so some students prefer following the steps outlined in the next example.
%
%\begin{example}{Completing the Square}{CompletingSquare}
%Solve $2x^2+12x-32=0$ by completing the square.
%\end{example}
%
%\begin{solution}
%In this instance, we will \ifont{not} divide by $2$ first (usually you would) in order to demonstrate what you should do when the `$a$' value is not $1$.
%
%\bigskip
%
%\begin{tabular}{rl}
%$2x^2+12x-32=0$ & Start with original equation.\\
%\\
%$2x^2+12x=32$ & Move the number over to the other side.\\
%\\
%$2(x^2+6x)=32$ & Factor out the $a$ from the $ax^2+bx$ expression.\\
%\\
%$6~~\to~~\frac{6}{2}=3~~\to~~3^2=\dfont{9}$ & Take the number in front of $x$, \\
%	&  \dfont{divide by $2$}, \\
%	&  then \dfont{square} it.\\
%\\
%$\ifont{2}(x^2+6x+\dfont{9})=32+\ifont{2}\cdot\dfont{9}$ & Add the result to both sides, \\
%	&  taking $a=2$ into account.\\
%\\
%$2(x+3)^2=50$ & Factor the resulting perfect square trinomial.\\
%\\
%~ & \ifont{You have now completed the square!}\\
%\\
%$(x+3)^2=25~~\to~~x=2 \mbox{ or } x=-8$ & To solve for $x$, simply divide by $a=2$ \\
%	& and take square roots.\\
%\end{tabular}
%\end{solution}

\begin{example}{Center and Radius of a Circle}{CenterRadius}
Find the centre and radius of the circle $y^2 + x^2 - 12x + 8y + 43 = 0$.
\end{example}

\begin{solution}
We need to complete the square twice, once for the $x$ terms and once for the $y$ terms.
We'll do both at the same time.
First let's collect the terms with $x$ together, the terms with $y$ together, and move the number to the other side.
%$$y^2 + x^2 - 12x + 8y + 43 = 0$$
$$(x^2-12x)+(y^2+8y)=-43$$
We add $36$ to both sides for the $x$ term ($-12\to \frac{-12}{2}=-6\to (-6)^2=36$), and $16$ to both sides for the $y$ term ($8\to \frac{8}{2}=4\to (4)^2=16$):
$$(x^2-12x+36)+(y^2+8y+16)=-43+36+16$$
Factoring gives:
$$(x-6)^2+(y+4)^2=3^2.$$
Therefore, the centre of the circle is $(6,-4)$ and the radius is $3$.
\end{solution}

\begin{example}{Type of Conic}{TypeConic}
What type of conic is $4x^2-y^2-8x+8=0$?
Put it in standard form.
\end{example}

\begin{solution}
Here we have $A=4$ and $C=-1$.
Since $AC<0$, the conic is a hyperbola.
Let us complete the square for the $x$ and $y$ terms.
First let's collect the terms with $x$ together, the terms with $y$ together, and move the number to the other side.
$$(4x^2-8x)-y^2=-8$$
Now we factor out $4$ from the $x$ terms.
$$4(x^2-2x)-y^2=-8$$
Notice that we don't need to complete the square for the $y$ terms (it is already completed!). To complete the square for the $x$ terms we add \dfont{1} ($-2\to\frac{-2}{2}=-1\to(-1)^2=1$), taking into consideration that the $a$ value is $4$:
$$\ifont{4}(x^2-2x+\dfont{1})-y^2=-8+\ifont{4}\cdot\dfont{1}$$
Factoring gives:
$$4(x-1)^2-y^2=-4$$
A hyperbola in standard form has $\pm1$ on the right side and a positive $x^2$ on the left side, thus, we must divide by $4$:
$$(x-1)^2-\frac{y^2}{4}=-1$$
Now we can see that the equation represents a vertical hyperbola with centre $(1,0)$ (and with $a$ value $\sqrt{1}=1$, and $b$ value $\sqrt{4}=2$).
\end{solution}

\begin{example}{Equation of Parabola}{EquationParabola}\label{EquationParabola}
Find an equation of the parabola with vertex $(1,-1)$ that passes through the points $(-4, 24)$ and $(7, 35)$.
\end{example}

\begin{solution}
We first need to determine if it is a vertical parabola or horizontal parabola.
See figure \ref{fig:EquationParabola} for a sketch of the three points $(1,-1)$, $(-4, 24)$ and $(7, 35)$ in the $xy$-plane.
\figure[!ht]
$$\includegraphics[width=60mm]{images/conics9}$$
\caption{Figure for Example \ref{EquationParabola}\label{fig:EquationParabola}}
\endfigure
Note that the vertex is $(1,-1)$.
Given the location of the vertex, the parabola cannot open downwards.
It also cannot open left or right (because the vertex is between the other two points - if it were to open to the right, every other point would need to be to the right of the vertex; if it were to open to the left, every other point would need to be to the left of the vertex).
Therefore, the parabola must open upwards and it is a vertical parabola.
It has an equation of
$$y-k=a(x-h)^2.$$
\begin{multicols}{2}
As the vertex is $(h,k)=(1,-1)$ we have:
$$y-(-1)=a(x-1)^2$$
To determine $a$, we substitute one of the points into the equation and solve.
Let us substitute the point $(x,y)=(-4,24)$ into the equation:\\
$$\begin{array}{rcl}
24-(-1) & = & a(-4-1)^2 \\
25 & = & 25a \\
a & = & 1
\end{array}$$

Therefore, the equation of the parabola is:
$$y+1=(x-1)^2.$$
Note that if we substituted $(7,35)$ into the equation instead, we would also get $a=1$.

$$\includegraphics[scale=0.4]{images/ExQuadratic}$$
\end{multicols}

\end{solution}


\documentclass[12pt,english]{article}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{mathtools}
\usepackage{relsize} % make some math large
\usepackage{bm} % make some math bold
\usepackage{bbm}
% url package

\usepackage{hyperref}
\hypersetup{
    colorlinks=true,
    linkcolor=blue,
    filecolor=magenta,
    urlcolor=cyan,
}

% Titling and Author
\title{Latex Example, Multiple Lines Equations}
\author{\href{https://fanwangecon.github.io/}{Fan Wang}\thanks{https://fanwangecon.github.io, repository: \href{https://fanwangecon.github.io/Tex4Econ/}{Tex4Econ}}}
\date{\today}

\begin{document}

\maketitle

\section{Indicator Function}
\begin{align}\label{eq:indi}
  \begin{gathered}
    \mathbbm{1}\left\{1 + 2 +3\right\}
    \\
    \mathlarger{\mathlarger{\mathlarger{
      \mathbbm{1}
    }}}
    \left\{
      \begin{array}{cc}
          1 + 2 + 3\\
          + 4 + 5\\
          > 0\\
      \end{array}
    \right\}
    \\
    \mathlarger{\mathlarger{\mathlarger{
    \mathlarger{\mathlarger{\mathlarger{
      \mathbbm{1}
    }}}}}}
    \left\{
      \begin{array}{cc}
          1 + 2 + 3\\
          + 4 + 5 + 6 \\
          + 7 + 8 + 9 \\
          > 0\\
      \end{array}
    \right\}
  \end{gathered}
\end{align}

\section{Multiple Lines of Equation}

Below, I demonstrate several possible ways of aligning and labeling four lines of equations. First, I label some of the four lines with own labels and align by equality sign. Second, we align by the equality sign but use one single label for all lines of equations. Third, we center align on page label each line using gather. Fourth, we center align on page with one label putting gather inside align.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%% A.1 Align and Split
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\pagebreak
\subsection{Align and Split}
\subsubsection{Align and Label Lines}
\begin{verbatim}
  \begin{align}
    x_1  =& 1        \label{eq:lineone}\\
    x_2  =& 1        \nonumber\\
    h(x) =& f(-20 + 15 + 17) \label{eq:linethree}\\
    h(x) =& f(12) \approx 1  \label{eq:linefour}
  \end{align}
\end{verbatim}
\begin{align}
  x_1  =& 1        \label{eq:lineone}\\
  x_2  =& 1        \nonumber\\
  h(x) =& f(-20 + 15 + 17) \label{eq:linethree}\\
  h(x) =& f(12) \approx 1  \label{eq:linefour}
\end{align}
\subsubsection{Align Split One Label}
\begin{verbatim}
\begin{align}\label{eq:split}
  \begin{split}
    x_1 &= 1 \\
    y &= 1        \\
    h(x) &= f(-20 + 10 + 20) \\
    g(x) &= f(12) \approx 2
  \end{split}
\end{align}
\end{verbatim}
\begin{align}\label{eq:split}
  \begin{split}
    x_1 &= 1 \\
    y &= 1        \\
    h(x) &= f(-20 + 10 + 20) \\
    g(x) &= f(12) \approx 2
  \end{split}
\end{align}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%% A.2 Align and Gather
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\pagebreak
\subsection{Align and Gather}
\subsubsection{Gather and Label Lines}
\begin{verbatim}
  \begin{gather}
    x_1  = 1        \label{eq:lineone}\\
    x_2  = 1        \nonumber\\
    h(x) = f(-20 + 15 + 17) \label{eq:linethree}\\
    h(x) = f(12) \approx 1  \label{eq:linefour}
\end{gather}
\end{verbatim}
\begin{gather}
  x_1  = 1        \label{eq:lineone}\\
  x_2  = 1        \nonumber\\
  h(x) = f(-20 + 15 + 17) \label{eq:linethree}\\
  h(x) = f(12) \approx 1  \label{eq:linefour}
\end{gather}
\subsubsection{Align Gathered}
\begin{verbatim}
  \label{eq:gathered}
  \begin{gathered}
    x_1 = 1       \\
    x_2 = 1        \\
    h(x) = f(-20 + 15 + 17) \\
    h(x) = f(12) \approx 1
  \end{gathered}
\end{verbatim}
\begin{align}
  \label{eq:gathered}
  \begin{gathered}
    x_1 = 1       \\
    x_2 = 1        \\
    h(x) = f(-20 + 15 + 17) \\
    h(x) = f(12) \approx 1
  \end{gathered}
\end{align}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%% B. Substack vs Array
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\pagebreak
\section{Substack vs Array}
\begin{verbatim}
\begin{align}
    \begin{split}
    \label{eq:Value}
    v_{ih}\left(a,z\right)
        =
        \max_{
            \substack{
                c>0\\
                a' \in \{0,[\bar{A}_{ih},\infty)\}
                }
            }
        u\left(c\right)+  \beta \int v_{ih}\left(a',z'\right)f(z'|z)dz'\\
    \label{eq:Value}
    v_{ih}\left(a,z\right)
        =
        \max_{
            \begin{array}{cc}
                c>0\\
                a' \in \{0,[\bar{A}_{ih},\infty)\}\\
            \end{array}
            }
        u\left(c\right)+  \beta \int v_{ih}\left(a',z'\right)f(z'|z)dz'\\
    \end{split}
\end{align}
\end{verbatim}

Using Substack, fonts are small under max:
\begin{align}
    \begin{split}
        \label{eq:Value}
        v_{ih}\left(a,z\right)
        =
        \max_{
            \substack{
                c>0\\
                a' \in \{0,[\bar{A}_{ih},\infty)\}
                }
            }
        u\left(c\right)+  \beta \int v_{ih}\left(a',z'\right)f(z'|z)dz'\\
    \end{split}
\end{align}

Using Array, fonts are larger under max:
\begin{align}
    \begin{split}
        \label{eq:Value}
        v_{ih}\left(a,z\right)
        =
        \max_{
            \begin{array}{cc}
                c>0\\
                a' \in \{0,[\bar{A}_{ih},\infty)\}\\
            \end{array}
            }
        u\left(c\right)+  \beta \int v_{ih}\left(a',z'\right)f(z'|z)dz'\\
    \end{split}
\end{align}

\pagebreak

\section{Multiple Lines}

\begin{verbatim}
\begin{align}
    x = y + z +
    \left\{
      \begin{array}{l}
            a \\
            + b \\
            + c \\
            + d \\
      \end{array}
    \right\}
\end{align}
\end{verbatim}

\begin{align}
    x = y + z +
    \left\{
      \begin{array}{l}
            a \\
            + b \\
            + c \\
            + d \\
      \end{array}
    \right\}
\end{align}

\section{Brackets}

Then in your document:

\[
\left(\rule{0cm}{2cm}
 \frac{\prod_0^\infty k}{\sum_0^\infty n}
\right)
\]

\end{document}


\documentclass[12pt]{cdblatex}
\usepackage{examples}

\begin{document}

\section*{Symmetry of the Ricci tensor}

This simple example shows that, for the metric connection, the Ricci tensor is symmetric, that is $R_{ab} = R_{ba}$.

\vspace{15pt}

\begin{cadabra}
   {a,b,c,d,e,f,i,j,k,l,m,n,o,p,q,r,s,t,u#}::Indices(position=independent).

   \partial{#}::PartialDerivative;

   g_{a b}::Symmetric;
   g^{a b}::Symmetric;

   g_{a b}::Depends(\partial{#});
   g^{a b}::Depends(\partial{#});

   dgab := \partial_{c}{g^{a b}} -> - g^{a e} g^{b f} \partial_{c}{g_{e f}}. # cdb (dgab,dgab)

   Gamma := \Gamma^{a}_{b c} ->
            (1/2) g^{a e} (   \partial_{b}{g_{e c}}
                            + \partial_{c}{g_{b e}}
                            - \partial_{e}{g_{b c}}).     # cdb (Chr,Gamma)

   Rabcd := R^{a}_{b c d} ->
           \partial_{c}{\Gamma^{a}_{b d}} + \Gamma^{a}_{e c} \Gamma^{e}_{b d}
         - \partial_{d}{\Gamma^{a}_{b c}} - \Gamma^{a}_{e d} \Gamma^{e}_{b c}.
                                                          # cdb (Rabcd,Rabcd)

   Rab := R_{a b} -> R^{c}_{a c b}.                       # cdb (Rab,Rab)

   eqn := 2 (R_{a b} - R_{b a}).

   substitute (eqn, Rab)
   substitute (eqn, Rabcd)
   substitute (eqn, Gamma)

   distribute   (eqn)
   product_rule (eqn)
   canonicalise (eqn)                                     # cdb (final1,eqn)

   substitute   (eqn,dgab)
   canonicalise (eqn)                                     # cdb (final2,eqn)
\end{cadabra}

\subsection*{Symmetry of the Ricci tensor}

\begin{minipage}[t]{0.60\textwidth}
\begin{align*}
   g^{a b}{}_{,c}   &:=\cdb{dgab}\\
   \Gamma^{a}_{bc}  &:=\cdb{Chr}\\
   R^a{}_{bcd}      &:=\cdb{Rabcd}\\
   R_{ab}           &:=\cdb{Rab}\\[10pt]
   2(R_{ab}-R_{ba}) &= \cdb{final1}\\
                    &= \cdb{final2}
\end{align*}
\end{minipage}
\hskip 1cm
\lower16pt\hbox{%
\begin{minipage}[t]{0.40\textwidth}
\begin{latex}
   \begin{align*}
      g^{a b}{}_{,c}   &:=\cdb{dgab}\\
      \Gamma^{a}_{bc}  &:=\cdb{Chr}\\
      R^a{}_{bcd}      &:=\cdb{Rabcd}\\
      R_{ab}           &:=\cdb{Rab}\\[10pt]
      2(R_{ab}-R_{ba}) &= \cdb{final1}\\
                       &= \cdb{final2}
   \end{align*}
\end{latex}
\end{minipage}}

\end{document}


% Copyright 2018 Melvin Eloy Irizarry-Gelpí
\chapter{Tri-Nilplex Numbers}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
An exo-3 real number has the form
\begin{equation}
    a_{0} + a_{1} W + a_{2} X + a_{3} WX + a_{4} Y + a_{5} WY + a_{6} XY + a_{7} WXY
\end{equation}
These follow from a parabolic Cayley-Dickson construct on the exo-2 reals.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Arithmetic Operations}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
...
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Multiplication}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
...
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Conjugate Operations}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
...
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Asterisk Conjugation}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
...
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Quadrance}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
...
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Cloak Conjugation}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
...
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Dagger Conjugation}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
...
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Hodge Star}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
...
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Differential Operators}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
...


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% shortestpath.tex last revised 29 Jan 2014 by Stuart Ambler.
% Explanation of correctness of bidirectional breadth first search algorithm for
% finding a shortest path between two nodes in an unweighted undirected graph.
% Copyright (c) 2014 Stuart Ambler.
% Permission is granted to copy, distribute and/or modify this document
% under the terms of the GNU Free Documentation License, Version 1.3
% or any later version published by the Free Software Foundation;
% with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.
% A copy of the license is included in the section entitled "GNU
% Free Documentation License".
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\documentclass{article}
\usepackage{amsmath,amsfonts,amssymb,amsthm,amsxtra,url}

% Part of GNU FDL LaTeX code (rest in section GNU Free Documentation License)
% Some lines commented out, marked with added trailing comment SA.
% This is set up to run with pdflatex.
%---------The file header---------------------------------------------
%\documentclass[a4paper,12pt]{book}  % SA

\usepackage[english]{babel} %language selection
\selectlanguage{english}

\pagenumbering{arabic}

\usepackage{hyperref}
\hypersetup{colorlinks,
           citecolor=black,
           filecolor=black,
           linkcolor=black,
           urlcolor=black,
           bookmarksopen=true,
           pdftex}

\hfuzz = .6pt % avoid black boxes

%\begin{document}  % SA
%---------------------------------------------------------------------
\begin{document}

\title{Why the Bidirectional Breadth First Search Algorithm Finds a Shortest Path}
\author{Stuart Ambler}
\maketitle

Suppose you have an unweighted undirected graph, given for every node $x$ of the graph the set $E_x$ of nodes on edges meeting $x$ (i.e. $E_x$ is the edgelist of $x$), and want a shortest path between two nodes $a$ and $b$.

A \textbf{unidirectional breadth first search} starting from $a$ will find one.  Let $A_0 = \{ a \}$, and for $k > 0$, $A_k = (\cup_{x \in A_{k-1}} E_x) \setminus \cup_{0 \le j < k} A_j$ (all the nodes in all the edgelists of all the nodes in $A_{k-1}$, but removing nodes in all previous $A_j$).  Define $d(x,a)$ as the distance from $x$ to $a$, the length of the shortest path from $x$ to $a$.  This is a metric, in particular for any $x, y, z$, $d(x,y) = 0 \Leftrightarrow x = y$ and $d(x,z) \le d(x,y) + d(y,z)$.  The latter is true because you can make a path from $x$ to $z$ by concatenating shortest paths from $x$ to $y$ and $y$ to $z$, though this concatenation may not be the shortest path from $x$ to $z$.

Claim: $A_k = \{ x | d(x, a) = k \}$.  If $d(a,x) = k$, then $x \in A_k$ because there is a path of length $k$ from $a$, and no shorter path (which would cause $x \in A_j$ for some $j < k$).

On the other hand, if $x \in A_k$, there is a path of length $k$ from $a$ to $x$, so $d(a,x) \le k$.  It's not possible that $d(a,x) = j < k$, because if it were, then $x \in A_j$ and thus not in $A_k$.  Thus $d(a,x) = k$.

A breadth first search for $b$ starting from $a$, implemented as in bfs1.py, in effect creates the sets $A_j$ for $j < k = d(a,b)$, and some part of $A_k$ until it encounters $b$ in that set.  This is because nodes are added to the queue from edgelists of nodes popped off the queue (which is initialized with $a$ and is FIFO), but only if they have not been previously encountered.  The code implicitly shifts from $A_j$ to $A_{j+1}$ when all the edgelists of nodes belonging to $A_{j-1}$ have been processed.

For every node in $A_j$ added to the queue, the code keeps track of which node in $A_{j-1}$ owned the edgelist on which it was first encountered.  A shortest path from $a$ to $b$ is then built by reversing what is obtained by following these parent relationships backwards.

A \textbf{bidirectional breadth first search} can be faster, due to the potentially exponential fan-out of breadth first search.  Define the $A_k$ as before, and the $B_k$ similarly starting from $B_0 = \{ b \}$.  Supposing that we start from the one end with $A_0$, from the other with $B_0$, that some choice is made at each iteration whether to move from $A_r$ to $A_{r+1}$ or $B_s$ to $B_{s+1}$, and that we stop when $A_r \cap B_s \ne \emptyset$ for the first time.  From the two paths from the ends we get a path between the endpoints $a$ and $b$ of length $r+s$.  In fact, there is no shorter path.  To see this, use the fact that if the equations below for $A_0$ through $A_k$ hold, then the logical equivalence for $A_{k+1}$ holds.
\begin{align}
&\text{If}  \notag \\
A_0 \cap B_0 &= \emptyset \notag \\
A_1 \cap B_0 &= \emptyset \notag \\
&\cdots \notag \\
A_k \cap B_0 &= \emptyset \notag \\
&\text{then}  \notag \\
A_{k+1} \cap B_0 &= \emptyset \Leftrightarrow A_r \cap B_s = \emptyset, \notag
\end{align}
for any nonnegative $r, s$ with $r + s = k + 1$.

This can be seen as follows. If $A_{k+1} \cap B_0 \ne \emptyset$, then recalling the properties of unidirectional breadth first search from $a$, there is a shortest path of length $k+1$ from $a$ to $b$.  This can be split into two pieces of lengths $r$ and $s$, reversing one of them, that are shortest paths of those lengths from $a$ to some $x$ and from $b$ to $x$, so that $x \in A_r \cap B_s$, which is thus nonempty.  Conversely, if there is an $x \in A_r \cap B_s$, by concatenating paths, one of them reversed, we get a path of length $r+s = k+1$ from $a$ to $b$, so that $d(a,b) \le k+1$.  But $d(a,b) \le k$ is false because of all the intersections empty by hypothesis.  Thus $d(a,b) = k+1$, and thus $A_{k+1} \cap B_0 \ne \emptyset$.

Then, to see that the path we construct that meets at a point in the first nonempty $A_r \cap B_s$ is a shortest path from $a$ to $b$, if $A_0 \cap B_0 \ne \emptyset$, then $a=b$ and the path is of length $0$.  Otherwise, use the above fact repeatedly, noting that we have some sequence of $A_{r_i} \cap B_{s_i} = \emptyset$, $r_i + s_i = k_i = 1,2,\dots,r+s-1$, implying $A_{k_i} \cap B_0 = \emptyset$ until finally we arrive at $A_r \cap B_s \ne \emptyset \Rightarrow A_{r+s} \cap B_0 \ne \emptyset \Rightarrow d(a,b) = r+s$.

The code in bfs2.py works with the $A_{r_i}$ and $B_{s_i}$ fairly explicitly, using the lists r\_level\_nodes and t\_level\_nodes for them (r for root or $a$, t for target or $b$).  It processes a whole level, or one of these sets, at a time, before going on to the next.  It finds a nonempty intersection of the two by finding a newly encountered node in the one that it's building, that is in the other that it's already built, and stops when this first happens.

\bigskip
\begin{quote}
    Copyright \copyright{} 2014 Stuart Ambler.
    Permission is granted to copy, distribute and/or modify this document
    under the terms of the GNU Free Documentation License, Version 1.3
    or any later version published by the Free Software Foundation;
    with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.
    A copy of the license is included in the section entitled ``GNU
    Free Documentation License''.
\end{quote}
\bigskip

\pagebreak
\section{GNU Free Documentation License}
%\chapter*{\rlap{GNU Free Documentation License}}  % SA
\phantomsection  % so hyperref creates bookmarks
\addcontentsline{toc}{chapter}{GNU Free Documentation License}
%\label{label_fdl}

 \begin{center}

       Version 1.3, 3 November 2008

 Copyright \copyright{} 2000, 2001, 2002, 2007, 2008  Free Software Foundation, Inc.

 \bigskip

%     \texttt{<http://fsf.org/>}  % SA
     \url{http://fsf.org/}

 \bigskip

 Everyone is permitted to copy and distribute verbatim copies
 of this license document, but changing it is not allowed.
\end{center}

\begin{center}
{\bf\large Preamble}
\end{center}

The purpose of this License is to make a manual, textbook, or other
functional and useful document ``free'' in the sense of freedom: to
assure everyone the effective freedom to copy and redistribute it,
with or without modifying it, either commercially or noncommercially.
Secondarily, this License preserves for the author and publisher a way
to get credit for their work, while not being considered responsible
for modifications made by others.

This License is a kind of ``copyleft'', which means that derivative
works of the document must themselves be free in the same sense.  It
complements the GNU General Public License, which is a copyleft
license designed for free software.

We have designed this License in order to use it for manuals for free
software, because free software needs free documentation: a free
program should come with manuals providing the same freedoms that the
software does.  But this License is not limited to software manuals;
it can be used for any textual work, regardless of subject matter or
whether it is published as a printed book.  We recommend this License
principally for works whose purpose is instruction or reference.

\begin{center}
{\Large\bf 1. APPLICABILITY AND DEFINITIONS\par}
\phantomsection
\addcontentsline{toc}{section}{1. APPLICABILITY AND DEFINITIONS}
\end{center}

This License applies to any manual or other work, in any medium, that
contains a notice placed by the copyright holder saying it can be
distributed under the terms of this License.  Such a notice grants a
world-wide, royalty-free license, unlimited in duration, to use that
work under the conditions stated herein.  The ``\textbf{Document}'', below,
refers to any such manual or work.  Any member of the public is a
licensee, and is addressed as ``\textbf{you}''.  You accept the license if you
copy, modify or distribute the work in a way requiring permission
under copyright law.

A ``\textbf{Modified Version}'' of the Document means any work containing the
Document or a portion of it, either copied verbatim, or with
modifications and/or translated into another language.

A ``\textbf{Secondary Section}'' is a named appendix or a front-matter section of
the Document that deals exclusively with the relationship of the
publishers or authors of the Document to the Document's overall subject
(or to related matters) and contains nothing that could fall directly
within that overall subject.  (Thus, if the Document is in part a
textbook of mathematics, a Secondary Section may not explain any
mathematics.)  The relationship could be a matter of historical
connection with the subject or with related matters, or of legal,
commercial, philosophical, ethical or political position regarding
them.

The ``\textbf{Invariant Sections}'' are certain Secondary Sections whose titles
are designated, as being those of Invariant Sections, in the notice
that says that the Document is released under this License.  If a
section does not fit the above definition of Secondary then it is not
allowed to be designated as Invariant.  The Document may contain zero
Invariant Sections.  If the Document does not identify any Invariant
Sections then there are none.

The ``\textbf{Cover Texts}'' are certain short passages of text that are listed,
as Front-Cover Texts or Back-Cover Texts, in the notice that says that
the Document is released under this License.  A Front-Cover Text may
be at most 5 words, and a Back-Cover Text may be at most 25 words.

A ``\textbf{Transparent}'' copy of the Document means a machine-readable copy,
represented in a format whose specification is available to the
general public, that is suitable for revising the document
straightforwardly with generic text editors or (for images composed of
pixels) generic paint programs or (for drawings) some widely available
drawing editor, and that is suitable for input to text formatters or
for automatic translation to a variety of formats suitable for input
to text formatters.  A copy made in an otherwise Transparent file
format whose markup, or absence of markup, has been arranged to thwart
or discourage subsequent modification by readers is not Transparent.
An image format is not Transparent if used for any substantial amount
of text.  A copy that is not ``Transparent'' is called ``\textbf{Opaque}''.

Examples of suitable formats for Transparent copies include plain
ASCII without markup, Texinfo input format, LaTeX input format, SGML
or XML using a publicly available DTD, and standard-conforming simple
HTML, PostScript or PDF designed for human modification.  Examples of
transparent image formats include PNG, XCF and JPG.  Opaque formats
include proprietary formats that can be read and edited only by
proprietary word processors, SGML or XML for which the DTD and/or
processing tools are not generally available, and the
machine-generated HTML, PostScript or PDF produced by some word
processors for output purposes only.

The ``\textbf{Title Page}'' means, for a printed book, the title page itself,
plus such following pages as are needed to hold, legibly, the material
this License requires to appear in the title page.  For works in
formats which do not have any title page as such, ``Title Page'' means
the text near the most prominent appearance of the work's title,
preceding the beginning of the body of the text.

The ``\textbf{publisher}'' means any person or entity that distributes
copies of the Document to the public.

A section ``\textbf{Entitled XYZ}'' means a named subunit of the Document whose
title either is precisely XYZ or contains XYZ in parentheses following
text that translates XYZ in another language.  (Here XYZ stands for a
specific section name mentioned below, such as ``\textbf{Acknowledgements}'',
``\textbf{Dedications}'', ``\textbf{Endorsements}'', or ``\textbf{History}''.)
To ``\textbf{Preserve the Title}''
of such a section when you modify the Document means that it remains a
section ``Entitled XYZ'' according to this definition.

The Document may include Warranty Disclaimers next to the notice which
states that this License applies to the Document.  These Warranty
Disclaimers are considered to be included by reference in this
License, but only as regards disclaiming warranties: any other
implication that these Warranty Disclaimers may have is void and has
no effect on the meaning of this License.

\begin{center}
{\Large\bf 2. VERBATIM COPYING\par}
\phantomsection
\addcontentsline{toc}{section}{2. VERBATIM COPYING}
\end{center}

You may copy and distribute the Document in any medium, either
commercially or noncommercially, provided that this License, the
copyright notices, and the license notice saying this License applies
to the Document are reproduced in all copies, and that you add no other
conditions whatsoever to those of this License.  You may not use
technical measures to obstruct or control the reading or further
copying of the copies you make or distribute.  However, you may accept
compensation in exchange for copies.  If you distribute a large enough
number of copies you must also follow the conditions in section~3.

You may also lend copies, under the same conditions stated above, and
you may publicly display copies.

\begin{center}
{\Large\bf 3. COPYING IN QUANTITY\par}
\phantomsection
\addcontentsline{toc}{section}{3. COPYING IN QUANTITY}
\end{center}

If you publish printed copies (or copies in media that commonly have
printed covers) of the Document, numbering more than 100, and the
Document's license notice requires Cover Texts, you must enclose the
copies in covers that carry, clearly and legibly, all these Cover
Texts: Front-Cover Texts on the front cover, and Back-Cover Texts on
the back cover.  Both covers must also clearly and legibly identify
you as the publisher of these copies.  The front cover must present
the full title with all words of the title equally prominent and
visible.  You may add other material on the covers in addition.
Copying with changes limited to the covers, as long as they preserve
the title of the Document and satisfy these conditions, can be treated
as verbatim copying in other respects.

If the required texts for either cover are too voluminous to fit
legibly, you should put the first ones listed (as many as fit
reasonably) on the actual cover, and continue the rest onto adjacent
pages.

If you publish or distribute Opaque copies of the Document numbering
more than 100, you must either include a machine-readable Transparent
copy along with each Opaque copy, or state in or with each Opaque copy
a computer-network location from which the general network-using
public has access to download using public-standard network protocols
a complete Transparent copy of the Document, free of added material.
If you use the latter option, you must take reasonably prudent steps,
when you begin distribution of Opaque copies in quantity, to ensure
that this Transparent copy will remain thus accessible at the stated
location until at least one year after the last time you distribute an
Opaque copy (directly or through your agents or retailers) of that
edition to the public.

It is requested, but not required, that you contact the authors of the
Document well before redistributing any large number of copies, to give
them a chance to provide you with an updated version of the Document.

\begin{center}
{\Large\bf 4. MODIFICATIONS\par}
\phantomsection
\addcontentsline{toc}{section}{4. MODIFICATIONS}
\end{center}

You may copy and distribute a Modified Version of the Document under
the conditions of sections 2 and 3 above, provided that you release
the Modified Version under precisely this License, with the Modified
Version filling the role of the Document, thus licensing distribution
and modification of the Modified Version to whoever possesses a copy
of it.  In addition, you must do these things in the Modified Version:

\begin{itemize}
\item[A.]
   Use in the Title Page (and on the covers, if any) a title distinct
   from that of the Document, and from those of previous versions
   (which should, if there were any, be listed in the History section
   of the Document).  You may use the same title as a previous version
   if the original publisher of that version gives permission.

\item[B.]
   List on the Title Page, as authors, one or more persons or entities
   responsible for authorship of the modifications in the Modified
   Version, together with at least five of the principal authors of the
   Document (all of its principal authors, if it has fewer than five),
   unless they release you from this requirement.

\item[C.]
   State on the Title page the name of the publisher of the
   Modified Version, as the publisher.

\item[D.]
   Preserve all the copyright notices of the Document.

\item[E.]
   Add an appropriate copyright notice for your modifications
   adjacent to the other copyright notices.

\item[F.]
   Include, immediately after the copyright notices, a license notice
   giving the public permission to use the Modified Version under the
   terms of this License, in the form shown in the Addendum below.

\item[G.]
   Preserve in that license notice the full lists of Invariant Sections
   and required Cover Texts given in the Document's license notice.

\item[H.]
   Include an unaltered copy of this License.

\item[I.]
   Preserve the section Entitled ``History'', Preserve its Title, and add
   to it an item stating at least the title, year, new authors, and
   publisher of the Modified Version as given on the Title Page.  If
   there is no section Entitled ``History'' in the Document, create one
   stating the title, year, authors, and publisher of the Document as
   given on its Title Page, then add an item describing the Modified
   Version as stated in the previous sentence.

\item[J.]
   Preserve the network location, if any, given in the Document for
   public access to a Transparent copy of the Document, and likewise
   the network locations given in the Document for previous versions
   it was based on.  These may be placed in the ``History'' section.
   You may omit a network location for a work that was published at
   least four years before the Document itself, or if the original
   publisher of the version it refers to gives permission.

\item[K.]
   For any section Entitled ``Acknowledgements'' or ``Dedications'',
   Preserve the Title of the section, and preserve in the section all
   the substance and tone of each of the contributor acknowledgements
   and/or dedications given therein.

\item[L.]
   Preserve all the Invariant Sections of the Document,
   unaltered in their text and in their titles.  Section numbers
   or the equivalent are not considered part of the section titles.

\item[M.]
   Delete any section Entitled ``Endorsements''.  Such a section
   may not be included in the Modified Version.

\item[N.]
   Do not retitle any existing section to be Entitled ``Endorsements''
   or to conflict in title with any Invariant Section.

\item[O.]
   Preserve any Warranty Disclaimers.
\end{itemize}

If the Modified Version includes new front-matter sections or
appendices that qualify as Secondary Sections and contain no material
copied from the Document, you may at your option designate some or all
of these sections as invariant.  To do this, add their titles to the
list of Invariant Sections in the Modified Version's license notice.
These titles must be distinct from any other section titles.

You may add a section Entitled ``Endorsements'', provided it contains
nothing but endorsements of your Modified Version by various
parties---for example, statements of peer review or that the text has
been approved by an organization as the authoritative definition of a
standard.

You may add a passage of up to five words as a Front-Cover Text, and a
passage of up to 25 words as a Back-Cover Text, to the end of the list
of Cover Texts in the Modified Version.  Only one passage of
Front-Cover Text and one of Back-Cover Text may be added by (or
through arrangements made by) any one entity.  If the Document already
includes a cover text for the same cover, previously added by you or
by arrangement made by the same entity you are acting on behalf of,
you may not add another; but you may replace the old one, on explicit
permission from the previous publisher that added the old one.

The author(s) and publisher(s) of the Document do not by this License
give permission to use their names for publicity for or to assert or
imply endorsement of any Modified Version.

\begin{center}
{\Large\bf 5. COMBINING DOCUMENTS\par}
\phantomsection
\addcontentsline{toc}{section}{5. COMBINING DOCUMENTS}
\end{center}

You may combine the Document with other documents released under this
License, under the terms defined in section~4 above for modified
versions, provided that you include in the combination all of the
Invariant Sections of all of the original documents, unmodified, and
list them all as Invariant Sections of your combined work in its
license notice, and that you preserve all their Warranty Disclaimers.

The combined work need only contain one copy of this License, and
multiple identical Invariant Sections may be replaced with a single
copy.  If there are multiple Invariant Sections with the same name but
different contents, make the title of each such section unique by
adding at the end of it, in parentheses, the name of the original
author or publisher of that section if known, or else a unique number.
Make the same adjustment to the section titles in the list of
Invariant Sections in the license notice of the combined work.

In the combination, you must combine any sections Entitled ``History''
in the various original documents, forming one section Entitled
``History''; likewise combine any sections Entitled ``Acknowledgements'',
and any sections Entitled ``Dedications''.  You must delete all sections
Entitled ``Endorsements''.

\begin{center}
{\Large\bf 6. COLLECTIONS OF DOCUMENTS\par}
\phantomsection
\addcontentsline{toc}{section}{6. COLLECTIONS OF DOCUMENTS}
\end{center}

You may make a collection consisting of the Document and other documents
released under this License, and replace the individual copies of this
License in the various documents with a single copy that is included in
the collection, provided that you follow the rules of this License for
verbatim copying of each of the documents in all other respects.

You may extract a single document from such a collection, and distribute
it individually under this License, provided you insert a copy of this
License into the extracted document, and follow this License in all
other respects regarding verbatim copying of that document.

\begin{center}
{\Large\bf 7. AGGREGATION WITH INDEPENDENT WORKS\par}
\phantomsection
\addcontentsline{toc}{section}{7. AGGREGATION WITH INDEPENDENT WORKS}
\end{center}

A compilation of the Document or its derivatives with other separate
and independent documents or works, in or on a volume of a storage or
distribution medium, is called an ``aggregate'' if the copyright
resulting from the compilation is not used to limit the legal rights
of the compilation's users beyond what the individual works permit.
When the Document is included in an aggregate, this License does not
apply to the other works in the aggregate which are not themselves
derivative works of the Document.

If the Cover Text requirement of section~3 is applicable to these
copies of the Document, then if the Document is less than one half of
the entire aggregate, the Document's Cover Texts may be placed on
covers that bracket the Document within the aggregate, or the
electronic equivalent of covers if the Document is in electronic form.
Otherwise they must appear on printed covers that bracket the whole
aggregate.

\begin{center}
{\Large\bf 8. TRANSLATION\par}
\phantomsection
\addcontentsline{toc}{section}{8. TRANSLATION}
\end{center}

Translation is considered a kind of modification, so you may
distribute translations of the Document under the terms of section~4.
Replacing Invariant Sections with translations requires special
permission from their copyright holders, but you may include
translations of some or all Invariant Sections in addition to the
original versions of these Invariant Sections.  You may include a
translation of this License, and all the license notices in the
Document, and any Warranty Disclaimers, provided that you also include
the original English version of this License and the original versions
of those notices and disclaimers.  In case of a disagreement between
the translation and the original version of this License or a notice
or disclaimer, the original version will prevail.

If a section in the Document is Entitled ``Acknowledgements'',
``Dedications'', or ``History'', the requirement (section~4) to Preserve
its Title (section~1) will typically require changing the actual
title.

\begin{center}
{\Large\bf 9. TERMINATION\par}
\phantomsection
\addcontentsline{toc}{section}{9. TERMINATION}
\end{center}

You may not copy, modify, sublicense, or distribute the Document
except as expressly provided under this License.  Any attempt
otherwise to copy, modify, sublicense, or distribute it is void, and
will automatically terminate your rights under this License.

However, if you cease all violation of this License, then your license
from a particular copyright holder is reinstated (a) provisionally,
unless and until the copyright holder explicitly and finally
terminates your license, and (b) permanently, if the copyright holder
fails to notify you of the violation by some reasonable means prior to
60 days after the cessation.

Moreover, your license from a particular copyright holder is
reinstated permanently if the copyright holder notifies you of the
violation by some reasonable means, this is the first time you have
received notice of violation of this License (for any work) from that
copyright holder, and you cure the violation prior to 30 days after
your receipt of the notice.

Termination of your rights under this section does not terminate the
licenses of parties who have received copies or rights from you under
this License.  If your rights have been terminated and not permanently
reinstated, receipt of a copy of some or all of the same material does
not give you any rights to use it.

\begin{center}
{\Large\bf 10. FUTURE REVISIONS OF THIS LICENSE\par}
\phantomsection
\addcontentsline{toc}{section}{10. FUTURE REVISIONS OF THIS LICENSE}
\end{center}

The Free Software Foundation may publish new, revised versions
of the GNU Free Documentation License from time to time.  Such new
versions will be similar in spirit to the present version, but may
differ in detail to address new problems or concerns.  See
% \texttt{http://www.gnu.org/copyleft/}.  % SA
\url{http://www.gnu.org/copyleft/}.

Each version of the License is given a distinguishing version number.
If the Document specifies that a particular numbered version of this
License ``or any later version'' applies to it, you have the option of
following the terms and conditions either of that specified version or
of any later version that has been published (not as a draft) by the
Free Software Foundation.  If the Document does not specify a version
number of this License, you may choose any version ever published (not
as a draft) by the Free Software Foundation.  If the Document
specifies that a proxy can decide which future versions of this
License can be used, that proxy's public statement of acceptance of a
version permanently authorizes you to choose that version for the
Document.

\begin{center}
{\Large\bf 11. RELICENSING\par}
\phantomsection
\addcontentsline{toc}{section}{11. RELICENSING}
\end{center}

``Massive Multiauthor Collaboration Site'' (or ``MMC Site'') means any
World Wide Web server that publishes copyrightable works and also
provides prominent facilities for anybody to edit those works.  A
public wiki that anybody can edit is an example of such a server.  A
``Massive Multiauthor Collaboration'' (or ``MMC'') contained in the
site means any set of copyrightable works thus published on the MMC
site.

``CC-BY-SA'' means the Creative Commons Attribution-Share Alike 3.0
license published by Creative Commons Corporation, a not-for-profit
corporation with a principal place of business in San Francisco,
California, as well as future copyleft versions of that license
published by that same organization.

``Incorporate'' means to publish or republish a Document, in whole or
in part, as part of another Document.

An MMC is ``eligible for relicensing'' if it is licensed under this
License, and if all works that were first published under this License
somewhere other than this MMC, and subsequently incorporated in whole
or in part into the MMC, (1) had no cover texts or invariant sections,
and (2) were thus incorporated prior to November 1, 2008.

The operator of an MMC Site may republish an MMC contained in the site
under CC-BY-SA on the same site at any time before August 1, 2009,
provided the MMC is eligible for relicensing.

\begin{center}
{\Large\bf ADDENDUM: How to use this License for your documents\par}
\phantomsection
\addcontentsline{toc}{section}{ADDENDUM: How to use this License for your documents}
\end{center}

To use this License in a document you have written, include a copy of
the License in the document and put the following copyright and
license notices just after the title page:

\bigskip
\begin{quote}
    Copyright \copyright{}  YEAR  YOUR NAME.
    Permission is granted to copy, distribute and/or modify this document
    under the terms of the GNU Free Documentation License, Version 1.3
    or any later version published by the Free Software Foundation;
    with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.
    A copy of the license is included in the section entitled ``GNU
    Free Documentation License''.
\end{quote}
\bigskip

If you have Invariant Sections, Front-Cover Texts and Back-Cover Texts,
replace the ``with \dots\ Texts.''\ line with this:

\bigskip
\begin{quote}
    with the Invariant Sections being LIST THEIR TITLES, with the
    Front-Cover Texts being LIST, and with the Back-Cover Texts being LIST.
\end{quote}
\bigskip

If you have Invariant Sections without Cover Texts, or some other
combination of the three, merge those two alternatives to suit the
situation.

If your document contains nontrivial examples of program code, we
recommend releasing these examples in parallel under your choice of
free software license, such as the GNU General Public License,
to permit their use in free software.

%---------------------------------------------------------------------
\end{document}


\documentclass[11pt]{amsbook}

\usepackage{../HBSuerDemir}	% ------------------------
\usepackage{graphicx,wrapfig,lipsum}
\setcounter{tocdepth}{3}

\usepackage{fancyhdr} % Header/Footer
\pagestyle{fancy}
\thispagestyle{fancy}
\fancyfoot{}
\fancyfoot[L]{\footnotesize
	Freshman Calculus by Suer \& Demir  \textbf{DRAFT} \\
	\LaTeX ~by Haluk Bingol
	\href{http://www.cmpe.boun.edu.tr/~bingol}
	{http://www.cmpe.boun.edu.tr/bingol}
	%\large
	%\footnotesize
	\today}
\fancyfoot[R]{{\thepage} of \pageref{LastPage}}

\begin{document}

% ++++++++++++++++++++++++++++++++++++++
\hPage{b1p2/344}
% ++++++++++++++++++++++++++++++++++++++
\\
\centerline{\thepage}

\section{METHODS OF INTEGRATION}
\par
In calculus there are essentialy two methods of integration :
"change of variables" and "by parts".

\subsection{Integration by change of variable (substitution)}\hfill
\par
Let the indefinite integral
$$I = \int f(x)~dx$$
to be evaluated.
\par
One makes (tries) the substitution
$$x = u(t) \quad \textrm{or} \quad t = u^{-1}(x) = v(x)$$
$$I = \int f(x) ~dx = \int f(u(t)).u'(t)~dt = \int g(t)~dt$$
If the substitution is properly selected the new integral is
more easily integrable than the original one, getting G(t) + c
and replacing t by v(x), one has
$$G(t) + c = G(v(x)) + c = F(x) + c$$
\begin{exmp}
	Evaluate
	$$I = \int \frac{ dx }{(1-x^2)^{3/2}}$$
\end{exmp}
\begin{wrapfigure}{r}{2.2cm}
	\includegraphics[width=0.25\textwidth]{images/b1p2-344-fig01}
	\centering
	sin Triangle\footnote{wrapfig package is used}
	\caption{}\label{wrap-fig:1}
\end{wrapfigure}
{
	\begin{sltn}
		Since square root is not involved, $1-x^2 > 0$ follows
		and the substitution $x = sin \theta $ may work
		$$I = \int \frac{cos \theta ~d\theta}{(1-sin^2\theta)^{3/2}}
		=\int \frac{cos \theta ~d\theta}{cos^3\theta}
		=\int sec^2\theta ~d\theta
		=tan\theta + c$$
	\end{sltn}
	\par
	The result is to be written in terms of x.
	Using the relation $x = sin \theta$ we have
	$$I = tan\theta + c = \frac{x}{\sqrt{1-x^2}} + c$$

\begin{exmp}
	Evaluate
	$$I = \int (x^3-2x+3)^{15}(3x^2-2)~dx$$\\
	\\
\end{exmp}
\begin{sltn}
	Observing that, $D(x^3-2x+3) = (3x^2-2)$ the proper
	substitution is $$u = (x^3-2x+3)\;\textrm{,} \quad du = (3x^2-2)~dx$$
\end{sltn}

% =======================================================
\end{document}