\chapter[Eigenvalues]{Eigenvalues}

\section{Introduction}

The eigenvalues and eigenvectors of a matrix are of fundamental
importance in many matrix problems: for example, in matrix
descriptions of physical models they often represent fundamental
physical quantities, like the energy levels of an atom or the
frequency of vibrations of a string.  In numerical analysis problems,
the eigenvalues determine the convergence conditions of many
iterations schemes, like Jacobi and Gauss-Seidel.

It is, therefore, important to be able to compute quickly and
accurately the eigenvalues and, eventually, the eigenvectors of a
square matrix.  This is an apparently simple problem that could be
tackled using standard nonlinear equations techniques.  In fact, the
eigenvalues $\lambda$ of a $n \times n$ square matrix $A$ are the $n$
roots of the characteristic polynomial of $A$,
%
\begin{equation*}
 \text{det}(A - \lambda \text{Id}) = 0 .
\end{equation*}
%
We could envisage using a root finding algorithm to detect all the
roots of the polynomial and, hence, all the eigenvalues of the matrix
$A$.  This is the procedure that is used in analytical calculations
when finding the eigenvalues of small ($2 \times 2$ or $3 \times 3$)
matrices.  However, roots of high order polynomials may be very
sensitive to the values of the coefficients: therefore, all the
unavoidable numerical errors involved in the lengthy calculation of
the coefficients of the characteristic polynomial of $A$ may well be
amplified at the stage of computing the roots and produce estimates
that bear very little relation to the eigenvalues of $A$.

\begin{Example}
  Polynomials may be badly conditioned, in the sense that a small
  change in the values of just one coefficient changes considerably
  the values of the roots.
\end{Example}

Consider the polynomial
%
\begin{align*}
 p(z) &= (z-1)\,(z-2)\,(z-3)\,(z-4)\,(z-5) \\
      &= -120 + 274 z - 225 z^2 + 85 z^3 - 15 z^4 + z^5 .
\end{align*}
%
The roots of this fifth order polynomial are quite obviously
$\{1,2,3,4,5\}$.  The roots of
%
\begin{equation*}
  p(z) = -120 + 274 z - 225 z^2 + 85 z^3 - 15 z^4 + 1.01 z^5
\end{equation*}
%
are $\{ 0.999, 2.067, 2.624, 4.580 \pm 0.966 \, i \, \}$ . \hfill
\rule{3mm}{3mm}

\smallskip

It is therefore necessary to develop techniques to identify the
eigenvalues of a matrix $A$ that do not rely on finding the roots of
the characteristic polynomial.

If we are interested in just a few eigenvalues then we can use one of
the many variations of the \textit{power method}.  Obtaining the full
spectrum is a much harder problem; an efficient algorithm to do so,
the $QR$-algorithm, was developed by J.G.F. Francis in 1961.  We
describe the power method first and give a relatively simple account
of the $QR$-algorithm after.  The chapter is based in its entirety on
the book by Kincaid and Cheney (chapter 5).%~\cite[Ch.~5]{kincaid96a}.

\section{The power method}

\subsection{The method}

The power method is an iterative procedure designed to compute the
dominant eigenvalue of a matrix and its corresponding eigenvector.
The method is based on two assumptions:

\begin{enumerate}
  %
\item There is a single eigenvalue of maximum modulus.
  %
\item There is a linearly independent set of $n$ eigenvectors, where
  $n$ is the size of the matrix.
  %
\end{enumerate}

According to the first assumption the eigenvalues of $A$ can be
labelled so that
%
\begin{equation*}
 |\lambda_1| > | \lambda_2 | \ge | \lambda_3 | \ge \ldots |\lambda_n|
\end{equation*}
%
According to the second assumption there is a basis of eigenvectors
$\{\bu_1, \bu_2, \ldots, \bu_n\}$ in $\bCn$ such that:
%
\begin{equation*}
 A \bu_j = \lambda_j \bu_j , \qquad 1 \le j \le n .
\end{equation*}

To start the iteration we need to define an initial vector $\bx^{(0)}$
and we have to require that its component along the eigenvector
$\bu_1$ is not zero:
%
\begin{equation*}
 \bx^{(0)} = \sum_{j=1}^n a_j \bu_j , \qquad a_1 \neq 0 .
\end{equation*}
%
Consider now the vector
%
\begin{equation*}
 \bx^{(k)} = A \bx^{(k-1)} = \ldots = A^k \bx^{(0)}.
\end{equation*}
%
Its decomposition on the basis of the eigenvectors is
%
\begin{equation*}
 \bx^{(k)} = \sum_{j=1}^n \lambda_j^k \bu_j =
             \lambda_1^k \left [ a_1 \bu_1 +
             \left ( \frac{\lambda_2}{\lambda_1} \right )^k a_2 \bu_2 +
             \left ( \frac{\lambda_3}{\lambda_1} \right )^k a_3 \bu_3 +
             \ldots
             \left ( \frac{\lambda_n}{\lambda_1} \right )^k a_n \bu_n
              \right ] .
\end{equation*}
%
We note that all the terms $\lambda_j/\lambda_1$, $j=2,3,\ldots,n$,
have modulus strictly smaller than one.  We can write this expression
as
%
\begin{equation}
  \bx^{(k)} = \lambda_1^k ( a_1 \bu_1 + \boldsymbol{\varepsilon}^{(k)} )
  \label{eq:Ev3}
\end{equation}
%
where
%
\begin{equation}
  \boldsymbol{\varepsilon}^{(k)} \equiv  \sum_{j=2}^n
  \left ( \frac{\lambda_j}{\lambda_1} \right )^k a_j \bu_j
  \label{eq:Ev5}
\end{equation}
%
is such that
%
\begin{equation}
  \| \boldsymbol{\varepsilon}^{(k)} \| =
  O \left ( \left | \frac{\lambda_j}{\lambda_1} \right |^k \right )
  \implies
  \lim_{k \to \infty} \| \boldsymbol{\varepsilon}^{(k)} \| = 0.
  \label{eq:Ev2}
\end{equation}
%
As we have assumed that $a_1 \neq 0$ we can write
%
\begin{equation}
  \bx^{(k)} = \lambda_1^k a_1 \bu_1 +
  O \left ( \left | \frac{\lambda_2}{\lambda_1} \right |^k \right ) .
  \label{eq:Ev6}
\end{equation}
%
This expression is at the heart of the power method: at each iteration
the vector $\bx^{(k)}$ aligns itself more and more along the
eigenvector $\bu_1$ and is multiplied by a (complex) number that gets
closer and closer to the eigenvalue $\lambda_1$.  In order to obtain a
quantitative estimate of the eigenvalue we need to pass from vectors
to numbers. As a simple example, we define the ratio
%
\begin{equation*}
 r_k = \frac{\| \bx^{(k+1)} \|}{\| \bx^{(k)} \|} = | \lambda_1 |
        \frac{\| a_1 \bu_1 + \boldsymbol{\varepsilon}^{(k+1)} \|}
             {\| a_1 \bu_1 + \boldsymbol{\varepsilon}^{(k)} \|} .
\end{equation*}
%
Using equation~(\ref{eq:Ev2}) we then have that
%
\begin{equation*}
 \lim_{k \to \infty} r_k = | \lambda_1 | .
\end{equation*}

\noindent
Therefore, a first approximate algorithm to implement the power method
is as follows:

\begin{enumerate}
  %
\item Chose an initial vector $\bx^{(0)}$ such that its component
  along $\bu_1$ is not zero.
  %
\item From the vector at iteration $k$, $\bx^{(k)}$, compute the
  vector at iteration $k+1$, $\bx^{(k+1)} = A \bx^{(k)}$.
  %
\item Compute the ratio $r_k = \| \bx^{(k+1)}\|/\| \bx^{(k)} \|$.
  %
\item If the ratio $r_k$ has converged to $| \lambda_1|$ stop;
  otherwise repeat from step 2.
  %
\end{enumerate}

\noindent
\textbf{Remark 1} - In practice, it is impossible to know a priori
that the vector $\bx_0$ satisfies the condition that its component
along $\bu_1$ is not zero.  However, a random choice of vector will,
with great probability, satisfy it.  In general, however, this is a
moot point: numerical errors will always induce a non-zero coefficient
$a_1$ that will be amplified at each iteration and the method will
ultimately converge to the dominant eigenvalue.

\smallskip

\noindent
\textbf{Remark 2} - From equation~(\ref{eq:Ev3}) we see that $\|
\bx^{(k)} \| \simeq | \lambda_1|^k$.  Therefore if $| \lambda_1 | > 1$
we have that $\| \bx^{(k)} \| \to \infty$ as $k \to \infty$.  This may
induce an overflow problem for sufficiently large $k$.  If, on the
other hand, $| \lambda_1 | < 1$, then $\| \bx^{(k)} \| \to 0$ as $k
\to \infty$.  This may induce an underflow problem for sufficiently
large $k$.  To avoid either pitfall it is convenient to normalise the
vectors $\bx^{(k)}$ at each iteration (or after a small number of
iterations).  A suitable algorithm is:

\begin{enumerate}
  %
\item Chose an initial vector $\bx^{(0)}$ such that its
  component along $\bu_1$ is not zero and $\| \bx^{(0)} \| = 1$.
  %
\item From the vector at iteration $k$, $\bx^{(k)}$, compute the
  vector at iteration $k+1$, $\bx^{(k+1)} = A \bx^{(k)}$.
  %
\item Compute  $r_k = \| \bx^{(k+1)}\|$ and assign $\|
  \bx^{(k+1)}\|/r_k \to \| \bx^{(k+1)}\|$.
  %
\item If the ratio $r_k$ has converged to $|\lambda_1|$ stop;
  otherwise repeat from step 2.
  %
\end{enumerate}

\smallskip

\noindent
\textbf{Remark 3} - The method as described so far returns only the
modulus of the eigenvalue.  However, it is fairly simple to modify it
so that it returns the eigenvalue itself.  We lose information on the
phase of the eigenvalue when we compute the ratio $r_k$ using the norm
of $\| \bx^{(k+1)}\|$ and $\| \bx^{(k)}\|$. We could use, instead, a
linear function $\phi$ from $\bC^n \to \bC$, that is, $\phi(\alpha \bx
+ \beta \by) = \alpha \phi(\bx) + \beta \phi(\by)$. (Note that
$\|\bx\|$ is not linear!)  A possible choice for $\phi(\bx)$ is, for
example, the sum of the components of $\bx$.  We can define the ratio
$r_k$ in terms of the functional $\phi(\bx)$ as
%
\begin{equation}
 r_k = \frac{\phi( \bx^{(k+1)} )}{\phi(\bx^{(k)})} = \lambda_1
        \frac{a_1 \phi(\bu_1) + \phi(\boldsymbol{\varepsilon}^{(k+1)})}
             {a_1 \phi(\bu_1) + \phi(\boldsymbol{\varepsilon}^{(k)})} .
 \label{eq:Ev7}
\end{equation}
%
Using equation~(\ref{eq:Ev2}) and the fact that $\lim_{\| \bx \| \to
  0} \phi(\bx) = 0$ we then have that
%
\begin{equation*}
 \lim_{k \to \infty} r_k = \lambda_k.
\end{equation*}
%
The algorithm that implements this scheme is as follows:

\begin{enumerate}
  %
\item Chose an initial vector $\bx^{(0)}$ such that its
  component along $\bu_1$ is not zero.
  %
\item At each iteration $k=0,1,2,\ldots$ compute $d_k = \|
  \bx^{(k)}\|$ and assign $\| \bx^{(k)}\|/d_k \to \| \bx^{(k)}\|$.
  %
\item From the vector at iteration $k$, $\bx^{(k)}$, compute the
  vector at iteration $k+1$, $\bx^{(k+1)} = A \bx^{(k)}$.
  %
\item Compute  $r_k = \phi(\bx^{(k+1)})/\phi(\bx^{(k)})$.
  %
\item If the ratio $r_k$ has converged to $\lambda_1$ stop;
  otherwise repeat from step 2.
  %
\end{enumerate}

\subsection{Rate of convergence}

We now show that the power method converges linearly.  We define $\mu
= {\lambda_2}/{\lambda_1}$.  This parameter controls the convergence
rate of the method.  We apply Taylor's theorem to
equation~(\ref{eq:Ev7}) and use equation~(\ref{eq:Ev2}) to obtain
%
\begin{equation*}
 r_k = \lambda_1
        \frac{a_1 \phi(\bu_1) + \phi(\boldsymbol{\varepsilon}^{(k+1)})}
             {a_1 \phi(\bu_1) + \phi(\boldsymbol{\varepsilon}^{(k)})}
     = \lambda_1
        \left [ 1 - \frac{\phi (\boldsymbol{\varepsilon}^{(k)} )}{a_1 \phi(\bu_1)}\right ] \,
        + O(\mu^{k+1}) .
\end{equation*}
%
Therefore the relative error of the approximation, defined as
%
\begin{equation}
  \epsilon^{(k)} \equiv \left |
    \frac{r_k - \lambda_1}{\lambda_1} \right | ,
  \label{eq:Ev4}
\end{equation}
%
is given by
%
\begin{equation}
  \epsilon^{(k)} =
  \left | \frac{\phi (\boldsymbol{\varepsilon}^{(k)} )}{a_1 \phi(\bu_1)} \right|  + O(\mu^{k+1}) =  c_k \mu^k
  \label{eq:Ev9}
\end{equation}
%
where $\{c_k\}$ is a bounded sequence.  Note that this equation
implies that $\epsilon^{(k)} \propto \mu \epsilon^{(k+1)}$, i.e.\ that
the rate of decrease of the error is linear.  In fact one can show
that
%
\begin{equation*}
  \frac{r^{(k+1)} - \lambda_1}{\lambda_1} = (c + \delta_k)
  \frac{r^{(k)} - \lambda_1}{\lambda_1}
\end{equation*}
%
where $|c| < 1$ and $\lim_{k \to \infty} \delta_k = 0$.

\smallskip

\smallskip

\noindent \textbf{Remark 1} - The condition that there should be a
single eigenvalue of maximum modulus should be qualified.  If the
eigenvalue is multiple then the power method will converge to it, but
it will not be able to return the eigenvectors associated to the
(multiple) eigenvalue; it will just return a random vector in the
eigenspace of the eigenvalue of maximum modulus.  If, on the other
hand, the matrix has distinct eigenvalues all with the same maximum
modulus (a rather common situation in problems with symmetry), the
method may fail to converge.

\smallskip

\noindent \textbf{Remark 2} - The convergence can be improved using
Aitken's acceleration, a technique similar to Richardson's
extrapolation  that uses the fact that the error decreases linearly to
improve the convergence.

\subsection{Variations on the power method}

\subsubsection{The inverse power method}

We can use a variation of the power method, called the \textit{inverse
power method}, to compute the smallest eigenvalue of a non singular
(and well conditioned) matrix.  The algorithm is based on the result
of linear algebra that if $\lambda$ is an eigenvalue of a nonsingular
matrix $A$, then $\lambda^{-1}$ is an eigenvalue of $A^{-1}$.

We assume as before that the eigenvectors of $A$ form a basis for
$\bC^n$, but we now require that the eigenvalue of smallest modulus is
isolated, i.e.\ that we can label the eigenvalues of $A$ so that
%
\begin{equation*}
  |\lambda_1| \ge |\lambda_2| \ge \ldots \ge |\lambda_{n-1}| >
  |\lambda_n| > 0 .
\end{equation*}
%
Therefore the eigenvalues of $A^{-1}$ are ordered as
%
\begin{equation*}
  |\lambda_n^{-1}| > |\lambda_{n-1}^{-1}| \ge \ldots \ge
  |\lambda_{1}^{-1}| > 0 .
\end{equation*}
%
We are now in the position to compute $\lambda_n^{-1}$ using the power
method applied to $A^{-1}$.

\smallskip

\noindent
\textbf{Remark} - It is \underline{not} a good idea to compute
$A^{-1}$ first and then use $\bx^{(k+1)} = A^{-1} \bx^{(k)}$.   It is
better to obtain $\bx^{(k+1)}$ by solving the linear system
%
\begin{equation*}
  A \bx^{(k+1)} = \bx^{(k)}
\end{equation*}
%
using an efficient (and accurate) linear solver.   The $LU$
decomposition is an excellent candidate for this problem.

\subsection{The shifted inverse power method}

The shifted inverse power method is used to compute the eigenvalue of
$A$ closest to a given complex number $\nu$.   Suppose that there is a
real number $\epsilon$ such that there is only one eigenvalue of $A$,
$\lambda_j$, such that
%
\begin{equation}
  |\lambda_j - \nu| \le \epsilon \quad \text{and} \quad
  |\lambda_i - \nu| > \epsilon \; \forall \, i \ne j.
  \label{eq:Ev10}
\end{equation}

Consider now the ``shifted'' matrix $A - \nu \text{Id}$: its
eigenvalues are given by $\lambda_i - \nu$.  Equation~(\ref{eq:Ev10})
implies that we can find $\lambda_j - \nu$ by applying the inverse
power method to the matrix $A - \nu \text{Id}$.  This algorithm is
called the shifted inverse power method.

\section{The \texorpdfstring{$QR$}{QR}-algorithm}

\subsection{Introduction}

The methods to compute the full spectrum of a matrix are based on the
idea that finding the eigenvalues is an easy problem for certain
classes of matrices, e.g.\ diagonal and triangular matrices: their
eigenvalues are the elements along the diagonal.  The trick
is to find operations that transform a matrix in an ``easy'' form
while not altering the spectrum.    The first step in doing so is to
identify classes of matrices that have the same spectrum and classes
of operations that do not alter the spectrum.

\begin{definition}
Two square matrices are similar if there exists a nonsingular matrix
$P$ such that $B = P A P^{-1}$.   We say that $B$ is obtained from $A$
using a similarity transformation.   If $P$ is unitary, i.e.\ $P =
P^\dagger$, the two matrices are unitary similar.
\end{definition}

\begin{theorem}
  \label{thr:1}
  Similar matrices have the same eigenvalues.
\end{theorem}
\noindent
\textbf{Proof}

\begin{equation*}
  A \bx = \lambda \bx \quad \Leftrightarrow \quad
  (P A P^{-1}) (P \bx) = P A \bx = \lambda (P \bx).
\end{equation*}
So if $\bx$ is an eigenvector of with eigenvalue $\lambda$ then $A$ $P
\bx$ is an eigenvector of $P A P^{-1}$ with the same
eigenvalue. Moreover, $P$ provides a bijection between the
eigenvectors of $A$ and the eigenvectors of $P A P^{-1}$. \hfill
\rule{3mm}{3mm}

\noindent
In other words, similar matrices have the same spectrum and similarity
transformations do not change the spectrum of a matrix.

\subsection{Schur's theorem}

Theorem~\ref{thr:1} suggests a strategy to find the eigenvalues of
matrix: find a similarity transformation $B = P A P^{-1}$ such that
$B$ is triangular.  The following theorem states that this strategy is
always (theoretically) possible.

\begin{theorem}[Schur]
  Every square matrix is unitary similar to a triangular matrix.
\end{theorem}

\noindent
In order to prove this theorem we need the following two lemmas. In
the remainder of this section, we always use the inner product
$(\bu,\bv)=\bu^\dagger\bv$ on $\bC^n$ and the associated norm
$\|\bu\|=\sqrt{\bu^\dagger\bu}$.

\begin{lemma}
  \label{ax:1}
  The matrix $\Id - \bu \bu^{\dagger}$ is unitary if and only if
  $\|\bu\| = \sqrt{2}$ or $\bu=0$.
\end{lemma}
\noindent
\textbf{Proof}

\begin{equation*}
  (\Id - \bu \bu^{\dagger})(\Id - \bu \bu^{\dagger})^\dagger
  = (\Id - \bu \bu^{\dagger})^2 = \Id - 2 \bu \bu^{\dagger}
  + \bu \bu^{\dagger} \bu \bu^{\dagger}
  = \Id +(\bu^{\dagger} \bu - 2) \bu \bu^{\dagger}
\end{equation*}
\null \hfill \rule{3mm}{3mm}

\begin{lemma}
  \label{ax:2}
  Let $\bx$ and $\by$ be two vectors such that $\|\bx\| = \|\by\|$ and
  $(\bx,\by)$ is real.  Then there exists an unitary matrix $U = \Id -
  \bu \bu^{\dagger}$ such that $U \bx = \by$.  The vector $\bu$ is
  given by:
  \begin{equation*}
    \bu = \sqrt{2} \frac{\bx - \by}{\|\bx - \by\|} .
  \end{equation*}
\end{lemma}
\noindent
\textbf{Proof}

\begin{align*}
  U \bx &= \left(\Id - \frac{2}{\|\bx - \by\|^2}(\bx - \by)(\bx - \by)^\dagger
  \right) \bx \\
  &= \bx - \frac{2 (\|\bx\|^2 - (\by,\bx))}{
    \|\bx\|^2 - (\bx,\by) - (\by,\bx) + \|\by\|^2} (\bx - \by) = \by
\end{align*}
\null \hfill \rule{3mm}{3mm}

\noindent
\textbf{Remark} - The matrix $U$ is called a \textit{Householder
  reflection} and its action can be interpreted geometrically as:
``the vector $\by$ can be obtained by reflecting the vector $\bx$ with
respect to a plane orthogonal to the vector $\bu$''.

\noindent
\textbf{Proof of Schur's theorem} - We prove this theorem for two
reasons: firstly, the proof is an example of the techniques used to
transform a given matrix in an ``easy'' matrix.  Secondly, the proof
clearly shows why this theorem is of little practical use.

In a nutshell the proof consists in constructing a succession of
unitary transformations that change the given $n \times n$ matrix $A$
in upper triangular form, one row and column at a time.

The proof of the theorem is by induction on the size $n$ of the
matrix.  The theorem is clearly true for $n=1$.  We now suppose that
it is true for all matrices of order $n-1$ and prove that it must also
hold for any matrix $A$ of order $n$.  The idea of the proof is to use
Lemma~\ref{ax:2} to find a unitary transformation $U$ that acts on the
first column of $A$ and transforms it in a scalar multiple of the
vector $\be_1 = (1,0,0,\ldots,0)$, i.e.
%
\begin{equation}
  U \left ( \begin{array}{c|ccc}
      a_{1 1} & a_{1 2} & \ldots & a_{1 n} \\ \hline
      a_{2 1} & a_{2 2} & \ldots & a_{2 n} \\
      \vdots & & \ddots & \\
      a_{n 1} & a{_n 2} & \ldots & a_{n n} \\
    \end{array} \right ) U^{\dagger} =
  \left ( \begin{array}{c|ccc}
      \mu & & \bw & \\ \hline
      0 & & & \\
      \vdots & & A_{n-1} & \\
      0 & & & \\
    \end{array} \right )
  \label{eq:Ev13}
\end{equation}
%
where $A_{n-1}$ is an $(n-1) \times (n-1)$ matrix, $\bw$ is a row
vector of dimension $n-1$ and $\mu$ is a number.  Unfortunately, in
order to build the transformation matrix $U$ we need one eigenvalue
$\lambda$ of $A$ with the corresponding eigenvector $\bv=(v_1, v_2,
\ldots, v_n)$, chosen such that $\| \bv \|=1$.  Following
Lemma~\ref{ax:2} we define the (complex) number
%
\begin{equation*}
  \beta =
  \begin{cases}
    \dfrac{v_1}{|v_1|} & v_1 \neq 0 \\*[4mm] 1 & v_1 = 0
  \end{cases}
\end{equation*}
%
and the vector
%
\begin{equation*}
  \bu = \frac{\sqrt{2}}{\|\bv - \beta \be_1\|} (\bv - \beta \be_1) .
\end{equation*}
%
Note that $(\bv, \beta \be_1) = v_1^{*} v_1/|v_1|=|v_1| \in \bR$ and
that $\|\bv\| = \|\beta \be_1\| = 1$ so that both the hypotheses of
Lemma~\ref{ax:2} are satisfied.  Therefore, the matrix
%
\begin{equation}
  U = \text{Id} - \bu \bu^{\dagger}
  \label{eq:Ev11}
\end{equation}
%
is unitary and such that
%
\begin{equation}
  U \bv = \beta \be_1 \implies U^{\dagger} \be_1 = \frac{1}{\beta} \, \bv .
  \label{eq:Ev12}
\end{equation}
%
The matrix defined by this equation is the transformation required
for~(\ref{eq:Ev13}) to be valid.  To verify that this is the case we
note that the product of any $n \times n$ matrix with $\be_1$ is its
first column vector.  We therefore proceed to verify that the first
column vector of $U A U^{\dagger}$ is a scalar multiple of $\be_1$:
%
\begin{equation*}
  U A U^{\dagger} \, \be_1 = U A \, \frac{1}{\beta} \bv =
  \frac{1}{\beta} U \lambda \bv = \frac{\lambda}{\beta} U \bv =
  \lambda \be_1 .
\end{equation*}
%
Hence the application of $U$ defined in~(\ref{eq:Ev11}) to $A$ reduces
this matrix to the form~(\ref{eq:Ev13}) with $\mu = \lambda$.

To complete the proof we note that by the induction hypothesis there
is a unitary matrix $Q_{n-1}$ such that $Q_{n-1} A_{n-1}
Q^{\dagger}_{n-1}$ is triangular.  Therefore, the unitary
transformation that reduces $A$ to triangular form is
%
\begin{equation*}
  V =
  \left ( \begin{array}{c|ccc}
      1 & & 0 \ldots 0 & \\ \hline
      0 & & & \\
      \vdots & & Q_{n-1} & \\
      0 & & & \\
    \end{array} \right ) \, U
\end{equation*}
\null \hfill \rule{3mm}{3mm}

\smallskip

\noindent
At each step of the proof we assume that the eigenvalues exist and use
them, but we do not address the question of how to compute them.  In
other words, if we know the eigenvalues then we can apply the method
described in the proof to find the unitary transformation required to
put $A$ in upper triangular form.  However, if we do not know them we
cannot apply the procedure and we are no nearer to solving the
original eigenvalue problem.

\noindent
\textbf{Remark} - One could think of using the power method to find
one eigenvalue of $A$ and thus build $U$ and $A_{n-1}$, and then apply
the same trick to $A_{n-1}$ and so on.  This method is called
\textit{deflation} and it works, but is rather sensitive to numerical
errors.

\begin{corollary}
  Every Hermitian matrix is unitary similar to a diagonal matrix.
\end{corollary}

\noindent
\textbf{Proof} - If $A$ is Hermitian then $A = A^{\dagger}$.  Let $U$
be the unitary matrix such that $U A U^{\dagger}$ is upper triangular.
Then $(U A U^{\dagger})^{\dagger}$ is lower triangular.  But
%
\begin{equation*}
  (U A U^{\dagger})^{\dagger} =
  (U^{\dagger})^{\dagger} A^{\dagger} U^{\dagger} =
  U A U^{\dagger} .
\end{equation*}
%
Thus, the matrix $U A U^{\dagger}$ is both upper and lower triangular;
hence, it must be a diagonal matrix.

\subsection{Householder's \texorpdfstring{$QR$}{QR}-factorisation}

In order to introduce the $QR$-algorithm of Francis to find the
eigenvalues of a matrix, we must first take a detour and discuss
\textit{orthogonal factorisations} of a matrix.  An orthogonal
factorisation consists in writing a given matrix $A$ as the product of
other matrices, some of which are orthogonal.  The most useful
orthogonal factorisation if the $QR$-factorisation developed by Alston
Householder: its objective is to factor an $m \times n$ matrix $A$
into a product
%
\begin{equation}
  A = Q \, R
  \label{eq:Ev15}
\end{equation}
%
where $Q$ is an $m \times m$ unitary matrix and $R$ is an $m \times n$
upper triangular matrix.

The $QR$-factorisation is used not only to find the eigenvalues of a
matrix, but also to solve least-squares problems and ill-posed
problems.  The algorithm produces the factorisation
%
\begin{equation}
  Q^{\dagger} A = R
  \label{eq:Ev16}
\end{equation}
%
where $Q^{\dagger}$ is built step by step as the product of unitary
matrices that at step $k=0,1,\ldots,m-1$ have the form
%
\begin{equation}
  U_k =
  \begin{pmatrix}
    \Id_{k-1} & 0 \\ 0 & \Id_{m-k+1} - \bu \bu^{\dagger}
  \end{pmatrix},
  \quad k = 1,2, \ldots, n-1,
  \label{eq:Ev17}
\end{equation}
%
with $\Id_k$ the $k \times k$ identity matrix and $\bu \in
\bC^{m-k+1}$.  These matrices are called \textit{reflections} or
\textit{Householder transformations}: the aim of each of them is to
transform the matrix $A$ into a matrix that looks more and more like
$R$, in a similar way that a triangular matrix is constructed in the
proof of Schur's theorem.

At the first step we wish to determine a vector $\bu \in \bC^m$ such
that $\Id_m - \bu^{\dagger} \bu$ is unitary and $(\Id_m -
\bu^{\dagger} \bu) A$ has first column of the form suitable for an
upper triangular matrix, i.e.\ of the type $(\beta, 0, \ldots, 0)^T$.
We indicate with $\ba_1$ the first column of $A$ and with $\be_1$ the
vector $(1,0,\ldots,0)$.  Following Lemma~\ref{ax:2} we define the
(complex) number
%
\begin{equation*}
  \beta =
  \begin{cases}
    -\dfrac{a_{11}}{|a_{11}|} \|\ba_1\|_2 & a_{11} \neq 0 \\*[4mm]
    \| \ba_1 \|_2 & a_{11} = 0
  \end{cases}
\end{equation*}
%
and the vector
%
\begin{equation*}
  \bu = \frac{\sqrt{2}}{\|\ba_1 - \beta \be_1\|} (\ba_1 - \beta \be_1) .
\end{equation*}
%
By Lemma~\ref{ax:2} the matrix
%
\begin{equation}
  U_1 = \text{Id} - \bu \bu^{\dagger}
  \label{eq:Ev18}
\end{equation}
%
is unitary and such that
%
\begin{equation}
  U_1 \ba_1 = \beta \be_1 .
  \label{eq:Ev19}
\end{equation}
%
Using the unitary matrix $U$ defined in equation~(\ref{eq:Ev18}) we
have that the matrix $U_0 A$ has form:
%
\begin{equation*}
  U_1 A =
  \left ( \begin{array}{c|ccc}
      \beta & & \bw_1 & \\ \hline
      0 & & & \\
      \vdots & & \tilde{A_1} & \\
      0 & & & \\
    \end{array} \right )  \equiv R_1
\end{equation*}
%
We can now repeat the same procedure for the first column of the
matrix $A_1$ and build the unitary operator $U_k$ defined in
equation~(\ref{eq:Ev17}) with $k=2$ and so on.  At stage $k$ in the
process, we shall have multiplied $A$ on the left by $k$ unitary
matrices and the result will be a matrix having its first $k$ columns
of the correct form, i.e.\ with $0$'s below the diagonal:
%
\begin{equation*}
  U_k \, U_{k-1} \, U_{k-2} \, \ldots \, U_1 A =
  \left ( \begin{array}{c|c} J & H \\ \hline 0 & W \end{array} \right )
\end{equation*}
%
where $J$ is an upper triangular $k \times k$ matrix, $0$ is an $(m-k)
\times k$ null matrix, $H$ is $k \times (n-k)$ and $W$ is $(m-k)
\times (m-k)$.

The process terminates at $k=(n-1)$ when $R_{n-1}$ is in the correct
form for equation~(\ref{eq:Ev15}), i.e.\ $R= R_{n-1}$.  Moreover,
%
\begin{equation*}
  Q^{\dagger} = U_{n-1} \ldots U_{1} \implies
  Q = U_1 \ldots U_{n-1} .
\end{equation*}

\subsection{The \texorpdfstring{$QR$}{QR}-Algorithm of Francis}

\subsubsection{The basic algorithm}

The $QR$-Algorithm of Francis is an iterative procedure that uses a
variant of the $QR$-factorisation to find the eigenvalues of a $n
\times n$ matrix $A$ by reducing to triangular form.

We can interpret the $QR$-algorithm as an extension of the power
method - from Epperson (2002):%~\cite{epperson02a}:

\begin{quote}
  One of the primary drawbacks to the power methods is that they work
  on a single vector at a time; if we want all the eigenpairs of a
  matrix, then we spend a lot of computation time in deflation [...].
  Why not do the power method on several vectors at once?  Let $Z_0$
  be an initial guess matrix and do the computation
  %
  \begin{equation*}
    Y_{k+1} = A Z_k, \qquad Z_{k+1} = Y_{k+1} D_{k+1}
  \end{equation*}
  %
  where $D_{k+1}$ is some properly chosen diagonal scaling matrix.
  The problem is that this won't work; the repeated multiplications by
  $A$ will simply cause all of the columns of $Y_{k+1}$ to line up on
  the direction of the dominant eigenvalue and so we will converge to
  $n$ copies of the dominant eigenpair - hardly an improvement on the
  power method.

  This algorithm does have something to recommend it, though.  If the
  $Z_k$ matrices are all \textit{orthogonal} matrices, then we get
  something that might work, because the orthogonality of the columns
  of $Z_k$ would prevent the columns of $Y_{k+1}$ from all lining up
  in the same direction.
\end{quote}

The $QR$-algorithm implements this idea using the $QR$ factorisation.
Additional information on its relation to the power method and
deflation techniques can be found in Burden \& Faires (2005,
$8^{\textrm{th}}$ ed.).

The $QR$-algorithm consists of the following steps:

\begin{enumerate}
  %
\item Start the iteration - Iteration index $k=1$, working matrix $A_k
  = A$.
  %
\item Compute the $QR$ factorisation of $A_k = Q_k R_k$, where $Q_k$
  is unitary and $R$ is upper triangular with non-negative diagonal.
  %
\item Compute $A_{k+1} = R_k Q_k$.  Under appropriate hypotheses, the
  matrix $A_{k+1}$ tends, as $k \to \infty$, to an upper triangular
  matrix whose diagonal elements are the eigenvalues of $A$ .  The
  procedure is repeated from step (2) until convergence has been
  attained.
\end{enumerate}

\noindent
\textbf{Remark 1} - All the matrices $A_k$ are similar to $A$:
%
\begin{equation*}
  A_k = Q_k R_k = (Q_k R_k) \, (Q_k Q_k^{\dagger}) =
  Q_k \, ( R_k Q_k) \, Q_k^{\dagger} =
  Q_k A_{k+1} Q_k^{\dagger} .
\end{equation*}

\noindent
\textbf{Remark 2} - If the matrix $A$ is real, then all the matrices
$A_k$ are real.  Therefore, if $A$ has non-real eigenvalues then at
most $A_k$ will converge to a ``triangular'' matrix with $2 \times 2$
minors on its diagonal.

\noindent
\textbf{Remark 3} - In order to reduce the burden of computation and
increase the speed of convergence the $QR$-algorithm is coupled with
other techniques, namely the reduction to upper Hessenberg form and an
origin shift (\textit{Shifted $QR$-factorisation}).

\section*{Further reading}

Topics covered here are also covered in
\begin{itemize}
\item Chapter 9 of Linz \& Wang, \textit{Exploring Numerical Methods}
  (QA297 LIN),
\item Chapter 5 of S{\"u}li \& Mayers, \textit{An Introduction to
    Numerical Analysis} (not in library),
\item Chapters 7 and 8 of the extremely detailed Golub \& Van Loan,
  \textit{Matrix Computations} (QA263 GOL).
\end{itemize}


\section{Minors and cofactors}

\begin{outcome}
  \begin{enumerate}
  \item Compute minors and cofactors of matrices.
  \item Use cofactor expansion to compute the determinant of an
    $n\times n$-matrix.
  \end{enumerate}
\end{outcome}

Determinants of larger matrices can be computed in terms of the
determinants of smaller matrices. We begin with the following
definition.

\begin{definition}{The $\ijth$ minor of a matrix}{ijth-minor}
  Let $A$ be an $n\times n$-matrix. The $\ijth$ \textbf{minor}%
  \index{matrix!minor}%
  \index{determinant!minor of a matrix}%
  \index{minor} of $A$, denoted by $\minor{A}{ij}$, is the
  determinant of the $(n-1)\times(n-1)$-matrix that is obtained by
  deleting the $i\th$ row and the $j\th$ column of $A$.
\end{definition}

Hence, there is a minor associated with each entry of $A$. The
following example illustrates this definition.

\begin{example}{Finding minors of a matrix}{finding-minors}
  Let
  \begin{equation*}
    A = \begin{mymatrix}{rrr}
      1 & 2 & 3 \\
      4 & 3 & 2 \\
      3 & 2 & 1
    \end{mymatrix}
  \end{equation*}
  Find the minors $\minor{A}{12}$ and $\minor{A}{33}$.
\end{example}

\begin{solution}
  First we will find $\minor{A}{12}$. By definition, this is the
  determinant of the $2\times 2$-matrix that results when we delete
  the first row and the second column of $A$. This minor is given by
  \begin{equation*}
    \minor{A}{12}
    ~=~
    \begin{absmatrix}{ccc}
      \strikeh{3.2em}{1} & \strikev{3.2em}{2} & 3 \\
      4 & 3 & 2 \\
      3 & 2 & 1 \\
    \end{absmatrix}
    ~=~
    \begin{absmatrix}{rr}
      4 & 2 \\
      3 & 1
    \end{absmatrix}
    ~=~ 4\cdot 1 - 3\cdot 2
    ~=~ -2.
  \end{equation*}
  Similarly, $\minor{A}{33}$ is the determinant of the
  $2\times 2$-matrix that is obtained by deleting the third row and
  the third column of $A$. This minor is therefore
  \begin{equation*}
    \minor{A}{33}
    ~=~
    \begin{absmatrix}{ccc}
      1 & 2 & \strikev{3.2em}{3} \\
      4 & 3 & 2 \\
      \strikeh{3.2em}{3} & 2 & 1 \\
    \end{absmatrix}
    ~=~
    \begin{absmatrix}{rr}
      1 & 2 \\
      4 & 3
    \end{absmatrix}
    ~=~ -5.
  \end{equation*}
\end{solution}

We now define the $\ijth$ cofactor of a matrix $A$, which is either
plus or minus the $\ijth$ minor.

\begin{definition}{The $\ijth$ cofactor of a matrix}{ijth-cofactor}
  Suppose $A$ is an $n\times n$-matrix. The $\ijth$ \textbf{cofactor}%
  \index{matrix!cofactor of}%
  \index{determinant!cofactor of a matrix}%
  \index{cofactor}, denoted by $\cofactor{A}{ij}$, is
  defined to be
  \begin{equation*}
    \cofactor{A}{ij} = (-1)^{i+j} \minor{A}{ij}
  \end{equation*}
\end{definition}

In other words, the $\ijth$ cofactor is equal to the corresponding
minor if $i+j$ is even, and the negative of the minor if $i+j$ is
odd. For remembering the signs, the following picture is sometimes
helpful:
\begin{equation*}
  \begin{mymatrix}{cccc}
    + & - & + & - \\
    - & + & - & + \\
    + & - & + & - \\
    - & + & - & + \\
  \end{mymatrix}.
\end{equation*}

\begin{example}{Finding cofactors of a matrix}{finding-cofactors}
  Consider the matrix
  \begin{equation*}
    A=\begin{mymatrix}{rrr}
      1 & 2 & 3 \\
      4 & 3 & 2 \\
      3 & 2 & 1
    \end{mymatrix}
  \end{equation*}
  Find the cofactors $\cofactor{A}{12}$ and $\cofactor{A}{33}$.
\end{example}

\begin{solution}
  We have already computed the corresponding minors in
  Example~\ref{exa:finding-minors}. For the cofactors, we have:
  \begin{eqnarray*}
    \cofactor{A}{12} &=& (-1)^{1+2}\minor{A}{12} ~=~ -\minor{A}{12} ~=~ -(-2) ~=~ 2, \\
    \cofactor{A}{33} &=& (-1)^{3+3}\minor{A}{33} ~=~ +\minor{A}{33} ~=~ +(-5) ~=~ -5.
  \end{eqnarray*}
  Note that $1+2$ is odd, so $\cofactor{A}{12}=-\minor{A}{12}$. On the
  other hand, $3+3$ is even, so $\cofactor{A}{33}=\minor{A}{33}$.
\end{solution}

You may wish to find the remaining cofactors of the above
matrix. Remember that there is a cofactor for every entry in the
matrix.

We have now established the tools we need to find the determinant of
an $n\times n$-matrix.

\begin{definition}{The determinant of an $n\times n$-matrix}{n-by-n-determinant}
  Let $A$ be an $n\times n$-matrix. Then $\det(A)$ is calculated by
  picking a row (or column) and taking the product of each entry in
  that row (column) with its cofactor and adding these products
  together. This process is known as \textbf{expanding along the
    $i\th$ row (or column)}%
  \index{determinant!expanding along row or column}.
  \bigskip

  In formulas, the process of expanding along the $i\th$ row is given
  as follows:
  \begin{equation*}
    \det(A) =
    a_{i1}\cofactor{A}{i1} + a_{i2}\cofactor{A}{i2} + \ldots + a_{in}\cofactor{A}{in}.
  \end{equation*}
  Similarly, the process of expanding along the $j\th$ column is:
  \begin{equation*}
    \det(A) =
    a_{1j}\cofactor{A}{1j} + a_{2j}\cofactor{A}{2j} + \ldots + a_{nj}\cofactor{A}{nj}.
  \end{equation*}

\end{definition}

When calculating the determinant, you can choose to expand any row or
any column. Regardless of which row or column you expand, you will
always get the same number, which is the determinant of the matrix
$A$.  This method of evaluating a determinant by expanding along a row
or a column is also called \textbf{cofactor expansion}%
\index{determinant!cofactor expansion}%
\index{cofactor expansion} or \textbf{Laplace expansion}%
\index{determinant!Laplace expansion}%
\index{Laplace expansion}.

\begin{example}{Finding a determinant by cofactor expansion}{determinant-three-by-three}
  Find $\det(A)$ using the method of cofactor expansion, where
  \begin{equation*}
    A = \begin{mymatrix}{rrr}
      1 & 2 & 3 \\
      4 & 3 & 2 \\
      3 & 2 & 1
    \end{mymatrix}.
  \end{equation*}
\end{example}

\begin{solution}
  First, we will calculate $\det(A)$ by expanding along the first
  column.  Using Definition~\ref{def:n-by-n-determinant}, the
  determinant is
  \begin{eqnarray*}
    \det(A)
    &=& a_{11}\cofactor{A}{11} + a_{21}\cofactor{A}{21} + a_{31}\cofactor{A}{31} \\
    &=& a_{11}\minor{A}{11} - a_{21}\minor{A}{21} + a_{31}\minor{A}{31} \\\\[-2ex]
    &=&
        1 \begin{absmatrix}{rr}
          3 & 2 \\
          2 & 1 \\
        \end{absmatrix}
        - 4 \begin{absmatrix}{rr}
          2 & 3 \\
          2 & 1 \\
        \end{absmatrix}
        + 3 \begin{absmatrix}{rr}
          2 & 3 \\
          3 & 2
        \end{absmatrix} \\\\[-1ex]
    &=& 1 (-1) - 4 (-4) + 3 (-5) \\
    &=& 0.
  \end{eqnarray*}
  As mentioned in Definition~\ref{def:n-by-n-determinant}, we
  can choose to expand along any row or column. Let's try now by
  expanding along the second row. The calculation is as follows.
  \begin{equation*}
    \det(A)
    ~=~ -4 \begin{absmatrix}{rr}
      2 & 3 \\
      2 & 1
    \end{absmatrix}
    + 3 \begin{absmatrix}{rr}
      1 & 3 \\
      3 & 1
    \end{absmatrix}
    - 2 \begin{absmatrix}{rr}
      1 & 2 \\
      3 & 2
    \end{absmatrix}
    ~=~ -4(-4) + 3(-8) - 2(-4)
    ~=~ 0.
  \end{equation*}
  You can see that for both methods, we obtained $\det(A) = 0$.
\end{solution}

You should try to compute the above determinant by expanding along
other rows and columns. This is a good way to check your work, because
you should come up with the same number each time!

\begin{theorem}{The determinant is well-defined}{well-defined-determinant}
  Expanding an $n\times n$-matrix along any row or column always gives
  the same answer, which is the determinant.
\end{theorem}

\begin{example}{Determinant of a four by four matrix}{four-by-four-determinant}
  Calculate
  $\begin{absmatrix}{rrrr}
    1 & 2 & 3 & 0 \\
    2 & 4 & 2 & 3 \\
    0 & 3 & 0 & 5 \\
    3 & 0 & 3 & 2
  \end{absmatrix}$.
\end{example}

\begin{solution}
  Using the cofactor method, we can expand this determinant along any
  row or column. But notice that the third row contains two zeros.
  This makes the cofactor expansion particularly convenient. So let us
  expand along the third row. We have:
  \begin{equation*}
    \begin{absmatrix}{rrrr}
      1 & 2 & 3 & 0 \\
      2 & 4 & 2 & 3 \\
      0 & 3 & 0 & 5 \\
      3 & 0 & 3 & 2
    \end{absmatrix}
    =
    0 \begin{absmatrix}{rrr}
      2 & 3 & 0 \\
      4 & 2 & 3 \\
      0 & 3 & 2
    \end{absmatrix}
    -3 \begin{absmatrix}{rrr}
      1 & 3 & 0 \\
      2 & 2 & 3 \\
      3 & 3 & 2
    \end{absmatrix}
    +0 \begin{absmatrix}{rrr}
      1 & 2 & 0 \\
      2 & 4 & 3 \\
      3 & 0 & 2
    \end{absmatrix}
    -5 \begin{absmatrix}{rrr}
      1 & 2 & 3 \\
      2 & 4 & 2 \\
      3 & 0 & 3
    \end{absmatrix}.
  \end{equation*}
  Note that we only need to compute two of the $3\times 3$
  determinants, since the remaining two are multiplied by $0$. We can
  compute each $3\times 3$ determinant using the method of
  Definition~\ref{def:3-by-3-determinant}. We find:
  \begin{equation*}
    \begin{absmatrix}{rrrr}
      1 & 2 & 3 & 0 \\
      2 & 4 & 2 & 3 \\
      0 & 3 & 0 & 5 \\
      3 & 0 & 3 & 2
    \end{absmatrix}
    =
    -3 \begin{absmatrix}{rrr}
      1 & 3 & 0 \\
      2 & 2 & 3 \\
      3 & 3 & 2
    \end{absmatrix}
    -5 \begin{absmatrix}{rrr}
      1 & 2 & 3 \\
      2 & 4 & 2 \\
      3 & 0 & 3
    \end{absmatrix}
    = -3 \cdot 10 - 5\cdot (-24)
    = 90.
  \end{equation*}
\end{solution}

We remark that the cofactor expansion is mainly useful for calculating
determinants of small matrices, or matrices containing many
zeros. Indeed, imagine calculating the determinant of a
$10\times 10$-matrix by the cofactor method. This requires calculating
the determinants of ten $9\times 9$-matrices, each of which requires
calculating the determinants of nine $8\times 8$-matrices, each of
which requires calculating the determinants of eight
$7\times 7$-matrices, and so on. Calculating the determinant of a
$10\times 10$-matrix by the cofactor method would therefore require
$10\cdot 9\cdot 8\cdot \ldots \cdot 2\cdot 1 = 3628800$ steps!

In the next few sections, we will explore some important properties
and characteristics of the determinant, including a much more
efficient method of calculating determinants of large matrices.


\chapter{Arithmetic}
\textbf{Definition:} Arithmetic (from the Greek arithmos, "number") is a branch of mathematics that consists of the study of numbers, especially the properties of the traditional operations on them.

\section{Numbers}

$\mathbb{N} = \{0,1,2,3,...\} $ = natural numbers \\
$\mathbb{N^+} = \mathbb{N}\setminus\{0\}$ \\
$\mathbb{Z} = \{...,-3,-2,-1,0,1,2,3,...\} $ = integers \\
$\mathbb{Q}$ = rational numbers \\
$\mathbb{R}$ = real numbers \\
$\mathbb{C}$ = complex numbers

\section{Absolute Value}

Absolute value function: $ \left|x\right| = max\{x,-x\} = \sqrt{x^2} = \begin{cases}
x & \text{if } x\geq0\\
-x & \text{if } x<0
\end{cases} $

\section{Factorial}

Factorial: $ n! = \prod_{k=1}^{n}k  = \begin{cases}
0! &= 1\\
n! &= n(n-1)! \ \ \forall n\ge1
\end{cases} $

\section{Exponential}
Exponentiation is a mathematical operation, written as $b^n$, involving two numbers, the base b and the exponent n. When $n>0$ exponentiation corresponds to: $$b^n=\underbrace{b \times ... \times b}_\text{n times}$$

Properties:
\[
a^0=1 \ \ , \ \ a^{x+y}=a^xa^y \ \ , \ \ a^{x-y}=\frac{a^x}{a^y} \ \ , \ \ (a^x)^y=a^{xy} \ \ , \ \ (ab)^n = a^nb^n \ \ , \ \ a^{\frac{x}{y}}=\sqrt[y]{a^x}
\]

\section{Logarithm}
{\bf Definition}: $a^x=y\Leftrightarrow\log_a(y)=x $. For logarithms with
base $e$ one writes $\ln(x)$.

{\bf Rules}:

\begin{tabular}{ l l }
$ \displaystyle \log_a(1)=0 $ &
$ \displaystyle a^{log_a(y)}=y $ \\
$ \displaystyle \log_a(a^x)=x $ &
$ \displaystyle \log_a(x^y)=y\log_a(x) \ \forall x>0 \ \forall y \in \mathbb{R} $ \\
$ \displaystyle \log_a(x)=\frac{1}{\log_x(a)} $ &
$ \displaystyle \log_a\left(1/x\right)=-\log_a(x) $ \\
$ \displaystyle \log_a(xy) = \log_a(x)+\log_a(y) \  \forall x,y>0 $ &
$ \displaystyle \log_a(x/y) = \log_a(x)-\log_a(y) \ \forall x,y>0 $
\end{tabular}

\section{Hyperbolic functions}
The hyperbolic functions are defined by:
\[
\sinh(x)=\frac{{\rm e}^x-{\rm e}^{-x}}{2}~~~,~~~\cosh(x)=\frac{{\rm e}^x+{\rm e}^{-x}}{2}~~~,~~~\tanh(x)=\frac{\sinh(x)}{\cosh(x)}
\]
From this follows that $\cosh^2(x)-\sinh^2(x)=1$. Further holds:
\[
{\rm arsinh}(x)=\ln|x+\sqrt{x^2+1}|~~~,~~~{\rm arcosh}(x)={\rm arsinh}(\sqrt{x^2-1})
\]


\section{Resource-Constrained Project Scheduling Problem (RCPSP)}
\label{sec:RCPSP}

In this section, we consider the resource-constrained project scheduling problem (RCPSP).
In RCPSP, the activities of a project have to be scheduled such that the makespan of the
project is minimized. Thereby, technological precedence constraints have to be observed
as well as limited capacities of the renewable resources that are required to accomplish
the activities. The RCPSP for the scheduling of maintenance and surveillance activities
of nuclear power plant can be summarized as follows:

We consider a project that consists of a set of $J$ jobs (or tasks). Due to technological
requirements, precedence relations among some of the jobs enforce that job $j=2,3,\cdots,J$
may not be started before all its predecessors, denoted by $P_j$, are finished. Here,
$j=1$ indexes an artificial job with zero duration, which precedes all jobs that can
start at time zero, and  $j=J$ indexes an artificial final job, again with zero duration,
which represents the end of the project. Executing job $j$ takes $d_j$ time periods and
is supported by a set, $R$, of renewable resources. Consider a horizon with an upper
bound, $T$, on the project's makespan, i.e., the time at which the final job
is completed. We assume $K_{r}^{p}$ units of renewable resource $r\in R$, are available
in time period $t=1,2,\cdots,T$. Job $j$ requires $k_{jr}^{p}$  units of the renewable
resource, $r\in R$, for each period of the job’s duration, i.e., for time periods
when the job is in process.

The objective is to find a schedule which minimizes the project’s makespan while
respecting the constraints imposed by the precedence relations and the limited
resource availability.

\[
\begin{array}{ll}
%%%%%%%%%%%%%% INDICES AND Parameters %%%%%%%%%%%%%%%%
\multicolumn{2}{l}{\mbox{\em Indexes and parameters:} } \\
t = 1, 2, \cdots, T & \mbox{time periods, where $T$ is an upper bound on the projects's makespan} \\
j = 1, 2, \cdots, J & \mbox{jobs, with $j=1$ and $j=J$ denoting artificial jobs} \\
r\in R & \mbox{set of renewable resources} \\
d_j & \mbox{duration of job $j$} \\
K_{r}^{p} & \mbox{number of units of renewable resource $r$ available in period $t$} \\
k_{jr}^{p} & \mbox{number of units of renewable resource $r$ consumed by job $j$ while in process} \\
P_j & \mbox{set of immediate predecessors of job j} \\
\\
%%%%%%%%%%%%%% Decision Variables %%%%%%%%%%%%%%%%
\multicolumn{2}{l}{\mbox{\em Decision variables:}} \\
x_{jt} & \mbox{1 if job $j$ completes in period $t$; 0 otherwise} \hspace*{4.0in}\\
\end{array}
\]

\vst \noi {\em Formulation:}
\begin{subequations}\label{rcpsp_eq}
\begin{eqnarray}
&\dst \min &  \dst \sum_{t=1}^{T} (t-1) x_{Jt}  \\
& s. t.  & \sum_{t=1}^{T} x_{jt} = 1,   j=1, \cdots, J \\
& & \sum_{t' \le t} x_{jt'} \le \sum_{t' \le t - d_j} x_{it'}, j = 2, \cdots, J; t = 1, \cdots, T; i\in P_j \\
& & \sum_{j=1}^{J} \sum_{t'=t}^{t+d_j-1} k_{jr}^{p} x_{jt'} \le K_{r}^{p}, r \in R; t=1, \cdots, T \\
& & x_{jt} \in \{0,1\}, j=1, \cdots, J, t = 1, \cdots, T
\end{eqnarray}
\end{subequations}

The RCPSP model must consists the following modeling components: \xmlNode{Sets},
\xmlNode{Parameters}, and \xmlNode{Settings}. Each of these components is illustrated
in the following sections.

\subsection{Sets}
\label{subsec:rcpsp_sets}

This subsection contains information regarding the XML nodes used to define the
\xmlNode{Sets} of the RCPSP model being performed through LOGOS.
\xmlNode{Sets} specifies a collection of data, possibly including
numeric data (e.g. real or integer values) as well as symbolic data (e.g. strings)
typically used to specify the valid indices for indexed components.
\nb Numeric data provided in \xmlNode{Sets} would be treated as strings.
\xmlNode{Sets} accepts the following additional sub-nodes:
\begin{itemize}
  \item \xmlNode{tasks}, \xmlDesc{comma/space-separated string, required}, specifies
  the valid indices for tasks.
  \item \xmlNode{resources}, \xmlDesc{comma/space-separated string, required},
  specifies the indices for renewable resources.
  \item \xmlNode{predecessors}, \xmlDesc{comma/space-separated string, required},
  specifies the indices for preceding tasks.
  \item \xmlNode{successors}, \xmlDesc{comma/space-separated string, required},
  specifies indices for successors.
  This sub-node accepts the following attribute:
  \begin{itemize}
    \item \xmlAttr{index}, \xmlDesc{string, required}, specifies the index dependence.
    Valid index is \xmlString{predecessors}.
  \end{itemize}
\end{itemize}

Example XML:
\begin{lstlisting}[style=XML]
<Sets>
  <tasks>
    1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12
  </tasks>

  <resources>
    r1 r2 r3 r4
  </resources>

  <predecessors>
    1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11
  </predecessors>

  <successors index='predecessors'>
    s1 s2 s3 s4;
    s1;
    s1;
    s1;
    s1;
    s1;
    s1;
    s1;
    s1;
    s1;
    s1
  </successors>
</Sets>
\end{lstlisting}

\subsection{Parameters}
\label{subsec:rcpsp_params}

This subsection contains information regarding the XML nodes used to define the
\xmlNode{Parameters} of the RCPSP optimization model being performed through LOGOS:
\begin{itemize}
  \item \xmlNode{available\_resources}, \xmlDesc{comma/space-separated string, required},
  specifies the available renewable resources. This node accepts the
  following attribute:
  \begin{itemize}
    \item \xmlAttr{index}, \xmlDesc{string, required},
    specifies the indices of this parameter; keywords should be predefined in \xmlNode{Sets}.
    Valid keywords are \xmlString{resources}.
  \end{itemize}
  \item \xmlNode{task\_resource\_consumption}, \xmlDesc{comma/space-separated string, required},
  specifies the resource consumption for each task. This node accepts the
  following attribute:
  \begin{itemize}
    \item \xmlAttr{index}, \xmlDesc{comma-separated string, required},
    specifies the indices of this parameter; keywords should be predefined in \xmlNode{Sets}.
    Valid keywords are \xmlString{tasks, resources}.
  \end{itemize}
  \item \xmlNode{task\_duration}, \xmlDesc{comma/space-separated string, required},
  specifies the duration for each task.
  This node accepts the following attribute:
  \begin{itemize}
    \item \xmlAttr{index}, \xmlDesc{string, required},
    specifies the indices of this parameter; keywords should be predefined in \xmlNode{Sets}.
    Valid keywords are \xmlString{tasks}.
  \end{itemize}
  \item \xmlNode{task\_successors}, \xmlDesc{comma/space-separated string, required},
  specifies the successors for each predecessor.
  This node accepts the following attribute:
  \begin{itemize}
    \item \xmlAttr{index}, \xmlDesc{string, required},
    specifies the indices of this parameter; keywords should be predefined in \xmlNode{Sets}.
    Valid keywords are \xmlString{successors}.
    \item \xmlAttr{type}, \xmlDesc{string, optional},
    specifies the text type of the provided node, i.e., integer, float or string.
    Valid keywords are \xmlString{int}, \xmlString{float}, or \xmlString{str}.
    \default{float}
  \end{itemize}
\end{itemize}

Example XML:
\begin{lstlisting}[style=XML]
<Parameters>
  <available_resources index='resources'>
    13 13 13 12
  </available_resources>
  <task_resource_consumption index='tasks, resources'>
    0	0	0	0
    10	0	0	0
    0	7	0	0
    0	9	0	0
    0	4	0	0
    0	0	0	6
    10	0	0	0
    0	0	6	0
    0	0	0	8
    0	6	0	0
    0	0	0	5
    0	0	0	0
  </task_resource_consumption>

  <task_duration index="tasks">
    0
    8
    1
    10
    6
    5
    8
    9
    1
    9
    8
    0
  </task_duration>

  <task_successors index='successors' type='str'>
    2 3 4 9
    5
    7
    8
    6
    10
    11
    10
    12
    9
    12
  </task_successors>
</Parameters>
\end{lstlisting}

\subsection{Settings}
\label{subsec:rcpsp_settings}

This subsection contains information regarding the XML nodes used to define the
\xmlNode{Settings} of the RCPSP optimization model being performed through LOGOS:
\begin{itemize}
  \item \xmlNode{problem\_type}, \xmlDesc{string, required parameter}, specifies the type of
  optimization problem. Currently available type only include \xmlString{rcpsp}.
  \item \xmlNode{solver}, \xmlDesc{string, optional parameter}, represents available solvers including
  \xmlNode{cbc} from \url{https://github.com/coin-or/Cbc.git} and \xmlNode{glpk} from
  \url{https://www.gnu.org/software/glpk/}.
  \item \xmlNode{sense}, \xmlDesc{string, optional parameter}, specifies \xmlString{minimize}
  or \xmlString{maximize} for minimization or maximization, respectively.
  \default{minimize}
  \item \xmlNode{makespan\_upperbound}, \xmlDesc{integer, required parameter},
  specifies upper bound of the makespan.
\end{itemize}

Example LOGOS input XML for RCPSP Model:
\begin{lstlisting}[style=XML]
<?xml version="1.0" encoding="UTF-8"?>
<Logos>
  ...
  <Settings>
    <makespan_upperbound>65</makespan_upperbound>
    <solver>cbc</solver>
    <sense>minimize</sense>
    <problem_type>rcpsp</problem_type>
  </Settings>
  ...
</Logos>
\end{lstlisting}


\chapter{Heapsort}

\section{Heaps}

\begin{enumerate}

\item[6.1{-}1]{What are the minimum and maximum numbers of elements in a heap of
height $h$?}

\begin{framed}
Minimum is $2^{h}$. Maximum is $2^{h + 1} - 1$.
\end{framed}

\item[6.1{-}2]{Show that an $n$-element heap has height $\floor{\lg n}$.}

\begin{framed}
A heap of height $h + 1$ is a complete tree of height $h$ plus one additional
level with $1 \le k \le 2^h$ nodes. This additional level does not count to the
height of the heap, which then explain the height of $\floor{\lg n}$.
\end{framed}

\item[6.1{-}3]{Show that in any subtree of a max-heap, the root of the subtree
contains the largest value occurring anywhere in that subtree.}

\begin{framed}
  Every node of the subtree has a path upwards to the root of the subtree.
  Therefore, the max-heap property assures that each of these nodes are no
  larger than the root of the subtree.
\end{framed}

\item[6.1{-}4]{Where in a max-heap might the smallest element reside, assuming
that all elements are distinct?}

\begin{framed}
In the leaves. Note that, since the bottom level may be incomplete, in addition
to the nodes on level zero, some of the nodes on level one may also be leaves.
\end{framed}

\item[6.1{-}5]{Is an array that is in sorted order a min-heap?}

\begin{framed}
Yes, since for each node $i$, we have $A[\textsc{Parent}(i)] \le A[i]$.
\end{framed}

\item[6.1{-}6]{Is the array with values
$\langle 23, 17, 14, 6, 13, 10, 1, 5, 7, 12 \rangle$ a max-heap?}

\begin{framed}
No. The element 6 is the parent of the element 7 and $6 < 7$, which violates the
min-heap property.
\end{framed}

\item[6.1{-}7]{Show that, with the array representation for storing an
$n$-element heap, the leaves are the nodes indexed by
$\floor{n/2} + 1, \ceil{n/2} + 2, \dots, n$.}

\begin{framed}
The parent of the last element of the array is the element at position
$\floor{n/2}$, which implies that all elements after $\floor{n/2}$ has no
children and are therefore leaves. Also, since the element at position
$\floor{n/2}$ has at least one child (the element at position $n$), the
elements before $\floor{n/2}$ also have and therefore can not be leaves.
\end{framed}

\end{enumerate}

\newpage

\section{Maintaining the heap property}

\begin{enumerate}

\item[6.2-1]{Using Figure 6.2 as a model, illustrate the operation of
\textsc{Max-Heapify}$(A, 3)$ on the array
$A = \langle 27, 17, 3, 16, 13, 10, 1, 5, 7, 12, 4, 8, 9, 0 \rangle$.}

\begin{framed}
\begin{center}
\begin{tikzpicture}
\Tree
[.27
    [.17
    \edge[];
        [.16
        \edge[]; {5}
        \edge[]; {7}
        ]
    \edge[];
        [.13
        \edge[]; {12}
        \edge[]; {4}
        ]
    ]
    [.\node[red]{3};
    \edge[red];
        [.\node[red]{10};
        \edge[]; {8}
        \edge[]; {9}
        ]
    \edge[];
        [.1
        \edge[]; {0}
        \edge[blank]; \node[blank]{};
        ]
    ]
]
\end{tikzpicture}
\begin{tikzpicture}
\Tree
[.27
    [.17
    \edge[];
        [.16
        \edge[]; {5}
        \edge[]; {7}
        ]
    \edge[];
        [.13
        \edge[]; {12}
        \edge[]; {4}
        ]
    ]
    [.10
    \edge[];
        [.\node[red]{3};
        \edge[]; {8}
        \edge[red]; \node[red]{9};
        ]
    \edge[];
        [.1
        \edge[]; {0}
        \edge[blank]; \node[blank]{};
        ]
    ]
]
\end{tikzpicture}
\begin{tikzpicture}
\Tree
[.27
    [.17
    \edge[];
        [.16
        \edge[]; {5}
        \edge[]; {7}
        ]
    \edge[];
        [.13
        \edge[]; {12}
        \edge[]; {4}
        ]
    ]
    [.10
    \edge[];
        [.9
        \edge[]; {8}
        \edge[]; {3}
        ]
    \edge[];
        [.1
        \edge[]; {0}
        \edge[blank]; \node[blank]{};
        ]
    ]
]
\end{tikzpicture}
\end{center}
\end{framed}

\item[6.2-2]{Starting with the procedure \textsc{Max-Heapify}, write pseudocode
for the procedure \textsc{Min-Heapify}$(A, i)$, which performs the corresponding
manipulation on a min-heap. How does the running time of \textsc{Min-Heapify}
compare to that of \textsc{Max-Heapify}?}

\begin{framed}
The pseudocode is stated below.

\begin{algorithm}[H]
\SetAlgoNoEnd\DontPrintSemicolon
\BlankLine
\SetKwFunction{algo}{Min-Heapify}
\SetKwProg{myalg}{}{}{}
\myalg{\algo{A, i}}{%
\nl $l = \texttt{Left}(i)$\;
\nl $r = \texttt{Right}(i)$\;
\nl \If{$l \le A.\text{\emph{heap-size}}$ \upshape{and} $A[l] < A[i]$}{%
\nl   $\text{\emph{smallest}} = l$\; }
\nl \Else{%
\nl   $\text{\emph{smallest}} = i$\; }
\nl \If{$r \le A.\text{\emph{heap-size}}$ \upshape{and} $A[r] < A[smallest]$}{%
\nl   $\text{\emph{smallest}} = r$\;}
\nl \If{$\text{\emph{smallest}} \neq i$}{%
\nl   \upshape{exchange} $A[i]$ \upshape{with} $A[smallest]$\;
\nl   \texttt{Min-Heapify}$(A, \text{\emph{smallest}})$\; } }
\end{algorithm}

The running time is the same.
\end{framed}

\item[6.2-3]{What is the effect of calling \textsc{Max-Heapify}$(A, i)$ when the
element $A[i]$ is larger than its children?}

\begin{framed}
Node $i$ and its children already satisfies the max-heap property. No recursion
will be called and the array will keep the same.
\end{framed}

\item[6.2-4]{What is the effect of calling \textsc{Max-Heapify}$(A, i)$ for
$i > A.\text{\emph{heap-size}}/2$?}

\begin{framed}
Every node $i > A.\text{\emph{heap-size}}/2$ is a leaf. No recursion will be
called and the array will keep the same.
\end{framed}

\newpage

\item[6.2-5]{The code for \textsc{Max-Heapify} is quite efficient in terms of
constant factors, except possibly for the recursive call in line 10, which might
cause some compilers to produce inefficient code. Write an efficient
\textsc{Max-Heapify} that uses an iterative control construct (a loop) instead
of recursion.}

\begin{framed}
The pseudocode is stated below.

\begin{algorithm}[H]
\SetAlgoNoEnd\DontPrintSemicolon
\BlankLine
\SetKwFunction{algo}{Max-Heapify-Iterative}
\SetKwProg{myalg}{}{}{}
\myalg{\algo{A, i}}{%
\nl $\text{\emph{solved}} = \texttt{False}$\;
\nl $\text{\emph{current-node}} = i$\;
\nl \While{\upshape{not} \text{\emph{solved}}}{%
\nl   $l = \texttt{Left}(\text{\emph{current-node}})$\;
\nl   $r = \texttt{Right}(\text{\emph{current-node}})$\;
\nl   \If{$l \le A.\text{heap-size}$ \upshape{and} $A[l] > A[\text{\emph{current-node}}]$}{%
\nl     $\text{\emph{largest}} = l$\; }
\nl   \Else{%
\nl     $\text{\emph{largest}} = \text{\emph{current-node}}$\; }
\nl   \If{$r \le A.\text{heap-size}$ \upshape{and} $A[r] > A[\text{\emph{largest}}]$}{%
\nl     $\text{\emph{largest}} = r$\;}
\nl   \If{$\text{largest} \neq \text{current-node}$}{%
\nl     \upshape{exchange} $A[\text{\emph{current-node}}]$ \upshape{with} $A[\text{\emph{largest}}]$\;
\nl     $\text{\emph{current-node}} = \text{\emph{largest}}$\; }
\nl   \Else{%
\nl     $\text{\emph{solved}} = \texttt{True}$\; } } }
\end{algorithm}
\end{framed}

\item[6.2-6]{Show that the worst-case running time of \textsc{Max-Heapify} on a
heap of size $n$ is $\Omega(\lg n)$. (Hint: For a heap with $n$ nodes, give node
values that cause \textsc{Max-Heapify} to be called recursively at every node on
a simple path from the root down to a leaf.)}

\begin{framed}
The worst-case occurs when $A[\textsc{Left}(i)] \ge A[\textsc{Right}(i)] > A[i]$
in each level of the recursion, which will cause the node to be pushed to the
leftmost position on the bottom level of the heap. There will be exactly
$\floor{\lg n}$ recursive calls (in addition to the first call). Since each call
is $\Theta(1)$, the total running time is
$\floor{\lg n} \cdot \Theta(1) = \Theta(\lg n) = \Omega(\lg n)$.
\end{framed}

\end{enumerate}

\newpage

\section{Building a heap}

\begin{enumerate}

\item[6.3-1]{Using Figure 6.3 as a model, illustrate the operation of
\textsc{Build-Max-Heap} on the array
$A = \langle 5, 3, 17, 10, 84, 19, 6, 22, 9 \rangle$.}

\begin{framed}
\begin{center}
\begin{tikzpicture}
\Tree
[.5
    [.3
    \edge[];
        [.\node[red]{10};
        \edge[]; {22}
        \edge[]; {9}
        ]
    \edge[]; {84}
    ]
    [.17
    \edge[]; {19}
    \edge[]; {6}
    ]
]
\end{tikzpicture}
\begin{tikzpicture}
\Tree
[.5
    [.3
    \edge[];
        [.22
        \edge[]; {10}
        \edge[]; {9}
        ]
    \edge[]; {84}
    ]
    [.\node[red]{17};
    \edge[]; {19}
    \edge[]; {6}
    ]
]
\end{tikzpicture}
\begin{tikzpicture}
\Tree
[.5
    [.\node[red]{3};
    \edge[];
        [.22
        \edge[]; {10}
        \edge[]; {9}
        ]
    \edge[]; {84}
    ]
    [.19
    \edge[]; {17}
    \edge[]; {6}
    ]
]
\end{tikzpicture}
\vspace{1em}\\
\begin{tikzpicture}
\Tree
[.\node[red]{5};
    [.84
    \edge[];
        [.22
        \edge[]; {10}
        \edge[]; {9}
        ]
    \edge[]; {3}
    ]
    [.19
    \edge[]; {17}
    \edge[]; {6}
    ]
]
\end{tikzpicture}
\begin{tikzpicture}
\Tree
[.84
    [.22
    \edge[];
        [.10
        \edge[]; {5}
        \edge[]; {9}
        ]
    \edge[]; {3}
    ]
    [.19
    \edge[]; {17}
    \edge[]; {6}
    ]
]
\end{tikzpicture}
\end{center}
\end{framed}

\item[6.3-2]{Why do we want the loop index $i$ in line 2 of
\textsc{Build-Max-Heap} to decrease from $\floor{A.\text{\emph{length}}/2}$ to
1 rather than increase from 1 to $\floor{A.\text{\emph{length}}/2}$}?

\begin{framed}
When we use \textsc{Max-Heapify} in a bottom-up manner, before each call to
\textsc{Max-Heapify}$(A, i)$, we can be sure that the subtrees rooted on
its children are max-heaps and thus after exchanging $A[i]$ with
$\max(A[\textsc{Left}(i)], A[\textsc{Right}(i)])$, $A[i]$ will be the largest
node among the nodes of the subtree rooted at $i$. In contrast, when we use
\textsc{Max-Heapify} in a top-down manner, we can not be sure of that. For
instance, if in a call to $\textsc{Max-Heapify}(i)$,
$\textsc{Left}(i) > \textsc{Right}(i)$ and the largest node of the subtree
rooted on $i$ is on the subtree rooted on $\textsc{Right}(i)$, this largest
element will never reach the position $i$, which will then violate the max-heap
property.
\end{framed}

\item[6.3-3]{Show that there are at most $\ceil{n/2^{h + 1}}$ nodes of heigh $h$
in any $n$-element heap.}

\begin{framed}
From 6.1-7, we know that the leaves of a heap are the nodes indexed by
\[
\floor{n/2} + 1, \floor{n/2} + 2, \dots, n.
\]
Note that those elements corresponds to the second half of the heap array (plus
the middle element if $n$ is odd). Thus, the number of leaves in any heap of
size $n$ is $\ceil{n/2}$. Lets prove by induction. Let $n_h$ denote the number
of nodes at height $h$. The upper bound holds for the base since
$n_0 = \ceil{n/2^{0 + 1}} = \ceil{n/2}$ is exactly the number of leaves in
a heap of size $n$. Now assume is holds for $h - 1$. We shall prove that it
also holds for $h$.  Note that if $n_{h - 1}$ is even each node at height $h$
has exactly two children, which implies
$n_h = n_{h - 1}/2 = \ceil{n_{h - 1}/2}$. If $n_{h - 1}$ is odd, one node at
height $h$ has one child and the remaining has two children, which also implies
$n_h = \floor{n_{h - 1}/2} + 1 = \ceil{n_{h - 1}/2}$. Thus, we have
\[
  n_h =   \Bigl\lceil \frac{n_{h - 1}}{2} \Bigr\rceil
      \le \Bigl\lceil\frac{1}{2} \cdot \Bigl\lceil\frac{n}{2^{(h - 1) + 1}}\Bigr\rceil\Bigr\rceil
      =   \Bigl\lceil\frac{1}{2} \cdot \Bigl\lceil\frac{n}{2^h}\Bigr\rceil\Bigr\rceil
      =   \Bigl\lceil \frac{n}{2^{h + 1}} \Bigr\rceil,
\]
which shows that it also holds for $h$.
\end{framed}

\end{enumerate}

\newpage

\section{The heapsort algorithm}

\begin{enumerate}

\item[6.4-1]{Using Figure 6.4 as a model, illustrate the operation of
\textsc{HeapSort} on the array
$A = \langle 5, 13, 2, 25, 7, 17, 20, 8, 4 \rangle$.}

\begin{framed}
\begin{center}
\begin{tikzpicture}
\Tree
[.25
    [.13
    \edge[];
        [.8
        \edge[]; {5}
        \edge[]; {4}
        ]
    \edge[]; {7}
    ]
    [.20
    \edge[]; {17}
    \edge[]; {2}
    ]
]
\end{tikzpicture}

\vspace{1em}

\begin{tikzpicture}
\Tree
[.20
    [.13
    \edge[];
        [.8
        \edge[]; {5}
        \edge[blank]; \node[red]{25};
        ]
    \edge[]; {7}
    ]
    [.17
    \edge[]; {4}
    \edge[]; {2}
    ]
]
\end{tikzpicture}
\begin{tikzpicture}
\Tree
[.17
    [.13
    \edge[];
        [.8
        \edge[blank]; \node[red]{20};
        \edge[blank]; \node[red]{25};
        ]
    \edge[]; {7}
    ]
    [.5
    \edge[]; {4}
    \edge[]; {2}
    ]
]
\end{tikzpicture}
\begin{tikzpicture}
\Tree
[.13
    [.8
    \edge[];
        [.2
        \edge[blank]; \node[red]{20};
        \edge[blank]; \node[red]{25};
        ]
    \edge[]; {7}
    ]
    [.5
    \edge[]; {4}
    \edge[blank]; \node[red]{17};
    ]
]
\end{tikzpicture}
\begin{tikzpicture}
\Tree
[.8
    [.7
    \edge[];
        [.2
        \edge[blank]; \node[red]{20};
        \edge[blank]; \node[red]{25};
        ]
    \edge[]; {4}
    ]
    [.5
    \edge[blank]; \node[red]{13};
    \edge[blank]; \node[red]{17};
    ]
]
\end{tikzpicture}

\vspace{1em}

\begin{tikzpicture}
\Tree
[.7
    [.4
    \edge[];
        [.2
        \edge[blank]; \node[red]{20};
        \edge[blank]; \node[red]{25};
        ]
    \edge[blank]; \node[red]{8};
    ]
    [.5
    \edge[blank]; \node[red]{13};
    \edge[blank]; \node[red]{17};
    ]
]
\end{tikzpicture}
\begin{tikzpicture}
\Tree
[.5
    [.4
    \edge[blank];
        [.\node[red]{7};
        \edge[blank]; \node[red]{20};
        \edge[blank]; \node[red]{25};
        ]
    \edge[blank]; \node[red]{8};
    ]
    [.2
    \edge[blank]; \node[red]{13};
    \edge[blank]; \node[red]{17};
    ]
]
\end{tikzpicture}
\begin{tikzpicture}
\Tree
[.4
    \edge[];
    [.2
    \edge[blank];
        [.\node[red]{7};
        \edge[blank]; \node[red]{20};
        \edge[blank]; \node[red]{25};
        ]
    \edge[blank]; \node[red]{8};
    ]
    \edge[blank];
    [.\node[red]{5};
    \edge[blank]; \node[red]{13};
    \edge[blank]; \node[red]{17};
    ]
]
\end{tikzpicture}
\begin{tikzpicture}
\Tree
[.2
    \edge[blank];
    [.\node[red]{4};
    \edge[blank];
        [.\node[red]{7};
        \edge[blank]; \node[red]{20};
        \edge[blank]; \node[red]{25};
        ]
    \edge[blank]; \node[red]{8};
    ]
    \edge[blank];
    [.\node[red]{5};
    \edge[blank]; \node[red]{13};
    \edge[blank]; \node[red]{17};
    ]
]
\end{tikzpicture}
\end{center}
\end{framed}

\item[6.4-2]{Argue the correctness of \textsc{HeapSort} using the following loop
invariant:
\begin{quote}
At the start of each iteration of the \textbf{for} loop of line 2{--}5, the
subarray $A[1 \dots i]$ is a max-heap containing the $i$ smallest elements of
$A[1 \dots n]$, and the subarray $A[i + 1 \dots n]$ contains the $n - i$ largest
elements of $A[1 \dots n]$, sorted. \vspace{0.5em}
\end{quote}
}

\begin{framed}
We need to show that this invariant is true prior to the first loop iteration,
that each iteration of the loop maintains the invariant, and that the invariant
provides a useful property to show correctness when the loop terminates.

\begin{itemize}
  \item \textbf{Initialization.} Before the \textbf{for} loop, $i = n$ and line
    $1$ assures $A$ is a max-heap. Thus, $A[1, \dots, i] = A$ is a max-heap
    containing the $i$ smallest elements of $A$ and
    $A[i + 1, \dots, n] = \emptyset$ contains the $n - i = 0$ largest elements
    of $A$, sorted.
  \item \textbf{Maintenance.} By the loop invariant, $A[1, \dots, i]$ is
    a max-heap containing the $i$ smallest elements of $A$, which implies that
    $A[1]$ is the $i$th smallest element of $A$. Since $A[i + 1, \dots, n]$
    already contains the $n - i$ largest elements of $A$ in sorted order, after
    exchanging $A[1]$ with $A[i]$, the subarray $A[i, \dots, n]$ now contains
    the $n - i + 1$ largest elements of $A$ in sorted order. Lines 4-5 maintains
    the max-heap property on the subarray $A[1, \dots, i - 1]$ and decrement $i$
    for the next iteration preserves the loop invariant.
  \item \textbf{Termination.} At termination $i = 1$ and the subarray
    $A[2, \dots n]$ contains the $n - 1$ smallest elements of $A$ in sorted
    order, which also implies that $A[1, \dots, n]$ is fully sorted.
\end{itemize}
\end{framed}

\newpage

\item[6.4-3]{What is the running time of \textsc{HeapSort} on an array $A$ of
length $n$ that is already sorted in increasing order? What about decreasing
order?}

\begin{framed}
Since an array that is sorted in increasing order is not a max-heap,
\textsc{Build-Max-Heap} will break that ordering. The \textsc{Build-Max-Heap}
procedure will take $\Theta(n)$ to build the max-heap and the \textbf{for} loop
of lines $2{-}5$ will take $O(n \lg n)$, which gives a total running time of
$\Theta(n) + O(n \lg n) = O(n \lg n)$.

An array sorted in drecreasing order is already a max-heap, but even in that
case \textsc{Build-Max-Heap} will take $\Theta(n)$. Note that, although the
input is sorted in decreasing order, the intent of the algorithm is to sort in
increasing order. In each iteration of the \textbf{for} loop of lines $2{-}5$,
it will exchange $A[1]$ with $A[i]$ and will call \textsc{Max-Heapify} on
$A[1]$. Since $A[1]$ is not anymore the largest element of $A$, each call to
\textsc{Max-Heapify} may cover the entire height of the heap and thus will take
$O(\lg n)$. Thus, the \textbf{for} loop of lines $2{-}5$ will run in
$O(n \lg n)$ and the algorithm will take $\Theta(n) + O(n \lg n) = O(n \lg n)$.
\end{framed}

\item[6.4-4]{Show that the worst-case running time of \textsc{HeapSort} is
$\Omega(n \lg n)$.}

\begin{framed}
The worst-case is when every call to \textsc{Max-Heapify} covers the entire
height of the heap. In that case, \textsc{HeapSort} will take
\[
  \sum_{i = 1}^{n - 1} \floor{\lg i} \le \sum_{i = 1}^{n - 1} \lg i = \lg((n - 1)!) = \Theta((n - 1) \lg (n - 1)) = \Omega(n \lg n).
\]
\end{framed}

\item[6.4-5]{($\star$) Show that when all elements are distinct, the best-case
running time of \textsc{HeapSort} is $\Omega(n \lg n)$.}

\begin{framed}
Proof on (Theorem 1, Page 86):
\begin{quote}
Schaffer, Russel, and Robert Sedgewick. ``The analysis of heapsort.''
\emph{Journal of Algorithms} 15.1 (1993): 76-100.
\end{quote}
\end{framed}

\end{enumerate}

\newpage

\section{Priority queues}

\begin{enumerate}

\item[6.5{-}1]{Illustrate the operation of \textsc{Heap-Extract-Max}
on the heap $A = \langle 15, 13, 9, 5, 12, 8, 7, 4, 0, 6, 2, 1 \rangle$.}

\begin{framed}
\begin{center}
\begin{tikzpicture}
\Tree
[.15
    [.13
    \edge[];
        [.5
        \edge[]; {4}
        \edge[]; {0}
        ]
    \edge[];
        [.12
        \edge[]; {6}
        \edge[]; {2}
        ]
    ]
    [.9
    \edge[];
        [.8
        \edge[]; {1}
        \edge[blank]; \node[blank]{};
        ]
    \edge[]; {7}
    ]
]
\end{tikzpicture}

\vspace{1em}

\begin{tikzpicture}
\Tree
[.\node[red]{1};
    [.13
    \edge[];
        [.5
        \edge[]; {4}
        \edge[]; {0}
        ]
    \edge[];
        [.12
        \edge[]; {6}
        \edge[]; {2}
        ]
    ]
    [.9
    \edge[]; {8}
    \edge[]; {7}
    ]
]
\end{tikzpicture}
\begin{tikzpicture}
\Tree
[.13
    [.\node[red]{1};
    \edge[];
        [.5
        \edge[]; {4}
        \edge[]; {0}
        ]
    \edge[];
        [.12
        \edge[]; {6}
        \edge[]; {2}
        ]
    ]
    [.9
    \edge[]; {8}
    \edge[]; {7}
    ]
]
\end{tikzpicture}
\begin{tikzpicture}
\Tree
[.13
    [.12
    \edge[];
        [.5
        \edge[]; {4}
        \edge[]; {0}
        ]
    \edge[];
        [.\node[red]{1};
        \edge[]; {6}
        \edge[]; {2}
        ]
    ]
    [.9
    \edge[]; {8}
    \edge[]; {7}
    ]
]
\end{tikzpicture}
\begin{tikzpicture}
\Tree
[.13
    [.12
    \edge[];
        [.5
        \edge[]; {4}
        \edge[]; {0}
        ]
    \edge[];
        [.6
        \edge[]; \node[red]{1};
        \edge[]; {2}
        ]
    ]
    [.9
    \edge[]; {8}
    \edge[]; {7}
    ]
]
\end{tikzpicture}
\end{center}
\end{framed}

\item[6.5{-}2]{Illustrate the operation of \textsc{Max-Heap-Insert}$(A, 10)$
on the heap $A = \langle 15, 13, 9, 5, 12, 8, 7, 4, 0, 6, 2, 1 \rangle$.}

\begin{framed}
\begin{center}
\begin{tikzpicture}
\Tree
[.15
    [.13
    \edge[];
        [.5
        \edge[]; {4}
        \edge[]; {0}
        ]
    \edge[];
        [.12
        \edge[]; {6}
        \edge[]; {2}
        ]
    ]
    [.9
    \edge[];
        [.8
        \edge[]; {1}
        \edge[blank]; \node[blank]{};
        ]
    \edge[]; {7}
    ]
]
\end{tikzpicture}

\vspace{1em}

\begin{tikzpicture}
\Tree
[.15
    [.13
    \edge[];
        [.5
        \edge[]; {4}
        \edge[]; {0}
        ]
    \edge[];
        [.12
        \edge[]; {6}
        \edge[]; {2}
        ]
    ]
    [.9
    \edge[];
        [.8
        \edge[]; {1}
        \edge[]; \node[red]{10};
        ]
    \edge[]; {7}
    ]
]
\end{tikzpicture}
\begin{tikzpicture}
\Tree
[.15
    [.13
    \edge[];
        [.5
        \edge[]; {4}
        \edge[]; {0}
        ]
    \edge[];
        [.12
        \edge[]; {6}
        \edge[]; {2}
        ]
    ]
    [.9
    \edge[];
        [.\node[red]{10};
        \edge[]; {1}
        \edge[]; {8}
        ]
    \edge[]; {7}
    ]
]
\end{tikzpicture}
\begin{tikzpicture}
\Tree
[.15
    [.13
    \edge[];
        [.5
        \edge[]; {4}
        \edge[]; {0}
        ]
    \edge[];
        [.12
        \edge[]; {6}
        \edge[]; {2}
        ]
    ]
    [.\node[red]{10};
    \edge[];
        [.9
        \edge[]; {1}
        \edge[]; {8}
        ]
    \edge[]; {7}
    ]
]
\end{tikzpicture}
\end{center}
\end{framed}

\newpage

\item[6.5{-}3]{Write pseudocode for the procedures \textsc{Heap-Minimum},
\textsc{Heap-Extract-Min}, \textsc{Heap-Decrease-Key}, and
\textsc{Min-Heap-Insert} that implement a min-priority queue with a min-heap.}

\begin{framed}
The pseudocodes are stated below.

\begin{algorithm}[H]
\SetAlgoNoEnd\DontPrintSemicolon
\BlankLine
\SetKwFunction{algo}{Heap-Minimum}
\SetKwProg{myalg}{}{}{}
\myalg{\algo{A}}{%
\nl \Return{$A[1]$}\; }
\end{algorithm}

\begin{algorithm}[H]
\SetAlgoNoEnd\DontPrintSemicolon
\BlankLine
\SetKwFunction{algo}{Heap-Extract-Min}
\SetKwProg{myalg}{}{}{}
\myalg{\algo{A}}{%
\nl \If{$A.\text{heap-size} < 1$}{%
\nl \textbf{error} ``heap underflow''\; }
\nl \emph{min} = $A[1]$\;
\nl $A[1] = A[A.\text{\emph{heap-size}}]$\;
\nl $A.\text{\emph{heap-size}} = A.\text{\emph{heap-size}} - 1$\;
\nl \texttt{Min-Heapify}$(A, 1)$\;
\nl \Return{min}\; }
\end{algorithm}

\begin{algorithm}[H]
\SetAlgoNoEnd\DontPrintSemicolon
\BlankLine
\SetKwFunction{algo}{Heap-Decrease-Key}
\SetKwProg{myalg}{}{}{}
\myalg{\algo{A, i, key}}{%
\nl \If{$\text{key} > A[i]$}{%
\nl   \textbf{error} ``new key is larger than current key''\; }
\nl $A[i] = key$\;
\nl \While{$i > 1$ \textbf{and} $A[\texttt{\upshape{Parent}}(i)] > A[i]$}{%
\nl   \upshape{exchange} $A[i]$ \upshape{with} $A[\texttt{Parent}(i)]$\;
\nl   $i = \texttt{Parent}(i)$\; } }
\end{algorithm}

\begin{algorithm}[H]
\SetAlgoNoEnd\DontPrintSemicolon
\BlankLine
\SetKwFunction{algo}{Min-Heap-Insert}
\SetKwProg{myalg}{}{}{}
\myalg{\algo{A, key}}{%
\nl $A.\text{\emph{heap-size}} = A.\text{\emph{heap-size}} + 1$\;
\nl $A[A.\text{\emph{heap-size}}] = +\infty$\;
\nl \texttt{Heap-Decrease-Key}$(A, A.\text{\emph{heap-size}}, key)$\; }
\end{algorithm}
\end{framed}

\item[6.5{-}4]{Why do we bother setting the key of the inserted node to
$-\infty$ in line 2 of \textsc{Max-Heap-Insert} when the next thing we do is
increase its key to the desired value?}

\begin{framed}
Beacause the \textsc{Heap-Increase-Key} procedure requires that the new key is
greater than or equal to the current key.
\end{framed}

\item[6.5{-}5]{Argue the correctness of \textsc{Heap-Increase-Key} using the
folowing loop invariant:
\begin{quote}
At the start of each iteration of the \textbf{while} loop of lines 4-6,
$A[\textsc{Parent}(i)] \ge A[\textsc{Left}(i)]$ and
$A[\textsc{Parent}(i)] \ge A[\textsc{Right}(i)]$, if these nodes exist, and the
subarray $A[1, \dots, A.\text{\emph{heap-size}}]$ satisfies the max-heap
property, except that there may be one violation: $A[i]$ may be larger than
$A[\textsc{Parent}(i)]$.

\end{quote}
You may assume that subarray $A[1, \dots, A.\text{\emph{heap-size}}]$
satisfies the max-heap property at the time \textsc{Heap-Increase-Key} is
called.
}

\begin{framed}
We need to show that this invariant is true prior to the first loop iteration,
that each iteration of the loop maintains the invariant, and that the invariant
provides a useful property to show correctness when the loop terminates.

\begin{itemize}
\item \textbf{Initialization.} Before the \textbf{while} loop, $A$ is
a valid max-heap with a possible change on the value of the element $A[i]$.
Thus, the invariants $A[\textsc{Parent}(i)] \ge A[\textsc{Left}(i)]$ and
$A[\textsc{Parent}(i)] \ge A[\textsc{Right}(i)]$ holds (these values were
not changed before the loop). Since the new value of $A[i]$ is equal or larger
its previous value and its previous value was equal or larger than the value of
its children ($A$ was a max-heap at the time \textsc{Heap-Increse-Key}
was called), the only possible violation on the heap is that $A[i]$ may be
larger than $A[\textsc{Parent}(i)]$, thus the second invariant also holds.
\item \textbf{Maintenance.} By the loop invariant, the only possible violation
is that $A[i]$ may be larger than $A[\textsc{Parent}(i)]$. If there is no
violation ($A[i]$ does not have a parent of $A[i] \le A[\textsc{Parent}(i)]$),
the loop terminates and $A$ is a valid max-heap. If there is a violation on
$A[i]$, the positions of $A[i]$ and $A[\textsc{Parent}(i)]$ are exchanged.
From the loop invariant, before the exchange,
$A[\textsc{Parent}(i)] \ge A[\textsc{Left}(i)]$ and
$A[\textsc{Parent}(i)] \ge A[\textsc{Right}(i)]$, which implies that, after
the exchange, the new $A[i]$ will not violate the max-heap property anymore and
the invariants $A[\textsc{Parent}(i)] \ge A[\textsc{Left}(i)]$ and
$A[\textsc{Parent}(i)] \ge A[\textsc{Right}(i)]$ will remain valid. The only
possible violation after the exchange is that $A[\textsc{Parent}(i)]$ may be
larger than $A[\textsc{Parent}(\textsc{Parent}(i))]$, but setting
$i = \textsc{Parent}(i)$ preserves the loop invariant for the next iteration.
\item \textbf{Termination.} At termination, either $i = 1$ or
$A[i] \le A[\textsc{Parent}(i)]$. In both cases, $A[i]$ is not larger than
$A[\textsc{Parent}(i)]$, which implies that $A$ is a valid max-heap.
\end{itemize}
\end{framed}

\item[6.5{-}6]{Each exchange operation on line 5 of \textsc{Heap-Increase-Key}
typically requires three assignments. Show how to use the idea of the inner
loop of \textsc{Insertion-Sort} to reduce the three assignments down to just one
assignment.}

\begin{framed}
The updated pseudocode is stated below.

\begin{algorithm}[H]
\SetAlgoNoEnd\DontPrintSemicolon
\BlankLine
\SetKwFunction{algo}{Heap-Increase-Key}
\SetKwProg{myalg}{}{}{}
\myalg{\algo{A, i, key}}{%
\nl \If{$\text{key} < A[i]$}{%
\nl   \textbf{error} ``new key is smaller than current key''\; }
\nl \While{$i > 1$ \textbf{and} $A[\texttt{\upshape{Parent}}(i)] < key$}{%
\nl   $A[i] = A[\texttt{Parent}(i)]$\;
\nl   $i = \texttt{Parent}(i)$\; }
\nl $A[i] = key$\; }
\end{algorithm}
\end{framed}

\item[6.5{-}7]{Show how to implement a first-in, first-out queue with a priority
queue. Show how to implement a stack with a priority queue. (Queues and stacks
are defined in Section 10.1.)}

\begin{framed}
A first-in, first-out queue can be implemented using a min-priority-queue, in
such a way that each heap element is a tuple (key, handle) and the key of a new
element is greater than the key of the current elements. The
\textsc{Extract-Min} operation will always return the oldest element (minimum
key value) and the \textsc{Insert} operation will keep the min-heap property.
A stack can be implemented similarly, but with a max-priority-heap instead of
a min-priority-heap.
\end{framed}

\item[6.5{-}8]{The operation \textsc{Heap-Delete}$(A, i)$ deletes the item in
node $i$ from heap $A$. Give an implementation of \textsc{Heap-Delete} that runs
in $O(\lg n)$ time for an $n$-element max-heap.}

\begin{framed}
The pseudocode is stated below.

\begin{algorithm}[H]
\SetAlgoNoEnd\DontPrintSemicolon
\BlankLine
\SetKwFunction{algo}{Heap-Delete}
\SetKwProg{myalg}{}{}{}
\myalg{\algo{A, i}}{%
\nl $A[i] = A[\text{\emph{heap-size}}]$\;
\nl $A.\text{\emph{heap-size}} = A.\text{\emph{heap-size}} - 1$\;
\nl \texttt{Max-Heapify}$(A, i)$\; }
\end{algorithm}
\end{framed}

\item[6.5{-}9]{Give an $O(n \lg k)$-time algorithm to merge $k$ sorted lists
into one sorted list, where $n$ is the total number of elements in all the input
lists. (Hint: Use a min-heap for $k$-way merging.)}

\begin{framed}
Let $T$ denote the final sorted list and $S_i$ the $i$th input list. The
pseudocode is stated below.

\begin{algorithm}[H]
\SetAlgoNoEnd\DontPrintSemicolon
\BlankLine
\SetKwFunction{algo}{Merge-Lists-Min-Heap}
\SetKwProg{myalg}{}{}{}
\myalg{\algo{$S_1, S_2, \dots, S_k$}}{%
\nl Let $T$ be a list\;
\nl Let $H$ be a min-heap of tuples in the form $(key, j)$\;
\nl \For{$i = 1$ \KwTo $k$}{%
\nl   $\texttt{Insert}(H, (S_i[1], i))$\;
\nl   $p_i = 2$\; }
\nl \While{$H.\text{heap-size} > 0$}{%
\nl   $(key, j) = \texttt{Extract-Min}(H)$\;
\nl   add $key$ to $T$\;
\nl   \If{$p_j \le S_j.\text{length}$}{%
\nl     $\texttt{Insert}(H, (S_j[p_j], j))$\;
\nl     $p_j = p_j + 1$\; } }
\nl \Return{T}\; }
\end{algorithm}

The \textbf{for} loop of lines 3{-}5 runs in $O(k \lg k)$. The \textbf{while}
loop of lines 6{-}11 will iterate $n$ times and each iteration takes $O(\lg k)$.
Since $n \ge k$, the algorithm runs in $O(n \lg k)$.
\end{framed}

\end{enumerate}

\newpage

\section*{Problems}
\addcontentsline{toc}{section}{\protect\numberline{}Problems}%

\begin{enumerate}

\item[6{-}1]{\textbf{\emph{Building a heap using insertion}}\\
We can build a heap by repeatedly calling \textsc{Max-Heap-Insert} to insert
the elements into the heap. Consider the following variation on the
\textsc{Build-Max-Heap} procedure:

\begin{algorithm}[H]
\SetAlgoNoEnd\DontPrintSemicolon
\BlankLine
\SetKwFunction{algo}{Build-Max-Heap'}
\SetKwProg{myalg}{}{}{}
\myalg{\algo{A}}{%
\nl $A.\text{\emph{heap-size}} = 1$\;
\nl \For{$i = 2$ \KwTo $A.\text{length}$}{%
\nl   $\textsc{Max-Heap-Insert}(A, A[i])$\; } }
\end{algorithm}

\begin{enumerate}
\item[\textbf{a.}]{Do the procedures \textsc{Build-Max-Heap} and
\textsc{Build-Max-Heap\texttt{'}} always create the same heap when run on the
same input array? Prove that they do, or provide a counterexample.}
\item[\textbf{b.}]{Show that in the worst-case,
\textsc{Build-Max-Heap\texttt{'}} requires $\Theta(n \lg n)$ time to build an
$n$-element heap.}
\end{enumerate}

}

\begin{framed}
\begin{enumerate}

\item No. Consider the array $A = [1, 2, 3]$. \textsc{Build-Max-Heap}$(A)$ will
create the heap
\begin{center}
\begin{tikzpicture}
\Tree
  [.\node[red]{1};
  \edge[]; {2}
  \edge[]; {3}
]
\end{tikzpicture}
\begin{tikzpicture}
\Tree
  [.3
  \edge[]; {2}
  \edge[]; {1}
]
\end{tikzpicture}
\end{center}

while \textsc{Build-Max-Heap\texttt{'}}$(A)$ will create the heap

\begin{center}
\begin{tikzpicture}
\Tree
  [.1
  \edge[]; \node[red]{2};
  \edge[]; {3}
]
\end{tikzpicture}
\begin{tikzpicture}
\Tree
  [.2
  \edge[]; {1}
  \edge[]; \node[red]{3};
]
\end{tikzpicture}
\begin{tikzpicture}
\Tree
  [.3
  \edge[]; {1}
  \edge[]; {2}
]
\end{tikzpicture}
\end{center}

\item Let $A$ be the input array. The worst-case occurs when there is an integer
$k$ such that $n = A.\text{\emph{length}} = 2^k - 1$ (that is, the elements of
$A$ forms a complete binary tree) and each call to \textsc{Max-Heap-Insert}
covers the entire height of the heap (occurs when $A$ is sorted). Since any heap
has $\ceil{n/2}$ leaves, the last $\ceil{n/2}$ elements of $A$ will be the
leaves of the final heap. Note that, at the time \textsc{Max-Heap-Insert} is
called on these last $\ceil{n/2}$ nodes, the height of the heap will be
$\floor{\lg n}$. Thus, considering only the last $\ceil{n/2}$ calls to
$\textsc{Max-Heap-Insert}$, the algorithm will take $\ceil{n/2} \cdot
\Theta(\floor{\lg n}) = \Theta(n \lg n)$, which implies that the worst-case
of \textsc{Build-Max-Heap\texttt{'}} runs in $\Omega(n \lg n)$. Also, note
that there will be exactly $n - 1$ calls to \textsc{Max-Heap-Insert} and
each call takes $O(\floor{\lg n})$. Thus, the worst-case of
\textsc{Build-Max-Heap\texttt{'}} runs in $\Theta(n \lg n)$.
\end{enumerate}
\end{framed}

\newpage

\item[6{-}2]{\textbf{\emph{Analysis of d-ary heaps}}\\
A \textbf{\emph{d-ary}} heap is like a binary heap, but (with one possible
exception) non-leaf nodes have $d$ children instead of 2 children.
\begin{enumerate}
\item[\textbf{a.}] How would you represent a $d$-ary heap in an array?
\item[\textbf{b.}] What is the height of a $d$-ary heap of $n$ elements in terms
of $n$ and $d$?
\item[\textbf{c.}] Give an efficient implementation of \textsc{Extract-Max} in
a $d$-ary max-heap. Analyze its running time in terms of $d$ and $n$.
\item[\textbf{d.}] Give an efficient implementation of \textsc{Insert} in
a $d$-ary max-heap. Analyze its running time in terms of $d$ and $n$.
\item[\textbf{e.}] Give an efficient implementation of
\textsc{Increase-Key}$(A, i, k)$, which flags an error if $k < A[i]$, but
otherwise sets $A[i] = k$ and then updates the $d$-ary max-heap structure
appropriately. Analyze its running time in terms of $d$ and $n$.
\end{enumerate}
}

\begin{framed}
\begin{enumerate}
\item The root occupies the first $d^0 = 1$ positions of the array, its
children occupies the next $d^1 = d$ positions of the array, and so on until
the bottom level. That way, the parent of a node with index $i$ will be at
position
\[
  \Bigl\lfloor{\frac{i - (d - 2)}{d}}\Bigr\rfloor
  = \Bigl\lfloor{\frac{i - d + 2}{d}}\Bigr\rfloor,
\]
and its $j$th children will be at position
\[
  di - (d - j) + 1 = di - d + j + 1 = d (i - 1) + j + 1.
\]

\item $h = \floor{\log_d n}$.

\item The only major modification is the \textsc{Max-Heapify} procedure. The
pseudocode is stated below.

\begin{algorithm}[H]
\SetAlgoNoEnd\DontPrintSemicolon
\BlankLine
\SetKwFunction{algo}{Child-D-Ary}
\SetKwProg{myalg}{}{}{}
\myalg{\algo{d, i, j}}{%
\nl \Return{$d (i - 1) + j + 1$}\; }
\end{algorithm}

\begin{algorithm}[H]
\SetAlgoNoEnd\DontPrintSemicolon
\BlankLine
\SetKwFunction{algo}{Extract-Max-D-Ary}
\SetKwProg{myalg}{}{}{}
\myalg{\algo{A, d}}{%
\nl \If{$A.\text{heap-size} < 1$}{%
\nl   \textbf{error} ``heap underflow''\; }
\nl $\text{\emph{max}} = A[1]$\;
\nl $A.\text{\emph{heap-size}} = A.\text{\emph{heap-size}} - 1$\;
\nl $\texttt{Max-Heapify-D-Ary}(A, d, 1)$\;
\nl \Return{\text{max}}\; }
\end{algorithm}

\begin{algorithm}[H]
\SetAlgoNoEnd\DontPrintSemicolon
\BlankLine
\SetKwFunction{algo}{Max-Heapify-D-Ary}
\SetKwProg{myalg}{}{}{}
\myalg{\algo{A, d, i}}{%
\nl $\text{\emph{largest}} = i$\;
\nl \For{$j = 1$ \KwTo $d$}{%
\nl   $\text{\emph{child-j}} = \texttt{Child}(d, i, j)$\;
\nl   \If{$\text{child-j} > A.\text{heap-size}$}{%
\nl     break\; }
\nl   \If{$A[\text{child-j}] > A[\text{largest}]$}{%
\nl     $\text{\emph{largest}} = \text{\emph{child-j}}$\; } }
\nl \If{$\text{largest} \neq i$}{%
\nl   exchange $A[i]$ with $A[\text{\emph{largest}}]$\;
\nl   $\texttt{Max-Heapify-D-Ary}(A, d, \text{\emph{largest}})$\; } }
\end{algorithm}

In the worst-case, \textsc{Max-Heapify-D-Ary} covers the height of the heap
(which is $\log_d n$).  Since each recursive call takes $O(d)$,
\textsc{Max-Heapify-D-Ary} runs in $O(d \log_d n)$. The running time of
\textsc{Extract-Max-D-Ary} is also $O(d \log_d n)$, since it performs a constant
amount of work on top of \textsc{Max-Heapify-D-Ary}.

\item The $\textsc{Max-Heap-Insert}$ procedure given in the text book also works
for a $d$-ary heap. The only modification is to use \textsc{Parent-D-Ary}
instead of \textsc{Parent} in the \textsc{Heap-Increase-Key} subroutine. In the
worst case, it will cover the height of the tree. Thus, the running time is
$O(\log_d n)$.

\item The $\textsc{Heap-Increase-Key}$ procedure given in the text book also
works for a $d$-ary heap. The only modification is to use \textsc{Parent-D-Ary}
instead of \textsc{Parent}. In the worst case, it will cover the height of the
tree. Thus, the running time is $O(\log_d n)$.
\end{enumerate}
\end{framed}

\newpage

\item[6{-}3]{\textbf{\emph{Young tableaus}}\\
An $m \times n$ \textbf{\emph{Young tableau}} is an $m \times n$ matrix such
that the entries of each row are in sorted order from left to right and the
entries of each column are in sorted order from top to bottom. Some of the
entries of a Young tableau may be $\infty$, which we treat as nonexistent
elements. Thus, a Young tableau can be used to hold $r \le mn$ finite numbers.
\begin{enumerate}
\item[\textbf{a.}] Draw a $4 \times 4$ Young tableau containing the elements
$\{9, 16, 3, 2, 4, 8, 5, 14, 12\}$.
\item[\textbf{b.}] Argue that an $m \times n$ Young tableau $Y$ is empty if
$Y[1, 1] = \infty$. Argue that $Y$ is full (contains $mn$ elements) if
$Y[m, n] < \infty$.
\item[\textbf{c.}] Give an algorithm to implement \textsc{Extract-Min} on
a non-empty $m \times n$ Young tableau that runs in $O(m + n)$ time. Your
algorithm should use a recursive subroutine that solves an $m \times n$ problem
by recursively solving either an $(m - 1) \times n$ or an $m \times (n - 1)$
subproblem. (Hint: Think about \textsc{Max-Heapify}.) Define $T(p)$, where
$p = m + n$, to be the maximum running time of \textsc{Extract-Min} on any
$m \times n$ Young tableau. Give and solve a recurrence for $T(p)$ that yields
the $O(m + n)$ time bound.
\item[\textbf{d.}] Show how to insert a new element into a nonfull $m \times n$
Young tableau in $O(m + n)$ time.
\item[\textbf{e.}] Using no other sorting method as a subroutine, show how to
use an $n \times n$ Young tableau to sort $n^2$ numbers in $O(n^3)$ time.
\item[\textbf{f.}] Give an $O(m + n)$-time algorithm to determine whether
a given number is stored in a given $m \times n$ Young tableau.
\end{enumerate}
}

\begin{framed}
\begin{enumerate}
\item Here it is:

\begin{center}
\begin{tabular}{cccc}
  2 & 3 & 4 & 5\\
  8 & 9 & 12 & 14\\
  16 & $\infty$ & $\infty$ & $\infty$\\
  $\infty$ & $\infty$ & $\infty$ & $\infty$
\end{tabular}
\end{center}

\item From the definition of a Young tableau, we have
\[
  Y[1, 1] \le Y[i, j]\;\Forall i, j,
\]
which implies
\[
  Y[1, 1] = \infty \rightarrow Y[i, j] = \infty\;\Forall i, j.
\]
Thus, $Y$ is full of $\infty$ and is therefore empty.

From the definition of a Young tableau, we have
\[
  Y[m, n] \ge Y[i, j]\;\Forall i, j,
\]
which implies
\[
  Y[m, n] < \infty \rightarrow Y[i, j] < \infty\;\Forall i, j.
\]
Thus, $Y$ does not have any $\infty$ and is therefore full.

\item The pseudocode is stated below.

\begin{algorithm}[H]
\SetAlgoNoEnd\DontPrintSemicolon
\BlankLine
\SetKwFunction{algo}{Extract-Min}
\SetKwProg{myalg}{}{}{}
\myalg{\algo{Y}}{%
\nl $\text{\emph{min}} = Y[1, 1]$\;
\nl \If{$\text{min} == \infty$}{%
\nl    \textbf{error} ``tableau underflow''\; }
\nl $Y[1, 1] = \infty$\;
\nl $\texttt{Min-Tableauify}(Y, 1, 1)$\;
\nl \Return{min}\; }
\end{algorithm}

\begin{algorithm}[H]
\SetAlgoNoEnd\DontPrintSemicolon
\BlankLine
\SetKwFunction{algo}{Min-Tableauify}
\SetKwProg{myalg}{}{}{}
\myalg{\algo{Y, i, j}}{%
\nl $\text{\emph{smallest-i}} = i$\;
\nl $\text{\emph{smallest-j}} = j$\;
\nl \If{$i + 1 \le Y.\text{rows}$ \textbf{\upshape{and}} $Y[i + 1, j] < Y[\text{smallest-i}, \text{smallest-j}]$}{%
\nl   $\text{\emph{smallest-i}} = i + 1$\; }
\nl \If{$j + 1 \le Y.\text{cols}$ \textbf{\upshape{and}} $Y[i, j + 1] < Y[\text{smallest-i}, \text{smallest-j}]$}{%
\nl   $\text{\emph{smallest-i}} = i$\;
\nl   $\text{\emph{smallest-j}} = j + 1$\; }
\nl \If{$i \neq \text{smallest-i}$ \textbf{\upshape{or}} $j \neq \text{smallest-j}$}{%
\nl   $Y[i, j] = Y[\text{\emph{smallest-i}}, \text{\emph{smallest-j}}]$\;
\nl   $Y[\text{\emph{smallest-i}}, \text{\emph{smallest-j}}] = \infty$\;
\nl   $\texttt{Min-Tableauify}(Y, \text{\emph{smallest-i}}, \text{\emph{smallest-j}})$\; } }
\end{algorithm}

The algorithm has the recurrence $T(p) \le T(p - 1) + \Theta(1) = O(p) + O(m + n)$.

\item The pseudocode is stated below.

\begin{algorithm}[H]
\SetAlgoNoEnd\DontPrintSemicolon
\BlankLine
\SetKwFunction{algo}{Tableau-Insert}
\SetKwProg{myalg}{}{}{}
\myalg{\algo{Y, key}}{%
\nl $i = Y.\text{\emph{rows}}$\;
\nl $j = Y.\text{\emph{cols}}$\;
\nl \If{$Y[i, j] == \infty$}{%
\nl   \textbf{error} ``tableau overflow''\; }
\nl $Y[i, j] = \text{\emph{key}}$\;
\nl \While{$i > 1$ \textbf{\upshape{or}} $j > 1$}{%
\nl   \If{$i > 1$ \textbf{\upshape{and}} $Y[i, j] < Y[i - 1, j]$}{%
\nl     exchange $Y[i, j]$ with $Y[i - 1, j]$\;
\nl     $i = i - 1$\; }
\nl   \ElseIf{$j > 1$ \textbf{\upshape{and}} $Y[i, j] < Y[i, j - 1]$}{%
\nl     exchange $Y[i, j]$ with $Y[i, j - 1]$\;
\nl     $j = j - 1$\; }
\nl   \Else{%
\nl     break\; } } }
\end{algorithm}

We first set the element to the position $A[m, n]$ and move it upwards and/or
leftwards until we find a valid position. The running time is $O(m + n)$

\item The pseudocode is stated below.

\begin{algorithm}[H]
\SetAlgoNoEnd\DontPrintSemicolon
\BlankLine
\SetKwFunction{algo}{Build-Young-Tableau}
\SetKwProg{myalg}{}{}{}
\myalg{\algo{A}}{%
\nl $n = \sqrt{A.\text{\emph{length}}}$\;
\nl Let $Y$ be an $n \times n$ array\;
\nl \For{$i = 1$ \KwTo $A.\text{\emph{length}}$}{%
\nl   \texttt{Tableau-Insert}$(Y, A[i])$\; }
\nl \Return{$Y$}\; }
\end{algorithm}

\begin{algorithm}[H]
\SetAlgoNoEnd\DontPrintSemicolon
\BlankLine
\SetKwFunction{algo}{Tableausort}
\SetKwProg{myalg}{}{}{}
\myalg{\algo{A}}{%
\nl $Y = \textsc{Build-Young-Tableau}(A)$\;
\nl \For{$i = 1$ \KwTo $A.\text{length}$}{%
\nl   $A[i] = \texttt{Extract-Min}(Y)$\; } }
\end{algorithm}

The \textsc{Build-Young-Tableau} procedure runs in
$n^2 \cdot O(n + n) = O(n^3)$. Each call to \textsc{Min-Tableauify} runs in
$O(n + n) = O(n)$. Thus, the algorithm runs in
$O(n^3) + n^2 \cdot O(n) = O(n^3)$.

\item The pseudocode is stated below.

\begin{algorithm}[H]
\SetAlgoNoEnd\DontPrintSemicolon
\BlankLine
\SetKwFunction{algo}{Tableau-Find}
\SetKwProg{myalg}{}{}{}
\myalg{\algo{Y, key}}{%
\nl $i = Y.\text{\emph{rows}}$\;
\nl $j = 1$\;
\nl \While{$i \ge 1$ \textbf{\upshape{and}} $j \le Y.\text{cols}$}{%
\nl   \If{$Y[i, j] == \text{\emph{key}}$}{%
\nl     \Return{\texttt{\upshape{True}}}\; }
\nl   \ElseIf{$Y[i, j] \le Y[i - 1, j]$}{%
\nl     $i = i - 1$\; }
\nl   \ElseIf{$Y[i, j] \ge Y[i, j + 1]$}{%
\nl     $j = j + 1$\; }
\nl   \Else{%
\nl     break\; }
\nl \Return{\texttt{\upshape{False}}}\; } }
\end{algorithm}

\end{enumerate}
\end{framed}

\end{enumerate}


\documentclass{article}
\usepackage{amsmath}
\usepackage[margin=1.0in]{geometry}
\usepackage{xcolor}

\begin{document}

\noindent
Does $\displaystyle \sum_{n=1}^\infty \frac{n^n}{n!}$
diverge, converge absolutely, or converge conditionally?

\subsection*{Solution}

\begin{align*}
L&=\lim_{n \to \infty} \left|\frac{a_{n+1}}{a_n}\right|\\
&= \lim_{n \to \infty} \left| \frac{(n+1)^{n+1}}{(n+1)!} \cdot \frac{n!}{n^n}\right|\\
&= \lim_{n \to \infty} \left| \left(\frac{n+1}{n}\right)^n\right|\\
&= \lim_{n \to \infty}  \left(\frac{n+1}{n}\right)^n.
\end{align*}
This is an indeterminate form, so we let $y = \left(\frac{n+1}{n}\right)^n$. So $\ln y = n \ln \frac{n+1}{n}$. Applying limit to both sides, we have
\[ \lim_{n \to \infty} \ln y =  \lim_{n \to \infty}  n \ln \frac{n+1}{n}.\]
The limit on the right can be computed using L'Hopital's rule (with a chain rule and quotent rule).
\begin{align*}
&\lim_{n \to \infty}  n \ln \frac{n+1}{n} \text{, which was the limit on the right side above}\\
&= \lim_{n \to \infty} \frac{ \ln \frac{n+1}{n}}{\frac1n}\\
&= \lim_{n \to \infty} \frac{ \frac{n}{n+1} \cdot \frac{n \cdot 1 - (n+1) \cdot 1}{n^2}}{\frac{-1}{n^2}} \text{ by L'Hopital's}\\
&= \lim_{n \to \infty} \frac{n}{n+1} \text{ after algebra simplification}\\
&= \lim_{n \to \infty} \frac11\\
&= 1.
\end{align*}
So, $\lim y = e^1 = e$. So $L=e$.

Since $L>1$, the series $\displaystyle \sum_{n=1}^\infty \frac{n^n}{n!}$ diverges by the Ratio Test.

\end{document}%%%%%%%%%%%%%%%%%

\begin{align*}
L&=\lim_{n \to \infty} \sqrt[n]{|a_n|}\\
&= \lim_{n \to \infty} \sqrt[n]{\left| \right|}\\
\end{align*}

\begin{align*}
L&=\lim_{n \to \infty} \left|\frac{a_{n+1}}{a_n}\right|\\
&= \lim_{n \to \infty} \left| \right|\\
\end{align*}

\begin{align*}
\lim_{n \to \infty} a_n
&= \lim_{n \to \infty} \\
\end{align*}

Since $\sum |a_n| = \sum a_n$, the series $\displaystyle \sum_{n=1}^\infty AAAAAAAAAAAAAA$ converges absolutely.

Since $|r| < 1$, the series ...  converges by the Geometric Series Test.

Since $|r| \geq 1$, the series ...  diverges by the Geometric Series Test.

The function $f(x)=\frac{}{}$ is continuous, positive, and decreasing on $[1,\infty)$.

\subsection*{Solution}