\chapter{Linear Models for Classification}

\section*{Exercise 4.1 $\star \star$}
Given a set of data points $\{\mathbf{x}_n\}$, we can define the
$\emph{convex hull}$ to be the data set of all points
$\mathbf{x}$ given by
\begin{equation}\label{eq:4.156}\tag{4.156}
    \mathbf{x} = \sum_{n} \alpha_n \mathbf{x}_n
\end{equation}
where $\alpha_n \geq 0$ and $\sum_{n} \alpha_n = 1$. Consider
a second set of points $\{\mathbf{y}_n\}$ together with
their corresponding convex hull. By definition,
the two set of points will be linearly separable
if there exists a vector $\widehat{\mathbf{w}}$ and
a scalar $w_0$ such that $\widehat{\mathbf{w}}^T\mathbf{x}_n + w_0 > 0$
for all $\mathbf{x}_n$, and $\widehat{\mathbf{w}}^T\mathbf{y}_n + w_0 < 0$
for all $\mathbf{y}_n$. Show that if their convex hulls intersect,
the two sets of points cannot be linearly separable, and conversely
that if they are linearly separable, their covex hulls do not intersect.

\vspace{1em}

\begin{proof}
    The vertices of the convex hulls are the data points $\{\mathbf{x}_n\}$ and $\{\mathbf{y}_n$\}.
    Therefore, the edges of the hulls will be represented by some segments beteween the data points.
    As a result, any point situated on the boundary of the hull can be written as a convex
    combination of the end-points of the segment it's contained by. Also,
    one can easily see that if two hulls intersect, they intersect in at least one
    point that is contained by the boundaries of both hulls.

    \vspace{1em}
    \textbf{1st Hypothesis}: If the hulls intersect, the two sets of
    points are not linearly separable
    \vspace{1em}

    Assume that the two hulls intersect in the point $\mathbf{z}$ situated
    on both hulls boundaries. From what we've seen above,
    the point $\mathbf{z}$ can be expressed as a convex
    combination between two data points of each set of data points. Therefore,
    there exist
    $\mathbf{x}_A, \mathbf{x}_B$ from $\{\mathbf{x}_n\}, \mathbf{y}_A, \mathbf{y}_B$
    from $\{\mathbf{y}_n\}$ and $\lambda_\mathbf{x}, \lambda_\mathbf{y} \in [0, 1]$ such that
    we can express $\mathbf{z}$ as
     \[
         \lambda_\mathbf{x} \mathbf{x}_A + (1 - \lambda_\mathbf{x}) \mathbf{x}_B =
         \lambda_\mathbf{y} \mathbf{y}_A + (1 - \lambda_\mathbf{y}) \mathbf{y}_B
    \]
    Suppose that the sets \{$\mathbf{x}_n$\} and $\{\mathbf{y}_n\}$ are linearly
    separable. Then there exists a discriminant function
    \[
        \theta(\mathbf{a}) = \widehat{\mathbf{w}}^T \mathbf{a} + w_0
    \]
    such that $\theta(\mathbf{x}_n) > 0$ for all $\mathbf{x}_n$ and $\theta(\mathbf{y}_n) < 0$
    for all $\mathbf{y}_n$. From the linearity of the discriminant function,
    and rewritting $\theta(\mathbf{z})$ using the convex combinations forms,
    we have that
    \[
        \lambda_\mathbf{x} \theta(\mathbf{x}_A) + (1 - \lambda_\mathbf{x}) \theta(\mathbf{x}_B)
        = \lambda_\mathbf{y} \theta(\mathbf{y}_A) + (1 - \lambda_\mathbf{y}) \theta(\mathbf{y}_B)
    \]
    Since $\theta(\mathbf{x}_A), \theta(\mathbf{x}_B) > 0$ and
    $\theta(\mathbf{y}_A), \theta(\mathbf{y}_B) < 0$, this
    expression is obviously false, since the left-hand side of the equality
    is positive and the right-hand one is negative. Therefore, our supposition
    that the data sets are linearly separable is false and our main hypothesis is true.

    \vspace{1em}
    \textbf{2nd Hypothesis}: If the two sets of points are linearly separable,
    then the hulls don't intersect.
    \vspace{0.25em}

    This hypothesis is the counterpositive of the 1st hypothesis. Therefore,
    it's valid too.
\end{proof}

\section*{Exercise 4.2 $\start \star$ \textbf{TODO}}
Consider the minimization of a sum-of-squares error
function ($\ref{eq:4.15}$), and suppose that all of the
target vectors in the training set satisfy a linear constraint
\begin{equation}\label{eq:4.157}\tag{4.157}
    \mathbf{a}^T\mathbf{t}_n + b = 0
\end{equation}
where $\mathbf{t}_n$ corresponds to the $n^\text{th}$ row of the matrix $\mathbf{T}$
in $(\ref{eq:4.15})$. Show that as a consequence of this constraint,
the elements of the model prediction $\mathbf{y}(\mathbf{x})$ given
by the least-squares solution ($\ref{eq:4.17}$) also satisfy
this constraint, so that
\begin{equation}\label{eq:4.158}\tag{4.158}
    \mathbf{a}^T\mathbf{y}(\mathbf{x}) + b = 0
\end{equation}
To do so, assume that one of the basis functions $\phi_0(\mathbf{x}) = 1$,
so that the corresponding parameter $w_0$ plays the role of a bias.

\section*{Exercise 4.4 $\star$}
Show that maximization of the class separation criterion given by
($\ref{eq:3.24}$) with respect to $\mathbf{w}$, using a Lagrange multiplier
to enforce the constraint $\mathbf{w}^T\mathbf{w} = 1$, leads to the
result that $\mathbf{w} \propto (\mathbf{m}_2 - \mathbf{m}_1)$.

\vspace{1em}

\begin{proof}
    Our goal is to maximize
    \begin{equation}\label{eq:3.24}\tag{3.24}
        m_2 - m_1 = \mathbf{w}^T(\mathbf{m}_2 - \mathbf{m}_1)
    \end{equation}
    with the constraint that $\mathbf{w}^T\mathbf{w} = 1$.
    The corresponding Lagrangian is given by
    \[
        \mathcal{L}(\mathbf{w}, \lambda)
        = \mathbf{w}^T(\mathbf{m}_2 - \mathbf{m}_1) + \lambda(\mathbf{w}^T\mathbf{w} - 1)
    \]
    By taking the gradient of this with respect to $\mathbf{w}$ and $\lambda$,
    we have that
    \[
        \nabla_{\mathbf{w}, \lambda} \mathcal{L}(\mathbf{w}, \lambda)
        = \begin{pmatrix}
            \mathbf{m}_2 - \mathbf{m}_1 + 2\lambda\mathbf{w} \\
            \mathbf{w}^T\mathbf{w} - 1
        \end{pmatrix}
    \]
    Setting to 0 the derivative with respect to $\mathbf{w}$ gives the initial result, that is
    \[
        \mathbf{w} = \frac{\mathbf{m}_1 - \mathbf{m}_2}{2\lambda}
    \]
    By replacing into the $\lambda$ derivative and setting it to 0, we'd obtain that
     \[
         \lambda = \frac{1}{4} ||\mathbf{m}_1 - \mathbf{m}_2||^2
    \]
    which gives
    \[
        \mathbf{w} = \frac{2(\mathbf{m}_1 - \mathbf{m}_2)}{||\mathbf{m}_1 - \mathbf{m}_2||^2}
        \propto (\mathbf{m}_2 - \mathbf{m}_1)
    \]
\end{proof}

\section*{Exercise 4.5 $\star$}
By making use of (4.20), ($\ref{eq:4.23}$), and (4.24), show that the Fischer
criterion ($\ref{eq:4.25}$) can be written in the form ($\ref{eq:4.26}$).

\vspace{1em}

\begin{proof}
    The Fisher criterion is defined to be the ratio of the between-class
    variance to the within-class variance and is given by
    \begin{equation}\label{eq:4.25}\tag{4.25}
        J(\mathbf{w}) = \frac{(m_2 - m_1)^2}{s_1^2 + s_2^2}
    \end{equation}
    where
    \begin{equation}\label{eq:4.23}\tag{4.23}
        m_k = \mathbf{w}^T\mathbf{m}_k
    \end{equation}
    and
    \begin{equation}\label{eq:4.24}\tag{4.24}
        s_k^2 = \sum_{n \in \mathcal{C}_k} (y_n - m_k)^2
    \end{equation}
    By substituting $(\ref{eq:4.23})$ into the numerator of the Fischer expression,
    \begin{align*}
        (m_2 - m_1)^2
        = (\mathbf{w}^T\mathbf{m}_2 - \mathbf{w}^T\mathbf{m}_1)^2
        = \mathbf{w}^T(\mathbf{m}_2 - \mathbf{m}_1)\mathbf{w}^T(\mathbf{m}_2 - \mathbf{m}_1)
        &= \mathbf{w}^T(\mathbf{m}_2 - \mathbf{m}_1)(\mathbf{m}_2 - \mathbf{m}_1)^T\mathbf{w} \\
        &= \mathbf{w}^T\mathbf{S}_B\mathbf{w}
    \end{align*}
    where $\mathbf{S}_B$ is the $\emph{between-class}$ covariance matrix and is given by
    \begin{equation}\label{eq:4.27}\tag{4.27}
        \mathbf{S}_B = (\mathbf{m}_2 - \mathbf{m}_1)(\mathbf{m}_2 - \mathbf{m}_1)^T
    \end{equation}
    Similarily, we use the fact that the projection of the D-dimensional input vector
    $\mathbf{w}$ to one dimension is given by
    \begin{equation}\label{eq:4.20}\tag{4.20}
        y = \mathbf{w}^T\mathbf{x}
    \end{equation}
    along with $(\ref{eq:4.23})$ and $(\ref{eq:4.24})$ to rewrite the denominator as
    \begin{align*}
        s_1^2 + s_2^2
        &= \sum_{n \in \mathcal{C}_1} (y_n - m_1)^2 + \sum_{n \in \mathcal{C}_2} (y_n - m_2)^2 \\
        &= \sum_{n \in \mathcal{C}_1} (\mathbf{w}^T\mathbf{x}_n - \mathbf{w}^T\mathbf{m}_1)^2
            + \sum_{n \in \mathcal{C}_2} (\mathbf{w}^T\mathbf{x}_n - \mathbf{w}^T\mathbf{m}_2)^2 \\
        &= \sum_{n \in \mathcal{C}_1} \mathbf{w}^T(\mathbf{x}_n - \mathbf{m}_1)
            (\mathbf{x}_n - \mathbf{m}_1)^T\mathbf{w}
            + \sum_{n \in \mathcal{C}_2} \mathbf{w}^T(\mathbf{x}_n - \mathbf{m}_2)
            (\mathbf{x}_n - \mathbf{m}_2)^T\mathbf{w} \\
        &= \mathbf{w}^T \bigg[\sum_{n \in \mathcal{C}_1}
            (\mathbf{x}_n - \mathbf{m}_1)(\mathbf{x}_n - \mathbf{m}_1)^T
            + \sum_{n \in \mathcal{C}_2}
            (\mathbf{x}_n - \mathbf{m}_2)(\mathbf{x}_n - \mathbf{m}_2)^T
            \bigg]\mathbf{w} \\
        &= \mathbf{w}^T\mathbf{S}_W\mathbf{w}
    \end{align*}
    where $\mathbf{S}_W$ is the $\emph{within-class}$ covariance matrix and is given by
    \begin{equation}\label{eq:4.28}\tag{4.28}
        \mathbf{S}_W =
           \sum_{n \in \mathcal{C}_1}
            (\mathbf{x}_n - \mathbf{m}_1)(\mathbf{x}_n - \mathbf{m}_1)^T
            + \sum_{n \in \mathcal{C}_2}
            (\mathbf{x}_n - \mathbf{m}_2)(\mathbf{x}_n - \mathbf{m}_2)^T
    \end{equation}
    Finally, by substituting the new expressions into ($\ref{eq:4.25}$), we can rewrite
    the Fischer criterion as
    \begin{equation}\label{eq:4.26}\tag{4.26}
        J(\mathbf{w}) = \frac{\mathbf{w}^T\mathbf{S}_B\mathbf{w}}{\mathbf{w}^T\mathbf{S}_W\mathbf{w}}
    \end{equation}
\end{proof}

\section*{Exercise 4.7 $\star$}
Show that the logistic sigmoid function $(\ref{eq:4.59})$ satisfies the property
$\sigma(-a) = 1 - \sigma(a)$ and that its inverse is given by
$\sigma^{-1}(y) = \ln\bigg(\displaystyle \frac{y}{1 - y}\bigg)$.

\vspace{1em}

\begin{proof}
    The sigmoid function is given by
    \begin{equation}\label{eq:4.59}\tag{4.59}
        \sigma(a) = \frac{1}{1 + e^{-a}}
    \end{equation}
    The symmetry property is easily satisfied, as
    \begin{equation}\label{eq:4.60}\tag{4.60}
        \sigma(-a)
        = \frac{1}{1 + e^a}
        = \frac{1 + e^a + 1 + e^{-a}}{(1 + e^{-a})(1 + e^a)}
        - \frac{1 + e^a}{(1 + e^{-a})(1 + e^a)}
        = 1 - \frac{1}{1 + e^{-a}}
        = 1 - \sigma(a)
    \end{equation}\label{eq:4.60}
    The sigmoid function is bijective, so inversable. Therefore, let $\sigma(x) = y$.
    Then,
    \[
        y = \frac{1}{1 + e^{-x}}
        \iff (1 + e^{-x})y = 1
        \iff e^{-x} &= \frac{1 - y}{y}
        \iff x = \ln\bigg(\frac{y}{1-y}\bigg)
    \]
    so the inverse of the sigmoid function is given by
    \[
        \sigma^{-1}(y) = \ln\bigg(\frac{y}{1 - y}\bigg)
    \]
\end{proof}

\section*{Exercise 4.8 $\star$}
Using ($\ref{eq:4.57}$) and $(\ref{eq:4.58})$, derive the result $(\ref{eq:4.65})$
for the posterior class probability in the two-class generative model with Gaussian
densities, and verify the results $(\ref{eq:4.66})$ and $(\ref{eq:4.67})$ for the
parameters $\mathbf{w}$ and $w_0$.

\vspace{1em}

\begin{proof}
    It is known that the posterior probability for class $\mathcal{C}_1$ can
    be written as
    \begin{equation}\label{eq:4.57}\tag{4.57}
        p(\mathcal{C}_1 | \mathbf{x}) = \sigma(a)
    \end{equation}
    where we have defined
    \begin{equation}\label{eq:4.58}\tag{4.58}
        a = \ln \frac{p(\mathbf{x} | \mathcal{C}_1)p(\mathcal{C}_1)}
            {p(\mathbf{x} | \mathcal{C}_2)p(\mathcal{C}_2)}
    \end{equation}
    and $\sigma$ is the logistic sigmoid function defined by $(\ref{eq:4.59})$.
    We start by expanding $a$ and rewritting it as
    \[
        a = \ln p(\mathbf{x} | \mathcal{C}_1) - \ln p(\mathbf{x} | \mathcal{C}_2)
            + \ln \frac{p(\mathcal{C}_1)}{p(\mathcal{C}_2)}
    \]
    Since the class-conditional densities are Gaussian, i.e. the density
    for a class $\mathcal{C}_k$ is given by
    \begin{equation}\label{eq:4.64}\tag{4.64}
        p(\mathbf{x} | \mathcal{C}_k) = \mathcal{N}(\mathbf{x} | \bm{\mu}_k, \mathbf{\Sigma})
        = \frac{1}{(2\pi)^{D/2}} \frac{1}{|\mathbf{\Sigma}|^{1/2}}
        \exp \bigg\{-\frac{1}{2} (\mathbf{x} - \bm{\mu}_k)^T\mathbf{\Sigma}^{-1}
        (\mathbf{x} - \bm{\mu}_k)\bigg\}
    \end{equation}
    one can easily obtain that
    \begin{align*}
        a
        &= \ln p(\mathbf{x} | \mathcal{C}_1) - \ln p(\mathbf{x} | \mathcal{C}_2)
        + \ln \frac{p(\mathcal{C}_1)}{p(\mathcal{C}_2)}\\
        &= \bm{\mu}_1^T\mathbf{\Sigma}^{-1}\mathbf{x}
        - \bm{\mu}_2^T\mathbf{\Sigma}^{-1}\mathbf{x}
        - \frac{1}{2} \bm{\mu}_1^T\mathbf{\Sigma}^{-1}\bm{\mu}_1
        + \frac{1}{2} \bm{\mu}_2^T\mathbf{\Sigma}^{-1}\bm{\mu}_2
        + \ln \frac{p(\mathcal{C}_1)}{p(\mathcal{C}_2)}\\
        &= (\bm{\mu}_1 - \bm{\mu}_2)^T \mathbf{\Sigma}^{-1} \mathbf{x}
        - \frac{1}{2} \bm{\mu}_1^T\mathbf{\Sigma}^{-1}\bm{\mu}_1
        + \frac{1}{2} \bm{\mu}_2^T\mathbf{\Sigma}^{-1}\bm{\mu}_2
        + \ln \frac{p(\mathcal{C}_1)}{p(\mathcal{C}_2)}\\
        &= \mathbf{w}^T\mathbf{x} + w_0
    \end{align*}
    where we have defined
    \begin{equation}\label{eq:4.66}\tag{4.66}
        \mathbf{w} = \mathbf{\Sigma}^{-1}(\bm{\mu}_1 - \bm{\mu}_2)
    \end{equation}
    \vspace{-1em}
    \begin{equation}\label{eq:4.67}\tag{4.67}
        w_0 =
        - \frac{1}{2} \bm{\mu}_1^T\mathbf{\Sigma}^{-1}\bm{\mu}_1
        + \frac{1}{2} \bm{\mu}_2^T\mathbf{\Sigma}^{-1}\bm{\mu}_2
        + \ln \frac{p(\mathcal{C}_1)}{p(\mathcal{C}_2)}
    \end{equation}
    Therefore, the posterior probability for class $\mathcal{C}_1$
    is given by
    \begin{equation}\label{eq:4.65}\tag{4.65}
        p(\mathcal{C}_1 | \mathbf{x}) = \sigma(\mathbf{w}^T\mathbf{x} + w_0)
    \end{equation}
\end{proof}

\section*{Exercise 4.9 $\star$}
Consider a generative classification model for $K$ classes
defined by prior class probabilities $p(\mathcal{C}_k) = \pi_k$
and general class-conditional densities  $p(\bm{\phi} | \mathcal{C}_k)$
where $\mathbf{\phi}$ is the input feature vector. Suppose
we are given a training set $\{\bm{\phi}_n, \mathbf{t}_n\}$ where
$n = 1,\ldots, N$, and $\mathbf{t}_n$ is a binary target vector
of length $K$ that use the 1-of-$K$ coding scheme, so
that it has components $t_{nj} = \mathbf{I}_{jk}$ if pattern
$n$ is from class $\mathcal{C}_k$. Assuming that the data points
are drawn independently from this model, show that the
maximum-likelihood solution for the prior probabilites is given
by
\begin{equation}\label{eq:4.159}\tag{4.159}
    \pi_k = \frac{N_k}{N}
\end{equation}
where $N_k$ is the number of data points assigned to class
$\mathcal{C}_k$.

\vspace{1em}

\begin{proof}
    Let $\mathbf{T}$ be the $N \times K$ matrix with the rows $\mathbf{t}_n^T$ and
    $\mathbf{\Phi}$ the $N \times M$ matrix with the rows $\bm{\phi}_n^T$.
    Also, let's define the column vector $\bm{\pi} = (\pi_1, \pi_2, \ldots, \pi_K)^T$.
    We have that
    \[
        p(\bm{\phi}_n, \mathcal{C}_k)
        = p(\mathcal{C}_k)p(\bm{\phi}_n | \mathcal{C}_k)
        = \pi_k p(\bm{\phi}_n | \mathcal{C}_k)
    \]
    so the likelihood function is given by
    \[
        p(\mathbf{T} | \mathbf{\Phi}, \bm{\pi})
        = \prod_{n = 1}^N p(\mathbf{t}_n | \mathbf{\Phi}, \bm{\pi})
        = \prod_{n = 1}^N \prod_{j = 1}^K
        \bigg[\pi_j p(\bm{\phi}_n | \mathcal{C}_j)\bigg]^{t_{nj}}
    \]
    The log likelihood is then easily derived as
    \[
        \ln p(\mathbf{T} | \mathbf{\Phi}, \bm{\pi})
        = \sum_{n=1}^{N} \sum_{j=1}^{K} \bigg(t_{nj} \ln \pi_j
        + t_{nj} \ln p(\bm{\phi}_n | \mathcal{C}_j)\bigg)
    \]
    We aim to minimize this with respect to $\pi_k$, while
    still maintaining the constraint $\sum_{k=1}^{N} \pi_k = 1$
    Therefore, by only keeping the terms depending on $\pi_k$, we obtain the Lagrangian
    \[
        \mathcal{L}(\bm{\pi}, \lambda)
        = \sum_{n=1}^{N} \sum_{j=1}^{K} t_{nj} \ln \pi_j
        + \lambda \bigg(\sum_{j=1}^{N} \pi_j - 1\bigg)
    \]
    with the gradient
    \[
        \nabla_{\pi_k, \lambda} \mathcal{L}(\bm{\pi}, \lambda)
        = \begin{pmatrix}
            \displaystyle \frac{1}{\pi_k} \sum_{n=1}^{N} t_{nk} + \lambda \\
            \\
            \displaystyle \sum_{k=1}^{N} \pi_k - 1
        \end{pmatrix}
    \]
    By setting this gradient to $0$, from the first relation we have that
    \[
        \pi_k\lambda = -\sum_{n=1}^{N} t_{nk} = -N_k
    \]
    Summing this over $k$, one can see that
     \[
         \lambda = -N
    \]
    After substituting this into the derivative and then setting it to 0,
    we obtain the maximum-likelihood solution for $\pi_k$, that is
    \[
        \pi_k = \frac{N_k}{N}
    \]
\end{proof}

\section*{Exercise 4.10 $\star \star$}
Consider the classification model of Exercise 4.9 and now suppose
that the class-conditional densities are given by Gaussian
distributions with a shared covariance matrix, so that
\begin{equation}\label{eq:4.160}\tag{4.160}
    p(\bm{\phi} | \mathcal{C}_k) = \mathcal{N}(\bm{\phi} | \bm{\mu}_k, \mathbf{\Sigma})
\end{equation}
Show that the maximum likelihood solution for the mean of the
Gaussian distribution for class $\mathcal{C}_k$ is given by
\begin{equation}\label{eq:4.161}\tag{4.161}
    \bm{\mu}_k = \frac{1}{N_k} \sum_{n=1}^{N} t_{nk} \bm{\phi}_n
\end{equation}
which represents the mean of those feature vectors assigned
to class $\mathcal{C}_k$. Similarly, show that the maximum likelihood
solution for the shared covariance matrix is given by
\begin{equation}\label{eq:4.162}\tag{4.162}
    \mathbf{\Sigma} = \sum_{k=1}^{K} \frac{N_k}{N}\mathbf{S}_k
\end{equation}
where
\begin{equation}\label{eq:4.163}\tag{4.163}
\mathbf{S}_k = \frac{1}{N_k} \sum_{n=1}^{N}
t_{nk}(\bm{\phi}_n - \bm{\mu_k})(\bm{\phi}_{k} - \bm{\mu}_k)^T
\end{equation}
Thus $\mathbf{\Sigma}$ is given by a weighted average of the covariances
of the data associated with each class, in which the weighting coefficients
are given by the prior probabilities of the classes.

\vspace{1em}

\begin{proof}
    Using the same notations as in the last exercise, we remember that the
    log likelihood is given by
    \[
        \ln p(\mathbf{T} | \mathbf{\Phi}, \bm{\pi})
        = \sum_{n=1}^{N} \sum_{j=1}^{K} \bigg(t_{nj} \ln \pi_j
        + t_{nj} \ln p(\bm{\phi}_n | \mathcal{C}_j)\bigg)
    \]
    By keeping only the parts depending on $\bm{\mu}_k$,
    \[
        \ln p(\mathbf{T} | \mathbf{\Phi}, \bm{\pi})
        = -\frac{1}{2}\sum_{n=1}^{N} \sum_{j=1}^{K}
        t_{nj} (\bm{\phi}_n - \bm{\mu}_k)^T\mathbf{\Sigma}^{-1}
            (\bm{\phi}_n - \bm{\mu}_k) + \text{const}
    \]
    For a symmetric matrix $\mathbf{W}$, one could show that
    \[
        \pdv{s}(\mathbf{x} - \mathbf{s})^T\mathbf{W}(\mathbf{x} - \mathbf{s})
        = -2 \mathbf{W}(\mathbf{x} - \mathbf{s})
    \]
    Therefore, the derivative with respect to $\mathbf{\mu}_k$ of the
    log-likelihood is given by
    \[
        \pdv{\bm{\mu}_k}
        \ln p(\mathbf{T} | \mathbf{\Phi}, \bm{\pi})
        = \sum_{n=1}^{N} t_{nk} \mathbf{\Sigma}^{-1}(\bm{\phi}_n - \bm{\mu}_k)
    \]
    Since $\sum_{n=1}^{N} t_{nk} = N_k$,
    by setting the derivative to 0 and rearranging the terms, we have that
    the solution for maximum likelihood is
    \[
        \bm{\mu}_k = \frac{1}{N_k} \sum_{n=1}^{N} t_{nk} \bm{\phi}_n
        \tag{4.161}
    \]
    Similarly, we do the same for the shared covariance matrix.
    By keeping only the terms depending on $\mathbf{\Sigma}$, the
    log likelihood is given by
    \begin{align*}
        \ln p(\mathbf{T} | \mathbf{\Phi}, \bm{\pi})
        &= -\frac{N}{2} \ln|\mathbf{\Sigma}| - \frac{1}{2}\sum_{n=1}^{N} \sum_{k=1}^{K}
            t_{nk} (\bm{\phi}_n - \bm{\mu}_k)^T\mathbf{\Sigma}^{-1}
            (\bm{\phi}_n - \bm{\mu}_k) + \text{const} \\
        &= -\frac{N}{2} \ln|\mathbf{\Sigma}| - \frac{1}{2}\sum_{n=1}^{N} \sum_{k=1}^{K}
            t_{nk} \big(\bm{\phi}_n^T\mathbf{\Sigma}^{-1}\bm{\phi}_n
            - 2\bm{\phi}_n^T\mathbf{\Sigma}^{-1}\bm{\mu}_k
            + \bm{\mu}_k^T\mathbf{\Sigma}^{-1}\bm{\mu}_k\big)
    \end{align*}
    By using (C.28) and
    \[
    \pdv{\mathbf{X}} \mathbf{a}^T\mathbf{X}^{-1}\mathbf{b}
    = -\mathbf{X}^{-T} \mathbf{ab}^T \mathbf{X}^{-T}
    \]
    we take the derivative of the log likelihood with respect to $\mathbf{\Sigma}$
    and obtain:
    \begin{align*}
        \pdv{\mathbf{\Sigma}} \ln p(\mathbf{T} | \mathbf{\Phi}, \bm{\pi})
        &= -\frac{N}{2} \mathbf{\Sigma}^{-1}
        + \frac{1}{2} \sum_{n=1}^{N} \sum_{j=k}^{K} t_{nk} \mathbf{\Sigma}^{-1}
        \big(\bm{\phi}_n\bm{\phi}_n^T - 2\bm{\phi}_n\bm{\mu}_k^T + \bm{\mu}_k\bm{\mu}_k^T\big)
        \mathbf{\Sigma}^{-1} \\
        &= -\frac{N}{2} \mathbf{\Sigma}^{-1}
        + \frac{1}{2} \sum_{n=1}^{N} \sum_{k=1}^{K} t_{nk} \mathbf{\Sigma}^{-1}
        (\bm{\phi}_n - \bm{\mu}_k)(\bm{\phi}_n - \bm{\mu}_k)^T \mathbf{\Sigma}^{-1} \\
        &= -\frac{N}{2} \mathbf{\Sigma}^{-1}
        + \frac{1}{2} \mathbf{\Sigma}^{-1}
        \bigg(\sum_{k=1}^{K} N_k\mathbf{S}_k\bigg) \mathbf{\Sigma}^{-1}
    \end{align*}
    where $\mathbf{S}_k$ is defined by $(\ref{eq:4.163})$. Therefore, by setting
    this derivative to 0 and rearranging the terms,
    we obtain the maximum-likelihood solution for the
    shared covariance matrix
    \[
        \mathbf{\Sigma} = \sum_{k=1}^{K} \frac{N_k}{N} \mathbf{S}_k \tag{4.162}
    \]
\end{proof}

\section*{Exercise 4.11 $\star \star$}
Consider a classification problem with $K$ classses for which the feature
vector $\mathbf{\phi}$ has $M$ components each of which can take $L$ discrete
states. Let the values of the components be represented by a 1-of-$L$ binary coding scheme.
Further suppose that, conditioned on the class $\mathcal{C}_k$, the $M$ components
of $\bm{\phi}$ are independent, so that the class-conditional density factorizes
with respect to the feature vector components. Show that the quantities given by
$(\ref{eq:4.63})$, which appear in the argument to the softmax function
describing the posterior class probabilties, are linear functions of the components
of $\bm{\phi}$. Note that this represents an example of the naive Bayes model
which is discussed in Section 8.2.2.

\vspace{1em}

\begin{proof}
    We've seen in Section 4.2 that the posterior probabilities
    can be written as $\emph{normalized}$ $\emph{exponentials}$:
    \begin{equation}\label{eq:4.62}\tag{4.62}
        p(\mathcal{C}_k | \bm{\phi})
        = \frac{\exp(a_k)}{\sum_{j} \exp(a_j)}
    \end{equation}
    where
    \begin{equation}\label{eq:4.63}\tag{4.63}
        a_k = \ln p(\bm{\phi} | \mathcal{C}_k)p(\mathcal{C}_k)
    \end{equation}
    Considering the setup of our classification problem,
    our class-conditional distribution will be of the form
    \[
        p(\bm{\phi} | \mathcal{C}_k)
        = \prod_{i = 1}^M \prod_{j = 1}^L
            \bm{\mu}_{kij}^{\bm{\phi}_{ij}}
    \]
    where $\bm{\mu}_k$ is given by $(\ref{eq:4.161})$.
    Therefore,
    by replacing into $(\ref{eq:4.63})$, the arguments
    of the softmax function are given by
    \[
        a_k = \ln p(\mathcal{C}_k)
        + \sum_{i=1}^{M} \sum_{j=1}^{L} \bm{\phi}_{ij} \ln \bm{\mu}_{kij}
    \]
    and are obviously linear functions of the components
    of $\bm{\phi}$.
\end{proof}

\section*{Exercise 4.12 $\star$}
Verify the relation ($\ref{eq:4.88}$) for the derivative of the logistic sigmoid
function defined by ($\ref{eq:4.59}$).

\vspace{1em}

\begin{proof}
    By taking the derivative of $(\ref{eq:4.59})$, we have that:
    \[
        \pdv{a} \sigma(a)
        = \pdv{a}\bigg(\frac{1}{1 + e^{-a}}\bigg)
        = \frac{e^{-a}}{(1 + e^{-a})^2}
        = \frac{1 + e^{-a}}{(1 + e^{-a})^2} - \frac{1}{(1 + e^{-a})^2}
        = \frac{1}{1 + e^{-a}} - \bigg(\frac{1}{1 + e^{-a}}\bigg)^2
    \]
    We recognize the expression of the logistic sigmoid function, so
    \begin{equation}\label{eq:4.88}\tag{4.88}
        \pdv{a} \sigma(a) = \sigma(a) - \sigma(a)^2 = \sigma(a) \big(1 - \sigma(a)\big)
    \end{equation}
\end{proof}

\section*{Exercise 4.13 $\star$}
By making use of the result ($\ref{eq:4.88}$) for the derivative of the logistic
sigmoid, show that the derivative of the error function ($\ref{eq:4.90}$) for
the logistic regression model is given by (\ref{eq:4.91}).

\vspace{1em}

\begin{proof}
    The error function for the logistic regression is given by
    \begin{equation}\label{eq:4.90}\tag{4.90}
        E(\mathbf{w})
        = -\ln p(\mathbf{t} | \mathbf{w})
        = -\sum_{n = 1}^{N} \{t_n \ln y_n + (1 - t_n) \ln(1 - y_n)\}
    \end{equation}
    where $y_n = \sigma(a_n)$ and $a_n = \mathbf{w}^T\bm{\phi}_n$.
    Taking the derivative of the log likelihood function with respect to $\mathbf{w}$
    gives
    \begin{align*}
        \nabla_{\mathbf{w}} \ln p(\mathbf{t} | \mathbf{w})
        &= \nabla_{\mathbf{w}} \sum_{n=1}^{N} \big\{
            t_n \ln y_n
            + (1 - t_n) \ln (1 - y_n)
            \big\} \\
        &= \sum_{n=1}^{N} \bigg[t_n \nabla_{\mathbf{w}} \ln y_n
            + (1 - t_n) \nabla_{\mathbf{w}} \ln (1 - y_n)\bigg] \\
        &= \sum_{n=1}^{N} \bigg[\frac{t_n}{y_n} \nabla_{\mathbf{w}} y_n
            + \frac{1 - t_n}{1 - y_n} \nabla_{\mathbf{w}} (1 - y_n) \bigg]\\
        &= \sum_{n=1}^{N} \frac{t_n(1 - y_n) - y_n(1 - t_n)}{y_n(1 - y_n)}
            \nabla_\mathbf{w} y_n \\
        &= \sum_{n=1}^{N} \frac{t_n - y_n}{y_n(1 - y_n)}
            \nabla_\mathbf{w} y_n \label{eq:4.13.1}\tag{4.13.1}
    \end{align*}
    Using ($\ref{eq:4.88}$), we can compute the gradient term:
    \begin{align*}
        \nabla_{\mathbf{w}} y_n
        = \nabla_\mathbf{w} \sigma(\mathbf{w}^T\bm{\phi}_n)
        = \pdv{\sigma}{a_n} \pdv{a_n}{\mathbf{w}} \big(\mathbf{w}^T\bm{\phi}_n\big)
        = y_n(1 - y_n) \bm{\phi}_n
    \end{align*}
    so the gradient of the log likelihood is
    \[
        \nabla_{\mathbf{w}} \ln p(\mathbf{t} | \mathbf{w})
        = \sum_{n=1}^{N} (t_n - y_n) \bm{\phi}_n
    \]
    and the gradient of the error function is then given by
    \begin{equation}\label{eq:4.91}\tag{4.91}
        \nabla_\mathbf{w} E(\mathbf{w})
        = -\nabla_\mathbf{w} \ln p(\mathbf{t} | \mathbf{w})
        = - \sum_{n=1}^{N} (t_n - y_n) \bm{\phi}_n
    \end{equation}
\end{proof}

\section*{Exercise 4.14 $\star$}
Show that for a linearly separable data set, the maximum likelihood
solution for the logistic regression model is obtained by finding a vector
$\mathbf{w}$ whose decision boundary $\mathbf{w}^T\bm{\phi}(\mathbf{x}) = 0$
separates the classes and then taking the magnitude of $\mathbf{w}$ to infinity.

\vspace{1em}

\begin{proof}
    Suppose there exists $\mathbf{w}$ such that the hyperplane
    $\mathbf{w}^T\bm{\phi} = 0$ separates the data set.
    Because the data set is linearly separable,
    $\mathbf{w}^T\bm{\phi}_a < 0$ and $\mathbf{w}^T\bm{\phi}_b > 0$
    for all $\bm{\phi}_a \in \mathcal{C}_1$ and $\bm{\phi}_b \in \mathcal{C}_2$.
    One can observe that the maximum likelihood is obtained
    when  $p(\mathcal{C}_1 | \bm{\phi}_a) = 1$ and $p(\mathcal{C}_2 | \bm{\phi}_b) = 1$
    for all $\bm{\phi}_a \in \mathcal{C}_1, \bm{\phi}_b \in \mathcal{C}_2$.
    Since our hyperplane is already chosen, there is a fixed angle $\theta_n$
    between each $\bm{\phi}_n$ and $\mathbf{w}$ such that $\cos \theta_n \neq 0$. Therefore,
    by using the geometric definition of the dot product
    \[
        \mathbf{w}^T\bm{\phi}_n
        = \Vert\mathbf{w}\Vert \Vert\bm{\phi}_n\Vert \cos \theta_n
    \]
    we see that our maximization is achieved by taking the magnitude
    of $\Vert\mathbf{w}\Vert$ to infinity, as
    \[
        \lim_{\Vert\mathbf{w}\Vert \to \infty} p(\mathcal{C}_1 | \bm{\phi}_a)
        = \lim_{\Vert\mathbf{w}\Vert \to \infty}
        \sigma(\Vert\mathbf{w}\Vert \Vert{\bm{\phi}_a}\Vert \cos \theta_a)
        = \lim_{\Vert\mathbf{w}\Vert \to \infty}
        \frac{1}{1 + \exp{-\Vert\mathbf{w}\Vert \Vert{\bm{\phi}_a}\Vert \cos \theta_a}}
        = 1
    \]
    and
    \[
        \lim_{\Vert\mathbf{w}\Vert \to \infty} p(\mathcal{C}_2 | \bm{\phi}_b)
        = \lim_{\Vert\mathbf{w}\Vert \to \infty}
        \sigma(\Vert\mathbf{w}\Vert \Vert{\bm{\phi}_b}\Vert \cos \theta_b)
        = \lim_{\Vert\mathbf{w}\Vert \to \infty}
        \frac{1}{1 + \exp{-\Vert\mathbf{w}\Vert \Vert{\bm{\phi}_b}\Vert \cos \theta_b}}
        = 0
    \]
    where $\bm{\phi} \in \mathcal{C}_1, \bm{\phi}_b \in \mathcal{C}_2$ and we've used the
    fact that $\mathbf{w}^T\bm{\phi}_a < 0$ and $ \mathbf{w}^T\bm{\phi}_b > 0$.
\end{proof}

\section*{Exercise 4.15 $\star \star$}
Show that the Hessian matrix $\mathbf{H}$ for the logistic regression
model, given by $(\ref{eq:4.97})$, is positive definite. Here $\mathbf{R}$
is a diagonal matrix with elements $y_n(1 - y_n)$, and $y_n$ is
the output of the logistic regression model for input vector $\mathbf{x}_n$.
Hence show that the error function is a convex function of $\mathbf{w}$
and it has an unique minimum.

\vspace{1em}

\begin{proof}
    The Hessian of the error function is given by
    \begin{equation}\label{eq:4.97}\tag{4.97}
        \mathbf{H}
        = \nabla \nabla E(\mathbf{w})
        = \sum_{n=1}^{N} y_n(1 - y_n)\bm{\phi}_n\bm{\phi}_n^T
        = \mathbf{\Phi}^T\mathbf{R}\mathbf{\Phi}
    \end{equation}
    Let $\mathbf{u}$ be a $M$-dimensional column vector.
    By using the sum formulation for the hessian matrix, we have that
    \[
        \mathbf{u}^T\mathbf{H}\mathbf{u}
        = \sum_{n=1}^{N} y_n(1 - y_n)\mathbf{u}^T\bm{\phi}_n\bm{\phi}_n^T\mathbf{u}
        = \sum_{n=1}^{N} y_n(1 - y_n) \big(\bm{\phi}_n^T\mathbf{u}\big)^T \bm{\phi}_n^T\mathbf{u}
        = \sum_{n=1}^{N} y_n(1 - y_n) \Vert\bm{\phi}_n^T \mathbf{u}\Vert^2
    \]
    which is $>0$ since $y_n$ is the output of the logistic sigmoid function,
    so $0 < y_n < 1$. Because $\mathbf{u}$ was chosen arbitrarily, we have that
    $\mathbf{H}$ is positive definite. As a result, the error function is convex and has an unique
    minimum.
\end{proof}

\section*{Exercise 4.16 $\star$}
Consider a binary classification problem in which each observation
$\mathbf{x}_n$ is known to belong to one of two classes, corresponding to $t=0$
and $t=1$, and suppose that the procedure for collecting training data
is imperfect, so that training points are sometimes mislabelled. For
every data point $\mathbf{x}_n$, instead of having a value $t$ for
the class label, we have instead a value $\pi_n$ representing the
probability that $t_n = 1$. Given a probabilistic model $p(t = 1 | \bm{\phi})$,
write down the log likelihood function appropiate for such a data set.

\vspace{1em}

\begin{proof}
    Straight away, we can see that $p(t = 0 | \bm{\phi}) = 1 - p(t = 1 | \bm{\phi})$.
    An fair approach would be to express $p(t_n | \bm{\phi})$ as a weighted
    average of $p(t_n = 0 | \bm{\phi})$ and $p(t_n = 1 | \bm{\phi}$ dictated by $\pi_n$.
    Therefore, the likelihood would be given by
    \[
        p(\mathbf{t} | \bm{\phi})
        = \prod_{n = 1}^N p(t_n | \bm{\phi})
        = \prod_{n = 1}^N p(t_n = 1 | \bm{\phi})^{\pi_n}
        p(t_n = 0 | \bm{\phi})^{1 - \pi_n}
        = \prod_{n = 1}^N p(t_n = 1 | \bm{\phi})^{\pi_n}
        \big\{1 - p(t_n = 1 | \bm{\phi})\big\}^{1 - \pi_n}
    \]
    which has the log likelihood
    \[
        \ln p(\mathbf{t} | \bm{\phi})
        = \sum_{n=1}^{N} \pi_n p(t_n = 1 | \bm{\phi})
        + (1 - \pi_n)\big\{1 - p(t_n = 1 | \bm{\phi})\big\}
    \]
\end{proof}

\section*{Exercise 4.17 $\star$}
Show that the derivatives of the softmax activation function
$(\ref{eq:4.104})$ where the $a_k$ are defined by $(\ref{eq:4.105})$, are
given by $(\ref{eq:4.106})$.

\vspace{1em}

\begin{proof}
    The softmax activation function is given by
    \begin{equation}\label{eq:4.104}\tag{4.104}
        y_k = \frac{\exp(a_k)}{\sum_{j} \exp(a_j)}
    \end{equation}
    where
    \begin{equation}\label{eq:4.105}\tag{4.106}
        a_k = \mathbf{w}_k^T\bm{\phi}
    \end{equation}
    Taking the derivative of $(\ref{eq:4.104})$ with respect to
    $a_j$ and applying the quotient rule gives
    \begin{align*}
        \pdv{y_k}{a_j}
        = \pdv{a_j} \bigg(\frac{\exp(a_k)}{\sum_{i} \exp(a_i)}\bigg)
        &= \frac{I_{kj} \exp(a_k) \sum_{i} \exp(a_i) - \exp(a_k)\exp(a_j)}
        {\bigg(\sum_{i} \exp(a_i)\bigg)^2} \\
        &= \frac{\exp(a_k)}{\sum_{j} \exp(a_j)}
        \bigg(I_{kj} - \frac{\exp(a_j)}{\sum_{j} \exp(a_j)}\bigg)
    \end{align*}
    which is equivalent to
    \begin{equation}\label{eq:4.106}\tag{4.106}
        \pdv{a_j} y_k = y_k(I_{kj} - y_j)
    \end{equation}
\end{proof}

\section*{Exercise 4.18 $\star$}
Using the result $(\ref{eq:4.106})$ for
the derivatives of the softmax activation function,
show that the gradients of the cross-entropy error $(\ref{eq:4.108})$
are given by ($\ref{eq:4.109}$).

\vspace{1em}

\begin{proof}
    The cross-entropy error is given by
    \begin{equation}\label{eq:4.108}\tag{4.108}
        E(\mathbf{w}_1, \ldots, \mathbf{w}_K)
        = -\ln p(\mathbf{T} | \mathbf{w}_1, \ldots, \mathbf{w}_K)
        = - \sum_{n=1}^{N} \sum_{k=1}^{K} t_{nk} \ln y_{nk}
    \end{equation}
    Taking its derivative with respect to $\mathbf{w}_j$ yields
    \begin{align*}
        \pdv{\mathbf{w}_j} E(\mathbf{w}_1, \ldots, \mathbf{w}_K)
        = \pdv{\mathbf{w}_j} \bigg(-\sum_{n=1}^{K} \sum_{k=1}^{K} t_{nk} \ln y_{nk}\bigg)
        = -\sum_{n=1}^{N} \sum_{k=1}^{K} t_{nk} \pdv{\mathbf{w}_j} \ln y_{nk}
    \end{align*}
    By using $(\ref{eq:4.106})$ and the chain rule, we have that
    \begin{align*}
        \pdv{\mathbf{w}_j} \ln y_{nk}
        = \frac{1}{y_{nk}} \pdv{y_{nk}}{\mathbf{w}_j}
        = \frac{1}{y_{nk}} \pdv{y_{nk}}{a_j} \pdv{a_j}{\mathbf{w}_j}
        = \frac{1}{y_{nk}} y_{nk}(I_{kj} - y_{nj}) \bm{\phi}_n
        = (I_{kj} - y_{nj})\bm{\phi}_n
    \end{align*}
    Replacing back into the gradient,
    \[
        \pdv{\mathbf{w}_j} E(\mathbf{w}_1, \ldots, \mathbf{w}_K)
        = -\sum_{n=1}^{N} \sum_{k=1}^{K} t_{nk} (I_{kj} - y_{nj}) \bm{\phi}_n
        = -\sum_{n=1}^{N} t_{nj}\bm{\phi}_n
        + \sum_{n=1}^{N} \bigg(\sum_{k=1}^{N} t_{nk}\bigg) y_{nj} \bm{\phi}_n
    \]
    gives the desired result
    \begin{equation}\label{eq:4.109}\tag{4.109}
        \pdv{\mathbf{w}_j}E(\mathbf{w}_1, \ldots, \mathbf{w}_K)
        = \sum_{n=1}^{N} (y_{nj} - t_{nj})\bm{\phi}_n
    \end{equation}
\end{proof}

\section*{Exercise 4.19 $\star$}
Write down expressions for the gradient of the log likelihood,
as well as the corresponding Hessian matrix, for the probit regression
model defined in Section 4.3.5. These are quantities that would be required to train such
a model using IRLS.

\vspace{1em}

\begin{proof}
    The probit function is given by
    \begin{equation}\label{eq:4.114}\tag{4.114}
        \Phi(a) = \int_{-\infty}^{a} \mathcal{N}(\theta | 0, 1) \diff \theta
    \end{equation}
    Therefore, from the fundamental theorem of calculus, we have that
    \[
        \pdv{a} \Phi(a)
        = \pdv{a} \int_{-\infty}^a \mathcal{N}(\theta | 0, 1) \diff \theta
        = \mathcal{N}(a | 0, 1) = \frac{1}{\sqrt{2\pi}} e^{-\frac{a^2}{2}}
    \]
    so
    \[
        \nabla_\mathbf{w} y_n
        = \nabla_\mathbf{w} \Phi(a_n)
        = \pdv{\Phi}{a_n} \pdv{a_n}{\mathbf{w}}
        = \frac{1}{\sqrt{2\pi}} \exp{-\frac{a_n^2}{2}}\bm{\phi}_n
    \]

    The probit regression model has a very similar form with what
    we've used for the logistic regression model in Section 4.3.2.
    More specifically, the log likelihood is still given by ($\ref{eq:4.90}$),
    but this time with $y_n = \Phi(a_n)$. Therefore, the general form for gradient of the log
    likelihood derived in Exercise 4.13, (\ref{eq:4.13.1}) can be used
    here too, so:
    \[
        \nabla_\mathbf{w} \ln p(\mathbf{t} | \mathbf{w})
        = \sum_{n=1}^{N} \frac{t_n - y_n}{y_n(1 - y_n)} \nabla_\mathbf{w} y_n
        = \frac{1}{\sqrt{2\pi}} \sum_{n=1}^{N} \frac{t_n - y_n}{y_n(1 - y_n)}
        \exp{-\frac{a_n^2}{2}}\bm{\phi}_n
    \]
    By taking the gradient of this again, we find the Hessian matrix
    using the chain rule:
    \begin{align*}
        \mathbf{H}
        = \nabla_\mathbf{w} \nabla_\mathbf{w} \ln p(\mathbf{t} | \mathbf{w})
        &= \frac{1}{\sqrt{2\pi}} \sum_{n=1}^{N}
        \nabla_\mathbf{w}\bigg[\frac{t_n - y_n}{y_n(1 - y_n)}\exp{-\frac{a_n^2}{2}}\bigg]\bm{\phi}_n \\
        &= \frac{1}{\sqrt{2\pi}} \sum_{n=1}^{N} \bigg[
            \bigg(\nabla_\mathbf{w} \frac{t_n - y_n}{y_n(1 - y_n)}\bigg)
            \exp{-\frac{a_n^2}{2}} + \frac{t_n - y_n}{y_n(1 - y_n)}
        \bigg(\nabla_\mathbf{w} \exp{-\frac{a_n^2}{2}}\bigg)\bigg]\bm{\phi}_n
    \end{align*}
    We compute each gradient term separately, so
    \begin{align*}
        \nabla_\mathbf{w} \frac{t_n - y_n}{y_n(1 - y_n)}
        &= -\frac{y_n(1 - y_n) + (t_n - y_n)(1 - 2y_n)}
        {y_n^2(1 - y_n)^2} \nabla_\mathbf{w} y_n
        = \frac{y_n^2 - 2t_ny_n + t_n}{y_n^2(1 - y_n)^2}\nabla_\mathbf{w} y_n \\
        &= \frac{y_n^2 - 2t_ny_n + t_n}{y_n^2(1 - y_n)^2\sqrt{2\pi}}\exp{-\frac{a_n^2}{2}}
        \bm{\phi}_n
    \end{align*}
    and
    \[
        \nabla_\mathbf{w} \exp{-\frac{a_n^2}{2}}
        = -a_n\exp{-\frac{a_n^2}{2}} \nabla_\mathbf{w} a_n
        = -a_n\exp{-\frac{a_n^2}{2}} \bm{\phi}_n
    \]
    Hence, the hessian matrix becomes
    \begin{align*}
        \mathbf{H}
        &= \frac{1}{\sqrt{2\pi}} \sum_{n=1}^{N} \bigg[
        \frac{y_n^2 - 2t_ny_n + t_n}{y_n^2(1 - y_n)^2\sqrt{2\pi}}\exp{-a_n^2}\bm{\phi}_n
        - \frac{a_n(t_n - y_n)}{y_n(1 - y_n)}\exp{-\frac{a_n^2}{2}}\bm{\phi}_n\bigg] \bm{\phi}_n
    \end{align*}
\end{proof}

\section*{Exercise 4.21 $\star$}
Show that the probit function ($\ref{eq:4.114}$) and the erf function $(\ref{eq:4.115})$ are
related by ($\ref{eq:4.116}$).

\vspace{1em}

\begin{proof}
    The error function is given by
    \begin{equation}\label{eq:4.115}\tag{4.115}
        \erf(a) = \frac{2}{\sqrt{\pi}} \int_{0}^{a} \exp{-\frac{\theta^2}{2}} \diff \theta
    \end{equation}
    By using the fact that the Gaussian is symmetric around the mean,
    the probit function can be rewritten as
    \begin{align*}
        \Phi(a)
        = \int_{-\infty}^{a} \mathcal{N}(\theta | 0, 1) \diff \theta
        = \int_{-\infty}^{0} \mathcal{N}(\theta | 0, 1) \diff \theta
        + \int_{0}^{a} \mathcal{N}(\theta | 0, 1) \diff \theta
        &= \frac{1}{2} + \frac{1}{\sqrt{2\pi}} \int_{0}^{a} \exp{-\frac{\theta^2}{2}} \diff \theta \\
        &= \frac{1}{2} \bigg\{1 + \frac{1}{\sqrt{2}} \erf(a)\bigg\} \label{eq:4.116}\tag{4.116}
    \end{align*}
\end{proof}

\section*{Exercise 4.22 $\star$}
Using the result (4.135), derive the expression
$(\ref{eq:4.137})$ for the log model evidence under the Laplace
approximation.

\vspace{1em}

\begin{proof}
    The proof of this is almost identical to the one in Section 4.4.1.
    From Bayes' theorem the model evidence is given by
    \begin{equation}\label{eq:4.136}\tag{4.136}
        p(\mathcal{D})
        = \int p(\mathcal{D} | \bm{\theta}) p(\bm{\theta}) \diff \bm{\theta}
    \end{equation}
    Identifying $f(\bm{\theta}) = p(\mathcal{D} | \bm{\theta})p(\bm{\theta})$
    and $Z = p(\mathcal{D})$, and applying the result (4.135),
    we obtain the model evidence under the Laplace approximation:
    \[
        p(\mathcal{D})
        = p(\mathcal{D} | \bm{\theta}_{MAP})p(\bm{\theta}_{MAP})
        \frac{(2\pi)^{M/2}}{|\mathbf{A}|^{1/2}}
    \]
    where $\bm{\theta}_{MAP}$ is the value of $\bm{\theta}$ at the mode
    of the posterior distribution, and $\mathbf{A}$ is the $\emph{Hessian}$
    matrix of second derivatives of the negative log posterior
    \begin{equation}\label{eq:4.138}\tag{4.138}
        \mathbf{A}
        = -\nabla \nabla \ln p(\mathcal{D} | \bm{\theta}_{MAP})p(\bm{\theta}_{MAP})
        = -\nabla \nabla \ln p(\bm{\theta}_{MAP} | \mathcal{D})
    \end{equation}

    Therefore, the log model evidence is given by
    \begin{equation}\label{eq:4.137}\tag{4.137}
        \ln p(\mathcal{D})
        = \ln p(\mathcal{D} | \bm{\theta}_{MAP}) + \ln p(\bm{\theta}_{MAP})
        + \frac{M}{2} \ln (2\pi) - \frac{1}{2} \ln |\mathbf{A}|
    \end{equation}
\end{proof}

\section*{Exercise 4.25 $\star \star$}
Suppose we wish to approximate the logistic sigmoid $\sigma(a)$
defined by $(\ref{eq:4.59})$ by a scaled probit function
$\Phi(\lambda a)$ where $\Phi(a)$ is defined by $(\ref{eq:4.114})$.
Show that if $\lambda$ is chosen so that the derivatives of the two
functions are equal at $a = 0$, then  $\lambda^2 = \pi/8$.

\vspace{1em}

\begin{proof}
    We start by evaluating both function's derivatives at $a=0$.
    We've seen in Exercise 4.19 that the derivative of the probit function
    is given by
     \[
         \pdv{a} \Phi(a) = \frac{1}{\sqrt{2\pi}} \exp{-\frac{a^2}{2}}
    \]
    so
    \[
        \pdv{a} \Phi(\lambda a)\bigg|_{a=0}
        = \frac{\lambda}{\sqrt{2\pi}} \exp{-\frac{a^2}{2}}\bigg|_{a=0}
        = \frac{\lambda}{\sqrt{2\pi}}
    \]
    From $(\ref{eq:4.88})$ we also obtain the derivative of the sigmoid function:
     \[
         \pdv{a} \sigma(a)\bigg|_{a=0} = \sigma(a)(1 - \sigma(a))\bigg|_{a = 0}
         = \frac{1}{4}
    \]
    Finally, by using the fact that the derivatives of the functions are
    equal at $a=0$, we quickly reach the result from the hypothesis,
    i.e. $\lambda^2 = \pi / 8$.
\end{proof}


\documentclass[../../main.tex]{subfiles}

\begin{document}

\subsection{Motivation}

In this work a tensor approximation problem is investigated. Low-rank decomposition of 2 and 3-dimensional tensors is performed using Tucker HOSVD and CPD algorithms. A gray scale image, an RGB image and a gray scale animation film are used as sample data. Relative error is measured in all cases.

\subsection{Theoretical background}

\subsubsection{Tucker, HOSVD}

\begin{figure}[h!]
\centering
\includegraphics[width=0.6\textwidth]{figures/tucker}
\caption{Tucker decomposition}
\label{fig:Lab1:1}
\end{figure}

Consider a 3-rd order tensor $\mathbf{\underline{X}} \in \mathbb{R}^{I \times J \times K}$. We define its Tucker Decomposition as

\begin{equation}
\begin{aligned}
\mathbf{\underline{X}} & \simeq \sum_{q=1}^{Q} \sum_{r=1}^{R} \sum_{p=1}^{P} g_{qrp} \mathbf{a}_q \circ \mathbf{b}_r \circ \mathbf{c}_p\\
& = \mathbf{\underline{G}} \times_1 \mathbf{A} \times_2 \mathbf{B} \times_3 \mathbf{C}\\
& = \Big[    \mathbf{\underline{G}} ;  \mathbf{A},  \mathbf{B}, \mathbf{C}      \Big]
\end{aligned}
\end{equation}

where $\mathbf{\underline{G}}$ is a dense core tensor with multi-linear rank $(Q,R,P)$ and $\mathbf{A} \in \mathbb{R}^{I \times Q}, \mathbf{B} \in \mathbb{R}^{J \times R}$ are factor matrices.

Consider a 3-rd order tensor $\mathbf{\underline{X}} \in \mathbb{R}^{I \times J \times K}$, decomposed in the Tucker format as

$$
\mathbf{\underline{X}} = \mathbf{\underline{G}} \times_1 \mathbf{A} \times_2 \mathbf{B} \times_3 \mathbf{C}
$$

The \textbf{HOSVD} is a special case of the Tucker decomposition, in which all the factor matrices are constrained to be orthogonal. They are computed as truncated version of the left singular matrices of all possible mode- unfoldings of tensor $\mathbf{\underline{X}}$:

\begin{equation}
\begin{aligned}
\mathbf{X}_{(1)} &= \mathbf{U}_1  \mathbf{\Sigma}_1 \mathbf{V}_1^T \quad \rightarrow \quad \mathbf{A} = \mathbf{U}_1[1:R_1]\\
\mathbf{X}_{(2)} &= \mathbf{U}_2  \mathbf{\Sigma}_2 \mathbf{V}_2^T \quad \rightarrow \quad \mathbf{B} = \mathbf{U}_2[1:R_2] \\
\mathbf{X}_{(3)} &= \mathbf{U}_3  \mathbf{\Sigma}_3 \mathbf{V}_3^T \quad \rightarrow \quad \mathbf{C} = \mathbf{U}_3[1:R_3] \\
\end{aligned}
\end{equation}

\break

\subsubsection{CPD}

\begin{figure}[h!]
\centering
\includegraphics[width=0.6\textwidth]{figures/cpd_decomp}
\caption{CPD decomposition}
\label{fig:Lab1:2}
\end{figure}

Consider a 3-rd order tensor $\mathbf{\underline{X}} \in \mathbb{R}^{I \times J \times K}$. We define its CPD Decomposition as

\begin{equation}
\begin{aligned}
\mathbf{\underline{X}} & \simeq \sum_{r=1}^{R} \lambda_r \mathbf{a}_r \circ \mathbf{b}_r \circ \mathbf{c}_r\\
& = \mathbf{\underline{\Lambda}} \times_1 \mathbf{A} \times_2 \mathbf{B} \times_3 \mathbf{C}\\
& = \Big[    \mathbf{\underline{\Lambda}} ;  \mathbf{A},  \mathbf{B}, \mathbf{C}      \Big]
\end{aligned}
\end{equation}

where $\mathbf{\underline{\Lambda}}$ is a 3-rd order core tensor having $\lambda_r$ as entries at positions $\mathbf{\underline{\Lambda}}[i, j, k]$ where $\mathbf{\underline{\Lambda}}[i, j, k]$ and zeros elsewhere; $\mathbf{A}, \mathbf{B}, \mathbf{C}$ are factor matrices obtained as the concatenation of the corresponding factor vectors, i.e $\mathbf{A} = \Big[    \mathbf{a}_1 \mathbf{a}_2  \cdots \mathbf{a}_R   \Big]$.

\subsection{Experiments}

\textbf{NB:} Jupyter Notebook with the code can be found at \url{https://github.com/evgeniishch/mm_forecasting}.

As an experiment both HOSVD and CPD decompositions are applied to 3-dimensional tensors.

\subsection{RGB image}

RGB image is formally a 3-dimensional tensor, however not completely honest as its RBG features are unordered and can be viewed as categorical. Yet it is good to start with.

A $100 \times 100 \times 3$ RGB image tensor is decomposed via HOSVD and CPD algorithms:

\begin{figure}[h!]
\centering
\includegraphics[width=0.6\textwidth]{figures/tucker_img}
\caption{Comparison of original image and image reconstructed from HOSVD decomposition with rank 5}
\label{fig:Lab1:3}
\end{figure}

\begin{figure}[h!]
\centering
\includegraphics[width=0.6\textwidth]{figures/cpd_img}
\caption{Comparison of original image and image reconstructed from CPD decomposition with rank 5}
\label{fig:Lab1:4}
\end{figure}

For each decomposed tensor a residual tensor is computed and relative error of approximation is calculated w.r.t. decomposition rank:

\begin{figure}[h!]
\centering
\includegraphics[width=0.6\textwidth]{figures/HOSVD_err}
\caption{Relative error of HOSVD decomposition w.r.t. decomposition rank}
\label{fig:Lab1:5}
\end{figure}

\begin{figure}[h!]
\centering
\includegraphics[width=0.6\textwidth]{figures/CPD_err}
\caption{Relative error of CPD decomposition w.r.t. decomposition rank}
\label{fig:Lab1:6}
\end{figure}

For entertainment purposes, a picture of a dog and a picture of a cake are decomposed with CPD algorithm. We offer the reader to distinguish between them at decomposition rank $2$:

\begin{figure}[h!]
\centering
\includegraphics[width=0.6\textwidth]{figures/cakedog}
\caption{Try to distinguish Chihua-hua and a cake}
\label{fig:Lab1:7}
\end{figure}

\subsubsection{Animation film}

Earlier, we considered images, which, although they are three-dimensional tensors, are of less interest for research, since the order is not defined on the features for one of the dimensions (RGB). Next, a short gray scale animation film is used as data for decomposition.

A $500 \times 333 \times 69$ frame sequence tensor is decomposed via CPD algorithm. A residual tensor is computed and relative error of approximation is calculated w.r.t. decomposition rank:

\begin{figure}[h!]
\centering
\includegraphics[width=0.6\textwidth]{figures/mickey_err}
\caption{Relative error of CPD decomposition w.r.t. decomposition rank}
\label{fig:Lab1:8}
\end{figure}

From visual observation a higher reconstruction quality is noticed for regions with high pixel contrast.

\end{document}


\chapter{Integer Division}

\section{The Division Algorithm}
\subsection{A Thought Experiment}

Suppose that you are playing a game of cards with your friends.  You have a deck of $n$ cards and there are $d$ players in the game.  In this particular game, the practice is to deal out an equal number of cards to each player and to deal out cards until it is no longer possible to do so.

Dealing cards is something most of us have done, but let's review the process.  When you deal cards, in order to make sure everyone gets the same number of cards, you do it in rounds.  On a given round, either you deal one card to each player or you don't deal to anyone.  This requires the dealer to compare the number of undealt to the number of players.

As all of the cards are to be dealt out, the prcess will continue until you are unable to deal.  This means you will come to a round where you cannot deal cards to every player.  Let $r$ be the number of cards that are left over.  It is possible that all of the cards have been dealt out, so that $r = 0$, but we know for certain that $r < d$ (recall $d$ is the number of players) because if $r \ge d$, we would be able to deal another round.

When we are done, each player will have the same number of cards.  Let's say this number is $q$.  Given this, we can now express $n$, the original number of cards, in the following way:
\[
n = d q + r
\]
where $0\le r < d$.

\subsection{The Division Algorithm}


\subsection{Efficient Trajectory Unrolling}
\label{text:approach/runtime/unrolling}
One of the most promising directions for decreasing the runtime of a trajectory optimization algorithm is to decrease the runtime of the objective and constraints evaluation as well as their gradients. As shown in sections \ref{text:approach/objective} and \ref{text:approach/constraint}, the majority of these optimization modules depend on the robot's trajectory $\x_{0:T}$, rather than on its control inputs $\u_{0:T}$. However, since control inputs are optimized, as described in Section \ref{text:approach/overview}, a computationally efficient transformation from controls and initial state to the trajectory is key for making the trajectory optimization real-time feasible.
\newline
Fortunately, we assumed that the robot follows double integrator dynamics, which are linear and markovian. Hence, they can be expressed as the following:

\begin{equation}
\x_{t+1} = A \x_t + B \u_t
\label{eq:dynamics}
\end{equation}

\begin{minipage}{0.5\textwidth}
$$A = \begin{bmatrix} 1 & 0 & \dt & 0 \\ 0 & 1 & 0 & \dt \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1\end{bmatrix}$$
\end{minipage}
\begin{minipage}{0.5\textwidth}
$$B = \begin{bmatrix} 0 & 0 & \dt & 0 \\ 0 & 0 & 0 & \dt \end{bmatrix}$$
\end{minipage}

Due to the linear (not linearized !) dynamics $\dt = \Delta t$ can be safely used. To fully "unroll" a trajectory, the linear dynamic equation \ref{eq:dynamics} would be applied iteratively for the length of the trajectory. Since the dynamics themselves are linear (not just a first-order approximation), this computation can be further batched and, in this way, speed up. In fact, the full trajectory can be derived merely based on the initial state $\x_0$ and the control input matrix $\u_{0:T}$, as shown in the following:

\begin{align}
\x_1 &= A \x_0 + B \ u_0 \\
\x_2 &= A \x_0 + B \ u_1 = A^2 \x_0 + A B \ u_0 + B \ u_1\\
\begin{bmatrix} \x_1 \\ \x_2 \\ \vdots \\ \x_n \end{bmatrix} &= \underbrace{\begin{bmatrix} A \\ A^2 \\ \vdots \\ A^n \end{bmatrix}}_{\substack{A_n}} \x_0 + \underbrace{\begin{bmatrix} B & 0 & \hdots & \hdots & 0 \\ AB & B & 0 & \hdots & 0 \\ \hdots & \hdots & \hdots & \hdots & \hdots \\ A^{n-1} B & A^{n-2} B & \hdots & \hdots & B \end{bmatrix}}_{\substack{B_n}} \begin{bmatrix} \u_0 \\ \u_1 \\ \vdots \\ \u_{n-1} \end{bmatrix}
\end{align}

In summary, we get the following (also linear) expression for computing the full robot trajectory at once. As demonstrated in \href{https://github.com/simon-schaefer/mantrap/blob/master/examples/timing.ipynb}{examples/tools/timing} using the fully batched formulation speeds up the trajectory "unrolling" by about a factor of 30.

\begin{equation}
\x_{1:n} = A_n \x_0 + B_n \u_{0:T-1}
\label{eq:dynamics_stacked}
\end{equation}


\documentclass[12pt]{article}

\usepackage[utf8]{inputenc}
\usepackage{latexsym,amsfonts,amssymb,amsthm,amsmath}

\setlength{\parindent}{0in}
\setlength{\oddsidemargin}{0in}
\setlength{\textwidth}{6.5in}
\setlength{\textheight}{8.8in}
\setlength{\topmargin}{0in}
\setlength{\headheight}{18pt}

\newcommand*{\QEDA}{\hfill\ensuremath{\blacksquare}}%
\newcommand*{\QEDB}{\hfill\ensuremath{\square}}%

\title{Advanced Algorithms - Homework 3}
\author{Kishlaya Jaiswal}

\begin{document}

\maketitle

\vspace{0.5in}

\subsection*{Exercise 1}

\begin{proof}
\textbf{i)} Given an instance of hitting set $(E, \mathcal{S})$ where $E$ is the ground set and $\mathcal{S}$ is a collection of subsets of $E$. We can reduce it to an instance of set cover as follows:

Let universe $\mathcal{U} = \mathcal{S}$ and subsets $\mathcal{S'} = E$ such that $S \in \mathcal{S'}$ contains all those elements $u \in \mathcal{U}$ where $S \in u$ in in the hitting set formulation.

Conversely, given $(\mathcal{U}, \mathcal{S'})$ an instance of set cover, reduce it to hitting set as follows:

Let ground set $E = \mathcal{S'}$ and subsets $\mathcal{S} = \mathcal{U}$ such that $S \in \mathcal{S}$ contains all those elements of $e \in E$ where $S \in e$ in set cover formulation.

Observe that these reductions are inverses of each other and hence there is a bijection between all instances of set cover and all instances of hitting set.

Now we show that the cost of a feasible solution remains preserved under these reductions.

Consider $(E, \mathcal{S})$ an instance of hitting set and let $F$ be feasible solution, so $F \cap S_i \neq \phi$ $\forall S_i \in \mathcal{S}$, then the set cover instance will be $(\mathcal{S}, E)$ and now $F$ corresponds to sets picked as $F \subseteq E$. $F$ covers the entire universe $\mathcal{S}$ as $F \cap S_i \neq \phi$.

Similarly, if $(\mathcal{U}, \mathcal{S'})$ is an instance of set cover with a feasible solution $\mathcal{T} \subseteq \mathcal{S'}$ such that $\bigcup_{T \in \mathcal{T}} T = \mathcal{U}$ then consider the equivalent hitting set instance $(\mathcal{S'}, \mathcal{U})$. Clearly, $T_i \cap \mathcal{U} \neq \phi$ as $\mathcal{T}$ covers the universe $\mathcal{U}$. Hence $\mathcal{T}$ is a feasible solution for hitting set.

Thus we have shown that there is a bijection between feasible solutions of same cost of of any equivalent instances of set cover and hitting set. Hence they are essentially the same problem.

Furthermore, from the cost preserving bijection we also get that $OPT(SC) = OPT(HS)$ for any two equivalent instances.

Since we have $\log n$ approximation for set cover we also get a $\log n$ approximation for hitting set.

Also, if know that size of each set $S \leq f$ in the hitting set instance then we also have a $f-$approximation.

\textbf{ii)} Let $U$ be the universe with $n$ elements. We pick an element from the universe with probability $p$ (to be decided later), that is we sample $F$ at random from the set of all possible subsets.

Let
$$Y_i =
    \begin{cases}
        1 & \text{if } S_i \text{ is satisfied} \\
        0 & \text{otherwise}
    \end{cases}
$$
Note that, $$Pr[Y_i = 1] = Pr[S_i \text{ is satisfied}] = Pr[|F \cup S_i| = 1] = |S_i|p(1-p)^{|S_i|-1}$$

Denote by $N = \sum Y_i$ number of satisfied sets. Then we have,

$$E(N) = \sum_{i=1}^m E(Y_i) = \sum_{i=1}^m Pr[Y_i = 1] = \sum_{i=1}^m |S_i|p(1-p)^{|S_i|-1}$$

Suppose the size of all sets is the same, say $k$. Set $p = \frac{1}{k}$ then $$E(N) = \sum_{i=1}^m k\left(\frac{1}{k}\right)\left(1-\frac{1}{k}\right)^{k-1} =  m\left(1-\frac{1}{k}\right)^{k-1}$$

\textbf{Case 1}: $k=1$. We set $p=1$ that is we pick all the elements. In this case, assuming the sets were all distinct, we have indeed achieved optimum as $E(N) = m = OPT$
\newline
\newline
\textbf{Case 2}: Since $\left(\frac{k}{k-1}\right)^{k-1} = \left(1+\frac{1}{k-1}\right)^{k-1} \leq e$ and Since, $OPT \leq m$,

we get $\boxed{E(N) \geq \frac{1}{e} OPT}$

\textbf{iii)} Say the universe $\mathcal{U}$ has $n$ elements. We consider the partition $(0,n] = (\frac{n}{2},n] \cup (\frac{n}{4},\frac{n}{2}] \cup (\frac{n}{8},\frac{n}{4}] \cup \ldots$. Now, size of each set $S_i$ lies in one of this partition. Since there are $\lg n$ many partitions, there exists one partition $(\frac{k}{2},k]$ such that atleast $\frac{m}{\lg n}$ many sets have size in $(\frac{k}{2},k]$.

We shall consider only these sets. Then repeating the experiment same as above, we get
$$E(N) \geq \sum_{i: |S_i| \in (k/2,k]} \frac{k}{2} \left(\frac{1}{k}\right)\left(1-\frac{1}{k}\right)^{k-1} \geq  \frac{m}{2\lg n} \left(1-\frac{1}{k}\right)^{k-1} \geq \frac{1}{2e\lg n} OPT$$

\textbf{iv)} We have the following LP formulation for the unique hitting set instance:
\begin{align*}
    \text{Maximize } \sum_{i=1}^m y_j \\
    \sum_{i \in S_j} x_i = y_j && \forall j \leq m \\
    x_i \in \{0,1\} && \forall i \leq n \\
    y_j \in \{0,1\} && \forall j \leq m \\
\end{align*}

The above LP has a optimal solution $OPT = m$ because it is given that given instance has the perfect hitting set property.

Relaxing the above LP, we get:

\begin{align*}
    \text{Maximize } \sum_{i=1}^m y_j \\
    \sum_{i \in S_j} x_i = y_j && \forall j \leq m \\
    0 \leq x_i \leq 1 && \forall i \leq n \\
    0 \leq y_j \leq 1 && \forall j \leq m \\
\end{align*}

Let $(x^*, y^*)$ be the optimal solution. Then we choose the $i^{th}$ element with probability $x_i^*$.

Let $q_k = \prod_{j \in S_k} (1- x_j^*)$, then probability that a set $S_k$ is satisfied is
$$\sum_{i \in S_k} x_i^* \prod_{\substack{j \in S_k \\ j \neq i}} (1-x_j^*) = \sum_{i \in S_k} \frac{x_i^*}{1 - x_i^*} q_k$$

Hence,
\begin{align*}
    \text{E(\# satisfied sets)} &= \sum_{k=1}^m \sum_{i \in S_k} \frac{x_i^*}{1 - x_i^*} q_k \\
    &= \sum_{k=1}^m \sum_{i \in S_k} \frac{q_k}{1 - x_i^*} x_i^* \\
    &\geq \sum_{k=1}^m \sum_{i \in S_k} \left(1-\frac{1}{e}\right) x_i^* \\
    &= \left(1-\frac{1}{e}\right) \sum_{k=1}^m \sum_{i \in S_k} x_i^* \\
    &= \left(1-\frac{1}{e}\right) \sum_{k=1}^m y_k^* \\
    &\geq \left(1-\frac{1}{e} \right) m \\
    &= \left(1-\frac{1}{e} \right) OPT
\end{align*}

\textbf{v)} Suppose all sets have size $2$. Let us call this problem as $2-$hitting set.

\textsl{Claim}: $2-$hitting set is as hard to approximate as vertex cover.

\textsl{Proof}: Given an instance $(V,E)$ of vertex cover. Consider it as an instance of $2-$hitting set with ground set $V$ and each $(u,v)$ edge as a subset.

Now clearly, if we have an $\alpha-$approximation for $2-$hitting set we get an $\alpha-$approximation for vertex cover. Furthermore, the above reduction is a poly-time reduction. Hence $2-$hitting set is also NP-hard. \QEDA

Suppose the sets satisfy $|S_i|=2$ $\forall S_i$ and it has the perfect hitting set property. Say the ground set is $V$ and subsets $S$. Then consider the graph $(V,S)$ where the vertices are the ground set and edges are $(u,v) \in S_i$ $\forall S_i \in S$.

Since it has the perfect hitting property, $\exists F \subseteq V$ such that $|F \cap S_i| = 1$

We claim that $(F, V \setminus F)$ is a bi-partition.

\textsl{Proof}: Suppose there is an edge $S_i$ between two elements of $F \implies |F \cap S_i| = 2$ contradiction to perfect hitting property .

Suppose there is an edge $S_i$ between two elements of $V \setminus F \implies |F \cap S_i| = \phi$ which is a contradiction. \QEDA

Hence the graph so obtained is a bipartite graph.

For bipartite graphs, we can find minimum vertex cover in poly-time (Konig-egervary theorem) hence we can find $F =$ minimum vertex cover, in poly-time.

Therefore, we can solve hitting set $(E, \{S_i\})$ instance with
\begin{itemize}
    \item $\forall S_i$ $|S_i| = 2$
    \item it has hitting property
\end{itemize}
in polynomial time.

\end{proof}

\subsection*{Exercise 2}
In class, we have seen a $\frac{1}{2}(\sqrt{5}-1)$-approximation algorithm for MAX SAT using biased coins and a randomized $(1-\frac{1}{e})$-approximation using randomized LP rounding. We show that choosing the better of the two solutions gives a $\frac{3}{4}$-approximation for MAX SAT.

\begin{proof}
Let $W_0$ denote the cost of the solution using biased coins randomized algorithm, $W_1$ denote the cost of the solution using unbiased coins randomized algorithm and $W_2$ denote the cost of the solution using randomized LP-rounding.

We recall that
$$E(W_0) \geq E(W_1)$$
$$E(W_1) \geq \sum_{j=1}^m w_j \left(1-\left(\frac{1}{2}\right)^{l_j}\right) $$
and
$$E(W_2) \geq \sum_{j=1}^m w_jy_j^* \left(1-\left(1-\frac{1}{l_j}\right)^{l_j}\right)$$
where $w_j$ is the weight of j-th clause and $y_j^*$ is optimal solution of relaxed LP.

Choosing better of the two solutions corresponds to $W = max(W_0, W_2)$, then we have $max(W_0, W_2) \geq \frac{1}{2}W_0+\frac{1}{2}W_2$. Thus,

\begin{align*}
    E(W) &\geq E\left(\frac{1}{2}W_0 + \frac{1}{2}W_2\right) \\
    &= \frac{1}{2}E(W_0) + \frac{1}{2}E(W_2) \\
    &\geq \frac{1}{2}E(W_1) + \frac{1}{2}E(W_2) \\
    &\geq \frac{1}{2}\left(\sum_{j=1}^m w_j \left(1-\left(\frac{1}{2}\right)^{l_j}\right)\right) + \frac{1}{2}\left(\sum_{j=1}^m w_jy_j^* \left(1-\left(1-\frac{1}{l_j}\right)^{l_j}\right)\right) \\
    &\geq \frac{1}{2}\left(\sum_{j=1}^m w_jy_j^* \left(1-\left(\frac{1}{2}\right)^{l_j}\right)\right) + \frac{1}{2}\left(\sum_{j=1}^m w_jy_j^* \left(1-\left(1-\frac{1}{l_j}\right)^{l_j}\right)\right) \\
    & (\text{because } 0 \leq y_j^* \leq 1) \\
    &= \frac{1}{2}\sum_{j=1}^m w_jy_j^* \left( \left(1-\left(\frac{1}{2}\right)^{l_j}\right) + \left(1-\left(1-\frac{1}{l_j}\right)^{l_j}\right)\right) \\
\end{align*}

Remains to show that $\forall n > 0$
$$\left(1-\left(\frac{1}{2}\right)^n\right) + \left(1-\left(1-\frac{1}{n}\right)^n\right) \geq \frac{3}{2}$$

Let $$f(x) = 2 - \left(\frac{1}{2}\right)^x - \left(1-\frac{1}{x}\right)^x$$
Then $f'(x) = - \left(\frac{1}{2}\right)^x\ln\frac{1}{2} - \left(1-\frac{1}{x}\right)^x\left(\ln(1-\frac{1}{x})+ \frac{1}{x-1}\right) > 0$. Hence $f$ is increasing for $x > 1$ and we get our desired result.

\end{proof}

\subsection*{Exercise 2}
\begin{proof}
Given a graph G, we perform the following experiment E, $n$ times on it:

Remove each edge of G with probability $p$ and retain it with probability $1-p$. If the resulting subgraph is disconnected, then set $X_i = 1$ otherwise, set $X_i = 0$.

Now, set $X = \frac{X_1+X_2+\cdots+X_t}{t}$

We have already seen that $E(X) = p_{fail} = \mu$ and $V(X) = \frac{\mu - \mu^2}{t} \leq \frac{\mu}{t}$ and so
$$Pr[|X-\mu|>\mu \epsilon] \leq \frac{1}{t\mu \epsilon^2} \leq \frac{poly(n)}{t \epsilon^2}$$
as $\mu = p_{fail} = \Omega\left(\frac{1}{poly(n)}\right)$ is given.

If we set this failure probability to atmost $\frac{1}{4}$ then we get $t \geq \frac{4}{\epsilon^2}poly(n)$.

Now, we repeat the above entire experiment $s = 2r+1$ times, to get i.i.d. random values $Y_1, Y_2, \ldots, Y_{s}$. Note that, we have $E(Y_i) = E(X)$ and $V(Y_i) = V(X)$ and so
$$Pr[|Y_i-\mu|>\mu \epsilon] \leq \frac{1}{t\mu \epsilon^2}$$

Let $Y = median(Y_1, Y_2, \ldots, Y_{2r+1})$.

We want to have failure probability atmost $\delta$ for $Y$ that is $Pr[|Y-\mu| > \mu \epsilon] \leq \delta$.

For that, we note that if $|Y-\mu| > \mu \epsilon$ then two of the following can happen:
\begin{itemize}
    \item $Y < (1-\epsilon)\mu$. Since $Y$ was the median, we must have $r+1$ many experiments to fail from below, that is there are $r+1$ many values for $i$ such that $Y_i < (1-\epsilon)\mu$.
    \item $Y > (1+\epsilon)\mu$. Since $Y$ was the median, we must have $r+1$ many experiments to fail from above, that is there are $r+1$ many values for $i$ such that $Y_i > (1-\epsilon)\mu$.
\end{itemize}

Now, we set $p = \frac{1}{t\mu\epsilon^2}$ and $q = 1-p$. Note that $p \leq \frac{1}{4} \implies q \geq \frac{3}{4}$.

We look at some inequalities first:
\begin{align}
    \forall k \leq r, \frac{q^kp^{2r+1-k}}{q^rp^r} = p\left(\frac{p}{q}\right)^{r-k} \leq 1  \\
    \sum_{k=0}^{r} {2r+1 \choose k} \leq \sum_{k=0}^{2r+1} {2r+1 \choose k} = 2^{2r+1} \\
    4q-4q^2 = 1 - (2q-1)^2 \leq 1 - \left(\frac{1}{2}\right)^2 \leq \frac{3}{4}
\end{align}

Hence, we get
\begin{align*}
    Pr[|Y-\mu| > \mu \epsilon] &= Pr[|Y_i-\mu| > \mu \epsilon \text{ for atleast } r+1 \text{ many values of } i] \\
    &= Pr[|Y_i-\mu| > \mu \epsilon \text{ for exactly } r+1 \text{ many values of } i] \\
    & + Pr[|Y_i-\mu| > \mu \epsilon \text{ for exactly } r+2 \text{ many values of } i] + \cdots \\
    & \cdots + Pr[|Y_i-\mu| > \mu \epsilon \text{ for exactly } 2r+1 \text{ many values of } i] \\
    &= \sum_{k=r+1}^{2r+1} {2r+1 \choose k} p^k q^{2r+1-k} \\
    &= \sum_{k=0}^{r} {2r+1 \choose k} q^k p^{2r+1-k} \\
    &\leq \sum_{k=0}^{r} {2r+1 \choose k} q^r p^r &\text{from }(1)\\
    &= (q-q^2)^r \sum_{k=0}^{r} {2r+1 \choose k} \\
    &\leq (q-q^2)^r 2^{2r} &\text{from }(2)\\
    &= (4q-4q^2)^r  \\
    &\leq \left(\frac{3}{4}\right)^r &\text{from }(3)\\
    &\leq \delta \\
\end{align*}

Now, we get $$r \geq \frac{\log\frac{1}{\delta}}{\log\frac{4}{3}} \implies s \geq \frac{2\log\frac{1}{\delta}}{\log\frac{4}{3}} + 1$$

Hence, $s = \Omega(\log\frac{1}{\delta})$.

So total time required is $t' = st \geq \frac{4}{\epsilon^2}poly(n)\log\frac{1}{\delta}$. Hence $t' = \Omega\left(\log \frac{1}{\delta}\right)$.
\end{proof}

\subsection*{Exercise 3}
\begin{proof}
For an assignment $a$, denote by $p(a)$ the probability of choosing that assignment where each variable is set to $1$ (or $0$) with probability $p$ (or $1-p$) independently. Further, for any given set $S$ of assignments, denote by $p(S) = \sum_{a \in S} p(a)$.

Let $S_i$ be the set of all satisfying assignments for the clause $C_i$ and set $\mathcal{S} = \bigcup_{i=1}^m S_i$.

Then $p(\mathcal{S}) = \sum_{a \in \mathcal{S}} p(a)$ is the number we need to approximate.

We first make a ($m$ x $|\mathcal{S}|$) table $T$ where the rows are indexed by the clauses and columns are indexed by the assignments $\mathcal{S}$. For each entry $(C_i, a)$ in the $T$, set it to $p(a)$ if $a$ satisfies $C_i$ otherwise set it to $0$. Mark the first non-zero entry in each column as special.

Now we perform the following experiment:

\begin{itemize}
    \item Choose a clause $C_i$ with probability $\frac{p(S_i)}{p(U)}$ where $p(U) = \sum p(S_i)$
    \item Having chosen a clause $C_i$, choose an assignment $a$ from set $S_i$ at random
    \item Set $X_n = 1$ iff the chosen assignment $a$ is special, otherwise $X_n = 0$
\end{itemize}

$E[X_n] = Pr[X_n = 1] = \frac{p(\mathcal{S})}{p(U)}$

Now let $X = \frac{\sum_{n \leq t} X_n}{t}$ and $Y = p(U) X$.

Then $Y$ approximates $p(\mathcal{S})$ as $E[Y] = p(\mathcal{S})$ and by law of large numbers, for large enough $t$, $Y \sim p(\mathcal{S})$.

[Note: We can compute $p(U) = \sum p(S_i)$ in polynomial time as $p(S_i) = \sum p^{p_i}(1-p)^{n_i}$ where the sum runs over all possibilities for the variables not occurring in $C_i$]
\end{proof}

\subsection*{Exercise 4}
\begin{proof}.

\begin{itemize}
    \item Probability two balls $B_i$ and $B_j$ fall into the same bin = $\sum_{k=1}^n$ Probability $B_i$ falls in bin $k$ and $B_j$ fall in bin $k$  = $\sum_{k=1}^n (1/n^2) = \boxed{\frac{1}{n}}$

    \item Let $X_{ij} = 1$ iff ball $B_i$ and ball $B_j$ collide otherwise $X_{ij} = 0$. Then the total number of collisions $X = \sum_{i \neq j} X_{ij}$.

    Hence, expected number of collisions = $E[X] = \sum_{i \neq j} E[X_{ij}] = \sum_{i \neq j} \frac{1}{n} = \boxed{{m \choose 2} \frac{1}{n}}$

    \item Probability that a particular bin is empty = $\boxed{\left(1-\frac{1}{n}\right)^m}$.

    Let $Y_k = 1$ iff bin $k$ is empty. Then total number of empty bins $Y = \sum_{k=1}^n Y_k$.

    Hence, expected number of empty bins = $E[Y] = \sum_{k=1}^n E[Y_k] = \sum_{k=1}^n (1-\frac{1}{n})^m = \boxed{n\left(1-\frac{1}{n}\right)^m}$.

    \item Probability that a particular bin has atleast $k$ balls = $\boxed{{m \choose k} \left(\frac{1}{n}\right)^k}$

    \item Probability that all bins have atmost $k$ balls = $1 -$ Probability that some bin has atleast $k+1$ balls $\geq$ $1 -$ Probability that some bin has atleast $k$ balls $\geq 1 - n$(Probability that a particular bin has atleast $k$ balls)

    Probability that a particular bin has atleast $k$ balls $\leq {m \choose k} (\frac{1}{n})^k \leq \left(\frac{me}{k}\right)^{k} \left(\frac{1}{n}\right)^{k} = \left(\frac{me}{nk}\right)^{k}$

    Assume that $\boxed{m \leq \frac{3}{e} n}$, then

    Probability that a particular bin has atleast $k$ balls $\leq \left(\frac{3}{k}\right)^{k}$

    Setting $k = \frac{3\ln n}{\ln \ln n}$, we get that, Probability that a particular bin has atleast $k$ balls $\leq \frac{1}{n^2}$

    Hence, probability that all bins have atmost $k$ balls $\geq 1 - \frac{1}{n}$

    \item Let $Z_i = $ number of balls in bin $i$ and $Z = max \{Z_i | 1\leq i \leq n \}$. Let $k = \frac{3\ln n}{\ln \ln n}$. We know that $P(Z<k) \geq 1-\frac{1}{n}$ and so $P(Z \geq k) \leq \frac{1}{n}$. Now,
        \begin{align*}
            E[Z] &= \sum_{i} i P(Z=i) \\
            &= \sum_{i<k} i P(Z=i) + \sum_{i \geq k} i P(Z=i) \\
            &\leq (k-1)P(Z < k) + nP(Z \geq k) \leq (k-1)P(Z < k) + 1 \\
            &\leq (k-1)+1 = k =  \frac{3\ln n}{\ln \ln n}
        \end{align*}
    Thus, $E[Z] = \frac{\ln n}{\ln \ln n}(1+O(1))$
\end{itemize}

\textbf{Analysis:}

1. Stirling's approximation:
$$e \sqrt{n}\left(\frac{n}{e}\right)^n \leq n! \leq \sqrt{2 \pi n}\left(\frac{n}{e}\right)^n$$

2. ${n \choose k} \leq (\frac{ne}{k})^k$
\begin{align*}
    {n \choose k} &\leq \frac{n!}{k!(n-k)!} \\
    &\leq \frac{e\sqrt{n}\left(\frac{n}{e}\right)^n}{\sqrt{2\pi k}\left(\frac{k}{e}\right)^k \sqrt{2 \pi (n-k)}\left(\frac{n-k}{e}\right)^{n-k}} \\
    &\leq \frac{e}{2\pi}\sqrt{\frac{n}{k(n-k)}} \left(\frac{n}{k}\right)^k \left(\frac{n}{n-k}\right)^{n-k} \\
    &\leq \frac{e}{\pi} \left(\frac{n}{k}\right)^k \left(1 + \frac{k}{n-k}\right)^{n-k} \\
    &\leq \frac{e}{\pi} \left(\frac{n}{k}\right)^k e^k \\
    &\leq \left(\frac{ne}{k}\right)^k \\
\end{align*}

3. $(\ln n)/n$ is a decreasing function $\forall n \geq 1$ (decreases to $0$) and hence $(\ln n)/n \leq \frac{1}{3}$ eventually.

4. If $k = \frac{3\ln n}{\ln \ln n}$, then $\left(\frac{3}{k}\right)^k \leq \frac{1}{n^2}$ for large enough $n$
\begin{align*}
    \left(\frac{3}{k}\right)^k &= \exp \left( k \ln \frac{3}{k} \right) \\
    &= \exp \left( \frac{3\ln n}{\ln \ln n} \ln \left(\frac{\ln \ln n}{\ln n}\right) \right) \\
    &= \exp \left( \frac{3\ln n}{\ln \ln n} (\ln\ln\ln n - \ln \ln n) \right) \\
    &= \exp \left( -3\ln n + \ln n \frac{3\ln\ln\ln n}{\ln\ln n} \right) \\
    &\leq \exp \left( -3\ln n +  \ln n \right) \\
    &= \exp \left( -2\ln n\right) \\
    &= \frac{1}{n^2}
\end{align*}
\end{proof}

\vspace{2in} %Leave more space for comments!

\end{document}


\chapter{Support Vector Machines}
\label{ch:svm}

Support vector machines (SVM) are another example of linear classifiers, similar to logistic or linear regression. However, SVM can overcome splitting the data by a plane by using the so-called \textit{kernel trick}. This means the hyperplane (decision boundary) can be transformed to a higher-dimensional space, which can fit the data nicely. In such a way, SVM becomes a non-linear classifier and can fit more complex data sets.

\begin{marginfigure}
    \includegraphics[width=50mm]{linear-regression.png}%
    \caption{Decision boundary of a linear regression classifier.}
  \end{marginfigure}

\begin{marginfigure}
    \includegraphics[width=50mm]{svm.png}%
    \caption{Decision boundary of a support vector machine classifier with an RBF kernel.}
\end{marginfigure}

The magic of SVM (and other methods that can use kernels, and are thus called kernel methods) is that they will implicitly find a transformation into a (usually infinite-dimensional) space, in which the distances between objects are such as prescribed by the kernel, and draw a hyperplane in this space.

\begin{figure}[h]
    \centering
    \includegraphics[scale=0.5]{workflow.png}
    \caption{$\;$}
\end{figure}

Abstract talking aside, SVM with different kernels can split the data not by ordinary hyperplanes, but with more complex curves. The complexity of the curve is decided by the kernel type and by the arguments given to the algorithm, like the degree and coefficients, and the penalty for misclassifications.

\begin{figure}[h]
    \centering
    \includegraphics[height=70mm]{svm-orange.png}
    \caption{$\;$}
\end{figure}


\section*{Problem 1 Solution}

For all parts of this problem, let $R$ be the rate of neutron capture by 1 g of $^{232}$Th, $2.0 \times 10^{11} \;\text{s}^{-1}$.\\

\begin{enumerate}[a)]

\item

First, convert all half-lives and irradiation times to seconds for consistency.\\
\-\\
\tab $\lambda_{\text{Th}} = \frac{\ln{2}}{1338\text{s}} = 5.18\times10^{-4}\text{s}^{-1}$; \tab $\lambda_{\text{Pa}}=\frac{\ln{2}}{2332800s}=2.971\times10^{-7}\text{s}^{-1}$; \tab 1.5 hr = 5400 s\\

\textbf{Thorium-233}\\
\-\\
The rate of change of the quantity of $^{233}$Th, $\frac{dN_{\text{th}}}{dt}$ is given by the production rate of $^{233}$Th, $R$, minus the decay rate (activity) of $^{233}$Th, $\lambda_{\text{Th}}N_{\text{Th}}$.
$$\frac{dN_{\text{Th}}}{dt} = R - \lambda_{\text{Th}}N_{\text{Th}}$$
We solve the differential equation, manipulating the equation so that the left side is only dependent on $N_{\text{Th}}$ and the right side is only dependent on $dt$. Then we integrate:
$$\int{\frac{dN_{\text{Th}}}{R-\lambda_{\text{Th}}N_{\text{Th}}}} = \int{dt}$$
$$\frac{-1}{\lambda_{\text{Th}}}[\ln(R-\lambda_{\text{Th}}N_{\text{Th}})] = t + C,\quad C=\text{const.}$$
$$R-\lambda_{\text{Th}}N_{\text{Th}} = e^{-\lambda_{\text{Th}}t -\lambda_{\text{Th}}C} = e^{-\lambda_{\text{Th}}t} e^{-\lambda_{\text{Th}}C}$$
We note that since $C$ is an arbitrary constant and $\lambda_{\text{Th}}$ is fixed, we could also say $e^{-\lambda_{\text{Th}}C}$ is an arbitrary constant, and just call it $C$ instead.
$$ R-\lambda_{\text{Th}}N_{\text{Th}} = Ce^{-\lambda_{\text{Th}}t} $$
We solve for $N_{\text{Th}}$ (explicitly including $N_{\text{Th}}$'s dependence on $t$), and get
$$ N_{\text{Th}}(t) = \frac{R - Ce^{-\lambda_{\text{Th}}t}}{\lambda_{\text{Th}}} .$$
At $t=0,\; N_{\text{Th}}(0) = \frac{R - C}{\lambda_{\text{Th}}}=0,$ since no $^{233}$\text{Th} has been formed. We find $C= R$, and use this in the general equation:
$$ N_{\text{Th}}(t) = R\frac{1 - e^{-\lambda_{\text{Th}}t}}{\lambda_{\text{Th}}}. $$
With this function of $N_{\text{Th}}$, we can determine the activity as a function of time, knowing that
$$ \mathcal{A}_{\text{Th}}(t) = \lambda_{\text{Th}}N_{\text{Th}}(t). $$
Substituting, we find
$$ \mathcal{A}_{\text{Th}}(t) = R(1-e^{-\lambda_{\text{Th}}t}) .$$
Using the numerical values for $R$, $\lambda_{\text{Th}}$, and $t$,
$$ \mathcal{A}_{\text{Th}}(1.5\text{ hr}) =(2.0\times10^{11}s^{-1})(1-e^{(-5.18\times10^{-4}\text{s}^{-1})(5400s)}) $$
$$ \mathcal{A}_{\text{Th}}(1.5\text{ hr}) = 1.878\times10^{11}\text{ Bq} .$$
Finally, we convert this to Curies,
$$\boxed{ \mathcal{A}_{\text{Th}}(1.5\text{ hr}) = 5.076\text{ Ci} }.$$

\textbf{Protactinium-233}\\
\-\\
We follow a similar procedure for $^{233}$Pa, noting that the production rate of $^{233}$Pa is just the activity of $^{233}$Th as it decays into $^{233}$Pa, $\mathcal{A}_{\text{Th}}$.
$$\frac{dN_{\text{Pa}}}{dt} = \mathcal{A}_{\text{Th}} - \lambda_{\text{Pa}}N_{\text{Pa}}$$
From above, we can substitute our function for $\mathcal{A}_{\text{Th}}(t)$,
$$\frac{dN_{\text{Pa}}}{dt} = R(1-e^{-\lambda_{\text{Th}}t}) - \lambda_{\text{Pa}}N_{\text{Pa}}$$
Since we cannot separate both sides to be dependent only on a single differential, we must try a different method of integration. We will use integrating factors. Still, we start in a similar fashion: collecting the terms dependent on $N_{\text{Pa}}$ on the same side.
$$ \frac{dN_{\text{Pa}}}{dt}+\lambda_{\text{Pa}}N_{\text{Pa}} = R(1-e^{-\lambda_{\text{Th}}t}) $$
The method of integrating factors suggests that we multiply both sides by an arbitrary exponential. We will use $e^{\lambda_{\text{Pa}}t}$.
$$ e^{\lambda_{\text{Pa}}t}\frac{dN_{\text{Pa}}}{dt} + \lambda_{\text{Pa}}e^{\lambda_{\text{Pa}}t}N_{\text{Pa}} = e^{\lambda_{\text{Pa}}t}R(1-e^{-\lambda_{\text{Th}}t}) $$
We can now observe that the left side of the equation appears to be the result of the product rule when the time derivative of $e^{\lambda_{\text{Pa}}t}N_{\text{Pa}}$ is found. We can then write the equation as
$$\frac{d}{dt}(e^{\lambda_{\text{Pa}}t}N_{\text{Pa}}) = e^{\lambda_{\text{Pa}}t}R(1-e^{-\lambda_{\text{Th}}t})$$
Moving the $dt$ term to the right side of the equation and using the distributive property, we have
$$ d(e^{\lambda_{\text{Pa}}t}N_{\text{Pa}}) = R(e^{\lambda_{\text{Pa}}t}-e^{\lambda_{\text{Pa}}t}e^{-\lambda_{\text{Th}}t})dt $$
or more simply (by exploiting properties of exponents)
$$ d(e^{\lambda_{\text{Pa}}t}N_{\text{Pa}}) = R(e^{\lambda_{\text{Pa}}t}-e^{\lambda_{\text{Pa}}t-\lambda_{\text{Th}}t})dt .$$
We integrate both sides,
$$ \int{d\left(e^{\lambda_{\text{Pa}}t}N_{\text{Pa}}\right)} = \int{ R(e^{\lambda_{\text{Pa}}t}-e^{\lambda_{\text{Pa}}t-\lambda_{\text{Th}}t})dt}, $$
separate the integral on the right side,
$$ \int{d\left(e^{\lambda_{\text{Pa}}t}N_{\text{Pa}}\right)} = R\int{ e^{\lambda_{\text{Pa}}t}dt}-R\int{e^{(\lambda_{\text{Pa}}-\lambda_{\text{Th}})t}dt} $$
and find
$$e^{\lambda_{\text{Pa}}t}N_{\text{Pa}} = \frac{R}{\lambda_{\text{Pa}}}e^{\lambda_{\text{Pa}}t}-\frac{R}{\lambda_{\text{Pa}}-\lambda_{\text{Th}}}e^{(\lambda_{\text{Pa}}-\lambda_{\text{Th}})t} +\;C,\; C=\text{const.}$$
Now we factor out the integrating factor back out from both sides and note explicitly the time dependence of $N_{\text{Pa}}$,
$$N_{\text{Pa}}(t) = \frac{R}{\lambda_{\text{Pa}}}-\frac{R}{\lambda_{\text{Pa}}-\lambda_{\text{Th}}}e^{-\lambda_{\text{Th}}t}+Ce^{-\lambda_{\text{Pa}}t}$$
At $t=0$,
$$ N_{\text{Pa}}(0) = \frac{R}{\lambda_{\text{Pa}}}-\frac{R}{\lambda_{\text{Pa}}-\lambda_{\text{Th}}}+C = 0, $$
 since no $^{233}$\text{Pa} has been formed. Solving for $C$, we find $C = \frac{R}{\lambda_{\text{Pa}}-\lambda_{\text{Th}}}-\frac{R}{\lambda_{\text{Pa}}}$. We plug this back into our equation above, and have the solution for $N_{\text{Pa}}(t)$:
$$N_{\text{Pa}}(t) = \frac{R}{\lambda_{\text{Pa}}}-\frac{R}{\lambda_{\text{Pa}}-\lambda_{\text{Th}}}e^{-\lambda_{\text{Th}}t}+\left( \frac{R}{\lambda_{\text{Pa}}-\lambda_{\text{Th}}}-\frac{R}{\lambda_{\text{Pa}}} \right)e^{-\lambda_{\text{Pa}}t}$$
and simplifying
$$ N_{\text{Pa}}(t) = \frac{R}{\lambda_{\text{Pa}}}(1-e^{\lambda_{\text{Pa}}t})+ \left(\frac{R}{\lambda_{\text{Pa}}-\lambda_{\text{Th}}}\right)(e^{-\lambda_{\text{Pa}}t}-e^{-\lambda_{\text{Th}}t}) .$$
With this function of $N_{\text{Pa}}$, we can determine the activity as a function of time, knowing that
$$ \mathcal{A}_{\text{Pa}}(t) = \lambda_{\text{Pa}}N_{\text{Pa}}(t) .$$
Substituting, we find
$$ \mathcal{A}_{\text{Pa}}(t) = R(1-e^{-\lambda_{\text{Pa}}t})+ \left(\frac{R \, \lambda_{\text{Pa}}}{\lambda_{\text{Pa}}-\lambda_{\text{Th}}}\right)(e^{-\lambda_{\text{Pa}}t}-e^{-\lambda_{\text{Th}}t}) .$$
Using the numerical values for $R$, $\lambda_{\text{Th}}$, $\lambda_{\text{Pa}}$, and $t$,
\begin{dmath*}
\mathcal{A}_{\text{Pa}}(1.5\text{ hr}) = (2.0\times10^{11}\text{ s}^{-1})\left(1-e^{(-2.971\times10^{-7}\text{s}^{-1})(5400\text{s})}\right)+ \left(\frac{2.0\times10^{11}\text{ s}^{-1}(2.791\times10^{-7}\text{s}^{-1})}{2.971\times10^{-7}\text{s}^{-1}-5.18\times10^{-4}\text{s}^{-1}}\right)\left(e^{(-2.971\times10^{-7}\text{s}^{-1})(5400\text{s})}-e^{(-5.18\times10^{-4}\text{s}^{-1})(5400\text{s})}\right)
\end{dmath*}
$$ \mathcal{A}_{\text{Pa}}(1.5\text{ hr}) = 2.195 \times 10^{8}\text{ Bq} $$
Finally, we convert this to Curies,
$$ \boxed{\mathcal{A}_{\text{Pa}}(1.5\text{ hr}) = 0.006\text{ Ci}}. $$

\item

Let's say that the 1.5 hour mark is now given by $t=t_0=1.5\text{ hr}$. We also note that 48 hours = 172,800 seconds.

\textbf{Thorium-233}\\
\-\\
Without irradiation, the rate of change of the quantity of $^{233}$Th is now just the decay rate.
$$ \frac{dN_{\text{Th}}}{dt} = -\lambda_{\text{Th}}N_{\text{Th}} $$
We separate the equation and integrate, arriving at the standard exponential decay formula, now including the explicit time dependence.
$$ \int_{N_{\text{Th}}(t_0)}^{N_{\text{Th}}(t)} \frac{-dN_{\text{Th}}'}{\lambda_{\text{Th}}N_{\text{Th}}'} = \int_{t_0}^{t} dt' $$
$$ \frac{-1}{\lambda_{\text{Th}}}\left[ \ln{N_{\text{Th}}} \right]_{N_{\text{Th}(t_0)}}^{N_{\text{Th}}(t)} = \left[t'\right]_{t_0}^{t} $$
$$ \ln\frac{N_{\text{Th}}(t)}{N_{\text{Th}}(t_0)}  = -\lambda_{\text{Th}}(t-t_0) $$
$$ \frac{N_{\text{Th}}(t)}{N_{\text{Th}}(t_0)}  = e^{-\lambda_{\text{Th}}(t-t_0)} $$
and we have
$$ N_{\text{Th}}(t) = N_{\text{Th}}(t_0) e^{-\lambda_{\text{Th}}\left(t-t_0\right)} $$
Given the definition of activity as $\mathcal{A} = \lambda N$ and noting $\lambda_{\text{Th}} N_{\text{Th}}(t_0) = \mathcal{A}_{\text{Th}}(t_0)$, we can write the activity of $^{233}$Th as
\begin{align*}
\mathcal{A}_{\text{Th}}(t)	&= \lambda_{\text{Th}} N_{\text{Th}}(t) \\
							&= \lambda_{\text{Th}} N_{\text{Th}}(t_0) e^{-\lambda_{\text{Th}}(t-t_0)} \\
							&= \mathcal{A}_{\text{Th}}(t_0) e^{-\lambda_{\text{Th}}(t-t_0)}
\end{align*}
Using the numerical values for $\lambda_{\text{Th}}$, $t$, and our answer from part (a) for the activity at $t_0=1.5$ hr, we find
$$ \mathcal{A}_{\text{Th}}(49.5\text{ hr}) = (5.076 \text{ Ci})e^{-5.18\times10^{-4}\text{s}^{-1} (172800\text{s})} $$
$$ \boxed{\mathcal{A}_{\text{Th}}(49.5\text{ hr}) = 6.787\times10^{-39}\text{ Ci}} $$

{\small Note: we could also have assumed that since 48 hours is many (more than 100) times longer than the half-life of $^{233}$Th, that the activity would be approximately zero.}
\-\\
\-\\

\textbf{Protactinium-233}\\
\-\\
We follow the example in part (a) for $^{233}$Pa, again using the activity of $^{233}$Th as the production rate of $^{233}$Pa.
$$ \frac{dN_{\text{Pa}}}{dt} = \mathcal{A}_{\text{Th}} - \lambda_{\text{Pa}}N_{\text{Pa}}. $$
We collect terms dependent on $N_{\text{Pa}}$ on one side,
$$ \frac{dN_{\text{Pa}}}{dt} + \lambda_{\text{Pa}}N_{\text{Pa}} = \mathcal{A}_{\text{Th}}(t_0) e^{-\lambda_{\text{Th}}(t-t_0)}, $$
multiply both sides by arbitrary exponential $e^{\lambda_{\text{Pa}}t}$,
$$ e^{\lambda_{\text{Pa}}t}\frac{dN_{\text{Pa}}}{dt} + \lambda_{\text{Pa}}e^{\lambda_{\text{Pa}}t}N_{\text{Pa}} = e^{\lambda_{\text{Pa}}t}\mathcal{A}_{\text{Th}}(t_0) e^{-\lambda_{\text{Th}}(t-t_0)}, $$
note that the left side is the result of the product rule when $\frac{d}{dt}$ is taken on $e^{\lambda_{\text{Pa}}t}N_{\text{Pa}}$
$$ \frac{d}{dt} (e^{\lambda_{\text{Pa}}t}N_{\text{Pa}}) = e^{\lambda_{\text{Pa}}t}\mathcal{A}_{\text{Th}}(t_0) e^{-\lambda_{\text{Th}}(t-t_0)}, $$
multiply by the differential, $dt$,
$$ d(e^{\lambda_{\text{Pa}}t}N_{\text{Pa}}) = e^{\lambda_{\text{Pa}}t}\mathcal{A}_{\text{Th}}(t_0) e^{-\lambda_{\text{Th}}(t-t_0)} dt ,$$
and rearrange,
$$ d(e^{\lambda_{\text{Pa}}t}N_{\text{Pa}}) = \mathcal{A}_{\text{Th}}(t_0) e^{(\lambda_{\text{Pa}} - \lambda_{\text{Th}})t + \lambda_{\text{Th}}t_0} dt .$$
Then we integrate,
$$ \int_{e^{\lambda_{\text{Pa}}t_0}N_{\text{Pa}}(t_0)}^{e^{\lambda_{\text{Pa}}t}N_{\text{Pa}}(t)} d(e^{\lambda_{\text{Pa}}t}N_{\text{Pa}}') = \mathcal{A}_{\text{Th}}(t_0)e^{\lambda_{\text{Th}}t_0} \int_{t_0}^{t} e^{(\lambda_{\text{Pa}} - \lambda_{\text{Th}})t'} dt' ,$$
and find,
$$ \left[e^{\lambda_{\text{Pa}}t}N_{\text{Pa}}'\right]_{e^{\lambda_{\text{Pa}}t_0}N_{\text{Pa}}(t_0)}^{e^{\lambda_{\text{Pa}}t}N_{\text{Pa}}(t)} = \frac{\mathcal{A}_{\text{Th}}(t_0)e^{\lambda_{\text{Th}}t_0}}{\lambda_{\text{Pa}}-\lambda_{\text{Th}}} \left[ e^{(\lambda_{\text{Pa}}-\lambda_{\text{Th}})t'}\right]_{t_0}^{t} $$
$$ e^{\lambda_{\text{Pa}}t}N_{\text{Pa}}(t) - e^{\lambda_{\text{Pa}}t_0}N_{\text{Pa}}(t_0) = \frac{\mathcal{A}_{\text{Th}}(t_0)e^{\lambda_{\text{Th}}t_0}}{\lambda_{\text{Pa}}-\lambda_{\text{Th}}} \left[ e^{(\lambda_{\text{Pa}}-\lambda_{\text{Th}})t} - e^{(\lambda_{\text{Pa}}-\lambda_{\text{Th}})t_0} \right] $$
Factoring out the integrating factor,
$$ N_{\text{Pa}}(t) - e^{-\lambda_{\text{Pa}}(t - t_0)}N_{\text{Pa}}(t_0) = \frac{\mathcal{A}_{\text{Th}}(t_0)e^{\lambda_{\text{Th}}t_0}}{\lambda_{\text{Pa}}-\lambda_{\text{Th}}} \left[ e^{-\lambda_{\text{Th}}t} - e^{-\lambda_{\text{Pa}}t}e^{(\lambda_{\text{Pa}}-\lambda_{\text{Th}})t_0} \right] $$
and solve for $ N_{\text{Pa}}(t)$,
$$ N_{\text{Pa}}(t) = \frac{\mathcal{A}_{\text{Th}}(t_0)e^{\lambda_{\text{Th}}t_0}}{\lambda_{\text{Pa}}-\lambda_{\text{Th}}} \left[ e^{-\lambda_{\text{Th}}t} - e^{-\lambda_{\text{Pa}}t}e^{(\lambda_{\text{Pa}}-\lambda_{\text{Th}})t_0} \right] + e^{-\lambda_{\text{Pa}}(t-t_0)}N_{\text{Pa}}(t_0) .$$
Rearranging,
$$ N_{\text{Pa}}(t) = \frac{\mathcal{A}_{\text{Th}}(t_0)}{\lambda_{\text{Pa}}-\lambda_{\text{Th}}} \left[ e^{-\lambda_{\text{Th}}(t - t_0)} - e^{-\lambda_{\text{Pa}}(t-t_0)} \right] + e^{-\lambda_{\text{Pa}}(t - t_0)}N_{\text{Pa}}(t_0) .$$
and so
$$ N_{\text{Pa}}(t) = \frac{\mathcal{A}_{\text{Th}}(t_0)}{\lambda_{\text{Pa}}-\lambda_{\text{Th}}} e^{-\lambda_{\text{Th}}(t-t_0)} + \left( N_{\text{Pa}}(t_0) - \frac{\mathcal{A}_{\text{Th}}(t_0)}{\lambda_{\text{Pa}}-\lambda_{\text{Th}}} \right) e^{-\lambda_{\text{Pa}}(t-t_0)} $$
Given the definition of activity as $\mathcal{A} = \lambda N$ and noting $\lambda_{\text{Pa}} N_{\text{Pa}}(t_0) = \mathcal{A}_{\text{Pa}}(t_0)$, we can write the activity of $^{233}$Pa as
\begin{align*}
\mathcal{A}_{\text{Pa}}(t)	&= \lambda_{\text{Pa}} N_{\text{Pa}}(t) \\
							&=\lambda_{\text{Pa}} \left(\frac{\mathcal{A}_{\text{Th}}(t_0)}{\lambda_{\text{Pa}}-\lambda_{\text{Th}}} e^{-\lambda_{\text{Th}}(t-t_0)} + \left( N_{\text{Pa}}(t_0) - \frac{\mathcal{A}_{\text{Th}}(t_0)}{\lambda_{\text{Pa}}-\lambda_{\text{Th}}} \right) e^{-\lambda_{\text{Pa}}(t-t_0)}\right) \\
							&= \frac{\mathcal{A}_{\text{Th}}(t_0) \lambda_{\text{Pa}}}{\lambda_{\text{Pa}}-\lambda_{\text{Th}}} e^{-\lambda_{\text{Th}}(t-t_0)} + \left( \mathcal{A}_{\text{Pa}}(t_0) - \frac{\mathcal{A}_{\text{Th}}(t_0) \lambda_{\text{Pa}}}{\lambda_{\text{Pa}}-\lambda_{\text{Th}}} \right) e^{-\lambda_{\text{Pa}}(t-t_0)} \\
							&= \mathcal{A}_{\text{Pa}}(t_0) e^{-\lambda_{\text{Pa}}(t-t_0)} +  \frac{\mathcal{A}_{\text{Th}}(t_0) \lambda_{\text{Pa}}}{\lambda_{\text{Pa}}-\lambda_{\text{Th}}} \left( e^{-\lambda_{\text{Th}}(t-t_0)} - e^{-\lambda_{\text{Pa}}(t-t_0)} \right) \\
\end{align*}
Using the numerical values for $\lambda_{\text{Th}}$, $\lambda_{\text{Pa}}$, $t$, and our answer from part (a) for the activities at $t=t_0=1.5$ hr, we find
\begin{dmath*}
\mathcal{A}_{\text{Pa}}(49.5\text{ hr}) = (0.006\text{ Ci}) e^{-(2.971\times10^{-7}\text{s}^{-1})(172800s)} +  \frac{(5.076\text{ Ci}) (2.971\times10^{-7}\text{s}^{-1})}{2.971\times10^{-7}\text{s}^{-1}-5.18\times10^{-4}\text{s}^{-1}} \left( e^{-(5.18\times10^{-4}\text{s}^{-1})(172800s)} - e^{-(2.971\times10^{-7}\text{s}^{-1})(172800s)} \right)
\end{dmath*}
$$\boxed{ \mathcal{A}_{\text{Pa}}(49.5\text{ hr}) = 0.008\text{ Ci} }$$

\item

Note that $\lambda_{\text{U}} = \frac{\ln 2}{5.024^{12}\text{s}} = 1.380\times10^{-13}\text{s}^{-1}$.

In just one day, the probability that any given $^{233}$Th nucleus survives is
$$ P = e^{-\lambda_{\text{Th}}t_d} $$
$$ P = e^{-(5.18\times10^{-4}\text{ s}^{-1})(86,400\text{ s})} \approx 10^{-20} $$
We can therefore assume that each thorium nucleus produced in the reactor has decayed into $^{233}$Pa by the end of the first day of storage. (There were only about $10^{15}$ thorium nuclei produced in total).
The probability that any one of these $^{233}$Pa atoms remains after one year---assumed to be 364 more days---is
$$ P = e^{-\lambda_{\text{Pa}}(t_y-t_d)} $$
$$ P = e^{-(2.971\times10^{-7}\text{ s}^{-1})(31449600\text{ s})} = 8.752\times10^{-5}  $$
While this probability indicates that some $^{233}$Pa nuclei will remain after the year of storage, they hardly make up a substantial fraction of the originally produced set of nuclei. We can assume that in one year, virtually every nucleus of $^{233}$Th has decayed at least into $^{233}$U.

For uranium-233 on the other hand, the probability of survival for a nucleus over a year is
$$ P = e^{-\lambda_{\text{U}}t_y} $$
$$ P = e^{-(1.380\times10^{-13})(31536000\text{ s})} $$
$$ P = 0.999996 $$
We see that while nearly every nucleus decays from $^{233}$Th into $^{233}$Pa and then into $^{233}$U, almost no nuclei seem to decay from $^{233}$U in the single year.
We can then use our simple formula for the activity of the uranium sample
$$ \mathcal{A}_{\text{U}} = \lambda_{\text{U}} N_{\text{U}}$$

Using our rate of production, there are $2.0\times10^{11}\text{ s}^{-1}*(3600\text{ s/hr}\times 1.5 \text{hr}) \approx 1.08\times10^{15}$ produced in the irradiation period. We will assume this is also equal to the number of nuclei of $^{233}$U present at the end of the 1 year storage period: $N_U \approx 10^{15}$. We can finally calculate the activity, as
$$ \mathcal{A}_{\text{U}} = (1.380\times10^{-13}\text{s}^{-1})(1.08\times10^{15}) .$$
This is ${149.04}\text{ Bq}$ or
$$\boxed{ 4.028\times10^{-9}\text{ Ci} }$$
\end{enumerate}


%
% Chapter 1.3
%

\section*{1.3 More advanced functions}

\subsection*{Vertical and Horizontal Shifts}

Suppose \(c>0\). Then
\begin{center}
\(y=f(x)+c\) is the graph of \(y=f(x)\) shifted \(c\) units upward.\\
\(y=f(x)-c\) is the graph of \(y=f(x)\) shifted \(c\) units downward.\\
\(y=f(x+c)\) is the graph of \(y=f(x)\) shifted \(c\) units to the left.\\
\(y=f(x-c)\) is the graph of \(y=f(x)\) shifted \(c\) units to the right.\\
\end{center}

\subsection*{Vertical and Horizontal Stretching and Reflecting}

Suppose \(c>1\). Then
\begin{center}
\(y=cf(x)\) is the graph of \(y=f(x)\) stretched vertically by a factor of \(c\).\\
\(y=(1/c)f(x)\) is the graph of \(y=f(x)\) shrunk vertically by a factor of \(c\).\\
\(y=f(cx)\) is the graph of \(y=f(x)\) shrunk horizontally by a factor of \(c\).\\
\(y=f(x/c)\) is the graph of \(y=f(x)\) stretched horizontally by a factor of \(c\).\\
\(y=-f(x)\) is the graph of \(y=f(x)\) reflected about the \(x\)-axis.\\
\(y=f(-x)\) is the graph of \(y=f(x)\) reflected about the \(y\)-axis.
\end{center}
Given two functions \(f\) and \(g\), the \textbf{composite function} \(f \circ g\) is defined by
$$(f \circ g)(x)=f(g(x))$$


\chapter{Heaps}

For the purposes of generality, instead of referring to
elements that are ``greater than" or ``less than" others,
we will simply say that they are ``better than" or ``worse than"
others. For any particular ordering, the ``best" element is desired first.
A Heap is a binary tree that satisfies the following properties:

\begin{itemize}
\item The root of a heap is better than its two children (the heap property)
\item The children of the root are also heaps
\item A heap is a complete binary tree (only the last level may not be full,
and all elements in the last level are on the left)
\end{itemize}

A heap differs from most binary trees in that it provides no particular
ordering of the elements, but rather guarantees that the best element is
at the root. Further, while heaps are usually discussed and defined
as binary trees, they need not be implemented as such.
A heap can in fact be implemented as an array with no performance reduction.
Contrary to standard theoretical convention, we will be using 0-indexed arrays
rather than 1-indexed arrays. That is to say, the indexing will reflect how
most programming languages perform indexing (0 is the first element, not 1).
To represent a heap as an array, we can implement a simple indexing.
If an element is at the $i$th
index in the array, its left and right child are at the $2i+1$th and $2i+2$th
indexes respectively. By placing the root at the $0$th index, all others follow.
This is depicted below.
{
  \includegraphics[scale=0.5]{heapTreeArray}
  %\caption{The mapping of a binary tree to an array}
  \label{fig:heapTreeArray}
}
Heaps are commonly used to implement priority queues, because they do the
minimum amount of work required to keep track of the best element.

\section{How to Implement a Heap}

A heap must support the following operations

\begin{itemize}
\item $best()$: returns the best element in the heap
\item $pop-best()$: removes the best element in the heap
\item $insert(x)$: inserts $x$ into the heap
\end{itemize}

From the definition of a heap, we know the we can easily implement
$best()$ by returning the root of the heap, which should take $O(1)$
time.  However the other operations are less obvious.

To $insert(x)$ recall that a heap must be complete, therefore, if a
new element is added, it must be added to the left-most available
space in the last row of the heap. However, the heap property may now
be violated.  If the new element $x$ is worse than its parent $y$, the
heap property is satisfied and we may stop. However if it is not
satisfied we may swap $x$ with $y$ and recurse on $x$'s new
position. This works because we know that $y$ is better than all of
$x$'s children, because the heap property was satisfied before $x$ was
added. Further, because $x$ is better than $y$, it is also better than
all of $y$'s children. However $x$ may still be better than its new
parent, so we must recurse. If $x$ is the new best element, it will
eventually reach the root. Because heaps are complete, this operation
will take $O(\log n)$ time, as this is the height of the heap.

To $pop-best()$, we may simply remove the root, however this
completely destroys the entire heap. Instead, we will swap the root
with the bottom-left-most element, $y$. Now removing the best element
leaves us with a still complete tree. However the heap property has
likely been violated once more. If $y$ is better than both its
children, then we may stop. However, if not, we shall swap $y$ with
its best child and recurse on $y's$ new position. Because the element
we swap with $y$ is better than both $y$ and the other child, the heap
property has been satisfied for this sub-heap. However, the heap
property may still be violated for $y's$ new sub-heap, so we must do
this again, until $y$ is the root of a valid heap. Once more, this
operation requires $O(\log n)$ time, as it must at worse traverse the
entire height of the tree.

Therefore, a heap may support $best()$ in $O(1)$ time, $insert(x)$
in $O(\log n)$ time, and $pop-best()$ in $O(\log n)$ time.

\section{Building a Heap}

Now that we can support all the operations that a heap must implement,
it would be nice to be able to actually construct one given a list of $n$
elements. A na\"ive approach is to simply call $insert(x)$ on every element
in the list. However, since $insert(x)$ requires $O(\log n)$ time, this will
require $O(n \log n)$ time. These seems pretty bad, considering one can find
the best element in a list by brute force in $O(n)$ time. Can we achieve a
construction time comparable to the brute force time?
Instead of building the heap top down with $insert$, we can build it from the
bottom up. Remark that a single element is a valid heap. If we were to try to
build from the bottom up, we could first take the last $n/2$ elements in the
list. All of these elements are their own valid and complete heaps,
and we therefore do not need to do anything to them.
To add the next $n/4$ elements, we simply perform
the procedure we did in $pop-best$ to fix the fact that the new root might
be violating the heap property, knowing that all the elements below it are
valid heaps. By repeating this process until we reach the first element in
the list, we will have created a valid heap on $n$ elements.

Because we are doing very little work for the majority of the elements, we end
up doing only $O(n)$ work over all, which is optimal, as this is the amount of
time required to find the best element.

\section{Heapsort}

Another nice property of a heap is that once one has been implemented,
it provides a very simple procedure for sort elements.
A simple algorithm to do this to construct a heap on the list and then simply
return $best$ and then call $pop-best$ over and over until there are no more
elements in the heap.
In fact, since our algorithm for building the heap is in-place and takes $O(n)$
time, and our remove method leaves the best element at the end of the array,
by simply building a heap on the input array and calling $pop-best$ $n$ times,
we will be left with a reverse sorted array in $O(n \log n)$ time.
This algorithm is particularly excellent because it requires no extra space,
runs deterministically, and is worst-case optimal.