Most hierarchical clustering algorithms
produce a binary tree over the data,
where each leaf corresponds to a data point.
Internal nodes correspond to groupings of descendant leaves.
This approach is simple, but can run into
problems when the hierarchical structure
in the data is ambiguous. Consider
a simple dataset with three points
that are equidistant from each other
or a dataset with three points
lying equally spaced apart in a line (see \autoref{fig:ambiguous-structure}).
A single binary tree is not sufficient
to describe the similarity relationships
between the points, as a single tree
will imply that some points are more similar
to each other than the other
when they are in fact not.
An initial solution is to extend clustering
algorithms to produce trees with arbitrary
branching factors.
This will help solve the
ambiguity problem
in the first dataset, but
not the second.
Furthermore, we can construct
adversarial datasets
that will always be ambiguous.
Consider $\numdata$ points lying on the unit circle, equally spaced apart.
Due to rotational symmetry, there will be several
equivalent trees with max branching factor less than $\numdata$.
Since an $\numdata$-ary tree over $\numdata$ data points
implies no underlying structure,
we require an approach that can
handle ambiguity without compromising
the capacity to discover structure.

Bayesian nonparametric hierarchical clustering (BNHC)
addresses the general issue of
ambiguity in hierarchical structure
by explicitly modeling any uncertainty with probability.
Rather than outputting a single tree
as in traditional hierarchical clustering,
a Bayesian method
returns a probability distribution
over hierarchies that explain the data.
In the ambiguous dataset with three points described earlier,
a Bayesian method would return
a distribution over the three hierarchies,
where each is assigned
a probability of $1/3$.
BNHC
also follows the standard Bayesian paradigm,
where Bayes rule enables
calculating a posterior distribution
given a prior and a likelihood model.

%The chapter is organized as follows.
%In section 2 we provide background information
%about traditional hierarchical clustering
%methods and Bayesian learning.
%In section 3 we lay down the fundamentals
%of Bayesian hierarchical clustering,
%describe several of the prominent
%models in BNHC, and provide
%algorithms to perform inference in the models.
%In section 4 we introduce ideas
%to improve BNHC with user interaction,
%and in section 5, we conclude and describe
%some areas for future work.

\section{Background}

In this section, we briefly discuss
some traditional hierarchical clustering algorithms
and provide some
background knowledge on latent variable modeling.

\subsection{Traditional hierarchical clustering}
Traditional hierarchical clustering algorithms
can be broadly divided into two categories:
\emph{agglomerative} (bottom-up) and \emph{divisive} (top-down).
In agglomerative clustering, the hierarchy
is built by iteratively merging clusters
that are most similar to each other, forming
larger and larger clusters at each level
of the tree.
In divisive clustering,
the hierarchy is built by recursively
splitting the data from the top,
forming a new pair of nodes with
each split \citep{Hastie2009}.

The input to hierarchical clustering algorithms
is a dataset, $\dataset$.
The output is a $\tree$,
a rooted tree with $\numdata$
labeled leaves,
each corresponding to one of the data points.
Typically, the tree is binary
and internal nodes correspond
to groupings of the leaves
of the tree.

An agglomerative clustering algorithm is fully specified
by a \emph{dissimilarity function} and a \emph{linkage criterion}.
A dissimilarity function,
$d(\x_i, \x_j)$ measures
how different two data points $\x_i$ and $\x_j$ are from each other;
for example, we may choose Euclidean distance
if our data are real valued vectors.
A linkage criterion $D(A, B)$ measures how different two clusters
$A$ and $B$
are from each other in terms of the pairwise
dissimilarities between the points in each cluster.
A popular linkage criterion is
\emph{single linkage}, which defines
cluster dissimilarity
as that
of the closest points in
each cluster,
i.e.
\begin{align}
  D_{SL}(A, B) = \min_{i\in A, j \in B} d(\x_i, \x_j)
\end{align}
Another common linkage function is \emph{average linkage},
where cluster dissimilarity is
the mean dissimilarity between
all pairs of points in each cluster.
\begin{align}
  D_{AL}(A, B) = \frac{1}{|A||B|}\sum_{i\in A, j \in B} d(\x_i, \x_j)
\end{align}
Finally, in \emph{complete linkage}, the cluster
dissimilarity
is that of the furthest points in each cluster,
the opposite of single linkage.
\begin{align}
  D_{CL}(A, B) = \max_{i\in A, j \in B} d(\x_i, \x_j)
\end{align}

Agglomerative clustering begins by instantiating
a singleton cluster $\{n\}$ for each $\x_n$, which
will be the leaves of our output hierarchy.
Every iteration of the algorithm,
we find
the two least dissimilar clusters $A$ and $B$
according to the linkage criterion
and merge them, producing a new cluster.
After $N - 1$ iterations, we are left
with a single cluster containing
all the data, and the process
creates a binary tree where each internal node
correspond to one of the merge operations.

See examples of clusterings with each
linkage criterion
in \autoref{fig:dendrograms}.

\begin{figure}[t]
  \includegraphics[width=\textwidth]{img/trees/dendrograms}
  \caption{Hierarchies over human tumor data produced by agglomerative clustering
    with three different linkage criterions.
  Image from \citet{Hastie2009}.}
\label{fig:dendrograms}
\end{figure}

In divisive clustering, the approach is flipped.
We begin with a cluster containing
every data point, corresponding
to the root of the output tree.
We recursively
partition the cluster, until
we are left with singleton clusters
at each leaf.
Partitioning a cluster
typically corresponds to solving
an optimization problem, such as minimizing
a cost function of a split.
For example, we could partition
a cluster using $k$-means, with $k = 2$,
optimizing the $k$-means cost function.
Alternatively in spectral clustering methods,
we create a similarity graph, $G$,
given a similarity function $s(\x, \x^\prime)$.
$G$ is an undirected, weighted graph
where there is a node for each data point,
and the weight for each edge $(i, j)$ is
$s(\x_i, \x_j)$ \citep{VonLuxburg2007}.
Bipartitioning a cluster
corresponds to finding a cut in its similarity graph,
and intuitively a good cut would avoid
edges between similar points.
Finding a good partition thus corresponds
to finding a minimum cut on $G$
for such cost functions as
RatioCut \citep{Hagen1992} and Ncut \citep{Shi2000}.

\iffalse
\subsection{Bayesian learning}
Bayesian learning is a probabilistic
approach to machine learning.
In Bayesian learning, our data
are represented as observed random variables,
which are generated conditionally
on a set of unobserved latent variables.
This is sometimes called a latent variable model.

In a latent variable model,
we have observed dataset $\dataset$
and assume there is some
unobserved set of latent variables $\globals$
responsible for generating $\dataset$.
The latent variables $\globals$
are distributed according
to a \emph{prior distribution} $\p(\globals)$,
and our observed data
are generated by a conditional
distribution $\p(\dataset \given \globals)$, also called
the \emph{likelihood}.
We are typically interested in the
\emph{posterior distribution},
$\p(\globals \given \data)$, which is the distribution
over latent variables given data.
Computing the posterior distribution is called \emph{inference}.
The posterior distribution can be calculated
via Bayes rule:

\begin{align}
  \p(\globals \given \dataset) = \frac{\p(\dataset \given \globals)\p(\globals)}{\p(\dataset)} = \frac{\p(\dataset \given \globals)\p(\globals)}{\int_\globals \p(\dataset \given \globals)\p(\globals)}
\end{align}

The choice of prior distribution and likelihood model
affect our ability to perform inference.
If our prior distribution is \emph{conjugate}
to our likelihood model, the posterior distribution
will be of the same form as the prior
and can be expressed analytically.
For example, if our prior over parameters
is Gaussian and likelihood model is Gaussian,
the posterior distribution will also be Gaussian.
This setup is common in classification problems
for real-valued data.
Similarly, if our prior over parameters is Dirichlet,
and the likelihood model is Multinomial,
the posterior distribution will be Dirichlet.
This setup is common in classification problems
when our data are bags-of-words.
In the general setting, however,
it is often impossible to compute an
analytical form for the posterior distribution.

After performing inference and obtaining
the posterior distribution $\p(\globals \given \dataset)$,
we are often tasked with prediction,
or calculating a distribution over a new, unobserved
data point. Let $\x^*$ be an unobserved test point.
The predictive distribution is defined
as $\p(\x^* \given \dataset)$, and can be obtained by
marginalizing the posterior distribution:

\begin{align}
  \p(\x^* \given \dataset) = \int_\globals \p(\x^*, \globals \given \dataset)d\globals =  \int_\globals \p(\x^* \given \globals, \dataset)\p(\globals \given \dataset)d\globals
\end{align}

%Statistical models are often represented
%as a directed graph, where nodes
%are random variables and edges
%represent dependency, called a \emph{graphical model}.

A classic example of a latent variable model
is the Gaussian mixture model (GMM).
GMM's are typically used in flat clustering scenarios,
and as such, we assume a fixed amount of clusters $K$.
A latent variable responsible for generating
data in a GMM are
the $K$ cluster centers, denoted by the set
$\bm{\mu} = \{\mu_1, \ldots, \mu_K\}$.
Each cluster has a weight, or a prior probability
of a data point belonging in the cluster,
represented by the vector $\bm{\pi} = \{\pi_1,\ldots,\pi_K\}$.
Each data point is generated by sampling a cluster
assignment $\z$ from the probability distribution $\bm{\pi}$,
and sampling a vector from a Gaussian distribution
centered at $\mu_\z$.
The graphical model for a GMM is pictured in \autoref{fig:gmm-model}.

\begin{figure}[H]
  \centering
  \includestandalone[width=0.2\textwidth]{tikz/gmm}
  \caption{A graphical model representation of a Gaussian mixture model. $\x_n$ is
  one of $\numdata$ data points and $\z_n$ is its cluster assignment. $\bm{\pi}$ are the weights
  for each cluster, and $\bm{\mu}$ are the centers of each cluster.}
\label{fig:gmm-model}
\end{figure}

Unfortunately, in the GMM,
and in many other latent variable models,
the posterior distribution $\p(\bm{\mu}, \bm{\pi} | X)$
is impossible to compute analytically.
Thankfully, there are plenty of
algorithms to approximate the posterior distribution,
such as
Markov chain
Monte Carlo (MCMC), or variational
inference.

Often times, we are not interested in the posterior
distribution itself, but the settings of the latent variables
that result in the highest posterior probability, i.e.
$\globals^* = \text{argmax}_\globals \p(\globals | X)$.
$\globals^*$ is called the \emph{maximum a posteriori}
and can be approximated
by such algorithms as
expectation-maximization \citep{Dempster1977}.
\fi

\section{The generative process}

Bayesian nonparametric hierarchical clustering (BNHC)
is a latent variable model where
data is generated in
in two stages.
First, a tree is generated by a
\emph{tree prior}.
Conditioned on the sample
from the tree prior,
data is generated with
a \emph{tree likelihood model}.

The simplest tree priors
model \emph{cladograms},
or rooted binary trees
with data at the leaves,
but very often the trees are imbued with
additional information.
For example:

\begin{enumerate}
  \item An ordering on the internal nodes of the tree.
    Typically, the root is given the lowest number and
    each node will have a higher number than its parent.
  \item Times associated with each node.
    Nodes higher up in the tree will typically have
    earlier times, creating an evolutionary
    interpretation to the hierarchy. Cladograms
    with times associated at each node
    are also called \emph{phylogenies}.
\end{enumerate}

If we are only interested in tree structure,
not ordering or node times, we can simply discard
all auxiliary information at the very end.

\begin{figure}[H]
  \centering
  \includegraphics{tikz/lvm}
  \caption{The graphical model used
  in Bayesian hierarchical clustering. $\tree$
  is a tree, sampled from a tree prior distribution.}
  \label{fig:bhc-gen-model}
\end{figure}

Formally,
let $\tree$ be a class
of tree structures (e.g. ordered cladograms),
and let $\p(\tree \in \trees)$
be a tree prior
for $\trees$.
We sample parameters $\globals$ for tree likelihood model,
conditional on the tree structure.
This involves assigning
latent parameters to each internal node in the $\tree$
and specifying a stochastic process in which they are generated.
Conditioned on a particular
structure $\tree$ sampled from $\p(\tree)$
and tree likelihood parameters $\globals$, dataset
$\dataset$
is generated
via probability distribution $\p(\dataset \given \tree, \globals)$.
This is visualized as a graphical model in \autoref{fig:bhc-gen-model}
and concisely expressed as follows:

\begin{align*}
  \tree &\sim \p(\tree) \\
  \globals &\sim \p(\globals \given \tree) \\
  \dataset &\sim \p(\dataset \given \tree, \globals)
\end{align*}

Inference involves computing the posterior
distribution over trees and parameters,
$\p(\tree, \globals \given \dataset)$, but we are sometimes interested
in the
posterior marginal distribution $\p(\tree \given \dataset)$.
Both of these distributions are typically
impossible to compute
analytically as marginalizing over $\tree$
is intractable.
In practice, most methods use Markov chain Monte Carlo
methods to sample $\p(\tree, \globals \given \dataset)$
and $\p(\tree \given \dataset)$.

In this section, we will cover two broad
classes of tree priors and touch on
some extraneous models. We will then explain
a few tree likelihood models, and
finish by explaining how to perform
inference in Bayesian hierarchical clustering.

\subsection{Tree priors}

Broadly speaking,
tree priors can be broadly broken down into
\emph{coalescent} models
and
\emph{diffusion} models.
Both of them share a core characteristic
of being sequential models.
In coalescent models, the tree is built sequentially
from the bottom up. We begin with each
data point in its own cluster,
and sequentially merge clusters until
there is only one. This idea is
very similar to agglomerative clustering.
Diffusion models take an inductive approach,
where we begin with a hierarchy over a single
data point, and sequentially add data to the hierarchy until
we have a tree with $\numdata$ leaves,
a fundamentally different paradigm.
Shared in both types of models is the property
of
\emph{exchangeability}.
A sequence of random variables, $\x_1, \ldots, \x_\numdata$,
is considered exchangeable
if their joint distribution
is invariant to permutations of the variables.
Exchangeability often enables computationally efficient
inference algorithms.

\textbf{Coalescent models:}
coalescent modeling was developed
in the early 1980s by John Kingman,
and achieved success in the field
of population genetics
\citep{Kingman1982}.
Coalescent models assume a
continuous time process,
wherein individuals of a population
are traced backwards in time
through their ancestry until
they all share a single common ancestor.
In terms of hierarchical clustering,
the individuals in the population
correspond to a dataset $\dataset$,
and their
ancestry backwards in time
is a hierarchy.

The classic coalescent model is
Kingman's coalescent \citep{Kingman1982}.
It assumes a countably infinite population
but has a consistency property
which allows it to be described
in terms of its marginal distribution
over finite populations.

Consider a dataset $\dataset$ with $\numdata$ points,
which we will call ``individuals''.
In Kingman's coalescent,
our current population exists at time $t = 0$,
and each individual in the population has a single
parent in the previous generation, which has its own parent,
continuing backwards in time until
$t = -\infty$.
At some time in the past,
the lineage of any two individuals will ``coalesce''
when they share an ancestor.
The lineages of the members of the population
can be concisely described
by the
\emph{genealogy} function, $\pi(t)$,
that maps between time $t$
and a partition of $\{1, \ldots, \numdata\}$
that groups individuals together
if their lineages have coalesced at time $t$.
$\pi(0) = \{\{1\}, \{2\},\ldots,\{\numdata\}\}$
represents
the current time, when no
lineages have coalesced.
$\pi(-\infty) = \{\{1, 2, \ldots, \numdata\}\}$
is the partition of all
individuals into a single group,
when they all share a common ancestor \citep{Teh2008}.

Kingman's coalescent
is a probability distribution
over genealogy functions $\pi(t)$
for populations of size $\infty$,
and the marginal distribution for
populations of size $\numdata$ is called the $\numdata$-coalescent.
The $\numdata$-coalescent
is a continuous-time
Markov process
over the space of partitions
of size $\numdata$,
starting at time $t = 0$ with
$\pi(0) = \{\{1\}, \{2\},\ldots,\{N\}\}$,
going backwards until  $t = -\infty$,
with
$\pi(-\infty) = \{\{1, 2, \ldots, N\}\}$.

\begin{figure}[H]
  \centering
  \includegraphics[width=\textwidth]{img/trees/coalescent}
  \caption{A sample from Kingman's coalescent. $\z$ is
    the common ancestor for data $\x_{1:4}$. Each $y_C$
  represents the common ancestor for data subset $C$.
  The axis represents time, and each $\delta_i$ represents elapsed time.
  $\pi(t)$ is
  the genealogy function. Image from \citet{Teh2008}.}
\label{fig:coalescent}
\end{figure}

The $\numdata$-coalescent
can be broken down into two
independent components.
The first is the \emph{jump process},
which models the discrete transitions
between partitions
before and after coalesce events.
The second is the \emph{time process},
which models the times in the past
at which coalesce events happen.
The jump process is very simple.
Each lineage has an equal probability of merging
with any other lineage. Thus, the
jump process is a Markov chain
where the transition matrix is uniform
for each pair in a partition coalescing.
The jump process has $\numdata - 1$ transitions, starting
with the partition with singleton groups
for each data, and ending with the
partition with all data in a single group.
The time process produces a series of times
$t_{\numdata - 1} < t_{\numdata - 2} < \cdots <  t_1$ when coalesce events happen,
ending with $t_0 = 0$.
The first step in the jump process
corresponds to coalesce time $t_1$, and so on.
Kingman's coalescent assumes
that each pair of lineages merges at a constant rate of $1$,
i.e.\
a pair lineages will eventually merge at
$t \sim \text{Exp}(1)$ where $\text{Exp}$ is the exponential distribution.
Given a set of $m$ lineages,
a pair of them will merge at
$t \sim \text{Exp}(\binom{m}{2})$.
Let the elapsed time before each jump $i$ be
$\delta_i = t_{i - 1} - t_i$
We start with a population of size $\numdata$,
and it decreases by $1$
in each step of the jump process.
Thus, $\delta_i \sim \text{Exp}(\binom{\numdata - i + 1}{2})$.
After sampling all the $\delta_i$'s, we can easily
compute the coalesce times,
completing the distribution over genealogy functions.
An example sample from Kingman's coalescent can be
seen in \autoref{fig:coalescent}.

\begin{figure}[H]
  \centering
  \includegraphics[width=0.8\textwidth]{img/trees/balanced}
  \caption{For the balanced tree on the left, there are two
  possible orderings in which clusters were merged, whereas there is only one possible ordering for the unbalanced tree on the right. Image taken from \citet{Boyles2012}.}
\label{fig:balanced}
\end{figure}

A genealogy sampled from Kingman's coalescent
is an ordered cladogram with times associated with every internal node.
The ordering comes from the order in which
internal nodes were created in the jump process.
The induced distribution over just ordered trees
is in fact uniform \citep{Teh2008}.
However, we often don't need order information
in hierarchical clustering.
\citet{Boyles2012} computes the induced distribution over
unordered rooted binary trees.
For a given unordered tree with $\numdata$ leaves, $\tree$,
there are $\frac{(\numdata - 1)!}{\prod_{\n = 1}^{\numdata - 1}m_\n}$ possible orderings
where $m_\n$ is the number of internal nodes in the subtree
of $\tree$ indexed by $\n$.
Intuitively, there
are several possible orderings to create a perfectly balanced binary tree,
but only one for an extremely unbalanced binary tree (see \autoref{fig:balanced}).
Thus, Kingman's coalescent induces a prior distribution
over unordered trees
that favors balanced trees over unbalanced trees,
and
marginalizing out the ordering of internal
nodes results in this distribution, also called
the time-marginalized coalescent \citep[TMC; ][]{Boyles2012}.

Kingman's coalescent assumes constant merge rates
for each pair of lineages in a genealogy.
Although this is an intuitive assumption,
there have been several papers generalizing Kingman's coalescent
to both $k$-ary trees, and more complex coalesce rates.
For example, in \citet{Pitman1999}, Pitman introduced the
$\Lambda$-coalescent, extending Kingman's coalescent
to have ``multiple collisions'', supporting
$k$-ary trees.
Rather than the coalesce rate
being $1$, it is instead $\lambda_i^k$
where $k$ is the maximum number of lineages
that can merge in a single event and $i$ is the total number
of lineages at the time of the event, resulting in $\delta_i \sim \text{Exp}(\lambda_i^k)$. $\lambda_i^k$ is calculated as
\begin{align}
  \lambda_i^k = \int_0^1 \gamma^{k - 2}(1 - \gamma)^{(i-k)}\Lambda(d\gamma)
\end{align}
where $\Lambda$ is a finite measure on $[0, 1]$. The
$\Lambda$-coalescent is identical to Kingman's coalescent
when $k = 2$ and $\Lambda$ is the Dirac delta.
Setting $\Lambda$ to a beta measure
results in the aptly named beta-coalescent \citep{Hu2013}.
A description of such coalescent rate functions
can be seen in \citet{Aldous1999}.

\textbf{Diffusion models:}
diffusion models adopt a different paradigm for generation.
In diffusion models, we begin with a
trivial tree over just one data point. We then
grow the tree by iteratively attaching new data
to branches in the tree, eventually creating
a tree with $\numdata$ leaves.
Similar to coalescent models, there is an underlying
continuous time process responsible for
the tree structure.

The first diffusion model, the Dirichlet diffusion tree (DDT),
was proposed in \citet{Neal2003}.
The DDT is an exchangeable model that models a sequence of data
$\x_1, \x_2,\ldots,\x_N$.
There is an underlying continuous time process
that lasts from time $t = 0$ to $t = 1$
In essence, each data point is generated in sequence by
a random walk beginning at $t = 0$,
reaching a final value at $t = 1$.
Each data point initially follows the previous
random walks, but eventually diverges and continues independently.

Let $\x_\n(t)$ be the value of data point $\x_\n$
at time $t$, defining a ``path'' associated
with each point from its start $\x_\n(0)$ to its final (observed) value $\x_\n(1)$.
In addition, the path of each data point $\x_\n(t)$ is conditioned
on all previous paths $\x_1(t), \x_2(t), \ldots, \x_{\n - 1}(t)$.
This process, when completed for all $\numdata$ data points,
induces a binary tree over the data.

The path for the first data point $\x_1$
is a Brownian motion
beginning at the origin.

\begin{align}
  \x_1(0) &= 0 \\
  \x_1(t + dt) &= \x_1(t) + \N(0, \sigma^2Idt)
\end{align}

The path for the second data point $\x_2(t)$
is exactly $\x_1(t)$ until
at some time $\tree$ it diverges,
creating a branching point for
$\x_1(t)$ and $\x_2(t)$.
$\tree$ is sampled according to an \emph{acquisition function}
$a(t)$ and
after divergence, $\x_2(t)$ is an
independent Brownian motion.

Now consider the inductive case.
We have already sampled $\n - 1$
paths, which form a binary tree
with internal nodes corresponding to
divergence events.
$\x_n(t)$ initially follows the same
path as the previous points,
picking branches to follow proportional
to the number of points that followed
the branch previously.
This scheme is called \emph{path reinforcement}
and encourages
big subtrees to grow even bigger.
If $\x_1(t)$ is traversing a path
followed by $m$ previous paths,
the probability that $\x_1(t)$ will
diverge along an infinitesimally small
interval $d$ is
$a(t)dt/m$.
In practice we work with the cumulative
divergence function $A(t) = \int_0^t a(u)du$.
Given an interval of time $(s, t)$,
that lies on a single branch of the DDT
that was previously traversed by $m$ paths,
the probability that the next point
does not diverge on the interval
is $e^{(A(s) - A(t))/m}$.
The acquisition function is chosen such that
$a(1) = \infty$. This guarantees that
all data points must diverge
before $t = 1$.
For a visualization of a DDT
for 1-dimensional data, see \autoref{fig:ddt-vis}.
Possible choices of acquisition function
are $a(t) = \frac{c}{1 - t}$ or $a(t) = b + \frac{d}{1 - t^2}$,
where $b$, $c$, and $d$ are constants.

\begin{figure}[H]
    \centering
    \begin{subfigure}[b]{0.45\textwidth}
        \centering
        \includegraphics[width=\textwidth]{img/trees/ddt}
    \end{subfigure}
    \hfill
    \begin{subfigure}[b]{0.45\textwidth}
        \centering
        \includegraphics[width=\textwidth]{img/trees/pydt}
    \end{subfigure}
    \caption{
      On the left is a sample Dirichlet diffusion tree.
      Black dots correspond to nodes in the tree. The image is taken from \citet{Vikram2016}.
      On the right is a sample from a Pitman-Yor diffusion tree with 4 points.
      The image is taken from \citet{Knowles2015}.
    }
    \label{fig:ddt-vis}
\end{figure}

The DDT, as proposed in \citet{Neal2003},
jointly models the tree prior and tree likelihood model,
as in it jointly samples tree structure
and real-valued data associated with the tree.
However, we can consider the same
path reinforcement model, without the Brownian motion
to obtain a prior over ordered cladograms
with times for each node.
We can obtain a sample from the DDT and just ignore the
internal values at each node where divergence occurred.

The DDT thus induces a prior over ordered cladograms
with times.
Unlike Kingman's coalescent, however, the DDT
does not, in general, induce a uniform prior over ordered cladograms.
This is due to the path-reinforcement element
of the prior and the choice of acquisition function.
It is worth noting that the Dirichlet diffusion tree has achieved
great success in the area of density estimation \citep{Adams2008}.

An extension to the Dirichlet diffusion tree
is the Pitman-Yor diffusion tree (PYDT),
which removes the DDT's restriction to just binary trees
\citep{Knowles2015}.
In a Pitman-Yor diffusion tree, the probability
of a data point diverging in an infinitesimally small
interval $dt$ is
\begin{align}
  \frac{a(t)\Gamma(m - \beta)}{\Gamma(m + 1 + \alpha)}
\end{align}
where $\alpha$ and $\beta$ are parameters. Setting $\alpha = \beta = 0$,
results in same probability as in the DDT.
In the PYDT, a data point can not only
diverge at any point in time,
but can also diverge at a branching point
in the tree, where in the DDT
a data point would always pick a branch.
When a data point reaches a branching point,
it picks branch $k$
with probability
\begin{align}
  \frac{b_k - \beta}{m + \alpha}
\end{align}
where $b_k$ is the number of paths
that previously traversed $k$
and $m$ is total the number of paths
that reached the branching point.
The probability that the data point
diverges, creating a new branch, is
\begin{align}
  \frac{\alpha + \beta K}{m + \alpha}
\end{align}
where $K$ is current number of branches.
This scheme allows for an arbitrary amount of
branching at each internal node of the tree,
but can be tuned by settings of $\alpha$ and $\beta$.
Note again that when $\alpha = \beta = 0$,
the generation scheme matches the DDT.

\subsection{Generalizations of coalescent and diffusion models}

The simplest distribution over (unordered) cladograms
is the uniform.
Kingman's coalescent is one step away,
as it
induces the uniform distribution
over ordered cladograms,
a distribution also called
the Yule model \citep{Harding1971}.
Marginalizing over possible orderings
results in the TMC \citep{Boyles2012}, a distribution over unordered cladograms biased towards balanced trees.

Generalizations of Kingman's coalescent
include the $\Lambda$-coalescent \citep{Pitman1999},
which extends coalescents to multifurcating trees,
and the Aldous $\beta$-splitting model \citep{Aldous1996}.
A more recent model, the Gibbs fragmentation tree,
is a generalization of the $\beta$-splitting model
to multifurcating trees,
and is considered the most general Markovian distribution over trees \citep{McCullagh2008}.

The diffusion models on the other hand,
include far more parameters.
The choice of acquisition function,
along with the choice of $\alpha$ and $\beta$ in the PYDT,
result in a wider variety of distributions.
The PYDT's induced distribution over tree
structure is, in fact, a specific case
of the Gibbs fragmentation tree.

An alternate generalization is the fragmentation-coagulation process (FCP),
which is a distribution over partition-valued
Markov chains \citep{Teh2011}.
Instead of partitions being
split up or merged monotonically as in
diffusion and coalescent models, the
FCP models both split and merge
transitions in the Markov chain, encompassing
both coalescent and diffusion models.

All of these relationships are pictured in \autoref{fig:familytree}.

\begin{figure}[H]
  \includegraphics[width=\textwidth]{img/trees/FamilyTree}
  \caption{A graph visualization of the relationships between various
  tree priors. Directed arrows represent
  specific cases of a distribution or model.}
\label{fig:familytree}
\end{figure}

\subsection{Other priors}

A popular line of work
follows extensions of the Chinese restaurant
process (CRP) and other related Bayesian nonparametric objects.
The CRP is a distribution
over partitions of $\{1, \ldots, N\}$ \citep{Aldous1985}.
Its name derives
from the nature of the stochastic process
by which partitions are sampled.
Imagine an empty restaurant with
a countably infinite number of tables
labeled $1, 2, \ldots$
with a line of customers waited to be seated.
The first customer will always pick
table $1$.
Let $\alpha$ be a hyperparameter.
The $n + 1$-th customer will pick the
the $i$-th table with probability
\begin{align}
    \frac{c_i}{n + \alpha}
\end{align}
where $c_i$ is the number
of customers already sitting at
table $i$ or
sit at a new, unoccupied table with probability
\begin{align}
    \frac{\alpha}{n + \alpha}
\end{align}
After $\numdata$ customers have been seated,
we have a partition over $\{1,\ldots, N\}$,
where customers are grouped
by the table they are seated at.

\begin{figure}[t]
  \includegraphics[width=\textwidth]{img/trees/ncrp}
  \caption{A visualization of a sampled from
  a nested Chinese
  restaurant process (NCRP). Only three levels
  for 5 customers are shown.
  Image from \citet{Blei2010}.}
\label{fig:ncrp}
\end{figure}

The CRP is an exchangeable model
and has a reinforcement property, where
tables with more people attract
even more people. It is
also the basis of the nested Chinese
Restaurant Process (nCRP) model
used for hierarchical clustering \citep{Blei2010}.
The nCRP extends the analogy to
an infinite chain of restaurants.
After a customer goes and sits
at the first Chinese restaurant,
they are directed to yet another
Chinese restaurant where the process
repeats ad infinitum. This defines
a structure where $\numdata$ customers, or data,
are distributed
across infinitely deep leaves of a tree.
Although the tree is infinite,
it can be represented in terms of its
finite induced distribution over
how the partition of customers is split
at each level of the tree.
This results in a distribution over rooted trees
with arbitrary branching factor and ordering over internal nodes.
See \autoref{fig:ncrp} for a visualization of the NCRP.
The NCRP
is closely related to the nested Dirichlet process (NDP),
the underlying de Finetti measure for the NCRP.
An extension
of the NCRP and NDP is the nested hierarchical Dirichlet
process (NHDP) \citep{Paisley2014}. Whereas each datum in the NDP and NCRP
are each represented as an infinitely deep path
down the tree, each datum in the NHDP
is represented as a \emph{mixture} of
infinitely deep paths down the tree.
The NCRP, NDP, and NHDP have been used to success
in topic modeling, finding hierarchies over
a corpus of documents.

The tree-structured stick-breaking process (TSSB)
is an alternate model
where data are allowed to live at any node
in the tree, as opposed to just leaves \citep{Adams2010}.
At the core of the TSSB is the
stick-breaking process, a Bayesian nonparametric object
closely related to the CRP.
The stick breaking process
iteratively carves up the unit interval
into smaller and smaller pieces,
resulting in an infinite amount of ``sticks'',
whose lengths sum up to one.
Let $\beta_i \sim \text{Beta}(1, \alpha)$ for
$i \in 1,2,\ldots$.
These will be fractions of the remainder
of the stick that we will carve up.
Now, let the ``sticks''
be $t_1 = \beta_1$ and $t_i = \beta_i \prod_{j = 1}^i (1 - \beta_j)$.
We now have a sequence of
random variables, $t_1, t_2, \ldots$
such that $\sum_{i = 1}^\infty t_i = 1$,
which we can treat as a distribution over
an infinite amount of events.
The TSSB is an extension of the stick-breaking process
to generate a distribution
over infinitely deep and wide hierarchies.
Associated with each internal node in the tree
is a stick breaking process,
which acts as a distribution over an infinite
amount of branches out of the node.
Data are generated by starting at the root
and sampling a branch. At any point,
data have a probability of stopping at a given node,
which distributed as a stick breaking process
over each possible path heading down the tree.
Although the tree is infinite, it induces a distribution
over trees where $\numdata$ data live across every node
in the tree, a fundamentally different tree structure
from the previous ones.

\subsection{Tree likelihood models}

Given a sample from a tree structure prior,
we now need a scheme to sample
hierarchically structured data.
Let the sample from the tree
structure prior be $\tree$.
Assume, for now, $\tree$ is
an rooted tree,
with times associated with each internal node (ordering does not matter).
The standard tree likelihood model is a diffusion model,
where there is a parameter $\globals_\n$ for each internal
node $\n$, which are generated from the root downwards.
Let $\globals_0$ be the parameter associated with the root node,
which will be sampled according to a prior distribution $\p_0(\globals)$.
We then define a transition kernel, $T(\globals' \given \globals)$.
The parameter for each node is generated conditionally
given its parent parameter in the tree,
i.e. for node $v$ and its parent $u$,
$\globals_v \sim T(\cdot \given \globals_u)$.
Data at leaves are also generated in this fashion.
We now enumerate some transition kernels
which accomodate different data types and tree structures.

\begin{enumerate}
    \item \textbf{Generalized Gaussian diffusion.} If our data
    is continuous and $\d$-dimensional, we can associate
    a latent vector $\globals_\n \in \R^\d$ with each internal node
    $\n$ in the tree. Let $u$ be an internal node
    and $v$ one of its children and
    let $\delta_{uv}$ be the elapsed time between $u$ and $v$.
    We sample $\globals_v \sim \N(\globals_u, \Sigma\delta_{uv})$,
    where $\Sigma$ is a positive definite covariance matrix.
    The root node value is assigned a prior
    $\N(\mu_0, \Sigma_0)$.
    In the case where there are no times associated with each node,
    we can assume elapsed time between nodes is always 1,
    and the hyperparameters in the model would be
    $\Sigma$, $\mu_0$, and $\Sigma_0$.
    This approach is used in \citet{Neal2003}, \citet{Teh2008}, \citet{Knowles2015},
    \citet{Adams2010}, \citet{Boyles2012} and \citet{Hu2013}.
    \item \textbf{Multinomial diffusion.} If our data
    consists of several categorical variables,
    we can model it with a transition matrix, $\tree$.
    For categorical feature $c$, let $T_c$
    be the row in $\tree$ for feature $c$.
    Let $\delta_{uv}$ be the elapsed time between parent $u$ and child $v$.
    As suggested by
    \citet{Teh2008},
    $T_{c} = e^{-\lambda_{c}\delta_{uv}}I + (1 -  e^{-\lambda_{c}\delta_{uv}})q_{c}^T\bm{1}$
    where $\lambda_{c}$ is a hyperparameter for the evolution rate for feature
    $c$, $q_{c}$ is a hyperparameter for the equilibrium
    distribution of $c$, and $\bm{1}$ is a vector of ones.
    \item \textbf{Dirichlet-Multinomial diffusion.}
    This tree likelihood model is for
    trees without times associated with each internal node.
    When data consists of counts of discrete events,
    such as the bag of words in topic modeling,
    leaves can be described as sampling
    a multinomial distribution.
    The transition kernel for
    just internal nodes as suggested by \citet{Adams2010} is $T(\globals' | \globals) = \text{Dirichlet}(\kappa\globals)$
    and the prior is $P_1(\globals) = \text{Dirichlet}(\kappa\bm{1})$,
    where $\kappa$ is a concentration hyperparameter and $\bm{1}$ is a
    vector of ones.
    Leaves are sampled via a multinomial distribution
    with its parent's vector as its parameter.
\end{enumerate}

\subsection{Inference}

We are interested
in the posterior distribution
over hierarchies given data.
Typically, this distribution
is intractable to compute
analytically, so approximate
methods are required.
A popular approach used
in \citet{Neal2003}, \citet{Knowles2015}, and \citet{Boyles2012}
is the Metropolis-Hastings
algorithm,
a Markov chain Monte Carlo
method where samples
are used as an approximation
to the posterior distribution.

In Metropolis-Hastings,
we are given a target
distribution, $\p(\x)$, and
define a \emph{proposal distribution}, $\q(\x^\prime \given \x)$.
A Markov chain is initialized with state $\x_0$.
In each iteration of the algorithm
we use current state $\x_t$
to sample a candidate $\x^\prime$ from $\q(\x^\prime \given \x_t)$.
and then calculate
the acceptance ratio,
\begin{align}
    \alpha = \frac{\p(\x^\prime)\q(\x_t \given \x^\prime)}{\p(\x_t)\q(\x^\prime \given \x_t)}
\end{align}
If $\alpha \ge 1$, we accept,
setting $\x_{t + 1} = \x^\prime$.
If $\alpha < 1$, we accept
with probability $\alpha$, and reject
otherwise, setting $\x_{t + 1} = \x_t$.
Provided some conditions on $\q(\x^\prime \given \x)$,
the Markov chain's stationary distribution
will be $\p(\x)$.

In BNHC, the probability distribution
of interest
is the posterior distribution $\p(\tree, \globals \given \dataset)$.
The state of a Markov chain in Metropolis-Hastings is therefore
a tuple $(\tree, \globals)$.
A proposal distribution $\q(\tree^\prime, \globals^\prime \given \tree, \globals, \dataset)$ would need to jointly
sample $\tree$ and $\globals$.
A simple strategy is to have two proposal distributions: a
tree proposal $\q(\tree^\prime \given \tree)$
and parameter proposal $\q(\globals^\prime \given \tree^\prime, \globals)$.
We would sample $\tree^\prime$ first,
and then we would sample $\globals^\prime$
additionally conditioned on $\tree^\prime$.
Alternatively, if we're only interested
the posterior distribution over trees,
we can often marginalize out $\globals$, either
analytically or via belief propagation,
resulting in the posterior marginal
distribution $\p(\tree \given \dataset)$.

\begin{figure}[t]
  \includegraphics[width=\textwidth]{img/trees/spr}
  \caption{Visualization of a subtree-prune and regraft move for a
  tree with four leaves. Image from \citet{Vikram2016}.}
\label{fig:spr}
\end{figure}

Designing a tree proposal distribution
involves modifying some tree $\tree$
randomly to form a new tree $\tree^\prime$.
One of the most popular proposals
is the \emph{subtree-prune and regraft} (SPR) move, proposed by \citet{Swofford1990}.
An SPR move consists of a \emph{prune} followed by a \emph{regraft}.
Let $\tree$ be a tree.
Let $s$ be a non-root
node in $\tree$ selected uniformly at random
and $S$ be its corresponding subtree.
We first \emph{prune}
$S$ from $\tree$ by
remove $s$'s parent $p$ from the tree,
replacing $p$ with $s$'s sibling.
To regraft $p$ to $\tree$,
we first select a branch in $\tree$,
$(u, v$), where $u$ is the parent of $v$.
$S$ is re-attached to $\tree$
by creating a new node $p$
with parent $u$ and children $v$ and $s$,
creating a new tree $\tree^\prime$.
This process is visualized in \autoref{fig:spr}.

The branch selected in the regraft move
is often sampled from a distribution
related to the tree prior.
For example, if we used a DDT,
we have a posterior distribution
over branches where a new data point would
diverge. If such a distribution
does not exist, the branch can simply
be selected uniformly at random.

Two alternatives to the SPR move
moves are the leaf move,
which restricts the SPR move
to just leaf nodes,
and nearest-neighbor interchange
moves, which interchange
two pairs of subtrees.
It is worth noting
that
the worst case mixing rate of
a Markov chain using
leaf moves to sample
the uniform distribution over
$\numdata$-leaf cladograms is $O(\numdata^3)$ \citep{Aldous2000}.

Sampling the parameters of
the tree likelihood model is much simpler
than sampling the tree prior.
The parameters $\globals$ often
have conditional dependence
structure that allows simple
ancestral sampling or Gibbs sampling.
For example, in the DDT and Kingman's coalescent,
each internal node in the tree stores
a latent vector, representing the intermediate value
of data higher up in the tree.
If the likelihood model is
generalized Gaussian diffusion,
all conditional distributions are Gaussian.
We can thus Gibbs sample each of the
latent vectors.

Metropolis-Hastings is perhaps the
simplest method to sample a general
BNHC model. However, alternative methods
have been proposed,
such as using slice sampling in \citet{Adams2010} and \citet{Boyles2012},
sequential Monte Carlo in \citet{Teh2008} and \citet{Hu2013},
Gibbs sampling in \citet{Blei2010},
and variational inference in \citet{Paisley2014}.
Furthermore, there are greedy algorithms
to approximate a MAP estimate
to the posterior distribution
of trees given data \citep{Teh2008, Hu2013}.


\subsection{Finding strong rules using the Apriori algorithm}

\subsubsection{Finding frequent patterns}

We can use search algorithms to find frequent patterns. Starting at the empty set.

\subsubsection{Apriori algorithm}

Breadth first search to generate candidate set of itemsets with support above some value.

Start with a \(1\)-itemset, and increase \(k\) once done.

Once we have found a frequent pattern, we can immediately identify other frequent patterns associated with it.

We can do this by looking at confidence, not support.


\lab{Job Search Model}{Job Search Model}

\section*{Discrete Choice (Threshold) Problems}\label{SecDiscrChoice}

One powerful application of dynamic programming is
to models that have both continuous and discrete state variables. These models are sometimes referred to as
discrete choice problems or optimal stopping problems.  Examples include models of employment that involve both
the choice of whether to work and how much to work, models of firm entry and exit that involve the choice of both
whether to produce and how much to produce, and models of marriage that involve the choice of whether to date
(get married or keep dating) and how much to date.
This application illustrates the versatility of dynamic programming as a dynamic solution method

In this lab, we follow a simple version of a standard job search model.
Assume that workers live infinitely long.
We will split a worker's life into discrete time periods, and in each period the worker is either
employed or unemployed, and receives a job offer. The worker must make a choice between discrete actions
(such as accepting or rejecting a job offer), with the goal of maximizing some utility function (hence,
this is a \emph{discrete choice} problem).

We can state this problem in terms of dynamic programming by defining an appropriate value function.
Let the value of entering a period with most recent wage $w$,
current job offer wage $w'$, and employment status $s$ be given by the following value function,
\begin{equation}\label{EqV}
   V(w,w',s) = \begin{cases}
                  V^E(w)    \quad&\text{if}\quad s = E \\
                  V^U(w,w') \quad&\text{if}\quad s = U \\
               \end{cases}
\end{equation}
where employment status is a binary variable $s\in\{E,U\}$ ($E$ indicates ``employed" and $U$ indicates ``unemployed");
a person can be either employed or unemployed.

As in the cake eating problem, the value function is calculated as the sum of some reward (based on the
current state) and the discounted value of entering the next period in some particular state.
The reward function, denoted (as usual) by $u$, gives the utility of spending available funds.
Assuming that a worker receives some wage $x$ in a given period and spends all available money in the period,
the utility of consumption is given by
\[
u(x).
\]
Calculating the value for the next period depends on the employment status in the current period, so we address
this separately for each case.

Let us first consider the case where the individual is unemployed ($s = U$). As is customary, let $s'$ denote the
employment status of the worker in the next period.
In this unemployed state, the worker receives unemployment benefits equal to a fraction of her most recent wage,
i.e. $\alpha w$, where $\alpha \in (0, 1)$.
Hence, the utility of consumption in the current unemployed state is given by
\[
u(\alpha w).
\]

The worker also receives one wage offer ($w'$) per period, and will obtain
this wage in the next period provided that he chooses to accept employment, i.e. provided $s' = E$.
The worker must decide whether to accept the current wage offer $w'$ or to remain unemployed in the next period,
i.e. he must decide on the value of $s'$. How
does he make this choice? He must weigh the value of entering the next period as an employed worker with wage
$w'$ (given by $V^E(w')$) versus the value of entering the next period as an unemployed worker with
previous wage $w$ and unknown wage offer $w''$ (given by $V^U(w,w'')$). Because the worker cannot know
what the future wage offer $w''$ will be, it is treated as a random variable with a particular probability
distribution. Hence, the worker must actually compute the \emph{expected} value of entering the next
period unemployed. This term is simply
\[
\mathbb{E}_{w''}V^U(w,w''),
\]
where $\mathbb{E}_{w''}$ denotes the expectation operator with respect to the probability distribution of future
wage offers $w''$.
To sum up, the worker chooses to accept the wage offer $w'$ or remain unemployed in the next period based on
which option gives the greater expected value, and the value of this decision is given by
\[
\max\Bigl\{V^E(w'), \,\, \mathbb{E}_{w''}V^U(w,w'')\Bigr\}.
\]

The overall value of the current unemployed state with previous wage $w$ and current wage offer $w'$ is
just the utility of consumption plus the discounted value of the next period, i.e.
\begin{equation}\label{EqVu}
V^U(w,w') = u(\alpha w) + \beta \max\Bigl\{V^E(w'), \,\, \mathbb{E}_{w''}V^U(w,w'')\Bigr\},
\end{equation}
where $\beta$ is the discount factor.

Now we turn to the case where the job status is employed ($s = E$).
In this case, the worker receives a wage $w$ in the current period, and so the utility of consumption is
just
\[
u(w).
\]
In the next period, the worker will have most recent wage $w$, she will receive wage offer $w''$, and will
have employment status $s'$. As in the unemployed case, $w''$ is unknown and treated as a random variable.
Unlike the unemployed case, however, the worker's future employment status $s'$ is not under her control,
but rather is also a random variable. The reason for this is that the worker will remain employed
until she loses the job, a random event that occurs with some fixed probability in each time period.
Hence, we must calculate the expected value of the next period with respect to both $w''$ and $s'$.
We may write the entire value function for the employed case as
\begin{equation}\label{EqVe1}
   V^E(w) = u(w) + \beta \mathbb{E}_{w'',s'}V(w,w'',s').
\end{equation}

To calculate the expectation term, we need to know the joint probability distribution over $w''$ and $s'$.
This can be characterized in the following way.
We assume that $s'$ and $w''$ are independent. Hence, we can split the joint expectation
operator into the composition of the two individual expectation operators:
\[
\mathbb{E}_{w'',s'} = \mathbb{E}_{w''}\mathbb{E}_{s'}.
\]
Let $\gamma$ represent the probability that an employed worker becomes unemployed in the next period,
so that $1-\gamma$ is the probability of remaining employed in the next period.
If the worker stays employed in the next period ($s' = E$), then next period's wage equals the current
period's wage, and the term inside the expectation is
\[
V(w,w'',E) = V^E(w).
\]
We then have
\begin{align*}
\mathbb{E}_{s'}V(w,w'',s') &= (1-\gamma)V(w,w'',E) + \gamma V(w,w'',U)\\
&= (1-\gamma)V^E(w) + \gamma V^U(w,w'').
\end{align*}
Notice that the term $(1-\gamma)V^E(w)$ is constant with respect to $w''$. Then
\begin{align*}
\mathbb{E}_{w''}\mathbb{E}_{s'}V(w,w'',s') &= \mathbb{E}_{w''}\left[(1-\gamma)V^E(w) + \gamma V^U(w,w'')\right]\\
&= \mathbb{E}_{w''}(1-\gamma)V^E(w) + \mathbb{E}_{w''}\gamma V^U(w,w'')\\
&= (1-\gamma)V^E(w) + \gamma \mathbb{E}_{w''}V^U(w,w'').
\end{align*}
Hence, we can rewrite \eqref{EqVe1} as follows:
\begin{equation}\label{EqVe2}
   V^E(w) = u(w) + \beta \Bigl[(1-\gamma)V^E(w) + \gamma \mathbb{E}_{w''}V^U(w,w'')\Bigr].
\end{equation}

We have now completely described the value function. What about the policy function?
The policy function for the unemployed worker gives his decision on whether to accept the job $s'=E$
or to reject the job $s'= U$.
This will be a function of both the most recent wage $w$ and the current wage offer $w'$.
The employment status $s'$ in the next period is determined by the policy function $\psi$:
\[
s' = \psi(w,w').
\]

These discrete choice problems are often called threshold
problems because the policy choice depends on whether the state variable is greater than or less than
some threshold level. That is, an unemployed worker will accept a job if and only if the offer wage is
above some set amount that depends on the most recent wage $w$. In the labor search model,
the threshold level is called the ``reservation wage'' $w_R'$. The reservation wage $w_R'$ is defined as
the wage offer such that the worker is indifferent between accepting the job $s' = E$ and
staying unemployed $s' = U$. Hence, this reservation wage satisfies the equation
\begin{equation}\label{EqWR}
   V^E(w_R') = E_{w''}\left[V^U(w,w'')\right].
\end{equation}
The policy function will then take the form of accepting the job if $w' \geq w_R'$ or
rejecting the job offer and remaining unemployed if $w' < w_R'$:
\begin{equation}\label{EqSprime}
   s' = \psi(w,w') = \begin{cases}
                      E \quad\text{if}\quad w' \geq w_R' \\
                      U \quad\text{if}\quad w' < w_R'.
                   \end{cases}
\end{equation}
Figure \ref{fig:disc_policy} shows an example of the discrete policy function.

\begin{figure}
\includegraphics[width=\textwidth]{disc_policy.pdf}
\caption{Here is the policy function for fixed $w = 50$.  Numerically we let 0 represent unemployment, $U$,
and 1 represent employment, $E$.  Thus we see that an individual will choose to take a new job, given
their old wage was 50, at a wage of roughly 35.  Thus for a previous wage of 50, we say the reservation wage is 35.}
\label{fig:disc_policy}
\end{figure}

In summary, the labor search discrete choice problem is characterized by the value functions \eqref{EqV}, \eqref{EqVu},
and \eqref{EqVe2}, the reservation wage \eqref{EqWR}, and the policy function \eqref{EqSprime}. Because wage offers
are distributed according to some given probability distribution (denote the cdf by $F(w')$),
and because the policy function takes the form of \eqref{EqSprime},
the probability that the unemployed worker receives a wage offer that he will reject is $F(w_R')$ and the probability
that he receives a wage offer that he will accept is $1 - F(w_R')$. Just like the continuous choice cake eating
problems, this problem can be solved by value function, policy function, or modified policy function iteration.

The value function iteration solution method for the equilibrium in the labor search problem is analogous to the
value function iteration from the previous labs. The only difference is that two value functions ($V^E$ and $V^U$)
must converge to a fixed point in this problem instead of just one value function converging in the previous problems.
Although there are two value functions to consider, there is only one policy function, since decisions are only made
in the unemployed state.  Thus, there is only one policy function on which to iterate in the case of policy or modified
policy iteration.

In the following problems, you will solve the job search problem using value function iteration and modified policy
function iteration. Assume that the consumption utility function $u$ is given by
\[
u(w) = \sqrt{w}.
\]
Assume that the probability of becoming unemployed in a given period is $\gamma = 0.10$, the fraction of wages paid
in unemployment benefits is $\alpha = 0.5$, and the discount factor is $\beta = 0.9$.

Assume that the log of wage offers are distributed normally.  We then say that offers are distributed
lognormally and write
\[
w'\sim \text{LogN}(\mu,\sigma).
\]
This is a convenient choice for the distribution of
wage offers.  Among other things, it guarantees that wage offers will be positive.
A mean of $20$ and variance of $200$ are typical parameters of such a wage distribution,
and we will use these parameters in the following problems.

As usual when dealing with continuous variables, we form a discrete approximation of
the possible wage values. In particular, approximate the wage values by a vector of
length $N = 500$ of equally-spaced values from $w_{min} = 0$ to $w_{max} = 100$, inclusive.
We then form a corresponding discrete approximation of the probability density function
$f(w')$ for the lognormal wage offers using code provided in the file \li{discretelognorm.py},
as follows, where \li{w} is the length-$N$ vector of wage values, $m$ is the mean, and $v$ is
the variance as specified above:
\begin{lstlisting}
>>> from discretelognorm import discretelognorm
>>> f = discretelognorm(w, m, v)
\end{lstlisting}
The function \li{discretelognorm} computes the discrete pdf of the specified lognormal distribution in much
the same way that you calculate the discrete normal pdf when solving the stochastic cake eating problem.

\begin{problem}
Solve the job search problem using value function iteration. Return the converged value functions
$V^E$ and $V^U$, as well as the converged policy function $\psi$. The following steps provide detailed
instructions. Note that there are multiple ways to proceed, and the following is simply one (fairly good)
possibility.
\begin{enumerate}

   \item As described above, represent the possible wage values by an array \li{w} of length
   $N$. Denote this array entrywise by
   \[
   w = (w_1,w_2,\ldots,w_N).
   \]
   Calculate the corresponding discrete lognormal pdf \li{f}, exactly as shown above.

   \item Note that $u(w)$ and $u(\alpha w)$ are needed when computing the value functions.
   Since these quantities do not change from one iteration to another, it is smart to
   compute them once at the outset. Denote $u(w)$ by \li{uw} and $u(\alpha w)$ by \li{uaw}.
   These are easily calculated as follows:
\begin{lstlisting}
>>> uw = u(w)
>>> uaw = u(alpha*w).reshape((N,1))
\end{lstlisting}
    where \li{u} is the square root function. We must reshape \li{uaw} because of array broadcasting
    issues that arise in the code snippets below.

   \item Since $V^E$ is a function of only $w$, it will be represented by a vector of length $N$, where
   the $i$-th entry gives $V^E(w_i)$. The unemployed value function $V^U$, however,
   is a function of both $w$ and $w'$, so it will be represented by an $N \times N$ array,
   where the $(i,j)$-th entry gives $V^U(w_i,w_j)$. Initialize the entries of these arrays to 0:
\begin{lstlisting}
>>> VE = np.zeros(N)        #employed value function
>>> VU = np.zeros((N,N))    #unemployed value function
\end{lstlisting}

   \item Note that $\mathbb{E}_{w''}V^U(w,w'')$  is needed to calculate both $V^E$ and $V^U$.
   This expectation depends on $w$, and so can be represented by a length $N$ array, where the
   $i$-th entry is $\mathbb{E}_{w''}V^U(w_i,w'')$.
   It is convenient to assign a variable to this array to keep track of it throughout the iterations.
   We denote the expectation by \li{EVU}, and initialize it to zeros:
\begin{lstlisting}
>>> EVU = np.zeros(N)
\end{lstlisting}

   \item For reasons that will soon become apparent, we will need to create a $N\times N$ helper array
   whose rows are equal to \li{VE} (call this array \li{MVE}), and a $N \times N$ helper array whose columns
   are equal to \li{EVU} (call this array \li{MEVU}).
   At the outset, simply initialize these arrays to zeros.

   \item Because job status is a binary variable, the policy function returns one of two possible values. It is
   convenient to represent ``employed" by $1$ and ``unemployed" by $0$. Now the policy function depends
   on $w$ and $w'$, so it will also be represented by an $N\times N$ array \li{PSI} of zeros and ones,
   where the $(i,j)$-th entry gives $\psi(w_i, w_j)$.

   \item Now we are ready to begin the iteration.
   A single iteration involves computing the updated value functions $V^E$ and $V^U$ from
   equations \eqref{EqVe2} and \eqref{EqVu} and then calculating the $2$-norm distance between
   both pairs of old and updated value functions to test for convergence. If both of these
   $2$-norm distances are less than $10^{-9}$, terminate the iteration.

   Before calculating the updated value functions, we first update our helper arrays \li{MVE} and
   \li{MEVU}. The rows of \li{MVE} need to equal \li{VE}. We can use array broadcasting:
\begin{lstlisting}
>>> MVE[:,:] = VE.reshape((1,N))
\end{lstlisting}
   The columns of \li{MEVU} need to equal \li{EVU}, so use a similar technique:
\begin{lstlisting}
>>> MEVU[:,:] = EVU.reshape((N,1))
\end{lstlisting}

   Now let us address how to compute the updated $V^U$, which we denote by \li{VU1}.
   Equation \eqref{EqVu} shows that it is the sum of
   two terms. The first, $u(\alpha w)$, we have already computed and stored in the variable \li{uaw}.
   The second term involves a maximization between two alternatives. One can imagine writing
   a double for-loop ranging over the values of $w'$ and $w$ to compute each individual
   $\max\{V^E(w'), \mathbb{E}_{w''}V^U(w,w'')\}$, but we can take advantage of the helper
   arrays \li{MVE} and \li{MEVU} to do this computation in one efficient line of code.
   Note that the $(i,j)$-th entry of \li{MVE} is just $V^E(w_j)$ and the $(i,j)$-th
   entry of \li{MEVU} is $\mathbb{E}_{w''}(w_i, w'')$, and
   \[
   V^U(w_i,w_j) = u(\alpha w_i) + \beta\max\{V^E(w_j), \mathbb{E}_{w''}V^U(w_i,w'')\}.
   \]
   Hence, taking the entrywise maximum of the arrays \li{MVE} and \li{MEVU} gives us the appropriate
   max term for $V^U$. To get the entrywise maximum of two arrays, stack the arrays along a new
   axis using \li{np.dstack}, and maximize along that axis. The computation for \li{VU}, then, is
   \begin{lstlisting}
>>> VU1 = uaw + beta*np.max(np.dstack([MEVU, MVE]), axis=2)
   \end{lstlisting}

   Calculating the updated $V^E$, denoted by \li{VE1}, is more straightforward.
   Equation \eqref{EqVe2} shows that it is just
   a particular linear combination of the arrays \li{uw}, \li{VE}, and \li{EVU}:
   \begin{lstlisting}
>>> VE1 = uw + beta*((1-gamma)*VE + gamma*EVU)
   \end{lstlisting}

   We can now calculate the 2-norm distances between old and updated value functions.
   It remains to update \li{VE}, \li{VU}, and \li{EVU}.
   The first two updates are trivial, and calculating \li{EVU} is equivalent to the
   matrix-vector multiplication of \li{VU} with \li{f}. This is similar to how we computed
   expectations in previous labs:
   \begin{lstlisting}
>>> EVU = np.dot(VU,f).ravel()
   \end{lstlisting}
   We use the \li{ravel} function to ensure that \li{EVU} is a flat array.

   \item Notice that it is not necessary to iteratively update the policy function, as it is not needed
   to update the value functions. Thus, we need only compute the policy function once, after convergence
   of the value functions has been achieved. This is done in a manner similar to calculating
   $\max\{V^E(w'), \mathbb{E}_{w''}V^U(w,w'')\}$ as described above, except we need to take the \emph{argmax}:
\begin{lstlisting}
>>> PSI = np.argmax(np.dstack([MEVU,MVE]), axis=2)
\end{lstlisting}

   \item Compute the reservation wage $w_R'$ as a function of the current wage $w$. It will be represented
   by a length $N$ array called \li{wr}. The reservation wage is the
   value of $w'$ where the policy function changes from zeros to ones (the optimal choice changes from remaining
   unemployed to accepting the job offer). We can calculate this as follows:
   \begin{lstlisting}
>>> wr_ind = np.argmax(np.diff(PSI), axis = 1)
>>> wr = w[wr_ind]
   \end{lstlisting}

   \item Plot the equilibrium reservation wage $w_R'$ of the converged problem as a function of the current
   wage $w$ with the current wage on the $x$-axis and the reservation wage $w_R'$ on the $y$-axis. This is
   the most common way to plot discrete choice policy functions. The reservation wage represents the wage
   that makes the unemployed worker indifferent between taking a job offer and rejecting it. So any wage
   above the reservation wage line represents $s' = E$ and any wage below the reservation wage line represents
   $s' = U$. Your plot should resemble that in Figure \ref{fig:res_wage}.

\end{enumerate}
\end{problem}

\begin{figure}
\includegraphics[width=\textwidth]{reservation_wage.pdf}
\caption{The reservation wage as a function of previous wage for the job search problem.}
\label{fig:res_wage}
\end{figure}

In the previous problem, it was necessary to iterate on two value functions.
Consequently the convergence is relatively slow.
We can improve upon this situation by using modified policy function iteration.

\begin{problem}
Solve the same problem, this time using modified policy function iteration with $15$ value function iterations
within each policy iteration.  You should be able to re-use much of your code from the previous problem.

Start off by initializing all of the same variables. Additionally, initialize your policy function array \li{PSI},
say
\begin{lstlisting}
>>> PSI = 2*np.ones((N,N))
\end{lstlisting}

Next comes the iteration.
Essentially, the iteration will consist of an outer while-loop (which terminates once the 2-norm distance
between successive policy functions passes below $10^{-9}$), and an inner for-loop (with $15$ loops).

The first step in the while-loop is to calculate the new policy function \li{PSI1}, just as in the previous
problem. Next, perform the inner for-loop, which consists simply of the value function iteration, but this
time using the current policy function. This means the line of code
\begin{lstlisting}
>>> VU = uaw + beta*np.max(np.dstack([MEVU, MVE]), axis=2)
\end{lstlisting}
is no longer valid, as it does not use the policy function. We must instead have
\begin{lstlisting}
>>> VU = uaw + beta*(MVE*PSI1 + MEVU*(1 - PSI1))
\end{lstlisting}
Why is this code correct?

Finally, after exiting the for-loop, calculate the 2-norm distance between the old and the new policy function,
and then update your old policy function, i.e.
\begin{lstlisting}
>>> PSI = PSI1
\end{lstlisting}

After convergence is achieved, once again compute the reservation wage array, and plot it as in the previous
problem. Then return the converged policy function.
\end{problem}

\begin{problem}
How many iterations did value function iteration take?
How many iterations did modified policy function iteration take?
Which was faster?
\end{problem}


\documentclass[bigger]{beamer}

\input{header-beam} % change to header-handout for handouts

% ====================
\title[Lecture 5]{Logic I F13 Lecture 5}
\date{September 24, 2013}
% ====================

\include{header}

\section[Expressive Power]{Expressive Power of the Boolean Connectives}

\subsec{Truth Tables for the Boolean Connectives}{

\[
\begin{array}{c|c}
P & \lnot P\\
\hline
\T & \F \\
\F & \T
\end{array}
\qquad
\begin{array}{cc|c}
P & Q & (P \land Q)\\
\hline
\T & \T & \T\\
\T & \F & \F\\
\F & \T & \F\\
\F & \F & \F
\end{array}
\qquad
\begin{array}{cc|c}
P & Q & (P \lor Q)\\
\hline
\T & \T & \T\\
\T & \F & \T\\
\F & \T & \T\\
\F & \F & \F
\end{array}
\]

}

\subsec{Neither \dots nor \dots}{

\bits
\item Neither a nor b is a cube
\item $\sf \lnot Cube(a) \land \lnot Cube(b)$
\item $\sf \lnot (Cube(a) \lor Cube(b))$
\eit

}

\subsec{Not Both}{

\bits
\item a and b are not both large
\item $\sf \lnot(Large(a) \land Large(b))$
\item $\sf \lnot Large(a) \lor \lnot Large(b)$
\eit

}

\subsec{Exclusive ``Or''}{

\bits
\item a or b is a cube (but not both)
\item $\sf (Cube(a) \lor Cube(b)) \land \lnot(Cube(a) \land Cube(b))$
\item $\sf (Cube(a) \land \lnot Cube(b)) \lor (\lnot Cube(a) \land Cube(b))$
\eit
}

\subsec{At Least One Of}{

\bits
\item At least one of a, b, and c is small
\item $\sf Small(a) \lor (Small(b) \lor Small(c))$
\eit

}

\subsec{At Least Two Of}{

\bits
\item At least two of a, b, and c are small
\item $\sf (Small(a) \land Small(b)) \lor (Small(a) \land Small(c)) \lor (Small(b) \land Small(c))$
\eit

}

\subsec{At Least As Large As and $\le$}{

\bits
\item a is at least as large as b
\bits
\item $\sf Larger(a, b) \lor SameSize(a, b)$
\item $\sf \lnot Smaller(a, b)$, $\sf\lnot Larger(b, a)$
\eit
\item $2 \le 3$
\bits
\item $(1+1) < (1+(1+1)) \lor (1+1) = (1+(1+1))$
\item $\lnot (1 + (1+1)) < (1+1)$
\eit\eit

}
\section{Step-by-Step Translations}

\subsec{Step-by-Step Method of Translation}{

\bit
\item a and b are either left of c or right of d, \emph{but}\\ e is neither left of c nor right of d.
\bit
\item a and b are either left of c or right of d
\item but
\item e is neither left of c nor right of d
\eit\eit
}

\subsec{Step-by-Step Method of Translation}{

\bits
\item a \emph{and} b are either left of c or right of d
\bit
\item a is left of c \emph{or} right of d
\bit
\item a is left of c or a is right of d\\
$\sf LeftOf(a, c) \lor RightOf(a, d)$
\eit
\item b is left of c \emph{or} right of d
\bit
\item b is left of c or b is right of d\\
$\sf LeftOf(b, c) \lor RightOf(b, d)$
\eit
\eit
\eit
\begin{align*}
\sf (LeftOf(a, c) \lor {} & \sf RightOf(a, d)) \land {}\\
\sf (LeftOf(b, c) \lor {} & \sf RightOf(b, d))
\end{align*}

}

\subsec{Step-by-Step Method of Translation}{

\bits
\item e is neither left of c nor right of d
\[\sf \lnot(LeftOf(e, c) \lor RightOf(e, d))\]
\eit
}

\subsec{Step-by-Step Method of Translation}{

\bit
\item a and b are either left of c or right of d, but\\ e is neither left of c nor right of d.
\bit
\item a and b are either left of c or right of d
\begin{align*}
\sf (LeftOf(a, c) \lor {} & \sf RightOf(a, d)) \land {}\\
\sf (LeftOf(b, c) \lor {} & \sf RightOf(b, d))
\end{align*}
\item but
$\land$
\item e is neither left of c nor right of d
\[\sf \lnot(LeftOf(e, c) \lor RightOf(e, d))\]
\eit\eit
\begin{align*}
\sf((LeftOf(a, c) \lor {} & \sf RightOf(a, d)) \land {} \\
\sf (LeftOf(b, c) \lor {} & \sf RightOf(b, d))) \land {}\\
\sf \lnot(LeftOf(e, c) \lor {} &\sf RightOf(e, d))
\end{align*}

}

\section{Ambiguity}

\subsec{Ambiguity in English}{

\bit
\item Lexical ambiguity: one word---many meanings \\
e.g., ``bank'', ``crane''
\item Syntactic ambiguity: one sentence---many readings\\
e.g.,
\bit
\item ``Flying planes can be dangerous'' (Chomsky)
\item ``One morning I shot an elephant in my pajamas.\\ How he got in my pajamas, I don't know.'' (Groucho Marx)
\eit
\eit

}

\subsec{Connectives and Ambiguity}{

\bits
\item a adjoins b and c or d
\item a adjoins [[b and c] or d]\\
$\sf (Adjoins(a, b) \land Adjoins(a, c)) \lor Adjoins(a, d)$
\item a adjoins [b and [c or d]]\\
$\sf Adjoins(a, b) \land (Adjoins(a, c) \lor Adjoins(a, d))$
\eit

}

\subsec{The Man Who Was Hanged by a Comma}{

\bit
\item Sir Roger Casement (1864--1916)
\item British consul to Congo and Peru
\item Tried to recruit Irish revolutionaries in Germany during WWI
\item Tried for treason
\eit

}

\subsec{Treason Act of 1351}{

\small
ITEM, Whereas divers Opinions have been before this Time in what Case
Treason shall be said, and in what not; the King, at the Request of
the Lords and of the Commons, hath made a Declaration in the Manner as
hereafter followeth, that is to say; When a Man doth compass or
imagine the Death of our Lord the King, or of our Lady his Queen or of
their eldest Son and Heir; or if a Man do violate the King's
Companion, or the King's eldest Daughter unmarried, or the Wife of the
King's eldest Son and Heir; or \textbf{if a Man do levy War against our Lord
the King in his Realm, or be adherent to the King's Enemies in his
Realm, giving to them Aid and Comfort in the Realm\onslide<2>{\emph{,}} or elsewhere}, and
thereof be probably attainted of open Deed by the People of their
Condition: \dots And it is to be
understood, that in the Cases above rehearsed, that ought to be judged
Treason which extends to our Lord the King, and his Royal Majesty:
\dots

}

\subsec{R v. Casement in the Blocks Language}{

\bits
\item a is a cube in front of b, or a tetrahedron in front of b[,] or in back of b.

\item Without comma:
\begin{align*}
\sf(Cube(a) \land {} & \sf FrontOf(a, b)) \lor {}\\
\sf(Tet(a) \land {} & \sf(FrontOf(a, b) \lor BackOf(a, b)))
\end{align*}
\item With comma:
\begin{align*}
\sf(Cube(a) \land {} & \sf(FrontOf(a, b) \lor BackOf(a, b))) \lor {}\\
\sf(Tet(a) \land {} & \sf(FrontOf(a, b) \lor BackOf(a, b))
\end{align*}
\eit
}

\subsec{More Scope Ambiguity}{

\bits
\item a and b are left of c or right of d
\item a is left of c or right of d, and so is b
\begin{align*}
\sf (LeftOf(a, c) \lor {} & \sf RightOf(a, d)) \land {}\\
\sf (LeftOf(b, c) \lor {} & \sf RightOf(b, d))
\end{align*}
\item a and b are both left of c, or a and b are both right of d
\begin{align*}
\sf (LeftOf(a, c) \land {} & \sf LeftOf(b, c)) \lor {}\\
\sf (RightOf(a, d) \land {} & \sf RightOf(b, d))
\end{align*}
\eit

}

\end{document}


\gotosection{1}{2}
\subsection{Introducing the actors: matrices}

\begin{exercise}{2}
  \begin{enumerate}
    \item $\begin{xmatrix}1 & 2 & 3 \\ 4 & 5 & 6\end{xmatrix}
           \begin{xmatrix}7 & 8 \\ 9 & 0 \\ 1 & 2\end{xmatrix} =
           \begin{xmatrix}
              7 + 18 + 3 &  8 + 0 +  6 \\
             28 + 45 + 6 & 32 + 0 + 12
           \end{xmatrix} =
           \begin{xmatrix}28 & 14 \\ 79 & 44\end{xmatrix}$

    \item Incompatible matrices: the first matrix has 2 columns while the second
          one has 3 rows.

    \item $\begin{aligned}[t]
            \begin{xmatrix}
               1 & -1 &  1 \\
              -1 &  0 &  2 \\
              -1 &  1 &  1
            \end{xmatrix}
            \begin{xmatrix}
               0 &  1 & -1 \\
              -1 &  1 &  2 \\
               2 &  0 & -2
            \end{xmatrix} &=
            \begin{xmatrix}
              0 + 1 + 2 &  1 - 1 + 0 & -1 - 2 - 2 \\
              0 + 0 + 4 & -1 + 0 + 0 &  1 + 0 - 4 \\
              0 - 1 + 2 & -1 + 1 + 0 &  1 + 2 - 2
            \end{xmatrix} \\ &=
            \begin{xmatrix}
              3 &  0 & -5 \\
              4 & -1 & -3 \\
              1 &  0 &  1
            \end{xmatrix}
          \end{aligned}$

    \item $\begin{xmatrix}7 & 1 \\ -1 & 0 \\ 2 & 3\end{xmatrix}
           \begin{xmatrix}5 \\ -4\end{xmatrix} =
           \begin{xmatrix}
             35 - 4 \\
             -5 + 0 \\
             10 - 12
           \end{xmatrix} =
           \begin{xmatrix}31 \\ -5 \\ -2\end{xmatrix}$

    \item \def \tmp{\begin{bmatrix*}[r]0 & 1 \\ -1 & 3\end{bmatrix*}}
          $\begin{aligned}[t]
            \begin{xmatrix}1 & 2 \\ 0 & 3\end{xmatrix}
            \begin{xmatrix}1 & 4 \\ 1 & 3\end{xmatrix}
            \tmp &=
            \begin{xmatrix}
              1 + 2 & 4 + 6 \\
              0 + 3 & 0 + 9
            \end{xmatrix}
            \tmp \\ &=
            \begin{xmatrix}3 & 10 \\ 3 & 9\end{xmatrix}
            \tmp \\ &=
            \begin{xmatrix}
              0 + 3 &   0 +  9 \\
              -3 + 9 & -10 + 27
            \end{xmatrix} \\
            &= \begin{xmatrix}3 & 9 \\ 6 & 17\end{xmatrix}
          \end{aligned}$

    \item Incompatible matrices: the first matrix has 3 columns while the second
          one has 2 rows.
  \end{enumerate}
\end{exercise}

\begin{exercise}{3}
  \begin{enumerate}
    \item $\begin{pmatrix}AB_{13}\\AB_{23}\end{pmatrix} =
           \begin{xmatrix}1 & 2 & 0\\3 & 1 & -1\end{xmatrix}
           \begin{pmatrix}1 \\ 2 \\ 3\end{pmatrix} =
           \begin{pmatrix}1+4 \\ 3+2-3\end{pmatrix} =
           \begin{pmatrix}5 \\ 2\end{pmatrix}$
    \item $\begin{aligned}[t]
            \begin{xmatrix}AB_{21} & AB_{22} & AB_{23}\end{xmatrix} &=
            \begin{xmatrix}3 & 1 & -1\end{xmatrix}
            \begin{xmatrix}
              2 & 5 & 1 \\
              1 & 4 & 2 \\
              1 & 3 & 3
            \end{xmatrix}
            \\ &=
            \begin{xmatrix}6 + 1 - 1 & 15 + 4 - 3 & 3 + 2 - 3\end{xmatrix}
            \\ &=
            \begin{xmatrix}6 & 16 & 2\end{xmatrix}
          \end{aligned}$
  \end{enumerate}
\end{exercise}

\begin{exercise}{5}
  Since $A$ is symmetric, $A = A^{\top}$.

  \begin{enumerate}
    \item True. $B^{\top}A = B^{\top}A^{\top} = (AB)^{\top}$ \QED
    \item True. $B^{\top}A^{\top} = (AB)^{\top} = (A^{\top}B)^{\top}$ \QED
    \item False. $(A^{\top}B)^{\top} = B^{\top}A \neq BA$,
          since $B^{\top} = B$ doesn't hold if $B$ is not symmetric.
    \item False. $(AB)^{\top} = B^{\top}A^{\top} \neq A^{\top}B^{\top}$,
          since $XY = YX$ iff $X$ is the inverse of $Y$.
  \end{enumerate}
\end{exercise}

\begin{exercise}{8}
  The only value of $a$ such that $AB = BA$ is $0$, as:
  \medskip

  $AB = \begin{xmatrix}1 & 1 \\ 1 & 0\end{xmatrix}
        \begin{xmatrix}1 & 0 \\ a & 1\end{xmatrix}
      = \begin{xmatrix}1 + a & 1 \\ 1 & 0\end{xmatrix}$
  \medskip

  $AB = \begin{xmatrix}1 & 0 \\ a & 1\end{xmatrix}
        \begin{xmatrix}1 & 1 \\ 1 & 0\end{xmatrix}
      = \begin{xmatrix}1 & 1 \\ 1 + a & a\end{xmatrix}$
  \medskip

  $\left\{
    \begin{aligned}
      1 + a &= 1 \\
          a &= 0
    \end{aligned}
  \right.\quad \Longrightarrow \quad a = 0$
\end{exercise}

\begin{exercise}{10}
  Follow the formula for the inverse of $2\times2$-matrix:

  \begin{align*}
    A^{-1} &= \frac{1}{a^2-0}\begin{xmatrix}a & -b\\0 & a\end{xmatrix} \\
           &= \begin{xmatrix}
                \frac{1}{a} & -\frac{b}{a^2} \\
                          0 &  \frac{1}{a}
              \end{xmatrix}
  \end{align*}
\end{exercise}

\begin{exercise}{15}
  Let $A = \begin{xmatrix}
             1 & a & b \\
             0 & 1 & c \\
             0 & 0 & 1
           \end{xmatrix}$,
  and $B = \begin{xmatrix}
             1 & x & y \\
             0 & 1 & z \\
             0 & 0 & 1
           \end{xmatrix}$.
  The product is

  $$AB =
    \begin{xmatrix}
      1 & x + a & y + az + b \\
      0 &     1 &      z + c \\
      0 &     0 &          1
    \end{xmatrix} =
    I_3 = \begin{xmatrix}1&0&0\\0&1&0\\0&0&1\end{xmatrix}$$

  Then we have:
  \medskip

  $\left\{
    \begin{aligned}
           x + a &= 0 \\
      y + az + b &= 0 \\
           z + c &= 0
    \end{aligned}
  \right.\quad \Longrightarrow \quad
  \left\{
    \begin{aligned}
               x &= -a \\
               z &= -c \\
      y - ac + b &= 0
    \end{aligned}
  \right.\quad \Longrightarrow \quad
  \left\{
    \begin{aligned}
      x &= -a     \\
      y &= ac - b \\
      z &= -c
    \end{aligned}
  \right.$
\end{exercise}

\begin{exercise}{16}
  From the definition of transpose and symmetric matrix,
  it is obvious that $(A^{\top})^{\top} = A$,
  and that if $A = A^{\top}$, then $A$ is symmetric.

  For any matrix $A$, since $(A^{\top}A)^{\top} = A^{\top}(A^{\top})^{\top} = A^{\top}A$,
  namely $A^{\top}A$ equal to its transpose, $A^{\top}A$ is symmetric. \QED
\end{exercise}


%---------------------------------------------------------------------------------
\chapter{Fitzhugh-Nagumo Model Example}
\label{chap:fitzhugh-nagumo}
%---------------------------------------------------------------------------------
\section{Background}
\label{sec:background}
We will look into Fitzhugh-Nagumo model as an application of the Python package. The Fitzhugh-Nagumo model describes an excitable system, such as the action potential of cardiac cells. The action potential was first described by Hodgkin and Huxley. Their model were then simplified to the Fitzhugh-Nagumo model, retaining the fast-slow phase and the excitability of the Hodgkin \& Huxley model. \cite{Keener2009}

This model is relevant to my D.Phil.~project as my project will be related to action potential of cardiac muscle cells, an excitable system. I will be working on sodium ion channels of cardiac muscle cells, studying the effect of the flow of sodium ions across the cell membrane on the cell's action potential. Moreover, Fitzhugh-Nagumo model captures all the important features of an action potential, the excitability and the fast-slow phase. Therefore, it is a good simple model to start with.

\section{Fitzhugh-Nagumo model}
\label{sec:FHN}
The definition of Fitzhugh-Nagumo model is
\begin{align}
\label{eqn:FHN}
    \epsilon \frac{dv}{dt} &= f(v) - w + I_{app} \\
    \frac{dw}{dt} &= v - \gamma w \label{eqn:FHN-end}
\end{align}
where $f(v) = v(1-v)(v-\alpha)$, $0 < \alpha < 1$, $\epsilon \ll 1$, $I_{app}$ is the applied current and $t$ is time. The fast $v$ is the excitation variable, while the slow $w$ is the recovery variable.

In this implementation, the parameters are chosen to be $\alpha = 0.1$, $\gamma = 0.5$, $\epsilon = 0.01$ and $I_{app} = 0.026$, taking reference from \cite{Chapwanya2018}.
The initial values are taken to be near the origin, which are $(v_0, w_0) = (0.01, 0.01)$. The model is solved for time $t$ from 0 to 1.

Fitzhugh-Nagumo model is solved with the various numerical methods implemented in the software that I have developed. The solutions of the model are in a notebook at the link:  \href{https://nbviewer.jupyter.org/github/FarmHJ/numerical-solver/blob/main/examples/fitzhugh_nagumo.ipynb}{\underline{\emph{Fitzhugh-Nagumo model notebook}}}.

\begin{figure}
    \includegraphics[width=0.95\columnwidth]{FHN_Euler_explicit}
    \caption{(\textit{A}) Phase plane of $v$ and $w$ by Euler's explicit method. (\textit{B}) Graph of $v$ and $w$ against time by Euler's explicit method.}
    \label{fig:FHN_Euler_explicit}
\end{figure}

From Figure \ref{fig:FHN_Euler_explicit} B, we can see that $v$, the excitation variable is excited in the early stage. While $v$ increases significantly, the change in $w$ is small. After $v$ reaches its peak and starts to reduce, $w$ increases slowly. This can be observed in both the phase plane (Figure \ref{fig:FHN_Euler_explicit} A) and the variable graph (Figure \ref{fig:FHN_Euler_explicit} B). When the variable $v$ starts to recover to its original value, $w$ is at its maximum. The scattering of points in the phase plane captures the feature of the model, where the change in $v$ is rapid while the change in $w$ is slow. When the change in $v$ is significantly larger than the change in $w$, the points are sparse. On the other hand, the points are packed when $w$ increases or decreases more than $v$. However, in the adaptive methods, such insights cannot be interpreted directly from the phase plane. Therefore, green triangles were plotted in Figure \ref{fig:FHN_adaptive} B to indicate the adapted mesh points. The mesh points are adapted towards large change in $v$ or $w$ over a short period of time.

\begin{figure}
    \includegraphics[width=0.95\columnwidth]{FHN_adaptive}
    \caption{(\textit{A}) Phase plane of $v$ and $w$ for adaptive method BS23. (\textit{B}) Graph of $v$ and $w$ against time for adaptive method BS23}
    \label{fig:FHN_adaptive}
\end{figure}

\section{Convergence of Fitzhugh-Nagumo model}
\label{sec:FHN-convergence}
A notebook (accessible from the link: \href{https://nbviewer.jupyter.org/github/FarmHJ/numerical-solver/blob/main/examples/fhn_model_convergence.ipynb}{\underline{\emph{Fitzhugh-Nagumo convergence notebook}}}) is created to test the convergence of the solution of the Fitzhugh-Nagumo model. Since the model has no analytical solution, it is tested against a reference solution. The reference solutions are assumed to be sufficiently accurate. For methods with fixed step size, which are the one-step methods and predictor-corrector method, the reference solutions are constructed by using a much smaller step size of $1^{-7}$, as compared to $1^{-5}$, the smallest step size for other numerical solutions. For methods with adaptive step size, reference solutions are obtained by using a much smaller tolerance value of $1^{-8}$, as compared to $1^{-5}$, the smallest tolerance value for other numerical solutions. This notebook shows the numerical solution computed for different methods at different step sizes or tolerance values. The numerical solutions are then compared with their respective reference solutions. In both methods, the error decreases as the step size or the tolerance value decreases, as shown in Figure \ref{fig:Euler_explicit_error} and Figure \ref{fig:adaptive_error}.

\begin{figure}
    \includegraphics[width=0.95\columnwidth]{FHN_Euler_explicit_error_behaviour}
    \caption{Error at a mesh point for Euler's explicit method. The Fitzhugh-Nagumo model Eqs.~\eqref{eqn:FHN}-\eqref{eqn:FHN-end} is solved with Euler's explicit method at different step sizes. The error is the absolute difference between reference solution and numerical solution at $x=0.7$. The reference solution is taken at step size of $1^{-7}$. Note that $\log(1^{-7})\simeq-16.118$.}
    \label{fig:Euler_explicit_error}
 \end{figure}
\begin{figure}
   \includegraphics[width=0.95\columnwidth]{FHN_adaptive_error_behaviour}
   \caption{Sum of error for adaptive method BS23. The Fitzhugh-Nagumo model Eqs.~\eqref{eqn:FHN}-\eqref{eqn:FHN-end} is solved with BS23 method at different absolute tolerance values. The relative tolerance is fixed at $1^{-15}$. The error is the sum of absolute difference between reference solution and numerical solution at all mesh points. The reference solution is taken at relative tolerance of $1^{-8}$. Note that $\log(1^{-8})\simeq-18.421$.}
   \label{fig:adaptive_error}
\end{figure}


\input{../../style/preamble}
\input{../../latex-math/basic-math}
\input{../../latex-math/basic-ml}

\newcommand{\titlefigure}{figure/lasso_ridge_enet_2d.png}
\newcommand{\learninggoals}{
  \item Know the elastic net as compromise between Ridge and Lasso regression
  \item Know regularized logistic regression
}

\title{Introduction to Machine Learning}
\date{}

\begin{document}

\lecturechapter{Elastic Net and Regularization for GLMs}
\lecture{Introduction to Machine Learning}

% \section{Elastic Net}

\begin{vbframe} {Elastic Net}

Elastic Net combines the $L_1$ and $L_2$ penalties:

$$
\mathcal{R}_{\text{elnet}}(\thetab) =  \sumin (\yi - \thetab^\top \xi)^2 + \lambda_1 \|\thetab\|_1 + \lambda_2 \|\thetab\|_2^2.
$$

\begin{figure}
\includegraphics[width=0.6\textwidth]{figure/lasso_ridge_enet_2d.png}\\
\end{figure}

\begin{itemize}
\item Correlated predictors tend to be either selected or zeroed out together.
\item Selection of more than $n$ features possible for $p>n$.
\end{itemize}

\framebreak
\footnotesize
Simulating two examples with each 50 data sets and 100 observations each: \\

$$\ydat =\Xmat \boldsymbol{\beta}+\sigma \epsilon, \quad \epsilon \sim N(0,1), \quad \sigma = 1$$

  \begin{columns}
\begin{column}{0.5\textwidth}
\begin{center}
\textbf{Ridge} performs better for: \\
$\boldsymbol{\beta}=(\underbrace{2,\ldots,2}_{5},\underbrace{0,\ldots,0}_{5})$\\
$ \operatorname{corr}(\Xmat_{i},\Xmat_{j})=0.8^{|i-j|}$ for all $i$ and $j$
  \end{center}
\end{column}
\begin{column}{0.5\textwidth}
\begin{center}
\textbf{Lasso} performs better for: \\
$\boldsymbol{\beta}=(2, 2, 2,\underbrace{0,\ldots,0}_{7})$ \\
$\operatorname{corr}(\Xmat_{i},\Xmat_{j})= 0$ for all $i \neq j$, otherwise 1
\end{center}
\end{column}
\end{columns}

\begin{figure}
\includegraphics[width=1\textwidth]{figure_man/elastic-net02.png}\\
\end{figure}

\framebreak

\begin{figure}
\includegraphics[width=0.9\textwidth]{figure_man/elastic-net03.png}\\
\end{figure}

\normalsize
Since Elastic Net offers a compromise between Ridge and Lasso, it is suitable for both data situations.

\end{vbframe}

% \section{Regularized Logistic Regression}

\begin{vbframe}{Regularized Logistic Regression}

Regularizers can be added very flexibly to basically any model which is based on ERM.

\lz

Hence, we can, e.g., construct $L_1$- or $L_2$-penalized logistic regression to enable coefficient shrinkage and variable selection in this model.

% \lz
% We can add a regularizer to the risk of logistic regression

\begin{align*}
\riskrt &= \risket + \lambda \cdot J(\thetab) \\
%&= \sumin \mathsf{log} \left[1 + \exp \left(-\yi f\left(\left.\xi~\right|~ \thetab\right)\right)\right] + \lambda \cdot J(\thetab) \\
&= \sumin \mathsf{log}\left[1 + \mathsf{exp}\left(-2\yi f\left(\left.\xi~\right|~ \thetab\right)\right)\right] + \lambda \cdot J(\thetab)
\end{align*}

% The other parts of the logistic regression remain exactly the same,
% except for the fitting algorithm to find \(\hat{\thetab}_{\text{reg}}\) (no closed-form solution, numerical optimization methods are necessary).

\end{frame}

\begin{frame}{Regularized Logistic Regression}

We fit a logistic regression model using polynomial features for \(x_1\)
and \(x_2\) with maximum degree of \(7\). We add an $L_2$ penalty. We
see for

\begin{itemize}

\item
  \(\lambda = 0\): The unregularized model seems to overfit.
\item
  \(\lambda = 0.0001\): Regularization helps to learn the underlying
  mechanism.
\item
  \(\lambda = 1\): The real data-generating process is captured very well.
\end{itemize}

\scriptsize

\begin{figure}
\includegraphics[width=0.8\textwidth]{figure_man/logistic-reg.png}\\
\end{figure}

\normalsize

\end{vbframe}

\endlecture
\end{document}


\subsection{Constraint Satisfaction Problem}

A CSP problem is one where we don't care about the path, we just want to identify the goal state.

For example, solving a sudoku

\subsection{Defining a CSP}

A CSP has:

\begin{itemize}
\item Variables \(X_i\).
\item Domain for each variable \(D_i\).
\item Constraints \(C_j\).
\end{itemize}

In a CSP there are a range of variables each with a domain. There are on top constraints on combinations of values.

A solution does not violate any constraint.

To solve, start with no allocations of variables. successor function assigns a value to an unassigned variable. goal test

Use heuristic minimum remaining value MRV: choose variable with fewest remaining legal values

Least constraining value: choose item in domain which constrains the least other moves

Forward checking. keep track of remaining legal moves for each variable. terminate if none left

After each move, update legal moves for each

Implement all this with recursive backtrack function, which returns a solution or failure. This is a depth first search

\subsection{Arc-consistency}

X-Y is arc consistent if all of domain of X is consistnet with some value of Y.

\subsection{Node-consistency}

X is node consistent if all of domain satisfies all unary constraint.

\subsection{Path-consistency}

Arc consistency for additional variables.

\subsection{Constraint propagation}

Constraint propagation can be used to prevent bad choices

We can check for:

\begin{itemize}
\item node-consistency
\item arc-consistency
\item path-consistency
\end{itemize}

\subsection{AC-3}

AC-3 algorihm makes a CSP arc-consistent

Take all arcs.

It may be possible to break the problem down into sub problems, making the problems much easier to solve.

Can do this before/after other algorithm.


\chapter{Networks from Distances}

So far we have been using existing network graphs. But we can also generate a graph from regular tabular data. Let us take the \emph{iris} data set. For now, we only have flowers as rows and their characteristics as columns. So how do we transform this to nodes and edges?

By looking at similarities. Rows will be nodes and the edge will be created between two rows if their similarity is above a certain threshold. Nice, isn't it?

\begin{figure}[h]
    \centering
    \includegraphics[width=\linewidth]{workflow.png}
    \caption{$\;$}
\end{figure}

First, we need distance matrix, which we can compute with the \widget{Distances} widget. Let's stick with Euclidean distances as it works well with this 4-dimensional data set.

Then pass the distance matrix to \widget{Network from Distances}. The widget enables setting the distances threshold, which we will set to 2.0 percentiles (meaning we keep the top 2\% of the edges that connect the most similar nodes).

\begin{figure}[h]
    \centering
    \includegraphics[scale=0.45]{net-from-distances.png}
    \caption{$\;$}
\end{figure}

\newpage

Finally, pass the constructed graph to \widget{Network Explorer}. Remember, this is not a scatter plot. The position of the points is not mapped to any sort of attribute. In the graph, you can simply see which elements are most similar to one another.

\begin{figure*}[h]
    \centering
    \includegraphics[width=\linewidth]{network-explorer.png}
    \caption{$\;$}
\end{figure*}


\section{Background}
\frame{\tableofcontents[currentsection, hideothersubsections]}

\begin{frame}
\frametitle{Background: MDP}
\textbf{The environment}, $E$: \\
a Markov decision process with a state space $S$, action space $A = \mathbb{R}^N$,
an initial state distribution $p(s_1)$, transition dynamics $p(s_{t+1}|s_t, a_t)$, and
reward function $r(s_t, a_t)$.
\vspace{2.5mm}

\textbf{At each discrete timestep} $t$, \\
the agent receives an observation $x_t = s_t$ (fully observable),
takes an action $a_t$ and receives a scalar reward $r_t$.
\vspace{2.5mm}

\textbf{The return from a state}: \\
$R_t= \sum_{i=t}^T  \gamma^{(i-t)} r(s_i, a_i)$ with a discounting factor $\gamma \in [0, 1]$.
\vspace{2.5mm}

\textbf{Goal}: \\
to learn a policy, $\pi: S \mapsto P(A)$, which
maximizes $J = \mathbb{E}_{r_i,s_i \sim E,a_i \sim \pi} [R_1]$.
\vspace{2.5mm}

\textbf{Action-value function}:\\
$Q^{\pi}(s_t,a_t) = \mathbb{E}_{r_{i \ge t},s_{i>t} \sim E,a_{i>t} \sim \pi} [R_t|s_t,a_t]$.
\end{frame}

\begin{frame}
\frametitle{Background: MDP with deterministic policy}
Bellman equation (the recursive relationship):
\begin{equation}
Q^{\pi} (s_t,a_t) = \mathbb{E}_{r_{t},s_{t+1} \sim E} \Big[ r(s_t,a_t) + \gamma \mathbb{E}_{a_{t+1} \sim \pi} [Q^{\pi}(s_{t+1},a_{t+1})] \Big]
\end{equation}

If the target policy is deterministic, $\mu: S \mapsto A$, then:
\begin{equation}
Q^{\mu} (s_t,a_t) = \mathbb{E}_{r_{t},s_{t+1} \sim E} \Big[ r(s_t,a_t) + \gamma Q^{\mu}(s_{t+1},\mu(s_{t+1})) \Big]
\end{equation}

Thus:
\begin{itemize}
\item the expectation depends only on the environment $E$,
\item possible to learn $Q^{\mu}$ off-policy, using transitions which
are generated from a different stochastic behavior policy $\beta$.
\end{itemize}
\end{frame}

\begin{frame}
\frametitle{Background: Fn approximator}
Consider fn approximators parameterized by $\theta^Q$, \\
which we optimize by minimizing the loss:
\begin{equation} \label{equ:qloss}
L(\theta^Q) =\mathbb{E}_{s_t \sim \rho^{\beta}, a \sim \beta, r_t \sim E} \Big[ \Big( Q (s_t,a_t|\theta^Q) - y_t \Big)^2 \Big]
\end{equation}
where:\\
$y_t = r(s_t,a_t) + \gamma Q(s_{t+1},\mu(s_{t+1}) | \theta^Q)$, and \\
$\rho^{\beta}$: the discounted state visitation distribution for a policy $\beta$.
\vspace{5mm}

Typically, ignore the fact that $y_t$ is also dependent on $\theta^Q$.\\
\end{frame}

\begin{frame}
\frametitle{Background: Actor-critic approach}
\begin{figure}
    \centering
    \includegraphics[scale=0.35]{actorcritic_arch}
\end{figure}
\end{frame}

% \begin{frame}
% \frametitle{Background: Actor-critic \cite{Sutton1998}}
% The Actor-Critic Algorithm is essentially a hybrid method to combine the policy gradient method and the value function method together.
% The policy function is known as the actor, while the value function is referred to as the critic.
% Essentially, the actor produces the action aa given the current state of the environment ss, while the critic produces a signal to criticizes the actions made by the actor.
% \end{frame}

% deep Q-network:
% \begin{itemize}
%   \item innovation:
%   \begin{itemize}
%     \item the network is trained off-policy with samples from a replay buffer to minimize correlations between samples;
%     the use of a replay buffer
%     \item the network is trained with a target Q network to give consistent targets during temporal difference backups.
%     a separate target network for calculating $y_t$
%   \end{itemize}
%   \item able to:
%   \begin{itemize}
%     \item solves problems with high-dimensional observation spaces,
%     \item can only handle discrete and low-dimensional action spaces.
%   \end{itemize}
% \end{itemize}


\subsection{Selection Based on Spline regression}
\begin{itemize}
\item Spline regression \cite{racine} was also considered as a basis for scatterplot clustering applying the following algorithm:
%\item In spline regression a curve $y=s(x)$ is represented as $\mathbf{y}_i=\mathbf{B}_i\mathbf{c}$ where
%\begin{itemize}
%\item $\mathbf{B}_i =\left[ B_{1p}\mathbf{x}_i,B_{2p}\mathbf{x}_i,\dots,B_{Lp}\mathbf{x}_i \right]$ the spline basis matrix and
%\item $\mathbf{c}$ is the vector of spline coefficients.
%\end{itemize}

%\item This suggests the following method (and algorithm) for detecting L--shaped genes based on \textbf{Clustering Spline Coefficients}:
\begin {enumerate}
\item Select genes with significant negative correlation.
\item For each selected gene fit a cubic splines regression model.
\item Obtain a distance matrix between all genes using the $1-\rho$ distance computed on spline coefficients.
\item Perform a hierarchical clustering and
\item Select genes in the \textit{L-shaped cluster(s)}.
\end{enumerate}
\end{itemize}


\documentclass{article}
\usepackage{amsmath}
\usepackage{amssymb}

\begin{document}
\section{2x2 complex matrix}
Consider the matrix
\[
  \left[
    \begin{array}{cc}
      \alpha & \beta\\
      \gamma & \delta
    \end{array}
  \right]
\]
The eigenvalues, $\lambda$, are solution of the characteristic equation
\[
  \alpha\delta(1-\lambda)^2 - \beta\gamma = \alpha\delta\lambda^2
  -2\alpha\delta\lambda + \alpha\delta - \beta\gamma = 0
\]
The solution depends only on the value of the product $\alpha\delta$
and we consider only the case $\delta=\alpha$ without loss of
generality.

\begin{align*}
  \lambda &= \frac{2\alpha^2 \pm \sqrt{4\alpha^4 -
            4\alpha^2(\alpha^2 - \beta\gamma)}}{2\alpha^2}\\
  &  = \alpha \pm \sqrt{\beta\gamma}
\end{align*}

There is a complex paire of eigenvalues if and only if $\beta$ and
$\gamma$ have opposite sign. The modulus of the eigenvalues is
\[
  |\lambda| = \sqrt{\alpha^2 + |\beta\gamma|}
\]
There is a paire of complex unit roots when $\alpha^2 + \beta\gamma =
1$ and the roots are smaller than one in modulus when $\alpha^2 + \beta\gamma <
1$
\begin{enumerate}
\item Generating a random 2x2 matrix with a paire of complex unit
  roots:
  \begin{itemize}
  \item Choose random $\alpha$ and $\beta$
  \item Set $\gamma = -\frac{1 - \alpha^2}{\beta}$
  \end{itemize}
\item Generating a random 2x2 matrxi with a paire of roots smalle than
  1 in modulus:
  \begin{itemize}
  \item Choose random $\alpha$ and $\beta$
  \item If $\beta < 0$, choose random $\gamma$ over $]0; -\frac{1 -
      \alpha^2}{\beta}[$,
  \item else choose random $\gamma$ over $]-\frac{1 -
      \alpha^2}{\beta}; 0[$
  \end{itemize}
\end{enumerate}
\end{document}

%%% Local Variables:
%%% mode: latex
%%% TeX-master: t
%%% End: