\part{Inference}

\chapter{Bayesian Data Analysis}\label{bayesian.chapter}

\noindent
\cite{GelmanEtAl:2013} provide the following
characterization of Bayesian data analysis.
%
\begin{quote}
  By Bayesian data analysis, we mean practical methods for making
  inferences from data using probability models for quantities we
  observe and about which we wish to learn.
\end{quote}
%
They go on to describe how Bayesian statistics differs from
frequentist approaches.
%
\begin{quote}
  The essential characteristic of Bayesian methods is their explicit
  use of probability for quantifying uncertainty in inferences based
  on statistical analysis.
\end{quote}
%
Because they view probability as the limit of relative frequencies of
observations, strict frequentists forbid probability statements about
parameters.  Parameters are considered fixed, not random.

Bayesians also treat parameters as fixed but unknown.  But unlike
frequentists, they make use of both prior distributions over
parameters and posterior distributions over parameters.  These prior
and posterior probabilities and posterior predictive probabilities are
intended to characterize knowledge about the parameters and future
observables.  Posterior distributions form the basis of Bayesian
inference, as described below.

\section{Bayesian Modeling}

\citep{GelmanEtAl:2013} break applied Bayesian modeling
into the following three steps.
%
\begin{enumerate}
\item  Set up a full probability model for all observable and
  unobservable quantities.  This model should be consistent with
  existing knowledge of the data being modeled and how it was
  collected.
\item Calculate the posterior probability of unknown quantities
  conditioned on observed quantities.  The unknowns may include
  unobservable quantities such as parameters and potentially
  observable quantities such as predictions for future observations.
\item Evaluate the model fit to the data.  This includes evaluating
  the implications of the posterior.
\end{enumerate}
%
Typically, this cycle will be repeated until a sufficient fit is
achieved in the third step.  Stan automates the calculations involved
in the second and third steps.

\section{Bayesian Inference}

\subsection{Basic Quantities}

The mechanics of Bayesian inference follow directly from Bayes's rule.
To fix notation, let $y$ represent observed quantities such as data
and let $\theta$ represent unknown quantities such as parameters and
future observations.  Both $y$ and $\theta$ will be modeled as random.
Let $x$ represent known, but unmodeled quantities such as constants,
hyperparameters, and predictors.

\subsection{Probability Functions}

The probability function $p(y,\theta)$ is the joint probability
function of the data $y$ and parameters $\theta$.  The constants and
predictors $x$ are implicitly understood as being part of the
conditioning.  The conditional probability function $p(y|\theta)$ of
the data $y$ given parameters $\theta$ and constants $x$ is called the
sampling probability function; it is also called the likelihood
function when viewed as a function of $\theta$ for fixed $y$ and $x$.

The probability function $p(\theta)$ over the parameters given the
constants $x$ is called the prior because it characterizes the probability
of the parameters before any data is observed.  The conditional
probability function $p(\theta|y)$ is called the posterior because
it characterizes the probability of parameters given observed data $y$
and constants $x$.

\subsection{Bayes's Rule}

The technical apparatus of Bayesian inference hinges on the following
chain of equations, known in various forms as Bayes's rule (where
again, the constants $x$ are implicit).
%
\[
\begin{array}{rcll}
p(\theta|y)  & =  & \displaystyle \frac{p(\theta,y)}{p(y)}
& \mbox{{} \ \ \ \ \ [definition of  conditional probability]}
\\[16pt]
& = & \displaystyle \frac{p(y|\theta) \, p(\theta)}{p(y)}
& \mbox{{} \ \ \ \ \ [chain rule]}
\\[16pt]
& = & \displaystyle \frac{p(y|\theta) \, p(\theta)}
                        {\int_{\Theta} p(y,\theta) \, d\theta}
& \mbox{{} \ \ \ \ \ [law of total probability]}
\\[16pt]
& = & \displaystyle \frac{p(y|\theta) \, p(\theta)}
                        {\int_{\Theta} p(y|\theta) \, p(\theta) \, d\theta}
& \mbox{{} \ \ \ \ \ [chain rule]}
\\[16pt]
& \propto & \displaystyle p(y|\theta) \, p(\theta)
& \mbox{{} \ \ \ \ \ [$y$ is fixed]}
\end{array}
\]
%
Bayes's rule ``inverts'' the probability of the posterior
$p(\theta|y)$, expressing it solely in terms of the likelihood
$p(y|\theta)$ and prior $p(\theta)$ (again, with constants and
predictors $x$ implicit).  The last step is important for Stan, which
only requires probability functions to be characterized up to a
constant multiplier.

\subsection{Predictive Inference}

The uncertainty in the estimation of parameters $\theta$ from the data
$y$ (given the model) is characterized by the posterior $p(\theta|y)$.
The posterior is thus crucial for Bayesian predictive inference.

If $\tilde{y}$ is taken to represent new, perhaps as yet unknown,
observations, along with corresponding constants and predictors
$\tilde{x}$, then the posterior predictive probability function is
given by
%
\[
p(\tilde{y}|y)
= \int_{\Theta} p(\tilde{y}|\theta)
                \, p(\theta|y) \, d\theta.
\]
Here, both the original constants and predictors $x$ and the new
constants and predictors $\tilde{x}$ are implicit.  Like the posterior
itself, predictive inference is characterized probabilistically.
Rather than using a point estimate of the parameters $\theta$,
predictions are made based on averaging the predictions over a range
of $\theta$ weighted by the posterior probability $p(\theta|y)$ of
$\theta$ given data $y$ (and constants $x$).

The posterior may also be used to estimate event probabilities.  For
instance, the probability that a parameter $\theta_k$ is greater than
zero is characterized probabilistically by
%
\[
\mbox{Pr}[\theta_k > 0]
= \int_{\Theta} \mbox{I}(\theta_k > 0) \, p(\theta|y) \, d\theta.
\]
%
The indicator function, $\mbox{I}(\phi)$, evaluates to one if the
proposition $\phi$ is true and evaluates to zero otherwise.

Comparisons involving future observables may be carried out in
the same way.  For example, the probability that $\tilde{y}_n >
\tilde{y}_{n'}$ can be characterized using the posterior predictive
probability function as
\[
\mbox{Pr}[\tilde{y}_n > \tilde{y}_{n'}]
= \int_{\Theta} \int_{Y} \mbox{I}(\tilde{y}_n > \tilde{y}_{n'}) \,
p(\tilde{y}|\theta) p(\theta|y) \, d\tilde{y} \, d\theta.
\]

\subsection{Posterior Predictive Checking}

After the parameters are fit to data, they can be used to simulate a
new data set by running the model inferences in the forward
direction.  These replicated data sets can then be compared to the
original data either visually or statistically to assess model fit
\citep[Chapter 6]{GelmanEtAl:2013}.

In Stan, posterior simulations can be generated in two ways.  The
first approach is to treat the predicted variables as parameters and
then define their distributions in the model block.  The second
approach, which also works for discrete variables, is to generate
replicated data using random-number generators in the generated
quantities block.

\chapter{Penalized Maximum Likelihood Point Estimation}\label{mle.chapter}

\noindent
This chapter defines the workhorses of non-Bayesian estimation,
maximum likelihood and penalized maximum likelihood, and relates them
to Bayesian point estimation based on posterior means, medians, and
modes.  Such estimates are called ``point estimates'' because they
are composed of a single value for the model parameters $\theta$
rather than a posterior distribution.

Stan's optimizer can be used to implement (penalized) maximum
likelihood estimation for any likelihood function and penalty function
that can be coded in Stan's modeling language.  Stan's optimizer can
also be used for point estimation in Bayesian settings based on
posterior modes.  Stan's Markov chain Monte Carlo samplers can be used
to implement point inference in Bayesian models based on posterior
means or medians.

\section{Maximum Likelihood Estimation}\label{mle.section}

Given a likelihood function $p(y|\theta)$ and a fixed data vector $y$,
the maximum likelihood estimate (MLE) is the parameter vector $\hat{\theta}$
that maximizes the likelihood, i.e.,
\[
\hat{\theta} = \mbox{argmax}_{\theta} \ p(y|\theta).
\]
It is usually more convenient to work on the log scale.
An equivalent%
%
\footnote{The equivalence follows from the fact that densities are
  positive and the log function is strictly monotonic, i.e.,
  $p(y|\theta) \geq 0$ and for all $a, b > 0$, $\log a > \log b$ if and
  only if $a > b$.}
%
formulation of the MLE is
%
\[
\hat{\theta} = \mbox{argmax}_{\theta} \ \log p(y|\theta).
\]

\subsection{Existence of Maximum Likelihood Estimates}

Because not all functions have unique maximum values, maximum
likelihood estimates are not guaranteed to exist.  As discussed in
\refchapter{problematic-posteriors}, this situation can arise when
%
\begin{itemize}
\item there is more than one point that maximizes the likelihood function,
\item the likelihood function is unbounded, or
\item the likelihood function is bounded by an asymptote that is never
  reached for legal parameter values.
\end{itemize}
%
These problems persist with the penalized maximum likelihood estimates
discussed in the next section, and Bayesian posterior modes as
discussed in the following section.

\subsection{Example: Linear Regression}

Consider an ordinary linear regression problem with an $N$-dimensional
vector of observations $y$, an $(N \times K)$-dimensional data matrix
$x$ of predictors, a $K$-dimensional parameter vector $\beta$ of
regression coefficients, and a real-valued noise scale $\sigma > 0$,
with log likelihood function
\[
\log p(y|\beta,x) = \sum_{n=1}^N \log \distro{Normal}(y_n|x_n \beta,
\sigma).
\]
%
The maximum likelihood estimate for $\theta = (\beta,\sigma)$ is just
\[
(\hat{\beta},\hat{\sigma})
\ = \
\mbox{argmax}_{\beta,\sigma}
\log p(y|\beta,\sigma,x) = \sum_{n=1}^N \log \distro{Normal}(y_n|x_n \beta, \sigma).
\]

\subsubsection{Squared Error}

A little algebra on the log likelihood function shows that the
marginal maximum likelihood estimate $\hat{\theta} =
(\hat{\beta},\hat{\sigma})$ can be equivalently formulated for
$\hat{\beta}$ in terms of least squares.  That is, $\hat{\beta}$ is
the value for the coefficient vector that minimizes the sum of squared
prediction errors,
%
\[
\hat{\beta}
\ = \
\mbox{argmin}_{\beta} \sum_{n=1}^N (y_n - x_n \beta)^2
\ = \
\mbox{argmin}_{\beta} (y - x \beta)^{\top} (y - x\beta).
\]
%
The residual error for data item $n$ is the difference between the
actual value and predicted value, $y_n - x_n \hat{\beta}$.  The
maximum likelihood estimate for the noise scale, $\hat{\sigma}$ is
just the square root of the average squared residual,
\[
\hat{\sigma}^2
\ = \
\frac{1}{N} \sum_{n=1}^N \left( y_n - x_n \hat{\beta} \right)^2
\ = \
\frac{1}{N} (y - x \hat{\beta})^{\top} (y - x\hat{\beta}).
\]

\subsubsection{Minimizing Squared Error in Stan}

The squared error approach to linear regression can be directly coded
in Stan with the following model.
%
\begin{stancode}
data {
  int<lower=0> N;
  int<lower=1> K;
  vector[N] y;
  matrix[N,K] x;
}
parameters {
  vector[K] beta;
}
transformed parameters {
  real<lower=0> squared_error;
  squared_error = dot_self(y - x * beta);
}
model {
  target += -squared_error;
}
generated quantities {
  real<lower=0> sigma_squared;
  sigma_squared = squared_error / N;
}
\end{stancode}
%
Running Stan's optimizer on this model produces the MLE for the linear
regression by directly minimizing the sum of squared errors and using
that to define the noise scale as a generated quantity.

By replacing \code{N} with \code{N-1} in the denominator of the
definition of \code{sigma\_squared}, the more commonly supplied
unbiased estimate of $\sigma^2$ can be calculated; see
\refsection{estimation-bias} for a definition of estimation bias and a
discussion of estimating variance.

\section{Penalized Maximum Likelihood Estimation}

There is nothing special about a likelihood function as far as the
ability to perform optimization is concerned.  It is common among
non-Bayesian statisticians to add so-called ``penalty'' functions
to log likelihoods and optimize the new function.  The penalized
maximum likelihood estimator for a log likelihood function
$\log p(y|\theta)$ and penalty function $r(\theta)$ is defined to be
\[
\hat{\theta} = \mbox{argmax}_{\theta} \log p(y|\theta) - r(\theta).
\]
The penalty function $r(\theta)$ is negated in the maximization so
that the estimate $\hat{\theta}$ balances maximizing the log
likelihood and minimizing the penalty.  Penalization is sometimes
called ``regularization.''

\subsection{Examples}\label{penalized-mle-examples}

\subsubsection{Ridge Regression}

Ridge regression \citep{HoerlKennard:1970} is based on penalizing the
Euclidean length of the coefficient vector $\beta$. The ridge penalty
function is
%
\[
r(\beta)
\ = \
\lambda \, \sum_{k=1}^K \beta_k^2
\ = \
\lambda \, \beta^{\top} \beta,
\]
%
where $\lambda$ is a constant tuning parameter that determines the
magnitude of the penalty.

Therefore, the penalized maximum likelihood estimate for ridge
regression is just
%
\[
(\hat{\beta},\hat{\sigma})
\ = \
\mbox{argmax}_{\beta,\sigma} \,
 \sum_{n=1}^N \log \distro{Normal}(y_n|x_n \beta, \sigma) - \lambda
 \sum_{k=1}^K \beta_k^2
\]
%
The ridge penalty is sometimes called L2 regularization or shrinkage,
because of its relation to the L2 norm.

Like the basic MLE for linear regression, the ridge regression
estimate for the coefficients $\beta$ can also be formulated in terms
of least squares,
%
\[
\hat{\beta}
\ = \
\mbox{argmin}_{\beta} \, \sum_{n=1}^N (y_n - x_n \beta)^2 + \sum_{k=1}^K \beta_k^2
\ = \
\mbox{argmin}_{\beta} \, (y - x\beta)^{\top} (y - x\beta) +
\lambda \beta^{\top} \beta.
\]

The effect of adding the ridge penalty function is that the ridge
regression estimate for $\beta$ is a vector of shorter length, or in
other words, $\hat{\beta}$ is shrunk.  The ridge estimate does not
necessarily have a smaller absolute $\beta_k$ for each $k$, nor does
the coefficient vector necessarily point in the same direction as the
maximum likelihood estimate.

In Stan, adding the ridge penalty involves adding its magnitude as a
data variable and the penalty itself to the model block,
%
\begin{stancode}
data {
  // ...
  real<lower=0> lambda;
}
// ...
model {
  // ...
  target += - lambda * dot_self(beta);
}
\end{stancode}
%
The noise term calculation remains the same.

\subsubsection{The Lasso}

The lasso \citep{Tibshirani:1996} is an alternative to ridge
regression that applies a penalty based on the sum of the absolute
coefficients, rather than the sum of their squares,
\[
r(\beta) = \lambda \sum_{k=1}^K | \beta_k |.
\]
The lasso is also called L1 shrinkage due to its relation to the L1
norm, which is also known as taxicab distance or Manhattan distance.

Because the derivative of the penalty does not depend on the value of
the $\beta_k$,
\[
\frac{d}{d\beta_k} \lambda \sum_{k=1}^K | \beta_k | =
\mbox{signum}(\beta_k),
\]
it has the effect of shrinking parameters all the way to 0 in maximum
likelihood estimates.  Thus it can be used for variable selection as
well as just shrinkage.%
%
\footnote{In practice, Stan's gradient-based optimizers are not
  guaranteed to produce exact zero values; see
  \cite{LangfordEtAl:2009} for a discussion of getting exactly zero
  values with gradient descent.}
%
The lasso can be implemented in Stan just as easily as ridge
regression, with the magnitude declared as data and the penalty added
to the model block,
%
\begin{stancode}
data {
  // ...
  real<lower=0> lambda;
}
// ...
model {
  // ...
  for (k in 1:K)
    target += - lambda * fabs(beta[k]);
}
\end{stancode}

\subsubsection{The Elastic Net}

The naive elastic net \citep{ZouHastie:2005} involves a weighted
average of ridge and lasso penalties, with a penalty function
\[
r(\beta)
= \lambda_1 \sum_{k=1}^K |\beta_k|
+ \lambda_2 \sum_{k=1}^K \beta_k^2.
\]
The naive elastic net combines properties of both ridge regression and
the lasso, providing both identification and variable selection.

The naive elastic net can be implemented directly in Stan by combining
implementations of ridge regression and the lasso, as
%
\begin{stancode}
data {
  real<lower=0> lambda1;
  real<lower=0> lambda2;
  // ...
}
// ...
model {
  // ...
  for (k in 1:K)
    target += -lambda1 * fabs(beta[k]);
  target += -lambda2 * dot_self(beta);
}
\end{stancode}
%
Note that the signs are negative in the program because $r(\beta)$ is
a penalty function.

The elastic net \citep{ZouHastie:2005} involves adjusting the final estimate for
$\beta$ based on the fit $\hat{\beta}$ produced by the naive elastic
net.  The elastic net estimate is
\[
\hat{\beta} = (1 + \lambda_2) \beta^*
\]
where $\beta^{*}$ is the naive elastic net estimate.

To implement the elastic net in Stan, the data, parameter, and model
blocks are the same as for the naive elastic net.  In addition, the
elastic net estimate is calculated in the generated quantities block.
%
\begin{stancode}
generated quantities {
  vector[K] beta_elastic_net;
  // ...
  beta_elastic_net = (1 + lambda2) * beta;
}
\end{stancode}
%
The error scale also needs to be calculated in the generated
quantities block based on the elastic net coefficients
\code{beta\_elastic\_net}.

\subsubsection{Other Penalized Regressions}

It is also common to use penalty functions that bias the coefficient
estimates toward values other than 0, as in the estimators of
\cite{JamesStein:1961}.  Penalty functions can also be used to bias
estimates toward population means; see
\citep{EfronMorris:1975,Efron:2012}.  This latter approach is similar
to the hierarchical models commonly employed in Bayesian statistics.

\section{Estimation Error, Bias, and Variance}\label{estimation-bias.section}

An estimate $\hat{\theta}$ depends on the particular data $y$ and
either the log likelihood function, $\log p(y|\theta)$, penalized log
likelihood function $\log p(y|\theta) - r(\theta)$, or log probability
function $\log p(y,\theta) = \log p(y,\theta) + \log p(\theta)$.  In
this section, the notation $\hat{\theta}$ is overloaded to indicate
the estimator, which is an implicit function of the data and
(penalized) likelihood or probability function.

\subsection{Estimation Error}

For a particular observed data set $y$ generated according to true
parameters $\theta$, the estimation error is the difference between
the estimated value and true value of the parameter,
\[
\mbox{err}(\hat{\theta}) = \hat{\theta} - \theta.
\]
%

\subsection{Estimation Bias}

For a particular true parameter value $\theta$ and a likelihood
function $p(y|\theta)$, the expected estimation error averaged over
possible data sets $y$ according to their density under the likelihood
is
%
\[
\mathbb{E}_{p(y|\theta)}[\hat{\theta}]
\ = \
\int \left( \mbox{argmax}_{\theta'} p(y|\theta') \right) p(y|\theta) dy.
\]

An estimator's bias is the expected estimation error,
%
\[
\mathbb{E}_{p(y|\theta)}[\hat{\theta} - \theta]
\ = \
\mathbb{E}_{p(y|\theta)}[\hat{\theta}] - \theta
\]
%
The bias is a multivariate quantity with the same dimensions as
$\theta$.  An estimator is unbiased if its expected estimation error
is zero and biased otherwise.

\subsubsection{Example: Estimating a Normal Distribution}

Suppose a data set of observations $y_n$ for $n \in 1{:}N$ drawn from
a normal distribution.  This presupposes a model $y_n \sim
\distro{Normal}(\mu,\sigma)$, where both $\mu$ and $\sigma > 0$ are
parameters.  The log likelihood is just
\[
\log p(y|\mu,\sigma) = \sum_{n=1}^N \log
\distro{Normal}(y_n|\mu,\sigma).
\]
The maximum likelihood estimator for $\mu$ is just the sample mean,
i.e., the average of the samples,
\[
\hat{\mu} = \frac{1}{N} \sum_{n=1}^N y_n.
\]
The maximum likelihood estimate for the mean is unbiased.

The maximum likelihood estimator for the variance $\sigma^2$ is the
average of the squared difference from the mean,
\[
\hat{\sigma}^2 = \frac{1}{N} \sum_{n=1}^N (y_n - \hat{\mu})^2.
\]
The maximum likelihood for the variance is biased on the low side,
i.e.,
%
\[
\mathbb{E}_{p(y|\mu,\sigma)}[\hat{\sigma}^2] < \sigma.
\]
%
The reason for this bias is that the maximum likelihood estimate is
based on the difference from the estimated mean $\hat{\mu}$.  Plugging
in the actual mean can lead to larger sum of squared differences;  if
$\mu \neq \hat{\mu}$, then
\[
\frac{1}{N} \sum_{n=1}^N (y_n - \mu)^2
>
\frac{1}{N} \sum_{n=1}^N (y_n - \hat{\mu})^2.
\]

An alternative estimate for the variance is the sample variance, which
is defined by
\[
\hat{\mu} = \frac{1}{N-1} \sum_{n=1}^N (y_n - \hat{\mu})^2.
\]
This value is larger than the maximum likelihood estimate by a factor
of $N/(N-1)$.

\subsection{Estimation Variance}

The variance of component $k$ of an estimator $\hat{\theta}$ is
computed like any other variance, as the expected squared difference
from its expectation,
%
\[
\mbox{var}_{p(y|\theta})[\hat{\theta}_k]
\ = \
\mathbb{E}_{p(y|\theta})[\, (\hat{\theta}_k -
\mathbb{E}_{p(y|\theta)}[\hat{\theta}_k])^2 \,].
\]
%
The full $K \times K$ covariance matrix for the estimator is thus
defined, as usual, by
%
\[
\mbox{covar}_{p(y|\theta)}[\hat{\theta}]
\ = \
\mathbb{E}_{p(y|\theta})[\, (\hat{\theta} - \mathbb{E}[\hat{\theta}]) \,
                         (\hat{\theta} -
                         \mathbb{E}[\hat{\theta}])^{\top} \, ].
\]

Continuing the example of estimating the mean and variance of a normal
distribution based on sample data, the maximum likelihood estimator
(i.e., the sample mean) is the unbiased estimator for the mean $\mu$
with the lowest variance; the Gauss-Markov theorem establishes this
result in some generality for least-squares estimation, or
equivalently, maximum likelihood estimation under an assumption of
normal noise; see \citep[Section~3.2.2]{HastieTibshiraniFriedman:2009}.

\chapter{Bayesian Point Estimation}

There are three common approaches to Bayesian point estimation based
on the posterior $p(\theta|y)$ of parameters $\theta$ given observed
data $y$: the mode (maximum), the mean, and the median.

\section{Posterior Mode Estimation}

This section covers estimates based on the parameters $\theta$ that
maximize the posterior density, and the next sections continue with
discussions of the mean and median.

An estimate based on a model's posterior mode can be defined by
%
\[
\hat{\theta} = \mbox{argmax}_{\theta} \, p(\theta|y).
\]
%
When it exists, $\hat{\theta}$ maximizes the posterior density of the
parameters given the data.  The posterior mode is sometimes called the
``maximum a posteriori'' (MAP) estimate.

As discussed in \refchapter{problematic-posteriors} and
\refsection{mle}, a unique posterior mode might not
exist---there may be no value that maximizes the posterior mode or
there may be more than one.  In these cases, the posterior mode
estimate is undefined.  Stan's optimizer, like most optimizers, will
have problems in these situations.  It may also return a locally
maximal value that is not the global maximum.

In cases where there is a posterior mode, it will correspond to a
penalized maximum likelihood estimate with a penalty function equal to
the negation of the log prior.  This is because Bayes's rule,
\[
p(\theta|y) = \frac{p(y|\theta) \, p(\theta)}{p(y)},
\]
ensures that
%
\begin{eqnarray*}
\mbox{argmax}_{\theta} \ p(\theta|y)
& = &
\mbox{argmax}_{\theta} \ \frac{p(y|\theta) \, p(\theta)}{p(y)}
\\[6pt]
& = &
\mbox{argmax}_{\theta} \ p(y|\theta) \, p(\theta),
\end{eqnarray*}
%
and the positiveness of densities and the strict monotonicity of log
ensure that
\[
\mbox{argmax}_{\theta} \ p(y|\theta) \, p(\theta)
\ = \
\mbox{argmax}_{\theta} \ \log p(y|\theta) + \log p(\theta).
\]
%

In the case where the prior (proper or improper) is uniform, the
posterior mode is equivalent to the maximum likelihood estimate.

For most commonly used penalty functions, there are probabilistic
equivalents.  For example, the ridge penalty function corresponds to a
normal prior on coefficients and the lasso to a Laplace prior.  The
reverse is always true---a negative prior can always be treated as a
penalty function.

\section{Posterior Mean Estimation}

A standard Bayesian approach to point estimation is to use the
posterior mean (assuming it exists), defined by
%
\[
\hat{\theta} = \int \theta \, p(\theta|y) \, d\theta.
\]
%
The posterior mean is often called {\it the}\ Bayesian estimator,
because it's the estimator that minimizes the expected square error of
the estimate.

An estimate of the posterior mean for each parameter is returned by
Stan's interfaces;  see the RStan, CmdStan, and PyStan user's guides
for details on the interfaces and data formats.

Posterior means exist in many situations where posterior
modes do not exist.  For example, in the $\distro{Beta}(0.1, 0.1)$
case, there is no posterior mode, but posterior mean is well defined
with value 0.5.

A situation where posterior means fail to exist but posterior modes do
exist is with a posterior with a Cauchy distribution
$\distro{Cauchy}(\mu,\tau)$.  The posterior mode is $\mu$, but the
integral expressing the posterior mean diverges.  Such diffuse priors
rarely arise in practical modeling applications; even with a Cauchy
Cauchy prior for some parameters, data will provide enough constraints
that the posterior is better behaved and means exist.

Sometimes when posterior means exist, they are not meaningful, as in
the case of a multimodal posterior arising from a mixture model or in
the case of a uniform distribution on a closed interval.

\section{Posterior Median Estimation}

The posterior median (i.e., 50th percentile or 0.5 quantile) is
another popular point estimate reported for Bayesian models.  The
posterior median minimizes the expected absolute error of estimates.
These estimates are returned in the various Stan interfaces;  see the
RStan, PyStan and CmdStan user's guides for more information on
format.

Although posterior medians may fail to be meaningful, they often exist
even where posterior means do not, as in the Cauchy distribution.

\chapter{Variational Inference}\label{vi-advanced.chapter}

\noindent
Stan implements an automatic variational inference algorithm based on
transforming variables to the unconstrained scale and using a normal
approximating distribution.

Classical variational inference algorithms are difficult to derive. We
must first define the family of approximating densities, and then
calculate model-specific quantities relative to that family to solve
the variational optimization problem.  Both steps require expert
knowledge.  The resulting algorithm is tied to both the model and the
chosen approximation.

We begin by briefly describing the classical variational inference framework.
For a thorough exposition, please refer to
\citet{Jordan:1999,Wainwright-Jordan:2008}; for a textbook presentation, please
see \citet{Bishop:2006}. We follow with a high-level description of Automatic
Differentiation Variational Inference (ADVI). For more details, see
\citep{Kucukelbir:2015}.

\section{Classical Variational Inference}

Variational inference approximates the
posterior $p(\theta \, | \, y)$ with a simple, parameterized distribution
$q(\theta \, | \, \phi)$. It matches the approximation to the
true posterior by minimizing the Kullback-Leibler (KL) divergence,
%
\[
  \phi^* = \argmin_\phi
  \KL{q(\theta \, | \, \phi) }{ p(\theta \mid y)}.
\]
%
Typically the KL divergence lacks an analytic, closed-form solution.
Instead we maximize a proxy to the KL divergence, the evidence lower bound
(ELBO)
%
\[
  \mathcal{L} (\phi)
  =
  \E_{q (\theta)} \big[ \log p (y,\theta) \big]
  -
  \E_{q (\theta)} \big[ \log q (\theta\, | \,\phi) \big].
\]
%
The first term is an expectation of the log
joint density under the approximation, and the second is the entropy of the
variational density. Maximizing the ELBO minimizes the KL
divergence \citep{Jordan:1999,Bishop:2006}.

\section{Automatic Variational Inference}

ADVI maximizes the ELBO in the real-coordinate space. Stan transforms the
parameters from (potentially) constrained domains to
the real-coordinate space. We denote the combined transformation as
$T:\theta \to \zeta$, with the $\zeta$ variables living in $\mathbb{R}^K$.
The variational objective (ELBO) becomes
%
\[
  \mathcal{L}(\phi)
  =
  \E_{q(\zeta\,|\,\phi)}
  \bigg[
  \log p (y, T^{-1}(\zeta))
  +
  \log \big| \det J_{T^{-1}}(\zeta) \big|
  \bigg]
  -
  \E_{q (\zeta\, | \,\phi)} \big[ \log q (\zeta\, | \,\phi) \big].
\]
%
Since the $\zeta$ variables live in the real-coordinate space, we can choose a
fixed family for the variational distribution. We choose a fully-factorized
Gaussian,
%
\[
  q(\zeta \, | \, \phi)
  =
  \distro{Normal}\left(\zeta \, | \, \mu, \sigma\right)
  =
  \prod_{k=1}^K
  \distro{Normal}
  \left(\zeta_k \, | \, \mu_k, \sigma_k\right),
\]
%
where the vector
$\phi = (\mu_{1},\cdots,\mu_{K}, \sigma_ {1},\cdots,\sigma_{K})$
concatenates the mean and standard deviation of each Gaussian factor.
This reflects the ``mean-field'' assumption in classical variational
inference algorithms; we will refer to this particular decomposition
as the \texttt{meanfield} option.

The transformation $T$ maps the support of the parameters to the real
coordinate space. Thus, its inverse $T^{-1}$ maps back to the support of the
latent variables. This implicitly defines the variational approximation in the
original latent variable space as
%
\[
\distro{Normal} \left(T(\theta) \, | \, \mu, \sigma\right)
\big| \det J_{T}(\theta) \big|.
\]
This is, in general, not a Gaussian distribution.
This choice may call to mind the Laplace approximation
technique, where a second-order Taylor expansion around the
maximum-a-posteriori estimate gives a Gaussian approximation to the
posterior. However, they are not the same \citep{Kucukelbir:2015}.

The variational objective (ELBO) that we maximize is,
\[
  \mathcal{L}(\phi)
  =
  \E_{q(\zeta\, | \,\phi)}
  \bigg[
  \log p (y, T^{-1}(\zeta))
  +
  \log \big| \det J_{T^{-1}}(\zeta) \big|
  \bigg]
  +
  \sum_{k=1}^K \log \sigma_k,
\]
where we plug in the analytic form for the Gaussian entropy and drop all terms
that do not depend on $\phi$. We discuss how we perform the maximization in
\refchapter{vi-algorithms}.


\documentclass[11pt, twoside]{article}

\usepackage{graphicx}
\usepackage{amsmath, amssymb}
\usepackage{enumerate}
\usepackage{titleps}
\usepackage[top=1.25in,bottom=1in,right=1in,left=1in]{geometry}
\usepackage[parfill]{parskip}
\usepackage{titling}
\usepackage[authoryear]{natbib}

\newpagestyle{ruled}
{\setfoot{}{\thepage}{} \footrule}
\pagestyle{ruled}

\setlength{\droptitle}{-4em}   % This is your set screw
\posttitle{\par\end{center}\vskip 0.5em}

\title{6.882 Proposal: Bayesian Reinforcement Learning}
\date{}
\author {Vickie Ye and Alexandr Wang}

\begin{document}
\maketitle

\section{Background and Motivation}
Because of the recent excitement regarding DeepMind's AlphaGo AI, we wanted to
explore reinforcement learning, in a Bayesian framework. We will use Strens's
ICML-2000 paper, \textit{A Bayesian Framework for Reinforcement Learning} and Engel's ICML-2005 paper, \textit{Reinforcement learning with Gaussian processes} as
guidelines as to how to reason about the reinforcement learning problems with Bayesian formulations.

We will study a toy reinforcement learning problem of navigating an unknown maze.
In the reinforcement learning problem, the agent must explore the surroundings
and determine the behavior that maximizes the expected return. In this case, our
agent must discover the structure of the maze while minimizing the time required
to find the end. A Markov decision process (MDP) model is traditionally used to
model the interactive system over $S$ the set of states, $A$ the set of actions,
$R: S \times A \rightarrow \mathbb{R}$ the reward function, and $T$ the transition
probabilities defined as
\begin{equation}
T(s, a, s') = P(X_{t+1} = s' | X_t = s, Y_t = a).
\end{equation}
We also define a quality function $Q$ as

\begin{equation}
Q(s, a) = \mathbb{E}[R(s, a)] + \gamma \sum_{s'} \textrm{max}_{a'} Q(s', a')
\end{equation}

where $\gamma$ is the time-discount factor \citep[p.3]{strens}.

Our optimal behavior is then the policy that maximizes the quality function. Many
approaches for finding the best policy uses dynamic programming in order to estimate
the transition probabilities. The maximum likelihood estimate for $T(s, a, s')$ is
the proportion of times the action $a$ in state $s$ leads to $s'$, and the estimate of
$\mathbb{E}[R(s, a)]$ is the average reward received whenever $(s, a)$ is taken. This
requires the keeping track of all pairs of states and actions - storage of large
sparse data. In \cite{strens}, the author places a prior over and estimates the
posterior transition probabilities. At each time step, the author uses the current
model over the system to estimate the quality function $Q^*$ and choose the best
action.

We will also use \cite{engel} as a guideline for implementing Bayesian policy evaluation using Gaussian processes. The algorithm presents a Bayesian solution to policy evaluation termed Gaussian Process Temporal Difference (GPTD). It then extends the algorithm to a State-Action-Reward-State-Action (SARSA) based extension of GPTD termed GPSARSA which is a fully functioning agent that can select actions and gradually improve its policies.

\section{Plans and Consideration}

For the two of us, our plan to implement these models will be in the following tasks:

\begin{enumerate}
\item Implement Bayesian MDPs as described in \cite{strens}. This includes implementing the representation of the posterior, generating the prior, and posterior updates (Vickie).
\item Implement hypothesis generation using the posteriors over the MDP parameters as described in \cite{strens} (Vickie).
\item Replicate paper results in three applications discussed in the paper (``Chain", ``Loop", and ``Maze" toy problems) (Alex).
\item Apply implementation to our maze problem and compare to non-Bayesian methods (in particular, traditional DPs) (Vickie).
\item Implement on-line Monte Carlo GPTD algorithm as described in \cite{engel} (Alex and Vickie).
\item Implement full GP State-Action-Reward-State-Action algorithm (GPSARSA) described in \cite{engel}, an extension of the GPTD which implements a fully learning agent (Alex).
\item Apply implementation of GPSARSA to our maze problem and compare to our Bayesian DP method and non-Bayesian methods from before (Alex).
\end{enumerate}

The major risks of this project come from successfully implementing the GP algorithms \cite{engel}. The algorithm is relatively complicated and involves learning of many parameters, and therefore is very prone to bugs. In general, it seems difficult to test the performance of our algorithms modularly without testing the performance of our agent as a whole. Since our reinforcement learning methods involve multiple techniques working together, we expect challenges in clean, efficient implementations that we can properly test on our toy problems. We anticipate that this proposal contains more work than we will be able to complete, and plan to reevaluate our goals at the halfway point in the project.

\bibliographystyle{apalike}
\bibliography{proposal}

\end{document}


\section{Dynamic Semantics for Modules}
\label{dynmod-sec}
\subsection{Reduced Syntax}
Since\index{58.1} signature expressions
are mostly dealt with in the static semantics,
the dynamic semantics need only take limited account of them.  Unlike types,
it cannot ignore them completely; the reason is that an explicit signature
ascription plays the role of restricting the ``view'' of a structure - that is,
restricting the domains of its component environments.  However, the types
and the sharing properties of structures and signatures are irrelevant to
dynamic evaluation; the syntax is therefore
reduced by the following transformations (in addition to those for the Core),
for the purpose of the dynamic semantics of Modules:
\begin{itemize}
\item Qualifications ``$\OF\ \ty$'' are omitted from exception descriptions.
\item Any specification of the form ``$\typespec$'', ``$\eqtypespec$'',
``$\DATATYPE$\ $\datdesc$'' or
``$\sharingspec$'' is replaced by the empty specification.
\item The Modules phrase classes TypDesc, DatDesc, ConDesc and SharEq
      are omitted.
%version 2:\item The Modules phrase classes $\typdesc$, $\datdesc$, $condesc$ and $\shareq$
%      are omitted.
\end{itemize}

\subsection{Compound Objects}
\label{dynmod-comp-obj-sec}
The\index{58.2} compound objects for the Modules dynamic semantics, extra to those for the
Core dynamic semantics, are shown in Figure~\ref{comp-dynmod-obj}.
\begin{figure}[h]
\vspace{2pt}
\begin{displaymath}
\begin{array}{rcl}
(\strid:\I,\strexp\langle:\I'\rangle,\B)
                & \in   & \FunctorClosure\\
                &       & \qquad  = (\StrId\times\Int)\times
                          (\StrExp\langle\times\Int\rangle)\times\Basis\\
(\IE,\vars,\exns)\ {\rm or}\ \I
                & \in   & \Int = \IntEnv\times\Fin(\Var)\times\Fin(\Exn)\\
        \IE     & \in   & \IntEnv = \finfun{\StrId}{\Int}\\
        \G 	& \in	& \SigEnv = \finfun{\SigId}{\Int}\\
        \F	& \in	& \FunEnv = \finfun{\FunId}{\FunctorClosure}\\
(\F,\G,\E)\ {\rm or}\ \B
                & \in   & \Basis = \FunEnv\times\SigEnv\times\Env\\
(\G,\IE)\ {\rm or}\ \IB
                & \in   & \IntBasis = \SigEnv\times\IntEnv
\end{array}
\end{displaymath}
\caption{Compound Semantic Objects}
\label{comp-dynmod-obj}
\vspace{3pt}
\end{figure}
%
%
An {\sl interface} $\I\in\Int$ represents a ``view'' of a structure.
Specifications and signature expressions will evaluate to interfaces;
moreover, during the evaluation of a specification or signature expression,
structures (to which a specification or signature expression may
refer via ``$\OPEN$'') are represented only by their interfaces.  To extract an
interface from a dynamic environment we define the operation
\[ \Inter\ :\ \Env\to\Int \]
as follows:
\[ \Inter(\SE,\VE,\EE)\ =\ (\IE,\Dom\VE,\Dom\EE)\]
where
\[ \IE\ =\ \{\strid\mapsto\Inter\E\ ;\ \SE(\strid)=\E\}\ .\]
An {\sl interface basis}\index{59.1} $\IB=(\G,\IE)$ is that part of a basis needed to
evaluate signature expressions and specifications.
The function $\Inter$ is extended to create an interface basis
from a basis $\B$ as follows:
\[ \Inter(\F,\G,\E)\ =\ (\G, \of{\IE}{(\Inter\E)}) \]

A further operation
\[ \downarrow\ :\ \Env\times\Int\to\Env\]
is required, to cut down an environment $\E$ to a given interface $\I$,
representing the effect of an explicit signature ascription.  It is defined
as follows:
\[ (\SE,\VE,\EE)\downarrow(\IE,\vars,\exns)\ =\ (\SE',\VE',\EE') \]
where
\[ \SE'\ =\ \{\strid\mapsto\E\downarrow\I\ ;\
          \SE(\strid)=\E\ {\rm and}\ \IE(\strid)=\I\} \]
and (taking $\downarrow$ now to mean restriction of a function domain)
\[\VE'=\VE\downarrow\vars,\ \EE'=\EE\downarrow\exns.\]

It is important to note that an interface is also a projection of the
{\sl static} value $\Sigma$ of a signature expression;
it is obtained by omitting structure names $\m$ and type environments
$\TE$, and replacing each variable environment $\VE$ and each
exception environment $\EE$ by its domain.
%it is obtained by omitting the
%structure names $\m$, type functions $\theta$ and type environments $\TE$.
Thus in an implementation interfaces would naturally be obtained from the
static elaboration; we choose to give separate rules here for obtaining them
in the dynamic semantics since we wish to maintain our separation of the
static and dynamic semantics, for reasons of presentation.

\subsection{Inference Rules}
The\index{59.2} semantic rules allow sentences  of the form
\[ \s,A\ts\phrase\ra A',\s' \]
to be inferred, where $A$ is either a basis or an interface basis or empty,
$A'$ is some semantic
object and $\s$,$\s'$ are the states before and after the evaluation
represented by the sentence.  Some hypotheses in rules are not of this form;
they are called {\sl side-conditions}.  The convention for options is
the same as for the Core static semantics.

The state and exception conventions are adopted as in the Core dynamic
semantics.  However, it may be shown that the only Modules phrases whose
evaluation
may cause a side-effect or generate an exception packet are of the form
$\strexp$, $\strdec$, $\strbind$ or $\topdec$.
%Also, as will be seen in Section~\ref{prog-sec}, a phrase of the
%form $\program$ can have side-effects, but not generate an
%exception packet.

%		SEMANTICS
%
%                       Structure Expressions
%
\rulesec{Structure Expressions}{\B\ts\strexp\ra \E/\p}
\begin{equation}	% generative strexp
%\label{generative-strexp-dyn-rule}
\frac{\B\ts\strdec\ra\E}
     {\B\ts\encstrexp\ra\E}\index{60.1}
\end{equation}

\begin{equation}	% longstrid
%\label{longstrid-strexp-dyn-rule}
\frac{\B(\longstrid)=\E}
     {\B\ts\longstrid\ra\E}
\end{equation}

\vspace{6pt}
\begin{equation}		% functor application
\label{functor-application-dyn-rule}
\frac{ \begin{array}{c}
        \B(\funid)=(\strid:\I,\strexp'\langle:\I'\rangle,\B')\\
        \B\ts\strexp\ra\E\qquad
       \B'+\{\strid\mapsto\E\downarrow\I\}\ts\strexp'\ra\E'\\
       \end{array}
     }
     {\B\ts\funappstr\ra\E'\langle\downarrow\I'\rangle}
\end{equation}

\vspace{6pt}
\begin{equation}	% let strexp
%\label{letstrexp-dyn-rule}
\frac{\B\ts\strdec\ra\E\qquad\B+\E\ts\strexp\ra\E'}
     {\B\ts\letstrexp\ra\E'}
\end{equation}
\comments
\begin{description}
\item{(\ref{functor-application-dyn-rule})}
Before the evaluation of the functor body $\strexp'$, the
actual argument $\E$ is cut down by the formal parameter
interface $\I$, so that any opening of $\strid$ resulting
from the evaluation of $\strexp'$ will produce no more components
than anticipated during the static elaboration.
\end{description}

\rulesec{Structure-level Declarations}{\B\ts\strdec\ra\E/\p}
   		% declarations
\begin{equation}                % core declaration
%\label{dec-dyn-rule}
\frac{ \of{\E}{\B}\ts\dec\ra\E' }
     { \B\ts\dec\ra\E' }\index{60.2}
\end{equation}

\vspace{6pt}
\begin{equation}        	% structure declaration
%\label{structure-decl-dyn-rule}
\frac{ \B\ts\strbind\ra\SE }
     { \B\ts\singstrdec\ra\SE\ \In\ \Env }
\end{equation}

\vspace{6pt}
\begin{equation}                % local structure-level declaration
%\label{local structure-level declaration-dyn-rule}
\frac{ \B\ts\strdec_1\ra\E_1\qquad
       \B+\E_1\ts\strdec_2\ra\E_2 }
     { \B\ts\localstrdec\ra\E_2 }
\end{equation}

\vspace{6pt}
\begin{equation}                % empty declaration
%\label{empty-strdec-dyn-rule}
\frac{}
     {\B\ts\emptystrdec\ra \emptymap{\rm\ in}\ \Env}
\end{equation}

\vspace{6pt}
\begin{equation}		% sequential declaration
%\label{sequential-strdec-dyn-rule}
\frac{ \B\ts\strdec_1\ra\E_1\qquad
       \B+\E_1\ts\strdec_2\ra\E_2 }
     { \B\ts\seqstrdec\ra\plusmap{\E_1}{\E_2} }
\end{equation}

\rulesec{Structure Bindings}{\B\ts\strbind\ra\SE/\p}
\begin{equation}                % structure binding
\frac{ \begin{array}{cl}
       \B\ts\strexp\ra\E\qquad\langle\Inter\B\ts\sigexp\ra\I\rangle\\
       \langle\langle\B\ts\strbind\ra\SE\rangle\rangle
       \end{array}
     }
     {\begin{array}{c}
      \B\ts\strbinder\ra\\
      \qquad\qquad\qquad\{\strid\mapsto\E\langle\downarrow\I\rangle\}
      \ \langle\langle +\ \SE\rangle\rangle
      \end{array}
     }\index{61.1}
\end{equation}
\comment As in the static semantics, when present, $\sigexp$ constrains the
``view'' of the structure. The restriction must be done in the
dynamic semantics to ensure that any dynamic opening of the structure
produces no more components than anticipated during the static
elaboration.
%
%                   Signature Rules
%

\rulesec{Signature Expressions}{\IB\ts\sigexp\ra\I}
\begin{equation}		% encapsulation sigexp
%\label{encapsulating-sigexp-dyn-rule}
\frac{\IB\ts\spec\ra\I }
     {\IB\ts\encsigexp\ra\I}\index{61.2}
\end{equation}

\begin{equation}		% signature identifier
%\label{signature-identifier-dyn-rule}
\frac{ \IB(\sigid)=\I}
     { \IB\ts\sigid\ra\I }
\end{equation}

\rulesec{Signature Declarations}{\IB\ts\sigdec\ra\G}
\begin{equation}	% single signature declaration
%\label{single-sigdec-dyn-rule}
\frac{ \IB\ts\sigbind\ra\G }
     { \IB\ts\singsigdec\ra\G }\index{61.3}
\end{equation}

\begin{equation}	% empty signature declaration
%\label{empty-sigdec-dyn-rule}
\frac{}
     { \IB\ts\emptysigdec\ra\emptymap }
\end{equation}

\begin{equation}	% sequential signature declaration
%\label{sequence-sigdec-dyn-rule}
\frac{ \IB\ts\sigdec_1\ra\G_1 \qquad \plusmap{\IB}{\G_1}\ts\sigdec_2\ra\G_2 }
     { \IB\ts\seqsigdec\ra\plusmap{\G_1}{\G_2} }
\end{equation}

\rulesec{Signature Bindings}{\IB\ts\sigbind\ra\G}

\begin{equation}	% signature binding
%\label{sigbind-dyn-rule}
\frac{ \IB\ts\sigexp\ra\I
        \qquad\langle\IB\ts\sigbind\ra\G\rangle }
     { \IB\ts\sigbinder\ra\{\sigid\mapsto\I\}
       \ \langle +\ \G\rangle }\index{61.4}
\end{equation}
%
                     % Specifications
%
\rulesec{Specifications}{\IB\ts\spec\ra\I}

\begin{equation}        % value specification
%\label{valspec-dyn-rule}
\frac{ \ts\valdesc\ra\vars }
     { \IB\ts\valspec\ra\vars\ \In\ \Int }\index{61.5}
\end{equation}

\begin{equation}        % exception specification
%\label{exceptionspec-dyn-rule}
\frac{ \ts\exndesc\ra\exns}
     { \IB\ts\exceptionspec\ra\exns\ \In\ \Int }
\end{equation}

\begin{equation}        % structure specification
\label{structurespec-dyn-rule}
\frac{ \IB\ts\strdesc\ra\IE }
     { \IB\ts\structurespec\ra\IE\ \In\ \Int }\index{62.1}
\end{equation}

\begin{equation}        % local specification
\label{localspec-dyn-rule}
\frac{ \IB\ts\spec_1\ra\I_1 \qquad
       \plusmap{\IB}{\of{\IE}{\I_1}}\ts\spec_2\ra\I_2 }
     { \IB\ts\localspec\ra\I_2 }
\end{equation}

\begin{equation}        % open specification
%\label{openspec-dyn-rule}
\frac{ \IB(\longstrid_1)=\I_1\quad\cdots\quad
       \IB(\longstrid_n)=\I_n }
     { \IB\ts\openspec\ra\I_1 + \cdots +\I_n }
\end{equation}

\begin{equation}        % include signature specification
%\label{inclspec-dyn-rule}
\frac{ \IB(\sigid_1)=\I_1 \quad\cdots\quad
       \IB(\sigid_n)=\I_n }
     { \IB\ts\inclspec\ra\I_1 + \cdots +\I_n }
\end{equation}

\begin{equation}        % empty specification
%\label{emptyspec-dyn-rule}
\frac{}
     { \IB\ts\emptyspec\ra\emptymap{\rm\ in}\ \Int }
\end{equation}

\begin{equation}        % sequential specification
\label{seqspec-dyn-rule}
\frac{ \IB\ts\spec_1\ra\I_1
       \qquad \plusmap{\IB}{\of{\IE}{\I_1}}\ts\spec_2\ra\I_2 }
     { \IB\ts\seqspec\ra\plusmap{\I_1}{\I_2} }
\end{equation}
\comments
\begin{description}
\item{(\ref{localspec-dyn-rule}),(\ref{seqspec-dyn-rule})}
  Note that $\of{\vars}{\I_1}$ and $\of{\exns}{\I_1}$ are
not needed for the evaluation of $\spec_2$.
\end{description}

                         % Descriptions

\rulesec{Value Descriptions}{\ts\valdesc\ra\vars}
\begin{equation}         % value description
%\label{valdesc-dyn-rule}
\frac{ \langle\ts\valdesc\ra\vars\rangle }
     { \ts\var\ \langle\AND\ \valdesc\rangle\ra
       \{\var\}\ \langle\cup\ \vars\rangle }\index{62.2}
\end{equation}

\rulesec{Exception Descriptions}{\ts\exndesc\ra\exns}
\begin{equation}         % exception description
%\label{exndesc-dyn-rule}
\frac{ \langle\ts\exndesc\ra\exns\rangle }
     { \ts\exn\ \langle\exndesc\rangle\ra\{\exn\}\ \langle\cup\ \exns\rangle }\index{62.3}
\end{equation}

\rulesec{Structure Descriptions}{\IB\ts\strdesc\ra\IE}
\begin{equation}
%\label{strdesc-dyn-rule}
\frac{ \IB\ts\sigexp\ra\I\qquad\langle\IB\ts\strdesc\ra\IE\rangle }
     { \IB\ts\strdescription\ra\{\strid\mapsto\I\}\ \langle +\ \IE\rangle }\index{62.4}
\end{equation}

%    			Functor and Program rules
%
\rulesec{Functor Bindings}{\B\ts\funbind\ra\F}
\begin{equation}	% functor binding
%\label{funbind-dyn-rule}
\frac{
      \begin{array}{c}
      \Inter\B\ts\sigexp\ra\I\qquad
      \langle\Inter\B+\{\strid\mapsto\I\} \ts\sigexp'\ra\I'\rangle \\
       \langle\langle\B\ts\funbind\ra\F\rangle\rangle
      \end{array}
     }
     {
      \begin{array}{c}
       \B\ts\funstrbinder\ \optfunbind\ra\\
       \qquad\qquad \qquad
              \{\funid\mapsto(\strid:\I,\strexp\langle:\I'\rangle,\B)\}
              \ \langle\langle +\ \F\rangle\rangle
      \end{array}
     }\index{62.5}
\end{equation}

\rulesec{Functor Declarations}{\B\ts\fundec\ra\F}
\begin{equation}        % single functor declaration
%\label{singfundec-dyn-rule}
\frac{ \B\ts\funbind\ra\F }
     { \B\ts\singfundec\ra\F }\index{63.1}
\end{equation}

\vspace{6pt}
\begin{equation}        % empty functor declaration
%\label{emptyfundec-dyn-rule}
\frac{}
     { \B\ts\emptyfundec\ra\emptymap }
\end{equation}

\vspace{6pt}
\begin{equation}        % sequential functor declaration
%\label{seqfundec-dyn-rule}
\frac{ \B\ts\fundec_1\ra\F_1\qquad
       \B+\F_1\ts\fundec_2\ra\F_2 }
     { \B\ts\seqfundec\ra\plusmap{\F_1}{\F_2} }
\end{equation}

\rulesec{Top-level Declarations}{\B\ts\topdec\ra\B'/\p}
\begin{equation}	% structure-level declaration
%\label{strdectopdec-dyn-rule}
\frac{\B\ts\strdec\ra\E}
     {\B\ts\strdec\ra\E\ \In\ \Basis
     }\index{63.2}
\end{equation}

\vspace{6pt}
\begin{equation}	% signature declaration
%\label{sigdectopdec-dyn-rule}
\frac{\Inter\B\ts\sigdec\ra\G}
     {\B\ts\sigdec\ra\G\ \In\ \Basis
     }
\end{equation}

\vspace{6pt}
\begin{equation}	% functor declaration
%\label{fundectopdec-dyn-rule}
\frac{\B\ts\fundec\ra\F}
     {\B\ts\fundec\ra\F\ \In\ \Basis
     }
\end{equation}

%from version 1
%\vspace{6pt}
%\begin{equation}	% sequential program
%\label{seqprog-dyn-rule}
%\frac{\B\ts\program_1\ra\B_1\qquad\
%      \plusmap{\B}{\B_1}\ts\program_2\ra\B_2
%     }
%     {\B\ts\seqprog\ra\plusmap{\B_1}{\B_2}}
%\end{equation}


\documentstyle[11pt,reduce]{article}
\title{NCPOLY: Computation in non--commutative polynomial ideals}
\date{}
\author{
Herbert Melenk\\[0.05in]
Konrad--Zuse--Zentrum f\"ur Informationstechnik Berlin \\
Takustrasse 7 \\
D--14195 Berlin -- Dahlem \\
Germany \\[0.05in]
E--mail: melenk@zib.de \\[0.1in]
Joachim Apel\\[0.05in]
Institut f\"ur Informatik \\[0.05in]
Universit\"at Leipzig \\[0.05in]
Augustusplatz 10--11\\[0.05in]
D--04109 Leipzig \\[0.05in]
Germany \\[0.05in]
E--mail: apel@informatik.uni--leipzig.de \\[0.05in]
}

\begin{document}
\maketitle

\index{Groebner Bases}
\section{Introduction}

{\small REDUCE} supports a very general mechanism for computing
with objects under a non--commutative multiplication, where
commutator relations must be introduced explicitly by rule
sets when needed. The package {\bf NCPOLY} allows you to
set up automatically a consistent environment for computing in an algebra
where the non--commutativity is defined by Lie-bracket commutators.
The package uses the {\small REDUCE} {\bf noncom}
mechanism for elementary polynomial arithmetic; the commutator
rules are automatically computed from the Lie brackets.
You can perform polynomial arithmetic directly, including
{\bf division} and {\bf factorization}.
Additionally {\bf NCPOLY} supports computations in a one sided ideal (left or right),
especially one sided {\bf Gr\"obner} bases and {\bf polynomial reduction}.

\section{Setup, Cleanup}

Before the computations can start the environment for a
non--commutative computation must be defined by a
call to {\bf nc\_setup}:
\begin{verbatim}
    nc_setup(<vars>[,<comms>][,<dir>]);
\end{verbatim}
where

$<vars>$ is a list of variables; these must include the
non--commutative quantities.

$<comms>$ is a list of equations \verb&<u>*<v> - <v>*<u>=<rh>&
where $<u>$ and $<v>$ are members of $<vars>$, and $<rh>$ is
a polynomial.

$<dir>$ is either $left$ or $right$ selecting a left or a
right one sided ideal. The initial direction is $left$.

{\bf nc\_setup} generates from $<comms>$ the necessary
rules to support an algebra where all monomials are
ordered corresponding to the given variable sequence.
All pairs of variables which are not explicitly covered in
the commutator set are considered as commutative and the
corresponding rules are also activated.

The second parameter in {\bf nc\_setup} may be
omitted if the operator is called for the second time,
e.g. with a reordered variable sequence. In such a case
the last commutator set is used again.

Remarks: \begin{itemize}
\item The variables need not be declared {\bf noncom} -
    {\bf nc\_setup} performs all necessary declarations.
\item The variables need not be formal operator expressions;
    {\bf nc\_setup} encapsulates a variable $x$ internally
    as \verb+nc!*(!_x)+ expressions anyway where the operator $nc!*$
    keeps the noncom property.
\item The commands {\bf order} and {\bf korder} should be avoided
    because {\bf nc\_setup} sets these such that the computation
    results are printed in the correct term order.
\end{itemize}

Example:
\begin {verbatim}
   nc_setup({KK,NN,k,n},
              {NN*n-n*NN= NN, KK*k-k*KK= KK});

   NN*n;            ->   NN*n
   n*NN;            ->   NN*n - NN
   nc_setup({k,n,KK,NN});
   NN*n - NN        ->   n*NN;

\end{verbatim}
Here $KK,NN,k,n$ are non--commutative variables where
the commutators are described as $[NN,n]=NN$, $[KK,k]=KK$.

The current term order must be compatible with the commutators:
the product $<u>*<v>$ must precede all terms on the right hand
side $<rh>$ under the current term order. Consequently
\begin{itemize}
\item the maximal degree of $<u>$ or $<v>$ in $<rh>$ is 1,
\item in a total degree ordering the total degree of $<rh>$ may be
not higher than 1,
\item in an elimination degree order (e.g. $lex$) all variables in
$<rh>$ must be below the minimum of $<u>$ and $<v>$.
\item If $<rh>$ does not contain any variables or has at most $<u>$ or
$<v>$, any term order can be selected.
\end{itemize}

If you want to use the non--commutative variables or results from
non--commutative computations later in commutative operations
it might be necessary to switch off the non--commutative
evaluation mode because not
all operators in REDUCE are prepared for that environment. In
such a case use the command
\begin{verbatim}
    nc_cleanup;
\end{verbatim}
without parameters. It removes all internal rules and definitions
which {\bf nc\_setup} had introduced. To reactive non--commutative
call {\bf nc\_setup} again.

\section{Left and right ideals}

A (polynomial) left ideal $L$ is defined by the axioms

$u \in L, v \in L \Longrightarrow u+v \in L$

$u \in L \Longrightarrow k*u \in L$ for an arbitrary polynomial $k$

where ``*'' is the non--commutative multiplication. Correspondingly,
a right ideal $R$ is defined by

$u \in R, v \in R \Longrightarrow u+v \in R$

$u \in R \Longrightarrow u*k \in R$ for an arbitrary polynomial $k$

\section{Gr\"obner bases}

When a non--commutative environment has been set up
by {\bf nc\_setup}, a basis for a left or right polynomial ideal
can be transformed into a Gr\"obner basis by the operator
{\bf nc\_groebner}:
\begin{verbatim}
   nc_groebner(<plist>);
\end{verbatim}
Note that the variable set and variable sequence must be
defined before in the {\bf nc\_setup} call. The term order
for the Gr\"obner calculation can be set by using the
{\bf torder} declaration. The internal steps of the
Gr\"obner calculation can be watched by setting the
switches {\bf trgroeb} (=list all internal basis polynomials)
or {\bf trgroebs} (=list additionally the $S$-polynomials)
\footnote{The command \verb+lisp(!*trgroebfull:=t);+ causes additionally
all elementary polynomial operations to be printed.}.

For details about {\bf torder}, {\bf trgroeb} and {\bf trgroebs}
see the {\bf {\small REDUCE} GROEBNER} manual.
\begin{verbatim}
2: nc_setup({k,n,NN,KK},{NN*n-n*NN=NN,KK*k-k*KK=KK},left);

3: p1 := (n-k+1)*NN - (n+1);

p1 :=  - k*nn + n*nn - n + nn - 1

4: p2 := (k+1)*KK -(n-k);

p2 := k*kk + k - n + kk

5: nc_groebner ({p1,p2});

{k*nn - n*nn + n - nn + 1,

 k*kk + k - n + kk,

 n*nn*kk - n*kk - n + nn*kk - kk - 1}

\end{verbatim}
Important: Do not use the operators of the GROEBNER
package directly as they would not consider the non--commutative
multiplication.

\section{Left or right polynomial division}

The operator {\bf nc\_divide} computes the one sided quotient and remainder of
two polynomials:
\begin{verbatim}
    nc_divide(<p1>,<p2>);
\end{verbatim}
The result is a list with quotient and remainder.
The division is performed as a pseudo--division, multiplying
$<p1>$ by coefficients if necessary. The result $\{<q>,<r>\}$
is defined by the relation

  $<c>*<p1>=<q>*<p2> + <r>$ for direction $left$ and

  $<c>*<p1>=<p2>*<q> + <r>$ for direction $right$,

where $<c>$ is an expression that does not contain any of the
ideal variables, and the leading term of $<r>$ is lower than
the leading term of $<p2>$ according to the actual term order.

\section{Left or right polynomial reduction}

For the computation of the one sided remainder of a polynomial
modulo a given set of other polynomials the operator
{\bf nc\_preduce} may be used:
\begin{verbatim}
    nc_preduce(<polynomial>,<plist>);
\end{verbatim}
The result of the reduction is unique (canonical) if
and only if $<plist>$ is a one sided Gr\"obner basis.
Then the computation is at the same time an ideal
membership test: if the result is zero, the
polynomial is member of the ideal, otherwise not.

\section{Factorization}

\subsection{Technique}

Polynomials in a non--commutative ring cannot be factored
using the ordinary {\bf factorize} command of {\small REDUCE}.
Instead one of the operators of this section must be used:
\begin{verbatim}
   nc_factorize(<polynomial>);
\end{verbatim}
The result is a list of factors of $<polynomial>$. A list
with the input expression is returned if it is irreducible.

As non--commutative factorization is not unique, there is
an additional operator which computes all possible factorizations
\begin{verbatim}
   nc_factorize_all(<polynomial>);
\end{verbatim}
The result is a list of factor decompositions of $<polynomial>$.
If there are no factors at all the result list has only one
member which is a list containing the input polynomial.

\subsection{Control of the factorization}

In contrast to factoring in commutative polynomial rings, the non--commutative
factorization is rather time consuming. Therefore two additional
operators allow you to reduce the amount of computing time when
you look only for isolated factors in special context, e.g. factors
with a limited degree or factors which contain only explicitly
specified variables:
\begin{verbatim}
    left_factor(<polynomial>[,<deg>[,<vars>]])
   right_factor(<polynomial>[,<deg>[,<vars>]])
    left_factors(<polynomial>[,<deg>[,<vars>]])
   right_factors(<polynomial>[,<deg>[,<vars>]])
\end{verbatim}
where $<polynomial>$ is the form under investigation,
$<vars>$ is an optional list of variables which must appear in the
factor, and $<deg>$
is an optional integer degree bound for the total degree of the
factor, a zero for an unbounded search, or a monomial
(product of powers of the variables) where each exponent
is an individual degree bound for its base variable; unmentioned
variables are allowed in arbitrary degree. The operators
$*\_factor$ stop when they have found one factor, while
the operators $*\_factors$ select all one--sided factors
within the given range. If there is no factor of the
desired type, an empty list is returned by $*\_factors$
while the routines $*\_factor$ return the input polynomial.

\subsection{Time of the factorization}

The share variable $nc\_factor\_time$ sets an upper limit
for the time to be spent for a call to the non--commutative
factorizer. If the value is a positive integer, a
factorization is terminated with an error message as soon
as the time limit is reached. The time units are milliseconds.

\subsection{Usage of SOLVE}

The factorizer internally uses $solve$, which is controlled
by the \REDUCE \ switch $varopt$. This switch (which per default
is set $on$) allows, to reorder the variable sequence, which is
favourable for the normal system. It should be avoided to set $varopt$
$off$, when using the non--commutative factorizer, unless very small
polynomials are used.

\section{Output of expressions}

It is often desirable to have the commutative parts (coefficients)
in a non--commutative operation condensed by factorization. The operator
\begin{verbatim}
   nc_compact(<polynomial>)
\end{verbatim}
collects the coefficients to the powers of the lowest possible
non-commutative variable.
\begin{verbatim}
load ncpoly;

nc_setup({n,NN},{NN*n-n*NN=NN})$
p1 := n**4 + n**2*nn + 4*n**2 + 4*n*nn + 4*nn + 4;

       4    2         2
p1 := n  + n *nn + 4*n  + 4*n*nn + 4*nn + 4

nc_compact p1;

  2     2          2
(n  + 2)  + (n + 2) *nn
\end{verbatim}
\end{document}


% !TEX root = ../main.tex
% \section{Modeling desiderata}
% \label{sec:desiderata}
% Our goal is to build a recommendation model that provably maximizes an
% evaluation metric. The model should also approximate other models that recommend
% items based on sets of attributes. We propose criteria for models that rank from
% sets, and then exhibit a model satisfying these desiderata:
% \begin{itemize}
% \item Order-invariance: if the input to the model includes an unordered set of
%   attributes, the output of the model should not depend on the order that the
%   set elements are fed to the model.
% \item Improved recall: training the model should provably improve the evaluation
%   metric of recall. Models that satisfy this desideratum make good recommenders.
%   (In contrast, it is unclear whether a model that reconstructs the training
%   data well will lead to maximum recall performance.)
% \item Universal approximation: the model should be able to approximate any other
%   order-invariant model. This reduces the need to compare to other models in
%   lieu of comparing parameterizations of a single model.
% \item Parameter-sharing: the model should share parameters across items with
%   similar sets of attributes. This enables the model to scale to large
%   real-world datasets.
% \end{itemize}
% The criterion of order-invariance is necessary due to the set-valued features
% associated with every item. The parameter-sharing desideratum further narrows
% the set of models and guides the development of architectures parameterizing a
% model. The desiderata of improved recall and universal approximation, if
% satisfied, frees practitioners to focus on building architectures for a single
% model. Intra-model comparisons, rather than arduous inter-model comparisons,
% enable the development of generic architectures that work across many datasets
% for ranking from sets.
\section{RankFromSets}
\label{sec:rankfromsets}
\acrfull{rfs} is a class of recommendation models that recommend items with attributes to users. Let~$u~\in~\{1, \ldots, N\}$ be a user, $m \in \{1, \ldots, M\}$ be an item, and $\yum~\in~\{0,1\}$ be a binary indicator where $1$ indicates user $u$ consumed item $m$. For each item $m$, there is an associated set of attributes $x_m \in \{0,1\}^{|V|}$ from a vocabulary of $V$ attributes. The observed data is a collection of user-item interactions $\{(u, m)\}$ and the sets of attributes associated with items $\{x_m\}$.

We assume that a recommendation model is given a budget of $K$ recommendations to be made for each user. In response, the recommender system produces a list of $K$ distinct recommendations $\mathbf{r}_u~=~(r_{u1}, \ldots, r_{uK})$ for each user. The goal of the recommendation task in this paper is to maximize the expected Recall@$K$,
\begin{equation}
\textrm{Recall}@K = \mathbb{E}_{u}\left[\frac{\sum_{r \in \mathbf{r}_u} y_{ur}}{\sum_{m} \yum}\right]\, ,
\label{eq:recall}
\end{equation}
with the expectation over users in the empirical distribution $\cD$.

We combine three techniques to maximize Recall@$K$ with \gls{rfs}. First, we cast recommendation as a classification task. Second, we learn user- and attribute-level embeddings. Statistical strength is shared between items with similar attributes by representing items as the mean of their attribute embeddings. Third, we scale \gls{rfs} to large datasets using a stochastic optimization-based negative sampling training procedure.

\gls{rfs} casts the recommendation problem as a classification task. Given a user-item pair $(u,m)$ and regression function $f$, \gls{rfs} learns to predict the probability that item~$m$~will be consumed by user $u$:
$$p(\yum = 1 \mid u, m) = \sigma\left( f \left (u, x_m\right) \right)\, ,$$
where $x_m$ is the set of attributes of item $m$ and $\sigma$ is the sigmoid function. Recommendations made by \gls{rfs} are the maximum likelihood set formed by ranking a set of items for a user according to the model $f(u, x_m)$. We motivate treating recommendation as classification with the following observation.
% \begin{equation}
% \label{eqn:argmax}
% \mathbf{r}_u(K) = \underset{\mathbf{r} \in \mathbb{N}^K}{\text{argmax}} \sum_{m\in\mathbf{r}} f \left( u, x_{m} \right) \, .
% \end{equation}
% Setup the problem here as a generic classification problem. the recommendations are going to be the argmax over the probabilities output by a classifier.
% \vspace{0.5cm}
\begin{prop}
\label{prop:maximizing-recall}
Let $u \in \mathcal{U}$ be a user, $m \in \mathcal{M}$ be an item, and
$y(u,m) \in \{0,1\}$ be an indicator of whether user $u$ logged item $m$. Let
$\mathcal{E}$ be the worst-case error for binary classifier $\hat{y}(u,m)$ on
any~$(u,m)$ pair drawn from the data $\mathcal{D}$,
\begin{equation*}
  \mathcal{E} = \max_{(u, m) \in \mathcal{D}} \mathbb{1}\left[ \hat y(u, m) \neq y(u, m) \right] \, .
\end{equation*}
A binary classifier with zero worst-case error ($\mathcal{E}=0$) maximizes
recommendation recall.
\end{prop}
\begin{proof}
  A model with zero worst-case error is a perfect classifier, assigning greater probability to data with positive labels than to data with negative labels. In other words, it ranks positive examples above negative examples. Recall@$K$ is measured by the fraction of items with positive labels in a ranking returned by the model. In a classifier that achieves zero worst-case error, positively-labeled datapoints must be ranked higher than other datapoints, maximizing recall.
\end{proof}

\Cref{prop:maximizing-recall} is simple, but conceptually important. Under the assumption that a perfect classifier exists, a consistent method for learning a classifier will be a consistent method for learning a recommendation system that targets expected recall. Put another way, recall is inherently binary: a model does or does not recall an item; an item is or is not in the top $K$ recommendations in the numerator of \Cref{eq:recall}. So the best one can hope to do if recall is used to assess recommendation performance is to train a binary classifier. In practice, as with any regression method, a perfect classifier is unachievable. \Cref{prop:maximizing-recall} is a guiding principle rather than a finite-sample guarantee of maximal performance. As we show in \Cref{sec:rfs-experiments}, the classification approach of \gls{rfs} performs well in practice.

For recommending items with attributes, \Cref{prop:maximizing-recall} says that building a classifier such as \gls{rfs} is optimal if we measure recommendation performance with recall. To parameterize the \gls{rfs} classifier, a regression function $f(u, x_m)$ is needed. A straightforward parameterization is an inner product,
\begin{equation}
\label{eqn:rankfromsets}
  f\left(u, x_m\right) = \theta_u^\top\left(\frac{1}{|x_m|}\sum_{j\in x_m}
  \beta_j + g(x_m)\right) + h(x_m) \, .
\end{equation}
Each element in the inner product regression function in \Cref{eqn:rankfromsets} has an intuitive interpretation. The user embedding $\theta_u \in \mathbb{R}^d$ captures the latent preferences for user $u$. This captures the individual-level tastes of a user and is analogous to the user preference vector in classical collaborative filtering or the row embedding in matrix factorization. The attribute embedding $\beta_j \in \mathbb{R}^d$ is the latent quality conveyed through item $m$ having attribute $j$. (The set $x_m$ contains only attributes with~$x_{mj}=1$. Attributes that are not associated with item $m$ are ignored.) The item embedding function~${g(x_m)\in\mathbb{R}^d}$~represents qualities not conveyed through the set of item attributes. This term in the regression function enables collaborative filtering by capturing unobserved patterns in item consumption such as popularity. We describe how to construct this function below. The scalar item intercept function $h(x_m) \in \mathbb{R}$ makes an item more or less likely due to availability.
\input{ch-rfs/table/nearest_meals}

To define scalable item embedding and item intercept functions, note that the parameterization of the item embedding function $g(x_m)$ depends on the size of the data. If the number of items is small, $g$ can function as a lookup for unique intercepts for every item. However, if the number of items is so large that unique item intercepts lead to overfitting, a scalable parameterization of item embeddings $g$ can be defined using additional information about every item. For example, if the data consists of foods in meals, we can define a meal intercept as the mean of food intercepts, yielding a scalable item intercept function. The item intercept function $h(x_m)$ that maps item attributes to scalars is constructed in the same way. We study both of these choices in \Cref{sec:rfs-experiments}.

The inner product regression function in \Cref{eqn:rankfromsets} has several benefits. It requires computing a sum over only the attributes with which each item is associated. This enables \gls{rfs} to scale to large attribute vocabularies where traditional matrix factorization methods are intractable. Second, the embed-and-average approach to set modeling is provably flexible as we show later. We now describe deep variants of \gls{rfs} and detail how \gls{rfs} can approximate other recommendation models.% Consider the example from \Cref{sec:rfs-introduction}
% of predicting whether a user
% would enjoy a meal. Each user has some preferences about which meals they enjoy.
% A meal is made up of foods or attributes. One does not need to consider foods
% that are not in a meal; unused foods can be ignored. In addition, a meal is more
% than its foods. For instance, a user trying to eat healthier may enjoy all the
% ingredients in a meal, but if it is deep fried, it will be unappealing. The
% regression function in \gls{rfs} should be able to learn such latent patterns in
% meal consumption. Finally, even if a meal is not very appealing to a user, it
% may be popular and available everywhere, making it more likely to be consumed.

% The regression function $f(u, x_m)$ parameterizes the generative process of the
% labels conditional on the item attributes and user,
% $\yum \sim \textrm{Bernoulli}\left(\yum; \sigma(f(u, x_m)\right)$.

% Specifically, recent results in deep learning for set-valued data show that
% such an approach is sufficient to model nearly any function
% \citep{DBLP:journals/corr/ZaheerKRPSS17}. In theory, it may be necessary to
% apply a nonlinearity, such as a deep neural network, to $f$ to learn any
% function. In preliminary experiments, we found increasing the embedding size
% $d$ in the embedding model offered a better tradeoff between parameter size
% and model performance than the additional parameters added by a neural
% network.

The \gls{rfs} inner product regression function in \Cref{eqn:rankfromsets} is a log-bilinear model. But there are several other choices of regression function, and we draw on the deep learning toolkit for classification to build two other example architectures. With finite data and finite compute, one architecture may outperform another, or prove insufficient to capture patterns in user consumption. (Later, we show that all architectures are equivalent under fewer assumptions.) First, as an alternative to the log-bilinear model in \Cref{eqn:rankfromsets}, we can use a deep neural network as a regression function:
\begin{align}
  f\left(u, x_m\right) = \phi\left(\theta_u, \frac{1}{|x_m|}\sum_{j\in x_m}
  \beta_j, g(x_m)\right) + h(x_m) \, ,
  \label{eqn:neural-network}
\end{align}
where the deep network $\phi$ has weights and biases and takes as inputs the user embedding, sum of attribute embeddings, and item intercept. Such a neural network can represent functions that may or may not include the inner product in \Cref{eqn:rankfromsets}; \emph{ex~ante}, it is unclear whether a finite-depth, finite-width neural network can represent the inner product.

Another regression function for \gls{rfs} is a combination of
\Cref{eqn:rankfromsets,eqn:neural-network}, using an idea borrowed from deep
residual networks for image classification~\citep{he2015deep}. In this
architecture, a neural network $\phi$ with the same inputs as in
\Cref{eqn:neural-network} learns the residual of the inner product model:
\begin{align}
  f\left(u, x_m\right) = \theta_u^\top\left(\frac{1}{|x_m|}\sum_{j\in x_m}
  \beta_j + g(x_m)\right) + \phi + h(x_m) \, .
  \label{eqn:residual}
\end{align}
The choice of regression function in \acrshort{rfs} depends on the data. On finite data, with finite compute, one parameterization of \gls{rfs} will outperform another. To demonstrate this, we simulated synthetic data from the same generative process \gls{rfs} employs with a ground-truth regression function (a square kernel), and found that the residual and deep parameterizations outperformed the inner product architecture. These results are included in \Cref{sec:simulation}, and motivate exploring other architectures than the three examples here.
% We conclude this section by showing that in the regime of infinite
% data and compute, all the \gls{rfs} architectures we propose, including the
% inner product, can approximate other recommendation models that operate on
% set-valued input such as matrix factorization.

Stepping back from the setting of finite data and compute, a bigger picture emerges, which reveals the choice of regression function in \gls{rfs} does not matter. We show that any \gls{rfs} architecture is sufficiently flexible to approximate recommendation models that operate on set-valued input. We define permutation-invariant models before deriving this result.

The regression function $f$ in \gls{rfs} operates on set-valued input: the unordered collection of item attributes $x_m$. A set is, by definition, permutation-invariant: it remains the same if we permute its elements. Functions that operate on set-valued inputs must also be permutation-invariant. \gls{rfs} is permutation-invariant; the set of attributes associated with an item enter into \Cref{eqn:rankfromsets,eqn:neural-network,eqn:residual} via summation. Other examples of permutation-invariant recommendation models are multiple matrix factorization, models based on word embeddings, and permutation-marginalized recurrent neural networks. These models are shown to be permutation-invariant in \Cref{sec:models} and evaluated in \Cref{sec:rfs-experiments}. We now show that \gls{rfs} can approximate other permutation-invariant recommendation models such as matrix factorization.

\begin{prop}
  Assume the vocabulary of attributes (set elements) is countable, $\lvert V \rvert < \lvert \bbN_0 \rvert$. Then \acrshort{rfs} can approximate any permutation-invariant recommendation model.
  \label{prop:universal-approximation}
\end{prop}
The proof follows directly from Theorem~2 in \citet{zaheer2017deep} and we will not
restate it here. (The only change to the proof is the mapping from set elements
to one-hot vectors, $c \colon V \to \left\{0, 1\right\}^{\lvert V \rvert}$ to
yield a unique representation of every object in the powerset.)
\Cref{prop:universal-approximation} means that any of the parameterizations in
\Cref{eqn:rankfromsets,eqn:neural-network,eqn:residual} is flexible enough to
approximate other principled recommendation models that leverage item
attributes, such as multiple matrix
factorization~\citep{gopalan2014content-based,wang2011collaborative}.% This proposition also supports
% exploring other parameterizations of \gls{rfs} that may have better
% computational or statistical properties.

The parameters for \gls{rfs} are learned by stochastic optimization. Denote
the full set of \gls{rfs} model parameters by $\mbgamma$, and let~$\cD_u$~be the
empirical data distribution for a user. Let $\lambda_u$ be a reweighting
parameter. The per-user maximum likelihood objective for \gls{rfs} is
\begin{equation}
  \label{eq:objective}
  \begin{aligned}
  \cL(\mbgamma, \lambda_u) = \E_u\big[ \E_{m \sim \cD_u \mid \yum = 1}%&
  \left
  [\log p(\yum =1 \mid x_m; \mbgamma)\right]
  + \lambda_u \E_{k \sim \cD_u \mid \yuk = 0}%&
  \left[\log p(\yuk = 0 \mid x_k;
  \mbgamma)\right]
                \big]
  \end{aligned}
\end{equation}

In traditional regression, altering the ratio of positive to negative examples
by reweighting leads to inconsistent parameter estimation. The inconsistency
stems from the randomness in the labels, given the features. However, Recall@$K$ assumes that each user, item attribute set pair ($u, x_m$) uniquely determines
whether the item was consumed or not (the label $\yum$). Here, all reweightings
produce the same result. This means that for any negative example weight
$\lambda_u$, the learned model will be the same. In practice we set $\lambda_u$
to balance the positive and negative examples for each user. We use stochastic
optimization to maximize \Cref{eq:objective}, and describe two negative sampling
schemes that are dependent on the choice of evaluation metric.
% for a single datapoint $(x_m, \yum=1)$ is
% \begin{align}
% \begin{split}
%   \cL(\theta, & (x_m, \yum), \{(x_k, \yuk)\}) =\\
%   &\log p(\yum = 1 \mid x_m; \theta) + \sum_{k=1}^L \log p(\yuk = 0 \mid x_k;
%   \theta),\\
%   &\textrm{with}~x_k\sim\textrm{Uniform}(I).
%   \label{eq:objective}
% \end{split}
% \end{align}
% As users only interact with a small number of items, this objective function is
% dominated by negatively-labeled datapoints with~$\yum = 0$. We use stochastic
% optimization to maximize \Cref{eq:objective} and learn parameters jointly.
% Stochastic optimization requires subsampling the terms in the objective with
% negative labels. Such subsampling directly leads to a binary classification
% binary classification loss function with negative samples, as in previous work
% on word embeddings~\citep{mikolov2013distributed} and recommender
% systems~\citep{he2017neural,song2018neural}.
% The \gls{rfs} parameterization in
% \Cref{eqn:prob_y_given_um,eqn:argmax,eqn:rankfromsets} is bi-convex and could be
% fit through alternating coordinate descent. That is, fixing the user embeddings,
% the problem is convex in each of the other parameters; similarly, the problem is
% convex in the user parameters if the other parameters are fixed. For each
% coordinate in the user step, such a procedure would require iterating over all
% items. The massive number of items in the datasets in \Cref{sec:rfs-experiments}
% makes this intractable; similar issues could also arise for datasets with many
% users.
% \paragraph{Negative sampling} All observed positive labels (i.e. instances where
% $\yum = 1$) are divided into mini-batches. For each mini-batch, each
% positively-labeled sample is paired with a negatively-labeled sample. The
% negatively-labeled sample has the same user as its paired sample, but the item
% is sampled uniformly from a set of items. This procedure is not unbiased, but
% still leads to a consistent classifier: note that any reweighting of items that
% assigns nonzero weight to all items can be arbitrarily biased, but still lead to
% a consistent classifier. As long as every item has nonzero weight, the infinite
% data limit ensures that the classifier will learn to discriminate between
% positively- and negatively-labeled datapoints. We decide to balance
% positively-labeled datapoints with negative samples in each mini-batch as it is
% easy to implement as shown in \Cref{sec:code}.
% We note that any negative sampling
% scheme that assigns nonzero weight to every item leads to a consistent
% classifier.

% Ideally, the sampling set should be the set of all items which the user has not
% consumed, where~$\yum~=~0$. But maintaining and sampling from user-specific sets
% of negative labels can be computationally prohibitive. When the item set is
% large and labels are sparse (that is, $\yum = 0$ for most user, item pairs),
% embedding models are often trained by drawing negative samples uniformly from
% the item set~\citep{mikolov2013distributed}. For training \gls{rfs}, we propose
% two different negative sampling distributions that trade off quality of the
% learned parameters for scalability of the training procedure.

Negative samples can be drawn uniformly over the entire corpus of items, which we define to be corpus sampling. If the item set is large, this can be an expensive procedure. This negative sampling scheme leads to objective functions used in other recommender systems~\citep{he2017neural,song2018neural}.

On large datasets, it is infeasible to calculate Recall@$K$ for evaluation, as this requires ranking every item for every user (e.g. in \Cref{sec:rfs-experiments} we study a dataset with over $10$M items). We define a scalable evaluation metric based on recall, and describe how it leads to a natural choice of negative sampling distribution.

Sampled recall is defined as follows. Consider held-out datapoints with positive labels,~${(x_m, \yum = 1)}$. For every held-out datapoint, $K-1$ datapoints with negative labels~${(x_k, \yuk = 0)}$ are sampled from the rest of the held-out data, which together yield a set of $K$ datapoints. A recommendation model is used to rank the $K$ datapoints $r_{u1}, \ldots, r_{uK}$. SampledRecall@$k$ is the fraction of the $K$ held-out datapoints that the model ranks in the top~$k$:
\begin{equation}
  \textrm{SampledRecall@}k = \frac{1}{K}{\E_{um}\left[\sum_{r\in
\{\mb{r}_{u1},
\ldots \mb{r}_{uk}\}} y_{ur}\right]}\, .
\label{eq:sampled-recall}
\end{equation}

The expectation is over users and items in the held-out set of datapoints. This evaluation metric is scalable: instead of using a model to rank every item, SampledRecall@$k$ requires ranking only $K$ items. Sampled recall is~$1$~if $k~=~K$, as the held-out datapoint with $\yum = 1$ is in each list of $K$ datapoints to be ranked. This metric is used in recommender systems when the number of items is large~\citep{ebesu2018collaborative,yang2018openrec:}.
% For
% example, with $M=10$, an average sampled recall@1 of $0.5$ means for 50\% of the
% held-out datapoints, the item consumed by the user was ranked first by the model
% (out of $9$~items randomly sampled from the held-out data of other users). For
% the sampled recall metric we choose $M=10$, meaning $9$ `fake' meals are sampled
% uniformly from other users' meals in the held-out validation and test sets.

When sampled recall is used as an evaluation metric, batch sampling is a natural way to draw negative samples. Sampled recall is calculated on items drawn from other user's data. We define batch sampling as generating negative samples by permuting mini-batch items. Besides corresponding to the sampled recall metric, this technique is memory-efficient, as it requires that only the current mini-batch be in memory.

In addition to scalability, both negative sampling procedures above have
the advantage of implicitly balancing the classifier. As shown in
\citet{veitch2019empirical}, using stochastic gradient descent with
negative sampling is equivalent to a Monte Carlo approximation of the
reweighted (balanced) classification loss.


\documentclass[letterpaper]{article}

\usepackage[utf8x]{luainputenc}
\usepackage{aeguill}
%\usepackage{nopageno}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{mathrsfs}
\usepackage{fullpage}
\usepackage{fancyhdr}
\setlength{\headheight}{12pt}
\pagestyle{fancy}
\chead{Linear Algebra}
\lhead{October 7, 2015}
\rhead{Jon Allen}
\allowdisplaybreaks

\newcommand{\abs}[1]{\left\lvert #1 \right\rvert}

\begin{document}
%\renewcommand{\labelenumii}{\alph{enumii}.}
%\renewcommand{\labelenumiii}{\alph{enumiii}.}
%\renewcommand{\labelenumi}{(\arabic{enumi})}
\section*{3.1}
%6, 8, 9(b,c), 11, 15, 16.
\begin{enumerate}
\setcounter{enumi}{5}
\item
  \begin{enumerate}
  \item
  Let $U$ and $V$
  \end{enumerate}
\setcounter{enumi}{7}
\item
Let $\mathbf{v}_1,\dots,\mathbf{v}_k\in \mathbb{R}^n$ and let $\mathbf{v}\in \mathbb{R}^n$. Prove that $\text{Span}(\mathbf{v}_1,\dots,\mathbf{v}_k)=\text{Span}(\mathbf{v}_1,\dots,\mathbf{v}_k,\mathbf{v})$ if and only if $\mathbf{v}\in \text{Span}(\mathbf{v}_1,\dots,\mathbf{v}_k)$.

Define $A=\text{Span}(\mathbf{v}_1,\dots,\mathbf{v}_k)$ and $B=\text{Span}(\mathbf{v}_1,\dots,\mathbf{v}_k,\mathbf{v})$.

We know that $A\le R^{n}$.
\item
Determine the intersection of the subspaces $\mathcal{P}_1$ and $\mathcal{P}_2$ in each case:
  \begin{enumerate}
  \setcounter{enumii}{1}
  \item
    $\mathcal{P}_1=\text{Span}\left((1,2,2),(0,1,1)\right),\mathcal{P}_2=\text{Span}\left((2,1,1),(1,0,0)\right)$
  \item
    $\mathcal{P}_1=\text{Span}\left((1,0,-1),(1,2,3)\right),\mathcal{P}_2=\{\mathbf{x}:x_1-x_2+x_3=0\}$
  \end{enumerate}
\setcounter{enumi}{10}
\item
Suppose $V$ and $W$ are orthogonal subspaces of $\mathbb{R}^n$, i.e., $\mathbf{v}\cdot\mathbf{w}=0$ for every $\mathbf{v}\in V$ and every $\mathbf{w}\in W$. Prove that $V\cap V^{\perp}=\{\mathbf{0}\}$
\setcounter{enumi}{14}
\item
Let $A$ be an $m\times n$ matrix. Let $V\subset \mathbb{R}^n$ and $W\subset \mathbb{R}^m$ be subspaces.
  \begin{enumerate}
  \item
  Show that $\{\mathbf{x}\in \mathbb{R}^n:A\mathbf{x}\in W\}$ is a subspace of $\mathbb{R}^n$.
  \item
  Show that $\{\mathbf{y}\in \mathbb{R}^m:\mathbf{y}=A\mathbf{x}\text{ for some }\mathbf{x}\in V\}$ is a subspace of $\mathbb{R}^m$
  \end{enumerate}
\item
Suppose $A$ is a symmetric $n\times n$ matrix. Let $V\subset \mathbb{R}^n$ be a  subspace with the property that $A\mathbf{x}\in V$ for every $\mathbf{x}\in V$.
\end{enumerate}
\end{document}