\chapter{System F}

% 2.1 Syntax
% A simple, powerful calculus
% mention here how powerful the language is, yada yada...
% 2.1.1 Types
% 2.1.2 Terms
% 2.1.3 Values
% 2.1.4 Contexts
% 2.1.5 Evaluation

% 2.2 Typing, a relation
% 2.2.1 T-Var
% 2.2.2 T-Abs
% 2.2.3 T-TApp
% 2.2.4 ......

% 2.3 Properties of

% Worth discussing why System F came about. Same as with Girard's qualms with System T, I retreat to System F for its expressivity and lack of improving expressivity.
\begin{singlespace}
\setlength{\epigraphwidth}{0.6\textwidth}
\epigraph{\textit{So we are led to endless improvement, in order to be able to consider, besides the booleans, the integers, lists, trees, etc. Of course, all this is done to the detriment of conceptual simplicity and modularity.}}{\textsc{Jean-Yves Girard \\  Proofs and Types (1989)}}
\end{singlespace}

\section{System T, then F}
In the Dialectica interpretation, Kurt G{\"o}del constructs System T. With it, he proves the consistency of Heyting arithmetic: a logical framework for reasoning about natural numbers \cite{avigad1998godel}. That is, the axioms for Heyting arithmetic do not lead to contradictions about natural numbers.

Despite success, logicians like Girard expressed misgivings about System T. In order to reason over natural numbers, System T explicitly introduces machinery for reasoning over things like booleans and pairs. Because they aren't inherently representable in System T, you end up with the endless improvements Girard mentions in the epigraph. To do something new in System T, you add something new. What results is a language which quickly loses "conceptual simplicity and modularity."

To amend these misgivings, Girard constructs System F \cite{girard1989proofs}. It's powerful enough to represent all machinery for the Dialectica interpretation, whilst preserving conceptual simplicity and modularity.

Similar misgivings about languages which learn programs led my retreat to System F. To learn something new, you add something new. Very quickly, these languages get complex. The key contribution of this work shows that not only does System F provide all machinery for the Dialectica interpretation, but also for learning from examples. To do something new in System F, I add nothing new.

The remainder of the chapter presents System F, with notation near identical to its presentation in \cite{pierce2002types}. My presentation is terse, in order to provide intuitions for readers unfamiliar with lambda calculi. These intuitions ground the work developed hereafter. For comprehensive coverage, I defer to \cite{pierce2002types}.\\

\begin{figure}[h]
\centering
\setlength{\tabcolsep}{12pt}
\begin{tabular}{l  r}
\specialrule{.1em}{0em}{.2em}

\specialrule{.1em}{0em}{1em}
    \Large \textsc{Syntax} & \\
    & \\
    \begin{math}
    \setlength{\jot}{-2pt}
    \begin{aligned}
    e ::= \;& && \hspace*{.25in} \textsc{terms:}\\
        & x && \hspace*{.25in} \textit{variable}\\
        & e_1e_2 && \hspace*{.25in} \textit{application}\\
        & \lam x {:} \tau.e && \hspace*{.25in} \textit{abstraction}\\
        & e\lceil\tau\rceil && \hspace*{.25in} \textit{type application}\\
        & \Lambda\alpha.e && \hspace*{.25in} \textit{type abstraction}\\
    \\
    v ::= \;& && \hspace*{.25in} \textsc{values:} \\
        & \lam x {:}\tau.e && \hspace*{.25in} \textit{abstraction}\\
        & \Lambda\alpha.e && \hspace*{.25in} \textit{type abstraction}\\
    \end{aligned}
    \end{math} &
    \begin{math}
    \setlength{\jot}{-2pt}
    \begin{aligned}
    \tau ::= \;& && \hspace*{.25in} \textsc{types:}\\
        & \tau_1 \to \tau_2 && \hspace*{.25in} \textit{function type}\\
        & \forall\alpha.\tau && \hspace*{.25in} \textit{polymorphic type}\\
        & \alpha && \hspace*{.25in} \textit{type variable}\\
    \\
    \Gamma ::= \;& && \hspace*{.25in} \textsc{contexts:}\\
        & \cdot && \hspace*{.25in} \textit{empty}\\
        & x{:}\tau,\Gamma && \hspace*{.25in} \textit{variable}\\
        & \alpha,\Gamma && \hspace*{.25in} \textit{type variable}
    \end{aligned}
    \end{math}\\
    &\\
\specialrule{.1em}{1em}{0em}
\end{tabular}
\caption{Syntax in System F}
    \label{fig:syntax}
\end{figure}

\section{Syntax}

A convenience of System F is its minimal syntax. It's only marginally more  complex than the simplest typed languages. Figure \ref{fig:syntax} presents its grammar in Backus-Naur form \cite{pierce2002types}. Beyond aesthetics, the minimalism is a mathematical convenience. Proofs about System F's behavior need only worry about a handful of language constructs.

To ease presentation, we use encodings for natural numbers from the start. These encodings are standard, and are shown in \cite{girard1989proofs}.

\subsection{Types}

Types describe the behavior of programs.

\begin{enumerate}[label=\alph*)]
\item $nat$ is the type for natural numbers. A program of this type is a natural number.
\item $nat \!\to\!nat$ is the function type from $nat$ to $nat$. A program of this type has an input of type $nat$ and an output of type $nat$.
\item $nat \!\to\!nat\!\to\!nat$ is the function type from $nat$ and $nat$ to $nat$. A program of this type has two inputs of type $nat$ and an output of type $nat$.
\item $\forall\alpha.\alpha\!\to\!\alpha\!\to\!\alpha$ is the polymorphic function type from $\alpha$ and $\alpha$ to $\alpha$. A program of this type has two inputs of the same type and an output of that type. Polymorphic abstraction, denoted by $\forall\alpha$, lets the function work for any type represented by $\alpha$.
\item $\forall\alpha.\alpha\!\to\!\alpha\!\to\!nat$ is the polymorphic function type from $\alpha$ and $\alpha$ to $nat$. A program of this type has two inputs of the same type and an output which is always type $nat$.
\end{enumerate}

\subsection{Terms}

Terms are programs.

\begin{enumerate}[label=\alph*)]
\item $\lam x{:}nat.x$ is a program which takes a natural number $x$ as input, and returns it. The variable which comes after $\lam$ in a term denotes the input. What comes after is the term's body, where its input is used for computation.
\item $(\lam x{:}nat.x)1$ applies the previous program to $1$.

\item $\Lambda\alpha.\lam x{:}\alpha.x$ is a program which takes an $x$ of any type, and returns it. It's a polymorphic, or generic, version of the first program.
\item $(\Lambda\alpha.\lam x{:}\alpha.x)\lceil nat\rceil$ applies the previous program to type $nat$.

\item $\lam f{:}nat\!\to\!nat.\lam x{:}nat.fx$ is a program which takes a function $f$ of type $nat\!\to\!nat$ and a natural number $x$ as input. It returns the application of $f$ to $x$.
\end{enumerate}

\subsection{Values}

Values are programs which have finished computing.
\begin{enumerate}[label=\alph*)]
\item $(\lam x{:}nat.x)1$ is not a value. The program is an application, which can't be a value. Applications means there's computing left to do, namely the application of the program to its argument.
\item $(\lam x{:}nat)$ is a value. The program is applied to no arguments. There's no computing left to do.
\item $1$ is a value.
\end{enumerate}

\subsection{Contexts}

Contexts carry type information about variables.
\begin{enumerate}[label=\alph*)]
\item $x{:}nat, y{:}nat$ is a valid context. It says $x$ and $y$ have type $nat$.
\item $\alpha$ is a valid context. It says $\alpha$ is a type variable.
\item $\cdot$ is a valid context. It says nothing. At times, when other information is present in the context, we omit $\cdot$---which technically, is always present in the context.
\end{enumerate}

\section{Evaluating, a relation}

System F's syntax tells us what programs look like. But doesn't say anything about how they run, how they compute. Suppose I have the program $(\lam x{:}nat.x)1$. It applies a program which returns its input to the number $1$. It should return $1$. But how exactly? We want a relation which gives us the following behavior.
$$(\lam x{:}nat.x)1 \to_\beta x[1/x] \to_\beta 1$$
First, $1$ is bound to the input $x$ of $\lam x{:}nat.x$. Then $1$ substitutes $x$, resulting in $1$. The notation $x[1/x]$ denotes $1$ replacing $x$ in the program $x$. A relation of this behavior is defined in Figure \ref{fig:evaluating}.

\begin{figure}[h]
\centering
\setlength{\tabcolsep}{12pt}
\begin{tabular}{l r  l r}
\specialrule{.1em}{0em}{.2em}
\specialrule{.1em}{0em}{1em}
    \Large \textsc{Evaluating} &
    &  & \fbox{ $e \to_\beta e'$}\\
    & & \\
    \multicolumn{2}{c}{
    \def\extraVskip{4pt}
    \def\labelSpacing{4pt}
    \def\defaultHypSeparation{\hskip .05in}
        \AxiomC{$e_1 \to_\beta e_1'$}
            \RightLabel{\textsc{(E-App1)}}
        \UnaryInfC{$e_1e_2 \to_\beta e_1'e_2$}
        \DisplayProof
    } &
    \multicolumn{2}{c}{
    \def\extraVskip{4pt}
    \def\labelSpacing{4pt}
    \def\defaultHypSeparation{\hskip .05in}
        \AxiomC{$e \to_\beta e'$}
            \RightLabel{\textsc{(E-TApp)}}
        \UnaryInfC{$e\lceil\tau\rceil \to_\beta e'\lceil\tau\rceil$}
        \DisplayProof
    }
    \\
    & &\\
    \multicolumn{2}{c}{
    \def\extraVskip{4pt}
    \def\labelSpacing{4pt}
    \def\defaultHypSeparation{\hskip .05in}
        \AxiomC{$e_2 \to_\beta e_2'$}
            \RightLabel{\textsc{(E-App2)}}
        \UnaryInfC{$e_1e_2 \to_\beta e_1e_2'$}
        \DisplayProof
    } &
    \multicolumn{2}{c}{
    \def\extraVskip{4pt}
    \def\labelSpacing{4pt}
    \def\defaultHypSeparation{\hskip .05in}
        \AxiomC{$(\Lambda\alpha.\lam x{:}\alpha.e)\lceil\tau\rceil \to_\beta (\lam x{:}\alpha.e)[\tau/\alpha]\,\,$\textsc{(E-TSub)}}
        \DisplayProof
    }
    \\
    & &\\
    \multicolumn{2}{c}{
    \def\extraVskip{4pt}
    \def\labelSpacing{4pt}
    \def\defaultHypSeparation{\hskip .05in}
        \AxiomC{$(\lam x{:}\tau.e)v \to_\beta e[v/x]\,\,$\textsc{(E-Sub)}}
        \DisplayProof
    } \\
    & \\
\specialrule{.1em}{1em}{0em}
\end{tabular}
\caption{Evaluating in System F}
    \label{fig:evaluating}
\end{figure}

\textsc{(E-App1)} says that if a program $e_1$ evaluates to $e_1'$, then $e_1e_2$ evaluates to $e_1'e_2$. For example:
\begin{prooftree}
\def\extraVskip{4pt}
\def\labelSpacing{4pt}
	\AxiomC{$(\lam x{:}nat\!\to\!nat.x)(\lam y{:}nat.y) \to_\beta (\lam y{:}nat.y)$}
	\RightLabel{\textsc{(E-App1)}}
	\UnaryInfC{$((\lam x{:}nat\!\to\!nat.x)(\lam y{:}nat.y))1 \to_\beta (\lam y{:}nat.y)1$}
\end{prooftree}

\textsc{(E-App2)} says that if a program $e_2$ evaluates to $e_2'$, then $e_1e_2$ evaluates to $e_1e_2'$. For example:
\begin{prooftree}
\def\extraVskip{4pt}
\def\labelSpacing{4pt}
	\AxiomC{$(\lam x{:}nat.x)1 \to_\beta 1$}
	\RightLabel{\textsc{(E-App2)}}
	\UnaryInfC{$(\lam x{:}nat.x)((\lam x{:}nat.x)1) \to_\beta (\lam x{:}nat.x)1$}
\end{prooftree}

\textsc{(E-Sub)} says that if a program $\lam x{:}\tau.e$ is applied to value $v$, then replace all instances of $x$ in $e$ with $v$. For example: \vspace*{-1.2em}
\begin{prooftree}
\def\extraVskip{4pt}
\def\labelSpacing{4pt}
	\AxiomC{$(\lam x{:}nat.x)1 \to_\beta x[1/x]\,\,$\textsc{(E-Sub)}}
\end{prooftree}

\textsc{(E-TApp)} says that if a program $e$ evaluates to $e'$, then $e\lceil\tau\rceil$ evaluates to $e'\lceil\tau\rceil$. For example:
\begin{prooftree}
\def\extraVskip{4pt}
\def\labelSpacing{4pt}
	\AxiomC{$\Lambda\alpha.(\lam x{:}\alpha\!\to\!\alpha.x)(\lam x{:}\alpha.x) \to_\beta \Lambda\alpha.\lam x{:}\alpha.x$}
	\RightLabel{\textsc{(E-TApp)}}
	\UnaryInfC{$(\Lambda\alpha.(\lam x{:}\alpha\!\to\!\alpha.x)(\lam x{:}\alpha.x))\lceil nat\rceil \to_\beta (\Lambda\alpha.\lam x{:}\alpha.x)\lceil nat\rceil$}
\end{prooftree}

\textsc{(E-TSub)} says that if a program $\Lambda\alpha.\lam x{:}\alpha.e$ is applied to type $\tau$, then replace all instances of $\alpha$ in $\lam x{:}\alpha.e$ with $\tau$. For example: \vspace*{-1.2em}
\begin{prooftree}
\def\extraVskip{4pt}
\def\labelSpacing{4pt}
	\AxiomC{$(\Lambda\alpha.\lam x{:}\alpha.x)\lceil nat\rceil \to_\beta (\lam x{:}\alpha.x)[nat/\alpha]\,\,$\textsc{(E-TSub)}}
\end{prooftree}

The evaluating relation comes in many forms. Here, we use what's referred to as a call-by-value evaluation strategy \cite{plotkin1975call}. More exist, and each impact how programs evaluate in System F, e.g. call-by-name and call-by-need \cite{plotkin1975call, ariola1997call}. These alternatives aren't discussed, but we do consider the reflexive, symmetric, transitive closure of the defined evaluating relation, $=_\beta$. It denotes program equality. Because this evaluating relation has the normalization property, deciding program equality is decidable. In upcoming chapters, we'll see how program equality is used to learn from examples.

\section{Typing, a relation}

With syntax, we saw what programs look like. With evaluation, we saw how programs execute. So then why do we need typing? Without typing, we can write the following program in System F. The grammar in Figure \ref{fig:syntax} permits it. \vspace*{-1.0em}
\begin{singlespace}
$$(\lam x{:}nat.x)(\lam y{:}nat.y)$$
\end{singlespace}
But this program is nonsense. The left-hand side of the application is a program which returns its argument $x$, which must be of type $nat$. The right-hand side is the same program. If we were to naively evaluate this program, it would substitute $\lam x{:}nat.x$ for the argument $x$ on the left-hand side. Yet the argument wouldn't be of type $nat$. $\lam x{:}nat.x$ is the function type $nat\!\to\!nat$. The types don't align for evaluation, to do computation.

To actually do something, the left-hand side needs an argument of the correct type. \vspace*{-1.0em}
\begin{singlespace}
$$(\lam x{:}nat\!\to\!nat.x)(\lam y{:}nat.y)$$
\end{singlespace}
This is why we need typing. It lets us build programs which make sense, for which types align to do computation. Like evaluation, we define typing as a relation.\\
\begin{figure}[h]
\centering
\setlength{\tabcolsep}{12pt}
\begin{tabular}{l r  l r}
\specialrule{.1em}{0em}{.2em}
\specialrule{.1em}{0em}{1em}
    \Large \textsc{Typing} &
    &  & \framebox{$\Gamma \vdash e : \tau$}\\
    & & \\
    \multicolumn{2}{c}{
    \def\extraVskip{4pt}
    \def\labelSpacing{4pt}
    \def\defaultHypSeparation{\hskip .05in}
        \AxiomC{$x:\tau \in \Gamma$}
            \RightLabel{\textsc{(T-Var)}}
        \UnaryInfC{$\Gamma \vdash x : \tau$}
        \DisplayProof
    } &
    \multicolumn{2}{c}{
    \def\extraVskip{4pt}
    \def\labelSpacing{4pt}
    \def\defaultHypSeparation{\hskip .05in}
        \AxiomC{$\Gamma,\alpha \vdash e : \tau$}
            \RightLabel{\textsc{(T-TAbs)}}
        \UnaryInfC{$\Gamma \vdash \Lambda \alpha.e:\forall\alpha.\tau$}
        \DisplayProof
    }
    \\
    & &\\
    \multicolumn{2}{c}{
    \def\extraVskip{4pt}
    \def\labelSpacing{4pt}
    \def\defaultHypSeparation{\hskip .05in}
        \AxiomC{$\Gamma,x{:}\tau_1 \vdash e_2 : \tau_2$}
            \RightLabel{\textsc{(T-Abs)}}
        \UnaryInfC{$\Gamma \vdash \lam x{:}\tau_1.e_2:\tau_1 \to \tau_2$}
        \DisplayProof
    } &
    \multicolumn{2}{c}{
    \def\extraVskip{4pt}
    \def\labelSpacing{4pt}
    \def\defaultHypSeparation{\hskip .05in}
        \AxiomC{$\Gamma \vdash e : \forall\alpha.\tau_1$}
            \RightLabel{\textsc{(T-TApp)}}
        \UnaryInfC{$\Gamma \vdash e\lceil\tau_2\rceil : [\tau_2/\alpha]\tau_1$}
        \DisplayProof
    }
    \\
    & &\\
    \multicolumn{2}{c}{
    \def\extraVskip{4pt}
    \def\labelSpacing{4pt}
    \def\defaultHypSeparation{\hskip .05in}
        \AxiomC{$\Gamma \vdash e_1 : \tau_1 \to \tau_2$}
        \AxiomC{$\Gamma \vdash e_2 : \tau_1$}
            \RightLabel{\textsc{(T-App)}}
        \BinaryInfC{$\Gamma \vdash e_1e_2 : \tau_2$}
        \DisplayProof
    } \\
    & \\
\specialrule{.1em}{1em}{0em}
\end{tabular}
\caption{Typing in System F}
    \label{fig:typing}
\end{figure}

\textsc{(T-Var)} says that if $x$ is bound to type $\tau$ in the context $\Gamma$, then the program $x$ is of type $\tau$.
\begin{prooftree}
\def\extraVskip{4pt}
\def\labelSpacing{4pt}
	\AxiomC{$x {:} nat \in x {:} nat$}
	\RightLabel{\textsc{(T-Var)}}
	\UnaryInfC{$x{:}nat \vdash x : nat$}
\end{prooftree}

\textsc{(T-Abs)} says that if $x$ is bound to type $\tau_1$ in the context $\Gamma$ and that a program $e_2$ is of type $\tau_2$, then the program $\lam x{:}\tau_1.e_2$ is of type $\tau_1 \!\to\! \tau_2$ and $x$ is removed from the context.
\begin{prooftree}
\def\extraVskip{4pt}
\def\labelSpacing{4pt}
	\AxiomC{$\Gamma,x{:}nat \vdash \lam y{:}nat.y : nat \!\to\! nat$}
	\RightLabel{\textsc{(T-Abs)}}
	\UnaryInfC{$\Gamma \vdash \lam x{:}nat.\lam y{:}nat.y : nat \!\to\! nat \!\to \!nat$}
\end{prooftree}

\textsc{(T-App)} says that if a program $e_1$ is of type $\tau_1 \!\to\! \tau_2$ and a program $e_2$ is of type $\tau_1$, then  $e_1e_2$ is of type $\tau_2$.
\begin{prooftree}
\def\extraVskip{4pt}
\def\labelSpacing{4pt}
	\AxiomC{$\Gamma \vdash \lam x{:}nat.x : nat \!\to\! nat$}
	\AxiomC{$\Gamma \vdash 1 : nat$}
	\RightLabel{\textsc{(T-App)}}
	\BinaryInfC{$\Gamma \vdash (\lam x{:}nat.x)1 : nat$}
\end{prooftree}

\textsc{(T-TAbs)} says that if $\alpha$ is in the context, and a program $e$ is of type $\tau$, then the program $\Lambda\alpha.e$ is of type $\forall\alpha.\tau$ and $\alpha$ is removed from the context.
\begin{prooftree}
\def\extraVskip{4pt}
\def\labelSpacing{4pt}
	\AxiomC{$\Gamma,\alpha \vdash \lam x{:}nat.x : \alpha \!\to\! \alpha$}
	\RightLabel{\textsc{(T-TAbs)}}
	\UnaryInfC{$\Gamma \vdash \Lambda\alpha.\lam x{:}\alpha.x : \forall\alpha.\alpha \!\to\! \alpha$}
\end{prooftree}

\textsc{(T-TApp)} says that if a program $e$ is of type $\forall\alpha.\tau_1$, then  the program $e\lceil\tau_2\rceil$ is of type $[\tau_2/\alpha]\tau_1$.
\begin{prooftree}
\def\extraVskip{4pt}
\def\labelSpacing{4pt}
	\AxiomC{$\Gamma \vdash \Lambda\alpha.\lam x{:}\alpha.x : \forall\alpha.\alpha\!\to\!\alpha$}
	\RightLabel{\textsc{(T-Abs)}}
	\UnaryInfC{$\Gamma \vdash (\Lambda\alpha.\lam x{:}\alpha.x)\lceil nat\rceil : nat\!\to\!nat$}
\end{prooftree}

As with evaluating, there are many ways to construct typing relations on System F. Each offers their own take on what constitutes a well-typed program in System F, e.g. System F extended with subtyping \cite{cardelli1991extension}. The behavior of the typing relation is especially important for learning, as learning will depend on key properties of the typing relation: namely progress and preservation.

\section{Metatheory}
Well-typed terms can be thought of as theorems constructed through the typing relation, whose proof are sound proof trees. Hence, statements about these theorems, or the typing relation in general, are metatheoretic statements. I briefly review key metatheoretic properties of System F essential for learning. Proofs ommitted, but easily found in \cite{girard1989proofs}.
\subsection{Programs don't get stuck}
\begin{theorem}[\textsc{Progress in Typing}]
If $e$ is a closed, well-typed program, then either $e$ is a value or else there is some program $e'$ such that $e \to_\beta e'$.
\label{progress-typing}
\end{theorem}

Programs shouldn't get stuck in the middle of computation. We want well-typed programs to always have something to do, even if they don't ever stop. Progress ensures this. And with preservation, it lets us prove typing is sound, that we can prove all true theorems implied by the typing relation.
\subsection{Programs don't change types}
\begin{theorem}[\textsc{Preservation in Typing}]
If $\,\Gamma \vdash e : \tau$ and $e \to_\beta e'$, then $\Gamma \vdash e' : \tau$.
\label{preservation-typing}
\end{theorem}

Programs shouldn't suddenly switch types in the middle of computation. Remember, the point of the type is to let us know kind of computation to expect from a program. If the type switches during computation, we're losing that information. With progress, preservation lets us prove typing is sound. By extension, they are essential to prove learning sound.

\subsection{Programs always halt}
\begin{theorem}[\textsc{Normalization in Evaluation}]
Well-typed programs in System F always evaluate to a value, to a normal form.
\label{normalization-evaluation}
\end{theorem}

Progress doesn't guarantee programs ever stop computing. They could get stuck in infinite loops. Under certain type systems it's impossible to guarantee programs ever stop, e.g. recursive types \cite{pierce2002types}. But in System F we know all programs halt, or stop. For learning from examples, introduced in subsequent chapters, this property is essential. With normalization, program equivalence becomes decidable---a key part of the learning procedure.

% Evaluation
% --Church-Rosser
% --Strongly Normalizing
% Typing
% --Progress
% --Preservation


\part{Inference}

\chapter{Bayesian Data Analysis}\label{bayesian.chapter}

\noindent
\cite{GelmanEtAl:2013} provide the following
characterization of Bayesian data analysis.
%
\begin{quote}
  By Bayesian data analysis, we mean practical methods for making
  inferences from data using probability models for quantities we
  observe and about which we wish to learn.
\end{quote}
%
They go on to describe how Bayesian statistics differs from
frequentist approaches.
%
\begin{quote}
  The essential characteristic of Bayesian methods is their explicit
  use of probability for quantifying uncertainty in inferences based
  on statistical analysis.
\end{quote}
%
Because they view probability as the limit of relative frequencies of
observations, strict frequentists forbid probability statements about
parameters.  Parameters are considered fixed, not random.

Bayesians also treat parameters as fixed but unknown.  But unlike
frequentists, they make use of both prior distributions over
parameters and posterior distributions over parameters.  These prior
and posterior probabilities and posterior predictive probabilities are
intended to characterize knowledge about the parameters and future
observables.  Posterior distributions form the basis of Bayesian
inference, as described below.

\section{Bayesian Modeling}

\citep{GelmanEtAl:2013} break applied Bayesian modeling
into the following three steps.
%
\begin{enumerate}
\item  Set up a full probability model for all observable and
  unobservable quantities.  This model should be consistent with
  existing knowledge of the data being modeled and how it was
  collected.
\item Calculate the posterior probability of unknown quantities
  conditioned on observed quantities.  The unknowns may include
  unobservable quantities such as parameters and potentially
  observable quantities such as predictions for future observations.
\item Evaluate the model fit to the data.  This includes evaluating
  the implications of the posterior.
\end{enumerate}
%
Typically, this cycle will be repeated until a sufficient fit is
achieved in the third step.  Stan automates the calculations involved
in the second and third steps.

\section{Bayesian Inference}

\subsection{Basic Quantities}

The mechanics of Bayesian inference follow directly from Bayes's rule.
To fix notation, let $y$ represent observed quantities such as data
and let $\theta$ represent unknown quantities such as parameters and
future observations.  Both $y$ and $\theta$ will be modeled as random.
Let $x$ represent known, but unmodeled quantities such as constants,
hyperparameters, and predictors.

\subsection{Probability Functions}

The probability function $p(y,\theta)$ is the joint probability
function of the data $y$ and parameters $\theta$.  The constants and
predictors $x$ are implicitly understood as being part of the
conditioning.  The conditional probability function $p(y|\theta)$ of
the data $y$ given parameters $\theta$ and constants $x$ is called the
sampling probability function; it is also called the likelihood
function when viewed as a function of $\theta$ for fixed $y$ and $x$.

The probability function $p(\theta)$ over the parameters given the
constants $x$ is called the prior because it characterizes the probability
of the parameters before any data is observed.  The conditional
probability function $p(\theta|y)$ is called the posterior because
it characterizes the probability of parameters given observed data $y$
and constants $x$.

\subsection{Bayes's Rule}

The technical apparatus of Bayesian inference hinges on the following
chain of equations, known in various forms as Bayes's rule (where
again, the constants $x$ are implicit).
%
\[
\begin{array}{rcll}
p(\theta|y)  & =  & \displaystyle \frac{p(\theta,y)}{p(y)}
& \mbox{{} \ \ \ \ \ [definition of  conditional probability]}
\\[16pt]
& = & \displaystyle \frac{p(y|\theta) \, p(\theta)}{p(y)}
& \mbox{{} \ \ \ \ \ [chain rule]}
\\[16pt]
& = & \displaystyle \frac{p(y|\theta) \, p(\theta)}
                        {\int_{\Theta} p(y,\theta) \, d\theta}
& \mbox{{} \ \ \ \ \ [law of total probability]}
\\[16pt]
& = & \displaystyle \frac{p(y|\theta) \, p(\theta)}
                        {\int_{\Theta} p(y|\theta) \, p(\theta) \, d\theta}
& \mbox{{} \ \ \ \ \ [chain rule]}
\\[16pt]
& \propto & \displaystyle p(y|\theta) \, p(\theta)
& \mbox{{} \ \ \ \ \ [$y$ is fixed]}
\end{array}
\]
%
Bayes's rule ``inverts'' the probability of the posterior
$p(\theta|y)$, expressing it solely in terms of the likelihood
$p(y|\theta)$ and prior $p(\theta)$ (again, with constants and
predictors $x$ implicit).  The last step is important for Stan, which
only requires probability functions to be characterized up to a
constant multiplier.

\subsection{Predictive Inference}

The uncertainty in the estimation of parameters $\theta$ from the data
$y$ (given the model) is characterized by the posterior $p(\theta|y)$.
The posterior is thus crucial for Bayesian predictive inference.

If $\tilde{y}$ is taken to represent new, perhaps as yet unknown,
observations, along with corresponding constants and predictors
$\tilde{x}$, then the posterior predictive probability function is
given by
%
\[
p(\tilde{y}|y)
= \int_{\Theta} p(\tilde{y}|\theta)
                \, p(\theta|y) \, d\theta.
\]
Here, both the original constants and predictors $x$ and the new
constants and predictors $\tilde{x}$ are implicit.  Like the posterior
itself, predictive inference is characterized probabilistically.
Rather than using a point estimate of the parameters $\theta$,
predictions are made based on averaging the predictions over a range
of $\theta$ weighted by the posterior probability $p(\theta|y)$ of
$\theta$ given data $y$ (and constants $x$).

The posterior may also be used to estimate event probabilities.  For
instance, the probability that a parameter $\theta_k$ is greater than
zero is characterized probabilistically by
%
\[
\mbox{Pr}[\theta_k > 0]
= \int_{\Theta} \mbox{I}(\theta_k > 0) \, p(\theta|y) \, d\theta.
\]
%
The indicator function, $\mbox{I}(\phi)$, evaluates to one if the
proposition $\phi$ is true and evaluates to zero otherwise.

Comparisons involving future observables may be carried out in
the same way.  For example, the probability that $\tilde{y}_n >
\tilde{y}_{n'}$ can be characterized using the posterior predictive
probability function as
\[
\mbox{Pr}[\tilde{y}_n > \tilde{y}_{n'}]
= \int_{\Theta} \int_{Y} \mbox{I}(\tilde{y}_n > \tilde{y}_{n'}) \,
p(\tilde{y}|\theta) p(\theta|y) \, d\tilde{y} \, d\theta.
\]

\subsection{Posterior Predictive Checking}

After the parameters are fit to data, they can be used to simulate a
new data set by running the model inferences in the forward
direction.  These replicated data sets can then be compared to the
original data either visually or statistically to assess model fit
\citep[Chapter 6]{GelmanEtAl:2013}.

In Stan, posterior simulations can be generated in two ways.  The
first approach is to treat the predicted variables as parameters and
then define their distributions in the model block.  The second
approach, which also works for discrete variables, is to generate
replicated data using random-number generators in the generated
quantities block.

\chapter{Markov Chain Monte Carlo Sampling}\label{mcmc.chapter}

\noindent
Stan uses Markov chain Monte Carlo (\MCMC) techniques to
generate samples from the posterior distribution for inference.

\section{Monte Carlo Sampling}

Monte Carlo methods were developed to numerically approximate
integrals that are not tractable analytically but for which evaluation
of the function being integrated is tractable
\citep{MetropolisUlam:1949}.

For example, the mean $\mu$ of a probability density $p(\theta)$ is
defined by the integral
\[
\mu = \int_{\Theta} \, \theta \times p(\theta) \, d\theta.
\]
For even a moderately complex Bayesian model, the posterior density
$p(\theta|y)$ leads to an integral that is impossible to evaluate
analytically.  The posterior also depends on the constants and
predictors $x$, but from here, they will just be elided and taken as
given.

Now suppose it is possible to draw independent samples from
$p(\theta)$ and let $\theta^{(1)},\theta^{(2)},\ldots,\theta^{(N)}$ be
$N$ such samples.  A Monte Carlo estimate $\hat{\mu}$ of the mean
$\mu$ of $p(\theta)$ is given by the sample average,
\[
\hat{\mu} = \frac{1}{N} \sum_{n=1}^N \theta^{(n)}.
\]

If the probability function $p(\theta)$ has a finite mean and
variance, the law of large numbers ensures the Monte Carlo estimate
converges to the correct value as the number of samples increases,
\[
\lim_{N \rightarrow \infty} \hat{\mu} = \mu.
\]
Assuming finite mean and variance, estimation error is governed by the
central limit theorem, so that estimation error decreases as the
square root of $N$,
\[
|\mu - \hat{\mu}| \propto \frac{1}{\sqrt{N}}.
\]
Therefore, estimating a mean to an extra decimal place of accuracy
requires one hundred times more samples; adding two decimal places
means ten thousand times as many samples.  This makes Monte Carlo
methods more useful for rough estimates to within a few decimal places
than highly precise estimates.  In practical applications, there is no
point estimating a quantity beyond the uncertainty of the data sample
on which it is based, so this lack of many decimal places of accuracy
is rarely a problem in practice for statistical models.

\section{Markov Chain Monte Carlo Sampling}

Markov chain Monte Carlo (\MCMC) methods were developed for situations
in which it is not straightforward to draw independent samples
\citep{Metropolis:1953}.

A Markov chain is a sequence of random variables $\theta^{(1)},
\theta^{(2)},\ldots$ where each variable is conditionally independent
of all other variables given the value of the previous value.  Thus if
$\theta = \theta^{(1)}, \theta^{(2)},\ldots, \theta^{(N)}$, then
\[
p(\theta) = p(\theta^{(1)}) \prod_{n=2}^N p(\theta^{(n)}|\theta^{(n-1)}).
\]
Stan uses Hamiltonian Monte Carlo to generate a next state in a manner
described in \refchapter{hmc}.

The Markov chains Stan and other \MCMC samplers generate are ergodic
in the sense required by the Markov chain central limit theorem,
meaning roughly that there is a reasonable chance of reaching
one value of $\theta$ from another.  The Markov chains are also
stationary, meaning that the transition probabilities do not change at
different positions in the chain, so that for $n, n' \geq 0$, the
probability function $p(\theta^{(n+1)}|\theta^{(n)})$ is the same as
$p(\theta^{(n'+1)}|\theta^{(n')})$ (following the convention of
overloading random and bound variables and picking out a probability
function by its arguments).

Stationary Markov chains have an equilibrium distribution on states in
which each has the same marginal probability function, so that
$p(\theta^{(n)})$ is the same probability function as
$p(\theta^{(n+1)})$.  In Stan, this equilibrium distribution
$p(\theta^{(n)})$ is the probability function $p(\theta)$ being
sampled, typically a Bayesian posterior density.

Using \MCMC methods introduces two difficulties that are not faced by
independent sample Monte Carlo methods.  The first problem is determining
when a randomly initialized Markov chain has converged to its
equilibrium distribution.  The second problem is that the draws from a
Markov chain are correlated, and thus the central limit theorem's
bound on estimation error no longer applies.  These problems are
addressed in the next two sections.

\section{Initialization and Convergence Monitoring}\label{convergence.section}

A Markov chain generates samples from the target distribution only
after it has converged to equilibrium.  Unfortunately, this is only
guaranteed in the limit in theory.  In practice, diagnostics must be
applied to monitor whether the Markov chain(s) have converged.

\subsection{Potential Scale Reduction}

One way to monitor whether a chain has converged to the equilibrium
distribution is to compare its behavior to other randomly initialized
chains.  This is the motivation for the \cite{GelmanRubin:1992}
potential scale reduction statistic, $\hat{R}$.  The $\hat{R}$
statistic measures the ratio of the average variance of samples within
each chain to the variance of the pooled samples across chains; if all
chains are at equilibrium, these will be the same and $\hat{R}$ will
be one.  If the chains have not converged to a common distribution,
the $\hat{R}$ statistic will be greater than one.

Gelman and Rubin's recommendation is that the independent Markov
chains be initialized with diffuse starting values for the parameters
and sampled until all values for $\hat{R}$ are below 1.1.  Stan
allows users to specify initial values for parameters and it is also
able to draw diffuse random initializations itself.

The $\hat{R}$ statistic is defined for a set of $M$ Markov chains,
$\theta_m$, each of which has $N$ samples $\theta^{(n)}_m$.  The
between-sample variance estimate is
\[
B
= \frac{N}{M-1} \, \sum_{m=1}^M (\bar{\theta}^{(\bullet)}_{m} - \bar{\theta}^{(\bullet)}_{\bullet})^2,
\]
%
where
%
\[
\bar{\theta}_m^{(\bullet)}
= \frac{1}{N} \sum_{n = 1}^N \theta_m^{(n)}
\ \ \ \ \
\mbox{and}
\ \ \ \ \
\bar{\theta}^{(\bullet)}_{\bullet}
= \frac{1}{M} \, \sum_{m=1}^M \bar{\theta}_m^{(\bullet)}.
\]
%
The within-sample variance is
\[
W
= \frac{1}{M} \, \sum_{m=1}^M s_m^2,
\]
where
\[
s_m^2 = \frac{1}{N-1} \, \sum_{n=1}^N (\theta^{(n)}_m - \bar{\theta}^{(\bullet)}_m)^2.
\]
%
The variance estimator is
\[
\widehat{\mbox{var}}^{+}\!(\theta|y)
= \frac{N-1}{N}\, W \, + \, \frac{1}{N} \, B.
\]
%
Finally, the potential scale reduction statistic is defined by
\[
\hat{R}
\, = \,
\sqrt{\frac{\widehat{\mbox{var}}^{+}\!(\theta|y)}{W}}.
\]

\subsection{Generalized $\hat{R}$ for Ragged Chains}

Now suppose that each chain may have a different number of samples.
Let $N_m$ be the number of samples in chain $m$.  Now the formula for
the within-chain mean for chain $m$ uses the size of the chain, $N_m$,
\[
\bar{\theta}_m^{(\bullet)}
= \frac{1}{N_m} \sum_{n = 1}^N \theta^{(m)}_n,
\]
as does the within-chain variance estimate,
\[
s_m^2 = \frac{1}{N_m-1} \, \sum_{n=1}^{N_m} (\theta^{(n)}_m - \bar{\theta}^{(\bullet)}_m)^2.
\]
The terms that average over chains, such as
$\bar{\theta}^{(\bullet)}_{\bullet}$, $B$, and $W$, have the same
definition as before to ensure that each chain has the same effect on
the estimate.  If the averages were weighted by size, a single long
chain would dominate the statistics and defeat the purpose of
monitoring convergence with multiple chains.

Because it contains the term $N$, the estimate $\widehat{var}^{+}$
must be generalized.  By expanding the first term,
\[
\frac{N-1}{N}\, W \,
\ = \
\frac{N-1}{N} \frac{1}{M} \, \sum_{m=1}^M
\frac{1}{N-1} \, \sum_{n=1}^N (\theta^{(n)}_m -
\bar{\theta}^{(\bullet)}_m)^2
\ = \
\frac{1}{M}
\sum_{m=1}^M
\frac{1}{N}
\sum_{n=1}^N (\theta^{(n)}_m -
\bar{\theta}^{(\bullet)}_m)^2,
\]
and the second term,
\[
\frac{1}{N}\, B
\ = \
\frac{1}{M-1} \, \sum_{m=1}^M (\bar{\theta}^{(\bullet)}_{m} - \bar{\theta}^{(\bullet)}_{\bullet})^2.
\]
the variance estimator naturally generalizes to
\[
\widehat{\mbox{var}}^{+}\!(\theta|y)
=
\frac{1}{M}
\sum_{m=1}^M
\frac{1}{N_m}
\sum_{n=1}^{N_m} (\theta^{(n)}_m -
\bar{\theta}^{(\bullet)}_m)^2
+
\frac{1}{M-1} \, \sum_{m=1}^M (\bar{\theta}^{(\bullet)}_{m} -
\bar{\theta}^{(\bullet)}_{\bullet})^2.
\]
%
If the chains are all the same length, this definition is equivalent
to the one in the last section.  This generalized variance estimator
and the within-chains variance estimates may be plugged directly into
the formula for $\hat{R}$ from the previous section.

\subsection{Split $\hat{R}$ for Detecting Non-Stationarity}

Before calculating the potential-scale-reduction statistic $\hat{R}$,
each chain may be split into two halves.  This provides an additional
means to detect non-stationarity in the chains.  If one chain involves
gradually increasing values and one involves gradually decreasing
values, they have not mixed well, but they can have $\hat{R}$ values
near unity.  In this case, splitting each chain into two parts leads
to $\hat{R}$ values substantially greater than 1 because the first
half of each chain has not mixed with the second half.

\subsection{Convergence is Global}

A question that often arises is whether it is acceptable to monitor
convergence of only a subset of the parameters or generated
quantities.  The short answer is ``no,'' but this is elaborated
further in this section.

For example, consider the value \code{lp\_\_}, which is the log
posterior density (up to a constant) It is a mistake to declare
convergence in any practical sense if \code{lp\_\_} has not converged,
because different chains are really in different parts of the space.
Yet measuring convergence for \code{lp\_\_} is particularly tricky, as
noted below.

\subsubsection{Asymptotics and transience vs.\ equilibrium}

Markov chain convergence is a global property in the sense that it
does not depend on the choice of function of the parameters that is
monitored.  There is no hard cutoff between pre-convergence
``transience'' and post-convergence ``equilibrium.''  What happens is
that as the number of states in the chain approaches infinity, the
distribution of possible states in the chain approaches the target
distribution and in that limit the expected value of the Monte Carlo
estimator of any integrable function converges to the true
expectation. There is nothing like warmup here, because in the limit,
the effects of initial state are completely washed out.

\subsubsection{Multivariate convergence of functions}

The $\hat{R}$ statistic considers the composition of a Markov chain
and a function, and if the Markov chain has converged then each Markov
chain and function composition will have converged. Multivariate
functions converge when all of their margins have converged by the
Cramer-Wold theorem.

The transformation from unconstrained space to constrained space is
just another function, so does not effect convergence.

Different functions may have different autocorrelations, but if the
Markov chain has equilibrated then all Markov chain plus function
compositions should be consistent with convergence. Formally, any
function that appears inconsistent is of concern and although it would
be unreasonable to test every function, \code{lp\_\_} and other
measured quantities should at least be consistent.

The obvious difference in \code{lp\_\_} is that it tends to vary
quickly with position and is consequently susceptible to outliers.

\subsubsection{Finite numbers of states}

The question is what happens for finite numbers of states? If we can
prove a strong geometric ergodicity property (which depends on the
sampler and the target distribution), then one can show that there
exists a finite time after which the chain forgets its initial state
with a large probability. This is both the autocorrelation time and
the warmup time.  But even if you can show it exists and is finite
(which is nigh impossible) you can't compute an actual value
analytically.

So what we do in practice is hope that the finite number of draws is
large enough for the expectations to be reasonably accurate. Removing
warmup iterations improves the accuracy of the expectations but there
is no guarantee that removing any finite number of samples will be
enough.

\subsubsection{Why inconsistent $\hat{R}$?}

There are two things to worry about here.

Firstly, as noted above, for any finite number of draws, there will
always be some residual effect of the initial state, which typically
manifests as some small (or large if the autocorrelation time is huge)
probability of having a large outlier. Functions robust to such
outliers (say, quantiles) will appear more stable and have better
$\hat{R}$. Functions vulnerable to such outliers may show fragility.

Secondly, use of the $\hat{R}$ statistic makes very strong
assumptions. In particular, it assumes that the functions being
considered are Gaussian or it only uses the first two moments and
assumes some kind of independence.  The point is that strong
assumptions are made that do not always hold. In particular, the
distribution for the log posterior density (\code{lp\_\_}) almost
never looks Gaussian, instead it features long tails that can lead to
large $\hat{R}$ even in the large $N$ limit.  Tweaks to $\hat{R}$,
such as using quantiles in place of raw values, have the flavor of
making the samples of interest more Gaussian and hence the $\hat{R}$
statistic more accurate.

\subsubsection{Final words on convergence monitoring}

``Convergence'' is a global property and holds for all integrable
functions at once, but employing the $\hat{R}$ statistic requires
additional assumptions and thus may not work for all functions equally
well.

Note that if you just compare the expectations between chains then we
can rely on the Markov chain asymptotics for Gaussian distributions
and can apply the standard tests.

\section{Effective Sample Size}\label{effective-sample-size.section}

The second technical difficulty posed by \MCMC methods is that the
samples will typically be autocorrelated within a chain.  This
increases the uncertainty of the estimation of posterior quantities of
interest, such as means, variances or quantiles.

A nice introductory reference for analyzing MCMC results in general
and effective sample size in particular is \citep{Geyer:2011}.  The
particular calculations used by Stan follow those for split-$\hat{R}$,
which involve both cross-chain (mean) and within-chain calculations
(autocorrelation); they were introduced in this manual and explained
in more detail in \citep{GelmanEtAl:2013}.

\subsection{Definition of Effective Sample Size}

The amount by which autocorrelation within the chains increases
uncertainty in estimates can be measured by effective sample size
({\sc ess}).  Given independent samples, the central limit theorem
bounds uncertainty in estimates based on the number of samples $N$.
Given dependent samples, the number of independent samples is replaced
with the effective sample size $N_{\mbox{\scriptsize eff}}$, which is
the number of independent samples with the same estimation power as
the $N$ autocorrelated samples.  For example, estimation error is
proportional to $1/\sqrt{N_{\mbox{\scriptsize eff}}}$ rather than
$1/\sqrt{N}$.

The effective sample size of a sequence is defined in terms of the
autocorrelations within the sequence at different lags.  The
autocorrelation $\rho_t$ at lag $t \geq 0$ for a chain with joint
probability function $p(\theta)$ with mean $\mu$ and variance
$\sigma^2$ is defined to be
\[
\rho_t
=
\frac{1}{\sigma^2} \, \int_{\Theta} (\theta^{(n)} - \mu)
(\theta^{(n+t)} - \mu) \, p(\theta) \, d\theta.
\]
This is just the correlation between the two chains offset by $t$
positions.  Because we know $\theta^{(n)}$ and $\theta^{(n+t)}$ have
the same marginal distribution in an \MCMC setting, multiplying the
two difference terms and reducing yields
\[
\rho_t
=
\frac{1}{\sigma^2} \, \int_{\Theta} \theta^{(n)} \, \theta^{(n+t)} \, p(\theta) \, d\theta.
\]

The effective sample size of $N$ samples generated by a process with
autocorrelations $\rho_t$ is defined by
\[
N_{\mbox{\scriptsize eff}}
\ = \
\frac{N}{\sum_{t = -\infty}^{\infty} \rho_t}
\ = \
\frac{N}{1 + 2 \sum_{t = 1}^{\infty} \rho_t}.
\]

\subsection{Estimation of Effective Sample Size}

In practice, the probability function in question cannot be tractably
integrated and thus the autocorrelation cannot be calculated, nor the
effective sample size.  Instead, these quantities must be estimated
from the samples themselves.  The rest of this section describes a
variogram-based estimator for autocorrelations, and hence effective sample
size, based on multiple chains. For simplicity, each chain
$\theta_m$ will be assumed to be of length $N$.

One way to estimate the effective sample size is based on the
variograms $V_t$ at lag $t \in \setlist{0,1\ldots}$.  The variograms are
defined as follows for (univariate) samples $\theta_m^{(n)}$, where $m \in
\setlist{1,\ldots,M}$ is the chain, and $N_m$ is the number of samples
in chain $m$.
\[
V_t =
\frac{1}{M}
\,
\sum_{m=1}^M
\
\left(
\frac{1}{N_m - t}
\sum_{n=t+1}^{N_m}
\left(
\theta_m^{(n)} - \theta_m^{(n-t)}
\right)^2
\right).
\]
%
The variogram along with the multi-chain variance estimate
$\widehat{\mbox{var}}^{+}$ introduced in the previous section can be
used to estimate the autocorrelation at lag $t$ as
\[
\hat{\rho}_t
= 1 - \frac{\displaystyle V_t}{
            \displaystyle 2 \, \widehat{\mbox{var}}^{+}}.
\]
If the chains have not converged, the variance estimator
$\widehat{\mbox{var}}^{+}$ will overestimate variance,
leading to an overestimate of autocorrelation and an underestimate
effective sample size.

Because of the noise in the correlation estimates $\hat{\rho}_t$ as $t$
increases, typically only the initial estimates of $\hat{\rho}_t$
where $\hat{\rho}_t > 0$ will be used.  Setting $T'$ to be the
first lag such that $\rho_{T' + 1} < 0$,
%
\[
T' = \arg\min_t \ \hat{\rho}_{t+1} < 0,
\]
the effective sample size estimator is defined as
\[
\hat{N}_{\mbox{\scriptsize eff}}
=
\frac{1}{2}
\,
\frac{MN}
     {1 + \sum_{t=1}^{T'} \hat{\rho}_t}.
\]
%
Exact autocorrelations can happen only on odd lags \citep{Geyer:2011}.
By summing over pairs, the paired autocorrelation is guaranteed to be
positive modulo estimator noise.  This is the motivation behind the
many termination criterion of \cite{Geyer:2011}. Stan does not (yet)
do the paired expectations because NUTS almost by construction avoids
the negative autocorrelation regime.  Thus terminating at the first
negative autocorrelation is a reasonable approximation for stopping
when the noise in the autocorrelation estimator dominates.

Stan carries out the autocorrelation computations for all lags
simultaneously using Eigen's fast Fourier transform (FFT) package with
appropriate padding; see \citep{Geyer:2011} for more detail on using
FFT for autocorrelation calculations.

\subsection{Thinning Samples}

In the typical situation, the autocorrelation, $\rho_t$, decreases as
the lag, $t$, increases.  When this happens, thinning the samples will
reduce the autocorrelation.  For instance, consider generating one
thousand samples in one of the following two ways.
%
\begin{enumerate}
\item Generate 1000 samples after convergence and save all of
  them.
\item Generate 10,000 samples after convergence and save every tenth
  sample.
\end{enumerate}
%
Even though both produce one thousand samples, the second approach
with thinning will produce more effective samples.  That's because the
autocorrelation $\rho_t$ for the thinned sequence is equivalent to
$\rho_{10t}$ in the unthinned samples, so the sum of the autocorrelations
will be lower and thus the effective sample size higher.

On the other hand, if memory and data storage are no object, saving all
ten thousand samples will have a higher effective sample size than
thinning to one thousand samples.

\chapter{Penalized Maximum Likelihood Point Estimation}\label{mle.chapter}

\noindent
This chapter defines the workhorses of non-Bayesian estimation,
maximum likelihood and penalized maximum likelihood, and relates them
to Bayesian point estimation based on posterior means, medians, and
modes.  Such estimates are called ``point estimates'' because they
are composed of a single value for the model parameters $\theta$
rather than a posterior distribution.

Stan's optimizer can be used to implement (penalized) maximum
likelihood estimation for any likelihood function and penalty function
that can be coded in Stan's modeling language.  Stan's optimizer can
also be used for point estimation in Bayesian settings based on
posterior modes.  Stan's Markov chain Monte Carlo samplers can be used
to implement point inference in Bayesian models based on posterior
means or medians.

\section{Maximum Likelihood Estimation}\label{mle.section}

Given a likelihood function $p(y|\theta)$ and a fixed data vector $y$,
the maximum likelihood estimate (MLE) is the parameter vector $\hat{\theta}$
that maximizes the likelihood, i.e.,
\[
\hat{\theta} = \mbox{argmax}_{\theta} \ p(y|\theta).
\]
It is usually more convenient to work on the log scale.
An equivalent%
%
\footnote{The equivalence follows from the fact that densities are
  positive and the log function is strictly monotonic, i.e.,
  $p(y|\theta) \geq 0$ and for all $a, b > 0$, $\log a > \log b$ if and
  only if $a > b$.}
%
formulation of the MLE is
%
\[
\hat{\theta} = \mbox{argmax}_{\theta} \ \log p(y|\theta).
\]

\subsection{Existence of Maximum Likelihood Estimates}

Because not all functions have unique maximum values, maximum
likelihood estimates are not guaranteed to exist.  As discussed in
\refchapter{problematic-posteriors}, this situation can arise when
%
\begin{itemize}
\item there is more than one point that maximizes the likelihood function,
\item the likelihood function is unbounded, or
\item the likelihood function is bounded by an asymptote that is never
  reached for legal parameter values.
\end{itemize}
%
These problems persist with the penalized maximum likelihood estimates
discussed in the next section, and Bayesian posterior modes as
discussed in the following section.

\subsection{Example: Linear Regression}

Consider an ordinary linear regression problem with an $N$-dimensional
vector of observations $y$, an $(N \times K)$-dimensional data matrix
$x$ of predictors, a $K$-dimensional parameter vector $\beta$ of
regression coefficients, and a real-valued noise scale $\sigma > 0$,
with log likelihood function
\[
\log p(y|\beta,x) = \sum_{n=1}^N \log \distro{Normal}(y_n|x_n \beta,
\sigma).
\]
%
The maximum likelihood estimate for $\theta = (\beta,\sigma)$ is just
\[
(\hat{\beta},\hat{\sigma})
\ = \
\mbox{argmax}_{\beta,\sigma}
\log p(y|\beta,\sigma,x) = \sum_{n=1}^N \log \distro{Normal}(y_n|x_n \beta, \sigma).
\]

\subsubsection{Squared Error}

A little algebra on the log likelihood function shows that the
marginal maximum likelihood estimate $\hat{\theta} =
(\hat{\beta},\hat{\sigma})$ can be equivalently formulated for
$\hat{\beta}$ in terms of least squares.  That is, $\hat{\beta}$ is
the value for the coefficient vector that minimizes the sum of squared
prediction errors,
%
\[
\hat{\beta}
\ = \
\mbox{argmin}_{\beta} \sum_{n=1}^N (y_n - x_n \beta)^2
\ = \
\mbox{argmin}_{\beta} (y - x \beta)^{\top} (y - x\beta).
\]
%
The residual error for data item $n$ is the difference between the
actual value and predicted value, $y_n - x_n \hat{\beta}$.  The
maximum likelihood estimate for the noise scale, $\hat{\sigma}$ is
just the square root of the average squared residual,
\[
\hat{\sigma}^2
\ = \
\frac{1}{N} \sum_{n=1}^N \left( y_n - x_n \hat{\beta} \right)^2
\ = \
\frac{1}{N} (y - x \hat{\beta})^{\top} (y - x\hat{\beta}).
\]

\subsubsection{Minimizing Squared Error in Stan}

The squared error approach to linear regression can be directly coded
in Stan with the following model.
%
\begin{stancode}
data {
  int<lower=0> N;
  int<lower=1> K;
  vector[N] y;
  matrix[N,K] x;
}
parameters {
  vector[K] beta;
}
transformed parameters {
  real<lower=0> squared_error;
  squared_error = dot_self(y - x * beta);
}
model {
  target += -squared_error;
}
generated quantities {
  real<lower=0> sigma_squared;
  sigma_squared = squared_error / N;
}
\end{stancode}
%
Running Stan's optimizer on this model produces the MLE for the linear
regression by directly minimizing the sum of squared errors and using
that to define the noise scale as a generated quantity.

By replacing \code{N} with \code{N-1} in the denominator of the
definition of \code{sigma\_squared}, the more commonly supplied
unbiased estimate of $\sigma^2$ can be calculated; see
\refsection{estimation-bias} for a definition of estimation bias and a
discussion of estimating variance.

\section{Penalized Maximum Likelihood Estimation}

There is nothing special about a likelihood function as far as the
ability to perform optimization is concerned.  It is common among
non-Bayesian statisticians to add so-called ``penalty'' functions
to log likelihoods and optimize the new function.  The penalized
maximum likelihood estimator for a log likelihood function
$\log p(y|\theta)$ and penalty function $r(\theta)$ is defined to be
\[
\hat{\theta} = \mbox{argmax}_{\theta} \log p(y|\theta) - r(\theta).
\]
The penalty function $r(\theta)$ is negated in the maximization so
that the estimate $\hat{\theta}$ balances maximizing the log
likelihood and minimizing the penalty.  Penalization is sometimes
called ``regularization.''

\subsection{Examples}\label{penalized-mle-examples}

\subsubsection{Ridge Regression}

Ridge regression \citep{HoerlKennard:1970} is based on penalizing the
Euclidean length of the coefficient vector $\beta$. The ridge penalty
function is
%
\[
r(\beta)
\ = \
\lambda \, \sum_{k=1}^K \beta_k^2
\ = \
\lambda \, \beta^{\top} \beta,
\]
%
where $\lambda$ is a constant tuning parameter that determines the
magnitude of the penalty.

Therefore, the penalized maximum likelihood estimate for ridge
regression is just
%
\[
(\hat{\beta},\hat{\sigma})
\ = \
\mbox{argmax}_{\beta,\sigma} \,
 \sum_{n=1}^N \log \distro{Normal}(y_n|x_n \beta, \sigma) - \lambda
 \sum_{k=1}^K \beta_k^2
\]
%
The ridge penalty is sometimes called L2 regularization or shrinkage,
because of its relation to the L2 norm.

Like the basic MLE for linear regression, the ridge regression
estimate for the coefficients $\beta$ can also be formulated in terms
of least squares,
%
\[
\hat{\beta}
\ = \
\mbox{argmin}_{\beta} \, \sum_{n=1}^N (y_n - x_n \beta)^2 + \sum_{k=1}^K \beta_k^2
\ = \
\mbox{argmin}_{\beta} \, (y - x\beta)^{\top} (y - x\beta) +
\lambda \beta^{\top} \beta.
\]

The effect of adding the ridge penalty function is that the ridge
regression estimate for $\beta$ is a vector of shorter length, or in
other words, $\hat{\beta}$ is shrunk.  The ridge estimate does not
necessarily have a smaller absolute $\beta_k$ for each $k$, nor does
the coefficient vector necessarily point in the same direction as the
maximum likelihood estimate.

In Stan, adding the ridge penalty involves adding its magnitude as a
data variable and the penalty itself to the model block,
%
\begin{stancode}
data {
  // ...
  real<lower=0> lambda;
}
// ...
model {
  // ...
  target += - lambda * dot_self(beta);
}
\end{stancode}
%
The noise term calculation remains the same.

\subsubsection{The Lasso}

The lasso \citep{Tibshirani:1996} is an alternative to ridge
regression that applies a penalty based on the sum of the absolute
coefficients, rather than the sum of their squares,
\[
r(\beta) = \lambda \sum_{k=1}^K | \beta_k |.
\]
The lasso is also called L1 shrinkage due to its relation to the L1
norm, which is also known as taxicab distance or Manhattan distance.

Because the derivative of the penalty does not depend on the value of
the $\beta_k$,
\[
\frac{d}{d\beta_k} \lambda \sum_{k=1}^K | \beta_k | =
\mbox{signum}(\beta_k),
\]
it has the effect of shrinking parameters all the way to 0 in maximum
likelihood estimates.  Thus it can be used for variable selection as
well as just shrinkage.%
%
\footnote{In practice, Stan's gradient-based optimizers are not
  guaranteed to produce exact zero values; see
  \cite{LangfordEtAl:2009} for a discussion of getting exactly zero
  values with gradient descent.}
%
The lasso can be implemented in Stan just as easily as ridge
regression, with the magnitude declared as data and the penalty added
to the model block,
%
\begin{stancode}
data {
  // ...
  real<lower=0> lambda;
}
// ...
model {
  // ...
  for (k in 1:K)
    target += - lambda * fabs(beta[k]);
}
\end{stancode}

\subsubsection{The Elastic Net}

The naive elastic net \citep{ZouHastie:2005} involves a weighted
average of ridge and lasso penalties, with a penalty function
\[
r(\beta)
= \lambda_1 \sum_{k=1}^K |\beta_k|
+ \lambda_2 \sum_{k=1}^K \beta_k^2.
\]
The naive elastic net combines properties of both ridge regression and
the lasso, providing both identification and variable selection.

The naive elastic net can be implemented directly in Stan by combining
implementations of ridge regression and the lasso, as
%
\begin{stancode}
data {
  real<lower=0> lambda1;
  real<lower=0> lambda2;
  // ...
}
// ...
model {
  // ...
  for (k in 1:K)
    target += -lambda1 * fabs(beta[k]);
  target += -lambda2 * dot_self(beta);
}
\end{stancode}
%
Note that the signs are negative in the program because $r(\beta)$ is
a penalty function.

The elastic net \citep{ZouHastie:2005} involves adjusting the final estimate for
$\beta$ based on the fit $\hat{\beta}$ produced by the naive elastic
net.  The elastic net estimate is
\[
\hat{\beta} = (1 + \lambda_2) \beta^*
\]
where $\beta^{*}$ is the naive elastic net estimate.

To implement the elastic net in Stan, the data, parameter, and model
blocks are the same as for the naive elastic net.  In addition, the
elastic net estimate is calculated in the generated quantities block.
%
\begin{stancode}
generated quantities {
  vector[K] beta_elastic_net;
  // ...
  beta_elastic_net = (1 + lambda2) * beta;
}
\end{stancode}
%
The error scale also needs to be calculated in the generated
quantities block based on the elastic net coefficients
\code{beta\_elastic\_net}.

\subsubsection{Other Penalized Regressions}

It is also common to use penalty functions that bias the coefficient
estimates toward values other than 0, as in the estimators of
\cite{JamesStein:1961}.  Penalty functions can also be used to bias
estimates toward population means; see
\citep{EfronMorris:1975,Efron:2012}.  This latter approach is similar
to the hierarchical models commonly employed in Bayesian statistics.

\section{Estimation Error, Bias, and Variance}\label{estimation-bias.section}

An estimate $\hat{\theta}$ depends on the particular data $y$ and
either the log likelihood function, $\log p(y|\theta)$, penalized log
likelihood function $\log p(y|\theta) - r(\theta)$, or log probability
function $\log p(y,\theta) = \log p(y,\theta) + \log p(\theta)$.  In
this section, the notation $\hat{\theta}$ is overloaded to indicate
the estimator, which is an implicit function of the data and
(penalized) likelihood or probability function.

\subsection{Estimation Error}

For a particular observed data set $y$ generated according to true
parameters $\theta$, the estimation error is the difference between
the estimated value and true value of the parameter,
\[
\mbox{err}(\hat{\theta}) = \hat{\theta} - \theta.
\]
%

\subsection{Estimation Bias}

For a particular true parameter value $\theta$ and a likelihood
function $p(y|\theta)$, the expected estimation error averaged over
possible data sets $y$ according to their density under the likelihood
is
%
\[
\mathbb{E}_{p(y|\theta)}[\hat{\theta}]
\ = \
\int \left( \mbox{argmax}_{\theta'} p(y|\theta') \right) p(y|\theta) dy.
\]

An estimator's bias is the expected estimation error,
%
\[
\mathbb{E}_{p(y|\theta)}[\hat{\theta} - \theta]
\ = \
\mathbb{E}_{p(y|\theta)}[\hat{\theta}] - \theta
\]
%
The bias is a multivariate quantity with the same dimensions as
$\theta$.  An estimator is unbiased if its expected estimation error
is zero and biased otherwise.

\subsubsection{Example: Estimating a Normal Distribution}

Suppose a data set of observations $y_n$ for $n \in 1{:}N$ drawn from
a normal distribution.  This presupposes a model $y_n \sim
\distro{Normal}(\mu,\sigma)$, where both $\mu$ and $\sigma > 0$ are
parameters.  The log likelihood is just
\[
\log p(y|\mu,\sigma) = \sum_{n=1}^N \log
\distro{Normal}(y_n|\mu,\sigma).
\]
The maximum likelihood estimator for $\mu$ is just the sample mean,
i.e., the average of the samples,
\[
\hat{\mu} = \frac{1}{N} \sum_{n=1}^N y_n.
\]
The maximum likelihood estimate for the mean is unbiased.

The maximum likelihood estimator for the variance $\sigma^2$ is the
average of the squared difference from the mean,
\[
\hat{\sigma}^2 = \frac{1}{N} \sum_{n=1}^N (y_n - \hat{\mu})^2.
\]
The maximum likelihood for the variance is biased on the low side,
i.e.,
%
\[
\mathbb{E}_{p(y|\mu,\sigma)}[\hat{\sigma}^2] < \sigma.
\]
%
The reason for this bias is that the maximum likelihood estimate is
based on the difference from the estimated mean $\hat{\mu}$.  Plugging
in the actual mean can lead to larger sum of squared differences;  if
$\mu \neq \hat{\mu}$, then
\[
\frac{1}{N} \sum_{n=1}^N (y_n - \mu)^2
>
\frac{1}{N} \sum_{n=1}^N (y_n - \hat{\mu})^2.
\]

An alternative estimate for the variance is the sample variance, which
is defined by
\[
\hat{\mu} = \frac{1}{N-1} \sum_{n=1}^N (y_n - \hat{\mu})^2.
\]
This value is larger than the maximum likelihood estimate by a factor
of $N/(N-1)$.

\subsection{Estimation Variance}

The variance of component $k$ of an estimator $\hat{\theta}$ is
computed like any other variance, as the expected squared difference
from its expectation,
%
\[
\mbox{var}_{p(y|\theta})[\hat{\theta}_k]
\ = \
\mathbb{E}_{p(y|\theta})[\, (\hat{\theta}_k -
\mathbb{E}_{p(y|\theta)}[\hat{\theta}_k])^2 \,].
\]
%
The full $K \times K$ covariance matrix for the estimator is thus
defined, as usual, by
%
\[
\mbox{covar}_{p(y|\theta)}[\hat{\theta}]
\ = \
\mathbb{E}_{p(y|\theta})[\, (\hat{\theta} - \mathbb{E}[\hat{\theta}]) \,
                         (\hat{\theta} -
                         \mathbb{E}[\hat{\theta}])^{\top} \, ].
\]

Continuing the example of estimating the mean and variance of a normal
distribution based on sample data, the maximum likelihood estimator
(i.e., the sample mean) is the unbiased estimator for the mean $\mu$
with the lowest variance; the Gauss-Markov theorem establishes this
result in some generality for least-squares estimation, or
equivalently, maximum likelihood estimation under an assumption of
normal noise; see \citep[Section~3.2.2]{HastieTibshiraniFriedman:2009}.

\chapter{Bayesian Point Estimation}

There are three common approaches to Bayesian point estimation based
on the posterior $p(\theta|y)$ of parameters $\theta$ given observed
data $y$: the mode (maximum), the mean, and the median.

\section{Posterior Mode Estimation}

This section covers estimates based on the parameters $\theta$ that
maximize the posterior density, and the next sections continue with
discussions of the mean and median.

An estimate based on a model's posterior mode can be defined by
%
\[
\hat{\theta} = \mbox{argmax}_{\theta} \, p(\theta|y).
\]
%
When it exists, $\hat{\theta}$ maximizes the posterior density of the
parameters given the data.  The posterior mode is sometimes called the
``maximum a posteriori'' (MAP) estimate.

As discussed in \refchapter{problematic-posteriors} and
\refsection{mle}, a unique posterior mode might not
exist---there may be no value that maximizes the posterior mode or
there may be more than one.  In these cases, the posterior mode
estimate is undefined.  Stan's optimizer, like most optimizers, will
have problems in these situations.  It may also return a locally
maximal value that is not the global maximum.

In cases where there is a posterior mode, it will correspond to a
penalized maximum likelihood estimate with a penalty function equal to
the negation of the log prior.  This is because Bayes's rule,
\[
p(\theta|y) = \frac{p(y|\theta) \, p(\theta)}{p(y)},
\]
ensures that
%
\begin{eqnarray*}
\mbox{argmax}_{\theta} \ p(\theta|y)
& = &
\mbox{argmax}_{\theta} \ \frac{p(y|\theta) \, p(\theta)}{p(y)}
\\[6pt]
& = &
\mbox{argmax}_{\theta} \ p(y|\theta) \, p(\theta),
\end{eqnarray*}
%
and the positiveness of densities and the strict monotonicity of log
ensure that
\[
\mbox{argmax}_{\theta} \ p(y|\theta) \, p(\theta)
\ = \
\mbox{argmax}_{\theta} \ \log p(y|\theta) + \log p(\theta).
\]
%

In the case where the prior (proper or improper) is uniform, the
posterior mode is equivalent to the maximum likelihood estimate.

For most commonly used penalty functions, there are probabilistic
equivalents.  For example, the ridge penalty function corresponds to a
normal prior on coefficients and the lasso to a Laplace prior.  The
reverse is always true---a negative prior can always be treated as a
penalty function.

\section{Posterior Mean Estimation}

A standard Bayesian approach to point estimation is to use the
posterior mean (assuming it exists), defined by
%
\[
\hat{\theta} = \int \theta \, p(\theta|y) \, d\theta.
\]
%
The posterior mean is often called {\it the}\ Bayesian estimator,
because it's the estimator that minimizes the expected square error of
the estimate.

An estimate of the posterior mean for each parameter is returned by
Stan's interfaces;  see the RStan, CmdStan, and PyStan user's guides
for details on the interfaces and data formats.

Posterior means exist in many situations where posterior
modes do not exist.  For example, in the $\distro{Beta}(0.1, 0.1)$
case, there is no posterior mode, but posterior mean is well defined
with value 0.5.

A situation where posterior means fail to exist but posterior modes do
exist is with a posterior with a Cauchy distribution
$\distro{Cauchy}(\mu,\tau)$.  The posterior mode is $\mu$, but the
integral expressing the posterior mean diverges.  Such diffuse priors
rarely arise in practical modeling applications; even with a Cauchy
Cauchy prior for some parameters, data will provide enough constraints
that the posterior is better behaved and means exist.

Sometimes when posterior means exist, they are not meaningful, as in
the case of a multimodal posterior arising from a mixture model or in
the case of a uniform distribution on a closed interval.

\section{Posterior Median Estimation}

The posterior median (i.e., 50th percentile or 0.5 quantile) is
another popular point estimate reported for Bayesian models.  The
posterior median minimizes the expected absolute error of estimates.
These estimates are returned in the various Stan interfaces;  see the
RStan, PyStan and CmdStan user's guides for more information on
format.

Although posterior medians may fail to be meaningful, they often exist
even where posterior means do not, as in the Cauchy distribution.

\chapter{Variational Inference}\label{vi-advanced.chapter}

\noindent
Stan implements an automatic variational inference algorithm that leverages
the transformations from \refchapter{variable-transforms}.

Classical variational inference algorithms are difficult to
derive. We must first define the family of approximating
densities, and then calculate model-specific quantities relative
to that family to solve the variational optimization problem.  Both
steps require expert knowledge.  The resulting algorithm is tied to
both the model and the chosen approximation.

We begin by briefly describing the classical variational inference framework.
For a thorough exposition, please refer to
\citet{Jordan:1999,Wainwright-Jordan:2008}; for a textbook presentation, please
see \citet{Bishop:2006}. We follow with a high-level description of Automatic
Differentiation Variational Inference (ADVI). For more details, see
\citep{Kucukelbir:2015}.

\section{Classical Variational Inference}

Variational inference approximates the
posterior $p(\theta \, | \, y)$ with a simple, parameterized distribution
$q(\theta \, | \, \phi)$. It matches the approximation to the
true posterior by minimizing the Kullback-Leibler (KL) divergence,
%
\[
  \phi^* = \argmin_\phi
  \KL{q(\theta \, | \, \phi) }{ p(\theta \mid y)}.
\]
%
Typically the KL divergence lacks an analytic, closed-form solution.
Instead we maximize a proxy to the KL divergence, the evidence lower bound
(ELBO)
%
\[
  \mathcal{L} (\phi)
  =
  \E_{q (\theta)} \big[ \log p (y,\theta) \big]
  -
  \E_{q (\theta)} \big[ \log q (\theta\, | \,\phi) \big].
\]
%
The first term is an expectation of the log
joint density under the approximation, and the second is the entropy of the
variational density. Maximizing the ELBO minimizes the KL
divergence \citep{Jordan:1999,Bishop:2006}.

\section{Automatic Variational Inference}

ADVI maximizes the ELBO in the real-coordinate space. Stan transforms the
parameters from (potentially) constrained domains to
the real-coordinate space. We denote the combined transformation as
$T:\theta \to \zeta$, with the $\zeta$ variables living in $\mathbb{R}^K$.
The variational objective (ELBO) becomes
%
\[
  \mathcal{L}(\phi)
  =
  \E_{q(\zeta\,|\,\phi)}
  \bigg[
  \log p (y, T^{-1}(\zeta))
  +
  \log \big| \det J_{T^{-1}}(\zeta) \big|
  \bigg]
  -
  \E_{q (\zeta\, | \,\phi)} \big[ \log q (\zeta\, | \,\phi) \big].
\]
%
Since the $\zeta$ variables live in the real-coordinate space, we can choose a
fixed family for the variational distribution. We choose a fully-factorized
Gaussian,
%
\[
  q(\zeta \, | \, \phi)
  =
  \distro{Normal}\left(\zeta \, | \, \mu, \sigma\right)
  =
  \prod_{k=1}^K
  \distro{Normal}
  \left(\zeta_k \, | \, \mu_k, \sigma_k\right),
\]
%
where the vector
$\phi = (\mu_{1},\cdots,\mu_{K}, \sigma_ {1},\cdots,\sigma_{K})$
concatenates the mean and standard deviation of each Gaussian factor.
This reflects the ``mean-field'' assumption in classical variational
inference algorithms; we will refer to this particular decomposition
as the \texttt{meanfield} option.

The transformation $T$ maps the support of the parameters to the real
coordinate space. Thus, its inverse $T^{-1}$ maps back to the support of the
latent variables. This implicitly defines the variational approximation in the
original latent variable space as
%
\[
\distro{Normal} \left(T(\theta) \, | \, \mu, \sigma\right)
\big| \det J_{T}(\theta) \big|.
\]
This is, in general, not a Gaussian distribution.
This choice may call to mind the Laplace approximation
technique, where a second-order Taylor expansion around the
maximum-a-posteriori estimate gives a Gaussian approximation to the
posterior. However, they are not the same \citep{Kucukelbir:2015}.

The variational objective (ELBO) that we maximize is,
\[
  \mathcal{L}(\phi)
  =
  \E_{q(\zeta\, | \,\phi)}
  \bigg[
  \log p (y, T^{-1}(\zeta))
  +
  \log \big| \det J_{T^{-1}}(\zeta) \big|
  \bigg]
  +
  \sum_{k=1}^K \log \sigma_k,
\]
where we plug in the analytic form for the Gaussian entropy and drop all terms
that do not depend on $\phi$. We discuss how we perform the maximization in
\refchapter{vi-algorithms}.


\documentclass[a4paper,11pt]{article}
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage{lmodern}
\usepackage[english]{babel}
\usepackage{amsmath}
\usepackage{graphicx}
\usepackage[labelformat=empty]{caption}
\usepackage[section]{placeins}
\usepackage{textcomp}
\usepackage{listings}

\lstdefinestyle{preformated}{
  language=C,
  stepnumber=1,
  numbersep=7pt,
  tabsize=2,
  showspaces=false,
  showstringspaces=false
}
\lstset{basicstyle=\small,style=preformated}

\title{Gelasia binary number representation.}
\author{Francisco Casas B.}

\begin{document}

\maketitle

\begin{abstract}
The present document shows a way to represent integer numbers of any size on a binary digits array, saving at the same time both it's value and it's size, this in order to store or send a group of well delimited numbers on a compact and scalable way, ensuring the small numbers to use less space. This can also be used when seting up standars, avoiding issues related to indexation (like, the overflow of a fixed space to countain addresses, or a very expensive memory reserve for a large number range, that's not totally used on most of the cases).
\end{abstract}

\tableofcontents

\section{Deduction}
\subsection{Previous analysis}
Normally, the amount of numbers that's possible to represent on a given space of $n$ bits is $2^n$. If we consider only positive integers (because starting from $0$ makes the deduction harder) the range of representable values is $[1,2^n]$ then, on a rough approach, if it's necessary to represent a number and delimitate it, we would take the number on it's binary form (after substracting $1$) and count the possition of the most significant 1 as the size of it, it's size has to be somehow stored, but that will be threated later, for now, the numbers will be represented this way:
\begin{center} \begin{lstlisting}
	 1 = [size:1] 0
	 2 = [size:1] 1
	 3 = [size:2] 10
	 4 = [size:2] 11
	 5 = [size:3] 100
	 6 = [size:3] 101
	 7 = [size:3] 110
	 8 = [size:3] 111
	 9 = [size:4] 1000
	10 = [size:4] 1001
	11 = [size:4] 1010
	12 = [size:4] 1011
	13 = [size:4] 1100
	14 = [size:4] 1101
	15 = [size:4] 1110
	16 = [size:4] 1111
	17 = [size:5] 10000
	      ...
\end{lstlisting} \end{center}

Then, if both the number and it's size are known, it's logical to think that if a number is big enough to requiere it's size, it's possible to dischard that the number has values that can be represented with smaller sizes:
\begin{center} \begin{lstlisting}
	 1 = [size:1] 0
	 2 = [size:1] 1
	 3 = [size:2] 00
	 4 = [size:2] 01
	 5 = [size:2] 10
	 6 = [size:2] 11
	 7 = [size:3] 000
	 8 = [size:3] 001
	 9 = [size:3] 010
	10 = [size:3] 011
	11 = [size:3] 100
	12 = [size:3] 101
	13 = [size:3] 110
	14 = [size:3] 111
	15 = [size:4] 0000
	16 = [size:4] 0001
	17 = [size:4] 0010
	      ...
\end{lstlisting} \end{center}
Now every possible value is being used, so, to get a number $N$ given it's size $S$ and this value $X$ (that will be called extra from now), the next formula will be used.
\begin{align*}
	N&= X +1 + \sum_{1\leq i < S}{2^i} \\
	&= X+1+ \frac{2^{S}-2}{2-1} \\
	&= X+2^{S}-1
\end{align*}
Now, since the size $S$ of a number, is also a number, it can be represented the same way, but, because $S>0$, it's better to store $S-1$. We can do so, until the size$-1$ of a size is $0$, The result is as follows:
\begin{center} \begin{lstlisting}
	 1 = [size:1] 0
	 2 = [size:1] 1
	 3 = [size:2] 00 = [size:1] 0 00
	 4 = [size:2] 01 = [size:1] 0 01
	 5 = [size:2] 10 = [size:1] 0 10
	 6 = [size:2] 11 = [size:1] 0 11
	 7 = [size:3] 000 = [size:1] 1 000
	 8 = [size:3] 001 = [size:1] 1 001
	 9 = [size:3] 010 = [size:1] 1 010
	10 = [size:3] 011 = [size:1] 1 011
	11 = [size:3] 100 = [size:1] 1 100
	12 = [size:3] 101 = [size:1] 1 101
	13 = [size:3] 110 = [size:1] 1 110
	14 = [size:3] 111 = [size:1] 1 111
	15 = [size:4] 0000 = [size:2] 00 0000 = [size:1] 0 00 0000
	16 = [size:4] 0001 = [size:2] 00 0001 = [size:1] 0 00 0001
	17 = [size:4] 0010 = [size:2] 00 0010 = [size:1] 0 00 0010
	      ...
\end{lstlisting} \end{center}
The only information that's then needed to recover a number besides the secuence of digits on the right is how many times the size-1 representation algorithm was applied (this will be called recursivity), since this number grows on a very slow rate it can be represented on the most primitive way that allows an infinite size, this is a series of $1$ ended by a $0$, where the amount of $1$'s is the number of times that the algorithm had to be applied:
\begin{center} \begin{lstlisting}
           1 = 0 0
           2 = 0 1
           3 = 10 0 00
           4 = 10 0 01
           5 = 10 0 10
           6 = 10 0 11
           7 = 10 1 000
           8 = 10 1 001
           9 = 10 1 010
          10 = 10 1 011
          11 = 10 1 100
          12 = 10 1 101
          13 = 10 1 110
          14 = 10 1 111
          15 = 110 0 00 0000
          16 = 110 0 00 0001
          17 = 110 0 00 0010
          18 = 110 0 00 0011
	      		...
\end{lstlisting} \end{center}
Not only the number it's stored on these bits, also it's size, that allows many numbers to be added contiguously on a bit array without need of a delimitator of additional data.
\subsection{Unsigned gelasia representation}
Now, with the last results, it's possible to substract $1$ to the values in order to have a representation for the $0$. This results on the following final representation:
\begin{center} \begin{lstlisting}
           0 = 0 0
           1 = 0 1
           2 = 10 0 00
           3 = 10 0 01
           4 = 10 0 10
           5 = 10 0 11
           6 = 10 1 000
           7 = 10 1 001
           8 = 10 1 010
           9 = 10 1 011
          10 = 10 1 100
          11 = 10 1 101
          12 = 10 1 110
          13 = 10 1 111
          14 = 110 0 00 0000
          25 = 110 0 00 1011
         125 = 110 0 10 111111
         625 = 110 1 001 001110011
        3125 = 110 1 011 10000110111
       15625 = 110 1 101 1110100001011
       78125 = 1110 0 00 0000 0011000100101111
      390625 = 1110 0 00 0010 011111010111100011
     1953125 = 1110 0 00 0100 11011100110101100111
     9765625 = 1110 0 00 0111 00101010000001011111011
    48828125 = 1110 0 00 1001 0111010010000111011011111
   244140625 = 1110 0 00 1011 110100011010100101001010011
  1220703125 = 1110 0 00 1110 001000110000100111001110010111
  6103515625 = 1110 0 01 00000 01101011110011000100000111101011
 30517578125 = 1110 0 01 00010 1100011010111111010100100110001111
152587890625 = 1110 0 01 00101 0001110000110111100100110111111000011
	             ...
\end{lstlisting} \end{center}
Where a number is determinated by the next formula, the calculation of the size $S$ will be shown later:
\begin{align*}
	N &= X+2^{S}-2
\end{align*}
\subsection{Signed gelasia representation}
It's also possible to add a sign bit when necessary, not substracting the $1$ above mentioned to the absolute value of the negative numbers, in order to get only one representation of the value $0$. This sign bit will be $1$ for negative numbers and $0$ otherwise:
\begin{center} \begin{lstlisting}
	      ...
	-15 = 1 110 0 00 0000
	-14 = 1 10 1 111
	-13 = 1 10 1 110
	-12 = 1 10 1 101
	-11 = 1 10 1 100
	-10 = 1 10 1 011
	 -9 = 1 10 1 010
	 -8 = 1 10 1 001
	 -7 = 1 10 1 000
	 -6 = 1 10 0 11
	 -5 = 1 10 0 10
	 -4 = 1 10 0 01
	 -3 = 1 10 0 00
	 -2 = 1 0 1
	 -1 = 1 0 0
	  0 = 0 0 0
	  1 = 0 0 1
	  2 = 0 10 0 00
	  3 = 0 10 0 01
	  4 = 0 10 0 10
	  5 = 0 10 0 11
	  6 = 0 10 1 000
	  7 = 0 10 1 001
	  8 = 0 10 1 010
	  9 = 0 10 1 011
	 10 = 0 10 1 100
	 11 = 0 10 1 101
	 12 = 0 10 1 110
	 13 = 0 10 1 111
	 14 = 0 110 0 00 0000
	        ...
\end{lstlisting} \end{center}
Then, if B is the sign bit of a number, it's value would be:
\begin{align*}
	N&= (X+2^{S}-2)(1-2B)-B
\end{align*}

\section{Calculation of a number's size}
For a positive number $N$ (before substracting $1$) the size $S$ is the position (from right to left, starting from $0$) of the most significant $1$ of the binary representation of the value $N+1$, this also applies to a negative number with it's absolute value. And is deducted from the equation $N = X+2^{S}-1$.
\\ Once the value $S$ is known, the following amounts of bits should be added to $S$ to complete the total size.
\begin{center}
	\begin{tabular}{ r || c| c || c}
		S & recursivity & size indication & total \\
		\hline
		1 & 1 & 0 & S+1 \\
		2-3 & 2 & 1 & S+3 \\
		4-7 & 3 & 3 & S+6 \\
		8-15 & 3 & 4 & S+7 \\
		16-31 & 4 & 7 & S+11 \\
		32-63 & 4 & 8 & S+12 \\
		64-127 & 4 & 9 & S+13 \\
		128-255 & 4 & 10 & S+14 \\
		256-511 & 4 & 12 & S+15 \\
		512-1023 & 4 & 13 & S+16 \\
		1024-2047 & 4 & 14 & S+17 \\
		2048-4095 & 4 & 15 & S+18 \\
		4096-8191 & 4 & 16 & S+19 \\
		8192-16383 & 4 & 17 & S+20 \\
		16384-32767 & 4 & 18 & S+21 \\
		32768-65535 & 4 & 19 & S+22 \\
		65536-131071 & 5 & 23 & S+28 \\
		131072-262143 & 5 & 24 & S+29
	\end{tabular}
\end{center}
Then, if an unsigned number is smaller than $2^{21}-1=2097151$ (before substracting $1$) it's granteed to use the same or less of the space that's possible to allocate on 32 bits.
	If the representation is signed, $1$ extra bit should be added.
\section{Possible variantions}
It's possible to set the minimal recursivity to $0$ instead of $1$, this will give a natural representation for the number $0$ that only requires one bit, but will imply for all the other values to have an aditional $1$ on it's recursivity header.
\begin{center} \begin{lstlisting}
           0 = 0
           1 = 10 0
           2 = 10 1
           3 = 110 0 00
           4 = 110 0 01
           5 = 110 0 10
           6 = 110 0 11
           7 = 110 1 000
           8 = 110 1 001
           9 = 110 1 010
          10 = 110 1 011
          11 = 110 1 100
          12 = 110 1 101
          13 = 110 1 110
          14 = 110 1 111
          15 = 1110 0 00 0000
          16 = 1110 0 00 0001
          17 = 1110 0 00 0010
          18 = 1110 0 00 0011
	      		...
\end{lstlisting} \end{center}
Also it's possible to add a base value to the size of all the numbers, for example $3$:
\begin{center} \begin{lstlisting}
           0 = 0 000
           1 = 0 001
           2 = 0 010
           3 = 0 011
           4 = 0 100
           5 = 0 101
           6 = 0 110
           7 = 0 111
           8 = 10 0 00 0000
           9 = 10 0 00 0001
          24 = 10 0 01 00000
           	   ...
\end{lstlisting} \end{center}
This modifications, among others, like applying an offset, can be made to ensure that the most frequent numbers use less space.
\end{document}


    \filetitle{weeksinyear}{Number of weeks in year}{dates/weeksinyear}

	\paragraph{Syntax}\label{syntax}

\begin{verbatim}
N = weeksinyear(Year)
\end{verbatim}

\paragraph{Input arguments}\label{input-arguments}

\begin{itemize}
\itemsep1pt\parskip0pt\parsep0pt
\item
  \texttt{Year} {[} numeric {]} - Year.
\end{itemize}

\paragraph{Output arguments}\label{output-arguments}

\begin{itemize}
\itemsep1pt\parskip0pt\parsep0pt
\item
  \texttt{N} {[} numeric {]} - Number of weeks in \texttt{Year}.
\end{itemize}

\paragraph{Description}\label{description}

The number of weeks in a year is either \texttt{52} or \texttt{53}, and
complies with the definition of the first week in a year in ISO 8601.
The first week of a year is the one that contains the 4th day of January
(in other words, has most of its days in that year).

\paragraph{Example}\label{example}

\begin{verbatim}
weeksinyear(2000:2010)
ans =
    52    52    52    52    53    52    52    52    52    53    52
\end{verbatim}


\documentclass[paper-main.tex]{subfiles}

\begin{document}

In this section, we explore how our apparatus can be used to teach a selection of signal processing techniques.
We use complex audio signals (such as music and speech) as natural successors to the constant and wandering tones used in Sections~\ref{sec:single_tone} and~\ref{sec:viterbi_wandering}, respectively.
As complex audio signals are not quasi-monochromatic, the Viterbi algorithm used in Section~\ref{sec:viterbi_wandering} is not directly applicable here.
Instead, we use a hierarchy of passive filters which suppress noise, yet do not assume any specific form of the signal, unlike the Fourier-based maximum likelihood filter which is tuned to the sinusoidal signals in Section~\ref{sec:single_tone} and Appendix~\ref{app:sinusoid_likelihood}.

We use the Michelson interferometer as an ``optical microphone'' to detect sound, replacing the components of a conventional microphone with a laser interferometer.
The only change to our apparatus is replacing the webcam with a photodiode to allow us to capture the higher frequencies necessary for speech and music (see Section~\ref{sec:photodiode}).
Optical microphones have precedence in the laser microphones~\cite{laser_microphone} which are (or were historically) used in the defense industry and operate on a variety of related principles.
Our objective is to play a recording of speech or music through the speaker attached to mirror M2 (see Fig.~\ref{fig:ifo_schematic_webcam}), record the resulting interference pattern, and then recover the original signal via a selection of signal processing techniques.

The apparatus serves as an independent demonstration for a broader physics and engineering audience, particularly in undergraduate laboratories.
We describe additional hardware components required for this demonstration in Section~\ref{sec:photodiode} and the initial results in Section~\ref{sec:initialResultsOpMic}.
We consider a selection of filter techniques, details of which, along with a summary of digital signal processing resources, can be found in the Supplementary Material.
In Section~\ref{sec:opticalMicResults}, we present the two best-performing techniques from the Supplementary Material.

\subsection{Hardware modifications for the optical microphone}
\label{sec:photodiode}
The human ear can hear frequencies in the range of $\sim 20\,{\rm Hz}$--$20\,{\rm kHz}$.
Speech intelligibility (the ability to understand speech) requires frequencies up to $3\,{\rm kHz}$ and music requires up to and beyond $8\,{\rm kHz}$.
Therefore, the optical microphone requires a sample rate of at least $16\,{\rm kHz}$ to capture both speech and music (adjusting for the Nyquist frequency).
This cannot be achieved with the webcam used in Sections~\ref{sec:single_tone} and ~\ref{sec:viterbi_wandering} as it has a sampling rate of $30\,{\rm Hz}$ and thus can only ``hear'' frequencies below $15\,{\rm Hz}$.
To overcome this issue, we use a photodiode~\footnote{A photodiode is an electrical component that acts as a regular diode when no light is incident on it, blocking any current flow in the reverse direction. As the intensity of incident light rises, it becomes increasingly conductive in the reverse direction.} at the output of the interferometer to achieve a sampling rate of $16\,{\rm kHz}$.

We place an OSRAM BPW21 photodiode in reverse-bias over an LM358 op-amp which together produce a voltage that depends on the incident intensity.
The photodiode records the interference pattern at roughly the same off-center position as the webcam in Sections~\ref{sec:single_tone} and~\ref{sec:viterbi_wandering}, again chosen arbitrarily.
The voltage signal from the photo-detector is captured by an MCP3008 $10$-bit analog-to-digital converter (ADC) connected to a Raspberry Pi Model 3 v1.2, which provides a convenient means to record the photodiode data.
Together, the circuit samples the signal at $\sim 16\,{\rm kHz}$.
Resources for using the Raspberry~Pi and photodiode, including a circuit diagram, are described in the Supplementary Material.

Sampling any frequency component of the analog signal above the Nyquist frequency of $8\,{\rm kHz}$ leads to aliasing (folding of frequencies greater than half the sampling rate) into the detected range. We include an anti-aliasing Sallen-Key filter with a cut-off frequency of $8\,{\rm kHz}$ before the ADC to prevent this from happening.
This component attenuates any frequencies above $8\,{\rm kHz}$ before they are digitally sampled. We also place a cloth screen over the face of the photodiode to reduce the incident intensity and avoid saturating the ADC -- an improvised, physical solution that could instead be replaced by scaling down the voltage electronically. This cloth screen was re-purposed grill cloth from a commercial speaker.

\subsection{Anti-aliased output}
\label{sec:initialResultsOpMic}

We test the optical microphone with a variety of recordings, including the speech of different people and music ranging from simple melodies and rhythms to songs.
During recordings, care is taken to minimize activity around the demonstration to reduce environmental noise coupling into the interferometer.
The timeseries data is then directly converted to a .wav file and played as an audio recording using the \texttt{scipy.io.wavfile.write} function in Python (see Appendix~\ref{app:code}).
When processing the results, we restrict our analysis to only the first $10\,{\rm s}$ of each observation (for efficiency), and only plot the first second here. \han{NOTE!}

\han{NOTES!}
The raw output of the optical microphone (with anti-aliasing) is noisy with a loud, continuous bass hum.
This can be explained by looking at the power spectral density (PSD) of the background noise (i.e., the output with the speaker switched off), shown in Fig.~\ref{fig:psd_noise}.
The spectrum is dominated by AC mains noise with power from the fundamental $50\,{\rm Hz}$ Australian mains power grid signal up to and beyond the $8$th harmonic.
The mains signal is also present, but far weaker, in the background spectrum taken with the photodiode in darkness, suggesting that ambient lighting has a large contribution.
Besides lighting, other possible contributions to the mains signal include air conditioning and the photodiode circuit itself.
The appearance of harmonics of the mains noise might be due to the non-linearity in the system discussed in Section~\ref{sec:ifo}.
The spectrum in Fig.~\ref{fig:psd_noise} also has a broad feature at around $750\,{\rm Hz}$, the origin of which is yet to be determined.
Environmental noise reduction for gravitational-wave detectors is an active area of research (see the Supplementary Material for further information and resources on this topic).

\begin{figure}
	\includegraphics[width=.5\textwidth]{figures/psd_podo_14_6.pdf}
	\caption{\label{fig:psd_noise}
Power spectral density (PSD) of background noise from the optical microphone (with the speaker off).
We see strong power from the $50\,{\rm Hz}$ mains hum and its harmonics (most likely from the photodiode circuit and the room’s lighting and cooling). Otherwise, the PSD is fairly white except for a peak at around 0.75~kHz.
}
\end{figure}

\subsection{Optical microphone results}
\label{sec:opticalMicResults}

\begin{figure*}
\begin{center}
\includegraphics[width=0.8\textwidth]{figures/combined_highlight_results_melatos_labelled.pdf}
\caption{\label{fig:notchWienerLogMMSEResults}
Timeseries (left column) and frequency spectrum (right column) results with the optical microphone.
The original input signal in the first row is a $1\,{\rm s}$ recording of an adult male voice (saying ``a cathode'').
 The input signal is shifted by 0.12 s to the right to synchronize the manual delay from starting the recording with the Raspberry Pi and starting to play the source through the speaker.
The second row shows the raw output from the optical microphone when the input from the first row is played.
The third row shows the result of applying the notch and Wiener filters combined.
The fourth row shows the result of applying the logMMSE estimator, where the rise at the start of the timeseries is an expected effect when filtering a signal of finite duration.
}
\end{center}
\end{figure*}

We explore several filters to remove the $50\,{\rm Hz}$ mains hum and harmonics and improve the speech intelligibility of the recording.
The Supplementary Material describes a range of analysis techniques that can be used as examples for the undergraduate laboratory.
All filters are tested on the same $1\,{\rm s}$ long speech recording.
The results of this section are shown in Fig.~\ref{fig:notchWienerLogMMSEResults}.
In the figure, the left and right columns show the timeseries and frequency spectrum, respectively.
The first row shows the input signal played through the speaker (see Fig.~\ref{fig:ifo_schematic_webcam}).
The second row shows the raw output from the photodiode recording.

In signal processing, the ideal filter would be one that:
(i) completely attenuates the undesired parts of the spectrum,
(ii) does not change the rest of the spectrum, and
(iii) smoothly transitions between these regions, as to not damage the time domain signal when seen under convolution.
However, these three conditions cannot all hold at once.
For example, if conditions (i) and (ii) hold, then the filter must be discontinuous at the boundary of the undesired region but this implies that the filter has ``infinite latency'' and so will affect (or damage) the time domain signal for infinite time.\cite{10.5555/151045}
Therefore, any filter must compromise between these three conditions.
For speech intelligibility, this means that either: (i) some noise remains in the filtered recording, (ii) some of the speech content is lost as certain important frequencies are attenuated, or (iii) the speech is somewhat distorted in time.
All three of these cases can, when taken to the extreme, make the speech intelligibility worse than the unfiltered recording.
Therefore, we choose filters that compromise between achieving the three conditions.

In this section, we present the results of two advanced signal processing techniques applied to the optical microphone recordings.
The techniques are only briefly described here and we refer the reader to the Supplementary Material for further details and other analysis techniques.

Firstly, we consider two signal processing techniques used in combination: the cascaded notch and the Wiener filter (see also the Supplementary Material).
A notch filter removes signals within a specific frequency range.
We want to remove the $50\,{\rm Hz}$ mains noise and harmonics, therefore we use a cascaded notch filter where each notch is centered on one of the harmonics.
The Wiener filter is an advanced statistical technique that makes use of statistical information from the speech data and noise.
It amplifies parts of the signal with a high signal-to-noise ratio while suppressing parts with a low signal-to-noise ratio.
The results of the combined cascaded notch and Wiener filter are shown in the third row in Fig.~\ref{fig:notchWienerLogMMSEResults}.
Most of the mains noise is removed; however, the recovered voice sounds muffled and is not understandable.

Secondly, we apply a speech enhancement technique.
Ref.~\cite{SubjectiveComparison} compares $13$ speech enhancement methods, finding the log minimum mean-square error (logMMSE) estimator to be the best, qualitatively, at recovering speech (see also the Supplementary Material).
We use an existing implementation of the logMMSE from Ref.~\cite{logmmse}.
The logMMSE estimator results are shown in the bottom panels in Fig.~\ref{fig:notchWienerLogMMSEResults}.
We see significant attenuation of the mains harmonics and general smoothing of the spectrum.
Most of the background noise is removed; however, the logMMSE still does not significantly enhance the speech as the voice sounds muffled and indistinct.

We find some improvement with music over speech.
Simple chords and drums can be heard after filtering, but more composite sounds and complex melodies cannot be heard clearly.
Our observations suggest that this is especially true for certain instruments; in particular flutes and violins sometimes can’t be heard at all.
This could be a perceptual effect or a frequency dependence somewhere in the optical microphone.
Speculating, perhaps the speaker-mirror coupling is stronger at low frequencies and thus instruments like electric bass and drums sound louder in the results.
To address these problems, we need to determine whether the signals that are audibly missing (the diction in the speech and complex melodies in music) are indeed being transmitted through the optical microphone at all.
To determine this requires a better understanding of the system, as discussed in Section~\ref{sec:future_work}.

\end{document}