\section{Proofs of Monitors}~\label{sec:proof}

In addition to proofs on the soundness of the Copilot compilation process, we
also support (in the {\tt Copilot.Theorem} module) some automated proving of
safety properties about Copilot specifications themselves. A proposition is a
Copilot value of type \lstinline{Prop Existential} or \lstinline{Prop Universal},
 which can be introduced using \lstinline{exists} and
\lstinline{forall}, respectively. These are functions taking as an argument a
normal Copilot stream of type \lstinline{Stream Bool}. Propositions can be added
to a specification using the \lstinline{prop} and \lstinline{theorem} functions,
where \lstinline{theorem} must also be passed a tactic for autmatically proving
the proposition. Consider this Copilot monitor specification for a version of
the fibonacci sequence:

\begin{lstlisting}[language = Copilot]
module Fib where

import Prelude ()
import Copilot.Language
import Copilot.Language.Reify
import Copilot.Theorem
import Copilot.Theorem.Prover.SMT

fib = do
  theorem "fibn_gre_n" (forall $ fibn >= n) $ kInduction def cvc4

  observer "fibn" fibn
  observer "n"    n
  where
    fibn :: Stream Word32
    fibn = [1, 1] ++ (fibn + drop 1 fibn)
    n = [0] ++ (n + 1)
\end{lstlisting}

In the specification above, in addition to observing the value of streams in
the specification, \lstinline{fibn_gre_n} names a theorem provable
automatically with induction using the Z3 SMT solver. This theorem can be
checked during reification:

\begin{code}
*Fib> reify fib
fibn_gre_n: valid (proved with k-induction (k = 3))
Finished: fibn_gre_n: proof checked successfully
\end{code}

The {\tt Copilot.Theorem} module provides two main mechanisms for interacting
with SMT solver backends: a generic backend at {\tt Copilot.Theorem.Prover.SMT}
that nominally supports a wide range of SMT solvers and {\tt
Copilot.Theorem.Prover.Z3}, a backend specialized to Z3. The example above uses
the generic {\tt SMT} backend with the CVC4 SMT solver.

See Figure~\ref{fig:solvers} for an overview of SMT solvers that have at least
been tested to work with our tool. The second column contains the value of type
{\tt Backend} from the {\tt Copilot.Theorem.Prover.SMT} module which must be
passed to the tactics in this module for executing the tactic using that SMT
solver (e.g., as is done in the second argument to {\tt kInduction} in the
example above). To use a certain SMT solver, the solver must be installed and
the executable should reside in one of the directories listed in the {\tt
\$PATH} environment variable.

\newcommand{\Yes}{\checkmark}
\newcommand{\No}{\textsf{X}}
\newcommand{\Some}{\textsf{Some}}

\begin{figure}
\begin{center}
\begin{tabular}{llllcccc}
Tool       & {\tt Backend} & Version & Interface & NL Real Arith.\ & Trig.\ funs.\ & Quantif. & Bitvec. \\
\toprule
Alt-Ergo   & {\tt altErgo} & 0.99.1  & SMTLib2   & \Yes{}          & \No{}         & \Yes{}      & \No{}      \\
CVC4       & {\tt cvc4}    & 1.4     & SMTLib2   & \Some{}         & \No{}         & \Yes{}      & \Yes{}     \\
DReal      & {\tt dReal}   & 2.15.01 & SMTLib2   & \Yes{}          & \Yes{}        & \No{}       & \No{}      \\
MathSAT    & {\tt mathSat} & 5.3.7   & SMTLib2   & \Some{}         & \No{}         & \No{}       & \Yes{}     \\
MetiTarski & {\tt metit}   & 2.5     & TPTP      & \Yes{}          & \Yes{}        & \Yes{}      & \No{}      \\
Yices      & {\tt yices}   & 2.4.0   & SMTLib2   & \Some{}         & \No{}         & \No{}       & \Yes{}     \\
Z3         & {\tt z3}      & 4.4.0   & SMTLib2   & \Yes{}          & \No{}         & \Yes{}      & \Yes{}     \\
\end{tabular}
\end{center}
\caption{An overview of some supported SMT solvers.}
\label{fig:solvers}
\end{figure}

As seen in the figure mentioned above, different SMT solvers have different
feature sets and the methods for using the various features are generally not
well standardized. As a result, we also have backend specialized to Z3, one of
the more feature-rich SMT solvers we've tested. The example above using the Z3
backend looks very similar to the version above:

\begin{lstlisting}[language = Copilot]
module Fib where

import Prelude ()
import Copilot.Language
import Copilot.Language.Reify
import Copilot.Theorem
import Copilot.Theorem.Prover.Z3

fib = do
  theorem "fibn_gre_n" (forall $ fibn >= n) $ kInduction def

  observer "fibn" fibn
  observer "n"    n
  where
    fibn :: Stream Word32
    fibn = [1, 1] ++ (fibn + drop 1 fibn)
    n = [0] ++ (n + 1)
\end{lstlisting}

Instead of importing {\tt Copilot.Theorem.Prover.SMT}, we import {\tt
Copilot.Theorem.Prover.Z3}, and we no longer need to pass the {\tt cvc4}
argument to {\tt kInduction} (because the SMT solver backend will be Z3). But
now when we go to reify {\tt fib}:

\begin{code}
*Fib> reify fib
fibn_gre_n: unknown (proof by k-induction failed)
Finished: fibn_gre_n: proof failed
Warning: failed to check some proofs.
\end{code}

This demonstrates an important limitation of the {\tt
Copilot.Theorem.Prover.SMT} backend: it encodes fixed-width integers using the
SMTLib {\tt Int} type (with some extra propositions about the bounds of {\tt
Int} variables)---it {\em does not model overflow}. The {\tt
Copilot.Theorem.Prover.Z3} backend, however, encodes Copilot's fixed-width
integer types using Z3's bitvectors. Now if we allow {\tt fibn} to overflow, the
theorem from the above example is clearly not true.

The above example contains a single {\tt theorem}. A {\tt theorem} takes three
arguments: the name of the theorem (which is only used to identify the theorem
in output), a proposition, and a proof ``tactic'' to use when trying to
automatically prove the proposition using SMT solvers. The {\tt
Copilot.Theorem.Prover.SMT} and {\tt Copilot.Theorem.Prover.Z3} modules both
export the same set of tactics\footnote{In fact, these modules share a lot a
duplicated code and desparately need to be refactored.}:
\begin{itemize}
\item {\tt onlySat}: check that the proposition is satisfiable.
\item {\tt onlyValidity}: check that the negation of the proposition is
unsatisfiable.
\item {\tt induction}: special case of $k$-induction, with $k = 0$.
\item {\tt kInduction}: version of induction where the induction hypothesis is
strengthened by including $k$ previous states.
\end{itemize}

Additionally, a few other tactics live in {\tt Copilot.Theorem.Tactics}:
\begin{itemize}
\item {\tt instantiate}: turn a proof for a universally quantified proposition
into a proof for an existentially quantified one.
\item {\tt assume}: prove a proposition true under an assumption.
\item {\tt admit}: prove anything.
\end{itemize}

Also, we support some older versions of the Kind2 model checker.


\documentclass{IEEEtran}

\usepackage{graphicx}
\usepackage{hyperref}

\title{A mutual information based medical image registration methodology based on JVHW entropy estimator}
\author{Sihan Li, Tsinghua University}

\begin{document}
  \maketitle

  \begin{abstract}
    Mutual information (MI) based medical image registration has gained much attention for its robustness. However, Jiao found that maximum likelihood estimator (MLE), although widely used to implement MI-based registration, is sub-optimal when the number of observations is comparable to the parameter dimension \cite{jiao2015minimax}. They proposed a new entropy estimator, JVHW estimator, which is claimed to outperform MLE estimator. Based on previous works, we propose a mutual information based registration methodology based on JVHW entropy estimator. The contribution of this paper can be divided into three parts: 1) A C++ version of JVHW estimators using only C++ standard library is effectively implemented, 2) a new similarity metric based on JVHW mutual information estimator is created as an elastix component, and 3) the new metric is evaluated and compared with other metrics. Dataset from Retrospective Image Registration Evaluation Project (RIRE) is used to evaluate and compare various implementations.

  \end{abstract}

  \section{Introduction}

  Image registration is a process of aligning two or more images geometrically. These images can be taken at different time, from different viewpoints or from different sensors. It is vital in image analysis when the data collected from different sources need to be combined. To achieve this, a variety of techniques have been developed and researched, aiming for different kinds of applications. \cite{brown1992survey, zitova2003image}

  Image registration methodologies consist of three major parts: \emph{Similarity measures}, which measure the degree of similarity between two images. \emph{Transforms}, which transform pixels/voxels from one image domain to another. \emph{Optimizations}, which minimize the similarity measures to achieve the registration.

  Similarity measure is a criteria to judge the degree of similarity of two images. It is know as \emph{cost function}, or \emph{registration basis} in Maintz's classification \cite{maintz1998survey}. Commonly used measures include intensity based measures, which analysis the images as a whole, and feature based measures, which detect and match features of two images.

  \subsection{Mutual information}

  Mutual information based registration methodologies use mutual information (MI) or its variant as the similarity measures. It is a kind of area-based metric. The Methodology based on a simple assumption that the gray level distributions of images to be registered are correlated. Mutual information is defined as:

  \begin{equation}
    I(A, B) = H(A) + H(B) - H(A, B)
  \end{equation}

  where $H(A)$ is the entropy of $A$. It describes the amount of uncertainty about image minus the uncertainty about $A$ when $B$ is known \cite{pluim2003mutual}. The most common entropy measure used in paper is Shannon entropy, which is defined as

  \begin{equation}
    H = -\sum_i{p_i\log{p_i}}
  \end{equation}

  where $p_i$ represents the probability of a event $e_i$. In order to register two images, the joint gray value distribution is calculated from them. Other entropy measures include Arimoto entropy \cite{li20153d}, Tsallis entropy \cite{khader2012information} etc.

  A popular variant of MI is Normalized mutual information (NMI), which is overlap invariant, which is defined as

  \begin{equation}
    NMI(A, B) = \frac{H(A) + H(B)}{H(A, B)}
  \end{equation}

  by Studholme in 1999 \cite{studholme1999overlap}. To achieve image registration, the MI/NMI between two images are maximized. Mutual information based measures are generally applicable, and does not need complex initialization \cite{pluim2003mutual}.

  % \subsection{Retrospective Image Registration Evaluation Project}

  % Retrospective Image Registration Evaluation Project (RIRE) is a project designed to compare different registration techniques. It contains a set of CT, PET and MR images. An example of the input images is shown in Figure~\ref{fig:before-registration}. Each MR-CT images pair and MR-PET images pair has been corrected using frames and markers, so the results of different registration methods can be compared with the ``gold standard'' registration results.

  % In RIRE dataset, a ``from'' image, which is a CT or PET image, is registered to a ``to'' image, which is a MR image. The relation is depicted in Figure~\ref{fig:RIRE-from-to}. An example of an image pair can be seen in Figure~\ref{fig:before-registration}. The transformations are expected to be rigid, and can be uploaded to compare with the ``gold standard'' transformations. The transformation is described by the transformed coordinates of eight corner in moving image. We are using RIRE to measure the performance of different registration methodologies.

  % \begin{figure}[htbp]
  %   \centering
  %   \includegraphics[width=0.7\columnwidth]{RIRE_from_to.jpg}
  %   \caption{Modalities of images in RIRE project. The registration is expected to be done from left (CT/PET) to right (different weighted MR). \cite{rireprotocol}}
  %   \label{fig:RIRE-from-to}
  % \end{figure}

  % \begin{figure}[htbp]
  %   \includegraphics[width=\columnwidth]{before_registration.png}
  %   \caption{A CT(green layer)-MR(red layer) image pair from RIRE project. Notice the two images are different in size.}
  %   \label{fig:before-registration}
  % \end{figure}

  \subsection{JVHW estimators}

  As we mentioned before, MI-based methodologies calculate the mutual information between two gray level distributions. However, in reality the distributions are unknown, so the exact MI can not be calculated and needs to be estimated. Most implementations simply use a straightforward maximum likelihood estimator (MLE) to estimate MI. However, Jiao pointed out that MLE and other previous prevailing approaches can be highly sub-optimal, and proposed a \emph{minimax} entropy/MI estimator, which means the estimator minimizes the maximum $L_2$ risk \cite{jiao2015minimax}. They showed that their estimator outperform MLE and many other estimators.

  \section{Implementations}

  \subsection{C++ version of JVHW estimator}

  In order to make a comparison between MLE and JVHW estimator, we are going to re-implement both of them in C++. To reuse as much as code as possible, we have to analyze both estimator and point out what they have in common. Actually, both of the estimator are linear estimators which can be expressed in the following way:

  \begin{equation}
    H = \sum_{i = 1}^n{a_jF_j},
  \end{equation}

  Where $F_j$ is called ``fingerprint'', representing the number of symbols that show up for $j$ times in the sample. The only different between two estimator is $a_j$. In MLE it follows the following form:

  \begin{equation}
    a_j = -\frac{j}{n}\log_2{\frac{j}{n}},
  \end{equation}

  and in JVHW estimator $a_j$ is calculated during run time. So instead of simply migrating MATLAB or Python code into C++ code, we divide the function into several sub-function. That is:

  \subsubsection{CalculateFingerprint}
  Collect the samples into a certain number of bins to get a histogram, then calculate the fingerprint from it.

  \subsubsection{est\_entro\_JVHW}
  Estimator entropy from an integer vector using JVHW estimator.

  \subsubsection{est\_entro\_MLE}
  The same as above, except using MLE.

  \subsubsection{NormalizeVector}
  Normalize the sample from a double vector to an integer vector.

  \subsubsection{CombineVector}
  Combine samples to form a joint-distributed sample.

  \subsubsection{est\_MI\_JVHW}
  Invoke above functions to calculate the MI.

  \subsubsection{est\_MI\_MLE}
  The same as above, except using MLE.

  To avoid file operations which will take lots of time, we embed \texttt{poly\_coeff\_r} into our source code. That is, we make it a static array, which will be loaded into the memory only once when the program starts. It will take about 314 KB memory. Given that metric function will be invoked continuously during registration, this memory consumption is really acceptable.

  \subsection{Elastix component}

  Elastix is built using CMake, and has been divided into small components. In order to add new metrics, optimizers, transformations or something else into elastix, one has to:

  \begin{enumerate}
    \item Implement an \texttt{itk} class, which does the actual works.
    \item Implement an \texttt{elastix} class, which takes care of things like parameter-setting.
    \item Put source files in the correct directory together with a \texttt{CMakeLists.txt}, which declares the component.
    \item Re-compile elastix.
  \end{enumerate}

  So in order to use JVHW entropy estimator in the metric, we have to implement two classes. Building the classes from nowhere can be a painful and error-prone, so we take most of the code from \texttt{AdvancedMeanSquares}, which is a simple metric that iterate over every voxels and compute the mean square error.

  \subsection{Optimizers and parameters}

  Elastix offers a branch of built-in optimizers, including \emph{Adaptive Stochastic Gradient Descent}, \emph{Standard Gradient Descent}, \emph{Quasi Newton LBFGS} etc. However, it should be noticed that most of the optimizers need to know the derivative of the cost function, which is especially hard to acquire when applying JVHW entropy estimator. Therefore, we will pick our optimizer from the ones that do not need to know the derivative of the cost function. Both \emph{CMA Evolution Strategy} and \emph{Simultaneous Perturbation} has been tested, and the latter one shows more robustness when applied to our problem.

  When it comes to parameters, \emph{Maximum Number Of Iterations} has been set to 500.

  The source code is available at \url{https://github.com/ThomasLee969/stanford-project}.

  \section{Results}

  To evaluate the two new metrics, we have to make sure that they are valid metric, and can be used to preform image registration. Thus, we perform registration on RIRE dataset using four metrics: \emph{Mattes MI}, \emph{NMI}, \emph{MLE} and \emph{JVHW}. The former two metrics are shipped with elastix. They use \emph{Parzen window method}, also known as \emph{kernel density estimate method}, to estimate the underlying distribution from a series of samples, as depicted in Figure~\ref{fig:parzenhistogram}

  \begin{figure}[htbp]
    \centering
    \includegraphics[width=\columnwidth]{parzenhistogram.png}
    \caption{Comparison of a histogram and a kernel density estimate.}
    \label{fig:parzenhistogram}
  \end{figure}

  Also, for \emph{Mattes MI}, \emph{NMI} method we also deploy a \emph{Adaptive Stochastic Gradient Descent} optimizer. The result can be seen in Figure~\ref{fig:CT_32bin} and Figure~\ref{fig:PET_32bin}

  \begin{figure}[htbp]
    \centering
    \includegraphics[width=\columnwidth]{CT_32bin.png}
    \caption{Quantile errors for CT to MR registration using 32 bins.}
    \label{fig:CT_32bin}
  \end{figure}

  \begin{figure}[htbp]
    \centering
    \includegraphics[width=\columnwidth]{PET_32bin.png}
    \caption{Quantile errors for PET to MR registration using 32 bins.}
    \label{fig:PET_32bin}
  \end{figure}

  We adopt the methodologies using in \cite{pluim2004f} to evaluate the accuracy of each implementation. The 0.5 and 0.9 quantile errors over all CT-MR and PET-MR pairs of 9 patients have been calculated. 0.9 quantile is used instead of the maximum value, because the maximum can be affected by a single outliers. It can be seen that we have implemented our metrics correctly. Moreover, our metrics outperform the built-in MI-based metrics.

  Having confirmed that we have implemented the metrics correctly, we are going to adjust the number of bins, which is one of the most important parameters in our metrics. The results can be seen in Figure~\ref{fig:CT-MLE-JVHW} and Figure~\ref{fig:PET-MLE-JVHW}.

  \begin{figure}[htbp]
    \centering
    \includegraphics[width=\columnwidth]{CT-MLE-JVHW.png}
    \caption{Quantile errors for CT to MR registration}
    \label{fig:CT-MLE-JVHW}
  \end{figure}

  \begin{figure}[htbp]
    \centering
    \includegraphics[width=\columnwidth]{PET-MLE-JVHW.png}
    \caption{Quantile errors for PET to MR registration}
    \label{fig:PET-MLE-JVHW}
  \end{figure}

  It can be seen that when there are few bins, MLE performs a little bit better than JVHW. But when the number of bins increase, the errors given by MLE increase faster than the errors given by JVHW. This is because when the number of samples is significantly larger than the support size, the MLE is essentially the optimal scheme. In our situation, the support size is about $10^3$, while the number of samples is about $10^6$. So it can be concluded that our JVHW-based metric outperform traditional MLE metric in general, especially when the number of bins are large.

  \section{Acknowledgment}

  The images and the standard transformation(s) were provided as part of the project, ``Retrospective Image Registration Evaluation'', National Institutes of Health, Project Number 8R01EB002124-03, Principal Investigator, J. Michael Fitzpatrick, Vanderbilt University, Nashville, TN.

  \bibliographystyle{IEEEtran}
  \bibliography{task4}

\end{document}


% !TEX root = ../main.tex
\section{Empirical Study}
\label{sec:case-study}
\label{sec:pvi-experiments}
\input{ch-pvi/tab_sbn}
We developed \acrfull{PVI}. We now empirically study \gls{PVI}, variational inference, and deterministic annealing~\citep{Katihara:2008}.\footnote{Source code for reproducibility is available at \url{https://github.com/altosaar/proximity_vi}.}

We first study sigmoid belief networks and find that \gls{PVI} improves over deterministic annealing and \gls{VI} in terms of held-out values of the \gls{ELBO} and marginal likelihood. We then study a variational autoencoder model of images.  Using an orthogonal proximity statistic, we show that \gls{PVI} improves over classical \gls{VI} by reducing overpruning. Finally, we study a deep generative model fit to a large corpus of text, where \gls{PVI} yields better predictive performance with little hyperparameter tuning.\footnote{We also compared \gls{PVI} to \citet{khan2015kullback}. Specifically, we tested \gls{PVI} on the Bayesian logistic regression model from that paper and with the same data.  Because Bayesian logistic regression has a single mode, all   methods performed equally well.  We note that we could not apply their algorithm to the sigmoid belief network because it would require approximating difficult iterated expectations.}

\paragraph{Hyperparameters.} For \gls{PVI}, we use the inverse Huber distance for $d$.\footnote{We define the inverse Huber distance   $d(x, y)$ to be $|x - y|$ if $|x - y| < 1$ and $0.5(x-y)^2 + 0.5$   otherwise. The constants ensure the function and its derivative are   continuous at $|x-y| = 1$.} The inverse Huber distance penalizes smaller values than the square difference. For \gls{PVI} \Cref{algo:global}, we set the exponential moving average decay constant for $\tmblambda$ to $\alpha=0.9999$. We set the constraint scale $k$ (or temperature parameter in deterministic annealing) to the initial absolute value of the \gls{ELBO} unless otherwise specified. We explore two annealing schedules for \gls{PVI} and deterministic annealing: a linear decay and an exponential decay. For the exponential decay, the value of the magnitude at iteration $t$ of $T$ total iterations is set to $k\cdot \gamma^{\frac{t}{T}}$ where $\gamma$ is the decay rate. We use the Adam optimizer~\citep{kingma2015adam:} unless otherwise specified.
\subsection{Sigmoid Belief Network}
The sigmoid belief network is a discrete latent variable model with layers of Bernoulli latent variables~\citep{neal1992connectionist,ranganath2015deep}. It is used to benchmark variational inference algorithms~\citep{Mnih:2016:VIM:3045390.3045621}. The approximate posterior is a collection of Bernoullis, parameterized by an inference network with weights and biases.  We fit these variational parameters with \gls{VI}, deterministic annealing~\citep{Katihara:2008}, or \gls{PVI}, and learn the model parameters (weights and biases) using variational expectation-maximization.

\input{ch-pvi/tab_sbn_3}
We learn the weights and biases of the model with gradient ascent. We use a step size of $\rho=10^{-3}$ and train for $4\times10^6$ iterations with a batch size of $20$.  For \gls{PVI} \Cref{algo:global} and deterministic annealing, we grid search over exponential decays with rates $\gamma\in \{10^{-5}, 10^{-6}, ..., 10^{-10}, 10^{-20}, 10^{-30}\}$ and report the best results for each algorithm.  (We also explored linear decays but they did not perform as well.)  To reduce the variance of the gradients, we use the leave-one-out control variate of~\citet{Mnih:2016:VIM:3045390.3045621} with $5$ samples. (This is an extension to the black box variational inference algorithm in \citet{ranganath2014black}.)

\paragraph{Results on MNIST.} We train a sigmoid belief network model on the binary MNIST dataset of handwritten digits~\citep{pmlr-v15-larochelle11a}. For evaluation, we compute the \gls{ELBO} and held-out marginal likelihood with importance sampling on the validation set of $10^4$ digits using $5000$ samples, as in \citet{rezende2014stochastic}. In Table~$1$ we show the results for a model with one layer of $200$ latent variables. \Cref{table:sbn_3_layer} displays similar results for a three-layer model with $200$~latent variables per layer. In both one and three-layer models the \gls{KL} proximity statistic performs worse than the mean/variance and entropy statistics; it requires different decay schedules. Overall, \gls{PVI} with the entropy and mean/variance proximity statistics yields improvements in the held-out marginal likelihood in comparison to deterministic annealing and \gls{VI}.

\input{ch-pvi/tab_dlgm}
\subsection{Variational Autoencoder}
\label{sec:variational_autoencoder}
To demonstrate the value of designing proximity statistics tailored to specific   models,  we study the variational autoencoder~ \citep{kingma2014autoencoding,rezende2014stochastic}. This model is difficult to optimize, and current optimization techniques yield solutions that do not use the full model capacity~\citep{Burda2016}. In \Cref{sec:proximity_examples} we designed an orthogonal proximity statistic to make backpropagation in neural networks easier. We show that this statistic enables us to find a better approximate posterior in the variational autoencoder by reducing overpruning.

We fit the variational autoencoder to binary MNIST data~\citep{pmlr-v15-larochelle11a} with variational expectation-maximization. The model has one layer of $100$ Gaussian latent variables. The inference network and generative network are chosen to have two hidden layers of size $200$ with rectified linear units. We use an orthogonal initialization for the inference network weights. The learning rate is set to $10^{-3}$ and we run \gls{VI} and \gls{PVI} for $5\times 10^4$ iterations. The orthogonal proximity statistic changes rapidly during optimization, so we use constraint magnitudes $k \in \{1, 10^{-1}, 10^{-2}, ..., 10^{-5}\}$, with no decay, and report the best result.

We compute the \gls{ELBO} and importance-sampled marginal likelihood estimates on the validation set. \Cref{table:dlgm2} shows that \gls{PVI} with the orthogonal proximity statistic on the weights of the inference network enables easier optimization and improves over \gls{VI}.

Why does \gls{PVI} improve upon \gls{VI} in the variational autoencoder? The choice of rectified linear units in the inference network allows us to  study overpruning of the latent code~\citep{mackay2001,Burda2016}. We study  the fraction of `dead units'--- the fraction of rectified linear units in each layer of the inference   neural network whose input is below zero. With \gls{PVI} \Cref{algo:global} and the orthogonal proximity constraint, the inference network has $1.6\%$ fewer dead units in the hidden layer and shows a $3.2\%$ reduction in the output layer than in the same model learned using classic variational inference.

Once the input to a rectified linear unit drops below zero, the unit stops receiving gradient updates. The output layer parametrizes the latent variable distribution, so this means \gls{PVI} reduced the pruning of the approximate posterior and led to the utilization of $3$ additional latent variables. This is the reason it outperformed a variational autoencoder fit with \gls{VI}.

\input{ch-pvi/tab_poisson}
\subsection{Deep Generative Model of Text}
\label{sec:poisson_factor_analysis}
Deep exponential family models, Bayesian analogues to neural networks, represent a flexible class of models~\citep{ranganath2015deep}. However, black box variational inference is commonly used to fit these models, which requires variance reduction~\citep{ranganath2014black}. Deep exponential family models with Poisson latent variables present a challenging approximate inference problem because they are discrete and high-variance. We demonstrate that \gls{PVI} with the mean/variance proximity constraint improves predictive performance in such an unsupervised model of text.

The generative process for a single-layer deep exponential family model of text, with Poisson latent variables and Poisson likelihood, is
\begin{align*}
\mbz &\sim \textrm{Poisson}(\mblambda) \\
\mbx &\sim \textrm{Poisson}(\mbz^\top g(W)) \,,
\end{align*}
where $W$ are real-valued model parameters and $g$ is an elementwise function that maps to the positive reals (we use the softplus function). The dimension of $\mbz$ is $K$, so the model parameters must have shape $(K, V)$ where $V$ is the cardinality of the count-valued observations $\mbx$. We use this as a model of documents, so $\mbx$ is the bag-of-words representation of word counts, $W$ represents the common factors in documents, and the per-document latent variable $\mbz$ captures factors prevalent in documents' language.

We study the performance of our method on a corpus of articles from the academic journal \emph{Science}. The corpus contains $138$k documents in the training set, $1$k documents in the test set, and $5.9$k terms. We set the latent dimension to $100$, and fit the variational Poisson parameters using black box variational inference~\citep{ranganath2014black} using minibatches of size $64$ and $32$~samples of the latent variables to estimate the gradients.

Poisson variables have high variance, so we use  the optimal control variate scaling developed in \citet{ranganath2014black} and estimate this scaling in a round-robin fashion as in \citet{Mnih:2016:VIM:3045390.3045621} for efficiency. We use the RMSProp adaptive gradient optimizer~\citep{Hinton} with a step size of $0.01$. For \gls{PVI}~\Cref{algo:global} with the mean/variance proximity statistic, we use an exponential decay for the constraint and test decay rates $\gamma$ of $10^{-5}$ and $10^{-10}$. We train for $10^6$ iterations on the \emph{Science} corpus, using variational expectation-maximization to learn the model parameters.

\input{ch-pvi/tab_poisson_topics}
For evaluation, we keep the model parameters fixed and hold out $90\%$ of the words in each document in the test set. Using the $10\%$ of observed words in each document, we learn the variational parameters using \gls{PVI} or variational inference with $300$ iterations per document. We compute perplexity on the held-out documents, which is given by
\begin{align*}
\exp\left(\frac{-\sum_{d\in{\textrm{docs}}} \sum_{w\in d} \log p(w \mid
\textrm{\# held-out in d})}{N_{\textrm{held-out words}}}\right).
\end{align*}
Conditional on the number of held-out words in a document, the distribution over held-out words is multinomial. The mean of the conditional multinomial is the normalized Poisson rate of the document matrix-multiplied with the softplus of the weights. This is the same evaluation metric as in \citet{ranganath2015deep}. The results of fitting the model to the corpus of \emph{Science} documents are reported in \Cref{table:poisson} and \Cref{table:poisson_topics}. While the topics found by models fit with both \gls{PVI} and \gls{VI} are similar, \gls{PVI} gives better predictive performance in terms of held-out perplexity.


\documentclass[14pt,aspectratio=169]{beamer}
\usepackage{polyglossia}
\setdefaultlanguage{english}
\usetheme{SaintPetersburg}

\usepackage{amsthm}
\usepackage{amssymb}
\usepackage{amsmath}
\usepackage{mathtools}
\usepackage{listings}
\usepackage{booktabs}
\usepackage{graphicx}
\usepackage{tikz}

\graphicspath{{figures/}}

\newcommand{\Fourier}[1]{\mathcal{F}\left\{#1\right\}}
\newcommand{\InverseFourier}[1]{\mathcal{F}^{-1}\left\{#1\right\}}
\newcommand{\Var}[1]{\sigma_{#1}^2}

\AtBeginSection[]{
%	\iffirstsection
%		\begin{frame}{Plan}
%			\tableofcontents
%		\end{frame}
%		\firstsectionfalse
%	\fi
%	\begin{frame}{Plan}
%		\tableofcontents[currentsection]
%	\end{frame}
	\frame{\sectionpage}
}

\title{Generating standing and propagating ocean waves with three-dimensional ARMA model}
\subtitle{Technical report}
\author{Ivan Gankevich}
\date{Aug 26, 2016}

\begin{document}

	\frame{\maketitle}

	\section{Two methods of finding wave's ACF}

	\begin{frame}
		\frametitle{Analytic method}
		\only<1>{%
			Apply Wiener---Khinchin theorem to a wave profile $\zeta$ to get ACF $K$:
			\begin{equation*}
				K(t) = \Fourier{\left| \zeta(t) \right|^2}.
				\label{eq:wiener-khinchin}
			\end{equation*}%
		}
		\only<2>{%
			\begin{example}
				Standing wave profile:
				\begin{equation*}
					\zeta(t, x, y) = A \sin (k_x x + k_y y) \sin (\sigma t).
					\label{eq:standing-wave}
				\end{equation*}
				Standing wave ACF:
				\begin{equation*}
					K(t,x,y) =
					\gamma
					\exp\left[-\alpha (|t|+|x|+|y|) \right]
					\cos \beta t
					\cos \left[ \beta x + \beta y \right].
					\label{eq:standing-wave-acf}
				\end{equation*}
			\end{example}%
		}
		\only<3>{%
			\begin{example}
				Propagating wave profile:
				\begin{equation*}
					\zeta(t, x, y) = A \cos (\sigma t + k_x x + k_y y).
					\label{eq:propagating-wave}
				\end{equation*}
				Propagating wave ACF:
				\begin{equation*}
					K(t,x,y) =
					\gamma
					\exp\left[-\alpha (|t|+|x|+|y|) \right]
					\cos\left[\beta (t+x+y) \right].
					\label{eq:propagating-wave-acf}
				\end{equation*}
			\end{example}%
		}
		\only<4>{%
			Some observations:
			\begin{itemize}
				\item Taking Fourier transform of sine/cosine wave profile requires
					  multiplying it by an decaying exponent to produce useful ACF.
				\item Fourier Transform of squared exponent (Gaussian) is another Gaussian.
			\end{itemize}
			\vfill\centering%
			\alert{Why use Fourier transform at all?}%
		}
	\end{frame}

	\begin{frame}
		\frametitle{Empirical method}
		The algorithm:
		\begin{enumerate}
			\item Multiply wave profile by an decaying exponent.
			\item Adjust sine/cosine phase to move maximum value to the origin
				  (or substitute sine with cosine to get the same effect).
		\end{enumerate}
		\vfill%
		In case of plain waves result is the same as for analitic method.
	\end{frame}

	\section{Governing equations for 3-dimensional ARMA process}

	\begin{frame}
		\frametitle{3-D ARMA process}
		Three-dimensional autoregressive moving average process is defined by
		\begin{equation*}
			\zeta_{i,j,k} =
			\sum\limits_{l=0}^{p_1}
			\sum\limits_{m=0}^{p_2}
			\sum\limits_{n=0}^{p_3}
			\Phi_{l,m,n} \zeta_{i-l,j-m,k-n}
			+
			\sum\limits_{l=0}^{q_1}
			\sum\limits_{m=0}^{q_2}
			\sum\limits_{n=0}^{q_3}
			\Theta_{l,m,n} \epsilon_{i-l,j-m,k-n}
			,
			\label{eq:arma-process}
		\end{equation*}
		\small{%
			where $\zeta$ --- wave elevation, $\Phi$ --- AR coefficients, $\Theta$ --- MA
			coefficients, \newline$\epsilon$ --- white noise with Gaussian distribution,
			$(p_1,p_2,p_3)$ --- AR process order, $(q_1,q_2,q_3)$ --- MA process order, and
			$\Phi_{0,0,0} \equiv 0$, $\Theta_{0,0,0} \equiv 0$.%
		}
	\end{frame}

	\begin{frame}
		\frametitle{Determining coefficients}
		\framesubtitle{AR process}
		\small%
		Solve linear system of equations (3-D Yule---Walker equations) for $\Phi$:
		\begin{equation*}
		    \Gamma
		    \left[
		        \begin{array}{l}
		            \Phi_{0,0,0}\\
		            \Phi_{0,0,1}\\
		            \vdotswithin{\Phi_{0,0,0}}\\
		            \Phi_{p_1,p_2,p_3}
		        \end{array}
		    \right]
		    =
		    \left[
		        \begin{array}{l}
		            K_{0,0,0}-\Var{\epsilon}\\
		            K_{0,0,1}\\
		            \vdotswithin{K_{0,0,0}}\\
		            K_{p_1,p_2,p_3}
		        \end{array}
		    \right],
		    \qquad
		    \Gamma=
		    \left[
		        \begin{array}{llll}
		            \Gamma_0 & \Gamma_1 & \cdots & \Gamma_{p_1} \\
		            \Gamma_1 & \Gamma_0 & \ddots & \vdotswithin{\Gamma_0} \\
		            \vdotswithin{\Gamma_0} & \ddots & \ddots & \Gamma_1 \\
		            \Gamma_{p_1} & \cdots & \Gamma_1 & \Gamma_0
		        \end{array}
		    \right],
		\end{equation*}
		\begin{equation*}
			\Gamma_i =
			\left[
			\begin{array}{llll}
				\Gamma^0_i & \Gamma^1_i & \cdots & \Gamma^{p_2}_i \\
				\Gamma^1_i & \Gamma^0_i & \ddots & \vdotswithin{\Gamma^0_i} \\
				\vdotswithin{\Gamma^0_i} & \ddots & \ddots & \Gamma^1_i \\
				\Gamma^{p_2}_i & \cdots & \Gamma^1_i & \Gamma^0_i
			\end{array}
			\right]
			\qquad
			\Gamma_i^j=
			\left[
			\begin{array}{llll}
				K_{i,j,0} & K_{i,j,1} & \cdots & K_{i,j,p_3} \\
				K_{i,j,1} & K_{i,j,0} & \ddots &x \vdotswithin{K_{i,j,0}} \\
				\vdotswithin{K_{i,j,0}} & \ddots & \ddots & K_{i,j,1} \\
				K_{i,j,p_3} & \cdots & K_{i,j,1} & K_{i,j,0}
			\end{array}
			\right].
		\end{equation*}
	\end{frame}

	\begin{frame}
		\frametitle{Determining coefficients}
		\framesubtitle{MA process}
		\small%
		Solve non-linear system of equations for $\Theta$:
		\begin{equation*}
			K_{i,j,k} =
			\left[
				\displaystyle
				\sum\limits_{l=i}^{q_1}
				\sum\limits_{m=j}^{q_2}
				\sum\limits_{n=k}^{q_3}
				\Theta_{l,m,n}\Theta_{l-i,m-j,n-k}
			\right]
			\Var{\epsilon}
		\end{equation*}
		via fixed-point iteration method:
		\begin{equation*}
			\theta_{i,j,k} =
				-\frac{K_{0,0,0}}{\Var{\epsilon}}
				+
				\sum\limits_{l=i}^{q_1}
				\sum\limits_{m=j}^{q_2}
				\sum\limits_{n=k}^{q_3}
				\Theta_{l,m,n} \Theta_{l-i,m-j,n-k}.
		\end{equation*}
	\end{frame}

	\begin{frame}
		\frametitle{Determining coefficients}
		\framesubtitle{ARMA process}
		To mix processes one needs to divide ACF between processes, and
		recompute one of the parts to match process properties (mean,
		variance etc.).
		\vfill%
		\begin{center}
			\alert{There is no recomputation formula for 3-D proccess.}
		\end{center}
	\end{frame}

	\begin{frame}
		\frametitle{Our approach}
		Use AR process for standing waves and MA process for
		propagating waves.
		\vfill%
		Supporting experimental results:
		\begin{itemize}
			\item It works that way in practice.
			\item It does not work the other way round
				  (processes diverge).
			\item Wavy surface integral characteristics
				  match the ones of real ocean waves.
		\end{itemize}
	\end{frame}

	\section{Evaluation and verification}

	\begin{frame}
		\frametitle{Experiment setup}
		\begin{itemize}
			\item Generate standing/propagating waves with
				  AR/MA processes respectively.
			\item Estimate distributions of integral
				  characteristics.
			\item Compare estimated distributions to the
				  known ones via QQ plots.
		\end{itemize}
		\vfill%
		\begin{center}
			\small
			\begin{tabular}{ll}
				\toprule
				Characteristic & Weibull shape ($k$) \\
				\midrule
				Wave height & 2 \\
				Wave length & 2.3 \\
				Crest length & 2.3 \\
				Wave period & 3 \\
				Wave slope & 2.5 \\
				Three-dimensionality & 2.5 \\
				\bottomrule
			\end{tabular}%
		\end{center}
	\end{frame}

%	\begin{frame}
%		\frametitle{Input ACFs (time slices)}
%		Standing wave ACF:
%		\vfill%
%		\begin{tabular}{llll}%
%			\includegraphics[scale=0.45]{standing-acf-0} &
%			\includegraphics[scale=0.45]{standing-acf-1} &
%			\includegraphics[scale=0.45]{standing-acf-3} &
%			\includegraphics[scale=0.45]{standing-acf-4} \\
%		\end{tabular}
%		\vfill%
%		Propagating wave ACF:
%		\vfill%
%		\begin{tabular}{llll}%
%			\includegraphics[scale=0.45]{propagating-acf-00} &
%			\includegraphics[scale=0.45]{propagating-acf-01} &
%			\includegraphics[scale=0.45]{propagating-acf-03} &
%			\includegraphics[scale=0.45]{propagating-acf-04} \\
%		\end{tabular}
%	\end{frame}

	\begin{frame}
		\frametitle{Verification results (QQ plots)}
		\small%
		\centering
		\begin{columns}
			\begin{column}{0.5\textwidth}
				\centering%
				Standing waves
				\begin{tabular}{ll}
					\includegraphics[scale=0.5]{standing-elevation} &
					\includegraphics[scale=0.5]{standing-wave-height-x} \\
					\addlinespace
					\includegraphics[scale=0.5]{standing-wave-length-x} &
					\includegraphics[scale=0.5]{standing-wave-period} \\
				\end{tabular}
			\end{column}
			\begin{column}{0.5\textwidth}
				\centering%
				Propagating waves
				\begin{tabular}{ll}
					\includegraphics[scale=0.5]{propagating-elevation} &
					\includegraphics[scale=0.5]{propagating-wave-height-x} \\
					\addlinespace
					\includegraphics[scale=0.5]{propagating-wave-length-x} &
					\includegraphics[scale=0.5]{propagating-wave-period} \\
				\end{tabular}
			\end{column}
		\end{columns}
	\end{frame}

\end{document}