%----------------------------------------------------------------------------
\chapter{Sequential modeling}

This chapter focuses on the theoretical background of processing sequential data. As relatively simple deep learning-based solutions are surprisingly good with sequential data, I concentrate on neural network-based approaches.

Sequential modeling is a powerful way to solve numerous tasks, like video processing, time series prediction, or natural language processing. Many tasks can be modeled sequentially. The key is to project the domain problem to the proper abstraction. The categorization is partly based on source \cite{SequentialLecture}. There are different types of sequential tasks:

\begin{itemize}
  \item Vanilla approach processes the input in a \textit{single-input-single-output} (SISO) manner: for example, one frame of a video is classified independently of others. This is the case when the model has a single input image and makes classification.
  \item In a \textit{single-input-multiple-output} (SIMO) model, there is a single input and multiple possible outputs. For example, when an image contains multiple characters, the input is a single image, and the output is the sequence of characters in it.
  \item \textit{multiple-input-single-output} (MISO) models receive multiple inputs and output a single prediction. For example, the inputs are words of a textual review, and the prediction is the writer's sentiment.
  \item \textit{multiple-input-multiple-output} (MIMO) models receive multiple inputs and predict multiple outputs. These approaches can be divided into further subcategories:
    \begin{itemize}
     \item Some models receive multiple inputs and immediately output the prediction based on that. In video classification, a model usually receives multiple frames and outputs the class for each time frame, but these are predicted simultaneously, depending on each other.
	   \item In another configuration, popular in machine translation, models receive an input text and predict the current translated text on the fly, based on the current input. Here, input and output sequences are shifted relative to each other (some input characters are needed for the first prediction). For example, Google Translate is a model of this kind.
    \end{itemize}
\end{itemize}

A summary of the different tasks is illustrated by Figure \ref{fig:task_types}. Sequential data can be processed with different network types. In the following sections, I present these variants.

\begin{figure}[htb]
 \centerline{\includegraphics[width=1.0\columnwidth]{.//Figure/Sequential/task_types.png}}
 \caption{The different types of sequential tasks.}
 \label{fig:task_types}
\end{figure}

\section{One-dimensional convolution}

One-dimensional convolution networks can solve sequential problems by applying convolution to the serial input. These approaches are popular for problems like time-series analysis or audio input processing.

When applying 1D convolution, the corresponding layer only sees the input in slices of the same width as the convolutional kernel. Pooling or strided convolution can be applied to reduce the length of the sequence, thus increasing the visible part of the input toward the deeper convolutional layers. Another approach uses \textit{dilated convolution}, where the kernel is inflated by inserting holes between its elements (Figure \ref{fig:dilated_convolution}). Dilation rate defines how much the kernel is widened. WaveNet\cite{WaveNet} is a successful dilated convolution-based MIMO model for sample-level audio synthesis. Its main data processing idea is illstrated by Figure \ref{fig:WaveNet_dilation}.

\begin{figure}[H]
 \centerline{\includegraphics[width=0.8\columnwidth]{.//Figure/Sequential/dilated_convolution.png}}
 \caption{Two-dimensional dilated convolution.}
 \label{fig:dilated_convolution}
\end{figure}

\begin{figure}[H]
 \centerline{\includegraphics[width=1.0\columnwidth]{.//Figure/Sequential/WaveNet_dilation.png}}
 \caption{Stack of one-dimensional dilated convolutional layers in WaveNet. Source: \cite{WaveNet}}
 \label{fig:WaveNet_dilation}
\end{figure}

These models use solely convolutional layers, thus being relatively simple. While training, the simple backpropagation algorithm is used, and their runtime is only constrained by the convolution operation: in the case of two dimensions, if the input size is ${n}\times{n}$ and the kernel size is ${m}\times{m}$, the runtime is $\mathcal{O}({n^2}\times{m^2})$. However, these models lack one crucial element to process long sequential inputs: memory.

\section{Recurrent neural networks}

Classical neural networks are directed acyclic graphs (DAG), which define computational nodes and their connections. \textit{Recurrent neural networks} (RNN) are dynamic models containing recurrent links (directed loops). These connections can model an inner/hidden state (memory), which makes them useful when sequential data is processed, where later values depend on earlier ones. In every time step, the same function is called, meaning that the RNN's nodes and weights are the same. Figure \ref{fig:RNN} represents the inference steps of an RNN. These models do not require prior knowledge of the input data, and they are generally robust to noise\cite{CTC}. The section below discusses how such a network is trained.

\begin{figure}[htb]
 \centerline{\includegraphics[width=1.0\columnwidth]{.//Figure/Sequential/RNN.png}}
 \caption{Unfolded interpretation of the RNN for a length-4 sequential input.}
 \label{fig:RNN}
\end{figure}

\subsection{Backpropagation through time}

In the classical backpropagation, a model input is first propagated forward the network, resulting in a prediction. When training the network, it is known what is expected from the model's output (ground truth). The difference between the prediction and the ground truth is calculated by the loss function, which results in the network's error. The error is then propagated backward the network: for each connection, a gradient is calculated based on how much it contributed to the local error of the following node. When each link has this value, the algorithm updates each connection weight based on its gradient and the learning rate.

Backpropagation through time\cite{SequentialLecture} (BPTT) works similarly. The difference is that the RNN is handled as an unfolded model. For each time step, the backpropagation algorithm is applied in a manner that the input of the \textit{Nth} step is the output of the \textit{N-1th} step (this wording is intentional, as we are moving backward in time). Gradient values are calculated for the whole network independently for each time step. When gradients are computed for a step, values are updated for each weight (Figure \ref{fig:BPTT}).

\begin{figure}[htb]
 \centerline{\includegraphics[width=1.0\columnwidth]{.//Figure/Sequential/BPTT.png}}
 \caption{Operation of the BPTT algorithm on the unfolded RNN.}
 \label{fig:BPTT}
\end{figure}

Usually, there is a truncation in the BPTT algorithm: the backward calculation only goes back to the last \textit{K} steps on the network unfolded by time. Otherwise, in the case of long series, the algorithm would take a lot of computational resources, significantly slowing down the training process.

\subsection{Limitations}

Many time steps are needed to create long-term inner memory. During BPTT, the error propagates back to these time steps on the same model. It means that we multiply the model's weight matrix multiple times, which is prone to the following problems:

\begin{itemize}
\item When the weight matrix singular value is high, it leads to gradient exploding. The weight updates are too big, so the model cannot converge.
\item When the weight matrix singular value is low, gradient vanishing happens. In this case, the gradients are too close to zero; weight updates are too small to improve the model.
\end{itemize}

For gradient exploding, possible solutions can be using saturating nonlinear activation functions or gradient clipping, where the values cannot go above a maximum. Skip connections can somehow compensate for the gradient vanishing problem, through which the error can flow freely back to the earlier time steps. However, these are not complete solutions.

\subsection{Long short-term memory}

Long short-term memory\cite{LSTM} (LSTM) is a type of RNN cell which uses a gated approach. It implements a short-term memory that can be long enough to handle longer sequences than simple RNNs can do. An LSTM cell has three inputs in each time step:

\begin{itemize}
\item \textit{C}: The cell state is the interpretable, optimized output of the cell.
\item \textit{H}: Hidden state, which is the inner state of the LSTM block.
\item \textit{X}: Input tensor, the input from the earlier network layers (for example, a feature map created by convolutional layers which we want to process sequentially, from left to right).
\end{itemize}

There are multiple trainable gates within an LSTM cell, manipulating the signal flow and thus modeling what information is needed from a specific input given a state. These gates are trainable to model memory. The gates are the following:

\begin{itemize}
\item \textit{forget}: Responsible for deleting certain things from the cell state based on the fused input of the hidden state and the input tensor. This gate was not part of the initial publication; it was added a few years later\cite{LSTM-ForgetGate}. Forget gate allows the cell to reset its state and was crucial to the success of LSTM.
\item \textit{input}: Decides what items of the fused input should be written to the cleaned cell state.
\item \textit{gate}: Applies nonlinearity to the fused input written to the cell state. This is the only gate that uses the tanh activation function instead of the sigmoid. This gate projects the values between [-1, 1] to provide the necessary abstraction switching of the signal while maximizing the output range, retaining the saturating behavior. The other gates use sigmoid activation because they mask the signal to be retained (where the minimum is 0 and the maximum is 1).
\item \textit{output}: Masks the hidden state output derived from the cell state.
\end{itemize}

In an LSTM cell, there are only saturating nonlinear activation functions, which can handle the gradient exploding problem. There is no activation function between the input and the output cell state, only one multiplication (forget gate) and one addition. This means that the input signal can propagate to the output without severe decrease, and thus the gradients can backpropagate without downscaling. This path forms a gradient highway across the unfolded model (containing the same LSTM cell of each time step) during BPTT training. This path is essential in solving the gradient vanishing problem induced by the numerous time steps. The visual representation of an LSTM cell is illustrated by Figure \ref{fig:LSTM}.

\begin{figure}[htb]
 \centerline{\includegraphics[width=1.0\columnwidth]{.//Figure/Sequential/LSTM.png}}
 \caption{Inner structure of an LSTM cell.}
 \label{fig:LSTM}
\end{figure}

LSTM is a relatively old approach from 1997, a mature standard tool for solving sequential problems. Over time, other solutions have emerged, such as the Gated Recurrent Unit\cite{GRU} (GRU) from 2014, which similarly models memory, but with fewer parameters (three gates instead of four). I decided to present LSTM because it is easier to understand thanks to its well-separated elements.

\section{Connectionist Temporal Classification}

RNNs are good at sequence learning, but they require thoroughly segmented and labeled data, which is time-consuming. Traditional loss functions like mean squared error cannot work well without properly segmented data because they assume that each item in the sequence is independent. This leads to limited model applicability.

Connectionist Temporal Classification\cite{CTC} is a loss function to handle sequential input, which is noisy and unsegmented. It assumes that the network output is a conditional probability distribution over the label sequences. This way, the loss function transforms the sequential problem into classification, where the model needs to select the most probable label for a given input sequence.

\subsection{Example}

I illustrate the function's applicability on an example with an input image in which we want to read a text. This example is partly based on \cite{CTCexp}. Reading the text in the image can be modeled as a sequential problem, where we want to process the picture from left to right. Only the input and the ground truth text label are needed when training with CTC. We discard the information where the particular characters are. CTC tries the possible alignments of the ground truth label in the image and sums all the scores. This way, it does not matter where the text is in the picture - the algorithm could find a suitable label alignment when the match value is high. This value should be maximized to the alignments outputting the ground truth label.

Because CTC tries to find the optimal label alignment, a good character encoding is needed to help distinguish good and bad ones. The characters are separated by a blank token (marked as ``|''), which separates the letters. This way, when a wide character is processed and triggers multiple predictions next to each other, we can merge them between the blank tokens to identify the actual letter. This also solves the problem of segregating the same characters next to each other. The encoding approach makes it possible to represent the same text multiple ways, allowing different alignments to match. The neural network is trained to predict the output in this encoded format.

\begin{itemize}
\item The word ``of'' is recognized when the output is ``|o|ff|'', ``|oooooo||||ffff||||'', or ``o|f''.
\item The word ``off'' is recognized when the output is ``||oo|f||f'', ``|ooo|fff|f|'', or ``o|f|f''.
\end{itemize}

To calculate the scores for an alignment, we have to consider all the possible paths and sum the ones producing the same decoded text. The added up probability is the overall score of the path. Each path text is then compared to the ground truth label, and the loss value is calculated weighted by the path probabilities. The loss function needs to be minimized. For that, CTC uses the negative sum of the log probabilities. The probability is maximized when this loss is minimized, so the function motivates the model to find good predictions.

The model outputs a matrix containing a score for each character at each time step. There are different approaches to decode this matrix. One method is called \textit{best path decoding}, where:

\begin{itemize}
\item The algorithm takes the maximum prediction at each step, producing the most probable path.
\item To decode the prediction, the same characters are removed next to each other, then the blank tokens are also removed. The remaining output is the predicted text.
\end{itemize}

This method is fast and usually good, but it is just an approximation. There are different decoding approaches where multiple paths can be considered (similarly when calculating the CTC loss for the different paths). One other algorithm is Word Beam Search\cite{WordBeamSearch}, where the number of nodes to consider at each time step can be defined. It can make decoding more accurate but also scalable.


\chapter{Clustering}

\citefull{Nguyen2009} analyse how the frequency characteristics of hippocampal ripple events change over the duration of the ripple. They derive the following signals from the electric potential recording (\cref{fig:Nguyen_traces,fig:Nguyen_avg}): the ripple band instantaneous amplitude (`AMP'), the multi-unit activity (`MUA'), the instantaneous frequency (`iFreq'), and the time derivative of this instantaneous frequency (the ``Frequency Modulation'' signal, `FM'). Per detected ripple event, they calculate two main features: ripple frequency and ripple FM. These are computed as the mean of the iFreq and the FM signals respectively, over a $\pm 10$ ms window around the ripple center, which they define as the largest positive peak of the ripple oscillation.

In the resulting ripple FM -- Freq scatter plot (\cref{fig:Nguyen_scatter}), the ripples are spread out in a continuum; no clusters are apparent. They go on to partition the ripples according to four quadrants in this scatter plot: negative FM vs positive FM (decreasing vs increasing frequency) and frequency below vs above 150 Hz. One thing of note is that higher frequency ripples correlate with larger multi-unit activity.

\begin{figure}
    \begin{subfigure}{0.6\textwidth}
        \caption{}
        \label{fig:Nguyen_scatter}
        \includegraphics[width=\textwidth]{Nguyen_scatter}
    \end{subfigure}
    \begin{subfigure}{0.65\textwidth}
        \caption{}
        \label{fig:Nguyen_avg}
        \includegraphics[width=\textwidth]{Nguyen_avg}
    \end{subfigure}
    \begin{subfigure}{0.8\textwidth}
        \caption{}
        \label{fig:Nguyen_traces}
        \includegraphics[width=\textwidth]{Nguyen_traces}
    \end{subfigure}
    \caption
    { Reproduced from \citefull{Nguyen2009}\protect\footnotemark.
    \subref{fig:Nguyen_scatter} Ripple frequency vs `ripple FM' (change in frequency) for all detected ripples, partitioned in four quadrants.
    \subref{fig:Nguyen_avg} Feature waveforms, averaged over all ripples of a quadrant. Ripples were centered around their largest positive ripple peak.
    \subref{fig:Nguyen_traces} Ripple examples for each quadrant. Traces from top to bottom: LFP, ripple band, multi-unit activity, instantaneous frequency, and `FM' (time derivative of instantaneous frequency) with a horizontal zero line.
    }
    \label{fig:Nguyen}
\end{figure}
\footnotetext{Article subject to a license agreement permitting unrestricted reproduction.}


\documentclass[aps,pra,notitlepage,amsmath,amssymb,letterpaper,12pt]{revtex4-1}
\usepackage{amsthm}
\usepackage{graphicx}
%  Above uses the Americal Physical Society template for Physical Review A
%  as a reasonable and fully-featured default template

%  Below define helpful commands to set up problem environments easily
\newenvironment{problem}[2][Problem]{\begin{trivlist}
\item[\hskip \labelsep {\bfseries #1}\hskip \labelsep {\bfseries #2.}]}{\end{trivlist}}
\newenvironment{solution}{\begin{proof}[Solution]}{\end{proof}}

% --------------------------------------------------------------
%                   Document Begins Here
% --------------------------------------------------------------

% In what follows, you can easily change text to see what happens to the document
% For example, replacing the text "Document X" inside the "\title{}" command will
% change the document title

\begin{document}

\title{About Classwork 12}
\author{Amelia and Gwyneth}
\affiliation{PHYS 220, Schmid College of Science and Technology, Chapman University}
\date{\today}

\maketitle

\section{About} % Specify main sections this way

% x.yz is the problem number
\begin{Observations}
In this assignment we observed a ball of mass $m$ with the horizontal coordinates $x$ rolling into a potential of $V(x) = x^4/4 - x^2/2$ and this is known as sombrero. This produces the force $f_{\text{hat}}(x) = -V'(x) = -x^3 + x$ on the ball.  The ball will also experience a drag force $f_{\text{drag}}(\dot{x}) = -\nu \dot{x}$. These forces would make the ball stop at some point.
\end{problem}

\begin{solution} In this assignment we used fourth order Runda Kutta for $m=1$, $\nu = 0.25$, and $\omega = 1$.
\end{solution}

\section{Analysis}
Using the fourth order Runga Kutta method, we could plot the position of the ball verses its velocity. As the force increases, the graph gains more points. This observation is shown in the figures below. The first has a Force value of 0.18. The second has a force value of 0.4.

\begin{figure}[h!] % h forces the figure to be placed here, in the text
  \includegraphics[width=0.8\textwidth]{sombrero.png}  % if pdflatex is used, jpg, pdf, and png are permitted
  \caption{Graphical representation of the sombrero potential with a Force value of 0.18.}
  \label{fig:figlabel}
\end{figure}

\begin{figure}[h!] % h forces the figure to be placed here, in the text
  \includegraphics[width=0.8\textwidth]{sombrero1.png}  % if pdflatex is used, jpg, pdf, and png are permitted
  \caption{Graphical representation of the sombrero potential with a Force value of 0.4.}
  \label{fig:figlabel}
\end{figure}

\section{Conclusions}

From that we know that the fourth order Runga Kutta gives very accurate results in computing differential equations. The motion of the ball starts out a messy, but eventually calms and follows a pattern.

\end{document}


%\documentclass[10pt,a4paper]{article}
\documentclass[10pt,letter]{article}
\usepackage{mathpartir,color,amsthm,amssymb,fullpage}
\usepackage{graphicx}

\newtheorem{theorem}{Theorem}
\newtheorem{lemma}[theorem]{Lemma}
\newtheorem{claim}[theorem]{Claim}
\newtheorem{corollary}[theorem]{Corollary}

\begin{document}

\title{More Flexible Definite Descriptions}
\author{Chris Stone}
\date{\today}
\maketitle

\section{Raw Syntax}

\newcommand{\I}{I}

\renewcommand{\t}{t}
\renewcommand{\u}{u}
\newcommand{\f}{f}
\newcommand{\x}{x}
\newcommand{\y}{y}
\newcommand{\tFun}[3]{#1({#2}_1,\ldots,{#2}_{#3})}
\newcommand{\tDesc}[3]{\rotatebox[origin=c]{180}{$\iota$} #1.\, #3}

%\newcommand{\p}{\psi}
%\newcommand{\q}{\varphi}
%\renewcommand{\r}{\chi}

\newcommand{\p}{P}
\newcommand{\q}{Q}
\renewcommand{\r}{R}

\newcommand{\pTrue}{\top}
\newcommand{\pFalse}{\bot}
% \newcommand{\pNot}[1]{\lnot #1}
\newcommand{\pExists}[3]{\exists #1.\, #3}
\newcommand{\pForall}[3]{\forall #1.\, #3}
\newcommand{\pAnd}[2]{#1 \wedge #2}
\newcommand{\pOr}[2]{#1 \vee #2}
\newcommand{\pImply}[2]{#1 \Rightarrow #2}
\newcommand{\pEqual}[2]{#1 = #2}
\newcommand{\pIff}[2]{#1 \Longleftrightarrow #2}

\newcommand{\D}{\Delta}
\newcommand{\emptyCtx}{\cdot}

\renewcommand{\wp}[1]{\mathop{\mathsf{WP}}(#1)}
\newcommand{\jwp}[1]{\mathop{\mathsf{wp}}(#1)}
\newcommand{\trans}[1]{\ulcorner #1 \urcorner}
\newcommand{\jtrans}[1]{\llcorner #1 \lrcorner}

\newcommand{\jwfterm}[3]{#1 \vdash #2\, :\, #3}
\newcommand{\jwfprop}[2]{#1 \vdash #2\, :\, \mathsf{Prop}}
\newcommand{\jpf}[2]{#1 \vdash #2}

\newcommand{\wfterm}[3]{#1 \,\triangleright\, #2\, :\, #3}
\newcommand{\wfprop}[2]{#1 \,\triangleright\, #2\,:\, \mathsf{Prop}}
\newcommand{\pf}[2]{#1 \,\triangleright\, #2}

\newcommand{\subst}[3]{#1[#2{\mapsto}#3]}

\[
\begin{array}{rl}
\t,\u ::= & \x\ |\ \y\ |\ \cdots\\
     | &\tFun{\f}{\t}{n}\\
     | &\tDesc{\x}{\I}{\p}\\[10pt]

\p,\q,\r ::= &\pTrue\\
| & \pFalse\\
| & \pAnd{\p}{\p}\\
| & \pOr{\p}{\p}\\
| & \pImply{\p}{\p}\\
| & \pForall{\x}{\I}{\p}\\
| & \pExists{\x}{\I}{\p}\\
| & \pEqual{\t}{\u}\\[10pt]

\D ::= & \emptyCtx\\
       |& \D,\x{:}\I\\
       |& \D,\p\\

\end{array}
\]

As usual, we consider two terms or propositions to be equal if they are identical modulo renaming of bound variables.
\section{Original JSL System}

\subsection{Well-Formedness}

\begin{mathpar}
\inferrule
  {\x{:}\I \in \D}
  { \jwfterm{\D}{\x}{\I} }

\inferrule
  { \jwfterm{\D}{\t_1}{\I}\\ \cdots \\ \jwfterm{\D}{\t_n}{\I} }
  { \jwfterm{\D}{\tFun{\f}{\t}{n}}{\I} }

\inferrule
  { \jpf{\D}{\pExists{\x}{\I}{\p(\x)}} \\
    \jpf{\D}{\pForall{\x}{\I}{\pForall{\y}{\I}
               {\pImply{\pAnd{\p(\x)}{\p(\y)}}{\x=\y}}}}}
  { \jwfterm{\D}{\tDesc{\x}{\I}{\p(\x)}}{\I} }
\end{mathpar}

\begin{mathpar}
\inferrule
  { }
  { \jwfprop{\D}{\pTrue} }

\inferrule
  { }
  { \jwfprop{\D}{\pFalse} }

%\inferrule
% { \jwfprop{\D}{\p} }
% { \jwfprop{\D}{\pNot{\p}} }

\inferrule
 { \jwfprop{\D}{\p} \\
   \jwfprop{\D}{\q} }
 { \jwfprop{\D}{\pOr{\p}{\q}} }

\inferrule
 { \jwfprop{\D}{\p} \\
   \jwfprop{\D,\p}{\q} }
 { \jwfprop{\D}{\pAnd{\p}{\q}} }

\inferrule
 { \jwfprop{\D}{\p} \\
   \jwfprop{\D,\p}{\q} }
 { \jwfprop{\D}{\pImply{\p}{\q}} }

\inferrule
 { \jwfprop{\D,\x{:}\I}{\p} }
 { \jwfprop{\D}{\pForall{\x}{\I}{\p}} }

\inferrule
 { \jwfprop{\D,\x{:}\I}{\p} }
 { \jwfprop{\D}{\pExists{\x}{\I}{\p}} }

\inferrule
 { \jwfterm{\D}{\t}{\I} \\ \jwfterm{\D}{\u}{\I} }
 { \jwfprop{\D}{\pEqual{\t}{\u}} }

\end{mathpar}

\subsection{Provability}

\begin{mathpar}
\inferrule
  {\p\in\D}
  {\jpf{\D}{\p}}

\inferrule
  {\jpf{\D}{\p} \\ \jpf{\D}{\q}}
  {\jpf{\D}{\pAnd{\p}{\q}} }

\inferrule
  {\jpf{\D}{\pAnd{\p}{\q}} }
  {\jpf{\D}{\p}}

\inferrule
  {\jpf{\D}{\pAnd{\p}{\q}} }
  {\jpf{\D}{\q}}

\fbox{
\inferrule
  { \jpf{\D}{\p} \\ \jwfprop{\D}{\pOr{\p}{\q}} }
  { \jpf{\D}{\pOr{\p}{\q}} }
}

\fbox{
\inferrule
  { \jpf{\D}{\q} \\ \jwfprop{\D}{\pOr{\p}{\q}} }
  { \jpf{\D}{\pOr{\p}{\q}} }
}

\fbox{
\inferrule
  { \jwfprop{\D}{\r} \\\\
    \jpf{\D}{\pOr{\p}{\q}} \\ \jpf{\D,\p}{\r} \\ \jpf{\D,\q}{\r} }
  { \jpf{\D}{\r} }
}

\iffalse
\inferrule
  { \jwfprop{\D}{\p} \\ \jpf{\D,
\fi

\inferrule
  {\jwfprop{\D}{\p} \\ \jpf{\D}{\pFalse}}
  {\jpf{\D}{\p}}

\inferrule
  {\jwfprop{\D}{\p} \\ \jpf{\D,\p}{\q}}
  {\jpf{\D}{\pImply{\p}{\q}}}

\inferrule
  { \jpf{\D}{\p} \\ \jpf{\D}{\pImply{\p}{\q}} }
  { \jpf{\D}{\q} }

\\

\inferrule
  { \jpf{\D,\x{:}\I}{\p(\x)} }
  { \jpf{\D}{\pForall{\x}{\I}{\p(\x)}} }

\inferrule
  { \jpf{\D}{\pForall{\x}{\I}{\p(\x)}} \\ \jwfterm{\D}{\t}{\I} }
  { \jpf{\D}{\p(\t)} }

\\

\fbox{
\inferrule
  { \jwfprop{\D}{\pExists{\x}{\I}{\p(\x)}} \\\\
    \jpf{\D}{\p(\t)} \\ \jwfterm{\D}{\t}{\I} }
  { \jpf{\D}{\pExists{\x}{\I}{\p(\x)}} }
}

\fbox{
\inferrule
  { \jwfprop{\D}{\q} \\\\
    \jpf{\D}{\pExists{\x}{\I}{\p(\x)}} \\
    \jpf{(\D,\x{:}\I,\p(\x))}{\q} \\
     }
  { \jpf{\D}{\q} }
}

\inferrule
  { \jpf{\D}{\pExists{\x}{\I}{\p(\x)}} \\
    \jpf{\D}{\pForall{\x}{\I}{\pForall{\y}{\I}
               {\pImply{\pAnd{\p(\x)}{\p(\y)}}{\x=\y}}}}}
  { \jpf{\D}{\p(\tDesc{\x}{\I}{\p(\x)})} }

\inferrule
  { \jwfprop{\t}{\I} }
  { \jpf{\D}{\pEqual{\t}{\t}} }

\inferrule
  { \jpf{\D}{\p(\t)} \\ \jpf{\D}{\pEqual{\t}{\u}} }
  { \jpf{\D}{\p(\u)} }

\end{mathpar}

\section{Lax System}

\subsection{Well-Formedness}

\begin{mathpar}
\inferrule
  {\x{:}\I \in \D}
  { \wfterm{\D}{\x}{\I} }

\inferrule
  { \wfterm{\D}{\t_1}{\I}\\ \cdots \\ \wfterm{\D}{\t_n}{\I} }
  { \wfterm{\D}{\tFun{\f}{\t}{n}}{\I} }

\inferrule
  { \pf{\D}{\pExists{\x}{\I}{\p(\x)}} \\
    \pf{\D}{\pForall{\x}{\I}{\pForall{\y}{\I}
               {\pImply{\pAnd{\p(\x)}{\p(\y)}}{\x=\y}}}}}
  { \wfterm{\D}{\tDesc{\x}{\I}{\p(\x)}}{\I} }
\end{mathpar}

\begin{mathpar}
\inferrule
  { }
  { \wfprop{\D}{\pTrue} }

\inferrule
  { }
  { \wfprop{\D}{\pFalse} }

%\inferrule
% { \wfprop{\D}{\p} }
% { \wfprop{\D}{\pNot{\p}} }

\inferrule
 { \wfprop{\D}{\p_1} \\
   \wfprop{(\D,\p_1)}{\p_2} }
 { \wfprop{\D}{\pAnd{\p_1}{\p_2}} }

\inferrule
 { \wfprop{\D}{\p_1} \\
   \wfprop{(\D,\p_1)}{\p_2} }
 { \wfprop{\D}{\pImply{\p_1}{\p_2}} }

\fbox{
\inferrule
 { \wfprop{(\D,\pFalse)}{\p_1} \\
   \wfprop{(\D,\pFalse)}{\p_2} }
 { \wfprop{\D}{\pOr{\p_1}{\p_2}} }
}

\inferrule
 { \wfprop{(\D,\x{:}\I)}{\p} }
 { \wfprop{\D}{\pForall{\x}{\I}{\p}} }

\fbox{
\inferrule
 { \wfprop{(\D,\x{:}\I,\pFalse)}{\p}}
 { \wfprop{\D}{\pExists{\x}{\I}{\p}} }
}

\inferrule
 { \wfterm{\D}{\t}{\I} \\ \wfterm{\D}{\u}{\I} }
 { \wfprop{\D}{\pEqual{\t}{\u}} }

\end{mathpar}

\subsection{Provability}

Unchanged except for replacement of $\vdash$ by $\triangleright$ in every judgment!

\section{Well-Formedness and Translation Functions:  Strict}

\[
\begin{array}{lll}
  \jwp{\x} & := & \pTrue\\
  \jwp{\tFun{\f}{\t}{n}} & := & \pAnd{\jwp{\t_1}}{\pAnd{\cdots}{\jwp{\t_n}}}\\
  \jwp{\tDesc{\x}{\I}{\p(\x)}} & := &
       (\pForall{\x}{\I}{\jwp{\p(\x)}}) \wedge
       (\pExists{\x}{\I}{\p(\x)} \wedge
       (\pForall{\y}{\I}
               {\pImply{\p(\y)}{\x=\y}}))\\[10pt]

  \jwp{\pTrue} & := & \pTrue\\
  \jwp{\pFalse} & := & \pTrue\\
%  \jwp{\pOr{\p}{\q}} & := & \pTrue\\
  \jwp{\pOr{\p}{\q}} & := & \pAnd{\jwp{\p}}{\jwp{\q}}\\
  \jwp{\pAnd{\p}{\q}} & := & \pAnd{\jwp{\p}}{(\pImply{\p}{\jwp{\q}})}\\
  \jwp{\pImply{\p}{\q}} & := & \pAnd{\jwp{\p}}{(\pImply{\p}{\jwp{\q}})}\\
  \jwp{\pForall{\x}{\I}{\p(\x)}} & := & \pForall{\x}{\I}{\jwp{\p(\x)}}\\
%  \jwp{\pExists{\x}{\I}{\p(\x)}} & := & \pTrue\\
  \jwp{\pExists{\x}{\I}{\p(\x)}} & := & \pForall{\x}{\I}{\jwp{\p(\x)}}\\
  \jwp{\pEqual{\t}{\u}} & := & \pAnd{\jwp{\t}}{\jwp{\u}}\\[20pt]

\iffalse
  \jtrans{\x} & := & \x\\
  \jtrans{\tFun{\f}{\t}{n}} & := & \f(\jtrans{\t_1},\ldots,\jtrans{\t_n})\\
  \jtrans{\tDesc{\x}{\I}{\p(\x)}} & := & \tDesc{\x}{\I}{\jtrans{\p(\x)}}\\[10pt]

  \jtrans{\pTrue} & := & \pTrue\\
  \jtrans{\pFalse} & := & \pFalse\\
  \jtrans{\pOr{\p}{\q}} & := & \pOr{(\pAnd{\jwp{\p}}{\jtrans{\p}})}{(\pAnd{\jwp{\q}}{\jtrans{\q}})}\\
  \jtrans{\pAnd{\p}{\q}} & := & \pAnd{\jtrans{\p}}{\jtrans{\q}}\\
  \jtrans{\pImply{\p}{\q}} & := & \pImply{\jtrans{\p}}{\jtrans{\q}}\\
  \jtrans{\pForall{\x}{\I}{\p}} & := & \pForall{\x}{\I}{\jtrans{\p}} \\
  \jtrans{\pExists{\x}{\I}{\p}} & := & \pExists{\x}{\I}{(\pAnd{\jwp{\p}}{\jtrans{\p}})}\\
  \jtrans{\pEqual{\t}{\u}} & := & \pEqual{\jtrans{\t}}{\jtrans{\u}}\\[10pt]

  \jtrans{\emptyCtx} & := & \emptyCtx\\
  \jtrans{\D,\p} & := & \jtrans{\D},\jtrans{\p}\\
\fi %\iffalse
\end{array}
\]

%\clearpage

\section{Well-Formedness and Translation Functions: Lax}

\[
\begin{array}{lll}
  \wp{\x} & := & \pTrue\\
  \wp{\tFun{\f}{\t}{n}} & := & \pAnd{\wp{\t_1}}{\pAnd{\cdots}{\wp{\t_n}}}\\
  \wp{\tDesc{\x}{\I}{\p(\x)}} & := &
%       (\pForall{\x}{\I}{\wp{\p(\x)}}) \wedge
\mbox{\fbox{$
        \pExists{\x}{\I}{(\pAnd{\p(\x)}{(\pForall{\y}{\I}
               {\pImply{\p(\y)}{\x=\y}})})}$}}\\[10pt]

  \wp{\pTrue} & := & \pTrue\\
  \wp{\pFalse} & := & \pTrue\\
  \wp{\pOr{\p}{\q}} & := & \mbox{\fbox{$\pTrue$}}\\
%  \wp{\pOr{\p}{\q}} & := & \pAnd{\wp{\p}}{\wp{\q}}\\
  \wp{\pAnd{\p}{\q}} & := & \pAnd{\wp{\p}}{(\pImply{\p}{\wp{\q}})}\\
  \wp{\pImply{\p}{\q}} & := & \pAnd{\wp{\p}}{(\pImply{\p}{\wp{\q}})}\\
  \wp{\pForall{\x}{\I}{\p(\x)}} & := & \pForall{\x}{\I}{\wp{\p(\x)}}\\
  \wp{\pExists{\x}{\I}{\p(\x)}} & := & \mbox{\fbox{$\pTrue$}}\\
  \wp{\pEqual{\t}{\u}} & := & \pAnd{\wp{\t}}{\wp{\u}}\\[20pt]

  \trans{\x} & := & \x\\
  \trans{\tFun{\f}{\t}{n}} & := & \f(\trans{\t_1},\ldots,\trans{\t_n})\\
  \trans{\tDesc{\x}{\I}{\p(\x)}} & := & \tDesc{\x}{\I}{\trans{\p(\x)}}\\[10pt]

  \trans{\pTrue} & := & \pTrue\\
  \trans{\pFalse} & := & \pFalse\\
  \trans{\pOr{\p}{\q}} & := & \pOr{(\pAnd{\wp{\p}}{\trans{\p}})}{(\pAnd{\wp{\q}}{\trans{\q}})}\\
  \trans{\pAnd{\p}{\q}} & := & \pAnd{\trans{\p}}{\trans{\q}}\\
  \trans{\pImply{\p}{\q}} & := & \pImply{\trans{\p}}{\trans{\q}}\\
  \trans{\pForall{\x}{\I}{\p}} & := & \pForall{\x}{\I}{\trans{\p}} \\
  \trans{\pExists{\x}{\I}{\p}} & := & \pExists{\x}{\I}{(\pAnd{\wp{\p}}{\trans{\p}})}\\
  \trans{\pEqual{\t}{\u}} & := & \pEqual{\trans{\t}}{\trans{\u}}\\[10pt]

  \trans{\emptyCtx} & := & \emptyCtx\\
  \trans{\D,\p} & := & \trans{\D},\trans{\p}\\
\end{array}
\]

\clearpage

\section{Metatheory}

\begin{lemma}[Substitution]
\label{lemma:subst}
\mbox{}
 \begin{enumerate}
	\item If $\jwfterm{(\D,\x{:}\I,\D')}{\t}{\I}$ and
	  $\jwfterm{(\D,\D')}{\emptyCtx}{\u}{\I}$ then
	  $\jwfterm{(\D,\subst{\D'}{\x}{\u})}{\subst{\t}{\x}{\u}}{\I}$.
	\item If $\jwfprop{(\D,\x{:}\I,\D')}{\p}$ and
	  $\jwfterm{\emptyCtx}{\u}{\I}$ then
	  $\jwfprop{(\D,\subst{\D'}{\x}{\u})}{\subst{\p}{\x}{\u}}$.
	\item If $\jpf{(\D,\x{:}\I,\D')}{\p}$ and
	  $\jwfterm{\emptyCtx}{\u}{\I}$ then
	  $\jpf{(\D,\subst{\D'}{\x}{\u})}{\subst{\p}{\x}{\u}}$.
 \end{enumerate}
\end{lemma}

\begin{proof}
To be checked.
\end{proof}

\begin{lemma}[Weakening]
\mbox{}
\label{lemma:weaken}
\begin{enumerate}
	\item If $\jwfterm{\D_1}{\t}{\I}$ then
	   $\jwfterm{(\D_1,\D_2)}{\t}{\I}$.
	\item If $\jwfprop{\D_1}{\p}$ then
	   $\jwfprop{(\D_1,\D_2)}{\p}$.
\end{enumerate}
\end{lemma}

\begin{proof}
To be checked.
\end{proof}

\begin{lemma}[Cut]
\label{lemma:cut}
\mbox{}
\begin{enumerate}
  \item If $\jwfterm{(\D,\p)}{\t}{\I}$ and
     $\jpf{\D}{\p}$ then
     $\jwfterm{\D}{\t}{\I}$.
\end{enumerate}
\end{lemma}

\begin{proof}
To be checked.
\end{proof}

\begin{lemma}[Logical weakening]
\label{lemma:logweak}
\mbox{}
\begin{enumerate}
\item If $\jpf{\D}{\p}$ and $\jwfprop{\p}{\q}$ then
  $\jwfprop{\D}{\q}$.
\end{enumerate}
\end{lemma}

\begin{proof}
  To be checked.
\end{proof}

\begin{lemma}
\mbox{}
\begin{enumerate}
  \item If $\jwfterm{\D}{\t}{\I}$ then
    $\wfterm{\D}{\t}{\I}$.
  \item If $\jwfprop{\D}{\p}$ then
    $\wfprop{\D}{\p}$.
  \item If $\jpf{\D}{\p}$ then
    $\pf{\D}{\p}$.
\end{enumerate}
\end{lemma}

\begin{proof}
By induction on derivations, using weakening and cut FOR THE LAX SYSTEM.
\end{proof}

\begin{lemma}[Well-formedness of  $\jwp{\cdot}$]
\label{lemma:wfwp}
\mbox{}
  \begin{enumerate}
  \item If $\jwfterm{\D}{\t}{\I}$ then
    $\jwfprop{\emptyCtx}{\jwp{\t}}$.
  \item If $\jwfprop{\D}{\p}$, we have
    $\jwfprop{\emptyCtx}{\jwp{\p}}$.
  \end{enumerate}
\end{lemma}

\begin{proof}
  \raggedright Induction on the proof of
  $\jwfterm{\D}{\t}{\I}$ or $\jwfprop{\D}{\p}$.
  \begin{itemize}
  \item Case: $\t = \x$ with $\x{:}\I\in\D$.
    \begin{enumerate}
      \item
        Here $\jwp{\t} = \pTrue$,
      \item
        and $\jwfprop{\emptyCtx}{\pTrue}$.
      \end{enumerate}
  \item Case: $\t = \tFun{\f}{\t}{n}$ with
    $\jwfterm{\D}{\t_i}{\I}$ for all $i\in1..n$.
    \begin{enumerate}
    \item
      Here $\jwp{\t} =
      \pAnd{\jwp{\t_1}}{\pAnd{\cdots}{\jwp{\t_n}}}$.
    \item
    	By induction,
      $\jwfprop{\emptyCtx}{\jwp{\t_1}}$ for
      all $i\in1..n$,
    \item
    	so
    $\jwfprop{\emptyCtx}{\pAnd{\jwp{\t_1}}{\pAnd{\cdots}{\jwp{\t_n}}}}$.
    \end{enumerate}

\goodbreak
  \item Case: $\t = \tDesc{\x}{\I}{\p(\x)}$ with
  $\jpf{\D}{\pExists{\x}{\I}{\p(\x)}}$ and
  $\jpf{\D}{\pForall{\x}{\I}{\pForall{\y}{\I}
               {\pImply{\pAnd{\p(\x)}{\p(\y)}}{\x=\y}}}}$.

    \begin{enumerate}
      \item
        Then
        $\jwp{\t} =
        \pAnd
          {(\pForall{\x}{\I}{\jwp{\p(\x)}})}
          {\pAnd
            {(\pExists{\x}{\I}{\p(\x)})}
            {(\pForall{\x}{\I}{\pForall{\y}{\I}
              {\pImply{\pAnd{\p(\x)}{\p(\y)}}{\x=\y}}})}}$.
      \item
        By the inductive hypothesis, $\jwfprop{\emptyCtx}{\jwp{\pExists{\x}{\I}{\p(\x)}}}$
\bigskip

{\bf BOGUS---can't apply the inducive hypothesis to a {\it logical} derivation!
}
      \item
      	and
        $\jwfprop{\emptyCtx}{\jwp{\pForall{\x}{\I}{\pForall{\y}{\I}
               {\pImply{\pAnd{\p(\x)}{\p(\y)}}{\x=\y}}}}}$.
      \item
      	That is,
	$\jwfprop{\emptyCtx}
	   {\pForall{\x}{\I}
	      {\pForall{\y}{\I}
	        {\pAnd
	          {\bigl(\pAnd
	             {\jwp{\p(\x)}}
	             {(\pImply{\p(\x)}{\jwp{\p(\y)}})}\bigr)}
	          {(\pImply
	             {\pAnd{\p(\x)}{\p(\y)}}
	             {\pAnd{\pTrue}{\pTrue}})} }}}$.
	  \item
	    By inversion,
	      $\jwfprop{(\D,\x{:}\I,\y{:}\I)}{\jwp{\p(\x)}}$,
      \item
        so by Lemma~\ref{lemma:subst} we have
        $\jwfprop{(\D,\x{:}\I)}{\jwp{\p(\x)}}$,
      \item
        and hence $\jwfprop{\emptyCtx}{\pForall{\x}{\I}{\jwp{\p(x)}}}$.
      \item
        Thus, $\jwfprop{\emptyCtx}{\jwp{\t}}$.
      \end{enumerate}

  \item Cases: $\p = \pTrue$, or $\p = \pFalse$, or $\p=(\pOr{\p_1}{\p_2})$,
      or $\p=(\pExists{\x}{\I}{\q})$.
      \begin{enumerate}
      \item
      	Here $\jwp{\p} = \pTrue$.
      \item
        Then $\jwfprop{\emptyCtx}{\pTrue}$.
      \end{enumerate}

  \item Case: $\p = (\pAnd{\p_1}{\p_2})$ or $\p = (\pImply{\p_1}{\p_2})$ with
       $\jwfprop{\D}{\p_1}$ and
              $\jwfprop{(\D,\p_1)}{\p_2}$.
       \begin{enumerate}
       \item
       		Then $\jwp{\p} = \pAnd{\jwp{\p_1}}{(\pImply{\p_1}{\jwp{\p_2}})}$.
       \item
       		By the inductive hypothesis,
         	$\jwfprop{\emptyCtx}{\jwp{\p_1}}$
         	and $\jwfprop{\emptyCtx}{\jwp{\p_2}}$.
       \item
         By Lemma~\ref{lemma:weaken} we have
         $\jwfprop{\p_1}{\jwp{\p_2}}$,
       \item so $\jwfprop{\emptyCtx}{(\pImply{\p_1}{\jwp{\p_2}})}$,

         {\bf Oops...this doesn't follow!  We didn't (and in
           general can't) show that $\jwfprop{\emptyCtx}{\p_1}$.
         Maybe we should define
\[
\begin{array}{lll}
\jwp{\pAnd{\p}{\q}} &:=& \pAnd{\jwp{\p}}{(\pImply{\trans\p}{\wp\q})}\\
\jwp{\pImply{\p}{\q}} &:=& \pAnd{\jwp{\p}}{(\pImply{\trans\p}{\wp\q})}\\
\end{array}
\]
instead?
}

       \item
         and hence $\jwfprop{\emptyCtx}{\jwp{\p}}$.
       \end{enumerate}
  \item Case: $\p=(\pForall{\x}{\I}{\q})$,
        with $\jwfprop{(\D,\x{:}\I)}{\q}$.
  	\begin{enumerate}
	\item
		Here $\jwp{\p} = \pForall{\x}{\I}{\jwp{\q}}$.
	\item
		By the inductive hypothesis,
	   $\jwfprop{(\D,\x{:}\I)}{\jwp{\q}}$.
	\item
		Thus
	   $\jwfprop{\emptyCtx}{\pForall{\x}{\I}{\jwp{\q}}}$ as desired.
	\end{enumerate}
  \item Case: $\p = (\pEqual{\t_1}{\t_2})$, with $\jwfterm{\D}{\t_1}{\I}$ and $\jwfterm{\D}{\t_2}{\I}$.
    	\begin{enumerate}
	\item
		Here $\jwp{\p} = \pAnd{\jwp{\t_1}}{\jwp{\t_2}}$.
	\item
		By the inductive hypotheses,
		$\jwfprop{\emptyCtx}{\jwp{\t_1}}$
	\item
		and $\jwfprop{\emptyCtx}{\jwp{\t_2}}$.
	\item
		Thus $\jwfprop{\emptyCtx}{\pAnd{\jwp{\t_1}}{\jwp{\t_2}}}$.
	\end{enumerate}
  \end{itemize}
\end{proof}

\begin{corollary}
\mbox{}
\begin{enumerate}
\item If $\jwfterm{\D}{\t}{\I}$ then
  $\jwfprop{\D}{\wp{\t}}$.
\item If $\jwfprop{\D}{\p}$ then
  $\jwfprop{\D}{\wp{\p}}$.
\end{enumerate}
\end{corollary}

\begin{proof}
To be checked.
\end{proof}

\clearpage

\begin{lemma}[Truth of $\wp{\cdot}$]
\mbox{}
  \begin{enumerate}
  \item If $\jwfterm{\D}{\t}{\I}$ then
    $\jpf{\D}{\wp{\t}}$.
  \item If $\jwfprop{\D}{\p}$, we have
    $\jpf{\D}{\wp{\p}}$.
  \end{enumerate}
\end{lemma}

\begin{proof}
  \raggedright By induction on the proof of
  $\jwfterm{\D}{\t}{\I}$ or $\jwfprop{\D}{\p}$.
%  In all cases we have $\jwfprop{\D}{\wp{\t}}$ or
%  $\jwfprop{\D}{\wp{\p}}$ as appropriate.
  \begin{itemize}
  \item Case: $\t = \x$ with $\x{:}\I\in\G$.
    \begin{enumerate}
      \item
        Here $\wp{\t} = \pTrue$.
      \item
        Then $\jpf{\emptyCtx}{\pTrue}$
      \end{enumerate}
  \item Case: $\t = \tFun{\f}{\t}{n}$ with
    $\jwfterm{\D}{\t_i}{\I}$ for all $i\in1..n$.
    \begin{enumerate}
    \item
      Here $\wp{\t} =
      \pAnd{\wp{\t_1}}{\pAnd{\cdots}{\wp{\t_n}}}$.
    \item
    	By induction,
      $\jpf{\D}{\wp{\t_1}}$ for
      all $i\in1..n$,
    \item
    	so
    $\jpf{\D}{\pAnd{\wp{\t_1}}{\pAnd{\cdots}{\wp{\t_n}}}}$.
    \end{enumerate}

  \item Case: $\t = \tDesc{\x}{\I}{\p(\x)}$ with
  $\jpf{\D}{\pExists{\x}{\I}{\p(\x)}}$ and
  $\jpf{\D}{\pForall{\x}{\I}{\pForall{\y}{\I}
               {\pImply{\pAnd{\p(\x)}{\p(\y)}}{\x=\y}}}}$.

    \begin{enumerate}
      \item
        Here
        $\wp{\t} =
        \pAnd
          {(\pForall{\x}{\I}{\wp{\p(\x)}})}
          {\pAnd
            {(\pExists{\x}{\I}{\p(\x)})}
            {(\pForall{\x}{\I}{\pForall{\y}{\I}
              {\pImply{\pAnd{\p(\x)}{\p(\y)}}{\x=\y}}})}}$.
      \item
        By the inductive hypothesis,
        $\jpf{\D}{\wp{\pForall{\x}{\I}{\pForall{\y}{\I}
               {\pImply{\pAnd{\p(\x)}{\p(\y)}}{\x=\y}}}}}$.
      \item
      	That is,
	$\jpf{\D}
	   {\pForall{\x}{\I}
	      {\pForall{\y}{\I}
	        {\pAnd
	          {\bigl(\pAnd
	             {\wp{\p(\x)}}
	             {(\pImply{\p(\x)}{\wp{\p(\y)}})}\bigr)}
	          {(\pImply
	             {\pAnd{\p(\x)}{\p(\y)}}
	             {\pAnd{\pTrue}{\pTrue}})} }}}$.
	  \item
	    From this (and conjunction elimination and introduction/elimination
	    for the universal quantifier) we can prove
	      $\jpf{\D}{\pForall{\x}{\I}{\wp{\p(\x)}}}$.
      \item
        Thus, $\jpf{\D}{\wp{\t}}$.
      \end{enumerate}

  \item Cases: $\p = \pTrue$, or $\p = \pFalse$, or $\p = (\pOr{\p_1}{\p_2})$, or $\p = (\pExists{\x}{\I}{\q})$.
      \begin{enumerate}
      \item
      	Then $\wp{\p} = \pTrue$,
      \item
        and $\jpf{\D}{\pTrue}$
      \end{enumerate}

  \item Case: $\p = (\pAnd{\p_1}{\p_2})$ or $\p = (\pImply{\p_1}{\p_2})$ with
       $\jwfprop{\D}{\p_1}$ and
              $\jwfprop{(\D,\p_1)}{\p_2}$.
       \begin{enumerate}
       \item Then $\wp{\p} = \pAnd{\wp{\p_1}}{(\pImply{\p_1}{\wp{\p_2}})}$.
       \item By the inductive hypothesis,
         $\jpf{\D}{\wp{\p_1}}$
       \item
         and $\jpf{\D,\p_1}{\wp{\p_2}}$.
       \item
         Thus $\jpf{\D}{\pImply{\p_1}{\wp{\p_2}}}$,
       \item
         so $\jpf{\D}{\wp{\p}}$.
       \end{enumerate}
  \item Case: $\p=(\pForall{\x}{\I}{\q})$,
        with $\jwfprop{(\G,\x{:}\I)}{\D}{\q}$.
  	\begin{enumerate}
	\item
		Here $\wp{\p} = \pForall{\x}{\I}{\wp{\q}}$.
	\item
		By the inductive hypothesis,
	   $\jpf{(\G,\x{:}\I)}{\D}{\wp{\q}}$.
	\item
		Thus
	   $\jpf{\D}{\pForall{\x}{\I}{\wp{\q}}}$ as desired.
	\end{enumerate}
  \item Case: $\p = (\pEqual{\t_1}{\t_2})$, with $\jwfterm{\D}{\t_1}{\I}$ and $\jwfterm{\D}{\t_2}{\I}$.
    	\begin{enumerate}
	\item
		Here $\wp{\p} = \pAnd{\wp{\t_1}}{\wp{\t_2}}$.
	\item
		By the inductive hypotheses,
		$\jpf{\D}{\wp{\t_1}}$
	\item
		and $\jpf{\D}{\wp{\t_2}}$.
	\item
		Thus $\jpf{\D}{\pAnd{\wp{\t_1}}{\wp{\t_2}}}$.
	\end{enumerate}
  \end{itemize}
\end{proof}

\clearpage

\begin{lemma}
\label{lemma:l9}
\mbox{}
  \begin{enumerate}
  \item If $\jwfterm{(\D,\pFalse)}{\t}{\I}$ then
    $\jwfterm{\wp{\t}}{\trans{\t}}{\I}$.
  \item If $\jwfprop{(\D,\pFalse)}{\p}$, then
    $\jwfprop{\wp{\p}}{\trans{\p}}$.
  \end{enumerate}
\end{lemma}

\begin{proof}
  \raggedright Induction on the proof of
  $\jwfterm{(\D,\pFalse)}{\t}{\I}$ or $\jwfprop{(\D,\pFalse)}{\p}$.
  \begin{itemize}
  \item Case: $\t = \x$ with $\x{:}\I\in\G$.
    \begin{enumerate}
      \item
        Here $\wp{\t} = \pTrue$ and $\trans{\t} = \x$.
      \item
        Then $\jwfterm{\pTrue}{\x}{\I}$.
      \end{enumerate}
  \item Case: $\t = \tFun{\f}{\t}{n}$ with
    $\jwfterm{(\D,\pFalse)}{\t_i}{\I}$ for all $i\in1..n$.
    \begin{enumerate}
    \item
      Here $\wp{\t} =
      \pAnd{\wp{\t_1}}{\pAnd{\cdots}{\wp{\t_n}}}$
      and $\trans{\t} = \f(\trans{\t_1},\ldots,\trans{\t_n})$.
    \item
    	By induction,
        $\jwfterm{\wp{\t_i}}{\trans{\t_i}}{\I}$ for
      all $i\in1..n$.
    \item
      By Lemma~\ref{lemma:weaken} we have
      $\jwfterm{(\wp{\t},\wp{\t_i})}{\trans{\t_i}}{\I}$
      for all $i\in1..n$.
    \item
      By Lemma~\ref{lemma:cut} we have
      $\jwfterm{\wp{\t}}{\trans{\t_i}}{\I}$ for
      all $i\in1..n$.
    \item Thus,
      $\jwfterm{\wp{\t}}{\f(\trans{\t_1},\ldots,\trans{\t_n})}{\I}$.
    \end{enumerate}

  \goodbreak

  \item Case: $\t = \tDesc{\x}{\I}{\p(\x)}$ with
  $\jpf{(\D,\pFalse)}{\pExists{\x}{\I}{\p(\x)}}$ and
  $\jpf{(\D,\pFalse)}{\pForall{\x}{\I}{\pForall{\y}{\I}
               {\pImply{\pAnd{\p(\x)}{\p(\y)}}{\x=\y}}}}$.

    \begin{enumerate}
      \item
        Then
        $\wp{\t} =
        \pAnd
          {(\pForall{\x}{\I}{\wp{\p(\x)}})}
          {\pAnd
            {(\pExists{\x}{\I}{\p(\x)})}
            {(\pForall{\x}{\I}{\pForall{\y}{\I}
              {\pImply{\pAnd{\p(\x)}{\p(\y)}}{\x=\y}}})}}$.
      \item
        and
        $\trans{\t} = \tDesc{\x}{\I}{\trans{\p(\x)}}$.
      \item
        By the inductive hypothesis
        $\jwfprop{\pTrue}
          {\pExists{\x}{\I}{\pAnd{\wp{\p(\x)}}{\trans{\p(\x)}}}}$
      \item
        and
        $\jwfprop{\pForall{\x}{\I}
                       {\pForall{\y}{\I}
                       	 {\pAnd{(\pAnd{\wp{\p(\x)}}{(\pImply{\p(\x)}{\wp{\p(\y)}})})}
	                           {(\pImply{\pAnd{\p(\x)}{\p(\y)}}
	                                    {\pAnd{\pTrue}{\pTrue}})}}}}
            {(\pForall{\x}{\I}{\pForall{\y}{\I}
              {\pImply{\pAnd{\p(\x)}{\p(\y)}}{\x=\y}}})}$.
        \item \ldots
      \end{enumerate}

  \item Cases: $\p = \pTrue$ or $\p = \pFalse$.
      \begin{enumerate}
      \item
      	Then $\wp{\p} = \pTrue$,
      \item
        and $\jwfprop{\emptyCtx}{\pTrue}$
      \item
        and $\jwfterm{\pTrue}{\x}{\I}$.
      \end{enumerate}

  \item Case: $\p = (\pOr{\p_1}{\p_2})$ with
       $\jwfprop{(\D,\pFalse)}{\p_1}$ and
              $\jwfprop{(\D,\pFalse)}{\p_2}$.
      	\begin{enumerate}
		\item
                  Here $\wp{\p} = \pTrue$
                \item
                  and $\trans{\p} =
                  \pOr{(\pAnd{\wp{\p_1}}{\trans{\p_1}})}{(\pAnd{\wp{\p_2}}{\trans{\p_2}})}$.
		\item
                  By the inductive hypothesis and \ref{lemma:weaken}
                  $\jwfprop{(\pTrue,\wp{\p_1})}{\trans{\p_2}}$
                \item
                  and
                  $\jwfprop{(\pTrue,\wp{\p_2})}{\trans{\p_2}}$
		\item
                  By Lemmas~\ref{lemma:wfwp} and \ref{lemma:weaken},
                  $\jwfprop{\pTrue}{\wp{\p_1}}$
                \item
                  and
                  $\jwfprop{\pTrue}{\wp{\p_2}}$.
		\item
                  Thus,
                  $\jwfprop{\pTrue}{\pAnd{\wp{\p_1}}{\trans{\p_1}}}$
                \item
                  and
                  $\jwfprop{\pTrue}{\pAnd{\wp{\p_2}}{\trans{\p_2}}}$
		\item
                  Therefore,
                  $\jwfprop{\wp{\p}}{\trans{\p}}$.
		\end{enumerate}

  \item Case: $\p = (\pAnd{\p_1}{\p_2})$ or $\p = (\pImply{\p_1}{\p_2})$ with
       $\jwfprop{(\D,\pFalse)}{\p_1}$ and
              $\jwfprop{(\D,\pFalse,\p_1)}{\p_2}$.
       \begin{enumerate}
       \item Then $\wp{\p} = \pAnd{\wp{\p_1}}{(\pImply{\p_1}{\wp{\p_2}})}$.
       \item By the inductive hypothesis,
         $\jwfprop{\emptyCtx}{\wp{\p_1}}$
         and $\jwfprop{\emptyCtx}{\wp{\p_2}}$.
       \item
       		By Lemma~\ref{lemma:weaken} we have
			$\jwfprop{\p_1}{\wp{\p_2}}$,
		\item
			so $\jwfprop{\emptyCtx}{(\pImply{\p_1}{\wp{\p_2}})}$,
		\item
		 	and hence $\jwfprop{\emptyCtx}{\p}$.
	    \item
	        Again by the inductive hypothesis,
	        $\jwfprop{\wp{\p_1}}{\p_1}$
	    \item
	        and $\jwfprop{\wp{\p_2}}{\p_2}$.
	    \item
	        By Lemma~\ref{lemma:weaken} we have
	        $\jwfprop
	           {(\pAnd{\wp{\p_1}}{(\pImply{\p_1}{\wp{\p_2}})},\wp{\p_1})}
	           {\p_1}$
	    \item
	        and since
	        $\jpf{\pAnd{\wp{\p_1}}{(\pImply{\p_1}{\wp{\p_2}})}}{\wp{\p_1}}$,
	    \item
	        by Lemma~\ref{lemma:cut} we have
	        	$\jwfprop
		       {\pAnd{\wp{\p_1}}{(\pImply{\p_1}{\wp{\p_2}})}}
	           {\p_1}$.
	    \item
	        That is, $\jwfprop{\wp{\p}}{\p_1}$.
	    \item
	        Similarly, by Lemma~\ref{lemma:weaken} we have
	        $\jwfprop
	           {(\pAnd{\wp{\p_1}}{(\pImply{\p_1}{\wp{\p_2}})},\p_1,\wp{\p_2})}
	           {\p_2}$.
	    \item
	        Then since
	        $\jpf
	           {(\pAnd{\wp{\p_1}}{(\pImply{\p_1}{\wp{\p_2}})},\p_1)}
	           {\wp{\p_2}}$,
	    \item
	        by Lemma~\ref{lemma:cut} we have
	        $\jwfprop
	           {(\pAnd{\wp{\p_1}}{(\pImply{\p_1}{\wp{\p_2}})},\p_1)}
	           {\p_2}$.
	    \item
	        That is,
	        $\jwfprop
	           {\wp{\p},\p_1}
	           {\p_2}$.
	    \item
	        Therefore,
	        $\jwfprop{\wp{\p}}{\p}$.
       \end{enumerate}
  \end{itemize}
\end{proof}

\begin{lemma}
  \label{lemma:l42}
  \mbox{}
  \begin{enumerate}
  \item If $\wfterm{\D}{\t}{\I}$ then
    $\jwfterm{\pFalse}{\t}{\I}$.
  \item If $\wfprop{\D}{\p}$ then
    $\jwfprop{\pFalse}{\p}$.
  \end{enumerate}
\end{lemma}

\begin{lemma}
  \label{lemma:wfproofs}
\mbox{}
\begin{enumerate}
\item If ``$D$ is well-formed'' and $\jpf{\D}{\p}$ then
  $\jwfprop{\D}{\p}$.
\item If ``$D$ is well-formed'' and $\pf{\D}{\p}$ then
  $\jwfprop{\D}{\p}$.
\end{enumerate}
\end{lemma}

\begin{proof}
  To be checked.
\end{proof}

\begin{lemma}
  \label{lemma:l41}
  \mbox{}
  \begin{enumerate}
  \item If $\wfterm{\D}{\t}{\I}$ then
    $\jwfterm{\trans{\D}}{\trans{\t}}{\I}$.
  \item If $\wfprop{\D}{\p}$ then
    $\jwfprop{\trans{\D}}{\trans{\p}}$.
  \item If $\pf{\D}{\p}$ then
    $\jpf{\trans{\D}}{\trans{\p}}$.
  \end{enumerate}
\end{lemma}

\begin{proof}
\mbox{}

\begin{enumerate}
\item % Part 1
  \begin{itemize}
  \item Case: $\t = \x$ with $\x{:}\I\in\G$.
    \begin{enumerate}
    \item
      Here $\trans{\t} = x$.
    \item
      Then $\jwfterm{\trans{\D}}{\x}{\I}$.
    \end{enumerate}
  \item Case: $\t = \tFun{\f}{\t}{n}$ with $\wfterm{\D}{\t_i}{\I}$ for every $i\in1..n$.
    \begin{enumerate}
    \item
      Here $\trans{\t} = \f(\trans{\t_1},\ldots,\trans{\t_n})$.
    \item
      By the induction hypothesis, $\jwfterm{\trans{\D}}{\trans{\t_i}}{\I}$ for every $i\in1..n$.
    \item
      Thus, $\jwfterm{\trans{\D}}{\f(\trans{\t_1},\ldots,\trans{\t_n})}{\I}$.
    \end{enumerate}
  \item Case: $\t = \tDesc{\x}{\I}{\p(\x)}$ with
     $\pf{\D}{\pExists{\x}{\I}{\p(\x)}}$ and
     $\pf{\D}{\pForall{\x}{\I}{\pForall{\y}{\I}{\pImply{(\pAnd{\p(\x)}{\p(\y)})}{\pEqual{\x}{\y}}}}}$.
     \begin{enumerate}
     \item
       \label{lemma:l41:proofline:a2}
       By the inductive hypothesis,
     $\jpf{\trans{\D}}{\pExists{\x}{\I}{\pAnd{\wp{\p(\x)}}{\p(\x)}}}$
     \item
       and
       $\jpf{\trans{\D}}{\pForall{\x}{\I}{\pForall{\y}{\I}{\pImply{(\pAnd{\trans{\p(\x)}}{\trans{\p(\y)}})}{\pEqual{\x}{\y}}}}}$.
     \item
       FOR SOME REASON, ``$\D$ will be well-formed''.
     \item
       Hence by Lemma~\ref{lemma:wfproofs} we have
       $\jwfprop{\trans{\D}}{\pForall{\x}{\I}{\pForall{\y}{\I}{\pImply{(\pAnd{\trans{\p(\x)}}{\trans{\p(\y)}})}{\pEqual{\x}{\y}}}}}$.
     \item
       By inversion,
       $\jwfprop{\G,\x{:}\I,\y{:}\I}{\trans{\D}}{\trans{\p(\x)}}$
       where $\y$ does is not free in $\D$.
     \item
       By Lemma~\ref{lemma:subst} we have
       $\jwfprop{\G,\x{:}\I}{\trans{\D}}{\trans{\p(\x)}}$.
     \item
       \label{lemma:l41:proofline:a1}
       Thus, $\wfprop{\trans{\D}}{\pExists{\x}{\I}{\trans{\p(\x)}}}$.
\smallskip

     \item
       Now
       $\jpf{\G,\x{:}\I}{(\trans\D,\pAnd{\wp{\p(\x)}}{\trans{\p(\x)}})}{\trans{\p(\x)}}$
     \item
       and
       $\jwfterm{\G,\x{:}\I}{(\trans\D,\pAnd{\wp{\p(\x)}}{\trans{\p(\x)}})}{\x}{\I}$
     \item
       \label{lemma:l41:proofline:a3}
       so by existential introduction we have
       $\jpf{\G,\x{:}\I}{(\trans\D,\pAnd{\wp{\p(\x)}}{\trans{\p(\x)}})}{\pExists{\x}{\I}{\trans{\p(\x)}}}$.
     \item
       Then by existential elimination
       applied to lines~\ref{lemma:l41:proofline:a1}, \ref{lemma:l41:proofline:a2}, and \ref{lemma:l41:proofline:a3},
       we have
       $\jpf{\trans\D}{\pExists{\x}{\I}{\trans{\p(\x)}}}$.
     \end{enumerate}
  \end{itemize}
\item % Part 2
  \begin{itemize}
  \item Cases: $\p = \pTrue$ or $\p = \pFalse$.
    \begin{enumerate}
    \item
      Here $\trans{\p} = \p$
    \item
      and $\jpf{\trans\D}{\p}$.
    \end{enumerate}
  \item Case: $\p = \pOr{\p_1}{\p_2}$ with
    $\wfprop{(\D,\pFalse)}{\p_1}$ and
    $\wfprop{(\D,\pTrue)}{\p_2}$.
    \begin{enumerate}
    \item
      Here $\trans\p =
      \pOr{(\pAnd{\wp{\p_1}}{\trans{\p_1}})}{(\pAnd{\wp{\p_2}}{\trans{\p_2}})}$.
    \item
      By Lemma~\ref{lemma:l42},
      $\jwfprop{(\D,\pFalse)}{\p_1}$
    \item
      and
      $\jwfprop{(\D,\pTrue)}{\p_2}$.
    \item
      Then by Lemma~\ref{lemma:wfwp} we have
      $\jwfprop{\emptyCtx}{\wp{\p_1}}$
    \item
      and
      $\jwfprop{\emptyCtx}{\wp{\p_2}}$,
    \item
      and by Lemma~\ref{lemma:l9} we have
      $\jwfprop{\wp{\p_1}}{\trans{\p_1}}$
    \item
      and
      $\jwfprop{\wp{\p_2}}{\trans{\p_2}}$.
    \item
      Thus,
      $\jwfprop{\emptyCtx}{\pAnd{\wp{\p_1}}{\trans{\p_1}}}$
    \item
      and
      $\jwfprop{\emptyCtx}{\pAnd{\wp{\p_2}}{\trans{\p_2}}}$,
    \item
      and hence
      $\jwfprop{\emptyCtx}{\trans{\p}}$.
    \item
      The desired result then follows by Lemma~\ref{lemma:weaken}.
    \end{enumerate}
  \item Cases: $\p = \pAnd{\p_1}{\p_2}$ or $\p = \pImply{\p_1}{\p_2}$, with
    $\wfprop{\D}{\p_1}$ and $\wfprop{\D,\p_1}{\p_2}$.
    \begin{enumerate}
    \item
      By the inductive hypothesis we have
      $\jwfprop{\trans\D}{\trans{\p_1}}$
    \item
      and
      $\jwfprop{\trans\D,\trans{\p_1}}{\trans{\p_2}}$.
    \item
      Thus $\jwfprop{\trans\D}{\pAnd{\trans{\p_1}}{\trans{\p_2}}}$
      or $\jwfprop{\trans\D}{\pImply{\trans{\p_1}}{\trans{\p_2}}}$, as
      appropriate.
    \item
      That is,
      $\jwfprop{\trans\D}{\trans\p}$.
    \end{enumerate}
  \item Case: $\p = \pExists{\x}{\I}{\q}$, with
    $\wfprop{(\G,\x{:}\I)}{(\D,\pFalse)}{\q}$.
    \begin{enumerate}
   \item
      Here $\trans\p =
      \pExists{\x}{\I}{\pAnd{\wp{\q}}{\trans{\q}}}$.
    \item
      By Lemma~\ref{lemma:l42},
      $\jwfprop{(\D,\pFalse)}{\q}$.
    \item
      Then by Lemma~\ref{lemma:wfwp} we have
      $\jwfprop{\emptyCtx}{\wp{\q}}$
    \item
      and by Lemma~\ref{lemma:l9} we have
      $\jwfprop{\wp{\q}}{\trans{\q}}$.
    \item
      Thus
      $\jwfprop{(\G,\x{:}\I)}{\emptyCtx}{\pAnd{\wp{\q}}{\trans\q}}$,
    \item
      so
      $\jwfprop{\emptyCtx}{\pExists{\x}{\I}{\pAnd{\wp{\q}}{\trans\q}}}$.
    \item
      That is,
      $\jwfprop{\emptyCtx}{\trans{\p}}$.
    \item
      The desired result follows by Lemma~\ref{lemma:weaken}.
    \end{enumerate}
  \end{itemize}

\item % Part 3

\end{enumerate}
\end{proof}

\begin{claim}
\mbox{}
\begin{enumerate}
  \item If $\jwfprop{\D}{\p}$ then
    $\jpf{\D}{\pImply{\pAnd{\wp{\p}}{\trans{\p}}}{\p}}$.
  \item If $\jwfprop{\D}{\p}$ then
    $\jpf{\D}{\pImply{\p}{\pAnd{\wp{\p}}{\trans{\p}}}}$.
\end{enumerate}
\end{claim}

\begin{proof}
  \begin{enumerate}
  \item Should be easy induction, with the premise strictly stronger
    than the conclusion.
  \item This is the tricky part!
  \end{enumerate}
\end{proof}

\begin{claim}
\mbox{}
\begin{enumerate}
  \item If $\jwfprop{\D}{\p}$ and $\pf{\D}{\p}$ then
    $\jpf{\D}{\p}$.
\end{enumerate}
\end{claim}

\end{document}

%%% Local Variables:
%%% mode: latex
%%% TeX-master: t
%%% End:


\documentclass{article}

\usepackage[verbose=true,letterpaper]{geometry}
\usepackage[numbers]{natbib}

\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc}    % use 8-bit T1 fonts
\usepackage{hyperref}       % hyperlinks
\usepackage{url}            % simple URL typesetting
\usepackage{booktabs}       % professional-quality tables
\usepackage{amsfonts}       % blackboard math symbols
\usepackage{nicefrac}       % compact symbols for 1/2, etc.
\usepackage{microtype}      % microtypography

\usepackage{amsmath,amsbsy,amssymb,amsfonts,amsthm}
\usepackage{bm}
\usepackage{graphicx}
\usepackage{subcaption}
\usepackage{xspace}
\usepackage{color}
\usepackage{algorithm, algpseudocode}
\usepackage{wrapfig}
\def\compactify{\itemsep=0pt \topsep=0pt \partopsep=0pt \parsep=0pt}

\usepackage{setspace}
\usepackage{enumitem}

\newtheorem{theorem}{Theorem}
\newtheorem{definition}[theorem]{Definition}
\newtheorem{assumption}[theorem]{Assumption}
\newtheorem{remark}[theorem]{Remark}
\newtheorem{claim}[theorem]{Claim}
\newtheorem{proposition}[theorem]{Proposition}
\newtheorem{lemma}[theorem]{Lemma}
\newtheorem{corollary}[theorem]{Corollary}

\newcommand{\E}{\mathbb{E}}
\newcommand{\Var}{\mathrm{Var}}
\newcommand{\mat}[1]{\bm{\mathit{#1}}}
\algdef{SE}[VARIABLES]{States}{EndStates}
   {\algorithmicvariables}
   {\algorithmicend\ \algorithmicvariables}
\algnewcommand{\algorithmicvariables}{\textbf{States}}
\algrenewcommand\Return{\State \algorithmicreturn{} }
\newcommand*{\AddNote}[4]{
    \begin{tikzpicture}[overlay, remember picture]
        \draw [decoration={brace,amplitude=0.5em},decorate,ultra thick,red]
            ($(#3)!(#1.north)!($(#3)-(0,1)$)$) --
            ($(#3)!(#2.south)!($(#3)-(0,1)$)$)
                node [align=center, text width=2.5cm, pos=0.5, anchor=west] {#4};
    \end{tikzpicture}
}

\title{\tuner and the Art of Momentum Tuning}

\author{
  Jian Zhang, Ioannis Mitliagkas, Christopher R\'e \\
  Department of Computer Science\\
  Stanford University\\
  \texttt{\{zjian,imit,chrismre\}@cs.stanford.edu} \\
  %% examples of more authors
  %% \And
  %% Coauthor \\
  %% Affiliation \\
  %% Address \\
  %% \texttt{email} \\
  %% \AND
  %% Coauthor \\
  %% Affiliation \\
  %% Address \\
  %% \texttt{email} \\
  %% \And
  %% Coauthor \\
  %% Affiliation \\
  %% Address \\
  %% \texttt{email} \\
  %% \And
  %% Coauthor \\
  %% Affiliation \\
  %% Address \\
  %% \texttt{email} \\
}

\newcommand{\fix}{\marginpar{FIX}}
\newcommand{\new}{\marginpar{NEW}}

\newcommand{\tuner}{\textsc{YellowFin}\xspace}
\newcommand{\asynctuner}{closed-loop \textsc{YellowFin}\xspace}
\newcommand{\Asynctuner}{Closed-loop \textsc{YellowFin}\xspace}

\newcommand{\yell}[1]{#1}
\newcommand{\outline}[1]{}
\newcommand{\forjian}[1]{{\color{magenta}FOR JIAN: #1}}
\newcommand{\notes}[1]{{\color{green}NOTES: #1}}
\newcommand{\jianedits}[1]{#1}

\setlength{\parskip}{1.2ex}
\setlength{\parindent}{0pt}

\begin{document}

\maketitle

\begin{abstract}
\noindent Hyperparameter tuning is one of the big costs of deep learning.
State-of-the-art optimizers, such as Adagrad, RMSProp and  Adam,
make things easier by adaptively tuning an individual learning rate for each variable.
This level of fine adaptation is understood to yield a more powerful method.
However, our experiments, as well as recent theory by \citet{wilson2017marginal},
show that hand-tuned stochastic gradient descent (SGD) achieves better results, at the same rate or faster.
The hypothesis put forth is that adaptive methods converge to different minima \cite{wilson2017marginal}.
Here we point out another factor: none of these methods tune their momentum parameter,
known to be very important for deep learning applications \cite{sutskever2013importance}.
Tuning the momentum parameter becomes even more important in asynchronous-parallel systems:
recent theory \cite{mitliagkas2016asynchrony} shows that asynchrony introduces momentum-like dynamics,
and that tuning down algorithmic momentum is important for efficient parallelization.

We revisit the simple momentum SGD algorithm and show that hand-tuning a single learning rate and momentum value makes it competitive with Adam.
We then analyze its robustness in learning rate misspecification and objective curvature variation.
Based on these insights, we design \tuner, an automatic tuner for both momentum and learning rate in SGD.
\tuner optionally uses a novel momentum-sensing component along with a negative-feedback loop mechanism to compensate for the added dynamics of asynchrony on the fly.
We empirically show \tuner converges in fewer iterations than Adam on large ResNet and LSTM models,
	with a speedup up to $2.8$x in synchronous and $2.7$x in asynchronous settings.% on ResNet and LSTM models.
\end{abstract}

\section{Introduction}

\outline{[Problem.]}
Accelerated forms of stochastic gradient descent (SGD), pioneered by
\citet{polyak1964some} and \citet{nesterov1983method}, are the de-facto
training algorithms for deep learning.
Their use requires a sane choice for their {\em hyperparameters}:
typically a {\em learning rate} and {\em momentum parameter} \citep{sutskever2013importance}.
However, tuning hyperparameters is arguably the most time-consuming part of deep learning, with thousands of productive hours sacrificed and many papers outlining best tuning practices written
\cite{bengio2012practical,orr2003neural,bengio2012deep,bottou2012stochastic}.

\outline{[previous approach]}
Deep learning researchers have proposed a number of methods to deal with hyperparameter optimization.
Na\"ive methods, like the grid-search,
are prohibitively expensive for all but the smallest problems.
Smart black-box methods \cite{bergstra2012random,snoek2012practical}
do not explicitly take into account the problem specifics and spend time testing multiple configurations.
Adaptive methods provide an attractive alternative.
They aim to tune a single run on the fly and have been largely successful in relieving practitioners of tuning the learning rate.
Algorithms like Adagrad \cite{duchi2011adaptive}, RMSProp \cite{tieleman2012lecture} and Adam \cite{kingma2014adam} use the magnitude of gradient elements to tune learning rates {\em individually for each variable}.
\yell{
This increased flexibility sounds great,
however our experiments and recent analysis in literature \cite{wilson2017marginal} suggest that methods that adapt multiple learning rates, yield marginal benefits compared to momentum SGD.
\citet{wilson2017marginal} argue that those methods also have worse generalization.
We make another argument: adaptive methods also suffer because they do not tune their momentum parameter.
}

Momentum is a fundamental parameter at the heart of accelerated optimization,
lending its name to the most ubiquitous accelerated method \cite{polyak1964some}, commonly referred to as {\em momentum}.
Classic \cite{polyak1964some} and recent results \cite{sutskever2013importance} alike show that proper momentum tuning has a significant impact on training speed.
Momentum becomes even more critical on distributed systems.
Recently, \citet{mitliagkas2016asynchrony} showed that, in asynchronous-parallel systems---a popular design for efficient distributed training without synchronization
\cite{recht2011hogwild,dean2012large,chilimbi2014project,hadjis2016omnivore}---the system introduces momentum-like dynamics into the optimization.
As a result,
one should reduce the amount of algorithmic momentum to get fast convergence~\cite{hadjis2016omnivore}.
As part of a collaboration with an industry affiliate and a big research lab, we were able to verify that tuning momentum improves convergence on thousand-node scales.
However, grid-searching momentum on very large cluster jobs becomes especially challenging.
Better understanding of momentum and its rich properties is interesting in its own right and could yield a next generation of adaptive methods that perform {\em automatic momentum tuning}.

\begin{wrapfigure}[13]{R}{0.47\textwidth}
\vspace{-2.0em}
\begin{minipage}{1.0\linewidth}
\begin{figure}[H]
	\includegraphics[width=1.\linewidth]{experiment_results/spotlight.pdf}
	\caption{\tuner in comparison to Adam on a ResNet (CIFAR100, cf.\ Section~\ref{sec:experiments}).
	}
\end{figure}
\end{minipage}
\end{wrapfigure}
We revisit the basic SGD update that uses Polyak's momentum and a single learning rate for all variables.
We empirically show that, when hand-tuned, momentum SGD achieves faster convergence than Adam for a large class of models.
We then formulate the optimization update as a dynamical system and study certain robustness properties of the momentum operator.
Building on our analysis, we design \tuner, an automatic hyperparameter tuner for momentum SGD.
\tuner tunes the learning rate and momentum on the fly, and uses a novel {\em closed-loop control architecture} to compensate for the added dynamics of asynchrony.
Specifically:
\begin{itemize}[leftmargin=2em]
\setlength\itemsep{0.2em}
\item
Our analysis in Section~\ref{sec:momentum_operator} shows that momentum is robust to learning rate misspecification and curvature variation,
two desirable properties for deep learning.
These properties stem from a known but obscure fact:
the momentum operator's spectral radius is constant in a large subset of the hyperparameter space.
\item
In Section~\ref{sec:sync_tuner}, we use these robustness insights and a simple quadratic model analysis to design \tuner, an automatic tuner for momentum SGD.
\tuner uses on-the-fly measurements from the gradients of the system to tune both learning rate and momentum.
\item In Section~\ref{sec:async_tuner} we present \asynctuner, suited for asynchronous training.
It measures the total momentum in a running system, including any asynchrony-induced momentum.
This measurement is used in a negative feedback loop to control the value of algorithmic momentum.% on the fly.

\end{itemize}

\outline{[empirical performance statement on \tuner]}
In Section~\ref{sec:experiments}, we demonstrate empirically that \yell{on ResNets and LSTMs}
\tuner  converges in fewer iterations compared to:
(i) hand-tuned momentum SGD (a speedup of up to 2.2x);
and (ii) hand-tuned Adam (up to 2.8x speedup).
Under asynchrony, we demonstrate that state-of-the-art adaptive methods suffer due to lack of momentum tuning.
In the same setting, the closed-loop control architecture speeds up \tuner by up to 2x,
making it at least 2.7x faster than Adam.
We conclude with related work in Section~\ref{sec:related} and discussion in Section~\ref{sec:discussion}.
We release a TensorFlow implementation of \tuner, that can be used as drop-in replacement for any optimizer\footnote{Our TensorFlow implementation can be accessed at \url{https://github.com/JianGoForIt/YellowFin}.}.
We are also planning a PyTorch release.

\section{The momentum operator}
\label{sec:momentum_operator}

\newcommand{\gc}{generalized curvature\xspace}
\newcommand{\Gc}{Generalized curvature\xspace}

In this section we identify the main technical insights that guide the design of \tuner.
We first establish some preliminaries on momentum gradient descent
and then perform a simple dynamical analysis.
We show that momentum is robust to learning rate misspecification and curvature variation for a class of non-convex objectives, two important properties for deep learning.

\subsection{Preliminaries}
\label{sec:robust_preliminaries}
We aim to minimize some objective $f(x)$.
In machine learning, $x$ is referred to as {\em the model} and the objective is some {\em loss function}.
A low loss implies a well-fit model.
Gradient descent-based procedures use the gradient of the objective function, $\nabla f(x)$, to update the model iteratively.
Polyak's momentum gradient descent update \citep{polyak1964some} is given by
\begin{align}
	x_{t+1}  &= x_t - \alpha \nabla f(x_t) + \mu (x_t - x_{t-1}),
	\label{eqn:momentum_gd}
\end{align}
where $\alpha$ denotes the learning rate and $\mu$ the value of momentum used.
Momentum's main appeal is its established ability to {\em accelerate convergence} \cite{polyak1964some}.
On a strongly convex smooth function with condition number $\kappa$, the optimal convergence rate of gradient descent ($\mu=0$)
 is $O(\frac{\kappa-1}{\kappa+1})$~\cite{nesterov2013introductory}.
\yell{On the other hand, for certain classes of strongly convex and smooth functions, like quadratics,  the optimal momentum value,
\begin{equation}
	\mu^* = \left(\frac{\sqrt{\kappa}-1}{\sqrt{\kappa}+1}\right)^2,
	\label{eqn:optimal_momentum}
\end{equation}
yields the optimal accelerated rate $O(\frac{\sqrt{\kappa}-1}{\sqrt{\kappa}+1})$}.\footnote{
\yell{
This guarantee does not generalize to arbitrary strongly convex functions \cite{lessard2016analysis}.
Nonetheless, acceleration is typically observable in practice (cf. Section~\ref{sec:robust_properties}).
}
}
This is the smallest value of momentum that {\bf ensures the same rate of convergence along all directions}.
This fact is often hidden away in proofs.
We shed light on some of its previously unknown implications in Section~\ref{sec:robust_properties}. % and use them in our tuner in Section~\ref{sec:sync_tuner}.

\subsection{Robustness properties of the momentum operator}
\label{sec:robust_properties}
In this section we analyze the dynamics of momentum on a simple class of one dimensional, non-convex objectives.
We first introduce the notion of {\em generalized curvature} and use it to describe the momentum operator.
Then we discuss some properties of the momentum operator.

\begin{definition}[\Gc]
\label{def:generalized_curvature}
The derivative of $f(x):\mathbb{R}\rightarrow\mathbb{R}$, can be written as
\begin{equation}
	 f'(x) = h(x) (x - x^*)
	\label{eqn:generalized_curvature}
\end{equation}
for some $h(x) \in \mathbb{R}$, where $x^*$ is the global minimum of $f(x)$.
We call $h(x)$ the {\em generalized curvature}.
\end{definition}

The \gc describes, in some sense, curvature with respect to the optimum,  $x^*$.
For quadratic objectives, it coincides with the standard definition of curvature.
It is the sole quantity related to the objective that influences the dynamics of gradient descent.
For example, the contraction of a gradient descent step is $1-\alpha h(x_t)$.
Let $\mat{A}_t$ denote the {\em momentum operator} at time $t$.
Using a state-space augmentation, we can express the momentum update as
\begin{align}
{\begin{pmatrix}
x_{t+1} - x^*\\
x_t - x^* \\
\end{pmatrix}}
=
{\begin{bmatrix}
1-\alpha h(x_t) + \mu & - \mu \\
1 & 0 \\
\end{bmatrix}}
{\begin{pmatrix}
x_t - x^* \\
x_{t-1} - x^*\\
\end{pmatrix}}
\triangleq
\mat{A}_t
{\begin{pmatrix}
x_t - x^* \\
x_{t-1} - x^*\\
\end{pmatrix}}.
\label{equ:one_dim_22_rec}
\end{align}

\begin{lemma}[Robustness of the momentum operator]
\label{lem:robustness}
If the generalized curvature, $h$, and hyperparameters $\alpha,\mu$ are in the {\em robust region}, that is:
\begin{align}
{(1-\sqrt{\mu})^2} &\leq \alpha h(x_t) \leq {(1+\sqrt{\mu})^2},
\label{eqn:robust_region}
\end{align}
then the spectral radius of the momentum operator only depends on  momentum: $	\rho(\mat{A}_t) = \sqrt{\mu}
$.
\end{lemma}
We can explain Lemma~\ref{lem:robustness} as robustness with respect to learning rate and to variations in curvature.

\begin{wrapfigure}[15]{R}{0.3\textwidth}
\vspace{-2.5em}
\begin{minipage}{1.0\linewidth}
\begin{figure}[H]
  \includegraphics[width=1.0\linewidth]{figures/spectral_radii}
\caption{
Spectral radius of momentum operator on scalar quadratic.
}
\label{fig:lr_robustness}
\end{figure}
\end{minipage}
\end{wrapfigure}
\paragraph{Momentum is robust to learning rate misspecification}
\label{sec:lr_robustness}
For a one dimensional strongly convex quadratic objective,
we get $h(x)=h$ for all $x$ and Lemma~\ref{lem:robustness} suggests that $\rho(\mat{A}_t)=\sqrt{\mu}$ as long as
\begin{align}
{(1-\sqrt{\mu})^2/h} &\leq \alpha \leq {(1+\sqrt{\mu})^2/h}.
\label{eqn:lr_robustness}
\end{align}

In Figure~\ref{fig:lr_robustness}, we plot $\rho(\mat{A}_t)$ for different $\alpha$ and $\mu$.
As we increase the value of momentum, the optimal rate of convergence $\sqrt{\mu}$ is achieved by an ever-widening range of learning rates.
Furthermore, for objectives with large condition number, higher values of momentum are {\em both faster and more robust}.
{\bf This property influences the design of our tuner:} as long as the learning rate satisfies \eqref{eqn:lr_robustness}, we are in the {\em robust region} and
expect the same asymptotics. For example a convergence rate of $\sqrt{\mu}$ for quadratics, independent of the learning rate.
Having established that, we can just focus on optimally tuning momentum.

\paragraph{Momentum is robust to curvature variation}
\label{sec:curvature_robustness}

As discussed in Section~\ref{sec:robust_preliminaries}, the intuition hidden in classic results
is that for strongly convex smooth objectives, the momentum value in \eqref{eqn:optimal_momentum} guarantees the same rate of convergence along all directions.
We extend this intuition to certain non-convex functions.
Lemma~\ref{lem:robustness} guarantees a constant, time-homogeneous spectral radius for the momentum operators $(\mat{A}_t)_t$ if
\eqref{eqn:robust_region} is satisfied at every step.
This motivates an extension of the condition number.
\begin{definition}[Generalized condition number]
We define the generalized condition number (GCN) of a scalar function, $f(x):\mathbb{R}\rightarrow \mathbb{R}$, to be the dynamic range of its generalized curvature, $h(x)$:
\begin{equation}
	\nu = \frac{\sup_{x \in dom(f)} h(x)}{ \inf_{x \in dom(f)} h(x)}
\end{equation}
\end{definition}
The GCN captures variations in generalized curvature along a scalar slice.
From Lemma~\ref{lem:robustness} we get
\begin{equation}
	\mu^* = \left(\frac{\sqrt{\nu}-1}{\sqrt{\nu}+1}\right)^2,
	\quad
	\alpha^* = \frac{(1-\sqrt{\mu})^2}{\inf_{x \in dom(f)}h(x)}
	\label{eqn:noiseless_tuning_rule}
\end{equation}
as the optimal hyperparameters.
Specifically, $\mu^*$ is the smallest momentum value that guarantees a homogeneous spectral radius of $\sqrt{\mu^*}$ for all $(\mat{A}_t)_t$.
The spectral radius of an operator describes its asymptotic behavior.
However, the product of a sequence of operators $\mat{A}_t\cdots\mat{A}_1$ all with spectral radius $\sqrt{\mu}$ does not necessarily follow the asymptotics of $\sqrt{\mu}^t$.
In other words, {\em we do not provide a convergence rate guarantee}.
Instead, we provide empirical evidence in support of this intuition.

For example, the non-convex objective in Figure~\ref{fig:curvature_robustness}(a),
composed of two quadratics with curvatures $1$ and $1000$, has a GCN of $1000$.
Using the tuning rule of \eqref{eqn:noiseless_tuning_rule}, and running the momentum algorithm (Figure~\ref{fig:curvature_robustness}(b)) yields a practically constant rate of convergence throughout.
In Figures~\ref{fig:curvature_robustness}(c,d) we demonstrate that for an LSTM,
the majority of variables follows a $\sqrt{\mu}$ convergence rate.
\textbf{This property influences the design of our tuner:}
in the next section we use the tuning rules of \eqref{eqn:noiseless_tuning_rule} in \tuner,
generalized appropriately to handle SGD noise.

\begin{figure}[t]
\centering
\begin{tabular}{c c c c}
  \includegraphics[width=0.23\linewidth]{figures/non_convex_toy} &
  \includegraphics[width=0.24\linewidth]{figures/non_convex_constant_rate} &
  \includegraphics[width=0.24\linewidth]{figures/constant_rate_09} &
  \includegraphics[width=0.19\linewidth]{figures/constant_rate_099} \\
  (a) & (b) & (c) &(d)
\end{tabular}
\caption{(a) Non-convex toy example;
(b) constant convergence rate achieved empirically on the objective of (a) tuned according to \eqref{eqn:noiseless_tuning_rule};
(c,d)
LSTM on MNIST: as momentum increases, more variables (shown in grey) fall in the robust region and follow the robust rate, $\sqrt{\mu}$.}
\label{fig:curvature_robustness}
\end{figure}

\input{tuner}

\section{\Asynctuner}
\label{sec:async_tuner}
To handle the momentum dynamics of asynchronous parallelism, we propose a {\em closed momentum loop} variant of \tuner.
After some preliminaries, we show the mechanism of the extension:
it measures the dynamics on a running system and controls momentum with a negative feedback loop.
\paragraph{Preliminaries}
Asynchrony is a popular parallelization technique \cite{recht2011hogwild} that avoids synchronization barriers.
When training on $M$ asynchronous workers, staleness (the number of model updates between a worker's read and write operations) is on average $\tau=M-1$,
i.e., the gradient in the SGD update is delayed by $\tau$ iterations as $\nabla f_{S_{t - \tau}}(x_{t - \tau} )$.
Asynchrony yields faster steps, but can
increase the number of iterations to achieve the same solution,
a tradeoff between hardware and statistical
efficiency~\citep{DBLP:journals/pvldb/ZhangR14}.
\citet{mitliagkas2016asynchrony} interpret asynchrony as added momentum dynamics.
Experiments in \citet{hadjis2016omnivore} support this finding, and demonstrate that reducing algorithmic momentum can compensate for asynchrony-induced momentum
and significantly reduce the number of iterations for convergence.
Motivated by that result, we use the model
in~\eqref{equ:exp_async_update}, where the total momentum, $\mu_T$, includes both asynchrony-induced and algorithmic  momentum, $\mu$, in~\eqref{eqn:momentum_gd}.
\begin{equation}
	\mathbb{E}[ x_{t+1} - x_t ]
	= \mu_T \mathbb{E}[x_t - x_{t-1}] - \alpha \mathbb{E}\nabla f(x_{t})
\label{equ:exp_async_update}
\end{equation}
We will use this expression to design an estimator for the value of total momentum, $\hat{\mu_T}$.
This estimator is a basic building block of \asynctuner, that {\em removes the need to manually compensate for the effects of asynchrony}.

\paragraph{Measuring the momentum dynamics}
\Asynctuner estimates total momentum $\mu_{T}$ on a running system and uses a negative feedback loop to adjust algorithmic momentum accordingly.
Equation~\eqref{equ:exp_async_update} gives an estimate of $\hat{\mu_T}$ on a system with staleness $\tau$, based on \eqref{equ:exp_async_update}.
\begin{align}
\hat{\mu_T}
					= \mathop{\mathsf{median}}\left(
							\frac{x_{t - \tau} - x_{t - \tau-1} + \alpha \nabla_{S_{t-\tau -1}} f(x_{t - \tau - 1} )}
							{x_{t - \tau-1} - x_{t - \tau-2}}
					\right)
\label{eqn:momentum_measurement}
\end{align}
We use $\tau$-stale model values to match the staleness of the gradient,  and perform all operations in an elementwise fashion.
This way we get a total momentum measurement from each variable;
the median combines them into a more robust estimate.

\label{subsec:closed_loop_YF}
\begin{figure}
\centering
\includegraphics[width=0.95\linewidth]{experiment_results/resnet/mom_dynamic_3.pdf}
	\caption{
	Momentum dynamics on CIFAR100 ResNet.
	Running \tuner, total momentum is equal to algorithmic momentum in a synchronous setting (left). Total momentum is greater than algorithmic momentum on 16 asynchronous workers, due to asynchrony-induced momentum (middle).
	Using the momentum feedback mechanism of \asynctuner, lowers algorithmic momentum and brings total momentum to match the target value on 16 asynchronous workers (right).
	Red dots are individual total momentum estimates, $\hat{\mu}_T$, at each iteration.
The solid red line is a running average of those estimates.
	}
	\label{fig:we-can-measure}
\end{figure}

\paragraph{Closing the asynchrony loop}
Given a reliable measurement of $\mu_{T}$,
we can use it to adjust the value of algorithmic momentum so that the total momentum matches the \emph{target momentum} as decided by \tuner in Algorithm~\ref{alg:basic-algo}.
\Asynctuner in Algorithm~\ref{alg:async-algo} %(in Appendix~\ref{sec:async_yf})
uses a simple negative feedback loop to achieve the adjustment.
Figure~\ref{fig:we-can-measure} demonstrates that under asynchrony the measured total momentum is strictly higher than the algorithmic momentum (middle plot), as expected from theory;
closing the feedback loop (right plot) leads to total momentum matching the target momentum.
Closing the loop, as we will see, improves performance significantly.
Note for asynchronous-parallel training, as the estimates and parameter tuning is unstable in the beginning when there are only a small number of iterations, we use initial learning $\frac{1}{\tau + 1}$ instead of $1.0$ to prevent overflow in the beginning.
\begin{algorithm}[H]
	\caption{\Asynctuner}
	\begin{algorithmic}[1]
	\State Input: $\mu\gets0$, $\alpha \gets \frac{1}{\tau + 1}$, $\gamma\gets0.01, \tau$ (staleness)
	\For { $t\gets1$ to $T$}
	\State $x_t\!\gets\!x_{t - 1} + \mu (x_{t - 1} - x_{t - 2} ) - \alpha \nabla_{S_t} f(x_{t - \tau - 1} )$
	\State $\mu^*,\alpha \gets \Call{\tuner}{\nabla_{S_t} f(x_{t - \tau - 1} ), \beta}$ %(get momentum from the dynamic range)
	\State $\hat{\mu_T}
					\gets \mathop{\mathsf{median}}\left(
							\frac{x_{t - \tau} - x_{t - \tau-1} + \alpha \nabla_{S_{t-\tau-1}} f(x_{t - \tau - 1} )}
							{x_{t - \tau-1} - x_{t - \tau-2}}
					\right)$ \Comment{Measuring total momentum}
	\State $\mu \leftarrow \mu + \gamma \cdot (\mu^* - \hat{\mu_T})$ \Comment{Closing the loop}
	\EndFor
\end{algorithmic}
\label{alg:async-algo}
\end{algorithm}

\input{experiments.tex}

\vspace{-0.25em}
\section{Related work}
\label{sec:related}
\vspace{-0.3em}
Many techniques have been proposed on tuning hyperparameters for optimizers.~\citet{bergstra2012random} investigate random search for general tuning of  hyperparameters.
Bayesian approaches~\cite{snoek2012practical} model evaluation metrics as samples from a Gaussian process guiding optimal hyperparameter search.
Another trend is the adaptive methods which require less manual tuning than SGD:
Adagrad~\cite{duchi2011adaptive} is one of the first method with per-dimension learning rate, followed by RMSProp~\cite{tieleman2012lecture} and Adam~\cite{chilimbi2014project} using different learning rate rules.
\citet{schaul2013no} use a noisy quadratic model similar to ours,
however they tune per-variable learning rates and
do not use momentum.

\vspace{-0.25em}
\section{Discussion}
\label{sec:discussion}
\vspace{-0.3em}
We presented \tuner, the first optimization method that automatically tunes momentum as well as the learning rate of momentum SGD.
\tuner outperforms the state-of-the-art in optimizing a large class of models both in synchronous and asynchronous settings.
It estimates statistics purely from the gradients of a running system,
and then tunes the hyperparameters of momentum SGD based on noisy, local quadratic approximations.
As future work, we believe that more accurate curvature estimation methods,
like the $bbprop$ method~\cite{martens2012estimating} can further improve \tuner.
We also believe that our closed-loop momentum control mechanism in Section~\ref{sec:async_tuner}
could be applied to other adaptive methods and improve performances in asynchronous-parallel settings.

\section{Acknowledgements}
We thank Bryan He, Paroma Varma, Chris De Sa and Theodoros Rekatsinas for helpful discussions. We gratefully acknowledge the support of the Defense Advanced Research Projects Agency (DARPA) SIMPLEX program under No. N66001-15-C-4043, the D3M program under No. FA8750-17-2-0095, the National Science Foundation (NSF) CAREER Award under No. IIS- 1353606, the Office of Naval Research (ONR) under awards No. N000141210041 and No. N000141310129, a Sloan Research Fellowship, the Moore Foundation, an Okawa Research Grant, Toshiba, and Intel. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of DARPA, NSF, ONR, or the U.S. government.

\bibliographystyle{unsrtnat}
\bibliography{arxiv}

\appendix
\input{main_lemma}
%\input{multi_dim}
%\input{dist_small_lstm}
\input{opt}
%\input{async_yf}
\input{model_spec}
\input{exp_spec}
%\input{test_perp}

\end{document}