% In this section we describe how \minesp works. To do so, we first introduce relevant definitions, followed by an overview of the algorithm. We subsequently discuss each component of the algorithm in detail.

% To describe how \minesp works, we first formally define time series and provide an overview of the algorithm. We subsequently discuss each component of the algorithm in detail.
To describe how \minesp works, we first provide an overview of the algorithm, then discuss each of its component in detail.

% % ------------------------------------------------
% \subsection{Definitions}
% % ------------------------------------------------
% % To both establish notation and clarify the sorts of data for which \minesp is applicable, we introduce the following definitions.
% % To facilitate a precise explanation of \minesp, we introduce the following definitions.
% \begin{Definition} \b{Sample.} A sample is a vector $\x \in \R^D$. $D$ is termed the sample's \b{dimensionality}. Each element of the sample is an integer represented using a number of bits $w$, termed the \b{bitwidth}. The bitwidth $w$ is shared by all elements.
% \end{Definition}
% \begin{Definition} \b{Time Series.} A time series $\X$ of length $T$ is a sequence of $T$ samples, $\x_1,\ldots,\x_T$. All samples $\x_t$ share the same bitwidth $w$ and dimensionality $D$. If $D = 1$, $\X$ is called \b{univariate}; otherwise it is \b{multivariate}.
% % , and the meaning of each dimension is consistent from sample to sample.
% \end{Definition}
% \begin{Definition} \b{Rows, Columns.} In a database context, we assume that each sample of a time series is one row and each dimension is one column. Because data arrives as samples and memory constraints may limit how many samples can be buffered, we assume that the data is stored in row-major order---i.e., such that each sample is stored contiguously.
% \end{Definition}

% ------------------------------------------------
\subsection{Overview}
% ------------------------------------------------

\minesp is a bit packing-based predictive coder. It consists of four components:
\begin{enumerate}
% \itemsep0em
\item \b{Forecasting.} \minesp employs a forecaster to predict each sample based on previous samples. It encodes the difference between the next sample and the predicted sample, which is typically closer to zero than the next sample itself.
\item \b{Bit packing.} \minesp then bit packs the errors as a ``payload'' and prepends a header with sufficient information to invert the bit packing.
\item \b{Run-length encoding.} If a block of errors is all zeros, \minesp waits for a block in which some error is nonzero and then writes out the number of all-zero blocks instead of the (otherwise empty) payload.
\item \b{Entropy coding.} \minesp Huffman codes the headers and payloads.
\end{enumerate}

These components are run on blocks of eight samples (motivated in Section~\ref{sec:bitpacking}), and can be modified to yield different compression-speed tradeoffs. Concretely, one can 1) skip entropy coding for greater speed and 2) choose between delta coding and our online learning method as forecasting algorithms. The latter is slightly slower but often improves compression.

We chose these steps since they allow for high speed and exploit the characteristics of time series. Forecasting leverages the high correlation of successive samples to reduce the entropy of the data. Run-length encoding allows for extreme compression in the (common) scenario that there is no change in the data---e.g., a user's smartphone may be stationary for many hours while the user is asleep. Our method of bit packing exploits temporal correlation in the variability of the data by using the same bitwidth for points that are within the same block. Huffman coding is not specific to time series but has low memory requirements and improves compression ratios.

 % in general is not especially geared towards time series, but our bit packing scheme exploits temporal correlation in the variability of the data by using the same bitwidth for points that are within the same block. Huffman coding is not specific to time series but has low memory requirements and improves compression ratios.

 % \textit{per se}, but provides a vectorizable and low-memory means of reducing the encoding size. Similarly, Huffman coding is not specific to time series but has low memory requirements and improves compression ratios.

% Before describing the four components in greater detail, we first provide an outline of the overall algorithms for compressing and decompressing blocks of data.

% We elaborate upon each of the four components in the following subsections, but first provide an outline of the overall compression and decompression functions.

\newcommand{\err}{\texttt{err}}
\newcommand{\nbits}{\texttt{nbits}}
\newcommand{\packed}{\texttt{packed}}
\newcommand{\buff}{$\texttt{buff}$}
\newcommand{\bytes}{\texttt{bytes}}
% \newcommand{\header}{\texttt{header}}
\newcommand{\payload}{\texttt{payload}}
\newcommand{\f}{\texttt{f}}
\newcommand{\fore}{\texttt{forecaster}}
\newcommand{\self}{\texttt{self}}

% ------------------------ compress overview

An overview of how \minesp compresses one block of samples is shown in Algorithm~\ref{algo:compress}. In lines \ref{line:bodyStart}-\ref{line:encPredictEnd}, \minesp predicts each sample based on the previous sample and any state stored by the forecasting algorithm. For the first sample in a block, the previous sample is the last element of the previous block, or zeros for the initial block. In lines \ref{line:eachColStart}-\ref{line:bodyEnd}, \minesp determines the number of bits required to store the largest error in each column and then bit packs the values in that column using that many bits. (Recall that each column is one variable of the time series). If all columns require 0 bits, \minesp continues reading in blocks until some error requires $>$0 bits (lines \ref{line:rleLoopStart}-\ref{line:rleLoopEnd}). At this point, it writes out a header of all 0s and then the number of all-zero blocks. Finally, it writes out the number of bits required by each column in the latest block as a header, and the bit packed data as a payload. Both header and payload are compressed with Huffman coding.

\begin{algorithm}[h]
% \caption{encodeBlock($\{\x_1, \ldots, \x_B \}, \vtheta $)}
\caption{encodeBlock($\{\x_1, \ldots, \x_B \}, \fore$)}
\label{algo:compress}
\begin{algorithmic}[1]

\State{Let \buff\sp be a temporary buffer}

\For {$i \leftarrow 1,\ldots,B$} \COMMENTT {For each sample} \label{line:bodyStart}
    % \State{$ \hat{\x}_i, \vtheta \leftarrow $predictAndTrain$(\x_{i-1}, \vtheta)  $}
    \State{$ \hat{\x}_i \leftarrow $ $\fore$.predict$(\x_{i-1})$}
    \State{$ \err_i \leftarrow \x_i - \hat{\x}_i  $}
    \State{$\fore$.train($\x_{i-1}$, $\x_i$, $\err_i$)} \label{line:encPredictEnd}
\EndFor
\For {$j \leftarrow 1,\ldots,D$} \COMMENTT {For each column} \label{line:eachColStart}
    \State{$ \nbits_j \leftarrow \max_i\{ $requiredNumBits$(\err_{ij}) \} $}
    \State{$ \packed_j \leftarrow $ bitPack$(\{\err_{1j},\ldots,\err_{Bj} \},\text{ }\nbits_j) $}  \label{line:bodyEnd}
\EndFor

\LineComment{Run-length encode if all errors are zero}
\If{$\nbits_j$ \texttt{==} $0$, $1 \le j \le D$}
    \Repeat  \COMMENT{Scan until end of run} \label{line:rleLoopStart}
        \State{Read in another block and run lines \ref{line:bodyStart}-\ref{line:bodyEnd} }
    \Until {$\exists_j [\nbits_j \neq 0 ]$} \label{line:rleLoopEnd}
    % \State{Read in another block and run lines \ref{line:bodyStart}-\ref{line:bodyEnd} until some nbits$_j$ != $0$}
    \State{Write $D$ $0$s as headers into \buff}
    \State{Write number of all-zero blocks as payload into \buff}
    \State{Output huffmanCode(\buff)}
\EndIf

\State{Write $\nbits_j$, $j = 1,\ldots,D$ as headers into \buff}
\State{Write $\packed_j$, $j = 1,\ldots,D$ as payload into \buff}
\State{Output huffmanCode(\buff)}

% \RETURN {$ odds_{event} - \max(odds_{noise}, odds_{next}) $}
\end{algorithmic}
\end{algorithm}

% ------------------------ decompress

\minesp begins decompression (Algorithm~\ref{algo:decomp}) by decoding the Huffman-coded bitstream into a header and a payload. Once decoded, these two components are easy to separate since the header is always first and of fixed size. If the header is all 0s, the payload indicates the length of a run of zero errors. In this case, \minesp runs the predictor until the corresponding number of samples have been predicted. Since the errors are zero, the forecaster's predictions are the true sample values. In the nonzero case, \minesp unpacks the payload using the number of bits specified for each column by the header.

\begin{algorithm}[h]
% \caption{decodeBlock(\bytes, $B$, $D$, $\vtheta$)}
\caption{decodeBlock(\bytes, $B$, $D$, $\fore$)}
\label{algo:decomp}
\begin{algorithmic}[1]

\State{$\nbits$, $\payload$ $\leftarrow$ huffmanDecode(\bytes, $B$, $D$) }

% \State{$\f \leftarrow \texttt{Forecaster()} $}
\If{$\nbits_j$ \texttt{==} $0$ $\forall j$} \COMMENT{Run-length encoded}
    \State{$\texttt{numblocks} \leftarrow $ readRunLength()}
    \For {$i \leftarrow 1,\ldots,(B $ $\cdot$ \texttt{numblocks})}
        % \State{$ \hat{\x}_i, \vtheta \leftarrow $ $\fore$.predict$(\x_{i-1}, \vtheta) $}
        \State{$ \x_i \leftarrow $ $\fore$.predict$(\x_{i-1})$}
        \State{Output $\x_i$}
        % \State{$\fore$.train($\x_{i-1}$, $\x_i$, 0)}
    \EndFor
    % \RETURN {}
% \EndIf
\Else \COMMENT{Not run-length encoded}
% \State{}
\For {$i \leftarrow 1,\ldots,B$}
    % \State{$ \hat{\x}_i, \vtheta \leftarrow $ $\fore$.predict$(\x_{i-1}, \vtheta)  $}
    \State{$ \hat{\x}_i \leftarrow $ $\fore$.predict$(\x_{i-1})$}
    \State{$ \err_i \leftarrow $unpackErrorVector$(i$, \nbits, \payload$) $}
    \State{$ \x_i \leftarrow \err_i + \hat{\x}_i  $}
    \State{Output $\x_i$}
    \State{$\fore$.train($\x_{i-1}$, $\x_i$, $\err_i$)}
\EndFor
\EndIf
\end{algorithmic}
\end{algorithm}

% ------------------------------------------------
\subsection{Forecasting}
% ------------------------------------------------

\minesp forecasting can use either delta coding or \justfire \text{ } (Fast Integer REgression), a novel online forecasting algorithm we introduce.

% ================================
\subsubsection{Delta Coding}
% ================================

Forecasting with delta coding consists of predicting each sample $\x_i$ to be equal to the previous sample $\x_{i-1}$, where $\x_{0} \triangleq \vec{0}$. This method is stateless given $\x_{i-1}$ and is extremely fast. It is particularly fast when combined with run-length encoding, since it yields a run of zero errors if and only if the data is constant. This means that decompression of runs requires only copying a fixed vector, with no additional forecasting or training. Moreover, when answering queries, one can sometimes avoid decompression entirely---e.g., one can compute the max of all samples in the run by computing the max of only the first value.

% ================================
\subsubsection{FIRE}
% ================================

Forecasting with \fire is slightly more expensive than delta coding but often yields better compression.
The basic idea of \fire is to model each value as a linear combination of a fixed number of previous values and learn the coefficients of this combination. Specifically, we learn an autoregressive model of the form:
% Consider a linear model for predicting a given value $x_i$ of some variable at time step $i$.
\begin{align}
    x_i = a x_{i-1} + b x_{i-2} + \eps_i
\end{align}
where $x_i$ denotes the value of some variable at time step $i$ and $\eps_i$ is a noise term.

Different values of $a$ and $b$ are suitable for different data characterisics. If $a = 2$, $b = -1$, we obtain double-delta coding, which extrapolates linearly from the previous two points and works well when the time series is smooth. If $a = 1$, $b = 0$, we recover delta coding, which models the data as a random walk. If $a = \frac{1}{2}$, $b = \frac{1}{2}$, we predict each value to be the average of the previous two values, which is optimal if the $x_i$ are i.i.d. Gaussians. In other words, these cases are appropriate for successively noisier data.

% The idea of \fire is to learn online what the best coefficients are.

The reason \fire is effective is that it learns online what the best coefficients are for each variable.
% \Fire learns appropriate coefficients online using gradient descent with L1 loss.
% To make prediction and learning as efficient as possible, however, \justfire
To make prediction and learning as efficient as possible, \justfire \text{} restricts the coefficients to lie within a useful subspace. Specifically, we exploit the observation that all of the above cases can be written as:
\begin{align}
    x_i = x_{i-1} + \alpha x_{i-1} - \alpha x_{i-2} + \eps_i
\end{align}
for $\alpha \in [-\frac{1}{2}, 1]$. Letting $\delta_i \triangleq x_i - x_{i-1}$ and subtracting $x_{i-1}$ from both sides, this is equivalent to
% \begin{align}
%     x_i = x_{i-1} + \delta_i + \alpha (x_{i-1} - x_{i-2}) + \eps_i
% \end{align}
% . Subtracting $x_{i-1}$ from both sides, we obtain
\begin{align}
    \delta_i &= \alpha \delta_{i-1} + \eps_i
\end{align}
% \begin{align}
%     \delta_i &= \delta_{i-1} + \alpha  + \eps_i \\
% \end{align}
% and therefore
% \begin{align}
%     \delta_i &= \alpha \delta_{i-1} \eps_i \\
% \end{align}
This means that we can capture all of the above cases by predicting the next delta as a rescaled version of the previous delta. This requires only a single addition and multiplication, and reduces the learning problem to that of finding a suitable value for a single parameter.

% % ------------------------------------------------
% \subsubsection{Details}
% % ------------------------------------------------
% Pseudocode for \fire prediction and training is given in Algorithm~\ref{algo:xff}.

% To make this underlying model practical and performant for integers,
% : \fire initialization (line~\ref{line:xffCtor}), prediction (line~\ref{line:xffPredict}), and training (line~\ref{line:xffTrain}).
To train and predict using this model, we use the functions shown in Algorithm~\ref{algo:xff}. First, to initialize a \fire forecaster, one must specify three values: the number of columns $D$, the learning rate $\eta$, and the bitwidth $w$ of the integers stored in the columns. Internally, the forecaster also maintains an accumulator for each column (line~\ref{line:counter}) and the difference (delta) between the two most recently seen samples (line~\ref{line:deltas}). The accumulator is a scaled version of the current $\alpha$ value with a bitwidth of $2w$. It enables fast updates of $\alpha$ with greater numerical precision than would be possible if modifying $\alpha$ directly. The accumulators and deltas are both initialized to zeros. % updates without the expense of division or multiplication instructions.

% ------------------------ xff pseudocode

\begin{algorithm}[h]
% \begin{struct}[h]
% \label{algo:xff}
% \floatname{algorithm}{Algorithm}
\caption{FIRE\_Forecaster Class} \label{algo:xff}
\begin{algorithmic}[1]

\Function{Init}{$D$, $\eta$, $w$} \label{line:xffCtor}
\State $\self$.learnShift $\leftarrow \lg(\eta)$
\State $\self$.bitWidth $\leftarrow w$ \COMMENT{8-bit or 16-bit}
\State $\self$.accumulators $\leftarrow $ zeros($D$) \label{line:counter}
\State $\self$.deltas $\leftarrow $ zeros($D$) \label{line:deltas}
\EndFunction

\Function{Predict}{$\x_{i-1}$} \label{line:xffPredict}
\State $\texttt{alphas} \leftarrow \self$.accumulators \rshift $\self$.learnShift
% \State $\texttt{alphas} \leftarrow (\texttt{alphas} \text{ }\rshift 4) \text{ }\lshift 4$
\State $\hat{\vdelta} \leftarrow$ (\texttt{alphas} $\odot$ $\self$.deltas) $\rshift \self$.bitWidth
\RETURN $\x_{i-1} + \hat{\vdelta}$
\EndFunction

\Function{Train}{$\x_{i-1}$, $\x_{i}$, $\err_i$} \label{line:xffTrain}
\State $\texttt{gradients} \leftarrow {-\sign(\err_i)} \odot \self$.deltas
% \State $\self$.accumulators $\leftarrow \self$.accumulators + $\texttt{gradients}$
% \State $\self$.accumulators \texttt{-=} $\texttt{gradients}$
\State $\self$.accumulators $\leftarrow$ $\self$.accumulators $-$ $\texttt{gradients}$
\State $\self$.deltas $\leftarrow \x_{i} - \x_{i-1}$
\EndFunction

\end{algorithmic}
\end{algorithm}

% To predict (line~\ref{line:xffPredict})
To predict, the forecaster first derives the coefficient $\alpha$ for each column based on the accumulator. By right shifting the accumulator $\log2(\eta)$ bits, the forecaster obtains a learning rate of $2^{-\log2(\eta)} = \eta$. It then estimates the next deltas as the elementwise product (denoted $\odot$) of these coefficients and the previous deltas. It predicts the next sample to be the previous sample plus these estimated deltas.

Because all values involved are integers, the multiplication is done using twice the bitwidth $w$ of the data type---e.g., using 16 bits for 8 bit data. The product is then right shifted by an amount equal to the bit width. This has the effect of performing a fixed-point multiplication with step size equal to $2^{-w}$.

% The forecaster trains (line~\ref{line:xffTrain})
The forecaster trains by performing a gradient update on the L1 loss between the true and predicted samples. I.e., given the loss:
\begin{align}
    \mathcal{L}(x_i, \hat{x}_{i}) = \abs{x_i - \hat{x}_i}
    = \abs{x_i - (x_{i-1} + \frac{\alpha}{2^w} \cdot \delta_{i-1})} \\
    = \abs{\delta_{i} - \frac{\alpha}{2^w} \cdot \delta_{i-1}}
\end{align}
for one column's value $x_i = \x_{ij}$ for some $j$ and coefficient $\alpha$, the gradient is:
\begin{align}
    % x = 5
    % \frac{\partial x}{\partial \alpha} x = 5
    % \mathcal{L}(\x_i, \hat{\x}_i)
        \frac{\partial }{\partial \alpha} \abs{\delta_{i} - \frac{\alpha}{2^w} \cdot \delta_{i-1}}
% \begin{equation*}
&= \begin{cases}
        -{2^{-w}}\vdelta_{i-1} & \x_{i} > \hat{\x}_{i} \\
        {2^{-w}}\vdelta_{i-1} & \x_{i} \le \hat{\x}_{i}
\end{cases} \\
&= -\sign(\eps) \cdot {2^{-w}}\vdelta_{i-1} \\
&\propto -\sign(\eps) \cdot \vdelta_{i-1}
% \end{equation*}
\end{align}
where we define $\eps \triangleq \x_{i} - \hat{\x}_{i}$ and ignore the $2^{-w}$ as a constant that can be absorbed into the learning rate. In all experiments reported here, we set the learning rate to $\frac{1}{2}$. This value is unlikely to be ideal for any particular dataset, but preliminary experiments showed that it consistently worked reasonably well. % and was large enough to avoid zeroing out small gradients.

In practice, \fire differs from the above pseudocode in three ways. First, instead of computing the coefficient for each sample, we compute it once at the start of each block. Second, instead of performing a gradient update after each sample, we average the gradients of all samples in each block and then perform one update. Finally, we only compute a gradient for every other sample, since this has little or no effect on the accuracy and slightly improves speed.

\begin{figure*}[t]
\begin{center}
    \includegraphics[width=\textwidth]{paper/overview}
    \caption{Overview of \minesp using a delta coding predictor.\textit{ a)} Delta coding of each column, followed by zigzag encoding of resulting errors. The maximum number of significant (nonzero) bits is computed for each column. \textit{b)} These numbers of bits are stored in a header, and the original data is stored as a (byte-aligned) payload, with leading zeros removed. When there are few columns, each column's data is stored contiguously. When there are many columns, each row is stored contiguously, possibly with padding to ensure alignment on a byte boundary.}
    % With blocks of eight samples (only four shown for clarity), this ensures that each column's data begins on a byte boundary.
    \label{fig:overview}
    \vspace{-5mm}
\end{center}
\end{figure*}

% ------------------------------------------------
\subsection{Bit Packing} \label{sec:bitpacking}
% ------------------------------------------------

An illustration of \mine's bit packing is given in Figure~\ref{fig:overview}. The prediction errors from delta coding or \fire are zigzag encoded \cite{zigzag} and then the minimum number of bits required is computed for each column. Zigzag encoding is an invertible transform that interleaves positive and negative integers such that each integer is represented by twice its absolute value, or twice its absolute value minus one for negative integers. This makes all values nonnegative and maps integers farther from zero to larger numbers.
% ). % We zigzag encode because it is extremely efficient and facilitates computing the number of bits necesasary.

Given the zigzag encoded errors, the number of bits $w^\prime$ required in each column can be computed as the bitwidth minus the fewest leading zeros in any of that column's errors. E.g., in Figure~\ref{fig:overview}a, the first column's largest encoded value is 16, represented as \texttt{00010000}, which has three leading zeros. This means that we require $w^\prime = 8 - 3 = 5$ bits to store the values in this column. One can find this value by ORing all the values in a column together and then using a built-in function such as GCC's $\texttt{\_\_builtin\_clz}$ to compute the number of leading zeros in a single assembly instruction (c.f. \cite{fastpfor}). This optimization motivates our use of zigzag encoding to make all values nonnegative.

Once the number of bits $w^\prime$ required for each column is known, the zigzag-encoded errors can be bit packed. First, \minesp writes out a header consisting of $D$ unsigned integers, one for each column, storing the bitwidths. Each integer is stored in $\log2(w)$ bits, where $w$ is the bitwidth of the data. Since there are $w+1$ possible values of $w^\prime$ (including 0), width $w-1$ is treated as a width of $w$ by both the encoder and decoder. E.g., 8-bit data that could only be compressed to 7 bits is both stored and decoded with a bitwidth of 8.

After writing the headers, \minesp takes the appropriate number of low bits from each element and packs them into the payload. When there are few columns, all the bits for a given column are stored contiguously (i.e., column-major order). When there are many columns, the bits for each \textit{sample} are stored contiguously (i.e., row-major order). In the latter case, up to seven bits of padding are added at the end of each row so that all rows begin on a byte boundary. This means that the data for each column begins at a fixed bit offset within each row, facilitating vectorization of the decompressor. The threshold for choosing between the two formats is a sample width of $32$ bits.

The reason for this threshold is as follows. Because the block begins in row-major order and we seek to reconstruct it the same way, the row-major bit packing case is more natural. For small numbers of columns, however, the row padding can significantly reduce the compression ratio. Indeed, for univariate 8-bit data, it makes compression ratios greater than 1 impossible. This gives rise to the column-major case; using a block size of eight samples and column-major order, each column's data always falls on a byte boundary without any padding. The downside of this approach is that both encoder and decoder must transpose the block. However, for up to four 8-bit columns or two 16-bit columns, this can be done quickly using SIMD shuffling instructions.\footnote{For recent processors with AVX-512 instructions, one could double these column counts, but we refrain from assuming that these instructions will be available.} This gives rise to the cutoff of 32 bit sample width for choosing between the formats.

As a minor bit packing optimization, one can store the headers for two or more blocks contiguously, so that there is one group of headers followed by one group of payloads. This allows many headers to share one set of padding bits between the headers and payload. Grouping headers does not require buffering more than one block of raw input, but it does require buffering the appropriate number of blocks of compressed output. In addition to slightly improving the compression ratio, it also enables more headers to be unpacked with a given number of vector instructions in the decompressor. Microbenchmarks show up to $10\%$ improvement in decompression speed as the number of blocks in a group grows towards eight. However, we use groups of only two in all reported experiments to ensure that our results tend towards pessimism and are applicable under even the most extreme buffer size constraints.

% ------------------------------------------------
\subsection{Entropy Coding}
% ------------------------------------------------

We entropy code the bit packed representation of each block using Huff0, an off-the-shelf Huffman coder \cite{fse}. This encoder treats individual bytes as symbols, regardless of the bitwidth of the original data. We use Huffman coding instead of Finite-State Entropy \cite{fse} or an arithmetic coding scheme since they are slower, and we never observed a meaningful increase in compression ratio.

The benefit of adding Huffman coding to bit packing stems from bit packing's inability to optimally encode individual bytes. For a given packed bitwidth $w$, bit packing models its input as being uniformly distributed over an interval of size $2^{w}$. Appropriately setting $w$ allows it to exploit the similar variances of nearby values, but does not optimally encode individual values (unless they truly are uniformly distributed within the interval). Huffman coding is complementary in that  it fails to capture relationships between nearby bytes but optimally encodes individual bytes.

We Huffman code after bit packing, instead of before, for two reasons. First, doing so is faster. This is because the bit packed block is usually shorter than the original data, so less data is fed to the Huffman coding routines. These routines are slower than the rest of \mine, so minimizing their input size is beneficial. Second, this approach increases compression. Bit packed bytes benefit from Huffman coding, but Huffman coded bytes do not benefit from bit packing, since they seldom contain large numbers of leading zeros. This absence of leading zeros is unsurprising since Huffman codes are not byte-aligned and use ones and zeros in nearly equal measure.

% 1) Huffman coding is slower than the rest of \mine, so running it on the smaller bit packed block is (usually) shorter than the original data.

% so applying it last is faster faster than Huffman coding the original data or the errors,

% ------------------------------------------------
\subsection{Vectorization}
% ------------------------------------------------

Much of \mine's speed comes from vectorization. For headers, the fixed bitwidths for each field and fixed number of fields allows for packing and unpacking with a mix of vectorized byte shuffles, shifts, and masks. For payloads, delta (de)coding, zigzag (de)coding, and \fire all operate on each column independently, and so naturally vectorize. Because the packed data for all rows is the same length and aligned to a byte boundary (in the high-dimensional case), the decoder can compute the bit offset of each column's data one time and then use this information repeatedly to unpack each row. In the low-dimensional case, all packed data fits in a single vector register which can be shuffled/masked appropriately for each possible number of columns. This is possible since there are at most four columns in this case. On an \texttt{x86} machine, bit packing and unpacking can be accelerated with the \texttt{pext} and \texttt{pdep} instructions, respectively.


\documentclass[11pt, letterpaper, twoside, final]{article}
\usepackage{amssymb, amsmath, amsthm}
\usepackage{mathtools, thmtools}
\usepackage{csquotes}
\usepackage{cleveref}
\usepackage[margin=1in]{geometry}
\newcommand*{\dto}{\overset{d}{\longrightarrow}}
\newcommand*{\pto}{\overset{p}{\longrightarrow}}
\newcommand*{\mvert}{\,\middle\vert\,}
\newcommand*{\ivert}{\,\vert\,}
\newcommand*{\W}{\mathcal{W}}
\DeclareMathOperator*{\Var}{\mathbb{V}ar}
\DeclareMathOperator*{\Cov}{\mathbb{C}ov}
\DeclareMathOperator*{\E}{\mathbb{E}}
\newtheorem{lemma}{theorem}

\author{Xu Cheng and Eric Renault and Paul Sangrey}
\title{Derivation of Asymptotic Covariance Matrix}

\date{\today}

\begin{document}

\maketitle

We want to estimate $\omega \coloneqq (\rho, c, \delta, \phi, \pi, \theta)'$ and get its asymptotic distribution.
We also have some auxiliary parameters: $\xi_1 \coloneqq (\beta, \gamma, \psi)'$ and $\xi_2 \coloneqq \phi^2$.
The way we do this by splitting $\omega$ in two parts.
The first $\omega_s$ is the vector of purely structural parameters: $(\phi, \pi, \theta)'$.
The second part $\omega_r$ is composed of parameters that we can estimate directly without model-based
cross-equation restrictions, but are of interest.
We collect all of the reduced parameters into a vector: $\xi \coloneqq (\omega_r', \xi_a)'$.
We start by constructing a GMM estimator for $\xi$, we will then show how to convert this into an estimator for
$\omega$.
Throughout, I will denote the partial derivative of some function $f$ with respect to some variable $x$ with
$f_x$.

\section{Stage 1}

We view estimating the first stage as a particular form of GMM.
From standard GMM theory, we know that the following holds, for some asymptotic covariance matrix $\Omega_{\xi}$.

\begin{equation}
    \sqrt{T} (\hat{\xi} - \xi)  \dto N\left(0, \Omega_{\xi}\right)
\end{equation}

We will construct $\Omega_{\xi}$ in three steps.
First, we will derive the asymptotic covariance matrices for each of $\omega_r, \xi_1, \xi_2$.
Then we will show how to combine them into one joint covariance matrix.

\subsection{$\omega_r$}\label{sec:omega_r}

\begin{equation}
    h\left(\sigma^2_{t},\sigma^2_{t+1} \mvert \omega_r\right) \coloneqq
\begin{bmatrix}
    \sigma^2_{t+1} - (c \delta + \rho \sigma^2_{t}) \\
%
    \sigma^2_{t} \left(\sigma^2_{t+1} - (c \delta + \rho \sigma^2_{t})\right)\\
%
    \sigma^4_{t+1} - \left(c^{2} \delta + 2 c \rho \sigma^2_{t} + \left(c \delta + \rho
    \sigma^2_{t}\right)^{2}\right)\\
%
    \sigma^2_{t} \left(\sigma^4_{t+1} - \left(c^{2} \delta + 2 c \rho \sigma^2_{t} + \left(c \delta + \rho
    \sigma^2_{t}\right)^{2}\right)\right)\\

    \sigma^4_{t} \left(\sigma^4_{t+1} - \left(c^{2} \delta + 2 c \rho \sigma^2_{t} + \left(c \delta + \rho
    \sigma^2_{t}\right)^{2}\right)\right)\\
\end{bmatrix}
\end{equation}

By standard GMM theory, the following holds, if the weighting matrix \newline $W_{\omega_r,T} \pto \E[h(\sigma^2_t,
\sigma^2_{t+1} \ivert \omega_r)' h(\sigma^2_t, \sigma^2_{t+1} \ivert \omega_r)]^{-1}$.
We have

\begin{equation}
    \sqrt{T}(\widehat{\omega}_r - \omega_r) \dto N\left(0, \Omega_{\omega_r}\right),
\end{equation}

\noindent where

\begin{equation}
    \Omega_{\omega_r} \coloneqq \left(\E\left[h_{\omega_r}(\sigma^2_{t}, \sigma^2_{t+1}, \xi_{1})\right]'
    \E[h(\sigma^2_t, \sigma^2_{t+1} \ivert \omega_r)' h(\sigma^2_t, \sigma^2_{t+1} \ivert \omega_r)]^{-1}
    \E\left[h_{\omega_r}(\sigma^2_{t}, \sigma^2_{t+1}, \xi_{1})\right]\right)^{-1}
\end{equation}

\noindent We estimate this by replacing the population expectations and covariances by their sample counterparts.

\subsection{$\xi_1$}\label{sec:est_xi2}

We estimate $\xi_1$ by weighted least squares, a special case of GMM.

\begin{equation}
    \E\left[r_{t+1} \mvert \sigma^2_t, \sigma^2_{t+1}\right]  = \gamma + \beta \sigma^2_t + \psi \sigma^2_{t+1}
\end{equation}

The only unusual part is we know that $\Var(r_{t+1} \ivert \sigma^2_t, \sigma^2_{t+1}) = (1-\phi^2)
\sigma^2_{t+1}$.
Consequently, the regression results  are more efficient if we adjust for heteroskedasticity.
However, we do not know $\phi$, and so this might seem impossible.
However, since time-invariant parts of heteroskedasticity adjustments cancel, reweighting by the inverse of
$\sigma^2_{t+1}$ achieves is equivalent to the optimal reweighting.
Also, since $\sigma^2_{t+1}$ is contained in the conditioning set, the fact that it is viewed as a random variable
in other parts  of the regression is irrelevant.
Let $u_t = \frac{r_{t+1} - (\gamma + \beta \sigma^2_{t} + \psi \sigma^2_{t+1}}{\sigma^2_{t+1}}$.

Since this regression is exactly identified, any positive-definite weight matrix, including the identity is
optimal.
Consequently, we have the following result, where the WLS covariance matrix has the standard form:

\begin{equation}
    \Omega_{\xi_1} = \E\left[\left(1, \sigma^2_{t}, \sigma^2_{t+1}\right) \left(1, \sigma^2_{t},
    \sigma^2_{t+1}\right)'\right]^{-1} \Var\left(u_t\right).
\end{equation}

\subsection{Step 3: $\xi_2$}

We know that $\Var(r_{t+1} \vert \sigma^2_{t+1} \sigma^2_t) = (1-\phi^2) \sigma^2_{t+1}$.
This implies $\Var(\frac{r_{t+1}}{\sigma_{t+1}} \vert \sigma^2_{t+1}, \sigma^2_t) = 1 - \phi^2$.
Since we consistently estimate the conditional mean of $r_{t+1}$ in \cref{sec:est_xi2}, the residuals ---
$\widehat{u}_t = \frac{r_{t+1} - \widehat{\gamma} - \widehat{\beta}\sigma^2_t - \widehat{\psi}
\sigma^2_{t+1}}{\sigma_{t+1}}$ --- satisfy $\frac{1}{T} \sum_{t=1}^T \hat{u}_t^2 \pto (1 - \phi^2)$.

Define $\widehat{\xi}_2 \coloneqq 1 - \frac{1}{T} \sum_{t=1}^T \hat{u}_t^2 \pto (1 - \phi^2)$.
Then $\widehat{\xi}_2 \dto N(0, \Omega_{\xi_2})$ for some covariance matrix $\Omega_{\xi_2}$, since this is a  GMM
estimator.
What is $\Omega_{\xi_2}$.
Since we are just identified, standard GMM theory says it is the covariance of the moment condition scaled by the
appropriate derivative.
Since we are estimating a mean shifted by a constant, the derivative equals $1$.
Consequently, $\Omega_{\xi_2} = \Var(\frac{u^2_t}{\sigma^2_{t+1}})$, which can be estimated by
$\frac{1}{T} \sum_{t=1}^T (\frac{\widehat{u}_t^2}{\sigma^2_{t+1}} - \frac{1}{T} \sum_{t=1}^T
\frac{\widehat{u}_t^2}{\sigma^2_{t+1}})^2$, i.e.\@ the sample covariance of the squared residuals.

\subsection{Combining $\Omega_{\omega_r}, \Omega_{\xi_2}$, and $\Omega_{\xi_2}$}

Each of $\Omega_{\xi_i}$ are of the form $(\E[h_{\xi_{i}}(\sigma^2_{t+1}, \sigma^2_t \ivert \xi_i)]'
\Var(h(\sigma^2_{t+1}, \sigma^2_t \ivert \xi_i) \ivert \xi_i)^{-1} \E[h_{\xi_{i}}(\sigma^2_{t+1}, \sigma^2_t
\ivert \xi_i) \ivert \xi_i])^{-1}$.
Consequently, the off-diagonal blocks of the joint covariance matrix $\Omega$ come from two places:
the derivatives and the covariance of the moments.
Since the moments in the first stage do not depend on the parameters in the second stage, and vice-versa, the
derivatives to not cause any co-movement.
The other cases are trickier, and so we consider them each in turn.

Consider the covariance between $h(\sigma^2_{t+1}, \sigma^2_t, \omega_r)$ and $h(\sigma^2_{t+1}, \sigma^2_t,
\xi_2)$, which we can rearrange since the moments are mean zero.

\begin{gather}
    \Cov\left(h\left(r_{t+1}, \sigma^2_{t+1}, \sigma^2_t \mvert \omega_{r} \right) ,
      h\left(\sigma^2_{t+1} \sigma^2_t \mvert \xi_2 \right) \right)
%
%
    = \E\left[h(r_{t+1},  \sigma^2_{t+1}, \sigma^2_t \mvert \omega_r) h\left(\sigma^2_{t+1}, \sigma^2_t \mvert
       \xi_2 \right) \right]
%
%
       \intertext{By the law of iterated expectations.}
%
%
    = \E\left[\E[h\left(r_{t+1},  \sigma^2_{t+1}, \sigma^2_t \mvert \omega_r\right) \mvert \sigma^2_{t+1},
       \sigma^2_t] \E\left[h\left(\sigma^2_{t+1}, \sigma^2_t \mvert \xi_2 \right) \mvert \sigma^2_{t+1},
       \sigma^2_t\right] \right]
\end{gather}

The first term in the expression above equals zero, and, hence, so does the entire expression.
In other words, the first two set of moment conditions are independent.
By an identical argument, the first and third moments are independent as well.

We now consider how the second and their sets of moments are related.
Since the derivatives are with respect to different parameters (and constant) no dependence arises from there.
The question is how are the moment conditions in the second and third steps related.
The second stage moment condition is a conditional mean and third stage moment is a conditional covariance.
Let $u_t$ denote the error term in that regression (as it did above).

\begin{equation}
    \E\left[\E\left[\frac{r_{t+1} - \E\left[r_{t+1}\mvert \sigma^2_{t+1} \sigma^2_t\right]}{\sigma_{t+1}} \right]
    \E\left[\frac{(r_{t+1} - \E\left[r_{t+1} \mvert \sigma^2_{t+1} \sigma^2_t\right])^2}{\sigma^2_{t+1}}\right]
    \right]
%
    = \E\left[\E\left[\frac{u_t u_t^2}{\sigma^2_{t+1}}\right] \mvert \sigma^2_t, \sigma^2_{t+1} \right]  = 0
\end{equation}

Since $u_t$ is conditionally Gaussian, its conditional third moment equals zero.
By the law of iterated expectations, its unconditional moment does as well.

In other words, the second and third set of moments are uncorrelated.
Now, the careful reader might be worried about filling in the population expectations instead of their estimators
in the regression above.
However, since the expectations are linear and consistently estimable, this error vanishes in the limit.
Intuitively, OLS mean and variance estimates are asymptotically independent.

In addition, since all three components are asymptotically independent; the inverse of a block-diagonal matrix is
block-diagonal, and we using optimal weighting matrices in each part, we are using an
optimal weighting matrix for $\xi$, not just its components.

\section{Stage 2}

In this stage, we convert the estimates for $\xi$ into estimates for $\omega$.
We do this by specifying a link function.
Since $(\rho, c, \delta)'$ shows up in both $\xi$ and $\omega$, the link function for those parameters is the
identify function.
The third set of parameters $\xi_3 = \phi^2$, and so we use that as a link function.
We also use the reduced-form estimates as link functions, as well, i.e.\@

\begin{equation}
    g(\xi, \omega) = \xi - (\rho, c, \delta, \beta(\rho, c, \phi, \pi, \theta), \gamma(\rho, \delta, c, \phi, \pi,
    \theta), \psi(\rho, c, \phi, \theta), \phi^2)'
\end{equation}

We specify $g(\xi, \omega)$ in this way because it gives us the correct off-diagonal elements.
If we estimated $(\rho, c, \delta)$ by themselves and just plugged them in here, we would have to relate the
off-diagonal terms in a separate step.
In addition, by using the optimal weight matrix for this stage, we estimate all of the parameters as efficiently
possible using our moment conditions.

Doing this implies we must be careful and treat the $\rho,c$, and $\delta$ on each sides of the equation as
different.
The 2nd-stage sample criterion function is

\begin{equation}
    Q_T(\omega) \coloneqq \frac{1}{2} g(\widehat{\xi}_T, \omega)' \W_{T} g(\widehat{\xi}_T, \omega)
\end{equation}

\noindent  with second stage weight matrix $\W_T$.
We need to estimate $\omega$, and so we differentiate and get the first-order condition at $\omega_0$, where $\W
\coloneqq \lim_{T \to \infty} \E[W_T]$.
This gives

\begin{equation}
    \frac{\partial Q_T}{\partial \omega}(\omega_0) =  g_{\omega}\left(\xi_0, \omega_0\right)  \W g\left(\xi_0,
    \omega_0\right) = 0.
\end{equation}

\noindent We now expand $\widehat{\xi}_T$ around $\xi_0$, for some $\widetilde{\xi}_T$ between $\xi_0$ and
$\widetilde{\xi}_T$,

\begin{align}
    \sqrt{T} \frac{\partial Q}{\partial \omega}(\omega_0)
%
    &= g_{\omega}\left(\omega_0, \widehat{\xi}_T\right) \W_T \left[\sqrt{T} g\left(\omega_{0},\xi_0\right) +
       g_{\xi}\left(\omega_{0}, \widetilde{\xi}_T\right)' \sqrt{T} \left(\widetilde{\xi}_T - \xi_0\right)\right]
%
%
    \intertext{The first term equals zero by assumption, and the derivative is the identity matrix.}
%
%
    &= g_{\omega}\left(\omega, \widehat{\xi}_T\right) \W_T \left[ \sqrt{T} \left(\widetilde{\xi}_T -
       \xi_0\right)\right]
\end{align}

\noindent We also need to compute the Hessian and evaluate it at $\widehat{\omega}_T$ a consistent estimator for
$\omega$.

\begin{align}
    \frac{\partial^2 Q}{\partial \omega \partial \omega'}(\widehat{\omega}_T) &=
%
    g_{\omega}\left(\widehat{\omega}_T, \widehat{\xi}_T\right)' \W_T g_{\omega} \left(\widehat{\omega}_T,
    \widehat{\xi}_T\right)+ g_{\omega, \omega}\left(\widehat{\omega}_T, \widehat{\xi}_T\right) \W_T
    g\left(\widehat{\omega}_T, \widehat{\xi}_T\right)' \\
%
    &\pto g_{\omega}\left(\omega, \xi_0\right)' \W g_{\omega} \left(\omega, \xi_0\right) + 0.
\end{align}

\noindent if we use the optimal weight matrix --- $\W \coloneqq \E[g_{\omega_s}(\theta_0, \xi_0)]$.
Standard extremum estimator theory gives

\begin{equation}
    \sqrt{T} \left(\widehat{\omega}_T - \omega_{0}\right)  \dto N\left(0, \left(\E[g_{\omega}(\theta_0, \xi_0)]'
    \Omega_{\xi}^{-1} \E[g_{\omega}(\theta_0, \xi_0)]\right)^{-1}\right).
\end{equation}

\noindent The covariance in the middle is GMM-covariance of the reduced-form parameters.
We estimate it by plugging $\widehat{\xi}$ into to the formulas above and their derivatives.
We estimate the asymptotic covariance matrix by replacing its components with their sample counterparts.

\section{Asymptotic Distribution of the Reduced-Form Parameter}

This section gives the asymptotic distribution of the reduced-form parameter.

Write $\omega =(\omega_{1},\omega_{2},\omega_{3}),$ where $\omega_{1}=(\rho ,c)$, $\omega_{2} = (\gamma ,\beta ,\psi)$, and $\omega_{3} = \phi ^{2}$.
Below we describe the estimator $\widehat{\omega}_{1},\widehat{\omega}_{2},\widehat{\omega}_{3}$ and provide
the asymptotic distribution of $\widehat{\omega} = (\widehat{\omega}_{1},\widehat{\omega
}_{2},\widehat{\omega}_{3})$.
We estimate these parameters separately because $\omega_{1}$ only shows up in the conditional mean and variance of
$r_{t+1};$ $\omega_{2}$ only shows up in the conditional mean of $\sigma_{t+1}^{2};$ and $\phi $ only shows up in
the conditional variance of $\sigma_{t+1}^{2}.$

We estimate $\omega_{1}$ by GMM based on the moment condition:

\begin{eqnarray}
    \mathbb{E}[h_{t}(\omega_{1,0}) & = &0,\text{ where}  \nonumber \\
%
    h_{t}(\omega_{1}) & = &\left(
        \begin{array}{c}
            \sigma_{t+1}^{2}-\left( c\delta +\rho \sigma_{t}^{2}\right)  \\
%
            \sigma_{t}^{2}\left( \sigma_{t+1}^{2}-\left( c\delta +\rho \sigma _{t}^{2}\right) \right)  \\
%
            \sigma_{t+1}^{4}-\left( c^{2}\delta +2c\rho \sigma_{t}^{2}+\left( c\delta +\sigma_{t+1}^{2}-\left(
            c\delta +\rho \sigma_{t}^{2}\right) ^{2}\right) \right)  \\
%
            \sigma_{t}^{2}\left( \sigma_{t+1}^{4}-\left( c^{2}\delta +2c\rho \sigma _{t}^{2}+\left( c\delta
            +\sigma_{t+1}^{2}-\left( c\delta +\rho \sigma _{t}^{2}\right) ^{2}\right) \right) \right)  \\
%
            \sigma_{t}^{4}\left( \sigma_{t+1}^{4}-\left( c^{2}\delta +2c\rho \sigma _{t}^{2}+\left( c\delta
            +\sigma_{t+1}^{2}-\left( c\delta +\rho \sigma _{t}^{2}\right) ^{2}\right) \right) \right)
        \end{array}\right).
\end{eqnarray}
%
The optimal GMM estimator is
%
\begin{eqnarray}
    \widehat{\omega}_{1} & = & \underset{\omega_{1}\in \Lambda_{1}}{\arg\min}
%
    \overline{h}_{T}(\omega_{1})^{\prime}W_{T}\overline{h}_{T}(\omega_{1}), \text{ where}  \nonumber \\
%
    \overline{h}_{T}(\omega_{1}) & = &T^{-1}\sum_{t = 1}^{T}h_{t}(\omega_{1}), \nonumber \\
%
    W_{T} & = &T^{-1}\sum_{t = 1}^{T}h_{t}(\widetilde{\omega}_{1})h_{t}(\widetilde{\omega}_{1})^{\prime
   }-\overline{h}_{T}(\widetilde{\omega}_{1})\overline{h} _{T}(\widetilde{\omega}_{1})^{\prime},
\end{eqnarray}
%
where $\widetilde{\omega}_{1}$ is the preliminary GMM estimator based on the identify covariance matrix.

We estimate $\omega_{2}$ by the GLS estimator because $\gamma, \beta, \psi$ are the intercept and linear
coefficients of the conditional mean function and the conditional variance is proportional to $\sigma_{t+1}^{2}$.
Define $x_{t} = \sigma_{t+1}^{-1}(1,\sigma_{t}^{2},\sigma_{t+1}^{2})$ and $y_{t} = \sigma_{t+1}^{-1}r_{t+1}$.
The GLS\ estimator of $\omega_{2}$ is
%
\begin{equation}
    \widehat{\omega}_{2} = \left( \sum_{t = 1}^{T}x_{t}x_{t}^{\prime}\right) ^{-1}\sum_{t = 1}^{T}x_{t}y_{t}.
\end{equation}

We estimate $\omega_{3}$ by the sample variance estimator.
Define
%
\begin{equation}
    \widehat{y}_{t} = x_{t}\widehat{\omega}_{2} = \sigma_{t+1}^{-1}(\widehat{\gamma}+\widehat{\beta
   }\sigma_{t}^{2}+\widehat{\psi}\sigma_{t+1}^{2}).
\end{equation}
%
The estimator of $\omega_{3}$ is
\begin{equation}
    \widehat{\omega}_{3} = \max \{1-T^{-1}\sum_{t = 1}^{T}\left( y_{t}-\widehat{y}_{t}\right) ^{2},0\}.
\end{equation}%
[**XC. In practice, do we need to impose the estimator is positive?]

The next lemma provides the asymptotic distribution of the estimator $\widehat{\omega}$.
Let $h_{\omega ,t}(\omega_{1})\in \mathbb{R}^{5\times 2}$ denote the derivative of $h_{t}(\omega_{1})$ w.r.t.\@
$\omega_{1}$.
Define
%
\begin{eqnarray}
    \Omega_{1} & = &\left \{ \mathbb{E}\left[ h_{\omega ,t}\left( \omega _{1,0}\right) \right]^{\prime}
        \mathbb{E}[h_{t}(\omega_{1,0})h_{t}(\omega _{1,0})^{\prime}]^{-1}\mathbb{E}\left[
    h_{\omega,t}\left(\omega_{1,0}\right) \right] \right \} ^{-1},  \notag \\
%
    \Omega_{2} & = &\mathbb{E}\left[ x_{t}x_{t}^{\prime}\right] ^{-1}\mathbb{E[(} y_{t}-x_{t}^{\prime
   }\omega_{2,0})^{2}],  \notag \\
%
    \Omega_{3} & = &\mathbb{V[}\left( y_{t}-x_{t}^{\prime}\omega_{2,0}\right)^{2}]
\end{eqnarray}

\begin{lemma}
Suppose Assumptions *** hold. Then

\begin{equation}
    T^{1/2}\left(
%
    \begin{array}{c}
        \widehat{\omega}_{1}-\omega_{1,0} \\
        \widehat{\omega}_{2}-\omega_{2,0} \\
        \widehat{\omega}_{3}-\omega_{3,0}%
    \end{array}\right)
%
    \rightarrow_{d}\xi_{\omega}  =
%
   \left(\begin{array}{c}
            \xi_{\omega 1} \\
            \xi_{\omega 2} \\
            \xi_{\omega 3}%
   \end{array}\right)
%
   \sim N\left( 0,
%
    \begin{array}{ccc}
        \Omega_{1} & 0 & 0 \\
        0 & \Omega_{2} & 0 \\
        0 & 0 & \Omega_{3}%
    \end{array}\right).
\end{equation}
\end{lemma}

\begin{proof}
    Will be added later.
\end{proof}

\section{Robust Inference for Risk Price}

The reduced-form parameters are $\omega  = (\rho ,c,\gamma ,\beta ,\psi ,\phi ^{2})$.
Using the conditional mean and conditional variance derived in the paper, we estimate $\, \omega_{1} = (\rho ,c)$
by the GMM estimator, estimate $\omega_{2} = (\psi ,\beta ,\gamma )$ by the GLS estimator, and estimate
$\omega_{3} = \phi^{2}$ by the method of moments estimator for the variance.

We can show that the estimator $\widehat{\omega}$ satisfies

\begin{equation}
    T^{1/2}(\widehat{\omega}-\omega_{0})\rightarrow_{d}\upsilon_{\omega}\sim N(0,V).
\end{equation}%

See the next section for details.
Note that these estimators do not involve the structural parameters $\theta $ and $\pi$.
We do not plug in $\beta ,\gamma ,\psi $ as functions of $\theta $ and $\pi .$ Instead, we treat $\beta ,\gamma
,\psi $ just as linear coefficients and estimate them by GLS.

We estimate the structural parameters $\theta $ and $\pi $ using $\widehat{ \omega}$ and the link functions
specified below.
First, we know that
%
\begin{equation}
    \label{psi_fn}
    \psi_{0}=\phi_{0}\left( c_{0}\left( 1+\rho_{0}\right) \right) ^{-1/2} - \left( 1-\phi_{0}^{2}\right)
    /2-(1-\phi_{0}^{2})\theta_{0}
\end{equation}
%
when all parameters are evaluated at the true values.
This equation strongly identifies $\theta_{0}$ because $\phi_{0}$ is assumed to be negative and bounded away from
1 in magnitude..
It follows from (\cref{psi_fn}) that

\begin{equation}
    \theta_{0} = L(\omega_{0})=-(1-\phi_{0}^{2})^{-1}\left[ \psi_{0}-\phi_{0}\left( c_{0}\left( 1+\rho_{0}\right)
    \right)^{-1/2}-\left( 1-\phi_{0}^{2}\right) /2\right] .
\end{equation}
%
Thus, we estimate $\theta_{0}$ by
%
\begin{equation}
    \widehat{\theta}=L(\widehat{\omega}).
\end{equation}
%
By the delta method, we know that
%
\begin{equation}
    T^{1/2}(\widehat{\theta}-\theta_{0})\rightarrow_{d}L_{\omega}(\omega_{0})^{\prime}\upsilon_{\omega},
\end{equation}
%
where $L_{\omega}(\omega )\in R^{d_{\omega}}$ denote the derivative of $L(\omega )$ w.r.t.\@ $\omega$.
The inference for $\theta $ is standard.
A confidence interval for $\theta $ can be obtained by inverting the $t$-statistic with a critical value obtained
from the standard normal distribution.

Next, we consider inference for the structural parameter $\pi$.
This is a non-standard problem because $\pi $ is potentially weakly identified.
Define
%
\begin{equation}
    g(\pi ,\omega )=\left(
%
    \begin{array}{c}
        \gamma -\left[ B\left( \pi +C\left( \theta_{L}-1\right) \right) -B\left( \pi +C\left( \theta_{L}\right)
        \right) \right]  \\
%
        \beta -\left[ A\left( \pi +C\left( \theta_{L}-1\right) \right) -A\left( \pi +C\left( \theta_{L}\right)
        \right) \right]
    \end{array}\right),
%
    \text{ where}\ \theta_{L} = L(\omega).
\end{equation}
%
We know
%
\begin{equation}
    g(\pi_{0}, \omega_{0})=0 \in \mathbb{R}^{2}.
\end{equation}
%
Inference on $\pi $ is based on the function $g(\pi ,\widehat{\omega})$ because $\widehat{\omega}$ is a
consistent estimator of $\omega_{0}$.
By the consistency of $\widehat{\omega}$,
%
\begin{equation}
    T^{1/2}\left[ g(\pi, \widehat{\omega}) - g(\pi,\omega_{0})\right] \Rightarrow \upsilon(\pi) = G(\pi,
    \omega_{0})^{\prime}\upsilon_{\omega},
\end{equation}%

\noindent where $G(\pi ,\omega )$ denote the derivative of $g(\pi ,\omega )$ w.r.t.\@ to $\omega$.
The Gaussian process $\upsilon(\pi)$ has covariance kernel
%
\begin{equation}
    \Sigma (\pi_{1},\pi_{2})=G(\pi_{1},\omega_{0})^{\prime}VG(\pi_{2},\omega_{0}).
\end{equation}
%
We can estimate $\Sigma (\pi_{1},\pi_{2})$ by
%
\begin{equation}
    \widehat{\Sigma}(\pi_{1},\pi_{2})=G(\pi_{1},\widehat{\omega})^{\prime}
    \widehat{V}G(\pi_{2},\widehat{\omega}),
\end{equation}

\noindent where $\widehat{V}$ is a consistent estimator of $V.$

We construct a confidence interval for $\pi $ by inverting tests $H_{0}:\pi =\pi_{0}$ vs $H_{0}:\pi \neq \pi_{0}$.
The test statistic is the QLR statistic:
%
\begin{equation}
    QLR=Tg(\pi_{0},\widehat{\omega})^{\prime}\widehat{\Sigma}(\pi_{0},\pi _{0})^{-1}g(\pi_{0},\widehat{\omega})
    - \underset{\pi \in \Pi}{\min}Tg(\pi ,\widehat{\omega})^{\prime}\widehat{\Sigma}(\pi,\pi)^{-1}
    g(\pi,\widehat{\omega}).
\end{equation}

To obtain the critical value, we follow the conditional inference approach in Andrews and Mikusheva (2016). To
this end, first construct a projection residual process:
%
\begin{equation}
    h(\pi ,\widehat{\omega})=g(\pi ,\widehat{\omega})-\widehat{\Sigma}(\pi ,\pi_{0})\widehat{\Sigma
   }(\pi_{0},\pi_{0})^{-1}g(\pi_{0},\widehat{\omega}).
\end{equation}
%
By construction, $h(\pi ,\widehat{\omega})$ and $g(\pi_{0},\widehat{\omega})$ are independent asymptotically.
Conditional on $h(\pi ,\widehat{\omega})$, we obtain the $1-\alpha $ quantile of the QLR statistic, denoted
$c_{\alpha}(h)$, by sampling from the asymptotic distributions of $g(\pi_{0},\widehat{\omega})$ under the null.
Specifically, we take independent draws $\upsilon^{\ast}\sim N(0,\Sigma (\pi_{0},\pi_{0}))$ and produce simulated
process:
%
\begin{equation}
    g^{\ast}(\pi ,\widehat{\omega}) = h\left(\pi ,\widehat{\omega}\right) + \widehat{\Sigma} (\pi
    ,\pi_{0})\widehat{\Sigma}(\pi_{0},\pi_{0})^{-1}\upsilon^{\ast}.
\end{equation}%
%
We then calculate
\begin{equation}
    QLR^{\ast}=Tg^{\ast}(\pi_{0},\widehat{\omega})^{\prime}\widehat{\Sigma} (\pi_{0},\pi_{0})^{-1}g^{\ast
   }(\pi_{0},\widehat{\omega})-\underset{\pi \in \Pi}{\min}Tg^{\ast}(\pi ,\widehat{\omega})^{\prime
   }\widehat{\Sigma} (\pi ,\pi )^{-1}g^{\ast}(\pi ,\widehat{\omega}),
\end{equation}
%
which is a random drawn from the conditional distribution of the $QLR$ statistic given $h_{T}(\pi ,\widehat{\omega
})$, when $g(\pi_{0},\widehat{ \omega})$ is drawn from its asymptotic distribution.
In practice, we repeat this process for a large number of times and obtain $c_{\alpha}(h)$ by simulation.

We reject the null $H_{0}:\pi =\pi_{0}$ if $QLR\geq c_{\alpha}(h)$.
The confidence interval for $\pi $ is the collection of null values that are not rejected as the null value.
Note that the construction of this CI\ does not involve estimation of $\pi$.

\end{document}


\chapter{\acro{gui} \fsm{} Testing and \acro{other}}

\label{app:test}

In the Kleene \acro{gui}, using the \texttt{test} window, you can type in
individual strings for generation or analysis, and see the output string
or strings displayed in the terminal window.  When testing \fsm{}s
containing \acro{other} (i.e.\@ unknown) labels, the output may at first
be unintuitive.

The first step is to understand that the \texttt{test} function implements
the formal algorithm for applying an \fsm{} to an input string.  For
generation using \verb!$testFst!

\begin{enumerate}
\item
The input string is built into a one-path \fsm{}; think of it as \verb!$input!.
\item
The input \fsm{} is composed ``on top of'' the \verb!$testFst!, i.e.\@
\verb!($input _o_ $testFst!)
\item
Then the output side (also known as the lower side) of the composition is extracted, i.e.\@
\verb!$^lowerside($input _o_ $testFst)!.  The result is an acceptor.
\item
Then, if the result acceptor is not empty, and the number of paths is finite, the strings are listed in the
terminal window.
\end{enumerate}

\noindent
Similarly for analysis, the result acceptor is computed as
\verb!$^upperside($testFst _o_ $input)!.

The second important point is that if the \fsm{} being tested contains
\acro{other\_nonid}, then any output strings containing unknown characters
will display as \acro{other\_id} rather than \acro{other\_nonid}.  This is
because when a projection (an acceptor) is extracted, the
\acro{other\_nonid} label cannot appear in an acceptor and is
automatically converted to \acro{other\_id}.

\begin{Verbatim}[fontsize=\small]
$testFst = .:. ;
draw $testFst ;
$generationresult = $^lowerside(z _o_ $testFst) ;
draw $result ;

test $testFst ;	// and enter z on top for generation
\end{Verbatim}

\noindent
In this example, the \verb!$testFst! \fsm{} will contain both
\acro{other\_id} and \acro{other\_nonid} labels, but the
\verb!$generationresult! \fsm{}, and the printed output of testing
\verb!$testFst! using the \texttt{test} facility, and generating
\texttt{z}, will contain \acro{other\_id} and not \acro{other\_nonid}.
Again, the reason is that the application algorithm extracts a projection
of the composition, and a projection is an acceptor and cannot contain
\acro{other\_nonid}.

The final important point in interpreting the output is to understand
that the meaning/coverage of \acro{other\_id} in the result should be
interpreted relative to the \verb!$generationresult! or
\verb!$analysisresult! \fsm{}, which is automatically created and
displayed when using \texttt{test}.

Other kinds of testing, using external runtime code, cannot reasonably
implement the formal composition-plus-projection-extraction method of
application, and the results will be different.  In the future, the
Kleene \texttt{test} facility may offer a choice between the current
formal method and alternative methods used in runtime code.


\documentclass{stdlocal}
\begin{document}
\section{Preliminaries} % (fold)
\label{sub:preliminaries}
  To systematically approach the implementation of PRNGs, basic knowledge in the topics of stochastics and statistics is administrable.
  Together, these topics will give a deeper understanding of randomness in deterministic computer systems, a formal description of pseudorandom sequences and generators, and the mathematical foundation of Monte Carlo algorithms.
  Based on them, we are capable of scientifically analyzing PRNGs concerning their randomness properties.
  Afterwards, we will give a brief overview of template mechanisms in the C++ programming language and the fundamentals of modern computer architecture.

  \subsection{Probability Theory} % (fold)
  \label{sub:stochastics}
    The observation of random experiments resulted in the construction of probability theory.
    But probability theory itself does not use a further formalized concept of randomness \autocite{schmidt2009}.
    % Randomness itself plays a minor role in probability theory and is used in form of realizations of random variables.
    In fact, it allows us to observe randomness without defining it \autocite{volchan2002}.
    % Actually, typical formalizations rely on probability theory.
    % This connection makes the development of RNGs possible.
    % Hence, in the following we will give only the formal definition of relevant structures without further discussions and will postpone an examination of randomness to the next section.
    Hence, we will postpone an examination of truly random sequences to section \ref{sec:pseudorandom_number_generators}.

    According to \textcite{schmidt2009}, Kolmogorov embedded probability theory in the theory of measurement and integration.
    Although it heavily relies on these theoretical structures, probability theory is one of the most important applications of measurement and integration theory.
    Therefore we will assume basic knowledge in this topic and refer to \textcite{schmidt2009} and \textcite{elstrodt2011} for a more detailed introduction to measure spaces, measurable functions, and integrals.
    Propositions and theorems will be given without proof.

    The underlying structure of probability theory, which connects it to measure theory, is the probability space.
    It is a measure space with a finite and normalized measure.
    This gives access to all the usual results of measure theory and furthermore unifies discrete and continuous distributions.
    \autocite[\ppno~193-195]{schmidt2009}

    \begin{definition}[Probability Space]
      A probability space is a measure space $\roundBrackets{\Omega, \mathscr{F}, P}$ such that $P(\Omega)=1$.
      In this case, we call $P$ the probability measure, $\mathscr{F}$ the set of all events, and $\Omega$ the set of all possible outcomes of a random experiment.
    \end{definition}
    % For our purposes, the set of possible outcomes $\Omega$ will be a finite or countable infinite set.
    % Hence, we can choose $\mathscr{F}$ to be the power set $\mathscr{P}(\Omega)$.
    Due to the complex definition
    % \footnote{Notation and symbols not directly defined are explained in the symbol table.}
    of a measure space, it is convenient to not have to explicitly specify the probability space when analyzing random experiments.
    Instead, we use random variables which are essentially measurable functions on a probability space \autocite[\pno~194]{schmidt2009}.
    For complicated cases, these will serve as observables for specific properties and will make the analysis much more intuitive.

    \begin{definition}[Random Variable]
      Let $(\Omega,\mathscr{F},P)$ be a probability space and $(\Sigma,\mathscr{A})$ a measurable space.
      A measurable function $\function{X}{\Omega}{\Sigma}$ is called a random variable.

      In this case, we denote with $P_X\define P\composition\inverse{X}$ the distribution and with $(\Sigma,\mathscr{A},P_X)$ the probability space of $X$.
      % We call $X(ω)$ for $ω\in\Omega$ a realization of $X$.
      % Let $\function{Y}{\Omega}{\Sigma}$ be another random variable.
      % $X$ and $Y$ are identically distributed if $P_X = P_Y$.
      Two random variables are identically distributed if they have the same distribution.
      Additionally, we say that $X$ is a real-valued random variable if $\Sigma = \setReal$ and $\mathscr{A} = \mathscr{B}(\setReal)$.
    \end{definition}
    From now on, if a random variable is defined then, if not stated otherwise, it is assumed there exists a proper probability space $(\Omega,\mathscr{F},P)$ and measurable space $(\Sigma, \mathscr{A})$.

    Another important concept of stochastics is known as independence.
    In \textcite{schmidt2009} it is defined for a family of events, a family of sets of events, and a family of random variables.
    If we think of random variables as observables then their independence means that their outcomes do not influence each other.
    % It makes only sense in the context of probability theory
    For our purposes, the general definition of all three forms of independence is distracting.
    In a computer, it makes no sense to talk about uncountably many elements.
    Therefore the following definition of independence takes only a countable sequence of random variables into account.
    Furthermore, to make it more understandable, this definition uses a theorem from \textcite[\pno~238]{schmidt2009} which characterizes the independence of random variables.
    % Here, we will use a simpler equivalent definition  because, for a computer, all we need are finite sequences of random variables.

    \begin{definition}[Independence]
      % Let $(\Omega,\mathscr{F},P)$ be a probability space.
      % Two events $A, B \in \mathscr{F}$ are independent if $P(A\cap B)=P(A)P(B)$.
      % Let $(\Sigma_i,\mathscr{A}_i)$ for $i\in\set{1,2}{}$ be measurable spaces and $X_i$ random variables with $X\define X_1\times X_2$.
      % They are called independent if $P_X = P_{X_1} \otimes P_{X_2}$.
      Let $I\subset\setNatural$ and $X_i$ be a random variable for all $i\in I$.
      Then these random variables are independent if the following equation holds for all finite subsets $J\subset I$ whereby we denote the respective random vector with $X_J \define \roundBrackets{X_i}_{i\in J}$.
      \[
        P_{X_J} = \bigotimes_{i\in J} P_{X_i}
      \]
    \end{definition}
    Typical observations of random sequences include the estimation of the expectation value and the variance.
    Both of these values are needed for analyzing PRNGs and the development of Monte Carlo simulations \autocite{landau2014}.
    Due to their deep connection to the integral, both of these moments are defined for real-valued random variables.
    We give the usual definitions based on \textcite[\ppno~274-276]{schmidt2009} in a simplified form.

    \begin{definition}[Expectation Value and Variance]
      Let $X$ be a real-valued random variable such that $\integral{\Omega}{}{\absolute{X}}{P}<\infty$.
      Then the expectation value $\expect X$ and variance $\var X$ of $X$ is defined in the following way.
      \[
        \expect X \define \integral{\Omega}{}{X(ω)}{P(ω)}
        \separate
        \var X \define \expect\roundBrackets{X - \expect X}^2
      \]
    \end{definition}
    To not rely on the underlying probability space directly, we want to be able to compute the expectation value through the respective distribution of the random variable.
    The theory of measure and integration gives the following proposition, also known as rule of substitution \autocite[\pno~276]{schmidt2009}.

    \begin{proposition}[Substitution]
    \label{proposition:substitution}
      Let $X$ be real-valued random variable and $\function{f}{\setReal}{\setReal}$ a measurable function such that $\integral{\Omega}{}{\absolute{f}}{P_X} < \infty$.
      Then the following equation holds.
      \[
        \expect(f\composition X) = \integral{\setReal}{}{f(x)}{P_X(x)}
      \]
      In particular, if $\expect \absolute{X} < \infty$ then the above equation can be reformulated as follows.
      \[
        \expect X = \integral{\setReal}{}{x}{P_X(x)}
      \]
    \end{proposition}
    The distribution of real-valued random variables is univariate and as a result can be described by so-called cumulative distribution functions (CDFs).
    The CDF intuitively characterizes the distribution and simplifies the analysis.
    Further, it can be proven that every CDF belongs to a univariate distribution.
    According to \textcite[\pno~246]{schmidt2009}, this is the theorem of correspondence.
    Sometimes it is even possible to define a probability density; a function that is the Lebesgue density of the respective distribution \autocite[\pno~255]{schmidt2009}.

    \begin{definition}[Probability Density and Cumulative Distribution Function]
      Let $X$ be a real-valued random variable.
      Then the respective cumulative distribution function is defined as follows.
      \[
        \function{F_X}{\setReal}{[0,1]}
        \separate
        F_X(x) \define P_X((-\infty,x])
      \]
      We call the function $\function{p}{\setReal}{[0,\infty)}$ a probability density of $X$ if for all $A\in\mathscr{B}(\setReal)$
      \[
        P_X(A) = \integral{A}{}{p(x)}{λ(x)}\ .
      \]
    \end{definition}
    % \begin{theorem}[Correspondence]
    %   Let $X$ be a real-valued random variable.
    %   There exists a unique monotone non-decreasing, right-continuous function $\function{F_X}{\setReal}{[0,1]}$ with
    %   \[
    %     \lim_{x\rightarrow -\infty} F_X(x) = 0
    %     \separate
    %     \lim_{x\rightarrow +\infty} F_X(x) = 1
    %   \]
    %   such that
    %   \[
    %     P_X((a,b]) = F(b) - F(a)
    %   \]
    % \end{theorem}
    As well as CDFs, probability densities can greatly simplify computations which are based on absolute continuous random variables.
    The following proposition, obtained from \textcite{schmidt2009}, shows the simplified computation of an expectation value through a Lebesgue integral.

    \begin{proposition}[Chaining]
    \label{proposition:chaining}
      Let $X$ be a real-valued random variable with $p$ as its probability density.
      If $\function{f}{\setReal}{\setReal}$ is a measurable function such that $\expect \absolute{f\circ X} < \infty$ then
      \[
        \expect \roundBrackets{f\composition X} = \integral{\setReal}{}{f(x)p(x)}{λ(x)}\ .
      \]
    \end{proposition}
    A last important theorem to name is the strong law of large numbers (SLLN).
    According to \textcite[\pno~13]{graham2013}, the principles of Monte Carlo methods are based on this theorem.
    It uses a sequence of identically and independently distributed (iid) random variables.
    Please note, there exist many more variations of this theorem.
    We will use a simplified version from \textcite{graham2013}.

    \begin{theorem}[Strong Law of Large Numbers]
    \label{theorem:slln}
      Let $(X_n)_{n\in\setNatural}$ be a sequence of iid real-valued random variables with finite expectation value $μ$.
      Then the following equation holds $P$-almost everywhere.
      \[
        \lim_{n\to\infty} \frac{1}{n}\sum_{i=1}^n X_i = μ
      \]
    \end{theorem}
  % subsection stochastics (end)

  % \subsection{Number Theory and Finite Fields} % (fold)
  % \label{ssub:finite_fields}

  % % subsection finite_fields (end)

  \subsection{The C++ Programming Language} % (fold)
  \label{sub:the_c_programming_language}
    As already told in the introductory section \ref{sec:introduction}, the C++ programming language is an adequate candidate for developing high-performance low-level structures while keeping the high degree of abstraction that makes the use of libraries easier and more consistent.
    C++ features multiple programming styles, like procedural programming, data abstraction, object-oriented programming, as well as generic programming which is also known as template metaprogramming \autocite{stroustrup2014,vandevoorde2018}.
    The type mechanism makes C++ a strongly typed language.
    To exploit this, we will always try to map problems to an abstract data structure.
    Furthermore, the built-in facilities of C++, such as templates, function overloading, type deduction, and lambda expressions, simplify the type handling and the generalization of algorithms.
    Additionally, C++ provides a standard library, called the standard template library (STL), consisting of header files providing default templates to use for a wide variety of problems.
    In this thesis, we will rely on the random utilities the STL exhibits.
    We will also assume basic knowledge in C++ and refer to \textcite{stroustrup2014} and \textcite{meyers2014} for a detailed introduction to the general usage of the language.
    A complete online reference of the language is given by \textcite{cppreference}.

    The C++ programming language keeps evolving by defining different language standards every three years which are published by the ISO C++ standardization committee.
    Newer standards typically introduce modern language features, fix old behavior and add new algorithms and templates to the STL.
    Hence, modern C++ separates into the standards C++11, C++14, and C++17 each published in the year 2011, 2014, and 2017, respectively.
    Currently, we are waiting for the C++20 standard specification which will provides even more advanced features concerning template metaprogramming and concurrency.
    At the time of writing this thesis, C++17 is the most modern specification and as a consequence it will be used throughout the code.
    \autocite{stroustrup2014,meyers2014,vandevoorde2018}

    To design the API of a library supplying vectorized RNGs and some advanced utilities, we will heavily rely on different template metaprogramming techniques.
    For getting deeper into the topic, we will refer to \textcite{vandevoorde2018} and again to \textcite{meyers2014}.
    Here, we will only be able to list the most important terms, techniques and rules that will be used throughout the code.

    \begin{description}
      \item[Template Argument Deduction]
        \textquote[\cite{cppreference}]{%
          In order to instantiate a function template, every template argument must be known, but not every template argument has to be specified.
          If possible, the compiler will deduce the missing template arguments from the function arguments.
        }
      \item[Overloading Function Templates]
        \textquote[\cite{vandevoorde2018}]{%
          Like ordinary functions, function templates can be overloaded.
          That is, you can have different function definitions with the same function name so that when that name is used in a function call, a C++ compiler must decide which one of the various candidates to call.
        }
      \item[Variadic Templates]
        \textquote[\cite{vandevoorde2018}]{%
          Since C++11, templates can have parameters that accept a variable number of template arguments.
          This feature allows the use of templates in places where you have to pass an arbitrary number of arguments of arbitrary types.
        }
      \item[Perfect Forwarding]
        C++11 introduced so-called move semantics to optimize specific copy operations by moving internal resources instead of creating a deep copy of them.
        Perfect forwarding is a template-based pattern that forwards the basic properties of a type concerning its reference and modification type.
        % \autocite{vandevoorde2018}
      \item[SFINAE]
        Substituting template arguments to resolve function template overloading could lead to errors by creating code that makes no sense.
        The principle \enquote{substitution failure is not an error} (SFINAE) states that in these circumstances the overload candidates with such substitution problems will be simply ignored.
        % \autocite{vandevoorde2018}
      \item[{\footnotesize \texttt{\textbf{decltype}}}]
        This is a specifier introducing an unevaluated context from which it is deducing the type of the given expression without actually evaluating the expression.
        % \autocite{stroustrup2014}
      \item[\texttt{\textbf{\footnotesize std::enable\_if\_t}}]
        This is a helper template from the STL to ignore function templates by using SFINAE under certain compile conditions.
        If the template argument evaluates to true then \code{std::enable\_if\_t} will evaluate to an actual type.
        Otherwise, it will have no meaning triggering the SFINAE principle for overloads.
        % \autocite{vandevoorde2018}
      \item[\texttt{\textbf{\footnotesize std::declval}}]
        The given STL function template is only declared, not defined and therefore cannot be called in evaluated contexts.
        It can be used as a placeholder for an object reference of a specific type.
        Typically, this routine will be inserted instead of a default constructor in the unevaluated context argument of \code{decltype}.
        % \autocite{vandevoorde2018}
      \item[Type Traits]
        Type traits are general functions defined over types to modify or evaluate them.
        In the STL a typical examples is given by \code{std::is_same_v} which is evaluating if two types are the same.
    \end{description}
  % subsection the_c_programming_language (end)
% section preliminaries (end)
\end{document}


%%=================================
%% Section 1.04: Asymptotics of K^+
%%=================================
\documentclass[../dissertation.tex]{subfiles}
\def\rest{\text{rest}}

\begin{document}

\section{Asymptotics of $K^+$}\label{sec1:AsympK}
A cursory inspection of the integrand of $G_\star^+$ ($\star = L \text{, or }R$)
indicates that $G_\star^+$ behavior
differs wildly for $x \in \mathbb R$ near zero, and $|x| > c$, where $c>0$ is any
fixed constant. While $1/p(\xi)$ decays roughly exponentially as $\xi \to -\infty$,
the fact that $1/p(\xi)$ decays as $1/\xi$ for $\xi \gg 1$ means the oscillatory term
$e^{ix\xi}$ in the integrand of $G_\star^+$ is imperative for the contour
integral in $G_\star^+$ of $e^{ix\xi} / p(\xi)$ to even have a chance for convergence.
In this section we study the ``nice'' case of $|x|>1$ and show that not only does
the integral in $G_\star^+$ converge in this scenario, it is rapidly decaying (as long
as $x$ stays away from zero). In the Section \ref{sec1:KSingularity}, we study
the behavior of $G_\star^+$ for $x$ near zero and show $G_\star^+$ has at worst a log
type singularity at $x=0$. Taken together, the results from this section
combined with the results from Section \ref{sec1:KSingularity} constitute a
proof of Theorem \ref{thm1:krep} from the introduction of this chapter.

Key to the analysis in both this section and Section \ref{sec1:KSingularity}
are the representation formulas \eqref{eq1:GFrep} proven in Section
\ref{sec1:GreensFunctions}. In particular, \eqref{eq1:GFrep} allows us
to reduce our analyses to a thorough study of $K^+$

To understand the properties of $K^+$, we study the convergence of the integral
\begin{align*}
	\int\limits_{\Sigma_{\sign(x)}} \frac{e^{ix\xi}}{p(\xi)} \, \mathrm{d}\xi,
\end{align*}
where $\Sigma_{\sign(x)} := \mathbb R + i \sign(x) \pi$.\label{sym1:SigRealLine}
Let $\Sigma\big(R,~\sign(x)\big)$\label{sym1:SigR} denote the contour
$(-R, R) + i \sign(x) \pi$. Recall
that $p(\xi)$ can be written as
\[
	p(\xi) = \xi - \zeta(\lambda)\( 1 - e^{-2\xi} \),
\]
which implies $p'(\xi) = 1 - 2 \zeta(\lambda) \, e^{-2\xi}$. In which case
\begin{align*}
	\int\limits_{\Sigma\big(R,~\sign(x)\big)} \frac{e^{ix\xi}}{p(\xi)} \, \mathrm{d}\xi
		&=  \left.\frac{e^{ix\xi}}{p(\xi)}\right|_{-R + i \sign(x)\pi}^{R + i \sign(x)\pi}
			- \frac{1}{ix} \int\limits_{\Sigma\big(R,~\sign(x)\big)} e^{ix\xi} \,
			\frac{p'(\xi)}{\big(p(\xi)\big)^2} \, \mathrm{d}\xi.
\end{align*}
Now
\begin{align*}
	p'(t \pm i \pi)
		= 1 - 2 \zeta(\lambda) e^{-2t} e^{\mp 2 \pi i}
		= 1 - 2 \zeta(\lambda) e^{-2t}
		= p'(t), \qquad t\in \RR.
\end{align*}
Further
\[
	\zeta(t \pm i \pi ) - \zeta(\lambda)
		= \frac{t}{1 - e^{-2t}} - \zeta(\lambda) \pm \frac{i\pi}{1 - e^{-2t}}
		= \zeta(t) - \zeta(\lambda) \pm \frac{i\pi}{1 - e^{-2t}}, \qquad t\in \RR,
\]
which implies
\begin{align} \label{eq1:shiftedP}
	p(t \pm i \pi)
		&= \big(\zeta(t \pm i \pi) - \zeta(\lambda) \big)\( 1- e^{-2t} \) \\
		&= \big(\zeta(t) - \zeta(\lambda)\big)\(1 - e^{-2t}\) \pm i\pi
		= p(t) \pm i \pi. \nonumber
\end{align}
Combining the two calculations above, we see that
\begin{align*}
	\frac{1}{ix} \int\limits_{\Sigma\big(R,~\sign(x)\big)} e^{ix\xi} \,
			\frac{p'(\xi)}{\big(p(\xi)\big)^2} \, \mathrm{d}\xi
		= \frac{e^{-|x|\pi}}{ix} \int_{-R}^R e^{ixt}
			\frac{p'(t)}
			{\big( p(t) + i \sign(x) \pi \big)^2} \, \mathrm{d}t.
\end{align*}
Thus,
\[
	\lim_{R\to\infty} \frac{1}{\left|p\big(\pm R + i \sign(x)\pi\big)\right|}=0
\]
and, formally,
\begin{align} \label{eq1:RemainderIntegralBnd}
	\left| \int\limits_{\Sigma_{\sign(x)}} \frac{e^{ix\xi}}{p(\xi)} \, \mathrm{d}\xi \right|
		\leq \frac{e^{-|x|\pi}}{|x|} \int_{\RR}
			\frac{\left| p'(t) \right|}{p(t)^2 + \pi^2} \, \mathrm{d}t,
\end{align}
where we proceed to establish the convergence of the integral on the right-hand
side of \eqref{eq1:RemainderIntegralBnd}. Note that $p'(t) = 0$ only when
$t = \frac{1}{4} \ln\big( \zeta(\lambda) \big)$. Further, given
$p''(t) = 4\zeta(\lambda)\,e^{-2t} > 0$ for all real $t$,
\[
	|p'(t)| =
		\begin{cases}
			-p'(t), & t < t_0 \\
			p'(t), & t \geq t_0,
		\end{cases}
\]
where $t_0:= \frac{1}{4} \ln\big( \zeta(\lambda) \big)$. Given this fact, we now proceed
to evaluate the integral on the right-hand side of \eqref{eq1:RemainderIntegralBnd}
through $u$-substitution by setting $u = p(t)$. Observe that
\begin{align}
	\int \frac{p'(t)}{p(t)^2 + \pi^2} \, \mathrm{d}t
		= \int \frac{1}{u^2 + \pi^2}
		= \frac{1}{\pi} \arctan\( \frac{u}{\pi} \) + C
		= \frac{1}{\pi} \arctan\( \frac{p(t)}{\pi} \) + C.
\end{align}
So, for $R > |t_0|$,
\[
	\int_{-R}^{t_0} \frac{|p'(t)|}{p(t)^2 + \pi^2}\,dt
		= - \int_{-R}^{t_0} \frac{p'(t)}{p(t)^2 + \pi^2}
		= -\frac{1}{\pi} \left[ \arctan\(\frac{p(t_0)}{\pi}\)
			- \arctan\(\frac{p(-R)}{\pi}\)\right],
\]
and
\[
	\int_{t_0}^R \frac{|p'(t)|}{p(t)^2 + \pi^2}\,dt
		= \frac{1}{\pi} \left[\arctan\(\frac{p(R)}{\pi}\)
			- \arctan\(\frac{p(t_0)}{\pi}\)\right].
\]
Now
\[
	\lim_{R\to \infty} p(\pm R)
		= \lim_{R\to \infty} \left[ \pm R - \zeta(\lambda) \( 1 - e^{\mp 2R} \) \right]
		= \infty,
\]
which implies that
\begin{align*}
	\lim_{R\to \infty} \int_{-R}^R \frac{|p'(t)|}{p(t)^2 + \pi^2}\,dt
		&= \lim_{R\to\infty} \(\int_{-R}^{t_0} \frac{|p'(t)|}{p(t)^2 + \pi^2}\,dt
			+ \int_{t_0}^{R} \frac{|p'(t)|}{p(t)^2 + \pi^2}\,dt \)\\
		&= \frac{1}{\pi}\left[ \lim_{R\to\infty} \arctan\(\frac{p(R)}{\pi}\)
				+\lim_{R\to\infty} \arctan\(\frac{p(-R)}{\pi}\) \right. \\
		&\qquad\quad- \left. 2 \arctan\(\frac{p(t_0)}{\pi}\) \right] \\
		&= 1 - \frac{2}{\pi} \arctan\( \frac{p(t_0)}{\pi} \)
\end{align*}

Since
\[
	p(t_0)
		= \frac{1}{4} \ln\zeta(\lambda) - \zeta(\lambda)
			+ \big(\zeta(\lambda) \big)^{\frac{1}{2}},
\]
$p(t_0) \to - \infty$ as $\lambda \to \pm \infty$, and
\[
	\lim_{\lambda\to\pm\infty} \int_\RR \frac{|p'(t)|}{p(t)^2 + \pi^2}\,dt = 2.
\]
Putting everything together, we see that
\begin{align}\label{eq1:Kasymp}
	K^+(x; \lambda) = \mc O\left( \frac{e^{-|x| \pi}}{|x|} \right)
\end{align}
for $|x| \geq 1$.
% \begin{align}\label{eq1:FinalGffromula}
% 	G_L^+(x, \lambda, 1) =
% 		\begin{cases}
% 			i \frac{1 - e^{2\lambda}}{2\lambda\,e^{2\lambda} - e^{2\lambda}+1}
% 				+ i \frac{1 - e^{2\lambda}}{2\lambda - e^{2\lambda}+1} e^{ix\lambda}
% 				+ \mc O\( \frac{e^{-|x| \pi}}{|x|} \), & x > 1 \\
% 			~\\
% 			\mc O\( \frac{e^{-|x| \pi}}{|x|} \), & x < -1,
% 		\end{cases}
% \end{align}
% where the $\mc O\( \frac{e^{-|x| \pi}}{|x|} \)$ terms are uniformly bounded in
% $\lambda$.

% Finally, since
% \[
% \begin{array}{ccc}
% 	\ds \lim_{\lambda\to-\infty} \frac{1 - e^{2\lambda}}{2 \lambda e^{2\lambda} - e^{2\lambda}+1} = 1,
% 	&\qquad&
% 	\ds \lim_{\lambda\to\infty} \frac{1 - e^{2\lambda}}{2 \lambda e^{2\lambda} - e^{2\lambda}+1} = 0, \\
% 	\ds \lim_{\lambda\to-\infty} \frac{1 - e^{2\lambda}}{2 \lambda - e^{2\lambda}+1} = 0,
% 	&\qquad&
% 	\ds \lim_{\lambda\to\infty} \frac{1 - e^{2\lambda}}{2 \lambda - e^{2\lambda}+1} = 1,
% \end{array}
% \]
% we see from \eqref{eq1:Gffraw} that $|G_L^+(x; \lambda)|$ is uniformly bounded in $\lambda$.

\end{document}


\section{Conclusion and Future Work}

In this paper, we present a novel approach for streamline estimation on uncertain steady vector fields. We model the particle tracing problem using a Bayesian framework and solve it using particle filtering. This solution has been tested on two flow field data sets. Compared to the previous methods, our method gives more concentrated traces, and can produce paths of high probability closer to the ground truth in similar computation times. There are several directions for future research. We will extend our framework to handle time-varying vector datasets, where the specific characteristics of pathlines and streaklines will be taken into account. Secondly, extensions to other flow visualization techniques like FTLE or LIC will be studied.


\chapter{Image processing and tracking of all the cells in the given $\textbf{\textit{E.coli}}$ movie using u-track 2.0, and perform data analysis} % Main chapter title

\label{Part1_chapter} % For referencing the chapter elsewhere, use \ref{Chapter1}

\section{Get data form movie}

For the video: 1 pixel = 0.65 $\mu m$, 1 frame = 0.1 second.\\
The data were extracted by u-track to track the cell by method of Single-Particles and Gaussian Mixture-model Fitting. There are more than 14000 tracks.

\newpage
\section{Single bacteria trajectory}

Form 14721 tracks, I chose the track with the biggest number of un-NAN values and removed all of the NAN value to plot the movement of this single bacteria.

%----------------------------------------------------------------------------------------

\begin{figure}[H]
\centering
\includegraphics[width=1\linewidth]{Figures/P3_fig1.png}
\caption{Single Bacteria Random Walk Trajectory}
\label{P2_fig1}
\end{figure}

\section{Bacteria population trajectory}
Also track all the cells from the data, to calculate the population motions. The statistical results are shown in Figure 3.2.
\begin{figure}[H]
\centering
\includegraphics[width=0.75\linewidth]{Figures/P3_fig2.png}
\caption{The ``mean displacement'' and ``mean displacement-square'' overtime for the cells in vedio}
\label{P2_fig2}
\end{figure}

\section{Fitting the Data with the Model}
For a series of scatter points $(x_i, y_i)(i=1,2,...,n)$, if a direct proportional function was used to fit the data, we have:\\
\begin{equation*}
\begin{aligned}
\centering
y     &= \beta x \\
\beta &= \sum_{i=1}^n x_i y_i \Big / \sum_{i=1}^n x_i^2 \\
\end{aligned}
\end{equation*}
As for our model: \\
\begin{equation*}
\begin{aligned}
\centering
\frac{1}{n} \sum_{i=1}^{n}x_i(t)^2 &= v_x^2t \\
\frac{1}{n} \sum_{i=1}^{n}y_i(t)^2 &= v_y^2t \\
\end{aligned}
\end{equation*}
So there are the fitting model: \\
\begin{equation*}
\begin{aligned}
\centering
\frac{1}{n} \sum_{i=1}^{n}x_i(t)^2 &= \beta_x t = v_x^2t \\
\frac{1}{n} \sum_{i=1}^{n}y_i(t)^2 &= \beta_y t = v_y^2t \\
\beta_x = \sum_{i=1}^n & t_i x_i \Big / \sum_{i=1}^n t_i^2 \\
\beta_y = \sum_{i=1}^n & t_i y_i \Big / \sum_{i=1}^n t_i^2 \\
\end{aligned}
\end{equation*}
We get: \\
\begin{equation*}
\begin{aligned}
\centering
\beta_x &= 0.9862 \mu m^2 /s^2 \\
v_x &= 0.9931 \mu m /s \\
\beta_y &= 0.8366 \mu m^2 /s^2 \\
v_y &= 0.9147 \mu m /s \\
\end{aligned}
\end{equation*}
%betaX = 0.9862
%betaY = 0.8366
\begin{figure}[H]
\centering
\includegraphics[width=1\linewidth]{Figures/P3_fig3.png}
\caption{Fitting the Data by Model}
\label{P3_fig3}
\end{figure}
%----------------------------------------------------------------------------------------

%----------------------------------------------------------------------------------------

%The \code{biblatex} package is used to format the bibliography and inserts references such as this one \parencite{Reference1}. The options used in the \file{main.tex} file mean that the in-text citations of references are formatted with the author(s) listed with the date of the publication. Multiple references are separated by semicolons (e.g. \parencite{Reference2, Reference1}) and references with more than three authors only show the first author with \emph{et al.} indicating there are more authors (e.g. \parencite{Reference3}). This is done automatically for you. To see how you use references, have a look at the \file{Chapter1.tex} source file. Many reference managers allow you to simply drag the reference into the document as you type.


\section{Basics of Plotting Functions}
\begin{figure}[!htbp]
    \centering
    \includegraphics[width=0.5\textwidth]{../src/lab_04_plot_1.pdf}
    \caption{Plots of Special Curves}
    \label{fig:special_curves}
\end{figure}

\newpage
\section{Set Properties for Plotting}
\begin{figure}[!htbp]
    \centering
    \includegraphics[width=0.5\textwidth]{../src/lab_04_plot_2.pdf}
    \caption{Plot of Sine Functions}
    \label{fig:sine_functions}
\end{figure}

\newpage
\section{Plotting Piecewise Function on Different Scales}
\begin{figure}[!htbp]
    \centering
    \includegraphics[width=0.5\textwidth]{../src/lab_04_plot_3.pdf}
    \caption{Plots of Piecewise Function on Different Scales}
    \label{fig:piecewise}
\end{figure}

\newpage
\section{Script and Function files}
\subsection{Script file}
\lstinputlisting[style=MATLAB]{../src/lab_04_script.m}


\section*{Preface}
A precise description of a programming language is a prerequisite for its
implementation and for its use.  The description can take many forms,
each suited to a different purpose.  A common form is a reference manual,
which is usually a careful narrative description of the meaning of each
construction in the language, often backed up with a formal presentation
of the grammar (for example, in Backus-Naur form).
This gives the programmer enough
understanding for many of his purposes.  But it is ill-suited for use
by an implementer, or by someone who wants to formulate laws for
equivalence of programs, or by a programmer who wants to design programs with
mathematical rigour.

This document is a formal description of both the {\sl grammar} and the
{\sl meaning} of a language which is both designed for large projects and
widely used.  As such, it aims to serve the whole community of people
seriously concerned with the language.  At a time when it is increasingly
understood that programs must withstand rigorous analysis, particular for
systems where safety is critical, a rigorous language presentation is even
important for negotiators and contractors; for a robust program
written in an insecure language is like a house built upon sand.

Most people have not looked at a rigorous language presentation before.
To help them particularly, but also to put the present work in perspective
for those more theoretically prepared,  it will be useful here to say something
about three things: the nature of Standard ML, the task of language definition
in general, and the form of the present Definition.

\subsubsection*{Standard ML}
Standard ML is a functional programming language, in the sense that the
full power of mathematical functions is present.  But it grew in response
to a particular programming task, for which it was equipped also
with full imperative power, and a sophisticated exception mechanism.
It has an advanced form of parametric modules, aimed at organised
development of large programs.  Finally it is strongly typed, and it was
the first language to provide a particular form of polymorphic type which
makes the strong typing remarkably flexible.  This combination of
ingredients has not made it unduly large, but their novelty has been
a fascinating challenge to semantic method (of which we say more below).

ML has evolved over fourteen
years as a fusion of many ideas from many people. This evolution is
described in some detail in Appendix E of the book, where also we
acknowledge all those who have contributed to it, both in design
and in implementation.

`ML' stands for
{\sl meta language}; this is the term logicians use for a language in which
other (formal or informal) languages are discussed and analysed.
Originally ML was conceived as a medium for finding and performing
proofs in a logical language.   Conducting rigorous argument as dialogue
between person and machine has been a strong research interest at Edinburgh
and elsewhere, throughout these fourteen years. The difficulties are enormous,
and make stern demands
upon the programming language which is used for this dialogue.   Those who are
not familiar with computer-assisted reasoning may be surprised that a
programming language, which was designed for this rather esoteric activity,
should ever lay claim to being {\sl generally} useful.
On reflection, they should not be surprised.  LISP is a prime example of
a language invented for esoteric purposes and becoming widely used.  LISP
was invented for use in artificial intelligence (AI); the important thing
about AI here is not that it is esoteric,  but that
it is difficult and varied; so much so, that anything which works well for
it must work well for many other applications too.

The same can be said about the initial purpose of ML, but with a different
emphasis.  Rigorous proofs are complex things, which need varied
and sophisticated presentation -- particularly on the screen in interactive
mode. Furthermore the proof methods, or
strategies, involved are some of the most complex algorithms which we know.
This all applies equally to AI, but one demand is made more strongly
by proof than perhaps by any other application: the demand for rigour.

This demand established the character of ML.  In order to be sure that,
when the user and the computer claim to have together performed a rigorous
argument, their claim is justified, it was seen
that the language must be strongly typed. On the other hand, to be
useful in a difficult  application, the type system had to be rather
flexible, and permit the machine to guide the user rather than impose
a burden upon him.   A reasonable solution was found, in which the machine
helps the user significantly  by
inferring his types for him. Thereby the machine also confers complete
reliability on his programs, in this sense:
If a program claims that a certain result
follows from the rules of reasoning which the user has supplied, then the
claim may be fully trusted.

The principle of inferring useful structural information about programs
is also represented, at the level of program modules, by the inference of
{\sl signatures}.
Signatures describe the interfaces between modules, and are vital for robust
large-scale programs.  When the user combines modules, the signature
discipline prevents him from mismatching their interfaces.  By programming
with interfaces and parametric modules, it becomes possible to focus on the
structure of a large system, and to compile parts of it in isolation from
one another -- even when the system is incomplete.

This emphasis on types and signatures has had a profound effect on the
language Definition. Over half this document is devoted to inferring types
and signatures for programs.  But the method used is exactly the same as for
inferring what {\sl values} a program delivers; indeed, a type or signature is
the result of a kind of abstract evaluation of a program phrase.

In designing ML,
the interplay among three activities -- language design, definition and
implementation -- was extremely close. This was particularly true for the
newest part, the parametric modules. This part of the language grew from an
initial proposal by David MacQueen, itself highly developed; but both
formal definition and implementation had a strong influence on the detailed
design.  In general, those who took part in the three activities cannot now
imagine how they could have been properly done separately.

\subsubsection*{Language Definition}
Every programming language presents  its own conceptual view of
computation.  This view is usually indicated by the names used for the phrase
classes of the language, or by its keywords: terms like package, module,
structure, exception, channel, type, procedure, reference, sharing, \ldots.
These terms also have their abstract counterparts,  which may be called
{\sl semantic objects}; these are what people really have in mind when they use
the language, or discuss it, or think in it.  Also, it is these objects,
not the syntax, which represent the particular conceptual view of each
language; they are the character of the language.  Therefore a definition
of the language must be in terms of these objects.

As is commonly done in programming language semantics, we shall loosely
talk of these semantic objects as {\sl meanings}. Of course, it is
perfectly possible to understand the semantic theory of a language, and
yet be unable to to understand the meaning of a particular program, in the
sense of its {\sl intention} or {\sl purpose}.  The aim of a language
definition is not to formalise everything which could possibly be called the
meaning of a program, but to establish a theory of semantic objects
upon which the understanding of particular programs may rest.

The job of a language-definer is twofold.  First -- as we have already
suggested -- he must create a world of meanings appropriate for the language,
and must find a way of saying
what these meanings precisely are.  Here, he meets a problem; notation
of {\sl some} kind must be used to denote and describe these meanings --
but not a {\sl programming language} notation, unless he is passing the
buck and defining one programming language in terms of another. Given
a concern for rigour, mathematical notation is an obvious choice.  Moreover,
it is not enough just to
write down mathematical definitions. The world of meanings only becomes
meaningful if the objects possess nice properties, which make them tractable.
So the language-definer really has to develop a small {\sl theory} of his
meanings, in the same way that a mathematician develops a theory.
Typically, after initially defining some objects, the mathematician goes on to
verify properties which indicate that they are objects worth studying.
It is this part, a kind of scene-setting, which the language-definer shares
with the
mathematician.  Of course he can take many objects and their theories
directly from mathematics,  such as functions, relations,
trees, sequences, \ldots. But he must also give some special theory for the
objects which make his language particular, as we do for types, structures and
signatures in this book; otherwise his language definition may be
formal but will give no insight.

The second part of the definer's job is to define {\sl evaluation} precisely.
This means that he must define at least {\sl what} meaning, $M$, results
from evaluating any phrase $P$ of his language (though he need not explain
exactly {\sl how} the meaning results; that is he need not give the full
detail of every computation).  This part of his job must be formal
to some extent, if only because the phrases $P$ of his language are indeed
formal objects.  But there is another reason for formality.  The task is
complex and error-prone, and therefore demands a high level of explicit
organisation (which is, largely, the meaning of `formality'); moreover,
it will be used to specify an equally complex, error-prone and formal
construction: an implementation.

We shall now explain the keystone of our semantic method.  First, we need a
slight but important refinement. A phrase $P$ is never evaluated {\sl in
vacuo} to a meaning $M$, but always {\sl against a background}; this
background -- call it $B$ -- is itself a semantic object,
being a distillation of the meanings
preserved from evaluation of earlier phrases (typically variable declarations,
procedure declarations, etc.).  In fact evaluation is background-dependent
-- $M$ depends upon $B$ as well as upon $P$.

The keystone of the method, then, is a certain kind of assertion about
evaluation; it takes the form
\[ B\vdash P\Rightarrow M\]
and may be pronounced: `Against the background $B$, the phrase $P$ evaluates
to the meaning $M$'.  {\sl The formal purpose of this Definition is no more,
and no less, than to decree exactly which assertions of this form are true.}
This could be achieved in many ways. We have chosen to do it in a structured
way, as others have, by giving rules which allow assertions about a
{\sl compound} phrase $P$ to be inferred from assertions about its
{\sl constituent} phrases $P_1,\ldots,P_n$.

\subsubsection*{The form of the Definition\footnote{
   The Definition has evolved through a sequence of three previous versions,
   circulated as Technical Reports.  For those who have followed the
   sequence, we should point out that the treatment of {\sl equality types}
   and of {\sl admissibility} has been slightly modified in this publication
   to meet the claim for principal signatures.  The changes are mainly
   in Sections~4.9, 5.5 and 5.13 and in the inference rules~19, 20,
   29 and 65.}}
We have written the Definition in a form suggested by the previous remarks.
That is, we have defined our semantic objects in mathematical
notation which is completely independent of Standard ML, and we have
developed just enough of their theory to give sense to our rules of
evaluation.
   Following another suggestion above, we have factored our task
by describing {\sl abstract} evaluation -- the inference and checking of
types and signatures (which can be done at compile-time) -- completely
separately from {\sl concrete} evaluation.
It really is a factorisation, because a {\sl full} value in all its glory --
you can think of it as a concrete object with a type
attached -- never has to be presented.

   The resulting document is, we hope, valuable as the essential point of
reference for Standard ML.  If it is to play this role well, it must
be supplemented by other literature.  Some expository books have already been
written, and this Definition will be useful as a
background reference for their readers. We have also become convinced, while
writing the Definition, that we could not discuss many questions without
making it far too long.  Such questions are: Why were certain design choices
made?  What are their implications for programming?
Was there a good alternative meaning for some constructs, or was our
hand forced?  What different forms of phrase are equivalent? What is the proof
of certain claims?  Many of these
questions will not be answered by pedagogic texts either. So we are writing
a Commentary on the Definition which will assist people in reading it, and
which will serve as a bridge between the Definition and other texts.
\begin{flushright} Edinburgh\\August 1989 \end{flushright}