\section{Traits}

\subsection{Linearization}
\begin{lstlisting}[language=Scala]
trait A { println(">>A")
def print = { println("A") }
}
trait B { println(">>B")
def print = { println("B") }
}
trait C extends A with B { println(">>C")
override def print = { println("C"); super.print }
}
trait D extends B with A { println(">>D")
override def print = { println("D"); super.print }
}
class E extends D with C { println(">>E")
override def print = { println("E"); super.print }
}
\end{lstlisting}

\textbf{Which text will be printed if a new instance of class E is created (which implicitly triggers the invocation of the constructors)?} \\
E | C B A | D A B => E C D A B
\begin{lstlisting}[language=Scala]
scala> val e = new E()
>>B
>>A
>>D
>>C
>>E
\end{lstlisting}

\textbf{Which text will be printed if method print is invoked on an instance of class E?} \\
A.print does not perform a super-call
\begin{lstlisting}[language=Scala]
scala> e.print
E
C
D
A
\end{lstlisting}

\textbf{What is the output when method print is invoked on an instance of class E if class C has been changed as follows:}
\begin{lstlisting}[language=Scala]
trait C extends A with B { println(">>C")
    override def print = { println("C"); super[B].print }
}
\end{lstlisting}

\begin{lstlisting}[language=Scala]
scala> e.print
E
C
B
\end{lstlisting}

\clearpage
\subsection{Observer Pattern with Traits}

\begin{lstlisting}
object ObserverTest1 {

  class Model(private var value: Int) {
    def getValue = value
    def setValue(value: Int) = { this.value = value }
  }

  trait Observer {
    def update(subject: Any)
  }

  trait Observable {
    private var observers: List[Observer] = Nil

    def addObserver(obs: Observer) = { observers = obs :: observers }
    def removeObserver(obs: Observer) = { bservers = observers.filterNot(_ == obs) }
    def notifyObservers = observers.foreach(_.update(this))
  }

  class Counter extends Observer {
    var counter = 0
    def update(subject: Any) = {
      println(s"update called with $subject")
      counter += 1
    }
  }

  def main(args: Array[String]): Unit = {
    val m = new Model(0) with Observable {
      override def setValue(x: Int) = {
        super.setValue(x)
        notifyObservers
      }
    }

    val c = new Counter
    m.addObserver(c)
    m.setValue(3);
    m.setValue(5);
    println(c.counter)
  }

}
\end{lstlisting}

\clearpage


\documentclass{article}
\usepackage{subfiles}
\input{settings.tex}

\title{Notes}
\author{Elijah Sheridan}

\begin{document}

\maketitle
\tableofcontents
\newpage

\section{Example Section}

And right here we have a \vocab{vocabulary word}.

\begin{define}[Test]
The quick brown fox jumped over that lazy dog over there. $p^2c^4 = e^2m^2 + m^2c^2$
\end{define}

\begin{theo}[Test]
The quick brown fox jumped over that lazy dog over there. $p^2c^4 = e^2m^2 + m^2c^2$

\end{theo}

\begin{res}[Test]
The quick brown fox jumped over that lazy dog over there. $p^2c^4 = e^2m^2 + m^2c^2$

\end{res}

\begin{cor}[Test]
The quick brown fox jumped over that lazy dog over there. $p^2c^4 = e^2m^2 + m^2c^2$

\end{cor}

\begin{ex}[Test]
The quick brown fox jumped over that lazy dog over there. $p^2c^4 = e^2m^2 + m^2c^2$
\end{ex}

\begin{moral}
Testing the moral
\end{moral}

\end{document}


\section{Metrical Synthesis}
\label{sec:opt}

In the parameter tuning phase of the algorithm we can leverage existing optimization techniques to find the best constants $\constants$ for a given program structure $p$.
One such optimization technique is gradient descent.
If the $\distFxn$ is not differentiable, for example using I/O, gradient descent may not be an appropriate option.
This stage can use any parameter tuning approach in place of gradient descent.


% When using TeXShop on the Mac, let it know the root document. The following must be one of the first 20 lines.
% !TEX root = ../design.tex

\chapter[Random Forests]{Random Forests: Classification and Regression}
\begin{moduleinfo}
\item[Author] \href{mailto:pjayaram@pivotal.io}{Preethi Jayaram}
\item[Author] \href{mailto:xfeng@pivotal.io}{Feng, Xixuan (Aaron)}
\item[History]
    \begin{modulehistory}
            \item[v0.1] Initial version: introduction, theory, and interface
    \end{modulehistory}
\end{moduleinfo}

% Abstract. What is the problem we want to solve?
\section{Introduction} % (fold)
\label{sec:introduction}

Tree-based methods provide a simple and human-interpretable model for classifying data.
Decision Trees, for example, use splitting rules to segment the predictor space into simple
regions that are then summarized in a tree form. However, they tend to over-fit the data,
resulting in poor prediction accuracies. One way to mitigate this problem is by building an
ensemble of classifiers, each of which produces a tree model on some combination of the
input data. The results of these models are then combined to yield a single prediction,
which, although at the expense of some loss in interpretation, have been found to be
highly accurate. Such methods of using multiple decision trees to make predictions
are called random forest methods.

\subsection{Basics of Random Forests} % (fold)
\label{sub:basics_of_random_forests}
Random forests operate by constructing many trees at training time using bootstrapped
samples from the training data. During prediction, they output the class that is the
mean (regression) or mode (classification) of the predictions output by the individual trees.

Each tree is grown as follows:
\begin{enumerate}
    \item If the number of training samples is N, we sample, with replacement, N cases at random.
    This forms the training set for the current tree.
    \item Each time a split is considered, a random sample of $m$ predictors is chosen as split
    candidates from the full set of $p$ predictors such that $m \ll p$. Typically, we
    choose $m \approx \sqrt{p}$. One of the m predictors is then used to split the node.
    \item The tree is grown to the maximum size without pruning.
\end{enumerate}

[10] proves that the generalization error, or the misclassification rate, has a limiting value
and that, random forests do not over-fit the data.

\subsection{Out-of-bag error (oob) error estimate}
The Out-of-bag error is an unbiased estimate of the test set error. While growing each tree during
training, about one-third of the samples are left out. These are used to get a test-set
classification for each sample in about one-third of the trees. The average prediction $c$ for each
sample is calculated as the average of the predictions output by individual trees.
The number of times $c$ is not equal to the true class of the sample, averaged
over all the samples, is the oob error estimate. Such an unbiased error estimate removes
the need for cross-validation or a separate test set for calculating test set error.

\subsection{Variable importance}
Random forests can be used to rank the importance of variables in the dataset.
For a given dataset $D_{n} = \{(X_i, Y_i)\}_{i=1}^{n}$, first, a random forest is fit to the data.
To measure the importance of the $j$-th feature after training, the values of the $j$-th feature are
permuted among the training data, and the oob error is calculated on this perturbed dataset. The
importance of the $j$-th feature is calculated by averaging the difference between the oob error
estimate before and after permuting the $j$-th feature. The average is normalized by the
standard deviations of the differences.

\subsection{Proximities}
Proximity matrices are a useful tool in clustering and outlier detection.
Random forests can be used to calculate pair-wise proximities between input samples.
If there are N training samples, the proximity matrix is created as an NxN matrix. After a
tree is grown, both the training and oob data are sent down the tree. If two different samples
end up in the same leaf node, their proximity is increased by one. After repeating this process
for all samples, the proximity values are normalized by diving by the number of trees.

\paragraph{Advantages of Random forests:}

\begin{enumerate}
    \item It is one of the most accurate learning algorithms available. For many data sets, it
    produces a highly accurate classifier.
    \item It runs efficiently on large databases.
    \item It can handle thousands of input variables without variable deletion.
    \item It gives estimates of what variables are important in the classification.
    \item It generates an internal unbiased estimate of the generalization error as the forest
    building progresses.
    \item It has an effective method for estimating missing data and maintains accuracy
    when a large proportion of the data are missing.
    \item It computes proximities between pairs of cases that can be used in clustering, locating
    outliers, or (by scaling) give interesting views of the data.
    \item The capabilities of the above can be extended to unlabeled data, leading to
    unsupervised clustering and outlier detection. Proximity matrix calculated from the random forest
    for unlabeled data can be used for clustering and outlier detection.
\end{enumerate}
% subsection basics_of_random_forests (end)

\section{Interface} % (fold)
\label{sec:interface}
The interface to the random forest functionality includes forest\_train and forest\_predict as
the training and prediction functions, and forest\_display as the visualization function.

\subsection{Training} % (fold)
\label{sub:training}
\begin{sql}
    SELECT forest_train(
            training_table_name,
            output_table_name,
            id_col_name,
            dependent_variable,
            list_of_features,
            list_of_features_to_exclude,
            grouping_cols,
            num_max_trees,
            num_random_features,
            max_depth,
            min_split,
            min_bucket,
            verbose
    )
\end{sql}

\paragraph{Arguments:}

The descriptions of the arguments are the same as found in Decision Trees with the exception
of the following:

\begin{itemize}
    \item \emph{num\_max\_trees}: Maximum number of trees to grow while building the random forest. Default: 500
    \item \emph{num\_random\_features}: Number of features to randomly select at each split. Default=$\sqrt{p}$ for
    classification, and p/3 for regression, where p is the total number of features.
    \item There will be no parameters for cross-validation and pruning as random forests build complete
    trees without pruning, and the technique of cross-validation is inherent.
\end{itemize}
% subsection training (end)

\subsection{Prediction} % (fold)
\label{sub:prediction}
\begin{sql}
    SELECT forest_predict(
            random_forest_model,
            new_data_table,
            output_table,
            type
    )
\end{sql}
The descriptions of all the arguments are the same as found in Decision Trees.
% subsection prediction (end)
\\
\subsection{Training Output} % (fold)
\label{sub:training_out}
The function forest\_train produces three output tables, one each for the model, summary
and groups, as described below:

\paragraph{Model table:}
The structure of the model table remains mostly similar to the one constructed by Decision trees.
For Random forests, we provide two additional fields: 'sample\_id' and 'group\_id'.\\
sample\_id refers to the id of the bootstrap sample, which is a number ranging from 1 to
num\_trees, where num\_trees is the number of trees that were actually grown.
group\_id is an integer assigned to each unique combination of values of grouping columns.
As one can see, a tree can uniquely be identified by using the sample\_id as well as
particular values for the grouping columns. To make it easier for the user to retrieve a tree,
we assign a group\_id instead of the user having to list all grouping columns and their values.
The total number of trees in the model table will be equal to the number of groups multiplied
by the number of bootstrap samples.

\begin{itemize}
    \item \emph{group\_id}: If grouping columns were provided, this refers to the Id of the group
    for which this tree was generated. The specific grouping columns and values can be
    obtained from the 'Groups' table.
    \item \emph{sample\_id}: Id of the (bootstrap) sample for which this tree was generated.
    \item \emph{tree}: Trained decision tree model stored in bytea8 format.
    \item \emph{cat\_levels\_in\_text}: Ordered levels of categorical variables.
    \item \emph{cat\_n\_levels}: Number of levels for each categorical variable.
    \item \emph{tree\_depth}: Depth of the tree.
\end{itemize}

\paragraph{Summary table:}
This table stores details that are common across all random forest models created by the
training function.
\begin{itemize}
    \item \emph{method\_name}: Name of training method, 'forest\_train' in this case.
    \item \emph{is\_classification}: Indicates if the model is for classification/regression.
    \item \emph{training\_table\_name}: Name of table containing training data.
    \item \emph{output\_table\_name}: Name of table containing the model.
    \item \emph{id\_col\_name}: The Id column name.
    \item \emph{dependent\_variable}: The dependent variable.
    \item \emph{features}: The independent variables.
    \item \emph{cat\_features\_str}: The list of categorical feature names as a comma-separated string.
    \item \emph{con\_features\_str}: The list of continuous feature names as a comma-separated string.
    \item \emph{grouping\_cols\_str}: Names of grouping columns.
    \item \emph{n\_groups}: Number of groups in training.
    \item \emph{failed\_groups}: Number of failed groups in training.
    \item \emph{n\_rows}: Total number of rows processed across all groups.
    \item \emph{n\_rows\_skipped}: Total number of rows skipped across groups.
    \item \emph{dep\_list\_str}: Distinct levels of dependent variable in the case of classification.
    \item \emph{dep\_type}: Type of dependent variable.
\end{itemize}

\paragraph{Groups table:}
This table stores statistics on a per-group basis. It has one row for each group (which in turn
corresponds to the random forest model for that group).
\begin{itemize}
    \item \emph{grouping\_cols}: Zero or more grouping columns depending upon user input.
    \item \emph{group\_id}: If grouping columns were provided, this refers to the Id of the group
    that this row of statistics corresponds to.
    \item \emph{num\_trees}: Actual number of trees constructed by the model. This should usually be the same
    as num\_max\_trees, but we could optimize the implementation to limit the number of trees to grow
    based on the observed oob error during construction of the random forest model for each group.
    \item \emph{oob\_error}: Out-of-bag error estimate for the model.
    \item \emph{variable\_importance}: A matrix containing the permutation-based importance measures.
    The rows correspond to the different predictors, and the columns correspond to various importance measures.
    Importance measures include Mean Decrease in MSE (for regression) and Mean Decrease in Classification
    Accuracy for classification. Additionally, we may be able to add Mean Decrease in Gini criterion for classification.
    \item \emph{proximity}: A matrix containing proximity measures among the input, based
    on the number of times pairs of input points end up in the same terminal node. Note that this
    feature being Priority 2, will not be available in the first iteration.
\end{itemize}
% subsection training_out (end)

\subsection{Prediction Output}
\label{sub:prediction_out}
The function forest\_predict produces one output table containing a list of predictions, which
are either class probabilities, or classification prediction outputs. In the case of
classification with multiple classes, an array of class probabilities are produced as output.
The output table is as described below:

\begin{itemize}
    \item \emph{id}: Id of the data point.
    \item \emph{estimated\_<column>}: Contains prediction for this data point. Can be one (regression)
    or more columns(classification).
\end{itemize}
% subsection prediction_out (end)
% section interface (end)

\subsection{Other functions}
\label{sub:others}
\subsubsection{Display} % (fold)
\label{sub:display}
\begin{sql}
    SELECT forest_display(
          random_forest_model,
          group_id,
          sample_id,
          is_dot_format)

\end{sql}

\begin{itemize}
    \item \emph{random\_forest\_model}: Table containing the random forest model.
    \item \emph{group\_id}: Id of the group that the tree to be displayed is a part of.
    \item \emph{sample\_id}: Id of the sample within the group.
    \item \emph{is\_dot\_format}: True for dot format, False for text format.
\end{itemize}

The output of the display function to output individual trees will follow the same format as used by
Decision Trees.

% subsection display (end)

\section{Implementation} % (fold)
\label{sec:implementation}

Let $X = \{X_1, X_2, \dots, X_N\}$ be a set of attributes with domains
$D_{X_1}, D_{X_2}, \dots, D_{X_N}$. Let $Y$ be an output with domain $D_Y$.
Given a dataset $D^* = \{(x_i, y_i) | x_i \in D_{X_1} \times D_{X_2} \times \dots D_{X_N}, y_i \in D_Y\}$,
the goal is to learn a random forest model that best approximates the true
distribution of $D^*$. The random forest will be composed of individual trees
each of which is learnt using a greedy top-down approach, as described in
Decision Trees.

In the below sections, we will only expand item[1] and item[5], which are specific
to building random forests. The others have already been covered in Decision Trees.
Any differences will be noted along in the process.

The following are the major steps for building a random forest model:
\begin{itemize}
\item[1.] Bootstrap, which includes sampling N samples from the training set.
\item[2.] Initialize, which includes binning and finding split candidates.
\item[3.] Finding best split for each node.
\item[4.] Finding prediction for leaf nodes in the individual decision trees.
\item[5.] Calculate random forest statistics.
\end{itemize}

\subsection{Bootstrapping}
Given the dataset $D^* = \{(x_i, y_i) | x_i \in D_{X_1} \times D_{X_2} \times \dots D_{X_N}, y_i \in D_Y\}$
we will generate B bootstrap samples $B_{1}, B_{2}, \dots B_{b}$ where each bootstrap
sample is obtained by sampling N times, with replacement, from the same dataset.
Each bootstrapped sample $B_{i}$ is treated as the training set for ~\ref{alg:buildRandomForest}.
We thought about a couple of approaches for achieving this:

\subsubsection{Using Row Identifiers}
The first method generates row ids for each row of the input table, which ranges from
  1..N. We then generate N random numbers between 1 and N, and store the results in a
  one-column table. A bootstrapped sample can now be obtained by joining this
  table with the input dataset. See \ref{sec:SamplingWithReplacement} for implementing this
  in sql. The problem, however, is that, the step which generates row ids for the input table
  would involve pulling in data from all segments to the master in order to be able to generate
  a sequential gap-less set of ids. This is an expensive operation which is overcome by the
  second method.

\subsubsection{Using Poisson sampling}
The process of bootstrapping can be thought of as sampling from a multinomial
  distribution where the probability of selecting any data point is uniform
  over the entire data set. While this requires scanning the entire data set, the second
  approach approximates the multinomial distribution by sampling from an identical
  Poisson distribution on each data point independently. Thus, we are able to generate
  counts indicative of the number of times the particular data point should
  be included in the bootstrap sample.

It has been decided that we will be using Poisson sampling for our bootstrapping step.
We will use a Poisson distribution with $\lambda=1$ where $\lambda$ is the average
fraction of input data that we want in each tree. In our case, this fraction = 1 as we want
N samples in each tree from the input where N is also the size of the data set.
\\
\begin{algorithm}[Bootstrapping$(D^*)$]\label{alg:bootstrap}
\alginput{Training dataset $D^*$}
\algoutput{Bootstrap sample $B$}
  \begin{algorithmic}[1]
   \For{$d \in D^*$}
     \State count = Poisson(1)
     \State id = user provided id for $d$
     \State Add $(count, id)$ to C where C is a two-column table
    \EndFor
    \State Output $Join(D^*,C)$
  \end{algorithmic}
\end{algorithm}

\subsection{Variable Importance}
``Subtract the number of votes for the correct class in the variable-m-permuted
oob data from the number of votes for the correct class in the untouched oob
data.''~\cite{random_forest_home}
According to the source code of R package randomForest~\cite{r_random_forest},
we define the normalization of the subtraction,
\[
D_{m,g,t,p} =
    \frac{
        \left( \sum_{n \in \mathit{oob}_{t,g}} \mathit{predict}(n) == y_n \right)
        - \left( \sum_{n \in \mathit{oob}_{t,g}} \mathit{predict}_p^m(n) == y_n \right)
    }{
        \mathit{oobsize}_{t,g}
    }.
\]
For regression,
\[
D_{m,g,t,p} =
    \frac{
        \left[ \sum_{n \in \mathit{oob}_{t,g}} \left( \mathit{predict}_p^m(n) - y_n \right)^2 \right]
        - \left[ \sum_{n \in \mathit{oob}_{t,g}} \left( \mathit{predict}(n) - y_n \right)^2 \right]
    }{
        \mathit{oobsize}_{t,g}
    }.
\]

``The average of this number over all trees in the forest is the raw importance
score for variable m.''~\cite{random_forest_home}
Considering multiple permutations, we have
\[
importance_{m,g} = \sum_{p=1}^P \sum_{t=1}^T \frac{D_{m,g,t,p}}{TP}
\]
for each variable $m$ per group $g$.

Note: R package randomForest~\cite{r_random_forest} computes the above
importance without considering grouping support and aggregating $p$ prior to
$t$.

In order to rank the importance of features in the input dataset, random forests
use a mechanism of calculating the average increase in misclassification rate
when the values of a particular feature are randomly permuted in the input set,
and comparing the classification accuracy against the original data set.
The algorithm is as follows:

 \begin{algorithm}[calculateVariableImportance$(R, F^*)$] \label{alg:variableImportance}
   \alginput{Random Forest model $R$}
   \alginput{Features $F^*$}
   \algoutput{Variable importance for each feature in $F$}
   \begin{algorithmic}[1]
   \For{$subtree T_i \in R$}
     \State $O_i$ = out-of-bag samples of $T_i$
     \State size = size of $O_i$
     \For{$f \in F$}
       \State Create one-column table C with values of $f$ randomly selected $size$ times
       \State Join $(O_i,C)$ replacing column f with values from C
       \For{$o \in O_i$}
         \State Find prediction for o, and increment count for (f, o, prediction)
       \EndFor
     \EndFor
   \EndFor
   \State Find majority prediction for each input point per feature
   \State Calculate misclassification rate per feature
  \end{algorithmic}
 \end{algorithm}

\subsection{Proximities}
Proximities help identify clusters within the input set. This is depicted
as an NxN matrix where N is the size of the data set.
The algorithm is as follows:

 \begin{algorithm}[calculateProximity$(R, D^*)$] \label{alg:proximity}
   \alginput{Random Forest model $R$}
   \algoutput{Proximity for data points in $D^*$}
   \begin{algorithmic}[1]
   \For{$subtree T_i \in R$}
     \For{$d \in D^*$}
       \State Find prediction p for d in $T_i$. Increment pair\-wise counts of d with all other data points with prediction p.
     \EndFor
   \EndFor
   \State Normalize proximity by dividing by number of trees.
  \end{algorithmic}
 \end{algorithm}

\subsection{Feature sampling}
Unlike in decision trees where all the features are used to grow the tree,
random forests will at each split, select at random, a subset of the
features to find the best split candidate. This would require some changes
in the current Decision tree implementation to both support and be able to
use the reduced feature set for an optimized implementation, by, for example,
only having to aggregate statistics for the feature subsample.

\subsection{Random forest and Decision Tree building algorithms}
\begin{algorithm}[BuildRandomForest$(D^*)$] \label{alg:buildRandomForest}
\alginput{Training dataset $D^*$}
\algoutput{A random forest model fit to the dataset $D$}
\begin{algorithmic}[1]
   \State Generate bootstrap samples $B_{1}, B_{2}, \dots B_{r}$
   using ~\ref{alg:bootstrap}
   \For {$i \in B_{i}$}
       \State Generate binning info $b_{i}$ using ~\ref{alg:findSplitBins}
       \State Build subtree using ~\ref{alg:buildSubtreeRF}
   \EndFor
   \end{algorithmic}
\end{algorithm}

\begin{algorithm}[buildSubtreeRandomForest$(D^*, M, n_l, B)$] \label{alg:buildSubtreeRF}
\alginput{Bootstrapped training dataset $B_{i}$,\\
          Binning info $b$}
\algoutput{Subtree built from $B_{i}$}
\begin{algorithmic}[1]
   \State Select $m$ features at random from the set of $p$ features
   \State On each segment, perform the following \Comment{Transition function}
    \For{$d \in B_{i}$}
        \For{$attr \in m$ \, in \, $d$}
            \State Find the bin index in $b$ that $attr\_value$ belongs to
            \State Add sufficient statistics $s$ to aggregate
        \EndFor
    \EndFor
    \State Merge aggregated statistics on different segments \Comment{Merge function}
    \State Find split in $m$ that maximizes impurity using ~\ref{alg:findBestSplits}\Comment{Final function}
\end{algorithmic}
\end{algorithm}

\subsection{Grouping Support}
Like other algorithms in MADlib, Random Forests will also support grouping. A lot of the
functionality that exists in Decision Trees to support grouping can be leveraged.
Two major steps to support grouping in random forests would be:
\begin{itemize}
    \item Generate bootstrap samples for a particular group before training a random forest
    on that group. See \ref{sec:SamplingWithReplacement} for implementation details of
    sampling with replacement involving groups, in the database.
    \item Aggregate necessary results on a per-group basis in order to output final statistics
    such as variable importance, proximity etc.
\end{itemize}

\section{Data Handling}
\label{sec:dataHandling}

In order to calculate metrics such as the oob error, variable importance and proximities,
data from all the trees need to aggregated. Three different alternatives were
experimented with, w.r.t storing and retrieving data.

\begin{enumerate}
    \item In the first approach, we construct two tables, one that stores the bootstrap samples
    of the current working set, i.e., the data needed to construct the current tree,
    and another table which accumulates both the training and oob samples for all trees.
    Metrics are then calculated at the end of training the random forest model by running
    aggregation queries on the table with all of the data. The query to accumulate all data
    simply appends the current sample to the aggregated sample.
    \item In the second approach, we store one big table, which is used for both training the
    current tree, as well as accumulating samples for all trees. The overall amount of data
    stored is reduced.
    \item In the final approach, we construct one table to store the current working set like in
    approach (1), and additionally, we store one other table to store various partially
    aggregated statistics from training each tree, such as the predictions for oob samples
    for each tree. These will be aggregated at the end of training to produce the final results.
    \item Based on preliminary tests on HAWQ (with half a rack), a table with 1M rows
    using approach (3) takes only half as much time as required for approaches (1) and (2)
\end{enumerate}

\section{Design Considerations}
\label{sec:designConsiderations}

\begin{itemize}
    \item Building a random forest can potentially be made embarrassingly parallel as each tree
    in the model can be built independently of the rest. This, however, requires that each segment
    has all the data necessary for training a single tree, and would involve copying large amounts
    of data over to the segments, which is a lot of overhead. It has therefore been decided that
    individual trees will be built in sequence, with each tree itself being built in parallel, as
    discussed under Decision Trees.
    \item Random forests are usually built without pruning, which could lead to trees that are very
    deep. That could potentially have an impact on performance. The current design proposes
    not to use pruning, but based on performance tests and accuracy results, this could be changed
    to use pruning and/or limit the maximum depth of the tree.
    \item num\_trees is currently an input parameter to the rf\_train method. If, however, we want
    to be able to determine the number of trees to use, one approach is to use a number that is
    typically used, such as few hundreds of trees. Another way might be to observe the out-of-bag
    error rate on each training sample on the current model, and stop constructing more trees when
    the improvement in the error rate falls below a certain threshold.
\end{itemize}

% section parallelism (end)


\subsection{Requirements}

    The following requirements seek to provide an estimate for the progress path the project will take.
    The priorities of each requirement are \textit{must > should > may > might}.
    Where these changed over time and where requirements were not met has been noted.

    \subsubsection{Solo Calculus}
        The first part of the system will be focused on text-based calculus expressions only.
        This will form a basis upon which to implement other features.
        This system:
        \begin{itemize}
            \item \textit{Must} support input and output of text representing expressions.
            \item \textit{Must} be able to perform reductions on inputted expressions and return these reductions.
            \item \textit{Should} provide this functionality though a accessible and useable user interface.
            \item \textit{Might} be implemented to evaluate expressions concurrently.
                \textit{This was not achieved over the course of the project.}
        \end{itemize}

    \subsubsection{Solo Diagrams}
        This will be the first extension of the system, allowing interoperability with diagrams.
        This system:
        \begin{itemize}
            \item \textit{Must} support input and output of diagrams.
                \textit{While this was achieved during the project, it was only in part.
                The diagrams used remained hard-coded in the application.
                It was hoped that this could have been achieved through inputting a calculus expression using the above, converting to a diagram and using it.}
            \item \textit{Should} provide a conversion between diagrams and expressions.
                \textit{Again, see above as to how this could have been achieved.}
            \item \textit{Should} provide the same reductions as are provided to calculus expressions.
            \item \textit{May} be able to produce step-by-step reductions.
            \item \textit{Might} be implemented to evaluate diagrams concurrently.
                \textit{Again, this was not achieved over the course of the project.
                The construction of diagrams allows for easily dividing into independent subdiagrams, which may be reduced with no interaction with one another.
                This would allow for a simple method to partition the dataset across multiple processes.}
        \end{itemize}

    \subsubsection{Diagram Visualisation}
        The second extension will build on the previous Solo Diagrams and provide a graphical interface for input/output/manipulation of diagrams.
        This should provide a clear visual real-time representation of how the calculus works.
        This system:
        \begin{itemize}
            \item \textit{Must} provide a visualisation of diagrams through a graphical user interface.
            \item \textit{Should} support input/output of a diagram from/to an expression.
                \textit{Again, see above as to how this could have been achieved.}
            \item \textit{Should} support manipulation of a diagram within the GUI.
            \item \textit{Should} be both intuitive and feature-complete to provide an obvious improvement over the text-based input of expressions as described above.
            \item \textit{May} be able to produce step-by-step reductions.
        \end{itemize}


% document properties
\documentclass[10pt,landscape]{article}
\title{Metrics Cheat Sheet}
\author{Travis Cao}

\usepackage{cheat-sheet}

\begin{document}

\raggedright
\footnotesize
\begin{multicols}{3}

% multicol parameters
% These lengths are set only within the two main columns
\setlength{\premulticols}{1pt}
\setlength{\postmulticols}{1pt}
\setlength{\multicolsep}{1pt}
\setlength{\columnsep}{2pt}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{center}
    \large{710a Metrics Notes} \\
    \vspace{.1cm}
    \normalsize{Travis Cao}
\end{center}

\vspace{-.8cm}

\section{Stats}

\subsection{Geometric series sum}
$\sum_{j = 0}^n r^j = \frac{1 - r^{n+1}}{1 - r}$

\subsection{Conditional expectation}
$E[Y] = \sum_l E[Y|Z = l] Pr(Z = l)$

\subsection{Block inversion formula}
$M$ is invertible iff $A-BD^{-1}C$ is invertible, and $M^{-1} = $
\begin{align*}
  \begin{bmatrix}
    (A-BD^{-1}C)^{-1} & -(A-BD^{-1}C)^{-1}BD^{-1} \\
    -D^{-1}C(A-BD^{-1}C)^{-1} & D^{-1}+D^{-1}C(A-BD^{-1}C)^{-1}BD^{-1}
  \end{bmatrix}
\end{align*}

\subsection{Sherman-Morrison formula}
$A+uv'$ is invertible iff $1+v'A^{-1}u \neq 0$, and
\begin{align*}
  (A+uv')^{-1} = A^{-1} - \frac{A^{-1}uv'A^{-1}}{1+v'A^{-1}u}
\end{align*}

\subsection{Partial and average partial effect}
\begin{itemize}
  \item Partial effect (at $X = x$): $\frac{\partial}{\partial x} E[Y|X = x]$
  \item Average partial effect: $E[\frac{\partial}{\partial X} E[Y|X]]$
\end{itemize}

\subsection{Inequalities}
\begin{itemize}
  \item Chebyshev's: $Pr (\abs{X - E[X]} \geq \epsilon) \leq \frac{Var(X)}{\epsilon^2}$ for all $\epsilon > 0$
  \item Markov: $Pr(X \geq \epsilon) \leq \frac{E[X]}{\epsilon}$ for all $\epsilon > 0$
  \item Jensen's: For convex function $g(\cdot)$, $g(E[X]) \leq E[g(X)]$
  \item Cauchy Schwarz: $E[XY]^2 \leq E[X^2] E[Y^2]$
\end{itemize}

\subsection{Martingale CLT}
If $\{ Z_t \}_{t=1}^T$ satisfy
\begin{itemize}
  \vspace{3pt}
  \item $\{ Z_t \}_{t=1}^T$ is strictly stationary
  \vspace{3pt}
  \item $E[Z_1^2] < \infty$
  \vspace{3pt}
  \item $E[Z_t | Z_{t-1}, Z_{t-2}, \ldots, Z_1] = 0$
  \vspace{2pt}
  \item $\frac{1}{T} \sum_{t=1}^T Z_t^2 \xrightarrow{p} E[Z_1^2]$
\end{itemize}
then $\frac{1}{\sqrt{T}} \sum_{t=1}^T Z_t \xrightarrow{d} N(0, E[Z_1^2])$ as $T \rightarrow \infty$

% \vfill\null
% \columnbreak

\section{IV}

Model: $Y_i = X_i'\beta + U_i$

Assumptions:
\begin{itemize}
  \item Validity of instrument
  \begin{itemize}
    \item Exogeneity: $E[U | Z] = 0$
    \item Relevance: $E[Z X']$ and $\sum_{i=1}^n Z_i X_i'$ are invertible
  \end{itemize}
  \item $E[|Y|^2 + ||X||^2 + ||Z||^2] < \infty$
  \item $\{ (Y_i, X_i', Z_i') \}$ are i.i.d.
\end{itemize}

IV estimator: $\hat{\beta}_{IV} = (\sum_{i=1}^n Z_i X_i')^{-1} (\sum_{i=1}^n Z_i Y_i')$

\subsection{Small sample property}
Model: $Y_i = \beta_0 + X_i \beta_1 + U_i$

\hspace{24pt} $X_i = \pi_0 + Z_i \pi_1 + V_i$

\vspace{3pt}
\hspace{24pt} $
  \begin{pmatrix}
    U_i \\
    V_i
  \end{pmatrix} \bigg| \enspace Z_i \sim N \left(
    \begin{pmatrix}
      0 \\
      0
    \end{pmatrix},
    \begin{pmatrix}
      \sigma_u^2 & \sigma_{uv} \\
      \sigma_{uv} & \sigma_v^2
    \end{pmatrix} \right)
$
\vspace{3pt}

$\Rightarrow E[\hat{\beta}_{IV} | \mathbbmss{X}, \mathbbmss{Z}] = \beta + \frac{\sigma_{uv}}{\sigma_v^2} \frac{\nu}{\frac{\hat{\pi}_1}{se(\hat{\pi}_1)} + \nu}$, where $\nu \sim N(0, 1)$

\subsection{Large sample property}

Large sample approximation is reasonable depends on both
\begin{itemize}
  \item sample size $n$ (the bigger, the better), and
  \item the covariance between the instrument and endogenous regressor $\pi_1$ (the bigger, the better)
\end{itemize}

Which means there's a need of testing whether the instrument is relevant (whether $\pi_1 \neq 0$)

\begin{adjustwidth}{.3cm}{0cm}
  Model: $X_1 = Z_1 \pi_1 + X_2' \pi_2 + V$. Let $\hat{\pi}_1$ be OLS estimate of $\pi_1$.

  Then the F-statistic is $F = \frac{(\hat{\pi}_1 - 0)^2}{se({\hat{\pi}_1})^2} = \frac{\hat{\pi}_1^2}{se({\hat{\pi}_1})^2}$

  Empirically, if $F \geq 10$, then some rough insurance with nominal $95\%$ conﬁdence intervals have actual coverage of at least $80\%$.
\end{adjustwidth}

\subsection{Weak instrument}

Model: $Y = X_1 \beta_1 + U$. Weak instrument is $Z_1$.

Assumptions on $Z_1$:
\begin{itemize}
  \item Exogeneity: $E[U|Z_1] = 0$
  \item Allow for relevance
  \item $E[Z_1^2]$ and $\frac{1}{n} \sum_i Z_{1i}^2$ are both non-zero (i.e. invertible)
\end{itemize}

Under weak instrument, want to test
\begin{align*}
  H_0 : \beta_1 = c \quad \quad H_1 : \beta_1 \neq c
\end{align*}

Identification under $H_0$: $E[Z_1(Y - X_1 c)] = 0$

$\Rightarrow T = \frac{1}{n} \sum_i Z_{1i} (Y_i - X_{1i}c)$. Reject $H_0$ when $\abs{T}$ is large

Anderson-Rubin Test:

\begin{adjustwidth}{.3cm}{0cm}
$AR \coloneqq \frac{\sqrt{n}T}{S} \xrightarrow{d} N(0, 1)$

where $S^2 = \frac{1}{n} Z_{1i}^2 (Y_i - X_{1i}c)^2 \xrightarrow{p} E[Z_1^2 U^2]$

With a size of $5\%$, reject $H_0$ when $\abs{AR} > 1.96$

Confidence set: $\{ c \in \RR : \abs{AR(c)} \leq 1.96 \}$
\end{adjustwidth}

\subsection{Optimal instrument}

Say optimal instrument is $h^*(Z)$. Identification yields from $E[h(Z) U] = 0$. So
$
  \hat{\beta}^h_{IV} = (\frac{1}{n} \sum_{i} h(Z_i)X_i' )^{-1} \frac{1}{n}\sum_{i} h(Z_i) Y_i
$

Under homoskedasticity, asymptotic variance (AVAR) is
\begin{align*}
  \Omega^h &= \frac{E[h(Z)^2]}{E[h(Z)X]^2} \sigma_u^2 = \frac{E[h(Z)^2]}{E[h(Z) E[X|Z]]^2} \sigma_u^2 \\
  &\geq \frac{E[h(Z)^2]}{E[h(Z)]^2 E[E[X|Z]^2]} \sigma_u^2 = \frac{\sigma_u^2}{E[E[X|Z]^2]}
\end{align*}
To achieve the lower bound of AVAR,
$
  h^*(Z) = E[X|Z]
$

\subsection{Random coefficient model}

Model: $Y = X'\beta_0 \cdot U = X'\beta_0 + X'\beta_0 \cdot (U - 1)$

Assumptions:
\begin{itemize}
  \item $E[U|Z] = 1$, and $E[ZX']$ invertible
  \item $Z$ independent of $Y$
  \item Let $X = (1, X)'$, $Z = (1, Z)'$, $Z \in \{ 0, 1 \}$, $X \in \{ 0, 1 \}$
\end{itemize}
\begin{align*}
  \hat{\beta}_{IV} \xrightarrow{p} \frac{Cov(Z, Y)}{Cov(Z, X)} = \frac{E[Y|Z = 1] - E[Y|Z = 0]}{E[X|Z = 1] - E[X|Z = 0]}
\end{align*}
For binary $X$, its response to $Z$ given unobservable $U$ is $X_U(Z)$
\begin{center}
  \begin{tabular}{c| c c}
    \toprule
    & $X_U(0) = 0$ & $X_U(0) = 1$ \\
    \midrule
    $X_U(1) = 0$ & Never taking & Defying \\
    $X_U(1) = 1$ & Complying & Always taking \\
    \bottomrule
  \end{tabular}
\end{center}
Assume no defyers, and $Pr(\text{Complying}) > 0$, then
\begin{align*}
  &E[X|Z = 1] - E[X|Z = 0] = E[X_U(1) - X_U(0)] \\
  &= Pr(X_U(1) - X_U(0) = 1) - Pr(X_U(1) - X_U(0) = -1) \\
  &= Pr(\text{Complying})\\
  &E[Y|Z = 1] - E[Y|Z = 0] = E[Y_U(1)X_U(1) + Y_U(0)(1-X_U(1))] \\
  &\quad - E[Y_U(1)X_U(0) + Y_U(0)(1-X_U(0))] \tag{since $Y = Y_U(1)X + Y_U(0)(1-X)$}\\
  &= E[(Y_U(1) - Y_U(0))(X_U(1) - X_U(0))] \\
  &= E[(Y_U(1) - Y_U(0))|X_U(1) - X_U(0) = 1] \times Pr(\text{Complying})
\end{align*}
Thus, $\hat{\beta}_{IV} \xrightarrow{p} E[\underbrace{(Y_U(1) - Y_U(0))}_{\text{mean response of $Y$ to $X$}}|\underbrace{X_U(1) - X_U(0) = 1}_{\text{for complyers}}]$

% \vfill\null
% \columnbreak

\section{Time Series}

\subsection{TS Models}
\begin{itemize}
  \item Static: $Y_t = \alpha_0 + X_t' \delta_0 + U_t$
  \item FDL($s$): $Y_t = \alpha_0 + X_t' \delta_0 + \ldots + X_{t-s}' \delta_s + U_t$
  \item AR($p$): $Y_t = \alpha_0 + Y_{t-1}\rho_1 + \ldots + Y_{t-p}\rho_p + U_t$
  \item MA($q$): $Y_t = \e_t + \theta_1 \e_{t-1} + \ldots + \theta_q \e_{t-q}$
  \item Trend:
    \begin{itemize}
      \item Linear time trend: $Y_t = \beta_0 + \beta_1 t + U_t$
      \item Exponential trend: $\log(Y_t) = \beta_0 + \beta_1 t + U_t$
    \end{itemize}
  \item Seasonality: $Y_t = \alpha_0 + \alpha_1 1_{\{t/12 \text{ is an integer}\}} + U_t$
\end{itemize}

\subsection{Stationarity}
\begin{itemize}
  \item Strict: $(Y_{t_1}, Y_{t_2}, \ldots, Y_{t_k}) \sim (Y_{t_1+l}, Y_{t_2+l}, \ldots, Y_{t_k+l})$

  \vspace{-1.5pt}
  (Joint distribution is $t$ independent)

  \item Weak (Covariance): For all $t$, $E[Y_t]$ and $\gamma(k) = Cov(Y_t, Y_{t+k})$ ($\leftarrow$ autocovariance function) are both independent of $t$
\end{itemize}

\end{multicols}

\end{document}


\hypertarget{mesh}{%
\section{Mesh}\label{mesh}}

The \texttt{Mesh} class provides support for meshes. Meshes may consist
of different kinds of element, including vertices, line elements, facets
or area elements, tetrahedra or volume elements.

To create a mesh, you can import it from a file:

\begin{lstlisting}
var m = Mesh("sphere.mesh")
\end{lstlisting}

or use one of the functions available in \texttt{meshtools} or
\texttt{implicitmesh} packages.

Each type of element is referred to as belonging to a different
\texttt{grade}. Point-like elements (vertices) are \emph{grade 0};
line-like elements (edges) are \emph{grade 1}; area-like elements
(facets; triangles) are \emph{grade 2} etc.

The \texttt{plot} package includes functions to visualize meshes.

\hypertarget{save}{%
\section{Save}\label{save}}

Saves a mesh as a .mesh file.

\begin{lstlisting}
m.save("new.mesh")
\end{lstlisting}

\hypertarget{vertexposition}{%
\section{Vertexposition}\label{vertexposition}}

Retrieves the position of a vertex given an id:

\begin{lstlisting}
print m.vertexposition(id)
\end{lstlisting}

\hypertarget{setvertexposition}{%
\section{Setvertexposition}\label{setvertexposition}}

Sets the position of a vertex given an id and a position vector:

\begin{lstlisting}
print m.setvertexposition(1, Matrix([0,0,0]))
\end{lstlisting}

\hypertarget{addgrade}{%
\section{Addgrade}\label{addgrade}}

Adds a new grade to a mesh. This is commonly used when, for example, a
mesh file includes facets but not edges. To add the missing edges:

\begin{lstlisting}
m.addgrade(1)
\end{lstlisting}

\hypertarget{addsymmetry}{%
\section{Addsymmetry}\label{addsymmetry}}

Adds a symmetry to a mesh. Experimental in version 0.5.

\hypertarget{maxgrade}{%
\section{Maxgrade}\label{maxgrade}}

Returns the highest grade element present:

\begin{lstlisting}
print m.maxgrade()
\end{lstlisting}

\hypertarget{count}{%
\section{Count}\label{count}}

Counts the number of elements. If no argument is provided, returns the
number of vertices. Otherwise, returns the number of elements present of
a given grade:

\begin{lstlisting}
print m.count(2) // Returns the number of area-like elements.
\end{lstlisting}


\newpage
\subsection{Nonlinear Parameters}

% Table
\begin{table}[h!]
    \begin{tabular}{*{2}{L{3cm}C{2cm}C{2cm}}}
        \hline\rowcolor{LightBlue}
        \btcell{Parameter}                  & \btcell{Value}    & \btcell{Unit}     &
        \btcell{Parameter}                  & \btcell{Value}    & \btcell{Unit}      \\\hline
    \end{tabular}
    % Poincare Results
    \ifpoincare
        \begin{tabular}{*{2}{L{3cm}C{3cm}C{1cm}}}
                \multicolumn{6}{l}{\gcell{\textbf{Poincar\'e Plot Parameters}}}     \\
                \gcell{SD1}                 & \sdone            & ms                &
                \gcell{SD2}                 & \sdone            & ms                \\
                \gcell{SD1/SD2}             & \sdratio          & ms                &
                \gcell{Ellipse Area S}      & \ellipsearea      & ms                \\\hline
        \end{tabular}
    \fi
    % DFA
    \ifdfa
        \begin{tabular}{*{2}{L{3cm}C{3cm}C{1cm}}}
            \multicolumn{6}{l}{\gcell{\textbf{Detrended Fluctuation Analysis}}}     \\
            \gcell{${\alpha}_1$}            & \dfaalphaone      & -                 &
            \gcell{${\alpha}_2$}            & \dfaalphatwo      & -                 \\\hline
        \end{tabular}
    \fi
    % Sample Entropy Results
    \ifsampen
        \begin{tabular}{*{2}{L{3cm}C{3cm}C{1cm}}}
            \multicolumn{6}{l}{\gcell{\textbf{Sample Entropy}}}                     \\
            \gcell{Sample Entropy}          & \sampen           & -                 &
            \gcell{-}                       & -                 & ms                \\\hline
        \end{tabular}
    \fi
\end{table}

% Poincare Plot
\ifpoincare
    \begin{minipage}{.475\textwidth}
        \centering
        \includegraphics[width=\linewidth]{figures/poincareplot.png}
    \end{minipage}%
    \hfill
\fi
% DFA Plot
\ifdfa
    \begin{minipage}{.457\textwidth}
        \centering
        \includegraphics[width=\linewidth]{figures/dfaplot.png}
    \end{minipage}
\else
    \hfill
\fi

\vfill


\newsection
\subsection{Theory}
\label{sec:theory}
\sectionauthors{Aditi Raghunathan, Sang Michael Xie, Ananya Kumar, Niladri Chatterji, Rohan Taori, Tatsunori Hashimoto, Tengyu Ma}

\newcommand\lpre{\ensuremath {\ell_{\text{pre}}}}
\newcommand\emppre{\ensuremath {\widehat{L}_{\text{pre}}}}
\newcommand\popadapt{\ensuremath {L_{\text{adapt}}}}
\newcommand\empadapt{\ensuremath {\widehat{L}_{\text{adapt}}}}
\newcommand\poppre{\ensuremath {L_{\text{pre}}}}
\newcommand\datapre{\ensuremath {p_{\text{pre}}}}
\newcommand\trainpre{\ensuremath {\hat{p}_{\text{pre}}}}
\newcommand\datatask{\ensuremath {p_{\text{task}}}}
\newcommand\traintask{\ensuremath {\hat{p}_{\text{task}}}}
\newcommand\thetaum{\ensuremath {\hat{\theta}_{\textup{FM}}}}
\newcommand\gammatask{\ensuremath {\gamma_{\text{task}}}}
\newcommand\ladapt{\ensuremath {\ell_{\text{adapt}}}}
\newcommand\poptest{\ensuremath {L_\text{task}}}

Rigorous mathematical theory plays a foundational role in many engineering and science disciplines (\eg information theory in electrical engineering). We believe that theory of foundation models can be particularly beneficial in guiding technical decisions and innovations because of the huge computational costs associated with experimenting on foundation models.
In addition, theoretical insights help elucidate fundamental limitations and explain surprising empirical phenomena.
However, the community currently has a limited theoretical understanding of foundation models, despite much recent progress~\cite{arora2019theoretical,haochen2021spectral,wei2021pretrained,wei2020theoretical,zhang2021inductive,saunshi2020mathematical,dao2019kernel,tosh2020contrastive,tosh2021contrastive,cai2021theory,lee2020predicting,zimmermann2021contrastive,bansal2020self,wang2020understanding,tsai2020self,tian2020makes,tian2020understanding, tripuraneni2020theory,du2020fewshot}.

Deep neural networks form the backbone of foundation models.
Even in the well-studied supervised learning setting, where the train and test scenarios have the same distribution, there are numerous open questions around deep nets such as understanding non-convex optimization, the implicit regularization effect of optimizers, and expressivity. Foundation models raise questions that significantly go beyond the supervised deep learning setting.
The core problem in theoretically analyzing foundation models is understanding why training on one distribution with a possibly unsupervised/self-supervised loss leads to good adaptation performance on \emph{different} downstream distributions and tasks.\footnote{The theory for foundation models closely relates to, but also goes beyond the theory for transfer learning (which is itself an underexplored area): foundation models are possibly trained with unlabeled data and will be adapted to many or all natural tasks, whereas transfer learning typically studies labeled source tasks and a fixed number of target tasks.}

We will discuss an intuitive modularization to analyze foundation models that lays bare the connections between supervised learning and foundation models, concrete and core technical questions, and some promising theoretical tools to address these questions. These new core questions can provide useful insight into foundation models and can be studied in parallel to supervised deep learning theory. While we focus on analyzing the downstream performance, the proposed modularization and tools could prove useful to analyze other metrics of interest such as robustness to distribution shifts (\refsec{robustness}) and security (\refsec{security}).

\begin{figure}[!ht]
\centering
\includegraphics[width=\linewidth]{technology/figures/Theory.png}
\caption{\label{fig:theory} The analysis of foundation models from pretraining on diverse data to downstream performance on adapted tasks involves capturing the relation between different loss terms as shown above. The main challenge is to analyze the highlighted pretraining-adaptation interface which requires reasoning carefully about the population losses in addition to the model architecture, losses and data distributions of the pretraining and adaptation stages (\refsec{theory-interface}). Analysis of generalization and optimization largely reduces to their analysis in standard supervised learning.}
\end{figure}

\subsubsection{Theoretical formulations and modularizations}
\label{sec:theory-modularizations}
Recall that foundation models are trained on a large amount of raw data (\refsec{training})  then adapted to specific tasks (\refsec{adaptation}) and therefore can be decomposed naturally into training and adaptation phases. We identify interfaces between them and disentangle parts specific to foundation models from parts that require standard deep learning theory, so that they can be independently worked on. We introduce a modularized analysis framework, which has also been implicitly or explicitly employed in recent works, \eg~\citet{arora2019theoretical,haochen2021spectral,wei2020theoretical,tripuraneni2020theory}. The crucial component in this modularized analysis turns out to be the \emph{pretrain-adaptation interface.} We first describe the modularization, and discuss why we find this modularization promising and finally some limitations.

We will refer to the training phase explicitly as ``pretraining'' to distinguish it from the adaptation phase that could also involve training on a few samples from a particular task.

\paragraph{Pretraining phase.}
The pretraining of foundation models often involves a data distribution $\datapre$ (\eg the distribution of natural text) and a \emph{pretraining loss} function $\lpre(x;\theta)$ that measures the loss (\eg language modeling loss in GPT-3) on an input $x$ for a model with parameters $\theta\in \Theta$. Let $\trainpre$ denote the empirical distribution over a large number of independent samples from $\datapre$.

Pretraining minimizes the loss $\lpre$ on $\trainpre$, which we call \textit{the empirical pretraining loss}, and produces a model $\thetaum$:
\begin{align}
    \emppre(\theta) \eqdef \E_{x\sim \trainpre}[\lpre(x;\theta)],
        ~~\text{and}~~
\thetaum \eqdef \argmin \limits_{\theta \in \Theta} \emppre(\theta).
\end{align}

We consider the corresponding loss on the population distribution $\datapre$, called the \textit{population pretraining loss}, as a central concept:
\begin{align}
   \poppre(\theta) \eqdef \E_{x\sim\datapre}[ \lpre(x; \theta)] .
\end{align}

\paragraph{Optimization-based adaptation phase.}
We frame adaptation as a general constrained optimization problem that depends on $\thetaum{}$, abstracting away those adaptation methods that are based on optimizing certain loss functions such as fine-tuning and prompt-tuning (see, \eg ~\citep{houlsby19adapter,li2021prefix,lester2021power}, and \refsec{adaptation}).

Since different adaptation methods could modify different subsets of the model parameters, we denote the space of adapted model parameters by some $\Gamma$. Given a downstream task distribution $\datatask$ (\eg question answering in a particular domain) and a few empirical samples $\traintask$ sampled from $\datatask$, we model the adaptation phase as minimizing some \textit{adaptation loss} $\ladapt$ on $\traintask$ w.r.t adapted parameters $\gamma \in \Gamma$:
\begin{align}
\gammatask(\thetaum) \eqdef \argmin \limits_{\gamma \in \Gamma, C(\gamma;\thetaum)\le c_0 }\empadapt(\gamma, \thetaum),
\end{align}
where $\empadapt(\gamma, \thetaum) \eqdef \E_{x\sim \traintask}[\ladapt(x;\gamma, \thetaum)]$ is the empirical adaptation loss, and $C(\gamma, \thetaum)\leq c_0$ is an optional constraint that controls the complexity of the adapted parameters, encompassing both explicit regularization (\eg model dimensionality and norm) and the implicit regularization of the adaptation process.

We list some common adaptation methods and discuss the corresponding adapted parameter $\gamma$ and constraints $C(\gamma, \thetaum)\leq c_0$.
\begin{enumerate}
    \item Linear probing: training a linear classifier on top of the representations from a foundation model. Here $\Gamma = \R^k$ is the set of linear classifiers on the representations of dimensionality $k$, and $C(\gamma, \thetaum)$ could be the $\ell_2$ or $\ell_1$ norm of $\gamma$.
    \item Fine-tuning: optimizing a randomly initialized linear head for a few steps, and all other parameters $\theta$ from the initialization of $\thetaum$. Here $\gamma$ is the concatenation of $\theta$ and the linear head. Such a process could correspond to some implicit regularization of $\gamma$ towards the initialization $\thetaum$ captured by $C(\gamma, \thetaum) \leq c_0$. The exact term $C(\gamma, \thetaum)$ would depend on the optimization algorithm used, and such a characterization of the implicit regularization of optimization is an area of active research study ~\citep[\eg][and references therein]{gunasekar2017implicit, soudry2018implicit,gunasekar2018implicit,arora2019implicit, blanc2019implicit,woodworth2020kernel, wei2020implicit, haochen2021shape,damian2021label}.\footnote{
    It may not always be feasible to characterize the inductive bias of adaptation via an explicit constraint $C(\gamma,\thetaum)\leq c_0$. The modularization we propose is also applicable in these cases, but for notational simplicity, we focus on the case where implicit regularization can be approximated via an explicit constraint.
    }
    \item Prompt-tuning: optimizing a small set of continuous task-specific vectors that prepend the task inputs. Here $\gamma$ is the continuous prompt vectors which often has small dimensionality, and we may optionally have a constraint on the norms of $\gamma$.
\end{enumerate}

One obvious limitation to note is that this formulation excludes adaptation methods such as in-context learning~\citep{brown2020gpt3} where there is no ``training'' (\ie the minimization of some empirical adaptation loss) during the adaptation phase. We discuss this and other limitations in \refsec{theory-incontext}.

Two central quantities for the adaptation phase are the \textit{population adaptation loss}
\begin{align}
    \popadapt(\gamma, \thetaum) = \E_{x\sim\datatask}[ \ladapt(x; \gamma,\thetaum)]
\end{align}
and the \emph{minimal adaptation loss}
\begin{align}
    \popadapt^\star(\thetaum) = \min_{\gamma \in \Gamma, C(\gamma;\thetaum)\le c_0}\popadapt(\gamma, \thetaum)
    \label{eqn:5}
\end{align}

\paragraph{Separate analysis for modularized phases.} Existing generalization theory for standard supervised learning aims to show that $\emppre\approx \poppre$ and $\empadapt\approx \popadapt$.
Addressing these questions specifically for deep nets is an active research area.
We can also leverage the standard learning theory decomposition to bound the final downstream task loss by the excess generalization error and the minimal adaptation loss as follows.
 \begin{align}
    \popadapt(\gammatask, \thetaum) \le \underbrace{\popadapt^\star(\thetaum)}_{\textup{minimal adaptation loss}} + \textup{~~generalization error}
    \label{eqn:6}
\end{align}
where the generalization error captures the closeness between $\popadapt$ and $\empadapt$.\footnote{More precisely, the generalization error term is the sum of  $\popadapt(\gammatask, \thetaum) -  \empadapt(\gammatask, \thetaum)$ and $\empadapt(\gammatask^\star, \thetaum)-\popadapt(\gamma^\star, \thetaum) = \empadapt(\gammatask^\star, \thetaum)-\popadapt^\star(\thetaum)$, where $\gammatask^\star$ is the minimizer of \refeqn{5}. \refeqn{6} follows easily be using $\empadapt(\gammatask, \thetaum)\le \empadapt(\gammatask^\star, \thetaum)$.}
The decomposition and relationship between these key quantities are shown in \reffig{theory}.
The generalization and optimization arrows, as argued above, largely reduce to deep learning theory in the supervised setting.
What we are left with is the main challenge with foundation models, which is to understand why the minimal adaptation loss $\popadapt^*(\thetaum)$ can be small as a result of a small pretraining population loss, which we dive into next in \refsec{theory-interface}.

The work of~\citet{arora2019theoretical} pioneered the pursuit of this question by bounding from above $\popadapt^\star(\thetaum)$ by $\poppre(\thetaum)$ in the context of contrastive learning, and ~\citet{haochen2021spectral,tosh2020contrastive,tosh2021contrastive} relax the data assumptions. Other pretraining methods successfully analyzed under this framework (implicitly or explicitly) include pretraining with language models ~\cite{wei2021pretrained} or self-supervision~\cite{lee2020predicting}, with self-training algorithms~\cite{wei2020theoretical,cai2021theory}, and with multiple supervised tasks ~\cite{tripuraneni2020theory,du2020fewshot}.

\subsubsection{Why is the pretraining-adaptation interface interesting?.}
\label{sec:theory-interface}
As shown in \reffig{theory}, the main missing link beyond standard supervised theory is:
\begin{center}
    \emph{Under what conditions does a small population pretraining loss $\poppre(\thetaum)$ imply a small minimal adaptation loss $\popadapt^\star(\thetaum)$} and why?
\end{center}
The conditions that lead to a successful interface could depend on several quantities such as the pretraining and adaptation distributions, objectives and training methods, as well as the model architecture. This question is beyond the scope of standard generalization theory, but it does narrow us down to a few important factors specific to foundation models, and captures the essence of various important open questions on foundation models as we argue below.

First, we note that this interface deals with population quantities that concern two \emph{different distributions}. Hence, the conditions for a successful interface are likely to involve special properties of the distributions, for example, the diversity of the pretraining distribution and structural shifts between the pretraining and adaptation data. This makes the analysis of the interface challenging (as discussed below in \refsec{theory-tools}) as we need to make careful modeling assumptions about how the two distributions relate to one another. However, this presents the possibility that tools and techniques developed to analyze such interfaces could be useful to understand the effect of distribution shifts and to predict when foundation models can improve robustness.

Second, the population losses and possibly the conditions of a successful interface depend on the \emph{model architecture}. This raises the challenge of opening up the black-box of the neural nets. What does a small pretraining loss on a particular distribution tell us about the properties of the intermediate layers? Such analyses would also guide us in designing new adaptation methods that more carefully exploit different intermediate representations.

Third, \emph{few-shot learning} or the sample efficiency of adaptation can be captured through the constraint on the complexity measure $C(\gamma, \thetaum) <c_0$ in the minimal adaptation loss.
We need to formally characterize these complexity measures (\eg by understanding the implicit regularization effect of the adaptation process) and further understand why a small population pretraining loss would imply a low-complexity adaptation parameters $\gammatask$.
A satisfactory answer to this question would likely allow us to improve the sample-efficiency of downstream adaptation.

Finally, and importantly, critical components of the interface are the choice of the \emph{pretraining and adaptation losses}. We want to understand how to best combine the pretraining and adaptation objectives for successful adaptation. It is possible that the pretraining objective that best guarantees successful adaptation differs from what is explicitly minimized during the pretraining process\dash{}the interface above allows one to use any surrogate population objective on the pretraining distribution. In addition, new surrogate objectives that provably lead to good adaptation across a broad set of tasks could shed light on the fundamental aspects that make foundation models successful.

To summarize, the interface precludes the issue of generalization and allows us to formally reason about the interaction between several important quantities of the pretraining and adaptation phases that can guide practice in important ways.

\subsubsection{Challenge: analysis of in-context learning and other emergent behavior}
\label{sec:theory-incontext}
GPT-3~\citep{brown2020gpt3} demonstrates the power of in-context learning, an adaptation method that does not need any parameter optimization. In the adaptation phase, the pretrained language foundation model takes in a prompt\dash{}a sequence of tokens that concatenates input-output examples from the task\dash{}followed by a test example and simply generates the label of the test example by conditioning on the sequence seen thus far (prompt plus test example).
In other words, there is no explicit training or change to the model parameters.
What is the mechanism by which the model ``learns'' from the different examples by simply executing with the examples as inputs?
The previous modularization does not directly apply because we do not obtain new model parameters during adaptation, but rather we only use the generative capabilities of the foundation model by executing on structurally-designed inputs.
More broadly, the proposed modularization provides a nice framework to gain useful theoretical insights into foundation models as discussed above. But it is possible that some emergent behavior like in-context learning and other capabilities yet to be discovered would require going beyond the modularization.

\subsubsection{Challenge: appropriate data assumptions and mathematical tools.}
 \label{sec:theory-tools}
Understanding the interface between pretraining and adaptation phases requires a more careful study of data distributions than in traditional supervised learning. This is because the pretraining and task adaptation distributions are inherently different. By definition, foundation models are trained on raw data that is typically extremely diverse and task-agnostic, while the adaptation data depends  heavily on the task. Similarly, in-context learning emerges as a result of learning to generate data that looks like the pretraining distribution, and thereby understanding in-context learning requires careful modeling of the pretraining data. Hence answering the central questions around foundation models requires realistic and interpretable assumptions that are also amenable to analysis.
Recent works either assume certain properties of the population data, \eg the expansion property in \citet{haochen2021spectral, wei2020theoretical}, or that the population data is generated from latent variable models with some structure~\citep{SaMa20mat,wei2021pretrained,arora2015latent,lee2020predicting,zhang20ont,tosh2021contrastive}.

We generally lack mathematical tools for relating properties of foundation models to the structure in the population data distribution.  ~\citet{haochen2021spectral} applies spectral graph theory to leverage the inner-class connectivity in the population distribution. More precise characterization of $\thetaum$ via probabilistic and analytical derivations is possible for latent variable models, but so far restricted to relatively simple ones. The community will significantly benefit from more systematic and general mathematical tools to address this question.

It is also highly desirable to define simple toy cases so that theoreticians can precisely compare the strengths of various tools and analyses. For example, \citet{haochen2021spectral} and \citet{wei2020theoretical} consider the mixture of manifolds problem which might potentially be a good simplified test bed for vision applications. We need more interesting test beds for discrete domains such as NLP.
We believe that tractable theoretical models which capture relevant properties of real datasets are a crucial step towards placing foundation models on solid theoretical footing.


\chapter{Model Design and Methods Used}

\section{Getting and Pre-processing the Data}

The data that was used in this thesis comes from our Computer Science
Undergraduate course, i.e. Analysis of Algorithms from $2^{nd}$ year during
the Fall 2015 semester. For that semester, we have a total record of 180
registered students, but we removed those whose attributes were not relevant.
Since we are interested in classifying students based on their exam grade,
also implying the classification in {\it passed} and {\it failed} classes,
we also removed those student that had a poor grade during the semester and
could not participate at the final exam. Having said this, we are left with a
total of 137 relevant student records. These records must further be split
into two sets: the training set - the set that is used for building the model
and the testing set - the set used for testing the accuracy and measure
the performance indicators of the model. There is not a standard recipe of
how to divide these two sets, but it is recommended that the training set
should have a proportion between 60\% and 80\% of the total dataset's number
of examples.

\subsection{Dataset Structure}

In Table \ref{table_1} there is a listing with all the attributes (features)
used for the training models. There are a total of 10 initial features
(will address later on this problem with the number of features). The weight
characteristic of the features represents the actual maximum points that a
student can get from that assignment (they are all equal, summing for a total
of 5 semester points). This was not used in the project, since the value is
constant for all features. From the table, we can see that the range for the
test and homework grades is not the same. This was resolved by dividing the
test grades range with a factor of 20, in order to have a better uniformization
of the datased before applying standard pre-processing techniques that are
required from learning models.

\begin{table}[ht]
\centering
\begin{tabular}{l*{10}{c}r}
\toprule
Feature            & Description & Type & Range & Weight \\
\hline
t\_1               & Test 1 grade & Float & 0-10 & 0.5 \\
t\_2               & Test 2 grade & Float & 0-10 & 0.5 \\
t\_3               & Test 3 grade & Float & 0-10 & 0.5 \\
t\_4               & Test 4 grade & Float & 0-10 & 0.5 \\
t\_5               & Test 5 grade & Float & 0-10 & 0.5 \\
t\_6               & Test 6 grade & Float & 0-10 & 0.5 \\
hw\_1              & Homework 1 grade & Float & 0-0.5 & 0.5 \\
hw\_2              & Homework 2 grade & Float & 0-0.5 & 0.5 \\
hw\_3              & Homework 3 grade & Float & 0-0.5 & 0.5 \\
hw\_4              & Homework 4 grade & Float & 0-0.5 & 0.5 \\
\bottomrule
\end{tabular}
\caption[Dataset Details]{Dataset Attribute Details}
\label{table_1}
\end{table}

Next, we need to provide the values used for labelling the examples. Since this
thesis treats both classification and regression problems, different types of
labels must be present in the dataset. For classification, we use the final
grade (integer between 4 and 10) and the exam grade as integer values. For
regression, only the exam grade (which is a real number between 0 and 4) is
used and it will be represented as a float number. Table \ref{table_2} gives a
structured view of how the students dataset labels are used in the studied models.
For classifying the students into 4 classes based on their exam points the
real continuous interval $(0, 4]$ was split into 4 equal subintervals and for
each subinterval was assigned a class label (0, 1, 2 and 3, respectively).

\begin{table}[ht]
\centering
\begin{tabular}{cccc}
\toprule
\multicolumn{1}{c}{Problem type} & \multicolumn{3}{c}{Label Description} \\
\cmidrule(r){2-4}
                        & Label type  & Label Value & Label used \\
\midrule
Binary Classification   & Integer     & 0|1         & Final grade \\
4-class Classification  & Integer     & 0|1|2|3     & Exam grade \\
Regression              & Float       & 0.0-4.0     & Exam grade \\
\bottomrule
\end{tabular}
\caption[Label Structure]{Label Structure}
\label{table_2}
\end{table}

\subsection{Data Standardization}

In Machine Learning, standardization of data is a common requierement for a
lot of models (e.g.: neural networks and SVMs). This means that, before
applying our learning models, we must first scale the features from our dataset.
This scaling implies that the data should have {\bf mean} 0 and {\bf standard
deviation} 1. This is done by subtracting the mean value of each feature, then
scale the data by dividing the features by their standard deviation (or variance,
because standard deviation is the square root of variance, so they are equal).

\begin{equation}
\centering
X_{scaled} = \frac{X-\overline{X}}{\sigma}
\end{equation}
where $X$ is a vector of values from one feature, $\overline{X}$ is the mean of $X$ and
$\sigma$ represents the standard deviation of $X$.

In Figure \ref{fig.figure2}\footnote{http://cs231n.github.io/neural-networks-2} we can
see a straight-forward visualization of how the data is represented before and after
the preprocessing techniques:

\insfigshw{figure2.png}%
    {Data preprocessing pipeline}%
    {Data preprocessing pipeline}%
    {fig.figure2}{0.8}

It is worth noting that the mean and standard deviation values must come only
from the training data and the transformation must be done on both training and
testing sets, for meaningful results. The reason is that, in general, the built
Machine Learning model is applied on unseen data (on real-time data), which is
not available when the model is built. So, for accurate calculation on the
model's performance and generalization, we must restrict the computation of
mean and variance only on the training examples.

\section{Using Polynomial Features}

\subsection{Curse of Dimensionality}

In general, the number of examples and the dimensionality of each example
(the number of features per example) are correlated, taking into consideration
the accuracy of the trained model. The {\it Hughes phenomenon}\cite{hughes}
tells us that if we have a constant number of training examples, the ability
of the model's prediction decreses when the dimensionality increses over an
optimal value. This is also called the {\bf Curse of Dimensionality} and it
can lead to {\it overfitting} the dataset - the model has a low power of
generalization. Finding the best number of features can be a very hard problem
(as it requires a lot of manual testing). Actually, this is an {\it intractable}
problem, because we need to generate all possible combinations of features and
find the optimal one. This could easily be avoided now by using Feature Selection
tools. For example, {\bf Random Forests} are very good models at selecting
features that provide the best accuracy to the model. This will be analyzed
with more details in the next sections of the thesis.

So, supposing that we have $M$ number of examples in the training set and
the feature tensor is one-dimensional, if we add another dimension to the
tensor (another feature), ideally, we need to square the training examples.
By induction, the number of training example grows exponentially with the
dimension of the feature tensor.

Figure \ref{fig.figure3}\footnote{http://www.visiondummy.com/2014/04/curse-dimensionality-affect-classification/} shows a representation of how the number of feature dimensions
affects the quality of the learning model. It can be seen that keeping the
training examples constant and only increasing the number of features, the
accuracy drops by an exponential rate.

\insfigshw{figure3.png}%
    {The Curse of Dimensionality}%
    {The Curse of Dimensionality}%
    {fig.figure3}{0.8}

\subsection{Adding Complexity to the Model}

There are situations in which is better to add complexity to our features.
For example, when our dataset in not linearly separable using one, two, or more
features, we can add extra dimensions to the feature tensor in order to make
that data separable. {\bf SVMs} make use of this approach, by using multiple
{\it kernel functions} that have the role to transform an input space in order
to easily process data. Intuitively, a kernel is a ``shortcut'' that allows us
to do certain computations, but without being directly involved in higher-dimensional
calculations. Kernels can be linear functions, polynomial functions or even sigmoid
functions. Using SVMs, we implicitly add complexity to our model.

Having tested some simple models on the dataset, such as Linear Regression or the
Perceptron, adding some complexity to the models was not at all a bad idea.
A good method was the one of using {\bf Polynomial Features}, that combines the
initial features into new nonlinear features. Polynomial Features adds more
dimensions to the feature space, but the key here is that they are correlated,
so this can help in achieving better prediction accuracy. Here, there are two different
options to consider:

\begin{itemize}
  \item The first one is generating a list of features (a polynomial of a certain
  degree) from the current features.
  Example: If we have the input given as $(X_1, X_2)$, after the polynomial
  tranformation the example becomes $(1, X_1, X_2, X_1*X_2, X_1^2, X_2^2)$
  \item The second approach was to consider only interactions between the
  features for building a polynomial with the same degree as the number of
  initial features.
  Example: The features $(X_1, X_2, X_3)$ are tranformed by the polynomial
  into: $(1, X_1, X_2, X_3, X_1*X_2, X_1*X_3, X_2*X_3, X_1*X_2*X_3)$, resulting
  in a total of $2^{N}$ final features, where $N$ is the original dimension of
  the input space.
\end{itemize}

So, for Linear Regression, the input features are transformed using the first
method (to generate non-linear functions like polynomials of degree 2 or 3).
This ``trick'' allows us to use simple linear models that are trained on actual
non-linear combinations of the data and are faster than other complex
non-linear models.
Supposing that we want to train our student's dataset using Linear Regression
with Polynomial Features:

Let $\tilde{y}$ be the output vector of the linear model, $x$ the input tensor,
$\omega \in \mathbb{R}^{M \cdot N}$ the coefficient tensor (a two-dimensional vector)
and $\beta$ the vector bias. The model computes the following equation, making use of
the ``least squares'' method for calculating
$\omega$ and $\beta$:
\begin{equation}
\label{lin_reg}
\centering
\tilde{y}(\omega, x) = \omega \cdot x + \beta
\end{equation}
where $x = (x_1,x_2,\dots,x_{10}).$
When we add the polynomial features, $x$ is transformed like this:
\begin{center}
$x_{nonlinear} = (x_1,x_2,\dots,x_{10},\dots,x_i \cdot x_j,\dots,x_1^2,x_2^2,\dots,x_{10}^2);$
 $i, j = \overline{1,10}, i < j$
\end{center}
The size of the new feature space is: $10+10+C_{10}^2=65$ \\
Substituting $x_{nonlinear}$ in Equation \ref{lin_reg}, we obtain:
\begin{equation}
\centering
\tilde{y}(\omega', x_{nonlinear}) = \omega' \cdot x_{nonlinear} + \beta'
\end{equation}
We can observe from the above equation that the linearity is still preserved and the
model can fit more complicated data now.

On the other hand, the Perceptron model was tested using both methods, although the
second method turned to be more appropiate, i.e. the {\it interaction features} method.
This method was also used for the MLP\footnote{Multi-Layer Perceptron} neural network
model. With the size of the feature space of 10, adding interaction features
provided a total of $2^{10}=1024$ features.

\section{Visualizing the dataset}


\documentclass[12pt]{cdblatex}
\usepackage{exercises}
\usepackage{fancyhdr}
\usepackage{footer}

\begin{document}

% --------------------------------------------------------------------------------------------
\section*{Exercise 5.2 Leading factors forbidden in patterns}

This exercise will raise a Cadabra run-time error -- the scale factor on the left
hand side of the rule (3 in this case) is not allowed.

\CdbSetup{action=verbatim}% the following code will raise a run time error

\begin{cadabra}
   {a,b,c,d,e,f,g,h,i,j,k,l,m,n,o,p,q,r,s,t,u#}::Indices(position=independent).

   expr := 2 V_{a b} - 3 V_{b a}.                  # cdb (ex-0502.100,expr)

   substitute (expr, $3 V_{b a} -> - 3 V_{a b}$)   # cdb (ex-0502.101,expr)

\end{cadabra}

\vskip 1cm
\bgroup
\lstset{numbers=none,backgroundcolor=\color{white}}
\begin{lstlisting}
   Traceback (most recent call last):
     File "/usr/local/bin/cadabra2", line 248, in <module>
       exec(cmp)
     File "ex-0502.py", line 18, in <module>
       substitute (expr, Ex(r'''3 V_{b a} -> - 3 V_{a b}''', False))
   RuntimeError: substitute: Index error in replacement rule.
                 substitute: No numerical pre-factors allowed on lhs of replacement rule.
\end{lstlisting}
\egroup

\end{document}


\chapter{Operators}\label{ch:operators}

The operators recognized by \aprepro{} are listed below. The letters \textbf{a}
and \textbf{b} can represent variables, numbers, functions, or expressions unless
otherwise noted. The tables below also list the precedence and associativity of
the operators. {\em Precedence} defines the order in which operations should be performed.
For example, in the expression:
\begin{apinp}
\{3 * 4 + 6 / 2\}
\end{apinp}
the multiplications and divisions are performed first, followed by the addition
because multiplication and division have higher precedence (10) than addition (9). The precedence
is listed from 1 to 14 with 1 being the lowest precedence and 14 being the highest.

{\em Associativity} defines which side of the expressions should be
simplified first.  For example the expression: \cmd{3 + 4 + 5} would
be evaluated as \cmd{(3 + 4) + 5} since addition is left associative;
in the expression \textbf{a = b / c}, the \cmd{b/c} would be evaluated
first followed by the assignment of that result to \cmd{a} since
equality is right associative

\section{Arithmetic Operators}

Arithmetic operators combine two or more algebraic expressions into a
single algebraic expression. These have obvious meanings except for
the pre- and post- increment and decrement operators. The
pre-increment and pre-decrement operators first increment or decrement
the value of the variable and then return the value. For example, if
\cmd{a = 1}, then \cmd{b=++a} will set both \cmd{b} and \cmd{a} equal to \cmd{2}. The post-increment and
post-decrement operators first return the value of the
variable and then increment or decrement the variable. For example, if
\cmd{a = 1}, then \cmd{b=a++} will set
\cmd{b} equal to \cmd{1} and \cmd{a} equal to \cmd{2}. The
modulus operator \cmd{\%} calculates the integer remainder. That is
both expressions are truncated an integer value and then the remainder
calculated. See the \cmd{fmod} function in Table~\ref{t:functions} for
the calculation of the floating point remainder. The tilde character
\cmd{\textasciitilde{}} is used as a synonym for multiplication to
improve the aesthetics of the unit conversion system (see
Chapter~\ref{ch:units}). It is more natural for some users to type
\cmd{12\textasciitilde{}metre} than \cmd{12*metre}.

\begin{longtable}{llcc}
\caption{Arithmetic Operators}\\
Syntax   & Description    & Precedence & Associativity\\
\hline
\cmd{a+b}      & Addition       &  9 & left \\
\cmd{a-b}      & Subtraction    &  9 & left \\
\cmd{a*b}, \cmd{a}\verb+~+\cmd{b} & Multiplication & 10 & left \\
\cmd{a/b}      & Division       & 10 & left \\
\cmd{a\^{}b}, \cmd{a**b} & Exponentiation & 12 & right \\
\cmd{a\%b}      & Modulus, (remainder) & 10 & left \\
\cmd{++a}, \cmd{a++} & Pre- and Post-increment a & 13 & left\\
\cmd{\texttt{--}a}, \cmd{a\texttt{--}} & Pre- and Post-decrement a & 13 & left\\
\end{longtable}

\section{Assignment Operators}

Assignment operators combine a variable and an algebraic expression into a single
algebraic expression, and also set the variable equal to the algebraic expression.
Only variables can be specified on the left-hand-side of the equal sign.

\begin{longtable}{llcc}
\caption{Assignment Operators}\\
Syntax & Description & Precedence & Associativity \\
\hline
\cmd{a=b}   & The value of $a$ is set equal to $b$ & 1 & right\\
\cmd{a+=b}  & The value of $a$ is set equal to $a + b$ & 2 & right\\
\cmd{a-=b}  & The value of $a$ is set equal to $a - b$ & 2 & right\\
\cmd{a*=b}  & The value of $a$ is set equal to $a * b$ & 3 & right\\
\cmd{a/=b}  & The value of $a$ is set equal to $a / b$ & 3 & right\\
\cmd{a}\^{}\cmd{=b}, a\cmd{**=b}& The value of $a$ is set equal to  $a^b$ & 4 & right\\
\end{longtable}

\section{Relational Operators}\label{sec:relationaloperators}

Relational operators combine two algebraic expressions into a single relational
expression. Relational expressions and operators can only be used before the question
mark (\cmd{?}) in a conditional expression.

\begin{longtable}{llcc}
\caption{Relational Operators}\\
Syntax & Description & Precedence & Associativity \\
\hline
\cmd{a \texttt{<} b} & true if $a$ is less than $b$ & 8 & left\\
\cmd{a \texttt{>} b} & true if $a$ is greater than $b$ & 8 & left\\
\cmd{a \texttt{<}= b} & true if $a$ is less than or equal to $b$ & 8 & left\\
\cmd{a \texttt{>}= b} & true if $a$ is greater than or equal to $b$ & 8 & left\\
\cmd{a ==  b} & true if $a$ is equal to $b$ & 8 & left\\
\cmd{a !=  b} & true if $a$ is not equal to $b$ & 8 & left\\
\end{longtable}

\section{Boolean Operators}

Boolean operators combine one or more relational expressions into a single relational
expression. If \cmd{la} and \cmd{lb} are two relational expressions, then:

\begin{longtable}{llcc}
\caption{Boolean Operators}\\
Syntax & Description & Precedence & Associativity \\
\hline
\cmd{la \textbar{}\textbar{} lb} & true if either $la$ or $lb$ are true. & 6 & left\\
\cmd{la \&\&  lb} & true if both $la$ and $lb$ are true. & 7 & left\\
\cmd{!la} & true if $la$ is false. & 11 & left\\
\end{longtable}
The evaluation of the expression is not short-circuited if the truth
value can be determined early; both sides of the expression are
evaluated and then the truth of the expression is returned.

\section{String Operators}

The only supported string operator at this time is string concatenation which is
denoted by \cmd{//}. For example,
\begin{apinp}
\{a = "Hello"\} \{b = "World"\}
\{c = a // " " //  b\}
\end{apinp}
sets \cmd{c} equal to \cmd{\texttt{"}Hello  World\texttt{"}.}  Concatenation
has precedence 14 and left associativity.