\documentclass[a4paper,10pt]{article}
\usepackage{graphicx,xspace}
\usepackage[utf8]{inputenc}
\usepackage[left=2.5cm,top=3cm,bottom=3cm,right=2.5cm]{geometry}
\newcommand{\R}{\textsf{R}\xspace}

\usepackage{/home/gavin/R/build/3.3-patched/share/texmf/tex/latex/Sweave}

\renewcommand{\abstractname}{Summary}

\begin{document}
\title{Introduction to R for the Geoscience: Stratigraphic \& Palaeo Data}
\author{Gavin Simpson}
\date{February , 2017}

\maketitle

\section{Rates of Change}
In this part of the practical you will compute some rate of change estimates
for a short core sequence from the Round Loch of Glenhead covering the last 140
years. There aren't any canned functions for computing rates of change in \R so
you'll perform all the neccessary steps yourself --- don't worry, they aren't
too onerous and you'll learn more about \R coding as you go along.

Start by loading the \textsf{rioja} package and the dataset needed
\begin{Schunk}
\begin{Sinput}
> library("rioja")                        # install.packages("rioja")
> ## load data and extract
> data(RLGH)
> spp <- RLGH$spec
> age <- RLGH$depth$Age
\end{Sinput}
\end{Schunk}

\subsection{Ordination-based rates of change}
The first method you'll use is the ordination based rate of change, for which
we need to smooth and interpolate the species data to a common time interval.
This can be done using the \texttt{interp.dataset()} function
\begin{Schunk}
\begin{Sinput}
> ## interpolate new dataset to every 5 years
> ## using a smoothing splint
> x.new <- seq(0, max(age), by=5)
> sp.interp <- interp.dataset(y=spp, x=age, xout=x.new,
+                             method = "sspline")
> rownames(sp.interp) <- x.new
\end{Sinput}
\end{Schunk}

Next you need to ordinate the data. Here we use DCA to follow the published version, but you could try other ordination techniques should you wish.
\begin{Schunk}
\begin{Sinput}
> ## ordinate
> library("vegan")
> ord <- decorana(sp.interp)
> ord
\end{Sinput}
\begin{Soutput}
Call:
decorana(veg = sp.interp)

Detrended correspondence analysis with 26 segments.
Rescaling of axes with 4 iterations.

                  DCA1    DCA2     DCA3     DCA4
Eigenvalues     0.1312 0.02112 0.007314 0.007082
Decorana values 0.1316 0.01130 0.005090 0.003085
Axis lengths    1.1056 0.53068 0.427959 0.374999
\end{Soutput}
\begin{Sinput}
> ## plot of points showing ages
> plot(ord, display = "sites", type = "text")
\end{Sinput}
\end{Schunk}

How many axes do you want to retain?

Finally, to compute the rate of change we need to extract the axis scores we
need and compute the distance between observations in ordination space. This
has to be done by hand for the most part.
\begin{Schunk}
\begin{Sinput}
> ## get scores
> scrs <- scores(ord, display = "sites", choices = 1:2)
> roc <- as.matrix(dist(scrs))
> roc <- roc[row(roc) == col(roc) + 1] ## extract off-diagonal
> roc
\end{Sinput}
\begin{Soutput}
 [1] 0.145791683 0.038388466 0.056476075 0.096765764 0.141675748 0.089316724
 [7] 0.148886370 0.065164981 0.123568620 0.180866597 0.135311608 0.072936429
[13] 0.009012417 0.054150418 0.058022625 0.065137156 0.081900194 0.093647074
[19] 0.102022525 0.106469426 0.102384652 0.095557955 0.078070381 0.054041243
[25] 0.024817872 0.004879166 0.010482557 0.017234419
\end{Soutput}
\end{Schunk}

Plot the rates of change for the core sequence
\begin{Schunk}
\begin{Sinput}
> ## plot roc against age
> plot(roc ~ head(x.new, -1), type = "l", ylab = "Rate of Change", xlab = "Age")
> points(roc ~ head(x.new, -1), type = "h")
\end{Sinput}
\end{Schunk}

Identify the periods with greatest rates of change. You'll compare these with
those identified using the dissimilarity-based approach in the next section.

\subsection{Dissimilarity-based rates of change}
Now you'll calculate rates of change for the same samples using the raw
dissimilarity method. In this method the data don't need to be interpolate,
instead we work with the actual species data themselves. Start by computing
the squared chord distances between all samples and extract the off-diagonal
elements (the pairwise dissimilarities)
\begin{Schunk}
\begin{Sinput}
> roc2 <- as.matrix(paldist(spp/100, dist.method="sq.chord"))
> roc2 <- roc2[row(roc2) == col(roc2) + 1] ## extract off-diagonal
\end{Sinput}
\end{Schunk}

Next scale these by the number of years between each sample. We can compute the
sample intervals in years using the \texttt{diff()} function
\begin{Schunk}
\begin{Sinput}
> intervals <- diff(age)
> roc2 <- roc2 / intervals
\end{Sinput}
\end{Schunk}

Plot the rates of change for the core sequence
\begin{Schunk}
\begin{Sinput}
> ## plot roc2 against age
> plot(roc2 ~ head(age, -1), type = "l", ylab = "Rate of Change", xlab = "Age")
> points(roc2 ~ head(age, -1), type = "h")
\end{Sinput}
\end{Schunk}

Identify the periods with greatest rates of change. How do these compare with
the rates extract using the ordination-based approach? Can you think why there
might be differences? Which method would you trust most? Why?

To aid you in your comparison, you can plot both rates of change on the same
device
\begin{Schunk}
\begin{Sinput}
> ## compare plots
> layout(matrix(1:2, nrow = 2))
> ## plot roc against new age
> xlim <- range(age, x.new)
> plot(roc ~ head(x.new, -1), type = "l", ylab = "Rate of Change", xlab = "Age",
+      main = "Ordination-based", xlim = xlim)
> points(roc ~ head(x.new, -1), type = "h")
> ## plot roc2 against age
> plot(roc2 ~ head(age, -1), type = "l", ylab = "Rate of Change", xlab = "Age",
+      main = "Dissimilarity-based", xlim = xlim)
> points(roc2 ~ head(age, -1), type = "h")
> layout(1)
\end{Sinput}
\end{Schunk}

\section{Chronological clustering}

Now we'll look at a coupld of chronological clustering methods that are available in \R.

\subsection{CONISS}

First we'll apply CONISS to the Abernethy Forest pollen record. We load the data set, convert the observations to proportions (from percentages) and apply a square root transformation. Ths means that when we compute the euclidean distances between samples using \texttt{dist()} we are computing hellinger distances instead. Finally we use the \texttt{chclust()} function to perform the CONISS clustering

\begin{Schunk}
\begin{Sinput}
> library("analogue")
> data(abernethy)
> aber <- abernethy[, seq_len(ncol(abernethy) - 2L)]
> diss <- dist(tran(aber, method = "pcent2prop"))
> clust <- chclust(diss)
\end{Sinput}
\end{Schunk}

We can plot the CONISS results and the variances ``explained'' by varying the number of zones.

\begin{Schunk}
\begin{Sinput}
> layout(matrix(1:2, ncol = 2))
> op <- par(mar = c(5,4,1,2) + 0.1)
> plot(clust, labels = abernethy$Age, hang=-1, horiz=TRUE, x.rev = TRUE, cex = 0.5)
> bstick(clust, 10)
> par(op)
> layout(1)
\end{Sinput}
\end{Schunk}

Using the screeplot method or Bennett's broken stick approach, suggest how many zones you would consider splitting the Abernethy Forest pollen record into?

\subsection{Binary splitting}

Unfortunately, the \textbf{mvpart} package that I used previously to do binary or recursive partitioning has been removed from CRAN as it contains problems that were never addressed by its author/maintainer. A new way of fitting almost the same model has become available in the \textbf{partykit} package. Rather than use cross-validation and cost-complexity pruning to choose the number of binary splits, it uses conditional inference trees. I'm not exactly sure how to describe this, but it is essentially using statistical methods to decide if it should make a new split at all. This is in contrast to standard tree models where they are grown very large and we use cross validation to prune the branches back.

In this section we'll take a quick look at doing clustering via binary splitting using the \textbf{partykit} package. First we load the package (you'll need to install it first) and then we remove some of the rare species (you should just do this casually --- I'm doing it here to make fitting the tree easier). We also need to clean up (remove spaces and hyphens) the taxon names so we can create the formula needed to fit the model

\begin{Schunk}
\begin{Sinput}
> library("partykit")                     # install.packages("partykit")
> aber2 <- chooseTaxa(aber, n.occ = 2, max.abun = 5)
> names(aber2) <- gsub(" ", "", names(aber2))
> names(aber2) <- gsub("-", "", names(aber2))
\end{Sinput}
\end{Schunk}

Now we can create the formula which describes the conditional tree we want to fit. All the species we didn't remove will end up on the left hand side of the formula, and \texttt{Age} will be the sole predictor. We add on the \texttt{Age} variable to the reduced set of species so all the variables are in one object.

\begin{Schunk}
\begin{Sinput}
> frm <- formula(paste0(paste(names(aber2), collapse = " + "), " ~ Age"))
> aber2 <- cbind(aber2, Age = abernethy$Age)
\end{Sinput}
\end{Schunk}

Now we can fit the conditional inference tree and plot the results

\begin{Schunk}
\begin{Sinput}
> sptree <- ctree(frm, data = aber2, teststat = "max", minsplit = 2)
> plot(sptree, terminal_panel = node_barplot, tp_args = list(rot = 45, just = c("right")))
\end{Sinput}
\end{Schunk}

The leaves of the tree are illustrated by barplots showing the abundances of each species. These barplots give you some idea of the species composition in each zone, but the code isn't flexible enough yet to allow you to mke nicer plots or show taxon names clearly (or I haven't worked out how to do it!) Note that here we just used the raw percentage abundances to fit the tree, you might get a different tree if you transformed or standardised the species data.

How many zones are suggested by the tree? Do these zones match up with the ones CONISS identified?

\section{Principal Curves}
In this section of the practical you'll explore fitting a principal curve to
the Abernethy Forest pollen data set. The functions you need are available in
the \textsf{analogue} package. Load it, and ignore the warnings for now.
\begin{Schunk}
\begin{Sinput}
> library("analogue")
> data(abernethy)
\end{Sinput}
\end{Schunk}

A simple summary of the data can be produced using the \texttt{Stratiplot()}
function
\begin{Schunk}
\begin{Sinput}
> (plt <- Stratiplot(Age ~ . - Depth, data =
+                    chooseTaxa(abernethy, max.abun = 15, n.occ = 10),
+                    type = c("h","g","l"), sort = "wa"))
\end{Sinput}
\end{Schunk}

Next, remove two variables from the data set (the samples depths and ages) and
then proceed to fit a PCA and a CA, as well as the principal curve to the data
\begin{Schunk}
\begin{Sinput}
> abernethy2 <- abernethy[, -(37:38)]
> aber.pca <- rda(abernethy2)
> aber.ca <- cca(abernethy2)
> aber.pc2 <- prcurve(abernethy2, method = "ca", trace = TRUE, plotit = TRUE,
+                     vary = TRUE, penalty = 1.4)
\end{Sinput}
\begin{Soutput}
   Determining initial DFs for each variable...

  |
  |                                                                      |   0%
  |
  |==                                                                    |   3%
  |
  |====                                                                  |   6%
  |
  |======                                                                |   8%
  |
  |========                                                              |  11%
  |
  |==========                                                            |  14%
  |
  |============                                                          |  17%
  |
  |==============                                                        |  19%
  |
  |================                                                      |  22%
  |
  |==================                                                    |  25%
  |
  |===================                                                   |  28%
  |
  |=====================                                                 |  31%
  |
  |=======================                                               |  33%
  |
  |=========================                                             |  36%
  |
  |===========================                                           |  39%
  |
  |=============================                                         |  42%
  |
  |===============================                                       |  44%
  |
  |=================================                                     |  47%
  |
  |===================================                                   |  50%
  |
  |=====================================                                 |  53%
  |
  |=======================================                               |  56%
  |
  |=========================================                             |  58%
  |
  |===========================================                           |  61%
  |
  |=============================================                         |  64%
  |
  |===============================================                       |  67%
  |
  |=================================================                     |  69%
  |
  |===================================================                   |  72%
  |
  |====================================================                  |  75%
  |
  |======================================================                |  78%
  |
  |========================================================              |  81%
  |
  |==========================================================            |  83%
  |
  |============================================================          |  86%
  |
  |==============================================================        |  89%
  |
  |================================================================      |  92%
  |
  |==================================================================    |  94%
  |
  |====================================================================  |  97%
  |
  |======================================================================| 100%

Fitting Principal Curve:

Initial curve: d.sq: 103233.450
Iteration   1: d.sq: 4283.431
Iteration   2: d.sq: 4312.298
Iteration   3: d.sq: 4340.691
Iteration   4: d.sq: 4355.388
Iteration   5: d.sq: 4366.497
Iteration   6: d.sq: 4369.944

PC Converged in 6 iterations.
\end{Soutput}
\end{Schunk}
The arguments to \texttt{prcurve()} tell it to start from a CA solution, show
it working by printing to the console and plotting the curve as it converges,
taxa are allowed differeing complexity smoothers and the penalty per degree of
freedom is increased by 40\% to help avoid over-fitting.

Look at the fitted object. How much of the variance in the dataset is explained
by the principal curve?
\begin{Schunk}
\begin{Sinput}
> aber.pc2
\end{Sinput}
\begin{Soutput}
	Principal Curve Fitting

Call: prcurve(X = abernethy2, method = "ca", vary = TRUE, trace = TRUE,
plotit = TRUE, penalty = 1.4)

Algorithm converged after 6 iterations

           SumSq Proportion
Total     103234      1.000
Explained  98864      0.958
Residual    4370      0.042

Fitted curve uses 218.3391 degrees of freedom.
\end{Soutput}
\end{Schunk}
Compare that with the variance explained by the PCA and CA
\begin{Schunk}
\begin{Sinput}
> varExpl(aber.pca)
\end{Sinput}
\begin{Soutput}
      PC1
0.4649883
\end{Soutput}
\begin{Sinput}
> varExpl(aber.ca)
\end{Sinput}
\begin{Soutput}
      CA1
0.3098955
\end{Soutput}
\end{Schunk}
How well has the principal curve done compared to the more traditional
techniques?

Plot the principal curve
\begin{Schunk}
\begin{Sinput}
> ## Plot the fitted curve
> plot(aber.pc2)
\end{Sinput}
\end{Schunk}
Can you see why the principal curve performs much better than the other two
ordination techniques?

A longer chunk of code is required to draw rates of change and compare the
various methods
\begin{Schunk}
\begin{Sinput}
> Depth <- abernethy$Depth
> Age <- abernethy$Age
> layout(matrix(c(1,2,2), ncol = 3))
> nseg <- nrow(abernethy2) - 1
> RoC <- unclass(diff(gradientDist(aber.pc2))) / diff(Age)
> plot(y = Age[-length(Age)],
+      x = RoC * 1000, type = "n",
+      ylim = rev(range(Age)),
+      ylab = "Age (Radiocarbon years BP)",
+      xlab  = expression("Rate of Change" ~ (kyr^{-1})),
+      main = "a", cex.main = 1.5)
> segments(x0 = rep(0, nseg), y0 = Age[-length(Age)],
+          x1 = RoC * 1000, y1 = Age[-length(Age)])
> plot(gradientDist(aber.pc2), orderBy = Age,
+      xlim = rev(range(Age)),
+      type = "o", flipAxes = TRUE, xlab = "Age (Radiocarbon years BP)",
+      main = "b",
+      cex = 0.8, pch = 21, col = "black", bg = "black", cex.main = 1.5)
> lines(gradientDist(aber.pca), orderBy = Age,
+       lty = "dashed", flipAxes = TRUE, type = "o",
+       cex = 0.8, pch = 22, col = "black", bg = "black")
> lines(1 - gradientDist(aber.ca), orderBy = Age,
+       lty = "dotdash", flipAxes = TRUE, type = "o",
+       cex = 0.8, pch = 23, col = "black", bg = "black")
> legend("topright", bty = "n", pch = 21:23,
+        legend = c(expression(PCurve), expression(PCA[1]), expression(CA[1])),
+        lty = c("solid","dashed","dotdash"),
+        col = c("black","black","black"),
+        pt.bg = c("black","black","black"), inset = 0.01, cex = 1.4,
+        seg.len = 4)
> layout(1)
\end{Sinput}
\end{Schunk}

The individual curves fitted to each taxon can be visualised, again with a long
chunk of code. Here, only the most abundant taxa are shown
\begin{Schunk}
\begin{Sinput}
> taxaWant <- chooseTaxa(aber, max.abun = 25, n.occ = 10, value = FALSE)
> layout(matrix(seq_len(sum(taxaWant)), ncol = 3))
> op <- par(mar = c(4,4,3,1) + 0.1)
> plot(sppResponse(aber.pc2), which = taxaWant)
> par(op)
> layout(1)
\end{Sinput}
\end{Schunk}

Using the two plots, do you feel the principal curve has done a good job of
describing floristic change in the Abernethy core?
\end{document}


\documentclass[landscape, letterpaper]{extarticle}

\usepackage[margin=.25in]{geometry}
\usepackage{fontspec}
\usepackage{amssymb}
\usepackage{mathtools}
\usepackage{multicol}
\usepackage{enumitem}
\usepackage{scrextend}
\usepackage{microtype}
\usepackage{bm}
\usepackage[super]{nth}

\renewcommand{\complement}[1]{{#1}^\mathsf{c}}
\DeclarePairedDelimiter{\floor}{\lfloor}{\rfloor}

\begin{document}
\begin{multicols}{3}
    \section*{ER Model}
    \begin{itemize}[noitemsep,nolistsep]
        \item \textbf{Entity:} A distinguishable object, e.g.\ person, thing, object.
        \item \textbf{Relationship:} Represents the fact that certain entities are related to each other.
        \item \textbf{Set:} A set of items with the same type
        \item \textbf{Attribute:} Property of an entity or relationship.
        \item \textbf{Key:} Minimal set of attributes that uniquely identifies entity in set.
        \item \textbf{Role:} Labels added when an entity serves multiple functions in the relationship set.
    \end{itemize}
    \subsection*{Diagram}
    \begin{itemize}[noitemsep,nolistsep]
        \item \textbf{Plain line:} Many to many
        \item \textbf{Bold line:} At least one
        \item \textbf{Arrow:} At most one
        \item \textbf{Bold arrow:} Exactly one
        \item \textbf{Bold arrow one way:} Many to one
              \begin{itemize}[noitemsep,nolistsep]
                  \item \boxed{Employees} -- WorksIn \bm{\rightarrow} \boxed{Departments}
                  \item Each employee can be part of at most one department
              \end{itemize}
        \item \textbf{Bold arrow two way:} One to one
              \begin{itemize}[noitemsep,nolistsep]
                  \item \boxed{Employees} \bm{\leftarrow} Manages \bm{\rightarrow} \boxed{Departments}
                  \item Only one manager per department
              \end{itemize}
    \end{itemize}
    \section*{Relational Algebra}
    \begin{itemize}[noitemsep,nolistsep]
        \item \textbf{Select: } \( \sigma_{condition}(relation) \)
              \begin{itemize}[noitemsep,nolistsep]
                  \item Operators: \( <, \leq, \geq, >, =, \neq \)
                  \item Condition: attribute operator constant OR attribute operator attribute
              \end{itemize}
        \item \textbf{Project: } \( \pi_{attribute~list}(relation) \)
        \item \textbf{Union:} \( A \cup B \)
        \item \textbf{Intersect:} \( A \cap B \)
        \item \textbf{Difference:} \( A - B\)
        \item \textbf{Cartesian Product:} \( A \times B \)
        \item \textbf{Rename:} \( \rho_{(p, q, \ldots, z)}relation\)
        \item \textbf{Join:} \( R \bowtie_{condition} S\)
              \begin{itemize}[noitemsep,nolistsep]
                  \item Equivalent to \(\sigma_{condition} (R \times S)\)
              \end{itemize}
        \item \textbf{Equijoin:} \( R \bowtie_{r = s} S\)
        \item \textbf{Natural join:} \(R \bowtie S \)
              \begin{itemize}[noitemsep,nolistsep]
                  \item Equijoin but with implicit condition
              \end{itemize}
        \item \textbf{Division:} \(R \div S\)
              \begin{itemize}[noitemsep,nolistsep]
                  \item Tuples in R that match all tuples in S
                  \item Derived: \(\pi_n (R) - \pi_n({({\pi_n(R) \times S}) -R})\)
              \end{itemize}
    \end{itemize}
    For set operations (Union, Intersect, Difference), the two relations need to be union compatible i.e.\ same column headers
    \section*{SQL}
    \begin{itemize}[noitemsep,nolistsep]
        \item \verb|CREATE TABLE name(prop type, constraint);|
        \item \verb|DROP TABLE name;|
        \item \verb|ALTER TABLE name action;|
              \begin{itemize}[noitemsep,nolistsep]
                  \item \verb|ADD/MODIFY (prop type/constraint)|
              \end{itemize}
        \item \verb|INSERT INTO name(properties) VALUES(vals);|
        \item \verb|UPDATE name SET prop=val WHERE condition;|
        \item \verb|DELETE FROM name WHERE condition;|
        \item \verb|CREATE VIEW AS SELECT ...;|
    \end{itemize}

    \section*{Disk}
    \textbf{Key:} Improve performance by reducing seek/rotation delays

    \begin{itemize}[noitemsep,nolistsep]
        \item Arrange files sequentially on disk.
              \begin{itemize}[noitemsep,nolistsep]
                  \item Exploit spatial locality.
              \end{itemize}
        \item Read/Write in bigger chunks/pages.
              \begin{itemize}[noitemsep,nolistsep]
                  \item Large page size: related data stored close by might get loaded, avoiding another page transfer
                  \item Small page size: Reduce transfer time, reduce required buffer in main memory
                  \item Typically: 4096 bytes
              \end{itemize}
        \item Buffering reads and writes.
              \begin{itemize}[noitemsep,nolistsep]
                  \item Cache requests so that you don't have to go to disk
                  \item When cache full, throw away least recently used
                  \item Keep track of clean/dirty so if clean no need to write back.
              \end{itemize}
    \end{itemize}
    \section*{Organization}
    \subsection*{Heap File}
    \begin{itemize}[noitemsep,nolistsep]
        \item Rows get appended to end of file as inserted (unordered)
        \item Deleted rows create gaps (Need compaction)
    \end{itemize}
    Performance:
    \begin{itemize}[noitemsep,nolistsep]
        \item Assume File has F pages
        \item Average search: \(F/2\) if exists, \(F\) otherwise
        \item Average delete: \(F/2 + 1\) if exists, \(F\) otherwise
    \end{itemize}
    \subsection*{Sorted File}
    \begin{itemize}[noitemsep,nolistsep]
        \item Rows get inserted based on specific row and everything to shift
              \begin{itemize}[noitemsep,nolistsep]
                  \item Can leave empty space around (fill factor)
                  \item Can use overflow pages (chains)
              \end{itemize}
        \item Deleted rows cause everything to get shifted
    \end{itemize}
    Performance:
    \begin{itemize}[noitemsep,nolistsep]
        \item Assume File has F pages
        \item Average search: \(\log_2 F\) can do binary search
        \item Average insert and delete: \(search + F\) Need to shift rest of file
    \end{itemize}
    \subsection*{Index File}
    Allows to locate rows without having to scan entire table using a search key (not candidate key, only used for search and may not be unique).

    Different structures:

    Integrated Structure:
    \begin{itemize}[noitemsep,nolistsep]
        \item Top contains mechanism for locating index entries.
        \item Bottom contains index entries with full records.
    \end{itemize}

    Secondary Index with Storage structure:
    \begin{itemize}[noitemsep,nolistsep]
        \item Separate file contains mechanism for locating index entries and index entries with pointers to storage.
        \item Storage contains records.
    \end{itemize}

    Index types:
    \begin{itemize}[noitemsep,nolistsep]
        \item Clustered: Data rows are same order as index.
        \item Unclustered: Opposite of above.
        \item Both can be either integrated or split.
        \item Dense: Index for ever record.
        \item Sparse: Index only for some records.
        \item Can be mixed in with the above.
    \end{itemize}

    \section*{Tree Structured Index}
    Can be used to do either an equality search or range search.

    Can be stored in binary tree, every operation will be O(height of tree), or we can use B+ tree instead and store multiple things per node. Complexity will be O(height) but height will be \(\log_{m}(Entries)\) instead of \(\log_{2}(Entries)\)

    \subsection*{ISAM}
    \begin{itemize}[noitemsep,nolistsep]
        \item Static tree, doesn't get updated when insert/delete
        \item Each node is a page
        \item When leaf pages full and you need more space, you add a new overflow page into the chain.
        \item Useful when not many updates
    \end{itemize}

    \subsection*{B+ Tree}
    \begin{itemize}[noitemsep,nolistsep]
        \item Dynamically updates whenever new data is added/removed
        \item Each node still a page.
        \item Search/Insert/Delete guaranteed to be \(\log_{F}(N)\)
        \item Leaf pages form a sequence set (from left to right, they are in order)
        \item Minimum 50\% occupancy, except for the root
        \item When split leaf, key is \textbf{copied} up. When split non-leaf, value is \textbf{moved} up.
        \item On deletion, if below minimum occupancy, rebalance
              \begin{itemize}
                  \item Try to redistribute by taking node with same parent
                  \item If fails, merge
                  \item If merge, delete node
                  \item Propagate
              \end{itemize}
    \end{itemize}

    \subsection*{Hashing}
    \begin{itemize}[noitemsep,nolistsep]
        \item Hash keys into page address
        \item \(H(K): K \rightarrow A\)
        \item Possible hashing functions:
              \begin{itemize}[noitemsep,nolistsep]
                  \item Let last digit of key to be page location
                  \item Folding: Replace text with ASCII code, combine into 4 digits then add, modulus to fit into address space
                  \item Midsquare: Square key and take middle
                  \item Radix transformation: Change radix then modulus
                  \item Same as folding, but with multiplication by index.
              \end{itemize}
        \item \textbf{Key Space:} Set of all possible values for keys
        \item \textbf{Address Space:} Set of all possible values for page address
        \item Address space is usually smaller than key space
        \item Address space must be able to store all records
        \item \textbf{Randomization:} Records should be spread randomly
        \item \textbf{Collision:} Two different keys may have same address
        \item Modulo prime number to increase likelihood of using all spaces
        \item Range and partial key search cannot be done
        \item Equality search is constant time
    \end{itemize}
    \subsection*{Collision}
    Static Methods:
    \begin{itemize}[noitemsep,nolistsep]
        \item Linear Probing
              \begin{itemize}[noitemsep,nolistsep]
                  \item Go down <step> addresses if collision
                  \item Wastes space as rows get deleted
              \end{itemize}
        \item Double Hashing
        \item Separate overflow
    \end{itemize}
    Dynamic Methods:
    \begin{itemize}[noitemsep,nolistsep]
        \item Extendable Hashing
              \begin{itemize}[noitemsep,nolistsep]
                  \item use first k bits of hash, increasing k as collisions happen.
              \end{itemize}
        \item Linear Hashing
    \end{itemize}
    \subsection*{In SQL}
    \begin{itemize}[noitemsep,nolistsep]
        \item Unclustered
              \begin{itemize}[noitemsep,nolistsep]
                  \item \verb|CREATE INDEX idx ON table(prop)|
                  \item Can add \verb|UNIQUE|
              \end{itemize}
        \item Clustered: create table with \verb|WITHOUT ROWID|
    \end{itemize}
    \section*{Normal Forms}
    \begin{itemize}[noitemsep,nolistsep]
        \item Functional Dependency \( X \rightarrow Y\)
              \begin{itemize}[noitemsep,nolistsep]
                  \item \( X \rightarrow YZ \equiv (X \rightarrow Y \text{ and } X \rightarrow Z\))
              \end{itemize}
        \item Armstrong's Axioms
              \begin{itemize}[noitemsep,nolistsep]
                  \item \textbf{Reflexivity:} \(X \rightarrow X \text{ or }XY \rightarrow X\)
                  \item \textbf{Augmentation:} \( X \rightarrow Y: XZ \rightarrow YZ\)
                  \item \textbf{Transitivity:} \( X\rightarrow Y, Y \rightarrow Z: X \rightarrow Z\)
              \end{itemize}
        \item Given a set \(F\) of FDs, the closure of \(F\), \(F^+\) is all the FDs implied from F.
        \item \(X^+\) is the set of all attributes that can be determined by \(X\)
        \item \textbf{Superkey:} Set of attributes that identify row.
        \item \textbf{Candidate Key:} Minimal superkey
        \item \textbf{Key:} Minimal and Unique
        \item \textbf{Unique:} If two tuples have same values, the rest of the attributes must always be the same.
        \item \textbf{Minimal:} Cannot eliminate any attribute without becoming not a key.
        \item \textbf{\nth{3} Normal Form:} All attributes are functionally dependent on solely the primary key.
              \begin{itemize}[noitemsep,nolistsep]
                  \item \(X \rightarrow A\)
                  \item \(X\) is super key of \(R\) or
                  \item \(Y\) is part of a key
              \end{itemize}
        \item \textbf{Boyce-Codd Normal Form:} Stricter 3NF
              \begin{itemize}[noitemsep,nolistsep]
                  \item \(X \rightarrow A\)
                  \item \(X\) is super key of \(R\)
              \end{itemize}
        \item \textbf{Lossless join BCNF:} Not always able to preserve, keep minimal cover
              \begin{itemize}[noitemsep,nolistsep]
                  \item \(X \rightarrow A\) in \(R(Z)\) that violates BCNF
                  \item Split into \(R1\left( X^+ \cap Z \right)\) and
                  \item \(R2\left( \left(Z - X^+\right) \cup X \right)\)
              \end{itemize}
        \item \textbf{Lossless join 3NF:} Start with lossless BCNF decomposition
              \begin{itemize}[noitemsep,nolistsep]
                  \item \(X \rightarrow A\) in \(R(Z)\) that isn't preserved, create relation with \(XA\)
              \end{itemize}
    \end{itemize}
    \section*{Database Tuning}
    \begin{itemize}[noitemsep,nolistsep]
        \item \textbf{Denormalize:} Avoid join but add redundancy
        \item \textbf{Performance Bottleneck:} Heavily used tables, deep index, wide rows
        \item \textbf{Parititioning:} Horizontal: Split rows; Vertical: Split columns + replicate keys
        \item \textbf{Clustered Index:} When point queries with large results, range queries, order-by
        \item \textbf{Unclustered Index:} When point queries with small results, nested loop joins
    \end{itemize}

    \section*{Examples}
    \newtheorem{example}{Example}
    \subsection*{Relational Algebra}
    \begin{example}
        SQL Division
        Define the following tables:

        R = (n, x), S = (x)

        SQL query:
        \begin{verbatim}
CREATE TABLE R(n INTEGER, x TEXT);

CREATE TABLE S(x TEXT);

SELECT n FROM R
EXCEPT
SELECT n FROM
     (SELECT R.n, S.x FROM R, S
     EXCEPT
     SELECT * FROM R);
        \end{verbatim}
    \end{example}
\end{multicols}
\end{document}


\subsection{Experiment} \label{experiment}

The proposed sliding window filter has several model parameters that need to be carefully tuned in order to achieve optimal performance. Depending on the application domain we need to select an appropriate window and step size, time series normalization, dissimilarity threshold, and filter criterion. In the following we describe all parameter settings that were assessed in our empirical study:

\begin{itemize}

\item
The \textbf{window size} determines the number of most recent measurements contained in the examined time series subsequences. We tested four different sizes that were learned from the training gesture, including \textbf{min}, \textbf{max}, and \textbf{avg} length as well as the \textbf{mid}-point of the range. \\

\item
The \textbf{step size} defines the gap between consecutive time series windows. As default setting we use one tenth of the window size. \\

\item
For online gesture recognition we employ the nearest neighbor classifier in combination with the DTW distance, where we evaluate 34 different Sakoe-Chiba \textbf{band} sizes, ranging from 1 \% to 100 \%.
Prior to pair-wise comparing sliding windows and training gestures, the corresponding time series should be normalized. We evaluate $\eta$, $z$, and no \textbf{normalization}. \\

\item
The dissimilarity \textbf{threshold} defines the time series distance at which a sliding window and a training gesture are considered to belong to the same class. We determine the threshold for an individual class by measuring the distances between all samples of that particular class and all instances of other classes. In our empirical study we evaluate the threshold influence for: (i) one half of the minimum distance - \textbf{HMinD}, (ii) one half of the average distance - \textbf{HAvgD}, and (iii) one half of the midpoint distance - \textbf{HMidD}. \\

\item
The \textbf{filter criterion} is an essential part of our proposed approach. In our empirical study we evaluate the performance of the two filter criteria, namely the sample variance \textbf{VAR} and the length normalized complexity estimate \textbf{LNCE} of a time series. Both filters are tested with different factors that increase the size of the filter interval from 100 \% to 300 \%.

\end{itemize}

Figure \ref{fig:experiment} visualizes the online gesture recognition results for a sample time series stream processed by our proposed sliding window filter,
after selecting the above described model parameters with help of the recorded training gestures.

\begin{figure}
    \resizebox {\textwidth} {!} {
        \begin{tikzpicture}
            \begin{axis}[
                xmin=0,
                xmax=2426,
                ymin=-16,
                ymax=16,
                width=10*\axisdefaultwidth,
                height=\axisdefaultheight,
                xticklabels={,,},
                yticklabels={,,}]
                \addplot[blue, mark=none, opacity=0.4] table[x=t, y=x] {../data/fig/experimentee_result2/exp1.dat};
                \addplot[red, mark=none, opacity=0.4] table[x=t, y=y] {../data/fig/experimentee_result2/exp1.dat};
                \addplot[green, mark=none, opacity=0.4] table[x=t, y=z] {../data/fig/experimentee_result2/exp1.dat};
                \addplot+[fill, opacity=0.5, red, mark=none] coordinates {(294, -16) (307, -16) (307, 16) (294, 16)} --cycle;
                \addplot+[fill, opacity=0.5, green, mark=none] coordinates {(307, -16) (357, -16) (357, 16) (307, 16)} --cycle;
                \addplot+[fill, opacity=0.5, red, mark=none] coordinates {(357, -16) (359, -16) (359, 16) (357, 16)} --cycle;
                \addplot+[fill, opacity=0.5, red, mark=none] coordinates {(497, -16) (508, -16) (508, 16) (497, 16)} --cycle;
                \addplot+[fill, opacity=0.5, green, mark=none] coordinates {(508, -16) (562, -16) (562, 16) (508, 16)} --cycle;
                \addplot+[fill, opacity=0.5, blue, mark=none] coordinates {(562, -16) (564, -16) (564, 16) (562, 16)} --cycle;
                \addplot+[fill, opacity=0.5, red, mark=none] coordinates {(712, -16) (722, -16) (722, 16) (712, 16)} --cycle;
                \addplot+[fill, opacity=0.5, green, mark=none] coordinates {(722, -16) (777, -16) (777, 16) (722, 16)} --cycle;
                \addplot+[fill, opacity=0.5, blue, mark=none] coordinates {(777, -16) (778, -16) (778, 16) (777, 16)} --cycle;
                \addplot+[fill, opacity=0.5, blue, mark=none] coordinates {(940, -16) (945, -16) (945, 16) (940, 16)} --cycle;
                \addplot+[fill, opacity=0.5, green, mark=none] coordinates {(945, -16) (1010, -16) (1010, 16) (945, 16)} --cycle;
                \addplot+[fill, opacity=0.5, blue, mark=none] coordinates {(1010, -16) (1023, -16) (1023, 16) (1010, 16)} --cycle;
                \addplot+[fill, opacity=0.5, red, mark=none] coordinates {(1310, -16) (1316, -16) (1316, 16) (1310, 16)} --cycle;
                \addplot+[fill, opacity=0.5, green, mark=none] coordinates {(1316, -16) (1367, -16) (1367, 16) (1316, 16)} --cycle;
                \addplot+[fill, opacity=0.5, red, mark=none] coordinates {(1367, -16) (1375, -16) (1375, 16) (1367, 16)} --cycle;
                \addplot+[fill, opacity=0.5, red, mark=none] coordinates {(1681, -16) (1689, -16) (1689, 16) (1681, 16)} --cycle;
                \addplot+[fill, opacity=0.5, green, mark=none] coordinates {(1689, -16) (1746, -16) (1746, 16) (1689, 16)} --cycle;
                \addplot+[fill, opacity=0.5, blue, mark=none] coordinates {(1746, -16) (1748, -16) (1748, 16) (1746, 16)} --cycle;
                \addplot+[fill, opacity=0.5, red, mark=none] coordinates {(2082, -16) (2090, -16) (2090, 16) (2082, 16)} --cycle;
                \addplot+[fill, opacity=0.5, green, mark=none] coordinates {(2090, -16) (2146, -16) (2146, 16) (2090, 16)} --cycle;
                \addplot+[fill, opacity=0.5, red, mark=none] coordinates {(2146, -16) (2147, -16) (2147, 16) (2146, 16)} --cycle;
                \addplot+[fill, opacity=0.5, blue, mark=none] coordinates {(2311, -16) (2388, -16) (2388, 16) (2311, 16)} --cycle;
                \addplot+[fill, opacity=0.5, red, mark=none] coordinates {(2297, -16) (2362, -16) (2362, 16) (2297, 16)} --cycle;
            \end{axis}
        \end{tikzpicture}
    }
    \caption{Visualized results of online gesture recognition for a sample time series stream. We highlight true positives in green, false positives in red, false negatives in blue, and true negatives in transparent.
    Although we see short false detection intervals before or after true positives, seven out of eight gestures were assigned to the correct class label.}
    \label{fig:experiment}
\end{figure}


\section{Comparison to other methods} \label{sec:compare}

\begin{table}
\caption{Table of $\theta_{\rm inf}$ - $\theta_{\rm true}$ and uncertainties for different CIGALE setups  }
\begin{center}
\begin{tabular}{ccccc} \toprule
set & CIGALE A & CIGALE B & CIGALE C & CIGALE D \\
$\Delta M_{tot}$ & 0.07 & 0.09 & 0.27 & 0.05\\
$M_{err}$ & 0.13 & 0.20 & 0.16 & 0.16\\
$\Delta$ Age & 1.78 & 1.59 & 1.97& 1.92\\
$Age_{err}$ & 2.36 & 2.63 & 2.47 & 2.47\\
$\Delta$ Z & 0.0037 & 0.0026 & 0.0027 & 0.0027\\
$Z_{err}$ & 0.0091 & 0.0085 & 0.0089 & 0.0089\\
\hline
\hline
\end{tabular} \label{tab:setups}
\end{center}
\end{table}

\begin{figure}
\begin{center}
\includegraphics[width=\textwidth]{figs/mini_mocha_cigale.png}
\caption{The properties inferred from CIGALE photometry fit as a function of true properties. Configuration CIGALE A, B, C, and D on one plot.
}
\label{fig:photo_cigaleALL}
\end{center}
\end{figure}


\documentclass[a4paper,twocolumn]{article}

\usepackage[english]{babel}
\usepackage[utf8]{inputenc}
\usepackage{graphicx}
\usepackage{fullpage}
\usepackage{amssymb}
\usepackage{caption}

\newenvironment{Figure}
  {\par\medskip\noindent\minipage{\linewidth}}
  {\endminipage\par\medskip}

\title{Progressive Neural Networks $-$ summary}
\author{Matěj Nikl}

\begin{document}
\maketitle
\noindent
Progressive Neural Networks (PNNs) are networks trying to address the task of using, transferring and not forgetting previously learned knowledge to learn yet another task, preferably with faster convergence. It retains a pool of trained models throughout training and learns connections to them to harvest potentially useful features while also learning new ones. What it means is that it has the ability to choose whether the previously learned knowledge is useful for the new task and thus to transfer it, or even to ignore it completely (by zeroing the lateral connections).

    \begin{figure}[!h]
        \centering
        \includegraphics[width=0.75\columnwidth]{PNN.png}
        \caption{A three column progressive network}
    \end{figure}

\subsection*{Basic definition}
A PNN starts with a single column $-$ a DNN with $L$ layers and hidden activations $h_i^{(1)} \in \mathbb{R}^{n_i}$, with $n_i$ the number of units at layer $i \le L$, and parameters $\Theta^{(1)}$ trained to convergence. When adding a second task, the parameters $\Theta^{(1)}$ are ``frozen'' and a new column with parameters $\Theta^{(2)}$ is instantiated (with random initialization), where layer $h_i^{(2)}$ receives input from both $h_{i-1}^{(2)}$ and $h_{i-1}^{(1)}$ via lateral connections. This generalizes to $K$ tasks as follows (biases omitted for clarity):
\[
    h_i^{(k)} = f \left( W_i^{(k)}h_{i-1}^{(k)} + \sum_{j<k} U_i^{(k,j)} h_{i-1}^{(j)} \right)
\]
where $W_i^{(k)} \in \mathbb{R}^{n_i \times n_{i-1}}$ is the weight matrix of layer $i$ of column $k$, $U_i^{(k,j)} \in \mathbb{R}^{n_i \times n_{i-1}}$ are the lateral connections from layer $i-1$ of column $j$, to layer $i$ of column $k$ and $h_0$ is the network input. $f$ is a element-wise non-linearity, $f(x) = \max (0, x)$ for all intermediate layers.

\paragraph{Adapters.} In practice, non-linear lateral connections called \textit{adapters} are used. The serve both to improve initial conditioning and perform dimensionality reduction, so that the number of parameters stemming from the lateral connections is in the same order as $\left| \Theta^{(1)} \right|$ as $k$ grows.

\subsection*{Principles of modularization}
The basic building block of PNNs is a \textit{column}. A column is a neural network with lateral connections to all previously trained columns, each trying to solve just one task. That might encourage the thinking of a column as a potential module. However, it cannot be taken away and used on its own as a some kind of a standalone unit $-$ it always needs all of its previous columns with whom it was trained with to function properly, because it might be extremely dependent on features (knowledge) extracted from input by them. So, even though we might like to see it as a module, it definitely cannot be used that way.

\subsection*{Principles of growing}
For each new task a new column is allocated. The knowledge from all previously learned tasks has the potential to be transferred, while being protected against forgetting, because all previous columns have frozen weights. This way, the PNN can theoretically grow for as many tasks as needed (while potentially transferring the knowledge from all of the previously learned ones), however in practice, some other techniques like pruning and online compression must be used during training for the number of parameters to be reasonable.

    \begin{figure*}[ht]
        \centering
        \includegraphics[width=0.65\textwidth]{2-column.png}
        \caption{A two column progressive network $-$ knowledge transfer}
    \end{figure*}

    \begin{figure*}[ht]
        \centering
        \includegraphics[width=0.9\textwidth]{3-column.png}
        \caption{A three column progressive network $-$ knowledge transfer to the rightmost task}
    \end{figure*}

    \begin{figure*}[ht]
        \centering
        \includegraphics[width=\textwidth]{perf.png}
        \captionsetup{singlelinecheck=off}
        \caption[trololo]{(a) Transfer matrix. Colors indicate transfer scores (clipped at 2). For progressive nets, the first column is trained on Pong, Noisy, or H-flip (table rows); the second column is trained on each of the other pong variants (table columns). (b) Example learning curves. \\
        \begin{itemize}
            \item \textbf{Baseline 2} $-$ a single column, pretrained on a source task and fine-tuned on the target task (output layer only)
            \item \textbf{Baseline 3} $-$ the same as baseline 2 but the whole model is fine-tuned
            \item \textbf{Baseline 4} $-$ a 2 column progressive architecture, with previous columns initialized randomly and frozen
        \end{itemize}
        }
    \end{figure*}

\end{document}


\subsubsection{Finite variance}\label{rectassumptionsvariance}

In the collision and messages plots we haven't trends but the behaviour is
blockwise, as we can see for example in \figref{fig:recttimevariance} with
swinging results and also in this case the order of magnitude of the residual is
lower than the one of the average predicted response, so we can conclude that
the assumption of finite variance is still valid.

As for other scenarios, to verify the assumption for the residuals of the 99th
percentile of the total broadcast time we need to perform a logarithmic
transformation of the predicted variable. The result is shown in
\figref{fig:recttimevariance}.

\begin{figure}[htb]
	\centering
	\includegraphics[width=0.6\textwidth]{img/rect/collisionvariance.png}
	\caption{Scatter plot of the variance of the residuals of the
	collisions. No trends but blockwise
	behaviour}\label{fig:recttimevariance}
\end{figure}


\section{EnsembleModel}
\label{sec:ensembleModel}

In most RAVEN MultiRun steps, a sampler provides a variety of inputs to a model, whose outputs then produce a
set of responses that can be used in many ways.  However, sometimes a single model is insufficient to produce
the response we want.  When this happens, the EnsembleModel opens up a new vista of simulation options.

The EnsembleModel is a powerful tool for chaining multiple models together, with outputs of some models
becoming inputs for others.  Whether this is adding a preprocessing or postprocessing model, or linking
multiple physics-based models together, the EnsembleModel provides the functionality to see many models
together as a single model in a MultiRun step.

Note that the EnsembleModel combines multiple models for \emph{each sample}; that is, the whole ensemble of
models is run for each sample taken by RAVEN.  If instead you want to postprocess a batch of sampled data, try
the PostProcessor models instead.

\subsection{Introduction: The EnsembleModel}
The key to the EnsembleModel is careful definition of the inputs and outputs of each model.  When an
EnsembleModel is defined in a RAVEN run, RAVEN will automatically scan the inputs and outputs of each model
and determine the right order to evaluate the models in (or \emph{graph}).

\subsection{Example: ballistics and impact}
% TODO add an image showing these two models, their inputs and outputs
As a basic example, consider two physics models.  The first is a ballistics code, used to determine the
kinetic energy $E$ of a ball when it hits the ground, with a path depending on its initial height $y_0$, initial velocity
$v_0$, mass $m$ and initial angle $\theta_0$.  We'll assume we're near the Earth's surface and aside from the initial
launch height, we're launching over a flat surface.  This can be represented as a functional $E(y_0,v_0,m,\theta_0)$.

The second physics model is an impact code that estimates the diameter of the crater $D$ made when a ball
impacts.  With a few simplifying assumptions ($D$ is under 1 km, the impact is near Earth's surface, the
impact is in dry sand with fixed density), size $D$ can be determined as a function of the ball's kinetic
energy when impacting $E$, its mass $m$, and its radius $r$: $D(E,m,r)$.

We can see that the input $E$ for the impact model is calculated by the ballistics code.  In RAVEN the
EnsembleModel will automatically detect this, and run the ballistics code before the impact code in each
iteration.

In general, the EnsembleModel also has some finite capacity for resolving circular dependencies using Picard
iteration.  See the manual for more information on this capability.

Setting up an EnsembleModel can be more complex than other models, so we'll walk through the input using our example
outlined above.  For
our purposes, we're going to let RAVEN perturb the codes using the GenericCode interface.  We'll call the
ballistics code \texttt{ballistics.py} with keyword input file \texttt{ballistics\_input.txt} and similarly the impact
code \texttt{impact.py} with keyword input file \texttt{impact\_input.txt}.

The RAVEN input file for this example is located at
\begin{verbatim}
raven/tests/framework/user_guide/EnsembleModel/basic.xml
\end{verbatim}
with run directory \texttt{run\_basic}.  The models and inputs are in the run directory.

Now we turn our attention to the RAVEN input file. We will discuss \xmlNode{DataObjects}, \xmlNode{Files},
\xmlNode{Models}, and \xmlNode{Steps} in detail; the rest are typical usage and need no special attention.

\subsubsection{DataObjects}
The key to a successful EnsembleModel is setting up the DataObject inputs and outputs.  Each model has a
\xmlNode{TargetEvaluation} DataObject that specifies what the inputs and outputs are for that model.  In
addition, any \xmlNode{Output} DataObjects in the \xmlNode{MultiRun} step can collect any or all of the
variables used throughout the EnsembleModel calculation, either as inputs or outputs.

In this
example, our convention is to name the \xmlNode{TargetEvaluation} DataObjects as \xmlString{model\_data}, replacing
\xmlString{model} with the model name.  Additionally, we add a DataObject to store the final results of the
\xmlNode{MultiRun} step:
\xmlExample{framework/user_guide/EnsembleModel/basic.xml}{DataObjects}
In this example, we see three DataObjects.  \xmlString{ballistics\_data} determines the inputs and outputs of the
\texttt{ballistics.py} model, so for its input space we have \texttt{y0,v0,ang,m}, which correspond to
$y_0,v_0,\theta_0,m$; for the output, we have \texttt{E} (for $E$).

The second DataObject, \xmlString{impact\_data}, delineates the inputs and outputs of the \texttt{impact.py}
model, so for its input space we have \texttt{E,m,r}, while in the output space we have \texttt{D}.  RAVEN
will use these first two DataObjects to map the order in which these two models should be run (first
ballistics, then impact) and transfer data from one to the next.

The third DataObject, \xmlString{final\_results}, can contain any information we want it to.  In this case,
we want to collect all the variables in their original spaces, so we take \texttt{y0,v0,ang,m,r} as inputs and
\texttt{E,D} as outputs.

\subsubsection{Files}
Since we know both of our models take input files to set the variable values, we inform RAVEN about the
template input files and
give those files RAVEN names in the \xmlNode{Files} block.  In this example, our input file for \texttt{ballistics.py} is
\texttt{ballistics\_template.txt}, which we simply name \xmlString{ballistics\_input}.  Similarly,
for \texttt{impact.py} the input file is \texttt{impact\_template.txt}, which we name
\xmlString{impact\_input}:
\xmlExample{framework/user_guide/EnsembleModel/basic.xml}{Files}
We will pause to note that the file \texttt{ballistics\_template.txt} is different than the file
\texttt{ballistics\_input.py}, and similarly for the impact code.  In the template file, we've replaced
variable values with the \texttt{\$RAVEN-\$} wildcard, since we will be using the Generic Code model for these
two models.  Having a template input file may not be required for all code interfaces, but it is required for
the Generic interface. See more about this interface in the Models section and in the user manual.

\subsubsection{Models}
Once we've specified the DataObjects and files, we are prepared to set up the \xmlNode{Models} block.  To set
up an EnsembleModel, we define each sub-module by itself as if it were a stand-alone RAVEN model, and then
combine them by defining an \xmlNode{EnsembleModel}:
\xmlExample{framework/user_guide/EnsembleModel/basic.xml}{Models}
The first model we will look at is the definition of the \xmlString{ballistics} model.  We note that is uses
the \xmlString{GenericCode} subType, so it will use the interface as described in the user manual, where
wildcards in the input file are replaced with RAVEN's sampled values.  The \xmlNode{clargs} (command line
arguments) include the \xmlString{python} command, the \xmlString{-i} flag to specify the input file, and the
\xmlString{-o} flag to specify the output file.  These nodes are specific to the Generic code interface and
will not apply to all codes or models. The \xmlString{Impact} model is defined in a similar manner.

With both Models defined, we can construct the \xmlNode{EnsembleModel}, which we've named
\xmlString{ballistic\_and\_impact} to describe the codes that are paired (you can name this whatever you
want).  There is no \xmlAttr{subType} for the \xmlNode{EnsembleModel} currently, so we leave it blank.

Within the \xmlNode{EnsembleModel} node there are two \xmlNode{Model} nodes listed, which will inform the
EnsembleModel which models need coupling.  The order listed does not matter; the EnsembleModel will sort out
the order based on the \xmlNode{TargetEvaluation} DataObject of each model.

The first \xmlNode{Model} we list within the \xmlNode{EnsembleModel} is the \texttt{ballistics} model.  Note
that the \xmlAttr{class} and \xmlAttr{type} are used along with the name to find the right model.  Note also
the name of the model goes in the \emph{text} of the \xmlNode{Model} node, right after the first closing angle
bracket.  The subnodes of the \xmlString{ballistics} model are the \xmlNode{Input}, which points to the input
template input file we defined in the \xmlNode{Files} block; and the \xmlNode{TargetEvaluation}, which points
to the \xmlNode{PointSet} we defined in the \xmlNode{DataObjects} block.  These are critical to allowing the
EnsembleModel both to run the \xmlString{ballistics} model correctly, as well as the entire ensemble in a
sensible manner.

In general, some models (such as a \xmlNode{ROM} or \xmlNode{ExternalModel}) do not require any files as
inputs.  In this case, the \xmlNode{Input} for an input-less model should be a placeholder DataObject,
usually with the same input space as the \xmlNode{TargetEvaluation} DataObject, and a placeholder output.
For more information, see examples running ExternalModels in these guides as well as the user manual.

After the \xmlString{ballistics} model, we see a similar structure pointing to the \xmlString{impact} model.
The only differences between this model specification and the ballistics model specificatoin are the names of
the \xmlNode{Input} file and \xmlNode{TargetEvaluation} DataObject.

With the two models defined, and the EnsembleModel assembled, no more work is required to
prepare these two models to run.  More complicated systems may require additional nodes in the EnsembleModel;
see the user manual for details.

\subsubsection{Steps}
Finally, to put all the pieces together, we consider the \xmlNode{Steps} node:
\xmlExample{framework/user_guide/EnsembleModel/basic.xml}{Steps}
We take two steps in this simulation, one \xmlNode{MultiRun} called \xmlString{sample} to sample the EnsembleModel on a grid,
and one \xmlNode{IOStep} to print the results.  These are constructed just as they would be with any other
model; as far as the \xmlNode{Steps} are concerned, the EnsembleModel is like any other model.  Note that we
include the two template input files as \xmlNode{Input} nodes for the sampling step.

We do not discuss the \xmlNode{Distributions}, \xmlNode{Samplers}, or \xmlNode{OutStreams}
here.  These operate in general as they would in any other RAVEN input file.  We will note, however, that the
\xmlNode{Grid} sampler named \xmlString{grid} samples all the inputs that are needed by either of the submodels,
unless the inputs are provided by another model.  In this case, that means we need to sample
$y_0,\theta_0,v_0,m,r$ but we do not sample $E$ since it is calculated by the \xmlString{ballistics} model.

We can find all the sampled and calculated values from both of the models in the output produced by this run,
found at
\begin{verbatim}
raven/tests/framework/user_guide/EnsembleModel/run_basic/results.csv
\end{verbatim}

\subsection{ExternalModel in EnsembleModel}
There are a few details that can make handling \xmlNode{ExternalModel} models challenging in EnsembleModel.
We'll cover a few of these to help smooth over some potential bumps.

\subsubsection{Input Placeholder DataObject}
Firstly, as noted above, both ExternalModel and ROM differ from Code models in a significant way: they
(usually) do not require any input files.  So what do you put as the \xmlNode{Input} for an ExternalModel or
ROM?  To maintain consistency, we leave the \xmlNode{Input} node in place.  Instead of specifying a file
however, a
placeholder DataObject is used instead.  A placeholder DataObject has variables listed in its input, but
for output either the \xmlNode{Output} node is omitted, or the special keyword \texttt{OutputPlaceHolder} is used
instead.  For example, see the excerpt from a regression test:
\xmlExample{framework/ensembleModelTests/index_input_output.xml}{DataObjects}
The inputs \xmlString{first\_in} and \xmlString{second\_in} are placeholder DataObject, with a scalar input
variable for the \xmlNode{Input} and placeholder OutputPlaceHolder as \xmlNode{Output}.  It is always a
\xmlNode{PointSet} type of DataObject.  The key
\texttt{OutputPlaceHolder} is recognized specially in RAVEN and translates into an empty output space.  This
type of DataObject is the only suitable input for an ExternalModel or ROM without an input file.

\subsubsection{DataSet}
Because of the complexity of EnsembleModel, it is possible to have mixed scalars and vectors as variable
inputs to a particular model in the Ensemble.  Furthermore, it is possible to have multiple vector variables
that depend on different independent variables (for example, ``time'' and ``space'').  To accommodate this,
the \xmlNode{DataSet} DataObject can be used.  More can be found on this data structure in the RAVEN user
manual. % TODO is this true?

\subsubsection{Scalar Variables}
Because of the flexibility of the EnsembleModel, and to maintain a level of consistency across all
variables, scalar variables in the EnsembleModel are stored as Numpy arrays with length 1.  In general this
should be transparent for most operations; however, occasionally it may be required to access a scalar
variable by accessing it at index 0.  For example, see the variable ``scalar2'' in a model used in the
regression tests for the EnsembleModel:
\begin{verbatim}
raven/tests/framework/ensembleModelTests/IndexInputOutput/model2.py
\end{verbatim}
\lstinputlisting{../../tests/framework/ensembleModelTests/IndexInputOutput/model2.py}

\subsubsection{Independent Variables}
One feature of ExternalModel is their capacity to take variables exactly as they are from RAVEN, without
going through file IO.  As a result, EnsembleModel frequently contain models that produce pivot-dependent data
that can be used as inputs for other models in the Ensemble. Several terms aptly describe these variables on
which other variables depend, including ``independent variables,'' ``indexes'', and ``pivots''.  In general,
we refer to them as ``indexes''.  While the index is often ``time'' or something
similar, it can be any monotonically-increasing variable.

However, when it is stored in RAVEN, the indexes are not stored like other variables,
because it is a coordinate axes that supports other variables.  This allows RAVEN to do a great many flexible
operations internally.  However, it also means that any time you want to pass an index variable from one model
to an ExternalModel, it can only come attached to another variable that depends on that input.

For example, consider two models.  In the first model, a path followed by a charged particle is traced out in
time.  The outputs of the first model are time $t$, lateral position $x$, and vertical position $y$.  The
second model calculates the energy of the particle at each point in time; however, the second model is written
such that it does not require any specific time values (it just reads them from the input).

To run these two models in Ensemble configuration, \xmlNode{TargetEvaluation} DataObject of the second model
must request one of the time-dependent variables from the first model, $x$ or $y$.  If $t$ is directly
requested, it will not be made available in the second ExternalModel.  Additionally, in the \xmlNode{Model}
definition, all variables (dependent or index) should be listed.  In this example, when defining the model,
$t$ should be listed as well as $x$ in the \xmlNode{variables} node in the second \xmlNode{Model} definition.

For an example of time dependent data passing, consider the regression test example at
\begin{verbatim}
raven/tests/framework/ensembleModelTests/index_intput_output.xml
\end{verbatim}