%%% Local Variables:
%%% mode: latex
%%% TeX-master: "../report"
%%% End:

We have been inspired by many other languages when designing and implementing Kite, most notably the Glasgow Haskell Compiler (GHC)\footnote{\url{http://www.haskell.org/ghc}}.

\subsection{Syntax and semantics}
In this section we will describe how to use the different features of Kite and how they work, i.e.\ the syntax and semantics of the language.

\subsubsection{Basic types and declarations}
Basic declaration of the variables in Kite, types are inferred by the analyzer:

\begin{kite}
one = 1
two = 2.0
truth = True
list = [1, 2, 3, 4]
char = 'a'
pair = (1, 'a')
\end{kite}

Functional languages often make extensive use of lists, which is indeed also the case for Kite. A list is an ordered array of items that can be transformed in many different ways. We use a common short-hand method for describing a list of things, using square brackets \code{[\ ]}. For instance, a \code{List(Int)} (pronounced ``List of Ints'') is denoted \code{[Int]} and a list of a type variables $a$ is \code{[a]}. Lists can be nested, allowing \code{List(List(Int))} denoted \code{[[Int]]}.

The pair type is also abbreviated using \code{( , )}, for instance the type \code{Pair(Int, Bool)} is written \code{(Int, Bool)}

Strings are represented as syntactic sugar for lists of characters. For
instance, \code{['f', 'o', 'o']} has the sugared version \code{"foo"}. This will be discussed later.

Basic arithmetic operators:
\begin{kite}
1 + 1  ----> 2
10 % 2 ----> 0
\end{kite}

\begin{table}[H]
  \centering
  \begin{tabular}{ll}
    \textbf{Operation} & \textbf{Meaning}   \\ \hline
    $\code{x + y}$     & Addition         \\ \hline
    $\code{x - y}$     & Subtraction        \\ \hline
    $\code{x * y}$     & Multiplication     \\ \hline
    $\code{x / y}$     & Division           \\ \hline
    $\code{x**y}$      & Exponentiation     \\ \hline
    $\code{x \% m}$    & Remainder of x / y \\ \hline
  \end{tabular}
  \caption{Basic arithmetic operators in Kite}
\label{tbl:kite-design-arith-op}
\end{table}

\clearpage

An example of two basic boolean operators:
\begin{kite}
1 < 2  ----> True
1 == 2 ----> False
\end{kite}

\begin{table}[H]
  \centering
  \begin{tabular}{ll}
    \textbf{Operation} & \textbf{Meaning} \\ \hline
    $\code{==}$ & Equal                   \\ \hline
    $\code{/=}$ & Not equal               \\ \hline
    $\code{<=}$ & Less than or equal than \\ \hline
    $\code{>=}$ & Greater than or equal   \\ \hline
    $\code{<}$  & Less than               \\ \hline
    $\code{>}$  & Greater than            \\ \hline
  \end{tabular}
  \caption{Basic boolean operators in Kite}
\label{tbl:kite-disign-bool-op}
\end{table}

\subsubsection{Function application}
\label{sec:kite-design-func-app}
The most general form of application is as follows:
\begin{kite}
func (arg1, ..., argN)
\end{kite}

Where the function \code{func}, having an arity of $n$, is applied to \emph{n} arguments.

A concrete example hereof is the application of \code{union}, which has an arity of 2, to two lists:
\begin{kite}
union ([1,2],[2,3]) ----> [1,2,3]
\end{kite}

Besides this form of application, it is also possible to use infix operators. The application of these is described in section~\ref{sec:kite-design-sugar} and the declaration of these is described in the following section.

Furthermore, Kite also allows for partial application, which is described in section~\ref{sec:kite-design-partial-app}.

\subsubsection{Function declaration}
\label{sec:func-decl}
Every function in Kite is declared as an anonymous function:

\begin{kite}
|param1, ..., paramN| -> {
  expr1
  ...
  exprN
}
\end{kite}

A function (lambda expression) is defined as a list of arguments and a block. A block is defined as a list of expressions optionally separated by \code{;}, where it is implicit that the last expression is returned as the result of the functions. A block introduces a new lexical scope, capturing its parent scope.

There are no named functions in Kite, but lambda expressions can be bound to identifers. A function is declared as follows:

\begin{kite}
name :: ArgType -> ... -> ReturnType
name = |arg1, ..., argN| -> { ... }
\end{kite}

Example of a valid function declaration in Kite, taking one parameter of the type \code{Int} and returning an \code{Int}:

\begin{kite}
foo :: Int -> Int
foo = |a| -> {
  a + 1
}
\end{kite}

Another type of function declaration is the declaration of \emph{infix operators}. This is generally done as follows:
\begin{kite}
{name} :: ArgType1 -> ArgType2 -> ReturnType
{name} = |arg1 , arg2| -> { ... }
\end{kite}

And a specific example is our implementation of the conjunction of boolean expressions:
\begin{kite}
{&&} :: Bool -> Bool -> Bool
{&&} = |l, r| -> {
  if l then r else False
}
\end{kite}

Finally, a runnable program must \emph{always} include the \code{main} function of type \code{Void -> a}, which will be used as the entry point for execution:

\begin{kite}
main = |Void| -> { ... }
\end{kite}

In order to use the preprocessor to include another code file, use the \code{\#include} keyword:
\begin{kite}
#include "foo.kite"
\end{kite}

\subsubsection{Type declaration}

Top-level declaration can optionally have a corresponding type declared. This can greatly increase readability of declarations as the inferred type might not be immediately clear.

Annotations are declared using the '\code{::}' operator. For instance the function \code{length} can have its type explicitly annotated as follows:

\begin{kite}
length :: [a] -> Int
length = |xs| -> {
  ...
}
\end{kite}

The type-checker will validate that an annotated type is at least as general as the inferred type. This means that a type annotation of \code{a} will match anything, and \code{a -> a} will match \code{Int -> Int} but not \code{Int -> Bool}

Types can be grouped to denote a higher-order function, for instance the type of map \code{(a -> b) -> [a] -> [b]} means that \code{map} accepts a \emph{function} as its first parameter, a list as its second and it produces a list.

\clearpage

\subsubsection{Syntactic sugar}
\label{sec:kite-design-sugar}
By using a sugaring module, Kite will be able to express certain functions more clearly and readable to the user. Syntactic sugar includes infix operators, multiple parameters on function calls, list of characters as strings and list comprehensions. Infix operators will allow infix use of binary functions (taking two parameters) in Kite. An example of infix operators is the basic $+$ arithmetic operation:

\begin{kite}
foo = 1 + 2
\end{kite}

Where the desugared version of the same expression is given as:

\begin{kite}
foo = ((+) (1)) (2)
\end{kite}

As Kite functions will only take one parameter, sugaring can simulate the use of multiple parameters in functions. Example hereof:

\begin{kite}
foo :: a -> a -> a
foo = |a, b| -> {
  a + b
}
\end{kite}

Which will be syntactic sugar for the following:

\begin{kite}
foo :: a -> a -> a
foo = |a| -> {
  return |b| -> {
    return ((+) (a)) (b)
  }
}
\end{kite}

\textbf{Parameterless functions} are also syntactic sugar. For instance the main function is often declared as follows:

\begin{kite}
main = -> {
  ...
}
\end{kite}

This is simply shorthand for

\begin{kite}
main = |Void| -> {
  ...
}
\end{kite}

As mentioned earlier, \textbf{Strings} are syntactic sugar for list of characters, \code{[Char]}, which will allow for easier representations of character lists. An example of a string:

\begin{kite}
str = "Hello"
\end{kite}

Where the corresponding desugar will be:
\begin{kite}
str = ['H', 'e', 'l', 'l', 'o']
\end{kite}
Note the distinction between strings using double-quotes ("...") and characters using single-quotes ('...').

\label{sec:ex-listcomp}
\textbf{List comprehensions} are syntactic sugar for manipulating lists. An example of a list comprehension is:

\begin{kite}
list = [ x*y | x <- range(3,5), y <- range(4,6) | (x+y) >= 10, y > 4]
\end{kite}

List comprehensions are composed of three elements: An \emph{output expression}, one or more \emph{draws} and optionally one or more \emph{guards}.

First, elements are drawn from the draws, which in the above example are \code{x <- range(3,5)} and \code{y <- range(4,6)}, yielding 9 potential elements to the output expression:

\begin{center}
\code{[[3,4],[3,5],[3,6],[4,4],[4,5],[4,6],[5,4],[5,5],[5,6]]}
\end{center}

These potential combinations are then filtered by the guards. The element \code{[3,3]} will not pass the guards as $3+3$ is not larger than or equal to 10, nor is 3 larger than 4. The element \code{[5,5]} will on the other hand pass the guards.

Finally, all the combinations that pass the guards are subjected to the output expression. The element \code{[5,5]} will thus become \code{25} in the resulting list.

The desugared version of the given example will look as follows:

\begin{kite}
list =
flatMap(|x| -> {
  flatMap (|y| -> {
    if (|x,y| -> {y > 4})(x,y) && (|x,y| -> {(x+y) >= 10})(x,y)
       then [(|x,y| -> {x*y})(x,y)]
       else []
  } , range(4,6) )
}, range(3,5) )
\end{kite}

Note that list comprehensions uses functions defined in section~\ref{foundation}, which will be discussed later.

\textbf{Infix functions} Kite allows for use of infix functions. Infix functions are syntactic sugar for prefix function calls, with some given parameters. Using '\code{\`}' (backtick) will imply the use of infix function calls

A common example of infix function calls, is the \code{elem}-function, which is defined as follows (from \nameref{foundation}):
\begin{kite}
elem :: a -> [a] -> Bool
elem = |el| -> {
  any ((el==))
}
\end{kite}

The following Kite example will be an example of usage of \code{elem}. \code{elem} is here used as an infix function call.
\begin{kite}
list = [1, 2, 3]
1 `elem list ----> True
\end{kite}
The resulting desugared version of same example will look as follows:

\begin{kite}
list = [1,2,3];
((elem) (1)) (list); ----> True
\end{kite}

\subsubsection{Recursion}
Kite supports recursion, which allows functions to call themselves in programs. Basic recursion will have the following syntax:
\begin{kite}
foo :: Int -> Int
foo = |a| -> {
  if a == 1 then 1
  else a + foo(a - 1)
}
\end{kite}

Kite also supports recursion in pattern matching, which will be discussed next.

\subsubsection{Conditionals}
\label{subsubsec:conditionals}
Kite features two conditional expressions; the if-expression and pattern matching.

The \textbf{if-expression} is not to be confused with an if-statement, as our implementation is not a control flow statement. It will always return an expression, and the return-types of the consequent and alternative must be the same.

\begin{kite}
if booleanExpr then consequenceExpr else alternativeExpr
\end{kite}

A more advanced form of conditional is \textbf{pattern matching}. It works by matching an expression to a list of cases, which can be either a normal expression or it can be a destruction\footnote{Destructuring means breaking up a data structure into its parts by using pattern matching. Kite features a simple implementation of this in its \code{match} expressions} of a list or a pair.

Below we show the matching of \code{n} on 0, which is an expression, and \code{otherwise}, which is written \code{"\_"}:

\begin{kite}
fibo :: a -> b -> Int -> a
fibo = |a, b, n| -> {
  match n {
    0 -> a,
    _ -> fibo(b, (a + b), (n - 1))
  }
}
\end{kite}

An example of pattern matching on lists is our implementation of the function \code{reverse} (\ref{foundation}), which matches \code{xs} with an expression, empty list, and the head and tail of \code{xs}:

\begin{kite}
reverse = |xs| -> {
  match xs {
    []    -> [],
    x,xs  -> reverse (xs) ++ [x]
  }
}
\end{kite}

\newpage

The final case is matching on pairs. The example below shows the implementation of \code{fst} (first) in Foundation (\ref{foundation}):
\begin{kite}
fst :: (a, b) -> a
fst = |p| -> {
  match p {
    (a , b) -> a
  }
}
\end{kite}

\subsubsection{Higher-order functions}
Kite features higher-order functions (HoF), and the \code{map} function is an example hereof. It takes a function as argument and applies it to every element of a list, and returns the resulting list:

\begin{kite}
map :: (a -> b) -> [a] -> [b]
map = |f, ls| -> {
  match ls {
    [] -> [],
    x,xs -> f (x) : map (f, xs)
  }
}
\end{kite}

\subsubsection{Currying}
\label{subsubsec:currying}
Beside every function in Kite being curried (which should be clear from the type-declarations), we have also implemented a \code{curry} (and \code{uncurry}) function, which given a function with a single pair as the argument, returns the same function taking two arguments corresponding to the values of the pair:

\begin{kite}
curry :: ((a, b) -> c) -> (a -> b -> c)
curry = |f| -> {
  |a,b| -> {
    f((a,b))
  }
}
\end{kite}

Uncurry, naturally does the opposite, meaning it takes a function taking two parameters and returns the same taking a single pair as an argument.

\begin{kite}
uncurry :: (a -> b -> c) -> ((a, b) -> c)
uncurry = |f| -> {
  |p| -> {
    match p {
      (a, b) -> f(a,b)
    }
  }
}
\end{kite}

We have chosen to include these two functions from~\nameref{foundation} in this section as they reflect an essential part of Kite and functional programming languages in general.

\subsubsection{Partial application}
\label{sec:kite-design-partial-app}
An example of partial application is the function \code{increment}, which partially applies addition to an infix operator:

\begin{kite}
increment :: Int -> Int
increment = (1+)
\end{kite}

Increment can then be applied to an integer:

\begin{kite}
increment(1) ----> 2
\end{kite}

It should be noted that the addition operator, \code{+}, has type \code{a
  -> a -> a}. Thus, partial application yields \code{a -> a} and since
it is partially applied with an integer, the type of \code{increment}
becomes \code{Int -> Int}

Another example is partial application of functions in the more general form, as mentioned in section \ref{sec:kite-design-func-app}. A function taking $n$ parameters can be partially applied to $k$ arguments, where $n > k > 0$

Here the syntax is as follows:
\begin{kite}
func(arg1, ..., argK)
\end{kite}
Which yields a function taking $n - k$ arguments.

\subsubsection{Error handling}
Kite supports termination of execution using the function \code{panic}. Using \code{panic} will stop the execution. It has the type signature \code{String -> a}, thus being allowed anywhere.

An example of the \code{panic} function in use:
\begin{kite}
head :: [a] -> a
head = |xs| -> {
  match xs {
    [] -> panic ("can't take head of empty list"),
    x,xs -> x
  }
}
\end{kite}
If a user tries to take the head of an empty list, the execution of Kite will stop and the error message will be outputted.

\subsection{Foundation}
As mentioned earlier in section~\ref{sec:requirements}, the \code{Foundation} of Kite is a standard library implemented in Kite consisting of often used functions. Kite's Foundation is inspired by Haskell's \code{Prelude}, and includes many functions found in
Prelude.

Below are two examples from Foundation.

\subsubsection{List concatenation}
List concatenation does as the name suggests, namely concatenating two lists into a single list:

\begin{kite}
{++} :: [a] -> [a] -> [a]
{++} = |l, r| -> {
  match l {
    [] -> r,
    x,xs -> x : (xs ++ r)
  }
}
\end{kite}

For example, as strings are simply lists of characters, \code{++} can be used as follows:

\begin{kite}
"Hello," ++ " World!" ----> "Hello, World!"
\end{kite}

\subsubsection{List indexing}
Lists in Kite are zero-indexed, meaning that the first element of a list is at 'index' zero. Indexing a list retrieves the $n$-th element of a list:

\begin{kite}
{#} :: [a] -> Int -> a
{#} = |xs, idx| -> {
  match xs {
    [] -> panic ("list index too large"),
    x,xs -> if idx == 0 then x else xs # (idx - 1)
  }
}
\end{kite}

A simple example of list indexing is as follows:

\begin{kite}
teens = range(13,19)
sweetSixteen = teens # 3 ----> 16
\end{kite}

For a full documentation of Foundation, please see section~\ref{foundation}.


% \documentclass[aps,preprint,twocolumn,groupedaddress]{revtex4}
\documentclass[aps,preprint,groupedaddress]{revtex4-1}
\usepackage[dvips]{graphicx}

\pagenumbering{arabic}

\begin{document}

\title{Automatically tuned libraries for native-dimension tensor transpose algorithms}

\author{Jeff R. Hammond}

% \affiliation{The James Franck Institute, Department of Chemistry and Department of Computer Science, The University of Chicago, Chicago, IL 60637}

\date{Revised June 2008}%\today}

\begin{abstract}

\end{abstract}

\maketitle

\section{Introduction}

Many numerical algorithms, particularly those of quantum many-body theory, rely heavily upon procedures called tensor contractions.  Tensor contractions (TCs) are the multi-dimensional generalization of matrix multiplication (MM).  Whereas in MM, one has only a single internal (contracted) index and the two external indices of the matrix,
\begin{equation}
 C^{i}_{j} = \sum_{k}~A^{i}_{k}~B^{k}_{j}
\end{equation}
a tensor contraction may have an arbitary number of both.  One simple example from quantum chemistry is
\begin{equation}\label{eqn:V2T2a}
 R^{a,b}_{i,j} = \sum_{c,d}~V^{a,b}_{c,d}~T^{c,d}_{i,j}.
\end{equation}
While Eqn.~\ref{eqn:V2T2a} is isomorphic to MM upon fusion of the three pairs of indices, other similar contractions with permuted indices are not.  An example of a TC that cannot be performed with MM alone is
\begin{equation}\label{eqn:V2T2b}
 R^{a,b}_{i,j} = \sum_{k,c}~\tilde{V}^{k,b}_{c,j}~T^{a,c}_{i,k},
\end{equation}
presuming that we are utilizing the straightforward layout of these objects in memory.  That the objects $R$ and $T$ in Eqns.~\ref{eqn:V2T2a} and~\ref{eqn:V2T2b} are the same, the best one can do is to choose a layout which is optimal for the most expensive TC and use a sub-optimal ordering for the other.  Alternatively, one can change the memory layout sub-optimally-ordered tensors via a transpose-like operation.  A much more complex approach is to use Morton-ordering~\cite{Morton} (also known as Z-ordering), or more generally, hierarchical tiling~\cite{HTiling}, to improve the performance of all tensor contractions, but then it is not possible to use existing implementations of MM, such as BLAS.

The complexity introduced by the transposition of indices in TCs presents a significant challenge to programmers.  If one hand-codes procedures which are not MM, then a significant performance loss is incured, as MM kernels are perhaps the most optimized in all numerical computation.  Alternatively, one can retain the use of fast MM kernels by realigning the memory layout such that operations like Eqn.~\ref{eqn:V2T2b} can be performed with MM.

The transformation of Eqn.~\ref{eqn:V2T2b} to a form which is consistent with matrix multiplication is as follows:
\begin{eqnarray}\label{eqn:transform_V2T2b}
 T^{a,c}_{i,k} &\rightarrow& T1(i,k,a,c) \\
 \tilde{V}^{k,b}_{c,j} &\rightarrow& V1(c,j,k,b) \\
 R^{a,b}_{i,j} &\rightarrow& R1(i,j,a,b) \\
 T2(i,a,k,c) &=& T1(i,k,a,c) \\
 V2((k,b),(c,j)) &=& V1((c,j),(k,b)) \\
 V3(k,c,b,j) &=& V2(k,b,c,j) \\
 R2((i,a),(b,j)) &=& {\rm SUM}[(k,c)]~V3((k,c),(b,j))*T2((i,a),(k,c)) \\
 R1(i,j,a,b) &=& R2(i,a,b,j)
\end{eqnarray}
where the matrix dimensions used in the matrix transpose and multiplication calls are denoted with parentheses.  Row-major ordering (the last index is stride-1) is presumed throughout.

For a rank-$n$ tensor, there are $n!$ possible permutations of the indices, and writing fast code for each of these procedures by hand is impractical for $n>4$.  The automatic generation of code for these procedures is the subject of this chapter.

\section{Background}

The target application for this project was the coupled-cluster codes within TCE module of the quantum chemistry package NWChem~\cite{NWChem}.  Because most of the code within the TCE module was written by a code generator, it employs a simple structure which is easily modified.  The TCE module also has few, if any, manually optimized procedures and thus suffers in performance with respect to the best hand-written packages.  In the particular case of tensor (array) permutation, four subroutine calls, \texttt{tce\_sortN} (\texttt{N=2,4,6,8}), were used to perform every permutation.  Nearly identical sort-acculuate calls (\texttt{tce\_sortaccN}, \texttt{N=2,4,6,8}) have use the same code except with ``\texttt{+=}'' instead of ``\texttt{=}''.  Replacing these procedure with faster ones would result in increased performance throughout the code.

Is it not unreasonable to question the utility of optimizing permutations at all.  The permutation of a $n$-d array is an $N^n$ flop procedure, where $N$ is the rank of each dimension, whereas the the contraction of two $n$-d arrays over $k$ indices is an $N^{n+k}$ flop procedure.  However, the mop cost of both procedures is $C \cdot N^n)$ where C is 2 for permutation (1 read, 1 write) and 3 for contraction (2 reads, 1 write).  On modern processors, mops are so expensive that some have said that flops can almost be ignored.  MM achieves a large percentage of machine peak by obscuring memory latency through data reuse, which is possible because flops/mops is large.  Since permutations are entirely memory bound, it is unreasonable to expect a large percentage of peak performance but at the same time, improper implementation of these procedures can be extraordinarily expensive.  Unlike MM, the flow of data during permutation is necessarily not optimal since at least half of the mops will not be stride-1.

If we assume flops are free and that performance is determined by the number and type of mops occuring, then permutation, not MM will be the more expensive procedure of the two.  Since theoretical analyses are rarely quantitative, the relative cost of the two procedures has been measured using profiling techniques.  Both the GNU profiler gprof~\cite{GPROF} and TAU~\cite{TAU} were used to profile the code to ensure correct measurements.

\section{Results}

All results are for a single water molecular at the equilibrium geometry.  Calculations were performed without point-group symmetry using spherical angular functions.  The tile size for the virtual orbitals (VO) was no greater than 32.  For the cc-pVDZ, cc-pVTZ and cc-pVQZ basis sets, there were 2, 4 and 8 VO tiles with average dimension 19, 26.5 and 27.5, respectively.  Using a larger tile size favors \texttt{dgemm} performance, while smaller favors \texttt{tce\_sortN}.

\subsection{Profiling of CCSD within NWChem}

First it was established empirically that the tensor transpose operation is a significant portion of the was time, as predicted by the aforementioned theoretical analysis.  In Table~\ref{tab:profile1}, the results of profiling are given for computing the CCSD ground-state energy.  When computing the CCSD energy, the coupled-cluster equations (described in previous chapters) are solved iteratively.  The energy evaluation two orders less expensive than the iterative procedure and does not contribute significantly to the computational cost.  The data given in Table~\ref{tab:profile1} shows that MM and transpose both contribute significantly to wall time.  However, the fraction of the wall time devoted to MM grows with the basis set, so it is not entirely clear that the optimality of the transpose will matter for larger systems.  It should be noted that the MM implementation used was from NETLIB.  A high-performance BLAS library such as GotoBLAS~\cite{GotoBLAS} or ATLAS~\cite{ATLAS} would greatly decrease the time spent on MM operations.

\begin{table}[!hp]
\caption{Profile (\texttt{gprof}) of the NWChem TCE module CCSD code for computing the ground-state energy.} \label{tab:profile1}
% \begin{ruledtabular}
% \begin{center}
\begin{tabular}{ccccc}
\hline\hline
& \multicolumn{2}{c}{Matrix multiplication} & \multicolumn{2}{c}{Tensor transpose} \\
 & \multicolumn{2}{c}{dgemm} & \multicolumn{2}{c}{tce\_sort4 \& tce\_sortacc4} \\
   \cline{2-3} \cline{4-5}
Basis & Time (s) & \% of Total & Time (s) & \% of Total \\
\hline
cc-pVDZ &   0.40 & 27.59 &   1.49 & 29.65 \\
cc-pVTZ &   8.70 & 30.87 &  34.15 & 37.22 \\
cc-pVQZ & 154.46 & 38.54 & 108.47 & 27.07 \\
\hline\hline
\end{tabular}
% \end{center}
% \end{ruledtabular}
\end{table}

As should be clear from previous chapters, computating the ground-state energy is but one of many possible tasks for a coupled-cluster code.  In Table~\ref{tab:profile2}, profiling information is given for the evaluation of all steps necessary to compute the hyperpolarizability using the method described in Chapter 8.  The number of difficult transposes required for the solution of the $\Lambda^{(0)}$, $T^{(1)}$ and $\Lambda^{(1)}$ equations is significantly larger than required just for $T^{(0)}$, which is affirmed by the data.  The relative amount of time spend in the transpose operations is approximately 50\% greater than that spent in MM for the cc-pVQZ basis set, and while the overall trend in the basis set is the same as Table~\ref{tab:profile1}, rate of which MM increases and transpose decrease is much less.

\begin{table}[!hp]
\caption{Profile (\texttt{gprof}) of the NWChem TCE module CCSD code for computing the hyperpolarizability.} \label{tab:profile2}
% \begin{ruledtabular}
% \begin{center}
\begin{tabular}{ccccc}
\hline\hline
 & \multicolumn{2}{c}{dgemm} & \multicolumn{2}{c}{tce\_sort4 \& tce\_sortacc4} \\
   \cline{2-3} \cline{4-5}
Basis & Time (s) & \% of Total & Time (s) & \% of Total \\
\hline
cc-pVDZ &    5.78 & 28.18 &   11.08 & 54.03 \\
cc-pVTZ &  111.10 & 28.44 &  192.45 & 49.26 \\
cc-pVQZ & 1389.05 & 29.16 & 2137.17 & 44.87 \\
\hline\hline
\end{tabular}
% \end{center}
% \end{ruledtabular}
\end{table}

\subsection{Autotuning transpose kernels}

It was determined that the primary reason transpose operations are slow is that they access memory in a less-than-optimal way, that is, strided access rather than sequential (stride-1) access.  While it is not possible to eliminate strided access, it is possible to minimize the cost of strided access by rearranging the loops such that the stride distance is minimal.  If the stride distance is small enough that cache reuse occurs, a significant performance increase will result.

While it is possible to determine optimal loop ordering using mathematical analysis, a much cruder approach --- exhaustive sampling --- is sufficient in this case.  In addition, sampling includes all possible hardware-specific factors which may not be available for integration into a performance modeling used in the analytic approach.  To determine the optimal loop-ordering for the 4-d transpose problem, a code-generator was developed which would produce source code for all possible implementations (24) for each of the 24 transposes, for a total of 576 cases.  Source code was generated in both Fortran 77 and ANSI C since the former is known to be more amenable to compiler optimization, while the latter allows a more complete set of compiler pragmas and is the language of choice of people who would further hand-tune these kernels.  A master program was instrumented to compile the source code into binary form using a variety of possible compiler flags to determine the effect of available optimization options.  Some of the optimizations sampled for the Intel compilers were loop-unrolling, auto-vectorization and auto-parallelization; compiler pragmas were also explored as a means to explicitly control unrolling and vectorization.  The master program built a self-contained binary for each possible transpose which, when executed, performed the timing and printed a complete table of results then identified the optimal loop-ordering.  It also prints the compiler flags which were used to generate the code to prevent data rot.

Table~\ref{tab:spaghetty1} shows the best improvement obtained with the automatically-generated code as compared to the original implementation within NWChem.  Four cases were considered: regular 4-d arrays of rank 20, 32 and 60 plus an irregular array.  The speed-up for the rank 20 case is significantly better than the others because both the input and output array (1,250 KB each) fit into cache on the machine tested (Intel Core2Duo, 4 MB L3 cache).  For larger dimensions, the arrays do not fit into cache.  This clearly indicates that L3 cache-blocking will significantly improve the performance transpose, although finding the optimal code with that additional level of complexity becomes harder.  Instead of performing an exhaustive search over just the space of loop-orderings or compiler options, an exhaustive search for the cache-blocking case involves exploring the tensor product space of blocking sizes and loop-orderings for each level of blocking.  The dimensionality here is too large to consider by brute force, and a space-pruning algorithm must be employed to make the solution achievable in a reasonable amount of time.

\begin{table}[!hp]
\caption{Best improvement relative to the original implementation by the automatically-generated ANSI C implementation of the transpose operations for a 4-d array.  The Intel 10.1 compiler flags used were \texttt{-O3 -xT -march=core2 -mtune=core2 -funroll-loops -align}.} \label{tab:spaghetty1}
% \begin{ruledtabular}
% \begin{center}
\begin{tabular}{ccccc}
\hline\hline
Transpose & $20^4$ & $32^4$ & $60^4$ & irregular \\
\hline
1234 & 7.250 & 2.500 & 1.946 & 3.769 \\
1243 & 7.667 & 2.345 & 2.257 & 2.733 \\
1324 & 5.000 & 1.828 & 1.861 & 2.667 \\
1342 & 5.250 & 2.379 & 2.173 & 2.923 \\
1423 & 7.000 & 2.448 & 2.272 & 2.929 \\
1432 & 5.250 & 2.000 & 2.372 & 3.154 \\
2134 & 5.250 & 2.000 & 1.967 & 2.583 \\
2143 & 8.334 & 2.586 & 2.108 & 2.786 \\
2314 & 5.250 & 2.000 & 2.028 & 2.583 \\
2341 & 5.000 & 2.267 & 2.179 & 3.000 \\
2413 & 7.000 & 2.571 & 2.390 & 2.857 \\
2431 & 5.000 & 1.889 & 2.756 & 3.385 \\
3124 & 5.250 & 1.862 & 1.966 & 2.538 \\
3142 & 5.000 & 3.233 & 2.216 & 3.000 \\
3214 & 5.250 & 2.143 & 2.104 & 2.833 \\
3241 & 7.000 & 1.971 & 2.219 & 2.571 \\
3412 & 5.000 & 1.838 & 2.208 & 2.692 \\
3421 & 6.334 & 1.976 & 2.281 & 2.429 \\
4123 & 6.333 & 2.655 & 2.228 & 2.875 \\
4132 & 4.750 & 1.944 & 2.202 & 3.000 \\
4213 & 5.250 & 2.821 & 2.326 & 2.786 \\
4231 & 5.250 & 2.195 & 2.299 & 3.077 \\
4312 & 4.750 & 1.973 & 2.120 & 3.308 \\
4321 & 4.500 & 1.767 & 2.163 & 3.077 \\
\hline\hline
\end{tabular}\\
irregular $= 41\times17\times24\times39$
% \end{center}
% \end{ruledtabular}
\end{table}

In addition to the 4-d case, exhaustive search was used to find the best implementation of the subset of 6-d transpose-accumulate operations used in CCSD(T).  Because of memory constraints imposed by the use triple-excitation amplitudes, dimensions of the arrays are much smaller.  Due to the smaller stride length, cache-blocking is less important and the performance improvement realized just by finding the optimal loop-ordering (among $6!=720$ possibilities) is quite good.  The performance improvement realized for the first-generation auto-tuning approach was between 3 and 4 times the original implementation, although preliminary results for a subsequent attempt were closer to 12 times.  This is the subject of ongoing research.

\section{Conclusions}

Tensor operations, which compose the overwhelming majority of quantum chemistry codes, require optimal implementations to take advantage of high-performance computers.  It was demonstrated that tensors transpose is a significant contribution to the wall time for coupled-cluster calculations and that a very simple approach decrease the time devoted to these operations by a factor of two.  The successful approach employed here did not employ cache-blocking or many other possible optimization techniques which will further improve the performance

Ultimately, this project demonstrates that the artificial seperation of transpose and MM in the implementation of tensor contractions is wholly inappropriate.  The original motivation for it was to take advantage of vendor-optimized BLAS libraries, but developments in autotuning over the past 10 years clearly indicate that it is possible to generate tensor contraction kernels directly.  The advantage is not only with respect to performance, but also in terms of mathematical elegance.  The many-body formalism of coupled-cluster theory is multidimensional and flattening the data structures used in such codes into matrices just to use BLAS should not be tolerated.

\begin{thebibliography}{99}

\bibitem{Morton}
G. M. Morton, ``A Computer Oriented Geodetic Data Base and a New Technique in File Sequencing,'' IBM technical report (1966).

\bibitem{HTiling}
% Hierarchical Tiling: A Methodology for High Performance
L. Carter, J. Ferrante, S. Flynn Hummel, B. Alpern, and K.S. Gatlin, UCSD Tech Report CS96-508, November 1996;
% Hierarchical Tiling for Improved Superscalar Performance
L. Carter, J. Ferrante and S. Flynn Hummel, \textit{Int. Par. Proc. Symp.}, April 1995;
% Efficient Multiprocessor Parallelism via Hierarchical Tiling
L. Carter, J. Ferrante and S. Flynn Hummel, \textit{SIAM Conf. on Par. Proc. for Sci. Comp.}, February 1995.

\bibitem{NWChem}
E. J. Bylaska, W. A. de Jong, N. Govind, K. Kowalski, T. P. Straatsma, M. Valiev, D. Wang, E. Apra, T. L. Windus, J. Hammond, P. Nichols, S. Hirata, M. T. Hackler, Y. Zhao, P.-D. Fan, R. J. Harrison, M. Dupuis, D. M. A. Smith, J. Nieplocha, V. Tipparaju, M. Krishnan, Q. Wu, T. Van Voorhis, A. A. Auer, M. Nooijen, E. Brown, G. Cisneros, G. I. Fann, H. Fruchtl, J. Garza, K. Hirao, R. Kendall, J. A. Nichols, K. Tsemekhman, K. Wolinski, J. Anchell, D. Bernholdt, P. Borowski, T. Clark, D. Clerc, H. Dachsel, M. Deegan, K. Dyall, D. Elwood, E. Glendening, M. Gutowski, A. Hess, J. Jaffe, B. Johnson, J. Ju, R. Kobayashi, R. Kutteh, Z. Lin, R. Littlefield, X. Long, B. Meng, T. Nakajima, S. Niu, L. Pollack, M. Rosing, G. Sandrone, M. Stave, H. Taylor, G. Thomas, J. van Lenthe, A. Wong, and Z. Zhang. ``NWChem, A Computational Chemistry Package for Parallel Computers, Version 5.1'' (2007), Pacific Northwest National Laboratory, Richland, Washington 99352-0999, USA.  A modified version.

\bibitem{GPROF}
gprof.  Jay Fenlason and Richard Stallman.  Copyright 1988, 92, 97, 98, 99, 2000, 2003 Free Software Foundation, Inc.

\bibitem{TAU}
TAU: The TAU Parallel Performance System.  S. Shende and A. D. Malony. \textit{International Journal of High Performance Computing Applications}, Volume 20 Number 2 Summer 2006. Pages 287-331.

\bibitem{GotoBLAS}
\texttt{http://www.tacc.utexas.edu/resources/software/}

\bibitem{ATLAS}
\texttt{http://math-atlas.sourceforge.net/}

% \bibitem{QingdaTranspose}
% Q. Lu, S. Krishnamoorthy and P. Sadayappan, ``Combining Analytical and Empirical Approaches in Tuning Matrix Transposition'', the \textit{15th international conference on Parallel architectures and compilation techniques (PACT)}, 2006.

\end{thebibliography}

\end{document}


\chapter{Algorithm Selection Model Training Algorithms}

\section{Introduction}
This chapter explains the implementation details of the algorithms used in training the AS model, which are \textit{pairwise random forest regression} (PRFR) and REINFORCE.

\section{Data Preparation}
\label{sec:dataprep}

Table \ref{tbl:graphs2015} displays the summary on GRAPHS-2015 dataset.

\begin{table}[H]
	\centering
	\begin{tabular}{|l|l|}
		\hline
		\textbf{Scenario ID} & GRAPHS-2015 \\ \hline
		\textbf{Source} & \citet{graphs2015} \\ \hline
		\textbf{Number of problem instances} & 5725 \\ \hline
		\textbf{Number of features} & 35 \\ \hline
		\textbf{Number of algorithms} & 7 \\ \hline
		\multirow{7}{*}{\textbf{Algorithm list}} & LAD \\ \cline{2-2}
		& SupplementalLAD \\ \cline{2-2}
		& VF2 \\ \cline{2-2}
		& GLASGOW1 \\ \cline{2-2}
		& GLASGOW2 \\ \cline{2-2}
		& GLASGOW3 \\ \cline{2-2}
		& GLASGOW4 \\ \hline
		\textbf{Performance measure} & Runtime (in milliseconds) \\ \hline
	\end{tabular}
	\caption{GRAPHS-2015 dataset summary}
	\label{tbl:graphs2015}
\end{table}

Only a subset of GRAPHS-2015 was used for training AS models. A technique called \textit{presolving} is used to eliminate the quickly solvable problem instances, leaving the harder problem instances for AS. The steps on removing presolved instances from GRAPHS-2015 were discussed in PORTSUB study. The flow and code is shown in Figures \ref{fig:presolver} and \ref{fig:extracthard}.

\begin{figure}[H]
	\centering
	\scalebox{.85}{\includegraphics{./img/presolver.png}}
	\caption[Filtering presolved instances from GRAPHS-2015]{Filtering presolved instances from GRAPHS-2015. 2336 hard problems are left for AS.}
	\label{fig:presolver}
\end{figure}

\begin{figure}[H]
	\centering
	\scalebox{.60}{\includegraphics{./img/extract_hard_instances.png}}
	\caption{R script for extracting hard instances from GRAPHS-2015}
	\label{fig:extracthard}
\end{figure}

\section{Pairwise Random Forest Regression}
The AS model used in PORTSUB study was trained using a supervised learning approach called PRFR. It works as follows: for each pair of algorithms in the algorithm portfolio, a model is trained to predict the performance difference between them. If the first algorithm is better than the second, the difference is positive, otherwise negative. The algorithm with the highest cumulative performance difference, i.e. the most positive difference over all other algorithms, is chosen to be run.

The LLAMA R package contains an implementation of pairwise random forest regression written by Lars Kotthoff, one of the PORTSUB authors. Its usage in R is shown on Figure \ref{fig:prfr}

\begin{figure}[H]
	\centering
	\scalebox{.65}{\includegraphics{./img/prfr.png}}
	\caption{R script for building AS model trained using PRFR}
	\label{fig:prfr}
\end{figure}

\section{REINFORCE}

The adaptation of REINFORCE algorithm to AS is composed of three main components:

\begin{enumerate}
	\item \textbf{Function approximator} \\
	An AS model can learn how algorithms match to problems with certain features. Patterns in problem features can be captured by function approximation methods, which translate feature values into parameters. For example, a simple function approximator such as linear regressor parameterizes inputs into slope and offset coefficients. Neural networks, a universal function approximator, represent parameters as network weights. Function approximation assists in generalizing AS model learning through discovery of feature patterns across problem instances.

	\item \textbf{Policy function} \\
	A policy function takes function approximator parameters as input and computes a probability for each algorithm. Algorithm probabilities change with respect to problem features. The computed probabilities serve as the basis of selection among algorithms.

	\item \textbf{Reward function} \\
	The reward value is equivalent to the observed performance of the selected algorithm. It serves as feedback to the AS model to improve its capability in predicting optimal algorithms. Optionally, the reward value can be normalized or transformed to control how much the policy behavior should change with respect to feedback.
\end{enumerate}

Figure \ref{fig:reinforce_flowchart} illustrates how an AS model is trained using \textit{REINFORCE} algorithm.

\begin{figure}[H]
	\centering
	\scalebox{.65}{\includegraphics{./img/REINFORCE_flowchart.png}}
	\caption{Implementation of AS model trained using REINFORCE}
	\label{fig:reinforce_flowchart}
\end{figure}

\begin{enumerate}
	\item The GRAPHS-2015 dataset is loaded into R workspace. Hard problem instances are extracted, which consists of 2336 problem instances with 42 attributes used for training: 35 problem features plus runtime data of 7 algorithms.

	\begin{figure}[H]
		\centering
		\scalebox{.75}{\includegraphics{./img/problem_instance.png}}
		\caption{Sample problem instance from GRAPHS-2015 dataset}
		\label{fig:problem_instance}
	\end{figure}

	\item Problem features are normalized using \textit{Z-score}. Normalization standardizes the range of data inputs which helps in resolving numerical issues when gradients are calculated during backpropagation. In effect, this speeds up neural network learning.

	\begin{equation}
	z_{ij} = \frac{x_{ij} - \mu_j}{\sigma_j}
	\end{equation}

	where:

	\begin{table}[H]
		\centering
		\begin{tabular}{rl}
			$z_{ij} =$ & normalized feature value\\
			$x_{ij} =$ & feature value \\
			$\mu_{j} =$ & mean of \textit{jth} feature across \textbf{P} problem instances \\
			$\sigma_j =$ & standard deviation of \textit{jth} feature across \textbf{P} problem instances \\
			$\mathit{i} =$ & \textit{ith} instance from \textbf{P} problem instances \\
			$\mathit{j} =$ & \textit{jth} feature out of 35 features \\
		\end{tabular}
	\end{table}

	The algorithm runtime data is rescaled to simplify the reward value used for computing the performance gradient. This enables the AS model to learn the most important information during problem solving, which is the ranking of algorithm performance from best to worst. This information is more important than the knowledge of actual algorithm runtimes, as the objective of AS is mainly not to characterize individual algorithm performance, but to determine which algorithms work best on problems. Table \ref{tbl:rewardscaling} shows how this study mapped algorithm ranking (determined by sorting runtime from shortest to longest) to reward values.

	\begin{table}[H]
		\centering
		\begin{tabular}{|c|c|}
			\hline
			\multicolumn{1}{|l|}{\textbf{Algorithm rank}} & \multicolumn{1}{l|}{\textbf{Assigned Reward Value}} \\ \hline
			1 & 16 \\ \hline
			2 & 4 \\ \hline
			3 & 1 \\ \hline
			4 & -4 \\ \hline
			5 & -16 \\ \hline
			6 & -64 \\ \hline
			7 & -256 \\ \hline
		\end{tabular}
		\caption{Algorithm ranking reward values}
		\label{tbl:rewardscaling}
	\end{table}

	\item A neural network is trained to map problem features (input) to algorithm scores (output). It is composed of an input layer with 35 inputs, 3 hidden layers, and an output layer with 7 outputs. Algorithm scores represent selection preferences, i.e., the larger the score, the more often that algorithm is taken.

	\item The algorithm scores are converted into probabilities using the softmax policy function. The probability assigned to an algorithm is proportional to its score, i.e., algorithms with the highest scores are given the highest probabilities of being selected. The softmax equation is given in the Equation below:

	\begin{equation}
	\pi(A | S_i, \theta) = \frac{\exp{H_A(S_i, \theta)}}{\sum_{b=1}^{7}\exp{H_b(S_i, \theta)}}
	\end{equation}

	where:

	\begin{table}[H]
		\centering
		\begin{tabular}{rl}
			$\pi(A | S_i, \theta) =$ & Probability of algorithm A \\
			$H_A(S_i, \theta) =$ & Algorithm score of algorithm A \\
			$H_B(S_i, \theta) =$ & Algorithm score of algorithm B \\
			$S_i =$ & features of problem instance \textit{i} \\
			$\theta =$ & neural network weights
		\end{tabular}
	\end{table}

	\item An algorithm is randomly selected based from the probabilities generated from the softmax policy function.

	\item The rescaled reward value corresponding to the selected algorithm is read from the dataset. This value, along with the probability of the selected algorithm, is used to calculate the performance gradient:

	\begin{equation}
	\nabla J(\theta_t) = G_t\ln{\pi(A | S_i, \theta_t)}
	\end{equation}

	where:

	\begin{table}[H]
		\centering
		\begin{tabular}{rl}
			$\nabla J(\theta_t) =$ & performance gradient \\
			$G_t =$ & reward \\
			$\pi(A | S_i, \theta_t) =$ & Probability of selected algorithm
		\end{tabular}
	\end{table}

	\item The performance gradient is used for backpropagation, a standard method of updating the weights of a neural network. Through this process, the neural network learns to adjust the algorithm scores in proportion to the reward value associated with the selected algorithm.

	\begin{equation}
	\theta_{t+1} \gets \theta_t + \alpha \nabla J(\theta_t)
	\end{equation}

	where:

	\begin{table}[H]
		\centering
		\begin{tabular}{rl}
			$\theta_{t+1} =$ & Updated neural network weights \\
			$\theta_t =$ & current neural network weights \\
			$\alpha =$ & learning rate \\
			$\nabla J(\theta_t) =$ & performance gradient
		\end{tabular}
	\end{table}
\end{enumerate}

\section{Model Evaluation}
The performance of PRFR and REINFORCE are evaluated using 10-fold cross validation. The 2336 hard instances from GRAPHS-2015 are randomly partitioned into 10 subsets of approximately equal size. Of the 10 subsets, 9 are combined to form the training set for the AS models, while the remaining subset is formed as a test set. Mean MCP (Equation \ref{eq:mcp}) is calculated on the test set to evaluate the AS model performance. This is repeated 10 times for all possible combinations of training and test sets. At the end of this process, the final mean MCP is calculated by averaging the mean MCPs from all test sets.

\section{R source code implementation}
The implementation of AS models in this study can be retrieved from here: \url{https://github.com/kvrigor/algosel-rl}


\section{Parameters}
\subsection{Non-age-stratified parameters}

\begin{longtable}[ht]{| >{\raggedright}p{4cm} | >{\raggedright}p{3cm} | p{6.8cm} |}
    \hline
    Parameter & Value & Rationale \\
    \endfirsthead
	\multicolumn{3}{c}{continuation of parameters table}\\
    \endhead
    \hline Incubation period & Calibration parameter, truncated normal distribution, mean 5.5 days & Estimates of the incubation period have included 5.1 days, 5.2 days and 4.8 days \cite{RN7, RN12, RN8, RN10}. A systematic review \cite{RN13} found that data are best fitted by a log-normal distribution (mean 5.8 days, CI 5.0 to 6.7, median 5.1 days). Our systematic review \cite{RN17} found that estimates of the mean incubation period have varied from 3.6 to 7.4 days. \\
    \hline
    Proportion of incubation period infectious & 50\% &     Infectiousness is considered to be present throughout a considerable proportion of the incubation period, based on analyses of confirmed source-secondary pairs \cite{RN23} and early findings that the incubation period was similar to the serial interval \cite{RN7}. The study of source-secondary pairs was also the primary reference cited by a review of the infectious period that identified studies that quantified the pre-symptomatic period, which concluded that the median pre-symptomatic period could range from less than one to four days \cite{RN14}. \\
    \hline
    Active period (regardless of detection/isolation, for clinical strata 1 to 3) &
    Calibration parameter, truncated normal distribution, mean 8 days &
    This quantity is difficult to estimate, given that identified cases are typically quarantined. Studies in settings of high case ascertainment and an effective public health response have suggested a duration of greater than 5.5 days \cite{RN10}. PCR positivity, which may continue for up to two to three weeks from the point of symptom onset \cite{RN23} \cite{RN14}, is difficult to interpret and does not necessarily indicate infectiousness. Consistent with these findings, the duration infectious for asymptomatic persons has been estimated at 6.5 to 9.5 days \cite{RN14} (although in our model, this would include the pre-symptomatic infectious period). \\
    \hline
    Proportion of infectious period before isolation or hospitalisation can occur &
    0.333 &
    Assumed \\
    \hline
    Disease duration prior to admission for hospitalised patients not critically unwell (i.e. early active sojourn time, stratum 4) &
    7.7 days &
    Mean value from ISARIC cohort, as reported on 4\textsuperscript{th} October 2020 in Table 6 \cite{RN22}, and similar to the expected mean from earlier reports from ISARIC \cite{RN16}. This cohort represents high-income countries better than low and middle-income countries, with the United Kingdom contributing data on the greatest number of patients, followed by France. Earlier estimates of this quantity from China included 4.4 days \cite{RN7}. \\
    \hline
    Duration of hospitalisation if not critically unwell (late active sojourn time, stratum 4) &
    12.8 days &
    Mean value from the ISARIC cohort, as reported on 4\textsuperscript{th} October 2020 in Table 6 \cite{RN22}. \\
    \hline
    ICU duration (late active sojourn time, stratum 5) & 10.5 days &
    Mean duration of stay in ICU/HDU from ISARIC cohort for patients with complete data, as reported on 10\textsuperscript{th} October 2020 Table 6 \cite{RN22}. Many other studies reporting on the average duration of ICU stay suffer from right-truncation issues, often estimating 7-10 days length of stay. \\
    \hline
    Duration of time prior to ICU for patients admitted to ICU &
    10.5 days &
    Calculated as the sum of the time from symptom onset to hospital admission (7.7 days above) plus the duration from hospital admission to ICU admission reported by October ISARIC report (2.8 days) \cite{RN22}. \\
    \hline
    Relative infectiousness of asymptomatic persons (per unit time with active disease) & 0.5 & Assumed \\
    \hline
    Relative infectiousness of persons admitted to hospital or ICU & 0.2 & Assumed \\
    \hline
    Relative infectiousness of identified persons in isolation & 0.2 & Assumed \\
    \hline
    Proportion of hospitalised patients ever admitted to ICU & 0.17 & Assumed \\
    \hline
	\caption{\textbf{Universal (non-age-stratified) model parameters.} Point estimates are used as model parameters except where ranges are indicated in calibration parameter table below in calibration table.}
	\title{Universal model parameters.}
	\label{tab:params}
\end{longtable}


\PassOptionsToPackage{dvipsnames}{xcolor}
\documentclass[12pt]{article}
%\documentclass[10pt,draft]{article}
\usepackage[utf8]{inputenc} % utf8 support

% === Document Layout ===
\usepackage[hmargin=0.75in,vmargin=0.75in]{geometry}
\usepackage{indentfirst}
\usepackage{lipsum}
\usepackage{url}
\usepackage[
    colorlinks,
    linkcolor=red!50!black,
    citecolor=blue!50!black,
    urlcolor=blue!50!black
]{hyperref}
\usepackage{amsmath}
\usepackage{pdfpages}
\usepackage{physics}
\usepackage{mathtools}
\usepackage{graphicx}
\usepackage{xcolor}
\usepackage{mathtools} % for the coloneq command
\usepackage{xparse}
\usepackage{ifthen} % For checking optional parameters
\usepackage{enumerate}
\usepackage{amsthm} % The AMS theorems package
\usepackage{amsthm}
\usepackage{caption}
\captionsetup{width=0.8\textwidth}
\usepackage{subcaption}
\usepackage{float}
\usepackage{tabularx}
\usepackage{authblk} % For affiliations
\usepackage{amsfonts}
\usepackage{indentfirst}
\usepackage{amssymb} % for ordering symbols
\usepackage{colonequals} % for the \ratio command

% amsthm
\newtheorem{theorem}{Theorem}[section]
\newtheorem{lemma}[theorem]{Lemma}
\newtheorem{proposition}[theorem]{Proposition}
\newtheorem{corollary}[theorem]{Corollary}
\newtheorem{conjecture}[theorem]{Conjecture}
\theoremstyle{definition}
\newtheorem{definition}{Definition}[section]
\theoremstyle{remark}
\newtheorem{remark}{Remark}

% === Bibliography ===
\usepackage[
    backend=biber,
    style=authoryear,
    citestyle=authoryear,
]{biblatex}
\addbibresource{references.bib}

\newcommand{\speccite}[1]{\citetitle{#1} \parencite{#1}}

\newcommand{\dt}[1]{\frac{\mathrm d #1}{\mathrm d t}}
\usepackage{pgfplots}
\usepackage{tikz}
\usepackage{tikz-3dplot}
\usetikzlibrary{calc}
\tikzset{every picture/.style={baseline={(current bounding box.center)}}}

\author[1,2]{Thomas C. Fraser}
\affil[1]{Perimeter Institute for Theoretical Physics, Waterloo, Ontario, Canada, N2L 2Y5}
\affil[2]{Dept. of Physics and Astronomy, University of Waterloo, Waterloo, Ontario, Canada, N2L 3G1}
\date{\today}

\title{On the Causality of Dynamical Systems}

\begin{document}
\maketitle

\section*{Comments}

These notes can be found at \url{https://github.com/tcfraser/pearl_and_dynamic_causality}.

\section{Overview}

\begin{enumerate}
    \item Origins in early chaos theory
    \begin{itemize}
        \item Lorenz's system \parencite{lorenz1963deterministic}
        \item Attractors
        \item Reconstructing attractors \parencite{packard1980geometry}
        \item Takens' Theorem \parencite{takens1981detecting}
    \end{itemize}
    \item Techniques for Dynamical causal discovery
    \begin{itemize}
        \item Convergent Cross-mapping (CCM) \parencite{sugihara2012detecting}
        \item Topological Causality (TC) \parencite{harnack2017topological}
        \item Dimensional Causality (DC) \parencite{benkHo2018exact}
    \end{itemize}
\end{enumerate}

The \textbf{dynamical systems} which will be considered here involved a finite number $n$ of variables $\{X_1, X_2, \ldots, X_n\}$ evolving according to a system of deterministic, first-order, time-independent, oridinary differential equations (discrete or continuous in time).
\[ \dt{X_j} = f_{j} (X_1(t), X_2(t), \ldots X_n(t)) \]

The solution to a dynamical system on a manifold $M$ (for example $\mathbb R^n$) is specified by a \textbf{flow}
\begin{equation}
    \Phi_{(\cdot)}(\cdot) : \mathbb R \times M \to M,
\end{equation}
where $\Phi_t$ is a diffeomorphism of $M$ for all $t \in \mathbb R$ such that
\begin{equation}
    \forall V \in M : \Phi_{0} ( V ) = V, \quad \text{and} \quad \forall s,t \in \mathbb R: \Phi_t \circ \Phi_s = \Phi_{s+t}.
\end{equation}

\section{A Chaotic Origin Story}

\subsection{Lorenz's Discovery}
In \speccite{lorenz1963deterministic}, \citeauthor{lorenz1963deterministic} (this is Edward Norton Lorenz, not to be confused with the physicist Ludvig Lorenz, or the actor Edward Norton) presented a simplified dynamical model for atmospheric convection, which today is known as the Lorenz system:
\begin{align}
    \begin{split}
        \dt{X} &= \sigma ( Y - X ), \\
        \dt{Y} &= X ( \rho - Z ) - Y, \\
        \dt{Z} &= XY - \beta Z.
    \end{split}
\end{align}
The Lorenz system was remarkable because despite its determinism and simplicity, it exhibits flows which are non-periodic, chaotic and yet bounded as in Figure~\ref{lorenz_system}. Moreover, Lorenz identifies the non-linearity of the equations as being essential for these features postulates their ubiquity.
\begin{figure}[H]
    \centering
    \includegraphics[scale=0.35]{figures/lorenz_system.png}
    \caption{The strange attractor of the Lorenz system for $\sigma = 10, \beta = 8/3, \rho = 28$ with $X(0) = 1.1, Y(0) = -1.9, Z(0) = 13.1$. Figure by \href{https://www.researchgate.net/figure/A-trajectory-of-the-Lorenz-system-in-phase-space-revealing-the-strange-attractor-The_fig1_262074522}{P. T. Clemson}.}
    \label{lorenz_system}
\end{figure}

Later in \speccite{ruelle1971nature}, \citeauthor{ruelle1971nature} develop a variety of mathematics to study such quasi-periodic behaviors and refers to them as \textit{strange attractors}. Importantly, they conjectured that universally, strange attractors where the cause of turbulent behaviors seen in fluid flow. This conjecture received further support after \speccite{gollub1975onset}.

\subsection{What are attractors anyway?}
What exactly is notion of an \textit{attractor}? Intuitively, attractors generalize the more familiar notions of stability for differential equations such as \textit{stable fixed points} and \textit{limit cycles} in the sense that an attractor $A$ is a compact subset such that points starting sufficiently close to $A$ will remain in a given neighborhood of $A$. For a detailed account of the various notions and formulations of an attractor, see \speccite{milnor1985concept}. Nevertheless, the following definition will be adequate.

An \textbf{attractor} is any $A \subseteq M$ which is
\begin{enumerate}
    \item forward invariant
        \[ \forall a \in A, \forall t \geq 0 : \Phi_{t} ( a ) \in A, \]
    \item admits an neighborhood $B(A)$, called the \textbf{basin of attraction}, such for every open neighborhood $N$ of $A$, there is a sufficiently large $T$ such that
        \[ \forall b \in B(A), \forall t \geq T : \Phi_{t} ( b ) \in N, \]
    \item and finally, there is no other proper subset $A' \subset A$ satisfying the above.
\end{enumerate}
Note that some alternative definitions demand that $A$ has non-zero measure (i.e. to eliminate stable fixed points) or that there exists a orbit of the flow that is dense inside $A$.

What makes a strange attractor so \textit{strange}? At the time, \citeauthor{ruelle1971nature} never articulated a concrete definition for strange attractors, instead is was simply an attractor that was ``strange'' to them. Today, strange attractors are defined to be attractors whose dimension is not an integer, i.e. a fractal. Of course, the Lorenz attractor is a strange attractor in this sense as well, as proven in \speccite{tucker2002rigorous}.

\begin{figure}[H]
    \centering
    \includegraphics[scale=0.6]{figures/strange_attractors.png}
    \caption{Some strange attractors. Figures from \href{http://mathworld.wolfram.com/StrangeAttractor.html}{E. W. Weisstein}.}
\end{figure}

\subsection{Reconstructing Attractors}
In \speccite{packard1980geometry}, \citeauthor{packard1980geometry} were interested in problem of \textit{dynamical reconstruction}: from experimental observations of a turbulent fluid flow, how does one show the existence of a low-dimensional chaotic dynamical system which explains the observed behaviors? Additionally, how does one determine the dimensionality of the underlying attractor? They demonstrate their ideas using another non-linear dynamical system studied by \parencite{rossler1976equation}:

\begin{align}
    \label{rossler}
    \begin{split}
        \dt{X} &= - ( Y + Z ), \\
        \dt{Y} &= X + 0.2 Y, \\
        \dt{Z} &= 0.4 + XZ - 5.7 Z.
    \end{split}
\end{align}

Their reconstruction method relied directly on the intuition that in order to specify the state of an $n$-dimensional system at any given time, it is sufficient to known the values of \textit{any} set of $n$ ``independent'' quantities. Moreover, they conjecture that the attractors associated with any such are all \textit{diffeomorphically} equivalent. To support this claim, they compare various trajectories in $3$-dimensions of the Rossler system: $\{ X(t), Y(t), Z(t)\}$, $\{ X(t), X(t - \tau), X(t - 2 \tau)\}$ (this example is credited to Ruelle by the authors), and $\{ X(t), \dot X(t), \ddot X(t) \}$ (by making $\tau$ very small and taking appropriate differences).

\begin{figure}[H]
    \centering
    \includegraphics[scale=0.25]{figures/rossler_reconstruction.png}
    \caption{The \citeauthor{rossler1976equation} system flow projected onto the $X, Y$ plane and the induced flow of $(X, \dot X, \ddot X)$ projected onto the $X, \dot X$ plane. Figure from \parencite{packard1980geometry}.}
\end{figure}

Evidently their heuristic seems to work and moreover it appears possible to reconstruct the dynamics from observing just a single coordinate. They further provide some tools for estimating the dimensionality of the attractor and how it relates to the number non-negative characteristic exponents.

\subsection{Takens' Theorem}

Shortly after this, in \speccite{takens1981detecting}, \citeauthor{takens1981detecting} formalized and proved the conjecture of \parencite{packard1980geometry} which has come to be known as \textit{Takens' Theorem}. The following is a summary of \href{https://www.math.sciences.univ-nantes.fr/~vitturi/talks/PG%20Colloquia/Takens_theorem.pdf}{Marco Vitturi's slides} on the subject.

\textbf{Whitney Embedding Theorem:}
Let $M$ be a compact manifold of (integer) dimension $d$. Then $M$ can be embedded smoothly in $\mathbb{R}^{2n}$ (i.e. no self-intersections).

\begin{figure}[H]
    \centering
    \includegraphics[scale=0.1]{figures/klein_bottle.png}
    \caption{The Klein bottle can be embedded smoothly in $\mathbb R^4$, but not in $\mathbb R^3$.}
\end{figure}

\textbf{Takens' Theorem:}
Let $M$ be a compact manifold of (integer) dimension $d$. Then for \textit{generic} pairs $(\phi, X)$, where
\begin{itemize}
    \item $\phi : M \to M$ is a $C^2$-diffeomorphism of $M$ in itself,
    \item $X : M \to \mathbb R$ is a $C^2$-differentiable function,
\end{itemize}
then the map $\Omega_{\phi, X} : M \to \mathbb R^{2d + 1}$ acting on $V \in M$ defined by
\[ \Omega_{\phi, X}(V) = ( X(V), X(\phi(V)), X(\phi^2(V)), \cdots, X(\phi^{2d}(V))) \]
is an embedding of $M$ into $\mathbb{R}^{2d+1}$ (i.e. injective and immersive).

Takens' theorem when applied to the flow of a dynamical system in $\mathbb R^n$ when $M$ is taken to be the attractor (assume manifold) $A$:
\begin{itemize}
    \item $\phi = \Phi_{-\tau} : A \to A$ for some $\tau > 0$,
    \item $X : A \to \mathbb R$ any coordinate of the system on the attractor,
    \item the \textbf{delay embedding} becomes
        \begin{align*}
            \Omega_{\Phi_{-\tau}, X}(V)
            &= ( X(V), X(\Phi_{-\tau}(V)), X(\Phi_{-2\tau}(V)), \cdots, X(\Phi_{-m\tau}(V))) \\
            &= ( X(t), X(t - \tau), X(t - 2 \tau), \cdots, X(t - m \tau))
        \end{align*}
    \item where $m$ needs to be sufficiently large, i.e. $m \geq 2 \mathrm{dim} (A)$ is sufficient.
    \item Oftentimes a smaller $m$ will still produce an embedding, but Takens' theorem does not guarantee this.
\end{itemize}
A similar result can be stated for reconstructing the orbit in $\mathbb{R}^d$ instead of the entire attractor, you can use $m = d - 1$. Therefore a single variable contains all the information about the orbit:
\[ t \mapsto ( X(t), X(t - \tau), X(t - 2 \tau), \cdots, X(t - (d-1) \tau)). \]

An analogous version for the case of non-integer (Hausdorff) dimensions $d_A$ is due to \speccite{sauer1991embedology}, where $m \geq \lceil 2 d_{A} \rceil}$.

Therefore, the attractor $A$ of the original system is diffeomorphic to the time-delayed attractors $A_X, A_Y$ and $A_Z$ where $A_X = \Omega_{\Phi_{-\tau}, X}(A)$.

Evidently, Takens' theorem reveals that the distinction between the kinematics and dynamics of a system is an arbitrary distinction as one can freely transform between the various phase-spaces in a diffeomorphic way. For similar instances of this arbitrary distinction, and more ideas on this subject, see \speccite{spekkens2015paradigm}.

\section{Selected References on Dynamic Causality}

\subsection{Granger Causality}

There is a pervasive notion of causality used frequently in economic forecasting, but also in neurology and ecology, known as \textit{Granger Causality} or sometimes \textit{G-Causality} named after Clive Granger. For evidence, the library of software tools \speccite{barnett2014mvgc} is highly cited.  Its origins can be traced back to a suggestion due to (Wiener, 1956) for using prediction as a proxy for causality, which was subsequently developed by Granger from 1963 to 1980. Granger was awarded the Novel Prize in Economics in 2003.

In \speccite{granger1988some}, \citeauthor{granger1988some} summarizes his notions of causality for time-series' which were based on two principles:
\begin{enumerate}
    \item The cause occurs before the effect.
    \item The causal series (say $Y(t)$) contains \textit{special information} about the series being caused (say $X(t)$) that is not available in the any other available series (say $W(t)$).
\end{enumerate}

Consider two sets of historical data:
\begin{align}
    J_t &= \{ (X(t-j), Y(t-j), W(t-j)) | j \geq 0 \} \\
    J_t' &= \{ (X(t-j), W(t-j)) | j \geq 0 \}.
\end{align}
His definitions pertain to the following: in the context of $W$, does $Y$ cause $X$?

Letting $P(X | J)$ denote the conditional distribution of $X$ given $J$, if $P(X(t+1) | J_t) = P(X(t+1) | J_t')$ then $Y(t)$ does \textit{not cause} $X(t)$, otherwise $Y(t)$ is a \textit{prima facie cause} of $X(t)$ and becomes a cause with certainty is $J_t$ contains all the information in the universe. Granger acknowledges a number of criticisms in his works.

Most relevant for the current discussion is the article \speccite{lusch2016inferring}, which simulated network systems of Kuramoto oscillators and then used the MGVC toolbox to reconstruct the network structure. They write ``Our results show a significant systematic disparity between the original and inferred network, unless the true structure is extremely sparse or dense.''

\begin{figure}[H]
    \centering
    \includegraphics[scale=0.3]{figures/granger_reconstruction.png}
    \caption{The methodology used by \speccite{lusch2016inferring}.}
\end{figure}

Clearly, Granger's notions of causality are inappropriate for the purposes of causal modeling, despite its widespread use.

\subsection{Convergent Cross Mapping (CCM)}

The seminal article titled \speccite{sugihara2012detecting} has been widely cited in other articles on dynamical causality and throughout ecological and economic communities. Its essential accomplishment, in my opinion, was the realization that nonlinear dynamical systems exhibit a kind of nonseparability which must be dealt with first in order analyze the underlying causal relations and moreover that Takens' theorem resolves this nonseparability.

It opens with an discussion around the notions of ephemeral or ``mirage'' correlations. The following discrete-time non-linear dynamical system,
\begin{align}
    \label{ecosystems}
    \begin{split}
        X(t+1) &= X(t) ( r_x - r_x X(t) - \beta_{x,y} Y(t) ) \\
        Y(t+1) &= Y(t) ( r_y - r_y Y(t) - \beta_{y,x} X(t) )
    \end{split}
\end{align}
exhibits periods of correlation, anticorrelation and no correlation.
\begin{figure}[H]
    \centering
    \includegraphics[scale=0.35]{figures/mirage_correlations.png}
    \caption{An illustration of mirage correlations from \speccite{sugihara2012detecting}.}
\end{figure}
Therefore, using correlation to infer causation is risky, especially in nonlinear dynamics. Moreover, this demonstrates why a lack of correlation does not imply a lack of causation because the former could simply be temporary. In other words, the principle of no fine-tuning only applies in the long term.

\citeauthor{sugihara2012detecting} attributes the inadequacy of Granger causality for dynamical systems to Takens' theorem. In particular, ``information about $X(t)$ that is relevant to predicting $Y$ is redundant in this system''. Stated another way, by algebraically rearranging Eq.~\ref{ecosystems}, it is possible to write $Y(t+1)$ in terms of $Y(t)$ and $Y(t-1)$; thus Granger causality predicts $X$ does not cause $Y$! In \speccite{harnack2017topological}, this nonseparability is referred to as an \textit{entanglement} between $X$ and $Y$.

The essential idea of \parencite{sugihara2012detecting} is that time-series variables $X$ and $Y$ are \textit{causally linked} if they share a common attractor manifold $A$; therefore, each variable can identify the other. Convergent cross mapping (CCM) tests for causal influence from $X$ to $Y$ by measuring the extent to which the history of $Y$ can reliably estimate the states of $X$. For example, a fish time series $Y$ can be used to estimate the weather $X$, but not conversely, thus $X$ causes $Y$; this is counter to Granger causality.

\begin{figure}[H]
    \centering
    \includegraphics[scale=0.4]{figures/convergent_cross_mapping.png}
    \caption{The method of convergent cross mapping (CCM) due to \speccite{sugihara2012detecting}.}
\end{figure}

\textbf{Sketching the CCM Algorithm:}
\begin{enumerate}
    \item Given two time series of length $L$,
        \[ \{ X(1), X(2), \ldots, X(L) \} \quad \text{and} \quad \{ Y(1), Y(2), \ldots, Y(L) \}, \]
        here is how to estimate $Y(t)$ using $X$.
    \item Compute the time lagged coordinate
        \[ \underline{X}(t) = ( X(t), X(t - \tau), X(t - 2 \tau), \ldots, X(t-(d-1)\tau)\}. \]
    \item Then find its $d+1$ nearest neighbors in $A_{X}$, denoted
        \[ (\underline{X}(t_1), \underline{X}(t_2), \ldots, \underline{X}(t_{d+1}) ), \]
        thus forming a $d$-dimensional simplex around $\underline{X}(t)$.
    \item Construct the estimate $\hat Y(t)$ for $Y(t)$ as
        \[ \hat Y(t) | A_X = \sum_{i=1}^{d+1} \omega_{i} Y(t_i) \]
        where the weight $\omega_{i}$ is based on the distance between $\underline X(t)$ and $\underline X(t_i)$.
    \item As the length $L$ increases, the attractor manifold is filled in more and more, and the nearest neighbors get closer and thus $\hat Y(t) | A_X$ should converge to $Y(t)$ (if $Y$ has causal influence on $X$) and vice versa.
\end{enumerate}

\begin{figure}[H]
    \centering
    \includegraphics[scale=0.55]{figures/convergence.png}
    \caption{Empirical results for CCM on Eq.~\ref{ecosystems}. (A) CCM where $\beta_{y,x} > \beta_{x,y}$; the asymmetry is reflected in the differing rates of convergence. (B) How the difference in cross-mapping estimates depends on $\beta_{x,y}$ and $\beta_{y,x}$.   (C,D) Estimates when $\beta_{x,y} = 0$, i.e. when $Y$ has no effect on $X$. From \speccite{sugihara2012detecting}.}
\end{figure}

\begin{figure}[H]
    \centering
    \includegraphics[scale=0.6]{figures/sardines_vs_anchovies.png}
    \caption{(C-F) CCM resolves a long-disputed causal relationship between populations of sardines and anchovies along with sea surface temperatures (SST) measured at Scripps Pier and Newport Pier in California; sardines and anchovies are not interacting, but share a common cause, namely SST. From \speccite{sugihara2012detecting}.}
\end{figure}

Future work demonstrated that CCM successfully distinguishes between direct and indirect causation in causal chains \parencite{ye2015distinguishing}.

\subsection{Topological Causality (TC)}

In \speccite{harnack2017topological}, the concepts of CCM are expanded upon further.
\begin{align*}
    \dt{X_1} &= f_1 ( X_1, w_{12} \mu_2 ( X_2 ) ) \\
    \dt{X_2} &= f_2 ( X_2, w_{21} \mu_1 ( X_1 ) )
\end{align*}
Theoretically, if the diffeomorphism between the attractor manifolds is differentiable, then it can be linearized at a fixed time $t$ (Jacobian matrix), denoted $M_{i\to j}^{t}$. The \textit{expansion} of this linear map, denoted
\[ e^{t}_{i \to j} = \prod_{k} \mathrm{max} (1, \sigma_{k} (M_{i\to j}^{t})) \]
is conjectured to be inversely proportional to the strength of the causal influence $X_j \to X_i$ ($\sigma_{k}(M_{i \to j}^{t})$ is the $k$-th singular value of $M_{i \to j}^{t}$). Unlike vanilla CCM, TC computes a \textit{graduated} measure causal influence which furthermore can be taken to be time-dependent.

\begin{figure}[H]
    \centering
    \includegraphics[scale=0.6]{figures/topological_causality.png}
    \caption{The intuition behind the inverse relationship between expansion and causal influence in \speccite{harnack2017topological}.}
\end{figure}

\subsection{Dimensional Causality (DC)}

The latest development in the field of dynamic causality based on Takens' theorem is the article \speccite{benkHo2018exact}, which presents a method called \textit{Dimensional Causality} (DC). While recognizing the utility of the CCM \parencite{sugihara2012detecting} and TC \parencite{harnack2017topological} techniques for detecting directed and bidirected causal relationships, \citeauthor{benkHo2018exact} identifies their inability to consistently and accurately identify latent common causes. To their best knowledge, ``the DC method is the first exact one which detects and distinguishes all possible causal relations of deterministic dynamical systems.''

The entire technique heavily relies on the ability to estimate the manifold of an attractor from observational data; the authors use the method in \speccite{szepesvari2007manifold} which depends only on a single parameter $k$ related to the resolution.

In order to detect the causal structure underlying two variables $X, Y$, there are three attractor manifolds of interest $A_X, A_Y$ and $A_J$ where $J = X \times Y$ is the joint system. In practice however, they construct the joint dynamics additively using $J' = a X + Y$ with suitably chosen irrational number (they use $a = \sqrt{29/31}$ as its close to unity).

If the two variables are independent, the joint dimension will equal the sum of the dimensions of the independent systems.
\[ \text{Independent case:}\qquad  X \perp Y \iff D_X + D_Y = D_J \]
Any interdependence between $X$ and $Y$ will lead to subadditivity in the manifold dimensions.Whenever there is unidirectional causation $X \to Y$, the information content in the consequent $Y$ is capable of determining the cause $X$, therefore the dimensions $D_X, D_Y$ unequivocally determine the direction of possible causal effect (see Figure~\ref{dim_cause} (A)),
\[ \text{Unidirectional case:}\qquad X \to Y \iff D_X < D_Y = D_J \]
The bidirected case follows naturally, as Takens' theorem proves $A_X$ and $A_Y$ are topologically equivalent and thus have the same dimension.
\[ \text{Bidirectional case:}\qquad X \leftrightarrow Y \iff D_X = D_Y = D_J \]
Detecting the presence of a common cause (without direct causal effect between $X$ and $Y$) is trickier to motivate.
\[ \text{Common cause case:}\qquad X \nwarrow\!\nearrow Y \iff \mathrm{max}(D_X, D_Y) < D_J < D_X + D_Y \]

Alternatively, these relationships become clearer when you realize that the authors are using the Rényi information dimension to estimate the dimensions of the attractors:
\[ d_X = \lim_{N \to \infty} \frac{1}{\log N} H ( [X]_N ) \]
where $H$ is the Shannon entropy and $[X]_N = \lfloor NX \rfloor  / N$ is the $N$-quantized discrete version of $X$. Elementary entropic inequalities give
\[ \mathrm{max}(H(X), H(Y)) \leq H(X, Y) \leq H(X) + H(Y), \]
which translates to the common cause condition above.

To showcase their method, they consider the logistic map,
\[ X_j[t+1] = r X_j[t] ( 1 - \sum_{l = 1}^{3} \beta_{j l} X_l[t] ) \]
with $j, l \in \{1,2,3\}$ and $r = 3.99$. The various causal scenarios are implemented as follows:
\begin{align*}
    \beta_{\to} = \begin{bmatrix} 1 & 0 & 0 \\ 0.5 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix} \qquad
    \beta_{\leftrightarrow} = \begin{bmatrix} 1 & 0.5 & 0 \\ 0.5 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix} \qquad
    \beta_{\nwarrow\!\nearrow} = \begin{bmatrix} 1 & 0 & 0.5 \\ 0 & 1 & 0.5 \\ 0 & 0 & 1 \end{bmatrix} \qquad
    \beta_{\perp} = \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix}
\end{align*}

\begin{figure}[H]
    \centering
    \includegraphics[scale=0.4]{figures/dimensional_causality.png}
    \caption{Dimensional Causality identifies a variety of causal structures. From \speccite{benkHo2018exact}.}
    \label{dim_cause}
\end{figure}

The also demonstrate their method on empirically obtained data and get interesting results. See scanned notes for more comments.

\begin{figure}[H]
    \centering
    \includegraphics[scale=0.5]{figures/inter-hemispherical_interactions.png}
    \caption{An application of the technique of dimensional causality in \speccite{benkHo2018exact}.}
\end{figure}

\section{Other Noteworthy Articles}

In \speccite{wagner1999causality}, \citeauthor{wagner1999causality} argues that a ``...notion of causality can only only be meaningfully defined for systems with \textit{linear} interactions among their variables. For the vastly more important class of \textit{nonlinear} systems, no such notion is likely to exist.'' \\

In \speccite{white2011linking}, \citeauthor{white2011linking} use the formalism of \textit{settable systems}, a generalization of Pearl's Causal Models \parencite{pearl2000causality}, to closely link the concepts of Granger causality to that of Pearl causality.\\

At a quick glance, \speccite{blom2019beyond} appears to be doing a similar thing but for systems at a stable fixed point.\\

Also \speccite{arbach2015dynamic}.

\nocite{*}
\printbibliography
\end{document}