\documentstyle[11pt]{article}
\title{REDUCE Meets CAMAL}
\author{J. P. Fitch \\
School of Mathematical Sciences\\
University of Bath\\
BATH, BA2 7AY, United Kingdom}
\def\today{}
\begin{document}\maketitle

\begin{abstract}
{\em It is generally accepted that special purpose algebraic systems
are more efficient than general purpose ones, but as machines get
faster this does not matter.  An experiment has been performed to see
if using the ideas of the special purpose algebra system CAMAL(F) it
is possible to make the general purpose system REDUCE perform
calculations in celestial mechanics as efficiently as CAMAL did twenty
years ago.  To this end a prototype Fourier module is created for
REDUCE, and it is tested on some small and medium-sized problems taken
from the CAMAL test suite. The largest calculation is the
determination of the Lunar Disturbing Function to the sixth order.  An
assessment is made as to the progress, or lack of it, which computer
algebra has made, and how efficiently we are using modern hardware.
}
\end{abstract}

\section{Introduction}

A number of years ago there emerged the divide between general-purpose
algebra systems and special purpose one.  Here we investigate how far
the improvements in software and more predominantly hardware have
enabled the general systems to perform as well as the earlier special
ones.  It is similar in some respects to the Possion program for
MACSYMA \cite{Fateman} which was written in response to a similar
challenge.

The particular subject for investigation is the Fourier series
manipulator which had its origins in the Cambridge University
Institute for Theoretical Astronomy, and later became the F subsystem
of CAMAL \cite{Barton67b,CAMALF}.  In the late 1960s this system was
used for both the Delaunay Lunar Theory \cite{Delaunay,Barton67a} and
the Hill Lunar Theory \cite{Bourne}, as well as other related
calculations.  Its particular area of application had a number of
peculiar operations on which the general speed depended.  These are
outlined below in the section describing how CAMAL worked.  There have
been a number of subsequent special systems for celestial mechanics,
but these tend to be restricted to the group of the originator.

The main body of the paper describes an experiment to create within
the REDUCE system a sub-system for the efficient manipulation of
Fourier series.  This prototype program is then assessed against both
the normal (general) REDUCE and the extant CAMAL results.  The tests
are run on a number of small problems typical of those for which CAMAL
was used, and one medium-sized problem, the calculation of the Lunar
Disturbing Function.  The mathematical background to this problem is
also presented for completeness.  It is important as a problem as it
is the first stage in the development of a Delaunay Lunar Theory.

The paper ends with an assessment of how close the performance of a
modern REDUCE on modern equipment is to the (almost) defunct CAMAL of
eighteen years ago.

\section{How CAMAL Worked}

The Cambridge Algebra System was initially written in assembler for
the Titan computer, but later was rewritten a number of times, and
matured in BCPL, a version which was ported to IBM mainframes and a
number of microcomputers.  In this section a brief review of the main
data structures and special algorithms is presented.

\subsection{CAMAL Data Structures}

CAMAL is a hierarchical system, with the representation of polynomials
being completely independent of the representations of the angular
parts.

The angular part had to represent a polynomial coefficient, either a
sine or cosine function and a linear sum of angles.  In the problems
for which CAMAL was designed there are 6 angles only, and so the
design restricted the number, initially to six on the 24 bit-halfword
TITAN, and later to eight angles on the 32-bit IBM 370, each with
fixed names (usually u through z).  All that is needed is to remember
the coefficients of the linear sum.  As typical problems are
perturbations, it was reasonable to restrict the coefficients to small
integers, as could be represented in a byte with a guard bit.  This
allowed the representation to pack everything into four words.
\begin{verbatim}
    [ NextTerm, Coefficient, Angles0-3, Angles4-7 ]
\end{verbatim}
The function was coded by a single bit in the {\tt Coefficient} field.  This
gives a particularly compact representation.  For example the Fourier
term $\sin(u-2v+w-3x)$ would be represented as
\begin{verbatim}
    [ NULL, "1"|0x1, 0x017e017d, 0x00000000 ]
or
    [ NULL, "1"|0x1, 1:-2:1:-3, 0:0:0:0 ]
\end{verbatim}
where {\tt "1"} is a pointer to the representation of the polynomial
1.  In all this representation of the term took 48 bytes.  As the
complexity of a term increased the store requirements to no grow much;
the expression $(7/4) a e^3 f^5 \cos(u-2v+3w-4x+5y+6z)$ also takes 48
bytes.  There is a canonicalisation operation to ensure that the
leading angle is positive, and $\sin(0)$ gets removed.  It should be
noted that $\cos(0)$ is a valid and necessary representation.

The polynomial part was similarly represented, as a chain of terms
with packed exponents for a fixed number of variables.  There is no
particular significance in this except that the terms were held in
{\em increasing} total order, rather than the decreasing order which
is normal in general purpose systems.  This had a number of important
effects on the efficiency of polynomial multiplication in the presence
of a truncation to a certain order.  We will return to this point
later.  Full details of the representation can be found in
\cite{LectureNotes}.

The space administration system was based on explicit return rather
than garbage collection.  This meant that the system was sometimes
harder to write, but it did mean that much attention was focussed on
efficient reuse of space.  It was possible for the user to assist in
this by marking when an expression was needed no longer, and the
compiler then arranged to recycle the space as part of the actual
operation.  This degree of control was another assistance in running
of large problems on relatively small machines.

\subsection{Automatic Linearisation}

In order to maintain Fourier series in a canonical form it is
necessary to apply the transformations for linearising products of
sine and cosines.  These will be familiar to readers of the REDUCE
test program as
\begin{eqnarray}
\cos \theta \cos \phi & \Rightarrow &
                (\cos(\theta+\phi)+\cos(\theta-\phi))/2, \\
\cos \theta \sin \phi & \Rightarrow &
                (\sin(\theta+\phi)-\sin(\theta-\phi))/2, \\
\sin \theta \sin \phi & \Rightarrow &
                (\cos(\theta-\phi)-\cos(\theta+\phi))/2, \\
\cos^2 \theta & \Rightarrow & (1+\cos(2\theta))/2,      \\
\sin^2 \theta & \Rightarrow & (1-\cos(2\theta))/2.
\end{eqnarray}
In CAMAL these transformations are coded directly into the
multiplication routines, and no action is necessary on the part of the
user to invoke them.  Of course they cannot be turned off either.

\subsection{Differentiation and Integration}

The differentiation of a Fourier series with respect to an angle is
particularly simple.  The integration of a Fourier series is a little
more interesting.  The terms like $\cos(n u + \ldots)$ are easily
integrated with respect to $u$, but the treatment of terms independent
of the angle would normally introduce a secular term.  By convention
in Fourier series these secular terms are ignored, and the constant of
integration is taken as just the terms independent of the angle in the
integrand.  This is equivalent to the substitution rules
\begin{eqnarray*}
\sin(n \theta) & \Rightarrow & -(1/n) \cos(n \theta) \\
\cos(n \theta) & \Rightarrow & (1/n) \sin(n \theta)
\end{eqnarray*}

In CAMAL these operations were coded directly, and independently of
the differentiation and integration of the polynomial coefficients.

\subsection{Harmonic Substitution}

An operation which is of great importance in Fourier operations is the
{\em harmonic substitution}.  This is the substitution of the sum of
some angles and a general expression for an angle.  In order to
preserve the format, the mechanism uses the translations
\begin{eqnarray*}
\sin(\theta + A) & \Rightarrow & \sin(\theta) \cos(A) +
                                 \cos(\theta) \sin(A) \\
\cos(\theta + A) & \Rightarrow & \cos(\theta) \cos(A) -
                                 \sin(\theta) \sin(A) \\
\end{eqnarray*}
and then assuming that the value $A$ is small it can be replaced by
its expansion:
\begin{eqnarray*}
\sin(\theta + A) & \Rightarrow & \sin(\theta) \{1 - A^2/2! + A^4/4!\ldots\} +\\
                 &             & \cos(\theta) \{A - A^3/3! + A^5/5!\ldots\} \\
\cos(\theta + A) & \Rightarrow & \cos(\theta) \{1 - A^2/2! + A^4/4!\ldots\} -\\
                 &             & \sin(\theta) \{A - A^3/3! + A^5/5! \ldots\} \\
\end{eqnarray*}
If a truncation is set for large powers of the polynomial variables
then the series will terminate.  In CAMAL the {\tt HSUB} operation
took five arguments; the original expression, the angle for which
there is a substitution, the new angular part, the expression part
($A$ in the above), and the number of terms required.

The actual coding of the operation was not as expressed above, but by
the use of Taylor's theorem.  As has been noted above the
differentiation of a harmonic series is particularly easy.

\subsection{Truncation of Series}

The main use of Fourier series systems is in generating perturbation
expansions, and this implies that the calculations are performed to
some degree of the small quantities.  In the original CAMAL all
variables were assumed to be equally small (a restriction removed in
later versions).  By maintaining polynomials in increasing maximum
order it is possible to truncate the multiplication of two
polynomials.  Assume that we are multiplying the two polynomials
\begin{eqnarray*}
        A = a_0 + a_1 + a_2 + \ldots \\
        B = b_0 + b_1 + b_2 + \ldots
\end{eqnarray*}
If we are generating the partial answer
\[
        a_i (b_0 + b_1 + b_2 + \ldots)
\]
then if for some $j$ the product $a_i b_j$ vanishes, then so will all
products $a_i b_k$ for $k>j$.  This means that the later terms need
not be generated.  In the product of $1+x+x^2+x^3+\ldots+x^{10}$ and
$1+y+y^2+y^3+\ldots+y^10$ to a total order of 10 instead of generating
100 term products only 55 are needed.  The ordering can also make the
merging of the new terms into the answer easier.

\section{Towards a CAMAL Module}

For the purposes of this work it was necessary to reproduce as many of
the ideas of CAMAL as feasible within the REDUCE framework and
philosophy.  It was not intended at this stage to produce a complete
product, and so for simplicity a number of compromises were made with
the ``no restrictions'' principle in REDUCE and the space and time
efficiency of CAMAL.  This section describes the basic design
decisions.

\subsection{Data Structures}

In a fashion similar to CAMAL a two level data representation is used.
The coefficients are the standard quotients of REDUCE, and their
representation need not concern us further.  The angular part is
similar to that of CAMAL, but the ability to pack angle multipliers
and use a single bit for the function are not readily available in
Standard LISP, so instead a longer vector is used.  Two versions were
written.  One used a balanced tree rather than a linear list for the
Fourier terms, this being a feature of CAMAL which was considered but
never coded.  The other uses a simple linear representation for sums.
The angle multipliers are held in a separate vector in order to allow
for future flexibility.  This leads to a representation as a vector of
length 6 or 4;
\begin{verbatim}
Version1:  [ BalanceBits, Coeff, Function, Angles, LeftTree, RightTree ]
Version2:  [ Coeff, Function, Angles, Next ]
\end{verbatim}
where the {\tt Angles} field is a vector of length 8, for the
multipliers.  It was decided to forego packing as for portability we
do not know how many to pack into a small integer.  The tree system
used is AVL, which needs 2 bits to maintain balance information, but
these are coded as a complete integer field in the vector.  We can
expect the improvements implicit in a binary tree to be advantageous
for large expressions, but the additional overhead may reduce its
utility for smaller expressions.

A separate vector is kept relating the position of an angle to its
print name, and on the property list of each angle the allocation of
its position is kept.  So long as the user declares which variables
are to be treated as angles this mechanism gives flexibility which was
lacking in CAMAL.

\subsection{Linearisation}

As in the CAMAL system the linearisation of products of sines and
cosines is done not by pattern matching but by direct calculation at
the heart of the product function, where the transformations (1)
through (3) are made in the product of terms function.  A side effect
of this is that there are no simple relations which can be used from
within the Fourier multiplication, and so a full addition of partial
products is required.  There is no need to apply linearisations
elsewhere as a special case.  Addition, differentiation and
integration cannot generate such products, and where they can occur in
substitution the natural algorithm uses the internal multiplication
function anyway.

\subsection{Substitution}

Substitution is the main operation of Fourier series.  It is useful to
consider three different cases of substitutions.
\begin{enumerate}
\item Angle Expression for Angle:
\item Angle Expression + Fourier Expression for Angle:
\item Fourier Expression for Polynomial Variable.
\end{enumerate}

The first of these is straightforward, and does not require any
further comment.  The second substitution requires a little more care,
but is not significantly difficult to implement.  The method follows
the algorithm used in CAMAL, using TAYLOR series.  Indeed this is the
main special case for substitution.

The problem is the last case.  Typically many variables used in a
Fourier series program have had a WEIGHT assigned to them.  This means
that substitution must take account of any possible WEIGHTs for
variables.  The standard code in REDUCE does this in effect by
translating the expression to prefix form, and recalculating the value.
A Fourier series has a large number of coefficients, and so this
operations are repeated rather too often.  At present this is the
largest problem area with the internal code, as will be seen in the
discussion of the Disturbing Function calculation.

\section{Integration with REDUCE}

The Fourier module needs to be seen as part of REDUCE rather than as a
separate language.  This can be seen as having internal and external
parts.

\subsection{Internal Interface}

The Fourier expressions need to co-exist with the normal REDUCE syntax
and semantics.  The prototype version does this by (ab)using the
module method, based in part on the TPS code \cite{Barnes}.  Of course
Fourier series are not constant, and so are not really domain
elements.  However by asserting that Fourier series form a ring of
constants REDUCE can arrange to direct basic operations to the Fourier
code for addition, subtraction, multiplication and the like.

The main interface which needs to be provided is a simplification
function for Fourier expressions.  This needs to provide compilation
for linear sums of angles, as well as constructing sine and cosine
functions, and creating canonical forms.

\subsection{User Interface}

The creation of {\tt HDIFF} and {\tt HINT} functions for
differentiation disguises this.  An unsatisfactory aspect of the
interface is that the tokens {\tt SIN} and {\tt COS} are already in
use.  The prototype uses the operator form
\begin{verbatim}
        fourier sin(u)
\end{verbatim}
to introduce harmonically represented sine functions.  An alternative of
using the tokens {\tt F\_SIN} and {\tt F\_COS} is also available.

It is necessary to declare the names of the angles, which is achieved
with the declaration
\begin{verbatim}
        harmonic theta, phi;
\end{verbatim}

At present there is no protection against using a variable as both an
angle and a polynomial varaible.  This will nooed to be done in a
user-oriented version.

\section{The Simple Experiments}

The REDUCE test file contains a simple example of a Fourier
calculation, determining the value of $(a_1 \cos({wt}) + a_3
\cos(3{wt}) + b_1 \sin({wt}) + b_3 \sin(3{wt}))^3$.  For the purposes
of this system this is too trivial to do more than confirm the correct
answers.

The simplest non-trivial calculation for a Fourier series manipulator
is to solve Kepler's equation for the eccentric anomoly E in terms of
the mean anomoly u, and the eccentricity of an orbit e, considered as a
small quantity
\[
        E = u + e \sin E
\]
The solution procedes by repeated approximation.  Clearly the initial
approximation is $E_0 = u$.  The $n^{th}$ approximation can be written
as $u + A_n$, and so $A_n$ can be calculated by
\[
        A_k = e \sin (u + A_{k-1})
\]
This is of course precisely the case for which the HSUB operation is
designed, and so in order to calculate $E_n - u$ all one requires is
the code
\begin{verbatim}
        bige := fourier 0;
        for k:=1:n do <<
          wtlevel k;
          bige:=fourier e * hsub(fourier(sin u), u, u, bige, k);
        >>;
        write "Kepler Eqn solution:", bige$
\end{verbatim}

It is possible to create a regular REDUCE program to simulate this (as
is done for example in Barton and Fitch\cite{Barton72}, page 254).
Comparing these two programs indicates substantial advantages to the
Fourier module, as could be expected.
\medskip
\begin{center}
\begin{tabular}{ | c | l l |}
\multicolumn{3}{c}{\bf Solving Kepler's Equation} \\
\hline
Order   &       REDUCE  &       Fourier Module \\
5       &       9.16    &       2.48    \\
6       &       17.40   &       4.56    \\
7       &       33.48   &       8.06    \\
8       &       62.76   &       13.54   \\
9       &       116.06  &       21.84   \\
10      &       212.12  &       34.54   \\
11      &       381.78  &       53.94   \\
12      &       692.56  &       82.96   \\
13      &       1247.54 &       125.86  \\
14      &       2298.08 &       187.20  \\
15      &       4176.04 &       275.60  \\
16      &       7504.80 &       398.62  \\
17      &       13459.80        &       569.26  \\
18      &       ***     &       800.00  \\
19      &       ***     &       1116.92 \\
20      &       ***     &       1536.40 \\
\hline
\end{tabular}
\end{center}
\medskip
These results were with the linear representation of Fourier series.
The tree representation was slightly slower.  The ten-fold speed-up
for the 13th order is most satisfactory.

\section{A Medium-Sized Problem}

Fourier series manipulators are primarily designed for large-scale
calculations, but for the demonstration purposes of this project a
medium problem is considered.  The first stage in calculating the
orbit of the Moon using the Delaunay theory (of perturbed elliptic
motion for the restricted 3-body problem) is to calculate the energy
of the Moon's motion about the Earth --- the Hamiltonian of the
system.   This is the calculation we use for comparisons.

\subsection{Mathematical Background}

The full calculation is described in detail in \cite{Brown}, but a
brief description is given here for completeness, and to grasp the
extent of the calculation.

Referring to the figure 1 which gives the cordinate system, the basic
equations are
\begin{eqnarray}
S  & = & (1-\gamma ^2)\cos(f + g +h -f' -g' -h')
+ \gamma ^2 cos(f + g -h +f' +g' +h') \\
r & = & a (1 - e \cos E) \\
l & = & E - e \sin E \\
a & = & r {{\bf d} E} \over {{\bf d} l} \\
r ^2 {{\bf d} f} \over {{\bf d} l} & = & a^2 (1 - e^2)^{1 \over 2}\\
R & = & m' {a^2 \over {a'} ^3} {{a'}\over {r
'}} \left \{ \left ({r \over a}\right )^2
\left ({{a'} \over {r'}}\right )^2 P_2(S) +
\left ({a \over {a'}}\right )\left
({r \over a}\right )^3 \left ({{a'} \over {r'}}\right )^3 P_3(S)
+ \ldots \right \}
\end{eqnarray}

There are similar equations to (7) to (10) for the quantities $r'$,
$a'$, $e'$, $l'$, $E'$ and $f'$ which refer to the position of the Sun
rather than the Moon.  The problem is to calculate the expression $R$
as an expansion in terms of the quantities $e$, $e'$, $\gamma$,
$a/a'$, $l$, $g$, $h$, $l'$, $g'$ and $h'$.  The first three
quantities are small quantities of the first order, and $a/a'$ is of
second order.

The steps required are
\begin{enumerate}
\item Solve the Kepler equation (8)
\item Substiture into (7) to give $r/a$ in terms of $e$ and $l$.
\item Calculate $a/r$ from (9) and $f$ from (10)
\item Substitute for $f$ and $f'$ into $S$ using (6)
\item Calculate $R$ from $S$, $a'/r'$ and $r/a$
\end{enumerate}

The program is given in the Appendix.

\subsection{Results}

The Lunar Disturbing function was calculated by a direct coding of the
previous sections' mathematics.  The program was taken from Barton
and Fitch \cite{Barton72} with just small changes to generalise it for
any order, and to make it acceptable for Reduce3.4.  The Fourier
program followed the same pattern, but obviously used the {\tt HSUB}
operation as appropriate and the harmonic integration.  It is very
similar to the CAMAL program in \cite{Barton72}.

The disturbing function was calculated to orders 2, 4 and 6 using
Cambridge LISP on an HLH Orion 1/05 (Intergraph Clipper), with the
three programs $\alpha$) Reduce3.4, $\beta$) Reduce3.4 + Camal Linear
Module and $\gamma$) Reduce3.4 + Camal AVL Module.  The timings for
CPU seconds (excluding garbage collection time) are summarised the
following table:
\medskip
\begin{center}
\begin{tabular}{ | c || l | l | l |}
\hline
Order of DDF    & Reduce        & Camal Linear  & Camal Tree \\
\hline
2       &       23.68   &       11.22   &       12.9    \\
4       &       429.44  &       213.56  &       260.64  \\
6       &       $>$7500 &       3084.62 &       3445.54 \\
\hline
%%% Linear n=4 138.72 (4Mb + unsafe vector access + recurrance)
%%% Linear n=6 1870.10 (4Mb + unsafe vector access + recurrance)
\end{tabular}
\end{center}
\medskip

If these numbers are normalised so REDUCE calculating the DDF is 100
units for each order the table becomes
\medskip
\begin{center}
\begin{tabular}{ | c || l | l | l |}
\hline
Order of DDF    & Reduce        & Camal Linear  & Camal Tree \\ \hline
2       &       100     &       47.38   &       54.48   \\
4       &       100     &       49.73   &       60.69   \\
6       &       100     &       $<$41.13        &       $<$45.94 \\
\hline
\end{tabular}
\end{center}
\medskip

From this we conclude that a doubling of speed is about correct, and
although the balanced tree system is slower as the problem size
increases the gap between it and the simpler linear system is
narrowing.

It is disappointing that the ratio is not better, nor the absolute
time less.  It is worth noting in this context that Jefferys claimed
that the sixth order DDF took 30s on a CDC6600 with TRIGMAN in 1970
\cite{Jefferys}, and Barton and Fitch took about 1s for the second
order DDF on TITAN with CAMAL \cite{Barton72}.  A closer look at the
relative times for individual sections of the program shows that the
substitution case of replacing a polynomial variable by a Fourier
series is only marginally faster than the simple REDUCE program.  In
the DDF program this operation is only used once in a major form,
substituting into the Legendre polynomials, which have been previously
calculated by Rodrigues formula.  This suggests that we replace this
with the recurrence relationship.

Making this change actually slows down the normal REDUCE by a small
amount but makes a significant change to the Fourier module; it
reduces the run time for the 6th order DDF from 3084.62s to 2002.02s.
This gives some indication of the problems with benchmarks.  What is
clear is that the current implementation of substitution of a Fourier
series for a polynomial variable is inadequate.

\section{Conclusion}

The Fourier module is far from complete.  The operations necessary for
the solution of Duffing's and Hill's equations are not yet written,
although they should not cause much problem.  The main defficiency is
the treatment of series truncation; at present it relies on the REDUCE
WTLEVEL mechanism, and this seems too coarse for efficient truncation.
It would be possible to re-write the polynomial manipulator as well,
while retaining the REDUCE syntax, but that seems rather more than one
would hope.

The real failure so far is the large time lag between the REDUCE-based
system on a modern workstation against a mainframe of 25 years ago
running a special system.  The CAMAL Disturbing function program could
calculate the tenth order with a maximum of 32K words (about
192Kbytes) whereas this system failed to calculate the eigth order in
4Mbytes (taking 2000s before failing).  I have in my archives the
output from the standard CAMAL test suite, which includes a sixth
order DDF on an IBM 370/165 run on 2 June 1978, taking 22.50s and
using a maximum of 15459 words of memory for heap --- or about
62Kbytes.  A rough estimate is that the Orion 1/05 is comparable in
speed to the 360/165, but with more real memory and virtual memory.

However, a simple Fourier manipulator has been created for REDUCE which
performs between twice and three times the speed of REDUCE using
pattern matching.  It has been shown that this system is capable of
performing the calculations of celestial mechanics, but it still
seriously lags behind the efficiency of the specialist systems of
twenty years before.  It is perhaps fortunate that it was not been
possible to compare it with a modern specialist system.

There is still work to do to provide a convenient user interface, but
it is intended to develop the system in this direction.  It would be
pleasant to have again a system of the efficiency of CAMAL(F).

I would like to thank Codemist Ltd for the provision of computing
resources for this project, and David Barton who taught be so much
about Fourier series and celstial mechanics.  Thank are also due to
the National Health Service, without whom this work and paper could not
have been produced.

\section*{Appendix: The DDF Function}
\begin{verbatim}
array p(n/2+2);
harmonic u,v,w,x,y,z;
weight e=1, b=1, d=1, a=1;

%% Generate Legendre Polynomials to sufficient order
for i:=2:n/2+2 do <<
  p(i):=(h*h-1)^i;
  for j:=1:i do p(i):=df(p(i),h)/(2j)
>>;

%%%%%%%%%%%%%%%% Step1: Solve Kepler equation
bige := fourier 0;
for k:=1:n do <<
  wtlevel k;
  bige:=fourier e * hsub(fourier(sin u), u, u, bige, k);
>>;

%% Ensure we do not calculate things of too high an order
wtlevel n;

%%%%%%%%%%%%%%%% Step 2: Calculate r/a in terms of e and l
dd:=-e*e; hh:=3/2; j:=1; cc := 1;
for i:=1:n/2 do <<
  j:=i*j; hh:=hh-1; cc:=cc+hh*(dd^i)/j
>>;
bb:=hsub(fourier(1-e*cos u), u, u, bige, n);
aa:=fourier 1+hdiff(bige,u); ff:=hint(aa*aa*fourier cc,u);

%%%%%%%%%%%%%%%% Step 3: a/r and f
uu := hsub(bb,u,v); uu:=hsub(uu,e,b);
vv := hsub(aa,u,v); vv:=hsub(vv,e,b);
ww := hsub(ff,u,v); ww:=hsub(ww,e,b);

%%%%%%%%%%%%%%%% Step 4: Substitute f and f' into S
yy:=ff-ww; zz:=ff+ww;
xx:=hsub(fourier((1-d*d)*cos(u)),u,u-v+w-x-y+z,yy,n)+
    hsub(fourier(d*d*cos(v)),v,u+v+w+x+y-z,zz,n);

%%%%%%%%%%%%%%%% Step 5: Calculate R
zz:=bb*vv; yy:=zz*zz*vv;

on fourier;
for i := 2:n/2+2 do <<
  wtlevel n+4-2i; p(i) := hsub(p(i), h, xx) >>;

wtlevel n;
for i:=n/2+2 step -1 until 3 do
    p(n/2+2):=fourier(a*a)*zz*p(n/2+2)+p(i-1);
yy*p(n/2+2);

\end{verbatim}
\newpage
\bibliographystyle{plain}
\bibliography{camal}

\end{document}


\chapter*{Introduction}
\addcontentsline{toc}{chapter}{Introduction}

Clustering is a commonly used technique for simplifying the work with complex data\-sets.
The main principle is to represent a dataset composed of a huge number of elements by a simplified (and significantly smaller) set of element groups which are commonly called clusters.
There are many various ways of constructing the clustering algorithms because each discipline typically requires a different kind of cluster similarity, it manipulates completely different data (texts, sequences, vectors) and it has different expectations about the organizing of the resulting data.

Clustering is commonly used to analyze datasets originating in single-cell cytometry~\cite{shapiro2005practical}, where grouping the cells by similar measured features often corresponds to creating clusters of all individual biological types of the cells. This greatly simplifies a data analysis and a distinction of different cell types. Many customized approaches for clustering cytometry data have been developed, including FlowSOM~\cite{van2015flowsom}, PhenoGraph~\cite{levine2015data}, SPADE~\cite{qiu2011extracting} or FlowGrid~\cite{ye2019ultrafast}. This thesis focuses on agglomerative hierarchical clustering with the Mahalanobis metric, originally developed by \citet{fivser2012detection} for the purpose of monitoring minimal residual decease in patients with leukemia.

The Mahalanobis clustering is, however, seriously restricted by the performance and complexity of the current implementation, which can cluster tens of thousands of cells on a common hardware.
The computation of these clusters is time demanding and consumes a lot of computer memory. Spe\-cifically for the Mahalanobis clustering, the maximum size of a dataset that can be processed on common hardware varies at about 100,000 cells. As a result, the performance of the Mahalanobis clustering is unsuitable for processing data from modern cytometers used in current experiments, which often produce datasets of more than several million cells.

The primary goal of this thesis is to research possibilities of accelerating the Mahalanobis clustering on a GPU, develop the implementation of the clustering accelerated on a GPU and measure its results. Many clustering algorithms have been already ported to a GPU device, eg.~DBSCAN~\cite{andrade2013g}, UPGMA~\cite{hua2017mgupgma}, etc.. The Mahalanobis clustering is extremely suitable for cytometry because it naturally forms elliptical clusters. A GPU-accelerated implementation that could improve the performance on large datasets has not been developed yet.

The thesis first discusses the hierarchical clustering (section~\ref{sec01:hierarch_clust}) and variations of algorithms that can be used (section~\ref{sec01:hca}). Then, an overview of the programming for the GPU is introduced (section~\ref{sec02:cuda}). Using this information, the thesis designs a parallelized algorithm of MHCA which can be run on a GPU; therefore, it is significantly faster. The combination of simple metric spaces with Minkowski distance specialization and enormous GPU throughput allows for a major memory requirements reduction. The total acceleration gain of the implementation varies from 20 times to 5000 times. The quality and speed of the result is demonstrated on flow and mass cytometry datasets. We hope that the results of the thesis will be possible to package and use for biologically relevant purposes.


\chapter{Machine Learning and \mbox{uncertainties}}
\label{methods_ML}

\vspace{-15pt} % one line spacing corresponds approx to 15 pts
\begin{tcolorbox}[enhanced,width=\textwidth,size=fbox,
        sharp corners,colframe=black!5!white,drop fuzzy shadow southeast,
        boxrule=3mm, parbox=false]

This chapter borrows from the article \citep{walch_big_2020}:

\qquad %\bibentry{walch_big_2020}
A. Walch, R. Castello, N. Mohajeri, and J.-L. Scartezzini (2020).
Big data mining for the estimation of hourly rooftop photovoltaic potential and its uncertainty.
\textit{Applied Energy}, 262:114404.
\href{https://doi.org/10.1016/j.apenergy.2019.114404}{doi:10.1016/j.apenergy.2019.114404}

and the conference proceedings \cite{walch_spatio-temporal_2019, walch_fast_2019}:

\quad A. Walch, R. Castello, N. Mohajeri, F. Guignard, M. Kanevski, and J.-L. Scartezzini (2019).
Spatio-temporal modelling and uncertainty estimation of hourly global solar irradianceusing Extreme Learning Machines.
\textit{Energy Procedia}, 158:6378–6383.\\
\href{https://doi.org/10.1016/j.egypro.2019.01.219}{doi:10.1016/j.egypro.2019.01.219}
% \quad \bibentry{walch_spatio-temporal_2019}

\quad % \bibentry{walch_fast_2019}
A. Walch, R. Castello, N. Mohajeri, and J.-L. Scartezzini (2019).
A Fast Machine Learning Model for Large-Scale Estimation of Annual Solar Irradiation on Rooftops.
\textit{ISES SWC2019 / SHC2019 Conference Proceedings}.
\href{https://doi.org/10.18086/swc.2019.45.12}{doi:10.18086/swc.2019.45.12}

\end{tcolorbox}

The large-scale estimation of RE potential at high spatial and temporal resolution challenges the analytical and geospatial models from Chapter~\ref{methods_physical}, for three main reasons.
First, some data required to perform the analytical and geospatial modelling may not be available across the entire region for which the potential is computed.
Second, the resolution or quality of some of the input datasets may be insufficient to assure a high-resolution potential estimation, but it could be improved through the use of additional information.
Third, these physics-based approaches may be too computationally intensive, inhibiting their application at the large scale.
The use of Machine Learning (ML) as a method to address these limitations of "classical" physics-based models has attracted increasing attention in recent years \cite{willard_integrating_2020}.

In this thesis, ML is used to address the above-mentioned limitations for the national-scale estimation of RPV (Chapter~\ref{solar}) and GSHP (Chaper~\ref{geothermal}) potential.
The ML models hereby learn and predict the relationship between a set of input data (\textit{features}) and output variables (\textit{targets}), known as \textit{supervised learning}, as summarized in Section~\ref{ML_supervised}.
For an in-depth introduction to Machine Learning, interested readers may refer for example to \citet{bishop_pattern_2006}.

The use of data-driven methods further requires the estimation of \textit{uncertainties} related to the ML predictions, which are essential for decision-making processes \cite{knusel_argument-based_2020}.
Uncertainties may arise for example from noise in input data sources or from the modelling approaches themselves, and their quantification is a vast field of research \cite{willard_integrating_2020,knusel_argument-based_2020}.
Uncertainties arising from data noise and from ML models are modelled here in the form of standard deviations, which are propagated through the analytical models presented in Chapter~\ref{methods_physical}.
Section~\ref{unc_ML} provides an overview of the applied methods for a systematic quantification of uncertainties through combined physics-based and data-driven modelling approaches.

\section{Supervised learning}
\label{ML_supervised}

Supervised learning algorithms are trained on a set of input features and corresponding output targets. The training procedure aims to minimise the error between an algorithm's prediction and the known ("true") target.
Several design choices can influence the performance of supervised learning algorithms: (i) the choice of the input features, (ii) the choice of an appropriate ML model for a given problem, (iii) the
choice of the parameters defining the model structure (hyper-parameters), and (iv) the size of the training data.
The selection of suitable input features varies largely between different ML applications and is often based on expert knowledge.
The full set of available features, obtained using expert knowledge or otherwise, may however contain features which are redundant or even detrimental to the ML model's performance.
Feature exploration and selection techniques may hence be used to enhance the quality of the feature set (see Section~\ref{ML_features}).

The choice of an appropriate ML model depends on the structure of the problem and the available data. Regression algorithms are used to model continuous targets, while classification algorithms are used for binary or multi-class targets (see Section~\ref{ML_models}). Furthermore, some algorithms such as Support Vector Machines (SVM) may be better suited for high-dimensional feature sets (i.e. many features), while other algorithms such as Extreme Learning Machines (ELM) are optimized for very large datasets (i.e. many samples). For image or natural language processing, convolutional neural networks (CNN) and recursive neural networks (RNN) have gained high popularity in recent years.

For any ML model, the optimization of its hyper-parameters (referred to as model \textit{tuning}) is performed during the training phase. For model training, the labelled data (all data samples for which both features and targets are known) are divided into three random subsets:
(i) The \textit{training} set, which is used to fit the model, (ii) the \textit{validation} set, which is used to assess the model's performance and to select the hyper-parameters, and (iii) the \textit{test} set, which is excluded from the tuning procedure and used for evaluating the general performance of the model.
The test set usually contains $20 - 33\%$ of the labelled data. The remaining data is then split again, keeping another $20 - 33\%$ for model validation.
To avoid biases in the choice of the training and validation sets, a $k$-fold cross-validation (CV) procedure is widely applied in the literature.
For this, the training data is randomly split into $k$ subsets (folds), from which $k$ ML models are trained. Each model uses ($k$ – 1) folds for training and the last fold for validation.
Unless mentioned otherwise, a 5-fold cross-validation ($k$ = 5) is used throughout this work.

\subsection{Feature exploration and selection techniques}
\label{ML_features}

A vast variety of methods exist in the literature for the exploration, analysis and selection of appropriate features for ML models, which are often highly dependent on the application. In this section, five concepts for designing ML feature sets are introduced, which have proven useful for the ML applications presented in this thesis.

\textbf{Correlation analysis} provides insights into potential redundancies between different features, and allows to obtain a first quantitative measure of the relevance of each feature towards predicting the target variable(s).
In addition to the widely used Pearson correlation coefficient, which measures linear correlations, other measures of non-linear dependencies between variables may be used, such as Spearman's rank correlation coefficient \cite{kokoska_crc_1999} or mutual information \cite{kozachenko_statistical_1987}. While low correlations are desirable between the different features, indicating low redundancy within the feature set, high correlations between the features and the target suggest a high importance towards predicting this target.

\textbf{Feature transformation} includes the normalisation of the features, typically to zero mean and unit variance, but it may further refer to non-linear transformations of the features or the target. While normalisation is standard practice and required for many ML algorithms, the suitability of non-linear transformations depends on the data. For example, log-transformations may significantly improve the predictions for features and targets at exponential scale. Visualisations like histograms or scatterplots can provide insights into suitable transformations.

\textbf{Principal Component Analysis (PCA)} \cite{tipping_probabilistic_1999} is an advanced feature transformation technique which yields a set of independent (principal) components. These components are ranked according to the percentage of explained variance in each component. To select features using PCA, the minimum percentage of explained variance may be defined, such that the last-ranked components up to the required percentage of explained variance are discarded.
The advantage of this method is that it yields only independent components, which is favourable for the performance of many ML models, and it can drastically reduce the number of features in high-dimensional problems. However, the PCA is based on the features only and contains no information on the relevance of the features to predicting the target. It further requires all available features as input, so it cannot be used to reduce the number of input variables.

\textbf{Sequential backward selection (SBS) }\cite{ferri_comparative_1994} is one method that can be applied to reduce the number of input variables. Its advantage is that it can be applied to any ML algorithm, using any metric of evaluation.
Starting from the complete feature set, the SBS iteratively excludes one feature at a time and computes the error for the prediction using the reduced feature set. The feature whose exclusion causes the lowest change in prediction error ($\Delta_{err}$) is permanently removed from the set of features. This procedure is repeated until only one feature remains, and the $\Delta_{err}$ is recorded for each iteration.
To reduce the number of features, a threshold for $\Delta_{err}$ can be applied, selecting all features that keep $\Delta_{err}$ below the threshold.
As the number of iterations increases exponentially with the number of features, this procedure may not be applicable for high-dimensional problems.
%
In this work, SBS is performed using the k-Nearest Neighbour algorithm (see Section~\ref{ML_models}) due to its fast computational time, in combination with a 5-fold CV (see above).

\textbf{Feature importance.} In addition to the above methods, which provide some insights into the importance of the features towards predicting the target, some ML algorithms provide a measure of feature importance as part of their algorithm's design.
Notably, decision tree algorithms (e.g. Random Forests) provide such a measure of feature importance. As the dataset is split at each node of the decision tree (see Fig.~\ref{fig:rf}) based on a threshold for one feature, the reduction of the impurity (or variance) in the targets due to each feature can be quantified \cite{breiman_random_2001}.
As redundancy between features may lead to misleading importance scores, this method should be combined with other analysis techniques.

\subsection{Regression and classification models}
\label{ML_models}
\label{RF}

\subsubsection{Regression}

\begin{table}[b]
\footnotesize
\centering
\caption[Error metrics in regression problems]{Error metrics in regression tasks for a dataset of $n$ samples, where $y_i$ is the target corresponding to the features $\mathbf{x}_i$, $\hat{y}_i$ is the prediction, $\bar{y}$ is the mean of $y_i$ and $\epsilon$ is a small positive number to avoid undefined MAPEs.} %  ($\bar{y}=\frac{1}{n} \sum_{i=0}^{n-1} y_{i}$)
\label{tab:error_metrics}
% \resizebox{\textwidth}{!}{%
\begin{tabular}{lllcl}
\hline
\textbf{Metric} & & \textbf{Acronym} & & \textbf{Definition} \\ \hline
&& &&\\[-2ex]
\begin{tabular}[c]{@{}l@{}}R$^2$-coefficient\\ of determination\end{tabular} & & \textit{R$^2$} &=& $\displaystyle 1-\frac{\sum_{i=0}^{n-1}\left(y_{i}-\hat{y}_{i}\right)^{2}}{\sum_{i=0}^{n-1}\left(y_{i}-\bar{y}\right)^{2}}$ \\[3ex]
\begin{tabular}[c]{@{}l@{}}Mean squared \\ error\end{tabular} &  & \textit{MSE} &=& $\displaystyle \frac{1}{n} \sum_{i=0}^{n-1}\left( y_i - \hat{y}_i \right)^2$ \\[3ex]
\begin{tabular}[c]{@{}l@{}}Root mean \\ squared error\end{tabular} & & \textit{RMSE} &=& $\displaystyle \left(\frac{1}{n} \sum_{i=0}^{n-1}\left( y_i - \hat{y}_i \right)^2\right)^{1/2}$ \\[3ex]
\begin{tabular}[c]{@{}l@{}}Mean absolute\\ error\end{tabular} & & \textit{MAE} &=& $\displaystyle \frac{1}{n} \sum_{i=0}^{n-1}\left| y_i - \hat{y}_i \right| $ \\[3ex]
\begin{tabular}[c]{@{}l@{}}Mean absolute \\ percentage error\end{tabular} & & \textit{MAPE} &=& $\displaystyle \frac{1}{n} \sum_{i=0}^{n-1} \frac{\left| y_i - \hat{y}_i \right|}{\max(\epsilon,y_i)}$ \\[3ex]
Mean bias error & & \textit{MBE} &=& $\displaystyle \frac{1}{n} \sum_{i=0}^{n-1}\left( y_i - \hat{y}_i \right) $\\[3ex]
\begin{tabular}[c]{@{}l@{}}Mean squared\\ logarithmic error\end{tabular} & & \textit{logMSE} &=& $\displaystyle \frac{1}{n} \sum_{i=0}^{n-1}\left( \log(1 + y_i) - \log(1 + \hat{y}_i) \right)^2$ \\[2ex] \hline
\end{tabular}
% }
\end{table}

Regression algorithms are used to predict continuous variables, such as solar radiation or shallow geothermal heat potential.
During model training, the (internal) parameters of the regression algorithms are optimized with the aim to minimise the error between the continuous targets and the model predictions.
This error (also referred to as the loss function) is often defined as the mean squared error (MSE), i.e. the mean of the squared difference between the predictions and the targets.
%
For tuning and performance evaluation, the error metrics shown in Table~\ref{tab:error_metrics} are used.
These are (i) the R$^2$-coefficient of determination, measuring the goodness of fit between predictions and targets, (ii) the MSE, (iii) the root-mean-squared error (RMSE), which has the same unit as the target, (iv) the mean average error (MAE), (v) the mean average percentage error (MAPE), (vi) the mean bias error (MBE), and (vii) the mean squared logarithmic error (logMSE), which is useful for targets with exponential growth \cite{pedregosa_scikit-learn:_2011}.

\subsubsection{Classification}
Classification algorithms are used for predicting discrete targets, either in a binary format (0 or 1) or as multiple classes, for example to differentiate between roof shapes \cite{mohajeri_city-scale_2018}.
The discrete nature of the targets requires different metrics to assess the prediction results.
In classification tasks, common loss functions used in the training procedure include cross-entropy, also known as negative log-loss \cite{bishop_pattern_2006}, and the gini impurity \cite{breiman_classification_1984}, among others. These measures are designed to maximise the likelihood of a correct classification result, whereby the gini impurity is primarily used in classification trees.

For model tuning and performance evaluation, the number of correct and incorrect predictions is evaluated and displayed in a so-called confusion matrix, shown in Table~\ref{tab:conf_matrix} for the case of binary classification. From this confusion matrix, several evaluation scores can be derived. These include (i) the overall accuracy (OA), namely the total number of correct classifications over the total number of samples, (ii) the precision, namely the correct predictions over the total predictions of a given class, (iii) the recall, i.e. the correct predictions over all true labels in a given class, and (iv) the f1-score, which is the weighted average of precision and recall ($2*\text{\textit{precision}}*\text{\textit{recall}}/(\text{\textit{precision}}+\text{\textit{recall}})$).
Assessing the ensemble of these error metrics is particularly relevant for cases of imbalanced classification, where the number of samples in one class is very small compared to the other class.

\begin{table}[hb]
\footnotesize
\centering
\caption[Structure of confusion matrix for binary classification and evaluation metrics]{Structure of confusion matrix for binary classification and evaluation metrics derived from the confusion matrix, namely precision, recall and accuracy (italic font), with $\sum = TN+TP+FN+FP$.}
\label{tab:conf_matrix}
\begin{tabular}{llccc}
\hline
 &  & \multicolumn{2}{c}{\textbf{Predicted class}} & \multirow{2}{*}{\textit{\textbf{Recall}}} \\
 &  & \textbf{0} & \textbf{1} &  \\ \hline
\multirow{2}{*}{\textbf{True class}} & \textbf{0} & True negatives (TN) & False positives (FP) & $TN/(TN + FP)$ \\
 & \textbf{1} & False negatives (FN) & True positives (TP) & $TP/(TP + FN)$ \\ \hline
\textit{\textbf{Precision}} & \textit{} & $TN/(TN + FN)$ & $TP/(TP + FP)$ & \textit{\textbf{OA} $= TP+TN/\sum$} \\ \hline
\end{tabular}
\end{table}

\subsubsection{ML model architectures}

Throughout this work, I use and compare the performance of six supervised ML models from different families of algorithms: (i) Linear Regression (LIN), (ii) K-Nearest Neighbors (KNN), (iii) Support Vector Machines (SVM), (iv) Random Forests (RF), Artificial Neural Networks (ANN), and (iv) Extreme Learning Machine Ensembles (ELM-E). The description of the algorithms provided below focuses on regression problems, which dominate in the use of ML throughout this thesis. All algorithms can however be equally applied to classification problems. The architecture of the six considered algorithms is shown in Fig. \ref{fig:ml_algorithms}.

\begin{figure}[tb] % !
\centering
\makebox[\linewidth][c]{
\begin{subfigure}[t]{.3\textwidth}
  \centering
  \fbox{\includegraphics[width=.9\linewidth]{images/Figs/lin.pdf} }
  \subcaption{\textbf{LIN}}
  \label{fig:lin}
\end{subfigure}
\begin{subfigure}[t]{.35\textwidth}
  \centering
  \includegraphics[width=.9\linewidth]{images/Figs/SVM.pdf}
  \subcaption{\textbf{SVM}}
  \label{fig:svm}
\end{subfigure}
\begin{subfigure}[t]{.35\textwidth}
  \centering
  \includegraphics[width=.95\linewidth]{images/Figs/ANN.pdf}
  \subcaption{\textbf{ANN}}
  \label{fig:ann}
\end{subfigure}
} \\ \vspace{.25cm}
\makebox[\linewidth][c]{
\begin{subfigure}[t]{.3\textwidth}
  \centering
  \fbox{\includegraphics[width=.9\linewidth]{images/Figs/knn.pdf} }
  \subcaption{\textbf{KNN}}
  \label{fig:knn}
\end{subfigure}
\begin{subfigure}[t]{.35\textwidth}
  \centering
  \includegraphics[width=.95\linewidth]{images/Figs/RF.pdf}
  \subcaption{\textbf{RF}}
  \label{fig:rf}
\end{subfigure}
\begin{subfigure}[t]{.35\textwidth}
  \centering
  \includegraphics[width=.95\linewidth]{images/Figs/ELM_E.pdf}
  \subcaption{\textbf{ELM-E}}
  \label{fig:elme}
\end{subfigure}
}
\caption[Visualisation of ML algorithms used in this work]{Visualisation of ML algorithms used in this work. The selected algorithms are (a) Linear regression (LIN), (b) K-Nearest Neighbor Regression (KNN), (c) Support Vector Machine (SVM), (d) Random Forest (RF), (e) Artificial Neural Network (ANN), (f) Extreme Learning Machine Ensemble (ELM-E).}
\label{fig:ml_algorithms}
\end{figure}

\textbf{Linear Regression }(Fig. \ref{fig:lin}) assumes that the target is a linear function of the inputs. The prediction is obtained from the linear combination of the features which minimizes the residual sum of squares between the target and predicted values \cite{scikit-learn_developers_11_2020}. It is fast and requires no tuning of hyper-parameters but shows a low accuracy for non-linear problems. The equivalent classification approach for the linear regression is logistic regression, which estimates the probability of a data sample to belong to a specific class.

\textbf{K-nearest Neighbor Regression} (Fig. \ref{fig:knn}) is an interpolation algorithm, which computes a prediction as the average of the targets of the $k$ training samples whose features are closest to the given inputs. In classification, the average is replaced by a "majority voting" of the $k$ neighbours \cite{scikit-learn_developers_16_2020}. The training dataset works as a look-up table for the predictions, which makes it effective for low-dimensional problems but inefficient for large datasets. I use the Euclidean distance as a measure of “closeness” and tune the number of neighbors ($k_{nn}$).

\textbf{Support Vector Machine} (Fig. \ref{fig:svm}), introduced by \citet{cortes_support-vector_1995}, is the most popular algorithm in the family of kernel methods. It exploits the \textit{kernel trick}, which projects the features to a higher-dimensional space that allows for linear modelling. Its structure makes it particularly effective for high-dimensional problems, but it does not scale well with the number of samples. In this work, I use $\varepsilon$-Support Vector Regression with a radial basis function as kernel and tune the kernel coefficient ($\gamma$), the penalty parameter (C) and the error tolerance ($\varepsilon$).

\textbf{Random Forest} (Fig. \ref{fig:rf}) is an ensemble (i.e. an aggregation) of decision trees, which was proposed by \citet{breiman_random_2001}. Each of the decision trees in the ensemble pass a training sample along a set of nodes based on a threshold (defined during training) until a leaf node is reached. The prediction of each tree is obtained by averaging (or majority voting) the target values in the respective leaves. It is a popular algorithm due to its good predictive power and high robustness. Its main hyper-parameters are the number of features considered for the optimization of each threshold ($m_\mathit{ftrs}$), the minimum number of samples in each leaf ($m_\mathit{leaf}$) and the number of trees in the ensemble ($n_{est}$).

\textbf{Artificial Neural Networks} (Fig. \ref{fig:ann}) consist of multiple layers of neurons, which are connected to the previous and the next layer through edges \cite{rumelhart_learning_1986}. From the input layer, which contains one neuron for each input, the data samples are passed through one or more hidden layers to the output layer with one neuron for each output. Each neuron computes a weighted linear combination of all nodes of the previous layer (for fully connected networks), which is then passed through a non-linear activation function. The weights within the network are optimised using a backpropagation procedure \cite{lecun_efficient_2012}. The hyperparameters of the ANN include the network architecture, namely the number of hidden layers and the number of neurons in each layer, the selected nonlinear activation function and a regularisation term.

\textbf{Extreme Learning Machine Ensemble }(Fig. \ref{fig:elme}) is a collection of single-layer neural networks (ELMs), which were developed by \citet{huang_extreme_2006}. Each ELM consists of a single hidden layer, which is trained a more efficient way than traditional neural networks (see above) by a single-step optimization using a least-squares approach. This avoids the need for back-propagation, resulting in a faster training time and a low number of hyper-parameters. The aggregation of $n$ ELMs in an ensemble further increases the robustness of the model, reduces the risk of overfitting and enables the estimation of uncertainties \cite{huang_trends_2015, guignard_uncertainty_2021}.
The main hyper-parameters to tune in the ELM-E are the size of the hidden layer ($m_{ELM}$) and the number of ELMs in the ensemble ($n_{ELM}$). Throughout this work, I use a sigmoid activation function, which is a common choice for regression problems \cite{huang_trends_2015}.

\textbf{ML ensemble models}, like the RF and the ELM-E, have a specific architecture which increases the robustness of these models and enables the estimation of uncertainty, as described in Section~\ref{unc_ML}. The ML ensembles are aggregates of ML models (e.g. decision trees or ELMs) - referred to as "ensemble members" - which are trained by applying a bootstrap-aggregating (bagging) approach \cite{breiman_bagging_1996}.
The bagging approach introduces randomness into the model, by training each ensemble member on one bootstrapped resample of the training data.
Each bootstrap replicate is obtained by resampling the $N$ training samples $N$ times uniformly and with replacement.
On average, every such resample contains 63.2\% of the original data, with some duplicated samples \cite{breiman_bagging_1996}.
The ensemble prediction is obtained by averaging the predictions of each ensemble member (regression) or by majority voting (classification).

The ML implementations of the \texttt{scikit-learn} library for python \cite{pedregosa_scikit-learn:_2011} are used for all models except the ELM-E. As no functional open-source code for the ELM-E was available when this work was carried out, I have implemented the ELM-E myself using the ML ensemble architecture described above (regression only). For this, I use the ELM model of the \texttt{HPELM} package for python \cite{akusok_high-performance_2015}, which supports acceleration on graphical processing units (GPUs).

\section{Uncertainties}
\subsection{Uncertainty estimation for ensemble models}
\label{unc_ML}

The architecture of ML ensemble models, described in Section~\ref{ML_models}, permits the estimation of the uncertainty arising from the modelling process (the model uncertainty, $\hat{\sigma}_M$).
The model uncertainty is quantified as the standard deviation of the predictions of all models in an ensemble, which is computed as \cite{heskes_practical_1997}:

\begin{equation}
\label{eq:model_unc}
  \hat{\sigma}_M^2 (\mathbf{x}_i) = \frac{1}{N} \sum_{n=1}^N (\hat{y}_i^n - \hat{y}_i)^2, \quad i=1,...,L
\end{equation}

where $\hat{\sigma}_M$ denotes the model standard deviation, referred to as model uncertainty, $N$ denotes the ensemble length, $\hat{y}_i$ is the ensemble prediction for sample $\mathbf{x}_i$ and $\hat{y}_i^n$ is the prediction of the $n$-th ensemble member.

The remaining residuals after the subtraction of the model variance are further used to estimate the uncertainty related to the data noise (the data uncertainty, $\hat{\sigma}_D$) \cite{akusok_per-sample_2019,guignard_uncertainty_2021}.
They are derived by subtracting the model variance from the squared difference between $\hat{y}_i$ and the targets $y_i$, as shown in Eq.~\eqref{eq:data_unc} below.
In order to reduce bias, the out-of-bag (OOB) model output is used, which is the mean of the ensemble members for which a given training data point was not considered (due to bootstrapping) \cite{heskes_practical_1997}.

\begin{equation}
\label{eq:data_unc}
  \hat{\sigma}_D^2 (\mathbf{x}_i) = \max \left\{ (y_i - \hat{y}_i)^2 - \hat{\sigma}_M^2 (\mathbf{x}_i) , 0 \right\}, \quad i=1,...,L
\end{equation}

 where $\hat{\sigma}_D$ denotes the standard deviation of data noise, referred to as data uncertainty, and $y_i$ is the target value for sample $\mathbf{x}_i$.
$\hat{\sigma}_D$ can only be computed if $y_i$ is known. Thus, a second ML model is trained from the residuals to predict $\hat{\sigma}_D$ for points with unknown targets. Unless mentioned otherwise, the same hyper-parameters are used in both models.

The total variance of a variable that is estimated using a ML model is the squared sum of its model uncertainty and its data uncertainty:

\begin{equation}
\label{eq:total_unc}
\sigma^2 (\mathbf{x}_i) = \hat{\sigma}_M^2 (\mathbf{x}_i) + \hat{\sigma}_D^2 (\mathbf{x}_i), \quad i=1,...,L
\end{equation}
This approach to estimate both the model and the data uncertainty enables a specific characterization of the uncertainty for each spatio-temporal point. It has been previously used to estimate for example wind speeds for wind power generation~\cite{wan_probabilistic_2014}.

\subsection{Uncertainty propagation}
\label{method_unc_prop}

For the estimation of renewable resource potential, different sources of uncertainties must be combined and propagated through physicals models.
For this propagation of uncertainties, the correlation among the random variables (RVs) plays a key role.
%
If statistical independence can be assumed, the mean and variance of the resulting variables can be computed from the mean and variance of each parameter only, which greatly simplifies the uncertainty propagation.
However, this assumption does not always hold, in which case correlations between RVs must be taken into account.
In the context of this thesis, the propagation of uncertainties over a set of sums and products of RVs is considered using error propagation theory \cite{jcgm_evaluation_2008, goodman_variance_1962}. This section summarises the error propagation formulas for summations and multiplications, applied in the following chapters, in their general form.

In error analysis, the variance of the sum of $N$ randomly distributed variables $X_i$ is obtained from the variances of each RV and their pairwise covariances, such that (cf. \cite{jcgm_evaluation_2008}):

\begin{equation}
\label{eq:unc_sum}
\mathrm{Var}\left(\sum_{i=1}^N X_i\right) = \sum_{i=1}^N\mathrm{Var}(X_i) + 2\sum_{i < j}\mathrm{Cov}(X_i, X_j)
\end{equation}
which simplifies to the sum of the variances of $X_i$ if the RVs are statistically independent.
%
The variance of the product of two RVs $X$ and $Y$ is defined as (cf. \cite{goodman_variance_1962}):

\begin{align}
\label{eq:var_mult_corr}
\mathrm{Var}(X Y)&=\mathrm{E}\left(X^{2} Y^{2}\right)-\mathrm{E}(X Y)^{2} \\
&= \mathrm{Cov}\left(X^{2}, Y^{2}\right)+\left[\mathrm{Var}(X)+\mathrm{E}(X)^{2}\right] \left[\mathrm{Var}(Y)+\mathrm{E}(Y)^{2}\right]-[\mathrm{Cov}\left(X, Y\right)+\mathrm{E}(X)  \mathrm{E}(Y)]^{2} \nonumber
\end{align}

If statistical independence between $X$ and $Y$ can be assumed, the covariance terms $\mathrm{Cov}\left(X^{2}, Y^{2}\right)$and $\mathrm{Cov}\left(X, Y\right)$ can be omitted, such that Eq.~\eqref{eq:var_mult_corr} becomes a function of the variances of $X$ and $Y$ and their expected value (cf. \cite{goodman_variance_1962}):
\begin{equation}
\label{eq:unc_prod}
\mathrm{Var}(XY) = \mathrm{Var}(X)\mathrm{Var}(Y) + \mathrm{Var}(X) \mathrm{E}(Y)^2 + \mathrm{Var}(Y) \mathrm{E}(X)^2
\end{equation}