% !TeX root = ../main.tex
% Add the above to each chapter to make compiling the PDF easier in some editors.

\chapter{Functional B-trees in Isabelle}\label{chapter:abs-set}

Proving higher level properties of data structures
tends to be easier on a functional level than on an imperative
level.
This is mostly due to the fact that many details of implementation
can be abstracted away or expressed in a simpler manner.
The work therefore begins with a functional specification of B-trees
in HOL that is not aware of the existence of heaps and uses
persistent data structures.

\section{Basic Definitions}
\label{sec:basic-defs}
%Definition used for this implementation.
%(esp. order)
%note that k refers to keys+subtrees rather than only keys/subtrees (false! we have k pairs)

% TODO all function equations similar to math equations, setting variables in italics

As discussed in \Cref{chapter:introduction},
we define B-trees recursively to either
be a \textit{Node}, comprising a list
of subtrees and keys or to be a \textit{Leaf}
that contains no information.
The only room for interpretation is how to actually
store the subtrees and separators.
For trees of variable but constantly bounded size,
an explicit constructor for each possible size may be given.
An example may be found for the 234-trees analysed by Nipkow\parencite{DBLP:conf/itp/Nipkow16},
where the number of subtrees is maximally 5.
However in B-trees, the number of subtrees
is bounded only by $k$, which is neither constant
nor bounded by a constant.

One possible solution is to make nodes
extensible by definition.
That is, instead of whole nodes,
node segments are defined that form a node together.
This resembles the definition of linked lists,
just integrating the shape into the node definition.
The result is that a node segment either has one subtree,
one element and a pointer to a subsequent segment of the node or
it is the last entry and has only one subtree.
This approach was implemented by Ernst \parencite{DBLP:journals/sosym/ErnstSR15}.
However, from our point of view, this approach has a number of downsides.
First of, all operations on the tree were defined co-recursively,
which adds a layer of complexity to all proofs.
Further, general lemmas for lists or similar data types can not
be used directly for this structure and would need to be transferred manually
or newly derived.

Therefore, we use an intermediate data structure to store the subtrees.
This approach is called nested recursive definition
and will become clearer in the actual code in \Cref{lst:btree-def}.
The intuitive idea is to choose sets as an abstract
data type to hold subtrees and keys, as was the first step by Fielding \parencite{Fielding80}.
However here we are limited as recursive nested definitions
only make sense for finitely bounded subtypes.\footnote{
    More information on this topic and potential remedies
    may be found in the tutorial to data type definition in Isabelle
    \parencite{isabelleDatatypeManual}.
}
Sets themselves are potentially of infinite cardinality
and hence not bounded and working with finite sets
introduces additional complexity that we want
to circumvent in the abstract definition.

Therefore we resort to the list data type,
defining each node to hold a list of key-node pairs.
By storing subtrees and keys next to each other,
properties that relate subtrees to their immediately
neighboring separators (i.e. sortedness)
can be defined locally on each pair in the list.
This also implicitly enforces the relationship of the number of subtrees
and separators and makes the explicit use of length and index relationships unnecessary.

A similar approach was chosen by Malecha \parencite{DBLP:conf/popl/MalechaMSW10}.
There, the B-tree data type was defined directly on the heap
and comprised the height of the tree, a pointer to the last subtree and
an array of fixed size that contains pairs of values and pointers.
We will store subtrees and keys in pairs too,
however, with a different in-pair order,
such that the subtree to the left of a key is the first element
in a tuple and the key is the second element,
as visualized in \Cref{fig:btree-basic}.\footnote{
    Writing this down in code or lemmas,
    this also appears as "subtree to the left on the left side of the tuple".
}
Moreover in a functional context, we will not make use of arrays
but rather resort to lists.
In \parencite{DBLP:conf/popl/MalechaMSW10} the height attribute was
used to inductively define a shape predicate that is only valid
if the height decreases for subtrees.
However, the finite size of the abstract tree is an implicit
result of its recursive definition.
We therefore also do not explicitly store the height of the tree.

\begin{lstlisting}[mathescape=true, language=Isabelle,label=lst:btree-def]
datatype 'a btree = Leaf | Node (( 'a btree * 'a ) list) ('a btree)
\end{lstlisting}

\begin{figure}
    \centering
    \includegraphics[width=0.5\linewidth]{figures/btree-basic.pdf}
    \caption[Visualization of the implementation choice for B-Trees]
    {The B-tree from figure \Cref{fig:btree-basic-nopair} as concretely specified.
    Subtrees and separators that lie in pairs in the implementation are surrounded by parentheses.}
    \label{fig:btree-basic}
\end{figure}

As said, this definition makes it easy to relate any separator
to the subtree to its left, as this is subtree in the same tuple.
If we need to access the subtree to the right,
we may obtain the head of the remaining list by pattern matching
and take the subtree of the first tuple.
In case the remaining list is empty,
we need to choose the last tree.

The balancedness of a B-tree is defined recursively, making use
of an intuitively defined height.\footnote{
    The operator $`$ denotes the image of all elements of a set for a given function.
    In this case we map all subtrees to their respective height.
}
The height of all subtrees must be equal.
However it is not completely intuitive
what would be the best way to express this property.
Either we fix the height of all trees
to some existentially quantified number or to a known value.
Since we know the last tree in the node exists and further
know that if all subtrees of the node have the same height,
then their height is equal to the last tree,
we simply choose the height of the last tree as an anchor point.

\begin{lstlisting}[mathescape=true, language=Isabelle]

fun height :: 'a btree $\Rightarrow$ nat where
  height Leaf = 0 |
  height (Node ts t) = 1 + Max (height ` (set (subtrees ts@[t]))

fun bal :: 'a btree $\Rightarrow$ bool where
    bal Leaf = True |
    bal (Node ts t) = (
        ($\forall$sub $\in$ set (subtrees ts). height sub = height t) $\wedge$
        ($\forall$sub $\in$ set (subtrees ts). bal sub) $\wedge$ bal t
    )
\end{lstlisting}

Another option, demonstrated by Fielding, is
to define balancedness as defined by Bayer
to be that all paths from the root to the node are of equal length,
or that the set of lengths of all paths be of size 1.
However reasoning on set cardinality is likely to be more difficult
in a mechanized proof than on paper
and the definition for the mechanized proof by Ernst resembles
ours more closely, so we did not experiment much with this option.

Further the order of the trees needs to be formally defined.
As discussed in \Cref{sec:data_structure_defs}, the most useful choice here is
to allow for at least $k$ and at most $2k+1$ subtrees.
Since the last subtree is fixed, and subtrees and children are residing as pairs in the same list,
we simply require the length of that list to be between $k$ and $2k$.
Note that we also need a special property $order^r$ for the root of the tree,
which has between one and $2k$ elements.
In mathematical equations, we will denote "order $k$ $t$" as "$\order_k t$"
for convenience (likewise for "order$^r$ $k$ $t$"), which is to be read as
"tree $t$ is of order $k$".

\begin{lstlisting}[mathescape=true, language=Isabelle]

fun order :: nat $\Rightarrow$ 'a btree $\Rightarrow$ bool where
    order k Leaf = True |
    order k (Node ts t) = (
        (length ts $\ge$ k)  $\wedge$
        (length ts $\le$ 2*k) $\wedge$
        ($\forall$sub $\in$ set (subtrees ts). order k sub) $\wedge$ order k t
    )

fun order$^r$ where
    order$^r$ k Leaf = True |
    order$^r$ k (Node ts t) = (
        (length ts $>$ 0) $\wedge$
        (length ts $\le$ 2*k) $\wedge$
        ($\forall$s $\in$ set (subtrees ts). order k s) $\wedge$ order k t
    )

\end{lstlisting}

We define the sortedness of a B-tree based on the \textit{inorder} of the tree,
which is the catenation of all elements in the tree in in-order traversal.
An example can be seen in \Cref{fig:inorder-basic}.
The library function $concat$ on lists of lists
is employed here to express catenation of
the inorder of subtrees and the separators.
This definition reads a bit inconveniently
as the node internal list is first mapped and then concatenated.
However since we make recursive use of the inorder-function
inside the mapping expression, mapping the subtree list is inconvenient to externalize.
We only later cover up this expression by using abbreviations.
In the following, we will not distinguish between lists,
pairs or trees when talking about their inorder representation.
The function with the correct type from the below listing is meant instead.

\begin{lstlisting}[mathescape=true, language=Isabelle]
fun inorder :: 'a btree $\Rightarrow$ 'a list where
    inorder Leaf = [] |
    inorder (Node ts t) =
        concat (map ($\lambda$ (sub, sep). inorder sub @ [sep]) ts) @ inorder t

abbreviation inorder_pair  $\equiv$ $\lambda$(sub,sep). inorder sub @ [sep]
abbreviation inorder_list ts $\equiv$ concat (map inorder_pair ts)
\end{lstlisting}

\begin{figure}
    \centering
    \includegraphics[width=0.3\linewidth]{figures/inorder-basic.pdf}
    \caption[Example inorder of a B-Tree]
    {The inorder view on the B-tree from figure \Cref{fig:btree-basic}.}
    \label{fig:inorder-basic}
\end{figure}

That way we can express sortedness of the tree $t$ as a simple $\sorted (\inorder t)$,
where $sorted$ is the property of being sorted strictly (with resprect to $<$)
in ascending order.
This definition is very compact and brings forward another benefit
pointed out by Nipkow \parencite{DBLP:conf/itp/Nipkow16}:

Many properties of search trees follow intuitively by considering
the inorder view on the tree.
Often, it is supposed to be invariant (i.e. stealing from the neighbor node to the right)
or only deviates in a manner we expect it to deviate given the inorder view
(i.e. insert an element at the correct position).
We will see later how the use of this fact comes in handy for proving
important properties of the implementation.

Since our B-tree definition really only makes sense for positive $k$,
we obtain the following overall invariant for B-trees.

\begin{definition}
    \label{def:btree-def}
    $k > 0 \Longrightarrow \btree_k t = \bal t \wedge \order^r_k t \wedge \sorted (\inorder t)$
\end{definition}

All trees that satisfy this invariant have a very small
height considering the number of inserted elements.
This fact is examined closer in the following section.
 % TODO note benefit over btree_sorted

\section{Height of B-Trees}

%Height etc.
As pointed out by Bayer \parencite{DBLP:journals/acta/BayerM72},
the height of B-trees is logarithmic with respect to the number
of nodes of the tree.
The original paper even gives a precise lower and upper bound,
the proof of which we will quickly sketch in the following.

First, we define the number of nodes in a tree:
\begin{lstlisting}[mathescape=true, language=Isabelle]
fun nodes :: 'a btree $\Rightarrow$ nat where
    nodes Leaf = 0 |
    nodes (Node ts t) =
        1 + $\sum$t$\leftarrow$subtrees ts. nodes t + nodes t
\end{lstlisting}

%To work with this equation, we just need to add one auxiliary lemmas to the standard canon
%\begin{lemma}
%\begin{lstlisting}[mathescape=true, language=Isabelle]
%    sum_list (replicate n c) = n*c
%\end{lstlisting}
%\end{lemma}

We obtain bounds
on the number of nodes of a subtree with respect to its height
by induction on the computation of the nodes function.

\begin{lemma}
    \label{lem:bound_internal_node}
    $\order_k t \wedge \bal t \longrightarrow$
    \begin{align}
        (k+1)^{\height t} - 1 &\le \nodes t * k \\
        \nodes t * 2k &\le (2k+1)^{\height t} - 1
    \end{align}
\end{lemma}

From \Cref{lem:bound_internal_node} we can almost directly obtain
the bounds on valid roots of B-trees.
The only difference to the bound of internal nodes occurs on the lower bound side.
The issue here is that a root node may contain less elements than
a valid internal node (namely only one), which yields two subtrees with known height
plus one for the node itself.
Note that these are the exact same bounds as obtained by Bayer \parencite{DBLP:journals/acta/BayerM72}\footnote{
    The operation $div$ denotes integer division here.
    It does not make much difference though as all terms are divisible
    without remainder.
},
except for the fact that we have generalized the equation,
incorporating whether $t$ is a tree or not.\footnote{
    If $t$ is a \textit{Leaf},
    $2((k+1)^{\height t - 1} - 1) \div k + (t \neq Leaf)$ becomes a fancy way of writing $0$.
}

\begin{theorem}
    \label{thm:bound_btree}
    $\rootorder_k t \wedge \bal t \wedge k > 0 \longrightarrow$
    \begin{equation}
        2((k+1)^{\height t - 1} - 1) \div k + (t \neq Leaf) \le \nodes t \le ((2k+1)^{\height t} - 1) \div 2k
    \end{equation}
\end{theorem}

These results are very interesting, because
the runtime of all further operations will be
directly proportional to the height of the tree.
Therefore, we are glad to see that the height is logarithmic
with respect to the number of nodes stored in the tree.

These bounds are sharp.
We prove this by providing in \Cref{lst:sharp-trees-def}.
functions that generate exactly those trees
that satisfy the requirements of B-trees, have a given height
and satisfy the inequality from \Cref{thm:bound_btree}
with equality.
As might be expected, these trees are simply those
that are minimally or maximally filled in each node with respect
to the order property.

%TODO (?) images of slim/full trees
\begin{figure}
\begin{lstlisting}[mathescape=true, language=Isabelle,label={lst:sharp-trees-def},
    caption={The functions generating trees with minimal size and maximal size for given height.}]

fun full_node where
  full_node k c 0 = Leaf |
  full_node k c (Suc n) = (
      Node
        (replicate (2*k) ((full_node k c n),c))
        (full_node k c n)
  )

fun slim_node where
  slim_node k c 0 = Leaf |
  slim_node k c (Suc n) = (
      Node
        (replicate k ((slim_node k c n),c))
        (slim_node k c n)
    )

definition full_tree = full_node

fun slim_tree where
  slim_tree k c 0 = Leaf |
  slim_tree k c (Suc h) =
    Node
        [(slim_node k c h, c)]
        (slim_node k c h)

\end{lstlisting}
\end{figure}

The proof for internal nodes follows by induction over the creation.

\begin{lemma} $t_f := \fullnode k\ a\ h \wedge t_s := \slimnode k\ a\ h \longrightarrow$
    \begin{align}
    h = \height t_s = \height t_f \wedge \\
    ((2k+1)^h - 1) &= \nodes t_f * (2k) &\wedge \order_k t_f \wedge \bal t_f \\
    ((k+1)^h - 1) &= \nodes t_s * k  &\wedge \order_k t_s \wedge \bal t_s
    \end{align}
\end{lemma}

The rule for the roots follows directly, making use of the lemma
for the internal nodes.
Note how for the root node the result is simply two times the
value for trees of one height less, which are the minimally
required two subtrees.

% TODO prove div version of thm
\begin{theorem}
    $k > 0 \wedge t_f = \fulltree k\ a\ h \wedge t_s = \slimtree k\ a\ h \longrightarrow$
    \begin{align*}
    h = \height t_s &= \height t_f &\wedge \\
        ((2k+1)^h - 1) \div 2k &= \nodes t_f &\wedge \rootorder_k t_f \wedge \bal t_f \\
        2((k+1)^{h-1} - 1) \div k + (t_s \neq Leaf) &= \nodes t_s &\wedge \rootorder_k t_s \wedge \bal t_s
    \end{align*}
\end{theorem}

\section{Set operations}

We say that B-trees store linearly ordered values
that could be stored in an abstract set,
and the that a B-tree is a concretization
of this set.
With this specification of B-trees we need to provide functions that allow
set-like operations on the trees -
membership queries, insertion and deletion.
In the Isabelle/HOL framework, there is a standard interface
for data structures that provide an implementation of sets.

\subsection{The Set Interface}
% Description of the set interface

As described by Nipkow \parencite{DBLP:conf/itp/Nipkow16},
an implementation \textit{'a t} of sets of elements of type \textit{'a} is required to provide the following
operations:

\begin{itemize}
    \itshape
    \item set :: 'a t $\Rightarrow$ 'a set
    \item inv :: 'a t $\Rightarrow$ bool
    \item empty :: 'a t \textup{\textbf{where}} set empty = $\emptyset$
    \item isin :: 'a t $\Rightarrow$ 'a $\Rightarrow$ bool \textup{\textbf{where}} inv t $\Longrightarrow$ isin x t = x $\in$ set t
    \item insert :: 'a $\Rightarrow$ 'a t $\Rightarrow$ 'a t \textup{\textbf{where}} inv t $\Longrightarrow$ set (insert t x) = set t $\cup$ {x}
    \item delete :: 'a $\Rightarrow$ 'a t $\Rightarrow$ 'a t \textup{\textbf{where}} inv t $\Longrightarrow$ set (delete t x) = set t $\backslash$ {x}
\end{itemize}

For this work, using the definition from \Cref{lst:btree-def},
we consider \textit{'a t = 'a btree}.
The standard approach is to provide functions on B-tree and show
that they have the same effect as membership test, insertion and deletion
in abstract sets with the same elements (specified above after "where").
Further, the invariant needs to remain valid for the results of the operations
as well as the $empty$ element (i.e. $inv\ t \Longrightarrow inv\ (insert\ x\ t)$).

We know from \Cref{sec:basic-defs} that one of the invariants
of the tree is that it is always sorted.
While this is expressed in a somewhat cumbersome manner
in the original definition by Bayer \parencite{DBLP:journals/acta/BayerM72},
this requirement is nothing else but a sortedness of the inorder view of the tree.
Knowing this, we resort to a specialized set interface
proposed by Nipkow \parencite{DBLP:conf/itp/Nipkow16}.
Their approach yielded automatic proofs for 2-3-trees
and 234-trees, which are specializations of B-trees.
The modified Set interface reasons based on the \textit{inorder}
view on the tree instead of the \textit{set} abstraction.

\begin{itemize}
    \itshape
    \item inorder :: 'a t $\Rightarrow$ 'a list
    \item inv :: 'a t $\Rightarrow$ bool
    \item empty :: 'a t \textup{\textbf{where}} inorder empty = []
    \item isin :: 'a t $\Rightarrow$ 'a $\Rightarrow$ bool \textup{\textbf{where}}\\
     inv t $\wedge$ sorted(inorder t) $\Longrightarrow$ isin x t = x $\in$ set (inorder t)
    \item insert :: 'a $\Rightarrow$ 'a t $\Rightarrow$ 'a t \textup{\textbf{where}}\\
     inv t $\wedge$ sorted(inorder t) $\Longrightarrow$ inorder (insert t x) = ins$_{list}$ (inorder t) x
    \item delete :: 'a $\Rightarrow$ 'a t $\Rightarrow$ 'a t \textup{\textbf{where}}\\
     inv t $\wedge$ sorted(inorder t) $\Longrightarrow$ inorder (delete t x) = del$_{list}$ (inorder t) x
\end{itemize}

The invariant and inorder functions for this set interface
follow directly from the definition of B-trees.
Hence, we specify $k > 0 \wedge \order^r_k t \wedge \bal t$
as the invariant of B-trees and use
the concatenation function from \Cref{sec:basic-defs} to specify the
list abstraction \textit{inorder}.
Note that sortedness is not part of the invariant
but will be required for all operations and preserved
due to the definition of $ins_{list}$ and $del_{list}$.
The interface is then implemented by providing set operations
and the proofs of invariant preservation
with respect to this inorder abstraction.

Some parts of the set specification are trivial.
In the following, \textit{Leaf} is an empty tree and hence represents
the empty set, satisfying the first part of the specification.
It can be seen directly that the leafs already satisfy all three
properties of the invariant.
The following sections will describe the implementation
and proofs of the non-trivial set operations.

\subsection{The split-Function}

\begin{figure}

\begin{lstlisting}[mathescape=true, language=Isabelle]
fun linear_split_help where
  linear_split_help [] x prev = (prev, []) |
  linear_split_help ((sub, sep)#xs) x prev = (
      if sep < x then
        linear_split_help xs x (prev @ [(sub, sep)])
      else
        (prev, (sub,sep)#xs)
  )

fun linear_split:: ('a btree$\times$'a) list $\Rightarrow$ 'a $\Rightarrow$ (_ list $\times$ _ list) where
  linear_split xs x = linear_split_help xs x []
\end{lstlisting}
\begin{lemma}
    \begin{equation*}
    \lsplitfun\ xs\ x = (\takewhile (\lambda(\_,s).\ s < x)\ xs,\ \dropwhile (\lambda(\_,s).\ s < x))
    \end{equation*}
\end{lemma}
\caption[An implementation of the abstract split function specifications.]
{An implementation of the abstract split function specifications.
This functions scans linearly through the list, returning the first tuple where the separator
or subtree could potentially contain the value $x$.
It is similar to a combination of library functions, simplifying proofs on its properties.}
\label{fig:linear_split}

\end{figure}
% Description of the implementation of the set interface.

Naturally, the set operations are defined recursively on the nodes of the tree.
Since each node contains a number of elements that is not bounded by a constant,
a generalized function to navigate to the correct separator and subtree
is central to all operations.\footnote{
    Opposing to that, in 234-trees determining the correct subtree
    and separator could be hard coded for every case, as done by Nipkow \parencite{DBLP:conf/itp/Nipkow16}.
}

We call this function \textit{split}-function.
It determines the "correct" position in the list of separators and subtrees
for recursion.
At this position, the range spanned by the subtree to the left
and the separator is exactly the range in which the desired element
must be contained if it is contained in the tree.
Hence, either the separator is equal to the desired value or
we need to recurse into the subtree to the left.
The approach of generalizing the tree navigation has
surprisingly little popularity in the implementations examined.
Notably, it was implemented in the work of Malecha and Fielding,
but not described in detail \parencite{DBLP:conf/popl/MalechaMSW10,Fielding80}.\footnote{
    In Fieldings approach the corresponding function is called \textit{index} and the final implementation is a linear search.
    Malecha calls it \textit{findSubtree} and looses no more words about it.
}
Usually however, it is integrated into the set-operation
by a linear search that is promised to be replaced by a more efficient binary search
in the actual implementation \parencite{DBLP:books/daglib/0023376,DBLP:journals/acta/BayerM72}
or left in the final code \parencite{DBLP:journals/sosym/ErnstSR15}.

The precise inner workings of the split function are not of interest here
and actually are not supposed to be interesting on the functional level.
Of course we need to know that \textit{some} kind of function
exists that correctly splits the key-value list.
An example is given in \Cref{fig:linear_split}.
In the process of implementing the set specifications,
this concrete function was used to explore the provability of the set methods.
However it quickly turned out that 1) only specific lemmas about the split
function are useful during proofs and 2) only relying on an abstract specification of
the split method would simplify integrating alternative splitting functions.
Most notably, the abstraction allows to later plug in an efficient splitting
function, e.g. based on binary search.

\begin{samepage}
Therefore all set functions are defined based on an abstract function
\textit{split} that fulfils the following requirements for a list of subtrees
and separators \textit{xs} and a partitioning value $p$:

\begin{itemize}
    \item $\splitfun xs\ p = (ls,rs) \Longrightarrow xs = ls @ rs$
    \item $\splitfun xs\ p = (ls@[(sub,sep)],rs) \wedge \sorted (\separators xs) \Longrightarrow sep < p$
    \item $\splitfun xs\ p = (ls,(sub,sep)\#rs) \wedge \sorted (\separators xs) \Longrightarrow p \le sep$
\end{itemize}
\end{samepage}

Described in natural language, the split function should return two lists
that concatenate to the original list
(we refer to them further as the \textit{first} and \textit{second} part of the list).
If the elements came in sorted order,
the list is split such that the key to the left is strictly
less than $p$ and the key to the right
should be less or equal than $p$.

Note how the split function really only needs to
consider the separators and not the subtrees themselves.
By requiring only sortedness of the separators,
we weaken the requirements to fulfil the split abstraction.
However these weakened assumptions are sufficient to guarantee
functional correctness.
Sortedness of the whole tree is part of the established invariant of B-trees.
If the whole tree is sorted, the separators are sorted too.
We prove this once and from it follows that
we may use the split function for sorted trees
and obtain the desired results.

\subsection{Membership tests}

%-------------------------------------------------
% TODO move to abstract description?
The simplest operation required in the set interface is
the \textit{isin} function.
It should return the same value for a B-tree
as a membership queries on the set abstraction of the tree.
% Note reference to i.e. Bauer retrieval algorithm
The definition in \Cref{lst:isin-fun} is straightforward
and also shows example usage of the split function.
In case the second part of the split list is non-empty,
we check the element at its head and recurse in the given subtree
if necessary.
Otherwise, we may directly recurse to the last tree in the node.\footnote{
    This function recurses on results of the split function.
    In order to show that this function terminates, we need to show
    the system that the obtained subtrees are of smaller size than the current tree.
    Adding this fact to the default termination simplification set resolves
    the issue for all coming functions.
}
%-------------------------------------------------

\begin{figure}
\begin{lstlisting}[mathescape=true, language=Isabelle, caption=The \textit{isin} function, label=lst:isin-fun]
fun isin :: 'a btree $\Rightarrow$ 'a $\Rightarrow$ bool where
    isin (Leaf) x = False |
    isin (Node ts t) x = (
        case split ts x of (_,(sub,sep)#rs) $\Rightarrow$ (
            if x = sep then
                True
            else
                isin sub x
        ) | (_,[]) $\Rightarrow$
            isin t x
    )
\end{lstlisting}
\end{figure}

By the standard set interface the operation is only required to work on
sorted, balanced trees of a certain order, however only the first property
is actually required for correctness.
The following lemma shows the required property of the function

\begin{theorem}
    \label{thm:isin-set}
    $\sorted (\inorder t) \Longrightarrow \isinfun t\ x = x \in \setfun (\inorder t)$
\end{theorem}

It follows by induction on the evaluation of the isin function.
To prove it, we invoke two specialized lemmas,
that simplify arguments about the choice of the node for recursion.
The lemmas specialize an idea proposed by Nipkow \parencite{DBLP:conf/itp/Nipkow16}
and similar lemmas will be used for the correctness proofs of
the \textit{insert} and \textit{delete} function.

\begin{lemma} $\sorted (\inorder (Node\ ts\ t) \wedge \splitfun ts\ x = (ls, rs) \Longrightarrow$ \\
    \begin{center}
    $x \in \setfun (\inorder (Node\ ts\ t)) = x \in \setfun (\inorder rs @ \inorder t)$
    \end{center}
\end{lemma}

The idea of this fact is to argue that, if the split function has provided
us with a given splitting, it is safe to limit the further search
to the second part of the split.
It follows directly from the requirements on the split function.
If $rs$ is empty, we can follow that the element has to reside in the inorder
of the last tree of the node.
When used in the inductive proof of \Cref{thm:isin-set}, we can then deduce that this is
equal to \textit{isin t x}, the branch taken in the isin-function
(see \Cref{lst:isin-fun}) if the second list in the split result is empty.
If $rs$ is not empty, we need an additional lemma.

\begin{lemma}
    $\sorted (\inorder (Node\ ts\ t)) \wedge \splitfun ts\ x = (ls, (sub,sep)\#rs) \wedge sep \neq x \Longrightarrow$ \\
    \begin{center}
    $x \in \setfun (\inorder ((sub,sep)\#rs) @ \inorder t) = x \in \setfun (\inorder sub)$
    \end{center}
\end{lemma}

With this lemma we know that the first subtree
in the second part of the split result
is the correct child to recurse into.
The requirement of $sep \neq x$ is because if $sep = x$,
no recursion is required at all.
The desired element was just found.

This operation has no effect on the tree, it only walks through it.
This fact follows directly due to the persistence of functional data structures.
Hence, no proof of invariant preservation is required,
in contrast to the following tree-modifying operations.

\subsection{Insertion}
\label{sec:abs-ins}

% TODO what is this for?
%There are not many provably correct functional implementations to our knowledge.
%However there is a relationship between 234-trees and B-trees,
%due to which the approach to the functional definition is inspired by the implementation
%in \parencite{DBLP:conf/itp/Nipkow16}.
%In that work, a correct implementation of the Set interface for 234-trees
%are provided.

The implementation of the insertion function
as described in \Cref{par:intro-ins} is documented in \Cref{lst:ins-fun}.
Rather than manually checking whether the lowest level was reached,
we recurse until we reach a leaf.
Inserting into it is defined to cause an overflow that will be handled
by the lowest internal nodes in the same manner as for other internal nodes.

The $ins$ function is a recursive helper function,
that recurses down into the correct leaf node for insertion.
As a possible intermediate result of insertion
may be two trees that are the result of an obligatory split,
it returns data in a new data type called \textit{up}$_i$.
This data type carries either a singleton valid B-tree or two trees and a separator
that could correctly be part of a node when placed next to each other.
The original subtree is replaced by
the result of the recursive call.
If an overflow occured in the lower node, the additional element and
subtree are inserted into the current node list
after the element that flowed over.
The resulting list may have an overflow itself.

To modularize insertion, the check for the overflow of
the nodes list has been
extracted to the function \textit{node}$_i$.
In case the node that was passed in violates the order invariant,
it performs the splitting and returns two trees and the additional separator.
This construction is useful for handling overflow
in a functional context and appears similarly in the
verified functional 234-tree implementation by Nipkow \parencite{DBLP:conf/itp/Nipkow16}.

Finally, the \textit{insert} function calls the \textit{ins} helper function
and transforms the result into a valid B-tree by allocating a new root if necessary.

\begin{figure}

\begin{lstlisting}[mathescape=true, language=Isabelle, label=lst:ins-fun, caption={
    The \textit{insert} function
}]
datatype 'b up$_i$ = T$_i$ 'b btree | Up$_i$ 'b btree 'b 'b btree

fun split_half where
  split_half xs = (take (length xs div 2) xs, drop (length xs div 2) xs)

fun node$_i$ :: nat $\Rightarrow$ ('a btree $\times$ 'a) list $\Rightarrow$ 'a btree $\Rightarrow$ 'a up$_i$ where
  node$_i$ k ts t = (
  if length ts $\le$ 2*k then T$_i$ (Node ts t)
  else (
    case split_half ts of (ls, (sub,sep)#rs) $\Rightarrow$
      Up$_i$ (Node ls sub) sep (Node rs t)
    )
  )

fun ins :: nat $\Rightarrow$ 'a $\Rightarrow$ 'a btree $\Rightarrow$ 'a up$_i$ where
  ins k x Leaf = (Up$_i$ Leaf x Leaf) |
  ins k x (Node ts t) = (
  case split ts x of
    (ls,(sub,sep)#rs) $\Rightarrow$
      (if sep = x then T$_i$ (Node ts t)
      else
        (case ins k x sub of
          Up$_i$ l a r $\Rightarrow$
            node$_i$ k (ls @ (l,a)#(r,sep)#rs) t |
          T$_i$ a $\Rightarrow$ T$_i$ (Node (ls @ (a,sep) # rs) t))) |
    (ls, []) $\Rightarrow$
      (case ins k x t of
         Up$_i$ l a r $\Rightarrow$
            node$_i$ k (ls@[(l,a)]) r |
         T$_i$ a $\Rightarrow$ T$_i$ (Node ls a)
  )
)

fun tree$_i$ :: 'a up$_i$ $\Rightarrow$ 'a btree where
  tree$_i$ (T$_i$ sub) = sub |
  tree$_i$ (Up$_i$ l a r) = (Node [(l,a)] r)

fun insert::nat $\Rightarrow$ 'a $\Rightarrow$ 'a btree $\Rightarrow$ 'a btree where
  insert k x t = tree$_i$ (ins k x t)
\end{lstlisting}

\end{figure}

\begin{figure}
    \centering
    \includegraphics[width=0.43\linewidth]{figures/btree-basic.pdf}\\
    \vspace*{1cm}
    \includegraphics[width=0.48\linewidth]{figures/btree-basic-ins9.pdf}\\
    \vspace*{1cm}
    \includegraphics[width=0.48\linewidth]{figures/btree-basic-ins9-ins1.pdf}
    \caption[An example B-Tree after two insertion operations]
    {The tree from \Cref{fig:btree-basic} (top) after
    successive insertion of $9$ (middle) and $1$ (bottom).}
    \label{fig:btree-basic-ins}
\end{figure}
%TODO

To fulfil the set interface requirements,
we need to show that this function preserves the invariants
and acts the same as inserting element $x$ into the inorder list of the tree.
Note that for these proofs, a separate notion of height, balancedness, order
and inorder had to be introduced for the \textit{up}$_i$ data type.
Height, balancedness and inorder are
defined such that they are invariant to splitting and merging of nodes.
For example, "$\height ts\ t = \height^{up_i} (\node_i k\ ts\ t)$".
The modified order function simply subsumes that all subtrees have
the given order.
Naturally, making a one-element tree of an up$_i$ element will thus
give a node of root-order, while inserting the elements
will not violate the recursive order requirement.

Proving that balancedness is invariant
under insertion requires an additional lemma,
namely that the height is also invariant under insertion.
Technically, height preservation is only required when modifying balanced
trees, however we have shown the stronger statement that this is even the case for non-balanced trees.
This follows by induction using the
associativity and commutativity of the maximum function.
With height invariance, the proof for balancedness follows much the same way.
Overall, the fact that the operation does preserve balancedness and
height should come at no surprise to the reader.
After all, the operations are never directly affecting the height of any tree,
and all trees generated by splitting a node comprise trees
that had been there before, now simply distributed along to two nodes
in the same level.

\begin{lemma}
    \begin{align*}
    \height t & = \height^{up_i} (\insfun_k x\ t) \\
    \bal t & \Longrightarrow \bal^{up_i} (\insfun_k x\ t)
    \end{align*}
\end{lemma}

The order invariant is the second property that needs
to be shown to stay invariant.
Since the central function to ensure order is the node$_i$ function,
we first show the following lemma:

\begin{lemma}
\label{lem:nodei-order}
    If all children have order $k$ and $\length ts \ge k$ and $\length ts \le 4k+1$
    then all trees in $\node_i k\ ts\ t$ have order $k$.
\end{lemma}

Even though in the case of insertion the length of the list
never exceeds $2k+1$, the statement holds up to $4k+1$.
The statement is proven by case distinction whether "$\length ts \le 2k$".
Looking at the definition in \Cref{lst:ins-fun}
we see that if this is the case nothing happens and the lemma is trivial.
In case "$\length ts > 2k$", a split occurs.
The median element is passed up as a seperator, leaving
$\lfloor\frac{\length ts}{2}\rfloor$ and $\lceil\frac{\length ts}{2}\rceil - 1$
elements for the node to the left or right respectively.
Given the constraint on the maximum length of $ts$
this will always be below or equal $2k$.
Further, as we require a size of at least $2k+1$ elements for a split to occur,
the resulting nodes each have at least $k$ elements.
This argument hardly changes comparing the upper bounds $2k+1$ and $4k+1$,
but with the weaker upper bound \textit{node}$_i$ can be used for merging nodes
in the deletion function as well.

Using the above, the order invariant for \textit{ins}
follows quite directly by induction.

\begin{lemma}
    $\order_k t \Longrightarrow \order^{up_i}_k (\insfun_k x\ t)$
\end{lemma}

Since the invariant on B-trees only requires root order of $k$ on the
tree, we had to additionally derive versions of
these invariants for this weaker order.
They are straightforward however as all trees in the inductive case
have order $k$, for which preservation was just proven.
Putting things together, we obtain preservation of the invariants.

\begin{theorem}
    \label{thm:ins-invar}
    $\order^r_k t \wedge \bal t \Longrightarrow
    \order^r_k (\insertfun_k x\ t) \wedge \bal (\insertfun_k x t)$
\end{theorem}

The set interface further requires that the insertion returns a tree
that has the same inorder view as the original inorder with the element
inserted at the correct position.

Looking at the \textit{ins} function in \Cref{lst:ins-fun},
we see that, in an inductive proof, the main obligation
is to argue for the choice of the subtree for recursion.
Similar to the proof of \Cref{thm:isin-set},
we use two auxiliary lemmas, that turn the remaining
proof in simple case distinctions and chains of equations.

\begin{lemma}
    \label{lem:sorted-left-x-in}
    $\sorted(\inorder (Node\ ts\ t)) \wedge \splitfun ts\ x = (ls, rs) \Longrightarrow$ \\
    \begin{center}
    $\inslist x\ (\inorder (Node\ ts\ t))) = \inorder ls\ @\ \inslist x\ (\inorder rs\ @\ \inorder t)$
    \end{center}
\end{lemma}

\begin{lemma}
    \label{lem:sorted-right-x-in}
    $\sorted (\inorder (Node\ ts\ t)) \wedge \splitfun ts\ x = (ls, (sub,sep)\#rs) \wedge sep \neq x \Longrightarrow$ \\
    \begin{center}
    $\inslist x\ (\inorder ((sub,sep)\#rs)\ @\ \inorder t)) =$\\
    $ (\inslist x\ (\inorder sub))\ @\ sep \# \inorder rs\ @\ \inorder t$
    \end{center}
\end{lemma}

The only case not covered by the above lemmas is the case $sep = x$.
In that case, the tree does not change.
This is due to the fact that $x$ is already in the set represented by the tree
and hence does not need to be inserted.
The same happens when inserting $x$ into a list that already contains $x$
(i.e. the inorder of the tree),
which follows simply by induction on the list.

\begin{lemma}
    \label{lem:sorted-x-in}
    $\sorted xs \wedge x \in \setfun xs \Longrightarrow \inslist x\ xs = xs$
\end{lemma}

\Cref{lem:sorted-left-x-in}, \Cref{lem:sorted-right-x-in} and \Cref{lem:sorted-x-in} are sufficient to show correctness
of the \textit{ins} function inductively.
The theorem for \textit{insert} follows again automatically,
which concludes the proof of the insertion part.

\begin{theorem}
    \label{thm:ins-set}
    $\sorted  (\inorder t)\Longrightarrow$\\
    \begin{center}
    $\inorder (\insertfun_k x\ t) = \insfun_{list} x\ (\inorder t)$
    \end{center}
\end{theorem}

\subsection{Deletion}

\begin{figure}
\begin{lstlisting}[mathescape=true, language=Isabelle,label={lst:rebalance-def},
    caption={The rebalancing functions}]
fun rebalance_middle_tree where
  rebalance_middle_tree k ls Leaf sep rs Leaf = (Node (ls@(Leaf,sep)#rs) Leaf) |
  rebalance_middle_tree k ls (Node mts mt) sep rs (Node tts tt) = (
    if length mts $\ge$ k $\wedge$ length tts $\ge$ k then
        Node (ls@(Node mts mt,sep)#rs) (Node tts tt)
    else (
        case rs of [] $\Rightarrow$ (
            case node$_i$ k (mts@(mt,sep)#tts) tt of
                T$_i$ u $\Rightarrow$ Node ls u |
                Up$_i$ l a r $\Rightarrow$ Node (ls@[(l,a)]) r
            ) |
        (Node rts rt,rsep)#rs $\Rightarrow$ (
            case node$_i$ k (mts@(mt,sep)#rts) rt of
                T$_i$ u $\Rightarrow$
                   Node (ls@(u,rsep)#rs) (Node tts tt) |
                Up$_i$ l a r $\Rightarrow$
                   Node (ls@(l,a)#(r,rsep)#rs) (Node tts tt)
            )
        )
    )

fun rebalance_last_tree where
    rebalance_last_tree k ts t = (
        case last ts of (sub,sep) $\Rightarrow$
            rebalance_middle_tree k (butlast ts) sub sep [] t
    )
\end{lstlisting}
\end{figure}

The procedures described in \Cref{par:intro-del}
are implemented here with a
certain flavor due to the definition of B-trees.
The \textit{rebalancing} as implemented in \Cref{lst:rebalance-def}
comprises merging and splitting neighboring nodes.
For simplicity of the function and proofs, we always
merge with the sibling to the right.
The only exception, which is unavoidable due to the asymmetry of the data structure,
is an underflow in the last subtree.
It will be merged with the second-to-last subtree, the last
tree in the variable length list of the node.
Note also how the tree does not change when asked to rebalance a leaf.
The reason is that rebalancing leafs may only happen if it is preceded
by deletion from a leaf - which does not change the tree at all.

The other detail is regarding the \textit{split\_max} function
employed when deleting elements from inner nodes.
It is defined in \Cref{lst:del-def}.
Here, some freedom exists whether to swap with the maximum lesser
or the minimum greater element in the tree.
In our definition, the last tree of the node is explicitly stored in each node
and the subtree to the left of a separator lies within the same
pair inside the node list.
Using pattern matching, it is hence significantly
easier to obtain the maximum of the subtree to the left,
than than to obtain the minimum of the subtree to the right.
Therefore, we will always swap with the former, maximal lesser element.
%---------------------------------------------------------

The most interesting property for the rebalancing operations is the order property,
which is meant to be restored after underflows.
It is important to note here that we can not guarantee
that at least one element remains in the node that
is passed back upwards.
This violates even the weakest order formulation up until now,
the root order invariant.
Hence before making the result of rebalancing
a final tree, we need to recover root order.
We can guarantee that all subtrees of the result are of order $k$
and that there are no more than $2k$ elements in the result,
which is sufficient.
We say that if a tree $t$ has at most $2k$ elements
and all subtrees of $t$ have order $k$, then $t$ has
\textit{almost order} $k$ ($\order^a_k t$).
Using this new definition we formulate the invariant for
the rebalancing operation.

\begin{lemma}
    If all subtrees in ls, rs are of order k
    and of sub and t, only one may have almost order k and the other has order k,
    and all subtrees have equal height, then \\
    \begin{center}
    $\order^a_k (\rebalancemt_k ls\ sub\ sep\ sep\ rs\ t)$
    \end{center}
\end{lemma}

It might be suprising that we require all subtrees to have equal height
for this operation.
However, a close look at \Cref{lst:rebalance-def} may clarify this requirement.
In order to successfully rebalance an internal node, we need neighboring
nodes that are not leafs.
As this fact is already expressed in the pattern matching of the function,
it is simply \textit{undefined} if not all subtrees have equal height.
But we know that B-trees are always balanced,
so we may assume equal subtree height for all proofs
involving \textit{rebalance\_middle\_tree}.
For this proof, we re-use \Cref{lem:nodei-order},
as two nodes of order $k$ will always contain
a maximum of $4k+1$ separators.

As mentioned, the result may have an empty list as key-value list.
This happens exactly when one of the two subtrees of a root
with minimal size underflowed.
After this operation, the tree shrinks in height.
The shrinked tree then is exactly the one remaining subtree in the original node.
See the operation \textit{reduce\_root} in \Cref{lst:del-def} for the exact implementation
of this step.
Since rebalancing always restores $\order k$ for all subtrees
of the current node and \textit{delete} employs \textit{reduce\_root},
the final result always has at least $\order^r k$.

\begin{figure}
\begin{lstlisting}[mathescape=true, language=Isabelle,label={lst:del-def},
    caption={The $delete$ function}]

fun split_max where
    split_max k (Node ts t) = (
        case t of Leaf $\Rightarrow$ (
            let (sub,sep) = last ts in (Node (butlast ts) sub, sep)
        ) |
        _ $\Rightarrow$ case split_max k t of (sub, sep) $\Rightarrow$
                (rebalance_last_tree k ts sub, sep)
        )

fun del where
    del k x Leaf = Leaf |
    del k x (Node ts t) = (
    case split ts x of (ls,[]) $\Rightarrow$
        rebalance_last_tree k ls (del k x t) |
    (ls,(sub,sep)#rs) $\Rightarrow$
        if sep $\neq$ x then
            rebalance_middle_tree k ls (del k x sub) sep rs t
        else if sub = Leaf then
            Node (ls@rs) t
        else let (sub_s, max_s) = split_max k sub in
            rebalance_middle_tree k ls sub_s max_s rs t
    )

fun reduce_root where
    reduce_root Leaf = Leaf |
    reduce_root (Node ts t) = (case ts of
        [] $\Rightarrow$ t |
        _ $\Rightarrow$ (Node ts t)
    )

fun delete where delete k x t = reduce_root (del k x t)
\end{lstlisting}
\end{figure}

\begin{figure}
    \centering
    \includegraphics[width=0.54\linewidth]{figures/btree-basic-ins9-ins1.pdf}\\
    \vspace*{1cm}
    \includegraphics[width=0.48\linewidth]{figures/btree-basic-ins9-ins1-del10.pdf}\\
    \vspace*{1cm}
    \includegraphics[width=0.48\linewidth]{figures/btree-basic-ins9-ins1-del10-del3.pdf}
    \caption[An example B-Tree after two deletion operations]{The tree from \Cref{fig:btree-basic-ins} (top) after
    successive deletion of $10$ (middle) and $3$ (bottom).
    For the first deletion, the value 10 and is swapped with value 9 before it is removed from the tree.
    The middle subtree is then in an underflow, resulting in a merge with the right subtree and the new separator 9.
    During the second deletion, an underflow is caused in the left subtree,
    resulting in a merge with the separator 5 and the middle subtree and a subsequent split.
    Note how the final result differs from approaches that would only steal single elements
    from neighbors to handle underflow.}
    \label{fig:btree-basic-del}
\end{figure}

The proofs for balancedness invariance follow similarly to
the ones in \Cref{sec:abs-ins} by first showing height invariance.
Both properties follow using the associativity
and commutativity of the maximum operation.
\footnote{
    When inspecting the proof documents,
    one will notice that the proof of height, balancedness and order
    are interleaved for the
    intermediate functions, especially
    \textit{split\_max} and \textit{rebalance\_middle\_tree}.
    This is due to the fact that these operations
    are only well defined for balanced trees.
    Moreove \textit{split\_max} requires at least
    one element in the node list, an equivalent
    to $\order^r k$.
}

With the basic operations covered,
the invariant preservation of the \textit{del} function
can be derived inductively.
Note that the function will return a tree of almost order $k$,
where the remaining root-underflow is coverered by \textit{reduce\_root}.
Moreover we do not need an additional lemma
for an input with nodes that have normal order $k$.
As $k > 0$, order $k$ implies a root order of $k$.
And even though it might seem to be beneficial to know
whether the input has normal order or root order, this is not the case.
The rebalancing functions only require
almost order $k$ and return a list of trees of valid order.
We are never interested in the length of that list itself
until reaching the root, so we are satisfied with statements
on the order of subtrees,
which are made by almost order $k$ sufficiently.

\begin{lemma}
    $k > 0 \wedge \order^r_k t \wedge \bal t \Longrightarrow
    \order^a_k (\delfun_k x\ t) \wedge \bal (\delfun_k x t)$
\end{lemma}

From this and our inspection of \textit{reduce\_root}
follows almost directly the fact about the invariant
preservation of \textit{delete}.

\begin{samepage}
\begin{theorem}
    $k > 0 \wedge \order^r_k t \wedge \bal t \Longrightarrow$ \\
    \begin{center}
    $\order^r_k (\deletefun_k x\ t) \wedge \bal (\deletefun_k x t)$
    \end{center}
\end{theorem}
\end{samepage}

Note that now (as opposed to insertion) $k > 0$ is required.
The main reason is that,
if $k = 0$, rebalancing would not always work anymore
as we could not be certain to have at least two subtrees per node.

In order to prove that \textit{delete} acts the same as
\textit{delete}$_{list}$ on the inorder of the tree,
we need to show some inorder properties for the intermediate functions.
For the \textit{del} helper function,
we use the same specialized lemmas
as for \Cref{thm:isin-set} and \Cref{thm:ins-set}
and induct over the execution of the \textit{del}-function.
% TODO state or no state?
Rather than stating the specialized lemmas again, we would like to point
out the benefits of the inorder method at this point.
In a first proof attempt, the whole deletion operation was
proven correct with respect to the standard set interface.
This attempt is compared in the following with a rough sketch
the proof worked in the inorder approach.

The reason the rebalancing operations preserve the sortedness and set properties
is plain when considering the inorder view of the tree:
it does not change at all.
A manual proof about the set and sortedness properties
required complicated, unnecessarily lengthy proofs,
that involved many case distinctions and sophisticated solvers
to argue for the validity of node merges.
For the inorder view,
since the definition can be easily unfolded
by the automatic simplifier,
the following property can be shown automatically.

\begin{lemma}
    \label{lem:rebalance-inorder}
    All subtrees have the same height $\Longrightarrow$ \\
    \begin{align*}
    \inorder&\ (\rebalancemt_k ls\ sub\ sep\ rs\ t) =\\
    \inorder&\ (Node\ (ls@(sub,sep)\#rs)\ t)
    \end{align*}
\end{lemma}

Considering the function split\_max,
the standard approach required showing
that the element returned is in fact the maximum
of the subtree to the left
and that the remaining tree union the maximum element
gives the whole original tree set.
Instead we simply show the following,
comprising both facts:

\begin{lemma}
    \label{lem:splitmax-inorder}
    If t has more than two subtrees, and the last two have equal height, then
    \begin{equation*}
    \inorder\ (\splitmax_k t) = \inorder t
    \end{equation*}
\end{lemma}

This fact follows easily by induction on the computation of split\_max,
using \Cref{lem:rebalance-inorder}.
The reason that these proofs follow so easily
is the intention behind the definition of both operations.
The idea is to obtain the same elements, in the same order,
just now in a configuration from which we can easily obtain
a new, valid B-tree.

Finally we obtain the last important property of the set interface.
\begin{theorem}
    $k > 0 \wedge \order^r_k t \wedge \bal t \wedge \sorted  (\inorder t)\Longrightarrow$\\
    \begin{center}
    $\inorder\ (\deletefun_k x\ t) = \delfun_{list} x\ (\inorder t)$
    \end{center}
\end{theorem}

With this, we have proven that
our implementation of B-trees is proven correct
fulfils the specifications of the set interface based on an inorder view.
The next step towards imperative B-trees is to implement
an imperative refinement of these set operations.


\subsection{2017 Free-Response Answers}

\begin{enumerate}
	\item \begin{enumerate}
		\item Using a left Riemann sum,
			\begin{equation*}
				\int_{0}^{10}{A(h)\d{h}} \approx (2-0)50.3 + (5-2)14.4 + (10-5)6.5 = 100.6 + 43.2 + 32.5 = 176.3.
 			\end{equation*}
 			So, we approximate that the volume of the tank is 176.3 cubic feet.
 		\item Since we are given that the function is decreasing over the interval, a left Riemann sum will overestimate the volume.
 		\item Integrating $f$ from $h=0$ to $h=10$,
 			\begin{equation*}
 				\int_{0}^{10}{\frac{50.3}{e^{0.2h}+h}\d{h}} \approx 101.325.
 			\end{equation*}
 			So, the volume of the tank given by $f$ is 101.325 cubic feet.
 		\item We know from (c) that
 			\begin{equation*}
 				V(h) = \int_{0}^{h}{f(x)\d{x}}.
 			\end{equation*}
 			Differentiating with respect to $t$, and applying the chain rule,
 			\begin{equation*}
 				\dd{V}{t} = f(h)\cdot\dd{h}{t}.
 			\end{equation*}
 			Plugging in $h=5$ and $\dd{h}{t}=0.26$,
 			\begin{equation*}
 				\dd{V}{t}_{h=5} = \frac{50.3}{e^1 + 5}\cdot 0.26 \approx 1.694.
 			\end{equation*}
 			So, when $h=5$ feet, the volume is changing at a rate of 1.694 cubic feet per minute.
	\end{enumerate}

	\item \begin{enumerate}
		\item Finding the area enclosed by $f$ from $\theta=0$ to $\theta=\pi/2$,
			\begin{equation*}
				A = \frac{1}{2}\int_{0}^{\pi/2}{\left(1+\sin{\theta}\cos{(2\theta)}\right)^2d\theta} \approx 0.648.
			\end{equation*}
		\item Our ray $\theta=k$ will represent a lower bound in one integral and an upper bound in the other.
			Finding equal areas between $g$ and $f$,
			\begin{equation*}
				\frac{1}{2}\int_{0}^{k}{\left(\left(2\cos{\theta}\right)^2-\left(1+\sin{\theta}\cos{(2\theta)}\right)^2\right)d\theta} = \frac{1}{2}\int_{k}^{\pi/2}{\left(\left(2\cos{\theta}\right)^2-\left(1+\sin{\theta}\cos{(2\theta)}\right)^2\right)d\theta}.
			\end{equation*}
		\item
			Since both $f$ and $g$ are polar functions evaluated at $\theta$, the distance between then will simply be the difference in their radii.
			\begin{equation*}
				w(\theta) = g(\theta) - f(\theta).
			\end{equation*}
			Finding the average of $w$,
			\begin{equation*}
				w_A = \frac{1}{\pi/2 - 0}\int_{0}^{\pi/2}{(2\cos{\theta}-(1+\sin{\theta}\cos{(2\theta)}))\d{\theta}} \approx 0.485.
			\end{equation*}
		\item Solving $w(\theta) = w_A$,
			\begin{equation*}
				2\cos{\theta}-(1+\sin{\theta}\cos{(2\theta)}) = 0.485 \implies \theta \approx 0.518.
			\end{equation*}
			Evaluating $w^\prime(0.518)$,
			\begin{equation*}
				w^\prime(0.518) \approx -0.581.
			\end{equation*}
			So, $w$ is decreasing at this value.
	\end{enumerate}

	\item \begin{enumerate}
		\item Applying the Fundamental Theorem of Calculus,
			\begin{align*}
				\int_{-6}^{-2}{f^\prime(x)\d{x}} &= f(-2) - f(-6) = 7 - f(-6) = 4 \implies f(-6) = 3 \\
				\int_{-2}^{5}{f^\prime(x)\d{x}} &= f(5) - f(-2) = f(5) - 7 = -2\pi + 3 \implies f(5) = 10 - 2\pi.
			\end{align*}
		\item $f$ is increasing when its derivative is positive.
			Looking at the graph of $f^\prime$, we see it is positive on $[-6,2) \cup (2,5)$.
			So, $f$ is increasing on $[-6,2] \cup [2,5]$\footnote{I personally don't think $f$ is ``increasing'' on the open endpoints where the derivative is 0. I think it's neither increasing nor decreasing. However, the test writers and graders feel differently, and I can't say their view is necessarily wrong.}.
		\item We see that the critical points of $f^\prime$ are $x=-2$ and $x=2$.
			A minimum can occur at a left/right endpoint that will/was decreasing.
			So, the two points we need to consider for the absolute minimum are $x=-6$ and $x=2$.
			We know from (a) that $f(-6)=3$.
			Applying the Fundamental Theorem of Calculus again,
			\begin{equation*}
				\int_{-2}^{2}{f^\prime(x)\d{x}} = f(2) - f(-2) = f(2) - 7 = 2\pi \implies f(2) = 7-2\pi.
			\end{equation*}
			Since  $7-2\pi < 3$, the absolute minimum of $f$ is at $x=2$.
		\item $f^{\prime\prime}(-5) = -\frac{1}{2}$ because $f$ is continuous at -5, and the left and right hand limits are both the slope of the line, which is $-\frac{1}{2}$.
			$f^{\prime\prime}(3) = \text{DNE}$.
			The left hand limit is the slope of the left hand line, which is 2, and the right hand limit is the left of the right hand line, which is -1.
			Since the limits do not agree, $f^\prime$ is not continuous, and hence not differentiable at $x=3$.
	\end{enumerate}

	\item \begin{enumerate}
		\item Writing the line in point-slope form and then converting to standard form,
			\begin{align*}
				y - y_0 &= m(t-t_0) \\
				y - H(0) &= \dd{H}{t}_{t=0}(t-0) \\
				y - 91 &= -16t \\
				y &= -16t + 91.
			\end{align*}
			Using this line to approximate $H(3)$,
			\begin{equation*}
				y = -16(3) + 91 = 43.
			\end{equation*}
			So, the tangent line at $t=0$ approximates $H(3)$ to be $43^\circ C$.
		\item Taking the derivative of $\dd{H}{t}$,
			\begin{equation*}
				\dd{^2H}{t^2} = -\frac{1}{4}.
			\end{equation*}
			Since the graph is concave down at every point, the tangent line at $t=0$ gives an overestimate of $H(3)$.
		\item This is a separable differential equation.
			\begin{align*}
				\dd{G}{t} &= -(G-27)^{2/3} \\
				\frac{\d{G}}{(G-27)^{2/3}} &= -1\d{t} \\
				\int{\frac{\d{G}}{(G-27)^{2/3}}} &= \int{-1\d{t}} \\
				3(G-27)^{1/3} &= -t + C \\
				(G-27)^{1/3} &= -t/3 + C \\
				G - 27 &= \left(-t/3 + C\right)^3 \\
				G &= \left(-t/3 + C\right)^3 + 27.
			\end{align*}
			Solving for $C$ using $G(0)=91$,
			\begin{align*}
				91 &= \left(-0/3 + C\right)^2 + 27 \\
				64 &= C^3 \\
				C &= 4.
			\end{align*}
			So, our overall solution for $G$ is
			\begin{equation*}
				G(t) = \left(4 - t/3\right)^3 + 27.
			\end{equation*}
			Evaluating at $t=3$,
			\begin{equation*}
				G(3) = \left(4 - 3/3\right)^3 + 27 = 3^3  + 27 = 54.
			\end{equation*}
			So, the internal temperature of the potato at $t=3$ minutes is $54^\circ C$.
	\end{enumerate}

	\item \begin{enumerate}
		\item Applying the quotient rule,
			\begin{equation*}
				f^\prime(x) = \frac{(2x^2-7x+5)(0)-3(4x-7)}{(2x^2-7x+5)^2} = \frac{-12x+21}{(2x^2-7x+5)^2}.
			\end{equation*}
			Evaluating at $x=3$,
			\begin{equation*}
				f^\prime(3) = \frac{-12(3)+21}{(2(3)^2-7(3)+5)^2} = -\frac{15}{(18-21+5)^2} = -\frac{15}{4}.
			\end{equation*}
		\item $f^\prime(x)=0$ only when the numerator is 0,
			\begin{equation*}
				-12x + 21 = 0 \implies x = \frac{7}{4}.
			\end{equation*}
			$f^\prime$ is negative to the left of $\frac{7}{4}$ and positive to the right of it.
			Therefore, $x=\frac{7}{4}$ is a relative maximum by the first derivative test.
		\item Evaluating the limit using the given partial fraction decomposition,
			\begin{align*}
				\int_{5}^{\infty}{f(x)\d{x}} &= \int_{5}^{\infty}{\left(\frac{2}{2x-5}-\frac{1}{x-1}\right)\d{x}} \\
				&= \ln{(2x-5)} - \ln{(x-1)} \biggr\rvert_{5}^{\infty} \\
				&= \ln{\left(\frac{2x-5}{x-1}\right)} \biggr\rvert_{5}^{\infty} \\
				&= \lim_{b\to\infty}{\ln{\left(\frac{2b-5}{b-1}\right)}} - \ln{\left(\frac{5}{4}\right)} \\
				&= \ln{2} - \ln{\left(\frac{5}{4}\right)} \\
				&= \ln{\left(\frac{8}{5}\right)}.
			\end{align*}
		\item For $n \geq 5$, $f(n)$ is positive and decreasing.
			Using our work from part (c), we know that the integral of $f$ from 5 to $\infty$ converges.
			So, by the Integral test, the series also converges.
	\end{enumerate}

	\item \begin{enumerate}
		\item Calculating the first four derivatives and the general derivative,
			\begin{align*}
				f(0) &= 0 \\
				f^\prime(0) &= 1 \\
				f^{\prime\prime}(0) &= -1\cdot f^\prime(0) = -1 \\
				f^{(3)}(0) &= -2\cdot f^{\prime\prime}(0) = 2 \\
				f^{(4)}(0) &= -3\cdot f^{(3)}(0) = -6.
				f^{(n+1)}(0) &= -n\cdot f^{(n)}(0) = (-1)^{n+1}n!.
			\end{align*}
			Applying the Maclaurin series formula,
			\begin{align*}
				P_n(x) &= \frac{f(0)}{0!} + \frac{f^\prime(0)}{1!}x + \frac{f^{\prime\prime}(0)}{2!}x^2 + \ldots + \frac{f^{(n)}(0)}{n!}x^n \\
				&= \frac{0}{1} + \frac{1}{1}x + \frac{-1}{2}x^2 + \frac{2}{6}x^3 + \frac{-6}{24}x^4 + \ldots + \frac{(-1)^{n+1}(n-1)!}{n!}x^n \\
				&= x - \frac{x^2}{2} + \frac{x^3}{3} - \frac{x^4}{4} + \ldots + (-1)^{n+1}\frac{x^n}{n}.
			\end{align*}
		\item At $x=1$, the series is
			\begin{equation*}
				\sum_{n=1}^{\infty}{(-1)^{n+1}\frac{1}{n}}.
			\end{equation*}
			This series converges by the Alternating Series test, which determines conditional converges.
			So, the series converges conditionally at $x=1$.
		\item Integrating term-by-term,
			\begin{align*}
				g(x) &= \int_{0}{x}{f(t)\d{t}} \\
				&= \int_{0}^{x}{\left(t-\frac{t^2}{2}+\frac{t^3}{3}-\frac{t^4}{4}+\ldots+(-1)^{n+1}\frac{t^n}{n}\right)\d{t}} \\
				&= \frac{t^2}{2} - \frac{t^3}{6} + \frac{t^4}{12} - \frac{t^5}{20} + \ldots + (-1)^{n+1}\frac{t^{n+1}}{n(n+1)} \biggr\rvert_{0}^{x} \\
				&= \frac{x^2}{2} - \frac{x^3}{6} + \frac{x^4}{12} - \frac{x^5}{20} + \ldots + (-1)^{n+1}\frac{x^{n+1}}{n(n+1)}.
			\end{align*}
		\item The Alternating Series Estimation Theorem tells us that the upper bound for the error is the absolute value of the first not included term.
			For $P_4$, this term is $-\frac{x^5}{20}$.
			So,
			\begin{align*}
				\biggr\lvert P_4\left(\frac{1}{2}\right) - g\left(\frac{1}{2}\right) \biggr\rvert < \biggr\lvert -\frac{\left(\frac{1}{2}\right)^5}{20} \biggr\rvert = \frac{1}{640} < \frac{1}{500}.
			\end{align*}
	\end{enumerate}

\end{enumerate}


\subsection{One-dimensional uniform dataset}

% We train a two-dimensional map over $25000$ samples drawn from a uniform distribution ($\mathcal{U}(0, 1)$). The map consists of $1024$ units (neurons) and it is being trained for $25000$ epochs (all the parameters can be found in Table~\ref{table:parameters}). Figure~\ref{fig:one-dim}{\bfseries \sffamily A} shows the neural space topology of the two-dimensional map after sampling a blue noise distribution and placing the neurons on the corresponding nodes. Figure~\ref{fig:one-dim}{\bfseries \sffamily B} indicates the learned representations after training. In this case the mapping of real numbers within the interval $[0, 1]$ is illustrated in color-code with blue color representing number zero and yellow representing one. It is clear that the map has organize the representations in a descending order starting from the upper left corner (one) of the map to the lower right corner (zero).

% Neurons of both the Kohonen and VSOM maps develop receptive filters during training. Each of these receptive fields usually captures a portion of the input space and learns the reciprocal representations. Therefore, we examine the receptive fields of neurons by evaluating their response to a stimulus. To this end, we feed the map with $6$ input samples after discretizing the interval $[0, 1]$. The activity of each neuron is  computed as the Euclidean distance between the input sample and the neuron's code word. Figures~\ref{fig:one-dim} {\sffamily \bfseries C}-{\sffamily \bfseries H}  show the neural activity on the map for $6$ input samples. It is apparent that different regions of the map respond to different stimuli, implying that the topographic organization has been successfully accomplished.

\begin{figure}
  \includegraphics[width=\columnwidth]{experiment-1D-uniform.pdf}
  \vspace{2mm}
  \centering
  \includegraphics[width=.975\columnwidth]{figures/colormap.pdf}
  %
  \caption{%
  %
  {\bfseries \sffamily One dimensional uniform dataset with holes (results)}
  %
  Randomized SOM made of $1024$ neurons with a $3$-nearest neighbors induced topology. Model has been trained for $25,000$ epochs on one-dimensional points drawn from a uniform distribution on the unit segment. \textbf{A} Map topology in neural space. \textbf{B} Map topology in data space. \textbf{C to H} Receptive field of the map for six samples.
  %
  }
  \label{fig:1D-uniform:results}
\end{figure}


\newpage
\section{Classification Algorithms and Regression Trees}
This is from the book by Breiman et. al.

At the university of California, San Diego Medical Center, when a
heart attack patient is admitted, 19 variables are measured during the
first 24 hours. They include BP, age and 17 other binary covariates
summarizing the medical symptoms considered as important indicators of
the patient's condition.

The goal of a medical study can be to develop a method to identify
high risk patients on the basis of the initial 24-hour data.

The next figure shows  a picture of a tree structured classification
rule that was produced in the study. The letter F means no high and
the letter G means high risk.

\centerline{\epsfig{figure=Plots/plot-07-06.ps,width=\textwidth}}

How can we use data to construct trees that give us useful
answers. There is a large amount of work done in this type of problem.
We will give a brief description in this section.

%%%Karl's notes
\subsection{Classifiers as Partitions}
Suppose we have a categorical outcome
$y \in \mathcal{C} = \{1, 2, \ldots, J\}$. We call ${\cal C}$ the set
of classes. Denote  with ${\cal X}$ the space
of all possible covariates.

We can define a classification rule as a function $d(\bx)$ defined on
${\cal X}$ so that for every $\bx$, $d(\bx)$ is equal to one of the numbers
$1,\dots,J$.

This could be considered a systematic way of predicting class
membership from the covariates.

Another way to define classifiers is to partition  ${\cal X}$
into disjoint sets $A_1,\dots,A_j$ with $d(\bx)=j$ for all $\bx \in A_j$.

But how do we construct these classifiers from data?

\subsection{What is truth?}
We are now going to describe how to construct classification rules
from data. The
data we use to construct the tree is called the training set $\cal L$
which is simply $\{ (\bx_1,j_1),\dots,(\bx_n,j_n) \}$.

Once a classification rule $d(X)$ is constructed how do we define it's
accuracy?

In this section we will define the {\it true misclassification rate}
$R^*(d)$.

One way to estimate $R^*(d)$ is to draw another very large subset
(virtually infinite) from the same population as $\cal L$ and observe
the rate of correct classification in that set. The proportion
misclassified by $d$ is our estimate of $R^*(d)$.

To make this definition more precise, define the space ${\cal X} \times
{\cal C}$ as the set of all couples
$(\bx,j)$ where $\bx \in {\cal X}$ and $j \in {\cal C}$. Let
$\Pr(A,j)$ be a probability distribution on  ${\cal X} \times {\cal
  C}$. Assume each element of $\cal L$ is an iid outcome
from this distribution.

We define the misclassification rate as
\begin{equation}
\label{mr}
R^*(d) = \Pr[d(\bx) \neq j | {\cal L}]
\end{equation}
with $(\bx,j)$ an outcome independent of $\cal L$.

How do we obtain an estimate of this?

The {\it substitution estimate} simply counts how many times we are
right with the data we have, i.e.
\[
R(d) = \frac{1}{N} \sum_{n=1}^N 1_{d(\bx_n) \neq j_n}.
\]
The problem with this estimate is that most classification algorithms
construct $d$ trying to minimize the above equation. If we have enough
covariates we can define a rule that always has $d(\bx_n) = j_n$ and
randomly allocates any other $\bx$. This has an $R(d)=0$ but one can
see that, in general, $R^*(d)$ will be much bigger.

Another popular approach is the {\it test sample} estimate. Here we
divide the data $\cal L$ into two groups ${\cal L}_1$ and
${\cal L}_2$. We then use ${\cal L}_1$ to define $d$ and ${\cal L}_2$
to estimate $R^*(d)$ with
\[
R(d) =\frac{1}{N_2} \sum_{\bx_n \in {\cal L}_2} 1_{d(\bx_n) \neq j_n}
\]
with $N_2$ the size of ${\cal L}_2$. A popular choice for $N_2$ is
1/3 of $N$, the size of $\cal L$.

A problem with this procedure is that we don't use 1/3 of the data
when constructing $d$. In situations where $N$ is very large this may
not be such a big problem.

The third approach is cross validation. We divide the data into many
subsets of equal (or approximately equal) size ${\cal L}_1,\dots{\cal
  L}_V$, define a $d_v$ for each of these groups, and use the estimate
\[
R(d) = \frac{1}{V}\sum_{v=1}^V \frac{1}{N_v} \sum_{\bx_n \in {\cal L}_v}
 1_{d(\bx_n) \neq j_n}.
\]

\subsection{Bayes Rule}
The major guide that has been used in the construction of classifiers
is the concept of the Bayes rule. A Bayes rule is the $d_B$ for which
\[
\Pr[d_B(\bx) \neq j) \leq \Pr[d(\bx) \neq y].
\]
for all classification rules $d(\bx)$.

If we assume that $\Pr(A | j)$ has a probability density  $f_j(\bx)$
such that
\[
\Pr(A|j) = \int_A f_j(\bx) \, dx
\]
then we can show
that
\[
d_B(\bx) = j \mbox{ on } A_j = \{\bx; f_j(\bx)\Pr(j) = \max_i
f_i(\bx)\Pr(j) \}.
\]

Discriminant analysis, kernel density estimation, and $k$-th nearest
neighbor smoothing attempt to estimate $f_j(\bx)$ and $\Pr(j)$ in
order to estimate the Bayes rule. They make many assumptions so the
methods are not always useful.

\subsection{Constructing tree classifiers}
Notice how big the space of all possible classifiers is. In the simple
case where ${\cal X} = \{0,1\}^p$ this space has $2^p$ elements.

Binary trees are a special case of this partition. Binary trees are
constructed by repeated splits of the subsets of ${\cal X}$ into two
descendant subsets, beginning with ${\cal X}$ itself.

\setlength{\unitlength}{1.0cm}
\begin{center}
\begin{picture}(12,11)

% circles
\thicklines
\put(6,10){\circle{2}}
\put(3.5,7.5){\circle{2}}
\put(8.5,7.5){\circle{2}}
\put(10.5,4.5){\circle{2}}

% boxes
\put(1.25,3.5){\framebox(1.5,1.5)[]{${\cal X}_5$}}
\put(4.25,3.5){\framebox(1.5,1.5)[]{${\cal X}_6$}}
\put(6.25,3.5){\framebox(1.5,1.5)[]{${\cal X}_7$}}
\put(8.25,0.5){\framebox(1.5,1.5)[]{${\cal X}_8$}}
\put(11.25,0.5){\framebox(1.5,1.5)[]{${\cal X}_9$}}

% lines
\put(5.5,9.5){\line(-1,-1){1.5}}
\put(6.5,9.5){\line(1,-1){1.5}}
\put(3,7){\line(-1,-2){1}}
\put(4,7){\line(1,-2){1}}
\put(8,7){\line(-1,-2){1}}
\put(9,7){\line(1,-2){1}}
\put(10,4){\line(-1,-2){1}}
\put(11,4){\line(1,-2){1}}

% text
\put(6,10){\makebox(0,0)[]{${\cal X}$}}
\put(3.5,7.5){\makebox(0,0)[]{${\cal X}_2$}}
\put(8.5,7.5){\makebox(0,0)[]{${\cal X}_3$}}
\put(10.5,4.5){\makebox(0,0)[]{${\cal X}_4$}}

\put(6,8.75){\makebox(0,0)[]{Split 1}}
\put(3.45,6){\makebox(0,0)[]{Split 2}}
\put(8.45,6){\makebox(0,0)[]{Split 3}}

\put(10.45,3){\makebox(0,0)[]{Split 4}}
\end{picture} \end{center}

The subsets created by the splits are called {\it nodes}. The subsets
which are not split are called terminal nodes.

Each terminal nodes gets
assigned to one of the classes. So if we had 3 classes we could get $A_1
= {\cal X}_5 \cup {\cal X}_9$, $A_2 ={\cal X}_6$ and $A_3 ={\cal X}_7 \cup {\cal X}_8$.
If we are using  the data we assign the class most frequently found in
that subset of ${\cal X}$. We call these classification tress.

Various question still remain to be answered
\begin{itemize}
\item How do we define truth?
\item How do we construct the trees from data?
\item How do we assess trees, i.e. what makes a good tree?
\end{itemize}

The first problem in tree construction is how to use $\cal L$ to
determine the binary splits of $\cal X$ into smaller and smaller
pieces. The fundamental idea is to select each split of a subset so that
the data in each of the descendant subsets are ``purer'' than the data
in the parent subset.

This can be implemented in the following way

\begin{itemize}
\item Define the node proportions $p(j|t)$ to be the proportion of
  cases $\bx_n \in t$ belonging to class $j$ so that $\sum_j p(j|t) =
  1$.

\item Define a measure of impurity $i(t)$ as a nonnegative function
  $\phi$ such that it reaches its maximum at $\phi(1/n,\dots,1/n)$,
$\phi(1,0,\dots,0)=0$,  and is symmetric with respect to its entries.

A popular example is the entropy
\[
i(t) = - \sum_{j=1}^J p(j|t) \log p(j|t),
\]
but there are many other choices.

\item Define a set $\cal S$ of binary splits $s$ at each node. Then we
  chose the split that minimize the impurity of the new left and right
  nodes
\[
\Delta i(s,t) = i(t) - p_L i(t_l) + p_R i(t_R)
\]

\end{itemize}

There are many different possible splits. For continuous variables
there are an infinite amount. We need to define the set of splits
$\cal S$ that we consider.

Most implementations require that the the splits are defined by only
one covariate, but fancier versions permit the use of linear combinations.

If the covariate is continuous or ordered then the split must be
defined by $x < c$ and $x \geq c$.

If the covariate is categorical then we simply consider all splits
that divide original set into two.

\setlength{\unitlength}{1.0cm}
\begin{center}
\begin{picture}(12,11)

% circles
\thicklines
\put(6,10){\circle{2}}
\put(3.5,7.5){\circle{2}}
\put(8.5,7.5){\circle{2}}
\put(10.5,4.5){\circle{2}}

% boxes
\put(1.25,3.5){\framebox(1.5,1.5)[]{1}}
\put(4.25,3.5){\framebox(1.5,1.5)[]{2}}
\put(6.25,3.5){\framebox(1.5,1.5)[]{1}}
\put(8.25,0.5){\framebox(1.5,1.5)[]{2}}
\put(11.25,0.5){\framebox(1.5,1.5)[]{3}}

% lines
\put(5.5,9.5){\line(-1,-1){1.5}}
\put(6.5,9.5){\line(1,-1){1.5}}
\put(3,7){\line(-1,-2){1}}
\put(4,7){\line(1,-2){1}}
\put(8,7){\line(-1,-2){1}}
\put(9,7){\line(1,-2){1}}
\put(10,4){\line(-1,-2){1}}
\put(11,4){\line(1,-2){1}}

% text
\put(6,10){\makebox(0,0)[]{$x_1$}}
\put(3.5,7.5){\makebox(0,0)[]{$x_2$}}
\put(8.5,7.5){\makebox(0,0)[]{$x_3$}}
\put(10.5,4.5){\makebox(0,0)[]{$x_2$}}

\put(4.25,9.25){\makebox(0,0)[]{$< 5$}}
\put(7.75,9.25){\makebox(0,0)[]{$\ge 5$}}
\put(1.75,6){\makebox(0,0)[]{$> 3$}}
\put(5.25,6){\makebox(0,0)[]{$\le 3$}}
\put(6.75,6){\makebox(0,0)[]{$= 2$}}
\put(10.25,6){\makebox(0,0)[]{$\ne 2$}}
\put(8.75,3){\makebox(0,0)[]{$> 1$}}
\put(12.25,3){\makebox(0,0)[]{$\le 1$}}
\end{picture}
\end{center}

Now all we need is a stopping rule and we are ready to create trees. A
simple stopping rule is that $\Delta i(s,t) < \delta$, but this does
not work well in practice.

What is usually done is that we let the trees grow to a size that is
bigger than what we think makes sense and then prune. We remove node
by node and compare the trees using estimates of $R^*(d)$.

Sometimes to save time and/or choose smaller trees we define a
penalized criterion based on  $R^*(d)$.

The big issue here is \emph{model selection}.  The model
selection problem consists of four orthogonal components.

\begin{enumerate}

\item Select a space of models

\item Search through model space

\item Compare models
  \begin{itemize}
    \item of the same size
    \item of different sizes (penalize complexity)
  \end{itemize}

\item Assess the performance of a procedure

\end{enumerate}

%\vsphalf

\textbf{Important points}:
\begin{itemize}
\item Components 2 and 3 are often confused (e.g., in
  stepwise regression).  That's bad.

\item People often forget component 1.

\item People almost always ignore component 4; it can be the hardest.

\end{itemize}

Better trees may be found by doing a one-step ``look ahead,'' but this
comes with the cost of a great increase in computation.

\subsection{Regression Trees}
If instead of classification we are interested in predicting we can
assign a predictive value to each of the terminal nodes. Notice that
this defines an estimate for the regression function
$\E(Y|X_1,\dots,X_n)$ that is like a multidimensional bin smoother.
We call these regression trees.

Regression trees are constructed in a similar way to classification
trees. They are used for the case where $Y$ is a
continuous random variable.

A regression tree partitions $x$-space into disjoint regions $A_k$
and provides a fitted value $\E(y | x \in A_k)$ within each region.

\setlength{\unitlength}{1.0cm}
\begin{center}
\begin{picture}(12,11)

% circles
\thicklines
\put(6,10){\circle{2}}
\put(3.5,7.5){\circle{2}}
\put(8.5,7.5){\circle{2}}
\put(10.5,4.5){\circle{2}}

% boxes
\put(1.25,3.5){\framebox(1.5,1.5)[]{13}}
\put(4.25,3.5){\framebox(1.5,1.5)[]{34}}
\put(6.25,3.5){\framebox(1.5,1.5)[]{77}}
\put(8.25,0.5){\framebox(1.5,1.5)[]{51}}
\put(11.25,0.5){\framebox(1.5,1.5)[]{26}}

% lines
\put(5.5,9.5){\line(-1,-1){1.5}}
\put(6.5,9.5){\line(1,-1){1.5}}
\put(3,7){\line(-1,-2){1}}
\put(4,7){\line(1,-2){1}}
\put(8,7){\line(-1,-2){1}}
\put(9,7){\line(1,-2){1}}
\put(10,4){\line(-1,-2){1}}
\put(11,4){\line(1,-2){1}}

% text
\put(6,10){\makebox(0,0)[]{$x_1$}}
\put(3.5,7.5){\makebox(0,0)[]{$x_2$}}
\put(8.5,7.5){\makebox(0,0)[]{$x_3$}}
\put(10.5,4.5){\makebox(0,0)[]{$x_2$}}

\put(4.25,9.25){\makebox(0,0)[]{$< 5$}}
\put(7.75,9.25){\makebox(0,0)[]{$\ge 5$}}
\put(1.75,6){\makebox(0,0)[]{$> 3$}}
\put(5.25,6){\makebox(0,0)[]{$\le 3$}}
\put(6.75,6){\makebox(0,0)[]{$= 2$}}
\put(10.25,6){\makebox(0,0)[]{$\ne 2$}}
\put(8.75,3){\makebox(0,0)[]{$> 1$}}
\put(12.25,3){\makebox(0,0)[]{$\le 1$}}
\end{picture} \end{center}

In other words, this is a decision tree where the outcome is a
fitted value for $y$.

We need a new definition $d(\bx)$ and $R^*(d)$.

Now $d(\bx_j)$ will simply be the average of the terminal node where
$\bx_j$ lies. So $d(\bx)$ defines a step-function ${\mathbb R}^p
\rightarrow {\mathbb R}$.

Instead of misclassification rate, we can define mean squared error
\[
R^*(d) = \E[ Y - d(\bx) ]^2
\]

The rest is pretty much the same.
\subsection{General points}

From Karl Broman's notes.

\begin{itemize}

\item This is most natural when the explanatory variables are
  categorical (and it is especially nice when they are \emph{binary}).

\item There is nothing special about the tree structure...the tree
  just partitions $x$-space, with a fitted value in each region.

\item \textbf{Advantage}: These models go after
  \emph{interactions\/} immediately, rather than as an afterthought.

\item \textbf{Advantage}: Trees can be easy to explain to
  non-statisticians.

\item \textbf{Disadvantage}: Tree-space is huge, so we may need
  \emph{a lot\/} of data.

\item \textbf{Disadvantage}: It can be hard to assess uncertainty in
  inference about trees.

\item \textbf{Disadvantage}: The results can be quite variable.  (Tree
  selection is not very \emph{stable}.)

\item \textbf{Disadvantage}: Actual \emph{additivity\/} becomes a mess
  in a binary tree.  This problem is somewhat alleviated by allowing
  splits of the form $x_1 + b x_2 < (\ge) \; d$.

\end{itemize}

\centerline{\textbf{Computing with trees}}

%\HRule

%\vsphalf

\textbf{R}: \verb|library(tree)|; \verb|library(rpart)|
[MASS, ch 10]

%\vspone

\textbf{An important issue}: Storing trees

%\vsphalf

Binary trees are composed of nodes (root node, internal nodes and
terminal nodes).

%\vsphalf

\emph{Root and internal nodes}:
\begin{itemize}
\item Splitting rule (variable + what goes to right)
\item Link to left and right daughter nodes
\item Possibly a link to the parent node (null if this is the root
  node)
\end{itemize}

\emph{Terminal nodes}:
\begin{itemize}
\item Fitted value
\item Possibly a link to the parent node
\end{itemize}

%\vspone

\textbf{C}: Use pointers and structures (\verb|struct|)

%\vsphalf

\textbf{R}: It beats me.  Take a look.

\textbf{Ref}: Breiman et al (1984) Classification and regression
trees.  Wadsworth.


\subsection{Well-balanced Property with Uncertain Topography}

In this section it is shown that the stochastic Galerkin model satisfies the well-balanced property for a lake-at-rest with uncertain topography.
A lake at rest is an idealised scenario where the free-surface is entirely flat and motionless, much like still water in a lake.
Assuming a lake-at-rest for the stochastic model, the mean free-surface elevation is constant and the mean discharge is zero, and there is no uncertainty in the free-surface elevation or discharge.
The bed elevation can have any spatial profile and stochastic profile.

Since the free-surface elevation is constant and the averaged bed elevation is continuous at interfaces then $\eta_{i+1/2,p}^- - z^\star_{i+1/2,p} = \eta_{i+1/2,p}^+ - z^\star_{i+1/2,p}$, hence $h^{-,\star}_{i+1/2,p} = h^{+,\star}_{i+1/2,p}$ for all $i = 0, \ldots, M$ and $p = 0, \ldots, P$.
Shorthand notation is introduced such that
$\eta_p$ is the $p$\textsuperscript{th} expansion coefficient of the spatially uniform free-surface elevation, and
$h^\star_{i+1/2,p} = h^{-,\star}_{i+1/2,p} = h^{+,\star}_{i+1/2,p}$.
Since $h^\star_{i+1/2,p}$ is continuous at interfaces and the discharge is zero then the numerical flux is equal to the physical flux.

In order to preserve a lake-at-rest solution in element $i$ then the ensemble average of the flux gradient must balance the ensemble average of the source term vector in equation~\eqref{eqn:swe-pc}.
The balance of mass and momentum components can be considered separately.
The mass continuity equation balances because the discharge is zero.
For the momentum equation, the following equality must hold for all $l = 0, \ldots, P$:
\begin{align}
\frac{\Ensemble{F_{i+1/2}^{(n)} \pcbasis_l} - \Ensemble{F_{i-1/2}^{(n)} \pcbasis_l}}{\Delta x}
-
\Ensemble{S_i^{(n)} \pcbasis_l}
= 0
\label{eqn:momentum-balance-separate}
\end{align}
where the ensemble average of the bed slope source term $\Ensemble{S_i^{(n)} \pcbasis_l}$ is given by equation~\eqref{eqn:pc-source} and the momentum flux $F_{i+1/2}^{(n)}$ is
\begin{align}
F_{i+1/2}^{(n)} = \frac{g}{2}
\left\{\sum_{p=0}^P h^\star_{i+1/2,p} \pcbasis_p\right\}
\left\{\sum_{s=0}^P h^\star_{i+1/2,s} \pcbasis_s\right\}
\end{align}
and similarly for $F_{i-1/2}^{(n)}$.
Due to the distributivity property of the basis (equation~\ref{eqn:distributive}), equation~\eqref{eqn:momentum-balance-separate} can be rewritten as
\begin{align}
\Ensemble{\frac{F_{i+1/2}^{(n)}
-
F_{i-1/2}^{(n)}}{\Delta x} \pcbasis_l}
=
\Ensemble{ S_i^{(n)} \pcbasis_l}
\label{eqn:momentum-balance}
\end{align}
The ensemble average of the momentum flux gradient is
\begin{align}
    \Ensemble{\frac{F_{i+1/2}^{(n)}
    -
    F_{i-1/2}^{(n)}}{\Delta x} \pcbasis_l}
    &=
    \frac{g}{2 \Delta x}
    \sum_{p=0}^P \sum_{\palt=0}^P
    \left( h^\star_{i+1/2,p} \, h^\star_{i+1/2,\palt} -
    h^\star_{i-1/2,p} \, h^\star_{i-1/2,\palt} \right)
    \Ensemble{\pcbasis_p \pcbasis_\palt \pcbasis_l}
    \label{eqn:momentum-flux-unfactorised}
%
\intertext{To be able to factorise equation~\eqref{eqn:momentum-flux-unfactorised}, the first step is to add then subtract the term $h^\star_{i-1/2,p} h^\star_{i+1/2,s}$ to yield}
%
    \Ensemble{\frac{F_{i+1/2}^{(n)}
    -
    F_{i-1/2}^{(n)}}{\Delta x} \pcbasis_l}
    &=
    \frac{g}{2 \Delta x}
    \sum_{p=0}^P \sum_{\palt=0}^P
    \left( h^\star_{i+1/2,p} \, h^\star_{i+1/2,\palt} -
    h^\star_{i-1/2,p} \, h^\star_{i-1/2,\palt} + \right. \nonumber \\
    &\left.
    \hspace{6em}
    h^\star_{i-1/2,p} \, h^\star_{i+1/2,s} -
    h^\star_{i-1/2,p} \, h^\star_{i+1/2,s} \right)
    \Ensemble{\pcbasis_p \pcbasis_\palt \pcbasis_l}
    \label{eqn:momentum-flux-4term}
%
\intertext{Using the distributivity property of multiplication, and associativity of summation, equation~\eqref{eqn:momentum-flux-4term} can be rewritten as}
%
    \Ensemble{\frac{F_{i+1/2}^{(n)}
    -
    F_{i-1/2}^{(n)}}{\Delta x} \pcbasis_l}
    &=
    \frac{g}{2 \Delta x}
    \left[
    \sum_{p=0}^P \sum_{\palt=0}^P
    \left( h^\star_{i+1/2,p} \, h^\star_{i+1/2,\palt} -
    h^\star_{i-1/2,p} \, h^\star_{i-1/2,\palt} +
    h^\star_{i-1/2,p} \, h^\star_{i+1/2,s}
    \right)
    \Ensemble{\pcbasis_p \pcbasis_\palt \pcbasis_l}
    \right.
    \nonumber \\
    &\left.
    \hspace{2.5em}
    -
    \sum_{p=0}^P \sum_{\palt=0}^P
    h^\star_{i-1/2,p} \, h^\star_{i+1/2,s}
    \Ensemble{\pcbasis_p \pcbasis_\palt \pcbasis_l} \right]
    \label{eqn:momentum-flux-split}
%
\intertext{
The final term in equation~\eqref{eqn:momentum-flux-split} can be rewritten as $\sum_{p=0}^P \sum_{s=0}^P h^\star_{i-1/2,s} \, h^\star_{i+1/2,p} \Ensemble{\pcbasis_p \pcbasis_s \pcbasis_l}$ due to commutativity of summation and product operators as well as the basis function (equation~\ref{eqn:commutative}).
Then, by associativity of summation:
}
%
    \Ensemble{\frac{F_{i+1/2}^{(n)}
    -
    F_{i-1/2}^{(n)}}{\Delta x} \pcbasis_l}
    &=
    \frac{g}{2 \Delta x}
    \sum_{p=0}^P \sum_{\palt=0}^P
    \left( h^\star_{i+1/2,p} \, h^\star_{i+1/2,\palt} -
    h^\star_{i-1/2,p} \, h^\star_{i-1/2,\palt} + \right. \nonumber \\
    &\left.
    \hspace{6em}
    h^\star_{i-1/2,p} \, h^\star_{i+1/2,s} -
    h^\star_{i-1/2,s} \, h^\star_{i+1/2,p} \right)
    \Ensemble{\pcbasis_p \pcbasis_\palt \pcbasis_l}
    \label{eqn:momentum-flux-4term-reordered}
%
\intertext{Finally, equation~\eqref{eqn:momentum-flux-4term-reordered} can be factorised as}
%
    \Ensemble{\frac{F_{i+1/2}^{(n)}
    -
    F_{i-1/2}^{(n)}}{\Delta x} \pcbasis_l}
    &=
    \frac{g}{2 \Delta x}
    \sum_{p=0}^P \sum_{\palt=0}^P
    \left( h^\star_{i-1/2,p} + h^\star_{i+1/2,p} \right)
    \left( h^\star_{i+1/2,s} - h^\star_{i-1/2,s} \right)
    \Ensemble{\pcbasis_p \pcbasis_\palt \pcbasis_l} \\
%
\intertext{and, given that $h^\star_{i+1/2,s} = \eta_s - \zmodified_{i+1/2,s}$ and $h^\star_{i-1/2,s} = \eta_s - \zmodified_{i-1/2,s}$, then}
    \Ensemble{\frac{F_{i+1/2}^{(n)}
    -
    F_{i-1/2}^{(n)}}{\Delta x} \pcbasis_l}
    &=
    -\frac{g}{\Delta x}
    \sum_{p=0}^P \sum_{\palt=0}^P
    \left(
    \frac{h^{+,\star}_{i-1/2,p} + h^{-,\star}_{i+1/2,p}}{2}
    \right)
    \left( \zmodified_{i+1/2,\palt} - \zmodified_{i-1/2,\palt} \right)
    \Ensemble{\pcbasis_p \pcbasis_\palt \pcbasis_l}
    \label{eqn:momentum-flux}
\end{align}
Equation~\eqref{eqn:momentum-flux} is equal to the ensemble average of the bed slope source term $\Ensemble{S_i^{(n)} \pcbasis_l}$ given in equation~\eqref{eqn:pc-source}, hence discrete balance is preserved.


\section{Rule Derivation}

Rules derived are stored in \textit{rule\_base.txt}, together with the definitions of fuzzy sets.

\subsection{Manual Rule Derivation}

After observing a couple of patterns from data in the prototyping set and reasoning about their validities, we come up with several heuristics.

First we notice that $Extent$ is a very good feature to start with: triangles have a $Extent$ ratio close to 0.5, rectangles and squares have a $Extent$ ratio close to 1, while circles and ellipses have a $Extent$ ratio of around 0.75 $\sim$ 0.85.

Then we can use $Thinness$ ratio to distinguish circles and ellipses, because circle has the smallest thinness ($=4\pi$) among all 2D figures,.

Similarily, when rectangles have $Thinness$ ratio close to 16, they become squares.

For the feature $Corners$, as mentioned in Section 3.3, it's a very shaky feature for our task. Therefore we decide not to use it.

Finally, we obtain 5 simple rules:

\begin{enumerate}

\item \textit{IF Thinness IS LIKE Circle AND Extent IS LIKE ellipse THEN Shape IS Circle}
\item \textit{IF Thinness IS NOT LIKE Circle AND Extent IS LIKE ellipse THEN Shape IS Ellipse}
\item \textit{IF Extent IS LIKE Triangle THEN Shape IS Triangle}
\item \textit{IF Thinness IS LIKE Square AND Extent IS LIKE rectangle THEN Shape IS Square}
\item \textit{IF Thinness IS NOT LIKE Square AND Extent IS LIKE rectangle THEN Shape IS Rectangle}

\end{enumerate}

\subsection{Autonomous Rule Derivation}

We will discuss the possibility of autonomous rule derivation in Section 7.


\chapter{Feature Selection}

\newthought{Sometimes, measuring features is expensive} or training a model needs to be fast yet accurate. It can also happen that a large number of features make the model difficult to interpret. Hence it is handy to know which features are actually important for the model and which are redundant. And in order to determine which are those features, we use a process called feature scoring.

\begin{wrapfigure}{o}{0.6\textwidth}
    \vspace{-0.5cm}
    \includegraphics[scale=0.4]{workflow.png}
\end{wrapfigure}

There are several scoring methods, some of which are univariate (they look at the relationship between a single feature and the target variable), some are multivariate (they can consider multiple features), and some are model-based (they use models to estimate feature importance).

One such scoring technique is called \textit{information gain}. Let us consider feature X and the target variable Y. Information gain is a univariate method measuring the amount of information gained Y knowing the value of X.

%box plot
A similar method is \textit{Gini decrease}, which also measures decrease in impurity after observing the variable (X). They both favor multilabel features, because they tend to split the data more resulting in purer subsets. This was the reason behind conception of \textit{Relief} (later improved to ReliefF), which handles multilabel and numeric data better. It also has one important advantage - it is a multivariate method, which means it can handle the so-called XOR problems\marginnote{An XOR problem is a problem where the final class would be a result of }

\begin{figure}[h]
    \centering
    \includegraphics[scale=0.4]{test_and_score.png}
    \caption{Cross validation splits the data sets into, say, 10 different non-overlapping subsets we call folds. In each iteration, one fold will be used for testing, while the data from all other folds will be used for training. In this way, each data instance will be used for testing exactly once.}
\end{figure}