\documentclass[../main.tex]{subfiles}

\begin{document}

\section{Z Notation Introduction}

The following subsections provide a high level overview of select properties of Z Notation based on
"The Z Notation: A Reference Manual" by J. M. Spivey. A copy
of this reference manual can be found at dave/docs/z/Z-notation reference manual.pdf.
In many cases, definitions will be pulled directly from the reference manual and when this occurs,
the relevant page number(s) will be included. For a proper introduction with tutorial examples, see
chapter 1, "Tutorial Introduction" from pages 1 to 23. For the $LaTeX$ symbols used to write Z,
see the reference document found at dave/docs/z/zed-csp-documentation.pdf.

\subsection{Decorations}
The following decorations are used throughout this document and are taken directly from the reference manual.
For a complete summary of the Syntax of Z, see chapter 6, Syntax Summary, starting on page 142.
\begin{argue}
  ' & indicates final state of an operation \\
  ? & indicates input to an operation \\
  ! & indicates output of an operation \\
  \Delta & indicates the schema results in a change to the state space \\
  \Xi & indicates the schema does not result in a change to the state space \\
  \pipe & indicates output of the left schema is input to the right schema
\end{argue}

\subsection{Types}
Objects have a type which characterizes them and distinguish them from other kinds of objects.
\begin{itemize}
\item Basic types are sets of objects which have no internal structure of interest meaning the concrete definition of the members is not relevant, only their shared type.
\item Free types are used to describe (potentially nested and/or recursive) sets of objects. In the most simple case, a free type can be an enumeration of constants.
\end{itemize}
Within the xAPI Formal Specification, both of these types are used to describe the
\href{https://github.com/adlnet/xAPI-Spec/blob/master/xAPI-Data.md#inversefunctional}{Inverse Functional Identifier}
property.
\begin{itemize}
\item Introduction of the basic types $MBOX$, $MBOX\_SHA1SUM$, $OPENID$ and $ACCOUNT$
  allows the specification to talk about these constraints within the xAPI
  specification without defining their exact structure
\item The free type $IFI$ is defined as one of the above basic types meaning an object
  of type $IFI$ is of type $MBOX$ or $MBOX\_SHA1SUM$ or $OPENID$ or $ACCOUNT$.
\end{itemize}
Types can be composed together to form composite types and thus complex objects.
\begin{zed}
  [MBOX, MBOX\_SHA1SUM, OPENID, ACCOUNT]
  \also
  IFI ::= MBOX \,|\, MBOX\_SHA1SUM \,|\, OPENID \,|\, ACCOUNT
\end{zed}
Within the xAPI Formal Specification, $IFI$ is used within the definition
of an $agent$ as presented in the schema $Agent$.

\begin{schema}{Agent}
  agent : AGENT \\
  objectType : OBJECTTYPE \\
  name : \finset_1 \#1 \\
  ifi : IFI
  \where
  objectType = Agent \\
  agent = \{ifi\} \cup \power \{name, objectType\}
\end{schema}
See section 2.2, pages 28 to 34, and chapter 3, pages 42 to 85, for more information about Schemas and the Z Language.

\subsection{Sets}
A collection of elements that all share a type. A set is characterized solely by which objects are members and which are not.
Both the order and repetition of objects are ignored. Sets are written in one of two ways:
\begin{itemize}
\item listing their elements
\item by a property which is characteristic of the elements of the set.
\end{itemize}
such that the following law from page 55 holds for some object y
$$y \in \{x_{1},...,x_{n}\} \iff y = x_{1} \lor ... \lor y = x_{n}$$

\subsection{Ordered Pairs}
Two objects $(x, y)$ where $x$ is paired with $y$. An n-tuple is
the pairing of n objects together such that equality between two n-tuple pairs
is given by the law from page 55
$$(x_{1},...,x_{n}) = (y_{1},...,y_{n}) \iff x_{1} = y_{1} \land ... \land x_{n} = y_{n}$$
When ordered pairs are used with respect to application (as seen on page 60)
$$f x \implies f(x) \iff (x,y) \in f$$
which states that $f(x)$ is defined if and only if there is a unique value $y$ which result from $f x$
Additionally, application associates to the left
$$f x y \implies (f x) y \implies (f(x), y)$$
meaning $f(x)$ results in a function which is then applied to $y$.

\subsection{Sequences}
A collection of elements where their ordering matters such that
$$\langle a_{1},...,a_{n} \rangle \implies \{1 \mapsto a_{1}, ..., n \mapsto a_{n}\}$$
as seen on page 115. Additionally, $\iseq$ is used to describe a sequence whose members are distinct.

\subsection{Bags}
A collection of elements where the number of times an element appears in the collection is meaningful.
$$\lbag a_{1},...,a_{n} \rbag \implies \{a_{1} \mapsto k_{1},...,k_{n} \mapsto k_{n}\}$$
As described on page 124, each element $a_{i}$ appears $k_{i}$ times in the list $a_{1},...,a_{n}$
such that the number of occurrences of $a_{i}$ within bag $A$ is returned by
$$count ~A ~a_{i} \ \equiv A ~\# ~a_{i}$$

\subsection{Maps}
This document introduces a named subcategory of sets, $map$ of the free type $KV$,
which are akin to sequences and bags. To enumerate the members of a $map$, $\ldata ... \rdata$ is used
but should not be confused with $d_{i}\ldata E_{i}[T]\rdata$ within a Free Type definition. The
distinction between the two usages is context dependent but in general, if $\ldata ... \rdata$
is used outside of a constructor declaration within a Free Type definition,
it should be assumed to represent a $map$.
$$KV ::= base ~| ~associate\ldata KV \cross X \cross Y \rdata$$
where
\begin{argue}
  base & is a constant which is the empty $KV \implies \ldata \rdata$ \\
  associate & is a constructor and is inferred to be an injection
\end{argue}
The full enumeration of all properties, constraints and functions
specific to a $map$ with type $KV$ will be defined elsewhere but
$associate$ can be understood to (in the most basic case) operate as follows.
$$associate(base, x_{i}, y_{i}) = \ldata (x_{i}, y_{i}) \rdata \implies \ldata x_{i} \mapsto y_{i} \rdata$$
The enumeration of a $map$ was chosen to be $\ldata ... \rdata$ as a $map$ is a collection of injections
such that if $M$ is the result of $associate(base, x_{i}, y_{i})$ from above then
$$atKey(M, x_{i}) = y_{i} \iff x_{i} \mapsto y_{i} ~\land (x_{i}, y_{i}) \in M$$

\subsection{Select Operations and Symbols}
The follow are defined in Chapter 4 (The Mathematical Tool-kit) within the reference manual
and are used extensively throughout this document. In many cases, the functions listed here
will serve as Operations in the context of Primitives and Algorithms.

\subsubsection{Functions}
\begin{argue}
  \pfun & relate each x $\in$ X to at most one y $\in$ Y, page 105 \\
  \fun & relate each x $\in$ X to exactly one y $\in$ Y, page 105 \\
  \pinj & map different elements of x to different y, page 105 \\
  \inj & $\pinj$ that are also $\fun$, page 105 \\
  \psurj & $X \pfun Y$ where whole of Y is the range, page 105 \\
  \surj & $X \pfun Y$ whole of X as domain and whole of Y as range, page 105 \\
  \bij & map x $\in$ X one-to-one with y $\in$ Y, page 105
\end{argue}
\begin{zed}
  X \pfun Y == \{~f : X \rel Y ~| ~(\forall x : X; y1, y2 : Y @ \\
  \t5 (x \mapsto y_{1} \in f \ \land \ (x \mapsto y_{2}) \in f \implies y_{1} = y_{2}))\} \\
  X \fun Y == \{~f : X \pfun Y ~| ~\dom f = X\} \\
  X \pinj Y == \{~f : X \pfun Y ~| ~(\forall x_{1}, x_{2} : \dom f @ f(x_{1}) = f(x_{2}) \implies x_{1} = x_{2})\} \\
  X \inj Y == (X \pinj Y) \cap (X \fun Y) \\
  X \psurj Y == \{~f : X \pinj Y ~| ~\ran f = Y\} \\
  X \surj Y == (X \psurj Y) \cap (X \fun Y) \\
  X \bij Y == (X \surj Y) \cap (X \inj Y)
\end{zed}

\subsubsection{Ordered Pairs, Maplet and Composition of Relations}
\begin{argue}
  first & returns the first element of an ordered pair, page 93 \\
  second & returns the second element of an ordered pair, page 93 \\
  \mapsto & maplet is a graphic way of expressing an ordered pair, page 95 \\
  \dom & set of all x $\in$ X related to at least one y $\in$ Y by R, page 96 \\
  \ran & set of all y $\in$ Y related to at least one x $\in$ X by R, page 96 \\
  \comp & The composition of two relationships, page 97 \\
  \circ & The backward composition of two relationships, page 97
\end{argue}

\begin{gendef}[X,Y]
  first: X \cross Y \fun X \\
  second: X \cross Y \fun Y
  \where
  \forall x: X; y: Y @ \\
  \t1 first(x,y) = x \ \land \\
  \t1 second(x,y) = y
\end{gendef}

\begin{gendef}[X,Y]
  \_\mapsto\_: X \cross Y \fun X \cross Y
  \where
  \forall x:X; y:Y @ \\
  x \mapsto y = (x, y)
\end{gendef}

\begin{gendef}[X,Y]
  \dom : (X \rel Y) \fun \power X \\
  \ran : (X \rel Y) \fun \power Y
  \where
  \forall R : X \rel Y @ \\
  \t1 \dom R = \{~x : X; ~y : Y ~| ~x \underline{R} y @ x\} ~\land \\
  \t1 \ran R = \{~x : X; ~y : Y ~| ~x \underline{R} y @ y\}
\end{gendef}

\begin{gendef}[X,Y,Z]
  \_\comp\_: (X \rel Y) \cross (Y \rel Z) \fun (X \rel Z) \\
  \_\circ\_: (Y \rel X) \cross (X \rel Y) \fun (X \rel X)
  \where
  \forall ~Q : X \rel Y; R : Y \rel Z @ \\
  \t1 Q \comp R = R \circ Q = \{~x : X; y : Y; z : Z | \\
  \t5  x ~\underline{Q} ~y \ \land \ y ~\underline{R} ~z @ x \mapsto z\}
\end{gendef}

\subsubsection{Numeric}

\begin{argue}
  succ & the next natural number, page 109\\
  .. & set of integers within a range, page 109 \\
  \# & number of members of a set, page 111 \\
  \min & smallest number in a set of numbers, page 113 \\
  \max & largest number in a set of numbers, page 113
\end{argue}

\begin{axdef}
  succ : \nat \fun \nat \\
  \_~..~\_ : \num \cross \num \fun \power \num
  \where
  \forall n : \nat @ succ(n) = n + 1 \\
  forall a,b : \num @ \\
  \t2 a~..~b = \{~k : \num ~| ~a \leq k \leq b\}
\end{axdef}

\begin{gendef}[X]
  \# : \finset X \fun \nat
  \where
  \forall S : \finset X @ \\
  \t1 \#S = (\mu ~n : \nat ~| ~(\exists f : 1~..~n \inj X @ \ran f = S))
\end{gendef}

\begin{axdef}
  \min : \power_1 \num \pfun \num \\
  \max : \power_1 \num \pfun \num
  \where
  \min = \{~S : \power_1 \num; m : \num ~| \\
  \t2 m \in S ~\land ~(\forall n : S @ m \leq n) @ S \mapsto m\} \\
  \max = \{~S : \power_1 \num; m : \num ~| \\
  \t2 m \in S ~\land ~(\forall n : S @ m \geq n) @ S \mapsto m\}
\end{axdef}

\subsubsection{Sequences}
\begin{argue}
  \cat & concatenation of two sequences, page 116 \\
  rev & reverse a sequence, page 116 \\
  head & first element of a sequence, page 117 \\
  last & last element of a sequence, page 117 \\
  tail & all elements of a sequence except for the first, page 117 \\
  front & all elements of a sequence except for the last, page 117 \\
  \extract & sub seq based on provided indices, order maintained, page 118\\
  \filter & sub seq based on provided condition, order maintained, page 118 \\
  squash & compacts a fn of positive integers into a sequence, page 118 \\
  \dcat & flatten seq of seqs into single seq, page 121 \\
  \disjoint & pairs of sets in family have empty intersection, page 122 \\
  \partition & union of all pairs of sets = the family set, page 122
\end{argue}

\begin{gendef}[X]
  \_~\cat~\_ : \seq X \cross \seq X \fun \seq X \\
  rev : \seq X \fun \seq X
  \where
  \forall s, t : \seq X @ \\
  \t1 s \cat t = s \cup \{~n : \dom t @ n + \# s \mapsto t(n)\} \\
  \forall s : \seq X @ \\
  \t1 rev s = (\lambda ~n : \dom s @ s(\#s - n + 1))
\end{gendef}

\begin{gendef}[X]
  head, last : \seq_1 X \fun X \\
  tail, front : \seq_1 X \fun \seq X
  \where
  \forall s : \seq_1 X @ \\
  \t1 head ~s = s(1) ~\land \\
  \t1 last ~s = s(\#s) ~\land \\
  \t1 tail ~s = (\lambda ~n : 1 ~..~\#s - 1 @ s(n + 1)) ~\land \\
  \t1 front ~s = (1~..~\#s - 1) \dres s
\end{gendef}

\begin{gendef}[X]
  \_~\extract~\_ : \power \nat_1 \cross \seq X \fun \seq X \\
  \_~\filter~\_ : \seq X \cross \power X \fun \seq X \\
  squash : (\nat_1 \pfun X) \fun \seq X
  \where
  \forall U : \power \nat_1; s : \seq X @ \\
  \t1 U \extract s = squash(U \dres s) \\
  \forall s : \seq X; V : \power X @ \\
  \t1 s \filter V = squash(s \rres V) \\
  \forall f : \nat_1 \pfun X @ \\
  \t1 squash f = f ~ \circ ~(\mu ~p : 1~..~\# f \bij \dom f ~| ~p \circ succ \circ p \inv \subseteq (\_ ~< ~\_))
\end{gendef}

\begin{gendef}[X]
  \dcat : \seq(\seq X) \fun \seq X
  \where
  \dcat \langle \rangle = \langle  \rangle \\
  \forall s : \seq X @ \dcat \langle s \rangle = s \\
  \forall q,r : \seq(\seq X) @ \\
  \t1 \dcat (q \cat r) = (\dcat q) \cat (\dcat r)
\end{gendef}

\begin{gendef}[I,X]
  \disjoint ~\_ : \power (I \pfun \power X) \\
  ~\_ ~\partition ~\_ : (I \pfun \power X) \rel \power X
  \where
  \forall S : I \pfun \power X; T : \power X @ \\
  \t1 (\disjoint S \iff \\
  \t2 (\forall i,j : \dom S ~| ~i \not= j @ S(i) \cap S(j) = \emptyset))~ \land \\
  \t1 (S \partition T \iff \\
  \t2 \disjoint S ~ \land ~ \bigcup \{~i : \dom S @ S(i)\} = T)
\end{gendef}

\subsubsection{Bags}

\begin{argue}
  count, \# & the number of times something appears in a bag, page 124 \\
  \otimes & scaling across a bag, page 124 \\
  \uplus & union of two bags, sum of occurrences, page 126 \\
  \uminus & bag difference, subtract occurrences or zero if negative, page 126 \\
  items & conversion from $\seq$ to $\bag$, page 127
\end{argue}

\begin{gendef}[X]
  count : \bag X \bij (X \fun \nat) \\
  \_ ~ \# ~ \_ : \bag X \cross X \fun \nat \\
  \_ ~ \otimes ~\_ : \nat \cross \bag X \fun \bag X
  \where
  \forall B : \bag X @ \\
  \t1 count B = (\lambda x : X @ 0) \oplus B \\
  \forall x : X; B : \bag x @ \\
  \t1 B ~\# ~x = count ~B ~x \\
  \forall n : \nat ; B : \bag X; x : X @ \\
  \t1 (n \otimes B) ~\# ~x = n * (B ~\# ~x)
\end{gendef}

\begin{gendef}[X]
  \_ ~\uplus ~\_ ~, ~\_ ~ \uminus ~\_ : \bag X \cross \bag X \fun \bag X
  \where
  \forall B ~, ~C : \bag X; x : X @ \\
  \t1 (B ~ \uplus ~C) ~\# ~x = B ~ \# ~ x + C ~ \# x ~ \land \\
  \t1 (B ~ \uminus ~C) ~\# ~ x = \max \{B ~\# x ~- ~C ~\# ~x, 0\}
\end{gendef}

\begin{gendef}[X]
  items : \seq X \fun \bag X
  \where
  \forall s : \seq X; x : X @ \\
  \t1 (items ~s) ~\# ~x = \#\{~i : \dom s ~| ~s(i) = x\}
\end{gendef}

\end{document}


\section{Sybil Resilience}
  One of our aims is to mitigate Sybil attacks \cite{sybilattack} whilst maintaining decentralized autonomy \cite{dionyziz}.
  We begin by extending the definition of indirect trust.
  \subimport{common/definitions/}{indirecttrustmultiplayer.tex}
  \noindent We now extend the Trust Flow theorem to many players.
  \subimport{common/theorems/}{multiplayertrustflowtheorem.tex}
  \subimport{common/proofs/}{multiplayertrustflowproof.tex}
  \noindent We now define several useful notions to tackle the problem of Sybil attacks. Let Eve be a possible attacker.
  \subimport{common/definitions/}{corrupted.tex}
  \subimport{common/definitions/}{sybil.tex}
  \subimport{common/definitions/}{collusion.tex}
  \subimport{common/figures/}{collusion.tikz}
  From a game theoretic point of view, players $\mathcal{V} \setminus (\mathcal{B} \cup \mathcal{C})$ perceive the collusion
  as independent players with a distinct strategy each, whereas in reality they are all subject to a single strategy dictated
  by Eve.
  \subimport{fc17/theorems/}{sybilrestheorem.tex}
  \subimport{fc17/proofsketches/}{sybilresproofsketch.tex}
  We have proven that controlling $|\mathcal{C}|$ is irrelevant for Eve, thus Sybil attacks are meaningless. Note that
  the theorem does not reassure against deception attacks. Specifically, a malicious player can create several identities, use
  them legitimately to inspire others to deposit direct trust to these identities and then switch to the evil strategy, thus
  defrauding everyone that trusted the fabricated identities. These identities correspond to the corrupted set of players and
  not to the Sybil set because they have direct incoming trust from outside the collusion.

  In conclusion, we have delivered on our promise of a Sybil-resilient decentralized financial trust system with invariant
  risk for purchases.


\section{Machine learning techniques}\label{sec:machine-learning-techniques}
In modern cyber security, there exist many different threats that potentially expose the system to being compromised.
Many of them are very sophisticated, some are pretty difficult to distinguish from legitimate operations.
Nowadays, there is a trend to use machine learning approaches to meet the high demands for the quality and reliability of security systems.

As stated by Tony Thomas et al. --- "Machine learning (\gls{ml}) may be defined as the ability of machines to learn without being explicitly programmed.
Using mathematical techniques across cyberdata, \gls{ml} algorithms can build models of behaviors and use those models as a basis for making predictions on newly input data"~\cite{thomas2020machine}.
This behavior of machine learning techniques is very convenient, especially when the domain of the problem and the borders between the data cannot be explicitly expressed by the written program.
Machine learning models can learn from their own mistakes and then recognize or even predict the future attacks~\cite{thomas2020machine}.

In scope of sequential data with natural timespan interpretation such us mouse actions, the recursive neural networks are natural choice.
As stated by Chong et al. --- "Given the sequential nature of mouse movement data, a recurrent neural network (\gls{rnn}), commonly employed for time series structure data, would be an intuitive choice for tackling this problem"~\cite{Main}.
However, if the data can be represented as a bit map or a picture, the convolutional neural networks become possible to use.
Due to the size of picture representation in a computer world, it is pretty expensive in the meaning of computational cost to use plain artificial neural networks because the number of weights in such a network becomes tremendous.
Convolutional neural networks address this issue and make the computation significantly faster and efficient.

Keiron O'Shea et. al in \cite{cnn-description} define an architecture of \gls{cnn}'s as a connected network of three types layers: convolutional layer, pooling layer and fully-connected layer.
The convolutional layer is combined out of filters that represent features of the images that the given filter should recognize.
These filters are then used on the different regions of images from the dataset by performing convolution.
The output reflects the found matches between the image that is recognized and the one described by the filter.
Pooling is a technique used to decrease the size of the image by grouping pixels and returning the representative value for this group.
As an example, the max-pooling bases on returning the pixel with the highest value in the group.
The fully-connected layer is a layer that builds plain artificial neural networks and it is built out of the neurons that are connected to the neurones from the previous layer, but they are not connected inside the current layer.
Thomas et. al add to these layers so-called rectified linear unit layer (\gls{relu}) and define its responsibility as "changing the negative pixel values in the image to zero, which gives us another stack of images with no negative values"~\cite{thomas2020machine}.
This layer is called an activation function because it activates the next layer only if the value of the pixel is positive.

The problem that this work raises can be specified as a binary problem because there exist only two possible categories for the data --- user's and bot actions.
In such situations, the measure of quality and correctness of the solution is commonly defined by a confusion matrix.
This matrix defines the performance of the model and consists of several measures: true positives, true negatives, false positives and false negatives.
True positives (\gls{tp}) are defined as the number of samples that the model assign to the positive category and the assignment is correct.
The opposite to them are false positives (\gls{fp}), which can be described as faulty categorized to the positive category.
The true negatives (\gls{tn}) and false negatives (\gls{fn}) are analogous, but the assigned category is negative.

Basing on the described measures, one can define relative measures --- false rejection rate (\gls{frr}) and false acceptance rate (\gls{far}).
These are defined as follows:

\begin{samepage}
\begin{equation}
    FRR = \frac{FN}{FN + TP}\label{eq:frr}
\end{equation}
\begin{equation}
    FAR = \frac{FP}{FP + TN}\label{eq:far}
\end{equation}
\end{samepage}

The values of \gls{frr} and \gls{far} are often expressed as a percentage value and the smaller their values, the better the performance of the model.
In authorization related problems such as raised in this work, there is a desire to have those values as low as possible because low \gls{far} defines the good security level of the system --- the lower it is, the less unauthorized users have access to the system, and accordingly \gls{frr} has an impact on authorization rejections of legitimate users.

In the presented solution, the authors use a transfer learning technique.
As stated in the "A survey of transfer learning" by Weiss et al. --- "In certain scenarios, obtaining training data that matches the feature space and predicted data distribution characteristics of the test data can be difficult and expensive.
Therefore, there is a need to create a high-performance learner for a target domain trained from a related source domain.
This is the motivation for transfer learning.
Transfer learning is used to improve a learner from one domain by transferring information from a related domain"~\cite{transfer-learning-def}.
This approach provides several conveniences, such as computational time and cost reduction.
The pretrained network does not require such a long time as in the case of training from scratch which also translates into the cost of calculations and in the end does not require a large computational grant.
The main advantage of such an approach is the reduction of the required amount of representative data.
In cases where the data gathering is difficult or the collected dataset is small, transfer learning seems to be a technique that is worth considering.
As a transfer learning model, the authors chose the InceptionV3\upperref{itm:inceptionV3} architecture from Tensorflow Hub\upperref{itm:tensorflow-hub}, which originally was trained on ImageNet\upperref{itm:image-net} dataset.
Basing on several benchmarks, it was considered that the described architecture fits well to the raised problem and the performance is great at the same time.
The architecture is build out of convolutional layers, average and max pooling, dropouts, concatenation layers and fully-connected layers.
Description of the InceptionV3 architecture is out of the scope of this work.

In order to perform the learning process of the model, the main dataset should be divided into two subsets --- training and test datasets.
The network basing on the output of the loss function is able to improve its predictions and make progress in classification.
In the case of the supervised learning, where the network learns from the given examples, there is also a requirement for the assignment of the labels that are considered as descriptive ones for the input sample.
When the gathered data consist of several categories, among which there is a dominant class in the meaning of volume of samples, such dataset is considered as an imbalanced one.
This issue has a negative impact on the learning process and should be resolved in order to improve the performance of the model.


\documentclass[a4paper,man,natbib]{apa6}

\usepackage[english]{babel}
\usepackage[utf8x]{inputenc}

% Common Packages - Delete Unused Ones %
\usepackage{setspace}
\usepackage{amsmath}

\usepackage[cache=false]{minted}
\usepackage{graphicx}
\usepackage{caption}
\graphicspath{ {./images/} }
% End Packages %

\title{Exercise Assignment 5}
\shorttitle{ES5}
\author{Brandon Hosley}
\date{2018 09 25}
\affiliation{Mike Davis}

%\abstract{}

\begin{document}
\maketitle
\singlespacing
\subsection{25a}
\emph{Assuming nine-bit two's complement representation, convert each of the following decimal numbers to binary, show the effect of the ASL operation on it, and
	then convert the result back to decimal. Repeat with the ASR operation:} \\
94 \\
BIN $\rightarrow$ 0 0101 1110 \\
ASL $\rightarrow$ 0 1011 1100 \\
DEC $\rightarrow$ 188 \\
ASR $\rightarrow$ 0 0101 1110 $\rightarrow$ 94 \\
\subsection{25c}
\emph{Assuming nine-bit two's complement representation, convert each of the following decimal numbers to binary, show the effect of the ASL operation on it, and
	then convert the result back to decimal. Repeat with the ASR operation:} \\
-62 \\
BIN $\rightarrow$ 1 1100 0010 \\
ASL $\rightarrow$ 1 1000 0100 \\
DEC $\rightarrow$ -124
ASR $\rightarrow$ 0 1100 0010$_{2}$ $\rightarrow$ 194$_{10}$
\subsection{26a}
\emph{Write the RTL specification for an arithmetic shift right on a six-bit cell.} \\
$c\rightarrow r\langle 0\rangle ,r \langle 0...5\rangle\rightarrow r\langle 1...6\rangle ,r\langle 6\rangle\rightarrow C$ \\
\subsection{26b}
\emph{Write the RTL specification for an arithmetic shift left on a 16-bit cell.} \\
$C\leftarrow r\langle 0\rangle ,r \langle 0...15\rangle\leftarrow rlangle 1...16\rangle ,r\langle 6\rangle\leftarrow c$
\subsection{28a}
\emph{C = 1, ROL 0 0110 1101} \\
C=0, 0 1101 1011 \\
\subsection{28b}
\emph{C = 0, ROL 0 0110 1101} \\
C=0, 0 1101 1010 \\
\subsection{35b}
\emph{Assuming nine-bit two's complement binary representation, convert the following numbers from hexadecimal to decimal. Remember to check the sign bit: 0F5} \\
0F5$_{16}$ $\rightarrow$ 0 1111 0101$_{2}$ $\rightarrow$ 245$_{10}$
\subsection{35c}
\emph{Assuming nine-bit two's complement binary representation, convert the following numbers from hexadecimal to decimal. Remember to check the sign bit: 100} \\
100$_{16}$ $\rightarrow$ 1 0000 0000$_{2}$ $\rightarrow$ -255$_{10}$
\subsection{53b}
\emph{For IEEE 754 single precision floating point, write the hexadecimal representation for the following decimal values: -1.0} \\
-1.0$_{10}$ $\rightarrow$ 1 0111 1111 000 0000 0000 0000 0000 0000$_{2}$ $\rightarrow$ BF800000$_{16}$
\subsection{53c}
\emph{For IEEE 754 single precision floating point, write the hexadecimal representation for the following decimal values: -0.0} \\
-0.0$_{10}$ $\rightarrow$ 1 0000 0000 000 0000 0000 0000 0000 0000$_{2}$ $\rightarrow$ 80000000$_{16}$

\nocite{warford10}
\bibliographystyle{apacite}
\bibliography{CS} %link to relevant .bib file

\end{document}


% !TeX root = RJwrapper.tex
\title{Connecting R with D3 for dynamic graphics, to explore multivariate data
with tours}
\author{by Michael Kipp, Ursula Laa, Dianne Cook}

\maketitle

\abstract{%
The tourr package in R has several algorithms and displays for showing
multivariate data as a sequence of low-dimensional projections. It can
display as a movie but has no capacity for interaction, such as stop/go,
change tour type, drop/add variables. The tourrGui package provides
these sorts of controls, but the interface is programmed with the dated
RGtk2 package. This work explores using custom messages to pass data
from R to D3 for viewing, using the Shiny framework. This is an approach
that can be generally used for creating all sorts of interactive
graphics.\\
}

\hypertarget{introduction}{%
\subsection{Introduction}\label{introduction}}

Did you know you can run \emph{any javascript you like} in a Shiny
application and you can pass \emph{whatever you want including JSON}
back and forth? This massively widens the scope of what you can do with
Shiny, and generating a tour of multivariate data with this approach is
a really good example of what is possible.

The tour algorithm (Cook et al. \protect\hyperlink{ref-gt_pp}{1995},
\protect\hyperlink{ref-gt_pp_mc}{2007}) is a way of systematically
generating and displaying projections of high-dimensional spaces in
order for the viewer to examine the multivariate distribution of data.
It can do this either randomly, or by picking projections judged
interesting according to some criterion or index function. The tourr
package (Wickham et al. \protect\hyperlink{ref-tourr}{2011}) provides
the computing and display in R (R Core Team
\protect\hyperlink{ref-R}{2018}; Ihaka and Gentleman
\protect\hyperlink{ref-ihaka:1996}{1996}) to make several types of
tours: grand, guided, little and local. The projection dimension can be
chosen between one and the number of variables in the data. The display,
though, has no capacity for interaction. The viewer can watch the tour
like a movie, but not pause it and restart, or change tour type, or
number of variables.

These interactive controls were provided with the tourrGui package
(Huang, Cook, and Wickham \protect\hyperlink{ref-tourrGui}{2012}), with
was programmed with the RGtk2 package (Lawrence and Temple Lang
\protect\hyperlink{ref-RGtk2}{2010}). This is not the toolkit of choice
today, and has been superceded with primarily web-capable tools, like
Shiny (Chang et al. \protect\hyperlink{ref-shiny}{2017}). To display
dynamic graphics though, is not straight-forward. This paper explains
how to use D3 (Bostock, Ogievetsky, and Heer
\protect\hyperlink{ref-D3}{2011}) as the display engine in a Shiny
graphical user interface (GUI), using custom message passing between
server and client.

\hypertarget{creating-a-tour-with-the-tourr-package}{%
\subsection{Creating a tour, with the tourr
package}\label{creating-a-tour-with-the-tourr-package}}

The \pkg{tourr} package (Wickham et al.
\protect\hyperlink{ref-tourr}{2011}) is an R implementation of the tour
algorithms discussed in Cook et al.
(\protect\hyperlink{ref-gt_pp_mc}{2007}). It includes methods for
geodesic interpolation and basis generation, as well as an
implementation of the simulated annealing algorithm to optimise
projection pursuit indices for the guided tour. The tour can be
displayed directly in the R graphics device, for example, the code below
generates a 1D density tour. Figure \ref{tour} shows snapshots.

\begin{Schunk}
\begin{Sinput}
library(tourr)
# quartz() # to display on a Mac; X11() # For windows; The Rstudio graphics
# device is not advised
animate_dist(flea[, 1:6], center = TRUE)
\end{Sinput}
\end{Schunk}

\begin{figure}[ht]
\centerline{\includegraphics[width=5cm]{figures/tour1.png}\includegraphics[width=5cm]{figures/tour2.png}\includegraphics[width=5cm]{figures/tour3.png}}
\caption{Three projections from a 1D tour of 6D data, displayed as a density. Full video can be seen at https://vimeo.com/255466661.}
\label{tour}
\end{figure}

A tour path is a smooth sequence of projection matrices, \(p\times d\),
that when combined with a matrix of n data points, \(n\times p\), and a
rendering method, produces a steady stream of \(d\)-dimensional views of
the data. Each tour is initialised with the \texttt{new\_tour()} method,
which instantiates a tour object and takes as arguments the data \(X\),
the tour method, e.g. \texttt{guided\_tour()}, and the starting basis.
Once initialised, a new target plane is chosen, and a series of steps
along a geodesic path from starting to target plane are generated by
interpolation.

This requires a series of calls to the tour object producing the series
of projections. The steps are discrete, of size given by
\(\omega/\Delta\), where \(\omega\) denotes the angular velocity of the
geodesic interpolation, and \(\Delta\) is a parameter denoting frames
per second, reflecting the rendering speed of the device in use. The
\(\Delta\) parameter can be thought of as the frames per second, while
\(\omega\) affects the speed at which the tour moves through the
projection space. For our purposes, \(\Delta\), \texttt{fps} in the
code, is set at 25, while the \(\omega\) can be adjusted by the user.

\hypertarget{connecting-the-tour-projections-to-d3-display-using-sendcustommessage}{%
\subsection{\texorpdfstring{Connecting the tour projections to D3
display using
\texttt{sendCustomMessage}}{Connecting the tour projections to D3 display using sendCustomMessage}}\label{connecting-the-tour-projections-to-d3-display-using-sendcustommessage}}

D3.js (Data-Driven Documents) (Bostock, Ogievetsky, and Heer
\protect\hyperlink{ref-D3}{2011}) is a JavaScript library for
manipulating documents based on data. The advantages of D3 are similar
to those provided by Shiny: namely, an industry standard with rich array
of powerful, easy to use methods and widgets that can be displayed on a
wide variety of devices, with a large user base. D3 works on data
objects in the JavaScript Object Notation (JSON) format, which are then
parsed and used to display customisable data visualisations.

The new implementation of the tour interface uses D3 to render each
projection step returned by R, focusing on 2D projections as a test
case. It does this by drawing and re-drawing a scatterplot with dots (or
circles in D3 language) and providing SVG objects for the web browser to
render. Figure \ref{tourrD3} shows the new GUI.

\begin{figure*}[ht]
\centerline{\includegraphics[width=15cm]{figures/TourrD3.png}}
\caption{Shiny GUI for the tour, with D3 as the display engine. GUI provides controls to select tour type, change speed, restart, and select variables to include.}
\label{tourrD3}
\end{figure*}

There are two functions provided by the Shiny framework to transport
data between R and JavaScript: \texttt{session\$sendCustomMessage()} in
R, and the corresponding \texttt{Shiny.addCustomMessageHandler()} in
JavaScript. Whenever the former is executed in R, the latter function
will execute a code block in JS. There are many examples of such
functions being used to pass arbitrary data from an R app to a JS
front-end, few examples exist of this basic functionality to update a D3
animation in real-time.

To set up the interface for the app, we need to load the relevant
scripts into the Shiny app and assign a section for the resulting plots.
This is done when setting up the user interface. We import D3 and our
plotting code via the \texttt{tags\$script} (for web links) and
\texttt{includeScript} (for reading from a full path). We use
\texttt{tags\$div} to assign an id for the output section that can be
accessed in the D3 code.

\begin{verbatim}
tags$script(src = "https://d3js.org/d3.v4.min.js"),
includeScript(system.file("js/d3anim.js", package = "tourrGUID3")),
tags$div(id = "d3_output")
\end{verbatim}

On the D3 side we can access the id defined in Shiny, and for example
assign it to a scalable vector graphics (svg) object to be filled in D3
and rendered onto the Shiny app.

\begin{verbatim}
var svg = d3.select("#d3_output")
    .append("svg")
    .attr("width", w)
    .attr("height", h);
\end{verbatim}

The data format expected by D3 is in JSON format, which combines two
basic programming paradigms: a collection of name/value pairs, and an
ordered list of values. R's preferred data formats include data frames,
vectors and matrices. Every time a new projection has been calculated
with the tour path, the resulting matrix needs to be converted to JSON
and sent to D3. Using a named list we can send multiple JSON datasets to
D3, e.g.~to draw both the data points (stored in dataframe d) and the
projection axes (stored in dataframe a). Converting dataframes will pass
the column names to JSON. The code to send the D3 data looks like this:

\begin{verbatim}
session$sendCustomMessage(type = "data", message = list(d = toJSON(d), a = toJSON(a)))
\end{verbatim}

This code is from the observe environment from the \texttt{server.R}
file. It converts the matrix of projected data points to JSON format,
and sends it to JavaScript with the id data. The list entries of the
``message'' can parsed in D3 by its \texttt{data()} method, e.g.
\texttt{data(message.d)} to access the projected data points, and we can
access each column through the column names assigned in the original
dataframe, and loop over all rows for rendering. All of the code
required to render the scatterplots and legends, along with colours, is
JavaScript code in the file \texttt{d3anim.js}. In particular, the data
from R is handled with the following code:

\begin{verbatim}
Shiny.addCustomMessageHandler("data",
    function(message) {
        /* D3 scatterplot is drawn and re-drawn using the
            data sent from the server. */
}
\end{verbatim}

Every time the message is sent (25 times per second), the code-block is
run.

\hypertarget{getting-projections}{%
\subsection{Getting projections}\label{getting-projections}}

The \texttt{observeEvent} Shiny method defines a code block to be run
whenever some input value changes. The following code snippet restarts a
tour using a random basis:

\begin{verbatim}
observeEvent(input$restart_random,
  {
    p <- length(input$variables)
    b <- matrix(runif(2*p), p, 2)
    rv$tour <-
      new_tour(as.matrix(rv$d[input$variables]),
              choose_tour(input$type,
              input$guidedIndex,
              c(rv$class[[1]])), b)
})
\end{verbatim}

The projections are calculated using the tour object in an
\texttt{observe()} environment, which re-executes the code whenever it
is invalidated. The invalidation is either by a change in reactive value
inside the code block, or we can schedule a re-execution by explicitly
invalidating the observer after a selected interval using
\texttt{invalidateLater()}. The projections are calculated using the
following code block:

\begin{Schunk}
\begin{Sinput}
observe({
    if (length(rv$mat[1, ]) < 3) {
      session$sendCustomMessage(type = "debug",
      message = "Error: Need >2 variables.")
    }
    aps <- rv$aps
    tour <- rv$tour
    step <- rv$tour(aps / fps)
    invalidateLater(1000 / fps)
    j <- center(rv$mat %*% step$proj)
    j <- cbind(j, class = rv$class)
    colnames(j) <- NULL
    session$sendCustomMessage(type = "data",
         message = list(d = toJSON(data.frame(pL=rv$pLabel[,1], x=j[,2],
                                              y=j[,1], c=j[,3])),
                        a = toJSON(data.frame(n=rv$vars, y=step$proj[,1],
                                              x=step$proj[,2]))))
})
\end{Sinput}
\end{Schunk}

\hypertarget{try-it}{%
\subsection{Try it}\label{try-it}}

You can try the app yourself using this code:

\begin{Schunk}
\begin{Sinput}
devtools::install_github("uschiLaa/tourrGUID3")
library(tourrGUID3)
launchApp(system.file("extdata", "geozoo.csv", package = "tourrGUID3"))
\end{Sinput}
\end{Schunk}

\hypertarget{troubleshooting}{%
\subsection{Troubleshooting}\label{troubleshooting}}

Fixing bugs in the JavaScript code can be cumbersome, as R and Shiny
will not report any errors. Tracing JavaScript errors can be done when
using the JavaScript console in the web browser. For example, in Google
Chrome the console can be accessed via the ``Developer Tools'' option
found under ``Moore Tools'' in the control menu. Typical errors that we
encountered were version dependent syntax in D3, e.g.~for axis
definitions or scaling.

\hypertarget{pros-and-cons}{%
\subsection{Pros and cons}\label{pros-and-cons}}

The D3 canvas makes for smooth drawing and re-drawing of the data
projections. Adding a GUI around the display is straightforward with the
Shiny package, e.g.~control elements such as stop/go, increase/decrease
speed, change tour type, add/remove variables from the mix.

The main disadvantage is that the speed is inconsistent, as server and
client play tag to keep up with each other, and the display cannot
handle many observations. Noticeable slow down was oberved with 2000
points, the main reason being the rendering time required for the large
number of SVG circle elements. The situation can be improved when using
a single HTML5 canvas element to draw the scatter points, significantly
reducing the rendering time.

Another disadvantage is that the displays needs to be coded anew. D3
provides mostly primitives, and example code, to make scatterplots, and
contours, but the data displays all need to be coded again.

\hypertarget{summary}{%
\subsection{Summary}\label{summary}}

The custom message tools from Shiny provide a way to share a tour path
with the D3 renderer, and embed it in a Shiny GUI providing controls
such as stop/go, increase/decrease speed, change tour type, add/remove
variables. However, the approach doesn't provide the smooth motion that
is needed for easy display of projections, and is slow for large numbers
of observations.

\hypertarget{code}{%
\subsection{Code}\label{code}}

The code is available at \url{https://github.com/uschiLaa/tourrGUID3},
and the source material for this paper is available at
\url{https://github.com/dicook/paper-tourrd3}.

\hypertarget{acknowledgements}{%
\subsection{Acknowledgements}\label{acknowledgements}}

Thanks to Yihui Xie for pointing out the custom message tools.

\hypertarget{references}{%
\subsection{References}\label{references}}

\hypertarget{refs}{}
\leavevmode\hypertarget{ref-D3}{}%
Bostock, Michael, Vadim Ogievetsky, and Jeffrey Heer. 2011. ``D3:
Data-Driven Documents.'' \emph{IEEE Transactions on Visualization and
Computer Graphics} 17 (12): 2301--9.
\url{http://vis.stanford.edu/papers/d3}.

\leavevmode\hypertarget{ref-shiny}{}%
Chang, Winston, Joe Cheng, JJ Allaire, Yihui Xie, and Jonathan
McPherson. 2017. \emph{Shiny: Web Application Framework for R}.
\url{https://CRAN.R-project.org/package=shiny}.

\leavevmode\hypertarget{ref-gt_pp}{}%
Cook, Dianne, Andreas Buja, Javier Cabrera, and Catherine Hurley. 1995.
``Grand Tour and Projection Pursuit.'' \emph{Journal of Computational
and Graphical Statistics} 4 (4): 155--72.

\leavevmode\hypertarget{ref-gt_pp_mc}{}%
Cook, Dianne, Andreas Buja, Eun-Kyung Lee, and Hadley Wickham. 2007.
``Grand Tours, Projection Pursuit Guided Tours and Manual Controls.''

\leavevmode\hypertarget{ref-tourrGui}{}%
Huang, Bei, Dianne Cook, and Hadley Wickham. 2012. ``TourrGui: A
gWidgets Gui for the Tour to Explore High-Dimensional Data Using
Low-Dimensional Projections.'' \emph{Journal of Statistical Software} 49
(6): 1--12.

\leavevmode\hypertarget{ref-ihaka:1996}{}%
Ihaka, Ross, and Robert Gentleman. 1996. ``R: A Language for Data
Analysis and Graphics.'' \emph{Journal of Computational and Graphical
Statistics} 5 (3): 299--314.

\leavevmode\hypertarget{ref-RGtk2}{}%
Lawrence, Michael, and Duncan Temple Lang. 2010. ``RGtk2: A Graphical
User Interface Toolkit for R.'' \emph{Journal of Statistical Software}
37 (8): 1--52. \url{http://www.jstatsoft.org/v37/i08/}.

\leavevmode\hypertarget{ref-R}{}%
R Core Team. 2018. \emph{R: A Language and Environment for Statistical
Computing}. Vienna, Austria: R Foundation for Statistical Computing.
\url{http://www.R-project.org/}.

\leavevmode\hypertarget{ref-tourr}{}%
Wickham, Hadley, Dianne Cook, Heike Hofmann, and Andreas Buja. 2011.
``Tourr: An R Package for Exploring Multivariate Data with
Projections.'' \emph{Journal of Statistical Software} 40 (2): 1--18.

\bibliography{references.bib}

\address{%
Michael Kipp\\
Monash University\\
Department of Econometrics and Business Statistics\\
}
\href{mailto:mkipp271@gmail.com}{\nolinkurl{mkipp271@gmail.com}}

\address{%
Ursula Laa\\
Monash University\\
School of Physics and Astronomy\\
}
\href{mailto:ursula.laa@monash.edu}{\nolinkurl{ursula.laa@monash.edu}}

\address{%
Dianne Cook\\
Monash University\\
Department of Econometrics and Business Statistics\\
}
\href{mailto:dicook@monash.edu}{\nolinkurl{dicook@monash.edu}}


\documentclass[10pt]{report}

\usepackage{enumerate} % for enumerate counter
\usepackage{subcaption} % for subfigures
\usepackage{amsthm} % for QED
\usepackage{mathtools} % for delimiter

\usepackage{listings} % for code
\lstset{
	language=R,
	basicstyle=\footnotesize\ttfamily,
	numbers=none,
	stepnumber=1,
	numbersep=8pt,
	showspaces=false,
	showstringspaces=false,
	showtabs=false,
	frame=single,
	tabsize=2,
	captionpos=t,
	breaklines=true,
	breakatwhitespace=false
}

\usepackage{float} % for figure [H]
\usepackage{booktabs} % for tabular
\usepackage{caption} % for \caption*
\usepackage[export]{adjustbox} % for valign=t
\usepackage{array} % for column type m
\usepackage{verbatim}
\usepackage{graphicx}
%\graphicspath{ {imgs/} }
\usepackage{fancyhdr}
\usepackage{amssymb}
\usepackage{amsmath}

%%%%%%Pagination
\setlength{\topmargin}{-.3 in}
\setlength{\oddsidemargin}{0in}
\setlength{\evensidemargin}{0in}
\setlength{\textheight}{9.in}
\setlength{\textwidth}{6.5in}

%Cover
\newcommand{\hwTitle}{Homework \#4}
\newcommand{\hwCourse}{Applied Statistics}
\newcommand{\hmClassInstructor}{Professor Lulu Kang}

\title{
	\vspace{2in}
	\textmd{\textbf{\hwCourse\\\hwTitle}}\\
	\vspace{0.3in}\large{\textit{\hmClassInstructor}}
	\vspace{3in}
}
\author{\textbf{Zhihao Ai}}
\date{}

%Header
\pagestyle{fancy}
\fancyhead[L]{Zhihao Ai}
\fancyhead[C]{Math 484}
\fancyhead[R]{Homework 4}
%%%%%%

%Global settings
%\everymath{\displaystyle}
\setlength\parindent{0pt}

%Custom commands
\newcommand{\ds}{\displaystyle}
\newcommand{\ts}{\textstyle}

\newcolumntype{N}{>$ c <$}
\newcolumntype{M}[1]{>{\centering\arraybackslash $}m{#1}<{$}}

\newcommand{\abs}[1] {\left| #1 \right|}

\DeclarePairedDelimiter\autoparen{(}{)}
\newcommand{\pa}[1]{\autoparen*{#1}}

\newcommand{\var} {\text{var}}

\newcommand{\m}[1] {\mathbf{#1}}

\begin{document}

\maketitle

\section*{Problem 1}
(Ex. 7.19) Refer to \textbf{Commercial properties}  Problem 6.18.
\begin{enumerate}[a.]
	\item
	Transform the variables by means of the correlation transformation (7.44) and fit the standardized regression model (7.45).

	By solving $\m{b} = \m{r}^{-1}_{XX} \m{r}^{}_{YX}$, we have
	\[
	\m{b} = [-0.5478526, 0.4236468, 0.04846136, 0.5027571]'
	\]
	So the standardized regression model is $Y^*_i = -0.5478526 X^*_{i1} + 0.4236468 X^*_{i2} + 0.04846136 X^*_{i3} + 0.5027571 X^*_{i4}$.

	\item
	Interpret the standardized regression coefficient $b^*_2$.

	With other predictor variables fixed, the rental rates ($Y$) will increase by 0.4236468 standard deviations if operating expenses and taxes ($X_2$) increase by 1 standard deviation.

	\item
	Transform the estimated standardized regression coefficients by means of (7.53) back to the ones for the fitted regression model in the original variables. Verify that they are the same as the ones obtained in Problem 6.18c.

	Employing the relations
	\begin{align*}
		b_k &= \pa{\frac{s_Y}{s_k}} b^*_k \quad (k=1,\dots,p-1)\\
		b_0 &= \bar{Y} - b_1 \bar{X}_1 - \dots - b_{p-1} \bar{X}_{p-1}
	\end{align*}
	We have
	\[
	\m{b} = [1.220059\mathrm{e}+01, -1.420336\mathrm{e}-01, 2.820165\mathrm{e}-01, 6.193435\mathrm{e}-01, 7.924302\mathrm{e}-06]'
	\]
	which is the same as the one ontained in Problem 6.18c.
\end{enumerate}

\section*{Problem 2}
(Ex. 7.24) Refer to \textbf{Brand preference}  Problem 6.5.
\begin{enumerate}[a.]
	\item
	Fit first-order simple linear regression model (2.1) for relating brand liking ($Y$) to moisture content ($X_1$). State the fitted regression function.

	Fitting the model for relating $Y$ to $X_1$, we have
	\lstinputlisting{p2/24a.txt}
	So the fitted regression function is $\hat{Y} = 50.775 + 4.425 x_1$.

	\item
	Compare the estimated regression coefficient for moisture content obtained in part (a) with the corresponding coefficient obtained in Problem 6.5b. What do you find?

	The fitted regression function in Problem 6.5b is $\hat{Y} = 37.65 + 4.425 x_1 + 4.375 x_2$. The $b_1$ coefficients are the same.

	\item
	Does $SSR(X_1)$ equal $SSR(X_1|X_2)$ here? If not, is the difference substantial?

	Yes, they both equal 1566.45.

	\item
	Refer to the correlation matrix obtained in Problem 6.5a. What bearing does this have on your findings in parts (b) and (c)?

	The correlation between $X_1$ and $X_2$ is 0, meaning the two predictor variables are uncorrelated. When they are uncorrelated, the effects ascribed to them by a first-order regression model are the same no matter which other of the variables are included in the model, so the corresponding coefficients in parts (b) are the same. Likewise, the marginal contribution of one variable in reducing the error sum of squares when the other variable is in the model is the same as when it is in the model alone, hence $SSR(X_1) = SSR(X_1|X_2)$.

\end{enumerate}

\section*{Problem 3}
(Ex. 8.24) \textbf{Assessed valuations}
\begin{enumerate}[a.]
	\item
	Plot the sample data for the two populations as a symbolic scatter plot. Does the regression relation appear to be the same for the two populations?

	Below is the symbolic scatter plot:
	\begin{figure}[H]
		\centering
		\includegraphics[width=.5\linewidth]{p3/24a.png}
	\end{figure}
	The regression relation does not appear to be the same.

	\item
	Test for identity of the regression functions for dwellings on corner lots and dwellings in the other locations; control the risk of Type I error at .05. State the alternatives, decision rule, and conclusion.

	The tentative model is given by
	\[
	Y_i = \beta_0 + \beta_1 X_{i1} + \beta_2 X_{i2} + \beta_3 X_{i1}X_{i2} + \epsilon_i
	\]
	where
	\begin{align*}
		X_{i1} &= \text{assessed valuation}\\
		X_{i2} &=
		\begin{cases}
		0 & \text{if on non-corner lots}\\
		1 & \text{if on corner lots}
		\end{cases}
	\end{align*}
	The alternatives are
	\begin{align*}
		H_0: &\ \beta_2 = \beta_3 = 0\\
		H_a: &\ \text{not both $\beta_2 = 0$ and $\beta_3 = 0$}
	\end{align*}
	The test statistic is:
	\[
	F^* = \frac{MSR(X_2, X_1 X_2 | X_1)}{MSE} = \frac{SSR(X_2 | X_1) + SSR(X_1 X_2 | X_1, X_2)}{2} \div MSE
	\]
	The decision rule is
	\begin{align*}
	\text{If } F^* \le F(0.95; 2, 60) = 3.150411, \text{ conclude } H_0\\
	\text{If } F^* > F(0.95; 2, 60) = 3.150411, \text{ conclude } H_a
	\end{align*}
	\lstinputlisting{p3/24b.txt}
	Using the results above, $F^* = (453.1+113.0)/2/15.2 = 18.62171 > 3.150411$. So we conclude $H_a$, that the regression functions for the two dwellings locations are not identical.

	\item
	Plot the estimated regression functions for the two populations and describe the nature of the differences between them.
	\begin{figure}[H]
		\centering
		\includegraphics[width=.5\linewidth]{p3/24c.png}
	\end{figure}
	The response functions are:
	\begin{align*}
		E(Y) &= \beta_0 + \beta_1 X_1 + \beta_2(0) + \beta_3(0) = \beta_0 + \beta_1 X_1 \quad \text{non-corner lots}\\
		E(Y) &= \beta_0 + \beta_1 X_1 + \beta_2(1) + \beta_3(X_1) = (\beta_0 + \beta_2) + (\beta_1 + \beta_3) X_1 \quad \text{corner lots}
	\end{align*}
	$\beta_2$ here indicated how much greater is the $Y$ intercept of the response function for the class coded 1 than that for the class coded 0. $\beta_3$ indicates how much smaller is the slope of the response function for the class coded 1 than that for the class coded 0. The nature of the differences is the effect of the class and the interaction of $X_1$ and $X_2$ on the regression.
\end{enumerate}

\end{document}


\documentclass[12pt]{article}
\usepackage{amsmath,amsfonts,amssymb}
\textheight=22cm \topmargin=-1cm

%%%%%Structure of Thms. etc.%%%%%%%%%%%%%%%
\newtheorem{thm}{Theorem}
\newtheorem{con}{Conjecture}
\newtheorem{lem}[thm]{Lemma}
\newtheorem{prop}{Proposition}
\newtheorem{cor}{Corollary}
\newtheorem{defn}{Definition}
\newtheorem{ex}{Example}
\newtheorem{exo}{Exercice}
\newtheorem{porism}{Porism}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\newcommand\la{\lambda}
\newcommand{\spf}{\noindent{\bf Second proof: \/}}
\newcommand{\fpf}{\noindent{\bf First proof: \/}}
\newcommand{\pr}{{\bf Proof: \/}}
\newcommand{\remark}{\noindent{\bf Remark: \/}}
%\newcommand{\eg}{\noindent{\bf For example: \/}}
\def\qed{\nobreak\quad\raise -2pt\hbox{\vrule\vbox to 10pt{\hrule width 6pt
\vfill\hrule}\vrule}\par\vspace{2ex}}
\def\qel{\nobreak\quad\raise -2pt\hbox{\vrule\vbox to 10pt{\hrule width 6pt
\vfill\hrule}\vrule}}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{document}

\begin{center}
{\Large\bf Applications of Waring's formula to some identities of
Chebyshev polynomials}
\end{center}

\vskip 2mm \centerline{ Jiang Zeng$^{1,2}$ and Jin Zhou$^2$}

\begin{center} \small $^1$ Institut Girard Desargues,
Universit\'e Claude Bernard (Lyon I)\\
69622 Villeurbanne Cedex, France \\
{\tt zeng@igd.univ-lyon1.fr}\\
and\\
$^2$ Center for Combinatorics, LPMC,
Nankai University\\
 Tianjin 300071, People's Republic of China\\
{\tt jinjinzhou@hotmail.com} \\
\vspace{10pt}
\small AMS Math Subject Classification Numbers: 11B39, 33C05, 05E05\\
\end{center}

 \noindent
\textbf{Abstract.} Some identities of Chebyshev polynomials are
deduced from Waring's formula on symmetric functions. In
particular, these formulae generalize some recent results of
Grabner and Prodinger.
\section{Introduction}

Given a set of variables $X=\{x_1,x_2,\ldots\}$, the $k$th ($k\geq
0$) \emph{elementary symmetric polynomial} $e_{k}(X)$ is defined
by $e_0(X)=1$,
$$
e_k(X)=\sum_{i_1<\ldots<i_k}x_{i_1}\ldots x_{i_k},\quad
\text{for}\quad k\geqslant 1,
$$
 and  the $k$th  ($k\geq
0$)\emph{power sum symmetric polynomial} $p_{k}(X)$ is defined by
$p_0(X)=1$,
$$
p_k(X)=\sum_ix_i^k,\quad \text{for}\quad k\geqslant 1.
$$
Let $\la=1^{m_1}2^{m_2}\ldots $ be a partition of $n$, i.e., $m_1
1+m_2 2+\ldots +m_n n=n$, where $m_i\geq 0$ for $i=1,2,\ldots n$.
Set $l(\la)=m_1+m_2+\ldots +m_n$. According to the
\emph{fundamental theorem of symmetric polynomials}, any symmetric
polynomial can be written uniquely as a polynomial of elementary
symmetric polynomials $e_i(X)$ ($i\geq 0$). In particular, for the
power sum $p_k(x)$, the corresponding formula  is usually
attributed to Waring~\cite{CLY,Mac} and reads as follows:
\begin{equation}\label{war}
p_k(X)=\sum_{\la}(-1)^{k-l(\lambda)}\frac{k(l(\lambda)-1)!}
{\prod_{i}{m_{i}!}}e_{1}(X)^{m_1}e_{2}(X)^{m_2}\ldots,
\end{equation}
where the sum is over all the partitions
$\la=1^{m_1}2^{m_2}\ldots$ of $k$.

In a recent paper~\cite{GP} Grabner and Prodinger proved some
identities about Chebyshev polynomials using generating functions,
the aim of this paper is to show that Waring's formula provides
a natural generalization of such kind of identities.

Let $U_n$ and $V_n$ be two
sequences defined by the following recurrence relations:
\begin{align}
U_n&=pU_{n-1}-U_{n-2},&U_0=0, U_1=1,\\
V_n&=pV_{n-1}-V_{n-2},&V_0=2, V_1=p.
\end{align}
Hence $U_n$ and $V_n$ are rescaled versions of the first and
second kind of Chebyshev polynomials ${\cal U}_n(x)$ and ${\cal
T}_n(x)$, respectively:
$$
{\cal U}_n(x)=U_{n+1}(2x),\quad {\cal T}_n(x)=\frac{1}{2}T_n(x).
$$
\begin{thm} For integers
$m,n\geq 0$, let $W_n=aU_n+bV_n$ and  $\Omega=a^2+4b^2-b^2p^2$. Then the following identity holds
\begin{equation}\label{eq:wm}
W_n^{2k}+W_{n+m}^{2k}=\sum_{r=0}^{k}\theta_{k,r}(m)\Omega^{k-r}W_n^rW_{n+m}^r,
\end{equation}
where
$$
\theta_{k,r}(m)=\sum_{0\leqslant 2j\leqslant
k}(-1)^j\frac{k(k-j-1)!}{j!(k-r)!(r-2j)!}V_m^{r-2j}U_m^{2k-2r}.
$$
\end{thm}
Note that  the identities of Grabner and Prodinger~\cite{GP} correspond to the $m=1$
and implicitly $m=2$ cases of Theorem~1 (cf. Section~3).
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Proof of Theorem~1}
We first check the $k=1$ case of (\ref{eq:wm}):
\begin{equation}\label{fund}
W_n^2+W_{n+m}^2=V_mW_nW_{n+m}+U_m^2\Omega.
\end{equation}
Set $\alpha=(p+\sqrt{p^2-4})/2$ and $\beta=(p-\sqrt{p^2-4})/2$
then it is easy to see that
$$
U_n=\frac{\alpha^{n}-\beta^{n}}{\alpha-\beta},\quad
V_{n}=\alpha^n+\beta^n,
$$
it follows that
\[
W_n=aU_n+bV_n=A\alpha^n+B\beta^n,
\]
where $A=b+a/(\alpha-\beta)$ and $B=b-a/(\alpha-\beta)$. Therefore
\begin{align*}
V_mW_nW_{n+m}+U_m^2\Omega
&=(\alpha^m+\beta^m)(A\alpha^n+B\beta^n)(A\alpha^{n+m}+B\beta^{n+m})\\
&+\left(\frac{\alpha^m-\beta^m}{\alpha-\beta}\right)^2(a^2+4b^2-b^2p^2),
\end{align*}
 which is readily seen to be equal to $W_n^2+W_{n+m}^2$.

Next we  take the alphabet $X=\{W_n^2, W_{n+m}^2\}$, then the left-hand
side of (\ref{eq:wm}) is the power sum $p_k(X)$. On the other
hand, since
$$
e_1(X)=W_n^2+W_{n+m}^2,\quad  e_2(X)=W_n^2W_{n+m}^2,\quad e_i(X)=0
\quad \textrm{if} \quad i\geqslant3,
$$
the summation at the right-hand side of (\ref{war}) reduces to the
partitions $\la=(1^{k-2j}\,2^j)$, with  $j\geq 0$. Now, using
(\ref{fund}) Waring's formula~(\ref{war}) infers that
\begin{align*}
W_n^{2k}&+W_{n+m}^{2k} \\
 &=\sum_{0\leqslant2 j\leqslant
k}(-1)^{j} \frac{k(k-j-1)!}{j!(k-2j)!}
(V_mW_nW_{n+m}+U_m^2\Omega)^{k-2j}(W_n^2W_{n+m}^2)^{j}\\
&=\sum_{0\leqslant2j\leqslant k}\sum_{i=0}^{k-2j}(-1)^j
\frac{k(k-j-1)!}{j!i!(k-2j-i)!}V_m^{k-2j-i}
U_m^{2i}\Omega^{i}(W_{n}W_{n+m})^{k-i}
\end{align*}
Setting $k-i=r$ and exchanging the order of summations yields
(\ref{eq:wm}). \qed

\section{Some special cases}
When $m=1$ or 2, as $U_1=1$, $V_1=p$ and $U_2=p$, $V_2=p^2-2$ the
coefficient $\theta_{k,r}(r)$ of Theorem~1 is much simpler.
\begin{cor} We have
\begin{eqnarray}
\theta_{k,r}(1)&=&\sum_{0\leqslant2j\leqslant
r}(-1)^j\frac{k(k-1-j)!}{(k-r)!j!(r-2j)!}p^{r-2j},\label{coeff1}\\
\theta_{k,r}(2) &=&\sum_{0\leqslant2j\leqslant
k}(-1)^j\frac{k(k-j-1)!}{j!(k-r)!(r-2j)!}
(p^2-2)^{r-2j}p^{2k-2r}.\label{coeff2}
\end{eqnarray}
\end{cor}

We notice  that (\ref{coeff1}) is exactly the formula given by Grabner and Prodinger~\cite{GP}
for $\theta_{k,r}(1)$, while for $\theta_{k,r}(2)$  they
give a more involved formula than  (\ref{coeff2})  as follows:
\begin{cor}[Grabner and Prodinger~\cite{GP}] There holds
\begin{equation}\label{coeffGP}
\theta_{k,r}(2) =\sum_{0\leqslant\lambda\leqslant
k}(-1)^{\lambda}p^{2k-2\lambda}
\frac{k(k-\lfloor\frac{\lambda}{2}\rfloor-1)!
2^{\lceil\frac{\lambda}{2}\rceil}}{(k-r)!\lambda!(r-\lambda)!}
\prod_{i=0}^{\lfloor\frac{\lambda}{2}\rfloor-1}
(2k-2\lceil\frac{\lambda}{2}\rceil-1-2i).
\end{equation}
\end{cor}
In order to identify (\ref{coeff2}) and (\ref{coeffGP}), we need
the following identity.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{lem}\label{key} We have
\begin{eqnarray}\label{fa}
&&\sum_{i=0}^{j/2}(-1)^{i}\frac{(k-i-1)!2^{j-2i}}{(j-2i)!i!}\nonumber\\
&&\hspace{1cm}= \frac{(k-\lfloor{j/2}\rfloor-1)!}{j!}2^{\lceil
j/2\rceil} \prod_{i=0}^{\lfloor j/2\rfloor-1}(2k-2\lceil
j/2\rceil-1-2i).
\end{eqnarray}
\end{lem}
\begin{pr} For $n\geq 0$ let $(a)_n=a(a+1)\ldots (a+n-1)$, then
the Chu-Vandermonde formula~\cite[p.212]{GKP} reads:
\begin{equation}\label{cv}
{}_2F_1( -n,a; c; 1):=\sum_{k\geqslant0}\frac{(-n)_k(a)_k}{(c)_k
k!}=\frac{(c-a)_n}{(c)_n}.
\end{equation}
Note that $n!=(1)_n$, so using the simple transformation formulae:
$$
(a)_{2n}=\left(\frac{a}{2}\right)_n\left(\frac{a+1}{2}\right)_{n}2^{2n},\quad
(a)_{2n+1}=\left(\frac{a}{2}\right)_{n+1}
\left(\frac{a+1}{2}\right)_{n}2^{2n+1},
$$
and
$$
(a)_{N-n}=\frac{(a)_N}{(a+N-n)_n}=(-1)^n\frac{(a)_N}{(-a-N+1)_n},
$$
 we can rewrite the left-hand side of identity~(\ref{fa}) as follows:
 \begin{eqnarray*}
  \begin{cases}
    \frac{(k-1)!}{(\frac{1}{2})_m(1)_m}
    \,{}_2F_1(-m, -m+\frac{1}{2};-k+1; 1)& \text{if $j=2m$}, \\
    \\
    \frac{(k-1)!}{{(\frac{1}{2})}_{m+1}(1)_m}
    \,{}_2F_1(-m, -m-\frac{1}{2};-k+1; 1)& \text{if $j=2m+1$},
  \end{cases}
\end{eqnarray*}
which is clearly equal to the right-hand side of (\ref{fa}) in
view of (\ref{cv}). \qed
\end{pr}

Now, expanding  the right-hand side of (\ref{coeff2}) by binomial
formula yields
$$
\sum_{0\leqslant2j\leqslant k}(-1)^j\frac{k(k-j-1)!}
{j!(k-r)!(r-2j)!}\sum_{i=0}^{r-2j}{r-2j\choose i}
p^{2i}(-2)^{r-2j-i}p^{2k-2r}.
$$
Writing  $\la=r-i$, so $\la\leq r\leq k$, and exchanging the order
of summations, the above quantity becomes
$$
\sum_{0\leqslant \la\leqslant k} (-1)^{\la} p^{2k-2\la
}\frac{k}{(k-r)!(r-\la)!}\sum_{0\leqslant j\leqslant
k/2}(-1)^j\frac{(k-j-1)!2^{\la-2j}}{(\la-2j)!j!},
$$
which yields (\ref{coeffGP}) by applying Lemma~\ref{key}.

\begin{thebibliography}{99}
\bibitem{CLY} William Y. C. Chen, Ko-Wei Lih, and  Yeong-Nan Yeh:
\emph{Cyclic Tableaux and Symmetric Functions}, Studies in Applied
Math., 94 (1995), 327-339.
\bibitem{GKP} Ronald L. Graham, Donald E. Knuth and Oren Patashnik:
\emph{Concrete Mathematics}, Addion-Wesley Pubilshing Co. 1989.
\bibitem{GP} Peter J. Grabner and Helmut Prodinger:
\emph{Some identities for Chebyshev polynomials}, Portugalia
Mathematicae 59 (2002), 311-314.
\bibitem{Mac} P. A. MacMahon:
\emph{Combinatory analysis}, Chelsea Publishing Co. New York,
1960.

\end{thebibliography}

\end{document}


\documentclass[a4paper, 12pt]{article}
\usepackage{preamble}

\begin{document}

\title{NOTES ON SOMETHING}
\author{AUTHOR}
\maketitle{}

\section{A section}
\blindtext[1]{}

\subsection{A subsection}
\blindtext[1]{}

Citing Equation~\ref{eq:sum}.

\begin{equation}
\label{eq:sum}
    a + b = c
\end{equation}

\section{Another section}
\blindtext[2]{}

\begin{table}[hbt!]
  \centering
  \renewcommand{\arraystretch}{1.1}
  \caption{Table caption}
  \begin{tabular}{p{0.3\linewidth}c{0.12\linewidth}c{0.12\linewidth}}
   \toprule
   \textbf{Column 1} & \textbf{Column 2} & \textbf{Column 3} \\
   \midrule
   Description 1 & 0.1 & 1.0 \\
   Description 2 & 0.2 & 2.0 \\
  \bottomrule
  \end{tabular}\label{tab:table}
  \renewcommand{\arraystretch}{1}
\end{table}

\begin{algorithm}[hbt!]
\caption{$\textrm{AlgorithmTitle}(q, P)$}
\textit{Input.} Algorithm input. \\
\textit{Output.} Algorithm output.
\begin{algorithmic}[1]
  \State{Draw a horizontal line $l_{0r}$ from $q$ to its left}\label{linea}
    \State{Move from infinity to $q$, following $l_{0r}$, and count the number
    $N$ of intersections of $l_{0r}$ with $P$}
    \If{$N$ is odd}
        \State{$q$ is inside $P$}
    \Else{}
        \State{$q$ is outside $P$}
    \EndIf{}
    \For{all edges $e_{ab}$ of $P$, where $a$ is the lower and $b$ is the upper
      point}
      \If{$a_{y} < q_{y}$ \textbf{and} $b_{y} > q_{y}$ \textbf{and}
      Equation~\ref{eq:sum} is satisfied}\label{lineb}
        \State{$N = N + 1$}
      \EndIf{}
    \EndFor{}
\end{algorithmic}\label{alg:example}
\end{algorithm}

Citing lines~\ref{linea} and~\ref{lineb} of Algorithm~\ref{alg:example}.

It is possible to include snippets from real code, like in Listing~\ref{list:snippet}.

\begin{listing}[hbf!]
  \caption{Code snippet}
  \label{list:snippet}
  \inputminted{cpp}{./snippets/hello.cpp}
\end{listing}

Explanation of code using highlighted syntax.

\code{cpp}{double x, y;}{declaration without initialization}
\code{cpp}{double x = 1.5;}{declaration with initialization}
\code{cpp}{double x(1.5);}{declaration with initialization}
\code{cpp}{const double x = 1.5;}{initializes a constant}

\begin{definition}{Lines intersections.}
\begin{enumerate}
  \item The lower point of $l_{ab}$ lies below $q$.
  \item The upper point of $l_{ab}$ lies above $q$.
  \item $q$ lies to the right from the line $\overrightarrow{l}_{ab}$, that
      goes from $a$ to $b$.
\end{enumerate}
\end{definition}

\begin{problem}{Point inclusion in a polygon}
Given a point $q$ and a polygon $P$, determine if $q$ is inside or outside $P$.
\end{problem}

\end{document}


\section{Structural Synthesis}
\label{sec:struct}

The structural synthesis stage of our algorithm consists of two parts.
First, we guess at an initial filter structure based on a preliminary analysis of the input/output audio examples.
In the second stage, which occurs during the synthesis loop, we iteratively pick new structures to try during metrical synthesis.
For this second stage we have implemented a greedy algorithm to pick the best structure out of the possible next choices.

\subsection{Initial Structure Construction}
\label{sec:initStruct}
In order to find an initial structure for our synthesized filter, we use an adaption of room impulse response measurement.
When measuring the impulse response of a room, we can reconstruct the band-pass filter exactly by playing a sound in the room, recording the sound, then examining the differences.
In DSP-PBE, we do not want to allow infinitely many bandpass filters, as the synthesized program should be relatively small and human-readable, in the same way as code would be readable if it had been manually written.
Thus, we mimic the room impulse response measurement technique using only the available \dspnode in our grammar.

In the grammar listed in Fig.~\ref{fig:grammar}, we have two filters which are similar to the bandpass filters used in measuring room impulse response - a low-pass filter, $LFP \ [0,20k]\ [0,1]$, and a high-pass filter $HPF\ [0,20k]\ [0,1]$.
In order to quickly discover approximate values for each \dspnode, we run an analysis of the frequencies present in the input example that have decreased in amplitude in the output example.
To describe the analysis, we write a formula that show how to find an initial threshold value for a lowpass filter in our synthesized code.
We use $t$ to represents the threshold of the lowpass filter, \texttt{spectrogram} to represent a function that runs a FFT on a sound sample and returns a list of peaks, $f_i$ to represent a frequency, and \texttt{amp()} to represent a function to retrieve the amplitude of the frequency.
%
\begin{align*}
&\text{Given input audio }i\text{ and output audio }o\text{, find }t\text{ such that} \\
&\forall f_1 \in  \texttt{spectrogram(i)}.\ \forall f_2 \in \texttt{spectrogram(o)}. \\
&(f_1 > t \land  f_2 > t \land f_1 == f_2) \implies \texttt{amp}(f_1) > \texttt{amp}(f_2)
\end{align*}

Similarly, we can build a formula to describe how to calculate an initial value for a high-pass.
Here we look for the lowest frequency where amplitude has decreased in the output audio - this is then a starting point for the threshold of a high pass filter.
%
\begin{align*}
&\text{Given input audio }i\text{ and output audio }o\text{, find }t\text{ such that} \\
&\forall f_1 \in \texttt{spectrogram(i)}.\ \forall f_2 \in \texttt{spectrogram(o)}. \\
&(f_1 < t \land f_2 < t \land f_1 == f_2) \implies \texttt{amp(}f_1\texttt{)} > \texttt{amp(} f_2 \texttt{)}
\end{align*}

\subsection{Structural Synthesis}

\begin{figure}
\begin{align*}
	LPF \ 10000 \ 0.5 \arrComp HPF \ 100 \ 0.5 & \\
	\ldots & \arrComp PitchShift \ 0.1 \ 0.1 \\
	\ldots & \parallelCompose PitchShift \ 0.1 \ 0.1 \\
	\ldots & \arrComp Reverb \ 0.1 \ 0 \ 0.1 \\
	\ldots & \parallelCompose Reverb \ 0.1 \ 0 \ 0.1 \\
	\ldots & \arrComp WhiteNoise \ 0.1 \\
	\ldots & \parallelCompose WhiteNoise \ 0.1
\end{align*}
\caption{New structural candidates are generated based on all compositions of unused \dspnode}
\label{fig:generation}
\end{figure}

During each loop of our synthesis procedure, we attempt to build a new structure based on the results of our previous attempts.
We compare several different variations on the current structure (Fig.~\ref{fig:generation}).
Variations are generated by composing all \dspnode that are not present in the current structure with sequential and parallel composition onto the existing filter program returned from our metrical synthesis (cf. Sec.~\ref{sec:opt}).
We keep the parameters for the existing \dspnode, but for the new \dspnode we initialize the parameters to small values.
Using small values (instead of zeroes) ensures that the filter has an observable effect on the sound.
We score each of these filter programs, $f$, by the distance, $\distFxn(f(i), o)$, from their output to the output example file.
We then select the candidate filter program with the best score.
We continue this process until we either exceed our maximum allowed structural attempts or have reached a program whose output is close enough to the desired output.


\breakpagebeforenextheadingtrue
\section{Implementation and Numerical Results}
\label{sec:73results}

\minitoc[0mm]{69mm}{8}

\parbox{1em}{}
\vspace{-3em}

\disableornamentsfornextheadingtrue
\subsection{Implementation}
\label{sec:731implementation}

\paragraph{Parameters, implementation, and geometry}

Details about implementational aspects of the model can be found in
\multicite{Sprenger15Continuum,Roehrle16Two,Valentin18Gradient},
for instance, values for the material parameters.
%
The constitutive law has been implemented in the CMISS software package
(an interactive computer program for Continuum Mechanics,
Image analysis, Signal processing and System identification%
\footnote{%
  \url{https://www.cmiss.org/}%
}).
The emerging PDEs are discretized using quadratic finite element basis
functions and the resulting linearized system is solved with CMISS.
%
The geometry of the human upper limb model is based on
the Visible Human Male's dataset \cite{Spitzer96Visible}.
Again, we refer to \multicite{Sprenger15Continuum,Roehrle16Two} for details
about the geometry.

\subsection{Reference and Sparse Grid Solution}
\label{sec:732solutionTypes}

\paragraph{Reference solution}

Since the model is only two-dimensional, we can compute a reference solution
on a full grid.
To this end, we evaluate the exerted muscle forces $\forceT$ and $\forceB$ on
the full grid
\begin{equation}
  \{\ang{10}, \ang{11}, \dotsc, \ang{150}\} \times \{0, 0.1, \dotsc, 1\}
  \ni (\elbang, \actX),\quad
  X \in \{\mathrm{T}, \mathrm{B}\}.
\end{equation}
The resulting \num{1551} grid points
are interpolated with bicubic full grid splines%
\footnote{%
  Computed with the Geometric Tools Engine \cite{Schneider03Geometric},
  see \url{https://www.geometrictools.com/}.%
}
to obtain \term{reference solutions}
$\forceTref, \forceBref\colon
\clint{\ang{10}, \ang{150}} \times \clint{0, 1} \to \real$,
which are shown in \cref{fig:biomech2ReferenceForce}.
Due to the high resolution of the full grid,
we may assume that the reference solutions are accurate enough
to ensure $\forceTref \approx \forceT$ and $\forceBref \approx \forceB$.
We refer to the resulting equilibrium elbow angle
with $\equielbangref{\forceL}$.
It is displayed in
\cref{fig:biomech2ReferenceEquilibriumAngle}
for the loads of $\forceL = \SI{22}{\newton}$,
$\SI{-60}{\newton}$, and $\SI{180}{\newton}$.

\begin{figure}
  \includegraphics{biomech2ReferenceForce_1}%
  \;\;%
  \includegraphics{biomech2ReferenceForce_2}%
  \hfill%
  \rlap{\raisebox{53mm}{\;$\forceXref$ [\si{\kilo\newton}]}}%
  \includegraphics{biomech2ReferenceForce_3}%
  \caption[Reference triceps and biceps forces]{%
    Reference triceps and biceps forces $\forceXref$
    ($X \in \{\mathrm{T}, \mathrm{B}\}$).%
  }%
  \label{fig:biomech2ReferenceForce}%
\end{figure}

\begin{figure}
  \includegraphics{biomech2ReferenceEquilibriumAngle_4}%
  \\[2mm]%
  \subcaptionbox{%
    $\forceL = \SI{22}{\newton}$%
  }[49mm]{%
    \includegraphics{biomech2ReferenceEquilibriumAngle_1}%
  }%
  \hfill%
  \subcaptionbox{%
    $\forceL = \SI{-60}{\newton}$%
  }[49mm]{%
    \includegraphics{biomech2ReferenceEquilibriumAngle_2}%
  }%
  \hfill%
  \subcaptionbox{%
    $\forceL = \SI{180}{\newton}$%
  }[49mm]{%
    \includegraphics{biomech2ReferenceEquilibriumAngle_3}%
  }%
  \caption[Reference equilibrium elbow angle]{%
    Reference equilibrium elbow angle $\equielbangref{\forceL}$
    for different loads $\forceL$.
    The empty areas correspond to activation pairs
    at which $\equielbangref{\forceL}$ is not well-defined
    (see \cref{eq:equilibriumAngle}).%
  }%
  \label{fig:biomech2ReferenceEquilibriumAngle}%
\end{figure}

\paragraph{Sparse grid solution}

Additionally, we evaluate $\forceT$ and $\forceB$ at the $\ngp = 49$
grid points
\begin{equation}
  \{(\elbang^{(k,\mathrm{unif})}, \actX^{(k,\mathrm{unif})}) \mid
  k = 1, \dotsc, \ngp\}
  \subset \clint{\ang{10}, \ang{150}} \times \clint{0, 1},\quad
  X \in \{\mathrm{T}, \mathrm{B}\},
\end{equation}
of the uniform regular sparse grid $\interiorregsgset{n}{d}$ of
level $n = 5$ in $d = 2$ dimensions
without boundary points (to reduce the number of samples)
and at the sparse Clenshaw--Curtis grid
\begin{equation}
  \{(\elbang^{(k,\cc)}, \actX^{(k,\cc)}) \mid
  k = 1, \dotsc, \ngp\}
  \subset \clint{\ang{10}, \ang{150}} \times \clint{0, 1},\quad
  X \in \{\mathrm{T}, \mathrm{B}\},
\end{equation}
of the same size and level.%
\footnote{%
  The domain $\clint{\ang{10}, \ang{150}} \times \clint{0, 1}$
  is assumed to be implicitly normalized to the unit square
  $\clint{\*0, \*1}$.%
}
These values are interpolated using three
different hierarchical B-spline bases of degree $p = 1$, $3$, and $5$:
modified hierarchical uniform B-splines
$\bspl[\modified]{\*l,\*i}{p}$
(see \cref{sec:313modification}),
modified hierarchical Clenshaw--Curtis B-splines
$\bspl[\cc,\modified]{\*l,\*i}{p}$
(see \cref{sec:314nonUniform}), and
modified hierarchical uniform not-a-knot B-splines
$\bspl[\nak,\modified]{\*l,\*i}{p}$
(see \cref{sec:323modifiedNAKBSplines}).
The implementation was done using the sparse grid toolbox
\sgpp{} \cite{Pflueger10Spatially}.%
\footnote{%
  \url{http://sgpp.sparsegrids.org/}%
}
The corresponding interpolants and resulting quantities
are denoted with the superscripts
``$\sparse,\!p$'', ``$\sparse,\!p,\!\cc$'', or ``$\sparse,\!p,\!\nak$'',
respectively.
A superscript of ``$\sparse$'' without any further specification
means one of the three hierarchical B-spline bases in general.
Note that the equilibrium elbow angle is \emph{not} interpolated
(neither in the full grid nor in the sparse grid case),
but rather obtained by inserting the interpolated muscle forces
into \eqref{eq:totalMomentSurrogate} and \eqref{eq:equilibriumAngleSurrogate}.

\subsection{Errors of Muscle Forces and Equilibrium Angle}
\label{sec:733errors}

\paragraph{Quality of reference interpolants}

Before we turn to the sparse grid interpolants,
we assess the quality of the reference interpolants on the full grid.
For this purpose, we evaluate the full grid interpolants
$\forceTintp, \forceBintp$
at the sparse grid points $(\elbang^{(k)}, \actX^{(k)})$
(which are not a subset of the full grid points!)
and compare the resulting values with the known exact values
$\forceT(\elbang^{(k)}, \actT^{(k)})$ and
$\forceB(\elbang^{(k)}, \actB^{(k)})$
of the muscle forces $\forceT, \forceB$.
We also incorporate the known values at the sparse
Clenshaw--Curtis grid points.
In particular, let $G$ be the union of
$\{(\elbang^{(k,\mathrm{unif})}, \actX^{(k,\mathrm{unif})}) \mid
k = 1, \dotsc, \ngp\}$ and
$\{(\elbang^{(k,\cc)}, \actX^{(k,\cc)}) \mid k = 1, \dotsc, \ngp\}$.
We then approximate the relative $\Ltwo$ interpolation error
of the reference interpolants by
\begin{equation}
  \frac{\normLtwo{\forceX - \forceXref}}{\normLtwo{\forceX}}
  \approx
  \frac{
    \setsize{G}^{-1/2}
    \norm[2]{
      (\forceX(\elbang, \actX) - \forceXref(\elbang, \actX))_
      {(\elbang, \actX) \in G}
    }
  }{
    \setsize{G}^{-1/2}
    \norm[2]{(\forceX(\elbang, \actX))_{(\elbang, \actX) \in G}}
  },\quad
  X \in \{\mathrm{T}, \mathrm{B}\},
\end{equation}
where $\norm[2]{\cdot}$ is the Euclidean norm.%
\footnote{%
  We have $\setsize{G} = 2\ngp - 1$, since sparse grids of
  uniform and Clenshaw--Curtis type only
  share the center point $(\elbang, \actX) = (\ang{80}, 0.5)$,
  if there are no boundary points.%
}
After inserting the known values $\forceX(\elbang, \actX)$ and
$\forceXref(\elbang, \actX)$ ($(\elbang, \actX) \in G$)
on the right-hand side, %\rhs
we obtain
\begin{equation}
  \frac{\normLtwo{\forceT - \forceTref}}{\normLtwo{\forceT}}
  \approx \SI{2.19}{\permille},\qquad
  \frac{\normLtwo{\forceB - \forceBref}}{\normLtwo{\forceB}}
  \approx \SI{2.06}{\permille}.
\end{equation}
These errors are very small, which justifies our assumption of
$\forceTref \approx \forceT$ and $\forceBref \approx \forceB$.

\paragraph{Error of sparse grid muscle forces}

\Cref{tbl:biomech2ErrorL2_1} contains the relative $\Ltwo$
interpolation errors
$\normLtwo{\forceXref - \forceXintp}/\normLtwo{\forceXref}$
($X \in \{\mathrm{T}, \mathrm{B}\}$) of the sparse grid interpolants
for all hierarchical bases and degrees $p = 1, 3, 5$.
All reported errors are relatively small
due to the smoothness of the original functions
(cf.\ $\forceXref$ in \cref{fig:biomech2ReferenceForce}).
All in all, the modified Clenshaw--Curtis B-splines perform best,
achieving relative $\Ltwo$ errors of below \SI{3.6}{\permille}
in the cubic case.
Surprisingly, the not-a-knot B-splines are the worst choice in our
comparison.
Their corresponding errors exceed \SI{1}{\percent} for the triceps
and $p > 1$.
The possible reasons are two-fold:
First, there might be slight noise in the given muscle force data,
which is visible in \cref{fig:biomech2ReferenceForce},
as there seems to be a kink in $\forceBref$ at $\elbang \approx \ang{25}$.
Second, the employed regular sparse grids might be too coarse
as the higher convergence order of not-a-knot B-splines
only pays off in the asymptotic range (see \cref{sec:541interpolation}).
The same observations hold for the degree $p$,
for which $p = 3$ seems to be the best choice,
as the errors increase again for $p = 5$.

\begin{table}
  \newcommand*{\bi}{$\bspl[\modified]{l,i}{p}$}
  \newcommand*{\bii}{$\bspl[\cc,\modified]{l,i}{p}$}
  \newcommand*{\biii}{$\bspl[\nak,\modified]{l,i}{p}$}
  \subcaptionbox{%
    $\normLtwo{\forceXref - \forceXintp}/\normLtwo{\forceXref}$
    [\si{\permille}] given as triceps/biceps pairs
    ($X \in \{\mathrm{T}, \mathrm{B}\}$).%
    \label{tbl:biomech2ErrorL2_1}%
  }[85.2mm]{%
    \setnumberoftableheaderrows{1}%
    \begin{tabular}{%
      >{\kern\tabcolsep}=l<{\kern2mm}%
      +c<{\kern-1mm}+c<{\kern-1mm}+c<{\kern\tabcolsep}%
    }
      \toprulec
      \headerrow
      $p$&   $1$&                  $3$&                  $5$\\
      \midrulec
      \bi&   $3.60,7.12$&          $3.05,7.00$&          $\mathbf{2.98},7.90$\\
      \bii&  $\mathbf{3.28},4.35$& $3.31,\mathbf{3.56}$& $3.35,3.64$\\
      \biii& $3.60,7.12$&          $3.09,10.0$&          $7.13,24.6$\\
      \bottomrulec
    \end{tabular}%
  }%
  \hfill%
  \subcaptionbox{%
    $\normLtwo{\equielbangref{\forceL} - \equielbangintp{\forceL}}/
    \normLtwo{\equielbangref{\forceL}}$
    [\si{\permille}] for $\forceL = \SI{22}{\newton}$.%
    \label{tbl:biomech2ErrorL2_2}%
  }[59mm]{%
    \setnumberoftableheaderrows{1}%
    \begin{tabular}{%
      >{\kern\tabcolsep}=l<{\kern2mm}%
      +c<{\kern-1mm}+c<{\kern-1mm}+c<{\kern\tabcolsep}%
    }
      \toprulec
      \headerrow
      $p$&   $1$&    $3$&             $5$\\
      \midrulec
      \bi&   $4.15$& $3.74$&          $3.72$\\
      \bii&  $3.42$& $\mathbf{2.83}$& $2.86$\\
      \biii& $4.15$& $4.06$&          $8.28$\\
      \bottomrulec
    \end{tabular}%
  }%
  \caption[Relative $L^2$ errors of forces and equilibrium elbow angle]{%
    Relative $\Ltwo$ errors of triceps/biceps force \emph{(left)} and
    equilibrium elbow angle \emph{(right)}
    for different hierarchical bases $\basis{\*l,\*i}$ and
    B-spline degrees $p$.
    Highlighted entries are the best among those with
    the same hierarchical basis or the same degree
    (similar to Nash equilibria).%
  }%
  \label{tbl:biomech2ErrorL2}%
\end{table}

\vspace{\fill}

\Cref{fig:biomech2ErrorForce} shows the pointwise absolute error
$\abs{\forceXref(\elbang, \actX) - \forceXintp(\elbang, \actX)}$
for the modified B-splines $\bspl[\modified]{l,i}{p}$ and
$\bspl[\cc,\modified]{l,i}{p}$
on uniform and Clenshaw--Curtis grids in the cubic case $p = 3$.
Note that in contrast to usual interpolation settings,
the absolute errors $\abs{\forceXref - \forceXintp}$
shown in \cref{fig:biomech2ErrorForce} do not vanish at the
sparse grid points $(\elbang^{(k)}, \actX^{(k)})$
($X \in \{\mathrm{T}, \mathrm{B}\}$, $k = 1, \dotsc, \ngp$),
since $\forceXintp$ does not interpolate $\forceXref$
at these points.%
\footnote{%
  It would have been possible to construct $\forceXintp$
  as a sparse grid interpolant of $\forceXref$.
  However, building a spline surrogate ($\forceXintp$)
  of another spline surrogate ($\forceXref$) would skew the results.%
}
As it is typical for (modified) sparse grid interpolants,
the error is the largest near the boundary of the domain.
However, the Clenshaw--Curtis points help to decrease the error
due to the higher density of grid points near the boundary.
In the Clenshaw--Curtis case, the maximal errors are
\begin{equation}
  \normLinfty{\forceTref - \forceTintp[p,\cc]}
  \approx \SI{10.6}{\newton},\qquad
  \normLinfty{\forceBref - \forceBintp[p,\cc]}
  \approx \SI{9.51}{\newton},
\end{equation}
where $\normLinfty{\forceXref - \forceXintp[p,\cc]}
\ceq \max_{(\elbang, \actX)}
\abs{\forceXref(\elbang, \actX) - \forceXintp[p,\cc](\elbang, \actX)}$
(since the functions are continuous).
If we restrict the domain to
$\clint{\ang{31}, \ang{129}} \times \clint{0.15, 0.85}$
by omitting \SI{15}{\percent} on each side of the original domain,
then the maximal absolute errors drop to only
\SI{6.73}{\newton} (triceps) and \SI{0.967}{\newton} (biceps),
which is small compared to maximal possible forces of
around \SI{1}{\kilo\newton}.

\begin{figure}
  \includegraphics{biomech2ErrorForce_7}%
  \\[2mm]%
  \subcaptionbox{%
    $\abs{\forceXref - \forceXintp[p]}$ for
    $X = \mathrm{T}$ \emph{(left)} and
    $X = \mathrm{B}$ \emph{(right).}%
  }[74mm]{%
    \includegraphics{biomech2ErrorForce_1}%
    \hfill%
    \includegraphics{biomech2ErrorForce_2}%
  }%
  \hfill%
  \subcaptionbox{%
    $\abs{\forceXref - \forceXintp[p,\cc]}$ for
    $X = \mathrm{T}$ \emph{(left)} and
    $X = \mathrm{B}$ \emph{(right).}%
  }[74mm]{%
    \includegraphics{biomech2ErrorForce_3}%
    \hfill%
    \includegraphics{biomech2ErrorForce_4}%
  }%
  \caption[Absolute error of muscle forces]{%
    Absolute error of muscle forces $\forceT, \forceB$ for
    modified cubic B-splines ($p = 3$)
    on sparse grids of uniform type \emph{(left two plots)} and
    of Clenshaw--Curtis type \emph{(right two plots)}
    together with the points of the sparse grid \emph{(dots).}%
  }%
  \label{fig:biomech2ErrorForce}%
\end{figure}

\pagebreak

\paragraph{Error of the equilibrium elbow angle}

The relative $\Ltwo$ errors
$\normLtwo{\equielbangref{\forceL} - \equielbangintp{\forceL}}/
\normLtwo{\equielbangref{\forceL}}$
of the equilibrium elbow angle function are shown in
\cref{tbl:biomech2ErrorL2_1} for the load of
$\forceL = \SI{22}{\newton}$.
Modified cubic Clenshaw--Curtis B-splines achieve the best results.
Therefore, we use this type of hierarchical basis
for the remainder of this chapter.
Pointwise plots of the absolute error
$\abs{\equielbangref{\forceL} - \equielbangintp[p,\cc]{\forceL}}$
are presented in \cref{fig:biomech2ErrorEquilibriumAngle}.
Again, the maximal error is comparatively small:
For $\forceL = \SI{22}{\newton}$, it is only \ang{0.886}.
If we restrict the domain to $\clint{0.15, 0.85}^2$,
then this maximal error drops to \ang{0.103} (or \ang{;6.18;}),
as the areas near the boundary of $\clint{\*0, \*1}$
contribute the most to the error.

\begin{figure}
  \includegraphics{biomech2ErrorEquilibriumAngle_5}%
  \\[2mm]%
  \subcaptionbox{%
    $\forceL = \SI{22}{\newton}$%
  }[49mm]{%
    \includegraphics{biomech2ErrorEquilibriumAngle_1}%
  }%
  \hfill%
  \subcaptionbox{%
    $\forceL = \SI{-60}{\newton}$%
  }[49mm]{%
    \includegraphics{biomech2ErrorEquilibriumAngle_2}%
  }%
  \hfill%
  \subcaptionbox{%
    $\forceL = \SI{180}{\newton}$%
  }[49mm]{%
    \includegraphics{biomech2ErrorEquilibriumAngle_3}%
  }%
  \caption[Absolute error of the equilibrium elbow angle]{%
    Absolute error
    $\abs{\equielbangref{\forceL} - \equielbangintp[p,\cc]{\forceL}}$
    of the equilibrium elbow angle for
    modified hierarchical cubic Clenshaw--Curtis B-splines ($p = 3$)
    for different loads $\forceL$.
    In the empty areas, at least one of
    $\equielbangref{\forceL}$ and $\equielbangintp[p,\cc]{\forceL}$
    is not well-defined (see \cref{eq:equilibriumAngle}).%
  }%
  \label{fig:biomech2ErrorEquilibriumAngle}%
\end{figure}

\subsection{Test Scenario}
\label{sec:734scenario}

\paragraph{Definition of the test scenario}

In the following, we want to assess the performance
of the sparse grid interpolants for the optimization problems
\ref{item:biomech2MinSum} and \ref{item:biomech2MinDist}.
For this goal, we create a test scenario \cite{Valentin18Gradient}
that simulates a pseudo-dynamic sequence of motions
by varying the load force and/or the target elbow angle
in discrete time steps $t$ as seen in \cref{fig:biomech2ScenarioA_1}.
The test scenario is as follows:
%
\begin{figure}
  \includegraphics{biomech2ScenarioA_5}%
  \\[2mm]%
  \subcaptionbox{%
    Load $\forceL$ and target elbow angle $\tarelbang$.%
    \label{fig:biomech2ScenarioA_1}%
  }[73.3mm]{%
    \hspace*{4.5mm}%
    \includegraphics{biomech2ScenarioA_1}%
    \hspace*{2.4mm}%
  }%
  \hfill%
  \subcaptionbox{%
    Optimal activation parameters $\actT$ and $\actB$.%
    \label{fig:biomech2ScenarioA_2}%
  }[73.3mm]{%
    \hspace*{0.0mm}%
    \includegraphics{biomech2ScenarioA_2}%
    \hspace*{9.8mm}%
  }%
  \\[1mm]%
  \subcaptionbox{%
    Deviation $\abs{\equielbang{\forceL} - \tarelbang}$
    of attained elbow angle to target and
    deviation \smash{$|\momentref|$} of the moment from equilibrium.%
    \label{fig:biomech2ScenarioA_3}%
  }[73.3mm]{%
    \includegraphics{biomech2ScenarioA_3}%
  }%
  \hfill%
  \subcaptionbox{%
    Number of evaluations of $\equielbang{\forceL}$
    and number of Newton iterations per evaluation
    of $\equielbang{\forceL}$.%
    \label{fig:biomech2ScenarioA_4}%
  }[73.3mm]{%
    \includegraphics{biomech2ScenarioA_4}%
  }%
  \caption[Settings and results of the test scenario]{%
    Setting (a) of the test scenario and corresponding results (b, c, d).%
  }%
  \label{fig:biomech2ScenarioA}%
\end{figure}
%
\begin{enumerate}
  \item
  Find a feasible initial solution for problem \ref{item:biomech2MinSum}
  with $\forceL(t_0) \ceq \SI{22}{\newton}$ and
  $\tarelbang(t_0) \ceq \ang{75}$.

  \item
  Apply \ref{item:biomech2MinSum} with $\forceL(t_1) \ceq \SI{22}{\newton}$ and
  $\tarelbang(t_1) \ceq \ang{75}$.

  \item
  Apply \ref{item:biomech2MinDist} with $\forceL(t_2) \ceq \SI{22}{\newton}$ and
  $\tarelbang(t_2) \ceq \ang{60}$ (changed target angle).

  \item
  Apply \ref{item:biomech2MinDist} with $\forceL(t_3) \ceq \SI{30}{\newton}$ and
  $\tarelbang(t_3) \ceq \ang{60}$ (changed load).

  \item
  Apply \ref{item:biomech2MinDist} with $\forceL(t_4) \ceq \SI{40}{\newton}$ and
  $\tarelbang(t_4) \ceq \ang{50}$ (changed load and target angle).
\end{enumerate}
%
For each of the steps 2 to 5, the activation levels $\actT, \actB$ obtained
in the previous step (i.e., either the feasible initial solution
of step 1 or the optimal solution of steps 2 to 4) are used
as the input of the optimization problem
\ref{item:biomech2MinSum} or \ref{item:biomech2MinDist}.
The feasible initial solution in step 1 is determined as explained
in \cref{sec:513gradientBasedConstrained}.

\paragraph{Solutions of problem \ref{item:biomech2MinSum}}

We note that independently of $\forceL$ and $\tarelbang$,
every solution $(\actT, \actB)$ of problem \ref{item:biomech2MinSum} will be
on the boundary part of the domain $\clint{\*0, \*1}$,
on which at least one activation parameter vanishes, i.e.,
\begin{equation}
  \{(\actT, \actB) \in \clint{\*0, \*1} \mid
  (\actT = 0) \lor (\actB = 0)\}.
\end{equation}
The reason is that the two muscles triceps and biceps are antagonistic
(see \cref{sec:711models}), meaning that they work against each other.
If both $\actT > 0$ and $\actB > 0$, then the body will waste energy,
as the same target elbow angle can be attained by reducing both
$\actT$ and $\actB$ simultaneously, thus requiring less energy.
A visual example for this is \cref{fig:biomech2ReferenceEquilibriumAngle},
where the contour lines generally go from the bottom left
(small $\actT, \actB$) to the top right (large $\actT, \actB$).
This issue may be prevented by either
more complicated musculoskeletal models with more
than two muscles or different optimization problems
such as problem \ref{item:biomech2MinDist},
where the objective function differs.

\paragraph{Plots of optimization results}

\Cref{fig:biomech2ScenarioA_2,fig:biomech2ScenarioA_3,fig:biomech2ScenarioA_4}
show the results of the test scenario using the muscle forces
$\forceXintp[p,\cc]$ obtained by interpolating with
modified hierarchical cubic Clenshaw--Curtis B-splines (solid lines, $p = 3$).
As comparison, we repeat the solution process
with the forces obtained by interpolating with the
corresponding hierarchical piecewise linear basis (dashed lines, $p = 1$) and
with the reference forces $\forceXref$ (dotted lines).
For the piecewise linear basis,
we use exactly the same method as for the cubic case
(Newton method for $\equielbangintp{\forceL}$,
Augmented Lagrangian with adaptive gradient descent for the
solution of problems \ref{item:biomech2MinSum} and \ref{item:biomech2MinDist}),
although the derivatives of the muscle forces are discontinuous.
For the reference forces, we use the fact that the reference surrogates
are full grid spline interpolants, which can be explicitly differentiated.
Without the full grid interpolants,
we would have to approximate the derivatives with finite differences.

\vspace*{\fill}
\pagebreak

\paragraph{Equilibrium elbow angle}

In \cref{fig:biomech2ScenarioA_2}, we see that the activation levels
of all three methods are more or less the same.
However, \cref{fig:biomech2ScenarioA_3} reveals that even these small
differences lead to deviations of the resulting equilibrium elbow angle
to the target angle that differ by up to two orders of magnitude.
The two green lines with filled markers at the bottom of
\cref{fig:biomech2ScenarioA_3} show the error of
the equilibrium elbow angle $\equielbangintp{\forceL}$
using sparse grid interpolation to the desired target angle $\tarelbang$.
Unsurprisingly, this error is very small as
it is minimized by the optimizer as part of the constraint.
The true error, which is obtained by
using the reference equilibrium elbow angle $\equielbangref{\forceL}$,
is in general much larger
(top two green lines in \cref{fig:biomech2ScenarioA_3}
with hollow markers).
We see that the cubic B-splines decrease the error
by up to two orders of magnitude compared to the
piecewise linear basis.
There are two reasons for this:
First, the error of $\equielbang{\forceL}$ is generally smaller
when using higher-order B-splines as we have seen above.
Second, higher-order B-splines are continuously differentiable,
which makes them suitable for gradient-based optimization.
In contrast, the surrogates obtained by piecewise linear interpolation
have kinks, which may complicate finding optimal points
in the augmented Lagrangian and Newton methods.

\paragraph{Number of evaluations and Newton iterations}

This is supported by \cref{fig:biomech2ScenarioA_4},
which shows the number of evaluations of $\equielbang{\forceL}$
during the optimization and the average number of Newton iterations
per evaluation.
While the number of total evaluations is similar for all three methods,
the number of required Newton iterations to achieve convergence
is in general around \SI{50}{\percent} larger for the piecewise linear
basis functions.

\subsection{Spatial Adaptivity}
\label{sec:735adaptivity}

\paragraph{Generation of a spatially adaptive sparse grid}

As mentioned in \cite{Valentin18Gradient}, spatial adaptivity
may be employed to reduce the number of necessary muscle force samples
even further,
especially for more complicated musculoskeletal systems with
more parameters.
To verify this statement, we remove all grid points
$(\elbang^{(k,\cc)}, \actX^{(k,\cc)})$
from the regular sparse Clenshaw--Curtis grid that satisfy
\begin{equation}
  \frac{
    \abs{\alpha_\mathrm{T}^{(k,p,\cc)}}
  }{
    \max_{k'} \abs{\alpha_\mathrm{T}^{(k',p,\cc)}}
  } < \SI{1}{\percent}
  \quad\text{and}\quad
  \frac{
    \abs{\alpha_\mathrm{B}^{(k,p,\cc)}}
  }{
    \max_{k'} \abs{\alpha_\mathrm{B}^{(k',p,\cc)}}
  } < \SI{1}{\percent},
\end{equation}
where $\alpha_X^{(k,p,\cc)}$ ($X \in \{\mathrm{T}, \mathrm{B}\}$)
is the hierarchical surplus of the basis function $\bspl[\cc,\modified]{k}{p}$
corresponding to $(\elbang^{(k,\cc)}, \actX^{(k,\cc)})$.
For higher-dimensional models,
one would of course not sample muscle data on a regular sparse grid
and then coarsen the data by removing points,
but rather use an a posteriori adaptivity criterion to
decide which grid points to refine iteratively.

\paragraph{Comparison with the regular case}

For the cubic case $p = 3$,
the resulting force interpolants $\forceXintp[p,\cc,\mathrm{adap}]$ together
with the spatially adaptive sparse grid
(which has been coarsened from 49 to 28 points) and
equilibrium elbow angle $\equielbangintp[p,\cc,\mathrm{adap}]{\forceL}$
for $\forceL = \SI{22}{\newton}$ are shown in
\cref{fig:biomech2SpatiallyAdaptive}.
The sparse grid is almost dimensionally adaptive,
as $\forceXref$ seems to be almost linear in the $\actX$ direction
for both $X = \mathrm{T}$ and $X = \mathrm{B}$.
The errors increase slightly:
The relative $\Ltwo$ force errors for $(\mathrm{T}, \mathrm{B})$ increase
from $(\SI{3.31}{\permille}, \SI{3.56}{\permille})$
to $(\SI{3.36}{\permille}, \SI{4.43}{\permille})$,
and the absolute $\Linfty$ errors increase
from $(\SI{10.6}{\newton}, \SI{9.51}{\newton})$
to $(\SI{12.3}{\newton}, \SI{9.57}{\newton})$.
In addition, the relative $\Ltwo$ and absolute $\Linfty$ errors
for $\equielbang{\forceL}$ increase
from \SI{2.83}{\permille} and \ang{0.886}
to \SI{4.12}{\permille} and \ang{1.09}, respectively.
While all these errors are somewhat larger than for the regular sparse grid,
they are still at an acceptable level,
but the number of necessary muscle force evaluations is halved compared
to the regular case.
Additionally, the solution of the test scenario doesn't change significantly
due to the similar errors of $\forceX$ and $\equielbang{\forceL}$.

\begin{figure}
  \hspace*{2mm}%
  \raisebox{0.2mm}{\includegraphics{biomech2ErrorForce_8}}%
  \hspace*{12mm}%
  \includegraphics{biomech2ErrorEquilibriumAngle_6}%
  \\[2mm]%
  \subcaptionbox{%
    $\abs{\forceTref - \forceTintp[p,\cc,\mathrm{adap}]}$%
  }[49mm]{%
    \raisebox{1.02mm}{\includegraphics{biomech2ErrorForce_5}}%
  }%
  \hfill%
  \subcaptionbox{%
    $\abs{\forceBref - \forceBintp[p,\cc,\mathrm{adap}]}$%
  }[49mm]{%
    \raisebox{1.02mm}{\includegraphics{biomech2ErrorForce_6}}%
  }%
  \hfill%
  \subcaptionbox{%
    $\abs{
      \equielbangref{\forceL} - \equielbangintp[p,\cc,\mathrm{adap}]{\forceL}
    }$
    for $\forceL = \SI{22}{\newton}$%
  }[49mm]{%
    \includegraphics{biomech2ErrorEquilibriumAngle_4}%
  }%
  \caption[%
    Errors of muscle forces and equilibrium angle
    for the spatially adaptive case%
  ]{%
    Errors of muscle forces and equilibrium elbow angle
    for the spatially adaptive case
    (modified hierarchical cubic Clenshaw--Curtis B-splines,
    i.e., $p = 3$) together with the points of the
    spatially adaptive sparse grid \emph{(dots).}%
  }%
  \label{fig:biomech2SpatiallyAdaptive}%
\end{figure}