\chapter{The Bow}\label{ch:bow}
The bow is an extremely interesting excitation mechanism from a simulation perspective.
A bow excites a string with a force due to friction, which introduces a nonlinear element into the system.
The string `sticks' to the bow and `slips' again when the restoring force of the string is too great and overcomes the friction force. This `stick-slip' behaviour, first coined by Bowden and Leben in 1939 \cite{Bowden1939} causes the string to move in a characteristic triangular motion where the corner of the triangle moves back and forth along the string (see Figure \ref{fig:helmholtz}). Herman Helmholtz was the first to discover this behaviour in the 19\th century, and later got named \textit{Helmholtz motion} in his honour \cite{Helmholtz1860}.\footnote{Also see \url{https://www.youtube.com/watch?v=6JeyiM0YNo4}}

\begin{figure}[h]
    \centering
    \begin{tikzpicture}[->,node distance=3cm,
        thick,main node/.style={circle,draw}, scale=0.96, every node/.style={scale=0.96}]

        \node[] (image) at (0,0) {\includegraphics[width=1\columnwidth]{figures/exciters/helmholtz.eps}};

        \draw[thick, ->] (-0.25,0.60) arc (110:430:0.6);

      \end{tikzpicture}
    \caption{Helmholtz motion. If the bow moves up on the left side of the string, the `Helmholtz corner' travels anti-clockwise. \label{fig:helmholtz}}
\end{figure}

The Helmholtz motion gives bowed string instruments, such as the violin and cello, their characteristic sound. An ideal case is shown in Figure \ref{fig:helmholtzTimedomain}, where, for a string described by $u(x,t)$, the location of a point $x_\text{out}$ along the string follows a sawtooth-like motion due to the `stick-slip' behaviour.
% Figure \ref{fig:helmholtzImplementation} for a time domain output of one point along a bowed stiff string. The string is bowed upwards, and the `stick-slip' behaviour is visible (stick for upwards, slip for downwards motion), creating a sawtooth pattern.
\begin{figure}[t]
    \centering
    \includegraphics[width=\textwidth]{figures/exciters/helmholtzTimedomain.eps}
    \caption{An example of the (ideal) Helmholtz motion of a location $x_\text{out}$ along a bowed string. \label{fig:helmholtzTimedomain}}
\end{figure}

The rest of this chapter is structured as follows: first, a brief history of bowed-string simulations is presented. Then, an introduction to interpolation and spreading operators, as well as an introduction to the Newton-Raphson method will be given, both of which are necessary to work with bow-string interaction. Finally, a static and a dynamic friction model are presented. The latter, the elasto-plastic friction model, was applied to a FD scheme of the stiff string during this project and is one of the contributions, published in paper \citeP[C].

\section{Brief history of bowed-string simulation}
The first nonlinear systems in the context of musical instrument simulations, including the bowed string, were presented by McIntyre, et al. in 1983 \cite{McIntyre1983}. The first real-time implementation of the bowed string was due to Smith in 1986 and used digital waveguides for the string and a look-up table for the friction model \cite{Smith1986}. Simultaneously, Florens et al. presented a real-time implementation of the bowed string, but instead, the string was modelled using mass-spring systems and the friction model used a static friction model \cite{Florens1986} (see Section \ref{sec:staticFricMod}). One of the most complex friction models applied in a musical context to-date is the elasto-plastic friction model due to Dupont \cite{Dupont2002}, which Serafin et al. \cite{Serafin2003, Serafin2004} applied to a digital waveguide implementation of the string.

The first appearance of FDTD methods in bowed string simulations was in a publication by Pitteroff and Woodhouse in \cite{Pitteroff1998}. Later, Maestre et al. in \cite{Maestre2014} used FDTD methods to implement a thermal friction model proposed by Woodhouse in \cite{Woodhouse2003}. In both cases, the string was implemented using digital waveguides. Desvages implemented a bowed string model using a static friction model and a two-polarisation FDTD model for the string in \cite{Desvages2016, Desvages2018}, but did not implement this in real-time. The first real-time implementation of a bowed stiff string fully modelled using FDTD methods was presented in paper \citeP[A]. Finally, paper \citeP[C] presented the first (real-time) implementation of the elasto-plastic friction model applied to a FD scheme in a musical context.

\section{Interpolation and spreading operators}\label{sec:interpolationSpreading}
To listen to, or interact with a FD scheme between grid points, it is necessary to use some form of interpolation.
% As should be clear by now, FD schemes are an approximation to continuous space (and time) using a finite number of points. Working with the system between grid points is not impossible, but requires an extra step. If one would like to listen to a location between specified grid points, one can use interpolation.
For this end, an \textit{interpolation operator} $I(x_\itxt)$ can be introduced and can be applied to a grid function \cite{theBible}. This operator is a function of $x_\itxt$, the (continuous) location of interest and can be defined in various levels of accuracy. In this section, a 1D system $u(x,t)$ is assumed where $x\in \D$ for spatial domain $\D$. The theory presented in this section will be extended to 2D in Section \ref{sec:interpolationSpreading2D}.

An interpolation operator can be applied to a grid function $\uln$, which performs the following operation:
\begin{equation}
    I_{l,o}(x_\itxt)u_l^n = \sum_{l\in d} I_{l,o}(x_\itxt)\cdot u_l^n.
\end{equation}
Here, $o$ is the order of the operator, and $l\in d$ with discrete domain $d$, which needs to be the same for both $I_{l,o}(x_\text{i})$ and $\uln$.

The simplest interpolation operator is of '0\textsuperscript{th}'-order and is defined as
\begin{equation}\label{eq:zerothOrderI}
    I_{l,0}(x_\itxt) = \begin{cases}
        1, & \text{if}\ l = l_\itxt,\\
        0, & \text{otherwise},
    \end{cases}
\end{equation}
where the grid location of interest is defined as $l_\itxt = \floor[x_\itxt / h]$ (see Figure \ref{fig:zerothOrderInterpol}). Instead of actually performing an interpolation operation, $I_0$ simply floors its input to the grid location below.
%As an example, for a 1D system of length $L=1$ described by $u(x,t)$, the location of interest is at $x_\itxt = 0.25$. If the grid spacing $h = 0.2$ the grid point of interest will be $l_\itxt = \floor[0.25/0.2] = \floor[1.25] = 1$. Writing this in operator form yields
% \begin{equation*}
%     u(0.25, t)\approxeq I_{l,0}(0.25)\uln = u_1^n.
% \end{equation*}
%
% \input{exciters/interpolation.tex}
%
A slightly more accurate way to perform 0\thOrder interpolation is to round $x_\itxt/h$ to the nearest neighbour, rather than using the flooring operation.

First-order or linear interpolation uses the fractional part of the flooring operation as well, according to $\alpha_\itxt = x_\itxt / h - l_\itxt$ and is defined as
\begin{equation}\label{eq:firstOrderI}
    I_{l,1}(x_\itxt) = \begin{cases}
        (1-\alpha_\itxt), & \text{if}\ l = l_\itxt,\\
        \alpha_\itxt , & \text{if}\ l = l_\itxt + 1,\\
        0 & \text{otherwise.}
    \end{cases}
\end{equation}
See Figure \ref{fig:firstOrderInterpol}.
% Using the same example as before, $\alpha = 1.25 - \floor[1.25] = 0.25$ and
% \begin{equation*}
%     u(0.25, t)\approxeq I_{l,1}(0.25)\uln = 0.75u_1^n + 0.25 u_2^n,
% \end{equation*}

The highest order interpolator used in this project is the Lagrange cubic interpolator:
\begin{equation}\label{eq:cubicI}
    I_{l,3}(x_\itxt) = \begin{cases}
        -\alpha_\itxt (\alpha_\itxt-1)(\alpha_\itxt-2)/6, & l = l_\itxt-1,\\
        (\alpha_\itxt-1)(\alpha_\itxt+1)(\alpha_\itxt-2)/2,  & l = l_\itxt,\\
        -\alpha_\itxt (\alpha_\itxt+1)(\alpha_\itxt-2)/2, & l = l_\itxt + 1,\\
        \alpha_\itxt (\alpha_\itxt+1)(\alpha_\itxt-1)/6, & l = l_\itxt + 2,\\
        0, & \text{otherwise.}
    \end{cases}
\end{equation}
See Figure \ref{fig:cubicInterpol}.
% Applied to a grid function using the same example as before, yields
% \begin{equation*}
%     u(0.25, t)\approxeq I_{l,3}(0.25)\uln = -0.0547u_0^n + 0.8203u_1^n + 0.2734u_2^n -0.0391 u_3^n.
% \end{equation*}
Notice that the sum of all values of $I_{l, o}(x_\itxt)$, regardless of the order or the value of $\alpha_\itxt$, add up to $1$.

One could potentially create higher-order interpolation operators, but as one is restricted to a finite domain, the flexibility of the implementation reduces. Notice that if $\alpha_\itxt = 0$, the higher-order interpolators reduce to the $0$\thOrder interpolator in Eq. \eqref{eq:zerothOrderI}.
\def\figWidth{0.32}
\begin{figure}[h]
    \centering
    \subfloat[$0$\thOrder interpolation.\label{fig:zerothOrderInterpol}]{\includegraphics[width=\figWidth\textwidth]{figures/exciters/noInterpol2.pdf}}\hfill
    \subfloat[Linear interpolation.\label{fig:firstOrderInterpol}]{\includegraphics[width=\figWidth\textwidth]{figures/exciters/linearInterpol2.pdf}}\hfill
    \subfloat[Cubic interpolation.\label{fig:cubicInterpol}]{\includegraphics[width=\figWidth\textwidth]{figures/exciters/cubicInterpol2.pdf}}
    \caption{Interpolation with varying orders of accuracy.\label{fig:interpolation}}
\end{figure}

Apart from interpolation operators, one may define \textit{spreading operators} which can be interpreted as an inverse interpolation operation. A spreading operator $J(x_\itxt)$ is used to interact with a distributed FD scheme in the form of an excitation or other interactions such as connections or collisions between multiple schemes (also see Part \ref{part:interactions}).

The spreading operators can be defined in the same way as the interpolation operators described above, yielding a $0$\thOrder spreading operator
\begin{equation}\label{eq:zerothOrderJ}
    J_{l,0}(x_\itxt) = \frac{1}{h}\begin{cases}
        1, & \text{if}\ l = l_\itxt,\\
        0, & \text{otherwise},
    \end{cases}
\end{equation}
a linear spreading operator
\begin{equation}\label{eq:firstOrderJ}
    J_{l,1}(x_\itxt) = \frac{1}{h}\begin{cases}
        (1-\alpha_\itxt), & \text{if}\ l = l_\itxt,\\
        \alpha_\itxt , & \text{if}\ l = l_\itxt + 1,\\
        0 & \text{otherwise,}
    \end{cases}
\end{equation}
and a Lagrange cubic spreading operator
\begin{equation}\label{eq:cubicJ}
    J_{l,3}(x_\itxt) = \frac{1}{h}\begin{cases}
        -\alpha_\itxt (\alpha_\itxt-1)(\alpha_\itxt-2)/6, & l = l_\itxt-1,\\
        (\alpha_\itxt-1)(\alpha_\itxt+1)(\alpha_\itxt-2)/2,  & l = l_\itxt,\\
        -\alpha_\itxt (\alpha_\itxt+1)(\alpha_\itxt-2)/2, & l = l_\itxt + 1,\\
        \alpha_\itxt (\alpha_\itxt+1)(\alpha_\itxt-1)/6, & l = l_\itxt + 2,\\
        0, & \text{otherwise.}
    \end{cases}
\end{equation}
Notice the scaling by $1/h$ which will be more elaborated on in Chapter \ref{ch:connections}. As is the case with the interpolation operators, higher-order spreading operators reduce to Eq. \eqref{eq:zerothOrderJ} if $\alpha_\itxt = 0$.

Spreading operators approximate the spatial Dirac delta function $\delta(x-x_\itxt)$ (in m$^{-1}$), which is a test function defined as
\begin{equation}\label{eq:spatialDirac}
    \delta(x)= \begin{cases}
        \infty, & x = 0,\\
        0, & \text{otherwise},
    \end{cases} \qaq \int_{-\infty}^{\infty} \delta(x)dx = 1,
\end{equation}
used in continuous time to locate an external force to a location $x_\itxt$ along a system distributed over space $x$. Note that the definition in \eqref{eq:spatialDirac} will not be used directly. Instead, it can be approximated using the spreading operators presented in this section.

\subsubsection{Identities}
The following identity is extremely useful when solving systems including interpolation and spreading operators of the same order $o$ for any (grid) function $f$ and discrete domain $d$:
\begin{equation}\label{eq:identityIJ}
    \langle f_l, J_{l,o}(x_\itxt) \rangle_d = I_{l,o}(x_\itxt) f_l,
\end{equation}
where $l\in d$. From this, it follows that taking the norm of a spreading operator over a given domain is identical to applying to its `dual' interpolation operator (of the same order $o$ and same input $x_\itxt$):
\begin{equation}\label{eq:JnormIJ}
    \langle  J_{l,o}(x_\itxt), J_{l,o}(x_\itxt)\rangle_d = \lVert J_{l,o}(x_\itxt)\rVert^2_d = I_{l,o}(x_\itxt)J_{l,o}(x_\itxt).
\end{equation}
See Section \ref{sec:innerProduct} for more details on the inner product and the norm.

\subsubsection{Other distributions}
This section presented interpolation and spreading operators that interact with the state of a FD scheme at a single location $x_\itxt$ and either interpolates or distributes over a range of points. Although multiple grid points might be used for these operations, the interpolation or spreading is not \textit{distributed}. Physical exciters such as mallets or bows have a non-zero width and thus interact with a larger part of the system. One could make an arbitrary distribution function $E$ with elements $e_l$ (in 1D) where $l\in d$ for discrete domain $d$ of the system at hand. The \textit{distribution} and spreading operators become
\begin{equation}\label{eq:otherDistribution}
    I_l = \frac{e_l}{\sum_d e_l} \qaq J_l = \frac{1}{h}I_l.
\end{equation}
Although any values for $E$ would work, to retain correct scaling, the sum of $E$ needs to be normalised to $1$ as shown in Eq. \eqref{eq:otherDistribution}.

\section{The Newton-Raphson method}\label{sec:newtonRaphson}
Before moving on to more complex nonlinear excitation mechanisms, it is useful to go over the process of how to solve some of these mechanisms using an iterative root-finding method called the \textit{Newton-Raphson} method (or Newton-Raphson for short).

If a FD scheme not be solved explicitly, due to an implicit dependence on a variable for example, Newton-Raphson can be used. For a continuous and differentiable function $f(x) = 0$, its root can be approached using the following iteration
\begin{equation}\label{eq:newtonRaphson}
    x_{i+1} = x_i - \frac{f(x_i)}{f'(x_i)}\ ,
\end{equation}
with iteration number $i$ and the tick is used to denote a derivation with respect to $x$.
This iteration will then be carried out until the difference between the values of two consecutive iterations is smaller than a given threshold:
\begin{equation}\label{eq:NRthreshold}
    |x_{i+1} - x_i| < \epsilon,
\end{equation}
where $\epsilon$ is small, but its value depends on the situation at hand. To prevent Newton Raphson from iterating endlessly (if the iteration can not converge), one can put a cap on the number of iterations allowed.

Preferably, the starting point of the iteration, $x_0$, should be close to the value of where the root is expected to be. This is especially the case for a higher-ordered function with multiple roots (non-uniqueness) or many local variations.

Algorithm \ref{alg:NR} shows an example of an implementation of Newton-Raphson using $f(x) = e^x -1\ \Rightarrow\ f'(x) = e^x $ and Figure \ref{fig:NR} visualises the iterative algorithm.
\\
\begin{minipage}{\textwidth}
\setlstMAT
\begin{lstlisting}[caption={Example of an implementation of the Newton-Raphson method using $f(x) = e^{x} - 1$.}, label=alg:NR]
% An example of the Newton Raphson method using f(x) = exp(x) - 1

x = 1;          % starting point
eps = 1e-4;     % threshold

% if the threshold has not been crossed before this number of
% iterations, do not iterate more
maxIterations = 100;

% loop until a maximum number of iterations
for i = 1:maxIterations

    % calculate next iteration (Eq. %*\eqrefMatlab[eq:newtonRaphson]*)
    xNext = x - (exp(x) - 1) / (exp(x));

    % threshold check (Eq. %*\eqrefMatlab[eq:NRthreshold]*)
    if abs(xNext - x) < eps
        break; % break out of the for loop
    end

    % update the value of x
    x = xNext;
end
disp("The root of f(x) is at x = " + xNext)
\end{lstlisting}
\end{minipage}

\captionsetup[subfigure]{labelformat=empty}
\def\figWidth{0.30}
\begin{figure}[h]
    \centering
    \subfloat[]{\includegraphics[width=\figWidth\textwidth]{figures/exciters/NR1.eps}}\hfill
    \subfloat[]{\includegraphics[width=\figWidth\textwidth]{figures/exciters/NR2.eps}}\hfill
    \subfloat[]{\includegraphics[width=\figWidth\textwidth]{figures/exciters/NR3.eps}}
    \vspace{-2em}\caption{The Newton-Raphson method. The $x$-value of the root of the tangent line at $f(x_i)$ is used to evaluate the next iteration.\label{fig:NR}}
\end{figure}
\captionsetup[subfigure]{labelformat=parens}

\subsection{Multivariate Newton-Raphson}\label{sec:multivariateNR}
For $M$ functions $f_m$ dependent on the same number of independent variables $x_m$ with $m = \{1, \hdots, M\}$, Newton-Raphson can be extended to
\begin{equation}\label{eq:multivariateNR}
    \mathbf{x}_{i+1} =
    \mathbf{x}_i -
    {\underbrace{\begin{bmatrix}
        \frac{\partial f_1(\mathbf{x}_i)}{\partial x_1} & \hdots & \frac{\partial f_1(\mathbf{x}_i)}{\partial x_M}\\
        \vdots & \ddots & \vdots\\
        \frac{\partial f_M(\mathbf{x}_i)}{\partial x_1} & \hdots & \frac{\partial f_M(\mathbf{x}_i)}{\partial x_M}
    \end{bmatrix}}_{\mathbf{J}(\mathbf{x}_i)}}^{-1}
    \begin{bmatrix}
        f_1(\mathbf{x}_i)\\
        \vdots\\
        f_M(\mathbf{x}_i)
    \end{bmatrix},
\end{equation}
where the column vector $ \mathbf{x} = [x_1, \hdots, x_M]^T$ is the collection of independent variables, and the iteration number is (again) denoted by $i$.
The matrix in Eq. \eqref{eq:multivariateNR} is referred to as the \textit{Jacobian matrix} $\mathbf{J}$ and contains the derivatives of all functions with respect to each individual independent variable.

As an example, consider the following system of equations\footnote{Taken from \url{http://fourier.eng.hmc.edu/e176/lectures/NM/node21.html}}:
\begin{subequations}
    \begin{align}
        f_1(\mathbf{x}) &= 3 x_1 - \cos(x_2x_3) - 3/2 = 0,\\
        f_2(\mathbf{x}) &= 4x_1^2 - 625 x_2^2 + 2x_3-1 = 0,\\
        f_3(\mathbf{x}) &= 20 x_3 + e^{-x_1x_2}+9 = 0,
    \end{align}
\end{subequations}
where $\mathbf{x} = [x_1, x_2, x_3]^T$. The Jacobian matrix will be
\begin{equation*}
    \mathbf{J}(\mathbf{x}) = \begin{bmatrix}
        3 & x_3 \sin (x_2x_3) & x_2 \sin(x_2x_3)\\
        8x_1 & -1250 x_2 & 2\\
        -x_2e^{-x_1x_2} & -x_1e^{-x_1x_2} & 20
    \end{bmatrix},
\end{equation*}
and its roots can be found by iteratively calculating Eq. \eqref{eq:multivariateNR}.

\section{Static friction models}\label{sec:staticFricMod}
A friction model is a nonlinear function that is (at least) dependent on the relative velocity $\vrel$ between the bow and the string. This function scales how much the bow force affects the bowed object.
In static friction models, the friction force is defined as a function of this relative velocity only.
The first mathematical description of friction was proposed by Coulomb in 1773 \cite{Coulomb1773}\todo{check references here} to which static friction, or \textit{stiction}, was added by Morin in 1833 \cite{Morin1833} and viscous friction, or velocity-dependent friction, by Reynolds in 1886 \cite{Reynolds1886}. In 1902, Stribeck found a smooth transition between the static and the coulomb part of the friction curve now referred to as the Stribeck effect \cite{Stribeck1902}. The latter is still the standard for static friction models today.

Many static friction models contain a discontinuity where the relative velocity between the  $\vrel = 0$ due to a multiplication with $\sgn(\vrel)$ in their definition. In this project, only the following static friction model has been used \cite{theBible}
\begin{equation}\label{eq:staticFriction}
    \Phi (\vrel) = \sqrt{2a}\vrel e^{-a\vrel^2 + 1/2},
\end{equation}
as it is continuous and differentiable, but still approximating discontinuous bow models. These characteristics make this model easier to work with in implementation. A definition for $\vrel$ will be given below.

\begin{figure}[h]
    \centering
    \includegraphics[width=0.8\textwidth]{figures/exciters/frictionCharacteristic.eps}
    \caption{The friction model in Eq. \eqref{eq:staticFriction} with $a = 100$. \label{fig:frictionCharacteristic}}
\end{figure}\todo{FULL DOC SWEEP: check figure centering}
\subsection{The bowed stiff string}\label{sec:bowedStiffStringStatic}
Consider a stiff string, its transverse displacement described by $u(x,t)$ defined for $x\in \D$ (see Chapter \ref{ch:stiffString}). The relative velocity between the string at bow position $x_\Btxt = x_\Btxt(t) \in \D$ and the bow can then be described as
\begin{equation}\label{eq:vrel}
    \vrel = \pt u(x_\Btxt, t) - v_\Btxt(t)
\end{equation}
(in m/s) with bow velocity $v_\Btxt = v_\Btxt(t)$ (in m/s).

Recalling the PDE of the stiff string in Eq. \eqref{eq:stiffStringPDE}
\begin{equation}
    \rho A \ptt u = T \pxx u - EI \pxxxx u - 2 \sz \rho A \pt u + 2 \so \rho A\pt \pxx u,
\end{equation}
one can add the bow force to the equation according to
\begin{equation}\label{eq:bowedStiffStringPDE}
    \rho A \ptt u = T \pxx u - EI \pxxxx u - 2 \sz \rho A \pt u + 2 \so \rho A\pt \pxx u - \delta(x-x_\Btxt)f_\Btxt\Phi(\vrel)
\end{equation}
where spatial Dirac delta function $\delta(x-x_\Btxt)$ (in m$^{-1}$) (see Section \ref{sec:interpolationSpreading}) positions the bow along the string and $f_\Btxt = f_\Btxt(t) \geq 0$ is the bow force (in N).\footnote{If the spatial Dirac delta function were omitted, the bow force would be applied to the entire string domain rather than only the bow location $x_\text{B}$.}

\subsubsection{Intuition}
From Eq. \eqref{eq:bowedStiffStringPDE} it can be seen that the bow force gets scaled by the friction model $\Phi(\vrel)$ shown in Figure \ref{fig:frictionCharacteristic}.
The figure shows that if $\vrel$ is too large (either positively or negatively) the bow term in \eqref{eq:bowedStiffStringPDE} becomes $0$. If, on the other hand, $\vrel$ is closer to $0$, the bow will have an effect on the string. This can be interpreted in terms of static and dynamic friction\footnote{`Static' and `dynamic' friction are unrelated to `static' and `dynamic' friction models.}. A stationary object requires more force to be moved than a moving object, i.e., the static friction coefficient is always higher than the dynamic friction coefficient. This is essentially what the friction model tries to simulate.

It might seem counter-intuitive that the bow term is subtracted from the scheme. This is due to the definition of the relative velocity in Eq. \eqref{eq:vrel}. For a negative bow velocity $v_\Btxt$, $\vrel$ becomes positive. As $\sgn\left(\Phi(\vrel)\right) = \sgn(\vrel)$ through Eq. \eqref{eq:staticFriction} and $f_\Btxt\geq 0$, the term $\delta(x-x_\Btxt)f_\Btxt\Phi(\vrel)$ will be positive for a positive $\vrel$. As a negative $v_\Btxt$ needs to have a downwards, or negative, effect on the string, the sign of the bow term also needs to be negative to achieve this.

\subsubsection{Discrete time}
Dividing all terms in Eq. \eqref{eq:bowedStiffStringFDS} by $\rho A$ and discretising the system yields
%
\begin{equation}\label{eq:bowedStiffStringFDS}
    \dtt \uln = c^2 \dxx \uln - \kappa^2 \dxxxx \uln - 2 \sz \dtd \uln + 2 \so\dtm\dxx \uln - J_l(x_\Btxt^n)F_\Btxt^n\Phi(\vrel^n),
\end{equation}
with $F_\Btxt^n = f_\Btxt^n/\rho A$ and spreading operator $J_l(x_\Btxt^n) = J_{l,o}(x_\Btxt^n)$ (in m$^{-1}$) as described in Section \ref{sec:interpolationSpreading}, where the order $o$ remains undetermined for now. As the bow position, bow velocity and bow force are time-dependent, these have a superscript $n$ in discrete time. These parameters are called \textit{control parameters} and will be supplied by the performer in an eventual implementation.

The relative velocity in Eq. \eqref{eq:vrel} is discretised using a centred difference operator according to
\begin{equation}\label{eq:vrelDisc}
    \vrel^n = I_l(x_\Btxt^n)\dtd \uln - v_\Btxt^n,
\end{equation}
with interpolation operator $I_l(x_\Btxt^n) = I_{l,o}(x_\Btxt^n)$ and is of the same order as $J_l(x_\Btxt^n)$. Equation \eqref{eq:vrelDisc} makes the scheme implicit due to the centred difference operator as the FD scheme in Eq. \eqref{eq:bowedStiffStringFDS} is now nonlinearly dependent on $u_l^{n+1}$. To solve Eq. \eqref{eq:bowedStiffStringFDS} for $u_l^{n+1}$, an iterative root-finding algorithm, such as Newton-Raphson described in Section \ref{sec:newtonRaphson}, is required. This process could be circumvented by using a backward  difference operator in Eq. \eqref{eq:vrelDisc}, but this will affect the accuracy and behaviour of the bow model.

\subsubsection{Solution}
To find a solution for $u_l^{n+1}$ at the bow location, an inner product of the scheme in Eq. \eqref{eq:bowedStiffStringFDS} with spreading operator $J_l(x_\Btxt^n)$ must be taken over the discrete domain of the string $d$ which isolates the scheme at the bowing location. Performing this operation and using identity \eqref{eq:identityIJ} yields
\begin{equation}\label{eq:bowLocFDS}
    \begin{aligned}
    I_l(x_\Btxt^n)\dtt\uln =\ &c^2 I_l(x_\Btxt^n)\dxx \uln - \kappa^2 I_l(x_\Btxt^n)\dxxxx \uln- 2 \sz I_l(x_\Btxt^n)\dtd \uln\\
    & + 2 \so I_l(x_\Btxt^n)\dtm\dxx \uln - \lVert J_l(x_\Btxt^n)\rVert^2_d F_\Btxt^n\Phi(\vrel^n).
    \end{aligned}
\end{equation}
One can rewrite Eq. \eqref{eq:vrelDisc} to
\begin{equation}\label{eq:vrelRewritten}
   I_l(x_\Btxt^n)\dtd \uln =  \vrel^n + v_\Btxt^n,
\end{equation}
and using identity \eqref{eq:identity1}, Eq. \eqref{eq:bowLocFDS} can be rewritten and assigned to a function $g(\vrel^n)$ according to
\begin{equation}\label{eq:staticFrictionNRfunc}
    g(\vrel^n) = \left(\frac{2}{k} + 2\sz\right)\vrel^n + \lVert J_l(x_\Btxt^n)\rVert^2_d F_\Btxt^n\Phi(\vrel^n) + b^n = 0,
\end{equation}
% Here, $b$ contains all terms not dependent on $\vrel$. As the operators can be interchanged, i.e., the order of application does not matter, could write $b$ according to
where the terms not dependent on $\vrel$ are collected in\footnote{An interpolation operator applied to a spatial derivative can be expanded in a similar fashion to when it is applied to a grid function. A first-order interpolator applied to $\dxx \uln$ thus yields $I_{l,1}(x_\itxt) \dxx \uln = (1-\alpha_\itxt)\dxx u_{l_\itxt}^n + \alpha \dxx u_{l_\itxt+1}^n$.}
\begin{equation}\label{eq:staticbDef}
    \begin{aligned}
        b^n =\ &-\frac{2}{k}I_l(x_\Btxt^n)\dtm \uln-c^2 I_l(x_\Btxt^n)\dxx \uln + \kappa^2 I_l(x_\Btxt^n)\dxxxx \uln \\
        &+ \left( \frac{2}{k} + 2 \sz \right)v_\Btxt^n - 2 \so I_l(x_\Btxt^n)\dtm\dxx \uln.
    \end{aligned}
    % b =&\ \left(-\frac{2}{k}\dtm -c^2 \dxx+ \kappa^2\dxxxx - 2 \so \dtm\dxx\right) I(x_\Btxt^n)\uln + \left( \frac{2}{k} + 2 \sz \right)v_\Btxt^n.
\end{equation}
%

One can then perform the Newton-Raphson method detailed in Section \ref{sec:newtonRaphson} to iteratively solve for $\vrel$
%
\begin{equation}
    (\vrel^n)_{i+1} = (\vrel^n)_i - \frac{g\left((\vrel^n)_i\right)}{g'\left((\vrel^n)_i\right)}\ ,
\end{equation}
where
\begin{equation*}
    g'(\vrel^n) = \frac{2}{k} + 2\sz + \lVert J_l(x_\Btxt^n)\rVert^2_\D F_\Btxt^n\Phi'(\vrel^n),
\end{equation*}
and
\begin{equation*}
    \begin{aligned}
        \Phi'(\vrel^n) %&= \sqrt{2a}e^{-a(\vrel^n)^2 + 1/2} + \sqrt{2a}\vrel^n * (-2a\vrel^n)e^{-a(\vrel^n)^2 + 1/2}\\
        %&
        = \sqrt{2a}\left(1 - 2a(\vrel^n)^2\right)e^{-a(\vrel^n)^2+1/2}.
    \end{aligned}
\end{equation*}
\subsection{Implementation and output}\label{sec:staticImplementation}
To implement the bowed stiff string, one must perform the Newton-Raphson iteration every sample. For the implementation shown in this section, the threshold in Eq. \eqref{eq:NRthreshold} has been set to $\epsilon = 10^{-7}$ and the maximum number of iterations to $100$. In the following, the parameters used for the string are listed in Table \ref{tab:stiffStringParams} with $T = 1000$ N and those for the bow are
\begin{equation*}
    x_\Btxt = \floor[0.125/h],\quad f_\Btxt = 1\ \text{N},\quad v_\Btxt = 0.2\  \text{m/s},\qaq a = 100.
\end{equation*}
Figure \ref{fig:staticBowImplementation} shows the behaviour of a bowed stiff string at the beginning of the simulation and shows the characteristic stick-slip behaviour. Figure \ref{fig:helmholtzState} shows the state of the same simulation 3 seconds later. One can observe the Helmholtz corner move in an anti-clockwise direction as presented in Figure \ref{fig:helmholtz}. Finally, the time domain output of the string at the bowing location is shown in Figure \ref{fig:helmholtzImplementation} generally follows the Helmholtz motion shown in Figure \ref{fig:helmholtzTimedomain}.

\begin{figure}[h]
    \centering
    \includegraphics[width=\textwidth]{figures/exciters/bowImplementation.eps}
    \caption{Behaviour of a bowed stiff string at the beginning of the simulation. \label{fig:staticBowImplementation}}
\end{figure}
\begin{figure}[h]
    \centering
    \includegraphics[width=\textwidth]{figures/exciters/helmholtzState.eps}
    \caption{Behaviour of a bowed stiff string simulation after 3 seconds. The string exhibits a Helmholtz motion as presented in Figure \ref{fig:helmholtz}. \label{fig:helmholtzState}}
\end{figure}
\begin{figure}[h]
    \centering
    \includegraphics[width=\textwidth]{figures/exciters/helmholtzImplementation.eps}
    \caption{Time domain output at the bowing location of a bowed stiff string using the static friction model in Eq. \eqref{eq:staticFriction}. \label{fig:helmholtzImplementation}}
\end{figure}
\subsection{Energy analysis}
Following the energy analysis of the stiff string presented in Section \ref{sec:energyAnalysisString}, taking an inner product of Eq. \eqref{eq:bowedStiffStringFDS} (after multiplication by $\rho A$) with $(\dtd \uln)$ over discrete domain $d$ one arrives at the following
\begin{align*}
    \dtp \h + \q &=  - \langle (\dtd \uln), J_l(x_\Btxt^n)f_\Btxt^n\Phi(\vrel^n) \rangle_d\\[-5pt]
    \xLeftrightarrow{\mystrut\ \text{Eq. \eqref{eq:identityIJ}}\ }\qquad &= -I_l(x_\Btxt^n)(\dtd \uln) f_\Btxt^n \Phi(\vrel^n) \\[-5pt]
    \xLeftrightarrow{\mystrut\ \text{Eq. \eqref{eq:vrelRewritten}}\ } \qquad &= \underbrace{-f_\Btxt^n \Phi(\vrel^n) \vrel^n}_{\text{loss}} - \underbrace{f_\Btxt^n \Phi(\vrel^n)v_\Btxt^n}_{\text{power}},
\end{align*}
where $\h$ and $\q$ are as defined in Eqs. \eqref{eq:energyBalanceStiffString} and \eqref{eq:dampingTermStiffString} respectively. As $\sgn\left(\Phi(\vrel^n)\right) = \sgn\left(\vrel^n\right)$ through Eq. \eqref{eq:staticFriction}, one can observe that the first term on the right-hand side always has a negative effect on the rate of change of the total energy. This term can therefore be interpreted as the loss of power through the bow (as indicated). The last term is of indeterminate sign and can thus be interpreted as the power supplied by the bow.

The final energy balance can thus be written as
\begin{equation}\label{eq:energyBalanceBow}
    \dtp \h = -\q - \q_\text{B} - \p
\end{equation}
where
\begin{equation*}
    \q_\text{B} = f_\Btxt^n \Phi(\vrel^n) \vrel^n \qaq \p = f_\Btxt^n \Phi(\vrel^n)v_\Btxt^n.
\end{equation*}
Figure \ref{fig:energyStaticBow} shows the energy of the bowed stiff string corresponding to the wave propagation in Figure \ref{fig:staticBowImplementation}. The bow only injects energy into the system when it sticks to the string.
% Although the normalised energy shows `jumpy` behaviour, correlated to the behaviour of the system, this decreases for a lower value of $\epsilon$.

\begin{figure}[h]
    \centering
    \begin{tikzpicture}[->,node distance=3cm,
        thick,main node/.style={circle,draw}]

        \node[] (image) at (0,0) {
        \includegraphics[width=\textwidth]{figures/exciters/energyBow.eps}
        };

        \node[] (he) at (0.2,0.5) {\small $\mathfrak{h}_\text{e}$};

        \node[] (h) at (-5.8, 1) {\small $\mathfrak{h}$};
        \node[] (v) at (-5.8, 0.5) {\small $\color{red}\mathfrak{v}$};
        \node[] (t) at (-5.8, 0) {\small $\color{blue}\mathfrak{t}$};
      \end{tikzpicture}
      \caption{The kinetic (blue), potential (red), and total (black) energy of the bowed stiff string. The right panel shows the normalised energy (according to Eq. \eqref{eq:normalisedEnergyDamping}) and shows that the deviation of the energy is within machine precision. \label{fig:energyStaticBow}}
\end{figure}

\section{Dynamic friction models}
As opposed to static friction models, dynamic friction models relate the relative velocity to the friction force using a differential equation. Dynamic friction models exhibit a phenomenon called \textit{hysteresis}, which is the dependence of a system on its history. This hysteresis loop is in the force versus velocity plane and has been confirmed by measurements using a bowing machine in \cite{Woodhouse2003}.

The first dynamic friction model was due to Dahl \cite{Dahl1968} and captured hysteresis effects. The Stribeck effect was, however, not taken into account. The LuGre model (named after the collaboration between Lund and Grenoble) was then proposed by Canudas de Wit et al. in \cite{Canudas1993, Canudas1995} and extended the Dahl model to take the Stribeck effect into account. The model assumes a large ensemble of bristles between the two sliding surfaces, each of which contributes a tiny amount to the total friction force. The drawback of this model is that it exhibits drift for extremely small external forces. In \cite{Dupont2002}, Dupont et al. extended the LuGre model by allowing for a purely elastic regime that solves the drift issue. This model is referred to as the \textit{elasto-plastic} friction model and is used in this project.

\subsection{The elasto-plastic friction model}\label{sec:elastoPlastic}
In a musical context, the elasto-plastic friction has been investigated in-depth by Serafin et al. in \cite{Serafin2003, Serafin2004, Avanzini2005}. Like the LuGre model, the elasto-plastic friction model assumes that the friction between the bow and the string is caused by a large quantity of bristles, all contributing a fraction of the total amount of friction. See Figure \ref{fig:elastoPlastic}.

\input{exciters/elastoPlasticFig.tex}

Unless denoted otherwise, this section follows the original model by Dupont et al. in \cite{Dupont2002}, but with the appropriate corrections added as presented in paper \citeP[C]. As opposed to the static friction model described in the previous section, the friction force $f$ (in N) is now dependent on the average bristle displacement $z = z(t)$ (in m) as well as the relative velocity $v = v(t)$ (in m/s). The friction force is defined as
\begin{equation}\label{eq:elastoPlasticForce}
    f(v, z) = s_0z + s_1\dot z + s_2v + s_3w,
\end{equation}
with bristle stiffness $s_0 \geq 0$ (in N/m), bristle damping $s_1 \geq 0$ (in kg/s), viscous friction $s_2 \geq 0$ (in kg/s) and, as presented in \cite{Serafin2004}, a dimensionless noise coefficient $s_3$ multiplied onto a pseudorandom function $w = w(t)$ (in N) generating values between $-1$ and $1$. Moreover, for a string defined over domain $\D$, the relative velocity between the string at bowing location $x_\Btxt = x_\Btxt(t) \in \D$ and the bow is (similar to Eq. \eqref{eq:vrel})
\begin{equation}\label{eq:vrelElasto}
    v = \pt u(x_\Btxt,t) - v_\Btxt,
\end{equation}
with bow velocity $v_\Btxt = v_\Btxt(t)$ (in m/s). Lastly, $\dot z$ is the rate of change of the bristle displacement (in m/s) and is related to $v$ according to
\begin{equation}\label{eq:zdot}
    \dot z = r(v, z) = v \bigg[ 1-  \alpha(v, z)\frac{z}{z_\text{ss}(v)}\bigg].
\end{equation}
Here, $z_\text{ss}$ is the steady-state function
\begin{equation}\label{eq:zss}
    z_\text{ss}(v) = \frac{\sgn(v)}{s_0}\left[f_\text{C}+(f_\text{S}-f_\text{C})e^{-(v/v_\text{S})^2}\right],
\end{equation}
where the $v_\text{S}$ is the Stribeck velocity (in m/s). Furthermore, using the normal force $f_\text{N} = f_\text{N}(t)$ (in N), the Coulomb force and stiction force can be calculated according to $f_\text{C} = f_\text{N}\mu_\text{C}$ and $f_\text{S} = f_\text{N}\mu_\text{S}$ respectively (both in N). In these definitions $\mu_\text{C}$ and $\mu_\text{S}$ are the dimensionless dynamic and static friction coefficients respectively. A plot of the steady state function can be found in Figure \ref{fig:zss}.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.8\textwidth]{figures/exciters/steadyState.eps}
    \caption{\label{fig:zss}The steady-state function $z_\text{ss}(v)$ plotted against relative velocity $v$ with $s_0 = 10^4$, $\mu_\text{C} = 0.3$, $\mu_\text{S} = 0.8$, $v_\text{S} = 0.1$ and $f_\text{N} = 5$.}
\end{figure}

Finally, $\alpha(v,z)$ in Eq. \eqref{eq:zdot} is an adhesion map between the bow and the string and is defined as

\begin{equation}\label{eq:adhesionMap}
\alpha(v, z) =
\begin{aligned}
    \begin{cases}
    \begin{rightcases}
        &0 & |z| \leq z_\text{ba}\\
        &\alpha_\text{m}(v,z)&\ \ \ \, z_\text{ba}<|z|<|z_\text{ss}(v)|\\        &1 &|z|\geq|z_\text{ss}(v)|
        \end{rightcases}

        &\text{if}\  \sgn(v)=\sgn(z)\\
        \,0&\!\,\text{if}\  \sgn(v)\neq\sgn(z),
    \end{cases}
    \end{aligned}
\end{equation}
where the transition between the elastic and plastic behaviour is defined as
\begin{equation}\label{eq:alphaM}
    \alpha_\text{m} = \frac{1}{2}\bigg[1+\sgn(z)\sin\bigg(\pi\frac{z-\sgn(z)\frac{1}{2}(|z_\text{ss}(v)|+z_\text{ba})}{|z_\text{ss}(v)|-z_\text{ba}}\bigg)\bigg],
\end{equation}
where the break-away displacement $z_\text{ba} = z_\text{ba}(t) = 0.7 f_\text{C}/s_0$ determines the value of $z$ before bristles start to break. The adhesion map is visualised in Figure \ref{fig:alphaPlot} and relates to Figure \ref{fig:elastoPlastic} as described in its caption.
\begin{figure}[ht]
    \centering
    \includegraphics[width=0.8\textwidth]{figures/exciters/drawAlpha3.eps}
    \caption{\label{fig:alphaPlot}A plot of the adhesion map $\alpha(v,z)$ in Eq. \eqref{eq:adhesionMap} plotted against $z$ when $\sgn(v)  = \sgn(z)$. The different coloured regions correspond to Figure \ref{fig:elastoPlastic} according to: yellow - a) \& b), orange - c) and red - d). (Adapted from paper \citeP[C].)}
\end{figure}

\subsubsection{Discrete time}
Equation \eqref{eq:elastoPlasticForce} can be discretised to
\begin{equation}\label{eq:discForceFunction}
    f(v^n,z^n) = s_0z^n + s_1r^n+s_2v^n+s_3w^n
\end{equation}
where Eq. \eqref{eq:vrelElasto} can be discretised to
\begin{equation}\label{eq:vrelElastoDisc}
    v^n = I_l(x_\Btxt^n)\dtd\uln - v_\Btxt^n,
\end{equation}
with interpolation operator $I_l(x_\Btxt^n) = I_{l,o}(x_\Btxt^n)$ (of unspecified order $o$) and
\begin{equation}\label{eq:r}
    r^n = r(v^n,z^n) = v^n\bigg[1-\alpha(v^n,z^n)\frac{z^n}{z_\text{ss}(v^n)}\bigg]
\end{equation}
is the discrete counterpart of Eq. \eqref{eq:zdot}. The discrete adhesion map is identical to the continuous definition given in Eqs. \eqref{eq:adhesionMap} and \eqref{eq:alphaM}, but with superscripts $n$ added for appearances of $v$ and $z$.

\subsection{Applied to a FDTD stiff string}
The first appearance of a contribution in this work is the elasto-plastic friction model applied to a stiff string implemented using FDTD methods. This has been presented in paper \citeP[C] and will be summarised in this section. \todo{state the stability stuff here?}

In the same way as done with the static friction model in Section \ref{sec:staticFricMod}, one can add the friction force to the stiff string PDE in Eq. \eqref{eq:stiffStringPDE} and discretise the system as follows:
%
\begin{equation}\label{eq:elastoStiffStringFDS}
    \dtt \uln = c^2 \dxx \uln - \kappa^2 \dxxxx \uln - 2 \sz \dtd \uln + 2 \so\dtm\dxx \uln - J_l(x_\Btxt^n)\frac{f(v^n,z^n)}{\rho A},
\end{equation}
where spreading operator $J_l(x_\Btxt^n) = J_{l,o}(x_\Btxt^n)$ and is of the same order as $I_l(x_\Btxt^n)$ in Eq. \eqref{eq:vrelElastoDisc}. Following the same procedure as for the static friction model in Section \ref{sec:staticFricMod}, one takes an inner product with $J_l(x_\Btxt^n)$ over discrete string domain $d$ and using identities \eqref{eq:identityIJ} and \eqref{eq:identity1}, one can rewrite this similar to the static friction model in Eq. \eqref{eq:staticFrictionNRfunc} as
%
\begin{equation}\label{eq:elastog1}
    g_1(v^n,z^n) = \left(\frac{2}{k} + 2\sz\right)v^n + \lVert J_l(x_\Btxt^n)\rVert_d^2\frac{f(v^n,z^n)}{\rho A}+ b^n= 0,
\end{equation}
where
\begin{align*}
    b^n =&-\frac{2}{k}I_l(x_\text{B}^n)\delta_{t-}u_l^n - c^2 I_l(x_\text{B}^n)\delta_{xx} u_l^n +\kappa^2I_l(x_\text{B}^n)\delta_{xxxx} u_l^n\\
    &+\left(\frac{2}{k}+ 2\sigma_0\right)v_\text{B}^n-2\sigma_1I_l(x_\text{B}^n)\delta_{t-}\delta_{xx}u_l^n.
\end{align*}
As $g_1$ contains two unknown variables $v^n$ and $z^n$ that need to be solved for, the multivariate Newton-Raphson method presented in \ref{sec:multivariateNR} must be performed. To be able to do this, an extra function must be included.

As $r$ describes $\dot z$ in Eq. \eqref{eq:zdot}, one can take another approach to approximate $\dot z$ using the trapezoid rule \cite{theBible}
\begin{equation}
    a^n = (\mtm)^{-1}\dtm z^n \quad \Longrightarrow\quad a^n = \frac{2}{k}(z^n - z^{n-1}) + a^{n-1}.
\end{equation}
As both $a^n$ and $r^n$ approximate $\dot z$, these can be used to create the second function necessary to solve the full system
\begin{equation}\label{eq:elastog2}
    g_2(v^n, z^n) = r^n - a^n = 0.
\end{equation}
Finally, one can use the multivariate Newton-Raphson method described in Section \ref{sec:multivariateNR}, which results in
\begin{equation}\label{eq:NRit}
    \begin{bmatrix}
    v^n\\
    z^n
    \end{bmatrix}_{i+1}
    =
    \begin{bmatrix}
    v^n\\
    z^n
    \end{bmatrix}_i
    -
    \begin{bmatrix}
    \frac{\partial g_1}{\partial v} & \frac{\partial g_1}{\partial z}\\
    \frac{\partial g_2}{\partial v} & \frac{\partial g_2}{\partial z}\\
    \end{bmatrix}^{-1}
    \begin{bmatrix}
    g_1\\
    g_2
    \end{bmatrix}\,
    .
\end{equation}
The derivatives in the Jacobian matrix are given in Appendix \ref{app:elastoDeriv}.

\subsection{Output}
In the following, the same parameters as presented for the static friction model in Section \ref{sec:staticImplementation} have been used for the string and the bow (where $f_N = f_\Btxt$).
Additional parameters used for the elasto-plastic friction model are given in paper \citeP[C].

Figure \ref{fig:hysteresis} shows that the implementation of the elasto-plastic friction model exhibits a hysteresis loop in the force versus velocity plane as desired from a dynamic friction model. The values around $v=0$ are due to sticking behaviour and the loop on the left is due to slipping behaviour.
Figure \ref{fig:elastoWaveform} shows the output of the implementation and follows the characteristic Helmholtz motion shown in Figure
\ref{fig:helmholtzTimedomain}. When compared to the output of the static friction model in Figure \ref{fig:helmholtzImplementation}, the output of the elasto-plastic implementation seems to be more `smooth' overall, which could be explained by the elasticity of the bristles.

\begin{figure}[ht]
    \centering
    \begin{tikzpicture}[->,node distance=3cm,
      thick,main node/.style={circle,draw}, scale=0.9, every node/.style={scale=0.9}]

      \node[anchor=south west,inner sep=0] (image) at (0,0) {\includegraphics[width=\textwidth]{figures/exciters/hysteresis.eps}};
      % \filldraw[black] (arr2B) (2,2) circle (1pt) node[anchor=center](topZ){};
      % \filldraw[black] (arr2E) (3,1.5) circle (1pt) node[anchor=center](topZ){};
      % (arr2B) edge[bend right] node [arrow] {} (arr2E);

    \node (arr1B) at (6.3, 4.35) {};
    \node (arr1E) at (8.2, 4.25) {};
    \node (arr2B) at (8.9, 1.2) {};
    \node (arr2E) at (6.7, 1.2) {};
    \node (arr3B) at (2.2, 2.05) {};
    \node (arr3E) at (2.2, 2.95) {};
    \path[every node/.style={font=\sffamily\small}]

      (arr1B) edge[bend left = 20] node [left] {} (arr1E)
      (arr2B) edge[bend left = 20] node [left] {} (arr2E)
      (arr3B) edge[bend left = 50] node [left] {} (arr3E);
      % \begin{scope}[x={(image.south east)},y={(image.north west)}]
      %     \draw[red,ultra thick,rounded corners] (0.62,0.65) rectangle (0.78,0.75);
      % \end{scope}
    \end{tikzpicture}
    \caption{Hysteresis loop showing 500 values up to $n=3\fs$. (Adapted from paper \citeP[C].) \label{fig:hysteresis}}
\end{figure}

\begin{figure}[h]
    \centering
    \includegraphics[width=\textwidth]{figures/exciters/elastoplasticWaveForm.eps}
    \caption{Time domain output at the bow location of a stiff string bowed using an elasto-plastic friction model. \label{fig:elastoWaveform}}
\end{figure}

\subsection{Stability through energy analysis}
As the elasto-plastic bow model is a differential equation in itself, its approximation will need to abide a stability condition as well.
As the system at hand is nonlinear, frequency domain analysis as described in Section \ref{sec:stabilityAnalysis} can not be performed. Energy analysis, on the other hand, can be used here to determine the necessary stability condition for this model. This section follows the concepts introduced in Section \ref{sec:stabilityAnalysisEnergy} to obtain a stability conditions for the elasto-plastic friction model. A similar process for finding stability for the LuGre model has been done by Olsson in continuous time \cite[p. 55]{Olsson1996}. The derivation below is inspired by his.

First, all terms of Eq. \eqref{eq:elastoStiffStringFDS} are multiplied by $\rho A$ to get the appropriate units for the analysis. Then, the inner product with $\dtd \uln$ over the string domain $\D$ is taken to get
\begin{equation}
    \dtp \h_\text{s} + \q_\text{s} = -\p_\Btxt
\end{equation}
where the definitions for the discrete Hamiltonian $\h_\text{s}$ and the damping term $\q_\text{s}$ for the string can be found in Section \ref{sec:energyAnalysisString}. The input power introduced by the bow is defined as (writing $f(v^n,z^n) = f^n$)
\begin{equation*}
    \p_\Btxt = \langle (\dtd \uln), J(x_\Btxt^n)f^n\rangle_\D
\end{equation*}
which, using identity \eqref{eq:identityIJ} can be written as
\begin{equation*}
    \p_\Btxt = I_l(x_\text{B}^n)\dtd \uln f^n.
\end{equation*}
Finally, using Eq. \eqref{eq:vrelElastoDisc} yields
\begin{equation}
    \p_\Btxt = f^nv^n + f^n v_\Btxt^n.
\end{equation}
The term $f^nv^n$ is the important one as $f^nv_\Btxt^n$ is a driving term, and is zero when the external bow velocity is zero. This means that this does not affect the internal stability of the system. In the following, the superscript $n$ is suppressed for brevity.

Substituting Eq. \eqref{eq:discForceFunction} into $fv$ and ignoring the noise term $s_3w^n$ for now, yields
\begin{equation}\label{eq:fv}
    \p_\Btxt = fv = \sigma_{0}zv +\sigma_{1}rv+\sigma_{2}v^2.
\end{equation}
The definition for $r^n$ in Eq. \eqref{eq:r} may be rewritten as
\begin{equation*}
    \begin{aligned}
        r &= v\bigg[1-\alpha\frac{z}{z_\text{ss}(v)}\bigg],\\
        r &= v-\frac{v\alpha z}{z_\text{ss}(v)},\\
        v &= r + \frac{v\alpha z}{z_\text{ss}(v)},
    \end{aligned}
\end{equation*}
and (following Olsson) may be substituted in Eq. \eqref{eq:fv} as
\begin{equation}
    \p_\Btxt = s_{0}z\left(r+ \frac{v\alpha z}{z_\text{ss}(v)}\right)+s_{1}r\left(r+ \frac{v\alpha z}{z_\text{ss}(v)}\right)+s_{2}v^2,
\end{equation}
or
\begin{equation}
    \p_\Btxt = s_0zr+s_1\left(r+\frac{v\alpha z}{2z_\text{ss}(v)}\right)^2+\frac{v\alpha z^2}{z_\text{ss}(v)}\left(s_0-\frac{s_1v\alpha}{4z_\text{ss}(v)}\right)+s_2v^2.
\end{equation}
The power introduced by the bow can then be subdivided into the total energy in the bristles and their damping. As $r$ approximates $\dot z$ the first term, one can rewrite this to
\begin{equation*}
    \dtp \h_\text{brist} + \q_\text{brist} \geq 0,
\end{equation*}
and needs to be non-negative for passivity.

Starting with the damping term, which is defined as
\begin{equation}
    \q_\text{brist} = s_1\left(r+\frac{v\alpha z}{2z_\text{ss}(v)}\right)^2+\frac{v\alpha z^2}{z_\text{ss}(v)}\left(s_0-\frac{s_1v\alpha}{4z_\text{ss}(v)}\right)+s_2v^2.
\end{equation}
one can show that, as all coefficients are non-negative and $\sgn(v) = \sgn(z_\text{ss}(v))$ (through a multiplication by $\sgn(v)$ in the definition of in Eq. \eqref{eq:zss}), $\q_\text{brist}$ is non-negative under the following condition:
\begin{equation}
    s_1 \leq \frac{4s_0z_\text{ss}(v)}{v}\ .
\end{equation}
This means that as long as one knows the limit of the velocity of the system, the coefficient $s_1$ can be set accordingly.

Using identity \eqref{eq:prodIdentity2}, the energy stored in the bristles can be shown to be
\begin{equation}
    \h_\text{brist} = \frac{s_0}{2}z^ne_{t-}z^n,
\end{equation}
which is not necessarily non-negative. In \cite{Olsson1996}, Olsson performs the analysis in continuous time, where the energy in the bristles is defined as to
\begin{equation}
    \mathfrak{H}_\text{brist} = \frac{s_0}{2} z^2,
\end{equation}
which is clearly non-negative. See Notes below for a discussion.

% A condition can, however, be retrieved from the damping term and is similar to the condition presented by Olsson in continuous time. The damping term is defined as

\subsubsection{\SWcomment[NOTES FOR STEFAN]}
As far as I see, to find a stability condition for the model, there are two options:
\begin{enumerate}
    \item The $s_0z$ term should be discretised using the centred averaging operator, such that the system is inherently stable.
    \begin{equation}
        s_0 (\mtd z^n)(\dtd z^n)\quad \overset{\text{Eq. \eqref{eq:prodIdentity4}}}{\Longrightarrow} \quad \frac{s_0}{2}\mtm(z^n)^2
    \end{equation}
    This introduces the issue of needing to deal with $z^{n+1}$ issue in the multivariate NR solve..
    \item If the bristles had some sort of mass, one could create a stability condition such as done for the mass-spring system in Section \ref{sec:stabilityAnalysisEnergy}. This requires a change the model.
    \item Or do you see something I don't...?
\end{enumerate}

\section{Discussion and conclusion}
This chapter presented the bow as a mechanism to excite stiff strings. Two friction models have been presented: a static friction model where the friction force is only a function of the relative velocity between the string and the bow, and a dynamic elasto-plastic friction model, which relates this relative velocity to the friction force using a differential equation. The latter has been presented in paper \citeP[C] where it was first applied to stiff strings based on FDTD methods.

Although a successful implementation of the elasto-plastic friction model has been made, it has not been used in the project beyond paper \citeP[C]. It was found that small changes in parameters, both control and model parameters, already yield large behavioural changes. An attempt was made at using the elasto-plastic friction model with a fully modelled instrument (the tromba marina presented in papers \citeP[D] and \citeP[E]), but did not yield the desired results, and the predictable static friction model was chosen instead. Future work includes tuning the many parameters that the elasto-plastic model relies on and apply an adjusted version to fully modelled instruments.

An in-depth comparison between the static and the elasto-plastic friction model in terms of perceptual differences, is also left for future work. A preliminary comparison has been carried out by Onofrei in \cite{Onofrei2021}, where the author noted that \textit{"the
elasto-plastic model seems to behave more smoothly"} than the static friction model. This is also what is observed when comparing the output of the friction models in Figures \ref{fig:helmholtzImplementation} and \ref{fig:elastoWaveform}. As said, these observations are preliminary, and further work needs to be done to compare the various friction models.

Finally, it must be noted that the applications of the bow are not limited to strings, and can very well be extended to other resonators. In [\hyperref[ch:listOfPublications]{S4}], for example, the authors apply the elasto-plastic friction model to a 2D drum membrane to simulate a friction drum inspired instrument.


\documentclass[12pt, a4paper]{scrartcl}
\usepackage[utf8]{inputenc}
\usepackage{graphicx}
\usepackage{amsmath, amsthm, amssymb, textcomp}
\usepackage{setspace}
\usepackage{paralist}
\usepackage{graphicx}
\usepackage{caption}
\graphicspath{{WSK_im/}} %Graphic is in a folder named WSK_im in the currend directory
\usepackage{float}
\usepackage{authblk}
\renewcommand\Authfont{\fontsize{12}{14.4}\selectfont}
\title{Bayesian probability theory - Lesson 7:\\
Parameter and model estimation and classification}

\author{Wolfgang von der Linden, Gerhard Dorn, Johanna Moser}
\date{Transcript}

\begin{document}
\setlength{\parindent}{0pt}
\maketitle
\onehalfspacing

Welcome to unit 7 of the course on Bayesian probability theory. My
name is Wolfgang von der Linden and I will enable you to help Captain
Bayes and her crew to fit data by \textbf{model functions} and to learn the basics of
\textbf{parameter estimation, prediction} and \textbf{classification}
of data as well as on \textbf{model selection}.
\begin{itemize}\item We will learn how to apply Bayes theorem to \textit{derive} the probability
distribution for the parameters of an assumed model,
\item We will learn how to apply this probability distribution in the \textit{interpolation}
of data to help Captain Venn find his treasure
\item We will learn how to \textit{classify} sea areas to be a nature reserve for
frogfish or a general fishing ground.
\end{itemize}

We begin with the classical \textbf{regression problem}. It is described as follows:\\
Let there be a \textit{set of measured data $d_{\nu}$} that depend on some \textbf{pivot points $x_{\nu}$}%7_1
\begin{figure}[H]
	\centering
	\includegraphics[width=0.75\textwidth]{7_1.png}
\end{figure}
These data points can be represented in an $x$-$y$ diagram.
The measurements are in general not exact but have some uncertainty that obey a certain distribution.
In most cases like in the situation of Captain Venn these
are additive \textbf{Gaussian uncertainties}.
Next we assume that without the experimental uncertainty there is a \textit{relation}
between intrinsic data and the pivot points that can be described by a
\textit{deterministic} function, for instance a physical law that depends on some parameters.\\
\[y_{\nu} = f(x_{\nu},\vec{a})\]

\textit{Then the measured data can be described by adding the measurement uncertainty to the deterministic
model function.}\[d_{\nu}=f(x_{\nu},\vec{a})+\eta_{\nu}\]

Now the goals of regression are twofold:\\

Firstly \textbf{Parameter estimation}: We would like to investigate the relation between
measurement values $d_{\nu}$ and pivot points $x_{\nu}$ and therefore \textit{determine
the model and its parameters $\vec{a}$, their uncertainty $\sigma_{\vec{a}}$ and covariance $\text{CoV}(a_i,a_j)$}.
In the case of Captain Venn and his starpath problem the goal is to find
the laws of how the stars or rather the earth move. The model could
include the rotation of the earth around a tilted axis with respect to
the elliptic orbit around the sun. The parameters could represent the
duration of such a rotation and the orbital period.\\

The second aim of regression is to \textbf{predict the values $y^*$} of the function at
additional pivot points $x^*$ and their corresponding uncertainty. These
additional pivot points could be such, that it was not possible to realize
a measurement at these points. (Remember the case of Captain Venn
who was not able to determine the angles of the star Arcturus in the
shortest night of the year due to bad weather.)\\

Another important application is to \textbf{extrapolate} physical properties from a small scale
experiment to a larger one as is for instance done in fusion plasma reactors.\\

\section*{Parameter estimation}
Now we come to parameter estimation.
For your orientation we recapitulate the \textit{classical regression procedure} before
we introduce the probabilistic approach.
As described before we assume we have a suitable model $y_{\nu}=f(x_{\nu},\vec{a})$, which is for instance given by physical
laws or is obvious due to a visible trend of the data.
A very prominent choice are \textbf{linear models} in which
one uses a linear combination of suitable ansatz functions $\phi_i (x)$ with
the coefficients being the parameters.
\[f(x,\vec{a})=\sum_i \phi_i(x)a_i\]

Based on the model function, a \textbf{cost function $\chi^2_{\vec{a}}$} is introduced that \textit{quantifies deviation between
model function and data}. A very common cost
function is the \textbf{root mean square function}.\\
\begin{equation*}\boxed{\chi^2_{\vec{a}}=\sum_{\nu=1}^N\frac{d_{\nu}-f(x_{\nu},\vec{a})^2}{\sigma_{\nu}^2}
}\end{equation*}\\
 Sometimes the individual contributions get a weight $\sigma_{\nu}^2$
according to the reliability of the measured data.
Outliers thus have a very strong impact on the cost function.\\

In the next step the cost function is \textit{minimized} with respect to the model parameters. This
is the so-called \textbf{least squares approach}. In the case of linear regression in combination
with a quadratic cost function the
minimization leads to a matrix equation where the columns of the matrix $M$ are given by the ansatz
functions evaluated at the pivot points.\\%7_2
\begin{figure}[H]
	\centering
	\includegraphics[width=0.75\textwidth]{7_2.png}
\end{figure}

If one is not entirely sure about the model function in a final step, one can evaluate the model function and
discuss whether is has a reasonable behaviour.
This would lead to the field of \textbf{hypothesis testing}.\\

Now we come to \textbf{Bayesian parameter estimation}.
We have already mentioned several times that probability theory represents the consistent
calculus for dealing with uncertainty. The general approach is the following:\\

First we have to \textit{fix all assumptions} like ``the measurement are given for certain pivot points'',
``the data can be described by an assumed model'', ``the experimental noise follows a given distribution''
and so on. Then we have to \textit{specify the questions} we are interested in, like: ``What
are the values of the parameters?'' ``What would be the value of the measured quantity at a new pivot point?''
``Is the assumed model the right one?'' ``What if there are \textbf{outliers}, which means there are individual data points that are \textit{corrupted
by external influences and do not obey the assumed error statistics}?''\\

Finally, we compute the corresponding probabilities.
We will see at the end of this section when and under which conditions the previously presented statistical procedure is justified. \\

\section*{Probability density of the parameters}
The \textbf{probability density of the parameters} $\vec{a}$ in the light of the measured data $\boldsymbol{D}$ is the \textbf{posterior probability} that can
be simplified by Bayes theorem.
\begin{equation*}\boxed{p(\vec{a}|\boldsymbol{D},X)=\frac 1Z p(\boldsymbol{D}|\vec{a},X)p(\vec{a})
}\end{equation*}\\
 The normalization $Z$ is independent of the parameters and can be determined in the end.
The prior probability $p(\vec{a})$ encodes our \textit{knowledge about the parameters before we take into account the experimental data}.
If nothing is known at all, the most ignorant prior is the \textbf{uniform prior}; maybe within
certain parameter ranges. We will discuss in the next unit how ignorant
priors can be determined purely based on considerations of symmetries and
invariances.\\
A crucial element in Bayes theorem is the \textbf{likelihood} $p(\boldsymbol{D}|\vec{a},X)$.
If the measurement errors are uncorrelated the likelihood factorizes.
\[ p(\boldsymbol{D}|\vec{a},X)=\prod_{\nu} p(\boldsymbol{d}_{\nu}|\vec{a},x_{\nu})\]
In the case of additive Gaussian noise with noise level $\sigma_v$ we can express the
likelihood of one data point given by the model function $f$ and the parameters $\vec{a}$.%7_3
\begin{figure}[H]
	\centering
	\includegraphics[width=0.75\textwidth]{7_3.png}
\end{figure}
Since we are interested in the probability density of the parameters $\vec{a}$,%%
we only need the terms that depend on them. We can replace the product
of the exponential by the exponential of the sum.
\begin{equation*}\boxed{ p(\boldsymbol{D}|\vec{a},X)\propto \exp\left[-\sum_{\nu}\frac{\boldsymbol{d}_{\nu}-f(x_{\nu},\vec{a})^2}{2\sigma_{\nu}^2}\right]=\exp\left[-\frac 12 \chi_{\vec{a}}^2\right]}\end{equation*}\\

This misfit term $\chi_{\vec{a}}^2$ corresponds to the \textbf{least squares cost function} which we
saw in the classical regression procedure.
If there are uncertainties $\sigma_{x,\nu}, \sigma_{y,\nu}$ in both coordinates and if the regression
function is linear in $x$ $f(x,\vec{a})=a_1x+a_2$ then the misfit is given by the following expression:
\[\chi_{\vec{a}}^2=\sum_{\nu}\frac{(\boldsymbol{d}_{\nu}-a_2-a_1x_{\nu})^2}{\sigma_{y,\nu}^2+a_1^2\sigma_{x,\nu}^2}\]

In summary we find the following probability density function for the parameters:
%7_4
\begin{figure}[H]
	\centering
	\includegraphics[width=0.75\textwidth]{7_4.png}
\end{figure}
More on the difference between pdfs and probability distributions will follow in the next unit.\\

An obvious way to decide on the parameters is to take the \textit{maximum of the posterior probability density} - the so-
called \textbf{MAP solution}. In the case of a flat prior this solution is
equal to the maximum likelihood. The\textbf{ Maximum Likelihood (ML)} choice of
the parameters is the basis of parameter estimation in conventional statistics.
If in addition the likelihood is \textit{Gaussian}, as in the present case, then the
MAP and ML solution is obtained by \textit{minimizing} $\chi_{\vec{a}}^2$, which is the
\textbf{Least-Squares} method that we discussed before.\\%7_5
\begin{figure}[H]
	\centering
	\includegraphics[width=0.75\textwidth]{7_5.png}
\end{figure}
\textit{This shows that the least squares approach is \textit{only applicable} in the case of Gaussian
experimental noise and if we have no (or if we ignore) prior knowledge about the parameters.} \\

In general the MAP and hence ML solution is not very robust and highly
susceptible to noise. In particular if the likelihood is not \textbf{unimodal}, which means it has \textit{several peaks},
the maximum can be misleading if it does not carry the \textit{dominant probability mass}.
A more reliable characterization of the parameters is given by the \textbf{posterior
mean}.
Here we finally need the normalization, which can be obtained by the following integral.%7_6
\begin{figure}[H]
	\centering
	\includegraphics[width=0.75\textwidth]{7_6.png}
\end{figure}

It cannot be overstressed that a single value for the parameters, be it given
by the MAP or the mean, is \textit{meaningless, if we do not know the \textbf{uncertainty}}.
The uncertainty can be quantified by the \textbf{variance}.
In many cases also the \textbf{covariance} is of great interest.%7_7
\begin{figure}[H]
	\centering
	\includegraphics[width=0.75\textwidth]{7_7.png}
\end{figure}

It tells us whether uncertainties are correlated. For instance it could be that
only a combination $a_1 + a_2$ is well defined by the data, while the individual
values are not. This can be seen in the covariance.\\

\section*{Predicting values}
Now we turn to the predictive aspect of regression. We are interested in the
function value $y^*$ at a new position $x^*$, given the data $\boldsymbol{D}$. If the pivot point
$x^*$ lies \textit{within the set of given pivot points}, we call this type of prediction an
\textbf{interpolation}, otherwise \textbf{extrapolation}. The naive and inconsistent approach
would be to replace the unknown parameters by the estimated obtained before
and just insert $x^*$ in the model function to obtain the function value
at the desired position. \\%7_8
\begin{figure}[H]
	\centering
	\includegraphics[width=0.75\textwidth]{7_8.png}
\end{figure}
The correct approach, however, is to \textit{determine the probability for the function
value $y^*$ given the data, so $P(y^*|\boldsymbol{D},X)$}.
We can compute $p(y^*|\vec{a},\boldsymbol{D},X)=\delta(y^*-f(x^*,\vec{a}))$ if we know the parameters. So we introduce
them via the marginalization rule.%
\begin{equation*}\boxed{p(y^*|\boldsymbol{D},X)=\int \text{d}V_{\vec{a}}p(y^*|\vec{a},\boldsymbol{D},X) \cdot p(\vec{a}|\boldsymbol{D},X)
}\end{equation*}\\
 Then the moments of $y^*$ are
 \begin{equation*}\boxed{\langle (y^*)^n\rangle=\int \text{d}V_{\vec{a}}(f(x^*,\vec{a}))^n \cdot p(\vec{a}|\boldsymbol{D},X)
}\end{equation*}\\
The result becomes particularly simple if the model function depends
linearly of the parameters, so $f(x,\vec{a})=M(x)\vec{a}$, where $M(x)$ is a row-vector containing the ansatz functions $M=(\Phi_1(x),\Phi_2(x),...)$
evaluated at $x$. Then the mean of the regression value $y^*$ is equal to the \textit{linear combination
of the ansatz functions with the mean of the parameters}.
\[\langle y^*\rangle = M(x^*)\langle \vec{a}\rangle\]
So in the linear case, this is in agreement with the naive approach, with the
parameters replaced by the posterior mean.
For the variance of the regression value we obtain
\[\langle (\Delta y^*)^2\rangle = M(x^*)CM^T(x^*)\]
The square root of the variance quantifies the uncertainty of the regression
value.\\%

Captain Venn wanted to know the azimutal or horizontal angles of the
star Arcturus at midnight on the shortest night of the year. Since the whether
was cloudy on that day of observation he wanted to infer the treasure's position
by interpolating his measurement values.\\
Captain Venn knows that the accuracy of his measurements is limited by
his time measurement which has an accuracy of about one minute.
The uncertainty of his angle measurements is given by the \textit{sum of
the deviations due to the time shifts plus the accuracy of his sextant and compass}.
From test measurements he knows the approximate deviations of the angles due to the time inaccuracy.
We assume that the deviations of the angles are normal distributed with the derived $\sigma$ values just mentioned.%7_9
\begin{figure}[H]
	\centering
	\includegraphics[width=0.75\textwidth]{7_9.png}
\end{figure}
Next we assume a linear model for which we can apply the formulas we just derived%7_10
\begin{figure}[H]
	\centering
	\includegraphics[width=0.75\textwidth]{7_10.png}
\end{figure}
formulas. Following the Bayesian regression steps we end up with a joint
probability density for the linear parameter and we can use them to derive a prediction for the desired angles.%7_11
\begin{figure}[H]
	\centering
	\includegraphics[width=0.75\textwidth]{7_11.png}
\end{figure}

\textit{Have a look at the interactive Pluto notebooks to get familiar with other
model functions and the numerical tricks to evaluate the integrals over the
whole parameter space! Here you can also explore the benefits of using a better
astronomic model to predict star orbits and check how well extrapolation
works within several months.}\\

\section*{Classification}
Now we turn to the classification problem.
Given the \textit{classification labels} for the two species of fish and their \textit{positions}, the goal
is to \textit{find the border between the fish reserves}, which shall be described by
a linear function.\\%7_12
\begin{figure}[H]
	\centering
	\includegraphics[width=0.75\textwidth]{7_12.png}
\end{figure}

The function depends on some parameters $\vec{a}$.
\[y_{border}=f(x,\vec{a})=M(x)\vec{a}\]
Here $M_{ij}(x)=\phi_j(x_i)$ defines the model, or rather
the \textit{set of ansatz functions that are linearly combined weighted by the parameters}. For instance we can have a linear model or a quadratic model. We can also construct an implicit function. \[f(x,y,\vec{a})=M(x,y)\vec{a}=0\]
When we insert the position $(x_i|y_i)$ of the fish, the implicit function yields 0 if the fish resides exactly on the boundary. Otherwise the sign of the function tells us on which side of the boundary the fish is located.
In order to derive the probability density for the model $p(\vec{a}|\boldsymbol{X},M,\boldsymbol{C})$
parameters given positions $\boldsymbol{X}$ and labels $\boldsymbol{C}$ of all fish and the model $M$, we proceed in a similar manner as in the regression case. Bayes’ theorem yields a proportionality to the product of likelihood and prior.\[p(\vec{a}|\boldsymbol{X},M,\boldsymbol{C})\propto p(\boldsymbol{X}|\vec{a},M,\boldsymbol{C})p(\vec{a}|M)\]

The classification label $\boldsymbol{C}$ has been omitted in the conditional part of the prior as it has no implications.
Moreover we assume that the likelihood terms are \textit{uncorrelated} resulting in a product form. \[p(\boldsymbol{X}|\vec{a},M,\boldsymbol{C})=\prod_{\nu}p(\boldsymbol{x}_{\nu}|\vec{a},M,\boldsymbol{C}_{\nu}\]
The individual likelihood factors are the \textit{probability to find a fish of a certain species at a specific position}.
Here, the model and the corresponding parameters are important because they define the boundary of the fish reserves.
The likelihood is a \textit{property of the fish} and is related to why and how they stick to their reserve.
The likelihood could in principle be quite different for the classification of other objects.
Nevertheless, the following considerations show how such classification problems are treated in
the framework of Bayesian probability theory in general.\\
As mentioned before, the sign of the implicit function $f(x_{\nu},y_{\nu},\vec{a})$ tells us on which side of the boundary the fish is located. \\

In the case of a linear implicit model function, the value of $f(x,y,\vec{a})=xa_1+ya_2+1$ can be used to easily determine the \textit{shortest distance to the boundary}. Just think of the Hessian normal form.%7_13
\begin{figure}[H]
	\centering
	\includegraphics[width=0.75\textwidth]{7_13.png}
\end{figure}
%Generally for a fish at position $x_v$, we define the shortest distance to the border by the following expression.\\%

The distance is defined as positive if the fish is on the correct site of the boundary and negative otherwise.
In order to obtain a probability that is constant for a fish in its own reserve and that decreases continuously to zero when crossing the border we choose the logistic function.
The likelihood then reads%7_14
\begin{figure}[H]
	\centering
	\includegraphics[width=0.75\textwidth]{7_14.png}
\end{figure}
where $\frac {1}{\beta}$ defines the width of the ``No Fish’s Land'' in which the probability
drops continuously to zero.
Its area is assumed to be \textit{very small} and the logistic function is almost a step function.
Hence the meaning of the \textbf{normalization} $Z$ is the \textit{size of the area of the two reserves}. For simplicity, we assume that the dependence of these areas on the parameters $\vec{a}$ and on the fish label $C_i$ can be ignored.
Then $Z$ is an unimportant constant.\\
It happens that not all fish abide by the rules. $\epsilon$ is the proportion of such fish and is related to the \textbf{slack variable}
in optimization problems.
$\epsilon$ makes the classification \textbf{outlier tolerant}.
So by now we have all terms of the probability for the model parameters.
From that we can determine the mean values of the parameters, as well as their uncertainty. %7_15
\begin{figure}[H]
	\centering
	\includegraphics[width=0.75\textwidth]{7_15.png}
\end{figure}

Finally we can determine the mean and the variance of the boundary.
\[\langle y\rangle =M(x)\langle \vec{a}\rangle \qquad C_y=\langle \Delta y \Delta y^T\rangle = M(x)C_{\vec{a}}M(x)^T\]

So let’s apply the Bayesian logistic classification approach to find the probability for the natural
boundary of the fish reserve. Using the data Bernoulli provided leads
to a probability distribution for the two parameters that can be visualized
by semitransparent borderlines.%7_16
\begin{figure}[H]
	\centering
	\includegraphics[width=0.75\textwidth]{7_16.png}
\end{figure}
 It can be seen from the formula that the likelihood function is maximized if the number of outliers is minimized. \\%7_17
 \begin{figure}[H]
 	\centering
	\includegraphics[width=0.75\textwidth]{7_17.png}
\end{figure}

\textit{See also the interactive pluto notebook to play with the slack variable $\epsilon$ and
other data sets.}\\

\section*{Model selection}
Another important question is the \textbf{model selection}, which answers the question ``\textit{What is the
probability that the correct form of the border is given by the model b, if we
know the position of all fish and their labels?}'' Here, Bayes theorem gives the following expression
\[P(M|\boldsymbol{X},\boldsymbol{C})=\frac{1}{Z'}p(\boldsymbol{X}|M,\boldsymbol{C}) \cdot P(M,\boldsymbol{C})\]
where the normalization can easily be computed in the end.
Now we use the marginalization rule to introduce the model parameters.
As long as we do not know better, we assume a uniform prior within certain
parameter ranges.\\%7_18
\begin{figure}[H]
	\centering
	\includegraphics[width=0.75\textwidth]{7_18.png}
\end{figure}

Here $V_M$ is the prior volume that depends on the model.
Then the probability for model M in the light of the data is given by the following expression. %7_19
\begin{figure}[H]
	\centering
	\includegraphics[width=0.75\textwidth]{7_19.png}
\end{figure}
Here $\Omega_M$ is the \textit{volume in parameter space} allowed by the model.
It should be noted that \textbf{Ockham's razor} is an integral part of the Bayesian expression
and it \textit{penalizes unnecessarily complex models} \textit{(The more complicated a model is, the larger is its parameter volume!)}.\\

This concludes unit 7. We have learned Bayesian regression and how to
apply it to parameter estimation and model selection.
We learned also how to predict unseen measurement values and their probabilities and we got insights into classification of data.
Now it’s your turn to actually find Captain Venn’s treasure in the pluto
notebooks where you can also work on the boundary issue of Claire and
Makabe.
Please feel free to ask questions in the forum and feel encouraged to test your
knowledge in the quiz!

\vspace{2cm}
\begin{minipage}[t]{1\textwidth}
	\raggedleft
	\centering
	\includegraphics[width = 0.20\textwidth]{CC-BY_icon}
	\vspace{0.2cm}

	\centering
	{\large ITPCP, TU Graz} \\
	https://creativecommons.org/licenses/by/4.0/legalcode
\end{minipage}
\end{document}


\SetPicSubDir{ch-Background}

\chapter{Background}
\label{ch:background}
\vspace{2em}
This section provides the background required to understand this work. Readers who already have an understanding of Deep Reinforcement Learning and Deep Deterministic Policy Gradient (DDPG) may skip this section.

We would like to note that a large portion of this section has been reproduced from OpenAI's Spinning Up in Deep RL~\cite{SpinningUp2018}. What is different in this reproduction, however, is that the background information on DRL has been adapted to fit the ideas and concepts related to map-less navigation found in this work.

\section{Deep Reinforcement Learning}
Deep reinforcement learning (DRL) is the combination of reinforcement learning and deep learning. This idea of using neural networks for reinforcement learning, however, is not new and can be dated all the way back to Teasauro's TD-Gammon~\cite{tesauro_temporal_nodate}. In the early 2010s, however, the field of deep learning began to find groundbreaking success, particularly in speech recognition~\cite{dahl_context-dependent_2012} and computer vision~\cite{krizhevsky_imagenet_2017}. This success, combined with the advances in computing power, allowed the revival in interest of using deep neural networks as universal function approximators for reinforcement learning - leading to deep reinforcement learning.

Deep reinforcement learning is useful when~\cite{SpinningUp2018}:
\begin{itemize}
\item we have a sequential decision-making problem (which we can represent as a Markov Decision Process)
\item we do not know the optimal behaviour (e.g. multi-modal problem)
\item but we can still evaluate whether behaviours are good or bad
\end{itemize}

\subsection{Markov Decision Process}
To train our agent with deep reinforcement learning, we must be able to model the relationship between the agent and its environment. A Markov Decision Process (MDP) is a mathematical object that describes our agent interacting with a stochastic environment. It is defined by the following components:
\begin{itemize}
\item $S$: \textbf{state space}, a set of states of the environment. This is the set of inputs for our robot which contain a view of the world (i.e. a stack of laser scans, the current speed of the robot, and the target position with respect to the robot's local frame)
\item $A$: \textbf{action space}, a set of actions, which the agent selects from each timestep. The actions taken by our robot to reach its target are its linear velocity and angular velocity, both of which are continuous-valued variables
\item $P(r, s' | s, a)$: \textbf{transition probability distribution}. For each state s and action $a$, $P$ specifies the probability that the environment will emit reward $r$ and transition to state $s'$. This transition function describes the relationship between states, actions, next states, and rewards in an environment.
\end{itemize}

\subsection{Policies}
The end goal is to find a policy $\pi$, which tells the agent what actions to take given a state. In DRL, we use parameterized policies: policies whose outputs are computable functions that depend on a set of parameters (e.g. the weights and biases of a neural network) which we can adjust to change the behavior via some optimization algorithm~\cite{SpinningUp2018}.

We denote the parameters of such a policy by $\theta$, and then write this as a subscript on the policy symbol to highlight the connection:

\begin{equation}
a_{t} \sim \pi_{\theta}(\cdot | s_{t})
\end{equation}

\subsection{Reward and Return}
\label{background:subsec:reward}
The reward function $R$ is critically important in reinforcement learning (and specifically for this work). It depends on the current state of the world, the action just taken, and the next state of the world:

\begin{equation}
r_{t} = R(s_{t}, a_{t}, s_{t+1})
\end{equation}

The goal of an agent is to maximise the cumulative reward over a trajectory. In this work, the type of return used is called an \textbf{infinite-horizon discounted return}, which is the sum of all rewards ever obtained by the agent, but discounted by how far off in the future they're obtained. This formulation of reward includes a discount factor $\gamma \in (0, 1)$

Specifically (prior to adding the reward feature proposed in this work), this work uses the reward function for map-less navigation proposed in Xie et. al's AsDDPG network~\cite{xie_learning_2018}:
\begin{equation*}
Reward, r_{t} =
\begin{cases}
  R_{crash}& \text{if robot crashes},\\
  R_{reach}& \text{if robot reaches the goal},\\
  \gamma{((d_{t-1} - d_{t})\Delta{t} - C)}& \text{otherwise}.\\
\end{cases}
\end{equation*}

\subsection{The RL Problem}
The goal of reinforcement learning is to select a policy which maximises \textbf{expected return} when the agent acts according to it. To talk about expected return, we first have to talk about probability distributions over trajectories.

Let’s suppose that both the environment transitions and the policy are stochastic. In this case, the probability of a $T$-step trajectory is:
\begin{equation}
P(\tau|\pi) = p_{0}(s_{0}) \prod_{t=0}^{T-1} P(s_{t-1}|s_{t}, a_{t}) \pi(a_{t}|s_{t})
\end{equation}

The expected return, denoted by $J(\pi)$, is then:
\begin{equation}
J(\pi) = \int_{\tau} P(\tau|\pi) R(\tau) = \tau \sim \pi R(\tau)
\end{equation}

The central optimization problem in RL can then be expressed by:
\begin{equation}
\pi^{*} = arg\underset{\pi}{max} J(\pi)
\end{equation}

with $\pi$ being the optimal policy.

\subsection{Value Functions}
It's often useful to know the \textbf{value} of a state, or state-action pair. By value, we mean the expected return if you start in
that state or state-action pair, and then act according to a particular policy forever after. Value functions are used, one way or another, in almost every RL algorithm.

There are four main functions of note:
\begin{enumerate}
\item The \textbf{On-Policy Value Function}, $V^{\pi}(s)$, which gives the expected return if you start in a state $s$ and always act according to policy $\pi$:
	\begin{equation}
	V^{\pi}(s) = \tau \sim \pi R(\tau)|s_{0} = s
	\end{equation}
\item The \textbf{On-Policy Action-Value Function}, $Q^{\pi}(s, a)$, which gives the gives the expected return if you start in state $s$, take an arbitrary action $a$ (which may not have come from the policy), and then forever after act according to policy
$\pi$:
	\begin{equation}
	Q^{\pi}(s, a) = \tau \sim \pi R(\tau)|s_{0} = s, a_{0} = a
	\end{equation}
\item The \textbf{Optimal Value Function}, $V^{*}(s)$, which gives the expected return if you start in state $s$ and always act according to the \textit{optimal} policy in the environment
	\begin{equation}
	V^{*}(s) = \underset{\pi}{max} \tau \sim \pi R(\tau)|s_{0} = s
	\end{equation}
\item The \textbf{Optimal Action-Value Function}, $Q^{*}(s, a)$, which gives the expected return if you start in state $s$, take an arbitrary action $a$, and then forever after act according to the \textit{optimal} policy in the environment:
	\begin{equation}
	Q^{*}(s, a) = \underset{\pi}{max} \tau \sim \pi R(\tau)|s_{0} = s, a_{0} = a
	\end{equation}
\end{enumerate}

\subsection{The Optimal Q-Function and the Optimal Action}
There is an important connection between the optimal action-value function $Q^{*}(s, a)$ and the action selected by the optimal policy. By definition, $Q^{*}(s, a)$ gives the expected return for starting in state $s$, taking (arbitrary) action $a$, and then acting according to the optimal policy forever after.

The optimal policy in $s$ will select whichever action maximises the expected return from starting in $s$. As a result, if we have $Q^{*}$, we can directly obtain the optimal action, $a^{*}(s)$, via:
\begin{equation}
	a^{*}(s) = \underset{a}{\argmax} Q^{*}(s, a)
\end{equation}

This connection is important; some RL algorithms learn by improving their policy (thereby directly obtaining "good" actions), whereas others learn by improving their Q-Function (which allows them to determine the best action from a set of actions, indirectly obtaining "good" actions).

\subsection{Bellman Equations}
All four of the value functions obey special self-consistency equations called \textbf{Bellman equations}. The basic idea behind the Bellman equations is this:
\begin{center}
The value of your starting point is the reward you expect to get from being there, plus the value of wherever you land next.
\end{center}

The Bellman equations for the on-policy value functions are:
\begin{equation}
	V^{\pi}(s) = \underset{\underset{s' \sim P}{a \sim \pi}}{E} [r(s, a) + \gamma V^{\pi}(s')]
	Q^{\pi}(s, a) = \underset{s' \sim P}{E} [r(s, a) + \gamma \underset{a' \sim \pi}{E}[Q^{\pi}(s', a')]]
\end{equation}

where $s' \sim P$ is shorthand for $s' \sim P(\cdot|s, a)$, indicating that the next state $s'$ is sampled from the environment's transition rules; $a \sim \pi$ is shorthand for $a \sim \pi(\cdot|s)$; and $a' \sim \pi$ is shorthand for $a' \sim \pi(\cdot|s')$.

The Bellman equations for the optimal value functions are:
\begin{equation}
	V^{*}(s) = \underset{a}{max} \underset{s' \sim P}{E} [r(s, a) + \gamma V^{*}(s')]
	Q^{*}(s, a) = \underset{s' \sim P}{E} [r(s, a) + \gamma \underset{a'}{max}[Q^{*}(s', a')]]
\end{equation}

The crucial difference between the Bellman equations for the on-policy value functions and the optimal value functions, is the absence or presence of the $max$ over actions. Its inclusion reflects the fact that whenever the agent gets to choose its action, in order to act optimally, it has to pick whichever action leads to the highest value.

\subsection{Advantage Functions}
Sometimes in RL, we don't need to describe how good an action is in an absolute sense, but only how much better it is than others on average. That is to say, we want to know the relative \textbf{advantage} of that action. We make this concept precise with the \textbf{advantage function}.

The advantage function $A^{\pi}(s, a)$ corresponding to a policy $\pi$ describes how much better it is to take a specific action $a$ in state $s$, over randomly selecting an action according to $\pi(\cdot|s)$, assuming you act according to $\pi$ forever after. Mathematically, the advantage function is defined by
\begin{equation}
A^{\pi}(s, a) = Q^{\pi}(s, a) - V^{\pi}(s)
\end{equation}

\section{Deep Deterministic Policy Gradient (DDPG)}
The network presented in this work is a variant of DDPG~\cite{xie_learning_2018}. As such, it is crucial that we also provide a background for what DDPG is, and how it works.

Deep Deterministic Policy Gradient (DDPG) is an algorithm which concurrently learns a Q-function and a policy. It uses off-policy data and the Bellman equation to learn the Q-function, and uses the Q-function to learn the policy.

This approach is closely connected to Q-learning, and is motivated the same way: if you know the optimal action-value function $Q^{*}(s,a)$, then in any given state, the optimal action $a^{*}(s)$ can be found by solving:

\begin{equation}
a^{*}(s) = \argmax_{a} Q^*(s,a)
\end{equation}

DDPG interleaves learning an approximator to $Q^*(s,a)$ with learning an approximator to $a^*(s)$, and it does so in a way which is specifically adapted for environments with continuous action spaces. But what does it mean that DDPG is adapted specifically for environments with continuous action spaces? It relates to how we compute the max over actions in $\max_a Q^*(s,a)$.

When there are a finite number of discrete actions, the max poses no problem, because we can just compute the Q-values for each action separately and directly compare them. (This also immediately gives us the action which maximizes the Q-value.) But when the action space is continuous, we can't exhaustively evaluate the space, and solving the optimization problem is highly non-trivial. Using a normal optimization algorithm would make calculating $\max_a Q^*(s,a)$ a painfully expensive subroutine. And since it would need to be run every time the agent wants to take an action in the environment, this is unacceptable.

Because the action space is continuous, the function $Q^*(s,a)$ is presumed to be differentiable with respect to the action argument. This allows us to set up an efficient, gradient-based learning rule for a policy $\mu(s)$ which exploits that fact. Then, instead of running an expensive optimization subroutine each time we wish to compute $\max_a Q(s,a)$, we can approximate it with $\max_a Q(s,a) \approx Q(s,\mu(s))$.

\subsection{Quick Facts}
\begin{itemize}
\item DDPG is an off-policy algorithm
\item DDPG can only be used for environments with continuous action spaces
\item DDPG can be thought of as being deep Q-learning for continuous action spaces
\end{itemize}

Next, we'll explain the math behind the two parts of DDPG: learning a Q function, and learning a policy.

\subsection{The Q-Learning Side of DDPG}

First, let's recap the Bellman equation describing the optimal action-value function, $Q^*(s,a)$. It's given by

\begin{equation}
Q^*(s,a) = \underset{s' \sim P}{E}[r(s, a) + \gamma \max_{a'} Q^*(s', a')]
\end{equation}

where $s' \sim P$ is shorthand for saying that the next state, $s'$, is sampled by the environment from a distribution $P(\cdot| s, a)$.

This Bellman equation is the starting point for learning an approximator to $Q^*(s,a)$. Suppose the approximator is a neural network $Q_{\phi}(s,a)$, with parameters $\phi$, and that we have collected a set ${\mathcal D}$ of transitions $(s,a,r,s',d)$ (where $d$ indicates whether state $s'$ is terminal). We can set up a mean-squared Bellman error (MSBE) function, which tells us roughly how closely $Q_{\phi}$ comes to satisfying the Bellman equation:

\begin{equation}
L(\phi, {\mathcal D}) = \underset{(s,a,r,s',d) \sim {\mathcal D}}{{\mathrm E}}\left[
    \Bigg( Q_{\phi}(s,a) - \left(r + \gamma (1 - d) \max_{a'} Q_{\phi}(s',a') \right) \Bigg)^2
    \right]
\end{equation}

Here, in evaluating $(1-d)$, we've used a Python convention of evaluating True to 1 and False to 0. Thus, when $d ==$ True---which is to say, when s' is a terminal state---the Q-function should show that the agent gets no additional rewards after the current state.

Q-learning algorithms for function approximators, such as DQN (and all its variants) and DDPG, are largely based on minimizing this MSBE loss function. There are two main tricks employed by all of them which are worth describing, and then a specific detail for DDPG.

\textbf{Trick One: Replay Buffers}. All standard algorithms for training a deep neural network to approximate $Q^*(s,a)$ make use of an experience replay buffer. This is the set ${\mathcal D}$ of previous experiences. In order for the algorithm to have stable behavior, the replay buffer should be large enough to contain a wide range of experiences, but it may not always be good to keep everything. If you only use the very-most recent data, you will overfit to that and things will break; if you use too much experience, you may slow down your learning. This may take some tuning to get right.

\textbf{Trick Two: Target Networks}. Q-learning algorithms make use of target networks. The term
\begin{equation}
r + \gamma (1 - d) \max_{a'} Q_{\phi}(s',a')
\end{equation}
is called the target, because when we minimize the MSBE loss, we are trying to make the Q-function be more like this target. Problematically, the target depends on the same parameters we are trying to train: $\phi$. This makes MSBE minimization unstable. The solution is to use a set of parameters which comes close to $\phi$, but with a time delay---that is to say, a second network, called the target network, which lags the first. The parameters of the target network are denoted $\phi_{\text{targ}}$.

In DQN-based algorithms, the target network is just copied over from the main network every some-fixed-number of steps. In DDPG-style algorithms, the target network is updated once per main network update by polyak averaging:
\begin{equation}
\phi_{\text{targ}} \leftarrow \rho \phi_{\text{targ}} + (1 - \rho) \phi,
\end{equation}
where $\rho$ is a hyperparameter between 0 and 1 (usually close to 1).

\textbf{DDPG Detail: Calculating the Max Over Actions in the Target}. As mentioned earlier: computing the maximum over actions in the target is a challenge in continuous action spaces. DDPG deals with this by using a target policy network to compute an action which approximately maximizes $Q_{\phi_{\text{targ}}}$. The target policy network is found the same way as the target Q-function: by polyak averaging the policy parameters over the course of training.

Putting it all together, Q-learning in DDPG is performed by minimizing the following MSBE loss with stochastic gradient descent:
\begin{equation}
L(\phi, {\mathcal D}) = \underset{(s,a,r,s',d) \sim {\mathcal D}}{{\mathrm E}}\left[
    \Bigg( Q_{\phi}(s,a) - \left(r + \gamma (1 - d) Q_{\phi_{\text{targ}}}(s', \mu_{\theta_{\text{targ}}}(s')) \right) \Bigg)^2
    \right],
\end{equation}

where $\mu_{\theta_{\text{targ}}}$ is the target policy.

\subsection{The Policy Learning Side of DDPG}

Policy learning in DDPG is fairly simple. We want to learn a deterministic policy $\mu_{\theta}(s)$ which gives the action that maximizes $Q_{\phi}(s,a)$. Because the action space is continuous, and we assume the Q-function is differentiable with respect to action, we can just perform gradient ascent (with respect to policy parameters only) to solve
\begin{equation}
\max_{\theta} \underset{s \sim {\mathcal D}}{{\mathrm E}}\left[ Q_{\phi}(s, \mu_{\theta}(s)) \right].
\end{equation}
Note that the Q-function parameters are treated as constants here.
Exploration vs. Exploitation

DDPG trains a deterministic policy in an off-policy way. Because the policy is deterministic, if the agent were to explore on-policy, in the beginning it would probably not try a wide enough variety of actions to find useful learning signals. To make DDPG policies explore better, we add noise to their actions at training time.

At test time, to see how well the policy exploits what it has learned, we do not add noise to the actions.