\section{Queueing}
\label{sec:queueing}
\index{queue}

Despite its didactic qualities, aggregate analysis (see
page~\pageref{par:aggregate}) is less frequently applied when the data
structures are not directly connected to numeration. We propose to
extend its scope by showing a compelling case study on
\emph{functional queues}
\citep{Burton_1982,Okasaki_1995,Okasaki_1998b}. A functional queue is
a linear data structure that is used in functional languages, whose
semantics force the programmer to model a \emph{queue} with two
stacks. Items can be pushed only on one stack and popped only on the
other:
\begin{equation*}
\begin{array}{@{}l@{}r@{\;}cc|c|c|c|c|c|cc@{\;}l@{}}
\cline{4-9}
& \text{Push, Pop (top)}
& \leftrightsquigarrow & & a & b & c & d & e &&&\\
\cline{4-9}
\end{array}
\end{equation*}
A queue is like a stack where items are added, or \emph{enqueued}, at
one end, called \emph{rear}, but taken out, or \emph{dequeued}, at the
other end, called \emph{front}:
\begin{equation*}
\begin{array}{@{}l@{}r@{\;}cc|c|c|c|c|c|cc@{\;}l@{}}
\cline{4-10}
& \text{Enqueue (rear end)}
                & \rightsquigarrow & & a & b & c & d & e &
& \rightsquigarrow & \text{Dequeue (front end).}\\
\cline{4-10}
\end{array}
\end{equation*}
Let us implement a queue with two stacks: one for enqueueing, called
the \emph{rear stack}\index{queue!rear stack}\index{stack!rear
  $\sim$|see{queue}}, and one for dequeueing, called the \emph{front
  stack}\index{queue!front stack}\index{stack!front
  $\sim$|see{queue}}. The previous ideal queue is equivalent to the
functional queue
\begin{equation*}
\begin{array}{r@{\;}cc|c|c|c|c|c|c|cc@{\;}l}
  \cline{3-6}\cline{8-10}
  \text{Enqueue (rear)} & \rightsquigarrow & & a & b & c & & d & e & &
  \rightsquigarrow & \text{Dequeue (front).}\\
  \cline{3-6}\cline{8-10}
\end{array}
\end{equation*}
Enqueueing is now pushing on the rear stack and dequeueing is popping
on the front stack. In the latter case, if the front stack is empty
and the rear stack is not, we swap the stacks and reverse the new
front stack. Graphically, dequeueing in the configuration
\begin{equation*}
\begin{array}{c|c|c|c|c|c} \cline{1-4}\cline{6-6} & a & b & c &
&\\ \cline{1-4}\cline{6-6}
\end{array}
\end{equation*}
requires first to make
\begin{equation*}
\begin{array}{c|c|c|c|c|c}
  \cline{1-1}\cline{3-6}
  & & a & b & c &\\
  \cline{1-1}\cline{3-6}
\end{array}
\end{equation*}
and then dequeue~\(c\).

Let us model a queue as we modelled the operation push by a function
\fun{cons/2}\index{cons@\fun{cons/2}} without a definition, on
page~\pageref{par:stacks}. We shall use the name
\fun{q/2}\index{q@\fun{q/2}} and the call \(\fun{q}(r,f)\) denotes a
functional queue with rear stack~\(r\) and front
stack~\(f\). Enqueueing is performed by the function
\fun{enq/2}\index{enq@\fun{enq/2}}:
\begin{equation}
\fun{enq}(x,\fun{q}(r,f)) \rightarrow \fun{q}(\cons{x}{r},f).
\label{def:enq}
\end{equation}
Dequeueing requires the result to be a \emph{pair}\index{functional
  language!pair} made of the dequeued item and the new queue without
it. Actually, the new queue is the first component of the pair, to fit
how the operation is depicted in the figures as a rightwards arrow.
%% We could denote the mathematical pair \((x,y)\) by
%% \(\fun{pair}(x,y)\), but a more symbolic notation is handy:
%% \(\pair{x}{y}\). The angle brackets allow us to avoid writing
%% \(f((x,y))\) when we mean \(f(\pair{x}{y})\).
Let us call \fun{deq/1}\index{deq@\fun{deq/1}} the
dequeueing function:
\begin{equation}
\begin{array}{@{}r@{\;}l@{\;}l@{}}
  \fun{deq}(\fun{q}(\cons{x}{r},\el))
& \xrightarrow{\smash{\theta}}
& \fun{deq}(\fun{q}(\el,\fun{rcat}(r,[x])));\\
  \fun{deq}(\fun{q}(r,\cons{x}{f}))
& \xrightarrow{\smash{\iota}}
& \pair{\fun{q}(r,f)}{x}.
\end{array}
\label{def:deq}
\end{equation}
See page~\pageref{def:rev} for the definition~\eqref{def:rev}
of~\fun{rcat/2}\index{rcat@\fun{rcat/2}}. We shall say that the queue
has size~\(n\) if the total number of items in both stacks
is~\(n\). The cost of enqueueing\index{enq@$\C{\fun{enq}}{n}$} is
\(\C{\fun{enq}}{n} = 1\). The minimum
cost\index{deq@$\B{\fun{deq}}{n}$} for dequeueing is
\(\B{\fun{deq}}{n} = 1\), by rule~\(\iota\). The maximum cost is
\(\W{\fun{deq}}{n} = n+2\), as seen with the trace
\(\theta\eta^{n-1}\zeta\iota\). Let~\(\A{n}\) be the cost of a
sequence of \(n\)~updates on a functional queue originally empty. A
first attempt at assessing~\(\A{n}\) consists in ignoring any
dependence on previous operations and take the maximum individual
cost. Since \(\C{\fun{enq}}{k} \leqslant \C{\fun{deq}}{k}\), we
consider a series of \(n\)~dequeueings in their worst case, that is,
with all the items located in the rear stack. Besides, after
\(k\)~updates, there may be at most \(k\)~items in the queue, so
\begin{equation*}
\A{n} \leqslant \sum_{k=1}^{n-1}{\W{\fun{deq}}{k}} =
\frac{1}{2}{(n-1)(n+4)} \sim \frac{1}{2}{n^2}.
\end{equation*}

\paragraph{Aggregate analysis}
\index{cost!amortised $\sim$}
\index{queue!amortised cost}

Actually, this is overly pessimistic and even unrealistic. First, one
cannot dequeue on an empty queue, therefore, at any time, the number
of enqueueings since the beginning is always greater or equal than the
number of dequeueings and the series must start with one
enqueueing. Second, when dequeueing with the front being empty, the
rear stack is reversed onto the front stack, so its items cannot be
reversed again during the next dequeueing, whose cost will
be~\(1\). Moreover, as remarked above, \(\C{\fun{enq}}{k} \leqslant
\C{\fun{deq}}{k}\), so the worst case for a series of
\(n\)~operations occurs when the number of dequeueings is maximum,
that is, when it is \(\lfloor{n/2}\rfloor\). If we denote by~\(e\) the
number of enqueueings and by~\(d\) the number of dequeueings, we have
the relationship \(n = e + d\) and the two requisites for a worst case
become \(e=d\) (\(n\)~even) or \(e=d+1\) (\(n\)~odd). The former
corresponds graphically to a \emph{Dyck path}\index{Dyck path} and the
latter to a \emph{Dyck meander}\index{Dyck meander}.

\paragraph{Dyck path}
\index{Dyck path}

Let us depict updates as in \fig~\vref{fig:enq_deq}.
\begin{figure}
\centering
\subfloat[Enqueue\label{fig:enqueue}]{%
  \includegraphics[bb=69 662 132 721,scale=0.75]{enqueue}
}
\qquad
\subfloat[Dequeue\label{fig:dequeue}]{%
  \includegraphics[bb=69 662 132 721,scale=0.75]{dequeue}
}
\caption{Graphical representations of operations on
  queues\label{fig:enq_deq}}
\end{figure}
Textually, we represent an enqueueing as an opening parenthesis and a
dequeueing as a closing parenthesis. For example,
\texttt{((()()(()))())} can be represented in \fig~\ref{fig:dyck_path}
\begin{figure}
\centering
\includegraphics[bb=75 570 508 727,scale=0.7]{dyck_path}
\caption{Dyck path modelling queue operations (cost \(24\))
\label{fig:dyck_path}}
\end{figure}
as a Dyck path\index{Dyck path}. For a broken line to qualify
as a Dyck path of length~\(n\), it has to start at the origin
\((0,0)\) and end at coordinates \((n,0)\). In terms of a \emph{Dyck
  language}, an enqueueing is called a \emph{rise}\index{Dyck
  path!rise} and a dequeueing is called a \emph{fall}\index{Dyck
  path!fall}. A rise followed by a fall, that is, \texttt{()}, is
called a \emph{peak}\index{Dyck path!peak}. For instance, in
\fig~\ref{fig:dyck_path}, there are four peaks. The number near each
rise or fall is the cost incurred by the corresponding operation. The
abscissa axis bears the ordinal of each operation.

When \(e=d\), the line is a Dyck path of length \(n=2e=2d\). In order
to deduce the total cost in this case, we must find a
\emph{decomposition}\index{Dyck path!decomposition} of the path, by
which we mean to identify patterns whose costs are easy to calculate
and which make up any path, or to associate any path to another path
whose cost is the same but easy to find. \Fig~\ref{fig:dyck_eq}
\begin{figure}[!b]
\centering
\includegraphics[bb=75 570 508 727,scale=0.7]{dyck_eq}
\caption{Dyck path equivalent to \fig~\ref{fig:dyck_path}
\label{fig:dyck_eq}}
\end{figure}
shows how the previous path is mapped to an equivalent path only made
of a series of isosceles triangles whose bases belong to the abscissa
axis. Let us call them \emph{mountains}\index{Dyck path!mountain} and
their series a \emph{range}\index{Dyck path!range}. The mapping is
simple: after the first series of falls, if we are back to the
abscissa axis, we have a mountain and we proceed with the rest of the
path. Otherwise, the next operation is a rise and we exchange it with
the first fall after it. This brings us down by~\(1\) and the process
resumes until the abscissas are reached. We call this méthod
\emph{rescheduling}\index{Dyck path!rescheduling} because it amounts,
in operational terms, to reordering subsequences of operations a
posteriori.

For instance, \fig~\vref{fig:rescheduling}
\begin{figure}
\centering
\subfloat[Initial\label{fig:initial}]{%
  \includegraphics[scale=0.75,bb=48 588 218 726]{mount0}}
\qquad
\subfloat[Swapping \(4\nearrow 5\) and \(5\searrow 6\)]{%
  \includegraphics[scale=0.75,bb=48 588 218 726]{mount1}}\\
\subfloat[Swapping \(5\nearrow 6\) and \(8 \searrow 9\)\label{fig:valley}]{%
  \includegraphics[scale=0.75,bb=48 588 218 726]{mount2}}
\qquad
\subfloat[Last one]{%
  \includegraphics[bb=13 645 180 782,scale=0.75]{mount3}}
\caption{Rescheduling of \fig~\ref{fig:dyck_path} into
  \fig~\ref{fig:dyck_eq}\label{fig:rescheduling}}
\end{figure}
displays the rescheduling of \fig~\vref{fig:dyck_path}. Note that two
different paths can be rescheduled into the same path. What makes
\fig~\ref{fig:valley} equivalent to \fig~\ref{fig:initial} is the
invariance of the cost because all operations have cost~\(1\). This
always holds because enqueueings always have cost~\(1\) and the
dequeueings involved in a rescheduling have cost~\(1\), because they
found the front stack non\hyp{}empty after a peak. We proved that all
paths are equivalent to a range with the same cost, therefore, the
maximum cost can be found on ranges alone.

Let us note \(e_1, e_2, \dots, e_k\) the maximal subsequences of
rises; for example, in \fig~\ref{fig:dyck_eq}, we have \(e_1=3\),
\(e_2 = 3\) and \(e_3 = 1\). Of course, \(e = e_1 + e_2 + \dots +
e_k\). The fall making up the \(i\)th peak incurs the
cost\index{deq@$\W{\fun{deq}}{n}$} \(\W{\fun{deq}}{e_i} = e_i + 2\),
due to the front being empty because we started the rises from the
abscissa axis. The next \(e_i-1\) falls have all cost~\(1\), because
the front is not empty. For the \(i\)th mountain, the cost is thus
\(e_i+(e_i+2)+(e_i-1) = 3e_i+1\). Then \(\A{e,k} =
\sum_{i=1}^{k}{(3e_i+1)} = 3e + k\). The maximum cost is obtained by
maximising \(\A{e,k}\) for a given~\(e\):
\begin{equation*}
\W{}{e,e} := \max_{1 \leqslant k \leqslant e}{\A{e,k}} = \A{e,e} = 4e
= 2n,\,\; \text{with \(n=e+d=2e\)},
\end{equation*}
where \(\W{}{e,e}\) is the maximum cost when there are
\(e\)~enqueueings and \({d=e}\) dequeueings. In other words, the worst
case when \(e=d=7\) is the saw\hyp{}toothed Dyck path shown in
\fig~\vref{fig:max_dyck}.
\begin{figure}
\centering
\includegraphics[bb=86 657 507 726,scale=0.7]{max_dyck}
\caption{Worst case when \(e=d=7\) (cost \(28\))
\label{fig:max_dyck}}
\end{figure}
Importantly, there are no other Dyck paths whose rescheduling lead to
this worst case and the reason is that the reverse transformation from
ranges to general Dyck paths works on dequeueings of cost~\(1\) and
the solution we found is the only one with no dequeueing equal
to~\(1\).

\paragraph{Dyck meander}
\index{Dyck meander}

Another worst case occurs if \(e = d + 1\) and the line is then a Dyck
meander whose extremity ends at ordinate \(e-d=1\). An example is
given in \fig~\ref{fig:meander1},
\begin{figure}[t]
\centering
\includegraphics[bb=75 570 480 727,scale=0.82]{meander1}
\caption{Dyck meander modelling queue operations (total cost~\(21\))\label{fig:meander1}}
\end{figure}
where the last operation is a dequeueing. The dotted line delineates
the result of applying the rescheduling we used on Dyck paths. Here,
the last operation becomes an enqueueing.

Another possibility is shown in \fig~\ref{fig:meander2},
\begin{figure}[t]
\centering
\includegraphics[bb=75 570 480 727,scale=0.82]{meander2}
\caption{Dyck meander modelling queue operations (total cost
  \(23\))\label{fig:meander2}}
\end{figure}
where the last operation is left unchanged. The difference between the
two examples lies in the fact that the original last dequeueing has,
in the former case, a cost of~\(1\) (thus is changed) and, in the
latter case, a cost greater than~\(1\) (thus is invariant). The third
kind of Dyck meander is one ending with an enqueueing, but because
this enqueueing must start from the abscissa axis, this is the same
situation as the result of rescheduling a meander ending with a
dequeueing with cost~\(1\) (see dotted line in \fig~\ref{fig:meander1}
again). Therefore, we are left to compare the results of rescheduling
meanders ending with a dequeueing, that is, we envisage two cases.
\begin{itemize}

  \item If we have a range of \(n-1\)~operations followed by an
    enqueueing, the maximum cost of the range is the cost of a
    saw\hyp{}toothed Dyck path, that is,
    \(\W{}{e-1,e-1}=4(e-1)=2n-2\), because \(n=e+d=2e-1\), followed by
    an enqueueing, totalling \(2n-1\).

  \item Otherwise, we have a range of \(n-3\)~operations followed by
    two rises and one fall (of cost~\(6\)). The cost is
    \(\W{}{e-2,e-2}+6=2n\), which is marginally greater than the
    previous case.

\end{itemize}

\mypar{Amortised cost}
\index{cost!amortised $\sim$}
\index{queue!amortised cost}

The cost~\(\A{n}\) of a series of \(n\)~queue updates, starting on an
empty queue, is tightly bounded as\index{queue@$\A{n}$}
\begin{equation*}
n \leqslant \A{n} \leqslant 2n,
\end{equation*}
where the lower bound is tight if all updates are enqueueings and the
upper bound when a saw\hyp{}toothed range is followed by one
enqueueing or else two enqueueings and one dequeueing. By definition,
the amortised cost of one operation is~\(\A{n}/n\) and lies between
\(1\)~and~\(2\), which is less than our first analysis (\(\sim
n/2\)). We published a slightly different analysis on the same
examples \citep{Rinderknecht_2011}.

\paragraph{Side note}

We can gain some more abstraction by using a dedicated constructor for
the empty queue, \(\fun{nilq/0}\), and changing accordingly the
definition of \fun{enq/2} in~\eqref{def:enq} \vpageref{def:enq} so it
handles this case:
\begin{equation*}
\begin{array}{r@{\;}l@{\;}l}
\fun{enq}(x,\fun{nilq}()) & \rightarrow & \fun{q}([x],\el);\\
\fun{enq}(x,\fun{q}(r,f)) & \rightarrow & \fun{q}(\cons{x}{r},f).
\end{array}
\end{equation*}
We can improve this a little by pushing~\(x\) directly into the front
stack:
\begin{equation*}
\begin{array}{r@{\;}l@{\;}l}
\fun{enq}(x,\fun{nilq}()) & \rightarrow & \fun{q}(\el,[x]);\\
\fun{enq}(x,\fun{q}(r,f)) & \rightarrow & \fun{q}(\cons{x}{r},f).
\end{array}
\end{equation*}

\paragraph{Exercises}
\begin{enumerate}

  \item Let \(\fun{nxt}(q)\)\index{nxt@\fun{nxt/1}} be the next item
    to be dequeued from~\(q\):
    \begin{equation*}
      \begin{array}{r@{\;}l@{\;}l}
        \fun{nxt}(\fun{q}(\cons{x}{r},\el)) & \rightarrow
        & \fun{nxt}(\fun{q}(\el,\fun{rcat}(r,[x])));\\
        \fun{nxt}(\fun{q}(r,\cons{x}{f})) & \rightarrow & x.
        \index{q@\fun{q/2}}
      \end{array}
    \end{equation*}
    Modify \fun{enq/2}\index{enq@\fun{enq/2}},
    \fun{deq/1}\index{deq@\fun{deq/1}} and
    \fun{nxt/1}\index{nxt@\fun{nxt/1}} in such a way that
    \(\C{\fun{nxt}}{n} = 1\)\index{nxt@$\C{\fun{nxt}}{n}$}, where
    \(n\)~is the number of items in the queue.

    \item Find~\(\A{n}\) using the slightly different definition
      \begin{equation*}
        \begin{array}{@{}r@{\;}l@{\;}l@{}}
          \fun{deq}(\fun{q}(\cons{x}{r},\el))
          & \rightarrow
          & \fun{deq}(\fun{q}(\el,\fun{rev}(\cons{x}{r})));\\
          \fun{deq}(\fun{q}(r,\cons{x}{f}))
          & \rightarrow
          & \pair{\fun{q}(r,f)}{x}.
        \end{array}
      \end{equation*}

\end{enumerate}


\subsection{Simulated annealing}

\subsubsection{Introduction}

We can use a version of Metropolis-Hastings to find the global maximum of a function \(f(x)\).

We start with an arbitrary point \(x_0\).

We move randomly from this to identiy a candidate point \(x_c\).

We accept this with probability depending on the the relationship between \(x_0\) and \(x_c\).

This process will converge on the global maximum.

\subsubsection{Hyperparameter}

There is a hyperparameter for selection. At the extreme this becomes a greedy function.


\chapter{Numerical Integrators} \label{Ch:NumericalIntegrators}

\section{Runge-Kutta Integrators} \index{Numerical
integrators!Runge-Kutta}

The Runge-Kutta integration scheme is a single step method used to
solve  differential equations for $n$ coupled variables of the
form

\[{{dr^i}\over{dt}} = f(t,r)\]

(The superscript in this discussion refers to the variables; hence
$f^i$ is the $i^{th}$ variable, and $r^{(n)}$ refers to all $n$
variables.) The method takes an integration step, $h$, by breaking
the  interval into several stages (usually of smaller size) and
calculating  estimates of the integration result at each stage.
The later stages use the  results of the earlier stages. The
cumulative effect of the integration is  an approximate total step
$\delta t$, accurate to a given order in the  series expansion of
the differential equation, for the state variables  $r_i(t+\delta
t)$ given the state $r_i(t)$.

The time increment for a given stage is given as a multiple $a_i$
of the  total time step desired; thus for the $i^{th}$ stage the
interval used  for the calculation is $a_i \delta t$; the estimate
of the integrated  state at this stage is given by

\[ k_i^{(n)} = \delta t f(t+a_i\delta t, r^{(n)}(t) + \sum_{j=1}^{i-1}b_{ij}k_j^{(n)}) \]

where $b_{ij}$ contains a set of coefficients specific to the
Runge-Kutta instance being calculated. Given the results of the
stage calculations, the  total integration step can be calculated
using another set of coefficients, $c_j$ and the formula

\[ r^{(n)}(t+\delta t) = r^{(n)}(t) + \sum_{j=1}^{stages}c_j k_j^{(n)} \]

The error control for these propagators is implemented by
comparing the  results of two different orders of integration. The
difference between the  two steps provides an estimate of the
accuracy of the step; a second set of  coefficients corresponding
to this second integration scheme can be used to obtain a solution

\[r'^{(n)}(t+\delta t) = r^{(n)}(t) + \sum_{j=1}^{stages}c_j^* k_j^{*(n)}\]

With care, the stage estimates $k_j$ and $k_j^*$ can be selected
so  that they are the same; in that case, the estimate of the
error in the  integration $\Delta^{(n)}$ can be written

\[ \Delta^{(n)} = \left| \sum_{j=1}^{stages}(c_j - c_j^*) k_j^{(n)} \right| \]

(The difference between the coefficients $c_j - c_j^*$ is the
array of  error estimate coefficients (ee) in this code.)

Once the estimated error has been calculated, the size of the
integration  step can be adapted to a size more appropriate to the
desired accuracy of the  integration. If the step results in a
solution that is not accurate enough, the step needs to be
recalculated with a smaller step size. Labeling the desired
accuracy $\alpha$ and the obtained accuracy $\epsilon$
(calculated, for instance, as the largest element of the array
$\Delta$), the new step used by the Runge-Kutta integrator is

\[\delta t_{new}= \sigma\delta t\left({{\alpha}\over{\epsilon}}\right)^{1/(m-1)}\]

where $m$ is the order of truncation of the series expansion of
the  differential equations being solved. The factor $\sigma$ is a
safety  factor incorporated into the calculation to avoid
unnecessary iteration over attempted steps. Common practice is to
set this factor to 0.9; that is the default value used in this
implementation.

Similarly, if the step taken does not result in the desired
accuracy, you may  want to increase the step size parameter for
the next integration step. The  new estimate for the desired
stepsize is given by

\[\delta t_{new}= \sigma\delta t\left({{\alpha}\over{\epsilon}}\right)^{1/(m)}\]

Sometimes you do not want to increase the stepsize in this manner;
for example,  you may want to keep the maximum step taken at some
fixed value. This  implementation provides a mechanism for
specifying a maximum allowed step.

Sometimes it is convenient to request steps of a specified size,
regardless  of the stepsize control algorithm or the calculation
of the \char`\"{}best step\char`\"{}  described above. This
implementation accomplishes that task by taking  multiple error
controlled steps is necessary to step across the requested
interval.

Both of these features are implemented using the boolean flags
described in  the base class for the integrators. See the
documentation for the {\bf Integrator} {\rm
(p.\,\pageref{classIntegrator})} class for more information about
these flags.

\subsection{Constructor \& Destructor Documentation}
\subsubsection{\setlength{\rightskip}{0pt plus 5cm}Runge\-Kutta::Runge\-Kutta (int {\em st}, int {\em order})}\label{classRungeKutta_a0}

Provides the greatest relative error in the state vector.

This method takes the state vector and calculates the error in
each  component. The error is then divided by the change in the
component. The function returns the largest of the resulting
relative errors.

Override this method if you want a different error estimate for
the stepsize  control. For example, we are using

\[error_i = \left({{\Delta_i(t+\delta t)}\over {r_i(t+\delta t) - r_i(t)}}\right)\]

Another popular approach is to divide the estimated error
$\Delta_i$ by  the norm of the corresponding 3-vector; for
instance, divide the error in x  by the magnitude of the
displacement in position for the step.

\section{Prince-Dormand Integrators}

\section{Adams Bashforth Moulton}  \index{Numerical
integrators!Adams-Bashforth-Moulton}

Implementation of the Adams-Bashford-Moulton Predictor-Corrector.

This code implements a fourth-order Adams-Bashford predictor /
Adams-Moulton corrector pair to integrate a set of first order
differential equations. The algorithm is found at  {\tt
http://chemical.caeds.eng.uml.edu/onlinec/white/math/s1/s1num/s1num.html}
or in Bate, Mueller and White, pp. 415-417.

The predictor step extrapolates the next state $r_{i+1}$ of the
variables using the the derivative information $(f)$ at the
current state and three  previous states of the variables, by
applying the equation

\[ r_{i+1}^{*j} = r_i^j + {{h}\over{24}}\left[55 f_n^j - 59 f_{n-1}^j + 37 f_{n-2}^j - 9 f_{n-3}^j \right] \]

The corrector uses derivative information evaluated for this
state, along  with the derivative information at the original
state and two preceding  states, to tune this state, giving the
final, corrected state:

\[ r_{i+1}^{j} = r_i^j + {{h}\over{24}}\left[9 f_{n+1}^{*j} + 19 f_{n}^j - 5 f_{n-1}^j + 1 f_{n-2}^j \right] \]

Bate, Mueller and White give the estimated accuracy of this
solution to be

\[ee = {{19}\over{270}} \left|r_{i+1}^{*j} - r_{i+1}^{j}\right|\]

Method used to fire the step refinement (the corrector phase).

\section{Bulirsch-Stoer} \index{Numerical integrators!Bulirsch-Stoer}

\section{Stopping Condition Algorithm} \index{Numerical integrators!Stopping condition algorithm}
\index{Stopping condition algorithm}

\section{Integrator Coefficients} \index{Numerical integrators!Coefficients}

\input{PD45Coefficients}
\input{PD56Coefficients}
\input{RKF56Coefficients}
%\clearpage
\input{PD78Coefficients}


\section{Math}
\subsection{Euler Function}
\lstinputlisting{"./math/euler.c"}

\subsection{Chinese Remainder Theorem}
\[
	x \equiv a_i \quad (\mathrm{mod}\;m_i)
\]
\lstinputlisting{"./math/crt.c"}

\subsection{FFT}
\lstinputlisting{"./math/fft.cc"}

\subsection{Number Theory Inverse}
\lstinputlisting{"./math/inv.cc"}

\subsection{Linear Programming}
\lstinputlisting{"./math/lp.cc"}


This chapter contains routines for geometric optics solutions of reflection and refraction of rays at dielectric interfaces. All of these solutions are fundamentally based on Snell's law. Specifically, these routines will compute the reflection or reflection point on an interface given the positions of a source and observer as inputs. These algorithms are especially useful for radar scattering and subsurface imaging problems. Many of these solutions are well known. In general, very fast, and much better, codes for geometric optics can be found in the computer graphics literature.

This chapter contains: 1) routines for computing the reflection point on a plane and sphere, 2) derivations and two routines for solving for the refraction point through the plane: one is analytical, one is iterative, 3) an analytical solution for the refraction point on a circular interface given a pair of interior and exterior source and observation points, 4) a vectorized iterative solution for refraction through a circular interface, and 5) a simple coordinate transform for the solution for the refraction point on a sphere.

\section{Snell's Law}

Snell's law of refraction at a flat dielectric half-space is given by
\eq{ \dfrac{\sin\theta_1}{\sin\theta_2} =  \dfrac{n_2}{n_1} }

\noindent where $\theta_1$ is the angle measured from normal in medium 1 with index of refraction, $n_1$, and $\theta_2$ is the angle measured from normal in medium 2 with index of refraction, $n_2$. Recall $n = \sqrt{\epsilon_r}$, where $\epsilon_r$ is the relative permittivity.

%
%\noindent This has the alternate form
%
%\[1 + \cot^2(\theta_t) = (1 + \cot^2(\theta_i)) \epsilon_r  \]

\section{Reflection Point - Flat Interface}

Here we find the ray-solution for the specular point for two arbitrary points above a flat surface. One point could be a source, the other a receiver. This can be solved a number of ways, including the image method or by enforcing equal incident and reflected angles relative to the normal of the specular point.

Let two points $\bb{r}_1$ and $\bb{r}_2$ be above a flat surface that is parallel to the XY plane at level $z = z_o$. Each half space has a different index of refraction. Define the following quantities
\ea{h_1 &=&z_1-z_o \\
h_2 &=& z_2-z_o \\
L &=& \sqrt{(x_2-x_1)^2 + (y_2-y_1)^2} \\
\hat{u} &=& \dfrac{(x_2-x_1)  \hat{x} + (y_2-y_1)\hat{y}}{L} }

\noindent where $\hat{u}$ is the transverse unit vector from $\bb{r}_1$ and $\bb{r}_2$. Define the transverse distance from $\bb{r}_1$ to the reflection point as $u_1$ and from $\bb{r}_2$ to the reflection point as $u_2$.  Then for equal incident and reflected angles relative to normal
\ea{ \tan\theta &=& \dfrac{u_1}{h_1} \\
\tan\theta &=& \dfrac{u_2}{h_2} }

\begin{figure}[h]
   \centering
   \includegraphics[width=3in]{ReflectionRefraction/Figures/reflectionPlane}
   \caption{Geometry for the solution of the planar reflection point, $\bb{r}$, which is unknown.}
\end{figure}

Equating the sines, using the fact that $L = u_1 +u_2$, and solving for $u_1$, we get
\eq{u_1 = \dfrac{h_1 L}{(h_2 + h_1)} }

From which the reflection point $\bb{r}$ is found by reprojecting
\eq{\bb{r} = x_1 \hat{x} + y_1 \hat{y} + u_1 \hat{u} + z_o \hat{z_o}}

The routine \texttt{reflectionPlane} takes as input the two points, $\bb{r}_1$, $\bb{r}_2$, and the level of the plane, $z_o$, and returns the reflection point, $\bb{r}$, as well as $\theta$ and the lengths of the vectors $l_1$ and $l_2$.  This routine is vectorized for any number of pairs of source and receiver points.

\begin{figure}[H]
   \centering
   \includegraphics[width=4in]{ReflectionRefraction/Figures/reflectionraysplane}
   \caption{Geometry for the solution of the planar reflection point, $\bb{r}$, which is unknown off a plane at level $z_o = 0$.}
\end{figure}

{\footnotesize
\VerbatimInput{\code/ReflectionRefraction/reflectionPlane.m}
}

\section{Reflection Point - Sphere}

Computing the reflection point on a sphere is often needed in Earth and planetary remote sensing, for example, when determining the specular point on a body between a source and receiver. The solution here is the one derived in \cite{reflection_sphere}, and summarized next. The reflection point on a circle can be found with this method by restricting the source and receiver points to a plane.

Let a sphere with radius $r$ be centered at the origin. $\bb{r}_1$ and $\bb{r}_2$ are radius-normalized vectors to exterior points 1 and 2, respectively, which can be a source and receiver.  Next, define the following coefficients
\ea{a &=& \bb{r}_1\cdot\bb{r}_1 \\
b &=& \bb{r}_1\cdot\bb{r}_2 \\
c &=& \bb{r}_2\cdot\bb{r}_2}

After enforcing the reflection condition on the sphere, the solution comes from finding the roots of the following quartic polynomial
\eq{\sum_{n=0}^{4} a_n y^n = 0}
\ea{a_4 &=& 4c(ac-b^2) \\
a_3 &=& -4(ac-b^2) \\
a_2 &=& a+2b+c-4ac \\
a_1 &=& 2(a-b) \\
a_0 &=& a-1}

Once the roots are known, the positive roots with zero imaginary part are kept, call these $\bar{y}$. This is used to compute a second quantity
\eq{\bar{x} = \dfrac{-2 c \bar{y}^2 + \bar{y} + 1}{2 b \bar{y} + 1}}

The solution is the pair $(\bar{x},\bar{y})$ for which both are positive. Then the reflection point on the sphere is
\eq{\bb{r} = r (\bar{x} \bb{r}_1 + \bar{y} \bb{r}_2 )}

\begin{figure}[h]
   \centering
   \includegraphics[width=4.5in]{ReflectionRefraction/Figures/reflectonraysphere}
   \caption{Geometry for the reflection point on a sphere.}
\end{figure}

\paragraph{Line-Sphere Intersection}

We exclude shadowed points by using the line-sphere intersection solution, \cite{linesphereintersect}. The equation for a line that originates at $\bb{r}_1$ and passes through $\bb{r}_2$ is
\eq{\bb{u}(l)  = \bb{r}_1 + l \hat{u} }

\noindent where $l$ is the distance along the line and $\hat{u}$ is
\eq{\hat{u} = \dfrac{\bb{r}_2 - \bb{r}_1}{\vert \bb{r}_2 - \bb{r}_1\vert}}

For a sphere with center, $\bb{c}$, and radius, $r$, the solutions for $l$ are
\ea{l &=& -\hat{u} \cdot(\bb{r}_1-\bb{c}) \pm \sqrt{\Delta} \\
\Delta &=& \left(\hat{u} \cdot(\bb{r}_1-\bb{c})\right)^2 - \left(\Vert \bb{r}_1 - \bb{c} \Vert^2 - r^2\right)}

If $\Delta < 0$ then there are no intersections. If $\Delta = 0$ then the line is tangent to the sphere. If $\Delta > 0$ then the line intersects the sphere at two places. In the last case, the reflection point is determined by checking which of the two intersection points is closer to the source.

\paragraph{Routine}
The routine \texttt{reflectionSphere} solves for the reflection point on a sphere that is centered at the origin. It takes as input the Cartesian points $\bb{r}_1$, $\bb{r}_2$, and the sphere radius, $r$, and returns the reflection point, $\bb{r}$, which lies on the surface of the sphere. The routine is vectorized to take multiple pairs of points at once, and returns \texttt{nan} if the points are shadowed or interior to the sphere. Shadowing is determined by the routine \texttt{lineIntersectSphere}, not copied here, that implements the line-sphere intersection solution above.

{\footnotesize
\VerbatimInput{\code/ReflectionRefraction/reflectionSphere.m}
}

\section{Refraction Point - Flat Interface}

We derive two solutions for the refraction point through a flat dielectric interface given two arbitrary points on either side of the interface. The first solution is in terms of the roots of a quartic polynomial, \cite{heliere2007radio}. The second is an iterative algorithm, \cite{lei20202}. Practical applications of this are subsurface SAR processing, e.g. radar sounding through ice, ground penetrating radar, or through-wall imaging, where the propagation phase between two points across a dielectric interface needs to be computed for focusing. In these cases, the refraction point is really an intermediate result in order to obtain the incidence and transmission angles and propagation distances along the bent-ray path.

The geometry is shown in Figure \ref{fig9a}. The dielectric interface is parallel to the $XY$ plane at level $z = z_o$. Let two points $\bb{r}_1$ and $\bb{r}_2$ be above and below the interface in mediums 1 and 2 each having an index of refraction, $n_1$ and $n_2$, respectively. Next define the following quantities,
\ea{h &=& z_1 - z_o \\
d &=& z_2 - z_o \\
L &=& \sqrt{(x_2-x_1)^2 + (y_2-y_1)^2} \\
\hat{u} &=& \dfrac{(x_2-x_1)  \hat{x} + (y_2-y_1)\hat{y} }{L}}

\noindent \noindent where $h$ is the height of $\bb{r}_1$ above the interface, $d$ is the depth of $\bb{r}_2$ below the interface, $\hat{u}$ is the transverse unit vector and $L$ is the transverse distance from point 1 to point 2. Finally, $u$ is the transverse distance along $\hat{u}$ between the projection of $\br_1$ on the interface and the refraction point.

\begin{figure}[h]
   \centering
   \includegraphics[width=3in]{ReflectionRefraction/Figures/refractionPlane}
   \caption{Geometry for the solution of the planar refraction point, $\bb{r}$, which is unknown.}
   \label{fig9a}
\end{figure}

\subsubsection{Quartic Solution}

Start with the observations that
\begin{eqnarray}
\sin\theta_1 &=& \dfrac{u}{\sqrt{h^2 + u^2}} \nonumber \\
\sin\theta_2 &=& \dfrac{L-u}{\sqrt{d^2 + (L-u)^2}} \nonumber
\end{eqnarray}

These are equated though Snell's law
\eq{\dfrac{u}{\sqrt{h^2 + u^2}}   = \dfrac{n_2}{n_1}\dfrac{L-u}{\sqrt{d^2 + (L-u)^2}} }

After squaring both sides, this can be rearranged as a fourth order polynomial in $u$:

\eq{ a u^4 + b u^3 + c u^2 + d u + e = 0}
\begin{eqnarray}
a &=& n_1^2 - n_2^2 \nonumber \\
b &=& -2 L (n_1^2 - n_2^2) \nonumber \\
c &=& n_1^2(L^2  + d^2) - n_2^2( L^2 + h^2) \nonumber \\
d &=& 2 n_2^2 L h^2 \nonumber \\
e &=& -n_2^2 L^2 h^2  \nonumber
\end{eqnarray}

The roots can be found with a root finding algorithm or analytically. The solution is the positive real root for which $u < L$.  This solution is good for either $n_1 > n_2$ or $n_1 < n_2$. Once found, $u$ is reprojected to give the refraction point:
\eq{\bb{r} = x_1 \hat{x} + y_1 \hat{y} + u \hat{u} + z_o \hat{z}}

\subsubsection{Iterative Solution}
For the iterative solution, we use the facts that
\begin{eqnarray}
u &=& h \tan \theta_1 \\
u &=& L - d \tan \theta_2
\end{eqnarray}

Start with the straight-ray approximation for $\theta_1$, then with the addition of Snell's law, we can build the following iteration
\begin{eqnarray}
\textrm{Initialize:} \quad \quad \theta_1 &\leftarrow &\arctan\left( \dfrac{L}{h + d}\right) \ \nonumber \\
\textrm{Iterate:} \quad \quad\theta_2 &\leftarrow &\arcsin\left(\dfrac{n_1}{n_2} \sin\theta_1\right) \nonumber  \\
u & \leftarrow &L - d \tan\theta_2 \nonumber  \\
\theta_1 &\leftarrow&  \arctan\left( \dfrac{u}{h}\right) \nonumber
\end{eqnarray}

This is an alternating sequence for $u$ (and $\theta_1$, $\theta_2$). The physical interpretation is that the value of $u$ bounces closer to the true solution at every iteration. This is effectively a Newton iteration of the transcendental equation for $\theta_1$ without a nonlinear optimization algorithm. The solution is lightning fast (requiring maybe 10 iterations), when it works. The solution does not always converge, which happens when $h$ is too close to the interface relative to values of $d$ and $L$. In other words, this solution is valid when $d,L \le h$. Also, as written, this should only be used when $n_1 < n_2$.

\begin{figure}[h]
   \centering
   \includegraphics[width=3.5in]{ReflectionRefraction/Figures/refractionraysplane}
   \caption{Refraction through a plane, $n_1 = 1$, $n_2 = 2$.}
\end{figure}
\subsubsection{Routine}

The routine \texttt{refractionPlane} takes as input the coordinates of $\bb{r}_1$, $\bb{r}_2$, $z_o$, $n_1$ and $n_2$, and returns the coordinates of the refraction point $\bb{r}$ on the interface, the angles $\theta_1$, $\theta_2$, and lengths of the vectors to the refraction point, $l_1$ and $l_2$. The quartic solution is used by default. Use the optional string switch \texttt{'it'} for the iterative solution. For the latter, the routine estimates the required number of iterations by testing a point with the largest free-space $\theta_1$, then applies that number of iterations to all other points in parallel.  The routine is vectorized to take one point, $\bb{r}_1$, while the coordinates of subsurface points, $\bb{r}_2$, can be arrays of any size (all the same size). Outputs are the same size as the $\bb{r}_2$ coordinates. This assumes $n_1$ and $n_2$ are real. The case when $\bb{r}_2$ is at or above the boundary is handled separately.

{\footnotesize
\VerbatimInput{\code/ReflectionRefraction/refractionPlane.m}
}

\section{Refraction Point - Circular Interface}
We provide two solutions for the refraction point on circular interface. The first is a semi-analytical solution in terms of the roots of a sixth degree polynomial. The second is a vectorized iterative solution that is valid for weak refraction.

\addtocontents{toc}{\protect\setcounter{tocdepth}{1}}
\subsection{Refraction Point - Circular Interface - Analytical}
\addtocontents{toc}{\protect\setcounter{tocdepth}{2}}

\label{sec:refractdir}
A semi-analytic solution for the refraction point on a circular interface is derived for an arbitrary interior and exterior point. This is based on satisfying Snell's law at the interface. Similar to the quartic solution for the refraction point on a flat surface, the solution is found in terms of the roots of a sixth degree polynomial. An alternative approach is to apply the principle of least action to the total electrical path length of the rays. Both approaches give the same result.

The geometry is shown in Figure \ref{refractthroughcircle}. Let a circle be centered at the origin with radius $r$. The indices of refraction outside and inside the circle are $n_1$ and $n_2$, respectively. Let the exterior and interior points be, $\bb{r}_1$, $\bb{r}_2$, and the refraction point on the circle, $\bb{r}$.  Arbitrary  exterior/interior points are rotated so that the exterior point is aligned with the $y$ axis. Once the refraction point is found in this geometry, it is rotated back. Start with
\ea{\bb{r}_1 &=& y_1 \hat{y} \\
\bb{r}_2 &=& x_2 \hat{x} + y_2 \hat{y} \\
\bb{r} &=& r_x \hat{x} + r_y \hat{y} }

\noindent where $y_1 > r$ and $r^2 = r_x^2 + r_y^2$.  Also, define the vectors $\bb{v}_1 = \bb{r}_1 - \bb{r}$ and $\bb{v}_2 = \bb{r}_2 - \bb{r}$.  The sine of the incident and transmission angles at the refraction point can be written as the following cross products:
\ea{\sin\theta_1 \hat{z} &=& \dfrac{\bb{r} \times \bb{v}_1}{\vert \bb{r} \vert \vert \bb{v}_1\vert } \\
\sin\theta_2 \hat{z} &=& \dfrac{(-\bb{r}) \times \bb{v}_2}{\vert \bb{r} \vert \vert \bb{v}_2\vert }  }

\begin{figure}[h]
   \centering
   \includegraphics[width=2.5in]{ReflectionRefraction/Figures/refractionCircle}
   \caption{Geometry for refraction through a circle. }
   \label{refractthroughcircle}
\end{figure}

Equating through Snell's law, expanding the cross products, and squaring both sides, we get
\eq{\dfrac{r_x^2 y_1^2}{r_x^2 + (y_1-r_y)^2} = \left(\dfrac{n_2}{n_1}\right)^2 \dfrac{(r_xy_2 - r_y x_2)^2}{(x_2 - r_x)^2 + (y_2-r_y)^2}}

Next, expand the squares, substitute $r_y^2 = r^2 - r_x^2$, and collect terms to get

%\eq{\dfrac{r_x^2 y_1^2}{r_x^2 + r_y^2 - 2r_y y_1 + y_1^2} = \left(\dfrac{n_2}{n_1}\right)^2 \dfrac{(r_xy_2 - r_y x_2)^2}{x_2^2 - 2x_2r_x + r_x^2 + r_y^2 - 2r_y y_2 + y_2^2}}
%
%\eq{\dfrac{(n_1^2/n_2^2)  y_1^2r_x^2 }{r^2 - 2r_y y_1 + y_1^2} =   \dfrac{ r_x^2y_2^2 -2r_y x_2r_xy_2 +  (r^2 - r_x^2) x_2^2}{x_2^2 - 2x_2r_x + r^2 - 2r_y y_2 + y_2^2}}
%
%\eq{\dfrac{(n_1^2/n_2^2)  y_1^2r_x^2}{r^2+ y_1^2 - 2 y_1r_y } =   \dfrac{ (y_2^2 -  x_2^2)r_x^2 -2 x_2y_2r_yr_x +  r^2x_2^2  }{x_2^2 + y_2^2+ r^2 - 2x_2 r_x  - 2y_2 r_y }}

\eq{\dfrac{c_1 r_x^2}{c_2  + c_3 r_y } =   \dfrac{ c_4 r_x^2 + c_5 r_yr_x +  c_6 }{c_7 + c_8 r_x  + c_9 r_y }}

\noindent where $c_n = \left\{(n_1^2/n_2^2)  y_1^2, r^2+ y_1^2, - 2y_1, y_2^2 -  x_2^2, -2 x_2y_2, r^2x_2^2, x_2^2 + y_2^2+ r^2, - 2x_2, - 2y_2\right\}, n = 1... 9$. To proceed, multiply through by the denominators, substitute $r_y^2$ again where it occurs, and collect $r_y$ on one side of the equation, then
\eq{b_1 r_x^3 + b_2 r_x^2  + b_3 r_x +b_4 = (b_5 r_x^2 + b_6 r_x + b_7 )r_y}

\noindent where $b_n = \left\{c_3 c_5 + c_1 c_8,  c_1 c_7 - c_2 c_4, -c_3 c_5 r^2, -c_2c_6, c_3c_4 - c_1 c_9, c_2c_5, c_3c_6\right\}, n = 1...7$.  Squaring both sides once more, substituting $r_y^2$ a third time, this can be written as a sixth degree polynomial in $r_x$ as
\eq{\sum_{n=0}^6 a_n r_x^n = 0}
\ea{a_6 &=& b_1^2 + b_5^2  \\
a_5 &=& 2 b_1 b_2 + 2 b_5 b_6  \\
a_4 &=&  b_2^2 - b_5^2 r^2 + 2 b_7 b_5 + b_6^2 + 2 b_1 b_3  \\
a_3 &=& - 2 b_5 b_6 r^2 + 2 b_1 b_4 + 2 b_2 b_3 + 2 b_6 b_7 \\
a_2 &=& b_3^2 - b_6^2 r^2 + b_7^2 - 2 b_5 b_7 r^2 + 2 b_2 b_4 \\
a_1 &=& - 2 b_6 b_7 r^2 + 2 b_3 b_4 \\
a_0 &=& b_4^2 - b_7^2 r^2 }

%\ea{c_1 &=& (n_1^2/n_2^2)  y_1^2  \\
%c_2 &=&  r^2+ y_1^2  \\
%c_3 &=& - 2y_1 \\
%c_4&=& y_2^2 -  x_2^2 \\
%c_5&=& -2 x_2y_2 \\
%c_6 &=& r^2x_2^2  \\
%c_7 &=& x_2^2 + y_2^2+ r^2 \\
%c_8 &=& - 2x_2 \\
%c_9 &=& - 2y_2}
%\ea{b_1 &=& c_3 c_5 + c_1 c_8  \\
%b_2 &=&  c_1 c_7 - c_2 c_4  \\
%b_3 &=& -c_3 c_5 r^2 \\
%b_4&=& -c_2c_6 \\
%b_5 &=& c_3c_4 - c_1 c_9 \\
%b_6 &=& c_2c_5  \\
%b_7 &=& c_3c_6 }

%\ea{a_6 &=& b_1^2 + b_5^2  \\
%a_5 &=& 2 b_1 b_2 + 2 b_5 b_6  \\
%a_4 &=&  b_2^2 - b_5^2 r^2 + 2 b_7 b_5 + b_6^2 + 2 b_1 b_3  \\
%a_3 &=& - 2 b_5 b_6 r^2 + 2 b_1 b_4 + 2 b_2 b_3 + 2 b_6 b_7 \\
%a_2 &=& b_3^2 - b_6^2 r^2 + b_7^2 - 2 b_5 b_7 r^2 + 2 b_2 b_4 \\
%a_1 &=& - 2 b_6 b_7 r^2 + 2 b_3 b_4 \\
%a_0 &=& b_4^2 - b_7^2 r^2 }

Once the roots are computed, they are sifted to find the valid solution, because squaring twice created extraneous solutions. Roots with a non-zero imaginary part are rejected. Then $r_y$ is computed and only solutions that satisfy Snell's law at the boundary are kept. In the event of a tie, the refraction point that gives the smallest electrical distance is kept, consistent with the principle of least action. Finally, we check that $\theta_1 < \pi/2$ to make sure the ray does not pass through the circle twice (this angle is computed with the dot product of $\bb{r}$ and $\bb{v}_1$ to safely handle $\pm \hat{x}$ solutions).

\begin{figure}[h]
   \centering
   \includegraphics[width=3.5in]{ReflectionRefraction/Figures/refractionrayscircle}
   \caption{Refraction into a circle between two points. Exterior index of refraction $n_1 = 1$, interior index of refraction $n_2 = 2$, and radius $r = 1$. }
\end{figure}

The routine \texttt{refractionCircle} takes as input the coordinates of an arbitrary exterior point, $\bb{r}_1$, an arbitrary interior point, $\bb{r}_2$, index of refractions $n_1$, $n_2$ and radius $r$, and returns the coordinates of the refraction point on the circle, the incident and transmission angles $\theta_1$ and $\theta_2$ in radians, and the magnitudes of the vectors $\bb{v}_1$ and $\bb{v}_2$. It first rotates the points to align them with the $y$ axis. It returns \texttt{nan} if no solution exists. It also assumes $n_1$ and $n_2$ are real.  Because this relies on \texttt{roots} it is not vectorized.

{\footnotesize
\VerbatimInput{\code/ReflectionRefraction/refractionCircle.m}
}

%
%\newpage
%
%\section{Refraction Point - Circular Interface}
%
%\ea{\bb{r}_1 &=& x_1 \hat{x} + y_1 \hat{y} \\
%\bb{r}_2 &=& x_2 \hat{x} + y_2 \hat{y} \\
%\bb{r} &=& r_x \hat{x} + r_y \hat{y} }
%
%\noindent where $r^2 = r_x^2 + r_y^2$.  Define the vectors
%\ea{\bb{v}_1 &=& \bb{r}_1 - \bb{r} \\
%\bb{v}_2 &=& \bb{r}_2 - \bb{r} }
%
%The total electrical path length is
%\eq{L = n_1 v_1 + n_2 v_2}
%
%We want the point $(r_x,r_y)$ that gives the minimum of $L$.  Writing out the magnitudes of the vectors, substituting $r_y = \sqrt{r^2 + r_x^2}$, differentiating with respect to $r_x$, setting the derivative to zero, moving one term to the other side of the equation, squaring both sides, and collecting terms, one can arrive at
%
%\eq{\dfrac{ c_1 r_x^2 + c_2 r_x \sqrt{r^2 - r_x^2}  + c_3}{c_4 r_x + c_5 \sqrt{r^2 - r_x^2} + c_6} = \dfrac{c_7 r_x^2 + c_8 r_x   \sqrt{r^2 - r_x^2}+ c_9}{c_{10} r_x + c_{11} \sqrt{r^2 - r_x^2} + c_{12}}}
%
%\ea{c_1 &=& n_1^2 x_1^2 - n_1^2 y_1^2 \\
%c_2 &=& 2 n_1^2 x_1 y_1 \\
%c_3 &=& -n_1^2 r^2 x_1^2 \\
%c_4 &=& 2 x_1 \\
%c_5 &=& 2 y_1 \\
%c_6 &=& -r^2 - x_1^2 - y_1^2 \\
%}
%
%\noindent where $c_7...c_{12}$ are the same as $c_1...c_6$ except subscript 1 is changed to 2.  Next, multiplying through by the denominators, simplifying, and collecting factors of $r_x$ and $\sqrt{r^2 - r_x^2}$, one can get
%\eq{b_1 r_x^3 + b_2 r_x^2 + b_3 r_x + b_4 =
%(b_5 r_x^2  + b_6 r_x + b_7) \sqrt{r^2 - r_x^2}}
%
%\ea{b_1 &=& c_1 c_{10} - c_4 c_7 - c_2 c_{11} + c_5 c_8 \\
%b_2 &=& c_1 c_{12} - c_6 c_7\\
%b_3 &=& c_3 c_{10} - c_4 c_9 + (c_2 c_{11} - c_5 c_8) r^2\\
%b_4 &=& c_3 c_{12} - c_6 c_9\\
%b_5 &=& c_4 c_8 + c_5 c_7 - c_1 c_{11} - c_2 c_{10}\\
%b_6 &=& c_6 c_8 - c_2 c_{12}\\
%b_7 &=& c_5 c_9 - c_3 c_{11}}
%
%Finally, squaring both sides again and collecting powers of $r_x$, this can be written as a sixth degree polynomial as
%
%\eq{\sum_{n=0}^6 a_n r_x^n = 0}
%\ea{a_6 &=& b_1^2 + b_5^2  \\
%a_5 &=& 2 (b_1 b_2 + b_5 b_6)\\
%a_4 &=& b_2^2 - b_5^2 r^2 + 2 b_7 b_5 + b_6^2 + 2 b_1 b_3\\
%a_3 &=& 2 (-b_5 b_6 r^2 + b_1 b_4 + b_2 b_3 + b_6 b_7)\\
%a_2 &=& 2 b_2 b_4 - r^2 (b_6^2 + 2 b_5 b_7) + b_3^2 + b_7^2\\
%a_1 &=& 2 (-b_6 b_7 r^2 + b_3 b_4)\\
%a_0 &=& b_4^2 - b_7^2 r^2 }
%
%(This was all done with a symbolic tool of course). Once the roots are found, they need to be sifted to find the valid solution, because the effect of squaring twice has introduced extraneous solutions. Roots with an imaginary part are rejected. Then $r_y$ is computed and then $\sin(\theta_1)$ and $\sin(\theta_2)$ are computed from the cross products of the vectors. Only solutions that satisfy Snell's law are kept. In the event of a tie, the refraction point that gives the smallest electrical distance is kept, consistent with the principle of least action. Finally, we check that $\theta_1 < \pi/2$ to make sure the ray does not pass through the circle twice, where the angle is computed with the dot product of $\bb{r}$ and $\bb{v}_1$ to safely handle $\pm$ coordinate solutions.

\addtocontents{toc}{\protect\setcounter{tocdepth}{1}}
\subsection{Refraction Point - Circular Interface - Iterative}
\addtocontents{toc}{\protect\setcounter{tocdepth}{2}}

\label{sec:refractit}
While the solution for refraction through a circular interface in the previous section is exact, the code could not be vectorized because it required the roots of a sixth degree polynomial. However, like the planar interface, a vectorized iterative solution can be derived. Its validity is restricted to mild refraction and small angles but it is lightning fast when it works. The primary application for this is subsurface SAR focusing in, for instance, radar sounding through ice at planetary bodies, where the body curvature needs to be included in the refraction for a large number of subsurface focal points.

Using the geometry in Figure \ref{refractthroughcircle}, define:
\ea{h &=& y_1 - r \\
d &=& r - y_2}

Using the fact that
\eq{\sin\theta_1 = \dfrac{r + h}{r}\sin\theta_l}

we can build an iteration as
\begin{eqnarray}
\textrm{Initialize:} \quad \quad \theta_l &\leftarrow &\arctan\left( \dfrac{x_2}{h + d}\right) \ \nonumber \\
\textrm{Iterate:} \quad \quad\theta_1 &\leftarrow &\arcsin\left( \dfrac{r + h}{r}\sin\theta_l \right) \nonumber  \\
\theta_2  &\leftarrow & \arcsin\left( \dfrac{n_2}{n_1} \sin\theta_1 \right) \\
\theta_p  &\leftarrow & \theta_2 - (\theta_1 - \theta_l) \\
(r_x, r_y) &\leftarrow & \textrm{ComputeIntersection}(\theta_p) \\
\theta_l  &\leftarrow &  \arctan\left( \dfrac{r_x}{h + r  - r_y}\right) \nonumber
\end{eqnarray}

\noindent where $\theta_p$ is the angle from the perpendicular at the refraction point to the interior point, and $\textrm{ComputeIntersection}(\theta_p)$ computes the point at which a line from the interior point with slope corresponding to $\theta_p$ intersects the circle.

This iteration starts with the straight ray approximation for the look angle $\theta_l$ from which it computes the incident angle on the circle, $\theta_1$. With this, the transmission angle, $\theta_2$, is computed via Snell's law. Next, $\theta_p$ is the portion of $\theta_2$ relative to the perpendicular.  Using this, we project a line from the interior point to intersect the circle. The slope of the line corresponds to the angle $\theta_p$. With the new point on the circle, the look angle is computed again and the process repeats.  This is again a Newton iteration of the transcendental solution for one of the variables. As before, the solution is not the most robust and should be used when the refraction is mild.

\begin{figure}[h]
   \centering
   \subfigure{\includegraphics[width=2in]{ReflectionRefraction/Figures/refractioncirclethetap}}
   \subfigure{\includegraphics[width=3in]{ReflectionRefraction/Figures/refractionrayscircleiterative}}
   \caption{Left: Angles at the refraction point for iterative solution. Right:Refraction into a circle between multiple points using the vectorized iterative solution. Exterior index of refraction $n_1 = 1$, interior index of refraction $n_2 = 2$, and radius $r = 1$.  }
\end{figure}

%
%\begin{figure}[h]
%   \centering
%   \includegraphics[width=2in]{ReflectionRefraction/Figures/refractioncirclethetap}
%   \caption{Angles at the refraction point for iterative solution.}
%\end{figure}
%
%
%\begin{figure}[h]
%   \centering
%   \includegraphics[width=3in]{ReflectionRefraction/Figures/refractionrayscircleiterative}
%   \caption{Refraction into a circle between multiple points using the vectorized iterative solution. Exterior index of refraction $n_1 = 1$, interior index of refraction $n_2 = 2$, and radius $r = 1$. }
%\end{figure}

\clearpage
\paragraph{Intersection of Circle and Line}

Given a line $y = mx + b$ and a circle $(x-p)^2 + (y-q)^2 = r^2$, the point of intersection $(X,Y)$ is found by substituting the first equation into the second and solving the resulting quadratic for $x$:
\begin{eqnarray}
A &=& m^2 + 1 \nonumber \\
B &=& 2(mb - mq - p) \nonumber \\
C &=& q^2 - r^2 + p^2 - 2bq +b^2 \nonumber \\
X &=& \dfrac{-B \pm \sqrt{B^2 - 4AC}}{2A} \nonumber \\
Y &=& mX + b \nonumber
\end{eqnarray}

\paragraph{ComputeIntersection($\theta_p$)}
Let the center of the circle be the origin $(p,q)=(0,0)$. The line with slope corresponding to angle $\theta_p$ passing though the interior point at $(x_2, y_2)$ is given by
\eq{y = -\cot\theta_p(x - x_2) + y_2  }

The parameters used in the equations above for the intersection of the line and circle are therefore
\begin{eqnarray}
m &=& -\cot \theta_p \nonumber \\
b &=& x_2\cot\theta_p + y_2 \nonumber \\
A &=& m^2 + 1 \nonumber \\
B &=& 2mb \nonumber \\
C &=& -r^2 + b^2 \nonumber
\end{eqnarray}

For this problem, the choice of quadratic root is determined by the sign of $x_2$. Furthermore, if the $y$ coordinate of the refraction point is negative at any point in the iteration, then the other root is chosen. Points that are collinear with the origin and the source, where there is no refraction, need to be handled separately. Finally, in this geometry, the look angle is given by
\eq{\tan\theta_l = \dfrac{r_x}{h + r - r_y} }

The routine \texttt{refractionCircleIterative} is a vectorized implementation of the routine above. It takes an input pairs of $\bb{r}_1$, $\bb{r}_2$, the index of refraction $n_1$, $n_2$, radius $r$, and number of iterations, \texttt{nit}, and returns the refraction points, $\bb{r}$, incident and transmission angles $\theta_1$, $\theta_2$, $\theta_l$, and the lengths of the vectors $\bb{v}_1$ and $\bb{v}_2$. The point arrays can be any size. This assumes $n_1$ and $n_2$ are real. This should only be used in its present form for large radii, and depths that are less than or equal to the heights. Set \texttt{nit} to be 10 or larger, it defaults to 40. The algorithm rotates points to align the exterior points to the $y$ axis, to be consistent with the solution as written, then rotates them back.

{\footnotesize
\VerbatimInput{\code/ReflectionRefraction/refractionCircleIterative.m}
}

\section{Refraction Point - Sphere}

We can adapt the solution for the refraction point over a circle to that for an arbitrary interior and exterior points over a sphere.  Let $\bb{r}_1$ be the exterior point, $\bb{r}_2$ be the interior point, and $\bb{r}$ be the point of refraction on the sphere.  The sphere has radius $r$ centered at the origin.  Define a new coordinate system $[\hat{u}, \hat{v}, \hat{w}]$ in the $\bb{r}_1$-$\bb{r}_2$ plane such that
\ea{\hat{v} &=& \hat{r}_1  \\
\hat{w} &=& \dfrac{\bb{r}_2 \times \hat{v}}{\vert \bb{r}_2 \times \hat{v} \vert }  \\
\hat{u} &=& \hat{v} \times \hat{w}
}

The exterior and interior points are represented in this frame as
\ea{\bb{r}_1' &=& \vert \bb{r}_1 \vert \hat{v}  \\
\bb{r}_2' &=& \left( \bb{r}_2 \cdot \hat{u}  \right)\hat{u}  + \left( \bb{r}_2 \cdot \hat{v}  \right)\hat{v}   }

The coordinates of $\bb{r}_1'$ and $\bb{r}_2'$ can now be used as inputs to either of the two circular refraction routines from Section \ref{sec:refractdir} or \ref{sec:refractit}. The magnitude of $\bb{r}_1'$ in the $\hat{v}$ direction corresponds to the $y$-coordinate of the exterior point in both routines, while $\left( \bb{r}_2 \cdot \hat{u}  \right) $ and $\left( \bb{r}_2 \cdot \hat{v}  \right)$ correspond to the $x$ and $y$ coordinates, respectively, of the interior point in both routines. Once solved, the $x$ and $y$ coordinates of the refraction point on the 2D circle, $r_x$ and $r_y$, are mapped back to the original 3D frame as
\eq{ \bb{r} = r_x \hat{u} + r_y \hat{v}}

The routine \texttt{refractionSphere} computes the refraction through a sphere that is centered at the origin. It takes as input the Cartesian coordinates of pairs of exterior and interior points of any array size, the index of refraction for the exterior and interior, $n_1$, $n_2$, respectively, and the radius of the sphere $r$. The outputs are the coordinates of the refraction point on the sphere, incident angle, transmission angle, and look angle which measured from the radial line of the exterior point, as well as the two path lengths inside and outside the sphere. The routine transforms the input coordinates using the mapping above in order to use the circle refraction algorithms, \texttt{refractionCircle} or \texttt{refractionCircleIterative}. The routine defaults to the direct circle refraction solution which loops over each pair of points. The direction solution can also be chosen using the string switch \texttt{'dir'}. Use string switch \texttt{'it'} to select the vectorized iterative circle refraction solution which can also take an optional input argument for the number of iterations.

\begin{figure}[H]
   \centering
   \includegraphics[width=4in]{ReflectionRefraction/Figures/refractionspheredirect}
   \caption{Refraction into a sphere between multiple points. Exterior index of refraction $n_1 = 1$, interior index of refraction $n_2 = 2$, and sphere radius $r = 1$. }
\end{figure}

{\footnotesize
\VerbatimInput{\code/ReflectionRefraction/refractionSphere.m}
}


\lab{Algorithms}{GMRES}{GMRES}
\label{lab:GMRES}
\objective{In this lab we will learn how to use the GMRES algorithm.}

The GMRES (``Generalized Minimal Residuals") algorithm solves linear systems of the form $Ax=b$ for nonsymmetric matrices that are so
large that either they cannot be stored in memory, or direct methods cannot run to completion in a reasonable time.
It is an iterative method that relies on Krylov subspaces to reduce a high-dimensional problem to a sequence of smaller dimensional
problems.

\section*{The Arnoldi Iteration and Approximate Solutions}
The basic idea of GMRES is as follows.
Let $A$ be an $m\times m$ matrix (real or complex), where $m$ is very large,
and let $b \in \mathbb{F}^m$ ($\mathbb{F}$ may either be the real or complex numbers).
Let $\mathcal{K}_n$ denote the order-$n$ Krylov subspace generated by $A$ and $b$.
In each iteration, we consider the least squares problem
\begin{equation}
\underset{x \in \mathcal{K}_n}{\text{minimize}}\qquad \|b-Ax\|_2.
\label{eq:GMRES_lstsq1}
\end{equation}
Now if $x \in K_n$, then $x$ can be expressed as a linear combination of basis vectors $b, Ab, \ldots, A^{n-1}b$, i.e.
\[
x = y_1b + y_2Ab + \cdots + y_nA^{n-1}b.
\]
If we let $K_n$ be the matrix whose columns are $b, Ab, A^{2}b, \cdots, A^{n-1}b$, then we can write this simply as
$x = K_n y$.
Then the solution of the least squares problem is the vector $K_{n}y$ such that $\|b-A K_{n}y\|_2$ is minimized.
\begin{figure}
\includegraphics[width=\textwidth]{LeastSquares}
\caption{GMRES involves solving the least squares problem repeatedly.}
\end{figure}

The major drawback of this approach is that it relies on the matrix $K_n$, which tends to be ill-conditioned due to its columns
being far from orthogonal (as discussed in Lab \ref{lab:kry_arnoldi}).
%To see this, suppose there is an eigenbasis for $A$ with associated eigenvalues, and suppose $\lambda$ is the largest eigenvalue.
%Then $\lambda^n$ is an eigenvalue of $A^n$.
%Since $\lambda$ is bigger than the other eigenvalues of $A$, $\lambda^n$ may be much, much bigger than the other eigenvalues of $A^n$,
%which are simply the eigenvalues of $A$ raised to the $n$th power.
%Thus the eigenvector associated with $\lambda$ often begins to dominate as we progress, with the end result being that the columns of
%$K_n$ become nearly linearly dependent.
The easiest fix in this situation is to use the Arnoldi iteration so that we have an orthonormal basis for $\mathcal{K}_n$ to work with.
Not only does this alleviate the problem of ill-conditioning, it also allows us to optimize in other ways, due to the special
structure of the matrices produced.

Let $q_1,\ldots, q_n$ be the orthonormal basis for $\mathcal{K}_n$ obtained by the Arnoldi iteration, and let $Q_n$ be the matrix
having these vectors as its columns.
Recall that $q_1 = b/\|b\|_2$.
Finally, let $H_n$ be the $(n+1)\times n$ upper Hessenberg matrix generated by the Arnoldi iteration, and let $e_1=(1,0,\cdots,0)$.

In this orthonormal basis, $x \in \mathcal{K}_n$ implies that there is some vector $y$ such that $x = Q_n y$.
Further, it is not hard to check that the matrices generated by the Arnoldi iteration satisfy the equation
\[
AQ_n = Q_{n+1}H_n.
\]
We also have the identity
\[
b = \|b\|_2q_1 = \|b\|_2Q_{n+1}e_1.
\]
Putting all of this together, we can rewrite the objective function of our least squares problem as follows:
\begin{align*}
\|b - Ax\|_2 &= \|Ax - b\|_2\\
&= \|AQ_ny - b\|_2\\
&= \|Q_{n+1}H_ny - \left(\|b\|_2Q_{n+1}e_1\right)\|_2\\
&= \|Q_{n+1}\left(H_n y - \|b\|_2e_1\right)\|_2.
\end{align*}

The matrix $Q_{n+1}$ has orthonormal columns, but it does not have enough columns to be a unitary matrix.
Let us extend the set $q_1,\ldots, q_{n+1}$ to an orthonormal basis $q_1,\ldots,q_{n+1},q_{n+2},\ldots,q_m$
of our space, and let $Q'$ be the matrix whose columns are equal to these vectors.
Then $Q'$ is now a unitary matrix, and hence preserves the norm, i.e. $\|Q'z\|_2 = \|z\|_2$ for all $z \in \mathbb{F}^m$.
Given $x \in \mathbb{F}^{n+1}$, if we define $x' \in \mathbb{F}^m$ to be
\[
x' =
\begin{bmatrix}
  x\\
  0\\
  \vdots\\
  0
\end{bmatrix},
\]
then you can easily check that
\[
Q'x' = Q_{n+1}x.
\]
From this, we deduce that
\begin{align*}
\|Q_{n+1}x\|_2 &= \|Q'x'\|_2\\
& = \|x'\|_2\\
&= \|x\|_2.
\end{align*}
Hence, we conclude that
\[
\|Q_{n+1}\left(H_n y - \|b\|_2e_1\right)\|_2 = \|H_n y - \|b\|_2e_1\|_2.
\]
Thus, the least squares problem given by \ref{eq:GMRES_lstsq1} is equivalent to the problem
\begin{equation}
\underset{y \in \mathbb{F}^n}{\text{minimize}}\qquad \|H_n y - \|b\|_2e_1\|_2.
\label{eq:GMRES_lstsq2}
\end{equation}
If $y$ is the solution to this problem, then the solution to \ref{eq:GMRES_lstsq1}, and hence an approximate
solution to $Ax = b$ is given by $x=Q_n y$.

We can measure how good our approximate solution is by considering the residual, which we define to be
\[
\frac{\|Ax-b\|_2}{\|b\|_2}.
\]
We can express this residual in terms of $y$ as follows:
\begin{equation}
\frac{\|H_n y - \|b\|_2e_1\|_2}{\|b\|_2}.
\label{eq:GMRES_residual}
\end{equation}
\section*{The GMRES Algorithm}
To summarize the discussion in the previous section, the GMRES algorithm combines the Arnoldi iteration with linear least squares,
generating successive approximations to the solution of $Ax = b$.
We fully present GMRES in Algorithm \ref{alg:gmres}.

\begin{algorithm}
\begin{algorithmic}[1]
\Procedure{gmres}{$Amul, b, k=100, tol=1E-8$}
	\State $m \gets b.size$						\Comment{Some initialization steps}
	\State $Q \gets \text{empty}\left(m, k+1\right)$
	\State $H \gets \text{zeros}\left(k+1, k\right)$
	\State $q_0 = b/\|b\|_2$
    \For{$n=1,2,\ldots, k$}
        \State Set entries of $Q$ and $H$ as in Arnoldi iteration.
        \State Calculate least squares solution $y$ of \ref{eq:GMRES_lstsq2}.
        \State Calculate the residual $r$ given by Equation \ref{eq:GMRES_residual}.
        \If{$r < tol$}
            \State \pseudoli{return} $Q_ny, \,\, r$
        \EndIf
    \EndFor
    \State \pseudoli{return} $Q_ny, \,\, r$
\EndProcedure
\end{algorithmic}
\caption{GMRES}
\label{alg:gmres}
\end{algorithm}

%\begin{warn}
%The Python function \li{linalg.lstsq} solves a least squares problem, returning not only the vector, $y$, but also the residual,
%the rank of the matrix, and the singular values.
%Be careful when you write your code that you access the correct results and don't just assume that \li{linalg.lstsq} returns the
%vector that you want.
%The least squares solver also returns a residual, but it's not the number we reference in this book, so be sure to take the square
%root of the residual reported by the solver and divide by $\norm{b}$ to get $res=\norm{Ax-b}/\norm{b}$.
%\end{warn}

\begin{problem}
Implement the GMRES algorithm as presented above. For simplicity, assume that we are only dealing with real-valued matrices and vectors.
Write a function \li{gmres} which accepts a function handle $Amul$ that computes the matrix-vector multiplication $Ax$ for any $x$,
a vector $b$, the maximum number of iterations to perform, and an error tolerance $tol$.
Use the GMRES algorithm to compute an approximate solution to the system of equations $Ax = b$, and return this solution, as well as
the residual.

Assume that the matrix is invertible, and ignore the possibility of dividing by zero for now, as this will occur very infrequently.
You may use the built-in least squares solver in \li{scipy.linalg} to solve the least squares problem, but check the documentation
for its correct use.

The following function generates an $n \times n$ array with randomly distributed real eigenvalues between $-1$ and $1$.
\begin{lstlisting}
import numpy as np
from numpy.random import rand
from scipy import linalg as la
def rand_eigs(n):
    """ Make an nxn array with random eigenvalues between -1 and 1. """
    A = rand(n) * 2 - 1
    X = rand(n, n)
    return la.lu_solve(la.lu_factor(X), (X * A).T, trans=1).T
\end{lstlisting}

Test your algorithm on a $100\times 100$ matrix with random eigenvalues and a random vector $b$ and $tol=10^{-3}$.
Allow only a maximum of 90 iterations.
Don't expect convergence before 90 iterations in this case, for reasons that will be explained shortly.
\label{prob:MyGMRES}
\end{problem}

\section*{Convergence of GMRES}
At the $n$-th iteration, GMRES computes the best approximate solution to $Ax = b$ within the order-$n$ Krylov subspace
$\mathcal{K}_n$.
If $A$ is full rank, then the order-$m$ Krylov subspace is all of  $\mathbb{F}^m$.
Therefore, we are guaranteed an exact solution, up to rounding error, after $m$ iterations.
This is not very useful, however, because $m$ is prohibitively large in practice.
Instead, we hope to converge to a close approximate solution in $n \ll m$ steps.

The convergence of GMRES for reasonably small $n$ depends highly on the eigenvalues of $A$, and we expect rapid convergence whenever
the eigenvalues are clumped together in several places in the complex plane and if the eigenvalues are not too close to zero.
The worst case is when the eigenvalues are uniformly distributed around the origin.
This is, in fact, the reason that the algorithm failed to converge even after ninety iterations in problem 1.
If you come across a problem with such poorly-distributed eigenvalues, it is best to look for another algorithm, such as conjugate
gradient applied to the normal equations (CGN).

\begin{problem}
Instead of testing GMRES on completely random matrices, test it on matrices with ``nice,'' clumped eigenvalues.
Here's one idea of how to design such a matrix for testing.
\begin{lstlisting}
# Generate a random matrix with nice eigenvalues
def rand_clumps(m, q, r):
    """ Generate a random mxm matrix with q distinct groups
    of eigenvalues around different natural numbers. """
    A = rand(m, m)
    Lambda = np.repeat(np.arange(2, q+2), m / q + 1)[:m]
    variation = (rand(m) - .5) * (2 * r)
    Lambda += variation
    Q = la.qr(A)[0]
    return (Q * Lambda).dot(Q.T)
\end{lstlisting}
Constructing a matrix this way often takes much longer than actually running GMRES to get the solution, especially once we
introduce restarts later on in the lab.
Test GMRES on matrices of this kind for $100\times 100$ matrices.
Convergence should now occur in relatively few iterations.
Try out larger matrices and different values of the parameters to see how this algorithm behaves for these kinds of matrices.
\label{prob:GMRESClumps}
\end{problem}

\section*{Breakdowns in GMRES}
One of the selling points of GMRES is that it can't break down unless it has reached an exact solution.
Why is this the case?
Breakdowns can occur as we try to expand to larger Krylov subspaces.
In each iteration, we have already converted $b,Ab,A^{2}b,\cdots, A^{n-1}b$ into an orthonormal set $q_0,q_1,\cdots,q_{n-1}$.
After computing $A^nb,$ we will try to make it orthogonal to each of the $q_i$.
But what if $A^{n}b$ is already a linear combination of the $q_i$? In other words, what if $A^{n}b$ lies in $\mathcal{K}_{n-1}$?
This means that
\[
\mathcal{K}_{n-1}=\mathcal{K}_n=\mathcal{K}_{n+1}=\cdots,
\]
so we have reached an \emph{invariant subspace},
and the algorithm cannot proceed without modification.
It also means that $A^{n}b = c_0 b + c_1 A b + \cdots + c_{n-1}A^{n-1}b$.
Rearranging this, we have that $c_0 b = c_1 A b + c_2 A^{2} b + \cdots + A^{n}b$, so that $b$ lies in $\mathcal{K}_{n+1}$.
Therefore, $b$ lies in $A\mathcal{K}_n$, so the least-squares problem will actually deliver an exact solution, up to rounding errors,
in this case.
To summarize: \emph{GMRES only breaks down when it has found an exact solution}.

\begin{problem}
If necessary, update your solution to \ref{prob:MyGMRES} to deal with breakdowns.
Consider any division by zero cases in your code.  Make the necessary adjustments to avoid any such cases.
\label{prob:GMRES3}
\end{problem}

\section*{Optimizing Least-Squares for GMRES (Optional)}
The Hessenberg structure and the Krylov subspace relations enable us to save time on the least-squares part of the problem if we use QR factorization.
Observe that if $H_n$ can be factored as $Q_n R_n$, where $Q_n$ is not the same matrix as above and $R_n$ is invertible upper triangular, we may solve the least squares problem by simply solving $R_n x_n=\norm{b} Q_{n}^{H}e_1$ via back substitution.
There are two ways in which we can speed up this process.
First, we take advantage of the Hessenberg structure by using the techniques from Problem \ref{prob:givens_hessenberg} in Lab \ref{lab:givens}.
Recall that the technique in this situation was to use Givens rotations to eliminate the subdiagonal elements one at a time.
This process, which was part of a previous lab, reduces the operation count from $O(n^3)$ to $O(n^2)$.
The second speedup comes from the fact that we already know the QR factorization for $H_{n-1}$ from the previous step of the algorithm.
This means that we can simply update the QR factorization from the previous step rather than computing it all over again.
Since $H_{n}$ has only one more column and row than $H_{n-1},$ all we need to do is update the last column by performing all previous Givens rotations on just the last column of $H_n$, which requires only $O(n)$ work.
Then we perform one final Givens rotation on $H_n$ to eliminate the new subdiagonal entry which was not present in $H_{n-1}$.
Thus, the QR factorization of $H_n$ can be reduced from an $O(n^3)$ process to only $O(n)$ using these special techniques.

The back substitution necessary to solve the least squares problem can also be reduced to an operation of $O(n)$.

The speedup from $O(n^3)$ to $O(n)$ is very good, but it can only partially alleviate the problems that come with a problem that is ill-suited for GMRES.
It may still be useful because it allows us to perform more iterations in a reasonable amount of time.
In many situations, the simple technique of the next section will keep $n$ low enough that the optimizations from this section are not critical.
%
% \begin{problem}
% \label{prob:GMRES2}
% (Optional) Modify MyGMRES to incorporate these optimizations, and call this program MyGMRES1.
% Run both programs on a series of five random $100\times 100$ matrices and compare the time each requires.
% Are the gains substantial?
% Try it again matrices of size $1000\times 1000$ or larger, and see how substantial the difference becomes.
% Try the same thing using the techniqes of the next section.
% Explain why the difference in performance is less dramatic this time.
% \end{problem}

\section*{GMRES(k)}
One of the serious drawbacks of GMRES is that it requires a lot of storage.
At step $n$, we need to store $H_n$ and $Q_n$, so the storage is $O(n^2)$.
It is also true that the complexity of the each iteration increases as $n$ increases, a fact that can substantially slow down the process.
If we want the iteration to proceed over many steps, these challenges can become prohibitive, so there is a modification called GMRES(k), or GMRES with restarts, that seeks to alleviate these difficulties by restarting the algorithm, but with an improved initial guess.
It then builds the Krylov subspaces generated by this improved initial guess and repeats the process as many times as needed.
Here's the outline of the algorithm:
\begin{enumerate}
\item \emph{Set} $k$, the maximum number of iterations before memory requirements or complexity are too large
\item \emph{Set} $b$, the initial guess
\item \emph{while} $i<k$ and convergence is not yet reached and the number of times through this while loop is less than a set maximum
\subitem Run GMRES for up to $k$ iterations, starting with initial guess
\subitem If convergence has not yet occurred, \emph{set} initial guess to equal the most recent estimate of the solution
\end{enumerate}

This process keeps storage under control and ensures that each iteration stays fast, but at the cost of reliability.
Thus, when GMRES(k) does converge, it may be much quicker and require less storage than GMRES without restarts, but in situations where the true solution, $x$ is nearly orthogonal to the first $k$ Krylov subspaces, not much improvement is made at each iteration, so convergence is slow or may fail entirely.
This is, however, an important variation of GMRES and is frequently used.

\begin{problem}
Adapt your prior implementation of GMRES to include restarts.
It should also take the additional argument $k$, the number of iterations before restarting.
Test its speed on the special $1000\times 1000$ matrices we constructed to test GMRES and compare its speed to GMRES without restarts.
No general statement can be made about the comparative speed of these two algorithms.
How do they compare in the case of these matrices with $k=10$ and $A$ has dimension $1000\times 1000$?
\label{prob:GMRES3}
\end{problem}

\section*{Industrial-grade GMRES package}
In practice, of course, you will most frequently use GMRES algorithms written by specialists rather than writing your own version.
In python, one option is the function \li{scipy.sparse.linalg.gmres}.
It also includes an option to specify whether restarts are needed.

\begin{problem}
Test your GMRES algorithm against SciPy's GMRES algorithm on random $500\times 500$ matrices, and report any difference on how quickly convergence is reached.
\label{prob:GMRES4}
\end{problem}

\section*{Summary}
In this lab, we developed the mathematical basis for GMRES, practiced implementing and working with the algorithm, and explored several variations and optimizations which, when used in conjunction and in the right situations, make GMRES the go-to algorithm for many, many applications.
Avoid falling into the trap of thinking that GMRES is a universal solution to all systems of equations dilemmas, since we have already seen that even for simple random matrices, GMRES has horrendous performance.
Learning and knowing when to apply a variety of algorithms is key to effective computing with large matrices.


%!TEX root = paper.tex

The problem we will analyze in this paper is that of a billiard ball bouncing around inside a square billiard table in the absence of any external forces (e.g. gravity) and/or any dissipative forces (e.g. friction). Before delving into the analysis, we must first clarify the problem statement.

\subsection{Setup}

The ball and table will be idealized in the following manner: we will represent the ball as a point moving around in $\R^2$ and the board as the unit square $[0,1]^2$. The ball will start with an initial position $\bvec{x}_0 \in [0,1]^2$ and initial velocity $\bvec{u}_0$. Because there are no external forces and/or dissipative forces, the speed of the ball is constant and irrelevant to the problem.

Any time the position of the ball (point) coincides with the edge of the table (unit square), we will say that the ball collides with that edge of the table. During this collision, the ball is reflected off the edge of the table in such a manner that the outgoing direction vector is a reflection of the incoming direction vector across the line perpendicular to the edge of the table at the point of collision. In more technical terms, the angle of incidence is equal to the angle of reflection in all ball-table collisions. Collisions at corners of the table are undefined and so we will ignore any such trajectories that intersect corners of the table. Figure \ref{fig:collision-angle} shows the general mechanics of a collision.

\begin{figure}[H]
  \begin{center}
    \includegraphics[keepaspectratio,width=2in]{particle_collision.png}
  \end{center}
  \vspace{-.2in} % corrects bad spacing
  \caption{\label{fig:collision-angle}Mechanics of the ball colliding with a table edge.}
\end{figure}

Introducing some notation to the problem, we will label the horizontal edges of the table $h$ and the vertical edges of the table $v$. Whenever the ball collides with a horizontal edge, we will call the resulting collision an $h$ collision. Likewise, collisions with vertical edges will be denoted as $v$ collisions.

\begin{definition}
  A \emph{collision sequence} ($\alpha$) for a ball is the sequence of sides that the ball collides with ($\alpha_i \in \cbracket{v, h}$). This sequence is ordered by increasing collision time. In this paper, ball trajectories are idealized and infinite, but we will only look at finite subsequences of the infinite collision sequences formed by these trajectories. Furthermore, we will only look at finite collision sequences that start and end with h.
\end{definition}

\subsection{An Example Collision Sequence}

For the reader to better understand the basic properties of collision sequences, we will consider an example trajectory and form a collision sequence from a short segment of the trajectory. Consider a ball with initial position $\bvec{x}_0 = (0.75, 0.75)$ and initial velocity $\bvec{u}_0 = (4.6, 1)$.

\begin{figure}[H]
  \begin{center}
    \includegraphics[keepaspectratio,width=3in]{example.png}
  \end{center}
  \vspace{-.2in} % corrects bad spacing
  \caption{\label{fig:example}Example trajectory, $x_0 = (0.75, 0.75)$, $u_0 = (4.6, 1)$.}
\end{figure}

The collision sequence corresponding to this trajectory segment is the following:

\begin{equation}
	\alpha = (v, h, v, v, v, v, v, h, v, v, v, v, v, h, v, v, v)
\end{equation}


%
% setting up at Nov. 2011
%
%
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\chapter{Introduction to Hermite Polynomials}
\label{sec:hermite_polynomials}

In the mathematics, Hermite polynomials is a series of orthogonal functions
that form an orthogonal basis of the Hilbert space, it's also a complete
orthogonal function system so that we can use it to expand a lot of functions.
In quantum chemistry, the Hermite polynomials are used in the integral
calculation. Hence here a brief summary for its properties will be given.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Definition}
The Hermite polynomials are defined as:
\begin{equation}
 \label{hermite_definition_eq:1}
H_{n}(x) = (-1)^{n}e^{x^{2}}\frac{d^{n}}{dx^{n}}e^{-x^{2}}
\end{equation}
This relation could be derived from a generation function:
\begin{equation}
\label{hermite_definition_eq:2}
 e^{2xt-t^{2}} = \sum_{t=0}^{\infty}H_{n}(x)\frac{t^{n}}{n!}
\end{equation}
when we make $n$th derivatives for the $e^{2xt-t^{2}}$, $H_{n}(x)$
could be generated:
\begin{equation}
 \label{hermite_definition_eq:3}
H_{n}(x) =  \left. \frac{\partial^{n} (e^{2xt-t^{2}}) }{\partial
t^{n}}\right|_{t=0}
\end{equation}
By setting the $e^{2xt-t^{2}} = e^{-(t-x)^{2} + x^{2}}$, then through
the $n$th derivatives we can get the \ref{hermite_definition_eq:1}.

For this polynomial we have some recursive relation from its definition:
\begin{equation}
 \label{hermite_definition_eq:4}
H_{n}^{'}(x) = 2xH_{n}(x) - H_{n+1}(x)
\end{equation}
By using the relation that $\frac{\partial
e^{2xt-t^{2}} }{\partial x} = 2te^{2xt-t^{2}}$, then we can bring this form
into \ref{hermite_definition_eq:3} and expand the higher derivatives for
$2te^{2xt-t^{2}}$; we get:
\begin{equation}
 \label{hermite_definition_eq:5}
H_{n}^{'}(x) = 2nH_{n-1}(x)
\end{equation}
Therefore we arrive at some final recursive expression:
\begin{equation}
 \label{hermite_definition_eq:6}
2nH_{n-1}(x) = 2xH_{n}(x) - H_{n+1}(x)
\end{equation}
we note that such derivative relation could also be expressed in terms of
its derivatives.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Orthogonality}
%
%
%
Hermite polynomials has one important property, is that they are orthogonal with
each other by a weight function:
\begin{equation}
\label{hermite_orthogonality_eq:1}
 \int^{+\infty}_{-\infty} H_{m}(x)H_{n}(x)e^{-x^{2}} =
 2^{n}n!\delta_{mn}
\end{equation}

The orthogonality could be derived through partial integral(suppose that $n\neq m$):
\begin{align}
 \label{hermite_orthogonality_eq:2}
 \int^{+\infty}_{-\infty} H_{m}(x)H_{n}(x)e^{-x^{2}} dx &=
(-1)^{n} \int^{+\infty}_{-\infty} H_{m}(x) \frac{d^{n}}{dx^{n}}e^{-x^{2}}dx \nonumber \\
&=  (-1)^{n-1} 2m\int^{+\infty}_{-\infty} H_{m-1}(x)\frac{d^{n-1}}{dx^{n-1}}e^{-x^{2}}dx
\nonumber \\
&= (-1)^{n-m} 2^{m}m!\int^{+\infty}_{-\infty} H_{0}(x)\frac{d^{n-m}}{dx^{n-m}}e^{-x^{2}}dx
\nonumber \\
&= 0
\end{align}
Such derivation uses the relation in the \ref{hermite_definition_eq:5}.


\subsection{Sparse Contrastive PCA}

The sparse contrastive PCA (scPCA) procedure applies SPCA with minimal
modifications \edit{to a pair of target and background datasets' contrastive
covariance matrix $\mathbf{C}_\gamma$}. The numerical solution to the SPCA
criterion of Equation \eqref{eq:spca} is obtained by the following
alternating algorithm until convergence of the sparse loadings \citep{Zou2006},
\edit{where $\mathbf{A}_{p\times k}$ is initialized as the matrix of loadings
corresponding to the $k$ leading principal components of $\mathbf{C}_\gamma$:}

\textbf{For fixed} $\mathbf{A}$:
\edit{Relying on the results of Equation \eqref{eq:cov_mat_spca}, the
elastic net solution for the $j^{\text{th}}$ loading vector is
\begin{equation*}
  \beta_j^\star = \argmin_{\beta_j} \lVert \mathbf{C}_\gamma^{\frac{1}{2}}\alpha_j - \mathbf{C}_\gamma^{\frac{1}{2}}
  \beta_j \rVert_2^2 + \lambda_0 \lVert \beta_j \rVert_2^2 + \lambda_{1, j}
  \lVert \beta_j \rVert_1.
\end{equation*}}
Generally, for ease of computation, $\lambda_{1, j} = \lambda_1$, for
$j=1, \ldots, k$. The entries of the loading matrix $\mathbf{B}$ are independent of
the choice for the $\ell_2$ penalty (ridge) parameter $\lambda_0$
\citep{Zou2006}, which is used only for numerical reasons. The ridge penalty is set to zero when
$\mathbf{C}^{\frac{1}{2}}_\gamma$ is of full rank; otherwise, a small constant
value is used to remedy issues of indeterminacy that arise when fitting the
elastic net.

\textbf{For fixed} $\mathbf{B}$: Only the first term of the SPCA criterion of
Equation \eqref{eq:spca} must be minimized with respect to $\mathbf{A}$. The
solution is given by the reduced rank form of the Procrustes rotation, computed
as $\mathbf{A}^\star = \mathbf{U}\mathbf{V}^\top$ \citep{Zou2006}. The matrices
of left and right singular vectors are obtained from the following singular
value decomposition:
\begin{equation*}
  \mathbf{C}_\gamma\mathbf{B} = \mathbf{U}\mathbf{D}\mathbf{V}^\top.
\end{equation*}
Generally, $\mathbf{C}_\gamma$ is not positive-semidefinite and its square root
is undefined. Instead, a positive-semidefinite matrix
$\widetilde{\mathbf{C}}_\gamma$, approximating $\mathbf{C}_\gamma$, is used.
$\widetilde{\mathbf{C}}_\gamma$ is obtained by replacing the diagonal matrix in
the eigendecomposition of $\mathbf{C}_\gamma$ by a diagonal matrix in which
negative eigenvalues are replaced by zeros \citep{elasticnet}:
\begin{align*}
  \mathbf{C}_\gamma &= \mathbf{V}\mathbf{\Lambda}\mathbf{V}^\top \\
  \widetilde{\mathbf{C}}_\gamma &= \mathbf{V}\mathbf{D}\mathbf{V}^\top \\
\end{align*}
\begin{equation*}
\text{where } \text{D}_{ii} =
  \begin{cases}
    \Lambda_{ii}, & \text{if $\Lambda_{ii} > 0$} \\
    0, & \text{otherwise}
  \end{cases},
  \qquad
  \text{for } i = 1, \ldots, p.
\end{equation*}

Thus, the directions of variation given by the negative eigenvalues of
$\mathbf{C}_\gamma$ are discarded, as they correspond to those which are dominated by the variance in the background dataset. This procedure can be viewed as a preliminary thresholding of the eigenvectors of $\mathbf{C}_\gamma$,
where the cutoff is an additional hyperparameter corresponding to a non-negative
real number. Explicitly defining a small positive threshold may prove useful for
datasets that possess many eigenvalues near zero, which correspond to sources of
technical and biological noise remaining after the contrastive step.
Empirically, however, providing a wide range of contrastive parameters $\gamma$ has been found to have a similar effect as using
multiple cutoff values, that is, larger values of $\gamma$ naturally produce
sparser matrices $\widetilde{\mathbf{C}}_{\gamma}$.

For the purpose of contrastive analysis, a direction's importance is
characterized by its target-background variance coupling; higher target variance
and lower background variance pairs produce the best directions \citep{Abid2018}
and correspond to the largest positive eigenvalues. The elimination of directions with
negative eigenvalues therefore guarantees that the sparse contrastive PCs (scPCs) are
rotations of the target data relying on the sparse directions most
variable in the target data but least variable in the background data, making
a cutoff of zero a natural choice for the thresholding operation.

\subsection{Framework for Hyperparameter Tuning}\label{hyp_tune}

The scPCA algorithm relies on two hyperparameters: the contrastive parameter $\gamma$ and the $\ell_1$ penalty parameter $\lambda_1$. To select the optimal combination of $\gamma$ and $\lambda_1$ from a grid of \textit{a priori} specified values, we propose to cluster the $n$ observations of the target dataset based on their first $k$ scPCs, selecting as optimal the combination $\{\gamma, \lambda_1\}$ producing the ``strongest'' cluster assignments. This framework casts the selection of $\{\gamma, \lambda_1\}$ in terms of a choice of clustering algorithm, distance metric (based on $\widetilde{\mathbf{C}}_{\gamma}$), and clustering strength criterion. For ease of application, we propose to select $\{\gamma, \lambda_1\}$ by maximization of the average silhouette width over clusterings of the reduced-dimension representation of the target data. This procedure implicitly requires the choice of a clustering algorithm, such as $k$-means~\citep{kmeans}, to be applied to the representation of the data in the first $k$ scPCs. Such methods require an appropriate choice for the number of clusters, which we contend will generally not be a limiting factor in the use of scPCA. Indeed, reasonable choices for the number of clusters can often be inferred in \textit{omics} settings from sample annotation variables accompanying the data or from previously available biological knowledge. In Section~\ref{results}, we empirically demonstrate that the results of the algorithm are robust to the choice of the number of clusters. Additionally, scPCA has no particular dependence on average silhouette width as a criterion, that is, alternative criteria for assessing clustering strength could be used when appropriate. %(e.g., when the data contains sub-clusters~\citep{Liu2010}).
Naturally, this proposed hyperparameter tuning approach can be applied to cPCA by setting $\lambda_1$ to zero.

To address concerns of overfitting and to avoid discovering non-generalizable patterns from the data, we propose the use of cross-validation. For a grid of \textit{a priori} specified contrastive parameters $\gamma$ and $\ell_1$ penalty parameters $\lambda_1$, \edit{$W$-fold} cross-validation may be performed as follows:
\begin{enumerate}
  \itemsep0pt
  \item Partition each of the target and
     background datasets into \edit{$W$} roughly
     equally-sized subsets.
   \item Randomly pair each of the target \edit{$W$} subsets with one of the
     background subsets; these pairs form the fold-specific \edit{validation sets}.
   \item Iteratively perform scPCA over the
     observations of the target and
     background data not contained in the validation set (i.e., the training sets),
     for each pair of contrastive parameters and $\ell_1$ penalty parameters in
     the hyperparameter grid.
  \item Project the target validation data onto the low-dimensional space using
     the loading matrices obtained in the prior step.
  \item Compute a clustering strength criterion (e.g., average
  silhouette width) for a clustering of the target validation data with the \textit{a priori}
  specified number of clusters.
  \item Finally, compute the cross-validated average of the clustering
  strength criteria (e.g., cross-validated average of average silhouette width)
  across the validation sets for each pair of hyperparameters, selecting the
  pair that maximizes the value of the criterion.
\end{enumerate}

\edit{We note two caveats of this cross-validation framework: (1) it increases the computation time, and (2) each fold is assumed to be representative of the process which generated the data. For large datasets and say 5 to 10 folds, (2) is not a grave concern; however, as the number of samples decreases, this assumption becomes less tenable. In such a situation, the non-cross-validated framework should be used, or the number of folds reduced. Examples and results are provided in Section~\ref{cv_algo_example}.}

\subsection{Algorithm and Software Implementation}

\edit{The implementation of the scPCA algorithm is presented in
Algorithm~\ref{algo1} of the supplement. Algorithm~\ref{algo2},
also in the supplement, details the cross-validated variant.}

\edit{Regarding the computational complexity of scPCA (and cPCA), we note that
the number of observations in the target and background data are not limiting
factors. Indeed, these algorithms are applied to a $p \times p$
contrastive covariance matrix, a function of the target and background covariance
matrices. The computation time of these covariance matrices increases linearly in
the number of observations $n$ and quadratically in the number of features $p$. The
methods' computational efficiency are therefore most impacted by the number of
features and the size of the hyperparameter grid. A note on cPCA's and
scPCA's running time is provided in Section~\ref{run_time}, as is a comparison
to competing methods.}

A free and open-source software implementation of scPCA is
available in the \texttt{scPCA} package for the \texttt{R} language and
environment for statistical computing~\citep{R}. The \texttt{scPCA} package
\edit{has been released} as part of the Bioconductor Project
\citep{gentleman2004bioconductor,gentleman2006bioinformatics,huber2015orchestrating} (\url{https://bioconductor.org/packages/scPCA}).

The code and data used to generate this manuscript are publicly available on
GitHub (\url{https://github.com/PhilBoileau/EHDBDscPCA}).