%!TEX root = ../thesis.tex
%*******************************************************************************
%****************************** Third Chapter **********************************
%*******************************************************************************
\chapter{Validating Reduced Order Models}

\ifpdf
    \graphicspath{{Chapter4/Figs/Raster/}{Chapter4/Figs/PDF/}{Chapter4/Figs/}}
\else
    \graphicspath{{Chapter4/Figs/Vector/}{Chapter4/Figs/}}
\fi

\section{Summary}
This chapter focuses on validating the hierarchy of models developed in Chapter 3. In the context of this work, model validation is the process of both estimating the parameters of models in such a way that they simultaneously minimize the difference between these models and experimental data, and give reasonable uncertainty in the model output and estimated parameters.

The chapter begins with a formulation of parameter estimation as a nonlinear optimization problem, followed by a discussion of different methods of optimization, with an emphasis on Particle Swarm Optimization (PSO), a review of challenges in optimization, and a discussion about how uncertainty of parameters can be quantified through confidence regions. We wrap up the algorithmic component of our discussion by presenting a novel optimization framework that is used to simultaneously validate models, establish confidence in our results, and determine the number of experiments needed to yield a user-defined balance of fitting models with a high degree of accuracy, and giving a meaningful distribution of estimated parameters and model outputs. Subsequently, we present a two part discussion on the results of our novel algorithm when applied to our reduced comb-drive actuator in electrolyte models, and the accompanying experiments. The first part of the discussion, focuses on observing the results of our algorithm in the scenario where we are concerned only with determining which of the reduced models fit the data with the greatest degree of fidelity. In this part of the discussion, we make use of all our experiments, and comment on the simplest model that best fits the data. The second part of our discussion examines the results of our algorithm more holistically. Here, we observe the confidence regions found by our algorithms when applied to different reduced order models, at different numbers of experiments and measurements. We then comment on how the confidence regions, as well as the distribution of model outputs, change for each model as a function of the amount of data used in the fit, and at a particular confidence, $1-\alpha$. Finally, we discuss how we can use the in-depth results of our algorithm to select the correct combination of reduced order models, and amount of data that allows us to explain the behavior of the comb-drive actuator with some balance of fit to data, and an appropriate distributions of model outputs and estimated parameters.

\section{Parameter Estimation as an Optimization Problem}
The problem of estimating the parameters of a model that best fits the data can be represented as the following nonlinear optimization problem

\begin{equation} \label{orig_optim_problm}
\begin{aligned}
& \underset{x}{\text{minimize}}
& & y(\mathbf{x},\mathbf{z}) = y(\mathbf{x}) = ||\mathbf{f}(\mathbf{x},\mathbf{z}) - \mathbf{g}(\mathbf{z})||_p \\
& \text{subject to}
& & x_{i,min} \leq x_i \leq x_{i,max} , \; i = 1, \ldots, m.
\end{aligned}
\end{equation}

where $\mathbf{x}$ is the vector of tune-able parameters that the model takes as in input, $\mathbf{z}$ is the vector of inputs to the model that are not tune-able, $\mathbf{f}$ is the vector of values that are output by the model given a particular set of $\mathbf{x}$ and $\mathbf{z}$, $\mathbf{g}$ is the vector of values that are output given $\mathbf{z}$, $p$ is an integer that indicates the type of norm we are taking (eg. $p=1 \implies \textrm{ L1 norm}$), $x_i$ is the $i_{th}$ component of the vector $\mathbf{x}$, and $m$ is the number of components in that vector. This is a standard bounded nonlinear optimization problem. In our case, we bound the parameters of the model, $\mathbf{x}$, so that our optimization framework only selects physically realizable values, and we select $p=2$ so that the above objective function becomes L2 (least-squares).

\section{Optimization Techniques}
This section focuses on reviewing different optimization techniques. We discuss gradient methods, with a focus on the trust region reflective algorithm, and stochastic methods, with a focus on particle swarm optimisation.

\subsection{Gradient Methods}

Gradient methods for optimization are among the most popular in literature. These methods use the gradients, or approximate gradients, of functions with respect to their tuneable inputs in order to find the value of these inputs that minimize or maximize said function. The most common gradient optimization method is gradient descent, which has gained particular favor in the machine learning community \textbf{CITE NG}. Given the optimization problem, \ref{orig_optim_problm}, and neglecting the constraints, gradient descent involves making an initial guess at the solution, $\mathbf{x}^{0}$, and iterating on that guess using,

\begin{equation} \label{grad_descent}
    \mathbf{x}^{n+1} = \mathbf{x}^{n} - \alpha g, \quad g = \nabla_\mathbf{x} y(\mathbf{x}^{n},\mathbf{z})
\end{equation}
where $\alpha$ is the learning rate, $x^{n}$ is the guess of the solution at the $nth$ iteration of gradient descent, $x^{n+1}$ is the guess at the $(n+1)th$ iteration based on the gradient of the objective function, and $\nabla_\mathbf{x} y(\mathbf{x}^{n},\mathbf{z})$ is the numeric or analytic gradient of $y$ with respect $\mathbf{x}$, evaluated at $\mathbf{x}^{n}$. While gradient descent is a powerful optimization method, it can be very expensive, especially when optimizing a nonlinear function. In particular, while the gradient moves in the direction of greatest change in the function, it does so at an arbitrary distance, making it more prone to "overreacting" to local curvature in a nonlinear function \textbf{illustration of this}.

In order to address this issue, a class of gradient based methods, termed Trust Region, methods were developed. These optimization methods in particular involve first determining the maximum magnitude, $\delta$, of change that can be imposed on the current guess of the solution. The size of $\delta$ is determined primarily by the extent to which the objective function being minimized reduces in the region around the current guess of a solution. There are a variety of algorithms for implementing this class of algorithms, the details of which are illustrated by Yuan et al., \textbf{CITE TRUST REGION}. To end our discussion on gradient methods, we will outline the general trust region problem.

The first step of the Trust-Region algorithm is making a local quadratic approximation of the objective function.

\begin{equation} \label{quad_approx}
    \tilde{y}(\mathbf{x}+\delta,\mathbf{z}) = y(\mathbf{x},\mathbf{z}) + g^T \delta + \delta^T B \delta
\end{equation}
where g is as defined in \ref{grad_descent}, $\delta$ is some perturbation to $\mathbf{x}$, and $B$ is an approximation of the hessian of the objective function.\textbf{CITE TRUST REGION} If Like gradient descent, the Trust Region Algorithm is iterative. It selects a vector step by which to change $\mathbf{x}^n$ by using \ref{quad_approx} to solve the following constrained optimization problem,

\begin{equation} \label{trust_region_optim_problm}
\begin{aligned}
& \underset{\delta}{\text{minimize}}
& & y_n(\mathbf{x}^n,\mathbf{z}) + g_n^T \delta + \delta^T B_n \delta\\
& \text{such that}
& & ||\delta||_2 \leq \Delta_n.
\end{aligned}
\end{equation}

where $g_n$ is the gradient of the objective function evaluated at $\mathbf{x}^n$, $B_n$ is the approximation of the Hessian at the $\mathbf{x}^n$, and $\Delta_n$ is the trust region at iteration $n$. The size of $\Delta_n$ is determined by the extent to which the objective function, $y$, can be minimized within this region. If on the previous iteration, $n-1$, the objective function is sufficiently reduced, then the trust region generally expands. If not, it reduces. Since we simply use a Python implementation of the Trust Region Algorithm, we do not go into specifics. Details on a variety of Trust Region Algorithms can be found in Yuan et al, \textbf{CITE TRUST REGION}.

\subsection{Stochastic Optimization Methods: Particle Swarm Optimization (PSO)}
Gradient based methods have two main limitations. The first is that they require the calculation of gradients, which can be expensive, and the second is that they are prone to local minima when given poor initial conditions. Stochastic optimization methods attempts to address both of these issues. Rather than taking gradients, stochastic optimization methods randomly search the solution space in order to minimize an objective function. Some of these algorithms have heuristically been shown to be more efficient at finding the global minimum of an objective function, \textbf{CITE PSO Thesis}. The stochastic optimization method of interest to us is Particle Swarm Optimization (PSO). In this section we will review PSO, and its key variants.

The basic version of PSO was developed by Kennedy et al. \textbf{Cite Basic PSO Paper}. In this version of PSO, the user initializes a "swarm" of $m$ guesses at an optimal solution, $(\mathbf{x}_1,...,\mathbf{x}_m)$, and evaluates the function at each of these guesses. The guess that results in the most optimal function, $\mathbf{x}_{Best}$, is stored. Each guess is called a particle, and the best solution each particle has ever seen during the optimization process is stored as $\mathbf{x}_{i,Best}$. During an iteration of particle swarm optimization, each particle's velocity is updated as

\begin{equation} \label{pso_vel}
    \begin{aligned}
    v_{i,j}(k+1) &= w v_{i,j}(k) + c_1 r_{1,j}((\mathbf{x}_{i,Best}(k))_j - x_{i,j}(k)) + c_2 r_{2,j} ((\mathbf{x}_{Best}(k))_{i,j} - x_{i,j}(k)), \\
    &i=1,...,m \quad j=1,...,n
    \end{aligned}
\end{equation}
where $v_{i,j}(k)$ is the velocity of the $j^{th}$ dimension of the $i^{th}$ particle at iteration $k$, $w$ is an inertial weight that states the extent to which the current velocity should impact the subsequent one, $c_1$ is a user determined constant, and $r_{1,j}$ and $r_{2,j}$ are random variables distributed uniformly in the interval $[0,1]$, i.e $r_1 \sim U(0,1)$ and $r_2 \sim U(0,1)$. This velocity equation is then used to update each solution guess (particle), as

\begin{equation} \label{pso_pos}
    x_{i,j}(k+1) = x_{i,j}(k) + \mathbf{x}_{i,j}(k)
\end{equation}
Algorithm \ref{pso_algorithm} sketches out the PSO in detail, assuming that the goal is to minimize some objective function $f$. A couple of things should be clarified from the algorithm sketch. Three of these clarifications involve the position and velocity of each particle. First, it is standard to guess the initial particles by sampling a multivariate uniform distribution in the domain defined by $\mathbf{x}_{max}$ and $\mathbf{x}_{min}$. Second, a maximum, $\mathbf{v}_{max}$, and minimum, $\mathbf{v}_{min}$, speed, is used to bound the velocity of each particle at each iteration of PSO. This is in order to help keep the particle dynamics stable, a point that will be expounded on shortly. Third, the re-initialization of particles that move outside of the solution domain is not well part of the original PSO algorithm. However, we added this in order to help insure that most of the particles remained within the defined solution domain.

The next set of points that need to be expanded upon are the values of user defined hyper-parameters, $c_1$, $c_2$, and $w$. These parameters control the relative impact of a particle's personal best solution, the best solution in the swarm, and the particle's own previous velocity on it's subsequent movement in solution space. There a variety of methods for determining the value of the inertial weights during optimization, including the strategy of decreasing the weight linearly as a function of iteration, \textbf{Cite Shi and Ebehart Emprical Study}. However, Shi et al. found that selecting values of inertial weight that satisfies $0.9 \leq w \leq 1.2$ tends to enhance the ability of PSO to converge globally \textbf{Cite Shi and Ebehart Modified PSO}. Additionally, it was found that choosing values of $c_1 \textrm{ and } c_2$ to satisfy $0 \leq c_1, c_2 \leq 2.0$ allows for the PSO performance. The values of these hyper-parameters are summarized in Table \ref{pso_hyperparam_vals}.

\begin{breakablealgorithm}
\caption{PSO}\label{pso_algorithm}
\begin{algorithmic}[1]
\State Define $n_{max}$ \Comment{Maximum iterations of NP-PSO}
\State Define $n_{samples}$ \Comment{Maximum iterations of sampling particles}
\State Initialize $tol$  \Comment{Minimum change in best solution}
\EndProcedure

\Procedure{Initialize Particle Positions and Velocities}{}
\State Define maximum and minimum allowed velocities and positions, $\mathbf{v}_{max}$, $\mathbf{v}_{min}$, $\mathbf{x}_{max}$ and $\mathbf{x}_{min}$
\State Randomly initialize solutions, $\mathbf{x}_i \quad i=1...m$ in the domain bounded by $\mathbf{x}_{max}$ and $\mathbf{x}_{min}$
\State Randomly initialize velocities, $\mathbf{v}_i \quad i=1...m$ to zero or in the domain bounded by $\mathbf{v}_{max}$ and $\mathbf{v}_{min}$
\EndProcedure

\Procedure{Initialize Best Particles}{}
\State $\mathbf{x}_{Best} \leftarrow \underset{\mathbf{x}_i}{\textrm{min}}(y(\mathbf{x}_1)...y(\mathby{x}_m))$
\For{$\mathbf{x}_i$ in $[\mathbf{x}_1,....\mathbf{x}_m]$}
    \State $x_{i,Best} \leftarrow \mathbf{x}_i$
\EndFor
\EndProcedure

\Procedure{Run PSO}{}
\State $k \leftarrow 0$
\While{$k<=n_{max} \text{ or } f_{Best}>tol$}
    \State Update each particle velocity using \ref{pso_vel}
    \If {$v_{i,j}(k+1) > \mathbf{v}_{max}_{i,j}$ or $v_{i,j}(k+1) < \mathbf{v}_{min}_{i,j}$}
    \State $v(k+1)_{i,j} = \textrm{max}(\textrm{min}(v(k+1)_{i,j},\mathbf{v}_{max}_{i,j}),\mathbf{v}_{min}_{i,j})$
    \EndIf
    \State Update each particle position using \ref{pso_pos}
    \If {$x_{i,j}(k+1) > \mathbf{x}_{max}_{i,j}$ or $x_{i,j}(k+1) < \mathbf{x}_{min}_{i,j}$}
    \State Randomly reinitialize particle $i$ in the domain bounded by
    \EndIf
    \For{$\mathbf{x}_i$ in $[\mathbf{x}_1,....\mathbf{x}_m]$}
        \If {$y(\mathbf{x}_i) < y_{i,Best}$ }
            \State $\mathbf{x}_{i,Best} = \mathbf{x}_i$
        \EndIf
        \If {$y(\mathbf{x}_i) < y_{Best}$ }
            \State $\mathbf{x}_{Best} = \mathbf{x}_i$
        \EndIf
    \EndFor
    $k \leftarrow k+1$
\EndWhile
\EndProcedure
\end{algorithmic}
\end{breakablealgorithm}

The initial PSO algorithm was promising, and has found applications ranging from tuning the hyper-parameters of neural networks, \textbf{DNN PSO Paper}, to estimating the value of circuit elements in PV Cells, \textbf{Cite PV Cells Paper}. Despite the promise of PSO, it has multiple issues. The first is that it is prone to premature convergence to local minima, and the second is that the dynamics of some particles tend toward instability \textbf{PSO Thesis}. More specifically, their velocities can grow exponentially, causing these particles to quickly move away from their defined solution domain. Multiple variants of PSO have been proposed to address these issues. In this work, we focus on the constricted PSO \textbf{Constriction PSO Paper}, and the Nonparametric PSO (NPSO) \textbf{NPSO Paper}.

\begin{table}[!htb]
\begin{center}
{\begin{tabular}{|c|c|}
	\hline
	\textbf{PSO Hyper-parameters} & \textbf{Recommended Range of Values} \\
    \hline
    $w$ & $0.9 \leq w \leq 1.2$ \\
    \hline
    $c_1$  & $0 \leq c_1 \leq 2.0$ \\
    \hline
    $c_2$  & $0 \leq c_2 \leq 2.0$ \\
    \hline
\end{tabular}}
\caption{Typical range of Hyper-parameters of the PSO algorithm found in literature}\label{pso_hyperparam_vals}
\end{center}
\end{table}

\subsection{Stochastic Optimization: Nonparametric Particle Swarm Optimization and Constricted PSO}
In this subsection we review the constricted PSO, and the NPSO optimization algorithms mentioned above. The constricted PSO was derived through Clerc et al.'s study of the stability of the PSO algorithm, \textbf{Cite Clerc et al}, assuming a function with a scalar solution, $x^*$. Before summarizing the work of Clerc et al, we introduce a few key expressions. The first is a combination of $x$ and $x_{Best}$,
\begin{equation} \label{combined_point}
    p = \frac{\phi_1 x + \phi_2 x_{Best}}{\phi} \quad \phi = \phi_1 + \phi_2
\end{equation}
where $\phi_1 = r_1 c_1$ and $\phi_2 = r_2 c_2$, which are both random quantities. However, for the purpose of the stability analysis, $\phi_1$ and $\phi_2$ are taken to be fixed quantities. Using this, Clerc et al. wrote a system of equations for the trajectories of the PSO particle as \textbf{Cite PSO Thesis}

\begin{equation} \label{continous_pso_states_clerc}
    \begin{aligned}
        v(k+1) &= v(k) + \phi (p-x(k)) \\
        x(k+1) &= x(k) + v(k+1)
    \end{aligned}
\end{equation}

This system is subsequently transformed to a discrete time dynamic system,
\begin{equation} \label{discrete_pso_states_clerc}
   \begin{aligned}
    v_{k+1} &= v_k + \phi z_k  \\
    z_{k+1} &= -v_{k} + (1-\phi)z_{k}
    \end{aligned}
\end{equation}
where $z_k = p - x_k$. Clerc et al. were able to show that the behavior of this system is governed by two eigenvalues, $\lambda_1$ and $\lambda_2$, where the system is guaranteed to converge when $max(|\lambda_1|,|\lambda_2|) < 1$. Clerc et al. wanted to be able to explicitly control this rate of convergence, so they further transformed the system into one in which the eigenvalues can be written as $\lambda_1' = \chi_1 \lambda_1$ and $\lambda_2' = \chi_2 \lambda_2$ with tune-able
coefficients, $\beta, \gamma, \delta, \eta, \textrm{ and } \alpha$.
\begin{equation} \label{discrete_controllable_pso_states_clerc}
   \begin{aligned}
    v_{k+1} &= \alpha v_k + \beta \phi z_k  \\
    z_{k+1} &= -\gamma v_{k} + (\delta- \eta \phi)z_{k}
    \end{aligned}
\end{equation}
Different choices of the above control parameters result in the system defined by \ref{discrete_controllable_pso_states_clerc} converging. Clerc et al. generalized the results of analyzing this system to the original Particle Swarm Optimization equations, \ref{pso_vel} and \ref{pso_pos}, in order to obtain a constricted velocity update equation,
\begin{equation} \label{pso_vel_constrict}
    \begin{aligned}
    v_{i,j}(k+1) &= \chi (v_{i,j}(k) + c_1 r_{1,j}((\mathbf{x}_{i,Best}(k))_j - x_{i,j}(k)) + c_2 r_{2,j} ((\mathbf{x}_{Best}(k))_{i,j} - x_{i,j}(k))), \\
    &i=1,...,m \quad j=1,...,n
    \end{aligned}
\end{equation}

where $\chi$ satisfies
\begin{equation}
    \chi = \frac{2}{|2 - \psi \sqrt{\psi^2 - 4\psi}|}, \quad \psi = c_1 + c_2
\end{equation}
and $c_1=c_2=4.05$. With this update equation, it is no longer necessary to bound the maximum and minimum velocity as in the original PSO algorithm, \ref{pso_algorithm}. Thus, the constriction PSO is simply algorithm  algorithm \ref{pso_algorithm}, where the velocity update equation is now governed by \ref{pso_vel_constrict}, and there is no need to bound particle velocities. Additionally, Clerc et al. performed a variety of experiments on test functions which showed that the constricted PSO alleviated the issues of premature convergence.

Despite this improvement, constricted PSO still has some issues with convergence. Moreover, like many PSO algorithms in literature, its dynamics are governed by a host of parameters that the user most choose, or tune. In order to address these issues, Beheshti et al. proposed a variant of PSO called the Nonparametric Particle Swarm Optimization (NP-PSO). \textbf{Cite NP-PSO Paper} The NP-PSO proposes three main changes. First, the standard PSO velocity update, \ref{pso_vel}, is changed to
\begin{equation} \label{pso_vel_nppso}
    \begin{aligned}
    v_{i,j}(k+1) &= v_{i,j}(k) + r_{1,j}((\mathbf{x}_{i,Best}(k))_j - x_{i,j}(k)) + r_{2,j} ((\mathbf{x}_{i,lBest}(k))_{i,j} - x_{i,j}(k)). \\
    &i=1,...,m \quad j=1,...,n
    \end{aligned}
\end{equation}
Recalling that $\mathbf{x}_{i,Best}$ is the best solution that particle $i$ has seen in it's history, and defining a neighborhood of size $2l$ of these as $({x}_{i-l,Best},...{x}_{i-1,Best},{x}_{i,Best},{x}_{i+1,Best},...{x}_{i-l,Best})$, $\mathbf{x}_{i,lBest}$ is the best of these particles. Additionally, the update equation, \ref{pso_vel_nppso}, removes the tuneable parameters $w$, $c_1$, and $c_2$.
Second, the standard PSO position update, \ref{pso_pos}, becomes
\begin{equation} \label{pso_pos_nppso}
    x_{i,j}(k+1) = x_{i,j}(k) + \mathbf{x}_{i,j}(k) + r_3 ((\mathbf{x}_{Best}(k))_{i,j} - x_{i,j}(k))
\end{equation}
where $r_3$ is uniformly distributed similarly to $r_1$ and $r_2$. This incorporates knowledge of the best solution found by the entire particle swarm into a particle's motion in solution space. Finally, the global best particle, $x_{Best}$, is updated differently at each iteration. The process of updating $\mathbf{x}_{Best}$ starts by randomly selecting two particles in the swarm, $(\mathbf{x}_J \textrm{ and } \mathbf{x}_K$, where we require that $\mathbf{x}_J \neq \mathbf{x}_K \neq \mathbf{x}_{Best}$. Two new particles, $\mathbf{x}_1'$ and $\mathbf{x}_2'
$, are introduced to the system using the following quadratic interpolations,
\begin{align} \label{nppso_interpolations}
    \mathbf{x}_1' &= \frac{1}{2} \frac{(\mathbf{x}_J^2 - \mathbf{x}_{Best}^2 - \mathbf{x}_K^2) \times y(\mathbf{x}_J) \times y(\mathbf{x}_K)}{(\mathbf{x}_J - \mathbf{x}_K) \times y(\mathbf{x}_{Best}) + (\mathbf{x}_K - \mathbf{x}_{Best}) \times y(\mathbf{x}_J) + (\mathbf{x}_{Best} - \mathbf{x}_J) \times y(\mathbf{x}_K)} \label{nppso_interp_1}\\ \nonumber\\
    \mathbf{x}_2' &= \frac{1}{2} \frac{(\mathbf{x}_J^2 - \mathbf{x}_K^2) \times y(\mathbf{x}_{Best}) + (\mathbf{x}_K^2 - \mathbf{x}_{Best}^2) \times y(\mathbf{x}_J) + (\mathbf{x}_{Best}^2 - \mathfb{x}_J^2) \times y(\mathbf{x}_K)}{(\mathbf{x}_J - \mathbf{x}_K) \times y(\mathbf{x}_{Best}) + (\mathbf{x}_K - \mathbf{x}_{Best}) \times y(\mathbf{x}_J) + (\mathbf{x}_{Best} - \mathbf{x}_J) \times y(\mathbf{x}_K)}  \label{nppso_interp_2}
\end{align}
These interpolations artificially expand the amount of solution space that our particle swarm is able to explore. We note that we have overloaded notation in equations \ref{nppso_interp_1} and \ref{nppso_interp_2}, and that the interpolation is performed separately on each element of the vectors. Subsequently, $\mathbf{x}_{Best}$ is first compared to $\mathbf{x}_1'$, and, if $\mathbf{x}_1'$ is more fit than $\mathbf{x}_{Best}$, $\mathbf{x}_{Best}$ is assigned to be $\mathbf{x}_1'$. The full NP-PSO algorithm is sketched out in \ref{nppso_algorithm}. We make an important point of clarification, during the step in which we sample new particles, $\mathbf{x}_J$ and $\mathbf{x}_K$. In order to meet the condition that $\mathbf{x}_J \neq \mathbf{x}_K \neq \mathbf{x}_{Best}$, we require that no elements of the vectors are equal. If it is the case that they are, we re-sample $\mathbf{x}_J$ and $\mathbf{x}_K$ for a maximum of $n_{samples}$ iterations. Behesti et al. tested NP-PSO on nineteen unimodal and  multimodal, and found that it outperformed PSO, constricted PSO, and four other popular variants of PSO in almost every scenario.

\begin{breakablealgorithm}
\caption{NP-PSO}
\label{nppso_algorithm}
\begin{algorithmic}[1]
\Procedure{Initialize Stopping Criterion}{}
\State Define $n_{max}$ \Comment{Maximum iterations of NP-PSO}
\State Define $n_{samples}$ \Comment{Maximum iterations of sampling particles}
\State Initialize $tol$  \Comment{Minimum change in best solution}
\EndProcedure

\Procedure{Initialize Particle Position and Velocities}{}
\State Define maximum and minimum allowed velocities and positions, $\mathbf{v}_{max}$, $\mathbf{v}_{min}$, $\mathbf{x}_{max}$ and $\mathbf{x}_{min}$
\State Randomly initialize velocities, $\mathbf{v}_i \quad i=1...m$ to zero or in the domain bounded by $\mathbf{v}_{max}$ and $\mathbf{v}_{min}$
\State Randomly initialize solutions, $\mathbf{x}_i \quad i=1...m$ in the domain bounded by $\mathbf{x}_{max}$ and $\mathbf{x}_{min}$
\EndProcedure

\Procedure{Initialize Best particles}{}
\State Define neighborhood size, $l$
\State $\mathbf{x}_{Best} \leftarrow \underset{\mathbf{x}_i}{\textrm{min}}(y(\mathbf{x}_1)...y(\mathbf{x}_m))$
\State $x_{i,Best} \leftarrow \mathbf{x}_i \quad i=1,..m$
\State $\mathbf{x}_{i,lBest} \leftarrow \underset{\mathbf{x}}{\textrm{min}}(y(\mathbf{x}_{i-l,Best}),...,y(\mathbf{x}_{i-1,Best}),y(\mathbf{x}_{i,Best}),y(\mathbf{x}_{i+1,Best}))  \quad i=1,..m$
\EndProcedure

\Procedure{Run NP-PSO}{}
\State $k \leftarrow 0$
\While{$k<=n_{max} \text{ or } f_{Best}>tol$}
    \For {$\mathbf{x}_i$ in $[\mathbf{x}_1,....\mathbf{x}_m]$}
        \State Update each particle velocity using \ref{pso_vel_nppso}
        \If {$v_{i,j}(k+1) > \mathbf{v}_{max}_{i,j}$ or $v_{i,j}(k+1) < \mathbf{v}_{min}_{i,j}$}
        \State $v(k+1)_{i,j} = \textrm{max}(\textrm{min}(v(k+1)_{i,j},\mathbf{v}_{max}_{i,j}),\mathbf{v}_{min}_{i,j})$
        \EndIf
        \State Update each particle position using \ref{pso_nppso}
        \If {$x_{i,j}(k+1) > \mathbf{x}_{max}_{i,j}$ or $x_{i,j}(k+1) < \mathbf{x}_{min}_{i,j}$}
        \State Randomly reinitialize particle $i$ in the domain bounded by
        \EndIf
        \If {$y(\mathbf{x}_i) < y_{i,Best}$ }
            \State $\mathbf{x}_{i,Best} = \mathbf{x}_i$
        \State $j \leftarrow 0$
        \State Randomly select particles $\mathbf{x}_J$ and $\mathbf{x}_K$ from the swarm
        \While{$j < n_{sample}_{max}$ and ($\mathbf{x}_J == \mathbf{x}_K$ or $\mathbf{x}_J == \mathbf{x}_{Best}$ or $\mathbf{x}_{Best} == \mathbf{x}_K$)}
            \State Randomly sample particles $\mathbf{x}_J$ and $\mathbf{x}_K$ from the swarm
            \State $j \leftarrow j+1$
        \EndWhile
        \If {$j < n_{samples}$}
            \State Calculate $\mathbf{x}_1'$ using \label{nppso_interp_1}
            \If {$y(\mathbf{x}_1') < y_{Best}$ }
                \State $\mathbf{x}_{Best} = \mathbf{x}_1'$
            \EndIf
            \State Calculate $\mathbf{x}_2'$ using \label{nppso_interp_2}
            \If {$y(\mathbf{x}_2') < y_{Best}$ }
                \State $\mathbf{x}_{Best} = \mathbf{x}_2'$
            \EndIf
        \EndIf
        \If {$y(\mathbf{x}_i) < y_{Best}$ }
            \State $\mathbf{x}_{Best} = \mathbf{x}_i$
        \EndIf
    \EndFor
    $k \leftarrow k+1$
\EndWhile
\EndProcedure
\end{algorithmic}
\end{breakablealgorithm}

\section{Hybrid Algorithms and Challenges to Optimization}
We have reviewed gradient based and stochastic optimization algorithms. We have seen that, while gradient methods are very accurate when they are initialized in the vicinity of global optima, they are very prone to local minima. Additionally, the computation of a numerical gradient can be very expensive. Stochastic algorithms were subsequently developed in order to remove the necessity of computing gradients during the process of optimization, and to allow for a more thorough exploration of solution space, so that there is a higher probability of finding the global optimum. While stochastic algorithms have a better chance of converging to the region of a global optima, they can require a large amount of extraneous function evaluations. Moreover, they can result in final values that are at least slightly sub-optimal, even in the vicinity of some global optimum.

In order to address some of these issues, a variety of researchers have proposed hybrid algorithms that combine Particle Swarm Optimization and gradient based methods. Recently, Han et al. and Salajegheh et al. proposed DGPSOGS and PSOG respectively, both of which are variants of PSO that use local gradient and hessian information in it's update equations, \textbf{CITE DGPSOGS and PSOG Papers}. Additionally, Chen et al. and Plevris et al. proposed hybrid algorithms that used standard PSO, algorithm \ref{pso_algorithm}, to initially explore the solution space. \textbf{Cite HybridPSOSQR and HybridPSOConjugate} The best solution from PSO is then used to initialize gradient methods that fine tune the optimization. All of the above hybrid methods were found to be more effective than PSO alone.

\section{Quantifying Uncertainty in Parameter Estimates}
We have, to this point, formulated parameter estimation of models as a nonlinear optimization problem, reviewed multiple approaches for optimization, and presented our algorithm for performing parameter estimation. An essential part of parameter estimation is quantifying the extent to which we trust the values obtained by our parameters. Two popular methods of quantifying this uncertainty are confidence intervals, and confidence regions.

Confidence intervals define the possible bounds on individual estimated parameters using equation

\begin{equation} \label{norm_conf_interval}
    x_i \pm t_{n-p}^{1-\frac{\alpha}{2}}(s^2 c_{ii})^{\frac{1}{2}}
\end{equation}

where $n$ is the total number of data points, $p$ is the number of parameters being estimated, $t_{n-p}^{1-\frac{\alpha}{2}}$ is a t-score with $n-p$ degrees of freedom at a confidence level of $1-\alpha$, $c_{ii}$ is the variance of parameter $x_i$ which corresponds to the diagonal of parameter covariance matrix $\mathbf{C}_\mathbf{x}$, and $s = \sqrt{\frac{y(\mathbf{x}_{Best})}{n-p}}$.

Confidence regions differ, because they define joint confidence in all of the parameters simultaneously. A common approach for defining the confidence region is as a hyper-ellipse that satisfies

\begin{equation} \label{norm_conf_region}
    (\mathbf{x}-\mathbf{x}_{Best})^T \mathbf{C}_\mathbf{x}^{-1}(\mathbf{x}-\mathbf{x}_{Best}) \leq \frac{p}{n-p} y(\mathbf{x}_{Best})F_{p,n-p}^{1-\alpha}
\end{equation}

where $F_{p,n-p}^{1-\alpha}$ is the F statistic with $p$ and $n-p$ degrees of freedom, at a confidence of $1-\alpha$. The definition of confidence intervals and confidence regions given by equations \ref{norm_conf_interval} and \ref{norm_conf_region}, both assume that the parameters and experiments follow a normal distribution. While this is a valid assumption for models that are linear in estimated parameters, this is generally not valid for nonlinear models, even if the experiments are normally distributed. In order to address this issue, Beale et al. proposed the following approximation for confidence regions of nonlinear models. \textbf{Cite Beale Paper}

\begin{equation} \label{nonlin_confidence_regions}
    y(\mathbf{x}) \leq y(\mathbf{x}_{Best})\bigg(1 + \frac{p}{n-p}F_{p,n-p}^{1-\alpha}\bigg)
\end{equation}

The above approximation gives an exact result for linear models, assuming that experiments are normally distributed. More generally, equation \ref{nonlin_confidence_regions} can be written as $ y(\mathbf{x}) \leq cy(\mathbf{x}_{Best})$, where $c$ is determined based on the distribution of parameters and experiments. However, it is difficult to determine the distribution of parameters and experiments. Instead, the assumption of normality in experimental errors is made in order to define $c$ as $c=\bigg(1 + \frac{p}{n-p}F_{p,n-p}^{1-\alpha}\bigg)$. A big advantage of equation \ref{nonlin_confidence_regions} is that it does not require the confidence regions to be ellipsoids, so it allows for more accurate approximations of confidence regions in the case of nonlinear models. Despite this advantage, equation \ref{nonlin_confidence_regions} is expensive, since it requires a large amount of objective function evaluations in order to form a reasonable confidence region.

Schwaab et al. addressed this issue using Particle Swarm Optimization (PSO). \textbf{cite Schwaab nonlinear}. The number of function evaluations necessary for PSO algorithms are usually viewed as a downside. However, Schwaab et al. presented work in which they used these function evaluations to construct confidence intervals. Specifically, Schwaab et al. kept track of every function evaluation done during the process of PSO, and all the parameter values that satisfy equation \ref{nonlin_confidence_regions} are used to construct the confidence region. In their work, Schwaab et al. demonstrated the ability of this method  to create non-ellipsoid confidence regions for a variety of nonlinear models. \textbf{cite Schwaab nonlinear} In order to do so, it was necessary for them to run PSO for an extended number of iterations, and to make sure to use enough particles to thoroughly explore the solution space. \textbf{cite Schwaab nonlinear} One thing worth noting is that, using the above method, it is straight forward to create confidence regions of combinations of relevant parameters. This will allow us to find the confidence regions of physical constants. We will use this technique described here to not only quantify uncertainty in our parameter estimates, but to perform physical model selection and experimental design.

\section{x-PSO-TRF-y: Parameter Estimation, Model Selection, and Experimental Design}
Up until this point we have reviewed a variety of methods for optimization, with a strong emphasis on PSO based methods. We also discussed a method for estimating the confidence in our optimal  We propose two hybrid algorithms, collectively called x-PSO-TRF-y, that make use of multiple variants of Particle Swarm Optimization (PSO), and can use information from all the particles in the gradient step. The x satisfies $\textrm{x} \in [\varnothing,\textrm{constrict},\textrm{NP}]$, and the y satisfies $\textrm{y} \in [1,2]$. In the scenario when y is 1, the relevant variant of PSO is run for some number of trials, and the best solution from each trial is used to initialize the TRF algorithm. When y is 2, rather than using the best solution of each swarm in each trial to initialize TRF, we run TRF using the best solution each particle has seen in it's history as an initial guess. We generally run these algorithms for a user defined number of trials, $n_{Trials}$. Every function evaluation run during the PSO phase of our  algorithm is stored, as are the final results of the TRF algorithm. Two separate confidence regions are then formed. One is formed using only the function evaluations stored during the PSO phase of our algorithms, and the other using both the function evaluations from PSO, and the results of the TRF. The outputs of this algorithm, which will be used in physical model selection and experimental design, is represented in the following plots:

\begin{enumerate}
    \item Plot of data, and model output from most optimal parameter
    \item Plot of data, and model outputs from all parameters in each confidence region
    \item Histogram of fitness of particles in each confidence region
    \item Plot of confidence regions of every pair of parameters
    \item Plot of confidence regions of physical constants of system that are constructed by combining the parameters
\end{enumerate}

In order to use this algorithm for physical model selection, and experimental design, we run the described algorithm for the reduced models under consideration, and at different numbers of experiments and data points within the experiments. The full procedure is given by algorithm \ref{xPSOTRFy_algorithm}.

\begin{breakablealgorithm}
\caption{x-PSO-TRF-y}
\label{xPSOTRFy_algorithm}
\begin{algorithmic}[1]
\Procedure{Initialize Experimental Conditions and confidence}{}
\State Define $n_{Experiments}$ \Comment{Define number of experiments run for physical system}
\State Define $n_y$  \Comment{Define number of measurements collected in each experiment}
\State Define confidence level, $1-\alpha$
\EndProcedure

\Procedure{Initialize Algorithm and Number of Trials}{}
\State Define $n_{Trials}$ \Comment{Define number of times PSO will be run}
\State Select variant of x-PSO, x,$\textrm{x} \in [\varnothing,\textrm{constrict},\textrm{NP}]$
\EndProcedure

\For{k=1:$\textrm{n}_{\textrm{Trials}}$}
\State Run x-PSO, and store all function evaluations, and the corresponding parameters
\State Define the confidence regions for only PSO as all the parameters,$\mathbf{x}$, seen during x-PSO that satisfies equation \ref{nonlin_confidence_regions}
\If{y=1}
\State Extract best solution from x-PSO, $\mathbf{x}_{Best}$.
\State Initialize guess for TRF as $\mathbf{x}_{Best}$, and run TRF
\State $\mathbf{x}_{Best} \leftarrow \underset{\mathbf{x}}{min}[y(\mathbf{x}_{Best}),y(\mathbf{x}_{Best-TRF})]$
\EndIf
\If{y=2}
\State Extract best solutions of each particle $\mathbf{x}_{i,Best}$
\For{$\mathbf{x}_{i,Best}$ in [$\mathbf{x}_{1,Best}$,...,$\mathbf{x}_{m,Best}$ ]}
\State Initialize guess for TRF as $\mathbf{x}_{i,Best}$, and run algorithm
\State Store Solution
\EndFor
\State $\mathbf{x}_{Best} \leftarrow \underset{\mathbf{x}}{min}[y(\mathbf{x}_{1,Best}),y(\mathbf{x}_{1,Best-TRF}),...,y(\mathbf{x}_{m,Best}),y(\mathbf{x}_{m,Best-TRF})]$
\EndIf
\State Define new confidence regions that satisfies equation \ref{nonlin_confidence_regions}
\EndFor
\end{algorithmic}
\end{breakablealgorithm}

\section{Selecting the Reduced Comb-Drive Model of Best Fit}
In this section, we analyze how well each of our reduced comb-drive models fits all the data, and select the simplest model that best explains the data. We do so, for the set of experiments described at the end of chapter 2. Briefly, we measure the displacement of the comb-drive actuator at concentrations of 0.1 $mM$, 0.5 $mM$, 1 $mM$, and 10 $mM$ KCl for two types of comb-drive actuators. One comb-drive actuator has 200 pairs of fingers with gaps of 2 $\mu m$, and one with 100 pairs of fingers wit gaps of 5 $\mu m$. These are done at 2 signals with magnitudes of 2 $V_{p-p}$.


\typeout{}\typeout{If latex fails to find aiaa-tc, read the README file!}
%

\documentclass[]{aiaa-tc}% insert '[draft]' option to show overfull boxes

\usepackage{amsmath}          % for formula writing (i.e. 'split', etc)
\usepackage{rotate}           %rotate/mirror images
\usepackage{cancel}           %draw lines through math to show "goes to zero"
\usepackage{xfrac}            %allows slated and side fractions
\usepackage{subcaption}       %allows captioning individual subfigures
\usepackage{multicol}         %enable environment with multiple columns
\usepackage[mode=buildnew]{standalone}% requires -shell-escape
  % compile with `pdflatex -shell-escape main` or `xelatex  -shell-escape main`

\usepackage{tikz}             %for creating vector graphics diagrams
\usetikzlibrary{backgrounds}  %put backgrounds behind tikz figures
\usetikzlibrary{calc}         %perform calculations within $$
\usetikzlibrary{positioning}  %position tikz elements using "right of, etc"
\usetikzlibrary{angles}       %label angles between lines with arcs
\usetikzlibrary{quotes}       %Put angle label in quotes
\usetikzlibrary{patterns}     %Patterns to fill shapes with

  \title{MAE 298 Aeroacoustics -- Homework \#2 \\ Lilley's Equation Solution and Application to Jet Noise}

\author{
  Logan D. Halstrom \\
  {\normalsize\itshape Graduate Student} \\
  {\normalsize\itshape Department of Mechanical and Aerospace Engineering} \\
  {\normalsize\itshape University of California, Davis, CA 95616}
       }

 % Define commands to assure consistent treatment throughout document
 \newcommand{\eqnref}[1]{(\ref{#1})}
 \newcommand{\class}[1]{\texttt{#1}}
 \newcommand{\package}[1]{\texttt{#1}}
 \newcommand{\file}[1]{\texttt{#1}}
 \newcommand{\BibTeX}{\textsc{Bib}\TeX}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{document}

\maketitle

% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% \begin{abstract}

% Abstract about homework

% \end{abstract}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section*{Nomenclature}

\begin{multicols}{2}

\begin{tabbing}
  XXX \= \kill% this line sets tab stop
  $\infty$            \> Ambient flow parameter subscript \\
  $j$                 \> Jet exit flow parameter subscript \\
  $r$                 \> Radial location in flow \\
  $R_j$               \> Axisymmetric jet radius \\
  $\theta$            \> Angular location in flow \\
  $z$                 \> Axial location in flow \\
  $M$                 \> Mach number \\
  $W(r)$              \> Radial distribution of mean axial velocity \\
  $\overline{a^2}(r)$ \> Radial distribution of speed of sound squared\\
  $\overline{\rho}$   \> Mean density \\
  $\gamma$            \> Specific heat ratio \\
  $\overline{p}$      \> Mean pressure \\
  $p'$                \> Perturbation pressure \\
  $P(r)$              \> Pressure profile in radial direction  \\
  $k$                 \> Wavenumber \\
  $\omega$            \> Wave oscillating frequency \\
  $n$                 \> Characteristic integer \\
  $i$                 \> Imaginary number $\sqrt{-1}$ \\
  $\exp$              \> Exponential ($e$) \\
  $E$                 \> Exponential term: $kz + n\theta -\omega t$ \\
  $\lambda$           \> Constant term in Bessel equation \\
  $J$                 \> First-order Bessel function \\
  $Y$                 \> Second-order Bessel function \\
  $H^{(n)}$           \> nth-order Hankel function \\
  $x$                 \> Placeholder variable for $\lambda r$ \\
  $A,B$               \> Arbitrary Bessel function constants \\
  $C,D$               \> Arbitrary Hankel constants \\
  $\vec{V}$           \> General velocity vector \\
  $V_r$               \> Velocity component in radial direction \\
  $\nu$               \> Constant velocity parameter \\
  $\chi$              \> Constant position parameter \\
  $\zeta$             \> Position of vortex sheet dividing inner/outer solution \\
  $+/-$               \> Outer/Inner solution, respectively \\

\end{tabbing}

\end{multicols}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Background} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

The following analysis will derive solutions to Lilley's equation for parallel axisymmetric flow (Eqn~\ref{Lilley}):

\begin{equation} \label{Lilley}
\left( \dfrac{\partial}{\partial t} + W \dfrac{\partial}{\partial z} \right)^3 p'
- \left( \dfrac{\partial}{\partial t} + W \dfrac{\partial}{\partial z} \right) \left( \overline{a^2} \nabla^2p' \right)
- \dfrac{d \overline{a^2}}{dr} \left( \dfrac{\partial}{\partial t} + W \dfrac{\partial}{\partial z} \right) \dfrac{dp'}{dr}
+ 2\overline{a^2} \dfrac{dW}{dr}\dfrac{\partial^2 p'}{\partial z \partial r}
= S(\vec{x}, t)
\end{equation}

\begin{center}
where $\nabla^2 \equiv \dfrac{1}{r}\dfrac{\partial}{\partial r} \left( r\dfrac{\partial}{\partial r} \right)
+ \dfrac{1}{r^2}\dfrac{\partial^2}{\partial \theta^2}
+ \dfrac{\partial^2}{\partial z^2}$
\end{center}

$W(r)$ is the radial distribution of axial velocity and $\overline{a^2}(r)$ is the radial distribution of speed of sound squared.

After a general solution is derived, it will be applied to find homogeneous solutions to the far-field and potential core regions of an axial jet.  Finally, the conditions for matching the solutions for this regions will be discussed.

\clearpage
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Problem 1 -- Solution to Lilley's Equation} \label{SecLilley}%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

In this section, we will seek solutions of Lilley's equation in the form:

\begin{equation} \label{SolnForm}
p'(r, \theta, z, t) \sim P(r) \exp\left[ i(kz + n\theta -\omega t) \right]
\end{equation}

\noindent We will assume isentropic flow and constant mean pressure:
\begin{enumerate}
  \item $\overline{a^2} = \dfrac{\gamma \overline{p}}{\overline{\rho}}$
  \item $\overline{p} = const$
\end{enumerate}

\noindent where $\overline{\rho}(r)$ is the radial distribution of the mean density.  With these assumptions, we will show that Lilley's equation reduces to (Eqn~\ref{LilleySoln}):

\begin{equation} \label{LilleySoln}
\dfrac{d^2 P}{dr^2}
+ \left\{ \dfrac{1}{r}
  - \dfrac{1}{\overline{\rho}} \dfrac{d\overline{\rho}}{dr}
  + \dfrac{2k}{(\omega - kW)} \dfrac{dW}{dr}
\right\} \dfrac{dP}{dr}
+ \left\{ \dfrac{(\omega - kW)^2}{\overline{a^2}} - k^2 - \dfrac{n^2}{r^2}
\right\} P = RHS
\end{equation}

To aid in this derivation, we find the results for the first-order partial derivatives of parameters relevant to the solution.  First, we compute the derivative of the perturbation pressure $p'$ with respect to (WRT) time $t$:

\newcommand\expterm{i(kz + n\theta -\omega t)}%terms in exponential

\begin{align*}
\dfrac{\partial p'}{\partial t} &= \dfrac{\partial}{\partial t}
  \left\{ P(r) \exp\left[ \expterm \right]\right\} \\
&= P \dfrac{\partial}{\partial t} [\expterm] \exp[\expterm] \\
&= iP \left[\cancelto{0}{\dfrac{\partial}{\partial t}(kz)      }
          + \cancelto{0}{\dfrac{\partial}{\partial t}(n\theta)}
                       - \dfrac{\partial}{\partial t}(\omega t)
    \right] \exp[\expterm]
\end{align*}

\begin{equation} \label{dpdt}
\boxed{\dfrac{\partial p'}{\partial t}
  = -iP\omega \exp[\expterm] = -iP\omega e^{iE} }
\end{equation}

\noindent where $E=kz + n\theta -\omega t$, $\dfrac{\partial E}{\partial t} = -\omega$, and $\dfrac{\partial E}{\partial z} = k$.  Next, we compute the derivative of $p'$ WRT the angular direction $\theta$:

\begin{align*}
\dfrac{\partial p'}{\partial \theta} &= \dfrac{\partial}{\partial \theta}
  \left\{ P(r) \exp\left[ \expterm \right]\right\} \\
&= P \dfrac{\partial}{\partial \theta} [\expterm] \exp[\expterm] \\
&= iP \left[\cancelto{0}{\dfrac{\partial}{\partial \theta}(kz)      }
                      + \dfrac{\partial}{\partial \theta}(n\theta)
          - \cancelto{0}{\dfrac{\partial}{\partial \theta}(\omega t)}
    \right] \exp[\expterm]
\end{align*}

\begin{equation} \label{dpdtheta}
\boxed{\dfrac{\partial p'}{\partial \theta} = iPn \exp[\expterm] = iPn e^{iE} }
\end{equation}

Next, we compute the derivative of $p'$ WRT the axial flow direction $z$:

\begin{align*}
\dfrac{\partial p'}{\partial z} &= \dfrac{\partial}{\partial z}
  \left\{ P(r) \exp\left[ \expterm \right]\right\} \\
&= P \dfrac{\partial}{\partial z} [\expterm] \exp[\expterm] \\
&= iP \left[             \dfrac{\partial}{\partial z}(kz)
          + \cancelto{0}{\dfrac{\partial}{\partial z}(n\theta)}
          - \cancelto{0}{\dfrac{\partial}{\partial z}(\omega t)}
    \right] \exp[\expterm]
\end{align*}

\begin{equation} \label{dpdz}
\boxed{\dfrac{\partial p'}{\partial z} = iPk \exp[\expterm] = iPk e^{iE} }
\end{equation}

Next, we compute the derivative of $p'$ WRT the radial direction $r$:

\begin{align*}
\dfrac{\partial p'}{\partial r} &= \dfrac{\partial}{\partial r}
  \left\{ P(r) \exp\left[ \expterm \right]\right\}
\end{align*}

\begin{equation} \label{dpdr}
\boxed{\dfrac{\partial p'}{\partial r} = \dfrac{dP}{dr} \exp[\expterm] = \dfrac{dP}{dr} e^{iE} }
\end{equation}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Term 1}

To simplify the derivation, we will apply the solution form individually to each term in Lilley's equation.  For the first term, we must apply the multi-derivative operator a total of three times:

% \begin{equation} \label{LilleyTerm1}
% \left( \dfrac{\partial}{\partial t} + W \dfrac{\partial}{\partial z} \right)^3 p'
% \end{equation}

\begin{align*}
\left( \dfrac{\partial}{\partial t} + W \dfrac{\partial}{\partial z} \right)^3 p'
&= \left( \dfrac{\partial}{\partial t} + W \dfrac{\partial}{\partial z}
   \right)^2
   \left( \dfrac{\partial}{\partial t} + W \dfrac{\partial}{\partial z}
   \right)p' \\
&= \left( \dfrac{\partial}{\partial t} + W \dfrac{\partial}{\partial z}
   \right)^2
   \left( \dfrac{\partial p'}{\partial t} + W \dfrac{\partial p'}{\partial z}
   \right) \\
&= \left( \dfrac{\partial}{\partial t} + W \dfrac{\partial}{\partial z}
   \right)^2
   \left( -P\omega ie^{iE} + PkW ie^{iE}
   \right) \\
&= \left( \dfrac{\partial}{\partial t} + W \dfrac{\partial}{\partial z}
   \right)
   \left( \dfrac{\partial}{\partial t} + W \dfrac{\partial}{\partial z}
   \right) (-\omega + kW)P i (e^{iE}) \\
&= \left( \dfrac{\partial}{\partial t} + W \dfrac{\partial}{\partial z}
   \right) (-\omega + kW)P i
   (-i\omega e^{iE} + ikW e^{iE}) \\
&= \left( \dfrac{\partial}{\partial t} + W \dfrac{\partial}{\partial z}
   \right) (-\omega + kW)^2 P i^2 (e^{iE}) \\
&= (-\omega + kW)^3 P i^3(e^{iE}) = (-\omega + kW)^3P(-i) (e^{iE}) \\
&= (\omega - kW)^3iP (e^{iE})
\end{align*}

The cubed imaginary number $i^3$ simplifies to $-i$ and the $-1$ is distributed into the cubed factor.  This results in the final expression for Term 1:

\begin{equation} \label{term1soln}
\boxed{ \left( \dfrac{\partial}{\partial t}
          + W \dfrac{\partial}{\partial z} \right)^3 p'
    =  i\exp[i(kz + n\theta -\omega t)] (\omega - kW)^3 P}
\end{equation}

\clearpage
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Term 2}

Application of the solution form to the second term (Eqn~\ref{term2}) of Lilley's equation is slightly more involved.

\begin{equation}\label{term2}
\left( \dfrac{\partial}{\partial t} + W \dfrac{\partial}{\partial z} \right) (\overline{a^2}\nabla^2p')
\end{equation}

Term 2 requires computing the double divergence of perturbation pressure $\nabla^2 p'$:

\begin{align*}
\nabla^2p'
  &= \dfrac{1}{r}\dfrac{\partial}{\partial r} \left( r\dfrac{\partial p'}{\partial r} \right)
      + \dfrac{1}{r^2} \dfrac{\partial^2 p'}{\partial \theta^2}
      + \dfrac{\partial^2 p'}{\partial z^2} \\
&= \dfrac{1}{r}\dfrac{\partial}{\partial r} \left( r \dfrac{dP}{dr} e^{iE} \right)
      + \dfrac{1}{r^2} \dfrac{\partial }{\partial \theta} \left( Pn ie^{iE} \right)
      + \dfrac{\partial}{\partial z} \left( Pk ie^{iE} \right) \\
&= \dfrac{1}{r} e^{iE} \left(
    \dfrac{dP}{dr} + r \dfrac{d^2P}{dr^2} \right)
      + \dfrac{1}{r^2} Pn^2 i^2 e^{iE}
      + Pk^2 i^2  e^{iE} \\
&= e^{iE} \dfrac{d^2P}{dr^2}
    + e^{iE} \dfrac{1}{r} \dfrac{dP}{dr}
    + i^2 e^{iE} \left( \dfrac{n^2}{r^2} + k^2 \right) P
\end{align*}

Thus, the double divergence of $p'$ can be expressed in the following expression, which is separated into like differential terms of $P$:

\begin{equation} \label{doubledivp}
\boxed{ \nabla^2p'= \exp[i(kz + n\theta -\omega t)] \left[
      \dfrac{d^2P}{dr^2}
    + \dfrac{1}{r} \dfrac{dP}{dr}
    - \left( \dfrac{n^2}{r^2} + k^2 \right) P \right] }
\end{equation}

Substituting Eqn~\ref{doubledivp} into Term 2 (Eqn~\ref{term2}), we can perform the multi-derivative expression to derive the final term.  All terms grouped with $P$ along with $a$ are constant WRT $t$ and $z$ and can be carried outside of the derivative expression.

\begin{align*}
\left( \dfrac{\partial}{\partial t}
  + W \dfrac{\partial}{\partial z} \right)
  (\overline{a^2}\nabla^2p')
&= \left( \dfrac{\partial}{\partial t}
  + W \dfrac{\partial}{\partial z}
  \right) \overline{a^2} e^{iE} \left[
      \dfrac{d^2P}{dr^2}
    + \dfrac{1}{r} \dfrac{dP}{dr}
    - \left( \dfrac{n^2}{r^2} + k^2 \right) P \right] \\
&= \overline{a^2} \left[ \dfrac{d^2P}{dr^2}
          + \dfrac{1}{r} \dfrac{dP}{dr}
          - \left( \dfrac{n^2}{r^2} + k^2 \right) P \right]
    \left( \dfrac{\partial}{\partial t}
  + W \dfrac{\partial}{\partial z}
  \right) e^{iE} \\
&= \overline{a^2} \left[ \dfrac{d^2P}{dr^2}
          + \dfrac{1}{r} \dfrac{dP}{dr}
          - \left( \dfrac{n^2}{r^2} + k^2 \right) P \right]
  \left( -\omega ie^{iE} + kW ie^{iE}\right) \\
&= - \overline{a^2} ie^{iE} (\omega - kW) \left[ \dfrac{d^2P}{dr^2}
          + \dfrac{1}{r} \dfrac{dP}{dr}
          - \left( \dfrac{n^2}{r^2} + k^2 \right) P \right]
\end{align*}

This results in the final expression for Term 2:

\begin{equation} \label{term2soln}
\boxed{ \left( \dfrac{\partial}{\partial t}
  + W \dfrac{\partial}{\partial z} \right)
  (\overline{a^2}\nabla^2p')
= - \overline{a^2} i\exp[i(kz + n\theta -\omega t)]
    (\omega - kW) \left[ \dfrac{d^2P}{dr^2}
    + \dfrac{1}{r} \dfrac{dP}{dr}
    - \left( \dfrac{n^2}{r^2} + k^2 \right) P \right] }
\end{equation}

\clearpage
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Term 3}

Now, we apply the solution form to third term of Lilley's equation:

\begin{align*}
\dfrac{d \overline{a^2}}{dr} \left( \dfrac{\partial}{\partial t} + W \dfrac{\partial}{\partial z} \right) \dfrac{dp'}{dr}
&= \dfrac{d \overline{a^2}}{dr}
   \left( \dfrac{\partial}{\partial t} + W \dfrac{\partial}{\partial z} \right)
   \dfrac{dP}{dr} e^{iE}  \\
&= \dfrac{d \overline{a^2}}{dr} \dfrac{dP}{dr}
   \left( \dfrac{\partial}{\partial t} + W \dfrac{\partial}{\partial z} \right) e^{iE} \\
&= \dfrac{d \overline{a^2}}{dr} \dfrac{dP}{dr}
    i(-\omega + kW) e^{iE}
\end{align*}

Apply the isentropic relationship assumption for speed of sound and taking the derivative WRT $r$:

\begin{align*}
\dfrac{d \overline{a^2}}{dr} \dfrac{dP}{dr}
    i(-\omega + kW) e^{iE}
&= \dfrac{d}{dr} \left(\dfrac{\gamma \overline{p}}{\overline{\rho}}\right)
    \dfrac{dP}{dr} i(-\omega + kW) e^{iE} \\
&= -\left(\dfrac{\gamma \overline{p}}{\overline{\rho}^2}\right) \frac{d \overline{\rho}}{dr}
    \dfrac{dP}{dr} i(-\omega + kW) e^{iE} \\
&= \left(\dfrac{\overline{a^2}}{\overline{\rho}}\right) \frac{d \overline{\rho}}{dr} \dfrac{dP}{dr} i(\omega - kW) e^{iE}
\end{align*}

This results in the final expression for Term 3:

\begin{equation} \label{term3soln}
\boxed{\dfrac{d \overline{a^2}}{dr} \left( \dfrac{\partial}{\partial t} + W \dfrac{\partial}{\partial z} \right) \dfrac{dp'}{dr}
= \overline{a^2} i\exp[i(kz + n\theta -\omega t)] (\omega - kW)
    \dfrac{1}{\overline{\rho}}\frac{d \overline{\rho}}{dr} \dfrac{dP}{dr} }
\end{equation}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Term 4}

Finally, we will apply the solution form to fourth term of Lilley's equation:

\begin{align*}
2\overline{a^2} \dfrac{dW}{dr}\dfrac{\partial^2 p'}{\partial z \partial r}
  &= 2\overline{a^2} \dfrac{dW}{dr} \dfrac{\partial }{\partial z}
    \dfrac{\partial p'}{\partial r}
    = 2\overline{a^2} \dfrac{dW}{dr} \dfrac{\partial }{\partial z}
    \left( \dfrac{dP}{dr} e^{iE} \right) \\
&= 2\overline{a^2} \dfrac{dW}{dr} \dfrac{dP}{dr}
    \dfrac{\partial }{\partial z} \left( e^{iE} \right)
    = 2\overline{a^2} \dfrac{dW}{dr} \dfrac{dP}{dr} ike^{iE}
\end{align*}

This results in the final expression for Term 4:

\begin{equation} \label{term4soln}
\boxed{2\overline{a^2} \dfrac{dW}{dr}\dfrac{\partial^2p'}{\partial z\partial r}
  = \overline{a^2} i\exp[i(kz + n\theta -\omega t)]
    2k \dfrac{dW}{dr} \dfrac{dP}{dr} }
\end{equation}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Lilley's Equation Solution}

To derive the final form of Lilley's equation, we combine Terms 1 through 4:

\begin{align*}
Eqn~\ref{term1soln} - Eqn~\ref{term2soln} - Eqn~\ref{term3soln} + Eqn~\ref{term4soln} = S(\vec{x}, t)
\end{align*}

Which becomes the following in expanded form:

\begin{align*}
\left\{ ie^{iE} (\omega - kW)^3 P \right\}
- \left\{ - \overline{a^2} ie^{iE}
    (\omega - kW) \left[ \dfrac{d^2P}{dr^2}
    + \dfrac{1}{r} \dfrac{dP}{dr}
    - \left( \dfrac{n^2}{r^2} + k^2 \right) P \right] \right\} \\
- \left\{ \overline{a^2} ie^{iE} (\omega - kW)
    \dfrac{1}{\overline{\rho}}\frac{d \overline{\rho}}{dr} \dfrac{dP}{dr} \right\}
+ \left\{ \overline{a^2} ie^{iE} 2k \dfrac{dW}{dr} \dfrac{dP}{dr} \right\}
= S(\vec{x}, t)
\end{align*}

Dividing both sides of the equation by the term $\overline{a^2} ie^{iE} (\omega - kW)$:

\begin{align*}
\dfrac{(\omega - kW)^2}{\overline{a^2}} P
+ \dfrac{d^2P}{dr^2}
+ \dfrac{1}{r} \dfrac{dP}{dr}
- \left( \dfrac{n^2}{r^2} + k^2 \right) P
- \dfrac{1}{\overline{\rho}}\frac{d \overline{\rho}}{dr} \dfrac{dP}{dr}
+ \dfrac{1}{\omega - kW} 2k \dfrac{dW}{dr} \dfrac{dP}{dr}
= \dfrac{S(\vec{x}, t)}{\overline{a^2} ie^{iE} (\omega - kW)} \\
\dfrac{d^2P}{dr^2}
+ \dfrac{1}{r} \dfrac{dP}{dr}
- \dfrac{1}{\overline{\rho}}\frac{d \overline{\rho}}{dr} \dfrac{dP}{dr}
+ \dfrac{1}{\omega - kW} 2k \dfrac{dW}{dr} \dfrac{dP}{dr}
+ \dfrac{(\omega - kW)^2}{\overline{a^2}} P
- \left( \dfrac{n^2}{r^2} + k^2 \right) P
= \dfrac{S(\vec{x}, t)}{\overline{a^2} ie^{iE} (\omega - kW)} \\
\end{align*}

Grouping like terms of $P$, we obtain the final form of the general solution of Lilley's Equation:

\begin{equation} \label{lilleysoln}
\boxed{
\dfrac{d^2P}{dr^2}
+ \left\{
    \dfrac{1}{r}
    - \dfrac{1}{\overline{\rho}}\frac{d \overline{\rho}}{dr}
    + \dfrac{1}{\omega - kW} 2k \dfrac{dW}{dr}
\right\} \dfrac{dP}{dr}
+ \left\{
    \dfrac{(\omega - kW)^2}{\overline{a^2}}
    - \left( \dfrac{n^2}{r^2} + k^2 \right)
\right\} P
= \dfrac{S(\vec{x}, t)}{\overline{a^2} ie^{iE} (\omega - kW)} }
\end{equation}

\noindent where $E=kz + n\theta -\omega t$.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Problem 2 -- General Solution for Jet Flow Far-Field} \label{secprob2}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Next, we will solve the homogeneous form of Lilley's equation (Eqn~\ref{lilleysoln}) to determine the general solution for the pressure fluctuation outside of the jet in the ambient medium where the sources vanish.  This region can be considered to be the far-field of the jet, so the solution must be chosen to ensure decaying solutions (outgoing waves).

Since this is far-field, ambient flow, we will assume that the mean velocity and density in this region are equal to the ambient mean values:

\begin{equation}
\begin{split}
&W(r)= W_{\infty}=const \\
&\overline{\rho} = \overline{\rho_0} = const
\end{split}
\end{equation}

\noindent which allows us to simplify the homogeneous equation:

\begin{align*}
\dfrac{d^2P}{dr^2}
+ \left\{
    \dfrac{1}{r}
    - \dfrac{1}{\overline{\rho}}\frac{d \overline{\rho}}{dr}
    + \dfrac{1}{\omega - kW} 2k \dfrac{dW}{dr}
\right\} \dfrac{dP}{dr}
+ \left\{
    \dfrac{(\omega - kW)^2}{\overline{a^2}}
    - \left( \dfrac{n^2}{r^2} + k^2 \right)
\right\} P = 0 \\
\dfrac{d^2P}{dr^2}
  + \left\{
    \dfrac{1}{r}
    - \dfrac{1}{\overline{\rho_0}} \cancelto{0}{\frac{d \overline{\rho_0}}{dr}}
    + \dfrac{1}{\omega - kW_{\infty}} 2k \cancelto{0}{\dfrac{dW}{dr}}
\right\} \dfrac{dP}{dr}
 + \left\{
    \underbrace{ \dfrac{(\omega - k W_{\infty} )^2}{\overline{a^2} }
    - k^2 }_{\lambda^2}
    - \dfrac{n^2}{r^2}
\right\} P = 0
% \dfrac{d^2P}{dr^2}
%   + \left\{
%     \dfrac{1}{r}
%   \right\} \dfrac{dP}{dr}
% + \left\{
%     \lambda^2
%     - \dfrac{n^2}{r^2}
% \right\} P = 0 \\
\end{align*}

Resulting in the homogeneous Bessel equation:

\begin{equation} \label{BesselEqn}
\boxed{ \dfrac{d^2P}{dr^2} + \left\{ \dfrac{1}{r} \right\} \dfrac{dP}{dr}
      + \left\{ \lambda_{\infty}^2 - \dfrac{n^2}{r^2} \right\} P = 0 }
\end{equation}

\noindent where the constant term is collected in $\lambda_{\infty}$:

\begin{align*}
\lambda_{\infty} &= \sqrt{\dfrac{(\omega - k W_{\infty} )^2}{\overline{a^2} } - k^2}
\end{align*}

Substituting the definition of wavenumber $k=\dfrac{\omega}{\overline{a}}$, the expression can be reduced:

\begin{align*}
\lambda_{\infty} &= \sqrt{
  \dfrac{ \left(\omega -\dfrac{\omega}{\overline{a}} W_{\infty} \right)^2}{\overline{a^2}}
  - \dfrac{\omega^2}{\overline{a^2}}
  } \\
&= \sqrt{
  \dfrac{ \omega^2 \left(1 -\dfrac{W_{\infty}}{\overline{a}} \right)^2}{\overline{a^2}}
  - \dfrac{\omega^2}{\overline{a^2}}
  } \\
&= \sqrt{ \dfrac{\omega^2}{\overline{a^2}}
    \left[ \left(1 -\dfrac{W_{\infty}}{\overline{a}} \right)^2 - 1  \right]
  } \\
&= \sqrt{ \dfrac{\omega^2}{\overline{a^2}}
    \left( 1 -2\dfrac{W_{\infty}}{\overline{a}} + \dfrac{W_{\infty}^2}{\overline{a^2}} - 1  \right) } \\
&= \sqrt{ \dfrac{\omega^2}{\overline{a^2}}
    \left( \dfrac{W_{\infty}^2}{\overline{a^2}} -2\dfrac{W_{\infty}}{\overline{a}} \right) } \\
&= \sqrt{ \dfrac{\omega^2}{\overline{a^2}} \dfrac{W_{\infty}}{\overline{a}}
    \left( \dfrac{W_{\infty}}{\overline{a}} - 2 \right) }
\end{align*}

Substituting the definition of Mach number $M_{\infty} = \dfrac{W_{\infty}}{\overline{a}}$, we obtain:

\begin{equation} \label{lambda}
\lambda_{\infty} = \sqrt{ \dfrac{\omega^2}{\overline{a^2}} M_{\infty} ( M_{\infty} - 2 ) }
\end{equation}

\noindent where $M_{\infty}$ is the freestream Mach number and $\lambda_{\infty}$ is an imaginary quantity for $M_{\infty} < 2$.

Eqn~\ref{BesselEqn} is of the same form as Bessel's ordinary differential equation and can be solved using the first $H_n^{(1)}(\lambda_{\infty} r)$ and second $H_n^{(2)}(\lambda_{\infty} r)$ order Hankel functions:

\begin{equation}
P = C H_n^{(1)}(\lambda_{\infty} r) + D H_n^{(2)}(\lambda_{\infty} r)
\end{equation}

\noindent where $C$ and $D$ are arbitrary constants and the first and second order Hankel functions are equal to the outgoing $H_n^{+}(x)$ and incoming $H_n^{+}(x)$ wave solutions, respectively, and $x = \lambda_{\infty} r$:

\begin{align}
H_n^{(1)}(\lambda_{\infty} r) = H_n^{+}(x) = J_n(x) + i Y_n(x) \\
H_n^{(2)}(\lambda_{\infty} r) = H_n^{-}(x) = J_n(x) - i Y_n(x)
\end{align}

\noindent where the Hankel functions are composed of the first and second order Bessel functions $J_n$ and $Y_n$, respectively.

Taking the limit of $H_n^{+}(x)$ and $H_n^{-}(x)$ as $x$ and $r$ approach infinity (far-field condition):

\begin{align*}
\lim_{x\to\infty} H_n^{+}(x) &= (-i)^{n+1} \dfrac{e^{ix}}{x} \\
\lim_{x\to\infty} H_n^{+}(x) &= i^{n+1}    \dfrac{e^{-ix}}{x}
\end{align*}

In the far-field, flow velocity is subsonic ($M_{\infty}<1$), making $\lambda_{\infty}$ imaginary according to Eqn~\ref{lambda}.  This, in turn, makes the limit of $H_n^{+}(x)$ a diminishing exponential and the limit of $H_n^{-}(x)$ an increasing exponential, which is impossible.  Thus, Eqn~\ref{BesselEqn} reduces to Eqn~\ref{BesselFar} for final solution of the far-field outside of the potential core of the jet:

\begin{equation} \label{BesselFar}
\boxed{P = C H_n^{(1)}(\lambda_{\infty} r)}
\end{equation}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Problem 3 -- General Solution for Jet Potential Core} \label{SecProb3}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

In this section, we will solve the homogeneous form of Lilley's equation (Eqn~\ref{lilleysoln}) to determine the general solution for the potential core region of the jet.  We will assume that the mean velocity and density in this region are constant and equal to the jet exit values:

\begin{align}
W(r)=W_j=const \\
\overline{\rho} = \overline{\rho_j} = const
\end{align}

These assumptions are similar to those made in Section~\ref{secprob2}, which allows the reduction of Eqn~\ref{lilleysoln} to the Bessel equation:

\begin{equation} \label{BesselEqnJet}
\boxed{ \dfrac{d^2P}{dr^2} + \left\{ \dfrac{1}{r} \right\} \dfrac{dP}{dr}
      + \left\{ \lambda_{j}^2 - \dfrac{n^2}{r^2} \right\} P = 0 }
\end{equation}

\noindent with

\begin{equation} \label{lambdaJ}
\lambda_{j} = \sqrt{ \dfrac{\omega^2}{\overline{a^2}} M_{j} ( M_{j} - 2 ) }
\end{equation}

\noindent where $M_{j}=\dfrac{W_j}{\overline{a}}$ is the jet exit Mach number and $\lambda_{j}$ is an imaginary quantity for $M_{j} < 2$.

For this potential core solution, we assume the Bessel equation is equivalent to the first $J_n$ and second $Y_n$ order Bessel functions:

\begin{equation}
P = A J_n(\lambda_j r) + B Y_n(\lambda_j r)
\end{equation}

\noindent where $A$ and $B$ are arbitrary constants.

The second order Bessel function $Y_n$ is defined as having a singularity where $\lambda_j r = 0$, so inside the potential core region where $0<r<R_j$, we can leave the singular $Y_n$ out of Eqn~\ref{BesselEqnJet}, resulting in the final expression for the solution inside of the potential core:

\begin{equation} \label{BesselCore}
\boxed{P = A J_n(\lambda_j r)}
\end{equation}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Problem 4 -- Matching of Far-Field and Potential Core Solutions}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

In Section~\ref{secprob2} we derived a solution for the pressure fluctuations ($P^+$) outside of the jet potential core in the far-field (Eqn~\ref{BesselFar}) and in Section~\ref{SecProb3} we derived a solution for the pressure fluctuations ($P^-$) inside the jet potential core (Eqn~\ref{BesselCore}).  Ideally, we would like to combine these near and far solutions by joining them at the edge of the potential core.

One method of accomplishing this solution is to replace the potential core boundary with a vortex sheet located at $r = R_j$.  To successfully join the two solutions, we require two matching boundary conditions at this vortex sheet, which can be described according the two following conditions:

\begin{enumerate}
  \item Pressure fluctuations on either edge of the vortex sheet must match:
        \begin{align} \label{constraint1}
          P^+ = P^-
        \end{align}
  \item Radial velocity fluctuations across the vortex sheet must match:
        \begin{align} \label{constraint2}
          V_r^+ = V_r^-
        \end{align}
\end{enumerate}

The first constraint (Eqn~\ref{constraint1}) of matched pressure fluctuation can be written using the Bessel equation solutions for the outer flow (Eqn~\ref{BesselFar}) and inner flow (Eqn~\ref{BesselCore}):

\begin{equation} \label{constantsEqn1}
\boxed{C H_n^{(1)}(\lambda_{\infty} r) = A J_n(\lambda_j r)}
\end{equation}

The second constraint will be achieved by relating the radial gradient of pressure $\dfrac{dP}{dr}$ with the radial velocity at the vortex sheet $V_r$:

\begin{equation}
V_r = \nu \exp[i(-\omega t + kz)]
\end{equation}

\noindent where $\nu$ is a constant value of velocity in the radial direction that is modulated by the expression $\exp[i(-\omega t + kz)]$, which is a function of time and the jet flow direction $z$.

We will first obtain an expression relating $V_r$ and the pressure fluctuation using the momentum equation:

\begin{equation} \label{momentum}
\overline{\rho_0} \dfrac{D\vec{V} }{Dt} = - \nabla P
\end{equation}

Because the matching conditions apply only across the vortex sheet in the radial direction, we look at the radial component of Eqn~\ref{momentum} by taking the dot product of both sides WRT the radial unit vector $\hat{r}$:

\begin{align*}
\overline{\rho_0} \dfrac{D\vec{V}}{Dt} \cdot \hat{r}
    &= - \nabla P \cdot \hat{r} \\
\overline{\rho_0} \dfrac{D V_r}{Dt} &= - \dfrac{\partial P}{\partial r}
\end{align*}

The total time derivative of $V_r$ is (see Section~\ref{SecLilley}):

\begin{align*}
\dfrac{D V_r}{Dt}
  &= \left(\dfrac{\partial}{\partial t} + W \dfrac{\partial}{\partial z}\right)
  V_r \\
  &= \left(\dfrac{\partial}{\partial t} + W \dfrac{\partial}{\partial z}\right)
  \nu \exp[i(-\omega t + kz)] \\
  &= - i( \omega - kW ) \nu\exp[i(-\omega t + kz)] \\
  &= - i( \omega - kW ) V_r
\end{align*}

Substituting $\dfrac{D V_r}{Dt}$ into Eqn~\ref{momentum}, we obtain:

\begin{align}
\overline{\rho_0} i( \omega - kW ) V_r = \dfrac{d P}{d r}
\end{align}

\noindent which can be rearranged in terms of $V_r$, allowing us to satisfy our second constraint:

\begin{align} \label{VrMomentum}
\boxed{V_r = \dfrac{\dfrac{d P}{d r}}{\overline{\rho_0} i( \omega - kW )}}
\end{align}

Next, we will define the radial velocity fluctuation $V_r$ instead as the time derivative of the radial displacement $\zeta$ of fluid particles located at the vortex sheet.

\begin{equation}
V_r = \dfrac{D\zeta}{Dt}
\end{equation}

\noindent where

\begin{equation}
\zeta(z,t) = \chi \exp[i(-\omega t + kz)]
\end{equation}

\noindent and where $\chi$ is a constant value of radial position that is modulated by the same expression $\exp[i(-\omega t + kz)]$, which governs the oscillation of the vortex sheet.

Taking the time derivative of $\zeta$ (see Section~\ref{SecLilley}):

\begin{align*}
V_r &= \dfrac{D\zeta}{Dt} =
    \left(\dfrac{\partial}{\partial t} + W \dfrac{\partial}{\partial z}\right)
    \zeta \\
    &=\left(\dfrac{\partial}{\partial t} + W \dfrac{\partial}{\partial z}\right)
    \chi \exp[i(-\omega t + kz)]  \\
&= -\chi \exp[i(-\omega t + kz)] i ( \omega - kW )  \\
&= -\zeta i ( \omega - kW )
\end{align*}

Applying the solution for $V_r$ to the outer and inner vortex sheet velocity fluctuations:

\begin{align}
V_r^+ &= -i(\omega - kW_{\infty} )\zeta  \label{Vplus}\\
V_r^- &= -i(\omega - kW_j )       \zeta  \label{Vminus}
\end{align}

Rearranging the solution of $V_r$ for $\zeta$, and setting Eqns~\ref{Vplus} and~\ref{Vminus} equal to each other:

\begin{align} \label{VrVel}
\boxed{\zeta = \dfrac{V_r^+}{-i(\omega - kW_{\infty})}
    = \dfrac{V_r^-}{-i(\omega - kW_{j})}}
\end{align}

Next, the second constraint (Eqn~\ref{constraint2}) can be put into terms of pressure rather than $V_r$ by substituting Eqn~\ref{VrMomentum} applied to the inner and outer solutions:

\begin{align*}
V_r^+ &=\dfrac{\dfrac{dP}{dr}^+}{\overline{\rho_0} i( \omega - kW_{\infty})} \\
V_r^- &= \dfrac{\dfrac{dP}{dr}^-}{\overline{\rho_0} i( \omega - kW_{j})}
\end{align*}

\noindent into Eqn~\ref{VrVel}:

\begin{align*}
\dfrac{V_r^+}{-i(\omega - kW_{\infty})}
    &= \dfrac{V_r^-}{-i(\omega - kW_{j})} \\
\dfrac{\dfrac{dP}{dr}^+}{-i^2(\omega - kW_{\infty})^2}
    &= \dfrac{\dfrac{dP}{dr}^-}{-i^2(\omega - kW_{j})^2} \\
\end{align*}

\begin{equation} \label{constraints}
\boxed{\dfrac{\dfrac{dP}{dr}^+}{(\omega - kW_{\infty})^2}
    = \dfrac{\dfrac{dP}{dr}^-}{(\omega - kW_{j})^2}}
\end{equation}

Finally, we can calculate the pressure differentials from the solutions for the outer flow (Eqn~\ref{BesselFar}) and inner flow (Eqn~\ref{BesselCore}):

\begin{align*}
\dfrac{dP}{dr}^+ &= \dfrac{d}{dr} \left[C H_n^{(1)}(\lambda_{\infty} r) \right]
    = C \dfrac{d}{dr}\left[H_n^{(1)}(\lambda_{\infty} r)\right] \\
\dfrac{dP}{dr}^- &= \dfrac{d}{dr} \left[A J_n(\lambda_j r) \right]
    = A \dfrac{d}{dr}\left[ J_n(\lambda_j r) \right]
\end{align*}

\noindent and substitute the resulting expressions into Eqn~\ref{constraints}:

\begin{align*}
\dfrac{C \dfrac{d}{dr}\left[H_n^{(1)}(\lambda_{\infty} r)\right]}{(\omega - kW_{\infty})^2}
    = \dfrac{A \dfrac{d}{dr}\left[ J_n(\lambda_j r) \right]}{(\omega - kW_{j})^2}
\end{align*}

\noindent Rearranging, we obtain a second equation containing the unknown constants $A$ and $C$.

\begin{equation} \label{constantsEqn2}
\boxed{
C(\omega - kW_{j})^2 \dfrac{d}{dr}\left[H_n^{(1)}(\lambda_{\infty} r)\right]
    = A(\omega - kW_{\infty})^2 \dfrac{d}{dr}\left[ J_n(\lambda_j r) \right]}
\end{equation}

Thus, achieving the two constraints set at the beginning of this section and substituting the known equations for pressure fluctuations inside and outside the potential core, we rewrite the resulting two equations (Eqns~\ref{constantsEqn1} and~\ref{constantsEqn2}) with two constant unknowns:

\begin{align*}
C H_n^{(1)}(\lambda_{\infty} r) &= A J_n(\lambda_j r) \\
C(\omega - kW_{j})^2 \dfrac{d}{dr}\left[H_n^{(1)}(\lambda_{\infty} r)\right]
    &= A(\omega - kW_{\infty})^2 \dfrac{d}{dr}\left[ J_n(\lambda_j r) \right]
\end{align*}

This system of ordinary differential equations can be solved to correctly determine $A$ and $C$ and thus provide an explicit solution for the combined interior and exterior axial jet flow.

It should be noted that this solution only applies for subsonic and supersonic jet flows with $M<2$ as previously demonstrated in Eqn~\ref{lambda}.

\end{document}


\documentclass{anstrans}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\title{Mixed Hybrid Finite Element Method Eddington Acceleration of Discrete Ordinates Source Iteration}
\author{Samuel S. Olivier*, Jim E. Morel}

\institute{Department of Nuclear Engineering, Texas A\&M University, College Station, TX 77843}

\email{smsolivier@tamu.edu}

%%%% packages and definitions (optional)
\usepackage{graphicx} % allows inclusion of graphics
\usepackage{booktabs} % nice rules (thick lines) for tables
\usepackage{microtype} % improves typography for PDF

\usepackage{xspace}
\usepackage{siunitx}

\newcommand{\SN}{S$_N$\xspace}
\renewcommand{\vec}[1]{\bm{#1}} %vector is bold italic
\newcommand{\vd}{\bm{\cdot}} % slightly bold vector dot
\newcommand{\grad}{\vec{\nabla}} % gradient
\newcommand{\ud}{\mathop{}\!\mathrm{d}} % upright derivative symbol
\newcommand{\pderiv}[2]{\frac{\partial #1}{\partial #2}}
\newcommand{\dderiv}[2]{\frac{\ud #1}{\ud #2}}
\newcommand{\edd}{\langle \mu^2 \rangle}

% add section on MHFEM system
% order of accuracy of accelerated system

\begin{document}
\section{Introduction}
	% Two of the most challenging computational tasks are radiation transport and hydrodynamics.
	One of the most challenging computational tasks is simulating the interaction of radiation with matter.
	A full description of a particle in flight includes three spatial variables ($x$,$y$ and $z$), two angular or direction of flight variables ($\mu =$ the cosine of the polar angle and $\gamma =$ the azimuthal angle), one energy variable ($E$) and one time variable ($t$). Numerical solutions require discretizing all seven variables leading to immense systems of algebraic equations. In addition, material properties can lead to vastly different solution behaviors making generalized numerical methods for radiation transport difficult to attain \cite{adams}.

	% The conservation of mass, momentum and energy in hydrodynamics simulations leads to a hyperbolic system of partial differential equations dependent on the time derivatives of velocity and two state variables. This results in five variables but only three equations leading to the requirement of additional equations to reach problem closure \cite{hydro}.

	% Radiation transport and hydrodynamics can be combined using operator splitting, where the radiation transport and hydrodynamics

	A national lab with whom we work has plans to develop a high-order radiation-hydrodynamics code. The hydrodynamics portion is discretized using the Mixed Hybrid Finite Element Method (MHFEM) where values are taken to be constant within a cell with discontinuous jumps at both cell edges \cite{mhfem}. MHFEM is particularly suited for hydrodynamics but not for radiation transport. This work seeks to develop an acceleration scheme capable of robustly reducing the number of iterations in Discrete Ordinates Source Iteration calculations while being compatible with MHFEM multiphysics.

\section{Background}
	The steady-state, mono-energetic, isotropically-scattering, fixed-source Linear Boltzmann Equation in slab geometry is:
		\begin{equation} \label{eq:bte}
			\mu \pderiv{\psi}{x}(x, \mu) + \Sigma_t(x) \psi(x,\mu) =
			\frac{\Sigma_s(x)}{2} \int_{-1}^{1} \psi(x, \mu') d\mu' + \frac{Q(x)}{2} \,,
		\end{equation}
	where $\mu = \cos\theta$ is the cosine of the angle of flight $\theta$ relative to the $x$--axis, $\Sigma_t(x)$ and $\Sigma_s(x)$ the total and scattering macroscopic cross sections, $Q(x)$ the isotropic fixed-source and $\psi(x, \mu)$ the angular flux \cite{adams}. The factors of 1/2 are consistent with the following definition of the scalar flux:
		\begin{equation} \label{eq:phiDef}
			\phi(x) = \int_{-1}^1 \psi(x, \mu) \ud \mu \,.
		\end{equation}
	Equation \ref{eq:bte} is an integro-differential equation due to the placement of the unknown, $\psi(x,\mu)$, under both a derivative and an integral.

	The Discrete Ordinates (\SN) angular discretization sets $\mu$ to discrete values stipulated by an $N$--point Gauss quadrature rule. The scalar flux is then
		\begin{equation} \label{eq:quad}
			\phi(x) = \int_{-1}^1 \psi(x, \mu) \ud\mu
				\xrightarrow{\text{S}_N} \sum_{n=1}^N w_n \psi_n(x) \,,
		\end{equation}
	where $\psi_n(x) = \psi(x,\mu_n)$ and the $w_n$ are the quadrature weights corresponding to each $\mu_n$ \cite{llnl}. To remain consistent with Eq. \ref{eq:phiDef}, the quadrature weights sum to 2. The \SN equations are then
		\begin{equation} \label{eq:sn}
			\mu_n \dderiv{\psi_n}{x}(x) + \Sigma_t(x) \psi_n(x) =
			\frac{\Sigma_s(x)}{2} \phi(x) + \frac{Q(x)}{2} \,,
		\end{equation}
	where $n = 1, 2, \dots, N$ and $\phi(x)$ is defined by Eq. \ref{eq:quad}. This is now a system of $N$ coupled, ordinary differential equations.

	The Source Iteration (SI) solution method decouples the \SN equations by lagging the right side of Eq. \ref{eq:si}. In other words,
		\begin{equation} \label{eq:si}
			\mu_n \dderiv{\psi_n^{\ell+1}}{x}(x) + \Sigma_t(x) \psi_n^{\ell+1}(x) =
			\frac{\Sigma_s(x)}{2} \phi^{\ell}(x) + \frac{Q(x)}{2} \,, 1 \leq n \leq N \,,
		\end{equation}
	where the superscripts indicate the iteration index.
	Equation \ref{eq:si} represents $N$ independent, first-order, ordinary differential equations each of which are easily solved by the well-known sweeping process.

	The iteration process begins with an initial guess for the scalar flux, $\phi^0(x)$. Equation \ref{eq:si} is solved, using $\phi^0(x)$ on the right side, for $\psi_n^1(x)$. $\phi^1(x)$ is then computed using Eq. \ref{eq:quad} and is used to update the right side of Eq. \ref{eq:si}.
	This process is repeated until
		\begin{equation} \label{eq:converg}
			\frac{\|\phi^{\ell+1}(x) - \phi^{\ell}(x)\|}{\|\phi^{\ell+1}(x)\|} < \epsilon \,,
		\end{equation}
	where $\epsilon$ is a sufficiently small tolerance.

	If $\phi^0(x) = 0$, then $\phi^\ell(x)$ is the scalar flux of particles that have undergone at most $\ell - 1$ collisions \cite{adams}. Thus, the number of iterations until convergence is directly linked to the number of collisions in a particle's lifetime. Typically, SI becomes increasingly slow to converge as the ratio of $\Sigma_s$ to $\Sigma_t$ approaches unity and the amount of particle leakage from the system goes to zero. SI is slowest in large, optically thick systems with small losses to absorption. In full radiation transport simulations each iteration could involve solving for hundreds of millions of unknowns. To minimize computational expense, acceleration schemes must be developed to rapidly increase the rate of convergence of SI.

	Fortunately, the regime where SI is slow to converge is also the regime where Diffusion Theory is most accurate. A popular method for accelerating SI is Diffusion Synthetic Acceleration (DSA) where each source iteration involves both a transport sweep and a diffusion solve. DSA requires carefully differencing the \SN and diffusion steps in a consistent manner to prevent instability in highly scattering media with coarse spatial grids \cite{alcouffe,morel}. DSA is not applicable in the setting of this presentation due to the incompatibility of MHFEM and \SN and the increased computational expense of solving consistently differenced diffusion. A new acceleration method is needed that avoids the consistency pitfall of DSA.

\section{Eddington Acceleration}
	The zeroth and first angular moments of Eq. \ref{eq:bte} are
		\begin{subequations}
		\begin{equation} \label{eq:zero}
			\dderiv{}{x} J(x) + \Sigma_a(x) \phi(x) = Q(x) \,,
		\end{equation}
		\begin{equation} \label{eq:first}
			\frac{\ud}{\ud x} \edd(x) \phi(x) + \Sigma_t(x) J(x) = 0 \,,
		\end{equation}
		\end{subequations}
	where $J(x) = \int_{-1}^{1} \mu \ \psi(x, \mu) \ud \mu$ is the current and
		\begin{equation} \label{eq:eddington}
			\edd(x) = \frac{\int_{-1}^1 \mu^2 \psi(x, \mu) \ud \mu}{\int_{-1}^1 \psi(x, \mu) \ud \mu}
			% \xrightarrow{\text{S}_N} \frac{\sum_{n=1}^N \mu_n^2 w_n\psi_n(x)}{\sum_{n=1}^N w_n \psi_n(x)}
		\end{equation}
	the Eddington factor. In \SN, the Eddington factor is
		\begin{equation} \label{eq:edd_sn}
			\edd(x) = \frac{\sum_{n=1}^N \mu_n^2 w_n\psi_n(x)}{\sum_{n=1}^N w_n \psi_n(x)} \,.
		\end{equation}
	Note that no approximations have been made to arrive at Eqs. \ref{eq:zero} and \ref{eq:first}. The Eddington factor is the true angular flux weighted average of $\mu^2$ and therefore Eqs. \ref{eq:zero} and \ref{eq:first} are just as accurate as Eq. \ref{eq:bte}.

	This formulation is beneficial because Eq. \ref{eq:zero} is a conservative balance equation and---if $\edd(x)$ is known---the moment equations' system of two first-order, ordinary differential equations can be solved directly with well-established methods. However, computing $\edd(x)$ requires knowledge of the angular flux.

	In Eddington Acceleration, \SN is used to compute the Eddington factor needed to solve the moment equations. Source iteration is then:
		\begin{enumerate}
			\item Given the previous estimate for the scalar flux, $\phi^{\ell}(x)$, solve Eq. \ref{eq:si} for $\psi_n^{\ell+1/2}(x)$.
			\item Compute $\edd^{\ell+1/2}(x)$ with Eq. \ref{eq:edd_sn}.
			% \item Interpolate $\edd(x)$ onto the MHFEM grid
			\item Solve the moment equations for $\phi^{\ell+1}(x)$ using $\edd^{\ell+1/2}(x)$.
			% \item Use the moment equations' $\phi(x)$ on the right hand side of Eq. \ref{eq:si}.
			\item Update the scalar flux estimate on the right side of Eq. \ref{eq:si} with $\phi^{\ell+1}(x)$ and repeat the iteration process until the scalar flux converges.
		\end{enumerate}
	% This process is one source iteration consisting of an \SN transport step to compute the Eddington factor and an MHFEM acceleration step to compute $\phi(x)$. The scalar flux from the acceleration step is used in the right hand side of Eq. \ref{eq:si} and steps 1--4 are repeated until the acceleration step's $\phi(x)$ converges according to Eq. \ref{eq:converg}.
W
	Acceleration occurs because the angular shape of the angular flux, and thus the Eddington factor, converges much faster than the scalar flux. In addition, the moment equations model the contributions of all scattering events at once, reducing the dependence on source iterations to introduce scattering information. The solution from the moment equations is then an approximation for the full flux and not the $\ell - 1$ collided flux as it was without acceleration.

	In addition to acceleration, this scheme allows the \SN equations and moment equations to be solved with arbitrarily different spatial discretization methods. \SN can be spatially discretized using normal methods, such as Linear Discontinuous Galerkin or Diamond Difference, while the moment equations can be solved with MHFEM.

\section{Results}
	\begin{figure}
		\centering
		\includegraphics[width=8.5cm]{figs/accel.pdf}
		\caption{A comparison of the number of iterations until convergence for unaccelerated, Eddington accelerated, and DSA S$_8$ SI. }
		\label{fig:comparison}
	\end{figure}

	As a proof of concept for Eddington Acceleration, a Diamond Differenced \SN code was created along with an MHFEM solver for Eqs. \ref{eq:zero} and \ref{eq:first}. The test problem of steady-state, one-group, isotropically-scattering, fixed-source radiation transport in slab geometry with a reflecting left boundary and vacuum right boundary was used to compare unaccelerated, Eddington accelerated, and DSA S$_8$ SI. The slab had a thickness of \SI{20}{cm} and was discretized into 100 spatial cells. The total macroscopic cross section was set to \SI{1}{cm^{-1}} leading to a total optical thickness of 20 and an optical thickness per cell of 0.2.

	Figure \ref{fig:comparison} shows the number of iterations until the convergence criterion in  Eq. \ref{eq:converg} was met with $\epsilon = \num{1e-6}$ for varying ratios of $\Sigma_s$ to $\Sigma_t$. Aside from $\Sigma_s/\Sigma_t = 0$ where acceleration is not possible, the ratio of unaccelerated to Eddington accelerated iterations ranged between 2.5 and 750. This suggests that acceleration is occurring and that Eddington Acceleration does not just do twice the amount of work in each iteration.

	\begin{figure}
		\centering
		\includegraphics[width=8.5cm]{figs/eddCon_si.pdf}
		\caption{The convergence rate of $\phi(x)$ compared to $\edd(x)$ for unaccelerated S$_8$ with $\Sigma_s/\Sigma_t = 0.75$. }
		\label{fig:conv_si}
	\end{figure}

	Figure \ref{fig:conv_si} shows the convergence criterion
		\begin{equation}
			\frac{\|f^{\ell+1} - f^{\ell}\|}{\|f^{\ell+1}\|}
		\end{equation}
	as a function of unaccelerated iteration number for $f = \phi(x)$ and $f = \edd(x)$. The large drop in the convergence criterion between the first and second iterations supports the claim that the angular shape of the angular flux, and thus the Eddington factor, converges much more rapidly than the scalar flux. When compared to Fig. \ref{fig:conv_edd}, a plot of the convergence criterion versus number of iterations for Eddington accelerated S$_8$, it is clear that Eddington Acceleration transfers the fast rate of convergence of $\edd(x)$ to $\phi(x)$.

	\begin{figure}
		\centering
		\includegraphics[width=8.5cm]{figs/eddCon_mu.pdf}
		\caption{The convergence rate of $\phi(x)$ compared to $\edd(x)$ for Eddington accelerated S$_8$ with $\Sigma_s/\Sigma_t = 0.75$. }
		\label{fig:conv_edd}
	\end{figure}

	% add convergence rate of phi v edd
	% include discussion on acceleration v just doing 2 times as much work. ie actual acceleration is happening

\section{Conclusions}
	The proposed acceleration scheme successfully accelerated S$_8$ source iteration calculations in slab geometry for a wide range of $\Sigma_s/\Sigma_t$. In the pure scattering regime ($\Sigma_s = \Sigma_t$), source iteration was accelerated by a factor of 750. This scheme is especially suited for multiphysics applications because the transport and acceleration steps do not need to be consistently differenced. In addition, the acceleration step produces a conservative solution that is computationally inexpensive compared to a transport sweep. Future work that will also be presented is the application of Eddington Acceleration to Linear Discontinuous Galerkin discretized \SN.

% \section{Acknowledgments}

\bibliographystyle{ans}
\bibliography{../bibliography.bib}
\end{document}


\chapter{Conclusion}\label{chap:conclusion}
In order to combat the issue of contamination of publicly accessible water supplies, namely fecal contamination, the \cp{} Biological Sciences Department teamed up with the \cp{} Computer Science Department to build a library-based \mst{} method, called \cplop{}.
Using the \ecoli{} isolated from fecal samples as \fiblong{}, \cp{} students pyrosequence the \pcr{}-amplified \itslong{} regions of the \ecoli{} and store the resulting vector, called a \pyro{}, in the \cplop{} database for later retrieval, analysis, and comparison.
This thesis investigates two \mst{} methodologies: a \dbased{} clustering method built for \cplop{} that clusters for \bslongs{} in order to classify an unknown \isol{} and the \kraplong{} (\krap{}), a set of four \knnlong{} list resolution strategies for data with multiple \compfuncs{}.

\section{Clustering for \BSlongs{}}

In this paper, we study the accuracy of a clustering-based MST approach which scales significantly better: the bacterial isolate information stored in \cplop{} is clustered using an efficient density-based clustering technique.
It clusters data by taking two parameters --- the minimum number of neighbors and an \eps{} range that those neighbors must be within to form a cluster --- and performs range queries in a spatial index that performs \bigo{\log{n}} look-up on neighbors using a comparison metric.
Compared to previous work \cite{DBLP:conf/bibm/McGovernDKBVG15, montana2013ontological}, it requires fewer comparisons to other \isols{} and computational resources, by being able to perform reasonably fast clusterings on a consumer laptop in minutes.

To ascertain how well this technique classifies, we build a notion of cluster purity.
By calculating the proportion of the entire cluster that the most-plural \spec{} makes up, we hope to understand how a density-based clustering algorithm clusters the \cplop{} data.
Furthermore, we inspect coverage and overall accuracy.

Results are that for \minneigh{} between 1 and 5, respectively, we are able to cluster between 72.9\% and 52.1\% of the \isols{} in \cplop{}, with between 51.0\% and 41.2\% falling into pure clusters and another 34.4\% to 43.8\% falling into clusters where their \spec{} is the most dominant.
Most clusters have high purity and low number of unique species, which is promising for using this for \mst{}.
Transient strains are also visible in the clustering technique, which will further aid the biologists working on \cplop{} in researching transient strains and improving \mst{} techniques.
Future work will leverage other \mst{} techniques designed for \cplop{} against this to make up for the lack of coverage and transient strains.
\section{\krap{} Effectiveness}
Generally, when using \kNN{}, it is preferred to use single digit $k$ values. Through our investigation of these various \kNN{} classification algorithms, we find that that general advice holds true. For our dataset, using $k\geq5$ does not produce significantly different results. Choosing $k<5$ is a dangerous notion, since it is likely that an outlier may make its way into the \knnlong{} list, confounding the results. Staying with $5\leq k\leq9$ appears to be a safe and reasonable option, providing a good balance between accuracy and filtering of isolates.

%\subsection{Choosing $\alpha$}
Outside of this study, we choose to differentiate between strains of \ecoli{} using $\alpha=0.99$. It appears that using this value is advantageous. There were, however, some exceptions to those results, motivating us to consider non-thresholded \knnlong{} lists when classifying an unknown isolate.

%\subsection{Choosing a Resolution Algorithm}
The four resolution algorithms --- meanwise, winner, union, and intersection --- each have their own quirks and behaviors as we alter $k$ and $\alpha$.

Meanwise, which currently uses the Euclidean norm to resolve different metrics, did not respond to the $\alpha$ threshold and completely stopped classifying anything for $\alpha$ near 1. This is very likely due to Euclidean norm mapping $([0,1],\dots,[0,1])\rightarrow [0,\sqrt{1+\dots+1}]$. To get around this, we multiplied the resulting norm by a factor of $\sqrt2$, which may have unexpected results. We may investigate this further, or choose a more natural norming method, like arithmetic or geometric mean. With no $\alpha$ filtering, it performed third best with an overall 73.2\% classification accuracy.

Winner performs worst, classifying accurately between 65\% and 68\% of the time. Some alterations to this algorithm may make it more reliable, such as only counting the species that appear in all lists.

Unionwise performs very well. Without filtering the \knnlong{} lists by $\alpha$, we find that the unionwise method classifies best, with an overall accuracy of 76.4\%. However, once we add in $\alpha$ filtering, the unionwise does not improve, staying relatively close to 76\%.

Intersection performs best when we use $\alpha$. This is likely due to the ``list'' actually being a set of common isolates. Overall, the accuracy was 74.7\%, 78\%, and 85.9\% for $\alpha = $0.00 (no filtering), 0.98, and 0.99 respectively.

Overall, we find that the intersection algorithm performs the best and recommend moving forward with it. While unionwise did perform well, it did not respond well to thresholding and still did not perform as well as the intersection algorithm overall. Meanwise and winner may be more useful with previously mentioned modifications and we may investigate these in the future.

%The high performance of the intersection method is likely due to the fundamental structure and strain differentiation methods we use in CPLOP. Give the pyroprint of two \ecoli{} cultures, a Pearson Correlation within 0.99 defines a strain. Restricting $\alpha$ to values near this limit gave us better overall accuracy, since we get are allowed to search a wide distance around the \isol{} to find \isol{}s that mathch both regions, and we filter out \isol{} that are a different strain. While we find that resolution by intersection is a very good method, we will likely still use variations of the other methods to aid in classification.

%\subsection{Species Representation}
Poorly distributed representation of species and environmental incomparabilities are issues endemic to library-based MST. CPLOP has an overabundance of Cow and Human \isol{}s, and an underrepresentation of many of the species in the database. This dilutes the \knnlong{} list considerably for species like the Chicken and Cat.

Library population issues aside, environmental limitations are another concern for accuracy. Nearly every sample in the library comes from a 30 mile radius around Cal Poly, making the collected \pyro{}s potentially incomparable to \pyro{}s collected from a different region.

%Interestingly, environment factors in in another way, which our data shows about Bats . Classifying Bats is highly accurate and may be due to their limited involvement in other species' environment and small size. Cows have a wide variety of \ecoli{} strains, which allows them to show up in many other species' \knnlong{} lists. Bats on the other hand only have 37 \isol{}s in CPLOP all from a single \host{}, yet manage to be above 95\%, regardless of the method. We hypothesize that this is due to their limited interaction and we may investigate further.

\section{Future Work}
Future work should incorporate \krap{} into \cplop{}, study the \dbscan{} clustering method using the overall clustering entropy, and investigate whether combining the strain-based and \isol{}-based approach improves \mst{} and is more efficient on the \cplop{} database.
Incorporating \krap{} into \cplop{} will provide researchers the ability to make transparent, repeatable assertions as to the \spec{} of an unknown \isol{}.
Investigating clustering entropy can give us more insight into the makeup of clusters and extending this concept to \krap{} classifications, to measure how ``close the competition is'' between \spec{} in the \knnlong{} lists.
Merging strains into the \krap{} methods will reduce the number of comparisons, since querying against the database will involve querying against clusters of \isols{}, as opposed to \isols{} themselves.
Whether this benefits the accuracy of \mst{} with \cplop{} needs to be investigated.

\subsection{\cplop{} Incorporation of \krap{}}
Future work needs to incorporate \krap{} into \cplop{} directly, so \cp{} researchers have direct access to the \mst{} methodologies.
\cplop{} researchers showed interest in viewing the \knnlong{} lists when using \krap{} on an \isol{}.
Building an interactive \mst{} workflow can give researchers a better insight into \spec{} determinations and help them avoid having to painstakingly perform \mst{} by hand.
Future work needs to at least make \krap{} available to researchers in \cplop{} and should consider building an visually interactive way of using it.

\subsection{Entropy}
Entropy \index{clustering entropy} is another validity measure that for clusterings, represents the ``degree to which each cluster consists of objects of a single class'' \cite{tan2006introduction} and for \krap{} can tell us how contentious the \spec{} determination was.
Similar to the cluster and clustering purity, cluster and clustering entropy can give us an idea of the nature of \bslongs{} created by a clustering method.
Ideally, a good clustering method minimizes the clustering entropy.

Consider a cluster $C$, consisting of datapoints with class labels from $\mathcal{L}$.
Given a class label $L\in\mathcal{L}$, the value $P_C(L)$ calculates the proportion of datapoints in $C$ that have class label $L$.
The individual \textit{cluster entropy} is:
\index{cluster entropy}
\begin{equation}
e(C) = \sum_{L\in\mathcal{L}}^{} P_C(L)\log_2{P_C(L)}
\end{equation}

In addition to computing the entropy of individual clusters we want to have an understanding of the overall entropy on the entire dataset for a given clustering.
As before, given a \textit{clustering} $\mathcal{C} = \{C_1,\dots,C_n\}$ on a dataset, we define the size $\mathcal{M}$ of the set of clusters:
\index{clustering}
\begin{equation}
\mathcal{M} = \sum_{i = 1}^{n} |C_i|
\end{equation}
The overall \textit{clustering entropy} is:
\index{clustering entropy}
\begin{equation}\label{eq:clustering:entropy}
\sum_{i=1}^{n} \frac{|C_i|}{\mathcal{M}}\cdot\nu(C_i)
\end{equation}
One can think of \eqref{eq:clustering:entropy} as a form of weighted arithmetic mean of the individual entropies.
Larger clusters, as a result, affect this overall value more.
Future work should use this metric to compare clusterings with different parameters or methodologies, seeking to minimize the entropy, and extend the analysis to \krap{} classifications.

\subsection{\kNClong{}}
Combining the strain-based and \isol{}-based method of classification is a natural next step.
Using \krap{} as a simple fallback method when the strain-based method fails to classify an unknown \spec{} \isol{} is one approach.
Incorporating strains directly into \krap{} is yet another, wherein the neighbors in the \knnlong{} list may be either \isols{} or strains --- which we can represent as clusters.
As a result, we would instead have a \kNClong{} classification algorithm, through which, we can still apply the resolution strategies from \krap{}.
Future work should consider different cluster weighting methods for the \spec{} in clusters that appear in the \knnlong{} list and determine whether any of them are useful for \mst{}.

\subsection{Efficiency Study}
The efficiency of the classification methodologies in this work and any combinations thereof needs to be investigated as well as their performance on the hardware that supports the \cplop{} database.
Offline determination of strains, be it through clustering or otherwise, can occur offline, speeding up the determination of strain membership during \mst{}.
Combining the two approaches in this work into a \kNClong{} methodology may also speed up \mst{}.
\kNN{}, and \krap{} as a result, compares the unknown \isol{} to every datapoint in the database, but if instead the datapoints might be clusters, fewer comparisons may need to be made.