\chapter{Introduction to the SSFM for simulating superfluid vortex systems}
\label{ch:splitop}
The Split-Step Fourier Method (SSFM) is an essential technique for simulating a variety of physical systems and is particularly useful for simulating the propagation of wave packets in single and multimode fibers~\cite{agrawal2000, sinkin2003, meirelles2005, min2003} and in various quantum systems~\cite{bayindir2015, weideman1986, wang2005}.
Though other methods, such as explicit and implicit Euler~\cite{butcher2016}, Crank-Nicholson~\cite{crank1947}, and Runge-Kutta~\cite{butcher2016}, can solve similar differential equations, the SSFM has distinct advantages over these methods.
For example, the SSFM relies on embarrassingly parallel element-wise matrix multiplications and Fast Fourier Transform (FFT) routines that have been optimized for parallel and distributed systems, but it also requires less memory than Runge-Kutta, which requires multiple arrays of the size of the wavefunction in memory~\cite{murray2011}.
The SSFM also provides a lower error bound than either the Euler or Crank-Nicholson methods, and does not require an implicit or tridiagonal solver \cite{conte2017, thomas1949} which are also not easily parallelizable~\cite{goddeke2010, wang1981, sweet1977}.
In addition, the SSFM is faster and requires fewer FFTs per step than similar methods, such as Runge-Kutta 4 in the Interaction Picture (RK4IP)~\cite{brehler2017}; however, the SSFM method's speed comes at a cost in accuracy.
Though much of this work focuses on using the SSFM to simulate superfluid vortex states, I will not be discussing alternative methods, such as point-vortex~\cite{benzi1992} or vortex-filament~\cite{schwarz1988} models in rigorous detail, as this work focuses primarily on engineering appropriate quantum states, while point-vortex and vortex filament methods focus primarily on vortex structures, themselves.

In the work presented in this chapter, I will be focusing on the application of the SSFM to superfluid vortex simulations and will use primarily physical arguments to understand the details of the method itself.
More details about General Purpose computing with Graphics Processing Units (GPGPU) and the GPUE (Graphics Processing Unit Gross-Pitaevskii Equation Solver) simulation software can be found in Chapter~\ref{ch:gpu}.
There, I will discuss several additional areas of interest for implementing similar solvers on GPUs, including distributed transposes and important considerations for traditional FFT routines for simulating quantum systems on multiple GPU devices.

This chapter will assume familiarity with basic principles of quantum mechanics and will focus on considerations for simulating quantum systems with the SSFM.
As such, I will also introduce important physical insights for understanding ultracold atomic systems that will be used throughout the rest of this work.
In particular, I will focus on understanding superfluid systems created by Bose--Einstein condensation and methods by which vortex structures can be generated and controlled in a Bose--Einstein Condensate (BEC).

\section{The SSFM}
Let us begin with the single-particle Schr\"odinger equation,

\begin{equation}
i\hbar \frac{\partial \Psi(\mathbf{r}, t)}{\partial t} = \left(\frac{\hat p^2}{2m} + V_0(\mathbf{r}) \right)\Psi(\mathbf{r},t)
\label{eqn:schrody}
\end{equation}

\noindent where $\hat p = -i\hbar\frac{\partial}{\partial \mathbf{r}}$ is the canonical momentum operator, $m$ is the mass, $V_0(\mathbf{r})$ is the potential, and $\Psi(\mathbf{r},t)$ is the single-particle wavefunction.
In this case, one often replaces most of the right-hand side of the equation with a Hamiltonian operator, which for this case would be $\mathcal{\hat H} = \frac{\hat p^2}{2m} + V_0(\mathbf{r})$.
Noticeably, this has two components, one acting in position-space, $\mathcal{\hat H}_v = V_0(\mathbf{r})$ and another in momentum-space, $\mathcal{\hat H}_p = \frac{\hat p^2}{2m}$.
For consistency, I will denote all variables in momentum-space with a $p$, and real-space with a $v$.
Additionally, any wavefunction can be expanded into a complete set of eigenkets of the Hamiltonian, with $\mathcal{\hat H}\ket{\phi_n} = E_n\ket{\phi_n}$,
which allows one to write

\begin{equation}
\ket{\Psi(\mathbf{r})} = \sum_{n=0}^\infty c_n \ket{\phi_n(\mathbf{r})},
\end{equation}

\noindent where $c_n$ is a constant for each constituent eigenfunction $\psi_n(\mathbf{r})$.

Simply stated, the SSFM splits the Hamiltonian into separate operators and uses a Fourier transform on the wavefunction to ensure that these operators are applied in the appropriate space.
In order to apply the Hamiltonian to the system, one first assumes a formal solution to the Schr\"odinger equation,

\begin{equation}
\Psi(\mathbf{r},t + dt) = \left[e^{-\frac{i\mathcal{\hat{H}}dt}{\hbar}}\right]\Psi(\mathbf{r},t) = \left[e^{-\frac{i(\mathcal{\hat{H}}_v + \mathcal{\hat{H}}_p)dt}{\hbar}}\right]\Psi(\mathbf{r},t),
\end{equation}

\noindent where $dt$ is a small timestep.
If the system is being simulated in a timestepping manor with a series of small timesteps, one can split this operation by using the Baker-Campbell-Hausdorff formula,

\begin{equation}
\Psi(\mathbf{r},t+dt) = \left[e^{-\frac{i\mathcal{\hat{H}}_vdt}{\hbar}}e^{-\frac{i\mathcal{\hat{H}}_pdt}{\hbar}}e^{-\frac{[i\hat{H}_v, i\hat{H}_p]dt^2}{2}}\right]\Psi(\mathbf{r},t).
\label{eqn:rsolve}
\end{equation}

\noindent If neglected, the commutation of the real and momentum-space components of the Hamiltonian will accrue an error on the order of $dt^2$.
This is noticeably high;
however, the $dt^2$ error can be decreased to $dt^3$ by performing a half-step in position space before doing a full-step in momentum space, through a process called Strang splitting~\cite{strang1968},

\begin{equation}
\Psi(\mathbf{r},t+dt) = \left[e^{-\frac{i\mathcal{\hat{H}}_vdt}{2\hbar}}e^{-\frac{i\mathcal{\hat{H}}_pdt}{\hbar}}e^{-\frac{i\mathcal{\hat{H}}_vdt}{2\hbar}} \right]\Psi(\mathbf{r},t) + \mathcal{O}(dt^3).
\end{equation}

Strang splitting can be best understood by performing a Taylor series expansion on both $e^{h(\mathbf{A}+\mathbf{B})}$ and $e^{h\mathbf{A}}e^{h\mathbf{B}}$, where $\mathbf{A}$ and $\mathbf{B}$ are matrices and $h$ is a defined step size~\cite{macnamara2016}.
When expanded,
\begin{equation}
e^{h(\mathbf{A}+\mathbf{B})} \approx \mathbf{I} + h(\mathbf{A} + \mathbf{B}) + \frac{1}{2}h^2(\mathbf{A} + \mathbf{B})^2,
\label{eqn:tstrang}
\end{equation}

\noindent where $\mathbf{I}$ is the identity matrix.
Here, the first-order terms for the expansion in Equation~\eqref{eqn:tstrang} are identical to the expansion of $e^{h\mathbf{A}}e^{h\mathbf{B}}$; however, when expanding the last term, one finds,
\begin{equation}
\frac{1}{2}h^2(\mathbf{A} + \mathbf{B})^2 = \frac{1}{2}\left( \mathbf{A}^2 + \mathbf{AB} + \mathbf{BA} + \mathbf{B}^2\right).
\end{equation}

\noindent Here, the $\mathbf{BA}$ term is missing in the expansion of $e^{h\mathbf{A}}e^{h\mathbf{B}}$ because $\mathbf{A}$ always comes before $\mathbf{B}$.
If a symmetric splitting is used instead, it is clear that all the terms up to the second order are the same, such that $e^{h(\mathbf{A}+\mathbf{B})} \approx e^{h\mathbf{A}/2}e^{h\mathbf{B}}e^{h\mathbf{A}/2}$.

Because position and momentum are conjugate domains, after Strang splitting one can address each part of this solution in chunks, first in position space, then in momentum space, then in position space again by using Fourier transforms.
\begin{equation}
\Psi(\mathbf{r}, t+dt) = \left[\hat{U}_r(dt)\mathcal{F}^{-1}\left[\hat{U}_p(dt) \mathcal{F} \left[\hat{U}_r(dt) \Psi(\mathbf{r},t) \right] \right] \right] + \mathcal{O}(dt^3)
\end{equation}
\noindent where $\hat{U}_r = e^{-\frac{i\mathcal{\hat{H}}_vdt}{2\hbar}}$, $\hat{U}_p = e^{-\frac{i\mathcal{\hat{H}}_pdt}{\hbar}}$, and $\mathcal{F}$ and $\mathcal{F}^{-1}$ indicate forward and inverse Fourier transforms.
In practice, these Fourier transforms are performed with Fast Fourier Transforms (FFTs), typically using a variation on the Cooley-Tukey method, which was first discovered by Gauss and later contemporized by Cooley and Tukey when they independently discovered it \cite{cooley1965}.
This method is not straightforwardly parallelizable; however, FFTs have become so fundamental to signal processing, that they have been incredibly well-optimized with several libraries, including FFTW~\cite{frigo1998} and CuFFT~\cite{fatica2008} for distributed and GPU calculations, respectively.
I will discuss optimal techniques for using FFTs with the SSFM method in Chapter~\ref{ch:gpu}.

Each timestep of the SSFM is essentially composed of the following steps:

\begin{enumerate}
\item Multiply the wavefunction in real space with the real-space operator by using a half-step in position space.
\item Flip to momentum space with an FFT operation on the wavefunction.
\item Multiply the momentum-space wavefuntion by the momentum-space operator.
\item Flip to position space with an inverse FFT on the wavefunction.
\item Repeat 1-4 until satisfied.
\end{enumerate}

With the method described so far, one can simulate simple, one-dimensional quantum systems.
For example, if one uses a Gaussian wavefunction which is offset from the center of a simple harmonic oscillator, one can simulate the wavefunction oscillation, as shown in Figure~\ref{fig:evolve}(a).

\begin{figure}

\center \includegraphics[width=0.6\textwidth]{data/splitop/SHO/SHO_gimp.pdf}

\caption{Evolution of a one-dimensional simple harmonic oscillator in (a) real, and (b) imaginary time after slightly shifting the trapping potential in the $\hat x$ direction.
In (c), the energy as a function of time is shown and one can see that the energy of the system when evolving in real time remains constant, but in imaginary time it will decay to the known ground-state energy of the simple harmonic oscillator.
The simulated results are from evolution with the SSFM after 10,000 steps.
Here, I use a $^{87}$Rb atom with $\omega_x = 10$ Hz on a 256-point grid of size 200 $\mu$m, at $t = 0$, where the trap has been shifted by 5 $\mu m$.
The wavefunction has been normalized such that $\int_{-\infty}^\infty|\Psi|^2 dx = 1$.
This simulation was performed with the GPUE codebase \cite{schloss2018}.}
\label{fig:evolve}
\end{figure}

In addition to this, one can find the lowest energy state of the system by performing a Wick rotation and using $\tau = it$ for the simulation instead of traditional units of time~\cite{wick1954}.
This changes the solution from the complex sinusoid shown in Equation~\eqref{eqn:rsolve} to an exponential decay,

\begin{equation}
\Psi(\mathbf{r},\tau + d\tau) = \left[e^{-\frac{\mathcal{\hat{H}}d\tau}{\hbar}}\right]\Psi(\mathbf{r},\tau) = \sum_{n=0}^N\left[c_n e^{-\frac{(E_n d\tau)}{\hbar}}\right]\phi_n(\mathbf{r},\tau).
\end{equation}

\noindent Note that it is not necessary to include the $c_n$ coefficients in the expansion as this form of time stepping is no longer unitary.
Overall, imaginary time evolution has two notable effects:
\begin{enumerate}
\item There will be an exponential decay of all energy states, with higher-energy states decaying faster than the ground-state.
\item The non-unitary evolution requires renormalization at every time step in imaginary time evolution.
\end{enumerate}
Let us start with a discussion on the first point, by showing a simulation of the same system as in Figure~\ref{fig:evolve}(a), but in imaginary time, shown in Figure~\ref{fig:evolve}(b).
Here, the wavefunction density is shifting to the center of the trap, and
in Figure~\ref{fig:evolve}(c), the energy is decaying to the known ground-state energy of a quantum harmonic oscillator of $\frac{1}{2}\hbar\omega = 3.31\times 10^{-33}$J for the chosen parameters.
This is because all eigenstates of the wavefunction are affected by the exponential decay, with the ground state decaying the slowest, and because renormalization occurs every timestep, only the ground state will survive imaginary time propagation.

The energy is computed as,
\begin{equation}
E = \braket{\Psi(\mathbf{r})|\hat H|\Psi(\mathbf{r})}.
\label{eqn:energy}
\end{equation}
For many systems, one can assume the simulation has reached the ground state when the energy converges to a fixed value.
More quantitatively, this means that the simulation can be stopped when the change in energy every step in imaginary time is below some provided threshold; however, this is not always the best course-of-action.
Further discussion on energy calculations performed in this work and the appropriate convergence criteria can be found in Chapter~\ref{ch:gpu}.

Now to discuss the second point that non-unitary evolution requires renormalization, which is problematic from a software perspective.
In practice, every step in imaginary time requires a relatively costly renormalization step,
\begin{equation}
    \label{eqn:norm}
    \int_\infty^\infty |\Psi(\mathbf{r},t)|^2 d\mathbf{r} = 1.
\end{equation}

\noindent Computationally, this operation is a summation, which is not well-optimized for GPU hardware; however, it is possible to perform a parallel reduction (summation), which allows for a considerable improvement on massively parallel systems~\cite{harris2007}.
Even so, the normalization is still a slow operation and should be used sparingly.

The implementation of the SSFM provided here assumes large-scale element-wise matrix multiplications in position and momentum-space; however, even though this implementation lends itself to parallelization, I will show other methods in Chapter~\ref{ch:gpu} when I discuss the implementation of the SSFM on graphics processing units.

It is important to note that the complexity of the SSFM is similar to the complexity of a convolution via the convolutional theorem,
\begin{equation}
f * g = \mathcal(F)^{-1}\left(\mathcal{F}(f)\cdot\mathcal{F}(g)\right).
\end{equation}
Here, $f$ and $g$ are arbitrarily chosen functions to be convolved.
In fact, interpreting the method as a series of convolutions can lead to important insight as to how the method operates.
For example, in imaginary time, the momentum-space operator becomes a Gaussian and the position-space operator becomes a more tightly-confining potential.
In this way, every time-step in imaginary time corresponds to a blurring operation with the momentum-space step and strong confinement with the position-space steps.
Real-time propagation can be interpreted in a similar way, as the momentum-space operator is similar to the Fourier transform of two Sobel filters, thereby performing two spatial derivatives.
For now, I will turn the focus to systems to be simulated via the SSFM throughout this work: ultracold atoms.

\section{Introduction to ultracold quantum systems}
\label{sec:intro}

When atomic systems are cooled to temperatures near zero Kelvin, it becomes easier to discern their quantum properties which vary drastically depending on whether the particles are bosonic or fermionic.
Because fermions have half-integer spin, they must obey the Pauli exclusion principle and are constrained to Fermi--Dirac statistics.
At zero temperature,
this creates a \textit{Fermi sea}, where the particles fill the single particle energy levels from the bottom-up with two particles of opposite spin per level.
On the other hand, bosons have integer spin and follow Bose--Einstein statistics.
They will condense into a single, macroscopic ground state when cooled~\cite{einstein1925, fetter2003}, and
this state of matter is known as a Bose--Einstein Condensate (BEC).
A BEC has the properties of a superfluid, which will be discussed more completely in the following section.

There are notable exceptions to these rules, such as the highly correlated Tonks--Girardeau gas where bosons may act as spinless, non-interacting fermions \cite{girardeau1960, schloss2016}.
It is also possible for interacting fermions to condense into a BEC-like state by forming molecules with integer spin~\cite{nozieres1985, bulgac2014}; however, I will not discuss fermionic systems further in this work.
For now, I will focus on BEC systems, but will also discuss Tonks--Girardeau gases later in Chapter~\ref{ch:1d}.

\subsection{Bose--Einstein condensation and the Gross--Pitaevskii Equation}

To motivate the formalism regularly used to describe BEC systems, I will follow a straightforward derivation using the second quantization~\cite{aversa2008}.
As mentioned in Section~\ref{sec:intro}, bosons in a BEC occupy a single ground state, meaning one must introduce a many-body Hamiltonian for the system and take inter-particle interactions into account.
Because experimental systems are dilute, I will only consider two-body interactions and assume any interactions between three or more atoms to be unlikely and negligible.
We can write the Hamiltonian with two body interactions in the second quantized form as
\begin{equation}
    \mathcal{\hat H} = \int \hat \Psi^\dagger(\mathbf{r})\left[-\frac{\hbar^2}{2m}\nabla^2 + V_0(\mathbf{r}) \right]\hat \Psi(\mathbf{r}) d\mathbf{r} + \frac{1}{2} \int  \hat \Psi^\dagger(\mathbf{r}) \hat \Psi^\dagger(\mathbf{r'}) V(\mathbf{r} - \mathbf{r'})\hat \Psi(\mathbf{r'}) \hat \Psi(\mathbf{r}) d\mathbf{r} d\mathbf{r'},
    \label{eqn:2nd}
\end{equation}
where $\mathbf{r}$ and $\mathbf{r'}$ are the positions of the two colliding particles, $V(\mathbf{r}-\mathbf{r'})$ is the interaction potential, and $\hat \Psi^\dagger(\mathbf{r})$ and $\hat \Psi(\mathbf{r})$ are the creation and annihilation operators for the atomic field that follow the bosonic commutation relations,
\begin{align}
 [\hat \Psi(\mathbf{r}),\hat \Psi^\dagger(\mathbf{r})] &= \delta(\mathbf{r} - \mathbf{r'}) \\
 [\hat \Psi^\dagger(\mathbf{r}),\hat \Psi^\dagger(\mathbf{r})] &= 0 \\
 [\hat \Psi(\mathbf{r}),\hat \Psi(\mathbf{r})] &= 0.
\end{align}

\noindent In the case of a BEC at $T\approx0$, one can perform a Bogoliubov expansion~\cite{bogoliubov1947, dalfovo1999}
\begin{equation}
    \hat \Psi (\mathbf{r}, t) = \Phi(\mathbf{r},t) + \delta \hat \Phi(\mathbf{r},t),
\label{eqn:bog1}
\end{equation}
where $\Phi(\mathbf{r},t) \equiv \langle \hat \Psi(\mathbf{r},t) \rangle$ is the wavefunction of the condensate known as the order parameter and $\delta \hat \Phi(\mathbf{r},t)$ represents quantum fluctuations of the BEC system;
therefore, the condensate density is defined as
\begin{equation}
    n(\mathbf{r},t) = |\Phi(\mathbf{r},t)|^2.
\end{equation}

Now one may use the Heisenberg equation of motion,
\begin{equation}
    i\hbar \frac{\partial}{\partial t}\hat \Psi(\mathbf{r},t) = [\hat \Psi, \hat H],
\end{equation}
    to determine the time evolution of the field operator $\hat \Psi(\mathbf{r},t)$ as
\begin{equation}
    \frac{\partial}{\partial t}\hat \Psi(\mathbf{r},t) = \frac{1}{i\hbar}\left[-\frac{\hbar^2}{2m}\nabla^2 + V_0(\mathbf{r}) + \int d\mathbf{r'} \hat \Psi^\dagger(\mathbf{r'}, t)V(\mathbf{r'} -\mathbf{r})\hat \Psi(\mathbf{r'},t)\right]\hat \Psi(\mathbf{r},t),
\end{equation}
which follows from Equation~\eqref{eqn:2nd} after integrating over $\mathbf{r}$.
If it is assumed that two bosons will only interact with a contact potential of the form
\begin{equation}
V(\mathbf{r'}-\mathbf{r}) = g\delta(\mathbf{r'} - \mathbf{r}),
\end{equation}
which has a strength given by
\begin{equation}
g = \frac{4 \pi \hbar^2 a_s}{m},
\end{equation}
where $a_s$ is the species and state-dependent s-wave scattering length,
we may write the evolution for the wavefunction as
\begin{equation}
    i\hbar \frac{\partial}{\partial t}\Phi(\mathbf{r},t) = \left( - \frac{\hbar^2}{2m} \nabla^2 + V_0(\mathbf{r}) + g |\Phi(\mathbf{r},t)|^2\right)\Phi(\mathbf{r},t).
\end{equation}

\noindent This is the celebrated the Gross--Pitaevskii Equation (GPE), which is known as the governing equation for the physics of dilute BEC systems.
When written in the time-independent form it determines the chemical potential $\mu$ of the condensate as~\cite{gross1961, pitaevskii1961}
\begin{equation}
    \mu\Phi(\mathbf{r}) = \left( - \frac{\hbar^2}{2m} \nabla^2 + V_0(\mathbf{r}) + g |\Phi(\mathbf{r})|^2\right)\Phi(\mathbf{r}).
    \label{eqn:GP}
\end{equation}

The time-dependent GPE allows one to determine the full dynamics of a BEC system and the numerical solutions will be discussed in subsequent chapters.
Similar derivations of the GPE can be found in many introductory texts on BEC physics~\cite{fetter2003, pethick2002, fetter2009}.
The main difference between this equation and the Schr\"odinger equation is the non-linear interaction term $g|\Psi|^2$, that accounts for the fact that BECs typically consist of $10^3$ to $10^6$ particles.
When solving this system in a simple harmonic oscillator in the limit where interactions are significant, one can find an analytical solution for the wavefunction density, known as a Thomas--Fermi (TF) distribution, shown in Figure~\ref{fig:TF}.
In this figure, the simulated wavefunction density is identical to the TF distribution, except in the tail region where the BEC tapers to zero density.

\begin{figure}
\center \includegraphics[width = \textwidth]{data/qs/SHO/SHO.pdf}

\caption{Slice of the ground state of a two-dimensional simple harmonic oscillator for the GPE equation with $g=1$ (purple, dashed) and the TF distribution (blue, solid).
Here, the GPE solution follows a TF distribution near the center of the trap, but the tail is slightly different.
This simulation was performed with GPUE~\cite{schloss2018} for a two-dimensional grid of $256^2$ elements of size $250 \mu m$ with a trapping potential of $\omega_x = \omega_y = 1$, and $5\times 10^4$ particles.}
\label{fig:TF}
\end{figure}

The TF distribution can be derived from the time-independent GPE in the limit where there are a large number of bosons in the condensate ($N \gg 1$) as~\cite{ueda2010}, so that the interaction energy exceeds the kinetic energy and it becomes,
\begin{equation}
\Psi_\text{TF}(\mathbf{r}) = \sqrt{\frac{[\mu-V(\mathbf{r})]\Theta(\mu-V(\mathbf{r}))}{g}}
\end{equation}
\noindent where $\Theta$ is the Heaviside step function that ensures the density stays positive.
The shape of this function is that of an inverted parabola for a harmonic trap, and the radius in any direction from the center can be found to be
\begin{equation}
R_\text{TF} = \sqrt{\frac{2\mu}{m\omega_i^2}},
\label{eqn:rtf}
\end{equation}
\noindent where $\omega_i$ is the trapping frequency in the $i$th direction.
Using the normalization condition, $\int_{-\infty}^\infty |\Psi(\mathbf{r},t)|^2 = N$, the chemical potential in the TF limit becomes
\begin{equation}
\mu_\text{TF} = \frac{h\omega_i}{2}\left(\frac{15 N a_s}{a} \right)^{2/5}.
\label{eqn:mutf}
\end{equation}
\noindent where $a=\sqrt{\hbar/m\bar\omega}$ and $\bar \omega$ is the average of all the harmonic trapping frequencies.
Using Equations~\eqref{eqn:mutf} and \eqref{eqn:rtf} one finds that
\begin{equation}
R_\text{TF} = a\left(\frac{15 N a_s}{a} \right)^{1/5}\frac{\bar \omega}{\omega_i}.
\end{equation}
This approximation is valid for stationary condensate solutions in simple harmonic oscillator trapping geometries.

It is important to note that a BEC acts like a superfluid, which is a state of matter that is similar to a classical fluid without viscosity.
This means that once a superfluid is set in motion, there is no retarding force to keep it from flowing.
There are a few known systems in which superfluidity can exist, such as $^4$He (sometimes called Helium II when in its superfluid phase)~\cite{allen1938}, neutron stars~\cite{migdal1960}, or BEC systems~\cite{einstein1925, anderson1995}.
BEC systems are generally cleaner experimental systems to create, as they do not have a classical fluid fraction, like $^4$He.
They are therefore well-suited systems to study excitations related to superfluid flow.

As a final note, the GPE is valid at zero temperature when the normal fluid component is small, and other models exist to describe BEC systems when finite temperature exists, such as the stochastic GPE~\cite{rooney2012} and the Zaremba-Nikuni-Griffin (ZNG) model~\cite{zaremba1999}.
In general, there is a small error between the GPE and experimental results, which can be slightly mitigated by performing fully three-dimensional simulations; however the direct cause of this error is not precisely known~\cite{savage2003}.
It is now important to discuss superfluids in more detail, focusing on vortex dynamics to be simulated in this work.

\section{Superfluid systems and vortex dynamics}

Next, I will focus on the differences between vortex dynamics in classical and superfluid systems before continuing to discuss three methods of vortex generation in superfluid systems: rotation, phase imprinting and artificial magnetic fields.
By rotating a fluid, it is possible to create a vortex around the axis of rotation; however, because of the viscosity of a classical fluid, the vortex will eventually disappear without constant driving.
In a superfluid, this is not necessarily the case.
For this discussion, it is worthwhile to start with the hydrodynamic description of a BEC, following the text of Pethick and Smith~\cite{pethick2002}.
To start, I will rewrite the condensate wavefunction as
\begin{equation}
\Psi(\mathbf{r},t) = \sqrt{\rho(\mathbf{r},t)}e^{i\psi(\mathbf{r},t)},
\label{eqn:ansatz}
\end{equation}
\noindent with
\begin{equation}
\rho(\mathbf{r},t)=\Psi(\mathbf{r},t)^*\Psi(\mathbf{r},t) = |\Psi(\mathbf{r},t)|^2,
\end{equation}
\noindent where $\psi(\mathbf{r},t)$ is the BEC phase.
By multiplying the GPE by $\Psi^*(\mathbf{r},t)$ and subtracting the complex conjugate, one can obtain the continuity equation~\cite{pethick2002},
\begin{equation}
\frac{\partial}{\partial t}\rho(\mathbf{r},t)+\nabla\cdot\mathbf{J}(\mathbf{r},t) = 0,
\end{equation}
\noindent where $\mathbf{j}$ is the current density of the condensate, defined as,
\begin{equation}
\mathbf{j}(\mathbf{r},t) = \frac{-i\hbar}{2m}\left( \Psi^*(\mathbf{r},t)\nabla\Psi(\mathbf{r},t)- \Psi(\mathbf{r},t)\nabla\Psi^*(\mathbf{r},t)\right).
\label{eqn:current_density}
\end{equation}
By substituting Equation~\eqref{eqn:ansatz} into Equation~\eqref{eqn:current_density}, the form of the current density for the GPE will be,
\begin{equation}
\mathbf{j}(\mathbf{r},t) = |\Psi(\mathbf{r},t)|^2\frac{\hbar}{m}\nabla\phi(\mathbf{r},t).
\end{equation}

The velocity of the superfluid is defined as a ratio of the current density to the density, itself, which is
\begin{equation}
\mathbf{v}(\mathbf{r},t)=\frac{\mathbf{j}(\mathbf{r},t)}{\rho(\mathbf{r},t)} = \frac{\hbar}{m}\nabla\phi(\mathbf{r},t).
\end{equation}
\noindent This can be interpreted to mean that the gradient of the phase determines the velocity of atoms in the BEC, indicating that the system is irrotational ($\nabla \times \mathbf{v} = 0$).
Because the condensate wavefunction has to be single-valued, any rotation in a finite system has to be quantized as,
\begin{equation}
\oint \mathbf{v} \cdot d\ell = \frac{\hbar}{m}2\pi\ell,
\label{eqn:quantized}
\end{equation}
\noindent where, $\ell$ is the integer charge of the circulation.
Equation~\eqref{eqn:quantized} shows the quantized nature of circulation in a superfluid with each vortex hosting multiples of $2\pi$ charges.
This means that every singly-charged vortex in a BEC will have a $2\pi$ phase winding, and an example of one vortex in a two-dimensional condensate can be seen in Figure~\ref{fig:rot} (a and c).
Equation~\eqref{eqn:quantized} also indicates that the phase is not defined at the center of the vortex; however, the condensate circumvents this problem by requiring the density at this point to be zero.
The density dip from the normal condensate density to zero happens over the scale of the healing length, which is
\begin{equation}
\xi=\frac{1}{\sqrt{8\pi\rho_ba_s}}
\end{equation}
\noindent for repulsive interactions, where $\rho_b$ is the bulk density of the condensate.

\begin{figure}

\includegraphics[width=\textwidth]{data/splitop/rot/WIP.pdf}

\caption{
Simulation of condensate under rotation that leads to a single vortex (a, c) and a vortex lattice (b, d).
The wavefunction density is shown in (a) and (b), while the corresponding phase is shown in (c) and (d).
An external rotation of $\Omega = 0.35\omega_x$ was used for (a) and (b), while a rotation of $\Omega = 0.99\omega_x$ was used for (c) and (d).
The system consists of $^87$ Rb atoms is used with a trapping frequency of $\omega_x = \omega_y = 2\pi$Hz on a 512-point grid of size 200$\times$200 $\mu$m.
This simulation was performed with the GPUE codebase \cite{schloss2018}, and the phase plots are created by multiplying the phase by the wavefunction density to remove anomalous noise beyond the BEC boundary.}
\label{fig:rot}
\end{figure}

In a cylindrically symmetric condensate with a single vortex at its center, the wavefunction can then be written as
\begin{equation}
\Psi(\mathbf{r}) = |\Psi(\mathbf{r})|e^{i\ell\phi},
\end{equation}
and the energy (Equation~\eqref{eqn:energy}) of the BEC is
\begin{equation}
E = \int_{-\infty}^\infty \frac{\hbar^2}{2m}\left( |\nabla\Psi(\mathbf{r})|^2 + \frac{|\Psi(\mathbf{r})|^2\ell^2 m\mathbf{v}^2}{2}\right) + V_0(\mathbf{r})|\Psi(\mathbf{r})|^2 + \frac{g}{2}|\Psi(\mathbf{r})|^4.
\end{equation}
\noindent Because $E \propto \ell^2$, as a superfluid is spun faster, a vortex will not grow in angular momentum, but multiple vortices with $\ell = 1$ will spawn instead~\cite{pethick2002}.
In other words, it is energetically favorable for two vortices of smaller angular momentum to form instead of a single vortex with a large amount of angular momentum;
therefore, as angular momentum increases and more vortices are introduced into the system, they will eventually arrange themselves in a triangular lattice structure known as an Abrikosov lattice~\cite{abrikosov1957, fetter2001}.
This behavior is identical to that of type II superconductors under the effects of a magnetic field.
An example of a vortex lattice and its phase can be seen in Figure~\ref{fig:rot} (b and d).

Until now, I have focused primarily on vortex structures in two-dimensions; however,
the three-dimensional properties of vortices in superfluid systems are also peculiar when compared to their classical counterparts.
Here, vortex lines are formed that must either end at the surface of the condensate~\cite{madison2000} or reconnect in the form of vortex rings or other, more complicated vortex structures~\cite{reichl2013, barenghi2014}.
Because the circulation around superfluid vortices is quantized, when two vortices approach each other with different velocity fields, they may reconnect into smaller, more energetically favorable vortex structures.
During this reconnection, the abrupt change in energy will create sound waves at the reconnection site~\cite{feynman1955}.

Three dimensional vortex structures in BECs are difficult to controllably generate experimentally, but I will discuss an experimentally viable method to generate, control, and detect vortex ring-like structures in superfluid BEC systems in Chapter~\ref{ch:vortex_states}.
In that chapter, I will also further discuss three-dimensional vortex motion.
On the other hand, in Chapter~\ref{ch:2d}, I will discuss important aspects of simulating two-dimensional condensate systems.
 For now, I will begin discussing three processes to generate vortex structures in superfluid systems: rotation, phase imprinting, and artificial magnetic fields.

\subsection{Rotation}

\label{sec:rot}
Rotation of a BEC system will provide vortex lines that follow the axis of rotation and start and end on the BEC boundary.
To simulate the effects of rotation, one simply need to append the angular momentum operator $L_z = -i\hbar(xp_y - yp_x)$ to the GPE in the rotating frame,

\begin{equation}
i \hbar \frac{\partial \Psi(\mathbf{r},t)}{\partial t} = \left(\frac{p^2}{2m} + V_0 + g |\Psi(\mathbf{r},t)|^2 -\Omega L_z \right)\Psi(\mathbf{r},t),
\label{eqn:GPErot}
\end{equation}

\noindent where $\Omega$ is the rotation frequency.

In order to generate a vortex via rotation in a harmonic trap, one must rotate faster than the critical velocity of $\Omega_c \approx 0.7 \omega_\perp$, where $\omega_\perp$ is the trapping frequency perpendicular to the axis of rotation~\cite{fetter2009}.
In addition, if the rotation frequency is greater than the trapping frequency, the atoms will no longer be bound by the trap due to centripetal forces.
As such, finding the appropriate rotation frequency for creating vortex lattices in BEC systems is a precarious balancing act.
Even so, large scale vortex lattices have been generated both experimentally and theoretically \cite{o2016, o2016topo, abo2001, schweikhard2004}.

Experimentally, rotation for a small number of vortices can be generated in a number of ways, such as a ``rotating bucket'' approach that has been extended to ultracold atomic systems~\cite{chevy2006}.
For this method, bosons are confined to a magnetic trap and an anisotropic potential is superimposed that rotates with the desired angular velocity~\cite{madison2000,abo2001,hodby2001,haljan2001}.
For rotation velocities close to the harmonic trapping frequency, additional methods must be used to ensure the atoms remain in-place.
These methods include adding an extra confining potential~\cite{bretin2004}, or the evaporative spin-up technique, where atoms with less angular momentum are evaporated such that the remaining atoms have a higher rotation speed~\cite{schweikhard2004,engels2003}.
Additionally, vortex ring-like structures have recently been generated experimentally via rotation~\cite{guo2019}.

In this work, I use rotation in a qualitatively similar way to Equation~\eqref{eqn:GPErot}; however, I will later introduce artificial magnetic fields, a broader framework that encompasses rotation that will be used in all simulations moving forward.
We will discuss this in more detail in Section~\ref{sec:implementation}.

\subsection{Phase imprinting}

Phase imprinting is a powerful tool to allow for the generation of various structures in atomic systems, including vortices~\cite{kumar2018, moulder2012, burger1999, denschlag2000, wu2002}.
To generate vortices in a BEC, this technique relies on imprinting a $2\pi$ phase winding onto a ground state condensate wavefunction, after which the density adjusts to zero at regions near the phase singularity.
Experimentally, phase imprinting can be done in a number of ways.
As an example, the phase could be imprinted with a two-photon Raman process to transfer orbital angular momentum to atoms from a Laguerre--Gaussian beam \cite{moulder2012, ryu2007}.
Another method is through pulsing a spatially dependent potential for a short time when compared to the trapping frequency, which will imprint its potential energy onto the phase of the system~\cite{kasevich1991} and can be used to generate solitons~\cite{denschlag2000}, vortices~\cite{gajda1999}, or other states with quantized circulation~\cite{kumar2018}.
Phase imprinting has also been used in theoretical studies to create a defect in a large vortex lattice by flipping the phase (and therefore rotation direction) of a selected vortex by imprinting a $-4\pi$ phase at the vortex's location~\cite{o2016topo}.
Note that if a phase greater than $|2\pi|$ is imprinted onto the system, the vortices are likely to decay into multiple vortices of $|2\pi|$ phase~\cite{shin2004}.

For the purposes of this work, I will only consider imprinting vortices in two-dimensional settings by applying phase imprinting operations, such that
\begin{equation}
\Psi_{\text{IMP}}(x,y,t) = |\Psi(x,y,t)|e^{i(\theta(x,y,t) + \theta_{\text{IMP}}(x,y))},
\end{equation}

\noindent where $\Psi_{\text{IMP}}(x,y,t)$ is the condensate wavefunction after phase imprinting.
This method allows one to apply a phase mask to any location in the transverse plane.

Phase imprinting has allowed for the generation of many interesting vortex topologies in theoretical and experimental studies~\cite{white2014, maucher2016}; however, it is a dynamical process that does not create eigenstates of the system.
As such, it is not as useful for engineering stable vortex structures, but is instead useful for dynamical studies, such as those found in Chapter~\ref{ch:2d}.

\subsection{Artificial magnetic fields}
\label{sec:gauge}

Magnetic fields are capable of generating rotational effects in charged systems through the Lorentz force, $F_l = q(\mathbf{E} + \mathbf{v} \times \mathbf{B})$, where $q$ is the charge of the system, $\mathbf{E}$ is the electric field, $\mathbf{B}$ is the magnetic field, and $\mathbf{v}$ is the velocity of the particle.
This effect allows for the creation of vortices in type II superconductors; however, it is not directly applicable to BEC systems because BECs are composed of neutral atoms.
Even so, it is possible to generate artificial magnetic fields with similar effects, and these artificial magnetic fields have  been shown to create vortices experimentally~\cite{lin2009}.
In addition, artificial magnetic fields create a broader framework that encompasses rotational effects previously shown in Section~\ref{sec:rot}, and I will use this framework instead of rotation for simulations in this work.
For this, a detailed introduction, similar to that given by Dalibard in~\cite{dalibard2015}, will be presented below.

If written in the Hamiltonian formalism, the Lorentz force law becomes

\begin{equation}
\mathcal{\hat{H}} = \frac{(\mathbf{\hat p} - q\mathbf{A}(\mathbf{r}))^2}{2m}
\end{equation}

\noindent where $\mathbf{A}$ is a vector potential such that the magnetic field is given by $\mathbf{B} = \nabla \times \mathbf{A}$ and $q$ is the charge of the particle.
Because cold atoms are neutral, one must find ways to simulate the effects of magnetic fields instead of using magnetic fields, themselves.

Firstly, let us describe how rotation can be considered to be an artificial vector potential and thus generate vortex structures in BEC systems.
Imagine a plane rotating with an angular velocity $\Omega$ around the $z$-axis ($\mathbf{\Omega} = \Omega \hat z$).
In this case, the Coriolis force is defined as
\begin{equation}
\mathbf{F}_{\text{Coriolis}} = 2m \mathbf{v} \times \mathbf{\Omega},
\end{equation}
which is formally similar to the Lorentz force law.
By applying the transformation $\mathcal{\hat H} = \mathcal{\hat H}_0 - \Omega \hat L_z$, where $\hat L_z = x\partial_y - y\partial_x$, one finds~\cite{bhat2008}
\begin{equation}
\begin{split}
\mathcal{\hat H} &= -\frac{\hbar^2}{2m}\nabla^2 + \frac 1 2 m \omega^2(x^2 + y^2) - \frac{\hbar \Omega}{i}(x\partial_y - y\partial_x) \\
 &= \frac{1}{2m}\left(\frac{\hbar}{i}\nabla - m(\mathbf{\Omega} \times \mathbf{r})\right)^2 + \frac m 2 \left( \omega^2 - \Omega^2 \right)r^2 \\
 &= \frac{(\hat{\mathbf{p}}-m\mathbf{A}(\mathbf{r}))^2}{2m}+ V_0(\mathbf{r}),
\end{split}
\end{equation}
where $\omega$ is the trapping frequency for a symmetric two-dimensional harmonic trap, $\mathbf{A} \equiv \mathbf{\Omega} \times \mathbf{r}$, and $V_0 = m/2 \left( \omega^2 - \Omega^2 \right)r^2$.
The final form is similar to that of the Lorentz force law and coincides with an effective magnetic field of $2 \mathbf \Omega \propto \mathbf B$.
In this way, one can recreate the rotation expected from the Lorentz force law in a cold atomic system with an artificial magnetic field~\cite{peshkin1989, madison2000, abo2001}.
Artificial magnetic fields provide a powerful tool to researchers who wish to generate and control complex vortex structures, and because of this, it is worth discussing them in further detail.
Important implementation details for how to use artificial magnetic fields with the SSFM will be discussed in Section~\ref{sec:implementation} and further discussions of how these modifications can be applied on GPU architecture can be found in Chapter~\ref{ch:gpu}.

\subsubsection{Geometric Gauge Fields}
\label{sec:geom}

As I have already described how rotation can act as an artificial Lorentz force, I will now turn the attention towards methods that might allow us to generate more general rotational effects and vortex structures.
In particular, I will discuss the adiabatic motion of free atoms undergoing geometric phase transformations through a Berry phase.
For this, one can assume that the system has an external parameter $\mathbf{\lambda}$ such that
\begin{equation}
\hat H(\mathbf{\lambda}) \ket{\psi_n(\mathbf{\lambda})} = E_n(\mathbf{\lambda})\ket{\psi_n(\mathbf{\lambda})},
\end{equation}
where the set of eigenstates $\left\{ \ket{\psi_n(\mathbf{\lambda})} \right\}$ allows us to define the time evolution of the system such that
\begin{equation}
\ket{\psi(t)} = \sum_n c_n(t) \ket{\psi_n(\mathbf{\lambda}(t))},
\end{equation}
where $\mathbf{\lambda}$ evolves slowly with time. If one considers the initial state of the system to be
\begin{equation}
c_l(0) = 1,
\qquad
c_n(0) = 0,
\qquad
\text{for all } n\neq l,
\end{equation}
the state of the system is proportional to $\ket{\psi_l(\mathbf{\lambda}(t))}$.
In this case, $c_l(t)$ is determined by the equation
\begin{equation}
i \hbar \dot{c}_l =  [E_l(t) - \dot{\mathbf{\lambda}} \cdot \mathbf{A}_l(\mathbf{\lambda})]c_l,
\label{Bcnx-1}
\end{equation}
where
\begin{equation}
\mathbf{A}_l(\mathbf{\lambda}) = i \hbar \braket{\psi_l|\nabla\psi_l}.
\label{eqn:Bcnx}
\end{equation}
This quantity is called the Berry connection, which is considered to be a vector potential, such that one can define a new artificial magnetic field, the Berry curvature as
\begin{equation}
\mathbf{B}_l = \mathbf{\nabla} \times \mathbf{A}_l.
\label{eqn:BC}
\end{equation}

Now imagine that the $\mathbf{\lambda}$ parameter follows the closed contour $C$ such that $\mathbf{\lambda}(T) = \mathbf{\lambda}(0)$.
By integrating Equation~\eqref{Bcnx-1}, one finds
\begin{equation}
c_l(t) = e^{i \Phi_{\text{dyn}}(t)}e^{i\Phi_{\text B}(T)}c_l(0),
\label{eqn:c}
\end{equation}
where
\begin{equation}
\begin{split}
\Phi_{\text{dyn}}(T) &= - \frac{1}{\hbar}\int_0^TE_l(t)dt \\
\Phi_{\text{Berry}} (T)&= \frac{1}{\hbar} \int_0 ^T \dot{\mathbf{\lambda}} \cdot \mathbf{A}_l(\mathbf{\lambda})dt = \frac{1}{\hbar}\oint\mathbf{A}_l(\mathbf{\lambda}) \cdot d\mathbf{\lambda}.
\end{split}
\end{equation}
In this case $\Phi_{\text{Berry}}$ is called the Berry phase and it only depends on the motion path of $\mathbf{\lambda}$.
It should be mentioned that both of the exponential terms in Equation~\eqref{eqn:c} are gauge invariant and thus remain unchanged when $\ket{\psi_n(\mathbf{\lambda})}$ is multiplied by a phase factor.
The Berry phase allows us to transfer angular momentum into a BEC and generate a vortex geometry, which
has been experimentally demonstrated in 2009 by Lin~\textit{et~al.}~\cite{lin2009}.
As a note, the vortex structures generated in this way follow the magnetic field lines, thus providing the capability to generate complex vortex structures beyond simple, straight lines.
A method to generate these gauge fields in an experimentally realizable way with the evanescent field of a dielectric system undergoing total internal reflection will be described in Chapter~\ref{ch:vortex_states}, and there are a wealth of applications of these fields in different areas of physics~\cite{niu2017}.
In addition, other field effects can be generated in a similar manner and with similar effects as the Berry phase~\cite{wu2005}, but these are not considered in this work.
For now I will discuss how these fields can be implemented in the SSFM, with an emphasis on their effects for superfluid simulations.

\section{Modifications to the SSFM for superfluid vortex simulations}
\label{sec:implementation}

As shown above, in order to simulate the effects of artificial magnetic fields on BEC systems, one must modify the Hamiltonian, such that

\begin{equation}
\mathcal{\hat H} = \frac{(p-m\mathbf{A})^2}{2m} + V_0 + g|\Psi(\mathbf{r},t)|^2,
\end{equation}

\noindent where $\mathbf{A}$ is an artificial vector potential.
As a note, some texts absorb the mass term into the artificial vector potential, but here I am  stating it explicitly for clarity.
When expanded, that the gauge field has a component in position space, $\frac{m\mathbf{A}^2}{2}$, which couples with the trapping potential, and another component that is partially in both position and momentum space, $-\left(\frac{p\mathbf{A} + \mathbf{A}p}{2}\right)$.

Firstly, let us discuss the component purely in position space.
Here, it is important to properly balance the trapping potential and the artificial vector potential when simulating BEC systems with artificial vector potentials, otherwise the trapping geometry will become warped.
In the case of rotation, this is effectively balanced by the centripetal force; however,
in the case of arbitrarily chosen vector potentials, this can create rather unusual potential geometries, as shown in Figure~\ref{fig:V_change} for a Gaussian $\mathbf{A_x}$ and $\mathbf{A_y}$.
Though one might be able to balance this warping with a centripetal force tailored to the gauge fields introduced, this was not considered this for the simulations within this work.

\begin{figure}

\center \includegraphics[width=0.6\textwidth]{data/splitop/gauge/check.pdf}

\caption{
(solid blue) Modified trap geometry due to the influence of high artificial vector potential strength.
The vector potential ($\frac{\mathbf{A}^2}{2}$) is shown in dotted blue and the original, harmonic trap is shown in dashed red.
}
\label{fig:V_change}
\end{figure}

The components of the artificial vector potential that are partially in position and momentum space are somewhat difficult to consider numerically.
For many physical examples, such as rotation, $\frac{\partial \mathbf{A}_i}{\partial i} = 0$ for $i \in {x, y, z}$, which means the only relevant term is $-\frac{\mathbf{A}_i p_i}{2}$.
This will effectively create a variable that resides in both position and momentum space, and multiplication with this operator can be performed with a one-dimensional FFT across $n$-dimensional data on the wavefunction first.
This means that if there are three operators, $\mathbf{A}_x p_x\hat x$, $\mathbf{A}_y p_y\hat y$, and $\mathbf{A}_z p_z\hat z$, one needs to perform an FFT on the wavefunction in the $\hat x$, $\hat y$, and $\hat z$ dimensions, respectively, before performing an element-wise multiplication.
As I will discuss in Chapter~\ref{ch:gpu}, this has significant performance penalties if one does not consider the underlying computational architecture.
This also requires the usage of more intricate FFT plans for the FFTW or CuFFT libraries, which are non-trivial for three-dimensional simulations.

If $\frac{\partial \mathbf{A}_i}{\partial i} \neq 0$, then the other operator, $-\frac{p_i\mathbf{A}_i}{2}$, must also be considered.
To perform this operation, a derivative must be performed on $\mathbf{A}$ before application to the wavefunction.
This operation doubles the complexity of the application of artificial magnetic fields to the system.

At this point, I have discussed the SSFM, itself, along with the primary system that will be simulated in this work; however, I have yet to discuss key techniques in quantum state engineering that are necessary to motivate several design decisions.
As such, I will consider two dynamic methods for quantum state engineering in the following chapter: shortcuts to adiabaticity and quantum optimal control.


%$Id: Rayleigh-Ritz.tex 68 2007-05-09 19:01:47Z jlconlin $
%$Author: jlconlin $
%$Date: 2007-05-09 13:01:47 -0600 (Wed, 09 May 2007) $
%$Revision: 68 $

\documentclass[10pt]{article}
\usepackage{geometry}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{graphicx}
\usepackage{theorem}
\usepackage[longnamesfirst]{natbib}
\usepackage{paralist}

\newtheorem{thm}{Theorem}[section]

\author{Jeremy Conlin}
\title{How Arnoldi's Method Estimates Eigenvalues of $A$}

\begin{document}
\maketitle
\abstract{In an effort to better understand the Mathematics behind Arnoldi's method, I am creating this document to write down some of the things I have discovered.  }

\section{Exercises}
Here I show the results of some exercises taken from \cite{Watkins2002:Funda}.  These exercises show how one can estimate eigenvalues with Ritz pairs.
\begin{description}
    \item[Exercise 6.1.4]\emph{Let $S$ be an invariant subspace of $A$ with basis $x_1, \ldots , x_k$, and let $\hat{X} = [x_1 \ldots]$.  By Theorem 6.1.3 there exists $\hat{B} \in \mathbb{F}^{k \times k}$ such that $A\hat{X} = \hat{X}\hat{B}$.}

    \begin{thm}[6.1.3 from \cite{Watkins2002:Funda}]
    Let $S$ be a subspace of $\mathbb{F}^n$ with a basis $x_1, \ldots , x_k$.  Thus $S = \mathrm{span}\{x_1, \ldots , x_k\}$.  Let $\hat{X} = [x_1 \ldots x_k] \in \mathbb{F}^{n\times k}$. Then $S$ is invariante under $A \in \mathbb{F}^{k \times k}$ such that
    \begin{equation}
        A\hat{X} = \hat{X}\hat{B}.
    \end{equation}
    \end{thm}

    \begin{itemize}[\textbf{Part }\bfseries a)]
    \item \emph{Show that if $\hat{v} \in \mathbb{F}^k$ is an eigenvector of $B$ with eigenvalue $\lambda$, then $v = \hat{X}\hat{v}$ is an eigenvector of $A$ with eigenvalue $\lambda$.  In particular, every eigenvalue of $\hat{B}$ is an eigenvalue of $A$.  What role does the linear independence of $x_1, \ldots , x_k$ play?}

    \begin{equation}
        \hat{B}\hat{v} = \lambda\hat{v}
    \end{equation}
    \begin{subequations}\begin{align}
        A\hat{X} &= \hat{X}\hat{B} \\
        A\hat{X}\hat{v} &= \hat{X}\hat{B}\hat{v} \\
        A\hat{x}\hat{v} &= \lambda \hat{x}\hat{v} \\[4mm]
        Av &= \lambda v.
    \end{align}\end{subequations}

    \item \emph{Show that $V \in S$.  Thus $v$ is an eigenvector of $A|_S$.}

    The columns of$\hat{x}$ span the space $S$,
    \begin{equation}
        S = \mathrm{span}\{x_1, \ldots , x_k\}
    \end{equation}
    so we can write any vector $v \in S$ as a linear combination of the basis vectors of $S$.
    \begin{equation}
        \hat{X}\hat{v} = v = x_1\hat{v_1}+\ldots+x_k\hat{v_k}.
    \end{equation}

    \end{itemize}

    \item[Exercise 6.3.35] \emph{Let $A \in \mathbb{C}^{n\times n}$, and let $S$ be a $k$-dimensional subspace of $\mathbb{C}^n$.  Then a vector $v \in S$ is called a {\em Ritz vector} of $A$ from $S$ if and only if there is a $\mu \in \mathbb{C}$ such that the {\em Rayleigh-Ritz-Galerkin} condition}
    \begin{equation}
        Av - \mu v \perp S
    \end{equation}
    \emph{holds, that is, $\langle Av - \mu v,s\rangle = 0$ for all $s \in S$.  The scalar $\mu$ is called the {\em Ritz value} of $A$ associated with $v$. Let $q_1, \ldots , q_k$ be an orthonormal basis of $S$, let $Q = [q_1 \ldots q_k]$, and let $B = Q^*AQ \in \mathbb{C}^{k\times k}$.}

    \begin{itemize}[\textbf{Part }\bfseries a)]
    \item \emph{Since $v \in S$, there is a uniqute $x \in \mathbb{C}^k$ such that $v = Qx$.  Show that $v$ is a Ritz vector of $A$ with associated Ritz value $\mu$ if and only if $x$ is an eigenvector of $B$ with associated eigenvalue $\mu$.  In particular, there are $k$ Ritz values of $A$ associated with $S$, namely the $k$ eigenvalues of $B$.}

    The vector $v$ is a linear combination of the columns of $Q$ (basis vectors of $S$);
    \begin{equation}
        v = Qx = x_1q_1+ \ldots +x_kq_k.
    \end{equation}
    Assume $(\mu,x)$ is an eigenpair of $B$,
    \begin{equation}
        Bx = \mu x
    \end{equation}
    \begin{subequations}\begin{align}
        Q^*AQ &= B \\
        A^*AQx &= Bx = \mu x \\
        AQx &= \mu x = \mu Qx \\
        Av &= \mu v.
    \end{align}\end{subequations}

    Now we go the other direction and begin by assuming $(v,\mu)$ is an eigenpair of $A$.
    \begin{equation}
        Av = \mu v
    \end{equation}
    \begin{subequations}\begin{align}
        AQx &= \mu Qx \\
        Q^*AQx &= \mu x \\
        Q^*AQx &= Bx \\
        \mu x = Q^*&AQ = Bx \\[4mm]
        Bx &= \mu x
    \end{align}\end{subequations}

    \item \emph{Show that if $v$ is a Ritz vector with Ritz value $\mu$, then$\mu = v^*Av/v^*v$.  That is, $\mu$ is the Rayleigh quotient of $v$.}

    \begin{subequations}\begin{align}
        Av &= \mu v \\
        v^*Av &= \mu v^*v \\[4mm]
        \mu &= \frac{v^*Av}{v^*v}
    \end{align}\end{subequations}
    \end{itemize}

    \item[Exercise 6.3.36] \emph{Let $(\mu, v)$ be an approzimate eigenpair of $A$ with $\|v\|_2 = 1$, and let $r = Av - \mu v$ (the residual).  Let $\epsilon = \|r\|_2$ and $E = -rv^*$.  Show that $(\mu, v)$ is an eigenpait of $A + E$, and $\|E\|_2 = \epsilon$.  This shows that if the residual norm $\|r\|_2$ is small, then $(\mu,v)$ is an exact eigenpair of a matrix that is close to $A$.  Thus $(\mu, v)$ is a good approximate eigenpair of $A$ in the sense of backward error.}

    \begin{equation}
        A + E = A - rv^*
    \end{equation}
    \begin{subequations}\begin{align}
        (A + E)v &= \mu v \\
        Av + Ev &= \mu v \\
        Av - \mu v &= -Ev \\
        r &= -Ev \\[4mm]
        E &= -rv^*
    \end{align}\end{subequations}

    \begin{equation}
        \|E\|_2 = \|rv^*\|_2 = \|r\|_2\|v^*\|_2 = \epsilon (1).
    \end{equation}

\end{description}
\bibliographystyle{plainnat}
\begin{thebibliography}{10}
\bibitem{Watkins2002:Funda}
D.~S. Watkins.
\newblock {\em Fundamentals of Matrix Computations}.
\newblock Wiley-Interscience, New York, second edition, 2002.
\end{thebibliography}

\end{document}


\section{Results}

\subsection{Minimal model of transcriptional regulation}\label{sec_model}

As a tractable circuit for which we have control over the parameters both
theoretically and experimentally, we chose the so-called simple repression
motif, a common regulatory scheme among prokaryotes \cite{Rydenfelt2014}. This
circuit consists of a single promoter with an RNA-polymerase (RNAP) binding
site and a single binding site for a transcriptional repressor
\cite{Garcia2011c}. The regulation due to the repressor occurs via exclusion of
the RNAP from its binding site when the repressor is bound, decreasing the
likelihood of having a transcription event. As with many important
macromolecules, we consider the repressor to be allosteric, meaning that it can
exist in two conformations, one in which the repressor is able to bind to the
specific binding site (active state) and one in which it cannot bind the
specific binding site (inactive state). The environmental signaling occurs via
passive import of an extracellular inducer that binds the repressor, shifting
the equilibrium between the two conformations of the repressor
\cite{Razo-Mejia2018}. In previous work we have extensively characterized the
mean response of this circuit under different conditions using equilibrium
based models \cite{Phillips2019}. Here we build upon these models to
characterize the full distribution of gene expression with parameters such as
repressor copy number and its affinity for the DNA being systematically varied.

As the copy number of molecular species is a discrete quantity, chemical master
equations have emerged as a useful tool to model their inherent probability
distribution \cite{Sanchez2013}. In \fref{fig2_minimal_model}(A) we show the
minimal model and the necessary set of parameters needed to compute the full
distribution of mRNA and its protein gene product. Specifically, we assume a
three-state model where the promoter can be found in a 1) transcriptionally
active state  ($A$ state), 2) a transcriptionally inactive state without the
repressor bound ($I$ state) and 3) a transcriptionally inactive state with the
repressor bound ($R$ state). We do not assume that the transition between the
active state $A$ and the inactive state $I$ occurs due to RNAP binding to the
promoter as the transcription initiation kinetics involve several more steps
than simple binding \cite{Browning2004}. We coarse-grain all these steps into
effective ``on" and ``off" states for the promoter, consistent with experiments
demonstrating the bursty nature of gene expression in {\it E. coli}
\cite{Golding2005}. These three states generate a system of coupled
differential equations for each of the three state distributions $P_A(m, p;
t)$, $P_I(m, p; t)$ and $P_R(m, p; t)$, where $m$ and $p$ are the mRNA and
protein count per cell, respectively and $t$ is time. Given the rates depicted
in \fref{fig2_minimal_model}(A) we define the system of ODEs for a specific $m$
and $p$. For the transcriptionally active state, we have
\begin{equation}
  \begin{aligned}
    \dt{P_A(m, p)} &=
    - \overbrace{\kpoff P_A(m, p)}^{A \rightarrow I} % A -> I
    + \overbrace{\kpon P_I(m, p)}^{I \rightarrow A}\\ % I -> A
    &+ \overbrace{r_m P_A(m-1, p)}^{m-1 \rightarrow m} % m-1 -> m
    - \overbrace{r_m P_A(m, p)}^{m \rightarrow m+1}% m -> m+1
    + \overbrace{\gm (m + 1) P_A(m+1 , p)}^{m+1 \rightarrow m} % m+1 -> m
    - \overbrace{\gm m P_A(m , p)}^{m \rightarrow m-1}\\ % m -> m-1
    &+ \overbrace{r_p m P_A(m, p - 1)}^{p-1 \rightarrow p} % p-1 -> p
    - \overbrace{r_p m P_A(m, p)}^{p \rightarrow p+1} % p -> p+1
    + \overbrace{\gp (p + 1) P_A(m, p + 1)}^{p + 1 \rightarrow p} % p+1 -> p
    - \overbrace{\gp p P_A(m, p)}^{p \rightarrow p-1}, % p -> p-1
  \end{aligned}
\end{equation}
where the state transitions for each term are labeled by overbraces. For the
transcriptionally inactive state $I$, we have
\begin{equation}
  \begin{aligned}
    \dt{P_I(m, p)} &=
    \overbrace{\kpoff P_A(m, p)}^{A \rightarrow I} % A -> I
    - \overbrace{\kpon P_I(m, p)}^{I \rightarrow A} % I -> A
    + \overbrace{\kroff P_R(m, p)}^{R \rightarrow I} % R -> I
    - \overbrace{\kron P_I(m, p)}^{I \rightarrow R}\\ % I -> R
    &+ \overbrace{\gm (m + 1) P_I(m+1 , p)}^{m+1 \rightarrow m} % m+1 -> m
    - \overbrace{\gm m P_I(m , p)}^{m \rightarrow m-1}\\ % m -> m-1
    &+ \overbrace{r_p m P_I(m, p - 1)}^{p-1 \rightarrow p} % p-1 -> p
    - \overbrace{r_p m P_I(m, p)}^{p \rightarrow p+1} % p -> p+1
    + \overbrace{\gp (p + 1) P_I(m, p + 1)}^{p + 1 \rightarrow p} % p+1 -> p
    - \overbrace{\gp p P_I(m, p)}^{p \rightarrow p-1}. % p -> p-1
  \end{aligned}
\end{equation}
And finally, for the repressor bound state $R$,
\begin{equation}
  \begin{aligned}
    \dt{P_R(m, p)} &=
    - \overbrace{\kroff P_R(m, p)}^{R \rightarrow I} % R -> I
    + \overbrace{\kron P_I(m, p)}^{I \rightarrow R}\\ % I -> R
    &+ \overbrace{\gm (m + 1) P_R(m+1 , p)}^{m+1 \rightarrow m} % m+1 -> m
    - \overbrace{\gm m P_R(m , p)}^{m \rightarrow m-1}\\ % m -> m-1
    &+ \overbrace{r_p m P_R(m, p - 1)}^{p-1 \rightarrow p} % p-1 -> p
    - \overbrace{r_p m P_R(m, p)}^{p \rightarrow p+1} % p -> p+1
    + \overbrace{\gp (p + 1) P_R(m, p + 1)}^{p + 1 \rightarrow p} % p+1 -> p
    - \overbrace{\gp p P_R(m, p)}^{p \rightarrow p-1}. % p -> p-1
  \end{aligned}
\end{equation}
As we will discuss later in \secref{sec_cell_cycle} the protein degradation
term $\gp$ is set to zero since active protein degradation is slow compared to
the cell cycle of exponentially growing bacteria, but rather we explicitly
implement binomial partitioning of the proteins into daughter cells upon
division \cite{Maurizi1992}.

It is convenient to rewrite these equations in a compact matrix notation
\cite{Sanchez2013}. For this we define the vector $\PP(m, p)$ as
\begin{equation}
  \PP(m, p) = (P_A(m, p), P_I(m, p), P_R(m, p))^T,
\end{equation}
where $^T$ is the transpose. By defining the matrices $\Km$ to contain the
promoter state transitions, $\Rm$ and $\Gm$ to contain the mRNA production and
degradation terms, respectively, and $\Rp$ and $\Gp$ to contain the protein
production and degradation terms, respectively, the system of ODEs can then be
written as (See \siref{supp_model} for full definition of these matrices)
\begin{equation}
  \begin{aligned}
    \dt{\PP(m, p)} &= \left( \Km -\Rm -m\Gm -m\Rp -p\Gp \right) \PP(m, p)\\
    &+ \Rm \PP(m-1, p)
    + (m + 1) \Gm \PP(m + 1, p)\\
    &+ m \Rp \PP(m, p - 1)
    + (p + 1) \Gp \PP(m, p + 1).
  \end{aligned}
  \label{eq_cme_matrix}
\end{equation}
Having defined the gene expression dynamics we now proceed to determine all
rate parameters in \eref{eq_cme_matrix}.

\subsection{Inferring parameters from published data sets}
\label{sec_param}

A decade of research in our group has characterized the simple repression motif
with an ever expanding array of predictions and corresponding experiments to
uncover the physics of this genetic circuit \cite{Phillips2019}. In doing so we
have come to understand the mean response of a single promoter in the presence
of varying levels of repressor copy numbers and repressor-DNA affinities
\cite{Garcia2011c}, due to the effect that competing binding sites and multiple
promoter copies impose \cite{Brewster2014}, and in recent work, assisted by the
Monod-Wyman-Changeux (MWC) model, we expanded the scope to the allosteric nature
of the repressor \cite{Razo-Mejia2018}. All of these studies have exploited the
simplicity and predictive power of equilibrium approximations to these
non-equilibrium systems \cite{Buchler2003}. We have also used a similar kinetic
model to that depicted in \fref{fig2_minimal_model}(A) to study the noise in
mRNA copy number \cite{Jones2014a}. Although these studies focus on the same
experimental system described by different theoretical frameworks, in earlier
work in our laboratory an attempt to unite parametric knowledge across studies
based on equilibrium and non-equilibrium models has not been performed
previously. As a test case of the depth of our theoretical understanding of this
simple transcriptional regulation system we combine all of the studies mentioned
above to inform the parameter values of the model presented in
\fref{fig2_minimal_model}(A). \fref{fig2_minimal_model}(B) schematizes the data
sets and experimental techniques used to measure gene expression along with the
parameters that can be inferred from them.

\siref{supp_param_inference} expands on the details of how the inference was
performed for each of the parameters. Briefly, the promoter activation and
inactivation rates $\kpon$ and $\kpoff$, as well as the transcription rate
$r_m$ were obtained in units of the mRNA degradation rate $\gm$ by fitting a
two-state promoter model (no state $R$ from \fref{fig2_minimal_model}(A))
\cite{Peccoud1995} to mRNA FISH data of an unregulated promoter (no  repressor
present in the cell) \cite{Jones2014a}. The repressor on rate is assumed to be
of the form $\kron = k_o [R]$ where $k_o$ is a diffusion-limited on rate and
$[R]$ is the concentration of active repressor in the cell \cite{Jones2014a}.
This concentration of active repressor is at the same time determined by the
repressor copy number in the cell, and the fraction of these repressors  that
are in the active state, i.e. able to bind DNA. Existing estimates of the
transition rates between conformations of allosteric molecules set them at the
microsecond scale \cite{Cui2008}. By considering this to be representative for
our repressor of interest, the separation of time-scales between the rapid
conformational changes of the repressor and the slower downstream processes
such as the open-complex formation processes allow us to model the probability
of the repressor being in the active state as an equilibrium MWC process. The
parameters of the MWC model $K_A$, $K_I$ and $\eAI$ were previously
characterized from video-microscopy and flow-cytometry data
\cite{Razo-Mejia2018}. For the repressor off rate, $\kroff$, we take advantage
of the fact that the mean mRNA copy number as derived from the model in
\fref{fig2_minimal_model}(A) cast in the language of rates is of the same
functional form as the equilibrium model cast in the language of binding
energies \cite{Phillips2015}. Therefore the value of the repressor-DNA binding
energy $\eR$ constrains the value of the repressor off rate $\kroff$. These
constraints on the rates allow us to make self-consistent predictions under
both the equilibrium and the kinetic framework. Having all parameters in hand,
we can now proceed to solve the gene expression dynamics.

\begin{figure}[h!]
	\centering \includegraphics
  {./fig/main/fig2_parameter_inference.pdf}
	\caption{\textbf{Minimal kinetic model of transcriptional regulation for a
	simple repression architecture.} (A) Three-state promoter stochastic model of
	transcriptional regulation by a repressor. The regulation by the repressor
	occurs via exclusion of the transcription initiation machinery, not allowing
	the promoter to transition to the transcriptionally active state. All
	parameters highlighted with colored boxes were determined from published
	datasets based on the same genetic circuit. Parameters in dashed boxes were
	taken directly from values reported in the literature or adjusted to satisfy
	known biological restrictions. (B) Data sets used to infer the parameter
	values. From left to right Garcia \& Phillips \cite{Garcia2011c} is used to
	determine $\kroff$ and $\kron$, Brewster et al. \cite{Brewster2014} is used
	to determine $\eAI$ and $\kron$, Razo-Mejia et al. \cite{Razo-Mejia2018} is
	used to determine $K_A$, $K_I$, and $\kron$ and Jones et al.
	\cite{Jones2014a} is used to determine $r_m$, $\kpon$, and $\kpoff$.}
  \label{fig2_minimal_model}
\end{figure}


\subsubsection{\stid{3.06} PETSc-TAO} \label{subsubsect:petsc}
\paragraph{Overview}

Algebraic solvers (generally nonlinear solvers that use sparse linear solvers) and integrators form the
core computation of many numerical simulations. No scalable ``black box'' sparse solvers or integrators
work for all applications, nor are there single implementations that work well for all problem sizes.
Hence, algebraic solver and integrator packages provide a wide variety of algorithms and implementations
that can be customized for the application and range of problem sizes. PETSc/TAO~\cite{petsc:homepage,petsc-man}
is a widely used numerical library for the scalable solution of linear, nonlinear, and variational systems,
for integration of ODE/DAE systems and computation of their adjoints, and for numerical optimization.
This project focuses on three topics: (1) partially matrix-free scalable solvers to efficiently use
many-core and GPU-based systems; (2) reduced synchronization algorithms that can scale to larger
concurrency than solvers with synchronization points; and (3) performance and data structure
optimizations for all the core data structures to better utilize many-core and GPU-based
systems as well as provide scalability to the exascale systems.

The availability of systems with over 100 times the processing power of today's machines compels the utilization
of these systems not just for a single ``forward solve'' (as discussed above), but rather within a tight loop
of optimization, sensitivity analysis (SA), and uncertain quantification (UQ). This requires the implementation
of a new scalable library for managing a dynamic hierarchical collection of running scalable simulations, where
the simulations directly feed results into the optimization, SA, and UQ solvers.  This library, which we call
libEnsemble, directs the multiple concurrent ``function evaluations'' through the tight coupling and
feedback. This work consist of two parts: (1) the development of libEnsemble; and (2) the development
of application-relevant algorithms to utilize libEnsemble.

\paragraph{Key Challenges}

A key challenge for scaling the PETSc/TAO numerical libraries to Exascale systems is that traditional
``sparse-matrix-based'' techniques for linear, nonlinear, and ODE solvers, as well as optimization
algorithms, are memory-bandwidth limited.  Another difficulty is that any synchronizations
required across all compute units---for example, an inner product or a norm---can
dramatically affect the scaling of the solvers.  Another challenge is the need to
support the variety of accelerators that will be available on the exascale systems
and the programming models that application teams use for performance
portability.

Running an ensemble of simulations requires a coordination layer that handles load balancing and
allows the collection of running simulations to grow and shrink based on feedback. Thus, our
libEnsemble library must be able to dynamically start simulations with different parameters,
resume simulations to obtain more accurate results, prune running simulations that the solvers
determine can no longer provide useful information, monitor the progress of the simulations,
and stop failed or hung simulations, and collect data from the individual simulations both
while they are running and at the end.

\paragraph{Solution Strategy}

To address the scalability of the numerical libraries, we implemented new solvers and data
structures including: pipeline Krylov methods that delay the use of the results of inner
products and norms, allowing overlapping of the reductions and other computation; partially
matrix-free solvers using high-order methods that have high floating-point-to-memory-access
ratios and good potential to use many-core and GPU-based systems; and in-node optimizations
of sparse matrix-matrix products needed by algebraic multigrid to better utilize many-core
systems.

Our strategy for coordinating ensemble computations has been to develop libEnsemble
to satisfy our needs.  This library should not be confused with workflow-based
scripting systems; rather it is a library that, through the tight coupling and
feedback, directs the multiple concurrent ``function evaluations'' needed by
optimization, SA, and UQ solvers.

\paragraph{Recent Progress}

In the past year, we have released PETSc/TAO 3.14 (available at \url{http://www.mcs.anl.gov/petsc}),
which features enhanced GPU support.  The library now supports CUDA-11 and HIP, along with CUDA-aware
MPI, which allows direct communication of data between Summit GPUs, bypassing the previously needed
step of first copying the data to the CPU memory. This enhancement reduces the latency of the
communication and improves bandwidth.  An experimental Kokkos backend for some matrix and
vector operations using KokkosKernels was also provided, as one step in the refactoring
process to support the variety of accelerators needed for exascale systems and the
programming models for performance portability wanted by applications.

\begin{figure}
\centering
\includegraphics[trim = 0in .2in 1.7in .2in, clip, width=0.9\textwidth]{projects/2.3.3-MathLibs/2.3.3.06-PETSc-TAO/petsc_arch}
\caption{The improved PETSc/TAO architecture enables users to utilize a variety of programming
models for GPUs independently of PETSc's internal programming model.}
\label{fig:petsc-tao-fig}
\end{figure}

We have also release libEnsemble 0.7.1 (available at \url{https://github.com/Libensemble/libensemble}).
This release includes new generator functions and examples, changes to become xSDK compatible, and
improved testing across available platforms.

\paragraph{Next Steps}

Our next efforts are:
\begin{enumerate}
  \item \textbf{Performance and application assessment}:
  We will provide updated performance reports of PETSc/TAO on the architectures available to us.
  We will work with our applications to assess the usage of our software technologies and our
  progress toward reaching our impact goals.
  We will add a libEnsemble guide for function writer users to the documentation and survey
  the libEnsemble user community.
  \item \textbf{PETSc/TAO release with full-functionality on available hardware}:
  We will release a version of PETSc/TAO that fully supports the hardware and software on the
  architectures available to us.
  We will begin testing important kernels using different backends and prepare more methods to utilize accelerators.
  \item \textbf{libEnsemble release with enhanced capabilities}:
  We will release a version of libEnsemble that implements a method for Bayesian calibration.
  We will connect libEnsemble to continuous integration tools and demonstrate capabilities.
  \item \textbf{PETSc/TAO release focused on performance on available hardware}:
  We will release a version of PETSc/TAO with performance improvements on the architectures available to us.
  We will continue testing important kernels using different backends and optimize more methods to
  utilize accelerators.
\end{enumerate}


\documentclass[fleqn,10pt]{wlscirep}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{listings}
\usepackage{xcolor}
\usepackage{booktabs}
\usepackage{float}
\usepackage{fancyhdr}
\usepackage{caption}
\usepackage{graphicx}

\graphicspath{ {./} }

\pagestyle{fancy}
\fancyhf{}
\fancyhead[LE,RO]{Dharak Vasavda}
\fancyhead[RE,LO]{Parallel Binary Search with OpenMP}
\fancyfoot[LE,RO]{\thepage}

\renewcommand{\headrulewidth}{2pt}
\renewcommand{\footrulewidth}{1pt}

\DeclareCaptionFont{white}{\color{white}}
\DeclareCaptionFormat{listing}{\colorbox{gray}{\parbox{\textwidth}{#1#2#3}}}
\captionsetup[lstlisting]{format=listing,labelfont=white,textfont=white}
\captionsetup{justification=raggedright,singlelinecheck=false}

% This concludes the preamble

\title{Parallel Binary Search Using OpenMP}

\author{Dharak Vasavda}
\affil{$d\_vasavda@u.pacific.edu$}

%\keywords{Keyword1, Keyword2, Keyword3}

\begin{document}

\maketitle
\flushbottom
\pagebreak
\raggedbottom

\section{Problem}
Binary Search is a divide-and-conquer searching algorithm which searches for an input value in a sorted array. The average-case time complexity for Binary Search is $\mathcal{O}(log{}n)$. Considering that it is a divide-and-conquer search where the input array is repeatedly divided into equal halves, can the search operation be made even faster using OpenMP parallelization?

\subsection*{Prior Solutions}
Parallel Binary Search implementations with OpenMP do exist, however; rarely is there any specifically real-world timing data comparison between a serial and parallel implementation. Due to such circumstance, this scientific paper aims to compare the real-world performance between a parallel and serial implementation. It also aims to investigate any real-world performance differences between a serial and parallel approach to Binary Search.

\section{Design}
\subsection*{Typical Serial Binary Search}
Typically, a serial approach to non-recursive Binary Search algorithm goes as follows:

\begin{lstlisting}[language=C, caption=Serial Binary Search]
    while (first <= last) {
        int middle = first + (last - first) / 2;

        // Check if search value is present at middle
        if (randomNums[middle] == searchVal)
            return middle;

        // If search value greater, ignore left half
        if (randomNums[middle] < searchVal) {
            first = middle + 1;
        }

        // If search value is smaller, ignore right half
        else
            last = middle - 1;
    }
\end{lstlisting}

\pagebreak

\subsection*{Parallel Binary Search Design}
The sample code block below may assist with understanding the parallel solution.

\begin{lstlisting}[language=C, caption=OpenMP Binary Search]
#pragma omp parallel num_threads(num_threads)
    {
#pragma omp sections
        {
        /* Function parameters:
            binarySearch_openmp(first_index, last_index, search_value);
        */

#pragma omp section
            thread_one = binarySearch_openmp(0, quarter_slice, searchVal);
#pragma omp section
            thread_two = binarySearch_openmp(quarter_slice + 1, middle, searchVal);
#pragma omp section
            thread_three = binarySearch_openmp(middle + 1, quarter_slice * 3, searchVal);
#pragma omp section
            thread_four = binarySearch_openmp((quarter_slice * 3) + 1, last, searchVal);
        }
    }
\end{lstlisting}

\subsubsection*{Parallel Algorithm Explanation}
Each call to the function responsible for binary search is now given the input of a new array size, where each input array is a quarter size of the original input array. From there, each thread will simultaneously perform a search operation on a quarter section of the array. Once a single thread finds the desired value to search for, all threads will stop the searching process and return the index of the search value.
\\
The pragma directive of choice here is using sections, so that each task can run on its own thread. The idea behind this algorithm is to reduce the search array size and reduce the number of operations, while each thread can perform each operation in parallel.

\pagebreak
\section{Experiments}
\subsection*{Data Sources}
All data was generated using an integer generator. All numbers are placed in ascending order into a file which will be used by the operation code. The data is inserted into an array in the source code to perform Binary Search experimentation upon.

\begin{table}[H]
\begin{tabular}{@{}ll@{}}
\toprule
\textbf{Data Type}   & Integer          \\ \midrule
\textbf{Size}        & 200,000,000      \\ \midrule
\textbf{Memory Type} & fixed-size array \\ \midrule
\textbf{Order}       & Ascending        \\ \midrule
\textbf{Sorted}      & Yes              \\
\bottomrule
\end{tabular}
    \caption{Data Source Information}
\end{table}

\subsection*{Results}
Below, contains both timing results between Parallel Binary Search and Parallel Binary Search using OpenMP. Timing data was recorded using \code{$omp\_get\_wtime()$}. Four runs were taken per entry and timing data was averaged. The operation was a Binary Search, where the user will input any number between 1 and 200,000,00 to search for.

\vspace*{20px}

\includegraphics[scale=0.6]{runtime.png}
\vspace*{20px}

\begin{table}[hbt!]
\begin{minipage}[b]{80mm}
\begin{tabular}{@{}ll@{}}
\toprule
\textbf{Thread Count} & \textbf{Run time (seconds)} \\ \midrule
1 (Serial)            & $\sim$0.000005  \\
\bottomrule
\\
\\
\\
\\
\end{tabular}
    \caption{Serial Binary Search Timing}
\end{minipage}
\begin{minipage}[b]{80mm}
\begin{tabular}{@{}ll@{}}
\toprule
\textbf{Thread Count} & \textbf{Run time (seconds)} \\ \midrule
1                     & 0.0000061            \\ \midrule
2                     & 0.0000269            \\ \midrule
4                     & 0.0000421            \\ \midrule
8                     & 0.0000564          \\
\bottomrule
\end{tabular}
    \caption{Parallel Binary Search Timing}
\end{minipage}
\end{table}

\\
\section{Analysis}

\begin{table}[hbt!]
\begin{tabular}{@{}ll@{}}
\toprule
\textbf{Parallel Time Complexity}                & $\mathcal{O}(log{}(n/k))$ \\ \midrule
\textbf{Speedup Estimate}                        & $\frac{log(n^2)}{k}$      \\ \midrule
\textbf{Efficiency Estimate}                     & $\frac{log(n^2)}{k^2}$    \\
\bottomrule
\end{tabular}
    \caption{Theoretical Estimates of Parallel Binary Search}
\end{table}

\subsection*{Real-World Results}
Referring to the data displayed in Table 2 and Table 3, the serial implementation of Binary Search is significantly faster than the parallel implementation. As the number of threads performing Binary Search increased, the operation took exponentially longer. There are a few reasons I can hypothesize as to why these results occurred.

\subsubsection*{Effectivity Limits of Parallel Solutions}
One crucial idea to note, when finding a parallel approach to a problem, is that the serial operation needs to be costly enough to computer resources to parallelize. Often times, complex calculations such as matrix multiplication or dot product are used to demonstrate parallel effectivity, for a specific reason. That is, the operation needs to be costly enough to outweigh the performance repercussions of a parallel solution. Specific performance repercussions such as thread spawning, excessive function calls, voltage and thermal limitations across several threads, and cache hierarchy issues can all play a part in slowing down performance. In the case of Binary Search, the operation simply is not costly enough to make useful out of a parallel approach. OpenMP is not a solution to all problems.

\section{Implementation}
\subsection*{Building and Running This Application}
To build the project and run experiments with it:
\begin{enumerate}
  \item cd into project directory
  \item Run the integer generator, which will output all integer data into input1.txt
  \item Compile the C file
  \item Run the compiled file
\end{enumerate}
\\
\begin{lstlisting}[language=bash]
> cd openmp-binary-search
> python3 integer_generator.py
> clang -Xpreprocessor -fopenmp -lomp binarySearch_openmp.c -o binarySearch_openmp
> ./binarySearch_openmp
\end{lstlisting}

\subsection*{Running Experiments}
There are several factors which need to be accounted for before running experiments and gathering data for a similar program:
\\
\begin{itemize}
  \item Ensure multiple data points are calculated for averaging
  \item Ensure you are running costly enough operations
  \item Ensure operation data set is large enough
  \item Keep aware that high-resolution timers are not always fully accurate for each floating point
  \item Therefore, do research before choosing a timer
  \item Create a stable thermal environment for the CPU before collecting data
\end{itemize}
\\
After following the above awareness points, the data collection goes as follows:
\\
\begin{enumerate}
  \item Generate all numbers using the random number generator
  \item Pick several numbers of threads in powers of 2
  \item Run the program with each thread count, four times total
  \item Average out each run
  \item Use the average as the calculated run time
\end{enumerate}
\\

\pagebreak
\section{Code Appendix}

\lstinputlisting[language=C, caption=$binarySearch\_openMP.c$]{binarySearch_openmp.c}
\pagebreak
\lstinputlisting[language=Python, caption=$integer\_generator.py$]{integer_generator.py}

\end{document}


\documentclass[11pt, a4paper]{article}
\usepackage[margin=1in]{geometry}
\usepackage[hidelinks]{hyperref}
\usepackage{amsfonts, amssymb, amsmath}
\usepackage{enumerate}
\usepackage{float}
\usepackage{tikz}
% \usepackage{pgfplots}
% \usetikzlibrary{decorations.pathreplacing}

\title{
    \textbf{Math 401 Notes} \\
    Lecture 1\&2 \\ (Statistics)
}

\author{Ahmed Nasser}
\date{\today}

\begin{document}
    \maketitle
    \tableofcontents
    \pagebreak

    \section{Statistics Steps}
    \begin{enumerate}
        \item Collect
        \item Organize
        \item Present
        \item Summarize
        \item Draw Conclusion
    \end{enumerate}

    \section{Types of Data}
    \begin{tikzpicture}[sibling distance=10em,
        every node/.style = {shape=rectangle, rounded corners,
          draw, align=center,
          top color=white, bottom color=blue!20}]]
        \node {Data}
          child { node {Qualitative} }
          child { node {Quantitative}
            child { node {discrete}}
            child { node {continuous} } };
      \end{tikzpicture}

      \section{Types of statistical studies}
      \begin{enumerate}
          \item \textbf{Descriptive statistics}
          \subitem collecting, organizing,
          presenting, and summarizing of data, in
          an informative way.
          \item \textbf{Inferential statistics}
          \subitem drawing conclusions and making
          decisions using the given data
      \end{enumerate}

      \section{Present (Graphs)}
      \subsection{Histogram}
      \begin{itemize}
          \item $X$-axis -- Class Limits
          \item $Y$-axis -- “$f$” or “$r. f.$”
      \end{itemize}
      \subsection{Ogive}
      \begin{itemize}
        \item $X$-axis -- Upper limits of each class limit
        \item $Y$-axis -- “$c.f.$” or “$c.r. f.$”
    \end{itemize}

      \section{Range \& Width}
      \begin{enumerate}
          \item Range = Max Val - Min Val
          \item Class Width = $\displaystyle \frac{\text{Range}+1}{\text{No. of classes}}$  {\ \ \ \ \ \small If this value is not an integer ROUND IT UP}
      \end{enumerate}

     \section{ grouped frequency distribution}
     \begin{itemize}
         \item If the \textbf{number of classes is given} and the given data values are integers: we
         proceed using the \textbf{direct method}
         \item If the \textbf{number of classes is NOT given} and the given data values are integers
or decimals
        \subitem If an \textbf{initial value is NOT given}, we proceed using the \textbf{Stem-leaf method}.
        \subitem If an \textbf{initial value is given}, we proceed using the \textbf{(
            $\sqrt{n}$ ) method}. (Width = $\frac{\text{Range}}{\text{No. of classes}}$)
     \end{itemize}

     \section{The Median}

     \subsection{Discrete Data}
     \begin{itemize}
         \item Sort the Data value
         \item MD = middle point  {\small (in case of odd num)}
         \item MD = (sum of the two middle points) / 2  {\small (in case of odd num)}
     \end{itemize}

     \subsection{Grouped data}
     $$\text{MD} = \frac{\frac{n}{2} - \text{pre.}"c.f"}{f}(W)+L_m$$
     where $W$ is the class width and $L_m$ is the lower bound \\
     \textbf{ex:} if the median is 80 then about 50\% of student achieved more than 80 and 50\% achieved less than 80

     \section{The Mode}
     \subsection{Discrete Data}
     \begin{itemize}
        \item The mode is a data value that has the highest frequency in a data set
        \item If all of data values have the same frequency then there is no mode

     \end{itemize}
     \subsection{Grouped data}
     \begin{itemize}
         \item We use Modal class instead of mode in grouped data
         \item The modal class is the class corresponds to the highest frequency
     \end{itemize}

     \section{The Mean}
     \subsection{Discrete Data}
     \begin{itemize}
         \item For a population
         $$\displaystyle \mu = \frac{\sum_{i = 1}^{N} X_i}{N}$$
         \item For a Sample
         $$\displaystyle \bar{X} = \frac{\sum_{i = 1}^{n} X_i}{n}$$
     \end{itemize}

     \subsection{Grouped data}
     $$\mu = \frac{\sum f.X_m}{\sum f}=\frac{\sum f.X_m}{N} $$
     Where $X_m$ is the class midpoint

     \subsection{Weighted data}
     $$\mu = \frac{\sum_{i = 1}^{n} W_iX_i}{\sum_{i = 1}^{n} W_i}$$

     \section{The Standard Deviation}
     \begin{itemize}
        \item For a population
        $$\sigma := \sqrt{\sigma^2}$$
        \item For a Sample
        $$S := \sqrt{S^2}$$
    \end{itemize}

    \section{The Population Variance}
    \subsection{Discrete Data}
    \begin{itemize}
        \item For a population
        $$\sigma^2 = \frac{\sum_{i=1}^N (X_i-\mu)^2}{N}$$
        \item For a Sample
        $$S^2 = \frac{\sum_{i=1}^n (X_i-\bar{X})^2}{n-1}$$
        $$S^2 = \frac{\sum_{i=1}^n X_i^2-\frac{\left(\sum_{i=1}^nX_i\right)^2}{n}}{n-1}$$

    \end{itemize}

    \subsection{Grouped data}
    \begin{itemize}
        \item For a population
        $$\sigma^2 = \frac{\sum f.(X_m-\mu)^2}{\sum f}$$
        \item For a Sample
        $$S^2 = \frac{\sum f.X_m^2-\frac{\left(\sum f.X_m\right)^2}{\sum f}}{\left(\sum f\right)-1}$$

    \end{itemize}

    \section{Chebyshev’s Theorem}
    $[\bar{x}-ks,\bar{x}+ks]$ is the smallest interval thant contains at least $\left(1-\frac{1}{k^2}\right)*100\%$ of the data

    \section{Percentiles}
    \begin{itemize}
        \item Sort your data values
        $$\text{Percentile} = \frac{0.5 + \text{No. of values below }X}{n}*100\% $$
        \item $n$ is number of data and $p$ is the percentile then evaluate m = $n.p$
        \subitem if $m$ is not integer round it up and then $p$ corresponds \textbf{data value}[m]
        \subitem if $m$ is integer then $p$ corresponds the average of \textbf{data value}[m] and \textbf{data value}[m+1]
    \end{itemize}

    \section{Quartiles}
    \begin{itemize}
     \item $Q_1 = 25^{\text{th}}$ percentile
     \item $Q_2 = 50^{\text{th}}$ percentile |or| median
     \item $Q_3 = 75^{\text{th}}$ percentile
    \end{itemize}
\end{document}


\subsection{Estimating the population mean}


\documentclass[letterpaper,12pt]{article}

\usepackage{amsmath, amsfonts, amscd, amssymb, amsthm}
\usepackage{graphicx}
%\usepackage{import}
\usepackage{versions}
\usepackage{crop}
\usepackage{multicol}
\usepackage{graphicx}
\usepackage{makeidx}
\usepackage{hyperref}
\usepackage{ifthen}
\usepackage[format=hang,font=normalsize,labelfont=bf]{caption}
\usepackage{setspace}
\usepackage{placeins}
\usepackage{framed}
\usepackage{enumitem}
\usepackage{threeparttable}
\usepackage{geometry}
\geometry{letterpaper,tmargin=1in,bmargin=1in,lmargin=1in,rmargin=1in}
\usepackage{multirow}
\usepackage[table]{xcolor}
\usepackage{array}
\usepackage{delarray}
\usepackage{lscape}
\usepackage{float,color, colortbl}
%\usepackage[pdftex]{graphicx}
\usepackage{hyperref}
\usepackage{tabu}

\include{thmstyle}
\providecommand{\abs}[1]{\lvert#1\rvert}
\providecommand{\norm}[1]{\lVert#1\rVert}
\newcommand{\ve}{\varepsilon}
\newcommand{\ip}[2]{\langle #1,#2 \rangle}

\crop
\makeindex

\begin{document}

\begin{center}
{\LARGE \bf Profit Shifting Model}
\end{center}

\begin{spacing}{1.5}

\section{Model with Perceived Risk}

	Assume mean-variance utility.  Two returns are uncorrelated.

	\begin{align}
		E\{r_D - r_F\} & = \gamma C\{r_D - r_F,r_P\} \nonumber \\
		\mu_D - \mu_F & = \gamma C\{r_D - r_F,\alpha r_D + (1-\alpha) r_F\} \nonumber \\
		\mu_D - \mu_F & = \gamma \left[ \alpha \sigma^2_D - (1-\alpha) \sigma^2_F \right] \nonumber \\
		\frac{1}{\gamma}\left[\mu_D - \mu_F\right] + \sigma^2_F & = \alpha \left[ \sigma^2_D + \sigma^2_F \right] \nonumber \\
		\alpha & = \frac{\tfrac{1}{\gamma}\left[\mu_D - \mu_F\right] + \sigma^2_F}{\sigma^2_D + \sigma^2_F} \nonumber \\
		\alpha & = \frac{\tfrac{1}{\gamma}\left[\bar r_{US}(1-\tau_{US}) - \bar r_{FOR}(1-\tau_{FOR})\right] + \sigma^2_F}{\sigma^2_D + \sigma^2_F} \nonumber \\
	\end{align}

\end{spacing}

\end{document}