We now give an overview of common data structures used in \BoxLib, followed by a simple
example written in both Fortran90 and C++ that makes use of these structures.
The example advances two scalar variables in time on a single level with multiple
grids (no AMR) and produces plotfiles.

\section{Overview of Data Structures}

\BoxLib\ contains the most fundamental objects used to construct parallel
block-structured AMR applications.
At each level of refinement, the region covered by that level is divided
into grids, or boxes.  The entire computational domain is covered by
the coarsest (base) level of refinement (called level $\ell=0$ in C++
and called level $\ell=1$ in Fortran90) and can be represented on one
grid or divided into many grids.
Higher levels of refinement have cells that are finer by a ``refinement ratio''
of either 2 or 4 (in C++) or 2 (in Fortran90).  The grids are properly nested in the sense that the union
of grids at level $\ell+1$ is contained in the union of grids at level $\ell$.
Furthermore, the containment is strict in the sense that, except at physical
boundaries (i.e., domain boundaries that are not periodic),
the level $\ell$ grids are large enough to guarantee that there is
a border at least $n_{\rm buffer}$ (typically 4) level $\ell$ cells wide surrounding each level
$\ell +1$ grid (grids at all levels are allowed to extend to the physical
boundaries so the proper nesting is not strict there).  See Figure \ref{fig:AMR}
for a sample two-dimensional grid structure.\\
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{figure}[tb]
\centering
\includegraphics[width=6.5in]{./Introduction/AMR}
\caption{\label{fig:AMR}Sample grid structure with two levels of refinement.  These
grids satisfy the requirements that the base grid covers the entire computational domain
and the grids are properly nested.  Note that refined grids are allowed to extend to physical
domain boundaries without coarser buffer cells.}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

On a grid, the data can be stored at cell-centers, faces, edges, or
corners.  In \BoxLib, data that is on an face is termed `nodal'
in that one direction (see Figure~\ref{fig:dataloc}).  In three-dimensions (not pictured),
data that is nodal in two directions is said to live on edges.  Data that is nodal in
all directions lives on the corners of cells (commonly referred to as the nodes).
\BoxLib\ uses $0$-based spatial indexing, and for data that is nodal in one or more direction,
the integer index corresponds to the lower boundary in that direction (see Figure~\ref{fig:dataloc}).
In our \BoxLib\ applications, the state data (velocity, density,
species, $\ldots$) is typically cell-centered.  Fluxes are typically nodal in exactly
one direction (i.e.~they are face-centered).  A few quantities are nodal in all
directions (e.g.~the pressure in the low Mach number projection methods).
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{figure}[tb]
\centering
\includegraphics[width=6.5in]{./Introduction/data_loc2}
\caption{\label{fig:dataloc} Some of the different data-centerings in two dimensions:
(a) cell-centered, (b) nodal in the $x$-direction only (face-centered), and (c) nodal in
both the $x$- and $y$-directions.  Note that for data that is nodal in one or more direction,
the integer index corresponds to the lower boundary in that direction.
Also note that \BoxLib\ uses $0$-based indexing, e.g., in each of these centerings,
the red point has the same indices:\ (1,2).
Not shown is the case where data is nodal in the $y$-direction only.
Also not shown is the three-dimensional edge-centered case, where the data
is nodal in exactly two directions.  }
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{itemize}
\item In C++ \BoxLib, we must specify the number of spatial dimensions (1, 2, or 3),
{\tt DIM}, at compile-time.  The code that will be built is specifically designed to
run only with that number of dimensions.
\item In Fortran90 \BoxLib, we build dimension-independent code at compile-time,
and tell the program the dimensionality of the problem via a runtime inputs file.
\end{itemize}

To simplify the description of the underlying AMR grid, \BoxLib\
provides a number of classes.  We now briefly summarize some of the major
classes.

\subsection{\IntVect}

\IntVect s are {\tt DIM}-tuples of integers that are used to define
indices in space.  In C++, an example of an \IntVect\ in 2D would be
(C++ source code will be shaded blue):
\begin{lstlisting}[backgroundcolor=\color{light-blue}]
IntVect iv(3,5);
\end{lstlisting}
In Fortran90, we don't use \IntVect s, but instead use standard
arrays of integers (Fortran90 source code will be shaded green):
\begin{lstlisting}[backgroundcolor=\color{light-green}]
integer :: iv(2)
iv(1) = 3
iv(2) = 5
\end{lstlisting}

\subsection{\BoxType}

A \BoxType\ is simply a rectangular domain in space and does not hold any data.
A \BoxType\ contains the indices of its low end and high end,
{\tt IntVect lo} and {\tt IntVect hi}.
\begin{itemize}
\item In C++, a \BoxType\ also
contains an {\tt IndexType} (cell-centered, face-centered, or nodal) for each
spatial direction.
\item In Fortran90, a \BoxType\ also contains the dimensionality
of the \BoxType.
\end{itemize}
To build a \BoxType\ in C++ use:
\begin{lstlisting}[backgroundcolor=\color{light-blue}]
IntVect iv_lo(0,0);
IntVect iv_hi(15,15);
Box bx(iv_lo,iv_hi);
\end{lstlisting}
To build a \BoxType\ in Fortran90 use:
\begin{lstlisting}[backgroundcolor=\color{light-green}]
type(box) :: bx
integer   :: iv_lo(2), iv_hi(2)
iv_lo(1:2) = 0
iv_hi(1:2) = 15
bx = make_box(lo,hi)
\end{lstlisting}

The computational domain is divided into non-overlapping grids.
The collection of grids at the same resolution comprise a level.
Figure~\ref{fig:boxes} shows three grids at the same level of
refinement.  Note that this figure cannot represent the base level of refinement,
since it would require that the grids span the problem domain.
The position of the grids is with respect to a global
index space that covers the entire domain at that level and uses 0-based indexing.
For example, the \BoxType\ associated with grid 1
in the figure has {\tt lo} = (2,6) and {\tt hi} = (5,13).
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{figure}[tb]
\centering
\includegraphics[width=4.0in]{./Introduction/index_grid2}
\caption{\label{fig:boxes} Three boxes that comprise a single level.
At this level of refinement, the domain is 16$\times$16 cells and
the global index space runs from 0 to 15 in each coordinate direction.
Note that these grids cannot be at the coarsest level,
since it would require that the grids span the problem domain.}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{itemize}
\item Example: For a simulation with 32 cells in each direction at the coarsest level,
the global index space for the coarsest level runs from 0 to 31 in each coordinate
direction.  Assuming refinement ratios of 2, the next finer level
will have a global index space running from
0 to 63 in each coordinate direction (corresponding to $64 \times
64$ zones if fully refined), and the next finer level will have a global index
space running from 0 to 127 in each coordinate direction
(corresponding to $128\times 128$ zones if fully refined).
\end{itemize}

\subsection{\BoxArray}

A \BoxArray\ is an array of \BoxType es.  The size of the array is the
number of \BoxType es in the \BoxArray.  Suppose your problem domain
has lo indices (0,0) and hi indices (15,15), and you want to define
a \BoxArray\ to contain four $8\times 8$ boxes to cover the problem domain.
In Fortran90, you could do the following:
\begin{lstlisting}[backgroundcolor=\color{light-green}]
integer        :: lo(2), hi(2)
type(box)      :: bx(4)
type(boxarray) :: ba
lo(1) = 0
lo(2) = 0
hi(1) = 7
hi(2) = 7
bx(1) = make_box(lo,hi)
lo(1) = 8
lo(2) = 0
hi(1) = 15
hi(2) = 7
bx(2) = make_box(lo,hi)
lo(1) = 0
lo(2) = 8
hi(1) = 7
hi(2) = 15
bx(3) = make_box(lo,hi)
lo(1) = 8
lo(2) = 8
hi(1) = 15
hi(2) = 15
bx(4) = make_box(lo,hi)
call boxarray_build_v(ba,bx)
\end{lstlisting}
This is rather cumbersome, so instead we use other \BoxLib\ functions to
build the same \BoxArray:
\begin{lstlisting}[backgroundcolor=\color{light-green}]
type(boxarray) :: ba
integer        :: lo(2), hi(2)
type(box)      :: bx
lo(1:2) = 0
hi(1:2) = 15
bx = make_box(lo,hi)
call boxarray_build_bx(ba,bx)  ! the boxarray has one 16^2 box
call boxarray_maxsize(ba,8)    ! the boxarray has four 8^2 boxes
\end{lstlisting}
The analogous code in C++ is
\begin{lstlisting}[backgroundcolor=\color{light-blue}]
IntVect lo(0,0), hi(15,15);
Box bx(lo,hi);
BoxArray ba(bx);  // the BoxArray has one 16^2 box
ba.maxSize(8);    // the BoxArray has four 8^2 boxes
\end{lstlisting}

\subsection{\layout\ (Fortran90 Only)}

A \layout\ is a more intelligent \BoxArray, since it contains a \BoxArray\ as well
as the associated processor assignments, \BoxType\ connectivity, and many other
parallel constructs.  In the simplest case, if we have a \BoxArray\ {\tt ba} (obtained
from the example above), a
\layout\ can be defined using:
\begin{lstlisting}[backgroundcolor=\color{light-green}]
type(layout) :: la
call layout_build_ba(la,ba)
\end{lstlisting}
In C++, the information that is contained in the Fortran90 \layout\ is part of
the \MultiFab\ class.

\subsection{\FArrayBox}

A \FArrayBox\ (or \Fab) is a ``Fortran array box'' that holds data.  It contains the
\BoxType\ that it is built on as well as a pointer to the data
that can be sent to a Fortran routine.
In Fortran90, \Fab\ data is stored in a four-dimensional array,
{\tt (nx,ny,nz,nc)} in size, regardless of the dimensionality of the
problem.  Here {\tt nc} is the number of components, for instance
representing different fluid variables.  For 2D problems, {\tt nz=1}.\\

In \BoxLib, we don't usually deal with
\Fab s alone, but rather through \MultiFab s, described next.

\subsection{Floating point data}

Floating point data in C++ is declared as {\tt Real} which is {\tt
typedef} to either {\tt float} or {\tt double} depending on how {\tt
PRECISION} is set in the {\tt GNUmakefile}.  This is defined in {\tt
REAL.H}.

In Fortran90, the {\tt bl\_types} module defines a type {\tt dp\_t} that
is double precision.  Floating point types should be declared using
{\tt real (kind=dp\_t)}.

\section{The \MultiFab}
\MultiFab s are so important that we will give them their own section.
A \MultiFab\ is a collection of all the \Fab s at the same level of
refinement.
\begin{itemize}
\item In C++, a \MultiFab\ is defined using a \BoxArray,
number of components, and number of ghost cells that each \Fab\
will have.
\item In Fortran90, a \MultiFab\ is defined using a \layout,
number of components, and number of ghost cells that each \Fab\
will have.
\end{itemize}
A \MultiFab\ has a ``valid'' region that is defined by
the \BoxArray~or \layout.  Each \Fab\ in the \MultiFab\ is built large enough
to hold valid data and ghost cell data, and thus the \BoxType\ associated with
each \Fab\ is a grown version of the corresponding \BoxType\ from the \BoxArray.
Thus, a \Fab\ has no concept
of ghost cells, it merely has a single \BoxType\ that identifies it.\\

To build a \MultiFab, we require a \layout\ (in Fortran90) or a \BoxArray\ (in C++).
In Fortran90, using the \layout\ built above, we build a \MultiFab\ using:
\begin{lstlisting}[backgroundcolor=\color{light-green}]
type(multifab) :: data
call multifab_build(data,la,2,6) ! build the multifab with 2 components
                                 ! and 6 ghost cells

...                              ! do fun stuff with the data

call multifab_destroy(data)      ! free up memory to prevent leaks
\end{lstlisting}
In C++, using the \BoxArray\ built above, you could either directly
build a \MultiFab\ using:
\begin{lstlisting}[backgroundcolor=\color{light-blue}]
MultiFab data(ba,2,6);                     // build a MultiFab
\end{lstlisting}
or a pointer to a \MultiFab\ using:
\begin{lstlisting}[backgroundcolor=\color{light-blue}]
MultiFab* data = new MultiFab(ba,2,6);     // build pointer to MultiFab

...                                        // do fun stuff with the data

delete data;                               // need to free the memory
\end{lstlisting}

\subsection{Accessing \MultiFab\ Data}
Here is some sample Fortran90 code to access data within a \MultiFab:
\begin{lstlisting}[backgroundcolor=\color{light-green}]
integer                  :: i,dm,ng,nc,lo(2),hi(2)
type(multifab)           :: data
real(kind=dp_t), pointer :: dp(:,:,:,:)

...    ! build multifab ``data'' as described above

dm = data%dim   ! dm is dimensionality
ng = data%ng    ! ng is number of ghost cells
nc = data%nc    ! nc is number of components

! loop over the grids owned by this processor
do i=1,nfabs(data)
   dp => dataptr(data,i)          ! dp points to data inside fab
   lo = lwb(get_box(data,i))      ! get lo indices of box
   hi = upb(get_box(data,i))      ! get hi indices of box
   select case(dm)
   case (2)
      call work_on_data_2d(dp(:,:,1,:), ng, nc, lo, hi)
   case (3)
      call work_on_data_3d(dp(:,:,:,:), ng, nc, lo, hi)
    end select
end do

! fill periodic domain boundary and neighboring grid ghost cells
call multifab_fill_boundary(data)

...

subroutine work_on_data_2d(data, ng, nc, lo, hi)

  integer          :: lo(2), hi(2), ng
  double precision :: data(lo(1)-ng:,lo(2)-ng:,:)

  ! local variables
  integer :: i,j,n

  do j=lo(2),hi(2)
     do i=lo(1),hi(1)
        do n=1,nc
           ! some silly function I made up
           data(i,j,n) = (i + j) * n
        end do
     end do
  end do

end subroutine work_on_data_2d
\end{lstlisting}

In C++:
\begin{lstlisting}[backgroundcolor=\color{light-blue}]
#if    defined(BL_FORT_USE_UPPERCASE)
#define FORT_WORK_ON_DATA            WORK_ON_DATA
#elif  defined(BL_FORT_USE_LOWERCASE)
#define FORT_WORK_ON_DATA            work_on_data
#elif  defined(BL_FORT_USE_UNDERSCORE)
#define FORT_WORK_ON_DATA            work_on_data_
#endif

extern "C"
{
    void FORT_WORK_ON_DATA (
        Real* data, const int* Ncomp, const int* ng,
        const int* lo, const int* hi);
}

int main()
{
  int Ncomp = 2, Nghost = 6;

  ...    // build pointer to MultiFab* ``data'' as described above

  // MFIter is a ``MultiFab Iterator'' that essentially
  // loops over grids
  for ( MFIter mfi(*data); mfi.isValid(); ++mfi )
  {
    const Box& bx = mfi.validbox();
    FORT_WORK_ON_DATA((*data)[mfi].dataPtr(),
                      &Nghost, &Ncomp, bx.loVect(), bx.hiVect());
  }
  // fill periodic domain boundary and neighboring grid ghost cells
  data->FillBoundary();
}
\end{lstlisting}
The {\tt FORT\_WORK\_ON\_DATA} calls a Fortran90 subroutine which is nearly
identical to the Fortran90 example given above.  The only difference is the subroutine
name cannot have the \_2d or \_3d in its name.  Thus, the 2d and 3d versions
are both named {\tt subroutine work\_on\_data}, and must be written in
different .f90 files, where the make system determines which version to compile
based on {\tt DIM}.\\

The {\tt multifab\_fill\_boundary} and {\tt FillBoundary} functions
fill all ghost cells on periodic domain boundaries, as well as interior
ghost cells with values that can simply be copied from the valid region of a neighboring
grid at the same level of refinement.  For single-level problems that are periodic
in all directions, these functions fill all ghost cells.  We will discuss
non-periodic domain boundaries and fine grid ghost cells near coarse-fine interfaces
in Chapter \ref{Sec:Boundary Conditions}.

\subsection{Other \MultiFab\ Functions}
{\tt setVal} is a simple subroutine that sets the \MultiFab\ data to a particular value.
In Fortran90, use:
\begin{lstlisting}[backgroundcolor=\color{light-green}]
! set all variables to 0.0; ``all=.true.'' means set the ghost cells also
call setval(data,0.d0,all=.true.)
\end{lstlisting}
In C++, use:
\begin{lstlisting}[backgroundcolor=\color{light-blue}]
data->setVal(0.0);  // set all variables to 0.0, including ghost cells
\end{lstlisting}
{\tt copy} is a simple subroutine that copies data from one \MultiFab\ to another.
In Fortran90, use:
\begin{lstlisting}[backgroundcolor=\color{light-green}]
! copy components 1 and 2 from data_src into data_dest,
! including the ghost cells. calling sequence is
! (1) destination multifab, (2) first component of destination,
! (3) source multifab, ! (4) first component of source,
! (5) number of components, (6) ghost cells
call multifab_copy_c(data_dest,1,data_src,1,2,6)
\end{lstlisting}
In C++, use:
\begin{lstlisting}[backgroundcolor=\color{light-blue}]
// copy components 0 and 1 from data_src into data_dest,
// including the ghost cells. calling sequence is
// (1) destination multifab, (2) source multifab,
// (3) first component of destination, (4) first component of source,
// (5) number of components, (6) ghost cells
MultiFab::Copy(*data_dest,*data_src,0,0,2,6)
\end{lstlisting}
There are many other subroutines available for adding, subtracting, multiplying, etc.,
components of \MultiFab s, finding the min/max value, norms, number of cells, etc.
Refer to {\tt BoxLib/Src/F\_BaseLib/multifab\_f.f90} or
{\tt BoxLib/Src/C\_BaseLib/MultiFab.H} for a complete listing.

\section{Simple Example - Fortran90}
We now provide a complete tutorial code that uses some concepts discussed above.
The code also writes plotfiles that can be viewed, and can be run in parallel if you
are working on a machine with MPI and/or OpenMP support.  The Fortran90 version of this
example is contained in {\tt BoxLib/Tutorials/HeatEquation\_EX1\_F/}.\\

In this example, we advance the equation:
\begin{equation}
\frac{\partial\phi}{\partial t} = \nabla^2 \phi; \quad \phi(t=0) = 1 + e^{-100r^2},
\end{equation}
on a domain from [-1,1] in each spatial direction, where $r$ is the distance
to the point $(x,y,z) = (0.25,0.25,0.25)$.  Note that we are placing the
initial Gaussian profile slightly off-center.  This asymmetry will be important
in later sections when we examine the effects of non-periodic boundary conditions.
We will assume that $\Delta x = \Delta y = \Delta z$ and use a fixed
time step with $\Delta t = 0.9\Delta x^2$.  We begin with a simple
single-level, forward Euler discretization, periodic boundary conditions,
and no refinement (i.e., only one level).\\

The basic time-advancement strategy uses the following temporal discretization:
\begin{equation}
\frac{\phi_{ij}^{n+1} - \phi_{ij}^n}{\Delta t} = \left[\nabla\cdot(\nabla\phi)\right]_{ij}.
\end{equation}
In the explicit case, we first compute $\nabla\phi$, at faces using:
\begin{equation}
(\nabla\phi)_{i+\myhalf,j} = \frac{\phi_{i+1,j}^n-\phi_{ij}^n}{\Delta x}.
\end{equation}
We will refer to these face-centered gradients as ``fluxes''.
Next, we compute the update by taking the divergence of these fluxes,
\begin{equation}
\left[\nabla\cdot(\nabla\phi)\right]_{ij} = \frac{(\nabla\phi)_{i+\myhalf,j}-(\nabla\phi)_{i-\myhalf,j}}{\Delta x} + \frac{(\nabla\phi)_{i,j+\myhalf}-(\nabla\phi)_{i,j-\myhalf}}{\Delta y}.
\end{equation}
We use a flux divergence formulation because it will enable a more natural
extension to multiple levels of refinement, where we will be concerned with
conservation across levels.  Note that in this explicit case, since $\Delta x = \Delta y$,
the Laplacian reduces to the standard five point stencil in two dimensions
(seven point stencil in three dimensions).\\

Since the fluxes live on faces, we need face-centered \MultiFab s, i.e.,
\MultiFab s that are nodal in one spatial direction.  In {\tt advance.f90},
we build them as follows:
\begin{lstlisting}[backgroundcolor=\color{light-green}]
! an array of multifabs; one for each direction
type(multifab) :: flux(phi%dim)

! build the flux(:) multifabs
do i=1,dm
   ! flux(i) has 1 component, 0 ghost cells, and is nodal in direction i
   call multifab_build_edge(flux(i),phi%la,1,0,i)
end do
\end{lstlisting}

In the problem directory, you will see the following files:
\begin{itemize}
\item {\tt GNUmakefile}

This contains compiler settings and directories required by the make system to build the code.

  \begin{itemize}

    \item {\tt BOXLIB\_HOME}

    Change this to point to the \BoxLib\ home directory.  Alternatively, you can define {\tt BOXLIB\_HOME}
    as an environment variable on your system.

    \item {\tt NDEBUG} ('{\tt t}' or '{\tt <blank>}') for TRUE or FALSE

    ``not debug'' (we know, confusing).  If '{\tt t}', modifies compiler flags to build a
    more optimized version of the code.  The program will run faster, but have fewer
    runtime error checks.

    \item {\tt MPI} ('{\tt t}' or '{\tt <blank>}')

    Indicate whether you want your executable to be MPI-compatible.  MPI must be installed on your
    machine in order to use this, and you must modify some of the make scripts, as will be
    discussed later.

    \item {\tt OMP} ('{\tt t}' or '{\tt <blank>}')

    Turns on OpenMP compiler flags to compile in any OpenMP directives in the code.
    We will discuss OpenMP further in Section \ref{Sec:OpenMP}.

    \item {\tt PROF} ('{\tt t}' or '{\tt <blank>}')

    Turns on timer compilation flags.  Timers are useful for optimizing your code since they tell you
    what subroutines are taking the most time and require more optimization.  Note that you still have
    to write timers into your code.  Documentation for timers is coming soon!

    \item {\tt COMP} ('{\tt gfortran}, {\tt Intel}, $\ldots$)'

    The Fortran compiler.  Supported options include {\tt gfortran}, {\tt Intel},
    {\tt PathScale}, {\tt PGI}, and {\tt Cray}.  The {\tt gfortran} compiler seems to be
    bug-free on all systems we've run on, so stick with this unless you have good reason to
    change.  {\tt Intel} after version 9 seems flaky.  {\tt PathScale} (available
    at OLCF and NERSC) seems to work as long as you don't turn the optimization flags
    too high, and seems to run the fastest if you can actually get it to work.
    {\tt Cray} seems to give similar performance as {\tt PathScale} (perhaps
    because Cray bought out PathScale recently).  {\tt PGI} (available at OLCF and NERSC)
    seems to work fine, but is slower than the others.

    \item {\tt MKVERBOSE} ('{\tt t}' or '{\tt <blank>}')

    Verbosity of compile-time output.

  \end{itemize}

\item {\tt GPackage.mak}

List of local files needed to be included in the build.  The {\tt GNUmakefile} points to this.

\item {\tt main.f90}, {\tt init\_phi.f90}, {\tt advance.f90}, {\tt write\_plotfile.f90}

Source code that is not within the {\tt BoxLib/Src/} tree.  Note that
if a file that exists in the
{\tt BoxLib/Src/} tree also exists in the local directory, the local copy takes precedence
as long as the {\tt GNUmakefile} lists your local directory as a {\tt VPATH\_LOCATIONS}
before the \BoxLib\ source code directory, {\tt BoxLib/Src/F\_BaseLib}.

\item {\tt inputs\_2d}, {\tt inputs\_3d}

Input files to customize the simulation parameters.

\end{itemize}

To build the code, edit the {\tt GNUmakefile} and simply type ``{\tt make}''.  An
executable will appear that has some indication (but not complete)
about what setting you used in the {\tt GNUmakefile}.  To run the code on a single processor,
simply type, for example (terminal commands and non-source code files are shaded in red),
\begin{lstlisting}[backgroundcolor=\color{light-red}]
./main.Linux.gfortran.exe inputs_2d
\end{lstlisting}
The program will complete and there will be a series of plotfiles, e.g., {\tt plt00000},
{\tt plt00100}, etc., in the run directory.  You can visualize the data
and make animations using {\tt VisIt} (available at {\tt https://wci.llnl.gov/codes/visit/});
refer to Section \ref{Sec:VisIt}.

\section{Simple Example - C++}
The C++ version of this example is contained in {\tt BoxLib/Tutorials/HeatEquation\_EX1\_C/}.

\begin{itemize}
\item {\tt GNUmakefile}

This contains compiler settings and directories required by the make system to build the code.

  \begin{itemize}

    \item {\tt BOXLIB\_HOME}

    Change this to point to the \BoxLib\ home directory.  Alternatively, you can define {\tt BOXLIB\_HOME}
    as an environment variables on your system.

    \item {\tt DEBUG} ('{\tt TRUE}' or '{\tt FALSE}')

    Debug mode.  If 'FALSE', modifies compiler flags to build a more optimized version of the code.
    The program will run faster, but have fewer runtime error checks.

    \item {\tt USE\_MPI} ('{\tt TRUE}' or '{\tt FALSE}')

    Indicate whether you want your executable to be MPI-compatible.  MPI must be installed on your
    machine in order to use this, and you must modify some of the make scripts, as will be
    discussed later.

    \item {\tt USE\_OMP} ('{\tt TRUE}' or '{\tt FALSE}')

    Turns on OpenMP compiler flags to compile in any OpenMP directives in the code.
    We will discuss OpenMP further in Chapter \ref{Sec:OpenMP}.

    \item {\tt PROFILE} ('{\tt TRUE}' or '{\tt FALSE}')

    Turns on timer compilation flags.  Timers are useful for optimizing your code since they tell you
    what subroutines are taking the most time and require more optimization.  Note that you still have
    to write timers into your code.  Documentation for timers is coming soon!

    \item {\tt COMP} ('{\tt g++}, {\tt Intel}, $\ldots$)'

    The C++ compiler.  Supported options include {\tt g++}, {\tt Intel}, {\tt PathScale},
    {\tt PGI}, and {\tt Cray}.  See compiler notes above.

    \item {\tt FCOMP} ('{\tt gfortran}, {\tt Intel}, $\ldots$)'

    The Fortran compiler.  See compiler notes above.

    \item {\tt DIM} ('1', '2', or '3')

    Dimensionality of the problem.  Unlike Fortran90, you need to set this in the C++ version.

    \item {\tt PRECISION} ('{\tt DOUBLE}' or '{\tt FLOAT}')

    Precision of real numbers.  You can use {\tt FLOAT} for single-precision real numbers to save memory.

    \item {\tt EBASE} ('{\tt main}', $\ldots$)

    The executable string will begin with this.

  \end{itemize}

\item {\tt Make.package}

List of local files needed to be included in the build.  The {\tt GNUmakefile} points to this.

\item {\tt main.f90}, {\tt writePlotFile.cpp}, {\tt writePlotFile.H}, {\tt init\_phi\_2d.f90},
      {\tt init\_phi\_3d.f90}, {\tt advance\_2d.f90}, {\tt advance\_3d.f90}

Source code that is not within the {\tt BoxLib/Src/} tree.  Note that if a file that
exists in the
{\tt BoxLib/Src/} tree also exists in the local directory, the local copy takes precedence
as long as the {\tt GNUmakefile} lists your local directory in the {\tt include} line before
before the \BoxLib\ source code directories.

\item {\tt inputs\_2d}, {\tt inputs\_3d}

Input files to customize the simulation parameters.

\end{itemize}

To build the code, simply type ``{\tt make}''.  An exectubale will appear that has some indication (but not complete)
about what setting you used in the {\tt GNUmakefile}.  To run the code on one processor, simply type, for example,
\begin{lstlisting}[backgroundcolor=\color{light-red}]
./main2d.Linux.g++.gfortran.ex inputs_2d
\end{lstlisting}
The program will complete and there will be a series of plotfiles, e.g., {\tt plt00000},
{\tt plt00100}, etc., in the run directory.  You can visualize the data
and make animations using {\tt VisIt} (available at {\tt https://wci.llnl.gov/codes/visit/});
refer to Section \ref{Sec:VisIt}.

\section{The \BoxLib\ Linear Solvers}

\label{ch:mg}

\subsection{\BoxLib\ Multigrid Philosophy}

Here we describe some of the general ideas behind the \BoxLib\
multigrid (MG).

We solve MG on an AMR hierarchy, which means in places we will encounter
C-F interfaces.  The \BoxLib\ MG will modify the stencil for the Laplacian
at C-F interfaces to ensure the correct solution to the Poisson equation
across these interfaces.  There is no correction step needed in this
case (this differs from Ricker, ApJs (2008).

The MG solver always works with jumps of $2\times$ between levels.  In
some codes, we can have jumps of $4\times$ between
levels.  \BoxLib\ uses a {\tt mini\_cycle} in these cases, effectively
inserting a multigrid level between the two AMR levels and doing a mini
V-cycle.

The MG solvers are located in {\tt BoxLib/Src/LinearSolvers/F\_MG/}.
There are two MG solvers, for cell-centered and nodal data.
Generally, the routines specific to the cell-centered solver will have
{\tt cc} in their name, and those specific to the nodal solver will have
{\tt nd} in their name.

Support for $\Delta x \ne \Delta y \ne \Delta z$

\subsection{Data Structures}

\subsubsection{\mgtower}

The \mgtower\ is a special Fortran derived type that carries all the
information required for a \BoxLib\ multigrid solve.

The following parameters are specified when building the \mgtower:
\begin{itemize}
\item {\tt smoother}: the type of smoother to use.  Choices are listed
  in {\tt mg\_tower.f90}.  Common options are {\tt
    MG\_SMOOTHER\_GS\_RB} for red-black Gauss-Seidel and {\tt
    MG\_SMOOTHER\_JACOBI} for Jacobi.

\item {\tt nu1}: The number of smoothings at each level on the way down
  the V-cycle.

\item {\tt nu2}: The number of smoothings at each level on the way up
  the V-cycle.

\item {\tt nub}: The number of smoothing before and after the bottom solver.

\item {\tt gamma}: \MarginPar{what does this mean?}

\item {\tt cycle\_type}: The type of multigrid to do, V-cycles ({\tt
  MG\_VCycle}), W-cycles ({\tt MG\_WCycle}), full multigrid ({\tt
  MG\_FCycle}).

\item {\tt omega}:  \MarginPar{description?}

\item {\tt bottom\_solver}: the type of bottom solver to use.  See the next
 section.

\item {\tt bottom\_max\_iter}: the maximum number of iterations for the
  bottom solver

\item {\tt bottom\_solver\_eps}: the tolerance used by the bottom
  solver.

\item {\tt max\_iter}: the maximum number of multigrid cycles.

\item {\tt max\_bottom\_nlevel}: additional coarsening if you use
  {\tt bottom\_solver} type 4 (see below)

\item {\tt min\_width}: minimum size of grid at coarsest multigrid level

\item {\tt rel\_solver\_eps}: the relative tolerance of the solver.

\item {\tt abs\_solver\_eps}: the absolute tolerance of the solver.

\item {\tt verbose}: the verbosity of the multigrid solver.  Higher
  numbers give more verbosity.

\item {\tt cg\_verbose}: the verbosity of the bottom solver. Higher
  numbers give more verbosity.

\end{itemize}

In addition to these parameters, the \mgtower\ carries a number of
\multifab s that carry the solution and stencil for the multigrid
solve.

\begin{itemize}

\item {\tt ss}: The stencil itself---for each zone, this gives the
  coefficients of the terms in the Laplacian, with the convection that
  the `0' term is located in the current cell and the other terms are
  the $\pm 1$ off the current cell in each direction.

\item {\tt cc}: scratch space (?)

\item {\tt ff}: The source (righthand side) of the elliptic equation
  we are solving.

\item {\tt dd}: The residual/defect, $f - L\phi$

\item {\tt uu}: The solution variable ($\phi$ when on the finest level)

\item {\tt mm}: For cell-centered, {\tt mm} takes a direction and
   tells you whether we are at a Dirichlet or Neumann boundary, or if
   we are skewed in that direction.

   For nodal, {\tt mm} simply tells us whether a point is Dirichlet
   or Neumann.  There is no skew in nodal.

\end{itemize}

\subsubsection{\bndryreg}

There are two types of \bndryreg\ objects, each of which is a set of
{\tt dim} multifabs.  The first, which is built with
the {\tt bndry\_reg\_build} call, is defined on all cells immediately outside
of each grid.  Each multifab within the \bndryreg\ contains all the fabs on both
the low and high faces for a given direction.  In the context of
multigrid, we fill the \bndryreg\ with coarse data before interpolating the data
to the correct locations to be used in the multigrid stencil.  The second type
of \bndryreg\ object is defined on the coarse cells immediately outside each
fine grid, and is defined with the  {\tt bndry\_reg\_rr\_build} call, is defined on all cells immediately outside
.  For this latter type, the option is available to include only cells
not covered by a different fine grid, but this is left as an option because it
requires additional calculations of box intersections.

In multigrid, we use the latter type of \bndryreg\ to calculate the
residual at coarse cells adjacent to fine grids that is used as the right hand side
in the relaxation step at the coarse level on the way down the V-cycle (or other).
The first type of \bndryreg is used to hold boundary conditions for the fine grids
that are interpolated from the coarse grid solution; this is filled on the way back up the V-cycle.

To compute the residual in a coarse cell adjacent to a fine grid, we first
compute the pure coarse residual, then subtract the contribution from the coarse
face underlying the coarse-fine interface, and add the contribution of the fine
faces at the coarse-fine interface.  The \bndryreg\ holds the cell-centered
contribution from the difference of these edge fluxes and is added to
the coarse residual multifab.

\subsection{Stencils}

There are several different stencil types that we can use for
the discretization.  For cell-centered, these are:
\begin{itemize}
\item {\tt CC\_CROSS\_STENCIL}: this is the standard cross-stencil---5 points
in 2-d and 7 points in 3-d.  For cell-centered MG, this is the default, and
is usually the best choice.

\item {\tt HO\_CROSS\_STENCIL}: this is a cross-stencil that uses 9 points
in 2-d and 11-points in 3-d.  For instance, it will use $(i,j)$;
$(i\pm1,j)$; $(i\pm2,j)$; $(i,j\pm1)$; $(i,j\pm2)$.  This is
higher-order accurate than {\tt CC\_CROSS\_STENCIL}.

\item {\tt HO\_DENSE\_STENCIL}: this is a dense-stencil---it uses all the
points (including corners) in $(i\pm1,j\pm1)$, resulting in a 9-point stencil
in 2-d and 27-point stencil in 3-d.
\end{itemize}

For the nodal solver, the choices are:
\begin{itemize}
\item {\tt ND\_CROSS\_STENCIL}: this is the standard cross-stencil.

\item {\tt ND\_DENSE\_STENCIL}: this is a dense stencil, using
  all the points in $(i\pm1,j\pm1)$.  The
  derivation of this stencil is based on finite-element ideas, defining
  basis functions on the nodes.  This is developed in 2-d in
  Almgren, Bell, and Szymczack (1996).

\item {\tt ND\_VATER\_STENCIL}: this is an alternate dense stencil derived
  using a similar finite-element idea as above, but a different control
  volume.
\end{itemize}
For the cell-centered solve, the coefficients for the stencil are computed
once, at the beginning of the solve.  For the nodal solver, the coefficients
are hard-coded into the smoothers.

\subsection{Smoothers}

The following smoothers are available (but not necessarily for both the
cell-centered and nodal solvers):
\begin{itemize}
\item {\tt MG\_SMOOTHER\_GS\_RB}: a red-black Gauss-Seidel smoother

\item {\tt MG\_SMOOTHER\_JACOBI}: a Jacobi smoother (not implemented for
the dense nodal stencil)

\item {\tt MG\_SMOOTHER\_MINION\_CROSS}

\item {\tt MG\_SMOOTHER\_MINION\_FULL}

\item {\tt MG\_SMOOTHER\_EFF\_RB}
\end{itemize}

\subsection{Cycling}

The default cycling is a V-cycle, but W-cycles and full multigrid are
supported as well.

\subsection{Bottom Solvers}

The multigrid cycling coarsens the grids as part of the solve.  When
the coarsest grid is reached, the individual boxes that comprise that
level are coarsened as much as then can, down to $2^3$ zones.  Depending
on the distribution of sizes of the grids, it may not be possible for
everything to reach this minimum size.  At this point, the bottom
solver is invoked.  Most of these will solve the linear system
on this collection of grids directly.  There is one special bottom
solver that will define a new box encompassing all of the coarsened
grids and then put the data on fewer boxes and processors and further
coarsen the problem, again until we get as close to $2^3$ as possible.
At that point, one of the other bottom solvers will be called upon
to solve the problem.

There are several bottom solvers available in \BoxLib.
The allowed values are:
\begin{itemize}

\item {\tt mg\_bottom\_solver} / {\tt hg\_bottom\_solver = 0}: smoothing only.

\item {\tt mg\_bottom\_solver} / {\tt hg\_bottom\_solver = 1}: biconjugate
  gradient stabilized---this is the default.

\item {\tt mg\_bottom\_solver} / {\tt hg\_bottom\_solver = 2}: conjugate
  gradient method

\item {\tt mg\_bottom\_solver} / {\tt hg\_bottom\_solver = 4}: a special
  bottom solver that extends the range of the multigrid coarsening
  by aggregrating coarse grids on the original mesh together and
  further coarsening.

\end{itemize}

You should use the special bottom solver (4) whenever possible, even
if it means changing your gridding strategy (as discussed below) to
make it more efficient.

\MarginPar{any simple discussion on why we might choose one of the
  other bottom solvers?}

\subsubsection{Special Bottom Solver}

The special solver takes the data from the coarsest level of the
original multigrid V-cycle and copies it onto a new grid structure with
the same number of total cells in each direction, but with a fewer
number of larger grids.  A new V-cycle begins from this point, so we
are essentially coarsening this ``new'' problem.  Now, the coarsest
level of the multigrid V-cycle in the ``new'' problem has fewer cells
and fewer grids as compared to the original coarsest level.

To enable this solver, set {\tt hg\_bottom\_solver = 4} (for the nodal
projections) and/or {\tt mg\_bottom\_solver = 4} (for the
cell-centered projections) in your inputs file.

To understand how this bottom solver works, the first thing you need
to know is what the grid structure of the coarsest level of your
multigrid V-cycle looks like.  Next, figure out the size of the box you
would need if you wanted it to fit all the data on the coarsest level.
Finally, figure out what the largest integer $n$ is so that you can evenly
divide the length of this box by $2^n$ in every coordinate direction.
If $n < 2$, the program will abort since the grid structure is not
suitable for this bottom solver.

The code will set up a ``new'' problem, using the data at the
coarsest level of the original problem as the initial data.  The grid
structure for this new problem has the same number of cells as the
coarsest level of the original problem, but the data is copied onto a
grid structure where each grid has $2^n$ cells on each side.  The new
V-cycle continues down to the new coarsest level, in which each grid
has 2 cells on each side.  If you wish to impose a limit on the
maximum value that $n$ can have, you can do so by setting {\tt
max\_mg\_bottom\_nlevs} equal to that value.

Some grid examples help make this clear:
\begin{itemize}
\item {\bf Example 1:} A 3D problem with $384^3$ cells divided into $32^3$
grids, i.e., there is a $12\times 12\times 12$ block of $32^3$ grids.
The coarsest level of the multigrid V-cycle contains $12\times
12\times 12$ grids that have $2^3$ cells, so the entire problem domain
has $24^3$ cells.  We see that $n=3$, and create a new problem domain
with a $3\times 3\times 3$ block of $8^3$ grids.  The coarsest level
of the multigrid V-cycle for the ``new'' problem will be a $3\times
3\times 3$ block of $2^3$ grids.

\item {\bf Example 2:} A 2D problem with $96\times 384$ cells divided into
$48^2$ grids, i.e., there is a $2\times 8$ block of $48^2$ grids.  The
coarsest level of the multigrid V-cycle contains $2\times 8$ grids
that have $3^2$ cells, so the entire problem domain has $6\times 24$
cells.  We see that $n=0$, so the program aborts since this grid
structure is not appropriate for the fancy bottom solver.
\end{itemize}

\subsection{Flowchart}

\subsubsection{Cell-Centered MG}

The flowchart below shows the structure of a cell-centered multigrid
solve using pure V-cycles.

\begin{itemize}

\item {\tt stencil\_fill\_cc\_all\_mglevels} / {\tt stencil\_fill\_cc}:
  Compute all of the stencil coefficients
  for the Laplacian operator at all cells.  At the C-F interfaces, the
  stencil coefficients are modified to know this.

\item {\tt ml\_cc}: The main driver for the cell-centered multigrid.
   Among other things, this computes the norm that will be used
   for convergence testing.

\item {\tt mg\_tower\_v\_cycle} (recursive):
\begin{itemize}

  \item {\em recursively descend V-cycle}

  \begin{itemize}
  \item {\tt cc\_mg\_tower\_smoother}: Smooth the
    problem at the current MG level using the desired smoother.

  \item {\tt compute\_defect}: Construct $f - L\phi$.

  \item {\tt mg\_tower\_restriction}:  Restrict
    the defect to the coarser level by conservative averaging.
  \end{itemize}

  \item {\tt mg\_tower\_bottom\_solve}:  Solve the coarsened problem
    using the chosen bottom solver.

  \item {\em ascend V-cycle}

  \begin{itemize}
  \item {\tt mg\_tower\_prolongation}: Take the solution at level $n-1$ and use it to
     correct the solution at level $n$ by representing the data on the finer grid.  This uses
     linear reconstruction for jumps by $2\times$ and piecewise-constant otherwise.

  \item {\tt cc\_mg\_tower\_smoother}:
  \end{itemize}

\end{itemize}
\item {\tt compute\_defect}: This is called multiple times, checking for
   convergence at each level.

\end{itemize}

\subsubsection{Nodal MG}

The flowchart below shows the structure of a cell-centered multigrid
solve using pure V-cycles.

\begin{itemize}

\item {\tt stencil\_fill\_cc\_all\_mglevels} / {\tt stencil\_fill\_cc}:
  For the nodal solver, this applies the weights to the
  coefficients. \MarginPar{nodal does things differently than cc---need more here}

\item {\tt ml\_nd}: The main driver for the nodal multigrid.

\item {\tt mg\_tower\_v\_cycle} (recursive):
\begin{itemize}

  \item {\em recursively descend V-cycle}
  \begin{itemize}
  \item {\tt nodal\_smoother\_?d}: Smooth the
    problem at the current MG level using the desired smoother.

  \item {\tt compute\_defect}: Construct $f - L\phi$.

  \item {\tt mg\_tower\_restriction}:  Restrict
    the defect to the coarser level by simply taking the fine value that
    lies at the same place as the coarse data.
  \end{itemize}

  \item {\tt mg\_tower\_bottom\_solve}:  Solve the coarsened problem
    using the chosen bottom solver.

  \item {\em ascend V-cycle}

  \begin{itemize}
  \item {\tt mg\_tower\_prolongation}: For nodal data, the fine grid
     will have some points at exactly the same place as the coarse data---these are
     simply copied to the fine grid.  The remain data is interpolated.

  \item {\tt modal\_smoother\_?d}:
  \end{itemize}

\end{itemize}
\item {\tt compute\_defect}: This is called multiple times, checking for
   convergence at each level.

\end{itemize}

\section{Visualization Using {\tt VisIt}}\label{Sec:VisIt}
First, download and install {\tt VisIt} from {\tt https://wci.llnl.gov/codes/visit/}.
To open a single plotfile, run {\tt VisIt}, then select ``File'' $\rightarrow$ ``Open file ...'',
then select the {\tt Header} file associated the the plotfile of interest (e.g., {\tt plt00000/Header}).
Assuming you ran the simulation in 2D, here are instructions for making a simple plot:
\begin{itemize}
\item To view the data, select ``Add'' $\rightarrow$ ``Pseudocolor'' $\rightarrow$ ``phi'', and then select
``Draw''.
\item To view the grid structure (not particularly interesting yet, but when we add AMR it will be), select
`` $\rightarrow$ ``subset'' $\rightarrow$ ``levels''.  Then double-click the text ``Subset - levels'',
enable the ``Wireframe'' option, select ``Apply'', select ``Dismiss'', and then select ``Draw''.
\item To save the image, select ``File'' $\rightarrow$ ``Set save options'', then customize the image format
to your liking, then click ``Save''.
\end{itemize}
Your image should look similar to the left side of Figure \ref{Fig:VisIt}.\\
\begin{figure}[tb]
\centering
\includegraphics[width=3.1in]{./GettingStarted/VisIt_2D}
\includegraphics[width=3.1in]{./GettingStarted/VisIt_3D}
\caption{(Left) 2D image generated with VisIt.  (Right) 3D image generated with VisIt.}
\label{Fig:VisIt}
\end{figure}

In 3D, you must apply the ``Operators'' $\rightarrow$ ``Slicing'' $\rightarrow$ ``ThreeSlice'', with the
``ThreeSlice operator attribute'' set to x=0.25, y=0.25, and z=0.25.  You can left-click and drag
over the image to rotate the image to generate something similar to right side of Figure \ref{Fig:VisIt}.\\

To make a movie, you must first create a text file named {\tt movie.visit} with a list of the {\tt Header}
files for the individual frames.  This can most easily be done using the command:
\begin{lstlisting}[backgroundcolor=\color{light-red}].
~/BoxLib/Tutorials/HeatEquation_EX1_F> ls -1 plt*/Header | tee movie.visit
plt00000/Header
plt01000/Header
plt02000/Header
plt03000/Header
plt04000/Header
plt05000/Header
plt06000/Header
plt07000/Header
plt08000/Header
plt09000/Header
plt10000/Header
\end{lstlisting}
The next step is to run {\tt VisIt}, select ``File'' $\rightarrow$ ``Open file ...'',
then select {\tt movie.visit}.  Create an image to your liking and press the ``play'' button
on the VCR-like control panel to preview all the frames.  To save the movie, choose
``File'' $\rightarrow$ ``Save movie ...'', and follow the on-screen instructions.

\section{Running in Parallel with MPI}
We will now demonstrate how to run the example in {\tt BoxLib/Tutorials/HeatEquation\_EX1\_F/}
using MPI parallelism.  To run in parallel using C++ \BoxLib\ is analogous.
On your local machine, if you have MPI installed, you must first configure
{\tt BoxLib/Tools/F\_mk/GMakeMPI.mak} and {\tt BoxLib/Tools/C\_mk/Make.mpi}
to your MPI installation.  Then, you
can simply build the executable as describe before, but with {\tt MPI=t}
in the {\tt GNUmakefile}.  Alternatively, you can override the settings in
{\tt GNUmakefile} at the command line using, e.g., ``{\tt make MPI=t}''.
An executable named {\tt main.Linux.gfortran.mpi.exe} will be built.
Then you can run the program in parallel using, e.g.,
\begin{lstlisting}[backgroundcolor=\color{light-red}].
mpiexec -n 4 main.Linux.gfortran.mpi.ex inputs_2d
\end{lstlisting}

To run in parallel on the hopper machine at NERSC, first copy the \BoxLib\ source code
into your home directory on hopper and go to the {\tt HeatEquation\_EX1\_F/} directory.
The default programming environment uses the {\tt PGI} compilers, so we will switch to the
{\tt gnu} programming environment to make {\tt g++} and {\tt gfortran} available
using the command:
\begin{lstlisting}[backgroundcolor=\color{light-red}].
module swap PrgEnv-pgi PrgEnv-gnu
\end{lstlisting}
Next, in {\tt GNUmakefile}, set {\tt MPI=t}, and then type ``{\tt make}''
(or alternatively, type ``{\tt make MPI=t}'').
An executable named {\tt main2d.Linux.g++.gfortran.mpi.exe} will be built.
You cannot submit jobs in your home directory, so change to a scratch space
(``{\tt cd \$SCRATCH}'' will typically do), and copy the executable and
{\tt inputs\_2d} into this directory.  Then you need to create a job script,
e.g., "{\tt hopper.run}", that has contents (for tcsh):
\lstinputlisting[backgroundcolor=\color{light-red}]{./GettingStarted/hopper.run}
Note that ``mppwidth'' and ``-n'' both indicate the number of cores you are requesting.
To run, simply type ``{\tt qsub hopper.run}''.  You can monitor the status of your job
using ``{\tt qstat -u <username>}'' and view your position in the queue
using ``{\tt showq}''.

\section{Running in Parallel with MPI/OpenMP (3D ONLY)}\label{Sec:OpenMP}
Both the C++ and Fortran versions of this tutorial also support hybrid MPI/OpenMP parallelism for {\bf three-dimensional problems only}.
You may add OMP parallelism to two-dimensional work loops, but be advised that subroutines within the \BoxLib~ infrastructure
are not threaded for two-dimensional problems.
To ``thread'' the code, we have simply added OpenMP directives (using the {\tt !\$omp parallel do}
construct) to any i/j/k loops we were interested in threading.  For example, in {\tt init\_phi.f90}:
\begin{lstlisting}[backgroundcolor=\color{light-green}]
!$omp parallel do private(i,j,k,x,y,z,r2)
do k=lo(3),hi(3)
   z = prob_lo(3) + (dble(k)+0.5d0) * dx
   do j=lo(2),hi(2)
      y = prob_lo(2) + (dble(j)+0.5d0) * dx
      do i=lo(1),hi(1)
         x = prob_lo(1) + (dble(i)+0.5d0) * dx

         r2 = ((x-0.25d0)**2 + (y-0.25d0)**2 + (z-0.25d0)**2) / 0.01d0
         phi(i,j,k) = 1.d0 + exp(-r2)

      end do
   end do
end do
!$omp end parallel do
\end{lstlisting}
This User's Guide is not a manual on OpenMP, so simply note that this particular
construct tells each thread to work on different values of {\tt k}, with each
thread getting its own local copy of {\tt i}, {\tt j}, {\tt x}, {\tt y}, {\tt z}, and {\tt r2}.\\

Finally, to tell the compiler that we would like to run with OpenMP, we make sure to
set {\tt OMP=t} (in Fortran) or {\tt USE\_OMP=TRUE} (in C++) in the {\tt GNUmakefile}.
Otherwise, all OpenMP directive are
simply ignored.  Note that at runtime you must have set the
{\tt OMP\_NUM\_THREADS} environment variable properly in order for threads to work.
Also, note that you can enable/disable MPI independently from the OMP flag.  Finally,
here is a sample hopper script (tcsh) for a hybrid MPI/OpenMP job:
\lstinputlisting[backgroundcolor=\color{light-red}]{./F_AdvancedTopics/hopper_omp.run}
\begin{itemize}
\item ``mppwidth'': how many total cores requested
\item ``-n'': total number of MPI tasks
\item ``-N'': number of MPI tasks per hopper node
\item ``-S'': number of MPI tasks per NUMA node
\item ``-ss'': demands strict memory containment per NUMA node
\item ``-d'': number of OpenMP threads per MPI task
\end{itemize}


\documentclass[12pt]{amsart}
\usepackage{geometry} % see geometry.pdf on how to lay out the page. There's lots.
\usepackage{bsymb}
\usepackage{unitb}
\usepackage{calculational}
\usepackage{ulem}
\usepackage{hyperref}
\normalem
\geometry{a4paper} % or letter or a5paper or ... etc
% \geometry{landscape} % rotated page geometry

% See the ``Article customise'' template for some common customisations

\title{}
\author{}
\date{} % delete this line to display the current date

%%% BEGIN DOCUMENT
\setcounter{tocdepth}{4}
\begin{document}

\maketitle
\tableofcontents

\newcommand{\G}{\text{G}}
\renewcommand{\H}{\text{H}}

\section{Initial model}

\begin{context}{ctx0}

\newset{\G}

\with{functions} \with{predcalc}
\dummy{ x,s,t : \pred[\G] }
\operator{;}{ seq : Pair [\pred[\G], \pred[\G]] \pfun \pred[\G] }
% \operator{\pneg}{ pneg : \G \pfun \G }
% \operator{\por}{ por : Pair [\G, \G] \pfun \G }
% \operator{\pand}{ pand : Pair [\G, \G] \pfun \G }
% \operator{\pimplies}{ pimplies : Pair [\G, \G] \pfun \G }
% \operator{\pequiv}{ pequiv : Pair [\G, \G] \pfun \G }
% \precedence{[[\pequiv],[\pimplies],[\por,\pand],[\pneg],[;]]}
\precedence{[[;],[\pneg],[\por,\pand],[\pimplies],[\pequiv]]}
%
\constant{ one, F, E : \pred [\G] }

\axiom{axm0}{ \qforall{x}{}{ \ew{ x ; one \1\pimplies x } } }
\theorem{CC:10i}{ \qforall{s}{}{ \ew{ s ; \pfalse \pequiv s \pand E } } }

\end{context}

\end{document}


\documentclass{article}

% math
\usepackage{amsmath}
\usepackage{amssymb}
\newcommand{\R}{\mathbb{R}}
\newcommand{\Q}{\mathbb{Q}}
\newcommand{\N}{\mathbb{N}}
\newcommand{\Z}{\mathbb{Z}}
\newcommand{\C}{\mathbb{C}}

% for double images
\usepackage{subcaption}

% Used for img import
\usepackage{import}
\usepackage{pdfpages}
\usepackage{transparent}
\usepackage{xcolor}

\newcommand{\incfig}[2][1]{%
    \def\svgwidth{#1\columnwidth}
    \import{./figures/}{#2.pdf_tex}
}

\pdfsuppresswarningpagegroup=1

\title{Calculus and Optimization for Machine Learning}
\date{2021-07-27}
\author{Seara}
\pagenumbering{gobble}

\begin{document}
\maketitle
\newpage
\pagenumbering{arabic}

\section{Week 1}

\subsection{Numerical sets and mappings}

\paragraph{Set.}
A {\it set} (is commonly called non-definable and fundamental) is an entity, a {\it collection} of some objects:
\begin{itemize}
    \item An object either belongs to the set or do not
    \item One object could be included in the set only one time
    \item There is {\it no order} (even if it is trivial) on objects of the set
\end{itemize}
Sets are usually denoted by capital letters: $X, Y, A, B, R, ...$ The fact of belonging to the set is denoted as: $a \in A$.
\\
Another essential concept concerning sets is subsets. Basically, the subset is any set of (not necessarily all) elements of the given set. We consider primarily numerical sets:
\begin{itemize}
    \item Natural numbers $\N= \{1, 2, 3, 4, ...\}$
    \item Integer numbers $\Z = \{0, 1, -1, 2, -2, ...\}$
    \item Rational numbers $\Q = \{\frac{m}{n} | m \in Z, n \in N\}$
    \item Real numbers $\R=\{a_0, a_1, ...\}$
\end{itemize}

\paragraph{Mapping.}
Assume that we have two sets $X$ and $Y$. A mapping between them is, generally speaking, ordered realtion between elements of $X$ and $Y$. Consider two logic notations: quantifiers "for all" $\forall$ and "exists" $\exists$.

\paragraph{Axiomatic definition of real numbers.}
\begin{itemize}
    \item $x+y=y+x, y \cdot x = x \cdot y$ - the commutative rule
    \item $(x+y)+z = x + (y+z), (x \cdot y) \cdot z = x \cdot (x \cdot z)$ - associativity
    \item $(x+y) \cdot z = x \cdot z + y \cdot z$ - distributivity
    \item Existence of two neutral elements 1 and 0: $a \cdot 1 = a, a+0=a$
    \item Existence of the inverse elements (exept 0): $a + (-a)=0,a \cdot a^{-1}=1$
    \item Non-triviality $0 \neq 1$
    \item For any two real numbers one is able to say $a>b,a<b$ or $a=b$
    \item This order is transitive: if $a<b,b<c$, then $a<c$
    \item Completeness: Assume the we consider some section of our set into two non-intersecting sets: $ \R = A \cup B, A \cap B = \O$, such as any element $a \in A$ is smaller than any element $b \in B$. Then there is a pivot real number $c \in \R$ that $a \leq c \leq b$ for any $a$ and $b$ elements from the corresponding sets.
\end{itemize}

\paragraph{Functions.}
Consider two sets $X$ and $Y$ and mapping $f: X \mapsto Y$. This mapping (relation) called {\it functional} (or a {\it function}) if and only it associates each element of the
$X$ set to exactly on element of the $Y$ set. $X$ set is called domain (set of arguments) and $Y$ set is called codomain (set of values).
\\
Function graph is a cetrain curve on the plane: the set of all points $(x,f(x))$ for all $x$ belonging to the function's domain.
\\
Domain - $D(f)$
\\
Codomain - $E(f)$
\\
Support - $supp(f) = \{x \in X : f(x) \neq 0\}$
\\
Composite function or composition - $g(f(x))$
\\
Vertical shift - $y = f(x) + c$
\\
Horizontal shift - $y=f(x+C)$
\\
Vertical contraction - $y=C \cdot f(x)$
\\
Horizontal contraction - $y=f(C \cdot x)$
\\
Absolute value - $y=|f(x)|$

\subsection{Limits, sequences}

\paragraph{Limit.}
Limit of the sequence - the real number that resembles our sequence the most as the element's number infinitely grows(approaches infinity). The notation:
\[
\lim_{n \to \infty} a_n = C
\]
$\forall \varepsilon > 0: \exists N \in \N \; that \; \forall n \in \N, n \geq N \Rightarrow |a_n -C| < \varepsilon$
\\
If a sequence has the limit equal to 0, it is called {\it infinitesimal}.
Привет
\end{document}

Courses/Trash/MathForDS[abandoned]/Calculus And Optimization For Machine Learning/calculus.tex


\section{Background estimation}
\label{sec:hmhzz_bkg}

In this analysis, 97\% of total expected background events are from irreducible ZZ backgrounds, which includes about 86\% quark-antiquark annihilation (\qqZZ), 10\% of gluon-induced production (\ggZZ) and around 1\% of EW vector boson scattering (\qqZZ EW) contribution.
For \qqZZ EW, although it has small contribution in total background events after analysis selection, it's important for VBF category with about 16\% contribution.

In addition to irreducible backgrounds, events from \Zjet and \ttbar processes, represent as reducible backgrounds, contribute at a few percent level and can be measured using data driven method that will later be described briefly.
Additional background called `Others', including ttV and triple-V (VVV) processes, has tiny contribution and is estimated from MC simulation directly.

\subsection{Irreducible backgrounds}
The Irreducible backgrounds have events with four prompt leptons.
The normalization of two dominant backgrounds \qqZZ and \ggZZ are taken from data by statistical fit, and the normalization of small \qqZZ EW background is taken directly from MC simulation.

The \mfl shapes of all three background components are taken from MC samples and then parameterized by an empirical function for each of them in each category respectively.
Details of background modellings are illustrated as below:

The empirical function used for background parameterization is:
\begin{equation}
    f(\mfl) = C_0 H(m_{0} - \mfl) f_{1}(\mfl) + H(\mfl - m_{0}) f_{2}(\mfl),
    \label{eq:bkg_model}
\end{equation}
where,
\begin{align*}
    f_1(x) &= \left( \frac{x - a_4}{a_3} \right)^{a_1 - 1} \left( 1 + \frac{x - a_4}{a_3} \right)^{-a_1 - a_2}, \\
    f_2(x) &= \exp \left[ b_0 \left( \frac{x - b_4}{b_3} \right)^{b_1 - 1} \left( 1 + \frac{x - b_4}{b_3} \right)^{-b_1 - b_2} \right], \\
    C_0    &= \frac{f_{2} (m_0)} {f_{1} (m_0)}.
    \label{eq:bkg_model_full}
\end{align*}

The function consists of two parts, the first part $f_{1}$ describes the \mfl spectrum in low mass region where both $Z$ bosons decay on-shell, while the second one $f_{2}$ covers distribution at high mass tail.
The transition between the low- and high- mass parts is presented in function~\ref{eq:bkg_model} by the Heaviside step function $H(x)$ at the transition point $m_0$.
The $m_0$ is chosen to optimize the smoothness of the function, and practically $m_0 = 260~(350)~\gev$ is used for \qqZZ (\ggZZ and \qqZZ EW).
Besides, the continuity of two functions at $m_0$ is ensured by the factor $C_0$ applied to $f_{1}$.
The coefficients $a_{i}$ in $f_{1}$ and $b_{i}$ in $f_{2}$ are shape parameters obtained by fitting to \mfl distribution from each MC simulated sample.

Figure~\ref{fig:qqZZ_m4l_shape_all_cut_based} to ~\ref{fig:qqZZEW_m4l_shape_all_cut_based} shows the fitting results of \qqZZ, \ggZZ, \qqZZ EW backgrounds in four cut-based categories (ggF-CBA-enriched-2$e$2$\mu$, ggF-CBA-enriched-4$e$, ggF-CBA-enriched-4$\mu$ and VBF-CBA-enriched).
Figure~\ref{fig:qqZZ_m4l_shape_all_DNN} to ~\ref{fig:qqZZEW_m4l_shape_all_DNN} shows the fitting results of those backgrounds in five MVA-based categories (ggF-MVA-high-2$e$2$\mu$, ggF-MVA-high-4$e$, ggF-MVA-high-4$\mu$, ggF-MVA-low and VBF-MVA-enriched).

\begin{figure}[htbp]
    \centering
    \includegraphics[width=0.32\textwidth]{figures/HMHZZ/background/cut_based/bkg_shape_qqZZ_ggF_4mu_190_to_2200_log.pdf}
    \includegraphics[width=0.32\textwidth]{figures/HMHZZ/background/cut_based/bkg_shape_qqZZ_ggF_4e_190_to_2200_log.pdf} \\
    \includegraphics[width=0.32\textwidth]{figures/HMHZZ/background/cut_based/bkg_shape_qqZZ_ggF_2mu2e_190_to_2200_log.pdf}
    \includegraphics[width=0.32\textwidth]{figures/HMHZZ/background/cut_based/bkg_shape_qqZZ_VBF_incl_190_to_2200_log.pdf}
    \caption{Distributions of the \mfl invariant mass fit projections of the \qqZZ background samples for the $4\mu$,
    $4e$ and $2\mu 2e$ final states in the ggF-CBA-enriched category, and the $4\ell$ inclusive VBF-CBA-enriched category.
    Cut-based categorization is used.}
    \label{fig:qqZZ_m4l_shape_all_cut_based}
\end{figure}

\begin{figure}[htbp]
    \centering
    \includegraphics[width=0.32\textwidth]{figures/HMHZZ/background/cut_based/bkg_shape_ggZZ_ggF_4mu_180_to_2200_log.pdf}
    \includegraphics[width=0.32\textwidth]{figures/HMHZZ/background/cut_based/bkg_shape_ggZZ_ggF_4e_185_to_2200_log.pdf} \\
    \includegraphics[width=0.32\textwidth]{figures/HMHZZ/background/cut_based/bkg_shape_ggZZ_ggF_2mu2e_185_to_2200_log.pdf}
    \includegraphics[width=0.32\textwidth]{figures/HMHZZ/background/cut_based/bkg_shape_ggZZ_VBF_incl_180_to_2200_log.pdf}
    \caption{Distributions of the \mfl invariant mass fit projections of the \ggZZ background samples for the $4\mu$,
    $4e$ and $2\mu 2e$ final states in the ggF-CBA-enriched category, and the $4\ell$ inclusive VBF-CBA-enriched category.
    Cut-based categorization is used.}
    \label{fig:ggZZ_m4l_shape_all_cut_based}
\end{figure}

\begin{figure}[htbp]
    \centering
    \includegraphics[width=0.32\textwidth]{figures/HMHZZ/background/cut_based/bkg_shape_qqZZEW_ggF_4mu_180_to_2200_log.pdf}
    \includegraphics[width=0.32\textwidth]{figures/HMHZZ/background/cut_based/bkg_shape_qqZZEW_ggF_4e_180_to_2200_log.pdf} \\
    \includegraphics[width=0.32\textwidth]{figures/HMHZZ/background/cut_based/bkg_shape_qqZZEW_ggF_2mu2e_180_to_2200_log.pdf}
    \includegraphics[width=0.32\textwidth]{figures/HMHZZ/background/cut_based/bkg_shape_qqZZEW_VBF_incl_180_to_2200_log.pdf}
    \caption{Distributions of the \mfl invariant mass fit projections of the \qqZZ (EW) background samples for the
    $4\mu$, $4e$ and $2\mu 2e$ final states in the ggF-CBA-enriched category, and the $4\ell$ inclusive VBF-CBA-enriched category.
    Cut-based categorization is used.}
    \label{fig:qqZZEW_m4l_shape_all_cut_based}
\end{figure}

%===========================================================================================================================
\begin{figure}[htbp]
    \centering
    \includegraphics[width=0.32\textwidth]{figures/HMHZZ/background/dnn/bkg_shape_qqZZ_ggF_4mu_190_to_2200_log.pdf}
    \includegraphics[width=0.32\textwidth]{figures/HMHZZ/background/dnn/bkg_shape_qqZZ_ggF_4e_190_to_2200_log.pdf}
    \includegraphics[width=0.32\textwidth]{figures/HMHZZ/background/dnn/bkg_shape_qqZZ_ggF_2mu2e_190_to_2200_log.pdf} \\
    \includegraphics[width=0.32\textwidth]{figures/HMHZZ/background/dnn/bkg_shape_qqZZ_VBF_incl_190_to_2200_log.pdf}
    \includegraphics[width=0.32\textwidth]{figures/HMHZZ/background/dnn/bkg_shape_qqZZ_rest_190_to_2200_log.pdf}
    \caption{Distributions of the \mfl invariant mass fit projections of the \qqZZ background samples for the $4\mu$,
    $4e$ and $2\mu 2e$ final states in the ggF-MVA-high category, the $4\ell$ inclusive ggF-MVA-low category and VBF-MVA-enriched category.
    DNN-based categorization is used.}
    \label{fig:qqZZ_m4l_shape_all_DNN}
\end{figure}

\begin{figure}[htbp]
    \centering
    \includegraphics[width=0.32\textwidth]{figures/HMHZZ/background/dnn/bkg_shape_ggZZ_ggF_4mu_180_to_2200_log.pdf}
    \includegraphics[width=0.32\textwidth]{figures/HMHZZ/background/dnn/bkg_shape_ggZZ_ggF_4e_185_to_2200_log.pdf}
    \includegraphics[width=0.32\textwidth]{figures/HMHZZ/background/dnn/bkg_shape_ggZZ_ggF_2mu2e_185_to_2200_log.pdf} \\
    \includegraphics[width=0.32\textwidth]{figures/HMHZZ/background/dnn/bkg_shape_ggZZ_VBF_incl_180_to_2200_log.pdf}
    \includegraphics[width=0.32\textwidth]{figures/HMHZZ/background/dnn/bkg_shape_ggZZ_rest_190_to_2200_log.pdf}
    \caption{Distributions of the \mfl invariant mass fit projections of the \ggZZ background samples for the $4\mu$,
    $4e$ and $2\mu 2e$ final states in the ggF-MVA-high category, the $4\ell$ inclusive ggF-MVA-low category and VBF-MVA-enriched category.
    DNN-based categorization is used.}
    \label{fig:ggZZ_m4l_shape_all_DNN}
\end{figure}

\begin{figure}[htbp]
    \centering
    \includegraphics[width=0.32\textwidth]{figures/HMHZZ/background/dnn/bkg_shape_qqZZEW_ggF_4mu_180_to_2200_log.pdf}
    \includegraphics[width=0.32\textwidth]{figures/HMHZZ/background/dnn/bkg_shape_qqZZEW_ggF_4e_180_to_2200_log.pdf}
    \includegraphics[width=0.32\textwidth]{figures/HMHZZ/background/dnn/bkg_shape_qqZZEW_ggF_2mu2e_180_to_2200_log.pdf} \\
    \includegraphics[width=0.32\textwidth]{figures/HMHZZ/background/dnn/bkg_shape_qqZZEW_VBF_incl_180_to_2200_log.pdf}
    \includegraphics[width=0.32\textwidth]{figures/HMHZZ/background/dnn/bkg_shape_qqZZEW_rest_180_to_2200_log.pdf}
    \caption{Distributions of the \mfl invariant mass fit projections of the \qqZZ (EW) background samples for the
    $4\mu$, $4e$ and $2\mu 2e$ final states in the ggF-MVA-high category, the $4\ell$ inclusive ggF-MVA-low category and VBF-MVA-enriched category.
    DNN-based categorization is used.}
    \label{fig:qqZZEW_m4l_shape_all_DNN}
\end{figure}

\subsection{Reducible backgrounds}

Similar to section~\ref{sec:background}, the reducible backgrounds include \Zjet (consisting of both heavy- and light-flavour jets), top quark pair, and $WZ$ production, which contain fake and non-isolated leptons.
The simulations are not very robust in terms of the selection efficiencies.
Thus, the data-driven method is applied to estimated the normalization of those processes in different control regions (CRs).
The estimations in this analysis are performed separately for \llmumu and \llee final states, with slightly different approaches for ``muon'' and ``electron'' backgrounds.

The ``electron'' backgrounds mostly come from process of a $Z$ boson with light-flavour jets ($Z$+LF) misidentified as electrons.
The large contribution of ``muon'' backgrounds come from heavy-flavour jets produced in association with a $Z$ boson ($Z$+HF) or in the decays of top quark.
The estimations are done following the common H4l studies without a specific \mfl range requirement~\cite{PhysRevD.91.012006}, and then the corresponding fraction of event yield in $\mfl > 200~\gev$ is calculated from MC simulation.

\textbf{\llmumu final states}

The normalizations of ``muon'' backgrounds are extracted from simultaneous fits of the leading lepton pair's invariant mass ($m_{12}$) in four orthogonal CRs:
\begin{itemize}
	\item \textbf{Inverted $d_{0}$ CR}: this CR is formed by inverting the $d_{0}$ selection for at least one lepton in subleading lepton pair while the leptons in leading pair are required to pass all standard selection.
This CR enhances $Z$+HF and \ttbar as leptons from heavy-flavour hadronic decays are characterised by large d0.
	\item \textbf{$e\mu+\mu\mu$ CR}: this CR is formed using an opposite-sign different-flavour dilepton in leading pair.
It aims to enhance \ttbar background as the leading lepton pair cannot come from $Z$ boson decay.
	\item \textbf{Inverted isolation CR}: in this CR, leptons in leading pair are required to satisfy all standard analysis selection, but for leptons in subleading pair, they are required to pass $d_{0}$ selection but have at least one of them failing isolation selection.
This CR enhances the events from $Z$+LF processes while suppress $Z$+HF by $d_{0}$ cut.
	\item \textbf{Same-sign CR}: in this CR, the leptons in subleading pair are required to have same-charge, while the leading pair still passes standard selection.
This CR is not dominant by any specific background since all reducible backgrounds could have sizable contribution to it.
\end{itemize}

The fit results of normalizations are then propagated to signal region (SR) by applying transfer factors to account for the difference of selection efficiencies between SR and CRs.
The transfer factors are computed using $Z+\mu$ MC samples.

\textbf{\llee final states}

The ``electron'' backgrounds are estimated in $3\ell+X$ CR, where $X$ denotes the lower \pt electron in the subleading pair.
The selection and identification criterias for $X$ are relaxed , while other three leptons must satisfy the standard selection.
In this case, $X$ could be a light-flavour jet, a photon conversion or an electron from heavy-flavour hadron decay.
Moreover, the subleading pair is required to have same charge dilepton to ensure the orthogonality to the signal region.
The normalization of backgrounds are obtained based on a fit to the number of hits in the innermost ID layer in CR,
and the transfer factors are computed from $Z+e$ simulated sample.

The \mfl shapes of reducible backgrounds are obtained from MC simulation in signal region, and then smoothed by an one-dimensional kernel estimation,
which models the input data as a superposition of Gaussian kernels, one for each data point with contributing $1/N$ to total integral $N$~\cite{Cranmer:2000du}.
The difference from using different smoothing strength ($\rho$) in kernel estimation is taken into account as additional shape uncertainties for these reducible backgrounds.


\documentclass{article}
\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{tikz}
\usepackage[all]{xy}
\usetikzlibrary{positioning,chains,fit,shapes,calc}

\begin{document}

\title{Homework 3}
\author{Josh Cai}

\maketitle
\section*{Section 9.5}

\textbf{30a)} The equivalence class of 010 is $[010]$, or all binary strings of 3 or more bits that start with 010.

\noindent\textbf{30d)}  The equivalence class of 01010101 is [010], or all binary strings of 3 or more bits that start with 010.
\\\\
\noindent\textbf{34b)} The equivalence class of 1011 is [1011], which consists of only the element 1011.

\noindent\textbf{34c)}  The equivalence class of 11111 is [11111], or all binary strings of 5 or more bits that start with 11111.
\\\\
\noindent\textbf{36a)} The congruence class $[4]_2$ is all the even numbers.

\noindent\textbf{36b)}  The congruence class $[4]_3$ is all numbers with remainder 1 after dividing by 3.

\noindent\textbf{36c)} The congruence class $[4]_6$ is all numbers with remainder 4 after dividing by 6.

\noindent\textbf{36d)}  The congruence class $[4]_8$ is all numbers with remainder 4 after dividing by 8.
\\\\
\noindent\textbf{42a)} Is a partition because has each of the elements of ${-3,-2,-1,0,1,2,3}$ in the subsets and each element only occurs once within all subsets.

\noindent\textbf{42d)}  Is not a partition because $0$ is not in the subsets.
\\\\
\noindent\textbf{44a)} Is a partition because every integer is either odd or even, so and no integer can be both odd and even.

\noindent\textbf{44b)}  Is not a partition because zero is not in either of the subsets of the partition.

\noindent\textbf{44e)}  Is not a partition because 2 is not divisible by 3 and is also even, and no element can exist in more than one subset of the partition.
\\\\

\noindent\textbf{46c)} Is not a partition because boundary points are contained within two subsets.

\noindent\textbf{46d)}  Is not a partition because the boundary points are not in any subsets.

\noindent\textbf{46e)}  Is a partition because every number exists in one and only one subset. The collection of subsets is the set of real numbers because $k$ is all integers.
\\\\
\noindent\textbf{48b)} \{(a,a),(b,b),(c,c),(c,d),(d,d),(d,c),(e,e),(e,f),(f,e),(f,f),(g,g)\}

\noindent\textbf{48d)}  \{(a,a),(a,c),(a,e),(a,g),(c,a),(c,c),(c,e),(c,g),(e,a),(e,c),(e,e),(e,g),(g,a), \\(g,c),(g,e),(g,g),(b,b),(b,d),(d,b),(d,d),(f,f)\}
\\\\
\noindent\textbf{56a)} If $R_1$ and $R_2$ are equivalence relations on $S$, then $R_1$ and $R_2$ are both reflexive, and therefore the union of the relations would be reflexive. $R_1$ and $R_2$ are both symmetric, if $(a,b)$ was in $R_1 \cup R_2$, then $(a,b)$ was in either $R_1, R_2$, or both. If it was in either $R_1$ or $R_2$, $(b,a)$ must exist in $R_1 \cup R_2$ since $(b,a)$ also existed in the relation that $R_1$ existed in. $R_1 \cup R_2$ is not necessarily transitive: if $(a,b)$ exists in $R_1$ and $(b,c)$ exists in $R_2$, $(a,c)$ need not necessarily exist.

\noindent\textbf{56c)}  It is not reflexive because since $R_1$ and $R_2$ were both reflexive, none of those elements can exist in $R_1 \oplus R_2$. It is symmetric because if $(a,b)$ existed in both sets, $(b,a)$ would exist in both sets, and neither would be in $R_1 \oplus R_2$. If $(a,b)$ and $(b,a)$ existed in one set, then both would exist in $R_1 \oplus R_2$. It is not necessarily transitive because both sets could have $(a,c)$ while one set has $(a,b)$ and another $(b,c)$.
\\\\
\noindent\textbf{60a)} $R$ is reflexive because $f(x)$ is trivially the same order as $f(x)$ (using the constant 1 for $C$ will show this). $R$ is symmetric because if $f = \Theta(g)$ then there exists constants $C_1, C_2$ such that $|f(x)|\le C_1|g(x)|$ and $|f(x)|\ge C_1|g(x)|$. The constants $C_1^{-1}, C_2^{-1}$ will satisfy the same conditions for $g$, so $g = \Theta(f)$. $R$ is transitive because if $f$ has the same order as $g$, and $g$ has the same order as $h$, then $f$ has the same order as $h$.

\noindent\textbf{60b)} The equivalence class of $f(n) = n^2$ is all functions of degree 2.
\\\\
\noindent\textbf{Bonus a)} There are $2^5 = 32$ different equivalence relations. Since each has 2 equivalence classes, we observe that the second equivalence class is simply the complement of the first equivalence class. Therefore, we can count the number of subsets of the set. The number of subsets of a set is $2^{\# of elements}$ because each element has the choice of being in the subset or not, and each choice is independent. However, we have overcounted with $2^6$, and we see that each subset is counted twice, once as a subset and the second as the complement of another subset. So we divide by 2, and get $2^5 = 32$ equivalence relations.

\noindent\textbf{Bonus b)}  There are $\binom{6}{2} = 15$ different equivalence relations. Since we have 5 equivalence classes, we know by Pigeonhole Principle one subset must have at least 2 elements. There cannot be a set with more than 2 elements, or there would be less than 5 equivalence classes. We see that the subset with 2 elements uniquely determines the equivalence relation, so we count all the different subsets of 2 elements in a set of 6 elements, which is just $\binom{6}{2} = 6 \times 5 / 2 = 15$.

\end{document}


\chapter{Assignment: Geolocations}
\label{ch:arheo_geolocations}

The Antikythera data set contains information on the geolocation of artefacts. The location is encoded as \textit{Xsugg}, \textit{Ysugg} coordinates in \textit{epsg:32634} system. We need to transform the data to the more standard \textit{EPSG: 4326} projection, which enables us to see the data on a map. We will do this with the help of a \widget{Python Script}.

\begin{figure}[h]
    \centering
    \includegraphics[scale=0.4]{python-script.png}
    \caption{$\;$} % empty caption for proper pagesetting
\end{figure}

\marginnote{For Python Script to work, one needs to have the `pyproj` package installed in the Orange environment. To bypass this, \href{http://file.biolab.si/datasets/pottery.tab}{here} are the preprocessed data you can load with File and connect directly to Geo Map.}

Now, we can plot the data on the map. We will use the Geo add-on for this. If you have not installed the add-on yet, go to Options --> Add-ons and install it. Restart Orange for the add-on to appear.

Connect \widget{Geo Map} to Python Script. The widget should automatically find latitude and longitude attributes, but if it doesn't, you can set them manually. The plot shows where the pottery shards were collected.

\begin{figure}[h]
    \centering
    \includegraphics[width=\textwidth]{workflow.png}
    \caption{$\;$} % empty caption for proper pagesetting
\end{figure}

\newpage

\subsection{Assignment}

\begin{figure*}[h]
    \centering
    \includegraphics[scale=0.4]{geo-map.png}
    \caption{$\;$} % empty caption for proper pagesetting
\end{figure*}

Use Geo Map widget and color the points by an appropriate attribute to answer the following questions:
\begin{itemize}
    \item Does the location of the shard correspond to the way it was collected?
    \item Use \widget{Select Rows} to keep only data instances with over 75\% certainty that they belong to the a certain period. Go through different periods (attributes from MNLN to Other) and observe if items from one period were found in a specific location of the island. Some periods will produce no results (no items with such high certainty), but that's ok. Skip them.
    \item Use \widget{Select Rows} to explore categorical variables. For example, set the filter to \textit{VesselPart is B} and observe where in the island the shards belonging to the body of the vessel were found. Explore variables of your interest and draw conclusions.
\end{itemize}


\section{Introduction}

Core-collapse supernovae (CCSNe) are the explosions of massive stars that end their lives.
They are directly or indirectly responsible for the lion's share of elements heavier than oxygen and play important roles in many astrophysical phenomena, such as neutron star and black hole formation.
Furthermore, these explosions occur at energies and densities relevant to address fundamental questions in nuclear, particle, and gravitational physics.
A solid theoretical framework for the CCSN explosion mechanism may help answer important questions in fundamental physics~\cite{janka_etal_2007}.

One essential part of the explosion mechanism is neutrino transport.
Neutrino energy deposition is believed to be the major driver of CCSN explosions, except in peculiar cases where rapid rotation is present and magnetohydrodynamic effects may dominate (for reviews, see~\cite{mezzacappa_2005,janka_2012,burrows_2013,muller_2016}).
Ideally, neutrino transport would be modeled by the Boltzmann transport equation, which is an integro-partial-differential equation evolving a phase-space distribution function $f$~(e.g., see~\cite{mezzacappaBruenn_1993a,mezzacappaBruenn_1993b,mezzacappaBruenn_1993c,mezzacappa_etal_2001,liebendoerfer_etal_2001,liebendoerfer_etal_2004,livne_etal_2004,liebendoerfer_etal_2005,ott_etal_2008,sumiyoshiYamada_2012,nagakura_etal_2014,sumiyoshi_etal_2014,nagakura_etal_2018} for studies of CCSN with Boltzmann transport in various approximate settings).
Simulating the neutrino transport implies finding a solution of the Boltzmann equation for a specific domain and period, with acceptable accuracy.

However, solving the Boltzmann transport equation with sufficient phase-space resolution and full weak interaction physics is at present too expensive.
To balance physical fidelity and computational expediency, an approximate method called the two-moment method has been adopted (e.g., see~\cite{kuroda_etal_2016,roberts_etal_2016,just_etal_2018,vartanyan_etal_2018}).
Using the two-moment method, the evolved variables are the zeroth and first angular moments of the distribution function $f$ -- the spectral particle density $\cJ$ and flux $\bcH$, respectively.
However, the equation of $\bcH$ includes the second angular moment $\bcK$.
Knowledge of $\bcK$ is needed to close the two-moment system.
Therefore, a closure that gives $\bcK$ consistent with $\cJ$ and $\bcH$ is needed.
The better the closure predicts $\bcK$, the more accurate the two-moment method will be.
The two-moment method has been widely applied in the CCSN modeling community with different algebraic closures, such as the Minerbo~\cite{minerbo_1978} closure (e.g.~{O'Connor} and {Couch}~\cite{oConnorCouch_2018}, Pan and et al.~\cite{pan_etal_2018}, Glas et al.~\cite{glas_etal_2018}, and Just et al.~\cite{just_etal_2018}) and the Levermore~\cite{levermore_1984} closure (e.g.~Vartanyan et al.~\cite{vartanyan_etal_2018}, Cabezon et al.~\cite{cabezon_etal_2018}, and Kuroda et al.~\cite{kuroda_etal_2016}).

Applying the two-moment method does simplify the problem, but doesn't guarantee an affordable solution.
To be precise, how to discretize the continuous system of equatio
ns given by the two-moment method and solve the discretized system efficiently remains a question.
In fact, the time scales of neutrino interactions with the background (they can be $\sim\mathcal{O}(10^{-10})$~second) is short compared to the duration of the CCSN explosion ($\sim\mathcal{O}(1)$~second).
This means that $\sim\mathcal{O}(10^{10})$ time steps could be needed for solving the system fully explicitly.
On the other hand, solving the moment equations fully implicitly requires inverting global band-structured matrices whose sizes depend on the phase-space discretization.
Such a global inversion is both expensive and unfriendly to parallelization.
To circumvent these challenges, implicit-explicit (IMEX) methods are taken into consideration.
By treating the transport terms in the two-moment equations explicitly and the collision terms implicitly, IMEX methods are subject only to a time step governed by the explicit transport terms, and the matrices to be inverted are block diagonal.
Therefore, IMEX methods require far fewer time steps compared with a fully explicit method, and the computation for each step is easily parallelizable.
For the relativistic setting that we have, where the fluid and the neutrinos have comparable propagation speeds, IMEX methods can be efficient.

To model neutrino transport using a two-moment method, two (or at least two) things need to be chosen carefully: an algebraic closure based on Fermi-Dirac statistics for closing the two-moment equations and a convex-invariant, diffusion-accurate IMEX scheme to ensure a physical result.
A convex-invariant scheme has the following property: if the solution $u^{n}\in W$ and $W$ is a convex set, then $u^{n+1}\in W$.
Since the neutrino distribution function is bounded ($f\in[0,1]$) by the Pauli exclusion principle, its moments as weighted integrals of a bounded function over the domain $\omega\in\bbS^{2}$ are also bounded.
We call the moments satisfying the constraints due to Fermi-Dirac statistics \textit{realizable moments} and the realizable $\cJ$ and $\bcH$ define a convex set~\cite{chu_etal_2018}.
The algebraic closure should give a realizable $\bcK$, and the well-posedness of the closure requires realizable $\cJ$ and $\bcH$.
This explains why an algebraic closure based on Fermi-Dirac statistics is needed.
Realizability of $\cJ$ and $\bcH$ after each time step requires a convex-invariant IMEX scheme.
Since the realizable moments form a convex set, it is possible to construct a realizability-preserving method using a convex-invariant IMEX scheme for two-moment neutrino transport.
In addition, the physics of neutrino transport in CCSNe requires the IMEX scheme to be diffusion-accurate.

The study of moment realizability and realizability-preserving methods with diffusion-accurate IMEX schemes motivates this work.
Gottlieb et al.~\cite{gottlieb_etal_2001} showed that standard strong-stability-preserving IMEX schemes cannot have an order higher than first without a restricted time step.
One way to obtain the second-order (or higher-order) accuracy is to add some correction steps after the standard step~\cite{huangShu_2017,hu_etal_2018}.
Unfortunately, the correction steps can deteriorate the accuracy of the IMEX scheme in the diffusion limit or restrict the time step.
To keep things simple, we focus on IMEX schemes without correction steps and require them to be high-order (second or higher order) in the streaming limit and diffusion-accurate.
We call these IMEX schemes \textit{PD-ARS}.

\texttt{thornado} is our toolkit for high-order neutrino-radiation hydrodynamics based on high-order Runge-Kutta Discontinuous Galerkin (RKDG) methods.
It is being developed at the University of Tennessee, Knoxville and Oak Ridge National Laboratory.
It currently includes solvers for the Euler equations for fluid dynamics and the two-moment approximation of the radiative transfer equation~\cite{endeve_etal_2018}.
In this paper, we focus on the transport methods in \texttt{thornado} with emphasis on IMEX.

This paper is organized as follows: Section~\ref{se:Two-MomentModel} discusses the mathematical model, algebraic closures, and the constraints on the moments and algebraic closures imposed by Fermi-Dirac statistics;
Section~\ref{se:SpatialDiscretization} gives a first-order finite-volume spatial discretization and shows how the spatial discretization preserves constraints in an IMEX step;
Section~\ref{se:TimeIntegration} discusses how to use convex combination to construct two PD-ARS schemes, one with second-order accuracy in the streaming limit and the other with third-order accuracy in the same limit;
Section~\ref{se:NumericalTests} presents the results of the numerical tests, which demonstrate the properties of the PD-ARS schemes; Section~\ref{se:Summary} summarizes the achievements of this paper and discusses future work.


\section{Approach}
	In this section we will present our aproach to tackle the speaker recognition problem.
\subsection{Erollment}
	\label{sec:approach_enrollment}
	An continuous speech of a user is collected during enrollment of that user. Depending on
	the number of users we plan to enroll and the accuracy of recognition system, duration
	of the utterance we need to collect may vary from 15 seconds to 30 seconds.

	Further processing of the utterance follows following steps:
	\begin{enumerate}
		\item \textbf{Voice Activity Detection (VAD)} \\
			Due to the channel difference and background noise may occur, which
			may degrade the performance of the system, non-speech part of the
			speech are removed from the utterance. This may lead to
			information loss (such as person's voice pause style), which in
			turns introduced the trade-off between loss of information and
			noise been modeled.

			As the corpus provided by teacher is nearly noise-free, we use a
			simple energy-based approach to remove the silence part.

		\item \textbf{Extract MFCC Feature } \\
			The utterance is then transformed to MFCC features with $20ms$
			frame-duration and $10ms$ frame-shift. Further more, we tuned the
			parameters of MFCC to yield better result. 40 filter banks and 19
			cepstral coefficent are parameters found to perform best.

			We have extensively examined MFCC feature. We found that, the
			distribution of a person's MFCC feature is truly a good representation
			of its vocal characteristics. Here are some plots of the first three
			dimensions of a spearker's MFCC feature.
			\begin{figure}[!ht]
				\begin{minipage}{0.48\linewidth}
					\centering
					\includegraphics[width=\linewidth]{res/m_002_03.trimed.png}
					\caption{The first three dimension of a man's MFCC feature}
				\end{minipage}
				\hfill
				\begin{minipage}{0.48\linewidth}
					\centering
					\includegraphics[width=\linewidth]{res/m_004_03.trimed.png}
					\caption{The first three dimension of another man's MFCC feature}
				\end{minipage}
				\vfill
				\begin{minipage}{0.48\linewidth}
					\centering
					\includegraphics[width=\linewidth]{res/f_001_03.trimed.png}
					\caption{The first three dimension of a woman's MFCC feature}
				\end{minipage}
				\hfill
				\begin{minipage}{0.48\linewidth}
					\centering
					\includegraphics[width=\linewidth]{res/f_003_03.trimed.png}
					\caption{The first three dimension of another woman's MFCC feature}
				\end{minipage}
			\end{figure}

			It is clearly showed in the figure that, women's MFCC feature distribution
			tends to have nice ecliptical shapes in first three-dimension,
			as women's voice pitch is higher than man. And the two men's MFCC figure
			shows great difference on density over space. It shows high correspondency
			of the theoretical analysis.

		\item \textbf{Gaussian Mixture Model}

			As the MFCC feature of a utterance of a user is a high-dimensional point
			set, which comprised a distribution, we model a user by modeling its
			MFCC feature distribution. The generative model we have primally adopted is
			Gaussian Mixture Model(GMM).

			In order to conduct a proof-of-concept experiments, we consequently conducted
			experiments using GMM from \textit{scikit-learn}\cite{scikit-learn}
			and \textit{PyPR}\cite{pypr}, but it turns out that these implementations
			are neither of efficiency, nor of decent accuracy. Therefore, we implemented
			our own GMM model with following improvements:
			\begin{itemize}
				\item As GMM is very sensitive to initialization, we use
					K-Means to initialize the configuration prior to GMM training.

				\item Further more, we use an improved algorithm of
					K-means++\cite{arthur2007k} named K-meansII\cite{bahmani2012scalable},
					which outperforms both naive K-means and K-means++ in both
					efficiency and accuracy. For detailed algorithm description,
					please consult the orignal paper.

				\item All above algorithms (K-means, K-means++, K-meansII and
					GMM) are implemented utilizing multi-core CPUs in
					thread level. Experiments shows that our implementation
					scales nearly linearly with number of cores.

				\item In order to futher speed up the training process,
					we examined the time consuming part of GMM algorithm,
					which lies in extensive use of exponential function.
					Therefore, we rewrite exponential function using 5-order polynomial
					approximation using \textbf{SSE} instructions, which only
					introduced an numerical error less than $10^{-7}$.
					The running time of the training process cut by half when using
					our optimization.
			\end{itemize}

			The efficiency we achieved enables us to build a
            GUI system with more powerful features, if embeded with our GMM training techniques.

			Further more, our implementation of GMM outperforms the GMM library
			aforementioned in terms of recognition accuracy.

			For detailed efficiency and perfomance benchmark, see \secref{result}

		\item \textbf{Continuous restricted Boltzmann Machine(CRBM)}

			Restricted Boltzmann Machine(RBM) is generative stochastic neural
			network that can learn a probability distribution over its set of
			inputs\cite{rbm_wiki}. But original version of RBM can only deal
			with binary inputs. \cite{chen2003continuous} proposed and extension called
			Continuous restricted Boltzmann Machine, which can deal with real-valued
			inputs. RBM is a two-layer neural network, in which one layer called
			visible layer (the input), and another called hidden layer. The neurons
			in hidden layer controls the model complexity and the performance of
			the network. The Gibbs sampling of hidden layer can be seen as a
			representation of the original data. Therefore RBMs can be used
			as an auto feature-extractor.

			Both RBM and CRBM can be trained using Contrastive Divergence learning.

			RBM has a ability to, given an input(visible layer), reconstruct a visible
			layer that is similar to the input. This demonstrates the modeling essence
			of RBM. \figref{crbm} illustrate original MFCC data and the sampled output of
			reconstructed data from CRBM.
			\begin{figure}[!ht]
				\begin{minipage}{0.48\linewidth}
					\centering
					\includegraphics[width=\linewidth]{res/all.trimed.png}
					\caption{The first three dimension of a woman's MFCC feature}
				\end{minipage}
				\hfill
				\begin{minipage}{0.48\linewidth}
					\centering
					\includegraphics[width=\linewidth]{res/50.trimed.png}
					\caption{The first three dimension of the same woman's MFCC feature
					recontructed by a CRBM with 50-neuron hidden layer. We can
					see that, the density of these two distributions are alike}
				\end{minipage}
                \caption{\label{fig:crbm}}
			\end{figure}

			It turns out that, although CRBM has good modeling ability, but
			due to the inconsiderate of the time-varing property of sequential
			signal, although experiments we conducted using CRBM indicate
			considerable performance on speaker recognition task, using CRBM
			naively as a substitution of GMM did not yield better result till now.

		\item \textbf{JFA}:

          \textbf{Factor Analysis} is a typical method which behave
          very well in classification problems, due to its ability to
          account for different types of variability in training data.
          Within all the factor analysis methods,
          Joint Factor Analysis (JFA)\cite{jfa2,jfa-se} was proved to outperform other method
          in the task of Speaker Recognition.

          JFA models the user by ``supervector'' , i.e. a $C\times F $ dimension vector, where $C$ is
          the number of components in the Universal Background Model, trained by GMM on all the training data,
          and $ F$ is the dimension of the acoustic feature vector. The supervector of an utterance is obtained by concatenate
          all the $C $ means vectors in the trained GMM model. The basic assumption of JFA on describing a supervector is:

          \[ \vec{M} = \vec{ m } + vy + dz + ux, \]

          where $m$ is a supervector usually selected to be the one trained from UBM, $v$ is a $ CF \times R_s$ dimension matrix,
          $ u$ is a $ CF \times R_c$ dimension matrix, and $d$ is a diagonal matrix.
          This four variables are considered independent of all kinds of variabilities and remain constant after training, and
          $x, y, z $ are matrixes computed for each utterance sample.
          In this formulation, $ m + vy + dz$ is commonly believed to account for the ``Inter-Speaker Variability'', and $ux $ accounts
          for the ``Inter-Channel Variability''.
          The parameter $ R_s $ and $ R_c$, also referred to as ``Speaker Rank'' and ``Channel Rank'', are two emprical constant selected as first.
          The training of JFA is to calculate the best $ u, v, d$ to fit all the training data.

          After our investigation, we found that the original algorithm \cite{jfa-se} for training JFA model is of
          too much complication and hard to implement.
          Therefore, we use the simpler algorithm presented in \cite{jfa-study}
          to train the JFA model. However, from the result, JFA does not seem to outperform our enhanced MFCC and GMM algorithms
          (but do outperform our old algorithms). It is suspected that the training of a JFA model needs more data than
          we have provided, since JFA needs data from various source to account for different types of variabilities.
          Therefore, we might need to add extra data on the training of JFA, but keep the same data scale in the stage of enrollment,
          to get a better result.

          It is also worth mentioning that the training of JFA will take much longer time than our old method,
          since the estimation process of $ u, v, d$ does not converge quickly. As a result, it might not be practical to add
          JFA approach to our GUI system. But we will still test further on the performance of it, compared to other methods.
	\end{enumerate}

\subsection{Recognition}
	Recognition procedure follows steps below:
	\begin{enumerate}
		\item Record a short utterance of the speaker (typically 5 seconds)
		\item Preprocess the utterance using first two steps described in
			\secref{approach_enrollment}, e.g, VAD and MFCC feature extraction.
		\item Compute the `score' of the MFCC featuresextraced for each model of persons
			enrolled, and adopt person corresponding the model which gives highest score to be the
			recognition result.

			A typical form of score is log-likelihood (or enery in RBM case).

	\end{enumerate}