\section{Matrix-free multigrid}
  \label{section:applications:matrix-free-multigrid}

\chapterDescription
  {
    1--2 days.
  }
  {
    Chapter \ref{chapter:quickstart}. It is advantageous if the reader has
    studied the implementation of the heat equation before.
  }

In this section, we sketch how to solve the convection-diffusion equation
\[
  - \nabla (\epsilon \nabla) u + \nabla (v\ u) = f \qquad \mbox{with } v \in
  \mathbb{R}^d, \epsilon \in \mathbb{R}^{d \times d}
\]
with various geometric multigrid solvers. $epsilon$ is a diagonal matrix with
entries $\epsilon _1, \epsilon _2$ or $\epsilon _1, \epsilon _2, \epsilon _3$,
respectively.
Our realisation is based upon a few design decisions:

\begin{enumerate}
  \item The solvers use the spacetree as computational grid.
  \item We use a finite element formalism with $d$-linear shape functions.
  \item The material parameters $\epsilon $ and $v$ are given per cell.
\end{enumerate}

\noindent
For the implementation, we use Peano's \texttt{matrixfree} toolbox.
This is a tiny little collection of helper classes to work with stencils and
local assembly matrix.
It is neither fast, i.e.~computationally mature, nor can it cope with real
stencil libraries, but it does the job.

\subsection{Setup}

We start with Peano's PDT and generate a project. We also link Peano's sources
into the project and unzip the \texttt{matrixfree} toolbox.
\begin{code}
  java -jar pdt.jar --create-project multigrid multigrid
  ln -s mypath/src/peano
  ln -s mypath/src/tarch
  cp mypath/tarballs/toolboxes/matrixfree.tar.gz .
  tar -xzvf matrixfree.tar.gz
\end{code}

\noindent
I recommend to hold \texttt{matrixfree} parallel to the \texttt{multigrid},
\texttt{peano} and \texttt{tarch} directory.
As soon as we use such a toolbox, we also might have to adopt our makefile
accordingly: we have to add the matrixfree directory to the find pathes when we
build up a list of source codes.
Furthermore, you might have to add an additional search directory. If you place
your toolbox parallel to \texttt{peano} and \texttt{tarch}, this however should
not be necessary.
Here's the corresponding excerp from the makefile\footnote{Please note that some hyphens might not be displayed properly}:
\begin{code}
files.mk:
    touch files.mk
    echo -n SOURCES= > files.mk
    find -H $(PEANO_HOME)/peano -name *.cpp | awk { printf "%s ", $$0 } >> files.mk
    find -H $(PEANO_HOME)/matrixfree -name *.cpp | awk { printf "%s ", $$0 } >> files.mk
    find $(PROJECT_HOME) -name *.cpp | awk { printf "%s ", $$0 } >> files.mk
\end{code}
% '*.cpp' | awk '{ printf "%s ", $$0 }' >> files.mk
%     find -H $(PEANO_HOME)/tarch -name '*.cpp' | awk '{ printf "%s ", $$0 }' >> files.mk
%     find -H $(PEANO_HOME)/matrixfree -name '*.cpp' | awk '{ printf "%s ", $$0 }' >> files.mk
%     find $(PROJECT_HOME) -name '*.cpp' | awk '{ printf "%s ", $$0 }' >> files.mk

\noindent
We next add our material parameters to the \texttt{Cell.def} file
\begin{code}
Packed-Type: short int;

Constant: DIMENSIONS;

class multigrid::records::Cell {
  persistent parallelise double   epsilon[DIMENSIONS];
  persistent parallelise double   v[DIMENSIONS];
};
\end{code}

\noindent
and create a simple first specification file:
\begin{code}
component: Multigrid

namespace: ::multigrid

vertex:
  dastgen-file: Vertex.def

cell:
  dastgen-file: Cell.def

state:
  dastgen-file: State.def

event-mapping:
  name: CreateGrid

event-mapping:
  name: PlotCells

adapter:
  name: CreateGrid
  merge-with-user-defined-mapping: CreateGrid
  merge-with-user-defined-mapping: PlotCells

\end{code}

\noindent
We run this specification file through the PDT

\begin{code}
java -jar <mypath>/pdt.jar --generate-gluecode multigrid/project.peano-specification multigrid
\end{code}

\noindent
and implement both the plotter and the creational mapping such that we have a
few characteristic setups.
It might however make sense to validate that make passes before we start any
PDE-specific coding:
\begin{code}
make -f multigrid/makefile
\end{code}

\noindent
We next introduce an operation
\begin{code}
matrixfree::stencil::ElementWiseAssemblyMatrix multigrid::Cell::getElementsAssemblyMatrix(
  const tarch::la::Vector<DIMENSIONS,double>&  h
) const {
  matrixfree::stencil::ElementWiseAssemblyMatrix result;

  const matrixfree::stencil::Stencil laplacianStencil =
    matrixfree::stencil::getLaplacian(_cellData.getEpsilon(), h);

  return matrixfree::stencil::getElementWiseAssemblyMatrix(laplacianStencil);
}
\end{code}
which returns the $\mathbf{R}^{2^d \times 2^d}$ local system matrix.
The method sets up the stencils that correspond to a regular Cartesian system
given the mesh size \texttt{h} and the material parameters.
Here, also the convective term has to be handled.
Finally, it uses \texttt{getElementWiseAssemblyMatrix} to extract the actual
matrix from this stencil.

\begin{remark}
  Peano supports all stencil/linear algebra operations for complex values.
\end{remark}

\subsection{Jacobi smoother}

The basic building block of all of our solvers is a simple Jacobi smoother
working on adaptive grids as well as on multiple scales.
Its realisation is an extension of the solver in \ref{section:applications:heat-equation}.
To make it work without the assembly of any global matrix, we associate each
vertex a residual value as well as the actual value.
There are a few other features such as boundary properties or some level
analysis that we either use later on or we pass to the used toolboxes.
Their exact semantics and rationale have to be taken from the source code.

\begin{code}
Packed-Type: short int;

class multigrid::records::Vertex {
  // Solution
  persistent parallelise double  u;

  // Rhs
  persistent parallelise double  f;

  // Residual
  persistent parallelise double   r;

  // Diagonal element
  persistent parallelise double   d;

  enum VertexType {
    Unknown, Dirichlet, Neumann
  };

  persistent VertexType vertexType;

  // some other attributes
};
\end{code}

\noindent
Besides a proper initialisation of the vertices, we extend the specification
similar to the heat equation.
\begin{code}
component: Multigrid

namespace: ::multigrid

vertex:
  dastgen-file: Vertex.def
  read scalar(double): U
  read scalar(double): R
  read scalar(double): D
  read scalar(double): F
  write scalar(double): U
  write scalar(double): R
  write scalar(double): D

...

adapter:
  name: CreateGrid
  merge-with-user-defined-mapping: CreateGrid
  merge-with-user-defined-mapping: PlotCells
  merge-with-predefined-mapping: VTKPlotVertexValue(u,getU,u)
\end{code}

\noindent
Once this code framework passes (only minor technical helper routines have to
be implemented, but by now this should be straightforward to any user), we can
introduce a new mapping/adapter \texttt{JacobiSmoother}, call this one a couple of hundred times in the runner and plot the result file then.
For the latter, it makes sense to use a predefined plotter.
The interesting new aspects can be found in three routines of the smoother.
The design of the code realises matrix-free element-wise mat-vecs 1:1:

\begin{enumerate}
  \item Each vertex carries a residual and a diagonal value attribute.
    They are cleared whenever the vertex is read the very first time in
    a traversal.
    \begin{code}
void multigrid::mappings::JacobiSmoother::touchVertexFirstTime(...) {
  logTraceInWith6Arguments( "touchVertexFirstTime(...)", ... );

  fineGridVertex.clearAccumulatedAttributes();

  logTraceOutWith1Argument( "touchVertexFirstTime(...)", fineGridVertex );
}

// in Vertex files

void multigrid::Vertex::clearAccumulatedAttributes() {
  _vertexData.setR(0.0);
  _vertexData.setD(0.0);
}
    \end{code}

    \noindent
    Our concept is that we accumulate the diagonal element $d$ and the residual
    $r$ within these (temporary) attributes per vertex throughout the traversal.

    \item When we use a vertex for the very last time, we may thus update the
    unknown according to the values of the residual and the diagonal value. This
    is the actual Jacobi smoothing step:
    \[
      u ^{(new)} \gets u ^{(old)} + \omega \frac{1}{d} r
    \]
    \begin{code}
void multigrid::mappings::JacobiSmoother::touchVertexLastTime(...) {
  const bool hasUpdated = fineGridVertex.performJacobiSmoothingStep( omega );

  ...
}

// in Vertex files

double multigrid::Vertex::getResidual() const {
  return _vertexData.getF() + _vertexData.getR();
}

void multigrid::Vertex::performJacobiSmoothingStep( double omega ) {
  if (_vertexData.getVertexType()== Records::Unknown) {
    assertion1( _vertexData.getD()>0.0, toString() );
    assertion2( omega>0.0, toString(), omega );
    _vertexData.setU( _vertexData.getU() + omega / _vertexData.getD() * getResidual() );
  }
}
    \end{code}

    \noindent
    Please note that the residual here is modelled as sum of the right-hand
    side and the accumulated value (cf.~helper operation
    \texttt{getResidual()}).
    We anticipate the minus from the definition
    \[ r = f - Au \]
    already in the accumulation, i.e.~sum up $-Au$ in the vertex attribute $r$.
    An additional pitfal is this context stems from the usage of a finite
    element method.
    It implies that we have to ensure that $f$ is scaled with $h^d$ ($h$ being the local mesh width), which is something we typically
    do already in the initialisation.

    In the present implementation, we make the Jacobi update return a flag that
    indicates whether a fine grid update has been done or not.
    Most codes will like to track global data such as a global residual or the
    maximum value of the solution and can use this flag to decide whether a
    vertex contributes to global data or not. Please study the accompanying
    source code for details.

    \item The most complicated part is obviously the evaluation of the local
    mat-vec contributions. Here, we rely on the cell's
    \texttt{getElementsAssemblyMatrix} as well as operations from \newline
    \texttt{VertexOperations}. All operations in this class are generated by the
    PDT and extract from \texttt{enterCell}'s vertices vectors: you hand in all
    fine grid data and extract a vector of all $u$ values, e.g. The other way
    round is supported as well. The operations within this helper class are all
    generated because of the read and write statements in the specification.
    \begin{code}
#include "multigrid/VertexOperations.h"

void multigrid::mappings::JacobiSmoother::enterCell(...) {
  logTraceInWith4Arguments( "enterCell(...)", fineGridCell, ... );

  const tarch::la::Vector<TWO_POWER_D,double> u    =
    VertexOperations::readU( fineGridVerticesEnumerator, fineGridVertices );
  const tarch::la::Vector<TWO_POWER_D,double> dOld    =
    VertexOperations::readD( fineGridVerticesEnumerator, fineGridVertices );
  const tarch::la::Vector<TWO_POWER_D,double> rOld =
    VertexOperations::readR( fineGridVerticesEnumerator, fineGridVertices );
  const matrixfree::stencil::ElementWiseAssemblyMatrix A =
    fineGridCell.getElementsAssemblyMatrix( fineGridVerticesEnumerator.getCellSize() );

  tarch::la::Vector<TWO_POWER_D,double> r = rOld - A * u;
  tarch::la::Vector<TWO_POWER_D,double> d = dOld + tarch::la::diag(A);

  VertexOperations::writeR( fineGridVerticesEnumerator, fineGridVertices, r );
  VertexOperations::writeD( fineGridVerticesEnumerator, fineGridVertices, d );

  logTraceOutWith1Argument( "enterCell(...)", fineGridCell );
}
    \end{code}
\end{enumerate}

\subsection{Environment}

The changes in the environment are straightforward once the smoother is
in place:
\begin{enumerate}
  \item We extend the runner such that it switches to the Jacobi smoother once
  the grid is set up and triggers a fixed number of iterations then.
  \item We add a logging device to the runner
  \begin{code}
    class multigrid::runners::Runner {
      private:
        static tarch::logging::Log  _log;
        ...
    };
  \end{code}
  and make the innermost loop plot residual and other statistics after each
  grid traversal:
  \begin{code}
  repository.switchToJacobiAndPlot();
  for (int i=0; i<100; i++) {
    repository.iterate();

    logInfo(
      "runAsMaster(...)",
      "#vertices=" << repository.getState().getNumberOfInnerLeafVertices() <<
      ",|res|_2=" << repository.getState().getResidualIn2Norm() <<
      ",|res|_max=" << repository.getState().getResidualInMaxNorm() <<
      ",|u|_L2=" << repository.getState().getSolutionInL2Norm() <<
      ",|u|_max=" << repository.getState().getSolutionInMaxNorm() <<
      ",#stencil-updates=" << repository.getState().getNumberOfStencilUpdates()
    );

    repository.getState().clearAccumulatedAttributes();
  }
  \end{code}
  \item To make the code work, we augment the state with the corresponding
  fields
  \begin{code}
Packed-Type: short int;

class multigrid::records::State {
  // Stores squared value, i.e. apply sqrt before returning it
  persistent parallelise double residual2Norm;
  persistent parallelise double residualMaxNorm;
  // Stores squared value, i.e. apply sqrt before returning it
  persistent parallelise double solutionL2Norm;
  persistent parallelise double solutionMaxNorm;
  persistent parallelise double numberOfStencilUpdates;
};
  \end{code}
  and realise the corresponding setters and getters. The design of the state
  methods (notably a method \texttt{clearAccumulatedAttributes()} in
  combination with \texttt{merge}) follows recommendations motivated in Section
  \ref{section:parallelisation:shared-memory}. For the time being, we do not
  discuss them further.
  \item Finally, we extend the \texttt{main} such that it can read in a
  well-suited relaxation parameter from the command line. It then sets this
  relaxation parameter (the static field) in the mapping:
  \begin{code}
    multigrid::mappings::JacobiSmoother::omega = atof( argv[2] );
  \end{code}
\end{enumerate}

\noindent
We may run this code for example the
well-known Poisson benchmark ($\epsilon = 1, v=0, f=d\ \pi ^2 \prod _i \sin
\left( x_i \pi \right) $ ), and observe the well-known dependency of Jacobi on
the mesh width (below grids with two, three or four compute grid levels):

\begin{center}
  \includegraphics[width=0.3\textwidth]{42_matrix-free-multigrid/Poisson3.png}
  \includegraphics[width=0.3\textwidth]{42_matrix-free-multigrid/Poisson4.png}
  \includegraphics[width=0.3\textwidth]{42_matrix-free-multigrid/Poisson5.png}
\end{center}

\subsection{Dynamically adaptive Jacobi mit FAC}

\noindent
We next implement a dynamically adaptive Jacobi solver that implements the FAC
scheme.
Its fundamental idea is that the Jacobi smootherr is applied on each and every
grid level in parallel.
Any hanging node's value is interpolated from coarser grids.
While we update all grid levels, we do overwrite coarse vertices with fine grid
values for all vertices that do exist on finer grid resolutions as well.
We inject the solution from the fine grids onto coarser grids.
This first solver is capable to handle dynamically adaptive grids with
arbitrary refinement pattern.
It also is a preliminary exercise how to implement multigrid full approximation
storage (FAS).

We extend the grid setup slightly such that it creates a very coarse grid even
if we prescribe a fine minimum mesh size.
Next, we validate that \texttt{enterCell} evaluates the stencil on each grid
level.
This leaves two tasks: interpolatation and injection.
The injection is basically the same we have used in the heat equation before.
\begin{code}
void multigrid::mappings::JacobiSmoother::touchVertexLastTime(...) {
 // see code snippets introduced before

 if (
  peano::grid::SingleLevelEnumerator::isVertexPositionAlsoACoarseVertexPosition(
    fineGridPositionOfVertex
  )
 ) {
  const peano::grid::SingleLevelEnumerator::LocalVertexIntegerIndex coarseGridPosition =
    peano::grid::SingleLevelEnumerator::getVertexPositionOnCoarserLevel
    (fineGridPositionOfVertex);
  coarseGridVertices[ coarseGridVerticesEnumerator(coarseGridPosition) ].inject(fineGridVertex);
 }
}

// in the vertex

void multigrid::Vertex::inject(const Vertex& fineGridVertex) {
  _vertexData.setU( fineGridVertex._vertexData.getU() );
}
\end{code}

\begin{remark}
  The \texttt{matrixfree} toolbox offers a type \texttt{solver::Smoother} that
  realises a Jacobi smoother that automatically tracks different residual and
  solution norms. In the present example we do not use this smoother while we
  use the toolbox's multigrid class. This is kind of inconsistent. It would
  probably be better to use the smoother as well.
\end{remark}

\noindent
The interpolation follows the idea of the heat equation solver, too.
However, we propose to rely on a premanufactured interpolation operation from
the \texttt{matrixfree} toolbox.
For this, we make our smoother mapping hold an instance of
\texttt{matrixfree::solver::Multigrid  \_multigrid}.
This object offers us an operation \texttt{getDLinearInterpolatedValue}:

\begin{code}
void multigrid::mappings::JacobiSmoother::createHangingVertex(...) {
  logTraceInWith6Arguments( "createHangingVertex(...)", ... );

  fineGridVertex.setU(
    _multigrid.getDLinearInterpolatedValue(
      VertexOperations::readU( coarseGridVerticesEnumerator, coarseGridVertices ),
      fineGridPositionOfVertex
    )
  );

  logTraceOutWith1Argument( "createHangingVertex(...)", fineGridVertex );
}
\end{code}

\noindent
Obviously, the vertex requires an additional \texttt{setU( double )} operation
to make this snippet work.
Its implementation is trivial.
This is the only additional extension required.
If we adopt the setup, we might solve the equation on grids alike below where
the mesh is resolved up to the finest level along the boundaries.
The latter is a proper choice for many problems, though inadequate and thus
only a proof of concept for the present case:

\begin{center}
  \includegraphics[width=0.3\textwidth]{42_matrix-free-multigrid/AdaptivePoisson3.png}
  \includegraphics[width=0.3\textwidth]{42_matrix-free-multigrid/AdaptivePoisson4.png}
\end{center}

\begin{remark}
  On some machines, the resulting code will fail in \texttt{Assert} mode as the
  plotters complain about nan for the right-hand side. In the release
  mode, i.e.~without assertions, no complaint is raised, but some visualisation
  software might complain about the nans. To fix this issue with the plotting,
  it is important to set the right-hand side $f$ to zero for hanging nodes and
  for boundary nodes. We propose to introduce a \texttt{clearF()} operation and
  to call it prior to any plotting (in the \texttt{CreateGrid} mapping for
  example) for hanging and boundary vertices.
\end{remark}

\noindent
We close our discussion on the Jacobi smoother with the introducton of a dynamic
refinement criterion.
For this, we rely on a predefined refinement criterion offered with the
matrixfree toolbox.
We use
\texttt{LinearSurplusRefinementCriterionWithFixedMeshSizes}
from \\
\texttt{matrixfree::adaptivitycriteria}.
There is an extensive documentation how to use it in its superclass' header, so
we do not reiterate this here.
Instead we just show some of the added code:

\begin{code}
multigrid::mappings::RefinementCriterion::RefinementCriterion():
  _refinementCriterion(
    0.1,                   // refinementPercentage,
    0.0,                   // deletePercentage,
    0.5,                   // minimumMeshSize,
    0.5                    // maximumMeshSize
  ) {
}

void multigrid::mappings::RefinementCriterion::touchVertexFirstTime(...) {
  VertexOperations::writeLinearSurplus(fineGridVertex,0.0);
}

void multigrid::mappings::RefinementCriterion::enterCell(...) {
  VertexOperations::writeLinearSurplus(
    fineGridVerticesEnumerator,
    fineGridVertices,
    _refinementCriterion.getNewLinearSurplus(
      VertexOperations::readU(fineGridVerticesEnumerator,fineGridVertices),
      VertexOperations::readLinearSurplus(fineGridVerticesEnumerator,fineGridVertices)
    )
  );
}

void multigrid::mappings::RefinementCriterion::touchVertexLastTime(...) {
  if ( fineGridVertex.isInside() ) {
    const tarch::la::Vector<TWO_POWER_D_TIMES_D,double > coarseGridLinearSurplus =
      VertexOperations::readLinearSurplus(coarseGridVerticesEnumerator, coarseGridVertices)
      +
      _refinementCriterion.getLinearSurplusContributionFromFineGrid(
        VertexOperations::readLinearSurplus( fineGridVertex ),
        fineGridVertex.getRefinementControl()==Vertex::Records::Unrefined,
        fineGridPositionOfVertex
      );

    VertexOperations::writeLinearSurplus( coarseGridVerticesEnumerator,
      coarseGridVertices, coarseGridLinearSurplus );

    switch (
      _refinementCriterion.analyse(
        VertexOperations::readLinearSurplus(fineGridVertex),
        fineGridVertex.getRefinementControl()==Vertex::Records::Refined,
        fineGridVertex.getRefinementControl()==Vertex::Records::Unrefined,
        fineGridH
      )
    ) {
      case matrixfree::adaptivitycriteria::LinearSurplusRefinementCriterion::Refine:
        fineGridVertex.refine();
        break;
      case matrixfree::adaptivitycriteria::LinearSurplusRefinementCriterion::Delete:
      case matrixfree::adaptivitycriteria::LinearSurplusRefinementCriterion::NoAction:
        break;
    }
  }
}
\end{code}

\noindent
The criterion's idea is to evaluate a stencil that allows us to compare the
linear interpoland of the solution within a vertex to the actual vertex's
solution value.
We anticipate what would happen if we could remove a particular vertex alone.
The other way round, we assume that vertices with a big difference are critical,
and we would benefit a lot if we would refine around this vertex.

The offered criterion class bucket sorts the linear surplus values. This is not
exact, but it does not require a real sort being in $\mathcal{O}(n \log
n)$---actually nothing is sorted, but each bucket is flagged (refine, coarse,
do nothing).
If a vertex falls into a particular bucket, the criterion returns the
corresponding flag.
More sophisticated criteria might yield better approximation patterns, but this
generic one works surprisingly well.

\begin{center}
  \includegraphics[width=0.24\textwidth]{42_matrix-free-multigrid/DynamicA00.png}
  \includegraphics[width=0.24\textwidth]{42_matrix-free-multigrid/DynamicA01.png}
  \includegraphics[width=0.24\textwidth]{42_matrix-free-multigrid/DynamicA02.png}
  \includegraphics[width=0.24\textwidth]{42_matrix-free-multigrid/DynamicA03.png}
\end{center}

\noindent
This code marks up to ten percent of the vertices for refinemenet and
consequently refines all the surrounding cells given that the residual in
the vertices underruns the given threshold \\
\texttt{\_convergenceThreshold}.
The latter magic constant is set to $10^{-2}$ for the plots above.

Now, one simple feature is worth trying: We have so far always used a
predefined visualisation routine that visualises the finest grid of a
simulation.
Among the set of standard visualisation routines also is a plotter that plots
the individual levels of a grid.
Given that we inject the solution to coarse levels anyway, this allows us to
visualise all data in a multilevel fashion.
To use it, we again extend our specification, regenerate the glue code and
recompile.

\begin{code}
adapter:
  name: JacobiAndPlot
  merge-with-user-defined-mapping: CreateGrid
  merge-with-user-defined-mapping: JacobiSmoother
  merge-with-user-defined-mapping: PlotCells
  merge-with-predefined-mapping: VTKPlotVertexValue(u,getU,u)
  merge-with-predefined-mapping: VTKPlotVertexMultilevelValue(multiscaleU,getU,u)
\end{code}

\noindent
Some minor remarks on proper initialisation that are often encountered shall
close the discussion.
First, the presented refinement criterion does not properly refine along the
domain's boundary, as it does not distinguish whether a stencil is applied along
the boundary.
In practice, it is reasonable to ensure that the boundary is always refined at
least as fine as the vertices next to the boundary.
In practive, it is reasonable to avoid hanging vertices along the boundary.
This can be achieved if we check within each cell whether there is a boundary
vertex and whether one vertex is refined.
If both properties hold, all unrefined boundary vertices have to be refined.
The class \texttt{peano::grid::aspects::VertexStateAnalysis} provides generic
helper routines to realise this behaviour with a few lines:

\begin{code}
#include "peano/grid/aspects/VertexStateAnalysis.h"
#include "peano/utils/Loop.h"

void multigrid::mappings::RefinementCriterion::enterCell(...) {
  ...
  if (
    fineGridCell.isRefined()
    &&
    peano::grid::aspects::VertexStateAnalysis::doesOneVertexCarryRefinementFlag(
      fineGridVertices, fineGridVerticesEnumerator, Vertex::Records::Unrefined
    )
  ) {
    bool isOneVertexABoundaryVertex = false;
    dfor2(k)
      isOneVertexABoundaryVertex |=
        fineGridVertices[ fineGridVerticesEnumerator(k) ].isBoundary();
    enddforx
    if (isOneVertexABoundaryVertex) {
      dfor2(k)
        if (fineGridVertices[ fineGridVerticesEnumerator(k) ].getRefinementControl()
          ==Vertex::Records::Unrefined) {
          fineGridVertices[ fineGridVerticesEnumerator(k) ].refine();
        }
      enddforx
    }
  }

}
\end{code}

\noindent
Second, many codes require a proper balancing.
As Peano relies on three-partitioning, a 3:1 balancing is something many codes
would like to have:
No two neighbouring cells differ in their maximum refinement level by more than
one.
This can be realised manually within a mapping.
Or you may decide to merge the predefined mapping \texttt{GridBalancer3to1} into
your adapters.

Finally, any dynamically adaptive code should properly initialise new vertices.
In the present example, a proper initialisation could rely on $d$-linear
interpolation again.
For multigrid, higher order schemes are desireable (though in practice I again
observed that linear interpolation is sufficient), but higher order
interpolation then has to be realised manually with helper variables or some
additional code applying the stencil to newly generated vertices.
We stick to the linear case here:
\begin{code}
void ...::createInnerVertex(...) {
  fineGridVertex.setU(
    _multigrid.getDLinearInterpolatedValue(
      VertexOperations::readU( coarseGridVerticesEnumerator, coarseGridVertices ),
      fineGridPositionOfVertex
    )
  );
}
\end{code}

\begin{center}
  \includegraphics[width=0.34\textwidth]{42_matrix-free-multigrid/DynamicB.png}
  \includegraphics[width=0.45\textwidth]{42_matrix-free-multigrid/MultiLevel.png}
\end{center}

\noindent
The right image above presents the result of the multilevel plotter:
Each grid resolution level is written to a separate file.
It is obvious that finer grid solution approximations overwrite coarser vertex
values, while the coarse values determine the hanging node values.
We also see that the grid is refined towards the boundary here.

\subsection{Multigrid}

We start with some metaphilosophical considerations.
Multigrid algorithms work in two types of spaces: the ansatz
(discretisation) space holding the solution and a set of correction spaces.
The latter either are updated all at once (additive multigrid) or one after
another (multiplicative) before their contribution is added back to the actual
representation.
In the present discussion, we restrict ourselves to geometrically inspired
multigrid where both the correction and the discretisation space are spanned by
the spacetree's grid levels.
On the previous pages, we have realised a Jacobi directly within the
discretisation space without assembling any bigger matrix.
Obviously, such an approach also works for the correction equations.

At first glance, we then however need two variable sets per vertex: one set for
the actual discretisation and one for the correction equations.
As long as we work on regular grids only, there is no need to store different
types of variables within the vertices: refined vertices hold correction
equations, fine grid vertices hold the discretisation.
This approach becomes problematic for adaptive grids.
Still, unrefined vertices of the fine grid hold the discretised PDE.
On the same level of an arbitrary unrefined level, there might however also be
an area that is refined further.
Here, we solve a correction equation.
If we realised both the correction equation and the discretisation with the same
set of variables, what would the right vertex values at the transition between
unrefined and correction areas be?
We have to distinguish carefully in each cell which type of equation we are
solving, unless \ldots

\ldots we solve also the correction equation in a discretisation space. This is
the idea of full approximation storage (FAS) which fits perfectly to our
injection that we realised anyway. Instead of solving
\[
  A_{3h} e = \hat f
\]
in areas that are refined further and thus hold a correction to the actual
solution iterate, we solve
\begin{eqnarray*}
  A_{3h} \left( Iu_h + e \right)  & = & \hat f +  A_{3h} Iu_h \qquad \mbox{added
  an additional term on both sides} \\
  & = & R r_h + RA_hPIu_h \\
  & = & R \left( r_h + A_hPIu_h \right) \\
  & = & R \left( f_h - A_hu_h + A_hPIu_h \right) \\
  & = & R \left( f_h - A_h\left( id - PI\right)u_h \right) \\
  & =: & R \left( f_h - A_h \hat u_h \right)  =: R \hat r_h
\end{eqnarray*}
i.e.~work with a coarsened solution as input to the correction solve
which is a formalism introduced by Griebel as HTMG.

So we can stick to our injections of the solution.
In each traversal, we have to determine the hierarchical surplus in each vertex
$\hat u = u - PIu_h$ which is simple, as $Iu_h$ is available anyway.
This hierarchical transform is a temporary helper.
We can throw it away after the traversal again.
We can model it as \texttt{discard} in the \texttt{Vertex.def}.

Once we have $\hat u$, we can simultaneously determine the residual $r$ as well
as its hierarchical counterpart $\hat r$.
With the residual $r$, we update the solution if we have to---a detail subject
of discussion in a minute when we finalise whether to realise a multiplicative
or additive scheme.
The hierarchical residual $\hat r$ in turn is not used to update any solution
directly.
It is restricted to the next coarser level to yield a right-hand side there.

We finally observe that we may not just update the solution with the residual.
If we did that we would loose the knowledge which correction to prolong to the
next finer level.
We therefore store the updates in another helper variable and use this update to
prolong it to a finer level later on.

\begin{remark}
  The update helper variable is not the only valid choice to
  distinguish coarse grid updates from fine grid representation. It is obviously
  that we also might store the hierarchical surplus persistently and then
  reconstruct the nodal value with the relation $u_h=\hat u_h+Pu_{3h}$. This is
  the implementation variant discussed in (Weinzierl:09), while a more detailed
  analysis of required helper variables for additive multigrid can be found in
  (Reps:16).
\end{remark}

\begin{code}
class multigrid::records::Vertex {
  persistent parallelise double  u;             // Solution
  persistent parallelise double  f;             // Rhs
  discard parallelise double     r;             // Residual
  discard parallelise double     d;             // Diagonal element
  discard parallelise double     hierarchicalU; // Hierarchical solution
  discard parallelise double     hierarchicalR; // Hierarchical residual
  persistent parallelise double  uUpdate;       // Update of solution
  discard parallelise double     linearSurplus[DIMENSIONS];

  enum VertexType {
    Unknown, Dirichlet, Neumann
  };
  persistent VertexType vertexType;
};
\end{code}

\noindent
The hierarchical surplus has to be determined in \texttt{touchVertexFirstTime}
where we also clear the temporary variables.
I decided to realise the generic multigrid operations in a mapping of its own.
It is called \texttt{HierarchicalTransformAndRHSRestriction}.
Once more, we rely on an instance of the Multigrid class.

\begin{code}
void
multigrid::mappings::HierarchicalTransformAndRHSRestriction::createHangingVertex(...) {
 fineGridVertex.clearHierarchicalValues();
}

void multigrid::mappings::HierarchicalTransformAndRHSRestriction::touchVertexFirstTime(...) {
 fineGridVertex.clearHierarchicalValues();

 if ( fineGridVertex.isInside() ) {
   const tarch::la::Vector<TWO_POWER_D,double > u_3h  =
     VertexOperations::readU(coarseGridVerticesEnumerator,coarseGridVertices);
   const tarch::la::Vector<TWO_POWER_D,double > e_3h  =
     VertexOperations::readUUpdate(coarseGridVerticesEnumerator,coarseGridVertices);
   const double                                 Pu_3h =
     _multigrid.getDLinearInterpolatedValue(u_3h,fineGridPositionOfVertex);
   const double                                 Pe_3h =
     _multigrid.getDLinearInterpolatedValue(e_3h,fineGridPositionOfVertex);

   fineGridVertex.correctU(Pe_3h);
   fineGridVertex.determineUHierarchical(Pu_3h);

   if (fineGridVertex.getRefinementControl()!=Vertex::Records::Unrefined) {
     fineGridVertex.clearF();
   }
 }

 fineGridVertex.clearAccumulatedAttributes();
}

void multigrid::Vertex::determineUHierarchical(double Pu_3h) {
  _vertexData.setHierarchicalU( _vertexData.getU()-Pu_3h );
}
\end{code}

\begin{remark}
  In this presentation, we make the solvers restrict residuals on all levels all
  the time and prolong all updates all the time.
  The actual distinction between additive and multiplicative multigrid stems from the decision which
  level is smoothed, i.e.~where we find an update. This is
  highly inefficient.
  Good solvers would compute only those residuals and right-hand sides that have changed and
  are required. But the purpose of this presentation is to introduce a quick,
  brief prototype. Speed is a different topic.
\end{remark}

\noindent
The accumulation of the hierarchical surplus itself is more or less a
cut-n-paste from the Jacobi smoother:
\begin{code}

void multigrid::mappings::HierarchicalTransformAndRHSRestriction::enterCell(...) {
  const tarch::la::Vector<TWO_POWER_D,double> hierarchicalU    =
     VertexOperations::readHierarchicalU( fineGridVerticesEnumerator, fineGridVertices );
   const tarch::la::Vector<TWO_POWER_D,double> hierarchicalROld =
     VertexOperations::readHierarchicalR( fineGridVerticesEnumerator, fineGridVertices );
   const matrixfree::stencil::ElementWiseAssemblyMatrix A =
     fineGridCell.getElementsAssemblyMatrix( fineGridVerticesEnumerator.getCellSize() );

   tarch::la::Vector<TWO_POWER_D,double> hierarchicalR =
     hierarchicalROld - A * hierarchicalU;

   VertexOperations::writeHierarchicalR
     ( fineGridVerticesEnumerator, fineGridVertices, hierarchicalR );
}
\end{code}

\noindent
For the restriction, we again rely on the multigrid object, where a specialised
prolongation function does exist that uses $d$-linear interpolation.
Furthermore, we stick to the convention $R=P^T$ and thus can write:

\begin{code}
void
multigrid::mappings::HierarchicalTransformAndRHSRestriction::touchVertexLastTime(...) {
  if ( fineGridVertex.isInside() ) {
    const tarch::la::Vector<TWO_POWER_D, double > P =
      _multigrid.calculateP(fineGridPositionOfVertex);

    dfor2(k)
      // There is no need to exclude boundary points here (the rhs does not play
      // there a role anyway), but it makes the visualisation nicer.
      if (
        coarseGridVertices[ coarseGridVerticesEnumerator(k) ].getRefinementControl()==
          Vertex::Records::Refined
        &&
        coarseGridVertices[ coarseGridVerticesEnumerator(k) ].isInside()
      ) {
        coarseGridVertices[ coarseGridVerticesEnumerator(k) ].incF(
          P(kScalar) * fineGridVertex.getHierarchicalResidual()
        );
      }
    enddforx
  }
}

void multigrid::Vertex::incF(double value) {
  _vertexData.setF( _vertexData.getF()+value );
}

double multigrid::Vertex::getHierarchicalResidual() const {
  return _vertexData.getF() + _vertexData.getHierarchicalR();
}
\end{code}

\noindent
We close the preparatory work with the remark that the specification has to be
augmented by additional readers and writers to make our code snippets work:

\begin{code}
component: Multigrid

namespace: ::multigrid

vertex:
  dastgen-file: Vertex.def
  read scalar(double): U
  read scalar(double): F
  read scalar(double): R
  read scalar(double): D
  read scalar(double): HierarchicalU
  read scalar(double): HierarchicalR
  write scalar(double): U
  write scalar(double): R
  write scalar(double): D
  write scalar(double): HierarchicalU
  write scalar(double): HierarchicalR

...
\end{code}

\begin{center}
  \includegraphics[width=0.8\textwidth]{42_matrix-free-multigrid/Hierarchical-Jacobi.png}
\end{center}

\noindent
We see above a Jacobi smoother (100 iterations) for the Poisson equation on the
unit square (left) as well as the hierarchical representation (right).
We also may augment the adapters with plots of the temporary data

\begin{code}
adapter:
  name: AnyAdapter
  ...
  merge-with-predefined-mapping: VTKPlotVertexValue(u,getU,u)
  merge-with-predefined-mapping: VTKPlotVertexMultilevelValue(multiscaleU,getU,u)
  merge-with-predefined-mapping: VTKPlotVertexMultilevelValue
    (multiscaleHierarchicalU,getHierarchicalU,u)
  merge-with-predefined-mapping: VTKPlotVertexMultilevelPointCloud
    (multiscaleResidual,getResidual,res)
  merge-with-predefined-mapping: VTKPlotVertexMultilevelPointCloud
    (multiscaleHierarchicalResidual,getHierarchicalResidual,res)
  merge-with-predefined-mapping: VTKPlotVertexMultilevelPointCloud(f,getF,f)
\end{code}

\noindent
and emphasise that we basically have almost all multigrid ingredients at hands.
Missing is a projection from coarse grid updates to fine grid values.
So far, we update the coarse grid values according to the right-hand side.
These updates however are not projected.
Instead, the updated approximation in a FAS sense are overwritten at the end of
the subsequent iteration with fine grid solution data.

\begin{remark}
  Most default VTK plotters plot the grid as well as values and (for
  consistency reasons imposed by the VTK data file format) have to plug into
  \texttt{touchVertexFirstTime}. For residuals and other temporary data, we
  typically however are interested to plot the value at the end of the
  iteration. A quick solution here is to plot the grid with one mapping and to
  use a second mapping that solely plots a point cloud with the vertex values.
  We then can visualise both data sets simultaneously.
\end{remark}

\subsection{Additive Geometric Multigrid and BPX}

Additive multigrid is relatively simple to realise as it fits naturally to our
tree traversals.
The most important and useful idea hereby is that you might want to shift the
actual multigrid cycle by half a tree traversal:
your start with the prolongation of the error terms in the very first grid sweep
(they are zero at this time) and you then trigger all the fine-to-coarse
operations in the remainder of the sweep.
It is not before the second grid sweep that the corrections introduced by the
fine-to-coarse operators are then added again to the finest grid solution.
So for $n$ additive multigrid sweeps, you need $n+1$ grid sweeps.

%  In additive multigrid, we may not use the same relaxation
% factor on each and every level.
% Instead, it is important that the coarser the level the smaller the contribution
% to the solution update.
% Otherwise, the solver tends to overshoot and become unstable.
% We propose to use relaxation factors $\omega ^k$ with $k=1$ on the fine grid
% level and increasing by one for each level up the hierarchy.
% $\omega $ obviously has to be adopted if the grid changes.
% It is thus straightforward to compute it on-the-fly.
% For this, we add an additional variable to the vertex
%
% \begin{code}
% class multigrid::records::Vertex {
%   persistent parallelise int  numberOfFinerLevelsAtSamePosition;
% };
% \end{code}
%
% \noindent
% and make the grid traversal determine the correct value of this attribute
% on the fly:
%
%
% \begin{code}
% void multigrid::Vertex::clearAccumulatedAttributes() {
%   _vertexData.setR(0.0);
%   _vertexData.setD(0.0);
%
%   _vertexData.setNumberOfFinerLevelsAtSamePosition( 1 );
% }
%
%
% void multigrid::mappings::AdditiveMGProlongation::touchVertexFirstTime(
%   multigrid::Vertex&               fineGridVertex,
%   const tarch::la::Vector<DIMENSIONS,double>&                          fineGridX,
%   const tarch::la::Vector<DIMENSIONS,double>&                          fineGridH,
%   multigrid::Vertex * const        coarseGridVertices,
%   const peano::grid::VertexEnumerator&                coarseGridVerticesEnumerator,
%   multigrid::Cell&                 coarseGridCell,
%   const tarch::la::Vector<DIMENSIONS,int>&                             fineGridPositionOfVertex
% ) {
%   logTraceInWith6Arguments( "touchVertexFirstTime(...)", fineGridVertex, fineGridX, fineGridH, coarseGridVerticesEnumerator.toString(), coarseGridCell, fineGridPositionOfVertex );
%
%   if (fineGridVertex.isInside()) {
%     const tarch::la::Vector<TWO_POWER_D,double > e_3h  = VertexOperations::readUUpdate(coarseGridVerticesEnumerator,coarseGridVertices);
%     const double                                 Pe_3h = _multigrid.getDLinearInterpolatedValue(e_3h,fineGridPositionOfVertex);
%
%     fineGridVertex.correctU(
%       Pe_3h
%       *
%       fineGridVertex.getDampingFactorForAdditiveCoarseGridCorrection(multigrid::mappings::JacobiSmoother::omega)
%     );
%   }
%
%   logTraceOutWith1Argument( "touchVertexFirstTime(...)", fineGridVertex );
% }
% \end{code}
%
%
% Update doch spezifisch
%
%
%
% inject ein bischen falsch. Eigentlich besser R.
%
% @todo Instabil. Relaxation mit aufnehmen
%
% @todo Kleiner Fehler mit der Injektion. Auf Bram verweisen, aber so lassen

To figure exactly out how information propagates through the grid levels is
straightforward as long as no adaptivity is involved.
Some tricks how to handle adaptivity are discussed in

 \begin{framed}
 \noindent
Reps, Bram and Weinzierl, Tobias: {\em Complex additive geometric
  multilevel solvers for Helmholtz equations on spacetrees}, ACM Transactions on
  Mathematical Software (TOMS), 44(1), 2017, 2:1--2:36
 \end{framed}

\noindent
A preprint is available as arXiv tech report (arXiv:1508.03954).
The report also discusses the extension of an additive multigrid solver into
BPX.
All code for this paper is available through the author's University webpage.

\subsection{Multiplicative Geometric Multigrid}

Multiplicative multigrid is perhaps easier to realise than an additive multigrid
algorithm. I usually run through the following steps:

\begin{itemize}
  \item I augment the state (\texttt{State.def}) with the current smoother step
  and an enum whether I'm currently smoothing, ascending or descending.
  \item The runner triggers one grid sweep per multigrid step and increments and
  decrements the smoothing levels. It also changes the state's smoothing state.
  \item I typically merge the restriction with the smoothing step on the next
  coarser level. Smoothing steps can accumulate the coarse grid residuals and
  need typically the new right-hand side not before they are about the touch a
  vertex for the very last time. So we can do both coarse grid residual
  accumulation and fine grid rhs computation in one mapping.
  \item I typically merge prolongation with the first smoothing step on the next
  finer grid. If the prolongation is implemented within
  \texttt{touchVertexFirstTime}, the actual prolongation is a preamble to the
  smoother and does not require a grid run-through of its own.
\end{itemize}

% \begin{remark}
% There is
% \end{remark}

Details on efficient realisations, the extension to algebraic operators,
as well as better smoothers are subject of dicussion in

 \begin{framed}
 \noindent
Weinzierl, Marion and Weinzierl, Tobias: {\em Quasi-matrix-free hybrid multigrid
on dynamically adaptive Cartesian grids}, arXiv:1607.00648
 \end{framed}

\noindent
All important source code fragments required to follow the techniques presented
therein are available through the matrixfree toolbox in the Peano repository.

\subsection*{Further reading}

\begin{itemize}
  \item Weinzierl, Marion and Weinzierl, Tobias: {\em Quasi-matrix-free hybrid
  multigrid on dynamically adaptive Cartesian grids}, arXiv:1607.00648
  \item  Reps, Bram and Weinzierl, Tobias: {\em Complex additive geometric
  multilevel solvers for Helmholtz equations on spacetrees}, ACM Transactions on
  Mathematical Software (TOMS), 44(1), 2017, 2:1--2:36, arXiv:1508.03954
  \item Weinzierl, Marion: {\em Hybrid Geometric-Algebraic Matrix-Free Multigrid on
Spacetrees}, Dissertation, Technische Universit\"at M\"unchen, 2013
  \item   Muntean, Ioan Lucian, Mehl, Miriam, Neckel, Tobias and Weinzierl,
  Tobias (2008). {\em Concepts for Efficient Flow Solvers Based on Adaptive
  Cartesian Grids}. In High Performance Computing in Science and Engineering,
  Garching 2007. Wagner, Siegfried, Steinmetz, Matthias, Bode, Arndt and Brehm,
  Matthias Berlin Heidelberg New York: Springer.
  \item Weinzierl, Tobias and K\"oppl, Tobias (2012). {\em A Geometric
  Space-time Multigrid Algorithm for the Heat Equation}. Numerical Mathematics:
  Theory, Methods and Applications 5(1): 110-130.
  \item Mehl, Miriam, Weinzierl, Tobias and Zenger, Christoph (2006). {\em A
  cache-oblivious self-adaptive full multigrid method}. Numerical Linear Algebra
  with Applications 13(2-3): 275-291.
\end{itemize}


\chapter{Categorical Data}\label{Ch.2}

Recall from Chapter 1 that a \textbf{categorical variable} is a variable that can take on one of a limited set of values (which are called \textit{levels}). In this chapter, we will discuss ways to analyze categorical variables. Throughout this chapter, we use the Titanic data set as a working example. This data set contains information about all of the people who were aboard the RMS Titanic when it sank in 1912, including both passengers and crew. Let's start by reading in this data set.

\begin{lstlisting}[language=Python]
import pandas as pd

data_dir = "http://dlsun.github.io/pods/data/"
df_titanic = pd.read_csv(data_dir + "titanic.csv")
df_titanic.head()
\end{lstlisting}

\small\begin{verbatim}
                             name  gender   age class embarked        country  \
0             Abbing, Mr. Anthony    male  42.0   3rd        S  United States
1       Abbott, Mr. Eugene Joseph    male  13.0   3rd        S  United States
2     Abbott, Mr. Rossmore Edward    male  16.0   3rd        S  United States
3  Abbott, Mrs. Rhoda Mary 'Rosa'  female  39.0   3rd        S        England
4     Abelseth, Miss. Karen Marie  female  16.0   3rd        S         Norway

   ticketno   fare  survived
0    5547.0   7.11         0
1    2673.0  20.05         0
2    2673.0  20.05         0
3    2673.0  20.05         1
4  348125.0   7.13         1
\end{verbatim}

The categorical variables in this data set are:
\begin{itemize}
\item
\verb|gender| (male or female)

\item
\verb|class|: what class they were in (1st, 2nd, or 3rd) or what type of crew member they were

\item
\verb|embarked|: where they embarked (Belfast, Southampton, Cherbourg, or Queenstown)

\item
\verb|country|: their country of origin

\item
\verb|ticketno|: the ticket number

\item
\verb|survived|: whether or not they survived the disaster

\end{itemize}

Note that \verb|age| and \verb|fare| are quantitative variables. It is tempting to consider \verb|name| a categorical variable, but it is not, since (almost) every person has a unique name. In order for a variable to be categorical, it must take on a \textit{limited} set of values, ideally with each level appearing multiple times in the data set. Otherwise, the analyses that we describe in this chapter will not be very meaningful.

\section{Analyzing One Categorical Variable}\label{2.1}

In this lesson, we focus on a single categorical variable. For a high-level summary of a categorical variable, we can use the \verb|.describe()| command. Note that the behavior of \verb|.describe()| will change, depending on whether \verb|pandas| thinks that the variable is quantitative or categorical, which is why it is important to cast categorical variables to the right type, as we discussed in the previous chapter.

\begin{lstlisting}[language=Python]
df_titanic["class"].describe()
\end{lstlisting}

To completely summarize a single categorical variable, we report the number of times each level appeared, or its \textbf{frequency}.

\begin{lstlisting}[language=Python]
class_counts = df_titanic["class"].value_counts()
class_counts
\end{lstlisting}

Notice that the levels are sorted in decreasing order of frequency by default. We can also report the levels in the order they appear in the data set...

\begin{lstlisting}[language=Python]
df_titanic["class"].value_counts(sort=False)
\end{lstlisting}

...or in alphabetical order, by sorting the index:

\begin{lstlisting}[language=Python]
class_counts.sort_index()
\end{lstlisting}

Note that this produces a \textit{new} \verb|Series| with the index sorted. It does not sort the original \verb|Series| \verb|class_counts|. To sort the original series, we need to specify that the sorting should be done in place, just like we did for \verb|.set_index()| in Chapter 1.

\begin{lstlisting}[language=Python]
class_counts.sort_index(inplace=True)
class_counts
\end{lstlisting}

Any other order would require selecting the levels manually, in the desired order.

\begin{lstlisting}[language=Python]
class_counts.loc[
    ["1st", "2nd", "3rd",
     "deck crew", "engineering crew", "victualling crew",
     "restaurant staff"]
]
\end{lstlisting}

This information can be visualized using a \textbf{bar chart}.

\begin{lstlisting}[language=Python]
class_counts.plot.bar()
\end{lstlisting}

Instead of reporting counts, we can also report proportions or probabilities, or the \textbf{relative frequencies}. We can calculate the relative frequencies by specifying \verb|normalize=True| in \verb|.value_counts()|.

\begin{lstlisting}[language=Python]
class_probs = df_titanic["class"].value_counts(normalize=True)
class_probs.sort_index()
\end{lstlisting}

This is equivalent to taking the counts and dividing by their sum.

\begin{lstlisting}[language=Python]
class_counts / class_counts.sum()
\end{lstlisting}

Notice that the relative frequencies add up to 1.0, by construction. We can report these relative frequencies using probability notation. For example:

$$ P(\text{1st class}) = 0.146806. $$

The complete collection of probabilities of all levels of a variable is called the \textbf{distribution} of that variable. So the code above calculates the distribution of "class" on the Titanic.

The bar chart for relative frequencies (i.e., probabilities) looks qualitatively the same as the bar chart for frequencies (i.e., counts). The only difference is that the scale on the $y$-axis is different.

\begin{lstlisting}[language=Python]
class_probs.sort_index().plot.bar()
\end{lstlisting}

In the next lesson, we will see why it is often more useful to plot the relative frequencies (i.e., probabilities) rather than the frequencies (i.e., counts).

\subsection{Transforming Categorical Variables}\label{2.1.1}

A categorical variable can be transformed by mapping its levels to new levels. For example, we may only be interested in whether a person on the titanic was a passenger or a crew member. The variable \verb|class| is too detailed. We can create a new variable, \verb|type|, that is derived from the existing variable \verb|class|. Observations with a \verb|class| of "1st", "2nd", or "3rd" get a value of "passenger", while observations with a \verb|class| of "deck crew", "engineering crew", or "deck crew" get a value of "crew".

\begin{lstlisting}[language=Python]
df_titanic["type"] = df_titanic["class"].map({
    "1st": "passenger",
    "2nd": "passenger",
    "3rd": "passenger",
    "victualling crew": "crew",
    "engineering crew": "crew",
    "deck crew": "crew"
})

df_titanic
\end{lstlisting}

Upon closer inspection of this \verb|DataFrame|, we see that we accidentally left out the level "restaurant staff" in the input to \verb|.map()|. Any levels that are unspecified will be mapped to the missing value \verb|NaN|.

This suggests a more concise way to define the new variable \verb|type|. We can specify only the levels for passengers in the mapping and then fill in the missing values afterwards.

\begin{lstlisting}[language=Python]
df_titanic["type"] = df_titanic["class"].map({
    "1st": "passenger",
    "2nd": "passenger",
    "3rd": "passenger"
})

# Replace all missing values by "crew"
df_titanic["type"].fillna("crew", inplace=True)

df_titanic
\end{lstlisting}

For more complex mappings, the \verb|.map()| method also accepts a function. So the above

\begin{lstlisting}[language=Python]
def class_to_type(c):
  if c in ["1st", "2nd", "3rd"]:
    return "passenger"
  else:
    return "crew"

df_titanic["class"].map(class_to_type)
\end{lstlisting}

We can apply the techniques we learned above to calculate the \textit{distribution} of this new variable, which only has two levels.

\begin{lstlisting}[language=Python]
df_titanic["type"].value_counts(normalize=True)
\end{lstlisting}

\subsection{Conditional Probabilities}\label{2.1.2}

What fraction of males were crew members? To answer questions like this, we have to filter the \verb|DataFrame| to include only males.

The standard way to filter a \verb|DataFrame| is to use a \textbf{boolean mask}. A boolean mask is simply a \verb|Series| of booleans whose index matches the index of the \verb|DataFrame|.

The easiest way to create a boolean mask is to use one of the standard comparison operators \verb|==|, \verb|<|, \verb|>|, and \verb|!=| on an existing column in the \verb|DataFrame|. For example, the following code produces a boolean mask that is equal to \verb|True| for the males on the Titanic and \verb|False| otherwise.

\begin{lstlisting}[language=Python]
df_titanic["gender"] == "male"
\end{lstlisting}

Notice the subtle way the equality operator \verb|==| is being used here! We are comparing an array with a string, i.e.,

\begin{align}
& \begin{bmatrix} \text{"male"} \\ \text{"male"} \\ \text{"male"} \\ \text{"female"} \\ \vdots \\ \text{"male"} \end{bmatrix} & \text{with} & & \text{"male"}.
\label{broadcast1}\end{align}

In most programming languages, this comparison would make no sense. An array of strings is obviously not equal to a string $\eqref{broadcast1}$. However, \verb|pandas| automatically applies the equality operator to \textit{each} element of the array. As a result, we get an entire array (i.e., a \verb|Series|) of booleans.

\begin{align}
\begin{bmatrix} \text{"male"} \\ \text{"male"} \\ \text{"male"} \\ \text{"female"} \\ \vdots \\ \text{"male"} \end{bmatrix} &== \text{"male"} &\Longrightarrow & &  \begin{bmatrix} \text{True} \\ \text{True} \\ \text{True} \\ \text{False} \\ \vdots \\ \text{True} \end{bmatrix}.
\label{broadcast2}\end{align}

When an operation is applied to each element of an array, it is said to be \textbf{broadcast} over that array $\eqref{broadcast2}$.

Now, we can use the boolean mask as a filter on the \verb|DataFrame| to extract the rows where the mask equals \verb|True|.

\begin{lstlisting}[language=Python]
df_male = df_titanic[df_titanic["gender"] == "male"]
df_male
\end{lstlisting}

Note that every person in this new \verb|DataFrame| is male. If you inspect the index, you will see that rows 4 and 5 are missing. That is because those passengers were female.

Finally, we can apply the methods we've learned in this lesson to \verb|df_male| to answer the original question: "What fraction of males were crew members?"

\begin{lstlisting}[language=Python]
df_male["type"].value_counts(normalize=True)
\end{lstlisting}

It appears that about \verb|50.4657%| of the males were crew members. We can notate this using \textit{conditional probability} notation:

$$ P(\text{crew} | \text{male}) = 0.504657. $$

The bar $|$ is read as "given". The information after the bar is the given information. In this case, we were interested in the probability a person was a crew member, "given" they were male. That is, after restricting to the male passengers (i.e., using a boolean mask as a filter), we want to know the relative frequency of crew members.

We can also filter on multiple criteria. For example, if we want to know the fraction of male \textit{survivors} who were crew members, we need to combine two boolean masks, one based on the column \verb|gender| and another based on the column \verb|survived|. The two masks can be combined using the logical operator \verb|\&|.

\begin{lstlisting}[language=Python]
(df_titanic["gender"] == "male") & (df_titanic["survived"] == 1)
\end{lstlisting}

Notice that the logical operator was \textit{broadcast} (that word again!) over the elements of the two \verb|Series|. In other words, the logical operator was applied to each element, producing a \verb|Series| of booleans.

Now we can use this new boolean mask to filter the \verb|DataFrame|, just as we did before.

\begin{lstlisting}[language=Python]
df_male_survivors = df_titanic[(df_titanic["gender"] == "male") & (df_titanic["survived"] == 1)]
df_male_survivors
\end{lstlisting}

Besides $\verb|&|$, there are two other logical operators that can be used to modify and combine boolean masks.
\begin{itemize}
\item
$\verb|&|$ means "and"

\item
$\verb|||$ means "or"

\item
$\verb|~|$ means "not"

\end{itemize}

Like $\verb|&|$, the operators $\verb|||$ and $\verb|~|$ are broadcast over the boolean masks.

\subsection{Exercises}\label{2.1.3}

\textit{Exercises 1-2 ask you to continue working with the Titanic data set explored in this lesson.}

1. What proportion of crew members were male? Express your answer using probability notation. How does this proportion differ from the proportion that was calculated in the lesson?

2. What is the distribution of gender among passengers on the Titanic? What is the distribution of gender among passengers on the Titanic who survived? Express your answers using probability notation.

\textit{Exercises 3-5 deal with the OKCupid data set, which consists of user profiles in the San Francisco Bay Area on the dating website OKCupid. This data set is available at the URL https://dlsun.github.io/pods/data/okcupid.csv.}

3. Make a visualization of the distribution of drinking status.

4. Create a new variable that indicates whether the user has, likes, or dislikes cats. Visualize the distribution of this variable.

5. If you were a heterosexual female interested in dating a non-smoker, how many options would you have in this data set?

\section{Two-Way Tables and Two Categorical Variables}\label{2.2}

Data science is all about relationships between variables. How do we summarize and visualize the relationship between two categorical variables?

For example, what can we say about the relationship between gender and survival on the Titanic?

\begin{lstlisting}[language=Python]
import pandas as pd
data_dir = "http://dlsun.github.io/pods/data/"
df_titanic = pd.read_csv(data_dir + "titanic.csv")
\end{lstlisting}

We can summarize each variable individually like we did in the previous lesson.

\begin{lstlisting}[language=Python]
df_titanic["gender"].value_counts()
\end{lstlisting}

\begin{lstlisting}[language=Python]
df_titanic["survived"].value_counts()
\end{lstlisting}

But this does not tell us how gender interacts with survival. To do that, we need to produce a \textit{cross-tabulation}, or "cross-tab" for short. (Statisticians tend to call this a \textit{contigency table} or a \textit{two-way table}.)

\begin{lstlisting}[language=Python]
pd.crosstab(df_titanic["survived"], df_titanic["gender"])
\end{lstlisting}

A cross-tabulation of two categorical variables is a two-dimensional array, with the levels of one variable along the rows and the levels of the other variable along the columns. Each cell in this array contains the number of observations that had a particular combination of levels. So in the Titanic data set, there were 359 females who survived and 1366 males who died. From the cross-tabulation, we can see that there were more females who survived than not, while there were more males who died than not. Clearly, gender had a strong influence on survival because of the Titanic's policy of "women and children first".

To get probabilities instead of counts, we specify \verb|normalize=True|.

\begin{lstlisting}[language=Python]
joint_survived_gender = pd.crosstab(df_titanic["survived"], df_titanic["gender"],
                                    normalize=True)
joint_survived_gender
\end{lstlisting}

Notice that the four probabilities in this table add up to 1.0. Each of these probabilities is called a joint probability and can be notated, for example, as

$$ P(\text{female}, \text{died}) = 0.058903.$$

Collectively, these probabilities make up the \textit{joint distribution} of the variables \textbf{survived} and \textbf{gender}.

\subsection{Marginal Distributions}\label{2.2.1}

Is it possible to recover the distribution of \textbf{gender} alone from the joint distribution of \textbf{survived} and \textbf{gender}?

Yes! We simply sum the probabilities for each \textbf{gender} over all the possible levels of \textbf{survived}.

\begin{align}
P(\text{female}) = P(\text{female}, \text{died}) + P(\text{female}, \text{survived}) &= 0.058903 + 0.162664 = 0.221567 \\
P(\text{male}) = P(\text{male}, \text{died}) + P(\text{male}, \text{survived}) &= 0.618940 + 0.159493 = 0.778433\end{align}

In code, this can be achieved by summing the \verb|DataFrame| \textit{over} one of the dimensions. We can specify which dimension to sum over, using the \verb|axis=| argument to \verb|.sum()|.
\begin{itemize}
\item
\verb|axis=0| refers to the rows. In the current example, \textbf{survived} is the variable along this axis.

\item
\verb|axis=1| refers to the columns. In the current example, \textbf{gender} is the variable along this axis.

\end{itemize}

Since we want to sum \textit{over} the \textbf{survived} variable, we specify \verb|.sum(axis=0)|.

\begin{lstlisting}[language=Python]
gender = joint_survived_gender.sum(axis=0)
gender
\end{lstlisting}

When calculated from a joint distribution, the distribution of one variable is called a \textit{marginal distribution}. So the above is the marginal distribution of \textbf{gender}.

The name "marginal distribution" comes from the fact that it is customary to write these totals in the \textit{margins} of the table. In fact \verb|pd.crosstab()| has an argument \verb|margins=| that automatically adds these margins to the cross-tabulation.

\begin{lstlisting}[language=Python]
pd.crosstab(df_titanic["survived"], df_titanic["gender"],
            normalize=True, margins=True)
\end{lstlisting}

While the margins are useful for display purposes, they actually make computations more difficult, since it is easy to mix up which numbers correspond to joint probabilities and which ones correspond to marginal probabilities.

Likewise, to obtain the marginal distribution of \textbf{survived}, we sum over the possible levels of \textbf{gender} (which is the variable along \verb|axis=1|).

\begin{lstlisting}[language=Python]
survived = joint_survived_gender.sum(axis=1)
survived
\end{lstlisting}

We can check this answer by calculating the distribution of \textbf{survived} directly from the original data, using the techniques from the previous lesson.

\begin{lstlisting}[language=Python]
df_titanic["survived"].value_counts(normalize=True)
\end{lstlisting}

\subsection{Conditional Distributions}\label{2.2.2}

Let's take another look at the joint distribution of \textbf{survived} and \textbf{gender}.

\begin{lstlisting}[language=Python]
joint_survived_gender
\end{lstlisting}

From the joint distribution, it is tempting to conclude that females and males did not differ too much in their survival rates, since

$$ P(\text{female}, \text{survived}) = 0.162664 $$

is not too different from

$$ P(\text{male}, \text{survived}) = 0.159493. $$

This is because there were 359 women and 352 men who survived, out of 2207 passengers.

But this is the wrong comparison. The joint probabilities are affected by the baseline gender probabilities, and over three-quarters of the people aboard the Titanic were men. $P(\text{male}, \text{survived})$ and $ P(\text{female}, \text{survived})$ should not even be close if men were just as likely to survive as women, simply because of the sheer number of men aboard.

A better comparison is between the conditional probabilities. We ought to compare

$$ P(\text{survived} | \text{female}) $$

to

$$ P(\text{survived} | \text{male}). $$

To calculate each conditional probability, we simply divide the joint probability by the marginal probability. That is,

\begin{align}
P(\text{survived} | \text{female}) = \frac{P(\text{female}, \text{survived})}{P(\text{female})} &= \frac{0.162664}{0.221568} = .7341 \\
P(\text{survived} | \text{male}) = \frac{P(\text{male}, \text{survived})}{P(\text{male})} &= \frac{0.159493}{0.778432} = .2049\end{align}

The conditional probabilities expose the stark difference in survival rates. One way to think about conditional probabilities is that they \textit{adjust} for the baseline gender probabilities. By dividing by $P(\text{male})$ and $P(\text{female})$, we adjust for the fact that there were more men and fewer women on the Titanic, thus enabling an apples-to-apples comparison.

In code, this can be achieved by dividing the joint distribution by the marginal distribution (of \textbf{gender}). However, we have to be careful:
\begin{itemize}
\item
The joint distribution is a two-dimensional array. It is stored as a \verb|DataFrame|.

\item
The marginal distribution (of \textbf{gender}) is a one-dimensional array. It is stored as a \verb|Series|.

\end{itemize}

How is it possible to divide a two-dimensional object by a one-dimensional object? Only if we \textit{broadcast} the one-dimensional object over the other dimension. A toy example is illustrated below.

\begin{align}
\begin{bmatrix} 1 & 2 \\ 3 & 4 \\ 5 & 6 \end{bmatrix} \Big/ \begin{bmatrix} 7 \\ 8 \end{bmatrix} &= \begin{bmatrix} 1 & 2 \\ 3 & 4 \\ 5 & 6 \end{bmatrix}  \Big/ \begin{bmatrix} 7 & 8 \\ 7 & 8 \\ 7 & 8 \end{bmatrix} \\
&= \begin{bmatrix} 1/7 & 2/8 \\ 3/7 & 4/8 \\ 5/7 & 6/8 \end{bmatrix}\end{align}

To do this in \verb|pandas|, we use the \verb|.divide()| method, specifying the dimension on which to align the \verb|Series| with the \verb|DataFrame|. Since \textbf{gender} is on \verb|axis=1| of \verb|joint_survived_gender|, we align the \verb|DataFrame| and \verb|Series| along \verb|axis=1|.

\begin{lstlisting}[language=Python]
cond_survived_gender = joint_survived_gender.divide(gender, axis=1)
# In this case, joint_survived_gender / gender would also haved worked,
# but better to play it safe and be explicit about the axis.

cond_survived_gender
\end{lstlisting}

Every probability in this table represents a conditional probability of gender given survival status. So from the table, we can read that

$$ P(\text{survived} | \text{female}) = 0.734151. $$

Notice that each row sums to $1.0$---as it must, since given the information that a person was female, there are only two possibilities: she either survived or died.

In other words, we have a distribution of \textbf{survived} for each level of \textbf{gender}. We might wish to compare these two distributions. When we call \verb|.plot.bar()| on the \verb|DataFrame|, it will plot the values in each column as a set of bars with its own color.

\begin{lstlisting}[language=Python]
cond_survived_gender.plot.bar()
\end{lstlisting}

A different way to visualize a conditional distribution is to use a stacked bar graph. Here, we want one bar for females and another for males, each one divided in proportion to the survival rates for that gender. First, let's take a look at the desired graph.

\begin{lstlisting}[language=Python]
cond_survived_gender.T.plot.bar(stacked=True)
\end{lstlisting}

Now, let's unpack the code that generated this graphic. Recall that \verb|.plot.bar()| plots each column of a \verb|DataFrame| in a different color. Here we want different colors for each level of \textbf{survived}, so we need to swap the rows and columns of \verb|cond_survived_gender|. In other words, we need the \textit{transpose} of the \verb|DataFrame|, which is accomplished using \verb|.T|.

\begin{lstlisting}[language=Python]
cond_survived_gender.T
\end{lstlisting}

When we call \verb|.plot.bar()| on this transposed \verb|DataFrame|, with \verb|stacked=True|, we obtain the stacked bar graph above.

\subsection{Exercises}\label{2.2.3}

Exercises 1-4 ask you to continue working with the Titanic data set explored in this lesson.

1. Filter the data to include passengers only. Calculate the joint distribution between a passenger's class and where they embarked.

2. Using the joint distribution that you calculated in Exercise 1, calculate the following:
\begin{itemize}
\item
the conditional distribution of their class given where they embarked

\item
the conditional distribution of where they embarked given their class

\end{itemize}

Use the conditional distributions that you calculate to answer the following questions:
\begin{itemize}
\item
What proportion of 3rd class passengers embarked at Southampton?

\item
What proportion of Southampton passengers were in 3rd class?

\end{itemize}

3. Make a visualization showing the distribution of a passenger's class, given where they embarked.

4. Compare the survival rates of crew members versus passengers. Which group appears to survive at higher rates?

(\textit{Hint:} You will have to transform the \textbf{class} variable to a variable that indicates whether a person was a passenger or a crew member. Refer to the previous lesson.)

Exercises 5-6 ask you to work with the Florida Death Penalty data set, which is available at  \verb|https://dlsun.github.io/pods/data/death_penalty.csv|. This data set contains information about the races of the defendant and the victim, as well as whether a death penalty verdict was rendered, in 674 homicide trials in Florida between 1976-1987.

5. Use the joint distribution to summarize the relationship between the defendant's and the victim's races in Florida homicides.

6. Does there appear to be a relationship between death penalty verdicts and the defendant's race? If so, in what direction?

\section{Multi-Way Tables and Simpson's Paradox}\label{2.3}

In the previous lesson, we summarized two categorical variables by cross-tabulating their frequencies.

\begin{lstlisting}[language=Python]
import pandas as pd
data_dir = "http://dlsun.github.io/pods/data/"
df_titanic = pd.read_csv(data_dir + "titanic.csv")

def class_to_type(c):
  if c in ["1st", "2nd", "3rd"]:
    return "passenger"
  else:
    return "crew"
df_titanic["type"] = df_titanic["class"].map(class_to_type)

joint_type_survived = pd.crosstab(
    df_titanic["type"],
    df_titanic["survived"],
    normalize=True
)
joint_type_survived
\end{lstlisting}

\small\begin{verbatim}
survived          0         1
type
crew       0.307657  0.095605
passenger  0.370186  0.226552
\end{verbatim}

Each number in this table represents a joint probability. For example:
$$ P(\text{crew}, \text{survived}) = 0.095605. $$

We might want to know whether crew members or passengers survived at higher rates. To do this, we have to compare the conditional probabilities
\begin{align}
P(\text{survived} | \text{crew}) & & \text{vs.} & & P(\text{survived} | \text{passenger}).\end{align}

In the last lesson, we learned to calculate conditional distributions using broadcasting.

\begin{lstlisting}[language=Python]
survived_given_type = joint_type_survived.divide(
    joint_type_survived.sum(axis=1),
    axis=0
)
survived_given_type
\end{lstlisting}

\small\begin{verbatim}
survived          0         1
type
crew       0.762921  0.237079
passenger  0.620349  0.379651
\end{verbatim}

From the table, it is apparent that passengers survived at much higher rates than crew members:
$$ P(\text{survived}|\text{crew}) = 0.237079 < 0.379651 = P(\text{survived}|\text{passenger}). $$

\subsection{Communication Corner: Reporting Differences in Probabilities}\label{2.3.1}

How do we report the difference between the two probabilities $23.71\%$ and $37.97\%$ above? There are a number of ways:
\begin{enumerate}
\item
As an \textit{additive change}: "Passengers were
$$ 37.97\% - 23.71\% = 14.26 \text{ percentage points} $$
more likely to survive than crew members."

\item
As a \textit{relative change} (or \textit{relative risk}): "Passengers were
$$ 37.97\% \big/ 23.71\% = 1.60 \text{ times} $$
as likely to survive as crew members."

\item
We can translate relative changes to \textit{percent changes} by subtracting $1$ and multiplying by $100\%$. So we can rephrase the above as: "Passengers were
$$ 100\% \times (1.60 - 1.00) = 60\% $$
more likely to survive than crew members."

\item
As an \textit{odds ratio}: "The odds of a passenger surviving was
$$ \frac{37.97\% \big/ (100\% - 37.97\%)}{23.71\% \big/ (100\% - 23.71\%)} = 1.97 \text{ times} $$
greater than the odds of a crew member surviving."

\end{enumerate}

Note that additive changes and percent changes should be compared to a baseline of 0.0 (and can be negative), while relative changes and odds ratios should be compared to a baseline of 1.0 (and cannot be negative).

Watch out: it is incorrect to say that passengers are $14.26\%$ more likely to survive than crew members, since the percent change is $60\%$. An additive change should always be reported in units of "percentage points".

\subsection{Controlling for a Variable}\label{2.3.2}

But is this the whole story? We know that survival rates for males and females were very different. Will the trend between the survival rates for crew and passengers still hold after we \textit{control} for \textbf{gender}?

To do this, let's determine the joint distribution of these two variables and a third variable, \textbf{gender}. In principle, the frequencies could be represented using a three-dimensional table, but it is difficult to visualize more than two dimensions on paper or on a screen. So we put two of the variables along one dimension and one variable along the other, creating a \textit{three-way table}.

\begin{lstlisting}[language=Python]
joint_gender_type_survived = pd.crosstab(
    [df_titanic["gender"], df_titanic["type"]],
    df_titanic["survived"],
    normalize=True
)
joint_gender_type_survived
\end{lstlisting}

\small\begin{verbatim}
survived                 0         1
gender type
female crew       0.001359  0.009062
       passenger  0.057544  0.153602
male   crew       0.306298  0.086543
       passenger  0.312642  0.072950
\end{verbatim}

Of course, we would have chosen any two of the variables to place along the rows, or had the two variables along the columns instead of the rows. The particular representation above was chosen because it makes it easy to survival rates for each gender and type, i.e.,
$$ P(\text{survived} | \textbf{gender}, \textbf{type}), $$
where \textbf{gender} is either "male" or "female" and \textbf{type} is either "crew" or "passenger". Recall that the conditional probability is calculated as
$$ P(\text{survived} | \textbf{gender}, \textbf{type}) = \frac{P(\text{survived}, \textbf{gender}, \textbf{type})}{P(\textbf{gender}, \textbf{type})}. $$
The numerator is the joint distribution above. The denominator can be calculated by summing over the possible values of \textbf{survived}---in other words, across each row, over the columns.

\begin{lstlisting}[language=Python]
joint_gender_type = joint_gender_type_survived.sum(axis=1)
joint_gender_type
\end{lstlisting}

\small\begin{verbatim}
gender  type
female  crew         0.010421
        passenger    0.211146
male    crew         0.392841
        passenger    0.385591
dtype: float64
\end{verbatim}

To obtain the conditional probabilities, we simply divide the joint distribution by the marginal.

\begin{lstlisting}[language=Python]
survived_given_gender_type = joint_gender_type_survived.divide(
    joint_gender_type,
    axis=0
)
survived_given_gender_type
\end{lstlisting}

\small\begin{verbatim}
survived                 0         1
gender type
female crew       0.130435  0.869565
       passenger  0.272532  0.727468
male   crew       0.779700  0.220300
       passenger  0.810811  0.189189
\end{verbatim}

Now, let's compare the survival rates of passengers and crew members for females and males separately.
\begin{itemize}
\item
For females, crew members survived at a higher rate:
$$ P(\text{survived} | \text{female}, \text{crew}) = 0.869565 > 0.727468 = P(\text{survived} | \text{female}, \text{passenger}) $$

\item
For males, crew members survived at a higher rate:
$$ P(\text{survived} | \text{male}, \text{crew}) = 0.220300 > 0.189189 = P(\text{survived} | \text{male}, \text{passenger}) $$

\end{itemize}

But remember, we found earlier that passengers survived at a higher rate overall:
$$ P(\text{survived} | \text{crew}) < P(\text{survived} | \text{passenger}). $$

How is it possible that both male and female crew members survived at a higher rate, yet crew members survived at a lower rate overall? This surprising phenomenon is known as \textit{Simpson's paradox}.

\subsection{Simpson's Paradox}\label{2.3.3}

Simpson's paradox is a phenomenon where a trend disappears or reverses when the data is aggregated. In the Titanic data set, both male and female crew members survived at higher rates, but when we aggregated over gender, the trend reversed.

In order to investigate Simpson's paradox, we first reorganize the probabilities. First, we keep only the survival rate, dropping the death rate (since it is redundant; it is just one minus the survival rate).

\begin{lstlisting}[language=Python]
survived_given_gender_type[1]
\end{lstlisting}

\small\begin{verbatim}
gender  type
female  crew         0.869565
        passenger    0.727468
male    crew         0.220300
        passenger    0.189189
Name: 1, dtype: float64
\end{verbatim}

Next, we rearrange these probabilities into a two-way table, with gender along one dimension and type along the other. This can be achieved in code by "unstacking" a level of the index. (There are two "levels": \textbf{gender} and \textbf{type}.)

\begin{lstlisting}[language=Python]
survival_rates_by_gender_type = (survived_given_gender_type[1].
                                 unstack(level="type"))
survival_rates_by_gender_type
\end{lstlisting}

\small\begin{verbatim}
type        crew  passenger
gender
female  0.869565   0.727468
male    0.220300   0.189189
\end{verbatim}

Caution: the probabilities in this table do not represent a distribution. They do not add up to 1.0. These probabilities originally came from the conditional distribution of \textbf{survived} given \textbf{gender} and \textbf{type}, but we dropped the death rates from the data.

Finally, we append the overall survival rates (which we calculated at the beginning of this lesson) to the last row of this \verb|DataFrame|.

\begin{lstlisting}[language=Python]
survival_rates_by_gender_type.append(survived_given_type[1])
\end{lstlisting}

\small\begin{verbatim}
type        crew  passenger
gender
female  0.869565   0.727468
male    0.220300   0.189189
1       0.237079   0.379651
\end{verbatim}

The overall survival rates are weighted averages of the survival rates for each gender. If we look at the survival rates for crew members:
\begin{itemize}
\item
The survival rate for female crew is 87.0%.

\item
The survival rate for male crew is 22.0%.

\item
The overall survival rate for all crew is 23.7%, which is between the gender-specific survival rates, but much closer to the survival rate for male crew.

\end{itemize}

Likewise, if we look at the survival rates for passengers:
\begin{itemize}
\item
The survival rate for female crew is 72.7%.

\item
The survival rate for male crew is 18.9%.

\item
The overall survival rate for all crew is 38.0%, which is closer to the middle of the gender-specific survival rates.

\end{itemize}

Why would the survival rate for crew members be so close to the survival rate for male crew? To answer this question, let's examine the weights that go into this weighted average.

In mathematical notation, the overall survival rate can be decomposed as:
$$ \underbrace{P(\text{survived} | \textbf{type})}_{\text{overall survival rate}} = \sum_{\textbf{gender}} \underbrace{P(\textbf{gender} | \textbf{type})}_{\text{weight}} \underbrace{P(\text{survived} | \textbf{gender}, \textbf{type})}_{\text{gender-specific survival rate}}. $$
So we see that the weights are $P(\textbf{gender} | \textbf{type})$.

First, we calculate this conditional distribution from the joint distribution of \textbf{gender} and \textbf{type}.

\begin{lstlisting}[language=Python]
joint_gender_type = pd.crosstab(
    df_titanic["gender"],
    df_titanic["type"],
    normalize=True
)

gender_given_type = joint_gender_type.divide(
    joint_gender_type.sum(axis=0),
    axis=1
)
gender_given_type
\end{lstlisting}

\small\begin{verbatim}
type        crew  passenger
gender
female  0.025843   0.353834
male    0.974157   0.646166
\end{verbatim}

Notice that 97.4% of crew members were male! So the lower male survival rate is going to dominate the weighted average when we calculate the overall survival rate for crew members. On the other hand, the gender ratio for passengers was more balanced, so their overall survival rate will end up being closer to the middle of the male and female survival rates.

Now, we calculate the weighted average, using the conditional distribution of gender as "weights" that we multiply by the survival rates. Then, we sum over the genders to get the weighted averages---i.e., the overall survival rates.

\begin{lstlisting}[language=Python]
(gender_given_type * survival_rates_by_gender_type).sum(axis=0)
\end{lstlisting}

\small\begin{verbatim}
type
crew         0.237079
passenger    0.379651
dtype: float64
\end{verbatim}

Check that these match the overall survival rates that we calculated above.

So the secret of Simpson's Paradox lies in two facts:
\begin{enumerate}
\item
Survival rates were generally much lower for men than for women.

\item
Because crew members were predominantly male, their survival rate was weighted towards the lower male survival rate, that their overall survival rate ended up being lower than the survival rate for passengers.

\end{enumerate}

Simpson's Paradox means that we have to be careful when comparing proportions from a two-way table, such as survival rates for crew and passengers. When we control for a third variable, such as \textbf{gender}, the direction of the effect could change!

\subsection{Exercises}\label{2.3.4}

1. Calculate the \textit{percent change} in survival rates between passengers and crew members, controlling for where they embarked. Does there appear to be a Simpson's paradox effect here?

Exercise 2 asks you to work with the Florida Death Penalty data set, which is available at  \verb|https://dlsun.github.io/pods/data/death_penalty.csv|. This data set contains information about the races of the defendant and the victim, as well as whether a death penalty verdict was rendered, in 674 homicide trials in Florida between 1976-1987.

2. Calculate the \textit{relative risk} of a death penalty verdict for black defendants (relative to white defendants), adjusting for the race of the victim. How does this compare to what you found at the end of the last chapter? Can you explain intuitively what is going on?