\chapter{Bundle Adjustment}
\label{ch:bundle_adjustment}

\definecolor{lgray}{gray}{0.95}

Satellite position and orientation errors have a direct effect on the
accuracy of digital elevation models produced by the Stereo Pipeline.
If they're not corrected, these uncertainties will result in
systematic errors in the overall position and slope of the \ac{DEM}.  Severe
distortions can occur as well, resulting in twisted or ``taco shaped''
\acp{DEM}, though in most cases these effects are quite subtle and hard to
detect.

The Stereo Pipeline includes a powerful suite of tools for correcting
camera position and orientation errors using a process called
\emph{bundle adjustment}. Bundle adjustment is the process of
simultaneously adjusting the properties of many cameras and the 3D
locations of the objects they see in order to minimize the error
between the estimated, back-projected pixel location of the 3D
objects and their actual measured location in the captured images.

That complex process can be boiled down to this simple idea: bundle
adjustment ensures that observations in multiple different images of a
single ground feature are self-consistent. If they are not consistent,
then the position and orientation of the cameras as well as the 3D
position of the feature must be adjusted until they are.  This
optimization is carried out along with thousands (or more) of similar
constraints involving many different features observed in other
images.  Bundle adjustment is very powerful and versatile: it can
operate on just two overlapping images, or on thousands.

\begin{figure}[bt]
  \centering
  \includegraphics[width=8cm]{images/ba_orig}
  \includegraphics[width=8cm]{images/ba_adjusted}
  \caption{Bundle adjustment is illustrated here using a color-mapped,
    hill-shaded DEM mosaic from Apollo 15 Orbit 33 imagery. (a)
    Prior to bundle adjustment, large discontinuities can exist between
    overlapping DEMs made from different images. (b) After bundle
    adjustment, DEM alignment errors are minimized, and no longer visible.}
  \label{fig:bundle_adjustment}
\end{figure}

Bundle adjustment can also take advantage of \acp{GCP}, which are
3D locations of features that are known a priori (often by measuring
them by hand in another existing \ac{DEM}). \acp{GCP} can improve the internal
consistency of your \ac{DEM} or align your \ac{DEM} to an existing data
product. Finally, even though bundle adjustment calculates the
locations of the 3D objects it views, only the final properties of
the cameras are recorded for use by the Ames Stereo Pipeline. Those
properties can be loaded into the \texttt{stereo} program which
uses its own method for triangulating 3D feature locations.

When using the Stereo Pipeline, bundle adjustment is an optional step
between the capture of images and the creation of \acp{DEM}. The bundle
adjustment process described below should be completed prior to
running the \texttt{stereo} command.

Although bundle adjustment is not a required step for generating
\acp{DEM}, it is {\em highly recommended} for users who plan to
create \acp{DEM} for scientific analysis and publication.  Incorporating
bundle adjustment into the stereo work flow not only results in
\acp{DEM} that are more internally consistent, it is also the correct
way to co-register your \acp{DEM} with other existing data sets and
geodetic control networks.

At the moment however, Bundle Adjustment does not automatically work
against outside DEMs from sources such as laser altimeters. Hand
picked \acp{GCP} are the only way for \acp{ASP} to register to those
types of sources.

\subsection{A deeper understanding}

In bundle adjustment the position and orientation of each camera
station are determined jointly with the 3D position of a set of image
tie-points points chosen in the overlapping regions between
images. Tie points, like they sound, tie individual camera images
together. Their physical manifestation would be a rock or small crater
than can be observed across multiple images.

Tie-points can be automatically extracted using Vision Workbench's
Interest Point module or through a number of outside methods such as
the famous SURF\citep{surf08}. Creating a tie point is a three step
process. First, all images are processed for their natural {\it
  'interesting'} points. In most algorithms, an interest point is
defined as a place where image gradients accumulate together into a
peak. These interesting points are then described by the texture that
surrounds them. Finally the described interesting points are matched
across the multiple images. A single matched pair of interest points
is now a tie point to use in bundle adjustment. In application, there
is also a little filtering of the tie points with RANSAC
\citep{fischler81}.

\begin{figure}[b!]
  \begin{center}
  \includegraphics[trim=20mm 20mm 20mm 15mm,clip,width=6in]{images/ba_feature_observation.pdf}
  \end{center}
  \caption{ A feature observation in bundle adjustment, from \citet{moore09} }
  \label{fig:ba_feature}
\end{figure}

Our bundle adjustment approach follows the method described
in~\cite{triggs00} and determines the best camera parameters that
minimize the projection error given by ${\bf \epsilon} =
\sum_k\sum_j(I_k-I(C_j, X_k))^2$ where $I_k$ are the tie points on the
image plane, $C_j$ are the camera parameters, and $X_k$ are the 3D
positions associated with features $I_k$. $I(C_j, X_k)$ is an image
formation model (i.e. forward projection) for a given camera and 3D
point. To recap, it projects the 3D point, $X_k$, into the camera with
parameters $C_j$. This produces a predicted image location for the 3D
point that is compared against the observed location, $I_k$. We reduce
this error with the Levenberg-Marquardt algorithm (LMA). Speed is
improved by using sparse methods as described in \citet{hartley04}.

Even though the arithmetic for bundle adjustment sounds clever. There
are faults with the base implementation. Imagine a case where all
cameras and 3D points were collapsed into a single point. If you
evaluate the above cost function, you'll find that the error is indeed
zero. Sadly, this is not the correct solution if the images were taken
from orbit. Another example is if a translation was applied equally to
all 3D points and camera locations. This again would not effect the
cost function. This fault comes from bundle adjustment's inability to
control scale and translation of the solution. It will correct
geometric shape of the problem. Yet it can not guarantee that solution
will have correct scale and translation.

We attempt to fix this problem by adding two additional cost functions
to bundle adjustment. First of which is ${\bf \epsilon} =
\sum_j(C_j^{initial}-C_j)^2$. This constrains camera parameters to
stay relatively close to their initial values. Second, a small handful
of 3D ground control points can be chosen by hand and added to the
error metric as ${\bf \epsilon} = \sum_k(X_k^{gcp}-X_k)^2$ to
constrain these points to known locations in the planetary coordinate
frame. A physical example of a ground control point could be the
location of a lander that has a well known location. In the cost
functions discussed above, errors are weighted by the inverse
covariance of the measurement that gave rise to the constraint.

Like other iterative optimization methods, there are several
conditions that will cause bundle adjustment to terminate.  When
updates to parameters become insignificantly small or when the error,
${\bf \epsilon}$, becomes insignificantly small, then the algorithm
has converged and the result is most likely as good as it will get.
However, the algorithm will also terminate when the number of
iterations becomes too large, in which case bundle adjustment may or
may not have finished refining the parameters of the cameras.

\section{Performing bundle adjustment with isis\_adjust}

First off, let it be known that USGS's \ac{ISIS} has its own bundle
adjustment software called \texttt{jigsaw}. It has a long history and
is supported by a team of skilled developers. Despite this, Ames
Stereo Pipeline provides an alternative called \texttt{isis\_adjust}
that has its own benefits.

Like \texttt{jigsaw}, the \texttt{isis\_adjust} program is designed to
perform bundle adjustment on images supported by \ac{ISIS} 3
software package.  The \texttt{isis\_adjust} program does not
discriminate based on camera type.  It can perform bundle adjustment
on images from line-scan imagers like \ac{MOC}, \ac{LROC} NAC,
\ac{HiRISE}, and the \ac{CTX}. The \texttt{isis\_adjust} program can
also perform bundle adjustment on traditional frame cameras
(e.g. Apollo Metric Camera). Theoretically it should also work with
push-frame imagers like the \ac{THEMIS} VIS, and \ac{LROC} WAC, though
this is untested.

The \texttt{isis\_adjust} program works by first converting all
pixel measurements in an image to measurements defined on the ideal
focal plane using millimeters and the \ac{ET}. The \ac{ET} is the
absolute second at which that pixel measurement was recorded on the
camera.  For a frame camera, all of the pixels are captured at the
same time so the \ac{ET} will be identical for all measurements on
the image.  For pushbroom, pushframe, or other cameras which build
up their `image' over time, different parts of the image will have
different ET values.  For example, on a MOC image between the first
and last line about 5 seconds of \ac{ET} will have elapsed.

When \texttt{isis\_adjust} calculates the partial derivatives of the
forward projection of a point, it uses an ideal pinhole camera model.
The properties of this model are defined as properties of the subject
camera at the specified \ac{ET} for the current measure plus the
correction function, $f(t)$, that \texttt{isis\_adjust} is solving
for.  Many forms of $f(t)$ could be used; the only limit is the
number of parameters in the equations. Some initial work hints that
anything greater than a second order polynomial becomes an ill-posed
problem, but we hope to investigate this further in the future.

Our \texttt{isis\_adjust} implements multiple bundle adjustment
algorithms which can be selected from the command line. It implements
the standard sparse algorithm that can be found in literature. It also
implements our research work into robust cost functions. This allows
the algorithm to ignore measurements that it thinks to be outliers.

Finally, \texttt{isis\_adjust} stores its solution for a camera as a
delta that can be added to the camera's original SPICE in a separate
file. Those files have the extension \texttt{*.isis\_adjust}. This is
less than ideal and one feature that we value in \ac{ISIS}'s
\texttt{jigsaw}. Their software overwrites the SPICE information in
the cube file. This means that \texttt{jigsaw} results can be used in
other \ac{ISIS} programs like \texttt{cam2map}. This is not the case
with \texttt{isis\_adjust}'s results.

\subsection{Options}

The following is a listing and explanation of the options that can be
given to \texttt{isis\_adjust} on the command line. These options are
not required.

\begin{description}

\item[\texttt{-\/-cnet|-c \textit{control-network-file}}] \hfill \\
  \emph{Optional.} This option will force {\tt isis\_adjust} to
  use a pre-built built control network. This control network can
  either be in the \ac{USGS} \ac{ISIS} ``cnet'' format or in the binary Vision
  Workbench format.

  If no control network is supplied using this option, {\tt
    isis\_adjust} will look for match files in the current working
  directory with base filenames that match the input images.  The
  \texttt{isis\_adjust} program will then create its own control
  network file and save it as \texttt{isis\_adjust.cnet}.

\item[\texttt{-\/-cost-function L1|L2|Cauchy|Huber|PseudoHuber(=L2)} ] \hfill \\
  Sets the cost function used for bundle adjustment. Default is \texttt{L2}
  which is the normal squared error. The full list of available options
  are:

  \begin{description}
    \item[L1] Proportional Error
    \item[L2] Squared Error and Default Option.
    \item[Cauchy]
    \item[Huber]
    \item[PseudoHuber]
  \end{description}

  The options towards the end of the list are robust cost functions
  that deal better with non-ideal data that has outliers. These robust
  cost functions are performed by post-weighting the original errors yet
  still using equations derived for squared error.

\item[\texttt{-\/-bundle-adjuster Ref|Sparse|RobustRef|RobustSparse|RobustSparseKGCP(=Sparse)}] \hfill \\
  Sets the bundle adjustment code to be used. The standard to use is
  \texttt{Sparse}, which is a traditional squared error derived method that
  utilizes sparse matrices to obtain speed. Here are the complete list
  of options:

  \begin{description}
    \item[Ref] Reference implementation that doesn't use sparse methods.
    \item[Sparse] Default implementation.
    \item[RobustRef]
    \item[RobustSparse]
    \item[RobustSparseKGCP]
  \end{description}

  The ending methods are experimental student-t derived bundle
  adjustment algorithms. Using the experimental robust algorithm
  overrides the {\tt cost-function} option. The last two bundle
  adjustment algorithms are still experimental and are a work in
  progress.

\item[\texttt{-\/-disable-camera-const}] \hfill \\
  \emph{Optional.} This disables the camera constraint error. Useful
  for debugging and just exploring what are the effects of this cost
  function.

\item[\texttt{-\/-disable-gcp-const}] \hfill \\
  \emph{Optional.} This disables the \ac{GCP} constraint error even
  if \acp{GCP} are provided. Useful for debugging and just exploring
  what the effects are of \acp{GCP}.

\item[\texttt{-\/-gcp-scalar \textit{multiplier(=1)}}] \hfill \\
  \emph{Optional.} Sets the multiplier that is used to adjust the
  sigma (or uncertainty) of the \aclp{GCP}. The sigmas of
  \aclp{GCP} are defined in the \ac{GCP} data file, so this
  option is useful when debugging for universally scaling \ac{GCP} sigmas
  up or down.

\item[\texttt{-\/-lambda|-l \textit{float}}] \hfill \\
  \emph{Optional.} This sets the starting value for $\lambda$: the
  parameter in the Levenberg Marquardt (LMA) optimization algorithm
  that selects between Gauss-Newton optimization and gradient
  descent. This parameter evolves over time on its own, but this
  argument can be used to override its initial value. \emph{This is an
  advanced setting, not recommended for normal use.}

\item[\texttt{-\/-min-matches \textit{integer(=5)}}] \hfill \\
  Set the minimum number of tie-points that are required between a
  pair of images for them to be included in the control network. This
  option is useful for eliminating tie-points from image pairs that
  have only a handful of poor or erroneous matches.

\item[\texttt{-\/-max-iterations \textit{integer(=25)}}] \hfill \\
  Sets the maximum number of iterations for bundle adjustment. The
  number of required iterations will vary by problem size, so this
  parameter allows the user to decide how much time they're willing
  to dedicate to the correction of the data.  We have found that 20
  iterations suffices for small problems with 10 or fewer images, and
  tens or hundreds of iterations may be required for problems with
  hundreds or thousands of images.

\item[\texttt{-\/-poly-order \textit{integer(=0)}}] \hfill \\
  \emph{Optional.} Sets the order of the polynomial that is used for
  adjustment. Using zero means only apply offset to the camera
  parameters that are not time dependent. That setting is recommend
  for frame cameras. Linescan imagers should using either a first
  order polynomial or a second order. Increasing this number too high can
  lead to a problem that is ill-defined and would prevent the algorithm
  from converging on a solution. \emph{Initial work suggest that anything
    beyond a 2nd order polynomial would be ill-defined. 3rd order may
    work with a good dose of ground control points.}

\item[\texttt{-\/-position-sigma \textit{float(=100)}}] \hfill \\
  Sets the sigma (or uncertainty) of the spacecraft position in
  units of meters.

\item[\texttt{-\/-pose-sigma \textit{float(=0.1)}}] \hfill \\
  Sets the sigma (or uncertainty) of the spacecraft pose in units
  of radians.

\item[\texttt{-\/-report-level|-r \textit{integer(=10)}}] \hfill \\
  \emph{Optional.} Sets the report level for the final bundle
  adjustment report.  This report is saved as
  \texttt{isis\_adjust.report}. Report levels available are:

  \begin{description}
    \item[0   - CommandLine Error and Final report]
    \item[10  - CommandLine Iteration Error (default)]
    \item[20  - Write Report file]
    \item[25  - \textnormal{\emph{In development}}]
    \item[30  - Write Stereo Triangulation Error]
    \item[35  - \textnormal{\emph{In development}}]
    \item[100 - Debug, Write Error Vectors (big human readable)]
    \item[110 - Debug, Write Jacobian Matrix (massive human readable)]
  \end{description}

\item[\texttt{-\/-robust-threshold \textit{float(=10)}}] \hfill \\
  Sets the robust threshold; an additional parameter specifically for
  the \texttt{PseudoHuber}, \texttt{Huber}, and \texttt{Cauchy} arguments to the   \texttt{-\/-cost-function} option.

\item[\texttt{-\/-save-iteration-data|-s}] \hfill \\
  \emph{Optional.} Use to write {\tt bundlevis} visualization files.

\item[\texttt{-\/-seed-with-previous}] \hfill \\
  \emph{Optional.} Loads up the previous {\tt isis\_adjust} session's
  adjustment file and uses them as a starting point for this session.

\item[\texttt{-\/-write-isis-cnet-also}] \hfill \\
  \emph{Optional.} Write an \ac{ISIS} \ac{PVL} style control network file to
  \texttt{isis\_adjust.net}. The output file is very large compared to
  the binary output, \texttt{isis\_adjust.cnet}, but is human readable
  and compatible with the \ac{ISIS} 3 \texttt{qnet} tool.

\item[\texttt{-\/-write-kml [0|1(=0)]}] \hfill \\
  \emph{Optional.} Providing this option with a zero will have the
  program write a \ac{KML} file showing the location of all the \acp{GCP}
  and be colored according to their final error. Providing this
  option with a one will have this perform as before, but also have it
  write all of the 3D point estimates. This is useful for debugging and
  for having a quick visualization of where stress points might exist in
  a bundle adjustment problem when there are many cameras.

\item[\texttt{-\/-help|-h}] \hfill \\
  Provides a shortened list of the above.

\end{description}

\section{Visualizing bundle adjustment with bundlevis}

The \texttt{bundlevis} program is used to visualize the process of
bundle adjustment. It will show an animated, fully interactive 3D
scene containing all the 3D points and cameras across all iterations
of bundle adjustment.  This tool is used to quickly determine if
bundle adjustment was successful.  If something does go wrong,
\texttt{bundlevis} can be a powerful debugging tool for identifying
the problem.

\begin{figure}[b!]
  \begin{center}
  \includegraphics[width=5in]{images/bundlevis_apollo.png}
  \end{center}
  \caption{ A screenshot of \texttt{bundlevis} visualizing the bundle
    adjustment of imagery from the Apollo 15 Metric Camera (orbit
    33). }
  \label{fig:bundlevis}
\end{figure}

Once \texttt{bundlevis} has loaded the data from a bundle adjustment run,
the user can click on and inspect the position of 3D points that are
in purple and play back the iterations using the keyboard. Double
clicking on a camera will cause lines to be drawn to each point viewed
by the camera. Clicking on a 3D point causes lines to be drawn to all
cameras that view the point.

Bundle adjustment can fail in a variety of different ways, but users
should be aware of two common failure modes.  The first is segmentation;
where tie points will split into two or more distinct groups,
producing cliffs between or clumps among points. This is usually
caused by insufficient matches between a pair of images in your
control network.  You may need to choose some tie-points between
these images by hand or add additional images that overlap with the
problem area.

The second common sign of a failure is a point cloud explosion (the
resulting terrain looks unrecognizably noisy). This most often results
from a high number of outlying, bad tie-point measurements that the
bundle adjustment algorithm can't recover from. These bad constraints
can be removed by hand or mitigated using one of the robust cost modes
(e.g. by using the \texttt{-\/-cost-function} argument for
\texttt{isis\_adjust}).

\subsection{Options}

The following is a listing and explanation of the options
that can be given to \texttt{bundlevis} from the command line.

\begin{description}

\item[\texttt{-\/-camera-iteration-file|-c \textit{bundlevis-camera-iteration-file}}] \hfill \\
  \emph{Optional.} Supply a camera iteration file that was produced by
       \texttt{isis\_adjust}.  \texttt{bundlevis} will only draw cameras
       if you supply a camera iteration file.

\item[\texttt{-\/-points-iteration-file|-p \textit{bundlevis-point-iteration-file}}] \hfill \\
  \emph{Optional.} Supply a point iteration file that was produced by
       \texttt{isis\_adjust}.  \texttt{bundlevis} will only draw 3D points
       if you supply a point iteration file.

\item[\texttt{-\/-control-network-file|-n \textit{Vision-Workbench-binary-control-network-file}}] \hfill \\
  \emph{Optional.} Supply a control network file that was produced by
       {\tt isis\_adjust}.  This allows {\tt bundlevis} to show the
       relationship between points and cameras when used in
       conjunction with \texttt{-\/-camera-iteration-file} and
       \texttt{-\/-points-iteration-file}.

\item[\texttt{-\/-additional-pnt-files \textit{bundlevis-point-iteration-files}}] \hfill \\
  \emph{Optional.} Supply additional points to be animated alongside
  the camera and 3D points.  The files given must be in the same
  format as a \texttt{bundlevis} point iteration file and have the same number
  of iterations.

\item[\texttt{-\/-fullscreen}] \hfill \\
  \emph{Optional.} Displays \texttt{bundlevis} using the entire screen;
  otherwise the program loads in a window. \emph{The fullscreen option
    does not work correctly with dual screen systems.}

\item[\texttt{-\/-stereo}] \hfill \\
  \emph{Optional.} Render the 3D scene in red/blue anaglyph mode.

\item[\texttt{-\/-show-moon}] \hfill \\
  \emph{Optional.} Draws a wireframe sphere with a radius of 1737.3~km
  that represents the Moon.

\item[\texttt{-\/-show-mars}] \hfill \\
  \emph{Optional.} Draws a wireframe sphere with a radius of 3397~km
  that represents Mars.

\item[\texttt{-\/-show-earth}] \hfill \\
  \emph{Optional.} Draws a wireframe sphere that represents the Earth.

\end{description}

\subsection{Controls}

Once \texttt{bundlevis} is running, there are several controls that
can be used to interface with the program. There are the playback
controls, jump-to-frame controls, and the mouse.

\paragraph{Playback Controls}

Playback controls are similar to those in the popular Winamp program;
arranged on the keyboard like the controls on a tape deck.

\newenvironment{myindentpar}[1]
               {\begin{list}{}
                   {\setlength{\leftmargin}{#1}}
                 \item[]
               }
               {\end{list}}

\begin{myindentpar}{3cm}
\begin{description}
  \item[Z] Step back one iteration
  \item[X] Play
  \item[C] Pause
  \item[V] Stop \emph{(which is the same as Pause except that
    bundlevis goes back to iteration 0)}
  \item[B] Step forward one iteration
\end{description}
\end{myindentpar}

\paragraph{Jump-to-Frame Controls}
Jump-to-Frame controls are the numbers \textbf{1-9} along the top of
the keyboard. Pressing \textbf{1} will display the very first
iteration. Pressing \textbf{9} will display the very last
iteration. Pressing \textbf{2} through \textbf{8} will display
iterations that are somewhere in between based on the value of the
number. Finally, pressing \textbf{0} will cause \texttt{bundlevis} to
display the points at their very last iteration with an additional
tail pointing back to the starting position of the points in the first
iteration.

\paragraph{Mouse}
The mouse is used to move around the model, and has the same controls
found in many 3D environments (e.g. \texttt{osgviewer}). Moving the
mouse with the \textbf{left mouse button} held down will cause the
model to rotate.  Moving with the \textbf{right mouse button} pressed
will zoom, and moving with the \textbf{middle mouse button} will
translate. Alternatively for systems where the mouse is
button-challenged, \textbf{option + mouse} is translation and
\textbf{command + mouse} is zoom.

Double clicking with the mouse on a point or camera will allow the
user to query entities in the model.  A double click will cause the
point number and camera number to be printed to the terminal. The
number identifier for a given camera or point will also appear when
the viewer is zoomed in on that entity.

\section{Examples of Use}
\label{sec:ba_example}

\subsection{Processing Mars Orbital Camera}

What follows is an example of bundle adjustment using two \ac{MOC}
images of the south Cydonia region. We use images M10/00254 and
R09/01059. These images are available from NASA's \ac{PDS} (the
\ac{ISIS} \texttt{mocproc} program will operate on either the IMQ
or IMG format files, we use the \texttt{.imq} below in the example).
For reference, the following \ac{ISIS} commands are how to convert
the \ac{MOC} images to \ac{ISIS} cubes.

\begin{verbatim}
  ISIS 3> mocproc from= m1000254.imq to= m1000254.cub mapping=no
  ISIS 3> mocproc from= r0901059.imq to= r0901059.cub mapping=no
\end{verbatim}

You will note that the resulting images are not map projected. Bundle
adjustment requires the ability to project arbitrary 3D points into
the camera frame. The process of map projecting an image dissociates
the camera model from the image. Map projecting can be perceived as
the generation of a new infinitely large camera sensor that is
perfectly parallel to the surface of its subject ( and thus spherical
). That makes it extremely hard to project a random point into
the camera's original model. The math would follow the transformation from
projection into the camera frame, then projected back down to surface
that ISIS uses, then finally up into the infinitely large sensor. The
\texttt{isis\_adjust} program does not support this.

At this point, we need to automatically generate tie-points between
these two images. This can be done using the \texttt{ipfind} and
\texttt{ipmatch} utilities. These tools do not reliably (currently)
work with photometrically calibrated images, so we must first convert
these images to a standard format using the \ac{ISIS} program
\texttt{isis2std} or \texttt{gdal\_translate}. The utility
\texttt{gdal\_translate} is the preferred option as it is able to
produce lossless compressed images.

\begin{verbatim}
  ISIS 3> isis2std from= m1000254.cub to= m1000254.tif format=TIFF
  ISIS 3> isis2std from= r0901059.cub to= r0901059.tif format=TIFF
\end{verbatim}

or ...

\begin{verbatim}
  ISIS 3> gdal_translate -of GTiff -scale -ot Byte
              -co COMPRESS=LZW -co TILED=yes m1000254.cub m1000254.tif
  ISIS 3> gdal_translate -of GTiff -scale -ot Byte
              -co COMPRESS=LZW -co TILED=yes r0901059.cub r0901059.tif
\end{verbatim}

Here is how to process those newly created TIFF files for tie-points
using the Interest Point Module tools from Vision Workbench.

\begin{verbatim}
  > ipfind  m1000254.tif r0901059.tif -g 1.2
  > ipmatch m1000254.tif r0901059.tif -d -r homography -i 30
\end{verbatim}

\begin{center}
\fcolorbox{black}{lgray}{ \begin{minipage}{5.5in}

    Be aware that the tie-point tools available in Vision Workbench do
    not always produce enough matches for bundle adjustment. An
    alternative would be to use outside code such as SURF. We have
    included a patch at the end of the book that converts SURF output
    to a Vision Workbench style match file. In extreme cases, users
    can be forced to make measurements themselves. This can be
    performed with \ac{ISIS}'s \texttt{qnet} progam.  \\ \\ For this
    example we found that \texttt{ipfind} and \texttt{ipmatch} work
    great. Expect to find approximately 100 matched points. Your
    results will be slightly different due to the random nature of
    RANSAC.

\end{minipage}}
\end{center}

Finally it is time to start bundle adjustment. There are many options
that can be used at this stage.  We have chosen those required to
create visualization data for \texttt{bundlevis}.  We have also set
the maximum iterations to 100 and chose the option to create a
detailed report file of \texttt{isis\_adjust}'s results.

\begin{verbatim}
  ISIS 3> isis_adjust *.cub -s --max 30 -r 50
\end{verbatim}

Before you hit enter, notice that we are not feeding the interest
point match files to \texttt{isis\_adjust}. Instead, we only provide
the camera files. During run time, the program will see that it
doesn't have any image measurements in the form of a control
network. In that case it will attempt to build its own by searching
the current working directory for \texttt{*.match} files.

Now you have permission to run \texttt{isis\_adjust}. You'll see the
command will produce considerable debugging output and will place many
output files in the current working directory. If you look through the
output in the terminal or alternatively in the output report file,
\texttt{isis\_adjust.report}, you'll see that the problem did not
converge but reduced most of the error in the first 40 iterations
(again, your results may vary slightly). The lack of convergence is
worrisome and a few (like 3) ground control points would probably help
considerably.  Notice that the error improved only slightly after
those first 40 iterations, but the shape of the pointcloud and the
camera paths changed considerably. This untwisting of the cameras can
be seen next in \texttt{bundlevis}.

Visualizing all of the data that was exported for \texttt{bundlevis}
can be carried out as follows:

\begin{verbatim}
  > bundlevis -p iterPointsParam.txt -c iterCameraParam.txt \
              -n isis_adjust.cnet
\end{verbatim}

Press escape to exit out of \texttt{bundlevis} when finished.  You may
also want to try viewing the data with a wire-frame of Mars to give
some perspective. Note, you will have to zoom in very far since the
size of a MOC frame is quite small relative to the size of Mars!

\begin{verbatim}
  > bundlevis -p iterPointsParam.txt -c iterCameraParam.txt \
              -n isis_adjust.cnet --show-mars
\end{verbatim}

Producing a DEM using the newly created corrections is the same as
covered in the Tutorial on page \pageref{ch:tutorial}, with one small
difference: \texttt{stereo} needs to know of the existence of the
correction files, \texttt{m1000254.isis\_adjust} and
\texttt{r0901059.isis\_adjust}.

\begin{verbatim}
  ISIS 3> stereo m1000254.cub r0901059.cub m1000254.isis_adjust \
                 r0901059.isis_adjust MOC_RESULTS/M1000254_R0901059
\end{verbatim}

The two new arguments (\texttt{*.isis\_adjust}) provide
\texttt{stereo} with the necessary corrections. When providing outside
camera models for images, they are always the 3rd and 4th argument for
\texttt{stereo}. This is also how \texttt{stereo} can be made to
operate on images from standard consumer cameras.

\subsection{Processing with Ground Control Points}

Ground control point files describe a single point in the world
that is seen by 1 or more cameras. How they are measured in the
first place is up to the user. We use a manual process of comparing
each image to a respected map projected image and then recording
the latitude, longitude, and altitude of the point(s). The maps to
register against can be anything, but it is recommended to register
against a product with a high amount of cartographic stability and
accuracy.  For terrestrial work, we would use a \ac{USGS} product
that can provide imagery that is registered to LIDAR height
measurements.

Unlike match files, ground control points must specifically be given
to \texttt{isis\_adjust} from the command line, but in no particular
order. Ground control point files are written with the extension
\texttt{.gcp}. Below is an example of a ground control point file that
was created to control a series of Apollo Metric Camera images from
several Apollo 15 orbits.

\begin{verbatim}
    -52.8452 27.2561 1735999 300 300 500
    sub4-AS15-M-2086.cub     210.9   3565.0
    sub4-AS15-M-2087.cub     1476.9  3579.0
    sub4-AS15-M-2088.cub     2798.9  3586.8
    sub4-AS15-M-2089.cub     4133.5  3588.6
    sub4-AS15-M-2344.cub     906.9   3874.8
    sub4-AS15-M-2345.cub     2204.2  3913.9
    sub4-AS15-M-2482.cub     939.8   4348.0
    sub4-AS15-M-2483.cub     2282.0  4340.7
    sub4-AS15-M-2484.cub     3642.1  4330.9
\end{verbatim}

The first line of a \texttt{.gcp} file is like a header line and
is different from the remaining lines.  The first line defines the
world location of the ground control point, and the rest of the
lines define the image locations of the ground control points. Here
are what the columns mean for the first line.

\begin{myindentpar}{2cm}
\begin{description}
  \item[Column 1:] Longitude in degrees
  \item[Column 2:] Latitude in degrees
  \item[Column 3:] Radius in meters
  \item[Column 4:] Sigma (or uncertainty) in meters for Local X axis
  \item[Column 5:] Sigma (or uncertainty) in meters for Local Y axis
  \item[Column 6:] Sigma (or uncertainty) in meters for Local Z axis
\end{description}
\end{myindentpar}

The other lines describe where this \ac{GCP} is found in each image:

\begin{myindentpar}{2cm}
\begin{description}
  \item[Column 1:] Image name
  \item[Column 2:] Sample (X) image measurement
  \item[Column 3:] Line (Y) image measurement
\end{description}
\end{myindentpar}

Make a {\tt .gcp} file for every ground control point, then be sure
to feed them as an input to {\tt isis\_adjust}. Remember that you
can scale the sigma of all ground control points by using the {\tt
-\/-gcp-scalar} flag. This can save time by allowing you to make
adjustments without needing to edit all of the files individually.

\subsection{Sharing Data with ISIS 3's qnet program}

\ac{ISIS} contains a program called \texttt{qnet} whose purpose is
to create and edit \ac{ISIS} style control network files. To share
a control network with \texttt{qnet}, you will need to save our
control network in the \ac{ISIS} format. If bundle adjustment has
already been performed once and if we want to simply convert the
control network for use in \texttt{qnet}, you can use this command
to save an \ac{ISIS} style control network:

\begin{verbatim}
  ISIS 3> isis_adjust -c isis_adjust.cnet --write-isis-cnet-also *.cub
\end{verbatim}

Otherwise if this is the first time performing bundle adjustment and a
control network does not already exist, use:

\begin{verbatim}
  ISIS 3> isis_adjust --write-isis-cnet-also *.cub
\end{verbatim}

There should now be an \texttt{isis\_adjust.net} file in the project's
directory. It will be quite a bit larger than the other control
network file since it is stored as ASCII text, but it can be read
and edited with a text editor. Before starting \texttt{qnet}, there
is one additional preparation that must be performed. \ac{ISIS}'s
\texttt{qnet} requires a text file listing of all the cubes used
by the control network. Here's how to create one:

\begin{verbatim}
  > ls *.cub > list_of_cubes.lis
\end{verbatim}

Now, start up \texttt{qnet} without any command line arguments. Click
File$\rightarrow$Open. It will first ask for the list of cubes. Refer
it to the newly created \texttt{list\_of\_cubes.lis}. Next it will ask for
the control network. Give it \texttt{isis\_adjust.net}.

\begin{center}
\fcolorbox{black}{lgray}{ \begin{minipage}{5.5in}

    At this time \texttt{qnet} does not work with the Apollo Metric
    Camera's cube files. When loading the text file listing of cubes
    it will issue an error about invalid serial numbers for the listed
    cube files.  % (03/11/09)

\end{minipage}}
\end{center}

When finished, save the new control network file. Here's how to use
the new control network in \texttt{isis\_adjust}:

\begin{verbatim}
  ISIS 3> isis_adjust -c the_new_control_network.net *.cub
\end{verbatim}

Take note that to distinguish \ac{ISIS} style control network files
from Stereo Pipeline style control network files is by the file
extension. \ac{ISIS} control networks have the extension of
\texttt{.net} and can be read with a text editor. Stereo Pipeline's
control networks have the extension \texttt{.cnet} and are binary
format files.


\documentclass{article}
\usepackage[a4paper, left=25.4mm, top=25.4mm, right=25.4mm, bottom=25.4mm]{geometry}
\usepackage[shortlabels]{enumitem}
\usepackage{adjustbox}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{amsthm}
\usepackage{authoraftertitle}
\usepackage{blindtext}
\usepackage{booktabs}
\usepackage{bussproofs}
\usepackage{cancel}
\usepackage{color}
\usepackage{fancyhdr}
\usepackage{graphicx}
\usepackage{latexsym}
\usepackage{listings}
\usepackage{multicol}
\usepackage{newpxtext}
%\usepackage{newpxmath}
\usepackage{subcaption}
\usepackage{textcomp}
\usepackage{textgreek}
\usepackage{vwcol}
\newcommand\tab[1]{\hspace*{#1}}
\newcommand\lowermidtilde{\raisebox{-0.8ex}{\textasciitilde}}
\newcommand\midtilde{\raisebox{-0.6ex}{\textasciitilde}}
\newcommand{\partfarrow}{\rightharpoonup}
\newcommand{\partf}{\rightharpoonup\!\!\!\!\!\!\!\!\!\raisebox{2.5pt}{$\rightharpoonup$}}
\renewcommand{\thesubsection}{\arabic{subsection}}
\renewcommand{\headrulewidth}{0pt}
\renewcommand{\ttdefault}{pcr}
\renewcommand{\sfdefault}{lmss}
\renewcommand\qedsymbol{Q.E.D.}
\definecolor{gray}{rgb}{0.5,0.5,0.5}
\lstset{
    basicstyle=\ttfamily,
    commentstyle=\color{gray}\ttfamily,
    keywordstyle=\ttfamily\bfseries,
    columns=flexible,
    numbers=left,
    numberstyle=\tiny,
    firstnumber=0,
    showstringspaces=false,
}
\setlist{
    listparindent=\parindent,
    align=parleft,
}

\author{Victor Zhao\\xz398@cam.ac.uk}

\begin{document}
\centering
\section*{Types\\CST Part II Paper 8 \& 9}
\MyAuthor

\justifying

\subsection{Simply-Typed $\lambda$-Calculus}

\subsection*{Syntax}

\hspace{-1em}
\begin{tabular}{lccl}
Types   & $T$ & $::=$ & $1\ |\ 0\ |\ T_1\times T_2\ |\ T_1+T_2\ |\ T_1\to T_2$\\[0.5em]
Terms   & $e$ & $::=$ & $x\ |\ \langle\rangle\ |\ \langle e_1,e_2\rangle\ |\ \textsf{fst }e\ |\ \textsf{snd }e\ |\ \textsf{L }e\ |\ \textsf{R }e\ |\ \textsf{case}(e,\textsf{L }x\to e_1,\textsf{R }y\to e_2)$\\
        & {}  & $|$   & $\lambda x:T.\ e\ |\ e_1\ e_2\ |\ \textsf{abort}$\\[0.5em]
Values  & $v$ & $::=$ & $\langle\rangle\ |\ \langle v_1,v_2\rangle\ |\ \lambda x:T.\ e\ |\ \textsf{L }v\ |\ \textsf{R }v$\\[0.5em]
Contexts & $\Gamma$ & $::=$ & $\cdot\ |\ \Gamma,x:T$
\end{tabular}

\subsection*{Typing rules}

(\textsc{I}: introduction rule, \textsc{E}: elimination rule, \textsc{Hyp}: hypothesis)
\medbreak
\noindent
\AxiomC{{\color{white} $\langle\rangle$}} % placeholder
\RightLabel{\textsc{1I}}
\UnaryInfC{$\Gamma\vdash\langle\rangle:1$}
\DisplayProof$\qquad$ %=========================================================
\AxiomC{$\Gamma\vdash e_1:T_1$}
\AxiomC{$\Gamma\vdash e_2:T_2$}
\RightLabel{\textsc{$\times$I}}
\BinaryInfC{$\Gamma\vdash\langle e_1, e_2\rangle:1$}
\DisplayProof$\qquad$ %=========================================================
\AxiomC{$\Gamma\vdash e:T_1\times T_2$}
\RightLabel{\textsc{$\times$E$_1$}}
\UnaryInfC{$\Gamma\vdash\textsf{fst }e:T_1$}
\DisplayProof$\qquad$ %=========================================================
\AxiomC{$\Gamma\vdash e:T_1\times T_2$}
\RightLabel{\textsc{$\times$E$_2$}}
\UnaryInfC{$\Gamma\vdash\textsf{snd }e:T_2$}
\DisplayProof\bigbreak %========================================================
\noindent
\AxiomC{$x:T\in\Gamma$}
\RightLabel{\textsc{Hyp}}
\UnaryInfC{$\Gamma\vdash x:T$}
\DisplayProof$\qquad$ %=========================================================
\AxiomC{$\Gamma,x:T\vdash e:T'$}
\RightLabel{\textsc{$\to$I}}
\UnaryInfC{$\Gamma\vdash\lambda x:T.\ e:T\to T'$}
\DisplayProof$\qquad$ %=========================================================
\AxiomC{$\Gamma\vdash e_1:T\to T'$}
\AxiomC{$\Gamma\vdash e_2:T$}
\RightLabel{\textsc{$\to$E}}
\BinaryInfC{$\Gamma\vdash e_1\ e_2:T'$}
\DisplayProof\bigbreak %========================================================
\noindent
\AxiomC{$\Gamma\vdash e:T_1$}
\RightLabel{\textsc{$+$I$_1$}}
\UnaryInfC{$\Gamma\vdash\textsf{L }e:T_1+T_2$}
\DisplayProof$\qquad$ %=========================================================
\AxiomC{$\Gamma\vdash e:T_2$}
\RightLabel{\textsc{$+$I$_2$}}
\UnaryInfC{$\Gamma\vdash\textsf{R }e:T_1+T_2$}
\DisplayProof\bigbreak %========================================================
\noindent
\AxiomC{$\Gamma\vdash e:T_1+T_2$}
\AxiomC{$\Gamma,x:X\vdash e_1:T$}
\AxiomC{$\Gamma,x:X\vdash e_2:T$}
\RightLabel{\textsc{$+$E}}
\TrinaryInfC{$\Gamma\vdash\textsf{case}(e,\textsf{L }x\to e_1,\textsf{R }y\to e_2):T$}
\DisplayProof\bigbreak %========================================================
\noindent
(No introduction for 0)$\qquad$
\AxiomC{$\Gamma\vdash e:0$}
\RightLabel{\textsc{0E}}
\UnaryInfC{$\Gamma\vdash\textsf{abort }e:T$}
\DisplayProof

\subsection*{Operational semantics}

(No rule for unit)$\qquad$
\AxiomC{$e_1\leadsto e_1'$}
\RightLabel{\textsc{Pair1}}
\UnaryInfC{$\langle e_1,e_2\rangle\leadsto\langle e_1',e_2\rangle$}
\DisplayProof$\qquad$ %=========================================================
\AxiomC{$e_2\leadsto e_2'$}
\RightLabel{\textsc{Pair2}}
\UnaryInfC{$\langle v,e_2\rangle\leadsto\langle v,e_2'\rangle$}
\DisplayProof\bigbreak %========================================================
\noindent
\AxiomC{{\color{white} $\langle\rangle$}} % placeholder
\RightLabel{\textsc{Proj1}}
\UnaryInfC{$\textsf{fst }\langle v_1,v_2\rangle\leadsto v_1$}
\DisplayProof$\quad$ %==========================================================
\AxiomC{{\color{white} $\langle\rangle$}} % placeholder
\RightLabel{\textsc{Proj2}}
\UnaryInfC{$\textsf{snd }\langle v_1,v_2\rangle\leadsto v_2$}
\DisplayProof$\quad$ %==========================================================
\AxiomC{$e\leadsto e'$}
\RightLabel{\textsc{Proj3}}
\UnaryInfC{$\textsf{fst }e\leadsto\textsf{fst }e'$}
\DisplayProof$\quad$ %==========================================================
\AxiomC{$e\leadsto e'$}
\RightLabel{\textsc{Proj4}}
\UnaryInfC{$\textsf{snd }e\leadsto\textsf{snd }e'$}
\DisplayProof\bigbreak %========================================================
\noindent
\AxiomC{$e\leadsto e'$}
\RightLabel{\textsc{Sum1}}
\UnaryInfC{$\textsf{L }e\leadsto\textsf{L }e'$}
\DisplayProof$\qquad$ %=========================================================
\AxiomC{$e\leadsto e'$}
\RightLabel{\textsc{Sum2}}
\UnaryInfC{$\textsf{R }e\leadsto\textsf{R }e'$}
\DisplayProof\bigbreak %========================================================
\noindent
\AxiomC{$e\leadsto e'$}
\RightLabel{\textsc{Case1}}
\UnaryInfC{$\textsf{case}(e,\textsf{L }x\to e_1,\textsf{R }y\to e_2)\leadsto\textsf{case}(e',\textsf{L }x\to e_1,\textsf{R }y\to e_2)$}
\DisplayProof\bigbreak %========================================================
\noindent
\AxiomC{{\color{white} $[]$}} %placeholder
\RightLabel{\textsc{Case2}}
\UnaryInfC{$\textsf{case}(\textsf{L }v,\textsf{L }x\to e_1,\textsf{R }y\to e_2)\leadsto[v/x]e_1$}
\DisplayProof$\qquad$ %=========================================================
\AxiomC{{\color{white} $[]$}} %placeholder
\RightLabel{\textsc{Case3}}
\UnaryInfC{$\textsf{case}(\textsf{R }v,\textsf{L }x\to e_1,\textsf{R }y\to e_2)\leadsto[v/y]e_2$}
\DisplayProof\bigbreak %========================================================
\noindent
\AxiomC{$e_1\leadsto e_1'$}
\RightLabel{\textsc{App1}}
\UnaryInfC{$e_1\ e_2\leadsto e_1'\ e_2$}
\DisplayProof$\qquad$ %=========================================================
\AxiomC{$e_2\leadsto e_2'$}
\RightLabel{\textsc{App2}}
\UnaryInfC{$v\ e_2\leadsto v\ e_2'$}
\DisplayProof$\qquad$ %=========================================================
\AxiomC{{\color{white} $[]$}} %placeholder
\RightLabel{\textsc{Fn}}
\UnaryInfC{$(\lambda x:T.\ e)\ v\leadsto[v/x]e$}
\DisplayProof\bigbreak %========================================================
\noindent
\AxiomC{$e\leadsto e'$}
\RightLabel{\textsc{Abort}}
\UnaryInfC{$\textsf{abort }e\leadsto\textsf{abort }e'$}
\DisplayProof\newpage %=========================================================

\subsection{Polymorphic $\lambda$-Calculus (System F)}

\subsection*{Syntax}

\hspace{-1em}
\begin{tabular}{lccl}
Types   & $T$ & $::=$ & $\alpha\ |\ T_1\to T_2\ |\ \forall\alpha.\ T\ |\ \exists\alpha.\ T$\\[0.5em]
Terms   & $e$ & $::=$ & $x\ |\ \lambda x:T.\ e\ |\ e_1\ e_2\ |\ \Lambda\alpha.\ e\ |\ e\ T\ |\ \textsf{pack}_{\alpha_\text{abs}.T_\text{sig}}(T_\text{conc},e_\text{impl})$\\
{}      & {}  &  $|$  & $\textsf{let pack}(\alpha,x)=e_\text{impl}\textsf{ in }e_\text{use}$\\[0.5em]
Values  & $v$ & $::=$ & $\lambda x:T.\ e\ |\ \Lambda\alpha.\ e\ |\ \textsf{pack}_{\alpha_\text{abs}.T_\text{sig}}(T_\text{conc},v_\text{impl})$\\[0.5em]
Type Contexts & $\Theta$ & $::=$ & $\cdot\ |\ \Theta,\alpha$\\[0.5em]
Term Contexts & $\Gamma$ & $::=$ & $\cdot\ |\ \Gamma,x:T$
\end{tabular}

\subsection*{Well-formedness of types}

\vspace{0.5em}
\AxiomC{$\alpha\in\Theta$}
\UnaryInfC{$\Theta\vdash\alpha\textsf{ type}$}
\DisplayProof$\qquad$ %=========================================================
\AxiomC{$\Theta\vdash T_1\textsf{ type}$}
\AxiomC{$\Theta\vdash T_2\textsf{ type}$}
\BinaryInfC{$\Theta\vdash T_1\to T_2\textsf{ type}$}
\DisplayProof$\qquad$ %=========================================================
\AxiomC{$\Theta,\alpha\vdash T\textsf{ type}$}
\UnaryInfC{$\Theta\vdash\forall\alpha.\ T\textsf{ type}$}
\DisplayProof

\subsection*{Well-formedness of term contexts}

\vspace{0.5em}
\AxiomC{{\color{white} $\Theta\vdash\Gamma$}} %placeholder
\UnaryInfC{$\Theta\vdash\cdot\textsf{ ctx}$}
\DisplayProof$\qquad$ %=========================================================
\AxiomC{$\Theta\vdash\Gamma\textsf{ ctx}$}
\AxiomC{$\Theta\vdash T\textsf{ type}$}
\BinaryInfC{$\Theta\vdash\Gamma,x:T\textsf{ ctx}$}
\DisplayProof

\subsection*{Typing rules}

\vspace{0.5em}
\AxiomC{$x:T\in\Gamma$}
\RightLabel{\textsc{Hyp}}
\UnaryInfC{$\Theta;\Gamma\vdash x:T$}
\DisplayProof$\qquad$ %==========================================================
\AxiomC{$\Theta\vdash T\textsf{ type}$}
\AxiomC{$\Theta;\Gamma,x:T\vdash e:T'$}
\RightLabel{\textsc{$\to$I}}
\BinaryInfC{$\Theta;\Gamma\vdash\lambda x:T.\ e:T\to T'$}
\DisplayProof\bigbreak %========================================================
\noindent
\AxiomC{$\Theta;\Gamma\vdash e_1:T\to T'$}
\AxiomC{$\Theta;\Gamma\vdash e_2:T$}
\RightLabel{\textsc{$\to$E}}
\BinaryInfC{$\Theta;\Gamma\vdash e_1\ e_2:T'$}
\DisplayProof\bigbreak %========================================================
\noindent
\AxiomC{$\Theta,\alpha;\Gamma\vdash e:T$}
\RightLabel{\textsc{$\forall$I}}
\UnaryInfC{$\Theta;\Gamma\vdash\Lambda\alpha.\ e:\forall\alpha.\ T$}
\DisplayProof$\qquad$ %=========================================================
\AxiomC{$\Theta;\Gamma\vdash e:\forall\alpha.\ T'$}
\AxiomC{$\Theta\vdash T\textsf{ type}$}
\RightLabel{\textsc{$\forall$E}}
\BinaryInfC{$\Theta;\Gamma\vdash e\ T:[T/\alpha]T'$}
\DisplayProof\bigbreak %========================================================
\noindent
\AxiomC{$\Theta,\alpha_\text{abs}\vdash T_\text{sig}\textsf{ type}$}
\AxiomC{$\Theta\vdash T_\text{conc}\textsf{ type}$}
\AxiomC{$\Theta;\Gamma\vdash e_\text{impl}:[T_\text{conc}/\alpha_\text{abs}]T_\text{sig}$}
\RightLabel{\textsc{$\exists$I}}
\TrinaryInfC{$\Theta;\Gamma\vdash\textsf{pack}_{\alpha_\text{abs}.T_\text{sig}}(T_\text{conc},e_\text{impl}):\exists\alpha_\text{abs}.\ T_\text{sig}$}
\DisplayProof\bigbreak %========================================================
\noindent
\AxiomC{$\Theta;\Gamma\vdash e_\text{impl}:\exists\alpha_\text{abs}.\ T_\text{sig}$}
\AxiomC{$\Theta,\alpha;\Gamma,x:[\alpha_\text{abs}/\alpha]T_\text{sig}\vdash e_\text{use}:T_\text{use}$}
\AxiomC{$\Theta\vdash T_\text{use}\textsf{ type}$}
\RightLabel{\textsc{$\exists$E}}
\TrinaryInfC{$\Theta;\Gamma\vdash\textsf{let pack}(\alpha,x)=e_\text{impl}\textsf{ in }e_\text{use}:T_\text{use}$}
\DisplayProof

\subsection*{Operational semantics}

(\textsc{Cong}: congruence rule, \textsc{Eval}: evaluation rule)
\medbreak
\noindent
\AxiomC{$e_1\leadsto e_1'$}
\RightLabel{\textsc{CongFun}}
\UnaryInfC{$e_1\ e_2\leadsto e_1'\ e_2$}
\DisplayProof$\qquad$ %=========================================================
\AxiomC{$e_2\leadsto e_2'$}
\RightLabel{\textsc{CongFunArg}}
\UnaryInfC{$v\ e_2\leadsto v\ e_2'$}
\DisplayProof$\qquad$ %=========================================================
\AxiomC{{\color{white} $[]$}} %placeholder
\RightLabel{\textsc{FunEval}}
\UnaryInfC{$(\lambda x:T.\ e)\ v\leadsto[v/x]e$}
\DisplayProof\bigbreak %========================================================
\noindent
\AxiomC{$e\leadsto e'$}
\RightLabel{\textsc{CongForall}}
\UnaryInfC{$e\ T\leadsto e'\ T$}
\DisplayProof$\qquad$ %=========================================================
\AxiomC{{\color{white} $[]$}} %placeholder
\RightLabel{\textsc{ForallEval}}
\UnaryInfC{$(\Lambda\alpha.\ e)\ T\leadsto[T/\alpha]e$}
\DisplayProof\bigbreak %========================================================
\noindent
\AxiomC{$e_\text{impl}\leadsto e'_\text{impl}$}
\RightLabel{\textsc{CongExists}}
\UnaryInfC{$\textsf{pack}_{\alpha_\text{abs}.T_\text{sig}}(T_\text{conc},e_\text{impl})\leadsto\textsf{pack}_{\alpha_\text{abs}.T_\text{sig}}(T_\text{conc},e'_\text{impl})$}
\DisplayProof\bigbreak %========================================================
\noindent
\AxiomC{$e_\text{impl}\leadsto e'_\text{impl}$}
\RightLabel{\textsc{CongExistsUnpack}}
\UnaryInfC{$\textsf{let pack}(\alpha,x)=e_\text{impl}\textsf{ in }e_\text{use}\leadsto\textsf{let pack}(\alpha,x)=e'_\text{impl}\textsf{ in }e_\text{use}$}
\DisplayProof\bigbreak %========================================================
\noindent
\AxiomC{{\color{white} $[]$}} %placeholder
\RightLabel{\textsc{ExistsEval}}
\UnaryInfC{$\textsf{let pack}(\alpha,x)=\textsf{pack}_{\alpha_\text{abs}.T_\text{sig}}(T_\text{conc},v_\text{impl})\textsf{ in }e_\text{use}\leadsto [T_\text{conc}/\alpha,v_\text{impl}/x]e_\text{use}$}
\DisplayProof

\subsection*{Church encodings}

\subsubsection*{Pairs}

\hspace{-1em}
\begin{tabular}{lcl}
$T_1\times T_2$ & \hspace{-0.5em}$\triangleq$ & \hspace{-0.5em}$\forall\alpha.\ (T_1\to T_2\to\alpha)\to\alpha$\\[0.5em]
$\langle e_1,e_2\rangle$ & \hspace{-0.5em}$\triangleq$ & \hspace{-0.5em}$\Lambda\alpha.\ \lambda k:T_1\to T_2\to\alpha.\ k\ e\ e'$\\[0.5em]
\textsf{fst }$e$ & \hspace{-0.5em}$\triangleq$ & \hspace{-0.5em}$e\ T_1\ (\lambda x:T_1.\ \lambda y:T_2.\ x)$\\[0.5em]
\textsf{snd }$e$ & \hspace{-0.5em}$\triangleq$ & \hspace{-0.5em}$e\ T_2\ (\lambda x:T_1.\ \lambda y:T_2.\ y)$
\end{tabular}

\subsubsection*{Sums}

\hspace{-1em}
\begin{tabular}{lcl}
$T_1+T_2$ & \hspace{-0.5em}$\triangleq$ & \hspace{-0.5em}$\forall\alpha.\ (T_1\to\alpha)\to(T_2\to\alpha)\to\alpha$\\[0.5em]
\textsf{L }$e$ & \hspace{-0.5em}$\triangleq$ & \hspace{-0.5em}$\Lambda\alpha.\ \lambda f:T_1\to\alpha.\ \lambda g:T_2\to\alpha.\ f\ e$\\[0.5em]
\textsf{R }$e$ & \hspace{-0.5em}$\triangleq$ & \hspace{-0.5em}$\Lambda\alpha.\ \lambda f:T_1\to\alpha.\ \lambda g:T_2\to\alpha.\ g\ e$\\[0.5em]
\end{tabular}\\
$\textsf{case}(e,\textsf{L }x\to e_1,\textsf{R }y\to e_2):T\ \triangleq\ e\ T\ (\lambda x:T_1\to T.\ e_1)\ (\lambda y:T_2\to T.\ e_2)$

\subsubsection*{Existential types}

$\exists\alpha.\ T_\text{sig}\ \triangleq\ \forall\beta.\ (\forall\alpha.\ T_\text{sig}\to\beta)\to\beta$\medbreak
\noindent
$\textsf{pack}_{\alpha_\text{abs}.T_\text{sig}}(T_\text{conc},e_\text{impl})\ \triangleq\ \Lambda\beta.\ \lambda k:\forall\alpha_\text{abs}.\ T_\text{sig}\to\beta.\ k\ T_\text{conc}\ e_\text{impl}$\medbreak
\noindent
$\textsf{let pack}(\alpha,x)=e_\text{impl}\textsf{ in }e_\text{use}:T_\text{use}\ \triangleq\ e_\text{impl}\ T_\text{use}\ (\Lambda\alpha.\ \lambda x:T_\text{sig}.\ e_\text{use})$

\subsubsection*{Booleans}

\hspace{-1em}
\begin{tabular}{lcl}
\textsf{bool} & \hspace{-0.5em}$\triangleq$ & \hspace{-0.5em}$\forall\alpha.\ \alpha\to\alpha\to\alpha$\\[0.5em]
\textsf{True} & \hspace{-0.5em}$\triangleq$ & \hspace{-0.5em}$\Lambda\alpha.\ \lambda x:\alpha.\ \lambda y:\alpha.\ x$\\[0.5em]
\textsf{False} & \hspace{-0.5em}$\triangleq$ & \hspace{-0.5em}$\Lambda\alpha.\ \lambda x:\alpha.\ \lambda y:\alpha.\ y$\\[0.5em]
\end{tabular}\\
\textsf{if }$e$\textsf{ then }$e_1$\textsf{ else }$e_2:T\ \triangleq\ e\ T\ e_1\ e_2$

\subsubsection*{Natural numbers}

\hspace{-1em}
\begin{tabular}{lcl}
$\mathbb{N}$ & \hspace{-0.5em}$\triangleq$ & \hspace{-0.5em}$\forall\alpha.\ \alpha\to(\alpha\to\alpha)\to\alpha$\\[0.5em]
\textsf{zero} & \hspace{-0.5em}$\triangleq$ & \hspace{-0.5em}$\Lambda\alpha.\ \lambda z:\alpha.\ \lambda s:\alpha\to\alpha.\ z$\\[0.5em]
\textsf{succ}$(e)$ & \hspace{-0.5em}$\triangleq$ & \hspace{-0.5em}$\Lambda\alpha.\ \lambda z:\alpha.\ \lambda s:\alpha\to\alpha.\ s\ (e\ \alpha\ z\ s)$\\[0.5em]
\end{tabular}\\
$\textsf{iter}(e,\textsf{zero}\to e_\textsf{z},\textsf{succ}(x)\to e_\textsf{s}):T\ \triangleq\ e\ T\ e_\textsf{z}\ (\lambda x:T.\ e_\textsf{s})$

\subsubsection*{Lists}

\hspace{-1em}
\begin{tabular}{lcl}
\textsf{list }$T$ & \hspace{-0.5em}$\triangleq$ & \hspace{-0.5em}$\forall\alpha.\ \alpha\to(T\to\alpha\to\alpha)\to\alpha$\\[0.5em]
$[]$ & \hspace{-0.5em}$\triangleq$ & \hspace{-0.5em}$\Lambda\alpha.\ \lambda n:\alpha.\ \lambda c:T\to\alpha\to\alpha.\ n$\\[0.5em]
$e::e'$ & \hspace{-0.5em}$\triangleq$ & \hspace{-0.5em}$\Lambda\alpha.\ \lambda n:\alpha.\ \lambda c:T\to\alpha\to\alpha.\ c\ e\ (e'\ \alpha\ n\ c)$\\[0.5em]
\end{tabular}\\
$\textsf{fold}(e,[]\to e_\textsf{n},x::r\to e_\textsf{c}):T'\ \triangleq\ e\ T'\ e_\textsf{n}\ (\lambda x:T.\ \lambda r:T'.\ e_\textsf{c})$

\newpage

\subsection{Monadic $\lambda$-Calculus for State}

\subsection*{Syntax}

\hspace{-1em}
\begin{tabular}{lccl}
Types          & $T$ & $::=$ & $1\ |\ \mathbb{N}\ |\ T_1\to T_2\ |\ \textsf{ref }T\ |\ \textsf{M }T$\\[0.5em]
Pure Terms     & $e$ & $::=$ & $\langle\rangle\ |\ n\ |\ \lambda x:T.\ e\ |\ e_1\ e_2\ |\ l\ |\ \{t\}$\\[0.5em]
Impure Terms   & $t$ & $::=$ & $\textsf{new }e\ |\ !e\ |\ e:=e'\ |\ \textsf{let }x=e;t\ |\ \textsf{return }e$\\[0.5em]
Values         & $v$ & $::=$ & $\langle\rangle\ |\ n\ |\ \lambda x:T.\ e\ |\ l\ |\ \{t\}$\\[0.5em]
Stores         & $\sigma$ & $::=$ & $\cdot\ |\ \sigma,l:v$\\[0.5em]
Contexts       & $\Gamma$ & $::=$ & $\cdot\ |\ \Gamma,x:T$\\[0.5em]
Store Typings  & $\Sigma$ & $::=$ & $\cdot\ |\ \Sigma,l:T$
\end{tabular}

\subsection*{Typing rules}

\subsubsection*{Pure terms}

\vspace{0.5em}
\AxiomC{$x:T\in\Gamma$}
\RightLabel{\textsc{Hyp}}
\UnaryInfC{$\Sigma;\Gamma\vdash x:T$}
\DisplayProof$\qquad$ %=========================================================
\AxiomC{{\color{white} $\langle\rangle$}} % placeholder
\RightLabel{\textsc{1I}}
\UnaryInfC{$\Sigma;\Gamma\vdash\langle\rangle:1$}
\DisplayProof$\qquad$ %=========================================================
\AxiomC{{\color{white} $\langle\rangle$}} % placeholder
\RightLabel{\textsc{$\mathbb{N}$I}}
\UnaryInfC{$\Sigma;\Gamma\vdash n:\mathbb{N}$}
\DisplayProof\bigbreak %========================================================
\noindent
\AxiomC{$\Sigma;\Gamma,x:T\vdash e:T'$}
\RightLabel{\textsc{$\to$I}}
\UnaryInfC{$\Sigma;\Gamma\vdash\lambda x:T.\ e:T\to T'$}
\DisplayProof$\qquad$ %=========================================================
\AxiomC{$\Sigma;\Gamma\vdash e_1:T\to T'$}
\AxiomC{$\Sigma;\Gamma\vdash e_2:T$}
\RightLabel{\textsc{$\to$E}}
\BinaryInfC{$\Sigma;\Gamma\vdash e_1\ e_2:T'$}
\DisplayProof\bigbreak %========================================================
\noindent
\AxiomC{$l:T\in\Sigma$}
\RightLabel{\textsc{RefBar}}
\UnaryInfC{$\Sigma;\Gamma\vdash l:\textsf{ref }T$}
\DisplayProof$\qquad$ %=========================================================
\AxiomC{$\Sigma;\Gamma\vdash t\div T$}
\RightLabel{\textsc{MI}}
\UnaryInfC{$\Sigma;\Gamma\vdash \{t\}:\textsf{M }T$}
\DisplayProof

\subsubsection*{Impure terms}

\vspace{0.5em}
\AxiomC{$\Sigma;\Gamma\vdash e:T$}
\RightLabel{\textsc{RefI}}
\UnaryInfC{$\Sigma;\Gamma\vdash \textsf{new }e\div\textsf{ref }T$}
\DisplayProof$\quad$ %==========================================================
\AxiomC{$\Sigma;\Gamma\vdash e:\textsf{ref }T$}
\RightLabel{\textsc{RefGet}}
\UnaryInfC{$\Sigma;\Gamma\vdash !e\div T$}
\DisplayProof$\quad$ %==========================================================
\AxiomC{$\Sigma;\Gamma\vdash e:\textsf{ref }T$}
\AxiomC{$\Sigma;\Gamma\vdash e':T$}
\RightLabel{\textsc{RefSet}}
\BinaryInfC{$\Sigma;\Gamma\vdash e:=e'\div 1$}
\DisplayProof\bigbreak %========================================================
\noindent
\AxiomC{$\Sigma;\Gamma\vdash e:T$}
\RightLabel{\textsc{MRet}}
\UnaryInfC{$\Sigma;\Gamma\vdash \textsf{return }e\div T$}
\DisplayProof$\qquad$ %=========================================================
\AxiomC{$\Sigma;\Gamma\vdash e:\textsf{M }T$}
\AxiomC{$\Sigma;\Gamma,x:T\vdash t\div T'$}
\RightLabel{\textsc{MLet}}
\BinaryInfC{$\Sigma;\Gamma\vdash \textsf{let }x=e;t\div T'$}
\DisplayProof

\subsubsection*{Store and configuration}

\vspace{0.5em}
\AxiomC{{\color{white} $\Sigma\vdash\cdot$}} %placeholder
\RightLabel{\textsc{StoreNil}}
\UnaryInfC{$\Sigma\vdash\cdot:\cdot$}
\DisplayProof$\quad$ %==========================================================
\AxiomC{$\Sigma\vdash\sigma':\Sigma'$}
\AxiomC{$\Sigma;\cdot\vdash v:T$}
\RightLabel{\textsc{StoreCons}}
\BinaryInfC{$\Sigma\vdash(\sigma',l:v):(\Sigma',l:T)$}
\DisplayProof$\quad$ %==========================================================
\AxiomC{$\Sigma\vdash\sigma:\Sigma$}
\AxiomC{$\Sigma;\cdot\vdash t\div T$}
\RightLabel{\textsc{ConfigOK}}
\BinaryInfC{$\langle\sigma;t\rangle:\langle\Sigma;T\rangle$}
\DisplayProof

\subsection*{Operational semantics}

\subsubsection*{Pure terms}

\vspace{0.5em}
\AxiomC{$e_1\leadsto e_1'$}
\UnaryInfC{$e_1\ e_2\leadsto e_1'\ e_2$}
\DisplayProof$\qquad$ %=========================================================
\AxiomC{$e_2\leadsto e_2'$}
\UnaryInfC{$v\ e_2\leadsto v\ e_2'$}
\DisplayProof$\qquad$ %=========================================================
\AxiomC{{\color{white} $[]$}} %placeholder
\UnaryInfC{$(\lambda x:T.\ e)\ v\leadsto[v/x]e$}
\DisplayProof

\subsubsection*{Impure terms}

\vspace{0.5em}
\AxiomC{$e\leadsto e'$}
\UnaryInfC{$\langle\sigma;\textsf{new }e\rangle\leadsto\langle\sigma;\textsf{new }e'\rangle$}
\DisplayProof$\qquad$ %=========================================================
\AxiomC{$l\notin\text{dom}(\sigma)$}
\UnaryInfC{$\langle\sigma;\textsf{new }v\rangle\leadsto\langle(\sigma,l:v);\textsf{return }l\rangle$}
\DisplayProof\bigbreak %========================================================
\noindent
\AxiomC{$e\leadsto e'$}
\UnaryInfC{$\langle\sigma;!e\rangle\leadsto\langle\sigma;!e'\rangle$}
\DisplayProof$\qquad$ %=========================================================
\AxiomC{$l:v\in\text{dom}(\sigma)$}
\UnaryInfC{$\langle\sigma;!l\rangle\leadsto\langle\sigma;\textsf{return }v\rangle$}
\DisplayProof\bigbreak %========================================================
\noindent
\AxiomC{$e_1\leadsto e_1'$}
\UnaryInfC{$\langle\sigma;e_1:=e_2\rangle\leadsto\langle\sigma;e_1':=e_2\rangle$}
\DisplayProof$\qquad$ %=========================================================
\AxiomC{$e_2\leadsto e_2'$}
\UnaryInfC{$\langle\sigma;v:=e_2\rangle\leadsto\langle\sigma;v:=e_2'\rangle$}
\DisplayProof\bigbreak %========================================================
\noindent
\AxiomC{{\color{white} $\langle\rangle$}} %placeholder
\UnaryInfC{$\langle(\sigma,l:v,\sigma');l:=v\rangle\leadsto\langle(\sigma,l:v',\sigma');\textsf{return }\langle\rangle\rangle$}
\DisplayProof\bigbreak %========================================================
\noindent
\AxiomC{$e\leadsto e'$}
\UnaryInfC{$\langle\sigma;\textsf{return }e\rangle\leadsto\langle\sigma;\textsf{return }e'\rangle$}
\DisplayProof$\qquad$ %=========================================================
\AxiomC{$e\leadsto e'$}
\UnaryInfC{$\langle\sigma;\textsf{let }x=e;t\rangle\leadsto\langle\sigma;\textsf{let }x=e';t\rangle$}
\DisplayProof\bigbreak %========================================================
\noindent
\AxiomC{{\color{white} $\langle\rangle$}} %placeholder
\UnaryInfC{$\langle\sigma;\textsf{let }x=\{\textsf{return v}\};t\rangle\leadsto\langle\sigma;[v/x]t\rangle$}
\DisplayProof$\qquad$ %=========================================================
\AxiomC{$\langle\sigma;t_1\rangle\leadsto\langle\sigma';t_1'\rangle$}
\UnaryInfC{$\langle\sigma;\textsf{let }x=\{t_1\};t_2\rangle\leadsto\langle\sigma';\textsf{let }x=\{t_1'\};t_2\rangle$}
\DisplayProof\bigbreak %========================================================

\subsection{Monadic $\lambda$-Calculus for I/O}

\subsection*{Syntax}

\hspace{-1em}
\begin{tabular}{lccl}
Types          & $T$ & $::=$ & $1\ |\ \mathbb{N}\ |\ T_1\to T_2\ |\ \textsf{M}_\textsf{IO}\ T$\\[0.5em]
Pure Terms     & $e$ & $::=$ & $\langle\rangle\ |\ n\ |\ \lambda x:T.\ e\ |\ e_1\ e_2\ |\ l\ |\ \{t\}$\\[0.5em]
Impure Terms   & $t$ & $::=$ & $\textsf{print }e\ |\ \textsf{let }x=e;t\ |\ \textsf{return }e$\\[0.5em]
Values         & $v$ & $::=$ & $\langle\rangle\ |\ n\ |\ \lambda x:T.\ e\ |\ \{t\}$\\[0.5em]
Output Tokens  & $\omega$ & $::=$ & $\cdot\ |\ n::\omega$\\[0.5em]
Contexts       & $\Gamma$ & $::=$ & $\cdot\ |\ \Gamma,x:T$
\end{tabular}

\subsection*{Typing rules}

\subsubsection*{Pure terms}

\vspace{0.5em}
\AxiomC{$x:T\in\Gamma$}
\RightLabel{\textsc{Hyp}}
\UnaryInfC{$\Gamma\vdash x:T$}
\DisplayProof$\qquad$ %=========================================================
\AxiomC{{\color{white} $\langle\rangle$}} % placeholder
\RightLabel{\textsc{1I}}
\UnaryInfC{$\Gamma\vdash\langle\rangle:1$}
\DisplayProof$\qquad$ %=========================================================
\AxiomC{{\color{white} $\langle\rangle$}} % placeholder
\RightLabel{\textsc{$\mathbb{N}$I}}
\UnaryInfC{$\Gamma\vdash n:\mathbb{N}$}
\DisplayProof\bigbreak %========================================================
\noindent
\AxiomC{$\Gamma,x:T\vdash e:T'$}
\RightLabel{\textsc{$\to$I}}
\UnaryInfC{$\Gamma\vdash\lambda x:T.\ e:T\to T'$}
\DisplayProof$\qquad$ %=========================================================
\AxiomC{$\Gamma\vdash e_1:T\to T'$}
\AxiomC{$\Gamma\vdash e_2:T$}
\RightLabel{\textsc{$\to$E}}
\BinaryInfC{$\Gamma\vdash e_1\ e_2:T'$}
\DisplayProof$\qquad$ %=========================================================
\AxiomC{$\Gamma\vdash t\div T$}
\RightLabel{\textsc{MI}}
\UnaryInfC{$\Gamma\vdash \{t\}:\textsf{M}_\textsf{IO}\ T$}
\DisplayProof

\subsubsection*{Impure terms}

\vspace{0.5em}
\AxiomC{$\Gamma\vdash e:\mathbb{N}$}
\RightLabel{\textsc{MPrint}}
\UnaryInfC{$\Gamma\vdash \textsf{print }e\div 1$}
\DisplayProof$\qquad$ %=========================================================
\AxiomC{$\Gamma\vdash e:T$}
\RightLabel{\textsc{MRet}}
\UnaryInfC{$\Gamma\vdash \textsf{return }e\div T$}
\DisplayProof$\qquad$ %=========================================================
\AxiomC{$\Gamma\vdash e:\textsf{M}_\textsf{IO}\ T$}
\AxiomC{$\Gamma,x:T\vdash t\div T'$}
\RightLabel{\textsc{MLet}}
\BinaryInfC{$\Gamma\vdash \textsf{let }x=e;t\div T'$}
\DisplayProof

\subsection*{Operational semantics}

\subsubsection*{Pure terms}

\vspace{0.5em}
\AxiomC{$e_1\leadsto e_1'$}
\UnaryInfC{$e_1\ e_2\leadsto e_1'\ e_2$}
\DisplayProof$\qquad$ %=========================================================
\AxiomC{$e_2\leadsto e_2'$}
\UnaryInfC{$v\ e_2\leadsto v\ e_2'$}
\DisplayProof$\qquad$ %=========================================================
\AxiomC{{\color{white} $[]$}} %placeholder
\UnaryInfC{$(\lambda x:T.\ e)\ v\leadsto[v/x]e$}
\DisplayProof

\subsubsection*{Impure terms}

\vspace{0.5em}
\AxiomC{$e\leadsto e'$}
\UnaryInfC{$\langle\omega;\textsf{print }e\rangle\leadsto\langle\omega;\textsf{print }e'\rangle$}
\DisplayProof$\qquad$ %=========================================================
\AxiomC{{\color{white} $\langle\rangle$}} %placeholder
\UnaryInfC{$\langle\omega;\textsf{print }n\rangle\leadsto\langle(n::\omega);\textsf{return }\langle\rangle\rangle$}
\DisplayProof\bigbreak %========================================================
\noindent
\AxiomC{$e\leadsto e'$}
\UnaryInfC{$\langle\omega;\textsf{return }e\rangle\leadsto\langle\omega;\textsf{return }e'\rangle$}
\DisplayProof$\qquad$ %=========================================================
\AxiomC{$e\leadsto e'$}
\UnaryInfC{$\langle\omega;\textsf{let }x=e;t\rangle\leadsto\langle\omega;\textsf{let }x=e';t\rangle$}
\DisplayProof\bigbreak %========================================================
\noindent
\AxiomC{{\color{white} $\langle\rangle$}} %placeholder
\UnaryInfC{$\langle\omega;\textsf{let }x=\{\textsf{return v}\};t\rangle\leadsto\langle\omega;[v/x]t\rangle$}
\DisplayProof$\qquad$ %=========================================================
\AxiomC{$\langle\omega;t_1\rangle\leadsto\langle\omega';t_1'\rangle$}
\UnaryInfC{$\langle\omega;\textsf{let }x=\{t_1\};t_2\rangle\leadsto\langle\omega';\textsf{let }x=\{t_1'\};t_2\rangle$}
\DisplayProof\newpage %=========================================================

\end{document}


\documentclass[../PHYS306Notes.tex]{subfiles}

\begin{document}
\section{Lecture 6}
\subsection{Lecture Notes - Charged Particles in EM Fields}
\subsubsection{Newton's Law in the presence of EM fields}
Newton says:
\[m\ddot{\v{r}} = q(\v{E} + \dot{\v{r}}\times \v{B})\] for an electric field $\v{E}$ and magnetic field $\v{B}$. This is just the sum of the Coloumb force and Lorentz force.

\subsubsection{Scalar and Vector Potentials}
We will now apply the ideas of vector potential (for the magnetic field) from PHYS 301 to this situation. Recall that we defined the vector potential $\v{A}$ as \[\v{B} = \curl{\v{A}}\].
We also recall Faraday's Law, which states (combined with the definition of the potential above) that:
\[\curl{\v{E}} = -\dpd{\v{B}}{t} = -\dpd{\curl{\v{A}}}{t}\]
Interchanging the order of taking the curl and taking the time derivative, we see that:
\[\curl{\left(\v{E} + \dpd{\v{A}}{t}\right)} = \v{0}\]
So there exists a scalar potential $V$ such that:
\[-\grad{V} = \v{E} + \dpd{\v{A}}{t}\]
Which comes from the fact that a curl of a gradient is zero. We can rewrite this to say that:
\[\v{E} = -\grad{V} - \dpd{\v{A}}{t}\]
Now, we construct $\LL$ such that we get the Lorentz Force. Generalize $U = qV$ to get:
\[U' = qV - q\dot{\v{r}}\cdot\v{A}\]
And we use $\LL' = T - U'$ as our Lagrangian.

Next week Monday, we will look at symmetry properties of the Lagrangian and how we can use this to derive conservation laws. On Wednesday we will look at the method of using Lagrange multipliers for constrained systems. On Friday we will review damped oscillators before moving into a discussion of coupled oscillators.

\end{document}


\documentclass{llncs}
\usepackage[dvips,final]{graphics}
\usepackage{amsmath}
\usepackage{times}
%
%
%
\begin{document}
\mainmatter

%
\title{An Image Inpainting Technique based on the Fast Marching Method}
\titlerunning{An Image Inpainting Technique}
\author{Alexandru Telea}

\authorrunning{Alexandru Telea}
\tocauthor{Alexandru Telea (Eindhoven University of Technology)}

\institute{Department of Mathematics and Computer Science,\\
Eindhoven University of Technology, \\
Den Dolech 2,Eindhoven 5600 MB, The Netherlands, \\
\email{alext@win.tue.nl},
\texttt{http://www.win.tue.nl/$\sim$alext}}
%
%
\maketitle
%
%
\begin{abstract}
%
  Digital inpainting provides a means for reconstruction of small
damaged portions of an image. Although the inpainting basics are
straightforward, most inpainting techniques published in
the literature are complex to understand and implement.
We present here a new algorithm for digital inpainting based on the fast
marching method for level set applications. Our algorithm is very
simple to implement, is fast, and produces nearly identical results as
more complex, and usually slower, known methods. Source code is available online.
\end{abstract}
%
%
\section{Introduction}
%
%
 Digital inpainting, the technique of reconstructing small damaged portions
of an image, has received considerable attention in the last years.
Digital inpainting serves a wide range of
applications, such as removing text and logos from still images or videos, reconstructing
scans of deteriorated images by removing scratches or stains, or creating
artistic effects.
 Most inpainting methods work as follows. First, the
image regions to be inpainted are selected, usually in a
manual way. Next, color information is propagated inwards from the region
boundaries, i.e. the known image information is used to fill-in the missing areas.
In order to produce a perceptually plausible reconstruction, an inpainting
technique should attempt to continue the
\textit{isophotes} (lines of equal gray value) as smoothly as possible inside the
reconstructed region. In other words, the missing region should be inpainted
such that the inpainted gray value and gradient extrapolate
the gray value and gradient outside this region.

Several inpainting methods are based on the above ideas.
In \cite{bertalmio1,bertalmio2}, the image smoothness
information, estimated by the image Laplacian, is propagated along the
isophotes directions, estimated by the image gradient rotated with 90
degrees. The \textit{Total Variational} (TV) model~\cite{chan} uses an
Euler-Lagrange equation coupled with anisotropic diffusion to maintain the
isophotes' directions. The \textit{Curvature-Driven Diffusion} (CCD)
model~\cite{chan2} enhances the TV method to drive diffusion along the isophotes' directions
and thus allows inpainting thicker regions. All above methods essentially
solve a partial differential equation (PDE) that describes
the color propagation inside the missing region, subject to various
heuristics that attempt to preserve the
isophotes' directions. Preserving isophotes is, however desirable, never
perfectly attained in practice. The main problem is that both isophote
estimation and information propagation are subject to numerical diffusion.
Diffusion is desirable as it stabilizes the PDEs to be solved, but leads
inevitably to a cetain amount of blurring of the inpainted area.

A second type of methods~\cite{oliveira}
repeatedly convolves a simple 3x3 filter over the missing regions to
diffuse known image information to the missing pixels.

However impressive, the above methods have several drawbacks that preclude
their direct use in practice. The PDE-based methods require implementing non-trivial iterative numerical
methods and techniques, such as anisotropic diffusion and multiresolution schemes \cite{bertalmio1}.
Little or no information is given on practical implementation details such as various thresholds or
discretization methods, although some steps are mentioned as numerically unstable.
Moreover, such methods are quite slow, e.g. a few minutes for the relatively
small inpainting region shown in Fig.~\ref{fig:example3}.
In contrast, the convolution-based method described in~\cite{oliveira}
is simple to implement and fast. However, this method has no provisions for
preserving the isophotes' directions.
High-gradient image areas must be manually selected before inpainting and
separately treated in order not to be blurred.

  We propose a new inpainting algorithm based on propagating an image smoothness estimator along the
image gradient, similarly to~\cite{bertalmio1}. We estimate the image
smoothness as a weighted average over a known image neighborhood of the pixel to
inpaint. We treat the missing regions as level sets and use
the fast marching method (FMM) described in~\cite{sethian} to propagate the
image information. Our approach has several advantages:
\begin{itemize}
  \item it is very simple to implement (the complete pseudocode is given here).
  \item it is considerably faster than other inpainting methods
       --- processing an 800x600 image (Fig.~\ref{fig:example3}) takes under 3 seconds on a 800 MHz PC.
  \item it produces very similar results as compared to the other methods.
  \item it can easily be customized to use different local inpainting strategies.
\end{itemize}
%
%
 In Section~\ref{sec:method}, we describe our method. Section~\ref{sec:discussion}
presents several results, details our method's advantages and limitations
in comparison to other methods, and discusses possible enhancements. Source
code of a sample method implementation is available online at the address
listed at the end of the paper.
%
%
	\begin{figure}[h] \centering
	\resizebox{1.0\textwidth}{!}{\includegraphics{EPS/example3.eps}}
	\caption{800x600 image inpainted in under 3 seconds}
	\label{fig:example3}
        \end{figure}
%
%
%
%
\section{Method}
\label{sec:method}
%
This section describes our inpainting method. First,
we introduce the mathematical model we base our inpainting
on (Sec.~\ref{sec:model}). Next, we describe how the missing regions are
inpainted using the FMM (Sec.~\ref{sec:fmm}). Finally, we detail the
implementation of inpainting one point on the missing region's boundary
(Sec.~\ref{sec:inp_function}).
%
\subsection{Mathematical model}
\label{sec:model}
%
 To explain our method, consider Fig.~\ref{fig:method}, in which one must
inpaint the point $p$ situated on the boundary $\partial\Omega$ of the
region to inpaint $\Omega$. Take a small neighborhood $B_{\varepsilon}(p)$
of size $\varepsilon$ of the known image around $p$
(Fig.~\ref{fig:method}~a). As described in~\cite{bertalmio1,oliveira,chan},
the inpainting of $p$ should be determined by the values
of the known image points close to $p$, i.e. in $B_{\varepsilon}(p)$.
We first consider gray value images, color images being a natural extension (see Sec.~\ref{sec:implem}).
%
	\begin{figure}[h] \centering
	\resizebox{0.65\textwidth}{!}{\includegraphics{EPS/method.eps}}
	\caption{The inpainting principle}
	\label{fig:method}
        \end{figure}
%
For $\varepsilon$ small enough, we consider a first order approximation $I_q(p)$
of the image in point $p$, given the image $I(q)$ and gradient $\nabla I(q)$
values of point $q$ (Fig~\ref{fig:method}~b):
%
%
\begin{equation}
   I_q(p) = I(q) + \nabla I(q) (p-q)
\label{eqn:linear}
\end{equation}
%
%
Next, we inpaint point $p$ as function of all points $q$ in
$B_{\varepsilon}(p)$ by summing the estimates of all points $q$,
weighted by a normalized weighting function $w(p,q)$:
%
%
\begin{equation}
  I(p) = \frac{\sum_{q \in B_{\varepsilon}(p)} w(p,q) [I(q) + \nabla I(q) (p-q)]}
              {\sum_{q \in B_{\varepsilon}(p)} w(p,q)}
\label{eqn:sum}
\end{equation}
%
%
The weighting function $w(p,q)$, detailed in Sec.~\ref{sec:inp_function},
is designed such that the inpainting of $p$ propagates the gray value as well
as the sharp details of the image over $B_{\varepsilon}(p)$.
%
\subsection{Adding inpainting to the FMM}
\label{sec:fmm}
%
%
 Section~\ref{sec:model} explained how to inpaint
a point on the unknown region's boundary $\partial\Omega$ as a function of
known image pixels only. To inpaint the whole $\Omega$, we iteratively apply
iteratively apply Equation~\ref{eqn:sum} to all the discrete pixels of
$\partial\Omega$, in increasing distance from $\partial\Omega$'s initial
position $\partial\Omega_i$, and advance the boundary inside $\Omega$ until the whole
regions has been inpainted (see pseudocode in Fig.~\ref{fig:pseudo}).
Inpainting points in increasing
distance order from $\partial\Omega_i$ ensures that areas closest to known
image points are filled in first, thus mimicking manual inpainting
techniques~\cite{bertalmio1,bertalmio2}.
%
	\begin{figure}[h] \centering
	\resizebox{0.45\textwidth}{!}{\includegraphics{EPS/pseudo.eps}}
	\caption{Inpainting algorithm}
	\label{fig:pseudo}
        \end{figure}
%

Implementing the above requires thus a method that propagates
$\partial\Omega$ into $\Omega$ by advancing the pixels of $\partial\Omega$
in order of their distance to the initial boundary $\partial\Omega_i$.
For this, we use the fast marching method (FMM) developed by Sethian et al.
\cite{sethian}. In brief, the FMM is an algorithm that solves the Eikonal equation:
%
%
\begin{equation}
   |\nabla T| = 1 \mbox{~~~~~on $\Omega$,~~~~~with $T=0$ on $\partial\Omega$}
\label{eqn:eikonal}
\end{equation}
%
The solution $T$ of Eqn.~\ref{eqn:eikonal} is the distance map of the $\Omega$ pixels to the boundary
$\partial\Omega$. The level sets, or isolines, of $T$
are exactly the successive boundaries $\partial\Omega$ of the shrinking
$\Omega$ that we need for inpainting. The normal $N$ to
$\partial\Omega$, also needed for inpainting, is exactly $\nabla T$. The FMM
guarantees that pixels of $\partial\Omega$ are always processed in
increasing order of their distance-to-boundary $T$~\cite{fmm_book}, i.e.
that we always inpaint the closest pixels to the known image area first.

We prefer the FMM over other 'distance transform' (DT) methods that compute the distance map $T$ to a boundary $\partial\Omega$
(e.g.~\cite{chamfer1,chamfer2,roerdinck}). The FMM's main advantage is that it
\textit{explicit} maintains the narrowband that
separates the known from the unknown image area \textit{and} specifies which is the next pixel to
inpaint. Other DT methods compute the distance map $T$ but do not maintain an
explicit narrowband. Adding a narrowband structure to these method would
complicate their implementation, whereas the FMM provides this structure by
default.

To explain in detail our use of the FMM  --- and since the FMM is not
straightforward to implement from the reference
literature~\cite{sethian,fmm_book} --- we provide next its complete pseudocode.
  The FMM maintains a so-called \textit{narrow band} of pixels, which is
exactly our inpainting boundary $\partial\Omega$. For every image pixel, we store
its value $T$, its image gray value $I$ (both represented as floating-point
values),  and a flag $f$ that may have three values:
\begin{itemize}
  \item\textit{BAND}: the pixel belongs to the narrow band. Its $T$ value
  undergoes update.
  \item\textit{KNOWN}: the pixel is outside $\partial\Omega$, in the known
  image area. Its $T$ and $I$ values are known.
  \item\textit{INSIDE}: the pixel is inside $\partial\Omega$, in the region to inpaint. Its $T$
  and $I$ values are not yet known.
\end{itemize}
%
%
  The FMM has an initialization and propagation phase, as follows.
First, we set $T$ to zero on and outside the boundary
$\partial\Omega$ of the region to inpaint and to some large value (in practice $10^6$)
inside, and initialize $f$ over the whole image as explained
above. All $BAND$ points are inserted in a heap \texttt{NarrowBand}
sorted in ascending order of their $T$ values.
Next, we propagate the $T$, $f$, and $I$ values using the code
shown in Fig.~\ref{fig:fmm_code}.
%
	\begin{figure}[h] \centering
	\resizebox{0.75\textwidth}{!}{\includegraphics{EPS/code.eps}}
	\caption{Fast marching method used for inpainting}
	\label{fig:fmm_code}
        \end{figure}
%
%
Step 1 extracts the $BAND$ point with the smallest $T$. Step 2 marches
the boundary inwards by adding new points to it. Step 3 performs the
inpainting (see Sec.~\ref{sec:inp_function}). Step 4 propagates the value
$T$ of point $(i,j)$ to its neighbors $(k,l)$ by solving the finite difference discretization of
Eqn.~\ref{eqn:eikonal} given by:
%
\begin{equation}
  \mbox{max}(D^{-x}T,-D^{+x}T,0)^2+\mbox{max}(D^{-y}T,-D^{+y}T,0)^2 = 1
\label{eqn:quad}
\end{equation}
%
where $D^{-x}T(i,j) = T(i,j)-T(i-1,j)$ and $D^{+x}T(i,j) = T(i+1,j)-T(i,j)$
and similarly for $y$. Following the upwind idea of Sethian~\cite{sethian},
we solve Eqn.~\ref{eqn:quad} for $(k,l)$'s four quadrants and retain
the smallest solution. Finally, step 5 (re)inserts $(k,l)$
with its new $T$ in the heap.

%
%
\subsection{Inpainting one point}
\label{sec:inp_function}
%
%
We consider now how to inpaint a newly discovered point $(k,l)$, as function of the
$KNOWN$ points around it, following the idea described in Sec.~\ref{sec:model}
(step 3 in Fig.~\ref{fig:fmm_code}, detailed in Fig.~\ref{fig:inpaint_code})).
%
	\begin{figure}[h] \centering
	\resizebox{0.7\textwidth}{!}{\includegraphics{EPS/inpaint_code.eps}}
	\caption{Inpainting one point}
	\label{fig:inpaint_code}
        \end{figure}
%
We iterate over the $KNOWN$ points in the neighborhood $B_{\varepsilon}$ of
the current point $(i,j)$ and compute $I(i,j)$ following
Eqn.~\ref{eqn:sum}. The image gradient $\nabla I$ (\texttt{gradI} in the
code) is estimated by central differences. As stated in Sec.~\ref{sec:model}, the design of the
weighting function $w(p,q)$ is crucial to propagate the sharp image details and the smooth
zones as such into the inpainted zone. We design $w(p,q) = dir(p,q) \cdot
dst(p,q) \cdot lev(p,q)$ as a product of three factors:
%
%
%\begin{equation}
%  w(p,q)=dir(p,q) \cdot dst(p,q) \cdot lev(p,q)
%\end{equation}
%
%
\begin{eqnarray*}
  dir(p,q) &=& \frac{p-q}{||p-q||} \cdot N(p)\\
  dst(p,q) &=& \frac{d_0^2}{||p-q||^2} \\
  lev(p,q) &=& \frac{T_0}{1+|T(p)-T(q)|}
\end{eqnarray*}
%
%
The \textit{directional} component $dir(p,q)$ ensures that the
contribution of the pixels close to the normal direction
$N=\nabla T$ (\texttt{gradT} in the code), i.e. close to the
FMM's information propagation direction, is higher than for those further from
$N$. The \textit{geometric distance} component $dst(p,q)$ decreases the
contribution of the pixels geometrically farther from $p$.
The \textit{level set} distance
component $lev(p,q)$ ensures that pixels close to the contour
through $p$ contribute more than further pixels. Both $dst$ and $lev$ are
relative with respect to the reference distances $d_0$ and $T_0$. In practice, we set
$d_0$ and $T_0$ to the inter-pixel distance, i.e. to 1.
Overall, the above factors model the manual inpainting heuristics~\cite{bertalmio1} that
describe how to paint a point by strokes bringing color from a small region
around it.

For $\varepsilon$ up to about
6 pixels, i.e. when inpainting thin regions, $dst$ and $lev$ have
a weak effect. For thicker regions to inpaint, such as
Fig.~\ref{fig:example2}~d where we used an $\varepsilon$ of 12 pixels,
using $dst$ and $lev$ provides better
results than using $dir$ alone. The above is clearly visible in
Fig.~\ref{fig:ellipse}, on a test image taken from~\cite{bertalmio1},  where
the missing ring shaped region is over 30 pixels thick.
Figure~\ref{fig:ellipse}~c shows, on an image detail, the effect of $dir$ alone. The results are
somewhat less blurry when $dir$ and $dst$ (Fig.~\ref{fig:ellipse}~d) or
$dir$ and $lev$ (Fig.~\ref{fig:ellipse}~e) are used together. The inpainting
is visually the best when all three components are used (Fig.~\ref{fig:ellipse}~f).
%
%
	\begin{figure}[h] \centering
	\resizebox{0.81\textwidth}{!}{\includegraphics{EPS/ellipse.eps}}
	\caption{Thick region to inpaint (a) and result (b). Effect of weighting functions:
         direction (c), direction and geometric distance (d), direction and level set distance (e),
         direction, geometric, and level set distance (f)}
	\label{fig:ellipse}
        \end{figure}
%
%
%
%
\subsection{Implementation details}
\label{sec:implem}
%
%
  Several implementation details are important. First, we compute the
boundary normal $N=\nabla T$ by numerical derivation of the field $T$
computed by the FMM. Derivating $T$ on the fly as it is computed by the
FMM is unstable, since we are not guaranteed that a large enough
neighborhood around the current point contains only $KNOWN$ points.
We first run the FMM outside the initial inpainting boundary
$\partial\Omega$ and obtain the distance field $T_{out}$. Since we use only
those points closer to $\partial\Omega$ than $\varepsilon$, we run the FMM
outside $\partial\Omega$ only until we reach $T > \varepsilon$. This
restricts the FMM computations to a band of thickness $\varepsilon$ around
$\partial\Omega$, thus speeding up the process. Next, we run the FMM
inside $\partial\Omega$ and obtain $T_{in}$. The field $T$ over the whole
image is given by:
%
\begin{equation}
T(p) = \left\{
      \begin{array}{rcl}
      &T_{in}&(p)~~~\mbox{if}~p \in \Omega\\
      -&T_{out}&(p)~~~\mbox{if}~p \notin \Omega.
\end{array}
\right.
\end{equation}
%
Next, we smooth $T$ by a 3x3 tent filter, and then compute $\nabla T$ by
central differences.

  The value of $\varepsilon$ giving the size of $B$ usually ranges from 3 to 10
pixels. This corresponds with the 'thickness' of the regions to inpaint,
which is usually less than 15 pixels. Higher values blur the sharp details
to be reconstructed by inpainting, although they are useful when inpainting
thicker regions.

  The test \texttt{f(k,l)!=OUTSIDE} in Fig.~\ref{fig:inpaint_code} that
restricts $B_{\varepsilon}$ to the known image points can be changed to
\texttt{f(k,l)==KNOWN}. The results are visually identical, as
$B_{\varepsilon}$ contains very few $BAND$ pixels. However, one would use the second test
if the initial $\partial\Omega$ corresponds to unknown image pixels.

The \texttt{NarrowBand} sorted heap (Sec.~\ref{sec:fmm}) is straightforwardly implemented using
the C++ STL \texttt{multimap} container~\cite{stl}. Finally, for color (RGB) images we apply the presented
method separately for each color channel.
%
\section{Discussion}
\label{sec:discussion}
%
%
   We have compared our inpainting method with the methods presented
by Bertalmio et al. in~\cite{bertalmio1} and Oliveira et al. in~\cite{oliveira}, further denoted by
BSCB and OBMC, by running it on the same input images (see Figures~\ref{fig:example1}
and~\ref{fig:example2}~a-c). For BSCB we used the implementation publicly available
at~\cite{implem}, whereas we reimplemented OBMC ourselves.
Our method produced visually nearly identical results with BSCB. Compare, for example, the
inpaintings in Fig.~\ref{fig:example1}~c,d (Fig.~\ref{fig:example1}~e,f in
detail) and Fig.~\ref{fig:example2}~b,c (Fig.~\ref{fig:example2}~g,h in detail).
In contrast, the results of OBMC (shown in~\cite{oliveira})
were visibly more blurry for regions thicker than 6 pixels.
%
	\begin{figure}[h] \centering
	\resizebox{1.05\textwidth}{!}{\includegraphics{EPS/example1.eps}}
	\caption{Lincoln 'cracked photo' inpainting}
	\label{fig:example1}
        \end{figure}
%
%
  The running time for our method was in all cases much lower than for
BSCB. Our C++ implementation took under 3 seconds on an 800 MHz PC for a 800x600
color image with about 15\% pixels to inpaint (Fig.~\ref{fig:example3}).
On the same input, the original BSCB method, published in~\cite{bertalmio1}, is reported to take under 5 minutes on a
300 MHz PC. The BSCB implementation we used~\cite{implem}, which is
however mentioned to be unoptimized by its authors, took between 2.5 and 3 minutes on the 800
MHz PC, depending on its various parameter settings.
In contrast, OBMC takes in all cases about the same time as our method.
The above matches the fact that both OBMC and our method are linear
in the inpainted region's size.

The main limitation of our method (applicable to the methods BSCB and
OBMC mentioned here too) is the blurring produced when inpainting
regions thicker than 10-15 pixels, especially visible when sharp isophotes
intersect the region's boundary almost tangentially. See for example the
inpainting in Fig.~\ref{fig:example2}~d,e (in
detail in Fig.~\ref{fig:example2}~i,j). The above is caused by the linear
and local character of our method. Techniques using an explicit nonlinear
and/or global image model, such as the TV~\cite{chan} and the CDD~\cite{chan2} methods,
achieve better results, at the cost of considerably more complex implementations.
%
	\begin{figure}[h] \centering
	\resizebox{1.05\textwidth}{!}{\includegraphics{EPS/example2.eps}}
	\caption{Inpainting examples. Damaged photo (a), inpainting by method BSCB (b),
         our method (c), and close-ups (f,g,h).
         Damaged photo (d), distance-weighted inpainting (e), and close-ups (i,j)}.
	\label{fig:example2}
        \end{figure}

Overall, the presented inpainting method is simple to implement (our complete C++ code is
about 500 lines), is fast, and easy to customize for different inpainting strategies.
We plan to extend the method by developing new inpainting functions that
are better able to preserve the isophotes' directions. One such way is to
integrate anisotropic diffusion, e.g. following~\cite{bertalmio1}, in the FMM boundary
evolution in order to reduce the blurring for inpainting thick regions. A
second extension would be to modulate the evolution speed of the FMM, now
equal to 1, by the image anisotropy, i.e. let inpainting 'work more' on the
high detail areas than on the smooth regions.
%
%
\section*{Acknowledgements}
%
  We are indebted to prof. J. J. van Wijk from the Department of
Mathematics and Computer Science of the Eindhoven University of Tehcnology
for his numerous suggestions on how to improve this paper.
%
\section*{Web Information}
%
  Source code of a sample C++ implementation of the inpainting method
described here is available at\\
\texttt{http://www.acm.org/jgt/papers/Telea03.html}
%
%
%
%-------------------------------------------------------------------------------
%
\begin{thebibliography}{13}
%
\bibitem {bertalmio1}
   {\sc M. Bertalmio, G. Sapiro, V. Caselles, and C. Ballester},
   {\em Image Inpainting}, Proc. SIGGRAPH 2000, ACM Press, pp. 417-424.
\bibitem {bertalmio2}
   {\sc M. Bertalmio, A. L. Bertozzi, and G. Sapiro},
   {\em Navier-Stokes, Fluid Dynamics, and Image and Video Inpainting},
   Proc. ICCV 2001, IEEE CS Press, vol. 1, pp. 1335-1362.
\bibitem{chamfer1}
   {\sc G. Borgefors},
   {\em Distance transformations in arbitrary images},
   Comp. Vision, Graphics, and Image Proc., 27(3), pp. 321-345, 1984.
\bibitem{chamfer2}
   {\sc G. Borgefors},
   {\em Distance transformations in digital images},
   Comp. Vision, Graphics, and Image Proc., 34(3), pp. 344-371, 1986.
\bibitem {oliveira}
   {\sc M. Oliveira, B. Bowen, R. McKenna, and Y.-S. Chang},
   {\em Fast Digital Image Inpainting},
   Proc. VIIP 2001 (Marbella, Spain), pp. 261-266.
\bibitem {chan}
   {\sc T. Chan, J. Shen},
   {\em Mathematical Models for Local Deterministic Inpaintings},
   Tech. report CAM 00-11, Image Processing Research Group, UCLA, 2000.
\bibitem {chan2}
   {\sc T. Chan, J. Shen},
   {\em Non-texture Inpainting by Curvature-Driven Diffusions (CDD)},
   Tech. report CAM 00-35, Image Processing Research Group, UCLA, 2000.
\bibitem{roerdinck}
   {\sc A. Meijster, J. Roerdink, W. Hesselink},
   {\em A general algorithm for computing distance transforms in linear time},
   Math. Morph. and its Appls. to Image and Signal Proc., pp. 331--340, Kluwer, 2000.
\bibitem {fmm_book}
   {\sc J. A. Sethian},
   {\em Level Set Methods and Fast Marching Methods},
   Cambridge Univ. Press, 2nd edition, 1999.
\bibitem {sethian}
   {\sc J. A. Sethian},
   {\em A Fast Marching Level Set Method for Monotonically Advancing Fronts},
   Proc. Nat. Acad. Sci. vol. 93, nr. 4, pp. 1591-1595, 1996.
\bibitem{implem}
   {\sc W. Yung, A. J. Shankar},
   {\em Image Inpainting Implementation Software}, \texttt{www.bantha.org/$\sim$aj/inpainting}
\bibitem{stl}
  {\sc D. R. Musser, A. Saini},
  {\em STL Tutorial and Reference Guide: C++
   Programming with the Standard Template Library},
  Addison-Wesley Professional Computing Series, 1996.

\end{thebibliography}

%-------------------------------------------------------------------------------
%
%
%
%
\end{document}
%


\documentclass[12]{scrartcl}
\usepackage{amssymb,amsmath,gensymb,dsfont,calc,multicol,fullpage}
\makeatletter
\newcommand\Aboxed[1]{
   \@Aboxed#1\ENDDNE}
\def\@Aboxed#1&#2\ENDDNE{%
   &
   \settowidth\@tempdima{$\displaystyle#1{}$}
   \setlength\@tempdima{\@tempdima+\fboxsep+\fboxrule}
   \kern-\@tempdima
   \boxed{#1#2}
}
\makeatother

\begin{document}

\title{Homework 26, Section 4.9: 1, 2, 6, 7, 12, 18}
\author{Alex Gordon}
\date{\today}
\maketitle
\section*{Homework}
\subsection*{1. A)}
\ \ N  \ M \\
$\begin{bmatrix}  .7 & .6 \\ .3 & .4  \end{bmatrix} $ News and Music
\subsection*{1. B)}
$\begin{bmatrix}  1\\ 0  \end{bmatrix} $
\subsection*{1. C)}
Approximately 33%
\subsection*{2. A)}
1, 2, and 3 respectively at the tops and it goes to 1, 2, and 3 on the side respectively.
$\begin{bmatrix}  .6 & .2 & .2 \\ .2 & .6 & .2 \\ .2 & .2 & .6  \end{bmatrix} $
\subsection*{2. B)}
Approximately .28
\subsection*{6.}
$\begin{bmatrix}  4/7 \\ 3/7  \end{bmatrix} $ is the steady state vector.
\subsection*{7.}
$\begin{bmatrix}  1/4 \\ 1/2 \\ 1/4  \end{bmatrix} $  is the steady state vector.
\subsection*{12.}
Each food will be preferred equally, because $\begin{bmatrix}  1/3 \\ 1/3 \\ 1/3  \end{bmatrix} $ of the equal numbers in the steady state vector.
\subsection*{18.}
If alpha = beta = 0, then $\begin{bmatrix}  1\\ 0  \end{bmatrix} $ and $\begin{bmatrix}  0\\ 1  \end{bmatrix} $ are the steady state vectors. If it is not 0, then it should equal $\frac{1}{\alpha + \beta} \begin{bmatrix}  \beta \\ \alpha  \end{bmatrix}$

\end{document}