\lab{GeoPandas}{GeoPandas}

\objective{GeoPandas is a package designed to organize and manipulate geographic data, It combines the data manipulation tools of pandas with the geometric capabilities of the Shapely package. In this lab, we explore the basic data structures of GeoSeries and GeoDataFrames and their functionalities.}

\section*{Installation} %============================
GeoPandas is a new package designed to combine the functionality of pandas with Shapely, a package used for geometric manipulation.
Using GeoPandas with geographic data is very useful as it allows the user to not only compare numerical data, but also geometric attributes.
GeoPandas can be installed via \li{pip}:
\begin{lstlisting}
>>> pip install geopandas
\end{lstlisting}
However, Geopandas can be notoriously difficult to install.
%It is very finicky, which is likely a by-product of it still being in development
This is especially the case if the python environment is not carefully maintained.
Some of its dependencies can also be very difficult to install on certain systems.
Because of this, using Colab for this lab is recommended; its environment is set up in a way that makes GeoPandas very easy to install.
Otherwise, the GeoPandas documentation contains some additional options that can be used if installation difficulties occur: \url{https://geopandas.org/install.html}.

\section*{GeoSeries} %============================

A GeoSeries is a pandas Series where each entry is a set of geometric objects.
There are three classes of geometric objects inherited from the Shapely package:
\begin{enumerate}
\item Points / Multi-Points
\item Lines / Multi-Lines
\item Polygons / Multi-Polygons
\end{enumerate}
A point is used to identify objects like coordinates, where there is one small instance of the object.
A line could be used to describe objects such as roads.
A polygon could be used to identify regions, such as a country.
Multipoints, multilines, and multipolygons contain lists of points, lines, and polygons, respectively.

Since each object in the GeoSeries is also a Shapely object, the GeoSeries inherits many methods and attributes of Shapely objects.
Some of the key attributes and methods are listed in Table \ref{shapely-methods}.
These attributes and methods can be used to calculate distances, find the sizes of countries, and determine whether coordinates are within country's boundaries.
The example below uses the attribute \li{bounds} to find the maximum and minimum coordinates of Egypt in a built-in GeoDataFrame.

\begin{table}[ht]
\begin{center}
\begin{tabular}{c|c}
Method/Attribute & Description\\
\hline
\li{distance(other)}&returns minimum distance from GeoSeries to \li{other}\\
\li{contains(other)}&returns \li{True} if shape contains \li{other}\\
\li{intersects(other)}&returns \li{True} if shape intersects \li{other}\\
\li{area}&returns shape area\\
\li{convex_hull}&returns convex shape around all points in the object\\
\li{bounds} &  returns the bounding x- and y-coordinates of the object
\end{tabular}
\end{center}
\caption{Attributes and Methods for GeoSeries}
\label{shapely-methods}
\end{table}

\begin{lstlisting}
>>> import geopandas as gpd
>>> world = gpd.read_file(gpd.datasets.get_path('naturalearth_lowres'))
# Get GeoSeries for Egypt
>>> egypt = world[world['name']=='Egypt']

# Find bounds of Egypt
>>> egypt.bounds
         minx    miny         maxx      maxy
47   24.70007    22.0     36.86623  31.58568
\end{lstlisting}

\section*{Creating GeoDataFrames} %============================

The main structure used in GeoPandas is a GeoDataFrame, which is similar to a pandas DataFrame.
A GeoDataFrame has one special column called \li{geometry}, which must be a GeoSeries.
This GeoSeries column is used when a spatial method, like \li{distance()}, is used on the GeoDataFrame.

A GeoDataFrame can be made from a pandas DataFrame.
At least one of the columns in the DataFrame should contain geometric information.
This column containing geometric information can be converted to a GeoSeries using the \li{apply()} method.
At this point, the Pandas DataFrame can be cast as a GeoDataFrame.
Assign which column will be the \li{geometry} using either the \li{geometry} keyword in the constructor or the \li{set_geometry()} method afterwards.

\begin{lstlisting}
>>> import pandas as pd
>>> import geopandas as gpd
>>> from shapely.geometry import Point, Polygon

# Create a Pandas DataFrame
>>> df = pd.DataFrame({'City': ['Seoul', 'Lima', 'Johannesburg'],
...                   'Country': ['South Korea', 'Peru', 'South Africa'],
...                   'Latitude': [37.57, -12.05, -26.20],
...                   'Longitude': [126.98, -77.04, 28.04]})

# Create geometry column
>>> df['Coordinates'] = list(zip(df.Longitude, df.Latitude))

# Make geometry column Shapely objects
>>> df['Coordinates'] = df['Coordinates'].apply(Point)

# Cast as GeoDataFrame
>>> gdf = gpd.GeoDataFrame(df, geometry='Coordinates')

# Equivalently, specify the geometry after construction
# Note that set_geometry() returns a new GeoDataFrame
>>> gdf = gpd.GeoDataFrame(df)
>>> gdf = gdf.set_geometry('Coordinates')

# Display the GeoDataFrame
>>> gdf
           City       Country  Latitude  Longitude                  Coordinates
0         Seoul   South Korea     37.57     126.98   POINT (126.98000 37.57000)
1          Lima          Peru    -12.05     -77.04  POINT (-77.04000 -12.05000)
2  Johannesburg  South Africa    -26.20      28.04   POINT (28.04000 -26.20000)

# Create a polygon with all three cities as points
>>> city_polygon = Polygon(list(zip(df.Longitude, df.Latitude)))
\end{lstlisting}

A \li{GeoDataFrame} can also be made directly from a dictionary.
If the dictionary already contains geometric objects, the corresponding column can be directly set as the \li{geometry} in the constructor.
Otherwise, a column containing geometry data can be created as in the above example and then set as the \li{geometry} with the \li{set_geometry()} method.

\begin{lstlisting}
# Both of these methods create the same GeoDataFrame as above
# Directly create the GeoDataFrame from the dictionary
>>> gdf = gpd.GeoDataFrame({'City': ['Seoul', 'Lima', 'Johannesburg'],
...                   'Country': ['South Korea', 'Peru', 'South Africa'],
...                   'Latitude': [37.57, -12.05, -26.20],
...                   'Longitude': [126.98, -77.04, 28.04]})
# Create geometry column and set as the geometry
>>> gdf['Coordinates'] = list(zip(df.Longitude, df.Latitude))
>>> gdf['Coordinates'] = df['Coordinates'].apply(Point)
# inplace=True modifies gdf itself rather than returning a copy
>>> gdf.set_geometry('Coordinates', inplace=True)

# Equivalently, using a dictionary that already contains geometry objects
>>> gdf = gpd.GeoDataFrame({'City': ['Seoul', 'Lima', 'Johannesburg'],
...                   'Country': ['South Korea', 'Peru', 'South Africa'],
...                   'Coordinates': [Point(126.98,37.57),
...                       Point(-77.04,-12.05), Point(28.04,-12.05)]},
...                   geometry='Coordinates')
\end{lstlisting}

\begin{info}
Longitude is the angular measurement starting at the Prime Meridian, $0\degree$, and going to $180\degree$ to the east and $-180\degree$ to the west.
Latitude is the angle between the equatorial plane and the normal line at a given point; a point along the Equator has latitude $0$, the North Pole has latitude $+90\degree$ or $90\degree N$, and the South Pole has latitude $-90\degree$ or $90\degree S$.
\end{info}

\section*{Plotting GeoDataFrames} %============================

Information from a GeoDataFrame is plotted based on the geometry column.
Data points are displayed as geometry objects.
The following example plots the shapes in the \li{world} GeoDataFrame.

\begin{lstlisting}
# Plot world GeoDataFrame
>>> world.plot()
\end{lstlisting}

\begin{figure}[H]
\begin{center}
\includegraphics[scale=.55]{figures/worldmap.pdf}
\end{center}
\label{world-map}
\caption{World map}
\end{figure}

Multiple GeoDataFrames can be plotted at once.
This can be done by by setting one GeoDataFrame as the base of the plot and ensuring that each layer uses the same axes.
In the following example, the file \li{airports.csv}, containing the coordinates of world airports, is loaded into a GeoDataFrame and plotted on top of the boundary of the \li{world} GeoDataFrame.

\begin{lstlisting}
# Set outline of world countries as base
>>> fig,ax = plt.subplots(figsize=(10,7), ncols=1, nrows=1)
>>> base = world.boundary.plot(edgecolor='black', ax=ax, linewidth=1)

# Load airport data and convert to a GeoDataFrame
>>> airports = pd.read_csv('airports.csv')
>>> airports['Coordinates'] = list(zip(airports.Longitude, airports.Latitude))
>>> airports['Coordinates'] = airports.Coordinates.apply(Point)
>>> airports = gpd.GeoDataFrame(airports, geometry='Coordinates')

# Plot airports on top of world map
>>> airports.plot(ax=base, marker='o', color='green', markersize=1)
>>> ax.set_xlabel('Longitude')
>>> ax.set_ylabel('Latitude')
>>> ax.set_title('World Airports')
\end{lstlisting}

\begin{figure}[H]
\begin{center}
\includegraphics[scale=.6]{figures/airports.pdf}
\end{center}
\label{airport-map}
\caption{Airport map}
\end{figure}

\begin{problem}
Read in the file \li{airports.csv} as a pandas DataFrame.
Create three convex hulls around the three sets of airports listed below.
This can be done by passing in lists of the airports' coordinates to a \li{shapely.geometry.Polygon} object.

Create a new GeoDataFrame with these three Polygons as entries.
Plot this GeoDataFrame on top of an outlined world map.
\begin{itemize} %These could also be listed by 4-character ICAO codes, but listing the names makes the problem more exciting/realistic. Probably.
	\item Maio Airport, Scatsta Airport, Stokmarknes Skagen Airport, Bekily Airport, K. D. Matanzima Airport, RAF Ascension Island
	\item Oiapoque Airport, Maio Airport, Zhezkazgan Airport, Walton Airport, RAF Ascension Island, Usiminas Airport, Piloto Osvaldo Marques Dias Airport
	\item Zhezkazgan Airport, Khanty Mansiysk Airport, Novy Urengoy Airport, Kalay Airport, Biju Patnaik Airport, Walton Airport
\end{itemize}
\end{problem}

\section*{Working with GeoDataFrames} %============================

As previously mentioned, GeoDataFrames contain many of the functionalities of pandas DataFrames.
For example, to create a new column, define a new column name in the GeoDataFrame with the needed information for each GeoSeries.

\begin{lstlisting}
# Create column in the world GeoDataFrame for gdp_per_capita
>>> world['gdp_per_cap'] = world.gdp_md_est / world.pop_est
\end{lstlisting}

GeoDataFrames can utilize many pandas functionalities, and they can also be parsed by geometric manipulations.
For example, a useful way to index GeoDataFrames is with the \li{cx} indexer.
This splits the GeoDataFrame by the coordinates of each geometric object.
It is used by calling the method \li{cx} on a GeoDataFrame, followed by a slicing argument, where the first element refers to the longitude and the second refers to latitude.

\begin{lstlisting}
# Create a GeoDataFrame containing the northern hemisphere
>>> north = world.cx[:, 0:]

# Create a GeoDataFrame containing the southeastern hemisphere
>>> south_east = world.cx[0:, :0]
\end{lstlisting}

GeoSeries objects in a GeoDataFrame can also be dissolved, or merged, together into one GeoSeries based on their geometry data.
For example, all countries on one continent could be merged to create a GeoSeries containing the information of that continent.
The method designed for this is called \li{dissolve}.
It receives two parameters, \li{by} and \li{aggfunc}.
\li{by} indicates which column to dissolve along, and \li{aggfunc} tells how to combine the information in all other columns.
The default \li{aggfunc} is \li{first}, which returns the first application entry.
In the following example, we use \li{sum} as the \li{aggfunc} so that each continent is the combination of its countries.

\begin{lstlisting}
>>> world = world[['continent', 'geometry', 'gdp_per_cap']]

# Dissolve world GeoDataFrame by continent
>>> continent = world.dissolve(by = 'continent', aggfunc='sum')
\end{lstlisting}

\section*{Projections and Coloring} %============================

When plotting, GeoPandas uses the CRS (coordinate reference system) of a GeoDataFrame.
This reference system indicates how coordinates should be spaced on a plot.
Two of the most commonly used CRSs are EPSG:4326 and EPSG:3395.
EPSG:4326 is the standard latitude-longitude projection used by GPS.
EPSG:3395, also known as the Mercator projection, is the standard navigational projection.

When creating a new GeoDataFrame, it is important to set the \li{crs} attribute of the GeoDataFrame.
This allows any plots to be shown correctly.
Furthermore, GeoDataFrames being layered need to have the same CRS.
To change the CRS, use the method \li{to_crs()}.

\begin{lstlisting}
# Check CRS of world GeoDataFrame
>>> print(world.crs)
epsg:4326

# Change CRS of world to Mercator
# inplace=True ensures that we modify world instead of returning a copy
>>> world.to_crs(3395, inplace=True)
>>> print(world.crs)
epsg:3395
\end{lstlisting}

GeoPandas accepts many different CRSs; a reference can be found at \url{www.spatialreference.org}.
% I think this is helpful information, but I'm not sure
Additionally, inspecting a given CRS object in the terminal without using \li{print()} or \li{str()} can be used to get additional information about a specific CRS:\footnote{This can also be accomplished using \li{print(repr(crs))}.}
\begin{lstlisting}[mathescape]
>>> world.crs
<Projected CRS: EPSG:3395>
Name: WGS 84 / World Mercator
Axis Info [cartesian]:
- E[east]: Easting (metre)
- N[north]: Northing (metre)
Area of Use:
- name: World between 80$^\circ$S and 84$^\circ$N.
- bounds: (-180.0, -80.0, 180.0, 84.0)
Coordinate Operation:
- name: World Mercator
- method: Mercator (variant A)
Datum: World Geodetic System 1984
- Ellipsoid: WGS 84
- Prime Meridian: Greenwich
\end{lstlisting}

GeoDataFrames can also be plotted using the values in the the other attributes of the GeoSeries.
The map plots the color of each geometry object according to the value of the column selected.
This is done by passing in the parameter \li{column} into the \li{plot()} method.

\begin{lstlisting}
>>> fig, ax = plt.subplots(1, figsize=(10,4))
# Plot world based on gdp
>>> world.plot(column='gdp_md_est', cmap='OrRd', legend=True,  ax=ax)
>>> ax.set_title('World Map based on GDP')
>>> ax.set_xlabel('Longitude')
>>> ax.set_ylabel('Latitude')
>>> plt.show()
\end{lstlisting}

\begin{figure}[H]
\begin{center}
\includegraphics[scale=.7]{figures/gdp.pdf}
\end{center}
\caption{World Map Based on GDP}
\label{figure:gdp_map}
\end{figure}

\begin{problem}
The file \li{county_data.gpkg.zip} contains information about US counties.\footnote{Source: \url{http://www2.census.gov/geo/tiger/GENZ2016/shp/cb_2016_us_county_5m.zip}}
After unzipping, use the command \li{gpd.read_file('county_data.gpkg')} to create a GeoDataFrame of this information.
Each county's shape is stored in the \li{geometry} column.
Use this to plot all US counties two times, first using the default CRS and then using EPSG:5071.

Next, create a new GeoDataFrame that merges all counties within a single state.
Drop regions with the following STATEFP codes: 02, 15, 60, 66, 69, 72, 78.
Plot this GeoDataFrame to see an outline of all 48 contiguous states.
Ensure a CRS  of EPSG:5071.
\label{problem:us_counties}
\end{problem}
\begin{info}
\li{.gpkg} files are actually structured as a directory that contains several files that each contain parts of the data.
For instance, \li{county_data.gpkg} consists of the files \li{county_data.cpg}, \li{county_data.dbf}, \li{county_data.prj}, \li{county_data.shp}, and \li{county_data.shx}.
Be sure that these files are placed directly in the first level of folders, and not in further subdirectories.\\
To use this file in Google Colab, upload the zipped file and extract it with the following code:
\begin{lstlisting}
county = files.upload()
!unzip county_data.gpkg.zip
\end{lstlisting}
It then can be loaded:
\begin{lstlisting}
county_df = gpd.read_file('county_data.gpkg')
\end{lstlisting}
\end{info}

\section*{Merging GeoDataFrames} %============================

Just as multiple pandas DataFrames can be merged, multiple GeoDataFrames can be merged with attribute joins or spatial joins.
An attribute join is similar to a merge in pandas.
It combines two GeoDataFrames on a column (not the geometry column) and then combines the rest of the data into one GeoDataFrame.

\begin{lstlisting}
>>> world = gpd.read_file(geopandas.datasets.get_path('naturalearth_lowres'))
>>> cities = gpd.read_file(geopandas.datasets.get_path('naturalearth_cities'))

# Create subsets of the world and cities GeoDataFrames
>>> world = world[['continent', 'name', 'iso_a3']]
>>> cities = cities[['name', 'iso_a3']]

# Merge the GeoDataFrames on their iso_a3 code
>>> countries = world.merge(cities, on='iso_a3')
\end{lstlisting}

A spatial join merges two GeoDataFrames based on their geometry data.
The function used for this is \li{sjoin}.
\li{sjoin} accepts two GeoDataFrames and then direction on how to merge.
It is imperative that two GeoDataFrames have the same CRS.
In the example below, we merge using an \li{inner} join with the option \li{intersects}.
The \li{inner} join means that we will only use keys in the intersection of both geometry columns, and we will retain only the left geometry column.
\li{intersects} tells the GeoDataFrames to merge on GeoSeries that intersect each other.
Other options include \li{contains} and \li{within}.

\begin{lstlisting}
# Combine countries and cities on their geographic location
>>> countries = gpd.sjoin(world, cities, how='inner', op='intersects')
\end{lstlisting}

\begin{problem}
Load in the file \li{nytimes.csv}\footnote{Source: \url{https://raw.githubusercontent.com/nytimes/covid-19-data/master/us-counties.csv}} as a DataFrame.
This file includes county-level data for the cumulative cases and deaths of Covid-19 in the US, starting with the first case in Snohomish County, Washington, on January 21, 2020.
Begin by converting the \li{date} column into a \li{DatetimeIndex}.

Next, use county FIPS codes to merge your GeoDataFrame from Problem \ref{problem:us_counties} with the DataFrame you just created.
A FIPS code is a 5-digit unique identifier for geographic locations.
Ignore rows in the Covid-19 DataFrame with unknown FIPS codes as well as all data from Hawaii and Alaska.

Note that the \li{fips} column of the Covid-19 DataFrame stores entries as floats, but the county GeoDataFrame stores FIPS codes as strings, with the first two digits in the \li{STATEFP} column and the last three in the \li{COUNTYFP} column.

Once you have completed the merge, plot the cases from March 21, 2020 on top of your state outline map from Problem \ref{problem:us_counties}, using the CRS of EPSG:5071.
Finally, print out the name of the county with the most cases on March 21, 2020 along with its case count.
\label{problem:merge_covid}
\end{problem}

\section*{Logarithmic Plotting Techniques} %============================

The color scheme of a graph can also help to communicate information clearly.
A good list of available colormaps can be found at \url{https://matplotlib.org/3.2.1/gallery/color/colormap_reference.html}.
Note also that you can reverse any colormap by adding \li{_r} to the end.
The following example demonstrates some plotting features, using country GDP as in Figure \ref{figure:gdp_map}.

\begin{lstlisting}
>>> fig, ax = plt.subplots(figsize=(15,7), ncols=1, nrows=1)
>>> world.plot(column='gdp_md_est', cmap='plasma_r',
...			ax=ax, legend=True, edgecolor='gray')

# Add title and remove axis tick marks
>>> ax.set_title('GDP on Linear Scale')
>>> ax.set_yticks([])
>>> ax.set_xticks([])
>>> plt.show()
\end{lstlisting}

\begin{figure}[H]
\begin{center}
\includegraphics[scale=.75]{figures/world_linear.pdf}
\end{center}
\caption{World map showing country GDP}
\label{figure:linear_world}
\end{figure}

Sometimes data can be much more informative when plotted on a logarithmic scale.
See how the world map changes when we add a \li{norm} argument in the code below.
Depending on the purpose of the graph, Figure \ref{figure:log_world} may be more informative than Figure \ref{figure:linear_world}.

\begin{lstlisting}
>>> from matplotlib.colors import LogNorm
>>> from matplotlib.cm import ScalarMappable
>>> fig, ax = plt.subplots(figsize=(15,6), ncols=1, nrows=1)

# Set the norm using data bounds
>>> data = world.gdp_md_est
>>> norm = LogNorm(vmin=min(data), vmax=max(data))

# Plot the graph using the norm
>>> world.plot(column='gdp_md_est', cmap='plasma_r', ax=ax,
...			edgecolor='gray', norm=norm)

# Create a custom colorbar
>>> cbar = fig.colorbar(ScalarMappable(norm=norm, cmap='plasma_r'),
...			ax=ax, orientation='horizontal', pad=0, label='GDP')

>>> ax.set_title('Country Area on a Log Scale')
>>> ax.set_yticks([])
>>> ax.set_xticks([])
>>> plt.show()
\end{lstlisting}

\begin{figure}[H]
\begin{center}
\includegraphics[scale=.65]{figures/world_log.pdf}
\end{center}
\caption{World map showing country GDP using a log scale}
\label{figure:log_world}
\end{figure}

\begin{problem}
As in Problem \ref{problem:merge_covid}, plot your state outline map from Problem \ref{problem:us_counties} on top of a map of the Covid-19 cases from March 21, 2020.
This time, however, use a log scale.
Use EPSG:5071 for the CRS.
Pick a good colormap (the counties with the most cases should generally be darkest) and be sure to display a colorbar.
\end{problem}

\begin{problem}
In this problem, you will create an animation of the spread of Covid-19 through US counties from January 21, 2020 to June 21, 2020.
Use a log scale and a good colormap, and be sure that you're using the same norm and colorbar for the whole animation.
Use EPSG:5071 for the projection.

As a reminder, below is a summary of what you will need in order to animate this map.
You may also find it helpful to refer to the animation section included with the Volume 4 lab manual.

%------------------------------------------- Is that ok to say?
\begin{enumerate}
	\item Set up your figure and norm. Be sure to use the highest case count for your \li{vmax} so that the scale remains uniform.
	\item Write your \li{update} function. This should plot the cases from a given day.
	\item Set up your colorbar. Do this outside the \li{update} function to avoid adding a new colorbar each day.
	\item Create the animation and embed it.
\end{enumerate}
\end{problem}


%!TEX root = thesis-kdyoung.tex

\chapter{Computational Experiments}
\label{chap:exp}

All experiments were run on a personal computer with
an Intel i5 5575R CPU (four cores each with 2.8GHz clock speed) and 16GB of main memory.
The scripts for the MIPs and the primary script used
to iterate through the Benders
decomposition algorithm was written in Python 3.4.
We employed Gurobi 7.0.2 as the solver
used for all MIP formulations.
As mentioned previously, \chuffed was the solver
we chose to solve all constraint programs with.
MiniZinc version 2.1.2 was used to compile the models
into the \chuffed FlatZinc format.
Unless stated otherwise, all tests were run
with a time limit of 30 minutes (1800 seconds).

We chose to only test \fsbf{2} and \scbf{2} for the
computational experiments of the previous work.
This is a result of our encoding of the \ssbf{2} model
returning infeasible solutions in our preliminary tests.
Due to time constraints we were unable to discern whether
the problem was our encoding of the model
or if the \ssbf{2} is ill-formed.

We now provide a brief outline of the computational
results reported in this chapter.
First a description of the data used for testing
is given in Section \ref{sec:exp:data}.
As the MIPs presented in \authciteb{Esmaeilbeigi2016}
have been untested, we investigate the best combination
of valid inequalities to include for these models
in Section \ref{sec:exp:mip}.
In Section \ref{sec:exp:cuts} we report on the best combination
of Benders cuts to use for our decomposition.
We provide a detailed investigation into the best configuration
of the CP sub-problem in Section \ref{sec:exp:cp}.
Once the superior sub-problem formulation is determined,
we close in Section \ref{sec:exp:benchmarkExp} of
this chapter by reporting on the results of our best
solution methodology on a set of benchmark instances.

\section{Data}
\label{sec:exp:data}
The primary dataset available for the \sua{} is
the SBF generated by \citeauthor{Scholl2013}
However, this dataset was created for the type-1 case in
particular and so is not immediately applicable to
the \sua{2}.
The SBF is divided into two halves, the SBF1 and SBF2,
where each half consists of a total of 1076 instances.
We first provide a brief explanation of how SBF1
was generated.

The basis of the SBF1 are a set of 269 instances
for the \sab{1} containing between 4 and 297 tasks.
When creating instances for the \sua{1}, the cycle time,
task times and precedence relations are all transferred from
the original instances.
We also require values of the forward and backward setup times \ap
so to create these the authors consider the average processing
time among all tasks, denoted $t_{av}$.
In practice the setup times are smaller than the execution time
of a task, so a parameter $\alpha$ is used when randomly generating
the setups.
An upper bound on the possible setup is defined as $\alpha\cdot t_{av}$
where $\alpha\in\{\:0.25,\:0.50,\:0.75,\:1.00\:\}$.
For example, when $\alpha=1.00$, the largest possible setup time
is 50\% of the average task time.
The 269 input instances are duplicated for each $\alpha$ value
and so 1076 instances of \sua{1} are generated for SBF1.

The SBF2 was generated in almost exactly the same way as SBF1.
However in this case,
care was taken when randomly generating the setup times to ensure
that at least one optimal solution to the original \sab{1} instance
is still feasible in the new \sua{1} instance.
This solution will necessarily be the optimal solution
of the \sua{2} as noted by \citeauthor{Scholl2013}
When storing the instance data, the authors also recorded the
optimal number of stations for the original \sab{1} instance.

The data we chose to run all our tests on was adapted from the SBF2
dataset.
As the optimal solutions were recorded,
we could take the optimal number of stations as input
and then solve for the cycle time.
This simple change allowed 1076 instances for testing our formulations.
\authciteb{Esmaeilbeigi2016} considered a subset of the SBF2 instances
which they divided into 3 classes.
We considered the same subset of instances for the type-2 problem
which are detailed in Table \ref{tab:exp:dataSBF2}.
In this table we give the number of tasks, stations, precedence
relations and order strength of the precedence graph.
In total 396 instances were considered in our tests.

The order strength of an instance is a measure of its complexity
where higher values correspond to more difficult instances.
However, we note that in the extreme, when $OS=1$, there is only one
precedence-feasible task sequence and thus the problem becomes trivial.

\begin{table}[tpb]
	\centering
	\caption{Dataset summary}
	\vspace{2mm}
	\begin{tabular}{llrrrrr}
		\toprule
		Class & Creator & \#Inst. & $n$ & $m$ & $|E|$ & Order Strength $(OS)$ \\
		\midrule\midrule
		1 &  & 108 & 7-21 & 2-8 & 6-27 & \\[1mm]
		 & {\tt mertens}  & 6 & 7 & 2-6 & 6 & medium (52.40)\\
		  & {\tt bowman8}  & 1 & 8 & 5 & 8 & high (75.00)\\
		  & {\tt jaeschke} & 5 & 9 & 3-8 & 11 & high (83.33)\\
		  & {\tt jackson}  & 6 & 11 & 3-8 & 13 & medium (58.18)\\
		  & {\tt mansoor}  & 3 & 11 & 2-4 & 11 & medium (60.00)\\
		  & {\tt mitchell} & 6 & 21 & 3-8 & 27 & high (70.95)\\\midrule
		2 &  & 112 & 25-30 & 3-14 & 32-40 & \\[1mm]
		  & {\tt roszieg} & 6 & 25 & 4-10 & 32 & high (71.67)\\
		  & {\tt heskia}   & 6 & 28 & 3-8 & 40 & low (22.49)\\
		  & {\tt buxey}    & 7 & 29 & 7-13 & 36 & medium (50.74)\\
		  & {\tt sawyer30} & 9 & 30 & 5-14 & 32 & low (44.83)\\\midrule
		3 &  & 176 & 32-58 & 3-31 & 38-82 & \\[1mm]
		  & {\tt lutz1}   & 6 & 32 & 6-11 & 38 & high (83.47)\\
		  & {\tt gunther}  & 7 & 35 & 7-14 & 43 & medium (59.50)\\
		  & {\tt kilbrid}  & 10 & 45 & 3-10 & 62 & low (44.60)\\
		  & {\tt hahn}     & 5 & 53 & 4-8 & 82 & high (83.82)\\
		  & {\tt warnecke} & 16 & 58 & 14-31 & 70 & medium (59.10)\\\midrule
		Overall & & 396 & 7-58 & 2-31 & 6-82 & \\
		\bottomrule
	\end{tabular}
	\label{tab:exp:dataSBF2}
\end{table}

% \section{Preliminary Tests}
% \label{sec:exp:prelim}

\section{MIP Formulations}
\label{sec:exp:mip}
We first report on how the MIP formulations presented
in Chapter \ref{chap:mip} performed.
In Table \ref{tab:exp:resultsFSBF} and \ref{tab:exp:resultsSCBF} we give the results
of \fsbf{2} and \scbf{2} respectively.
Each is tested on each combination of the valid inequalities from
Section \ref{sec:mip:validIneqs} against
all 3 classes of the subset
of instances from SBF2.
The following datapoints are summarized for each test:
the number of nodes explored by \gurobi's \bab tree,
the average gap for all instances,
the number of instances where no feasible solution was found,
the fraction of instances found to optimality out of the whole class,
the percentage of optimal instances and finally
the average runtime taken by \gurobi.

\begin{table}[tpb]
	\caption{FSBF-2 formulations tested on classes 1,2 and 3}
	\centering
	\vspace{2mm}
	\begin{tabular}{clrrrrrr}
		\toprule
		Class & Ineqs. & \#Nodes & \%Gap & \#No.s. & \#Opt. & \%Opt. & Rt.(s) \\\midrule\midrule
		1 & -- & 40,181 & 0.56 & 0 & 98/108 & 90.74 & 172.90 \\
		 & (\ref{eq:mip:valIneq1}) & 43,437 & 0.16 & 0 & \bf{101}/108 & 93.52 & 166.57 \\
		 & (\ref{eq:mip:valIneq3}) & 50,987 & 1.14 & 0 & 84/108 & 77.78 & 628.47 \\
		 & (\ref{eq:mip:valIneq1}),(\ref{eq:mip:valIneq3}) & 51,734 & 1.12 & 0 & 85/108 & 78.73 & 645.02 \\\midrule
		2 & -- & 171,847 & 14.81 & 0 & 0/112 & 0.00 & 1801.08 \\
		 & (\ref{eq:mip:valIneq1}) & 168,899 & 13.76 & 0 & 0/112 & 0.00 & 1800.66 \\
		 & (\ref{eq:mip:valIneq3}) & 205,752 & 18.04 & 0 & 0/112 & 0.00 & 1801.52 \\
		 & (\ref{eq:mip:valIneq1}),(\ref{eq:mip:valIneq3}) & 202,980 & 17.90 & 0 & 0/112 & 0.00 & 1801.27 \\\midrule
		3 & -- & 48,927 & 19.75 & 9 & 7/176 & 3.98 & 1763.45 \\
		 & (\ref{eq:mip:valIneq1}) & 46,060 & 19.56 & 9 & 7/176 & 3.98 & 1759.67 \\
		 & (\ref{eq:mip:valIneq3}) & 62,845 & 25.64 & 16 & 0/176 & 0.00 & 1800.23 \\
		 & (\ref{eq:mip:valIneq1}),(\ref{eq:mip:valIneq3}) & 63,002 & 24.72 & 15 & 0/176 & 0.00 & 1801.12 \\
		\bottomrule
	\end{tabular}
	\label{tab:exp:resultsFSBF}
\end{table}

\begin{table}[tpb]
	\caption{SCBF-2 formulations tested on classes 1,2 and 3}
	\centering
	\vspace{2mm}
	\begin{tabular}{clrrrrrr}
		\toprule
		Class & Ineqs. & \#Nodes & \%Gap & \#No.s. & \#Opt. & \%Opt. & Rt.(s) \\\midrule\midrule
		1 & -- & 221,842 & 1.62 & 0 & \bf{92}/108 & 86.11 & 374.88 \\
		 & (\ref{eq:mip:valIneq1}) & 244,811 & 1.97 & 0 & 84/108 & 77.78 & 403.17 \\
		 & (\ref{eq:mip:valIneq3}) & 238,515 & 1.75 & 0 & 86/108 & 76.93 & 377.05  \\
		 & (\ref{eq:mip:valIneq1}),(\ref{eq:mip:valIneq3}) & 215,866 & 1.75 & 0 & 86/108 & 76.93 & 380.82  \\\midrule
		2 & -- & 651,880 & 18.01 & 28 & 0/112 & 0.00 & 1800.33 \\
		 & (\ref{eq:mip:valIneq1}) & 701,617 & 19.75 & 31 & 0/112 & 0.00 & 1800.47 \\
		 & (\ref{eq:mip:valIneq3}) & 686,566 & 18.98 & 30 & 0/112 & 0.00 & 1801.03 \\
		 & (\ref{eq:mip:valIneq1}),(\ref{eq:mip:valIneq3}) & 612,442 & 18.57 & 31 & 0/112 & 0.00 & 1800.99 \\\midrule
		3 & -- & 301,055 & 10.09 & 109 & 0/176 & 0.00 & 1800.88 \\
		 & (\ref{eq:mip:valIneq1}) & 326,284 & 12.20 & 117 & 0/176 & 0.00 & 1801.00 \\
		 & (\ref{eq:mip:valIneq3}) & 324,878 & 12.15 & 112 & 0/176 & 0.00 & 1801.21 \\
		 & (\ref{eq:mip:valIneq1}),(\ref{eq:mip:valIneq3}) & 319,908 & 12.06 & 117 & 0/176 & 0.00 & 1801.01 \\
		\bottomrule
	\end{tabular}
	\label{tab:exp:resultsSCBF}
\end{table}

By inspecting the data we find that using the
valid inequality (\ref{eq:mip:valIneq1})
without (\ref{eq:mip:valIneq3}) is the best
formulation of \fsbf{2}.
As for \scbf{2}, we found that the best formulation
was when neither of the two valid inequalities were included in
the program.

For both the MIPs, optimality was proven for almost none of the instances of
class 2 and 3.
As a result, all the average runtime values are close
to the time limit of 30 minutes.
In the case of \scbf{2}, no feasible solution was found for 66.5\% of the instances
in class 3.
Even for the smallest class of data, where there is no more
than 21 tasks, neither MIP was able to prove
optimality for all instances.
We note that when calculating the gap value, we only average
the instances where a feasible solution was found.
With a 30 minute time limit, both MIPs could not achieve
an average gap value less than 10\% for class 2 and 3.

Here we tested the full mixed-integer programming approach
on the smallest subset of instances of the SBF2.
The poor results we have found make it clear that these programs
are unsatisfactory exact solution methods for tackling
the \sua{2}.

\section{Cuts}
\label{sec:exp:cuts}
The cutting strategy used in the iteration of the Benders
algorithm can have a large impact on its overall performance.
Even when either the master or sub-problem is exceptionally
difficult to solve, if the Benders cuts added to the master
allow us to find optimality in very few iterations then the problem
can still be tractable.

\subsection{Nogood Cuts}
\label{sec:exp:NGcuts}
In the preliminary testing phase, we considered feasibility sub-problems
using the nogood cuts presented in Section \ref{sec:bend:NGcuts} of the
previous chapter.
These results are briefly summarized in Table \ref{tab:exp:resultsNGcut}.
Even on the smallest class 1, the Benders
algorithm failed to prove optimality for 66 of the instances tested.
Due to this, we felt that further investigation into the use
of nogood cuts was not the most fruitful research direction.
Note that we did not make use of the possible parallel computing capabilities
detailed earlier in Section \ref{sec:bend:SPfeas},
where the feasibility sub-problems could be run concurrently.
This potential avenue was not explored as the percentage of computation
time spent solving the sub-problems was dwarfed by the time spent solving
the master problem; as we will see in the coming results.

For all remaining experiments optimality sub-problems were used
unless stated otherwise.

\begin{table}[tpb]
	\caption{Benders algorithm tested on class 1 with nogood cuts}
	\centering
	\vspace{2mm}
	\begin{tabular}{ccrrrr}
		\toprule
		Cuts & Class & \#No sol. & \#Opt. & \%Opt. & Runtime(s) \\\midrule\midrule
		$\mathcal{C}_{ng}$ & 1 & 66 & 42/108 & 38.89 & 1456.57 \\
		\bottomrule
	\end{tabular}
	\label{tab:exp:resultsNGcut}
\end{table}

\subsection{Cutting Procedure}
\label{sec:exp:cuttngProcedure}
In Tables \ref{tab:exp:resultsCutsClass1} and \ref{tab:exp:resultsCutsClass2}
we provide the primary comparison of the possible cutting strategies
that we considered.
The first table compares the 9 possible cut combinations on the smallest
class of data (class 1).
The variation in the results of Table \ref{tab:exp:resultsCutsClass1},
makes it difficult to discern a
clear best cutting procedure.
So we further tested 6 cut combinations
on class 2 in the Table \ref{tab:exp:resultsCutsClass2}.
The choice of solving technology for the sub-problems
will only effect the time taken to solve all scheduling problems
but not the cutting procedure.
Thus we arbitrarily chose to test all combinations of cuts using MIP
sub-problems.

The two tables comparing the cut combinations contain
the following data:
average number of nodes explored by \gurobi's
\bab tree (total combined between the \rmp{} and sub-problems),
average number iterations of the Benders algorithm,
average number of cuts generated,
average Benders gap,
total number of instances where no feasible solution was found,
percentage of the instances proved optimal and
the average runtime of all instances.

By inspecting the results of Table \ref{tab:exp:resultsCutsClass1}
it is difficult to reason about which cutting strategy is superior
as all instances were proved optimal for every combination.
These results immediately tell us that the Benders decomposition
we present here has outperformed the pure MIP formulations
previously presented by the literature.
We note that when the global bound in not enforced after
a globally feasible solution is found, the Benders gap
remains substantially large and so it is difficult
to measure the convergence of the Benders algorithm.
Thus for the remaining experiments we run, the global bound
will always be used unless stated otherwise.
The $3^{\text{rd}}$ version of the inference cut ($\mathcal{C}_{iii}$) appears to
be a key ingredient in the Benders algorithm terminating efficiently,
but to confirm this suspicion we test our solution
methodology on class 2.

\begin{table}[tpb]
	\caption{Benders cutting strategies compared on class 1}
	\centering
	\vspace{2mm}
	\begin{tabular}{cccccrrrrrrr}
		\toprule
		\multicolumn{5}{c}{Cuts and Bound}  &  &  &  &  &  &  &   \\\cmidrule(rl){1-5}
		$\mathcal{C}_{i}$ & $\mathcal{C}_{ii}$ & $\mathcal{C}_{iii}$ & $\mathcal{C}_{l}$ & $\mathcal{C}_{gb}$& \#Nodes & \#Its. & $|\mathcal{C}|$ & \%Gap & \#No.s. & \%Opt. & Rt.(s) \\\midrule\midrule
		\checkmark &  &  &  &  & 4,718 & 18 & 35 & 0.00 & 0 & 100.00 & 5.8 \\
		 & \checkmark &  &  &  & 2,845 & 15 & 28 & 0.00 & 0 & 100.00 & 4.1 \\
		 &  & \checkmark &  &  & 170 & 4 & 8 & 0.00 & 0 & 100.00 & 1.1 \\\midrule
		\checkmark &  &  &  & \checkmark & 3,790 & 16 & 32 & 0.00 & 0 & 100.00 & 2.9 \\
		 & \checkmark &  &  & \checkmark & 2,064 & 12 & 26 & 0.00 & 0 & 100.00 & 1.9 \\
		 &  & \checkmark &  & \checkmark & 128 & 3 & 9 & 0.00 & 0 & 100.00 & 0.4 \\\midrule
		\checkmark &  &  & \checkmark & \checkmark & 766 & 17 & 34 & 0.00 & 0 & 100.00 & 2.6 \\
		 & \checkmark &  & \checkmark & \checkmark & 896 & 17 & 34 & 0.00 & 0 & 100.00 & 2.8 \\
		 &  & \checkmark & \checkmark & \checkmark & 56 & 4 & 12 & 0.00 & 0 & 100.00 & 0.5 \\
		\bottomrule
	\end{tabular}
	\label{tab:exp:resultsCutsClass1}
\end{table}

In Table \ref{tab:exp:resultsCutsClass2} we can see that
class 2 proved more challenging for the Benders decomposition
to solve the problem as in many cases optimality was
proven for only $\sim30\%$ of the instances.
These results make it clear that $\mathcal{C}_{iii}$
is the strongest infer cut as it is able to
solve almost twice as many instances to optimality
that the other 2 infer cuts.
Further to this, it needed half as many iterations
and similarly half as many cuts in total.
For the instances where the optimal solution was not found,
$\mathcal{C}_{iii}$ provided the lowest Benders gap values
as well.

We must also examine whether including the logic cut, $\mathcal{C}_{l}$,
provided a benefit.
When considering the logic cut, a global bound on the cycle time is instituted
as an upper bound on the load of each station's sub-problem,
possibly leading to feasibility sub-problems, rather than optimality.
The logic cut was formulated to provide the \rmp{} with detailed information
about the assignment variables whilst also reducing the time
needed to solve a sub-problem; however it also has a notable drawback.
When a logic cut is generated by a sub-problem, that sub-problem
has not been solved to optimality.
This results in us not being able to generate an optimality cut for
this sub-problem, such as $\mathcal{C}_{iii}$.
The results of Table \ref{tab:exp:resultsCutsClass2} indicate
that on average, not being able to perform an infer cut has not outweighed the beneficial
information provided by the logic cut.

\begin{table}[tpb]
	\caption{Benders cutting strategies compared on class 2}
	\centering
	\vspace{2mm}
	\begin{tabular}{cccccrrrrrrr}
		\toprule
		\multicolumn{5}{c}{Cuts and Bound}  &  &  &  &  &  &  &   \\\cmidrule(rl){1-5}
		$\mathcal{C}_{i}$ & $\mathcal{C}_{ii}$ & $\mathcal{C}_{iii}$ & $\mathcal{C}_{l}$ & $\mathcal{C}_{gb}$ & \#Nodes & \#Its. & $|\mathcal{C}|$ & \%Gap & \#No.s. & \%Opt. & Rt.(s) \\\midrule\midrule
		\checkmark &  &  &  & \checkmark & 1,522k & 99 & 396 & 11.63 & 1 & 31.25 & 1339.7 \\
		 & \checkmark &  &  & \checkmark & 1,430k & 99 & 399 & 10.79 & 1 & 32.14 & 1348.5 \\
		 &  & \checkmark &  & \checkmark & 1,024k & 47 & 196 & 8.42 & 1 & \bf{60.71} & 959.9 \\\midrule
		\checkmark &  &  & \checkmark & \checkmark & 918k & 112 & 439 & 11.38 & 1 & 33.04 & 1318.4 \\
		 & \checkmark &  & \checkmark & \checkmark & 964k & 114 & 443 & 11.33 & 1 & 31.25 & 1342.3 \\
		 &  & \checkmark & \checkmark & \checkmark & 898k & 68 & 225 & 9.03 & 1 & 58.04 & 956.8 \\
		\bottomrule
	\end{tabular}
	\label{tab:exp:resultsCutsClass2}
\end{table}

Hence, for all remaining experiments we used the infer cut $\mathcal{C}_{iii}$
in combination with the global bound $\mathcal{C}_{gb}$ as the cutting
procedure, unless stated otherwise.

We mention here that combining different
types of infer cuts together in a single cutting procedure
was not explored.
This is because every inference made by $\mathcal{C}_{i}$ is
contained within $\mathcal{C}_{ii}$ and every inference made by $\mathcal{C}_{ii}$
is similarly contained within $\mathcal{C}_{iii}$.
Thus combining these infer cuts will not lead to a
substantial reduction in runtime.

\section{CP Sub-Problem}
\label{sec:exp:cp}
The CP solver \chuffed provides the
user with numerous input parameters to
tailor the propagation and search procedure
to the problem at hand.
In this section, we report on the comparisons
between the model formulation
and search strategies in order to find the best suited
configuration for the \sua{2} instances.

As we are only concerned with optimizing the sub-problems'
solving time, the runtime values we present here will
only average the time used by the CP model,
\ie excluding the runtime of the master problem.
Similarly, this also applies to the average number of nodes
explored by the CP sub-problems..

% The upper bound cut reduces the space complexity of the sub-problems
% very helpful for cp solver whose difficultly is very closely
% related to the inference that can be done on the finite domains.

\subsection{Model Formulation}
\label{sec:exp:cpForm}
Two main types of CP models were considered, so we
first check which of these is superior before moving
on to test other input parameters.
$CP_1$ represents the model detailed in Section \ref{sec:bend:cpModel}
from the previous chapter.
$CP_2$ represents the same model but with the additional decision
variable $\sigma_{ij}$ and the \cumu global constraint.

In Table \ref{tab:exp:resultsCPmodel}, we compare these two models
against each other on the class 1 and
class 2 instances where $\alpha$ is fixed to $1.00$.
The search strategy used for these tests was simply the default
search of \chuffed.
These tests indicate that there is a reduction
in sub-problem runtime when the \cumu constraint
is used in the CP formulation.
For all remaining experiments where CP sub-problems
were used, $CP_2$ was the formulation utilized
unless stated otherwise.

\begin{table}[tpb]
	\caption{CP formulations compared on classes 1 and 2 when $\alpha=1.00$}
	\centering
	\vspace{2mm}
	\begin{tabular}{ccrrrrrr}
		\toprule
		Class & Model & \#SP Nodes & \%Gap & \%Opt. & SP Runtime(s) \\\midrule\midrule
		1 & $CP_1$ & 32,755 & 0.00 & 100.00 & 3.83  \\
		 & $CP_2$ & 29,225 & 0.00 & 100.00 & \bf{2.84} \\\midrule
		2 & $CP_1$ & 2,768,871 & 15.45 & 25.00 & 386.71 \\
		 & $CP_2$ & 2,503,183 & 14.03 & 25.00 & \bf{329.60} \\
		\bottomrule
	\end{tabular}
	\label{tab:exp:resultsCPmodel}
\end{table}

\subsection{Search Strategy}
\label{sec:exp:cpSearch}
Earlier in Section \ref{sec:bend:cpForm}
of the previous Chapter, we detailed a range of possible search procedures
which could be employed when solving our constraint program.
In Table \ref{tab:exp:resultsCPsearch} we present the results
of testing the CP sub-problems on a selection of these procedures.
The names of the searches refer to the following procedures:
{\tt def} is the default search, {\tt start} is the basic search on start time variables,
{\tt start\_ff\_Then\_sigma} is a sequential search of start then $\sigma$ variables.
Four priority searches were considered which only differ by the variable selection
strategy used to choose the next sequential procedure.
% ; these are
% {\tt priority\_io} using {\tt input\_order} variable selection,
% {\tt priority\_sm} using {\tt smallest},
% {\tt priority\_sml} using {\tt smallest\_largest} and
% {\tt priority\_ff} using {\tt first\_fail}.

From the results of Table \ref{tab:exp:resultsCPsearch},
we can immediately conclude that the default search
is inferior to the search strategies we have tailored for
the \sua{2}.
Although, among the search procedures we devised, there
is very little variations in the results.
Each procedure was able to find the optimal solution
for the same number of instances,
while the runtime was very similar.
The basic search {\tt start} needed to explore the fewest
number of nodes in the search tree, however the priority search
{\tt priority\_ff} provided the best average runtime.
For all remaining tests where CP sub-problems
were utilized, the priority based search strategy using
the {\tt first\_fail} variable selection was used.

\begin{table}[tpb]
	\caption{CP search strategies compared on class 2 when $\alpha=1.00$}
	\centering
	\vspace{2mm}
	\begin{tabular}{clrrrrrr}
		\toprule
		Class & Search & \#SP Nodes & \%Gap & \%Opt. & SP Runtime(s) \\\midrule\midrule
		2 & {\tt default}			& 2,503k & 14.03 & 25.00 & 329.60 \\
		 & {\tt start}				& 1,884k & 8.97 & 42.86 & 246.57 \\
		 & {\tt start\_ff\_Then\_sigma}	& 2,391k & 9.69 & 42.86 & 254.23 \\
		 & {\tt priority\_io}		& 2,448k & 9.12 & 42.86 & 253.70 \\
		 & {\tt priority\_sm}		& 2,408k & 9.06 & 42.86 & 252.91 \\
		 & {\tt priority\_sml}		& 2,470k & 9.04 & 42.86 & 252.63 \\
		 & {\tt priority\_ff}		& 2,426k & 8.90 & 42.86 & \bf{245.74} \\
		\bottomrule
	\end{tabular}
	\label{tab:exp:resultsCPsearch}
\end{table}

\section{Benchmark Experiments}
\label{sec:exp:benchmarkExp}
We close this chapter by comparing the best
configuration of our logic-based Benders decomposition
against the results of the MIP formulations for all benchmark
instances from the subset of SBF2 we have used.
Before presenting this final result, we must first
decide on the best sub-problem formulation for our
Benders decomposition.

Earlier in Section \ref{sec:lit:cpChuffed}, we gave a
brief description of the CP solver, \chuffed, we chose to utilize when
tackling the constraint programs we formulated.
As mentioned in that section, \chuffed does not currently have
the ability to be run in parallel.
This handicap has hampered its ability to effectively compete
with \gurobi, which can easily run in parallel on all four cores
of the computer we used for testing.

In Table \ref{tab:exp:resultsSPform} we present a comparison
of our Benders decomposition algorithm tested on the class 2
instances when using MIP and CP sub-problems.
There are many more nodes explored when using CP sub-problems
due to the different search procedure of a CP solver
when compared to a MIP solver.
The results of the two formulations are similar, but we can
conclude that our best configuration of the CP model
has provided a reduction in runtime when compared to the MIP
formulation.

\begin{table}[tpb]
	\caption{MIP and CP sub-problem formulations compared on class 2}
	\centering
	\vspace{2mm}
	\begin{tabular}{ccrrrrrr}
		\toprule
		Class & Sub-Problem & \#SP Nodes & \%Gap & \%Opt. & SP Runtime(s) \\\midrule\midrule
		2 & \spmip{} & 630k & 8.42 & 60.71 & 173.05 \\
		 & \spcp{} & 2,489k & 8.21 & 61.61 & \bf{165.26} \\
		\bottomrule
	\end{tabular}
	\label{tab:exp:resultsSPform}
\end{table}

Now using the best formulation of our Benders decomposition
we present the results of our final benchmark experiments
on all 3 classes from the subset of the SBF2 instances.
These results can be found in Table \ref{tab:exp:resultsBestBenders}
where we provide the average number of iterations,
cuts, average gap, number optimal and runtime breakdown.
The instances from classes 2 and 3 proved to be more
challenging than the smaller instances from class 1.
Most notably, the total runtime taken by
the Benders algorithm is almost entirely due to the complexity of the \rmp{}.

More cuts are to be expected when the difficultly of the problem
is increased, but we also mention that for the instances of class 3
there is also a larger number of stations.
This will lead to more cuts being made for every iteration of the algorithm.

\begin{table}[tpb]
	\caption{Benchmark results of our final Benders decomposition}
	\centering
	\vspace{2mm}
	\begin{tabular}{crrrrrr}
		\toprule
		Class & \#Its. & \#Cuts. & \%Gap & \#Opt. & RMP.Runtime(s) & SP.Runtime(s) \\\midrule\midrule
		1 & 3 & 9 & 0.00 & 108/108 & 0.36 (87.95\%) & 0.04 (10.98\%) \\
		2 & 47 & 196 & 8.21 & 69/112 & 781.29 (82.33\%) & 165.26 (17.42\%) \\
		3 & 92 & 468 & 15.76 & 27/176 & 1462.01 (88.00\%) & 173.63 (10.45\%) \\
		\bottomrule
	\end{tabular}
	\label{tab:exp:resultsBestBenders}
\end{table}

Table \ref{tab:exp:resultsBenchmark} compares how our final results perform
when compared to the pure MIP formulations of the \sua{2}.
For each class of instances our solution methodology beats the best formulation
of the MIPs.
In the case of class 2 where the Benders decomposition could not prove optimality for
all instances, the average Benders gap at termination is superior to
the best average gap found by \gurobi when solving either \fsbf{2} and \scbf{2}.
For class 3, our algorithm could only find the optimal solution for 15.3\% of the 176 instances.
Also the \scbf{2} model returned superior average gap values for class 3, even though
it was unable to find any optimal solutions.
Although the gap value of \scbf{2} for the class 3 instances
may be misleading, as it only calculated from
the 59 of 176 instances where a feasible solution was found.

\begin{table}[tpb]
	\caption{Benders decomposition compared to pure MIP formulations}
	\centering
	\vspace{2mm}
	\begin{tabular}{crrrrrrrr}
		\toprule
		 & \multicolumn{4}{c}{Benders decomposition}  & \multicolumn{2}{c}{\fsbf{2}} & \multicolumn{2}{c}{\scbf{2}}  \\
		 \cmidrule(lr){2-5} \cmidrule(lr){6-7} \cmidrule(lr){8-9}
		Class & \%Gap & \#Opt. & \%Opt. & Rt.(s) & \%Gap & \%Opt. & \%Gap & \%Opt. \\\midrule\midrule
		1 & 0.00 & 108/108 & 100.00 & 0.41 & 0.16 & 93.52 & 1.62 & 77.78 \\
		2 & 8.21 & 69/112 & 61.61 & 948.93 & 13.76 & 0.00 & 18.01 & 0.00 \\
		3 & 15.76 & 27/176 & 15.34 & 1661.93 & 19.56 & 3.98 & 10.09 & 0.00 \\
		\bottomrule
	\end{tabular}
	\label{tab:exp:resultsBenchmark}
\end{table}

Our results indicate that the \sua{2} is a difficult problem to solve
when the instance size becomes large.
However, we can see from these results that decomposing the problem using a
logic-based Benders decomposition framework provides
the most optimal solutions when compared to the other available exact
solution procedures.

% \section{Summary}
% \label{sec:exp:summary}

% blah


\chapter{Shape-from-Shading}
\label{ch:sfs}

ASP provides a tool, named \texttt{sfs}, that can improve the level of
detail of DEMs created by ASP or any other source using
\textit{shape-from-shading} (SfS). The tool takes as input one or more
camera images, a DEM at roughly the same resolution as the images, and
returns a refined DEM.

\texttt{sfs} works only with ISIS cub images. It has been tested
thoroughly with Lunar LRO NAC datasets, and some experiments were done
with Mars HiRISE images and with pictures from Charon, Pluto's moon. As
seen later in the text, it returns reasonable results on the Moon as far
as $85^\circ$ South.

Currently, \texttt{sfs} is computationally expensive, and is practical
only for DEMs whose width and height are several thousand pixels. It can be
sensitive to errors in the position and orientation of the cameras, the
accuracy of the initial DEM, and to the value of the two weights it uses.
Yet, with some effort, it can work quite well.

A tool named \texttt{parallel\_sfs} is provided (section \ref{psfs})
that parallelizes \texttt{sfs} using multiple processes (optionally on
multiple machines) by splitting the input DEM into tiles with padding,
running \texttt{sfs} on each tile, and then blending the results.

The \texttt{sfs} program can model position-dependent albedo, different
exposure values for each camera, shadows in the input images, and regions
in the DEM occluded from the Sun. It can refine the positions and orientations
of the cameras.

The tool works by minimizing the cost function
\begin{equation}\label{cost}
%\begin{multline}\label{cost}
\int\!\! \int \! \sum_k \left[ I_k(\phi)(x, y) - T_k A(x, y)
 R_k(\phi)(x, y) \right]^2\,
% R_k(\phi)(x, y) \right]^2\,  \\
+ \mu \left\|\nabla^2 \phi(x, y) \right\|^2
+ \lambda  \left[ \phi(x, y) - \phi_0(x, y) \right]^2
\, dx\, dy.
\end{equation}
%\end{multline}

Here, $I_k(\phi)(x, y)$ is the $k$-th camera image interpolated at
pixels obtained by projecting into the camera 3D points from the terrain
$\phi(x, y)$, $T_k$ is the $k$-th image exposure, $A(x, y)$ is the
per-pixel albedo, $R_k(\phi)(x, y)$ is the reflectance computed from the
terrain for $k$-th image, $\left\|\nabla^2 \phi(x, y) \right\|^2 $ is the sum
of squares of all second-order partial derivatives of $\phi$, $\mu > 0$
is a smoothing term, and $\lambda > 0$ determines how close we should
stay to the input terrain $\phi_0$ (smaller $\mu$ will show more detail
but may introduce some artifacts, and smaller $\lambda$ may allow for
more flexibility in optimization but the terrain may move too far from the input).

We use either the regular Lambertian reflectance model,
or the Lunar-Lambertian model \cite{mcewen1991photometric}, more specifically as given in
\cite{lohse2006derivation} (equations (3) and (4)).
Also supported is the Hapke model,
\cite{johnson2006spectrophotometric}, \cite{fernando2013surface},
\cite{hapke2008bidirectional}, \cite{hapke1993opposition}.
Custom values for the coefficients of these models can be passed to the program.

\section{How to get good test imagery}

We obtain the images from
\url{http://wms.lroc.asu.edu/lroc/search} (we search for EDR images of
type NACL and NACR).

A faster (but not as complete) interface is provided by
\url{http://ode.rsl.wustl.edu/moon/indexproductsearch.aspx}. The related
site
\url{http://ode.rsl.wustl.edu/moon/indextools.aspx?displaypage=lolardr}
can provide LOLA datasets which can be used as (sparse) ground truth.

We advise the following strategy for picking images. First choose a small
longitude-latitude window in which to perform a search for imagery. Pick
two images that are very close in time and with a big amount of overlap
(ideally they would have consecutive orbit numbers).
Those can be passed to ASP's \texttt{stereo} tool to create an initial DEM. Then, search for
other images close to the center of the maximum overlap of the first
two images. Pick one or more of those, ideally with different illumination
conditions than the first two. Those (together with one of the first two images) can be used for SfS.

To locate the area of spatial overlap, the images can be map-projected (either
with \texttt{cam2map} with a coarse resolution) or with
\texttt{mapproject}, using for example the LOLA DEM as the terrain to project onto,
or the DEM obtained from running \texttt{stereo} on those images. Then the images can be overlayed
in \texttt{stereo\_gui}. A good sanity check is to examine the shadows in
various images. If they point in different directions in the images and perhaps
also have different lengths, that means that illumination conditions are
different enough, which will help constrain the \texttt{sfs} problem better.

\section{Running sfs at 1 meter/pixel using a single image}

In both this and the next sections we will work with LRO NAC images taken
close to the Lunar South Pole, at a latitude of $85^\circ$ South (the tool was
tested on equatorial regions as well). We will use four images,
M139939938LE, M139946735RE, M173004270LE, and M122270273LE.

We first retrieve the data sets.
\begin{verbatim}
  wget http://lroc.sese.asu.edu/data/LRO-L-LROC-2-EDR-V1.0/\
LROLRC_0005/DATA/SCI/2010267/NAC/M139939938LE.IMG
  wget http://lroc.sese.asu.edu/data/LRO-L-LROC-2-EDR-V1.0/\
LROLRC_0005/DATA/SCI/2010267/NAC/M139946735RE.IMG
  wget http://lroc.sese.asu.edu/data/LRO-L-LROC-2-EDR-V1.0/\
LROLRC_0009/DATA/SCI/2011284/NAC/M173004270LE.IMG
  wget http://lroc.sese.asu.edu/data/LRO-L-LROC-2-EDR-V1.0/\
LROLRC_0002/DATA/MAP/2010062/NAC/M122270273LE.IMG
\end{verbatim}

Then we convert them to ISIS cubes, initialize the SPICE kernels, and
perform radiometric calibration and echo correction. Here are the steps,
illustrated on the first image:
\begin{verbatim}
  lronac2isis from = M139939938LE.IMG     to = M139939938LE.cub
  spiceinit from   = M139939938LE.cub
  lronaccal from   = M139939938LE.cub     to = M139939938LE.cal.cub
  lronacecho from  = M139939938LE.cal.cub to = M139939938LE.cal.echo.cub
\end{verbatim}
We rename, for simplicity, the obtained four processed datasets to
A.cub, B.cub, C.cub, and D.cub.

The first step is to run stereo to create an initial guess DEM. We
picked for this the first two of these images. These form a stereo pair,
that is, they have a reasonable baseline and sufficiently close times of
acquisition (hence very similar illuminations). These conditions are
necessary to obtain a good stereo result.
\begin{verbatim}
parallel_stereo --job-size-w 1024 --job-size-h 1024 A.cub B.cub    \
                --left-image-crop-win 0 7998 2728 2696             \
                --right-image-crop-win 0 9377 2733 2505            \
                --threads 16 --corr-seed-mode 1  --subpixel-mode 3 \
                run_full1/run
\end{verbatim}
Next we create a DEM at 1 meter/pixel, which is about the resolution
of the input images. We use the stereographic projection since this
dataset is very close to the South Pole. Then we crop it to the region
we'd like to do SfS on.
\begin{verbatim}
  point2dem -r moon --stereographic --proj-lon 0 \
    --proj-lat -90 run_full1/run-PC.tif
  gdal_translate -projwin -15471.9 150986 -14986.7 150549  \
    run_full1/run-DEM.tif run_full1/run-crop-DEM.tif
\end{verbatim}
This creates a DEM of size $456 \times 410$ pixels.

Then we run \texttt{sfs}:
\begin{verbatim}
  sfs -i run_full1/run-crop-DEM.tif A.cub -o sfs_ref1/run           \
     --reflectance-type 1                                           \
    --smoothness-weight 0.08 --initial-dem-constraint-weight 0.0001 \
    --max-iterations 10 --use-approx-camera-models                  \
    --use-rpc-approximation --crop-input-images
\end{verbatim}

The smoothness weight is a parameter that needs tuning. If it is too
small, SfS will return noisy results, if it is too large, too much
detail will be blurred. Here we used the Lunar Lambertian model. The meaning of the other
\texttt{sfs} options can be looked up in section \ref{sfs}.

We show the results of running this program in figure
\ref{fig:sfs1}. The left-most figure is the hill-shaded original DEM,
which was obtained by running:
\begin{verbatim}
  hillshade --azimuth 300 --elevation 20 run_full1/run-crop-DEM.tif \
    -o run_full1/run-crop-hill.tif
\end{verbatim}
The second image is the hill-shaded DEM obtained after running
\texttt{sfs} for 10 iterations.

The third image is, for comparison, the map-projection of A.cub onto the
original DEM, obtained via the command:
\begin{verbatim}
  mapproject --tr 1 run_full1/run-crop-DEM.tif A.cub A_map.tif --tile-size 128
\end{verbatim}
The forth image is the colored absolute difference between the original
DEM and the SfS output, obtained by running:
\begin{verbatim}
  geodiff --absolute sfs_ref1/run-DEM-final.tif run_full1/run-crop-DEM.tif
  colormap --min 0 --max 2 --colormap-style binary-red-blue \
    run-DEM-final__run-crop-DEM-diff.tif
\end{verbatim}
\begin{figure}[h!]
\begin{center}
\includegraphics[width=7in]{images/sfs1.jpg}
\caption[sfs]{An illustration of \texttt{sfs}. The images are, from
  left to right, the original hill-shaded DEM, the hill-shaded DEM obtained
from \texttt{sfs}, the image A.cub map-projected onto the original DEM,
and the absolute difference of the original and final DEM, where the brightest
shade of red corresponds to a 2 meter height difference.}
\label{fig:sfs1}
\end{center}
\end{figure}

It can be seen that the optimized DEM provides a wealth of detail and
looks quite similar to the input image. It also did not diverge
significantly from the input DEM. We will see in the next section that
SfS is in fact able to make the refined DEM more accurate than the
initial guess (as compared to some known ground truth), though that is
not guaranteed, and most likely did not happen here where just one image
was used.

\section{SfS with multiple images in the presence of shadows}

In this section we will run \texttt{sfs} with multiple images. We would
like to be able to see if SfS improves the accuracy of the DEM rather
than just adding detail to it. We evaluate this using the following
(admittedly imperfect) approach. We resample the original images by a
factor of 10, run stereo with them, followed by SfS using the stereo
result as an initial guess and with the resampled images. As ground
truth, we create a DEM from the original images at 1 meter/pixel, which
we bring closer to the initial guess for SfS using
\texttt{pc\_align}. We would like to know if running SfS brings us even
closer to this ``ground truth'' DEM.

The most significant challenge in running SfS with multiple images is
that shape-from-shading is highly sensitive to errors in camera position
and orientation. The \texttt{sfs} tool can improve these by floating
them during optimization and by using a coarse-to-fine scheme, where
the problem is first solved using subsampled images and terrain then
it is successively refined.

If possible, it may still be desirable to bundle-adjust the cameras first
(section \ref{bundleadjust}). It is important to note that bundle adjustment may
fail if the images have sufficiently different illumination, as it will
not be able to find matches among images. A solution to this is discussed
in section \ref{sfs-lola}.

To make bundle adjustment and stereo faster, we first crop the images,
such as shown below (the crop parameters can be determined via
\texttt{stereo\_gui}).
\begin{verbatim}
  crop from = A.cub to = A_crop.cub sample = 1 line = 6644 nsamples = 2192 nlines = 4982
  crop from = B.cub to = B_crop.cub sample = 1 line = 7013 nsamples = 2531 nlines = 7337
  crop from = C.cub to = C_crop.cub sample = 1 line = 1 nsamples = 2531 nlines = 8305
  crop from = D.cub to = D_crop.cub sample = 1 line = 1 nsamples = 2531 nlines = 2740
\end{verbatim}
Then we bundle-adjust and run stereo
\begin{verbatim}
  bundle_adjust A_crop.cub B_crop.cub C_crop.cub D_crop.cub    \
    --min-matches 1 -o run_ba/run
  stereo A_crop.cub B_crop.cub run_full2/run --subpixel-mode 3 \
    --bundle-adjust-prefix run_ba/run
\end{verbatim}

This will result in a point cloud, \verb#run_full2/run-PC.tif#, which will
lead us to the ``ground truth'' DEM. As mentioned
before, we'll in fact run SfS with images subsampled by a factor of
10. Subsampling is done by running the ISIS \texttt{reduce} command
\begin{verbatim}
  for f in A B C D; do
    reduce from = ${f}_crop.cub to = ${f}_crop_sub10.cub sscale = 10 lscale = 10
  done
\end{verbatim}

We run bundle adjustment and stereo with the subsampled images using
commands analogous to the above:
\begin{verbatim}
  bundle_adjust A_crop_sub10.cub B_crop_sub10.cub C_crop_sub10.cub D_crop_sub10.cub \
    --min-matches 1 -o run_ba_sub10/run --ip-per-tile 100000
 stereo A_crop_sub10.cub B_crop_sub10.cub run_sub10/run --subpixel-mode 3           \
   --bundle-adjust-prefix run_ba_sub10/run
\end{verbatim}
We'll obtain a point cloud named \verb#run_sub10/run-PC.tif#.

We'll bring the ``ground truth'' point cloud closer to the initial guess
for SfS using \texttt{pc\_align}:
\begin{verbatim}
  pc_align --max-displacement 200 run_full2/run-PC.tif run_sub10/run-PC.tif \
    -o run_full2/run --save-inv-transformed-reference-points
\end{verbatim}
This step is extremely important. Since we ran two bundle adjustment
steps, and both were without ground control points, the resulting clouds
may differ by a large translation, which we correct here. Hence we would like to
make the ``ground truth'' terrain aligned with the datasets on which we
will perform SfS.

Next we create the ``ground truth'' DEM from the aligned high-resolution
point cloud, and crop it to a desired region:
\begin{verbatim}
  point2dem -r moon --tr 10 --stereographic --proj-lon 0 --proj-lat -90 \
    run_full2/run-trans_reference.tif
  gdal_translate -projwin -15540.7 151403 -14554.5 150473               \
    run_full2/run-trans_reference-DEM.tif run_full2/run-crop-DEM.tif
\end{verbatim}
We repeat the same steps for the initial guess for SfS:
\begin{verbatim}
  point2dem -r moon --tr 10 --stereographic --proj-lon 0 --proj-lat -90 \
    run_sub10/run-PC.tif
  gdal_translate -projwin -15540.7 151403 -14554.5 150473               \
    run_sub10/run-DEM.tif run_sub10/run-crop-DEM.tif
\end{verbatim}
After this, we run \texttt{sfs} itself. Since our dataset has many shadows, we found
that specifying the shadow thresholds for the tool improves the
results. The thresholds can be determined using \texttt{stereo\_gui}.
This can be done by turning on shadow-threshold mode from the GUI menu,
and then clicking on a few points in the shadows.  Then the thresholded images
can be visualized/updated from the menu as well, and this process can be iterated.
\begin{verbatim}
  sfs -i run_sub10/run-crop-DEM.tif A_crop_sub10.cub C_crop_sub10.cub \
    D_crop_sub10.cub -o sfs_sub10_ref1/run --threads 4                \
    --smoothness-weight 0.12 --initial-dem-constraint-weight 0.0001   \
    --reflectance-type 1 --float-exposure                             \
    --float-cameras --use-approx-camera-models                        \
    --max-iterations 10 --use-approx-camera-models                    \
    --use-rpc-approximation --crop-input-images                       \
    --bundle-adjust-prefix run_ba_sub10/run                           \
    --shadow-thresholds "0.00162484 0.0012166 0.000781663"
\end{verbatim}
We compare the initial guess to \texttt{sfs} to the ``ground truth'' DEM
obtained earlier and the same for the final refined DEM using
\texttt{geodiff} as in the previous section. Before SfS:

\begin{verbatim}
  geodiff --absolute run_full2/run-crop-DEM.tif run_sub10/run-crop-DEM.tif
  gdalinfo -stats run-crop-DEM__run-crop-DEM-diff.tif | grep Mean=
\end{verbatim}
and after SfS:
\begin{verbatim}
  geodiff --absolute run_full2/run-crop-DEM.tif sfs_sub10_ref1/run-DEM-final.tif
  gdalinfo -stats run-crop-DEM__run-DEM-final-diff.tif | grep Mean=
\end{verbatim}

The mean error goes from 2.64~m to 1.29~m, while the standard deviation
decreases from 2.50~m to 1.29~m. Visually the refined DEM looks more detailed
as well as seen in figure \ref{fig:sfs2}. The same experiment can be
repeated with the Lambertian reflectance model (reflectance-type 0), and
then it is seen that it performs a little worse.

We also show in this figure the first of the images used for SfS,
\verb#A_crop_sub10.cub#, map-projected upon the optimized DEM. Note that
we use the previously computed bundle-adjusted cameras when
map-projecting, otherwise the image will show as shifted from its true
location:
\begin{verbatim}
  mapproject sfs_sub10_ref1/run-DEM-final.tif A_crop_sub10.cub A_crop_sub10_map.tif \
    --bundle-adjust-prefix run_ba_sub10/run
\end{verbatim}
\begin{figure}[h!]
\begin{center}
\includegraphics[width=7in]{images/sfs2.jpg}
\caption[sfs]{An illustration of \texttt{sfs}. The images are, from
  left to right, the hill-shaded initial guess DEM for SfS, the hill-shaded DEM obtained
from \texttt{sfs}, the ``ground truth'' DEM, and the first of the
images used in SfS map-projected onto the optimized DEM.}
\label{fig:sfs2}
\end{center}
\end{figure}

\section{Dealing with large camera errors and LOLA comparison}
\label{sfs-lola}

SfS is very sensitive to errors in camera positions and
orientations. These can be optimized as part of the problem, but if they
are too far off, the solution will not be correct. In the previous
section we used bundle adjustment to correct these errors, and then we
passed the adjusted cameras to \texttt{sfs}. However, bundle adjustment may
often fail, simply because the illumination conditions can be very
different among the images, and interest point matching may not succeed.

The option \texttt{-\/--coarse-levels \it{int}} can be passed to
\texttt{sfs}, to solve for the terrain using a multi-resolution
approach, first starting at a coarse level, where camera errors have
less of an impact, and then jointly optimizing the cameras and
the terrain at ever increasing levels of resolution. Yet, this may still
fail if the terrain does not have large and pronounced features on the
scale bigger than the errors in the cameras.

The approach that we found to work all the time is to manually select
interest points in the images, as the human eye is much more skilled at
identifying a given landmark in multiple images, even when the lightning
changes drastically. Picking about 4 landmarks in each image is
sufficient. Ideally they should be positioned far from each other, to
improve the accuracy.

Below is one example of how we manually select interest points, run SfS,
and then how we compare to LOLA, which is an independently acquired
sparse dataset of 3D points on the Moon. According to
\cite{smith2011results}, the LOLA accuracy is on the order of 1~m. To
ensure a meaningful comparison of stereo and SfS with LOLA, we resample
the LRO NAC images by a factor of 4, making them nominally 4
m/pixel. This is not strictly necessary, the same exercise can be
repeated with the original images, but it is easier to see the
improvement due to SfS when comparing to LOLA when the images are
coarser than the LOLA error itself.

We work with the same images as before. To resample them, we do:
\begin{verbatim}
  for f in A B C D; do
    reduce from = ${f}_crop.cub to = ${f}_crop_sub4.cub sscale=4 lscale=4
  done
\end{verbatim}

We run \texttt{stereo} and \texttt{point2dem} to get a first cut DEM. We don't do bundle
adjustment at this stage yet.
\begin{verbatim}
  stereo A_crop_sub4.cub B_crop_sub4.cub run_stereo_noba_sub4/run --subpixel-mode 3
  point2dem --stereographic --proj-lon -5.7113451 --proj-lat -85.000351 \
    run_stereo_noba_sub4/run-PC.tif --tr 4
\end{verbatim}

We would like now to manually pick interest points for the purpose of
doing bundle adjustment.  We found it it much easier to locate the landmarks if we
first map-project the images, which brings them all into the same
perspective. We then pick interest points in \texttt{stereo\_gui}, and then project
them back into the original cameras and do bundle adjustment. Here are
the steps:
\begin{verbatim}
  for f in A B C D; do
    mapproject --tr 4 run_stereo_noba_sub4/run-DEM.tif ${f}_crop_sub4.cub \
      ${f}_crop_sub4_v1.tif --tile-size 128
  done
  stereo_gui A_crop_sub4_v1.tif B_crop_sub4_v1.tif C_crop_sub4_v1.tif \
    D_crop_sub4_v1.tif run_ba_sub4/run
\end{verbatim}
Interest points are selected by zooming and right-clicking with the
mouse, one point at a time, from left to right, and then saving them. An
illustration is shown in Figure \ref{fig:sfs3}.

\begin{figure}[t!]
\begin{center}
\includegraphics[width=7in]{images/sfs3.jpg}
\caption[sfs]{An illustration of how interest points are picked manually for the purpose of bundle adjustment and then SfS.}
\label{fig:sfs3}
\end{center}
\end{figure}

Then bundle adjustment happens:
\begin{verbatim}
  P='A_crop_sub4_v1.tif B_crop_sub4_v1.tif' # to avoid long lines below
  Q='C_crop_sub4_v1.tif D_crop_sub4_v1.tif run_stereo_noba_sub4/run-DEM.tif'
  bundle_adjust A_crop_sub4.cub B_crop_sub4.cub C_crop_sub4.cub D_crop_sub4.cub  \
    -o run_ba_sub4/run --mapprojected-data  "$P $Q"                              \
    --min-matches 1
\end{verbatim}

A good sanity check to ensure that at this stage cameras are aligned properly is to map-project
using the newly obtained camera adjustments and then overlay the obtained images in the GUI.
The features in all images should be perfectly on top of each other.
\begin{verbatim}
  for f in A B C D; do
    mapproject --tr 4 run_stereo_noba_sub4/run-DEM.tif ${f}_crop_sub4.cub  \
     ${f}_crop_sub4_v2.tif --tile-size 128 --bundle-adjust-prefix run_ba_sub4/run
  done
\end{verbatim}
This will also show where the images overlap, and hence on what portion of the DEM we
can run SfS.

Then we run stereo, followed by SfS.
\begin{verbatim}
  stereo A_crop_sub4.cub B_crop_sub4.cub run_stereo_yesba_sub4/run             \
    --subpixel-mode 3 --bundle-adjust-prefix run_ba_sub4/run
  point2dem --stereographic --proj-lon -5.7113451 --proj-lat -85.000351        \
    run_stereo_yesba_sub4/run-PC.tif --tr 4
  gdal_translate -srcwin 138 347 273 506 run_stereo_yesba_sub4/run-DEM.tif     \
    run_stereo_yesba_sub4/run-crop1-DEM.tif
  sfs -i run_stereo_yesba_sub4/run-crop1-DEM.tif A_crop_sub4.cub               \
    C_crop_sub4.cub D_crop_sub4.cub -o sfs_sub4_ref1_th_reg0.12_wt0.001/run    \
    --shadow-thresholds '0.00149055 0.00138248 0.000747531'                    \
    --threads 4 --smoothness-weight 0.12 --initial-dem-constraint-weight 0.001 \
    --reflectance-type 1 --float-exposure --float-cameras --max-iterations 20  \
    --use-approx-camera-models --use-rpc-approximation --crop-input-images     \
    --bundle-adjust-prefix run_ba_sub4/run
\end{verbatim}

We fetch the portion of the LOLA dataset around the current DEM from the
site described earlier, and save it as
\verb#RDR_354E355E_85p5S84SPointPerRow_csv_table.csv#. It is necessary
to align our stereo DEM with this dataset to be able to compare
them. We choose to bring the LOLA dataset into the coordinate system
of the DEM, using:
\begin{verbatim}
  pc_align --max-displacement 280 run_stereo_yesba_sub4/run-DEM.tif             \
    RDR_354E355E_85p5S84SPointPerRow_csv_table.csv -o run_stereo_yesba_sub4/run \
    --save-transformed-source-points
\end{verbatim}

Then we compare to the aligned LOLA dataset the input to SfS and its output:
\begin{verbatim}
  geodiff --absolute -o beg --csv-format '1:lon 2:lat 3:radius_km' \
    run_stereo_yesba_sub4/run-crop1-DEM.tif run_stereo_yesba_sub4/run-trans_source.csv
  geodiff --absolute -o end --csv-format '1:lon 2:lat 3:radius_km' \
    sfs_sub4_ref1_th_reg0.12_wt0.001/run-DEM-final.tif             \
    run_stereo_yesba_sub4/run-trans_source.csv
\end{verbatim}

We see that the mean error between the DEM and LOLA goes down, after SfS,
from  1.14~m to 0.90~m, while the standard deviation decreases from
 1.18~m to 1.06~m.

\section{Running SfS with an external initial guess DEM}

Sometimes it is convenient to run SfS with a DEM not created using ASP's
\texttt{stereo}. For example, for the Moon, the LOLA gridded DEM is
available. It is somewhat noisy and the nominal resolution is on the
order of 10 m/pixel, but it is available even for permanently shadowed
regions.

The main challenge in such a situation is that the images, such as
coming from LRO NAC, may not be aligned well to this external DEM and
among themselves, and then SfS will fail. To get an initial alignment,
what worked for us to coarsen both this DEM and the images to about 40
meters/pixel (using the ISIS \texttt{reduce} command and
\texttt{dem\_mosaic} with the options to blur and change the grid size),
and then run SfS over a reasonably large area (say about $500 \times
500$ pixels) with pronounced terrain and where images have a lot of
overlap with the option \texttt{-\/-float-all-cameras} (and of course
\texttt{-\/-float-exposure}, and an appropriate \texttt{-\/-shadow-threshold}).
Then, the SfS program will find
adjustments to the cameras, writing them in the output directory. The
\texttt{mapproject} tool can be used to map-project the coarse images
onto the input DEM using \texttt{-\/-bundle-adjust-prefix} pointing to
the \texttt{sfs} output prefix. This should be used to verify that images
are now aligned correctly, for example by overlaying them in
\texttt{stereo\_gui}. If so, these adjustments can be used as input for
SfS with images at finer levels of resolution (after appropriately renaming the
adjustment files and using the \texttt{-\/-bundle-adjust-prefix} option of \texttt{sfs}).

Here, a higher value can be used for \texttt{-\/-initial-dem-constraint-weight}
to ensure we cameras have more motivation to align to the terrain.

When it comes to running SfS with many large high-resolution
images, one runs very fast into memory constraints. It is then necessary
to parallelize the problem using \texttt{parallel\_sfs}, which will run
\texttt{sfs} on multiple tiles. Yet, the camera adjustments need to be
determined before running this tool, and then kept fixed when this tool
is run, as otherwise each single tile will optimize its cameras
independently, and as result there will be discontinuities at tile
boundaries.

If the camera adjustments are determined by first running \texttt{sfs}
on a clip representative of the entire terrain, far from that clip the
cameras will start to disagree. For that reason, \texttt{sfs} makes it
possible to optimize the cameras on an entire collection of clips,
chosen so they are reasonably spread out over the entire terrain,
and that each camera image is covered by at least a handful of such clips.
The clips can be passed to \texttt{sfs} as a quoted string via the \texttt{-i} option.
As before, at the end the \texttt{mapproject}
program can be used to verify that the camera images mapprojected using the
obtained camera adjustments are perfectly on top of each other, and if not,
more clips can be added to the joint optimization problem. When a good
enough set of camera adjustments is obtained, \texttt{parallel\_sfs} can
be run as before.

\section{Insights for getting the most of SfS}
\label{sfs:insights}

Here are a few suggestions we have found helpful when running \texttt{sfs}:

\begin{itemize}{}

\item First determine the appropriate smoothing weight $\mu$ by running a
small clip, and using just one image. A value between 0.06 and 0.12 seems to work
all the time with LRO NAC, even when the images are subsampled. The other weight, $\lambda,$ can
be set to something small, like $0.0001.$ This can be increased to $0.001$ if noticing
that the output DEM strays too far.

\item As stated before, more images with more diverse illumination conditions
result in more accurate terrain. Ideally there should be at least 3 images, with the shadows
being, respectively, on the left, right, and then perhaps missing or small.

\item Bundle-adjustment for multiple images is crucial, to eliminate
  camera errors which will result in \texttt{sfs} converging to a local
  minimum. This is described in section \ref{sfs-lola}.

\item Floating the albedo (option \texttt{-\/-float-albedo}) can
  introduce instability and divergence, it should be avoided unless
  obvious albedo variation is seen in the images.

\item Floating the DEM at the boundary (option
  \texttt{-\/-float-dem-at-boundary}) is also suggested to be avoided.

\item Overall, the best strategy is to first use SfS for a single image
  and not float any variables except the DEM being optimized, and then
  gradually add images and float more variables and select whichever
  approach seems to give better results.

\item If an input DEM is large, it may not be completely covered by a
single set of imagery with various illumination conditions.  It should
then be broken up into smaller regions (with overlap), the SfS problem
can be solved on each region, and then every output terrain can be
transformed using \texttt{pc\_align} into LOLA's global coordinate
system, where they can be mosaicked together using
\texttt{dem\_mosaic}. Or, \texttt{sfs} can be run not on one clip, but
on an entire collection of clips covering this area to get the
adjustments, and then \texttt{parallel\_sfs} can be run as described in
the previous section.

  The easier case is when at least the two images in the stereo pair
  cover the entire terrain. Then, portions of this terrain can be used
  as an initial guess for each SfS sub-problem (even as the other images
  used for SfS change), the results can be mosaicked, and the alignment
  to LOLA can happen just once, after mosaicking. This approach is
  preferable, if feasible, as alignment to LOLA is more accurate if the
  terrain to align is larger in extent.

\item The \texttt{mapproject} program can be used to map-project each image
onto the resulting SfS DEM (with the camera adjustments solved using
SfS). These orthoimages can be mosaicked using \texttt{dem\_mosaic}. If the
\texttt{-\/-max} option is used with this tool, it create a mosaic with
the most illuminated pixels from this image. If during SfS the camera
adjustments were solved accurately, this mosaic should have little or no
blur. An alterantive is to use the \texttt{-\/-block-max} option
which will pick the most lit of the images per each block, with the latter
being specified via \texttt{-\/-block-size}.
\end{itemize}


\section{Interpretation and Discussion of the Results}
%In this section, first the challenges of the given problem are summarized and an ideal model to solve the problem is defined. Second, all properties of the ideal model are discussed with the relation to selected models and their performance. Last but not the least, conclusion is made on which of the three models is the best for classification of glass fragments.

\subsection{Classification Challenges and the Ideal Classifier}
Building a good model to accurately predict glass fragments from a data set of $214$ data points turned out to be demanding. The core challenges were:

%\vspace{10pt}
\begin{enumerate}
    \item Little amount of data for 6-class classification problem
    \item Skewed class distribution
    \item Class overlap
\end{enumerate}
%\vspace{10pt}

In the light of the model being used in criminal investigation processes, an ideal model would be transparent in a way that the decision-making is comprehensible for humans. Furthermore, the model should be well-suited to tackle the challenges mentioned.

%Before a discussion of particular models, it is important to highlight the main challenges of the given classification problem and based on it define ideal model. First, certain classes of glass, such as $1$, $2$ and $3$, had a large overlap given its similar features. This implies a need for a complex model capable of handling such overlap and still being able to generalize well. Second, distribution of classes within the training and test split of data was skewed. Thus, minority classes might be very hard to predict. This goes hand in hand with the size of the training data-set which was only 149 records. Therefore, optimal model should be able to learn even from a small training data-set. Last but not the least, since the output of the model might be used as part of crime analysis, ideal model should be also interpretable, i.e. it should be clear how it decides.

\subsection{Comparison of Model Performances}
The three models evaluated within this project can be ranked by their performance (measured by the expected out-of-sample accuracy) as follows:

\begin{enumerate}
    \item \class{RandomForestClassifier()}
    \item \class{NeuralNetworkClassifier()}
    \item \class{DecisionTreeClassifier()}
\end{enumerate}

Looking at the individual classification reports, all models were able to almost perfectly predict class $7$ which goes hand in hand with the findings from the EDA and Figure \ref{pca}. In contrast, the prediction of the overlapping classes 1, 2 and 3 turned out to be generally challenging. In order to classify these classes correctly, the model needed to establish a complex decision boundary, that at the same time generalises well. The \class{RandomForestClassifier()} did the best job at separating the classes 1 and 2 from each other (Table \ref{random_forest_evaluation}), giving one indication of its overall good performance. The \class{DecisionTreeClassifier()} struggled the most at separating the overlapping classes (Table \ref{dt_evaluation}).
Another noticeable, common pattern from the classification reports is that the minority classes were generally more difficult to predict. That is reasonable, since a lack of training examples makes it difficult to learn class-specific properties.

\subsection{Interpretation of Model Performances}
The \class{RandomForestClassifier()}, being an ensemble of a large number of, uncorrelated decision trees, as well as the \class{NeuralNetworkClassifier()} are complex models, with longer training times, but more accurate predictions. This explains, why both models outperform the simpler \class{DecisionTreeClassifier()}.
The fact that the \class{RandomForestClassifier()} gives better results than the \class{NeuralNetworkClassifier()} is probably due to the fact that neural networks usually need larger amounts of training data to give good results. In fact, neural networks perform best on complex classification problems with large amount of data to learn from. Since this classification problem is quite the opposite of that, the neural network is not performing as well as the \class{RandomForestClassifier()}.

%- 1. difference between high performant, but slow/ less performant, but fast
%- 2. neural net not as good, because usually needs more data points

%- 3. however, when choosing a model, dt might stil be intersting, since transparent model (main advantage not only speed, but also interpretability)

\begin{comment}
First, all models were able to almost perfectly predict class $7$ which goes hand in hand with the findings from EDA and figure $2$ where even using only first two principal components, it was possible to see the clear separation of class $7$ from others. Second, more challenging proved to be a prediction of classes $1$ and $2$. In order to classify them correctly, the model needed to establish complex decision boundary. From this perspective, it was expected that \class{DecisionTreeClassifier()} will perform the worst as it can only divide feature space with straight boundaries. This expectation was wrong as \class{NeuralNetwork()} (both implementations) had similar results. One possible explanation is that \class{NeuralNetwork()} got stuck in a local minimum during training and even advanced optimizer (Adam) did not help to overcome this. More interestingly, \class{RandomForestClassifier()} performed way better than the other two models. Its core advantage is concealed in its ability to rely on a majority vote of a large and diverse set of \class{DecisionTreeClassifier()} trained only on a subsample of original data. Thanks to this diversity, it is capable of fitting complex patterns but even more importantly generalize well. In other words, it is less likely to make an error due to small variation in data.

\subsection{Learning from a small data set}
In the given data-set, there were three minor classes, namely $3$, $5$ and $6$. From EDA (Figure 3), it was clear that it will be very hard to classify them correctly as they all overlapped with the other majority classes. This proved to be a problem especially for \class{DecisionTreeClassifier()}. A good explanation can be found from its visualization and following the splits of class 3 for example. In majority of cases (3 out of 5), it ends being in a leaf node either with class 1 or 2 with which it has a large overlap. Further, the nodes are almost pure, thus the model has no further incentive to split it. In addition, since these classes have more records, they are also more likely to be predicted. Finally, compare to other two methods, \class{NeuralNetwork()} is parametric and given the chosen architecture, it has a large number of parameters to be tuned. Therefore, it also needs a larger training data-set in order for it to work well. This might be a possible explanation why it outperformed by \class{RandomForestClassifier()} for prediction of classes $1$ and $2$.

%\subsection{Interpretability}
%\class{DecisionTreeClassifier()}'s core advantage compare to the other two models is that its decision can be simply translated into just if-else statements. For example, can see that within its first two levels (root is level 1), it only uses first or second principal component to decide. This in fact makes sense as the first two components explain most of the variability of the given features and thus it is a good way to separate the classes. On the opposite site, trying to understand how a \class{RandomForestClassifier()} decided is way more difficult and infeasible as it would require to go through all the 100 trees.
\end{comment}

\subsection{Conclusion}
The analysis has shown that a model chosen solely on the basis of best performance (\class{RandomForestClassifier()}) has an expected out-of-sample performance of over 80\%, which is a solid result given the challenges of the problem at hand. However, in the light of the model being used in assisting criminal investigations, transparency in the way decision are being made might be relevant, i.e. in court trials. Thus, if interpretability of the model is a core requirement for the final model, a  \class{DecisionTreeClassifier()} should be considered.


\documentclass{article}
\usepackage[T1]{fontenc}%
\usepackage[utf8]{inputenc}%
\usepackage{lmodern}%
\usepackage{textcomp}%
\usepackage{lastpage}%
\input{common_symbols_and_format.tex}

\begin{document}
\logo
\rulename{Simple Moving Averages}
\tblofcontents

\ruledescription{
These representations have the position dependent on comparing two moving average windows one is short and the other is Long when the short window average length $\averagelengthshort$ is greater than the long one average length $\averagelengthlong$ so the model will go for the long position and when the short $\averagelengthshort$ is less than the long one $\averagelengthlong$, the position will be short}
\\

\ruleparameters
{Long Window size}{10}{This is the number of time steps over which exponential contributions are sourced}{$\averagelengthlong$}
{Short Window Size}{5}{This is the decay factor that reduces older contributions from the price series}{$\averagelengthshort$}
\stoptable
\section{Equation}
\begin{equation}
\sum_{\dummyiterator}^{\averagelengthlong} \price_{\currenttime - \dummyiterator} \\
\label{eq1}
\end{equation}

\begin{equation}
\sum_{\dummyiterator}^{\averagelengthshort} \price_{\currenttime - \dummyiterator} \\
\label{eq2}
\end{equation}

\[ \position_\currenttime = \begin{cases} \mbox{1,} & \mbox{if } equ.\ref{eq2}>equ.\ref{eq1} \\ \mbox{0,} & \mbox{if } equ.\ref{eq1}>equ.\ref{eq2} \end{cases} \]

\hspace{200mm}

where $\price_\currenttime$ is the price at time $\currenttime$,

\hspace{200mm}
\hspace{200mm}

\keyterms
\furtherlinks %The footer
\end{document}


\documentclass[a4paper, 11 pt, article, accentcolor=tud7b]{tudreport}

\usepackage[utf8]{inputenc}
\usepackage{amsmath}
\usepackage{placeins}
\usepackage{tabularx}
\usepackage{subcaption}

\title{CNuVS Exercise 9}
\author{Nils Rollshausen, Daniel Drodt}
\subtitle{Nils Rollshausen, Daniel Drodt}

\begin{document}
	\maketitle
	\section{TCP Congestion control}
	\subsection*{a) Congestion control vs flow control}
	Flow control intends to protect the receiver from overload while congestion control protects the network. TCP specifically uses a window-based approach to congestion control.

	\subsection*{b) Congestion window}
	The amount added to the congestion window with each ack is given as $inc = MSS \cdot \frac{MSS}{window}$. So, on the first acknowledgement, we increment the window by $1320 \cdot \frac{1320}{13200} = 132$ bytes to $13332$ bytes. On the second ack, we increment by $1320 \cdot \frac{1320}{13332} = 130.69$ byte to a new congestion window of $13462.69$ byte.

	\subsection*{c) Slow start}
	After establishing a connection or pausing the connection when congestion is detected, it would take a long time for TCP to incrementally adjust the congestion window to its proper value. To speed this process up, the congestion window is increased exponentially (as opposed to linearly in the regular congestion avoidance phase) until the desired window size is reached (half the previous window if a packet loss has been detected) or a packet is lost.

	\section{Three-Way-Handshake}

	\subsection*{a) Handshake}
	The three-way-handshake consists of the initiating party sending a SYN packet to the receiving party. The receiving party responds to the SYN request with a SYN-ACK packet acknowledging the receipt of the SYN request. Finally, the initiating party acknowledges the receipt of the SYN-ACK with an ACK packet.

	\subsection*{b) TCP Segment}

	In addition to the fields already given, the TCP header contains the following values:

	\begin{table}[h]
	  \centering
	  \begin{tabular}{|l|l|}
	    \hline
	    Source port & 0x076c \\ \hline
	    Destination port & 0x07cd \\ \hline
	    Data offset & 0x5 \\ \hline
	    Checksum & 0x3c8f \\ \hline
	  \end{tabular}
	  \caption{Remaining TCP header fields}
	\end{table}

	The application data is \verb|0x4e6f 7274 6865 726e 204c 6967 6874 7300|. The checksum is calculated as the inverse two's-complement sum as follows:

	\begin{align*}
	  0xFFFF - (&0x076c + 0x07cd + 0x0000 + 0x1337 + 0x0000 + 0x0000 \\
	            &+ 0x5020 + 0x5000 + 0x0000 + 0x0000 + 0x4e6f + 0x7274 \\
	            &+ 0x6865 + 0x726e + 0x204c + 0x6967 + 0x6874 + 0x7300) \\
	          = &0xFFFF - 0xC370 = 0x3C8F
	\end{align*}

	\newpage

	\subsection*{c) TCP header fields}

	\begin{table}[h]
	  \centering
	  \begin{tabularx}{\linewidth}{|l|X|}
	    \hline
	    Source port & The port from which the packet is sent \\ \hline
	    Destination port & The port to which the packet is sent \\ \hline
	    Sequence number & Sequence number of the packet, corresponds to bytes already sent \\ \hline
	    Acknowledgement number & If the ACK flag is set, this acknowledges the receipt of all bytes up to the acknowledgement number \\ \hline
	    Data offset & The length of the header in 32 bit words, used to determine where the payload starts \\ \hline
	    Reserved & Reserved bits intended for future extensions of the protocol \\ \hline
	    URG & Indicates that the Urgent Data Pointer field is valid and should be considered \\ \hline
	    ACK & Indicates that the acknowledgement number is valid \\ \hline
	    PSH & Indicated that received data should be pushed to the receiving process immediately \\ \hline
	    RST & Resets the connection \\ \hline
	    SYN & Indicates a connection request \\ \hline
	    FIN & Indicates a connection termination \\ \hline
	    Window Size & The amount of bytes the sender of the packet is currently willing to receive, starting at the value in the acknowledgement field \\ \hline
	    Checksum & The 16bit twos-complement checksum over the entire packet \\ \hline
	    Urgent Pointer & If URG is set, this indicates that the first n bytes from the message start are urgent \\ \hline
	  \end{tabularx}
	  \caption{TCP header fields}
	\end{table}

	\subsection*{d) Retransmission}
	\begin{figure}[h]
    \begin{subfigure}[b]{0.3\textwidth}
      \includegraphics[width=\textwidth]{retrans1.png}
      \caption{Lost ACK}
      \label{fig:1}
    \end{subfigure}
    %
    \begin{subfigure}[b]{0.3\textwidth}
      \includegraphics[width=\textwidth]{retrans2.png}
      \caption{Premature Timeout}
      \label{fig:2}
    \end{subfigure}
    %
    \begin{subfigure}[b]{0.3\textwidth}
      \includegraphics[width=\textwidth]{retrans3.png}
      \caption{Cumulative ACK}
      \label{fig:2}
    \end{subfigure}
  \end{figure}
	When an ACK packet is lost, the sender would, after a timeout, consider the packet whose acknowledgement got dropped to be lost and re-send it. The receiver recognizes the re-sent packet as a duplicate, discards it, and responds with an up-to-date acknowledgement.

	\FloatBarrier

	\section{RED, State Chart \& Nagle}

	\subsection*{a) Random Early Detection}
	Random Early Detection is necessary for routers and other network equipment to signal to end nodes that they are running close to their maximum capacity. As the routers should be agnostic to the choice of transport protocol, there is no protocol-inherent mechanism for them to communicate their capacity. As a solution, routers communicate implicitly by randomly dropping packets when running close to maximum load to reduce the network load before actually running completely out of capacity, which would lead to heavy congestion.

	\subsection*{TIME\_WAIT and ESTABLISHED}
	The TIME\_WAIT state is used by the party terminating a connection to ensure that the other party was able to terminate the connection properly (i.e. it received the acknowledgement to its FIN). A duplicate received FIN in this state indicates that the previous ACK was lost and should be retransmitted. After a sufficient amount of time passes, it can be assumed that no such duplicate FIN will arrive and the remote party was able to terminate the connection. \\ \medskip
	The ESTABLISHED state is the normal state of a TCP connection in which both parties have established the connection and are able to send and receive data.

	\subsection*{c) Nagle's Algorithm}
	Nagle's Algorithm deals with the problem network congestion caused by acknowledging small amounts of data and thus creating very high network overhead for comparatively little data. It states that data should only be sent if either the amount of data to send is larger or equal to the maximum segment size or there is no unacknowledged data packet still in-flight to the receiver. Using this algorithm, non-full packets are only sent when it is assumed that the receiver is currently idle.

	\end{document}