%-----------------------------------------------------------------------------------------------
\section{Conclusions}

    This     work     has     illustrated     the     validity     of      Carnot's      general
    proposition~\cite{1897-ThurstonRH-Wiley} by using a parametric, reversible  Stirling  engine
    model from the literature~\cite{2012-ChengCH+YangHS-ApEnergy} that takes into account engine
    geometry and working fluid properties. Moreover, the analytic nature of the model, with  its
    exact solution, helps in keeping the illustration concise and exact, and generic up  to  the
    model's modeling space.

    In one hand, the example illustrated herein is less generic and less conceptual (i.e.,  more
    instantiated) than the canonical proof by contradiction using only general concepts, usually
    presented  in   thermodynamics   courses   and   textbooks~\cite{2013-CengelYA+BolesMA-AMGH,
    2002-MoranMJ+ShapiroHN-LTC, 1986-JonesJB+HawkinsGA-Wiley}, and therefore, evidently does not
    serve as a general replacement of the proof by contradiction.

    On the other hand, the illustration based on a reversible model of a \emph{real}  engine  is
    more defined and hopefully more palpable and interesting  to  inquiring  students.  One  can
    argue that at least the illustration complements the proof by contradiction  and  may  boost
    interest by the more concrete approach.

%-----------------------------------------------------------------------------------------------


\chapter{Introduction}
The energy difference between an uncharged species and its negative ion, referred to as an electron affinity (EA), is an important property of atoms and molecules. \cite{oldpaper}
The application of EA spreads over many chemical areas including gas-phase ion chemistry \cite{gasion}, pure chemistry \cite{pure}, material science and environmental chemistry \cite{environment}. EA also plays an important role in silicon \cite{silicon} and quantum dot \cite{quantumdot} semiconductor chemistry and polymer photo-luminescence \cite{luminescence}.

Experimentally, EA can be measured by a photon source of energy $h\nu$ to detach an electron from molecular anion $\text{A}^{-}$.
By determining the minimum photon energy needed to detach an electron, one determines the EA.
The most direct technique for determining negative EAs is to use electron transmission spectroscopy in which a beam of electrons having kinetic energy impinges on a neutral molecule.
If the negative kinetic energy matches the (negative) EA of the molecule, one of the beam’s electrons can be captured into an empty orbital of the neutral to form the metastable anion, which can be detected either by measuring attenuation of the incident electron beam or by probing the electron ejected at right
angles to this beam.

However, theoretical study of EA is still important.
First, many of chemical substances are constituents in many ionic compounds and materials, so their stability, spectra, sizes, and other chemical properties need to be characterized, and often it is difficult to study them experimentally.
For example, the ubiquitous sulfate anion $\text{SO}_4^{2-}$ cannot be studied experimentally as an isolated species because it undergoes spontaneous electron loss within a very short time (less than $10^{-14}$ s), but it has been studied theoretically \cite{SO4} using special tools to handle its metastable character.

In addition, anions bind their outermost electrons rather weakly , and hence their valence-range electron densities are diffuse.
This causes them to interact strongly with their environment (e.g., in solution or in crystals), making their behavior strongly influenced by the surroundings and causing them to be excellent probes of their environment.

However, theoretical study of EA is not a easy task compared with ground state energy.
The reason is that EA is only a very small part of total ground state energy.
Thus, the energies of neutron and anion must be determined to high orders.
For example, EA usually lies between 0-5 eV, while the ground state energy of very small molecule like $\text{C}_2\text{H}_6$ is more than 2000 eV.

In history, the early work of Pekeris obtained a theoretical EA for a single hydrogen atom, which matched really well with experiments. \cite{pekeris}
However, a lot of difficulties have been met when it comes to EA of other atoms or molecules, ending up with many negative EA results.
As is pointed out by Blondel \cite{blondel}, the existence of negative ions depends significantly upon the instantaneous correlation of the motions of electrons, which means Hartree-Fock will not give even qualitatively correct result.

By far, the most popular method for EA is density functional theory (DFT) because of its wide application to many problems, high reliability and also because it is easy to use and computationally cheap.
However, DFT does not give very good result compared with high-level electronic structure methods like Coupled Cluster (CC).
CC explicitly include consideration of electron affinity and can thus give very impressive results.
However, CC is a computationally very expensive method because nonlinear equations need to be solved.
Thus, applications of CC are limited to medium sized systems with up to 60 atoms.

Another prominent quantum chemistry method to study EA of small and medium-sized molecules is the algebraic diagrammatic construction scheme (ADC).. \cite{implementation, ADCpp}
Due to its size-consistency and Hermitian structure, the ADC scheme is known to be an accurate and reliable approach for the calculation of anion states and their properties.
Generally, the approach is based on a Green’s function formalism in combination with the typical M{\o}ller-Plesset (MP) partitioning of the Hamiltonian.
I implemented restricted calculation of Non-dyson ADC algorithm into a developmental version of \emph{adcman}, which is a part of Q-Chem program package.
The results match well with that of CC, while consuming less time.

The thesis is organized as follows:

In chapter 2, I will present important and popular quantum chemistry methods and give a comparison between them from computational complexity, size consistency and performance to include electron correlation.
In chapter 3, I will introduce the theory of ADC from Green function and perturbation theory.
I will discuss both Dyson approach and Non-dyson approach and give a comparison of both methods.
I will also introduce intermediate state representation (ISR) for a better understanding of the efficiency and size consistency of ADC.
In chapter 4, I will give an overview of Q-chem and how does \emph{adcman} work.
I will also discuss my implementation of EA Non-dyson ADC to Q-Chem and the results of benchmark calculations.


\chapter{Introduction}
Digits recognition is a well-known and thoroughly explained subject in the image processing field. The outcome of those studies has significant impact for many aspects in people's lives. We used to provide credit curd number by hand some time ago, until the cell phones started to do it for us with the help of the camera. Automatic opening the gate basing on the plates numbers for entitled cars in the parking or motorway gates for those whose owners paid the toll upfront. These examples are just a pick of an iceberg of the potential digits recognition use cases.

Image recognition and deep learning algorithms became globally available not only for scientists, but also for ordinary individuals, especially because of high level libraries like Tensorflow or PyTorch where the exact knowledge of how the particular algorithm works is often irrelevant for the sake of the API. Developer has to invoke proper function with the desired parameters, but the function itself which for example creates and trains neural network is in the form of the black box. Enormous amount of tutorials and resources also make the entry threshold relatively low for the person without prior knowledge in the topic.
\paragraph{MNIST}
Crucial role in many if not the most image recognition algorithms and deep learning plays a dataset. It is necessary to gather significant amount of training data to properly teach for example the neural network and subsequently test in order to evaluate the quality of recognition. The process of gathering such a huge amount of data is more often the not tedious and takes a lot of time. This way MNIST database emerged for the sake of digits recognition. It contains 70000 handwritten digits, where 60000 and 10000 are consequently images for training and testing purposes \cite{lecun-mnisthandwrittendigit-2010}. Digits were written by high school students and employees of the United States Census Bureau. \figurename{} \ref{fig:mnist_example} shows several images from the MNIST dataset.
\begin{figure}[H]
	\begin{center}
		\scalebox{.7}{\includegraphics{./pictures/mnist_example.png}}
	\end{center}
	\caption{MNIST dataset example}

	\label{fig:mnist_example}
\end{figure}
To give an impression of the current level of digits recognition algorithms basing on MNIST dataset, several machine learning methods are juxtaposed on the \tablename{}~\ref{tab:mnist_comparison}.
\begin{table}[h!]
\centering
\begin{tabular}{    |l|c|c|c|  }
\hline
  Type & Classifier & Error rate (\%)  \\
 \hline
  Linear classifier & Pairwise linear classifier & \num{7.6} \cite{b:Lecun} \\
 \hline
  K-Nearest Neighbors & K-NN with non-linear deformation (P2DHMDM) & \num{0.52} \cite{b:Keysers} \\
 \hline
   Deep neural network (DNN) & 2-layer 784-800-10 & \num{1.6} \cite{b:Simard} \\
 \hline
   Deep neural network (DNN) & 6-layer 784-2500-2000-1500-1000-500-10 & \num{0.35} \cite{b:Cire_an_2010} \\
 \hline
\end{tabular}
 \caption{Several deep learning algorithms comparison in the digits recognition basing on MNIST database}
\label{tab:mnist_comparison}
\end{table}
It is very easy to take an advantage of this database, whilst working with for example Tensorflow, because the dataset can be downloaded directly using the library API. In this paper MNIST dataset will be exploited only for the training purposes.

\paragraph{How it works}
To goal is to recognise individual handwritten digits (outside the MNIST database) in real time by the camera. The images are captured in Python, resized to the 28x28 size and sent to the embedded system through UART, where deep feed forward neural network is implemented. Detected digit is displayed on the 7 segment display. System provides two ways of preprocessing the images. Adaptive Gaussian thresholding with the help of Open CV's API and Otsu thresholding implemented from the basis both in pure Python and C (in embedded system). Desired preprocessing might be chosen in the main script of communication application (main.py):
\begin{minted}[frame=single, linenos]{python}
class Preprocessing():
    ADAPTIVE_THRESHOLD = 1
    OTSU = 2
    OTSU_ORIGINAL_IMAGE_NO_SERIAL = 3 # no data transfer to the uC -- just
                                      # for the visualisation. OTSU
                                      # on not resized image, high quality
    OTSU_EMBEDDED = 4

###################################################
# configuration part:
###################################################
SERIAL = False
PREPROCESSING = Preprocessing.OTSU_ORIGINAL_IMAGE_NO_SERIAL
\end{minted}
Two variables are to be adapted by the user. If the SERIAL flag is true, data are being sent to the microcontroller, otherwise all process of recognition is done on the PC. User chooses the preprocessing algorithm with the PREPROCESSING variable.

Results evaluation and the whole process of recognition is thoroughly discussed in the following parts of the paper.

\paragraph{Project structure}
Figure \ref{fig:project structure} presents project structure. It is divided into two main sub projects -- neural\_network with Python and C feed forward implementation in two version: one and two hidden layers, and video, which is an communication project -- main.py contains aforementioned SERIAL and PREPROCESSING configuration, so it directly sends an image content to the microcontroller. 1\_hidden and 2\_hidden contains the same content, but with different amount of neural network hidden layers.
\begin{figure}[H]
	\begin{center}
		\scalebox{.7}{\includegraphics{./pictures/structure.pdf}}
	\end{center}
	\caption{Project structure}

	\label{fig:project structure}
\end{figure}

\begingroup
\renewcommand{\cleardoublepage}{}
\renewcommand{\clearpage}{}
\chapter{Preprocessing}
\endgroup
Training images are provided in the 28x28 array already preprocessed in the MNIST database, however images provided in the real time even though conceptually the same are significantly different. Two first pictures from the left of the \figurename{} \ref{fig:7_images} present digit 7. First one comes from the MNIST database, second one is a raw picture taken by the camera. The difference is in the background of the digit, which is not purely white, what introduces huge distortion in the process of recognition. Background of the original MNIST image in gray scale is zero, background of the raw image is bigger than zero, maybe not high, but still, it makes the neural network which was trained on the MNIST dataset incapable of detecting the digits correctly. Therefore thresholding is necessary.
\begin{figure}[H]
	\begin{center}
		\scalebox{.09}{\includegraphics[origin=c]{./pictures/7_images.png}}
	\end{center}
	\caption{Set of preprocessed images. From the left: MNIST example, taken by the camera (raw), adaptive thresholding, Otsu on the original image, Otsu on the previously resized.}
	\label{fig:7_images}
\end{figure}
Image in the centre shows the outcome of adaptive thresholding. This algorithm helps when the lighting conditions are not always the same, which is exactly the case here. The idea is to mute the background and express the digit. Algorithm calculates the threshold for every pixel based on a small region around it. More narrowly it is a gaussian-weighted sum of the pixel's neighbourhood. Pixel with higher value than the threshold is cast to 255 (black), with lower to 0 (white). Open CV provides an API to take an advantage of this thresholding:
\begin{minted}[frame=single, linenos]{python}
arr   = np.asarray(cv2.imread('7.jpg', cv2.IMREAD_GRAYSCALE))
arr2  = cv2.resize(arr, (28, 28))
trunc = cv2.adaptiveThreshold(arr2, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C,
    cv2.THRESH_BINARY_INV, 11, 9)
\end{minted}

\begingroup
\renewcommand{\cleardoublepage}{}
\renewcommand{\clearpage}{}
\section{Otsu}
\endgroup
Otsu method is named after its author Nobuyuki Otsu. Calculated threshold separates the pixels into two classes foreground and background \cite{b:otsu}. The method bases on minimising the between class variance or equally by maximising within class variance, but the first one is significantly faster. \figurename{} \ref{fig:otsu_example} shows the result of Otsu thresholding for the image captured by laptop build-in camera.
\begin{figure}[H]
	\begin{center}
		\scalebox{.5}{\includegraphics[origin=c]{./pictures/otsu_full_quality.png}}
	\end{center}
	\caption{Image captured by build-in laptop camera processed by Otsu}
	\label{fig:otsu_example}
\end{figure}
\subsection{Algorithm}
The outcome of the method comes down to the division of the whole set of pixels into two classes, foreground and background. Otsu proposed to separate the pixel clusters by maximising the withing class variances, what means maximising the clusters separation. The variance is calculated on the distribution of the pixel intensities, so the histogram of the image, what leads to the probability distribution. This is given by the formula \ref{eq:otsu_1}, where $\sigma_W$ represents within class variance, $\sigma_{b}^{2}$ and $\sigma_{f}^{2}$ variances of the background and foreground classes, $W_b$ and $W_f$ are the probabilities that the classes are separated by the threshold T.
\begin{equation} \label{eq:otsu_1}
\sigma_{W}^{2}(T) = W_b(T)\sigma_{b}^{2}(T) + W_f(T)\sigma_{f}^{2}(T)
\end{equation}
Between class variance is given by the relation \ref{eq:otsu_2}
\begin{equation} \label{eq:otsu_2}
    \sigma_{B}^{2}(T) = \sigma^{2} - \sigma_{W}^{2}(T)
\end{equation}
What finally results (after the calculations) in the relation \ref{eq:otsu_3}, where $\mu_b$ and $\mu_f$ are the weighted means of the particular cluster.
\begin{equation} \label{eq:otsu_3}
    \sigma_{B}^{2}(T) = W_b(T)W_f(T)(\mu_{b}(T) - \mu_{f}(T))^2
\end{equation}
The following listing shows the algorithm implementation in C:
\begin{minted}[frame=single, linenos]{C}
void otsu(int image[], int size) {
    int histogram[BINS_NUMBER] = {0};
    calculate_histogram(histogram, image, size);

    float sum = 0;
    for(int i = 0; i < BINS_NUMBER; i++) sum += i*histogram[i];

    float sumB = 0, wB = 0, wF = 0, varMax = 0,
    int threshold = 0;
    for(int t = 0; t < size; t++) {
        wB += histogram[t];
        if(wB == 0) continue;
        wF = size - wB;
        if(wF == 0) break;

        sumB += (float)t*histogram[t];
        float mB = sumB / wB;
        float mF = (sum-sumB) / wF;
        float varBetween = wB * wF * (mB-mF)*(mB-mF);

        if(varBetween > varMax) {
            varMax = varBetween;
            threshold = t;
        }
    }
    printf("threshold = %d\n", threshold);
    for(int i = 0; i < size; i++) image[i] = image[i] > threshold ? 0 : 255;
}
\end{minted}
Where the histogram is calculated in the following way:
\begin{minted}[frame=single, linenos]{C}
void calculate_histogram(int histogram[], int image[], int size) {
	for(int i = 0; i < size; i++) {
		histogram[image[i]]++;
	}
}
\end{minted}
Two last images from the picture \ref{fig:7_images} present consecutively the result of original image Otsu thresholding and the same image, but previously resized to 28x28 pixels. Adaptive threshold for this image gives better results, however it is not always the case.
\begingroup
\renewcommand{\cleardoublepage}{}
\renewcommand{\clearpage}{}
\chapter{Neural network}
\endgroup
Deep feed forward was chosen as a neural network. Feed forward pass was implemented in pure Python without high level libraries like NumPy or Tensorflow, then reimplemented in C in order to be placed in the microcontroller. However the process of training was outsourced to the Tensorflow in version 2. Weights and biases from the trained multilayer perceptron then are saved in the C header files.

\section{Feedforward neural network}
Feedforward neural network is a multilayer perceptron with several hidden layers extensively used in machine learning along with deep convolutional neural networks and recurrent neural networks. The forward pass, so the process of thinking is relatively simple and performance effective. \figurename{} \ref{fig:feedforward} shows deep feedforward neural network with multiple inputs and outputs. In the feed forward pass all inputs are multiplied by weights in the connection with the first hidden node what results in the net input $z_1[0]=i[0]w_1[0][0]+i[1]w_1[1][0]+\dots+i[k]w_1[k][0] + b_1[0]$. Subsequently the net input $z$ is applied to the activation function. The process is performed for every node in every hidden and output layer, where output from the particular node is an input of the next node. It is worth noting that the number of nodes in the hidden layers not necessarily has to be equal. Formal process of the feed forward pass for particular the node is written in \ref{eq:feedforward}, where $f_k$ represents activation function for the layer $k$ which is either ReLU for hidden layers or sigmoid for the output one. Rectified linear unit (ReLU) has an advantage over sigmoid function used in the output layer, because it does not have vanishing gradient, which prevents neural network from efficient learning. In fact ReLU has the constant gradient. Sigmoid activation function is given by \ref{eq:sigmoid}, whereas ReLU by \ref{eq:ReLU}.

\begin{equation} \label{eq:feedforward}
\begin{aligned}
    z_k[0]&=n_{k-1}[0]w_k[0][0]+n_{k-1}[1]w_k[1][0]+\dots+n_{k-1}[m]w_k[m][0] + b_k[0] \\
    n_k[0]&=f_k(z_k[0]) \\
\end{aligned}
\end{equation}

\begin{equation} \label{eq:sigmoid}
f(z)=\frac{1}{1-e^{-z}}
\end{equation}

\begin{equation} \label{eq:ReLU}
f(z)=max(0, z)
\end{equation}

\begin{figure}[H]
	\begin{center}
		\scalebox{.6}{\includegraphics{./pictures/feedforward.pdf}}
	\end{center}
	\caption{Feedforward neural network with multiple inputs and outputs. On the bottom of every layer, activation function was presented.}

	\label{fig:feedforward}
\end{figure}
Python implementation of the feedforward pass is presented on the following listing. This function is invoked for every hidden and output layer. Then the result of particular layer is passed to the consecutive one.
\begin{minted}[frame=single, linenos]{python}
def feedForward(self, inputs, weights_matrix, biases, activationFunction):
    input_length = len(weights_matrix)
    output_length = len(weights_matrix[0])
    x = inputs
    y = []
    z = []  # without activation

    for o in range(0, output_length):  # calculate each output in order
        y.append(0)
        z.append(0)
        for i in range(0, input_length):
            y[o] += weights_matrix[i][o] * x[i]

        z[o] = y[o] + biases[o]

    for o in range(0, output_length):
        y[o] = activationFunction(z[o])

    return y, z
\end{minted}
And the process of thinking, so the way of feedForward invocation for the consecutive layers. In the example 1 hidden layer.
\begin{minted}[frame=single, linenos]{python}
def think(self, x):
    (self.l1_output, self.l1_z) = self.feedForward(x, self.w1, self.b1,
        self.ReLU)
    (self.output, self.output_z) = self.feedForward(self.l1_output, self.w2,
        self.b2, self.sigmoid)
    return self.output
\end{minted}
Following two listings show the implementation in C which are directly used by the microcontroller. Moving from Python to C is not trivial in this case, because two dimensional weights matrix is expressed in one dimension. Otherwise it would not be possible to pass different sizes of this matrix. Particular weight is obtained in the line 9. Activation function is passed as a function pointer.
\begin{minted}[frame=single, linenos]{C}
void feedforward(const double x[], double z[], const double weights_matrix[],
    int ROW, int COL, const double biases[], double y[],
    double (*activationFunction)(double)) {
	int o, i;
	double cur_w;

	for(o = 0; o < COL; o++) {
		for(i = 0; i < ROW; i++) {
			cur_w = *(weights_matrix + i*COL + o);
			y[o] += cur_w * x[i];
		}

		z[o] = y[o] + biases[o];
	}

	for(o = 0; o < COL; o++) {
		y[o] = (*activationFunction)(z[o]);
	}
}
\end{minted}
And the process of thinking:
\begin{minted}[frame=single, linenos]{C}
void think(const double x[], double y[], double z[]) {
	double l1_output[128] = {0}, z1[128] = {0};
	feedforward(x        , z1, w1[0], 784, 128, b1, l1_output, &ReLU);
	feedforward(l1_output, z,  w2[0], 128, 10,  b2, y, &sigmoid);
}
\end{minted}
\section{Training process}
Training was performed using Tensorflow library in the version 2.2 however implementation in pure Python is also provided (check Appendix \ref{ch:back}). Time of the training varies between several seconds to roughly a few minutes, depending on the amount of epochs, nodes in the particular layer and the layers itself. Neural network is fully connected. As a lost function sparse categorical cross entropy was used, because the classes are mutually exclusive. It is given by the formula \ref{eq:bce}, where $y_a$ and $y_d$ are the actual outputs of the neural network respectively and $L$ is the loss function.
\begin{equation} \label{eq:bce}
L(y_a,y_d)=-\frac{1}{N}\sum_{i=1}^{N}[y_{a_i}log(y_{d_i})+(1-y_{a_i})log(1-y_{d_i})]
\end{equation}
The training process starts with the model definition, goes through the training itself and finally finishes on the model saving as well as the weights and biases in the proper format. The following listing shows the training implementation for 1 hidden layer with 128 nodes, input with $28\cdot28=784$ nodes and 10 output neurons.
\begin{minted}[frame=single, linenos]{python}
def createModel(self):
    self.model = keras.Sequential([ # layers in sequence
                 keras.layers.Flatten(input_shape=(28, 28)),
                 keras.layers.Dense(128, activation="relu"),  # fully connected
                 keras.layers.Dense(10, activation="sigmoid")
            ])
    self.model.compile(optimizer="adam",
    loss="sparse_categorical_crossentropy", metrics=["accuracy"])

def trainModel(self, epochs_num, save=True):
    self.model.fit(self.train_data, self.train_labels, epochs=epochs_num)
    test_loss, test_acc = self.model.evaluate(self.test_data, self.test_labels)
\end{minted}
Training takes an advantage of MNIST database as it was already aforementioned. When loaded, images are scaled to the range of 0-1.
\begin{minted}[frame=single, linenos]{python}
(self.train_data, self.train_labels), (self.test_data, self.test_labels) =
    keras.datasets.mnist.load_data()
self.train_data = self.train_data / 255.0
self.test_data = self.test_data / 255.0
\end{minted}
Weights and biases are saved in the proper format, more narrowly they are exported to the .txt file along with the whole model and then transformed to the header files with the help of regexp and specially designed bash script.

\begingroup
\renewcommand{\cleardoublepage}{}
\renewcommand{\clearpage}{}
\chapter{PC side}
\endgroup
In this project camera is directly connected to the PC, but may be external one. The images are captured in real time, resized and depending on the need directly sent to the microcontroller or previously preprocessed. All of that is implemented in Python 3 with the help of OpenCV and serial package. The following listing shows the serial configuration.
\begin{minted}[frame=single, linenos]{python}
ser = serial.Serial(
    port='/dev/ttyACM0',
    baudrate=115200,
    parity=serial.PARITY_NONE,
    stopbits=serial.STOPBITS_ONE,
    bytesize=serial.EIGHTBITS
)
\end{minted}
When the picture is ready to be sent it is flattened, that is changed to one dimensional array. Then every pixel is packed in 3 bytes format and sent. Synchronisation with the microcontroller is sygnalised by 2 special values: "aaa" stands for the beginning of the transfer and "bbb" for the end. Right after the beginning characters 3 bytes distinguishing the process of preprocessing are sent -- "OOO" in case of Otsu implemented on embedded system, "AAA" otherwise (outside microcontroller) for example adaptive thresholding.
\begin{minted}[frame=single, linenos]{python}
def send(ser, data, type):
    ser.write("aaa".encode())
    if type == Preprocessing.OTSU_EMBEDDED:
        ser.write("OOO".encode())
    else:
        ser.write("AAA".encode())

    for i in data.flatten():
        to_send = '{:3d}'.format(i)
        ser.write(to_send.encode())
    ser.write("bbb".encode())
\end{minted}
It is worth to mention, that the bottleneck of the system is exactly the communication part. Even with the maximum baud rate accessible for UART, sending roughly $28\cdot28\cdot3 + 6$ bytes takes significant amount of time (around \num{1.5} second) which stops systems fluency. Time of neural network computation comparing to this is negligible.

\begingroup
\renewcommand{\cleardoublepage}{}
\renewcommand{\clearpage}{}
\chapter{Microcontroller}
\endgroup
Neural network and the whole process of recognition is implemented on the microcontroller STM32H743ZIT6U. Embedded system is placed on the Nucleo board, which is directly connected to the computer through USB. The board was chosen mostly because of high volume of memory. It has 2 Mbytes of Flash memory, where weights and biases resides in the constant arrays and 1 Mbyte SRAM where is the rest of the program, image array for instance. Moreover the core of this device is highly efficient ARM Cortex-M7 at 480MHz, which is desired from the neural network calculations point of view. Figure \ref{fig:STM32H743ZIT6U} presents the Nucleo board with 7 segment display showing number 5.
\begin{figure}[H]
	\begin{center}
		\scalebox{.18}{\includegraphics[origin=c]{./pictures/uC.jpg}}
	\end{center}
	\caption{STM32H743ZIT6U with 7 segment display, which shows number 5 as a result of the image recognition}
	\label{fig:STM32H743ZIT6U}
\end{figure}

\begin{figure}[H]
	\begin{center}
		\scalebox{.4}{\includegraphics[origin=c]{./pictures/cube_config.png}}
	\end{center}
	\caption{Peripherals STM32CubeMX configuration}
	\label{fig:cube}
\end{figure}
The explanation of the whole process of peripherals configuration is omitted to focus mainly on the program logic. Nevertheless STM32CubeMX peripherals assignment is presented on the \figurename{} \ref{fig:cube}.
Microcontroller's main loop is responsible mainly for the neural network feed forward pass. Nevertheless apart from that there are a few operation worth to explain. First of all "done" flag indicates the end of the image transfer through UART. This flag is set in the interrupt callback, hence volatile. Subsequently Otsu preprocessing is done depending if it is requested via UART message ("OOO" bytes), this flag is also set in the interrupt callback handler. When it is done, image is rescaled and the process of recognition is performed. The result is displayed on the 7 segment display and finally the readiness of transfer is signalised by setting done flag to false.
\begin{minted}[frame=single, linenos]{C}
#define IMAGE_SIZE 784
uint8_t Received[3];
double image[IMAGE_SIZE];
volatile bool done = false;
volatile bool is_otsu = false;
////
while (1) {
    if(done) {
        if(is_otsu) {
            otsu(image, IMAGE_SIZE);
        }

        for(int i = 0; i < IMAGE_SIZE; i++) {
            image[i] = image[i] / 255.0;
        }

        think(image, y, z);
        display_digit(max_idx(y));
        done = false;
    }
}
\end{minted}
Interrupt handler for the UART receive event is presented on the following listing. The logic is similar to the process of sending the messages in the communication program, discussed previously. Start and finish flag, "a" (first letter of "aaa"), "b" consecutively. Type of the preprocessing, and the image content itself. Passing around 2000 bytes through UART is a bottleneck of the system as it was already stated. Interrupts introduce the notion of overlapping of the two images transmission -- during the detection of the first one, second image may be being transmitted. However the question is why not to exploit the DMA? Unfortunately there are several issues with DMA in this type of STM32 \footnote{\url{https://community.st.com/s/article/FAQ-DMA-is-not-working-on-STM32H7-devices}}, so the notion of using it was abandoned.
\begin{minted}[frame=single, linenos]{C}
int counter = 0;

void HAL_UART_RxCpltCallback(UART_HandleTypeDef *huart) {
	if(Received[0] == 'a') {
		counter = 0;
	} else
	if(Received[0] == 'b') {
		done = true;
	} else
	if(Received[0] == 'O') {
		is_otsu = true;
	} else
	if(Received[0] == 'A') {
		is_otsu = false;
	} else {
		int value = atoi((const char*)Received);
		image[counter++] = (double)value;
	}

	HAL_UART_Receive_IT(&huart3, Received, 3);
}
\end{minted}

\begingroup
\renewcommand{\cleardoublepage}{}
\renewcommand{\clearpage}{}
\chapter{Experiments and results}
\endgroup
Feed forward with one hidden and 2 hidden layers was tested. More layers could not fit in the microcontroller's flash memory. First of all accuracy of the training process on the MNIST database is presented on the \tablename{}~\ref{tab:mnist_comparison2}.
\begin{table}[h!]
\centering
\begin{tabular}{    |l|c|c|c|c|  }
\hline
  Type & Classifier & Accuracy (\%)  & Epochs number & Loss \\
 \hline
   Feed forward & 1-layer 784-800-10 & \num{0.9721} & 5 & \num{0.0864} \\
 \hline
   Deep feed forward & 2-layer 784-512-128-10 & \num{0.9793} & 5 & \num{0.0727} \\
 \hline
\end{tabular}
 \caption{1 hidden and 2 hidden layers training results}
\label{tab:mnist_comparison2}
\end{table}
Admittedly the training quality rates, namely accuracy and loss (sparse categorical cross entropy) are similar. However one has to remember, that the training was performed on the MNIST database, but the real testing is in the real time, with hand written, subsequently preprocessed images. From the quality point of view, so how two neural networks behave in such conditions, they are capable of detecting the digits correctly. Deep feed forward neural network from the user experience behaves better than 1 hidden layer neural network, in the same time characterising by the similar performance. It is difficult to evaluate the efficiency quantitatively without huge set of images. Same applies to the preprocessing algorithms. Both of them are able to preprocessed the images satisfactory nonetheless. However as it might have been observed on the Figure \ref{fig:7_images}, Otsu is highly vulnerable for lighting differences, hence from time to time some part of the digit may not be exposed correctly (last two images on the Figure \ref{fig:7_images}). As an attachment to this report several test recordings may be found in the projects repository\footnote{\url{https://gitlab-stud.elka.pw.edu.pl/jwieczo1/numbers_detection}}.

\begingroup
\renewcommand{\cleardoublepage}{}
\renewcommand{\clearpage}{}
\chapter{Conclusions}
\endgroup
In this project deep feed forward neural network was implemented on the embedded system for the sake of visual recognition. Nowadays the biggest obstacle for deep learning algorithms is not the performance of the microcontroller, but the memory volume. Even with 2 Mbytes of the flash memory, it was possible to place there only 2 hidden layers. The bottleneck of the system is the image transfer to the STM32, therefore in the future improvement USB protocol is worth considering. Neural networks were trained basing on MNIST dataset instead of specially prepared images. Nevertheless preprocessing approximated the images captured in the real time, so it was not a problem for the detection. It is worth noting, that a digit should be placed in the centre of the camera to be correctly read.

In addition to the neural network two preprocessing algorithms were compared, where Otsu was implemented from the basis and placed onto microcontroller. Adaptive threshold is more resistant to the light differences, so it behaves slightly better.

\begin{appendices}

\begingroup
\renewcommand{\cleardoublepage}{}
\renewcommand{\clearpage}{}
\chapter{Back propagation} \label{ch:back}
\endgroup
The following appendix shows an implementation of the one hidden layer feed forward back propagation algorithm. Without high level libraries, just pure Python. The difficulties in algorithm implementation grow with the number of layers, because back propagation bases on the gradient calculation, where particular gradient in the layer depends on many factors in the following layers. Thus, for the need of the project back propagation was outsourced to the Tensorflow.
\section{Output layer}
\begin{minted}[frame=single, linenos]{python}
def gradientsOutput(self, inputs, dedo, w, dActivationFunction, acti_arg1):
    """
    @param inputs, The outputs of the last hidden layer
    @param dedo,   Error gradient with respect to the output
    @param w,      An array of weights between output and the
                   hidden layer. Is constant
    @param dActivationFunction in the case of this layer is
                   dSigmoid
    @param acti_arg1 Argument for dActivationFunction. In this case
                   an output of the last layer
    """
    outputs_length = len(w[0])
    inputs_length = len(w)

    dedw = [[0 for o in range(outputs_length)] for i in range(inputs_length)]
    dedb = [0 for o in range(outputs_length)]
    for o in range(0, outputs_length):
        for i in range(0, inputs_length):
            dedw[i][o] = dedo[o] * dActivationFunction(acti_arg1[o])*inputs[i]

        dedb[o] = dedo[o] * dActivationFunction(acti_arg1[o]) * 1
    return dedw, dedb
\end{minted}
\section{Hidden layer}
\begin{minted}[frame=single, linenos]{python}
def gradients(self, inputs, dedo, w, w2, dActivationFunction, acti_arg1,
    dActivationFunction_2, acti_arg1_2):
    """
    @param inputs, The input of the nn
    @param dedo,   Error gradient with respect to the output
    @param w,      An array of weights between input and the
                   hidden layer. Is constant
    @param w2,     An array of weights between hidden layer
                   and the output. Gradient of w depends on them.
                   Is constant
    @param dActivationFunction activation for the hidden layer
    @param acti_arg1 Argument for dActivationFunction. If ReLU then
           should be net inputs z.
    @param dActivationFunction2 activation for the output layer
    @param acti_arg2 Argument for dActivationFunction2.
    """
    outputs_length = len(w[0]) # of w!
    inputs_length = len(w)

    dedw = [[0 for o in range(outputs_length)] for i in range(inputs_length)]
    dedb = [0 for o in range(outputs_length)]

    dedohw = [[0 for o in range(outputs_length)] for i in range(inputs_length)]
    dedohb = [0 for o in range(outputs_length)]

    outputs_length2 = len(w2[0])
    inputs_length2 = len(w2)

    for o2 in range(0, outputs_length2):
        dEdActiv = dedo[o2] * dActivationFunction_2(acti_arg1_2[o2])
        for o in range(0, outputs_length):
            dedoh = dEdActiv * w2[o][o2]
            for i in range(0, inputs_length):
                dedohw[i][o] += dedoh

            dedohb[o] += dedoh

    for o in range(0, outputs_length):
        dActiv = dActivationFunction(acti_arg1[o])
        for i in range(0, inputs_length):
            dedw[i][o] = dedohw[i][o] * dActiv * inputs[i]

        dedb[o] = dedohb[o] * dActiv * 1

    return dedw, dedb
\end{minted}
\section{Back propagation chain}
\begin{minted}[frame=single, linenos]{python}
def learn(self, x, y_desired):
    y = self.think(x)
    dedo = self.dSqe(y, y_desired)

    (dedw2, dedb2) = self.gradientsOutput(self.l1_output, dedo, self.w2,
        self.dSigmoid, self.output)
    (dedw1, dedb1) = self.gradients(x, dedo, self.w1, self.w2,
        self.dReLU, self.l1_z, self.dSigmoid, self.output)

    self.update(self.w2, self.b2, dedw2, dedb2)
    self.update(self.w1, self.b1, dedw1, dedb1)
\end{minted}

\section{Update function}
\begin{minted}[frame=single, linenos]{python}
def update(self, w, b, dedw, dedb):
    outputs_length = len(w[0])
    inputs_length = len(w)

    for o in range(0, outputs_length):
        for i in range(0, inputs_length):
            w[i][o] += dedw[i][o] * self.lr

        b[o] += dedb[o] * self.lr
\end{minted}

\end{appendices}


\section{Perils of disregard}
\label{sec:graph-importance}

As stated in Section \ref{sec:intro}, transportation demand models are used to evaluate the impact of policies on a certain
transportation demand related outcome. As an example, consider the proposals from the following fictitious scenarios:
\begin{itemize}
 \item Based on input from the public, a certain Department of Transportation (DOT) is considering implementing a new parking policy.
 This proposed policy would change prices and restrict availability to discourage individuals from parking in the central business district.
 The DOT and its constituents believe that this policy will encourage people to use more active transportation modes (e.g., walking, biking, etc.) and cause them to drive less.
 \item The DOT is considering constructing a streetcar (trolley, tram) line in a low/mid-income area in its jurisdiction.
 Citing examples from other cities and countries, the DOT claims that the new streetcar line will create more transit oriented developments and increase economic activity in the areas surrounding the proposed project.
 \end{itemize}

%% TODO: Present an example or two of real life project proposals?

Thinking more closely about these proposals, it is clear that they assume a causal relationship between the proposed project/policy and the desired goals.
The DOT in question analyzed data, concluding that such a policy or project would cause the desired output and achieve the desired goal.
In the presented scenarios, the DOT claims that implementing the new parking policy will cause an increase the share of
active transportation modes in the central business district and that constructing the proposed streetcar line will
cause more transit oriented developments and economic activity.

Policymakers base their analyses and conclusions on hypotheses or beliefs of how the world operates.
In other words, the data analysis is based on specific beliefs about the data generating process.
However, policymakers often do not present these beliefs in a clear and concise manner.
As a result, these proposals maintain an obscure representation of how the policy or project will achieve their desired goals.

DAGs allow one to clearly encode their assumptions about the data generating process and the problem at hand.
Researchers and practitioners have made use of DAGs in fields ranging from medicine and epidemiology (\citet{shrier_2008_reducing, sung_2012_reducing}) to economics (\citet{white_2011_covariate}) and have found them to be practical.

Likewise, DAGs could prove useful in addressing transportation policy questions.
\citet{brathwaite_2018_causal} have proposed a framework illustrating how practitioners and researchers can use DAGs to answer such transportation modelling questions in a causal context.
However, \citet{brathwaite_2018_causal} did not show an empirical application of their framework.

In this section, we will present an empirical example illustrating the importance of using DAGs in transportation demand modelling.
Specifically, we will illustrate how different assumptions about the data generating process result in different conclusions, even while assuming the same outcome model.
Before going any further, we note that this example is not meant to be all-encompassing.
That is, it does not reflect all complexities in a typical transportation choice modelling problem.
Indeed, it is not our primary goal in this section to recover the causal effect of the proposed intervention.
Instead, we are most interested in showing how different DAGs would result in different conclusions about the effect of the proposed intervention.

Let us assume that a company wants to reduce its workforce carbon footprint by moving its employees closer to their campus.
We would like to forecast how such an intervention would change the share of employees driving to work.
We model this travel mode choice problem based on a dataset from \citet{brathwaite_asymmetric}.
This dataset is based on the 2012 California Household Travel Survey, and it
contains approximately 4000 home-based school or work tours made by approximately 3850 individuals in the San Francisco Bay Area, California.
The dataset includes eight travel modes.
For our illustrative purposes, we focus on the following car-centric modes:

\begin{itemize}
   \item Drive Alone: The individual uses a private vehicle to make the trip
   \item Shared Ride 2: The individual shares an automobile ride with one more individual
   \item Shared Ride 3+: The individual shares an automobile ride with two or more individuals
\end{itemize}

Readers interested in a more detailed description of the dataset can refer to \citet{brathwaite_asymmetric}.
For the purposes of this exercise, we treat the multinomial logit model (MNL) defined in \citet{brathwaite_asymmetric} as the true outcome generating model.

In this model, the systematic utility equations of the car-centric modes defined above are specified as follows:
\begin{equation}
   \begin{aligned}
   \label{eq:DA}
   \textrm{Utility} \left(\textrm{Drive Alone}\right) &= \beta_{\textrm{travel\_time}} \times \textrm{Travel\_Time} + \beta_{\textrm{cost\_per\_distance\_drive\_alone}} \times \textrm{Cost\_per\_Distance}_{\textrm{da}} \\
   &\quad + \beta_{\textrm{autos\_per\_licensed\_driver}} \times \textrm{Autos\_per\_Licensed\_Driver} \\
   &\quad + \beta_{\textrm{cross\_bay\_da}} \times \textrm{Cross\_Bay}
   \end{aligned}
\end{equation}

\begin{equation}
   \begin{aligned}
   \label{eq:SR2}
   \textrm{Utility} \left(\textrm{Shared Ride 2}\right) &= ASC_{\textrm{shared\_ride\_2}} + \beta_{\textrm{time\_drive}} \times \textrm{Travel\_Time} \\
   &\quad + \beta_{\textrm{cost\_per\_distance\_shared\_ride\_2}} \times \textrm{Cost\_per\_Distance}_{ \textrm{sr2} } \\
   &\quad + \beta_{\textrm{autos\_per\_licensed\_driver}}  \times \textrm{Autos\_per\_Licensed\_Driver} \\
   &\quad + \beta_{\textrm{cross\_bay\_shared\_ride}} \times \textrm{Cross\_Bay} \\
   &\quad + \beta_{\textrm{hh\_size}} \times \textrm{Household\_Size} \\
   &\quad + \beta_{\textrm{n\_kids\_hh}} \times \textrm{Number\_of\_kids}
   \end{aligned}
\end{equation}

\begin{equation}
   \begin{aligned}
   \label{eq:SR3+}
   \textrm{Utility} \left(\textrm{Shared Ride 3+}\right) &= ASC_{\textrm{sr3+}} + \beta_{\textrm{time\_drive}} \times \textrm{Travel\_Time} \\
   &\quad + \beta_{\textrm{cost\_per\_distance\_sr3+}} \times \textrm{Cost\_per\_Distance}_{\textrm{sr3+}} \\
   &\quad + \beta_{\textrm{autos\_per\_licensed\_driver}}  \times \textrm{Autos\_per\_Licensed\_Driver} \\
   &\quad + \beta_{\textrm{cross\_bay\_shared\_ride}} \times \textrm{Cross\_Bay} \\
   &\quad + \beta_{\textrm{hh\_size}} \times \textrm{Household\_Size} \\
   &\quad + \beta_{\textrm{n\_kids\_hh}} \times \textrm{Number\_of\_kids}
   \end{aligned}
\end{equation}

Note that, since we consider this model to be the true outcome model, there are no latent variables we need to account for.
Below is a description of the key variables included in the model:

\begin{itemize}
  \item Total Travel Distance: the total travel distance for individual i and mode j, for all available modes for individual i during trip t of tour l.
  \item Total Travel Cost: the travel cost in dollars for individual i and mode j, for all available modes for individual i during trip t of tour l.
  \item Total travel time: the travel time in minutes for individual i and mode j, for all available modes for individual i during trip t tour l.
  \item Number of Autos: the number of automobiles owned by individual i's household.
  \item Number of Licensed Drivers: is the number of licensed drives in individual i's household.
  \item Number of Kids: the number of kids in individual i's household.
  \item Cross-bay trip: a binary variable indicating whether the trip t in tour l for individual i is a cross-bay trip.
\end{itemize}

Figure \ref{fig:IND_GRAPH} illustrates the DAG where all explanatory variables in each utility equation are marginally independent.
This DAG is equivalent to what an analyst assumes when they only update the variables directly impacted by a given policy, without considering the dependencies between the explanatory variables.
To illustrate the problem with this approach, consider the case where the ``true'' data generating process that reflects the dependencies between the covariates is as shown in Figures \ref{fig:DA_causal_2} through \ref{fig:SR3_causal_2}.
Under this generative model, intervening on one variable would also result in changes to other variables that are dependent on it.
The goal of our simulation exercise is to show that ignoring the true generative model of the data (Figures \ref{fig:DA_causal_2} through \ref{fig:SR3_causal_2}) can result in arbitrarily biased treatment effects, even if the analyst knows the true outcome model (Equations \ref{eq:DA} through \ref{eq:SR3+}).

\begin{figure}
\centering
\begin{subfigure}[b]{0.75\textwidth}
   \includegraphics{Independent_graph}
   \caption{Causal Graph with Independent Covariates}
   \label{fig:IND_GRAPH}
\end{subfigure}

\begin{subfigure}[b]{0.75\textwidth}
   \includegraphics{DA_interacting_graph}
   \caption{Causal Graph for the Drive Alone Utility Function}
   \label{fig:DA_causal_2}
\end{subfigure}

\begin{subfigure}[b]{0.75\textwidth}
   \includegraphics{SR2_interacting_graph}
   \caption{Causal Graph for the Shared Ride 2 Utility Function}
   \label{fig:SR2_causal_2}
\end{subfigure}

\begin{subfigure}[b]{0.75\textwidth}
   \includegraphics{SR3_interacting_graph}
   \caption{Causal Graph for the Shared Ride 3+ Utility Function}
   \label{fig:SR3_causal_2}
\end{subfigure}

\end{figure}

To achieve this goal, we follow these simulation steps, using the same outcome model defined in Equations \ref{eq:DA} to \ref{eq:SR3+}:
\begin{itemize}
   \item Simulate data from the DAGs shown in Figure \ref{fig:DA_causal_2} through Figure \ref{fig:SR3_causal_2}.
   \item In one scenario, only modify the travel distance variable in all graphs to emulate a company's decision to move its employees closer to campus. This is similar to assuming the DAG in Figure \ref{fig:IND_GRAPH} as the true data generating model.
   \item In the other scenario, modify the travel distance variable in all utility graphs, as well as all the variables that are dependent on it based on the DAGs in Figure \ref{fig:DA_causal_2} through Figure \ref{fig:SR3_causal_2}. This case is meant to reflect the ``correct'' approach needed to quantify the effects of the company's policy.
   \item Predict the probabilities of choosing car-centric modes before and after modifying the data to emulate the proposed intervention under each of the two scenarios above, and compute the differences in mode choice probabilities.
\end{itemize}

Readers interested in exploring the details of our simulation exercise can refer to our GitHub repository \citep{brathwaite_etal_2020}.

We then plot histograms of the computed differences between the average probability of an individual
in our sample choosing a car centric mode before and after implementing a policy or intervention
aimed at reducing travel distance.
Recall that these differences are plotted under the assumptions that the outcome model is the same in both scenarios, and the only difference between the two scenarios is the set of variables assumed to be affected by the proposed intervention.
This difference in the set of variables affected by the proposed intervention is a result of the two different causal graphs representing each scenario (Figures \ref{fig:IND_GRAPH} - \ref{fig:SR3_causal_2}).
Figure \ref{fig:histogram_probability} highlights the resulting bias between the estimated probability of an average individual choosing a car centric mode.
The histograms show that the distribution of inferred treatment effects based on the ``true'' causal graph includes both positive and negative treatment effects.
With low but non-trivial probability, using the wrong causal graph in one's analyses could result in not only wrong magnitudes of the treatment effect of interest but also the wrong sign.
This plotted discrepancy shows the importance of using causal graphs in the estimation of treatment effects.
Of course, the importance of a correct causal graph does not and should not take away from the importance of correctly specified outcome choice models.

\begin{figure}[h!]
   \centering
   \includegraphics[width=0.95\textwidth]{histogram_selection_on_obs}
   \caption{Histograms of the probability of choosing Car Centric Modes under Different Data Generating processes.}
   \label{fig:histogram_probability}
\end{figure}

We have shown in this simulation exercise how causal graphs can be helpful in avoiding biased causal effect inferences.
Specifically, two situations highlight this fact; having mediating or confounding variables relative to the variable being intervened on in our outcome model.
In the case of mediating variables relative to the node the treatment intervenes on, we need to be careful about constructing our treatment effect estimator.
In particular, we need to ensure that we appropriately use total effect estimators instead of natural direct effect estimators.
Enacting a certain treatment shows its effect through the mediating variable.
Therefore, when ``changing'' the value of the intervention node, we need to make sure the values and distributions of any downstream nodes reflect this change.
For more information on mediating mechanisms, please refer to work by \citet{pearl_2012_mediation} and references therein.
In the case of confounding variables, we need to consider the treatment assignment mechanism when building choice models.
As \citep{hahn_2020_bayesian} show, one way to account for the treatment assignment mechanism is to estimate the propensity score in our sample and adjust for that in our outcome model.

In contrast to the example shown in this section, the data generating process might not be easily distinguishable in the majority of situations, mainly due to the complexity of the real world.
Therefore, constructing a causal graph that represents the data generating process as closely as possible is not an easy task.
To prepare readers to create causal graphs, the next section will provide a brief overview of them and their history in choice modelling.
Then, Section \ref{sec:graph-construction} explores this topic and includes detailed guidance on how to build causal graphs representing the researchers' beliefs about the data generating process.
Section \ref{sec:graph-testing} follows up with guidance on how to test one's causal graphs against one's data.


\section{Conditions and sub-boxes}

In this section we expand on some topics mentioned briefly in Section 1.  As such, it would be useful to look again at
FIXME(Definitions 1.8, 1.12, and 1.22, and Remark 1.9),
where ${\cal P}$, ${\cal T}$ and their partners (under exponentiation)
${\cal W}$, ${\cal S}$ are introduced, and at
FIXME(Definition 1.18)
 where the notion of a killerword is introduced (the
definition is phrased in terms of ${\cal P}$, but the definition also makes sense for ${\cal W}$).  Note that working with
the region
${\cal P}$ is intuitively appealing, but working with the box ${\cal W}$ is vastly superior computationally (see Remark
1.21).

FIXME(Theorem 0.2),
hence
FIXME(Theorem 0.1), follows from
FIXME(Propositions 1.28, 2.8, 3.1, and 3.2).
  The computational aspects of the proofs of each of these propositions are similar.  We will focus on
FIXME(Proposition 1.28)
 here.
Its proof   amounts to decomposing
 ${\cal W}$ into a collection of sub-boxes of two types:
\begin{itemize}
\item[1)]   The 11 sub-boxes which comprise the
exceptional boxes $X_0, X_1, \ldots, X_6.$

\item[2)]   Sub-boxes each of which has an associated {\it condition} that will describe how to kill that entire sub-box,
perhaps with the help of a killerword.  To {\it kill} a sub-box means to show that
${\cal S} - \bigcup_{n = 0,\dots, 6} X_n$ has no point in the sub-box.
\end{itemize}
\noindent The set-up for efficiently describing these sub-boxes will be given in Construction FIXME(5.3).

We now list the conditions used to kill the nonexceptional sub-boxes.  There are two types of conditions: the trivial and
the interesting.  The trivial conditions kill sub-boxes in ${\cal W}$ since the sub-boxes in question miss
$\exp({\cal P}).$  The interesting conditions are where the real work is done, and they require a killerword in $f, w,
f^{-1}, w^{-1}$ to work their magic (see
FIXME(Remark 1.17)
).

To be consistent with the computer program {\it verify} we use the following notation: $L^{\prime} = z_0 + i z_3$,
$D^{\prime} = z_1 + i z_4,$ and $R^{\prime} = z_2 + i z_5.$  Here $(L^{\prime}, D^{\prime}, R^{\prime}) \in {\cal W}$ and
$L^{\prime} = \exp(L) = \exp(l+it),\   D^{\prime} = \exp(D) = \exp(d+ib),\
R^{\prime} = \exp(R) = \exp(r+ia).$
\vglue10pt
 {{\bf The trivial conditions
FIXME(5.1)
}}.
\vglue6pt
{\it Condition} `s' ({\it short}):  Tests that all points in the sub-box have $|z_0 + i z_3| < 1.10274.$  This ensures that
$$\exp(l) = |\exp(L)| = |L^{\prime}| = |z_0 + i z_3| < 1.10274 < \exp(0.0978),$$  and
FIXME(Definition 1.12)
 tells us that we are
outside of $\exp({\cal P}).$
\vglue6pt
{\it Condition} `l' ({\it long}): Tests that all points in the sub-box have $|z_0 + i z_3| > 3.63201.$  This ensures that
$$\exp(l) = |\exp(L)| = |L^{\prime}| = |z_0 + i z_3| > 3.63201 > \exp(1.289785)$$ and we are outside of $\exp({\cal P}).$
\vglue6pt
{\it Condition} `n' ({\it near}): Tests that all points in the sub-box have $|z_1 + i z_4| < 1.$  This ensures that
$$\exp(d) = |\exp(D)| = |D^{\prime}| = |z_1 + i z_4| <1= \exp(0)$$ and we are outside of $\exp({\cal P}).$
\vglue6pt
{\it Condition} `f' ({\it far}): Tests that all points in the sub-box have $|z_1 + i z_4| > 3.$  This ensures that
$$\exp(d) = |\exp(D)| = |D^{\prime}| = |z_1 + i z_4|  > 3= \exp(\ln 3)$$ and we are outside of $\exp({\cal P}).$
\eject

{\it Condition} `w' ({\it whirle  big}): Tests that all points in the sub-box have $|z_2 + i z_5|^2 > |z_0 + i z_3| $  This
ensures that $$\exp(r) = |\exp(R)| = |R^{\prime}| = |z_2 + i z_5|  > \sqrt{|z_0 + i z_3|} =  \sqrt {\exp(l)} = \exp(l/2)$$ and
we are outside of $\exp({\cal P}).$
\vglue6pt
{\it Condition} `W' ({\it whirle  small}): Tests that all points in the sub-box have $|z_2 + i z_5| < 1. $  This ensures that
$$\exp(r) = |\exp(R)| = |R^{\prime}| = |z_2 + i z_5|  < 1 = \exp(0)$$ and we are outside of $\exp({\cal P}).$

\vglue12pt
 {{\bf The interesting conditions
FIXME(5.2)
}}.
\vglue6pt
{\it Condition}  `L': This condition comes equipped with a killerword $k$ in $f, w, f^{-1}, w^{-1},$ and tests that all
points in the sub-box have $|\exp({\rm length}(k))| <  |L^{\prime}| = |\exp(L)|,$ where ${\rm length}(k)$ means the length
of the isometry determined by $k.$   This, of course, contradicts the fact that $L$ is the length of the shortest geodesic.

It is easy to carry out the test $|\exp({\rm length}(k))| <  |L^{\prime}|$ because
Lemma FIXME(1.25a)) can be used.
Note that in {\it verify} the function which computes $\exp({\rm length})$ is called {\it length}.

Of course, Condition `L' also checks that the isometry corresponding to the word $k$ is not the identity.
\vglue6pt
{\it Condition}   `O':   This condition comes equipped with a killerword
$k$ in $f, w, f^{-1}, w^{-1},$ and tests that all points in the sub-box have
$$|\exp({\rm distance}(k(B_{(0,\infty)}),
B_{(0,\infty)}))| <  |D^{\prime}| = |\exp(D)|.$$    (Recall that
$B_{(0;\infty)}$ denotes the oriented geodesic $\{(0,0,z): 0< z < \infty \}$,
with negative endpoint $(0,0,0).$)
This, of course, contradicts the ``nearest" condition.

It is easy to carry out the test $|\exp({\rm distance}(k(B_{(0,\infty)}), B_{(0,\infty)}))| <  |D^{\prime}|$ because
Lemma FIXME(1.25b)) can be used. Note that in {\it verify} the function which computes the quantity
$\exp({\rm distance}(k(B_{(0,\infty)}), B_{(0,\infty)}))$ is called {\it orthodist}.

Also, Condition `O' checks that the isometry corresponding to the word $k$ does not take the axis of $f$ to itself.
\vglue6pt
{\it Condition}   `2':  This is just the `L' condition without the ``not-the-identity" check, but with the additional
proviso that the killerword $k$ is of the form $f^p w^q.$ This ensures that $k$ is not the identity, because for $k$ to be
the identity $f$ and $w$ would have to have the same axis, which contradicts the fact that $d$ can be taken to be
greater than or equal to $l/4.$
\eject

{\it Condition}  `conjugate':  There is one other condition that is used to eliminate points in ${\cal W}.$  Following
FIXME(Definition 1.12)
 (and
FIXME(Lemma 1.13)
) we eliminate all boxes with $0 < t \le \pi.$  Of course, after exponentiating $L = l+it,$
this corresponds to eliminating all boxes with $z_3 > 0.$ Specifically, we toss all sub-boxes of ${\cal W}$ whose fourth
entry is a 1. This condition does not appear in {\it verify} because it is applied ``outside" of these
programs, as described in Construction
FIXME(5.3).
\vglue6pt
 {\it Construction} FIXME(5.3):
We now give the method for describing the roughly 930 million sub-boxes that the initial box ${\cal W}$ is subdivided into.

All sub-boxes are obtained by subdivision of a previous sub-box along a real
hyper-plane midway between parallel faces of the sub-box before
subdivision.  Of course, these midway planes are of the form $x_i = $ a
constant.   We use $0$'s and $1$'s to describe which half of a subdivided
sub-box to take ($0$ corresponds to lesser $x_i$ values).  For example, 0
describes
the sub-box $${\cal W} \cap \{(x_0,x_1,x_2,x_3,x_4,x_5): x_0 \le 0 \},$$
$010$ describes the sub-box
$$W \cap \{(x_0,x_1,x_2,x_3, x_4,x_5) : x_0 \le 0,\ x_1 \ge 0,\ x_2 \le 0 \},$$
and so on.

In this way, we get a one-to-one correspondence
between strings and sub-boxes.
If $s$ is a string of $0$'s and $1$'s, then let $Z(s)$ denote
the sub-box corresponding to $s$.
The range of values for the $i^{\rm th}$ coordinate in the sub-box $Z(s)$ is related
to the binary fraction $0.s_{i}s_{i+6}\ldots s_{i+6k}$.
The two sub-boxes gotten from
subdividing $Z(s)$ are $Z(s0)$ and $Z(s1)$.

The directions of subdivision cycle among the various coordinate axes:
the $n^{\rm th}$ subdivision is across the $(n \bmod 6)^{\rm th}$ axis.
The dimensions of the top-level box ${\cal W}$ were chosen so that subdivision
is always done across the longest dimension of the box,
and so that all of the sub-boxes are similar.
The dimensions of ${\cal W}$ have the beneficial effect of making the sub-boxes as ``round" as possible, hence making the Taylor approximation calculations efficient and fast.
This explains the factor of $2^{(5-i)/6}$ in Definition
FIXME(1.22)
.

To kill a sub-box $Z(s)$, the checker program has two (recursive) options:
use a condition and, if necessary, an associated killerword  to kill $Z(s)$ directly, or first kill $Z(s0)$ and then kill $Z(s1)$.
At this point, it may seem as if the second option is not necessary,
because surely a condition which kills two halves also kills the whole.
The answer to this has been hinted at in
FIXME(Remarks 1.17 and 1.35)
 where
it is noted that our evaluation of a function arising
from a killerword  is via first-order Taylor
approximation, complete with remainder/error term.
(Note that the remainder/error term incorporates bounds on
both the theoretical error arising from using a first-order Taylor
approximation to approximate a function, and the accumulated
round-off error; see
FIXME(Sections 6, 7,  and 8)
.)
Even if a killerword could theoretically kill off a sub-box,
it is quite possible that our
first-order Taylor approximation approach would not be
able to prove this
because its remainder/error term is too large.
However, if we subdivide the sub-box, then the first-order
Taylor approximations on the two halves should be more accurate.
Thus,
we want to have the recursive subdivision option at our
disposal.
Note that because the checker program does in fact do such
recursive subdivisions, the actual number of sub-boxes in the
ultimate subdivision is larger (perhaps substantially larger)
than the 930 million sub-boxes of the initial data tree.

It is also possible that the checker program will employ
neither of the two options described in the previous
paragraph, and will instead employ
a third option: do not kill $Z(s)$, and instead
mark $s$ as omitted.
Any omitted sub-boxes are checked with another instance
of the checker program,
unless the sub-box is one of the 11 exceptional
sub-boxes (which produce the seven exceptional boxes after joining abutters).
Note that according to the definition of ``kill" given at the beginning of this section, the exceptional boxes are
automatically killed.

Thus, a typical output from {\it verify} would be
\begin{eqnarray*}
&&{\rm verified} 000000111101111111 \\
&&\qquad\qquad\quad-
\{ 0000001111011111110 \ \ 000000111101111111110 \} .
\end{eqnarray*}
which means that the sub-box $Z(000000111101111111)$ was killed except for its sub-boxes
$Z(0000001111011111110)$  and $Z(000000111101111111110).$  The output
\begin{eqnarray*}
&&{\rm verified} 0000001111011111110 - \{\ \}.\\
\noalign{\noindent and}
&&{\rm verified} 000000111101111111110 - \{\ \}.
\end{eqnarray*} shows that these sub-boxes were subsequently killed as well, and thus the entire sub-box
$Z(000000111101111111)$ has been killed.

Instead of immediately working on killing the top-level box, we subdivide in the six co-ordinate directions to get the 64 sub-boxes $$Z(000000),Z(000001),Z(000010),Z(000011),\ldots, Z(111111).$$
\noindent We then throw out
the ones with fourth co-ordinate equal to 1 (see condition `conjugate'), leaving  the 32 sub-boxes
$$Z(000000),Z(000001),Z(000010),Z(000011),\ldots, Z(111011).$$  \noindent We then use {\it verify} to kill these.
\eject

The choices in {\it verify} are made for it by a sequence
of integers given as input.  The sequence of integers containing the directions for killing $Z(000000)$ is contained in the file data/000000 (actually, data/000000.d).
In such a sequence,
$0$ tells {\it verify} to subdivide the present box (by $x_i = c$),  to position itself on the ``left-hand" box ($x_i \le c$)
created by that subdivision, and to read in the next integer in the sequence. A  positive integer $n$ tells {\it verify} to
kill directly the sub-box it is positioned at, using the condition (and killerword, if necessary) on line $n$ in the
``conditionlist" file,  and then to position itself at the ``next" natural sub-box.  Now,
$-1$ tells {\it verify} to omit the sub-box, and mark it as skipped
(the sequence of integers used in killing the skipped box $Z(s)$ is contained in a file data/s).

The checker program {\it verify}, its inputs, and the list of conditions
are available from the {\it Annals of Mathematics} web site.
\vglue7pt
FIXME({\it Example} 5.4.)
To illustrate the checking process in action, this is a (non-representative)
example, which shows how the sub-box $Z(s)$ (minus a hole) is killed,
where
\begin{eqnarray*}
&&s=0010001100011101110011110001011111111011111\\
&&\hskip2in 00111001111000001111011110111.
\end{eqnarray*}

The input associated with this sub-box is
$$(0, 0, 0, 1929, 12304, 0, 0, 7, 0, 1965, 0, 1929, 1929, 1996, -1),$$
which causes the program to kill $Z(s)$ in the following fashion:
{\noindent\obeylines
\def\>{\hskip 0.05in}
 kill $Z(s)$:
\>kill $Z(s0)$:
\>\>kill $Z(s00)$:
\>\>\>kill $Z(s000)$ with condition 1929 = ``L(FwFWFWfWFWFwFwfww)''
\>\>\>kill $Z(s001)$ with condition 12304 = ``L(FwfWFFWFwFwfwfWfwfw)''
\>\>kill $Z(s01)$:
\>\>\>kill $Z(s010)$:
\>\>\>\>kill $Z(s0100)$ with condition 7 = ``L(w)''
\>\>\>\>kill $Z(s0101)$:
\>\>\>\>\>kill $Z(s01010)$ with condition 1965 = ``L(fwFwFWFFWFwFwfwww)''
\>\>\>\>\>kill $Z(s01011)$:
\>\>\>\>\>\>kill $Z(s010110)$ with condition 1929
\>\>\>\>\>\>kill $Z(s010111)$ with condition 1929
\>\>\>kill $Z(s011)$ with condition 1996 = ``L(FwFwFWFWfWFWFwFwfww)''
\>omit $Z(s1)$
\noindent as shown in
FIXME(Figure 5.1).
}

 \centerline{\BoxedEPSF{fig5.1.eps scaled 1250}}

 \begin{quote}{
FIXME(Figure 5.1)
.  Six levels of subdivision, in two projections,
 with all the trimmings.}
 \end{quote}

$Z(s1)$ is ignored, so the checker would indicate this omission
in its report.  In fact, $Z(s1)$ is one of the 11 exceptional sub-boxes (seven boxes after joining abutters), specifically
$X_{5a},$ hence killed automatically.

The use of condition ``L(w)'' so deep in the tree is unusual.
In this case, it is because the manifold in the exceptional sub-box
has ${\rm length}(f) = {\rm length}(w)$, so that the program will frequently
come to places where it can bound ${\rm length}(f) > {\rm length}(w)$ nearby.

The sequence $\ 0, 1929, 1929\ $ in the
input for $Z(s)$ tells the checker to
subdivide $Z(s01011)$, and then use the condition 1929
on both halves to kill   them separately,
thereby killing $Z(s01011)$.
As mentioned above,
the reason for carrying out this  subdivision
is that the remainder/error bound in the calculation
 for $Z(s01011)$ using condition 1929 was not good enough
to prove that the sub-box is killed directly.

In the input for $Z(s)$ we could have
replaced
 $\ 0, 1929, 1929\ $ with 1929 alone
and then the checker
program itself would be smart enough to carry out
the subdivision after 1929 failed to kill the sub-box
$Z(s01011).$
This recursive subdivision tool is quite useful when dealing with the
remainder/error term---if a killerword barely misses killing off a sub-box, then recursively subdivide the sub-box and
use the same killerword on the pieces until it succeeds.   We note that the theoretical error arising from using a
first-order Taylor approximation is likely to be significantly improved by subdivision, whereas the
round-off error is relatively unaffected because the killerword
used is unchanged (hence the number of mathematical operations
performed is unchanged).

The binary numbers used by the computer require too much space to print.
In the example calculation which follows, we instead use decimal
representations (although we print fewer digits than could
be gotten from the 53 binary digits used for the actual calculations).

The sub-box $Z(s01011)$ is the region where
$$\left(\begin{array}{c}
-1.381589027741\ldots  \le  {\rm Re}(L')  \le  -1.379848991182\ldots\cr
-1.378124546093\ldots  \le  {\rm Re}(D')  \le  -1.376574349753\ldots\cr
0.999893182771\ldots  \le {\rm Re}(R')  \le  1.001274250703\ldots\cr
-2.535837191243\ldots  \le  {\rm Im}(L')  \le  -2.534606799593\ldots\cr
\phantom{-}2.535404997792\ldots  \le  {\rm Im}(D')  \le  -2.534308843448\ldots\cr
-0.001953125000\ldots  \le  {\rm Im}(R')  \le  0.000000000000\ldots\phantom{-}\cr
\end{array}\right)$$

At this point, we would like to compute $$f,\ w,\ g = f^{-1}wf^{-1}w^{-1}f^{-1}w^{-1}fw^{-1}f^{-1}w^{-1}f^{-1}wf^{-1}wfww, \ {\rm length}(g),$$
\noindent
and so on.
However, these items take on values over an entire sub-box and thus are computed via AffApproxes (first-order Taylor
approximations with remainder/error bounds), which are not formally defined until the next section.  We
complete
FIXME(Example 5.4)
 at the end of
FIXME(Section 6).

\vglue9pt
FIXME({\it Remark} 5.5)
.  For those planning on looking at the program {\it verify}
we now tie in the above description of its workings to a portion of the actual code in the program.  We note that the CWeb version of {\it verify} is extensively documented, and is organized so that the most important details are presented first.

If the executable version of {\it verify} is called {\it verify} and we are in the correct place with respect to the location of the data, then a typical UNIX command line would be
$$
\hbox{\tt zcat data/000000.gz | verify 000000 > output000000}
$$
This would run {\it verify} at the node 000000, and, when needed, would pipe in the unzipped data from \hbox{\tt
data/000000.gz}.  This unzipped data contains the tree decomposition of the parameter space at the sub-box 000000.
The output from {\it verify} would be redirected to the file \hbox{\tt output000000}.

In {\it verify}, \hbox{\tt main} would check for syntax errors in the command line, and if there were no such errors, would read the location 000000 into the character array \hbox{\tt where}  and compute that the \hbox{\tt depth} of \hbox{\tt where} was 6, which means that 000000 contains
six subdivisions.   It would then immediately print
$$
\hbox{\tt verified 000000 - } \{$$
into the file \hbox{\tt output000000}, and then call the function {\it verify}, as follows:
$$
\hbox{\tt verify(where, depth, 0);}
$$

The function \hbox{\tt verify(where, depth, autocode)} is now invoked; this time with \hbox{\tt autocode} equal to 0.  {\it Verify} would first check that \hbox{\tt depth} was not too deep.  Next, {\it verify} checks if \hbox{\tt autocode} is equal to 0, which it is, so it reads in the next (in this case, the first) integer from the unzipped file \hbox{\tt data/000000.gz}, and sets \hbox{\tt code} equal to this integer.  Now, {\it verify} recursively calls itself on the left child (0000000) of the \hbox{\tt where} box and the right child (0000001) of the
 \hbox{\tt where} box:
\begin{eqnarray*}
\noalign{\vskip-12pt}
&&
\hbox{\tt where[depth] = \hbox{`}0\hbox{'};}
\\
&& \hbox{\tt verify(where, depth + 1, code);}
\\
&&
\hbox{\tt where[depth] = \hbox{`}1\hbox{'};}
\\
&&\hbox{\tt verify(where, depth + 1, code);}
\end{eqnarray*}
In general, \hbox{\tt verify(where, depth, autocode)} does the following.  It checks to see that \hbox{\tt depth} is not too deep.  Then if \hbox{\tt autocode} is equal to 0, it recursively calls itself on its left and right children.  If \hbox{\tt autocode} is not equal to 0, then \hbox{\tt code} is set equal to \hbox{\tt autocode}, and
three possibilities can occur.  Either,
\vglue9pt
1)  \hbox{\tt code} is less than zero, in which case we are at
a sub-box to be skipped,
 and {\it verify} prints out its location  (\hbox{\tt where}) in \hbox{\tt output000000} and recursively moves on to the next node in the tree, or
\vglue9pt
2) \hbox{\tt code} is greater than zero and it invokes a condition/killerword from the file {\it conditionlist} which kills the entire sub-box \hbox{\tt where}
in which case \hbox{\tt verify} simply recursively moves on to the next node in the tree, or
\vglue9pt
3) \hbox{\tt code} is greater than zero and it invokes a condition/killerword from the file {\it conditionlist} which does not kill the entire sub-box \hbox{\tt where}, in which case
\hbox{\tt verify} subdivides the sub-box \hbox{\tt where} and recursively calls itself on the left child and the right child, using the same \hbox{\tt code}:
\begin{eqnarray*}
&&\hbox{\tt where[depth] = \hbox{`}0\hbox{'};}
\\ &&\hbox{\tt verify(where, depth + 1, code);}
\\ &&\hbox{\tt where[depth] =\hbox{`}1\hbox{'};}
\\ &&\hbox{\tt verify(where, depth + 1, code);}
\end{eqnarray*}
In this way {\it verify} tests the entire starting box, in this case the sub-box 000000, and if successful at killing
it minus the omissions which it prints out, it finishes \hbox{\tt main} by printing out a right bracket into \hbox{\tt output000000}.