% !TeX root = ../smc-report.tex
% !TeX encoding = UTF-8
% !TeX spellcheck = en_GB

\section*{LINFA: Smart drugs restocking}

  \subsection*{Smart drugs restocking}

    The pharmaceutical logistic chain is a very complex system, as it includes several actors and critical points (such as elevated drug costs, need for transport at monitored temperature, chance of expiration of goods, stock management, irregularity of demands, several possible logistic approaches, etc), which render the problem hard to optimize without the proper tools for decision support. In this context, possible unavailability could lead to critical situations, sometime even catastrophic, as it wouldn't be possible any more to guarantee the correct execution of one or more healthcare protocol, thus affecting the patients' health. Furthermore, orders are typically carried out on a daily basis and in a manual fashion, without the support of any decision support system. All these reasons makes restocking schedule hard, thus leading to unproductive stocks and higher stocking costs.

    In this scenario fits the project \textit{LINFA} (i.e. \textit{Logistica INtelligent del FArmaco}, or \textit{Smart Drug Logistic}, from Italian), that aims to develop an IT system for support to processes of drugs logistic management, in the context of healthcare or local companies. LINFA aims to increase efficiency, effectiveness and predictability of the process of drugs and medical devices restocking, within healthcare structures, through methods of predictive analysis and optimization, advanced logistic techniques and tracking features through the use of RFID technologies or the integration of healthcare and administrative information stream.

    For sake of simplicity, the following assumptions will be made in order to build a model:

    \begin{itemize}
      \item a single ward is present;
      \item the ward has a fixed number of beds (40) and a fixed maximum storage capacity (40);
      \item patients are indistinguishable;
      \item there is only one type of drug;
      \item patients can arrive through emergencies (i.e. according to a random variable) or scheduled examinations (i.e. according to a fixed constant);
      \item each day, patients can leave the ward with a certain probability;
      \item each day, patients can consume up to 3 units of drug;
      \item for each missing drug, an urgent order is issued, which arrives immediately;
      \item restock orders are issued at the end of each day and arrive immediately;
      \item the drug can be restocked in quantities of 0 (i.e. no order), 10, 20, 30 or 40.
    \end{itemize}

    The model in Code \ref{lst:ward} models a generic ward with the above specifications and three days of lookahead. The model is intended to characterise the evolution of the ward for the three subsequent days in order to support decision to the restock order of the current day, so it starts at the end of the current day (i.e. after patients arrival/discharge and drug consumption, but before the restock order). The arrival of patients and the consumption of drugs have been modelled through binomial distributions, as well as drug restock order choices, with different success probabilities. Sojourn time inside the ward for each patient is instead modelled through a geometric distribution.

    \begin{center}
      \lstinputlisting[language=prism, caption={PRISM code of the probabilistic model of a hospital ward.}, label={lst:ward}]{code/ward_dtmc.sm}
    \end{center}

    \question{Look at the model in Code \ref{lst:ward} and describe how it works.\\}
    \answer{
      The first thing that can be notice about the model is that it is defined as a DTMC. This is due the fact that the model is fully probabilistic (i.e. there is no non-determinism) and that the time is divided into discrete steps.

      In the first part of the code, before the beginning of the main module (lines 3-19), constants and probabilities are defined. Distributions for patients arrival through emergency, drug consumption and drug restock are defined as binomial distribution with different success probabilities (lines 8-11). Upper bounds of such distributions are defined as $4$ for emergency arrivals (\prism{maxER}), $3$ for drug consumption (\prism{maxConsume}) and $4$ for drug reorder choices. Arrivals of scheduled patients for the three subsequent days of lookahead are defined as fixed variables (lines 17-19). Also, cost weights are defined for the order cost of a single drug unit, the cost for urgent drug reorders and the stocking cost for keeping each drug unit in stock in the right conditions each day (lines 13-15).

      The main module \prism{hospitalWard} firstly defines several internal variables (lines 37-42), such as the number of hospitalised patients \prism{n}, the number of current drugs in stock \prism{stockDrugs}, the current day \prism{day}, or support variables, such as \prism{s}, \prism{tmp} or \prism{tmp2}. In particular \prism{s} is used to describe the internal state for each day of the ward.

      States from \prism{s=0} to \prism{s=3} (lines 31-46) are used to model the drug order for the current day and for the first and second following days. The module starts at the end of the current day (i.e. after patients arrival/discharge and drug consumption, but before the restock order) in order to include as much real information as possible from the real ward when deciding how to reorder. Lines 34-37 are used to chose probabilistically which kind of order to issue, according to the probability \prism{probOrder} defined. Lines 39-43 are then used to execute the corresponding order, effectively increasing the stocked drugs. Also, each order has a label associated, that will be used in the cost reward computation. After the order choice has been made, the \prism{day} variable is incremented and the \prism{tmp} support variable is set to the number of patients currently hospitalised, \prism{n}.

      States from \prism{s=10} to \prism{s=12} are used to model the arrival and discharge of patients in the ward. State \prism{s=10} (lines 49-52) of the ward is used to cycle through all the currently hospitalised patients to decide which of them will leave the ward, with probability \prism{probExit}. In this scenario the \prism{tmp} variable is used to cycle through all the patients. State \prism{s=11} (lines 55-57) is used to model the arrival of scheduled patients, the exact number of which is indicated by three constants. State \prism{s=12} (lines 60-63) instead is used to model the arrival of new patients through emergencies, modelled through a binomial distribution with success probability \prism{probER} defined.

      State \prism{s=20} is used to model drug consumption by the currently hospitalised patients. A binomial distribution with success probability \prism{probConsume} is used and a cycle is implemented through the use of the \prism{tmp} (lines 66-69). On top of that, for each patient, if after the drug consumption some drugs result to be missing (i.e. some drug was needed but there wasn't enough in stock), then the special action with label \prism{missingDrugs} is executed (line 71), which is needed to compute the correct cost reward. When drugs have been administered to all the patients (line 73) the module will then either start again (line 76) or stop definitely (line 77) in case the three following days have been completely modelled.

      Lastly, the only reward defined, \prism{totalCost}, is used to cumulate the total cost coming from different sources (lines 85-95). In particular, every time the consumption of drug is finished for a certain day, the remaining drugs in stock produce a cost determined by the weight \prism{costStorage} (line 86). Each time one or more drugs result missing during administration, and therefore an urgent order has to be issued, each missing drug produce a cost defined by the weight \prism{costUrgentOrder} (line 88). Regular orders also produce a cost, in particular a cost of $1$ for each drug ordered (lines 90-94).
    }

    \question{
      Add the following property, representing the expected \prism{totalCost} at the end of the three subsequent days:

      \lstinputlisting[language=prism, numbers=none]{code/totalCost.pctl}

      Then, create a new experiment to evaluate the best reorder strategy for wards that have different probabilities of drug consumption. For this experiment, set the probability constants \prism{probExit} and \prism{probER} to $0.5$ and $0.3$ respectively.
    }
    \answer{
      The results of the experiment, that plots the expected \prism{totalCost} after the three days for different consumption and reorder probabilities, are shown in Figure \ref{fig:totalCost_probOrder_probConsume_plot}.

    	\begin{figure}[h!]
    		\begin{center}
    			\includegraphics[scale=0.45]{totalCost_probOrder_probConsume_plot.eps}
    		\end{center}
    		\caption{Experiment plot of the expected total cost after the three subsequent days as \prism{probOrder} varies for different values of \prism{probConsume}.}
    		\label{fig:totalCost_probOrder_probConsume_plot}
    	\end{figure}

      As expected, lower values of \prism{probOrder} result in a higher expected \prism{totalCost} after three days for higher probabilities of consumption. This is clearly due the fact that higher consumption probabilities means a higher demand of drugs, which otherwise will deplete soon, therefore policies that tend to restock big quantities of drugs are advised in these situations. These curves, however, are not monotone, as they seem to cross around $0.6$, which seems to be also the minimum for most curves. After this point, which seems to be the best policy to be taken on average, the situation changes, meaning that higher probabilities of reordering are favoured in scenarios with higher probabilities of consumption. It is also worth noticing that the curve relative to \prism{probConsume}$=0$ is the only strictly increasing, due to the fact that if no drugs are ever consumed (i.e. never required), then reordering drugs is always just a cost and never an investment. So, as expected, low values of \prism{probConsume} requires also low values of \prism{probOrder} and viceversa. A good strategy, on average, would be to reorder with probability between $0.5$ and $0.6$.
    }

    \question{Perform now an experiment similar to the previous one, this time studying the best reorder strategy for wards that have different average sojourn times (i.e. probability of patient discharge). Set instead the value of \prism{probConsume} to $0.2$}
    \answer{
      The results of the experiment are shown in Figure \ref{fig:totalCost_probOrder_probExit_probConsume-0-2_plot}.

    	\begin{figure}[h!]
    		\begin{center}
    			\includegraphics[scale=0.45]{totalCost_probOrder_probExit_probConsume-0-2_plot.eps}
    		\end{center}
    		\caption{Experiment plot of the expected total cost after the three subsequent days as \prism{probOrder} varies for different values of \prism{probExit}, with \prism{probConsume}$=0.2$.}
    		\label{fig:totalCost_probOrder_probExit_probConsume-0-2_plot}
    	\end{figure}

      The results show that, as expected, lower values of \prism{probExit} (i.e. patients stay longer, on average, in the ward) require higher values of \prism{probOrder}, while higher values of the discharge probability require a smaller reorder probability. As seen in the previous experiment, the curves cross, this time between $0.3$ and $0.35$. In this scenario, if the probability of patients discharge is unknown, the safest strategy would be with a value of \prism{probOrder} between $0.25$ and $0.3$.
    }

    \question{Repeat the previous experiment, this time with \prism{probConsume}$=0.8$. Which conclusions can you draw comparing the results of the two experiments?}
    \answer{
      The results of the experiment are shown in Figure \ref{fig:totalCost_probOrder_probExit_probConsume-0-8_plot}.

    	\begin{figure}[h!]
    		\begin{center}
    			\includegraphics[scale=0.45]{totalCost_probOrder_probExit_probConsume-0-8_plot.eps}
    		\end{center}
    		\caption{Experiment plot of the expected total cost after the three subsequent days as \prism{probOrder} varies for different values of \prism{probExit}, with \prism{probConsume}$=0.8$.}
    		\label{fig:totalCost_probOrder_probExit_probConsume-0-8_plot}
    	\end{figure}

      The results of this last experiment are very similar to the results of the previous one, meaning that lower values of \prism{probExit} favour higher values of \prism{probOrder} and viceversa. This time, instead, the inversion point of the curves results shifted to the right, around $0.6$. This shift is due the fact that when drugs are consumed more frequently, ordering more drugs keeps being the optimal choice even if patients leave the hospital quickly, because even in the short time they spend in the ward, they are expected to consume high quantities of drugs. Also, the range of the expected \prism{totalCost} varies greatly between the two experiments: in the previous one the maximum was $636$, while for this was is $2543$. This can simply be explained by the fact that if more drugs are consumed on average, the simple cost of having to restock more drug units makes the total cost skyrocket.
    }

  \dashedrule

  \subsection*{Adding non-determinism}
    One of the most useful features of PRISM is the optimal strategy extraction of an MDP model. \textit{MDPs} (\textit{Markov Decision Process}) are probabilistic models that also include non-deterministic choices, in order to model transitions between states with no probabilistic information attached to them. A strategy, for an MDP, is a set of specific actions for all the non-deterministic choices that resolve the non-determinism of the model, turning it into a fully probabilistic model. Among all the possible strategies, the optimal one is the strategy that minimises/maximises a certain reward function.

    \question{Modify the model in Code \ref{lst:ward} in order to make the drug reorder choice non-deterministic.}
    \answer{
      Changing the reorder choice in Code \ref{lst:ward} from a probabilistic choice to a non-deterministic one is quite straightforward. The first thing do is, clearly, to change the keyword \prism{dtmc} in line 1 to \prism{mdp}, so that the model is correctly built as an MDP. Then, it's simply necessary to replace the lines 31-46 in Code \ref{lst:ward} with the following code:

      \lstinputlisting[language=prism, numbers=none]{code/ward_mdp_update.sm}

      The above code allows all the reorder choices to be enabled at the same time with no probability information attached to them, thus making the reorder choice non-deterministic. Also, the constant \prism{probOrder} (line 11) and the variable \prism{drugsToOrder} (line 28) can be removed from the code, as they are not used any more.
    }

    \question{
      Save the new MDP model as \bash{ward\_mdp.sm}, setting \prism{probExit}, \prism{probER} and \prism{probConsume} to be $0.5$, $0.3$ and $0.6$ respectively. Then execute the following command to \href{http://www.prismmodelchecker.org/manual/RunningPRISM/Adversaries}{extract the optimal adversary}, states and labels:

      \lstinputlisting[numbers=none]{code/extractadv.bat}

      Following the PRISM manual, determine which is the optimal strategy for the current day reorder with the initial setting of \prism{stockDrugs}$=10$ and \prism{n}$=10$.
    }
    \answer{
      According to the manual, once the suggested command have been executed, the file \bash{adv.lab} indicates which is the ID of the initial state, in this case $370$. By exploring, then, the \bash{adv.tra} file, the following line can be found:

      \lstinputlisting[numbers=none]{code/adv_tra_example.tra}

      The above line indicates that, when in the state with ID $370$ (i.e. the initial state, which in this case represents the scenario with 10 patients and 10 stocked drugs), the best strategy to follow, in order to minimise on average the total cost after three days, is to order 10 units, as indicated by the label of the action \prism{order10}. The above line indicates also that, by taking this action, the model will move to the state with ID $2191$, which, according to the file \bash{adv.sta}, corresponds to the scenario with 10 patients, 20 stocked drugs and variable \prism{s}$=1$, as expected.
    }

    \question{Perform now an experiment, either manually or through a custom script, varying the initial state for both \prism{n} and \prism{stockDrugs}. Show and comment the results.}
    \answer{
      The results of the experiment are shown in Figure \ref{fig:sd_n_plot}.

    	\begin{figure}[h!]
    		\begin{center}
    			\includegraphics[scale=1]{sd_n_plot.pdf}
    		\end{center}
    		\caption{Experiment plot of the optimal reorder strategy as \prism{stockDrugs} varies for different values of \prism{n}.}
    		\label{fig:sd_n_plot}
    	\end{figure}

    	From Figure \ref{fig:sd_n_plot} it can be seen that, as the value of \prism{stockDrugs} increases, the suggested optimal reorder amount decreases, as with higher amounts of drugs already in stock, the need for even more drugs gets weaker and weaker. When the number of hospitalised patients \prism{n}, instead, is increased, the suggested optimal amount results also increased, because, as expected, with more patients in the hospital more drug consumptions are expected, thus requiring a fuller stock. In Figure \ref{fig:sd_n_plot} the curves \prism{n}$=30$ and \prism{n}$=40$ overlaps.

    	The experiment have been conducted through a simple custom script, used to speed up the experimentation\footnote{The script source code can be found at the following link:\\ \url{https://github.com/oddlord/uni/tree/master/phd/courses/smc/experiments}.}.
    }

    \question{Perform a similar experiment, this time varying \prism{stockDrugs} and \prism{costStorage}, while keeping the initial value of \prism{n} fixed to $20$.}
    \answer{
      The results of the experiment are shown in Figure \ref{fig:sd_costStorage_0-40_plot}.

    	\begin{figure}[h!]
    		\begin{center}
    			\includegraphics[scale=1]{sd_costStorage_0-40_plot.pdf}
    		\end{center}
    		\caption{Experiment plot of the optimal reorder strategy as \prism{stockDrugs} varies for different values of \prism{costStorage}.}
    		\label{fig:sd_costStorage_0-40_plot}
    	\end{figure}

      The results in Figure \ref{fig:sd_costStorage_0-40_plot} show that, similarly to the results of the previous experiment, the suggested optimal reorder amount decreases as the amount of drugs in stock increases. Also, it can be observed how, with the increase of the storage cost for each drug for each day, the suggested amount decreases: this is due the fact that, when the cost is low (or even zero) the system tends to suggest to completely refill the storage, in order to cope even with extreme emergency situation; on the other hand, if the storage cost weights more, PRISM is forced to find a more suitable trade-off between the storage cost and the actual need for drugs. An interesting detail is also that the gap between two subsequent curves seems to get smaller and smaller as \prism{costStorage} increases: this remarks the fact that, as \prism{costStorage} increases, the suggested optimal reorder amount tends to a limit, representing the best trade-off between the need to store as less drugs as possible (due to the high storage cost) and the need to also store as much drugs as possible (in order to avoid urgent orders).
    }


\section{Image classification by MgNet}\label{sec:CNN-Results}
In the following Table \ref{tab:MgNet-CIFAR10-Results}, Table \ref{tab:MgNet-CIFAR100-Results}, Table \ref{tab:MgNet-ImageNet-Results}, we summarize the test accuracy of MgNet on popular datasets CIFAR10, CIFAR100, ImageNet, respectively. For all MgNet models, we use take $c_f=c_u$.

%\begin{table}[!htbp]
%	\begin{center}
%		\begin{tabular}{|c|c|c|c|}
%			\hline
%			Dataset     & Model                 & Test accuracy   & Number of parameters \tabularnewline
%			\hline
%			CIFAR10      &MgNet      &  96.00      &   8.2M      \tabularnewline
%			\hline
%			CIFAR100     &MgNet      &  79.94      &   8.3M      \tabularnewline
%			\hline
%			ImageNet     &MgNet      &  71.01      &   8.5M    \tabularnewline
%			\hline
%		\end{tabular}
%	\end{center}
%	\label{tab:MgNet-Results}
%	\caption{Test accuracy of MgNet on popular datasets.}
%\end{table}
\newpage
\begin{table}[!htbp]
	\begin{center}
		\begin{tabular}{|c|c|c|c|}
			\hline
			Dataset     & Model, [$\nu_1,\nu_2,...$], $c_u$                  &  Test accuracy   &  Number of parameters \tabularnewline
			\hline
			CIFAR10     &MgNet, [2,2,2,2], 256      &  96.00      &   8.2M      \tabularnewline
			\hline
			CIFAR10     &MgNet, [3,4,6,3], 256      &  95.98      &   8.3M      \tabularnewline
			\hline
			CIFAR10     &MgNet, [3,4,6,3], 512      &  96.48      &   33.1M     \tabularnewline
			\hline
		\end{tabular}
	\end{center}
	\label{tab:MgNet-CIFAR10-Results}
	\caption{Test accuracy of MgNet on CIFAR10.}
\end{table}

\begin{table}[!htbp]
	\begin{center}
		\begin{tabular}{|c|c|c|c|}
			\hline
			Dataset     & Model, [$\nu_1,\nu_2,...$], $c_u$                  &  Test accuracy   &  Number of parameters \tabularnewline
			\hline
			CIFAR100     &MgNet, [2,2,2,2], 256            &  79.94      &   8.3M      \tabularnewline
			\hline
			CIFAR100     &MgNet, [2,2,2,2], 512            &  81.35      &   33.1M      \tabularnewline
			\hline
			CIFAR100     &MgNet, [2,2,2,2], 1024           &  82.46      &   132.2M      \tabularnewline
			\hline
		\end{tabular}
	\end{center}
	\label{tab:MgNet-CIFAR100-Results}
	\caption{Test accuracy of MgNet on CIFAR100.}
\end{table}

\begin{table}[!htbp]
	\begin{center}
		\begin{tabular}{|c|c|c|c|}
			\hline
			Dataset     & Model, [$\nu_1,\nu_2,...$], $c_u$                  &  Test accuracy   &  Number of parameters \tabularnewline
			\hline
			ImageNet     &MgNet, [3,4,6,3], [64,128,256,512]          &  73.78      &   9.9M      \tabularnewline
			\hline
			ImageNet     &MgNet, [2,2,4,2], [128,256,512,1024]        &  77.58      &   38.5M      \tabularnewline
			\hline
		\end{tabular}
	\end{center}
\label{tab:MgNet-ImageNet-Results}
\caption{Test accuracy of MgNet on ImageNet.}
\end{table}


\documentclass[letterpaper]{article}
\usepackage{amsmath}
\usepackage{fullpage}
\usepackage{natbib}
\usepackage{textcomp}
\usepackage{color}
\definecolor{darkblue}{rgb}{0.0,0.0,0.9}
\usepackage[colorlinks=true,urlcolor=darkblue,linkcolor=black,citecolor=darkblue]{hyperref}

\newcommand{\deriv}[2]{\frac{\mathrm{d}#1}{\mathrm{d}#2}}

\begin{document}
\newpage
\pagestyle{plain}
\setcounter{page}{1}

\begin{center}
\huge{DJLES: Dubreil-Jacotin-Long Equation Solver}\\[1em]
\large{Feb 8, 2019}\\
Michael Dunphy
\end{center}

\section{Introduction}
DJLES is a MATLAB/Octave package that finds a mode-one solution to the DJL equation. \cite{StastnaLamb2002} describe the problem setup, weakly nonlinear theory, fully nonlinear theory, and iterative solution procedure. DJLES was inspired by \verb+soliw+, a DJL solving C package resulting from the work of~\cite{StastnaLamb2002}.

DJLES differs from \verb+soliw+ by being MATLAB based, which avoids the need to compile C code and manage the resulting raw binary files. When used with MATLAB, it also takes advantage of the built in parallelisation of FFTs which accelerates solution on multi-core computers. Lastly, it easily employs the resolution refinement approach of~\cite{DunphySubichStastna2011}, a feature not available in \verb+soliw+.

If you find this code useful, please cite~\cite{DunphySubichStastna2011} as it will encourage the author to maintain/improve DJLES. Bug reports, fixes/improvements and additional case files are welcome; please contact \href{mailto:mdunphy@uwaterloo.ca}{mdunphy@uwaterloo.ca} for inclusion in a future version.

DJLES has benefited from contributions from Marek Stastna, Kevin Lamb, Chris Subich, Derek Steinmoeller, Jorge Magalh\~aes and Jos\'e da Silva.

The remainder of this document is a user guide for the code.

\section{Quickstart}
Download the code and run one of the case files in either MATLAB or Octave. The case files take less than one minute to run on a recent desktop computer, and may need 2-3 minutes on an older system. Upon completion the code will produce a plot of the wave.

\section{Usage}
Typical usage is to set problem parameters, then call \verb+djles_refine_solution+ to solve the DJL equation, which returns the isopycnal displacement field $\eta$ and wave speed $c$ in MATLAB variables \verb+eta+ and \verb+c+, respectively. The required problem parameters are:
\begin{itemize}
 \item $L$ and $H$ -- Domain length and depth (m)
 \item $N_x$ and $N_z$ -- Number of grid points in $x$ and $z$
 \item $\bar{\rho}(z)$ and $\deriv{}{z} \bar{\rho}(z)$
       -- Background density profile and first derivative
       \begin{itemize}
        \item If using full density, also provide reference density parameter \verb+rho0+.
        \item If using non-dimensional density, DJLES will assume \verb+rho0=1+.
       \end{itemize}
 \item $A$ -- Target available potential energy (per unit in $y$)
       \begin{itemize}
        \item If using full density, units are kg m/s$^2$.
        \item If using non-dimensional density, units are m$^4$/s$^2$.
       \end{itemize}
 \item $U_{bg}(z)$, $\deriv{}{z} U_{bg}(z)$, and $\deriv{^2}{z^2} U_{bg}(z)$
       -- Background velocity profile (m/s) and first two derivatives
\end{itemize}
If a background velocity is not desired, set it to zero (see test cases).
Once the solution is complete, \verb+djles_diagnostics+ computes the associated velocity fields, vorticity, etc., \verb+djles_pressure+ computes the pressure, and \verb+djles_plot+ produces a simple plot of these fields.

During the first call to \verb+djles_refine_solution+, we use weakly nonlinear theory to find an initial guess for the solution, which is then iterated to convergence. Subsequent calls to \verb+djles_refine_solution+ use the previous fully nonlinear solution as the initial guess. This allows for the strategy of successively adjusting parameters (resolution, wave APE, etc) to ease/accelerate finding a solution. The various test cases demonstrate several solution scenarios.

\subsection{Tuning knobs}
There are a few tuning knobs. Set them in the case file to override the default.
\begin{itemize}
 \item \verb+min_iterate+ specifies the minimum number of iterations that the solver will take, default is 10.
 \item \verb+max_iterate+ specifies the maximum number of iterations that the solver will take, default is 2000. If you use a very small \verb+epsilon+ or \verb+relax+ you may need to increase this.
 \item \verb+NL+ is the number of Legendre points to use for the Gauss quadrature used for the APE integral. Values below 5 are probably too small, values in 15--25 are quite good, and values higher than 25 are likely unnecessary. Default is 20.
 \item \verb+relax+ is the underrelaxation factor, $0 < \verb+relax+ <= 1$.
        The value is the fraction of the new iteration, $\eta^{k*}$ to retain
        after performing an iteration, that is,
        $\eta^k = \eta^{k-1} + \verb+relax+(\eta^{k*} - \eta^{k-1})$.
        The default of 0.5 works well in most cases, and a value of 1 disables underrelaxation.
        Using a small value can help stabilise the solver at the expense of needing more iterations.
 \item \verb+g+ is the gravitational acceleration constant, default value is 9.81 m/s$^2$.
 \item \verb+verbose+ controls the verbosity of status updates. A value of 0 disables printing, a value $\ge 1$ enables printing updates after each iteration, and a value $\ge 2$ enables printing of timing data upon completion. Default is 1.
\item \verb+epsilon+ controls the stop condition, which is the relative difference between successive iterations. Default $1\times10^{-4}$. Smaller values will require more solver iterations.
\end{itemize}

\subsection{Miscellaneous}

A successful solve will have a relative error of roughly \verb+epsilon+/\verb+relax+ decimal places in $\eta$. Using a small value of \verb+relax+ should be accompanied with a reduction in \verb+epsilon+.

If the solver fails to converge,
\begin{enumerate}
 \item Ensure that your density profile $\rho(z)$ is stably stratified.
 \item Run the solver with a smaller underrelaxation factor. A value of 0.1 or even 0.01 may suppress oscillations and stabilise the solver.
 \item Ensure that your density profile (and first derivative) is smooth.
 \item Ensure that the domain width $L$ is wide enough to contain the wave. The boundary conditions for the problem are $\eta=0$ on all boundaries; if your domain is too narrow it will be enforcing that condition at the wrong location.
\end{enumerate}

\subsection{Compatibility notes}
\begin{itemize}
 \item Tested with MATLAB R2013a (Linux 64-bit and Windows), R2012a (OSX 10.6.8).
 \item Tested with Octave v3.8.1, v4.0.1 and v4.4.1 (Linux 64-bit).
 \item The initial guess part uses \verb+polyeig.m+, and Octave added it in v3.8.0, so it's unlikely to work with Octave older than v3.8.0.
 \item MATLAB R2014a prints warnings about deprecating \verb+ppval+, you can suppress the warnings with \\ \verb+warning('off','MATLAB:interp1:ppGriddedInterpolant');+
\end{itemize}

\subsection{Numerical Grid}
The domain is the rectangular region $[0,L] \times [-H,0]$ and is discretized into $N_x\times N_z$ grid boxes. The boundary conditions for the problem are zero, and we use the cell centres for the coordinates, that is,
\begin{align}
x_i^c &= \frac{\Delta x}{2} + i \Delta x, \quad i \in 0,1,2,\ldots,N_x-1,\\
z_j^c &= -H + \frac{\Delta z}{2} + j \Delta z, \quad j \in 0,1,2,\ldots,N_z-1,
\end{align}
where $(\Delta x, \Delta z) = (L/N_x, H/N_z)$. The grid shifting routine \verb+djles_shift_grid.m+ shifts the data from the cell centres to cell edges,
\begin{align}
x_i^e &= i \Delta x, \quad i \in 0,1,2,\ldots,N_x\\
z_j^e &= -H + j \Delta z, \quad j  \in 0,1,2,\ldots,N_z.
\end{align}
The array sizes are $N_x+1 \times N_z+1$ on the cell edges grid.

\section{Description of M-files}
The code consists of a handful of functions, scripts, and test cases.

\subsection{Scripts}
The scripts are
\begin{verbatim}
Filename                  - Description
-----------------------------------------------------------------------------------------
djles_common.m            - Sets default parameters and generates grid and wavenumbers.
djles_diagnostics.m       - Computes wave velocities, vorticity, etc from eta and c.
djles_initial_guess.m     - Used by djles_refine_solution.m to get an initial guess for
                            eta & c from WNL theory if an initial guess is not provided.
djles_plot.m              - Plots the solution and some diagnostics.
djles_pressure.m          - Computes the non-hydrostatic pressure and residuals in the
                            governing equations. Uses results from djles_diagnostics.m.
djles_refine_solution.m   - This is the main m-file that each case files calls. It
                            performs the iterative procedure to find a wave solution.
\end{verbatim}

\subsection{Helper Functions}
The helper functions are
\begin{verbatim}
djles_change_resolution.m - Changes the resolution of the solution.
djles_compute_apedens.m   - Computes the available potential energy density.
djles_diffmatrix.m        - Generates a finite difference differentiation matrix for use
                            in the initial guess procedure.
djles_extend.m            - Extends the input data using specified symmetry.
djles_gradient.m          - Computes the gradient of input data.
djles_quadweights.m       - Returns the 1-dimensional interior grid quadrature weights.
djles_residual.m          - Computes the residual in the DJL equation.
djles_shift_grid.m        - Shifts the input data from cell centres to cell endpoints.
djles_sinequadrature.m    - Computes the interior grid sine quadrature weights for
                            area integrals.
djles_wavelength.m        - Computes an estimate of the wavelength.
\end{verbatim}

\subsection{Test cases}
Included are eight test cases that demonstrate how to use DJLES. A brief description of each follows.

\subsubsection{Small APE}
Test case: \verb"case_small_ape.m" \\\\
This test case demonstrates the parameter regime of no background current, a smooth pycnocline, and small wave APE.
The initial guess obtained from weakly nonlinear theory will be ``close'' to the fully nonlinear DJL solution, so the solver readily converges.
Once we find the low resolution wave (32$\times$32), we increase the resolution to 512$\times$256 and iterate to convergence.
The low resolution solution is a very good initial guess for the high resolution problem, so the high resolution wave is also readily solved.

\subsubsection{Large APE}
Test case: \verb"case_large_ape.m" \\\\
This test case demonstrates the parameter regime of no background current, a smooth pycnocline, and large wave APE.
The initial guess from weakly nonlinear theory will be rather poor.
The solution strategy we use is to find a wave with a small APE, and supply that wave as the initial guess to find a wave with a larger APE.
Applying this several times yields the final wave with a large APE.
In this case we start at one percent of the target APE and raise the APE in five steps.
Lastly, we increase the resolution once we reach the target APE.
The result is a broad flat crested wave.

\subsubsection{Background current}
Test case: \verb"case_ubg.m" \\\\
We use the same successive solution strategy here by solving without a background current first,
and then raise the background current strength in increments until we reach the full background current.
We increase epsilon for the intermediate solutions to accelerate the process as we only need them to be of ``initial guess'' quality.
Lastly, we reduce epsilon, and increase the resolution in two increments.

\subsubsection{Sharp pycnocline}
Test case: \verb"case_sharp_pycnocline.m" \\\\
Obtaining a solution with a sharp pycnocline can be difficult for the solver to find directly, and the successive solution strategy works here as well.
We begin with a low-resolution wide-pycnocline that is readily solved and begin successively reducing the pycnocline thickness.
As we sharpen the pycnocline, we also increase the resolution such that the pycnocline is not under-resolved.

\subsubsection{Synthetic data}
Test case: \verb"case_synthetic_data.m" \\\\
This case matches the parameters of the background current case, except that we use synthetic mooring data as the background profiles.
We sample the analytic functions at 25 evenly spaced points and construct continuous profiles using linear interpolating polynomials.
Second order differentiation provides the gradients, which are linearly interpolated in the same fashion.
The \verb+linear+ and \verb+pchip+ interpolation methods preserve monotonocity which is important to ensure the interpolated density profile is stable.
The \verb+spline+ method does not preserve monotonicity so be careful if you use it.
As in \verb"case_ubg.m", we raise the background velocity in four successive steps, followed by three resolution refinements.

\subsubsection{Lake Erie data}
Test case: \verb"case_lakeerie.m" \&  \verb"case_lakeerie.txt" \\\\
Here we use time-averaged temperature profile data collected from Lake Erie (K.~Lamb 2014, pers.~comm.).
The text file contains two columns, depth (m) and temperature (\textdegree C), and we convert the temperature to density with a linear equation of state.
There is no surface value so we extrapolate to find a value for $z=0$.
We construct the density and density gradient in the same fashion as in the synthetic data case, and use no background current.
We solve the wave at low resolution and then raise the resolution in two steps.

\subsubsection{Pineda et al. (2015)}
Test case: \verb"case_pineda.m" \&  \verb"case_pineda_cast1.txt" \\\\
This case is intended as a typical shallow continental shelf case where we reproduce the DJL solutions shown in~\cite{PinedaEtAl2015}.
The text file contains two columns, depth (m) and density (kg/m$^3$), and is an approximate reproduction (courtesy Jorge Magalh\~aes and Jos\'e da Silva) of the smoothed curve of Fig 9a.
We use the full density and set the reference density \verb+rho0+ as the maximum value, and construct the density and first derivative functions as in the synthetic data case.

The first part of the script finds the wave showcased in Figure 11.
After raising the target APE in a few steps at low resolution and large epsilon, we apply several steps of resolution refinement with epsilon reductions.
The relative residual of the final wave is order $4\times 10^{-7}$, and matches~\cite{PinedaEtAl2015} to three significant figures in wave height ($-14.1176$ m vs. $-14.1$ m) and two significant figures in phase speed (0.585978 m/s vs. 0.585 m/s).

The second part of the script reproduces the solid curves of Figure 10.
We sweep over a range of target APE values and record the wave height, phase speed, and velocity and pressure one metre above the bottom.
Each wave is reused as the initial guess for the next APE value as the sweep progresses.

Lastly, we note there is a difference in amplitude between the pressure reported in~\cite{PinedaEtAl2015} and that computed by DJLES (Figures 10c and 11c).
We compute residuals by substituting the wave fields into the Boussinesq equations moving with the wave (see bottom part of \verb+djles_pressure.m+), and find that the residuals are appropriately small, which indicates that the pressure computed by DJLES is correct.

\subsubsection{Amazon River mouth data}
Test case: \verb"case_amazon.m" \&  \verb"case_amazon.txt" \\\\
This case is intended to find the maximum wave amplitude for internal solitary waves on the continental shelf near the mouth of the Amazon River.
The included text file contains three columns: depth (m), density (kg/m$^3$) and velocity (m/s) projected along the wave direction of propagation (data courtesy of Jorge Magalh\~aes and Jos\'e da Silva).

The density profile has inversions, so we make ad-hoc adjustments to ensure that the first derivative is always negative (and thus $N^2(z)$ is always positive).
Much like the other cases, we start with a small amplitude at low resolution, and incrementally raise the values until we reach a high resolution wave of 35.3 m amplitude.
This wave is near its amplitude limit; further increases to the APE will expand the wave as a broad flat crested wave and a larger domain is needed to calculate such a wave.

% References
\bibliographystyle{kbib}
\bibliography{djles}

\end{document}


\section{Description of datasets}
\label{sec:datasets}

Most of the VQA datasets have strong biases. This allow models to learn strategies without reasoning about the visual input~\cite{Santoro2017ASN}.
The CLEVR dataset~\cite{johnson2017clevr} was developed to address those issues and come back to the core challenge of visual QA which is testing reasoning abilities.
CLEVR contains images of 3D-rendered objects; each image comes with a number of highly compositional questions that fall into different categories.
Those categories fall into 5 classes of tasks: Exist, Count, Compare Integer, Query Attribute and Compare Attribute.
The CLEVR dataset consists of:
\begin{itemize}
\item 	A training set of 70k images and 700k questions,
\item	A validation set of 15k images and 150k questions,
\item	A test  set of 15k images and 150k questions about objects,
\item	Answers, scene graphs and functional programs for all train and val images and questions.
\end{itemize}
Each object present in the scene, aside of position, is characterized by a set of four attributes:
\begin{itemize}
\item 2 sizes: large, small,
\item 3 shapes: square, cylinder, sphere,
\item 2 material types: rubber, metal,
\item 8 color types: gray, blue, brown, yellow, red, green, purple, cyan,
\end{itemize}
resulting in 96 unique combinations.

Along with CLEVR, the authors~\cite{johnson2017clevr} introduced  CLEVR-CoGenT (Compositional Generalization Test, CoGenT in short), with a goal of evaluating how well the models can generalize, learn relations and compositional concepts.
This dataset is generated in the same way as CLEVR with two additional conditions.
As shown in \tableref{tab:cogent_conditions}, in Condition A all cubes are gray, blue, brown, or yellow, whereas all cylinders are red, green, purple, or cyan; in Condition B cubes and cylinders swap color palettes.
For both conditions spheres can be any colors.

\begin{table}[b!]
	\centering
	\begin{tabular}{cccc}
		\toprule
		Dataset        & Cubes              & Cylinders &  Spheres         \\
		\midrule
		CLEVR   &  any color &  any color        &    any color    \\
		%\midrule
		CLEVR CoGenT A & gray / blue / brown / yellow  & red / green / purple / cyan       &    any color  \\
		CLEVR CoGenT B  & red / green / purple / cyan &   gray / blue / brown / yellow       &      any color  \\
		\bottomrule
	\end{tabular}
	\caption{Colors/shapes combinations present in CLEVR, CoGenT-A and CoGenT-B datasets.}
	\label{tab:cogent_conditions}
\end{table}

The CoGenT dataset contains:
\begin{itemize}
\item	Training set of 70,000 images and 699,960 questions in Condition A,
\item	Validation set of 15,000 images and 149,991 questions in Condition A,
\item	Test set of 15,000 images and 149,980 questions in Condition A (without answers),
\item	Validation set of 15,000 images and 150,000 questions in Condition B,
\item	Test set of 15,000 images and 149,992 questions in Condition B (without answers),
\item	Scene graphs and functional programs for all training/validation images/questions.
\end{itemize}


\chapter{Cart-Pole \& Mountain-Car}
\section{Cart-Pole}
The cart-pole environment, which is an implementation of the famous cart-pole problem [7], is a popular AI benchmark that simulates a pole that needs to be balanced on the flat rectangle (cart) that it's attached to. The observation object of this environment includes the velocity and position of the cart, the velocity at the tip of the pole, and the angle of the pole. Initially, these parameters are assigned a random value in the range $[-0.05, 0.05]$, which results in the pole standing almost upright with a slight tilt. The environment only allows two discrete actions (left and right) which the agent uses to balance the pole before it falls over, by pushing the cart without causing it to go off-screen. The environment terminates if the agent fails in either of those tasks, or after the maximum number of time-steps (200) is reached. The agent receives $1$ reward on every time-step until one of these termination criteria is met. Finally, this environment is considered to be solved if the agent achieves an average reward of greater than or equal to $195.0$ over 100 consecutive trials. More details about the various environmental parameters can be found in the environment’s official GitHub Wiki page\footnote{\url{https://github.com/openai/gym/wiki/CartPole-v0}}.

\subsection{Experiment 1: Binary Programs}
The goal of the first experiment was to implement the simplest GP algorithm that performs better than a random agent. The simplest language that can encode the actions allowed in this environment is one of 0’s and 1’s, where 0’s represent ‘left’ and 1’s represent ‘right’. The program structure, therefore, included the terminal set \verb+T = {0, 1}+ and an empty function set. The intuition for this structure was that, in principle, there has to exist a string of left-right actions that manages to balance the pole on the cart most of the time.

This GP agent did not managed to achieve better-than-random performance: a random agent achieves an average reward of $\approx22.0$ and the best programs the GP algorithm generated achieved an average reward of $16.76$. The most likely reason for this is that these programs are static, which means they don’t use information about the state of the environment to compute the next action to take. This limitation would not be a problem if the initial state of the environment was always the same, because that would make the state transitions deterministic and, therefore, there would exist a static program that would reliably balance the pole. However, the initial state of the environment is randomised each time the environment starts running and the range of possible values for the various parameters of the environment is sufficiently large, such that no static strategy can perform well.

\subsection{Experiment 2: Known Strategy}
The main goal of the second experiment was to enable the agent to utilise the environment state to choose actions dynamically. At the same time, I wanted to test the capabilities of the GP algorithm, since it would be used to find increasingly complex programs as the environments proved to be more difficult to solve. To achieve both of these goals, a manual strategy was discovered that performed well by utilising the environment state. The program structure of the GP algorithm was also adjusted to allow it to discover the strategy automatically. The chosen strategy can be described as follows:

\begin{verbatim}
    if pole_angle <= 0 then push_left() else push_right()
\end{verbatim}

Intuitively, this strategy describes the behaviour "if the pole is leaning to the left, push the cart to the left; if the pole is leaning to the right, push the cart to the right". This should keep the pole balanced at all times. To test this intuition, I implemented an agent that follows this strategy and tested it on the environment. The agent achieved an average score of $42.0$, a major improvement over the previous agent. Then, I introduced a simple environment-specific language to replace the binary language of the previous experiment, which included variables to represent the environment state and the actions the agent can take, flow-control (if-statements) and comparison operators to allow the agent to make decisions dynamically, and constants for calculations. The function and terminal set used for this experiment were \verb+F = {IFLTE}+ and \verb+T = {pa, 0, L, R}+ respectively.

The function set included a single function, \verb+IFLTE+ (if-less-than-or-equal), which accepts 4 inputs; if the first input is less than or equal to the second input, it outputs the third input; otherwise, it outputs the fourth input. This functionality seemed sufficient to encode the strategy described above. The terminal set included four elements: the pole angle (\verb+pa+) used to encode the corresponding environment observation, the numerical constant \verb+0+ which was required for the comparison performed in the strategy, and the actions \verb+L+ and \verb+R+, left and right respectively. The program structure was quite restricted in this first version of the algorithm to make it easier to evaluate and troubleshoot before applying it to more difficult environments that would require a more complex program structure. Specifically, each program was restricted to consist of a single \verb+IFLTE+ function, which could only accept \verb+pa+ and \verb+0+ as its first two arguments, and \verb+L+ and \verb+R+ as its third and fourth arguments (figure \ref{fig:iflte_tree}). It was important to restrict the third and fourth arguments to actions since the output of the program had to be the action the agent would perform in the environment. Therefore, a simple type system was implemented to perform this check. Using this simple functional language, the manually-discovered strategy could be encoded as \verb+IFLTE(pa, 0, L, R)+.

\begin{figure}[ht]
    \centering
    \includegraphics[width=12cm]{images/simple_iflte_program.png}
    \caption{Tree representation of the program structure of the Known Strategy agent}
    \label{fig:iflte_tree}
\end{figure}

With these structural restrictions, it was easy for the GP algorithm to converge to the intended strategy (it only required a single generation, and mutation was not needed) because the program space was very small. The exact parameters used for the experiment are summarised in table \ref{tab:cartpole_exp2_params}.

\begin{table}[ht]
    \centering
    \begin{tabular}{|l|c|}
        \hline
        \textbf{Parameters} & \textbf{Values} \\
        \hline
        Population size     & 10  \\
        Max generations     & 5   \\
        Max program depth   & 1  \\
        Terminal fitness & 195.0 \\
        Number of runs      & 1   \\
        Number of episodes  & 100 \\
        Episode length      & 200  \\
        \hline
    \end{tabular}
    \caption{Experiment parameters (cart-pole experiment 2)}
    \label{tab:cartpole_exp2_params}
\end{table}

\subsection{Experiment 3: Exploration}
\subsubsection{Setup and Motivation}
The goal of the final experiment was to expand the program space to encourage exploration and allow the algorithm to discover better strategies than the one found manually. The terminal set was extended to include the second observation of the environment, pole velocity (\verb+pv+), to allow the algorithm to incorporate it in the solution. Additionally, I experimented with constants other than \verb+0+ and it turned out that the performance was very sensitive to even very small deviations from \verb+0+. Specifically, I discovered that including the constant \verb+0.025+ in the terminal set increased the performance of the discovered solutions significantly. Finally, the program structure was adapted to allow for the generation of programs that utilised higher-order functions (i.e. functions that accept functions as arguments) which can be represented as a trees of varying depth (see figure \ref{fig:deep_iflte_tree} for an example). The extended terminal set was \verb+T = {pa, pv, 0, 0.025, L, R}+. The parameters used for this experiment are summarised in table \ref{tab:cartpole_exp3_params}.

\begin{figure}[ht]
    \centering
    \includegraphics[width=12cm]{images/complex_iflte_program.png}
    \caption{Tree representation of the program structure of the Exploration agent}
    \label{fig:deep_iflte_tree}
\end{figure}

\begin{table}[ht]
    \centering
    \begin{tabular}{|l|c|}
        \hline
        \textbf{Parameters} & \textbf{Values} \\
        \hline
        Population size     & 100  \\
        Max generations     & 5  \\
        Max program depth   & 2  \\
        Terminal fitness & 195.0 \\
        Number of runs      & 1   \\
        Number of episodes  & 100 \\
        Episode length      & 200  \\
        \hline
    \end{tabular}
    \caption{Experiment parameters (cart-pole experiment 3)}
    \label{tab:cartpole_exp3_params}
\end{table}

\subsubsection{Results and Discussion}
The initial results of this experiment showed that the second observation, pole velocity, was more useful than the pole angle, so programs using \verb+pv+ instead of \verb+pa+ tended to dominate the population during the first few generations. The first run of GP converged on the program \verb+IFLTE(pv, 0, L, R)+, which achieved an average reward of $181.78$. This improvement could be attributed to the fact that the velocity of the pole provided the agent with more useful information than its angle. Using the first strategy (\verb+IFLTE(pa, 0, L, R)+), if the pole leaned towards, say, the right at the beginning of the run, the agent would begin to push the cart to the right to cause the pole to lean towards the left, bringing it back to the center. The problem with this strategy was that by the time the pole angle became negative (meaning the pole leaned towards the left) it had accumulated too much momentum for the agent to manage to push left to balance it on time. The agent was essentially over-correcting. Using the new strategy, the agent detected which direction the pole was accelerating towards (using the sign of its velocity) and pushed the cart towards that direction for just long enough to prevent the pole from falling over, without over-correcting. However, this strategy was still problematic (as is evident from the imperfect score it achieved) because the agent stopped pushing the cart as soon as the sign of the pole velocity was flipped, so the pole didn't have enough momentum to lean towards the other side, thereby letting it fall towards the same side again. The result of this was that the agent was now under-correcting, balancing the pole perfectly, but at an angle, causing the cart to move until it eventually exited the screen and the environment terminated. It was evident at this point that a solution would involve information about both the pole angle and its velocity.

Increasing the maximum number of generations from 5 to 10 allowed for more complex solutions to arise. Eventually, the algorithm consistently produced solutions that achieved the reward specified as the solution criterion for the environment. These solutions included programs using nested if statements and combining both observation variables. An example of such a program is:
\begin{verbatim}IFLTE(0.025, pa, IFLTE(pv, pv, R, L), IFLTE(0.025, pv, R, L))\end{verbatim}
This program achieves an average reward of $198.74$ over 100 consecutive trials. The following is an equivalent, but more readable, version of the strategy described by this program:

\begin{verbatim}
if pa > 0.025 then R else (if pv > 0.025 then R else L)
\end{verbatim}

Following this strategy, the agent pushed the cart to the right to balance the pole if it was leaning at a right angle greater than some threshold ($0.025$). If the pole was leaning at a left angle or a small right angle, the agent checked the pole's velocity to make its decision: if the pole was accelerating towards the right (\verb+pv > 0.025+) then the agent pushed right to counteract it; otherwise it pushed left for the same reason.

This extended version of the program structure allowed the agent to achieve the goal of utilising exploration to find new and better solutions by exploring a larger program space. At this stage, the GP algorithm can incorporate new language extensions as well as additional genetic operators, both of which will be required in the following environments. So, Cart Pole was a useful starting point in showcasing the potential of this approach and establishing a framework for applying it to other RL environments.

\section{Mountain-Car}
The second RL environment attempted, was mountain-car. This environment simulates a car that attempts to climb a steep hill (figure \ref{fig:mountain-car}). There are two versions of this environment, one in which the actions the agent can take are discrete (similar to cart-pole) and one in which they are continuous (the action is a number that represents the magnitude of the force the agent applies to the car). The former is, in principle, easier to solve because there are fewer possible actions and, therefore, fewer strategies to explore before discovering a solution. The discrete version of the environment was attempted first with the hope that a solution to the simpler environment could be used as a basis for the more complex one.

\subsection{Experiment 1: Discrete Actions}
For the discrete version of this environment, I used the same parameters as in cart-pole experiment 3, and a similar program structure, adapted to the details of this environment:

\begin{verbatim}
F = {IFLTE}
T = {position, velocity, 0.0, 0, 1, 2}
\end{verbatim}
The terminals \verb+position+ and \verb+velocity+ represent the corresponding environment observations, \verb+0.0+ is a numerical constant used for comparisons (similar to cart-pole), and \verb+0+, \verb+1+, and \verb+2+ encode the discrete action space of the environment (push left, no action, and push right respectively).

In fewer than $10$ generations, the GP algorithm found the simple and intuitive program \\\verb+IFLTE(0.0, velocity, 2, 0)+, which achieves a fitness score of approximately $-120$, which is very close to the requirement for a solution, $-110$. This program encodes the strategy “if the velocity of the car is positive, push the car to the right (i.e. increase its velocity), otherwise push it to the left (i.e. decrease its velocity).” The agent effectively swings the car back and forth, maximising its momentum, until it can reach a high enough velocity to climb the hill and reach the goal.

\subsection{Experiment 2: Continuous Actions}
The interpretable nature of the strategy found for the discrete version of this environment makes it reasonable to hypothesise that the same strategy should perform well on the continuous version of this environment. The same principle of applying a force proportional to the car’s velocity can be used, so the only required modification is changing the value of this applied force to a continuous value in the range $[0.0, 1.0]$, which defines the action space of this environment.

The problem then becomes a simple numerical optimisation problem with one variable. Evaluating the program using all values in $[0.0, 1.0]$ with a $0.001$ increment, reveals that the optimal force is $0.180$, which produces an average fitness score of $98.80$, which is greater than the requirement for a solution ($90$). The complete set of results is visualised in figure \ref{fig:mountain_car_cont}, which shows that values between $0.0$ and $0.179$ produce negative fitness scores (the car is unable to reach the goal), and values after $0.180$ linearly decrease performance. The reason for the negative values at the beginning of the range is that the force applied to the car is not great enough to eventually allow it to climb the hill, so it only receives negative rewards from the environment on every time step. The optimal value is the smallest amount of force necessary to eventually allow the car to climb the hill, which satisfies the requirement of the environment (reaching the goal with the minimum amount of effort). Every value greater than the optimal increases the amount of effort beyond what is necessary for the goal to be achieved and, therefore, reduces the overall fitness score.

\begin{figure}[ht]
    \centering
    \includegraphics[width=12cm]{images/mountaincarcont.png}
    \caption{The x-axis represents the magnitude of the for applied to the car (multiplied by 1000) and the y-axis represents the average reward achieved using that magnitude.}
    \label{fig:mountain_car_cont}
\end{figure}

The simple and intuitive nature of this solution is further confirmation that GP is a suitable approach for solving (at least simple) RL environments. Another interesting feature of this program is its similarity to the solution for cart-pole, despite the differences between the environments. This is an indication that environments that share common characteristics (e.g. classic control environments) might be solvable using similar solutions.


\documentclass[main.tex]{subfiles}
\begin{document}

\section{Initial Data Problem}

\marginpar{Tuesday\\ 2021-6-1, \\ compiled \\ \today}

It is perhaps the most difficult problem currently.

The problem is stated as
%
\begin{align}
C_0 &= R + K^2 - K_{ij} K^{ij} - 16 \pi E = 0  \\
C_i &= D_j K^{j}_{i} - D_i K - 8 \pi P_i = 0
\,,
\end{align}
%
and we want to calculate \(\gamma_{ij}\) and \(K_{ij} \) on \(\Sigma _0\) such that
\begin{enumerate}
    \item the constraints are satisfied;
    \item the solution is physically meaningful: it really describes a BH, BBH, \(n\)-BH, NS spacetime.
\end{enumerate}

We will assume for today that the matter fields are given or zero --- in general one would have to solve the hydrodynamic equations for them as well.

We have 4 equations for 12 unknowns!
We will need to prescribe 8 quantities.

The problem is split in two: a problem of constrained data (4)
and one of free data (8).

The choice of free data is guided by a few principles:
\begin{enumerate}
    \item we need to define our (astro)physical expectations for \(\Sigma_0 \) and verify that they are matched;
    \item we need some heuristic or intuition for the various fields;
    \item mathematical (/ computational?) necessity: the equations \(C_\alpha = 0\) should be as nice as possible --- linearity, decoupling, well-posedness are all desirable.
\end{enumerate}

The \textbf{Conformal Decomposition} allows us to implement these.
Within it, there are two main formalisms: one is the ``conformal/transverse traceless'' formalism (CTT), due to York in 1973; the other is the ``conformal/thin sandwich'' (CTS), also due to York in 1999.
Both are based off of early work by Lichnerowicz.

The CTS formalism is often used to generate binary systems --- it makes it easier to implement the quasi-symmetries there.

We start from the Lichnerowicz equation (the Hamiltonian constraint):
%
\begin{align}
C_0 &= \widetilde{D}_i \widetilde{D}^{i} \psi
- \frac{1}{8} \hat{A}_{ij} \hat{A}^{ij} \psi^{-7} +
\qty(- \frac{K^2}{12} + 2 \pi E) \psi^{5} &= 0
\,,
\end{align}
%
with \(p = -10\).
Let us solve it as a standalone equation: it is a Poisson-like one, since \(\widetilde{D}_i \widetilde{D}^{i}\) is similar to a Laplacian.
Then, this is some sort of elliptic operator applied to \(\psi \) equated to some powers of \(\psi \).

If \(K^2= 0\) the equation simplifies, and the BVP can be studied: there are several known results about the well-posedness of this equation under the hypothesis \(K = \const\).

These are known as Constant Mean Curvature (CMC) spacetimes.

If we take an Asymptotically Flat, CMC spacetime with \(K = 0\), and with \(E = 0\), then the BVP with the Lichnerowicz equation is \emph{solvable} for a ``large'' class of metrics \(\widetilde{\gamma}_{ij}\) (a ``positive Yamabi class'').

There is a ``prototype equation'' for the Lichnerowicz one if we set \(K = 0\):
%
\begin{align}
\triangle u = f^2 u^{p}
\,,
\end{align}
%
in flat spacetime, for some function \(f\).

These theorems are found in somewhat specialized PDE literature.
In the linear case \(p = 1\), and we take \(\eval{u}_{\partial \Omega } = 0\), then the solution, identically zero, is unique.

In the nonlinear case, the solution is unique iff \(p > 0\) (or specifically, the same sign of the coefficient of the \(f^2 u^{p}\) term).

So, what about the Lichnerowicz equation?
The easiest thing to do is to linearize the equation: we write it as
%
\begin{align}
\widetilde{\triangle} \psi + H (\psi ) = 0
\,.
\end{align}

We write \(\psi = \psi_0 + \epsilon \), and expand
%
\begin{align}
H(\psi ) = H(\psi_0 ) + \eval{\pdv{H}{\epsilon }}_{0} \epsilon + \order{\epsilon^2}
\,,
\end{align}
%
so we find an equation in the form \(\widetilde{\triangle} \epsilon = f \epsilon \) with:
%
\begin{align}
f = \frac{1}{8 } \widetilde{R} + \frac{7}{8} \hat{A}_{ij} \hat{A}^{ij} \psi_0^{-8} - 10 \pi E \psi_0^{4}
\,.
\end{align}

This immediately shows us the problem: if \(K =0 \) then \(\widetilde{R} > 0\), but we can check that \(f\) is not positive, because of the ``matter term''!

However, we insist on solving this equation, and we can do so as long as we \emph{rescale} the matter term.

If we define \(\widetilde{E} = \psi^{s} E\), then, the term becomes
%
\begin{align}
- 5 \times 2 \pi E \psi_0^{4} \to - (s \times 5) 2 \pi \widetilde{E} \psi^{4-s}
\,,
\end{align}
%
so this is \(> 0\) for any \(s > 5\).
This is \emph{not} a trick: in practice, these elliptic equations are solved by iteration --- specifying a guess, and then iterating.

Specifying the ``correct'' RHS therefore is key if we want to find a solution.
So, what do we use?
The best answer turns out to be 8: if we do this, we can express the dominant energy condition in conformal variables: if
%
\begin{align}
\widetilde{E} = \psi^{8} E
\qquad \text{and} \qquad
\widetilde{P}^{i} = \psi^{10} P^{i}
\,,
\end{align}
%
then \(\widetilde{E}^2 \geq \widetilde{P}^2\) implies \(E^2 \geq P^2\). In general this would read
%
\begin{align}
\psi^{2s} E^2 \geq \psi^{-4} \psi^{10} \psi^{10} P^2
\,.
\end{align}

\subsection{Maximal Slicing}

What does it mean to impose \(K = 0\)?
As we will see shortly, this is a gauge condition which extremizes the volume of \(\Sigma _t\).

We just need to remember the definition of the trace: it is an identity that
%
\begin{align}
K = \gamma^{ab} K_{ab} = - \frac{1}{2 \alpha } \gamma^{ij} \mathscr{L}_m \gamma_{ij} = - \frac{1}{2 \alpha } \mathscr{L}_m \log \gamma
\,,
\end{align}
%
which can be nicely written as
%
\begin{align}
K = - \frac{1}{\alpha } \mathscr{L}_m \log \sqrt{\gamma }
\,,
\end{align}
%
so we recover the volume element on \(\Sigma _t\), \(\sqrt{\gamma }\)!

The volume in a certain region is \(V = \int \sqrt{\gamma } \dd[3]{x}\), so if we perform a variation \(v^{a} = \delta t \qty(\alpha n^a + \beta^{a})\), so that \(\eval{v^{a}}_{S} = 0\) (the deformation is zero at the boundary), the volume changes as such:
%
\begin{align}
\fdv{V}{t} &= \int \partial_{t} \sqrt{\gamma } \dd[3]{x}
= \int \qty(- \alpha K + D_i \beta^{i}) \dd[3]{x}
\marginnote{Using the kinematic equation.}  \\
&= - \int_V \alpha K \sqrt{\gamma } \dd[3]{x} + \underbrace{\oint_S \beta^{i} S_i}_{= 0}
= - \int \alpha K \sqrt{\gamma } \dd[3]{x}
\,,
\end{align}
%
therefore \(K = 0\) yields \(\delta V = 0\): this is a maximum.

It is the exact same geometric problem as a film of soap.
If the soap is held at a ring, it stays flat.

The only difference from that case is that here we are in a Lorentzian geometry: in the Euclidean case we have a minimum, here we have a maximum.

This is an example of a singularity-avoiding gauge, used by many works.

\subsection{The Conformal Transverse Traceless formalism}

We use the \(p = -10\) scaling of \(K_{ij}\), and on top of the Lichnerowicz equation we derive an equation from the momentum constraint \(C_i = 0\), for a vector \(x^{i}\) which is obtained with a further decomposition of \(\hat{A}^{ij}\):
%
\begin{align}
\hat{A}^{ij} = \hat{A}^{ij}_L + \hat{A}^{ij}_{TT}
\,,
\end{align}
%
where the TT part is transverse and traceless: \(\widetilde{\gamma}_{ij} \hat{A}^{ij}_{TT} = 0\) and \(\widetilde{D}_j \hat{A}^{ij}_{TT} = 0\).

The longitudinal part on the other hand is expressed as follows:
%
\begin{align}
\qty(\widetilde{L} x)^{ij} = \widetilde{D}^{i} x^{j} + \widetilde{D}^{j} x^{i} - \frac{2}{3} \widetilde{D}_k x^{k} \widetilde{\gamma}^{ij}
\,.
\end{align}

This seems like a rather strange expression: \(\widetilde{L}\) is called the \textbf{Conformal Killing operator} on \(x^{i}\).
It is determined as follows:
%
\begin{align}
\widetilde{D}_j  \hat{A}^{ij} =
\widetilde{D}_j  \qty(\widetilde{L} x)^{ij}
= \widetilde{D}_j  \widetilde{D}^{j} x^{i}
+ \frac{1}{3} \widetilde{D}^{i} \widetilde{D}_j x^{i} + \widetilde{R}^{i}{}_j x^{i} = \widetilde{\triangle}_L x^{i}
\,,
\end{align}
%
where \(\widetilde{\triangle}_L x^{i}\) is the \textbf{conformal vector Laplacian operator}.

There exists a unique L + TT decomposition of \(\hat{A}^{ij}\) iff there exists a unique solution of the conformal Laplacian equation:
%
\begin{align}
\widetilde{\triangle}_L x^{i} = \widetilde{D}_j \hat{A}^{ij}
\,.
\end{align}

This is useful because there is a theorem by Cantor in 1979 which tells us that if \(\Sigma \) is asymptotically flat and we have \(\partial_{k} \partial_{l} \widetilde{\gamma}_{ij} = \order{r^{-3}}\),
then existence and uniqueness are guaranteed.

With this decomposition, we obtain
%
\begin{align}
\widetilde{\triangle}_L x^{i} - \frac{2}{3} \widetilde{D}^{i} K \psi^{6} - 8 \pi \widetilde{P}^{i} = 0
\,,
\end{align}
%
which can be solved together with the Lichnerowicz equation.
This allows us to constrain \(\psi\) and \(x^{i}\), as long as we determine the free data: \(\widetilde{\gamma}_{ij}\), \(\hat{A}^{ij}_{TT}\) and \(K\).

Under maximal slicing, \(K = 0\), the two equations decouple!
They also partially decouple if we take \(K = \const\), since we can solve one and then the other.

Also, it is nice that the free data are the conformal metric: \(\widetilde{\gamma}_{ij}\) and \(\hat{A}_{TT}^{ij}\) are useful in heuristically determining the GW content of \(\Sigma \).

Let us give an example of CTT data under the four hypotheses of conformal flatness, asymptotic flatness, and maximal slicing.

Asymptotic flatness is chosen because we want the solution to exist.
This is the simplest choice of free data: we take \(\hat{\gamma}_{ij} = f_{ij}\) (conformal flatness) and \(K = 0\) (maximal slicing).

Further, we consider \(E = 0 = P^{i}\) (vacuum).

Without justification we also assume that \(\hat{A}^{ij}_{TT} = 0\).

Under all these assumptions we have
%
\begin{align}
\widetilde{D}_i = D_i
\,,
\end{align}
%
therefore
%
\begin{align}
\widetilde{D}_i \widetilde{D}^{i} = D_i D^{i} = \triangle
\,
\end{align}
%
is simply the flat elliptic operator, also \(\widetilde{R} = 0\) and \(\widetilde{L} = L\) is the one calculated in the flat case.

So, the CTT equations become:
%
\begin{align}
\triangle \psi + \frac{1}{8} \qty(Lx )_{ij} \qty(Lx)^{ij} \psi^{-7} &= 0  \\
\triangle_L x^{i} = \triangle x^{i} + \frac{1}{3} D_j D^{i} x^{j} &= 0
\,,
\end{align}
%
which are decoupled, so they can be solved independently of one another.
They are ``easy'', since they use the flat Euclidean operators.
If we want a boundary value problem we also need to specify the boundaries: we specify asymptotic flatness: \(\psi = 1\) and \(x^{i} = 0\) at \(\iota_0\).

Optionally we can impose strong-field inner boundary conditions: for example we can fix the topology.

Our \textbf{case 1} is to take \(\Sigma_0 = \mathbb{R}^{3}\) (no inner boundary condition): then, the solution of the second CTT equation is \(x^{i}= 0\), and the first one reads \(\triangle \psi = 0\) together with \(\psi = 1\) at \(\iota_0 \): this implies \(\psi \equiv 1\).
As expected, we recover flat spacetime.

Can we also get a nontrivial solution?
For that, we need an inner boundary.

\end{document}


% BReviewofLinearAlgebra.tex
% Fund Science! & Help Ernest finish his Physics Research! : quantum super-A-polynomials - a thesis by Ernest Yeung
%
% http://igg.me/at/ernestyalumni2014
%
% Facebook     : ernestyalumni
% github       : ernestyalumni
% gmail        : ernestyalumni
% google       : ernestyalumni
% linkedin     : ernestyalumni
% tumblr       : ernestyalumni
% twitter      : ernestyalumni
% youtube      : ernestyalumni
% indiegogo    : ernestyalumni
%
% Ernest Yeung was supported by Mr. and Mrs. C.W. Yeung, Prof. Robert A. Rosenstone, Michael Drown, Arvid Kingl, Mr. and Mrs. Valerie Cheng, and the Foundation for Polish Sciences, Warsaw University.

\section{Review of Linear Algebra }

\subsection{Linear Maps}

\exercisehead{B.1}
\begin{enumerate}
  \item[(a)]
\item[(b)]
\item[(c)]
\item[(d)] \textbf{Want}: if $(v_1 \dots v_k)$ linearly dependent $k$-tuple in $V$, $v_1 \neq 0$, \\
then some $v_i = \sum_{j=1}^{i-1} c^j v_j$ \\
\begin{proof}
$(v_1 \dots v_k)$ linearly dependent, so if $\sum_{i=1}^k a^i v_i = 0$, $a^i$ not all equal to $0$.  \\

Suppose for fixed $i$, $2\leq i \leq k$, $a^{i+1} = \dots =a^k=0$. \\
\quad Indeed, suppose for $\sum_{i=1}^k a^i v_i = 0$, $a^2 = \dots = a^k = 0$.  $a^1 v_1 =0$, $v_1 \neq 0$, $a^1 =0$.  Then $(v_1 \dots v_k)$ linearly independent.  Contradiction.

if $i=2$, \\
\phantom{\quad} $a^1 v_1 + a^2 v_2 =0$ \\
\phantom{\quad \,} $\Longrightarrow v_2  = \frac{-a^1}{a^2} v_1$ \\

if $i=k$, $v_k = \frac{ -\sum_{i=1}^{k-1} a^i v_i }{ a^k}$  \\

So in general, $v_i = \frac{ -\sum_{j=1}^{i-1} a^j v_j }{ a^i}$
\end{proof}
\end{enumerate}

\exercisehead{B.9}
given $(E_1 \dots E_n)$ basis for $V$ \\
\phantom{ \quad } $\exists \, \lbrace i_1 \dots i_k \rbrace \subset \lbrace 1 \dots n \rbrace$ s.t. \\
\phantom{ \quad \, } $ \text{span}(E_{i_1} \dots E_{i_k} )$ is complement to $S$ \\

Hence $\forall \, $ subspace $S \subseteq V$, $\exists \, $ complementary subspace $T$ in $V$, so $V = S \oplus T$

\begin{proof}
$\forall \, $ subspace $S$ is itself a vector space, closed under addition and multiplication. \\
\phantom{ \quad } Hence $S$ has basis $(F_1 \dots F_m)$ with $\text{dim}S = m$ \\

Consider ordered $(m+n)$-tuple
\[
(F_1 \dots F_m, E_1 \dots E_n)
\]

$(F_1 \dots F_m, E_1 \dots E_n)$ linearly dependent in $V$, by linear algebra.

For $j_1 \in \lbrace 1 \dots n \rbrace$, $E_{j_1}$ linear combination of previous vectors (cf. Exercise B.1(d)) \\
\phantom{ \quad } eliminate $E_{j_1}$ : $(F_1 \dots F_m, E_1 \dots \widehat{E}_{j_1} \dots E_n )$. \\

Repeat, until there are $n-m$ $E$ basis vectors left, labeled $i_1 \dots i_{n-m}$ (hence \textbf{use Exercise B.1(d)} many and enough times, $m$ times)
\[
\Longrightarrow (F_1 \dots F_m, E_{i_1} \dots E_{i_{n-m}} )
\]
By linear algebra, $(F_1 \dots F_m, E_{i_1} \dots E_{i_{n-m}})$ a basis for $V$, linearly independent. The procedure wouldn't have ``overshot'' by a Thm. (see Apostol's \textbf{Calculus} Vol. 2, first few chapters, linear algebra part) \\

$\forall \, v\in V$, $v= a^iF_i  + b^{i_j} E_{i_j}$ with $a^i F_i \in S$.  Then $b^{i_j} E_{i_j} \in T$.  \\
\phantom{ \quad } since $V = S\oplus T$, $T$ complement to $S$

$\Longrightarrow $ given fixed basis of $V$, $(E_1 \dots E_n)$, subspace $S \subseteq V$, $S$ having basis $(F_1 \dots F_m)$, $S$ has complementary subspace in $V$, $T$, \\
\phantom{ \quad } s.t. basis of $T$ is $(E_{i_1} \dots E_{i_{n-m}})$ and $V = S\oplus T$

\end{proof}

\exercisehead{B.13}  Suppose $\exists \, $ linear $T: V \to W $ s.t. $T(E_i) = w_i$, \quad \, $i = 1 \dots n$ \\
Suppose $\exists \, $ linear $T':V \to W$ s.t. $T'(E_i)= w_i$, \quad \, $i = 1 \dots n$

Let $x\in V$, so $x  = x^i E_i$ (the key idea is that with a basis, the vector space is completely determined, vectors in the vector space are spanned by the basis elements)
\[
(T-T')(x) \equiv T(x) - T'(x)  = x^i w_i - x^i w_i = 0
\]
$T(x) = T'(x)$\quad \, $\forall \, x \in V$ \\
so $T=T'$.  $T$ unique.

$T$ exists by construction.
\hrulefill

\textbf{ affine subspace } of $V$ parallel to $S$, linear subspace $S \subseteq V$, $v + S = \lbrace v + w | w \in S \rbrace$, some fixed $v \in V$ \\

\textbf{affine map } $F: V \to W$ if $F(v) = w + Tv$ for some $T: V \to W$, some fixed $w\in W$ \\

\exercisehead{B.16}

Let $a,b \in \mathbb{C}$, $x, y \in F(V)$ \\

Now
\[
F(V) = \lbrace y | y = w + Tv = F(v), \, v \in V, \text{ fixed } w \in W , \text{ some } T \rbrace
\]

\subsubsection{Change of Basis}

\exercisehead{B.22} Suppose $V,W, X$ finite-dim. vector spaces \\
$S:V\to W$, \, $T:W \to X$

\begin{enumerate}
\item[(a)] $\text{rank}S \leq \text{dim}V$ \quad \, $\text{rank}S = \text{dim}V$ iff $S$ injective
\item[(b)] $\text{rank}S \leq \text{dim}W$ \quad \, $\text{rank}S = \text{dim}W$ iff $S$ surjective
\item[(c)] if $\text{dim}V = \text{dim}W$ and $S$ either injective or surjective, then $S$ isomorphism
\item[(d)] $\text{rank}TS \leq \text{rank}S$ \quad \, $\text{rank}TS = \text{rank}S$ iff $\text{im}S \bigcap \text{ker}T = 0$
\item[(e)] $\text{rank}TS \leq \text{rank}T$ \quad \, $\text{rank}TS = \text{rank}T$ iff $\text{im}S + \text{ker}T = W$
\item[(f)] if $S$ isomorphism, then $\text{rank}TS = \text{rank}T$
\item[(g)] if $T$ isomorphism, then $\text{rank}TS = \text{rank}S$
\end{enumerate}

EY : Exercise B.22(d) is useful for showing the chart and atlas of a Grassmannian manifold, found in the More examples, for smooth manifolds.

\begin{proof}
\begin{enumerate}
\item[(a)]
\item[(b)]
\item[(c)]
\item[(d)] Now
\[
\begin{aligned}
  & \text{dim}V = \text{rank}TS + \text{nullity}TS \\
  &  \text{dim}V = \text{rank}S + \text{nullity}S
\end{aligned}
\]
$\text{ker}S \subseteq \text{ker}TS$, clearly, so $\text{nullity}S \leq \text{nullity}TS$
\[
\Longrightarrow \boxed{ \text{rank}TS \leq \text{rank}S }
\]

If $\text{rank}TS = \text{rank}S$, \\
\phantom{ \quad } then $\text{nullity}S = \text{nullity}TS$ \\
\phantom{ \, } Suppose $w \in \text{Im}S \bigcap \text{ker}T$, $w \neq 0$ \\
\phantom{ \quad } Then $\exists \,  v\in S$, s.t. $w = S(v)$ and $T(w)=0$ \\
\phantom{ \quad \, } Then $T(w) = TS(v) =0$.  So $v\in \text{ker}TS$ \\
\phantom{ \quad \quad \, } $v\notin \text{ker}S$ since $w = S(v) \neq 0$ \\
\phantom{ \quad \quad \, } This implies $\text{nullity}TS > \text{nullity}S$.  Contradiction. \\
$\Longrightarrow \text{Im}S \bigcap \text{ker}T =0$ \\

If $\text{Im}S \bigcap \text{ker}T =0$, \\
\phantom{ \quad } Consider $v \in \text{ker}TS$.  Then $TS(v)=0$.  \\
\phantom{ \quad  Consider $v \in \text{ker}TS$}.  Then $S(v)  \in \text{ker}T$ \\
\phantom{ \quad  } $S(v) =0$; otherwise, $S(v) \in \text{Im}S$, contradicting given $\text{Im}S \bigcap \text{ker}T =0$ \\
\phantom{ \quad \quad } $v\in \text{ker}S$ \\

$\text{ker}TS \subseteq \text{ker}S$\\
$\Longrightarrow \text{ker}TS = \text{ker}S$ \\
So $\text{nullity}TS = \text{nullity}S$  \\
$\Longrightarrow \text{rank}TS = \text{rank}S$

\item[(e)]
\item[(f)]
\item[(g)]
\end{enumerate}
\end{proof}

\subsection*{Inner Products and Norms}

\subsubsection*{Norms}

If $V$ real vector space, \\
\phantom{\quad } norm on $V$, $v \mapsto |v| \in \mathbb{R}$ s.t.
\begin{enumerate}
\item[(i)] \textsc{Positivity} $|v| \geq 0$, \, $\forall \, v \in V$, $|v| =0$ iff $v=0$
\item[(ii)] \textsc{Homogeneity} $|cv| = |c| |v|$ \quad \, $\forall \, c \in \mathbb{R}$, $v\in V$
\item[(iii)] \textsc{Triangle Inequality} $|v+w | \leq |v| + |w|$, \, $\forall \, v,w \in V$
\end{enumerate}

2 norms $| \, \cdot \, |_1$, $| \, \cdot \, |_2$ on vector space $V$ equivalent if $\exists \, $ constants $c,C >0$ s.t.
\[
c|v|_1 \leq |v|_2 \leq C|v|_1 \quad \quad \, \forall \, v \in V
\]

\exercisehead{B.49} $\forall \, x \in V$, \\
Consider $B_r(x) = \{ y \in V | |y-x|_2 < r \}$ \\
\phantom{ \quad } Note $y-x \in V$ as $V$ is a vector space

\[
\begin{gathered}
  c|y-x|_1 \leq |y-x|_2 \leq C |y-x|_1 \\
 c|y-x|_1 \leq |y-x_2| < r \\
 |y-x|_1 < \frac{r}{c}
\end{gathered}
\]
Now suppose $S \subseteq V$ open in $| \, \cdot \, |_2$ \\
\phantom{ \quad } But $S$ also open in $| \, \cdot \, |_1$ as $\exists \, \frac{r}{c'} >0$ s.t. $B_{\frac{r}{c'}}(x) \subseteq S$ for the exact same pts. as $S$ \\
so $| \, \cdot \, |_1, | \, \cdot \, |_2$ equivalent norms yield the same metric topology.

EY : 20141220

\subsection*{Direct Products and Direct Sums}


\section{Introduction}
	The aim of this project is to familiarize with different methods to solve an Asymmetric Travelling Salesman Problem.
	\begin{itemize}
		\item Exact method that gives the optimal solution. This is implemented with the CPLEX API.
		\item Meta heuristic methods: local search and tabu search.
	\end{itemize}

	\subsection{Problem}
		The combinatorial optimization problem was:
		\begin{quote}
				A company produces boards with holes used to build electric frames.  Boards are positioned over a machines and a drill moves over the board, stops at the desired positions and makes the holes.  Once a board is drilled, a new board is positioned and the process is iterated many times.  Given the position of the holes on the board, the company asks us to determine the hole sequence that minimizes the total drilling time, taking into account that the time needed for making an hole is the same and constant for all the holes.
		\end{quote}

		This problem can be modelled as an Asymmetric Travelling Salesman Problem. The salesman is represent by the drill and cities by the holes in the board.

	\subsection{Attachments}
		I attach to this report the folders:
		\begin{itemize}
			\item \verb|RealInstances|: it contains the \verb|.dat| and the information pdf of the real instances tested;
			\item \verb|Results|: it contains the pdf of all results taken, the most interesting data is included in this report. It should not be necessary to consult it, I attached them for completeness;
			\item \verb|Programs|: it contains the source code and executables of the two programs require by the assignment.
		\end{itemize}

	\subsection{System of testing}
		All the computation are be made on the following system:
		\begin{itemize}
			\item OS: Ubuntu 16.04 LTS (64 bit);
			\item Hardware: RAM 8 GB, CPU Intel Core i5-7500 3.40 GHz.
		\end{itemize}

	\subsection{Document structure}
		The first part of reports show how the exact method, the local search and the tabu search were been implemented with C++ code. In the second part it describes the test done and it shows the results obtained. There are two test sections, one with instances originated from random generator and one with real instances generated from real gerber (the standard format for represent PBCs) files.


\section{Configuring a Neural Network}

\coderef{Nn/NeuralNetwork.hh}

Squirrel supports a variety of different neural network architectures, including CNNs and recurrent neural networks. In the following, details on configuring neural networks in Squirrel are provided.

In order to set up a neural network, first its architecture needs to be defined. Therefore, two major components need to be specified: Connections and layers. Connections are initially just a list of names that are later filled with meaning. So, configuring a network with two layers requires to define the connections between the layers first:

\begin{verbatim}
    [neural-network]
    connections = connection-0-1, connection-1-2

    [neural-network.connection-0-1]
    from = network-input
    to   = layer-1
    type = weight-connection

    [neural-network.connection-1-2]
    from = layer-1
    to   = layer-2
    type = weight-connection
\end{verbatim}

For each of the specified connections, the source and destination layer are specified, also by arbitrary names, here \texttt{layer-1} and \texttt{layer-2}. The first connection usually has the \texttt{network-input} as source and forwards it to the first layer. There are different types of connections (see below). The \texttt{weight-connection} simply multiplies a weight matrix to the input. The layer types still need to be added to the configuration:

\begin{verbatim}
    [neural-network.layer-1]
    type = rectified
    number-of-units = 128

    [neural-network.layer-2]
    type = softmax
    number-of-units = 10
\end{verbatim}

A list of possible layers and their description can be found below. Be aware of the \texttt{connections} list as this specifies the topology of the network. Particularly when building network with recurrent skip connections, the order of the list is important.

\subsection{Layer Types}

\subsubsection*{Layer Base Class}
\coderef{Nn/Layer.hh}

All layers inherit from a base layer class. Parameters accessible for all layers are:
\begin{itemize}
    \item \texttt{number-of-units} the number of units. Default is $ 0 $, mandatory to be specified unless the connection leading to the layer is a convolutional connection.
    \item \texttt{dropout-probability} amount of units that are randomly set to zero for dropout (default: 0.0).
    \item \texttt{use-bias} add a bias for this layer if true, else omit bias (default: true).
    \item \texttt{is-bias-trainable} bias is trained if true or left unchanged if false (default: true).
    \item \texttt{bias-initialization} Initialization if bias can not be loaded from file. Possible options: \texttt{random, zero} (default: \texttt{random}).
    \item \texttt{random-bias-min} and \texttt{random-bias-max} min and max value for the random (uniform) initialization (default: $ -0.1 $ and $ 0.1 $).
    \item \texttt{learning-rate-factor} train this layer stronger (factor $ > 1 $) or weaker (factor $ < 1 $) than other layers (default: 1.0).
\end{itemize}

\subsubsection*{Identity Layer}

Does not change the layer units at all (apart from adding a bias if \texttt{use-bias} is true, which is done by all layers).

\subsubsection*{Sigmoid Layer}
\coderef{Nn/ActivationLayer.hh}

Applies the sigmoid function $ \sigma(x) = \frac{1}{1+\exp(-x)} $ to each input unit. Use by setting \texttt{type = sigmoid}.

\subsubsection*{Tanh Layer}
\coderef{Nn/ActivationLayer.hh}

Applies the $ \mathrm{tanh} $ to each input unit. Use by setting \texttt{type = tanh}.

\subsubsection*{Rectified Layer}
\coderef{Nn/ActivationLayer.hh}

Applies the rectifier $ \mathrm{relu}(x) = \max(0, x) $ to each input unit. Use by setting \texttt{type = rectified}.

\subsubsection*{Average-Pooling and Max-Pooling Layer}
\coderef{Nn/MultiPortLayer.hh}

Applies average-pooling/max-pooling the the input units. Usually used for CNNs. Use by setting \texttt{type = avg-pooling} or \texttt{type = max-pooling}. Parameters are
\begin{itemize}
    \item \texttt{grid-size} the region to apply pooling to, \eg 3 for a $ 3 \times 3 $ grid. Default: 2.
    \item \texttt{stride} spatial stride of the pooling region. Default: 2.
\end{itemize}

\subsubsection*{Batch-Normalization Layer}
\coderef{Nn/MultiPortLayer.hh}

Applies batch normalization to the input units. Use by setting \texttt{type = batch-normalization}. Parameters are
\begin{itemize}
    \item \texttt{is-spatial} apply batch normalization spatially if true, else apply batch normalization to each input unit. Default: true.
    \item \texttt{is-inference} use inference mode of batch-normalization (load mean and variance from file) if true, else use training mode (estimate running mean and variance)
\end{itemize}

\subsubsection*{Gated Recurrent Unit Layer}
\coderef{Nn/MultiPortLayer.hh}

Implementation of gated recurrent units. Use by setting \texttt{type = gated-recurrent-unit}. Note that this layer is inherently recurrent and does not need the specification of any recurrent connection (\ie a connection with same source and target layer).

\subsubsection*{Softmax Layer}
\coderef{Nn/ActivationLayer.hh}

Applies the $ \mathrm{softmax} $ function to the input units. Although usually used as output layer, it can also be used as an internal layer in Squirrel. Use by setting \texttt{type = softmax}.

\subsubsection*{Clipped Layer}
\coderef{Nn/ActivationLayer.hh}

A generalization of the rectified layer. Clips each input unit at a lower and upper bound. Use by setting \texttt{type = clipped}. Parameters are
\begin{itemize}
    \item \texttt{left-threshold} lower bound for clipping (default: 0.0)
    \item \texttt{right-threshold} upper bound for clipping (default: 1.0)
\end{itemize}
If the lower bound is zero and the upper bound is infinity, the layer is a rectifier.

\subsubsection*{L2-Normalization Layer}
\coderef{Nn/ActivationLayer.hh}

Applies $ \ell_2 $-normalization to the input, \ie divides each input unit by the $ \ell_2 $ norm of all input units. Use by setting \texttt{type = l2-normalization}.

\subsubsection*{Power-Normalization Layer}
\coderef{Nn/ActivationLayer.hh}

Applies power-normalization to each input unit, \ie applies the function $ \mathrm{power}(x) = \mathrm{sign}(x) \cdot x^p $. Use by setting \texttt{type = power-normalization}.
Set the value $ p $ using the parameter \texttt{power} (default: 0.5).

\subsubsection*{Sequence length normalization layer}
\coderef{Nn/ActivationLayer.hh}

Only useful for recurrent networks. Divides the input units at the last timeframe of the sequence by the sequence length. Use by setting \texttt{type = sequence-length-normalization}.

\subsubsection*{Temporal Reversion Layer}
\coderef{Nn/ActivationLayer.hh}

Only useful for recurrent networks. Reverts the temporal order of the input units. Can be used to define bi-directional recurrent networks. Use by setting \texttt{type = temporal-reversion}.

\subsection{Connection Types}

\subsubsection*{Connection Base Class}

All connections inherit from a base connection class. Parameters accessible for all connections are
\begin{itemize}
    \item \texttt{learning-rate-factor} train this connection stronger (factor $ > 1 $) or weaker (factor $ < 1 $) than other connections (default: 1.0).
    \item \texttt{is-recurrent} force the connection to be treated as recurrent even if it is configured as a standard forward from \texttt{layer-from} to \texttt{layer-to}. If true, the input to \texttt{layer-to} at time $ t $ is a linear transformation (matrix multiplicaiton or convolution) of the output of \texttt{layer-from} at time $ t - 1 $. If false, the input to \texttt{layer-to} at time $ t $ is a linear transformation (matrix multiplication or convolution) of the output of \texttt{layer-from} at time $ t $ (default: false).
\end{itemize}

\subsubsection*{Weight Connection}

Causes the input of \texttt{layer-to} to be the output of \texttt{layer-from} multiplied by the connections weight matrix. Use by setting \texttt{type = weight-connection}. This type of connection is also used if no \texttt{type} value has been set. Parameters are
\begin{itemize}
    \item \texttt{is-trainable} train the weights of this connection if true, else leave them unchanged during training (default: true)
    \item \texttt{weight-initialization} if the weights can not be loaded from a file, initialize them by one of these: \texttt{random, zero, identity, glorot}. \texttt{random} is a unifrom random initialization, \texttt{zero} initializes with zeros, \texttt{identity} initializes with the identity matrix (if matrix is not square, all remaining columns/rows are initialized with zeros), and \texttt{glorot} initializes based on a strategy proposed by Glorot et.\ al.\ (default: \texttt{random})
    \item \texttt{random-weight-min} and \texttt{random-weight-max} the interval to uniformly sample values from in case of \texttt{random} initialization (default: $ -0.1 $ and $ 0.1 $).
\end{itemize}

\subsubsection*{Convolutional Connection}

Causes the input of \texttt{layer-to} to be a convolution of the output of \texttt{layer-from} with the kernels of this connections. Use by setting \texttt{type = convolutional-connection}. Parameters are the same as for the weight connections plus
\begin{itemize}
    \item \texttt{kernel-height} and \texttt{kernel-width} the height and width of the kernels (default: $ 3 $ for both)
    \item \texttt{dest-channels} the number of destination channels/feature maps (mandatory to be set)
    \item \texttt{stride-x} and \texttt{stride-y} the stride in $ x $- and $ y $-direction (default: $ 1 $ for both)
\end{itemize}

\subsubsection*{Plain Connection}

Does not apply any linear transformation, \ie input of \texttt{layer-to} is equal to output of \texttt{layer-from}. Requires \texttt{layer-from} and \texttt{layer-to} to have the same number of units.

\section{Training Neural Network}
\coderef{examples/mnist/config/training.config}

\subsection{General Settings}

Neural network training requires some general settings. These are:
\begin{itemize}
    \item \texttt{source-type} and \texttt{target-type}, \ie if they are single vectors/images/labels or sequences of vectors/images/labels. Use \texttt{single} for the first, \texttt{sequence} for the latter (default: \texttt{single}).
    \item \texttt{batch-size} the batch size to use (default: 1).
    \item \texttt{trainer} the kind of trainer to use (\texttt{feed-forward-trainer, rnn-trainer}).
    \item \texttt{training-criterion} the criterion to use for training, \eg \texttt{cross-entropy, squared-error}, see \textit{Nn/TrainingCriteria.hh} for more options.
\end{itemize}

\subsection{Loading Data}

In order to provide data for the network, the \textit{Features} module is used. Provide the input and target data by specifying an \texttt{aligend-feature-reader}:
\begin{verbatim}
    [features.aligned-feature-reader]
    feature-cache = <input feature file>
    target-cache  = <target label/feature file>
\end{verbatim}
For details \eg on how to shuffle data, see Chapter~\ref{ch:features}. If you only want to forward data without using a target-cache, use the \texttt{feature-reader} instead of the \texttt{aligned-feature-reader}. As the network needs some information on the feature dimension, also specify the following parameters:
\begin{verbatim}
    [neural-network]
    input-dimension = <input-feature-dimension>
    source-width    = <width of input images/videos>
    source-height   = <height of input images/videos>
    source-channels = <number of input channels (e.g. 3 for rgb, 1 for gray)>
\end{verbatim}

Note that the last three parameters are only required if the input is images or videos.

\section{Further Configuration}

We refer to the \texttt{config} files in the \texttt{examples} directory for more example configuration of both, CNNs and recurrent neural networks.


\section{Evaluation}\label{sec:evaluation:krap}

Evaluating \krap{} requires an understanding of how well it classifies the \spec{} of an \isol{}.
There are a few areas of focus that we have when interpreting the results of \krap{}:
\begin{itemize}
\item What size $k$ achieves the best results?
\item What size $\alpha$ achieves the best results?
\item Which metric resolution algorithm achieves the best results?
\end{itemize}
Indeed we can define ``best'' in many ways, but we choose to look at two metrics, recall and precision, and a combination of the two, the $F$-measure. The metrics look at the accuracy of the classification on the object and the object on the classification respectively, while \fmeasure{} hopes to represent a balance between the two.
We test \krap{} by performing cross validation with holdout.

\subsection{Cross Validation with Holdout}
\index{cross validation with holdout}
To gauge the effectiveness of \krap{} at classifying the \spec{} of an \isol{}, we cross-validated against the library by separately holding out each \isol{} in CPLOP from CPLOP, classifying it against CPLOP, and verifying whether it is correct.
Since each \isol{} in CPLOP has the correct \spec{}, we know whether a classification is correct or not.

\subsection{Recall}
In our study, recall tracks how well we are able to discover all isolates from a given category, i.e. with a given host species.
Given a category (\spec{} name), the recall for that host species is the percentage of isolates taken from this host species that have been properly identified.
For example, if our database had 100 cat isolates, and 74 of them were classified by our method as having come from a cat, the recall would be 74\%.
In this study, we compute both overall recall (what percentage of \isols{} were classified as their proper \spec{} label) as well as \spec{}-level recall (what percentage of isolates that came from dogs/humans/sheep/etc. were classified
as their proper label).

\subsection{Precision}
Precision tracks how well our method avoids misclassification errors.
Given a category and a list of isolates our method classified as belonging to it, the precision of the method on the
category is the percent of isolates from the list that has the correct label.
For example, if our method returned 100 isolates labelled ``Dog'' of which 77 isolates really did come from dogs, the precision of the method is 77\%. As with recall, we compute both overall precision, as well as the precision for each category/species label.

\subsection{\textit{F}-Measure}
The \fmeasure{}, $F_1$, is the \textit{harmonic mean} of the precision, $P$ and the recall, $R$:
\begin{equation*}
    F_1
    =
    \frac{2}{\frac{1}{P}
    +
    \frac{1}{R}}
    = 2\cdot
    \frac{P\cdot R}
    {P + R}
\end{equation*}
While we prefer maximizing this value, a value near 0.5 means we are doing well.