\chapter{Introduction} \label{sec:intro}

Large databases of images exist in various fields. Indexing and searching these using traditional textual descriptors is generally not effective \cite[p.657]{forsyth2012}. Visual search offers an alternative to this, by allowing a user to search an image collection using an image as the query, with the aim being to return similar images. This is achieved by generating and comparing numeric, vector based, image descriptors which can be treated similar to words.

This report presents the implementation of such a system in \gls{matlab}. Several different methodologies will be considered, and their results compared and contrasted.

In order to test the system, the Microsoft Research Cambridge, object recognition image database, version 2.0 will be used. This is a freely available image database which is split into 20 rows, with each row containing similar images \cite{criminisi2004}.

For the purposes of evaluating this system, similarity will be defined using the 20 categories which the image set is sorted into. This is not ideal since the categories encode level 2 meaning, whilst the descriptors used will perform level 1 operations \cite{eakins1998}. For this reason, it is not expected that the system will have exceptional performance.

In order to calculate the distance between descriptors, the $L_2$ norm is used. This distance measure, also known as Euclidean distance, returns the straight line distance between two points in a vector space. It is discussed in greater detail, and performance evaluated against other distance measures, in Appendix \ref{app:dist-measures}.

\section{System Design} \label{sec:sys-design}
A common framework was used to apply generate descriptors from all of the images, compare them, and display results. This framework uses a \gls{matlab} function handle to input both the descriptor and distance measure. This method allows the core code-body to be generic, so that in order to run a different test only the external descriptor and/or distance measure functions need to be changed.

A block diagram of the operation of the system is shown in Figure \ref{fig:sys-operation}. In this figure shaded blocks represent functions necessary for the system's core operation, white blocks represent auxillary functions, and dashed blocks represent functions or data which may be omitted. The arrows show the transitions of data.

\begin{figure}[ht]
	\centering
	\tikzstyle{generalRect} = [rectangle, rounded corners, minimum width=2cm, minimum height=1cm, text centered, draw=black, fill=black!40]
	\tikzstyle{arrow} = [thick,->,>=stealth]
	\begin{tikzpicture}[node distance=1cm]
		\node (compute) [generalRect] {Compute Descriptors};
		\node (descriptor) [generalRect, right = of compute] {Descriptor Function};
		\node (pca) [generalRect, fill=white, above = 0.5cm of descriptor, dashed] {PCA Function};
		\node (compare) [generalRect, below= of compute] {Compare Descriptors};
		\node (query) [generalRect, fill=white, left = of compare, dashed] {Query Image};
		\node (dist) [generalRect, right = of compare] {Distance Measure};
		\node (display) [generalRect, below= of compare, fill=white] {Display Results};
		\node (plotPR) [generalRect, below= of display, fill=white] {\shortstack{Plot Precision-Recall Graph}};

		\draw [arrow] (compute) -- (compare);
		\draw [arrow, <->] (compute) -- (descriptor);
		\draw [arrow, <->, dashed] (pca.west) -- (compute.east);
		\draw [arrow] (compare) -- (display);
		\draw [arrow, <->] (compare) -- (dist);
		\draw [arrow, dashed] (query) -- (compare);
		\draw [arrow] (display) -- (plotPR);
	\end{tikzpicture}
	\caption{System design}
	\label{fig:sys-operation}
\end{figure}

\chapter{Description Techniques} \label{sec:techniques-implemented}
This Chapter discusses the different descriptor generation techniques implemented, along with the theory of their operation and the difficulties which were encountered whilst implementing them. Details of how each technique was tested is given in Appendix \ref{app:test-methodology}.

\section{Global Colour Histogram} \label{sec:color-histogram}
A global colour histogram quantises the RGB space into a series of bins. There are $Q$ bins for each dimension, and therefore $Q^3$ bins for the total image. Each pixel in the image is then sorted into the appropriate bin, and the sum of the bin totals is normalised to 1. This vector histogram, of length $Q^3$, can therefore be used to describe the image. A diagrammatic representation of this technique is shown in Figure \ref{fig:rgb-hist-diag}. This Figure shows the RGB space quantised with a quantisation level of $Q=2$, yielding $2^2=4$ bins. $P_1$, $P_2$, and $P_3$ represent points in the RGB space which will be quantised to their containing boxes.

\begin{figure}[ht]
	\centering
	\tdplotsetmaincoords{65}{135}
	\newcommand{\histbox}[3]
	{
		\draw[dashed, color=#3] (#1) -- (#2x);
		\draw[dashed, color=#3] (#1) -- (#2y);
		\draw[dashed, color=#3] (#1) -- (#2z);
		\draw[dashed, color=#3] (#2x) -- (#2xy);
		\draw[dashed, color=#3] (#2y) -- (#2xy);
		\draw[dashed, color=#3] (#2x) -- (#2xz);
		\draw[dashed, color=#3] (#2z) -- (#2xz);
		\draw[dashed, color=#3] (#2y) -- (#2yz);
		\draw[dashed, color=#3] (#2z) -- (#2yz);
		\draw[dashed, color=#3] (#2xy) -- (#2);
		\draw[dashed, color=#3] (#2xz) -- (#2);
		\draw[dashed, color=#3] (#2yz) -- (#2);
	}
	\begin{tikzpicture}[scale=2,tdplot_main_coords]
	\coordinate (O) at (0,0,0);
	\tdplotgetpolarcoords{1}{1}{1}
	\tdplotsetcoord{A}{1.73205}{\tdplotrestheta}{\tdplotresphi}
	\tdplotgetpolarcoords{2}{1}{1}
	\tdplotsetcoord{B}{2.44949}{\tdplotrestheta}{\tdplotresphi}
	\tdplotgetpolarcoords{1}{2}{1}
	\tdplotsetcoord{C}{2.44949}{\tdplotrestheta}{\tdplotresphi}
	\tdplotgetpolarcoords{1}{1}{2}
	\tdplotsetcoord{D}{2.44949}{\tdplotrestheta}{\tdplotresphi}

	\tdplotgetpolarcoords{2}{2}{1}
	\tdplotsetcoord{E}{3}{\tdplotrestheta}{\tdplotresphi}
	\tdplotgetpolarcoords{2}{1}{2}
	\tdplotsetcoord{F}{3}{\tdplotrestheta}{\tdplotresphi}
	\tdplotgetpolarcoords{1}{2}{2}
	\tdplotsetcoord{G}{3}{\tdplotrestheta}{\tdplotresphi}

	\tdplotgetpolarcoords{2}{2}{2}
	\tdplotsetcoord{H}{3.464102}{\tdplotrestheta}{\tdplotresphi}

	\draw[thick,->] (0,0,0) -- (2.25,0,0) node[anchor=north east]{R};
	\draw[thick,->] (0,0,0) -- (0,2.25,0) node[anchor=north west]{G};
	\draw[thick,->] (0,0,0) -- (0,0,2.25) node[anchor=south]{B};

	\draw plot [mark=*, mark size=2, mark options={color={rgb:red,1.5;green,0.5;blue,0.5}}] coordinates{(1.5,0.5,0.5)};
	\draw plot [mark=*, mark size=2, mark options={color={rgb:red,0.3;green,1.6;blue,1.7}}] coordinates{(0.3,1.6,1.7)};
	\draw plot [mark=*, mark size=2, mark options={color={rgb:red,1.8;green,1.8;blue,0.2}}] coordinates{(1.8,1.8,0.2)};
	\histbox{O}{A}{blue}
	\histbox{O}{B}{blue}
	\histbox{O}{C}{blue}
	\histbox{O}{D}{blue}
	\histbox{O}{E}{blue}
	\histbox{O}{F}{blue}
	\histbox{O}{G}{blue}
	\histbox{O}{H}{blue}

	\node[left] at (1.5,0.5,0.5) {$P_1$};
	\node[below] at(0.3,1.6,1.7) {$P_2$};
	\node[right] at(1.8,1.8,0.2) {$P_3$};

	\end{tikzpicture}
	\caption{Visualisation of RGB histogram}
	\label{fig:rgb-hist-diag}
\end{figure}

The implementation of this function was relatively straightforward, since example code for the function had been provided \cite{collomosse2016}. The only modifications made to the code were to explicitly state the histogram bin edges. This was necessary to provide a robust descriptor since the provided code normalised the histogram to the range of the input. Therefore an image containing only values in one area of the RGB space, would appear similar to an image with uniform distribution. The function is implemented as \texttt{H =  vs\_compute\_rgb\_histogram(img, Q)} where \texttt{H} is the descriptor returned, \texttt{img} is the image input, and \texttt{Q} is the quantisation level for each image dimension. The only variable parameter is therefore $Q$.

\section{Gridding} \label{sec:gridding-concatenation}
Gridding of features allows descriptors to apply locally to parts of an image. The implementation of this is fairly simple in \gls{matlab}. The image can be split using the native array indexing of \gls{matlab} to select a subset of the image array,  and the descriptor function can then be applied to each of these sub images. The resultant descriptors are then concatenated into a matrix.

Gridding is implemented as \texttt{F = vs\_grid(img, h\_level, v\_level, compute\_function)}, where \texttt{img} is the image input, \texttt{h\_level} is the horizontal quantisation level, \texttt{v\_level} is the vertical quantisation level and \texttt{compute\_function} is the descriptor generation function. This configuration therefore gives two independent variables, \texttt{h\_level} and \texttt{v\_level}.

However, since most of the images in the dataset have an aspect ratio of approximately 1.5:1, it makes sense to set the vertical quantisation level to 1.5 times the horizontal quantisation level. Therefore the overall quantisation level, $Q$ can be defined, where \texttt{h\_level = ceil(1.5*Q)}, and \texttt{v\_level = Q}.

\section{Edge Orientation Histogram} \label{sec:texture-histogram}
Texture is an important concept in computer vision, but is difficult to define. Texture loosely corresponds to a repeating pattern of edge characteristics, examples of which would include grass pebbles, and hair \cite[p. 194]{forsyth2012}.

One method of creating a texture based descriptor is to use an edge orientation histogram. This works as follows:
\begin{enumerate}
	\item Edges are detected in the image.
	\item An estimation of the orientation of each edge is calculated.
	\item A histogram is created of the edge orientations.
\end{enumerate}

This process is implemented using two functions: an edge orientation function, and a histogram generation function. The edge orientation function returns a two dimensional matrix where each element contains a value corresponding to an estimate of the edge orientation at that location. This estimate is in the range $-\pi$ to $+\pi$, but is normalised to $0$ to $1$. Weak edges are set to a normalised value of 0.5, corresponding to \SI{0}{\radian}.

In order to perform the edge detection, and edge angle estimation, the Sobel operator is used \cite{sobel2015}. The Sobel operator is formed of two kernels which are convolved with input image in order to perform edge detection in the x and y directions respectively. The kernels are defined in Equation \ref{eq:sobel-kernels}.

\begin{equation}
	K_x =
	\begin{bmatrix}
	1 & 0 & -1 \\
	2 & 0 & -2 \\
	1 & 0 & -1 \\
	\end{bmatrix}
	K_y =
	\begin{bmatrix}
	1 & 2 & 1 \\
	0 & 0 & 0 \\
	-1 & -2 & -1 \\
	\end{bmatrix}
	\label{eq:sobel-kernels}
\end{equation}

The edge magnitude and angle are calculated as described in Equations \ref{eq:sobel-mag} and \ref{eq:sobel-ang} respectively. It should be noted that the operators in these equations apply on an element-by-element basis, not to the matrices as a whole. In addition, the $K_y$ value is inverted in order to allow the performance to match the \gls{matlab} library function \texttt{imgradient}, this allows for easier testing. The pixels with edge magnitudes above a certain threshold are passed to the histogram generation function.
\begin{equation}
\text{Mag} = \sqrt{{K_x}^2 + {K_y}^2}
\label{eq:sobel-mag}
\end{equation}
\begin{equation}
\theta = \atantwo(-K_y, K_x)
\label{eq:sobel-ang}
\end{equation}

The edge orientation function is implemented as \texttt{F = vs\_edge\_detect(img, compute\_function, strength)} where \texttt{F} is the descriptor returned, \texttt{img} is the image input, \texttt{compute\_function} is the histogram generation function, and \texttt{strength} is the minimum edge strength to consider (range 0--1).

The histogram generation function is \texttt{H = vs\_compute\_histogram(img, Q)} where \texttt{H} is the histogram returned, \texttt{img} is the image input and \texttt{Q} is the quantisation factor. When \texttt{vs\_compute\_histogram} is used as the \texttt{compute\_function} in \texttt{vs\_edge\_detect}, the system can produce an edge orientation histogram, with two independent parameters. These parameters determine the minimum strength edge to consider, as well as the quantisation level of the histogram. These parameters will be referred to as $E$ and $Q$ respectively.

Additionally, since texture is an inherently local feature, the image will be gridded. The descriptor function handle is therefore : \texttt{@(x)vs\_grid(x, v\_level, h\_level, @(x)vs\_edge\_detect(x, @(x)vs\_compute\_histogram(x,Q),E))}.

\section{Concatenation of Descriptors} \label{sec:conc-desc}
Descriptors focusing upon an individual aspect of an image, such as colour or texture have already been discussed, as well as how the can be applied to multiple sections of an image rather than globally.

It stands to reason therefore that multiple descriptors could be combined to produce a descriptor of even greater effectiveness, such as a combination of an RGB histogram and a texture histogram. This is possible in \gls{matlab} using the native array indexing interface, but it is clearer to use the \texttt{horzcat} function.

A combined descriptor can therefore be invoked using the function \texttt{@(x)horzcat( func\_1(x) .* M, func\_2(x))} where \texttt{texture\_func(x)} and \texttt{color\_func(x)} are function handles for two descriptor functions with a single parameter, \texttt{x} for the input image. The only other parameter in this function is \texttt{M}. \texttt{M} defines a weighting factor between the two descriptors, and can be used to give a greater weighting to one over the other. This works by using a multiplier value to either spread all of the values in the descriptor further apart, or bring them closer together.

\chapter{Experimental Results} \label{sec:results}

In order to evaluate the performance of each descriptor, two test images have been chosen from the data set: 9\_23\_s and 13\_1\_s. These images have been chosen because they represent two contrasting examples of what could be presented to the algorithm. Image 9\_23\_s shows a sheep in a field. This image has two strongly dominant colours: the green of the grass, and the cream of the sheep's wool, as well as two regions of largely uniform texture: the sheep's body, and the grass. Image 13\_1\_s strongly contrasts this. This image shows a collection of differently coloured books on a shelf. The image therefore contains a large variety of colours distributed around the image. In addition to this the transitions between the books, as well as the text on the spine of the books means that a large amount of varied texture is distributed across the image.

The fact that these images contrast in many different ways means that they should be ideal to evaluate the performance of each descriptor in a fair way, as well as determine the optimum parameters for each descriptor function. The performance of the descriptors across the wider images in the dataset will then be presented in Section \ref{sec:results}.

In order to evaluate performance, a precision-recall characteristic is plotted for each test. This characteristic is a plot showing the proportion of the returned results which have been relevant (precision), vs the number of relevant results returned (recall). The ideal precision-recall characteristic would be a horizontal line at precision = 1, a system which produced this characteristic would return all the relevant images, and none of the irrelevant images. Most practical systems produce a characteristic which slopes from the top left to the bottom right, this occurs because systems first return the results they are most sure about. These results are likely to be correct, and so precision will be high, but since not many results have yet been returned, recall will be low. As the system returns an increasing number of results, an increasing number of irrelevant results will be returned also, causing precision to decrease as recall increases.

\section{Global Colour Histogram} \label{sec:global-colour-histogram-results}
The global colour descriptor has one parameter the quantisation level, $Q$, which produces a descriptor of length $Q^3$, as discussed in Section \ref{sec:color-histogram}.

In order to analyse the effect of quantisation level on performance, results will be generated for various values of $Q$, and the performance of each image analysed for each test value. The expected result is that higher values of $Q$ will produce better results, up to a certain point, after which performance will decrease. This should happen because at very high quantisation levels values, there are too many possible bins, and as such nominally similar colours will be sorted into different bins. Since each bin is a different dimension in the descriptor, they will not therefore be considered similar by the distance measure function.

\begin{figure}[ht]
	\begin{minipage}[]{0.3\linewidth}
		\centering
		\includegraphics[width = 0.9\linewidth]{figs/msrc/9_23_s}
		\subcaption{Query image}
	\end{minipage}
	\begin{minipage}[]{0.7\linewidth}
		\centering
		\begin{tikzpicture}
		\prplot{data/global_colour_hist/9_23_s/pr_2.txt}{Q=2}
		\prplotadd{data/global_colour_hist/9_23_s/pr_3.txt}{Q=3}
		\prplotadd{data/global_colour_hist/9_23_s/pr_4.txt}{Q=4}
		%\prplotadd{data/global_colour_hist/9_23_s/pr_5.txt}{Q=5}
		\prplotadd{data/global_colour_hist/9_23_s/pr_6.txt}{Q=6}
		%\prplotadd{data/global_colour_hist/9_23_s/pr_7.txt}{Q=7}
		\prplotadd{data/global_colour_hist/9_23_s/pr_8.txt}{Q=8}
		%\prplotadd{data/global_colour_hist/9_23_s/pr_9.txt}{Q=9}
		\prplotadd{data/global_colour_hist/9_23_s/pr_10.txt}{Q=10}
		\prplotclose
		\end{tikzpicture}
		\subcaption{Resultant graph}
	\end{minipage}
	\caption{Global colour histogram results for query image 9\_23\_s}
	\label{fig:colour-hist-sheep}
\end{figure}

\begin{figure}[ht]
	\begin{minipage}[]{0.3\linewidth}
		\centering
		\includegraphics[width = 0.9\linewidth]{figs/msrc/13_1_s}
		\subcaption{Query image}
	\end{minipage}
	\begin{minipage}[]{0.7\linewidth}
		\centering
		\begin{tikzpicture}
		\prplot{data/global_colour_hist/13_1_s/pr_2.txt}{Q=2}
		\prplotadd{data/global_colour_hist/13_1_s/pr_3.txt}{Q=3}
		\prplotadd{data/global_colour_hist/13_1_s/pr_4.txt}{Q=4}
		%\prplotadd{data/global_colour_hist/13_1_s/pr_5.txt}{Q=5}
		\prplotadd{data/global_colour_hist/13_1_s/pr_6.txt}{Q=6}
		%\prplotadd{data/global_colour_hist/13_1_s/pr_7.txt}{Q=$}
		\prplotadd{data/global_colour_hist/13_1_s/pr_8.txt}{Q=8}
		%\prplotadd{data/global_colour_hist/13_1_s/pr_9.txt}{Q=9}
		\prplotadd{data/global_colour_hist/13_1_s/pr_10.txt}{Q=10}
		\prplotclose
		\end{tikzpicture}
		\subcaption{Resultant graph}
	\end{minipage}
	\caption{Global colour histogram results for query image 13\_1\_s}
	\label{fig:colour-hist-books}
\end{figure}

The results shown in Figures \ref{fig:colour-hist-sheep} and \ref{fig:colour-hist-books} match expectations suggesting that a very low quantisation value does not deliver good results, yet performance also drops off at higher quantisation levels. Both test images suggest that a quantisation level of $Q=4$, yielding $64$ bins, gives the best result, since the line joining these data points is closest to the top right hand corner.

The results also show that query image 13\_1\_s performs better than 9\_23\_s. The reason for this is likely due to the fact that 13\_1\_s has a much more varied colour distribution, allowing it to have a much more unique histogram signature. This means that similar images can be better matched. In contrast to this, the green field will dominate the histogram of image 9\_23\_s meaning the system may confuse it with other images containing a green background, such as those of other farm animals in a field.

\FloatBarrier
\section{Gridding} \label{sec:gridding-results}

As discussed in Section \ref{sec:gridding-concatenation}, the independent parameters for the gridding function can be reduced to a single quantisation factor $Q$. In a similar manner to that of Section \ref{sec:global-colour-histogram-results} the optimum quantisation factor will be determined by computing descriptors for various values of $Q$ and then, using the two test images, determining which has best results.

The same effect of increasing Quantisation factor as experienced with the global colour histogram is expected. Increasing the value should increase performance up to a maximum, at this point the performance should decrease with increasing quantisation factor. One reason for this is that tighter grids mean that small movements in the object could potentially result in the image being quantised in a very different way, meaning visually similar images would be distant from each other in the quantised space.

\begin{figure}[ht]
	\begin{minipage}[]{0.3\linewidth}
		\centering
		\includegraphics[width = 0.9\linewidth]{figs/msrc/9_23_s}
		\subcaption{Query image}
		\label{fig:grid-color-sheep-img}
	\end{minipage}
	\begin{minipage}[]{0.7\linewidth}
		\centering
		\begin{tikzpicture}
		\prplot{data/gridding/color_hist/9_23_s/pr_1.txt}{Q=1}
		\prplotadd{data/gridding/color_hist/9_23_s/pr_2.txt}{Q=2}
		\prplotadd{data/gridding/color_hist/9_23_s/pr_3.txt}{Q=3}
		\prplotadd{data/gridding/color_hist/9_23_s/pr_4.txt}{Q=4}
		\prplotadd{data/gridding/color_hist/9_23_s/pr_8.txt}{Q=8}
		\prplotadd{data/gridding/color_hist/9_23_s/pr_15.txt}{Q=15}
		\prplotadd{data/global_colour_hist/9_23_s/pr_4.txt}{No grid}

		\prplotclose
		\end{tikzpicture}
		\subcaption{Resultant graph}
		\label{fig:grid-color-sheep-graph}
	\end{minipage}
	\caption{Gridded global colour histogram results for query image 9\_23\_s}
	\label{fig:grid-color-sheep}
\end{figure}

\begin{figure}[ht]
	\begin{minipage}[]{0.3\linewidth}
		\centering
		\includegraphics[width = 0.9\linewidth]{figs/msrc/13_1_s}
		\subcaption{Query image}
		\label{fig:grid-color-books-img}
	\end{minipage}
	\begin{minipage}[]{0.7\linewidth}
		\centering
		\begin{tikzpicture}
		\prplot{data/gridding/color_hist/13_1_s/pr_1.txt}{Q=1}
		\prplotadd{data/gridding/color_hist/13_1_s/pr_2.txt}{Q=2}
		\prplotadd{data/gridding/color_hist/13_1_s/pr_3.txt}{Q=3}
		\prplotadd{data/gridding/color_hist/13_1_s/pr_4.txt}{Q=4}
		\prplotadd{data/gridding/color_hist/13_1_s/pr_8.txt}{Q=8}
		\prplotadd{data/gridding/color_hist/13_1_s/pr_15.txt}{Q=15}
		\prplotadd{data/global_colour_hist/13_1_s/pr_4.txt}{No grid}

		\prplotclose
		\end{tikzpicture}
		\subcaption{Resultant graph}
		\label{fig:grid-color-books-graph}
	\end{minipage}
	\caption{Gridded global colour histogram results for query image 13\_1\_s}
	\label{fig:grid-color-books}
\end{figure}

Figures \ref{fig:grid-color-sheep} and \ref{fig:grid-color-books} show the results of the gridding process.

Figure \ref{fig:grid-color-sheep} shows little variation across gridding quantisation levels, this is likely due to the fact that the image has only two dominant colours, so all grids have similar colour histograms. Gridding does, in the general case, show slight performance improvement with this image however. Notably increasing precision values for higher recall levels.

Figure \ref{fig:grid-color-books} on the other hand shows great variation in the result with grid size. Increasing the grid quantisation level to 2 greatly increases the performance up to a recall of around 0.4, and a grid quantisation level of 1 also increases the performance by a large amount also. In addition to this, much higher quantisation steps, such as 8 and 15 show greatly decreased performance.

The optimum quantisation factor is therefore $Q=2$, yielding \texttt{h\_level = 3}, and \texttt{v\_level = 2} using the formulae defined in Section \ref{sec:gridding-concatenation}.

\FloatBarrier
\section{Edge Orientation Histogram} \label{sec:texture-histogram-results}
As discussed in Section \ref{sec:texture-histogram}, the gridded edge orientation histogram implementation has two variable parameters, $E$ and $Q$ (setting the grid quantisation factors to the optimised values from Section \ref{sec:gridding-results}). Evaluating the effect of changing both together is computationally expensive, and the results are hard to visualise, since a 3-axis plot would be required. Therefore the two variables will be evaluated separately.

Initially the effect of changing the histogram quantisation level, $Q$, will be investigated using a constant $E$ of $0$. This will leave all edges in the image for the consideration of the histogram. The expected result is similar to increasing the quantisation level of the RGB histogram -- increasing $Q$ will increase performance up to a point, but increasing past that point will result in poorer results

\begin{figure}[ht]
	\begin{minipage}[]{0.3\linewidth}
		\centering
		\includegraphics[width = 0.9\linewidth]{figs/msrc/9_23_s}
		\subcaption{Query image}
	\end{minipage}
	\begin{minipage}[]{0.7\linewidth}
		\centering
		\begin{tikzpicture}
		\prplot{data/grid_text_hist/E_0/9_23_s/pr_4.txt}{Q=4}
		\prplotadd{data/grid_text_hist/E_0/9_23_s/pr_8.txt}{Q=8}
		\prplotadd{data/grid_text_hist/E_0/9_23_s/pr_10.txt}{Q=10}
		\prplotadd{data/grid_text_hist/E_0/9_23_s/pr_12.txt}{Q=12}
		\prplotadd{data/grid_text_hist/E_0/9_23_s/pr_15.txt}{Q=16}
		\prplotadd{data/grid_text_hist/E_0/9_23_s/pr_18.txt}{Q=18}
		\prplotadd{data/grid_text_hist/E_0/9_23_s/pr_20.txt}{Q=20}
		\prplotclose
		\end{tikzpicture}
		\subcaption{Resultant graph}
	\end{minipage}
	\caption{Gridded texture histogram results for query image 9\_23\_s, with $E=0$}
	\label{fig:grid-text-sheep}
\end{figure}

\begin{figure}[ht]
	\begin{minipage}[]{0.3\linewidth}
		\centering
		\includegraphics[width = 0.9\linewidth]{figs/msrc/13_1_s}
		\subcaption{Query image}
	\end{minipage}
	\begin{minipage}[]{0.7\linewidth}
		\centering
		\begin{tikzpicture}
		\prplot{data/grid_text_hist/E_0/13_1_s/pr_4.txt}{Q=4}
		\prplotadd{data/grid_text_hist/E_0/13_1_s/pr_8.txt}{Q=8}
		\prplotadd{data/grid_text_hist/E_0/13_1_s/pr_10.txt}{Q=10}
		\prplotadd{data/grid_text_hist/E_0/13_1_s/pr_12.txt}{Q=12}
		\prplotadd{data/grid_text_hist/E_0/13_1_s/pr_15.txt}{Q=16}
		\prplotadd{data/grid_text_hist/E_0/13_1_s/pr_18.txt}{Q=18}
		\prplotadd{data/grid_text_hist/E_0/13_1_s/pr_20.txt}{Q=20}
		\prplotclose
		\end{tikzpicture}
		\subcaption{Resultant graph}
	\end{minipage}
	\caption{Gridded texture histogram results for query image 13\_1\_s, with $E=0$}
	\label{fig:grid-text-books}
\end{figure}

Figure \ref{fig:grid-text-sheep} shows that for the sheep image, there is very little difference in performance with quantisation levels greater than approximately 4. Figure \ref{fig:grid-text-books} shows that for the books, on the other hand, performance is strongly dependant on quantisation level. Higher quantisation levels continue to improve performance, up to approximately $Q=16$, after this point performance decreases.

\begin{figure}[ht]
	\begin{minipage}[]{0.3\linewidth}
		\centering
		\includegraphics[width = 0.9\linewidth]{figs/msrc/9_23_s}
		\subcaption{Query image}
	\end{minipage}
	\begin{minipage}[]{0.7\linewidth}
		\centering
		\begin{tikzpicture}
		\prplot{data/grid_text_hist/Q_16/9_23_s/pr_0.txt}{E=0}
		\prplotadd{data/grid_text_hist/Q_16/9_23_s/pr_1.txt}{E=0.1}
		\prplotadd{data/grid_text_hist/Q_16/9_23_s/pr_2.txt}{E=0.2}
		\prplotadd{data/grid_text_hist/Q_16/9_23_s/pr_3.txt}{E=0.3}
%		\prplotadd{data/grid_text_hist/Q_16/9_23_s/pr_4.txt}{E=0.4}
		\prplotadd{data/grid_text_hist/Q_16/9_23_s/pr_5.txt}{E=0.5}
%		\prplotadd{data/grid_text_hist/Q_16/9_23_s/pr_6.txt}{E=0.6}
		\prplotadd{data/grid_text_hist/Q_16/9_23_s/pr_7.txt}{E=0.7}
%		\prplotadd{data/grid_text_hist/Q_16/9_23_s/pr_8.txt}{E=0.8}
		\prplotadd{data/grid_text_hist/Q_16/9_23_s/pr_9.txt}{E=0.9}
%		\prplotadd{data/grid_text_hist/Q_16/9_23_s/pr_10.txt}{E=1.0}
		\prplotclose
		\end{tikzpicture}
		\subcaption{Resultant graph}
	\end{minipage}
	\caption{Gridded texture histogram results for query image 9\_23\_s, with $Q=16$}
	\label{fig:grid-text-sheep-q16}
\end{figure}

\begin{figure}[ht]
	\begin{minipage}[]{0.3\linewidth}
		\centering
		\includegraphics[width = 0.9\linewidth]{figs/msrc/13_1_s}
		\subcaption{Query image}
	\end{minipage}
	\begin{minipage}[]{0.7\linewidth}
		\centering
		\begin{tikzpicture}
		\prplot{data/grid_text_hist/Q_16/13_1_s/pr_0.txt}{E=0}
		\prplotadd{data/grid_text_hist/Q_16/13_1_s/pr_1.txt}{E=0.1}
		\prplotadd{data/grid_text_hist/Q_16/13_1_s/pr_2.txt}{E=0.2}
		\prplotadd{data/grid_text_hist/Q_16/13_1_s/pr_3.txt}{E=0.3}
%		\prplotadd{data/grid_text_hist/Q_16/13_1_s/pr_4.txt}{E=0.4}
		\prplotadd{data/grid_text_hist/Q_16/13_1_s/pr_5.txt}{E=0.5}
%		\prplotadd{data/grid_text_hist/Q_16/13_1_s/pr_6.txt}{E=0.6}
		\prplotadd{data/grid_text_hist/Q_16/13_1_s/pr_7.txt}{E=0.7}
%		\prplotadd{data/grid_text_hist/Q_16/13_1_s/pr_8.txt}{E=0.8}
		\prplotadd{data/grid_text_hist/Q_16/13_1_s/pr_9.txt}{E=0.9}
%		\prplotadd{data/grid_text_hist/Q_16/13_1_s/pr_10.txt}{E=1.0}
		\prplotclose
		\end{tikzpicture}
		\subcaption{Resultant graph}
	\end{minipage}
	\caption{Gridded texture histogram results for query image 13\_1\_s, with $Q=16$}
	\label{fig:grid-text-books-q16}
\end{figure}

Figures \ref{fig:grid-text-sheep-q16} and \ref{fig:grid-text-books-q16} show the result of varying $E$, whilst keeping $Q$ at a constant value of $16$. The results are interesting. For the bookshelves, increasing $E$ above $0$ improves performance up to approximately $E=0.3$, however, for the sheep increasing $E$ above $0$ greatly hinders performance. This is likely due to the fact that whilst the books have lots of strong edges (at the edges of the books), which will remain with increasing values of $E$, the sheep image likely lacks these strong edges, and so increasing $E$ removes almost all of the relevant edges from the generated descriptor.

\FloatBarrier
\section{Concatenation of Descriptors} \label{sec:conc-desc-results}
Sections \ref{sec:gridding-results} and \ref{sec:texture-histogram-results} demonstrate the effectiveness of gridded colour histogram and texture descriptors. It stands to reason therefore that these descriptors could be combined to produce a descriptor of even greater effectiveness. This is possible using the methodology described in Section \ref{sec:conc-desc}.

In order to combine the gridded texture and gridded global colour histogram results, the parameters of these functions which were found to be optimal in Sections \ref{sec:gridding-results} and \ref{sec:texture-histogram-results} will be used. The only free parameter of the concatenation function is therefore $M$. $M$ is a multiplier that is applied to the texture function in order to adjust the weighting between the texture and colour descriptors. When $M$ is $1$ the two descriptors are equally weighted, however since the colour descriptor is 4 times as long as the texture descriptor (a 64 bin histogram, vs and 8 bin histogram), it is effectively weighted more highly since there are many more dimensions in which values can be distant. Whilst weighting of $M=4$ should therefore make up for this difference in descriptor length, a different value may provide better results.

\begin{figure}[ht]
	\begin{minipage}[]{0.3\linewidth}
		\centering
		\includegraphics[width = 0.9\linewidth]{figs/msrc/9_23_s}
		\subcaption{Query image}
	\end{minipage}
	\begin{minipage}[]{0.7\linewidth}
		\centering
		\begin{tikzpicture}
		\prplot{data/grid_text_hist/Q_16/9_23_s/pr_0.txt}{Texture only}
		\prplotadd{data/gridding/color_hist/9_23_s/pr_2.txt}{Colour only}
		\prplotadd{data/concat/9_23_s/pr_1.txt}{W = 1}
		\prplotadd{data/concat/9_23_s/pr_4.txt}{W = 4}
		\prplotadd{data/concat/9_23_s/pr_10.txt}{W = 10}
		\prplotclose
		\end{tikzpicture}
		\subcaption{Resultant graph}
	\end{minipage}
	\caption{Concatenated descriptor results for query image 9\_23\_s}
	\label{fig:concat-sheep}
\end{figure}

\begin{figure}[ht]
	\begin{minipage}[]{0.3\linewidth}
		\centering
		\includegraphics[width = 0.9\linewidth]{figs/msrc/13_1_s}
		\subcaption{Query image}
	\end{minipage}
	\begin{minipage}[]{0.7\linewidth}
		\centering
		\begin{tikzpicture}
		\prplot{data/grid_text_hist/Q_16/13_1_s/pr_0.txt}{Texture only}
		\prplotadd{data/gridding/color_hist/13_1_s/pr_2.txt}{Colour only}
		\prplotadd{data/concat/13_1_s/pr_1.txt}{W = 1}
		\prplotadd{data/concat/13_1_s/pr_4.txt}{W = 4}
		\prplotadd{data/concat/13_1_s/pr_10.txt}{W = 10}
		\prplotclose
		\end{tikzpicture}
		\subcaption{Resultant graph}
	\end{minipage}
	\caption{Concatenated descriptor results for query image 13\_1\_s}
	\label{fig:concat-books}
\end{figure}

Figure \ref{fig:concat-sheep} shows that the concatenated descriptor remains very close to the plot of only the RGB histogram for all values of $W$ plotted. This implies that all of the descriptors are very close in the texture plot, leading to the colour plot dominating the results.

Figure \ref{fig:concat-books} on the other hand shows that the concatenated descriptor does vary between the two plots with differing values of $W$. This implies that both descriptors are discriminative. The results show that at a value of $W=1$, the results are close to the RGB histogram, but at $W=10$, the results are closer to the edge orientation histogram. A value of $W=4$ provides a compromise between the two as expected. The results are poorer than the RGB histogram for this particular query image, but this is unlikely to hold true for all query images since some images will likely be described by colour well, and others by texture well.

\section{Overall Results} \label{sec:overall-results}

Following the characterisation each algorithm, discussed in the previous parts of this section, it has been possible to test the performance of each descriptor against the entire database of images. This is achieved by calculating \gls{map}. \gls{map} is a single numeric measure for determining the performance of a descriptor across a dataset. Is is the mean of \gls{ap}, which is defined in Equation \ref{eq:map}. In this equation P$(i)$ represents the precision at that value of $i$, and Rel$(i)$ is $1$ if the result is relevant, or $0$ if it is not.

\begin{equation}
\text{\glsentryshort{ap}} = \frac{\sum_{i=1}^{n} \left( \text{P}(i) \times \text{Rel}(i) \right) }{\sum_{i=1}^{n}  \text{Rel}(i) } \label{eq:map}
\end{equation}

Table \ref{tbl:map-stats} tabulates the \gls{map} statistics for all images in the dataset, both within each category, as well as globally. These results show \gls{map} across the entire dataset, so a lot of the numbers are low. The reason for this is that whilst the images have similar Level 2 content, some of the images are very different in terms of colour and texture content. In order to account for this, Table \ref{tbl:map-stats-at-10} tabulates \gls{map} statistics for the first 10 relevant recalled images. This data removes the effect of the aforementioned `outlier' images in each category, and better demonstrates how well each type of image content responds to each descriptor.

The results in Table \ref{tbl:map-stats-at-10} show that the image sets respond well to some descriptors, and some respond poorly. One example of this is category 4, where the \gls{map} is 0.71 for the edge orientation histogram, but 0.25 and 0.44 for the global colour histogram and gridded global colour histogram. This can be explained by looking at the type of image category 4 is. Category 4 contains images of aeroplanes. Almost all of these images have a smooth aeroplane body in the middle of the image, with a clear sky at the top, and plain grass at the bottom. This distinctive texture signature is what allows this high \gls{ap} figure. Other image sets which respond particularly well to the edge orientation histogram are categories 7 and 13 respectively. Category 7 contains cars, and category 13 books on shelves, both of these have distinct texture signatures.

In general, most of the categories responded better to the gridded colour histogram than the global colour histogram. This is expected since encoding the spatial aspect of the colour data should allow for a more discriminative descriptor. A notable example which does not is category 18. Category 18 is a series of images which all contain water, however the water is in a different location in almost every image, this kind of spatial variance cannot be encoded in the gridded descriptor, so it performs poorly.

Two categories which responded particularly well to the colour descriptor are categories 2 and 13. These are images of trees, mostly on a background of a washed-out sky, and bookshelves of colourful books respectively. Both images likely work well with this descriptor because they contain a variety of colours which are both common across most of the images, and distinctive in the image set itself.

In general, concatenation of colour and texture descriptors improved performance over the individual descriptors. The cases where they did not are cases where one descriptor had a much greater performance over the other, such as category 4, which responded particularly well to the texture descriptor and category 2 which responded particularly well to the colour descriptors. One example of a category which showed great improvement with descriptor concatenation are categories 7 and 10. These categories contain differently coloured cars in different orientations and differently coloured flowers, with varying shapes. Concatenation helps with the cars because the variance of colour across the set means that the similarly coloured cars strongly match each other in a colour histogram, but all others are distant, whereas all of the cars are likely moderate matches with the edge orientation histogram. The flowers have a similar effect.

One category which performed particularly poorly in all cases is category 16. This category contains photographs of dogs, of various sizes and colours, all taken from different camera angles and on different backgrounds. This is an example of Level 2 concepts which do not translate well at all to Level 1 attempts at interpretation.

\begin{table}[ht]
	\rowcolors{2}{gray!25}{white}
	\caption{\glsentryshort{map} statistics for all methods and categories}
	\label{tbl:map-stats}
	\centering
	\pgfplotstableset{ % Makes bold a whole row of a table
		highlightrow/.style={
			postproc cell content/.append code={
				\count0=\pgfplotstablerow
				\advance\count0 by1
				\ifnum\count0=#1
				\pgfkeysalso{@cell content/.add={$\bf}{$}}
				\fi
			},
		},
	}
	\pgfplotstabletypeset[
	col sep=comma,
	string type,
	columns/set/.style={column name={\textbf{Image category}}, column type={r}},
	columns/gch/.style={column name={\textbf{\shortstack{Global colour\\histogram}}}, column type={r}},
	columns/gridch/.style={column name={\textbf{\shortstack{Gridded\\ colour\\histogram}}}, column type={r}},
	columns/eoh/.style={column name={\textbf{\shortstack{Gridded\\edge orienatation\\histogram}}}, column type={r}},
	columns/concat/.style={column name={\textbf{\shortstack{Concatenated\\descriptors}}}, column type={r}},
	every head row/.style={before row=\toprule, after row=\midrule},
	every last row/.style={after row=\bottomrule},
	highlightrow={21},
	]{data/global_stats/map.txt}
\end{table}

\begin{table}[ht]
	\rowcolors{2}{gray!25}{white}
	\caption{\glsentryshort{map} statistics for first 10 images in each set, over all methods and categories}
	\label{tbl:map-stats-at-10}
	\centering
	\pgfplotstableset{ % Makes bold a whole row of a table
		highlightrow/.style={
			postproc cell content/.append code={
				\count0=\pgfplotstablerow
				\advance\count0 by1
				\ifnum\count0=#1
				\pgfkeysalso{@cell content/.add={$\bf}{$}}
				\fi
			},
		},
	}
	\pgfplotstabletypeset[
	col sep=comma,
	string type,
	columns/set/.style={column name={\textbf{Image category}}, column type={r}},
	columns/gch/.style={column name={\textbf{\shortstack{Global colour\\histogram}}}, column type={r}},
	columns/gridch/.style={column name={\textbf{\shortstack{Gridded\\ colour\\histogram}}}, column type={r}},
	columns/eoh/.style={column name={\textbf{\shortstack{Gridded\\edge orienatation\\histogram}}}, column type={r}},
	columns/concat/.style={column name={\textbf{\shortstack{Concatenated\\descriptors}}}, column type={r}},
	every head row/.style={before row=\toprule, after row=\midrule},
	every last row/.style={after row=\bottomrule},
	highlightrow={21},
	]{data/global_stats/map_at_10.txt}
\end{table}

\chapter{Conclusion} \label{sec:conclusion}
This report presents several methods of generating descriptors for images, and uses test data to  set their adjustable parameters, the results of which is presented in Section \ref{sec:results}. The overall results show that the \gls{matlab} system is capable of performing with a reasonable \gls{map} over the first 10 images in each set. However performance does suffer when considered over the whole dataset. Overall the concatenated descriptors perform best over the entire dataset with a \gls{map} of 0.21, or 0.36 if only the first 10 images are considered. A global colour histogram performs most poorly overall with a \gls{map} of 0.16 over the entire dataset, or 0.26 if only the first 10 images are considered.

The Appendices present further results which extend the investigation in areas other than investigation of different descriptors. Appendix \ref{app:pca} investigates \gls{pca} and shows how descriptors can be made much more compact without impact on performance. The investigated descriptor, a gridded colour histogram, was able to be reduced to 26\% of it's original size without noticeable performance impact.

Appendix \ref{app:dist-measures} investigates various distance measures which can be used to compare descriptors and tests their relative performance. The $L_2$ norm, which has been used throughout the remainder of this investigation was found to be the strongest overall performer.

In addition Appendix \ref{app:test-methodology} presents the test methodology for verification of the functionality of the descriptor generation functions.

To conclude the system presented by the report is a modular and extensible visual search system, with characterised performance, and scope to be extended beyond its current functionality.


\documentclass{article}
\usepackage{nips15submit_e,times}
\usepackage{hyperref}
\usepackage{url}
\usepackage{graphicx}
\usepackage{latexsym}

\title{Weekly Report(March.19,2019-April.17,2019)}

\newcommand{\fix}{\marginpar{FIX}}
\newcommand{\new}{\marginpar{NEW}}

\begin{document}

\maketitle
\begin{abstract}
	During these weeks, I finished assignment3. I learned RNN, LSTM and some other interesting models. And I also modified the last part of Pytorch.ipnb in assignment2.
\end{abstract}

\section{Learned}
After finishing all the thing about them, I have to say, it's easy to finish code if you know the formula. But it'e hard to understand how they work and why they can make it.
\subsection{RNN}
\subsubsection{Introduction}
Recurrent Neural Network. It's a typical model to address squential data, such as videos, sentences, music and so on.\\
\begin{figure}[h]
	\centering  %插入的图片居中表示
	\includegraphics[width=1\textwidth]{1.png}
	\caption{Different kinds of RNN}  %图片的名称
	\label{fig:f1}   %标签，用作引用
\end{figure}

Formula:

for each hidden layer:
\begin{center}
${\displaystyle h_t=f_w(h_{t-1}, x_t)=tanh(Wh_{t-1}*h_{t-1}+Wx*X+b)}$\\
${y_t=W_{hy}h_t}$
\end{center}
${h_t}$is the new state, while ${h_{t-1}}$is the old state, and ${x_t}$ is input of this timestep. ${f_w}$ is some function with parameters W, the right part of the equation is the function used in homework. ${y_t}$ will be used when calculating loss.

\begin{figure}[htbp]
	\centering  %插入的图片居中表示
	\includegraphics[width=0.8\textwidth]{2.jpg}
	\caption{Typical many-to-many model}  %图片的名称
	\label{fig:f2}   %标签，用作引用
\end{figure}

\subsubsection{My work}
Actually It's much more easier than other parts, so I will just show the result.

\begin{figure}[htbp]
	\centering  %插入的图片居中表示
	\includegraphics[width=0.8\textwidth]{3.png}
	\caption{loss}  %图片的名称
	\label{fig:f3}   %标签，用作引用
\end{figure}

\begin{figure}[htbp]
	\centering
	\begin{minipage}[t]{0.5\textwidth}
		\centering
		\includegraphics[width=1.0\textwidth]{4.png}
	%	\caption{ab}
	\end{minipage}
	\begin{minipage}[t]{0.45\textwidth}
		\centering
		\includegraphics[width=1.0\textwidth]{5.png}
	\end{minipage}
	\caption{trained result}
\end{figure}

\subsection{LSTM}
\subsubsection{Introduction}
Long Short Term Memory, is slightly fancier recurrence relationfor RNN. It's designed to solve the problem of vanishing and exploding gradients.

Formula:

\begin{center}
	$\left( \begin{array}{ccc} i\\f\\o\\g\end{array} \right)
	=\left( \begin{array}{ccc} \sigma\\\sigma\\\sigma\\tanh\end{array} \right)
	W\left( \begin{array}{ccc} h_{t-1}\\x_t \end{array} \right)$

	${\displaystyle c_t=f\odot c_{t-1}+i\odot g}$

	${\displaystyle h_t=o \odot tanh(c_t)}$
\end{center}

In this formulas, \textsl{i} means \textsl{input\_gate}, \textsl{f} means \textsl{forget\_gate}, \textsl{o} means \textsl{output\_gate}, while \textsl{g} doesn't have a good name.

\subsubsection{My work}
The implementation is the same as RNN, so I will still say nothing about it.

\begin{figure}[htbp]
	\centering  %插入的图片居中表示
	\includegraphics[width=.7\textwidth]{6.png}
	\caption{Final result}  %图片的名称
	\label{fig:f6}   %标签，用作引用
\end{figure}

\begin{figure}[htbp]
	\centering
	\begin{minipage}[t]{0.5\textwidth}
		\centering
		\includegraphics[width=1.0\textwidth]{7.png}
	\end{minipage}
	\begin{minipage}[t]{0.4\textwidth}
		\centering
		\includegraphics[width=1.0\textwidth]{8.png}
	\end{minipage}
	\caption{trained results}
\end{figure}

\section{Modification}
After finishing assignment3, I think back "open-ended challenge" in assignment2. Since it really attracted me and I think I can find a better model to get better grade.

I try to modify the model for several trials, I finally get about 80\% accuracy. But I have some questions showed below.

\begin{figure}[htbp]
	\centering  %插入的图片居中表示
	\includegraphics[width=0.8\textwidth]{9.png}
	\caption{open-ended challenge accuracy}  %图片的名称
	\label{fig:f9}   %标签，用作引用
\end{figure}

\section{Understanding}
RNN addresses sequential words in the homework. In order to predict next layer's state, it will use all of the previous state as input. It works very well. However, it has the problem of vanishing and exploding gradients. That's why LSTM exists. LSTM will forget part of previous hidden state, which makes the network work better.

\textbf{Question}: As training goes on, I come to confuse about the relation between an actual layer and its functions. For example, why adding convolutional layers in a CNN will get better performance? I know there must be some scientific basis, so how to use this basis if I want to improve a model?

\section{Plan}
I need to focus on my professional courses.
\end{document}


\section*{Representing Time}

\subsection*{\\
The underspecification of time representations in OWL axioms}

Both an axiom in the OBO representational syntax \todo{more on OBO}
%
\begin{equation}
\begin{split}
\mclass{CellMembrane}\;\mirel{continuant\_part\_of}\;\mclass{Cell}
\end{split}
\label{eq:cellobo}
\end{equation}
%
and the OWL axiom (often considered equivalent)
%
\begin{equation}
\begin{split}
\mclass{CellMembrane}\;\mathtt{subClassOf}\;\mclass{continuant\_part\_of}\;\mathtt{some}\;\mclass{Cell}
\end{split}
\label{eq:celldl}
\end{equation}
%
do not exhibit any explicit commitment with regard to time. This is not a side issue, because it makes a
difference whether, for example, a cell membrane is \emph{always} part of some cell or
only \emph{at some time}.
The lack of temporal definition of OWL statements is especially unsatisfactory
when it comes to transitive properties like $\mirel{continuant\_part\_of}$ or $\mirel{located\_in}$,
where the suppression of the temporal factor can produce plainly wrong entailments,
especially at the level of individuals, such as in the following example:
%
\begin{equation}
\begin{split}
\mirel{located\_in} (\mirel{Thrombus\#398}, \mirel{Heart\#431})  \\
\mirel{located\_in} (\mirel{Heart\#431}, \mirel{Patient\#900})
\end{split}
\label{eq:trans}
\end{equation}
%
If we now assume that \mirel{Heart\#431} is later transplanted to \mirel{Patient\#115}, we also have

\begin{equation}
\begin{split}
\mirel{located\_in} (\mirel{Heart\#431}, \mirel{Patient\#115})
\end{split}
\label{eq:trans}
\end{equation}

This would entail that one of the patients is located within the other, which is plainly wrong.
Also, if we assume that the thrombus was no longer in the heart when it was transplanted to Patient\#115, then the inference to
%
\begin{equation}
\begin{split}
\mirel{located\_in} (\mirel{Thrombus\#398}, \mirel{Patient\#115})
\end{split}
\label{eq:transEntailment}
\end{equation}
%
which follows from the transitivity of the relation $\mirel{located\_in}$, is obviously invalid.

This has a far-reaching consequence: by default, OWL statements are temporally ambiguous in a way that may entail unintended consequences when domain ontologies are used for reasoning with real data. (A lack of an explicit treatment of temporal context has previously been implicated in an assessment of the quality of existential restrictions in OBO Foundry ontologies \cite{boeker2011}.)

\subsection*{Strengths of Relatedness}

We can now propose a distinction between different temporal \emph{strengths} of relatedness. These are to be satisfied by temporally qualified such as parthood defined in the OBO Relation Ontology  and apply to relations between continuants only.
We begin by characterising the ontological status of regions of time , henceforth referred to by the symbol $t$.
Times are either time intervals or time points. %, assuming that the latter can be approximated by infinitesimally small intervals.
We restrict ourselves to those ternary relations for which it can be assumed that whenever they hold for a time interval,
they also hold for any of its subintervals, including time points. For example, if a car is located in a garage during a day, then it is located therein at any time interval during that day. If a patient's cholesterol level is elevated for a whole year, it is elevated over any duration of time during this year.
In the following, ternary relations are marked by the superscript $^t$, like in $\mrelt{rel}$, from which binary relations are  distinguished by
$^b$, like in $\mrelb{rel}$.

Formally:
\begin{equation}
\begin{split}
\forall a, b, t, t^\prime:\; \mrelt{rel}(a, b, t) \wedge \mirel{within}(t^\prime,t) \rightarrow
\mrelt{rel}(a,b,t^\prime)
\end{split}
\label{eq:temporarily:temp}
\end{equation}

These are called time-distributive relations.

\subsubsection*{Temporary Generic Relatedness (TGR)}

Informally: for all instances $a$ of \mclass{A} there is some time $t$ and some instance $b$ of
\mclass{B} such that $a$ is related to $b$ at $t$. Examples:
\begin{enumerate}[(a)]
\item for all apple seeds there is
some apple such that the seed is part of the apple at some time;
\item for all
trees there is some leaf such that the leaf is part of the tree at some time.
\end{enumerate}

Formally:
\begin{equation}
\begin{split}
\mclass{TemporarilyRelated}(\mclass{A},\mclass{B}) =_{def}&\;
\forall a, t:\; \mrelt{inst}(\mclass{A}, a, t) \\
&\ \rightarrow
\exists b, t^\prime:\;(\mrelt{inst}(\mclass{B},b,t^\prime) \wedge
\mrelt{rel}(a,b,t^\prime) \wedge \mirel{within}(t^\prime,t))
\end{split}
\label{eq:temporarily:cls}
\end{equation}

\subsubsection*{Permanent Generic Relatedness (PGR)}

Informally: for all instances $a$ of \mclass{A} there is, at all times $t$ for which
$a$ exists,
some instance $b$ of \mclass{B} such that $a$ is related to $b$ at $t$, but not necessarily
always the same $b$ at all times $t$. Examples:
\begin{enumerate}[(a)]
\item all cells have a water molecule as
part at all times, but not always the same water molecule;
\item every bacterial colony has some bacteria as parts at all times, but not
always the same bacteria.
\end{enumerate}

\begin{equation}
\begin{split}
\mclass{PermanentlyGenericallyRelated}(\mclass{A},\mclass{B}) =_{def}&\;
\forall a, t:\; \mrelt{inst}(\mclass{A}, a, t) \\
&\ \rightarrow
\exists b:\;(\mrelt{inst}(\mclass{B},b,t) \wedge
\mrelt{rel}(a,b,t))
\end{split}
\label{eq:generically:cls}
\end{equation}

It is as PGR relations that the standard interpretation of OBO binary class-level relations are understood
such as in formula (\ref{eq:cellobo}) above, according to \cite{OBO:RO}.

\subsubsection*{Permanent Specific Relatedness (PSR)}

Informally, for all instances $a$ of \mclass{A} there is, at all times $t$ that $a$ exists, an
instance $b$ of \mclass{B} such that $a$ is related to $b$ at $t$; in this case it is the
same $b$ at all times $t$. Examples:
\begin{enumerate}[(a)]
\item a human being has a brain as part at all times, and it is necessarily the same brain;
\item a radioactively marked molecule of DNA has the radioactive isotope as part
at all times, and it is necessarily the same radioactive isotope.
\end{enumerate}

\begin{equation}
\begin{split}
\mclass{Permanently}&\mclass{SpecificallyRelated}(\mclass{A},\mclass{B}) =_{def}\;
\forall a, t:\; \mrelt{inst}(\mclass{A}, a, t) \\
&\ \rightarrow
\exists b:\;\big(\mrelt{inst}(\mclass{B},b,t) \wedge
\mrelt{rel}(a,b,t))
\\
&\quad\quad \wedge \forall t^\prime: (\mrelt{inst}(\mclass{A},a,t^\prime)
\rightarrow (\mrelt{rel}(a,b,t^\prime) \wedge
\mrelt{inst}(\mclass{B},b,t^\prime))\big)
\end{split}
\label{eq:specifically:cls}
\end{equation}


\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage[acronym, toc]{glossaries}

\makeglossaries

\input{Glossary.tex}

\begin{document}

\tableofcontents

\section{First Section}

The \Gls{latex} typesetting markup language is specially suitable for documents that include {maths}. {formula} are rendered properly an easily once one gets used to the commands.

%\clearpage

\section{Second Section}

\vspace{5mm}

Given a set of numbers, there are elementary methods to compute its {gcd}, which is abbreviated {gcd}. This process is similar to that used for the {lcm}.

%\clearpage

\printglossary

%\clearpage

\printglossary[type=\acronymtype]

\end{document}


\chapter{Weights table for Higgs sample}
\section{Windowed re-weighting results}
\label{apdx:window_wgts}
\begin{table}[thb]
    \centering
    \footnotesize
    \makebox[\textwidth]{\begin{tabular}{|c|c|c|c|c|c|c|c|c|c|c|c|c|}
\hline
(100 GeV) & 2-3 & 3-5 & 5-6 & 6-7 & 7-8 & 8-9 & 9-10 & 10-15 & 15-20 & 20-25 & 25-30 & 30-80 \\ \hline
30&0.0572 & 0.1082 & 0.0737 & 0.0686 & 0.0742 & 0.0857 & 0.1015 & 0.1431 & 0.1832 & 0.2573 & 0.4859 & 0.74 \\ \hline
25&0.0531 & 0.1011 & 0.071 & 0.0659 & 0.0734 & 0.083 & 0.1001 & 0.1461 & 0.2169 & 0.3597 & 0.3971 & 0.1964 \\ \hline
20&0.0545 & 0.0913 & 0.0659 & 0.0633 & 0.0713 & 0.084 & 0.1021 & 0.1599 & 0.2889 & 0.3043 & 0.0933 & 0.0481 \\ \hline
15&0.0643 & 0.1118 & 0.0829 & 0.0812 & 0.0948 & 0.113 & 0.1447 & 0.2395 & 0.2847 & 0.0732 & 0.022 & 0.0141 \\ \hline
10&0.0242 & 0.0493 & 0.044 & 0.0529 & 0.0752 & 0.1179 & 0.1896 & 0.1752 & 0.0141 & 0.0028 & 0.0008 & 0.0008 \\ \hline
9&0.0238 & 0.0468 & 0.0465 & 0.0602 & 0.1028 & 0.1791 & 0.2064 & 0.0832 & 0.0066 & 0.0013 & 0.0004 & 0.0003 \\ \hline
8&0.0205 & 0.045 & 0.0592 & 0.0941 & 0.1991 & 0.2324 & 0.1099 & 0.0341 & 0.0031 & 0.0007 & 0.0002 & 0.0001 \\ \hline
7&0.0207 & 0.0478 & 0.076 & 0.1965 & 0.2409 & 0.0787 & 0.0324 & 0.012 & 0.0014 & 0.0003 & 0.0001 & 0.0001 \\ \hline
6&0.0212 & 0.0637 & 0.2502 & 0.2875 & 0.0573 & 0.0205 & 0.0098 & 0.0048 & 0.0007 & 0.0002 & 0.0 & 0.0001 \\ \hline
5&0.014 & 0.0705 & 0.217 & 0.0251 & 0.0083 & 0.0041 & 0.0023 & 0.0014 & 0.0002 & 0.0 & 0.0 & 0.0 \\ \hline
3&0.2708 & 0.2386 & 0.0111 & 0.0037 & 0.002 & 0.0012 & 0.0008 & 0.0006 & 0.0001 & 0.0 & 0.0 & 0.0 \\ \hline
2&0.3757 & 0.0259 & 0.0026 & 0.001 & 0.0006 & 0.0004 & 0.0003 & 0.0001 & 0.0 & 0.0 & 0.0 & 0.0 \\ \hline
\end{tabular}}
\caption{Windowed re-weigthing result for ggZZ samples.}
\end{table}

\begin{table}[thb]
    \centering
    \footnotesize
    \makebox[\textwidth]{\begin{tabular}{|c|c|c|c|c|c|c|c|c|c|c|c|c|c|}
\hline
(100 GeV) & 2-3 & 3-4 & 4-5 & 5-6 & 6-7 & 7-8 & 8-9 & 9-10 & 10-15 & 15-20 & 20-25 & 25-30 & 30-80 \\ \hline
30                       & 0.0013  & 0.0024  & 0.0021  & 0.0035  & 0.003   & 0.005   & 0.01    & 0.0112   & 0.0265    & 0.0764    & 0.1583    & 0.3873    & 0.6233    \\ \hline
25                       & 0.0017  & 0.0028  & 0.0023  & 0.0046  & 0.004   & 0.0063  & 0.0121  & 0.0149   & 0.039     & 0.1193    & 0.2795    & 0.3844    & 0.2218    \\ \hline
20                       & 0.0015  & 0.0044  & 0.0038  & 0.0067  & 0.0065  & 0.0096  & 0.0211  & 0.0256   & 0.0709    & 0.2562    & 0.3658    & 0.1491    & 0.0904    \\ \hline
15                       & 0.0045  & 0.0088  & 0.0075  & 0.015   & 0.0131  & 0.0218  & 0.0448  & 0.06     & 0.1727    & 0.4011    & 0.1484    & 0.0569    & 0.0434    \\ \hline
10                       & 0.0052  & 0.0077  & 0.0092  & 0.0184  & 0.0208  & 0.0464  & 0.0152  & 0.0085   & 0.2471    & 0.0459    & 0.0136    & 0.0058    & 0.0055    \\ \hline
9                        & 0.0054  & 0.0142  & 0.0167  & 0.0378  & 0.0449  & 0.0975  & 0.332   & 0.4022   & 0.2029    & 0.0389    & 0.0122    & 0.0058    & 0.0052    \\ \hline
8                        & 0.0107  & 0.0196  & 0.0252  & 0.0623  & 0.0998  & 0.3037  & 0.187   & 0.2898   & 0.1195    & 0.0255    & 0.0085    & 0.0036    & 0.0042    \\ \hline
7                        & 0.0087  & 0.0226  & 0.0299  & 0.1043  & 0.2057  & 0.2715  & 0.2067  & 0.0905   & 0.0481    & 0.0125    & 0.0047    & 0.0023    & 0.0022    \\ \hline
6                        & 0.0176  & 0.0482  & 0.0882  & 0.0557  & 0.4805  & 0.157   & 0.1003  & 0.0499   & 0.0348    & 0.0108    & 0.0039    & 0.002     & 0.0017    \\ \hline
5                        & 0.029   & 0.1134  & 0.2939  & 0.5247  & 0.1041  & 0.0499  & 0.0397  & 0.025    & 0.0188    & 0.0065    & 0.0026    & 0.0015    & 0.0013    \\ \hline
4                        & 0.0634  & 0.336   & 0.4353  & 0.1231  & 0.001   & 0.0183  & 0.0174  & 0.0129   & 0.0111    & 0.0042    & 0.0015    & 0.0008    & 0.0007    \\ \hline
3                        & 0.1082  & 0.3288  & 0.0725  & 0.0305  & 0.0114  & 0.0086  & 0.009   & 0.0064   & 0.0058    & 0.0022    & 0.0009    & 0.0005    & 0.0004    \\ \hline
2                        & 0.7428  & 0.0912  & 0.0134  & 0.0132  & 0.005   & 0.0042  & 0.0047  & 0.0031   & 0.0028    & 0.0004    & 0.0       & 0.0       & 0.0       \\ \hline
\end{tabular}}
\caption{Windowed re-weigthing result for VBF samples.}
\end{table}
\pagebreak

\section{Sample mass factors for ggZZ and VBF\_ZZ samples}
\label{apdx:mass_wgts}
\begin{table}[]
    \centering
\begin{tabular}{|c|c|c|c|}
\hline
\multicolumn{2}{|c|}{ggZZ} & \multicolumn{2}{c|}{VBF\_ZZ} \\ \hline
Mass (GeV)     & Factor    & Mass (GeV)      & Factor     \\ \hline
200            & 1         & 200             & 1          \\ \hline
300            & 0.8798    & 300             & 0.8731     \\ \hline
500            & 1.1068    & 400             & 0.7873     \\ \hline
600            & 0.7327    & 500             & 0.7106     \\ \hline
700            & 0.6833    & 600             & 0.6361     \\ \hline
800            & 0.6370    & 700             & 0.5754     \\ \hline
900            & 0.6001    & 800             & 0.5217     \\ \hline
1000           & 0.5658    & 900             & 0.4818     \\ \hline
1500           & 0.4769    & 1000            & 0.4257     \\ \hline
2000           & 0.4300    & 1500            & 0.3255     \\ \hline
2500           & 0.4204    & 2000            & 0.2968     \\ \hline
3000           & 0.4256    & 2500            & 0.2898     \\ \hline
               &           & 3000            & 0.2916     \\ \hline
\end{tabular}
\caption{Sample mass re-weighting factors for both signal samples.}
\end{table}

\chapter{Additional figures}
\pagebreak
\section{1D Histograms of variables used for constructing 2d templates histograms}
\label{sec:1dhists}
\begin{figure}[htb]
\begin{center}
\subfloat[]{\includegraphics[width=.4\linewidth]{fig/MT_ZZ_Nj_0.png}}
\subfloat[]{\includegraphics[width=.4\linewidth]{fig/MT_ZZ_Nj_1.png}}\\
\subfloat[]{\includegraphics[width=.5\linewidth]{fig/MT_ZZ_Nj_2.png}}
\end{center}
\caption{Distribution (integral not to be taken literal) of $\met$ for backgrounds and signals with
different $N_\mathrm{jets}$. Here sig a3 is a scaled up version of the signal (a1) and VVV corresponds to
WWW or WWZ or WZZ}
\label{fig:1d_hists_mtzz}
\end{figure}

\begin{figure}[htb]
\begin{center}
\subfloat[]{\includegraphics[width=.4\linewidth]{fig/MET_Nj_0.png}}
\subfloat[]{\includegraphics[width=.4\linewidth]{fig/MET_Nj_1.png}}\\
\subfloat[]{\includegraphics[width=.5\linewidth]{fig/MET_Nj_2.png}}
\end{center}
\caption{Distribution (integral not to be taken literal) of $\met$ for backgrounds and signals with
different $N_\mathrm{jets}$. Here sig a3 is a scaled up version of the signal (a1) and VVV corresponds to
WWW or WWZ or WZZ}
\label{fig:1d_hists_met}
\end{figure}

\begin{figure}[htb]
\begin{center}
\includegraphics[width=.7\linewidth]{fig/DJJ_VBF_Nj_2.png}
\end{center}
\caption{Distribution (integral not to be taken literal) of $D_{j j}^{V B F}$ for backgrounds and signals with
$N_\mathrm{jets}=2$. Here sig a3 is a scaled up version of the signal (a1) and VVV corresponds to
WWW or WWZ or WZZ}
\label{fig:1d_hists_djjvbf}
\end{figure}
\section{Signal sample fitting template histogram of background and signal (z-axis scaled by \num{1e5})}
\begin{figure}[hbt]
\begin{center}
\subfloat[]{\includegraphics[width=.33\linewidth]{fig/counts_templates/ggZZ_2e2nu_nj0_Bkg.pdf}}
\subfloat[]{\includegraphics[width=.33\linewidth]{fig/counts_templates/ggZZ_2e2nu_nj0_Sig.pdf}}
\subfloat[]{\includegraphics[width=.33\linewidth]{fig/counts_templates/ggZZ_2e2nu_nj1_Bkg.pdf}}\\
\subfloat[]{\includegraphics[width=.33\linewidth]{fig/counts_templates/ggZZ_2e2nu_nj1_Sig.pdf}}
\subfloat[]{\includegraphics[width=.33\linewidth]{fig/counts_templates/ggZZ_2e2nu_nj2_Bkg.pdf}}
\subfloat[]{\includegraphics[width=.33\linewidth]{fig/counts_templates/ggZZ_2e2nu_nj2_Sig.pdf}}\\
\subfloat[]{\includegraphics[width=.33\linewidth]{fig/counts_templates/ggZZ_2e2nu_nj3p_Bkg.pdf}}
\subfloat[]{\includegraphics[width=.33\linewidth]{fig/counts_templates/ggZZ_2e2nu_nj3p_Sig.pdf}}
\end{center}
\end{figure}

\begin{figure}[hbt]
\begin{center}
\subfloat[]{\includegraphics[width=.33\linewidth]{fig/counts_templates/VVZZ_2e2nu_nj0_Bkg.pdf}}
\subfloat[]{\includegraphics[width=.33\linewidth]{fig/counts_templates/VVZZ_2e2nu_nj0_Sig.pdf}}
\subfloat[]{\includegraphics[width=.33\linewidth]{fig/counts_templates/VVZZ_2e2nu_nj1_Bkg.pdf}}\\
\subfloat[]{\includegraphics[width=.33\linewidth]{fig/counts_templates/VVZZ_2e2nu_nj1_Sig.pdf}}
\subfloat[]{\includegraphics[width=.33\linewidth]{fig/counts_templates/VVZZ_2e2nu_nj2_Bkg.pdf}}
\subfloat[]{\includegraphics[width=.33\linewidth]{fig/counts_templates/VVZZ_2e2nu_nj2_Sig.pdf}}\\
\subfloat[]{\includegraphics[width=.33\linewidth]{fig/counts_templates/VVZZ_2e2nu_nj3p_Bkg.pdf}}
\subfloat[]{\includegraphics[width=.33\linewidth]{fig/counts_templates/VVZZ_2e2nu_nj3p_Sig.pdf}}
\end{center}
\end{figure}

\begin{figure}[hbt]
\begin{center}
\subfloat[]{\includegraphics[width=.33\linewidth]{fig/counts_templates/qqZZ_2e2nu_nj0_count.pdf}}
\subfloat[]{\includegraphics[width=.33\linewidth]{fig/counts_templates/qqZZ_2mu2nu_nj0_count.pdf}}
\subfloat[]{\includegraphics[width=.33\linewidth]{fig/counts_templates/qqZZ_2e2nu_nj1_count.pdf}}\\
\subfloat[]{\includegraphics[width=.33\linewidth]{fig/counts_templates/qqZZ_2mu2nu_nj1_count.pdf}}
\subfloat[]{\includegraphics[width=.33\linewidth]{fig/counts_templates/qqZZ_2e2nu_nj2_count.pdf}}
\subfloat[]{\includegraphics[width=.33\linewidth]{fig/counts_templates/qqZZ_2mu2nu_nj2_count.pdf}}\\
\subfloat[]{\includegraphics[width=.33\linewidth]{fig/counts_templates/qqZZ_2e2nu_nj3p_count.pdf}}
\subfloat[]{\includegraphics[width=.33\linewidth]{fig/counts_templates/qqZZ_2mu2nu_nj3p_count.pdf}}
\end{center}
\end{figure}
\clearpage

\section{Signal sample fitting template histogram relative error of background and signal}
\label{apdx:templates_hist}
\begin{figure}[hbt]
\begin{center}
\subfloat[]{\includegraphics[width=.33\linewidth]{fig/ggZZ_templates/ggZZ_2e2nu_nj0_Bkg.pdf}}
\subfloat[]{\includegraphics[width=.33\linewidth]{fig/ggZZ_templates/ggZZ_2e2nu_nj0_Sig.pdf}}
\subfloat[]{\includegraphics[width=.33\linewidth]{fig/ggZZ_templates/ggZZ_2e2nu_nj1_Bkg.pdf}}\\
\subfloat[]{\includegraphics[width=.33\linewidth]{fig/ggZZ_templates/ggZZ_2e2nu_nj1_Sig.pdf}}
\subfloat[]{\includegraphics[width=.33\linewidth]{fig/ggZZ_templates/ggZZ_2e2nu_nj2_Bkg.pdf}}
\subfloat[]{\includegraphics[width=.33\linewidth]{fig/ggZZ_templates/ggZZ_2e2nu_nj2_Sig.pdf}}\\
\subfloat[]{\includegraphics[width=.33\linewidth]{fig/ggZZ_templates/ggZZ_2e2nu_nj3p_Bkg.pdf}}
\subfloat[]{\includegraphics[width=.33\linewidth]{fig/ggZZ_templates/ggZZ_2e2nu_nj3p_Sig.pdf}}
\end{center}
\end{figure}

\begin{figure}[htb]
\begin{center}
\subfloat[]{\includegraphics[width=.33\linewidth]{fig/ggZZ_templates/ggZZ_2mu2nu_nj0_Bkg.pdf}}
\subfloat[]{\includegraphics[width=.33\linewidth]{fig/ggZZ_templates/ggZZ_2mu2nu_nj0_Sig.pdf}}
\subfloat[]{\includegraphics[width=.33\linewidth]{fig/ggZZ_templates/ggZZ_2mu2nu_nj1_Bkg.pdf}}\\
\subfloat[]{\includegraphics[width=.33\linewidth]{fig/ggZZ_templates/ggZZ_2mu2nu_nj1_Sig.pdf}}
\subfloat[]{\includegraphics[width=.33\linewidth]{fig/ggZZ_templates/ggZZ_2mu2nu_nj2_Bkg.pdf}}
\subfloat[]{\includegraphics[width=.33\linewidth]{fig/ggZZ_templates/ggZZ_2mu2nu_nj2_Sig.pdf}}\\
\subfloat[]{\includegraphics[width=.33\linewidth]{fig/ggZZ_templates/ggZZ_2mu2nu_nj3p_Bkg.pdf}}
\subfloat[]{\includegraphics[width=.33\linewidth]{fig/ggZZ_templates/ggZZ_2mu2nu_nj3p_Sig.pdf}}
\end{center}
\end{figure}

\begin{figure}[htb]
\begin{center}
\subfloat[]{\includegraphics[width=.33\linewidth]{fig/VVZZ_templates/VVZZ_2mu2nu_nj0_Bkg.pdf}}
\subfloat[]{\includegraphics[width=.33\linewidth]{fig/VVZZ_templates/VVZZ_2mu2nu_nj0_Sig.pdf}}
\subfloat[]{\includegraphics[width=.33\linewidth]{fig/VVZZ_templates/VVZZ_2mu2nu_nj1_Bkg.pdf}}\\
\subfloat[]{\includegraphics[width=.33\linewidth]{fig/VVZZ_templates/VVZZ_2mu2nu_nj1_Sig.pdf}}
\subfloat[]{\includegraphics[width=.33\linewidth]{fig/VVZZ_templates/VVZZ_2mu2nu_nj2_Bkg.pdf}}
\subfloat[]{\includegraphics[width=.33\linewidth]{fig/VVZZ_templates/VVZZ_2mu2nu_nj2_Sig.pdf}}\\
\subfloat[]{\includegraphics[width=.33\linewidth]{fig/VVZZ_templates/VVZZ_2mu2nu_nj3p_Bkg.pdf}}
\subfloat[]{\includegraphics[width=.33\linewidth]{fig/VVZZ_templates/VVZZ_2mu2nu_nj3p_Sig.pdf}}
\end{center}
\end{figure}

\begin{figure}[htb]
\begin{center}
\subfloat[]{\includegraphics[width=.33\linewidth]{fig/VVZZ_templates/VVZZ_2mu2nu_nj0_Bkg.pdf}}
\subfloat[]{\includegraphics[width=.33\linewidth]{fig/VVZZ_templates/VVZZ_2mu2nu_nj0_Sig.pdf}}
\subfloat[]{\includegraphics[width=.33\linewidth]{fig/VVZZ_templates/VVZZ_2mu2nu_nj1_Bkg.pdf}}\\
\subfloat[]{\includegraphics[width=.33\linewidth]{fig/VVZZ_templates/VVZZ_2mu2nu_nj1_Sig.pdf}}
\subfloat[]{\includegraphics[width=.33\linewidth]{fig/VVZZ_templates/VVZZ_2mu2nu_nj2_Bkg.pdf}}
\subfloat[]{\includegraphics[width=.33\linewidth]{fig/VVZZ_templates/VVZZ_2mu2nu_nj2_Sig.pdf}}\\
\subfloat[]{\includegraphics[width=.33\linewidth]{fig/VVZZ_templates/VVZZ_2mu2nu_nj3p_Bkg.pdf}}
\subfloat[]{\includegraphics[width=.33\linewidth]{fig/VVZZ_templates/VVZZ_2mu2nu_nj3p_Sig.pdf}}
\end{center}
\end{figure}


\section{Results and discussions}
\label{section:results_and_discussions}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
In this section, five DL models of semantic segmentation approach including  Res-UNet, VGG16 encoder-decoder, FCN-DenseNet, PSPNet, and GCN were evaluated on exemplary three damage scenarios of an RMS of the numerically calculated full wavefield interpolated at the bottom surface of the plate in order to identify the delamination.
Additionally, an experimental scenario was also used to evaluate the performance of the models to show the DL capabilities of generalization.
For each model, the mean and the max \(IoU\) are calculated and presented as a metric of comparison.
Moreover, the Accuracy of classification, Precision, Recall, and F1 score depicted in Eqs~(\ref{accuracy}-\ref{f1_score}) respectively, were calculated for each model.
\begin{equation}
	\rm Accuracy\ = \frac{TP+TN}{(Total\ number\ of\ tested\ samples)}
	\label{accuracy}
\end{equation}

\begin{equation}
	\rm Pricision\ =\frac{TP}{TP+FP}
	\label{pricisoin}
\end{equation}
\begin{equation}
	\rm Recall\ = \frac{TP}{TP+ FN}
	\label{recall}
\end{equation}
\begin{equation}
	\rm F1 score =\frac{2 \times (Pricision\times Recall)}{(Pricision + Recall)}
	\label{f1_score}
\end{equation}
where the Positive/Negative refers to the predicted output as (damage or non-damage) respectively, True Positive (TP) and True Negative (TN)  represent the correct classification, and the False Positive (FP) and False Negative (FN) represent the incorrect classification.

In this work, all semantic segmentation models were implemented and trained with Keras API~\cite{chollet2015keras} running on top of TensorFlow.
Further, we have replaced the previous NVIDIA RTX2080 GPU utilised in our work~\cite{Ijjeh2021} with a superior NVIDIA Tesla V100 GPU with \(32\)GB memory that enables us to increase the total number of learnable parameters of the DL models that will allow them to learn and extract more features.
For training purposes, we have used the K-fold cross-validation technique with five folds.
Accordingly, each model has trained for \(5\) iterations.
For each iteration, we calculated the mean accuracy of all folds.
Further, the number of epochs equals \(20\) per iteration.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Numerical scenarios}
In the following, three scenarios of numerical data regarding delamination of different locations, shapes, and angles are presented.
The first exemplary delamination scenario is shown in Fig.~\ref{fig:softmax_448}.
The delamination is located at the left edge of the plate, and it is surrounded by a line to represent its shape and location as shown in Fig.~\ref{fig:RMS_flat_shell_Vz_448}.
Figures~\ref{fig:unet_pred_448}-~\ref{fig:gcn_pred_448} show the predicted output of the Res-UNet, VGG16 encoder-decoder, PSPNet, FCN-DenseNet and GCN models, respectively.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
All models properly indicate the location of the delamination.
Moreover, pixels related to delamination location are clustered into a single spot without any additional noise.
VGG16 encoder-decoder and GCN models better visually resemble the actual shape of the delamination than other models.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{figure}[!h]
	\centering
	\begin{subfigure}[b]{0.47\textwidth}
		\centering
		\includegraphics[scale=1.0]{RMS_flat_shell_Vz_448_with_square_label.png}
		\caption{RMS bottom with label}
		\label{fig:RMS_flat_shell_Vz_448}
	\end{subfigure}
	\hfill
%	\begin{subfigure}[b]{0.47\textwidth}
%		\centering
%		\includegraphics[scale=1.0]{m1_rand_single_delam_448.png}
%		\caption{Ground truth}
%		\label{fig:m1_rand_single_delam_448}
%	\end{subfigure}
%	\hfil
	\begin{subfigure}[b]{0.47\textwidth}
		\centering
		\includegraphics[scale=1.0]{residual_unet_num_269.png}
		\caption{Res-UNet}
		\label{fig:unet_pred_448}
	\end{subfigure}
	\hfill
	\begin{subfigure}[b]{0.47\textwidth}
		\centering
		\includegraphics[scale=1.0]{VGG16_ecoder_decoder_num_269.png}
		\caption{VGG16 encoder-decoder}
		\label{fig:vgg16_pred_448}
	\end{subfigure}
	\hfill
	\begin{subfigure}[b]{0.47\textwidth}
		\centering
		\includegraphics[scale=1.0]{PSPNet_num_269.png}
		\caption{PSPNet}
		\label{fig:pspnet_pred_448}
	\end{subfigure}
	\hfill
	\begin{subfigure}[b]{0.47\textwidth}
		\centering
		\includegraphics[scale=1.0]{FCN_DenseNet_num_269.png}
		\caption{FCN-DenseNet}
		\label{fig:fcn_densenet_pred_448}
	\end{subfigure}
	\hfill
	\begin{subfigure}[b]{0.47\textwidth}
		\centering
		\includegraphics[scale=1.0]{GCN_num_269.png}
		\caption{GCN}
		\label{fig:gcn_pred_448}
	\end{subfigure}
	\caption{First delamination scenario based on numerical data}
	\label{fig:softmax_448}
\end{figure}
\clearpage
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
In the second delamination scenario, shown in Fig.~\ref{fig:385_softmax}, the delamination is located at the upper left corner of the plate, and it is surrounded by an ellipse to represent its shape and location as shown in Fig.~\ref{fig:RMS_flat_shell_Vz_385}.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
This is the most challenging damage scenario because of reflections coming from plate edges overshadows reflections from damage.
As a result, changes in RMS patterns are barely visible.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Figures~\ref{fig:Unet_Pred__softmax_385} -~\ref{fig:gcn_pred_385} show the predicted output of the Res-UNet, VGG16 encoder-decoder, PSPNet, FCN-DenseNet and GCN models, respectively.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
All models perform reasonably well, considering the difficult damage scenario.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{figure}[!h]
	\centering
	\begin{subfigure}[b]{0.47\textwidth}
		\centering
		\includegraphics[scale=1.0]{RMS_flat_shell_Vz_27_385_with_label.png}
		\caption{RMS bottom with label}
		\label{fig:RMS_flat_shell_Vz_385}
	\end{subfigure}
	\hfill
%	\begin{subfigure}[b]{0.47\textwidth}
%		\centering
%		\includegraphics[scale=1.0]{m1_rand_single_delam_385.png}
%		\caption{Ground truth}
%		\label{fig:m1_rand_single_delam_385}
%	\end{subfigure}
	\begin{subfigure}[b]{0.47\textwidth}
		\centering
		\includegraphics[scale=1.0]{residual_unet_num_17.png}
		\caption{Res-UNet}
		\label{fig:Unet_Pred__softmax_385}
	\end{subfigure}
	\hfill
	\begin{subfigure}[b]{0.47\textwidth}
		\centering
		\includegraphics[scale=1.0]{VGG16_ecoder_decoder_num_17.png}
		\caption{VGG16 encoder-decoder}			\label{fig:vgg16_pred__softmax_385}
	\end{subfigure}
	\hfill
	\begin{subfigure}[b]{0.47\textwidth}
		\centering
		\includegraphics[scale=1.0]{PSPNet_num_17.png}
		\caption{PSPNet}
		\label{fig:pspnet_pred__softmax_385}
	\end{subfigure}
	\hfill
	\begin{subfigure}[b]{0.47\textwidth}
		\centering
		\includegraphics[scale=1.0]{FCN_DenseNet_num_17.png}
		\caption{FCN-DenseNet}
		\label{fig:fcn_densenet_pred__softmax_385}
	\end{subfigure}
	\hfill
	\begin{subfigure}[b]{0.47\textwidth}
		\centering
		\includegraphics[scale=1.0]{GCN_num_17.png}
		\caption{GCN}
		\label{fig:gcn_pred_385}
	\end{subfigure}
	\caption{Second delamination scenario based on numerical data}
	\label{fig:385_softmax}
\end{figure}
\clearpage
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
The third delamination scenario is shown in Figure~\ref{fig:475_softmax}.
The delamination is located at the upper middle of the plate and it is surrounded by an ellipse to represent its shape and location as shown in Fig.~\ref{fig:RMS_flat_shell_Vz_475}.
Figures~\ref{fig:Unet_Pred__softmax_475} -~\ref{fig:gcn_pred_475} show the predicted output of the Res-UNet, VGG16 encoder-decoder, PSPNet, FCN-DenseNet and GCN models, respectively.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
In this case, the elliptical shape of delamination is best preserved by Res-UNet and FCN-DenseNet.
However, the highest value of IoU is obtained by GCN (see Table~\ref{tab:table_numerical_scenarios}).
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{figure}[!h]
	\centering
	\begin{subfigure}[b]{0.47\textwidth}
		\centering
		\includegraphics[scale=1.0]{RMS_flat_shell_Vz_27_475_with_label.png}
		\caption{RMS bottom with label}
		\label{fig:RMS_flat_shell_Vz_475}
	\end{subfigure}
	\hfill
%	\begin{subfigure}[b]{0.47\textwidth}
%		\centering
%		\includegraphics[scale=1.0]{m1_rand_single_delam_475.png}
%		\caption{Ground truth}
%		\label{fig:m1_rand_single_delam_475}
%	\end{subfigure}
	\begin{subfigure}[b]{0.47\textwidth}
		\centering
		\includegraphics[scale=1.0]{residual_unet_num_377.png}
		\caption{Res-UNet}
		\label{fig:Unet_Pred__softmax_475}
	\end{subfigure}
	\hfill
	\begin{subfigure}[b]{0.47\textwidth}
		\centering
		\includegraphics[scale=1.0]{VGG16_ecoder_decoder_num_377.png}
		\caption{VGG16 encoder-decoder}			\label{fig:vgg16_pred__softmax_475}
	\end{subfigure}
	\hfill
	\begin{subfigure}[b]{0.47\textwidth}
		\centering
		\includegraphics[scale=1.0]{PSPNet_num_377.png}
		\caption{PSPNet}
		\label{fig:pspnet_pred__softmax_475}
	\end{subfigure}
	\hfill
	\begin{subfigure}[b]{0.47\textwidth}
		\centering
		\includegraphics[scale=1.0]{FCN_DenseNet_num_377.png}
		\caption{FCN-DenseNet}
		\label{fig:fcn_densenet_pred__softmax_475}
	\end{subfigure}
	\hfill
	\begin{subfigure}[b]{0.47\textwidth}
		\centering
		\includegraphics[scale=1.0]{GCN_num_377.png}
		\caption{GCN}
		\label{fig:gcn_pred_475}
	\end{subfigure}
	\caption{Third delamination scenario based on numerical data}
	\label{fig:475_softmax}
\end{figure}
\clearpage
The \(IoU\) values for all models regarding the predicted delamination are presented in Table.~\ref{tab:table_numerical_scenarios}.
For the first and third scenarios, the GCN model has the highest \(IoU\) compared to the other models, and for the second scenario, the VGG16 encoder-decoder model has the highest \(IoU\) compared to the other models.
Further, in all models, the predicted outputs have no noise regarding delamination identification.
\begin{table}[]
	\centering
	\caption{\(IoU\) of Numerical scenarios}
	\label{tab:table_numerical_scenarios}
	\resizebox{\textwidth}{!}
	{
		\begin{tabular}{cccc}\hline
			Model & 1st scenario & 2nd scenario & 3rd scenario \\ \hline
			Res-UNet & \(0.498\) & \(0.782\) & \(0.816\)  \\
			VGG16 encoder-decoder & \(0.512\) & \(0.787\)  & \(0.662\)  \\
			FCN-DenseNet & \(0.734\)  & \(0.612\)  & \(0.866\)  \\
			PSPNet & \(0.389\) & \(0.496\) & \(0.646\)  \\
			GCN & \(0.791\) & \(0.696\) & \(0.875\) \\ \hline
		\end{tabular}
	}
\end{table}
Table~\ref{tab:table_iou} presents the mean and maximum values of \(IoU\) calculated for the previously unseen numerical test set (380 cases) for all models.
Table~\ref{tab:table_iou} shows that all models have a relatively high \(IoU\), indicating their ability to detect and localise the delamination, which is higher compared to the traditional signal processing techniques such as the adaptive wavenumber filtering presented in our previous work~\cite{Ijjeh2021}.
The mean \(IoU\) for the adaptive wavenumber filtering technique regarding the whole testing samples was \(0.373\) compared to the previous FCN-DenseNet model, which has a mean \(IoU\) of \(0.623\).
\begin{table}[]
	\centering
	\caption{Analysis of numerical data}
	\label{tab:table_iou}
	\begin{tabular}{ccc}\hline
		Model & mean \(IoU\) & max \(IoU\) \\ \hline
		Res-UNet & \(0.664\) & \(0.888\) \\
		VGG16 encoder-decoder & \(0.572\) & \(0.841\) \\
		FCN-DenseNet & \(0.680\) & \(0.920\) \\
		PSPNet & \(0.549\) & \(0.914\) \\
		GCN & \(0.763\) & \(0.931\) \\ \hline
	\end{tabular}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Further, in Table~\ref{tab:table_performance} the TP, TN, FP, and FN are presented for all models regarding the test set.
\begin{table}[]
	\centering
	\caption{Model classification performance}
	\label{tab:table_performance}
	\resizebox{\textwidth}{!}
	{
		\begin{tabular}{ccccc} \hline
			Model& True Positive & True Negative & False Positive & False Negative \\ \hline
			Res-UNet & 376 & 376 & 4 & 0 \\
			VGG16 encoder-decoder & 373 & 373 & 7 & 0 \\
			FCN-DenseNet & 378 & 378 & 2 & 0 \\
			PSPNet & 368 & 368 & 12 & 0 \\
			GCN & 380 & 380 & 0 & 0 \\ \hline
		\end{tabular}
	}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Moreover, Table~\ref{tab:evaluation_metric} presents the classification accuracy, precision, recall, and the F1-score values for all models as additional evaluation metrics.
As shown in Table~\ref{tab:evaluation_metric}, all models have high classification accuracy, which indicates that all the presented models are capable of predicting the presence of the delamination in all the numerically generated cases.
However, the best performance was achieved by GCN.
\begin{table}[]
	\centering
	\caption{Evaluation metric}
	\label{tab:evaluation_metric}
	\resizebox{\textwidth}{!}
	{
		\begin{tabular}{ccccc} \hline
			Model& Accuracy & Precision & Recall & F1-Score \\ \hline
			Res-UNet & \(0.994\)  & \(0.989\) &  \(1.00\)  & \(0.994\)  \\
			VGG16 encoder-decoder & \(0.991\)  & \(0.981\) & \(1.00\) &  \(0.991\)\\
			FCN-DenseNet & \(0.997\)  & \(0.994\) & \(1.00\)  & \(0.994\) \\
			PSPNet & \(0.984\) & \(0.968\) & \(1.00\) & \(0.984\) \\
			GCN & \(1.00\) & \(1.00\) & \(1.00\) & \(1.00\) \\ \hline
		\end{tabular}
	}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%Figures~\ref{fig:res_unet_iou_loss}-\ref{fig:GCN_iou_loss} show the accuracy and the loss graphs of the training and validation phases during epochs for Res-UNet, VGG16 encoder-decoder, FCN-DenseNet, PSPNet and GCN models, respectively.
%\begin{figure} [!h]
%	\centering
%	%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%	\begin{subfigure}[b]{0.47\textwidth}
%	 \centering		\includegraphics[width=\textwidth]{Unet_kfold_iou_per_epochs_softmax.png}	\caption{}
%	 \label{fig:unet_accuracy_metric}
%	\end{subfigure}
%	\hfill
%	\begin{subfigure}[b]{0.47\textwidth}
%	 \centering
%	 \includegraphics[width=\textwidth]{Unet_kfold_loss_per_epochs_softmax.png}
%	 \caption{}
%	 \label{fig:unet_loss_metric}
%	\end{subfigure}
%	\caption{Accuracy and loss of the Res-UNet model}
%	\label{fig:res_unet_iou_loss}
%\end{figure}
%	%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%\begin{figure}[!h]
%	\centering
%	\begin{subfigure}[b]{0.47\textwidth}
%		\centering
%		\includegraphics[width=\textwidth]{FCN_VGG16_iou_per_epochs_softmax.png}
%		\caption{}
%		\label{fig:vgg16_accuracy_metric}
%	\end{subfigure}
%	\hfill
%	\begin{subfigure}[b]{0.47\textwidth}
%		\centering
%		\includegraphics[width=\textwidth]{FCN_VGG16_loss_per_epochs_softmax.png}
%		\caption{}
%		\label{fig:vgg16_loss_metric}
%	\end{subfigure}
%	\caption{Accuracy and loss of the VGG16 encoder-decoder model}
%	\label{fig:Vgg16_iou_loss}
%\end{figure}
%	%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%\begin{figure}[!h]
%	\begin{subfigure}[b]{0.47\textwidth}
%	\centering
%	\includegraphics[width=\textwidth]{FCN_DenseNet_iou_per_epochs_softmax.png}
%	\caption{}
%	\label{fig:fcn_densenet_accuracy_metric}
%	\end{subfigure}
%	\hfill
%	\begin{subfigure}[b]{0.47\textwidth}
%	\centering
%	\includegraphics[width=\textwidth]{FCN_DenseNet_loss_per_epochs_softmax.png}
%	\caption{}
%	\label{fig:fcn_densenet_loss_metric}
%	\end{subfigure}
%\caption{Accuracy and loss of the FCN-DenseNet model}
%\label{fig:FCN_DenseNet_iou_loss}
%\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%\begin{figure} [!h]
%	\centering
%	\begin{subfigure}[b]{0.47\textwidth}
%		\centering
%		\includegraphics[width=\textwidth]{PSPNet_kfold_iou_per_epochs_softmax.png}
%		\caption{}
%		\label{fig:psp_accuracy_metric}
%	\end{subfigure}
%	\hfill
%	\begin{subfigure}[b]{0.47\textwidth}
%		\centering
%		\includegraphics[width=\textwidth]{PSPNet_kfold_loss_per_epochs_softmax.png}
%	\caption{}
%	\label{fig:psp_loss_metric}
%	\end{subfigure}
%	\caption{Accuracy and loss of the PSPNet model}
%	\label{fig:PSPNet_iou_loss}
%\end{figure}
%	%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%\begin{figure} [!h]
%	\centering
%	\begin{subfigure}[b]{0.47\textwidth}
%		\centering
%		\includegraphics[width=\textwidth]{GCN_kfold_iou_per_epochs_softmax.png}
%		\caption{}
%		\label{fig:gcn_accuracy_metric}
%	\end{subfigure}
%	\hfill
%	\begin{subfigure}[b]{0.47\textwidth}
%		\centering
%		\includegraphics[width=\textwidth]{GCN_kfold_loss_per_epochs_softmax.png}
%		\caption{}
%		\label{fig:gcn_loss_metric}
%	\end{subfigure}
%	\caption{Accuracy and loss of the GCN model}
%	\label{fig:GCN_iou_loss}
%\end{figure}
%\clearpage
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Moreover, the total number of parameters in any DL model is a sum of the trainable parameters (weights of convolution filters) and non-trainable parameters (biases and pooling filters).
Trainable parameters are continuously updated until we reach the minimum loss value, while the non-trainable parameters are not changed during the whole training process.
Table~\ref{tab:table_parameters} shows the total number of parameters for all implemented models.
Further, the total number of parameters can reflect the computation complexity of the model.
It can be noted that as the number of total parameters increases, the required time for training increases.
\begin{table}[]
	\centering
	\caption{Model parameters}
	\label{tab:table_parameters}
		\begin{tabular}{cc}\hline
			Model &  Total parameters (\(\approx\)) \\ \hline
			Res-UNet & \(52\times 10^6\) \\
			VGG16 encoder-decoder & \(37.3\times 10^6\)  \\
			FCN-DenseNet & \(2.5\times 10^6\) \\
			PSPNet & \(6.6\times 10^6\) \\
			GCN & \(36\times 10^6\) \\ \hline
		\end{tabular}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Experimental scenario}
In this scenario, an experimental case of CFRP with Teflon insert as artificial delamination is investigated presented in Fig.~\ref{fig:Exp_ERMS_teflon}.
Similar to the synthetic data set, we applied a frequency of \(50\) kHz to excite a signal in a transducer placed at the centre of the plate.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
A0 mode wavelength for this particular CFRP material at such frequency is about 20 mm.
The measurements were performed by Polytec PSV-400 SLDV on a bottom surface of the plate of dimensions 500 by 500 mm.
The measurement grid spacing was 1 mm and sampling frequency 512 kHz.
The measured full wavefield was further processed by an energy compensated RMS takes into account wave attenuation.
The results of such operation are shown in Fig.~\ref{fig:Delamination}.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
The delamination is surrounded by a square frame representing its shape and location.
Figures~(\ref{fig:unet_exp_7_} - \ref{fig:gcn_exp}) shows delamination prediction maps for Res-UNet, VGG16 encoder-decoder, PSPNet, FCN-DenseNet and GCN models, receptively.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
The performance of the models were evaluated by using IoU metric which takes into account not only damage location but also its shape and size.
The following results were obtained: the Res-UNet IoU = 0.577, the VGG16 encoder-decoder \(IoU = 0.624\), the
PSPNet \(IoU = 0.488\), the FCN-DenseNet \(IoU = 0.537\) and the GCN \(IoU =
0.723\).
Similarly to the numerical dataset, the best accuracy was achieved by using GCN.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
As shown, the models are capable of precise detection and localisation of the delamination.
We can see that the models can identify the delamination with almost free noise, which indicates the models are capable of generalising and detecting the delamination on previously unseen data.
Considering the presented models were trained only on the numerically generated dataset, the models show high generalisation capability.
The performance of the models can be further improved when they are trained on the experimental and the numerical data, as new features will be learned.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{figure} [!h]
	\centering
	\begin{subfigure}[b]{0.47\textwidth}
		\centering
		\includegraphics[scale=1]{ERMS_with_label.png}
		\caption{ERMS CFRP Teflon inserted \& Label}
		\label{fig:Delamination}
	\end{subfigure}
	\hfill
	\begin{subfigure}[b]{0.47\textwidth}
		\centering
		\includegraphics[scale=1]{residual_unet_decoder_exp_7.png}
		\caption{Res-UNet}
		\label{fig:unet_exp_7_}
	\end{subfigure}
	\hfill
	\begin{subfigure}[b]{0.47\textwidth}
		\centering
		\includegraphics[scale=1]{VGG16_ecoder_decoder_exp_7.png}
		\caption{VGG16 encoder-decoder}
		\label{fig:vgg16_exp_7_}
	\end{subfigure}
	\hfill
	\begin{subfigure}[b]{0.47\textwidth}
		\centering
		\includegraphics[scale=1]{pspnet_exp_7.png}
		\caption{PSPNet}
		\label{fig:pspnet_exp_7_}
	\end{subfigure}
	\hfill
	\begin{subfigure}[b]{0.47\textwidth}
		\centering
		\includegraphics[scale=1]{FCN_DenseNet_exp_7.png}
		\caption{FCN-DenseNet}
		\label{fig:fcn_densenet_exp}
	\end{subfigure}
	\hfill
	\begin{subfigure}[b]{0.47\textwidth}
		\centering
		\includegraphics[scale=1]{GCN_exp_7.png}
		\caption{GCN}
		\label{fig:gcn_exp}
	\end{subfigure}
	\caption{Experimental results}
	\label{fig:Exp_ERMS_teflon}
\end{figure}
\clearpage
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


\section{sub2 with spaces}
\label{sec:sub2 with spaces}

\begin{equation}
  f(x) = 42
  \label{eq:included sub with spaces is working}
\end{equation}


%======================================================================
\chapter{Conclusions and Future Considerations}
%======================================================================

A summary of the main conclusions from the development of the up-scaling framework as well as the implementation and testing of the framework are presented here. These conclusions represent a successful completion of the research objectives for the thesis. That being said, there are substantial limitations to this research. A series of recommendations are provided to address some of these limitations and to provide guidance on how to extend this research.

%----------------------------------------------------------------------
\section{Conclusions}
%----------------------------------------------------------------------
A multi-scale framework for up-scaling \acrshort{dem} simulations has been developed to address the computational demands of simulating microscale phenomenon in a macroscale domain in the context of NFR. Up-scaling is achieved by matching homogenized stress-strain curves from REV-scale \acrshort{dem} simulations to single element continuum models using \acrshort{pso} and LMA optimization algorithms. A Drucker-Prager plasticity model with ductile damage is implemented in the \acrshort{cdm} model to empirically capture the effect of the degradation (damage) of the NFR as deformation takes place.

\subsection*{1. Deformable DEM Homogenization}

Homogenization algorithms were developed for homogenizing \acrshort{dem} simulations with deformable bodies to assess the spatially averaged stress-strain behaviour of the REV from the microscale displacements, strains, and stresses. In this homogenization process, the resultant inter-block contact forces and block displacement from the \acrshort{dem} simulations are converted to average stresses and strains. To apply the homogenization algorithms, a method of automatically assessing a suitable REV given a sufficiently large domain was developed. These algorithms were implemented in Python\textsuperscript{TM} as the HODS software, which was used as a module for MOUSE.

\subsection*{2. Parameterization Methodology}

Two examples of the parameterization methodology are presented. Here, the key parameters required to capture the salient features of the model are isolated in order to be able to run the parameter estimation algorithms effectively. the main aspect of this parameterization methodology is the functional assumptions of the hardening/softening and damage evolution functions. In addition, the parameters are rewritten in term of physically meaningful parameters to provide more insight into the mechanics. The Drucker-Prager model with ductile damage is shown to be a reasonable \acrshort{cdm} model approach to represent NFR in a continuum context, including effects of pressure dependent yield and the triaxiality based damage initiation criterion. Compared to a full \acrshort{dem} simulation, the \acrshort{cdm} model shows a good fit pre-damage, but is unable to emulate the subtle post-yield oscillations arising from non-continuous yielding in the NFR.

\subsection*{3. Up-Scaling Framework}

MOUSE software was created and written in Python\textsuperscript{TM} to provide an implementation of the up-scaling framework presented in this thesis using in house and third-party software modules. The software itself provides a platform through which the four software elements of the up-scaling framework (\acrshort{dem} module, homogenization module, parameter estimation module, and macroscale module) can communicate with each other. The communication is facilitated by MOUSE through modules which wrap the third party software in such a way that the I/O routines to and from the modules are performed in a consistent capacity regardless of the third party software being used. A consistent set of data protocols were developed for the modules to effectively transfer data between them.

\subsection*{4. Framework Verification}

The parameter estimation module was tested and yielded an appropriate parameter set that both matched the \acrshort{dem} data and provided realistic parameters. Additionally, the Drucker-Prager model with ductile damage was found to provide a far superior fit than the damage plasticity model for quasi-brittle materials. Most importantly, the DNS of the slope stability analysis showed that with this up-scaling framework, very significant computational gains can be had with an acceptable error. Very comparable results ($<5\%$ error) to full \acrshort{dem} solutions were obtained with the \acrshort{cdm} method but required two orders of magnitude less computational time. The computational demands were again able to be reduced by another order of magnitude by using a selectively refined continuum mesh at the locations in the domain with high stress gradients.

%----------------------------------------------------------------------
\section{Recommendations}
%----------------------------------------------------------------------

The main limitation of the presented up-scaling implementation is the macroscale constitutive model module. In the current ABAQUS\textsuperscript{TM} module, the constitutive models consider only isotropic behaviour. In addition, the model does not consider the effects of pore pressure in the rock mass or fluid flow in any capacity. Though the isotropic assumptions for the elasto-plastic constitutive relationships are likely sufficiently accurate, future implementations of the macroscale constitutive model should consider anisotropic damage behaviour, as anisotropic implementations are found to be completely insufficient. In the case of the Drucker-Prager model with ductile damage, the exponential Johnson-Cook triaxiality based damage initiation criterion provides an excellent fit for monotonic loading, but does not model cyclic loading well. Here, it would be ideal for the cyclic loading capacity of the damage plasticity model for quasi-brittle materials to be incorporated as well. Ultimately, the available damage material subroutines in ABAQUS\textsuperscript{TM} are insufficient for the key physical characteristics in the system to be captured. As such, a custom anisotropic damage implementation is recommended for the macroscale constitutive model.

Retrospectively, the functional form of the hardening curve is overly complex. Though it is often necessary to model the softening of the material in the plasticity model, with \acrshort{cdm} the damage can implicitly model the softening behaviour. Here, it is observed that for the Barcelona model used for the hardening/softening curve, only the hardening portion of the curve is ever used. As such, for future implementations of the plasticity hardening functions, a simpler exponential function could be applied which would also have the benefit of decreasing the number of parameters that need to be estimated, leading to more consistent solutions and faster convergence of the optimization algorithms.

Furthermore, it is speculated by the author that portions of the parameterization methodology could be modified to yield more consistent solutions. In the parameterization formulations presented in this thesis, too much emphasis was placed on creating physically meaningful parameters rather than numerically consistent parameters. This inconsistency is the case with certain paired parameters if one of the parameters is highly sensitive.

Additionally, effectively searching a 11+ dimensional parameter space is computationally expensive. By dividing the problem and exploiting features of the curves and constitutive models, it may be possible to increase the convergence rate and effectively get a better, faster solution. Instead of searching the entire parameter space, it is possible to split the parameters into groups (e.g. pre-damage and post damage). Here, the damage parameters don't actually affect the plasticity calculations until damage is initiated. As such, the plasticity parameters can be estimated using the pre-damage curve and the damage parameters can subsequently be estimated using the post-damage curve.

A more rigorous examination of other available optimization routines and associated optimization parameters would be another way to potentially reduce the computational cost and increase the accuracy of the parameter estimation process. \acrshort{pso} and LMA were used in this research, but Dynamically Dimensioned Search (DDS),
Real-Coded Genetic Algorithm (RGA) and Simulated Annealing (SA) were also briefly investigated. As previously stated, rigorously assessing the most effective algorithm was not a priority of this research, but the \acrshort{pso} + LMA combination was chosen for its simplicity and effectiveness. The other algorithms, if properly applied may provide a faster and more accurate solution.

For this up-scaling methodology to be more accurate, 3D \acrshort{dem} simulations are required to capture accurate physical responses of these complex systems. In addition to modifying the \acrshort{dem} simulations, the associated homogenization algorithms would have to be modified to provide the 3D stress and strain tensors.

Hydro-mechanically coupled \acrshort{dem} simulations are also an important consideration for more accurate simulations. Here, the homogenization module should be modified to assess the homogenized fluid properties such as pore pressure and flow velocity vectors and the macroscale model needs to be modified to simulate poroelastic physics.

Ultimately, rigorous validation studies should be done with real-world applications to show the viability of this approach. Incorporating some of the above reccomendations will allow this up-scaling framework to validly be applied to complex geomechanical problems in order to avoid the computational demands of \acrshort{dem} modelling.