\chapter{Effect of measurement uncertainty}
\label{cha:error}
% **************************** Define Graphics Path **************************
\ifpdf
    \graphicspath{{Chapter4/Figs/Raster/}{Chapter4/Figs/PDF/}{Chapter4/Figs/}}
\else
    \graphicspath{{Chapter4/Figs/Vector/}{Chapter4/Figs/}}
\fi

% **************************** Chapter Abstract ******************************
\leftskip=1cm
\noindent
\emph{Observations are inevitably contaminated with measurement uncertainty, which is a predominant source of uncertainty in some cases. In reliability analysis, probabilistic models are typically fitted to measurements without considering this uncertainty. Hence, this chapter intends to explore the effect of this simplification on structural reliability and to provide recommendations on its treatment. Statistical and interval-based approaches are used to quantify and to propagate measurement uncertainty. They are critically compared by analyzing ground snow measurements, which are often affected by large measurement uncertainty. It is propagated through the mechanical model of a generic structure to investigate its effect on reliability.
%Parametric studies facilitate to analyze the effect of key parameters, such as measurement uncertainty, coefficient of variation of ground snow load, and distribution type. The interval analysis is performed as a hybrid interval-probabilistic analysis. Measurements are represented as intervals and probabilistic model is then fitted to them. Thus, snow parameters and the reliability index are also interval variables; other random variables are described by standard probabilistic distributions. Implementation of the statistical approach is based on the frequentist paradigm where the contamination mechanism is expressed in terms of random variables. This approach allows decoupling measurement uncertainty from a variable of interest.
The results indicate that measurement uncertainty may lead to significant (order of magnitude) underestimation of failure probability and should be taken into account in reliability analysis.
%If more information than interval endpoints is available, a statistical approach is recommended; otherwise the interval representation should be used.
Ranges of the key parameters are identified where measurement uncertainty should be considered. For practical applications, the lower interval bound and predictive reliability index are recommended as point estimates using interval and statistical analysis, respectively. The point estimates should be accompanied by uncertainty intervals, which convey valuable information about the credibility of results.
%Although general recommendations are given, treatment of measurement uncertainty should be handled on a case-specific basis.
}

\leftskip=0pt\rightskip=0pt

%****************************************************************************************
%****************************************************************************************
\section{Problem statement and the state of the art}

Models accounting for all uncertainties are of a considerable interest in structural reliability since these are the bases of design specifications, hence impacting the building and structure stocks of large regions. Snow is particularly important for light-weight structures for which it is typically the governing action. To our knowledge, the effect of snow measurement uncertainty on structural reliability has not yet been studied and other probabilistic models are treated similarly in civil engineering. For instance, neither the joint European research on snow actions \citep{Sanpaolesi1998} or the JCSS Probabilistic Model Code \citep{JCSS_basis} provides any information on the treatment of measurement uncertainty and its effect. Therefore, the aim of this chapter is to explore the effect of this simplification on structural reliability and to provide recommendations on its treatment.

Observations are inevitably contaminated with measurement uncertainty (MU), which is a predominant source of uncertainty in some cases. Uncertainty is understood here as the lack of knowledge (epistemic) and natural variability (aleatory)\footnote{This division is subjective as conditioned on the selected ``model universe''.}  not including known systematic error, which are assumed to be adjusted. In reliability analysis, probabilistic models are typically fitted to measurements without considering their uncertainty. This is the case for snow measurements where often only the snow depth is measured and the applied techniques makes the derived loads highly uncertain, for example, the uncertainty range can reach 50\% of the measured depth\footnote{Based on a personal correspondence with a meteorologist.}.

The World Meteorological Organization conducted a comprehensive comparative study on the then available solid precipitation measurement techniques and experimentally confirmed that measurements should be adjusted for wetting loss, evaporation loss, and wind induced undercatch \citep{Goodison1998}. They found that the snow catch ratio of the four most widely used gauges ranges from 20\% to 70\% at 6 m/s wind speed. Even for automated systems, measurement error in solid precipitation can vary from 20\% to 50\% due to undercatch in windy conditions \citep{Rasmussen2012}. Although these mainly contribute to systematic error they indicate uncertainties in snow measurements as these errors cannot be exactly corrected. For instance coefficient of determination ($R^2$) values vary from 0.40 to 0.80 for the fitted wind correction equations at certain sites; these are associated with about 10\% standard error in catchment ratio. Additional uncertainty may be introduced if no site specific auxiliary data, e.g. wind speed measurements, are available \citep{Goodison1998}. These issues are not limited to snow measurements but valid for all evidence based models -- that is for every model -- although their importance may vary.

%****************************************************************************************
%****************************************************************************************
\section{Solution strategy}
%*****************************************************************************************
\subsection{Adopted approaches}

We assume that measurements are corrected for known systematic errors. Additionally, the following model is assumed to describe the connection between observed ($Y$) and real, true, physical ($X$) values, i.e. the variable of interest:
\begin{equation}
\label{eq:ro_link}
	\mathrm{(true, real\footnotemark)} X \xrightarrow[]{h(X,E)} Y \mathrm{(observed)}.
\end{equation}
\footnotetext{Herein we tacitly assume the existence of some objective reality independent of the observer.}
The $h(X, E)$ function represents the mathematical relationship between the true and observed random variables referred hereinafter as reality-observation link. $E$ covers the unknown processes contributing to measurement uncertainty. The recommended probabilistic models -- typically distributions -- in the literature are almost exclusively given for the true variable and not for the observed, potentially contaminated one. Possible reasons for this are that:
\begin{itemize}
	\item The contamination is commonly site- and measuring technique-dependent, thus no general recommendations can be given for the distribution of $Y$.
	\item The model type is often selected based on theoretical arguments considering the physical phenomena generating $X$, for example Normal distribution if $X$ is the result of summation; Lognormal if $X$ is the product of random variables; extreme value distribution if $X$ is related to extremes.
	\item Structural reliability ultimately depends on $X$ and not on $Y$, although we are limited to access only $Y$.
\end{itemize}

The last point is especially important since structures are subjected to actions coming from $X$ and not from $Y$; the latter is affected by our ignorance or inability to make accurate measurements (epistemic uncertainty). In a broader sense this also applies for $X$, but for now we remain in the commonly accepted model universe of engineering and treat $X$ as a random variable. If the distribution type of $X$ is known or agreed, then the reality-observation link uniquely determines the distribution of $Y$. Hence, if any measurement uncertainty is present, its distribution type almost certainly differs from the distribution of $X$. This is prevalently neglected while fitting distributions in civil engineering -- $Y$ is assumed to be distributed as $X$. This simplification is acceptable in some practical cases. This method is termed hereinafter as Approach~1 while it is referred to as Approach~2 when the difference between distributions is appreciated:
\begin{description}
	\item[Approach~1] Use the probabilistic model of true random variable ($X$) and treat the observations -- contaminated with measurement uncertainty -- ($\mathbf{y}$) as the realizations of this model: $\mathbf{y} \sim X$.
	\item[Approach~2] Differentiate between the distribution of true and observed random variables. Within this, the following two sub-approaches are considered:
	\begin{description}
		\item[Approach~2a] Representation of measurement uncertainty with intervals at the level of observations and propagating them to the derived parameters via interval analysis. As interval representation by nature contains no information about the reality-observation link, the decontamination of observations is not possible.
		\item[Approach~2b] Representation of measurement uncertainty with a probability distribution. Use a mathematical model ($h(X, E)$) to describe the connection between measurement uncertainty ($E$), true phenomenon ($X$), and observed phenomenon ($Y$). Based on this model and on observations ($\mathbf{y}$), infer the parameters of the true random variable ($X$). These issues are referred to as \textit{measurement error problems} in the literature \citep{Kondlo2010}.
	\end{description}
\end{description}

\noindent Additional assumptions for all considered approaches:
\begin{itemize}
	\item ${\mathbf{y}} = \left\{ {{y_1},{y_2},...,{y_n}} \right\}$ and ${\mathbf{x}} = \left\{ {{x_1},{x_2},...,{x_n}} \right\}$ each are independent, identically distributed realizations; $\mathbf{y}$ is contaminated with measurement uncertainty.
	\item The realizations of the true phenomenon ($\mathbf{x}$) and measurement uncertainty ($\boldsymbol{\epsilon}$) are mutually independent.
	\item The true phenomenon ($X$) follows arbitrary, but known distribution type.
	\item Only for Approach~2b: the measurement uncertainty ($E$) follows arbitrary, but known distribution type, and the reality-observation link is also known.
\end{itemize}

%Fig 1

%****************************************************************************************
%****************************************************************************************
\subsection{Uncertainty representation and propagation}
\label{sec:uncertainty_rep_prop}

%*****************************************************************************************
\subsubsection{Interval analysis}

Interval representation is one possible approach to quantify uncertainty in an observed variable: the width of the interval expresses our uncertainty (Figure \ref{fig:obs_with_interval}). In this concept the true value is certainly within the interval but we know nothing about how likely it takes a particular value from that. In other words no probability distribution function is assumed over the interval, thus it expresses greater ignorance than probability distributions can \citep{Huber2010}.
The basic objective of interval analysis is to propagate the interval uncertainty of input variables to the outputs. Its main challenge is to calculate the interval bounds without overestimating them. This typically occurs if floating point computations are simply replaced by intervals and caused by interval dependency \citep{Moore2009}. Since the operators are typically not known explicitly and are non-monotonic, special algorithms are needed to obtain sufficiently narrow approximate interval bounds.
Interval analysis is traditionally used to model floating point truncation error in numerical computations; however, it is also successfully applied to various civil engineering issues, for instance, reliability of structures \citep{Qiu2008} and systems \citep{Qiu2007}. \citet{Rao2015} analyzed the effect of incorrect fitting on trusses and frames using mixed interval finite element formulation, using intervals to model fabrication errors. \citet{Muhanna2015} demonstrated the feasibility of non-linear interval finite element analysis for beam-column structures. In their study geometric, material and load uncertainties are modeled with intervals.
In this study the general definition of interval variables is used and constrained numerical optimization is applied to find the interval endpoints. This is motivated by the readily available optimization algorithms, and its feasibility due to the analyzed simple, computationally cheap examples. For computationally demanding models more efficient algorithms are available \citep{Zhang2010, Alibrandi2015, Muhanna2015}.
Intervals in this study are defined by midpoint and radius ($\epsilon_\mathrm{r}$), the midpoint is taken as the observed value, $y_i$, see Figure \ref{fig:obs_with_interval}. In this approach, the true value is assumed to be certainly within the interval given the modeling assumptions are valid.
\begin{figure}[htbp!]
	\centering
	\includegraphics[]{obs_with_interval.pdf}
	\includegraphics[]{MU_interval.pdf}
	\caption{Interval representation of measurement uncertainty (black) on a sorted random sample (red). The sample is generated from $Q_1$ with properties given in Table \ref{tab:prob_models_mu} and $CV_{Q_1} = 0.2$.}
	\label{fig:obs_with_interval}
\end{figure}

%*****************************************************************************************
\subsubsection{Statistical analysis}
An alternative approach to represent measurement uncertainty is statistical by means of probability distributions. The likelihood function depends on the reality-observation link (Eq.\ref{eq:ro_link}). This connection is also uncertain, but for simplicity, known relationship is assumed here and a possible treatment of this uncertainty is discussed in Section \ref{sec:discussion_mu}. Algebra of random variables can be used to obtain the likelihood function reflecting the distribution of involved random variables and the reality-observation link:
\begin{equation}
\label{eq:mu_like}
	L\left( {{{\boldsymbol{\theta }}_X},{{\boldsymbol{\theta }}_E}|{\mathbf{x}},{\boldsymbol{\epsilon }}} \right) = \prod\limits_{i = 1}^n {p\left( {h({x_i},{\epsilon _i})|{{\boldsymbol{\theta }}_X},{{\boldsymbol{\theta }}_E}} \right)}
\end{equation}
where ${\boldsymbol{\theta }}_X$ and ${\boldsymbol{\theta }}_E$ are the parameters of true and measurement uncertainty random variables, respectively.
In Approach~1, this means no additional complication because the observations are assumed to be distributed as the true random variable since the reality-observation link is neglected. However, in Approach~2b the likelihood function should be constructed to remove the effect of measurement uncertainty ($E$) from the variable of interest ($X$).
The \textit{measurement error problem} arises in many areas where only the contaminated values are attainable to the observer but the interest lays in the inference of true, uncontaminated values. Among others, these areas include astronomy, econometrics, biometrics, medical statistics, and image reconstruction \citep{Stefanski2000, Koen2009, Meister2009}. A straightforward solution is to construct the likelihood function (Eq.\ref{eq:mu_like}) and to infer the parameters of the variable of interest ($X$) by a selected method. To our knowledge this approach has not been applied in civil engineering yet.
Maximum likelihood method is used herein to infer the parameters in the statistical formulation of the measurement uncertainty problem.
Additive and multiplicative reality-observation links are considered. For the additive relationship: $Y = X + E$, the density function of the sum of two independent, continuous random variables is obtained by convolution:
\begin{equation}
\label{eq:conv}
	{f_Y}\left( y \right) = \left( {{f_X} * {f_E}} \right)\left( y \right) = \int\limits_{ - \infty }^\infty  {{f_X}\left( {y - x} \right)}  \cdot {f_E}\left( x \right) \cdot {\mathrm{d}}x.
\end{equation}
Here, for convenience the $p(.)$ notation of density functions is replaced with one that identifies the function elsewhere than in the argument, $f_X(x) \equiv p(x)$.
The integral can be efficiently solved by utilizing Fourier transformation since afterwards it reduces to a point-wise multiplication. Here, the fast-Fourier transformation is used to accomplish this task. For the multiplicative relationship: $Y = X \cdot E$, the density function of the product of two independent, continuous random variables is obtained by computing the following integral:
\begin{equation}
\label{eq:prod}
	{f_Y}\left( y \right) = \int\limits_{ - \infty }^\infty  {{f_X}\left( x \right)}  \cdot {f_E}\left( {\frac{y}{x}} \right) \cdot \frac{1}{{\left| x \right|}} \cdot {\mathrm{d}}x.
\end{equation}
This can be efficiently solved by Mellin transformation but here the integral is directly calculated due to the small computational burden.
Sampling variability (parameter estimation uncertainty) is accounted for by using the predictive reliability index, $\tilde \beta$ \citep{Kiureghian1989}:
\begin{equation}
\label{eq:predi_beta}
	 \tilde \beta  = \frac{{{{\mathrm{mean}}_B}}}{{\sqrt {1 + {{\mathrm{std}}_B}^2} }} \approx \frac{{{\mathrm{median}_B}}}{{\sqrt {1 + {{\left( {1.483 \cdot {{\mathrm{mad}}_B}} \right)}^2}} }}
\end{equation}
where $B$ is the posterior reliability index, std and mad are the standard deviation and median absolute deviation of $B$, respectively. The formulation with median and mad are used in this chapter, as that is more robust to outliers. Eq.\ref{eq:predi_beta} is an approximation as it is valid only for Normal distributed $B$. Additionally, the statistics are estimated from repeated analyses, and no Bayesian formulation of the reliability problem is used, even though that was used to derive the formula. For this study it is deemed sufficiently accurate to indicate tendencies and to identify critical cases.
\begin{figure}[htbp!]
	\centering
	\includegraphics[]{obs_with_prob_error.pdf}
	\includegraphics[]{MU_prob_distr.pdf}
	\caption{Illustration of probability distribution representation of measurement uncertainty (black) on a sorted random sample (red). The sample is generated from $Q_1$ with properties given in Table \ref{tab:prob_models_mu} and $CV_{Q_1} = 0.2$.}
	\label{fig:obs_with_prob_error}
\end{figure}

%****************************************************************************************
%****************************************************************************************
\section{Example: reliability of a generic structure}

%*****************************************************************************************
\subsection{Model description}
The reliability of a simple structural member is analyzed using a generic limit state function:
\begin{equation}
	 g(\mathbf{X}) = R - \left( {G + {Q_{50}}} \right).
	 %({\mathbf{X}})
\end{equation}

It represents a structure subjected to permanent ($G$) and variable ($Q_{50}$) actions, where the subscript 50 indicates 50-year reference period (common design working life). The probabilistic model of involved random variables are based on the recommendations of \citep{JCSS_basis} and summarized in Table \ref{tab:prob_models_mu}. For simplicity only the variable action is assumed to be affected by measurement uncertainty; it could be easily extended to more variables. Coefficient of variations 0.2, 0.4 for $Q_1$ represent annual snow maxima of mountains, highlands, while 0.6 characterizes lowlands in the Carpathian Region. The Lognormal model for snow maxima is typically adopted in the USA \citep{ASCE2010}, while the Gumbel model is widespread in Europe \citep{Sanpaolesi1998, JCSS_load}. The Normal and Gumbel distributions are light-tailed while the Lognormal is heavy-tailed. The adopted distributions and parameter ranges cover also other variable actions such as wind and thermal actions, thus the results can be readily generalized.

The annual maxima are assumed to be independent:
\begin{equation}
\label{eq:gfun_mu}
	{F_{50}}\left( q \right) = {F_1}{\left( q \right)^{50}}
\end{equation}
where $F(.)$ is the cumulative distribution function.

\begin{table}[htbp!]
\caption{Probabilistic models.}
\centering
\label{tab:prob_models_mu}
\small
	\begin{threeparttable}
    \begin{tabular}{llll}
    \toprule
    Variable name (symbol)  & Distribution & Mean & CV \\
    \midrule
    \rowcolor{lightgrey} Resistance ($R$)  & Lognormal & \tnote{*} & 0.10  \\
    Permanent action ($G$) & Normal &  8  & 0.10 \\
    \rowcolor{lightgrey} Variable action ($Q_1$)\tnote{\textdagger}  & Normal, Lognormal, Gumbel & 10 & $[0.20, 0.40, 0.60]$ \\
    \bottomrule
    \end{tabular}
    \begin{tablenotes}
    	\item[*] set to reach $\beta_\mathrm{target} = 3.8$ for each combination of inputs.
	    \item[\textdagger] the specified parameters are used to generate 50-element sample and the parameters of the model used in reliability analysis are inferred from it.
   	\end{tablenotes}
   	\end{threeparttable}
\end{table}

\subsection{Interval and reliability analysis}
\label{subsec:interval_reli}
To model the effect of measurement uncertainty, 50 random observations are generated from $Q_1$, these are treated as observed ($Y$) values as the reality-observation link by definition is unknown in interval representation (Figure \ref{fig:int_alg}). Then intervals are centered at observations and various interval radiuses are considered. Using these interval variables, the distribution of $Q_1$ is fitted by the method of moments, which is a widely used approach in civil engineering \citep{Sanpaolesi1998} and was proved to be robust e.g. for modeling hydrological extremes \citep{Madsen1997}. The hybrid interval-probabilistic reliability problem is solved using optimization and first order reliability method (FORM). An outcome of the analysis is an interval reliability index.

The upper bound of it is irrelevant from safety point of view and the lower bound is recommended for practical applications \citep{Qiu2007}. This is due to the special nature of intervals and how they represent uncertainty: the real value can be anything within the interval but one cannot assume that all points are equally likely (principle of indifference) at least because the consequence of specific values are not equal. Hence we chose the recommended, careful engineering approach and use the lower endpoint of the reliability index interval as representative value.
\begin{figure}[htbp!]
	\centering
	\includegraphics[]{interval_analysis_algorithm_.pdf}
	\caption{Algorithm of analyzing the effect of interval measurement uncertainty on reliability.}
	\label{fig:int_alg}
\end{figure}

\subsubsection{Full and approximate propagation of interval uncertainty}
As measurement uncertainty is expressed at the level of individual observations its full propagation yields to two distinct 50-dimensional constrained optimization problems that can be computationally demanding if each iteration step involves fitting a distribution function and solving a reliability problem. The computational burden can be considerably lessened by a two-step approximate technique where first the distribution parameters are fitted to the interval observations. Then only the interval representation of distribution parameters are used in further reliability analysis. Thus, the optimization with reliability analysis is reduced to a two-dimensional search space. Moreover, our experience show that the optimum is at the bounds so as it can be found by considering only the possible permutations of the parameter bounds.
The accuracy of full and two-step approximate uncertainty propagations are compared using Gumbel distributed $Q_1$. The results in terms of reliability indices are presented in Figure \ref{fig:beta_interval_full_approx}. The interval uncertainty is expressed as the ratio of interval radius and mean of annual maxima ($Q_1$). 0-10\% range is covered and it is assumed that all observations are contaminated with the same radius. For each coefficient of variation the mean of the resistance is set to reach the 3.8 target reliability level. This is performed by considering no measurement uncertainty ($\epsilon_\mathrm{r} = 0$) and using the parameters given in Table \ref{tab:prob_models_mu}, thus sampling variability has no effect. The calculated upper and lower reliability index endpoints are presented in the plots with solid and dashed lines for two-step and full propagations, respectively. Figure \ref{fig:beta_interval_full_approx} shows also the reliability index obtained by Approach~1. This is illustrated with a dotted line and is not affected by the assumed measurement uncertainty interval.
\begin{figure}[htbp!]
	\centering
	\includegraphics[]{beta_interval_full_approx.pdf}
	\caption{Reliability index intervals as the function of normalized measurement uncertainty radius ($\epsilon_\mathrm{r}/\mu_{Q_1}$) with full and approximate propagation of interval uncertainty.}
	\label{fig:beta_interval_full_approx}
\end{figure}

The plots show that the approximate technique slightly overestimates the accurate (full) reliability intervals, the largest difference is observed for $CV_{Q_1} = 0.2$ with large measurement uncertainty. Since in general the overestimation of the approximate technique is small, it is used in all further analysis. The sensitivity factor of the 50-year reference period maxima ($\alpha_{Q_{50}}$) is also displayed on the plots. It corresponds to a model without uncertainty in measurement and parameters. The decreasing interval range of $\beta$ with increasing $CV_\mathrm{Q1}$ is explained by the decreasing contribution of interval uncertainty to the full uncertainty of $Q_1$, i.e. aleatory uncertainty becomes dominating. Figure \ref{fig:explain_decr_beta_int} illustrates this shrinkage of uncertainty interval by comparing the transformed cumulative distribution functions with different coefficient of variations. The plots correspond to 50 particular random realizations; the same pattern is observed for other sets of random realizations.
\begin{figure}[htbp!]
	\centering
	\includegraphics[]{explain_decr_beta_int.pdf}
	\caption{Illustration of the shrinkage of uncertainty interval with increasing coefficient of variation but constant measurement uncertainty interval.}
	\label{fig:explain_decr_beta_int}
\end{figure}

\subsubsection{Effect on reliability index and required resistance}
Eq.\ref{eq:gfun_mu} is solved for Normal, Lognormal and Gumbel distributed variable action ($Q_1$) using the two-step approximation technique. The results are summarized in Figure \ref{fig:beta_interval_small_multiples}; they have the same rationale as is given for Figure \ref{fig:beta_interval_full_approx}. The light gray lines show the opening reliability interval with increasing measurement uncertainty for 20 random samples, each with 50 realizations. These are indicative of the effect of sampling variability: in this case this is entirely parameter estimation uncertainty due to the finite sample size. The results show that sampling variability -- with 50 realizations, which is typical for maxima model of climatic actions -- has significant effect on reliability. It is dominating over measurement uncertainty for small interval radiuses and comparable for larger values. The thick black lines are the median of the 20 sample sets. The reliability index without considering measurement uncertainty can be seen at the common starting point of the lower and upper bound lines. The difference of this value and the lower bound is of interest here as it indicates the extent of the non-conservative neglect of measurement uncertainty. Based on our experience, the difference is deemed significant if it is larger than 0.5. This level is indicated by a dashed horizontal line while the significant range with a red half line. With the selected target reliability level, this corresponds to more than six-fold increase in failure probability.
\begin{figure}[htbp!]
	\centering
	\includegraphics[]{beta_interval_small_multiples.pdf}
	\caption{Reliability index intervals as the function of the normalized measurement uncertainty radius ($\epsilon_\mathrm{r}/\mu_{Q_1}$). The gray lines represent 20 random samples, indicating sampling variability. The black lines are the median lower and upper interval endpoints of the reliability index. The red half line indicates the range where the lower endpoint of the reliability interval is significantly lower ($>0.5$) than the reliability calculated without measurement uncertainty ($\epsilon_\mathrm{r} = 0$).}
	\label{fig:beta_interval_small_multiples}
\end{figure}

The results suggest that moderate $\pm 4\%$ measurement uncertainty can lead to significant reduction of reliability level for mountains and highlands represented by $CV_{Q_1} = 0.2-0.4$. For the largest considered value of $CV_{Q_1} = 0.6$, the Gumbel model does not reach the limiting value. This indicates that for lowlands even a quite large $\pm 10\%$ measurement uncertainty has no practically significant effect. The reliability interval ranges indicate that even a small $\pm 2\%$ measurement uncertainty can lead to an order of magnitude uncertainty in the failure probability, see for instance the Lognormal distribution with $CV_{Q_1} = 0.2$. For larger measurement uncertainties, the width of the reliability intervals can be larger than 2.0; the widths are quite considerable for large $CV_{Q_1} = 0.6$ models too.
Measurement uncertainty thus seems to have a marked effect on structural reliability. The practical question then arises: what are its implications on design and how it should be accounted for? To examine this, we calculated the mean resistance ($\mu_R$) required to reach the target reliability with the lower bound of the reliability interval (Approach~2a). Then this value is compared to the $\mu_R$ required to reach the target reliability without explicit consideration of measurement uncertainty (Approach~1). The ratios of the mean values (with interval MU/without explicit MU) are illustrated in Figure \ref{fig:dspt_ratio_small_multiples}. These indicate how large adjustment might be needed in representative resistance values to meet target reliability in the presence of measurement uncertainty. The plots are structured and have the same rationale as Figure \ref{fig:beta_interval_small_multiples}. Based on our expertise, the ratio is deemed practically significant if it is larger than 1.1. This level is indicated by a dashed horizontal line while the significant range with a red half line. The small effect of sampling variability for Normal distribution is likely due to the small sensitivity factor of $Q_{50}$. On the contrary, sampling variability is quite considerable for Lognormal distribution. The selected threshold is reached for all distributions. The Lognormal model shows opposite trend, this might be attributed to its heavy tail. For this distribution the 1.1 threshold is reached at about 4\% normalized radius and the ratio can be over 1.4 for larger radiuses, which is a huge potential adjustment. The Gumbel distribution illustrates decreasing ratio with increasing coefficient of variation. For $CV_{Q_1} = 0.2$ (mountains), moderate $\pm 4\%$ measurement uncertainty can lead to significant $\mu_R$ ratio. For lowlands ($CV_{Q_1} = 0.6$), the ratio is over the selected threshold only for excessive measurement uncertainty $\pm 9\%$, which suggests that measurement uncertainty can be neglected for large values of $CV_{Q_1}$.
\begin{figure}[htbp!]
	\centering
	\includegraphics[]{dspt_ratio_small_multiples.pdf}
	\caption{Mean resistance ($\mu_R$) ratio for the variable action with and without measurement uncertainty as the function of the normalized measurement uncertainty radius ($\epsilon_\mathrm{r}/\mu_{Q_1}$). The red half line indicates the significant range where the ratio is larger than 1.1.}
	\label{fig:dspt_ratio_small_multiples}
\end{figure}

\subsection{Statistical and reliability analysis}
This section presents the statistical approach to quantify and propagate measurement uncertainty (Approach~2b). To model the effect of measurement uncertainty, 50 random observations are generated from $Q_1$ and treated as true ($X$) values. Then by using the assumed reality-observation link they are contaminated with measurement uncertainty. This is generated from a known, independent distribution ($E$). The algorithm is outlined in Figure \ref{fig:stat_alg}. Additive and multiplicative reality-observation links are assumed and the measurement uncertainty is taken as normally distributed with zero mean (unbiased). After the contamination of data, the information about the parameters of the underlying generating models -- with the exception of the zero mean of measurement uncertainty -- is disregarded and the maximum likelihood method is applied to decouple true values from measurement uncertainty. Finally, the model of decontaminated observations is used in reliability analysis. The sampling variability is again indicated by 20 samples and taken into account in an approximate manner through the predictive reliability index (Eq.\ref{eq:predi_beta}). The median and mean absolute deviation are calculated.
\begin{figure}[htbp!]
	\centering
	\includegraphics[]{statistical_analysis_algorithm_.pdf}
	\caption{Algorithm of analyzing the effect of measurement uncertainty on reliability using statistical technique.}
	\label{fig:stat_alg}
\end{figure}

\subsubsection{Decontamination of observations}
To illustrate the technique and the effect of decontamination, random realizations are generated from Gumbel distribution -- with parameters given in Table \ref{tab:prob_models_mu} -- and contaminated with measurement uncertainty (Figure \ref{fig:Gumbel_additive_me_smallm}). First, additive reality-observation relationship is assumed and Approach~1 and Approach~2b are used to infer the model parameters. The maximum likelihood method is used to obtain point estimates and the delta method is applied to construct $90\%$ confidence intervals to illustrate parameter estimation uncertainty \citep{Coles2001}.\mynote{ref to chapter 2!} The results for three cases of Gumbel distribution and two cases of measurement uncertainty with varying standard deviation are discussed only. The realizations and the fitted models are shown in Figure \ref{fig:Gumbel_additive_me_smallm}. It comprises return value-return period plots transformed to Gumbel space where the Gumbel distribution appears as a straight line. Though the plots are corresponding to a particular set of realizations, they convey reliably the trends and expected differences: (\textit{i}) Approach~1 typically overestimates the fractiles thus leading to lower reliability level and being conservative; (\textit{ii}) the difference between models increases with increasing return period. Due to the small sample size, the difference is affected by large sampling variability.
The calculations are repeated with multiplicative measurement uncertainty (Figure \ref{fig:Gumbel_prod_me_smallm}). The results correspond well with those obtained for the additive format. Furthermore, wider confidence intervals of Approach~2b compared with Approach~1 are observed. This is due to the larger model space where the same sample size allows less certain inference. This effect is less pronounced for the additive model.
For both models, Approach~1 is inherently biased since it is not using the correct likelihood function, while Approach~2b asymptotically converges to the true model. Thus, in the long run -- from theoretical point of view -- Approach~2b is better; however, Approach~1 seems to be generally conservative for the considered reality-observation links. This latter aspect is analyzed in more detail in the following section focusing on reliability index as a quantity of practical interest.
\begin{figure}[htbp!]
	\centering
	\includegraphics[]{Gumbel_additive_me_smallm_50_rng23.pdf}
	\caption{Gumbel distributions fitted to random realizations contaminated with additive measurement uncertainty using Approach~1 and Approach~2b. The point estimates (dashed lines) are accompanied by 90\% confidence intervals (dotted lines).}
	\label{fig:Gumbel_additive_me_smallm}
\end{figure}
\begin{figure}[htbp!]
	\centering
	\includegraphics[]{Gumbel_prod_me_smallm_50_rng14.pdf}
	\caption{Gumbel distributions fitted to random realizations contaminated with multiplicative measurement uncertainty using Approach~1 and Approach~2b. The point estimates (dashed lines) are accompanied by 90\% confidence intervals (dotted lines).}
	\label{fig:Gumbel_prod_me_smallm}
\end{figure}

\subsubsection{Effect on reliability index}
The effect of measurement uncertainty on reliability index is analyzed for the additive relationship considering Normal, Lognormal and Gumbel distributed true values and with coefficient of variation ranging from 0.2 to 0.6. The measurement uncertainty has Normal distribution with known zero mean and varying standard deviation. Consistently with the interval analysis in Section \ref{subsec:interval_reli}, the mean value of the resistance is determined to reach the target reliability without measurement uncertainty and parameter estimation uncertainty. Then the algorithm presented in Figure \ref{fig:stat_alg} is applied to generate contaminated observations, to decontaminate them, and to calculate the reliability index using the inferred parameters. The calculations are repeated for 20 samples with sample size of 50. The results in terms of reliability indices are shown in Figure \ref{fig:Normal_lognormal_additive_me_beta_smallm} and in Figure \ref{fig:Gumbel_additive_me_beta_smallm}. Grey and light blue solid lines are representing the 20 samples, and the corresponding thick solid and dashed lines are the median and predictive reliability indices, respectively. The difference between these latter two lines expresses the effect of parameter estimation uncertainty. For Normal distribution, this effect is small compared to Lognormal and Gumbel for which it is increasing with increasing standard deviation of measurement uncertainty. For Approach~2b it is typically larger than Approach~1 as the larger model space allows less certain inference with the same sample size. In case of Lognormal and Gumbel models, the ratio of the predictive and median failure probabilities can be as large as an order of magnitude or larger. Additionally, the plots show that the reliability index can considerably be overestimated when parameter estimation uncertainty is neglected. The salient large scatter of Approach~2b might be partially attributed to the unstable maximum likelihood estimators for small samples \citep{Hosking1985, Martins2000}.
\begin{figure}[htbp!]
	\centering
	\includegraphics[]{Normal_additive_me_beta_smallm_666_median.pdf}
	\includegraphics[]{Lognormal_additive_me_beta_smallm_666_median.pdf}
	\caption{Reliability indices as the function of the normalized standard deviation of measurement uncertainty ($\sigma_{E}/\mu_{Q_1}$). The thick solid lines are the median of the reliability indices while the thick dashed lines are the approximate predictive reliability indices.}
	\label{fig:Normal_lognormal_additive_me_beta_smallm}
\end{figure}

Comparing Approach~1 with Approach~2b for Normal and Gumbel distributions, the former approach yields to systematically lower reliability indices. The opposite trend observed for Lognormal distribution might be attributed to its heavy-tail. For Normal distribution, Approach~1 seems to be overly conservative, the median is well below the target reliability level.
For all distributions, Approach~1 is reasonably conservative with the exception of Lognormal distribution and coefficient of variation of 0.6. However, even in this case the predictive reliability index corrects the overestimation. Though for Normal distribution it is too conservative, the currently prevalent Approach~1 appears to be safely applicable to measurement uncertainty problems in case of additive reality-observation relationship. Approach~2b is sound from theoretical point of view; however, its median overestimates reliability level, thus the predictive reliability index is to be used to avoid underestimation of failure probability. Its larger parameter estimation uncertainty can lead large reduction in reliability index for small sample sizes.
\begin{figure}[htbp!]
	\centering
	\includegraphics[]{Gumbel_additive_me_beta_smallm_666_median.pdf}
	\caption{Reliability indices as the function of the normalized standard deviation of measurement uncertainty ($\sigma_{E}/\mu_{Q_1}$). The thick solid lines are the median of the reliability indices while the thick dashed lines are the approximate predictive reliability indices.}
	\label{fig:Gumbel_additive_me_beta_smallm}
\end{figure}

%****************************************************************************************
%****************************************************************************************
\section{Application example: Turbine hall of Paks Nuclear Power Plant}
\label{sec:turbine_mu}

The turbine hall of Paks Nuclear Power Plant is selected to demonstrate the effect of measurement uncertainty on a real life example. The structure is introduced in Annex~\ref{sec:paks}, here only the essential details, which are required to interpret the results, are provided. The interval approach is used to represent measurement uncertainty and the two-step approximation is applied to propagate interval uncertainty. Annual ground snow maxima related to the location are contaminated with measurement uncertainty, the rest of random variables are represented by probability distributions. The results in terms of reliability indices and failure probabilities are given in Figure~\ref{fig:interval_beta_turbine_hall}. One year reference period is used for all calculations of the frame.

\begin{figure}[htbp!]
	\centering
	\includegraphics[]{interval_beta_turbine_hall.pdf}
	\caption{Reliability index and failure probability intervals as the function of normalized measurement uncertainty radius ($\epsilon_\mathrm{r}/\mu_{Q_1}$). Comparison of Approach~1 and Approach~2a for the turbine hall of Paks Nuclear Power Plant.}
	\label{fig:interval_beta_turbine_hall}
\end{figure}

The widest reliability index interval is 0.2, which is obtained for the largest measurement uncertainty error radius. This small value is in agreement with the trend observed in the previous example (Figure~\ref{fig:beta_interval_small_multiples}). It is attributed to the large coefficient of variation of annual ground snow load that is dominating over measurement uncertainty. In this example, neglect of measurement uncertainty leads to 70\% underestimation of failure probability at worst; hence, the effect is negligible.

%****************************************************************************************
%****************************************************************************************
\section{Discussion}
\label{sec:discussion_mu}
One can distinguish two components of measurement uncertainty: the reality-observation link and the nature of the contamination $E$. The three approaches considered here differ in how they treat these two components. The present prevalent approach (Approach~1) neglects both components, thus entirely ignores the possibility that the real values are greater or smaller than the observed due to measurement uncertainty.

The interval approach (Approach~2a) expresses full ignorance in respect of reality-observation link and represents measurement uncertainty with intervals. Therefore, no decoupling of true values from measurement uncertainty is possible. Intervals should be used with caution because by definition values outside of the interval are impossible. This assumption is rarely met in civil engineering. Measurement uncertainty is often described on the basis of expert judgment and wide intervals are applied to almost surely capture real, unobserved values.

The statistical approach (Approach~2b) requires the knowledge of the reality-observation link and represents measurement uncertainty with distribution function. This is the only approach that can decontaminate the observations and can directly infer the variable of interest, true variable. This is important since structural reliability is dependent on the true variable. The statistical and interval analysis based approaches are conceptually different, thus they are only comparable on that level but not quantitatively. Their uncertainty representation is inherently distinct, thus there is no equivalency between interval and distribution based representations.

Additionally, it must be emphasized that another type of uncertainty – statistical uncertainty in parameter estimation and selection of distribution function – often needs to be taken into account in reliability analysis as it may be even more important as measurement uncertainty investigated here \citep{RozsasIABSE2015, RozsasESREL2015}. Bayesian paradigm is a natural choice to incorporate this uncertainty, consequently that is recommended for practical applications. For example, in real-life situations, the reality-observation link cannot be established with certainty. Yet, this uncertainty can be captured by using multiple models and averaging them with respect their goodness of describing the data, this can be achieved for example by Bayesian model averaging \citep{Hoeting1999}.

Although this study is limited by the considered distribution types, reality-observation functions, and parameter range, it is believed to cover many practically relevant random variables. The presented approaches and algorithms can be easily used for other distribution types and measurement error structure. An additional limitation of this study is that measurement uncertainty is considered only for the dominant variable action. However, it is anticipated that for other random variables the effect is smaller due to their typically smaller sensitivity factor. Moreover, measurement uncertainty is much smaller for other than climatic actions such as resistance and permanent actions. Furthermore, the effect of sample size should be analyzed in later works. It is believed that the outcomes would be similar for sample sizes ranging from 20 to couple of hundreds, which cover the majority of cases in civil engineering. More data would allow more certain model identification.

%\section{Application example}

\section{Summary and conclusions}
The current practice in probabilistic engineering treats observed data contaminated with measurement uncertainty as realizations of the true distribution, thus neglecting the contamination mechanism. Statistical and interval-based analyses are thus conducted to investigate the effect of this simplification on structural reliability. Extensive parametric analyses -- based on 50 realizations, which is a typical length of records for climatic actions -- reveal that:\\
If interval representation of measurement uncertainty is used:
\begin{itemize}
	\item Sampling variability (parameter estimation uncertainty) has significant effect on reliability: it is dominant over measurement uncertainty for small interval radiuses and comparable for large radiuses.
	\item For mountains and highlands, moderate $\pm 4\%$ measurement uncertainty -- relative to value of an observed variable -- can lead to significant reduction of reliability level. For lowlands, even a large $\pm 10\%$ measurement uncertainty has no significant effect. An effect is deemed significant if it yields to greater than six fold increase in failure probability compared with Approach~1 (neglect of measurement uncertainty).
	\item Reliability interval ranges indicate that a small $\pm 2\%$ measurement uncertainty can lead to reduction of 0.6 in reliability index. For larger measurement uncertainties, the width of the reliability intervals can be larger than 2.0.
	\item The effect of measurement uncertainty is more pronounced for low variability random variables where its contribution to the total uncertainty increases.
	\item Parameter ranges where Approach~1 often overestimates the reliability index are identified.
\end{itemize}
If statistical (distribution function) representation of measurement uncertainty is used:
\begin{itemize}
	\item It is demonstrated that the statistical approach can be used to decontaminate the observations, thus to access the variable of interest.
	\item The ratio of the predictive and median failure probabilities can be as large as an order of magnitude or larger ($\sim30$ for Lognormal distribution).
	\item If parameter estimation uncertainty is disregarded, the reliability index can be considerably overestimated.
	\item For all considered distributions with additive measurement uncertainty, Approach~1 is reasonably conservative in most cases.
\end{itemize}

Practical recommendations:
\begin{itemize}
	\item Figure \ref{fig:beta_interval_small_multiples} and Figure \ref{fig:dspt_ratio_small_multiples} can be used to identify cases when Approach~1 significantly overestimates reliability index. In such cases and when no or very limited information on measurement uncertainty is available, then interval analysis could be used, considering the lower bound of the reliability interval.
	\item If the reality-observation link is known then the statistical approach is recommended. For small and moderate sample sizes ($<100$), the predictive reliability index is recommended. For additive measurement uncertainty, Approach~1 is conservative.
	\item For point estimates, such as median, the reliability index should be accompanied by uncertainty intervals to indicate the credibility of results.
	\item For ground snow extremes at lowlands, Approach~1 provides a reasonable approximation, thus the effect of measurement uncertainty can be neglected. Otherwise more advanced analysis is recommended.
\end{itemize}

Assessment of measurement uncertainty should be region- and case-specific accounting for measuring techniques, and applied correction equations, thus involvement of meteorologists, analysts or other experts is beneficial. Moreover, the selected approach to propagate measurement uncertainty should always be based on the particular issue in question, acknowledging ``the degree of precision to which the nature of the subject admits''.


\chapter{Model's nomenclature}
\label{ap:nomen}

\textit{In this appendix we will report the nomenclature used in Chapter \ref{ch::model} and in Appendix \ref{ap:model} for model description.} \\

~\\

\begin{longtable}{ r p{11cm} }
  %\hline
  %\multicolumn{1}{|c|}{\textbf{Symbol}} & \multicolumn{1}{c|}{\textbf{Description}}                                    \\~\\
  %\hline
  $\left( x_w, y_w, z_w \right)$        & Real coordinates of the point in the world reference system.                  \\~\\
  %\hline
  $\left( x_f, y_f, z_f \right)$        & Projection of the principal point on the laser plane reference system.        \\~\\
  %\hline
  $\left( x_s, y_s, z_s \right)$        & Point coordinates in the tilted image plane reference system.                 \\~\\
  %\hline
  $\left( x_p, y_p, z_p \right)$        & Points coordinates in the image plane parallel to the sensor plane.           \\~\\
  %\hline
  $\left( x_p^d, y_p^d \right)$         & Distorted points coordinates in the image plane parallel to the sensor plane. \\~\\
  %\hline
  $\left( x_c, y_c \right)$             & Discrete points coordinates in the sensor reference system                    \\~\\
  %\hline
  $\phi$                                & Triangulation angle                                                           \\~\\
  %\hline
  $\alpha$                              & Angle offset in $y$ direction, with respect to $\phi$                         \\~\\
  %\hline
  $\beta$                               & Angle offset in $x$ direction, with respect to the optical axis               \\~\\
  %\hline
  $\rho$                                & Laser roll angle                                                              \\~\\
  %\hline
  $\gamma$                              & Laser pitch angle                                                             \\~\\
  %\hline
  $\upsilon$                            & Lens rotation in $y$ direction, due to Scheimpflug principle                  \\~\\
  %\hline
  $\chi$                                & Lens rotation in $x$ direction, due to Scheimpflug principle                  \\~\\
  %\hline
  $Roll$                                & Rotation along $x$ axis                                                       \\~\\
  %\hline
  $Pitch$                               & Rotation along $y$ axis                                                       \\~\\
  %\hline
  $Yaw$                                 & Rotation along $z$ axis                                                       \\~\\
  %\hline

  %\caption{Model's nomenclature and descriptions}
  %\label{tab:ap-nomen}
\end{longtable}


\chapter{Concept}

In this chapter the concept of transferring \gls{AutoML} concepts to clustering will be described.
For this, it will first be explained how meta-learning and the hyperparameter optimization methods were used to optimize the parameters of a clustering algorithm.
After this, the changes that had to made to the concept to also tackle the problem of algorithm selection  will be explained.

\section{Hyperparameter Optimization for Clustering}

\begin{itemize}
    \item First applied concepts on KMeans.
    \item Goal is to predict the number of clusters.
    \item For this, analyze which hyperparameter optimization methods are suitable.
    \item Also analyze which metrics are suitable for the different methods.
    \item So use internal/external metrics for the optimizers.
    \item Although external cannot be applied for new unseen datasets since they do not have class labels.
    \item But meta-learning can be used to warmstart optimizers.
    \item Warmstart means that the optimizers do not start with random models but with models that are suggestes by meta-learning model.
    \item For this, offline phase necessary.
    \item Assuming in the offline phase the number of clusters is known, it is also possible to use external metrics.
    \item Basic architecture shown in \cref{fig:concept_arch}.
    \begin{figure}
        \centering
        \includegraphics[width=\textwidth]{graphics/concept_architecture.png}
        \caption{General architecture of the concept divided in Online and Offline phase.}
        \label{fig:concept_arch}
    \end{figure}
    \item Offline phase: Train ``meta-model''
    \item Use optimizer to get configuration. Run KMeans on them and evaluate resulting according to metric.
    \item Then save result to Meta-DB.
    \item Also save the evaluation of the dataset together with the \textit{meta-features}.
    \item Meta-DB builds the meta-model.
    \item For the meta-model a k-NN classifier with $k=1$ is used.
    \item If new dataset has to be clustered in online phase, then extract meta-features (same as in offline phase).
    \item Then find the ``nearest'' dataset.
    This is basically where 1-NN is used to find the nearest dataset based on the $L1$ distance.
    \item Input is meta-features of the new dataset and these are used to build the 1-NN and find the nearest dataset.
    \item From the nearest dataset it is looked up for which configurations the lowest k deviation was measured in the offline phase.
    \item This configurations are then used to \textit{warmstart} the optimizer, which means the optimizer first evaluates this configurations.

\end{itemize}

\section{Algorithm Selection for Clustering}

\begin{itemize}
    \item Basically same as for Hyperparameter Selection.
    \item Instead of only using parameters of that one algorithm, the optimizer gets one additional parameter.
    \item Additional parameter is name of algorithm.
    \item So the optimizer predicts the name of the algorithm and the hyperparameters like done in \cite{ThorntonAuto-WEKA:Algorithms}.
    \item Limited to partitional clustering algorithms.
    \item Basically Bayes would also be applicable to other clustering algorithms, but for multi-arm bandit based methods the budget has to be defined somehow.
    To get a meaningful metric result one needs labels for all data points after each iteration if the budget is the number of iterations or even for time budget.
    This holds partitional clustering algorithms but not necessarily for other kinds of clustering algorithms.
    \item The architecture shown in \cref{fig:concept_arch} changes in that way that the optimizer does not only gives the k value that should be executed next, but also the algorithm.
    \item New architecture can be seen in \cref{fig:archAlgoSelection}.
    \begin{figure}
        \centering
        \includegraphics[width=\textwidth]{main/concept_architecture_algo_selection.png}
        \caption{General architecture of the concept to also include the algorithm selection.}
        \label{fig:archAlgoSelection}
    \end{figure}

\end{itemize}


\section{ General Framework for Group Deviation Detection
}
 \label{Sec:Framework}
In the last section, we clearly defined the problem of detecting group deviations in static and dynamic situations.
  This section elaborates  upon the general framework and underlying structure for GAD and GCD techniques. Discovering significant group deviations is related to three sub-problems.
\begin{enumerate}[1.]
\item {\it Group Structures}:  The definition of groups and the relationship between data instances is important to understand. %  clustering algorithm aggregate data instances into group structures.
\item {\it Statistical Properties}:
A variety of statistical properties may characterise group deviations. % For example, a user is able to examine statistical properties of interest such as proportions, location or dependence.
 If statistical properties  are  not adequately quantified then relevant group deviations cannot be identified.
\item  {\it Model Design}:
 The design of GAD and GCD techniques include data inputs,   model assumptions, learning approaches and output information.
\end{enumerate}

\subsection{Group Structures}
We now highlight examples of different problems associated with group structures.   A group is a collection of two or more related data instances where Tan et al.  \cite{Tan} state that a data instance is synonymous with terms such as  vector, sample, entity, observation, etc. %Many GAD techniques assume that features from members in a group are independent and identically distributed (iid) where
Members in a group can be related by  external information of known group labels such as in topic modelling where Xiong et al. \cite{FGM} consider a document as a group of  words or a corpus as a group of documents. When group structures are not previously known, clustering algorithm aggregate data instances based on similarity criterion.
 For example, Xiong et al. \cite{MGM} infer a spatial cluster of galaxies with distances closer than 1 megaparsecs while Wong et al. \cite{wong-rule} examine a demographic group of patients  based on certain categorical features.
%   The interpretation of results relies on initial definition and construction of group structures.

When group structures are unknown a priori,  clustering algorithms are applied. Common procedures minimise the cumulative  distance  between  members in a group to a central reference point. %These algorithms such as $k$-means or  nearest neighbors
 Clustering algorithm  for numerical values  are discussed in Jain \cite{jain2010} where a group of data instances contains members with similar features based on minimising distance metrics.
Steinbach \cite{steinbach2004} discusses issues with distance-based clustering algorithms as data points become difficult to differentiate in higher dimensions.   More sophisticated techniques have been specifically developed for group
%In particular,
 group deviation detection   %techniques infer clusters based on different criterion
 such as:
\begin{enumerate}[-]
\item Chen et al.  \cite{GLETS}  apply density-based spatial clustering based on Pearson's correlation and Euclidean distance between values of times series.
%\item Chen et al.  \cite{Chen2014} examine Pearson's correlation and Euclidean distance between values of times series.
\item    Yu et al. \cite{GLAD} examine individual features as well as pairwise connection data.  %infer groups containing a mixture of social roles for social media applications.
\item Soleimani and   Miller \cite{ATD} infer anomalous clusters of documents based on likelihood probabilities.
\item Dai et al. \cite{ERACD} cluster based on the ranking of feature values.
\end{enumerate}
Therefore there are many procedures for clustering data instances when group structures are previously unknown.

\begin{figure}[h]
\centering
   \begin{subfigure}{1\linewidth} \centering
     \includegraphics[width=6cm, height=4cm,trim=5cm 9.5cm 4.5cm 9cm]{FIGURES/ECGa}
     \caption{A collective anomaly (red dotted circle) in the ECG reading of a single patient.}
   \end{subfigure}
   \begin{subfigure}{1\linewidth} \centering
     \includegraphics[width=6cm,  height=4cm,trim=5cm 9.5cm 4.5cm 8.5cm]{FIGURES/ECG}
     \caption{A group anomaly (Patient 3) is observed when comparing several patients.}
   \end{subfigure}
  % \vspace{-1cm}
\caption{ In (a), a collective anomaly is a collection of related data instances that is anomalous with respect to time-dependent observations from  a single patient.
In (b), a group anomaly is a specific example of a collective anomaly where  the entire dataset involves a group of observations from multiple patients.
 } \label{Fig:ECG}
\end{figure}
%\vspace{1cm}

In our thesis, we do not use group anomaly and  collective anomaly interchangeably.  Chandola \cite{Chandola} defines a collective anomaly  as "a collection of related data instances that is anomalous with respect to the entire dataset."
A group anomaly is specific example of a collective anomaly where
the entire dataset involves multiple groups. % in the dataset.
To differentiate these terms, consider the electrocardiogram (ECG) example from Goldberger et al. \cite{Goldberger}.  Figure \ref{Fig:ECG} (a) highlights a collective anomaly (in the dotted red circle) that exhibits an irregular pattern as compared to the  entire dataset (time series of a single patient). For a GAD application,  Figure \ref{Fig:ECG} (b) illustrates ECG readings from several patients  where  Patient 3 represents a group anomaly as the statistical properties are significantly different.  %however this is also a collective anomaly as the entire dataset can be considered as information from all of the patients.
 A realistic GCD case  would involve a group of patients with ECG readings similar to Figure \ref{Fig:ECG} (a) where a large-scale phenomena affects all patients around $\tau=10$.

\subsection{Statistical Properties}
% SP - known vs unknown
 In some cases, a domain expert is interested in particular  statistical properties of groups. Statistical properties are measured and compared for group deviation detection. In terms of point-based and distributed-based group behaviours, point-based group deviations are characterised by a central location while distributed-based group deviations are characterised by statistical properties other than location such as scale, shape or dependence. In practice, group deviations are difficult to detect as a combination of statistical properties may  significantly differ. If a user is able to specify statistical properties of interest, group deviations are more easily detected and interpreted.

 Since there are also many ways to quantify statistical properties of groups, we describe common parametric and non-parametric measures for statistical properties in Table \ref{Tab:Des} where non-parametric measurements are more robust to individual outliers. %Statistical properties in Table \ref{Tab:Des}, capture group  behaviours  for continuous variables however other metrics are more suitable for discrete or categorical datasets.
   Given prior knowledge of statistical properties that characterise group deviations for GAD or GCD applications, a suitable method for discriminating groups is easily constructed. However when the nature of significant group deviations in terms of statistical properties is unknown, more   specialised techniques are required.

		\begin{table}[h]
%\renewcommand{\arraystretch}{1}
	\tabcolsep=0.1cm
	\begin{center}
   \scalebox{0.88}{
	\begin{tabular}{lccc  }
	\hline\\[-2mm]
		%\multirow{ 2}{*}{} & & & \\[2mm]
 Statistical & %Function &
	 Parametric Measures% in $\boldsymbol \alpha$
	 &  Non-parametric Measures %  in  $\boldsymbol \gamma$
	 \\[-1mm]  Property & & &
	 \\[2mm] \hline\\[-2mm]
		\multirow{ 2}{*}{Location } & %	\multirow{ 2}{*}{$h_1$ }&
	  $ \displaystyle\bar{ {X}}_v=\frac{1}{N} \sum_{n=1}^N X_{nv}$ & $  \displaystyle \hat{q}_v({0.5})$  \\
	 & \hspace{5mm}(mean) & (median)  \\%[2mm]
	  %
	\multirow{ 2}{*}{Scale} & %\multirow{ 2}{*}{$h_2$} &
	 $ \hat\sigma^2_v = \displaystyle	\frac{1}{N-1}\sum_{n=1}^{N} \big(X_{nv}-\bar {X}_v \big)^2$  &  % \displaystyle s^2 =	\(\stackunder[1pt]{$ \mbox{mediai}$}{	\scalebox{0.8}{$ 1\le i\le N$}} \) $\big (| X_n - q_{0.5}| \big)$
	$\displaystyle \hat{q}_v({0.75}) - \hat{q}_v({0.25})$ \\ %[-1mm]
	 & (variance) &(interquartile range)   \\[1mm]
	 Skewness & % $h_3$ &
	$\displaystyle\frac{1}{N} \sum_{n=1}^N \frac{( X_{nv}-\bar{X}_v)^3}{ \hat\sigma_v^3}  $  &
	$ \displaystyle \frac{\hat{q}_v({0.9}) + \hat{q}_v({0.1}) -2 \hat{q}_v({0.5}) }{  \hat{q}_v({0.9}) - \hat{q}_v({0.1})  }$\\[4mm] %\displaystyle  \hat{\mathcal{S}}=
	 Kurtosis &  %$h_4$ &
	 $\displaystyle\frac{1}{N} \sum_{n=1}^N \frac{( X_{nv}-\bar{X}_v)^4}{ \hat\sigma_v^4}  $ &
	$ \displaystyle\frac{\hat{q}_v({0.975}) -\hat{q}_v({0.025}) }{\hat{q}_v({0.75}) -\hat{q}_v({0.25}) }  $ \\[4mm] %\displaystyle \hat{\kappa} =
	 	\multirow{ 2}{*}{Dependence} &  	%\multirow{ 2}{*}{$H$} &
	  $ \; %\hat \rho =
	   \displaystyle\frac{ \sum_{n} \big(X_{n1}- \bar X_{1} \big) \big(X_{n2} - \bar X_{2}\big) } {\sqrt{ \sum_n \big(X_{n1}- \bar X_{1} \big) ^2 \sum_n \big(X_{n2} - \bar X_{2}\big)^2 }   } $ &
	\;\; $\displaystyle   \frac{ \sum_{n} \big(R_{n1} - \bar R_{1} \big) \big(R_{n2} - \bar R_{2}\big) } { \sqrt{ \sum_n \big(R_{n1} - \bar R_{1} \big) ^2 \sum_n \big(R_{n2} - \bar R_{2}\big)^2  }} $ \\
	 & (Pearson's correlation ) & (Spearman's rank correlation \cite{Spearman})\\[1mm] % \hat{\rho} =
	 \hline\\[-2mm]
	 \end{tabular}
	 }
	\end{center}
	\caption{ Given a group ${\bf G} =(X_{nv})  \in \mathbb{R}^{ N \times V}$,  the $\beta$-quantile  $\hat{q}_v(\beta)$ is estimated from  the empirical distribution of the $v$th column of random variables. Also
$R_{\cdot v}  \in \mathbb{R}^{ N }$  denotes ranked values  of $X_{\cdot v}$ with average column rank $\bar R_v$.  % Note Pearson's correlation captures a linear relationship between variables whereas Spearman's rank correlation \cite{Spearman}  measures a monotonic (possibly non-linear) dependence.
Non-parametric measures of skewness and kurtosis are respectively described in
Hinkley   \cite{hinkley1975}
and Moors \cite{RobustK}.
}
 \label{Tab:Des}
\end{table}

% SP - characterise, measure
In most applications, statistical properties that characterises group deviations are usually known. Without prior information, it is difficult to differentiate a group deviation  as there may be a significant difference in a combination of statistical properties. %Also if an incorrect selection of statistical properties are analysed then significant group deviations cannot be identified.
  Guevara et al. \cite{SMDD} explore the GAD application and find that more complicated group behaviours are not adequately characterised by single quantities such as location estimates of group distributions. %Their study investigates examples of Gaussian mixtures where a group anomaly is generated from a different proportion of distributions.
 % Another issue occurs for high dimensional datasets as common measures of statistical properties in Table \ref{Tab:Des} may not properly characterise the behaviour of groups.
   Topic models are applied for GAD problems  where  statistical properties of  groups represent  proportions of inferred topic  variables.  % high number of dimensions where  models extract .
  We  discuss topic models in more detail for  GAD applications in Chapter \ref{sec:staticGAD}.

\subsection{Model Design}
After understanding problems associated with group structures and statistical properties of interest, a domain expert can construct suitable solutions for GAD and GCD problems. Firstly models are designed to be compatible for specific types of input data such as continuous or categorical variables. To appropriately identify group deviations, discriminative methods do not impose data assumptions  while generative models assume how data is generated.  Given availability of labeled group behaviours, models either apply  supervised or unsupervised learning. Another important aspect of model design is  interpreting outputs with scores or labels.  The model design of group deviation detection techniques requires a clear understanding.

\subsubsection{Input Data }
Certain models are compatible for specific types of input data.   As specified by Equation (\ref{Eqn:Domain}) in the problem definition, data types that are explored in GAD and GCD applications include  discrete, continuous or categorical features.
Data types also influence the appropriateness of statistical properties for characterising  groups behaviours. In particular, generative models are flexible in assuming a Gaussian distribution for continuous real-valued data whereas categorical features are modelled by categorical distributions.
 Pairwise network connections are also a possible type of input data and are usually  incorporated for clustering when group structures are previously unknown.  Similarly, certain clustering algorithms are only compatible for specific data types.

\subsubsection{Assumptions}
 The assumptions of GAD and GCD techniques  are  discussed in terms of discriminative and generative models.  %where supervised or   unsupervised learning is applied depending on the availability of labeled data.
Discriminative approaches are useful for directly classifying groups into regular and anomalous behaviours without knowledge of how data is generated.  %Due to the lack of regular and anomalous group labels, discriminative GAD techniques classify regular behaviour based on the predominant group pattern.
 On the other hand, generative models assume specific probability density functions over   variables.  Hypothesis tests are a special type of generative model that further classify group deviations. %GAD and GCD  techniques  are classified as either  discriminative or generative models however hypothesis tests are also elaborated on.
% Common advantages and disadvantages of
Table \ref{Tab:DG} lists common advantages of  discriminative methods, generative models as well as hypothesis tests. % for  GAD and GCD  applications.

\begin{table}[H]
	\tabcolsep=0.3cm
	\renewcommand{\arraystretch}{1.2}
	\begin{center}\scalebox{1}{
	\begin{tabular}{lcccccccccccccccc  }
	\hline\\[-6mm]
 Procedure & $\mathcal{A}1$ & $\mathcal{A}2$ & $\mathcal{A}3$ & $\mathcal{A}4$ & $\mathcal{A}5$  \\[-1mm] \hline \\[-8mm] \hline\\[-6mm]
  Discriminative Methods   & \yeah & \nope & \yeah  & \nope & \nope \\
  Generative Models   & \nope & \yeah & \nope  & \yeah & \nope \\
 Hypothesis Tests  & \yeah & \yeah & \nope  & \yeah & \yeah\\[3mm]
\hline
	 \end{tabular}
	}
	\end{center}
	\medskip
		\caption{  Summary of advantages of   group deviation detection techniques in terms of 	 discriminative methods, generative models and hypothesis tests. If a procedure has a particular advantage, a tick label is present whereas if a procedure lacks an advantage, a cross is displayed.  }
	%\vspace{-5mm}
 \label{Tab:DG}
\end{table}

\begin{enumerate}[{$\mathcal{A}$}1] \setlength\itemsep{5pt}
\item   {\it Direct classification}:  discriminative models and hypothesis tests provide explicit boundaries between regular   behaviours and significant group deviations.
 \item  {\it Rich Interpretation}: Results from discriminative methods  are difficult to interpret due to the complex representation between group   variables. For explanatory purposes,
  generative models offer a rich interpretation of  groups and inferred statistical properties.    The flexible structure of generative models is also useful for incorporating prior information.
  \item  {\it  Minimal  Assumptions}:
  Discriminative approaches assume that group behaviours can be differentiated based on certain optimisation criteria.   Generative models further impose distributional assumptions which are not  appropriate for all datasets.
  \item  {\it Prevents Overfitting}: Discriminative methods are prone to overfitting model parameters especially on  training data with smaller sample sizes.
 Generally, generative models experience less overfitting  when the model assumptions are appropriate for given datasets.
   \item  {\it Statistical Significance}: In addition,  hypothesis tests determine whether the statistical properties of a group is significantly different. In many cases, generative models arbitrarily classify    significant group deviations based on highest anomalous scores.
\end{enumerate}

\subsubsection{Learning}
Supervised learning requires previously labelled group behaviours while
unsupervised approaches learn the dominant pattern in the dataset.
When surveying current state-of-the-art group deviation detection
 techniques, most discriminative  and generative models employ unsupervised learning.
 Unsupervised learning is preferred as  ground truth labels of regular or irregular behaviours are not usually available. In a  comparison study conducted by Laskov et al. \cite{laskov2005learning},  unsupervised methods achieve a higher accuracy than  supervised algorithms when learned behaviours from a training data do not account for unknown patterns in a test set.  Unsupervised methods are beneficial for  discovering novel group patterns.

\subsubsection{Output}
The output from group deviation detection techniques  is given by classification labels or  scores indicating significant group deviations.
 Discriminative methods produce binary labels for regular or anomalous classes in GAD while GCD methods estimate times of significant change in a dynamic group over time.   Generative models  tend to compute scores that quantifies the degree that a  group is significantly different as compared to other groups in a dataset. Scores from generative models are often converted to a classification by a threshold chosen by a user however  this selection is subjective and equivocal.  Hypothesis tests are advantageous for obtaining classification labels based on the statistical significance of group deviations.

\section{Challenges}
We now discuss challenges associated with group deviation detection. There are many issues that arise from inadequate group definitions in a dataset. Datasets involving group structures are more difficult to understand and analyse than many pointwise data problems with potential absence of group labels. Benchmark datasets are currently unavailable and thus comparison studies are difficult to conduct. Even though detecting group deviations may seem like a straightforward task, results require validation and careful interpretation. The challenges in  GAD and GCD applications  include:

\begin{enumerate}[\textbullet]
\item {\it  Defining Groups}:
 F\~{a}rber et al. \cite{ClusterEval}   highlight that known group labels may not represent natural clustering patterns in a dataset.
 However when group memberships are unknown a priori,   clustering method possibly lead to inadequate group representations. Group structures  may be improved by incorporating additional information such as known regular group behaviour or pairwise relationships between data instances.
 \item {\it Defining Group Deviations}: Group deviations can be defined based on the number of groups or the number of  data instances within groups. Usually when groups possess similar sizes, group deviations occur as a minority of group observations.
  % Consider  a single group that contains more observations than the total number of other groups.
\item {\it Capturing Statistical Properties}: Group deviations may occur in a variety of statistical properties such as location, scale, shape, etc.    Borgatti et al. \cite{GroupSocialMedia} explain that more complicated patterns exist when each member contributes to behaviour of a group. Thus an effective detection method has to adequately capture properties of groups in order to identify significant deviations.
% \item {\it Evolving Patterns}: In many domains, the notion and definition of group deviations also changes over a period of time. Algorithms that can adapt to identifying evolving unknown group patterns  are preferable in many applications.
\item {\it Not statistically significant}: Many methods classify or score group behaviours however they do not quantify statistical significance. In many cases, it is difficult to distinguish noisy group observations from a significant group deviation.
\item {\it Absence of Group Labels}: When group memberships are unknown a priori, clustering  induces additional uncertainty in analysis and subsequent results. Halkidi et al. \cite{ClusterValidity} explain that it is also difficult to evaluate the effectiveness of inferred clusters without sufficient ground truth labels.
\item {\it Absence of Ground Truth Labels}: Like other anomaly detection applications, ground truth labels are usually unavailable. To investigate group deviations rather than a single  instance, requires more time and effort for obtaining  ground truth labels.
\item {\it Absence of Benchmark Datasets}: Since there is a lack of benchmark group datasets, many methods resort to anomaly injection. Anomaly injection involves contaminating a real-world dataset with significant deviations and subsequently comparing detected instances with  ground truth labels. This does not account for  anomalies that are naturally present in a dataset such that injected anomalies should possess higher deviations than naturally occurring anomalies.
\item {\it Absence of Robust Comparison Studies}: Evaluative metrics that assess  group deviation detection datasets would be useful for a robust comparison study. For pointwise anomaly detection methods, Campos et al. \cite{Campos2016} propose two measures for a dataset; difficulty of detecting different types of anomalies and diversity or agreement between scores computed from methods.  A similar evaluative process is recommended for group deviation datasets to provide a more robust comparison of state-of-the-art techniques.
\end{enumerate}
There is no single solution that overcomes all of these challenges in GAD and GCD research.  We further elaborate on related work for group deviation detection techniques in static and dynamic scenarios.

\section{Related Work}
Due to continual research involving  anomaly detection and change detection,  there are additional problems and  techniques that are proposed after the publication of many papers. Extensive reviews on pointwise anomaly detection techniques are conducted by Hodge and  Austin \cite{Hodge}  as well as Chandola et al. \cite{Chandola}.   Since change detection is a general topic, many techniques are domain specific where
Singh \cite{singh1989review} examines the application to remote sensor data while Reeves et al. \cite{reeves2007review} explore change detection techniques for climate data. An overview of temporal outlier detection is provided by Gupta et al. \cite{Gupta2013}. Many of these papers  briefly discuss  group applications however GAD and GCD are emerging areas of research where most state-of-the-art techniques have been more recently developed.  Yu et al. \cite{SurveySocialMedia} and  Xiong \cite{Collective}  provide descriptions of current state-of-the-art GAD methods.  %

{
GAD is closely related to zero-shot learning (ZSL) where data instances are classified however their behaviour may not be seen in training. % different classes even when unseen instances in a test set   that are not present in the training set.
%There is an overlap between GAD and zero-shot learning (ZSL) techniques however there are fundamental difference.
Surveys on ZSL have been conducted by Xian et al. \cite{ZSLsurvey} where ZSL assumes a subset of classes (groups) are known during training whereas GAD techniques are more general as none of regular  group behaviours (classes) may be available.
%GAD is more general than ZSL as classes (groups) may not be available while ZSL assume classes are known during training.
 %ZSL accounts for the simple case where training and test set are disjoint while classes in  training and test set may overlap for generalised ZSL.
  Without known classes,  Kodirov et al. \cite{unsupervisedZSL} propose an unsupervised ZSL technique that incorporates auxiliary textual information.   ZSL techniques are specfically formulated for certain domains such as image classification \cite{ZSLanomaly} and network intrusion  \cite{perez2016} however we focus on GAD techniques that are applicable for more general domains.

%Group change detection
Similarly the GCD problem has been  specifically formulated for many real-world applications. In video change detection, a video can be modelled as a group of color pixels or visual features where  significant deviations in video frames are detected over time. Lienhart
\cite{VideoSurvey} provides a survey of video transition (gradual change) detection techniques while a comparison study of performance for video-shot-change (abrupt  change) detection algorithms is conducted in Gargi et al. \cite{gargi2000}. %Other methods such as  Yuan et al. \cite{yuan2017anomaly} identify subtle changes in  traffic scenes while  %Yuan et  al.   \cite{yuan2016hyperspectral} examine  reflectance spectrum data.
%Rout et al. \cite{rout2018}  detect changing  positions of dynamic objects in an underwater video.
  In other applications, Sakaki et al. \cite{sakaki2010} detect an earthquake event by monitoring a group of keywords on Twitter and  Xie et al. \cite{xie2013} explore  sequential change-point detection in multiple sensor readings over time. % the average value of  sensor readings over time. % and also estimate the affected proportion  of sensors at a specific time step.
Many of these GCD techniques are only applicable in  specific domains and are not flexible for detecting changes in a variety of statistical properties.
}