\subsection{PostProcessor}
\label{sec:models_postProcessor}
A Post-Processor (PP) can be considered as an action performed on a set of data
or other type of objects.
%
Most of the post-processors contained in RAVEN, employ a mathematical operation
on the data given as ``input''.
%
RAVEN supports several different types of PPs.

Currently, the following types are available in RAVEN:
\begin{itemize}
  \itemsep0em
  \item \textbf{BasicStatistics}
  \item \textbf{ComparisonStatistics}
  \item \textbf{ImportanceRank}
  \item \textbf{SafestPoint}
  \item \textbf{LimitSurface}
  \item \textbf{LimitSurfaceIntegral}
  \item \textbf{External}
  \item \textbf{TopologicalDecomposition}
  \item \textbf{DataMining}
  \item \textbf{HistorySetDelay}
  \item \textbf{HS2PS}
  \item \textbf{HStoPSOperator}
  \item \textbf{HistorySetSampling}
  \item \textbf{HistorySetSnapShot}
  \item \textbf{HistorySetSync}
  \item \textbf{TypicalHistoryFromHistorySet}
  \item \textbf{dataObjectLabelFilter}
  \item \textbf{Metric}
  \item \textbf{CrossValidation}
  \item \textbf{ValueDuration}
  \item \textbf{FastFourierTransform}
  \item \textbf{SampleSelector}
  \item \textbf{ParetoFrontier}
  \item \textbf{EconomicRatio}
  \item \textbf{Validation}
  \item \textbf{TSACharacterizer}
  %\item \textbf{PrintCSV}
  %\item \textbf{LoadCsvIntoInternalObject}
\end{itemize}

The specifications of these types must be defined within the XML block
\xmlNode{PostProcessor}.
%
This XML node needs to contain the attributes:
\vspace{-5mm}
\begin{itemize}
  \itemsep0em
  \item \xmlAttr{name}, \xmlDesc{required string attribute}, user-defined
  identifier of this post-processor.
  %
  \nb As with other objects, this is the name that can be used to refer to this
  specific entity from other input XML blocks.
  \item \xmlAttr{subType}, \xmlDesc{required string attribute}, defines which of
  the post-processors needs to be used, choosing among the previously reported
  types.
  %
  This choice conditions the subsequent required and/or optional
  \xmlNode{PostProcessor} sub nodes.
  %
\end{itemize}
\vspace{-5mm}

As already mentioned, all the types and meaning of the remaining sub-nodes
depend on the post-processor type specified in the attribute \xmlAttr{subType}.
%
In the following sections the specifications of each type are reported.

%%%%% PP BasicStatistics %%%%%%%
\subsubsection{BasicStatistics}
\label{BasicStatistics}
The \textbf{BasicStatistics} post-processor is the container of the algorithms
to compute many of the most important statistical quantities. It is important to notice that this
post-processor can accept as input both \textit{\textbf{PointSet}} and \textit{\textbf{HistorySet}}
data objects, depending on the type of statistics the user wants to compute:
\begin{itemize}
  \item \textit{\textbf{PointSet}}: Static Statistics;
  \item \textit{\textbf{HistorySet}}: Dynamic Statistics. Depending on a ``pivot parameter'' (e.g. time)
  the post-processor is going to compute the statistics for each value of it (e.g. for each time step).
  In case an \textbf{HistorySet} is provided as Input, the Histories needs to be synchronized (use
    \textit{\textbf{Interfaced}} post-processor of type  \textbf{HistorySetSync}).
\end{itemize}
%
\ppType{BasicStatistics post-processor}{BasicStatistics}
\begin{itemize}
  \item \xmlNode{"metric"}, \xmlDesc{comma separated string or node list, required field},
    specifications for the metric to be calculated.  The name of each node is the requested metric.  There are
    two forms for specifying the requested parameters of the metric.  For scalar values such as
    \xmlNode{expectedValue} and \xmlNode{variance}, the text of the node is a comma-separated list of the
    parameters for which the metric should be calculated.  For matrix values such as \xmlNode{sensitivty} and
    \xmlNode{covariance}, the matrix node requires two sub-nodes, \xmlNode{targets} and \xmlNode{features},
    each of which is a comma-separated list of the targets for which the metric should be calculated, and the
    features for which the metric should be calculated for that target.  See the example below.

    \nb When defining the metrics to use, it is possible to have multiple nodes with the same name.  For
    example, if a problem has inputs $W$, $X$, $Y$, and $Z$, and the responses are $A$, $B$, and $C$, it is possible that
    the desired metrics are the \xmlNode{sensitivity} of $A$ and $B$ to $X$ and $Y$, as well as the
    \xmlNode{sensitivity} of $C$ to $W$ and $Z$, but not the sensitivity of $A$ to $W$.   In this event, two
    copies of the \xmlNode{sensitivity} node are added to the input.  The first has targets $A,B$ and features
    $X,Y$, while the second node has target $C$ and features $W,Z$.  This could reduce some computation effort
    in problems with many responses or inputs.  An example of this is shown below.
  %
  \\ Currently the scalar quantities available for request are:
  \begin{itemize}
    \item \textbf{expectedValue}: expected value or mean
    \item \textbf{minimum}: The minimum value of the samples.
    \item \textbf{maximum}: The maximum value of the samples.
    \item \textbf{median}:  The weighted median of the samples ( $50\%$ weighted percentile). If probability weights are not assigned, uniform distribution will be assigned. The median $x_k$ satisfying:
    \begin{equation}
      \sum_{i = 1}^{k - 1} w_i \le 1/2  and \sum_{i = k + 1}^{n} w_i \le 1/2
    \end{equation}
    \item \textbf{variance}: variance
    \item \textbf{sigma}: standard deviation
    \item \textbf{percentile}: the percentile. If this quantity is inputted as \textit{percentile} the $5\%$ and $95\%$ percentile(s) are going to be computed.
                               Otherwise the user can specify this quantity with a parameter \textit{percent='X'}, where the \textit{X} represents the requested
                               percentile (a floating point value between 0.0 and 100.0)
    \item \textbf{variationCoefficient}: coefficient of variation, i.e. \textbf{sigma}/\textbf{expectedValue}. \nb If the \textbf{expectedValue} is zero,
    the \textbf{variationCoefficient} will be \textbf{INF}.
    \item \textbf{skewness}: skewness
    \item \textbf{kurtosis}: excess kurtosis (also known as Fisher's kurtosis)
    \item \textbf{samples}: the number of samples in the data set used to determine the statistics.
  \end{itemize}
  The matrix quantities available for request are:
  \begin{itemize}
    \item \textbf{sensitivity}: matrix of sensitivity coefficients, computed via linear regression method. (\nb The condition number is computed every time this quantity is requsted. If it results
    to be greater then $30$, a multicollinearity problem exists and the sensitivity coefficients
    might be incorrect and a Warning is spooned by the code)
    \item \textbf{covariance}: covariance matrix
    \item \textbf{pearson}: matrix of correlation coefficients
    \item \textbf{spearman}: matrix of spearman ranking coefficients. This matrix is computed in its
    weighted form (see RAVEN theory manual at \cite{RAVENtheoryManual}):
    \begin{equation}
\boldsymbol{\mathrm{P}}(\boldsymbol{X},\boldsymbol{Y}) = \frac{\boldsymbol{\Sigma}(\boldsymbol{R(X)},\boldsymbol{R(Y)})}{\sigma_{R(x)} \sigma_{R(y)}}
\end{equation}
    \item \textbf{NormalizedSensitivity}: matrix of normalized sensitivity
    coefficients. \nb{It is the matrix of normalized VarianceDependentSensitivity}
    \item \textbf{VarianceDependentSensitivity}: matrix of sensitivity coefficients dependent on the variance of the variables
  \end{itemize}
  This XML node needs to contain the attribute:
  \begin{itemize}
    \itemsep0em
    \item \xmlAttr{prefix}, \xmlDesc{required string attribute}, user-defined prefix for the given \textbf{metric}.
      For scalar quantifies, RAVEN will define a variable with name defined as:  ``prefix'' + ``\_'' + ``parameter name''.
      For example, if we define ``mean'' as the prefix for \textbf{expectedValue}, and parameter ``x'', then variable
      ``mean\_x'' will be defined by RAVEN. For \textbf{percentile}, RAVEN will define a variable with name defined as:
      ``prefix'' + ``\_'' + ``percent'' + ``\_'' + ``parameter name''. For example, if we define ``perc'' as the prefix
      for \textbf{percentile}, percent as ``10'', and parameter ``x'', then variable ``perc\_10\_x'' will
      be defined by RAVEN.
      For matrix quantities, RAVEN will define a variable with name defined as: ``prefix'' + ``\_'' + ``target parameter name'' + ``\_'' + ``feature parameter name''.
      For example, if we define ``sen'' as the prefix for \textbf{sensitivity}, target ``y'' and feature ``x'', then
      variable ``sen\_y\_x'' will be defined by RAVEN.
      \nb These variable will be used by RAVEN for the internal calculations. It is also accessible by the user through
      \textbf{DataObjects} and \textbf{OutStreams}.
  \end{itemize}
   %
  \nb If the weights are present in the system then weighted quantities are calculated automatically. In addition, if a matrix quantity is requested (e.g. Covariance matrix, etc.), only the weights in the output space are going to be used for both input and output space (the computation of the joint probability between input and output spaces is not implemented yet).
  \\
  \nb Certain ROMs provide their own statistical information (e.g., those using
  the sparse grid collocation sampler such as: \xmlString{GaussPolynomialRom}
  and \xmlString{HDMRRom}) which can be obtained by printing the ROM to file
  (xml). For these ROMs, computing the basic statistics on data generated from
  one of these sampler/ROM combinations may not provide the information that the
  user expects.
  \\
  In addition, RAVEN will automatically calculate the standard errors on the following scalar quantities:
  \begin{itemize}
    \item \textbf{expectedValue}
    \item \textbf{median}
    \item \textbf{variance}
    \item \textbf{sigma}
    \item \textbf{skewness}
    \item \textbf{kurtosis}
    \item \textbf{percentile}
  \end{itemize}
  RAVEN will define a variable with name defined as: ``prefix for given \textbf{metric}'' + ``\_ste\_'' + ``parameter name'' to
  store standard error of given \textbf{metric} with respect to given parameter. This information will be stored in the DataObjects,
  i.e. \textbf{PointSet} and \textbf{HistorySet}, and by default will be printed out in the ``CSV'' output files by the
  \textbf{OutStreams}. Option node \xmlNode{what} can be used in the \textbf{OutStreams} to select the information that
  the users want to print.
  In the case when the users want to store all the calculations results in general \textbf{DataSets}, RAVEN will employ a variable
  with name defined as: ``\textbf{metric}'' + ``\_ste'' to store standard error with respect to all target parameters. An additional
  index ``target'' will added in the \textbf{DataSets} with respect to these variables. All these quantities will be automatically
  computed and stored in the given \textbf{DataSet}, and the users do not need to specify these quantities in their RAVEN input files.
  %
   \item \xmlNode{pivotParameter}, \xmlDesc{string, optional field}, name of the parameter that needs
   to be used for the computation of the Dynamic BasicStatistics (e.g. time). This node needs to
   be inputted just in case an \textbf{HistorySet} is used as Input. It represents the reference
   monotonic variable based on which the statistics is going to be computed (e.g. time-dependent
   statistical moments).
    \default{None}
  %
  \item \xmlNode{biased}, \xmlDesc{string (boolean), optional field}, if \textit{True} biased
  quantities are going to be calculated, if \textit{False} unbiased.
  \default{False}
  %
  \item \xmlNode{dataset}, \xmlDesc{boolean, optional field}, if \textit{True} \xmlString{DataSet}
    will be used to store the calculation results, if \textit{False} \xmlString{PointSet} or \xmlString{HistorySet}
    will be used to store the calculation results.
    \nb The optional \xmlString{DataSet} is added only to this PostProcessor, one can still use the \xmlString{OutStreams}
    to print the variables available in the \textit{DataSet}. The \xmlString{"metric"} names are used as the
    variable names, i.e. variable names listed in \xmlNode{Input} or \xmlNode{Output} in the defined \xmlString{DataSet}.
    In addition, the extra node \xmlNode{Index} is required, and the value for \xmlAttr{var} can be found in the following:
    \begin{itemize}
      \item scalar metrics, such as \xmlNode{expectedValue} and \xmlNode{variance},
        are requested, the index variable \xmlString{targets} will be required.
      \item vector metrics, such as \xmlNode{covariance} and \xmlNode{sensitivity}, are requested, the index variables
        \xmlString{targets} and \xmlString{features} will be required.
      \item If \xmlNode{percentile} is requested, an additional index variable \xmlString{percent} should be added.
      \item when dynamic BasicStatistics (e.g. time) is requested, the index variable \xmlString{time}  will be required.
    \end{itemize}
  \default{False}
  %
\item \xmlNode{multipleFeatures}, \xmlDesc(boolean, optional field), if \textbf{False}, this node can be used when
    the users want to compute sensitivities based on one target variable with respect to one feature variable,
    i.e. the sensitivity calculations are directly computed using the \textbf{Linear Regression} or
    \textbf{Best Linear Predictor} method with single feature. This method can be useful when the input features
    depend on each other. The default value is \textbf{True}, which means the sensitivity calculations are performed
    using \textbf{Linear Regression} or \textbf{Best Linear Predictor} method with multiple features. If the input
    features are not fully correlated, the default value for \xmlNode{multipleFeatures} is always recommended.
    \nb this node only affects the calculations of metrics such as \xmlNode{sensitivity},
    \xmlNode{VarianceDependentSensitivity} and \xmlNode{NormalizedSensitivity}.
  \default{True}
\end{itemize}
\textbf{Example (Static Statistics):}  This example demonstrates how to request the expected value of
\xmlString{x01} and \xmlString{x02}, along with the sensitivity of both \xmlString{x01} and \xmlString{x02} to
\xmlString{a} and \xmlString{b}.
\begin{lstlisting}[style=XML,morekeywords={name,subType,debug}]
<Simulation>
  ...
  <Models>
    ...
    <PostProcessor name='aUserDefinedName' subType='BasicStatistics' verbosity='debug'>
      <expectedValue prefix='mean'>x01,x02</expectedValue>
      <sensitivity prefix='sen'>
        <targets>x01,x02</targets>
        <features>a,b</features>
      </sensitivity>
    </PostProcessor>
    ...
  </Models>
  ...
</Simulation>
\end{lstlisting}

In this case, the RAVEN variables ``mean\_x01, mean\_x02, sen\_x01\_a, sen\_x02\_a, sen\_x01\_b, sen\_x02\_b''
will be created and accessible for the RAVEN entities \textbf{DataObjects} and \textbf{OutStreams}.

\textbf{Example (Static, multiple matrix nodes):} This example shows how multiple nodes can specify
particular metrics multiple times to include different target/feature combinations.  This PostProcessor
calculates the expected value of $A$, $B$, and $C$, as well as the sensitivity of both $A$ and $B$ to $X$ and
$Y$ as well as the sensitivity of $C$ to $W$ and $Z$.
\begin{lstlisting}[style=XML,morekeywords={name,subType,debug}]
<Simulation>
  ...
  <Models>
    ...
    <PostProcessor name='aUserDefinedName' subType='BasicStatistics' verbosity='debug'>
      <expectedValue prefix='mean'>A,B,C</expectedValue>
      <sensitivity prefix='sen1'>
        <targets>A,B</targets>
        <features>x,y</features>
      </sensitivity>
      <sensitivity prefix='sen2'>
        <targets>C</targets>
        <features>w,z</features>
      </sensitivity>
    </PostProcessor>
    ...
  </Models>
  ...
</Simulation>
\end{lstlisting}
\textbf{Example (Dynamic Statistics):}
\begin{lstlisting}[style=XML,morekeywords={name,subType,debug}]
<Simulation>
  ...
  <Models>
    ...
    <PostProcessor name='aUserDefinedNameForDynamicPP' subType='BasicStatistics' verbosity='debug'>
      <expectedValue prefix='mean'>x01,x02</expectedValue>
      <sensitivity prefix='sen'>
        <targets>x01,x02</targets>
        <features>a,b</features>
      </sensitivity>
      <pivotParameter>time</pivotParameter>
    </PostProcessor>
    ...
  </Models>
  ...
  <HistorySet name='basicStatHistorySet'>
    <Output>
      mean_x01,mean_x02,
      sen_x01_a, sen_x01_b,
      sen_x02_a, sen_x02_b
    </Output>
    <options>
      <pivotParameter>time</pivotParameter>
    </options>
  </HistorySet>
</Simulation>
\end{lstlisting}

\textbf{Example (Dumping the results into DataSet):}
\begin{lstlisting}[style=XML,morekeywords={name,subType,debug}]
<Simulation>
  ...
  <Models>
    ...
    <PostProcessor name='aUserDefinedNameForDynamicPP' subType='BasicStatistics' verbosity='debug'>
      <dataset>True</dataset>
      <expectedValue prefix='mean'>x01,x02</expectedValue>
      <sensitivity prefix='sen'>
        <targets>x01,x02</targets>
        <features>a,b</features>
      </sensitivity>
      <pivotParameter>time</pivotParameter>
    </PostProcessor>
    ...
  </Models>
  ...
  <DataObjects>
    <DataSet name='basicStatDataSet'>
      <Output>expectedValue,sensitivity</Output>
      <Index var='time'>expectedValue,sensitivity</Index>
      <Index var='targets'>expectedValue,sensitivity</Index>
      <Index var='features'>sensitivity</Index>
    </DataSet>
  </DataObjects>
</Simulation>
\end{lstlisting}
%%%%% PP ComparisonStatistics %%%%%%%
\subsubsection{ComparisonStatistics}
\label{ComparisonStatistics}
The \textbf{ComparisonStatistics} post-processor computes statistics
for comparing two different dataObjects.  This is an experimental
post-processor, and it will definitely change as it is further
developed.

There are four nodes that are used in the post-processor.

\begin{itemize}
\item \xmlNode{kind}: specifies information to use for comparing the
  data that is provided.  This takes either uniformBins which makes
  the bin width uniform or equalProbability which makes the number
  of counts in each bin equal.  It can take the following attributes:
  \begin{itemize}
  \item \xmlAttr{numBins} which takes a number that directly
    specifies the number of bins
  \item \xmlAttr{binMethod} which takes a string that specifies the
    method used to calculate the number of bins.  This can be either
    square-root or sturges.
  \end{itemize}
\item \xmlNode{compare}: specifies the data to use for comparison.
  This can either be a normal distribution or a dataObjects:
  \begin{itemize}
  \item \xmlNode{data}: This will specify the data that is used.  The
    different parts are separated by $|$'s.
  \item \xmlNode{reference}: This specifies a reference distribution
    to be used.  It takes distribution to use that is defined in the
    distributions block.  A name parameter is used to tell which
    distribution is used.
  \end{itemize}
\item \xmlNode{fz}: If the text is true, then extra comparison
  statistics for using the $f_z$ function are generated.  These take
  extra time, so are not on by default.
\item \xmlNode{interpolation}: This switches the interpolation used
  for the cdf and the pdf functions between the default of quadratic
  or linear.
\end{itemize}

The \textbf{ComparisonStatistics} post-processor generates a variety
of data.  First for each data provided, it calculates bin boundaries,
and counts the numbers of data points in each bin.  From the numbers
in each bin, it creates a cdf function numerically, and from the cdf
takes the derivative to generate a pdf.  It also calculates statistics
of the data such as mean and standard deviation. The post-processor
can generate a CSV file only.

The post-processor uses the generated pdf and cdf function to
calculate various statistics.  The first is the cdf area difference which is:
\begin{equation}
  cdf\_area\_difference = \int_{-\infty}^{\infty}{\|CDF_a(x)-CDF_b(x)\|dx}
\end{equation}
This given an idea about how far apart the two pieces of data are, and
it will have units of $x$.

The common area between the two pdfs is calculated.  If there is
perfect overlap, this will be 1.0, if there is no overlap, this will
be 0.0.  The formula used is:
\begin{equation}
  pdf\_common\_area = \int_{-\infty}^{\infty}{\min(PDF_a(x),PDF_b(x))}dx
\end{equation}

The difference pdf between the two pdfs is calculated.  This is calculated as:
\begin{equation}
  f_Z(z) = \int_{-\infty}^{\infty}f_X(x)f_Y(x-z)dx
\end{equation}
This produces a pdf that contains information about the difference
between the two pdfs.  The mean can be calculated as (and will be
calculated only if fz is true):
\begin{equation}
  \bar{z} = \int_{-\infty}^{\infty}{z f_Z(z)dz}
\end{equation}
The mean can be used to get an signed difference between the pdfs,
which shows how their means compare.

The variance of the difference pdf can be calculated as (and will be
calculated only if fz is true):
\begin{equation}
  var = \int_{-\infty}^{\infty}{(z-\bar{z})^2 f_Z(z)dz}
\end{equation}

The sum of the difference function is calculated if fz is true, and is:
\begin{equation}
  sum = \int_{-\infty}^{\infty}{f_z(z)dz}
\end{equation}
This should be 1.0, and if it is different that
points to approximations in the calculation.

\textbf{Example:}
\begin{lstlisting}[style=XML]
<Simulation>
   ...
   <Models>
      ...
      <PostProcessor name="stat_stuff" subType="ComparisonStatistics">
      <kind binMethod='sturges'>uniformBins</kind>
      <compare>
        <data>OriData|Output|tsin_TEMPERATURE</data>
        <reference name='normal_410_2' />
      </compare>
      <compare>
        <data>OriData|Output|tsin_TEMPERATURE</data>
        <data>OriData|Output|tsout_TEMPERATURE</data>
      </compare>
      </PostProcessor>
      <PostProcessor name="stat_stuff2" subType="ComparisonStatistics">
        <kind numBins="6">equalProbability</kind>
        <compare>
          <data>OriData|Output|tsin_TEMPERATURE</data>
        </compare>
        <Distribution class='Distributions' type='Normal'>normal_410_2</Distribution>
      </PostProcessor>
      ...
   </Models>
   ...
   <Distributions>
      <Normal name='normal_410_2'>
         <mean>410.0</mean>
         <sigma>2.0</sigma>
      </Normal>
   </Distributions>
</Simulation>
\end{lstlisting}

%%%%% PP ImportanceRank %%%%%%%
\subsubsection{ImportanceRank}
\label{ImportanceRank}
The \textbf{ImportanceRank} PostProcessor is specifically used
to compute sensitivity indices and importance indices with respect to input parameters
associated with multivariate normal distributions. In addition, the user can also request the transformation
matrix and the inverse transformation matrix when the PCA reduction is used.
%
\ppType{ImportanceRank}{ImportanceRank}
%
\begin{itemize}
  \item \xmlNode{what}, \xmlDesc{comma separated string, required field},
  %
  List of quantities to be computed.
  %
  Currently the quantities available are:
  \begin{itemize}
    \item \xmlString{SensitivityIndex}: used to measure the impact of sensitivities on the model.
    \item \xmlString{ImportanceIndex}: used to measure the impact of sensitivities and input uncertainties on the model.
    \item \xmlString{PCAIndex}: the indices of principal component directions, used to measure the impact
    of principal component directions on input covariance matrix.
    \nb \xmlString{PCAIndex} can be only requested when subnode \xmlNode{latent} is defined in \xmlNode{features}.
    \item \xmlString{transformation}: the transformation matrix used to map the latent variables to the manifest variables in the original input space.
    \item \xmlString{InverseTransformation}: the inverse transformation matrix used to map the manifest variables to the latent variables in the transformed space.
    \item \xmlString{ManifestSensitivity}: the sensitivity coefficients of \xmlNode{target} with respect to \xmlNode{manifest} variables defined in \xmlNode{features}.

    \nb In order to request \xmlString{transformation} matrix or \xmlString{InverseTransformation} matrix or \xmlString{ManifestSensitivity},
    the subnodes \xmlNode{latent} and \xmlNode{manifest} under \xmlNode{features} are required (more details can be found in the following).
    %
  \end{itemize}
  %
  \nb For each computed quantity, RAVEN will define a unique variable name so that the data can be accessible by the users
  through RAVEN entities \textbf{DataObjects} and \textbf{OutStreams}. These variable names are defined as follows:
  \begin{itemize}
    \item \xmlString{SensitivityIndex}: `sensitivityIndex' + `\_' + `targetVariableName' + `\_' + `latentFeatureVariableName'
    \item \xmlString{ImportanceIndex}: `importanceIndex' + `\_' + `targetVariableName' + `\_' + `latentFeatureVariableName'
    \item \xmlString{PCAIndex}: `pcaIndex' + `\_' + `latentFeatureVariableName'
    \item \xmlString{transformation}: `transformation' + `\_' + `manifestFeatureVariableName' + `\_' + `latentFeatureVariableName'
    \item \xmlString{InverseTransformation}: `inverseTransformation' + `\_' + `latentFeatureVariableName' + `\_' + `manifestFeatureVariableName'
    \item \xmlString{ManifestSensitivity}: `manifestSensitivity' + `\_' + `targetVariableName' + `\_' + `manifestFeatureVariableName'
  \end{itemize}
  %
  If all the quantities need to be computed, the user can input in the body of \xmlNode{what} the string \xmlString{all}.
  \nb \xmlString{all} equivalent to \xmlString {SensitivityIndex, ImportanceIndex, PCAIndex}.

  Since the transformation and InverseTransformation matrix can be very large, they are not printed with option \xmlString{all}.
  In order to request the transformation matrix (or inverse transformation matrix) from this post processor,
  the user need to specify \xmlString{transformation} or \xmlString{InverseTransformation} in \xmlNode{what}. In addition,
  both  \xmlNode{manifest} and \xmlNode{latent} subnodes are required and should be defined in node \xmlNode{features}. For example, let $\mathbf{L, P}$ represent
  the transformation and inverse transformation matrices, respectively. We will define vectors $\mathbf x$ as manifest variables and vectors $\mathbf y$
  as latent variables. If a absolute covariance matrix is used in given distribution, the following equation will be used:

  $
  \mathbf{\delta x} = \mathbf L * \mathbf y
  $

  $
  \mathbf y = \mathbf P * \mathbf \delta \mathbf x
  $

  If a relative covariance matrix is used in given distribution, the following equation will be used:

  $
  \frac{\mathbf \delta \mathbf x}{\mathbf \mu} = \mathbf L * \mathbf y
  $

  $
  \mathbf y = \mathbf P * {\frac{\mathbf \delta \mathbf x}{\mathbf \mu}}
  $

  where $\mathbf{\delta x}$ denotes the changes in the input vector $\mathbf x$, and $\mathbf \mu$ denotes the mean values of the input vector $\mathbf x$.

  %
  %
  \item \xmlNode{features}, \xmlDesc{XML node, required parameter}, used to specify the information for the input variables.
  In this xml-node, the following xml sub-nodes need to be specified:
    \begin{itemize}
      \item \xmlNode{manifest},\xmlDesc{XML node, optional parameter}, used to indicate the input variables belongs to the original input space.
      It can accept the following child node:
        \begin{itemize}
          \item \xmlNode{variables},\xmlDesc{comma separated string, required field}, lists manifest variables.
          \item \xmlNode{dimensions}, \xmlDesc{comma separated integer, optional field}, lists the dimensions corresponding to the manifest variables.
          If not provided, the dimensions are determined by the order indices of given manifest variables.
        \end{itemize}
      \item \xmlNode{latent},\xmlDesc{XML node, optional parameter}, used to indicate the input variables belongs to the transformed space.
      It can accept the following child node:
        \begin{itemize}
          \item \xmlNode{variables},\xmlDesc{comma separated string, required field}, lists latent variables.
          \item \xmlNode{dimensions}, \xmlDesc{comma separated integer, optional field}, lists the dimensions corresponding to the latent variables.
          If not provided, the dimensions are determined by the order indices of given latent variables.
        \end{itemize}
      \nb At least one of the subnodes, i.e. \xmlNode{manifest} and \xmlNode{latent} needs to be specified.
    \end{itemize}
  %
  \item \xmlNode{targets}, \xmlDesc{comma separated string, required field}, lists output responses.
  %
  \item \xmlNode{mvnDistribution}, \xmlDesc{string, required field}, specifies the
  multivariate normal distribution name. The \xmlNode{MultivariateNormal} node must be present. It requires two attributes:
    \begin{itemize}
      \item \xmlAttr{class}, \xmlDesc{required string attribute}, is the main
        ``class'' the listed object is from, the only acceptable class for
        this post-processor is \xmlString{Distributions};
      \item \xmlAttr{type}, \xmlDesc{required string attribute}, is the type of distributions,
        the only acceptable type is \xmlString{MultivariateNormal}
    \end{itemize}
\end{itemize}
  %
  %
  Here is an example to show the user how to request the transformation matrix, the inverse transformation matrix, the
  manifest sensitivities and other quantities.
  %

\textbf{Example:}
\begin{lstlisting}[style=XML,morekeywords={name,subType,debug}]
<Simulation>
  ...
  <Models>
    ...
    <PostProcessor name='aUserDefinedName' subType='ImportanceRank'>
      <what>SensitivityIndex,ImportanceIndex,Transformation, InverseTransformation,ManifestSensitivity</what>
      <features>
        <manifest>
          <variables>x1,x2</variables>
          <dimensions>1,2</dimensions>
        </manifest>
        <latent>
          <variables>latent1</variables>
          <dimensions>1</dimensions>
        </latent>
      </features>
      <targets>y</targets>
      <mvnDistribution>MVN</mvnDistribution>
    </PostProcessor>
    ...
  </Models>
  ...
</Simulation>
\end{lstlisting}

The calculation results can be accessible via variables ``sensitivityIndex\_y\_latent1, importanceIndex\_y\_latent1,
manifestSensitivity\_y\_x1, manifestSensitivity\_y\_x2, transformation\_x1\_latent1, transformation\_x2\_latent1,
inverseTransformation\_latnet1\_x1, inverseTransformation\_laent1\_x2'' through RAVEN entities \textbf{DataObjects}
and \textbf{OutStreams}.

%%%%% PP SafestPoint %%%%%%%
\subsubsection{SafestPoint}
\label{SafestPoint}
The \textbf{SafestPoint} PostProcessor provides the coordinates of the farthest
point from the limit surface that is given as an input.
%
The safest point coordinates are expected values of the coordinates of the
farthest points from the limit surface in the space of the ``controllable''
variables based on the probability distributions of the ``non-controllable''
variables.

The term ``controllable'' identifies those variables that are under control
during the system operation, while the ``non-controllable'' variables are
stochastic parameters affecting the system behavior randomly.

The ``SafestPoint'' post-processor requires the set of points belonging to the
limit surface, which must be given as an input.
%
The probability distributions as ``Assembler Objects'' are required in the
``Distribution'' section for both ``controllable'' and ``non-controllable''
variables.

The sampling method used by the ``SafestPoint'' is a ``value'' or ``CDF'' grid.
%
At present only the ``equal'' grid type is available.

\ppType{Safest Point}{SafestPoint}

\begin{itemize}
  \item \xmlNode{Distribution}, \xmlDesc{Required}, represents the probability
  distributions of the ``controllable'' and ``non-controllable'' variables.
  %
  These are \textbf{Assembler Objects}, each of these nodes must contain 2
  attributes that are used to identify those within the simulation framework:
        \begin{itemize}
    \item \xmlAttr{class}, \xmlDesc{required string attribute}, is the main
    ``class'' the listed object is from.
                \item \xmlAttr{type}, \xmlDesc{required string attribute}, is the object
    identifier or sub-type.
        \end{itemize}
             \item  \xmlNode{outputName}, \xmlDesc{string, required field}, specifies the name of the output variable where the probability is going to be stored.
               \nb This variable name must be listed in the \xmlNode{Output} field of the Output DataObject
        \item \xmlNode{controllable}, \xmlDesc{XML node, required field},  lists the controllable variables.
  %
  Each variable is associated with its name and the two items below:
        \begin{itemize}
                \item \xmlNode{distribution} names the probability distribution associated
    with the controllable variable.
    %
                \item \xmlNode{grid} specifies the \xmlAttr{type}, \xmlAttr{steps}, and
    tolerance of the sampling grid.
    %
        \end{itemize}
        \item \xmlNode{non-controllable}, \xmlDesc{XML node, required field}, lists the non-controllable variables.
  %
  Each variable is associated with its name and the two items below:
        \begin{itemize}
                \item \xmlNode{distribution} names the probability distribution associated
    with the non-controllable variable.
    %
                \item \xmlNode{grid} specifies the \xmlAttr{type}, \xmlAttr{steps}, and
    tolerance of the sampling grid.
    %
                \end{itemize}
\end{itemize}

\textbf{Example:}
\begin{lstlisting}[style=XML,morekeywords={name,subType,class,type,steps}]
<Simulation>
  ...
    <Models>
    ...
    <PostProcessor name='SP' subType='SafestPoint'>
      <Distribution  class='Distributions'  type='Normal'>x1_dst</Distribution>
      <Distribution  class='Distributions'  type='Normal'>x2_dst</Distribution>
      <Distribution  class='Distributions'  type='Normal'>gammay_dst</Distribution>
      <controllable>
        <variable name='x1'>
          <distribution>x1_dst</distribution>
          <grid type='value' steps='20'>1</grid>
        </variable>
        <variable name='x2'>
          <distribution>x2_dst</distribution>
          <grid type='value' steps='20'>1</grid>
        </variable>
      </controllable>
      <non-controllable>
        <variable name='gammay'>
          <distribution>gammay_dst</distribution>
          <grid type='value' steps='20'>2</grid>
        </variable>
      </non-controllable>
    </PostProcessor>
    ...
  </Models>
  ...
</Simulation>
\end{lstlisting}
%%%%% PP LimitSurface %%%%%%%
\subsubsection{LimitSurface}
\label{LimitSurface}
The \textbf{LimitSurface} PostProcessor is aimed to identify the transition
zones that determine a change in the status of the system (Limit Surface).

\ppType{LimitSurface}{LimitSurface}

\begin{itemize}
  \item \xmlNode{parameters}, \xmlDesc{comma separated string, required field},
  lists the parameters that define the uncertain domain and from which the LS
  needs to be computed.
  \item \xmlNode{tolerance}, \xmlDesc{float, optional field}, sets the absolute
  value (in CDF) of the convergence tolerance.
 %
  This value defines the coarseness of the evaluation grid.
 %
 \default{1.0e-4}
  \item \xmlNode{side}, \xmlDesc{string, optional field}, in this node the user can specify
  which side of the limit surface needs to be computed. Three options are available:
  \\ \textit{negative},  Limit Surface corresponding to the goal function value of ``-1'';
  \\ \textit{positive}, Limit Surface corresponding to the goal function value of ``1'';
  \\ \textit{both}, either positive and negative Limit Surface is going to be computed.
  %
  %
\default{negative}
  % Assembler Objects
  \item \textbf{Assembler Objects} These objects are either required or optional
  depending on the functionality of the Adaptive Sampler.
  %
  The objects must be listed with a rigorous syntax that, except for the xml
  node tag, is common among all the objects.
  %
  Each of these nodes must contain 2 attributes that are used to map those
  within the simulation framework:
   \begin{itemize}
    \item \xmlAttr{class}, \xmlDesc{required string attribute}, is the main
    ``class'' of the listed object.
    %
    For example, it can be ``Models,'' ``Functions,'' etc.
    \item \xmlAttr{type}, \xmlDesc{required string attribute}, is the object
    identifier or sub-type.
    %
    For example, it can be ``ROM,'' ``External,'' etc.
    %
  \end{itemize}
  The \textbf{LimitSurface} post-processor requires or optionally accepts the
  following objects' types:
   \begin{itemize}
    \item \xmlNode{ROM}, \xmlDesc{string, optional field}, body of this xml
    node must contain the name of a ROM defined in the \xmlNode{Models} block
    (see section \ref{subsec:models_ROM}).
    \item \xmlNode{Function}, \xmlDesc{string, required field}, the body of
    this xml block needs to contain the name of an External Function defined
    within the \xmlNode{Functions} main block (see section \ref{sec:functions}).
    %
    This object represents the boolean function that defines the transition
    boundaries.
    %
    This function must implement a method called
    \textit{\_\_residuumSign(self)}, that returns either -1 or 1, depending on
    the system conditions (see section \ref{sec:functions}).
    %
    \end{itemize}
\end{itemize}

\textbf{Example:}
\begin{lstlisting}[style=XML,morekeywords={name,subType,debug,class,type}]
<Simulation>
 ...
 <Models>
  ...
    <PostProcessor name="computeLimitSurface" subType='LimitSurface' verbosity='debug'>
      <parameters>x0,y0</parameters>
      <ROM class='Models' type='ROM'>Acc</ROM>
      <!-- Here, you can add a ROM defined in Models block.
           If it is not Present, a nearest neighbor algorithm
           will be used.
       -->
      <Function class='Functions' type='External'>
        goalFunctionForLimitSurface
      </Function>
    </PostProcessor>
    ...
  </Models>
  ...
</Simulation>
\end{lstlisting}

%%%%% PP LimitSurfaceIntegral %%%%%%%

\subsubsection{LimitSurfaceIntegral}
\label{LimitSurfaceIntegral}
The \textbf{LimitSurfaceIntegral} PostProcessor computes the likelihood (probability) of the event, whose boundaries are
represented by the Limit Surface (either from the LimitSurface post-processor or Adaptive sampling strategies).
The inputted Limit Surface needs to be, in the  \textbf{PostProcess} step, of type  \textbf{PointSet} and needs to contain
both boundary sides (-1.0, +1.0).
%\\ The \textbf{LimitSurfaceIntegral} post-processor accepts as outputs both files (CSV) and/or  \textbf{PointSet}s.
\\ The \textbf{LimitSurfaceIntegral} post-processor accepts as output  \textbf{PointSet}s only.

\ppType{LimitSurfaceIntegral}{LimitSurfaceIntegral}
\begin{itemize}
\item \variableDescription
 \variableChildIntro
 \begin{itemize}
     \item  \xmlNode{outputName}, \xmlDesc{string, required field}, specifies the name of the output variable where the probability is going to be stored.
               \nb This variable name must be listed in the \xmlNode{Output} field of the Output DataObject
    \item   \xmlNode{distribution}, \xmlDesc{string,
               optional field}, name of the distribution that is associated to this variable.
              Its name needs to be contained in the \xmlNode{Distributions} block explained
              in Section \ref{sec:distributions}. If this node is not present, the  \xmlNode{lowerBound}
              and  \xmlNode{upperBound} XML nodes must be inputted. It requires the following two attributes:
            \begin{itemize}
              \item \xmlAttr{class}, \xmlDesc{required string attribute}, is the main
              ``class'' the listed object is from, the only acceptable class for
              this post-processor is \xmlString{Distributions};
              \item \xmlAttr{type}, \xmlDesc{required string attribute}, is the type of distributions,
                i.e. Normal, Uniform.
            \end{itemize}
   \item   \xmlNode{lowerBound}, \xmlDesc{float,
               optional field}, lower limit of integration domain for this dimension (variable).
               If this node is not present, the  \xmlNode{distribution} XML node must be inputted.
   \item   \xmlNode{upperBound}, \xmlDesc{float,
               optional field}, upper limit of integration domain for this dimension (variable).
               If this node is not present, the  \xmlNode{distribution} XML node must be inputted.
  \end{itemize}

    \item  \xmlNode{tolerance}, \xmlDesc{float, optional field}, specifies the tolerance for
               numerical integration confidence.
                \default{1.0e-4}
     \item  \xmlNode{integralType}, \xmlDesc{string, optional field}, specifies the type of integrations that
                need to be used. Currently only MonteCarlo integration is available
                \default{MonteCarlo}
    \item  \xmlNode{computeBounds}, \xmlDesc{bool, optional field},
    activates the computation of the bounding error of the limit
    surface integral ( maximum error in the identification of the
    limit surface location). If True, the bounding error is stored
    in a variable named as \xmlNode{outputName} appending the suffix
    ``\_err''. For example, if \xmlNode{outputName} is
    ``EventProbability'', the bounding error will be stored as
    ``EventProbability\_err'' (this variable name must be listed as
    variable in the output DataObject).
                \default{False}
     \item  \xmlNode{seed}, \xmlDesc{integer, optional field}, specifies the random number generator seed.
                \default{20021986}
     \item  \xmlNode{target}, \xmlDesc{string, optional field}, specifies the target name that represents
                the $f\left ( \bar{x} \right )$ that needs to be integrated.
                \default{last output found in the inputted PointSet}
\end{itemize}

\textbf{Example:}
\begin{lstlisting}[style=XML,morekeywords={name,subType,debug,class,type}]
<Simulation>
 ...
 <Models>
  ...
    <PostProcessor name="LimitSurfaceIntegralDistributions" subType='LimitSurfaceIntegral'>
        <tolerance>0.0001</tolerance>
        <integralType>MonteCarlo</integralType>
        <seed>20021986</seed>
        <target>goalFunctionOutput</target>
        <outputName>EventProbability</outputName>
        <variable name='x0'>
          <distribution>x0_distrib</distribution>
        </variable>
        <variable name='y0'>
          <distribution>y0_distrib</distribution>
        </variable>
    </PostProcessor>
    <PostProcessor name="LimitSurfaceIntegralLowerUpperBounds" subType='LimitSurfaceIntegral'>
        <tolerance>0.0001</tolerance>
        <integralType>MonteCarlo</integralType>
        <seed>20021986</seed>
        <target>goalFunctionOutput</target>
        <outputName>EventProbability</outputName>
        <variable name='x0'>
          <lowerBound>-2.0</lowerBound>
          <upperBound>12.0</upperBound>
        </variable>
        <variable name='y0'>
            <lowerBound>-1.0</lowerBound>
            <upperBound>11.0</upperBound>
        </variable>
    </PostProcessor>
    ...
  </Models>
  ...
</Simulation>
\end{lstlisting}

%%%%% PP External %%%%%%%
\subsubsection{External}
\label{External}
The \textbf{External} post-processor will execute an arbitrary python function
defined externally using the \textit{Functions} interface (see
Section~\ref{sec:functions} for more details).
%

\ppType{External}{External}

\begin{itemize}
  \item \xmlNode{method}, \xmlDesc{comma separated string, required field},
  lists the method names of an external Function that will be computed (each
  returning a post-processing value). \nb New variable names will be defined as:
  ``Function Name in this post-processor'' + ``\_`` + ``variable name in XML
  node \xmlNode{method}''. These new variables will be used to store the computed
  values from the list of methods, and can be accessed by the users through RAVEN
  entities \textbf{DataObjects} and \textbf{OutStreams}.
  \item \xmlNode{Function}, \xmlDesc{xml node, required string field}, specifies
  the name of a Function where the \textit{methods} listed above are defined.
  %
  \nb This name should match one of the Functions defined in the
  \xmlNode{Functions} block of the input file.
  %
  The objects must be listed with a rigorous syntax that, except for the XML
  node tag, is common among all the objects.
  %
  Each of these sub-nodes must contain 2 attributes that are used to map them
  within the simulation framework:

   \begin{itemize}
     \item \xmlAttr{class}, \xmlDesc{required string attribute}, is the main
     ``class'' the listed object is from, the only acceptable class for
     this post-processor is \xmlString{Functions};
     \item \xmlAttr{type}, \xmlDesc{required string attribute}, is the object
     identifier or sub-type, the only acceptable type for this post-processor is
     \xmlString{External}.
  \end{itemize}
\end{itemize}

  This Post-Processor accepts as Input/Output both \xmlString{PointSet} and \xmlString{HistorySet}:
   \begin{itemize}
    \item If a \xmlString{PointSet}  is used as Input, the parameters are passed in the external  \xmlString{Function}
  as numpy arrays. The methods' return type must be either a new array or a scalar. In the following it is reported an example
  with two methods, one that returns a scalar and the other one that returns an array:
      \begin{lstlisting}[language=python]
import numpy as np
def sum(self):
  return np.sum(self.aParameterInPointSet)

def sumTwoArraysAndReturnAnotherone(self):
  return self.aParamInPointSet1+self.aParamInPointSet2
      \end{lstlisting}
    \item If a \xmlString{HistorySet}  is used as Input, the parameters are passed in the external  \xmlString{Function}
     as a list of numpy arrays. The methods' return type must be either a new list of arrays (if the Output is another
     \xmlString{HistorySet}), a scalar or a single array (if the  Output is  \xmlString{PointSet} . In the following it
     is reported an example
     with two methods, one that returns a new list of arrays (Output = HistorySet) and the other one that returns an array (Output =
     PointSet):
      \begin{lstlisting}[language=python]
import numpy as np
def newHistorySetParameter(self):
  x = []*len(self.time)
  for history in range(len(self.time)):
    for ts in range(len(self.time[history])):
      if self.time[history][ts] >= 0.001: break
    x[history] = self.x[history][ts:]
  return x

def aNewPointSetParameter(self):
  x = []*len(self.time)
  for history in range(len(self.time)):
    x[history] = self.x[history][-1]
  return x
      \end{lstlisting}
   \end{itemize}

\textbf{Example:}
\begin{lstlisting}[style=XML,morekeywords={subType,debug,name,class,type}]
<Simulation>
  ...
  <Models>
    ...
    <PostProcessor name="externalPP" subType='External' verbosity='debug'>
      <method>Delta,Sum</method>
      <Function class='Functions' type='External'>operators</Function>
        <!-- Here, you can add a Function defined in the
             Functions block. This should be present or
             else RAVEN will not know where to find the
             defined methods. -->
    </PostProcessor>
    ...
  </Models>
  ...
</Simulation>
\end{lstlisting}

\nb The calculation results from this post-processor are stored in the internal variables. These variables
are accessible by the users through RAVEN entities \textbf{DataObjects} and \textbf{OutStreams}. The names
of these variables are defined as: ``Function Name in this post-processor'' + ``\_`` + ``variable name in XML
node \xmlNode{method}''. For example, in previous case, variables ``operators\_Delta'' and ``operators\_Sum''
are defined by RAVEN to store the outputs of this post-processor.

%%%%% PP TopologicalDecomposition %%%%%%%
\subsubsection{TopologicalDecomposition}
\label{TopologicalDecomposition}
The \textbf{TopologicalDecomposition} post-processor will compute an
approximated hierarchical Morse-Smale complex which will add two columns to a
dataset, namely \texttt{minLabel} and \texttt{maxLabel} that can be used to
decompose a dataset.
%

The topological post-processor can also be run in `interactive' mode, that is
by passing the keyword \texttt{interactive} to the command line of RAVEN's
driver.
%
In this way, RAVEN will initiate an interactive UI that allows one to explore
the topological hierarchy in real-time and adjust the simplification setting
before adjusting a dataset. Use in interactive mode will replace the parameter
\xmlNode{simplification} described below with whatever setting is set in the UI
upon exiting it.

In order to use the \textbf{TopologicalDecomposition} post-processor, the user
needs to set the attribute \xmlAttr{subType}:
\xmlNode{PostProcessor \xmlAttr{subType}=\xmlString{TopologicalDecomposition}}.
The following is a list of acceptable sub-nodes:
\begin{itemize}
  \item \xmlNode{graph} \xmlDesc{, string, optional field}, specifies the type
  of neighborhood graph used in the algorithm, available options are:
  \begin{itemize}
    \item \texttt{beta skeleton}
    \item \texttt{relaxed beta skeleton}
    \item \texttt{approximate knn}
    %\item Delaunay \textit{(disabled)}
  \end{itemize}
  \default{\texttt{beta skeleton}}
  \item \xmlNode{gradient}, \xmlDesc{string, optional field}, specifies the
  method used for estimating the gradient, available options are:
  \begin{itemize}
    \item \texttt{steepest}
    %\item \xmlString{maxflow} \textit{(disabled)}
  \end{itemize}
  \default{\texttt{steepest}}
  \item \xmlNode{beta}, \xmlDesc{float in the range: (0,2], optional field}, is
  only used when the \xmlNode{graph} is set to \texttt{beta skeleton} or
  \texttt{relaxed beta skeleton}.
  \default{1.0}
  \item \xmlNode{knn}, \xmlDesc{integer, optional field}, is the number of
  neighbors when using the \xmlString{approximate knn} for the \xmlNode{graph}
  sub-node and used to speed up the computation of other graphs by using the
  approximate knn graph as a starting point for pruning. -1 means use a fully
  connected graph.
  \default{-1}
  \item \xmlNode{weighted}, \xmlDesc{boolean, optional}, a flag that specifies
  whether the regression models should be probability weighted.
  \default{False}
  \item \xmlNode{interactive}, if this node is present \emph{and} the user has
  specified the keyword \texttt{interactive} at the command line, then this will
  initiate a graphical interface for exploring the different simplification
  levels of the topological hierarchy. Upon exit of the graphical interface, the
  specified simplification level will be updated to use the last value of the
  graphical interface before writing any ``output'' results.
  \item \xmlNode{persistence}, \xmlDesc{string, optional field}, specifies how
  to define the hierarchical simplification by assigning a value to each local
  minimum and maximum according to the one of the strategy options below:
  \begin{itemize}
    \item \texttt{difference} - The function value difference between the
    extremum and its closest-valued neighboring saddle.
    \item \texttt{probability} - The probability integral computed as the
    sum of the probability of each point in a cluster divided by the count of
    the cluster.
    \item \texttt{count} - The count of points that flow to or from the
    extremum.
    % \item \xmlString{area} - The area enclosed by the manifold that flows to
    % or from the extremum.
  \end{itemize}
  \default{\texttt{difference}}
  \item \xmlNode{simplification}, \xmlDesc{float, optional field}, specifies the
  amount of noise reduction to apply before returning labels.
  \default{0}
  \item \xmlNode{parameters}, \xmlDesc{comma separated string, required field},
  lists the parameters defining the input space.
  \item \xmlNode{response}, \xmlDesc{string, required field}, is a single
  variable name defining the scalar output space.
\end{itemize}
\textbf{Example:}
\begin{lstlisting}[style=XML,morekeywords={subType}]
<Simulation>
  ...
  <Models>
    ...
    <PostProcessor name="***" subType='TopologicalDecomposition'>
      <graph>beta skeleton</graph>
      <gradient>steepest</gradient>
      <beta>1</beta>
      <knn>8</knn>
      <normalization>None</normalization>
      <parameters>X,Y</parameters>
      <response>Z</response>
      <weighted>true</weighted>
      <simplification>0.3</simplification>
      <persistence>difference</persistence>
    </PostProcessor>
    ...
  <Models>
  ...
<Simulation>
\end{lstlisting}

%%%%% PP DataMining %%%%%%%
\input{PostProcessors/DataMining.tex}

%%%%%%%%%%%%%% ParetoFrontier PP %%%%%%%%%%%%%%%%%%%

\subsubsection{ParetoFrontier}
\label{ParetoFrontierPP}
The \textbf{ParetoFrontier} PostProcessor is designed to identify the points lying on the Pareto Frontier in a multi-dimensional trade-space.
This post-processor receives as input a \textbf{DataObject} (a PointSet only) which contains all data points in the trade-space space and it
returns the subset of points lying in the Pareto Frontier as a PointSet.

It is here assumed that each data point of the input PointSet is a realization of the system under consideration for a
specific configuration to which corresponds several objective variables (e.g., cost and value).

%
\ppType{ParetoFrontier}{ParetoFrontier}
%
\begin{itemize}
  \item   \xmlNode{objective},\xmlDesc{string, required parameter}, ID of the objective variable that represents a dimension of the trade-space space.
          The \xmlNode{costID} requires one identifying attribute:
          \begin{itemize}
            \item \xmlAttr{goal}, \xmlDesc{string, required field}, Goal of the objective variable characteristic: minimization (min) or maximization (max)
            \item \xmlAttr{upperLimit}, \xmlDesc{string, optional field}, Desired upper limit of the objective variable for the points in the Pareto frontier
            \item \xmlAttr{lowerLimit}, \xmlDesc{string, optional field}, Desired lower limit of the objective variable for the points in the Pareto frontier
          \end{itemize}
\end{itemize}

The following is an example where a set of realizations (the ``candidates'' PointSet) has been generated by changing two parameters
(var1 and var2) which produced two output variables: cost (which it is desired to be minimized) and value (which it is desired to be maximized).
The \textbf{ParetoFrontier} post-processor takes the ``candidates'' PointSet and populates a Point similar in structure
(the ``paretoPoints'' PointSet).

\textbf{Example:}
\begin{lstlisting}[style=XML,morekeywords={anAttribute},caption=ParetoFrontier input example (no expand)., label=lst:ParetoFrontier_PP_InputExample]
  <Models>
    <PostProcessor name="paretoPP" subType="ParetoFrontier">
      <objective goal='min' upperLimit='0.5'>cost</objective>
      <objective goal='max' lowerLimit='0.5'>value</objective>
    </PostProcessor>
  </Models>

  <Steps>
    <PostProcess name="PP">
      <Input     class="DataObjects"  type="PointSet"        >candidates</Input>
      <Model     class="Models"       type="PostProcessor"   >paretoPP</Model>
      <Output    class="DataObjects"  type="PointSet"        >paretoPoints</Output>
    </PostProcess>
  </Steps>

  <DataObjects>
    <PointSet name="candidates">
      <Input>var1,var2</Input>
      <Output>cost,value</Output>
    </PointSet>
    <PointSet name="paretoPoints">
      <Input>var1,var2</Input>
      <Output>cost,value</Output>
    </PointSet>
  </DataObjects>
\end{lstlisting}

\nb it is possible to specify both upper and lower limits for each objective variable.
When one or both of these limits are specified, then the Pareto frontier is filtered such that all Pareto frontier points that
satisfy those limits are preserved.

%%%%%%%%%%%%%% Metric PP %%%%%%%%%%%%%%%%%%%

\subsubsection{Metric}
\label{MetricPP}
The \textbf{Metric} PostProcessor is specifically used to calculate the distance values among points from PointSets and histories from HistorySets,
while the \textbf{Metrics} block (See Chapter \ref{sec:Metrics}) allows the user to specify the similarity/dissimilarity metrics to be used in this
post-processor. Both \textbf{PointSet} and \textbf{HistorySet} can be accepted by this post-processor.
If the name of given variable is unique, it can be used directly, otherwise the variable can be specified
with $DataObjectName|InputOrOutput|VariableName$ like other places in RAVEN.
Some of the Metrics also accept distributions to calculate the distance against.
These are specified by using the name of the distribution.
%
\ppType{Metric}{Metric}
%
\begin{itemize}
  \item \xmlNode{Features}, \xmlDesc{comma separated string, required field}, specifies the names of the features.
    This xml-node accepts the following attribute:
    \begin{itemize}
      \item \xmlAttr{type}, \xmlDesc{required string attribute}, the type of provided features. Currently only
        accept `variable'.
    \end{itemize}
  \item \xmlNode{Targets}, \xmlDesc{comma separated string, required field}, contains a comma separated list of
    the targets. \nb Each target is paired with a feature listed in xml node \xmlNode{Features}. In this case, the
    number of targets should be equal to the number of features.
    This xml-node accepts the following attribute:
    \begin{itemize}
      \item \xmlAttr{type}, \xmlDesc{required string attribute}, the type of provided features. Currently only
        accept `variable'.
    \end{itemize}
  \item \xmlNode{multiOutput}, \xmlDesc{optional string attribute}, only used when \textbf{HistorySet} is used as
    input. Defines aggregating of time-dependent metrics' calculations. Available options include:
    \textbf{mean, max, min, raw\_values} over the time. For example, when `mean' is used, the metrics' calculations
    will be averaged over the time. When `raw\_values' is used, the full set of  metrics' calculations will be dumped.
    \default{raw\_values}
  \item \xmlNode{weight}, \xmlDesc{comma separated floats, optional field}, when `mean' is provided for \xmlNode{multiOutput},
    the user can specify the weights that can be used for the average calculation of all outputs.
  \item \xmlNode{pivotParameter}, \xmlDesc{optional string attribute}, only used when \textbf{HistorySet}
    is used as input. The pivotParameter for given metrics' calculations.
    \default{time}
  \item \xmlNode{Metric}, \xmlDesc{string, required field}, specifies the \textbf{Metric} name that is defined via
    \textbf{Metrics} entity. In this xml-node, the following xml attributes need to be specified:
    \begin{itemize}
      \item \xmlAttr{class}, \xmlDesc{required string attribute}, the class of this metric (e.g. Metrics)
      \item \xmlAttr{type}, \xmlDesc{required string attribute}, the sub-type of this Metric (e.g. SKL, Minkowski)
    \end{itemize}
\end{itemize}

\textbf{Example:}

\begin{lstlisting}[style=XML]
<Simulation>
 ...
  <Models>
    ...
    <PostProcessor name="pp1" subType="Metric">
      <Features type="variable">ans</Features>
      <Targets type="variable">ans2</Targets>
      <Metric class="Metrics" type="SKL">euclidean</Metric>
      <Metric class="Metrics" type="SKL">cosine</Metric>
      <Metric class="Metrics" type="SKL">manhattan</Metric>
      <Metric class="Metrics" type="ScipyMetric">braycurtis</Metric>
      <Metric class="Metrics" type="ScipyMetric">canberra</Metric>
      <Metric class="Metrics" type="ScipyMetric">correlation</Metric>
      <Metric class="Metrics" type="ScipyMetric">minkowski</Metric>
    </PostProcessor>
    ...
  </Models>
 ...
</Simulation>
\end{lstlisting}

In order to access the results from this post-processor, RAVEN will define the variables as ``MetricName'' +
``\_'' + ``TargetVariableName'' + ``\_'' + ``FeatureVariableName'' to store the calculation results, and these
variables are also accessible by the users through RAVEN entities \textbf{DataObjects} and \textbf{OutStreams}.
\nb We will replace ``|'' in ``TargetVariableName'' and ``FeatureVariableName'' with ``\_''.
In previous example, variables such as \textit{euclidean\_ans2\_ans, cosine\_ans2\_ans, poly\_ans2\_ans} are accessible
by the users.

%%%%%%%%%%%%%% Cross Validation PP %%%%%%%%%%%%%%%%%%%

\subsubsection{CrossValidation}
\label{CVPP}
The \textbf{CrossValidation} post-processor is specifically used to evaluate estimator (i.e., ROMs) performance.
Cross-validation is a statistical method of evaluating and comparing learning algorithms by dividing data into
two portions: one used to `train' a surrogate model and the other used to validate the model, based on specific
scoring metrics. In typical cross-validation, the training and validation sets must crossover in successive
rounds such that each data point has a chance of being validated against the various sets. The basic form of
cross-validation is k-fold cross-validation. Other forms of cross-validation are special cases of k-fold or involve
repeated rounds of k-fold cross-validation. \nb It is important to notice that this post-processor currently can
only accept \textbf{PointSet} data object.
%
\ppType{CrossValidation}{CrossValidation}
%
\begin{itemize}
  \item \xmlNode{SciKitLearn}, \xmlDesc{string, required field}, the subnodes specifies the necessary information
    for the algorithm to be used in the post-processor. `SciKitLearn' is based on algorithms in SciKit-Learn
    library, and currently it performs cross-validation over \textbf{PointSet} only.
  \item \xmlNode{Metric}, \xmlDesc{string, required field}, specifies the \textbf{Metric} name that is defined via
    \textbf{Metrics} entity. In this xml-node, the following xml attributes need to be specified:
    \begin{itemize}
      \item \xmlAttr{class}, \xmlDesc{required string attribute}, the class of this metric (e.g. Metrics)
      \item \xmlAttr{type}, \xmlDesc{required string attribute}, the sub-type of this Metric (e.g. SKL, Minkowski)
    \end{itemize}
    \nb Currently, cross-validation post-processor only accepts \xmlNode{SKL} metrics with \xmlNode{metricType}
    \xmlString{mean\_absolute\_error}, \xmlString{explained\_variance\_score}, \xmlString{r2\_score},
    \xmlString{mean\_squared\_error}, and \xmlString{median\_absolute\_error}.
\end{itemize}

\textbf{Example:}

\begin{lstlisting}[style=XML]
<Simulation>
 ...
  <Files>
    <Input name="output_cv" type="">output_cv.xml</Input>
    <Input name="output_cv.csv" type="">output_cv.csv</Input>
  </Files>
  <Models>
    ...
    <ROM name="surrogate" subType="SciKitLearn">
      <SKLtype>linear_model|LinearRegression</SKLtype>
      <Features>x1,x2</Features>
      <Target>ans</Target>
      <fit_intercept>True</fit_intercept>
      <normalize>True</normalize>
    </ROM>
    <PostProcessor name="pp1" subType="CrossValidation">
        <SciKitLearn>
            <SKLtype>KFold</SKLtype>
            <n_splits>3</n_splits>
            <shuffle>False</shuffle>
        </SciKitLearn>
        <Metric class="Metrics" type="SKL">m1</Metric>
    </PostProcessor>
    ...
  </Models>
  <Metrics>
    <SKL name="m1">
      <metricType>mean_absolute_error</metricType>
    </SKL>
  </Metrics>
  <Steps>
    <PostProcess name="PP1">
        <Input class="DataObjects" type="PointSet">outputDataMC</Input>
        <Input class="Models" type="ROM">surrogate</Input>
        <Model class="Models" type="PostProcessor">pp1</Model>
        <Output class="Files" type="">output_cv</Output>
        <Output class="Files" type="">output_cv.csv</Output>
    </PostProcess>
  </Steps>
 ...
</Simulation>
\end{lstlisting}

In order to access the results from this post-processor, RAVEN will define the variables as ``cv'' +
``\_'' + ``MetricName'' + ``\_'' + ``ROMTargetVariable'' to store the calculation results, and these
variables are also accessible by the users through RAVEN entities \textbf{DataObjects} and \textbf{OutStreams}.
In previous example, variable \textit{cv\_m1\_ans} are accessible by the users.

\paragraph{SciKitLearn}

The algorithm for cross-validation is chosen by the subnode \xmlNode{SKLtype} under the parent node \xmlNode{SciKitLearn}.
In addition, a special subnode \xmlNode{average} can be used to obtain the average cross validation results.

\begin{itemize}
  \item \xmlNode{SKLtype}, \xmlDesc{string, required field}, contains a string that
    represents the cross-validation algorithm to be used. As mentioned, its format is:

    \xmlNode{SKLtype}algorithm\xmlNode{/SKLtype}.
  \item \xmlNode{average}, \xmlDesc{boolean, optional field}, if `True`, dump the average cross validation results into the
    output files.
\end{itemize}

Based on the \xmlNode{SKLtype} several different algorithms are available. In the following paragraphs a brief
explanation and the input requirements are reported for each of them.

\paragraph{K-fold}
\textbf{KFold} divides all the samples in $k$ groups of samples, called folds (if $k=n$, this is equivalent to the
\textbf{Leave One Out} strategy), of equal sizes (if possible). The prediction function is learned using $k-1$ folds,
and fold left out is used for test.
In order to use this algorithm, the user needs to set the subnode:
\xmlNode{SKLtype}KFold\xmlNode{/SKLtype}.
In addition to this XML node, several others are available:
\begin{itemize}
  \item \xmlNode{n\_splits}, \xmlDesc{integer, optional field}, number of folds, must be at least 2. \default{3}
  \item \xmlNode{shuffle}, \xmlDesc{boolean, optional field}, whether to shuffle the data before splitting into
    batches.
  \item \xmlNode{random\_state}, \xmlDesc{integer, optional field}, when shuffle=True,
    pseudo-random number generator state used for shuffling. If not present, use default numpy RNG for shuffling.
\end{itemize}

\paragraph{Stratified k-fold}
\textbf{StratifiedKFold} is a variation of \textit{k-fold} which returns stratified folds: each set contains approximately
the same percentage of samples of each target class as the complete set.
In order to use this algorithm, the user needs to set the subnode:

\xmlNode{SKLtype}StratifiedKFold\xmlNode{/SKLtype}.

In addition to this XML node, several others are available:
\begin{itemize}
  \item \xmlNode{labels}, \xmlDesc{list of integers, (n\_samples), required field}, contains a label for each sample.
  \item \xmlNode{n\_splits}, \xmlDesc{integer, optional field}, number of folds, must be at least 2. \default{3}
  \item \xmlNode{shuffle}, \xmlDesc{boolean, optional field}, whether to shuffle the data before splitting into
    batches.
  \item \xmlNode{random\_state}, \xmlDesc{integer, optional field}, when shuffle=True,
    pseudo-random number generator state used for shuffling. If not present, use default numpy RNG for shuffling.
\end{itemize}

\paragraph{Label k-fold}
\textbf{LabelKFold} is a variation of \textit{k-fold} which ensures that the same label is not in both testing and
training sets. This is necessary for example if you obtained data from different subjects and you want to avoid
over-fitting (i.e., learning person specific features) by testing and training on different subjects.
In order to use this algorithm, the user needs to set the subnode:

\xmlNode{SKLtype}LabelKFold\xmlNode{/SKLtype}.

In addition to this XML node, several others are available:
\begin{itemize}
  \item \xmlNode{labels}, \xmlDesc{list of integers with length (n\_samples, ), required field}, contains a label for
    each sample. The folds are built so that the same label does not appear in two different folds.
  \item \xmlNode{n\_splits}, \xmlDesc{integer, optional field}, number of folds, must be at least 2. \default{3}
\end{itemize}

\paragraph{Leave-One-Out - LOO}
\textbf{LeaveOneOut} (or LOO) is a simple cross-validation. Each learning set is created by taking all the samples
except one, the test set being the sample left out. Thus, for $n$ samples, we have $n$ different training sets and
$n$ different tests set. This is cross-validation procedure does not waste much data as only one sample is removed from
the training set.
In order to use this algorithm, the user needs to set the subnode:

\xmlNode{SKLtype}LeaveOneOut\xmlNode{/SKLtype}.

\paragraph{Leave-P-Out - LPO}
\textbf{LeavePOut} is very similar to \textbf{LeaveOneOut} as it creates all the possible training/test sets by removing
$p$ samples from the complete set. For $n$ samples, this produces $(^n_p)$ train-test pairs. Unlike \textbf{LeaveOneOut}
and \textbf{KFold}, the test sets will overlap for $p > 1$.
In order to use this algorithm, the user needs to set the subnode:

\xmlNode{SKLtype}LeavePOut\xmlNode{/SKLtype}.

In addition to this XML node, several others are available:
\begin{itemize}
  \item \xmlNode{p}, \xmlDesc{integer, required field}, size of the test sets
\end{itemize}

\paragraph{Leave-One-Label-Out - LOLO}
\textbf{LeaveOneLabelOut} (LOLO) is a cross-validation scheme which holds out the samples according to a third-party
provided array of integer labels. This label information can be used to encode arbitrary domain specific pre-defined
cross-validation folds. Each training set is thus constituted by all samples except the ones related to a specific
label.
In order to use this algorithm, the user needs to set the subnode:

\xmlNode{SKLtype}LeaveOneLabelOut\xmlNode{/SKLtype}.

In addition to this XML node, several others are available:
\begin{itemize}
  \item \xmlNode{labels}, \xmlDesc{list of integers, (n\_samples,), required field}, arbitrary
    domain-specific stratificatioin of the data to be used to draw the splits.
\end{itemize}

\paragraph{Leave-P-Label-Out}
\textbf{LeavePLabelOut} is imilar as \textit{Leave-One-Label-Out}, but removes samples related to $P$ labels for
each training/test set.
In order to use this algorithm, the user needs to set the subnode:

\xmlNode{SKLtype}LeavePLabelOut\xmlNode{/SKLtype}.

In addition to this XML node, several others are available:
\begin{itemize}
  \item \xmlNode{labels}, \xmlDesc{list of integers, (n\_samples,), required field}, arbitrary
    domain-specific stratificatioin of the data to be used to draw the splits.
  \item \xmlNode{n\_groups}, \xmlDesc{integer, optional field}, number of samples to leave out in the test split.
\end{itemize}

\paragraph{ShuffleSplit}
\textbf{ShuffleSplit} iterator will generate a user defined number of independent train/test dataset splits. Samples
are first shuffled and then split into a pair of train and test sets. it is possible to control the randomness for
reproducibility of the results by explicitly seeding the \xmlNode{random\_state} pseudo random number generator.
In order to use this algorithm, the user needs to set the subnode:

\xmlNode{SKLtype}ShuffleSplit\xmlNode{/SKLtype}.

In addition to this XML node, several others are available:
\begin{itemize}
  \item \xmlNode{n\_splits}, \xmlDesc{integer, optional field}, number of re-shuffling and splitting iterations
    \default{10}.
  \item \xmlNode{test\_size}, \xmlDesc{float or integer, optional field}, if float, should be between 0.0 and 1.0 and
    represent the proportion of the dataset to include in the test split. \default{0.1}
    If integer, represents the absolute number of test samples. If not present, the value is automatically set to
    the complement of the train size.
  \item \xmlNode{train\_size}, \xmlDesc{float or integer, optional field}, if float, should be between 0.0 and 1.0 and represent
    the proportion of the dataset to include in the train split. If integer, represents the absolute number of train
    samples. If not present, the value is automatically set to the complement of the test size.
  \item \xmlNode{random\_state}, \xmlDesc{integer, optional field}, when shuffle=True,
    pseudo-random number generator state used for shuffling. If not present, use default numpy RNG for shuffling.
\end{itemize}

\paragraph{Label-Shuffle-Split}
\textbf{LabelShuffleSplit} iterator behaves as a combination of \textbf{ShuffleSplit} and \textbf{LeavePLabelOut},
and generates a sequence of randomized partitions in which a subset of labels are held out for each split.
In order to use this algorithm, the user needs to set the subnode:

\xmlNode{SKLtype}LabelShuffleSplit\xmlNode{/SKLtype}.

In addition to this XML node, several others are available:
\begin{itemize}
  \item \xmlNode{labels}, \xmlDesc{list of integers, (n\_samples)}, labels of samples.
  \item \xmlNode{n\_splits}, \xmlDesc{integer, optional field}, number of re-shuffling and splitting iterations
    \default{10}.
  \item \xmlNode{test\_size}, \xmlDesc{float or integer, optional field}, if float, should be between 0.0 and 1.0 and
    represent the proportion of the dataset to include in the test split. \default{0.1}
    If integer, represents the absolute number of test samples. If not present, the value is automatically set to
    the complement of the train size.
  \item \xmlNode{train\_size}, \xmlDesc{float or integer, optional field}, if float, should be between 0.0 and 1.0 and represent
    the proportion of the dataset to include in the train split. If integer, represents the absolute number of train
    samples. If not present, the value is automatically set to the complement of the test size.
  \item \xmlNode{random\_state}, \xmlDesc{integer, optional field}, when shuffle=True,
    pseudo-random number generator state used for shuffling. If not present, use default numpy RNG for shuffling.
\end{itemize}

%%%%%%%%%%%%%% ValueDuration %%%%%%%%%%%%%%%%%%%
\subsubsection{ValueDuration}
\label{ValueDurationPP}
The \xmlNode{ValueDuration} PostProcessor is a tool to construct a particular kind of histogram, where the
independent variable is the number of times a variable exceeds a particular value, and the dependent variable
is the values themselves.  An example of this is the Load Duration Curve in energy modeling. This approach is
similar to that used in Lebesgue integration. Note that for each realization in the input
\xmlNode{HistorySet}, a separate load duration curve will be created for each target.

The \xmlNode{ValueDuration} PostProcessor can only act on \xmlNode{HistorySet} data objects, and generates a
\xmlNode{HistorySet} in return.  Two output variables are created for each \xmlAttr{target}:
\xmlString{counts\_x} and \xmlString{bins\_x}, where \xmlString{x} is replaced by the name of the target.
These must be specified in the output data object in order to be collected.

To plot a traditional Load Duration Curve, the x-axis should be the bins variable, and the y-axis should be
the counts variable.

\ppType{ValueDuration}{ValueDuration}
%
\begin{itemize}
  \item \xmlNode{target}, \xmlDesc{comma separated strings, required field}, specifies the names of the
    target(s) for which Value Duration histograms should be generated.
  \item \xmlNode{bins}, \xmlDesc{integer, required field}, specifies the number of bins that the values of the
    targets should be counted into.
\end{itemize}

\textbf{Example:}

\begin{lstlisting}[style=XML]
<Simulation>
 ...
  <Models>
    ...
    <PostProcessor name="pp" subType="ValueDuration">
      <target>x, y</target>
      <bins>100</bins>
    </PostProcessor>
    ...
  </Models>
 ...
</Simulation>
\end{lstlisting}

%%%%%%%%%%%%%% FastFourierTransform %%%%%%%%%%%%%%%%%%%
\subsubsection{FastFourierTransform}
\label{FastFourierTransformPP}
The \xmlNode{FastFourierTransform} PostProcessor provides access to the Numpy fast Fourier transform function
\texttt{numpy.fft.fft}
and provides the frequencies, periods, and amplitudes from performing the transform. The periods are simply
the inverse of the frequencies, and the frequency units are the deltas between pivot values in the provided
input. For example, if data is collected every 3600 seconds, the units of frequency are per-hour.  This
PostProcessor expects uniformly-spaced pivot values. Note that for each realization in the input data object,
a separate fft will be created for each target.

The \xmlNode{FastFourierTransform} PostProcessor can act on any target in a DataObject that depends on a
single index, and generates three histories per sample per target: an independent variable
\xmlString{target\_fft\_frequency}, and two dependent values \xmlString{target\_fft\_period} and
\xmlString{target\_fft\_amplitude}, which both depend on the frequency by default. In all three outputs,
\emph{target} is replaced by the name of the target for which the fft was requested.

\ppType{FastFourierTransform}{FastFourierTransform}
%
\begin{itemize}
  \item \xmlNode{target}, \xmlDesc{comma separated strings, required field}, specifies the names of the
    target(s) for which the fast Fourier transform should be calculated.
 \end{itemize}
\textbf{Example:}

\begin{lstlisting}[style=XML]
<Simulation>
 ...
  <Models>
    ...
    <PostProcessor name="pp" subType="FastFourierTransform">
      <target>x, y</target>
    </PostProcessor>
    ...
  </Models>
 ...
</Simulation>
\end{lstlisting}
%%%%%%%%%%%%%% SampleSelector %%%%%%%%%%%%%%%%%%%
\subsubsection{SampleSelector}
\label{SampleSelectorPP}
The \xmlNode{SampleSelector} PostProcessor is a tool to select a row from a dataset, depending on different
criteria. The different criteria that can be used are listed below.

The \xmlNode{SampleSelector} PostProcessor can  act on any \xmlNode{DataObjects}, and generates a
\xmlNode{DataObject} with a single realization in return.

\ppType{SampleSelector}{SampleSelector}
%
\begin{itemize}
  \item \xmlNode{criterion}, \xmlDesc{string, required field}, specifies the criterion to select the
    realization from the input DataObject. Options are as follows:
    \begin{itemize}
      \item \xmlString{min}, choose the realization that has the lowest value of the \xmlNode{target}
        variable. The target must be scalar.
      \item \xmlString{max}, choose the realization that has the highest value of the \xmlNode{target}
        variable. The target must be scalar.
      \item \xmlString{index}, choose the realization that has the provided index. The index must be an
        integer and is zero-based, meaning the first entry is at index 0, the second entry is at index 1, etc.
        The realization order is taken from the order in which they were entered originally into the input
        DataObject. If this option is used, the \xmlNode{criterion} node must have an \xmlAttr{value}
        attribute that gives the index.
    \end{itemize}
  \item \xmlNode{target}, \xmlDesc{string, optional field}, required if the criterion targets a particular
    variable (such as the minimum and maximum criteria). Specifies the name of the
    target for which the criterion should be evaluated.
\end{itemize}

\textbf{Example:}

\begin{lstlisting}[style=XML]
<Simulation>
 ...
  <Models>
    ...
    <PostProcessor name="select_min" subType="SampleSelector">
      <target>x</target>
      <criterion>min</criterion>
    </PostProcessor>
    ...
    <PostProcessor name="select_index" subType="SampleSelector">
      <criterion value='3'>index</criterion>
    </PostProcessor>
    ...
  </Models>
 ...
</Simulation>
\end{lstlisting}

%%%%% PP Validation %%%%%%%
\subsubsection{Validation}
\label{subsubsec:Validation}

The \xmlNode{Validation} PostProcessor represents a gate
for applying a different range of algorithms to validate (e.g. compare)
dataset and/or models (e.g. Distributions).
The post-processor is in charge of deploying a common infrastructure
for the user of  \textbf{Validation} problems.
Several algorithms are available within this post-processor:
\begin{itemize}
  \item  \textbf{Probabilistic}, for Static and Time-dependent data
  \item  \textbf{PPDSS}, for Time-dependent data
  % \item  \textbf{Representativity}
  % \item  \textbf{PCM}
\end{itemize}
%

The \textbf{Validation} post-processor makes use of the \textbf{Metric} system (See Chapter \ref{sec:Metrics}) to, in conjunction with the specific algorithm chosen from the list above,
to report validation scores for both static and time-dependent data.
Indeed, Both \textbf{PointSet} and \textbf{HistorySet} can be accepted by this post-processor (depending on which algorithm is chosen).
If the name of given variable to be compared is unique, it can be used directly, otherwise the variable can be specified
with $DataObjectName|InputOrOutput|VariableName$ nomenclature.

%
\ppType{Validation}{Validation}
%
\begin{itemize}
  \item \xmlNode{Features}, \xmlDesc{comma separated string, required field}, specifies the names of the features.
  \item \xmlNode{Targets}, \xmlDesc{comma separated string, required field}, contains a comma separated list of
     targets. \nb Each target is paired with a feature listed in xml node \xmlNode{Features}. In this case, the
    number of targets should be equal to the number of features.
    \item \xmlNode{Metric}, \xmlDesc{string, required field}, specifies the \textbf{Metric} name that is defined via
    \textbf{Metrics} entity. In this xml-node, the following xml attributes need to be specified:
    \begin{itemize}
      \item \xmlAttr{class}, \xmlDesc{required string attribute}, the class of this metric (e.g., Metrics)
      \item \xmlAttr{type}, \xmlDesc{required string attribute}, the sub-type of this Metric (e.g., SKL, Minkowski)
    \end{itemize}
    The choice of the available metrics depends on the specific validation algorithm that is chosen (see table \ref{tab:ValidationAlgorithms})
\end{itemize}

In addition to the nodes above, the user must use the sub-nodes specific to the chosen validation algorithm:
\begin{itemize}
  \item Probabilistic: specify that the validation needs to be performed
  using the Probabilistic metrics: \textbf{CDFAreaDifference} (see \ref{subsubsec:metric_CDFAreaDifference})  or \textbf{PDFCommonArea} (see \ref{subsubsec:metric_PDFCommonArea})
  This validation algorithm must have the following sub-nodes for HistorySet:
    \begin{itemize}
      \item \xmlNode{pivotParameter}, \xmlDesc{string, required field}, specifies the pivotParameter for a <HistorySet>. The pivot parameter is the shared index of the output variables in the data object.
    \end{itemize}
  \item PPDSS: specify that the validation needs to be performed
  using the PPDSS metrics: \textbf{DSS} (\ref{subsection:DSS})
  This validation algorithm must have the following sub-nodes:
    \begin{itemize}
      \item \xmlNode{pivotParameterFeature}, \xmlDesc{string, required field}, specifies the pivotParameter for a feature <HistorySet>. The feature pivot parameter is the shared index of the output variables in the data object.
      \item \xmlNode{pivotParameterTarget}, \xmlDesc{string, required field}, specifies the pivotParameter for a target <HistorySet>. The target pivot parameter is the shared index of the output variables in the data object.
      \item \xmlNode{multiOutput}, \xmlDesc{string, required field}, to extract raw values for the HistorySet. The user must use ‘raw values’ for the full set of metrics’ calculations to be dumped.
      \item \xmlNode{scale}, \xmlDesc{string, required field}, specifies the type of time scaling. The following are the options for scaling (specific definitions for each scaling type is provided in \ref{sec:dssdoc}):
        \begin{itemize}
          \item \textbf{DataSynthesis}, calculating the distortion for two data sets without applying other scaling ratios.
          \item \textbf{2\_2\_affine}, calculating the distortion for two data sets with scaling ratios for parameter of interest and agent of changes.
          \item \textbf{dilation}, calculating the distortion for two data sets with scaling ratios for parameter of interest and agent of changes.
          \item \textbf{beta\_strain}, calculating the distortion for two data sets with scaling ratio for parameter of interest.
          \item \textbf{omega\_strain}, calculating the distortion for two data sets with scaling ratios for agent of changes.
          \item \textbf{identity}, calculating the distortion for two data sets with scaling ratios of 1.
        \end{itemize}
      \item \xmlNode{scaleBeta}, \xmlDesc{float or comma separated list of floats, required field}, specifies the parameter of interest scaling ratio between the feature and target.
      To provide more than one scaling factor, separate by adding a comma in between each number. Providing more than one scaling factor presumes there are more than one parameter to be post-processed.
      If so, \xmlNode{Features}, \xmlNode{Targets}, and \xmlNode{scaleOmega} must have the same number scaling factors.
      \item \xmlNode{scaleOmega}, \xmlDesc{float or comma separated list of floats, required field}, specifies the agents of change scaling ratio between the feature and target.
      To provide more than one scaling factor, separate by adding a comma in between each number. Providing more than one scaling factor presumes there are more than one parameter to be post-processed.
      If so, \xmlNode{Features}, \xmlNode{Targets}, and \xmlNode{scaleBeta} must have the same number scaling factors.
    \end{itemize}
  %\item \xmlNode{DSS}, \xmlDesc{XML node, optional field}, specify that the validation needs to be performed via DSS.
  %This xml-node accepts the following attribute:
  %  \begin{itemize}
  %    \item \xmlAttr{ name}, \xmlDesc{required string attribute}, the  user defined name of the validation algorithm used as prefix for the output results.
  %  \end{itemize}
  %  The following subnodes must be inputted:
  %   \begin{itemize}
  %     \item \xmlNode{myNode}, \xmlDesc{comma separated string, required field}, DESCRIPTION
  %  \end{itemize}
\end{itemize}

\begin{table}[]
\caption{Validation Algorithms and respective available metrics and DataObjects}
\label{tab:ValidationAlgorithms}
\begin{tabular}{|c|c|c|}
\hline
\textbf{Validation Algorithm} & \textbf{DataObject}                                            & \textbf{Available Metrics}                                                   \\ \hline
Probabilistic                 & \begin{tabular}[c]{@{}c@{}}PointSet \\ HistorySet\end{tabular} & \begin{tabular}[c]{@{}c@{}}CDFAreaDifference\\ \\ PDFCommonArea\end{tabular} \\ \hline
PPDSS                         & HistorySet                                                     & DSS                                                           \\ \hline
\end{tabular}
\end{table}

\textbf{Example:}
\begin{lstlisting}[style=XML,morekeywords={subType}]
<Simulation>
  ...
  <Models>
    ...
    <PostProcessor name="pp1" subType="Probabilistic">
      <Features>outputDataMC1|ans</Features>
      <Targets>outputDataMC2|ans2</Targets>
      <Metric class="Metrics" type="CDFAreaDifference">cdf_diff</Metric>
      <Metric class="Metrics" type="PDFCommonArea">pdf_area</Metric>
    </PostProcessor>
    ...
  <Models>
  ...
<Simulation>
\end{lstlisting}

%%%%% PP EconomicRatio %%%%%%%
\input{PostProcessors/EconomicRatio.tex}

%%%%% HistorySetDelay %%%%%%
\input{PostProcessors/HistorySetDelay.tex}

%%%%% HStoPSOperator %%%%%%
\input{PostProcessors/HStoPSOperator.tex}

%%%%% HistorySetSampling %%%%%%
\input{PostProcessors/HistorySetSampling.tex}

%%%%% HistorySetSync %%%%%%
\input{PostProcessors/HistorySetSync.tex}

%%%%% HistorySetSnapShot %%%%%%
\input{PostProcessors/HistorySetSnapShot.tex}

%%%%% HS2PS %%%%%%
\input{PostProcessors/HS2PS.tex}

%%%%% TypicalHistoryFromHistorySet %%%%%%
\input{PostProcessors/TypicalHistoryFromHistorySet.tex}

%%%%% dataObjectLabelFilter %%%%%%
\input{PostProcessors/dataObjectLabelFilter.tex}

%%%%%%%%%%%%%% InterfacedPostProcessors %%%%%%%%%%%%%%%%
% To be replaced by the PostProcessor Plugin
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%\input{PostProcessors/InterfacedPostProcessors.tex}

\subsubsection{TSACharacterizer}
\label{TSACharacterizer}
The \xmlNode{TSACharacterizer} PostProcessor is a tool to characterize sets of histories using
the Time Series Analysis (TSA) module. It takes each history realization from the input data object
and returns characterizations. Note the characterizations are entirely stored in the \texttt{metadata}
portion of the output data object; no data is store in the input or output.

The \xmlNode{SampleSelector} PostProcessor can only act on \xmlNode{HistorySet} \xmlNode{DataObjects},
and generates a \xmlNode{PointSet} \xmlNode{DataObject} in return with as many realizations as the input
history set.

\ppType{TSACharacterizer}{TSACharacterizer}
%
\begin{itemize}
  \item \xmlNode{pivotParameter}, \xmlDesc{string, required field}, specifies the name of the time-like
    monotonic variable used for the signals in the input history set.
\end{itemize}

\tsaList

\textbf{TSACharacterizer Example:}

\begin{lstlisting}[style=XML]
<Simulation>
 ...
  <Models>
    ...
    <PostProcessor name="chz" subType="TSACharacterizer">
      <pivotParameter>pivot</pivotParameter>
      <fourier target='signal_f, signal_fa'>
        <periods>2, 5, 10</periods>
      </fourier>
      <arma target="signal_a, signal_fa" seed='42'>
        <SignalLag>2</SignalLag>
        <NoiseLag>3</NoiseLag>
      </arma>
    </PostProcessor>
    ...
  </Models>
 ...
</Simulation>
\end{lstlisting}


\subsection{Modeling}

\begin{frame}
  \frametitle{Model-Checking - Modeling}

  In MC we model the behavior of a system with the
  notion of {\bf state}. A state is a configuration of
  the system at a particular time instant
  \vfill
  The system can change state by means of a {\bf transition}
  \vfill
  We are interested in a {\bf property} of the system
  \vfill
  \pause
  Example:
  \begin{itemize}
    \item System: a washing machine
    \item A state: ``the door is open and the engine is off''
    \item A transition: ``if the door is open then close the door''
    \item A property: ``When the engine is on, the door is closed''
  \end{itemize}

\end{frame}

\begin{frame}
  \frametitle{System to Model}

  \begin{center}
  \includegraphics[scale=.67]{imgs/cheap-washing-machine-edit.png}
  \end{center}

\end{frame}

\begin{frame}
  \frametitle{Modeling - States}

  State variables can be used to describe a particular state
  \vfill
  \begin{center}
  \begin{tabular}{ll}
    \hline
    State variable & Values \\
    \hline
    door           & open, closed \\
    tray           & empty, filled \\
    engine         & off, on \\
    \hline
  \end{tabular}
  \end{center}
  \vfill\pause
  E.g.:
  \begin{center}
  \input{ex_1}
  \end{center}
  which stands for ``the door is open, the engine is on, and the tray is empty''.\pause
  How many different states can we describe with our state variables ?

\end{frame}

\begin{frame}
  \frametitle{Modeling - States}

  \begin{center}
  \input{model_1}
  \end{center}

  Some states are called {\bf initial} (green). Initial states
  are the configurations of the system at time $0$

\end{frame}

\begin{frame}
  \frametitle{Modeling - Transitions}

  Transitions describe the evolution of the system. They transform
  the ``current'' state into a ``next'' state
  \vfill\pause
  \begin{center}
  \begin{tabular}{llllcl}
    \hline
    \multicolumn{4}{l}{Transition}   & ~~~ & Name \\
    \hline
    if & door=open  & then & door'=closed & & [close\_door] \\
    \\
    if & tray=empty & then & tray'=full   & & [fill\_tray] \\
    \\
    \multirow{2}{*}{if} & engine=off  & \multirow{2}{*}{then} & engine'=on  & & \multirow{2}{*}{[start\_wash]} \\
                        & door=closed &                       & tray'=empty & &                              \\
    \\
    \multirow{2}{*}{if} & \multirow{2}{*}{door=closed} & \multirow{2}{*}{then} & door'=open  & & \multirow{2}{*}{[open\_door]} \\
                        &                              &                       & engine'=off & &                             \\
    \hline
  \end{tabular}
  \end{center}
  \vfill
  var' indicates the value of var in the next state

\end{frame}

\begin{frame}
  \frametitle{Modeling - Transitions}

  \scriptsize

  \begin{tabular}{llllcl}
    if & door=open  & then & door'=closed & ~~~ & [close\_door]
  \end{tabular}

  \begin{center}
  \scalebox{.85}{\input{model_2_3}}
  \end{center}
  \pause

  \begin{tabular}{llllcl}
    if & tray=empty  & then & tray'=full & ~~~ & [fill\_tray]
  \end{tabular}

  \begin{center}
  \scalebox{.85}{\input{model_2_1}}
  \end{center}
  \pause

  \begin{tabular}{llllcl}
    \multirow{2}{*}{if} & \multirow{2}{*}{door=closed} & \multirow{2}{*}{then} & door'=open  & & \multirow{2}{*}{[open\_door]} \\
                        &                              &                       & engine'=off & &                             \\
  \end{tabular}

  \begin{center}
  \scalebox{.85}{\input{model_2_2}}
  \end{center}
  \pause

  \begin{tabular}{llllcl}
    \multirow{2}{*}{if} & door=closed & \multirow{2}{*}{then} & tray'=empty & & \multirow{2}{*}{[start\_washing]} \\
                        & engine=off  &                       & engine'=on  & &                                   \\
  \end{tabular}

  \begin{center}
  \scalebox{.85}{\input{model_2_4}}
  \end{center}

\end{frame}

\begin{frame}
  \frametitle{Modeling - (Safety) Property}

  Last step, we need to model the property
  \begin{center}
    ``when the engine is on the door is closed''
  \end{center}
  It is a {\bf safety} property: they are easy
  to define as they are properties of the states
  \vfill\pause
  We call {\bf bad state} (or unsafe state) a state
  that does not satisfy the property
  \vfill
  \begin{center}
  \input{model_3}
  \end{center}

\end{frame}


\subsection{Fuel salt reprocessing system}

\begin{frame}
  \frametitle{Fuel salt reprocessing system overview: gas separation}
  Gaseous fission products (e.g., Xe, Kr) must be removed from the fuel salt
  to avoid reactor
poisoning.

      \begin{columns}
      	\column[t]{4.7cm}
    \begin{block}{Noble gas removal}
      \begin{enumerate}
      	\item bubble generator injects He bubbles in the salt stream
      	\item noble gas migrate to the He bubbles
      	\item gas separator discharges the poison-rich bubbles
      \end{enumerate}
    \end{block}

     	\column[t]{8cm}
  \begin{figure}[t]
	  \centering
	  		\vspace{-8mm}
		\includegraphics[width=\textwidth]{../figures/gas_separation.pdf}
	\caption{Schematic flow diagram of the \gls{MSBR} gas separation system
	(figure reproduced from Robertson \emph{et al.}
	\cite{robertson_conceptual_1971}).}
    \end{figure}

	\end{columns}
\end{frame}

\begin{frame}
  \frametitle{Mathematical model for gas separation efficiency}
  		\vspace{-1mm}
Xenon removal efficiency ($\epsilon_{Xe}$) in a gas separation system is
\cite{peebles_removal_1968}:
\begin{align}
& \qquad\qquad \epsilon_{Xe} = \frac{1-e^{-\beta}}{1+\alpha} \nonumber \\
\alpha &= \frac{RTQ_{L}}{HQ_{G}} \nonumber \\
\beta &= \frac{K_L a A_C L (1+\alpha)}{Q_{L}} \nonumber \\
Q_{L}&= \mbox{volumetric salt flow rate} \nonumber \\
Q_{G}&= \mbox{volumetric helium flow rate} \nonumber \\
H &= \mbox{Henry's law constant} \nonumber \\
a &= \mbox{gas-liquid interfacial area} \nonumber \\
K_L &= \mbox{liquid phase mass transfer coefficient.} \nonumber
\end{align}
		\vspace{-5mm}
  \begin{figure}[t]
	\includegraphics[width=0.66\textwidth]{./images/pipeline_contactor.png}
	\vspace{-2mm}
	\caption{Flow diagram for gas separator (figure reproduced from Peebles
		\emph{et al.} \cite{peebles_removal_1968}).}
\end{figure}

\end{frame}

\begin{frame}
\frametitle{Fuel processing system overview: rare earths and Pa removal}
	\begin{figure}[htp!] % replace 't' with 'b' to
		\centering
			\includegraphics[width=0.57\textwidth]{../figures/flowsheet.pdf}
			\vspace{-2mm}
		\caption{Liquid metal (Bi) extraction system for the \gls{MSBR}
		(reproduced from Sorensen \cite{sorensen_one-fluid_2006}).}
	\end{figure}

\end{frame}

\begin{frame}
\frametitle{Fuel processing system overview: TAP concept}
	\vspace{-2mm}
\begin{figure}[htp!] % replace 't' with 'b' to
	\centering
	\includegraphics[width=0.75\textwidth]{../figures/tap_primary_loop.png}
	\caption{Simplified \gls{TAP} primary loop design including off-gas system
		(blue), nickel filter (orange) and liquid metal extraction system
		(green) \cite{transatomic_power_transatomic_2019}.}
\end{figure}

\end{frame}

\begin{frame}
\frametitle{SaltProc demonstration for TAP concept input data}
	  \begin{textblock*}{12.5cm}(0.5cm,1.5cm) % {block width} (coords)
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{table}[htbp!]
	\fontsize{6}{9}\selectfont
	\centering
	\caption{The effective cycle times for fission products removal  from the
		\gls{TAP} reactor \cite{betzler_implementation_2017}.}
	\begin{tabular}{p{0.14\textwidth} p{0.3\textwidth} p{0.11\textwidth}
			p{0.11\textwidth}}
		\hline
		\textbf{Processing group} & \qquad\qquad\qquad \textbf{Nuclides} &
		\textbf{Removal Rate (s$^{-1}$)} & \textbf{Cycle time (at full power)}
		\\ \hline
		\multicolumn{3}{c}{\textit{Elements removed in \gls{MSBR} concept and
				adopted for the \gls{TAP}} \cite{robertson_conceptual_1971}} \\
		Noble gases & Xe, Kr								  & 5.00E-2 & 20
		sec \\
		Noble metals & Se, Nb, Mo, Tc, Ru, Rh, Pd, Ag, Sb, Te & 5.00E-2 & 20
		sec \\
		Seminoble metals & Zr, Cd, In, Sn	  				  & 5.79E-8 & 200
		days\\
		Volatile fluorides & Br, I 							  & 1.93E-7 & 60
		days\\
		Rare earths & Y, La, Ce, Pr, Nd, Pm, Sm, Gd           & 2.31E-7 & 50
		days\\
		\qquad & Eu & 2.32E-8 & 500 days \\
		Discard & Rb, Sr, Cs, Ba & 3.37E-9 & 3435 days \\
		\hline
		\multicolumn{3}{c}{\textit{Additional elements removed}
			\cite{betzler_implementation_2017,
			transatomic_power_corporation_neutronics_2016}} \\
		Noble gases & H								  	& 5.00E-2 & 20
		sec    \\
		Noble metals & Ti, V, Cr, Cu						& 3.37E-9 & 3435
		days \\
		Seminoble metals & Mn, Fe, Co, Ni, Zn, Ga, Ge, As   & 3.37E-9 & 3435
		days \\
		Rare earths & Sc									& 3.37E-9 & 3435
		days \\
		Discard & Ca										& 3.37E-9 & 3435
		days \\
		\hline
	\end{tabular}
	\label{tab:reprocessing_list}
\end{table}
	\begin{itemize}
		\item Noble gas removal efficiency: variable, described using
		mathematical
		correlation
		\item Other FP removal efficiency: fixed, non-ideal, based on
		Table~\ref{tab:reprocessing_list}
	\end{itemize}
	\end{textblock*}
\end{frame}

\subsection{Proposed tool design}

\begin{frame}
\frametitle{SaltProc class architecture}
	\begin{itemize}
		\item \textit{Simulation} class
			\begin{itemize}
				\item Manages simulation process
				\item Stores data into the HDF5 database
				\item Tracks time
			\end{itemize}
		\item \textit{Depcode} class
			\begin{itemize}
				\item Contains attributes and methods for reading user's input
				\item Creates input files for depletion code
				\item Parses depletion code output
			\end{itemize}
		\item \textit{Process} class
			\begin{itemize}
				\item Represents fuel processing system component
				\item Contains attributes of the component ($\epsilon_e$, throughput rate, capacity)
				\item Tracks waste stream
			\end{itemize}
		\item \textit{MaterialFlow} class
			\begin{itemize}
				\item Instances of that class represents the material flowing between processes
			\end{itemize}
	\end{itemize}
		\vspace{3mm}
	\begin{figure}[ht!] % replace 't' with 'b' to
		\centering
		\includegraphics[width=0.6\textwidth]{../figures/materialflow.pdf}
		\vspace{-0.1in}
		\caption{Schematic for passing material data between fuel processing
		system components.}
	\end{figure}

\end{frame}

\begin{frame}
\frametitle{SaltProc flowchart}
\vspace{-2mm}
\begin{figure}[ht!] % replace 't' with 'b' to \centering
	\centering
	\includegraphics[width=1.05\textwidth]{./images/saltproc_flowchart.pdf}
	\caption{Tentative generic flowchart for SaltProc v1.0 python package.}
\end{figure}

\end{frame}