\subsection{A witness-oriented approach}

\begin{frame}[fragile]
\frametitle{A witness-oriented approach}

Subset-mode often works in practice, but does not seem ideal
\begin{itemize}
\item Strange normal form that affects all booleans
\item Strange iff-rewrites needed for all UBDD-making functions
\item Free variables in transitivity and the preservation of membership
\item Rules about \Code{q-ite} seem somehow fragile
\end{itemize}

\SmallSkip
\Highlight{Witness-mode} is a more advanced alternative
\begin{itemize}
\item Intuitively, ``Pick all of the probably-relevant points''
\item Casts everything in terms of \Code{eval-bdd}
\item Works with existing normal forms
\end{itemize}
\end{frame}

\begin{frame}[fragile]
\frametitle{The witness approach, graphically}

\includegraphics[width=12cm]{venn-diagrams}

\end{frame}

\begin{frame}[fragile]
\frametitle{The basic transformation}

Hypothesis: \Code{x} $\neq$ \Code{y}  (or \Code{x})
\begin{itemize}
\item Means $\exists$ \Code{v} : \Code{(eval-bdd x v)} $\neq$ \Code{(eval-bdd y v)}
\item Introduce a new variable, \Code{v}
\item Replace the hyp with \Code{(eval-bdd x v)} $\neq$ \Code{(eval-bdd y v)}
\end{itemize}

\SmallSkip

Hypothesis: \Code{x} $=$ \Code{y}   (or \Code{(not x)})
\begin{itemize}
\item Means $\forall$ \Code{v} : \Code{(eval-bdd x v)} $=$ \Code{(eval-bdd y v)}
\item Collect all \Code{v} occurring in the clause
\item Replace the hyp with \Code{(eval-bdd x v)} $=$ \Code{(eval-bdd y v)}
\end{itemize}

\end{frame}

\begin{frame}[fragile]
\frametitle{Transformation example}

\begin{verbatim}
  (IMPLIES (AND ;; (NORMP C)
                ;; (NORMP HYP)
                (Q-ITE C HYP NIL)
                (NOT (EQUAL (Q-ITE C HYP NIL) HYP))
                HYP
                (NOT (EQUAL C T))
                (NOT (Q-ITE C NIL HYP))
                (NOT (EVAL-BDD C ARBITRARY-VALUES)))
           (NOT (EVAL-BDD HYP ARBITRARY-VALUES))))
\end{verbatim}

\end{frame}

\begin{frame}[fragile]

\begin{verbatim}
  (IMPLIES (AND (NOT (EQUAL (EVAL-BDD (Q-ITE C HYP NIL) V1)
                            (EVAL-BDD NIL V1)))
                (NOT (EQUAL (EVAL-BDD (Q-ITE C HYP NIL) V2)
                            (EVAL-BDD HYP V2)))
                (NOT (EQUAL (EVAL-BDD HYP V3)
                            (EVAL-BDD NIL V3)))
                (NOT (EQUAL (EVAL-BDD C V4)
                            (EVAL-BDD T V4)))
                (NOT (Q-ITE C NIL HYP))
                (NOT (EVAL-BDD C ARBITRARY-VALUES)))
           (NOT (EVAL-BDD HYP ARBITRARY-VALUES))))
\end{verbatim}

Values: \Code{V1}, \Code{V2}, \Code{V3}, \Code{V4}, \Code{ARBITRARY-VALUES}

\end{frame}

\begin{frame}[fragile]

{\footnotesize \begin{verbatim}
  (IMPLIES (AND (NOT (EQUAL (EVAL-BDD (Q-ITE C HYP NIL) V1)
                            (EVAL-BDD NIL V1)))
                (NOT (EQUAL (EVAL-BDD (Q-ITE C HYP NIL) V2)
                            (EVAL-BDD HYP V2)))
                (NOT (EQUAL (EVAL-BDD HYP V3)
                            (EVAL-BDD NIL V3)))
                (NOT (EQUAL (EVAL-BDD C V4)
                            (EVAL-BDD T V4)))

                (EQUAL (EVAL-BDD (Q-ITE C NIL HYP) V1)
                       (EVAL-BDD NIL V1))
                (EQUAL (EVAL-BDD (Q-ITE C NIL HYP) V2)
                       (EVAL-BDD NIL V2))
                (EQUAL (EVAL-BDD (Q-ITE C NIL HYP) V3)
                       (EVAL-BDD NIL V3))
                (EQUAL (EVAL-BDD (Q-ITE C NIL HYP) V4)
                       (EVAL-BDD NIL V4))
                (EQUAL (EVAL-BDD (Q-ITE C NIL HYP) ARBITRARY-VALUES)
                       (EVAL-BDD NIL ARBITRARY-VALUES))

                (NOT (EVAL-BDD C ARBITRARY-VALUES)))
           (NOT (EVAL-BDD HYP ARBITRARY-VALUES))))
\end{verbatim}}

\end{frame}

\begin{frame}[fragile]

{\small \begin{verbatim}
  (IMPLIES (AND (EVAL-BDD (Q-ITE C HYP NIL) V1)
                (NOT (EQUAL (EVAL-BDD (Q-ITE C HYP NIL) V2)
                            (EVAL-BDD HYP V2)))
                (EVAL-BDD HYP V3)
                (NOT (EVAL-BDD C V4))

                (NOT (EVAL-BDD (Q-ITE C NIL HYP) V1))
                (NOT (EVAL-BDD (Q-ITE C NIL HYP) V2))
                (NOT (EVAL-BDD (Q-ITE C NIL HYP) V3))
                (NOT (EVAL-BDD (Q-ITE C NIL HYP) V4))
                (NOT (EVAL-BDD (Q-ITE C NIL HYP) ARBITRARY-VALUES))

                (NOT (EVAL-BDD C ARBITRARY-VALUES)))
           (NOT (EVAL-BDD HYP ARBITRARY-VALUES))))
\end{verbatim}}

Follows from cases introduced by \Code{eval-bdd-of-q-ite}

\end{frame}

\begin{frame}[fragile]
\frametitle{The {\tt eval-bdd-cp} clause processor (1/2)}

\Code{(diff x y)}
\begin{itemize}
\item When \Code{x} $\neq$ \Code{y},
       \Code{(eval-bdd x (diff x y))} $\neq$ \Code{(eval-bdd y (diff x y))}
\end{itemize}

\SmallSkip
{\bf 1a.}.  Gather hyps of the form \Code{x} $\neq$ \Code{y}, where
\Code{x}, \Code{y} are (likely) UBDDs
\begin{itemize}
\item A hyp which is just \Code{x} also counts: \Code{x} $\neq$ \Code{NIL}
\end{itemize}

\SmallSkip
{\bf 1b.}.  For each \Code{x} $\neq$ \Code{y} found, replace the hyp with
\begin{center}
\Code{(implies (and (normp x) (normp y))) \qquad \qquad \qquad \qquad \qquad \qquad \quad \;} \\
\qquad \Code{(eval-bdd x (diff x y))} $\neq$ \Code{(eval-bdd y (diff x y))}
\end{center}

This is sound
\begin{itemize}
\item In the \Code{normp} case, the clauses are equivalent
\item Otherwise, the new clause implies the original
\end{itemize}

\end{frame}

\begin{frame}[fragile]
\frametitle{The {\tt eval-bdd-cp} clause processor (2/2)}

{\bf 2.} As a convenience, generalize away all \Code{(diff x y)} terms just
introduced with fresh variables. (trivially sound)

\SmallSkip
{\bf 3.} Gather up all \Code{v} which are used, anywhere, as arguments to
\Code{eval-bdd}, i.e., \Code{(eval-bdd x v)}.

\SmallSkip
{\bf 4a.} Gather hyps of the form \Code{x} $=$ \Code{y} found, where \Code{x},
\Code{y} are (likely) UBDDs
\begin{itemize}
\item A hyp which is \Code{(not x)} also counts: \Code{x} $=$ \Code{NIL}
\end{itemize}

\SmallSkip
{\bf 4b.} Replace these hyps with \Code{(eval-bdd x v)} $=$ \Code{(eval-bdd y v)},
for all \Code{v} found in step 3. (trivially sound)

\end{frame}

\begin{frame}[fragile]
\frametitle{Automating {\tt eval-bdd-cp}}

We use a default hint

\begin{itemize}
\item The clause must be \Code{stable-under-simplificationp}
\item The definition of \Code{eval-bdd-cp-hint} must be enabled
\item The transformation must modify the clause
\end{itemize}

\SmallSkip
The hint we give
\begin{verbatim}
(:or (:clause-processor ...)
     (:no-op t))
\end{verbatim}

\end{frame}


\section{Introduction} \label{intro}

% Big picture.

Economics is thinking in models that abstract from the world by imposing assumptions on economic relations. These assumptions are defendable as long as they are sufficiently realistic for the research question at hand \citep{F53}, prompting the question what this means. Often, assumptions are decided upon by mathematical convenience: do the assumptions grant that the model is solvable, i.e. whether a solution exists in closed form and whether this solution is unique.

% Economics is thinking in models that abstract from the actually existing economic systems by imposing assumptions on economic relations. Those assumptions are defendable as long as they are sufficiently realistic for the research question at hand \citep{F66}, what prompts the question, what sufficiently realistic means. Often, assumptions are decided upon by mathematical convenience: do the assumptions grant that the model is solvable, i.e. whether a solution exists in closed form and whether this solution is unique.

% Narrow it down.

With the advent of numerical methods, we are less coerced to impose insufficiently realistic assumptions. Even if the model at hand does not have an analytical solution, one can still obtain an approximate solution by applying numerical methods \citep{MF04}.

% Due to the advancements in computational tractability we are less coerced to impose insufficiently realistic assumptions if numerical methods can be applied. That is, since even if the model at hand does not have an analytical solution, one can still obtain an approximate solution by applying numerical methods \citep{MF04}.

Yet, if predictions shall be derived from a model in which the outcome is modelled as a function of inputs, it is crucial to understand how sensitive the model’s output is to changes in its inputs. Sensitivity analysis plays a major role in structural econometrics where the estimated model is oftentimes used to investigate counterfactual policies by quantifying the effects of a policy on some output \citep{LM17}. If these models are actually used for policy making, the structural model and its assumptions have real-world consequences. The question arises whether the imposed model assumptions are sufficiently realistic to inform policy. Thus, thorough modelling is crucial for structural econometrics.

% Now explain sensitivity analysis?

Consider the following model
\begin{equation*}
Y = f(X),
\end{equation*}

\noindent where $Y$ is the response variable that depends on the values of some independent variables $X$. Let $X$ denote the vector $X = (X_1, \dots, X_k)' \in \mathbb{R}^k$, where $k$ is the number of independent variables. Let $x$ be the vector of specific values assigned to $X$, i.e. one realisation of $X$. Let $f(\cdot)$ denote a function that describes in which way $Y$ depends on $X$. This function may be some complex function (e.g. computer code) or a model that can be solved by numerical methods only. Thus, $f$ is generally not available in closed form and, hence, the relationship between $Y$ and $X$ is considered a black box.

In the remainder of this work, I name $Y$ the (model) output and the vector $X$ the (model) inputs. In this thesis, I will consider $X$ as being stochastic, that is, the $X_i$, $i = 1,\dots, k$, follow a joint cumulative distribution function $G(X)$. Thus, although $f$ is assumed to be a deterministic function of $X$, the output $Y$ is also stochastic due to the uncertainty in $X$ \citep{SNS16}. Sensitivity analysis sheds light on this input-output relationship \citep{BP16}.

Any assumption we impose can be such an input. One common objective of sensitivity analysis is Factor Fixing (FF), where we want to know whether an input can be fixed without influencing the output. For example, if we assume an estimated parameter to be deterministic by assigning a value to it, we implicitly neglect that this estimate is subject to uncertainty \citep{R21}. This assumption is troubling if the model output is very sensitive to the value assigned to this input. Such an input we call \textit{important}. Sensitivity analysis can guide research by identifying these important inputs \citep{R21}. For example, we could focus on estimating important inputs more precisely, thus reducing output uncertainty.

Generally, sensitivity analysis can be structured into local and global methods. Local methods conduct sensitivity analysis around a certain point, or base case, $x_0$, in a deterministic framework, i.e. no probability distribution is assigned to $X$ \citep{BP16}. In contrast, when performing global sensitivity analysis, we find ourselves in a stochastic context, which requires knowledge of the distribution of $X$, be it joint or marginal, with or without dependence between the inputs \citep{ST02}. The result of performing sensitivity analysis is some sensitivity measure, that depends on the sensitivity method we apply to the context we find ourselves in. The methods applied in this thesis yield global sensitivity measures.

% Research Gap.

One popular method to conduct global sensitivity analysis are Shapley effects. They are a well-suited sensitivity method that is easily interpreted and can be applied to both independently and dependently distributed inputs. Despite these and other appealing features, estimation of Shapley effects can be computationally demanding. This is where the Morris method, a qualitative sensitivity method first introduced by \citet{M91}, comes into play since it promises to serve similar purposes as the Shapley effects, but to come at a lower computational cost.

% Short summary of your paper and main results.

This paper assesses the predictive validity of the Morris method by applying it to a classical structural econometric model, the single-agent dynamic stochastic model of discrete choice introduced by \citet{R87}. The Morris method employs a one-at-a-time algorithm to assess input importance: it uses the relative change in the model output due to a change in a model input. If inputs are dependent, the original Morris method can be subject to Type II error, i.e. it can identify inputs as uninfluential even if they are not, since it does not account for influence due to dependence among inputs. I compare the performance of the Morris method for \textit{dependent} inputs as proposed by \citet{GM17} to Shapley effects, which serve as the benchmark. I compare them based on the input ranking they imply.

I find that the Morris method is a substitute for Shapley effects only if the resulting indices are interpreted with caution. Morris indices are very volatile. % since they perform very well in identifying uninfluential inputs and in ranking the inputs accordingly for the Rust model.
My results suggest further that the extension of Morris indices for dependent inputs by \citet{GM17} is an important contribution since their proposed measure reduces the Type II error in identifying uninfluential inputs.

Although the \textit{extended} Morris method has been applied in some cases \citep[e.g.][]{MMA18, RZY19}, to my knowledge their predictive validity has been assessed in the original paper by \citet{GM17} only. In economics, structured global sensitivity analysis is not a common practice: \citet{HMSW19} cite only few articles in economics that applied global sensitivity analysis. The contribution of this thesis is that it assesses the extended Morris method by applying it to an economic model.

% In the context of structural econometrics sensitivity analysis has an important role. Structural econometrics imposes structure on the relationship between economic parameters \citep{LM17}. The structural model is then taken to the data and deep economic parameters are estimated \citep{LM17}. Structural econometrics can be used to investigate counterfactual policies by quantifying the effects of a policy on some output \citep{LM17}. If these models are actually used for policy making, the structural model and its assumptions have real-world consequences. The question arises whether the imposed model assumptions are realistic enough to inform policy. Therefore, thorough modelling is crucial for structural econometrics.

% Sensitivity analysis analyses how sensitive the model outcome is to model inputs \citep{R21}.

% In this work, I assess the predictive validity of Morris indices, a qualitative sensitivity method introduced by \citet{M91}. Qualitative sensitivity methods aim at identifying uninfluential inputs and ranking inputs with respect to their importance \citep{BP16}. The Morris method employs a one-at-a-time algorithm to assess input importance: it uses the relative change in the model output due to a change in a model input \citep{M91}. I compare the performance of the Morris method for dependent inputs as proposed by \citet{GM17} to Shapley effects, a variance-based quantitative sensitivity method \citep{O14}.

% Variance-based sensitivity analysis, a method that evaluates the importance of an input by assigning to it the expected reduction in output variance if this input was known with certainty \citep{BP16}. Shapley values are a concept from game theory introduced by \citet{S53}. The main advantages of Shapley values for sensitivity analysis are that they are as easily applied to dependent inputs as they are interpreted. Furthermore, Shapley values satisfy a range of desirable properties, the most important being efficiency and the null-player property \citet{S53, O14}. In the context of sensitivity analysis, \citet{SNS16} use the term Shapley effects. Despite of these appealing features, estimation of Shapley effects can be computationally demanding \citep{SNS16}.

%  I consider the \textit{extended} Morris method introduced by \citet{GM17} that handles correlated inputs.

% Say something about the literature.

% In this paper I estimate Morris indices for  and compare these to Shapley effects. Interestingly, I find that the Morris method is a worthy substitute for Shapley effects since they perform very well in identifying uninfluential inputs and in ranking the inputs accordingly for the Rust model. I find that the extension of Morris indices for dependent inputs by \citet{GM17} is an important contribution since their proposed measure reduces the Type-II error in identifying influential inputs.

% One set of common variance-based sensitivity measure are the total and first-order effects, or Sobol' indices, introduced by \citet{S93} for independently distributed input variables, which are based on an Analysis of Variances (henceforth ANOVA) decomposition. One obvious drawback of the Sobol' indices in the original definition is that they rely on the assumption of inputs independence. Even if the ANOVA decomposition is adapted to handle dependent inputs, they suffer from problems \citep{OP17} which are discussed in \cref{var_based_sa}. \citet{OP17} suggest using Shapley values instead, which were suggested in the context

% Overview over thesis.

The remainder of this thesis is organised as follows: \cref{rust_model} introduces the Rust model for which I conduct sensitivity analysis. In \cref{comp_shap}, I estimate Shapley effects as a benchmark for the assessment. \Cref{comp_morris} gives details on how the extended Morris method is used for sensitivity analysis, while \cref{comparison} discusses the results. \Cref{conclusion} offers some concluding remarks.


\documentclass[./\jobname.tex]{subfiles}
\begin{document}
\chapter {Experiment 0: Serial Memetic JADE}
\label{chap:experimet_0}

This chapter describes the results obtained by the most basic adaption of JADE for solving \gls{pde}s. The algorithm used here builds on the concepts described in \cite{chaquet_using_2019}. Aside from substituting some parameters, the only main difference is the usage of JADE instead of a \gls{cma_es}.

\section{Hypotheses}
A memetic algorithm was first mentioned by \cite{moscato_evolution_2000}. In essence, it is a hybridisation of a population-based evolutionary algorithm and a deterministic direct or local search. The pseudocode \ref{algo: memeticJADE} below shows the implementation of such a memetic JADE. At first, JADE performs the global search and places its population around the optimum. Then the Downhill Simplex (\gls{ds}) exploits this area. The budget of \gls{nfe} is split into two parts: JADE takes nearly all \gls{nfe} and leaves $2\cdot dim$ \gls{nfe} for the \gls{ds}. The SciPy implementation of the \gls{ds} is used (\cite{scipy_scipyoptimizefmin_2020}).

\begin{algorithm}[h]
	\SetAlgoNoLine
	\DontPrintSemicolon
	\SetKwFunction{FmJADE}{memeticJADE}
	\SetKwProg{Fn}{Function}{:}{}
	\Fn{\FmJADE{$\mathbf{X}$, $funct$, $minErr$, $maxFE$}}{
		$dim$, $popsize$ $\gets size(\mathbf{X})$\;
		$p \gets 0.3$\;
		$c \gets 0.5$\;
		$pop$, $FE$, $F$, $CR$ $\gets JADE($$\mathbf{X}$, $p$, $c$, $funct$, $minErr$, $maxFE - 2 dim$ $)$\;
		$bestIndex = argmin(FE)$\;
		$bestSol = pop[bestIndex]$\;
		$pop$, $FE$ $ = downhill\text{ }simplex($$funct$, $bestSol$, $minErr$, $2 dim)$\;
		\Return $pop$, $FE$, $F$, $CR$
	}
	\unterschrift{Pseudocode of memetic JADE}{}{}
	\label{algo: memeticJADE}
\end{algorithm}

This experiment provides a first insight into the performance of the proposed algorithm. It tries to answer the question if JADE is a suitable surrogate algorithm for a \gls{cma_es}. Further, the memory usage and solving time is compared to the \gls{fem} results obtained in chapter \ref{chap:fem_baseline_results}.

\section{Experiment Setup}
The standard parameters from table \ref{tab:ci_parameter} are taken. The memetic JADE is limited to either $10^4$ \gls{nfe} or $10^6$ \gls{nfe}. The experiment is done on two different machines. The first try with $10^4$ \gls{nfe} is run on the same machine ($\rightarrow$ machine 1) as the \gls{fem} experiment. This allows a fair memory and solving time comparison. This comparison can not be performed with the data obtained by using $10^6$ \gls{nfe} ($\rightarrow$ machine 2). Further, 5 \gls{gak} are used which results in a dimension of 20 parameters. Thus, the population consists of 40 individuals.

To validate the results, a statistical significance test is performed. The test function is based on the Wilcoxon test. The Wilcoxon test checks for significance at $\alpha = 0.05$. To decide if the results are better or worse, the mean and the median are compared. If both are smaller, the results are better. Contrary, if both are larger, the results are worse. If only one of the values is smaller, the results are undecided. The implementation is further explained in appendix \ref{chap:apendix_post_proc}.

\section{Results}
\label{chap:experimet_0_results}
Table \ref{tab:results_literature_comparison} compares the obtained results with results from previous research papers. Therefore, the common testbed functions \gls{pde} 2 and \gls{pde} 3 are used. It is important to notice that the parameters used to obtain the results might not coincide. \cite{tsoulos_solving_2006} also use these two \gls{pde}s, but their paper did not provide any usable error metric.
\begin{table}[H]
	\centering
	\noindent\adjustbox{max width=\linewidth}{
		\begin{tabular}{|c|c|c|c|}

			\hline
			\rowcolor[HTML]{\farbeTabA}

			Paper & Parameter & RMSE \gls{pde} 2 & RMSE \gls{pde} 3 \\ \hline
			\cite{chaquet_using_2019} & \multilinecell{4 kernel \\ max \gls{nfe}=$10^6$ \\ 50 replications } & $(1.75 \pm 1.14) 10^{-4}$ & $(1.09 \pm 0.846) 10^{-5}$  \\ \hline
			\cite{chaquet_solving_2012} & \multilinecell{10 harmonics \\ max \gls{nfe} = $G \cdot \lambda$ = $1.2 \cdot 10^6$ \\ 10 replications} & $(6.37 \pm 0.733) 10^{-3}$ & $(5.90 \pm 0.799)10^{-3}$ \\ \hline
			\cite{sobester_genetic_2008}& \multilinecell{50 max tree length \\ 12 generations \\ 20 replications} & $(6.9 \pm 8.3)10^{-4}$ & N/A \\ \hline
			\cite{panagant_solving_2014}& \multilinecell{unknowns: N/A \\ \gls{nfe}=$5\cdot 10^5$ \\ replications: N/A} & $7.256 10^{-4}$ & $9.489 10^{-6}$ \\ \hline
			serial memetic JADE & \multilinecell{5 kernel \\ max \gls{nfe} = $10^6$ \\ 20 replications} & $(2.9798 \pm 1.5541)10^{-2}$ & $(3.8225 \pm 1.9438)10^{-2}$ \\ \hline

		\end{tabular}
	}
	\unterschrift{This table compares the results obtained with the serial memetic JADE to the numerical results obtained by similar work in literature. The same metric must be used, thus the RMSE as defined in equation \eqref{eq:rmse_chaquet} is calculated.}{}{}
	\label{tab:results_literature_comparison}
\end{table}

The following table \ref{tab:serial_jade_compare_10^6_10^4} lists the smallest L2 norm reached after $10^4$ \gls{nfe} and $10^6$ \gls{nfe}, respectively. A standard Wilcoxon test, as described in the appendix \ref{chap:apendix_post_proc}, is performed. Remarkable is that the results on \gls{pde} 5 get significantly worse when more \gls{nfe} are used.

\begin{table}[H]
	\centering
	\noindent\adjustbox{max width=\linewidth}{
		\begin{tabular}{|c|c|c|c|c|l|}

			\hline
			\rowcolor[HTML]{\farbeTabA}

			\gls{nfe} & \multicolumn{2}{|c|}{$10^4$} & \multicolumn{2}{|c|}{$10^6$} & \\ \hline
			stat & mean & median & mean & median & Wilcoxon Test \\ \hline \hline
			\gls{pde} 0A & 1.9415 $\pm$ 0.3321 & 1.8844 & 0.6596 $\pm$ 0.5510 & 0.9285 & sig. better \\ \hline
			\gls{pde} 0B & 0.7137 $\pm$ 0.1979 & 0.6354 & 0.2027 $\pm$ 0.1302 & 0.1516 & sig. better \\ \hline
			\gls{pde}  1 & 0.1874 $\pm$ 0.0408 & 0.1938 & 0.0149 $\pm$ 0.0049 & 0.0151 & sig. better \\ \hline
			\gls{pde}  2 & 0.0890 $\pm$ 0.0334 & 0.0760 & 0.0257 $\pm$ 0.0140 & 0.0224 & sig. better \\ \hline
			\gls{pde}  3 & 0.2409 $\pm$ 0.1051 & 0.2309 & 0.0328 $\pm$ 0.0169 & 0.0285 & sig. better \\ \hline
			\gls{pde}  4 & 0.1102 $\pm$ 0.0367 & 0.0985 & 0.0378 $\pm$ 0.0083 & 0.0352 & sig. better \\ \hline
			\gls{pde}  5 & 0.6645 $\pm$ 0.1930 & 0.6263 & 1.1968 $\pm$ 0.0286 & 1.2056 & sig. worse \\ \hline
			\gls{pde}  6 & 1.9660 $\pm$ 1.3845 & 1.6540 & 0.4135 $\pm$ 1.2133 & 0.0018 & sig. better \\ \hline
			\gls{pde}  7 & 0.0457 $\pm$ 0.0137 & 0.0452 & 0.0221 $\pm$ 0.0019 & 0.0223 & sig. better \\ \hline
			\gls{pde}  8 & 0.2186 $\pm$ 0.0045 & 0.2191 & 0.2170 $\pm$ 0.0019 & 0.2175 & unsig. better \\ \hline
			\gls{pde}  9 & 0.0525 $\pm$ 0.0147 & 0.0516 & 0.0451 $\pm$ 0.0119 & 0.0459 & unsig. better \\ \hline

		\end{tabular}
	}
	\unterschrift{L2 norm reached with serial JADE at $10^4$ \gls{nfe} and $10^6$ \gls{nfe}}{}{}
	\label{tab:serial_jade_compare_10^6_10^4}
\end{table}

The following two images \ref{fig:serial_jade_time_boxplot} and \ref{fig:serial_jade_memory_boxplot} show the time and memory usage for solving the testbed with $10^4$ \gls{nfe} in relation to the \gls{fem} solver. The images highlight the varying complexity of the testbed-\gls{pde}s and their corresponding fitness function. Although these results are not obtained for $10^6$ \gls{nfe}, they provide insight on how the solver scales with more \gls{nfe}.

\begin{figure}[H]
	\centering
	\noindent\adjustbox{max width=0.66\linewidth}{
		\includegraphics[width=\textwidth]{../../code/experiments/experiment_0/time_boxplot_ci_exp0.pdf}
	}
	\unterschrift{Relative solving time results of memetic JADE after $10^4$ \gls{nfe}.}{}{}
	\label{fig:serial_jade_time_boxplot}
\end{figure}

\begin{figure}[H]
	\centering
	\noindent\adjustbox{max width=0.66\linewidth}{
		\includegraphics[width=\textwidth]{../../code/experiments/experiment_0/mem_boxplot_ci_exp0.pdf}
	}
	\unterschrift{Relative memory usage results of memetic JADE after $10^4$ \gls{nfe}.}{}{}
	\label{fig:serial_jade_memory_boxplot}
\end{figure}

\section{Discussion}

The results presented in chapter \ref{chap:experimet_0_results} above are discussed on the following pages. A comparison to the current literature is drawn. The solving time and memory consumption in relation to the \gls{fem} solver NGSolve is examined. Especially interesting are the results obtained on \gls{pde} 0A and \gls{pde} 5.

\subsection{Comparison to Literature}

The \gls{rmse} reached on the \gls{pde} 2 and 3, are clearly not as good as the results obtained in previous papers, especially the results presented in \cite{chaquet_using_2019}. There might be a few reasons for that:
\begin{itemize}
	\item As stated in the previous table \ref{tab:ci_parameter}, the penalty and weighting factor on the collocation points are different. These settings have been chosen on the basis of preliminary experiments, but there is no guarantee that the choice of these parameters is optimal. Setting these values is not a trivial task and must be tailored to the optimisation algorithm and the differential equation.
	\item Since \gls{pde} 0A is defined as a combination of 5 \gls{gak}, at least that many kernels must be provided to the solver. This setup results in a greater search dimension, as compared in table \ref{tab:results_literature_comparison}. In general, more kernels result in a better solution, which is confirmed by experiments in \cite{chaquet_using_2019} and in chapter \ref{chap:pde 2 3 4 7}.
	To reduce the computational effort per generation, and thus allow JADE to adapt the internal parameters for a longer period, the population size is set to $2 \cdot dim$. According to \cite{mallipeddi_empirical_2008}, the population size should not be lower than that. Again, choosing these parameters is not necessarily simple. The combination of larger problem dimension and smaller population size could influence the convergence to the worse.
	\item Compared to \cite{chaquet_using_2019}, the \gls{nfe}-budget for the local \gls{ds} search is smaller. Thus, the algorithm puts more emphasis on the exploration. However, due to the internal parameter-adaption JADE can perform both - exploration and exploitation. Preliminary experiments have shown that the direct search converges fast with very little progress after more than 100 \gls{nfe}.
	\item JADE might simply be not as well suited for the problem as e.g. a \gls{cma_es}.
\end{itemize}

%%%%%%%%%%%%%%%%%%%%%%%%%
%        time/mem       %
%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Solving Time/Memory Usage}

The results from table \ref{tab:serial_jade_compare_10^6_10^4} show that the \gls{ci} solver can not nearly compete with the results obtained by the \gls{fem} solver from table \ref{tab:fem_sol_quality}.
However, more interesting is the comparison of time and memory usage.
In the current implementation, the population and the corresponding function values as well as the F and the CR history are recorded at every generation. Therefore, the memory usage scales linearly with the number of function evaluations used. This information is not necessary for the actual algorithm, but helpful for evaluating the results. In later implementations, this could be disregarded in order to even further reduce the memory consumption. Depending on the testbed \gls{pde}, only 1.5 to 4.0 percent of the memory used by the \gls{fem} solver is needed. The \gls{ci} solver uses less memory on all problems (figure \ref{fig:serial_jade_memory_boxplot}).

The \gls{ci} solver takes between 3 and 40 times as long as the \gls{fem} solver (figure \ref{fig:serial_jade_time_boxplot}) to perform $10^4$ \gls{nfe}. Since the solving time scales linearly, $10^6$ \gls{nfe} take about 10 times longer. An interesting observation is the distribution of the solving time within the testbed. The more operations a fitness function needs, the more expensive is one function evaluation and thus the longer is its solving time. However, this does not necessarily correspond with the quality of the solutions. For example, \gls{pde} 0B requires the longest absolute evaluation time ($29 \cdot 5 \text{ s} \approx 145\text{ s}$), compared to the other \gls{pde}s, it reaches a ``fairly'' good quality.

%%%%%%%%%%%%%%%%%%%%%%%%%
%         PDE0A         %
%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{PDE 0A}
\label{chap: experiment_0_pde_0A}

The purpose of this \gls{pde} is to show that the solver would converge globally towards the analytical solution, if it can be represented by a finite number of kernels. However, his can not be confirmed with the current implementation. While more function evaluations do tend to generate better results (as confirmed by the Wilcoxon test in table \ref{tab:serial_jade_compare_10^6_10^4}), it is common for the \gls{ci} solver to result in different functions. A typical phenomenon is that the obtained approximation has some of its Gauss ``bumps'' outside of the domain $\Omega$. The comparison of two solutions with $10^4$ \gls{nfe} and $10^6$ \gls{nfe} in figure \ref{fig:serial_jade_pde0a_sol_comparison} shows this behaviour. Since the results improve from $10^4$ to $10^6$, it is possible that the results from $10^6$ \gls{nfe} can be refined with an even larger \gls{nfe} budget. However, this is not tested due to the already extensive computational effort.

\begin{figure}[H]
	\centering
	\begin{subfigure}[b]{0.45\linewidth}
		\centering
		\includegraphics[width=1\textwidth]{../../code/experiments/experiment_0/pde0a_missing_bump_sol_10_4.pdf}
		\caption{\gls{pde} 0A error; $10^4$ \gls{nfe}: \\ L2 norm: 2.5075 \\ FE value: 4.4129 \\ RMSE: 0.5699}
		\label{fig:pde0a_sol_10_4}
	\end{subfigure}%
	%
	\begin{subfigure}[b]{0.45\linewidth}
		\centering
		\includegraphics[width=1\textwidth]{../../code/experiments/experiment_0/pde0a_missing_bump_sol_10_6.pdf}
		\caption{\gls{pde} 0A error; $10^6$ \gls{nfe}: \\ L2 norm: 0.9281 \\ FE value: 1.3762 \\ RMSE: 0.2111}
		\label{fig:pde0a_sol_10_6}
	\end{subfigure}%
	\unterschrift{Comparison of two typical \gls{pde} 0A solutions. }{}{}%
	\label{fig:serial_jade_pde0a_sol_comparison}
\end{figure}

\subsection{PDE 5}
\label{chap:ex0_pde5}
%%%%%%%%%%%%%%%%%%%%%%%%%
%         PDE5          %
%%%%%%%%%%%%%%%%%%%%%%%%%
A fascinating property of \gls{pde} 5 is observed: more function evaluations (from $10^4$ to $10^6$) result in a significantly worse solution quality. This is confirmed by a Wilcoxon test, as seen in table \ref{tab:serial_jade_compare_10^6_10^4}.
This property can even be concluded from a visual perspective. A comparison between the best solution after $10^4$ \gls{nfe} and the best solution after $10^6$ \gls{nfe} is shown in figure \ref{fig:serial_jade_pde5_sol_comparison}. The solution after $10^4$ \gls{nfe} describes the global structure more accurately. It seems that the correct description of the boundary points gets lost with more \gls{nfe}.

\begin{figure}[H]
	\centering
	\begin{subfigure}[b]{0.3333\linewidth}
		\centering
		\includegraphics[width=1\textwidth]{../../code/experiments/experiment_0/pde5_best_sol_10_4.pdf}
		\caption{best solution $10^4$ \gls{nfe} \\ L2 norm: 0.4283 \\ FE value: 2980.96}
		\label{fig:pde5_sol_10_4}
	\end{subfigure}%
	%
	\begin{subfigure}[b]{0.3333\linewidth}
		\centering
		\includegraphics[width=1\textwidth]{../../code/experiments/experiment_0/pde5_best_sol_10_6.pdf}
		\caption{best solution $10^6$ \gls{nfe} \\ L2 norm: 1.1095 \\ FE value: 1007.44}
		\label{fig:pde5_sol_10_6}
	\end{subfigure}%
	%
	\begin{subfigure}[b]{0.3333\linewidth}
		\centering
		\includegraphics[width=1\textwidth]{../../code/testbed/pde5/sol_pde_5.pdf}
		\caption{analytical solution \\to testbed \\ \gls{pde} 5.}
		\label{fig:pde5_analytical_solution}
	\end{subfigure}%
	\unterschrift{Comparison of best solution after $10^4$ and $10^6$ \gls{nfe}.}{}{}%
	\label{fig:serial_jade_pde5_sol_comparison}
\end{figure}

The distributions of the achieved fitness value after $10^4$ and $10^6$ \gls{nfe} is shown in figure \ref{fig:pde5_fitness_histogram}. They are clearly separated with distinct mean and median. As expected, both the mean and the median after $10^6$ \gls{nfe} are significantly smaller than these values after $10^4$ \gls{nfe}. The corresponding L2 norm distributions are plotted in figure \ref{fig:pde5_norm_histogram}. As described by the Wilcoxon test, the results are inverted. The mean and the median of the L2 norm after $10^4$ \gls{nfe} are smaller than the ones after $10^6$ \gls{nfe}.

\begin{figure}[H]
	\centering
	\begin{subfigure}[b]{0.5\linewidth}
		\centering
		\includegraphics[width=1\textwidth]{../../code/experiments/experiment_0/pde5_fit_histogram.pdf}
		\caption{Histogram of the fitness value reached on \gls{pde} 5.}
		\label{fig:pde5_fitness_histogram}
	\end{subfigure}%
	%
	\begin{subfigure}[b]{0.5\linewidth}
		\centering
		\includegraphics[width=1\textwidth]{../../code/experiments/experiment_0/pde5_norm_histogram.pdf}
		\caption{Histogram of the L2 norm reached on \gls{pde} 5.}
		\label{fig:pde5_norm_histogram}
	\end{subfigure}%
	\unterschrift{Histograms of the fitness and the L2 norm reached with $10^4$ \gls{nfe} and $10^6$ \gls{nfe} on \gls{pde} 5.}{}{}%
	\label{fig:pde5_histograms}
\end{figure}

This phenomenon can be explained by the structural difference between the fitness function and the L2 norm. The fitness of a candidate solution can only decrease or stay the same, due to the greedy selection used in JADE (line 15 in the pseudocode \ref{algo: jade}). The monotonically decreasing fitness value at every generation of an exemplary individual within the population is plotted in figure \ref{fig:ex0_pde5_gak_fit_vs_l2}. Because the L2 norm is not the property that gets optimised, the quality of an individual at every generation is not necessarily monotonically decreasing. The L2 norm and the fitness at every generation of one individual from the population is plotted in figure \ref{fig:ex0_pde5_gak_fit_vs_l2}.

\begin{figure}[H]
	\centering
	\noindent\adjustbox{max width=0.7\linewidth}{
		\includegraphics[width=\textwidth]{../../code/experiments/misc/pde5_gak_fit_l2_history.pdf}
	}
	\unterschrift{Fitness value and L2 Norm of one individual at every generation on \mbox{\gls{pde} 5.}}{}{}
	\label{fig:ex0_pde5_gak_fit_vs_l2}
\end{figure}

This indicates that the current fitness function does not fully describe the optimisation problem. Some possible solutions to this problem could be:
\begin{itemize}
	\item An adaptive number of kernels reduces the dimensionality of the problem and only increases the kernel number if necessary.
	\item A different kernel type that better resembles the structure of the solution could be used.
	\item A denser grid of collocation points alters the fitness function. This introduces more points that might sit at more interesting positions within the domain. However, this also increases the computational effort to evaluate the fitness function. Alternatively, a scheme that adapts the collocation points during the optimisation process could balance out the computational effort and interesting probing locations.
\end{itemize}

An adaptive kernel scheme is proposed in chapter \ref{chap:experimet_2}. The \gls{gsk} kernel type is examined in chapter \ref{chap:experimet_3}.

\end{document}


\section{Eigenvalues and eigenvectors of infinite-dimensional vectors}


\section{Z-test}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%                                                                 %
%   MINUIT User Guide -- LaTeX Source                             %
%                                                                 %
%   Chapter 3                                                     %
%                                                                 %
%   The following external EPS files are referenced:              %
%                                                                 %
%   Editor: Michel Goossens / CN-AS                               %
%   Last Mod.:  1 June 1992 10:10 mg                              %
%                                                                 %
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\chapter{How to Use Minuit}

\section{The Function FCN.}
The user must always supply a Fortran subroutine which calculates
the function value to be minimized or analyzed.
\medskip
\Shubr{FCN}{(NPAR,GRAD,FVAL,XVAL,IFLAG,FUTIL)}

\begin{DLtt}{1234567}
\item[{\rm\bf Input parameters}] \mbox{}
\item[NPAR]  number of currently variable parameters.
\item[XVAL]  vector of (constant and variable) parameters.
\item[IFLAG] Indicates what is to be calculated (see example below).
\item[FUTIL] Name of utilitary routine (if needed, it
             must be declared \Lit{EXTERNAL} and provided by the user).
\item[{\rm\bf Output parameters}] \mbox{}
\item[FVAL]  The calculated function value.
\item[GRAD]  The (optional) vector of first derivatives).
\end{DLtt}

Note that when Minuit is being used through an intermediate package such as
HBOOK or PAW, then the \Rind{FCN} may be supplied by the this package.

\begin{XMPt}{Example of \Lit{FCN} routine}
        SUBROUTINE FCN(NPAR,GRAD,FVAL,XVAL,IFLAG,FUTIL)
        IMPLICIT DOUBLE PRECISION (A-H,O-Z)  !  for 32-bit machines
        DIMENSION GRAD(*),XVAL(*)
        EXTERNAL FUTIL   !    (if needed and supplied by user)
C-
        IF (IFLAG .EQ. 1)  THEN
C           read input data,
C           calculate any necessary constants, etc.
        ENDIF
        IF (IFLAG .EQ. 2)  THEN
C           calculate GRAD, the first derivatives of FVAL
C           (this is optional)
        ENDIF
C             Always calculate the value of the function, FVAL,
C             which is usually a chisquare or log likelihood.
C                  Optionally, calculation of FVAL may involve
        FTHEO = FUTIL(....)
C                  It is responsability of user to pass
C                  any parameter values needed by FUTIL,
C                  either through arguments, or in a COMMON block
         IF (IFLAG .EQ. 3)  THEN
C            will come here only after the fit is finished.
C            Perform any final calculations, output fitted data, etc.
        ENDIF
        RETURN
        END
\end{XMPt}

The name of the subroutine may be chosen freely (in documentation we
give it the generic name \Rind{FCN}) and must be declared \Lit{EXTERNAL} in the
user's program which calls Minuit (in data-driven mode) or calls
Minuit subroutines (in Fortran-callable mode).
The meaning of the parameters \Lit{XVAL} is of course defined by
the user, who uses the values of those parameters to calculate his
function value.
The starting values must be specified by the user
(either by supplying parameter
definitions from a file, or typing them at the terminal,
in data-driven mode; or
by calling subroutine \Rind{MNPARM} in Fortran-callable mode),
and later values are determined by Minuit as it searches for the
minimum or performs whatever analysis is requested by the user.
\Rind{FUTIL} represents the name of a function or subroutine which may
be defined and supplied by the user and called from \Rind{FCN}.
If the user does not use the \Rind{FUTIL} feature, the last argument
may be given as zero, but if used, the name of \Rind{FUTIL} must
be declared \Lit{EXTERNAL} and a subprogram of that name must be
supplied at loading time.

It is possible, by giving them different names, to analyze several
different \Rind{FCN}s in one job.
However, one analysis must be completed before the next is started.
In order to avoid interference between the analyses of two different
\Rind{FCN}s, the user should call Minuit (in data-driven mode) or \Rind{MNINIT}
(in Fortran-callable mode) each time a new \Rind{FCN} is to be studied.

\section{Running Minuit in Data-driven Mode.}

Minuit can be run in two different modes:
{\bf Data-driven mode}
\index{data driven mode}
\index{mode!data driven}
means that the user drives Minuit with data, either typed
interactively from a terminal or from a data file in batch; and
\index{Fortran-callable mode}
\index{mode!Fortran-callable}
{\bf Fortran-callable mode}
means that Minuit is driven directly from Fortran subroutine
calls, without data.
To some extent, the two modes may also be mixed.
This section describes the first mode, and is valid for both
interactive and batch running.
The differences between interactive and batch are described in
a separate subsection below.

\index{data driven mode}
\index{mode!data driven}
In {\bf data-driven mode}, the user must supply,
in addition to the subroutine \Rind{FCN}, a
{\bf main program} which includes the following
statements (the statements in upper case are required, those
given in lower case are optional):

\begin{XMPt}{Example of main program when using Minuit in data driven mode}
      EXTERNAL FCN
      external futil
      call mintio(ird,iwr,isav)
      CALL MINUIT(FCN,futil)
\end{XMPt}

The name of \Rind{FCN} may be chosen freely, and is communicated
to Minuit as its first argument.
\Rind{FUTIL} is the generic name of a function or subroutine which the
user may optionally call from \Rind{FCN}, and if he does call such
a routine, he must declare it external and communicate its
name to Minuit as well.  If \Rind{FUTIL} is not used, then the second
argument may be put equal to \Lit{0},
and need not be declared \Lit{EXTERNAL}; if \Rind{FUTIL} is
declared \Lit{EXTERNAL}, it must be supplied in the loading process.

\newpage

\Shubr{MINTIO}{(IREAD,IWRITE,ISAVE)}

\medskip\Action
The purpose of \Rind{MINTIO} is to communicate to Minuit the I/O units.

\begin{DLtt}{1234567}
\item[{\rm\bf Input parameters}] \mbox{}
\item[IREAD]  Fortran unit number for reading (default 5).
\item[IWRITE] Fortran unit number for writing (default 6).
\item[Isave]  Fortran unit number for saving (default 7).
\end{DLtt}

\index{input/output units}

If the default values are acceptable, then it is not necessary to
call \Rind{MINTIO}.
It is the user's responsibility that the I/O units are properly
opened for the appropriate operations.

\subsection*{Note}

In data-driven mode, that is with \Lit{CALL}~\Rind{MINUIT}, you should
{\bf not call \Rind{MNINIT}}, since Minuit takes care of all
initialization.
To change unit numbers, call \Rind{MINTIO}
{\bf before calling \Rind{MINUIT}}.

In order that control returns to the user program after
\Lit{CALL MINUIT}, the last command in the corresponding Data Block
should be \Cind[RETurn]{RETURN}.
If the last command is \Cind{EXIT} or \Cind{STOP},
then Minuit will execute a Fortran \Lit{STOP}, and if the last
command is \Cind{END}, Minuit will read a new Data Block from the current
input unit.

\Subsection{10cm}{Data to drive Minuit}

In data-driven mode, either interactively or in batch,
Minuit reads the following data provided by the user:

\begin{UL}
\item {\bf Title:} (a string of 50 characters or less)
      which can be chosen freely by the user, to help identify the job.
\item {\bf Parameter definitions:} for each parameter one record giving:
      \begin{OL}
      \item {\bf The parameter number.}
            This is the index in the array \Lit{XVAL} by which the
            user function \Rind{FCN} will access the value of the parameter.
      \item {\bf The parameter name.}
            A string of ten characters to help the user in
            reading the Minuit output.
      \item {\bf The starting value} of the parameter.
      \item {\bf The starting step size,}
             or expected uncertainty in this parameter,
             if it is to be a variable parameter. Otherwise blank or zero if the
             value is to be constant.
      \item [Optional] The {\bf lower bound}
            (limit) below which the parameter value must not vary.
      \item [Optional] The {\bf upper bound}
            (limit) above which the parameter value must not vary.
      \end{OL}
      Normally the user should {\bf not} specify limits on the parameters, that is
      both should be left blank. If one limit is specified, then BOTH must
      be specified. The properties of limits are explained elsewhere
      in this document.

      The format of the parameter definitions may be either
      fixed-field (each item in a field of width ten columns),
      or in free-field format.
      In the free-field format, items are separated by blanks or one comma,
      and the parameter name must be given between single quotes.
      The program assumes free-field format if it finds two single
      quotes in the line.
      Parameter names will be blank-padded or truncated to be
      ten characters long.
\item {\bf A blank record:} indicates the end of parameter definitions.
\item If the user \Rind{FCN} reads input data from the same input stream as the
       Minuit data (the default stream is \Lit{UNIT 5}),
       then the \Lit{FCN} data should appear here.
\item {\bf Minuit commands:} these specify actions which should be performed by Minuit.
       Commands must not contain leading or embedded blanks, but may be
       truncated to three characters, and may be given in upper or lower case.
       Some commands have numerical arguments, and these may be given in
       free-field format, separated by blank(s) or one comma\footnote{%
       In older versions of Minuit, there was a special format for the \Cind{MINOs}
       command, when specifying a list of parameters; the new Minuit reads
       the \Cind{MINOs} command with the same free-field format as the other
       commands, so if parameter numbers are specified, they must now
       be separated by a blank or comma.}.
       The list of recognized commands is given and explained below.
       The command \Cind{HELP} causes Minuit to write to the output stream a list
       of currently recognized commands.
       The command \Cind{HELP SHOw} lists the available \Cind{SET}
       and \Cind{SHOw} commands.
\end{UL}

Any or all of the above data read by Minuit
can reside on one or more different files,
and Minuit can be instructed
to switch to reading a different file with the \Cind[SET INPut]{SET INPUT} command.
Optionally, the {\bf title} record may be preceeded by a record
beginning with the characters \Cind[SET TITle]{SET TITLE}, and the
{\bf parameter definitions} may be preceeded by a record
beginning with the characters \Cind[PARameters]{PARAMETERS}.
It is in fact recommended always to include these optional
records when preparing a data file, since the file can then be
read at any time (not just at the beginning of a Minuit run)
and will always be interpreted correctly by Minuit.

\begin{XMPt}{Example of a typical Minuit data set}
SET TITLE
Fit to time distribution of K decays, Expt NA94
PARAMETERS
1 'Real(X)'  0.  .1
2 'Imag(X)'  0.  .1
5 'Delta M'  .535   .01
10 'K Short LT'  .892
11 'K Long LT'   518.3

fix 5
migrad
set print 0
minos
restore
migrad
minos
fix 5
set param  5   0.535
contour 1 2
stop
\end{XMPt}

\subsection{Batch and interactive running.}

\index{batch run}
\index{interactive session}
In its initialization phase, Minuit attempts to determine whether
or not it is running interactively, by calling the logical function
\Rind{INTRAC}, a routine in the CERN Program Library which can
be provided for all commonly used computers.
For our purposes, we define ``running interactively'' as meaning that
input is coming from a terminal under the control of an intelligent
being, able to make decisions based on the output he receives at
the terminal. It is not always easy for \Rind{INTRAC} to know whether this
is the case, so, depending on your operating system, Minuit can be
fooled in certain cases. When this happens, the user can always override
the beliefs of \Rind{INTRAC} with the commands \Cind{SET BATch} and
\Cind{SET INTeractive}.
The command \Cind{SHOw INTeractive} informs the user of the current mode.

According to whether or not it believes it is running interactively,
Minuit behaves differently in the following ways:

\begin{UL}
\item If interactive, the user is prompted before each data record is read.
\item If interactive, Minuit recovers from many error conditions
      and prompts the user to enter correct data or to specify
      additional required input.
      If the same error conditions occur in batch mode, the program either
      exits (if no corrective action seems possible) or ignores the incorrect
      data (for example, a command it cannot interpret) and continues.
\item The default page size for output is a typical terminal dimension
      (80 by 24) if interactive, and a typical printed page size (120 by 56)
      if batch, but these can be overridden with the commands
      \Cind[SET WIDthpage]{SET WIDth} and \Cind[SET LINesperpage]{SET LINes}.
\end{UL}

When an interactive user requests Minuit to read
further input from an external file (the \Cind{SET INPut} command),
then further input is considered to be temporarily
in batch mode, until input reverts to the primary input stream.

\Section{5cm}{Running Minuit in Fortran-callable mode.}

The following Minuit subroutines are provided in order to allow
the user to communicate with Minuit and perform all Minuit
functions (define parameters, execute commands, etc.) directly
from Fortran through subroutine calls.
In the following list of subroutines, output arguments are indicated
by appending a star \Lit{*} to its name.
It should also be noted that for the Double Precision version of
Minuit (recommended for all 32-bit machines such as IBM, Vax,
Unix workstations, etc.), all the \Lit{REAL} arguments given below must be
declared \Lit{DOUBLE PRECISION}.

\Subsection{4cm}{Initialize Minuit}

\Shubr{MNINIT}{(IRD,IWR,ISAV)}

\begin{DLtt}{123456}
\item[{\rm\bf Input parameters:}]
\item[IRD]  Unit number for input to Minuit.
\item[IWR]  Unit number for output from Minuit.
\item[ISAV] Unit number for use of the SAVE command.
\end{DLtt}

\Subsection{4cm}{Specify a title for a problem}

\Shubr{MNSETI}{(CTITLE)}

\begin{DLtt}{123456}
\item[{\rm\bf Input parameter:}]
\item[CTITLE] Character string of up to 50 characters containing
              an identification text for the present job or fit.
\end{DLtt}

\Subsection{4cm}{Define a parameter}

\Shubr{MNPARM}{(NUM,CHNAM,STVAL,STEP,BND1,BND2,IERFLG*)}

\begin{DLtt}{123456}
\item[{\rm\bf Input parameters:}]
\item[NUM]    Parameter number as referenced by user in \Rind{FCN}.
\item[CHNAM]  Character string of up to 10 characters containing
              the name which the user assigned to the given parameter.
\item[STVAL]  Starting value
\item[STEP]   Starting step size or approximate parameter error.
\item[BND1]   Lower bound (limit) on parameter value, if any (see below).
\item[BND2]   Upper bound (limit) on parameter value, if any (see below).
\item[{\rm\bf Output parameter:}]
\item[IERFLG] Error return code: \Lit{0} if no error, \Lit{>0} if request failed.
\end{DLtt}

If \Lit{BND1=BND2=0.}, then the parameter is considered unbounded, which
is recommended unless limits are needed to make things behave well.

\Subsection{4cm}{Execute a Minuit command}

\Shubr{MNEXCM}{(FCN,CHCOM,ARGLIS,NARG,IERFLG,FUTIL)}

\begin{DLtt}{123456}
\item[{\rm\bf Input parameters:}]
\item[FCN]    Name fo the function being analyzed (to be declared \Lit{EXTERNAL})
\item[CHCOM]  Character string containing the name of the Minuit
              command to be executed (see below).
\item[ARGLIS] Array of dimension \Lit{MAXARG}, containing the numeric arguments
              to the command (if any),
\item[NARG]   Number of arguments specified (\Lit{NARG}$\leq${MAXARG}),
\item[FUTIL]  Name fo a function called by \Rind{FCN} (or \Lit{=0} if not used).
              If used this function must be declared \Lit{EXTERNAL}.
\item[{\rm\bf Output parameter:}]
\item[IERFLG] Error return code: \Lit{0} if the command was executed normally,
              \Lit{>0} otherwise.
\end{DLtt}

Executing a command by calling \Rind{MNEXCM} has exactly the same
effect as reading the same command in data-driven mode, except that
a few commands would make no sense and are not available in
Fortran-callable mode (e.g. \Cind[SET INPut]{SET INPUT}).
The other difference is that {\bf control always returns to the
calling routine from \Rind{MNEXCM}}, even after commands \Cind{END},
\Cind{EXIT}, and \Cind{STOP}.

\Subsection{4cm}{Get the current value of a parameter}

This routine is the inverse of \Rind{MNPARM} and
can for instance be used after a fit.

\Shubr{MNPOUT}{(NUM,CHNAM*,VAL*,ERROR*,BND1*,BND2*,IVARBL*)}

\begin{DLtt}{123456}
\item[{\rm\bf Input parameter:}]
\item[NUM]    Parameter number as referenced by user in \Rind{FCN} and
              about which information is required.
\item[{\rm\bf Output parameters:}]
\item[CHNAM]  Character string of up to 10 characters containing
              the name which the user assigned to the given parameter.
\item[VAL]    Current parameter value (fitted value if fit has converged),
\item[ERROR]  Current estimate of parameter uncertainty (or zero if constant)
\item[BND1]   Lower limit on parameter value, if any (otherwise zero).
\item[BND2]   Upper limit on parameter value, if any (otherwise zero).
\item[IVARBL] Internal parameter number if parameter is variable, or
              zero if parameter is constant, or negative if parameter is undefined.
\end{DLtt}

\Subsection{4cm}{Get the current status of minimization}

\Shubr{MNSTAT}{(FMIN*,FEDM*,ERRDEF*,NPARI*,NPARX*,ISTAT*)}

\begin{DLtt}{123456}
\item[{\rm\bf Output parameters:}]
\item[FMIN]   The best function value found so far
\item[FEDM]   The estimated vertical distance remaining to minimum
\item[ERRDEF] The value of \Lit{UP} defining parameter uncertainties
\item[NPARI]  The number of currently variable parameters
\item[NPARX]  The highest (external) parameter number defined by user
\item[ISTAT]  A status integer indicating how good is the covariance matrix:
              \begin{DLtt}{1}
                \item[0] Not calculated at all
                \item[1] Diagonal approximation only, not accurate
                \item[2] Full matrix, but forced positive-definite
                \item[3] Full accurate covariance matrix (After \Cind[MIGrad]{MIGRAD},
                         this is the indication of normal convergence.)
              \end{DLtt}
\end{DLtt}

\Subsection{4cm}{Get the current value of the covariance matrix}

\Shubr{MNEMAT}{(EMAT*,NDIM)}

\begin{DLtt}{123456}
\item[{\rm\bf Input parameter:}]
\item[NDIM]   Integer variable specifying the number of rows and columns
              the suer has reserved in \Lit{EMAT} to store the matrix elements.
              \Lit{NDIM} should be at least as large as the number of parameters
              variable at the time of the call, otherwise the user will get
              only part of the full matrix.
\item[{\rm\bf Output parameter:}]
\item[EMAT]   Array declared as \Lit{DIMENSION EMAT(NDIM,NDIM)} which
              is to be filled with the (external) covariance matrix.
\end{DLtt}

\Subsection{4cm}{Access current parameter errors}

\Shubr{MNERRS}{(NUM,EPLUS*,EMINUS*,EPARAB*,GLOBCC*)}

\begin{DLtt}{123456}
\item[{\rm\bf Input parameter:}]
\item[NUM]    Parameter number.
              If \Lit{NUM>0}, this is taken to be an external parameter number;
              if \Lit{NUM<0}, it is the negative of an internal parameter number.
\item[{\rm\bf Output parameters:}]
\item[EPLUS]  The positive \Cind[MINOs]{MINOS} error of parameter \Lit{NUM}.
\item[EMINUS] The negative \Cind[MINOs]{MINOS} error (a negative number).
\item[EPARAB] The ``parabolic'' parameter error, from the error matrix.
\item[GLOBCC] The global correlation coefficient for parameter \Lit{NUM}.
              This is a number between zero and one which gives the correlation
              between parameter \Lit{NUM} and that linear combination of all other
              parameters which is most strongly correlated with \Lit{NUM}.
\end{DLtt}

Note that this call does not cause the errors to be
calculated, it merely returns the current existing values.
If any of the requested values has not been calculated, or has
been destroyed (for example, by a redefinition of parameter values)
\Rind{MNERRS} returns a value of zero for that argument.
Thus the call to \Rind{MNERRS} will normally follow the execution of
commands \Cind[MIGrad]{MIGRAD}, \Cind[HESse]{HESSE},
\Cind{MNContour}, and/or \Rind[MINOs]{MINOS}.

\Subsection{4cm}{Find a function contour with the MNContour method}

\Shubr{MNCONT}{(FCN,NUM1,NUM2,NPT,XPT*,YPT*,NFOUND*,FUTIL)}

\begin{DLtt}{123456}
\item[{\rm\bf Input parameters:}]
\item[FCN]    Name of the function being treated (to be declared \Lit{EXTERNAL})
\item[NUM1/2] Parameter numbers with respect to which the contour
              is to be determined (external).
\item[NPT]    The number of points required on the contour (\Lit{>4}).
\item[FUTIL]  Name of a function called by \Rind{FCN} (or =0 if not used).
              If used this function must be declared \Lit{EXTERNAL}.
\item[{\rm\bf Output parameters:}]
\item[XPT]    Array of x-coordinates of contour points with
              values for parameter \Lit{NUM1}.
              It must be declared with a \Lit{DIMENSION XPT(NPT)}.
\item[YPT]    Array of y-coordinates of contour points with
              values for parameter \Lit{NUM2}.
              It must be declared with a \Lit{DIMENSION YPT(NPT)}.
\item[NFOUND] The number of points actually found on the contour.
              If all goes well, this will be equal to \Lit{NPT}, but it can be
              negative (if the input arguments are not valid), or zero if
              less than four points have been found, or less than \Lit{NPT} if the
              program could not find \Lit{NPT} points.
\end{DLtt}

Note that alternatively \Cind{MNContour} can be calculated
by calling \Rind{MNEXCM} to issue the \Cind{MNContour} command,
but then the user does not have Fortran access to the actual
point coordinates \Lit{XPT} and \Lit{YPT}.

\Subsection{4cm}{Switch to command-reading mode}

This facility can be useful when one wants to continue interactively.

\Shubr{MNINTR}{(FCN,FUTIL)}

\begin{DLtt}{123456}
\item[{\rm\bf Input parameters:}]
\item[FCN]    Name of the function being treated (to be declared \Lit{EXTERNAL})
\item[FUTIL]  Name of a function called by \Rind{FCN} (or \Lit{=0} if not used).
              If used this function must be declared \Lit{EXTERNAL}.
\end{DLtt}

The call to \Rind{MNINTR} will cause Minuit to read commands from
the unit \Lit{IRD} (originally specified by the user in his call to \Rind{MNINIT},
\Lit{IRD} is usually 5 by default,
which in turn is usually the terminal by default).
Minuit then reads and executes commands until it encounters
a command \Cind{END}, \Cind{EXIT}, \Cind{RETurn}, or \Cind{STOP},
or an end-of-file on input
(or an unrecoverable error condition while reading
or trying to execute a command), in which case control returns
to the program which called \Rind{MNINTR}.