\documentclass[english]{article}
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage{csquotes, caption}
\usepackage[colorlinks=true, citecolor = blue]{hyperref}      % hyperlinks
\usepackage{multirow, amssymb, amsmath, graphicx, arydshln, url}
\usepackage[T1]{fontenc}
\usepackage{natbib}

\title{Supporting Information for ``Modelling publication bias and \textit{p}-hacking''\\ by Jonas Moss and Riccardo De Bin}

\author{}

\date{}

\begin{document}

\maketitle

\section*{Web Appendix A}

As mentioned in Section 2, any \textit{p}-hacking model can be written on the form of a selection model. Observe that
\begin{eqnarray*}
\int_{[0,1]}f_\alpha^{\star}(x_{i}\mid\theta_{i},\eta_{i}, u_i)d\omega(\alpha) & = & \int_{[0,1]}f(x_{i}\mid\theta_{i})P(u_i\in\left[0,\alpha\right]\mid\theta_{i},\eta_{i})^{-1}d\omega(\alpha)\\
 & = & f(x_{i}\mid\theta_{i})\int_{[0,u_i)}P(u_i\in\left[0,\alpha\right]\mid\theta_{i},\eta_{i})^{-1}d\omega(\alpha).
\end{eqnarray*}
where $f_\alpha^{\star}$ is the density $f^{\star}$ truncated so that the \textit{p}-value associated to $x_i$, $u_i$, lies in the interval $\left[0,\alpha\right)$. This is a publication bias model if $$h(u_i)=\int_{[0,u_i]}P(u_i\in\left[0,\alpha\right]\mid\theta_{i},\eta_{i})^{-1}d\omega(\alpha)$$ is bounded for each $u_i$ and $h(u_i)$ is independent of $\theta_{i},\eta_{i}$. While $h(u_i)$ can be bounded, it is typically dependent of $\theta_{i},\eta_{i}$, with the fixed effect model under complete selection for significance being a notable exception.

On the other hand, any selection model $f(x_{i};\theta_{i},\eta_{i})\rho(u_i)$ with $I =\int f(x;\theta_{i},\eta_{i})\rho(u_i)du_i<\infty$ can be written as a mixture model. For then there is a finite measure $d\omega(\alpha;\theta_{i},\eta_{i})$ satisfying
\[
\rho(u_i)=\int_{[0,u_i)}\frac{1}{P(u_i\in\left[0,\alpha\right)\mid\theta,\eta)}d\omega(\alpha;\theta_{i},\eta_{i})
\]
Just take $d\omega(\alpha;\theta_{i},\eta_{i})=d\rho(\alpha)P(u_i\in\left[0,\alpha\right)\mid\theta_{i},\eta_{i})$, where $d\rho(\alpha)$ is defined by $\int_{0}^{u_i}d\rho(\alpha)=\rho(u_i)$. The size of the measure is
\begin{eqnarray*}
\int_{0}^{1}d\omega(\alpha;\theta_{i},\eta_{i}) & = & \int_{0}^{1}P(u_i\in\left[0,\alpha\right)\mid\theta_{i},\eta_{i})d\rho(\alpha)\\
 & = & \int_{0}^{1}f(u_i;\theta_{i},\eta_{i})\int_{0}^{u_i}d\rho(\alpha)du_i\\
 & = & I
\end{eqnarray*}
Hence $I_{\theta,\eta}d\omega'(\alpha;\theta_{i},\eta_{i})$ is a probability measure. This probability measure makes
\[
I^{-1}f(x_{i};\theta_{i},\eta_{i})\rho(u_i)=\int_{[0,1]}f_\alpha(x_{i};\theta_{i},\eta_{i})d\omega'(\alpha)
\]
as can be seen by the following computation,
\begin{eqnarray*}
I^{-1}f(x_{i};\theta_{i},\eta_{i})\rho(u_i) & = & I^{-1}\int_{[0,u_i)}\frac{f(x_{i};\theta_{i},\eta_{i})}{P(u_i\in\left[0,\alpha\right)\mid\theta_{i},\eta_{i})}d\omega(\alpha)\\
 & = & I^{-1}\int_{[0,1]}\frac{f(x_{i};\theta,\eta)1_{\left[0,\alpha\right)}(u_i)}{P(u_i\in\left[0,\alpha\right)\mid\theta_{i},\eta_{i})}d\omega(\alpha)\\
 & = & I^{-1}\int_{[0,1]}f_\alpha(x_{i};\theta_{i},\eta_{i})d\omega(\alpha)\\
 & = & \int_{[0,1]}f_\alpha(x;\theta_{i},\eta_{i})d\omega'(\alpha)
\end{eqnarray*}

Proposition 1 shows the form of the one-sided normal step function selection probability publication bias model when it is written as a mixture model of the form (5). But most such mixture models are not true \textit{p}-hacking models, as the mixing probabilities $\pi_{i}^{\star}$ depend on $\theta$. There is no way for the \textit{p}-hacker to know
$\theta$, so we cannot regard the publication bias model as a \textit{p}-hacking model.

\newpage

\section*{Web Appendix B}

%\subsection*{Inverse Gamma prior on $\tau^2$}

\begin{table}[ht]
\centering
\caption*{\noindent Table C.1: {\bf Inverse Gamma prior, no publication bias, no
                    \textit{p}-hacking.} Posterior means and
                    standard deviations from the \textit{p}-hacking and
                    publication bias models when the data are simulated
                    from the normal random effects meta-analysis model.}
\label{tab:Simulation_ph}
\begin{tabular}{lllrrrrrr}
   \multicolumn{3}{r}{\textbf{True values}} &
       \multicolumn{2}{c}{\textbf{\textit{p}-hacking model}} &
       \multicolumn{2}{c}{\textbf{Public. bias model}} &
       \multicolumn{2}{c}{\textbf{Classical model}}\\$\tau$ & $\theta_0$ & $n$ & \multicolumn{1}{c}{$\widehat{\theta_0}$} & \multicolumn{1}{c}{$\widehat{\tau}$} & \multicolumn{1}{c}{$\widehat{\theta_0}$} & \multicolumn{1}{c}{$\widehat{\tau}$} & \multicolumn{1}{c}{$\widehat{\theta_0}$} & \multicolumn{1}{c}{$\widehat{\tau}$} \\
   \hline
  \multirow{9}{*}{$0.1$} & \multirow{3}{*}{$0$} & 5 & -0.03 (0.08) & 0.18 (0.08) & -0.06 (0.06) & 0.14 (0.07) & 0.00 (0.07) & 0.20 (0.09) \\
  & & 30 & -0.02 (0.03) & 0.08 (0.03) & -0.02 (0.03) & 0.07 (0.03) & 0.00 (0.03) & 0.10 (0.04) \\
  & & 100 & -0.01 (0.02) & 0.08 (0.02) & -0.02 (0.02) & 0.07 (0.02) & 0.00 (0.02) & 0.09 (0.02) \\
   \cdashline{3-9}
 & \multirow{3}{*}{$0.2$} & 5 & 0.12 (0.08) & 0.20 (0.07) & 0.08 (0.06) & 0.15 (0.06) & 0.19 (0.08) & 0.18 (0.07) \\
  & & 30 & 0.17 (0.04) & 0.09 (0.04) & 0.15 (0.03) & 0.08 (0.04) & 0.20 (0.03) & 0.09 (0.03) \\
  & & 100 & 0.18 (0.02) & 0.09 (0.03) & 0.17 (0.02) & 0.09 (0.03) & 0.20 (0.02) & 0.09 (0.02) \\
   \cdashline{3-9}
 & \multirow{3}{*}{$0.8$} & 5 & 0.78 (0.07) & 0.20 (0.10) & 0.64 (0.13) & 0.32 (0.14) & 0.79 (0.07) & 0.18 (0.08) \\
  & & 30 & 0.80 (0.04) & 0.10 (0.04) & 0.79 (0.04) & 0.10 (0.04) & 0.80 (0.03) & 0.10 (0.04) \\
  & & 100 & 0.80 (0.02) & 0.09 (0.03) & 0.80 (0.02) & 0.09 (0.03) & 0.80 (0.02) & 0.09 (0.03) \\
  \cline{2-9}
\multirow{9}{*}{$0.5$} & \multirow{3}{*}{$0$} & 5 & -0.08 (0.20) & 0.56 (0.22) & -0.24 (0.18) & 0.50 (0.21) & -0.03 (0.21) & 0.58 (0.21) \\
  & & 30 & -0.03 (0.08) & 0.50 (0.07) & -0.14 (0.08) & 0.46 (0.07) & 0.00 (0.08) & 0.50 (0.07) \\
  & & 100 & -0.02 (0.05) & 0.50 (0.04) & -0.08 (0.06) & 0.48 (0.04) & 0.00 (0.05) & 0.50 (0.04) \\
   \cdashline{3-9}
 & \multirow{3}{*}{$0.2$} & 5 & 0.16 (0.24) & 0.60 (0.19) & -0.04 (0.20) & 0.58 (0.2) & 0.20 (0.24) & 0.60 (0.19) \\
  & & 30 & 0.16 (0.08) & 0.52 (0.07) & 0.03 (0.09) & 0.50 (0.07) & 0.19 (0.08) & 0.52 (0.07) \\
  & & 100 & 0.18 (0.06) & 0.51 (0.04) & 0.10 (0.06) & 0.50 (0.04) & 0.20 (0.06) & 0.50 (0.04) \\
   \cdashline{3-9}
 & \multirow{3}{*}{$0.8$} & 5 & 0.73 (0.26) & 0.61 (0.17) & 0.40 (0.28) & 0.74 (0.18) & 0.75 (0.25) & 0.58 (0.17) \\
  & & 30 & 0.77 (0.09) & 0.53 (0.07) & 0.60 (0.12) & 0.60 (0.09) & 0.79 (0.08) & 0.51 (0.07) \\
  & & 100 & 0.80 (0.06) & 0.52 (0.04) & 0.71 (0.07) & 0.55 (0.04) & 0.81 (0.05) & 0.51 (0.04) \\
  \hline
\end{tabular}
\end{table}

\begin{table}[ht]
\centering
\caption*{\noindent Table C.2: {\bf Inverse Gamma prior, publication bias.}
                    Posterior means and standard deviations from the
                    \textit{p}-hacking, publication bias, and uncorrected models
                    when the data are simulated from the publication
                    bias model with cutoffs at $0.025$ and $0.05$,
                    with selection probabilities equal to $1$, $0.7$,
                    and $0.1$ in the intervals $[0, 0.025)$, $[0.025, 0.05)$,
                    and $[0.5, 1]$.}
\label{tab:Simulation_pb}
\begin{tabular}{lllrrrrrr}
   \multicolumn{3}{r}{\textbf{True values}} &
       \multicolumn{2}{c}{\textbf{\textit{p}-hacking model}} &
       \multicolumn{2}{c}{\textbf{Public.\ bias model}} &
       \multicolumn{2}{c}{\textbf{Uncorrected model}}\\$\tau$ & $\theta_0$ & $n$ & \multicolumn{1}{c}{$\widehat{\theta_0}$} & \multicolumn{1}{c}{$\widehat{\tau}$} & \multicolumn{1}{c}{$\widehat{\theta_0}$} & \multicolumn{1}{c}{$\widehat{\tau}$} & \multicolumn{1}{c}{$\widehat{\theta_0}$} & \multicolumn{1}{c}{$\widehat{\tau}$} \\
   \hline
  \multirow{9}{*}{$0.1$} & \multirow{3}{*}{$0$} & 5 & -0.02 (0.11) & 0.21 (0.08) & 0.00 (0.08) & 0.16 (0.07) & 0.12 (0.08) & 0.22 (0.10) \\
  & & 30 & 0.03 (0.05) & 0.12 (0.04) & 0.02 (0.04) & 0.10 (0.03) & 0.14 (0.04) & 0.15 (0.04) \\
  & & 100 & 0.02 (0.02) & 0.11 (0.03) & 0.01 (0.02) & 0.09 (0.02) & 0.13 (0.02) & 0.16 (0.02) \\
   \cdashline{3-9}
 & \multirow{3}{*}{$0.2$} & 5 & 0.12 (0.13) & 0.29 (0.10) & 0.10 (0.07) & 0.21 (0.09) & 0.33 (0.06) & 0.16 (0.07) \\
  & & 30 & 0.22 (0.05) & 0.12 (0.05) & 0.19 (0.05) & 0.10 (0.04) & 0.33 (0.03) & 0.06 (0.03) \\
  & & 100 & 0.23 (0.03) & 0.11 (0.03) & 0.19 (0.05) & 0.09 (0.03) & 0.33 (0.02) & 0.04 (0.02) \\
   \cdashline{3-9}
 & \multirow{3}{*}{$0.8$} & 5 & 0.79 (0.11) & 0.21 (0.10) & 0.65 (0.17) & 0.31 (0.13) & 0.80 (0.10) & 0.19 (0.07) \\
  & & 30 & 0.80 (0.03) & 0.10 (0.04) & 0.80 (0.03) & 0.10 (0.05) & 0.80 (0.03) & 0.10 (0.04) \\
  & & 100 & 0.80 (0.02) & 0.09 (0.03) & 0.80 (0.02) & 0.10 (0.03) & 0.80 (0.02) & 0.09 (0.03) \\
  \cline{2-9}
\multirow{9}{*}{$0.5$} & \multirow{3}{*}{$0$} & 5 & 0.30 (0.26) & 0.53 (0.20) & 0.03 (0.22) & 0.55 (0.19) & 0.40 (0.22) & 0.48 (0.20) \\
  & & 30 & 0.34 (0.10) & 0.47 (0.08) & 0.00 (0.17) & 0.49 (0.08) & 0.42 (0.09) & 0.42 (0.09) \\
  & & 100 & 0.35 (0.05) & 0.46 (0.04) & 0.00 (0.10) & 0.49 (0.05) & 0.43 (0.04) & 0.42 (0.04) \\
   \cdashline{3-9}
 & \multirow{3}{*}{$0.2$} & 5 & 0.48 (0.22) & 0.55 (0.19) & 0.17 (0.22) & 0.61 (0.19) & 0.55 (0.19) & 0.47 (0.21) \\
  & & 30 & 0.52 (0.09) & 0.43 (0.08) & 0.18 (0.16) & 0.51 (0.09) & 0.57 (0.08) & 0.37 (0.08) \\
  & & 100 & 0.50 (0.05) & 0.43 (0.05) & 0.20 (0.13) & 0.50 (0.05) & 0.56 (0.04) & 0.38 (0.04) \\
   \cdashline{3-9}
 & \multirow{3}{*}{$0.8$} & 5 & 0.81 (0.22) & 0.52 (0.20) & 0.48 (0.27) & 0.68 (0.21) & 0.85 (0.20) & 0.47 (0.18) \\
  & & 30 & 0.91 (0.09) & 0.45 (0.07) & 0.69 (0.18) & 0.57 (0.11) & 0.93 (0.08) & 0.42 (0.07) \\
  & & 100 & 0.91 (0.05) & 0.44 (0.04) & 0.75 (0.13) & 0.53 (0.08) & 0.93 (0.05) & 0.41 (0.04) \\ \hline
\end{tabular}
\end{table}

\begin{table}[ht]
\centering
\caption*{\noindent Table C.3: {\bf  Inverse Gamma prior, \textit{p}-hacking.} Posterior means and
                    standard deviations from the \textit{p}-hacking,
                    publication bias, and uncorrected models when the data are simulated
                    from the \textit{p}-hacking model with cutoffs at
                    $0.025$ and $0.05$, with \textit{p}-hacking probabilities
                    equal to $0.6$, $0.3$, and $0.1$ for $\alpha = 0.025, 0.05$, and $1$}
\label{tab:Simulation_ph}
\begin{tabular}{lllrrrrrr}
   \multicolumn{3}{r}{\textbf{True values}} &
       \multicolumn{2}{c}{\textbf{\textit{p}-hacking model}} &
       \multicolumn{2}{c}{\textbf{Public. bias model}} &
       \multicolumn{2}{c}{\textbf{Uncorrected model}}\\$\tau$ & $\theta_0$ & $n$ & \multicolumn{1}{c}{$\widehat{\theta_0}$} & \multicolumn{1}{c}{$\widehat{\tau}$} & \multicolumn{1}{c}{$\widehat{\theta_0}$} & \multicolumn{1}{c}{$\widehat{\tau}$} & \multicolumn{1}{c}{$\widehat{\theta_0}$} & \multicolumn{1}{c}{$\widehat{\tau}$} \\
   \hline
  \multirow{9}{*}{$0.1$} & \multirow{3}{*}{$0$} & 5 & -0.06 (0.13) & 0.28 (0.07) & 0.04 (0.07) & 0.17 (0.05) & 0.29 (0.07) & 0.15 (0.08) \\
  & & 30 & -0.02 (0.07) & 0.13 (0.04) & 0.02 (0.06) & 0.06 (0.02) & 0.28 (0.02) & 0.06 (0.03) \\
  & & 100 & -0.01 (0.04) & 0.10 (0.04) & -0.01 (0.05) & 0.05 (0.03) & 0.29 (0.01) & 0.03 (0.02) \\
   \cdashline{3-9}
 & \multirow{3}{*}{$0.2$} & 5 & 0.13 (0.15) & 0.29 (0.09) & 0.09 (0.07) & 0.22 (0.08) & 0.35 (0.06) & 0.14 (0.06) \\
  & & 30 & 0.18 (0.06) & 0.11 (0.04) & 0.15 (0.05) & 0.09 (0.03) & 0.34 (0.02) & 0.04 (0.01) \\
  & & 100 & 0.20 (0.03) & 0.09 (0.04) & 0.16 (0.04) & 0.08 (0.03) & 0.34 (0.01) & 0.02 (0.01) \\
   \cdashline{3-9}
 & \multirow{3}{*}{$0.8$} & 5 & 0.78 (0.08) & 0.20 (0.08) & 0.63 (0.14) & 0.32 (0.12) & 0.78 (0.08) & 0.19 (0.07) \\
  & & 30 & 0.79 (0.04) & 0.10 (0.04) & 0.79 (0.04) & 0.10 (0.04) & 0.79 (0.03) & 0.10 (0.04) \\
  & & 100 & 0.80 (0.02) & 0.10 (0.02) & 0.80 (0.02) & 0.10 (0.03) & 0.80 (0.02) & 0.09 (0.02) \\
  \cline{2-9}
\multirow{9}{*}{$0.5$} & \multirow{3}{*}{$0$} & 5 & 0.06 (0.20) & 0.47 (0.19) & 0.02 (0.10) & 0.35 (0.21) & 0.37 (0.12) & 0.27 (0.18) \\
  & & 30 & 0.07 (0.09) & 0.44 (0.09) & -0.25 (0.19) & 0.36 (0.11) & 0.37 (0.05) & 0.24 (0.10) \\
  & & 100 & 0.06 (0.06) & 0.44 (0.04) & -0.36 (0.15) & 0.37 (0.06) & 0.36 (0.03) & 0.25 (0.05) \\
   \cdashline{3-9}
 & \multirow{3}{*}{$0.2$} & 5 & 0.24 (0.21) & 0.52 (0.20) & 0.05 (0.12) & 0.47 (0.21) & 0.45 (0.13) & 0.33 (0.18) \\
  & & 30 & 0.24 (0.10) & 0.47 (0.08) & -0.20 (0.20) & 0.46 (0.11) & 0.45 (0.06) & 0.29 (0.08) \\
  & & 100 & 0.23 (0.06) & 0.47 (0.04) & -0.30 (0.15) & 0.48 (0.06) & 0.46 (0.04) & 0.28 (0.04) \\
   \cdashline{3-9}
 & \multirow{3}{*}{$0.8$} & 5 & 0.69 (0.18) & 0.60 (0.20) & 0.31 (0.19) & 0.74 (0.19) & 0.76 (0.14) & 0.51 (0.18) \\
  & & 30 & 0.80 (0.10) & 0.51 (0.08) & 0.39 (0.25) & 0.66 (0.12) & 0.85 (0.08) & 0.44 (0.07) \\
  & & 100 & 0.81 (0.05) & 0.50 (0.04) & 0.43 (0.19) & 0.65 (0.09) & 0.86 (0.04) & 0.43 (0.03) \\
  \hline
\end{tabular}
\end{table}

\newpage

%\subsection*{Uniform prior on $\tau$}

\begin{table}[ht]
\centering
\caption*{\noindent Table C.4: {\bf Uniform prior, no publication bias, no
                    \textit{p}-hacking.} Posterior means and
                    standard deviations from the \textit{p}-hacking and
                    publication bias models when the data are simulated
                    from the normal random effects meta-analysis model.}
\label{tab:Simulation_ph}
\begin{tabular}{lllrrrrrr}
   \multicolumn{3}{r}{\textbf{True values}} &
       \multicolumn{2}{c}{\textbf{\textit{p}-hacking model}} &
       \multicolumn{2}{c}{\textbf{Public. bias model}} &
       \multicolumn{2}{c}{\textbf{Classical model}}\\$\tau$ & $\theta_0$ & $n$ & \multicolumn{1}{c}{$\widehat{\theta_0}$} & \multicolumn{1}{c}{$\widehat{\tau}$} & \multicolumn{1}{c}{$\widehat{\theta_0}$} & \multicolumn{1}{c}{$\widehat{\tau}$} & \multicolumn{1}{c}{$\widehat{\theta_0}$} & \multicolumn{1}{c}{$\widehat{\tau}$} \\
   \hline
  \multirow{9}{*}{$0.1$} & \multirow{3}{*}{$0$} & 5 & -0.03 (0.07) & 0.17 (0.06) & -0.06 (0.07) & 0.12 (0.05) & -0.01 (0.08) & 0.18 (0.07) \\
  & & 30 & -0.02 (0.03) & 0.08 (0.03) & -0.02 (0.03) & 0.07 (0.03) & 0.00 (0.04) & 0.10 (0.03) \\
  & & 100 & -0.01 (0.02) & 0.08 (0.03) & -0.02 (0.02) & 0.07 (0.03) & 0.00 (0.02) & 0.10 (0.03) \\
   \cdashline{3-9}
 & \multirow{3}{*}{$0.2$} & 5 & 0.12 (0.09) & 0.21 (0.09) & 0.09 (0.07) & 0.16 (0.08) & 0.19 (0.08) & 0.18 (0.08) \\
  & & 30 & 0.17 (0.03) & 0.09 (0.04) & 0.15 (0.03) & 0.09 (0.03) & 0.20 (0.03) & 0.10 (0.03) \\
  & & 100 & 0.18 (0.02) & 0.09 (0.03) & 0.17 (0.02) & 0.09 (0.03) & 0.20 (0.02) & 0.09 (0.02) \\
   \cdashline{3-9}
 & \multirow{3}{*}{$0.8$} & 5 & 0.77 (0.09) & 0.20 (0.08) & 0.62 (0.14) & 0.31 (0.13) & 0.77 (0.08) & 0.18 (0.07) \\
  & & 30 & 0.80 (0.04) & 0.10 (0.04) & 0.80 (0.04) & 0.10 (0.04) & 0.80 (0.03) & 0.10 (0.04) \\
  & & 100 & 0.80 (0.02) & 0.10 (0.02) & 0.80 (0.02) & 0.10 (0.03) & 0.80 (0.02) & 0.10 (0.02) \\
  \cline{2-9}
\multirow{9}{*}{$0.5$} & \multirow{3}{*}{$0$} & 5 & -0.03 (0.20) & 0.55 (0.19) & -0.21 (0.17) & 0.49 (0.19) & 0.01 (0.20) & 0.57 (0.19) \\
  & & 30 & -0.02 (0.10) & 0.51 (0.08) & -0.13 (0.10) & 0.48 (0.08) & 0.00 (0.10) & 0.51 (0.08) \\
  & & 100 & -0.02 (0.05) & 0.50 (0.04) & -0.08 (0.05) & 0.48 (0.04) & 0.00 (0.05) & 0.50 (0.04) \\
   \cdashline{3-9}
 & \multirow{3}{*}{$0.2$} & 5 & 0.20 (0.22) & 0.57 (0.19) & -0.02 (0.17) & 0.57 (0.19) & 0.24 (0.21) & 0.57 (0.18) \\
  & & 30 & 0.17 (0.10) & 0.52 (0.08) & 0.05 (0.10) & 0.50 (0.08) & 0.20 (0.10) & 0.51 (0.08) \\
  & & 100 & 0.19 (0.06) & 0.51 (0.04) & 0.11 (0.06) & 0.50 (0.04) & 0.20 (0.06) & 0.50 (0.04) \\
   \cdashline{3-9}
 & \multirow{3}{*}{$0.8$} & 5 & 0.74 (0.23) & 0.61 (0.23) & 0.41 (0.24) & 0.75 (0.23) & 0.77 (0.22) & 0.58 (0.22) \\
  & & 30 & 0.79 (0.10) & 0.53 (0.07) & 0.60 (0.14) & 0.61 (0.08) & 0.81 (0.10) & 0.51 (0.07) \\
  & & 100 & 0.78 (0.06) & 0.52 (0.04) & 0.70 (0.07) & 0.55 (0.04) & 0.80 (0.06) & 0.51 (0.04) \\
  \hline
\end{tabular}
\end{table}

\begin{table}[ht]
\centering
\caption*{\noindent Table C.5: {\bf Uniform prior, publication bias.}
                    Posterior means and standard deviations from the
                    \textit{p}-hacking, publication bias, and uncorrected models
                    when the data are simulated from the publication
                    bias model with cutoffs at $0.025$ and $0.05$,
                    with selection probabilities equal to $1$, $0.7$,
                    and $0.1$ in the intervals $[0, 0.025)$, $[0.025, 0.05)$,
                    and $[0.5, 1]$.}
\label{tab:Simulation_pb}
\begin{tabular}{lllrrrrrr}
   \multicolumn{3}{r}{\textbf{True values}} &
       \multicolumn{2}{c}{\textbf{\textit{p}-hacking model}} &
       \multicolumn{2}{c}{\textbf{Public.\ bias model}} &
       \multicolumn{2}{c}{\textbf{Uncorrected model}}\\$\tau$ & $\theta_0$ & $n$ & \multicolumn{1}{c}{$\widehat{\theta_0}$} & \multicolumn{1}{c}{$\widehat{\tau}$} & \multicolumn{1}{c}{$\widehat{\theta_0}$} & \multicolumn{1}{c}{$\widehat{\tau}$} & \multicolumn{1}{c}{$\widehat{\theta_0}$} & \multicolumn{1}{c}{$\widehat{\tau}$} \\
   \hline
  \multirow{9}{*}{$0.1$} & \multirow{3}{*}{$0$} & 5 & -0.01 (0.11) & 0.24 (0.09) & -0.01 (0.08) & 0.18 (0.08) & 0.13 (0.09) & 0.24 (0.10) \\
  & & 30 & 0.02 (0.05) & 0.12 (0.04) & 0.01 (0.04) & 0.10 (0.03) & 0.13 (0.04) & 0.16 (0.04) \\
  & & 100 & 0.03 (0.03) & 0.12 (0.03) & 0.01 (0.02) & 0.10 (0.02) & 0.14 (0.02) & 0.16 (0.02) \\
   \cdashline{3-9}
 & \multirow{3}{*}{$0.2$} & 5 & 0.16 (0.13) & 0.27 (0.08) & 0.12 (0.06) & 0.21 (0.06) & 0.33 (0.06) & 0.15 (0.06) \\
  & & 30 & 0.23 (0.05) & 0.12 (0.05) & 0.19 (0.06) & 0.10 (0.04) & 0.33 (0.03) & 0.06 (0.02) \\
  & & 100 & 0.24 (0.03) & 0.10 (0.03) & 0.20 (0.04) & 0.10 (0.03) & 0.33 (0.02) & 0.04 (0.02) \\
   \cdashline{3-9}
 & \multirow{3}{*}{$0.8$} & 5 & 0.80 (0.07) & 0.19 (0.08) & 0.66 (0.13) & 0.31 (0.13) & 0.80 (0.07) & 0.18 (0.07) \\
  & & 30 & 0.80 (0.03) & 0.10 (0.04) & 0.79 (0.04) & 0.11 (0.04) & 0.80 (0.03) & 0.10 (0.04) \\
  & & 100 & 0.80 (0.02) & 0.09 (0.03) & 0.80 (0.02) & 0.09 (0.02) & 0.80 (0.02) & 0.09 (0.02) \\
  \cline{2-9}
\multirow{9}{*}{$0.5$} & \multirow{3}{*}{$0$} & 5 & 0.36 (0.24) & 0.54 (0.20) & 0.07 (0.23) & 0.57 (0.18) & 0.45 (0.21) & 0.47 (0.22) \\
  & & 30 & 0.39 (0.10) & 0.47 (0.08) & 0.04 (0.17) & 0.51 (0.08) & 0.45 (0.08) & 0.42 (0.08) \\
  & & 100 & 0.36 (0.06) & 0.48 (0.04) & 0.00 (0.12) & 0.51 (0.05) & 0.43 (0.05) & 0.43 (0.05) \\
   \cdashline{3-9}
 & \multirow{3}{*}{$0.2$} & 5 & 0.41 (0.22) & 0.53 (0.18) & 0.11 (0.20) & 0.58 (0.18) & 0.49 (0.17) & 0.46 (0.20) \\
  & & 30 & 0.50 (0.08) & 0.45 (0.08) & 0.15 (0.18) & 0.52 (0.09) & 0.55 (0.07) & 0.39 (0.08) \\
  & & 100 & 0.51 (0.05) & 0.43 (0.05) & 0.17 (0.13) & 0.51 (0.06) & 0.56 (0.04) & 0.38 (0.05) \\
  \cdashline{3-9}
 & \multirow{3}{*}{$0.8$} & 5 & 0.86 (0.16) & 0.52 (0.19) & 0.51 (0.23) & 0.70 (0.20) & 0.89 (0.15) & 0.49 (0.17) \\
  & & 30 & 0.90 (0.09) & 0.46 (0.07) & 0.67 (0.19) & 0.58 (0.11) & 0.92 (0.08) & 0.43 (0.06) \\
  & & 100 & 0.91 (0.05) & 0.44 (0.04) & 0.76 (0.11) & 0.52 (0.07) & 0.93 (0.04) & 0.41 (0.03) \\ \hline
\end{tabular}
\end{table}

\begin{table}[ht]
\centering
\caption*{\noindent Table C.6: {\bf Uniform prior, \textit{p}-hacking.} Posterior means and
                    standard deviations from the \textit{p}-hacking,
                    publication bias, and uncorrected models when the data are simulated
                    from the \textit{p}-hacking model with cutoffs at
                    $0.025$ and $0.05$, with \textit{p}-hacking probabilities
                    equal to $0.6$, $0.3$, and $0.1$ for $\alpha = 0.025, 0.05$, and $1$}
\label{tab:Simulation_ph}
\begin{tabular}{lllrrrrrr}
   \multicolumn{3}{r}{\textbf{True values}} &
       \multicolumn{2}{c}{\textbf{\textit{p}-hacking model}} &
       \multicolumn{2}{c}{\textbf{Public. bias model}} &
       \multicolumn{2}{c}{\textbf{Uncorrected model}}\\$\tau$ & $\theta_0$ & $n$ & \multicolumn{1}{c}{$\widehat{\theta_0}$} & \multicolumn{1}{c}{$\widehat{\tau}$} & \multicolumn{1}{c}{$\widehat{\theta_0}$} & \multicolumn{1}{c}{$\widehat{\tau}$} & \multicolumn{1}{c}{$\widehat{\theta_0}$} & \multicolumn{1}{c}{$\widehat{\tau}$} \\
   \hline
  \multirow{9}{*}{$0.1$} & \multirow{3}{*}{$0$} & 5 & -0.07 (0.14) & 0.28 (0.08) & 0.04 (0.07) & 0.17 (0.07) & 0.29 (0.06) & 0.15 (0.09) \\
  & & 30 & -0.02 (0.08) & 0.13 (0.05) & 0.00 (0.07) & 0.07 (0.03) & 0.29 (0.03) & 0.05 (0.04) \\
  & & 100 & -0.01 (0.04) & 0.11 (0.04) & -0.02 (0.05) & 0.06 (0.02) & 0.28 (0.01) & 0.03 (0.02) \\
   \cdashline{3-9}
 & \multirow{3}{*}{$0.2$} & 5 & 0.11 (0.13) & 0.28 (0.09) & 0.09 (0.05) & 0.20 (0.05) & 0.34 (0.05) & 0.13 (0.05) \\
  & & 30 & 0.20 (0.05) & 0.11 (0.05) & 0.16 (0.06) & 0.09 (0.04) & 0.34 (0.02) & 0.04 (0.02) \\
  & & 100 & 0.20 (0.03) & 0.09 (0.03) & 0.17 (0.04) & 0.08 (0.03) & 0.34 (0.01) & 0.02 (0.01) \\
   \cdashline{3-9}
 & \multirow{3}{*}{$0.8$} & 5 & 0.79 (0.09) & 0.20 (0.08) & 0.64 (0.15) & 0.32 (0.13) & 0.80 (0.08) & 0.19 (0.07) \\
  & & 30 & 0.79 (0.03) & 0.09 (0.04) & 0.79 (0.03) & 0.10 (0.04) & 0.80 (0.03) & 0.09 (0.04) \\
  & & 100 & 0.80 (0.02) & 0.09 (0.03) & 0.80 (0.02) & 0.10 (0.02) & 0.80 (0.02) & 0.09 (0.02) \\
  \cline{2-9}
\multirow{9}{*}{$0.5$} & \multirow{3}{*}{$0$} & 5 & 0.04 (0.23) & 0.47 (0.20) & -0.01 (0.14) & 0.37 (0.22) & 0.34 (0.12) & 0.30 (0.23) \\
  & & 30 & 0.08 (0.10) & 0.43 (0.08) & -0.23 (0.17) & 0.35 (0.10) & 0.37 (0.06) & 0.23 (0.09) \\
  & & 100 & 0.06 (0.06) & 0.44 (0.05) & -0.36 (0.14) & 0.37 (0.06) & 0.37 (0.03) & 0.25 (0.05) \\
   \cdashline{3-9}
 & \multirow{3}{*}{$0.2$} & 5 & 0.22 (0.22) & 0.53 (0.18) & 0.06 (0.11) & 0.45 (0.22) & 0.44 (0.13) & 0.33 (0.19) \\
  & & 30 & 0.24 (0.10) & 0.47 (0.07) & -0.18 (0.18) & 0.45 (0.09) & 0.45 (0.05) & 0.28 (0.08) \\
  & & 100 & 0.23 (0.05) & 0.47 (0.04) & -0.31 (0.18) & 0.47 (0.06) & 0.45 (0.03) & 0.28 (0.04) \\
   \cdashline{3-9}
 & \multirow{3}{*}{$0.8$} & 5 & 0.71 (0.23) & 0.61 (0.18) & 0.34 (0.25) & 0.74 (0.18) & 0.78 (0.18) & 0.53 (0.16) \\
  & & 30 & 0.82 (0.09) & 0.50 (0.08) & 0.43 (0.24) & 0.66 (0.12) & 0.87 (0.08) & 0.43 (0.06) \\
  & & 100 & 0.79 (0.05) & 0.50 (0.04) & 0.41 (0.17) & 0.65 (0.08) & 0.84 (0.05) & 0.42 (0.03) \\
\hline
\end{tabular}
\end{table}

\newpage

\section*{Web Appendix C}

Recall that a density $f(x;\theta)$ is identifiable if $f(x;\theta_{1})=f(x;\theta_{2})$
for all $x$ implies that $\theta_{1}=\theta_{2}.$ Call a density
$f(x;\theta)$ \textit{strongly identifiable} if $f(x;\theta_{1})/f(x;\theta_{2})$
being constant for all $x$ in an open interval $I$ implies that
$\theta_{1}=\theta_{2}$.
\paragraph{Proposition A.}\label{prop:identifiable}

Let $f(x;\theta)$ be a family of densities on $\mathbb{R}$, $-\infty=a_{1}<a_{2}<\ldots<a_{k+1}=\infty$
a sequence of cutoffs, and $f_{[a_{i},a_{i+1})}(x;\theta)$ the density
$f$ truncated to $[a_{i},a_{i+1})$. Let $\lambda_{i},k=1,\ldots k$
be positive numbers satisfying $\sum_{i=1}^{k}\lambda_{i}=1$. Then
the mixture
\[
g(x;\lambda,\theta)=\sum_{i=1}^{k}\lambda_{i}f_{[a_{i},a_{i+1})}(x;\theta)
\]
is identifiable in $(\lambda,\theta)$ if $f(x;\theta)$ is strongly
identifiable in $\theta$.

\paragraph{Proof.}
Assume that $g(x;\lambda_{1},\theta_{1})=g(x;\lambda_{2},\theta_{2})$.
Then $$\lambda_{1i}f_{[a_{i},a_{i+1})}(x;\theta_{1})=\lambda_{2i}f_{[a_{i},a_{i+1})}(x;\theta_{2})$$
for all $i$, thus
\[
\frac{\lambda_{1i}}{\lambda_{2i}}=\frac{f_{[a_{i},a_{i+1})}(x;\theta_{1})}{f_{[a_{i},a_{i+1})}(x;\theta_{2})}.
\]
This implies that $f(x;\theta_{1})/f(x;\theta_{2})$ is constant for
$x\in[a_{i},a_{i+1})$. But since $f(x;\theta)$ is strongly identifiable,
$\theta_{1}=\theta_{2}$, and, consequently, $\lambda_1 = \lambda_2$.

If $f(x;\theta)$ is real analytic and nowhere zero, $f(x;\theta_{1})/f(x;\theta_{2})$
is also real analytic and nowhere zero. By the Identity Theorem \citep[Corollary 1.2.6]{Krantz2002-bt}, if the ratio
$f(x;\theta_{1})/f(x;\theta_{2})$ is constant on some interval $I$,
then $f(x;\theta_{1})/f(x;\theta_{2})$ is constant everywhere, hence
$f(x;\theta_{1})=f(x;\theta_{2})$ everywhere. Thus a family of real
analytic nowhere zero densities is identifiable if and only if it is
strongly identifiable. Every exponential family of densities on the form
\[
f(x;\theta)=h(x)\exp(\eta(\theta)^{T}T(x)-A(\theta))
\]
satisfies this property, provided only that $h$ is nowhere zero real analytic
and $T$ is real analytic. In particular, the normal family satisfies
the properties.

Not every density is strongly identifiable. For instance, mixtures of uniforms are not strongly identifiable. And indeed, Proposition A fails when $f$ is a mixture of uniforms.

% \section*{Web Appendix C}

% \subsubsection*{Graphical representation of selection models}

% It is handy to visualize selection models and their selection sets using directed acyclic graphs. To this end recall that a Bayesian network is a directed acyclic graph $G$ together with a probability density $p$ satisfying the property that $p(x)=\prod_{v\in V(G)}p(x_{v}\mid x_{\textrm{pa}(v)})$, where $V(G)$ is the set of vertices in $G$ and $x_{\textrm{pa}(v)}$ are the parents of $x_{v}$ in $G$ \citep{Pearl2014}. Transforming a Bayesian network for $p$ into a Bayesian network for $q_{H}$ is easy, just add the following to $G$: (i) The selection variable vertex $s$, and (ii) arrows $x$ to $s$ for each $x$ that $s$ depends on. Then
% \begin{equation}
% q_{H}(x\mid s=1)=\frac{p(s=1\mid x_{\textrm{pa}(s)})}{p(s=1\mid H^{c})}\prod_{v\in V(G)}p(x_{v}\mid x_{\textrm{pa}(v)})\label{eq:DAG, selection model}.
% \end{equation}

% To visualize the selection set $H$, start by drawing a dashed plate around the vertices in $H$. In plate notation \citep{buntine1994operations}, a solid plate represents variables that are sampled together. The dashed plate does almost the same, for recall that $H$ contains all the elements that are sampled together until $s=1$. The semantic difference between a dashed and a solid plate is that every sample in a solid plate is observed, but only one of potentially many samples in a dashed plate is observed. The following bare-bones example should make things clear.

% Figure \ref{fig:Plate notation, simple example} displays the directed acyclic graphs of $p$ and the selection models when $H = \emptyset$, $H=\left\{ x,\theta\right\}$, and $H=\left\{ x\right\}$, respectively. The marginal distribution of $\theta$ is not the same for $H=\left\{ x\right\}$ and $H=\left\{ x,\theta\right\} $, as $q_{\left\{ x,\theta\right\} }(\theta) = p(\theta)\frac{p(s=1\mid\theta)}{p(s=1)}$ and $q_{x}(\theta) = \int\frac{p(s=1\mid x)}{p(s=1\mid\theta)}p(x,\theta)dx=p(\theta)$, i.e., it is affected by the selection mechanism $s$.

% \subsubsection*{Example}\label{exa:Marginal density of theta}
% Let $p(x,\theta)=p(x\mid\theta)p(\theta)$ be a density and $s$ be a function of $x$ only, so that $p(s=1\mid x,\theta)=p(s=1\mid x)$.
% The possible selection models are
% \begin{eqnarray*}
% q_{\emptyset}(x,\theta) = q_{\theta}(x,\theta) & = & p(x,\theta)\\
% q_{(x,\theta)}(x,\theta) & = & \frac{p(s=1\mid x)}{p(s=1)}p(x,\theta)\\
% q_{x}(x,\theta) & = & \frac{p(s=1\mid x)}{p(s=1\mid\theta)}p(x,\theta)
% \end{eqnarray*}

% \begin{figure}
% \begin{center}
%  \includegraphics{plots/figure_A.jpg}
% \end{center}
% \caption{\label{fig:Plate notation, simple example} Three simple selection models. {\bf (left)} the original $p(x,\theta)$; {\bf (middle)} the model $q_{\left\{ x,\theta\right\} }$, where $\theta$ and $x$ are
% sampled together until $s=1$; {\bf (right)} the model $q_{x}$, where only $x$ is sampled until $s=1$.}
% \end{figure}

% Let $f(x,\theta\mid\eta)=f(x\mid\theta,\eta)p(\theta)$ be the joint density of a random effects meta-analysis, where $x$ is the effect size, $\theta$ is the study-specific parameter of interest, and $\eta$ is a study-specific nuisance parameter such as the sample size of the study. The left plot of Figure \ref{fig:Plate notation, simple example} is a visualization of $f(x,\theta)$. If we have more than one study to analyse, we will have to work with product density $\prod_{i=1}^{n}f(x_{i},\theta_{i}\mid\eta_{i})$ instead of the stand-alone density $f(x,\theta\mid\eta)$. This is visualised in the middle plot of Figure \ref{fig:Plate notation, simple example}  by drawing a solid plate around the pair $(x,\theta)$. When we are dealing with a fixed effects meta-analysis, in which $\theta$ is fixed, the plate should be drawn around $x$ only (Figure \ref{fig:Plate notation, simple example}, right graph).

% \subsubsection*{Graphical representation of the publication bias and $p$-hacking models}

% To visualize selection models based on \textit{p}-values, we must make some modifications to the original graph: (i) Add the \textit{p}-value node $u$; (ii) Add an arrow from $x$ to $u$; (iii) Since the \textit{p}-value $u$ usually depends on more information than just $x$, such as the standard deviation of $x$, add an arrow from $\eta$ (which represents the extra information) to $u$ as well; (iv) Add the selection node $s$ and an arrow from $u$ to $s$. If $u$ is the only parent of $s$, we are dealing with selection models only based on \textit{p}-values.

% The placement of dashed and solid plates depends on which model we want to use. The idea behind the publication bias model is that a completely new study is done whenever the last one failed to be published. This implies that $\theta$ and $x$ are sampled together. The left plot of Figure \ref{fig:Plate notation, publication bias and p-hacking} shows the direct acyclic graph of the normal publication bias model defined in Proposition 3 in the main text. In this particular case, $\eta$ corresponds to $\sigma$, the standard deviation. Moreover, $u$ is a \textit{p}-value, $\theta_{0}$ is the mean of the effect size distribution, $\tau$ is the standard deviation of the effect size distribution, and $\rho$ is the selection probability function. The variable $Z$ lives on the unit interval, and encodes the editor's decision to publish: If the observed \textit{p}-value is less than $Z$, the study is published. Importantly, $Z$ is placed inside the selection set because a new \textit{p}-value cut-off decision is made for each study received. Since $x$ and $\theta$ are sampled together, the selection mechanism modifies $p(\theta)$.%, as can be seen in the example \ref{exa:Publication bias, theta distribution} here below.

% In the \textit{p}-hacking scenario, the \textit{p}-hacker will hack his study all the way to significance, regardless of $\theta$. This means that $\theta$ and $x$ are sampled separately and $\theta$ must be placed outside the selection set. Moreover, the decision of how much to \textit{p}-hack is not reevaluated at each attempt. Consequently, the random variable that controls the \textit{p}-hacking decisions, analogously to the publication bias model, $Z$, is also placed outside the selection graph. This is the case, for example, of an author who decides to \textit{p}-hack to level $\alpha$ ($Z = \alpha$): he acts on $x$ to obtain the desired \textit{p}-value, whatever the sampled $\theta$ is. The graphical representation of this model is shown in the right plot of Figure \ref{fig:Plate notation, publication bias and p-hacking}. Since $x$ and $\theta$ are not sampled together, the selection mechanism does not modify $p(\theta)$.

% \begin{figure}
% \begin{center}
% \includegraphics{plots/figure_B.jpg}
% \end{center}
% \caption{\label{fig:Plate notation, publication bias and p-hacking} Directed acyclic graphs for: {\bf (left)}
% the publication bias model; {\bf (right)} the \textit{p}-hacking model. The dashed plates enclose the selection sets and the the solid plates enclose variables that are repeated together.}
% \end{figure}

\bibliographystyle{biom}
\bibliography{main.bib}

\end{document}


\documentclass[landscape,twocolumn]{article}
\usepackage{biblatex, booktabs, hyperref, graphicx, mathtools, pgfplots, pgfplotstable}
\addbibresource{references.bib}
\pgfplotsset{compat = 1.16}
\pgfplotstableread[col sep=comma]{../Output/results.csv}\results{}
\pgfplotstableread[col sep=comma]{../Output/Naive Bayes.csv}\naivebayes{}
\pgfplotstableread[col sep=comma]{../Output/Logistic Regression.csv}\logisticregression{}
\pgfplotstableread[col sep=comma]{../Output/SVM.csv}\svm{}
\title{COMP5318 Assignment 1}
\author{Nicholas Grasevski (ngra5777, 500710654)}
\begin{document}
\maketitle
\begin{abstract}
	In this report several image classifiers are benchmarked against a grayscale image dataset. With the help of some feature engineering, a Support Vector Machine with Radial Basis Function kernel is the highest performing classifier with 89.5\% accuracy on the test set.
\end{abstract}

\section{Introduction}
Object recognition is one of the biggest subfields of computer vision, with many practical applications such as computer aided diagnosis~\cite{doi2007computer}, face detection~\cite{hjelmaas2001face}, Optical Character Recognition~\cite{mori1999optical} and so on.

The current state of the art in object recognition is Convolutional Neural Networks~\cite{iandola2016squeezenet}. The convolutional neural network takes in the raw pixel data and applies several layers of `convolution' transformations, which aggregate nearby pixels into higher order features in a hierarchical fashion. Prior to the prevalence of GPU based neural network training, the previous state of the art was to apply some hand crafted transformations to the input image to extract higher order features~\cite{rybski2010visual}, and then pass the encoded features through a classical machine learning algorithm such as a generalized linear model~\cite{ebrahimzadeh2014efficient}.

The aim of this study is to compare and contrast several different machine learning algorithms when applied to an object detection problem. This should provide insight into their relative ability to capture the information in the data, with the help of some standard hand crafted feature transformations.

\section{Methods}
\begin{figure}[ht]
	\includegraphics[width=\linewidth]{../Dataset_image}
	\caption{Grayscale $28 \times 28$ images of clothing.}\label{fig:images}
\end{figure}

Several machine learning algorithms were benchmarked against a corpus of $28 \times 28$ grayscale images. Examples of the images can bee seen in figure~\ref{fig:images}. There are 10 classes in total:
\begin{enumerate}
	\item T-shirt/Top
	\item Trouser
	\item Pullover
	\item Dress
	\item Coat
	\item Sandal
	\item Shirt
	\item Sneaker
	\item Bag
	\item Ankle Boot
\end{enumerate}

30,000 of these images are used for training and hyperparameter tuning using 10-fold cross-validation, and 2,000 are used as a hold-out set to evaluate the best classifier according to the cross validation results.

The performance is evaluated primarily based on top-1 accuracy, as defined in equation~\ref{eq:accuracy}:

\begin{equation}
	\label{eq:accuracy}
	\text{Accuracy} = \frac{\text{Number of correct classifications}}{\text{Total number of test examples used}}
\end{equation}

The training and inference time cost of each algorithm is also evaluated. The experiment was performed on a 16-inch 2019 MackBook Pro, with a 2.3GHz 8-Core Intel Core i9 and 32 GB 2667 MHz DDR4 RAM\@.

\subsection{Preprocessing}
The images were preprocessed by first getting the Histogram of Oriented Gradients descriptor, and then passing the descriptor into a Principal Component Analysis.

\subsubsection{Histogram of Oriented Gradients}
\begin{figure}[ht]
	\includegraphics[width=\linewidth]{../Output/hog}
	\caption{Histogram of Oriented Gradients transformation.}\label{fig:hog}
\end{figure}

The input data was preprocessed using a feature descriptor called `Histogram of Oriented Gradients' (HOG)~\cite{dalal2005histograms}. It performs the following steps:
\begin{description}
	\item[Gamma compression] First, the square root of the raw pixel data is taken to reduce the influence of illumination effects.
	\item[Gradient computation] The horizontal and vertical gradients between adjacent pixels are calculated, resulting in each cell having a 2D gradient vector.
	\item[Orientation binning] The orientations of these vectors are discretized, into 8 bins in our case, corresponding to bins from 0 to 180 degrees.
	\item[Cell histogram] These discretized orientations are then aggregated by `cells', which in our case are $4 \times 4$ pixel areas, and collected in a histogram, with each pixel orientation contributing to the histogram according to its magnitude.
	\item[Descriptor blocks] The cells are then grouped into `blocks', which in our case are $3 \times 3$ cell areas, in a sliding window fashion.
	\item[Block normalization] The cells in each block are normalized against one another using the $L_1$ norm in our case.
	\item[Object detection] Finally, the overlapping normalized blocks are flattened into a feature vector, to be fed into a machine learning algorithm.
\end{description}

The HOG descriptor achieves the following tasks:

\begin{description}
	\item[Normalization] The block descriptor normalization provides better invariance to illumination, shadowing, and edge contrast.
	\item[Feature Extraction] The raw pixel data is distilled to a more meaningful representation of the object edges, as can be seen in figure~\ref{fig:hog}.
\end{description}

\subsubsection{Principal Component Analysis}
The output dimension of the HOG descriptor with the aforementioned parameters is 1800 ($\text{9 cells per block} \times \text{25 blocks} \times \text{8 orientations}$) which is more than double the original flattened dimension ($28 \times 28$). This can cause:

\begin{description}
	\item[Slow training] It is computationally intensive to work with these additional dimensions.
	\item[High variance] Having extra artificial dimensions increases the chances of overfitting.
\end{description}

To mitigate these problems, a Principal Component Analysis (PCA)~\cite{wold1987principal} is performed, wherein the input data is factorized into a set of 3 matrices, known as Singular Value Decomposition~\cite{golub1971singular}, as shown in equation~\ref{eq:svd}:

\begin{equation}
	\label{eq:svd}
	\textbf{X}=\textbf{U}\Sigma\textbf{W}^T
\end{equation}

Where $\textbf{X}$ is the input features, $\textbf{U}$ is an n-by-n matrix, $\Sigma$ is an n-by-p rectangular diagonal matrix of positive numbers $\sigma_{\left(k\right)}$, and $\textbf{W}$ is a p-by-p matrix. We can then truncate the $\textbf{W}$ matrix to include only the $L$ largest singular values and their singular vectors, as in equation~\ref{eq:trunc}:

\begin{equation}
	\label{eq:trunc}
	\textbf{T}_L=\textbf{X}\textbf{W}_L
\end{equation}

From this truncated form, we will take the first 128 principal components of the data, thus resulting in a net reduction of the dimension of the original data by a factor of 6.

\subsection{Algorithms}
\begin{table*}
	\begin{tabular}{ccccc}
		\textbf{Name} & \textbf{Train Time} & \textbf{Train Space} & \textbf{Predict Time} & \textbf{Predict Space} \\\toprule
		Logistic Regression & $\mathcal{O}\left(n^2d\right)$ & $\mathcal{O}\left(nd\right) $ & $\mathcal{O}\left(d\right)$ & $\mathcal{O}\left(d\right)$\\
		Naive Bayes & $\mathcal{O}\left(nd\right)$ & $\mathcal{O}\left(d\right)$ & $\mathcal{O}\left(d\right)$ & $\mathcal{O}\left(d\right)$\\
		SVM & $\mathcal{O}\left(n^3d\right)$ & $\mathcal{O}\left(nd\right)$ & $\mathcal{O}\left(nd\right)$ & $\mathcal{O}\left(nd\right)$\\
	\end{tabular}
	\caption{Computational complexity of different algorithms for $n$ features and $d$ dimensions.}\label{tab:algorithms}
\end{table*}

Several different algorithms are applied in turn to the preprocessed data, and a grid of their hyperparameters are explored. A stratified 10-fold cross validation is performed on each hyperparameter combination and the highest scoring (algorithm, hyperparameter) combination is retrained on the entire training set and evaluated on the test set.

In real world applications, time and memory are not infinite and the machine learning is bound by the physical limitations of the hardware. Table~\ref{tab:algorithms} shows the computational complexity of the various algorithms.

\subsubsection{Logistic Regression}
Logistic Regression is a generalization of linear regression to estimate the probability of a given class~\cite{wright1995logistic}. It models a probability by applying the softmax function (or sigmoid in the binary case) to a linear model, as in equation~\ref{eq:lr}:

\begin{equation}
	\label{eq:lr}
	\Pr\left(Y_i=c\right)=\frac{e^{\beta_c \cdot \textbf{X}_i}}{\sum_h{e^{\beta_h \cdot \textbf{X}_i}}}
\end{equation}

Where $Y_i$ is the label of a given sample, $\textbf{X}_i$ is the input vector, and $\beta_c$ is the weight vector from the fitted model for the given class $c$.

It fits a linear decision boundary such that the logistic loss is minimized, as defined in equation~\ref{eq:logisticregression}:

\begin{equation}
	\label{eq:logisticregression}
	L\left(y,p\right)=-\sum{y\log{p}}
\end{equation}

Where $L$ is the loss, $y$ is the ground truth (as a one hot class vector) and $p$ is the predicted probability (as a softmax vector). It is normally trained using batch gradient descent. Although the training for Logistic Regression is slower and more memory intensive than Naive Bayes, the inference complexity is identical, making it a relatively efficient machine learning algorithm.

\subsubsection{Naive Bayes}
Naive Bayes is a generative model that uses Bayes' Theorem to calculate the probability of a given class given the input variables~\cite{rish2001empirical}. The algorithm assumes that the input variables are conditionally independent, ie $\Pr\left(A,B|Y\right)=\Pr\left(A|Y\right)\Pr\left(B|Y\right)$, yielding a closed-form solution to the probability calculation, as can be seen in equation~\ref{eq:naivebayes}:

\begin{equation}
	\label{eq:naivebayes}
	\Pr\left(C_k|x_1,\ldots,x_n\right)=\frac{1}{Z}\Pr\left(C_k\right)\prod_{i=1}^n{\Pr\left(x_i|C_k\right)}
\end{equation}

For class $C$, where the evidence $Z=\Pr\left(\textbf{X}\right)=\sum_k{\Pr\left(C_k\right)\Pr\left(\textbf{x}|C_k\right)}$ is a scaling factor dependent only on $x_1,\ldots,x_n$, that is, a constant if the values of the feature variables are known.

This simple formulation makes training Naive Bayes very computationally efficient and also easily parallelizable, thanks to the conditional independence assumption. However, the assumption rarely holds in practise, leading to poor performance compared to more sophisticated techniques.

\subsubsection{Support Vector Machine}
Support Vector Machine is a supervised learning algorithm which produces a non-probabilistic classifier, attempting to separate the classes by a set of hyperplanes. It can be used as a linear classifier, or for any nonlinear `kernel' satisfying Mercer's condition~\cite{noble2006support}.

This gives it more predictive power compared to Naive Bayes (which can only predict based on statistically independent variables) and Logistic Regression (which can only predict a linear decision boundary). However the additional power afforded by nonlinearity comes at a cost, as the resultant optimization problem must be solved using a quadratic programming algorithm such as coordinate descent, which typically has cubic time complexity.

Furthermore the inference time can be an order of magnitude slower compared to Naive Bayes and Logistic Regression. More than one weight vector is required, and the number of weight vectors required is proportional to the size of the input data.

\section{Experiments and results}
\begin{figure}
	\begin{tikzpicture}
		\begin{axis}[
			title={Accuracy},
			ybar, nodes near coords,
			legend style={legend pos={outer north east}},
			xlabel={Model},
			xtick=data,
			xticklabels from table={\results}{name},
			x tick label style={rotate=90,anchor=east},
			ylabel={Accuracy (\%)}
			]
			\addplot table[x expr=\coordindex, y expr=100*\thisrow{test_acc}]{\results};
			\addplot table[x expr=\coordindex, y expr=100*\thisrow{cv_acc}]{\results};
			\legend{Test,CV}
		\end{axis}
	\end{tikzpicture}
	\caption{Comparison of accuracy.}\label{fig:accuracy}
\end{figure}

\begin{figure}
	\begin{tikzpicture}
		\begin{axis}[
			title={Time},
			ybar, nodes near coords,
			legend style={legend pos={outer north east}},
			xlabel={Model},
			xtick=data,
			xticklabels from table={\results}{name},
			x tick label style={rotate=90,anchor=east},
			ylabel={Time (s)}
			]
			\addplot table[x expr=\coordindex, y expr=\thisrow{fit_time_seconds}]{\results};
			\addplot table[x expr=\coordindex, y expr=\thisrow{inf_time_seconds}]{\results};
			\legend{Fit,Predict}
		\end{axis}
	\end{tikzpicture}
	\caption{Comparison of time.}\label{fig:time}
\end{figure}

\begin{table*}
	\pgfplotstabletypeset[
	columns={name,test_acc,cv_acc,fit_time_seconds,inf_time_seconds},
	columns/name/.style={string type,column name=Model},
	columns/test_acc/.style={multiply by=100,column name={Test Acc\%}},
	columns/cv_acc/.style={multiply by=100,column name={CV Acc\%}},
	columns/fit_time_seconds/.style={column name={Fit Time (s)}},
	columns/inf_time_seconds/.style={column name={Inf Time (s)}},
	every head row/.style={after row=\toprule},
	]{\results}
	\caption{Comparison of machine learning algorithms.}\label{tab:results}
\end{table*}

As can be seen in table~\ref{tab:results}, there is a tradeoff between training speed and accuracy. Naive Bayes is by far the fastest to train, but does not achieve great accuracy. SVM is the slowest to train, but is highly accurate by comparison. Logistic Regression is slow to train, but notably has similar inference speed to Naive Bayes, due to the same algorithmic complexity during inference.

Although naive bayes has the lowest accuracy, it is still decent. This is testament to the effectiveness of the feature engineering and preprocessing that was done to normalize the data and extract meaningful independent features.

\subsection{Logistic Regression}
\begin{table*}
	\pgfplotstabletypeset[
	columns={param_C,param_penalty,mean_test_score,std_test_score,mean_fit_time,std_fit_time,mean_score_time,std_score_time},
	columns/param_C/.style={column name=C},
	columns/param_penalty/.style={string type,column name=Penalty},
	columns/mean_test_score/.style={multiply by=100,column name={Acc\%}},
	columns/std_test_score/.style={multiply by=100,column name=STD},
	columns/mean_fit_time/.style={column name={Fit Time (s)}},
	columns/std_fit_time/.style={column name=STD},
	columns/mean_score_time/.style={column name={Score Time (s)}},
	columns/std_score_time/.style={column name=STD},
	every head row/.style={after row=\toprule},
	]{\logisticregression}
	\caption{Logistic Regression cross validation results.}\label{tab:logisticregression}
\end{table*}

As per table~\ref{tab:logisticregression}, two parameters were tuned for logistic regression:

\begin{description}
	\item[C] Inverse of regularization strength. A regularization strength of 1 (corresponding to the uniform prior) performed best, making a good balance between bias and variance.
	\item[Penalty] What type of regularization to use. $L_1$ and $L_2$ regularization were both attempted (corresponding to assumptions of laplacian and gaussian noise respectively), with $L_2$ performing consistently better. Perhaps the images naturally had some gaussian noise, allowing the $L_2$ penalty to account for that noise more effectively than the $L_1$ penalty. It is also conceivable that the sparseness enforced by the $L_1$ penalty is undesirable as the input data is continuous numerical data, so taking individual points (pixel groups in this case, due to the HOG transform) and discarding the surrounding context may cause overfitting when generalizing to the test data.
\end{description}

\subsection{Naive Bayes}
\begin{table*}
	\pgfplotstabletypeset[
	columns={mean_test_score,std_test_score,mean_fit_time,std_fit_time,mean_score_time,std_score_time},
	columns/mean_test_score/.style={multiply by=100,column name={Acc\%}},
	columns/std_test_score/.style={multiply by=100,column name=STD},
	columns/mean_fit_time/.style={column name={Fit Time (s)}},
	columns/std_fit_time/.style={column name=STD},
	columns/mean_score_time/.style={column name={Score Time (s)}},
	columns/std_score_time/.style={column name=STD},
	every head row/.style={after row=\toprule},
	]{\naivebayes}
	\caption{Naive Bayes cross validation results.}\label{tab:naivebayes}
\end{table*}

Being a simple algorithm, no hyperparameters were tuned for Naive Bayes. As per table~\ref{tab:naivebayes}, Both training and inference time were infinitesimal, thanks to the linear time complexity of the algorithm.

\subsection{Support Vector Machine}
\begin{table*}
	\pgfplotstabletypeset[
	columns={param_C,param_kernel,mean_test_score,std_test_score,mean_fit_time,std_fit_time,mean_score_time,std_score_time},
	columns/param_C/.style={column name=C},
	columns/param_kernel/.style={string type,column name=Kernel},
	columns/mean_test_score/.style={multiply by=100,column name={Acc\%}},
	columns/std_test_score/.style={multiply by=100,column name=STD},
	columns/mean_fit_time/.style={column name={Fit Time (s)}},
	columns/std_fit_time/.style={column name=STD},
	columns/mean_score_time/.style={column name={Score Time (s)}},
	columns/std_score_time/.style={column name=STD},
	every head row/.style={after row=\toprule},
	]{\svm}
	\caption{SVM cross validation results.}\label{tab:svm}
\end{table*}

As per table~\ref{tab:svm}, two main hyperparameters were tested for SVM:\@

\begin{description}
	\item[C] Inverse of regularization strength. In this case, a regularization strength of 10 performed marginally better than other settings.
	\item[Kernel] Which kernel type to use. The Radial Basis Function performed the best. This makes sense because intuitively it is grouping together images which have edges in similar locations, whereas the linear kernel (ie no kernel) would mostly classify based on the overall contrast of each group of pixels in the image, with no consideration for the proximity of said groups of pixels. The polynomial kernel also performed well, the default setting is degree 3, perhaps degree 2 could provide better results. The sigmoid kernel performed worst.
\end{description}

\section{Conclusion}
Out of the algorithms applied, Support Vector Machine achieved the highest accuracy at the expense of a relatively long training time. Naive Bayes was by far the fastest in both training and inference, due to the efficient time complexity of the algorithm. Further improvements would be to consider convolutional neural networks in order to automatically extract similar features to the HOG descriptor via backpropagation and stochastic gradient descent. Furthermore, GPU acceleration could be utilized to achieve fast training and inference by parallelizing the pixel-wise operations which are very common in computer vision.

\printbibliography\appendix
\section{Running the code}
\begin{description}
	\item[Dependencies] \texttt{pip3 install h5py pandas scikit-image scikit-learn}
	\item[Usage] Run the steps in the notebook sequentially to get the tuning and evaluation results.
\end{description}

Detailed cross validation results will be written as csv files to \texttt{Output} for each respective classifier, along with the final predictions as \texttt{Output/predicted\_labels.h5}. Note that the \texttt{ALGORITHMS} parameter can be modified at will to change the grid search process, for example to speed up the runtime of the script by excluding certain algorithms or hyperparameter combinations.

\end{document}


    \foldertitle{sstatelang}{Steady-state file language}{sstatelang/Contents}

	Steady-state (sstate) files are a complementary tool to solve and
 analyse the steady states of more complex models. They allow you to
 create steady-state files independent of the original model files, and
 write the steady-state equations in different ways, manipulate their
 structure or order, split the problem into subblocks, and combine
 numerical and symbolic solutions. Using then the
 \href{sstatelang/Contents}{sstate objects}, you can compile and run
 stand-alone steady-state m-file functions based on your steady-state
 files.

 \paragraph{Input parameters}

 \begin{itemize}
 \item
   \href{sstatelang/input}{\texttt{!input}} - List of input parameters or
   variables.
 \item
   \href{sstatelang/growthnames}{\texttt{!growthnames}} - Pattern for
   creating growth names.
 \end{itemize}

 \paragraph{Equations and assignments}

 \begin{itemize}
 \item
   \href{sstatelang/equations}{\texttt{!equations}} - Block of equations
   or assignments.
 \item
   \href{sstatelang/growthnames}{\texttt{!growthnames2imag}} - Pattern
   for creating growth names.
 \item
   \href{sstatelang/solvefor}{\texttt{!solvefor}} - List of variables for
   which the current equations block will be solved.
 \item
   \href{sstatelang/symbolic}{\texttt{!symbolic}} - Attempt to solve the
   current equations block symbolically using the Symbolic Maths Toolbox.
 \end{itemize}

 \paragraph{Variables with steady state restricted to be positive}

 \begin{itemize}
 \item
   \href{sstatelang/logvariables}{\texttt{!log\_variables}} - Restrict
   the steady state of some of the variables to be positive.
 \item
   \href{sstatelang/allbut}{\texttt{!all\_but}} - Inverse list of
   variables with positive steady states.
 \end{itemize}

 \paragraph{Getting on-line help on sstate file language}

 \begin{verbatim}
 help sstatelang
 help sstatelang/keyword
 \end{verbatim}


% Copyright 2021 UChicago Argonne, LLC
% Author:
% - Haoyu Wang and Roberto Ponciroli, Argonne National Laboratory
% - Andrea Alfonsi, Idaho National Laboratory

% Licensed under the Apache License, Version 2.0 (the "License");
% you may not use this file except in compliance with the License.
% You may obtain a copy of the License at

%   https://www.apache.org/licenses/LICENSE-2.0.txt

% Unless required by applicable law or agreed to in writing, software
% distributed under the License is distributed on an "AS IS" BASIS,
% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
% See the License for the specific language governing permissions and
% limitations under the License.

\section{Input of FARM for RAVEN}

In this section, the external matrices file structure, FARM model definition, and input/output file example will be
discussed. In order to run FARM, two XML files are required:
\begin{itemize}
  \item External Matrices File (See Section \ref{ExtMatFileStructure}), and
  \item Input File (See Section \ref{InputFile})
\end{itemize}
Currently two version of Reference Governor SIMO are provided: paramterized version, and unparameterized version.
\begin{itemize}
  \item Parameterized Version: Multiple state-space representations profiles are provided in the external matrices file,
  labeled by a parameter "ActuatorParameter". The FARM will select the closest profile based on the value of input
  variable "PwrSet".
  \item Unparameterized Version: Single state-space representation profile is provided, and will be used in all the time.
\end{itemize}

\subsection{External Matrices File Structure}
\label{ExtMatFileStructure}
As discussed in \ref{RG_SIMO}, the state space matrices \begin{math} \textbf{A} \end{math},
\begin{math} \textbf{B} \end{math}, \begin{math} \textbf{C} \end{math} are required for the FARM to run.
In addition, the norminal values of
\begin{math} u \end{math},
\begin{math} \overrightarrow{x} \end{math},
\begin{math} \overrightarrow{y} \end{math} needs to be known for the following equations at
\begin{math}k=0 \end{math} to be satisfied:
\begin{itemize}
  \item \begin{math} \overrightarrow{x}[k+1]=\textbf{A}*\overrightarrow{0}+\textbf{B}*0 \end{math}
  \item \begin{math} \overrightarrow{0}=\textbf{C}*\overrightarrow{0} \end{math}
\end{itemize}
In addition, the latest value of \begin{math} \overrightarrow{x} \end{math} needs to be known as the current system state
if no other values are provided in the Input File.\newline

\subsubsection{Parameterized External Matrices File}
\label{ParaExtMatFile}
The following Listing \ref{lst:ParaExtXMLExample} is a example external matrices file for Parameterized Reference
Governor with necessary information:

\begin{lstlisting}[style=XML,morekeywords={anAttribute},
  caption=Parameterized External Matrices File Example., label=lst:ParaExtXMLExample]
<DataObjectMetadata name="rom_stats">
  <DMDrom type="Static">
    <DMDcModel>
      <dmdTimeScale>1800 1810
      <UNorm>
        <realization ActuatorParameter="-1.15e+01" sample="0">-4.68e-02</realization>
        <realization ActuatorParameter="-3.45e+01" sample="1">-4.30e-01</realization>
      </UNorm>
      <XNorm>
        <realization ActuatorParameter="-1.15e+01" sample="0">2.55e+01 0.00e+00</realization>
        <realization ActuatorParameter="-3.45e+01" sample="1">2.55e+01 0.00e+00</realization>
      </XNorm>
      <YNorm>
        <realization ActuatorParameter="-1.15e+01" sample="0">3.20e+01 2.55e+01</realization>
        <realization ActuatorParameter="-3.45e+01" sample="1">3.20e+01 2.55e+01</realization>
      </YNorm>
      <XLast>
        <realization ActuatorParameter="-1.15e+01" sample="0">2.44e+01 4.12e+01</realization>
        <realization ActuatorParameter="-3.45e+01" sample="1">2.19e+01 1.38e+02</realization>
      </XLast>
      <Atilde>
        <realization ActuatorParameter="-1.15e+01" sample="0">
          <imaginary>0.00e+00 0.00e+00 0.00e+00 0.00e+00</imaginary>
          <matrixShape>2,2</matrixShape>
          <real>1.00e+00 -1.18e-01 -6.25e-05 4.14e-03</real>
        </realization>
        <realization ActuatorParameter="-3.45e+01" sample="1">
          <imaginary>0.00e+00 0.00e+00 0.00e+00 0.00e+00</imaginary>
          <matrixShape>2,2</matrixShape>
          <real>1.00e+00 -5.61e-02 -6.90e-05 1.32e-01</real>
        </realization>
      </Atilde>
      <Btilde>
        <realization ActuatorParameter="-1.15e+01" sample="0">
          <imaginary>0.00e+00 0.00e+00</imaginary>
          <matrixShape>2,1</matrixShape>
          <real>-2.01e-04 -4.21e+00</real>
        </realization>
        <realization ActuatorParameter="-3.45e+01" sample="1">
          <imaginary>0.00e+00 0.00e+00</imaginary>
          <matrixShape>2,1</matrixShape>
          <real>-2.31e-04 -3.71e+00</real>
        </realization>
      </Btilde>
      <Ctilde>
        <realization ActuatorParameter="-1.15e+01" sample="0">
          <imaginary>0.00e+00 0.00e+00 0.00e+00 0.00e+00</imaginary>
          <matrixShape>2,2</matrixShape>
          <real>-1.00e+00 1.00e+00 -3.48e-10 1.10e-16</real>
        </realization>
        <realization ActuatorParameter="-3.45e+01" sample="1">
          <imaginary>0.00e+00 0.00e+00 0.00e+00 0.00e+00</imaginary>
          <matrixShape>2,2</matrixShape>
          <real>-1.00e+00 1.00e+00 -5.85e-10 7.81e-18</real>
        </realization>
      </Ctilde>
    </DMDcModel>
  </DMDrom>
</DataObjectMetadata>
\end{lstlisting}

As one can see, all the required information for FARM are included in the \xmlNode{DMDcModel} block.
Indeed, the name of this block and its parent blocks is flexible:
\begin{itemize}
  \item \xmlNode{DataObjectMetadata}, Root element of XML file, other names are acceptable.
  \item \xmlNode{DMDrom}, Child 1 level element of XML file, other names are acceptable.
  \item \xmlNode{DMDcModel}, Child 2 level element of XML file, other names are acceptable.
\end{itemize}

As long as the following keywords and inputs are the Child 3 level elements of this XML file, the information can
be processed for FARM execution:
\begin{itemize}
  \item \xmlNode{dmdTimeScale}, \xmlDesc{float vector, required parameter},
  the time step marks used in DMDc calculation.
  At least two consecutive time step value are required to calculate the discrete time interval;
\end{itemize}

Some items (\xmlNode{UNorm}, \xmlNode{XNorm}, \xmlNode{YNorm}, \xmlNode{XLast}, \xmlNode{Atilde}, \xmlNode{Btilde},
\xmlNode{Ctilde}) contains multiple \xmlNode{realization} nodes, in the format of
\xmlNode{realization ActuatorParameter="some float value" sample="some integer value"}, and the corresponding
values are indexed by this ActuatorParameter.
\begin{itemize}
  \item
  \begin{itemize}
    \item The \xmlNode{ActuatorParameter} attribute share the same unit with the input variable "PwrSet", and the
    parameterized Reference Governor SIMO will select the realization whose \xmlNode{ActuatorParameter} value is
    closest to the input variable "PwrSet";
    \item The \xmlNode{sample} attribute contains a consecutive integer value starting from 0.
  \end{itemize}
\end{itemize}
All the following items should share the same list of \xmlNode{ActuatorParameter} attribute:
\begin{itemize}
  \item \xmlNode{UNorm}, \xmlDesc{float scalar, required parameter},
  the norminal value of system actuation variable \begin{math} u \end{math};
  \item \xmlNode{XNorm}, \xmlDesc{float vector, required parameter},
  the norminal value of system state vector \begin{math} \overrightarrow{x}\in\mathbb{R}^n \end{math};
  \item \xmlNode{YNorm}, \xmlDesc{float vector, required parameter},
  the norminal value of system output vector \begin{math} \overrightarrow{y}\in\mathbb{R}^p \end{math};
  \begin{itemize}
    \item Note: the \xmlNode{YNorm} must have at least realization that is within the range defined by
    "Min\_Target" and "Max\_Target" in the definition of Reference Governor SIMO EnternalModel.
    See Listing \ref{lst:Para_RGSIMOExample} in Section \ref{ParaInputFile}.
  \end{itemize}
  \item \xmlNode{XLast}, \xmlDesc{float vector, required parameter},
  the latest value of system state vector \begin{math} \overrightarrow{x}\in\mathbb{R}^n \end{math},
  \textbf{without norminal value subtraction};

  \item \xmlNode{Atilde}, \xmlDesc{complex matrix, required parameter},
  the state matrix \begin{math} \textbf{A}\in\mathbb{C}^{n \times n} \end{math}:
  \begin{itemize}
    \item \xmlNode{imaginary}, \xmlDesc{float matrix, optional parameter},
    the imaginary part of state matrix \begin{math} \textbf{A} \end{math}. The matrix are flattened
    column-by-column, i.e. the order is

    \begin{math} A_{11} A_{21} ... A_{n1} \end{math} \begin{math}A_{12} A_{22} ... A_{n2} ... \end{math}
    \item \xmlNode{matrixShape}, \xmlDesc{int vector, required parameter},
    the shape of matrix, defined as \xmlNode{matrixShape}Number of Rows,Number of Columns\xmlNode{/matrixShape}
    \item \xmlNode{real}, \xmlDesc{float matrix, required parameter},
    the real part of state matrix \begin{math} \textbf{A} \end{math}. The matrix are flattened
    column-by-column, i.e. the order is

    \begin{math} A_{11} A_{21} ... A_{n1} \end{math} \begin{math}A_{12} A_{22} ... A_{n2} ... \end{math}
  \end{itemize}

  \item \xmlNode{Btilde}, \xmlDesc{complex matrix, required parameter},
  the input matrix \begin{math} \textbf{B}\in\mathbb{C}^{n \times 1} \end{math}:
  \begin{itemize}
    \item \xmlNode{real}, \xmlDesc{float matrix, required parameter},
    the real part of input matrix \begin{math} \textbf{B} \end{math}. The matrix are flattened
    column-by-column, i.e. the order is

    \begin{math} B_{11} B_{21} ... B_{n1} \end{math} \begin{math}B_{12} B_{22} ... B_{n2} ... \end{math}
    \item \xmlNode{matrixShape}, \xmlDesc{int vector, required parameter},
    the shape of matrix, defined as \xmlNode{matrixShape}Number of Rows,Number of Columns\xmlNode{/matrixShape}
    \item \xmlNode{imaginary}, \xmlDesc{float matrix, optional parameter},
    the imaginary part of input matrix \begin{math} \textbf{B} \end{math}. The matrix are flattened
    column-by-column, i.e. the order is

    \begin{math} B_{11} B_{21} ... B_{n1} \end{math} \begin{math}B_{12} B_{22} ... B_{n2} ... \end{math}
  \end{itemize}

  \item \xmlNode{Ctilde}, \xmlDesc{complex matrix, required parameter},
  the output matrix \begin{math} \textbf{C}\in\mathbb{C}^{p \times n} \end{math}:
  \begin{itemize}
    \item \xmlNode{real}, \xmlDesc{float matrix, required parameter},
    the real part of output matrix \begin{math} \textbf{C} \end{math}. The matrix are flattened
    column-by-column, i.e. the order is

    \begin{math} C_{11} C_{21} ... C_{n1} \end{math} \begin{math}C_{12} C_{22} ... C_{n2} ... \end{math}
    \item \xmlNode{matrixShape}, \xmlDesc{int vector, required parameter},
    the shape of matrix, defined as \xmlNode{matrixShape}Number of Rows,Number of Columns\xmlNode{/matrixShape}
    \item \xmlNode{imaginary}, \xmlDesc{float matrix, optional parameter},
    the imaginary part of output matrix \begin{math} \textbf{C} \end{math}. The matrix are flattened
    column-by-column, i.e. the order is

    \begin{math} C_{11} C_{21} ... C_{n1} \end{math} \begin{math}C_{12} C_{22} ... C_{n2} ... \end{math}
  \end{itemize}

\end{itemize}

This external matrices file example can be found in the following directory, which is automatically generated by a
Dynamic Mode Decomposition with Control (DMDC)(\cite{proctor2016dynamic} and \cite{wang2020DMDc}) post processor,
and contains some information that is not necessary for FARM:
\begin{itemize}
  \item /raven/plugins/FARM/tests/RefGov\_para\_xmlABC\_Test/DMDcCxCoeff\_TES\_para.xml
\end{itemize}

This external matrices file represents a thermal energy storage (TES) model, with:
\begin{itemize}
  \item One(1) system actuation variable \begin{math} u \end{math}
  \begin{itemize}
    \item Power set point, measured in Mega Watts (MW);
  \end{itemize}
  \item Two(2)-element system state vector \begin{math} \overrightarrow{x}\in\mathbb{R}^2 \end{math}
  \item Two(2)-element system output vector \begin{math} \overrightarrow{y}\in\mathbb{R}^2 \end{math}
  \begin{itemize}
    \item \begin{math} y_1 \end{math}, Hot fluid tank height, measured in Meter (m);
    \item \begin{math} y_2 \end{math}, Cold fluid tank height, measured in Meter (m);
  \end{itemize}
\end{itemize}

In addition, the state-space representations are parameterized by 20 charging(-)/discharging(+) power levels,
and the values in \xmlNode{ActuatorParameter} attribute are measured in Mega Watts (MW).

\subsubsection{Unparameterized External Matrices File}
\label{UnparaExtMatFile}
The following Listing \ref{lst:UnparaExtXMLExample} is a example external matrices file for Unparameterized Reference
Governor with necessary information:

\begin{lstlisting}[style=XML,morekeywords={anAttribute},
  caption=Unparameterized External Matrices File Example., label=lst:UnparaExtXMLExample]
<DataObjectMetadata name="rom_stats">
  <DMDrom type="Static">
    <DMDcModel>
      <dmdTimeScale>1800 1810
      <UNorm>
        <realization sample="0">-3.50e+00</realization>
      </UNorm>
      <XNorm>
        <realization sample="0">2.55e+01 0.00e+00</realization>
      </XNorm>
      <XLast>
        <realization sample="0">2.09e+00 9.15e+02</realization>
      </XLast>
      <YNorm>
        <realization sample="0">3.20e+01 2.55e+01</realization>
      </YNorm>
      <Atilde>
        <realization sample="0">
          <imaginary>0.00e+00 0.00e+00 0.00e+00 0.00e+00</imaginary>
          <matrixShape>2,2</matrixShape>
          <real>1.00e+00 -1.33e-02 -5.56e-05 -4.57e-01</real>
        </realization>
      </Atilde>
      <Btilde>
        <realization sample="0">
          <imaginary>0.00e+00 0.00e+00</imaginary>
          <matrixShape>2,1</matrixShape>
          <real>-1.75e-04 -6.25e+00</real>
        </realization>
      </Btilde>
      <Ctilde>
        <realization sample="0">
          <imaginary>0.00e+00 0.00e+00 0.00e+00 0.00e+00</imaginary>
          <matrixShape>2,2</matrixShape>
          <real>-1.00e+00 1.00e+00 -3.76e-11 -4.86e-17</real>
        </realization>
      </Ctilde>
    </DMDcModel>
  </DMDrom>
</DataObjectMetadata>
\end{lstlisting}

As one can see, all the required information for FARM are included in the \xmlNode{DMDcModel} block.
Indeed, the name of this block and its parent blocks is flexible:
\begin{itemize}
  \item \xmlNode{DataObjectMetadata}, Root element of XML file, other names are acceptable.
  \item \xmlNode{DMDrom}, Child 1 level element of XML file, other names are acceptable.
  \item \xmlNode{DMDcModel}, Child 2 level element of XML file, other names are acceptable.
\end{itemize}

As long as the following keywords and inputs are the Child 3 level elements of this XML file, the information can
be processed for FARM execution:
\begin{itemize}
  \item \xmlNode{dmdTimeScale}, \xmlDesc{float vector, required parameter},
  the time step marks used in DMDc calculation.
  At least two consecutive time step value are required to calculate the discrete time interval;
\end{itemize}

Some items (\xmlNode{UNorm}, \xmlNode{XNorm}, \xmlNode{YNorm}, \xmlNode{XLast}, \xmlNode{Atilde}, \xmlNode{Btilde},
\xmlNode{Ctilde}) contains one \xmlNode{realization} nodes, in the format of
\xmlNode{realization sample="0"}, and the corresponding value are contained in the \xmlNode{realization} node.

All the following items have the \xmlNode{realization sample="0"} node:
\begin{itemize}
  \item \xmlNode{UNorm}, \xmlDesc{float scalar, required parameter},
  the norminal value of system actuation variable \begin{math} u \end{math};
  \item \xmlNode{XNorm}, \xmlDesc{float vector, required parameter},
  the norminal value of system state vector \begin{math} \overrightarrow{x}\in\mathbb{R}^n \end{math};
  \item \xmlNode{YNorm}, \xmlDesc{float vector, required parameter},
  the norminal value of system output vector \begin{math} \overrightarrow{y}\in\mathbb{R}^p \end{math};
  \begin{itemize}
    \item Note: the \xmlNode{YNorm} values must be within the range defined by "Min\_Target" and "Max\_Target" in the
    definition of Reference Governor SIMO EnternalModel. See Listing \ref{lst:Unpara_RGSIMOExample} in Section
    \ref{UnParaInputFile}.
  \end{itemize}
  \item \xmlNode{XLast}, \xmlDesc{float vector, required parameter},
  the latest value of system state vector \begin{math} \overrightarrow{x}\in\mathbb{R}^n \end{math},
  \textbf{without norminal value subtraction};

  \item \xmlNode{Atilde}, \xmlDesc{complex matrix, required parameter},
  the state matrix \begin{math} \textbf{A}\in\mathbb{C}^{n \times n} \end{math}:
  \begin{itemize}
    \item \xmlNode{imaginary}, \xmlDesc{float matrix, optional parameter},
    the imaginary part of state matrix \begin{math} \textbf{A} \end{math}. The matrix are flattened
    column-by-column, i.e. the order is

    \begin{math} A_{11} A_{21} ... A_{n1} \end{math} \begin{math}A_{12} A_{22} ... A_{n2} ... \end{math}
    \item \xmlNode{matrixShape}, \xmlDesc{int vector, required parameter},
    the shape of matrix, defined as \xmlNode{matrixShape}Number of Rows,Number of Columns\xmlNode{/matrixShape}
    \item \xmlNode{real}, \xmlDesc{float matrix, required parameter},
    the real part of state matrix \begin{math} \textbf{A} \end{math}. The matrix are flattened
    column-by-column, i.e. the order is

    \begin{math} A_{11} A_{21} ... A_{n1} \end{math} \begin{math}A_{12} A_{22} ... A_{n2} ... \end{math}
  \end{itemize}

  \item \xmlNode{Btilde}, \xmlDesc{complex matrix, required parameter},
  the input matrix \begin{math} \textbf{B}\in\mathbb{C}^{n \times 1} \end{math}:
  \begin{itemize}
    \item \xmlNode{real}, \xmlDesc{float matrix, required parameter},
    the real part of input matrix \begin{math} \textbf{B} \end{math}. The matrix are flattened
    column-by-column, i.e. the order is

    \begin{math} B_{11} B_{21} ... B_{n1} \end{math} \begin{math}B_{12} B_{22} ... B_{n2} ... \end{math}
    \item \xmlNode{matrixShape}, \xmlDesc{int vector, required parameter},
    the shape of matrix, defined as \xmlNode{matrixShape}Number of Rows,Number of Columns\xmlNode{/matrixShape}
    \item \xmlNode{imaginary}, \xmlDesc{float matrix, optional parameter},
    the imaginary part of input matrix \begin{math} \textbf{B} \end{math}. The matrix are flattened
    column-by-column, i.e. the order is

    \begin{math} B_{11} B_{21} ... B_{n1} \end{math} \begin{math}B_{12} B_{22} ... B_{n2} ... \end{math}
  \end{itemize}

  \item \xmlNode{Ctilde}, \xmlDesc{complex matrix, required parameter},
  the output matrix \begin{math} \textbf{C}\in\mathbb{C}^{p \times n} \end{math}:
  \begin{itemize}
    \item \xmlNode{real}, \xmlDesc{float matrix, required parameter},
    the real part of output matrix \begin{math} \textbf{C} \end{math}. The matrix are flattened
    column-by-column, i.e. the order is

    \begin{math} C_{11} C_{21} ... C_{n1} \end{math} \begin{math}C_{12} C_{22} ... C_{n2} ... \end{math}
    \item \xmlNode{matrixShape}, \xmlDesc{int vector, required parameter},
    the shape of matrix, defined as \xmlNode{matrixShape}Number of Rows,Number of Columns\xmlNode{/matrixShape}
    \item \xmlNode{imaginary}, \xmlDesc{float matrix, optional parameter},
    the imaginary part of output matrix \begin{math} \textbf{C} \end{math}. The matrix are flattened
    column-by-column, i.e. the order is

    \begin{math} C_{11} C_{21} ... C_{n1} \end{math} \begin{math}C_{12} C_{22} ... C_{n2} ... \end{math}
  \end{itemize}

\end{itemize}

This external matrices file example can be found in the following directory, which is automatically generated by a
Dynamic Mode Decomposition with Control (DMDC)(\cite{proctor2016dynamic} and \cite{wang2020DMDc}) post processor,
and contains some information that is not necessary for FARM:
\begin{itemize}
  \item /raven/plugins/FARM/tests/RefGov\_unpara\_xmlABC\_Test/DMDcCxCoeff\_TES\_unpara.xml
\end{itemize}

This external matrices file represents a thermal energy storage (TES) model, with:
\begin{itemize}
  \item One(1) system actuation variable \begin{math} u \end{math}
  \begin{itemize}
    \item Charging(-)/Discharging(+) Power setpoint, measured in Mega Watts (MW);
  \end{itemize}
  \item Two(2)-element system state vector \begin{math} \overrightarrow{x}\in\mathbb{R}^2 \end{math}
  \item Two(2)-element system output vector \begin{math} \overrightarrow{y}\in\mathbb{R}^2 \end{math}
  \begin{itemize}
    \item \begin{math} y_1 \end{math}, Hot fluid tank height, measured in Meter (m);
    \item \begin{math} y_2 \end{math}, Cold fluid tank height, measured in Meter (m);
  \end{itemize}
\end{itemize}

The state-space representation listed in this external matrices file is linearized at a charging power of 218.5 MW.

\subsection{Input of Reference Governor SIMO ExternalModel}
\label{InputFile}
The input of Reference Governor SIMO is an XML file.

\subsubsection{Input File for Parameterized Reference Governor SIMO ExternalModel}
\label{ParaInputFile}
An example of the input structure is given in Listing \ref{lst:Para_RGSIMOExample}.
The following section will discuss the different keywords in the input and describe how they are used in this FARM plugin.

\begin{lstlisting}[style=XML,morekeywords={anAttribute},
caption=Reference Governor SIMO ExternalModel Example., label=lst:Para_RGSIMOExample]
  <ExternalModel name="RG1" subType="FARM.RefGov_parameterized_SIMO">
    <outputVariables>V, V_min, V_max</outputVariables>
    <variables>PwrSet, V, V_min, V_max</variables>
    <constant varName="MOASsteps"> 360 </constant>
    <constant varName="Min_Target1"> 2.5 </constant>
    <constant varName="Max_Target1"> 55. </constant>
    <constant varName="Min_Target2"> 2.5 </constant>
    <constant varName="Max_Target2"> 55. </constant>
    <constant varName="Sys_State_x"> 30., 0 </constant>
  </ExternalModel>
\end{lstlisting}

As one can see, all the specifications of the Reference Governor SIMO plugin are given in the \xmlNode{ExternalModel}
block.
\begin{itemize}
  \item The \xmlNode{subType} attribute should be set as "FARM.RefGov\_parameterized\_SIMO" to call the parameterized
  Reference Governor SIMO external model.
\end{itemize}

Inside the \xmlNode{ExternalModel} block, the XML nodes that belong to this plugin only (and not to the ExternalModel)
are:
\begin{itemize}
  \item \xmlNode{outputVariables}, \xmlDesc{3-element string, required parameter}, the names of output variables:
  \begin{itemize}
    \item Adjusted actuation variable \begin{math} u \end{math},
    any user-defined name is allowed(e.g. V);
    \item Lower limit of adjusted actuation variable \begin{math} u \end{math},
    any user-defined name is allowed (e.g. V\_min);
    \item Upper limit of adjusted actuation variable \begin{math} u \end{math},
    any user-defined name is allowed (e.g. V\_max);
  \end{itemize}
  The order of the 3 variables should be strictly followed.

  \item \xmlNode{variables}, \xmlDesc{4-element string, required parameter}, the names of:
  \begin{itemize}
    \item Original actuation variable \begin{math} r \end{math},
    any user-defined name is allowed (e.g. PwrSet);
    \item The 3 output variables in \xmlNode{outputVariables}
  \end{itemize}
  The order of the 4 variables should be strictly followed.

  \item \xmlNode{constant varName="MOASsteps"}, \xmlDesc{integer, required parameter},
  the $\textbf{g}$ value of steps to predict in the future (e.g. 360);

  \item And 2*p blocks of \xmlNode{constant} regarding the constraints on system output vector
  \begin{math} \overrightarrow{y}\in\mathbb{R}^p \end{math}
  \begin{itemize}
    \item \xmlNode{constant varName="Min\_Target1"}, \xmlDesc{float, required parameter},
    the lower constraint on output \begin{math} y_1 \end{math} (e.g. 2.5);
    \item \xmlNode{constant varName="Max\_Target1"}, \xmlDesc{float, required parameter},
    the upper constraint on output \begin{math} y_1 \end{math} (e.g. 50.);
    \item \xmlNode{constant varName="Min\_Target2"}, \xmlDesc{float, required parameter},
    the lower constraint on output \begin{math} y_2 \end{math} (e.g. 2.5);
    \item \xmlNode{constant varName="Max\_Target2"}, \xmlDesc{float, required parameter},
    the upper constraint on output \begin{math} y_2 \end{math} (e.g. 50.);
    \item ...
    \item \xmlNode{constant varName="Min\_Targetp"}, \xmlDesc{float, required parameter},
    the lower constraint on output \begin{math} y_p \end{math} (not shown in Listing
    \ref{lst:Para_RGSIMOExample} due to $p=2$);
    \item \xmlNode{constant varName="Max\_Targetp"}, \xmlDesc{float, required parameter},
    the upper constraint on output \begin{math} y_p \end{math} (not shown in Listing
    \ref{lst:Para_RGSIMOExample} due to $p=2$);
  \end{itemize}
  \item \xmlNode{constant varName="Sys\_State\_x"}, \xmlDesc{float vector, optional parameter},
  the current system state vector \begin{math} x \end{math} (e.g. 30., 0) before subtracting
  the value in \xmlNode{XNorm} in the External Matrice File (See Section \ref{ParaExtMatFile}).
  If not supplied, the FARM will automatically use the the value in \xmlNode{XLast} in the
  External Matrice File (See Section \ref{ParaExtMatFile}).
\end{itemize}
This external model definition example can be found in the following directory:
\begin{itemize}
  \item /raven/plugins/FARM/tests/test\_RefGov\_para\_xmlABC.xml
\end{itemize}

and it is designed for a thermal energy storage (TES) simulation, with:
\begin{itemize}
  \item One(1) input variable:
  \begin{itemize}
    \item PwrSet, Charging(-)/Discharging(+) Power setpoint, measured in Mega Watts (MW);
  \end{itemize}
  \item Three(3) output variables:
  \begin{itemize}
    \item V, Adjusted Charging(-)/Discharging(+) Power setpoint, measured in Mega Watts (MW);
    \item V\_min, Admissible power setpoint lower limit, measured in Mega Watts (MW);
    \item V\_max, Admissible power setpoint upper limit, measured in Mega Watts (MW);
  \end{itemize}
  \item MOASsteps \begin{math} g=360 \end{math}, will project for the next 360 discrete time steps;
  \item Two(2) pairs of system output constraints:
  \begin{itemize}
    \item \begin{math} y_1 \end{math}, Hot fluid tank height, measured in Meter (m);
    \item \begin{math} y_2 \end{math}, Cold fluid tank height, measured in Meter (m);
  \end{itemize}
\end{itemize}

\subsubsection{Input File for Unparameterized Reference Governor SIMO ExternalModel}
\label{UnparaInputFile}
An example of the input structure is given in Listing \ref{lst:Unpara_RGSIMOExample}.
The following section will discuss the different keywords in the input and describe how they are used in this FARM plugin.

\begin{lstlisting}[style=XML,morekeywords={anAttribute},
caption=Reference Governor SIMO ExternalModel Example., label=lst:Unpara_RGSIMOExample]
  <ExternalModel name="RG1" subType="FARM.RefGov_unparameterized_SIMO">
    <outputVariables>V, V_min, V_max</outputVariables>
    <variables>PwrSet, V, V_min, V_max</variables>
    <constant varName="MOASsteps"> 360 </constant>
    <constant varName="Min_Target1"> 2.5 </constant>
    <constant varName="Max_Target1"> 55. </constant>
    <constant varName="Min_Target2"> 2.5 </constant>
    <constant varName="Max_Target2"> 55. </constant>
    <constant varName="Sys_State_x"> 30., 0 </constant>
  </ExternalModel>
\end{lstlisting}

Being similar to Listing \ref{lst:Para_RGSIMOExample}, all the specifications of the Reference Governor SIMO plugin
are given in the \xmlNode{ExternalModel} block. The only difference is
\begin{itemize}
  \item The \xmlNode{subType} attribute should be set as "FARM.RefGov\_unparameterized\_SIMO" to call the unparameterized
  Reference Governor SIMO external model.
\end{itemize}

This external model definition example can be found in the following directory:
\begin{itemize}
  \item /raven/plugins/FARM/tests/test\_RefGov\_unpara\_xmlABC.xml
\end{itemize}

and it is designed for the same thermal energy storage (TES) simulation described in Section \ref{ParaInputFile}.

\subsection{Input File Example with FARM.RefGov\_parameterized\_SIMO ExternalModel}

A complete input file example using the external model "FARM.RefGov\_parameterized\_SIMO" can be found in the
following directory, and it will be discussed in this section:
\begin{itemize}
  \item /raven/plugins/FARM/tests/test\_RefGov\_para\_xmlABC.xml
\end{itemize}
This RAVEN input file simulates the feedback from RefGov\_parameterized\_SIMO ExternalModel. The key factors are:
\begin{itemize}
  \item External matrices file in Listing \ref{lst:ParaExtXMLExample} is used;
  \item External model defined in Listing \ref{lst:Para_RGSIMOExample} is used;
  \item Twenty(20) random input values (uniformly distributed over range [-2000, +2000] MegaWatts) will be generated,
  and fed into the external model as "PwrSet";
  \item The external model will perform the calculation, and print the three(3) outputs(V, V\_min, V\_max) associated
  with each input value of PwrSet to a file "RefGovOutput.csv".
\end{itemize}
The example input file is shown in Listing \ref{lst:InputRGSIMOExample}:

\begin{lstlisting}[style=XML,morekeywords={anAttribute},
caption=Input File Example using RefGov.RefGov\_SIMO ExternalModel., label=lst:InputRGSIMOExample]
<Simulation verbosity="silent">
  <TestInfo>
    <name>plugins/FARM.RefGov_parameterized_SIMO</name>
    <author>HaoyuWang</author>
    <created>2021-02-01</created>
    <classesTested>Models.ExternalModel</classesTested>
    <description>
      This is a test run of parameterized reference governor. It loads ABC matrices from external xml file and calculate the feedback from RefGov_parameterized_SIMO external model.
    </description>
    <requirements> </requirements>
  </TestInfo>
  <!-- TestInfo is the description part of input xml file, won't run -->

  <RunInfo>
    <WorkingDir>RefGov_para_xmlABC_Test</WorkingDir>
    <Sequence>
      RGrun,
      printTOfile
    </Sequence>
  </RunInfo>

  <Files>
    <Input name="ABCMatrices" type="">DMDcCxCoeff_TES_para.xml</Input>
  </Files>

  <Models>
    <ExternalModel name="RG1" subType="FARM.RefGov_parameterized_SIMO">
      <!-- 3 output variables -->
      <outputVariables>V, V_min, V_max </outputVariables>
      <!-- 4 variables: Issued Setpoint(PwrSet), Adjusted Setpoint(V1), bounds of V1(V1min & V1max) -->
      <variables> PwrSet, V, V_min, V_max </variables>
      <!-- steps in MOAS calculation, "g" value -->
      <constant varName="MOASsteps"> 360 </constant>
      <!-- lower and upper bounds for y vector, will be internally checked -->
      <constant varName="Min_Target1"> 2.5 </constant>
      <constant varName="Max_Target1"> 55. </constant>
      <constant varName="Min_Target2"> 2.5 </constant>
      <constant varName="Max_Target2"> 55. </constant>
      <!-- System state vector "x", optional, with elements separated by comma(,) -->
      <constant varName="Sys_State_x"> 30.,0 </constant>
    </ExternalModel>
  </Models>

  <Distributions>
    <Uniform name="one">
      <lowerBound>-2000</lowerBound>
      <upperBound>2000</upperBound>
    </Uniform>
  <!-- distribution for PwrSet sampling -->
  </Distributions>

  <Samplers>
    <MonteCarlo name="RG_Sampler">
      <samplerInit>
        <limit>20</limit>
      </samplerInit>
      <variable name="PwrSet">
        <distribution>one</distribution>
      </variable>
    </MonteCarlo>
  <!-- A MonteCarlo sampler for PwrSet sampling  -->
  </Samplers>

  <DataObjects>
    <PointSet name="RGInput">
      <Input>PwrSet </Input>
      <Output>OutputPlaceHolder</Output>
    </PointSet>
    <PointSet name="RGOutput">
      <Input>PwrSet </Input>
      <Output>V, V_min, V_max </Output>
    </PointSet>
  <!-- input and output pointsets for RG -->
  </DataObjects>

  <Steps>
    <MultiRun name="RGrun">
      <Input  class="DataObjects" type="PointSet">RGInput</Input>
      <Input  class="Files" type="">ABCMatrices</Input>
      <Model  class="Models"  type="ExternalModel">RG1</Model>
      <Sampler  class="Samplers"  type="MonteCarlo">RG_Sampler</Sampler>
      <Output class="DataObjects" type="PointSet">RGOutput</Output>
    </MultiRun>
    <!-- MultiRun step to execute the plugin for multiple times -->
    <IOStep name="printTOfile">
      <Input  class="DataObjects" type="PointSet">RGOutput</Input>
      <Output class="OutStreams"  type="Print">RefGovOutput</Output>
  </IOStep>
  <!-- IOStep to dump the RGOutput to RefGovOutput.csv-->
  </Steps>

  <OutStreams>
    <Print name="RefGovOutput">
      <type>csv</type>
      <source>RGOutput</source>
      <what>input,output</what>
    </Print>
  </OutStreams>
</Simulation>
\end{lstlisting}
The output is a csv file, containing 10 rows of input-outputs (PwrSet - V - V\_min - V\_max). The data and plot are included
in Figure \ref{fig:RGSIMOoutput}.

As one can see, the V\_min and V\_max placed limits on the adjusted actuation variable V:
\begin{itemize}
  \item when original actuation variable(PwrSet) exceeds the upper limit(V\_max), the actuation variable(V) is adjusted to
  V\_max;
  \item when original actuation variable(PwrSet) undergoes the lower limit(V\_min), the actuation variable(V) is adjusted to
  V\_min;
  \item when original actuation variable(PwrSet) is within the range of [V\_min, V\_max], the actuation variable V has the
  same value as PwrSet.
\end{itemize}

\begin{figure}[h]
  \centerline{\includegraphics[width=6in]{include/RefGovOutput.png}}
  \caption{Data and Plot from example RefGov\_parameterized\_SIMO input file.}
  \label{fig:RGSIMOoutput}
\end{figure}