\section{Empirical Application}
In this section we are going to apply our CV model averaging method and other related methods to forecasting the quarterly GDP growth rate for the U.S. and Taiwan. These two series are shown in figure~\ref{fig1}. The U.S. data is obtained from the Bureau of Economic Analysis\footnote{\url{http://www.bea.gov/}}. The data for Taiwan is from National Statistics\footnote{\url{http://eng.stat.gov.tw/}}. The U.S. series is shown in the blue colored solid line while the Taiwan series is denoted in red colored dash line. Note that the data for Taiwan is shorter than that of the the U.S. This is because Taiwan officially starts its modernization with American aid in the early 1950s.

For the U.S. series, we can see that the growth rate becomes less volatile toward the end of the sample. This is the so called Great Moderation phenomenon, see Stock \cite{Stock_DEECON2004} and Stock and Watson \cite{Stock_Watson_JEL2003}. Stock and Watson argue that the forecasting relationship for the GDP growth is time-varying and combination forecasts reliably improve upon the AR benchmark. They claim ``\emph{From the perspective of forecasting methods, this evidence of sporadic predictive content poses the challenge of developing methods that provide reliable forecasts in the face of time-varying relations...the finding that averaging individually unreliable forecasts produces a reliable combination forecast is not readily explained by the standard theory of forecast combination, which relies on information pooling in a stationary environment...fully articulated statistical or economic models consistent with this observation could help to produce combination forecasts with even lower MSFEs.}'' In the next section, we demonstrate that our theory based CV model averaging method performs better than Mallows' weight, Bayesian weight, and most importantly, the equal weight method in terms of smaller root MSFE.

For the Taiwan series, we can see that it has two interesting features. First, it looks like that the average growth rate has dropped toward the end of the sample. This could be explained by the economic growth theory in Macroeconomics that during the early period of modernization or industrialization, a country tends to have high GDP growth rate. But as time goes, the growth rate approaches to the lower equilibrium rate. Second, it seems like the series becomes more volatile toward the end of the sample compared with the U.S. data. This phenomenon contrasts with many other developed counties as they exhibit the similar Great Moderation pattern shown in the U.S. data, for example, Canada and Germany. This motivates us to include Taiwan as an additional empirical example to evaluate our method.
\begin{figure} \centering
	\caption{U.S. and Taiwan Quarterly GDP Growth Rate}
    \includegraphics[scale=0.7]{gdp_ts.pdf}
	\label{fig1}
\end{figure}
\subsection{Forecast U.S. GDP Growth}
Here we apply our method to forecasting U.S. GDP growth rate\footnote{The data we use for this application are from Bruce Hansen's website:\url{http://www.ssc.wisc.edu/~bhansen/cbc/}. Hansen has several exogenous predictors included in his data in addition to the GDP growth data.} out-of-sample and compare its performance with others. We have quarterly data from 1960:1 to 2012:1, 209 total observations. The variable we are interested in forecasting is the U.S. quarterly GDP growth rate. Predictors included in some models are the quarterly change of U.S. 3-month treasury rate ($\Delta\mathrm{SR}$), the quarterly change of U.S. 10-year treasury rate ($\Delta\mathrm{LR}$) and the quarterly change of default premium ($\Delta\mathrm{DP}$)\footnote{In the database, these predictors are given in levels, but we have converted them to rates of change, so the actual sample size in all models except for the AR(1) is 207. The default premium is calculated by the difference between the AAA bond rate and BAA bond rate.}.

We consider five models, from small to large they are:
\begin{subequations}
\begin{align}
\Delta\mathrm{GDP}_{t} & = \beta_{0} + \beta_{1}\Delta\mathrm{GDP}_{t-1} + \epsilon_{t} \label{md:1}\\
\Delta\mathrm{GDP}_{t} & = \beta_{0} + \beta_{1}\Delta\mathrm{GDP}_{t-1} + \beta_{2}\Delta\mathrm{GDP}_{t-2} + \epsilon_{t}\label{md:2} \\
\Delta\mathrm{GDP}_{t} & = \beta_{0} + \beta_{1}\Delta\mathrm{GDP}_{t-1} + \beta_{2}\Delta\mathrm{SR}_{t-1} + \epsilon_{t}\label{md:3} \\
\Delta\mathrm{GDP}_{t} & = \beta_{0} + \beta_{1}\Delta\mathrm{GDP}_{t-1} + \beta_{2}\Delta\mathrm{SR}_{t-1} + \beta_{3}\Delta\mathrm{LR}_{t-1} + \epsilon_{t}\label{md:4} \\
\Delta\mathrm{GDP}_{t} & = \beta_{0} + \beta_{1}\Delta\mathrm{GDP}_{t-1} + \beta_{2}\Delta\mathrm{SR}_{t-1} + \beta_{3}\Delta\mathrm{LR}_{t-1} + \beta_{4}\Delta\mathrm{DP}_{t-1} + \epsilon_{t}\label{md:5}
\end{align}
\end{subequations}
Since in reality we do not know the ``true'' econometric model specification, we consider the above five candidate models. For each model, we examine the performance of various model averaging methods. Consistent with what is done in the previous simulation section, for each model we use the recursive window to forecast out-of-sample. To better examine the OOS performance, in each case we generate a sequence of RMSFE by varying the evaluation sample size, from 20 to 50, with increments of 5. Forecast results from all five models are presented in table~\ref{ntb:3}. For simplicity and ease of comparison, following our Monte Carlo simulation we pick the equal weight method as the benchmark and normalize all OOS forecasting performance around $1$. If the value of relative RMSFE is below $1$, then the OOS forecasts perform better than that of the equal weight method.

We can see that in all five models approximating the DGP, CV forecasts better than SIC, Cp and equal weight under recursive window across all evaluation sample sizes. Additionally, CV is the only method which beats equal weight. The forecast gains of CV relative to equal weight range from about $1\%$ to $6\%$ across evaluation sample sizes and models. CV solves the forecast combination puzzle in this application.

\begin{table}
    \caption{U.S. Quarterly GDP Growth Rate Forecast Comparison} \label{ntb:3}
    \centering
    \begin{adjustbox}{width=\textwidth,totalheight=\textheight,keepaspectratio}
    \begin{threeparttable}
    \begin{tabular}{lccccccccccccccc}
    \toprule
     & \multicolumn{3}{c}{Model a} & \multicolumn{3}{c}{Model b} & \multicolumn{3}{c}{Model c} & \multicolumn{3}{c}{Model d} & \multicolumn{3}{c}{Model e}\\%[0.3em]
    \cmidrule(r){2-4}
    \cmidrule(r){5-7}
    \cmidrule(r){8-10}
    \cmidrule(r){11-13}
    \cmidrule(r){14-16}\\
           & Cp    & CV    & SIC  & Cp    & CV    & SIC & Cp    & CV    & SIC & Cp    & CV    & SIC & Cp    & CV    & SIC \\
    P = 20 & 1.044 & 0.967 & 0.999& 1.031 & 0.983 &0.999& 1.017 & 0.987 &0.999& 1.038 & 0.970 &0.998& 1.043 & 0.960 &0.997 \\
    P = 25 & 1.038 & 0.968 & 0.999& 1.021 & 0.984 &0.999& 1.036 & 0.976 &0.999& 1.038 & 0.969 &0.998& 1.017 & 0.967 &0.998 \\
    P = 30 & 1.022 & 0.977 & 0.999& 1.022 & 0.983 &0.999& 1.007 & 0.996 &1.000& 1.013 & 0.991 &0.998& 1.032 & 0.975 &0.998 \\
    P = 35 & 1.020 & 0.980 & 1.000& 1.036 & 0.996 &0.999& 1.022 & 0.983 &0.999& 1.024 & 0.983 &0.999& 1.034 & 0.973 &0.998 \\
    P = 40 & 1.022 & 0.979 & 0.999& 1.012 & 0.987 &1.000& 1.024 & 0.982 &0.999& 1.025 & 0.982 &0.999& 1.033 & 0.974 &0.998 \\
    P = 45 & 1.024 & 0.978 & 1.000& 1.014 & 0.986 &1.000& 1.025 & 0.982 &0.999& 1.026 & 0.981 &0.999& 1.037 & 0.974 &0.998 \\
    P = 50 & 1.021 & 0.987 & 1.000& 1.011 & 0.989 &1.000& 1.027 & 0.984 &0.999& 1.023 & 0.987 &0.999& 1.022 & 0.988 &0.999 \\
    \bottomrule
    \end{tabular}
    \begin{tablenotes}[para, flushleft] \footnotesize
    Notes: Quarterly data from 1960:1 to 2012:1. $\mathrm{P}$ is the evaluation sample size. Equal weight is chosen as the benchmark and the numbers in the table represent the RMSFE ratio between each individual method and equal weight. Smaller number indicates better forecasting performance. Cp: Mallows' weights. CV: cross-validation weights. SIC: Schwarz-Bayesian weights.
    \newline Model a: AR(1)
    \newline Model b: AR(2)
    \newline Model c: AR(1) + SR
    \newline Model d: AR(1) + SR + LR
    \newline Model e: AR(1) + SR + LR + DP
    \end{tablenotes}
    \end{threeparttable}
    \end{adjustbox}
\end{table}
%Since in this section we are only interested in comparing our methods with others and in trying to somehow solve the forecast combination puzzle, we do not select one best model approximating %the DGP among those five candidates listed above. In practice, researcher may select a best
\subsection{Forecast Taiwan GDP Growth}
Here we apply our method to forecasting Taiwan GDP growth rate out-of-sample and compare its performance with others. We have quarterly data from 1962:1 to 2013:4, 208 total observations. The variable we are interested in forecasting is the Taiwan quarterly GDP growth rate. Since we do not have any exogenous predictors available, we only consider two AR forecasting models, namely, the AR(1) model and the AR(2) model. We continue the general setting outlined in the previous application. For each model, we generate a sequence of RMSFE by varying the evaluation sample size, from 20 to 50, with increments of 5. Forecast results from all five models are shown in table~\ref{ntb:5}. For simplicity and ease of comparison, following our Monte Carlo simulation we pick the equal weight method as the benchmark and normalize all OOS forecasting performance around $1$. If the value of relative RMSFE is below $1$, then the OOS forecasts perform better than that of the equal weight method.

From table~\ref{ntb:5}, we can see that now the Mallows' weights do better than the equal weight in both models and across all evaluation sample sizes. For the AR(1) model, all three methods perform roughly the same but CV has the smallest RMSFE ratio across all evaluation sample sizes. For the AR(2) model, CV performs significantly better than the other two methods.
\begin{table}
    \caption{Taiwan Quarterly GDP Growth Rate Forecast Comparison} \label{ntb:5}
    \centering
    \begin{adjustbox}{keepaspectratio}
    \begin{threeparttable}
    \begin{tabular}{lcccccc}
    \toprule
     & \multicolumn{3}{c}{Model AR(1)} & \multicolumn{3}{c}{Model AR(2)}\\%[0.3em]
    \cmidrule(r){2-4}
    \cmidrule(r){5-7}\\
           & Cp    & CV    & SIC  & Cp    & CV    & SIC  \\
    P = 20 & 0.991 & 0.947 & 0.999& 0.968 & 0.944 &1.000\\
    P = 25 & 0.998 & 0.994 & 1.000& 0.972 & 0.942 &1.000\\
    P = 30 & 0.998 & 0.995 & 1.000& 0.973 & 0.943 &1.000\\
    P = 35 & 0.999 & 0.995 & 1.000& 0.974 & 0.945 &1.000\\
    P = 40 & 0.998 & 0.993 & 1.000& 0.976 & 0.948 &1.000\\
    P = 45 & 0.998 & 0.993 & 1.000& 0.982 & 0.961 &1.000\\
    P = 50 & 0.997 & 0.996 & 1.000& 0.984 & 0.962 &1.000\\
    \bottomrule
    \end{tabular}
    \begin{tablenotes}[para, flushleft] \footnotesize
    Notes: Quarterly data from 1962:1 to 2013:4. $\mathrm{P}$ is the evaluation sample size. Equal weight is chosen as the benchmark and the numbers in the table represent the RMSFE ratio between each individual method and equal weight. Smaller number indicates better forecasting performance. Cp: Mallows' weights. CV: cross-validation weights. SIC: Schwarz-Bayesian weights.
    \end{tablenotes}
    \end{threeparttable}
    \end{adjustbox}
\end{table}


\subsection{All Adinkras in terms of $\ell$- and $\tilde{\ell}$-equivalence Classes}
By direct calculation, we find that their are 48 $\ell$-equivalence classes and 48 $\tilde{\ell}$-equivalence classes.
The $\ell$ ($\tilde{\ell}$) coefficients for the 48 $\ell$-equivalence classes ($\tilde{\ell}$-equivalence classes) of adinkras with the same $\ell$  ($\tilde{\ell}$)  coefficients are shown in Table~\ref{t:ell} (Table~\ref{t:elltilde}). Here we comment on some general results displayed in these tables. First, all cis- representations ($\chi_0 = +1$) have $\ell$ or $\tilde{\ell}$ coefficients which belong to the $\alpha$ $su(2)$ representation and all trans-representations belong to the $\beta$ $su(2)$ representation. Owing to the trace orthogonality relationship, Eq.~(\ref{e:abtraces}), the gadget value between all cis- and trans- representations is zero. This is confirmed by direct gadget calculation in section~\ref{s:gadgets}.
%group numbers are 1-48 as in the code, here they are listed 1-48 top to bottom, left to right
\begin{table}[!h]
	\label{t:ell}
	\begin{tabular}{|c|c|c|c|}
	\hline
		%%%%%%%%%%%%%%A set: Groups 1-8
			flop & flip & $i\ell_{IJ}^1, i\ell_{IJ}^2, i\ell_{IJ}^3 $ & $\chi_0$\\
			\hline
			%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\chi_0 = -1
			\multirow{8}{*}{$\mathcal{V}$}
			& $14,8,2,4$ & $ \beta ^1,\beta ^3,\beta ^2 $ & \multirow{4}{*}{$-1$}\\
			\cline{2-3}
			& 2, 4, 14, 8 & $ -\beta ^1,\beta ^3,-\beta ^2$ & \\
			\cline{2-3}
			&4, 2, 8, 14& $ -\beta ^1,-\beta ^3,\beta ^2 $& \\
			\cline{2-3}
			&8, 14, 4, 2& $ \beta ^1,-\beta ^3,-\beta ^2$ & \\
			%%%%%%%%%%%%%%%%%%%%%%%%%%%%%chi_0 = +1
			\cline{2-4}
			&6, 0, 10, 12&  $-\alpha ^1,\alpha ^2,\alpha ^3  $  & \multirow{4}{*}{$+1$} \\
			\cline{2-3}
			&10, 12, 6, 0& $\alpha ^1,\alpha ^2,-\alpha ^3 $   & \\
			\cline{2-3}
			&12, 10, 0, 6& $\alpha ^1,-\alpha ^2,\alpha ^3$    & \\
			\cline{2-3}
			&0, 6, 12, 10& $-\alpha ^1,-\alpha ^2,-\alpha ^3$& \\
			\hline
			\hline
			%%%%%%%%%%%%%%B set: Groups 9-16
			%%%%%%%%%%%%%%%%%%%\chi_0=+1
			\multirow{8}{*}{$(12)\mathcal{V}$} & 14, 4, 2, 8 & $\alpha ^3,-\alpha ^2,-\alpha ^1$ & \multirow{4}{*}{$+1$}\\
			\cline{2-3}
			&2, 8, 14, 4& $-\alpha ^3,-\alpha ^2,\alpha ^1$& \\
			\cline{2-3}
			&4, 14, 8, 2& $\alpha ^3,\alpha ^2,\alpha ^1$& \\
			\cline{2-3}
			&8, 2, 4, 14&  $-\alpha ^3,\alpha ^2,-\alpha ^1$  & \\
			%%%%%%%%%%%%%%%%%%%\chi_0=-1
			\cline{2-4}
			&6, 12, 10, 0& $\beta ^2,-\beta ^3,\beta ^1$   & \multirow{4}{*}{$-1$}\\
			\cline{2-3}
			&10, 0, 6, 12& $-\beta ^2,-\beta ^3,-\beta ^1$   & \\
			\cline{2-3}
			&12, 6, 0, 10& $\beta ^2,\beta ^3,-\beta ^1$   & \\
			\cline{2-3}
			&0, 10, 12, 6& $-\beta ^2,\beta ^3,\beta ^1$& \\
			\hline
			\hline
			%%%%%%%%%%%%%C set: Groups 17-24
			%%%%%%%%%%%%%%\chi_0=+1
			\multirow{8}{*}{$(13)\mathcal{V}$} & 14, 2, 8, 4 & $-\alpha ^2,\alpha ^1,-\alpha ^3$& \multirow{4}{*}{$+1$}\\
			\cline{2-3}
			&2, 14, 4, 8& $-\alpha ^2,-\alpha ^1,\alpha ^3$& \\
			\cline{2-3}
			&4, 8, 2, 14& $\alpha ^2,-\alpha ^1,-\alpha ^3$& \\
			\cline{2-3}
			&8, 4, 14, 2& $\alpha ^2,\alpha ^1,\alpha ^3$   & \\
			%%%%%%%%%%%%%%\chi_0=-1
			\cline{2-4}
			&6, 10, 0, 12& $-\beta ^3,-\beta ^1,-\beta ^2$   & \multirow{4}{*}{$-1$} \\
			\cline{2-3}
			&10, 6, 12, 0& $-\beta ^3,\beta ^1,\beta ^2$   & \\
			\cline{2-3}
			&12, 0, 10, 6& $\beta ^3,\beta ^1,-\beta ^2$   & \\
			\cline{2-3}
			&0, 12, 6, 10& $\beta ^3,-\beta ^1,\beta ^2$& \\
			\hline
			\hline
	\end{tabular}
	\quad
	\begin{tabular}{|c|c|c|c|}
		\hline
		%%%%%%%%%%%%%D set: Groups 25-32
		flop & flip & $i\ell_{IJ}^1, i\ell_{IJ}^2, i\ell_{IJ}^3 $ & $\chi_0$ \\
			%%%%%%%%%%%%%%%%%%\chi_0=+1
			\hline
			\multirow{8}{*}{$(23)\mathcal{V}$} & 2, 4, 8, 14 & $-\alpha ^1,-\alpha ^3,\alpha ^2$& \multirow{4}{*}{+1}\\
			\cline{2-3}
			&14, 8, 4, 2& $\alpha ^1,\alpha ^3,\alpha ^2$& \\
			\cline{2-3}
			&8, 14, 2, 4& $\alpha ^1,-\alpha^3,-\alpha ^2$ & \\
			\cline{2-3}
			&4, 2, 14, 8& $-\alpha ^1,\alpha ^3,-\alpha ^2$  & \\
			%%%%%%%%%%%%%%%%%%\chi_0=-1
			\cline{2-4}
			&10, 12, 0, 6& $\beta ^1,-\beta ^2,\beta ^3$   & \multirow{4}{*}{$-1$} \\
			\cline{2-3}
			&6, 0, 12, 10&  $-\beta ^1,\beta ^2,\beta ^3$  & \\
			\cline{2-3}
			&0, 6, 10, 12&  $-\beta ^1,-\beta ^2,-\beta ^3$  & \\
			\cline{2-3}
			&12, 10, 6, 0& $\beta ^1,\beta ^2,-\beta ^3$& \\
			\hline
			\hline
			%%%%%%%%%%%%%%E set: Groups 33-40
			%%%%%%%%%%%%%%%%%%\chi_0 = -1
			\multirow{8}{*}{$(123)\mathcal{V}$} & 14, 4, 8, 2 & $-\beta ^2,\beta ^1,-\beta ^3$ & \multirow{4}{*}{$-1$}\\
			\cline{2-3}
			&2, 8, 4, 14& $\beta ^2,-\beta ^1,-\beta ^3$& \\
			\cline{2-3}
			&4, 14, 2, 8& $-\beta ^2,-\beta ^1,\beta ^3$ & \\
			\cline{2-3}
			&8, 2, 14, 4&  $\beta ^2,\beta ^1,\beta ^3$  & \\
			%%%%%%%%%%%%%%%%%%\chi_0 = +1
			\cline{2-4}
			&6, 12, 0, 10& $-\alpha ^3,-\alpha ^1,-\alpha ^2$   & \multirow{4}{*}{$+1$} \\
			\cline{2-3}
			&10, 0, 12, 6& $\alpha ^3,\alpha ^1,-\alpha ^2$   & \\
			\cline{2-3}
			&12, 6, 10, 0& $-\alpha ^3,\alpha ^1,\alpha ^2$   & \\
			\cline{2-3}
			&0, 10, 6, 12& $\alpha ^3,-\alpha ^1,\alpha ^2$ & \\
			\hline
			\hline
			%%%%%%%%%%%%%F set: Groups 41-48
			%%%%%%%%%%%%%%%%%%\chi_0 = -1
			\multirow{8}{*}{$(132)\mathcal{V}$} & 14, 2, 4, 8 & $\beta ^3,-\beta ^2,-\beta ^1$ & \multirow{4}{*}{$-1$}\\
			\cline{2-3}
			&2, 14, 8, 4& $\beta ^3,\beta ^2,\beta ^1$ & \\
			\cline{2-3}
			&4, 8, 14, 2& $\beta ^3,\beta ^2,\beta ^1$ & \\
			\cline{2-3}
			&8, 4, 2, 14&  $-\beta ^3,\beta ^2,-\beta ^1$   & \\
			%%%%%%%%%%%%%%%%%%\chi_0 = +1
			\cline{2-4}
			&6, 10, 12, 0&  $\alpha ^2,-\alpha ^3,\alpha ^1$  & \multirow{4}{*}{$+1$}\\
			\cline{2-3}
			&10, 6, 0, 12&  $\alpha ^2,\alpha ^3,-\alpha ^1$  & \\
			\cline{2-3}
			&12, 0, 6, 10&  $-\alpha ^2,-\alpha ^3,-\alpha ^1$  & \\
			\cline{2-3}
			&0, 12, 10, 6& $-\alpha ^2,\alpha ^3,\alpha ^1$ & \\
			\hline
			\hline
	\end{tabular}
	\caption{$\ell$ coefficients for all 4 color, 4 node adinkras generated from an even worded cis-seed representation of each slice of the pizza. Subscripts indicating which space the elements act over have been suppressed for space purposes. The flops and flips are over the color space, with the color flop being performed before the color flip as in the example above. The numbers encoding the color flip refer to the boolean notation as in appendix~\ref{a:pop} but operating on the colors of the adinkras. For instance, the first row on the left of color flop $\mathcal{V}$ and flip $14, 8, 2, 4$ refers to the equivalence class in Eq.~(\ref{e:FirstEquivClass}) with $\ell^{\hat{a}}_{IJ}$ values as in Eq.~(\ref{e:FirstElls}).}
\end{table}

%group numbers are 1-48 as in the code, here they are listed 1-48 top to bottom, left to right
\begin{table}[!htbp]
	\label{t:elltilde}
	\begin{tabular}{|c|c|c|c|}
	\hline
		%%%%%%%%%%%%%%A set: Groups 1-8
			flop & flip & $i\tilde{\ell}_{IJ}^1, i\tilde{\ell}_{IJ}^2, i\tilde{\ell}_{IJ}^3 $ & $\chi_0$\\
			\hline
			%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\chi_0 = +1
			\multirow{8}{*}{$\mathcal{V}$} & $14,8,2,4$ & $ \alpha ^1,\alpha ^3,\alpha ^2 $ & \multirow{4}{*}{$+1$}\\
			\cline{2-3}
			& 2, 4, 14, 8 & $-\alpha ^1,-\alpha ^3,\alpha ^2$ & \\
			\cline{2-3}
			&4, 2, 8, 14& $-\alpha ^1,\alpha ^3,-\alpha ^2$& \\
			\cline{2-3}
			&8, 14, 4, 2& $ \alpha ^1,-\alpha ^3,-\alpha ^2$ & \\
			%%%%%%%%%%%%%%%%%%%%%%%%%%%%%chi_0 = -1
			\cline{2-4}
			&6, 0, 10, 12&  $-\beta ^1,\beta ^2,\beta ^3$  & \multirow{4}{*}{$-1$} \\
			\cline{2-3}
			&10, 12, 6, 0& $\beta ^1,-\beta ^2,\beta ^3$   & \\
			\cline{2-3}
			&12, 10, 0, 6& $\beta ^1,\beta ^2,-\beta ^3$    & \\
			\cline{2-3}
			&0, 6, 12, 10& $-\beta ^1,-\beta ^2,-\beta ^3$& \\
			\hline
			\hline
			%%%%%%%%%%%%%%B set: Groups 9-16
			%%%%%%%%%%%%%%%%%%%\chi_0=-1
			\multirow{8}{*}{$(12)\mathcal{V}$} & 14, 4, 2, 8 & $-\beta ^2,\beta ^1,-\beta ^3$ & \multirow{4}{*}{$-1$}\\
			\cline{2-3}
			&2, 8, 14, 4& $\beta ^2,-\beta ^1,-\beta ^3$& \\
			\cline{2-3}
			&4, 14, 8, 2& $-\beta ^2,-\beta ^1,\beta ^3$& \\
			\cline{2-3}
			&8, 2, 4, 14&  $\beta ^2,\beta ^1,\beta ^3$  & \\
			%%%%%%%%%%%%%%%%%%%\chi_0=+1
			\cline{2-4}
			&6, 12, 10, 0& $-\alpha ^3,-\alpha ^1,-\alpha ^2$   & \multirow{4}{*}{$+1$}\\
			\cline{2-3}
			&10, 0, 6, 12& $\alpha ^3,\alpha ^1,-\alpha ^2$   & \\
			\cline{2-3}
			&12, 6, 0, 10& $-\alpha ^3,\alpha ^1,\alpha ^2$   & \\
			\cline{2-3}
			&0, 10, 12, 6& $\alpha ^3,-\alpha ^1,\alpha ^2$& \\
			\hline
			\hline
			%%%%%%%%%%%%%C set: Groups 17-24
			%%%%%%%%%%%%%%\chi_0=-1
			\multirow{8}{*}{$(13)\mathcal{V}$} & 14, 2, 8, 4 & $\beta ^3,-\beta ^2,-\beta ^1$& \multirow{4}{*}{$-1$}\\
			\cline{2-3}
			&2, 14, 4, 8& $\beta ^3,\beta ^2,\beta ^1$& \\
			\cline{2-3}
			&4, 8, 2, 14& $-\beta ^3,-\beta ^2,\beta ^1$& \\
			\cline{2-3}
			&8, 4, 14, 2& $-\beta ^3,\beta ^2,-\beta ^1$   & \\
			%%%%%%%%%%%%%%\chi_0=+1
			\cline{2-4}
			&6, 10, 0, 12& $\alpha ^2,-\alpha ^3,\alpha ^1$   & \multirow{4}{*}{$+1$} \\
			\cline{2-3}
			&10, 6, 12, 0& $\alpha ^2,\alpha ^3,-\alpha ^1$   & \\
			\cline{2-3}
			&12, 0, 10, 6& $-\alpha ^2,-\alpha ^3,-\alpha ^1$   & \\
			\cline{2-3}
			&0, 12, 6, 10& $-\alpha ^2,\alpha ^3,\alpha ^1$& \\
			\hline
			\hline
	\end{tabular}
	\quad
	\begin{tabular}{|c|c|c|c|}
		\hline
		%%%%%%%%%%%%%D set: Groups 25-32
		flop & flip & $i\tilde{\ell}_{IJ}^1, i\tilde{\ell}_{IJ}^2, i\tilde{\ell}_{IJ}^3 $ & $\chi_0$ \\
			%%%%%%%%%%%%%%%%%%\chi_0=-1
			\hline
			\multirow{8}{*}{$(23)\mathcal{V}$} & 2, 4, 8, 14 & $-\beta ^1,-\beta ^3,\beta ^2$& \multirow{4}{*}{-1}\\
			\cline{2-3}
			&14, 8, 4, 2& $\beta ^1,-\beta ^3,-\beta ^2$& \\
			\cline{2-3}
			&8, 14, 2, 4& $\beta ^1,\beta ^3,\beta ^2$ & \\
			\cline{2-3}
			&4, 2, 14, 8& $-\beta ^1,\beta ^3,-\beta ^2$  & \\
			%%%%%%%%%%%%%%%%%%\chi_0=+1
			\cline{2-4}
			&10, 12, 0, 6& $\alpha ^1,-\alpha ^2,\alpha ^3$   & \multirow{4}{*}{$+1$} \\
			\cline{2-3}
			&6, 0, 12, 10&  $-\alpha ^1,-\alpha ^2,-\alpha ^3$  & \\
			\cline{2-3}
			&0, 6, 10, 12&  $-\alpha ^1,\alpha ^2,\alpha ^3$  & \\
			\cline{2-3}
			&12, 10, 6, 0& $\alpha ^1,\alpha ^2,-\alpha ^3$& \\
			\hline
			\hline
			%%%%%%%%%%%%%%E set: Groups 33-40
			%%%%%%%%%%%%%%%%%%\chi_0 = +1
			\multirow{8}{*}{$(123)\mathcal{V}$} & 14, 4, 8, 2 & $\alpha ^3,-\alpha ^2,-\alpha ^1$ & \multirow{4}{*}{$+1$}\\
			\cline{2-3}
			&2, 8, 4, 14& $-\alpha ^3,-\alpha ^2,\alpha ^1$& \\
			\cline{2-3}
			&4, 14, 2, 8& $\alpha ^3,\alpha ^2,\alpha ^1$ & \\
			\cline{2-3}
			&8, 2, 14, 4&  $-\alpha ^3,\alpha ^2,-\alpha ^1$  & \\
			%%%%%%%%%%%%%%%%%%\chi_0 = -1
			\cline{2-4}
			&6, 12, 0, 10& $\beta ^2,-\beta ^3,\beta ^1$   & \multirow{4}{*}{$-1$} \\
			\cline{2-3}
			&10, 0, 12, 6& $-\beta ^2,-\beta ^3,-\beta ^1$   & \\
			\cline{2-3}
			&12, 6, 10, 0& $\beta ^2,\beta ^3,-\beta ^1$   & \\
			\cline{2-3}
			&0, 10, 6, 12& $-\beta ^2,\beta ^3,\beta ^1$ & \\
			\hline
			\hline
			%%%%%%%%%%%%%F set: Groups 41-48
			%%%%%%%%%%%%%%%%%%\chi_0 = +1
			\multirow{8}{*}{$(132)\mathcal{V}$} & 14, 2, 4, 8 & $-\alpha ^2,\alpha ^1,-\alpha ^3$ & \multirow{4}{*}{$+1$}\\
			\cline{2-3}
			&2, 14, 8, 4& $-\alpha ^2,-\alpha ^1,\alpha ^3$ & \\
			\cline{2-3}
			&4, 8, 14, 2& $\alpha ^2,-\alpha ^1,-\alpha ^3$ & \\
			\cline{2-3}
			&8, 4, 2, 14&  $\alpha ^2,\alpha ^1,\alpha ^3$   & \\
			%%%%%%%%%%%%%%%%%%\chi_0 = -1
			\cline{2-4}
			&6, 10, 12, 0&  $-\beta ^3,-\beta ^1,-\beta ^2$  & \multirow{4}{*}{$-1$}\\
			\cline{2-3}
			&10, 6, 0, 12&  $-\beta ^3,\beta ^1,\beta ^2$  & \\
			\cline{2-3}
			&12, 0, 6, 10&  $\beta ^3,\beta ^1,-\beta ^2$  & \\
			\cline{2-3}
			&0, 12, 10, 6& $\beta ^3,-\beta ^1,\beta ^2$ & \\
			\hline
			\hline
	\end{tabular}
	\caption{$\tilde{\ell}$ coefficients for all 4 color, 4 node adinkras generated from an even worded trans-seed representation of each slice of the pizza. }
\end{table}


\documentclass[11pt, a4paper]{article}

\usepackage{amsmath,amssymb,bm,braket,float,fontspec,geometry,graphicx,listings,minted,siunitx,url,xcolor}
\usepackage[USenglish]{babel}

%% package settings
\geometry{bottom=2.5cm, top=2.5cm, left=2.5cm, right=2.5cm}
\definecolor{bg4python}{rgb}{0.99, 0.97, 0.97}
\setminted[python]{bgcolor=bg4python,breaklines,mathescape,texcl,linenos}

%% self-defined settings
\setlength\parindent{0pt}

\title{ELEC5210 Midterm Project\\2D Ising model with Monte Carlo simulation}
\author{name, SID}
\date{}

\begin{document}
\maketitle
\bibliographystyle{unsrt}

\section{Ising Model Theory}
Ising model, named after Ernst Ising who first investigated and solved the one-dimensional Ising model\cite{Ising1925}, is a physical model that generally describes a lattice system for spin-1/2 (any two-state system) particles with only two states for each particle: spin up and spin down. We can use $+1$ and $-1$ to denote the two states. There is interaction between neighboring particles, e.g. the four particles on the left, right, above and below one specific particle. The particles can also be subject to an external magnetic filed. The most common Ising model is two-dimensional square lattice system with zero B filed as it can be analytically solved and easy for numerical simulation. Ising model is the simplest physical model that has phase transition\cite{gallavotti2013statistical}.

\subsection{2D square lattice Ising model}
For a 2D square lattice Ising model, assume $N$ by $N$ particles are aligned in a square lattice. The interaction strength between each pair of neighboring particles $S_i$ and $S_i$ is $J_{ij}$, and the interaction strength between one particle $S_i$ and the external B filed is $B_i$. We can write down the Hamiltonian for the system\cite{IsingWiki}:
\begin{equation}
H = -\sum\limits_{i, j}J_{ij}S_i S_j - \sum\limits_{i} B_i S_i \label{eq:Raw Hamiltonian}
\end{equation}

Generally, we consider the Ising model with zero B filed, so equation (\ref{eq:Raw Hamiltonian}) is reduced to a more common format:
\begin{equation}
H = -\sum\limits_{i, j}J_{ij}S_i S_j \label{eq:Common Hamiltonian}
\end{equation}
Here the interaction parameter $J_{ij}$ can be positive (corresponding to ferromagnetic interaction) or negative (corresponding to antiferromagnetic interaction). Since the Hamiltonian corresponds to the energy which is supposed to be minimized for a stable system. For simplicity we can set $J_{ij} = \pm 1$ meaning the interactions between neighboring particles are uniform and for ferromagnetic system we take $J = 1$. Hence the Hamiltonian we will investigate below is finally:
\begin{equation}
H = -\sum\limits_{i, j}S_i S_j \label{eq:Final Hamiltonian}
\end{equation}

For ferromagnetic system, when the temperature is low, the particles tend to have identical spins to lower the total energy; when the temperature is gradually increased, small fluctuations will happen and some noises of the spin alignment will emerge in the system; when the temperature is high enough, the fluctuations due to the thermal movement becomes dominant and finally the system is randomized. There is a phase transition when the temperature is increasing and the temperature for the phase transition is called critical temperature $T_c$.

\subsection{Onsager's solution}
Onsager developed the analytical solution of the Helmholtz free energy $F$ for 2D Ising model\cite{Onsager}:
\begin{equation}
-\beta F = \ln(2) + \frac{1}{8\pi^2} \int_{0}^{2\pi} \int_{0}^{2\pi} \ln\left[\cosh^2(2\beta) - \sinh(2\beta)\cos(\theta_1) - \sinh(2\beta)\cos(\theta_2)\right] \text{d}\theta_1 \text{d}\theta_2 \label{eq:Onsager F}
\end{equation}

here we already take $J = 1$ and $\beta = \frac{1}{k_BT}$ where $k_B = 1$ for simplicity. From Onsager's solution, we can obtain various thermal dynamics parameters among which the most important one is the critical temperature:
\begin{equation}
T_c = \frac{2}{\ln(1+\sqrt{2})} \approx 2.269 \label{eq:Onsager solution}
\end{equation}

To investigate the characteristics of a system, a traditional thermal dynamics method is to calculate the partition function of the system, and then obtain every thermal dynamics characteristics from the partition function. But this method is not suitable for large scale system as the calculation of the partition function requires knowing microscopic configurations, which is nearly impossible for human since the system size is growing exponentially. Therefore, we seek the help from computers and thus the Monte Carlo method is introduced.

\section{Monte Carlo}
Monte Carlo method, named after a casino in Monaco, is a statistical method using probability (especially with random numbers) to simulate and solve real world problem\cite{MCWiki}. The basic principle of Monte Carlo method is to generate random numbers for a specific state of a system, and then calculate certain values based on the randomly generated number, and this update is accepted or rejected based on some criteria. Here we have a simple example of estimating $\pi$ using Monte Carlo method.

\subsection{An example for estimating $\pi$}

Within a $1\times 1$ square area, $N$ points with coordinates $(x_i, y_i)$ are generated randomly. Then the distance between each point and the origin point is calculated and compared with 1. If the distance is smaller than 1, it means the point falls inside the quarter of circle area and if larger, the point falls outside the circle. We count the number of points falling inside of the circle ($n$), and when $N$ goes to considerably large, we can extract $\pi$ from the estimated area: $\lim\limits_{N\to \infty} \frac{n}{N} = \frac{\pi}{4}$, thus $\pi \approx 4\frac{n}{N}$ when $N\to\infty$.
\begin{figure}[H]
\centering
\includegraphics[width=0.7\textwidth]{fig/piEstimation.pdf}\vspace*{-4mm}
\caption{Monte Carlo estimation of $\pi$}\label{fig:Monte Carlo Pi}
\end{figure}

Fig (\ref{fig:Monte Carlo Pi}) illustrates the Monte Carlo simulation result of $N=30000$, and the estimated $\pi_e$ is very close to the real value.

\subsection{Implementation on 2D Ising model}
Consider a $N\times N$ lattice with total particle number $N^2$. Each particle has the spin of $+1$ or $-1$. We can calculate the interaction energy of each particle by the summation of its neighboring four particles, \textit{i.e.}
\begin{equation}
E_{i,j} = -S_{i,j} (S_{i-1, j} + S_{i+1, j} + S_{i, j-1} + S_{i, j+1}) \label{eq:neighboring energy}
\end{equation}

Then we add up all the interaction energy for single particle to get the total energy.

\subsubsection{Initialization}

At first the system can be initialized into a uniform state meaning all the particles have the same spin. From thermal dynamics we know the microscopic state depends on system temperature. The higher the temperature is, the more random the system is. Thus we can throw a random number at one certain temperature $T$ to flip a particle's spin and to see whether the total energy drops or not. Since the total energy is supposed to be minimized, if the total energy of the new state is lower than the older state, the new state is accepted; but if the new energy is higher than the older one, instead of simply rejecting it, we apply the Metropolis method\cite{metropolis}.

\subsubsection{Metropolis method}
When the total energy of the new system is higher than that of the older one (assuming the energy difference is $\Delta E$), we accept it with the probability $e^{-\beta\Delta E = e^{-\frac{\Delta E}{k_BT}}} = e^{-\frac{\Delta E}{T}}$. By ``with the probability of $e^{-\Delta E}$" we mean generating a random number within $[0,1)$ and compare it with $e^{-\Delta E}$; if the generated random number is smaller than $e^{-\Delta E}$ we accept this state.

\subsubsection{Periodic boundary conditions}
We cannot simulate real world crystal as the computation resources for simulation is finite so we must cut off at the edge of the square lattice. The particles inside the square lattice has four interacting neighbor particles, however the particles at the edge have only three or even two neighboring particles. Thus we need to apply periodic boundary conditions (PBC) for boundary particles. When the calculation happens on the boundary particles, we ``borrow'' one or two particles from the opposite border and treat them as the neighboring particles of the boundary particles in order to complete the calculation.

When we have completed all the configurations of the Ising model, we can do the simulation and extract some thermal dynamics parameters from the model, as listed below:
\begin{align}
\braket{E} &= \frac{1}{N^2} \sum\limits_{i} E_i\\
\braket{S} &= \frac{1}{N^2} \ln(\Omega) = \frac{1}{N^2} \ln(\text{C}_{N^2}^{n_{+1}})\\
\braket{C} &= \frac{1}{T^2}\left(\braket{E^2} - \braket{E}^2\right)\\
\braket{M} &= \frac{1}{N^2} \sum\limits_{i} S_i
\end{align}
here the $\Omega$ is the number of possible microscopic states and it can be calculated by the combination number of total particles number and number of spin up particles.

\section{Simulation Results}

Fig (\ref{fig:Results random}) shows the Ising model with a random initial state at different temperature. Noted that at low temperature, the system tend to be aligned at the same direction and several domain emerge. At the critical temperature, the fluctuations of the system is huge and the whole system can no longer remain ordered after iteration. When temperature is high, the system is completely in disordered phase.

\begin{figure}[!h]
\centering
\includegraphics[width=0.9\textwidth]{fig/system-T.pdf}
\caption{Ising model with random initial state}\label{fig:Results random}
\end{figure}

From Fig (\ref{fig:Results uniform}), we can verify the results further. The energy, entropy, specific heat and magnetization are plotted separately. We can see clearly the phase transition happens on the critical temperature $T_c$.

\begin{figure}[!h]
\centering
\includegraphics[width=0.98\textwidth]{fig/ESCM-T-size100-iter1000000.pdf}
\vspace*{-4mm}
\caption{Ising model simulation results with uniform initial state}\label{fig:Results uniform}
\end{figure}

%% Bibliography
\newpage
\bibliography{ref}

%% Appendix
\newpage
\appendix
\section{Appendix: Codes}
\subsection{Monte Carlo estimation of $\pi$}
\inputminted[firstline=5,lastline=9]{python}{code/mcpi.py}
\subsection[Monte Carlo simulation of 2D Ising model]{Monte Carlo simulation of 2D Ising model\protect\footnote{This code is updated from this GitHub repo: \url{https://github.com/bdhammel/ising-model}}}
\inputminted[firstline=8,lastline=79]{python}{code/isingTemperature.py}
\subsection{Plotting simulation data}
\inputminted[firstline=6,lastline=24]{python}{code/plotDataFile.py}

\end{document}


\subsubsection{Cone}
A cone is a transition state between one-sheet and two-sheet hyperboloids. When the constant term in the hyperboloid's equation is 0, the top and bottom surfaces are only connected at a single point.

[INSERT IMAGE]


\section{Experiments}
\label{sec:experiments}

\begin{figure}[t]
    \vspace*{-2px}
    \centering

    \begin{minipage}[t]{0.225\textwidth}
        \vspace*{0px}

        \centering
        \includegraphics[width=1\textwidth]{fig_msvhn_corr_advtrain}
        \vspace*{-14px}

        \includegraphics[width=1\textwidth]{fig_msvhn_succ_advtrain}
    \end{minipage}
    \begin{minipage}[t]{0.225\textwidth}
        \vspace*{0px}

        \centering
        \includegraphics[width=1\textwidth]{fig_msvhn_corr_ours10}
        \vspace*{-14px}

        \includegraphics[width=1\textwidth]{fig_msvhn_succ_ours10}
    \end{minipage}

    \vspace*{-8px}
    \caption{\textbf{Confidence Histograms.} On SVHN, for \AdvTrain ($50\%$/$50\%$ adversarial training, left) and \ConfTrain (right), we show confidence histograms corresponding to \emph{correctly classified} test examples (top) and adversarial examples (bottom). We consider the worst-case adversarial examples across all $L_\infty$ attacks for $\epsilon = 0.03$. While the confidence of adversarial examples is reduced slightly for \AdvTrain, \ConfTrain is able to distinguish the majority of adversarial examples from (clean) test examples by confidence thresholding (in \textcolor{colorbrewer1}{red}).}
    \label{fig:experiments-histograms}
    \vspace*{-8px}
\end{figure}

We evaluate \ConfTrain in comparison with \AdvTrain \cite{MadryICLR2018} and related work \citep{MainiICML2020,ZhangICML2019} on MNIST \citep{LecunIEEE1998}, SVHN \citep{NetzerNIPS2011} and Cifar10 \citep{Krizhevsky2009} as well as MNIST-C \cite{MuICMLWORK2019} and Cifar10-C \cite{HendrycksICLR2019} with corrupted examples (\eg, blur, noise, compression, transforms \etc). We report \textbf{\textit{confidence-thresholded} test error (\TE; $\downarrow$ lower is better)} and \textbf{\textit{confidence-thresholded} robust test error (\RTE; $\downarrow$ lower is better)} for a confidence-threshold $\tau$ corresponding to $99\%$ true positive rate (TPR); we omit $\tau$ for brevity. We note that normal and standard adversarial training (\AdvTrain) are also allowed to reject examples by confidence thresholding. \TE is computed on $9000$ test examples. \RTE is computed on $1000$ test examples. The confidence threshold $\tau$ depends \emph{only} on correctly classified clean examples and is fixed at $99\%$TPR on the held-out \emph{last} $1000$ test examples.

\textbf{Attacks:}
%
For thorough evaluation, we consider $7$ different $L_p$ attacks for $p \in \{\infty, 2, 1, 0\}$. As white-box attacks, we use \PGD to maximize the objectives \eqnref{eq:attack} and \eqref{eq:conf-attack}, referred to as \PGD-\FCE and \PGD-\FConf. We use $T = 1000$ iterations and $10$ random restarts with random initialization plus one restart with zero initialization for \PGD-\FConf, and $T = 200$ with $50$ random restarts for \PGD-\FCE. For $L_\infty$, $L_2$, $L_1$ and $L_0$ attacks, we set \textbf{$\boldsymbol{\epsilon}$ to $\boldsymbol{0.3, 3, 18, 15}$ (MNIST) or $\boldsymbol{0.03, 2, 24, 10}$ (SVHN/Cifar10)}.

As black-box attacks, we additionally use the Query Limited (QL) attack \cite{IlyasICML2018} adapted with momentum and backtracking for $T = 1000$ iterations with $10$ restarts, the Simple attack \citep{NarodytskaCVPRWORK2017} for $T = 1000$ iterations and $10$ restarts, and the Square attack ($L_\infty$ and $L_2$) \citep{AndriushchenkoARXIV2019} with $T = 5000$ iterations. In the case of $L_0$ we also use Corner Search (CS) \citep{CroceICCV2019}. For all $L_p$, $p \in \{\infty, 2, 1, 0\}$, we consider $5000$ (uniform) random samples from the $L_p$-ball and the Geometry attack \cite{KhouryARXIV2018}. Except for CS, all black-box attacks use \eqnref{eq:conf-attack} as objective:

{\footnotesize
\vskip -2px
\begin{tabular}{| l | l | c | c |}
	\hline
	Attack & Objective & $T$ & Restarts\\\hline\hline
	\PGD-\FCE & \eqnref{eq:attack}, random init. & 200 & 50\\
	\PGD-\FConf & \eqnref{eq:conf-attack}, zero + random init. & 1000 & 11\\
	QL$^{\boldsymbol{\dagger}}$ & \eqnref{eq:conf-attack}, zero + random init. & 1000 & 11\\
	Simple$^{\boldsymbol{\dagger}}$ & \eqnref{eq:conf-attack} & 1000 & 10\\
	Square$^{\boldsymbol{\dagger}}$ & \eqnref{eq:conf-attack}, $L_\infty$, $L_2$ only & 5000 & 1\\
	CS$^{\boldsymbol{\dagger}}$ & \eqnref{eq:attack}, $L_0$ only & 200 & 1\\
	Geometry$^{\boldsymbol{\dagger}}$ & \eqnref{eq:conf-attack} & 1000 & 1\\
	Random$^{\boldsymbol{\dagger}}$ & \eqnref{eq:conf-attack} & -- & 5000\\
	\hline
    \multicolumn{4}{l}{\scriptsize $\boldsymbol{\dagger}$ Black-box attacks.}
\end{tabular}
\vskip -2px
}

Additionally, we consider adversarial frames and distal adversarial examples: Adversarial frames \cite{ZajaxAAAIWORK2019} allow a $2$ (MNIST) or $3$ (SVHN/Cifar10) pixel border to be manipulated arbitrarily within $[0,1]$ to maximize \eqnref{eq:conf-attack} using \PGD. Distal adversarial examples start with a (uniform) random image and use \PGD to maximize \eqref{eq:conf-attack} within a $L_\infty$-ball of size $\epsilon = 0.3$ (MNIST) or $\epsilon = 0.03$ (SVHN/Cifar10).

\begin{table}[t]
    \centering
    \footnotesize
    \input{tab_msvhn_per_attack_99tpr}
    %\vskip -4px
    \caption{\textbf{Per-Example Worst-Case Evaluation.} We compare confidence-thresholded \RTE with $\tau$@$99\%$TPR for the per-example worst-case and the top-$5$ individual attacks/restarts among $7$ attacks with $84$ restarts in total. Multiple restarts are \emph{necessary} to effectively attack \ConfTrain, while a single attack and restart is nearly sufficient against \AdvTrainHalf. This demonstrates that \ConfTrain is more difficult to ``crack''.}
    \label{tab:experiments-per-attack}
    %\vspace*{-4px}
\end{table}

\textbf{Training:}
%
We train $50\%$/$50\%$ \AdvTrain (\AdvTrainHalf) and \ConfTrain as well as $100\%$ \AdvTrain (\AdvTrainFull) with $L_\infty$ attacks using $T = 40$ iterations for \PGD-\FCE and \PGD-\FConf, respectively, and $\epsilon = 0.3$ (MNIST) or $\epsilon = 0.03$ (SVHN/Cifar10). We use ResNet-20 \citep{HeCVPR2016}, implemented in PyTorch \citep{PaszkeNIPSWORK2017}, trained using stochastic gradient descent. For \ConfTrain, we use $\rho = 10$.

\textbf{Baselines:}
%
We compare to multi-steepest descent (\Wong) adversarial training \cite{MainiICML2020} using the pre-trained LeNet on MNIST and pre-activation ResNet-18 on Cifar10 trained with $L_\infty$, $L_2$ and $L_1$ adversarial examples and $\epsilon$ set to $0.3, 1.5, 12$ and $0.03, 0.5, 12$, respectively. The $L_2$ and $L_1$ attacks in \tabref{tab:experiments-main} (larger $\epsilon$) are unseen. For \TRADES \cite{ZhangICML2019}, we use the pre-trained convolutional network \cite{CarliniSP2017} on MNIST and WRN-10-28 \cite{ZagoruykoBMVC2016} on Cifar10, trained on $L_\infty$ adversarial examples with $\epsilon=0.3$ and $\epsilon=0.03$, respectively. On Cifar10, we further consider the pre-trained ResNet-50 of \cite{MadryICLR2018} (\MadryAT, $L_\infty$ adversarial examples with $\epsilon = 0.03$). We also consider the Mahalanobis (\Lee) \cite{LeeNIPS2018} and local intrinsic dimensionality (\Ma) detectors \cite{MaICLR2018} using the provided pre-trained ResNet-34 on SVHN/Cifar10.

\begin{figure}[t]%{0.99\textwidth}
    \vspace*{-1px}

    \begin{minipage}[t]{0.235\textwidth}
        \vspace*{-4px}

        \includegraphics[width=1\textwidth]{fig_msvhn_attack_error_advtrain_0}
    \end{minipage}
    \begin{minipage}[t]{0.235\textwidth}
        \vspace*{-4px}

        \includegraphics[width=1\textwidth]{fig_msvhn_attack_error_advtrain_2}
    \end{minipage}
    \\
    %\begin{center}
    \fbox{
        \hspace*{0.9cm}\includegraphics[width=0.35\textwidth]{fig_attack_legend}\hspace*{0.9cm}
    }
    %\end{center}
    \vspace*{-14px}
    \caption{\textbf{Backtracking.} Our $L_\infty$ \PGD-\FConf attack, \ie, \PGD maximizing \eqnref{eq:conf-attack}, using $40$ iterations with momentum and our developed backtracking scheme (left) and without both (right) on SVHN. We plot \eqnref{eq:conf-attack} over iterations for the first $5$ test examples corresponding to different colors. Backtracking avoids oscillation and obtains higher overall objective values within the same number of iterations.}
    \label{fig:experiments-momback}
    \vspace*{-4px}
\end{figure}

\subsection{Ablation Study}
\label{subsec:experiments-ablation}

\textbf{Evaluation Metrics:}
%
\figref{fig:experiments-evaluation} shows ROC curves, \ie, how well adversarial examples can be rejected by confidence. As marked in \textcolor{colorbrewer1}{red}, we are only interested in the FPR for the conservative choice of $99\%$TPR, yielding the confidence threshold $\tau$. The \RTE curves highlight how robustness is influenced by the threshold: \AdvTrain also benefits from a reject option, however, not as much as \ConfTrain which has been explicitly designed for rejecting adversarial examples.

\begin{table*}[t]
	%\centering
	\footnotesize
	\hspace*{-0.2cm}
	\begin{subfigure}{0.82\textwidth}
		\centering
		\input{tab_mmnist_main4_99tpr}
	\end{subfigure}
	\begin{subfigure}{0.085\textwidth}
		\centering
		\input{tab_mmnist_distal3_main_99tpr}
	\end{subfigure}
	\begin{subfigure}{0.08\textwidth}
		\centering
		\input{tab_mmnist_corrupted3_main_99tpr}
	\end{subfigure}
	\vskip 2px
	\hspace*{-0.2cm}
	\begin{subfigure}[t]{0.82\textwidth}
		\centering
		\vspace*{0px}

		\input{tab_msvhn_main4_99tpr}
	\end{subfigure}
	\begin{subfigure}[t]{0.08\textwidth}
		\centering
		\vspace*{0px}

		\input{tab_msvhn_distal3_main_99tpr}
	\end{subfigure}
	\hfill
	\vskip 2px
	\hspace*{-0.2cm}
	\begin{subfigure}[t]{0.82\textwidth}
		\centering
		\vspace*{0px}

		\input{tab_mcifar10_main4_99tpr}
	\end{subfigure}
	\begin{subfigure}[t]{0.085\textwidth}
		\centering
		\vspace*{0px}

		\input{tab_mcifar10_distal3_main_99tpr}
	\end{subfigure}
	\begin{subfigure}[t]{0.08\textwidth}
		\centering
		\vspace*{0px}

		\input{tab_mcifar10_corrupted3_main_99tpr}
	\end{subfigure}
	\vskip -6px
	\caption{\textbf{Main Results: Generalizing Robustness.} For $L_\infty$, $L_2$, $L_1$, $L_0$ attacks and adversarial frames, we report per-example worst-case (confidence-thresholded) \TE and \RTE at $99\%$TPR across all attacks; $\epsilon$ is reported in the corresponding columns. For distal adversarial examples and corrupted examples, we report FPR and \TE, respectively. $L_\infty$ attacks with $\epsilon{=}0.3$ on MNIST and $\epsilon = 0.03$ on SVHN/Cifar10 were used for training (\textcolor{colorbrewer3}{seen}). The remaining attacks were not encountered during training (\textbf{\textcolor{colorbrewer1}{unseen}}). \ConfTrain outperforms \AdvTrain and the other baselines regarding robustness against unseen attacks. FPRs included in the supplementary material. \textbf{*}~Pre-trained models with different architecture, \Ma/\Lee use the same model.}
	\label{tab:experiments-main}
    \vspace*{-2px}
\end{table*}

\textbf{Worst-Case Evaluation:}
%
\tabref{tab:experiments-per-attack} illustrates the importance of worst-case evaluation on SVHN, showing that \ConfTrain is significantly ``harder'' to attack than \AdvTrain. We show the worst-case \RTE over all $L_\infty$ attacks as well as the top-$5$ individual attacks (each restart treated as separate attack). For \AdvTrainHalf, a single restart of \PGD-\FConf with $T=1000$ iterations is highly successful, with $52.1\%$ \RTE close to the overall worst-case of $56\%$. For \ConfTrain, in contrast, multiple restarts are crucial as the best individual attack, \PGD-\FConf with $T=1000$ iterations and zero initialization obtains only $23.6\%$ \RTE compared to the overall worst-case of $39.1\%$.

\textbf{Backtracking:}
%
\figref{fig:experiments-momback} illustrates the advantage of backtracking for \PGD-\FConf with $T{=}40$ iterations on $5$ test examples of SVHN. Backtracking results in better objective values and avoids oscillation, \ie, a stronger attack for training and testing. In addition, while $T{=}200$ iterations are sufficient against \AdvTrain, we needed up to $T{=}1000$ iterations for \ConfTrain.

\subsection{Main Results (Table  \ref{tab:experiments-main})}
\label{subsec:experiments-main}

\textbf{Robustness Against \textcolor{colorbrewer3}{seen} $L_\infty$ Attacks:}
%
Considering \tabref{tab:experiments-main} and $L_\infty$ adversarial examples as \textcolor{colorbrewer3}{seen} during training, \ConfTrain exhibits comparable robustness to \AdvTrain. With $7.4\%$/$67.9\%$ \RTE on MNIST/Cifar10, \ConfTrain lacks behind \AdvTrainHalf ($1.7\%$/$62.7\%$) only slightly. On SVHN, in contrast \ConfTrain outperforms \AdvTrainHalf and \AdvTrainFull significantly with $39.1\%$ vs. $56.0\%$ and $48.3\%$. We note that \ConfTrain and \AdvTrainHalf are trained on $50\%$ clean / $50\%$ adversarial examples. This is in contrast to \AdvTrainFull trained on $100\%$ adversarial examples, which improves robustness slightly, \eg, from $56\%$/$62.7\%$ to $48.3\%$/$59.9\%$ on SVHN/Cifar10.

\textbf{Robustness Against \textcolor{colorbrewer1}{unseen} $L_p$ Attacks:}
%
Regarding \textcolor{colorbrewer1}{unseen} attacks, \AdvTrain's robustness deteriorates quickly while \ConfTrain is able to generalize robustness to novel threat models. On SVHN, for example, \RTE of \AdvTrainHalf goes up to $88.4\%$, $99.4\%$, $99.5\%$ and $73.6\%$ for larger $L_\infty$, $L_2$, $L_1$ and $L_0$ attacks. In contrast, \ConfTrain's robustness generalizes to these unseen attacks significantly better, with $53.1\%$, $29\%$, $31.7\%$ and $3.5\%$, respectively. The results on MNIST and Cifar10 or for \AdvTrainFull tell a similar story. However, \AdvTrain generalizes better to $L_1$ and $L_0$ attacks on MNIST, possibly due to the large $L_\infty$-ball used during training ($\epsilon = 0.3$). Here, training purely on adversarial examples, \ie, \AdvTrainFull is beneficial. On Cifar10, \ConfTrain has more difficulties with large $L_\infty$ attacks ($\epsilon = 0.06$) with $92\%$ \RTE.
As detailed in the supplementary material, these observations are supported by considering FPR. \AdvTrain benefits from considering FPR as clean \TE is not taken into account. On Cifar10, for example, $47.6\%$ FPR compared to $62.7\%$ \RTE for \AdvTrainHalf. This is less pronounced for \ConfTrain due to the improved \TE compared to \AdvTrain. Overall, \ConfTrain improves robustness against arbitrary (unseen) $L_p$ attacks, demonstrating that \ConfTrain indeed extrapolates near-uniform predictions beyond the $L_\infty$ $\epsilon$-ball used during training.

\textbf{Comparison to \Wong and \TRADES:}
%
\TRADES is able to outperform \ConfTrain alongside \AdvTrain (including \MadryAT) on Cifar10 with respect to the $L_\infty$ adversarial examples \textcolor{colorbrewer3}{seen} during training: $43.5\%$ \RTE compared to $68.4\%$ for \ConfTrain. This might be a result of training on $100\%$ adversarial examples and using more complex models: \TRADES uses a WRN-10-28 with roughly $46.1\text{M}$ weighs, in contrast to our ResNet-18 with $4.3\text{M}$ (and ResNet-20 with $11.1\text{M}$ for \Wong). However, regarding \textcolor{colorbrewer1}{unseen} $L_2$, $L_1$ and $L_0$ attacks, \ConfTrain outperforms \TRADES with $52.2\%$, $58.8\%$ and $23\%$ compared to $70.9\%$, $96.9\%$ and $36.9\%$ in terms of \RTE. Similarly, \ConfTrain outperforms \Wong. This is surprising, as \Wong trains on both $L_2$ and $L_1$ attacks with smaller $\epsilon$, while \ConfTrain does not. Only against larger $L_\infty$ adversarial examples with $\epsilon = 0.06$, \TRADES reduces \RTE from $92.4\%$ (\ConfTrain) to $81\%$. Similar to \AdvTrain, \TRADES also generalizes better to $L_2$, $L_1$ or $L_0$ on MNIST, while \Wong is not able to compete. Overall, compared to \Wong and \TRADES, the robustness obtained by \ConfTrain generalizes better to previously unseen attacks. We also note that, on MNIST, \ConfTrain outperforms the robust Analysis-by-Synthesis (ABS) approach of \cite{SchottICLR2019} \wrt $L_\infty$, $L_2$, and $L_0$ attacks.

\textbf{Detection Baselines:}
%
The detection methods \Ma and \Lee are outperformed by \ConfTrain across all datasets and threat models. On SVHN, for example, \Lee obtains $73\%$ \RTE against the seen $L_\infty$ attacks and $79.5\%$, $78.1\%$, $67.5\%$ and $41.5\%$ \RTE for the unseen $L_\infty$, $L_2$, $L_1$ and $L_0$ attacks. \Ma is consistently outperformed by \Lee on SVHN. This is striking, as we \emph{only} used \PGD-\FCE and \PGD-\FConf to attack these approaches and emphasizes the importance of training \emph{adversarially} against an adaptive attack to successfully reject adversarial examples.

\textbf{Robustness Against Unconventional Attacks:}
%
Against adversarial frames, robustness of \AdvTrain reduces to $73.7\%$ /$62.3\%$ \RTE (AT-$50\%$/$100\%$), even on MNIST, while \ConfTrain achieves $0.2\%$. \Wong, in contrast, is able to preserve robustness better with $8.8\%$ \RTE, which might be due to the $L_2$ and $L_1$ attacks seen during training. \ConfTrain outperforms both approaches with $0.2\%$ \RTE, as does \TRADES. On SVHN and Cifar10, however, \ConfTrain outperforms all approaches, including \TRADES, considering adversarial frames.
Against distal adversarial examples, \ConfTrain outperforms all approaches significantly, with $0\%$ FPR, compared to the second-best of $72.5\%$ for \AdvTrainFull on Cifar10. Only the detection baselines \Ma and \Lee are competitive, reaching close to $0\%$ FPR.
This means that \ConfTrain is able to extrapolate low-confidence distributions to far-away regions of the input space.
Finally, we consider corrupted examples (\eg, blur, noise, transforms \etc) where \ConfTrain also improves results, \ie, mean \TE across all corruptions. On Cifar10-C, for example, \ConfTrain achieves $8.5\%$ compared $12.9\%$ for \MadryAT and $12.3\%$ for normal training. On MNIST-C, only \Wong yields a comparably low \TE: $6\%$ vs. $5.7\%$ for \ConfTrain.

\textbf{Improved Test Error:}
%
\ConfTrain also outperforms \AdvTrain regarding \TE, coming close to that of normal training. On all datasets, \emph{confidence-thresholded} \TE for \ConfTrain is better or equal than that of normal training. On Cifar10, only \Ma/\Lee achieve a better standard and confidence-thresholded \TE using a ResNet-34 compared to our ResNet-20 for \ConfTrain ($21.2\text{M}$ vs. $4.3\text{M}$ weights). In total the performance of \ConfTrain shows that the robustness-generalization trade-off can be improved significantly.

Our \textbf{supplementary material} includes detailed descriptions of our \PGD-\FConf attack (including pseudo-code), a discussion of our confidence-thresholded \RTE, and more details regarding baselines. We also include results for confidence thresholds at $98\%$ and $95\%$TPR, which improves results only slightly, at the cost of ``throwing away'' significantly more clean examples. Furthermore, we provide ablation studies, qualitative examples and per-attack results.


%!TEX TS-program = xelatex

\newcommand{\Semester}{WiSe 2016/2017, Term 2}
\newcommand{\fach}{Fourier and wavelet analysis}
\newcommand{\prof}{Prof.\ Mohammad Asadzadeh}

\input{../!config/VorlagenTim/preambel.tex}

\begin{document}

\maketitle
\cleardoubleoddemptypage

\pagenumbering{Alph}
\section*{foreword --- cooperation}
This document is a transcript of the lecture \enquote{\fach, \Semester}, by \prof.
It mainly contains the written content of the lecture. I will not assume any responsibility for the correctness of the content! For questions, remarks and mistakes please write an email to \href{mailto:keil.menden@web.de}{\nolinkurl{keil.menden@web.de}}. I'm grateful for every email.
\newpage

\newpage

\tableofcontents
\cleardoubleoddemptypage
\pagenumbering{arabic}
\setcounter{page}{1}

\section{Introduction}
\label{sec:introduction}
\begin{definition*}[Fouriertransforms]
	We remember the Fouriertransformation for $\mathbb{R}^n$
	\[
		\hat{f}(\xi) = \int_{\mathbb{R}^n}^{}e^{-2 \pi i x \cdot \xi} f(x) \,\mathrm{d}x.
	\]
	where
	\[
		x \cdot \xi = \sum^{n}_{i=1}x_i \xi_j
	\]
\end{definition*}
The course will be about
\begin{itemize}
	\item The Fourier transforms
	\item Distribution theory
	\item Transforms related to Fouriertransforms. (Radon, Hankel z-transforms)
	\item The Wavelet transforms
	\item Multiresolution analysis
	\item Discrete Fouriertransforms, Sampling
	\item Applications (3 Lab obligus)
\end{itemize}
\begin{description}
	\item[Notation]
	\[
		\begin{cases}
			\hat{f}(\xi) &= (\mathcal{F}f)(\xi) \\
			f(x) &\stackrel{\mathcal{F}}{\supset} \hat{f}(\xi)
		\end{cases}
	\]
\item[Basic properties]
\begin{description}
	\item[Linearity] \begin{align*}
		f + g &\supset \hat{f} + \hat{g} \\
		\alpha f &\supset \alpha \hat{f} \qquad (\alpha \in \mathbb{C}, \alpha \in \mathbb{R})
	\end{align*}
	\item[Scaling] \[
		f \supset \hat{f} \qquad \Leftrightarrow \qquad \frac{1}{a}f ( \frac{x}{a}) \stackrel{\mathcal{F}}{\supset} \hat{f}(a \xi)
	\]
\end{description}
If $f$ is "resonably regular/smoth" so is $\hat{f}$.
\item[ouriertransform in $L_2(\mathbb{R})$]
\[
	L_2(\mathbb{R}):= \set[f]{f \text{ is measurable and } \int_{\mathbb{R}}^{}\abs{f(x)}^2 \,\mathrm{d}x < \infty}
\]
\item[Scalar product]
With $f,g :\mathbb{R} \to \mathbb{C}$ we have
\[
	\skal{f}{g} = \int_{\mathbb{R}}^{} f(x) \overline{g(x)} \,\mathrm{d}x.
\]
\item[Parsevals formula]
\[
	\int_{\mathbb{R}}^{}\abs{f(x)}^2 \,\mathrm{d}x = \int_{\mathbb{R}}^{} \abs{\hat{f}(\xi)}^2 \,\mathrm{d}\xi.
\]
\end{description}
By "regular/smoth" we will mean Schwartz class $S$. \\
Often we will have that $f(x)$ is a signal where $f(t)= sin(\omega t)$.
\begin{definition*}[Hevyside]
	We call
	\[
		f(t) = H(x) := \begin{cases}
			1, &x \geq 0 \\
			0, &x <0
		\end{cases}
	\]
	the Hevyside-function. \\
	We have also
	\[
		f(t) = \delta(t)
	\]
	which is called the Diracs "$\delta $-function" and it is the derivative of the Hevyside function. The Dirac functions are not in $L_2$ (nor in $S$).
\end{definition*}
\begin{fundamental}[The Fourier inversion forumula]
	\[
		\hat{f}(\xi) = \int_{\mathbb{R}}^{} e^{-2 \pi i x \xi} f(x) \,\mathrm{d}x \qquad \Leftrightarrow \qquad f(x) = \int_{\mathbb{R}}^{}e^{2 \pi i x \xi}\hat{f}(\xi) \,\mathrm{d}\xi.
	\]
\end{fundamental}
\begin{beispiel}
	Consider
	\[
		\mathcal{F}(e^{-\pi x^2}) = e^{- \pi \xi^2}.
	\]
	We want to scale with $a=2$. Then we get
	\[
		\frac{1}{2} e^{- \pi \left( \frac{x}{2} \right)^2} \stackrel{\mathcal{F}}{\supset } e^{-\pi (2 \xi)^2}.
	\]
\end{beispiel}
\minisec{Other transforms:}
\begin{description}
	\item[Hankel:]
	\[
		\hat{f}(\xi_1,\xi_2) = \int_{\mathbb{R}^2}^{} e^{- 2 \pi i (x_1 \xi_1 + x_2 \xi_2)} f(x_1,x_2) \,\mathrm{d}x_1 \,\mathrm{d}x_2
	\]
	Let
	\[
		f(x_1,x_2) = F( \sqrt{x_1^2+ x_2^2}) = F(r), \qquad \begin{cases}
			x_1 &= r \cos \theta, \\
			x_2 &= r \sin \theta
		\end{cases}
	\]
	Then
	\[
		\hat{f}(\xi_1,\xi_2) = \tilde F(\sqrt{\xi_1^2+ \xi_2^2}) = \tilde F(\rho), \qquad \begin{cases}
			% \xi_1 &= r \cos \theta, \\
% 			\xi_2 &= r \sin \theta
		\end{cases}
	\]
	\[
		F(r) \mapsto \tilde F( \rho)
	\]
	\item[Raden] IF $f(x_1,x_2)$ is the density of "the head" at point $(x_1,x_2)$. Then
	\[
		\int_{L_{r,\omega}}^{} f(x_1,x_2) \,\mathrm{d}t
	\]
	gives a measure of damping along $L_{r,\omega}$ (which can be measured).
	\[
		\mathcal{R}_{\omega} = \int_{L_{r,\omega}}^{} f(x_1,x_2) \,\mathrm{d}t
	\]
	is the Radon transform. It has an application in computer tomography.
\end{description}
\subsection{Functions with Fouriertransform}
\label{sub:functions_with_fouriertransform}
What functions do have a Fouriertransform?
\begin{align*}
	L_2(\mathbb{R}) = \\
	\mathcal{S} \qquad \text{(smooth functions with rapid dacay)} \\
	L_1(\mathbb{R}) = \set[f]{\int_{\mathbb{R}}^{}\abs{f(x)} \,\mathrm{d}x < \infty}
\end{align*}
Note that if
\[
	f \in L_1(\mathbb{R})
\]
we get
\begin{align*}
	\abs{\hat{f}f(\xi)} = \abs{\int_{\mathbb{R}}^{} e^{-2 \pi x \xi} f(x) \,\mathrm{d}x} \\
	\leq \int_{\mathbb{R}}^{} \underset{= \abs{f(x)}}{\underbrace{\abs{e^{-2 \pi i x \xi}f(x)}}} \,\mathrm{d}x < \infty
\end{align*}
So we have in general
\[
	f \in L_1(\mathbb{R}) \qquad \Rightarrow \qquad \hat{f} \in L_{\infty}(\mathbb{R})
\]
And more general we have it for $L_p$ and $L_q$ for $\frac{1}{p}+ \frac{1}{q} =1$.
\begin{theorem}[Hausdorff-Young's inequality]
	For $1 \leq p \leq 2$ with $\frac{1}{p} + \frac{1}{q} = 1$ we have
	\[
		f \in L_p(\mathbb{R}) \qquad \Rightarrow \qquad \hat{f} \in L_q(\mathbb{R})
	\]
	This is a variation of Hölder's inequality.
\end{theorem}
\begin{theorem}[Convolution Theorem]
	Let \[
		f(x) \stackrel{\mathcal{F}}{\supset} \hat{f}(\xi) \text{ and } g(x) \stackrel{\mathcal{F}}{\supset} \hat{g}(\xi)
	\]
	Then we define
	\[
		(f * g)(x) = \int_{\mathbb{R}}^{} f(x-y)g(y) \,\mathrm{d}y \qquad \Leftrightarrow \qquad \supset \hat{f}(\xi)\hat{g}(\xi)
	\]
	with the properties
	\begin{description}
		\item[commutative:] $f * g= g * f$
		\item[associative] $f *( g * h) = (f*g) * h$
		\item[distributive] $f * (g + h) = f *g + f *h$
	\end{description}
\end{theorem}
\minisec{Translation:}
Let $\tau_a$ be a translation such that $\tau_a: f(.) \mapsto f(. -a)$. Then we have
\begin{align*}
	\mathcal{F}(\tau_a f)(\xi) &= \int_{\mathbb{R}}^{}e ^{-2 \pi i \xi x}f(\underset{x-a}{\underbrace{=y}}) \,\mathrm{d}x \\
	&= \int_{\mathbb{R}}^{} e^{-2 \pi i x \xi(x+y)} f(y) \,\mathrm{d}y \\
	&= e^{-2 \pi i x \xi a} \int_{\mathbb{R}}^{} e^{-2 \pi i \xi y} f(y) \,\mathrm{d}y
\end{align*}
and also
\begin{align*}
	\tau_a \hat{f}(\xi) &= \int_{\mathbb{R}}^{} e^{-2 \pi i x ( \xi - a )} f(x) \,\mathrm{d}x \\
	&= \int_{\mathbb{R}}^{} e^{-2 \pi i x \xi} \left( e^{2 \pi i x a }f(x) \right) \,\mathrm{d}x
\end{align*}
\minisec{Derivation:}
We have
\[
	f \stackrel{\mathcal{F}}{\supset} \hat{f} \qquad \Rightarrow \qquad \begin{cases}
		Df &\stackrel{\mathcal{F}}{\supset} 2 \pi i \xi \hat{f}(\xi) \\
		- 2 \pi i (.)f &\stackrel{\mathcal{F}}{\supset} D \hat{f}
	\end{cases}
\]
\begin{beweis}
	We have
	\[
		\hat{f}(\xi) = \int_{\mathbb{R}}^{} f(x) e^{-2 \pi i x \xi} \,\mathrm{d}x
	\]
	Then
	\[
		D \hat{f}(\xi) = \int_{\mathbb{R}}^{} \left( -2 \pi i x f(x) \right) e^{-2 pi i x \xi} \,\mathrm{d}x.
	\]
	And also with partial differentiation
	\begin{align*}
		\mathcal{F}(Df) &= \int_{\mathbb{R}}^{} e^{-2 \pi i x \xi} f'(x) \,\mathrm{d}x \\
		&= - \int_{\mathbb{R}}^{} - 2 \pi i \xi e^{-2 \pi i x \xi}f(x) \,\mathrm{d}x.
	\end{align*}
\end{beweis}
\subsection{The schwartz classes $\mathcal{S}$ and $\mathcal{S}'$}
\label{sub:the_classes_mathcal_s_and_mathcal_s}
[Distributions (generalized functions)]

\begin{definition}[Schwartz-class]
	The function class $\mathcal{S}$ are complex-valued functions $f$ of a real variable such that $f: \mathbb{R} \to \mathbb{C}$ satisfies
	\[
		\sup_{x \in \mathbb{R}} \abs{ \abs{x}^{\alpha} D^{\beta}f(x)} < \infty.
	\]
	for any choice of $\alpha \geq 0$ and $\beta \geq 0$.
\end{definition}
\begin{beispiele}
	\begin{itemize}
		\item For $a >0$ \[
		f(x) = e^{- a x^2}
	\]
	What if $a = i$ or $a <0$?
	\item \[
		f(x) = \begin{cases}
			e^{- \frac{1}{(x-2)^2} - \frac{1}{(x-b)^2}}, &a < x < b \\
			0, &\text{else}
		\end{cases}
	\]
	$f$ has compact support $f \in C^{\infty}$ "but not! real analytic" (i.e. its power series is not convergent)
	\end{itemize}
\end{beispiele}

%% missing

We will now state properties of $\mathcal{S}$.
\begin{lemma}[properties of $\mathcal{S}$]
	\begin{enumerate}
		\item if $f \in \mathcal{S}$ and if for $\alpha, \beta \in \mathbb{Z}^+$ $g(x) = x^{\alpha} D^{\beta}f(x)$ then $g \in \mathcal{S}$
		\item $f \in  \mathcal{S} \qquad \Rightarrow \qquad \hat{f} \in S$.
	\end{enumerate}
\end{lemma}

\cleardoubleoddemptypage
\pagenumbering{Alph}
\setcounter{page}{1}

\end{document}


%Should reference stone as well as koopman
In this section we present some of the tools that are used in the methodology of Chapter \ref{chap:targetLocalisation}. Many of the techniques are used in the approaches noted in the literature review.

\subsection{Hidden Markov Models}\label{subsec:BGHMM}
\input{Chapters/BackgroundKnowledgeAndRelatedWork/MultiAgentTargetDetectionBackground/HMM.tex}

\subsection{Dynamic Bayesian Networks}\label{subsec:BGDBN}
\input{Chapters/BackgroundKnowledgeAndRelatedWork/MultiAgentTargetDetectionBackground/DynamicBayesNets.tex}

\subsection{Inference Algorithms}\label{subsec:BGInfAlgos}
\input{Chapters/BackgroundKnowledgeAndRelatedWork/MultiAgentTargetDetectionBackground/InferenceAlgorithms.tex}

%\subsection{Prediction Algorithms}
%\input{Chapters/BackgroundKnowledgeAndRelatedWork/MultiAgentTargetDetectionBackground/PredictionAlgorithms.tex}

\subsection{Sequential Statistical Hypothesis Testing}\label{subsec:SPRT}
\input{Chapters/BackgroundKnowledgeAndRelatedWork/MultiAgentTargetDetectionBackground/SPRT.tex}

%\subsection{Related Literature}
%\input{Chapters/BackgroundKnowledgeAndRelatedWork/MultiAgentTargetDetectionBackground/RelatedLiterature.tex}


\subsection{GRASP implementation}

\subsubsection{Constructive phase}

The GRASP meta-heuristic has a constructive phase that is concerned to build up a feasible solution. This phase can be deterministic or include a certain amount of randomness by controlling a paremeter value $\alpha$.  This means
that for every execution, different solution could emerge. We use the basic GRASP construction phase described in \cite{grasp}. The specific parts of the implementation are the initialization of the candidate set $C$, implemented in the function $initializeCandidates$ depicted in $Algorithm$ $1$, and the greedy cost function shown in the next subsection.

\begin{algorithm}[H]
\KwIn{nNurses, hours, schedule\_constraints}
%\Parameter{Some parameter}
\KwOut{candidate list C}

\SetKwData{Left}{left}
\SetKwData{This}{this}
\SetKwData{Up}{up}
\SetKwFunction{Union}{Union}
\SetKwFunction{FindCompress}{FindCompress}

$C$ $\leftarrow$ $\emptyset$ \\
\ForEach{$h \in hours$}{

	$E$ $\leftarrow$ $initializeEmptySchedules(10, hours, schedule_constraints)$ \\
	\ForEach{$hindex \in [h+1, hours]$}{
		$modulo\_param = 2$ \\
		\ForEach{$e \in E$}{
			\If{$hindex - h \bmod  modulo\_param > 0$}{
				addWorkingHour(e, hindex) \\
				\If{notValidConstraints(e)}{
					removeWorkingHour(e,hindex) \\
					addRestingHour(e, hindex)  \\
				}
			}
			\Else{
				addRestingHour(e, hindex) \\
				\If{notValidConstraints(e)}{
					removeRestigHour(e,hindex) \\
					addWorkingHour(e, hindex) \\
				}
			}

			\If{notValidConstraints(e, hindex - h)}{

				$E$ $\leftarrow$ $E \cap e$ \\
			}
			$modulo\_param += 1$
		}
	}
	$C$ $\leftarrow$ $C \cup E$
}
$\textbf{return}$ <$C$>
\caption{initializeCandidates}\label{alg.mainLoop}
\end{algorithm}

The ComputeCandidateElements function takes as input the total number of nurses, the number of hours to schedule and the rest of the constraints(maximum presence hours, consecutive hours, total hours and minimum hours). The output is a list of multiple schedules that each nurse can be assigned to. A schedule is the list of hours in which a nurse works must work.\\
First, the algorithm initializes an empty candidate set.
Then, for each hour in the schedule, it creates 10 different types of schedules beginning a this specific hour(line3). The difference between the 10 types of schedules is the compactness of the working hours. This is controlled by a parameter used to do the modulo with current hour index in the built schedule. This allows to create from the most compact schedule with all working hours consecutive until the constraints allows to do it ($hindex - h \bmod hours + 1$), to the most sparse schedule consisting of alternating working and resting hours (using $ hindex -h  \bmod 2$).
The different schedules, started at different hours are built incrementally by adding work or rest hours depending on the modulo parameter (line 7), and always taking into consideration the validity of the resulting partial schedule (line 9, 16), in which case the validity is temptatively fixed. If no more options remain and the schedule becomes invalid for the constraints of the problem(line 21), it is removed from the set $E$ (line 22) and so it is not later saved to the candidate set $C$ (line 27).

\subsubsection{Greedy cost function}

As all the nurses are equal in this scenario, there's only one thing that the greedy cost function can determine, the number of hours of demand that a single nurse schedule contributes to. We are able to do this because all schedule candidates produced are valid (they follow the constraints). We conside $e$ to be a candidate schedule for a single nurse, being $e_{h} = 1$ if the nurse has to work or 0 otherwise. We also use a big constant number K, that should be bigger than the value $hours$ (for example $nNurses$) and we consider $remaining\_demand_h$ to be the demand that is not covered by any other schedule that is present in the solution at the hour $h$	.

\begin{center}
 $ gc(e) = K - \sum_{h=1}^{Hours} remaining\_demand_h \times e_{h}$
\end{center}

\subsubsection{Other problem specific details}

There are some specific modifications of the GRASP constructie phase of \cite{grasp} applied to this problem. After updating the candidate set, in \cite{grasp} page 2, we test the feasibility of the updated solution in each iteration of the constructive phase, and in the case we are having a feasible solution, we leave the loop. If not, we continue looping until no candidate schedules are available or no more candidate schedules can be added (all nurses have a schedule). A solution is feasible if for each hour, the demand is less or equal to the number of nurses working at this hour. Another improvement introduced in the basic version of the algorithm, is the fact that instead of removing candidate elements(single nurse schedules) from the candidate set and adding them to the solution, we generate a basic list of possible candidates and each time a nurse is assigned one of them, we don't remove it from the candidate set. What we do is recompute each time the greedy cost of each candidate when the solution is updated. This is possible thanks to the fact that all nurses are equal under the problem assumptions and constraints. That way we reduce the number of candidates that we have to generate and sort.

\subsubsection{Local Search}

Once the constructive phase ends, the local search tries to improve the given solution by looking at its neighbourhood,i.e. the near solutions will be searched. We show the pseudocode in ($Algorithm$ $2$). We perform a lightweight local search after every GRASP constructive phase, and then when $maxIterations$ of GRASP constructive phases have been performed, we execute a more intensive local search.

\begin{algorithm}[H]
\KwIn{solution, nNurses, hours, demand}
%\Parameter{Some parameter}
\KwOut{solution}

\SetKwData{Left}{left}
\SetKwData{This}{this}
\SetKwData{Up}{up}
\SetKwFunction{Union}{Union}
\SetKwFunction{FindCompress}{FindCompress}

$sol \leftarrow removeExceedingWorkingHours(solution)$ \\
improved = True \\
\While{improved}{

	improved = False \\
	$G \leftarrow findValidScheduleGaps(sol)$ \\
	\While{ $G \neq \emptyset $}{
		\ForEach{$n \in nNurses $}{
			$sol' \leftarrow sol$ \\
			$G_{aux} \leftarrow G$ \\
			\ForEach{$h \in hours$}{
				\If{$sol'(n,h) \neq 1$ }{
					continue \\
				}
				$sol' \leftarrow swapHourAssignment(sol', n, G(h), h) $ \\
				$G \leftarrow removeFirstScheduleGapAtHourH(G,h) $ \\

			}

			\If{$isValidSolution(sol')$ and $improvesCost(sol', sol)$}{
				$sol \leftarrow sol' $ \\
				Improved = True
			}\Else{
				$G \leftarrow G_{aux}$ \\
			}
		}
	}
}

solution $\leftarrow$ sol\\
$\textbf{return}$ solution
\caption{Local Search}\label{alg.mainLoop}
\end{algorithm}

The local search algorithm starts by removing any work assignment that is not needed to fulfill the demand of the problem (line 1). This will make room for later swaping of hour assignments (and sometimes frees some nurses). Then we initialize the improvement while loop, a loop that will stop when no nurse can be freed anymore. Inside the loop(line 5), we first save all the gaps of the solution. That means all hours that a working nurse is resting but that he/she could work and its schedule would still be valid. The gap list $G$ contains for each possible hour of the schedule, a list of nurses that have a valid gap at these specific hour. A new loop is created until there's no more valid gap left (line 6). Inside this loop, we try, for each nurse, to assign its working hours to the first nurse that is listed in the gap list for each hour(line 14, 15).
If after swaping all the hours of a nurse the resulting solution is valid and improves the solution, then the solution is updated (line 17). If not, we undo all the changes to the gap list (line 22) and we don't update the current solution. At the end, the returned solution contains either the same solution as the beginning, or a new solution with less working nurses but all valid and fullfiling the demand at each hour.

To transform the firts improvement local search shown previously into an intensive local search,  we can add room for performing iterations of the local search even if the solution has not improved. We could limit that with a variable called $failedIterations$. Thats the approach we choose to perform on the last localsearch we perform after the main GRASP loop (see first algorithm in \cite{grasp} on page 2).
Usually, there's also the possibility to implement a first improvement local search and a best improvement local search. There is no such possibility with this implementation. We would require to start swaping nurse hour assignments from different nurses instead of following always the same order. This turned to be too costly in terms of algorithmic cost and we decided to go with this simpler approach.