\documentclass[journal]{IEEEtran}

\usepackage[pdfpagemode={UseOutlines},bookmarks=true,bookmarksopen=true,
   bookmarksopenlevel=0,bookmarksnumbered=true,hypertexnames=false,
   colorlinks,linkcolor={blue},citecolor={blue},urlcolor={blue},
   pdfstartview={FitV},unicode,breaklinks=true]{hyperref}

% *** MATH PACKAGES ***
\usepackage{amsmath,amssymb,bm}

% *** GRAPHICS RELATED PACKAGES ***
\ifCLASSINFOpdf
	\usepackage[pdftex]{graphicx}
	\graphicspath{{images/}}
	% \DeclareGraphicsExtensions{.pdf,.jpeg,.png}
\else
	\usepackage[dvips]{graphicx}
	\graphicspath{{images/}}
	% \DeclareGraphicsExtensions{.eps}
\fi

% *** SUBFIGURE PACKAGES ***
\ifCLASSOPTIONcompsoc
  \usepackage[caption=false,font=normalsize,labelfont=sf,textfont=sf]{subfig}
\else
  \usepackage[caption=false,font=footnotesize]{subfig}
\fi

\usepackage[pdftex,dvipsnames]{xcolor}
\usepackage{listings,xargs}
\usepackage[colorinlistoftodos,prependcaption,textsize=tiny]{todonotes}

\definecolor{dkgreen}{rgb}{0,0.6,0}
\definecolor{gray}{rgb}{0.5,0.5,0.5}
\definecolor{mauve}{rgb}{0.58,0,0.82}
\newcommand{\fsize}{\scriptsize}
%\newcommand{\fsize}{\tiny}
\newcommand{\tabsize}{4}

\lstdefinelanguage{diff}{
  morecomment=[f][\color{blue}]{@@},		% group identifier
  morecomment=[f][\color{red}]-,		% deleted lines
  morecomment=[f][\color{dkgreen}]+,		% added lines
  morecomment=[f][\color{magenta}]{---},	% Diff header lines (must appear after +,-)
  morecomment=[f][\color{magenta}]{+++},
}

\lstset{frame=tb,
  language=C++,
%  aboveskip=0mm,
  belowskip=0mm,
  showstringspaces=false,
  columns=flexible,
  basicstyle={\fsize\ttfamily},
  numberstyle=\fsize\color{gray},
%  numbers=left,
  keywordstyle=\color{blue},
  commentstyle=\color{dkgreen},
  stringstyle=\color{mauve},
  breaklines=true,
  breakatwhitespace=true,
  tabsize=\tabsize
}

% correct bad hyphenation here
\hyphenation{op-tical net-works semi-conduc-tor}

\newcommand{\fref}[1]{\figurename~\ref{#1}}
\newcommand{\eref}[1]{(\ref{#1})}
\newcommand{\tref}[1]{\tablename~\ref{#1}}
\newcommand{\lref}[1]{LISTING~\ref{#1}}

\newcommandx{\improv}[2][1=]{\todo[linecolor=Plum,backgroundcolor=Plum!25,bordercolor=Plum,#1]{#2}}
\newcommand{\improvi}[1]{\improv[inline]{#1}}

\begin{document}

% paper title
\title{Acceleration of An SVM Classifier}

% author names and IEEE memberships
\author{Group 6: Yubo~Zhi (yz4116), Jiayang~Sun (js11815)}

% The paper headers
\markboth{ADSD Design Coursework}%
{ADSD Design Coursework}

% make the title area
\maketitle

\begin{abstract}
This course work aims to design and optimise a hardware Support Vector Machine (SVM) classifier on an embedded system using High Level Synthesis (HLS) language. The hardware implementation was optimised by various directives in HLS. In addition, the performance of software and hardware implementations were also assessed and compared in this excise.
\end{abstract}

\section{Introduction}

Dedicated hardware accelerators can perform a particular task faster then generalised software design. However, there are a few trade-offs associated with using hardware accelerators. In this exercise, an SVM classifier IP core was developed and optimised using Vivado HLS toolkit. The hardware platform used in this exercise is Zedboard, which include a Xilinx Zynq SoC FPGA. This FPGA has an ARM Cortex-A9 core together with the FPGA fabric. Hardware acceleration modules developed on the FPGA fabric can communicate with the ARM core through AXI interconnect. The procedure of designing and optimising the accelerator will be explained in details. Finally, the performance differences between hardware and software implementations were evaluated and discussed.

\hfill \today

\section{IP core implementation}

\subsection{Overview}

\fref{fig:ip} shows the purposed SVM classifier accelerator architecture, designed according to \eref{eq:cls}.

\begin{align}
	f(\bm{x}) &= sgn \left( bias + \sum_{i=1}^{N_{SV}} \alpha_i k(\bm{{sv}_i}, \bm{x}) \right)
	\label{eq:cls}\\
	\text{where}~k(\bm{x}, \bm{y}) &= tanh(2 ( \bm{x} \cdot \bm{y} ) )
\end{align}

\begin{figure}[t]
	\centering
	\includegraphics[width=0.6\columnwidth]{IP}
	\caption{Flow chart of purposed SVM classifier IP core}
	\label{fig:ip}
\end{figure}

According to the provided sample data, vectors have a width of 16 elements and there are 1050 support vectors. These numbers were replaced by macro definitions in the design, which can be easily adjusted to meet the properties of potentially other sample data.

\subsection{Dot product calculation}

Dot product between 2 vectors was calculated using the module designed previously in Part I.

\subsection{\texttt{tanh} calculation}

COordinate Rotation DIgital Computer (CORDIC) \cite{volder1959cordic} was used to calculate the \texttt{tanh} function.

\subsubsection{CORDIC}

CORDIC calculates trigonometric functions through vector rotation in 2-dimension space. It can be generalised to calculate \texttt{sinh} and \texttt{cosh} functions. The computations involved with CORDIC are simple integer addition, subtraction and shifting. This makes it suitable to use on a resource constrained embedded platform.

\subsubsection{Range and precision}\label{subsubsec:range}

The precision of CORDIC can be improved by allowing more iterations, however, the input range of CORDIC is non-ideal. The CORDIC implementation used in this design has an input range of about $\pm 1.10$ radians, but the sample dataset can give an input as large as $83.6$ radians.

In order to improve the performance of the \texttt{tanh()}, the algorithm for optimising range detection for CORDIC based on the trigonometric rules were implemented.

\begin{align}
\sinh(\theta + \alpha) = \cosh(\theta)\sinh(\alpha) + \sinh(\theta)\cosh(\alpha)
\label{eq:tri}\\
\cosh(\theta + \alpha) = \cosh(\theta)\cosh(\alpha) + \sinh(\theta)\sinh(\alpha)
\end{align}

As equations \eref{eq:tri} describe, the $\theta$ can be used as a offset in order to increase the range of CORDIC.  In addition, declare a flag (\texttt{neg})to store whether the input is negative. If it is negative set the flag to one and invert the value to positive. Furthermore, it is easy to observe that $tanh()$ is infinitely approximate to 1 when the angle lager than 6 \texttt{rads}. As the result, when the input angle is larger than 6 \texttt{rads}, the result will be directly equal to \texttt{1}. Otherwise, the \texttt{trigo\_index} will be equal to the \texttt{round(theta)}. Also, \texttt{trigo\_index} is the index of look up table(LUT) of \texttt{lut\_sinh[]} and \texttt{lut\_cosh[]}. These value is used as the offset $\theta$ in the previous equation \eref{eq:tri}. After that the rest part of the result will be calculated inside \texttt{cordic()}. The results from \texttt{cordic()} can be used as $\alpha$ in equation \eref{eq:tri}. Moreover, do the operation from trigonometric rules inside \lref{lst:ext}, it will combine the results from CORDIC and offset in order to get the final output. Finally, if the \texttt{neg} is one, invert the output to negative. \cite{llamocca2007fixed}

\begin{lstlisting}[float,floatplacement=h,caption={Extending the range of CORDIC},captionpos=b,label=lst:ext]
	if (theta >= (mdata_t)6.0) {
		result = 1.0;
	} else {
		// Trigo index to extend range
		int trigo_index = theta;
		theta = theta - trigo_index;
		// Call Cordic function
		ldata_t outcosh, outsinh;
		cordic(theta, &outcosh, &outsinh);
		// Trigo rules
		result_sinh = (lut_sinh[trigo_index] * outcosh + lut_cosh[trigo_index] * outsinh);
		result_cosh = (lut_cosh[trigo_index] * outcosh + lut_sinh[trigo_index] * outsinh);
		result = result_sinh / result_cosh;
		*output = neg ? (ldata_t)-result : result;
	}
\end{lstlisting}

\subsection{SVM classifier}

The top-level SVM classifier was designed according to the flow chart \fref{fig:ip}, as shown in \lref{lst:clas}. It iterates through all support vectors, accumulates each result, finally output a boolean value according to the sign of accumulator after bias removal. The function \texttt{k()} is the same as the k inside of equation \eref{eq:cls}. It will calculate the dot product of two vectors, and apply \texttt{tanh()} to the dot product in order to get the output. Finally, output will be checked whether it is negative. If it is positive, then output 0, otherwise output 1.

\begin{lstlisting}[float,floatplacement=h,caption={Top-level SVM classifier},captionpos=b,label=lst:clas]
static void k(ldata_t u[N], ldata_t v[N], ldata_t *output)
{
	mdata_t res;
	dotp(u, v, &res);
	res = res << 1u;	// * 2
	fp_tanh(res, output);
}

void classifier(ldata_t x[N], int *output)
{
	hdata_t sum = 0;
	for (int i = 0; i != ASIZE(alpha); i++) {
		ldata_t res;
		k(SVs[i], x, &res);
		sum += res * alpha[i];
	}
	*output = sum + bias >= 0 ? 0 : 1;
}
\end{lstlisting}

\subsection{Hardware verification}

\subsubsection{Test bench}

The design of SVM classifier was verified using C simulation test benches. The test bench iterates through all \texttt{testData} vectors from the sample dataset and executes the classifier. The results from classifier were compared with \texttt{testDataLabel} ground truths from the same sample dataset. Any difference will be recorded, and an overall error rate will be calculated at the end of simulation.

\subsubsection{Reference generation}

A double precision example classifier implementation was available from the dataset. It produces 170 incorrect results from all 2000 ground truths values, corresponding to an error rate of $8.5 \%$. The results including errors from the double precision were considered the reference output, therefore it can reflect any precision mismatches between the hardware implementation and software double precision implementation.

\subsection{Design evaluation}

The original implementation without any optimisation directives gives a maximum latency of 164877 clock cycles, as shown by \tref{tbl:res_unopt}.

\begin{table}[ht]
	% increase table row spacing, adjust to taste
	\renewcommand{\arraystretch}{1.3}
	\caption{Performance and resource usage of unoptimised version}
	\label{tbl:res_unopt}
	\centering
	% Some packages, such as MDW tools, offer better commands for making tables
	% than the plain LaTeX2e tabular which is used here.
	\begin{tabular}{llll}
		\hline
		Item			& Unoptimised	\\
		\hline
		Estimated clock timing	& $8.77$	\\
		Latency			& $164877$	\\
		Interval		& $164878$	\\
		Co-simulation (average)	& $203$		\\
		\hline
		BRAM\_18K		& $36~(12\%)$	\\
		DSP48E			& $6~(2\%)$	\\
		FF			& $1436~(1\%)$	\\
		LUT			& $4086~(7\%)$	\\
		\hline
	\end{tabular}
\end{table}

Running RTL co-simulation using all 2000 test vectors with this order of delay interval between every test vector will be very time-consuming and not very useful. Therefore, only C simulation was used to ensure the correctness of the unoptimised design. RTL co-simulations were still used in later optimised design.

\section{Performance optimisation}

\subsection{Interface optimisation}

The interface for specified input vectors was optimised by reducing the interface width to a minimum and using the \texttt{ARRAY\_RESHAPE} directive. This directive can combine multiple elements of the input vector to a single 32-bit buffer, reduces the number of data transfers needed between the ARM core and the hardware accelerator.

\subsection{Resource usage optimisation}

In order to balanced the resources and accuracy, the bits of fix point number should be controlled by different bits of \texttt{ap\_fixed} type, as shown in \lref{lst:data_t}. The number of fractional bits for all types is 12. The only changed part is integer part, in order to have enough range to represent different data. For example, \texttt{hdata\_t} it is used for the accumulation, final result, and bias. Multiple times experimental tests are set in order to get exact number of bits for integer part of different arrays due to different range of data.

\begin{lstlisting}[float,floatplacement=h,caption={Optimised data type},captionpos=b,label=lst:data_t]
typedef ap_fixed<22, 10> hdata_t;
typedef ap_fixed<19, 7> mdata_t;
typedef ap_fixed<16, 4> data_t;
typedef ap_fixed<16, 4> ldata_t;
\end{lstlisting}

\subsection{Throughput optimisation}

\subsubsection{Pipelining}

By using the technique of pipelining, the throughput can be greatly improved without significant increases of hardware resource usages. This can be achieved by applying the \texttt{PIPELINE} directive. Vivado HLS will automatically insert pipeline stage registers in appropriate locations. An iteration interval of 1 clock cycles was achieved by pipelining. With the addition of a few pipeline stages filling the entire pipeline beforehand, the iteration of all 1050 support vectors now takes only over 1100 clock cycles.

\subsubsection{Loop unrolling}

There are also some small iteration loops in the design, such as the iteration inside dot product calculation which only iterates about 16 times. These loops can be unrolled to allow parallel computation of iterations, at the cost of more hardware resource usages. This can be achieved by applying the \texttt{UNROLL} directive. Vivado HLS is capable of duplicating the loop body multiple times, and applying map-reduce techniques to resolve dependencies automatically. Together with pipelining, this can reduce the number of pipeline stages.

\subsubsection{Parallel chains}

The latency can be further reduced by executing multiple iterations of the support vector loop inside the main classifier function in parallel. Instead of sequentially iterates through all 1050 support vectors, ideally this parallel execution can reduce the latency by a factor of parallel chains.
The disadvantage of this method is, the hardware resource usage will be increased by the same factor in order to implement multiple parallel chains. A trade-off between throughput and resource usage is implied.
This parallel execution was done by unrolling the main loop while specifying unroll factor. All constant values such as support vectors and alpha implemented by ROMs need to be partitioned by the same factor to prevent Vivado duplicating the entire ROM for each parallel chain. The array partition was done by specifying the \texttt{ARRAY\_PARTITION} directive.
\lref{lst:opt_cla} shows the classifier function with optimisations applied.

\begin{lstlisting}[float,floatplacement=h,caption={Optimised classifier top-level function},captionpos=b,label=lst:opt_cla]
void classifier(ldata_t x[N], int *output)
{
#pragma HLS ARRAY_RESHAPE variable=x cyclic factor=16 dim=1
#pragma HLS ARRAY_PARTITION variable=SVs cyclic factor=10 dim=1
#pragma HLS ARRAY_RESHAPE variable=SVs cyclic factor=16 dim=2
#pragma HLS ARRAY_PARTITION variable=alpha cyclic factor=10 dim=1
#pragma HLS INTERFACE s_axilite port=output
#pragma HLS INTERFACE s_axilite port=return
#pragma HLS INTERFACE s_axilite port=x
	hdata_t sum = 0, sums[12] = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};
#pragma HLS ARRAY_PARTITION variable=sums cyclic factor=10 dim=1
loop:	for (int i = 0; i != ASIZE(alpha); i++) {
#pragma HLS PIPELINE
#pragma HLS UNROLL factor=10
		ldata_t res;
		k(SVs[i], x, &res);
		sums[i % 10] += res * alpha[i];
	}
	for (int i = 0; i != 10; i++) {
#pragma HLS PIPELINE
#pragma HLS UNROLL
		sum += sums[i];
	}
	*output = sum + bias >= 0 ? 0 : 1;
}
\end{lstlisting}

The \texttt{ARRAY\_PARTITION} directive is used for cutting the array into small parts in order to parallel the program. \texttt{ARRAY\_RESHAPE} also can separate the array, however, it will combine the array together after the progress so it is better to use inside the dot product in order to save resources. The main \texttt{for} loop is optimised by \texttt{PIPELINE} and \texttt{UNTROLL} together. The factor of \texttt{UNROLL} means the number of threads paralleled is 10. The maximum number of threads is ten because of the limitation of hard ware resources.

\subsection{Clock timing issue}

\begin{figure*}[!t]
	\centering
	\includegraphics[width=0.9\textwidth]{bd}
	\caption{Block diagram design of the entire system}
	\label{fig:bd}
\end{figure*}

An unexpected issue occurred when applying unroll optimisations. The target clock period specified for this design is $10 ns$, but Vivado kept generating paths exceeding that clock constrain. By analysing the warning messages about the critical path, it is possible to resolve this issue by separating the arithmetic involved in the critical path, letting Vivado insert pipeline registers between them.

An example of resolving the timing issue is given by \lref{lst:tim}. The original multiplication and accumulation operation was broken into 10 separate \texttt{sums} variables as caches, partitioned to match the loop unroll factor. So that each parallel chain has a dedicated accumulator, summarised together in later stages. This removes the need for accumulate all 10 parallel chains in each iteration, hence reduces the length of the critical path.

\begin{lstlisting}[float,floatplacement=h,language=diff,caption={Code modification to resolve timing issue},captionpos=b,label=lst:tim]
--- old/classifier.cpp
+++ new/classifier.cpp
@@ -62,12 +62,19 @@ void classifier(ldata_t x[N],
 #pragma HLS INTERFACE s_axilite port=output
 #pragma HLS INTERFACE s_axilite port=return
     hdata_t sum = 0;
+    hdata_t sums[10] = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0};
+#pragma HLS ARRAY_PARTITION variable=sums cyclic factor=10 dim=1
     for (int i = 0; i != ASIZE(alpha); i++) {
 #pragma HLS PIPELINE
 #pragma HLS UNROLL factor=10
        ldata_t res;
        k(SVs[i], x, &res);
-       sum += res * alpha[i];
+       sums[i % 10] += res * alpha[i];
+    }
+    for (int i = 0; i != 10; i++) {
+#pragma HLS PIPELINE
+#pragma HLS UNROLL
+       sum += sums[i];
     }
     *output = sum + bias >= 0 ? 0 : 1;
 }
\end{lstlisting}

\subsection{New \texttt{tanh()} reflection}

As the subsection \ref{subsubsec:range} mentioned, there is a new technique to enhance the range and accuracy of the \texttt{tanh()} fuction. However, it needs more resource and more instructions to complete, which means that it has higher latency than the previous CORDIC (133 cycles). However, the final latency of the design is shown in \fref{fig:latency}, which is 150 cycle. Although there are 17 more cycles to complete the calculation, it gets as accurate as the double precision answer generated by software test bench. Because if using the previous method to complete \texttt{tanh()} there are in total 169 errors, nonetheless it is different from the result come from the test bench which is 170 errors. As the result, the final implementation choose a more accuracy method in order to get enhanced classification.

\subsection{Design verification}

Correctness verifications by C simulations were done after every optimisations and code modifications, using the golden reference output generated from double precision implementation.

An unexpected discrepancy between Vivado HLS' C language and ISO C language standard (6.7.8.19 \cite{iso1999iec}) was discovered. All array elements must be explicitly initialised in Vivado HLS for a defined behaviour, whereas in ISO C standard, uninitialised elements in a partially initialised array should have the value 0, the same as static storage initialisation. Hence the 10 zeros written in the initialisation of \texttt{sums} in \lref{lst:tim}.

After applied all optimisations, the latency reduced significantly. RTL co-simulation was done at this stage before exporting as an IP core, to ensure the exported RTL implementation has the same functionality as the C implementation.

\subsection{Design evaluation}

\fref{fig:latency} shows the latency and varies aspects of hardware resources versus main loop unrolling factor.

\begin{figure}[ht]
	\centering
	\includegraphics[width=0.9\columnwidth]{latency}
	\caption{Latency and resource usage versus loop unrolling. unopt: Unoptimised version}
	\label{fig:latency}
\end{figure}

As loop unroll factor increases, hardware resource usages increase linearly. However, the effect on latency decreases exponentially. The latency can be estimated by \eref{eq:latency}.

\begin{equation}
	\text{latency} = \frac{N_{SV}}{\text{factor}} + \text{stages}
	\label{eq:latency}
\end{equation}

$N_{SV}$ equals the number of support vectors, which is 1050. Stages is the depth of pipeline, which is 44 in most cases, given by the loop latency value.

The equation shows, there is always some factors that can not be parallelised. Therefore the effect on performance will become more and more insignificant. Moreover, with a loop unroll factor of 10, DSP resource usage had already reached $95.45 \%$, impossible for further adjustments.

\section{System implementation}

After multiple optimisations, the improved implementation was packaged into Vivado as an IP core. The setup of Vivado and SDK were the same as in previous dot product exercise. \fref{fig:bd} shows the block diagram design of the system on FPGA. \fref{fig:sdk} describes an abstract structural overview of the testing program running on the ARM core, developed on the SDK.

\begin{figure}[t]
	\centering
	\includegraphics[width=0.3\columnwidth]{sdk}
	\caption{Overview of the testing program}
	\label{fig:sdk}
\end{figure}

\section{System evaluation}

Using the global timer inside ARM core, it is possible to get cycle accurate profiling information.

\lref{lst:test_int} shows one example of the tests. This test aims to record the time consumption of different procedures of the whole system. The whole progress can be divided into three parts which include data transmission from ARM core to IP core, starting, and calculation. After running this program, the total time and the time for each part will be print in the terminal.

\begin{lstlisting}[float,floatplacement=h,caption={Code for recoding execution time for different procedures},captionpos=b,label=lst:test_int]
unsigned int test_cls_hls_interrupt()
{
	XTime tick[4], perf[3] = {0, 0, 0};
	unsigned int err = 0;
	int *label = &testDataLabel[0];
	int16_t *x = &testDataI[0][0];
	for (size_t ix = ASIZE(testDataLabel); ix != 0; ix--) {
		XTime_GetTime(&tick[0]);
		XClassifier_Set_x_V(&cls, *(XClassifier_X_v *)x);
		XTime_GetTime(&tick[1]);
		XClassifier_Start(&cls);
		XTime_GetTime(&tick[2]);
		interrupt_wait();
		XTime_GetTime(&tick[3]);
		err += (!XClassifier_Get_output_r(&cls)) != (!*label++);
		perf[0] += tick[1] - tick[0];
		perf[1] += tick[2] - tick[1];
		perf[2] += tick[3] - tick[2];
		x += N;
	}
	printf("Data %llu, starting %llu, result %llu\r\n", perf[0], perf[1], perf[2]);
	return err;
}
\end{lstlisting}

An example output of the test program was given by \lref{lst:test_int_out}.

\begin{lstlisting}[float,floatplacement=h,language=,caption={Example output from testing program},captionpos=b,label=lst:test_int_out]
Build @ Mar 16 2017 22:15:28
HLS peripheral is ready
Converting data...
Conversion finished, 863684 ticks

Starting tests...
<double precision classifier> finished, 450697222 ticks, error count 170
<fixed point classifier> finished, 200704648 ticks, error count 170
Data 1097219, starting 266571, result 1100144
<HLS classifier by polling> finished, 2651697 ticks, error count 170
Data 1089646, starting 267273, result 1287449
<HLS classifier by interrupt> finished, 2826942 ticks, error count 170
SVM classifiers testing done.
\end{lstlisting}

\subsection{Performance and resource usage}

The support vector classifier was also implemented as software using double precision and fixed point CORDIC inside the ARM core, as a performance comparison to hardware implementation.

The performance data of software and notable hardware implementations are summarised in \tref{tbl:throughput}. \fref{fig:throughput} shows detailed hardware implementation throughputs and resource usage values. Both polling and interrupt methods for waiting results ready were implemented, however, only performance data from polling method was shown. The polling method gives slightly better overall performance values, probably due to some interfacing overheads of supporting interrupt.

\begin{table}[ht]
	% increase table row spacing, adjust to taste
	\renewcommand{\arraystretch}{1.3}
	\caption{Performance and resource usages of system implementations}
	\label{tbl:throughput}
	\centering
	% Some packages, such as MDW tools, offer better commands for making tables
	% than the plain LaTeX2e tabular which is used here.
	\begin{tabular}{llll}
		\hline
		Type			& Throughput (k vectors / second)	\\
		\hline
		Double precision	& $2.96$	\\
		Fixed precision CORDIC	& $6.64$	\\
		\hline
		Unoptimised hardware	& $1.47$	\\
		Pipelined hardware	& $152.17$	\\
		Unrolled with factor 10	& $541.18$	\\
		\hline
	\end{tabular}
\end{table}

It can be seen from the table, the final optimised hardware implementation gives a speed-up of 182 times over the double precision and 80.5 times over the fixed precision software implementation, quite substantial. However, as the figure suggests, the throughput almost saturates as loop unroll factor increases above 9.

\begin{figure}[ht]
	\centering
	\includegraphics[width=0.9\columnwidth]{throughput}
	\caption{System throughput and resource usages versus loop unroll factor}
	\label{fig:throughput}
\end{figure}

\fref{fig:time} shows a detailed breakdown of total time spent on each stage of interfacing the hardware SVM classifier. The time values shown on the figure were summarised for all 2000 test vectors.

\begin{figure}[ht]
	\centering
	\includegraphics[width=0.9\columnwidth]{time}
	\caption{Detailed time usage breakdown of interfacing hardware classifier}
	\label{fig:time}
\end{figure}

It can be seen from the figure that transferring the 16 elements input vectors (\texttt{tData}) and starting the classifier (\texttt{tStart}) are very inefficient and unavoidable, taking almost the same time as actual computation (\texttt{tResult}) in the final version where loop unroll factor is 10. These unavoidable overheads limit the maximum throughput achievable, result in the saturation.

\subsection{Reflection and Limitations}

Different from previous dot product exercise, the SVM classifier hardware accelerator designed in this exercise improves the system throughput by a large factor. The benefits of using a massively paralleled hardware accelerator greatly overcome the overheads of interfacing the accelerator, based on the number of computations offloaded onto hardware each time. The processor core is also freed to do some other sequential computations while waiting for the results to be available, further increases system performance and efficiency.

\section{Conclusion}

By evaluating the performance data of software and hardware implementation of the same algorithm, the advantages, use case and limiting factors of using hardware accelerators were investigated. Although the computation speed of dedicated hardware accelerator can be faster than software implementation, overheads such as data transfer performance may limit the actual efficiency.

% References section
\bibliographystyle{IEEEtran}
\bibliography{Reference}

% that's all folks
\end{document}


\documentclass{article}

\usepackage{amsmath,amssymb,amsthm,bm,graphicx,enumerate}
\usepackage{color}

\newcommand{\be}{\begin{equation}}
\newcommand{\ee}{\end{equation}}

\usepackage{geometry}
 \geometry{
 a4paper,
 total={210mm,297mm},
 left=35mm,
 right=35mm,
 top=30mm,
 bottom=30mm,
 }

 % make a note
\newcommand{\mynote}[1]{\noindent \textcolor{red} {{$\blacktriangleright$
   \small{\textsf{#1}} $\blacktriangleleft$}}}

\title{Covid-19 epidemic control}
\author{Lauri Neuvonen, Matthias Wildemeersch}
\date{}

\begin{document}

\maketitle
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Scenarios of interest}
\begin{enumerate}[-]
\item Limitations of the current model: (i) All asymptomatic become symptomatic (ii) recovered cannot turn susceptible again. These shortcomings can be resolved rather easily, but the corresponding transition rates are unknown and would require a sensitivity analysis. Possible extension in a later phase.
\item Distinction between lock-down and quarantine: Known infected people are put in quarantine with contact probability $\lambda^Q$; people in lockdown have a contact probability $\lambda^{LD}$; baseline contact probability is $\lambda$; $\lambda^Q < \lambda^{LD}<\lambda$.
\item Economic output
\be
Y = NQ + \frac{\lambda^{LD}}{\lambda} LD + \frac{\lambda^Q}{\lambda} Q \, ,
\ee
with NQ, Q, and LD representing the number of people not in quarantine, in quarantine, and in lockdown.
\mynote{MW: need to think if we need another scaling factor in this formula} Loss in output can be calculated then as $Y_\mathrm{loss} = Y_\mathrm{baseline} - Y$.
\item number of objectives: (i) output or loss of output/working hours, (ii) number of deaths / number of hospitalized people, (iii) cost of the strategy. The loss of output could potentially be combined with the cost of the strategy. Another alternative is to look at the loss of working hours over the cost of the strategy.
\item scenarios of interest
\begin{enumerate}[i.]
\item generalized lock-down: Control strength of the lock-down by means of the parameter $\lambda^{LD}$; testing only for people who start developing symptoms. Trade-off between economic cost vs health cost as a function of $\lambda^{LD}$. Testing cost small.
\item Paul Romer scenario: massive population-scale testing without a generalized lock-down. Only people who test positive are put in quarantine; tradeoff related to sensitivity of diagnostic testing, where false negatives can freely circulate. Cost of strategy is high. This approach requires implementing repeated testing (see below). Testing cost high.
\item test-and-trace: No generalized lockdown. Only people who develop symptoms are tested, and if positive track the (on average) 10 (?sensitivity?) people with whom infected person was in contact. These 10 people should be distributed over different compartments; requires some tweaks in the model.  \mynote{MW: to be worked out in the coming weeks} Test-and-trace cost moderately high.
\end{enumerate}
\item repeated testing: requires transition from NA* to NA, and transition from NA* to IA without star. The transition from NA* to NA happens on average in 7 days (sensitivity), so transition probability is $\sigma = 1/7$. Fixes are marked in excel file in green.
\end{enumerate}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Test and trace}
In this section, we present two methods that capture test-and-trace for compartmental models. Test-and-trace can easily be implemented in individual based models, which keep track of the history and individual contacts of each agent. In compartmental models, working with expected values and perfect mixing, test-and-trace is less straightforward. The aim of this section is to quantify how many undiagnosed people will get traced and quarantined. The test-and-trace can be implemented in two different ways, (i) people that are reached through contact tracing go into quarantine without diagnosis, or (ii) people that are traced get tested if the testing capacity is sufficient, and get quarantined based the test result.

\subsection{Method 1}
In the baseline model, the following compartments are affected by test-and-trace, NA-NQ (undiagnosed and diagnosed), IA-NQ undiagnosed, and FN-NQ. All said compartments transition to the quarantined state if they get traced. The transition probability from compartment $X$ to $Y$ can be written as
\be
\xi_{X \to Y} = \xi_\mathrm{TT} f^X \, ,
\ee
where $\xi_\mathrm{TT}$ is a generic test-and-trace efficiency parameter which is scaled by the fraction $f^X$ of the compartment with respect to the unquarantined population
\be
f^X = \frac{N_t^X}{N_t^\mathrm{U}} = \frac{N_t^X}{N_t^\mathrm{NA-NQ-U}+N_t^\mathrm{NA-NQ-D} + N_t^\mathrm{IA-NQ-U} + N_t^\mathrm{FN-NQ}} \, .
\ee
This method has as pro that it is easy and that we consider the test-and-trace strength over a large interval. (The scaling parameter could even be disregarded) As disadvantage, the method is coarse and it does not take into account changes of tracing efforts over time as a result of the number of newly identified cases. Delays between testing and sequential tracing can be included by introducing additional compartments for the different phases in the incubation and symptomatic period.

\subsection{Method 2}
We aim to describe the efficiency of test-and-trace as a function of time, in other words, as a function of the trajectory of the epidemic. Assuming approach (i) of contact tracing where contact-traced people go into quarantine, we can define the transition probability between the non-quarantined compartment $X$ and the quarantined compartment $Y$ as $\xi_{X \to Y}^\mathrm{CT}$, which can be expressed as
\begin{align}
\xi_{X \to Y}^\mathrm{CT}(t) &= P(\mathrm{CT} \, | \,C_\mathrm{T}) \cdot P(C_\mathrm{T}) \nonumber \\
& = \eta \cdot P(C_\mathrm{T}) \, ,
\end{align}
where $\eta$ represents the efficiency of the test-and-trace strategy, the probability that an individual get contact-traced conditioned on the event $C_\mathrm{T}$ that this person was in contact with a diagnosed/tested person. The probability $P(C_\mathrm{T})$ stands for the contact probability with a diagnozed person. We can further write
\begin{align}
\xi_{X \to Y}^\mathrm{CT}(t) &= \eta \cdot \Big( P(C_\mathrm{I}) \cdot P(\mathrm{IT}) + P(C_\mathrm{N}) \cdot P(\mathrm{FP}) \Big)\nonumber \\
&= \eta \cdot \Big( \lambda^X \, \pi_t^I \cdot (\pi_t^\mathrm{IST} + \pi_t^\mathrm{IAT}) + \lambda^X \pi_t^N \cdot \pi_t^\mathrm{FP} \Big) \, ,
\end{align}
where $\pi_t^I$ and $\pi_t^N$ are the probabilities to meet an infected and non-infected individual conditioned on the meeting an individual, $\lambda^X$ stands for the meeting rate and depends on the compartment $X$, $\pi_t^\mathrm{IST}$ and $\pi_t^{IAT}$ represent the probability at time $t$ that an infected person is tested in the case of  symptomatic and asymptomatic individuals, and $\pi_t^\mathrm{FP}$ is the probability at time $t$ that a non-infected person falsely tested positive.
\begin{align}
\pi_t^\mathrm{IST} & = \frac{ \delta (\lambda^\mathrm{Q} \, M^\mathrm{IA-Q}_{t-1} + \lambda \, M^\mathrm{IA-NQ}_{t-1})}{M_t^\mathrm{I}} \nonumber \\
\pi_t^\mathrm{IAT} &= \frac{\lambda \, M_{t-1}^\mathrm{IA-NQ}\cdot \tau \,q^{sens} + \lambda^\mathrm{Q} \, M_{t-1}^\mathrm{IA-Q}\cdot \tau \,q^{sens}}{M_t^\mathrm{I}} \nonumber \\
\pi_t^\mathrm{FP} &= \frac{\lambda \, M_{t-1}^\mathrm{NA-NQ}\cdot \tau \,(1-q^{spec}) + \lambda^\mathrm{Q} \, M_{t-1}^\mathrm{NA-Q}\cdot \tau \,(1- q^{spec})}{M_t^\mathrm{N}} \, .
\end{align}
In order to avoid double counting, only the new infected symptomatic individuals at time $t$ are accounted in the formulation of the diagnosis probability $\pi_t^\mathrm{IST}$, which depends on multiple compartments at time $t-1$. The same idea is used in the formulation of $\pi_t^\mathrm{IAT}$ and $\pi_t^\mathrm{FP}$. We assume here that a symptomatic person is correctly diagnosed with probability 1. Note that by including test-and-trace into the model, the difference equations can now be written as $X_{t+1} = f(X_t, X_{t-1})$.

The quarantine rate $\xi^{CT}$ would work for a situation, where the share of infected people  in those who have been contact traced would be the same that in total population. This is however not the case as both the risk of infection and chance of getting traced increases when you meet an infected person. Therefore, the transition rate $\xi^{CT}$ must be split into two different rates, $\xi^{CT, I}$ and $\xi^{CT, N}$ for infected and non-infected compartments respectively. This can be modeled by adjusting the rates as $\xi^{CT, I} = w^I \xi^{CT}$ and $\xi^{CT, N} = w^N \xi^{CT}$

\subsection{Lauri's new formulation}
One thing to note is that tracing probability and infection probability are affected by the same meeting. This means that the meeting rate should only be counted once for those who get traced, they don't neeed a separate meeting for getting traced after having met the person who gives (or doesn't) them the infection.

The following transition paths can now be defined from NA-NQ:

\textbf{Path 1: meeting a newly diagnosed infected person and getting infected and traced}: The chance of having met such a person on last time step, given a meeting, is
\begin{align}
	\pi_t^\mathrm{IST} &= \frac{\delta (\lambda^\mathrm{Q} \, M^\mathrm{IA-Q}_{t-1} + \lambda \, M^\mathrm{IA-NQ}_{t-1})}{ M_t^\mathrm{total}}\nonumber  \\
	\pi_t^\mathrm{IAT} &= \frac{\lambda \, M_{t-1}^\mathrm{IA-NQ}\cdot \tau \,q^{sens} + \lambda^\mathrm{Q} \, M_{t-1}^\mathrm{IA-Q}\cdot (\tau + \tau^{CT}) \,q^{sens}}{M_t^\mathrm{total}} \nonumber \\
	\pi_t^\mathrm{FP} &= \frac{\lambda \, M_{t-1}^\mathrm{NA-NQ}\cdot \tau \,(1-q^{spec}) + \lambda^\mathrm{Q} \, M_{t-1}^\mathrm{NA-Q}\cdot (\tau + \tau^{CT}) \,(1- q^{spec})}{M_t^\mathrm{total}}
\end{align}
Meeting such a person (out of all meetings) can lead to infections  at conditional rate $\rho^S$, giving an infection rate per random meeting:
\begin{equation}
  \alpha^{IS}_{t-1} = \pi^{IST}_{t-1}\rho^S + \pi^{IAT}_{t-1}\rho^A
\end{equation}
At this point these persons will move to IANQ compartment, at rate:
\begin{equation}
	\lambda\alpha^{IS}_{t-1} = \lambda \pi^{IST}_{t-1}\rho^S
\end{equation} and the mass calculated below. This mass is the one infected mass that can be traced on the next time step. It is a 'compartment in compartment' within IA-NQ.
\begin{equation}
	M_t^{TI} = \lambda\alpha^{IS}_{t-1} M_{t-1}^{NA-NQ} = \lambda (\pi^{IST}_{t-1}\rho^S + \pi^{IAT}_{t-1}\rho^A)M_{t-1}^{NA-NQ}
\end{equation}
In our model, this transition happens as part of the total transition from NA-NQ to IA-NQ, happening at rate
\begin{equation}
	\lambda \alpha_{t-1} = \lambda \pi_t^I[\pi_t^{IS} \rho^S + \pi_t^{IA} \rho^A]
\end{equation}
from where they may proceed to IA-Q, based on the tracing efficiency $\eta$.

This means that only part of the 'new arrivals' , and therefore also only part of the total mass in IA-NQ are subject to potential tracing efforts. Also, at this point the chance that they have already met an infected person is accounted for (because we know they have, that's how they got infected). Considering the share of these new members in the whole group of infected persons in IANQ leads to transition rate of:
\begin{equation}
\xi^{CT, I} = \eta   M^\mathrm{IT}_t  / M^\mathrm{IA-NQ}_t
\end{equation}
to IA-Q.
Fully extended, this becomes:
\begin{equation}
\xi^{CT, I} = \eta \cdot \lambda \rho^S   \frac{\lambda^{Q} (\delta  + (\tau + \tau^{CT}) \,q^{sens}) M^{IA-Q}_{t-2} + \lambda(\delta  + \tau \,q^{sens} ) M^{IA-NQ}_{t-2}) }{M_{t-1}^{total}} \frac{M_{t-1}^{NA-NQ}}{M^{IA-NQ}_t}
\end{equation}
\\
\\
\textbf{Path 1: meeting a newly diagnosed infected person and getting traced but not infected}:

This process moves on similarly to the one above. Instead of infection chance, one only needs to consider the complement, and instead of target compartment IA-NQ, the compartment NA-NQ. This gives:
\begin{equation}
	\xi^{CT, N} = \eta  M^\mathrm{NT}_t / M^\mathrm{NA-NQ}_t
\end{equation}

One way to think about this would be to build pass-through compartments for 'traceable' IA-NQ and NA-NQ that would be kept separate from the 'main' IN-NQ and NA-NQ compartments. The only transitions to quarantine through test and trace would then be from these 'traceable' compartments with rate $\eta$ (rest would go to main compartments because their 'traceability' would be lost). These both structures (with and without extra compartments) should lead to same transition \emph{massess} to quarantine compartments. This might clarify this process but is perhaps not necessary.

\end{document}


% ---------------------------------------------------------------------
\documentclass{article}

\usepackage{nberpreamble}
\usepackage{tikz}
\usetikzlibrary{
  arrows,
  patterns,
  positioning,
  calc,
  fit,
  intersections,
  decorations.text,
  decorations.markings,
  decorations.pathmorphing,
  shadows.blur
}

\title{\bfseries Notes on Parametric Power \\ for Clustered Randomization}
\author{\sffamily Prepared by Mauricio C\'aceres}
\date{\sffamily \today}

\renewcommand{\displayoptions}{
  \maketitle
  \pagenumbering{arabic}
}

% ---------------------------------------------------------------------
\begin{document}
\displayoptions

%----------------------------------------------------------------------
\section{MDE and Sample Size with a Cluster Design}
\label{sec:mde_and_sample_size_with_a_cluster_design}

Recall \Cref{eq:cluster_reg}; without controls we have
\begin{equation}
\label{eq:cluster_reg}
Y_{ij} = \alpha + \beta T_j + \varepsilon_{ij},
\end{equation}

with $E (\varepsilon_{ij} \varepsilon_{i^\prime j}) \ne 0$ and the randomization $T_j$ at the cluster level ($j = 1, \ldots, J$ clusters and $n_j$ individuals per cluster, $\sum^{J}_{j = 1} n_j = N$). Following \citet[p. 3921-2]{DufloGlennersterKremer2007}, suppose we can additively decompose the error term $\varepsilon_{ij}$ as
\begin{equation}
\label{eq:cluster_reg_additive}
Y_{ij} = \alpha + \beta T_j + v_j + u_{ij},
\end{equation}

with $v_j \stackrel{iid}{\sim} (0, \sigma_v^2), u_{ij} \stackrel{iid}{\sim} (0, \sigma_u^2)$. \citeauthor{DufloGlennersterKremer2007} provide a formula for the case when we have equal cluster sizes, but not the case when cluster sizes vary. Let us further assume $n_j \stackrel{iid}{\sim} (\mu_n, \sigma^2_n)$ and $\rho \equiv \sigma_v^2 / \sigma_\varepsilon^2$ so that $E (\varepsilon_{ij} \varepsilon_{i^\prime j}) = \rho \sigma_\varepsilon^2$. For significance level $\alpha$, power $\kappa$, and proportion randomized $P$ we have
\begin{equation}
  \begin{array}{rl}
    J   & = \left(\dfrac{t_{1 - \kappa} + t_{\alpha / 2}}{MDE}\right)^2 \dfrac{DE \cdot \sigma_\varepsilon^2}{\mu_n P(1 - P)}
          = N_0 \dfrac{DE}{\mu_n} \\[9pt]
    MDE & = \Fabs{t_{1 - \kappa} + t_{\alpha / 2}} \sqrt{\dfrac{DE \cdot \sigma_\varepsilon^2}{J \mu_n P(1 - P)}}
          = MDE_0 \cdot \sqrt{DE},
  \end{array}
\end{equation}

where $MDE_0, N_0$ are the MDE and sample size required if the model used individual data and $DE$ is the so-called design effect (DE) or variance inflation factor (VIF):
\begin{equation}
  DE =
  \begin{cases}
    1 + \rho (\mu_n - 1) & \sigma^2_n = 0 \\
    1 + \rho ((\sigma^2_n / \mu_n^2 + 1) \mu_n - 1) & \sigma^2_n > 0
  \end{cases}.
\end{equation}

The formulas above are a slight modification of equations (1) and (3) in \citet{manatunga_sample_2001} to account for the case when the proportion randomized $P \ne 0.5$. Note we're leveraging the fact that testing the significance of $\widehat{\beta}_{OLS}$ is equivalent to a paired $t$-test in this case. If $Y_{ij}$ is also binary then we need to adjust the variance. Equation (3) in \citet{kong_sample_2003} gives
\begin{equation}
J = \left(\dfrac{t_{1 - \kappa} + t_{\alpha / 2}}{MDE}\right)^2 \dfrac{DE}{\mu_n P(1 - P)}
    \Big(\mu_T (1 - \mu_T) (1 - P) + \mu_C (1 - \mu_C) P\Big).
\end{equation}

Again, we modify the formula slightly so we account for $P \ne 0.5$. Note that the variance of $\widehat{\beta}_{OLS}$ is
\begin{equation}
\begin{array}{rl}
    V_{\widehat{\beta}} & = \widehat{V}_T + \widehat{V}_C \\
    V_{k} & = \dfrac{\sum^{J}_{j = 1} n_{ik} \left(1 + (n_{jk} - 1) \rho\right)}{\left(\sum^{J}_{j = 1} n_{jk}\right)^2} \sigma^2_{\varepsilon}
        \quad\quad k = T, C.
  \end{array}
\end{equation}

$J V_{\widehat{\beta}} \xrightarrow{P} \sigma^2_{\varepsilon} DE / \mu_n$ as $J \to \infty$, meaning we can estimate $DE$ if the cluster sizes are known, given an estimate of $\rho$. \citeauthor{kong_sample_2003} suggests using an ANOVA-based estimate. Intuitively, from \cref{eq:cluster_reg_additive} we see that a random effects model $Y_{ij} = \alpha + v_j + u_{ij}$ is the true model under the null of $H_0: \beta = 0$; then $\rho = \sigma_v^2 / (\sigma_u^2 + \sigma_v^2)$.

%----------------------------------------------------------------------
\section{The Effect of Covariates}
\label{sec:the_effect_of_covariates}

Adding controls has the effect of absorbing some of the variation in the error terms, thereby reducing $\sigma_\varepsilon^2$ and $\rho$ and improving the precision of our estimates. The proportion of the unexplained variation absorbed by the covariates will be $R^2$, meaning we can adjust our parametric estimates to account for covariates by multiplying the variance by $1 - R^2$.

For the intra-cluster correlation $\rho$ there is no trivial way of accounting for an arbitrary number of covariates---however, it is possible to make a simple adjustment for any one covariate. We follow the approach outlined in \citet{stanish_estimation_1983} and adjust the ANOVA-based estimate for $\rho$ to account for the lag of the outcome variable. Taking $MSB$ and $MSW$ as they are usually defined, we have
\begin{align*}
  n & = \dfrac{1}{J - 1} \left[N - \sum^{}_{j} n_j^2 / N \right] \\
  k & = \dfrac{1}{J - 1} \left[\dfrac{\sum^{}_{j} n_j^2 (\overline{x}_j - \overline{x})^2}{\sum^{}_{j} \sum^{}_{i} (x_{ij} - \overline{x})^2}\right] \\
  \widehat{\rho} & = \dfrac{MSB - MSW}{MSB + (n - k - 1) MSW},
\end{align*}

where the unadjusted $\widehat{\rho}$ would simply use $k = 0$.

%----------------------------------------------------------------------
% \clearpage

\bibliographystyle{apalike}
\bibliography{power.bib}

%----------------------------------------------------------------------
\end{document}


\documentclass{article}

\usepackage[T1]{fontenc}
\usepackage[osf]{libertine}
\usepackage[scaled=0.8]{beramono}
\usepackage[margin=1.5in]{geometry}
\usepackage{url}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{nicefrac}
\usepackage{microtype}
\usepackage{subcaption}
\usepackage{bm}

\usepackage{sectsty}
\sectionfont{\large}
\subsectionfont{\normalsize}

\usepackage{titlesec}
\titlespacing{\section}{0pt}{10pt plus 2pt minus 2pt}{0pt plus 2pt minus 0pt}
\titlespacing{\subsection}{0pt}{5pt plus 2pt minus 2pt}{0pt plus 2pt minus 0pt}

\usepackage{pgfplots}
\pgfplotsset{
  compat=newest,
  plot coordinates/math parser=false,
  tick label style={font=\footnotesize, /pgf/number format/fixed},
  label style={font=\small},
  legend style={font=\small},
  every axis/.append style={
    tick align=outside,
    clip mode=individual,
    scaled ticks=false,
    thick,
    tick style={semithick, black}
  }
}

\pgfkeys{/pgf/number format/.cd, set thousands separator={\,}}

\usepgfplotslibrary{external}
\tikzexternalize[prefix=tikz/]

\newlength\figurewidth
\newlength\figureheight

\setlength{\figurewidth}{12cm}
\setlength{\figureheight}{6cm}

\newlength\squarefigurewidth
\newlength\squarefigureheight

\setlength{\squarefigurewidth}{4cm}
\setlength{\squarefigureheight}{4cm}

\newlength\smallsquarefigurewidth
\newlength\smallsquarefigureheight

\setlength{\smallsquarefigurewidth}{3.25cm}
\setlength{\smallsquarefigureheight}{3.25cm}

\newlength\smallfigurewidth
\newlength\smallfigureheight

\setlength{\smallfigurewidth}{6.25cm}
\setlength{\smallfigureheight}{4cm}

\setlength{\parindent}{0pt}
\setlength{\parskip}{1ex}

\newcommand{\acro}[1]{\textsc{\MakeLowercase{#1}}}
\newcommand{\given}{\mid}
\newcommand{\mc}[1]{\mathcal{#1}}
\newcommand{\data}{\mc{D}}
\newcommand{\intd}[1]{\,\mathrm{d}{#1}}
\newcommand{\inv}{^{-1}}
\newcommand{\trans}{^\top}
\newcommand{\mat}[1]{\bm{\mathrm{#1}}}
\renewcommand{\vec}[1]{\bm{\mathrm{#1}}}
\newcommand{\R}{\mathbb{R}}
\renewcommand{\epsilon}{\varepsilon}

\DeclareMathOperator{\var}{var}
\DeclareMathOperator{\cov}{cov}
\DeclareMathOperator{\diag}{diag}
\DeclareMathOperator*{\argmin}{arg\,min}
\DeclareMathOperator*{\argmax}{arg\,max}

\begin{document}

\section*{The Kernel Trick}

Consider the assumption of linear regression with an explicit feature
expansion $\phi\colon \vec{x} \mapsto \phi(\vec{x})$:
\begin{equation*}
  y(\vec{x}) = \phi(\vec{x})\trans \vec{w} + \epsilon(\vec{x}).
\end{equation*}
Given training data $\data = \bigl\{ (\vec{x}, y) \bigr\} = (\mat{X},
\vec{y})$, recall we modeled the residuals $\epsilon(\vec{x})$ as
zero-mean independent, identically distributed Gaussians with variance
$\sigma^2$:
\begin{equation*}
  p(\vec{\epsilon})
  =
  \mc{N}(\vec{\epsilon}; \vec{0}, \sigma^2 \mat{I}),
\end{equation*}
giving rise to the following likelihood:
\begin{equation*}
  p(\vec{y} \given \mat{X}, \vec{w}, \sigma^2)
  =
  \mc{N}(\vec{y};\mat{\Phi}\trans \vec{w}, \sigma^2 \mat{I}).
\end{equation*}
where we have defined $\mat{\Phi} = \phi(\mat{X})$.

In Bayesian linear regression, we further chose a multivariate
Gaussian prior for $\vec{w}$:
\begin{equation*}
  p(\vec{w}) = \mc{N}(\vec{w}; \vec{\mu}, \mat{\Sigma}).
\end{equation*}
For simplicity, in the below we assume the prior mean for $\vec{w}$ is
$\vec{\mu} = \vec{0}$.

Given these assumptions, we were able to derive the posterior
distribution of $\vec{w}$ given the data $\data$:
\begin{equation*}
  p(\vec{w} \given \data, \sigma^2)
  =
  \mc{N}(\vec{w};
  \vec{\mu}_{\vec{w}\given\data},
  \mat{\Sigma}_{\vec{w}\given\data}
  ),
\end{equation*}
where
\begin{align*}
  \vec{\mu}_{\vec{w}\given\data}
  &=
  \mat{\Sigma}
  \mat{\Phi}\trans
  (\mat{\Phi}\mat{\Sigma}\mat{\Phi}\trans + \sigma^2 \mat{I})\inv
  \vec{y};
  \\
  \mat{\Sigma}_{\vec{w}\given\data}
  &=
  \mat{\Sigma}
  -
  \mat{\Sigma}
  \mat{\Phi}\trans
  (\mat{\Phi}\mat{\Sigma}\mat{\Phi}\trans + \sigma^2 \mat{I})\inv
  \mat{\Phi}
  \mat{\Sigma},
\end{align*}

If we wish to use our model to predict the outputs $\vec{y}_\ast$
associated with a set of inputs $\mat{X}_\ast$, we previously derived:
\begin{equation*}
  p(\vec{y}_\ast \given \mat{X}_\ast, \data, \sigma^2)
  =
  \mc{N}(
  \vec{y}_\ast;
  \mat{\Phi}_\ast \vec{\mu}_{\vec{w}\given\data},
  \mat{\Phi}_\ast \mat{\Sigma}_{\vec{w}\given\data} \mat{\Phi}_\ast\trans + \sigma^2 \mat{I}).
\end{equation*}
Examining the forms of these expressions, we see that the feature
expansion $\phi$ always appears in one of the following expressions:
\begin{equation*}
  \mat{\Phi}\mat{\Sigma}\mat{\Phi}\trans
  \qquad
  \mat{\Phi}_\ast\mat{\Sigma}\mat{\Phi}\trans
  \qquad
  \mat{\Phi}\mat{\Sigma}\mat{\Phi}_\ast\trans
  \qquad
  \mat{\Phi}_\ast\mat{\Sigma}\mat{\Phi}_\ast\trans.
\end{equation*}
The entries of these matrices are always of the form
$\phi(\vec{x})\trans \mat{\Sigma} \phi(\vec{x}')$, where $\vec{x}$ and
$\vec{x}'$ are two arbitrary inputs.  To simply our expressions, we
define a function
\begin{equation*}
  K(\vec{x}, \vec{x}')
  =
  \phi(\vec{x})\trans\mat{\Sigma}\phi(\vec{x}').
\end{equation*}
Because $\mat{\Sigma}$ is positive definite, it has a ``matrix square
root,'' $\mat{\Sigma}^{\nicefrac{1}{2}}$ with the property
$(\mat{\Sigma}^{\nicefrac{1}{2}})^2 = \mat{\Sigma}$.\footnote{You can
  prove this via the singular value decomposition (\acro{SVD}): write
  $\mat{\Sigma} = \mat{U}\mat{D}\mat{U}\trans$, where $\mat{U}$ is
  unitary and $\mat{D}$ is diagonal with positive entries (because
  $\mat{\Sigma}$ is positive definite), then define
  $\mat{\Sigma}^{\nicefrac{1}{2}} =
  \mat{U}\mat{D}^{\nicefrac{1}{2}}\mat{U}\trans$.} If we define the
function $\psi(\vec{x}) =
\mat{\Sigma}^{\nicefrac{1}{2}}\phi(\vec{x})$, we can see that
$K$ is simply an inner product:
\begin{equation*}
  K(\vec{x}, \vec{x}')
  =
  \psi(\vec{x})\trans \psi(\vec{x}').
\end{equation*}
Such a function $K$ is guaranteed to always produce positive-definite
Gram matrices (a \emph{Gram matrix} is a square matrix of inner
products between pairs of elements), and is called a \emph{kernel} or
\emph{covariance function.}

Sometimes it is possible to specify a covariance function $K$ directly
without ever computing the feature map explicitly.  With such a
function, we could perform efficient Bayesian linear regression even
with a high-dimensional (or even infinite dimensional!) feature
expansion $\phi$ \emph{implicitly.} This idea of computing inner
products in a feature space directly is called the \emph{kernel trick}
and has been the basis of a large amount of work in the
machine-learning community.  Effectively, any algorithm that operates
purely in terms of inner products between input vectors can be made
nonlinear by replacing normal inner products with the evaluation of a
kernel.

With this definition, we may rewrite the predictive distribution
for $\vec{y}_\ast$:
\begin{equation*}
  p(\vec{y}_\ast \given \mat{X}_\ast, \data, \sigma^2)
  =
  \mc{N}(
  \vec{y}_\ast;
  \mu_{\vec{y}_\ast \given \data},
  K_{\vec{y}_\ast \given \data}),
\end{equation*}
where
\begin{align*}
  \vec{\mu}_{\vec{y}_\ast\given\data}
  &=
  K(\mat{X}_\ast, \mat{X})
  \bigl(K(\mat{X}, \mat{X}) + \sigma^2 \mat{I}\bigr)\inv
  \vec{y};
  \\
  K_{\vec{y}_\ast\given\data}
  &=
  K(\vec{X}_\ast, \vec{X}_\ast)
  -
  K(\mat{X}_\ast, \mat{X})
  \bigl(K(\mat{X}, \mat{X}) + \sigma^2 \mat{I}\bigr)\inv
  K(\mat{X}, \mat{X}_\ast).
\end{align*}

\subsection*{Examples}

Perhaps the most-commonly used kernel is the \emph{squared exponential
  covariance function:}
\begin{equation*}
  K(\vec{x}, \vec{x}'; \lambda, \ell)
  =
  \lambda^2
  \exp\biggl(-\frac{\lVert \vec{x} - \vec{x}' \rVert^2}{2\ell^2}\biggr),
\end{equation*}
where $\lambda$ and $\ell$ are parameters.  The former is simply a
multiplicative scaling constant (you can think of this as an implicit
scalar multiplication in the implicit feature map $\phi$).  The latter
takes the role of a \emph{length scale;} vectors separated by more
than a couple length scales will have a kernel value near zero.

An example of Bayesian linear regression using this kernel function is
shown in Figure \ref{kernel_example}.  We see that the use of this
kernel function allowed us to achieve nice nonlinear regression
without computing explicit basis expansions.  In fact, you can show
that the squared exponential kernel corresponds to an
\emph{infinite-dimensional} basis expansion, where we use a Gaussian
basis function \emph{centered on every point.}  Such a feature
expansion would be impossible to use if we attempted to use explicit
feature computation.

\begin{figure}
  \centering
  \input{figures/kernel_example.tex}
  \caption{Example of Bayesian linear regression using the squared
    exponential covariance function.  The true function is $f =
    \sin(x)$.  The kernel parameters are $\lambda = \ell = 1$, and the
    noise variance was set to $\sigma^2 = 0.1^2$.}
  \label{kernel_example}
\end{figure}

Sometimes thinking in terms of the kernel can help even when you have
an explicit feature expansion on hand.  As an example, imagine our
inputs are binary vectors of length $n$ (so each input $\vec{x}$ is a
subset, a member of the power set $\mc{P}(n)$).  One rather expensive
feature expansion we could try would be to enumerate every member of
$\mc{P}(n)$ and define $\phi(\vec{x})_i = \vec{s}_i \subset \vec{x}$,
where $\vec{s}_i$ is the $i$th element of the power set.  So we
represent our set $\vec{x}$ by a feature vector of length $2^n$
indicating every subset of $\vec{x}$.  This is a very expensive
feature expansion, requiring exponential space to store for each
input.  However, if we take $\mat{\Sigma} = \mat{I}$, we can compute
the dot product as:
\begin{equation}
  K(\vec{x}, \vec{x}') = 2^{\lvert x \cap x' \rvert},
\end{equation}
which only requires time and space linear in $n$!

\end{document}


\documentclass[10pt]{article}
\usepackage{algorithm}
\usepackage{algpseudocode}
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage{mathptmx}
\begin{document}

\title{Supplementary Information for ``A Theoretically-Sufficient And Computationally-Practical Technique For Deterministic Frequency Seriation''}
\author{Carl P. Lipo, Mark E. Madsen, and Robert C. Dunnell}
\date{March 2015}
\maketitle

\section*{S1 Text:  Pseudo-code representation of the IDSS algorithm}

\begin{algorithm}[h]
\caption{Algorithm for IDSS seriation}\label{alg:idss}
\begin{algorithmic}[1]

 \Require  Input file $I$ format:
 \State Tab-delimited text, column 0 contains assemblage name
 \State Remaining columns contain type counts as integers

\Procedure{IDSS}{$I$}

   \State Read input file $I$
   \State Calculate relative frequency of each type
   \State Calculate max frequency difference between assemblage pairs
   \State Create list of assemblages $A$

    \ForAll{triplets of assemblages $T$}
        \If{using continuity threshold $t_c$}
            \If{max frequency difference $> t_c$ for pairs $\in T$}
            \State Skip triplet
            \EndIf
        \EndIf

        \If{triplet $T$ is valid given unimodality for all types}
            \State Store triplet in candidate solutions $\mathbf{C}$
        \EndIf

        \State $R \gets$ assemblages $\notin \mathbf{C}$
        \Comment Remaining assemblages

        \Repeat
            \ForAll{assemblages $ a \in R$}
                \If{using continuity threshold $t_c$}
                    \If{max freq difference $> t_c$ for $a$ and all $\mathbf{C}$}
                        \State Skip assemblage $a$ for this loop
                    \EndIf
                \EndIf

                \If{assemblage $a$ + candidate solution $c \in \mathbf{C}$}
                    \State Replace $c$ in $\mathbf{C}$ with $c+a$
                    \State Remove $a$ from $R$
                    \Comment Grow existing solutions
                \EndIf

            \EndFor

        \Until{ $R = \empty$ or loop repeats with no changes }

            \ForAll{candidate solutions $c \in \mathbf{C}$}
                \If{$c$ is strict subset of another solution in $\mathbf{C}$}
                    \State Remove $c$ from $\mathbf{C}$
                \EndIf
            \EndFor

    \EndFor
    \Comment $\mathbf{C}$ now contains the set of solutions

    \State Output $\mathbf{C}$ in various formats

\EndProcedure
\end{algorithmic}
\end{algorithm}

\end{document}


\subsection{Down sampling}


\chapter{Setting up the linear system}
\label{section:setup-linear-system}
\par
Our typical user is interested in solving $A X = Y$,
where $A$ is square, large and sparse,
and $X$ and $Y$ are dense matrices with one or more columns.
{\bf SPOOLES} is a very large sophisticated library with
a commensurate learning curve to master its functionality.
But what is the bare minimum a user has to know to obtain a
solution to their linear system?
\begin{itemize}
\item
They need to construct an {\tt InpMtx} object that holds
the entries of $A$. ({\tt InpMtx} stands for
{\tt Inp}ut {\tt m}a{\tt t}ri{\tt x},
for it is an easy to use object that one uses to input,
assemble, sort and manipulate entries in a sparse matrix.)
\item
They need to construct a {\tt DenseMtx} object that holds
the entries of $Y$.
\item
They need to construct a {\tt DenseMtx} object to hold
the entries of $X$.
\end{itemize}
These two objects encapsulate the minimal interface to the
{\bf SPOOLES} library.
the application program needs to know how to construct
the {\tt InpMtx} and {\tt DenseMtx} objects, either directly inside
an application program, or by reading in a custom matrix file.
This is what we now describe.
\par
\section{Constructing an {\tt InpMtx} object}
\label{subsection:construct-InpMtx}
\par
The {\tt InpMtx} object is more of an ``Input'' object
than a ``Matrix'' object.
It descended from an out-of-core assembly code that assembled and
sorted entries of a sparse matrix.
Simplicity and functionality are its goals, at some expense of
efficiency in storage and computation.
{\it Note: all indices are zero-based as in C, not 1-based as in
FORTRAN.}
\par
The {\tt InpMtx} object is simplest understood as a ``bag'' of
triples $\langle r(i,j),c(i,j),a_{i,j}\rangle$,
where $r()$ and $c()$ are some
functions that define the first and second coordinates.
Each {\tt InpMtx} object has a ``coordinate type'', one of
\begin{itemize}
\item
{\tt INPMTX\_BY\_ROWS}, where $r(i,j) = i$, $c(i,j) = j$.
\item
{\tt INPMTX\_BY\_COLUMNS}, where $r(i,j) = j$, $c(i,j) = i$.
\item
{\tt INPMTX\_BY\_CHEVRONS},
where $r(i,j) = \min(i,j)$, $c(i,j) = j - i$.
\end{itemize}
Rows and columns are self-explanatory, the first coordinate
$r(i,j)$ is either the row or column of $a_{i,j}$.
The $j$-th ``chevron'' is composed of the diagonal entry $a_{j,j}$,
entries in the $j$-th row of the upper triangle,
and entries in the $j$-th column of the lower triangle.
It is the natural data structure for the assembly of the matrix
entries into the ``fronts'' used to factor the matrix.
\par
% ``Entries'' of the {\tt InpMtx} object can be one of three types.
The {\tt InpMtx} object can hold one of three types of entries as
``indices only'' (no entries are present),
real entries, or complex entries.
The type is specified by the {\tt inputMode}
parameter to the {\tt InpMtx\_init()} method.
\begin{itemize}
\item
{\tt INPMTX\_INDICES\_ONLY} where the triples
$langle r(i,j),c(i,j),-\rangle$ are really only pairs,
i.e., no numerical values are present.
This mode is useful for assembling graphs.
\item
{\tt SPOOLES\_REAL} where $a_{i,j}$ is a real number,
a {\tt double} value.
\item
{\tt SPOOLES\_COMPLEX} where $a_{i,j}$ is a complex number,
really two consecutive {\tt double} values.
\end{itemize}
``Coodinate type'' and ``input mode'' (equivalently, the type of
entries) are the two parameters that must be specified when
initializing an {\tt InpMtx} object.
\begin{verbatim}
InpMtx   *mtxA = InpMtx_new() ;
InpMtx_init(mtxA, coordType, inputMode, 0, 0) ;
\end{verbatim}
Every object in the {\bf SPOOLES} library is initialized
via an {\tt {\it ObjectName}\_new()} method, which allocates space
for the object and sets its fields to default values.
If you wish to use an {\it automatic} variable, then one must
explicitly set the default fields, as follows.
\begin{verbatim}
InpMtx   mtxA ;
InpMtx_setDefaultFields(&mtxA) ;
InpMtx_init(&mtxA, coordType, inputMode, 0, 0) ;
\end{verbatim}
Only the coordinate type and input mode are necessary.
The fourth and fifth arguments are upper bounds on the number of
entries and vectors for the object. (More on vectors in just a
moment.) The user does not need to know values for the number of
entries or vectors, for the object resizes itself as necessary
as information is placed into it.
\par
``Vectors'' is one way that the entries can be stored.
There are actually three ways, specified by the
{\tt storageMode} field of the {\tt InpMtx} object.
\begin{itemize}
\item
{\tt INPMTX\_RAW\_DATA}, where the pairs or triples are stored in
unordered form.
\item
{\tt INPMTX\_SORTED}, where the pairs or triples are stored in
ascending lexicographic order of the first two coordinates.
\item
{\tt INPMTX\_BY\_VECTORS}, where the pairs or triples are sorted
and stored in vectors defined by their first coordinate.
\end{itemize}
The storage mode can be changed via a call to
{\tt InpMtx\_changeStorageMode()}.
\par
The user does not really need to know about this
``storage mode''. Fill the {\tt InpMtx} object with data in any way
at all (we will describe this shortly).
The wrapper method will check that the data is in the form it needs.
If is isn't, the object will be transformed as necessary.
The ``sort'' operation is really ``sort-and-compress'', the pairs
or triples are sorted into ascending order, and then the list is
scanned duplicates are ``merged'' together, i.e., if real or
complex entries are present, they are added together.
(This allows us to assemble finite element matrices.)
The knowledgeable user can change the storage mode as necessary,
and thus avoiding expensive sorts when possible.
For example, after reading in the matrix data from the CSAR-Nastran
file,
the entries are already in sorted form, and the explicit sort
can be avoided.
\par
Now let us see how we ``input'' information into the {\tt InpMtx}
object.
There are several input methods, e.g., single entries, rows,
columns, and submatrices, and each input method has three types of
input, e.g, indices only, real entries, or complex entries.
Here are the prototypes below.
\begin{itemize}
\item Input methods for ``indices only'' mode.
\begin{verbatim}
void InpMtx_inputEntry ( InpMtx *mtxA, int row, int col ) ;
void InpMtx_inputRow ( InpMtx *mtxA, int row, int rowsize, int rowind[] ) ;
void InpMtx_inputColumn ( InpMtx *mtxA, int col, int colsize, int colind[] ) ;
void InpMtx_inputMatrix ( InpMtx *mtxA, int nrow, int ncol, int rowstride,
                          int colstride, int rowind[], colind[] ) ;
\end{verbatim}
\item Input methods for real entries.
\begin{verbatim}
void InpMtx_inputRealEntry ( InpMtx *mtxA, int row, int col, double value ) ;
void InpMtx_inputRealRow ( InpMtx *mtxA, int row, int rowsize,
                           int rowind[], double rowent[] ) ;
void InpMtx_inputRealColumn ( InpMtx *mtxA, int col, int colsize,
                              int colind[], double colent[] ) ;
void InpMtx_inputRealMatrix ( InpMtx *mtxA, int nrow, int ncol, int rowstride,
                    int colstride, int rowind[], colind[], double mtxent[] ) ;
\end{verbatim}
\item Input methods for complex entries.
\begin{verbatim}
void InpMtx_inputComplexEntry ( InpMtx *mtxA, int row, int col,
                                double real, double imag ) ;
void InpMtx_inputComplexRow ( InpMtx *mtxA, int row, int rowsize,
                               int rowind[], double rowent[] ) ;
void InpMtx_inputComplexColumn ( InpMtx *mtxA, int col, int colsize,
                                 int colind[], double colent[] ) ;
void InpMtx_inputComplexMatrix ( InpMtx *mtxA, int nrow, int ncol, int rowstride,
              int colstride, int rowind[], colind[], double mtxent[] ) ;
\end{verbatim}
\end{itemize}
The {\tt rowind[]} row indices and {\tt colind[]} column indices
are precisely that.
Don't worry about what coordinate type the {\tt InpMtx} object has,
the translation from row and column indices into the particular
coordinate is done inside the input methods.
\par
Let us look at a particular example, where we have a
${\tt n1} \times {\tt n2}$ grid and we want to have a
$\left \lbrack \begin{array}{ccc}
   & -1 &    \\
-1 &  4 & -1 \\
   & -1 &
\end{array} \right \rbrack$
5-point operator at each grid point.
Note, this matrix is symmetric, so we need input only the upper
triangle (or the lower triangle) of the matrix.
\begin{verbatim}
mtxA = InpMtx_new() ;
InpMtx_init(mtxA, INPMTX_BY_ROWS, SPOOLES_REAL, 0, 0) ;
for ( ii = 0 ; ii < n1 ; ii++ ) {
   for ( jj = 0 ; jj < n2 ; jj++ ) {
      ij = ii + jj*n1 ;
      indices[0] = ij ;
      entries[0] = 4.0 ;
      count = 1 ;
      if ( ii < n1 ) {
         indices[count] = ij + 1 ;
         entries[count] = -1.0 ;
         count++ ;
      }
      if ( jj < n2 ) {
         indices[count] = ij + n1 ;
         entries[count] = -1.0 ;
         count++ ;
      }
      InpMtx_inputRealRow(mtxA, ij, count, indices, entries) ;
   }
}
InpMtx_changeStorageMode(mtxA, INPMTX_BY_VECTORS) ;
\end{verbatim}
The process begins by allocating an {\tt InpMtx} object {\tt mtxA}
using the {\tt InpMtx\_new()} method,
initializing it with the {\tt InpMtx\_init()} method,
and filling it with matrix entries with the {\tt
InpMtx\_inputRealRow()} method.
The last method, {\tt InpMtx\_changeStorageMode()},
``assembles'' the data (not really necessary because the entries
are disjoint,
``sorts'' the data (again not necessary since the entries were
input in ascending order,
and creates a vector structure inside the {\tt InpMtx} object that
allows easy access to each individual row.
\par
We could have input all the entries and treated it as a
nonsymmetric matrix, but that would not be efficient with respect
to storage or factorization cost.
Alternatively, we could have input all the entries and called the
{\tt InpMtx\_dropLowerTriangle()} method to drop the lower
triangular entries.
\par
\section{Constructing an {\tt DenseMtx} object}
\label{subsection:construct-DenseMtx}
\par
The {\tt DenseMtx} stores a real or complex dense matrix.
It is not just an array of numbers, it also has row indices and
column indices.
This allows it to exist in a distributed MPI environment where each
processors has only a submatrix of the matrix.
Here is how to initialize a {\tt DenseMtx} object.
\begin{verbatim}
int        type, rowid, colid, nrow, ncol, inc1, inc2 ;
DenseMtx   *mtx = DenseMtx_new() ;
DenseMtx_init(mtx, type, rowid, colid, nrow, ncol, inc1, inc2) ;
\end{verbatim}
\begin{itemize}
\item
The {\tt type} is either {\tt SPOOLES\_REAL}
or {\tt SPOOLES\_COMPLEX}.
\item
The {\tt rowid} and {\tt colid} values are used to identify a
{\tt DenseMtx} as a submatrix of a larger matrix.
Any values are suitable.
\item
{\tt nrow} and {\tt ncol} are the number of rows and columns in the
matrix, respectively.
\item
The entries of the matrix can be stored in either row major or
column major form.
For row major, use {\tt inc1 = ncol} and {\tt inc2 = 1}.
For column major, use {\tt inc1 = 1} and {\tt inc2 = nrow}.
Note, all solve and matrix-matrix multiply methods require that the
{\tt DenseMtx} object be column major.
\end{itemize}
For example, here is the call to initialize a {\tt DenseMtx} object
to have real entries, 100 rows and 5 columns, entries column major.
\begin{verbatim}
DenseMtx_init(mtx, SPOOLES_REAL, 0, 0, 100, 5, 1, 100) ;
\end{verbatim}
During the initialization,
the row indices are set to $0, 1, \dots, \mathtt{nrow - 1}$
and the column indices are set to $0, 1, \dots, \mathtt{ncol - 1}$.
The entries are {\bf not} initialized.
Zero the entries with a call to {\tt DenseMtx\_zero()}.
(This is crucial when loading a sparse right hand side into
the {\tt DenseMtx} object.)
\par
Once we have the {\tt DenseMtx} object initialized, we want to be
able to access the row indices, the column indices and the entries.
We do this through instance methods.
\begin{verbatim}
void DenseMtx_rowIndices ( DenseMtx *mtx, int *pnrow, int *prowind ) ;
void DenseMtx_columnIndices ( DenseMtx *mtx, int *pncol, int *pcolind ) ;
double * DenseMtx_entries ( DenseMtx *mtx ) ;
\end{verbatim}
We would use them as follows.
\begin{verbatim}
double   *entries ;
int      ncol, nrow, *colind, *rowind ;

DenseMtx_rowIndices(mtx, &nrow, &rowind) ;
DenseMtx_columnIndices(mtx, &ncol, &colind) ;
entries = DenseMtx_entries(mtx) ;
\end{verbatim}
We can now fill the indices or the entries.
The location of the {\tt (irow,jcol)} entry is found
at {\tt offset = irow*inc1 + jcol*inc2}.
The row and column increments can be found as follows.
\begin{verbatim}
int inc1 = DenseMtx_rowIncrement(mtx) ;
int inc2 = DenseMtx_columnIncrement(mtx) ;
\end{verbatim}
\par
To avoid dealing with row and column increments,
we can retrieve and set values of a particular entry.
\begin{verbatim}
double   value, real, imag ;
int      irow, jcol ;

DenseMtx_realEntry(mtx, irow, jcol, &value) ;
DenseMtx_complexEntry(mtx, irow, jcol, &real, &imag) ;
DenseMtx_setRealEntry(mtx, irow, jcol, value + 10.) ;
DenseMtx_setComplexEntry(mtx, irow, jcol, real + 1., imag + 2.) ;
\end{verbatim}
As a real example, consider the ${\tt n1} \times {\tt n2}$ grid
from the previous subsection, where we assembled a finite
difference matrix.
Assume that the right hand side is zero except for points where
{\tt (n1-1,0:n2-1)}, where a unit load is applied.
Here is the code to generate the {\tt DenseMtx} object.
\begin{verbatim}
mtxY = DenseMtx_new();
DenseMtx_init(mtxY, SPOOLES_REAL, 0, 0, n1*n2, 1, 1, n1*n2) ;
DenseMtx_zero(mtxY) ;
ii = n1 - 1 ;
for ( jj = 0 ; jj < n2 ; jj++ ) {
   ij = ii + jj*n1 ;
   DenseMtx_setRealEntry(mtxY, ij, 1, 1.0) ;
}
\end{verbatim}
Do not forget to zero the entries in {\tt mtxY} before setting any
entries.
\par
\section{IO for the {\tt InpMtx} and {\tt DenseMtx} objects}
\label{subsection:IO}
\par
The three driver programs that we describe in the next sections
read $A$ and $Y$ from files and write $X$ to a file.
So the first thing we know is that the {\tt InpMtx} and {\tt
DenseMtx} objects can read and write themselves from and to files.
This convention is supported by most of the objects in the
{\bf SPOOLES} library.
In fact, there is a common {\it protocol} that is followed.
Let us take a look at the common IO methods for the {\tt InpMtx}.
\begin{itemize}
\item
{\tt int InpMtx\_readFromFile ( InpMtx *obj, char *filename ) ;}
\item
{\tt int InpMtx\_readFromFormattedFile ( InpMtx *obj, FILE *fp ) ;}
\item
{\tt int InpMtx\_readFromBinaryFile ( InpMtx *obj, FILE *fp ) ;}
\item
{\tt int InpMtx\_writeToFile ( InpMtx *obj, char *filename ) ;}
\item
{\tt int InpMtx\_writeToFormattedFile ( InpMtx *obj, FILE *fp ) ;}
\item
{\tt int InpMtx\_writeToBinaryFile ( InpMtx *obj, FILE *fp ) ;}
\item
{\tt int InpMtx\_writeForHumanEye ( InpMtx *obj, FILE *fp ) ;}
\end{itemize}
There are corresponding methods for the {\tt DenseMtx} object,
just replace ``{\tt Inp}'' by ``{\tt Dense}'' in the above prototypes.
\par
Two methods take as input {\tt char *} file names. Each object can
be archived in its own file with a particular suffix.
For example, {\tt InpMtx} objects can be read from and written to
files of the form {\tt *.inpmtxf} for a formatted file and
{\tt *.inpmtxb} for a binary file.
For a {\tt DenseMtx} object, the file names are
{\tt *.densemtxf} and {\tt *.densemtxb}.
The {\tt InpMtx\_readFromFile()} method looks at the {\tt filename}
argument, and calls the binary or formatted read methods, depending on
the suffix of {\tt filename}.
A normal return code is {\tt 1}.
If the suffix does not match either {\tt *.inpmtxf} or {\tt *.inpmtxb},
an error message is printed and the return code is {\tt 0}.
Something similar works for writing an {\tt InpMtx} object to a
file using {\tt InpMtx\_writeToFile()}, except if {\tt filename}'s
suffix does not match, the {\tt InpMtx\_writeForHumanEye()} method
is called.
\par
Here are three approaches to link $A$ and $Y$ from an application
code to the {\tt InpMtx} and {\tt DenseMtx} objects demanded by the
{\bf SPOOLES} application.
\begin{itemize}
\item
An application could take the simple approach of creating an {\tt
InpMtx} and {\tt DenseMtx} object to hold $A$ and $Y$, write them
to a file, and then call a totally separate code that functions
much like our drivers, reading in $A$ and $Y$, computing $X$ and
writing $X$ to a file, which is then read in by the application code.
\item
A second approach, one that was taken during the first integration
of the {\bf SPOOLES} library into CSAR-Nastran, was to have the
CSAR-Nastran code generate two files for $A$ and $Y$ in CSAR-Nastran
format.
(This way CSAR-Nastran did not need to know any of the {\bf SPOOLES}
interface.)
Two custom routines were written to read in the entries of $A$ and
$Y$ from the CSAR-Nastran files and construct {\tt InpMtx} and {\tt
DenseMtx} objects.
The wrapper routines we describe in the next three chapters were
called to solve for $X$ which was then written to a CSAR-Nastran file.
\item
A third approach would be to generate the {\tt InpMtx} and {\tt
DenseMtx} objects in the application program, and then call the
wrapper methods to solve for $X$, i.e., no IO would be necessary.
\end{itemize}


\section{Homogenization and the Two-Scale Approach}
\label{sec:61homogenization}

\minitoc{70mm}{4}

\noindent
We roughly follow the presentation given in
\multicite{%
  Huebner14Mehrdimensionale,%
  Valentin14Hierarchische,%
  Valentin16Hierarchical%
}.
The necessary notation is summarized in
\cref{tbl:glossaryTopologyOptimization}.

\begin{table}
  \setnumberoftableheaderrows{0}%
  \newcommand*{\pnst}[1]{\printnotationsymbol{#1}&\printnotationtext{#1}}%
  \begin{tabular}{%
    >{\kern\tabcolsep}=l<{\kern-1.5mm}+l<{\kern2.9mm}+l<{\kern-1.5mm}+l%
    <{\kern2.9mm}+l<{\kern-1.25mm}+l<{\kern\tabcolsep}%
  }
    \toprulec
    \pnst{\objdomain}&       \pnst{\force}&        \pnst{\densglobal}\\
    \pnst{\dimobjdomain}&    \pnst{\displacement}& \pnst{\denscell}\\
    $d$&\#micro-cell param.& \pnst{\compliance}&   \pnst{\densub}\\
    \pnst{\mcp}&             \pnst{\vol}&          \pnst{\etensor}\\
    &&                       \pnst{\voldens}&      \pnst{\cholfactor}\\
    \bottomrulec
  \end{tabular}%
  \caption[Glossary for topology optimization]{%
    Glossary of the notation for topology optimization.%
  }%
  \label{tbl:glossaryTopologyOptimization}%
\end{table}

\subsection{Homogenization}
\label{sec:611homogenization}

\paragraph{Density function}

Let $\objdomain \subset \real^{\dimobjdomain}$ be the object domain.%
\footnote{%
  We use tildes to denote variables and quantities
  that correspond to the object domain $\objdomain$
  (e.g., $\tilde{\*x}$ is a point in $\objdomain$).
  In contrast, variables without a tilde will correspond
  to the sparse grid domain $\clint{\*0, \*1} = \clint{0, 1}^d$
  (e.g., $\gp{\*l,\*i} \in \clint{\*0, \*1}$ will be a sparse grid point).%
}
Usually, we assume $\dimobjdomain = 2$ or $\dimobjdomain = 3$,
although the method can be generalized to
arbitrary dimensionalities $\dimobjdomain \in \nat$.
Shapes and topologies are described by \term{density functions}
$\densglobal\colon \objdomain \to \clint{0, 1}$.
The function values $\densglobal(\tilde{\*x}) \in \clint{0, 1}$
tell if $\tilde{\*x}$ is contained in the object (value of one) or
not (value of zero).
The \term{homogenization} approach also allows values between
zero and one, giving the physical density of the material in $\tilde{\*x}$.

\paragraph{Optimization of compliance values}

Furthermore, for every density function $\densglobal$,
let $\compliance(\densglobal)$ be an objective function value.
In our setting, which is shown in \cref{fig:topoOptExample},
we exert a force $\force$ on the object,
measure the resulting deformation, and
compute the \term{compliance} (i.e., the inverse of the stiffness) as
the objective function value~$\compliance(\densglobal)$:
\begin{equation}
  \compliance(\densglobal)
  = \int_{\objdomain} \tr{\force} \displacement_{\densglobal}(\tilde{\*x})
  \diff\tilde{\*x},
\end{equation}
where the \term{displacement function}
$\displacement_{\densglobal}\colon \objdomain \to \real^{\dimobjdomain}$
depends on the density \cite{Huebner14Mehrdimensionale}.
We want to find the density function
with the minimal compliance value:
\begin{equation}
  \label{eq:topoOptProblemContinuous}
  \min_{\densglobal}\, \compliance(\densglobal).
\end{equation}
If we do not impose additional conditions,
then there are often uninteresting trivial solutions.
For example, choosing $\densglobal :\equiv 1$
(i.e., filling the entire domain $\objdomain$ with material)
usually leads to the topology with the
highest stiffness and, thus, the smallest displacement and compliance value.
Therefore, we introduce the following volume constraint:
\begin{equation}
  \frac{\voldens{\densglobal}{\objdomain}}{\vol{\objdomain}} \le \densub,\quad
  \voldens{\densglobal}{\objdomain}
  \ceq \int_{\objdomain} \densglobal(\tilde{\*x}) \diff\tilde{\*x},\quad
  \vol{\objdomain}
  \ceq \voldens{1}{\objdomain},
\end{equation}
where $\vol{\objdomain} = \int_{\objdomain} 1 \diff\tilde{\*x}$
is the volume of the object domain and
$\densub \in \clint{0, 1}$ is an upper bound on the volume fraction.

\begin{SCfigure}
  \includegraphics{topoOptExample_1}%
  \caption[%
    Example scenario for topology optimization%
  ]{%
    Example scenario for topology optimization.
    An object \emph{\textcolor{hellblau}{(light blue)}}
    is fixed on the left side
    of the object domain $\objdomain$
    \emph{\textcolor{mittelblau!50}{(darker blue)}}
    and deformed by a force $\force$, resulting in a displaced object
    \emph{(dashed).}
    The density function $\densglobal(\tilde{\*x})$ is one inside the object
    and zero outside.%
  }%
  \label{fig:topoOptExample}%
\end{SCfigure}

\subsection{Two-Scale Approach}
\label{sec:612twoScale}

\paragraph{Discretization and two-scale approach}

Of course, we cannot solve the problem \eqref{eq:topoOptProblemContinuous}
numerically,
as there are infinitely many density functions $\densglobal$.
For simplicity, we assume that $\objdomain$ is some hyper-rectangle
$\clint{\tilde{\*a}, \tilde{\*b}}
= \clint{\tilde{a}_1, \tilde{b}_1} \times \dotsb \times
\clint{\tilde{a}_{\dimobjdomain}, \tilde{b}_{\dimobjdomain}}$;
if it is not, we replace $\objdomain$ with its bounding box.
The object domain $\objdomain$ can then be split into
$M_1 \times \dotsb \times M_{\dimobjdomain}$
equally-sized and axis-aligned sub-hyper-rectangles,
which we call \term{macro-cells}
(where $M_1, \dotsc, M_{\dimobjdomain} \in \nat$).

In the \term{two-scale approach,}
we assume the material of the macro-cells to be
repetitions of infinitesimally small periodic structures
(i.e., identical for each macro-cell),
called \term{micro-cells.}
These micro-cells have a specific shape, which is parameterized by
$d$ \term{micro-cell parameters} $x_1, \dotsc, x_d$,
normalized to values in the unit interval $\clint{0, 1}$.
For instance, in two dimensions,
this shape may be an axis-aligned cross
with thicknesses $x_1$ and $x_2$, as shown in \cref{fig:twoScale}.
The choice of a suitable \term{micro-cell model}
(parametrization of the micro-cells)
depends on the optimization scenario and has to be done a priori.

\begin{figure}
  \includegraphics{twoScale_1}%
  \caption[%
    Two-scale approach for topology optimization%
  ]{%
    Two-scale approach to discretize the homogenized topology
    optimization problem in two dimensions ($\dimobjdomain = 2$).
    \emph{Left:} The object domain $\objdomain$ is
    subdivided into $M_1 \times M_2$ macro-cells,
    each with its own density \emph{(gray squares).}
    \emph{Center:} Every macro-cell is the repetition of infinitesimally small
    periodic micro-cells.
    \emph{Right:} The shape of the structure in every micro-cell is
    described by a micro-cell model with $d$ parameters $x_1, \dotsc, x_d$.
    Here, the micro-cell model is a cross with two parameters
    that represent the thickness of each crossbar.%
  }%
  \label{fig:twoScale}%
\end{figure}

\paragraph{Elasticity tensors}

Note that while the shape of all micro-cells in one macro-cell is identical,
the micro-cell parameters corresponding to different macro-cells differ
in general.
This enables varying densities in different regions of $\objdomain$.
We denote the micro-cell parameters corresponding to the $q$-th macro-cell
with $\mcp{q} = (\mcpentry{1}{q}, \dotsc, \mcpentry{d}{q}) \in
\clint{\*0, \*1} = \clint{0, 1}^d$,
where $q = 1, \dotsc, M$ and
$M \ceq M_1 \dotsm M_{\dimobjdomain}$ is the number of macro-cells.
With linear elasticity,
one can compute so-called \term{elasticity tensors} $\etensor^{(q)}$,
which encode information about the material properties
of the different macro-cells.
The elasticity tensors can be written as symmetric matrices
in $\real^{3 \times 3}$ (for $\dimobjdomain = 2$) or
in $\real^{6 \times 6}$ (for $\dimobjdomain = 3$).%
\footnote{%
  In general, the elasticity tensor is a fourth-order tensor in
  $\real^{\dimobjdomain \times \dimobjdomain \times \dimobjdomain \times \dimobjdomain}$.
  One can reduce the size of the tensor by exploiting various symmetries
  \cite{Huebner14Mehrdimensionale}
  to obtain $6$ or $21$ stiffness coefficients
  in two or three dimensions, respectively.
  These coefficients can then be expressed as a symmetric matrix.%
}
To simplify the following considerations,
we assume that $\dimobjdomain = 3$, i.e.,
$\etensor^{(q)} \in \real^{6 \times 6}$.
The elasticity tensors are usually computed as the solution of a \fem problem
\term{(micro-problem).}
Once all $\etensor^{(q)}$ are known,
we can compute the compliance value
by solving another \fem problem \term{(macro-problem),}
see \cite{Allaire04Topology} and \cite{Huebner14Mehrdimensionale}.

\vspace{-0.5em}

\paragraph{Discretized optimization problem}

The new optimization problem emerging from the
two-scale discretization process has the form
\begin{subequations}
  \label{eq:topoOptProblemDiscrete}
  \setlength{\belowdisplayskip}{5pt}%
  \begin{gather}
    \min J(\mcp{1}, \dotsc, \mcp{M}),\quad
    \mcp{1}, \dotsc, \mcp{M} \in \clint{\*0, \*1}
    \quad\text{s.t.}\quad
    \densmean(\mcp{1}, \dotsc, \mcp{M}) \le \densub,\\
    \densmean(\mcp{1}, \dotsc, \mcp{M})
    \ceq \frac{1}{M} \sum_{q=1}^M \denscell^{(q)}(\mcp{q}).
  \end{gather}
\end{subequations}
Here, $\denscell^{(q)}(\mcp{q}) \in \clint{0, 1}$ is the
density of the $q$-th macro-cell with micro-cell parameter $\mcp{q}$
(i.e., the fraction of material volume of one micro-cell
with respect to its total volume)
and $\densmean(\mcp{1}, \dotsc, \mcp{M}) \in \clint{0, 1}$
is the resulting total mean density.
This discretized optimization problem can now be implemented and
solved numerically.