\section{$\mathbf{A} = \mathbf{V\Lambda V^{-1}}$ Spectral Theorems, Diagonalization}
\label{sec:diagonalization}

Spectral theorems deal with diagonalizable linear operators.
\\

A diagonalization of a matrix $\mathbf{A}$ is always possible when a matrix is square, and refers to a decomposition of the matrix into the matrix of eigenvectors $\mathbf{V}$ and eigenvalues $\mathbf{\Lambda}$ as

\begin{equation}
\mathbf{A} = \mathbf{V\Lambda V^{-1}}
\end{equation}

\subsection{$\mathbf{A} = \mathbf{V\Lambda V}^T$ Eigendecomposition of Symmetric Matrices}
A hermitian matrix $\mathbf{A}$ has orthogonal eigenvectors, which means that $\mathbf{V}$ is unitary, meaning that $\mathbf{V}^{-1} = \mathbf{V}^{\dagger}$. In that case, the diagonalization is:

\begin{equation}
\mathbf{A} = \mathbf{V\Lambda V}^{\dagger} = \left[\begin{array}{cccc}
\vrule&\vrule&\hdots&\vrule\\
v_1&v_2&\ddots&v_n\\
\vrule&\vrule&\hdots&\vrule
\end{array}\right]\left[\begin{array}{cccc}
\lambda_1&0&\hdots&0\\
0&\lambda_2&\hdots&0\\
\vdots&\vdots&\ddots&\vdots\\
0&\hdots&\hdots&\lambda_n
\end{array}\right]
\left[
\begin{array}{ccc}
\rule[.5ex]{3.5em}{0.4pt}&v_1^{\dagger}&\rule[.5ex]{3.5em}{0.4pt}\\
\rule[.5ex]{3.5em}{0.4pt}&v_2^{\dagger}&\rule[.5ex]{3.5em}{0.4pt}\\
\vdots&\ddots&\vdots\\
\rule[.5ex]{3.5em}{0.4pt}&v_n^{\dagger}&\rule[.5ex]{3.5em}{0.4pt}\\
\end{array}
\right]
\end{equation}

Which is the same as saying that all hermitian matrices are \textit{similar} to a diagonal matrix (cf. section \ref{sec:similiarity}, two matrices $\mathbf{A}$ and $\mathbf{B}$ are similar if they are transmutable using unitary transformations as $\mathbf{A} = \mathbf{UBU^{\dagger}}$). The diagonal representation also shows that in order for $\mathbf{A}$ to satisfy the hermitian property $\mathbf{A}^{\dagger}=\mathbf{A}$ its eigenvalues $\lambda_i$ must be real. Further, it means that if $\mathbf{A}$ is hermitian, $\mathbf{A}$ can be written in terms of projections on the eigenvectors:

\begin{equation}
\mathbf{A} = \sum_i \lambda_i (v_i \otimes v_i)
\end{equation}

Where $\otimes$ is the (complex) outer product $v_i\otimes v_i = v_i v_i^{\dagger}$. I have seen the existence of this representation of a hermitian matrix be described as synonmous with \textit{spectral theorem}. Since the eigenvectors $\mathbf{V}$ are an orthonormal basis, $\sum_i v_i \otimes v_i = \mathbb{I}$.

\subsection{$\mathbf{H} = \mathbf{U\Lambda U}^T$ Eigendecomposition of Hermitian Matrices}

Similarly, a hermitian matrix $\mathbf{H} \in \mathbb{C}^{n\times n}$ (the complex equivalent to a symmetric matrix) has real eigenvalues and the matrix of eigenvectors is unitary, so that $\mathbf{H} = \mathbf{U\Lambda U}^T$.

\subsection{Eigenvalue Sensitivity and Accuracy}

\subsubsection{General Case}

In general, the values of the eigenvalues of a square matrix $\mathbf{A}$ may vary wildly under a slight cange of $\mathbf{A} \rightarrow \mathbf{A}+\delta\mathbf{A}$. The sensitivity of the eigenvalues to a change in $\mathbf{A}$ can be investigated using matrix norms \cite{mathworkseig}. Let $||\cdot||$ denote a submultiplicative matrix norm, then:

\begin{equation}
\begin{array}{rl}
\Lambda + \delta\Lambda &= \mathbf{X^{-1}}\left( \mathbf{A} + \delta\mathbf{A} \right)\mathbf{X} \\
\delta\Lambda &= \mathbf{X^{-1}} \delta \mathbf{A} \mathbf{X}\\
||\delta\Lambda || &= ||\mathbf{X^{-1}} \delta \mathbf{A} \mathbf{X} || \leq  ||\mathbf{X^{-1}}|| ||\mathbf{X}|| ||\delta\mathbf{A}||
\end{array}
\end{equation}

When $||\cdot||$ is chosen to be the operator norm with respect to $L^2$, $||\cdot||_{(2)}||$, then $||\mathbf{X^{-1}}|| = \sigma_1$ and $||\mathbf{X}|| = \frac{1}{\sigma_n}$ where $\sigma_1$ and $\sigma_2$ are the square roots of the largest and the smallest eigenvalue of $\mathbf{X^{\dagger}}\mathbf{X}$ respectively (cf. section \ref{sec:norms} on matrix norms). In that case, the sensitivity of the eigenvalues to a change in $\mathbf{A}$ is:

\begin{equation}
||\delta\mathbf{\Lambda}||_{(2)} \leq \frac{\sigma_1}{\sigma_n} ||\delta\mathbf{A}||_{(2)} = \kappa(\mathbf{X})||\delta\mathbf{A}||_{(2)}
\end{equation}

Where $\kappa(\mathbf{X})$ is the conditioning number of the matrix $\mathbf{X}$. Upper bounds on the error on individual eigenvalues can also be derived quite easily, which is shown in \ref{mathworkseig} pp 10-12.

\subsubsection{Hermitian Matrices}
For hermitian (or orthogonal) matrices, the conditioning number for the individual eigenvalues  $\kappa(\lambda_i,\mathbf{H}) = 1$, so that the error on an individual eigenvalue $||\lambda_i||_{(2)} \leq \kappa(\lambda_i,\mathbf{H}) ||\mathbf{H}||_{(2)} = 1\times ||\mathbf{H}||_{(2)}$.


\section{Natural Smoothing Splines}
Natural splines add the constrain that the function must be linear after
the knots at the end points. This forces 2 more restrictions since
$f''$ must be 0 at the end points, i.e the space has $k + 4 - 2$
parameters because of
this extra 2 constrains.

So where do we put the knots? How many do we use? There are some
data-driven procedures for doing this. Natural Smoothing Splines
provide another approach.

What happens if the knots coincide with the dependent variables
$\{X_i\}$. Then there is a function $g \in \cal G$, the space of cubic
splines with knots at $(x_1,\dots,x_n)$, with
$g(x_i) = y_i, i,\dots,n$, i.e. we haven't smoothed at all.

Consider the following problem: among all functions
$g$ with two  continuous first two derivatives, find one that minimizes
the penalized residual sum of squares
\[
\sum_{i=1}^n \{ y_i - g(x_i) \}^2 + \lambda \int_a^b \{g''(t)\}^2 \,
dt
\]
where $\lambda$ is a fixed constant, and $a \leq x_1 \leq \dots \leq
x_n \leq b$. It can be shown (Reinsch 1967) that the solution to this
problem is a
natural cubic spline with knots at the values of $x_i$ (so there are
$n-2$ interior knots and $n-1$ intervals). Here $a$ and
$b$ are arbitrary as long as they contain the data.

It seems that this procedure is over-parameterized since a
natural cubic spline as this one will have $n$ degrees of
freedom. However we will see that the penalty makes this go down.

\subsection{Computational Aspects}
We use the fact that the solution is a natural cubic spline and write
the possible answers as
\[
g(x) = \sum_{j=1}^{n} \theta_j B_j(x)
\]
where $\theta_j$ are the coefficients and $B_j(x)$ are the basis
functions. Notice that if these were cubic splines the functions lie
in a $n+2$ dimensional space, but the natural splines are an $n$
dimensional subspace.

Let $\bB$ be the $n \times n$ matrix defined by
\[
B_{ij} = B_j(x_i)
\]
and a penalty matrix $\bg{\Omega}$ by
\[
\Omega_{ij} = \int_a^b B_i''(t)B_j''(t) \, dt
\]
now we can write the penalized criterion as
\[
(\by - \bB\bg{\theta})'(\by - \bB\bg{\theta}) +
\lambda\bg{\theta}'\bg{\Omega}\bg{\theta}
\]
It seems there are no boundary derivatives constraints but they are
implicitly imposed by the penalty term.

Setting derivatives with respect to $\bg{\theta}$ equal to 0 gives
the estimating equation:
\[
(\bB'\bB + \lambda\bg{\Omega})\bg{\theta} = \bB'\by.
\]
The $\hat{\bg{\theta}}$ that solves this equation will give us the
estimate $\hat{\g} = \bB \hat{\bg{\theta}}$.

Is this a linear smoother?

Write:
\[
\hat{\g} = \bB \bg{\theta} = \bB(\bB'\bB + \lambda \bg{\Omega})^{-1}
\bB'\by =  ({\mathbf I} + \lambda {\mathbf K})^{-1}\by
\]
where ${\mathbf K} = \bB -1 ' \bg{\Omega} \bB^{-1}$. Notice we can
write the criterion as
\[
(\by - \g)'(\by - \g) + \lambda \g' {\mathbf K} \g
\]

If we look at the ``kernel'' of this linear smoother we will see that
it is similar to the other smoothers presented in this class.

\begin{figure}[h]
\centerline{\epsfig{figure=Plots/plot-04-02.ps,width=.8\textwidth}}
\caption{Smoothing spline fitted using different penalties.}
\end{figure}


% !TeX root = main.tex

\chapter{Fast Fourier Transform}
\glsresetall
\label{chapter:fft}

Performing the \gls{dft} directly using matrix-vector multiply requires $\mathcal{O}(n^2)$ multiply and add operations, for an input signal with $n$ samples.  It is possible to reduce the complexity by exploiting the structure of the constant coefficients in the matrix.  This $S$ matrix encodes the coefficients of the \gls{dft}; each row of this matrix corresponds to a fixed number of rotations around the complex unit circle (please refer to Chapter \ref{sec:DFTbackground} for more detailed information). These values have a significant amount of redundancy, and that can be exploited to reduce the complexity of the algorithm.

\begin{aside}
The 'Big O' notation used here describes the general order of complexity of an algorithm based on the size of the input data.  For a complete description of Big O notation and its use in analyzing algorithms, see \cite{CLR}.
\end{aside}

The \gls{fft} uses a divide-and-conquer approach based on the symmetry of the $S$ matrix.  The \gls{fft} was made popular by the Cooley-Tukey algorithm \cite{cooley65}, which requires $\mathcal{O}(n \log n)$ operations to compute the same function as the \gls{dft}.  This can provide a substantial speedup, especially when performing the Fourier transform on large signals. %Other optimizations are also possible based on the fact that each element of the $S$ matrix has magnitude '1'.  We will mention some of these optimizations, but generally focus on the architectural tradeoffs in implementing the \gls{fft}.

\begin{aside}
The divide-and-conquer approach to computing the \gls{dft} was initially developed by Karl Friedrich Gauss in the early 19th century. However, since Gauss' work on this was not published during his lifetime and only appeared in a collected works after his death, it was relegated to obscurity.  Heideman et al.~\cite{heideman84} provide a nice background on the history of the \gls{fft}.
\end{aside}

The focus of this chapter is to provide the reader with a good understanding of the \gls{fft} algorithm since that is an important part of creating an optimized hardware design. Thus, we start by giving a mathematical treatment of the \gls{fft}. This discussion focuses on small \gls{fft} sizes to give some basic intuition on the core ideas. After that, we focus on different hardware implementation strategies.

\section{Background}

The \gls{fft} brings about a reduction in complexity by taking advantage of symmetries in the \gls{dft} calculation. To better understand how to do this, let us look at \gls{dft} with a small number of points, starting with the 2 point \gls{dft}.  Recall that the \gls{dft} performs a matrix vector multiplication, i.e., $G[] = S[][] \cdot g[]$, where $g[]$ is the input data, $G[]$ is the frequency domain output data, and $S[][]$ are the \gls{dft} coefficients. We follow the same notation for the coefficient matrix, and the input and output vectors as described in Chapter \ref{sec:DFTbackground}.

For a 2 point \gls{dft}, the values of $S$ are:
\begin{equation}
S =
 \begin{bmatrix}
  W^{0 0}_2 & W^{0 1}_2  \\
  W^{1 0}_2 & W^{1 1}_2 \\
 \end{bmatrix}
 \end{equation}
Here we use the notation $W = e^{-j 2 \pi}$. The superscript on $W$ denotes values that are added to the numerator and the subscript on the $W$ indicates those values added in the denominator of the complex exponential. For example, $W^{2 3}_4 = e^{\frac{-j 2 \pi \cdot 2 \cdot 3}{4}}$. This is similar to the $s$ value used in the \gls{dft} discussion (Chapter \ref{sec:DFTbackground}) where $s = e^{\frac{-j 2 \pi}{N}}$. The relationship between $s$ and $W$ is $s = W_N$.

\begin{aside}
The $e^{-j 2 \pi}$ or $W$ terms are often called \term{twiddle factors}. This term has its origin in the 1966 paper by Gentleman and Sande \cite{gentleman1966fast}.
\end{aside}

\begin{equation}
\begin{bmatrix}
G[0] \\
G[1] \\
\end{bmatrix} =
 \begin{bmatrix}
  W^{0 0}_2 & W^{0 1}_2  \\
  W^{1 0}_2 & W^{1 1}_2 \\
 \end{bmatrix}
 \cdot
  \begin{bmatrix}
  g[0] \\
  g[1]\\
\end{bmatrix}
\end{equation}

Expanding the two equations for a 2 point \gls{dft} gives us:
\begin{equation}
\begin{array} {lll}
G[0] & = & g[0] \cdot e^{\frac{-j 2 \pi \cdot 0 \cdot 0}{2}} + g[1] \cdot e^{\frac{-j 2 \pi \cdot 0 \cdot 1}{2}} \\
 & = & g[0] + g[1] \\
\end{array}
\label{eq:2ptlower}
\end{equation} due to the fact that since  $e^{0}  =  1$. The second frequency term
\begin{equation}
\begin{array} {lll}
G[1] & = & g[0] \cdot e^{\frac{-j 2 \pi \cdot 1 \cdot 0}{2}} + g[1] \cdot e^{\frac{-j 2 \pi \cdot 1 \cdot 1}{2}} \\
 & = & g[0] - g[1] \\
\end{array}
\label{eq:2pthigher}
\end{equation} since  $e^{\frac{-j 2 \pi \cdot 1 \cdot 1}{2}}  = e^{-j \pi } = -1$.

Figure \ref{fig:2pointFFT} provides two different representations for this computation.  Part a) is the data flow graph for the 2 point \gls{dft}. It is the familiar view that we have used to represent computation throughout this book. Part b) shows a butterfly structure for the same computation. This is a typical structure used in digital signal processing, in particular, to represent the computations in an \gls{fft}.

The butterfly structure is a more compact representation that is useful to represent large data flow graphs. When two lines come together this indicates an addition operation. Any label on the line itself indicates a multiplication of that label by the value on that line. There are two labels in this figure. The `$-$' sign on the bottom horizontal line indicates that this value should be negated. This followed by the addition denoted by the two lines intersecting is the same as subtraction. The second label is $W^0_2$. While this is a multiplication is unnecessary (since $W^0_2 = 1$ this means it is multiplying by the value `$1$'), we show it here since it is a common structure that appears in higher point \gls{fft}s.

\begin{figure}
\centering
\includegraphics[width= 0.8 \textwidth]{images/2pointFFT}
\caption{Part a) is a data flow graph for a 2 point \gls{dft}/\gls{fft}. Part b) shows the same computation, but viewed as a butterfly structure. This is a common representation for the computation of an \gls{fft} in the digital signal processing domain.}
\label{fig:2pointFFT}
\end{figure}

Now let us consider a slightly larger \gls{dft} -- a 4 point \gls{dft}, i.e., one that has 4 inputs, 4 outputs, and a $4 \times 4$ $S$ matrix. The values of $S$ for a 4 point \gls{dft} are:
\begin{equation}
S =
 \begin{bmatrix}
  W^{0 0}_4 & W^{0 1}_4 & W^{0 2}_4 & W^{0 3}_4 \\
  W^{1 0}_4 & W^{1 1}_4 & W^{1 2}_4 & W^{1 3}_4 \\
  W^{2 0}_4 & W^{2 1}_4 & W^{2 2}_4 & W^{2 3}_4 \\
  W^{3 0}_4 & W^{3 1}_4 & W^{3 2}_4 & W^{3 3}_4 \\
 \end{bmatrix}
 \end{equation}
And the \gls{dft} equation to compute the frequency output terms are:
\begin{equation}
\begin{bmatrix}
G[0] \\
G[1] \\
G[2] \\
G[3] \\
\end{bmatrix} =
 \begin{bmatrix}
  W^{0 0}_4 & W^{0 1}_4 & W^{0 2}_4 & W^{0 3}_4 \\
  W^{1 0}_4 & W^{1 1}_4 & W^{1 2}_4 & W^{1 3}_4 \\
  W^{2 0}_4 & W^{2 1}_4 & W^{2 2}_4 & W^{2 3}_4 \\
  W^{3 0}_4 & W^{3 1}_4 & W^{3 2}_4 & W^{3 3}_4 \\
 \end{bmatrix}
 \cdot
  \begin{bmatrix}
  g[0] \\
  g[1]\\
  g[2]\\
  g[3]\\
\end{bmatrix}
\end{equation}

Now we write out the equations for each of the frequency domain values in $G[]$ one-by-one. The equation for G[0] is:
\begin{equation}
\begin{array} {lll}
G[0] & = & g[0] \cdot e^{\frac{-j 2 \pi \cdot 0 \cdot 0}{4}} + g[1] \cdot e^{\frac{-j 2 \pi \cdot 0 \cdot 1}{4}} + g[2] \cdot e^{\frac{-j 2 \pi \cdot 0 \cdot 2}{4}} + g[3] \cdot e^{\frac{-j 2 \pi \cdot 0 \cdot 3}{4}}\\
 & = & g[0] + g[1] + g[2] + g[3] \\
\end{array}
\end{equation} since $e^0 = 1$.

The equation for $G[1]$ is:
\begin{equation}
\begin{array} {lll}
G[1] & = & g[0] \cdot e^{\frac{-j 2 \pi \cdot 1 \cdot 0}{4}} + g[1] \cdot e^{\frac{-j 2 \pi \cdot 1 \cdot 1}{4}} + g[2] \cdot e^{\frac{-j 2 \pi \cdot 1 \cdot 2}{4}} + g[3] \cdot e^{\frac{-j 2 \pi \cdot 1 \cdot 3}{4}}\\
 & = & g[0] + g[1] \cdot e^{\frac{-j 2 \pi}{4}} + g[2] \cdot e^{\frac{-j 4 \pi}{4}} + g[3] \cdot e^{\frac{-j 6 \pi}{4}}\\
 & = & g[0] + g[1] \cdot e^{\frac{-j 2 \pi}{4}} + g[2] \cdot e^{-j \pi}   + g[3] \cdot e^{\frac{-j 2 \pi}{4}} e^{-j \pi} \\
 & = & g[0] + g[1] \cdot e^{\frac{-j 2 \pi}{4}} - g[2] - g[3] \cdot e^{\frac{-j 2 \pi}{4}}\\
\end{array}
\end{equation} The reductions were done based upon the fact that $e^{-j \pi} = -1$.

The equation for $G[2]$ is:
\begin{equation}
\begin{array} {lll}
G[2] & = & g[0] \cdot e^{\frac{-j 2 \pi \cdot 2 \cdot 0}{4}} + g[1] \cdot e^{\frac{-j 2 \pi \cdot 2 \cdot 1}{4}} + g[2] \cdot e^{\frac{-j 2 \pi \cdot 2 \cdot 2}{4}} + g[3] \cdot e^{\frac{-j 2 \pi \cdot 2 \cdot 3}{4}}\\
 & = & g[0] + g[1] \cdot e^{\frac{-j 4 \pi}{4}} + g[2] \cdot e^{\frac{-j 8 \pi}{4}} + g[3] \cdot e^{\frac{-j 12 \pi}{4}}\\
 & = & g[0] - g[1]  + g[2] -  g[3] \\
\end{array}
\end{equation} The reductions were done by simplifications based upon rotations. E.g., $e^{\frac{-j 8 \pi}{4}} = 1$ and $e^{\frac{-12 j \pi}{4}} = -1$ since in both cases use the fact that $e^{-j 2\pi}$ is equal to $1$. In other words, any complex exponential with a rotation by $2 \pi$ is equal.

Finally, the equation for $G[3]$ is:
\begin{equation}
\begin{array} {lll}
G[3] & = & g[0] \cdot e^{\frac{-j 2 \pi \cdot 3 \cdot 0}{4}} + g[1] \cdot e^{\frac{-j 2 \pi \cdot 3 \cdot 1}{4}} + g[2] \cdot e^{\frac{-j 2 \pi \cdot 3 \cdot 2}{4}} + g[3] \cdot e^{\frac{-j 2 \pi \cdot 3 \cdot 3}{4}}\\
 & = & g[0] + g[1] \cdot e^{\frac{-j  6 \pi}{4}} + g[2] \cdot e^{\frac{-j  12 \pi}{4}} + g[3] \cdot e^{\frac{-j 18 \pi}{4}}\\
 & = & g[0] + g[1] \cdot e^{\frac{-j 6  \pi }{4}}  - g[2] +  g[3] \cdot e^{\frac{-j 10  \pi}{4}}\\
  & = & g[0] + g[1] \cdot e^{\frac{-j 6  \pi }{4}}  - g[2] -  g[3] \cdot e^{\frac{-j 6  \pi}{4}}\\
\end{array}
\end{equation} Most of the reductions that we have not seen yet deal with the last term. It starts out as $e^{\frac{-j 18 \pi}{4}}$. It is reduced to $e^{\frac{-j 10 \pi}{4}}$ since these are equivalent based upon a $2 \pi$ rotation, or, equivalently, $e^{\frac{-j 10 \pi}{4}} \cdot e^{\frac{-j 8 \pi}{4}}$ and the second term $e^{\frac{-j 8 \pi}{4}} = 1$. Finally, a rotation of $\pi$, which is equal to $-1$, brings it to $e^{\frac{-j 6  \pi}{4}}$. Another way of viewing this is $e^{\frac{-j 6 \pi}{4}} \cdot e^{\frac{-j 4 \pi}{4}}$ and $e^{\frac{-j 4 \pi}{4}} = -1$. We leave this term in this unreduced state in order to demonstrate symmetries in the following equations.

With a bit of reordering, we can view these four equations as:
\begin{equation}
\begin{array} {lll}
G[0] & = & (g[0] + g[2]) + e^{\frac{-j 2 \pi 0}{4}} (g[1] + g[3])\\
G[1] & = & (g[0] - g[2]) + e^{\frac{-j 2 \pi 1}{4}} (g[1] - g[3])\\
G[2] & = & (g[0] + g[2]) + e^{\frac{-j 2 \pi 2}{4}} (g[1] + g[3])\\
G[3] & = & (g[0] - g[2]) + e^{\frac{-j 2 \pi 3}{4}} (g[1] - g[3])\\
\end{array}
\end{equation}

Several different symmetries are starting to emerge. First, the input data can be partitioned into even and odd elements, i.e., similar operations are done on the elements $g[0]$ and $g[2]$, and the same is true for the odd elements $g[1]$ and $g[3]$. Furthermore we can see that there are addition and subtraction symmetries on these even and odd elements. During the calculations of the output frequencies $G[0]$ and $G[2]$, the even and odd elements are summed together. The even and odd input elements are subtracted when calculating the frequencies $G[1]$ and $G[3]$.  Finally, the odd elements in every frequency term are multiplied by a constant complex exponential $W^i_4$ where $i$ denotes the index for the frequency output, i.e., $G[i]$.

Looking at the terms in the parentheses, we see that they are 2 point \gls{fft}. For example, consider the terms corresponding to the even input values $g[0]$ and $g[2]$. If we perform a 2 point \gls{fft} on these even terms, the lower frequency (DC value) is $g[0] + g[2]$ (see Equation \ref{eq:2ptlower}), and the higher frequency is calculated as $g[0] - g[2]$ (see Equation \ref{eq:2pthigher}). The same is true for the odd input values $g[1]$ and $g[3]$.

We perform one more transformation on these equations.
\begin{equation}
\begin{array} {lll}
G[0] & = & (g[0] + g[2]) + e^{\frac{-j 2 \pi 0}{4}} (g[1] + g[3])\\
G[1] & = & (g[0] - g[2]) + e^{\frac{-j 2 \pi 1}{4}} (g[1] - g[3])\\
G[2] & = & (g[0] + g[2]) - e^{\frac{-j 2 \pi 0}{4}} (g[1] + g[3])\\
G[3] & = & (g[0] - g[2]) - e^{\frac{-j 2 \pi 1}{4}} (g[1] - g[3])\\
\end{array}
\label{eq:reduced4point}
\end{equation}
The twiddle factors in the last two equations are modified from $e^{\frac{-j 2 \pi 2}{4}} = -e^{\frac{-j 2 \pi 0}{4}}$ and $e^{\frac{-j 2 \pi 3}{4}} = -e^{\frac{-j 2 \pi 1}{4}}$. This allows for a reduction in the complexity of the multiplications since we can share multiplications across two terms.

\begin{figure}
\centering
\includegraphics[width=  .5 \textwidth]{images/4ptFFT}
\caption{A four point \gls{fft} divided into two stages. Stage 1 has uses two 2 point \gls{fft}s -- one 2 point \gls{fft} for the even input values and the other 2 point \gls{fft} for the odd input values. Stage 2 performs the remaining operations to complete the \gls{fft} computation as detailed in Equation \ref{eq:reduced4point}. }
\label{fig:4ptFFT}
\end{figure}

Figure \ref{fig:4ptFFT} shows the butterfly diagram for the four point \gls{fft}. We can see that the first stage is two 2 point \gls{fft} operations performed on the even (top butterfly) and odd (bottom butterfly) input values. The output of the odd 2 point \gls{fft}s are multiplied by the appropriate twiddle factor. We can use two twiddle factors for all four output terms by using the reduction shown in Equation \ref{eq:reduced4point}.

We are seeing the beginning of trend that allows the a reduction in complexity from $\mathcal{O}(n^2)$ operations for the \gls{dft} to $\mathcal{O}(n \log n)$ operations for the \gls{fft}. The key idea is building the computation through recursion. The 4 point \gls{fft} uses two 2 point \gls{fft}s. This extends to larger \gls{fft} sizes. For example, an 8 point \gls{fft} uses two 4 point \gls{fft}s, which in turn each use two 2 point \gls{fft}s (for a total of four 2 point \gls{fft}s). An 16 point \gls{fft} uses two 8 point \gls{fft}s, and so on.

\begin{exercise}
How many 2 point \gls{fft}s are used in a 32 point \gls{fft}? How many are there in a 64 point \gls{fft}? How many 4 point \gls{fft}s are required for a 64 point \gls{fft}? How about a 128 point \gls{fft}?  What is the general formula for 2 point, 4 point, and 8 point \gls{fft}s in an $N$ point \gls{fft} (where $N > 8$)?
\end{exercise}

Now let us formally derive the relationship, which provides a general way to describe the recursive structure of the \gls{fft}.  Assume that we are calculating an $N$ point \gls{fft}. The formula for calculating the frequency domain values $G[]$ given the input values $g[]$ is:
\begin{equation}
G[k] = \displaystyle\sum\limits_{n=0}^{N-1} g[n] \cdot e^{\frac{-j 2 \pi k n}{N}} \text{ for } k = 0,\dots, N-1
\label{eq:fft-full}
\end{equation}

We can divide this equation into two parts, one that sums the even components and one that sums the odd components.
\begin{equation}
G[k] = \displaystyle\sum\limits_{n=0}^{N/2-1} g[2n] \cdot e^{\frac{-j 2 \pi k (2n)}{N}} + \displaystyle\sum\limits_{n=0}^{N/2-1} g[2n+1] \cdot e^{\frac{-j 2 \pi k (2n+1)}{N}}
\label{eq:fft-split}
\end{equation}
The first part of this equation deals with the even inputs, hence the $2n$ terms in both $g[]$ and in the exponent of $e$. The second part corresponds to the odd inputs with $2n +1$ in both places. Also note that the sums now go to $N/2 -1$ in both cases which should make sense since we have divided them into two halves.

We transform Equation \ref{eq:fft-split} to the following:
\begin{equation}
G[k] = \displaystyle\sum\limits_{n=0}^{N/2-1} g[2n] \cdot e^{\frac{-j 2 \pi k n}{N/2}} + \displaystyle\sum\limits_{n=0}^{N/2-1} g[2n+1] \cdot e^{\frac{-j 2 \pi k (2n)}{N}} \cdot e^{\frac{-j 2 \pi k}{N}}
\label{eq:fft-split-2}
\end{equation}
In the first summation (even inputs), we simply move the $2$ into the denominator so that it is now $N/2$. The second summation (odd inputs) uses the power rule to separate the $+1$ leaving two complex exponentials. We can further modify this equation to
\begin{equation}
G[k] = \displaystyle\sum\limits_{n=0}^{N/2-1} g[2n] \cdot e^{\frac{-j 2 \pi k n}{N/2}} + e^{\frac{-j 2 \pi k}{N}} \cdot \displaystyle\sum\limits_{n=0}^{N/2-1} g[2n+1] \cdot e^{\frac{-j 2 \pi k n}{N/2}}
\label{eq:fft-split-3}
\end{equation}
Here we only modify the second summation. First we pull one of the complex exponentials outside of the summation since it does not depend upon $n$. And we also move the $2$ into the denominator as we did before in the first summation. Note that both summations now have the same complex exponential $e^{\frac{-j 2 \pi k n}{N/2}}$. Finally, we simplify this to
\begin{equation}
G[k] = A_k + W_N^k B_k
\label{eq:fft-split-4}
\end{equation} where $A_k$ and $B_k$ are the first and second summations, respectively. And recall that $W = e^{-j 2 \pi}$. This completely describes an N point \gls{fft} by separating even and odd terms into two summations.

For reasons that will become clear soon, let us assume that we only want to use Equation \ref{eq:fft-split-4} to calculate the first $N/2$ terms, i.e., $G[0]$ through $G[N/2 -1]$. And we will derive the remaining $N/2$ terms, i.e., those from $G[N/2]$ to $G[N-1]$ using a different equation. While this may seem counterintuitive or even foolish (why do more math than necessary?), you will see that this will allow us to take advantage of even more symmetry, and derive a pattern as we have seen in the 4 point \gls{fft}.

In order to calculate the higher frequencies $G[N/2]$ to $G[N-1]$, let us derive the same equations but this time using $k = N/2, N/2 + 1, \dots, N/2 -1$.  Thus, we wish to calculate
\begin{equation}
G[k + N/2] = \displaystyle\sum\limits_{n=0}^{N-1} g[n] \cdot e^{\frac{-j 2 \pi (k + N/2) n}{N}} \text{ for } k = 0, \dots, N/2 - 1
\label{eq:fft-upper}
\end{equation}
This is similar to Equation \ref{eq:fft-full} with different indices, i.e., we replace $k$ from Equation \ref{eq:fft-full} with $k + N/2$. Using the same set of transformations that we did previously, we can move directly to the equivalent to Equation \ref{eq:fft-split-3}, but replacing all instances of $k$ with $k + N/2$ which yields
\begin{equation}
G[k + N/2] = \displaystyle\sum\limits_{n=0}^{N/2-1} g[2n] \cdot e^{\frac{-j 2 \pi (k + N/2) n}{N/2}} + e^{\frac{-j 2 \pi (k + N/2)}{N}} \cdot \displaystyle\sum\limits_{n=0}^{N/2-1} g[2n+1] \cdot e^{\frac{-j 2 \pi (k + N/2) n}{N/2}}
\label{eq:fft-split-3-upper}
\end{equation}

We can reduce the complex exponential in the summations as follows:
\begin{equation}
e^{\frac{-j 2 \pi (k + N/2) n}{N/2}} = e^{\frac{-j 2 \pi k n}{N/2}} \cdot e^{\frac{-j 2 \pi (N/2) n}{N/2}} = e^{\frac{-j 2 \pi k n}{N/2}} \cdot e^{-j 2 \pi n} = e^{\frac{-j 2 \pi k n}{N/2}} \cdot 1
\label{eq:fft-split-4-upper}
\end{equation}
The first reduction uses the power rule to split the exponential. The second reduction cancels the term $N/2$ in the second exponential. The final reduction uses that fact that $n$ is a non-negative integer, and thus $e^{-j 2 \pi n}$ will always be a rotation of multiple of $2 \pi$. This means that this term is always equal to $1$.

Now let us tackle the second complex exponential
\begin{equation}
e^{\frac{-j 2 \pi (k + N/2)}{N}} = e^{\frac{-j 2 \pi k }{N}} \cdot e^{\frac{-j 2 \pi N/2 }{N}} = e^{\frac{-j 2 \pi k }{N}} \cdot e^{-j  \pi} = - e^{\frac{-j 2 \pi k }{N}}
\label{eq:fft-split-5-upper}
\end{equation}
The first reduction splits the exponential using the power rule. The second reduction does some simplifications on the second exponential. We get the final term by realizing that $e^{-j \pi} = -1$.

By substituting Equations \ref{eq:fft-split-4-upper} and \ref{eq:fft-split-5-upper} into Equation \ref{eq:fft-split-3-upper}, we get
\begin{equation}
G[k + N/2] = \displaystyle\sum\limits_{n=0}^{N/2-1} g[2n] \cdot e^{\frac{-j 2 \pi k n}{N/2}} - e^{\frac{-j 2 \pi k}{N}} \cdot \displaystyle\sum\limits_{n=0}^{N/2-1} g[2n+1] \cdot e^{\frac{-j 2 \pi k n}{N/2}}
\label{eq:fft-split-6-upper}
\end{equation}
Note the similarity to Equation \ref{eq:fft-split-3}. We can put it in terms of Equation \ref{eq:fft-split-4} as
\begin{equation}
G[k + N/2] = A_k - W_N^k B_k
\label{eq:fft-split-7-upper}
\end{equation}

We can use Equations \ref{eq:fft-split-4} and \ref{eq:fft-split-7-upper} to create an $N$ point \gls{fft} from two $N/2$ point \gls{fft}s. Remember that $A_k$ corresponds to the even input values, and $B_k$ is a function of the odd input values. Equation \ref{eq:fft-split-4} covers the first $N/2$ terms, and Equation \ref{eq:fft-split-7-upper} corresponds to the higher $N/2$ frequencies.

\begin{figure}
\centering
\includegraphics[width=  .8 \textwidth]{images/NptFFT}
\caption{Building an $N$ point \gls{fft} from two $N/2$ point \gls{fft}s. The upper $N/2$ point \gls{fft} is performed on the even inputs; the lower $N/2$ \gls{fft} uses the odd inputs. }
\label{fig:NptFFT}
\end{figure}

Figure \ref{fig:NptFFT} shows an $N$ point \gls{fft} derived from two $N/2$ point \gls{fft}s. $A_k$ corresponds to the top $N/2$ \gls{fft}, and $B_k$ is the bottom $N/2$ \gls{fft}.  The output terms $G[0]$ through $G[N/2-1]$ are multiplied by $W_N^0$ while the output terms $G[N/2]$ through $G[N-1]$ are multiplied by $-W_N^0$. Note that the inputs $g[]$ are divided into even and odd elements feeding into the top and bottom $n/2$ point \gls{fft}s, respectively.

We can use the general formula for creating the \gls{fft} that was just derived to recursively create the $N/2$ point \gls{fft}. That is, each of the $N/2$ point \gls{fft}s can be implemented using two $N/4$ point \gls{fft}s. And each $N/4$ point \gls{fft} uses two $N/8$ point \gls{fft}s, and so on until we reach the base case, a 2 point \gls{fft}.

Figure \ref{fig:8ptFFT} shows an 8 point \gls{fft} and highlights this recursive structure. The boxes with the dotted lines indicate different sizes of \gls{fft}. The outermost box indicates an 8 point \gls{fft}. This is composed by two 4 point \gls{fft}s. Each of these 4 point \gls{fft}s have two 2 point \gls{fft}s for a total of four 2 point \gls{fft}s.

\begin{figure}
\centering
\includegraphics[width=  \textwidth]{images/8ptFFT}
\caption{An 8 point \gls{fft} built recursively. There are two 4 point \gls{fft}s, which each use two 2 point \gls{fft}s. The inputs must be reordered to even and odd elements twice. This results in reordering based upon the bit reversal of the indices.}
\label{fig:8ptFFT}
\end{figure}

Also note that the inputs must be reordered before they are feed into the 8 point \gls{fft}. This is due to the fact that the different $N/2$ point \gls{fft}s take even and odd inputs. The upper four inputs correspond to even inputs and the lower four inputs have odd indices. However, they are reordered twice. If we separate the even and odd inputs once we have the even set $\{g[0], g[2], g[4], g[6] \}$ and the odd set $\{g[1], g[3], g[5], g[7] \}$. Now let us reorder the even set once again. In the even set $g[0]$ and $g[4]$ are the even elements, and $g[2]$ and $g[6]$ are the odd elements. Thus reordering it results in the set $\{g[0], g[4], g[2], g[6] \}$. The same can be done for the initial odd set yielding the reordered set $\{g[1], g[5], g[3], g[7] \}$.

The final reordering is done by swapping values whose indices are in bit reversed order. Table \ref{table:bit_reverse} shows the indices and their three bit binary values. The table shows the eight indices for the 8 point \gls{fft}, and the corresponding binary value for each of those indices in the second column. The third column is the bit reversed binary value of the second column. And the last column is the decimal number corresponding the reversed binary number.

\begin{table}[htbp]
\caption{The index, three bit binary value for that index, bit reversed binary value, and the resulting bit reversed index.}
\begin{center}
\begin{tabular}{|c|c|c|c|}
\hline
Index & Binary & Reversed & Reversed \\
 & & Binary & Index \\
\hline
0 & 000 & 000 & 0 \\
1 & 001 & 100 & 4 \\
2 & 010 & 010 & 2 \\
3 & 011 & 110 & 6 \\
4 & 100 & 001 & 1 \\
5 & 101 & 101 & 5 \\
6 & 110 & 011 & 3 \\
7 & 111 & 111 & 7 \\
\hline
\end{tabular}
\end{center}
\label{table:bit_reverse}
\end{table}

Looking at the first row, the initial index $0$, has a binary value of $000$, which when reversed remains $000$. Thus this index does not need to be swapped. Looking at Figure \ref{fig:8ptFFT} we see that this is true. $g[0]$ remains in the same location. In the second row, the index 1 has a binary value $001$. When reversed this is $100$ or $4$. Thus, the data that initially started at index 1, i.e., $g[1]$ should end up in the fourth location. And looking at index 4, we see the bit reversed value is $1$. Thus $g[1]$ and $g[4]$ are swapped.

This bit reversal process works regardless of the input size of the \gls{fft}, assuming that the \gls{fft} is a power of two. \gls{fft} are commonly a power of two since this allows them to be recursively implemented.

\begin{exercise}
In an 32 point \gls{fft}, index 1 is swapped with which index? Which index is index 2 is swapped with?
\end{exercise}

This completes our mathematical treatment of the \gls{fft}. There are plenty of more details about the \gls{fft}, and how to optimize it. You may think that we spent too much time already discussing the finer details of the \gls{fft}; this is a book on parallel programming for FPGAs and not on digital signal processing. This highlights an important part of creating an optimum hardware implementation -- the designer must have a good understanding of the algorithm under development. Without that, it is difficult to create a good implementation. The next section deals with how to create a good \gls{fft} implementation.

\section{Baseline Implementation}

In the remainder of this chapter, we discuss different methods to implement the Cooley-Tukey \gls{fft} \cite{cooley65} algorithm using the \VHLS tool. This is the same algorithm that we described in the previous section. We start with a common version of the code, and then describe how to restructure it to achieve a better hardware design.

When performed sequentially, the $\mathcal{O}(n \log n)$ operations in the \gls{fft} require $\mathcal{O}(n \log n)$ time steps.   Typically, a parallel implementation will perform some portion of the \gls{fft} in parallel.  One common way of parallelizing the \gls{fft} is to organize the computation into $\log n$ stages, as shown in Figure \ref{fig:fftstages}.  The operations in each stage are dependent on the operations of the previous stage, naturally leading to a pipelining across the tasks.  Such an architecture allows $\log n$ \glspl{fft} to be computed simultaneously with a task interval determined by the architecture of each stage.  We discuss task pipelining using the \lstinline|dataflow| directive in Section \ref{sec:fft_task_pipelining}.

Each stage in the \gls{fft} also contains significant parallelism, since each butterfly computation is independent of other butterfly computations in the same stage.  In the limit, performing $n/2$ butterfly computations every clock cycle with a Task Interval of 1 can allow the entire stage to be computed with a Task Interval of 1.  When combined with a dataflow architecture, all of the parallelism in the \gls{fft} algorithm can be exploited.  Note, however that although such an architecture can be constructed, it is almost never used except for very small signals, since an entire new block of \lstinline|SIZE| samples must be provided every clock cycle to keep the pipeline fully utilized.  For instance, a 1024-point \gls{fft} of complex 32-bit floating point values, running at 250 MHz would require 1024 \text{points}*(8 {bytes}/{point})*250*$10^9$ Hz = 1Terabyte/second of data into the FPGA.   In practice, a designer must match the computation architecture to the data rate required in a system.

\begin{exercise}
Assuming a clock rate of 250 MHz and one sample received every clock cycle, approximately how many butterfly computations must be implemented to process every sample with a 1024-point \gls{fft}?  What about for a 16384-point \gls{fft}?
\end{exercise}

In the remainder of this section, we describe the optimization of an \gls{fft} with the function prototype \lstinline|void fft(DTYPE X_R[SIZE], DTYPE X_I[SIZE])| where \lstinline|DTYPE| is a user customizable data type for the representation of the input data. This may be \lstinline|int|, \lstinline|float|, or a fixed point type. For example, \lstinline|#define DTYPE int| defines \lstinline|DTYPE| as an \lstinline|int|. Note that we choose to implement the real and imaginary parts of the complex numbers in two separate arrays. The \lstinline|X_R| array holds the real input values, and the \lstinline|X_I| array holds the imaginary values. \lstinline|X_R[i]| and \lstinline|X_I[i]| hold the $i$th complex number in separate real and imaginary parts.

\begin{aside}
There is one change in the \gls{fft} implementation that we describe in this section. Here we perform an \gls{fft} on complex numbers. The previous section uses only real numbers. While this may seem like a major change, the core ideas stay the same. The only differences are that the data has two values (corresponding to the real and imaginary part of the complex number), and the operations (add, multiply, etc.) are complex operations.
\end{aside}

This function prototype forces an in-place implementation. That is, the output data is stored in the same array as the input data. This eliminates the need for additional arrays for the output data, which reduces the amount of memory that is required for the implementation. However, this may limit the performance due to the fact that we must read the input data and write the output data to the same arrays. Using separate arrays for the output data is reasonable if it can increase the performance. There is always a tradeoff between resource usage and performance; the same is true here. The best implementation depends upon the application requirements (e.g., high throughput, low power, size of FPGA, size of the \gls{fft}, etc.).

%\section{Initial ``Software'' \gls{fft} Implementation}

We start with code for an \gls{fft} that would be typical for a software implementation. Figure \ref{fig:fft_sw} shows a nested three \lstinline|for| loop structure. The outer \lstinline|for| loop, labeled \lstinline|stage_loop| implements one stage of the \gls{fft} during each iteration. There are $log_2(N)$ stages where $N$ is the number of input samples. The stages are clearly labeled in Figure \ref{fig:8ptFFT}; this 8 point \gls{fft} has $log_2(8) = 3$ stages. You can see that each stage performs the same amount of computation, or the same number of butterfly operations. In the 8 point \gls{fft}, each stage has four butterfly operations.

\begin{figure}
\lstinputlisting[format=none,firstline=34]{examples/fft_sw.cpp}
\caption{ A common implementation for the \gls{fft} using three nested \lstinline|for| loops.  While this may work well running as software on a processor, it is far from optimal for a hardware implementation.}
\label{fig:fft_sw}
\end{figure}

\begin{exercise}
For an $N$ point \gls{fft}, how many butterfly operations are there in each stage? How many total butterfly operations are there for the entire \gls{fft}?
\end{exercise}

The second \lstinline|for| loop, labeled \lstinline|butterfly_loop|, performs all of the butterfly operations for the current stage. \lstinline|butterfly_loop| has another nested \lstinline|for| loop, labeled \lstinline|dft_loop|. Each iteration of \lstinline|dft_loop| performs one butterfly operation. Remember that we are dealing with complex numbers and must perform complex additions and multiplications.

The first line in \lstinline|dft_loop| determines the offset of the butterfly. Note that the ``width'' of the butterfly operations changes depending upon the stage. Looking at Figure \ref{fig:8ptFFT}, Stage 1 performs butterfly operations on adjacent elements, Stage 2 performs butterfly operations on elements with index differing by two, and Stage 3 performs butterfly operations on elements with index differing by four. This difference is computed and stored in the \lstinline|i_lower| variable. Notice that this offset, stored in the variable \lstinline|numBF|, is different in every stage.

The remaining operations in \lstinline|dft_loop| perform multiplication by the twiddle factor and an addition or subtraction operation. The variables \lstinline|temp_R| and \lstinline|temp_I| hold the real and imaginary portions of the data after multiplication by the twiddle factor $W$. The variables \lstinline|c| and \lstinline|s| are the real and imaginary parts of $W$, which is calculated using the \lstinline|sin()| and \lstinline|cos()| builtin functions. We could also use the CORDIC, such as the one developed in Chapter \ref{chapter:cordic}, to have more control over the implementation. Twiddle factors are also commonly precomputed and stored in on-chip memory for moderate array sizes.  Lastly, elements of the \lstinline|X_R[]| and \lstinline|X_I[]| arrays are updated with the result of the butterfly computation.

\lstinline|dft_loop| and \lstinline|butterfly_loop| each execute a different number of times depending upon the stage. However the total number of times that the body of \lstinline|dft_loop| is executed in one stage is constant. The number of iterations for the \lstinline|butterfly for| loop depends upon the number of unique $W$ twiddle factors in that stage. Referring again to Figure \ref{fig:8ptFFT}, we can see that Stage 1 uses only one twiddle factor, in this case $W_8^0$. Stage 2 uses two unique twiddle factors and Stage 3 uses four different $W$ values. Thus, \lstinline|butterfly_loop| has only one iteration in Stage 1, 2 iterations in stage 2, and four iterations in stage 3. Similarly, the number of iterations of \lstinline|dft_loop| changes. It iterates four times for an 8 point \gls{fft} in Stage 1, two times in Stage 2, and only one time in stage 3. However in every stage, the body of \lstinline|dft_loop| is executed the same number of times in total, executing a total of four butterfly operations for each stage an 8 point \gls{fft}.

\begin{aside}
\VHLS performs significant static analysis on each synthesized function, including computing bounds on the number of times each loop can execute.  This information comes from many sources, including variable bitwidths, ranges, and \lstinline|assert()| functions in the code. When combined with the loop II, \VHLS can compute bounds on the latency or interval of the \gls{fft} function.  In some cases (usually when loop bounds are variable or contain conditional constructs), the tool is unable to compute the latency or interval of the code and returns `'?'.  When synthesizing the code in Figure \ref{fig:fft_sw}, \VHLS may not be able to determine the number of times that \lstinline|butterfly_loop| and \lstinline|dft_loop| iterate because these loops have variable bounds.

The \lstinline|tripcount| directive enables the user to specify to the \VHLS tool more information about the number of times a loop is executed which can be used by the tool to analyze the performance of the design. It takes three optional arguments \lstinline|min|, \lstinline|max|, and \lstinline|average|. In this code, we could add a directive to \lstinline|dft_loop|. By applying this directive, the \VHLS tool can calculate bounds on the latency and interval value for the loop and the overall design.  Note that since the \VHLS tool uses the numbers that you provide, if you give the tool an incorrect tripcount then the reported task latency and task interval will be incorrect -- garbage in, garbage out.
\end{aside}

\begin{exercise}
What is the appropriate way to use the \lstinline|trip count| directive for the \gls{fft} in Figure \ref{fig:fft_sw}? Should you set the \lstinline|max|, \lstinline|min|, and/or \lstinline|average| arguments? Would you need to modify the tripcount arguments if the size of the \gls{fft} changes?
\end{exercise}

\section{Bit Reversal}
\label{sec:fft_bit_reversal}

We have not talked about the bit reverse function, which swaps the input data values so that we can perform an in-place \gls{fft}. This means that the inputs values are mixed, and the result is that the output data is in the correct order. We discuss that function in some detail now.

Figure \ref{fig:fft_bit_reverse} shows one possible implementation of the bit reverse function. It divides the code into two functions. The first is the bit reversal function (\lstinline|bit_reverse|), which reorders data in the given arrays so that each data is in located at a different index in the array. This function calls another function, \lstinline|reverse_bits|, which takes an input integer and returns the bit reversed value of that input.

\begin{figure}
\lstinputlisting[lastline=33]{examples/fft_sw.cpp}
\caption{ The first stage in our \gls{fft} implementation reorders the input data. This is done by swapping the value at index $i$ in the input array with the value at the bit reversed index corresponding to $i$. The function \lstinline|reverse_bits| gives the bit reversed value corresponding to the \lstinline|input| argument. And the function \lstinline|bit_reverse| swaps the values in the input array. }
\label{fig:fft_bit_reverse}
\end{figure}

Let us start with a brief overview of the \lstinline|reverse_bits| function. The function goes bit by bit through the \lstinline|input| variable and shifts it into the \lstinline|rev| variable. The \lstinline|for| loop body consists of a few bitwise operations that reorder the bits of the input. Although these operations are individually not terribly complex, the intention of this code is that the for loop is completely unrolled and \VHLS can identify that the bits of the input can simply be wired to the output.  As a result, the implementation of the \lstinline|reverse_bits| function should require no logic resources at all, but only wires.  This is a case where unrolling loops greatly simplifies the operations that must be performed.  Without unrolling the loop, the individual `or' operations must be performed sequentially.  Although this loop can be pipelined, the `or' operation would still be implemented in logic resources in the FPGA and executing the the loop would have a latency determined by the number of bits being reversed (\lstinline|\gls{fft}_BITS| in this case).

\begin{exercise}
What is the latency of the \lstinline|reverse_bits| function when no directives are applied? What is the latency when the loop is pipelined?  What is the latency when the whole function is pipelined?
\end{exercise}

\begin{aside}
It is tempting to ``blindly'' apply directives in order to achieve a better design. However, this can be counterproductive. The best designer has a deep understanding of both the application and the available optimizations and carefully considers these together to achieve the best results.
\end{aside}

Now let us optimize the parent \lstinline|bit_reverse| function. This function has a single \lstinline{for} loop that iterates through each index of the input arrays. Note that there are two input arrays \lstinline{X_R[]} and \lstinline{X_I[]}. Since we are dealing with complex numbers, we must store both the real portion (in the array \lstinline{X_R[]}), and the imaginary portion (in the array \lstinline{X_I[]}). \lstinline{X_R[i]} and \lstinline{X_I[i]} holds the real and imaginary values of the i-th input.
In each iteration of the \lstinline{for} loop, we find the index reversed value by calling the \lstinline{reverse_bits} function. Then we swap both the real and imaginary values stored in the index \lstinline{i} and the index returned by the function \lstinline{reverse_bits}. Note that as we go through all \lstinline{SIZE} indices, we will eventually hit the reversed index for every value. Thus, the code only swaps values the first time based on the condition \lstinline{if(i < reversed)}.

%To me this doesn't say much that this point...  it would be better to discuss this in a case where it really matters
%We can optimize this function in a number of different ways. Since the \lstinline{reverse_bits} function is small and efficient, it make sense to use the \lstinline{inline} pragma. This essentially copies all of the code from within the \lstinline{reverse_bits} function directly into the \lstinline{bit_reverse} function. Thus there is no function call, and any overhead associated with it goes away. Furthermore, this allows the \VHLS tool to optimize the code within the \lstinline{reverse_bits} function along with the code in the \lstinline{bit_reverse} function, which can lead to some additional efficiencies. That being said, it may be the case when the \lstinline{inline} directive does not help.

\section{Task Pipelining}
\label{sec:fft_task_pipelining}

Dividing the \gls{fft} algorithm into stages enables \VHLS to generate an implementation where different stages of the algorithm are operating on different data sets. This optimization, called \gls{taskpipelining} is enabled using the \lstinline{dataflow} directive. This is a common hardware optimization, and thus is relevant across a range of applications.

\begin{figure}
\lstinputlisting[format=none,firstline=35]{examples/fft_stages.cpp}
\caption{Code implementing an 8 point \gls{fft} divided into stages, each of which is implemented by a separate function. The \lstinline{bit_reverse} function is the first stage. Each following stage performs implements butterfly operations. }
\label{fig:fft_stages_code}
\end{figure}

We can naturally divide the \gls{fft} algorithm into $\log_2(N+1)$ stages where $N$ is the number of points of the \gls{fft}.  The first stage swaps each element in the input array with the element located at the bit reversed address in the array. After this bit reverse stage, we perform $\log_2(N)$ stages of butterfly operations. Each of these butterfly stages has the same computational complexity.  Figure \ref{fig:fft_stages_code} describes how to divide an 8 point \gls{fft} into four separate tasks. The code has separate function call for each of the tasks: one call to \lstinline{bit_reverse}, and three calls to \lstinline{fft_stage}. Each stage has two input arrays and two output arrays: one for the real portion and one for the imaginary portion of the complex numbers. Assume that the \lstinline{DTYPE} is defined elsewhere, e.g., as an \lstinline{int}, \lstinline{float} or a fixed point data type.

Refactoring the \gls{fft} code allows us to perform \gls{taskpipelining}. Figure \ref{fig:fftstages} gives an example of this. In this execution, rather than wait for the first task to complete all four function calls in the code before the second task can begin, the second task can start after the first task has only completed the first function \lstinline{bit_reverse}. The first task continues to execute each stage in the pipeline in order, followed by the remaining tasks in order.  Once the pipeline is full, all four subfunctions are executing concurrently, but each one is operating on different input data. Similarly, there are four 8 point \glspl{fft} being computed simultaneously, each one executing on a different component of the hardware. This shown in the middle portion of Figure \ref{fig:fftstages}. Each of the vertical four stages represents one 8 point \gls{fft}. And the horizontal denotes increasing time. Thus, once we start the fourth 8 point \gls{fft}, we have four \glspl{fft} running simultaneously.  Note that for this to work, each call to the \lstinline{fft_stage} function must be implemented with independent hardware.  In addition, enough storage is required to contain the intermediate computations of each \gls{fft} being computed simultaneously.

\begin{figure}
\centering
%\includegraphics[width=  \textwidth]{images/fftstages}
{\scriptsize \includesvg{fftstages}}
\includesvg{fft_dataflow_behavior}
\caption{ Dividing the \gls{fft} into different stages allows for task pipelining across each of these stages. The figure shows an example with three \gls{fft} stages (i.e., an 8 point \gls{fft}). The figure shows four 8 point \gls{fft} executing at the same time. }
\label{fig:fftstages}
\end{figure}

The \lstinline|dataflow| directive can construct separate pipeline stages (often called \glspl{process}) from both functions and loops. The code in Figure \ref{fig:fft_stages_code} uses functions only, but we could achieve a similar result with four loops instead of four functions.  In fact, this result could be achieved by unrolling the outer \lstinline|stage_loop| in the original code either explicitly or using \lstinline|#pragma HLS unroll|.  Such a code structure has several advantages.  Firstly, it is closer to the structure of the original algorithmic code, reducing the amount of code changes which need to be made.   Secondly, the code is less verbose, making it somewhat easier to write.  Thirdly, the code is again parameterized, supporting different values of size with the same code.  Code with a loop is shown in Figure \ref{fig:fft_stages_loop_code}.

\begin{figure}
\lstinputlisting[format=none,firstline=64]{examples/fft_stages_loop.cpp}
\caption{Code implementing an arbitrary-sized FFT using a loop. After the loop is unrolled, each function call in the unrolled loop becomes a dataflow process. }
\label{fig:fft_stages_loop_code}
\end{figure}

The \lstinline{dataflow} directive and the \lstinline{pipeline} directive both generate circuits capable of pipelined execution.  The key difference is in the granularity of the pipeline. The \lstinline{pipeline} directive constructs an architecture that is efficiently pipelined at the cycle level and is characterized by the II of the pipeline.  Operators are statically scheduled and if the II is greater than one, then operations can be shared on the same operator.  The \lstinline{dataflow} directive constructs an architecture that is efficiently pipelined for operations that take a (possibly unknown) number of clock cycles, such as the behavior of a loop operating on a block of data.  These coarse-grained operations are not statically scheduled and the behavior is controlled dynamically by the handshake of data through the pipeline.  In the case of the \gls{fft}, each stage is an operation on a block of data (the whole array) which takes a large number of cycles.  Within each stage, loops execute individual operations on the data in a block.  Hence, this is a case where it often makes sense to use the \lstinline|dataflow| directive at the toplevel to form a coarse-grained pipeline, combined with the \lstinline|pipeline| directive within each loop to form fine-grained pipelines of the operations on each individual data element.

The \lstinline{dataflow} directive must implement memories to pass data between different processes. In the case when \VHLS can determine that processes access data in sequential order, it implements the memory using a FIFO. This requires that data is written into an array in the same order that it is read from the array.  If the is not the case, or if \VHLS can not determine if this streaming condition is met, then the memory can be implemented using a ping-pong buffer instead.   The ping-pong buffer consists of two (or more) conceptual blocks of data, each the size of the original array. One of the blocks can be written by the source process while another block is read by the destination process. The term ``ping-pong'' comes from the fact that the reading and writing to each block of data alternates in every execution of the task. That is, the source process will write to one block and then switch to the other block before beginning the next task. The destination process reads from the block that the producer is not writing to. As a result, the source and destination processes can never writing and reading from the same block at the same time.

A ping-pong buffer requires enough memory to store each communication array at least twice.   FIFOs can often be significantly smaller, although determining a minimal size for each fifo is often a difficult design problem.  Unlike a FIFO, however, the data in a ping-pong buffer can be written to and read from in any order. Thus, FIFOs are generally the best choice when the data is produced and consumed in sequential order and ping-pong buffers are a better choice when there is not such regular data access patterns.

Using the \lstinline{dataflow} directive effectively still requires the behavior of each individual process to be optimized. Each individual process in the pipeline can still be optimized using techniques we have seen previously such as code restructuring, pipelining, and unrolling.  For example, we have already discussed some optimizations for the \lstinline{bit_reverse} function in Section \ref{sec:fft_bit_reversal}.  In general, it is important to optimize the individual tasks while considering overall toplevel performance goals. Many times it is best to start with small functions and understand how to optimize them in isolation. As a designer, it is often easier to comprehend what is going on in a small piece of code and hopefully determine the best optimizations quickly. After optimizing each individual function, then you can move up the hierarchy considering larger functions given particular implementations of low level functions, eventually reaching the toplevel function.

However, the local optimizations must be considered in the overall scope of the goals. In particular for dataflow designs the achieved interval for the overall pipeline can never be smaller than the interval of each individual process. Looking again at Figure \ref{fig:fft_stages_code}, assume that \lstinline{bit_reverse} has an interval of 8 cycles, \lstinline{fft_stage(1,...)} has an interval of 12 cycles, \lstinline{fft_stage(2,...)} has an interval of 12 cycles, and \lstinline{fft_stage(3,...)} has an interval of 14 cycles. When using \lstinline{dataflow}, the overal task interval is 14, determined by the maximum of all of the tasks/functions. This means that you should be careful in balancing optimizations across different processes with the goal of creating a balanced pipeline where the interval of each process is approximately the same. In this example, improving the interval of the \lstinline{bit_reverse} function cannot improve the overall interval of the \lstinline|fft| function. In fact, it might be beneficial to increase the latency of the \lstinline{bit_reverse} function, if it can be achieved with significantly fewer resources.

\section{Conclusion}
\label{sec:fft_conclusion}

The overall goal is to create the most optimal design, which is a function of your application needs. This may be to create the smallest implementation. Or the goal could be creating something that can perform the highest throughput implementation regardless of the size of the FPGA or the power/energy constraints. Or the latency of delivering the results may matter if the application has real-time constraints. All of the optimizations change these factors in different ways.

In general, there is no one algorithm on how to optimize your design. It is a complex function of the application, design constraints, and the inherent abilities of the designer himself. Yet, it is important that the designer have a deep understanding of the application itself, the design constraints, and the abilities of the synthesis tool.

We attempted to illustrate these bits of wisdom in this chapter. While the \gls{fft} is a well studied algorithm, with a large number of known hardware implementation tricks, it still serves as a good exemplar for high-level synthesis. We certainly did not give all of the tricks for optimization. %e leave that as an exercise in Chapter \ref{chapter:ofdm} where we task the designer to create an simple orthogonal frequency-division multiplexing receiver. The core of this an \gls{fft}.
Regardless, we attempted to provide some insight into the key optimizations here, which we hope serve as a guide to how to optimize the \gls{fft} using the \VHLS tool.

First and foremost, understand the algorithm. We spent a lot of time explaining the basics of the \gls{fft}, and how it relates to the \gls{dft}. We hope that the reader understands that this is the most important part of building optimal hardware. Certainly, the designer could translate C/MATLAB/Java/Python code into \VHLS and get an working implementation. And that same designer could somewhat blindly apply directives to achieve better results. But that designer is not going to get anywhere close to optimal results without a deep understanding of the algorithm itself.

Second, we provide an introduction to  task level pipelining using the \lstinline{dataflow} directive. This is a powerful optimization that is not possible through code restructuring. I.e., the designer must use this optimization to get such a design. Thus, it is important that the designer understand its power, drawbacks, and usage.

Additionally, we give build upon some of the optimizations from previous chapters, e.g., loop unrolling and pipelining. All of these are important to get an optimized \gls{fft} hardware design. While we did not spend too much time on these optimizations, they are extremely important.

Finally, we tried to impress on the reader that these optimizations cannot be done in isolation. Sometimes the optimizations are independent, and they can be done in isolation. For example, we can focus on one of the tasks (e.g., in the \lstinline{bit_reverse} function as we did in Section \ref{sec:fft_bit_reversal}). But many times different optimizations will effect another. For example, the \lstinline{inline} directive will effect the way the pipelining of a function. And in particular, the way that we optimize tasks/functions can propagate itself up through the hierarchy of functions.  The takeaway is that it is extremely important that the designer understand the effects of the optimizations on the algorithm, both locally and globally.


\textbf{Modify the code to} \verb+diffusion_eq1_FD.m+  \textbf{with sparse matrices}

\vspace{0.3in}

In this problem we want to show the need to take advantage of some properties of matrices. In particular, the need to use the fact that in some problems the matrices involved have a majority of its entries being zero. This type of matrices are called sparse matrices. If we code using this fact, the computer will store the value and position of the nonzero entries instead of the whole matrix, resulting in a much short computation time, as we will show. The code generates the following solution to the diffusion differential equation (for the case of $N=100$ nodes).
\begin{figure}[H]
\centering     %%% not \center
{\includegraphics[scale=0.75]{P2_surf1.eps}}
\caption{Solution of the diffusion equation, $N=100$.}
\end{figure}

We have measured the time used to calculate the solution for different number of nodes. In figure 2 we can see how using sparse matrices makes our code run faster and the bigger $N$ the bigger the difference between the two speeds.
\begin{figure}[H]
\centering     %%% not \center
{\includegraphics[scale=0.75]{P2_times.eps}}
\caption{Elapsed time for regular and sparse matrices.}
\end{figure}

In the next table we can see the measured times for both cases for different values for $N$ as well as the ratio between the two. As we could observe in figure 2, the ratio gets bigger as we increase the dimesion of the matrices, which proves the importance of taking advantage of sparse matrices specially for bigger systems.

\begin{table}[H]
\centering
\begin{tabular}{c|c|c|c}
%\hline
%\multicolumn{3}{|c|}{Datos}\\
N  & Regular Matrices & Sparse Matrices & Ratio\\
\hline
$100$ & $0.6716$ & $   0.3347$ & $   2.0064$\\
$200$ & $3.0701$ & $    1.4052$ & $    2.1848$\\
$300$ & $8.3650$ & $    3.4260$ & $    2.4416$\\
$400$ & $18.4907$ & $    6.4504$ & $    2.8666$\\
$500$ & $36.2399$ & $   11.1142$ & $    3.2607$\\
$600$ & $62.5694$ & $   16.6995$ & $    3.7468$\\
$700$ & $118.2346$ & $   24.1097$ & $    4.9040$\\
$800$ & $  217.2509$ & $   34.0877$ & $    6.3733$\\
\end{tabular}
\caption{$L_2$ error norm of different methods.}
\end{table}

\subsection*{Matlab code for this problem}
\begin{verbatim}
%% Problem 2
clear all; close all
legendfontsize=14;
axisfontsize=14;
labelfontsize=16;
%% Original code, non-sparse matrices.
time=zeros(8,1);
for j=1:8
    tic
    N = 100*j;
    x = linspace(-1,1,N)';
    dx = x(2)-x(1);

    unos = ones(N-2,1);
    D2 = ( diag(unos(1:end-1),-1)+diag(unos(1:end-1),1)-2*diag(unos,0) )/dx^2;
    B = zeros(N-2,2);
    B(1,1) = 1/dx^2; B(end,end) =1/dx^2;
    % initial condition
    u0 = sin(pi*x)+0.2*sin(5*pi*x);
    % boundary conditions
    g1 = @(t) .5*sin(50*t);
    g2 = @(t) 0*t;

    t = 0:0.001:.5;

    [T,U] = ode45(@(t,u) D2*u+B*[g2(t);g1(t)], t, u0(2:end-1));
%     for k = 2:length(t)
%         plot(x,[0;U(k,:)';g1(t(k))],'*-')
%         ylim([-1.5 1.5])
%         shg
%         drawnow
%     end
    time(j)=toc;
    if N==100
        figure
        [T,X]= meshgrid(t,x(2:end-1));
        surf(T,X,U','edgecolor','none')
        xlabel('$x$','fontsize',labelfontsize,...
            'interpreter','latex')
        ylabel('$y$','fontsize',labelfontsize,...
            'interpreter','latex')
        zlabel('$\phi_V(x,y)$','fontsize',labelfontsize,...
            'interpreter','latex')
        set(gca,'fontsize',axisfontsize)
        txt='Latex/FIGURES/P2_surf1';
        saveas(gcf,txt,'epsc')
    end
end

%% Modified code, sparse matrices.
sparsetime=zeros(8,1);
for j=1:8
    tic
    N = 100*j;
    x = linspace(-1,1,N)';
    dx = x(2)-x(1);

    unos = ones(N-2,1);
    D2 = sparse(( diag(unos(1:end-1),-1)+diag(unos(1:end-1),1)-2*diag(unos,0) )/dx^2);
    B = sparse(zeros(N-2,2));
    B(1,1) = 1/dx^2; B(end,end) =1/dx^2;
    % initial condition
    u0 = sin(pi*x)+0.2*sin(5*pi*x);
    % boundary conditions
    g1 = @(t) .5*sin(50*t);
    g2 = @(t) 0*t;

    t = 0:0.001:.5;

    [T,U] = ode45(@(t,u) D2*u+B*[g2(t);g1(t)], t, u0(2:end-1));
%     for k = 2:length(t)
%         plot(x,[0;U(k,:)';g1(t(k))],'*-')
%         ylim([-1.5 1.5])
%         shg
%         drawnow
%     end
    sparsetime(j)=toc;
    if N==100
        figure
        [T,X]= meshgrid(t,x(2:end-1));
        surf(T,X,U','edgecolor','none')
        txt='Latex/FIGURES/P2_surf2';
        saveas(gcf,txt,'epsc')
    end
end
%%
figure
Nj=100:100:800;
plot(Nj,time,'r*',Nj,sparsetime,'b*')
legend('Non-Sparse','Sparse')
grid on
txt='Latex/FIGURES/P2_times';
saveas(gcf,txt,'epsc')
\end{verbatim}


\section{Greatest Common Divisor}

\frame{
{Part 2: Greatest Common Divisors}

\tableofcontents[currentsection,hideallsubsections, firstsection=1, sections={1-5}]
}

\begin{frame}{Greatest Common Divisor (GCD)}

  The \structure{Greatest Common Divisor} of $a$ and $b$ ($gcd(a,b)$), is the largest $c$ so that $c|a \land c|b$.\bigskip

  Examples:
  \begin{itemize}
  \item $gcd(10,12)$ = 2 \hfill $2|10 \land 2|12$
  \item $gcd(13,12)$ = 1 \hfill 13 and 12 have no common factors, and $1|x, \forall x$
  \item $gcd(17,17)$ = 17
  \item $gcd(0,n)$ = n \hfill $\forall n\in\mathbb{N}, n > 0, n|0$
  \end{itemize}
\end{frame}

\subsection{Computing the GCD}
\begin{frame}{Euclidean Algorithm}{The Remainder Lemma}

  The GCD is easy to compute, by using the \structure{Remainder Lemma}:\bigskip

  \begin{equation*}
    gcd(a,b) = gcd(b, \text{remainder}(a,b)), \text{ for } b\neq 0
  \end{equation*}\bigskip

  \begin{block}{Proof Idea}

    \begin{itemize}
    \item The division axiom states that: $a = qb + r, 0\leq r < b$;
    \item $c = gcd(a,b) \implies c|a \land c|qb$;
    \item $r = a - qb$, and $c|a \land c|qb \implies c|(a + (-1)*qb)$
    \item So $c|r$
    \end{itemize}
  \end{block}
\end{frame}

\begin{frame}{Euclidean Algorithm}{Calculating with the Remainder Lemma}

  To calculate the GCD of $a$ and $b$, we repeatedly calculate the remainder
  of $a$ and $b$, and replace $a$ with the remainder.\bigskip

  \begin{center}
    GCD(899, 493) -- a = 899, b = 493
  \end{center}\bigskip

  \begin{itemize}
  \item $899 = 493 \times 1 + 406$ \hfill \structure{division axiom}
  \item GCD(899, 493) = GCD(493, 406) \hfill \structure{remainder lemma}
  \item GCD(493, 406) = GCD(406, 87) \hfill \structure{$493 =
    406\times 1 + 87$}
  \item GCD(406,87) = GCD(87,58) \hfill \structure{$406 = 87\times 4 + 58$}
  \item GCD(87,58) = GCD(58,29) = GCD(29,0) = \alert{29}
  \end{itemize}
\end{frame}

\subsection{Proof of correctness}

\begin{frame}{Euclidean Algorithm}{State Machine}
  Let's use a State Machine to prove the correctness of Euclidean Algorithm:\bigskip

  \begin{itemize}
  \item \structure{States::=} $\mathbb{N}\times \mathbb{N}$\hspace{2cm} (values of $a$ and $b$)\bigskip
  \item \structure{Start State::=} $(a,b)$ \bigskip
  \item \structure{State Transitions::=}
    $(x,y) \rightarrow (y,rem(x,y))$ if $y\neq 0$ \bigskip
  \item \structure{End State::=} $y = 0$
  \end{itemize}\bigskip

  Remember that to prove correctness, we have to:
  \begin{itemize}
    \item Prove \structure{partial correctness}
    \item Prove \structure{termination}
  \end{itemize}
\end{frame}

\begin{frame}{Proof of GCD}{Proof of Partial Correctness}

  To prove partial correctness, we have to prove the \structure{preserved invariant} $P((x,y)) ::= [gcd(x,y) = gcd(a,b)]$.

  \begin{proof}
    We prove the preserved invariant $P((x,y))$ by induction:

    \begin{itemize}
      \item P(start) is trivial: $gcd(a,b) = gcd(a,b)$
      \item If $P((x,y))$ is true, then it is still true for any transition.
      \begin{itemize}
        \item There is only one transition: $(x,y) \to (y,\text{rem}(x,y))$
        \item The \structure{Remainder Lemma} says that: gcd(y,y) = gcd(y,rem(x,y));
      \end{itemize}
    \end{itemize}
    By these two items, we proved that $P()$ holds for any state in the machine.
  \end{proof}
\end{frame}

\begin{frame}{Proof of GCD}{Proof of Termination}
  \begin{itemize}
    \item The transition is $(x,y) \to (y,\text{rem}(x,y))$;
    \item By the division axiom, $0 \leq \text{ rem}(x,y) < y$;
    \item So $y$ becomes smaller after every transition;
    \item When $y = 0$, \structure{the machine halts};
  \end{itemize}\bigskip

  By the way, when the machine stops, the state is $(x_e,0)$. Because the preserved invariant state that $gcd(a,b) = gcd(x,y)$, and $gcd(x,0) = x$, the final result of the algorithm is that $gcd(a,b) = x_e$.
\end{frame}

\begin{frame}{How fast is the GCD?}

  Analysing the state machine, we can calculate how fast the GCD terminates:\bigskip

  \begin{itemize}
    \item At each transition, x is replaced by y. There are two cases:
      \begin{enumerate}
      \item $y \leq x/2$: so x is halved this step.\medskip
      \item $y > x/2$: so rem$(x,y) = x - y$ and x will get
        halved at the \emph{next} step.
      \end{enumerate}\bigskip
    \item So every two steps, x gets halved (or even smaller)
    \item This means that after $\leq 2\log_2x$ steps, the algorithm stops.
  \end{itemize}\bigskip

  So GCD$(a,b)$ is calculated after $2\log_2a$ steps.
\end{frame}

\subsection{The Pulverizer}
\begin{frame}{GCD and Linear Combinations}

  Remember that we showed that a divisor $c$ of $a$ and $b$ is also a divisor of $(sa+tb)$.\bigskip

  Note that $c$ is a divisor of itself too, so we can represent $c$ as $(sa+tb)$ for some $s$ and $t$.\bigskip

  The \structure{Extended Euclid Algorithm} (also called "The Pulverizer") can calculate $s$ and $t$ so that gcd$(a,b) = sa + tb$.\bigskip

  It is useful to note also that the gcd$(a,b)$ divides {\bf every linear combination} of $a$ and $b$.
\end{frame}

\begin{frame}{The Pulverizer Algorithm}

    \structure{Calculate} Euclid's algorithm as usual:
    \begin{itemize}
    \item GCD(x,y) = GCD(y,rem(x,y)\hspace{2cm} {\bf Start}: GCD(a,b)
    \end{itemize}

    \bigskip

    As we calculate GCD, \structure{keep track of} four coefficients: {\bf c,d,e,f}
    \begin{itemize}
    \item $x = ca+db$ and $y = ea+fb$
    \item {\bf at start:} $x = 1a + 0b$, $y = 0a+1b$
    \item {\bf update:} $x_{\text{next}} = y = ea + fb$
    \item $y_{\text{next}} =
      \text{rem}(x,y) = x - qy =
      ca+db-q(ea+fb)$
    \item $y_{\text{next}} = (c-qe)a+(d-qf)b$
    \end{itemize}
\end{frame}

\begin{frame}{The Pulverizer Algorithm}{Example}

    {\bf a = 899, b = 493}\\
    \hfill (remember: $e_1 = c_0-q_0e_0, f_1 = d_0 - q_0f_0$)

    \vfill

    \begin{tabular}{l|l|l|l|r|r|r|r}
      a & b & q & rem(a,b) & c & d & e & f\\
      \hline
      899 & 493 & 1 & 406 & 1 & 0 & 0 & 1\\
      493 & 406 & 1 & 87 & 0 & 1 & 1 & -1\\
      406 & 87 & 4 & 58 & 1 & -1 & -1 & 2\\
      87 & 58 & 1 & 29 & -1 & 2 & 5 & -9\\
      58 & 29 & 2 & 0 & 5 & -9 & -6 & 11\\
      29 & 0 & - & - & -6 & 11 & - & -\\
    \end{tabular}

    \begin{center}
      GCD(899,493) = 29 = $-6\times 899 +11\times 493$
    \end{center}
\end{frame}

\begin{frame}{The Pulverizer Algorithm}{Positive coefficients}

  So the Pulverizer calculates one linear combination of $a$ and $b$ which is equal to the GCD:

  \begin{equation*}
  \text{GCD}(899, 493) = -6\times 899 + 11\times 493
  \end{equation*}\bigskip

  It is possible to obtain other linear combinations, by defining $s$ and $t$ as follows:

  \begin{equation*}
    \text{GCD}(899, 493) = (-6 + 493k)899 + (11 - 899k)493, \text{ for any }k
  \end{equation*}\bigskip

  For example, if we set $k = 1$, we can find the following coefficients for $s$ and $t$:

  \begin{equation*}
    \text{GCD}(899, 493) = 487\times 899 - 888\times 493
  \end{equation*}

\end{frame}

\begin{frame}{The Pulverizer Algorithm}{Positive coefficients}

  Remember the robot from last class? The position of the robot was $x = 5a - 3b$. For any $x$, we can use the pulverizer to find $a$ and $b$.\bigskip

  \begin{itemize}
    \item Let's say that we want to find the path of the robot for $x = 8$
    \item gcd$(5,3) = 1 = 2\times5 - 3\times 3$, using the pulverizer.
    \item $8 = 8\times 1 = 8\times(2\times 5 - 3\times3) = (8\times2)5 - (8\times 3)3$
    \item Result: 16 moves forward, 24 moves back.
  \end{itemize}\bigskip

  This may not be the best solution for $x=8$, but it is an easy and fast algorithm to calculate a solution.
\end{frame}


\documentclass{ximera}
\input{../preamble}
\title{Exercises: Sequences}
%%%%%\author{Philip T. Gressman}

\begin{document}
\begin{abstract}
Exercises relating to sequences.
\end{abstract}
\maketitle

\begin{exercise}
The sequence $a_n = 1/n^3$ has limit $L = 0$. Suppose $\epsilon = 1/64$; find a threshold $N$ such that
\[ |a_n -L| < \epsilon \]
is guaranteed to hold for all $n > N$ (take your value of $N$ as small as possible).
\[ N = \answer{4}. \]
\end{exercise}

\begin{exercise}
The sequence $a_n = (2n^2 + (-1)^n)/n^2$ has limit $L = 2$. Suppose $\epsilon = 1/100$; find a threshold $N$ such that
\[ |a_n -L| < \epsilon \]
is guaranteed to hold for all $n > N$ (take your value of $N$ as small as possible).
\[ N = \answer{10}. \]
\end{exercise}

\begin{exercise}%[APEX0801SEQS09]
Determine the \(n^\text{th}\) term of the given sequence. $a_1 = 4$, $a_2 = 7$, $a_3 = 10$, $a_4 = 13$, $a_5 = 16$, \(\ldots\)
\[ a_n = \answer{3n+1}. \]
\end{exercise}
\begin{exercise}%[APEX0801SEQS09]
Determine the \(n^\text{th}\) term of the given sequence. $a_1 = 3$, $a_2 = -5/2$, $a_3 = 7/4$, $a_4 = -9/8$, $a_5 = 11/16$, \(\ldots\)
\[ a_n = \answer{\frac{(-1)^{n-1} (2n+1)}{2^{n-1}}}. \]
\end{exercise}

\begin{exercise}%[APEX0801SEQS17]
Determine whether the sequence converges or diverges. If convergent, give the limit of the sequence.
\[ a_n = \left\{(-1)^n\frac{n}{n+1}\right\} \]
The sequence \wordChoice{\choice{converges}\choice[correct]{diverges}} to $\answer{N/A}$ (enter N/A if the sequence does not converge to a finite answer).
\end{exercise}

\begin{exercise}%[APEX0801SEQS18]
Determine whether the sequence converges or diverges. If convergent, give the limit of the sequence.
\[ a_n = \frac{4n^2-n+5}{3n^2+1} \]
The sequence \wordChoice{\choice[correct]{converges}\choice{diverges}} to $\answer{4/3}$ (enter N/A if the sequence does not converge to a finite answer).
\end{exercise}

\begin{exercise}%[APEX0801SEQS19]
Determine whether the sequence converges or diverges. If convergent, give the limit of the sequence.
\[ a_n = \frac{4^n}{5^n} \]
The sequence \wordChoice{\choice[correct]{converges}\choice{diverges}} to $\answer{0}$ (enter N/A if the sequence does not converge to a finite answer).
\end{exercise}

\begin{exercise}
Determine whether the sequence converges or diverges. If convergent, give the limit of the sequence.
\[ a_n = \left(1 - \frac{3}{n} \right)^{-n} \]
The sequence \wordChoice{\choice[correct]{converges}\choice{diverges}} to $\answer{e^3}$ (enter N/A if the sequence does not converge to a finite answer).
\begin{hint}
Take the reciprocal and compare to your reference list of commonly-occurring limits.
\end{hint}
\end{exercise}

\begin{exercise}
Determine whether the sequence converges or diverges. If convergent, give the limit of the sequence.
\[ a_n = \left(1 - \frac{3}{n}\right)^{-n^2} \]
The sequence \wordChoice{\choice{converges}\choice[correct]{diverges}} to $\answer{N/A}$ (enter N/A if the sequence does not converge to a finite answer).
\begin{hint}
If a sequence $b_n$ tends to $e^3$, what will the sequence $(b_n)^n$ do?
\end{hint}
\end{exercise}

\begin{exercise}
Determine whether the sequence converges or diverges. If convergent, give the limit of the sequence.
\[ a_n = \frac{(1.1)^n}{n} \]
The sequence \wordChoice{\choice{converges}\choice[correct]{diverges}} to $\answer{N/A}$ (enter N/A if the sequence does not converge to a finite answer).
\begin{hint}
What are the relative orders of growth of numerator versus denominator?
\end{hint}
\end{exercise}

\begin{exercise}
Determine whether the sequence converges or diverges. If convergent, give the limit of the sequence.
\[ a_n = \frac{(0.9)^n}{n} \]
The sequence \wordChoice{\choice[correct]{converges}\choice{diverges}} to $\answer{0}$ (enter N/A if the sequence does not converge to a finite answer).
\end{exercise}

\begin{exercise}
Determine whether the sequence converges or diverges. If convergent, give the limit of the sequence.
\[ a_n = n^{1000000} (0.9)^n \]
The sequence \wordChoice{\choice[correct]{converges}\choice{diverges}} to $\answer{0}$ (enter N/A if the sequence does not converge to a finite answer).
\end{exercise}

\begin{exercise}
Determine whether the sequence converges or diverges. If convergent, give the limit of the sequence.
\[ a_n = \frac{\ln n}{n} \]
The sequence \wordChoice{\choice[correct]{converges}\choice{diverges}} to $\answer{0}$ (enter N/A if the sequence does not converge to a finite answer).
\end{exercise}

\begin{exercise}
Determine whether the sequence converges or diverges. If convergent, give the limit of the sequence.
\[ a_n = \frac{\ln n}{n^{0.00001}} \]
The sequence \wordChoice{\choice[correct]{converges}\choice{diverges}} to $\answer{0}$ (enter N/A if the sequence does not converge to a finite answer).
\end{exercise}

\begin{exercise}
Let $b_n$ be the sequence given by
\[ b_1 = 0 \ \text{ and } \ b_{n+1} = \frac{2 + b_n}{3} \ \text{ for } n \geq 1 \]
converges. Compute its limit.
\[ \lim_{n \rightarrow \infty} b_n = \answer{1}. \]
\end{exercise}

\begin{exercise}
Determine whether the sequence converges or diverges. If convergent, give the limit of the sequence.
\[ a_1 = 1 \ \text{ and } \ a_{n+1} = a_n + \frac{1}{a_n} \ \text{ for } n \geq 1 \]
The sequence \wordChoice{\choice{converges}\choice[correct]{diverges}} to $\answer{N/A}$ (enter N/A if the sequence does not converge to a finite answer).
\end{exercise}

\begin{exercise}
Determine whether the sequence converges or diverges. If convergent, give the limit of the sequence.
\[ a_1 = 1 \ \text{ and } \ a_{n+1} = a_n + \frac{1}{4} (4 - a_n^2) \ \text{ for } n \geq 1 \]
The sequence \wordChoice{\choice[correct]{converges}\choice{diverges}} to $\answer{2}$ (enter N/A if the sequence does not converge to a finite answer).

\end{exercise}

\begin{exercise}
Let $a_n$ be the sequence given by
\[ a_1 = \frac{1}{4}, \ \text{ and }  \ a_{n+1} = 2 a_n(1-a_n) \ \text{ for } n \geq 1 \]
converges. Compute its limit.
\[ \lim_{n \rightarrow \infty} a_n = \answer{\frac{1}{2}}. \]
\begin{hint}
If $a_n$ happens to be positive and less than $1/2$, then $2 (1-a_n) > 1$, so this forces $2 a_n(1-a_n) > a_n$ (meaning that the term after $a_n$ will be larger than $a_n$.
\begin{hint}
The function $2 x(1-x)$ is nonnegative on the interval $[0,1]$ and has a maximum value of $1/2$ attained at $x = 1/2$. This means that if $a_n$ is anything between $0$ and $1$, the next term of the sequence will always be between $0$ and $1/2$.
\end{hint}
\end{hint}
\end{exercise}

\section*{Sample Quiz Questions}

\begin{question}%%%%%[SeqConvSQRT01]

Find the limit of the sequence
\[\lim_{n \rightarrow \infty} \sqrt{\frac{2n-2}{2n^2-4n+3}}.\] \offline{Justify your response.}
(Hints will not be revealed until after you choose a response.)
\begin{multiplechoice}
\choice[correct]{\(\displaystyle 0\)}
\choice{\(\displaystyle \frac{1}{3}\)}
\choice{\(\displaystyle \frac{1}{2}\)}
\choice{\(\displaystyle 1\)}
\choice{\(\displaystyle 2\)}
\choice{\(\displaystyle 3\)}
\end{multiplechoice}
\begin{feedback}
Because the square root function is continuous, you can pass the limit through it and compute
\[ \sqrt{\lim_{n \rightarrow \infty} \frac{2n-2}{2n^2-4n+3}}. \]
\begin{hint}
Reduce numerator and denominator to the dominant terms (in the regime $n \rightarrow \infty$).
\begin{hint}
\[\begin{aligned}
\lim_{n \rightarrow \infty} \sqrt{\frac{2n-2}{2n^2-4n+3}} & = \lim_{n \rightarrow \infty} \sqrt{\frac{2n^{-1}-2n^{-2}}{2-4n^{-1}+3n^{-2}}} \\
 & = \sqrt{\frac{\lim_{n \rightarrow \infty} 2n^{-1}-2n^{-2}}{\lim_{n \rightarrow \infty} 2-4n^{-1}+3n^{-2}}} \\
 & = \sqrt{\frac{0}{2}} = 0.
\end{aligned}\]
\end{hint}
\end{hint}
\end{feedback}

\end{question}

\begin{question}%%%%%[SeqConvORDER19]

Determine whether the limit below exists. If it exists, find its value.
\[\lim_{n \rightarrow \infty} \frac{(-1)^{n+1}n^2 - 2^{-n-1}}{(-1)^{n}n^2 + 4^{-n}}.\] \offline{Justify your response.}
(Hints will not be revealed until after you choose a response.)
\begin{multiplechoice}
\choice[correct]{\(\displaystyle -1\)}
\choice{\(\displaystyle 0\)}
\choice{\(\displaystyle \frac{1}{2}\)}
\choice{\(\displaystyle 2\)}
\choice{\(\displaystyle 3\)}
\choice{limit does not exist}
\end{multiplechoice}
\begin{feedback}
Comparing the orders of growth of the terms in the numerator, the first term dominates because \(|-1|>|1/2|\).
\begin{hint} Likewise the first term dominates in the denominator because  \(|-1|>|1/4|\). \begin{hint} Neglecting non-dominant terms leads to the limit \[\lim_{n \rightarrow \infty} \frac{(-1)^{n+1}n^2}{(-1)^{n}n^2}\] which simply equals \(-1\). \end{hint} \end{hint}
\end{feedback}

\end{question}

\begin{question}%%%%%[SeqConvEXPN43]

Determine whether the limit below exists. If it exists, find its value.
\[\lim_{n \rightarrow \infty} \left(\frac{4n - 3}{4n + 1}\right)^{n}.\] \offline{Justify your response.}
(Hints will not be revealed until after you choose a response.)
\begin{multiplechoice}
\choice{\(\displaystyle 0\)}
\choice{\(\displaystyle 1\)}
\choice[correct]{\(\displaystyle e^{-1}\)}
\choice{\(\displaystyle e\)}
\choice{\(\displaystyle e^2\)}
\choice{limit does not exist}
\end{multiplechoice}
\begin{feedback}
First observe that
\[\frac{4n - 3}{4n + 1} = 1 - \frac{4}{4n + 1} \rightarrow 1\]
as \(n \rightarrow \infty\). \begin{hint} Next, in light of the known limit \((1+x/k)^{k} \rightarrow e^x\) as \(k \rightarrow \infty\), manipulate exponents to see that  \[\left(1 - \frac{4}{4n + 1}\right)^n = \left( \left(1 - \frac{4}{4n + 1}\right)^{4n + 1} \right)^{ 1/4} \left(1 - \frac{4}{4n + 1}\right)^{-1/4}. \] \begin{hint} As \(n \rightarrow \infty\), the first term on the right-hand side tends to \(e^{-1}\) and the second term tends to \(1\). Thus the original sequence tends to \(e^{-1}\) as well. \end{hint} \end{hint}
\end{feedback}

\end{question}

\begin{question}%%%%%[SeqConvEXPN39]

Determine whether the limit below exists. If it exists, find its value.
\[\lim_{n \rightarrow \infty} \left(\frac{n + 3}{2n - 2}\right)^{n^2}.\] \offline{Justify your response.}
(Hints will not be revealed until after you choose a response.)
\begin{multiplechoice}
\choice[correct]{\(\displaystyle 0\)}
\choice{\(\displaystyle 1\)}
\choice{\(\displaystyle e^{-1}\)}
\choice{\(\displaystyle e\)}
\choice{\(\displaystyle e^2\)}
\choice{limit does not exist}
\end{multiplechoice}
\begin{feedback}
First observe that
\[\frac{n + 3}{2n - 2} = \frac{1}{2} + \frac{2}{n - 1} \rightarrow \frac{1}{2}\]
as \(n \rightarrow \infty\). \begin{hint} Since the limit is positive and less than one, raising this expression to increasingly large powers generates a sequence which converges rapidly to zero. \end{hint}
\end{feedback}

\end{question}

\section*{Sample Exam Questions}

\begin{question}%%%%%[2016C.08]

Determine whether the sequence \(\displaystyle a_n = (-1)^{n-1} \frac{n^2}{1 + n^2 + n^3}\) converges or diverges. If it converges, find its limit.
\begin{multiplechoice}
\choice{divergent, \(\displaystyle \lim_{n \rightarrow \infty} a_n = 0\)}
\choice{convergent, \(\displaystyle \lim_{n \rightarrow \infty} a_n = 1\)}
\choice[correct]{convergent, \(\displaystyle \lim_{n \rightarrow \infty} a_n = 0\)}
\choice{convergent, \(\displaystyle \lim_{n \rightarrow \infty} a_n = -1\)}
\choice{divergent, \(\displaystyle \lim_{n \rightarrow \infty} a_n = \infty\)}
\choice{divergent, limit doesn't exist}
\end{multiplechoice}

\end{question}

\end{document}


\subsection{Types of Matrices}
\noindent
Below is a list of different types of matrices and their special properties.
\begin{enumerate}[label=]
	\item A square matrix has the same number of rows as columns.
		\begin{equation*}
			\begin{bmatrix}
				1 & 3 & 7 \\
				0 & 2 & -1 \\
				2 & 7 & 9
			\end{bmatrix}
		\end{equation*}
	\item Row vectors have one column. Column vectors have one row.
		\begin{equation*}
			\begin{bmatrix}
			1 & 3 & 0
			\end{bmatrix} \text{, } \begin{bmatrix}
			1 \\
			3 \\
			0
			\end{bmatrix}
		\end{equation*}
	\item Upper triangular matrices have all 0's below the main diagonal. Lower triangular matrices have all 0's above the main diagonal.
		\begin{equation*}
			\begin{bmatrix}
				1 & 3 & 7 \\
				0 & 2 & -1 \\
				0 & 0 & 9
			\end{bmatrix} \text{, } \begin{bmatrix}
				1 & 0 & 0 \\
				0 & 2 & 0 \\
				2 & 7 & 9
			\end{bmatrix}
		\end{equation*}
	\item Diagonal matrices are both upper and lower triangular. They only have non-zero entries on the main diagonal.
		\begin{equation*}
			\begin{bmatrix}
				1 & 0 & 0 \\
				0 & 2 & 0 \\
				0 & 0 & 9
			\end{bmatrix}
		\end{equation*}
	\item The identity matrix is one of the most common matrices. It is square, diagonal, and has all 1's on the main diagonal. It's the multiplicative identity for matrices.
		\begin{equation*}
			I_3 = \begin{bmatrix}
				1 & 0 & 0 \\
				0 & 1 & 0 \\
				0 & 0 & 1
			\end{bmatrix}
		\end{equation*}
	\item The inverse matrix of $A$, $A^{-1}$, is such that
		\begin{equation*}
			A^{-1}A = AA^{-1} = I
		\end{equation*}
	\item The transpose matrix of $A$, $A^T$, is where the rows and columns of $A$ are swapped.
		\begin{equation*}
			A = \begin{bmatrix}
				1 & 3 & 2 & -1 \\
				-5 & 7 & 3 & 0 \\
			\end{bmatrix} \implies A^T = \begin{bmatrix}
				1 & -5 \\
				3 & 7 \\
				2 & 3 \\
				-1 & 0
			\end{bmatrix}
		\end{equation*}
\end{enumerate}


\section{Mallow's $C_p$}
Mallow's  $\mbox{C}_p$  is a technique for model
selection in regression (Mallows 1973). The $\mbox{C}_p$ statistic is
defined  as a criteria
to assess fits when models with different numbers of parameters are being
compared. It is given by
\begin{equation}
\label{Cpdef}
\mbox{C}_p = \frac{\mbox{RSS}(p)}{\sigma^2} - N + 2p
\end{equation}
If model(p) is correct then  $\mbox{C}_p$ will tend to be close to or
smaller than $p$. Therefore a simple plot of $\mbox{C}_p$ versus $p$ can be used
to decide amongst models.

In the case of ordinary linear regression, Mallow's method is based on
estimating the mean squared error (MSE) of
the estimator $\hat{\bg{\beta}\:}\!_p = {(\mathbf{X}_p'\mathbf{X}_p)^{-1}\mathbf{X}_p'\mathbf{Y}}$,
\[
\E [\hat{\bg{\beta}\:}\!_p - \bg{\beta}]^2
\]
via a quantity based on the residual sum of squares (RSS)
\begin{eqnarray*}
\mbox{RSS}(p) &=& \sum_{n=1}^N (y_n
-\mathbf{x}_n\hat{\bg{\beta}\:}\!_p)^2\\
&=& (\mathbf{Y}-\mathbf{X}_p\hat{\bg{\beta}\:}\!_p)'(\mathbf{Y}-\mathbf{X}_p\hat{\bg{\beta}\:}\!_p)\\
&=& \mathbf{Y}' ( \mathbf{I}_N - \mathbf{X}_p(\mathbf{X}_p'\mathbf{X}_p)^{-1}\mathbf{X}_p') \mathbf{Y}
\end{eqnarray*}
Here $\mathbf{I}_N$ is an $N\times N$ identity matrix.
By using a result for quadratic forms, presented for example as
Theorem 1.17 in Seber's book, page 13, namely
\[
\E[\mathbf{Y'AY}] = \E\mathbf{[Y']A}\E[\mathbf{Y}] + \tr[\bg{\Sigma}
\mathbf{A}]
\]
$\bg{\Sigma}$ being the variance matrix of $\mathbf{Y}$,
we find that
\begin{eqnarray*}
\E [\mbox{RSS}(p)]
&=& \E[\mathbf{Y}' ( \mathbf{I}_N - \mathbf{X}_p(\mathbf{X}_p'\mathbf{X}_p)^{-1}\mathbf{X}_p') \mathbf{Y}]\\
&=&\E [\hat{\bg{\beta}\:}\!_p - \bg{\beta}]^2 + \tr\left[\mathbf{I}_N - \mathbf{X}_p(\mathbf{X}_p'\mathbf{X}_p)^{-1}\mathbf{X}_p'\right]\sigma^2 \\
&=& \E[ \hat{\bg{\beta}\:}\!_p - \bg{\beta}]^2 + \sigma^2\left(N - \tr
\left[{(\mathbf{X}_p'\mathbf{X}_p)(\mathbf{X}_p'\mathbf{X}_p)^{-1}}\right]\right)\\
&=& \E [\hat{\bg{\beta}\:}\!_p - \bg{\beta}]^2 + \sigma^2(N - p)
\end{eqnarray*}
where $N$ is the number of observations and $p$ is the number of
parameters. Notice that when the true model has $p$ parameters
$\E[\mbox{C}_p] = p$.
This shows why, if model(p) is correct, $\mbox{C}_p$ will
tend to be close to $p$.

One problem with the $\mbox{C}_p$  criterion is that
we have to find an
appropriate estimate of $\sigma^2$ to use for all values of $p$.

\subsection{$C_p$ for smoothers}
A more direct way of constructing an estimate of PSE is to correct the
ASR. It is easy to show that
\[
\E\{\mbox{ASR}(\lambda)\} = \left\{ 1 - n^{-1}\tr(2\bS_{\lambda} -
    \bS_{\lambda} \bS_{\lambda}') \right\} \sigma^2 + n^{-1}
    \bv_{\lambda}'\bv_{\lambda}
\]
notice that
\[
\mbox{PSE}(\lambda) - \E\{\mbox{ASR}(\lambda)\} = n^{-1}
2\tr(\bS_{\lambda}) \sigma^2
\]

This means that if we knew $\sigma^2$ we could find a ``corrected''
ASR
\[
\mbox{ASR}(\lambda) + 2\tr(\bS_{\lambda}) \sigma^2
\]
with the right expected value.

For linear regression $\tr(\bS_{\lambda})$ is the number of parameters
so we could think of $2\tr(\bS_{\lambda}) \sigma^2$ as a penalty
for large number of parameters or for un-smooth estimates.

How do we obtain an estimate for $\sigma^2$? If we had a $\lambda^*$
for which the bias is 0, then the usual unbiased estimate is
\[
\frac{\sum_{i=1}^n \{y_i - f_{\lambda^*}(x_i)\}^2}{n -
  \tr(2\bS_{\lambda^*} -   \bS_{\lambda^*} \bS_{\lambda^*}')}
\]
The usual trick is to chose one a $\lambda^*$ that does little
smoothing and consider the above estimate. Another estimate that has
been proposed it the first order difference estimate
\[
\frac{1}{2(n-1)} \sum_{i=1}^{n-1} (y_{i+1} - y_i)^2
\]

Once we have an estimate $\hat{\sigma}^2$ then we can define
\[
C_p = \mbox{ASR}(\lambda) + n^{-1} 2\tr(S_{\lambda})\hat{\sigma}^2
\]
Notice that the $p$ usually means number of parameters so it should be
$C_{\lambda}$.

Notice this motivates a definition for degrees of freedoms.