\title{FYS-STK4155 \\
Project 1}
\author{Lars Johan Brodtkorb}
\date{ }

\subsection*{Abstract}

In this exercise I have tried to figure out which method is to be preferred of ordinary least squares, Ridge and Lasso regressions. The results show a better prediction error for the ridge regression, but that may be due to a faulty implementation of the Lasso method.

\section{Introduction}

The main aim of this project is to study in more detail various
regression methods, including the Ordinary Least Squares (OLS) method,
Ridge regression and finally Lasso regression.
The methods are in turn combined with resampling techniques.

We will first study how
to fit polynomials to a specific two-dimensional function called
\href{{http://www.dtic.mil/dtic/tr/fulltext/u2/a081688.pdf}}{Franke's
	function}.  This
is a function which has been widely used when testing various  interpolation and fitting
algorithms. Furthermore, after having etsablished the model and the
method, we will employ resampling techniques such as the  cross-validation and/or
the bootstrap methods, in order to perform a proper assessment of our models.

The Franke function, which is a weighted sum of four exponentials  reads as follows
\begin{align*}
f(x,y) &= \frac{3}{4}\exp{\left(-\frac{(9x-2)^2}{4} - \frac{(9y-2)^2}{4}\right)}+\frac{3}{4}\exp{\left(-\frac{(9x+1)^2}{49}- \frac{(9y+1)}{10}\right)} \\
&+\frac{1}{2}\exp{\left(-\frac{(9x-7)^2}{4} - \frac{(9y-3)^2}{4}\right)} -\frac{1}{5}\exp{\left(-(9x-4)^2 - (9y-7)^2\right) }.
\end{align*}

The function will be defined for $x,y\in [0,1]$.  Our first step will
be to perform an OLS regression analysis of this function, trying out
a polynomial fit with an $x$ and $y$ dependence of the form $[x, y,
x^2, y^2, xy, \dots]$. We will also include cross-validation and
bootstrap as resampling techniques.  As in homeworks 1 and 2, we
can use a uniform distribution to set up the arrays of values for $x$
and $y$, or as in the example below just a fix values for $x$ and $y$ with a given step size.
In this case we will have two predictors and need to fit a
function (for example a polynomial) of $x$ and $y$.  Thereafter we will
repeat much of the same procedure using the the Ridge and
Lasso regression methods, introducing thus a dependence on the bias
(penalty) $\lambda$.

Thereafter we are going to use (real) digital terrain data and try to
reproduce these data using the same methods. We will also try to go
beyond the second-order polynomials metioned above and explore
which polynomial fits the data best.

\section{Methods}

\subsection{Measures of prediction accuracy}

Find the confidence intervals of the $\beta$ parameters by computing their variances, evaluate the Mean Squared error (MSE)
\[ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n}
\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2,
\]
and the $R^2$ score function.
If $\tilde{\hat{y}}_i$ is the predicted value of the $i-th$ sample and $y_i$ is the corresponding true value, then the score $R^2$ is defined as
\[
R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2},
\]
where we have defined the mean value  of $\hat{y}$ as
\[
\bar{y} =  \frac{1}{n} \sum_{i=0}^{n - 1} y_i.
\]

\subsection{Ordinary least square regression}

We have an input vector $X^T = (X_1,X_2,X_3,...,X_p)$ and want to predict a real-valued output Y. The linear regression model has the form: (\cite{elementsstat}, chapter 3.2)

\begin{equation}
f(X) = \beta_0 + \sum_{j=1}^{p}X_j\beta_j
\end{equation}

The linear model either assumes that the regression function $E(Y |X)$ is
linear, or that the linear model is a reasonable approximation. The $\beta_j$ ’s are unknown parameters or coefficients, and the variables $X_j$ can come
from many different sources. One of them is that of basis expansions such as

$X_2 = X_1^2, X_3 = X_1^3$, leading to a polynomial
representation;

We can use the basis expansions as a device to achieve more flexible representations for f(X). Polynomials are an example of this. Having great flexibility in approximation allows the basis expansions to work for many occurences, but their variability may increase heavily if they are applied outside their intended scope. The coefficients to achieve a functional form
in one region can cause the function to flap about madly in remote regions (\cite{elementsstat}, chapter 5.1, p. 140).

The polynomial method produces a dictionary $D$ consisting of typically a very large number of basis functions $|D|$, far more than we can afford to fit to our data. Along
with the dictionary we require a method for controlling the complexity
of our model, using basis functions from the dictionary. There are three
common approaches:
Restriction, selection and regularization methods, which we will revisit in the Ridge and Lasso segment.

No matter the source of the $X_j$, the model is linear in the parameters. Typically we have a set of training data $(x_1, y_1)...(x_N , y_N )$ from which
to estimate the parameters $\beta$. Each $x_i = (x_{i1}, x_{i2},...,x_{ip})^T$ is a vector of feature measurements for the i-th case.

Using linear least squared fittings for $X \in \R^2$ we can get the linear function of X that minimizes the sum of the sum of squared residuals of Y.

A unique solution for $\beta$ has the form:

\begin{align*}
\beta =(\textbf{X}^T\textbf{X})^{-1}\textbf{X}^T\textbf{y}
\end{align*}
\medskip

We have made minimal assumptions about the true distribution
of the data. In order to pin down the sampling properties of $\hat{\beta}$ , we now
assume that the observations $y_i$ are uncorrelated and have constant variance
$\sigma^2$, and that the $x_i$ are fixed (non random). The $\hyphenation{variance covariance}$
matrix of the least squares parameter estimates is then given by:

\begin{align*}
Var(\hat{\beta}) =(\textbf{X}^T\textbf{X})^{-1}\sigma^2
\end{align*}
\medskip

\subsection{K-fold cross-validation algorithm}

Cross-validation is probably the simplest and most widely used method for estimating prediction
error. This method directly estimates the expected
extra-sample error $Err = E[L(Y, \hat{f(X)})]$, the average generalization error when the method $\hat{f(X)}$ is applied to an independent test sample from the joint distribution of X and Y. As mentioned earlier, we might hope that cross-validation estimates the conditional error, with the training set T held fixed. Unfortunately, cross-validation typically estimates well only the expected prediction error (\cite{elementsstat}, chapter 7.10 - 7.12).

K-fold crossvalidation uses part of the available data to fit the model, and a different part to test it. We split the data into K roughly equal-sized parts. For the kth part, we fit the model to the other $K-1$ parts of the data, and calculate the prediction error of the fitted model when predicting the kth part of the data. We do this for k = 1, 2,...,K and
combine the K estimates of prediction error.

Given a set of models $f(x,\alpha)$ indexed by a tuning parameter $\alpha$, denote
$\hat{f}^{-\kappa(x,\alpha)}$ by the $\alpha$-th model fitted with the k-th part of the data removed. Then for this set of models we define:

\begin{equation}
CV(\hat{f},\alpha) = \frac{1}{N}\sum_{i=1}^{N}L(y_i,\hat{f}^{-\kappa(i)}(x_i,\alpha))
\end{equation}
\medskip

where L is the loss function, $\kappa$ is the removed data.

The function $CV(\hat{f},\alpha)$ provides an estimate of the test error curve, and we
find the tuning parameter $\hat{\alpha}$ that minimizes it. Our final chosen model is
$f(x,\hat{\alpha})$, which we then fit to all the data.
\medskip
\medskip

The steps of the cross-validation method: \newline
(a) Find a subset of “good” predictors that show fairly strong (univariate) correlation with the class labels, using all of the samples except those in fold k.\newline
(b) Using just this subset of predictors, build a multivariate classifier, using all of the samples except those in fold k.\newline
(c) Use the classifier to predict the class labels for the samples in
fold k.

\subsection{Bootstrap}

The bootstrap method has been used in this project to estimate the variance and bias.
As with cross-validation, the bootstrap seeks to estimate the conditional error, but typically estimates well only the expected prediction error.

Suppose we have a model fit to a set of training data. We denote the
training set by $Z = (z_1, z_2,...,z_N )$ where $z_i = (x_i, y_i).$ The basic idea is
to randomly draw datasets with replacement from the training data, each
sample the same size as the original training set. This is done B times
(B = 200 in my application), producing B bootstrap datasets.
Then we refit the model to each of the bootstrap datasets, and examine
the behavior of the fits over the B replications (\cite{elementsstat} p.249)

\subsection{Ridge and Lasso regression}

\subsubsection*{Ridge regression}
Ridge regression is a simple example of a regularization approach, while the lasso is both a regularization and selection method.

Selection methods adaptively scan the dictionary and include only those basis functions $h_m$ that contribute significantly to the fit of
the model.

Regularization methods use the entire dictionary but restrict the coefficients.  (\cite{elementsstat}, chapter 5.1 p.141)

The method used follows \cite{ridge} quite closely. In chapter 1.4.2 on page 8 in \cite{ridge} we find an estimator for $\beta$ and variance in the ridge regression:

\begin{equation*}
\beta=(\textbf{X}^T\textbf{X}+\lambda\textbf{I})^{-1}\textbf{X}^T\textbf{y}
\end{equation*}

\medskip
\begin{equation*}
Var(\hat{\beta}) =(\textbf{X}^T\textbf{X}+\lambda\textbf{I})^{-1}\sigma^2
\end{equation*}
\medskip

Ridge regression shrinks the regression coefficients by imposing a penalty on their size. The ridge coefficients minimize a penalized residual sum of squares.

\begin{equation}
\hat{\beta}_{ridge} =\argmin_{\beta}({\sum_{i=1}^{N}(y_i - \beta_0 -\sum_{j=1}^{p}x_{ij}\beta_j)^2 + \lambda \sum_{j=1}^{p}\beta_j^2})
\end{equation}
\medskip

Here $\lambda\ge 0 $ is a complexity parameter that controls the amount of shrinkage. The idea of penalizing by the sum-of-squares of the parameters is also used in neural networks,
where it is known as weight decay (\cite{elementsstat} p. 63)

\subsection*{Lasso regression}
The Lasso method is a shrinking method like Ridge. Just as in ridge regression, we can re-parametrize the constant $\beta_0$ by standardizing the predictors; the solution for $\hat{\beta_0}$ is $\hat{y}$, and thereafter we fit a model without an intercept. In the signal processing literature,
the Lasso is also known as basis pursuit (Chen et al., 1998).
We can also write the Lasso problem in the equivalent Lagrangian form:

\begin{equation}
\hat{\beta}_{lasso} =\argmin_{\beta}(\frac{1}{2}{\sum_{i=1}^{N}(y_i - \beta_0 -\sum_{j=1}^{p}x_{ij}\beta_j)^2 + \lambda \sum_{j=1}^{p}\abs{\beta_j}})
\end{equation}
\medskip

The difference from the ridge method is that the ridge penalty $\sum_{j=1}^{p}\beta_j^2$ is replaced with the Lasso penalty $\sum_{j=1}^{p}\abs{\beta_j}$

\section{Code implementation}
For most of the code implementation see github repository: \href{https://github.com/larsjbro/FYS-STK3155/tree/master/project1}{https://github.com/larsjbro/FYS-STK3155/tree/master/project1}. I will include the bootstrap method with comments in the report.

\subsection{Bootstrap}

The bootstrap method was implemented with the following code, which was used to generate the prediction error, bias and variance shown in the figures.\newline

\lstinputlisting[language=Python]{bootstrap.py}

\medskip

\section{Analysis}

\subsection{Results}

\subsubsection{OLS}

Table \ref{Table:1} shows the fitted coefficients for the best ordinary least squares method fitted to the Frankfunkction along with its standard deviation, z-score and 95 percent confidence interval. As we can se from the table, only a few of the coefficients are significantly different from 0. In total table \ref{Table:1} shows there are only 5 such coefficients. The other coefficients could have possibly been set to zero.

\FloatBarrier
\begin{table}

	\begin{tabular}{lrrrrr}
		\toprule
		{} &         coef &          std &   z\_score &        Confidence interval 0.025 &        Confidence interval 0.975 \\
		\midrule
		0  &    -1.648196 &     0.532137 & -3.097317 &    -2.691165 &    -0.605227 \\
		1  &    -1.919886 &     1.848958 & -1.038361 &    -5.543776 &     1.704005 \\
		2  &     6.699335 &     3.720084 &  1.800856 &    -0.591897 &    13.990566 \\
		3  &     1.577407 &    10.841403 &  0.145498 &   -19.671353 &    22.826166 \\
		4  &     0.248456 &     0.463744 &  0.535762 &    -0.660466 &     1.157378 \\
		5  &     3.048496 &     2.491046 &  1.223781 &    -1.833865 &     7.930857 \\
		6  &   -20.088662 &    12.870111 & -1.560877 &   -45.313616 &     5.136292 \\
		7  &   -17.914271 &    16.822026 & -1.064929 &   -50.884836 &    15.056294 \\
		8  &    81.253441 &    62.107653 &  1.308268 &   -40.475323 &   202.982204 \\
		9  &     0.006444 &     1.659116 &  0.003884 &    -3.245363 &     3.258252 \\
		10 &   -10.048598 &    13.844107 & -0.725839 &   -37.182548 &    17.085353 \\
		11 &   122.772857 &    58.256501 &  2.107453 &     8.592213 &   236.953501 \\
		12 &    18.202467 &    89.033985 &  0.204444 &  -156.300937 &   192.705871 \\
		13 &  -464.346238 &   295.293981 & -1.572488 & -1043.111806 &   114.419331 \\
		14 &    -5.577415 &     2.836466 & -1.966325 &   -11.136786 &    -0.018044 \\
		15 &   -17.666890 &    15.364312 & -1.149865 &   -47.780388 &    12.446608 \\
		16 &   188.349237 &    79.758452 &  2.361496 &    32.025544 &   344.672930 \\
		17 &   127.373969 &   100.448526 &  1.268052 &   -69.501525 &   324.249463 \\
		18 &  -781.831660 &   379.979896 & -2.057561 & -1526.578570 &   -37.084750 \\
		19 &     0.140168 &     8.835873 &  0.015864 &   -17.177825 &    17.458162 \\
		20 &    72.613196 &    63.561066 &  1.142416 &   -51.964203 &   197.190596 \\
		21 &  -603.170312 &   289.455482 & -2.083810 & -1170.492631 &   -35.847993 \\
		22 &  -176.329738 &   401.060283 & -0.439659 &  -962.393449 &   609.733972 \\
		23 &  2420.894666 &  1422.585633 &  1.701757 &  -367.321940 &  5209.111272 \\
		\bottomrule

	\end{tabular}
	\caption{Values for the best fit ordinary least squares method for the data from the Frankefunction with m=300 and sigma=0.5}
	\label{Table:1}
\end{table}
\FloatBarrier

Below are figures from the ordinary least squares method:
\FloatBarrier
\begin{figure}[!ht]
	\centering
	\FloatBarrier
    \includegraphics[width=1\textwidth]{plot_ols_without_r2/olsprediction_error_vs_degrees_m300_l300_s0.png}

	\caption{This shows the bias-variance tradeoff as function of the maximum polynomial degree used in the OLS method, with a sample size of 300.}
	\label{fig:1}
\end{figure}
\FloatBarrier

\medskip

\FloatBarrier
\begin{figure}[!ht]
	\centering
	\FloatBarrier
	\includegraphics[width=1\textwidth]{plot_ols_without_r2/olsprediction_error_vs_degrees_m300_l300_s10.png}

	\caption{This shows the bias-variance tradeoff as function of the maximum polynomial degree used in the OLS method, with a sample size of 300 and a random gaussian noise with standard deviation of 0.1.}
	\label{fig:1}
\end{figure}
\FloatBarrier

\medskip

\FloatBarrier
\begin{figure}[!ht]
	\centering
	\FloatBarrier
	\includegraphics[width=1\textwidth]{plot_ols_without_r2/olsprediction_error_vs_degrees_m300_l300_s50.png}

	\caption{This shows the bias-variance tradeoff as function of the maximum polynomial degree used in the OLS method, with a sample size of 300 and a random gaussian noise with standard deviation of 0.5.}
	\label{fig:1}
\end{figure}
\FloatBarrier

\medskip

\subsubsection{Ridge}

Below are the figures for varying sample sizes, degree of polynomials and penalty with Ridge.

\FloatBarrier
\begin{figure}[!ht]
	\centering
	\FloatBarrier
	\includegraphics[width=1\textwidth]{plot_ridge_without_r2/ridgeprediction_error_m300_d5_s0.png}

	\caption{This shows the bias-variance tradeoff as function of the regularization parameter, lambda,for Ridge regression with a sample size of 300 and a polynomial degree of 5}
	\label{fig:2}
\end{figure}
\FloatBarrier

\medskip

\FloatBarrier
\begin{figure}[!ht]
	\centering
	\FloatBarrier
	\includegraphics[width=1\textwidth]{plot_ridge_without_r2/ridgeprediction_error_m300_d5_s10.png}

	\caption{This shows the bias-variance tradeoff as function of the regularization parameter, lambda,for Ridge regression with a sample size of 300, polynomial degree of 5 and standard deviation of 0.1}
	\label{fig:2}
\end{figure}
\FloatBarrier

\medskip

\FloatBarrier
\begin{figure}[!ht]
	\centering
	\FloatBarrier
	\includegraphics[width=1\textwidth]{plot_ridge_without_r2/ridgeprediction_error_m300_d5_s50.png}

	\caption{This shows the bias-variance tradeoff as function of the regularization parameter, lambda,for Ridge regression with a sample size of 300, polynomial degree of 5 and standard deviation of 0.5}
	\label{fig:2}
\end{figure}
\FloatBarrier

\medskip

\subsubsection{Lasso}
Below are the figures for varying sample sizes, polynomials and penalty with Lasso.
\FloatBarrier
\begin{figure}[!ht]
	\centering
	\FloatBarrier
	\includegraphics[width=1\textwidth]{lasso_prediction_vs_lambda/lassoprediction_error_m300_d5_s0.png}

	\caption{This shows the bias-variance tradeoff as function of the regularization parameter, lambda, for Lasso regression method with a sample size of 300 and a polynomial degree of 5}
	\label{fig:3}
\end{figure}
\FloatBarrier

\medskip

\FloatBarrier
\begin{figure}[!ht]
	\centering
	\FloatBarrier
	\includegraphics[width=1\textwidth]{lasso_prediction_vs_lambda/lassoprediction_error_m300_d5_s10.png}

	\caption{This shows the bias-variance tradeoff as function of the regularization parameter, lambda, for Lasso regression method with a sample size of 300, polynomial degree of 5 and standard deviation of 0.1.}
	\label{fig:3}
\end{figure}
\FloatBarrier

\medskip

\FloatBarrier
\begin{figure}[!ht]
	\centering
	\FloatBarrier
	\includegraphics[width=1\textwidth]{lasso_prediction_vs_lambda/lassoprediction_error_m300_d5_s50.png}

	\caption{This shows the bias-variance tradeoff as function of the regularization parameter, lambda, for Lasso regression method with a sample size of 300, a polynomial degree of 5 and standard deviation of 0.5}
	\label{fig:3}
\end{figure}
\FloatBarrier

\medskip

\subsection{Discussion}
From all the figures I will now consider prediction error, bias and variance for the various methods.

\subsubsection{OLS}
For the OLS I can see that the minimum prediction error is for a polynomial of degree 4 for all standard deviations 0, 0.1 and 0.5. IN all cases the value of the prediction error is about 0.004. The bias is smallest at 5 degrees in all cases. The variance is smallest for a degree 4 polynomial for standard deviations of 0 and 0.1. For standard deviation of 0.5 the smallest variance is at 3. For a degree 1 polynomial, we do have a low variance, but the bias is really high.
\medskip

Table \ref{Table:1} shows that there were only 5 significant coefficients, the other ones could possibly have been set to 0, but that was not implemented. In principle I could have found a best subset model here.

\subsubsection{Ridge}
For the ridge regression I can see a low prediction error for lambda being $10^{-4}$ and $10^{-10}$ for the 0 and 0.1 standard deviation cases, with a value of the prediction error of about 0.0035 and 0.002. The lowest value being for 0.1 standard deviation and lambda $10^{-4}$. For the 0.5 standard deviation case, the prediction error is the smallest at about 0.25 for lambda $10^{-4}$.

\subsubsection{Lasso}
For the lasso regression at standard deviation of 0 I can see a minimum prediction error of about 0.005 for lambda $10^{-7}$. For standard deviation 0.1 the prediction error is up to 0.015 and for standard deviation 0.5 it is all the way up to 0.25. This tells me that something might have gone wrong with the implementation of lasso.

\section{Conclusion}

The results show that the Ridge regression gives a better prediction error than the other two methods. That does not agree with the expected, since the Lasso method should have been better. I tried to figure out the reason why that happened, and I came up with the following explanation:

The formulation of Lasso is the same as for Ridge, but for the extra $\beta$ term in Ridge. It seems clear from the results that the Ridge method was to be preferred, but on further inspection, it seemed that the Lasso method may have been worse because the standardization method was not done correctly for Lasso. I suspect this was the reason for Lasso being worse than the Ridge regression.

I then implemented this standardization, and still found that the Lasso regression was inferior to the ridge regression, which I can not fully explain.

\section*{Improvements}
I would have liked to complete the data analysis, and also figured out why Ridge regression showed superior results compared to the Lasso Regression. For the OLS method I could have found a best subset model after having found the significant coefficients.

\section{Bibliography}
\begin{thebibliography}{99}
	\bibitem{projecttext}$\href{https://github.com/CompPhysics/MachineLearning/blob/master/doc/Projects/2018/Project1/pdf/Project1.pdf}{https://github.com/CompPhysics/MachineLearning/blob/master/doc/Projects/2018/Project1/pdf/Project1.pdf}$

	\bibitem{elementsstat}$\href{https://github.com/CompPhysics/MachineLearning/blob/master/doc/Textbooks/elementsstat.pdf}{https://github.com/CompPhysics/MachineLearning/blob/master/doc/Textbooks/elementsstat.pdf}$

	\bibitem{ridge}$\href{https://arxiv.org/pdf/1509.09169.pdf}{https://arxiv.org/pdf/1509.09169.pdf}$

	\bibitem{franke}$\href{http://www.dtic.mil/dtic/tr/fulltext/u2/a081688.pdf}{http://www.dtic.mil/dtic/tr/fulltext/u2/a081688.pdf}$

\end{thebibliography}


%%\documentclass[handout]{beamer}
%\documentclass[aspectratio=169,13pt]{beamer}

%\input{./preamble}

\subtitle{Fast Fourier Transform}

\date{}
\begin{document}

\begin{frame}
  \titlepage
\end{frame}

\section{Model Problem}

\begin{frame}{Sparse Linear Systems and Time-independent PDEs}

\begin{itemize}
\item The Poisson equation serves as a model problem for numerical methods:

\lgcond{
\begin{itemize}
\sitem the 2D Poisson problem and resulting Kronecker product linear system are a common benchmark,
\sitem this system has the form $\B T \otimes \B I + \B I \otimes \B T$ where $\B T$ is tridiagonal.
\end{itemize}
}

\item Dense, sparse direct, iterative, FFT, and Multigrid methods provide increasingly good complexity for the problem:

\lgcond{
\begin{itemize}
\sitem dense linear system solve costs $O(n^3)$ naively,
\sitem nested dissection with Cholesky has $O(n^{3/2})$ complexity and $O(n\log n)$ memory
\sitem Conjugate-Gradient gives $O(n^{3/2})$ complexity with $O(n)$ memory
\sitem FFT achieves $O(n \log n)$ cost and multigrid achieves $O(n)$.
\end{itemize}
}

\end{itemize}

\end{frame}

\section{Multigrid}

\begin{frame}{Multigrid}

\begin{itemize}
\item Multigrid employs a hierarchy of grids to accelerate iterative methods:

\lgcond{
\begin{itemize}
\item the residual equation $\B A \B{\hat{x}} = \B r$ on each fine grid, is approximately solved on the next coarser grid,
\mitem the equation is \coloremph{restricted} by projection matrix $\B P$, so that $\B P \B A \B P^T \B P \B{\hat{x}} = \B P \B r$
\mitem the interpolation operator (often given by $\B P^T$) is used to obtain an approximate $\B{\hat{x}}$ based on the coarse grid approximate solution,
\mitem at each level we perform some smoothing operations (e.g. Jacobi or Conjugate Gradient) before restriction and after interpolation,
\mitem at the coarsest level we typically solve directly.
\end{itemize}
}
\mdcond{}

\item The multigrid method works by resolving high-frequency error components on finer-grids and low-frequency error components on coarser grids:

\mdcond{
\begin{itemize}
\item smoothers are usually effective at reducing local error, but slow at resolving global (low-frequency) components of the error,
\mitem on coarser grids, the low frequency error may be resolved more quickly.
\end{itemize}
}

\end{itemize}

\end{frame}

\begin{frame}{Multigrid}

\begin{itemize}
\item Consider the Galerkin approximation with linear finite elements to the Poisson equation $u''=f(t)$ with boundary conditions $u(a)=u(b)=0$:
\[%\forall i \in \{1,n\} , \quad
\phi_i^{(h)}(t) = \begin{cases} (t-t_{i-1})/h &: t \in [t_{i-1},t_i] \\ (t_{i+1}-t)/h &: t\in [t_{i},t_{i+1}] \\ 0 &: \text{otherwise} \end{cases}\]
where $t_0=t_1=a$ and $t_{n+1}=t_n=b$.
\tlgcond{
The weak form with grid spacing of $h$ is
\begin{align*}
\int_a^b f(t) \phi^{(h)}_i(t) dt &=  - \sum_{j=1}^n x_j\int_a^b {\phi_j^{(h)}}'(t){\phi_i^{(h)}}'(t) dt.
\end{align*}
in multigrid, we define a coarse grid basis of $(n-1)/2$ functions, which are hat functions of twice the width,
\begin{align*}
\phi_i^{(2h)}(t) = \frac 12\phi_{2i-2}^{(h)}(t) + \phi_{2i-1}^{(h)}(t) + \frac 12\phi_{2i}^{(h)}(t) = \begin{cases} (t-t_{i-2})/2h &: t \in [t_{i-2},t_i] \\ (t_{i+2}-t)/2h &: t\in [t_{i},t_{i+2}] \\ 0 &: \text{otherwise} \end{cases}
\end{align*}
}
\lgcond{}
\end{itemize}
\end{frame}

\begin{frame}{Coarse Grid Matrix}

\begin{itemize}
\item Multigrid restricts the residual equation on the fine grid $\B A^{(h)}\B x = \B r^{(h)}$ to the coarse grid:
\tlgcond{
Let $\B{\phi}^{(2h)}=\begin{bmatrix} \phi_1^{(2h)} & \cdots &  \phi_{(n-1)/2}^{(2h)} \end{bmatrix}$ and
$\B{\phi}^{(h)}=\begin{bmatrix} \phi_1^{(h)} & \cdots &  \phi_{n}^{(h)} \end{bmatrix}$ and define \coloremph{restriction matrix} $\B P$ so that $\B{\phi}^{(2h)}=\B P \B{\phi}^{(h)}$, i.e.,
\[\B P = \frac 12\begin{bmatrix} 1 & 2 & 1 & & \\ & 1 & 2 & 1 & \\ & & \ddots & \ddots & \ddots \end{bmatrix} = \begin{bmatrix} {\B{p}^{(1)}} \\ {\B{p}^{(2)}} \\ \vdots \end{bmatrix}.\]
The coarse grid stiffness matrix is given by
\begin{align*}
a^{(2h)}_{ij} &= - \int_a^b {\phi_j^{(2h)}}'(t){\phi_i^{(2h)}}'(t) dt \\
&= - {\B{p}^{(i)}}\underbrace{\bigg(\int_a^b  {\B \phi^{(h)}}'(t) {{\B{\phi}^{(h)}}'}{}^T(t) dt\bigg)}_{-\B A^{(h)}}{\B p^{(j)}}^T, \\
\B A^{(2h)} &= \B P \B A^{(h)} \B P^T.
% &=  - \int_a^b ({\phi_{2i-2}^{(h)}}'(t) + 2{\phi_{2i-1}^{(h)}}'(t) + {\phi_{2i}^{(h)}(t)}'(t))
% ({\phi_{2j-2}^{(h)}}'(t) + 2{\phi_{2j-1}^{(h)}}'(t) + {\phi_{2j}^{(h)}(t)}'(t))dt
\end{align*}

%\begin{align*}
%\int_a^b f(t) \phi^{(h)}_i(t) dt &=  - \sum_i x_i\int_a^b {\phi_j^{(h)}}'(t){\phi_i^{(h)}}'(t) dt.
%\end{align*}
%
%To do so, we need to obtain a suitable basis transformation from $\{\phi^{(h)}_i\}$ to $\{\phi^{(2h)}_i\}$.
%We would like a correct answer at each $t$ and can obtain a system of equations by integrating w.r.t. test functions $\phi^{(h)}_j(h)$,
%\begin{align*}
%\sum_{i=1}^{n/2} x^{(2h)}_i \int_a^b\phi_i^{(2h)}(t)  \phi_j^{(h)}(t)dt = \sum_{i=1}^n x^{(h)}_i\int_a^b \phi_i^{(h)}(t) \phi_j^{(h)}(t)dt
%\end{align*}
}
\lgcond{}
\end{itemize}
\end{frame}

\begin{frame}{Restricting the Residual Equation}

\begin{itemize}
\item Given the fine-grid residual $\B r^{(h)}$, we seek to use the coarse grid to approximate $\B x^{(h)}$ so that $\B A \B x^{(h)} \approx \B r^{(h)}$

\lgcond{
\begin{itemize}
\item
%Let $\B t = \begin{bmatrix} a+h & a+2h & \cdots & b-h \end{bmatrix}^T$.
Given a function in the coarse grid basis, $u^{(2h)} = {\B x^{(2h)}}^T\B{\phi}^{(2h)}$, we can express it in the fine-grid basis via
\begin{align*}
u^{(2h)} &= {\B x^{(2h)}}^T \underbrace{\B P \B{\phi}^{(h)}}_{\B{\phi}^{(2h)}} = \underbrace{{\B x^{(2h)}}^T \B P}_{{\B x^{(h)}}^T} \B{\phi}^{(h)}.
\end{align*}
\item
Consequently, the solution to the restricted residual equation $\B A^{(2h)}\B x^{(2h)} = \B r^{(2h)}$ will lead to an approximate residual equation solution on the fine grid
with $\B x^{(h)} =\B P^T\B x^{(2h)}$.
\mitem
Noting this, we derive the form of the coarse grid residual, %form the coarse grid problem via
\begin{align*}
\B r^{(2h)} &= \B A^{(2h)}\B x^{(2h)} \\
&= \B P\B A^{(h)}\B P^T \B x^{(2h)}  = \B P\B A^{(h)}\B x^{(h)}  \\
&= \B P\B r^{(h)}.
\end{align*}
\end{itemize}
%\[w_i=u^{(2h)}(t_i)={\B x^{(2h)}}^T\B{\phi}^{(2h)}(t_i)={\B x^{(2h)}}^T\B P \underbrace{\B{\phi}^{(h)}(t_i)}_{\B e_i}\]
% so $\B w = \B P^T{\B x^{(2h)}}$.
%Note that for a function in the fine grid basis, $u^{(h)} = {\B x^{(h)}}^T\B{\phi}^{(h)}$, we have $u^{(2h)}(t_i) = x^{(h)}_i$.
%Consequently, we can express the coarse grid function in the fine grid basis via
%\begin{align*}
%u^{(h)}(t) &= \underbrace{{\B x^{(2h)}}^T \B P}_{\B x^{(h)}^T} \B{\phi}^{(h)}
%\end{align*}
}
\lgcond{}
\end{itemize}

\end{frame}

%\begin{frame}{Multigrid Restriction}
%
%\begin{itemize}
%\item Multigrid restricts the residual equation on the fine grid to the coarse grid:
%\tlgcond{
%To do so, we need to obtain a suitable basis transformation from $\{\phi^{(h)}_i\}$ to $\{\phi^{(2h)}_i\}$,
%\begin{align*}
%\B V(\{\phi_{2i-1}^{(2h)}\}_{i=1}^{(n-1)/2},\{t_i\}_{i=1}^n) \B x^{(2h)} \cong  \B V(\{\phi_i\}_{i=1}^n,\{t_i\}_{i=1}^n)  \B x^{(h)}
%\end{align*}
%For linear finite elements $\B V(\{\phi_i^{(h)}\}_{i=1}^n,\{t_i\}_{i=1}^n)= \B I$ so we obtain,
%\begin{align*}
% \B V(\{\phi_{2i-1}^{(2h)}\}_{i=1}^{(n-1)/2},\{t_i\}_{i=1}^n)\B x^{(2h)} =  \B x^{(h)}.
%\end{align*}
%Multigrid requires values of the residual $\{r(t_{2i-1}\}_{i=1}^{(n-1)/2}$ on the coarse grid, whose coefficients $\B x^{(2h)}$ are then given by
%\begin{align*}
%\B r^{(2h)} = \underbrace{\B V(\{\phi_{i}^{(h)}\}_{i=1}^{n},\{t_{2i-1}\}_{i=1}^{(n-1)/2})}_{\B I}\B x^{(h)} &=  \B V(\{\phi_{2i-1}^{(2h)}\}_{i=1}^{(n-1)/2},\{t_i\}_{i=1}^n)\B x^{(2h)}.
%\end{align*}
%}
%\end{itemize}
%\end{frame}

\section{Fast Fourier Transform}

\begin{frame}{Discrete Fourier Transform}

\begin{itemize}
\item The solutions to hyperbolic PDEs like Poisson are wave-like and take on simple representations in the frequency basis, both for continuous and discretized equations. We define the \coloremph{discrete Fourier transform} using
$$
\omega_{(n)} = \cos(2\pi/n) - i\sin(2\pi/n) = e^{-2\pi i/n}.
$$
\lgcond{
The DFT matrix $\B F\in\mathbb{R}^{n\times n}$ is given by
$f_{ij}=\omega_{(n)}^{ij}$,
$$
\B{F} =
\left[ \begin{matrix}
1 & 1 & 1 & 1 \cr
1 & \omega_{(4)}^{1} & \omega_{(4)}^{2} & \omega_{(4)}^{3} \cr
1 & \omega_{(4)}^{2} & \omega_{(4)}^{4} & \omega_{(4)}^{6} \cr
1 & \omega_{(4)}^{3} & \omega_{(4)}^{6} & \omega_{(4)}^{9} \end{matrix}
\right]$$

\begin{itemize}
\item it is complex and symmetric (not Hermitian),
\item it is unitary modulo scaling $\B F^{*} = n\B F^{-1}$.
\end{itemize}
The discrete Fourier transform of vector $\B v$ is $\B F \B v$.
}
\lgcond{}

\end{itemize}

\end{frame}

\begin{frame}{Fast Fourier Transform (FFT)}

\begin{itemize}
\item Consider $\B b=\B{F}\B a$, we have
\[\forall j\in[0,n-1] \quad b_j=\sum_{k=0}^{n-1}\omega_{(n)}^{jk}a_k,\]
the FFT computes this recursively via 2 FFTs of dimension $n/2$, using $\omega_{(n/2)}=\omega_{(n)}^2$,
\lgcond{
\begin{align*}
%\forall j\in[0,n-1] \quad
b_j&=\sum_{k=0}^{n/2-1}\omega_{(n)}^{j(2k)}a_{2k}+\sum_{k=0}^{n/2-1}\omega_{(n)}^{j(2k+1)}a_{2k+1} \\
&=\sum_{k=0}^{n/2-1}\omega_{(n/2)}^{jk}a_{2k}+\omega_{(n)}^j\sum_{k=0}^{n/2-1}\omega_{(n/2)}^{jk}a_{2k+1}
\end{align*}
}
\lgcond{}
%, so
%\[\forall j\in[1,n/2] \quad b(2j)=\sum_{k=0}^{n/2-1}\omega_{(n/2)}^{jk}a_k\]
\end{itemize}
\end{frame}

\begin{frame}{Fast Fourier Transform Derivation}

\begin{itemize}
\item The FFT leverages similarity between the first and second half of the output,
\begin{align*}
b_j&=\underbrace{\sum_{k=0}^{n/2-1}\omega_{(n/2)}^{jk}a_{2k}}_{u_j}+\omega_{(n)}^j\underbrace{\sum_{k=0}^{n/2-1}\omega_{(n/2)}^{jk}a_{2k+1}}_{v_j}
\end{align*}
corresponds closely to the entry shifted by $n/2$,
\begin{align*}
 b_{j+n/2}&=\sum_{k=0}^{n/2-1}\omega_{(n/2)}^{(j+n/2)k}a_{2k}+\omega_{(n)}^{j+n/2}\sum_{k=0}^{n/2-1}\omega_{(n/2)}^{(j+n/2)k}a_{2k+1}
\end{align*}
\lgcond{
Now $\omega_{(n/2)}^{(j+n/2)k}=\omega_{(n/2)}^{jk}$ since $(\omega_{(n/2)}^{n/2})^k=1^k=1$ and using $\omega_{(n)}^{n/2}=-1$,
%$\forall j\in[0,n/2-1]$
\begin{align*}
b_{j+n/2}
&=\underbrace{\sum_{k=0}^{n/2-1}\omega_{(n/2)}^{jk}a_{2k}}_{u_j}-\omega_{(n)}^{j}\underbrace{\sum_{k=0}^{n/2-1}\omega_{(n/2)}^{jk}a_{2k+1}}_{v_j}
\end{align*}
}
\end{itemize}
\end{frame}

\begin{frame}{FFT Algorithm Summary}

%Each of these two summation can be done recursively with an FFT %of dimension $n/2$
% which can then be combined to compute the upper and lower halves of $b$
\begin{itemize}
\item Let vectors $\B u$ and $\B v$ be two recursive FFTs, $\forall j\in[0,n/2-1]$
\begin{align*}
u_j&=\sum_{k=0}^{n/2-1}\omega_{(n/2)}^{jk}a_{2k}, \quad
v_j=\sum_{k=0}^{n/2-1}\omega_{(n/2)}^{jk}a_{2k+1}
\end{align*}
\lgcond{
\begin{itemize}
%\item We can make these two recursive calls simultaneously and without any work
\item Given $\B u$ and $\B v$ scale using "twiddle factors" $z_j=\omega_{(n)}^j\cdot v_j$
\item Then it suffices to combine the vectors as follows
\(\B b=\begin{bmatrix} \B u+\B z \\ \B u-\B z\end{bmatrix}\)
%\item This recombination is an FFT of dimension 2
%\[\B b=\begin{bmatrix} \B{b}_1 \\ \B{b}_2 \end{bmatrix} = \vc{\begin{bmatrix} \B b_1 & \B b_2\end{bmatrix}} =
%\vcop\bigg(\begin{bmatrix} \B u & \B z \end{bmatrix} \underbrace{\begin{bmatrix} 1 & 1 \\ 1 & -1 \end{bmatrix}}_{\B{F_4}[0:2,0:2]}\bigg) %=
%%\vc{\begin{bmatrix} \B u & \B z \end{bmatrix} \B{F_2}}
%\]
%\vspace{-.2in}
%\item Radix-$r$ algorithm for any $\B{A}\in\mathbb{R}^{n/r\times r}$
%\[
%%\vc{\B{B}}=
%\B{F}_{n}\vc{\B{A}} = \vc{\big([\B{F}_{n}[0:r,0:r] \odot (\B{F}_r \B{A})]\B{F}_{n/r})^T}\]
%%\ \text{ if and only if } \ \B{B} = \B{F}_{n/r} \B{A} \B{F}_r\]
\end{itemize}
}
\item The FFT has $O(n \log n)$ cost complexity:

\lgcond{
There are two recursive calls of dimension $n/2$ and $O(n)$ work for application to twiddle factors and final summation, thus
$$T(n)=2T(n)+O(n)=O(n\log n).$$
}
\end{itemize}
\end{frame}

\begin{frame}{Applications of the FFT}
\begin{itemize}
\item We can rapidly multiply degree $n$ polynomials by considering their values $\omega_{(2n-1)}^i$ for $i\in\{0,\ldots,2n-1\}$
\lgcond{
\[p_c(\omega_{(2n-1)}^i)=p_a(\omega_{(2n-1)}^i)p_b(\omega_{(2n-1)}^i)\]
\begin{itemize}
\item The product of coefficients of $p_a,p_b$ with Vandermonde matrix $v_{ij}=(\omega_{(2n-1)}^{i})^{j}$, which is the DFT matrix, gives values of polynomials at $2n-1$ nodes.
\mitem Interpolation to compute coefficients of $p_c$ from the products of values of $p_a$ and $p_b$ at those nodes is multiplication by the inverted DFT matrix and is exact since $p_c$ is degree $2n-2$.
\end{itemize}
}

\item More generally the DFT can be used to solve any Toeplitz linear system (convolution):
\lgcond{
\begin{itemize}
\item
A standard convolution has the form,
\(\forall k\in [0,n-1] \quad c_k = \sum_{j=0}^k a_jb_{k-j}.\)
\mitem Convolution is equivalent to multiplications of polynomials with degree $n/2-1$ and coefficients  $\Vec a$ and $\Vec b$, where
%\[p_a(x) = \sum_{k=0}^{n/2-1} a_kx^k,\quad
%p_b(x) = \sum_{k=0}^{n/2-1} a_kx^k\]
the convolution computes the coefficients $\Vec c$ of the product of the two polynomials.
%\[p_c(x) = p_a(x)p_b(x) = \sum_{k=0}^{n-1} c_kx^k\]
\end{itemize}
}
\end{itemize}
\end{frame}

\begin{frame}{Convolution via DFT}

\begin{itemize}
\item The Fourier transform method for computing a convolution is given by
\begin{align*}
c_k &= %\sum_s D^{-1}_{ks} \Big(\sum_j D_{sj} a_j\Big)\Big(\sum_t D_{st} b_t\Big)
  \frac 1n \sum_s \omega^{-ks}_{(n)} \Big(\sum_j \omega^{sj}_{(n)} a_j\Big)\Big(\sum_t \omega_{(n)}^{st} b_t\Big)
\end{align*}
\lgcond{
\begin{itemize}
\item Rearrange the order of the summations to see what happens to every product of $a$ and $b$
\begin{align*}
c_k &%= \frac 1n \sum_s\sum_j\sum_t \omega^{-ks}_{(n)} \omega_{(n)}^{sj}\omega^{st}_{(n)} a_j b_t
 = \frac 1n \sum_s\sum_j\sum_t \omega^{(j+t-k)s}_{(n)} a_j b_t
\end{align*}
\item For any $u=j+t-k\neq 0$, we observe $\sum_s(\omega_{(n)}^u)^s=0$ %, as for $\Mat F\Mat F^{-1}$
\mitem When $j+t-k=0$ the products $\omega^{(s+t-j)k}_{(n)}=1$, so there are $n$ nonzero terms $a_jb_{k-j}$ in the summation
\end{itemize}
}
\lgcond{}
\end{itemize}
\end{frame}

\begin{frame}{Solving Numerical PDEs with the FFT}

\urcornerlinkdemo{12-fft}{Fast Fourier Transform}

\begin{itemize}
\item 1D finite-difference schemes on a regular grid correspond to convolutions:

\mdcond{
1D model problem is simply convolution with vector $[1, -2, 1]$.
}

\item For the 1D Poisson model problem, the eigenvectors of $\B T$ corresponds to the imaginary part of a minor of a $2(n+1)$-dimensional DFT matrix:

\mdcond{
\begin{itemize}
\item
In particular, $\B T =\B X \B D \B X^{-1}$ where $x_{ij}$ is the imaginary part of $f_{i+1,j+1}$ with $\B X\in \mathbb{R}^{n\times n}$ and $\B F \in \mathbb{R}^{2(n+1)\times 2(n+1)}$.
\mitem
Consequently, $\B T$ can be diagonalized and the overall system solved by FFT with $O(n\log n)$ cost.
\end{itemize}
}

\item Multidimensional Poisson can be handled with multidimensional FFT:

\mdcond{
For example 2D FFT (1D FFT of each row then 1D FFT of each column) suffices to solve the 2D Poisson problem.
}

\end{itemize}

\end{frame}

%\end{document}


% !TEX options=--shell-escape
\documentclass [12pt]{article}

\usepackage {amsmath}
\usepackage {amsthm}
\usepackage {amssymb}
\usepackage {graphicx}
\usepackage {float}
\usepackage {multirow}
\usepackage {xcolor}
\usepackage {algorithmic}
\usepackage [ruled,vlined,commentsnumbered,titlenotnumbered]{algorithm2e} \usepackage {array}
\usepackage {booktabs}
\usepackage {url}
\usepackage {parskip}
\usepackage [margin=1in]{geometry}
\usepackage [T1]{fontenc}
\usepackage {cmbright}
\usepackage [many]{tcolorbox}
\usepackage [colorlinks = true,
            linkcolor = blue,
            urlcolor  = blue,
            citecolor = blue,
            anchorcolor = blue]{hyperref}
\usepackage {enumitem}
\usepackage {xparse}
\usepackage {verbatim}
\usepackage{listings}
\usepackage{xcolor}
\usepackage{csquotes}
\usepackage[cache=false]{minted}
\usepackage{mdframed}
\newtheorem{theorem}{Theorem}

\BeforeBeginEnvironment{minted}{\begin{mdframed}}
\AfterEndEnvironment{minted}{\end{mdframed}}

\DeclareTColorBox {Solution}{}{breakable, title={Solution}}
\DeclareTColorBox {Solution*}{}{breakable, title={Solution (provided)}}
\DeclareTColorBox {Instruction}{}{boxrule=0pt, boxsep=0pt, left=0.5em, right=0.5em, top=0.5em, bottom=0.5em, arc=0pt, toprule=1pt, bottomrule=1pt}
\DeclareDocumentCommand {\Expecting }{+m}{\textbf {[We are expecting:} #1\textbf {]}}
\DeclareDocumentCommand {\Points }{m}{\textbf {(#1 pt.)}}

\begin {document}

{\LARGE \textbf {COMP 285 (NC A\&T, Spr `22)}\hfill \textbf {Homework 3} }
\vspace {1em}
\begin {Instruction}

\paragraph {Due.} Tuesday, February 9th, 2022 @ 11:59 PM!
\end {Instruction}

\vspace {1em}
\begin {Instruction} \paragraph {Homework Expectations:} Please see \href{https://www.comp285.ml/homework/#general-homework-information}{Homework}.
\end {Instruction}

\vspace {1em}
\begin {Instruction}

\paragraph {Exercises} The following questions are exercises. We encourage you to work with a group and discuss solutions to make sure you understand the material.

\paragraph {Points} This assignment is graded out of 100 points. However, you can get up to 120 points if you complete everything. These are not bonus points, but rather points to help make-up any parts you miss.

\end {Instruction}

\begin{centering}
\section*{Fun with Divide and Conquer, Sorting, and Medians}
\end{centering}

\begin{Instruction}

\paragraph{Written Problems} The following questions are to be submitted in written/typed form to gradescope.

\end{Instruction}

\section{Getting to Know You \Points{10}}

As part of this homework assignment, I'd like to get to know each of you.

Do the following:

\begin{itemize}
    \item \Points{5} Schedule a 10 minute conversation with me in the available slots \href{https://calendly.com/comp285-prof/get-to-know-you?month=2022-02}{here}.
    \item \Points{5} When you're done with the homework, complete this \href{https://forms.gle/KpC8RJ3gvkPgQPYY7}{form}.
\end{itemize}

\Expecting{You should have scheduled a meeting with me for an available slot and submit the form. All of this is tracked and this is how we'll confirm to give you credit for this question.}

\pagebreak
\section{Exercise: A faster MergeSort? \Points{10}}
Inspired by all of our discussion on recursion and sorting, Miah, Ryan, and Jordan get together to try and figure out a faster\footnote{We will not cover in class, but it has been proven that there is no deterministic algorithm faster than $\theta(n \log n)$ for sorting numbers using comparisons.} way to sort! After many attempts, they write down the following sorting function:

\vspace{2em}
\begin{minted}[linenos=true]{cpp}
// MergeSort algorithm!
std::vector<int> mergeSortN(const std::vector<int>& input) {
    const std::size_t n = input.size();
    if (n <= 1) {
        return input;
    }
    std::vector<std::vector<int>> sortedSubvectors = {};
    for (int i = 0; i < n; i++) {
        std::vector<int> temp = mergeSortN(
            {input.begin() + i, input.begin() + i + 1}
        );
        sortedSubvectors.push_back(temp);
    }
    std::vector<int> sortedA = {};
    for (const std::vector<int> subVector : sortedSubvectors) {
        sortedA = merge(sortedA, subVector);
    }
    return sortedA;
}
\end{minted}

\subsection{What is this doing!? \Points{5} }
In your own words, describe that the algorithm above is doing. This should not take more than a few sentences, but should highlight how this is different from the normal implementation of \texttt{MergeSort}.

\Expecting{A few sentences describing, in plain English, what the algorithm above does.}

\subsection{How fast is it?! \Points{5}}
Jordan claims this version of \texttt{MergeSort} is much faster than any versions we've seen before. This is his argument:

\begin{displayquote}
    This modified \texttt{MergeSort} splits the array into $n$ subproblems of size $O(1)$ immediately. We therefore don’t waste time with the `$log(n)$ levels’ worth of splitting. Additionally, in this modified sort, we’re calling \texttt{Merge} on a bunch of subvectors of size $1$. Each merge would therefore take time

    $$
        O(\text{size of subvector A}) + O(\text{size of subvector B}) = O(1) + O(1) = O(1)
    $$
    That's contant time per merge! Yay! We therefore have an algorithm that sorts the input array in $O(n)$ time - $n$ merges of $O(1)$ each!
\end{displayquote}

Sadly, Jordan's analysis is \textbf{wrong}. It was a valiant try, though. Let's help Miah, Ryan, and Jordan out! What mistake are they making in their analysis above, and what is the true runtime of their modified \texttt{MergeSort}.

\Expecting{An explanation (one sentence) that summarizes the mistake in the above reasoning and the correct runtime of \texttt{mergeSortN} along with an explanation (another sentence).}

\pagebreak
\section{Interview Practice: Finding Matching Elements \Points{10}}
Caleb is working on a ``search'' algorithm at Foogle. He is given an array \texttt{input} of $n$ integers that is sorted in ascending order and contains only unique elements. That is to say, if his array is
$$
a = [a_1, a_2, \cdots, a_{n-1}, a_n]
$$

then $a_1 < a_2 < \cdots < a_{n-1} < a_n$.

His project is to search the above array and find the element such that its index is equal to its value, or returns that such $i$ does not exist. That is to say, the algorithm finds $i$ such that $i = a_i$.

\subsection{A first proposal. \Points{5}}
\label{subsec:first_proposal}
Caleb proposes the following pseudo-code for an algorithm that solves the problem described above.

\begin{verbatim}
Algorithm:
    Input: A vector "input" sorted in ascending order with no duplicates.
    Outputs: The index i such that the i-th index contains the value i.

    n = len(input)
    for i = 0 to i = n - 1
        if (i == input[i]):
            return i

    throw "No such index exists"
\end{verbatim}

Describe in plain English how Caleb's algorithm works, and provide the big-Oh running time.

\Expecting{A few sentences describing how the algorithm Caleb wrote works in plain English, as well as $O(\cdots)$ where $\cdots$ is filled in with the proper running time.}

\subsection{Your Proposal. \Points{5}}
\label{subsec:second_proposal}
After looking at Caleb's proposed algorithm, you realize that he's not taking advantage of the fact that the input array is sorted. Give an $O(\log n)$ algorithm that solves the same problem (eg, it finds the index $i$ such that $i = a_i$ or returns that such $i$ does not exist). If there are multiple such $i$s, your algorithm can return any of them \footnote{Consider a divide-and-conquer approach, similar to binary search! If you'd like to refresh your memory on binary search, see this \href{https://www.youtube.com/watch?v=P3YID7liBug}{video}.}.

\Expecting{Pseudocode for you algorithm, similar to what Caleb wrote in Part \ref{subsec:first_proposal}, as well as a plain English description of what your algorithm is doing and why it is correct. You \textbf{do not} need to proof correcness.}

\pagebreak
\section{Interview Practice: Finding Medians \Points{30}}
You are given two arrays, each of length $n$. Your goal is to find the median of all elements of the two arrays.

\subsection{Unsorted Arrays \Points{10}}
If the arrays \textbf{are not sorted}, give a $\Theta(n \log n)$ algorithm that returns the median of the combined arrays.

\Expecting {A short English description of your approach, the pseudocode (similar to Part \ref{subsec:second_proposal}), and an explanation of your algorithm's runtime.}.

\subsection{Unsorted Arrays But Faster \Points{10}}

If the arrays \textbf{are not sorted}, give an $\Theta(n)$ algorithm that returns the median of the combined arrays\footnote{Hint: You can use `Select(A, k)' directly, without explanation, since this was given in lecture}.

\Expecting {A short English description of your approach, the pseudocode (similar to Part \ref{subsec:second_proposal}), and an explanation of your algorithm's runtime.}.

\subsection{Sorted Arrays \Points{10}}

If the arrays \textbf{are sorted}, give an $O(\log n)$ algorithm that returns the median of the combined arrays. Here are a few hints to help you. Try to look at these only after you've given the problem a bit of thought on your own.

\begin{enumerate}[label=Hint \arabic*]
    \item You probably want to use divide-and-conquer.
    \item Calculating the median of a single sorted array can be done in $O(1)$ time (how?).
    \item If the biggest element in one array (eg, the last element) is smaller than the smallest element in another array (eg, the first element), you can also compute the median $O(1)$ time (how?).
    \item If you look at the medians of the two arrays, how does that help you split the problem into parts? What do you know if the median of the two arrays are equal? What if one is bigger than the other?
\end{enumerate}

\Expecting {A short English description of your approach, the pseudocode (similar to Part \ref{subsec:second_proposal}), and an explanation of your algorithm's runtime.}.

\pagebreak
\section {Exercise: Randomized Algorithms \Points {10}}
\label{sec:last}

In this exercise, we'll explore different types of randomized algorithms. We say that a randomized algorithm is a \textbf {Las Vegas algorithm} if it is always correct (that is, it returns the right answer with probability 1), but the running time is a random variable. We say that a randomized algorithm is a \textbf {Monte Carlo algorithm} if there is some probability that it is incorrect. For example, QuickSort (with a random pivot) is a Las Vegas algorithm, since it always returns a sorted array, but it might be slow if we get very unlucky.

We will visit the Majority Element problem to get more insight on randomized algorithms. The Majority Element problem is to find the element in a collection of elements that occurs at least half of the time. See Algorithm \ref{alg:isMajority} for how we would check if an element is the majority.

We will assume that such an element always exists.

\vspace{2em}
\begin{minted}[linenos=true]{cpp}
// Returns 2, because it occurs 4 times which is greater
// than 6/2 = 3.
findMajorityElement({2, 2, 2, 2, 3, 5});
// Returns 10, because it occurs 3 times which is greater
// than 5/2 = 2.5
findMajorityElement({10, 10, 10, 2, 4});
// Returns 4 because it occurs 3 times, which is greater
// than 4/2 = 2.
findMajorityElement({4, 4, 2, 4})
\end{minted}

\vspace{2em}
\begin{center}
\begin{tabular}{|c|p{3cm}|p{2cm}|p{2cm}|p{4cm}|}
\hline
Algorithm & Monte Carlo or Las Vegas? & Expected running time & Worst-case running time & Probability of returning a majority element \\
\hline
\textbf {Algorithm 1} & & & & \\
\hline
\textbf {Algorithm 2} & & & & \\
\hline
\textbf {Algorithm 3} & & & & \\
\hline
\end{tabular}
\end{center}

\Expecting {Your filled in-table. You may use asymptotic notation for the running times.} \newpage

\begin{algorithm}[H]
    \KwIn {A population $P$ of $n$ elements}
    \While {true} {
        Choose a random $p \in P$\;
        \If {\textsc {isMajority}($P$, $p$)}{
            \Return { $p$}\;
        }
    }
    \caption {\textsc {findMajorityElement1}}
\end{algorithm}

\begin{algorithm}[H]
    \KwIn {A population $P$ of $n$ elements}
    \For {100 iterations} {
        Choose a random $p \in P$\;
        \If {\textsc {isMajority}($P$, $p$)}{
            \Return { $p$ }\;
        }
    }
    \Return { $P[0]$\;}
    \caption {\textsc {findMajorityElement2}}
\end{algorithm}

\begin{algorithm}[H]
    \KwIn {A population $P$ of $n$ elements}
    Put the elements in $P$ in a random order.\;
    \tcc{Assume it takes time $\Theta (n)$ to put the $n$ elements in a random order}
    \For { $p \in P$ } {
        \If {\textsc {isMajority}($P$, $p$)}{
            \Return { $p$ }\;
        }
    }
    \caption {\textsc {findMajorityElement3}}
\end{algorithm}

\begin{algorithm}[H]
    \KwIn {A population $P$ of $n$ elements and a element $p \in P$}
    \KwOut {True if $p$ is a member of a majority species}
    count $\gets $ 0\;
    \For { $q \in P$ } {
        \If { $p = q$ }{
            count ++\;
        }
    }
    \If { count $> n/2$ }{
        \Return {True}\;
    } \Else {
        \Return {False}\;
    }
    \caption {\textsc {isMajority}}
    \label{alg:isMajority}
\end{algorithm}

\pagebreak
\begin{Instruction}

\paragraph{Coding Problems} The following questions are to be submitted as a ".zip" file on Gradescope.

\end{Instruction}

\section{Coding \Points{50}}
\Points{50} After completing the written portion of the assignment, you should submit to \href{https://www.gradescope.com/courses/350304}{Gradescope}.

You can get your starter code for the coding portion \href{https://replit.com/team/COMP285/HW3-Code}{here}.

Note that the starter code also include a few test cases you can run on repl.it. However, the full test suite is the one run on Gradescope.

Please reference the \texttt{README.md} included in your starter code for detailed instructions.

\section*{Submitting the Assignment}

This assignment is a combination of written and programming questions. Both portions of the assignment should be submitted through \href{https://www.gradescope.com/courses/350304}{Gradescope}.

The "Homework 3: Fun with Divide and Conquer, Sorting, and Medians" assignment is the written portion, for which you should submit a \textbf{typed} response to the non-coding questions (questions 1-\ref{sec:last}). Each response should clearly be marked with its corresponding number. You are free to use the provided templates, print the questions and write your answers, or to simply type your responses on a blank document (whatever works for you).

The "Homework 3: Coding" is the programming portion of the assignment. For this portion, download the ".zip" file from replit and upload this ".zip" file as your answer to \href{https://www.gradescope.com/courses/350304}{Gradescope}. You can upload the assignment as many times as you want.

\end {document}