\chapter{Winding Surface Optimization with Adjoint \texttt{REGCOIL}}
\label{ch:adjoint}

In this section we describe winding surface optimization using the adjoint \texttt{REGCOIL} method.

\section{Overview}

If an adjoint equation is solved in \texttt{REGCOIL}, (\parlink{sensitivity\_option} $> 1$), then analytic derivatives of $\chi^2_B$, $\norm{\bm{K}}_2$, or $K_{\max}$ are computed with respect to the Fourier coefficients defining the winding surface using the adjoint method. These derivatives are used for a gradient-based optimization method to find a winding surface which minimizes a user-defined objective function. The target plasma surface is held fixed during the optimization. The user has the option of holding a target function fixed during the optimization (such as $\chi^2_B$, $\norm{\bm{K}}_2$, or $K_{\max}$) to fix the regularization parameter $\lambda$. There are also options to impose constraints, such as on the minimum coil-plasma distance. The \texttt{NESCIN} convention is used, and the result of the optimization is a \texttt{NESCIN} file with the optimal winding surface Fourier coefficients.

\section{Optimization scripts}
Additional parameters must be included in the \texttt{REGCOIL} input file outside the \texttt{regcoil\_nml} Fortran namelist. The parameters in this namelist are read by either the \texttt{scipy\_optimize} or \texttt{nlopt\_optimize} python scripts, found in the \texttt{windingSurfaceOptimization} directory. These scripts are called with the \texttt{REGCOIL} input file as an argument. The \texttt{REGCOIL} input file in addition to any geometry files (\texttt{nescin\_filename}, \texttt{efit\_filename}, \texttt{bnorm\_filename}, \texttt{shape\_filename\_plasma}) must be located in the directory from which these scripts are called.

The \texttt{nlopt} package must be installed in order to call \texttt{nlopt\_optimize}. See the \href{https://nlopt.readthedocs.io/en/latest/#download-and-installation}{nlopt documentation} for installation instructions. In general \texttt{nlopt\_optimize} should be used if one wants to perform constrained optimization. Once \texttt{nlopt} is installed, ensure that your \texttt{\$PYTHONPATH} includes the directory containing \texttt{libnlopt.so}. The directory where this is located is specified by \texttt{libdir} in the file \texttt{libnlopt.la}. At this time \texttt{nlopt\_optimize} has been used with \texttt{nlopt} 2.4.2. The \texttt{scipy\_optimze} script utilizes the \texttt{scipy.optimize} package. Details on installation of \texttt{scipy} can be found \href{https://scipy.org/install.html}{here}. The parameters relevant to each of these scripts are detailed below.

Each time that \texttt{REGCOIL} is called from one of the scripts, an \texttt{eval\_} directory will be created. The script will print the objective function and constraint functions diagnostics with each evaluation to standard output. The \texttt{compareRegcoilSurface} script (found in \texttt{regcoil/coilOptimizationTools}) can be called on two \texttt{REGCOIL} output files to compare the winding surfaces at 2 evaluations.
\begin{verbatim}
compareRegcoilSurface eval_0/regcoil_out.w7x.nc eval_10/regcoil_out.w7x.nc
\end{verbatim}
Several example input files can be found in the \texttt{adjointRegcoilExamples} directory.

While \texttt{nlopt\_optimize} and \texttt{scipy\_optimize} are serial optimizers, the gradient computation in \texttt{REGCOIL} is performed in parallel with OpenMP. Multithreading is controlled with the \texttt{OMP\_NUM\_THREADS} environment variable.

To run \texttt{scipy\_optimize} or \texttt{nlopt\_optimize} from any directory, add the \\ \texttt{regcoil/coilOptimizationTools} directory to your \texttt{\$PATH} and to your \texttt{\$PYTHONPATH}, and add the \texttt{regcoil} directory to your \texttt{\$PATH}.

\section{Required \texttt{\&regcoil\_nml} namelist parameters}
The following items in the \texttt{\&regcoil} namelist should be used when running adjoint \texttt{REGCOIL}.
\begin{itemize}
\item \parlink{geometry\_option\_coil} = 3 or 4
    \begin{itemize}
    \item A \parlink{nescin\_filename} must be specified. It is assumed that the $m=0$ mode only includes $n\geq0$ modes.
    \end{itemize}
\item \parlink{sensitivity\_option} $>1$ denotes an adjoint solve must be performed. If $\chi^2_B$, $\norm{\bm{K}}_2$ or $K_{\text{max}}$ are included in the objective function, \parlink{sensitivity\_option} should be $>2$. If finite difference derivatives are used by setting \parlink{grad\_option} = 1 or if $\chi^2_B$, $\norm{\bm{K}}_2$ or $K_{\text{max}}$ are not included in the objective function, \parlink{sensitivity\_option} can be set to 1.
\item \parlink{nmax\_sensitivity} should be set to the largest value of $n$ that should be varied in the \texttt{NESCIN} file. This matters if \parlink{sensitivity\_option} $>1$.
\item \parlink{nmax\_sensitivity} should be set to the largest value of $m$ that should be varied in the \texttt{NESCIN} file. This matters if \parlink{sensitivity\_option} $>1$.
\item \parlink{sensitivity\_symmetry\_option} should be set to reflect the symmetry desired in the optimized winding surface.
\item If the coil-plasma distance is to be included in the objective function or constraints, then \parlink{coil\_plasma\_dist\_lse\_p} should be set to the desired value for the log-sum-exponent approximation. A value in the range $10^{2}$ - $10^4$ is typically sufficient. At very large values the function has very steep gradients, while at small values it does not approximate the minimum function well.
\item If the gradients of $\chi^2_B$, $\norm{\bm{K}}_2$ or $K_{\text{max}}$  are to be computed at fixed target function (specified by \parlink{target\_option}) (rather than at fixed $\lambda$), \parlink{fixed\_norm\_sensitivity\_option} should be $>1$. The following parameters matter when \parlink{fixed\_norm\_sensitivity\_option} $>1$.
	\begin{itemize}
	\item \parlink{target\_option} must be \texttt{"max\_K\_lse"}, \texttt{"lp\_norm\_K"}, or \texttt{"chi2\_B"}
	\item \parlink{target\_option\_p} is a parameter in the norm defined by \parlink{target\_option}
	\end{itemize}
\end{itemize}

\section{Coil-winding Surface Optimization Parameters}

\myhrule

The parameters related to winding surface optimization are defined in the input file outside the \texttt{regcoil\_nml} namelist.

\myhrule

\section{Winding Surface Optimization}

The following objective function is used when \texttt{nlopt\_optimize} or \texttt{scipy\_optimize} is called.
\begin{multline}
f = \texttt{scale\_factor} \Bigg( -\alpha_V V_{\text{coil}} + \alpha_S S_p - \alpha_D d_{\text{min}} + \alpha_B \chi^2_B + \alpha_K \norm{\bm{K}}_2 \\ + \alpha_{D,\tanh} \left(1 + \tanh\left( \left( d_{\min}-\texttt{d\_min\_target} \right)/\texttt{alpha\_D\_tanh\_scale} \right) \right) \Bigg)
\end{multline}
Here $S_p$ is the spectral width,
\begin{gather}
S_p = \sum_{m,n} m^p \left( \left(r_{mn}^c\right)^2 + \left(z_{mn}^s\right)^2 \right),
\label{spectral_width}
\end{gather}
$d_{\text{min}}$ is the minimum coil-plasma distance,
\begin{gather}
d_{\text{min}} = \min \left( \sqrt{ \left(\bm{r}_{\text{coil}} - \bm{r}_{\text{plasma}} \right)^2 } \right),
\end{gather}
and $\norm{\bm{K}}_2$ is the root-mean-squared current density,
\begin{gather}
\norm{\bm{K}}_2 = \sqrt{\chi^2_K/A_{\text{coil}}}.
\end{gather}
The coefficients in $f$ are defined by the user.

\myhrule

\param{alphaV}
{float}
{0}
{When \texttt{nlopt\_optimize} or \texttt{scipy\_optimize} is being called.}
{Scaling factor for $V_{\text{coil}}$ in the objective function.}

\myhrule

\param{alphaS}
{float}
{0}
{When \texttt{nlopt\_optimize} or \texttt{scipy\_optimize} is being called.}
{Scaling factor for $S_p$ in the objective function.}

\myhrule

\param{alphaD}
{float}
{0}
{When \texttt{nlopt\_optimize} or \texttt{scipy\_optimize} is being called.}
{Scaling factor for $d_{\text{min}}$ in the objective function.}

\myhrule

\param{alphaB}
{float}
{0}
{When \texttt{nlopt\_optimize} or \texttt{scipy\_optimize} is being called.}
{Scaling factor for $\chi^2_B$ in the objective function.}

\myhrule

\param{alphaK}
{float}
{0}
{When \texttt{nlopt\_optimize} or \texttt{scipy\_optimize} is being called.}
{Scaling factor for $\norm{\bm{K}}_2$ in the objective function.}

\myhrule

\param{alphaD\_tanh}
{float}
{0}
{When \texttt{nlopt\_optimize} or \texttt{scipy\_optimize} is being called.}
{Scaling factor for the $\tanh$ function in the objective function, which acts as a `wall` in parameter space when $d_{\min}$ reaches \parlink{d\_min\_target}. The scaling is set by \parlink{alphaD\_tanh\_scale}.}

\myhrule

\param{alphaD\_tanh\_scale}
{float}
{1.0}
{When \texttt{nlopt\_optimize} or \texttt{scipy\_optimize} is being called and \parlink{alphaD\_tanh\_scale} is non-zero.}
{Sets the scale length for the $\tanh$ function in the objective function. When this value is large, the gradients are less sharp.}

\myhrule

\param{d\_min\_target}
{float}
{0.1}
{When \texttt{nlopt\_optimize} or \texttt{scipy\_optimize} is being called and \parlink{alphaD\_tanh\_scale} is non-zero.}
{Sets the location of the `wall' in parameter space due to the $\tanh$ function.}

\myhrule

\param{scaleFactor}
{float}
{1}
{When \texttt{nlopt\_optimize} or \texttt{scipy\_optimize} is being called.}
{Scaling factor for objective function.}

\myhrule

\subsection{Scipy Optimize}

The following parameters are read if \texttt{scipy\_optimize} is being called.

\param{scipy\_optimize\_method}
{string}
{CG}
{When \texttt{scipy\_optimize} is being called.}
{The method used by \texttt{scipy\_optimize}. The following gradient-based methods are available: CG, BFGS, Newton-CG, L-BFGS-B, TNC, SLSQP, dogleg, and trust-ncp. See the \\
\href{https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.minimize.html}{scipy.optimize.minimize} documentation for more information.}

\myhrule

\param{grad\_option}
{integer}
{1}
{When \texttt{scipy\_optimize} is being called.}
{When \parlink{grad\_option} $ == 1$, the gradients computed by REGCOIL are used by \\ \texttt{scipy.optimize.minimize}. If \parlink{grad\_option} $ == 0$, a gradient function handle is not passed to \texttt{scipy.optimize}, and finite differencing is used.}

\myhrule

\param {maxiter}
{integer}
{1000}
{When \texttt{scipy\_optimize} is being called.}
{Maximum number of iteration to be taken by \texttt{scipy.optimize.minimize}.}

\myhrule

\param{norm}
{integer}
{2}
{When \texttt{scipy\_optimize} is being called.}
{Order of norm of the gradient used by \texttt{scipy.optimize.minimize} to determine successful termination.}

\myhrule

\param{gtol}
{float}
{$10^{-5}$}
{When \texttt{scipy\_optimize} is being called.}
{Tolerance for gradient norm required for termination.}

\myhrule

\param{nmax}
{integer}
{none}
{When \texttt{scipy\_optimize} is being called.}
{Maximum $n$ for Fourier modes of coil winding surface parameterization in \parlink{nescin\_filename}.}

\myhrule

\param{mmax}
{integer}
{none}
{When \texttt{scipy\_optimize} is being called.}
{Maximum $m$ for Fourier modes of coil winding surface parameterization in \parlink{nescin\_filename}.}

\myhrule

\param{nmax}
{integer}
{none}
{When \texttt{nlopt\_optimize} is being called.}
{Maximum $n$ value for winding surface Fourier modes in \texttt{nescin\_filename}.}

\myhrule

\param{mmax}
{integer}
{none}
{When \texttt{nlopt\_optimize} is being called.}
{Maximum $m$ value for winding surface Fourier modes in \texttt{nescin\_filename}.}

\subsection{NLOPT Optimize}

The following parameters are read if \texttt{nlopt\_optimize} is being called.

\param{constraint\_min}
{integer}
{0}
{When \texttt{nlopt\_optimize} is being called.}
{\texttt{constraint\_min} = 0: No constraint on a minimum coil-plasma distance is enforced. \\
 \texttt{constraint\_min} = 1: Minimum coil-plasma distance is constrained to be $\leq$ \texttt{d\_min}. }

\myhrule

\param{d\_min}
{float}
{0.2}
{When \texttt{nlopt\_optimize} is being called and \texttt{contraint\_min} = 1.}
{Minimum coil-plasma allowed for optimized winding surface.}

 \myhrule

\param{constraint\_max\_K}
{integer}
{0}
{When \texttt{nlopt\_optimize} is being called.}
{\texttt{constraint\_max\_K} = 0: No constraint on $\max K$. \\
 \texttt{constraint\_max\_K} = 1: Maximum current density is constraint to be $\leq$ \texttt{max\_K}}

\myhrule

\param{max\_K}
{float}
{7.1e6}
{When \texttt{nlopt\_optimize} is being called and \texttt{contraint\_max\_K} = 1.}
{Maximum current density allowed during winding surface optimization. }

\myhrule

\param{constraint\_rms\_K}
{integer}
{0}
{When \texttt{nlopt\_optimize} is being called.}
{\texttt{constraint\_rms\_K} = 0: No constraint on $\norm{K}_2$. \\
 \texttt{constraint\_rms\_K} = 1: Maximum current density is constraint to be $\leq$ \texttt{rms\_K}}

\myhrule

\param{rms\_K}
{float}
{2.36e6}
{When \texttt{nlopt\_optimize} is being called and \texttt{contraint\_rms\_K} = 1.}
{Maximum current density allowed during winding surface optimization. }

\myhrule

\param{nlopt\_method}
{string}
{none}
{When \texttt{nlopt\_optimize} is being called. }
{Algorithm used for gradient based winding surface optimization. The following options are supported.
\begin{itemize}
\item \texttt{nlopt.G\_MLSL\_LDS}
\item \texttt{nlopt.LD\_LBFGS}
\item \texttt{nlopt.LD\_MMA}
\item \texttt{nlopt.LD\_SLSQP}
\item \texttt{nlopt.LD\_CCSAQ}
\item \texttt{nlopt.LD\_TNEWTON\_PRECOND\_RESTART}
\item \texttt{nlopt.LD\_VAR1}
\end{itemize}
}

\myhrule

\param{omega\_min}
{float}
{-7}
{When \texttt{nlopt\_optimize} is being called. }
{Minimum value for $r_{mn}^c$ or $z_{mn}^s$ allowed for winding surface optimization.}

\myhrule

\param{omega\_max}
{float}
{7}
{When \texttt{nlopt\_optimize} is being called. }
{Maximum value for $r_{mn}^c$ or $z_{mn}^s$ allowed for winding surface optimization.}

\myhrule

\param{constraint\_tol}
{float}
{1e-6}
{When \texttt{nlopt\_optimize} is being called and \texttt{constraint\_min} =1 or \texttt{constraint\_max\_K} = 1 or \texttt{constraint\_rms\_K}.}
{Tolerance allowed for constraint equation to be satisfied.}

\myhrule

\param{ftol\_rel}
{float}
{1e-6}
{When \texttt{nlopt\_optimize} is being called.}
{Optimization will stop when the relative change in the objective function $f$ is less than \texttt{ftol\_rel} in successive steps.}

\myhrule

\section{General considerations and tips}
\begin{itemize}
\item The results of the optimization vary widely with the input parameters. It is suggested that a user perform low-resolution optimizations with varying parameters (such as $\alpha_B$, $\alpha_S$, $\alpha_{\max{K}}$, and $\alpha_V$). To begin, one can run the optimization for a few evaluations to ensure that it is descending in the desired direction.
\item If \parlink{general\_option} = 4 or 5 (a $\lambda$ search is performed for a target function), it is not always possible to obtain a solution for $\lambda$ if the current density is too low or high. In this case, the optimization scripts adjust the \parlink{target\_current\_density} and will print a message to standard output. If you see that the \parlink{target\_current\_density} is readjusted several times, it is probably a good idea to begin the optimization with a different \parlink{target\_current\_density} .
\item The SLSQP and CCSAQ algorithms in \texttt{nlopt} can be sensitive to the selection of \parlink{omega\_min} and \parlink{omega\_max}, as these set the initial step size for the optimization. If the winding surface wanders to far from the initial surface, these should be adjusted.
\end{itemize}


\subsection{Tuning Rules According to Ziegler-Nichols (Oscillation Method)}

This method relies on a plant that can potentially become unstable. That is to
say,  the  plant  must be of higher order such  that  the  phase  crosses  the
\SI{180}{\degree} threshold at some point. If this is the case, then the plant
can be  placed  in a closed loop with a single gain element (P-Controller) and
the gain can be  increased  until  the  system's  output  begins to oscillate.

\begin{figure}[H]
    \centering
    \includegraphics[width=\imagewidth]{images/osc_method.png}
\end{figure}

The gain at which this occurs  is  $K_{p,crit}$  and  the  period  at which it
oscillates is $\tau_{crit}$.

The full procedure of the Ziegler-Nichols method is:

\begin{enumerate}
    \item Use a pure P controller and set a small gain $K_{P}$.
    \item Increasing the gain $K_{P}$ until an undamped oscillation occurs.
    \item Indentify the critical gain $K_{p,crit}$ and the critical period $\tau_{crit}$
    \item According to table \ref{tab:ziegler_nichols} specifying the controllers parameter.
\end{enumerate}

\begin{center}
    \begin{threeparttable}
        \begin{tabular}{cccc}
            \toprule
            Type & $K_{P}$                   &  $T_{i}$                   &  $T_{d}$ \\
            \midrule
            P    &  $0.5  \cdot K_{P,crit}$  &  -                         &  -                         \\
            PI   &  $0.45 \cdot K_{P,crit}$  &  $0.85 \cdot \tau_{crit}$  &  -                         \\
            PID  &  $0.6  \cdot K_{P,crit}$  &  $0.5 \cdot \tau_{crit}$   &  $0.12 \cdot \tau_{crit}$  \\
            \bottomrule
        \end{tabular}
        \caption{Table with controller parameters, according to the Ziegler-Nichols method (good disturbance rejection).}
        \label{tab:ziegler_nichols}
    \end{threeparttable}
\end{center}

The advantage of the Ziegler-Nichols method is  that it is simple to apply and
because of that also easy to understand. On the  other  hand  it is very risky
because the control loop must be operated close to instability.  Dependong  on
the  plant  being  measured  this  can  be  very dangerous and very expensive.

A  disadvantage  is  the  need  for  the  plant to  be  potentially  unstable.
Theoretically not all plants can be made to oscillate in a closed  loop with a
P-controller. The  practical  significance  of  this  method  is thus limited.


We present three benchmakrs in the poster:

\subsection{Standard Poisson systems}

    \subsubsection{Description}
    This benchmark provides a reference to users whose CFD codes solve standard Poisson systems.
    For this purpose, we measured only the run time of solving the system, that is, the calls to
    \lstinline[language=C++, basicstyle=\ttfamily]|KSPSolve| and
    \lstinline[language=C++, basicstyle=\ttfamily]|solve(x, rhs)|.
    The  function \lstinline[language=C++, basicstyle=\ttfamily]|solve(x, rhs)|
    covers MPI communications and data conversion and transfer,
    so timing the call to this
    function gives a performance indication of our wrapper, rather than only
    of the solution behavior on GPUs.

    The benchmarks use conjugate-gradient solvers on both CPUs and GPUs.
    The preconditioner for the CPU solver is Hypre BoomerAMG
    (through PETSc's interface),
    whereas the preconditioner of the GPU solver is classical algebraic multigrid --
    described as a GPU implementation of Hypre BoomerAMG in the AmgX
    manual\cite[see][p.130]{amgx-manual}.

    \subsubsection{Results}
    With 25M unknowns,
    four\footnotemark[2] NVIDIA K20s can compete with about 100\footnotemark[3]
    CPU cores in 2D problems;
    in 3D, eight K20s can compete with about 256 CPUs.
    For 2D problems with 100M unknowns and 3D problems with 50M unknowns,
    32 K20s can compete with about 400 CPU cores.

\subsection{Flying snake simulations -- an application of PetIBM}

    \subsubsection{Description}
    This benchmark intends to show how multi-GPU computing can accelerate a real
    PETSc-based CFD code through an actual application.
    The CFD code we chose is PetIBM\cite{petibm-repo},
    and the application is simulating the flow around a flying-snake's cross-section\cite{Krishnan-2013-ID33}.

    PetIBM, our group's PETSc-based CFD code, offers a Navier-Stokes solver based on
    the formulation of Taira and Colonius\cite{Taira-2007-ID69}, in which a modified
    Poisson system is solved at each time step.
    Solving this modified Poisson system takes over 90\% of run rime on CPUs.
    We only applied multi-GPU linear solvers on this part, while keeping all other
    calculations on CPUs.

    We attempt a fair comparison using the same nodes of the cluster, with
    the GPU runs exploiting two available NVIDIA K20s on each node
    (each node has 12 physical CPU cores).
    We also ran the benchmark on a workstation, having 6 physical
    CPU cores and up to two NVIDIA K40c GPUs.

    We obtain the best run times on the GPU cases with conjugate-gradient solvers and
    aggregation multigrid preconditioners.
    For the CPU cases, we used stabilized biconjugate-gradient solvers and GAMG
    preconditioners, given that (to the best of our knowledge) there is no
    counterpart implementation of aggregation multigrid in PETSc,
    and the combination of stabilized biconjugate-gradient and GAMG gave
    the best result in this case.

    \subsubsection{Results}
    The overall speed-up provided by only one GPU node is
    competitive with about 20 nodes of a CPU cluster.
    On a workstation, one K40c GPU can compete with about 16 nodes of a CPU cluster.

\subsection{Amazon EC2}

    \subsubsection{Description}
    The previous benchmark shows that enabling multi-GPU computing in a CFD code can
    size down the clusters required for running simulations.
    This comes with a cost saving to research based on CFD simulations.
    We experimented with the same flying-snake simulations on Amazon EC2 to give
    an example of the cost savings.
    We compared the simulation executed on an 8-node CPU cluster (c4.8xlarge)
    versus a single GPU node (g2.8xlarge).

    \subsubsection{Results}
    The result indicates a 3.1x speed-up and a 16x cost saving on Amazon EC2
    when exploiting the GPU nodes.

\footnotetext[2]{
    The number of GPUs presented in the text is the minimum number of GPUs needed
    due to limited memory on a single GPU.
}

\footnotetext[3]{
    The comparable numbers of CPU cores are estimated based on
    good scaling shown in the benchmarks (see figures on the poster).
    This is a conservative estimate for our purposes (i.e., favors the CPU case).
}


\documentclass[a4paper]{article}

\input{temp}

\begin{document}

\title{Revision Questions}
\date{Easter 2016}

\maketitle

\newpage

\tableofcontents

\newpage

\section{Introduction}
There are many tripos questions which are far more than enough for anyone who is keen to practice. These questions are, however, usually targeted at the fundamental ideas, and can be used as a check list when you think you have finished revision on a course.

\newpage

\section{Groups}

\newpage

\section{Vector Calculus}

\subsection{Problem 1}
Explain what is meant by saying that $f$ is a \emph{scalar field}.\\
Let $f$ be a scalar field, $\mathbf{a}$ be a point in the space where $f$ is defined. Show that the direction in which $f$ is increasing at the greatest rate is the direction of $\nabla f$ at that point.\\
Let a surface $S$ be defined by $f=c$ where $c$ is a constant, and let $\mathbf{b}$ be a point on the surface. Show that $\nabla f$ at $\mathbf{b}$ is normal to the surface.

\subsection{Problem 2}
Deduce the relationship between the basis vectors $\mathbf{e}_r,\mathbf{e}_\varphi,\mathbf{e}_\theta$ of the spherical polar coordinate system and $\mathbf{e}_x,\mathbf{e}_y,\mathbf{e}_z$ of the Cartesian coordinate system.\\
Let $S$ be a surface of sphere of radius $a$. Show that the scalar area element is given by
\begin{equation*}
\begin{aligned}
dS = a^2 \sin\theta d\theta d\varphi.
\end{aligned}
\end{equation*}

\subsection{Problem 3}
State what is the \emph{curl} of a vector field $\mathbf{F}$ in 3 dimensions.\\
State Green's theorem.\\
State Stoke's theorem.\\
Prove that Green's theorem and Stoke's theorem are equivalent.

\subsection{Problem 4}
Let $\mathbf{F}$ be a vector field in $\R^3$. Show that the following three statements are equivalent to each other:\\
i) $\mathbf{F}=\nabla f$ for some scalar field $f$;\\
ii) $\int_C \mathbf{F}\cdot d\mathbf{r}$ is independent of $C$ for fixed endpoints and orientation;\\
iii) $\nabla\times\mathbf{F}=0$.

\subsection{Problem 5}
Let $C$ be a curve in $\R^n$. State what is meant by a \emph{parameterisation} of it.\\
Explain why
\begin{equation*}
\begin{aligned}
\mathbf{r}:[0,4\pi) \to \R^2
\end{aligned}
\end{equation*}
with
\begin{equation*}
\begin{aligned}
\theta \to \left(\cos\theta,\sin\theta\right)
\end{aligned}
\end{equation*}
is \emph{not} a parameterisation of the unit circle centred at the origin.\\
State what is the \emph{arclength} of $C$.\\
Show that the arclength of $C$ is not changed under different parameterisations.

\subsection{Problem 6}
Let $f$ be a scalar field and $\mathbf{F}$ be a vector field in $\R^3$.\\
Consider $f\left(\mathbf{r}\left(u,v,w\right)\right)$ in an orthogonal curvilinear coordinate system. Show that
\begin{equation*}
\begin{aligned}
\nabla f = \frac{1}{h_u} \frac{\partial f}{\partial u} \mathbf{e}_u + \frac{1}{h_v} \frac{\partial f}{\partial v} \mathbf{e}_v + \frac{1}{h_w} \frac{\partial f}{\partial w} \mathbf{e}_w
\end{aligned}
\end{equation*}
where $h_u, h_v, h_w$ satisfy
\begin{equation*}
\begin{aligned}
\frac{\partial \mathbf{r}}{\partial u} = h_u \mathbf{e}_u, \frac{\partial \mathbf{r}}{\partial v} = h_v \mathbf{e}_v, \frac{\partial \mathbf{r}}{\partial w} = h_w \mathbf{e}_w.
\end{aligned}
\end{equation*}
(non-examinable?)\\
Deduce the formula of $\nabla f$, $\nabla \cdot \mathbf{F}$, $\nabla \times \mathbf{F}$ in cylindrical polar coordinates.\\
Deduce the formula of $\nabla f$, $\nabla \cdot \mathbf{F}$, $\nabla \times \mathbf{F}$ in spherical polar coordinates.

\subsection{Problem 7}
State Gauss' Law of gravitation.\\
Assuming mass is distributed with spherical symmetry, deduce Newton's law of gravitation for point masses from Gauss' Law of gravitation.\\
Deduce that
\begin{equation*}
\begin{aligned}
\nabla\cdot \mathbf{g} = -4\pi G \rho.
\end{aligned}
\end{equation*}
Deduce that
\begin{equation*}
\begin{aligned}
\nabla^2 \mathbf{F} = \nabla\left(\nabla\cdot\mathbf{F}\right) - \nabla \times \left(\nabla\times\mathbf{F}\right).
\end{aligned}
\end{equation*}
Hence a gravitational potential $\varphi$ can be chosen such that
\begin{equation*}
\begin{aligned}
\nabla^2 \varphi = 4\pi G\rho
\end{aligned}
\end{equation*}
with $-\nabla \varphi = \mathbf{g}$.

\subsection{Problem 8}
Describe what is a \emph{Dirichlet condition} and what is a \emph{Neumann condition}.\\
State and prove the Uniqueness theorem.\\
Prove Green's First and Second identities:
\begin{equation*}
\begin{aligned}
\int_S \left(u\nabla v\right) \cdot dS = \int_V \left(\nabla u\right) \cdot\left(\nabla v\right) dV + \int_V u\nabla^2 v dV,\\
\int_S \left(u\nabla v - v\nabla u\right) \cdot d\mathbf{S} = \int_V \left(u\nabla^2 v - v\nabla^2 u \right) dV.
\end{aligned}
\end{equation*}

\subsection{Problem 9}
Explain what is a \emph{harmonic function}.\\
State and prove the mean value property for harmonic functions.\\
If $\varphi$ is a harmonic function in a region $V$, show that it can't take a minimum or maximum at an interior point of $V$.

\subsection{Problem 10}
State the Maxwell's equations.\\
Show that
\begin{equation*}
\begin{aligned}
\frac{\partial \rho}{\partial t}+ \nabla \cdot \mathbf{j} = 0.
\end{aligned}
\end{equation*}

\subsection{Problem 11}
Explain what is a \emph{tensor}. Explain what is the \emph{rank} of a tensor.\\
State the \emph{tensor transformation rule}.\\
Let $\mathbf{u},\mathbf{v},...,\mathbf{w}$ be n vectors. Explain why
\begin{equation*}
\begin{aligned}
T_{ij...k} = u_i v_j ... w_k
\end{aligned}
\end{equation*}
is a rank $n$ tensor.

In the following problems, the tensors are
by default Cartesian tensors.

\subsection{Problem 12}
Define the \emph{tensor product}, and show that the tensor product of two tensors is still a tensor.\\
Explain what is meant by saying that a tensor is \emph{antisymmetric} in some two indices.

\subsection{Problem 13}
Show that a rank $n$ tensor is equivalent to a multilinear map from $n$ vectors to $\R$.

\subsection{Problem 14}
State and prove the quotient rule of tensors. Prove that its converse is also true.

\subsection{Problem 15}
Explain what is a \emph{tensor field}.\\
Let $T_{ij...k}$ be a rank $n$ tensor. Show that
\begin{equation*}
\begin{aligned}
\frac{\partial}{\partial x_p} T_{ij...k}
\end{aligned}
\end{equation*}
is a rank $n+1$ tensor.

\subsection{Problem 16}
State and prove the divergence theorem for tensors.

\newpage

\section{Differential Equation}

\newpage

\section{Probability}

\newpage

\section{Vectors and Matrices}

\newpage

\section{Analysis}

\newpage

\section{Numbers and Sets}

\newpage

\section{Dynamics and Relativity}

\subsection{Problem 1}
Define \emph{central force}.\\
State the formula of \emph{angular momentum}. Prove that it is conserved by a central force.

\subsection{Problem 2}
Define \emph{gravitational potential energy} and \emph{gravitational potential}.\\
For a planet with radius $R$ and mass $M$, find its escape velocity.

\subsection{Problem 3}
State the formula for the force experienced by a particle with charge $q$ and velocity $\mathbf{v}$ in an electric field $\mathbf{E}$ and magnetic field $\mathbf{B}$.\\
Show that, for time independent electric field and magnetic field, the energy
\begin{equation*}
\begin{aligned}
E=\frac{1}{2}m|\mathbf{v}|^2 + q \phi_e
\end{aligned}
\end{equation*}
is conserved.

\subsection{Problem 4}

\newpage

\end{document}


\documentclass[11 pt]{scrartcl}
\usepackage[header, margin, koma]{tyler}

\newcommand{\hwtitle}{Discussion 1B Recap}

\pagestyle{fancy}
\fancyhf{}
\fancyhead[l]{\hwtitle{}}
\fancyhead[r]{Tyler Zhu}
\cfoot{\thepage}

\begin{document}
\title{\Large \hwtitle{}}
\author{\large Tyler Zhu}
\date{\large September 4, 2020}

\maketitle

\section{Induction}

Induction is typically most helpful when we try to prove statements of the form: $(\forall n\in \NN), P(n)$ is true. This happens in three steps:
\itemnum
    \ii \styl{Base Case}: Prove that $P(0)$ is true.
    \ii \styl{Induction Hypothesis}: Assume that for any $k \geq 0$, $P(k)$ is true.
    \ii \styl{Inductive Step}: Prove that $P(k+1)$ is true, showing that $P(k) \implies P(k+1)$.
\itemend

When writing induction proofs for homework or for exams, be sure to state these three steps clearly for maximal points.

Induction used like this is typically referred to as \emph{weak} induction, in contrast with \emph{strong} induction, where in the IH we instead make the assumption that for all $0\leq k' \leq k$, $P(k')$ is true.

Let me reinforce the relationship between weak and strong induction. If we have a statement $P(n)$ for integer $n \geq 0$, then weak induction proves $P(0)$ and $P(k) \implies P(k+1)$. So in this framework, if a statement $P(5)$ is true, it must have been proved through the connection $P(0) \implies P(1) \implies P(2) \implies \dots \implies P(5)$, meaning that if $P(5)$ is true, $P(k')$ is true for $0\leq k' \leq 4$.

But that's precisely what strong induction is! To prove $P(k+1)$, we assume that $P(k')$ is true for $0 \leq k' \leq k$. Hence, strong and weak induction are the same, aside from their assumptions. So you don't need to worry about whether you should use strong or weak induction and just worry about what base assumptions you need in order to prove $P(k+1)$.

\section{Tips}
\itemnum
    \ii To show the IS, take $k+1$ case, break it down to the $k$ case, apply the induction hypothesis, and re-extend to the $k+1$ case. In other words, start with the statement of $P(k+1)$ and manipulate until you can apply the induction hypothesis of $P(k)$, then show that $P(k+1)$ follows. This is the standard way to use induction, and is more useful than you'd expect.
    \ii For purely algebraic proofs, you can feel free to manipulate from the LHS to the RHS or vice versa, but do not modify both sides at the same time. This works because algebra is symmetric.
    \ii If you need more room for bounds or arguing in the IS, consider \emph{strengthening the hypothesis}, i.e. make a claim which is more specific which implies your statement. It's counterintuitive that we would try to instead prove a harder statement, but the extra assumptions often help in induction.
\itemend

\end{document}


\documentclass{article}

\usepackage{algpseudocode}
\usepackage{booktabs}
\usepackage{caption}
\usepackage{enumitem}
\usepackage{fontspec}
\usepackage{graphicx}
\usepackage{mathtools}
\usepackage{microtype}
\usepackage{sectsty}
\usepackage[table,xcdraw]{xcolor}

\allsectionsfont{\sffamily}
\captionsetup{font=small,labelfont={bf,sf},margin=1.5em}

\title{Balanced Binary Search Trees}
\author{Florian Kretlow}

\begin{document}

\section{Red-Black-Tree}

\subsection{Introduction and general properties}
A red-black-tree is a self-balancing binary search tree. In addition to the invariants
of the latter, the former garantees that the tree remains roughly balanced even when it
receives input sequences that constitute pathological cases for normal binary search
trees (sorted input).

Every node in a red-black-tree is assigned one of two colors (traditionally red and
black). The following invariants hold after each insertion into, or deletion from a
red-black-tree.

\begin{enumerate}[label=(\arabic*)]
\item The root node of the tree is black. Every other node is either red or black.
\item If a node is red, it doesn't have a red child.
\item All paths from the root to a leaf go through the same number of black nodes.
\end{enumerate}

If \(b\) is the number of black nodes on every path from the root to a leaf, then the
shortest possible path from the root to a leaf (which contains no red nodes) has a
length of \(b\), and the longest possible path (where black and red nodes alternate) has
a length of \(2b\).  Thus the invariants ensure that the lengths of any two paths from
the root to a leaf (i.e. the depths of any two leaf nodes) differ by at most a factor of
2.

\[
\forall \text{ leaf nodes } l_{i}, l_{j} \in T : \text{depth}(l_i) \leq 2 \cdot
\text{depth}(l_j)
\]

The balance is weaker than in an AVL tree (where the difference between the heights of
the left and right sub-trees of any sub-tree is at most 1). Consequently, operations on
a red-black-tree require fewer tree rotations, so they’re faster. Both red-black-trees
and AVL trees get, insert, and delete elements in logarithmic time (\(\Theta(\log n)\)).

\subsection{Groups of Nodes}
Let a \emph{group} of nodes in a red-black-tree denote \emph{any black node together
with all its direct red child nodes.} Every node in a red-black-tree belongs to exactly
one group: Every black node is the ‘head’ of its own group. Every red node belongs
to the group of its black parent. Using the fact that every group contains exactly one
black node, we can rephrase invariant 3 as follows:

\begin{enumerate}[label=(\arabic*)]
\setcounter{enumi}{2}
\item All paths from the root to a leaf go through the same number of groups.
\end{enumerate}

The \emph{weight} of a group \(g\) is the number of nodes it contains, obviously \(1
\leq \text{weight}(g) \leq 3\). If \(\text{weight}(g) = 1\), \(g\) is said to be
\emph{empty:} it doesn't contain any red child nodes.  If \(\text{weight}(g) = 3\),
\(g\) is said to be \emph{full:} it contains two red child nodes and it's not possible
to add another node to the group.  If \(\text{weight}(g) = 2\), it's possible to rotate
\(g\) in such a way that the former red child becomes the black root of the group, and
the former black root becomes a red child on the opposite site; this rotation does not
affect any other group than \(g\).

\begin{figure}
\begin{centering}
    \includegraphics[scale=2]{img/groups.pdf}
    \caption{Possible groups in a red-black-tree. The direct ancestor of the head of a
    group can be both red and black. Every direct descendant below a group is black.}
\end{centering}
\end{figure}

\subsection{Insertion}
Insertion into a normal binary search tree is straightforward. You start at the root,
and then you always go left if the new value is less than the current value, or you go
right if it is greater. When you've nowhere left to go (i.e. you'd need to go left but
there's no left child, or the other way around) you simply add the value at that
position.

In order to preserve the invariants, inserting a node into a red-black-tree must not add
a new group to the tree (except at the very root where the addition affects all paths
equally).  Therefore insertion is not possible if the direct parent of the to-be-added
node is part of a full group. The solution is to reduce the weight of that group: If \(
\text{weight}(g) \leq 2\), it is possible to add a node to \(g\).

\begin{small}
\begin{verbatim}
procedure decrease-weight(n):
    p := parent of n
    pp := grandparent of n
    l, r := children of n

    // assume that n is black and l, r are red

    case 1: n is the root of the tree
        recolor:
            l, r: black

    case 2: p is black
        recolor:
            n: red
            l, r: black

    case 3: p is red
        case 3.1: pp has 1 red child (p)
            case 3.1a: pp->p->n is right-right or left-left
                right-right: rotate-left(pp)
                left-left: rotate-right(pp)
                recolor:
                    pp: red
                    p: black
                    n: red
                    l, r: black
            case 3.1b: pp->p->n is right-left or left-right
                right-left:
                    rotate-right(p)
                    rotate-left(pp)
                left-right:
                    rotate-left(p)
                    rotate-right(pp)
                recolor:
                    pp: red
                    l, r: black

        case 3.2: pp has 2 red children
            decrease-weight(pp)
            decrease-weight(n) // try again

procedure insert(n, v):
    p := parent of n

    if v == n.value:
        return
    else if v < n.value and n has a left child:
        insert(n.left, v)
        return
    else if v > n.value and n has a right child:
        insert(n.right, v)
        return

    case 1: n is black
        if v < n.value:
            n.left = new node(v, red)
            n.left.parent = n
        else:
            n.right = new node(v, red)
            n.right.parent = n

    case 2: n is red
        case 2.1: weight(p) = 2
            decrease-weight(p)
            // now n can be anywhere, go again:
            insert(n, v)

        case 2.2: weight(p) = 1
            case 2.2.1: v < n.value and n = p.left
                rotate-right(p)
                n.left = new node(v, red)
            case 2.2.2: v > n.value and n = p.right
                rotate-left(p)
                n.right = new node(v, red)
            case 2.2.3: v < n.value and n = p.right
                p.left = new node(v, red)
                swap(p, p.left)
            case 2.2.4: v > n.value and n = p.left
                p.right = new node(v, red)
                swap(p, p.right)
\end{verbatim}
\end{small}

\subsection{Deletion}

Deletion is only possible if the group we delete from is not empty. Likewise, deleting a
node from the tree must not remove a group entirely (again, except at the root).

\begin{small}
\begin{verbatim}
procedure increase-weight(n):
    p := parent of n
    s := sibling of n

    // case 1: ancestor group is not empty
    if p is red or weight(p) >= 2:
        if p is black:
            if n == p.right:
                rotate-right(p); s = p.left
            else:
                rotate-left(p); s = p.right

        // case 1.1: s is empty
        if weight(s) == 1:
            recolor:
                p: black
                n, s: red

        // case 1.2: s is not empty
        else:
            // make sure s has an outer child
            if s == p.left and s.left is black:
                rotate-left(s); s = p.left
            else if s == p.right and s.right is black:
                rotate-right(s); s = p.right

            if n == p.right:
                rotate-right(p)
            else:
                rotate-left(p)

            recolor:
                p: black
                n, s: red
                outer child of s: black

    // case 2: ancestor group is empty
    else:
        // case 2.1: s is empty
        if weight(s) == 1:
            if p is root:
                recolor:
                    n, s: red
            else:
                increase-weight(p)
                increase-weight(n)

        // case 2.2: s is not empty
        else:
            // make sure s has an outer child
            if s == p.left and s.left is black:
                rotate-left(s); s = p.left
            else if s == p.right and s.right is black:
                rotate-right(s); s = p.right

            if n == p.right:
                rotate-right(p)
            else:
                rotate-left(p)

            recolor:
                n: red
                outer child of s: black

procedure delete(n, v):
    while v != n.value:
        if v < n.value:
            n = n.left
        else if v > n.value:
            n = n.right
        if n == nil: return // not found

    if n is not in a leaf position:
        successor := n.right
        while successor has a left child:
            successor = successor.left
        swap(n, successor)
        // now n is where successor was, in a leaf position

    if n is black and not the root and weight(n) == 1:
        increase-weight(n)

    if n is black:
        if n has a left child:
            rotate-right(n)
        else:
            rotate-left(n)

    remove n from the tree

\end{verbatim}
\end{small}

\end{document}


\documentclass[aspectratio=169]{beamer}
\usepackage[utf8]{inputenc}
\usepackage{hyperref}
\usepackage{amsmath,amsfonts,amsthm,bm}
\usepackage{color}
\usepackage{minted}
\usepackage{graphicx} % Allows including images
\usepackage{booktabs} % Allows the use of \toprule, \midrule and \bottomrule in tables
\usepackage{tikz}
\usepackage[version=3]{mhchem}
\usepackage{pgfplots}
\pgfplotsset{compat=1.16}
\setminted{fontsize=\scriptsize}

\hypersetup{
    colorlinks=true,
    linkcolor=red,
    filecolor=magenta,
    urlcolor=red,
}

\DeclareMathOperator*{\argmax}{argmax}
\DeclareMathOperator*{\argmin}{argmin}
\let \vec \mathbf

\mode<presentation> {
    \usetheme{CambridgeUS}
    \setbeamertemplate{footline}[page number]
    \setbeamertemplate{navigation symbols}{}
}

\title[Linear Methods]{Linear Methods}

\author{Shyue Ping Ong}
\institute[UCSD]{University of California, San Diego\\
\medskip
}
\date{NANO281}

\begin{document}

\begin{frame}
    \titlepage % Print the title page as the first slide
\end{frame}

\begin{frame}{Overview}
    \tableofcontents
\end{frame}

\section{Preliminaries}

\begin{frame}{Preliminaries}
    \begin{itemize}
        \item We will go very deep into linear models.
        \item Most of you probably have seen linear models in some form, but we will start from scratch to further illustrate key concepts such as bias and variance.
        \item We will then discuss techniques such as regularization and transformation of inputs in the context of linear methods.
    \end{itemize}
\end{frame}

\begin{frame}{Notation}
    \begin{itemize}
        \item Capital letters, e.g., $X$ denote variables.
        \item Lower-case letters e.g., $x$, denote observations.
        \item Dummy index $j$ to denotes different variables, e.g., $X_j$
        \item Dummy index $i$ to denotes different observations, e.g., $x_i$
        \item Bolded variables are vector/matrices, e.g., $\vec{y}$, $\vec{X}$
    \end{itemize}
\end{frame}

\section{Linear regression}

\begin{frame}{Linear Regression}
    \Huge{\centerline{Linear Regression}}
\end{frame}

\begin{frame}{Simplest possible model between target and feature}
    \begin{equation*}
        Y=f(X_1,X_2,...,X_p)= \beta_0 + \sum_{j=1}^p \beta_j X_j
    \end{equation*}
    $X_j$ can be:
    \begin{itemize}
        \item Quantitative inputs
        \item Transformations of quantitative inputs, e.g., log, exp, powers, etc.
        Basis expansions, e.g., $X_2 = X_1^2$, $X_3 = X_1^3$
        \item Interactions between variables
        \item Encoding of levels of inputs
    \end{itemize}
\end{frame}

\begin{frame}{Supervised learning}
    \begin{itemize}
        \item Given a set of paired observations $\{x_{ij}, y_i\}$, what are the model parameters (in this case, the coefficients $\beta_j$) that are ``optimal''?
        \item ``Optimal'' is typically defined as minimization of some \textbf{loss function} (also known as \textbf{cost function}) that measures the error of the model.
    \end{itemize}
\end{frame}

\begin{frame}{Least squares regression}
    Consider the simple case of
    \begin{equation*}
        Y = \beta_0 + \beta_1 X_1
    \end{equation*}
    In least squares regression, the loss function is defined as the sum squared error given the $N$ observations:
    \begin{eqnarray*}
        L(Y, \hat{f}(X)) & = & \sum_{i=1}^N (y_i - f(x_i))^2 \\
        & = & \sum_{i=1}^N (y_i - \beta_0 - \beta_1 x_{i1})^2
    \end{eqnarray*}
    \end{frame}

    \begin{frame}{What are the optimal parameters $\beta_0$ and $\beta_1$?}
    \begin{eqnarray*}
        \frac{\partial L}{\partial\beta_0} & = & \sum_{i=1}^N 2 (y_i - \beta_0 - \beta_1 x_{i1})(-1) = 0\\
        \implies & & \sum_{i=1}^N y_i = N \beta_0 + \sum_{i=1}^N \beta_1 x_{i1} \\
        \implies & & \beta_0 = \bar{y} - \beta_1 \bar{x_{1}} \\
        \frac{\partial L}{\partial\beta_1} & = & \sum_{i=1}^N 2 (y_i - \beta_0 - \beta_1 x_{i1}) (-x_{i1}) = 0\\
        \implies & & \beta_1 = \frac{\sum_{i=1}^N x_{i1} y_i - N \bar{x_1}\bar{y}}{\sum_{i=1}^N x_{i1}^2 - N \bar{x_1}^2}\\
    \end{eqnarray*}
    \end{frame}

    \begin{frame}{Reformulating the general multiple linear regression as a vector equation…}
    Considering $N$ observations of
    \begin{equation*}
        y_i = \beta_0 + \beta_1 x_{i1} + + \beta_2 x_{i2} + ... + \beta_p x_{ip}
    \end{equation*}
    Let
    \begin{equation*}
        \vec{y} = \begin{pmatrix}y_1\\y_2\\...\\y_n\end{pmatrix}, \bm{\beta} = \begin{pmatrix}\beta_0\\\beta_1\\...\\\beta_p\end{pmatrix}, \vec{X} = \begin{pmatrix}1 & x_{11} & x_{12} & ... & x_{1p}\\
        1 & x_{21} & x_{22} & ... & x_{2p}\\
        \vdots & & & & \\
        1 & x_{N1} & x_{N2} & ... & x_{Np}\end{pmatrix},
    \end{equation*}
    So,
    \begin{equation*}
        \vec{y} = \vec{X}\bm{\beta}
    \end{equation*}
    Note that $\vec{y}$ is a $N \times 1$ vector,
    $\bm{\beta}$ is a $(p+1) \times 1$ vector, and $\vec{X}$ is a  $N \times (p+1)$ matrix.
\end{frame}

\begin{frame}{Reformulating the general multiple linear regression as a vector equation…}
    \begin{equation*}
        L = RSS = (\vec{y} - \vec{X}\bm{\beta})^T(\vec{y} - \vec{X}\bm{\beta})
    \end{equation*}
    Assuming (for the moment) that $\vec{X}$ has full column rank, and hence $\vec{X}^T\vec{X}$ is positive definite, It can be shown using the same principles that the following unique solution for $\bm{\beta}$ is:
    \begin{eqnarray*}
        \hat{\bm{\beta}} &=& (\vec{X}^T \vec{X})^{-1} \vec{X}^T \vec{y} \\
        \hat{\vec{y}} & = & \vec{X} \hat{\bm{\beta}} = \vec{X}(\vec{X}^T \vec{X})^{-1} \vec{X}^T \vec{y}
    \end{eqnarray*}
\end{frame}

\begin{frame}{Graphic representation of MLR with two dependent variables}
    \begin{figure}
        \centering
        \includegraphics[width=0.3\textwidth]{figures/fig3-1.pdf}
        \includegraphics[width=0.3\textwidth]{figures/fig3-2.pdf}
        \caption{MLR minimizes sum square of residuals. The projection $\vec{\hat{y}}$ represents the vector of the least squares predictions onto the hyperplane spanned by the input vectors $\vec{x_1}$ and $\vec{x_2}$. \cite{hastieElementsStatisticalLearning2016}.}
    \end{figure}
\end{frame}

\begin{frame}{Validity of least squares criterion}
    \begin{itemize}
        \item Observations are independently drawn at random.
        \item Variance of $\vec{y}$ is constant given by $\sigma^2$.
    \begin{equation*}
        \mathrm{var}(\hat{\bm{\beta}}) = (\vec{X}^T \vec{X})^{-1} \sigma^2
    \end{equation*}
    \item and $\sigma$ is estimated using:
    \begin{equation*}
        \sigma^2 = \frac{1}{N-p-1}\sum_{i=1}^N (y_i - \hat{y_i})^2
    \end{equation*}
    \end{itemize}
\end{frame}

\begin{frame}{Example materials data}
    \begin{columns}
    \begin{column}{0.6\textwidth}
        \begin{itemize}
            \item Target: Bulk modulus of elements (from Materials Project)
            \item Candidate features:
            \begin{itemize}
                \item Melting point (MP)
                \item Boiling point (MP)
                \item Atomic number (Z)
                \item Electronegativity ($\chi$)
                \item Atomic radius ($r$)
            \end{itemize}
            \item Question: Why these features?
            \item We will add some transformations of these inputs as well, i.e., the square and square root of the electronegativity and atomic radius.
    \end{itemize}
    \end{column}
    \begin{column}{0.3\textwidth}
        \begin{figure}
        \centering
        \includegraphics[width=\textwidth]{figures/elementdata.png}
    \end{figure}
    \end{column}
\end{columns}
\end{frame}

\begin{frame}[fragile]{Using pandas for easy data manipulation}
    \inputminted{python}{example_pandas_data_manipulation.py}
\end{frame}

\begin{frame}[fragile]{MLR in scikit-learn}
\inputminted{python}{example_sklearn_mlr.py}
\begin{itemize}
    \item Note that x should contain the features only - there is no need to add a 1 column for the intercept. By default, the parameter fit\_intercept in sklearn.linear\_model.LinearRegression is True. You can set it to False to do a MLR without intercept.
    \item Documentation: \href{https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html}{link}.
\end{itemize}

\end{frame}

\begin{frame}{Hypothesis Testing for Coefficients}
    \begin{itemize}
        \item To derive insights into a model, we often want to know which of the input parameters are the most relevant to the target.
        \item Under assumptions of the errors in $y$ follow a Gaussian distribution $N(0, \sigma^2)$, the errors in $\hat{\bm{\beta}}$ also have a Gaussian distribution $N(\beta, (\vec{X}^T \vec{X})^{-1} \sigma^2)$
        \item Hypothesis testing can be carried out for whether a particular $\beta_j$ is 0 using the following test statistic:
        \begin{equation*}
        t_j = \frac{\hat{\bm{\beta_j}}}{\sigma\sqrt{v_j}}
        \end{equation*}
        where $v_j$ is the $j$th diagonal element of $(\vec{X}^T \vec{X})^{-1}$. $t_j$ has a $t$ distribution with $N-p-1$ degrees of freedom (dof).
    \end{itemize}
\end{frame}

\begin{frame}{Hypothesis Testing for Groups of Coefficients}
    \begin{itemize}
        \item More often, we want to test groups of coefficient for significance. E.g., to the $k$ levels of a categorical variable.
        \item We will use the following $F$ statistic:
        \begin{equation*}
        F = \frac{(\mathrm{RSS}_0 - \mathrm{RSS}_1)/(p_1-p_0)}{\mathrm{RSS}_1/(N-p_1-1)}
        \end{equation*}
        where $\mathrm{RSS}_0$ is the RSS of the larger model with $p_0 + 1$ parameters and $\mathrm{RSS}_1$ is the RSS of the smaller model with $p_1 + 1$ parameters with $p_0 - p_1$ parameters set to zero. The $F$ statistic has a distribution of $F_{p_1-p_0,N-p_1-1}$.
    \end{itemize}
\end{frame}

\begin{frame}{Gauss-Markov Theorem}
    \begin{itemize}
        \item Consider the estimator $\hat{\theta}$ for a variable $\theta$.
        \begin{eqnarray*}
            \mathrm{MSE} & = & E(\hat{\theta} - \theta)^2 \\
            & = & \mathrm{var}(\hat{\theta}) + [E(\hat{\theta}) - \theta]^2
        \end{eqnarray*}
        \item The MSE can be broken down into the variance of the estimate itself and the square of the bias.
        \begin{block}{Gauss-Markov Theorem}
        The least squares estimator has the smallest variance among all linear \textit{unbiased} estimators.
        \end{block}
        \item However, there can be estimators that are biased with smaller MSE.
    \end{itemize}
\end{frame}

\section{Model selection}

\begin{frame}{Model selection}
    \Huge{\centerline{Model selection}}
\end{frame}

\begin{frame}{Model performance}
    \begin{itemize}
        \item We will take a brief digression into model assessment and selection before continuing on to other linear methods.
        \item Model performance is related to its performance on \textit{independent test data}, i.e., one cannot simply report a model's performance on training data alone.
        \item Note that this section is deliberately limited to high level concepts that are needed to continue further in exploration of linear methods. A more detailed discussion will be performed in later lectures.
    \end{itemize}
\end{frame}

\begin{frame}{Typical measures of model performance}
    \begin{itemize}
        \item Mean squared error (MSE):
            \begin{equation*}
                L(Y, \hat{f}(X)) = \frac{1}{N}\sum_{i=1}^N (y_i - f(x_i))^2
            \end{equation*}
        \item Mean absolute error (MAE):
            \begin{equation*}
                L(Y, \hat{f}(X)) = \frac{1}{N}\sum_{i=1}^N \left| y_i - f(x_i) \right|
            \end{equation*}
        \item Test error: $L$ over independent test set.
        \item Training error: $L$ over training set.
    \end{itemize}
\end{frame}

\begin{frame}{Training and test errors with model complexity}
    \begin{itemize}
        \item Model complexity increases as the number of parameters increases (e.g., number of independent variables in MLR).
        \item Training errors \textbf{always} decrease with increasing model complexity.
        \item However, test errors do not have a monotonic relationship with model complexity. Test errors are high when model complexity is too low (underfitting) or too high (overfitting).
    \end{itemize}
    \begin{figure}
        \centering
        \includegraphics[width=0.35\textwidth]{figures/fig7-1.pdf}
    \end{figure}
\end{frame}

\begin{frame}{Training, validation and test data}
    \begin{itemize}
        \item Model selection: estimating the performance of different models in order to choose the best one.
        \item Model assessment: having chosen a final model, estimating its prediction error (generalization error) on new data.
        \item Ideal data-rich situation: Divide data into three parts:
        \begin{itemize}
            \item Training set: For training the model.
            \item Validation set: For estimating prediction error to select the model.
            \item Test set: For assessing the generalization error of the final model.
        \end{itemize}
        \item Typical training:validation:test split is 50:25:25 or 80:10:10, or in very data-poor situations, maybe even 90:5:5.
        \item Note that at no point in the model fitting process should the test set be ``seen''.
    \end{itemize}
\end{frame}

\begin{frame}{$K$-fold cross validation (CV)}
    \begin{itemize}
        \item Simplest and most widely used approach for model validation.
        \item Data set is split into $K$ buckets (usually by random).
        \item Typical values of $K$ is 5 or 10. $K = N$ is known as ``leave-one-out'' CV.
        \begin{table}
        \begin{tabular}{|p{1.7cm}|p{1.7cm}|p{1.7cm}|p{1.7cm}|p{1.7cm}|}
            \hline
            \Large{Train} & \Large{Train} & \textcolor{red}{\Large{Validate}} & \Large{Train} & \Large{Train}\\
            \hline
        \end{tabular}
        \end{table}
        \item CV score is computed on the validate data set after training on the train data:
        \begin{equation*}
                CV(\hat{f}^{-k(i)},\alpha) = \frac{1}{N_{k(i)}}\sum_{i=1}^{N_{k(i)}} L(y_i, \hat{f}^{-k(i)}(x_i,\alpha))
        \end{equation*}
        \item assuming the $k^{th}$ data bucket has $N_{k(i)}$ data points and $\hat{f}^{-k(i)}$ refers to the model fitted with the $k^{th}$ data left out ($N-N_{k(i)}$ data in fitting).
    \end{itemize}
\end{frame}

\begin{frame}[fragile]{CV in scikit-learn}
\inputminted{python}{example_sklearn_cv.py}
\begin{itemize}
    \item Note that we have customized the KFold object passed to the cross\_validate method. The reason is that our element data is non-random by default. So we want to perform shuffling prior to doing the splits.
    \item Documentation: \href{https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_validate.html?highlight=cross_validate#sklearn.model_selection.cross_validate}{link}.
\end{itemize}
\end{frame}

\begin{frame}{Characteristics of the example materials dataset}
    \begin{itemize}
        \item Before proceeding further, let us try to tease out some aspects of the dataset.
        \item Quite clearly, there are correlations between some sets of variables.
        \item In other words, the input features are \textbf{non-orthonormal} with each other.
        \begin{figure}
            \centering
            \includegraphics[width=0.35\textwidth]{figures/pairplot-materialsdata.png}
            \includegraphics[width=0.45\textwidth]{figures/paircorrelations-materialsdata.png}
        \end{figure}
    \end{itemize}
\end{frame}

\section{Beyond least squares}

\begin{frame}{Beyond least squares}
    \Huge{\centerline{Beyond least squares}}
\end{frame}

\begin{frame}{Model selection}
    \begin{itemize}
        \item Often, we want to improve on the least squares model.
        \begin{itemize}
            \item To improve prediction accuracy by sacrificing some bias for reduced variance.
            \item To improve interpretability by reducing number of features or descriptors.
        \end{itemize}
        \item Three main approaches:
        \begin{enumerate}
            \item Subset selection
            \item Shrinkage methods
            \item Dimension reduction
        \end{enumerate}
    \end{itemize}
\end{frame}

\subsection{Subset selection}

\begin{frame}{Subset selection}
    Best subset selection
    \begin{itemize}
        \item Brute force approach.
        \item From $p$ parameters, find the subset of $k$ parameters that results in the smallest RSS.
        \item Combinatorially expensive for large $p$ and large $k$.
        \item Note that the best subset for a larger $k$ does not necessarily include the best subset for a smaller $k$.
    \end{itemize}
    Forward- or backward-stepwise selection
    \begin{itemize}
        \item Forward: Start with intercept, and iteratively add feature that most improves the fit.
        \item Backward: Start with full model, and sequentially deletes the feature with least impact on the fit.
    \end{itemize}
\end{frame}

\subsection{Shrinkage}

\begin{frame}{Shrinkage methods}
    \begin{itemize}
        \item Subset methods is discrete, i.e., retains/discards variables, and tends to exhibit high variance.
        \item Shrinkage methods are more continuous and do not suffer as much from high variability.
        \item Basic concept: instead of finding the parameters that minimizes the RSS only, we add a penalty term that penalizes more complex models, e.g., models with larger coefficients or larger number of coefficients. This ``shrinks'' the coefficients, in some cases, to 0.
    \end{itemize}
\end{frame}

\begin{frame}{Ridge regression ($L_2$ regularization)}
    \begin{equation*}
        \hat{\beta^{ridge}} = \argmin_\beta \left \{ \sum_{i=1}^N (y_i - \beta_0 - \sum_{j=1}^p \beta_j x_j)^2 + \lambda \sum_{j=1}^p \beta_j^2 \right \}
    \end{equation*}
    \begin{itemize}
        \item $\lambda \geq 0$ is the shrinkage parameter. The larger the $\lambda$, the greater the shrinkage.
        \item Also equivalent to:
        \begin{eqnarray*}
        \hat{\beta^{ridge}} = \argmin_\beta \sum_{i=1}^N (y_i - \beta_0 - \sum_{j=1}^p \beta_j x_j)^2\\
        \mathrm{subject~to} \sum_{j=1}^p \beta_j^2 \leq t
        \end{eqnarray*}
    \end{itemize}
\end{frame}

\begin{frame}{Ridge regression - Key details}
    \begin{itemize}
        \item Intercept ($\beta_0$) is not part of penalty term.
        \item Inputs should be scaled prior to performing ridge regression, typically by centering to the mean and scaling to unit variance:
        \begin{equation*}
            z_j = \frac{x_j - \mu_{x_j}}{s_{x_j}}
        \end{equation*}
    \end{itemize}
\end{frame}

\begin{frame}{LASSO ($L_1$ regularization)}
    \begin{equation*}
        \hat{\beta^{LASSO}} = \argmin_\beta \left \{ \sum_{i=1}^N (y_i - \beta_0 - \sum_{j=1}^p \beta_j x_j)^2 + \lambda \sum_{j=1}^p |\beta_j| \right \}
    \end{equation*}
    \begin{itemize}
        \item Least Absolute Shrinkage and Selection Operator
        \item $\lambda \geq 0$ is the shrinkage parameter. The larger the $\lambda$, the greater the shrinkage.
        \item Also equivalent to:
        \begin{eqnarray*}
        \hat{\beta^{LASSO}} = \argmin_\beta \sum_{i=1}^N (y_i - \beta_0 - \sum_{j=1}^p \beta_j x_j)^2\\
        \mathrm{subject~to} \sum_{j=1}^p |\beta_j| \leq t
        \end{eqnarray*}
    \end{itemize}
\end{frame}

\begin{frame}{LASSO regression - Key details}
    \begin{itemize}
        \item Intercept ($\beta_0$) is not part of penalty term.
        \item Inputs should be scaled prior to performing lasso regression, just as in ridge regression.
    \end{itemize}
\end{frame}

\begin{frame}{Subset vs ridge vs LASSO}
    \begin{itemize}
        \item Consider a set of orthonormal features.
        \begin{itemize}
            \item Ridge: proportional shrinkage. No coefficients are set to zero.
            \item LASSO: ``soft'' thresholding. Translates coefficients by a factor, truncating at zero.
            \item Best-subset: ``hard'' thresholding. Drops all coefficients below a certain threshold.
        \end{itemize}
    \end{itemize}
    \begin{figure}
    \begin{tikzpicture}[scale=0.35]
	\begin{axis}[title=Best subset, grid=major]
	\addplot[color=black, dashed] coordinates {
		(-9,-9)
		(9,9)
	};
	\addplot[color=red] coordinates {
		(-9,-9)
		(-3,-3)
		(-3,0)
		(3,0)
		(3,3)
		(9,9)
	};
	\end{axis}
	\end{tikzpicture}
    \begin{tikzpicture}[scale=0.35]
	\begin{axis}[title=Ridge, grid=major]
	\addplot[color=black, dashed] coordinates {
		(-9,-9)
		(9,9)
	};
	\addplot[color=red] coordinates {
		(-9,-7)
		(9,7)
	};
	\end{axis}
    \end{tikzpicture}
    \begin{tikzpicture}[scale=0.35]
	\begin{axis}[title=LASSO, grid=major]
	\addplot[color=black, dashed] coordinates {
		(-9,-9)
		(9,9)
	};
	\addplot[color=red] coordinates {
		(-9,-7)
		(-2,0)
		(2,0)
		(9,7)
	};
	\end{axis}
    \end{tikzpicture}
        \includegraphics[width=0.35\textwidth]{figures/fig3-14.pdf}
    \end{figure}
\end{frame}

\begin{frame}{Other variants of shrinkage methods}
    \begin{itemize}
        \item Elastic net penalty:
        \begin{eqnarray*}
            \lambda \left( \alpha \sum_{j=1}^p \beta_j^2+ (1-\alpha) \sum_{j=1}^p |\beta_j| \right)
        \end{eqnarray*}
        \item Least angle regression
    \end{itemize}
\end{frame}

\subsection{Derived input directions}

\begin{frame}{Derived input directions}
    \begin{itemize}
        \item General concept: transforms input $\vec{X}$ into a smaller subset of $\vec{z_m}$ and regress on $\vec{z_m}$
        \item Principal component regression:
        \begin{itemize}
            \item Transform non-orthonormal features into orthonormal directions using Principal Component Analysis (PCA).
            \item Choose $M$ directions that have the highest eigenvalues (explains the most variance) and discards the rest.
            \item Will revisit at a later lecture.
        \end{itemize}
    \end{itemize}
\end{frame}

\begin{frame}{Partial Least Squares (PLS)}
    \begin{itemize}
        \item Algorithm:
        \begin{enumerate}
            \item Compute $\phi_{1j} = <\vec{x_j}, \vec{y}>$ for each $j$.
            \item First transformed direction $\vec{z_1} = \sum_j \phi_{1j} \vec{x_j}$, i.e., each direction is weighted by strength of effect on $\vec{y}$.
            \item Regress $\vec{y}$ on $\vec{z_1}$ to obtain $\theta_1$, orthogonalize $\vec{x_1}, ... \vec{x_p}$ wrt $\vec{z_1}$ via $x_j' = x_j - \frac{<\vec{z_1}, \vec{x_j}>}{<\vec{z_1}, \vec{z_1}>}\vec{z_1}$.
            \item Repeat until $M \leq p$ coefficients are obtained.
        \end{enumerate}
        \item Finds directions with high variance and high correlation with response.
    \end{itemize}
\end{frame}

\section{Extending linear methods}

\begin{frame}{Preliminaries}
    \begin{itemize}
        \item It is highly unlikely that the true function $f(X)$ is linear in $X$.
        \item In some cases, linearity is a reasonable assumption, e.g., a first order Taylor series expansion:
        \begin{equation*}
            f(x) = f(a) + f'(a) (x-a) + f''(a) \frac{(x-a)^2}{2!} + f'''(a) \frac{(x-a)^3}{3!} + ...
        \end{equation*}
        \item Examples where this is used in materials science - linear elasticity (Hooke's law), etc.
        \item More frequently, we perform a transformation of inputs to create a linear basis expansion.
    \end{itemize}
\end{frame}

\section{Transformation of inputs}

\begin{frame}{General concept}
    \begin{itemize}
        \item Express:
        \begin{equation*}
            f(X) = \sum_{m=1}^M \beta_m h_m(X)
        \end{equation*}
        where $h_m$ is the $m^{th}$ transformation of $X$.
        \item This is known as a linear basis expansion in $X$.
        \item The key lies in choice of the basis functions $h_m$.
    \end{itemize}
\end{frame}

\begin{frame}{Examples of basis expansions}
    \begin{itemize}
        \item $h_m(X) = X_j^2, h_m(X) = X_i X_j$
        \begin{itemize}
            \item Polynomial expansion to higher-order Taylor series terms.
            \item No. of terms increases exponentially with degree of polynomial. For $p$ variables, we have $O(p^2)$ square and cross-product terms in a quadratic model. For a degree $d$ polynomial, we have $O(p^d)$.
        \end{itemize}
        \item $h_m(X) = log(X_j), sqrt(X_j), exp(i X_j)$: non-linear transformations in $X$.
        \item $h_m(X) = I(L_m \leq X_k < U_m)$: Piece-wise division of regions of $X$. E.g., cubic splines.
        \item $h_m(X) = RBF(||X-X_m||)$: radial basis function, e.g., Gaussian.
        \item Typically, basis functions are used simply to allow a more flexible representation of the data. The basis functions can span a very large (sometimes infinite) set, from which a selection has to be made:
        \begin{itemize}
            \item Restriction - Truncate the choice of basis functions using some criteria.
            \item Selection - Choose basis functions that contribute significantly to the fit.
            \item Regularization - Use the whole and/or very large subset and apply regularization techniques (e.g., ridge or LASSO) to restrict coefficients.
        \end{itemize}
    \end{itemize}
\end{frame}

\begin{frame}{Linearization from physical laws}
    \begin{itemize}
        \item Arrhenius law:
        \begin{equation*}
            r = A \exp(-\frac{E_a}{RT}) \longrightarrow log(r) = log(A) - \frac{E_a}{RT}
        \end{equation*}
        \item Ising model:
        \begin{equation*}
            H(\sigma) = - \sum_{<i, j>} J_{ij}\sigma_i \sigma_j - \mu \sum_j h_j \sigma_j
        \end{equation*}
        \begin{figure}
            \centering
            \includegraphics[width=0.3\textwidth]{figures/ising.png}
        \end{figure}
    \end{itemize}
\end{frame}

\begin{frame}{Compressive sensing for cluster expansions}
    \begin{itemize}
        \item Cluster expansion of energy on lattice points:
        \begin{equation*}
            H(\sigma) = E_0 + \sum_f J_f \prod_f(\sigma)
        \end{equation*}
        \item $\sigma$ is the vector representing occupation of lattice sites, $\prod_f$ are the cluster basis functions, $J_f$ are effective cluster interactions (ECIs).
        \item Compressive sensing: essentially a LASSO to solve for ECIs.\cite{nelsonCompressiveSensingParadigm2013}
        \begin{figure}
            \centering
            \includegraphics[width=0.3\textwidth]{figures/numberofclusters.png}
            \includegraphics[width=0.3\textwidth]{figures/ecifit.pdf}
        \end{figure}
    \end{itemize}
\end{frame}

\section{Piece-wise polynomials}

\begin{frame}{Piecewise polynomials}
    \begin{equation*}
        h_1(X) = I(X < \xi_1), h_2(X) = I(\xi_1 \leq X < \xi_2), h_3(X) = I(X \geq \xi_2)
    \end{equation*}
    \begin{columns}
    \column{0.3\textwidth}
    \begin{figure}
        \centering
        \includegraphics[width=\textwidth]{figures/piecewisefits.pdf}
    \end{figure}
    \column{0.7\textwidth}
    Parameters:
    \begin{itemize}
        \item No. of knots
        \item Order of polynomial
        \item Continuity at knots (value, first derivative, second derivative, etc.). For a polynomial of order $N$, we usually want all derivatives $< N$ to be continuous.
    \end{itemize}
    \end{columns}
\end{frame}

\begin{frame}{Cubic splines}
    \begin{columns}
    \column{0.5\textwidth}
    \begin{figure}
        \centering
        \includegraphics[width=0.7\textwidth]{figures/piecewisecubic.pdf}
    \end{figure}
    \column{0.5\textwidth}
    \begin{itemize}
        \item Probably the most commonly used.
        \item Continuous 1st and 2nd derivatives.
        \item Natural cubic spline: polynomial is linear beyond boundaries.
        \item Smoothing spline: Use  regularization to control complexity:
        \begin{eqnarray*}
            RSS(f, \lambda) = \sum_{i=1}^N \{y_i - f(x_i)\} ^ 2 \\
            + \lambda \int \{f''(t)\}^2 dt
        \end{eqnarray*}
    \end{itemize}
    \end{columns}
\end{frame}

\begin{frame}{Examples of cubic spline fitting}
    \begin{itemize}
        \item Spline-based Modified Embedded Atom Method (MEAM)
        \begin{eqnarray*}
            E = \sum_{i <j} \phi(r_{ij}) + \sum_i U(n_i), \\
            n_i = \sum_j \rho(r_{ij}) + \sum_{i < k, j,k!=i} f(r_{ij}) f(r_{ik})g[cos(\theta_{jik})]
        \end{eqnarray*}
        where $\phi$, $U$, $\rho$, $f$ and $g$ can be approximated by cubic splines.
        \begin{figure}
            \centering
            \includegraphics[width=0.3\textwidth]{figures/meam-phi.png}
            \includegraphics[width=0.3\textwidth]{figures/meam-u.png}

        \end{figure}
    \end{itemize}
\end{frame}

\begin{frame}[fragile]{Demo: Cubic spline fitting in scipy}
    \inputminted{python}{example_sklearn_spline.py}
\end{frame}

\section{Gaussian basis functions}

\begin{frame}{Gaussian basis functions}
    \begin{equation*}
        h_m(x) = \exp(-k(x - x_m) ^ 2)
    \end{equation*}
    \begin{itemize}
        \item Gaussian functions centered at $x_m$.
        \item Other similar types of functions include Lorentzian ($h_m(x) = \frac{1}{1 + kx^2}$), Gaussian-Lorentzian, Voigtian, Pearson type IV, and beta profiles.
    \end{itemize}
\end{frame}

\begin{frame}{Example: Rietveld refinement}
\begin{figure}
    \centering
    \includegraphics[width=0.45\textwidth]{figures/rietveld.pdf}
    \caption{Neutron powder diffraction diagram of \ce{CaUO4}}
\end{figure}
    \begin{itemize}
        \item Least squares fitting of theoretical line profile to match a measured diffraction pattern (e.g., X-ray, neutron).\cite{rietveldProfileRefinementMethod1969}
    \end{itemize}
\end{frame}

\begin{frame}{Example: Rietveld refinement, contd.}
    \begin{itemize}
        \item Peak shape function:
        \begin{equation*}
            PSF(\theta) = \Omega(\theta) \otimes \Lambda(\theta) \otimes \Psi(\theta) + b(\theta)
        \end{equation*}
        \item $\Omega$: Instrument broadening, $\Lambda$: Wavelength dispersion, $\Psi$: Specimen function.
        \item For single phase, minimize:
        \begin{equation*}
            \Phi = \sum_{i=1}^N w_i \left ( Y_i^{obs} - \left ( b_i + K \sum_{j=1}^m I_jy_j(x_j)\right )\right )^2
        \end{equation*}
        \item where $y_j(x_j)$ is typically a pseudo-Voigt (mix of Gaussian and Lorentizan function) function.
        \item Note that the background ($b_i$) holds no useful structural information and should be minimized in experiments.
    \end{itemize}
\end{frame}

\section{Wavelet and Fourier basis functions}

\begin{frame}
\frametitle{Wavelet smoothing}
\begin{columns}
\column{0.4\textwidth}
\begin{figure}
    \centering
    \includegraphics[width=\textwidth]{figures/wavelets.pdf}
\end{figure}
\column{0.6\textwidth}
    \begin{itemize}
        \item Complete orthonormal basis
        \item Shrink and select toward \textbf{sparse} representation.
        \item Able to represent both time and frequency localization efficiently (Fourier basis can only do frequency localization).
    \end{itemize}
\end{columns}
\end{frame}

\begin{frame}{Example: NMR Spectroscopy}
\begin{columns}
\column{0.4\textwidth}
\begin{figure}
    \centering
    \includegraphics[width=0.45\textwidth]{figures/nmrwavelet.pdf}
    \caption{Subtraction of a large spectral line: (top) the original spectrum of polyethylene, (bottom) reconstructed spectrum after removal of \ce{CH2} peak.\cite{baracheContinuousWaveletTransform1997}}
\end{figure}
\column{0.6\textwidth}
    Applications:
    \begin{itemize}
        \item Suppression of large unwanted spectral line (left).
        \item Rephasing spectrum perturbed by time-dependent magnetic field.
        \item Noise filtering
        \item Detecting phases in a mixture
    \end{itemize}
\end{columns}
\end{frame}

\begin{frame}{Example: Fourier transform for analysis of extended X-ray absorption fine structure (EXAFS)}

    \begin{columns}
    \column{0.2\textwidth}
    \begin{figure}
        \centering
    \includegraphics[width=\textwidth]{figures/EXAFS-1.png}
    \includegraphics[width=\textwidth]{figures/EXAFS-2.png}
    \includegraphics[width=\textwidth]{figures/EXAFS-3.png}
    \end{figure}
    \column{0.77\textwidth}
    \begin{itemize}
        \item (a) The extended edge (orange part) contains information of atom chemical environment.
        \item (b) Subtract the background, convert energy to k-space unit, and multiply the normalized intensity by $k^2$
        \item (c) Fourier transform $k$-space information to real space and obtain the first shell bond length.
    \end{itemize}
    \end{columns}
\end{frame}

\begin{frame}[allowframebreaks]{Bibliography}
    \bibliographystyle{unsrt}
    \bibliography{refs}
\end{frame}

\begin{frame}
    \Huge{\centerline{The End}}
\end{frame}

\end{document}


\chapter{Conclusion}\label{cha:conclusion}

In this thesis, the graph convolutional neural network presented in Gasse et al. (2019) \cite{gasse2019exact} was iteratively ablated. The ablations resulted in five models, among which two were graph convolutional neural networks and three were pure multi-layer perceptrons. The \gls{GCNN} models used the bipartite graph nature of the constraint-variable relationship, while the \gls{MLP}s only used the features of the candidate variables. The models were trained, tested, and evaluated on generated \gls{MILP} problems from the problem classes combinatorial auctions and set covering. All resulting models were tested for accuracy on predicting the optimal branching variable according to the strong branching algorithm. The efficiency of the \gls{ML}-enhanced solvers were evaluated by running the models on both a \gls{GPU} and \gls{CPU}. These experiments were chosen in order to gain insight into the model and help future researchers make informed choices on \gls{ML} model selection.

% Ecole
The experiments were implemented in the new framework \textit{\gls{Ecole}} \cite{prouvost2020ecole}. The framework provides an interface for the \gls{BnB} solver \gls{SCIP}, inspired by \textit{OpenAI Gym} \cite{brockman2016openai}.
This thesis is the first article to use \gls{Ecole} except for the introductory papers by Provoust et al. (2020) \cite{prouvost2020ecole} and Cappart et al. (2021) \cite{cappart2021combinatorial}. The framework was evaluated to be useful, especially as it is improved upon in the future.

% accuracy ...
The accuracy of the models consistently decreased as layers were removed from the original model. There was a significant loss of accuracy after removing the graph convolutional component as well as after removing all hidden layers of the model. Both problem sets showed the same tendency, though the degradation of accuracy was more dramatic for the set covering problems, where there is a larger number of constraints.
%The iterative ablations showed a consistent decrease in accuracy for both problem classes. The notable decreases in accuracy occurred after removing the graph convolutional modules (resulting in a pure Multi-Layer Perceptron) and after all of the hidden layers of the model were removed, resulting in a linear model. A larger loss of accuracy was found for the set covering problems than the combinatorial auction problems after the removal of the graph convolutions. This is assumed to be because of the higher number of constraints in the set covering problems compared to the combinatorial auction problems.

% efficiency ...
The computation time per variable decision also decreased with the ablations. Significant reductions in time per node were found after removing the graph convolutions and after removing all hidden layers. When the \gls{SCIP} solver was run on test problems by performing the variable selection with the learned models, all models showed competitive efficiency with a selection of classical branching algorithms. When the models were run on the \gls{CPU}, the more complex models suffered a large loss of efficiency. This particularly affected the models containing graph convolutions. Results were also indicative of the problem formulation being relevant for the viability of running \gls{GCNN} models on the \gls{CPU}.
%All ablations resulted in decreased computation time per variable branching decision. All models were competitive with the classical branching strategies (Full Strong branching, Pseudo-cost branching, Reliability Pseudo-cost branching) when run on the \gls{GPU}. On the \gls{CPU}, there was a significant decrease in performance for the models containing graph convolutions. This performance reduction was greater for the set covering problems, indicating that the number of constraints might exacerbate problems with running graph convolutional models on restricted hardware. Reducing the number of hidden layers did not impact the computation time per node in particular, and is therefore not considered a viable alternative for reducing the computation time.

% Implications ...
The main implications of the results are the importance of considering the hardware the \gls{ML} enhanced solver will be deployed on, as well as the problem types the models are tested on. The relation between the running times of the different models on both hardware implied non-trivial relations, which urges caution for future attempts at developing \gls{ML} models for this purpose. The results in this thesis as well as the project \textit{Multi-Layer Perceptrons for Branching in Mixed-Integer Linear Programming} (2020) and the article by Gupta et al. (2020) \cite{gupta2020hybrid} stipulate a shift toward less computationally complex models with richer input features (observation functions). These models will be more universally applicable and predictable on the various hardware that will run \gls{BnB} algorithms. In addition, the most recent attempt at learning to branch by Zarpellon et al. (2020) \cite{zarpellon2020parameterizing} is consistent with this observation by training models that generalize across problem classes.

% Future work ...
Future work should take into account the implications of this thesis in terms of both hardware and problem class. The analysis of how problem formulations can be detrimental to model performance is also highly relevant and should be taken into consideration when presenting data-driven methods with the implication of universally useful. Following the trend of the last few years, models pre-trained with imitation learning and improved with reinforcement learning can provide further advances in the field of \gls{ML} enhanced \gls{BnB}. These models will yield insights into the greater topic of machine-created algorithms, and may revolutionize how the hardest computational problems are solved in the future.

%\begin{chapquote}{Thomas Aquinas, \textit{Summa Theologica}}
%``Quod potest compleri per pauciora principia, non fit per plura.\footnote{It is superfluous %to suppose that what can be accounted for by a few principles has been produced by many.}''
%\end{chapquote}

% Diskusjonskapittel
% Rød tråd: Accuracy vs. efficiency


\chapter{Conclusion and Outlook}
\label{chap:conclusion}

\section{Conclusion}

    In this thesis, existing procedures have been combined, such that a new penalizing metric is computed for provided graphs.
    This new metric compensates the popularity of routes' edges in the graph, because these higher popularities causes to overload a graph's underlying network, when many routes are computed.
    This is achieved with the help of an existing procedure from~\cite{barth:alternative_multicriteria_routes}, that interpretes optimum paths found by Dijkstra in the underlying cost-space.
    With help of this procedure, several alternative routes for a provided pair of source and destination are enumerated.
    Because of the new metric, the number of found alternative routes is increased significantly while a user-provided tolerance (with respect to travel-time) holds.

    Advantages of the presented process to create the new metric is the intuitive idea behind it.
    While other approaches for finding alternative routes lack in performance, parameter-tuning, complexity or diversity of found alternative routes, the method in this thesis can be summarized in two iterations.
    The first iteration updates the graph's new metric by favoring popular routes.
    The second and last iteration considers this preference negatively and updates the graph's new metric, respectively trying to avoid these popular routes.
    In the end, the average of both updates results in a penalizing metric, that can be used in existing routing-algorithms for graphs of multidimensional metrics.
    The new metric leads to more spreaded routes, as shown in this thesis with street-networks from OpenStreetMap.

\section{Future work}

    Remaining issues with the procedure are mainly performance-related, although preprocessing-methods as contraction-hierarchies are already used to speed the route-queries significantly up.

    One performance-improvement would bring a better choice of sources and destinations for the user-provided set of \glspl{stpair}.
    Yet, sources and destinations are chosen \gls{uar}\ from the graph's vertices.
    This requires a high number of \glspl{stpair} to actually overload the graph.
    Heuristics, as one described in~\cite{bakillah:population_from_osm} to approximate the population based on OpenStreetMap-data, may help to weight the vertices in a more realistic way, which is usually less \gls{uar}\ when thinking of rush-hour-scenarios.

    One slighter performance-improvement might be the flattening of shortcuts, that are created by the contraction-hierarchies, right after all found routes are collected, not right after a route is found.
    Another improvement refers the implemented graph-parser, that assumes, that drivers are okay with choosing even dirt tracks for their paths, leading to more edges in the graph.
    As the tolerance for travel-time holds, this shouldn't affect the spread a lot, but the routing-algorithms (especially the contraction-hierarchies) are much affected by the number of edges.
    Further, the contraction-hierarchies contracts just almost all vertices, which takes much less time than actually creating the new metric.
    A higher contraction-rate to reduce the creation-time might improve the total runtime here.

    Although the results from the shown experiments are satisfying, only one metric is created.
    Maybe, the results would be even better with several artificial penalization-metrics being created, since the averaging-approach behaves accordingly to this idea.
    In addition, once a graph is balanced, the graph's balancing-progress is more or less fixed.
    Here, it would be more practical to have the graph being balanced right after each \gls{stpair} is processed.
    Despite the metric-update, that relies on the mean over all workloads, a remaining issue would be the adaption of contraction-hierarchies.
    This could be done by a replacement-strategy, where the last contracted graph is used for answering incoming queries and a copy of it is further balanced in the meanwhile.
    At some point, the contracted graph, that is answering incoming queries, is replaced by the new graph, after this is contracted.
    This would fit better for use-cases in the real world, especially with large capacities, as Google can provide.
    Google additionally has enough user-queries to produce valid workloads.

    Another aspect is the influence of crossroads on travel-time.
    The tolerance holds for travel-time, which is based solely on speed-limits and distances, not on behaviour or circumstances at crossroads.
    Here, the hop-distance would be important, too.
    On the other hand, the idea behind all of this is the reduction of occuring traffic-jams.
    Probably, circumstances at crossroads might be the better option than standing in a traffic-jam.


\documentclass{article}%
\usepackage[T1]{fontenc}%
\usepackage[utf8]{inputenc}%
\usepackage{lmodern}%
\usepackage{textcomp}%
\usepackage{lastpage}%
\usepackage{amsmath}%
%
\title{2019 AIME II Problems}%
\author{MAA}%
\date{\today}%
%
\begin{document}%
\normalsize%
\maketitle%
\section{Problems}%
\label{sec:Problems}%
\begin{enumerate}%
\item%
Two different points, $C$ and $D$, lie on the same side of line $AB$ so that $\triangle ABC$ and $\triangle BAD$ are congruent with $AB = 9$, $BC=AD=10$, and $CA=DB=17$. The intersection of these two triangular regions has area $\frac mn$, where $m$ and $n$ are relatively prime positive integers. Find $m+n$.
%
\item%
Lily pads $1,2,3,\ldots$ lie in a row on a pond. A frog makes a sequence of jumps starting on pad $1$. From any pad $k$ the frog jumps to either pad $k+1$ or pad $k+2$ chosen randomly with probability $\frac{1}{2}$ and independently of other jumps. The probability that the frog visits pad $7$ is $\frac{p}{q}$, where $p$ and $q$ are relatively prime positive integers. Find $p+q$.
%
\item%
Find the number of $7$-tuples of positive integers $(a,b,c,d,e,f,g)$ that satisfy the following systems of equations:
\begin{align*} abc&=70,\\ cde&=71,\\ efg&=72. \end{align*}
%
\item%
A standard six-sided fair die is rolled four times. The probability that the product of all four numbers rolled is a perfect square is $\frac{m}{n}$, where $m$ and $n$ are relatively prime positive integers. Find $m+n$.
%
\item%
Four ambassadors and one advisor for each of them are to be seated at a round table with $12$ chairs numbered in order $1$ to $12$. Each ambassador must sit in an even-numbered chair. Each advisor must sit in a chair adjacent to his or her ambassador. There are $N$ ways for the $8$ people to be seated at the table under these conditions. Find the remainder when $N$ is divided by $1000$.
%
\item%
In a Martian civilization, all logarithms whose bases are not specified as assumed to be base $b$, for some fixed $b\ge2$. A Martian student writes down
\[3\log(\sqrt{x}\log x)=56\]
\[\log_{\log x}(x)=54\]
and finds that this system of equations has a single real number solution $x>1$. Find $b$.
%
\item%
Triangle $ABC$ has side lengths $AB=120,BC=220$, and $AC=180$. Lines $\ell_A,\ell_B$, and $\ell_C$ are drawn parallel to $\overline{BC},\overline{AC}$, and $\overline{AB}$, respectively, such that the intersections of $\ell_A,\ell_B$, and $\ell_C$ with the interior of $\triangle ABC$ are segments of lengths $55,45$, and $15$, respectively. Find the perimeter of the triangle whose sides lie on lines $\ell_A,\ell_B$, and $\ell_C$.
%
\item%
The polynomial $f(z)=az^{2018}+bz^{2017}+cz^{2016}$ has real coefficients not exceeding $2019$, and $f\left(\frac{1+\sqrt{3}i}{2}\right)=2015+2019\sqrt{3}i$. Find the remainder when $f(1)$ is divided by $1000$.
%
\item%
Call a positive integer $n$ $k$-pretty if $n$ has exactly $k$ positive divisors and $n$ is divisible by $k$. For example, $18$ is $6$-pretty. Let $S$ be the sum of positive integers less than $2019$ that are $20$-pretty. Find $\frac{S}{20}$.
%
\item%
There is a unique angle $\theta$ between $0^{\circ}$ and $90^{\circ}$ such that for nonnegative integers $n$, the value of $\tan{\left(2^{n}\theta\right)}$ is positive when $n$ is a multiple of $3$, and negative otherwise. The degree measure of $\theta$ is $\frac{p}{q}$, where $p$ and $q$ are relatively prime integers. Find $p+q$.
%
\item%
Triangle $ABC$ has side lengths $AB=7, BC=8,$ and $CA=9.$ Circle $\omega_1$ passes through $B$ and is tangent to line $AC$ at $A.$ Circle $\omega_2$ passes through $C$ and is tangent to line $AB$ at $A.$ Let $K$ be the intersection of circles $\omega_1$ and $\omega_2$ not equal to $A.$ Then $AK=\frac mn,$ where $m$ and $n$ are relatively prime positive integers. Find $m+n.$
%
\item%
For $n \ge 1$ call a finite sequence $(a_1, a_2 \ldots a_n)$ of positive integers progressive if $a_i < a_{i+1}$ and $a_i$ divides $a_{i+1}$ for all $1 \le i \le n-1$. Find the number of progressive sequences such that the sum of the terms in the sequence is equal to $360$.
%
\item%
Regular octagon $A_1A_2A_3A_4A_5A_6A_7A_8$ is inscribed in a circle of area $1.$ Point $P$ lies inside the circle so that the region bounded by $\overline{PA_1},\overline{PA_2},$ and the minor arc $\widehat{A_1A_2}$ of the circle has area $\frac{1}{7},$ while the region bounded by $\overline{PA_3},\overline{PA_4},$ and the minor arc $\widehat{A_3A_4}$ of the circle has area $\frac{1}{9}.$ There is a positive integer $n$ such that the area of the region bounded by $\overline{PA_6},\overline{PA_7},$ and the minor arc $\widehat{A_6A_7}$ of the circle is equal to $\frac{1}{8}-\frac{\sqrt2}{n}.$ Find $n.$
%
\item%
Find the sum of all positive integers $n$ such that, given an unlimited supply of stamps of denominations $5,n,$ and $n+1$ cents, $91$ cents is the greatest postage that cannot be formed.
%
\item%
In acute triangle $ABC$ points $P$ and $Q$ are the feet of the perpendiculars from $C$ to $\overline{AB}$ and from $B$ to $\overline{AC}$, respectively. Line $PQ$ intersects the circumcircle of $\triangle ABC$ in two distinct points, $X$ and $Y$. Suppose $XP=10$, $PQ=25$, and $QY=15$. The value of $AB\cdot AC$ can be written in the form $m\sqrt n$ where $m$ and $n$ are positive integers, and $n$ is not divisible by the square of any prime. Find $m+n$.
%
\end{enumerate}

%
\end{document}