\chapter{Introduction}
A common problem in applied statistics is that one has an independent
variable or outcome $Y$ and various dependent variable or covariates
$X_1,\dots,X_p$. One usually observes these variables for various
``subjects''.

One may be interested in various things: What effects do the
covariates have on the outcome? How well can we describe these
effects? Can we predict the outcome using the
covariates?, etc..

Statisticians usually assume that $Y$ and the $X$'s are random
variables. Then one can summarize the above question by asking: what
is $\E[ Y | X_1,\dots,X_p]$? We usually call $f(X_1,\dots,X_p) = \E[Y
| X_1,\dots, X_p]$ the {\it regression function}.

It should be noted that for some designed experiments it does not make
sense to assume the $X$ are random variables. In this case we usually
assume we have ``design points'' $x_{1i}, \dots, x_{pi}, i=1,\dots,n$
and non-IID observations $Y_1,\dots,Y_n$ for each design
point. In most cases, the
theory for both these cases is very similar if not the same. These are
called the {\it random design model} and {\it fixed design model} respectively.

How do we learn about $E[Y|X_1,\dots,X_p]$?

A common procedure is linear regression. One typically assumes
\[
\E[Y | X_1, \dots, X_p] = \sum_{j=1}^p X_j \beta_j.
\]
Assuming that the conditional probability of $Y$ is normal is quite
common. However, if the range of this expectation is not continuous one can
generalize to:
\[
g ( \E[Y | X_1, \dots, X_p] ) = \sum_{j=1}^p X_j \beta_j
\]
with $g$ a link function with real-valued range. It is typical to
assume the conditional distribution of $Y$  is part of an exponential
family, e.g. binomial, Poisson, gamma, etc.... Many times the link
function is chosen for mathematical convenience.

These model have the convenience that the parameters $\beta$ usually
have direct interpretation with scientific meaning. For example
$\beta=5$ may mean that a man that is one inch higher than another is
expected to weigh 5 more pounds.

Another advantage is that, once an appropriate model is in place, the
estimates have many desirable properties.

A drawback of these models is that they are quite
restrictive. Linearity and additivity are two very strong
assumptions. This may have practical consequences. For example, by
assuming  linearity one may never notice that a covariate
has an effect that increases and then decreases. We will see various
example of this in class.

In this class we will
\begin{itemize}

\item Start by introducing various smoothers useful for smoothing
  scatter plots $\{ (X_i, Y_i), i=1,\dots,n \}$ where both $X$ and $Y$
  are continuous variables.
\item Set down precise models and outline the proofs of asymptotic
  results.
\item Introduce local regression (loess).
\item Examine spline models and some of the theory behind splines.
\item Some smoothers are more flexible than others. However with
  flexibility comes variance. We will talk about the bias-variance
  trade-off and how one can use resampling methods to estimate bias
  and variance.
\item After explaining all these smoothers we will make a connection
  between them. We will also make connections to other statistical
  procedures.
\item We will examine the case were one has many covariates. One can
  relax the linearity assumption,
  assume additivity and use additive models. One can also forget the
  additivity assumption and
  use regression trees.
\item After all this we will be ready to consider the case where $Y$
  is not necessarily continuous. We will generalize to this case and
  look at Generalized Additive Models and Local Likelihood.
\item While examining all these subjects we will be considering
  various models for one data set. We will briefly discuss techniques
  that can be used to aid in the choice of such models.
\item Finally we will look at a brief introduction of times series
  analysis.
\end{itemize}

By relaxing assumptions we loose some of the nice properties of
estimates. There is an on going debate about specification
vs. estimation.

We will begin the class talking about the case were the regression
function $f$ will depend on a
single, real-valued predictor $X$ ranging over some possibly infinite
interval of the real line, $I \subset \mathbb{R}$. Therefore, the
(mean) dependence of $Y$ on $X$ is given by
\begin{equation}
\label{regfunc}
f(x) = \E[Y|X=x], x \in I \subset \mathbb{R}.
\end{equation}

Sometimes, the need to estimate $f$ arises when investigators have to
decide among various explanations for a physical phenomenon,
and existing subject-knowledge or scientific theory says nothing about
$f$. In this case we collect data to see what it says. Exhibiting some
aspect of $f$ may then imply the confirmation or
revision of a given theory.

The data to
support  such investigations are typically a set of $n$ paired
observations $(\bX_1,Y_1),\dots,(\bX_n,Y_n)$. These can be either a
random sample of  the joint distribution of $(\bX,Y)$ as is the case
for observational studies, or fixed input values $\{ \bx_i \}$, arising perhaps from a designed experiment.

So once we have the data what do we do?

If we are going to ``model'' (\ref{regfunc}), we gain insight into the
important features of the relationship
between $Y$ and $\bX$ by entertaining various descriptions of or
models for $f$. Through this exercise we might:
\begin{itemize}
\item  identify the width and height of peaks
\item  explore the overall shape of $f$ in some neighborhood
\item find  areas of sharp increase or regions exhibiting little
  curvature.
\end{itemize}

The first three chapters of the class deal with this problem. We will
then move on to the case where we have many covariates, then cases
where the
expectation needs to be transformed, and various other generalization.

Through out all these subjects we will be talking about finite sample
theory, asymptotics, practical aspects, and computational
consideration.


\documentclass[../PHYS306Notes.tex]{subfiles}

\begin{document}
\section{Lecture 30}
\subsection{Lecture Notes - Continuum Mechanics}
\subsubsection{Setting Up The Continuum Limit}
Recall HW5, where we considered an infinitely long chain of particles. Then, defining $u_s$ to be the displacement of the $s$th photon from the equilibrium position, we solved (by Newton's law):
\[m\ddot{u}_s = m\omega_0^2(u_{s+1} + u_{s-1} - 2u_s)\]
Where $\omega_0^2 = \frac{k}{m}$. The forces depend on the neighbouring interactions and displacements from equilibrium. Now, it becomes a bit of a subtle issue to how we take limits to infinity without getting things to blow up.
\newline We start with $N$ masses which are separated by a distance $l$ (chain of SHOs).
\begin{itemize}
    \item  We want to take $N \rightarrow \infty$ and $l \rightarrow 0$ at the same time. We do this in such a way that the total length $L = Nl$ of the string remains constant.
    \item  As we increase the number of particles, they also get lighter, so we take $m \rightarrow 0$ and $l \rightarrow 0$ such that the line mass density $\lambda = \frac{m}{l}$ of the string also is a constant.
\end{itemize}
Question: When we double the number of masses and reduce the equilibrium length by a factor of 2, what must happen with the spring constant $k$?
\begin{center}
    \includegraphics[scale=1]{Lecture-30/l30-img1.png}
\end{center}
\begin{s}
Since $F = kx$, if $x \mapsto \frac{x}{2}$, then $k \mapsto 2k$ in order to give the same force between the particles as the original configuration.
\end{s}
\begin{itemize}
    \item So another condition on our limit; we will take $k \rightarrow \infty$ and $l \rightarrow 0$ with $kl$ held constant.
\end{itemize}
The idea is we will go from the position of a single particle $u_s(t)$ and take it to the continuum limit of $u(x, t)$, now considering a field.
\begin{center}
    \includegraphics[scale=1]{Lecture-30/l30-img2.png}
\end{center}

\subsubsection{Deriving the Wave Equation}
Replacing the discrete positions with a continuous variable in the Newton's law equation we had in the section above, we have:
\[\dpd[2]{u(x, t)}{t} = \omega_0^2\left(u(x + l, t) + u(x - l, t) - 2u(x, t)\right)\]
But there is an issue; $\omega_0^2 = \frac{k}{m}$ diverges as $k \rightarrow \infty$ and $m \rightarrow 0$! How do we rescue this? We multiply the above equation by $1$ in a clever way, multiplying and dividing by $l^2$:
\[\dpd[2]{u(x, t)}{t} = \omega_0^2l^2\left(\frac{u(x + l, t) + u(x - l, t) - 2u(x, t)}{l}\right)\]
But this is quite convenient, as $\omega_0^2l^2 = kl\frac{l}{m} = \frac{kl}{\lambda}$ which is constant! Let us therefore denote $\omega_0^2l^2 = c^2$ where $c$ si the speed of sound. We can therefore write the above equation in the following way:
\[\dpd[2]{u(x, t)}{t} = \frac{c^2}{l}\left[\frac{u(x +l) - u(x, t)}{l} - \frac{u(x, t) - u(x - l, t)}{l}\right]\]
We recognize that the two terms in the bracket are two first order spatial derivatives. Further taking another derivative by considering the difference of the two first order derivatives by taking $l \rightarrow 0$, the RHS becomes a single second order spatial derivative:
\[\boxed{\dpd[2]{u(x, t)}{t} = c^2\dpd[2]{u(x, t)}{x}}\]
Which we recognize is a wave equation!

\subsubsection{Wave equation Solutions and Dispersion Relation}
One prime example of a solution to the wave equation is:
\[u(x, t) = A\sin(kx - \omega t)\]
Where $k$ is the wavevector and $\omega$ the frequency. Plugging this into the wave equation, we immediately obtain the (familiar) dispersion relation:
\[\omega = ck\]
Recall that in HW5 we found a relation that was slightly more complicated, and was given as:
\[\omega = 2\omega_0\abs{\sin(\frac{kl}{2})}\]
Which graphically looks like:
\begin{center}
    \includegraphics[scale=0.8]{Lecture-30/l30-img3.pdf}
\end{center}
So if we consider the limit of this dispersion for $kl \ll 1$, we can Taylor expand the sine and get:
\[\omega = 2\omega_0 \frac{kl}{2} = \omega_0 kl = ck\]
So we therefore have that the slope of the above graph for small $k$ is equal to $c$. We then see that we can recover the linear dispersion relationship that we obtained in the continuum limit!

\subsubsection{Generalization to 3D}
Suppose we work in three dimensions, and instead have $P = p(x, y, z)$ (pressure, or some other variable). We then have that:
\[\dpd[2]{p}{t} = c^2\left(\dpd[2]{p}{x} + \dpd[2]{p}{y} + \dpd[2]{p}{z}\right) = c^2\nabla^2 p\]
Where $\nabla^2$ is the laplacian.

\subsubsection{Volume and Surface Forces}
Since we now work with materials with a finite extension, we have more types of forces to consider. We first have \textbf{volume forces}, which are proportional to $dV$. Typical examples are gravity and electrostatic forces:
\[\v{F}_g = \rho_0 \v{g}dV\]
\[\v{F}_e = \rho_e \v{E}dV\]
$\rho_0, \rho_e$ represent mass and charge densities repectively. These are in a sense "boring" and just come from external fields.
\begin{center}
    \includegraphics[scale=0.9]{Lecture-30/l30-img4.png}
\end{center}
Slightly more interesting are surface forces, which are proportional to dA. We then have Pressure, Tension, and Shear forces, as pictured below. Already things become more complicated as they depend not just on the magnitude of the force as well as the orientation of the body.
\begin{center}
    \includegraphics[scale=0.9]{Lecture-30/l30-img5.png}
\end{center}
We could of course build up to something like torsion from combinations of the above "elementary" surface forces.
\newline A question might be is where does this resistance to these surface forces come from? The answer is the intermolecular forces between the atoms in the material.

\subsubsection{Stress \& Strain - Basic Definitions}
\begin{itemize}
    \item Stress = Force / Area = Pressure (Fluid)
    \item Stress = Tension / Area (Wire)
    \item Stress = Shear Force / Area (Shear)
    \item Strain = dV/V (Fluid)
    \item Strain = dl/l (Wire)
    \item Strain = dy/dx (Shear)
\end{itemize}
One can think of the stress as a pressure, and the Strain as a "fractional deformation"/relative change.

\subsubsection{Hooke's Law for Solids (Linear Elasticity)}
For small deformations, materials will also follow Hooke's law. We start with the wire case. If we change the wire by length $dl$, we have that the force $dF$ is given by:
\[dF = k\cdot dl\]
However, it is more useful to divide both sides by the area $A$ and add a factor of $l$ on the RHS, which gives:
\[\frac{dF}{A} = \frac{kl}{A}\frac{dl}{l}\]
Therein, the term on the LHS is the tensile stress, the rightmost term is the tensile strain (could be compressive if $dl < 0$, but let us assume for now that $dl > 0$) and define:
\[\frac{lk}{A} = Y\]
Which is Young's modulus, which is a characteristic of the material/material property.
\newline Next, we look at a change of pressure in a fluid. We then have:
\[dp = -B\frac{dV}{V}\]
Where if we pull/expand a fluid, the pressure decreases. The $B$ is known as the bulk modulus, which tells us the response to a volumetric change (the proportionality constant to the strain). Note that in general liquids are non-compressible and hence $B$ for something like water is extremely high (but not infinite!)
\newline Finally, considering a shear, we have
\[\frac{F}{A} = G\frac{dy}{dx}\]
Where $G$ is the shear modulus. As a visual, consider shearing a material by displacing it. The amount by which we shear is $dy$. The amount of tilt is the shear strain, and the coefficient of the force that fights back is the shear modulus.
\begin{center}
    \includegraphics[]{Lecture-30/l30-img6.png}
\end{center}

\subsubsection{The Stress Tensor}
When we think of solids, we worry about two things; we both worry about the force (which has three components) as well as the orientation of the surface in which we apply the force, as the material acts differently depending along which surface we apply the force. The stress is specified by the direction of action and the orientation of the surface. This yields $3 \cdot 3 = 9$ possible components, yielding the stress tensor!
\begin{center}
    \includegraphics[scale=0.5]{Lecture-30/l30-img7.png}
\end{center}
Question: A fluid is an isotropic molecule that cannot sustain any shearing forces in equilibrium. What must be true for the stress tensor of the fluid?
\begin{s}
    The tensor must be digonal (no shear forces) with one unique component (isotropic). The idea is if we push, we feel the same pressure on all sides. There is only one scalar value which quantifies the response. For the fluid, the stress tensor will have a simple form:
    \[\sigma = \m{-P & 0 & 0 \\ 0 & -P & 0 \\ 0 & 0 & -P}, \quad  \sigma_{ij} = -P\delta_{ij}\]
    Note that this is not true of all liquids, e.g. with viscous liquids like honey we can expect shear forces when we stir.
\end{s}
Next day we will look at the counterpart of the stress tensor, the strain tensor, and hopefully get to equations of motion that describe the motion of objects with finite extension.

\end{document}


\chapter{Nonparametric regression}
This chapter is under \work.
\section{Kernel regression}
Let $(x_1,y_1),\ldots,(x_n,y_n)$ be a bivariate sample of size $n$. Assuming that the relationship between $x$ and $y$ can be expressed as:
\begin{equation}
y_i=m(x_i)+\epsilon_i,
\end{equation}

where $\epsilon_1,\epsilon_2,\ldots$ are IID with mean 0 and constant variance $\sigma^2$. Our goal is to estimate the {\it regression function (or regression curve)} $m(x)$.

{\it Nonparametric regression} is a collection of techniques for estimating the regression function without making strong assumptions about the shape of the function. Beside kernel regression, other nonparametric regression techniques include spline regression, wavelet regression and tree regression.

Kernel regression methods can be broadly classified as:
\begin{asparaenum}[(a)]
\item {\it Local polynomial kernel regression methods.}

\item {\it Others}, e.g. the {\it Gasser-Muller method} and the {\it Priestley-Chao method}.
\end{asparaenum}
We will focus mainly on the local polynomial kernel regression methods because they perform better. Moreover, with respect to mean squared error, the Gasser-Muller and Priestley-Chao methods are asymptotically inadmissible.

\subsection{Local polynomial kernel regression}
The estimated value of the regression function at a particular point, say $x^*$, is obtained by locally fitting a $p^{\textrm{th}}$ degree polynomial to the data via weighted least squares:
\begin{asparaenum}[(a)]
\item Let:
\begin{equation}
m_p(x,x^*)=\beta_0+\beta_1(x-x^*)+\beta_2(x-x^*)^2+\ldots+\beta_p(x-x^*)^p.
\end{equation}
\item Estimate the coefficients, $\beta_0,\beta_1,\ldots,\beta_p$ by weighted least squares:
\begin{equation}
(\hat{\beta}_0,\hat{\beta}_1,\ldots,\hat{\beta}_p)=\arg\min\sum^{n}_{i=1}K_h(x_i-x^*)[y_i-m_p(x_i,x^*)]^2.
\end{equation}
\item Let:
\begin{equation}
\hat{m}_{p,h}(x,x^*)=\hat{\beta}_0+\hat{\beta}_1(x-x^*)+\hat{\beta}_2(x-x^*)^2+\ldots+\hat{\beta}_p(x-x^*)^p.
\end{equation}
\item The estimated value of the regression function at $x^*$ is given by:
\begin{equation}
\hat{m}(x^*)=\hat{m}_{p,h}(x^*,x^*)=\hat{\beta}_0
\end{equation}
%\begin{flushright}   $\boxbox$ \end{flushright}
\end{asparaenum}
Note that the points are first shifted in the $x$-axis, so that the origin is at $x^*$, and then the polynomial is fitted to the shifted points by weighted least squares. The weights are provided by a kernel centred on $x^*$, i.e. $K_h(x-x^*)$. The consequence of the shift is that the estimate of the regression function at $x^*$ is now simply the estimate of the $y$-intercept, i.e. $\hat{\beta}_0$.

To obtain the regression coefficients by weighted least squares, let:
$$B=\left(\begin{array}{c}\beta_0\\\beta_1\\\textrm{M}\\\beta_p\end{array}\right),\space
Y=\left(\begin{array}{c}y_1\\\textrm{M}\\y_n\end{array}\right),\space
X_{x^*}=\left(\begin{array}{cccc}1&(x_1-x^*)&\Lambda &(x_1-x^*)^p\\\textrm{M}&\textrm{M}&\textrm{M}&\textrm{M}\\1&(x_n-x^*)&\Lambda &(x_n-x^*)^p
\end{array}\right)$$
$$W_{x^*}=\left(\begin{array}{cccc}
K_k(x_1-x^*)&0&\Lambda&0\\
0&K_k(x_2-x^*)&&\textrm{M}\\
\textrm{M}&&\textrm{O}&0\\
0&\Lambda&0&K_k(x_n-x^*)\\
\end{array}\right).$$
Then:
\begin{equation}
\hat{B}=(X^T_{x^*}W_{x^*}X_{x^*})^{-1}X^T_{x^*}W_{x^*}Y.
\end{equation}

If $e_1$ is the $(p + 1)$-column vector with 1 in the first row and 0 elsewhere, then:
\begin{equation}
\hat{m}(x^*)=e_1^T(X^T_{x^*}W_{x^*}X_{x^*})^{-1}X^T_{x^*}W_{x^*}Y.
\end{equation}

Special cases of local polynomial kernel regression include:

\begin{asparaenum}[(a)]
\item {\it Nadaraya-Watson method.} This corresponds to $p = 0$, i.e. fit a constant locally to the data. The estimate of the regression function at $x^*$ can be expressed as:
\begin{equation}
\hat{m}(x^*)=\frac{\sum^n_{i=1}K_h(x_i-x^*)y_i}{\sum^n_{j=1}K_h(x_j-x^*)}.
\end{equation}
\item {\it Local linear kernel regression.} This corresponds to $p = 1$, i.e. fit a straight line locally to the data. The estimate of the regression function at $x^*$ can be expressed as:
\begin{equation}
\hat{m}(x^*)=\frac{\sum^n_{i=1}[\hat{s}_2(x^*)-\hat{s}_1(x^*)(x_i-x^*)]K_h(x_i-x^*)y_i}{\hat{s}_2(x^*)\hat{s}_0(x^*)-\hat{s}_1(x^*)^2},
\end{equation}
where:
\begin{equation}
\hat{s}_r(x^*)=\sum^n_{i=1}(x_i-x^*)^rK_h(x_i-x^*),
\end{equation}
which may be thought of as a weighted $r^{\textrm{th}}$ moment about $x^*$.
%\begin{flushright}   $\boxbox$ \end{flushright}
\end{asparaenum}

See WJ\footnote{Wand, M.P. and Jones, M.C. (1995), {\it Kernel Smoothing}, Chapman \& Hall.}: Figure 5.1 for a visualisation of this regression.
%\begin{flushright}   $\boxbox$ \end{flushright}
Notice that for $p = 0$ and $p = 1$, the estimates of the regression function have the form:
\begin{equation}
\hat{m}(x^*)=\sum^n_{i=1}\omega_i(x^*)y_i,
\end{equation}
where:
\begin{equation}
\sum^n_{i=1}\omega_i(x^*)=1,
\end{equation}

i.e. the estimates are weighted (linear) combinations of the $y$ values. In general, this is true of local polynomial kernel regression for any $p$. The weights, $\omega_1(x^*),\ldots,\omega_n(x^*)$, are called {\it equivalent kernel weights} and they depend on the point $x^*$ at which the regression function is being estimated.

See WJ: Figures 5.6 and 5.7 for a visualisation of this method.
%\begin{flushright}   $\boxbox$ \end{flushright}
For routine use, local linear kernel regression is recommended because it has good asymptotic properties and behaves well at boundaries.

\subsection{Choice of kernel bandwidth}

The bandwidth selection methods for kernel density estimation can be adapted for kernel regression.

One approach is to consider the optimal bandwidth that minimises the asymptotic conditional mean of the integrated weighted squared error:
\begin{equation}
\lim_{n\rightarrow\infty}E[\int\{\hat{m}(x)-m(x)\}^2f(x)dx|x_1,\ldots,x_n],
\end{equation}

where $f$ is the density of the predictors.

Assuming that the perturbation variance $\sigma^2$ is constant and that the predictors are supported on the bounded interval $[a, b]$, the optimal bandwidth for local linear kernel regression is given by\footnote{Ruppert, D., Sheather, S. J. and Wand, M. P. (1995), \textquotedblleft An effective bandwidth selector for local least squares regression", {\it Journal of the American Statistical Association 90, 1257-1270}.}:
\begin{equation}
h=[\frac{c_K\sigma^2(b-a)}{c_mn}]^{1/5},
\end{equation}
where:
\begin{equation}
c_K=\frac{\int K(x)^2dx}{[\int x^2K(x)dx]^2},
\end{equation}

depends on the kernel $K$, and:
\begin{equation}
c_m=\int m^{\prime\prime}(x)^2f(x)dx,
\end{equation}

depends on the regression function $m$.

For the Gaussian kernel, $c_K=(2\sqrt{\pi})^{-1}$, and for the Epanechnikov kernel, $c_K=15$.

To compute the optimal bandwidth, plug-in estimates of $\sigma^2$ and $c_m$ are required. One way to do this is implemented in the \Matlab function {\tt loclinbw}:
\begin{VrbM}
% loclinbw.m
% Bandwidth selection for local linear kernel regression.
% Estimates the optimal asymptotic mean integrated weighted squared error
% by the rule-of-thumb method.
%
% Assumptions: Homoscesdastic, bounded predictor support.
%
% Usage: bw = loclinbw(x,y,kertype);
%
% Inputs: x = column vector of predictors.
%             y = column vector of responses.
%             kertype = kernel type:
%                             1 = Gaussian,
%                             2 = Epanechnikov.
%
% Output: bw = bandwidth.
%
% Reference: Ruppert, D., Sheather, S. J. and Wand, M. P. (1995),
% "An effective bandwidth selector for local least squares regression",
% Journal of the American Statistical Association 90, 1257-1270.
\end{VrbM}
This bandwidth should be used as an initial guide to what the appropriate bandwidth should be, by exploring other values close to it.
%\begin{flushright}   $\boxbox$ \end{flushright}

See WJ: Figure 5.2 for a depiction of this.
%\begin{flushright}   $\boxbox$ \end{flushright}

\begin{labwork}
The data in {\tt automobile.txt} are the horsepower (HP, in column 1) and miles per gallon (MPG, in column 2) of 82 automobiles. Estimate the regression function for MPG against HP using local linear Gaussian kernel regression.

\begin{VrbM}
data = load('automobile.txt');
x = data(:,1);
y = data(:,2);
bw = loclinbw(x,y,1)
s = [40:330]';
n = length(x);
ns = length(s);
yfit = zeros(ns,1);
for i = 1:ns
    K = normpdf(x,s(i),bw);
    xdev = x - s(i);
    xdevK = xdev .* K;
    s0 = sum(K);
    s1 = sum(xdevK);
    s2 = sum(xdev .* xdevK);
    w = ((s2 - s1 * xdev) .* K) / (s0 * s2 - s1 * s1);
    yfit(i) = w' * y;
end
plot(x,y,'.b',s,yfit,'-r')
xlabel('HP'), ylabel('MPG')
title(['Local linear Gaussian kernel regression with bw = ' num2str(bw)])
\end{VrbM}
Results:
\begin{VrbM}
bw = 13.3923
\end{VrbM}
%\includegraphics
Notice the gap in the HP values that causes the estimated regression curve to look suspicious - we should expect MPG to decrease monotonically with HP. We can therefore increase the kernel bandwidth slowly until the estimated regression curve is approximately monotone. A bandwidth of about 34 appears to be adequate.
%\includegraphics
%\begin{flushright}   $\boxbox$ \end{flushright}
\end{labwork}

\section{Extension to multivariate predictors}
Suppose that the predictors $x_1,\ldots,x_n$ are multivariate, say $d$-dimensional, but the responses $y_1,\ldots,y_n$ are still univariate. Once again, assume that the relationship between $x$ and $y$ can be expressed as:
\begin{equation}
y_i=m(x_i)+\epsilon_i=m(x_i(1),\ldots,x_i(d))+\epsilon_i,
\end{equation}

where $x_i(j)$ denotes the $j^{\textrm{th}}$ component of $x_i$, and $\epsilon_1,\epsilon_2,\ldots$ are IID with mean 0 and constant variance $\sigma^2$.

\subsection{Multivariate kernel regression}
We will describe the extension of local linear kernel regression to multivariate predictors. The estimated value of the regression function at a point $x^*$ is obtained by locally fitting a hyperplane to the data via weighted least squares, with weights provided by a multivariate kernel:
\begin{asparaenum}[(a)]
\item Let $m_1(x,x^*)=\beta_0+\beta_1(1)[x(1)-x^*(1)]+\Lambda+\beta_1(d)[x(d)-x^*(d)]$.

\item Estimate the regression coefficients by weighted least squares:
$$(\hat{\beta}_0,\hat{\beta}_1(1),\ldots,\hat{\beta}_1(d))=\arg\min\sum^n_{i=1}K_H(x_i-x^*)[y_i-m_1(x_i,x^*)]^2.$$

\item Let $\hat{m}_{1,H}(x,x^*)=\hat{\beta}_0+\hat{\beta}_1(1)[x(1)-x^*(1)]+\Lambda+\hat{\beta}_1(d)[x(d)-x^*(d)]$.

\item The estimated value of the regression function at $x^*$ is given by:

$$\hat{m}(x^*)=\hat{m}_{1,H}(x^*,x^*)=\hat{\beta}_0,$$

which can be computed directly from:
$$\hat{m}(x^*)=e_1^T(X^T_{x^*}W_{x^*}X_{x^*})^{-1}X^T_{x^*}W_{x^*}Y,$$

where $e_1$ is the $(d + 1)$-column vector with 1 in the first row and 0 elsewhere:
$$X_{x^*}=\left(\begin{array}{cccc}1&(x_1(1)-x^*(1))&\Lambda &(x_1(d)-x^*(d))\\\textrm{M}&\textrm{M}&\textrm{M}&\textrm{M}\\1&(x_n(1)-x^*(1))&\Lambda &(x_n(d)-x^*(d))
\end{array}\right)$$
and:
$$W_{x^*}=\left(\begin{array}{cccc}
K_H(x_1-x^*)&0&\Lambda&0\\
0&K_H(x_2-x^*)&&\textrm{M}\\
\textrm{M}&&\textrm{O}&0\\
0&\Lambda&0&K_H(x_n-x^*)\\
\end{array}\right).$$
%\begin{flushright}   $\boxbox$ \end{flushright}
\end{asparaenum}
Clearly, the extension from univariate predictors to multivariate ones is straightforward. Unfortunately, to get a good estimate of the regression function as $d$ increases requires the sample size to grow much faster than $d$ - this is known as the {\it curse of dimensionality}.

\subsection{Additive regression}
This is an alternative approach that models the multivariate regression function as a sum of univariate regression functions, one for each dimension, plus a constant term $m_0$:
\begin{equation}
y_i=m(x_i)+\epsilon_i=m_0+\sum^d_{j=1}m_j(x_i(j))+\epsilon_i.
\end{equation}

Thus, $m_j$ is a univariate regression function for the $j^{\textrm{th}}$ component of the predictor. Finding the regression functions is not the same as performing d separate univariate regressions because the other dimensions must be taken into account when fitting for one dimension. This involves an iterative procedure known as {\it backfitting}, where the residuals from the current iteration are used to improve the fit in the next iteration.

Additive regression performs well if the multivariate regression function is represented well by the assumed additive model, and if the components of the predictor are not strongly correlated.
%\begin{flushright}   $\boxbox$ \end{flushright}

See Scott\footnote{Scott, D.W. (1992), {\it Multivariate Density Estimation: Theory, Practice and Visualization,} Wiley.}: Figures 8.7 and 8.9 for a graphic illustration.

\section{Exercises}
\begin{exercise}
Work through Example 6.1.7 for the automobile data.
\end{exercise}
\begin{exercise}
The data in {\tt car.txt} are the miles per gallon (MPG) (column 1) and top speed (column 2) of 82 cars.
\begin{asparaenum}[(a)]
\item Perform a regression of top speed (response) on MPG (predictor) using local linear Gaussian kernel regression with the bandwidth that minimises the asymptotic conditional mean integrated weighted squared error. Produce a scatter plot of the data points with the estimated regression curve superimposed.
\item Suppose that the relationship between top speed and MPG is actually monotone. Adjust the bandwidth to get a regression curve that reflects this but does not oversmooth. Produce a scatter plot of the data points with the new regression curve superimposed.
\end{asparaenum}
\end{exercise}

\begin{exercise}
The second column of {\tt glass.txt} contains measurements of the refractive index of 214 glass specimens collected in forensic work. The fifth column contains the aluminium contents of the glass specimens.
\begin{asparaenum}[(a)]
\item Perform a local linear Gaussian kernel regression of refractive index $(y)$ against aluminium content $(x)$, with four times the bandwidth that minimises the asymptotic conditional mean integrated weighted squared error. Produce a scatter plot of the data points with the estimated regression curve superimposed.
\item Obtain a point-wise 0.95 BCA confidence band for the regression function by bootstrapping residuals with 1000 bootstrap replicates.
\end{asparaenum}
\end{exercise}

\begin{exercise}
The data in {\tt motorcycle.txt} are from 94 simulated motorcycle accidents. The first column contains the times in milliseconds since impact; the second column contains the head accelerations in gs.
\begin{asparaenum}[(a)]
\item Perform a local linear Gaussian kernel regression of head acceleration $(y)$ versus time $(x)$, with the bandwidth that minimises the asymptotic conditional mean integrated weighted squared error. Produce a scatter plot of the data points with the estimated regression curve superimposed.
\item Obtain a point-wise 0.95 BCA confidence band for the regression function by bootstrapping residuals with 1000 bootstrap replicates.
\end{asparaenum}
\end{exercise}


\section{Model fitting}

\subsection{Hough transform}

x-y space to a-b parameter space

\includegraphics[width=0.8\columnwidth]{pictures/hough}

$$b = (-x)*a+y$$

Alternative: x-y space to $\theta$-$\rho$ (or d) space
$$x cos(\theta) + y sin(\theta) = \rho$$

For each point, render the curve ($\theta$, d) and count the parameter in bins with similar values.

Difficulties: How big should the bins be (too big, and we cannot distinguish between quite different lines; too small, and noise causes lines to be missed)

Hardly ever satisfactory in practice, because problems with noise and bin size defeat it

\subsection{Line fitting}
\subsubsection{Incremental line fitting}
\begin{lstlisting}[frame=single]
Put all points on a curve list, in order along the curve
Empty the line point list
Empty the line list
Until there are too few points on the curve
	Transfer first few points on the curve tot the line point list
	Fit line to line point list
	While fitted line is good enough
		Transfer the next point on the curve to tthe line point list and refit the line
	end
	Transfer last point(s) back to curve
	Refit line
	Attach line to line list
end
\end{lstlisting}

\subsubsection{K-means}
\begin{lstlisting}[frame=single]
Hypothesize k lines (perhabps uniformly at random)
or
Hypothesize an assignment of lines to points and then fit lines using this assingment

Until convergence
	Allocate each point to the closest line
	Refit lines
end
\end{lstlisting}

Problem can be stuck in local optima

\subsubsection{Probabilistic}
- EM-algorithm (expectation maximum)
- M-estimators
- RANSAC

\subsubsection{RANSAC}

- Choose a small subset uniformly at random
- Fit to that
- Anything that is close to result is signal; all others are noise
- Refit
- Do this many times and choose the best\\

\textbf{Issues}
\begin{itemize}
	\item How many times? Often enough that we are likely to have a good line
	\item How big a subset? Smallest possible
	\item What does close mean? Depends on the problem
	\item What is a good line? One where the number of nearby points is so big it is unlikely to be all outliers
\end{itemize}

\begin{lstlisting}[frame=single]
Determine:
	n-the smallest number of points required
	k-the number of iterations required
	t-the threshold used to identify a point that fits well
	d-the number of nearby points required to assert a model fits well
Until k iterations have occured
	Draw a sample of n points from the data uniformly and at random
	Fit to that set of n points
	For each data point outside the sample
		Test the distance from the point to the line against t; if the distance from the point to the line is less than t, the point is close
	end
	If there are d or more points close to the line then there is a good fit. Refit the line using all these points.
end
Use the best fit from this collection, using the fitting error as criterium
\end{lstlisting}

$$ k = \frac{log(1-p)}{log(1-w^s)} $$
k - number of iterations\\
p - probability at least one sample free from outliers\\
w - fraction of inliers = d/n\\
s - model dimension (2 for a line, 3 for plane)

\subsection{Missing variable problems}

In many vision problems, if some variables were known the maximum likelihood inference problem would be easy
\begin{itemize}
	\item fitting; if we knew which line each token came from, it would be easy to determine line parameters
	\item segmentation; if we knew the segment each pixel came from, it would be easy to determine the segment parameters
	\item fundamental matrix estimation; if we knew which feature corresponded to which, it would be easy to determine the fundamental matrix
\end{itemize}
This sort of thing happens in statistics, too - strategy:
\begin{enumerate}
	\item estimate missing variables
	\item plug these in, now estimate parameters
	\item re-estimate appropriate values for missing variables - converges to local extremum (like k-means)
\end{enumerate}

\subsection{Cross validation}
-Split data set into two pieces, fit to one, and compute negative loglikelihood on the other
-Average over multiple different splits
-Choose the model with the smallest value of this average
-The difference in averages for two different models is an estimate of the difference in KL divergence of the models from the source of the data


\documentclass[a4paper, 12pt]{article}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{dsfont}
\usepackage[left=1.5cm, right=1.5cm, bottom=2cm, top=2cm]{geometry}
\usepackage{graphicx}
\usepackage{hyperref}
\usepackage[utf8]{inputenc}
\usepackage{microtype}
\usepackage{natbib}
\newcommand{\given}{\,|\,}

\title{Notes}
\author{Brendon J. Brewer}
\date{}

\begin{document}
\maketitle

\abstract{}

% Need this after the abstract
\setlength{\parindent}{0pt}
\setlength{\parskip}{8pt}

\section*{Unknown True Metallicities}
Let $\{Z_i\}$ be the unknown true metallicities and let $\{z_i\}$ be the
measured metallicities, with the reported errorbars $\{b_i\}$, such that
\begin{align}
z_i \given Z_i &\sim \textnormal{Normal}\left(z_i, b_i^2\right).
\end{align}
Let the other parameters be $\theta$ and the data be $D$. We seek the
posterior
\begin{align}
p(\theta, \boldsymbol{Z} \given D) &\propto p(\theta, \boldsymbol{Z})
                                            p(D \given \theta, \boldsymbol{Z}) \\
    &= p(\theta)p(\boldsymbol{Z}) p(D \given \theta, \boldsymbol{Z}),
\end{align}
assuming that the true metallicities do not tell us anything about the other
parameters. Then, we can marginalise out the true metallicities:
\begin{align}
p(\theta \given D)
    &= \int p(\theta, \boldsymbol{Z} \given D) \, d\boldsymbol{Z} \\
    &\propto \int p(\theta)p(\boldsymbol{Z}) p(D \given \theta, \boldsymbol{Z})
               \, d\boldsymbol{Z}
\end{align}

\end{document}


\documentclass{homework}
\course{Math 5522H}
\author{Jim Fowler}
\input{preamble}

\begin{document}
\maketitle

\begin{inspiration}
  On April 9, 1975, Congressman Robert Michel brandished a list of new
  NSF grants on the floor of the House of Representatives and selected
  a few that he thought might represent a waste of the taxpayers'
  money. One of them \ldots was called ``Studies in Complex
  Analysis. Michel's comment was, `` `Simple Analysis' would,
  hopefully, be cheaper.''
  \byline{Gerald B. Folland} % the American Mathematical Monthly (vol 780, Oct 1998, pg. 780)
\end{inspiration}

\section{Terminology}

\begin{problem}
  What is a \textbf{simply-connected} domain?
\end{problem}

\begin{problem}
  What does it mean to say that two curves $\alpha$ and $\beta$ are
  \textbf{homologous}?
\end{problem}

\begin{problem}
  What is a \textbf{null homotopic} path?
\end{problem}

\begin{problem}
  Given a function $f : U \to \C$, what is meant by a \textbf{branch} of the log of $f$?
\end{problem}

\section{Numericals}

\begin{problem}
  Let $f(z) = (z^2-z-1)/z^3$, and consider the curve
  $\gamma : [0,2\pi] \to \C$ by $\gamma(\theta) = f(e^{i\theta})$.
  Compute the winding number $n(\gamma,0)$.
\end{problem}

\begin{problem}
  Evaluate the \textbf{Dirichlet integral}
  \[
    \int_0^\infty \frac{\sin x}{x} \, dx.
  \]
\end{problem}

\begin{problem}
  Define $\gamma : [0,2\pi] \to \C$ by $\gamma(\theta) = e^{i\theta}$.
  Suppose $p \in \C[z]$ is a degree $n$ polynomial with distinct roots
  $z_1,\ldots,z_n$ in the unit disk, and evaluate
  \[
    \int_\gamma \frac{z^m \, p'(z)}{p(z)} \, dz.
  \]
\end{problem}

\begin{problem}
  Again define $\gamma : [0,2\pi] \to \C$ by
  $\gamma(\theta) = e^{i\theta}$, and then evaluate
  \[
    \int_\gamma \frac{\conj{w}}{w - z} \, dw
  \] for $z \in \C$ with $\abs{z} < 1$.
\end{problem}

\begin{problem}
  Consider subsets of the real line $A = (-\infty,-1] \cup [1,\infty)$
  and $B = [-1,1]$.  Is it possible to define a branch of the
  logarithm of $z^2 - 1$ on $\C \setminus A$?  What about on
  $\C \setminus B$?
\end{problem}

\section{Exploration}

\begin{problem}
  If we can find a branch of log $f$, then we can define a
  single-valued $\sqrt{f(z)}$ via $e^{(1/2) \, \log f}$.  Does the
  converse hold?  Find an open set $U \subset \C$ and a holomorphic
  function $f : U \to \C$ so that there is \textit{no} branch of log
  of $f$, but it is nevertheless possible to define a single-valued
  $\sqrt{f(z)}$.
\end{problem}

\begin{problem}
  Liouville's theorem (\ref{liouville-theorem}) states that a
  nonconstant holomorphic function $f : \C \to \C$ is not bounded.
  Seeing \ref{entire-is-dense}, we might ask how ``small'' can the
  image of holomorphic function be.  Find a holomorphic function
  $f : \C \to \C$ with image equal to $\C \setminus \{w\}$.
  (Incidentally, by \textbf{Picard's little theorem}, you will have
  trouble finding an entire function with image missing two points.)
\end{problem}

\begin{problem}\label{cauchy-inequalities-2}Suppose
  $f(z) = \sum_{n=0}^\infty a_n z^n$ is holomorphic in the disk
  $B_R(0)$.  If $0 < r < R$, show that
  \[
    \abs{a_n} \leq \frac{1}{r^n} \displaystyle\sup_{z \in \partial B_r(0)} \abs{f(z)}.
  \]
  This extends \ref{cauchy-inequalities}.
\end{problem}

\begin{problem}
  Consider a nonconstant polynomial $p \in \mathbb{C}[z]$ of degree $n$.  Compute
  \[
    \lim_{r \to \infty} \int_{\partial B_r(0)} \left( \frac{p'(z)}{p(z)} - \frac{n}{z} \right) dz.
  \]
  Can you perform this computation \textit{without} appealing to the
  fundamental theorem of algebra?
\end{problem}

\begin{problem}\label{uniformly-approximate-conj}For $\epsilon > 0$, is there a polynomial $p \in \mathbb{C}[z]$ so
  that $\abs{p(z) - \conj{z}} < \epsilon$ for $z \in B_1(0)$?  In
  other words, can we uniformly approximate $\conj{z}$ by a polynomial
  in $z$?
\end{problem}

\section{Prove or Disprove and Salvage if Possible}

\begin{problem}\label{entire-is-dense}If $f : \C \to \C$ is
  holomorphic, then the image of $f$ is dense in $\C$.
\end{problem}

\begin{problem}\label{identity-dominate-entire}Suppose $f : \C \to \C$
  is holomorphic and for all $z \in \C$ we have
  $\abs{f(z)} \leq \abs{z}$.  Then $f(z) = \lambda z$ for some
  $\lambda \in \C$.
\end{problem}

\begin{problem}
  There is a nonconstant holomorphic function $f : \C \to \C$ so that
  for all $n \in \Z$ we have $f(z + n) = f(z)$.  (Such a function is
  \textbf{periodic}.)
\end{problem}

\begin{problem}\label{doubly-periodic}There is a nonconstant holomorphic function $f : \C \to \C$ so that
  for all $\omega \in \Z[i]$ we have $f(z + \omega) = f(z)$.  (Such a
  function is \textbf{doubly periodic}.)
\end{problem}

\begin{problem}\label{maximum-modulus-principle}Consider a connected open subset $U \subset \C$ and a holomorphic
  function $f : U \to \C$.  If there is a point $z_0 \in U$ so that
  for all $z \in U$ we have $\abs{f(z_0)} \geq \abs{f(z)}$, then $f$
  is constant.  \textit{Hint:} show that $g(z) = \abs{f(z)}$ is
  constant and invoke \ref{open-mapping-theorem-preview}.  This is the
  \textbf{maximum modulus principle}.
\end{problem}

\end{document}


\documentclass{ximera}
\input{../preamble}
\title{Exercises: Taylor Series}
%%%%%\author{Philip T. Gressman}

\begin{document}
\begin{abstract}
Exercises relating to Taylor series and their computation.
\end{abstract}
\maketitle

\begin{exercise}
Use the Maclaurin series for $\sin x$ to give a complete formula for the Maclaurin series for $x \sin x^2$:
\[ x \sin (x^2) = \sum_{m=0}^\infty  \answer{\frac{(-1)^m}{(2m+1)!}} x^{\answer{4m+3}} \]
\end{exercise}

\begin{exercise}
Use the Maclaurin series for $e^x$ and $e^{-x}$ to give a complete formula for the Maclaurin series for $\cosh x$:
\[ \cosh x = \sum_{n=0}^\infty \answer{\frac{1}{(2n)!}} x^{\answer{2n}}. \]
\begin{hint}
Your formula should account for the fact that there are only even powers of $x$.
\end{hint}
\end{exercise}

\begin{exercise}
Compute the Taylor series of the function $f(x) = \ln x$ centered at the point $x_0 = 2$.
\[ \answer{ \ln 2} + \answer{\frac{1}{2}} (x-2) + \answer{- \frac{1}{8}}(x-2)^2 + \answer{\frac{1}{24}}(x-2)^3 + \cdots = \ln 2 + \sum_{n=1}^\infty \answer{\frac{(-1)^{n-1}2^{-n}}{n}} (x-2)^n \]
\end{exercise}

\begin{exercise}
Compute the Maclaurin series of the function identified below.
\[ \int_0^x \frac{\sin t}{t} dt = \sum_{n=0}^\infty \answer{\frac{(-1)^n}{(2n+1)! (2n+1)}} x^{\answer{2n+1}} \]
\end{exercise}

\section*{Sample Quiz Questions}

\begin{question}%%%%%[TaylorExpand001]

Compute the first \(4\) nonzero terms in the Taylor series
at \(x=0\) of the function \[\frac{d}{dx} \left[ x e^{x^2} \right].\]
\begin{multiplechoice}
\choice[correct]{\( \displaystyle 1 + 3x^{2} + \frac{5}{2}x^{4} + \frac{7}{6}x^{6}\)}
\choice{\( \displaystyle  - 1 - 3x^{2} + \frac{5}{2}x^{4} + \frac{7}{6}x^{6}\)}
\choice{\( \displaystyle 1 - 6x^{2} - \frac{5}{4}x^{4} + \frac{7}{6}x^{6}\)}
\choice{\( \displaystyle  - 1 - 6x^{2} + \frac{5}{4}x^{4} + \frac{7}{6}x^{6}\)}
\choice{\( \displaystyle 2 + 3x^{2} + \frac{5}{4}x^{4} + \frac{7}{6}x^{6}\)}
\choice{\( \displaystyle 2 - 3x^{2} - \frac{5}{4}x^{4} + \frac{7}{6}x^{6}\)}
\end{multiplechoice}
\begin{feedback}
Compute the series in stages beginning with substitution into known series:
\[e^{x^2} = 1 + x^{2} + \frac{1}{2}x^{4} + \frac{1}{6}x^{6} + \cdots \]
\[x e^{x^2} = x + x^{3} + \frac{1}{2}x^{5} + \frac{1}{6}x^{7} + \cdots \]
\[\frac{d}{dx} \left[ x e^{x^2} \right] = 1 + 3x^{2} + \frac{5}{2}x^{4} + \frac{7}{6}x^{6} + \cdots \]
\end{feedback}

\end{question}

\begin{question}%%%%%[TaylorExpand011]

Compute the first \(4\) nonzero terms in the Taylor series
at \(x=0\) of the function \[\int_0^x \left( x \ln (1-x) \right) ~ dx.\]
\begin{multiplechoice}
\choice{\( \displaystyle  - \frac{1}{6}x^{3} - \frac{1}{8}x^{4} - \frac{2}{15}x^{5} - \frac{1}{24}x^{6}\)}
\choice{\( \displaystyle \frac{1}{6}x^{3} + \frac{1}{8}x^{4} - \frac{2}{15}x^{5} - \frac{1}{24}x^{6}\)}
\choice{\( \displaystyle  - \frac{1}{6}x^{3} + \frac{1}{4}x^{4} + \frac{1}{15}x^{5} - \frac{1}{24}x^{6}\)}
\choice{\( \displaystyle \frac{1}{6}x^{3} + \frac{1}{4}x^{4} - \frac{1}{15}x^{5} - \frac{1}{24}x^{6}\)}
\choice[correct]{\( \displaystyle  - \frac{1}{3}x^{3} - \frac{1}{8}x^{4} - \frac{1}{15}x^{5} - \frac{1}{24}x^{6}\)}
\choice{\( \displaystyle  - \frac{1}{3}x^{3} + \frac{1}{8}x^{4} + \frac{1}{15}x^{5} - \frac{1}{24}x^{6}\)}
\end{multiplechoice}
\begin{feedback}
Compute the series in stages beginning with substitution into known series:
\[\ln (1-x) =  - x - \frac{1}{2}x^{2} - \frac{1}{3}x^{3} - \frac{1}{4}x^{4} + \cdots \]
\[x \ln (1-x) =  - x^{2} - \frac{1}{2}x^{3} - \frac{1}{3}x^{4} - \frac{1}{4}x^{5} + \cdots \]
\[\int_0^x \left( x \ln (1-x) \right) ~ dx =  - \frac{1}{3}x^{3} - \frac{1}{8}x^{4} - \frac{1}{15}x^{5} - \frac{1}{24}x^{6} + \cdots \]
\end{feedback}

\end{question}

\section*{Sample Exam Questions}

\begin{question}%%%%%[2015C.01]

The first few nonzero terms of the Maclaurin series for \(f(x) = \ln ( 1 + \sin x)\) are:
\begin{multiplechoice}
\choice{\(\displaystyle 1 + \frac{1}{2} x - \frac{1}{8} x^2 + \frac{1}{24} x^3 + \cdots\)}
\choice{\(\displaystyle 1 + \frac{1}{2} x - \frac{1}{8} x^2 - \frac{1}{48} x^3 + \cdots\)}
\choice{\(\displaystyle x - \frac{1}{2} x^2 + \frac{1}{8} x^3 - \frac{1}{24} x^4 + \cdots\)}
\choice{\(\displaystyle 1 + x + \frac{1}{2} x^2 + \frac{1}{3} x^3 + \frac{1}{6} x^4 \cdots\)}
\choice[correct]{\(\displaystyle x - \frac{1}{2} x^2 + \frac{1}{6} x^3 - \frac{1}{12} x^4 + \cdots\)}
\choice{\(\displaystyle 1 + x + \frac{1}{2} x^2 + \frac{1}{3} x^3 - \frac{1}{12} x^4 + \cdots\)}
\end{multiplechoice}
(Hints will not be revealed until you choose your response.)
\begin{feedback}
The first few derivatives of \(f(x)\) are:
\[\begin{aligned}
f(x) & = \ln (1 + \sin(x)) \\
f'(x) & = \frac{ \cos x}{1 + \sin x} \\
f''(x) & = - \frac{\sin x}{1 + \sin x} - \frac{\cos^2 x}{(1 + \sin x)^2} \\
f'''(x) & = - \frac{\cos x}{1 + \sin x} + \frac{\sin x \cos x}{(1 + \sin x)^2} - \frac{2 \sin x \cos x}{(1 + \sin x)^2} + 2 \frac{\cos^3 x}{(1 + \sin x)^3}
\end{aligned}\]
\begin{hint}
Evaluating at \(x = 0\) gives \(f(0) = \ln 1 = 0\), \(f'(0) = 1\), \(f''(0) = -1\), and \(f'''(0) = 1\). Therefore the series starts with the terms \(x - \frac{1}{2} x^2 + \frac{1}{6} x^3 + \cdots\).
\end{hint}
\end{feedback}

\end{question}

\begin{question}%%%%%[2016C.12]

Find the Taylor polynomial of degree 2 for \(f(x) = \sqrt{x+16}\) centered at \(x=9\).
\begin{multiplechoice}
\choice{\(\displaystyle 5 + \frac{4}{5} x + \frac{9}{250} x^2\)}
\choice{\(\displaystyle 5 - \frac{3}{5} (x-5) + \frac{1}{125} (x-5)^2\)}
\choice[correct]{\(\displaystyle 5 + \frac{1}{10} (x-9) - \frac{1}{1000} (x-9)^2\)}
\choice{\(\displaystyle 5 + \frac{3}{5} (x-5) + \frac{8}{125} (x-5)^2\)}
\choice{\(\displaystyle 5 + \frac{1}{5} (x-9) + \frac{16}{125} (x-9)^2\)}
\choice{none of these}
\end{multiplechoice}

\end{question}

\begin{question}%%%%%[2017C.14]

Use the Taylor polynomial of degree \(3\) for \(f(x) = \ln (1+x)\) centered at \(x_0 = 0\) to approximate the value of \(\displaystyle \ln \left( \frac{3}{2} \right)\).
\begin{multiplechoice}
\choice{\(\displaystyle \frac{2}{3}\)}
\choice{\(\displaystyle \frac{3}{2}\)}
\choice{\(\displaystyle \frac{15}{4}\)}
\choice[correct]{\(\displaystyle \frac{5}{12}\)}
\choice{\(\displaystyle \frac{9}{24}\)}
\choice{\(\displaystyle \frac{11}{24}\)}
\end{multiplechoice}

\end{question}

\begin{question}%%%%%[2017C.15]

Let \(F(x)\) be the unique function that satisfies \(F(0) = 0\), \(F'(0) = 0\), and \(F'(x) = \frac{1}{x} \sin x^3\) for all \(x \neq 0\). Find the Taylor Series of \(F(x)\) centered at \(x_0 = 0\).
\begin{multiplechoice}
\choice{\(\displaystyle \sum_{n=0}^\infty \frac{(-1)^n x^{6n+3}}{(2n+1)!}\)}
\choice{\(\displaystyle \sum_{n=0}^\infty \frac{(-1)^n (6n+3) x^{6n+2}}{(2n+1)!}\)}
\choice[correct]{\(\displaystyle \sum_{n=0}^\infty \frac{(-1)^n x^{6n+3}}{(6n+3)(2n+1)!}\)}
\choice{\(\displaystyle \sum_{n=0}^\infty \frac{(-1)^n x^{6n+2}}{(2n+1)!}\)}
\choice{\(\displaystyle \sum_{n=0}^\infty \frac{(-1)^n (6n+2) x^{6n+2}}{(2n+1)!}\)}
\choice{\(\displaystyle \sum_{n=0}^\infty \frac{(-1)^n x^{2n+3}}{(6n+3)(2n+1)!}\)}
\end{multiplechoice}

\end{question}

\end{document}


% Copyright 2018 Markus J. Pflaum and Daniel Spiegel, licensed under GNU FDL v1.3
% lectures notes taken by Daniel Spiegel based on a lecture by Markus J.Pflaum
%
\section{Spectral theory of bounded operators}
\para
We now apply the foundations of Hilbert space theory built in the previous sections to
spectral theory. For the moment we will sacrifice generality and work only with bounded
linear operators. The spectral theory of unbounded linear operators will be treated later.

Let us a recall that a linear map $A: \hilbertH_1 \rightarrow \hilbertH_2$ between Hilbert spaces
is continuous if and only if it is bounded, i.e.\ has finite operator norm, and that
$\blinOps (\hilbertH_1,\hilbertH_2)$ is a Banach space with the operator norm.
For the rest of this section, $\hilbertH$, $\hilbertH_1$, $\hilbertH_2, \ldots$ will always
denote complex Hilbert spaces and $A$, $B$ bounded linear operators.
We will also now fix the base field to be complex, i.e.\ $\fldK = \C$.
Last we agree on writing $I_\hilbertH$ or just $I$ for the identity operator on a Hilbert space
$\hilbertH$.

\subsec{Spectrum and Resolvent}

\begin{definition}
  Let $A:\hilbertH \to \hilbertH$ be a bounded linear operator.  A complex number $\lambda$
  is then called an \emph{eigenvalue} of $A$ if there exists a nonzero $v \in H$ such that
  $Av = \lambda v$. For every $\lambda \in \C$ one defines the $\lambda$-\emph{eigenspace} of
  $A$ as
  \[
   \operatorname{Eig}_\lambda  (A) = \big\{ v \in H \bigmid Av = \lambda v \big\} \subset \hilbertH,
  \]
 which is clearly a linear subspace of $\hilbertH$.
\end{definition}

\para
  By definition it is immediately clear that
 \[
   \operatorname{Eig}_\lambda (A) = \ker(A - \lambda),
 \]
 where the $\lambda$ on the right stands for the operator $\lambda I$.
 In other words this means that
 $\lambda\in \C$ is an eigenvalue of $A$ if and only if $A - \lambda$ is not injective.

\begin{definition}
Let $A \in \blinOps(\hilbertH)$. We make the following definitions.
\begin{romanlist}
  \item A \emph{regular value} of $A$ is a complex number $\lambda$ such that $A-\lambda$ is invertible.
  \item The set of all regular values is the \emph{resolvent} of $A$, denoted $\rset(A)$.
  \item A \emph{spectral value} of $A$ is a complex number $\lambda$ such that $A - \lambda$ is not
        invertible.
  \item The set of all spectral values is the \emph{spectrum} of $A$, denoted $\sigma(A)$.
  \item The \emph{point} or \emph{eigenspectrum} of $A$ is the set
	\[
	\sigma_\textup{p}(A) = \big\{\lambda \in \C \bigmid \ker(A - \lambda) \neq \{0 \} \big\}.
	\]
  \item An \emph{approximate eigenvalue} of $A$ is a complex number $\lambda$ for which there exists
        a sequence of unit vectors $(v_n)_{n\in \N}\subset \hilbertH$ such that
	\[
   	   \lim_{n \rightarrow \infty} (A - \lambda)v_n = 0.
	\]
	The set $\sigma_\textup{ap}(A)$ is the set of all approximate eigenvalues.
\end{romanlist}
\para
Evidently, $\sigma(A) = \C \setminus \rset(A)$ and $\sigma_\textup{p}(A) \subset \sigma_\textup{ap}(A) \subset \sigma(A)$,
and these may all be strict inclusions. Note that $A - \lambda$ is bounded for any $\lambda \in \C$,
so the open mapping theorem \ref{} implies that $(A - \lambda)^{-1} \in \blinOps(\hilbertH)$ when $\lambda \in \rset(A)$.
We call the map
\[
  R_\bullet (A ):\rset(A) \rightarrow \blinOps(\hilbertH), \quad R_\lambda (A) = (A-\lambda)^{-1}
\]
the \emph{resolvent} of $A$, not to be confused with the resolvent set $\rset(A)$.
To keep the notation clean, we often briefly write $R_\lambda$ for $R_\lambda(A)$ and leave implicit that $R_\lambda$ depends on $A$.
\end{definition}
First, we prove some topological properties of the spectrum and resolvent. Recall the following lemma, which generalizes the geometric series.

\begin{lemma}[Carl Neumann]\label{thm:neumann-series} Let $A \in \blinOps(\hilbertH)$. If $\norm{A} < 1$, then $I - A$ is invertible,
\[
(I - A)^{-1} = \sum_{n=0}^\infty A^n,
\]
and
\[
\norm{(I-A)^{-1}} \leq \frac{1}{1 - \norm{A}}.
\]
\end{lemma}

\begin{proof}
  Since $\norm{A} < 1$ and $\norm{A^n} \leq \norm{A}^n$ by submultiplicativity of the operator norm, we know
  $\sum_{n=0}^\infty \norm{A^n} < \infty$. This implies that the  family $(A^n)_{n\in\N}$ is absolutely summable,
  so $\sum_{n=0}^\infty A^n$ exists. Furthermore, for every $N \in \N$ we have
  \begin{align*}
    (I-A)\sum_{n=0}^N A^n = \left(\sum_{n=0}^N A^n \right)(I-A)= \sum_{n=0}^N A^n - \sum_{n=1}^{N+1}A^n = I - A^{N+1},
  \end{align*}
which implies that
\[
\lim_{N \rightarrow \infty}(I-A)\sum_{n=0}^N A^n = \lim_{N \rightarrow \infty} \left(\sum_{n=0}^N A^n\right) (I-A) = I.
\]
By continuity of multiplication in $\blinOps(\hilbertH)$ one gets
\[
(I-A) \sum_{n=0}^\infty A^n = \left(\sum_{n=0}^\infty A^n\right) (I-A) = I,
\]
which proves that $I-A$ is invertible and $(I-A)^{-1} = \sum_{n=0}^\infty A^n$.

Finally,  one concludes by the triangle inequality and submultiplicativity of the operator norm
\[
  \norm{(I - A)^{-1}} \leq \sum_{n=0}^\infty \norm{A^n} \leq \sum_{n=0}^\infty \norm{A}^n = \frac{1}{1 - \norm{A}}.
\]
\end{proof}

\begin{proposition}\label{thm:resolvent-topological-properties}
Let $A \in \blinOps(\hilbertH)$.
\begin{romanlist}
\item For any $\lambda \in \rset(A)$, one has
	\[
	B_{\norm{R_{\lambda}}^{-1}}(\lambda) \subset \rset(A) \ .
	\]
	Hence, $\rset(A) \subset \C$ is open.
\item The spectrum $\sigma(A)$ is compact and
	\[
	\sigma(A) \subset \closure{B}_{\norm{A}}(0) \ .
	\]
\item\label{ite:resolvent-expansion-large-argument} If the complex number $\lambda$ satisfies $\abs{\lambda} > \norm{A}$, then $\lambda \in \rset(A)$ and
      \[
	R_\lambda = - \frac{1}{\lambda} - \sum_{n=1}^\infty \lambda^{-n-1}A^n \ ,
      \]
      where convergence is with respect to the operator norm.
\end{romanlist}
\end{proposition}

\begin{proof}
 \begin{adromanlist}
 \item
 Fix $\lambda \in \rset(A)$ and set $r = \norm{R_{\lambda}}^{-1}$. Let $\mu \in B_r(\lambda)$. Then
\begin{align*}
 \norm{(\mu - \lambda)R_{\lambda}} = \abs{\mu - \lambda} \norm{R_{\lambda}} < 1.
\end{align*}
Thus, by Lemma \ref{thm:neumann-series}, one knows that $I - (\mu - \lambda)R_{\lambda}$ is invertible. Since $A - \lambda$ is invertible,
the composition
\[
(A - \lambda) \, \big( I - (\mu - \lambda)R_{\lambda} \big) = A - \mu
\]
is invertible, which proves that $\mu \in \rset(A)$. Hence $\rset(A)$ is open.

\item Since $\rset(A)$ is open, the complement $\sigma(A) = \C \setminus \rset(A)$ is closed. Furthermore, if $\abs{\lambda} > \norm{A}$, then $\norm{\lambda^{-1}A} < 1$, so $I - \lambda^{-1}A$ and hence $A - \lambda$ are invertible by Lemma \ref{thm:neumann-series}. This implies that $\lambda \in \rset(A)$, so $\sigma(A) \subset \closure{B}_{\norm{A}}(0)$. Since $\sigma(A)$ is closed and bounded, it is compact.
\item If $\abs{\lambda} > \norm{A}$, then $I - \lambda^{-1}A$ is invertible by Lemma \ref{thm:neumann-series} and
\[
(I - \lambda^{-1}A)^{-1} = \sum_{n=0}^\infty \lambda^{-n}A^n.
\]
Since $-\lambda(A - \lambda)^{-1} = (I - \lambda^{-1}A)^{-1}$, one obtains
\[
R_\lambda = -\frac{1}{\lambda}\sum_{n=0}^\infty \lambda^{-n}A^n = -\frac{1}{\lambda} - \sum_{n=1}^\infty \lambda^{-n-1}A^n,
\]
as desired.
\end{adromanlist}
\end{proof}

Next, we prove some algebraic properties of the resolvent. Hereby, $[A,B] = AB - BA$ denotes the commutator of two operators,
as usual.

\begin{proposition}\label{thm:resolvent-algebraic-properties}
Let $A,B \in \blinOps(\hilbertH)$. Then the following holds true.
\begin{romanlist}
\item\label{ite:commutativity-resolvent-operator} The resolvent commutes with the operator which means that
  \[ [A, R_\lambda(A) ] = 0 \quad\text{for all } \lambda \in \rset(A) \ . \]
\item\label{ite:commutativity-resolvent-itself}
   The values of the resolvent commute with each other that is
 \[ [R_\lambda (A), R_\mu (A)] = 0 \quad\text{for all } \lambda,\mu \in \rset(A) \ . \]
\item\label{ite:first-resolvent-identity}\textup{({\sffamily First resolvent identity})} For all $\lambda, \mu \in \rset(A)$
  \[ R_\lambda(A) - R_\mu (A)= (\lambda - \mu)R_\lambda(A) R_\mu(A) \ . \]
\item\textup{({\sffamily Second resolvent identity})} For all $\lambda \in \rset(A)\cap \rset(B)$
  \[ R_\lambda (A)- R_\lambda (B) = R_\lambda(A)\, (B-A) \, R_\lambda (B) \ . \]
\end{romanlist}
\end{proposition}

\begin{proof}
\begin{adromanlist}
\item Obviously  $[A, A - \lambda] = 0$, so
  \[
    0 = R_\lambda [A, A - \lambda]R_\lambda = R_\lambda A - AR_\lambda,
  \]
as desired.
\setcounter{enumi}{2}
\item We compute
\begin{align*}
(R_\lambda - R_\mu)(A - \mu)(A - \lambda) &= (R_\lambda A - \mu R_\lambda)(A - \lambda) - (A - \lambda)\\
&= (A - \mu)R_\lambda (A - \lambda) - (A - \lambda)\\
&= \lambda - \mu ,
\end{align*}
where we used part \ref{ite:commutativity-resolvent-operator} to commute $R_\lambda$ past $A$ in the second step.
Now multiplying both sides with $R_\lambda R_\mu $ from the right yields the desired equality.
\setcounter{enumi}{1}
\item
  For $\lambda = \mu$, one obviously has $[A_\lambda, A_\mu] = 0$. For $\lambda \neq \mu$, one concludes from
  \ref{ite:commutativity-resolvent-itself}
  \[
     R_\mu R_\lambda = \frac{R_\mu - R_\lambda}{\mu - \lambda} = \frac{R_\lambda - R_\mu}{\lambda - \mu} = R_\lambda R_\mu,
  \]
  so $[R_\lambda, R_\mu] = 0$ for $\lambda \neq \mu $ as well.
\setcounter{enumi}{3}
\item  The  last equality follows by
  \[ R_\lambda(A)\, (B-A) \, R_\lambda (B) =  R_\lambda(A)\, \big((B-\lambda)-(A-\lambda) \big) \, R_\lambda (B) =
     R_\lambda(A) - R_\lambda (B) \ . \]
\end{adromanlist}
\end{proof}

The resolvent $R_\bullet (A)$ also has some nice analytic properties which we are going to prove next.
\begin{proposition}\label{thm:resolvent-analytic-properties}
  The resolvent $R_\bullet (A) :\rset(A) \rightarrow \blinOps(\hilbertH)$, $\lambda \mapsto R_\lambda$ is continuous and complex
  differentiable with derivative given by
  \[
    R_\bullet(A)' :\: \rset(A) \rightarrow \blinOps(\hilbertH),  \: \lambda \mapsto \lim_{\mu \rightarrow \lambda} \frac{R_\mu - R_\lambda}{\mu - \lambda} =  R_\lambda^2
  \]
\end{proposition}

\begin{proof}
Fix $\lambda \in \rset(A)$ and $\varepsilon > 0$. Let $0 < \abs{\mu - \lambda} < \delta$, where
\[
  \delta = \min\left( \frac{\varepsilon}{2\norm{R_\lambda}^2},\, \frac{1}{2\norm{R_\lambda}} \right) \ .
\]
Note that $\mu \in \rset(A)$ by Proposition \ref{thm:resolvent-topological-properties}. Moreover, $\norm{(\mu - \lambda)R_\lambda} <1$, so $I - (\mu - \lambda)R_\lambda$ is invertible with norm less than $(1 - \norm{(\mu - \lambda) R_\lambda})^{-1}$ by Lemma \ref{thm:neumann-series}.
Now observe that the first resolvent identity can be rearranged to
\[
R_\mu =  R_\lambda[I - (\mu - \lambda)R_\lambda]^{-1} \ .
\]
Hence
\begin{align*}
\norm{R_\mu - R_\lambda} &\leq \abs{\mu - \lambda}\norm{R_\mu}\norm{R_\lambda} \\
&\leq \abs{\mu - \lambda} \norm{R_\lambda}^2 \norm{(I - (\mu - \lambda)R_\lambda)^{-1}}\\
&\leq \frac{\abs{\mu - \lambda} \norm{R_\lambda}^2}{1 - \norm{(\mu - \lambda)R_\lambda}}\\
&< \frac{\varepsilon/2}{1-1/2} = \varepsilon \ .
\end{align*}
This proves that $\lambda \mapsto R_\lambda$ is continuous.

As for complex differentiability, we simply use the first resolvent identity and continuity to conclude
\[
\lim_{\mu \rightarrow \lambda} \frac{R_\mu - R_\lambda}{\mu - \lambda} = \lim_{\mu \rightarrow \lambda} R_\mu R_\lambda = R_\lambda^2.
\]
\end{proof}

\begin{proposition}\label{thm:resolvent-limes-infinity}
Let $A \in \blinOps(\hilbertH)$. Then $\lambda R_\lambda \rightarrow -I$ as $\abs{\lambda} \rightarrow \infty$. In particular, $R_\lambda \rightarrow 0$ as $\abs{\lambda} \rightarrow \infty$.
\end{proposition}

\begin{proof}
Fix $\varepsilon > 0$.  For $\abs{\lambda} > \norm{A}$, we have by \Cref{thm:resolvent-topological-properties} \ref{ite:resolvent-expansion-large-argument}
\[
\lambda R_\lambda = -I - \sum_{n=1}^\infty \lambda^{-n}A^n.
\]
Since
\[
\norm{\sum_{n=1}^\infty \lambda^{-n}A^n} \leq  \frac{\norm{A}}{\abs{\lambda}-\norm{A}},
\]
one sees that $\lambda R_\lambda \rightarrow - I$ as $\abs{\lambda} \rightarrow \infty$. Similarly, for $\abs{\lambda} > \norm{A}$ one has
\[
\norm{R_\lambda} \leq \frac{1}{\abs{\lambda}} + \frac{1}{\abs{\lambda}}\sum_{n=1}^\infty \norm{\lambda^{-n}A^n} \leq \frac{1}{\abs{\lambda}} + \frac{1}{\abs{\lambda}} \frac{\norm{A}}{\abs{\lambda} - \norm{A}},
\]
which shows that $R_\lambda \rightarrow 0$ as $\abs{\lambda} \rightarrow \infty$.
\end{proof}

\begin{proposition}
For all $v, w \in \hilbertH$, the map
\[
 \inprod{ R_\bullet (A) v,  w} : \: \rset(A) \rightarrow \C , \: \lambda \mapsto \inprod{ R_\lambda v, w}
\]
is holomorphic with derivative
\[
 \inprod{ R_\bullet (A) v, w}' : \: \rset(A) \rightarrow \C , \: \lambda \mapsto \inprod{R_\lambda^2 v,  w}.
\]
\end{proposition}

\begin{proof}
Given $\lambda \in \rset(A)$, we compute
\begin{align*}
  \lim_{\mu \rightarrow \lambda} \frac{\inprod{R_\mu v, w} - \inprod{R_\lambda v,  w}}{\mu - \lambda} =
  \lim_{\mu \rightarrow \lambda} \frac{\inprod{(\mu - \lambda)R_\mu R_\lambda v,  w}}{\mu - \lambda} =
  \lim_{\mu \rightarrow \lambda} \inprod{ R_\mu R_\lambda v, w} = \inprod{R_\lambda^2v , w},
\end{align*}
where we have used the first resolvent identity in the first step and continuity of the inner product in the last.
\end{proof}

\begin{proposition}
The spectrum of an operator $A \in \blinOps(\hilbertH)$ is nonempty.
\end{proposition}

\begin{proof}
Suppose $\sigma(A) = \emptyset$, hence $\rset(A) = \C$. The map
\[
\C \to \C , \: \lambda \mapsto \inprod{R_\lambda v, w}
\]
then is entire for every $v, w \in \hilbertH$. Furthermore, one has  for $\norm{v}, \norm{w} \leq 1$
\[
  \abs{\inprod{R_\lambda v,  w}} \leq  \norm{R_\lambda} \norm{v} \norm{w} \leq \norm{R_\lambda} \ .
\]
Since $\lambda \mapsto \norm{R_\lambda}$ is continuous and $\norm{R_\lambda} \rightarrow 0$ as $\abs{\lambda} \rightarrow \infty$, one sees
that $\norm{R_\lambda}$ is bounded. Hence $\inprod{R_\bullet v,  w}$ is a bounded entire function, which  by Liouville's theorem implies
that it is zero for every pair $v, w \in \hilbertH$ with $\norm{v} = \norm{w} = 1$.
This entails that $R_\lambda = 0$ for every $\lambda \in \C$, which
is a contradiction to $R_\lambda$ being invertible. Hence $\sigma(A) \neq \emptyset$.
\end{proof}


\section{Stabilization and Weak Formulation}
\label{app:stab}

This appendix details the weak formulation of the Navier Stokes
equations instantiated in the software GRINS, as well as providing a
derivation of the $\tau$ stabilization terms.

In brief, our process is the following:

\begin{itemize}
 \item Cast Navier Stokes + Boussinesq equations into weak form
 \item Prepare as an operator $Lc=f$
 \item Calculate Fr\'echet derivative
 \item Separate into differential (P) and constant (Z) components,
       $L'[c] = P + Z$
 \item Choose stabilization operator such that $S = -P^*$
 \item Then stabilization has form, $a_h(c,\phi) = a(c,\phi) + \langle
       Lc,S\phi \rangle_\tau$
\end{itemize}

This is essentially the least-squares stabilization proposed by Hughes
and extended to natural convection by Becker and Braack.

%
% start the real work
%

\subsection{Weak Formulation of Equations of Interest}

We begin with the incompressible Navier-Stokes equations with Bousinesq
buoyancy,
\begin{align}
 \nabla \cdot u &= \, 0 \label{eq_cont}\\
 \frac{\partial u}{\partial t} + u \cdot \nabla u &= -\frac{1}{\rho}
 \nabla p + \nu \nabla^2 u + g \frac{T'}{T_0} \label{eq_mom}\\
 \rho c_p \frac{\partial T}{\partial t} + u \cdot \nabla T &= \nabla
 \cdot (k \nabla T) \label{eq_energy}
\end{align}
e.g. the continuity, momentum and energy equations, respectively. Our
state vector is $c =  \left[p,u,T \right]$. To cast these into
weak form we multiply by appropriate test
functions $\phi = \left[q,v,w \right] \in H^1_0(\Omega)$ and integrate over
the domain, $\Omega \in \mathbb{R}^n$. Our system of equations now
appears as,
\begin{align}
  \bigintsss_\Omega q \nabla \cdot u \, dx &= 0 \\
 \bigintsss_\Omega \dot u \cdot v \, dx +
 \bigintsss_\Omega  (u \cdot \nabla) \, u \cdot v \, dx &=
 \bigintsss_\Omega \frac{p}{\rho} \nabla \cdot v \, dx - \nu \bigintsss_\Omega \nabla u \cdot \nabla v
 \,dx + \bigintsss_\Omega g \frac{T'}{T_0} \cdot v \, dx \\
 \rho c_p \bigintsss_\Omega \dot T \cdot w \, dx + \bigintsss_\Omega (u
 \cdot \nabla) T \cdot w \, dx  &= -\bigintsss_\Omega (k \nabla T) \cdot
 \nabla w \, dx
\end{align}

where an ``over-dot'' denotes time diffentiation, e.g. $\dot u =
\frac{\partial u}{\partial t}$. Note that both the pressure term as well
as the viscous term were integrated by parts to reduce the required
order of the solution on those state variables.

We define the inner product by the shorthand notation $(u,v) =
\bigintsss_\Omega u\cdot v dx $, giving our equations the form,
\begin{align}
 (\nabla \cdot u, q) &= 0 \\
 (\dot u,v) + (u \cdot \nabla u, v) - (p,\nabla \cdot v) + \nu (\nabla
 u, \nabla v) &= (g \frac{T'}{T_0},v) \\
 \rho c_p (\dot T,w) + (u \cdot \nabla T,w) + (k \nabla T,\nabla w) &= 0.
\end{align}

This defines our weak form operator $a(c,\phi)$. Our full equations will
also include a stabilization term such that,
\begin{equation}
 a_h(c,\phi) = a(c,\phi) +  \langle Lc,S\phi \rangle_\tau.
\end{equation}

The subsequent section will define the operators L and S, so that we
might then fully define the stabilization term $\langle Lc,S\phi
\rangle_\tau$.

%
% subsection
%
\subsection{The operators L and S}

To form the stabilization terms,

\begin{equation}
 \langle Lc,S\phi \rangle_\tau
\end{equation}

we must define the operators L and S. The operator L is simply the PDEs
in Equations \ref{eq_cont} - \ref{eq_energy} written in operator form. S is
defined as the negative adjoint of the differential terms in L, e.g.
\begin{align}
 L'[c] = P + Z \\
 S = -P^*.
\end{align}
Where P are the differential terms, and Z the constant terms.

Our objective is now to construct the adjoint operator of L. This is
accomplished using the Fr\'echet derivative, which defines the
functional derivative on L. In general this is accomplished by taking
the first variation of a function $\Pi(u)$ around a base state, $u$,
\begin{equation}
 \delta\, \Pi(u) = \lim_{\epsilon \to 0} \frac{\Pi(u+\epsilon \hat u) -
  \Pi(u)}{\epsilon} =
  \frac{\partial \Pi(u +\epsilon \hat u)}{\partial \epsilon}
  \bigg|_{\epsilon = 0}
\end{equation}
for all $\hat u$ and $\epsilon > 0$ with $u + \epsilon \hat u \in
H^1_0(\Omega)$. This is recognizable as the G\^{a}teaux
derivative of the functional.

We now consider the first variation of state for the momentum equation
term by term. The convective term is,
\begin{align}
 \frac{\partial}{\partial \epsilon} (u + \epsilon \hat u) &\cdot \nabla
  (u + \epsilon \hat u) \\
 = \lim_{\epsilon \to 0} \hat u &\cdot \nabla (u + \epsilon \hat u) \\
 = \hat u & \cdot \nabla u \\
 = - u &\cdot \nabla \hat u
\end{align}
and the viscous term is,
\begin{align}
 \frac{\partial}{\partial \epsilon} \nabla^2 (u + \epsilon \hat u) \\
 = \nabla^2 \hat u
\end{align}
while the buoyancy term is,
\begin{align}
 \delta \left(-g \frac{T'}{T_0}\right) &= \delta \left( -g
 \frac{T-T_0}{T_0} \right) \\
 &= -g \frac{\partial}{\partial \epsilon} \left( \frac{T-T_0+\epsilon
 \hat T}{T_0} \right) \\
 &= -g \left( \frac{\hat T}{T_0} \right)
\end{align}

%Lagrangian is therefore, $\mathcal{L}$
and thus the full adjoint equation for momentum appears as
\begin{align}
 - u &\cdot \nabla \hat u - \nabla^2 \hat u = -\frac{1}{\rho} \nabla p.
\label{eq_adjmom}
\end{align}

The continuity equation is straightforward,
\begin{align}
 \frac{\partial}{\partial \epsilon} \nabla \cdot (u + \epsilon \hat
 u) = 0 \\
\nabla \cdot \hat u = 0.
\label{eq_adjcont}
\end{align}

Finally, consider the convective term of the energy equation,
\begin{align}
 \frac{\partial}{\partial \epsilon} u \cdot \nabla (T + \epsilon \hat T)
 = u \cdot \nabla \hat T
\end{align}
and the thermal diffusion term,
\begin{align}
 \frac{\partial}{\partial \epsilon} \cdot (-k \nabla (T + \epsilon \hat
 T)) = \nabla \cdot (-k \nabla \hat T).
\end{align}
The full adjoint energy equation is therefore,
\begin{align}
 u \cdot \nabla \hat T + \nabla \cdot (k \nabla \hat T) =0.
\label{eq_adjen}
\end{align}

We are now in a position to define the matricies L and S. L comes
directly from the PDEs in Equations \ref{eq_cont} - \ref{eq_energy} and is
defined as thus,

\begin{equation}
\renewcommand\arraystretch{2}
 L =
  \begin{pmatrix}
    0 & \nabla \cdot () & 0   \\
    \nabla \,() & u \cdot \nabla() - \nu \nabla^2() & -g \frac{()}{T_0}  \\
    0 & 0 & u \cdot \nabla() - \nabla \cdot (k \nabla() \,)
  \end{pmatrix}.
\end{equation}

While the S matrix is constructed from Equations
\ref{eq_adjmom}, \ref{eq_adjcont}, and \ref{eq_adjen} must be,
\begin{equation}
\renewcommand\arraystretch{2}
 S = -P^* =
  \begin{pmatrix}
    0 & \nabla \cdot () & 0   \\
    \nabla \,() & u \cdot \nabla() + \nu \nabla^2() &  -g \frac{()}{T_0}  \\
    0 & 0 & u \cdot \nabla() + \nabla \cdot k \nabla()
  \end{pmatrix}.
\end{equation}

%
% tau!
%
\subsection{Tau stabilization terms}

Finally, we may now form the $\tau$ stabilization terms,
\begin{equation}
 \langle Lc,S\phi \rangle_\tau.
\end{equation}
Where the operator $ \langle \cdot,\cdot \rangle_\tau. $ is shorthand
and denotes
\begin{equation}
 \langle u,v \rangle_\tau = \sum_K \tau_K (u,v)_K.
\end{equation}
Where $K$ denotes the FEM cells. Now, through what Becker and Braack
contemptibly referred to as ``elementary calculus'', we now discover our
stabilization terms,
\begin{align*}
 \langle Lc,S\phi \rangle_\tau = \sum_K \{ \quad &\tau_p (\nabla \cdot u,
 \nabla \cdot v) \\
 +\quad &\tau_u \,(\nabla p + u \cdot \nabla u - \nu \nabla^2 - g \frac{T'}{T_0},
 \nabla q) \\
 +\quad &\tau_u \, (\nabla p + u \cdot \nabla u - \nu \nabla^2 - g \frac{T'}{T_0},
 \nabla u \cdot \nabla v + \nu \nabla^2 v) \\
 +\quad &\tau_T \, (u \cdot T - \nabla \cdot (k \nabla T), \nabla u \cdot \nabla
 w + \nabla \cdot (k \nabla w)) \}.
\end{align*}


\documentclass[]{article}

\author{Braden Fineberg}
\usepackage[pdftex]{graphicx}
\usepackage{achicago}
\usepackage{fullpage}
\usepackage{ amssymb }
\usepackage{amsmath}
%\usepackage[top=1in, bottom=1in, left= 1 in, right=1in]{geometry}
\usepackage{setspace}
\usepackage[nopar]{lipsum} % for dummy text
\newcommand{\HRule}{\rule{\linewidth}{0.25mm}}

\usepackage{listings}
\usepackage{subfig}
\usepackage{color} %red, green, blue, yellow, cyan, magenta, black, white
\definecolor{mygreen}{RGB}{28,172,0} % color values Red, Green, Blue
\definecolor{mylilas}{RGB}{170,55,241}
\renewcommand\thesection{\Alph{section}}

\begin{document}

	\lstset{language=Matlab,%
		%basicstyle=\color{red},
		breaklines=true,%
		morekeywords={matlab2tikz},
		keywordstyle=\color{blue},%
		morekeywords=[2]{1}, keywordstyle=[2]{\color{black}},
		identifierstyle=\color{black},%
		stringstyle=\color{mylilas},
		commentstyle=\color{mygreen},%
		showstringspaces=false,%without this there will be a symbol in the places where there is a space
		numbers=left,%
		numberstyle={\tiny \color{black}},% size of the numbers
		numbersep=9pt, % this defines how far the numbers are from the text
		emph=[1]{for,end,break},emphstyle=[1]\color{red}, %some words to emphasise
		%emph=[2]{word1,word2}, emphstyle=[2]{style},
		basicstyle=\tiny,
	}

\input{../title.tex}

\section{CTMC states}
Given the integer nature of the problem, $X(t)$ will be integer values between 0 and max.

\section{Transition times out of given state}
The transition time will be the first occurrence of anything that changes the value of X. Given that the chance of a claim, premium or dividend is paid are all exponentially distributed, the transition time will also be exponential. Lack of transition is the case in which no event happens. Exponential distributions are additive, therefore, $$T_{leave} = min\{T_p, T_c, T_d\} \qquad T_{stay} = e^{-(\lambda + \alpha + \beta)t}$$
\begin{center}
	\begin{tabular}{|| c | c | c||}
		\hline
		Range & Event & $v_x$ \\ [0.5ex]
		\hline\hline
		A & premium & $\lambda$ \\
		\hline
		B & premium \& claim paid @ $X(t)$& $\lambda + \alpha$ \\
		\hline
		C & premium \& claim paid & $\lambda + \alpha$ \\
		\hline
		E & claim, dividend & $\lambda + \beta$ \\ [.5ex]
		\hline
	\end{tabular}
\end{center}

\section{Possible states going out of X(t) = x}
Given that there are only four options for each transition out of x,
\begin{center}
	\begin{tabular}{|| c | c | c||}
		\hline
		Range & Event & Possible States \\ [0.5ex]
		\hline\hline
		A & premium & x+1 \\
		\hline
		B & premium \& claim paid @ $X(t)$& x+1, net 0 \\
		\hline
		C & premium \& claim paid & x+1-c \\
		\hline
		E & claim, dividend & x-c-d\\ [.5ex]
		\hline
	\end{tabular}
\end{center}

\section{Transition probabilities}
Given Baye's theory,
\begin{center}
	P\{premium payment$|$ a transition happens\} =$\frac{\text{	P\{premium payment\}}\ \cap \text{ P\{transition happens\}}}{\text{P\{transition happens\}}}$
\end{center}
$$= \frac{\lambda}{\lambda+\alpha+\beta }$$
\begin{center}
	\begin{tabular}{|| c | c | c||}
		\hline
		Range & State j & Transition P \\ [0.5ex]
		\hline\hline
		A & 1 & $\frac{\lambda}{\lambda}$ \\
		\hline
		B & x+1 & $\frac{\lambda}{\lambda + \alpha}$ \\
		\hline
		B & 0 & $\frac{\alpha}{\lambda + \alpha}$ \\
		\hline
		C & x+1 & $\frac{\lambda}{\lambda + \alpha}$ \\
		\hline
		C & x-c & $\frac{\alpha}{\lambda + \alpha}$ \\
		\hline
		E & x-c & $\frac{\alpha}{\beta + \alpha}$\\
		\hline
		E & x-d & $\frac{\beta}{\beta + \alpha}$\\[.5ex]
		\hline
	\end{tabular}
\end{center}
\section{Simulation}
\begin{lstlisting}
function [X,T] = cfSim(lambda,beta, alpha, X0, c, d, Xr, Xmax, Tmax)

i = 1;
T(i) = 0;
X(i) = X0;
add = [1, -c, -d];

while T(i) < Tmax
	x=X(i);
	tp = exprnd(1/lambda);
	tc = exprnd(1/alpha);
	td = exprnd(1/beta);

	if x==0
		T(i+1) = T(i) + tp;
		X(i+1)=x+1;
	elseif  0<x && x<Xr
		[m, ind]=min([tp,tc]);
		T(i+1) = T(i) + m;
		X(i+1) = x + add(ind);
	elseif Xr <= x && x < Xmax
		[m, ind]=min([tp,tc,td]);
		T(i+1) = T(i) + m;
		X(i+1) = x + add(ind);
	elseif x==Xmax
		[m, ind] = min([realmax,tc,td]);
		T(i+1) = T(i) + m;
		X(i+1) = x + add(ind);
	else
		break;
	end
	i=i+1;
end
\end{lstlisting}

The code above was used to generate:

\begin{figure}[ht]
	\centering
	\includegraphics[width=0.7\linewidth]{figures/Cashflow}
	\caption{Cash On Hand @ time t}
	\label{fig:cashflow}
\end{figure}

\section{Kolmogorov's Forward Equation}
Given the lecture notes,
\begin{center}
	\begin{tabular}{|| c | c | c | c ||}
		\hline
		Range & State j @ t+1 & $v_x\cdot p_{xy}$ & $q_{xy}$\\ [0.5ex]
		\hline\hline
		A & 1 & $\frac{\lambda}{\lambda} \cdot \lambda$ & $\lambda$ \\
		\hline
		B & x+1 & $\frac{\lambda}{\lambda + \alpha} \cdot (\lambda + \alpha)$ & $\lambda$\\
		\hline
		B & 0 & $\frac{\alpha}{\lambda + \alpha}\cdot (\lambda + \alpha)$ & $\alpha$ \\
		\hline
		C & x+1 & $\frac{\lambda}{\lambda + \alpha} \cdot (\lambda + \alpha)$ & $\lambda$\\
		\hline
		C & x-c & $\frac{\alpha}{\lambda + \alpha}\cdot (\lambda + \alpha)$ & $\alpha$\\
		\hline
		D & x-c & $\frac{\alpha}{\lambda + \beta + \alpha}\cdot (\lambda + \alpha + \beta)$& $\alpha$\\
		\hline
		D & x+1 & $\frac{\lambda}{\lambda + \beta + \alpha}\cdot (\lambda + \alpha + \beta)$& $\lambda$\\
		\hline
		D & x-d & $\frac{\beta}{\lambda + \beta + \alpha}\cdot (\lambda + \alpha + \beta)$& $\beta$\\
		\hline
		E & x-d & $\frac{\beta}{\beta + \alpha}\cdot (\beta + \alpha)$& $\beta$\\
		\hline
		E & x-c & $\frac{\alpha}{\beta + \alpha}\cdot (\beta + \alpha)$& $\alpha$\\[.5ex]
		\hline
	\end{tabular}
\end{center}
Therefore, by plugging the various probabilities into the Kolmogorov's equations, I can generate the following:
\subsection{Range A, Y = 0}
$$\alpha\sum_{k=1}^{c} P_{xk}-\lambda P_{xy}$$
\subsection{Range B, 0 $<$ Y $<$ c}
$$\lambda P_{x,Y-1} + \alpha P_{x, Y+c} - (\lambda + \alpha)P_{xY}$$
\subsection{Range C, c $\leq$ Y $<$ X$_r$}
Given the dividend cannot surpass X$_r$,
$$\lambda P_{x,Y-1} + \alpha P_{x, Y+c} - (\lambda + \alpha)P_{xY}$$
Given the dividend can surpass X$_r$,
$$\lambda P_{x,Y-1} + \alpha P_{x, Y+c} + \beta P_{x, Y+d} - (\lambda + \alpha)P_{xY}$$
\subsection{Range D, X$_r$ $<$ Y $\leq$ Xmax}
The transition is close enough to Y that a claim or dividend can cause the transition:
$$\lambda P_{x,Y-1} + \alpha P_{x, Y+c} + \beta P_{x, Y+d} - (\lambda + \alpha + \beta)P_{xY}$$
When the transition cannot be caused by a claim or dividend:
$$\lambda P_{x,Y-1} - (\lambda + \alpha + \beta)P_{xY}$$
When a transition can only be caused a claim or premium:
$$\lambda P_{x,Y-1} + \alpha P_{x, Y+c} - (\lambda + \alpha + \beta)P_{xY}$$
When the transition can only be caused by a dividend or premium:
$$\lambda P_{x,Y-1} + \beta P_{x, Y+d} - (\lambda + \alpha + \beta)P_{xY}$$
\subsection{Range E, Y = Xmax}
$$\lambda P_{x,Y-1} - (\alpha + \beta)P_{xY}$$

\section{Kolmogorov's Backwards Equations}
\subsection{Range A, X = 0}
$$\lambda P_{X+1, y} - P_{X,y}$$
\subsection{Range B, 0 < X < c}
$$\lambda P_{X+1, y} + \alpha P_{0, y} - (\lambda + \alpha) P_{X,y}$$
\subsection{Range C, c $\leq$ X < X$_r$}
$$\lambda P_{X+1, y} + \alpha P_{x-c, y} - (\lambda + \alpha) P_{X,y}$$
\subsection{Range D, X$_r$ $\leq$ X $<$ Xmax}
$$\lambda P_{X+1, y} + \alpha P_{x-c, y} + \beta P_{x-d, y} - (\lambda + \alpha + \beta) P_{X,y}$$
\subsection{Range E, X = Xmax}
$$\alpha P_{x-c, y} + \beta P_{x-d, y} - (\alpha + \beta) P_{X,y}$$

\section{Solution of Kolmogorov’s equations}
\begin{lstlisting}
function [R]=Kolm(lambda,alpha,beta,c,d,Xr,Xmax)
	%UNTITLED2 Summary of this function goes here
	%   Detailed explanation goes here
	R=zeros(Xmax+1);
	R(1, 1) = -lambda;
	R(Xmax+1, Xmax)=lambda;
	R(Xmax+1, Xmax+1)=-(alpha+beta);

	for i=1:Xmax+1
		if i>=2 && i <=Xr-d
			R(i,i)=-(lambda+alpha);
			R(i, i-1) = lambda;
			R(i, i+c) = alpha;
		elseif i>=Xr-d+1 && i<=Xmax-d+1
			R(i, i-1) = lambda;
			R(i,i+c)=alpha;
			R(i,i+d)=beta;
			if i>= Xr+1
				R(i,i)=-(lambda+alpha+beta);
			else
				R(i,i)=-(lambda+alpha);
		end
			elseif i >= Xmax-d+2 && i<= Xmax-c+1
			R(i,i-1)=lambda;
			R(i,i+c)=alpha;
			R(i,i)=-(lambda+alpha+beta);
		elseif i>= Xmax-c+2 && i<=Xmax
			R(i,i-1)=lambda;
			R(i,i)=-(lambda+alpha+beta);
		end
	end
end
\end{lstlisting}
The code above was used to generate the following:
\begin{figure}[!ht]
	\centering
	\includegraphics[width=0.7\linewidth]{figures/pmf}
	\caption{Probability Mass Function of Realizations}
	\label{fig:pmf}
\end{figure}
\pagebreak
\section{PMF of Dividends}
\begin{lstlisting}
s = 0;
for i=1:1000
[x,~] = cfSim(lambda,beta, alpha, X0, c, d, Xr, Xmax, Tmax);
a = x(2:end);
diff = a-x(1:end-1);
if ~isempty(find(diff == -d, 1))
s = s+1;
end
end
disp(s/100);

%%
pmf = zeros(100, 21);
for i=1:100
	[x,t] = cfSim(lambda,beta, alpha, X0, c, d, Xr, Xmax, Tmax);
	a = x(2:end);
	diff = a-x(1:end-1);

	T = .25:.25:5;
	edges = zeros(1, length(T));
	for tm = 1:length(T)
		[~,ind] = find(t<=T(tm), 1, 'last');
		edges(tm) = ind;
		end

		edges(edges > length(diff)) = length(diff);

		edges = [1,edges];
		p = zeros(1, 21);
		for e=2:21
		p(e-1) = length(find(diff(edges(e-1):edges(e)) == -d));
	end
	pmf(i, :) = p/sum(p);
end
pmf(any(isnan(pmf), 2),:)=[];
avg = mean(pmf);
stairs(0:20, avg);
grid on;
title('average pmf of paying a dividend at in quarter q (100 trials)');
xlabel('q');
ylabel('pmf');
saveas(gcf, 'pmfDividend.png');
\end{lstlisting}
The code above was used to generate the following:
\begin{figure}[ht]
	\centering
	\includegraphics[width=0.7\linewidth]{figures/pmfDividend}
	\caption{Average pmf, 100 trials}
	\label{fig:pmfdividend}
\end{figure}
\pagebreak
\\
This indicates that over 20 quarters, the insurance company will almost always pay a dividend. However, the probability that the company will pay a dividend in quarter q varies from approx. 7.5\% to 3.75\%

\end{document}