\section{Elliptic systems: linear elasticity}

In this section we review the foundations of the analysis of elliptic
(systems of) partial differential equations (\putindex{PDE}) and apply
them, again with the purpose of reminding us of the analytical tools
in mind, to the Lamé-Navier-equations of linear elasticity. While this
is expected to be knowledge you already bring into this class, it will
help us putting the analysis of mixed finite element methods into
perspective.  A short derivation of the Lamé-Navier equations is in
\cref{sec:lame-navier} in the appendix. For comparison,
consider~\cite{Braess97,Braess13}.

\subsection{Example: weak form of the Lamé-Navier equations}
\begin{Definition}{weak-lame-navier}
  The weak formulation of the Lamé-Navier boundary value problem in
  linear elasticity with homogeneous displacement boundary conditions,
  namely
  \begin{gather}
    \label{eq:mixedintro:lame1}
    \begin{aligned}
      -\div \sigma(\vx) &= \vf(\vx) & \vx&\in\domain,\\
      \sigma(\vx) &= 2\mu \strain \vu(\vx) + \lambda \operatorname{tr} \strain \vu(\vx) \id\\
%      \vu(\vx) &= 0 & x&\in\Gamma_D,
    \end{aligned}
  \end{gather}
  is: find $\vu\in \vV = \vH^1_{0}(\domain;\R^d)$ such that
  \begin{gather}
    \label{eq:mixedintro:lame2}
    a(\vu,\vv) \equiv 2\mu\form(\strain \vu, \strain \vv)
    + \lambda \form(\div \vu, \div \vv)
    = \form(\vf,\vv)
    \quad\forall \vv\in \vV.
  \end{gather}
    Here, $\strain \vu = \nicefrac12\bigl(\nabla u + (\nabla u)^\transpose\bigr)$ is the symmetric gradient.
\end{Definition}

\begin{Problem}{frobenius}
  Given the vector space of square matrices $X = \R^{d\times d}$ with the
  Frobenius inner product
  \begin{gather}
    \label{eq:mixedintro:frobenius}
    \scal(A,B) = A:B = \sum_{ij} a_{ij}b_{ij}.
  \end{gather}
  Show that the subspaces of symmetric and skew-symmetric matrices,
  respectively, are orthogonal to each other and $X$ is the direct sum
  of those.
\begin{solution}
\begin{enumerate}
\item First, we note that for every matrix $A$, there holds
  \begin{gather}
    A = A_S + A_A, \quad A_S = \frac{A+A^\transpose}2, \quad A_A = \frac{A-A^\transpose}2.
  \end{gather}
  Furthermore, we see that $A_S$ is symmetric and $A_A$ is
  skew-symmetric. The sum of two symmetric matrices is symmetric,
  the same for skew-symmetric. Thus, the sets $X_S$ and $X_A$ of
  symmetric and skew-symmetric matrices, respectively, are vector
  spaces and $X=X_S+X_A$.

\item The only matrix which is symmetric and skew-symmetric is
  zero. Thus, $X=X_S\oplus X_A$.

\item It remains to show orthogonality. To this end, we first note
  that for any skew-symmetric matrix $A$, there holds $a_{ii}=0$
  for all diagonal elements. Thus, the inner product of $A$ with a
  symmetric matrix $S$ is
  \begin{align}
    \scal(A,S) &= \sum_{i\neq j} a_{ij}s_{ij}\\
               &= \sum_{i<j} (a_{ij}s_{ij} + a_{ji}s_{ji})\\
               &= \sum_{i<j} (a_{ij}-a_{ji})s_{ij}\\
               &= 0.
  \end{align}
\end{enumerate}
\end{solution}
\end{Problem}

\begin{Problem}{weak-lame-navier}
  Show that the weak formulation equation~\eqref{eq:mixedintro:lame2}
  in \slideref{Definition}{weak-lame-navier} indeed is indeed obtained
  from the classical formulation ~\eqref{eq:mixedintro:lame2} in that
  definition by integration by parts and the application of boundary
  conditions.
\end{Problem}

\begin{Definition}{lame-galerkin}
  The \define{conforming} \define{Galerkin approximation} of the weak
  formulation in \slideref{Definition}{weak-lame-navier} consists of the following steps:
  \begin{enumerate}
  \item Choose a finite dimensional subspace $\vV_h\subset \vV$
  \item Find $\vu_h \in \vV_h$, such that for all $\vv\in \vV_h$ there holds
  \begin{gather}
    \label{eq:mixedintro:lame3}
    a(\vu_h,\vv_h) \equiv 2\mu\form(\strain{\vu_h}, \strain{\vv_h})
    + \lambda \form(\div{\vu_h}, \div{\vv_h})
    = \form(\vf,\vv_h).
  \end{gather}
  \end{enumerate}
\end{Definition}

\begin{Problem}{weak-lame-well-posedness}
  Without reading any further in the notes, try to remember the means
  of analyzing the well-posedness of this weak formulation. What are
  the assumptions on the weak formulation?

  Also, try to remember the means of analyzing well-posedness and
  approximation properties for the conforming Galerkin
  approximation. What are the assumptions on the weak formulation?
\end{Problem}

\subsection{The finite element toolbox}

We review the abstract setting of Galerkin approximations and the
finite element method and apply those to the Lamé-Navier equations.

\begin{Notation}{vh-finite}
  Spaces with subscript $h$ in these notes always denote finite
  dimensional spaces used for discretization.

  If nothing else is stated, we denote the solution of the PDE
  boundary value problem by $u$\index{u@$u, \vu$} or $\vu$, while
  $u_h$ or $\vu_h$ \index{uh@$u_h, \vu_h$} denote its finite element
  discretization.

  The index $h$ is related to a finite element mesh $\mesh_h$, where
  $h$ indicates the mesh size. We use this notation even for adaptive
  and high order finite elements, where there is no single mesh size
  characterizing the discretization.
\end{Notation}

\begin{Definition}{elliptic-abstract-weak}
  The \define{weak formulation} of a \define{boundary value problem}
  for a PDE in abstract form involves a function space $V$, a bilinear
  form $a(\cdot,\cdot)$ on $V$, and a right hand side $f\in V^*$. It
  reads: find $u\in V$ such that
  \begin{gather}
    \label{eq:elliptic-abstract-weak}
    a(u,v) = f(v) \qquad\forall v\in V.
  \end{gather}
  For its \define{Galerkin approximation}, choose a subspace
  $V_h\subset V$ and solve: find $u_h\in V_h$ such that
  \begin{gather}
    \label{eq:elliptic-abstract-galerkin}
    a(u_h,v_h) = f(v_h) \qquad\forall v_h\in V_h.
  \end{gather}
\end{Definition}

It turns out that the analysis of the weak formulation as well as its
Galerkin approximation hinges on a single set of assumptions, which we
present here.

\begin{Assumption}{coercive}
  Let $V$ be a Hilbert space and let $a(.,.)$ be a \putindex{bilinear
    form} on $V$.  The bilinear form is
  \textbf{bounded}\index{bilinear form!bounded}, i. e. there is a
  constant $M$ such that
  \begin{gather}
    a(u,v) \le M \norm{u}_V \norm{v}_V \qquad \forall u,v\in V.
  \end{gather}
  The bilinear form is \textbf{coercive}\index{bilinear form!coercive}
  or \textbf{elliptic}\index{bilinear form!elliptic}, i.e. there is a
  constant $\ellipa > 0$ such that
  \begin{gather}
    \label{eq:infsup:elliptic}
    a(u,u) \ge \ellipa \norm{u}_V^2 \qquad \forall u\in V.
  \end{gather}
\end{Assumption}

Note that typically the space $V$ and its inner product are chosen
such that the assumption holds, not the other way round.

The analytical results can be summarized by

\begin{Theorem*}{fem-toolbox}{The finite element toolbox (elliptic)}
  Let \slideref{Assumption}{coercive} hold. Let furthermore
  $V_h\subset V$. Then, there holds
  \begin{enumerate}
  \item (\putindex{Lax-Milgram lemma}) There is a unique solution to
    the weak formulation in
    \slideref{Definition}{elliptic-abstract-weak}. By the subspace
    property, this result is inherited by the Galerkin
    approximation. There holds
    \begin{gather}
      \norm{u}_V \le \tfrac1{\ellipa} \norm{f}_{V^*},
      \qquad
      \norm{u_h}_V \le \tfrac1{\ellipa} \norm{f}_{V^*}.
    \end{gather}
  \item (\putindex{Céa's lemma}) The error of the Galerkin
    approximation admits the estimate (\putindex{quasi-optimality})
    \begin{gather}
      \norm{u-u_h}_V \le \tfrac{M}{\ellipa}\inf_{v_h\in V_h}\norm{u-v_h}_V.
    \end{gather}
  \end{enumerate}
\end{Theorem*}

\begin{intro}
  The form $a(\cdot,\cdot)$ is symmetric and thus semi-definite on $V$. It can
  also be bounded easily by the $H^1$-norm. But, for well-posedness of
  the weak formulation, we also require ellipticity. This question is
  indeed not trivial and rests on the fact that for a function
  $u\in V$, such that $\nabla u$ is skew-symmetric everywhere, there
  holds $\strain u\equiv 0$. Thus, such functions must be excluded by
  the boundary conditions. Note, that in particular for rigid body
  translations and rotations $\strain u = 0$. Therefore, the Dirichlet
  boundary conditions must exclude such solutions.

  The condition needed for well-posedness is called Korn inequality,
  and it will be posed as an assumption. We will give a proof for a
  simple case and refer the readers to a plethora of articles on more
  complicated cases.
\end{intro}

% \begin{Lemma*}{korn-inequality}{Korn inequality}
%   There is a constant $c_K>0$ such that
%   \begin{gather}
%     \label{eq:mixedintro:korn}
%     c_K \norm{\vv}_{\vH^1(\domain)}
%     \le \norm{\strain \vv}_{\vL^2(\Omega)}
%     \qquad\forall \vv\in \vH^1_0(\domain) \cap \vH^2(\domain).
%   \end{gather}
% \end{Lemma*}

\begin{Problem*}{korn-inequality}{Korn inequality}
  Prove that the inequality
  \begin{gather}
    \label{eq:mixedintro:korn}
    c_K \norm{\vv}_{\vH^1(\domain)}
    \le \norm{\strain \vv}_{\vL^2(\Omega)}
    \qquad\forall \vv\in \vH^1_0(\domain) \cap \vH^2(\domain).
  \end{gather}
  holds with a constant
  \begin{gather}
    c_K = \frac1{\sqrt2(1+c_P)},
  \end{gather}
  where $c_P$ is the constant of the \putindex{Poincaré-Friedrichs inequality}.
  Use the intermediate result
  \begin{gather}
    2 \norm{\strain u}^2_{L^2(\domain)} = \norm{\nabla u}_{L^2(\domain)}^2
    + \norm{\div u}_{L^2(\domain)}^2,
  \end{gather}
  and prove it by integration by parts.
\end{Problem*}

While the inequality is of high importance in the mathematical and
numerical analysis of problems in continuum mechanics, it a peripheral
topic to this class. We point out, that there is a whole family of
Korn inequalities ensuring the definiteness of the strain bilinear
form on certain spaces. We have only stated the simplest version
here. See~\cite{DuvautLions76,DesvillettesVillani02} for some indepth
analysis.

\begin{Problem}{elasticity-standard-h1}
  Let the space $V=H^1_0(\domain;\R^d)$ be equipped with its standard inner
  product $\scal(\vu,\vv)$ with the bilinear form of the
  Lamé-Navier equations and the corresponding norm $\norm{.}_{\vH^1}$.

  Show using the elliptic finite element toolbox in \slideref{Theorem}{fem-toolbox}
  \begin{enumerate}
  \item The weak formulation has a unique solution for which there holds
    \begin{gather}
      \norm{\vu}_{\vH^1} \le \frac1{2c_K\mu} \norm{\vf}_{\vH^{-1}}.
    \end{gather}
  \item There holds the error estimate
    \begin{gather}
      \norm{\vu-\vu_h}_{\vH^1}
      \le \frac{2\mu+d\lambda}{2c_K\mu}
      \inf_{\vv_h\in V_h} \norm{\vu-\vv_h}_{\vH^1}.
    \end{gather}
    Hint: there are divergence-free functions ($\div \vv=0$) and others.
  \end{enumerate}
  \begin{solution}
    \begin{enumerate}
    \item For well-posedness, we want to apply the Lax-Milgram
      lemma. Thus, we have to show that the bilinear form is bounded
      and elliptic.\marginpar{Todo}
   \item For the $H^1$-error estimate we make use of Korn's inequality to note\marginpar{Todo: this is outdated}
   \begin{align}
    \norm{u}_V^2=2\mu\norm{\strain u}_0^2+ \lambda \norm{\div u}_0^2\geq
    2\mu(c_K^2\norm{u}_1^2-\norm{u}_0^2)
   \end{align}
   and on the other hand
   \begin{align}
    \norm{u}_V^2=2\mu\norm{\strain u}_0^2+ \lambda \norm{\div u}_0^2\leq
    (2\mu+\lambda d^2)\norm{u}_1^2
   \end{align}
   where we used
   \begin{align}
    \norm{\nabla \cdot u}_0^2 = \norm{\sum_i \partial_i u_i}_0^2 &\leq (\sum_{i,j} \norm{\partial_j u_i}_0\delta_{i,j})^2\\
    & \leq \sum_{i,j} \left(\norm{\partial_j u_i}_0^2\right) d^2 \leq \norm{u}_1^2 d^2.
   \end{align}
   Using Poincaré's inequality
   \begin{align}
    \norm{u}_0\leq C_P |u|_1
   \end{align}
   we further notice
   \begin{align}
    \norm{u}_V^2+2\mu\norm{u}_0^2\leq (1+C_P^2) \norm{u}_V^2.
   \end{align}
   Combining these estimates gives
   \begin{align}
   2\mu c_K^2\norm{u}_1^2\leq
    \norm{u}_V^2+2\mu \norm{u}_0^2\leq
    (1+C_P^2)\norm{u}_V^2\leq
    (1+C_P^2)(2\mu+\lambda d^2)\norm{u}_1^2.
   \end{align}
   and therefore
   \begin{align}
    \norm{u-u_h}_1^2\leq \inf_{v\in V_h} \frac{(1+C_P^2)(2\mu+\lambda d^2)}{2\mu c_K^2}\norm{u-v_h}_1^2.
   \end{align}
   If the solution is even $H_0^1(\Omega)$-regular, we can improve the estimate to
   \begin{align}
    \norm{u}_1^2\leq \inf_{v\in V_h}\frac{(\mu+(\lambda+\mu) d^2)}{\mu}\norm{u-v_h}_1^2.
   \end{align}
   by using the identity
   \begin{align}
    (\strain u, \strain v) = (\nabla u, \nabla v) + (\nabla \cdot u, \nabla \cdot v).
   \end{align}

   \item A more refined estimate would be
   \begin{align}
    \norm{u-u_h}_1^2\leq \inf_{v\in V_h}\frac{(1+C_P^2)}{2\mu c_K^2}\left(2\mu\norm{u-v_h}_1^2+\lambda \norm{\nabla\cdot (u-v_h)}_0^2\right).
   \end{align}
   Choosing a divergence-preserving interpolation operator, the second term vanishes.
  \end{enumerate}
\end{solution}
\end{Problem}

If the bilinear form is symmetric, there is an even more elegant way of
doing the analysis: we can use the bilinear form as an inner product
itself. This leads to our second, albeit somewhat more restricted toolbox.

\begin{Theorem*}{fem-toolbox-riesz}{The finite element toolbox (symmetric)}
  Let $a(\cdot,\cdot)$ be a symmetric, positive definite bilinear form
  on a space $V$. We equip $V$ with this bilinear form and the \define{energy norm}:
  \begin{gather}
    \scal(u,v)_V = a(u,v)
    ,\qquad \norm{v}_V = \sqrt{a(v,v)} \qquad \forall u,v\in V.
  \end{gather}
  Let furthermore $V_h\subset V$. Then, there holds
  \begin{enumerate}
  \item (Riesz representation theorem) There is a unique solution to
    the continuous and discrete weak formulations in
    \slideref{Definition}{elliptic-abstract-weak}, respectively. There holds
    \begin{gather}
      \norm{u}_V = \norm{f}_{V^*},
      \qquad
      \norm{u_h}_V = \norm{f}_{V^*}.
    \end{gather}
  \item (\putindex{Céa's lemma}) The approximation error admits the
    estimate
    \begin{gather}
      \norm{u-u_h}_V \le \inf_{v_h\in V_h}\norm{u-v_h}_V.
    \end{gather}
  \end{enumerate}
\end{Theorem*}

\begin{Problem}{elasticity-standard}
  Let the space $V=H^1_0(\domain;\R^d)$ be equipped with the inner
  product $\scal(\vu,\vv) = a(\vu,\vv)$, the bilinear form of the
  Lamé-Navier equations and the corresponding norm $\norm{.}_\vV$.

  Show using the symmetric finite element toolbox \slideref{Theorem}{fem-toolbox-riesz}
  \begin{enumerate}
  \item The weak formulation has a unique solution for which there holds
    \begin{gather}
      \norm{\vu}_\vV \le \norm{\vf}_{\vV^*}.
    \end{gather}
  \item The ``energy estimate'' for conforming finite element
    approximation with a space $\vV_h\subset \vV$
    \begin{gather}
      \norm{\vu-\vu_h}_\vV = \inf_{\vv_h\in \vV_h} \norm{\vu-\vv_h}_\vV.
    \end{gather}
  \end{enumerate}
\begin{solution}
We consider the problem:
  Find $u\in V = H^1_{\Gamma_D}(\domain;\R^d)$ such that
  \begin{gather}
    a(u,v) \equiv 2\mu\form(\strain u, \strain v)
    + \lambda \form(\div u, \div v)
    = \form(f,v)
    \quad\forall v\in V.
  \end{gather}
  where the norm on $V$ is given through $a$ as
  \begin{align}
    \norm{u}_V^2=a(u,u)=2\mu\norm{\strain u}_0^2+ \lambda \norm{\div u}_0^2
    \end{align}
  \begin{enumerate}
   \item Testing symmetrically ($v=u$) gives
   \begin{align}
   \norm{u}_V=\frac{a(u,u)}{\norm{u}_V}= \frac{\form(f,u)}{\norm{u}_V} \leq \norm{f}_{V^*}.
   \end{align}
   \item Applying Céa's lemma gives
   \begin{align}
   \norm{u-u_h}_V&=\frac{a(u-u_h, u-u_h)}{\norm{u-u_h}_V}\\
               &=\frac{a(u-u_h, u-v_h)}{\norm{u-u_h}_V}\leq \norm{u-v_h}_V\quad \forall v_h \in V_h
   \end{align} and hence $\norm{u-u_h}_V = \inf_{v_h\in V_h} \norm{u-v_h}_V$.
   \item For the $H^1$-error estimate we make use of Korn's inequality to note
   \begin{align}
    \norm{u}_V^2=2\mu\norm{\strain u}_0^2+ \lambda \norm{\div u}_0^2\geq
    2\mu(c_K^2\norm{u}_1^2-\norm{u}_0^2)
   \end{align}
   and on the other hand
   \begin{align}
    \norm{u}_V^2=2\mu\norm{\strain u}_0^2+ \lambda \norm{\div u}_0^2\leq
    (2\mu+\lambda d^2)\norm{u}_1^2
   \end{align}
   where we used
   \begin{align}
    \norm{\nabla \cdot u}_0^2 = \norm{\sum_i \partial_i u_i}_0^2 &\leq (\sum_{i,j} \norm{\partial_j u_i}_0\delta_{i,j})^2\\
    & \leq \sum_{i,j} \left(\norm{\partial_j u_i}_0^2\right) d^2 \leq \norm{u}_1^2 d^2.
   \end{align}
   Using Poincaré's inequality
   \begin{align}
    \norm{u}_0\leq C_P |u|_1
   \end{align}
   we further notice
   \begin{align}
    \norm{u}_V^2+2\mu\norm{u}_0^2\leq (1+C_P^2) \norm{u}_V^2.
   \end{align}
   Combining these estimates gives
   \begin{align}
   2\mu c_K^2\norm{u}_1^2\leq
    \norm{u}_V^2+2\mu \norm{u}_0^2\leq
    (1+C_P^2)\norm{u}_V^2\leq
    (1+C_P^2)(2\mu+\lambda d^2)\norm{u}_1^2.
   \end{align}
   and therefore
   \begin{align}
    \norm{u-u_h}_1^2\leq \inf_{v\in V_h} \frac{(1+C_P^2)(2\mu+\lambda d^2)}{2\mu c_K^2}\norm{u-v_h}_1^2.
   \end{align}
   If the solution is even $H_0^1(\Omega)$-regular, we can improve the estimate to
   \begin{align}
    \norm{u}_1^2\leq \inf_{v\in V_h}\frac{(\mu+(\lambda+\mu) d^2)}{\mu}\norm{u-v_h}_1^2.
   \end{align}
   by using the identity
   \begin{align}
    (\strain u, \strain v) = (\nabla u, \nabla v) + (\nabla \cdot u, \nabla \cdot v).
   \end{align}

   \item A more refined estimate would be
   \begin{align}
    \norm{u-u_h}_1^2\leq \inf_{v\in V_h}\frac{(1+C_P^2)}{2\mu c_K^2}\left(2\mu\norm{u-v_h}_1^2+\lambda \norm{\nabla\cdot (u-v_h)}_0^2\right).
   \end{align}
   Choosing a divergence-preserving interpolation operator, the second term vanishes.
  \end{enumerate}
\end{solution}
\end{Problem}

\begin{Theorem*}{interpolation-elliptic}{Interpolation estimate}
  Let $\mesh_h$ be a finite element mesh of size $h$, that is, the
  maximal diameter of a mesh cell is $h$. Let the finite element space
  be chosen piecewise polynomial such that
  \begin{gather}
    V_h = \bigl\{ v \in H^1_0(\domain) \;\big|\;
    v_{|\cell} \in \mathcal P(\cell) \;\forall \cell\in\mesh_h \bigr\}.
  \end{gather}
  For each cell $\cell\in\mesh_h$, let $\mathcal P(\cell)$ be a
  polynomial space containing $\P_k$. Then, there is an interpolation
  operator $I_h\colon H^1_0(\domain)\cap H^{k+1}(\domain)\to V_h$,
  such that for $m\le k$ there holds
  \begin{gather}
    \abs{u-I_h u}_{m} \le c_I h^{k+1-m}\abs{u}_{k+1},
  \end{gather}
  with an \define{interpolation constant} $c_I$ independent of $h$ and
  $u$, but depending on $k$, $m$, and \putindex{shape regularity} of the mesh.
\end{Theorem*}

\begin{Theorem}{elliptic-convergence}
  On a sequence of meshes $\mesh_h$ with $h\to 0$ and uniform shape
  regularity, the error of the finite element solution $\vu_h$ of the
  Lamé-Navier equations is bounded by
  \begin{gather}
    \norm{\vu-\vu_h}_{1} \le c_I h^k \frac{2\mu+d\lambda}{2c_K\mu}
    \abs{\vu}_{k+1},
  \end{gather}
  if the true solution $\vu$ is sufficiently regular.
\end{Theorem}

\subsection{Where things go wrong}
\begin{example}
  We study the finite element approximation of the following problem:
  a square sheet of elastic material is hanging from the top, subject
  to gravity acting as body force pointing downward. We choose $\mu=1$
  and vary $\lambda$ from 1 to
  $10^5$. Figure~\ref{fig:elasticity-compressibility} shows
  approximations with standard bilinear finite elements (red) and a
  ``good'' approximation (blue). In fact, the red solution for
  $\lambda = 10^5$ is almost identical with the undeformed
  configuration, although the material is not as hard. This phenomenon
  was discovered early in finite element history and is called
  ``locking''.
\end{example}

\begin{figure}[tp]
  \centering
  \subfigure[{$\lambda = 1$}]{\includegraphics[width=.45\textwidth]{./graph/elasticity/stalactite-0}}
  \subfigure[{$\lambda = 10$}]{\includegraphics[width=.45\textwidth]{./graph/elasticity/stalactite-1}}
  \subfigure[{$\lambda = 100$}]{\includegraphics[width=.45\textwidth]{./graph/elasticity/stalactite-2}}
  \subfigure[{$\lambda = 1000$}]{\includegraphics[width=.45\textwidth]{./graph/elasticity/stalactite-3}}
  \subfigure[{$\lambda = 10000$}]{\includegraphics[width=.45\textwidth]{./graph/elasticity/stalactite-4}}
  \subfigure[{$\lambda = 100000$}]{\includegraphics[width=.45\textwidth]{./graph/elasticity/stalactite-5}}
  \caption{Approximation with standard finite elements (red) and
    ``good'' elements (blue) for $\mu=1$ and different values of
    $\lambda$.}
  \label{fig:elasticity-compressibility}
\end{figure}

\begin{example}
  In a second example, we cook up a right hand side function $\vf$
  such that we know the solution to the Lamé-Navier equations. Indeed,
  we take a given solution $\vu$ and apply the differential operator
  to compute $\vf$.

  The \cref{fig:elasticity-locking} shows the $H^1$-seminorm of the
  error over the mesh size (represented by its negative dyadic
  logarithm). The black triangle denotes the convergence order $h$,
  the expected convergence of the bilinear finite elements used in
  this example.

  We see that for $\lambda=1$ and $\lambda=10$, we get the expected
  curves with the correct convergence rate. For larger values of
  $\lambda$ things seem to go wrong. There is no convergence in the
  beginning, but it seems that the expected rates are recovered on
  finer meshes.
\end{example}

\begin{figure}[tp]
  \centering
  \includegraphics[width=.9\textwidth]{mixed/graph/elasticity/locking}
  \caption{Errors for a manufactured solution with different values of $\lambda$.}
  \label{fig:elasticity-locking}
\end{figure}

\begin{intro}
  As we could see in the preceding examples, approximation
  of the solution to the Lamé-Navier equations becomes difficult, if
  $\lambda \gg \mu$. In this case, the material is called almost
  incompressible, since the divergence measures compression or
  dilation and the dominating divergence term forces the divergence of
  the solution to be small. These cases are important in engineering
  and they initiated a lot of the research that resulted in the topics
  of this class. The phenomenon observed here, that the approximation
  of the solution to an elasticity problem deteriorates with
  increasing $\lambda$ is called \define{locking}.
\end{intro}

\begin{Problem}{divergence-q1}
  On $\domain = (-1,1)^2$ use the finite element mesh
  \begin{center}
  \includegraphics[width=.25\textwidth]{./fig/patch1m.tikz},
  \end{center}
  and on each cell the bilinear shape function space $\Q_1$, that is
  \begin{gather}
    \vV_h = \bigl\{ \vv\in \vH^1_0(\domain) \;\big|\;
    \vv_{|\cell_i} \in \Q_1\quad i=1,2,3,4\bigr\}.
  \end{gather}
  Show that for functions in $\vV_h$ there holds
  \begin{gather}
    \norm{\div \vv_h}_{L^2(\domain)} = 0
    \qquad\Rightarrow\qquad
    \vv_h = 0.
  \end{gather}
\end{Problem}

\subsection{A mixed formulation}

\begin{Definition}{robust-discretization}
  We call a discretization of a parameterized (system of) PDE robust
  with respect to the parameters or simply \define{robust}, if in an
  error estimate of the abstract form
  \begin{gather}
    \norm{u-u_h}_X \le c h^\alpha \norm{u}_Y
  \end{gather}
  the constant $c$ is independent of the parameters.
\end{Definition}

A way to approach this problem is the introduction of an auxiliary variable
\begin{gather}
  p = -\lambda \div u.
\end{gather}
Entering this definition into the Lamé-Navier equations, we obtain
the following weak formulation.

\begin{Definition}{displacement-pressure}
  The \define{displacement-pressure formulation} of the Lamé-Navier
  equations reads: find a pair $(u,p) \in V\times Q$ such that
  \begin{subequations}
  \begin{gather}
    \label{eq:lame-navier-mixed}
    \begin{aligned}
      2\mu\form(\strain u, \strain v) &- \form(p,\div v) &=&\form(f,v)
      &\forall v&\in V = \vH^1_0(\domain)\\
      -\form(q,\div u) &-\tfrac1\lambda \form(p,q) &=&0
      &\forall q&\in Q \subset L^2(\domain).\\
    \end{aligned}
  \end{gather}
  \pause
  Equivalently, we write this in a single equation as
  \begin{multline}
    2\mu\form(\strain u, \strain v) - \form(p,\div v)
    - \form(q,\div u) -\tfrac1\lambda \form(p,q)
    = \form(f,v)
    \\
    \qquad\forall v\in V, q\in Q,
  \end{multline}
  \pause
  or in the nonsymmetric, (semi-)definite version
  \begin{multline}
    2\mu\form(\strain u, \strain v) + \form(p,\div v)
    - \form(q,\div u) +\tfrac1\lambda \form(p,q)
    = \form(f,v)
    \\
    \qquad\forall v\in V, q\in Q,
  \end{multline}
  \end{subequations}
\end{Definition}

\begin{remark}
  The three forms have different purposes and will be used
  accordingly. The first one highlights the fact that we now have a
  system of equations, each equation tested with its own test
  function. The second and third stress the fact that we now have a
  bilinear form on the product space $X=V\times Q$.

  The second form is symmetric, but we will see later that is
  indefinite. Thus, non of our tools from functional analysis
  apply. In contrast, the third form is nonsymmetric, but we have
  \begin{multline}
    2\mu\form(\strain u, \strain u) + \form(p,\div u)
    - \form(p,\div u) +\tfrac1\lambda \form(p,p)
    \\
    = 2\mu\form(\strain u, \strain u) + \tfrac1\lambda \form(p,p)
    \ge \tfrac{2\mu}{c_K} \norm{u}_{H^1}^2 + \tfrac1\lambda \form(p,p).
  \end{multline}
  Thus, we have ellipticity with respect to the norm
  \begin{gather}
    \norm{(u,p)}_X^2 = \norm{u}_{H^1}^2 + \norm{p}_{L^2}^2.
  \end{gather}
  Nevertheless, the ellipticity constant depends on $\lambda$, and for
  large $\lambda$, we loose sharpness of estimates again.
\end{remark}

\begin{Definition}{lame-navier-strong}
  Integrating the first equation by parts, we obtain the
  \textbf{strong form} of the \putindex{displacement-pressure
    formulation}
  \begin{gather}
    \arraycolsep.2ex
    \begin{matrix}
% Check signs
      - 2\mu \div \strain u &+& \nabla p &=& f \\
      \div u &+& \tfrac1\lambda p &=& 0
    \end{matrix}
  \end{gather}
\end{Definition}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Stokes equations}

\begin{intro}
  When we write the Lamé-Navier equations in \putindex{displacement-pressure formulation} according to
  \slideref{Definition}{displacement-pressure}, there is no
  parameter $\lambda$ tending to infinity when the material becomes
  less and less compressible. Instead, there is the parameter
  $1/\lambda$ tending to zero. Thus, we can simply consider the case
  of incompressible material by setting $1/\lambda=0$ or, in our
  abstract framework~\eqref{eq:mixedintro:1} setting $c(\cdot,\cdot) = 0$. The
  resulting system is not only important for incompressible
  elasticity, but also models the slow flow of a very viscous liquid,
  so called \putindex{creeping flow}.
\end{intro}

\begin{Definition}{stokes-eq1}
  The \define{Stokes equations} in strong form are
    \begin{gather}
      \label{eq:mixedintro:stokes-strong1}
      \arraycolsep.2ex
      \begin{matrix}
        -2\mu \div \strain \vu &+& \nabla p &=& \vf \\
        \div \vu && &=& 0.
      \end{matrix}
    \end{gather}
    In weak form, they are: find $\vu\in \vV \subset \vH^1(\domain)$
    and $p\in Q \subset L^2(\domain)$ such that
  \begin{gather}
    \label{eq:mixedintro:stokes-weak1}
    \begin{aligned}
      2\mu\form(\strain \vu, \strain \vv) &- \form(\div \vv,p) &=&\form(\vf,\vv)
      + \text{bdry}
      &\forall \vv&\in \vV\\
      -\form(\div \vu,q) & &=&0+ \text{bdry}
      &\forall q&\in Q.\\
    \end{aligned}
  \end{gather}
  The subspaces $\vV$ and $Q$ are determined by boundary conditions.
\end{Definition}

\begin{Definition}{solenoidal}
  A vector-valued function $\vu$ is called \define{divergence-free} or
  \define{solenoidal}, if there holds
  \begin{gather}
    \div \vu = 0.
  \end{gather}
  Flow described by a solenoidal function is called
  \define{incompressible}.
\end{Definition}

\begin{Lemma}{stokes-equivalence}
  Let $\vV=\vH^1_0(\domain)\cap \vH^2(\domain)$. Then, for any
  solenoidal function $\vu\in vV$ there holds
  \begin{gather}
    \label{eq:mixedintro:6}
    2\mu\form(\strain \vu, \strain \vv) = \mu\form(\nabla \vu, \nabla \vv)
    \qquad\forall \vv\in \vV.
  \end{gather}
\end{Lemma}

\begin{proof}
  We have ('$:$' denoting the \putindex{double contraction} of
  \putindex{Frobenius inner product})
  \begin{align}
    \strain \vu: \strain \vv
    &= \frac14\sum_{i,j=1}^d\left[ (\d_i u_j+\d_j u_i) (\d_i v_j+\d_j v_i)
      \right]
    \\
    &= \frac12\sum_{i,j=1}^d\left[ \d_i u_j\d_iv_j + \d_i u_j\d_j v_i\right].
  \end{align}
  The first term is the desired result, thus we have to eliminate the
  other one. We integrate by parts to obtain
  \begin{align}
    \int_\domain \d_i u_j\d_j v_i \dx
    &= - \int_\domain \d_{ij}u_j v_i \dx
%     \\ &
    = \int_\domain \d_j u_j \d_i v_i \dx
  \end{align}
  Entering in the
  previous equation and summing over $i$ and $j$, we obtain
  \begin{gather}
    2(\strain \vu,\strain \vv)
    = (\nabla \vu,\nabla \vv)+(\div \vu,\div \vv)
                           = (\nabla \vu,\nabla \vv).
  \end{gather}
\end{proof}

\begin{intro}
  In order to simplify subsequent discussion, it is customary to use
  the previous lemma to simplify the Stokes equations and to replace
  the strain tensor by the gradient. As a result, we can avoid the use
  of a Korn inequality and operate directly with the inner product of
  $\vH^1$. We note though that this formulation, while mathematically
  simpler, is physically wrong if $\vu\neq \vH^1_0(\domain)$, that is,
  having nonzero boundary values.
\end{intro}

\begin{Definition}{stokes-eq2}
  The simplified \define{Stokes equations} in strong form are
    \begin{gather}
      \label{eq:mixedintro:stokes-strong2}
      \arraycolsep.2ex
      \begin{matrix}
        - \nu \Delta \vu &+& \nabla p &=& \vf \\
        \div \vu && &=& 0.
      \end{matrix}
    \end{gather}
    In weak form, they are: find $\vu\in \vV \subset \vH^1(\domain)$
    and $p\in Q \subset L^2(\domain)$ such that
  \begin{gather}
    \label{eq:mixedintro:stokes-weak2}
    \begin{aligned}
      \nu\form(\nabla \vu, \nabla \vv) &- \form(\div \vv,p) &=&\form(\vf,\vv)
      + \text{bdry}
      &\forall \vv&\in \vV\\
      -\form(\div \vu,q) & &=&0+ \text{bdry}
      &\forall q&\in Q.\\
    \end{aligned}
  \end{gather}
  The subspaces $V$ and $Q$ are determined by boundary conditions.
\end{Definition}

\begin{Definition}{stokes-boundary2}
  Typical boundary conditions for the Stokes problem are
  \begin{xalignat}3
    \text{no-slip:}&& \vu &= 0,\\
    \text{free:}&& \d_n \vu + p \n &= 0 ,\\
    \text{slip:}&& \vu_n &= 0 & \d_n \vu_\tau &= 0,\\
    \text{friction:}&& \vu_n &= 0 & \d_n \vu_\tau &= \alpha \vu_\tau.
  \end{xalignat}
  Here, $\vu_n$ and $\vu_\tau$ are the normal and tangential components of
  $\vu$ at the boundary.
\end{Definition}

\begin{remark}
  Very much the same way as for elliptic equations, boundary
  conditions on the function $\vu$ itself are essential boundary
  conditions which have to be incorporated into the space $V$, while
  those involving normal derivatives are the result of integration by
  parts and thus of the type of natural boundary conditions.

  All of the conditions above can also be imposed with nonzero data,
  where the physical meaning of such a condition might be debatable in
  some cases. Mathematically, inhomogeneous essential boundary
  conditions are achieved by lifting an arbitrary function with this
  boundary condition and modifying the right hand side of the
  equation, while inhomogeneous conditions on the normal derivative
  are implemented by boundary integrals on the right hand side.
\end{remark}

\begin{remark}
  Physically, the condition $\vu_n=0$ models an impermeable wall. It
  says in particular, that no mass is lost through this boundary and
  is thus related to the first principle of mass conservation.

  The conditions on the tangential velocity model the fact that
  molecules very close to the wall stick to the wall. While this claim
  is not supported by a first principle, it has been verified by
  measurements to very high accuracy. Nevertheless, in the study of
  turbulent flow, a friction condition comes up quite naturally.
\end{remark}

\begin{Lemma}{divergence-compatibility}
  For any solenoidal $\vu$ function there holds
  \begin{gather}
    \int_{\d\domain} \vu\cdot \n\ds = 0.
  \end{gather}
  Furthermore, if the space $\vV$ in the Stokes equations is chosen such that for any $\vv\in \vV$ there holds $\vv_n=0$ on the whole
  boundary $\d\domain$, then the pressure $p$ is determined by the
  Stokes equations only up to a constant.
\end{Lemma}

\begin{proof}
  The first statement is simple application of the Gauss theorem
  \begin{gather}
    \int_{\domain} \div \vu\dx = \int_{\d\domain} \vu\cdot \n\ds.
  \end{gather}
  For the second statement, we note that the only term in the
  equations which determines the pressure is $\form(\div \vv,
  p)$. Integrating by parts and using the boundary condition, we
  obtain
  \begin{gather}
    \int_\domain \div \vv\,p\dx
    = -\int_\domain \vv\cdot\nabla p\dx +
    \int_{\d\domain} \vv\cdot \n \,p\ds
    = -\int_\domain \vv\cdot \nabla p\dx.
  \end{gather}
  Since the gradient of a constant is zero, we can add any constant
  function to a given solution $p$ without changing the term
  $\form(\div \vv,p)$, thus leaving $p$ determined only up to a
  constant.
\end{proof}

\begin{Notation}{pressure-constant}
  If in Definition~\ref{Definition:stokes-eq2} the space $\vV$ is chosen
  such that for all $\vv\in \vV$ there holds $\vv\cdot \n =0$ on the whole
  boundary $\d\domain$, then the pressure solution $p$ cannot be
  determined uniquely in $L^2(\domain)$. In such cases, we choose the
  pressure space
  \begin{gather}
    L^2_0(\domain) = L^2(\domain)/\R
    = \biggl\{q\in L^2(\domain)\bigg| \int_\domain q\dx = 0\bigg\}.
  \end{gather}
\end{Notation}

\begin{remark}
  As we could see from the preceding lemma, solvability of
  equation~\eqref{eq:mixedintro:stokes-weak2} depends on some
  compatibility of the spaces $\vV$ and $Q$ we had not seen in the
  elliptic case. Indeed, we need a whole new tool from functional
  analysis to replace the Lax-Milgram lemma. This tool will be studied
  in Chapter~\ref{sec:mixed-wellposedness}.
\end{remark}

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "main"
%%% End:


\documentclass{article}

\usepackage[stdmargin, noindent]{../../rajeev}
\usepackage{mathtools}
\usepackage{dirtytalk}

\pagestyle{fancy}
\rhead{\today}
\lhead{Math 291H HW \#2}

\begin{document}

\begin{center}
    \Large \textbf{Math 291H Homework \#2}
\end{center}
\begin{center}
    \Large Rajeev Atla
\end{center}

Honors Pledge Statement: \say{The writeup of this submission is my own work alone.}

\problem{1.19}

\subsection*{a}

\begin{align*}
  \pars{1, 1, 0} + t \pars{1, -1, 2} &= \pars{2, 0, 2} + s \pars{-1, 1, 0} \\
  \pars{t, -t, 2t} - \pars{-s, s, 0} &= \pars{1, -1, 2} \\
  \pars{t + s, -t-s, 2t} &= \pars{1, -1, 2} \\
\end{align*}

This implies that $t=1$, which means that $s=0$.
Resubstituting, we find the point of intersection to be \boxed{\pars{2, 0, 2}}.

\subsection*{b}

Let $\bm{v}_1 = \pars{1, -1, 2}$ and $\bm{v}_2 = \pars{-1, 1, 0}$.
Both these vectors are in the plane.
We next find the normal vector $\bm{n}$.

$$
\bm{n} = \bm{v}_1 \times \bm{v}_2 = \pars{-2, -2, 0}
$$

The general form of a plane is $\bm{n} \cdot \bm{x} = d$.
Substituting $\bm{x} = \pars{1, 1, 0}$, we see that $d=-4$.
The equation for the plane is therefore

$$
-2 x + -2y = -4
$$

We can simplify this a little to get

$$
\boxed{x + y = 2}
$$

\problem{1.20}
Defining $\bm{n} := \pars{2, -1, 3}$ and $\bm{x}_0 := \pars{2, 0, 0}$, we can write the given equation as

$$
\bm{n} \cdot \pars{\bm{x} - \bm{x}_0} = 0
$$

We normalize $\bm{n}$, getting

$$
\bm{u} = \frac{1}{\sqrt{14}} \pars{2, -1, 3}
$$

The minimal distance between this plane and $\bm{p}$ is then

$$
\abs{\pars{\bm{x}_0 - \bm{p}} \cdot \bm{u}} = \frac{1}{\sqrt{14}} \abs{\pars{2, 3, 0} \cdot \pars{2, -1, 3}} = \boxed{\frac{1}{\sqrt{14}}}
$$

\problem{1.24}

Let $k \in \RR$, so that

$$
h_{\bm{u}} \pars{\bm{x}} = k \bm{e}_1
$$

Since $\norm{h_{\bm{u}} \pars{\bm{x}}} = \norm{\bm{x}}$, \boxed{k= \pm 7}.

By construction of the Householder reflection,

\begin{align*}
  \bm{u} &= \pm \frac{\bm{x} - k \bm{e}_1 }{\norm{\bm{x} - k \bm{e}_1}} \\
         &= \pm \frac{\pars{5 \pm 7, 2, 4, 2}}{\norm{\pars{5 \pm 7, 2, 4, 2}}} \\
  &= \boxed{\frac{\pars{6, 1, 2, 1}}{\sqrt{42}}, -\frac{\pars{6, 1, 2, 1}}{\sqrt{42}}, \frac{\pars{-1, 1, 2, 1}}{\sqrt{10}}, - \frac{\pars{-1, 1, 2, 1}}{\sqrt{10}} }\\
\end{align*}

(In the first two lines, the outer $\pm$ signs are independent of the inner ones.)

\problem{1.27}
We define the orthonormal basis

\begin{align*}
  \bm{u}_1 &= \frac{\bm{v}_1 \times \bm{v}_2}{\norm{\bm{v}_1 \times \bm{v}_2}} \\
           &= \frac{1}{\sqrt{14}}\pars{-3, -2, 1} \\
  \bm{u}_2 &= \frac{1}{\norm{\bm{v}_1}} \bm{v}_1 \times \bm{u}_1 \\
           &= \frac{1}{ \sqrt{10}} \pars{-1, 0, -3} \\
  \bm{u}_3 &= \bm{u}_1 \times \bm{u}_2 \\
           &= \frac{1}{\sqrt{35}} \pars{3, -5, -1} \\
\end{align*}

We can then define

$$
\bm{b} = \bm{x}_1 - \bm{x}_2 = \pars{0, 3, 1}
$$

We then have

$$
t = \frac{\bm{b} \cdot \bm{u}_2}{\bm{v}_2 \cdot \bm{u}_2} = \frac{-5}{10} = - \frac{1}{2}
$$

The point on the first line that corresponds to this $t$-value is
$$
\boxed{\pars{\frac{3}{2}, -\frac{1}{2}, -\frac{5}{2}}}
$$

Similarly,

$$
s = \frac{ t \pars{\bm{v}_2 - \bm{b} } \cdot \bm{u}_3 }{\bm{v}_1 \cdot \bm{u}_3} = \frac{- \frac{1}{2} \pars{-1, 0, 2} \cdot \pars{3, -5, -1}}{\pars{1, -4, -2} \cdot \pars{3, -5, -1}} = \frac{\frac{5}{2}}{25}  = \frac{1}{10}
$$

$$
\boxed{\pars{\frac{33}{10}, \frac{3}{2}, \frac{9}{10}}}
$$

The distance between these points is

$$
\abs{\bm{b} \cdot \bm{u}_1} = \boxed{\frac{5}{\sqrt{14}}}
$$

\problem{1.29}

\subsection*{a}

It's easy to see that $\norm{\bm{u}_i} = 1$.
In addition, we can compute

\begin{align*}
  \bm{u}_1 \cdot \bm{u}_2 &= \frac{1}{9} \pars{1 \cdot 2 + 2 \cdot 1 - 2 \cdot 2} = 0 \\
  \bm{u}_1 \cdot \bm{u}_3 &= \frac{1}{9} \pars{1 \cdot 2 + 2 \cdot \pars{-2} -2 \cdot \pars{-1}} = 0 \\
  \bm{u}_2 \cdot \bm{u}_3 &= \frac{1}{9} \pars{2 \cdot 2 + 1 \cdot \pars{-2} + 2 \cdot \pars{-1}} = 0 \\
\end{align*}

We see that

$$
\bm{u}_1 \times \bm{u}_2 = \frac{1}{3} \pars{2, -2, -1} = \bm{u}_3
$$

Therefore, this is a right-handed orthonormal basis.

\subsection*{b}

$$
\bm{u} = \frac{\bm{u}_1 - \bm{e}_1}{\norm{\bm{u}_1 - \bm{e}_1}} = \frac{\frac{1}{3} \pars{-2, 2, -2}}{\frac{1}{3} \norm{\pars{-2, 2, -2}}} = \boxed{\frac{1}{\sqrt{3}} \pars{-1, 1, -1}}
$$

\subsection*{c}

\begin{align*}
  h_{\bm{u}} \pars{\bm{u}_2} &= \bm{u}_2 - 2 \pars{\bm{u}_2 \cdot \bm{u}} \bm{u} \\
                             &= \frac{1}{3} \pars{2, 1, 2} - \frac{2}{9} \pars{-3} \pars{-1, 1, -1} \\
                             &= \pars{\frac{2}{3}, \frac{2}{3}, \frac{1}{3}} + \pars{-\frac{2}{3}, \frac{2}{3}, -\frac{2}{3}} \\
                             &= \boxed{\pars{0, 1, 0} = \bm{e}_2} \\
  h_{\bm{u}} \pars{\bm{u}_3} &= \bm{u}_3 - 2 \pars{\bm{u}_3 \cdot \bm{u}} \bm{u} \\
                             &= \frac{1}{3} \pars{2, -2, -1} - \frac{2}{9} \pars{-3} \pars{-1, 1, -1} \\
                             &= \pars{\frac{2}{3}, - \frac{2}{3}, - \frac{1}{3}} + \pars{- \frac{2}{3}, \frac{2}{3}, - \frac{2}{3}} \\
                             &= \boxed{\pars{0, 0, 1} = \bm{e}_1} \\
\end{align*}

\problem{1.32}

\subsection*{a}

Define $V := V_1 \cap V_2$.
Suppose we have vectors $\bm{r}_1, \bm{r}_2 \in V$ and scalars $a, b \in \RR$.
By definition, $\bm{r}_1 \in V_1, V_2$ and $\bm{r}_2 \in V_1, V_2$.
By definition of a subspace, $a \bm{r}_1 \in V_1, V_2$ and $b \bm{r}_1 \in V_1, V_2$.
Since these vectors are in both subspaces, we can say that

$$
a \bm{r}_1 + b \bm{r}_2 \in V_1, V_2
$$

This is what we wanted to show.

\subsection*{b}

If we scale up $\bm{z}$ by a factor of $\alpha$, we get

$$
\alpha \bm{z} = \alpha \bm{x} + \alpha \bm{y}
$$

Since $\alpha \bm{x} \in V_1$ and $\alpha \bm{x} \in V_2$ by the definition of a subspace, $\alpha \bm{z} \in V_1 + V_2$ by the construction of $V_1 + V_2$.
Similarly, let

$$
\bm{z'} = \bm{z}_1 + \bm{z}_2 = \bm{x}_1 + \bm{y}_1 + \bm{x}_2 + \bm{y}_2 = \pars{\bm{x}_1 + \bm{x}_1} + \pars{\bm{y}_1 + \bm{y}_2}
$$

By the definition of a subspace, $\bm{x}_1 + \bm{x}_2 \in V_1$.
Similarly, $\bm{y}_1 + \bm{y}_2 \in V_2$.
Therefore, by construction, $\bm{z}_1 + \bm{z}_2 \in V_1 + V_2$.

\problem{1.33}

Define $V_3 := V_1 + V_2$ and $V := V_1 \cap V_2$.
We then define an orthonormal basis for $V$.

$$
\set{\bm{u}_1, \dots, \bm{u}_p}
$$

Therefore, $\dim \pars{V} = p$.
We can extend $\set{\bm{u}_1, \dots, \bm{u}_p}$ to find orthonormal bases for $V_1$ and $V_2$.
By definition, $V \subseteq V_1$ and $V \subseteq V_2$.
Therefore, $\spn V \subseteq \spn V_1$ and $\spn V \subseteq \spn V_2$.
Similarly, $\spn V_1 \subseteq \spn V_3$ and $\spn V_2 \subseteq V_3$.
We can therefore write and define

\begin{align*}
  S &:= \set{\bm{u}_1, \cdots, \bm{u}_p, \bm{v}_1, \cdots, \bm{v}_q} \\
  T &:= \set{\bm{u}_1, \cdots, \bm{u}_p, \bm{w}_1, \cdots, \bm{w}_r} \\
  V_1 &= \spn \set{\bm{u}_1, \cdots, \bm{u}_p, \bm{v}_1, \cdots, \bm{v}_q} \\
  V_2 &= \spn \set{\bm{u}_1, \cdots, \bm{u}_p, \bm{w}_1, \cdots, \bm{w}_r} \\
\end{align*}

It's easy to see that $V_1 \subseteq \spn \pars{S \cup T}$ and $V_2 \subseteq \spn \pars{S \cup T}$.
Therefore, $V_3 \subseteq \spn \pars{S \cup T} $.
In addition, $\pars{S \cup T} \subseteq V_3$, so $\spn \pars{S \cup T} \subseteq V_3$.
Therefore,

\begin{align*}
  V_3 &= \spn \pars{S \cup T} \\
      &= \spn \set{\bm{u}_1, \cdots, \bm{u}_p, \bm{v}_1, \cdots, \bm{v}_q, \bm{w}_1, \cdots, \bm{w}_r} \\
  \dim V_3 &= p+q+r \\
\end{align*}

We can now verify that

\begin{align*}
  \dim V_3 + \dim V &= \dim V_1 + \dim V_2 \\
  \pars{p + q + r} + \pars{p} &= \pars{p+q} + \pars{p+r} \\
\end{align*}

\end{document}


\section*{Problem 1: Gaussian Random Projections and Inner Products}

Let $\phi(x) = \frac{1}{\sqrt{m}}Ax$ represent our random projection of
$x \in \mathbb{R}^d$, with $A$ an $m \times d$ projection matrix with each entry
sampled i.i.d from $N\left(0,1\right)$. Note that each row of $A$ is a random
projection vector, $v^{(i)}$.

The \emph{norm preservation theorem} states that for all $x \in \mathbb{R}^d,$
the norm of the random projection $\phi\left(x\right)$ approximately maintains
the norm of the original $x$ with high probability:
\begin{equation}
  \mathbb{P}\left(
    (1 - \epsilon) \lVert x \rVert^2
    \leq
    \lVert \phi\left(x\right) \rVert^2
    \leq
    \left(1 + \epsilon\right) \lVert x \rVert^2
  \right)
  \geq 1 - 2\exp\left(-\left(\epsilon^2 - \epsilon^3\right)m/4\right),
  \label{eqn:1thm}
\end{equation}
where $\epsilon \in \left(0,1/2\right).$

Using the norm preservation theorem, prove that for any $u, v \in \mathbb{R}^d$
such that $\lVert u \rVert \leq 1$ and $\lVert v \rVert \leq 1$,
\begin{equation}
  \mathbb{P}\left(\lvert u \cdot v - \phi\left(u\right)\cdot\phi\left(v\right)\rvert \geq \epsilon\right) \leq 4\exp\left(
    -\left(\epsilon^2 - \epsilon^3\right)m/4
  \right)
  \label{eqn:1}
\end{equation}

\subsection*{Solution}
\begin{proof}
  First note that
  \begin{equation*}
      (1 - \epsilon) \lVert u + v \rVert^2
      \leq
      \lVert \phi\left(u + v\right) \rVert^2
      \leq
      \left(1 + \epsilon\right) \lVert u + v \rVert^2
    \end{equation*}
    implies that
    \begin{equation}
      \lVert u + v \rVert^2 - 2\epsilon
      \leq
      \lVert \phi\left(u + v\right) \rVert^2
      \leq
      \lVert u + v \rVert^2 + 2\epsilon
      \label{eqn:1sup}
    \end{equation}
    by triangle inequality and the assumption of the norms of $u$ and $v$.

    Thus, the probability of the event in Equation \ref{eqn:1sup} than that of
    Equation \ref{eqn:1thm}.

    Using this and taking the additive inverse, we have that
  \begin{align}
    \mathbb{P}\left(
    \left\lVert \phi(u + v) \right\rVert^2
    \not\in
    \left[
    \lVert u + v \rVert^2 - 2\epsilon,
    \lVert u + v \rVert^2 + 2\epsilon
    \right]
    \right)
    &\leq 2\exp\left(-\left(\epsilon^2 - \epsilon^3\right)m/4\right)
      \label{eqn:1conda}\\
    \mathbb{P}\left(
    \left\lVert \phi(u - v) \right\rVert^2
    \not\in
    \left[
    \lVert u - v \rVert^2 - 2\epsilon,
    \lVert u - v \rVert^2 + 2\epsilon
    \right]
    \right)
    &\leq 2\exp\left(-\left(\epsilon^2 - \epsilon^3\right)m/4\right).
      \label{eqn:1condb}
  \end{align}

  By the countable sub-additivity property of probability distributions, we have
  that the probability of both these events occurring is at most
  $4\exp\left(-\left(\epsilon^2 - \epsilon^3\right)m/4\right)$. Thus, we are
  done if we can show
  $\left\{\left\lvert u \cdot v - \phi(u) \cdot \phi(v) \right\rvert \geq
    \epsilon\right\}$ subsets these two conditions.

  If we have the pair
  \begin{align}
    \left\lVert \phi(u + v) \right\rVert^2
    &\leq \left\lVert
      u + v
      \right\rVert^2 - 2\epsilon \Rightarrow
      \left\lVert
      u + v
      \right\rVert^2 -
      \left\lVert \phi(u + v) \right\rVert^2
      \geq
      2\epsilon
    \\
    \left\lVert \phi(u - v) \right\rVert^2
    &\geq \left\lVert
      u - v
      \right\rVert^2 + 2\epsilon \Rightarrow
      \left\lVert
      u - v
      \right\rVert^2 -
      \left\lVert \phi(u - v) \right\rVert^2
      \geq
      2\epsilon,
  \end{align}
  we can use the linearity of $\phi$ and the expansion
  $\lVert u \pm v \rVert^2 = \lVert u \rVert^2 + \lVert v \rVert^2 \pm 2u \cdot
  v$, we can add the two inequalities to obtain
  \begin{align}
    4\left(
    u \cdot v
    -
    \phi\left(u\right)\cdot\phi\left(v\right)
    \right)
    \geq 4\epsilon.
    \nonumber
  \end{align}

  Thus, we have that the conditions in Equations \ref{eqn:1conda} and
  \ref{eqn:1condb} imply
  $u \cdot v - \phi\left(u\right)\cdot\phi\left(v\right) \geq \epsilon$.

  Similarly, we show that the pair
  \begin{align}
    \left\lVert \phi(u + v) \right\rVert^2
    &\geq \left\lVert
      u + v
      \right\rVert^2 + 2\epsilon
    \\
    \left\lVert \phi(u - v) \right\rVert^2
    &\leq \left\lVert
      u - v
      \right\rVert^2 - 2\epsilon
  \end{align}
  implies
  $u \cdot v - \phi\left(u\right)\cdot\phi\left(v\right) \leq -\epsilon$, which
  gives us
  $\left\lvert u \cdot v - \phi\left(u\right)\cdot\phi\left(v\right)
  \right\rvert \geq \epsilon.$
\end{proof}


\chapter{Shortest Path}

Given a weighted graph, one interesting problem is to find the
path with minimal weight.

\section{Single Source}

This instance of the shortest path problem begins at a specific vertex
in the graph, hence ``Single Source.''

\subsection{Dijkstra's Algorithm}

Given an acyclic graph $G=(V,E)$ in which every edge has a positive
weight and a single vertex $s$ from which to start the path, we begin
by labeling all vertices $u \in E | weight(u) = \infty$, then label
$weight(s) = 0$.  We also create a min-heap of the vertices using
weight as key.

We pop-best a vertex $v$ from our heap.  For each edge $e$ incident to
$v$ and some other vertex $u$, we check if $weight(v) + weight(e)$ is
less than $weight(u)$, in which case we set $weight(u) = weight(v) +
weight(e)$ and decrease the key of $u$ in our heap.  We continue this
process until no vertices remain in the heap.

It should be easy to see that if we have a particular target vertex,
we can halt the algorithm as soon as our target is the root of the
heap, because we have found the shortest path to it.

\subsection{Analysis of Dijkstra's Algorithm}

Intuitively, this algorithm has an upper bound of $O(|V|)$ times
however long it takes us to extract the minimum vertex from our heap,
plus $O(|E|)$ times however long it takes us to decrease the key,
since we have to check every vertex and we may have to check every
edge.  Using a standard heap, this gives us $O(|V|log|V| +
|E|log|V|)$.

\section{All Sources}

This instance of the shortest path problem is concerned with several
sources, in which case Dijkstra's Algorithm does not suffice.

\subsection{Floyd-Warshall Algorithm}

The Floyd-Warshall Algorithm is
\hyperlink{sec:floyd_warshall}{discussed in detail in the chapter on
  Dynamic Programming}.

\section{Arbitrary Weights}

A keen reader will have noticed that both Dijkstra's and Floyd
Warshall Algorithms assume positive edge weights.  Neither of which
are generally useful when there may be edges of negative weight.

\subsection{Bellman-Ford Algorithm}

The Bellman-Ford Algorithm can be applied with edges of arbitrary
weight.  Note that even Bellman-Ford doesn't deal with cycles of
negative weight, since these can be used to make any path have an
arbitrarily small total weight.

The Bellman-Ford Algorithm is beyond the scope of these notes, for
more information please see
\href{https://en.wikipedia.org/wiki/Bellman-Ford}{Wikipedia's entry on
  the Bellman-Ford Algorithm}, or CLRS.


%These notes summarize the lecture notes from the Linear Modelling course at Sheffield's School of Mathematics and Statistics, MSc degree programme. The original notes were written by Dr.\ Kevin Walters and Dr.\ Jeremy Oakley. This summary is completely derived from these notes and from other MSc sources. Any errors are most probably mine.

%Everything is in matrix form unless a lower case letter with a subscript (such as $x_i$) is used (even there, I might deviate from this convention if I need to index sub-matrices; it's best to look at the context to decide what is meant).

\section{Background}

\subsection{Derivations of combinations of functions}

\begin{equation}
  (uv)' = uv' + vu'
\end{equation}

\begin{equation}
(u/v)' = 	\frac{vu' - uv'}{v^2}
\end{equation}

\subsection{Some key distributional results}

\subsection{Some very basic matrix algebra facts}

\textbf{Inverse (2x2)}:

$\begin{pmatrix}
a & b \\
c & d\\
\end{pmatrix}^{-1}
= \frac{1}{ad-bc}
\begin{pmatrix}
d & -b \\
-c & a\\
\end{pmatrix}$

\textbf{Inverse of non-singular matrices}.
If A and B are non-singular matrices then $(AB)^{?1} = B^{-1}A^{-1}$.

\textbf{Symmetric square matrix}: $A=A^T$.

If a symmetric matrix A is non-singular then $A^{-1}$ is also symmetric.

$AA^{-1}=A^{-1}A=I$ given A is square and invertible.

 If the symmetric matrix $A$ is non-singular then $A^{-1}$ is also symmetric.

 \textbf{Multiplication is distributive}: Multiplication is distributive over addition and subtraction, so $(A-B)(C-D)=AC -BC-AD+BD$.

\textbf{Transpose}: $(A+B)^T =A^T+B^T$ and $(AB)^T =B^TA^T$

\textbf{Sum of squares}: $\sum x_i^2 = \mathbf{x}^T \mathbf{x}$.

\textbf{Symmetry under multiplication}: If A is $n\times p$, then $AA^T$ and $A^TA$ are symmetric.

\textbf{Trace of a square matrix}:

\begin{enumerate}
\item
$tr(A)=\sum a_{ii}$
\item
$tr(A + B) = tr(A) + tr(B)$
\item
$tr(cA) = ctr(A)$
\item
$tr(AB) = tr(BA)$
\end{enumerate}

\textbf{Idempotent}:  $A^2 = AA = A$. Example: $A = I_n$; this is the only non-singular idempotent matrix.

If A is idempotent and if $A\neq I_n$, then A is singular and its trace is equal to its rank $n-p$, for some $p>0$.

\textbf{Inverse of a matrix product}:
If A and B are non-singular matrices then $(AB)^{-1} = B^{-1}A^{-1}$.

\textbf{Rank}: the number of linearly independent columns or rows of A.

How to determine linear independence:

\section{Basic facts}

\begin{equation}
y=X\beta+\epsilon
\end{equation}

\begin{tabular}{@{}ll@{}ll@{}}
$E(y) = X\beta = \mu$ &  & $E(\epsilon)=0$  \\
$Var(y) = \sigma^2 I_n $ & & $Var(\epsilon) = \sigma^2 I_n$ \\
%y+X\hat{\beta} + e$ \\
%%$E(y) = X\beta = \mu$ & Var\\
% $\epsilon \sim N_p (0,\sigma^2I_n)$  & No \verb!\part! divisions. \\
%\verb!article! & No \verb!\part! or \verb!\chapter! divisions. \\
%\verb!letter!  & Letter (?). \\
%\verb!slides!  & Large sans-serif font.
\end{tabular}

\begin{equation}
y = X\hat{\beta} + e
\end{equation}

\begin{fmpage}{\linewidth}
Note that $S_{xx}=\frac{1}{(X'X)^{-1}}$, and
$G=(X'X)^{-1}$, so that $S_{xx}=\frac{1}{G}$.
\end{fmpage}

\begin{tabular}{@{}ll@{}ll@{}}
Results for $\hat{\beta}$ & Results for $e$\\
$E(\hat{\beta}) = \beta$ & $E(e) = 0$\\
$Var(\hat{\beta}) = \sigma^2 (X^T X)^{-1} = \frac{\sigma^2}{S_{xx}}=\sigma^2G$ & $Var(e)=\sigma^2 M$  \\
$\hat{\beta} \sim N_p(\beta,\sigma^2 (X^T X)^{-1})$ & $Var(e_i)=\sigma^2 m_{ii} $\\
&  $E(e_i^2)= \sigma^2 m_{ii}$\\
$\hat{\beta} = (X^T X)^{-1} X^T y$, $X$ has full rank  & $E(\sum e_i^2) = \sigma^2 (n-p)$\\
\end{tabular}

\medskip
\textbf{Sum of Squares}:

\begin{equation}
S(\hat{\beta}) = \sum e_i^2 = e^T e = (y-X\hat{\beta})^T (y-X\hat{\beta})  = y^T y - y^T X \hat{\beta} = S_r
\end{equation}

Alternatively: $S_r= y^Ty - \hat{\beta}^T X^T X\hat{\beta}=y^Ty - \hat{\beta}^T X^T y$ (see review exercises 2).

\textbf{Estimation of error variance: $e=My$}

\begin{equation}
e = y - X\hat{\beta} = y - X (X^T X)^{-1} X^T y = My
\end{equation}

\noindent
where

\begin{equation}
M = I_n -  X (X^T X)^{-1} X^T
\end{equation}

M is symmetric, idempotent $n\times n$.

Note that $MX=0$, which means that

\begin{equation}
E(e)=E(My) = ME(y)= MX\beta = 0
\end{equation}

Also, $Var(e) = Var(My) = M Var(y) M^T = \sigma^2 I_n M$.

\medskip
\textbf{Important properties of M}:

\begin{itemize}
\item $M$ is singular because every idempotent matrix except $I_n$ is singular.
\item $trace(M)=rank(M)=n-p$.
\end{itemize}

\medskip

\textbf{Residual mean square}:
\begin{equation}
\hat{\sigma}^2 = \frac{\sum e_i^ 2}{n-p} \quad E(\hat{\sigma}^2)=\sigma^2
\end{equation}

The square root of $\hat{\sigma}^2$, $\hat{\sigma}$ is the \textbf{residual standard error}.
Note: The phrase ``standard error'' here should not be misinterpreted to mean standard error in the sense of ``SE''.

\textbf{Variance-covariance matrix}:

In a model like \begin{verbatim}fm<-lm(Maint ~ Age, data = data)\end{verbatim}, the variance-covariance matrix is:

\begin{equation}
\begin{pmatrix}
Var(\hat{\beta}_0) & Cov(\hat{\beta}_0,\hat{\beta}_1) \\
Cov(\hat{\beta}_0,\hat{\beta}_1) & Var(\hat{\beta}_1)\\
\end{pmatrix}
\end{equation}

The correlation between the two parameter estimates is therefore:

\begin{equation}
Corr(\hat{\beta}_0,\hat{\beta}_1) = \frac{Cov(\hat{\beta}_0,\hat{\beta}_1)}{SE(\hat{\beta}_0) SE(\hat{\beta}_1)}
\end{equation}

Example (tractor data):

\begin{verbatim}
> vcov(fm)
            (Intercept)     Age
(Intercept)       21591 -4624.0
Age               -4624  1267.9
\end{verbatim}

We can check the correlation calculation using

\begin{verbatim}
> cov2cor(vcov(fm))
            (Intercept)      Age
(Intercept)     1.00000 -0.88378
Age            -0.88378  1.00000
\end{verbatim}

\subsection{Some short-cuts for hand-calculations}

\begin{tabular}{@{}ll@{}}
$S_{xx} = \sum (x_i - \bar{x})^2$  & $= \sum x_i^2 - n\bar{x}^2$\\
$S_{yy} = \sum (y_i - \bar{y})^2$ & $= \sum y_i^2 - n\bar{y}^2$\\
$S_{xy} = \sum (x_i -\bar{x})(y_i -\bar{y})$ & = $\sum x_i y_i - n\bar{x}\bar{y}$\\
\end{tabular}

\begin{equation}
\hat{\beta} = (X^T X)^{-1} X^T y =
\begin{pmatrix}
\bar{y} - \bar{x} \frac{S_{xy}}{S_{xx}}\\
\frac{S_{xy}}{S_{xx}}
\end{pmatrix}
\end{equation}

\begin{equation}
X^T X = \begin{pmatrix}
n & \sum_{i=1}^n x_i\\
\sum_{i=1}^n x_i & \sum_{i=1}^n x_i^2\\
\end{pmatrix}
\end{equation}

\begin{equation}
(X^T X)^{-1} = \frac{1}{nS_{xx}}
\begin{pmatrix}
 S_{xx}+n\bar{x}^2 & -n\bar{x} \\
 -n\bar{x} & n\\
\end{pmatrix}
\end{equation}

\noindent
Note that $\sum_{i=1}^n x_i = n\bar{x}$.

\begin{equation}
X^T  y = \begin{pmatrix}
n\bar{y} \\
S_{xy} + n\bar{x}\bar{y}
\end{pmatrix}
\end{equation}

See \cite[25]{DraperSmith} for a full exposition.

\subsection{Gauss-Markov conditions}

This imposes distributional assumptions on $\epsilon = y - X \beta$.

$E(\epsilon)=0$ and $Var(\epsilon)=\sigma^2 I_n$,

\subsection{Gauss-Markov theorem}

Let $a$ be any $p \times 1$ vector and suppose that $X$ has rank $p$. Of all estimators of $\theta = a^T \beta$ that are unbiased and linear functions of $y$, the estimator $\hat{\theta} = a^T \hat{\beta}$ has minimum variance. Note that $\theta$ is a scalar.

Note: no normality assumption required! But if $\epsilon \sim N(0,\sigma^2)$, $\hat{\beta}$ have smaller variances than any other estimators.

\textbf{Minimum variance unbiased linear estimators}:
to-do

\subsection{$R^2$ or Coefficient of determination}

\begin{tabular}{@{}ll@{}}
$S_{TOTAL} = (y-\bar{y})^T(y-\bar{y})$  $= y^T y - n\bar{y}^2$ & \\
$S_{REG} = (X\hat{\beta}-\bar{y})^T (X\hat{\beta}-\bar{y})$ & \\
$S_r = \sum e_i^2 = (y-X\hat{\beta})^T (y-X\hat{\beta})$ & \\
\end{tabular}

\begin{equation}
S_{TOTAL} = S_{REG}+ S_r
\end{equation}

\begin{equation}
R^2 = \frac{S_{TOTAL}-S_r}{S_{TOTAL}} = \frac{S_{REG}}{S_{TOTAL}}
\end{equation}

For $y = 1_n \beta_0 + \epsilon$, then $R^2 = \frac{S_{REG}}{S_{TOTAL}} = 0$ because $X\hat{\beta} = \bar{y}$. So $S_{REG} = (X\hat{\beta} - \bar{y})^T (X\hat{\beta} - \bar{y}) = 0$.

In simple linear regression, $R^2 = r^2$.  $R^2$ is a generalization of $r^2$.

Adjusted $R^2= R_{Adj}^2$.  $R_{Adj}^2= 1-\frac{S_r/(n-p)}{S_{TOTAL}/(n-1)}$.

$R^2$ increases with increasing numbers of explanatory variables, therefore $R_{Adj}^2$ is better.

\section{Hypothesis testing}

\subsection{Some theoretical background}

\textbf{Multivariate normal}:

Let $X^T = < X_1,\dots,X_p>$, where $X_i$ are univariate random variables.

X has a multivariate normal distribution if and only if every component of $X$ has a univariate normal distribution.

\textbf{Linear transformations}:

Let $A, b$ be constants. Then, $Ax + b\sim N_q (A\mu + b, A\Sigma A^T)$.

\textbf{Standardization}:

Note that $\Sigma$ is positive definite (it's a variance covariance matrix), so $\Sigma = CC^T$.
$C$ is like a square root (not necessarily unique).

It follows ``immediately'' that

\begin{equation}
C^{-1} (X-\mu) \sim N_p (0_p, I_p)
\end{equation}

If $\Sigma$ is a diagonal matrix, then $X_1,\dots,X_n$ are independent and uncorrelated.

\textbf{Quadratic forms}:

Recall distributional result: If we have $n$ independent standard normal random variables, their sum of squares is $\chi_n^2$.

Lt $z = C^{-1} (X-\mu)$, and $\Sigma=CC^T$. The sum of squares $z^T z$ is:

\begin{equation}
\begin{split}
z^T z & = [C^{-1} (X-\mu)]^T [C^{-1} (X-\mu)]\\
& = (X-\mu)^T [C^{-1}]^T [C^{-1}](X-\mu) \quad \dots (AB)^T=B^T A^T\\
\end{split}
\end{equation}

Note that $ [C^{-1}]^T =  [C^{T}]^{-1}$. Therefore,

\begin{equation}
\begin{split}
[C^{-1}]^T [C^{-1}] & = [C^T]^{-1} [C^{-1}]\\
& = (C^T C)^{-1}\\
& = (C C^T)^{-1}\\
& = \Sigma^{-1}\\
\end{split}
\end{equation}

Therefore: $z^T z = (X-\mu)^T  \Sigma^{-1} (X-\mu)\sim \chi_p^2$, where $p$ is the number of parameters.

\textbf{Quadratic expressions involving idempotent matrices}

Given a matrix $K$ that is idempotent, symmetric. Then:

\begin{equation}
x^T K x = x^T K^2 x = x^T K^T K x
\end{equation}

Let $x\sim N_n(\mu,\sigma^2 I_n)$, and let $K$ be a symmetric, idempotent $n \times n$ matrix such that $K\mu=0$. Let $r$ be the rank or trace of $K$. Then we have the

\begin{fmpage}{\linewidth}
\textbf{sum of squares property}:

\begin{equation}
x^T K x \sim \sigma^2 \chi_r^2
\end{equation}
\end{fmpage}

The above generalizes the fact that if we have $n$ independent standard normal random variables, their sum of squares is $\chi_n^2$.

Two points about the sum of squares property:
\begin{itemize}
\item
Recall that the expectation of a chi-squared random variable is its degrees of freedom. It follows that:

\begin{equation}
E(x^T K x) =  \sigma^2 r
\end{equation}

If $K\mu\neq 0$, $E(x^T K x) =  \sigma^2 r+\mu^T K\mu$.

\item If $K$ is idempotent, so is $I-K$. This allows us to split $x^T x$ into two components sums of squares:

\begin{equation}
x^T x = x^T K x+x^T (I-K) x
\end{equation}
\end{itemize}

\textbf{Partition sum of squares}:

[helps prove independence of SSs]

\begin{enumerate}
\item Let $K_1, K_2,\dots, K_q$ be \textbf{symmetric idempotent $n \times n$ matrices} such that
\item $\sum K_i= I_n$ and
\item $K_iK_j =0$, for all $i\neq j $.
\item Let $x\sim N_n(\mu, \sigma^2I_n)$.
 \end{enumerate}

 Then we have the following partitioning into \textbf{independent} sums of squares:

  \begin{equation}
x^T x = \sum x^T K_i x
\end{equation}

If $K_i \mu = 0$, then $ x^T K_i x\sim \sigma^2 \chi_{r_i}^2$, where $r_i$ is the rank of $K_i$.

\textbf{Example}:

\begin{equation}
y^T y  = y^T M y + y^T (I-M) y
\end{equation}

Let $K_1 = M$ and $K_2 = (I-M)$. It is easy to check that all four conditions above are satisfied; therefore the sums of squares are independent.

Note that

\begin{equation}
 y^T M y  = e^T e \sim \chi^2_{n-p}
\end{equation}

and

\begin{equation}
 y^T (I-M y)  = \hat{\beta}^T (X^T X) \hat{\beta} \sim \chi^2_{p}
\end{equation}

Recall distributional result: if $X\sim \chi_v^2, Y\sim \chi_w^2$ and $X,Y$ independent then $\frac{X/v}{Y/w}\sim F_{v,w}$.

Therefore, $ \frac{\frac{y^T (I-M y)}{n-p}}{\frac{y^T M y}{p}}\sim F_{p,n-p}$.

\subsection{Confidence intervals for $\hat{\beta}$}

Note that $\hat{\beta} \sim N_p (\beta,\sigma^2 (X^T X)^{-1})$, and that
$\frac{\hat{\sigma}^2}{\sigma^2} \sim  \frac{\chi^2_{n-p}}{n-p}$.

From distributional theory we know that $T=\frac{X}{\sqrt{Y/v}}$, when $X\sim N(0,1)$ and $Y\sim \chi^2_{v}$.

Let
 $x_i$ be a column vector containing the values of the explanatory/regressor variables for a new observation $i$. Then if we define:

\begin{equation}
X=\frac{x_i^T \hat{\beta} - x_i^T \beta}{\sqrt{\sigma^2 x_i^T (X^T X)^{-1}x_i}} \sim N(0,1)
\end{equation}

\noindent
and

\begin{equation}
Y=\frac{\hat{\sigma}^2}{\sigma^2} \sim  \frac{\chi^2_{n-p}}{n-p}
\end{equation}

It follows that  $T=\frac{X}{\sqrt{Y/v}}$:

\begin{equation}
T=  \frac{x_i^T \hat{\beta} - x_i^T \beta}{\sqrt{\hat{\sigma}^2 x_i^T (X^T X)^{-1}x_i}} =
\frac{  \frac{x_i^T \hat{\beta} - x_i^T \beta}{\sqrt{\sigma^2 x_i^T (X^T X)^{-1}x_i}}}{\sqrt{\frac{\hat{\sigma}^2}{\sigma^2}}}
 \sim t_{n-p}
\end{equation}

I.e., a 95\% CI:

\begin{equation}
x_i^T \hat{\beta} \pm t_{n-p,1-\alpha/2}\sqrt{\hat{\sigma}^2 x_i^T(X^T X)^{-1}x_i}
\end{equation}

Cf.\ a prediction interval:

\begin{equation}
x_i^T \hat{\beta} \pm t_{n-p,1-\alpha/2}\sqrt{\hat{\sigma}^2 (1+x_i^T(X^T X)^{-1}x_i)}
\end{equation}

Note that

\begin{enumerate}
\item
A prediction interval will be wider about the edges; this is because the term $\hat{\sigma}^2 (1+x_i^T(X^T X)^{-1}x_i)$ in the prediction interval formula is minimized at the mean value of the
predictor variable. When $x_i = \bar{x}$ we have the smallest value for the term, and so the further away the $x_i$ value from $\bar{x}$, the larger the interval.
\item
The width of the prediction interval stays much more constant around the range of observed values.
This is because 1 is much larger than $x_i^T(X^T X)^{-1}x_i)$; so if $x_i$ is near the mean value for $x$ then this term will not change much.

\end{enumerate}

\subsection{Distributions of estimators and residuals}

Covar$(\hat{\beta},e)=0$:

         Var$\begin{pmatrix}
	 \hat{\beta} \\
	e \\
	\end{pmatrix}
	=
	\begin{pmatrix}
	 Var(\hat{\beta}) & 0 \\
	 0 & Var(e) \\
	\end{pmatrix}
	=
	\begin{pmatrix}
	 \sigma^2 (X^T X)^{-1} & 0 \\
	 0 & \sigma^2 M \\
	\end{pmatrix}
	$.

\textbf{Confidence intervals for components of $\beta$}

Let $G=(X^T X)^{-1}$, and $g_{ii}$ the $i$-th diagonal element.

\begin{equation}
\hat{\beta}_i \sim N(\beta_i, \sigma^2 g_{ii})
\end{equation}

Since $\hat{\beta}$ and $S_r$ are independent, we have:

\begin{equation}
\frac{\hat{\beta}_i - \beta_i}{\hat{\sigma}\sqrt{g_{ii}}} \sim t_{n-p}
\end{equation}

The 95\% CI:

\begin{equation}
\hat{\beta}_i \pm t_{n-p,(1-\alpha)/2} \hat{\sigma} \sqrt{g_{ii}}
\end{equation}

\subsection{Maximum likelihood estimators}

For $\sigma^2$:

Let $X_i$, $i=1,\dots,n$ be a random variable with PDF $f(x; \sigma) = \frac{1}{2\sigma} exp (-\frac{\mid x \mid}{\sigma})$. Find $\hat \sigma$, the MLE of $\sigma$.

\begin{equation}
	L(\sigma) = \prod f(x_i; \sigma) = \frac{1}{(2\sigma)^n} exp (-\sum \frac{(x_i - \mu)^2}{\sigma^2})
\end{equation}

Let $\ell$ be log likelihood. Then:

\begin{equation}
	\ell (x; \sigma) = - n\log 2 - n\log \sigma - \sum (x_i - \mu)^2 /\sigma^2
\end{equation}

Differentiating and equating to zero to find maximum:

\begin{equation}
	\ell ' (\sigma) =  - \frac{n}{\sigma} +  \sum (x_i - \mu)^2 /\sigma^3 =
	 0
\end{equation}

Rearranging the above, the MLE for $\sigma$ is:

\begin{equation}
	\hat \sigma^2 =  \sum (x_i - \mu)^2 /n
\end{equation}

Since $S_r\sim \chi^2_{n-p}$, $E(S_r)=\sigma^2 (n-p)$. So we need to correct $S_r$ as $S_r/n-p$ to get $E(S_r)=\sigma^2$.

\subsection{Hypothesis testing}

A general format for specifying null hypotheses: $H_0: C\beta = c$, where $C$ is a $q\times p$ matrix and $c$ is a $q\times 1$ vector of known constants. The matrix $C$ effectively asserts specific values for $q$ linear functions of $\beta$. In other words, it asserts $q$ null hypotheses stated in terms of (components of) the parameter vector $\beta$.

 E.g., given:

\begin{equation}
y_i = \beta_0 + \beta_1 x_i + \beta_2 x_i^2+\epsilon_i
\end{equation}

\noindent
we can test $H_0: \beta_1=1, \beta_2=2$ by setting

$C=\begin{pmatrix}
0 & 1 & 0\\
0 & 0 & 1\\
\end{pmatrix}$
and $c=\begin{pmatrix}
1\\
2\\
\end{pmatrix}$.

The alternative is usually the negation of the null, i.e., $H_1: C\beta\neq c$, which means that at least one of the $q$ linear functions does not take its hypothesized value.

\textbf{Constructing a test}:

\begin{equation}
C\hat{\beta} \sim N_q (c,\sigma^2 C (X^T X)^{-1} C^T)
\end{equation}

So, if $H_0$ is true, by sum of squares property:

\begin{equation}
(C\hat{\beta} - c)^T [C (X^T X)^{-1} C^T]^{-1} (C\hat{\beta} - c) \sim \sigma^2 \chi_q^2
\end{equation}

In other words:

\begin{equation}
\frac{(C\hat{\beta} - c)^T [C (X^T X)^{-1} C^T]^{-1} (C\hat{\beta} - c)}{ \sigma^2} \sim \chi_q^2
\end{equation}

Note that $\hat{\beta}$ is independent of $\hat{\sigma}^2$, and recall that

\begin{equation}
\frac{\hat{\sigma}^2 }{\sigma^2} \sim \frac{\chi_{n-p}^2}{n-p}  \Leftrightarrow
\frac{\hat{\sigma}^2 (n-p)}{\sigma^2} \sim \chi_{n-p}^2
\end{equation}

Recall distributional result: if $X\sim \chi_v^2, Y\sim \chi_w^2$ and $X,Y$ independent then $\frac{X/v}{Y/w}\sim F_{v,w}$.

It follows that if $H_0$ is true,  and setting
$X=\frac{(C\hat{\beta} - c)^T [C (X^T X)^{-1} C^T]^{-1} (C\hat{\beta} - c)}{ \sigma^2}$,
$Y=\frac{\hat{\sigma}^2 (n-p)}{\sigma^2}$, and setting the degrees of freedom to $v=q$ and $w=n-p$:

\begin{equation}
\frac{X/v}{Y/w}=
\frac{\frac{(C\hat{\beta} - c)^T [C (X^T X)^{-1} C^T]^{-1} (C\hat{\beta} - c)}{ \sigma^2}/q}{\frac{\hat{\sigma}^2 (n-p)}{\sigma^2}/(n-p)}
\end{equation}

Simplifying:

\begin{equation}
\frac{(C\hat{\beta} - c)^T [C (X^T X)^{-1} C^T]^{-1} (C\hat{\beta} - c)}{q\hat{\sigma}^2} \sim F_{q,n-p}
\end{equation}

%The above test is the \textbf{generalized likelihood ratio test}.

This is a \textbf{one-sided test} even though the original alternative was two-sided.

\textbf{Special cases of hypothesis tests}:

When $q$ is 1, we have only one hypothesis to test, the $i$-th element of $\beta$. Given:

\begin{equation}
y_i = \beta_0 + \beta_1 x_i + \beta_2 x_i^2+\epsilon_i
\end{equation}

\noindent
we can test $H_0: \beta_1=0$ by setting

$C=\begin{pmatrix}
0 & 1 & 0\\
\end{pmatrix}$
and $c=0$.

Using the fact that $X\sim t(v)\Leftrightarrow X^2 \sim F(1,v)$, we have

\begin{equation}
\frac{\hat{\beta}_i - c_i}{\hat{\sigma}\sqrt{g_{ii}}} \sim t_{n-p}
\end{equation}

\subsection{Sum of squares}

This is a very important section!

\begin{fmpage}{\linewidth}

Recall:
If $K$ is idempotent, so is $I-K$. This allows us to split $x^T x$ into two components sums of squares:

\begin{equation}
x^T x = x^T K x+x^T (I-K) x
\end{equation}

 Let $K_1, K_2,\dots, K_q$ be symmetric idempotent $n \times n$ matrices such that
 $\sum K_i= I_n$ and $K_iK_j =0$, for all $i\neq j $. Let $x\sim N_n(\mu, \sigma^2)$.
 Then we have the following partitioning into independent sums of squares:

  \begin{equation}
x^T x = \sum x^T K_i x
\end{equation}

If $K_i \mu = 0$, then $ x^T K_i x\sim \sigma^2 \chi_{r_i}^2$, where $r_i$ is the rank of $K_i$.
\end{fmpage}

We can use the sum of squares property just in case $K$ is idempotent, and $K\mu =0$ . Below, $K=M$ and $\mu=E(y)=X\beta$.

Consider the sum of squares partition:

\begin{equation}
y^T y = \explain{\underline{y^T M y}}{S_r= e^T e} + \explain{\underline{y^T (I-M) y}}{\hat{\beta}^T (X^T X)\hat{\beta}}
\end{equation}

Note that the preconditions for sums of squares partitioning are satisfied:
\begin{enumerate}
\item $M$ is idempotent  (and symmetric), rank=trace=$n-p$.
\item $I-M$ is idempotent (and symmetric), rank=trace=$p$.
\item $ME(y) = 0$ because $ME(y)=MX\beta$ and $MX=0$.
\end{enumerate}

We can therefore partition the sum of squares into two independent sums of squares:

\begin{equation}
y^T y = \explain{\underline{y^T M y}}{e^T e \sim \sigma^2 \chi_{n-p}^2} \hbox{~~~~~~~~}+\hbox{~~~~~~~~}
\explain{\underline{y^T (I-M) y}}{ \sim \sigma^2 \chi_p^2 \newline \hbox{ iff } X\beta=0, i.e., \beta=0}
\end{equation}

So, iff we have $H_0: \beta=0$, we can partition sum of squares as above. Saying that $\beta=0$ is equivalent to saying that $X$ has rank $p$ and $X\beta=0$.

\subsection{Testing the effect of a subset of regressor variables}

Let:

\begin{equation}
C= (0_{p-q} I_q) \quad c=0, \hbox{ and } \beta=\begin{pmatrix} \beta_1\\ \beta_2 \end{pmatrix}
\end{equation}

Here, $\beta_{1,2}$ are vectors (sub-vectors?), not components of the $\beta$ vector.
Then, $C\times \beta = \beta_2$ and $H_0: \beta_2=0$. Note that order of elements in $\beta$ is arbitrary; i.e., any subset of $\beta$ can be tested.

Since  $C\times \beta = \beta_2$ and $c=0$, we can construct a sum of squares:

\begin{equation}
(C\hat{\beta} - c)^T [C (X^T X)^{-1} C^T]^{-1} (C\hat{\beta} - c) \sim \sigma^2 \chi_q^2
\end{equation}

This becomes (since $C\beta=\hat{\beta}_2$):

\begin{equation}
\hat{\beta}_2^T [C (X^T X)^{-1} C^T]^{-1} \hat{\beta}_2 \sim \sigma^2 \chi_q^2
\end{equation}

We can rewrite this as: $\hat{\beta}_2^T G_{qq}^{-1} \hat{\beta}_2$, where $G_{qq}= C (X^T X)^{-1} C^T$ ($G_{qq}$ should not be confused with $g_{ii}$) is a $q\times q$ submatrix of $G=(X^T X)^{-1}$.

Note that $\hat{\beta}$ is independent of $\hat{\sigma}^2$, and
recall that  $\frac{\hat{\sigma}^2 (n-p)}{\sigma^2} \sim \chi_{n-p}^2$. We can now construct the F-test as before:

\begin{equation}
\frac{\hat{\beta}_2^T C (X^T X)^{-1} C^T \hat{\beta}_2}{q\hat{\sigma}^2} =
\frac{\hat{\beta}_2^T G \hat{\beta}_2}{q\hat{\sigma}^2}
 \sim F_{q,n-p}
\end{equation}

\textbf{Sums of squares}:

We can construct three idempotent matrices:

\begin{itemize}
\item
$M = I_n - X(X^T X)^{-1} X^T$
\item
$M_1 =  X(X^T X)^{-1} X^T -   [X(X^T X)^{-1} C^T]  [\explain{\underline{C (X^T X)^{-1} C^T}}{G}]^{-1}
[C(X^T X)^{-1} X^T]$

(that is: $M_1 =  X(X^T X)^{-1} X^T - M_2$)

\item $M_2 = [X(X^T X)^{-1} C^T]  [\explain{\underline{C (X^T X)^{-1} C^T}}{G}]^{-1}
[C(X^T X)^{-1} X^T]$
\end{itemize}

Note that $M+M_1+M_2=I_n$ and $MM_1=MM_2=M_1M_2=0$. I.e., sum of squares partition property applies. We have three independent sums of squares:

\begin{enumerate}
\item $S_r = y^T M y$
\item $S_1 = y^T M_1 y = \hat{\beta}^T X^T X \hat{\beta}-  \hat{\beta}_2^T G_{qq}^{-1} \hat{\beta}_2$
\item $S_2 = y^T M_2 y =  \hat{\beta}_2^T G_{qq}^{-1} \hat{\beta}_2$
\end{enumerate}

So: $y^T y = S_r + S_1 + S_2$.  Then:

\begin{itemize}
\item It is unconditionally true that $S_r \sim \sigma^2 \chi^2_{n-p}$.
\item If $H_0: \beta=0$ is true, then $E(\hat{\beta}_2) = \beta_2 = 0$. It follows from the sum of squares property that $S_2 \sim \sigma^2 \chi_q^2$.
\item Regarding $S_1$:
We can prove that $M_1 = X_1 (X_1^T X_1)^{-1}X_1^T$, where $X_1$ contains the first $p-q$ columns of $X$. It follows that:

$S_1 = y^T M_1 y =y^T X_1 (X_1^T X_1)^{-1}X_1^T  y$

Note that $X_1 (X_1^T X_1)^{-1}X_1^T$ is idempotent. If $\beta=0$, i.e., if $E(y) =X\beta = 0$, we can use the  sum of squares property and conclude that

$S_1 \sim \sigma^2 \chi_{p-q}^2$

The degrees of freedom are $p-q$ because the rank=trace of  $X_1 (X_1^T X_1)^{-1}X_1^T$ is $n-p$.

\textbf{Thus, $S_1$ is testing $\beta_1=0$ but under the assumption that $\beta_2=0$}.

\end{itemize}

\textbf{Analysis of variance}

%\begin{table}[htdp]
%\caption{default}
%\begin{center}
\begin{tabular}{|l|c|c|c|c|}
\hline
Sources & SS & df & MS & MS ratio\\
 of variation & & & & \\
\hline
Due to $X_1$  & $S_1$ & $p-q$ & $S_1/(p-q)$ & $F_1$ \\
if $\beta_2=0$   d& & & & $F_{p-q,n-p}$\\
\hline
Due to $X_2$ & $S_2$ & $q$ & $S_2/q$ & $F_2$\\
& & & &  $F_{q,n-p}$\\
\hline
Residuals   & $S_r$ & $n-p$ & $\hat{\sigma}^2$ & \\
\hline
Total           & $y^T y$  & n &  &\\
\hline
\end{tabular}
%\end{center}
%\label{default}
%\end{table}%

Note:

\begin{enumerate}
\item
The ANOVA tests are \textbf{performed in order}:  First we test $H_0: \beta_2=0$. Then, if this test does not reject the null, we test $H_0: \beta_1 = 0$ \textbf{on the assumption (which may or may not be true)} that $\beta_2=0$.
\item What happens if we reject the first hypothesis?
\end{enumerate}

\textbf{The null or minimal model (constant term)}

We can set $C=I_p$ and $c=0$. This tests whether all coefficients are zero. But this states that $E(y)=0$, whereas it should have a non-zero value (e.g., reading times).  We include the constant term to accommodate this desire to have $E(y)=\mu=\neq 0$. In matrix format: let $\beta$ be the parameter vector; then, $\beta_1=\mu$ is the first, constant, term, and the rest of the parameters are the vector $\beta_2$ ($p-1\times 1$).
The first column of $X$ will be $X_1=1_n$.

\begin{enumerate}
\item
$S_1=y^T (X_1^T X_1)^{-1} X_1^T y = (\sum y)^2/n = n\bar{y}^2$
\item
$S_r = y^Ty - \hat{\beta}^T X^T X\hat{\beta}$
\item
$S_2 = y^T y -S_1 - S_r = \hat{\beta}^T X^T X\hat{\beta}-n\bar{y}^2$
\end{enumerate}

It is normal to omit the row in the ANOVA table corresponding to the constant term.

\medskip
\textbf{Testing whether all predictors (besides the constant term) are zero}

To test whether $p$ predictor variables have any effect on $y$,we set $q=p-1$, and our anova table looks like this:

\begin{tabular}{|l|l|l|l|l|}
\hline
Sources & SS & df & MS & MS \\
of variation  & & & & ratio \\
\hline
%Due to $X_1$ if $\beta_2=0$ & $S_1$ & $p-q$ & $S_1/(p-q)$ & ($F_1$) $  F_{p-q,n-p}$\\
%\hline
Due & $S_2$ & $p-1$ & $\frac{S_2}{(p-1)}$ & $F_2$\\
 to regressors & & & &  $F_{p-1,n-p}$\\
\hline
Residuals   & $S_r$ & $n-p$ & $\hat{\sigma}^2$ & \\
\hline
Total        & $S_{yy}=$  & n-1 &  &\\
 (\textbf{adjusted})    & $(y-\bar{y})^T(y-\bar{y})$ & & & \\
& $=y^T y - n\bar{y}^2$ & & & \\
\hline
\end{tabular}

Note that $S_{yy}=\sum (y_i - \bar{y})^2$ is the residual sum of squares that we get after fitting the constant $\hat{\mu}=\bar{y}$.

\medskip
\textbf{Testing a subset of predictors $\beta_2$}

\begin{tabular}{|l|l|l|l|l|}
\hline
Sources & SS & df & MS & MS \\
of variation  & & & & ratio \\
\hline
Due to $X_1$   & $S_1$ & $p-q-1$ & $\frac{S_1}{(p-q-1)}$ & ($F_1$) \\
if $\beta_2=0$ & & & & $F_{p-q-1,n-p}$\\
(test of $\beta_1$) & & & & \\
\hline
Due & $S_2$ & $q$ & $\frac{S_2}{q}$ & $F_2$\\
 to $X_2$ & & & &  $F_{q,n-p}$\\
(test of $\beta_2$) & & & & \\
\hline
Residuals   & $S_r$ & $n-p$ & $\hat{\sigma}^2$ & \\
\hline
Total        & $S_{yy}$  & n-1 &  &\\
\hline
\end{tabular}
%Note: the lecture notes have total SS as $y^T y$ but I think that's a typo.

%Used at the very beginning of a document:
%\verb!\documentclass{!\textit{class}\verb!}!.  Use
%\verb!\begin{document}! to start contents and \verb!\end{document}! to
%end the document.

\section{Checking model assumptions}

\subsection{Standardized residuals (\texttt{stdres} in R)}

Recall that $Var(e)=\sigma^2 M$, where $M = I_n -  X (X^T X)^{-1} X^T \quad \hbox{M is symmetric, idempotent } n\times n$. The diagonals of $M$ are all less than 1, and are not all equal (i.e., not equal variance), and off-diagonals are not 0 (i.e., the residuals are correlated). Correcting for unequal variance is done by the \textbf{scaled residual}:

\begin{equation}
e_i* = \frac{e_i}{\sqrt{m_{ii}}}
\end{equation}

Note: $Var(e_i*) = \sigma^2$ because $e_i \sim N(0,\sigma^2 m_{ii})$, therefore $e_i=\frac{e_i}{\sqrt{m_{ii}}}*\sim N(0,\sigma^2)$.

The \textbf{standardized residuals} are

\begin{equation}
s_i = \frac{e_i*}{\hat{\sigma}}
\end{equation}

This is approximately $t_{n-p}$ (approximately because $e_i*$ and  $\hat{\sigma}$ are not independent).
Since $s_i\sim t_{n-p}$, we can designate a residual as an outlier if $\mid s_i \mid > t_{crit}$ where $t_{crit}$ is the critical t-value.

\subsection{Standardized deletion residuals (\texttt{studres} in R)}

This is a more exact way to test for outliers than the above discussion. Define:

\begin{equation}
\hat{\beta}_{-i} = (X_{-i}^T X_{-i})^{-1} X_{-i}^T y_{-i}
\end{equation}

\noindent
where the $-i$ refers to removing data point $i$.
Standardized deletion residuals are

\begin{equation}
s_{-i} = \frac{e_i}{\hat{\sigma}_{-i}\sqrt{m_{ii}}}
\end{equation}

We can compute $s_{-i}$ from $s_{i}$:

\begin{equation}
s_{-i} = \frac{s_i \sqrt{n-p-1}}{\sqrt{n-p-s_{i}^2}} \sim t_{n-p-1}
\end{equation}

If $n$ is large, $s_{-i}\approx s_i$.

\subsection{Correcting for multiple testing}

\v{S}id\'ak correction:
``suppose we are performing $n$ tests and in each test we specify the probability of making a type I error to be $\beta$ (note: don't confuse this as type II error). Then, if the tests are independent, the probability of at least one false positive claim in the $n$ tests is given by

\begin{equation}
1-(1-\beta)^n = \alpha \Leftrightarrow \beta = 1-(1-\alpha)^{1/n}
\end{equation}

This correction ``has a stronger bound [than the Bonferroni] and so has greater statistical power.''

\subsection{Checks}

\begin{enumerate}
\item Normality: qqnorm etc. Hist is a useful addition to qqplot in large samples. For small samples, use scaled or standardized residuals if sample size is small (not sure why).
\item Independence: index-plots: residuals against observation number. Not useful for small samples. Or: compute correlation between $e_i, e_{i+1}$ pairs of residuals.
\item Homoscedasticity: residuals against fitted. Fan out suggests violation. A quadratic trend in a plot of residuals against predictor x could suggest that a quadratic predictor term is needed; note that $X^T e = 0$. (review exercises 3), so we will never have a perfect straight line in such a plot. Alternative: Bartlett's test.
\end{enumerate}

\subsection{Formal tests of normality}

Komogorov-Smirnov and Shapiro-Wilk. Only useful for large samples ; not very powerful and not much better than diagnostic plots. Tests may be useful as follow-ups if non-normality is suspected.

\subsection{Influence and leverage (\texttt{lm.influence\$hat} in R)}

A point can influence the parameter estimates without being an exceptional outlier. Influence does not depend on ``outlyingness''. Potential to influence (e.g., by being an extreme x value) is called leverage; once the y value is also extreme, we have influence. I.e., it takes an extreme x and y value to be influential, and it takes only an extreme x value to have leverage.

Leverage more formally defined: recall that $M = I_n -  X (X^T X)^{-1} X^T$. Define a hat matrix $H=I-M=X (X^T X)^{-1} X^T$. It's called a hat matrix because it puts a hat on y: $\hat{y} = X \hat{\beta} = Hy$.
Since $x_i^T$  is the $i$-th row of $X$, we have $h_{ii} = x_i^T (X^T X)^{-1}x_i$. The measure for leverage is:

\begin{equation}
h_{ii} = 1 - m_{ii}
\end{equation}

Notice that $h_{ii}$ is a scalar, so $\hbox{trace}(h_{ii}=h_{ii}$.
So (because for a square matrix A,B, tr(AB)= tr(BA)):

\begin{equation}
h_{ii} = tr(x_i^T (X^T X)^{-1}x_i)=tr(x_i^T x_i (X^T X)^{-1})
\end{equation}

Since $X^T X = \sum_{i=1}^n x_i x_i^T$, $h_{ii}$ represents the magnitude of $ x_i x_i^T$ relative to the sum of the values for all observations. Note that $h_{ii}$ only depends on X.

Also note that

\begin{equation}
\sum_{i=1}^n h_{ii} = tr(X^T X (X^T X)^{-1}) = tr(I_p)=p \quad mean(h_{ii})=p/n
\end{equation}

$h_{ii}$ measures leverage because $Var(e_i)=\sigma^2 m_{ii} = \sigma^2(1-h_{ii})$ and $Var(\hat{y}_i) = \sigma^2 h_{ii}$. Therefore $h_{ii}$ has to lie between 0 and 1. When it is close to one, the fitted value will be close to the actual value of $y_i$---signalling potential for leverage (aside by SV: the explanation sounds circular to me---this statement says it has leverage by definition. Also, I don't know why I should care that a data point has \textit{potential} to influence the estimates).

A cutoff one can use to identify high leverage points is $h_{ii} > 2p/n$ or $h_{ii} > 3p/n$.

The leverage of a data point is directly related to how far away it is from the mean:

\begin{equation}
h_{ii} = n^{-1} + \frac{(x_i - \bar{x})^2}{S_{xx}}
\end{equation}

In \texttt{lm.influence}, ``coefficients is the matrix whose i-th row contains the change in the estimated coefficients which results when the i-th case is dropped from the regression. sigma is a vector whose i-th element contains the estimate of the residual standard error obtained when the i-th case is dropped from the regression'' (p.\ 71 of lecture notes).

\subsection{Cook's distance D: A measure of influence}

Let $s_i$ be the i-th standardized residual, $\hat{\beta}_{-i}$ the estimate of the vector of parameters with the i-th row removed.

\begin{equation}
D_i =  \frac{(\hat{\beta}-\hat{\beta}_{-i})^T(X^T X)^{-1}(\hat{\beta}-\hat{\beta}_{-i})}{p\hat{\sigma}^2} = \frac{s_i^2 h_{ii}}{p(1-h_{ii})}
\end{equation}

A data point is influential if it is outlying as well as high leverage. Cutoff for Cook's distance is $\frac{4}{n}$.

\textbf{Procedure for checking model fit}:
to-do, see p 73

\subsection{Transformations}

Suppose $Y$ is a random variable whose variance depends on its mean. I.e., $E(y)=\mu, Var(y)=g(\mu)$.
The function $g(\cdot)$ is known.

We seek a transformation from $y$ to $z = f(y)$ such that $Var(z)$ is (approximately) constant.

Expand $f(\cdot)$ in a Taylor series expansion, keeping only the first-order term:

\begin{equation}
z= f(y)\approx f(\mu) + (y-\mu)f'(\mu)
\end{equation}

Then: $E(z)=f(\mu)$ and $Var(z)=g(\mu) f'(\mu)^2$. The variance needs to be constant at, say, $k^2$:

\begin{equation}
k^2 = g(\mu) f'(\mu)^2 \Rightarrow f'(\mu)=\frac{k}{\sqrt{g(\mu)}}
\end{equation}

So,

\begin{equation}
f(\mu) = \int f'(\mu) = k\int \frac{1}{\sqrt{g(\mu)}}
\end{equation}

\begin{fmpage}{\linewidth}
\begin{equation}
f(\mu) =  k\int [\sqrt{g(\mu)}]^{-1/2}\, d\mu
\end{equation}
\end{fmpage}

\textbf{Example 1}: Let $g(\mu)=a\mu$; then $f(\mu)=2k\sqrt{\frac{\mu}{a}}$. So, $z=\sqrt{\mu}$.

\textbf{Example 2}: Let $g(\mu)=a\mu^2$; then $f(\mu)=k\sqrt{\frac{1}{a}}\log \mu$. So, $z=\log \mu$.

\textbf{Estimating a transformation}: $\exists \lambda$ such that

\begin{equation}
f_\lambda (y_i) = x_i^T \beta + \epsilon_i \quad \epsilon_i \sim N(0, \sigma^2)
\end{equation}

We use maximum likelihood estimation to estimate $\lambda$. Note that

$L(\beta_\lambda, \sigma^2_\lambda, \lambda; y) \propto$

\begin{equation}
(\frac{1}{\sigma})^n \exp [-\frac{1}{2\sigma^2} \sum [f_\lambda(y_i)-  x_i^T \beta ]^2] [\prod \explain{f'_\lambda(y_i)}{\hbox{Jacobian}}]
\end{equation}

For fixed $\lambda$, we estimate $\hat{\beta}$ and $\hat{\sigma}^2$ in the usual MLE way, and then we turn our attention to $\lambda$:

\begin{equation}
L(\hat{\beta}_\lambda, \hat{\sigma}^2_\lambda, \lambda; y) = S_\lambda^{-n/2}\prod f'_\lambda(y_i)
\end{equation}

Taking logs:

\begin{equation}
\ell = c-\frac{n}{2} \log S_\lambda + \sum \log f'_\lambda(y_i)
\end{equation}

\textbf{Box-Cox family}:

\begin{equation}
f_\lambda (y) = \left\{
\begin{array}{l l}
       \frac{y^\lambda - 1}{\lambda}   & \lambda \neq 0\\
       \log y & \quad \lambda=0\\
\end{array}
\right.
\end{equation}

We assume that $f_\lambda (y) \sim N(x_i^T \beta,\sigma^2)$. So we have to just estimate $\lambda$ by MLE, along with $\beta$.

\textbf{Box-Cox by hand}:

Since $f_\lambda=\frac{y^\lambda-1}{\lambda}$, it follows that $f'_\lambda(y)= y^{\lambda-1}$.

Now, for different $\lambda$ you can figure out the log likelihoods by hand by solving this equation:

\begin{equation}
\ell = c-\frac{n}{2} \log \explain{S_\lambda}{\hbox{Residual sum of squares}} + (\lambda-1)\sum \log (y_i)
\end{equation}

\section{Factors}

\subsection{Overcoming multicollinearity through parameterization}

to-do: multicollinearity explanation

If the model matrix $X$ is not full rank (this is true when we include a column for the intercept), then we can put constraints on the predictors (through parameterization). E.g., treatment contrasts (corner-point constraints), sum contrasts, etc.

to-do constraints on p.\ 94-95

%\subsection{Merging factor levels}

%This is done to increase group size.

\subsection{Model selection}

\textbf{$S_r$ and $R^2$ can't be used for model selection}:
``$S_r$ will always decrease when we add more regressor variables, so the best fitting model is always the full model which contains all the possible regressor variables - so $S_r$ is not a good model selection tool. For the same reason, the coefficient of determination $R^2$ is not a useful measure in model selection. $R^2$ will not decrease as the number of parameters increases (i.e. it is a non-decreasing function of the number of parameters in the model).''

\textbf{Penalized likelihood methods for model comparison}:

We can compare models using log likelihood:

\begin{equation}
\ell = n \log \hat{\sigma}^2 + z(p)
\end{equation}

\noindent
where $z$ is some penalty function. ``Then we declare that the optimal model is that which minimizes $\ell$. We can think of $z$ as an ad hoc adjustment that tries to give simpler models credit for having fewer regressor variables.''

AIC etc cannot be used to compare across datasets, but can be used to compare non-nested models (cf.\ ANOVA, which allows only nested models to be compared).

\textbf{AIC}: here, $z(p)= 2p$. To calculate AIC:

\begin{equation}
AIC = 2p + n\log \frac{S_r}{n}
\end{equation}

[Note: does not match up with the AIC function output in R.]

\textbf{Where does AIC come from?} From the fact that maximum likelihood of $\hat{\sigma}^2$ is:

\begin{equation}
  L(\sigma) \propto  (\hat{\sigma}^2)^{-n/2}
\end{equation}

$-2\times loglik=n\log \hat{\sigma}^2=n\log \frac{S_r}{n}$ is a good model selection tool: smaller values (smaller $S_r$) will mean better fit.

\textbf{BIC}:  $z(p)= p\log n$. This penalty will be large for large $n$, compared to AIC.

\textbf{Mallow's $C_p$}:

\begin{equation}
C_p = \frac{S_r}{\hat{\sigma}_f^2} - n + 2p_r
\end{equation}

$\hat{\sigma}_f^2$ is the residual mean square of the full model, $S_r$ is residual
sums of squares of  reduced model, $p_r$ is the number of regressors in the reduced model.

We want a small $C_p$ and $C_p \approx p$.

\textbf{Best subsets method}: in leaps library, regsubsets command.

\begin{verbatim}
b<-regsubsets(model specification)
summary(b)$rsq, $cp, $bic
\end{verbatim}

\textbf{Backward elimination}: fit full model, and remove the t-value that's smallest, and so on.
Forward elimination goes in the other direction.

\textbf{Stepwise selection}: step function in MASS. Incrementally add a predictor as above, but once one is added, try removing other predictors with smallest t-value. Repeat until nothing can be added or deleted.

\begin{verbatim}
step(fm,scope=list(upper/lower=formula),
        direction="forward/backward/both")
\end{verbatim}

\section{Generalized least squares}

Let $Var(\epsilon)=\sigma^2 \Sigma$, where  $\Sigma$ is known and non-singular. If $\Sigma\neq I_n$ then we either have correlation or non-equal variance or both. If $\Sigma$ is known, we only need to estimate $\sigma$ and we are back in least squares theory, with some modification:

\begin{equation}
y \sim N(X\beta,\sigma^2 \Sigma)
\end{equation}

Likelihood $L(\beta,\sigma^2; y)$ is now:

\begin{equation}
 (2\pi)^{-n/2}\mid \sigma^2 \Sigma\mid^{-1/2} \exp[-\frac{1}{2\sigma^2}(y-X\beta)^T \Sigma^{-1}  (y-X\beta)]
\end{equation}

The MLE of $\beta$ minimizes:

\begin{equation}
S=(y-X\beta)^T \Sigma^{-1}  (y-X\beta)
\end{equation}

instead of $(y-X\beta)^T (y-X\beta)$.

\textbf{Least squares estimators}:

$\hat{\beta}: (X^T \Sigma^{-1} X)^{-1}X^T \Sigma^{-1} y$

$E(\hat{\beta})=\beta$

$Var(\hat{\beta})=\sigma^2 (X^T \Sigma^{-1}X)^{-1}$

Estimator of $\sigma^2$ is $\frac{S_v}{n-p}$, where

$S_v=y^T \Sigma^{-1} y - \hat{\beta}^T X^T \Sigma^{-1} X\hat{\beta} = y^T \Sigma^{-1} y - \hat{\beta}^T X^T \Sigma^{-1} y$.

\subsection{Weighted least squares}

Suppose $\Sigma=diag(c1,\dots,cn)$ (uncorrelated, but not homoscedastic) and so $\Sigma^{-1}=diag(1/
c1,\dots,1/cn)$. Let $w_i = 1/c_i$.

The sum of squares will be
$S=\sum_{i=1}^n w_i (y_i - x_i^T \beta)^2$. So each squared residual is weighted by that observation's variance; observations with large variance are less reliable, and are down-weighted.

Compared to $X^T X = \sum x_i x_i^T$ in WLS we have  $X^T \Sigma^{-1} X = \sum w_i x_i x_i^T$.
And instead of $X^T y = \sum x_i y_i$ in WLS we have $X^T \Sigma^{-1}y = \sum w_i x_i y_i$.

So $\hat{\beta}=(X^T \Sigma^{-1}  X)^{-1}X^T \Sigma^{-1}y= ( \sum w_i x_i x_i^T)^{-1} (\sum w_i x_i y_i)$

``Weighted LS is appealing because it allows us to adjust for different (known) variances in the observations in an intuitive way that is easy to implement. The variances of the different observations might be known from pilot or previous studies'' (or are estimated from data).

[SV: I don't get this ``known variance'' business. How can we ever \textbf{know} what the variance is? Pilot or previous data will only yield estimates.]

The main disadvantage of WLS is that weights have to specified in advance.

\subsection{OLS vs WLS}

With dataset wls1.txt, if we fit:

\begin{verbatim}
summary(m0<-lm(y~X.x,data))
summary(m0.wls<-lm(y~X.x,data,
                   weights=I(1/X.x^2)))
\end{verbatim}

The coefs will be the same in each, but SEs of coefs will be smaller in WLS fit (because $S_r$ will be down-weighted.

\textbf{Effect of scaling weights}:

Multiplying the weights by some constant will change residual standard error, but leaves SEs and coefs unchanged. This is because $Var(\hat{\beta})=\sigma^2 (X^T \Sigma^{-1} X)^{-1}$, so whatever factor $\sigma^2$ gets multiplied by, it will be cancelled out because it also appears in $ \Sigma^{-1}$.

Differently put:

``Let $w_i'$ be the scaled weights, $\hat{\sigma}'$ be the residual standard error in the analysis with the scaled weights, $S_r'$ be the residual sum of squares for the scaled analysis and $(\Sigma^{-1})'$ be the weight matrix for the scaled analysis. Then if $w_i' = 16w_i$, we see that $\hat{\sigma}' = 4\hat{\sigma}$ since $S_r' = 16S_r$.''

The SEs will not change because: ``If
$(\Sigma^{-1})'=16\Sigma^{-1}$ and
$\hat{\sigma}' = 4\hat{\sigma}$
then $Var(\hat{\beta}') =
(\hat{\sigma}^2)' (X^T (\Sigma^{-1})'X)^{-1} =
(\hat{\sigma}^2)  (X^T (\Sigma^{-1})X)^{-1}$.
So the standard errors don’t change.''

\begin{verbatim}
summary(m0.wls<-lm(y~X.x,data,
                   weights=I(16*1/X.x^2)))
\end{verbatim}

\begin{enumerate}
\item
If you get the weights wrong, SEs will increase. So, if unsure about weights use OLS.
\item
``If looking at the standardized residuals shows that there are observations that may be outliers and they reside in regions that will be given high weight then it may be safer to use OLS rather than WLS.''
\item One can estimate the weights from the data, but one needs lots of replicates for this, with fewer replicates the SEs will increase.
\item If you don't have enough replicates, you can group x values close to each other.
\item Outliers can dramatically influence estimates in WLS.
\end{enumerate}

Conclusion: WLS is a powerful tool, if weights are known, but outliers must be studied carefully.

\subsection{Using group means in WLS with replicated data}

Without replicates in the data, we have:

\begin{equation}
X^T y = \sum_i^n x_i y_i
\end{equation}

If we have $k$ replicates:

\begin{equation}
X^T y = \sum_{i=1}^k \sum_{j=1}^{n_i} x_i y_{ij} =
\sum_{i=1}^k x_i  \sum_{j=1}^{n_i}  y_{ij}
= \sum_{i=1}^k n_i x_i \bar{y}_i
\end{equation}

This is equivalent to having $\bar{y}_i$ as observations for the replicate sets, and $n_i$ as weights in WLS.
If we had $\bar{y}_i$ as observations, then their variances would be $\sigma^2/n_i$, so we would have unequal variances and would use WLS with $w_i = n_i$. \textbf{Example}: tractor data.

\subsection{Replication}

Define replicates here as repeated measurements that are mutually independent (cf.\ replicates which are not independent, as in linear mixed model theory).
Since all $x_i^T$ within a replicate set come from the same distribution, their variance should be an estimate of $\sigma^2$.

Let $y_{ij}$ be the $j$th observation in the $i$th replicate set, where $j=1,\dots,n_i$ and $n_i$ is the size of the $i$th replicate set ($i=1,\dots,k$). When $n_i = 1$ we have no replication, and for higher values we have replication. \textit{Within} each replicate, we can produce an estimator:

\begin{equation}
\hat{\sigma}^2 =  (n_i - 1)^{-1}  \sum_{j=1}^{n_{i}} (y_{ij} - \bar{y}_i)^2 = (n_i - 1)^{-1}S_i
\end{equation}

$\bar{y}_i=n_i^{-1} \sum_{j=1}^{n_i} y_{ij}$  is the mean of the replicate set.
$S_i \sim \sigma^2 \chi_{n_i - 1}^2$.

In a one-factor model, $S_r = \sum_{i=1}^k S_i$ and dfs are $n-k = \sum_{i=1}^k (n_i -1)$.
So, in the general case,  $df_r=n-k$,  $S_r = \sum_{i=1}^k S_i\sim \chi_{df_R}^2$. The ratio $\hat{\sigma}^2/df_R$ is an unbiased estimator of $\sigma^2$.

The distributional fact being used here is that the sum of independent chi-squared distributions has a chi-squared distribution, and the degrees of freedom is the sum of dfs of the RVs being summed.

\textbf{The replication estimator}: $\hat{\sigma}_R^2=\frac{S_R}{df_R}$ is the replication estimator of $\sigma^2$.

``The $\hat{\sigma}_R^2$ is independent of the particular form that we have used for the model. We obtain the same replication sum of squares with the same degrees of freedom whether we postulate a linear, quadratic, or some other relationship between Maintenance cost and Age.'' (to-do: don't get this). This is the strength of the replication sum of squares.

The $S_R$ is in general not equal to $S_r$.

\subsection{Partitioning replication sum of squares}

to-do

\bibliographystyle{plain}
\bibliography{/Users/shravanvasishth/Dropbox/Bibliography/bibcleaned}


\section{Options, Part 2}

\subsection*{Binomial model: risk-neutral pricing}

Solve by replication $\delta$ shares of the stock, $b$ dollars of riskless bond

$\delta u S_0 + b (1+r) = C_u$ \\
$\delta d S_0 + b (1+r) = C_d$

Solution: $\delta = \frac{C_u-C_d}{(u-d)S_0}$ ,
$b=\frac{1}{1+r}\frac{uC_d-dC_u}{u-d}$ \\
Then: $C_0 = \delta S_0 + b$ \\
$ C_0 = \frac{C_u-C_d}{(u-d)} + \frac{1}{1+r} \frac{uC_d-dC_u}{u-d}$ \\

\subsection*{Risk neutral probability}

$q_u=\frac{(1+r)-d}{u-d}$, $q_d = \frac{u-(1+r)}{u-d}$ Then

$C_0=\frac{q_u C_u + q_d C_d}{1+r} = \frac{E^Q[C_T]}{1+r}$
where $E^Q[\cdot]$ is the expectation under probability $Q=(1,1-q)$, which is
called the risk-neutral probability

\subsection*{State prices and risk-neutral probabilities}
$\Phi_u = \frac{q}{1+r}$ , $\Phi_d = \frac{1-q}{1+r}$

$\Phi_{uu} = \frac{q^2}{(1+r)^2}$ , $\Phi_{ud} = \Phi_{du} = \frac{q(1-q)}{(1+r)^2}$ , $\Phi_{dd} = \frac{(1-q)^2}{(1+r)^2}$
With state prices, can price any state-contingent payoff as a portfolio of
state-contingent claims: mathematically equivalent to the risk-neutral
valuation formula.

\subsection*{Implementing binominal model}

\begin{itemize}
	\item As we reduce the length of the time step, holding the maturity fixed, the
	binomial distribution of log returns converges to Normal distribution.
	\item Key model parameters $u$, and $d$ need to be chosen to reflect the
	distribution of the stock return
\end{itemize}

Once choice is:  $u=exp(\sigma \frac{T}{n})$, $d=\frac{1}{u}$, $p=\frac{1}{2} + \frac{1}{2} \frac{u}{\sigma} \sqrt{\frac{T}{n}}$

\subsection*{Black-Scholes-Merton formula}

$C_0 = S_0 N(x) - K e^{-rT} N (x-\sigma \sqrt(T))$ \\
$x = \frac{ln(\frac{S_0}{K e^{-rT}})}{\sigma \sqrt{T}} + \frac{1}{2} \sigma \sqrt{T}$ \\

In Excel \texttt{$N(x)$=NORM.S.DIST(x,TRUE)}
The call is equivalent to a levered long position in the stock;
 $S_0 N(x) $ is the amount invested in the stock;
$K e^{-rT} N (x-\sigma \sqrt(T))$ is the dollar amount borrowed.

Equivalent formulation:
$ C(S_t, t) = N(d_1)S_t - N(d_2)Ke^{-r(T - t)} $ \\
$ d_1 = \frac{1}{\sigma\sqrt{T - t}}\left[\ln\left(\frac{S_t}{K}\right) + \left(r + \frac{\sigma^2}{2}\right)(T - t)\right] $\\
$ d_2 = d_1 - \sigma\sqrt{T - t} $

The price of a corresponding put option based on put-call parity  is:
$ P(S_t, t) = Ke^{-r(T - t)} - S_t + C(S_t, t) =  N(-d_2) Ke^{-r(T - t)} - N(-d_1) S_t $

\subsection*{Option Greeks}

Delta: $\delta = \frac{\partial C}{\partial S}$
Omega: $\Omega = \frac{\partial C}{\partial S} \frac{S}{C}$
Gamma: $\Gamma = \frac{\partial \delta}{\partial S} = \frac{\partial^2 C}{\partial S^2} $
Theta: $\Theta = \frac{\partial C}{\partial S}$
Vega:  $\mathcal{V} = \frac{\partial C}{\partial S}$