\chapter{Numerical methods for PDE}

PDE stands for Partial Differential Equations, and we are going to study in particular Finite Difference Methods, not Finite Element Methods.

PDEs can be divided in three categories: elliptic, parabolic and high-parabolic. We will see a test problem on each of these categories.

\section{Poisson 2D problem with Dirichlet boundary conditions}
\begin{equation}
\text{Poisson equation:\quad} \Delta u(x, y) = f(x, y), \quad (x, y) \in \Omega
\end{equation}

\begin{equation}
\text{Dirichlet Boundary condition:\quad} u(x, y) = g(x, y), \quad (x, y) \in \Gamma
\end{equation}

where $\Omega = (0, 1)^2$

% im:I

Example: the electric potential inside a domain with a defined potential inside, $f$, and a fixed potential, $g$, at the border.

The Poisson 2D problem can be solved analytically using \textit{separation of variables} by searching for solutions of this type:

\begin{equation}
	u(x, y) = X(x) Y(y)
\end{equation}

Discretization: fix N positive integer and put $h = \frac{1}{N}$ as the stepsize.

Now we discretize the plane $\mathbb{R}^2$ by the mesh:

\begin{equation}
	\mathbb{R}^2_h = \{(mh, nh): m,n \in \mathbb{Z}\}
\end{equation}

% im:2

For a mesh point $(mh, nh)$, with $m, n \in \mathbb{Z}$ we can define four neighbors:

% im:3

The discrete version of $\Omega$ is

\begin{equation}
	\Omega_h := \Omega \cap \mathbb{R}^2_h
\end{equation}

% 4

while the discrete version of $\Gamma$ is the set of points $\Gamma_h$ in $\mathbb{R}^2_h$ which are not in $\Omega_h$, but have a neighbor in $\Omega_h$.

Notice how $\Gamma_h \neq \Gamma \cap \mathbb{R}^2_h$ because the four corners $\{(0,0), (0,1), (1,0), (1,1)\}$ are not included.

We also define $\overline{\Omega_h} = \Omega_h \cup \Gamma_h$: this is the discrete version of $\overline{\Omega_h}  = \Omega \cup \Gamma$.\\

The discrete version of the Poisson problem with Dirichlet Boundary conditions is:

\begin{equation}
\text{discrete Poisson equation:\enspace} \Delta_h u_h(x, y) = f(x, y), (x, y) \in \Omega_h
\end{equation}

\begin{equation}
\text{discrete Dirichlet Boundary condition:\enspace} u_h(x,y) = g(x, y), (x, y) \in \Gamma_h
\end{equation}

An operator is a map/function, whose domains and codomains are set of functions.

$\Delta_h$ is a discrete Laplacian, i.e. and operator such that:
\begin{itemize}
	\item it associates (as a map) to a mesh function $\overline{\Omega_h} \rightarrow \mathbb{R}$ an interior mesh function $\Omega_h \rightarrow \mathbb{R}$;

	\item it approximates the Laplacian, in the sense that, for function $v: \overline{\Omega_h} \rightarrow \mathbb{R}$ sufficiently smooth, which means $C^2(\overline{\Omega_h})$:

		\begin{equation}
			\Delta_h v |_{\overline{\Omega_h}} \; (x,y) \approx \Delta v(x, y) \quad (x, y) \in \Omega_h
		\end{equation}

	where $|_{\overline{\Omega_h}}$ is the restriction of $v$ to $\overline{\Omega_h}  \subseteq \overline{\Omega}$.
\end{itemize}

We are going to show how to construct a discrete Laplacian.\\

Consider a function $v$ of only one variable $t$. We discretize $v''(t)$ by:

\begin{equation}
\frac{v(t - h) - 2 v(t) + v(t + h)}{h^2}
\end{equation}
.

%5

\begin{equation}
\frac{\frac{v(t + h) - v(t)}{h} - \frac{v(t) - v(t - h)}{h}}{h} \approx\\
\frac{v'(t) - v'(t-h)}{h} \approx\\
v''(t-h) \approx v''(t)
\end{equation}

By Taylor approximations under the assumptions that v is $C^4$:

\begin{equation}
v(t-h) = v(t) - v'(t) h + \frac{1}{2} v''(t) h^2 - \frac{1}{6} v'''(t) h^3 + \frac{1}{24} v^{(4)}(\alpha_h) h^4
\label{t-h}
\end{equation}

where $\alpha_h \in [t-h, t]$;

\begin{equation}
v(t+h) = v(t) + v'(t) h + \frac{1}{2} v''(t) h^2 + \frac{1}{6} v'''(t) h^3 + \frac{1}{24} v^{(4)}(\beta_h) h^4
\label{t+h}
\end{equation}

where $\beta_h \in [t, t+h]$.

By summing \ref{t-h} and \ref{t+h} we obtain:
\begin{equation}
v(t-h) + v(t+h) = 2 v(t) + v''(t) h^2 + \frac{1}{24} (v^{(4)}(\alpha_h) + v^{(4)}(\beta_h)) h^4
\end{equation}

\begin{equation}
v(t-h) - 2 v(t) + v(t+h) = v''(t) h^2 + \frac{1}{24} (v^{(4)}(\alpha_h) + v^{(4)}(\beta_h)) h^4
\end{equation}
\begin{equation}
\frac{v(t-h) - 2 v(t) + v(t+h)}{h^2} = v''(t) + \frac{1}{24} (v^{(4)}(\alpha_h) + v^{(4)}(\beta_h)) h^2
\end{equation}

So we got an approximation of the second derivative of the function $v$.

\subsection{Discretization of the Laplacian}

For a smooth function $v: \overline{\Omega_h} \rightarrow \mathbb{R}$ we approximate the Laplacian:\\

% 6

by:

\begin{equation}
\Delta_h v(x, y) = \frac{v(x-h, y) - 2v(x,y) + v(x+h, y)}{h^2} + \frac{v(x, y-h) - 2v(x,y) + v(x, y+h)}{h^2}
\end{equation}

at $(x, y) \in \Omega_h$.

% 7

The computation of $\Delta_h v(x, y)$ requires $v(x-h, y)$ and $v(x, y+h)$ with $(x-h, y), (x, y+h) \in \Gamma_h$.

This discretization of the Laplacian is called the \textit{Five-points discretization}:

% 8

\subsection{Assess approximation of the discrete Laplacian}

Consider $v: \overline{\Omega} \rightarrow \mathbb{R}$ of class $C^4$. For $(x, y) \in \Omega_h$:

\begin{equation}
|\Delta_h v {|}_{\overline{\Omega_h}} (x,y) \Delta v(x,y)|
\end{equation}

\begin{equation}
|\frac{v(x-h, y)  - 2v(x, y) + v(x+h, y)}{h} + \frac{v(x, y-h) - 2v(x, y) + v(x, y+h)}{h} - \frac{\partial^2 v}{\partial x^2}(x, y) - \frac{\partial^2 v}{\partial y^2}(x, y)|
\end{equation}

which is $\leq$ than:

\begin{equation}
|\frac{v(x-h, y) - 2v(x, y) + v(x+h, y)}{h^2} - \frac{\partial^2 v}{\partial x^2}(x, y)| + |\frac{v(x, y-h) - 2v(x, y) + v(x, y+h)}{h^2} - \frac{\partial^2 v}{\partial y^2}(x, y)|
\label{buh}
\end{equation}
.

\ref{buh} is equal to:

\begin{equation}
|\frac{1}{24} ( \frac{\partial^4 v}{\partial x^4}(\alpha_h, y) + \frac{\partial^4 v}{\partial x^4}(\beta_h, y) \; h^2) | + |\frac{1}{24} ( \frac{\partial^4 v}{\partial y^4}(x, \gamma_h) + \frac{\partial^4 v}{\partial y^4}(x, \delta_h) \; h^2) |
\label{buh2}
\end{equation}

where $\alpha_h \in [x-h, x], \beta_h \in [x, x+h], \gamma_h \in [y-h, y], \delta_h \in [y, y+h]$.

\ref{buh2} (the error) can be bounded ($\leq$) by:

\begin{equation}
\frac{1}{6} \max\{ \max_{(x, y) \in \overline{\Omega}} |\frac{\partial^4 v}{\partial x^4} (x, y)|, \max_{(x, y) \in \overline{\Omega}} |\frac{\partial^4 v}{\partial y^4} (x, y)| \} \; h^2
\end{equation}

So we can conclude that:

\begin{equation}
\max_{(x,y) \in \Omega_h}  |\Delta_h v {|}_{\overline{\Omega_h}} (x,y) \Delta v(x,y)| \leq \frac{1}{6} \max\{ \max_{(x, y) \in \overline{\Omega}} |\frac{\partial^4 v}{\partial x^4} (x, y)|, \max_{(x, y) \in \overline{\Omega}} |\frac{\partial^4 v}{\partial y^4} (x, y)| \} \; h^2
\end{equation}

This is called the \textit{consistency error} of the five point approximation of the discrete Laplacian.

\subsection{Exercise}

Given $v \in C^6(\Omega)$, find  a function $C(x,y)$ of the point $(x, y) \in \overline{\Omega}$ and $M \ge 0$ such that
\begin{equation*}
\max_{(x,y) \in \Omega_h}  |\Delta_h v {|}_{\overline{\Omega_h}} (x,y) - \Delta v(x,y) - C(x, y) h^2| \le M h^4
\end{equation*}

\begin{equation*}
%\max_{(x,y) \in \Omega_h}  |
\Delta_h v {|}_{\overline{\Omega_h}} (x,y) = \Delta v'(x,y) + \text{error}
\end{equation*}

with $\text{error} = O(h^2)(*) = C(x, y) h^2 + O(h^4) (**)$.

Remind that:
\begin{equation}
v''(t) \approx \frac{v(t - h) - 2 v(t) + v(t + h)}{h^2}
\end{equation}

we can continue with the Taylor expansion:
\begin{equation*}
v(t-h) = v(t) - v'(t) h + \frac{v''(t)}{2} h^2 - \frac{v'''(t)}{6} h^3 + \frac{v^{4}(t)}{24} h^4  - \frac{v^{5}(t)}{120} h^5 + \frac{v^{6}(t)}{6!} h^6
\end{equation*}

\begin{equation*}
v(t-h) = v(t) + v'(t) h + \frac{v''(t)}{2} h^2 + \frac{v'''(t)}{6} h^3 + \frac{v^{4}(t)}{24} h^4  + \frac{v^{5}(t)}{120} h^5 + \frac{v^{6}(t)}{6!} h^6
\end{equation*}

where $\alpha \in [t-h, t]$ and $\beta \in [t, t+h]$.

\begin{equation*}
v(t-h) + v(t+h) = 2v(t) + v''(t) h^2 + \frac{v^{4}(t)}{12} h^4 + \frac{v^{6}(\alpha)}{6!} h^6 + \frac{v^{6}(\beta)}{6!} h^6
\end{equation*}

\begin{equation*}
\frac{v(t-h) - 2v(t)+ v(t+h)}{h^2} - v''(t) h^2 = \frac{v^{4}(t)}{12} h^4 + \frac{v^{6}(\alpha)}{6!} h^6 + \frac{v^{6}(\beta)}{6!} h^6
\end{equation*}

\dots

The discrete problem:
\begin{equation*}
\text{Poisson equation:\quad} \Delta u_h(x, y) = f(x, y), (x, y) \in \Omega_h
\end{equation*}

\begin{equation*}
\text{Dirichlet Boundary condition:\quad} u_h(x, y) = g(x, y), (x, y) \in \Gamma_h
\end{equation*}

is a linear system of $(N-1)^2$ unknowns (where $N$ is the number of subdivisions we do in each dimension of the square)  and $(N-1)^2$ equations.

The unknowns are $u_{ij} := u_h (ih, jh)$ with $i \in \{1, \dots, N-1 \}$ and $j \in \{1, \dots, N-1 \}$.

% 1 : 2018/03/15

.. we don't consider the values on the borders as unknowns because they are known, from the second equation of the Dirichlet Boundary condition.

On the border $\Gamma_h$:

\begin{equation*}
u_{ij} = u_h (ih, jh) = h_{ij} := g(ih, jh)
\end{equation*}

where
\begin{equation*}
(i,j) \in \{0\} \times \{1, \dots, N-1 \} \cup \{1, \dots, N-1 \} \times \{0\} \cup \{N\} \times  \{1, \dots, N-1 \} \cup  \{1, \dots, N-1 \} \times \{N\}
\end{equation*}

The equations are
\begin{equation*}
\Delta_h u_h (x, y) = f(x, y), \quad \in \Omega_h
\end{equation*}

each equation corresponds to a point in $\Omega_h$ with $(N-1)^2$ equations.

The equation corresponding to the point $(ih, jh)$, with $i \in  \{1, \dots, N-1 \}, j \in  \{1, \dots, N-1 \}$ is:

\begin{equation*}
\frac{u_{i-1\; j} - 2 u_{ij} + u_{i+1\; j}}{h^2} + \frac{u_{i\; j-1} - 2 u_{ij} + u_{i\; j+1}}{h^2} = f(ih, jh) = f_{ij}
\end{equation*}

\begin{equation*}
u_{i\; j-1} + u_{i-1\; j} -4 u_{ij} +  u_{i+1\; j} + u_{i\; j+1} = h^2 f_{ij}
\end{equation*}

In order to apply a linear system of the form $A u = b$, we need to re-arrange the equations of the points of the square (which are 2D) into a 1D vector.

Consider $N=4$:

% 2
\dots

We want to rewrite our matrix of unknowns $u$:

\begin{equation*}
u = \begin{bmatrix}
u_{11} & u_{21} & u_{31} \\
u_{12} & u_{22} & u_{32} \\
u_{13} & u_{23} & u_{33} \\
\end{bmatrix}
\end{equation*}

as a vector:

\begin{equation*}
u = \begin{bmatrix}
u_{11} \\
u_{21} \\
u_{31} \\
u_{12} \\
u_{22} \\
u_{32} \\
u_{13} \\
u_{23} \\
u_{33} \\
\end{bmatrix}
\end{equation*}

Then we can rewrite our equation $u_{i\; j-1} + u_{i-1\; j} -4 u_{ij} +  u_{i+1\; j} + u_{i\; j+1} = h^2 f_{ij}$ as (for the point $(1,1)$):

\begin{equation*}
u_{10} + u_{01} - 4u_{11} + u_{21}  + u_{12} = h^2 f_{11}
\end{equation*}

where $u_{10}$ and $u_{01}$ correspond to $g_{10}$ and $g_{01}$ respectively.

\begin{equation*}
- 4u_{11} + u_{21}  + u_{12} = h^2 f_{11} - u_{10} - u_{01}
\end{equation*}

For the point $(2,1)$:

\begin{equation*}
u_{20} + u_{11} - 4u_{21} + u_{31}  + u_{22} = h^2 f_{22}
\end{equation*}

rewritten as:
\begin{equation*}
u_{11} - 4u_{21} + u_{31} + u_{22} = h^2 f_{22} - g_{20}
\end{equation*}

We can write a matrix $\mathcal{A}$:

\begin{equation*}
\mathcal{A} = \begin{bmatrix}
-4 &  1 &  0 &  1 &  0 &  0 &  0 &  0 &  0 \\
 1 & -4 &  1 &  0 &  1 &  0 &  0 &  0 &  0 \\
 0 &  1 & -4 &  0 &  0 &  1 &  0 &  0 &  0 \\
 1 &  0 &  0 & -4 &  1 &  0 &  1 &  0 &  0 \\
 0 &  1 &  0 &  1 & -4 &  1 &  0 &  1 &  0 \\
 0 &  0 &  1 &  0 &  1 & -4 &  0 &  0 &  1 \\
 0 &  0 &  0 &  1 &  0 &  0 & -4 &  1 &  0 \\
 0 &  0 &  0 &  0 &  1 &  0 &  1 & -4 &  1 \\
 0 &  0 &  0 &  0 &  0 &  1 &  0 &  1 & -4 \\
\end{bmatrix}
\end{equation*}

and the unknown vector $u$:

\begin{equation*}
u = \begin{bmatrix}
h^2 f_{11} - g_{01} - g_{10} \\
h^2 f_{21} - g_{20}          \\
h^2 f_{31} - g_{14}          \\
h^2 f_{12} - g_{02}          \\
h^2 f_{22}                   \\
h^2 f_{32} - g_{24}          \\
h^2 f_{13} - g_{03} - g_{14} \\
h^2 f_{23} - g_{24}          \\
h^2 f_{33} - g_{34} - g_{43} \\
\end{bmatrix}
\end{equation*}

$\mathcal{A}$ can be rewritten as a $3x3$ blocks matrix with blocks of size $3x3$:
\begin{equation*}
\mathcal{A} = \begin{bmatrix}
  A & I_3 &   0 \\
I_3 &   A & I_3 \\
  0 & I_3 &   A \\
\end{bmatrix}
\end{equation*}

For a general $N$, $\mathcal{A}$ is a $(N - 1) \times (N - 1)$ block matrix:
\begin{equation*}
\mathcal{A} = \begin{bmatrix}
      A & I_{N-1} &        &         &         \\
I_{N-1} &       A & \ddots &         &         \\
        &  \ddots & \ddots &  \ddots &         \\
%
        &         & \ddots &       A & I_{N-1} \\
        &         &        & I_{N-1} &       A \\
\end{bmatrix}
\end{equation*}

% -4  1
%  1 -4

A generic matrix with dimension $n \times n$ is called ``sparse" if the number of non-zero elements is $O(n)$.

The matrix $\mathcal{A}$ is sparse, block \textit{tridiagonal} (it has elements on the diagonal, on the diagonal of the upper matrix and on the diagonal of the lower matrix), symmetric ($\mathcal{A}^T = \mathcal{A}$) and negative definite (all the eigenvalues are negative).

Consider a symmetric (real eigen-values) matrix B $n \times n$, B is negative definite when:

\begin{equation*}
<x, Bx> = x^T B x = \sum_{i=1}^{m} \sum_{j=1}^{n} b_{ij} x_i x_j
\end{equation*}

is $ \le 0 \;\; \forall x: n \times 1$ vectors (negative semi-definite) and\\
$< 0$ for $x \ne 0$.

\subsection{$\mathcal{A}$ is negative definite}
We are going to prove that $\mathcal{A}$ is negative definite, which then means that it is a non-singular matrix, which means that it can be inverted, which means that our system can be solved.

First we prove that:

\begin{equation*}
<v, \mathcal{A}v> \le -2 \; {||v||}^2
\end{equation*}
for all $v \in \mathbb{R}^{N-1}$.

\begin{equation*}
<v, \mathcal{A}v> = \sum_{i=1}^{N-1} \sum_{j=1}^{N-1} a_{ij} v_i v_j =
\end{equation*}
\begin{equation*}
-4 v_{1}^{2} + v_1 v_2 + v_2 v_1 -4 v_2^2 + v_2 v_3 + v_3 v_2 - 4 v_3^2 + \dots -4 v_{N-2}^2 + v_{N-2} v_{N-1} + v_{N-1} v_{N-2} -4 v_{N-1}^2 =
\end{equation*}
\begin{equation*}
-4 v_1^2 -4 v_2^2 -4 v_3^2 + \dots -4 v_{N-2}^2 - 4 v_{N-1}^2 + 2 v_1 v_2 + 2 v_2 v_3 + \dots + 2 v_{N-2} v_{N-1} =
\end{equation*}
\begin{equation*}
-3 v_1^2 - 2 v_2^2 - 2 v_3^2 + \dots - 2 v_{N-2}^2 - 3 v_{N-1}^2 - (v_1^2 - 2 v_1 v_2 + v_2^2) - (v_2^2 - 2 v_2 v_3 + v_3^2) + \dots + ( v_{N-2}^2 - 2 v_{N-2} v_{N-1} + v_{N-1}^2) =
\end{equation*}
\begin{equation*}
-3 v_1^2 - 2 v_2^2 - 2 v_3^2 + \dots - 2 v_{N-2}^2 - 3 v_{N-1}^2 - (v_1 - v_2)^2 - (v_2 - v_3)^2 + \dots - (v_{N-2} - v_{N-1})^2 \le
\end{equation*}
\begin{equation*}
-2 v_1^2 - 2 v_2^2 - 2 v_3^2 + \dots - 2 v_{N-2}^2 - 2 v_{N-1}^2 = -2 \underbrace{(v_1^2 + v_2^2 + v_3^2 + \dots + v_{N-2}^2 + v_{N-1}^2)} %{= || v ||_2^2}
\end{equation*}

Consider $v = (v_1, \dots, v_{N-1}) \in \mathbb{R}^{(N-1)^2}$, where $v_1, \dots, v_{N-1} \in \mathbb{R}^{N-1}$.

\begin{equation*}
<v, \mathcal{A}v> = v^T \mathcal{A} v = [v_1^T, v_2^T, \dots, v_{N-1}^T]
\end{equation*}

\begin{equation*}
<x, Bx> = \sum_{i=1}^{m} \sum_{j=1}^{n} b_{ij} x_i x_j = \sum_{i=1}^{m} \sum_{j=1}^{n} <v_i, \mathcal{A}_ij v_j> =
\end{equation*}
\begin{equation*}
<v_1, A v_1> + <v_1, I v_2> + <v_2, I v_1> + <v_2, A v_2> + <v_2, I v_3> + <v_3, I v_2> + <v_3, A v_3> + \dots +
\end{equation*}
\begin{equation*}
+ <v_{N-2}, A v_{N-2}> + <v_{N-2}, v_{N-1}> + <v_{N-1}. v_{N-2}> + <v_{N-1}. A v_{N - 1}> =
\end{equation*}
\begin{equation*}
\overbrace{<v_1, A v_1>}^{\le -2 ||v_1||_2^2} + \overbrace{<v_1, I v_2>}^{\le -2 || v_2 ||_2^2} + \overbrace{<v_2, I v_1>}^{\le -2 || v_3 ||_2^2} + \dots + \overbrace{<v_{N-2}, A v_{N-2}>}^{\le -2 || v_{N-2} ||_2^2} + \overbrace{<v_{N-1}, A v_{N - 1}>}^{\le -2 || v_{N-1} ||_2^2} + 2 \dots\dots \le
\end{equation*}
\begin{equation*}
-2 ||v_1||_2^2 -2 ||v_2||_2^2 +\dots -2 ||v_{N-1}||_2^2 \dots\dots =
\end{equation*}
\begin{equation*}
= - ||v_1||_2^2 - (||v_1||_2^2 -2 <v_1, v_2> + ||v_2||_2^2) - (||v_2||_2^2 -2 <v_2, v_3> + ||v_3||_2^2) + \dots - (||v_{N-2}||_2^2 -2 <v_{N-2}, v_{N-1}> + ||v_{N-1}||_2^2) - ||v_{N-1}||_2^2 =
\end{equation*}
\begin{equation*}
= - ||v_1||_2^2 - ||v_1 - v_2||_2^2 - ||v_2 - v_3||_2^2 + \dots - ||v_{N-2} - v_{N-1}||_2^2 - ||v_{N-1}||_2^2 \le 0
\end{equation*}

Now we show that $<v, \mathcal{A}v> < 0$ for  $v \ne 0$.

Equivalently, we show that $<v, \mathcal{A}v> = 0 \implies v = 0$.
When $<v, \mathcal{A}v> = 0$, also

\begin{equation*}
- ||v_1||_2^2 - ||v_1 - v_2||_2^2 - ||v_2 - v_3||_2^2 + \dots - ||v_{N-2} - v_{N-1}||_2^2 = 0
\end{equation*}

\dots

\begin{equation*}
||v_1||_2 = ||v_1 - v_2||_2 = \dots = ||v_{N-2} - v_{N-1}||_2 = ||v_{N-1}||_2 = 0
\end{equation*}
and so:
\begin{equation*}
v_1 = 0, v_1 = v_2, \dots, v_{N-1} = 0
\end{equation*}

and so:
\begin{equation*}
v_1 = v_2 = \dots = v_{N-1} = 0
\end{equation*}
and so:
\begin{equation*}
v = 0
\end{equation*}

\subsubsection*{Exercise}

Write the 1D Poisson problem with Dirichlet Boundary condition on $\Omega = (0, 1)$.

By definition, the Laplacian is: $\frac{\partial^2}{\partial x^2} (x,y) + \frac{\partial^2 u}{\partial y^2} (x,y)$.

The 1D version of the Poisson problem is: $u''(x) = f(x), \quad x \in \Omega = (0,1)$
$u(x) = g(x), \quad x \in \Gamma$ : $u(0) = g(0), u(1) = g(1)$.

Solve analytically this problem:

$u'(x) = u'(0) + \int_{0}^{x} \underbrace{u''(s)}_{f(s)} ds$, \quad $x \in [0, 1]$.

\begin{equation*}
u(x) = u(0) + \int_{0}^{x} u'(s) ds = u(0) + \int_{0}^{x} (u'(0) + (\int_{0}^{s} f(\delta) d\delta) \;\, ds =
\end{equation*}
\begin{equation*}
u(0) + u'(0) \int_{0}^{x} ds + \int_{0}^{x} \int_{0}^{s} f(\delta) d\delta ds =
\end{equation*}

\begin{equation*}
u(x) = u(0) + u'(0) x + \int_{0}^{x} \int_{0}^{s} f(\delta) d\delta ds \quad x \in [0, 1]
\end{equation*}

$u(0) = g(0)$
$u(1) = u(0) + u'(0) + \int_{0}^{1} \int_{0}^{s} f(\delta) d\delta ds \quad x \in [0, 1]$

\dots\dots

Propose a corresponding discrete problem and write the associated linear system:

$\Omega = (0, 1)$, $h = 1/N$ with $N$ positive integer.
$\Omega_h = {h, 2h, \dots, (N-1) h}$

The discrete problem is:

\begin{equation*}
\Delta_h u_h (x) = f(x), \quad x \in \Omega_h
u_h (0) = g(0)
u_h (1) = g(1)
\end{equation*}

$\Delta_h$ associates to a function $\overline{\Omega_h} \rightarrow \mathbb{R}$ a function $\Omega_h \rightarrow \mathbb{R}$ that approximates $\Delta u = u''$.

For a function $v_h: \overline{\Omega_h} \rightarrow \mathbb{R}$, we define:
\begin{equation*}
\Delta_h v_h(x) = \frac{v_h(x-h) - 2 v_h(x) + v_h(x+h)}{h^2}, \quad x \in \Omega_h
\end{equation*}

We can rewrite the discrete problem as:
\begin{equation*}
u_i = u_h(ih), \quad i \in {1, \dots, N-1}
\end{equation*}
\begin{equation*}
f_i = f(ih)
\end{equation*}
\begin{equation*}
g_0 = g(0), \; g_N = g(1)
\end{equation*}

\begin{equation*}
\frac{u_{i-1} - 2 u_i + u_i+1}{h^2} = f_i, \quad i \in {1, \dots, N-1}
\end{equation*}

with $u_0 = g_0, \quad u_N = g_N$,

\begin{equation*}
u_{i-1} - 2 u_i + u_i+1 = h^2 f_i, \quad i \in {1, \dots, N-1}
\end{equation*}

with $u_0 = g_0, \quad u_N = g_N$.

The linear system is:

We can write a matrix:

\begin{equation*}
\begin{bmatrix}
-2 &  1 &    &    &    &    &    \\
 1 & -2 &  1 &    &    &    &    \\
   &  1 & \ddots &  \ddots &    &    &    \\
 & & \ddots &   &   & & \\

   &    &    &    &  1 & -2 &  1 \\
   &    &    &    &    &  1 & -2 \\
\end{bmatrix}
\end{equation*}

and the unknown vector is:

\begin{equation*}
u = \begin{bmatrix}
h^2 f_{1} - g_{0}   \\
h^2 f_{2}           \\
h^2 f_{3}           \\
\dots               \\
h^2 f_{N-2}         \\
h^2 f_{N-1} - g_{N} \\
\end{bmatrix}
\end{equation*}

For $v \in \mathbb{R}^{N-1}$: $<v, \mathcal{A}v$ = $-2 v_1^2 + v_1 v_2 + v_2 v_1 - 2 v_2^2 + \dots$\\
$. \qquad \quad \qquad \qquad \qquad \qquad \qquad \qquad  + v_{N-2} v_{N-1} + v_{N-1} v_{N-2} + 2 v_N^2 =$

\begin{equation*}
- v_1^2 - (v_1 - v_2)^2 - (v_2 - v_3)^2 + \dots - (v_{N-2} - v_{N-1})^2 - v_{N-1}^2 \le 0
\end{equation*}

If $<v, \mathcal{A}v> = - v_1^2 - (v_1 - v_2)^2 + \dots - (v_{N-2} - v_{N-1})^2 - v_{N-1}^2 = 0$ then $v_1 = 0, v_1 - v_2 = 0, \dots, v_{N-2} - v_{N-1} = 0, v_{N-1} = 0$ and so $v = 0$.

\subsubsection*{Exercise}

Consider the 3D Poisson problem with Dirichlet boundary conditions on $\Omega = (0, 1)^3$.
Propose a discrete Laplacian and a consequent discrete problem.

\begin{equation*}
\Delta u(x, y, z) = f(x, y, z), \quad (x, y, z) \in \Omega = (0, 1)^3
\end{equation*}

\begin{equation*}
u(x, y, z) = g(x, y, z), \quad (x, y, z) \in \Gamma
\end{equation*}

\begin{equation*}
\Delta u(x, y, z) = \frac{\partial^2 u}{\partial x^2} (x, y, z) + \frac{\partial^2 u}{\partial y^2} (x, y, z) + \frac{\partial^2 u}{\partial z^2} (x, y, z)
\end{equation*}

\begin{equation*}
\Omega_h = \Omega \cap \mathbb{R}^3_h
\end{equation*}

\begin{equation*}
\mathbb{R}^3_h = {(mh, uh, ph) : m,n,p \in \mathbb{Z}}
\end{equation*}

$\Gamma_h$ in 2D was the border without the corners. In 3D is the set of faces, without the edges and the corners.
$\Gamma_h = $ "points of $\mathbb{R}^3_h \cap \Gamma$ with a neighbor in $\Omega_h$"

The discrete problem then is

\begin{equation*}
\Delta_h u(x, y, z) = f(x, y, z), \quad (x, y, z) \in \Omega_h
\end{equation*}

\begin{equation*}
u_h(x, y, z) = g(x, y, z), \quad (x, y, z) \in \Gamma_h
\end{equation*}

The discrete Laplacian is given by a mesh function $v: \overline{\Omega_h} = \Omega_h \cup \Gamma_h \rightarrow \mathbb{R}$

\begin{equation*}
\Delta_h v_h(x, y, z) = \frac{v(x-h, y, z) - 2v(x,y,z) + v(x+h, y, z)}{h^2} +\\
\frac{v(x, y-h, z) - 2v(x,y,z) + v(x, y+h, z)}{h^2} + \frac{v(x, y, z-h) - 2v(x,y,z) + v(x, y, z+h)}{h^2} \quad (x, y, z) \in \Omega_h
\end{equation*}

Describe in the matrix of the linear system associated to the discrete problem the row corresponding to a mesh point in $\Omega_h$ with nearest neighbors in $\Omega_h$.

In order to write our problem in this way: $\mathcal{A} U = b$ we need to order the unknowns.
There are $(N-1)^3$ unknowns, where $h = \frac{1}{N}$; therefore $\mathcal{A}$ is a $(N-1)^3 \times (N-1)^3$ matrix.

What is the equation of a point whose neighbors are part of $\Omega_h$?

We construct the matrix from the equation $\Delta_h u_h(x, y, z) = f(x, y, z)$; which becomes $h^2 \Delta_h u_h(x, y, z) = h^2 f(x, y, z)$

%\begin{equation*}
$
-6 \overbrace{u(x, y, z)}^{= u_{ijk} = U_l} + \overbrace{u_h(x-h, y, z)}^{= u_{l-1}} + \overbrace{u_h(x+h, y, z)}^{= u_{l+1}} +\\
.\qquad\qquad\qquad \; \; + \overbrace{u_h(x, y-h, z)}^{= u_{l-(N-1)}} + \overbrace{u_h(x, y+h, z)}^{= u_{l+(N-1)}} +\\
.\qquad\qquad\qquad \; \; + \overbrace{u_h(x, y, z-h)}^{= u_{l- (N-1)^2}} + \overbrace{u_h(x, y, z+h)}^{= u_{l+ (N-1)^2}} =
h^2 f(x, y, z)
$
%\end{equation*}

\begin{equation*}
%U_{l-(N-1)^2} + U_{l-(N-1)^} + U_{l-1} - 6 U_{l} + U_{l+1} + U_{l+(N-1)} + U_{l+(N-1)^2} = h^2 f_l
\end{equation*}

\dots

\section{Convergence analysis of the discrete Poisson problem (with D. b.c)}

We assume that the continuous problem has a unique solution $u$.

We want to prove that the discrete problem has a unique solution $u_h$ (already done, but we obtain this by using another technique) and we want to give a bound on the error:

\begin{equation*}
\max_{(x,y) \in \overline{\Omega}_h} |u_h (x, y) - u(x, y)|
\end{equation*}

This bound will prove the convergence of $\mu_h$ to $u$ as $h \rightarrow 0$.

\subsection{Theorem: Discrete maximum principle}

Let $v_h: \overline{\Omega}_h \rightarrow \mathbb{R}$ such that:

\begin{equation*}
\Delta_h v_h \ge 0 \qquad \forall (x, y) \in \Omega_h
\end{equation*}

Then:
\begin{equation*}
\max_{(x,y) \in \Omega_h} v_h (x, y) \le \max_{(x,y) \in \Gamma_h} v_h (x, y)
\end{equation*}

with equality iff $v_h$ is constant.

\subsubsection{Continuous maximum principle}

Let $v: \overline{\Omega} \rightarrow \mathbb{R}$ of class $C^2$ such that:

\begin{equation*}
\Delta v (x,y) \ge 0 \qquad \forall (x, y) \in \Omega
\end{equation*}

Then:
\begin{equation*}
\sup_{(x,y) \in \Omega} v(x, y) \le \max_{(x,y) \in \Gamma} v(x, y)
\end{equation*}

\subsubsection{Proof of the Discrete maximum principle theorem}

We have to prove that:
\begin{equation*}
\max_{(x,y) \in \Omega_h} v_h(x, y) \le \max_{(x,y) \in \Gamma_h} v_h(x, y)
\end{equation*}

with $=$ iff $v_h$ is constant.

We begin by observing if $v_h$ is constant, then:
\begin{equation*}
\max_{(x,y) \in \Omega_h} v_h(x, y) = \max_{(x,y) \in \Gamma_h} v_h(x, y)
\end{equation*}

Given two logical sentences $A$ and $B$, $A \implies B$ is equivalent to $\neg B \implies \neg A$

When $v_h$ is not constant ($A$), then ($B$):
\begin{equation*}
\max_{(x,y) \in \Omega_h} v_h(x, y) < \max_{(x,y) \in \Gamma_h} v_h(x, y)
\end{equation*}

Therefore we prove the counter-opposite ($\neg B$) of the previous equation.
\begin{equation*}
\max_{(x,y) \in \Omega_h} v_h(x, y) > \max_{(x,y) \in \Gamma_h} v_h(x, y)
\end{equation*}
which means that $v_h$ is not constant ($\neg A$).

Suppose:
\begin{equation*}
\max_{(x,y) \in \Omega_h} v_h(x, y) \ge \max_{(x,y) \in \Gamma_h} v_h(x, y)
\end{equation*}

So:
\begin{equation*}
\max_{(x,y) \in \Omega_h} v_h(x, y) = \max_{(x,y) \in \Gamma_h} v_h(x, y)
\end{equation*}

Consider now a point $(x_0, y_0) \in \Omega_h$ where $v_h$ is the maximum.

We prove that all $(x_0 - h, y_0), (x_0 + h, y_0), (x_0, y_0 - h), (x_0, y_0 + h)$ are maximum points for $v_h$.

We have:
\begin{equation*}
h^2 \Delta_h v_h(x_0, y_0) \ge 0
\end{equation*}
\begin{equation*}
= v_h(x_0, y_0 - h) + v_h(x_0 - h, y_0) - 4 v_h(x_0, y_0) + v_h(x_0 + h, y_0) + v_h(x_0, y_0 + h)
\end{equation*}

and then:
\begin{equation*}
4 v_h(x_0, y_0) \le \overbrace{v_h(x_0, y_0 - h)}^{v_h (x_0, y_0)} + \overbrace{v_h(x_0 - h, y_0)}^{v_h (x_0, y_0)} + \overbrace{v_h(x_0 + h, y_0)}^{v_h (x_0, y_0)} + \overbrace{v_h(x_0, y_0 + h)}^{v_h (x_0, y_0)}
\end{equation*}
which means that the second member must be $\le 4 v_h (x_0, y_0)$.

Thus:
\begin{equation*}
4 v_h(x_0, y_0) = v_h(x_0, y_0 - h) + v_h(x_0 - h, y_0) + v_h(x_0 + h, y_0) + v_h(x_0, y_0 + h)
\end{equation*}

and it follows that:
\begin{equation*}
v_h(x_0, y_0 - h) = v_h(x_0 - h, y_0) = v_h(x_0 + h, y_0) = v_h(x_0, y_0 + h) = v_h(x_0, y_0)
\end{equation*}

We have proved that $(x_0 - h, y_0), (x_0 + h, y_0), (x_0, y_0 - h), (x_0, y_0 + h)$ are maximum points for $v_h$; this argument can be repeated recursively for all the neighbors, which means that they all have value $v_{max}$; we obtain that all the mesh points obtain the maximum value $v_{max}$, and $v_h$ is constant. $\blacksquare$

\subsection{Corollary of the Discrete maximum principle}

Let $v_h: \overline{\Omega}_h \rightarrow \mathbb{R}$ such that:

\begin{equation*}
\Delta_h v_h (x,y) \le 0 \qquad \forall (x, y) \in \Omega_h
\end{equation*}

\begin{equation*}
\min_{(x,y) \in \Omega_h} v_h(x, y) \ge \min_{(x,y) \in \Gamma_h} v_h(x, y)
\end{equation*}

\subsubsection{Proof}
Consider $w_h = - v_h$. We have:

\begin{equation*}
\Delta_h w_h (x, y) = - \Delta_h v_h (x, y) \ge 0 \forall (x, y) \in \Omega_h
\end{equation*}

From the discrete maximum principle, we have:
\begin{equation*}
\max_{(x, y) \in \Omega_h} \overbrace{w_h (x, y)}^{= - v_h(x, y)} \le \max_{(x, y) \in \Gamma_h} \overbrace{w_h (x, y)}^{= - v_h(x, y)}
\end{equation*}
$\blacksquare$

\subsection{Theorem:}

The discrete Poisson problem (with D. b.c.) has a unique solution.

\subsubsection{Proof:}

The discrete problem is written as a linear system: $\mathcal{A} U = b$.

Existence and uniqueness of the solution of the discrete problem follows by the non singularity of the matrix $\mathcal{A}$.

$\mathcal{A}$ is non singular if:

\begin{equation*}
\mathcal{A} U = 0 \implies U = 0
\end{equation*}

We are now going to prove this implication.

$\mathcal{A} U = 0$ is obtained for the discrete problem:

\begin{equation*}
\Delta_h u_h(x, y) = 0, \quad (x, y) \in \Omega_h
\end{equation*}

\begin{equation*}
u_h(x, y) = 0, \quad (x, y) \in \Gamma_h
\end{equation*}

Now we show that $u_h = 0$ (i.e. $U = 0$).

We have:

\begin{equation*}
\Delta_h u_h(x, y) \ge 0, \quad (x, y) \in \Omega_h
\end{equation*}

and from the discrete max principle:

\begin{equation*}
\max_{(x, y) \in \Omega_h} u_h(x, y) \le \max_{(x, y) \in \Gamma_h} u_h(x, y) = 0
\end{equation*}

Moreover, we have:
\begin{equation*}
\Delta_h u_h(x, y) \le 0, \quad (x, y) \in \Omega_h
\end{equation*}

and then from the discrete min principle:

\begin{equation*}
\min_{(x, y) \in \Omega_h} u_h(x, y) \ge \min_{(x, y) \in \Gamma_h} u_h(x, y) = 0
\end{equation*}

For any $(x, y) \in \Omega_h$ we have: $0 \le u_h(x, y) \le 0$, and so: $u_h(x, y) = 0$. $\blacksquare$

\section{$\infty$-norm and $L^\infty$ norm}

Given a function $f: \mathcal{A} \rightarrow \mathbb{R}$, we set:

\begin{equation*}
||f||_{L^\infty(\mathcal{A})} := \sup_{x \in \mathcal{A}} |f(x)|
\end{equation*}

When A is finite or $\mathcal{A} sub \mathbb{R}^d$ closed and bounded (i.e. compact) then:

\begin{equation*}
||f||_{L^\infty(\mathcal{A})} := \max_{x \in \mathcal{A}} |f(x)|
\end{equation*}

$L^\infty(\mathcal{A})$ is the set of functions $f: \mathcal{A} \rightarrow \mathbb{R}$ such that $||f||_{L^\infty(\mathcal{A})} < +\infty$.

We have, for $v: \overline{\Omega} \rightarrow \mathbb{R}$ of class $C^4$:

\begin{equation*}
\max_{(x, y) \in \Omega_h} {|\Delta_h v|}_{\overline{\Omega_h}} (x, y) - \Delta v(x, y)| \le \frac{h^2}{6} \max\{\max_{(x, y) \in \overline{\Omega}} |\frac{\partial^4 v}{\partial x^4} (x, y)|, \max_{(x, y) \in \overline{\Omega}} |\frac{\partial^4 v}{\partial y^4} (x, y)|\}
\end{equation*}

\begin{equation*}
||\Delta_h v{|}_{\overline{\Omega_h}} (x, y) - \Delta v(x, y)||_{L^\infty(\Omega_h)} \le \frac{h^2}{6} \max\{ ||\frac{\partial^4 v}{\partial x^4}||_{L^\infty(\overline{\Omega_h})}, ||\frac{\partial^4 v}{\partial y^4}||_{L^\infty(\overline{\Omega_h})} \}
\end{equation*}

The stability result says that the map: $(f, g) \rightarrowtail u_h$ (underbraces: data for the discrete problem ... solution of the discrete problem) is uniformly bounded with respect to h, i.e.:

\begin{equation*}
||u_h||_{L^\infty(\overline{\Omega_h})} \le C \cdot \max \{ ||f||_{L^\infty(\Omega)}, ||g||_{L^\infty(\Gamma)} \}
\end{equation*}

with a constant C independent of $f$, $g$ and $h$.

\subsection{Theorem (stability problem)}

Let $f \in L^\infty(\Omega)$ and let $g \in L^\infty(\Gamma)$ and let $u_h$ be the solution of the discrete problem:

\begin{equation*}
\Delta_h u_h = f(x, y), \qquad (x, y) \in \Omega_f
\end{equation*}

\begin{equation*}
u(x, y) = g(x, y) \qquad (x, y) \in \Gamma_h
\end{equation*}

Then we have:

\begin{equation*}
{||u_h||}_{L^\infty(\overline{\Omega})} \le \frac{1}{8} \overbrace{{||f||}_{L^\infty(\Omega)}}^{\le max} + \overbrace{{||g|}_{L^\infty(\Gamma)}}^{\le max}
\end{equation*}

\subsubsection{Proof:}
We introduce the mesh function:

\begin{equation*}
\Phi(x, y) = \frac{(x - \frac{1}{2})^2 + (y - \frac{1}{2})^2}{4} = \frac{d^2}{4}, \quad (x, y) \in \overline{\Omega_h}
\end{equation*}

We have:

\begin{equation*}
\Delta_h \Phi(x, y) = 1, \quad (x, y) \in \Omega_h
\end{equation*}

\begin{equation*}
\Delta_h \Phi(x, y) = \frac{\Phi(x, y-h) + \Phi(x - h, y) - 4 \Phi(x, y) + \Phi(x + h, y) + \Phi(x, y + h)}{h^2}
\end{equation*}

\begin{equation*}
\Phi(x, y - h) = \frac{(x - \frac{1}{2})^2 + (y - h - \frac{1}{2})^2}{4} = \frac{(x - \frac{1}{2})^2 + (y - \frac{1}{2})^2 - 2(y - \frac{1}{2})h + h^2}{4}
\end{equation*}

\begin{equation*}
\Phi(x - h, y) = \frac{(x - \frac{1}{2})^2 + (y - \frac{1}{2})^2 - 2(x - \frac{1}{2})h + h^2}{4}
\end{equation*}

\begin{equation*}
-4 \Phi(x, y) = \frac{-4(x - \frac{1}{2})^2 - 4(y - \frac{1}{2})^2}{4}
\end{equation*}

\begin{equation*}
\Phi(x, y + h) = \frac{(x - \frac{1}{2})^2 + (y - \frac{1}{2})^2 + 2(x - \frac{1}{2})h + h^2}{4}
\end{equation*}

\begin{equation*}
\Phi(x + h, y) = \frac{(x - \frac{1}{2})^2 + (y - \frac{1}{2})^2 + 2(y - \frac{1}{2})h + h^2}{4}
\end{equation*}

which means
\begin{equation*}
\Delta_h \Phi(x, y) = \frac{\frac{4 h^2}{4}}{h^2} = 1
\end{equation*}

We have:

\begin{equation*}
0 \le \Phi(x,y) \le \frac{1}{8}, \quad (x, y) \in \overline{\Omega_h}
\end{equation*}

\begin{equation*}
\Phi(x, y) = \frac{d^2}{4} \le \frac{l^2}{4} = \frac{(\frac{\sqrt{2}}{2})^2}{4} = \frac{(\frac{1}{\sqrt{2}})^2}{4}
\end{equation*}

where $l$ is the diagonal from the center of the square to a corner:

We prove that $\Delta_h$ is a linear operator, i.e. for $v_h, w_h: \overline{\Omega_h} \rightarrow \mathbb{R}$ we have:

\begin{equation*}
\Delta_h(v_h + w_h) = \Delta_h v_h + \Delta_h w_h
\end{equation*}

for $v_h: \overline{\Omega_h} \rightarrow \mathbb{R}$ and $\alpha \in \mathbb{R}$ we have:

\begin{equation*}
\Delta_h \dots
\end{equation*}

For $(x, y) \in \Omega_h$:

\begin{equation*}
\Delta_h(v_h + w_h)(x, y) = ( v_h(x, y-h) + w_h(x, y-h) + v_h(x-h, y) + w_h(x-h, y) - 4 v_h(x,y) - 4 w_h(x, y) + v_h(x, y+h) + w_h(x, y+h) + v_h(x+h, y) + w_h(x+h, y) ) \cdot \frac{1}{h^2} =
\end{equation*}

\begin{equation*}
= ( v_h(x, y-h)+ v_h(x-h, y) - 4 v_h(x,y) + v_h(x, y+h) + v_h(x+h, y)) \cdot \frac{1}{h^2} + ( w_h(x, y-h) + w_h(x-h, y) - 4 w_h(x, y) + w_h(x, y+h) + w_h(x+h, y) ) \cdot \frac{1}{h^2} =
\end{equation*}

\begin{equation*}
= \Delta_h v_h(x, y) + \Delta_h w_h(x, y)
\end{equation*}

\begin{equation*}
\Delta_h(\alpha v_h)(x,y) = ( \alpha v_h(x, y-h) + \alpha v_h(x-h, y) - \alpha 4 v_h(x,y) + \alpha v_h(x, y+h) + \alpha v_h(x+h, y)) \cdot \frac{1}{h^2} = \alpha (v_h(x, y)) \dots
\end{equation*}

Set:
\begin{equation*}
A := {||f||}_{L^\infty(\Omega)}, B := {||g||}_{L^\infty(\Gamma)}
\end{equation*}

For $(x, y) \in \Omega_h$:
\begin{equation*}
\Delta_h (u_h + A\Phi) (x, y) = \Delta_h u_h(x, y) + A \underbrace{\Delta_h \Phi(x, y)}_{=1} = \Delta_h u_h(x, y) + A = f(x, y) + A \ge - |f(x, y)| + \underbrace{A}_{= \sup_{(x, y) \in \Omega} |f(x, y)} \ge 0
\end{equation*}

\begin{equation*}
\Delta_h (u_h - A\Phi) (x, y) = \Delta_h u_h(x, y) - A \underbrace{\Delta_h \Phi(x, y)}_{=1} = \Delta_h u_h(x, y) - A = f(x, y) - A \le |f(x, y)| - \underbrace{A}_{= \sup_{(x, y) \in \Omega} |f(x, y)} \le 0
\end{equation*}

The discrete maximum principle says that:
\begin{equation*}
\max_{(x, y) \in \Omega_h} (u_h + A \Phi) (x, y) \le \max_{(x, y) \in \Gamma_h} (u_h + A \Phi) (x, y)
\end{equation*}

\begin{equation*}
\min_{(x, y) \in \Omega_h} (u_h - A \Phi) (x, y) \ge \max_{(x, y) \in \Gamma_h} (u_h - A \Phi) (x, y)
\end{equation*}

\begin{equation*}
\max_{(x, y) \in \Omega_h} u_h(x, y) \le \max_{(x, y) \in \Omega_h} (\underbrace{u_h + \overbrace{A \Phi(x, y)}^{\ge 0}}_{= (u_h + A \Phi) (x, y)}) \le
\end{equation*}

\begin{equation*}
\le \max_{(x, y) \in \Gamma_h} (\underbrace{u_h + A \Phi(x, y)}_{= (u_h + A \Phi) (x, y)}) = \max_{(x, y) \in \Gamma_h} (g(x, y) + A \underbrace{\Phi(x,y)}_{\le \frac{1}{8}}) \le \max_{(x, y) \in \Gamma_h}(g(x, y) - \frac{1}{8}A) = \max_{(x, y) \in \Gamma_h} \underbrace{g(x, y)}_{\ge -|g(x, y)|} + \frac{1}{8}A \le - \max_{(x, y) \in \Gamma_h} |g(x, y)| - \frac{1}{8}A \le - \dots
\end{equation*}

The opposite with the $\min$.

We have proved, for $(x, y) \in \overline{\Omega_h}$:

\begin{equation*}
-\frac{1}{8}A - B \le u_h(x, y) \le \frac{1}{8}A + B
\end{equation*}

This is exactly the same of:

\begin{equation*}
|u_h(x, y)| \le \frac{1}{8}A + B
\end{equation*}

In other words:
\begin{equation*}
||u_h||_{L^\infty()} \dots
\end{equation*}

\begin{equation*}
|| \Delta_h v|_{\Omega_h} - (\Delta_h v)|_{\Omega_h} ||_{L^\infty(\Omega_h)}
\end{equation*}

\begin{equation*}
|| u_h - u|_{\overline{\Omega_h}}  ||
\end{equation*}

\subsection{Theorem (the convergence theorem)}

Let $u$ be the solution of:

\begin{equation*}
\Delta u(x, y) = f(x, y) \quad (x, y) \in \Omega
\end{equation*}

\begin{equation*}
u(x, y) = g(x, y) \quad (x, y) \in \Gamma
\end{equation*}

and let $u_h$ be the solution of:
\begin{equation*}
\Delta u_h(x, y) = f(x, y) \quad (x, y) \in \Omega_h
\end{equation*}

\begin{equation*}
u_h(x, y) = g(x, y) \quad (x, y) \in \Gamma_h
\end{equation*}

We have:

\begin{equation*}
|| u_h - u|_{\Omega_h} ||_{L^\infty(\overline{\Omega_h})} \le \frac{1}{8} ||\Delta_h u|_{\Omega_h} - (\Delta u)|_{\Omega_h} ||_{L^\infty(\Omega_h)}
\end{equation*}

\subsubsection{Proof:}

We set:
\begin{equation*}
l_h := u_h - u|_{\overline{\Omega_h}}
\end{equation*}

We have:
\begin{equation*}
\Delta_h l_h = \Delta_h (u_h - u|_{\overline{\Omega_h}}) = \Delta_h u_h - \Delta_h u|_{\overline{\Omega_h}} = \Delta_h u_h - (\Delta u)|_{\Omega_h} + (\Delta u)|_{\Omega_h} - \Delta_h u|_{\Omega_h}
\end{equation*}

We have, for $(x, y) \in \Omega_h$
\begin{equation*}
\Delta_h u_h(x, y) - \Delta u (x, y) = g(x, y) - f(x, y) = 0
\end{equation*}

We have:
\begin{equation*}
\Delta_h l_h = - \epsilon_h
\end{equation*}

where
\begin{equation*}
\epsilon_h = \Delta_h u|_{\Omega_h} - (\Delta u)|_{\Omega_h}
\end{equation*}

For $(x, y) \in \Gamma_h$:
\begin{equation*}
l_h (x, y) = u_h (x, y) - u(x, y) = \dots
\end{equation*}

By concluding, we can say that the important error $l_h$ satisfies the discrete problem:
\begin{equation*}
\Delta_h \epsilon_h (x, y) = - \epsilon_h(x, y) \qquad (x, y) \in \Omega_h
\end{equation*}

\begin{equation*}
l_h(x, y) = 0 \qquad (x, y) \in \Gamma_h
\end{equation*}

By means of the previous theorem:
\begin{equation*}
||l_h||_{L^\infty(\overline{\Omega_h})} \le \frac{1}{8} \underbrace{|| -\epsilon_h||_{L^\infty(\Omega_h)}}_{= || \epsilon_h ||_{L^\infty(\Omega_h)} } + \underbrace{||o||_{L^\infty(\Gamma_h)}}_{= 0} \le \frac{1}{8} ||\epsilon_h||_{L^\infty(\Omega_h)} + \dots
\end{equation*}
$\blacksquare$

We know that:
\begin{equation*}
|| \Delta_h u|_{\Omega_h} - (\Delta u)|_{\Omega_h} ||_{L^\infty(\Omega_h)} \le \frac{h^2}{6} \max \{ ||\frac{\partial^4 u}{\partial x^4}||_{L^\infty(\overline{\Omega_h})}, ||\frac{\partial^4 v}{\partial y^4}||_{L^\infty(\overline{\Omega_h})} \}
\end{equation*}

and then:
\begin{equation*}
|| u_h - u|_{\overline{\Omega_h}} ||_{L^\infty(\overline{\Omega_h})} \le \frac{h^2}{48} \max \{ ||\frac{\partial^4 u}{\partial x^4}||_{L^\infty(\overline{\Omega_h})}, ||\frac{\partial^4 v}{\partial y^4}||_{L^\infty(\overline{\Omega_h})} \}
\end{equation*}
when $u \in C^4$.

\subsection{Some final remarks}

Consider:
\begin{itemize}
	\item $u$ is the solution of the continuous problem;
	\item $u_h$ is the solution of the discrete problem;
	\item $\Delta_h$ is a discrete Laplacian, not necessarily the five-point discretization.
\end{itemize}

We define:
\begin{equation*}
l_h := u_h - u|_{\overline{\Omega_h}}
\end{equation*}

$l_h$ is called the \textit{convergence error}:
\begin{equation*}
\epsilon_h := \Delta_h u|_{\Omega_h} - (\Delta u)|_{\Omega_h}
\end{equation*}

$\epsilon_h$ is called the \textit{consistency error}.

We are interested on $l_h$, since $\epsilon_h$ is a "surrogate" of $l_h$.\\

We say that, given $p$ a positive integer:

\begin{itemize}
	\item the discretization is \textit{consistent of order $p$} if:
		\begin{equation*}
		|| \epsilon_h ||_{L^\infty(\Omega_h)} = O(h^p) \qquad N \rightarrow \infty
		\end{equation*}
	\item the discretization is \textit{convergent of order $p$} if
		\begin{equation*}
		|| l_h ||_{L^\infty(\overline{\Omega_h})} = O(h^p) \qquad N \rightarrow \infty
		\end{equation*}

	\item the discretization is \textit{stable} if the solution $v_h$ of the discrete problem:
		\begin{equation*}
		\Delta_h v_h(x, y) = l(x,y), \qquad (x, y) \in \Omega_h
		\end{equation*}
		\begin{equation*}
		v_h(x, y) = 0
		\end{equation*}

		we have:
		\begin{equation*}
		|| v_h ||_{L^\infty(\overline{\Omega_h})} \le C || l ||_{L^\infty(\Omega)}
		\end{equation*}

		where $C$ is independent of $l$ and $h$.
\end{itemize}

We have: consistency of order $p$ and stability $\implies$ convergence of order $p$.

We have:
\begin{equation*}
\Delta_h \l_h = \epsilon_h in \Omega_h
\end{equation*}

\begin{equation*}
l_h = 0 in \Gamma_h
\end{equation*}

If $||\epsilon_h||_{L^\infty(\Omega_h)} = O(h^p)$ (consistency) and $||l_h||_{L^\infty(\overline{\Omega_h})} \le C || \epsilon_h ||_{L^\infty(\Omega_h)}$ then $||L_h||_{L^\infty(\overline{\Omega_h})} = O(h^p)$ (convergence).


\subsection{Typing a Big Formula}

How to type a big formula step-by-step. By the end of this example you will have typed the formula:

\[
    \sum_{i = 1}^{ \left[ \frac{n}{2} \right] }
    \binom{ x_{i, i + 1}^{i^{2}} }
        { \left[ \frac{i + 3}{3} \right] }
    \frac{ \sqrt{ \mu(i)^{ \frac{3}{2}} (i^{2} - 1) } }
        { \sqrt[3]{\rho(i) - 2} + \sqrt[3]{\rho(i) - 1} }
\]

Type each step, and compile to double check that there are no mistakes.

\begin{enumerate}
    \item Start with the $\left[ \frac{n}{2} \right]$\\
    \begin{verbatim}
$\left[ \frac{n}{2} \right]$
    \end{verbatim}
    \item Now type the sum:\[
        \sum_{i+1}^{ \left[ \frac{n}{2} \right] }
    \]\\
    For the superscript you can copy the formula above (removing the ``\$'')\\
    \begin{verbatim}
\sum_{i+1}^{ \left[ \frac{n}{2} \right] }
    \end{verbatim}
    \item Next the two formulas in the binomial\[
        x_{i, i + 1}^{i^{2}} \qquad \left[ \frac{i + 3}{3} \right]
    \] Type as two separated formulas
    \begin{verbatim}
x_{i, i + 1}^{i^{2}} \qquad \left[ \frac{i + 3}{3} \right]
    \end{verbatim}
    \item Modify the formula above, by removing the \texttt{qquad}, and adding \texttt{binom} in the beginning
    \begin{verbatim}
\binom{ x_{i,i + 1}^{i^{2}} }{ \left[ \frac{i + 3}{3} \right] }
    \end{verbatim}
    You will get the binomial formula:
    \[
        \binom{ x_{i,i + 1}^{i^{2}} }{ \left[ \frac{i + 3}{3} \right] }
    \]
    \item Next type the formula under the square root $\mu(i)^{ \frac{3}{2} } (i^{2} - 1)$ as
    \begin{verbatim}
\mu(i)^{ \frac{3}{2} } (i^{2} - 1)
    \end{verbatim}
    Then add the square root $\sqrt{ \mu(i)^{ \frac{3}{2} } (i^{2} - 1) }$
    \begin{verbatim}
\sqrt{ \mu(i)^{ \frac{3}{2} } (i^{2} - 1) }
    \end{verbatim}
    \item Type the two cubic roots: $\sqrt[3]{ \rho(i) - 2 }$, and $\sqrt[3]{ \rho(i) - 1 }$
    \begin{verbatim}
\sqrt[3]{ \rho(i) - 2 }  \sqrt[3]{ \rho(i) - 1 }
    \end{verbatim}
    \item Now get the fraction
    \[
    \frac{ \sqrt{ \mu(i)^{ \frac{3}{2}} (i^{2} -1) } }
        { \sqrt[3]{ \rho(i) - 2 }  + \sqrt[3]{ \rho(i) - 1 } }
    \]
    \begin{verbatim}
\frac{ \sqrt{ \mu(i)^{ \frac{3}{2}} (i^{2} -1) } }
    { \sqrt[3]{ \rho(i) - 2 }  + \sqrt[3]{ \rho(i) - 1 } }
    \end{verbatim}
    \item Finally put everything together by copying-and-pasting from the steps above
    \begin{verbatim}
    \sum_{i = 1}^{ \left[ \frac{n}{2} \right] }
    \binom{ x_{i, i + 1}^{i^{2}} }
        { \left[ \frac{i + 3}{3} \right] }
    \frac{ \sqrt{ \mu(i)^{ \frac{3}{2}} (i^{2} - 1) } }
        { \sqrt[3]{\rho(i) - 2} + \sqrt[3]{\rho(i) - 1} }
    \end{verbatim}
    To get the complete formula
    \[
    \sum_{i = 1}^{ \left[ \frac{n}{2} \right] }
    \binom{ x_{i, i + 1}^{i^{2}} }
        { \left[ \frac{i + 3}{3} \right] }
    \frac{ \sqrt{ \mu(i)^{ \frac{3}{2}} (i^{2} - 1) } }
        { \sqrt[3]{\rho(i) - 2} + \sqrt[3]{\rho(i) - 1} }
    \]
\end{enumerate}

Notice the use of:
\begin{itemize}
    \item \emph{spacing} to help distinguish the braces (Overleaf can help balancing the braces).
    \item Separated lines for each major part of the formula.
\end{itemize}

The points are for \emph{your} readability, \LaTeX{} itself doesn't care about spaces, but try to find the error in the formula below

\begin{verbatim}
\sum_{i=1}^{\left[\frac{n}{2}\right]}\binom{x_{i,i+1}^{i^{2}}}
{\left[\frac{i+3}{3}\right]}\frac{\sqrtt{\mu(i)^{\frac{3}{2}}(i^{2}-1)}}
{\sqrt[3]{\rho(i)-2}+\sqrt[3]{\rho(i)-1}}
\end{verbatim}


\newpage
\part{Regression Analysis}
\section{Definition \& Terminology}
Regressions identify \textbf{relationship between dependent and independent variables}.
\begin{itemize}
	\item Association between dependent \& independent variables,
	\item Impact of independent variables on dependent variables.
	\item Formulation of association/impact in functional form.
	\item Used for
	\begin{itemize}
		\item descriptive analysis
		\item numerical prediction
		\item time series forcasting
	\end{itemize}
\end{itemize}
\subsection*{Terminology}
\begin{itemize}
	\item \textbf{measurement tuples}: data streams  $(x_1,y_1), \dots, (x_n,y_n)$
	\item \textbf{predictor (independent variable, feature, regressor, covariate)}: $x_i$
	\item \textbf{response (dependent variable, outcome)}: $y_i$
	\item \textbf{regression function}: $\eta(x) = E(y|x)$
\end{itemize}

\section{Linear Regression}
\subsection{First Order Linear Model}

$$Y = \beta_0 + \beta_1 X + \varepsilon$$

\begin{itemize}
	\item Y: response variable (from measurement tuples)
	\item X: predictor variable (from measurement tuples)
	\item $\beta_0$: y-Axis intercept, \textbf{unknown, to be estimated}
	\item $\beta_1$: slope, \textbf{unknown, to be estimated}
	\item $\varepsilon$: \textbf{residual}, random error
\end{itemize}

\subsection{Multiple Linear Regression Model}
$$Y = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + \dots + \beta_k X_k + \varepsilon$$
Formulation in \textbf{matrix form}:
$$\begin{bmatrix}
y_1 \\ y_2 \\ \vdots \\ y_m
\end{bmatrix} = \begin{bmatrix}
1 & X_11 &X_12 &\dots &X_1k \\
1 & X_21 &X_22 &\dots &X_2k \\
\vdots & \vdots &\vdots &\vdots &\vdots \\
1 & X_m1 &X_m2 &\dots &X_mk
\end{bmatrix} \begin{bmatrix}
\beta_0 \\ \beta_1 \\ \beta_2 \\ \vdots \\ \beta_k
\end{bmatrix} + \begin{bmatrix}
\varepsilon_1 \\ \varepsilon_2 \\ \vdots \\ \varepsilon_m
\end{bmatrix}$$
$$[m \times 1] = [m \times (k+1)] \cdot [(k+1) \times 1] + [m \times 1]$$

\textbf{OLS Estimator} for multiple linear regression model: minimize the RSS
\begin{itemize}
	\item Model:
	\begin{align*}
		RSS  &= e^{T} e = (y - X\hat{\beta})^T (y - X\hat{\beta}) \rightarrow \min \\
		\rightarrow  &\frac{\partial RSS}{\partial \beta} = -2X^Ty + 2X^TX\beta= 0
	\end{align*}

	\item Solution:
	\begin{align*}
		\hat{\beta} &= (X^TX)^{-1}X^Ty \\ \ \\
		\hat{y} &= X(X^TX)^{-1}X^Ty
	\end{align*}

	\item Projection: minimizing RSS $\rightarrow \vec{e}$ is orthogonal to the subspace spanned by all independent variables
	\begin{figure}[H]
		\centering
		\includegraphics[width=0.4\textwidth]{projection.png}
	\end{figure}
\end{itemize}

\subsection{Estimation of Coefficients: Ordinary Least Square Estimator}
\begin{itemize}
	\item input: data streams $(x_i, y_i)$

		  model: linear regression $Y = \beta_0 + \beta_1 X + \varepsilon$

		  output: $\hat{\beta}_0$, $\hat{\beta}_1$
	\item Goal of OLS estimator: \textbf{minimize the sum of squared residuals (RSS) }

	Outlier will have larger value input when residual squared.
	$$\hat{y} = \hat{\beta}_0 + \hat{\beta}_1 x$$
	$$\min{\Sigma_i e_i^2} = \min{\Sigma_i (y - \hat{y})^2} = \min{\Sigma_i (y - (\hat{\beta}_0 + \hat{\beta}_1 x))^2} $$

\end{itemize}
\subsection{Quality Metrics of the Model}
To measure how well the model fits the data, you can analyze the quality model-wise or coefficient-wise.
\begin{itemize}
	\item Model-wise:
	\begin{itemize}
		\item Residual Sum of Squares (RSS)
		\item Total Sum of Squares (TSS)
		\item $R^2$
	\end{itemize}
	\item Coefficient-wise:
	\begin{itemize}
		\item Significance-Test on each coefficient in the model
	\end{itemize}
\end{itemize}
\subsubsection{Residual Sum of Squares (RSS)}
An \textbf{unbiased} estimator of RSS of the population is given by
$$RSS = \Sigma_{i=1}^{n} (y_i - \hat{y}_i)^2$$
\begin{itemize}
	\item RSS = 0: the model fits 100\% of the data
	\item RSS $\neq$ 0: distance between true $y$ and $\hat{y}$ : residual $e$
	\item RSS $\downarrow$, model-fit quality $\uparrow$
\end{itemize}
\subsubsection{Mean Squared Error (MSE)}
$$MSE = \frac{RSS}{N}$$
\subsubsection{Root Mean Squared Error (RMSE)}
$$RMSE = \sqrt{MSE}$$
\subsubsection{Total Sum of Squares (TSS)}
\textbf{Total Sum of Squares(TSS)} is the sum of Explained Sum of Squares(ESS) and Residual Sum of Squares(RSS).
\begin{align*}
	\Sigma (y - \bar{y})^2 &= \Sigma (\hat{y} - \bar{y})^2 + \Sigma (y - \hat{y})^2 \\
	TSS &= ESS + RSS
\end{align*}
\subsubsection{$\mathbf{R^2}$}
$R^2$ measures the proportion of the variation in y that is explained by the variation in x. $\rightarrow$ the \textbf{proportion of Explained Sum of Squares(ESS).}
$$R^2 = \frac{TSS - RSS}{TSS} = 1 - \frac{RSS}{TSS} = \frac{ESS}{TSS}$$
\begin{itemize}
	\item range of $R^2$: [0,1]
	\begin{itemize}
		\item $R^2 = 0$: ESS = 0, RSS = $\infty$. No linear relationship between x and y.
		\item $R^2 = 1$: ESS = TSS, RSS = 0. Perfect match between x and y.
	\end{itemize}
\end{itemize}

\subsubsection{Significance Test of the coefficients: T-Test}
\begin{itemize}
	\item Test the significance of the coefficients in alternative hypothesis($H_1$):
	$$H_0: \beta_1 = 0$$
	$$H_1: \beta_1 \neq 0$$
	\item Distribution: T-Distribution
	\item Degree of Freedom: if \textbf{error variable} is \textbf{normally distributed}, then $$df = n - 2 $$
	\item test statistic:
	\large{$$t = \frac{\hat{\beta}_1 - 0}{SE(\hat{\beta}_1)} = \dfrac{\hat{\beta}_1}{\sqrt{\frac{RSS}{\Sigma_{i=1}^n (x_i - \bar{x})^2} \cdot \frac{1}{n-2}}}$$}
	\item Conclusion:

	two-sided test, reject $H_0$, if $|t| > t^c_{1-\frac{\alpha}{2}}$ or $p < \alpha$

\end{itemize}
\subsubsection{Other Metrics}
\begin{itemize}
	\item Adjusted $R^2$: allows models \textbf{with different number of variables} to be compared
	\item F-statistic: indicates linear relationship between y and \textbf{at least one} of the xs
	\item T-test of each partial regression coefficient: significance of a single coefficient while controlling others
\end{itemize}

\subsection{Model Specification}
The process of developing a regression model. It's a repeated process. You need to try different combinations or test the significance of variables to find an optimal regression model.

\begin{itemize}
	\item selection of an appropriate functional form (linear, quadratic, log-linear, interaction terms, etc.)
	\item choosing which variables to include (might include irrelevant or omit relevant variables)
\end{itemize}

\subsubsection{Functional Form of Linear Model}
\begin{itemize}
	\item standard normal linear model(first-order/multiple)
	\item nominal variables(categorical): 0 or 1
	\item quadratic models: $y = \beta_0 + \beta_1 x_1 + \beta_2 x_2^2 \rightarrow z_2 = x_2^2$
	\item interaction terms: $y = \beta_0 + \beta_1 x_1 + \beta_2 x_1x_2 \rightarrow z_2 = x_1x_2$
	\item exponential terms into logarithm terms: $y = \alpha x^{\beta}\varepsilon \rightarrow \ln(y) = \ln(\alpha) + \beta\ln{x} + \ln(\varepsilon)$
\end{itemize}

\subsubsection{Choosing Variables to Include}
\begin{itemize}
	\item Idea: The initial model might include large set of irrelevant variables or omit some relevant variables.
	\item Goal: find an optimal combination that explains variation in Y with \textbf{a small and meaningful predictors Xs} $\rightarrow$ feature selection
	\item Methods:
	\begin{itemize}
		\item \textbf{Best Subset}:

		Test \textbf{all combinations} ($2^n$) and find out best subset
		\item \textbf{Backward Elimination}: (top-down)

		Start with\textbf{ full model with all variables}, test the significance(t-Test) of the variables.

		Consider the predictor with lowest t-statistic/highest p-value: remove the variable if $p > \alpha$ (can't reject $H_0$)
		\item \textbf{Forward Selection}: (buttom-up)

		Start with \textbf{only one variable}, test the significance(t-Test) of the variable.

		Only consider the variable with highest t-statistic/lowest p-value: add the variable if $p < \alpha$ (reject $H_0$)
		\item \textbf{Stepwise Regression}: combination of forward/backward selection.
	\end{itemize}
\end{itemize}

\subsection{Model Interpretation}
Result of a model fitting can be retrieved by ''summary(model)''
\begin{itemize}
	\item Hypothesis:
	\begin{itemize}
		\item Parameters:

		$H_0$: $\beta_i = 0$

		$H_1$: $\beta_i \neq 0$
		\item Whole Model:
		$H_0$: All predictors are not able to explain the model.

		$H_1$: The whole model is statistically significant.
	\end{itemize}
	\item Conclusion:

	for an $\alpha = 0.05$ level, $p > \alpha, \rightarrow$ cannot reject $H_0$, the parameter x is not significant from 0.
\end{itemize}
\begin{figure}[H]
	\centering
	\includegraphics[width=0.9\textwidth]{lm_R.png}
\end{figure}

\subsubsection*{Interpretation}
\begin{itemize}
	\item \textbf{predictors, intercepts -- coefficient \& p-value}:
	\begin{itemize}
		\item Significance: significant? at a significance level of $\alpha$ = ?
		\item $\beta_0$: intercept, the value when all predictors are 0.
		\item $\beta_i$: slope, coefficient positive/negative influence of predictors on response?
		\item \textbf{Transformed} predictors/response
	\end{itemize}
	\item \textbf{whole model -- F-statistic \& p-value}: significant? at a significance level of $\alpha$ = ?
	\item \textbf{Explanatory power of the model -- Adjusted $R^2$}: high/low? why?

\end{itemize}


\section{Linear Regression}
\smallskip \hrule height 2pt \smallskip

The loss is $|| y - Xw ||^2 + \lambda || w ||^2$ \hfill \\
(this corresponds to the sum of squared error loss + L2 regularization)\hfill \\
\hfill \\

\underline{Ordinary Least Squares} \hfill \\

Notation:
\begin{itemize}
	\item \textbf{$x_i$}: an input data point.  \_\_ rows by \_\_ columns.
	\item \textbf{$y_i$}: a predicted output
	\item \textbf{$\widehat{y_i}$}: a predicted output
	\item \textbf{$\widehat{y}$}:
	\item \textbf{$w_k$}: weight k
	\item \textbf{$\bm{w}^*$}: the vector of weights found in regression.
	\item \textbf{$f_k(x_i)$}
	\item \textbf{$t$}: what we want to regress against
	\item \textbf{$t_j$}: the output variable that you either have data for or are predicting.
	\item \textbf{$t(\bm{x})$}: Data.  "Mapping from x to t(x)"
	\item \textbf{$H$}: $H = \{ h_1, \dots, h_K \}$.  Basis functions.  In the simplest case, they can just be the value of an input variable/feature or a constant (for bias).
	\item \textbf{$L_2$}: The $L_2$.  Can appear as a loss function to describe the deviation from data or as a penalty.
		% Erick clarification
	\item \textbf{$ || \widehat{w} ||_1$}: "$L_1$" penalty.  The "Manhattan distance".  Like traveling a, b in a pythagorean triangle.  $\sum |x_i|$
	\item \textbf{$ || \widehat{w} ||_2$}.  "$L_2$" penalty.  Euclidean length of a vector.  Like c in a pythagorean triangle.  $\sqrt{\sum |x_i|^2}$
\end{itemize}

\underline{Vocab}:
\begin{itemize}
	\item \textbf {bias-variance tradeoff} - the problem of simultaneously minimizing two sources of error that prevent
			supervised learning algorithms from generalizing beyond their training set.  % https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff
			\begin{itemize}
				\item The bias is error from erroneous assumptions in the learning algorithm.
					High bias can cause an algorithm to miss the relevant relations between features
					and target outputs (underfitting).
				\item The variance is error from sensitivity to small fluctuations in the training set.
					High variance can cause overfitting: modeling the random noise in the training
					data, rather than the intended outputs.
			\end{itemize}
	\item \textbf{basis function}
	\item \textbf{bias} (parameter): like the intercept in a linear equation.  The part that doesn't depend on the features.
	\item \textbf{bias} (learning bias):  (?? "inductive bias" ??)  -
	\item \textbf{hyperplane} - a plane, usually with more than 2 dimensions.
	\item \textbf{input variable} - a.k.a. feature.  % https://en.wikipedia.org/wiki/Dependent_and_independent_variables
		E.g. a column like CEO salary for rows of data corresponding to different companies.
	\item \textbf{response variable} - synonyms: "dependent variable", "regressand", "predicted variable", "measured variable", "explained variable", "experimental variable", "responding variable", "outcome variable", and "output variable".   E.g. a predicted stock price.
	\item \textbf{regularization} -  introducing additional information in order to solve an ill-posed problem or to prevent overfitting.
	% https://en.wikipedia.org/wiki/Regularization_(mathematics)
	E.g. applying a penalty for large parameters in the model.
	\item \textbf{ridge regression} -
	\item \textbf{vector norm}: put in a vector and get out a number like length or size.  Real valued function of some sort of vector or matrix quantity.
	\item \textbf{hyperparameters}: \hfill \\
	$\lambda$, not $w$.  Parameters that control your actual parameters.  \hfill \\ \hfill \\
	% Wrong?!?   Norm 1 and norm 2 are not "parameters".  $\dots$ model design.  \hfill \\
	% Wrong?!?   Different values of lambda won't change the complexity of the model.  \hfill \\

	% Erick definition:
	In Bayesian analysis, the parameters that don't touch the data.
	Like the parameters for the prior on the prior.
	Called the ridge regression $\lambda$ a hyperparameter, though this is a stretch in the terminology.  \hfill \\
	Class version:
	\item \textbf{feature selection}: explicitly select features that can go into your model instead of throwing all features in.
	\item \textbf{loss function}:
		% wikipedia:
		 A function that maps an event or values of one or more variables onto a real number intuitively representing some 			"cost" associated with the event.
		 An optimization problem seeks to minimize a loss function. \hfill \\
		% week 4 typed notes near bottom.
		$\sum_j (t(x_j) - \sum w_i h(x_i))^2$. (least-squares error ($L_2$))  (??? = training error??)
	\item \textbf{training set error}:  *doesn't include the regularization penalty!*.  A.k.a. "training error".
			Sum of squares error divided by the number of points.   See formula later.
%	\item \textbf{training error}:  sononomous
\end{itemize}

\subsection{Ordinary Least Squares}
total error = $\displaystyle \sum_i (y_i-\hat{y_i})^2 = \sum_i(y_i - \sum_k w_k f_k(x_i))^2$ \hfill \\
$i$ is for each data point, $k$ is for each of the k basis functions.

 \hfill \\ \hfill \\

Under the additional assumption that the errors be normally distributed, OLS is the maximum likelihood estimator. \hfill \\ % https://en.wikipedia.org/wiki/Ordinary_least_squares
?? Use words to describe what subset of regression in general this is.  What is ordinary? What are we limiting?  \hfill \\
 \hfill \\

The regression problem: \hfill \\
Given basis functions $\{ h_1, \dots, h_K \}$  with $h_i(\bf{x}) \in \mathbb{R}$,  \hfill \\
	find coefficients $\bm{w} = \{ w_1, \dots, w_k \}$.  \hfill \\%
$t(\bm{x}) \approx \widehat{f}(\bm{x}) = \sum_i w_i h_i(\bm{x})$     \hfill \\  \hfill \\

This is called linear regression b/c it is linear in the parameters.
We can still fit to nonlinear functions by using nonlinear basis functions;  $f_k \rightarrow  h_i$
Minimize the \textbf{residual squared error}: \hfill \\
$ \displaystyle \bm{w}* = \argmin_{\bm{w}}  \sum_j (t(\bm{x}_j) - \sum_i w_i h_i(\bm{x}_j))^2$  \hfill \\
$j$ for each data point, $i$ for the number of weights, which is the number of basis functions.  \hfill \\
\hfill \\  \hfill \\

For fitting a line in 2D space, your basis functions are $\{ h_1(x) = x, h_2(x) = 1 \}$.  $h_2(x)$ = 1 is the (constant) bias basis function.  The size of $w_2$ controls the effect in prediction.    \hfill \\  \hfill \\

To fit a parabola, your basis functions could be $\{ h_1(x) = x^2, h_2(x)=x, h_3(x)=1 \}$.   \hfill \\
Want a 2D parabola? Use $\{ h_1(x) = x_1^2, h_2(x)=x_2^2, h_3(x)=x_1 x_2, \dots \}$. \hfill \\
Can define any basis functions $h_i(\bm{x})$ for n-dimensional input $\bm{x} = <x_1, \dots, x_n>$
\hfill \\  \hfill \\

\underline{Linear in feature space vs linear in the parameter space}:  \hfill \\
When we want to fit a parabola, we are still using a function that is linear in \underline{parameter} space.
The weight matrix is still all constants.
When we use nonlinear basis functions, we are nonlinear in \underline{feature} space.  The basis functions may be transformations of the input parameters.
 \hfill \\ \hfill \\

\subsection{Regression: matrix notation}
\begin{align*}
	\bm{w}* &= \argmin_w \sum_j(t(\bm{x}_j - \sum_i w_i h_i(\bm{x}_j))^2  \\
	\bm{w}* &= \argmin_w (\bm{Hw} -\bm{t})^T (\bm{Hw} -\bm{t})
\end{align*}
$  (\bm{Hw} -\bm{t})^T (\bm{Hw} -\bm{t})$ is the residual error.
\includegraphics[width=3in]{figures/Least_squares_matricies.pdf}

\subsection{Regression: closed form solution}
 % derivation: http://courses.cs.washington.edu/courses/cse446/16wi/Slides/4_LinearRegression.pdf
\begin{align*}
	\bm{w}^* = \argmin_w (\bm{Hw} -\bm{t})^T (\bm{Hw} -\bm{t})  & \\
	\bm{F}(\bm{w}) =  \argmin_w (\bm{Hw} -\bm{t})^T (\bm{Hw} -\bm{t}) & \\
	\triangledown_{\bm{w}}\bm{F}(\bm{w}) = 0 \\
	2 \bm{H}^T (\bm{H}\bm{w}-\bm{t}) = 0  & \\
	(\bm{H}^T\bm{H}\bm{w}) - \bm{H}^T\bm{t} = 0 & \\
	\bm{w}^* = (\bm{H}^T\bm{H})^{-1}\bm{H}^T\bm{t} &
\end{align*}

\includegraphics[width=3in]{figures/Regression_matrix_math.pdf}

Dimensions:
\begin{itemize}
	\item $\bm{t}$:  N-dimensional  (N = \# of input data points)
	\item $\bm{w}$:
	\item $\bm{H}$: k + 1 by N.  N is \# of rows.
\end{itemize}
\hfill \\

\subsubsection{Linear regression prediction is a linear function plus Gaussian noise}
%______TA e-mail 3/15________________________
\underline{Casual explanation:} \hfill \\
If we assume that our data $y_i$ is drawn from a linear function with some zero-mean Gaussian noise, i.e.

$$y_i = \sum_j w_j X_ij + \epsilon_i$$
(where $\epsilon_i$ is zero-mean gaussian noise drawn from Normal(0, $\sigma$)

Then we can show that the MLE estimates of $w_j$ are exactly the optimal weights obtained by minimizing the SSE: $\sum_i (y_i - w^T X_i)^2$

(note that the variance $\sigma$ actually doesn't matter for the derivation, you can just assume it's some positive number). \hfill \\ %%
\hfill \\
%______________________________
\underline{More formal explanation:} \hfill \\
We can model as linear combination of basis functions + noise $\epsilon$.
It's safe to assume epsilon comes from Gaussian distribution.  \hfill \\

$t(\bm{x}) = \sum_i w_i h_i(\bm{x}) + \epsilon $ \hfill \\
Note: no $\mu$ because we set it to zero.  \hfill \\

We can learn $\bf{w}$ using MLE:
$P(t | x, w, \sigma) = \frac{1}{\sigma \sqrt{2 \pi}} e^\frac{-[t - \sum_i w_i h_i(x)]^2}{2 \sigma^2}$
Take the log and maximize with respect to w:  (maximizing log-likelihood with respect to w) \hfill \\
$\displaystyle \ln P(D | \bm{w}, \sigma) = \ln(\frac{1}{\sigma \sqrt{2 \pi}})^N \prod_{j=1}^N e^\frac{-[t_j - \sum_i w_i h_i(x_j)]^2}{2 \sigma^2}$ \hfill \\
Now find the w that maximizes this: \hfill \\
$\argmax_w \ln(\frac{1}{\sigma \sqrt{2 \pi}})^N + \sum_{j=1}^N \frac{-[t_j - \sum_i w_i h_i(x_j)]^2}{2 \sigma^2}$ \hfill \\
the first term isn't impacted by $w$ so  \hfill \\
$= \argmax_w  \sum_{j=1}^N \frac{-[t_j - \sum_i w_i h_i(x_j)]^2}{2 \sigma^2}$ \hfill \\
switch to $\argmin_w$ when we divide by -1.  The numerator is constant.:  \hfill \\
$= \argmin_w  [t_j - \sum_i w_i h_i(x_j)]^2 $ \hfill \\

\textbf{Least-squares Linear Regression is MLE for Gaussians!!!}  \hfill \\ \hfill \\

If you have a polynomial you are fitting, how many basis functions are there (??)  \hfill \\ % asked Wed 1/27

\subsection{OLS Protocol (incomplete)}  \hfill \\
\begin{enumerate}
	\item Chose your basis functions $h_i$.  (Requires expertise).  Can be nonlinear, e.g. $x_1^2, \sin(x)$, etc.
		\begin {itemize}
			\item
				The number of parameters is len(H) + 1 (bias).
				E.g. for fitting a parabola (formula $y = ax^2 + bx + c$),
				you have 3 parameters: weights for basis functions $x$, $x^2$, and bias.
				Typically the \# of basis functions is $<$ the \# of features.
		\end {itemize}
	\item Chose a regularization method so your weights don't get too big.  (see below)
	\item Plug them in to regression to get the weights ($w_i$s).  (Form the sum (residual squared error + regularization) that you want to minimize, then minimize.)
	\item Make sure your weights aren't too big.

\end{enumerate}

\subsection{Regularization in Linear Regression}  \hfill \\
You need to regularize to prevent parameters from growing too large.
Both of these were built from the same set of basis functions; the right one is clearly over-fit. \hfill \\
\includegraphics[width=1.2in]{figures/need_to_regularize.pdf}

\subsubsection{Ridge Regression}  \hfill \\
Ridge Regression is the most famous form of linear regression
Here is our old "ordinary" least squares objective function:   \hfill \\
$\displaystyle \widehat{w} = \argmin_w \sum_{j=1}^N [t(x_j) - (w_0 + \sum_{i=1}^k w_i h_i(x_j))]^2$   \hfill \\
It is the same as the previous ones but $i=0$ is pulled out.    \hfill \\
Now for ridge regression, we use that same notation.  \hfill \\
And we add a penalty term that isn't applied to the bias feature:
\begin{align*}
	\widehat{w}_{ridge} &= \argmin_w \sum_{j=1}^N [t(x_j) - (w_0 + \sum_{i=1}^k w_i h_i(x_j))]^2 + \lambda \sum_{i=1}^k w_i^2   \\
	&= \argmin_w (\bm{H}\bm{w} - \bm{t})^T(\bm{H}\bm{w}-\bm{t}) + \lambda \bm{w}^T I_{0+k} \bm{w}
\end{align*}
That $I_{0+k}$ matrix is this:
\includegraphics[width=1.0in]{figures/ridge_identity_matrix_with_zero.pdf}  \hfill \\
% Erick hasn't seen this notation.
Allows you to multiply the whole weight array without getting the bias term in there.

Note: $W^TW$ is $w_i^2$ or $|| w_i ||^2$

A similar derivation leads to a closed form solution:  \hfill \\
% http://courses.cs.washington.edu/courses/cse446/16wi/Slides/4_LinearRegression.pdf
$w_{ridge}^* = (\bm{H}^T\bm{H} + \lambda I_{0+k})^{-1}\bm{H}^T\bm{t}$ \hfill \\
(Recall that un-regularized regression was $w^* = (\bm{H}^T\bm{H})^{-1}\bm{H}^T\bm{t}$).  \hfill \\   \hfill \\

How do you chose how large $\lambda$ is? \hfill \\
* As $\lambda \rightarrow 0$, becomes same as MLE: unregularized.  Large magnitudes of coefficients. \hfill \\
* As $\lambda \rightarrow \infty$, all weights become 0.  \hfill \\   \hfill \\

\subsubsection{Experiment cycle}
\begin{enumerate}
	\item select a hypothesis $f$ to best match the training set.
		(??) Is this the same as choosing your basis functions (?)
	\item isolate a held-out data set if you have enough data, or do K-fold cross-validation if not enough data.
	\begin{itemize}
		\item tune hyperparameters ($\lambda$) on the held-out set or via cross-validation.
			(Try many values of $\lambda$ and chose the best one.)
		\item You can use the same held-out data set each time if that set is big.
		\item If doing K-fold, divide the data into k subsets.
				Repeatedly train on k-1 and test on the remaining one.
				Average the results.
		\item find the $w$ that minimizes the error.
			(Do so by taking the derivative and setting = 0);
			see ridge regression notes.
	\end{itemize}
	\item Select basis functions
\end{enumerate}

\subsection{Regularization options: Ridge \& Lasso}  \hfill \\
Ridge:
\begin{itemize}
	\item $ \displaystyle \widehat{w}_{ridge} = \argmin_w \sum_{j=1}^N [t(x_j) - (w_0 + \sum_{i=1}^k w_i h_i(x_j))]^2 + \lambda \sum_{i=1}^k w_i^2   $
	\item $L_2$ penalty.  ("$L_2$ norm of $\bm{w}$).
		Large distances get penalized more.  $Y = x^2$:
		don't want errors to cancel each other; differentiable.
\end{itemize}
Lasso: \hfill \\
\begin{itemize}
	\item$ \displaystyle \widehat{w}_{ridge} = \argmin_w \sum_{j=1}^N [t(x_j) - (w_0 + \sum_{i=1}^k w_i h_i(x_j))]^2 + \lambda \sum_{i=1}^k |w_i|   $
	\item L1 penalty: linear penalty pushes more weights to zero.  Allows for a type of feature selection.  But it is not differentiable and there is no closed form solution.
	\item L1 is absolute value: don't need to square it.
	\item Lasso may be more useful when you have too many features for the amount of data you get.
		Example: 100k parameters about companies to predict stock prices but only 100 data points.
		Could tune lambda until you have about 100 nonzero weights.
\end{itemize}

Your chose of penalty has huge effects on the algorithm! \hfill \\
Sometimes $L_1$ will do better than $L_2$ and vice versa, but usually $L_2$ is more powerful than $L_1$. \hfill \\
Would be better to use a combination of the penalties than to first reduce the number of features with $L_2$ before applying $L_1$.    \hfill \\
You can generate a lot of basis functions and use $L_1$ to chose the good ones. \hfill \\
\hfill \\

\includegraphics[width=3in]{figures/lasso_and_ridge_geometry.pdf}

This figure shows:
\begin{itemize}
	\item The contour lines represent the maximum likelihood of the vector of weights.
		All points on the contour have equal likelihood.
	\item The two axes represent different parameters for two of the weights.  (regression coefficients)
	\item Circles are characteristic of ridge regression (with L2 penalty): Penalty = the magnitude of the vector.
	\item Shapes that are pointy on the axes are characteristic of Lasso (with L1 penalty): the vector components get added.
	\item Where the likelihood function touches in this $w_1, w_2$ space represents the coefficients of the weights.
		For Ridge Regression, we see that small but nonzero values of the coefficients can be obtained.
		For Lasso Regression, the curves are most likely to touch the diamond on the axes,
		resulting in coefficients that are truly zero.
\end{itemize}

\includegraphics[width=1.8in]{figures/lambda_with_w2.pdf}  \includegraphics[width=1.6in]{figures/lambda_with_w1.pdf}
Don't compare coefficient magnitudes at given $\lambda$s,
but do note that for Ridge the gradually come away from the zero axis and in Lasso they are zero until they pop out.   \hfill \\ \hfill \\

\subsection{Bias-Variance Tradeoff}
\includegraphics[width=2.0in]{figures/biasvariance.png}

Your choice of hypothesis class (e.g. degree of polynomial) introduces learning bias.  \hfill \\
The more complex the model, the more the \_\_\_\_ set accuracy goes down  \hfill \\  % had "training" with ?
\textbf{A more complex class } $\rightarrow$ less bias and more variance.   \hfill \\ \hfill \\

From https://www.youtube.com/watch?v=Rm6s6gmLTdg: hfill \\
High bias = inability to represent the true function in the class of functions we are willing to tolerate.
Pulls us to a particular function or class of functions regardless of the data.
Can be from looking only at a specific type of function, or by having strong regularization  \hfill \\
High variance = extremely dependent on the exact data they were trained on.
Might fit the data well, but may do poorly on a different data self.  \hfill \\

From Wikipedia: \hfill \\  % https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff
Ideally, one wants to choose a model that both accurately captures the regularities in its training data,
but also generalizes well to unseen data. Unfortunately, it is typically impossible to do both simultaneously.
High-variance learning methods may be able to represent their training set well, but are at risk of overfitting
to noisy or unrepresentative training data.
In contrast, algorithms with high bias typically produce simpler models that don't tend to overfit,
but may underfit their training data, failing to capture important regularities. \hfill \\  \hfill \\

Models with low bias are usually more complex (e.g. higher-order regression polynomials),
enabling them to represent the training set more accurately.
In the process, however, they may also represent a large noise component in the training set,
making their predictions less accurate - despite their added complexity.
In contrast, models with higher bias tend to be relatively simple (low-order or even linear regression polynomials),
but may produce lower variance predictions when applied beyond the training set. \hfill \\ \hfill \\

Adding features (predictors) tends to decrease bias, at the expense of introducing additional variance.   \hfill \\
\hfill \\

Solutions:
\begin{itemize}
	\item Dimensionality reduction and feature selection can decrease variance by simplifying models.
	\item Similarly, a larger training set tends to decrease variance.
	\item Use tunable modeling parameters such as applying more significant regularization.
\end{itemize}

\textbf{If held-out is too small, we can end up over-fitting.}

\hfill \\  \hfill \\

\subsection{Error Definitions}

\underline{True ("Prediction") Error}:   \hfill \\
Since the training set error can be a poor measure of the "quality" of the solution, we can use prediction error ("true error").
The error over all possibilities.   Instead of sum, take expectation.
\begin{align*}
	error_{true}(\bm{w}) &= E_X[(t(\bm{x_j})-\sum_{i} w_i h_i(\bm{x_j}))^2] \\
		& \mbox{Gold Standard:} \\
	error_{true}(\bm{w}) &= \int_x (t(\bm{x_j})-\sum_{i} w_i h_i(\bm{x_j}))^2 p(\bm{x}) d\bm{x}
\end{align*}

% TA 3/15/2016:
$p(x)$ is assumed to be the "true" distribution of the data, which we don't actually know.

How to get $p(\bm{x})$?  Need to know the true distribution of the data (?).
You almost never know how to compute $p(x)$.
And, the integral is a very big sum. \hfill \\  \hfill \\

% Find $p(x)$ using monte-carlo integration. \hfill \\

% TA 3/15/2016:
To do this, you need to split your data into training and test set.
If we have a dataset which is randomly sampled from $p(x)$, which we assume our data is, we can estimate the true error using the average of the samples.
Since we train on our training set, the error might be a bad estimate of the true error (we're biased to decrease the error), so we use a separate hold-out testing set to approximate the true error in instead.

Sample a set of i.i.d. points  $\{ \bm{x}_1, \dots, \bm{x}_M \}$ from $p(x)$.   \hfill \\
Approximate the integral with the sample average:
$$ error_{true}(\bm{w}) \approx \frac{1}{M} \sum_{j=1}^M \left( t(\bm{x}_j - \sum_i w_i h_i(\bm{x}_j) \right) $$
That is the sampling approximation of the predicted error: \hfill \\
That leads to a fair approximation.   \hfill \\

\hfill \\  \hfill \\

The true prediction error is the expectation over \textbf{future} test cases you don't have.
Since you don't have the x values, you go to probability.
Pick a point from the distribution, and calculate the \_\_\_\_\_.     \hfill \\  \hfill \\

Don't use the training data to predict true error; you've already trained to that data!  You would have too optimistic of a prediction for true error.

Prediction error is high when the model is too simple \underline{and} too complex, unlike training set error which only penalizes too simple.  \hfill \\
\hfill \\

\underline{Training Set Error}:  \textbf{optimistically biased}  (a.k.a. training error) \hfill \\
$\displaystyle  error_{train}(\bm{w}) = \frac{1}{N_{train}} \sum_{j=1}^{N_{train}}(t(\bm{x_j})-\sum_{i} w_i h_i(\bm{x_j}))^2$ \hfill \\
Decreases exponentially with model complexity.   \hfill \\
\textbf{Training error is a poor prediction of prediction error!} \hfill \\
You expect to see training error to decrease with complexity, but that doesn't mean you have a good solution!  \hfill \\
\hfill \\  \hfill \\
% http://courses.cs.washington.edu/courses/cse446/16wi/Slides/4_LinearRegression.pdf

\underline{Test Error}: (our final measure)  \hfill \\
Uses the same formula as prediction error, except that we have never observed the test data.  See formula below.
We expect the true error to be smile shaped if the x-axis is model-complexity.
\hfill \\ \hfill \\

\underline{\textbf{Testing is for the user of your algorithm. }}
The user doesn't care about the values in your model.  They just care about how well it works.
 They dont' care about the value of your hyperparameters.

\subsubsection{Effects of $\lambda$ value on model}
% week 4 typed notes.
 \begin{itemize}
 	\item high lambda $\rightarrow$ simple model $\rightarrow$ lots of zeros $\rightarrow$ high error.
	\item with lambda = 0 $\rightarrow$ converges ridge regression to regular regression.
	\item with enough traiing data and lambda = 0, we expect overfitting $\rightarrow$ small training error.
\end{itemize}

\subsubsection{Choosing $\lambda$}
How to find lambdas?
\begin{itemize}
	\item try a bunch and find which does best on the held out data set.
		Can't touch test data, even w/ stick so use held-out set.
	\item Do k-fold cross validation to pick the lambda that gives minimum error. \hfill \\
		Average over the loss curves Loss($fold_i$, $\lambda$) \hfill \\
		Minimum error = lowest loss on the hold-out set (the curve should look U shaped, so you just find the bottom of the U).  \hfill \\
		May not find the bottom-of-the-U shape lambda.    \hfill \\
		But it is ok to be off a bit; the red U might be pretty flat at the bottoms. \hfill \\
\end{itemize}
This use of the held-out data doesn't count as training.
Lambda is fixed each time, and we train separately.
Training on training data, just watching the number on the held-out set. \hfill \\
\textbf{The model's parameters are not determined by the held-out data} \hfill \\
\hfill \\

How do you chose the range of lambda? (practical solution)
\begin{itemize}
	\item You are limited to a grid search over values of lambda.
	\item You need a strategy for sweeping that space.
		Could do a "binary search", or something like a gradient search:
		take a step until we screw it up, and step back smaller.
		If it gets worse, you take a smaller step.
	\item From your loss, take the value of your loss (compute it).  \hfill \\
		Loss = $(t(x) - \sum w_i h(x_i))^2$
        		\textbf{"You should be able to find loss given w}
            	Can compute norm of W, too.
		If your loss is on the order of 1000 and your norm is on the order of 1, you can use these for defining search space.
                	Firs try lamba = 1, 10, 100, 1000.
		Find the best from there then explore around there.
		If 100 was good, try 200, etc.
		You just want to know in which order the loss function is going to appear.
\end{itemize}

\subsubsection{How to handle error calculations:}
Given a dataset, randomly split it into two parts:  \hfill \\
* training data: $\{ \bm{x_1}, \dots, \bm{x_{N_train}} \}$   \hfill \\
* test data: $\{ \bm{x_1}, \dots, \bm{x_{N_test}} \}$   \hfill \\
Use the training data to optimize parameters $\bm{w}$. \hfill \\
To calculate the test set error, you use the final solution $\bm{w^*}$ and calculate \hfill \\
$\displaystyle error_{test}(\bm{w}) \approx \frac{1}{N_{test}} \sum_{j=1}^{N_{test}} (t(\bm{x_j} - \sum_i w_i h_i(\bm{x_j}))^2$  \hfill \\

\includegraphics[width=3in]{figures/errors_as_f_of_complexity.pdf}   \hfill \\
(blue is train, red is true, green is test) \hfill \\

\subsubsection{overfitting}
Assume: \hfill \\
* Data generated from distribution D(X,Y). \hfill \\
* A hypothesis space H \hfill \\
Define:  \hfill \\
* Training error: $error_{train}(h)$ \hfill \\
* Data (true) error: $error_{true}(h)$ \hfill \\
We say $h$ \textbf{overfits} the training data if there exists an $h' \in H$ such that:  \hfill \\
* $error_{train}(h) < error_{train}(h)'$ and $error_{true}(h) > error_{true}(h)'$

\includegraphics[width=2in]{figures/Error--vs_lambda--fixed_model_complexity.pdf}   \hfill \\
Lambda (the regularization parameter) is fixed.  Blue = training set error.
Leads to overfitting/too-large-parameters when $\lambda$ --> 0.
Red = true error, which likes a happy medium $\lambda$.  \hfill \\  \hfill \\

\textbf{Warning}: your test set is only unbiased if you never ever ever do \textbf{any} learning on the data.
This includes using the test data to select for the degree of the polynomial you fit.   \hfill \\  \hfill \\
(Recall, you can create a held-out/validation set from your training data or do k-fold validation.)  \hfill \\
\hfill \\

The height of the true error (red curve) is the "bias".  \hfill \\

\subsubsection{What you need to know}
Regression:
\begin{itemize}
	\item Basis functions = features
	\item Optimizing a sum of squared errors
	\item Relationship between regression and Gaussians
\end{itemize}
Regularization
\begin{itemize}
	\item Ridge regression math
	\item LASSO formulation
	\item How to set lambda
\end{itemize}
Bias-Variance trade-off


\paragraph{Simplicial convolution.}

A convolutional layer is of the form $\psi\circ(f\ast \varphi_W)$, where $\ast$ denotes convolution, $\varphi_W$ is a function
\emph{with small support} parameterized by learnable weights $W$, and $\psi$ is some nonlinearity and bias. This formulation of CNNs lends itself to a spectral interpretation that we exploit to extend CNNs to a much more general setting.

Following~\cite{defferrard2016convolutional} and motivated by the fact that the discrete Fourier transform of a real-valued function on an $n$-dimensional cubical grid coincides with its decomposition into a linear combination of the eigenfunctions of the graph Laplacian for that grid, we define the Fourier transform of real $p$-cochains on a simplicial complex with Laplacians $\mathcal{L}_p$ as
\begin{align*}
  &\mathcal{F}_p: C^p(K) \to \mathbb{R}^{\lvert K_p \rvert} \\
  &\mathcal{F}_p(c) = \left(\ip{c}{e_1}_p, \ip{c}{e_2}_p, \dotsc, \ip{c}{e_{\lvert K_p \rvert}}_p\right),
\end{align*}
where the $e_i$'s are the eigencochains of $\mathcal{L}_p$ ordered by eigenvalues $\lambda_1\leq\dotsm\leq\lambda_{\lvert K_p \rvert}$. The function $\mathcal{F}_p$ is invertible since $\mathcal{L}_p$ is diagonalizable; explicitly, if we write $U\diag(\Lambda)U\transpose$ for a normalized eigendecomposition, the orthonormal matrices $U$ and $U\transpose$ represent $\mathcal{F}\inv_p$ and $\mathcal{F}_p$, respectively. This is the foundation for Barbarossa's development of signal processing on simplicial complexes~\cite{barbarossa2018learning}.

Recall that on the function classes for which it is defined, the classical Fourier transform satisfies $\mathcal{F}(f\ast g)=\mathcal{F}(f)\mathcal{F}(g)$, where the right-hand side denotes pointwise multiplication. This will be our definition of convolution in the simplicial setting. Indeed, for cochains $c,c'\in C^p(K)$ we \emph{define} their convolution as the cochain
\begin{equation*}
  c\ast_p c' = \mathcal{F}_p\inv\left(\mathcal{F}_p(c)\mathcal{F}_p(c')\right).
\end{equation*}

Within this framework, we are led to define a \emph{simplicial convolutional layer} with input $p$-cochain $c$ and weights $W$ as being of the form
\begin{equation*}
  \psi\circ\left(\mathcal{F}\inv_p(\varphi_W)\ast_p c\right)
  \end{equation*}
for some as of yet unspecified $\varphi_W\in\mathbb{R}^{\lvert K_p \rvert}$. To ensure the central property that a convolutional layer be localizing, we demand that $\varphi_W$ be a low-degree polynomial in $\Lambda=(\lambda_1, \dotsc, \lambda_{\lvert K_p \rvert})$, namely
\begin{equation*}
  \varphi_W = \sum_{i=0}^N W_i\Lambda^i = \sum_{i=0}^N W_i(\lambda^i_1, \lambda^i_2, \dotsc, \lambda^i_{\lvert K_p \rvert}),
\end{equation*}
for small $N$. In signal processing parlance, one would say that such a convolutional layer \emph{learns filters that are low-degree polynomials in the frequency domain}.

The reason for restricting the filters to be these low-degree polynomials is best appreciated when writing out the convolutional layer in a basis. Let $L^i_p$ denote the $i$'th power of the matrix for $\mathcal{L}_p$ in, say, the standard basis for $C^p(K)$, and similarly for $c$. Then (ignoring $\psi$),
\begin{equation*}
  \mathcal{F}\inv_p(\varphi_W)\ast_p c = \sum_{i=0}^N W_iU\diag(\Lambda^i)U\transpose c = \sum_{i=0}^N W_i\left(U\diag(\Lambda)U\transpose\right)^i c = \sum_{i=0}^NW_iL^i_pc. %\label{eq:filter}
\end{equation*}
This is important for three reasons, like for traditional CNNs.
First, the convolution can be efficiently implemented by $N$ sparse matrix-vector multiplications: This reduces the computational complexity from $\mathcal{O}(\lvert K_p\rvert^2)$ to $\mathcal{O}(\xi\lvert K_p\rvert)$ where $\xi$ is the density factor.
Second, the number of weights to be learned is reduced from $\mathcal{O}(\lvert K_p\rvert)$ to $\mathcal{O}(1)$.
Third, the operation is $N$-localizing in the sense that if two simplices $\sigma,\tau$ are more than $N$ hops apart, then a degree-$N$ convolutional layer does not cause interaction between $c(\sigma)$ and $c(\tau)$ in its output (see the supplementary material).
Those local interactions (in the spatial domain) can be interpreted as message-passing between simplices.%~\cite{gilmer2017NeuralMP}.

%In practice we implement \refeq{filter} using Chebyshev polynomials, as suggested for GNNs in~\cite{defferrard2016convolutional}.
%\mdeff{Not very important: The choice of polynomial (monomials, Chebyshev, Legendre, etc.) doesn't make much difference in practice.}


\lab{Algorithms}{Temporal Complexity and Sparse Matrices}{Complexity and Sparse Matrices}

\objective{It also introduces the concept of temporal complexity.
Finally, it explores SciPy's special methods for working with sparse matrices.}

\section*{Temporal Complexity}
One of the most important questions in scientific computing is:
How long will this operation take?
The concept of temporal complexity attempts to answer this question by determining
how much time a function needs to operate on a given size of input.
For example suppose calculating the inverse of a matrix of size $n$ requires the following number of calculations.
\begin{equation*}
f(n) = \frac{3n^3}{2} + 75n^2 + 250n + 30
\end{equation*}
What is the most important part of this expression?
When our input gets very large the only relevant term in this equation is $n^3$.
For this reason we say that $f(n) \in O(n^3)$, or more commonly that $f(n)$ is $O(n^3)$
(spoken ``Big O of n cubed'' or ``order of n cubed").
This notation is borrowed from analysis. The notation captures the salient behavior of our temporal complexity,
or more precisely the growth rate we can expect of the execution time of our algorithm.
We also refer to this as \emph{asymptotic notation}.
We will discuss this concept later, but this is a simple introduction to the notion of complexity and Big O.
\begin{definition}[Temporal Complexity]
Let $n$ represent the problem size.  Let $f(n)$ and $g(n)$ be functions mapping natural numbers to positive real numbers.
$f \in O(g(n))$ if $\exists c > 0, \forall n > n_0, f(n) \leq cg(n)$.
When this is satisfied, we say that $g(n)$ is an asymptotic upper bound for $f(n)$.
The notation $f = O(g(n))$ is also common.
\end{definition}

\begin{problem}
Verify the following relationships
\begin{itemize}
\item $O(\log{n}) \subset O(n) \subset O(n\log{n})$
\item $O(n^k) \subset O(n^{k+1})$
\item $O(n^p) \subset O(2^n)$ for all $p \in \mathbb{N}$
\end{itemize}
\end{problem}

What does this have to do with computer programs?
Every algorithm executed on a computer corresponds to a function
that returns the number of steps taken (and therefore the time)
given an input of size n.  Certain problems can be solved with fewer steps,
whilst others require many more.  For example, given two matrices of size n,
it only takes about n steps to add the entries together and thus matrix addition is $O(n)$.
On the other hand, given a matrix of size $n$, it takes about
$n^3$ steps to calculate its inverse,
and thus inverting a matrix is $O(n^3)$.
There is something inherently more difficult about inverting a matrix;
there is complexity that simply isn't there with the straightforward operation of addition.

How do we determine the temporal complexity of a piece of code?
Calculating the exact temporal complexity of a formula is fairly difficult.
However, to asymptotically analyze a function is fairly straightforward.
\begin{lstlisting}
s = 0
for i in xrange(100):
    s = s + i
\end{lstlisting}
The code above is $O(n)$ because it takes approximately $n$ steps to complete its work.
For-loops are a good indicator of the complexity is.  A double for-loop strongly suggests
$O(n^2)$ or worse.  The reason that array addition is $O(n^2)$ is because we have to add
$n^2$ elements in and $n \times n$ array.  Asymptotic analysis reveals that
if we have an array of size $n \times n$ and it takes $x$ seconds to execute a $O(n^2)$ algorithm,
then if we increase the size of the array to $2n \times 2n$, we should expect the same algorithm
to terminate in about $(2)^2 x = 4x$ seconds.  Sometimes, when you don't have code to look at,
you can determine the asymptotic growth of a function by timing it on various sized inputs.

\begin{problem}
Determine the asymptotic growth of the following code that calculates all the partial sums
up to 1000.
\begin{lstlisting}
t = 0
for n in xrange(1000):
    t += sum(xrange(n))
\end{lstlisting}
%the asymptotic growth is $O(n^2)$
\end{problem}

We can also classify an algorithm by the amount of memory it needs to execute.  We will use the term \emph{spatial complexity} to measure the amount of memory an algorithm uses.  The definition of spatial complexity is the same as the definition for temporal complexity with the obvious difference that $n$ refers to space rather than time.

\section*{Sparse Matrices}
A sparse matrix is a matrix that has relatively few nonzero elements.
SciPy has several different ways of storing sparse matrices.
Each way has it pros and cons (the reader is encouraged to read the help for each way).

\begin{table}[h!]
\centering
\begin{tabular}{|l|l|}
\hline
Function & Description \\
\hline
\li{sparse.bsr()} & Compressed Block Sparse Row\\
\li{sparse.coo()} & Coordinate\\
\li{sparse.csc()} & Compressed Sparse Column\\
\li{sparse.csr()} & Compress Sparse Row\\
\li{sparse.dia()} & Sparse Diagonal\\
\li{sparse.dok()} & Dictionary of Keys\\
\li{sparse.lil()} & Linked List\\
\hline
\end{tabular}
\caption{Sparse matrix representations in SciPy}
\end{table}
Type the following into IPython.
\begin{lstlisting}
: from scipy import sparse
: A = np.diagflat([2,3,4])
: B = sparse.csc_matrix(A)
: C = B.todense()
\end{lstlisting}
Notice that the matrix $A$ has only three non-zero entries, and so we can consider it sparse.
In memory, an array stores a bit of data (be it an integer, float, or complex number)
each entry, meaning that a $3 \times 3$ matrix requires a total 9 blocks of memory.
However, if we leverage the sparsity of $A$ we realize that we only need to store 3 numbers.
The \li{sparse} methods do exactly this: they store only the non-zero entries and their locations in the matrix.
No longer are we working with an array.  SciPy has many methods for performing operations on sparse matrices.
To convert back to a dense matrix, we use the \li{todense()} method of the sparse matrix.
We can also convert between the different types sparse matrices.

We remark that if you want to make a sparse diagonal matrix, the
best way to do it isn't to use \li{diagflat()} followed by \li{sparse},
it's actually better to use the \li{sparse.spdiags()} method:
\begin{lstlisting}
: sparse.spdiags([2,3,4],0,3,3)
\end{lstlisting}
Often, when we are using sparse matrices, we are dealing with matrices
that are too large to be handled efficiently when represented in full form.

\section*{Banded Matrices}
A banded matrix is one whose only non-zero entries are diagonal
strips.  For example, the matrix
\begin{equation*}
A = \begin{pmatrix}
1 & 2 & 0 & 0 \\
3 & 4 & 5 & 0 \\
0 & 6 & 7 & 8 \\
0 & 0 & 9 & 10
\end{pmatrix}
\end{equation*}
is banded because there are three nonzero diagonals.  This
particular type of banded matrix is called a tri-diagonal matrix.

You can easily create banded matrices using the \li{diagflat()} method.
For example, the matrix $A$ above can be created by entering
\begin{lstlisting}
: np.diagflat([3, 6, 9], -1) + np.diagflat([1, 4, 7, 10], 0) + np.diagflat([2, 5, 8], 1)
\end{lstlisting}

Often a better way to create a tri-diagonal is it use the \li{sparse.spdiags()} method.
This is because many diagonal matrices are sparse.
For example, we create the same matrix in Python (while designating that it is sparse) using the command:
\begin{lstlisting}
: Z = np.array([[3, 1, 0], [6, 4, 2], [9, 7, 5], [0, 10, 8]]).T
: sparse.spdiags(Z, [-1, 0, 1], 4, 4)
\end{lstlisting}

For more information, check the documentation by typing \li{spar.spdiags?}.
For example we create a tri-diagonal array with uniformly distributed random entries.
This example also demonstrates the efficiency of using sparse arrays.
\begin{lstlisting}
: B = np.rand(3,10000)
: A = sparse.spdiags(B, range(-1, 2), 10000, 10000)
: denseA = A.todense()  #only do this step if you have _lots_ of memory!
: A.data.nbytes
240000          #about 0.24MB of memory
: denseA.nbytes
800000000       #about 762.9MB of memory!
\end{lstlisting}
We can't use the \li{full} command in this case because the computer will almost
certainly run out of memory (the matrix is $10000 \times 10000$).
However, we can still visualize this matrix using the \li{plt.spy()} command from Matplotlib,
which essentially shows the location of non-zero entries in a matrix.
The output of \li{plt.spy(A)} in this case is shown in Figure \ref{fig:mpl_spy}.
\begin{figure}[h]
\centering
\includegraphics[width=\textwidth]{spy.pdf}
\caption{The output of the \li{spy()} command}
\label{fig:mpl_spy}
\end{figure}

\section*{Using Sparse Matrices}
Consider the linear system $A x = b$, where $A$ is a $100000\times 100000$ tri-diagonal matrix.
To store a full matrix of that size in your computer, it would normally require 10
billion double-precision floating-point numbers.  Since it takes 8
bytes to store a double, it would take roughly 80GB to store the
full matrix.  For most desktop computers, that fact alone makes the
system numerically prohibitive to solve.
The temporal complexity of the problem is even more problematic.
Methods for directly solving an arbitrary linear system are usually $O(n^3)$.
As a result, even if the computer could store an 80GB matrix in RAM, it
would still take several weeks to solve the system.  However, since
we don't have computers with that much available RAM, most of the
matrix would have to be stored on the hard drive, so the computation
would probably take between $6$ months to a year.

The point is that even the next generation of computers will
struggle with solving arbitrary linear systems of this size in a
reasonable period of time.  However, if we take advantage of the
sparse structure of the tri-diagonal matrix, we can solve the linear
system, even with a modest modern computer.  This is because all of
those zeros don't need to be stored and we don't need to do as many
operations to row reduce the tri-diagonal system.

Let's first compute the spatial complexity of the above system when
considered as a sparse matrix.  There are three diagonals that have
roughly $100000$ non-zero entries.  That's $300000$
double-precision floating point numbers, which is about 2.4 MB (Less
storage than your favorite song).  As a result, it will easily
fit into the computer's RAM.  Furthermore, the temporal complexity for solving
a tri-diagonal matrix is $O(n)$. Let's see how long it takes to
solve the system for random data:
\begin{lstlisting}
: from scipy.sparse import linalg as sparla
: D = sp.rand(3, 100000)
: b = sp.rand(1, 100000)
: A = spar.spdiags(D,[-1,0,1],100000,100000)
: def solSys():
:     return sparla.spsolve(A,b)
: %timeit solSys()
\end{lstlisting}

\begin{problem}
Write a function that returns a full $n\times n$
tri-diagonal array with $2$'s along the diagonal and $-1$'s along
the two sub-diagonals above and below the diagonal.
Hint: Use the \li{la.toeplitz()} method.
\label{prob:full_tridiag}
\end{problem}

\begin{problem}
Write another function that builds the same array as above, but as a sparse array.
You must build this as a sparse matrix from the beginning.
Hint: Use the \li{spar.spdiags()} method.
\label{prob:sparse_tridiag}
\end{problem}

\begin{problem}
Solve the linear system $Ax = b$ where $A$ is the $n\times n$
tri-diagonal array from Problem \ref{prob:full_tridiag} and Problem \ref{prob:sparse_tridiag} and $b$ is randomly
generated.  How high can you go for each method?  Make a table for
several different values of $n$ and the time it took to solve for
each run.  What conclusions can you draw?
\end{problem}

\begin{problem}
Using the sparse array above and the method \li{la.eigs()},
calculate the smallest eigenvalue $\lambda$ of the array as the array's size goes to infinity.
What value does $\lambda n^2$ approach?
Hint: It's the square of an important number.
This is related to operator theory: the second derivative operator has this eigenvalue in certain cases.
\end{problem}

\section*{Other Sparse Commands}
One important method of sparse matrix objects is the \li{nonzero()} method,
which is related to the number of nonzero entries in the matrix.
This number is important because it is an indicator of the amount of time and space
that is required to operate on the sparse matrix.
You should be aware that there is some overhead to using and storing the sparse matrix data structure.
Sparsely represented matrices are very beneficial when the number of
nonzero entries is relatively small compared to the total number of entries.
When the matrix has many nonzero entries, a sparse representation becomes disadvantageous.
To see this, create and execute a script with the following code:
\begin{lstlisting}
: A = np.random.rand(600,600); B = spar.csc_matrix(A)
: def square(A): return sp.power(A, 2)
: with timer() as t:
:     t.time(square, A)
:     t.time(square, B)
\end{lstlisting}
Run the script and note the two different runtimes.
Notice that it takes much longer to square the sparse matrix.
This is because the sparse matrix data structure is optimized for matrices that are actually sparse.
The array $A$ is entirely nonzero.
Thus, you incur the overhead of the sparse array representation without any
benefits since there are no entries you are not required to store or compute.
To summarize, only use a sparse matrix when your matrix is in fact sparse.
Using sparse matrices for mostly nonzero arrays will negatively impact performance and memory requirements.

Just as with dense arrays, we can pre-allocate sparse matrices.
Sometimes it is necessary to create sparse matrices that do not have a nice banded pattern.
We initialize a sparse matrix just like any other array.
The most efficient sparse matrix for pre-allocation is LIL.
Once you are done constructing you sparse matrix and wish to perform calculations,
you should convert to a more efficient sparse matrix (CSR or CSC).
\begin{lstlisting}
: Z = spar.lil_matrix((400,300))
<400x300 sparse matrix of type '<type 'numpy.float64'>'
        with 0 stored elements in LInked List format>
: Z[1,34] = 23
: Z[23,32] = 56
: Z[2,:] = 13.2
\end{lstlisting}
This code snippet creates a $400 \times 300$ LIL sparse matrix.
We can then work with the sparse matrix as though it were a dense array.
When the matrix is initialized all of the entries are assumed to be zero.