\section{Deep Neural Networks}

In this section, we will talk about some recent extension version of DNN models which
actually comes form the generalization of convolutional neural networks.

First, the so-called one hidden layer neural networks can be defined by
\begin{equation}
DNN_1=span\{\sigma(wx+b), w\in\mathbb{R}^n ,b\in\mathbb{R} \}
\end{equation}
with
$$
\theta(x) = wx+b \in\mathbb{R}^m, w=(w_i,\cdots,w_m)^T,\ w_i\in\mathbb{R}^{1\times n}
$$
and
\begin{equation*}
\theta(x) = 	wx+b=\left(\begin{array}{c}
		w_1x+b \\
		\vdots \\
		w_m x+b_m
	\end{array}
	\right)
\end{equation*}
Let $\sigma$ be the activation function, say Heaviside, sigmoid, $\tanh x$, ReLU. The deep neural network is defined as
\begin{equation*}
	\begin{aligned}
		&x^0 = x,\\
		&f^{n+1}=\sigma(w^n x^n+b^n)=(\sigma\circ\theta)(f^n)\\
		&DNN_i = \{w^i f^i +b^i \}
	\end{aligned}
\end{equation*}
Then we have
\begin{equation}
  \label{eq:1}
DNN_{i+1} = \{\theta(f^{i+1}) \}
=\{\theta(\sigma\circ\theta)(f^i) \}
=(\theta\circ\sigma) DNN_{i}
\end{equation}
and $\#$ of layers = $\#$ of $\sigma$ applied.

More general DNN:
\begin{equation}
	\begin{aligned}
		DNN_0&=\{wx+b\}\\
		DNN_J&=\{(\theta\circ\sigma)\circ f_{J-1}, \ f_{J-1}\in DNN_{J-1}\}
	\end{aligned}
\end{equation}
DNN contains ResNet, iResNet, DenseNet, MgNet.

The question is whether $DNN_k\subset DNN_{k+1}$ is true. The answer is that $DNN_k\subset DNN_{k+1}$ is only true for the case that the activation function is ReLU, since $x=ReLU(x)-ReLU(-x)$.

\chapter{The Sparse Grid Method}
It is well-known that a smooth function defined on $[0,1]^d$
can be pointwisely approximated with $O(h^2)$ accuracy by
a piecewise bilinear function in a subspace $V_h$ of dimension $O(h^{-d})$.
In the socalled sparse grid method proposed by
Zenger \cite{zenger1990sparse}, an $O(h^2|\log h|^{d-1})$ pointwise
accuracy can be achieved by using a  substantially smaller
subspace $S_h\subset V_h$ of dimension $O(h^{-1}|\log h|^{d-1})$.
As a result, a function $u$ on a general domain in $\R^d$
can be approximated with $O(h|\log h|^{d-1})$ in $H^1$ norm
with only $O(h^{-1}|\log h|^{d-1})$ number of operations.

\section{Multi-linear finite elements}

For simplicity, we take
$$
D_s=(0,1)^s,\ 1\leq s\leq d.
$$
For $D_1$, let the first grid $\mathcal{T}_0^1$ be itself, namely $\mathcal T_0^1=\{(0, 1)\}$. Divide each
element of $\mathcal{T}_k^1$ into two equal intervals and obtain the
next level grid $\mathcal{T}_{k+1}^1$ . For $s=1$, denote the basis function of nodal value interpolation to the vertex $x_{k,i}=ih_k$ by $\phi_{k,i}(x)$ with $0\leq i\leq n_k$.
\begin{figure}[!ht]
\begin{center}
\begin{tikzpicture}[xscale=4,yscale=4]
\draw[-] (0,0.9) -- (1,0.9);
\draw[-] (0,0) -- (1,0);
\draw[-] (0,0.3) -- (1,0.3);
\draw[-] (0,0.6) -- (1,0.6);
\fill(0,0.9) circle(0.5pt);
\fill(1,0.9) circle(0.5pt);

\fill(0,0.6) circle(0.5pt);
\fill(0.5,0.6) circle(0.5pt);
\fill(1,0.6) circle(0.5pt);

\fill(0,0.3) circle(0.5pt);
\fill(0.25,0.3) circle(0.5pt);
\fill(0.5,0.3) circle(0.5pt);
\fill(0.75,0.3) circle(0.5pt);
\fill(1,0.3) circle(0.5pt);

\fill(0,0) circle(0.5pt);
\fill(0.125,0) circle(0.5pt);
\fill(0.25,0) circle(0.5pt);
\fill(0.375,0) circle(0.5pt);
\fill(0.5,0) circle(0.5pt);
\fill(0.625,0) circle(0.5pt);
\fill(0.75,0) circle(0.5pt);
\fill(0.875,0) circle(0.5pt);
\fill(1,0) circle(0.5pt);

\node at (-0.15,0) {$\mathcal{T}_3^1$};
\node at (-0.15,0.3) {$\mathcal{T}_2^1$};
\node at (-0.15,0.6) {$\mathcal{T}_1^1$};
\node at (-0.15,0.9) {$\mathcal{T}_0^1$};
\node at (-0.05,0) {$0$};
\node at (-0.05,0.3) {$0$};
\node at (-0.05,0.6) {$0$};
\node at (-0.05,0.9) {$0$};
\node at (1.05,0.9) {$1$};
\node at (1.05,0) {$1$};
\node at (1.05,0.3) {$1$};
\node at (1.05,0.6) {$1$};
\end{tikzpicture}
\end{center}
\end{figure}

Consider a cubic grid $
\mathcal{T}_k^s$ of the domain $D_s$, which is a tensor product of
$\mathcal{T}_k^1$. The vertice of elements there are $(i_1h_k, \cdots
, i_s h_k)$, with $0\leq i_1, \cdots, i_s\leq n_k$, $n_k=h_k^{-1}$
and $h_k=2^k$. For $1\leq s\leq d$, let
$$
\textbf{i}=(i_1,\cdots,i_s) \text{ and } x_{k,\textbf{i}}=(i_1h_k,\cdots, i_sh_k)
$$
and $\psi_{k, \textbf{i}}^s (x_1,\cdots,x_s)$ be the basis function of the nodal value interpolation(bilinear interpolation) corresponding to the vertex $(i_1h_k, \cdots , i_s h_k)$ on the grid $\mathcal{V}_k$. Then
\begin{equation}
\psi_{k, \textbf{i}}^s(x)=\prod_{j=1}^s\phi_{k,i_j}^j(x_j),
\quad 0\le i_j\le n_k,\quad 1\le s\le d.
\end{equation}
Here we add a superscript $j$ to each basis function $\phi_{k,i_j}$ to represent that the corresponding function is the basis function with respect to $x_j$ of $x=(x_1,\cdots,x_s)$. In other words, the grid $\mathcal{T}_k^s$ is obtained by cutting each cubic of $\mathcal{T}_{k-1}^s$ into $2^s$ equal cubics. Then the number of elements in $\mathcal{T}_k^s$ is $(n_k)^s=2^{ks}$ and the number of interior nodes is $(n_k-1)^s=(2^k-1)^s$.

\begin{lemma}\Label{lm:1}
Assume that $\Pi_k^s: C(\bar D_s)\mapsto \mathcal{T}_k^s(D_s)$ is the nodal value interpolant on $\mathcal{T}_k$ and $I_k^s$ is the nodal value interpolant with respect to the variable $x_s$.  Then
%: C(\bar D_1)\mapsto \mathcal{T}_k^s(D_1)
$$\Pi_k^d u=\prod_{s=1}^d I^s_k u.$$
\end{lemma}
\begin{proof}
For each $1\leq s\leq d$, let
$$
\mathcal{I}_k^s=\{\textbf{i}=(i_1,\cdots,i_s),\ 0\leq i_1,\cdots ,i_s\leq n_k\}.
$$
\begin{equation}
\begin{split}
\Pi_k^d u&= \sum_{\textbf{i}\in \mathcal{I}_k^d} u(x_{k,\textbf{i}}) \psi_{k,\textbf{i}}^d\\
&= \sum_{\textbf{i}\in \mathcal{I}_k^d} u(x_{k,\textbf{i}})  \prod_{s=1}^d\phi_{k,i_s}^s(x_s)\\
&= \sum_{\textbf{i}\in \mathcal{I}_k^d}  \prod_{s=1}^{d-1}\phi_{i_s}^s(x_{s})\sum_{1\leq i_d\leq n_k-1} u(x_{k,\textbf{i}})\phi_{i_d}^d(x_d)\\
&=\sum_{\textbf{i}\in \mathcal{I}_k^{d-1}}  \prod_{s=1}^{d-1}\phi_{i_s}^s(x_{s})  (I_k^d u)(x_{k,\textbf{i}}) \\
&=(\prod_{s=1}^d I_k^s) u.
\end{split}
\end{equation}
\end{proof}
\input{3FEM/1DHBasis}

From this one dimensional hierarchical basis, a multi-dimensional discrete space on the $d$-dimensional unite cube $D_d$ is obtain by a tensor product construction:
\begin{equation}
\begin{aligned}
\m_J^d &= \otimes_{s=1}^d \m_J\\%\overbrace{\m_J\otimes \m_J\otimes \cdots \otimes \m_J}^{d}\\
&= \bigoplus_{k_1,k_2,\cdots k_d=0}^J  \otimes_{s=1}^d V_{k_s}^s%( \mathcal V_{k_1}^1 \otimes  \mathcal V_{k_2}^2\otimes\cdots \otimes  \mathcal V_{k_d}^d)\\
&= \bigoplus_{j=0}^{dJ}\bigoplus_{k_1+k_2+\cdots + k_d= j}  \otimes_{s=1}^d V_{k_s}^s.%( \mathcal V_{k_1}^1 \otimes  \mathcal V_{k_2}^2\otimes\cdots \otimes  \mathcal V_{k_d}^d).
\end{aligned}
\end{equation}
Here the superscript $i$ of $V_{k_i}^i$ represents that this is the subspace for the $i$-th coordinate space.
Let $\mathbf{k}=(k_1,k_2,\cdots,k_d)$ be a multi-index. Denote
$$
\mathcal V_{\mathbf{k}} = \mathcal V_{k_1}^1 \otimes  \mathcal V_{k_2}^2\otimes\cdots \otimes  \mathcal V_{k_d}^d.
$$
The discrete space $\m_J^d$ can be written as
\begin{equation}
\begin{aligned}
\m_J^d &= \bigoplus_{|\mathbf{k}|_\infty=J} \mathcal V_{\mathbf{k}} \\
&= \bigoplus_{j=0}^{dJ}\bigoplus_{|\mathbf{k}|_1=j} \mathcal V_{\mathbf{k}}.
\end{aligned}
\end{equation}
Let
$$
\mathbf{I_l}=\{\mathbf{i}\in \mathbb{N}^d: \mathbf{1}\le \mathbf{i}\le 2^\mathbf{l}-1, i_j \text{ is odd for all }1\le j\le d\},
$$
and the hierarchical basis
$$
\{\psi_\mathbf{k, i}^d: \mathbf{i}\in \mathbf{I_k}, \mathbf{k}\le \mathbf{l}\}\quad \mbox{with}\quad \psi_\mathbf{k, i}=\Pi_{s=1}^d\phi_{k_s,x_s}^s.
$$
For the $d$-dimensional hierarchical basis functions with $d\neq 2$, it is often more convenient to use the scaled HB
(hierarchical basis)
as follows
\begin{equation}
\label{shb}
\{\psi_\mathbf{k, i} \in {\cal V}_{\mathbf{k}}: \psi_\mathbf{k, i}=2^{-(|\mathbf{k}|_1+d)}\psi_{ \textbf{k, i}}^d, |\mathbf{k}|_\infty\le J\}.
\end{equation}
With a proper ordering, we shall denote the scaled HB by $\{\psi_i,
i=1:N\}$. The HB in multiple dimensions is formally a direct generalization of the
HB in the one dimensional case.  But the property for the corresponding
stiffness matrix in multiple dimensions is not at all as clear as in one
dimension where the stiffness matrix is an identity matrix in some
special cases.  In this section, we shall show that at least in two
dimensions, a hierarchical basis is still very useful.

For the linear finite element,
$
\|u-u_I\|_0\lesssim h^2|u|_2.
$
Since $h\approx N^{-1/d}$, where $N$ is the number of the total degree of freedom, we have
$$
\|u-u_I\|_0\lesssim N^{-2/d}|u|_2.
$$
It implies that for a fixed number of degrees of freedom $N$, a large dimension results in pretty low accuracy.

In order to overcome the curse of dimensionality, we intend to choose a small subspace properly and throw away a large number of useless information. In the following, we introduce an interpolation operator $R_h$ which interpolates functions into finite element spaces on coarse meshes. Denote the range of $R_k^d$ by $S_k^d$, which is the desirable subspace.

Next we consider the approximation property and dimension  of the following subspace:
$$
S_{J,r}^d = \bigoplus_{|\mathbf{k}|_1 \le r}\mathcal V_{\mathbf{k}}.
$$
Denote
$$
\mathcal F_{J,r}=\{\mathbf k=(k_1,\cdots,k_d): 0\leq k_i\leq J, \ |\mathbf k|_1\leq r\},
$$
$$
\mathcal F_{J,r}^c=\{\mathbf k=(k_1,\cdots,k_d): 0\leq k_i\leq J, \ |\mathbf k|_1> r\},
$$
$$
\mathcal F_{J,r}^o=\{\mathbf k=(k_1,\cdots,k_d): 0\leq k_i\leq J, \ |\mathbf k|_1= r\}.
$$

%Next, we discuss the number $2^{-(2-m)r}\#\  \mathcal F_{J,r}^o$.
%\begin{enumerate}
%\item $r\leq J$: $\# \ \mathcal F_{J,r}^o=\#\ \{\mathbf k:  |\mathbf k|_1= r\}=C_{r+d-1}^r$. In this case, $r$ is usually small, we have
%$$
%\# \ \mathcal F_{J,r}^o= \frac{d(d+1)\cdots (d+r-1)}{r!}\approx d^r,\ \text{ and }2^{-(2-m)r}\#\  \mathcal F_{J,r}^o\approx 2^{-(2-m)r}d^r;
%$$
%\item $J<r\leq Jd$: For 2 dimensional case, let $k_i=a_i^2$, $r=r_0^2$. Then
%$$
%\#\  \mathcal F_{J,r}^o=\# \{\mathbf{a}=(a_1,\cdots,a_d): \sum_{i=1}^da_i^2=r_0^2,\ 0\leq a_i\leq \sqrt{J}\}.
%$$
%We consider the length $l_\theta=r_0(2 \theta-\frac{\pi}{2})$ with $r_0\sin \theta=\sqrt{J}$. Then
%$$
%l_\theta=\frac{\sqrt{J}}{\sin \theta}(2 \theta-\frac{\pi}{2}).
%$$
%A simple computation gives
%$$
%\sin \theta l_\theta'+\cos \theta l_\theta=2\sqrt{J},
%$$
%thus,
%$$
%\sin \theta l_\theta=\frac{\pi\sqrt{J}}{2} -2\sqrt{J}(\frac{\pi}{2}-\theta)=2\sqrt{J}(\theta-\frac{\pi}{4}),
%$$
%namely $l_\theta= 2\sqrt{J}(\theta-\frac{\pi}{4})/\sin \theta$. As $r$ grows larger, $l_\theta$ approximates $2\sqrt{J}$.
%
%\item $r> Jd$: $\# \ \mathcal F_{J,r}^o=0$.
%\end{enumerate}

\begin{lemma}\Label{lm:2}
Assume that
$I^s_k: C(\bar D)\mapsto \m_l(D_1)$ is the nodal value interpolant
with respect to the variable $x_s$.  Define
$$
F_{J,r}^d=\sum_{\mathbf k\in \mathcal{F}_{J,r}}\prod_{s=1}^d(I^s_{k_s}-I^s_{k_{s}-1})
$$
and
$$
S_{J,r}^d=F_{J,r}^d\m.
$$
Then for $m=0,1$,
$$
\inf_{\chi\in S_{J,r}^d}\nm{v-\chi}{m, D_d}\le C h_J^{(2-m)(r+1)/J}|\log h_J|^{d-1}\max_{|\mathbf k|_1> r}\nm {\partial^{2\text{sign}(\mathbf k)}v}{0,D_d}.
$$
\end{lemma}
\begin{proof}
Using the obvious identity (with $I_{-1}^s=0$)
$$I^s_J=\sum_{k=0}^J(I^s_k-I^s_{k-1}),$$
it follows from the commutative property and Lemma \ref{lm:1} that
$$
\Pi_J^d=\prod_{s=1}^d\sum_{k=0}^J(I^s_{k}-I^s_{k-1})
=\sum_{k_1,k_2,\cdots, k_d=0}^J\prod_{s=1}^d(I^s_{k_s}-I^s_{k_s-1}).
$$
Thus,
\begin{equation}\label{multiNote1}
(\Pi_J^d-F_{J,r}^d)v=
\sum_{\mathbf k\in \mathcal{F}_{J,r}^c}\prod_{s=1}^d(I^s_{k_s}-I^s_{k_s-1})v.
\end{equation}
Note that if $k_1\neq 0$ and $m\le 2$,
\begin{equation}\label{multiNote201}
\begin{split}
\nm { \prod_{s=1}^d(I^s_{k_s}-I^s_{k_s-1})v}{m,D_d}
&\lesssim h^{2-m}_{k_1}\nm {\partial^2_{x_1}\prod_{s=2}^d(I^s_{k_s}-I^s_{k_s-1})v}{0, D_d} \\
&=  h^{2-m}_{k_1}\nm {\prod_{s=2}^d(I^s_{k_s}-I^s_{k_s-1}) \partial^2_{x_1} v}{0, D_d} \\
&=2^{-k_1(2-m)}\nm {\prod_{s=2}^d(I^s_{k_s}-I^s_{k_s-1}) \partial^2_{x_1} v}{0, D_d}.
%&\lesssim  h^{2-m}_{k_1}h^{2-m}_{k_2}\nm {\prod_{s=3}^d(I^s_{k_s}-I^s_{k_s-1}) \partial^2_{x_1} v}{L^{p}(D_d)} \\
%&\le Ch^{2-m}_{k_1}h^{2-m}_{k_2}\cdots h^{2-m}_{k_d}
%\nm{\partial^2_{x_1}\partial^2_{x_2}\cdots \partial^2_{x_d}v}{L^p(D_d)}.
\end{split}
\end{equation}
If $k_1=0$ and $m\le 2$,
\begin{equation}\label{multiNote202}
\begin{split}
\nm { \prod_{s=1}^d(I^s_{k_s}-I^s_{k_s-1})v}{m,D_d}
&= \nm { I_0^1 \prod_{s=2}^d(I^s_{k_s}-I^s_{k_s-1})v}{m,D_d} \\
&=2^{-k_1(2-m)}\nm {\prod_{s=2}^d(I^s_{k_s}-I^s_{k_s-1})  I_0^1  v}{m,D_d}.
\end{split}
\end{equation}
A combination of \eqref{multiNote201} and \eqref{multiNote202} gives
\begin{equation}\label{multiNote2}
\nm {\prod_{s=1}^d(I^s_{k_s}-I^s_{k_s-1})v}{0, D_d}
\lesssim 2^{-(2-m)|\mathbf k |_1}\nm {\partial^{2\text{sign}(\mathbf k)}v}{0,D_d}.
\end{equation}
An elementary calculation shows that
\begin{equation}\label{multiNote3}
\begin{split}
\sum_{\mathbf k\in \mathcal{F}_{J,r}^c}2^{-(2-m)|\mathbf k |_1}\nm {\partial^{2\text{sign}(\mathbf k)}v}{0,D_d}
&=\sum_{j=r+1}^{Jd}\sum_{\mathbf k \in \mathcal F_{J,j}^o}2^{-(2-m)|\mathbf k |_1}\nm {\partial^{2\text{sign}(\mathbf k)}v}{0,D_d}\\
&\leq\sum_{j=r+1}^{Jd} 2^{-(2-m)j}  \max_{\mathbf k \in \mathcal F_{J,j}^o}\nm {\partial^{2\text{sign}(\mathbf k)}v}{0,D_d}\sum_{\mathbf k \in \mathcal F_{J,j}^o}1\\
&\le J^{d-1}\sum_{j=r+1}^{Jd} 2^{-(2-m)j} \max_{\mathbf k\in \mathcal{F}_{J,r}^c}\nm {\partial^{2\text{sign}(\mathbf k)}v}{0,D_d}\\
&=\frac{1-2^{-(2-m)(Jd-r-1)}}{1-2^{-(2-m)}} J^{d-1}2^{-(r+1)(2-m)}\max_{\mathbf k\in \mathcal{F}_{J,r}^c}\nm {\partial^{2\text{sign}(\mathbf k)}v}{0,D_d}\\
&\le C h_J^{(2-m)(r+1)/J}|\log h_J|^{d-1}\max_{\mathbf k\in \mathcal{F}_{J,r}^c}\nm {\partial^{2\text{sign}(\mathbf k)}v}{0,D_d}.
\end{split}
\end{equation}
%Given an integer $p\ge1$, an elementary calculation shows that
%\begin{equation}\label{multiNote2}
%\begin{split}
%\sum_{k_1+k_2+\cdots+ k_d\ge p}h^{2}_{k_1}h^{2}_{k_2}\cdots h^{2}_{k_d}\\
%&=\sum_{k_1+k_2+\cdots+ k_d\ge p}2^{-(k_1+k_2+\cdots+k_d)}\\
%&=\sum_{j=p}^{Jd}\sum_{k_1+k_2+\cdots+ k_d= j}2^{-(k_1+k_2+\cdots+k_d)}\\
%&=\sum_{j=p}^{Jd}\sum_{k_1+k_2+\cdots+ k_d= j}2^{-j}\\
%&=\frac{1-h^{(2)(d-1)}}{1-\gamma^{2(2)}} J^{d-1}\gamma^{2J(2)}\le C h^{2}|\log h|^{d-1}.
%\end{split}
%\end{equation}
A combination of \eqref{multiNote1}, \eqref{multiNote2} and \eqref{multiNote3} yields
\begin{equation}
\begin{split}
\| v-F_{J,r}^d v\|_{m, D_d}&\leq \| v-\Pi_k^d v\|_{m,D_d} + \| \Pi_k^d v-F_{J,r}^d v\|_{m,D_d}
\\
&\le C h_J^{(2-m)(r+1)/J}|\log h_J|^{d-1}\max_{|\mathbf k|_1> r}\nm {\partial^{2\text{sign}(\mathbf k)}v}{0,D_d}.
\end{split}
\end{equation}
which completes the proof.
\end{proof}
\begin{remark}
Let $ \lceil p/2\rceil$ be the smallest integer larger than $p/2$. For any positive integer $p$, if $r\ge \lceil p/2\rceil J$, there are at least $\lceil p/2\rceil$ components of $\mathbf{k}$ which are nonzero, namely, $2\text{sign} (\mathbf{k})> p$. Thus, for any polynomial $v$ with degree not larger than $p$, $\max_{\mathbf k\in \mathcal{F}_{J,r}^c}\nm {\partial^{2\text{sign}(\mathbf k)}v}{0,D_d}=0.$
\end{remark}

%\newcommand{S_h}{{\cal S}_h}
\begin{lemma}\label{lm:sparseN}
By the definition of $S_{J,r}^d$, we have
$$
S_{J,r}^d={\rm span}\{\prod_{s=1}^d\phi_{k_s, i_s}^s,\quad
|\mathbf{k}|_1< r,\quad i_sh_{k_s}\in {\cal N}_{k_s}\setminus {\cal N}_{k_s-1}\},
$$
and
$$
\dim(S_{J,r}^d)=O(h_J^{-(r+1)/J}|\log h_J|^{d-1}).
$$
\end{lemma}
\begin{proof}
It is easy to see that
\begin{equation*}
\begin{split}
\dim(S_{J,r}^d) &\le C\sum_{|\mathbf k|_1\leq r}2^{k_1+k_2+\cdots+k_d}\\
&= C \sum_{j=0}^r 2^{j}\# \mathcal F_{J, j}^o\\
&\leq 2^{r+1}J^{d-1}\\
&\leq Ch_J^{-(r+1)/J}|\log h_J|^{d-1}.
\end{split}
\end{equation*}
\end{proof}

A combination of Lemma \ref{lm:2} and Lemma \ref{lm:sparseN} leads to the following result.
\begin{theorem}
$$
\inf_{\chi\in S_{J,r}^d}\nm{v-\chi}{m, D_d}\le C N^{-(2-m)}|\log N|^{d-1}\max_{|\mathbf k|_1> r}\nm {\partial^{2\text{sign}(\mathbf k)}v}{0,D_d},
$$
where $N$ is the number of the degree of freedom.
\end{theorem}
This indicates that the sparse grid method overcomes the curse of dimensionality.

\input{6DL/SparseGridModify}

\section{The sparse grid method using high order polynomials}
In this section, we consider the high order method. Let $\mathcal{M}_k, k =0,1,\cdots $ denote the one-dimensional Lagrange finite element spaces of order $p$ on the grids $\mathcal{T}_k$.  Let $I_k$ be the Lagrange interpolant of order $p$. Thus,
$$
\mathcal{M}_k = I_k\m.
$$
Note that the number of elements on  $\mathcal{T}_k, k = 0,1,\cdots$ is $2^k$. Thus, the number of degrees of freedom of the space $\mathcal{M}_k$ is
$$
\text{Dim}(\mathcal{M}_k) = p2^k+1 \quad \text{ for } k\ge 1,
$$
and $\text{Dim}(\mathcal{M}_0)=p+1$.
Setting $\mathcal{V}_0= \mathcal{M}_0$ and
$$
\mathcal{V}_k=(I_k - I_{k-1}) \mathcal{M}_k  \quad \text{ for } k\ge 1.
$$
Thus, the dimension of the spaces is
$$
\text{Dim}(\mathcal{V}_k) = p2^{k-1} \quad \text{ for } k\ge 1,
$$
and $\text{Dim}(\mathcal{V}_0)=p+1$. It is easy to see that
\begin{equation}\label{hbk}
\mathcal V_k=\{\phi_{k,i}: x_{k,i}\in {\cal N}_k\setminus {\cal N}_{k-1}\},
\end{equation}
where ${\cal N}_k$ denotes the set of nodes of Lagrange finite element space of order $k$.  The above subspaces obviously give rise to a direct sum decomposition of the space $\m_J$ as follows:
$$
        \mathcal M_\infty=\bigoplus_{k=0}^\infty \mathcal V_k.
$$

Consider the $d$-dimensional unite cube $D_d$, let $\Pi_k^s: C(\bar D_s)\mapsto \mathcal{T}_k^s(D_s)$ is the Lagrange interpolant on $\mathcal{T}_k$ of order $p$.  A multi-dimensional basis on the $d$-dimensional unite cube $D_d$ is obtain by a tensor product construction:
\begin{equation}
\begin{aligned}
\m^d &= \overbrace{\m \otimes \m \otimes \cdots \otimes \m }^{d}\\
&= \bigoplus_{k_1,k_2,\cdots k_d=0}^\infty ( \mathcal V_{k_1}^1 \otimes  \mathcal V_{k_2}^2\otimes\cdots \otimes  \mathcal V_{k_d}^d)\\
&= \bigoplus_{j=0}^{\infty}\bigoplus_{k_1+k_2+\cdots + k_d= j} ( \mathcal V_{k_1}^1 \otimes  \mathcal V_{k_2}^2\otimes\cdots \otimes  \mathcal V_{k_d}^d)\\
\end{aligned}
\end{equation}
Let $\mathbf{k}=(k_1,k_2,\cdots,k_d)$ be a multi-index. Denote
$
\mathcal V_{\mathbf{k}} = \mathcal V_{k_1}^1 \otimes  \mathcal V_{k_2}^2\otimes\cdots \otimes  \mathcal V_{k_d}^d.
$
We make the following truncation:
$$
\mathcal{M}^d = \left(\bigoplus_{|\mathbf{k}|_1 < L}\mathcal{V}_{\mathbf{k}} \right) \bigoplus \left(\bigoplus_{|\mathbf{k}|_1 \ge L}\mathcal{V}_{\mathbf{k}} \right).
$$
Next we consider the approximation property and dimension  of  $\mathcal{M}^{d,L,p} : =\bigoplus_{|\mathbf{k}|_1 < L}\mathcal{V}_{\mathbf{k}}$.

\begin{lemma}
It holds that
\begin{equation}
\text{Dim}(\mathcal{M}^{d,L,p}) =\mathcal{O}\left( (p+1)^d L^{-1}(L+2)^d 2^{L-d}\right).
\end{equation}
\end{lemma}
\begin{proof}
\begin{equation}
\begin{aligned}
\text{Dim}(\mathcal{M}^{d,L,p})& = \text{Dim}(\bigoplus_{|\mathbf{k}|_1 < L} \mathcal V_{k_1}^1 \otimes  \mathcal V_{k_2}^2\otimes\cdots \otimes  \mathcal V_{k_d}^d )\\
& = \sum_{i=0}^d C_d^i (p+1)^i \sum_{k_1+\cdots +k_{d-i} <L}^{k_j \ge 1} p^{d-i}2^{k_1+\cdots+k_{d-i} -(d-i)} \\
& = \sum_{i=0}^d C_d^i (p+1)^i \sum_{k_1+\cdots +k_{d-i} <L-(d-i)}^{k_j \ge 0} p^{d-i}2^{k_1+\cdots+k_{d-i} } \\
& \le \sum_{i=0}^d  C_d^i    (p+1)^i p^{d-i} (L-d+i)^{d-i-1}2^{L-d+i}\\
&\le (p+1)^d L^{-1}2^{L-d} \sum_{i=0}^d C_d^i L^{d-i} 2^i\\
&\le (p+1)^d L^{-1}(L+2)^d 2^{L-d}.
\end{aligned}
\end{equation}
In the first inequality of above estimation, we use the fact that
$$
\sum_{k_1+\cdots+k_d < L }2^{k_1+\cdots +k_d} = \sum_{j=0}^{L-1} \sum_{k_1+\cdots+k_d =j}2^j =  \sum_{j=0}^{L-1} C_{j+d-1}^{d-1}2^j,
$$
and
$$
\frac{(L-1+d-1)!}{(d-1)!(L-1)!} 2^{L-1} < \sum_{j=0}^{L-1} C_{j+d-1}^{d-1}2^j <  \sum_{j=0}^{L-1} L^{d-1}2^j =L^{d-1}2^L.
$$
Thus,
$$
\sum_{k_1+\cdots+k_d < L }2^{k_1+\cdots +k_d} \approx  L^{d-1}2^L.
$$

\end{proof}

\begin{lemma}\label{lm:spare_grid_decay}
It holds that
\begin{equation}
\|\Pi_{i=1}^d (I_{k_i}^i -I_{k_i-1}^i) v\|_0  \le C^{d} 2^{-(p+1)|\mathbf{k}|_1}\|\partial^{(p+1)\text{sign}(\mathbf{k})}v\|_0.
\end{equation}
Here $I_{k_i}^i$ is the Lagrange interpolant onto $\m_{k_i}$ with respect to the $i$-th variable, and $C$ is a constant independent of $p,d,\mathbf{k}$.
\end{lemma}
\begin{proof}
Similar to the analysis in Lemma \ref{lm:2},
\begin{equation}
\begin{aligned}
\|\Pi_{i=1}^d (I_{k_i}^i -I_{k_i-1}^i) v\|_0  &\le Ch_{k_i}^{p+1} \|\Pi_{i=2}^d (I_{k_i}^i -I_{k_i-1}^i)  \partial^{(p+1)\text{sign}(k_1)}v \|_0\\
&\le C^{d} 2^{-(p+1)|\mathbf{k}|_1}\|\partial^{(p+1)\text{sign}(\mathbf{k})}v\|_0.
\end{aligned}
\end{equation}
\end{proof}

For any $v$, we denote $v_{\mathbf{k}} = \Pi_{i=1}^d (I_{k_i}^i -I_{k_i-1}^i) v$. It is obvious that
$$
v \approx \sum_{\mathbf{k} } v_{\mathbf{k}}.
$$
The above lemma indicates that  $v_{\mathbf{k}}$ decays as  $C^{d} 2^{-(p+1)|\mathbf{k}|_1}$.

\begin{lemma}
Setting  $v_{\mathbf{k}} = \Pi_{i=1}^d (I_{k_i}^i -I_{k_i-1}^i) v$. It hold that
\begin{equation}
\|\sum_{|\mathbf{k}|_1 \ge L } v_{\mathbf{k}}\|_0 \le C^d 2^{-(p+1)L + \frac{1}{2^{p+1}\ln 2}d}\sum_{i=0}^{d-1}\frac{\left((1-\frac{1}{2^{p+1}})L\right)^{i}}{i!} \max_{|\mathbf{k}|_1\ge L}\|\partial^{(p+1)\text{sign}(\mathbf{k})} v\|_0
\end{equation}
\end{lemma}
\begin{proof}
By Lemma \ref{lm:spare_grid_decay}, we have
\begin{equation}\label{pest}
\|\sum_{|\mathbf{k}|_1 \ge L } v_{\mathbf{k}}\|_0 \le C^d \max_{|\mathbf{k}|_1\ge L }\|\partial^{(p+1)\text{sign}(\mathbf{k})} v\|_0  \sum_{|\mathbf{k}|_1 \ge L} 2^{-(p+1)|\mathbf{k}|_1}.
\end{equation}
Setting
$
s_{d, L} = \sum_{|\mathbf{k}|_1 \ge L} 2^{-(p+1)|\mathbf{k}|_1}.
$
Note that
$$
\begin{aligned}
s_{d, L} &= \sum_{|\mathbf{k}|_1 \ge L} 2^{-(p+1)|\mathbf{k}|_1}\\
&= \sum_{k_2+\cdots+k_d\ge L-1, k_1\neq 0} 2^{-(p+1) (k_1+1 +k_2+\cdots+k_d)}  + \sum_{k_2+\cdots+k_d \ge L, k_1=0} 2^{-(p+1) ( 0+k_2+\cdots+k_d)}\\
& =  2^{-(p+1)} s_{d,L-1} +s_{d-1,L}
\end{aligned}
$$
and $s_{1,L} = \frac{2^{-(p+1)L}}{1 - 2^{-p-1}}$, $s_{d,0} = (\frac{1}{1 - 2^{-p-1}})^d$.
Let $t_{d,L} = 2^{(p+1)L} s_{d,L}$, $\gamma = \frac{1}{1 - 2^{-p-1}}$. We have the following induction
$$
t_{d,L} =t_{d,L-1} +t_{d-1,L}\quad \text{with}\quad  t_{1,L} = \gamma, t_{d,0} = \gamma^d.
$$
By induction, we have
$$
t_{d,L}\lesssim \gamma^d(1 + \frac{L}{\gamma } +\cdots +\frac{ L^{d-1} }{(d-1)!\gamma^{d-1}}).
$$
Thus,
$$
s_{d,L} \lesssim 2^{-(p+1)L} (1-2^{-p-1})^{-d} (1 +\frac{(1-2^{-p-1})L}{1}+\cdots + \frac{ \left((1-2^{-p-1}) L\right)^{d-1} }{(d-1)!}  )
$$
Since $-\ln(1-x)\le x$, we have
$$
(1 - 2^{-p-1})^{-d} = 2^{-d \ln (1-2^{-p-1}) /\ln 2} \le 2^{ \frac{1}{2^{p+1}\ln 2}d}.
$$
A combination of this and \eqref{pest} completes  the proof.
\end{proof}


\documentclass[11pt]{article}
\renewcommand{\baselinestretch}{1.05}
\usepackage{amsmath,amsthm,verbatim,amssymb,amsfonts,amscd, graphicx}
\usepackage{graphics}
\topmargin0.0cm
\headheight0.0cm
\headsep0.0cm
\oddsidemargin0.0cm
\textheight23.0cm
\textwidth16.5cm
\footskip1.0cm

\begin{document}

\title{Assignment 3 Solution}
\author{Rochan Avlur Venkat}
\maketitle

\section{}
Given that $$f(x,y) = \frac{x^{2}y^{2}}{x^{2}y^{2} + (x-y)^{2}}$$

whenever $x^{2}y^{2} + (x - y)^{2} \neq 0$. Now, applying the first limit, we get

$$\lim_{y\to0}f = \frac{x^{2}y^{2}}{x^{2}y^{2} + (x-y)^{2}} = \frac{0}{0 + (x)^{2}} = 0$$

Now, applying the outer limit we get

$$\lim_{x\to0}\lim_{y\to0}f = 0$$

Similarly, we get

$$\lim_{x\to0}f = \frac{x^{2}y^{2}}{x^{2}y^{2} + (x-y)^{2}} = \frac{0}{0 + (y)^{2}} = 0$$

And applying the outer limit we get

$$\lim_{y\to0}\lim_{x\to0}f = 0$$

Hence, $$\lim_{y\to0}\lim_{x\to0}f = \lim_{x\to0}\lim_{y\to0}f$$

Now, $\lim_{(x,y)\to(0,0)}f(x,y)$ along the $y=x$ line, we get

$$\lim_{(x,y)\to(0,0)}f = \frac{x^{4}}{x^{4}} = 1$$

Similarly, $\lim_{(x,y)\to(0,0)}f(x,y)$ along the $y=2x$ line, we get

$$\lim_{(x,y)\to(0,0)}f = \frac{4x^{4}}{4x^{4} + x^{2}} = \frac{4x^{2}}{4x^{2} + 1} = 0$$

Hence, limit does not exits at $(x,y) \to (0,0)$.

\section{}

See appended section.

\section{}
Given that $$f(x,y) = \frac{x^{2} - y^{2}}{x^{2} + y^{2}}$$
\\
We need to find the limit along $y = mx$ as $(x,y) \rightarrow (0,0)$
\\
\\
Replacing the value of $y$ with $mx$ in the given function, we get $$f = \frac{x^{2} - m^{2}x^{2}}{x^{2} + m^{2}x^{2}}$$
\\
Solving the limit, we get $$\lim_{x\to0} \frac{x^{2} - m^{2}x^{2}}{x^{2} + m^{2}x^{2}} = \frac{1 - m^{2}}{1 + m^{2}}$$
\\
In order to define a $f(x,y)$ so as to make it continuous at $(0,0)$, $$\lim_{(x,y)\to(0,0)} \frac{x^{2} - y^{2}}{x^{2} + y^{2}}$$ should be equal along every path $y = mx + c$ taken. Since this is not true in the given function, we cannot define $f(x,y)$ so as to make it continuous at $(0,0)$.

\section{}
Given a scalar field
$$f(\textbf{x}) = ||\textbf{x}||^{4}$$

Now, assume that

$$g(t) = f(\textbf{x} + t\textbf{y})$$

Since

$$f(\textbf{x}) = (\textbf{x}\cdot\textbf{x})\times(\textbf{x}\cdot\textbf{x})$$

We get

$$g(t) = (\textbf{x} + t\textbf{y})\cdot(\textbf{x} + t\textbf{y})\times(\textbf{x} + t\textbf{y})\cdot(\textbf{x} + t\textbf{y})$$

$$g(t) = (\textbf{x}\cdot\textbf{x}+2t\textbf{x}\cdot\textbf{y}+t^{2}\textbf{y}\cdot\textbf{y})\times(\textbf{x}\cdot\textbf{x}+2t\textbf{x}\cdot\textbf{y}+t^{2}\textbf{y}\cdot\textbf{y})$$

$$g\prime(0) = f\prime(\textbf{x};\textbf{y}) = 4||\textbf{x}||^{2}(\textbf{x}\cdot\textbf{y})$$

\section{}
\subsection{}

Given that

$$f(x,y) = \frac{x}{\sqrt{x^{2} + y^{2}}}$$

Then the first order partial derivative can be calculated as

$$\frac{\partial f}{\partial x} = \frac{\partial \frac{x}{\sqrt{x^{2} + y^{2}}}}{\partial x}$$

Using division rule, we get

$$\frac{\partial f}{\partial x} = \frac{\frac{\partial \:}{\partial \:x}\left(x\right)\sqrt{x^2+y^2}-\frac{\partial \:}{\partial \:x}\left(\sqrt{x^2+y^2}\right)x}{\left(\sqrt{x^2+y^2}\right)^2}$$

$$\frac{\partial f}{\partial x} = \frac{1\cdot \sqrt{x^2+y^2}-\frac{x}{\sqrt{x^2+y^2}}x}{\left(\sqrt{x^2+y^2}\right)^2} = \frac{y^2}{\left(x^2+y^2\right)\sqrt{x^2+y^2}}$$

The partial derivative with respect to y is

$$\frac{\partial f}{\partial y} = \frac{\partial \frac{x}{\sqrt{x^{2} + y^{2}}}}{\partial y}$$

$$\frac{\partial f}{\partial y} = x\frac{\partial \:}{\partial \:y}\left(\left(x^2+y^2\right)^{-\frac{1}{2}}\right)$$

And using chain rule, replacing $u = x^{2} + y^{2}$, we get

$$\frac{\partial f}{\partial y} = x\frac{\partial \:}{\partial \:u}\left(u^{-\frac{1}{2}}\right)\frac{\partial \:}{\partial \:y}\left(x^2+y^2\right)$$

$$\frac{\partial f}{\partial y} = x\left(-\frac{1}{2u^{\frac{3}{2}}}\right)\cdot \:2y$$

Replacing the back the value of $u$, we get

$$\frac{\partial f}{\partial y} = -\frac{xy}{\left(x^2+y^2\right)^{\frac{3}{2}}}$$

\subsection{}

Given $$f(x) = \vec{a}.\vec{x}$$

$\vec{a}$ being fixed, $f$ is defined on $R^{n}$ and $\vec{a} = a_{1}i + a_{2}j + ...$

Then, the partial derivative in the $x$ direction is

$$\frac{\partial f}{\partial x} = \lim_{h\to0}\frac{f((x,y,z...) + h(1,0,0,0...)) - f(x,y,z...)}{h}$$

$$= \lim_{h\to0}\frac{\vec{a}\cdot(x+h,y,z..) - \vec{a}\cdot(x,y,z...)}{h}$$

$$= a_{1}$$

Then, the partial derivative in the $y$ direction is

$$\frac{\partial f}{\partial y} = \lim_{h\to0}\frac{f((x,y,z...) + h(0,1,0,0...)) - f(x,y,z...)}{h}$$

$$= \lim_{h\to0}\frac{\vec{a}\cdot(x,y+h,z..) - \vec{a}\cdot(x,y,z...)}{h}$$

$$= a_{2}$$

And so on.

\section{}

Given the function $$f(x,y) = \frac{1}{y}\cos{x^{2}}$$

We have $$D_{2}f = \frac{\partial \frac{1}{y}\cos{x^{2}}}{\partial y} = \frac{-1}{y^{2}}\cos{x^{2}}$$ $$D_{1}f = \frac{\partial \frac{1}{y}\cos{x^{2}}}{\partial x} = \frac{-2x}{y}\sin{x^{2}}$$

Then, the mixed partial derivatives $D_{1}(D_{2}f)$ and $D_{2}(D_{1}f)$ are given by
$$D_{1}(D_{2}f) = \frac{2x}{y^{2}}\sin{x^{2}}$$

$$D_{2}(D_{1}f) = \frac{2x}{y^{2}}\sin{x^{2}}$$

Since $D_{1}(D_{2}f) = D_{2}(D_{1}f)$ for all values of $(x,y)$
\\
Hence, Proved

\section{}

Given the scalar field $$f(x,y,z) = x^{2} + 2y^{2} + 3z^{2}$$

and the unit vector $\frac{i - j + 2k}{\sqrt{6}}$

The directional derivative at $(1, 1, 0)$ in the direction of $\vec{v} = i - j + 2k$ equals to

$$DD = f((1,1,0);(\frac{1}{\sqrt{6}}, \frac{-1}{\sqrt{6}}, \frac{2}{\sqrt{6}})) = \lim_{h\to0}\frac{f((1,1,0) + h(\frac{1}{\sqrt{6}}, \frac{-1}{\sqrt{6}}, \frac{2}{\sqrt{6}})) - f(1,1,0)}{h}$$

$$= \lim_{h\to0}\frac{\frac{(h + \sqrt{6})^{2}}{6} + \frac{(-h + \sqrt{6})^{2}}{6} + \frac{12h^{2}}{6} - 3}{h}$$

Applying L' Hopitals rule, we get

$$\lim_{h\to0}\frac{15h^{2} - 2\sqrt{6}h}{6h} = \frac{-2}{\sqrt{6}}$$

\section{}

Given the scalar field $$f(x,y,z) = axy^{2} + byz + cz^{2}x^{3}$$
  has a maximum value of 64 in a direction parallel
to the z -axis.

The directional derivative at $(1, 2, −1)$ in the direction parallel to the $z$ axis being $\vec{v} = k$ equals to

$${\Big(\frac{\partial f}{\partial x}i + \frac{\partial f}{\partial y}j + \frac{\partial f}{\partial z}k}\Big)\cdot\big({k}\big)$$

Replacing values, we get

$$\Big((ay^{2} + 3cz^{2}x{2})i + (2axy + bz)j + (by + 2czx^{3})k\Big)\cdot\big({k}\big)$$

where $(x,y,z) \rightarrow (0,0,1)$, we get

$$\Big((4a + 3c)i + (4a - b)j + (2b - 2c)k\Big)\cdot\big({k}\big) = \big(\frac{2b - 2c}{\sqrt{1}}\big) = 64$$

Hence, we get

$$b - c = 32$$
\\
\\
If maximum occurs along a direction, then the minimum occurs along a direction perpendicular to it.

$$4a + 3c = 0$$
$$4a - b = 0$$

Solving these equations, we get

$$a = 6, b = 24, c = -8$$

\section{}

Given
$$\textbf{r}(x,y,z) = xi + yj + zk$$ and $$r(x,y,z) = ||\textbf{r}(x,y,z)|| = \sqrt{x^{2} + y^{2} + z^{2}}$$

Also
$$r^{n} = \sqrt[n/2]{x^{2} + y^{2} + z^{2}}$$

Computing for some value $n$, a positive integer, the gradient of $r^{n}$

$$ \nabla(r^{n}) = \Big(\frac{\partial r^{n}}{\partial x}i + \frac{\partial r^{n}}{\partial y}j + \frac{\partial r^{n}}{\partial z}k\Big)$$

$$\nabla(r^{n}) = \Big(\frac{2nx\sqrt[(n-2)/2]{x^{2} + y^{2} + z^{2}}}{2}i + \frac{2ny\sqrt[(n-2)/2]{x^{2} + y^{2} + z^{2}}}{2}j + \frac{2nz\sqrt[(n-2)/2]{x^{2} + y^{2} + z^{2}}}{2}k\Big)$$

$$\nabla(r^{n}) = \Big(nx\sqrt[(n-2)/2]{x^{2} + y^{2} + z^{2}}i + ny\sqrt[(n-2)/2]{x^{2} + y^{2} + z^{2}}j + nz\sqrt[(n-2)/2]{x^{2} + y^{2} + z^{2}}k\Big)$$

$$\nabla(r^{n}) = \Big(n\sqrt[(n-2)/2]{x^{2} + y^{2} + z^{2}}\Big)\times\Big(xi + yj + zk\Big)$$

$$\nabla(r^{n}) = \Big(nr^{n-2}\Big)\times\Big(xi + yj + zk\Big)$$

$$\nabla(r^{n}) = nr^{n-2}\textbf{r}$$

\section{}

Given a function $u = f(x,y)$, $x = X(t)$, $y = Y(t)$ define $u$ as a function of $t$, say $u = F(t)$

\subsection{}

Given $$f(x,y) = x^{2} + y^{2} , X(t) = t, Y(t) = t^{2}$$

Replacing the given values of $(x,y)$ as $(X(t),Y(t))$, we get

$$u(t) = t^{2} + t^{4}$$

$F\prime(t)$ can be calculated as

$$F\prime(t) = \frac{du(t)}{dt}$$

$$F\prime(t) = \frac{d(t^{2} + t^{4})}{dt}$$

$$F\prime(t) = 2t + 4t^{3}$$

$F\prime\prime(t)$ can be calculated as

$$F\prime\prime(t) = \frac{dF\prime(t)}{dt}$$

$$F\prime\prime(t) = \frac{d(2t + 4t^{3})}{dt}$$

$$F\prime\prime(t) = 2 + 12t^{2}$$

\subsection{}

Given $$f(x,y) = e^{xy}\cos(xy^{2}), X(t) = \cos(t), Y(t) = \sin(t)$$

Replacing the given values of $(x,y)$ as $(X(t),Y(t))$, we get

$$u(t) = e^{\cos(t)\sin(t)}\cos(\cos(t)(\sin(t))^{2})$$

$F\prime(t)$ can be calculated as

$$F\prime(t) = \frac{du(t)}{dt}$$

$$F\prime(t) = \frac{d(e^{\cos(t)\sin(t)}\cos(\cos(t)(\sin(t))^{2}))}{dt}$$

Have to add.

\section{}
See appended section
\section{}
See appended section
\section{}
See appended section
\section{}
See appended section
\section{}
See appended section
\section{}
See appended section
\section{}
See appended section
\section{}

\subsection{}

Given $$f(x,y,z) = (y^{2} − z^{2})i + 2yzj − x^{2}k$$

along the path described by $\alpha(t) = ti + t^{2}j + t^{3}k$
\\
\\
The line integral of the vector field is

$$\int{f(\alpha(t))}\cdot{d\alpha} = \int {((t^{4} − t^{6})i + 2t^{5}j − t^{2}k)}\cdot{(i + 2tj + 3t^{2}k)}dt$$

$$= \int{(t^{4} - t^{6}) + 4t^{6} - 3t^{4}}dt$$

$$= \Big(\frac{-2t^{5}}{5} + \frac{3t^{7}}{7}\Big)$$

\subsection{}

See appended section

\section{}

\subsection{}

Given $$\int_{C}(x^{2} - 2xy)dx + (y^{2} - 2xy)dy$$

where $C$ is a path from $(−2,4)$ to $(1,1)$ along the parabola $C$ $y = x^{2}$.
\\
\\
The parametric equation of the curve is $x = t$ and $y = t^{2}$, then the line integral is

$$= \int_{-2}^{1}(t^{2} - 2t^{3})dt + 2(t^{5} - 2t^{4})dt$$

$$= \int_{-2}^{1}(t^{2} - 2t^{3} + 2t^{5} - 4t^{4})dt$$

$$= \Big(\frac{t^{3}}{3} - \frac{t^{4}}{2} + \frac{t^{6}}{3} - \frac{4t^{5}}{5}\Big)_{-2}^{1}$$

$$= \frac{396}{10}$$

\subsection{}

Given $$\int_{C}\frac{(x+y)dx - (x-y)dy}{x^{2} + y^{2}}$$

where $C$ is is the circle $x^{2} + y^{2} = a^{2}$ traversed once in a counter-clockwise direction.
\\
\\
The parametric equation of the curve is $x = a\cos{t}$ and $y = a\sin{t}$, then the line integral is

$$= \int_{0}^{2\pi}\frac{-a\sin{t}(\cos{t}+\sin{t})dt - a\cos{t}(\cos{t}-\sin{t})dt}{a^{2}({\cos{t}}^{2} + {\sin{t}}^{2})}$$

$$= \int_{0}^{2\pi}\frac{-a({\sin{t}}^{2} + {\cos{t}}^{2})dt}{a^{2}}$$

$$= \int_{0}^{2\pi}\frac{-dt}{a}$$

$$= \int_{0}^{2\pi}\frac{-dt}{a}$$

$$= \Big(\frac{-t}{a}\Big)_{0}^{2\pi}$$

$$= \frac{-2\pi}{a}$$
\section{}
\subsection{}
Given the vector field $$f(x, y) = (2xe^{y} + y)i + (x^{2}e^{y} + x − 2y)j$$

We have $$f_{1}(x,y) = 2xe^{y} + y$$ and $$f_{2}(x,y) = x^{2}e^{y} + x − 2y$$

Then, the partial derivatives $D_{2}f_{1}$ and $D_{1}f_{2}$ are given by
$$D_{2}f_{1} = 2xe^{y} + 1$$

and

$$D_{1}f_{2} = 2xe^{y} + 1$$

Since $D_{2}f_{1} = D_{1}f_{2}$ for all values of $(x,y)$, this vector field is a gradient on any open subset of $R^{2}$.
\\
We know that $$\frac{\partial \phi}{\partial x} = 2xe^{y} + 1$$ and $$\frac{\partial \phi}{\partial y} = 2xe^{y} + 1$$

Using indefinite integrals and integrating the first of these equations with respect to x (holding y constant) we find

$$\phi(x,y) = \int(2xe^{y} + 1)dx + A(y) = x^{2}e^{y} + x + A(y)$$

and

$$\phi(x,y) = \int(2xe^{y} + 1)dy + B(x) = 2xe^{y} + y + B(x)$$

\subsection{}

Given the vector field $$f(x,y,z) = 2xy^{3}i + x^{2}z^{3}j + 3x^{2}yz^{2}k$$

We have $$f_{1}(x,y) = 2xy^{3}$$ $$f_{2}(x,z) = x^{2}z^{3}$$ $$f_{3}(x,y,z) = 3x^{2}yz^{2}$$

Then, the partial derivatives $D_{3}f_{1}$, $D_{2}f_{2}$ and $D_{1}f_{3}$ are given by
$$D_{3}f_{1} = 0$$

$$D_{2}f_{2} = 0$$

$$D_{1}f_{3} = 6xyz^{2}$$

Since $D_{3}f_{1} = D_{2}f_{2} = D_{1}f_{3}$ only when either $x, y$ or $z$ is equal to $0$

Hence, this vector is not a gradient of a scalar field $\phi$

\newpage

\section{Appendix}

Due to time constrains, I haven't been able to type all the solutions for the assignments in LaTeX. The answers that refer to this section are in the other document.

\end{document}


\section{Simulation}

We need to simulate a data set to train, tune and selecting an algorithm for stock picking. As described in section \ref{sec:model} the returns on the stocks are drawn from a distribution (here assumed to be normal):

\begin{equation}
    \rr^{(t)} \sim N(\mu^{(t)}, \Omega^{(t)})
\end{equation}

\subsection{Sampling of expected returns}

Investigating the distribution of means, we find it is reasonable to assume the expected return for an individual stocks follows a normal distribution:

\begin{equation}
    \mu^{(t)}_{i} \sim N(\xi, \sigma_{r})
\end{equation}

To estimate the parameters $\xi$, $\sigma_r$ we perform a log-likelihood estimation which yields the following values: $\xi=0.000461$ and $\sigma_r = 0.00123$. Figure \ref{fig:distmeans} plots the fitted distribution vs. the empirical distribution, and even though the fourth moment of the distribution seems not to exactly follow the normal distribution, the normal distribution appears to be a good approximation of the data generating process of $\mu_i^{(t)}$.

\begin{figure}[ht]
\centering
\includegraphics[scale=0.3]{figures/distribution_means.png}
\caption{Distribution of individual means $\mu_i^{(t)}$}
\label{fig:distmeans}
\end{figure}

\subsection{Sampling of covariance matrix}

Sampling the covariance matrix is more involved than sampling $\mu$. I go the following way about in in this paper: Realize that a transformation from a covariance matrix to a correlation matrix is possible:

\begin{equation}
    \text{corr}_{x, y} = \frac{\sigma_{x,y}^{2}}{\sigma_x \cdot \sigma_y}
\end{equation}

Therefore we first need to sample a correlation matrix, then a vector of variances, then transform the correlation matrix to the covariance matrix $\Omega^{(t)}$.

\subsubsection{Sampling of correlation matrix}

Sampling a correlation matrix is not as straightforward as sampling the individual stock returns, $\mu_i^{(t)}$. This is due to the fact, the correlation matrix should adhere to multiple criteria:

\begin{enumerate}
    \item A correlation matrix is a positive definite matrix.
    \item The individual values (apart from the diagonal) of the correlation matrix should follow a normal distribution $\rho_{i}^{(t)} \sim N(\kappa, \sigma_\rho)$.
\end{enumerate}

In this paper to accomplish the desired I use the following sampling scheme:

\begin{enumerate}
    \item Initialize an empty matrix $M$ of size $k \times k$.
    \item sample individual observations $\rho_{i,j}=\rho_{j,i}$ from a fitted normal distribtuion.
    \item Replace the diagonal of $M$ with 1.
    \item Save the matrix $M$ if all eigenvalues are positive, i.e. $M$ is positive definite.
\end{enumerate}

Figure \ref{fig:distcorrs} shows the fitted distribution plotted against the empirical distribution with regards to the individual $\rho_{i,j}^{(t)}$ not considering the diagonal. Again log-likelihood estimation is used to estimate the parameters: $\kappa=0.313, \sigma_\rho=0.188$. Again we see that, the actual distribution does not perfectly follow a normal distribution, but passes for a good approximation. It should be noted that when sampling from this distribution, the distribution is truncated at $(-1, 1)$.

\begin{figure}[ht]
\centering
\includegraphics[scale=0.3]{figures/correlation_distribution.png}
\caption{Distribution of individual correlates $\rho_{i,j}^{(t)}$}
\label{fig:distcorrs}
\end{figure}

\subsubsection{Sampling of variances}

The variances does not follow a normal distribution, this is obvious, since the variance by definition is a positive measure. I model the variances to be exponentially distributed: $\sigma^2_{i} \sim expon(\lambda)$. Using log-likelihood estimation we find the $\lambda=0.000346$. Figure \ref{fig:distvars} displays the fitted distribution against the empirical distribution. I conclude that the exponential distribution is a reasonable approximation of the true distribution.

\begin{figure}[ht]
\centering
\includegraphics[scale=0.3]{figures/distribution_variances.png}
\caption{Distribution of individual variances}
\label{fig:distvars}
\end{figure}

\subsubsection{The simulated covariance distribution vs. the empirical distribution}

Using the scheme described above, I simulate a set of covariance matrices. Figure \ref{fig:distcovars} displays the simulated distribution of individual covariances vs. the empirical distribution of individual covariances. From the figure, we see that the simulated distribution, seems slightly biased towards 0. However I conclude that for the purpose of this paper the simulation passes as a good approximation.

\begin{figure}[ht]
\centering
\includegraphics[scale=0.3]{figures/dist_simulated_covariances.png}
\caption{Distribution of individual covariances}
\label{fig:distcovars}
\end{figure}

\subsubsection{The simulated data set}

Now having simulated covariance matrices, and returns, we can simulate an entire data set, that corresponds to our model. We simulate a data set of $11$ columns and $2.000.0000$ rows, which is more than necessary for the further investigation in this paper. We set the parameter $p=0.047$, so we have just slightly less then $10.000$ structural breaks in the simulated data set. Since we have the underlying $\mu^{(t)}$ and $\Omega^{(t)}$ in each period, we can calculate the Sharpe ratios for each of the simulated stocks throughout the period assuming the risk free asset having a return of $\bar{r}=2 \%$ annually.

Summing it all up: A dataset is now constructed based on a structural model. The data set consists of not only the simulated returns, but also the Sharpe ratios of the individual stocks, which in the real data set are latent variables. The simulated data set allows for training, tuning and selecting between algorithms.


% !TEX root = main.tex

%-------------------------------------------------
\section{Signed variables}\label{sec:}

Random variables that take both positive and negative values are called \emph{signed variables}.

\begin{definition}
The \emph{positive part} and \emph{negative part} of a random variable $X$ are
\begin{align*}
X^{+}(\omega)
	& = \begin{cases}
			X(\omega)	& \text{if } X(\omega)\geq 0, \\
	   		0 		 	& \text{if } X(\omega) < 0, \text{ and}
	\end{cases} \\[2ex]
X^{-}(\omega)
	& = \begin{cases}
			-X(\omega)	& \text{if } X(\omega) < 0, \\
			0 		 	& \text{if } X(\omega) \geq 0.
		\end{cases}
\end{align*}
respectively.
\end{definition}

Note that $X^{+}$ and $X^{-}$ are both non-negative random variables with
\[
X = X^{+} - X^{-}
\qquad\text{and}\qquad
|X| = X^{+} + X^{-}.
\]

\begin{definition}[Expectation of signed random variables]
The \emph{expectation} of a signed random variable $X$ is
\[
\expe(X) = \expe(X^{+}) - \expe(X^{-})
\]
provided that $\expe(X^{+})$ and $\expe(X^{-})$ are not both infinite.
\end{definition}

\begin{remark}
If $\expe(X^{+})$ and $\expe(X^{-})$ are both infinite, the expectation of $X$ is \emph{undefined} because we cannot make sense of the expression ``$\infty - \infty$''. In this case we say that the expectation of $X$ \emph{does not exist}.
\end{remark}

\begin{example} % discrete, undefined
Let $X$ be a discrete random variable with the following PMF:
\[
f(k) = \begin{cases}
	\displaystyle\frac{3}{\pi^2 k^2} 	& \text{if }\ k\in\{\pm 1,\pm 2, \ldots\} \\[1ex]
	0						& \text{otherwise.}
\end{cases}
\]
Show that $\expe(X)$ is undefined.
\begin{solution}
$X$ is a signed random variable, so we must deal with its positive and negative parts separately:
\[
X^{+}	= \begin{cases}  X & \text{ if } X \geq 0, \\ 0 & \text{otherwise.}\end{cases}
\qquad\text{and}\qquad
X^{-}	= \begin{cases} -X & \text{ if } X <    0, \\ 0 & \text{otherwise.}\end{cases}
\]
The expected values of $X^{+}$ and $X^{-}$ are
\[\begin{array}{lll}
\expe(X^{+})
	& = \displaystyle \sum_{k=1}^{\infty} k \left(\frac{3}{\pi^2 k^2}\right)
	& = \displaystyle \frac{3}{\pi^2} \sum_{k=1}^{\infty}\frac{1}{k} = \infty \\[2ex]
\expe(X^{-})
	& = \displaystyle \sum_{k=-\infty}^{-1} (-k) \left(\frac{3}{\pi^2 k^2}\right)
	& = \displaystyle \frac{3}{\pi^2} \sum_{k=1}^{\infty}\frac{1}{k} = \infty \\
\end{array}\]
so $\expe(X) = \expe(X^{+}) - \expe(X^{-})$ is undefined.
\end{solution}
\end{example}

\begin{example}[Cauchy distribution]% continuous - undefined
Let $X$ be a continuous random variable having the following PDF,
\[
f(x) = \frac{1}{\pi(1+x^2)}\qquad\text{for all $x\in\R$.}
\]
Show that $\expe(X)$ is undefined.
\begin{solution}
$X$ is a signed variable, so we must deal with its positive and negative parts separately:

\[\begin{array}{lll}
\expe(X^{+})
	& = \displaystyle\int_0^\infty xf(x)\,dx
	& = \displaystyle\frac{1}{\pi}\int_0^\infty \frac{x}{1+x^2}\,dx  \\[2ex]
\expe(X^{-})
	& = \displaystyle\int_{-\infty}^0 (-x)f(x)\,dx
	& = \displaystyle\frac{1}{\pi}\int_0^{\infty} \frac{x}{1+x^2}\,dx
\end{array}\]
Thus $\expe(X) = \expe(X^{+}) - \expe(X^{-})$ is given by
\begin{align*}
\expe(X)
%	& = \expe(X^{+}) - \expe(X^{-}) \\
	& = \frac{1}{\pi}\left[\int_0^{\infty} \frac{x}{1+x^2}\,dx - \int_0^{\infty} \frac{x}{1+x^2}\,dx\right].
\end{align*}

At this point, it might be tempting to conclude that $\expe(X)=0$. However if $x>1$ then $x^2 > 1$ and therefore $2x^2 > 1+x^2$, so
\[
\frac{x}{1+x^2} > \frac{1}{2x} \qquad\text{for all } x > 1
\]
Consequently,
\[
\int_{0}^{\infty}\frac{x}{1+x^2}\,dx
	> \int_{1}^{\infty}\frac{x}{1+x^2}\,dx
	> \frac{1}{2}\int_{1}^{\infty}\frac{1}{x}\,dx
	= \infty,
\]
so $\expe(X)$ is undefined.
\end{solution}
\end{example}


% LinearSpace.tex
\documentclass[12pt]{article}
\usepackage[T1]{fontenc}
\usepackage[margin=1in]{geometry}
\usepackage{amsmath, amsfonts, amsthm}
\usepackage{verbatim}

\newcommand{\xmax}{x_{\max}}
\newcommand{\xmin}{x_{\min}}
\newcommand{\xmid}{x_{\mathrm{mid}}}
\newcommand{\xt}{x-\xmid}
\newcommand{\xg}{2\left[ \frac{x-\xmin}{\xmax - \xmin} \right] - 1}
\newcommand{\dd}{\mathop{}\!\mathrm{d}}
\newcommand{\Pc}{\overline{Pc}}

\title{Hand Calculation of a Linear Space Source Scoring}
\author{Jeremy Lloyd Conlin}

\begin{document}
\maketitle
In this document, I will try to figure out why my LinearSpace scoring isn't working properly.  The calculations will be done ``by hand'' or at least manually instead of a C++ code.

\section{Approximating $f(x)$}
We will approximate $f(x)$ as a linear combination of polynomials
\begin{equation}
    \tilde{f}(x) = \sum_{n=1}^N a_n \Psi_n(x) \hspace{0.5in} \xmin \leq x \leq \xmax
    \label{eq:fApproximation}
\end{equation}
where $a_n$ is an expansion coefficient and $\Psi_n(x)$ is some polynomial which has a few properties
\begin{equation}
    \int_{\xmin}^{\xmax} \Psi_m(x)\Psi_n(x) \dd x =
    \begin{cases}
        1 & m = n \\
        0 & m \neq n.
    \end{cases}
    \label{eq:Orthonormalized}
\end{equation}
Equation \ref{eq:Orthonormalized} states that the polynomials $\Psi_n(x)$ must be normalized and orthogonal to each other.

We will make our polynomials the Legendre polynomials $\Psi_n(x) = P_n(x)$.

\subsection{Determining the Expansion Coefficients}
We can determine the expansion coefficients as
\begin{align}
    \begin{split}
        \int_{\xmin}^{\xmax} P_m(x)\tilde{f}(x) \dd x  &=\\
        &= \int_{\xmin}^{\xmax} P_m(x)\sum_{n=1}^{N_p} a_n \Psi_n(x)\dd x \\
        &= \int_{\xmin}^{\xmax} P_m(x)\sum_{n=1}^{N_p} a_n P_n(x)\dd x \\
        &= \sum_{n=1}^{N_p}a_n\int_{\xmin}^{\xmax} P_m(x) P_n(x)\dd x \\
        &= a_n\int_{\xmin}^{\xmax}P_n(x)^2 \dd x.
    \end{split}
    \intertext{Therefore}
    a_n &= \frac{1}{k_n}\int_{\xmin}^{\xmax} P_n(x)\tilde{f}(x) \dd x,
    \label{eq:CoefficientDefined}
    \intertext{where}
    k_n &= \left( \int_{\xmin}^{\xmax} P_n(x)^2 \dd x \right)^{1/2}.
    \label{eq:NormalizingConstant}
\end{align}
Equation \ref{eq:CoefficientDefined} uses the orthonormalizing definition in equation \ref{eq:Orthonormalized}.

Equation \ref{eq:CoefficientDefined} is---by definition---the expected value of the function $P_n(x)/k_n$ where samples of $x$ are taken from $\tilde{f}(x)$.  It is much simpler to sample from a uniform distribution.  It is a simple change of variable to allow this;
\begin{equation}
        a_n = \frac{1}{k_n}\int_{\xmin}^{\xmax} P_n(x)\omega(x)U(x) \dd x \\
    \label{eq:WeightedCoefficient}
\end{equation}
where
\begin{subequations}
    \begin{align}
        \omega(x) &= \frac{\tilde{f}(x)}{U(x)}
        \label{eq:WeightingFunction}
        \intertext{is a weighting function, and}
        U(x) &= \frac{1}{\xmax-\xmin}
        \label{eq:UniformDistribution}
    \end{align}
\end{subequations}
is a uniformly distributed PDF.

A careful comparison between equations \ref{eq:CoefficientDefined} and \ref{eq:WeightedCoefficient} reveals we have only multiplied the integrand by $U(x)/U(x) = 1$, but we have changed the nature of the problem to make it simpler; we can now sample from a uniform distribution and score the weighted Legendre polynomial $P_n(x)\omega(x)$.  We can sample and score $N_i$ times; the mean of these evaluations is an estimate of the expansion coefficient:
\begin{equation}
    \hat{a}_n = \frac{1}{N_i}\frac{1}{k_n} \sum_{i=1}^{N_i} P_n(x_i) \omega(x_i).
    \label{eq:MCCoefficient}
\end{equation}

In this paper I will only use expansions with two basis functions, $N=2$ so we can write out the functions that we will need.
\begin{subequations}
    \begin{align}
        P_0(x) & = 1 \\
        P_1(x) & = x \\
    \end{align}
    \label{eq:LegendrePolynomials}
\end{subequations}
\begin{subequations}
    \begin{align}
        k_0 = \int_{\xmin}^{\xmax} P_0(x) \dd x = \int_{\xmin}^{\xmax}  \dd x &= \boxed{\left(\xmax - \xmin\right) = k_0} \\
        k_1 = \int_{\xmin}^{\xmax} P_1(x) \dd x = \int_{\xmin}^{\xmax} x^2 \dd x &= \boxed{\frac{1}{3}\left(\xmax^3 - \xmin^3\right) = k_1}
    \end{align}
    \label{eq:ConstantsDefined}
\end{subequations}

\section{Simple Example}
\begin{equation}
    f(x) = \frac{1}{2}\left( x+1 \right), \hspace{0.5in} -1 \leq x \leq 1.
    \label{eq:original}
\end{equation}
$f(x)$ is already normalized so it is ready for sampling.  The normalization constants for this problem are
\begin{subequations}
    \begin{align}
        k_0 &= \int_{\xmin}^{\xmax}  \dd x = \left(\xmax - \xmin\right) = \left(1 - (-1)\right) = 2 \\
        k_1 &= \int_{\xmin}^{\xmax} x^2 \dd x = \frac{1}{3}\left(\xmax^3 - \xmin^3\right) = \frac{1}{3}\left(1^3 - (-1)^3\right) = \frac{2}{3}.
    \end{align}
\end{subequations}
We can also define the uniform distribution and weighting functions, respectively
\begin{subequations}
    \begin{align}
        U(x) &= \frac{1}{\xmax - \xmin} = \frac{1}{2} \\
        \omega(x) &= \frac{\tilde{f}(x)}{U(x)} = x+1.
    \end{align}
\end{subequations}
The expansion coefficients:
\begin{subequations}
    \begin{align}
        \hat{a}_0 &= \frac{1}{N_i}\frac{1}{k_0} \sum_{i=1}^{N_i} P_0(x_i)\omega(x_i) = \frac{1}{N_i}\frac{1}{2} \sum_{i=1}^{N_i} \omega(x_i) \\
        \hat{a}_1 &= \frac{1}{N_i}\frac{1}{k_1} \sum_{i=1}^{N_i} P_1(x_i)\omega(x_i) = \frac{1}{N_i}\frac{3}{2} \sum_{i=1}^{N_i} x_i\,\omega(x_i).
    \end{align}
    \label{eq:MCCoeffSimple}
\end{subequations}

We learned that you can't perform a simulation by performing this calculation by hand, that is, you can't do just a few points and expect the answer to be correct.  The variance of this is just too large.  We can evaluate the coefficients many times easily with a computer then we can have enough data points to make it good enough.

It works!

\section{Harder Example}
This section has a function that is not on the traditional range $[-1,1]$.  The function used for this example is:
\begin{equation}
    f(x) = x-1, \hspace{0.5in} 1 \leq x \leq 5.
    \label{eq:original}
\end{equation}
Normalized such that
\begin{equation}
    \int_1^5 f(x)^2 \dd x = 1,
\end{equation}
Equation \ref{eq:original} becomes
\begin{equation}
    f(x) = 0.125\left( x-1 \right) \hspace{0.5in} 1 \leq x \leq 5.
    \label{eq:normalized}
\end{equation}
The midpoint of this ``bin'' is
\begin{equation}
    \xmid = \frac{\xmax + \xmin}{2.0} = \frac{5 + 1}{2.0} = 3.0
    \label{eq:xmid}
\end{equation}
and we will define
\begin{equation}
    \tilde{x} \equiv x - \xmid. \label{eq:xTilde}
\end{equation}

The thing that makes this function more difficult is the fact that the Legendre polynomials are not naturally orthogonal to each other on the range $[1,5]$.  We can show that they are orthogonal to each other if the range is centered at 0 (at least we can show that the 0th and 1st basis functions are orthogonal to each other if they are centered at 0).  To do so, we shift the Legendre polynomial by subtracting the midpoint of the range from the x point and then we can integrate:

Now we need to be a bit more careful with this new derivation.  So I'll try to be a bit more formal.

\subsection{More formal}
We can define our function as an expansion of basis functions as before,
\begin{equation}
    \tilde{f}(x) = \sum a_m P_m(x).
    \label{eq:fExpansion}
\end{equation}

Finding the expansion coefficients is similar to how this is done before, but we must shift our Legendre polynomials and $\tilde{f}(x)$ so that the range is centered about zero.  The simplest way to do this is to subtract the midpoint of the original range as I defined as $\tilde{x}$.  Now we can proceed as usual:
\begin{equation}
    \begin{split}
        \int_{\xmin}^{\xmin} P_n(\xt)\tilde{f}(\xt) \dd x &= \\
        &= \int_{\xmin}^{\xmin} P_n(\tilde{x})\tilde{f}(\tilde{x}) \dd x \\
        &= \int_{\xmin}^{\xmin} P_n(\tilde{x})\sum a_m P_m(\tilde{x}) \dd x \\
        &= \sum a_m \int_{\xmin}^{\xmin} P_n(\tilde{x})P_m(\tilde{x}) \dd x \\
        &= a_n k_n
    \end{split}
    \label{eq:Coefficients}
\end{equation}
where
\begin{equation}
    k_n = \int_{\xmin}^{\xmax} P_n(\xt)^2 \dd x.
    \label{eq:knDefined}
\end{equation}
Solving equation \eqref{eq:Coefficients} for $a_n$ we get
\begin{equation}
    a_n = \frac{1}{k_n}\int_{\xmin}^{\xmin} P_n(\xt)\tilde{f}(\xt) \dd x.
    \label{eq:CoeffDefined}
\end{equation}
Of course we want to be able to sample from a uniform distribution when doing our Monte Carlo so now we rewrite equation \eqref{eq:CoeffDefined} as
\begin{equation}
    a_n = \frac{1}{k_n}\int_{\xmin}^{\xmin} P_n(\xt)\underbrace{\frac{\tilde{f}(\xt)}{U(\xt)}}_{\omega(\xt)}U(\xt) \dd x.
    \label{eq:CoeffDefined}
\end{equation}
Our Monte Carlo approximation for this coefficient is then
\begin{equation}
    \hat{a}_n = \frac{1}{k_n}\frac{1}{N} \sum_{i=1}^N P_n(x_i - \xmid) \omega(x_i - \xmid)
    \label{eq:MCCoefficients}
\end{equation}
where the $x_i$s are sampled from the uniform distribution $U(x)$.

This derivation requires that the shifted Legendre polynomials are orthogonal to each other on the range $[\xmin, \xmax]$.  We can do this for $n=1$ and $m=0$:
\begin{equation}
    \begin{split}
        \int_{\xmin}^{\xmax} P_0(x-\xmid)P_1(x-\xmid) \dd x &= \\
         &=\int_{\xmin}^{\xmax} (1) \left( x-\xmid \right) \dd x \\
         &=\int_{\xmin}^{\xmax} \left( x-\xmid \right) \dd x \\
         &=\left[\frac{1}{2}x^2-\xmid\; x\right]_{\xmin}^{\xmax} \\
         &=\frac{1}{2}\left(\xmax^2 - \xmin^2\right)-\xmid\left(\xmax-\xmin\right)\\
         &=\frac{1}{2}\left(\xmax^2 - \xmin^2\right)-\left(\frac{\xmax+\xmin}{2}\right)\left(\xmax-\xmin\right)\\
         &=\frac{1}{2}\left(\xmax^2 - \xmin^2\right)-\frac{1}{2}\left(\xmax^2-\xmin^2\right)\\
         &= 0.
    \end{split}
\end{equation}
This can be done in general as
\begin{equation}
    \int_{\xmin}^{\xmax}P_n(\xt)P_m(\xt) \dd x.
\end{equation}
We first need to make a change of variable
\begin{equation}
   u = \xt, \qquad  \dd u = \dd x
   \label{eq:VariableChange}
\end{equation}
\begin{subequations}
    \label{eq:uLimits}
    \begin{alignat}{4}
        u_{max} &= u(\xmax) &= \xmax - \xmid &= \frac{\xmax - \xmin}{2} & &=t \\
        u_{min} &= u(\xmin) &= \xmin - \xmid &= \frac{\xmin - \xmax}{2} &=-\frac{\xmax - \xmin}{2} &= -t
    \end{alignat}
\end{subequations}
Inserting equations \eqref{eq:VariableChange} and \eqref{eq:uLimits} we obtain
\begin{equation}
    \int_{-t}^{t}P_n(u)P_m(u) \dd u.
    \label{eq:NewIntegral}
\end{equation}
Equation \eqref{eq:NewIntegral} is valid for any value of $t$.  We know that for $t=1$ this integral evaluates to zero.  However this is not true for any $n$ and $m$.  It is true for $n+m = 2i+1 = \mathrm{odd}$.  This can be seen by noting that $P_n(x)$ is a polynomial of degree $n$ and that $P_n(x)P_m(x)$ is a polynomial of degree $n+m$.  A polynomial with an odd degree is an odd function and therefore will integrate to zero if the limits of integration are centered about zero.  If the degree is even, the polynomial is an even function and the integral will not be zero.  Therefore shifting the Legendre polynomials will not work in general this way.  I will have to move to David Griesheimer's version of the shifted polynomials.  My method will work if I am just using a second order expansion.

\section{Griesheimer's Method of Shifting}
Griesheimer defined his expansion not too differently from what I have been doing;
\begin{equation}
    \tilde{f}(x) = \sum a_mP_m\!\left(\hat{x}(x)(x)\right).
    \label{eq:GrExpansion}
\end{equation}
In Dave's thesis, he (correctly) shifts his Legendre polynomials so that they are not only centered at zero, but also fall in the range $[-1,1]$.  To do this it is a simple scaling
\begin{equation}
    \hat{x}(x) = \xg.
    \label{eq:xScaled}
\end{equation}
With this scaling it is easy to see that the range of $\hat{x}(x)$ is what we desire.

We can determine our expansion coefficients similarly to what we have already done
\begin{equation}
    \begin{split}
        \int_{\xmin}^{\xmax} P_n\!\left[\hat{x}(x)\right]\tilde{f}(x) \dd x &= \\
        &= \int_{\xmin}^{\xmax} P_n\!\left[\hat{x}(x)\right] \sum a_mP_m\!\left[ \hat{x}(x) \right] \dd x \\
         &=  \sum a_m\int_{\xmin}^{\xmax} P_n\!\left[ \hat{x}(x) \right]P_m\!\left[ \hat{x}(x) \right] \dd x \\
         &=  a_nk_n.
    \end{split}
    \label{eq:CoeffsG}
\end{equation}
Now $k_n$ is non-zero whenever $m = n$.  We can evaluate the integral by a change of variable from $x$ to $\hat{x}(x)$.
\begin{equation}
    k_n = \int_{\xmin}^{\xmax} P_n^2\left[ \hat{x}(x) \right] \dd x \label{eq:kn}
\end{equation}
Let
\begin{alignat}{2}
    y &= \hat{x}(x) & \qquad y\!\left(\xmin\right) &= -1 \\
    \dd y &= \frac{2}{\xmax - \xmin} \dd x & \qquad y\!\left(\xmax\right) &= 1.
\end{alignat}
Equation \eqref{eq:kn} becomes
\begin{equation}
    k_n = \frac{\xmax-\xmin}{2}\int_{-1}^{1} P_n^2\left( y \right) \dd y = \frac{\xmax-\xmin}{2}\frac{2}{2n+1} = \frac{\xmax-\xmin}{2n+1}.
\end{equation}

Solving equation \eqref{eq:CoeffsG} for $a_n$ we obtain
\begin{equation}
    \begin{split}
        a_n &= \frac{1}{k_n}\int_{\xmin}^{\xmax}P_n(\hat{x}(x))\tilde{f}(x) \dd x \\
         &= \frac{1}{k_n}\int_{\xmin}^{\xmax}P_n\!\left( \xg \right)\tilde{f}(x) \dd x \\
    \end{split}
\end{equation}
We want this weighted as before so we can sample from a uniform distribution
\begin{equation}
    a_n = \frac{1}{k_n} \int_{\xmin}^{\xmax} P_n\left[ \hat{x}(x) \right] \underbrace{\frac{\tilde{f}(x)}{U(x)} U(x)}_{\omega(x)} \dd x \\
\end{equation}

\begin{equation}
    a_n = \frac{1}{k_n}\frac{1}{N}\sum_{i=1}^{N}P_n\!\left( 2\left[\frac{x_i-\xmin}{\xmax-\xmin}\right]-1 \right)\omega_n\!\left(x_i\right)
\end{equation}
where of course the $x_i$s have been sampled from the uniform distribution $U(x)$.

Now we want our expansion to use just the first two Legendre polynomials, $P_0(x)$ and $P_1(x)$.  We need two expansion coefficients
\begin{subequations}
    \begin{align}
        \begin{split}
            a_0 &= \frac{1}{k_0}\frac{1}{N}\sum_{i=1}^N P_0\!\left( 2\left[\frac{x_i-\xmin}{\xmax-\xmin}\right]-1 \right)\omega(x_i) \\
             &=  \frac{1}{k_0}\frac{1}{N}\sum_{i=1}^N \omega(x_i) \\
        \end{split}\\
        \begin{split}
            a_1 &= \frac{1}{k_1}\frac{1}{N}\sum_{i=1}^N P_1\!\left( 2\left[\frac{x_i-\xmin}{\xmax-\xmin}\right]-1 \right)\omega(x_i) \\
             &= \frac{1}{k_1}\frac{1}{N}\sum_{i=1}^N  \left( 2\left[\frac{x_i-\xmin}{\xmax-\xmin}\right]-1 \right)\omega(x_i)\\
        \end{split}
    \end{align}
\end{subequations}
Our expanded function becomes
\begin{equation}
    \begin{split}
        \tilde{f}(x) &= a_0P_0\left(\hat{x}\right) + a_1P_1\left(\hat{x}\right) \\
         &= a_0 + a_1\left( \xg \right).
    \end{split}
    \label{eq:2TermExpansion}
\end{equation}
What we would really like is a function of the traditional form \[ \tilde{f}(x) = mx+b.\]  This can be obtained from equation \eqref{eq:2TermExpansion} with just a bit of algebra
\begin{equation}
    \tilde{f}(x) = a_1\left(\frac{2}{\xmax-\xmin}\right)x + \left[a_0 - a_1\left(\frac{\xmax+\xmin}{\xmax-\xmin}\right)\right].
    \label{eq:2TermPointSlope}
\end{equation}
\end{document}


\subsection{Step Response Characterisation}

The first step to calculating $T$, $r$, and $n$ is to  understand  how the order
$n$ influences the shape of the transfer function's step response.

It is important to realise that no matter how a step response function is scaled
or  offset  (defined  by the parameters $K_s$ for amplitude, $T$ for time scale,
$x_0$ and $y_0$ for offset) the actual \textit{shape}  always  remains the same.
Thus, only its shape tells us something about its complexity.

Therein  lies  the  key. A method  needs  to  be  devised  for  determining  how
``simple''  or  how ``complex'' a step response is -- independent of  scale  and
offset -- before  it  is possible to start modelling and fitting a system to it.

This ``complexity'' is \textbf{directly related}  to  the  required order $n$ of
the transfer function.

\subsubsection*{P. Hudzovic's Approach}

P. Hudzovic proposed the following method (see figure \ref{fig:tu_tg}):

\begin{itemize}
    \item
Find  the  point  of  inflection of the step function. This  typically  involves
calculating the derivative and searching for a maximum.
    \item
Place a tangent  in  said  point and find the intersections with the minimum and
maximum horizontal lines.
    \item
The distance between the two intersections  is  referred  to  as  $T_g$, and the
distance between  the minimum intersection point and the beginning of the signal
is referred to as $T_u$.
\end{itemize}

The ``complexity'' of the  step  response  is  defined by the ratio of $T_u$ and
$T_g$ and is written as:

\begin{equation}
    \textrm{plant}_{Tu/Tg} = \frac{T_u}{T_g}
    \label{eq:tu_tg}
\end{equation}

\subsubsection*{L. Sani's Approach}

L.  Sani  proposed   a  different  method  (see  figure  \ref{fig:t10_t50_t90}):
Determine the  times  $t_{10}$,  $t_{50}$ and $t_{90}$ required for reaching the
values   at    \SI{10}{\percent},    \SI{50}{\percent}    and   \SI{90}{percent}
respectively.

The   ``complexity''   of  the  step  response  is  defined  by  the  ratio   of
$t_{90}-t_{10}$   and   $t_{50}$,   otherwise   referred   to   as    $\lambda$:

\begin{equation}
    \textrm{plant}_{\lambda} = \frac{t_{90}-t_{10}}{t_{50}}
    \label{eq:t10_t50_t90}
\end{equation}

\subsubsection*{Short Visual Explanation}

Visually, one can see how decreasing $T_g$ in figure \ref{fig:tu_tg}  causes the
step  response  to become steeper (i.e. it becomes  more  ``complex''  and  thus
requires  a  higher  order  $n$)  and the value of $\textrm{plant}_{T_u/T_g}$ in
equation  \ref{eq:tu_tg}  increases.   Similarly,   decreasing   the  difference
$t_{90}-t_{10}$ in figure \ref{fig:t10_t50_t90} also causes the step response to
become  steeper  and causes the value of  $\textrm{plant}_{\lambda}$  increases.

On the other hand, one can  also see how increasing $T_u$ and $t_{50}$ increases
the delay  time  of  the  step  response,  which  similarly  leads  to  a higher
``complexity'', and thus, a higher order $n$.

How $n$ is calculated will become clear in the next section.

\begin{figure}[t]
    \includegraphics[width=\linewidth]{images/step_response_tu_tg}
    \caption{Method of P. Hudzovic, determine Tu and Tg}
    \label{fig:tu_tg}
\end{figure}
\begin{figure}[t]
    \includegraphics[width=\linewidth]{images/step_response_t10_t50_t90}
    \caption{Method of L. Sani, determine t10, t50 and t90}
    \label{fig:t10_t50_t90}
\end{figure}


\subsection{Slater's condition}

\subsubsection{Strong duality}

Strong duality is where the duality gap is \(0\).
\subsubsection{Slater's condition}

Slater's condition says that strong duality holds if there is an input where the inequality constraints are satisified strictly.

That is they are \(g(x)<0\), not \(g(x)\le 0\)

This means that the conditions are slack.

This only applies if the problem is convex. That is, if Slater's condition holds, and the problem is convex, then strong duality holds.


\subsection{Trigonometric polynomials}\label{subsec:trigonometric_polynomials}

\begin{definition}\label{def:ring_of_laurent_polynomials}
  The ring of \term{Laurent polynomials} in the indeterminates \( \mscrX \) over the integral domain \( D \) is obtained from the \hyperref[def:polynomial_algebra]{polynomial ring} \( D[\mscrX] \) by \hyperref[thm:adjoining_elements_to_field]{adjoining} to \( D[\mscrX] \) the set
  \begin{equation*}
    \set*{ \frac 1 X \given* X \in \mscrX }
  \end{equation*}
  of reciprocals of the indeterminates from the field of rational functions \( D(\mscrX) \).

  If \( \mscrX = \set{ X_1, \ldots, X_n } \), this ring is denoted by \( R[X_1^\pm, \ldots, X_n^\pm] \) or \( R[X_1, X_1^{-1}, \ldots, X_n, X_n^{-1}] \). Individual polynomials are written as
  \begin{equation*}
    p(X_1, \ldots, X_n) = \sum_{k_1=-\infty}^\infty \cdots \sum_{k_n=-\infty}^\infty a_{k_1, \ldots, k_n} X^{k_1} \cdots X_n^{k_n}.
  \end{equation*}
\end{definition}

\begin{definition}\label{def:trigonometric_polynomial}
  We define the \term{trigonometric polynomials} over \( \BbbC \) as the \hyperref[def:ring_of_laurent_polynomials]{Laurent polynomials} \( \BbbC[e^{iz}] \). A trigonometric polynomial \( p \in \BbbC[e^{iz}] \) can be written as
  \begin{equation}\label{def:trigonometric_polynomial/exponential}
    p(z) = \sum_{k \in \BbbZ} c_k e^{ikz}
  \end{equation}
  or, using \hyperref[thm:exponential_trigonometric_identities/eulers_formula]{Euler's formula}, rewritten in the more conventional notation (see \cite[1]{Боянов2008} or \cite[88]{Rudin1987RealAndComplex}):
  \begin{equation}\label{def:trigonometric_polynomial/trigonometric}
    p(z) = a_0 + \sum_{k=1}^\infty [ a_k \cos(kz) + b_k \sin(kz) ],
  \end{equation}
  where we denote \( a_k \coloneqq c_k \) and \( b_k \coloneqq ic_k \).

  In particular, when using \fullref{def:trigonometric_polynomial/trigonometric}, we may regard the coefficients \( \{ a_k \}_{k=0}^\infty \) and \( \{ b_k \}_{k=1}^\infty \) as either real or complex, which is a downside of \fullref{def:trigonometric_polynomial/exponential}.

  Denote by \( \tau_n(\BbbK) \) the vector space of all trigonometric polynomials of degree at most \( n \) with coefficients in \( \BbbK \). We also introduce the subspaces \( \tau_n^\alpha{\BbbK} \) of those polynomials which \( a_0 = 0 \).
\end{definition}


\lab{Algorithms}{Moore-Penrose Pseudo-Inverse}{Moore-Penrose}

\objective{This section explains several methods for numerically computing the Moore-Penrose Pseudo-Inverse. It also compares performance of the different methods.}

The generalized inverse of a matrix $A$ is a matrix (which we will denote $A^\dagger$ for now) that satisfies the following:
\begin{equation} \label{cond:one}
AA^\dagger A = A
\end{equation}

\begin{equation} \label{cond:two}
A^\dagger A A^\dagger = A^\dagger
\end{equation}

For any matrix $A$ there exists a generalized inverse. Note that in the case that $A$ is invertible, $A^{-1}$ satisfies the conditions \ref{cond:one} and \ref{cond:two}. However, it is not generally unique. For this reason, we usually add additional conditions that make the generalized inverse unique.

For example, we can specify the following two additional conditions:

\begin{equation} \label{cond:three}
(AA^\dagger)^* = AA^\dagger
\end{equation}

\begin{equation} \label{cond:four}
(A^\dagger A)^* = A^\dagger A
\end{equation}

These four conditions guarantee both the existence and uniqueness of the matrix $A^\dagger$ for any matrix $A$. The matrix $A^\dagger$ is known as the Moore-Penrose inverse. In some settings it is simply called the pseudo-inverse.\footnote{For the rest of this section we will use the notation $A^\dagger$ to represent the Moore-Penrose inverse, although in some books that notation is used a generalized inverse.}

%{\bf We could add some exposition here about examples of the MP inverse for specific matrices, but I don't know if Jeff is doing that in the book or not...}

\subsection{Calculating the Moore-Penrose Inverse}

There are several different methods for calculating the Moore-Penrose Inverse. We will consider the most general method first, and then compare its performance to a few specialized methods.

Recall that the SVD of a matrix $A$ is of the form

\[
A = U \Sigma V^*
\]

Where $U$ and $V$ are unitary (i.e. $U^*U = I$) and $\Sigma$ is diagonal. Consider the matrix

\[
B = V \Sigma^\dagger U^*
\]

Where $\Sigma^\dagger$ is the pseudo-inverse of $\Sigma$. Since $\Sigma$ is diagonal, the pseudo-inverse is simply found by replacing each non-zero entry with its multiplicative inverse and transposing the matrix. We write, for ease of notation $\Sigma^\dagger \Sigma = I_A$ Now we have the following:

\[
ABA = U \Sigma V^* = A
\]

\[
BAB = V \Sigma^\dagger U^* = B
\]

\[
(AB)^* = U \Sigma^{\dagger *} V^* V \Sigma^* U^* = U (\Sigma \Sigma^\dagger)^* U^* = U \Sigma \Sigma^\dagger U^* = U \Sigma V^* V \Sigma^\dagger U^* = AB
\]

\[
(BA)^* =  V \Sigma^* U^* U \Sigma^{\dagger *} V^* = V (\Sigma^\dagger \Sigma)^* V^* = V  \Sigma^\dagger \Sigma V^* =  V \Sigma^\dagger U^* U \Sigma V^* = BA
\]

Thus $B$ satisfies all of the properties of the pseudo-inverse, and since the pseudo-inverse is unique it is the only such matrix.

Therefore, we can easily calculate the pseudo-inverse of a matrix using the SVD. Write the following code in Python:

\begin{lstlisting}[style=python]
U,s,Vh = la.svd(A,full_matrices=False)
S = sp.diag(s)
\end{lstlisting}
We can then calculate the pseudo-inverse by using the following line of code:

\begin{lstlisting}[style=python]
Apinv = sp.dot(sp.dot(Vh.T,sp.diag(1./s)),U.T)
\end{lstlisting}

This line uses the matrices that \li{svd} created to calculate the pseudo-inverse, using the formula we established above. Note that we used \li{diag} twice to invert just the singular values (and not the zeros).
\begin{problem}
Python has the built-in functions \li{la.pinv} and \li{la.pinv2} to calculate the pseudo-inverse. Write a script that compares the performance of \li{la.pinv},\li{la.pinv2}, and the implementation that we demonstrated above. Use a matrix generated by  \li{sp.rand}.
 Try the following matrix sizes:
\begin{itemize}
\item $1000 \times 1$
\item $1000 \times 20$
\item $1000 \times 500$
\item $1000 \times 1000$
\end{itemize}
How do the performances compare?
\end{problem}

\begin{problem}
{\bf Continuity of the pseudo-inverse:} This problem demonstrates why the pseudo-inverse is not generally a continous operation. Create a random $5 \times 5$ matrix $A$, and find its SVD using the code:
%should we use 'econ' here? I haven't read whatever section that comes from
\begin{lstlisting}[style=python]
U,s,Vh = la.svd(A,full_matrices=False)
V = Vh.T
S = sp.diag(s)
\end{lstlisting}
Set the bottom right entry of $S$ to zero, and then set $A = U*S*V'$. Now set $S(5,5)$ to be $.01$. Now calculate \li{la.norm(A - sp.dot(sp.dot(U,S),Vh))}. The value should be $.01$. This is because the matrix $2$-norm is simply the value of the largest singular value. Now calculate \li{la.norm(la.pinv(A) - la.pinv(sp.dot(sp.dot(U,S),Vh)))}. You should get $100$. Why is this? Hint: Think of how we calculated the pseudo-inverse. This example, and your explanation, should explain why two matrices, that are arbitrarily close together, can have pseudo-inverses that are arbitrarily far apart. Thus the pseudo-inverse is not a continuous operator.
\end{problem}

\subsection*{Other methods for calculating the Pseudo-Inverse}

The SVD is a powerful tool for calculating the pseudo-inverse. However, for large matrices it can be costly to calculate the SVD. Accordingly, there are several other methods for calculating the pseudo-inverse of a matrix.

One method is an iterative approach established by Ben-Israel and Cohen\footnote{Ben-Israel, Adi; Cohen, Dan (1966). "On Iterative Computation of Generalized Inverses and Associated Projections". SIAM Journal on Numerical Analysis 3: 410–419.}. This method uses the recursive sequence

\[
A_{i+1} = 2A_i - A_i A A_i
\]

With $A_0$ satisfying the equation $A_0 A = (A_0 A)^*$. The convergence of this sequence eventually becomes quadractic, which is a very desirable property. The simplest choice for $A_0$ is $\alpha A^*$, with $0 < \alpha < 2/\sigma_1^2(A)$, where $\sigma_1$ denotes the largest singular value (the condition on $\alpha$ is a technical condition established in the original paper to guarantee fast convergence).

One tricky part of developing this method is finding an appropriate $\alpha$. We don't want to waste much time calculating $\sigma_1$. Luckily there is an easy equivalence that we can leverage:

\[
||A||_2 = \sigma_1(A) = \sqrt{\lambda_{max} (A^* A)}
\]
%Python: no eigs function -- but it appears to be a topic of interest in the community, perhaps a future addition
Thus, we can calculate the largest singular value of $A$ quickly in some cases using. We stress that this is not always the fastest way. For example, if $A$ is a million by one column vector then calculating the 2-norm of $A$ is much faster than calculating the largest eigenvalue of a million by million matrix. However, with these equivalences we have some tools at our disposal to easily calculate the largest singular value of $A$.

Now, using these tools, we can write a simple function to calculate the pseudo-inverse using this iterative method:

\begin{lstlisting}[style=python]
function out = IterativePInv(A)

alpha = 2/(1.1*eigs(A'*A,1));
out = alpha*A';
C = out + 1;
while max(sum(abs(C-out))) > 1e-4
    C = out;
    out = 2*out - out*A*out;
end
\end{lstlisting}
As you can see, the implementation of this algorithm is fairly straightforward. We used the matrix $1$-norm (the maximum absolute column sum) to detect convergence of our sequence, since it is very fast to compute.

\begin{problem}
The implementation we wrote above is naive in a few ways. Make the following three adaptations to your code:
\begin{itemize}
\item Allow variable tolerance in the detection of convergence of the sequence. Use \li{*args} so that the user only specifies the tolerance if they want to.
\item We used the value $1.1$ rather arbitrarily in our selection of $\alpha$. The original paper proves that the optimal value is $2/(\sigma_1^2 + \sigma_r^2)$, where $\sigma_r$ is the smallest non-zero singular value. Use again to find the value of $\sigma_r$, and use that value to set $\alpha$.
\item If $A$ isn't close to square then it is not advantageous to use at all, as we explained above. Adapt the code to only use \li{eigs} as you think it's appropriate. Remember that you can use the matrix $2$-norm and a constant like $1.1$ in cases where  isn't appropriate.
\end{itemize}

Now test your code against the svd method, using the same cases that you did for that method. How does it perform?
\end{problem}

It should be noted that this method, although powerful, sometimes performs worse than the svd method. Specifically, when the matrix is ill-conditioned, it may take a long time for the sequence to enter the region of quadratic convergence, making this method a poor choice.

\subsection*{The QR method}

This final method, while only applicable in certain cases, offers a very fast method for calculating the pseudo-inverse. Suppose that $A$ has full column rank. We could show that the pseudo-inverse is then:
\[
A^\dagger = (A^* A)^{-1} A^*
\]

Direct calculation of the pseudo-inverse using this formula is not very helpful, since matrix inversion is so costly. However, since $A$ is full rank, then $A^*A$ is positive definite, and thus has a cholesky decomposition. Therefore, we can rewrite this equation as:

\[
R^*R A^\dagger = A^*
\]

with $R$ being upper-triangular. This equation can be solved quickly by using forward and backward substitution.

We can speed calculation even further by noting that we don't even have to calculate $A^* A$, which can be costly (again, imagine the million by one vector). We can calculate R using the QR-decomposition of A:

\[
A^* A = R^*Q^*QR = R^* R
\]

These equalities hold since $Q$ is orthogonal. Further, in the QR decomposition $R$ is upper-triangular, and thus we can calculate the cholesky factorization easily using the QR decomposition.

Combining these facts we can write the following code to calculate the pseudo-inverse for a matrix that has full column rank as follows:

\begin{lstlisting}[style=python]
R = la.qr(A)[1];

Ainv = la.lstsq(R,la.lstsq(R.T,A.T)[0])[0];
\end{lstlisting}

The first line performs QR decomposition for $R$. The second line conducts the backwards and forwards substitutions.

\begin{problem}
%This Problem did not exactly work in Python, my function was slower when A was square when using la.lstsq
Compare the run-time of this code against SciPy's \li{la.pinv}, for the values used in the other problems. Also, make sure to use a non-square matrix for A, or for square matricies change the \li{la.lstsq}'s to \li{la.solve}'s to maximize performance.  How does it compare? It should be a lot faster. This demonstrates that by leveraging the correct information (full column rank) we can significantly improve the speed of this operation. Note that we can beat SciPy's own implementation significantly in this case.

\end{problem}

\begin{problem}
This method can also be adapted to use for matrices that have full row rank. In the case of full column rank the following formula holds:
\[
A^\dagger = A^*(A A^*)^{-1}
\]
Use the same technique to write a function that finds the pseudo-inverse for a matrix of full row rank. Note that you will have to find the QR-decomposition of $A^*$ this time, and that it may be helpful to take the transpose so that you can use the backslash operator.
\end{problem}