%----------------------------------------------------------------------------------------
% What is RBI
%----------------------------------------------------------------------------------------

\section{What is Randomization Based Inference}
\subsection{Definition}
The general explanation goes something like this: If the data we are analyzing falls to meet our model conditions but passes the randomness assumption, we can randomly “shuffle” or permute our response variable (or treatment groups, or our multiple explanatory variables, etc.) $n$ times to estimate a distribution of test statistics. From this distribution, we calculate the number of test statistics that are as extreme or more extreme than our original test statistic, coming from the original model we created. This proportion is then our p-value. We can trust this p-value and use it to make a judgement on whatever to reject or fail to reject our null hypothesis.
\newline\\
Algorithmically, suppose we have some randomly sampled dataset, $A$, with test statistic, $S_0$. We can make $J$-many permutations of the column of responses in $A$ where these permuted matrices are $A_j^*$ with test statistic $S_j$.
$$
    A=\begin{pmatrix}
        x_{1,1} & x_{1,2} & \cdots & y_{1} \\
        x_{2,1} & x_{2,2} & \cdots & y_{2} \\
        \vdots  & \vdots  & \ddots & \vdots  \\
        x_{n,1} & x_{n,2} & \cdots & y_{n}
    \end{pmatrix}\longrightarrow
    A_j^*=\begin{pmatrix}
        x_{1,1} & x_{2,2} & \cdots & y_{i} \\
        x_{2,1} & x_{2,2} & \cdots & y_{i} \\
        \vdots  & \vdots  & \ddots & \vdots  \\
        x_{n,1} & x_{n,2} & \cdots & y_{i}
    \end{pmatrix}
$$
\\
Where our p-value is calculated as

$$
\hat{p}=\frac{\sum_{j=1}^{J}I(|S_j|\geq|S_0|)}{J} % add source from the article for this
$$
\\
\subsection{In Practice}
There is often more to this picture than just the above definition. Different kind of data require different kinds of permutation methods to perform statistical inference. Take a two-sample t-test for example.  We start with the sampling process. Let us imagine that we have two treatments, $A$ and $B$ with $n_1$ and $n_2$ randomly assigned patients, respectively.
\\
$$
    A=\begin{pmatrix}
        x_{1}\\
        x_{2}\\
        \vdots\\
        x_{n_1}
    \end{pmatrix}\quad\quad
    B=\begin{pmatrix}
        x_{n_1+1}\\
        x_{n_1+2}\\
        \vdots\\
        x_{n_1+n_2}
    \end{pmatrix}
$$
\newline\\
If from here we randomly sample $n_1$ patients into $A$ without replacement and put the rest into $B$, we will have ensured that our new data is random as the treatments were randomly assigned in our initial sample. That is, under the assumption of the null hypothesis, if there was no difference in treatments $A$ and $B$, we could assign each data point from the whole study to each treatment randomly.
\\
$$
    A^*=\begin{pmatrix}
        x_{k}\\
        x_{k}\\
        \vdots\\
        x_{k}
    \end{pmatrix}\quad\quad
    B^*=\begin{pmatrix}
        x_{k}\\
        x_{k}\\
        \vdots\\
        x_{k}
    \end{pmatrix}
$$
\newline\\
From here, we don’t have to check our model conditions anymore because we have committed to moving forward with randomization based inference. The next step in the process will be to calculate our test statistic from this permutation of the data. For this example with two treatments, we could calculate a difference in sample means, generating a t-statistic. Following this, we repeat the initial sampling procedure and extract a difference in means t-statistic $N$-many times.
\\
$$
    A_n^*=\begin{pmatrix}
        x_{k}\\
        x_{k}\\
        \vdots\\
        x_{k}
    \end{pmatrix}\quad\quad
    B_n^*=\begin{pmatrix}
        x_{k}\\
        x_{k}\\
        \vdots\\
        x_{k}
    \end{pmatrix}
$$
\begin{center}
    For $k\in \{1,2,3\dots,n_1+n_2\}$ for $N$-many permutations.
\end{center}
\newline\\
When this process is done, we take our t-statistics and plot them on a dotplot. The result we get is called a randomization or reference distribution. Consequently, that distribution is centered on the null hypothesis of the given test we are conducting meaning that the reference distribution we created is also a null distribution. Using this distribution, we locate where our original test statistic calculated using $A$ and $B$ would fall on this distribution. We then count the number of new test statistics that are as or more extreme than the original one and divide that by the number of total test statistics calculated. This proportion that is yielded is our p-value. The error rate of this p-value has been maintained at the original significance level we started with as well. We can then take this p-value and use it to either reject or fail to reject the null hypothesis of the test we conducted.


Like we introduced Lasso and Ridge regression to avoid over fitting in Project 1, we can add a penalty term to the cost function in equation \eqref{eq:Geron_cost}.

\subsubsection{\texorpdfstring{$L^1$}{L1} regularization}
The $L^1$ regularization utilizes the Taxi-cab metric,
\begin{align*}
    ||\bm{a}||_1 = |a_0| + \dots + |a_{n-1}| = \sum^{n-1}_{i=0}|a_i|,
\end{align*}
and is defined as
\begin{align}
    \lambda||\bm{w}||_1,
    \label{eq:l1-reg}
\end{align}
with $\bm{w}$ being the weight matrix. This is equivalent with $\beta$ as seen in the logistic regression. Its derivative is given as,
\begin{align}
    \lambda \sign{(\bm{w})}
    \label{eq:l1-reg-derivative}
\end{align}
where $\sign$ is simple the sign of $\bm{w}$.

For logistic regression, this becomes in the gradient of the Geron cost function \eqref{eq:Geron_cost},
\begin{align}
    \frac{\partial J(\beta)}{\partial \beta} &=\frac{1}{N}\bm{X}^T(\bm{p}-\bm{y})\ + \lambda\cdot{\sign(\beta)}
    \label{eq:geron-cost-l1}
\end{align}

% In our project we used two different penalties: $L^1 = \lambda|\beta|$ and $L^2 = \lambda||\beta||^2$. When fitting the model we need to include the derivatives of the penalty term in Equation \ref{eqT:diff_cost}.} \husk{The gradient with the penalty term is,
% \begin{equation}
% \begin{split}
% \frac{\partial J(\beta)}{\partial \beta} &=\frac{1}{N}\bm{X}^T(\bm{p}-\bm{y})\ + \lambda\cdot{sign(\beta)},\ for\ L1\ regularization\\
%  &\mathrm{or}\\
% \frac{\partial J(\beta)}{\partial \beta} &=\frac{1}{N}\bm{X}^T(\bm{p}-\bm{y})\ + \lambda\cdot{2\beta},\ for\ L2\ regularization.
% \end{split}
% \end{equation}
% }

\subsubsection{\texorpdfstring{$L^2$}{L2} regularization}
The $L^2$ regularization is given as the Euclidean norm of the weight matrix,
\begin{align}
    ||\bm{a}||_2 = \left(\sum^{n-1}_{i=0} a_i^2 \right)^{1/2},
\end{align}
and is given as
\begin{align}
    \lambda||\bm{w}||^2_2,
    \label{eq:l2-reg}
\end{align}
with its following derivative
\begin{align}
    \lambda\cdot{2\bm{w}}.
    \label{eq:l2-reg-derivative}
\end{align}
The 2 in front is often offset by redefining the $L^2$ norm with a factor half, and will not affect the final outcome.

Implementing the $L^2$ norm in the gradient of the Geron cost function\eqref{eq:Geron_cost}, we get
\begin{align}
    \frac{\partial J(\beta)}{\partial \beta} &=\frac{1}{N}\bm{X}^T(\bm{p}-\bm{y})\ + \lambda\cdot{2\beta}
    \label{eq:geron-cost-l2}
\end{align}

\subsubsection{Elastic net regularization}
Elastic net regularization utilizes a linear combination of $L^1$ and $L^2$ regularization, and consists of adding a term
\begin{align}
    \lambda_1 ||\bm{w}||_1 + \lambda_2 + ||\bm{w}||_2^2
    \label{eq:elastic_net}
\end{align}
The derivative of this w.r.t. $\bm{w}$ is simply the combined derivatives of $L^1$ and $L^2$. For our purposes we will set $\lambda_1=\lambda_2$ in order to avoid having the hyper parameter space become too large.


\subsection{Logistic distribution}

The logistic distribution has the cumulative distribution function:

\(F(x)=\dfrac{1}{1+e^{-\dfrac{x-\mu }{s}}} \)


\section*{Exercises}

\begin{ex}
  For each matrix $A$, find the products $(-2)A$, $0A$, and $3A$.
  \begin{enumerate}
  \item
    $A = \begin{mymatrix}{rr}
      1 & 2 \\
      2 & 1
    \end{mymatrix}$
  \item
    $A = \begin{mymatrix}{rr}
      -2 & 3 \\
      0 & 2
    \end{mymatrix}$
  \item
    $A = \begin{mymatrix}{rrr}
      0 & 1 & 2 \\
      1 & -1 & 3 \\
      4 & 2 & 0
    \end{mymatrix}$
  \end{enumerate}
  % \begin{sol}
  % \end{sol}
\end{ex}

\begin{ex}
  Find scalars $x,y,z,w$ such that
  \begin{equation*}
    x\begin{mymatrix}{rr} 1 & 2 \\ 3 & 5\end{mymatrix}
    ~+~ y\begin{mymatrix}{rr} 0 & 1 \\ 1 & 1\end{mymatrix}
    ~+~ z\begin{mymatrix}{rr} 0 & 2 \\ -1 & 2\end{mymatrix}
    ~+~ w\begin{mymatrix}{rr} 1 & 0 \\ 0 & 2\end{mymatrix}
    ~=~ \begin{mymatrix}{rr} 1 & 2 \\ 6 & 5\end{mymatrix}.
  \end{equation*}
  \begin{sol}
    By definition of scalar multiplication, this is equivalent to the
    matrix equation
    \begin{equation*}
      \begin{mymatrix}{rr} x+0y+0z+w & 2x+y+2z+0w \\ 3x+y-z+0w & 5x+y+2z+w\end{mymatrix}
      ~=~ \begin{mymatrix}{rr} 1 & 2 \\ 6 & 5\end{mymatrix}.
    \end{equation*}
    By definition of equality of matrices, this is equivalent to the
    system of four scalar equations
    $x+0y+0z+w=1$, $2x+y+2z+0w=2$, $3x+y-z+0w=6$,
    $5x+y+2z+w=5$. Solving the system of equations, we find $x=1$,
    $y=2$, $z=-1$, $w=0$.
  \end{sol}
\end{ex}

\begin{ex}
  Using only the properties given in
  Propositions~\ref{prop:properties-of-addition} and
  {\ref{prop:properties-scalar-multiplication}}, show that $0A=0$. Here the $0$
  on the left is the scalar $0$ and the $0$ on the right is the zero
  matrix of appropriate size.
  \begin{sol}
    $0A=(0+0)A=0A+0A$. Now add $-(0A)$ to both sides. Then $0=0A$.
  \end{sol}
\end{ex}


\section{Introduction}
We assume basic knowledge of commutative algebra. Let $R$ be a unital commutative ring.
\subsection{Algebras}
\dfi{Algebras}{Let $A$ be an $R$ module. We call $A$ an $R$ algebra if there is an $R$ linear map
$$\mu : A \otimes_R A \longrightarrow A.$$
We call $A$ associative if the following diagram commutes:
$$\xymatrix{A \otimes_R A \otimes_R A\ar[rr]^{\mu\otimes id_A}\ar[d]_{id_A \otimes \mu}&&A\otimes_R A\ar[d]^\mu\\A\otimes_R A \ar[rr]_{\mu}&&A.\\}$$
We call $A$ unital if there is an $R$ linear map $\eta : R \longrightarrow A$ such that
$$\xymatrix{R\otimes_R A \ar[rr]^{\eta \otimes id_A}\ar[rrd]_\simeq&&A \otimes_R A\ar[d]^\mu&&A \otimes_R R\ar[ll]_{id_A \otimes \eta}\ar[lld]^\simeq\\&&A&&\\}$$ commutes. We call $A$ commutative if
$$\xymatrix{A \otimes_R A \ar[r]^\tau\ar[rd]_\mu&A\otimes_R A\ar[d]^\mu\\&A\\}$$commutes. Here $\tau = \tau_{A\otimes A} : A \otimes_RA \longrightarrow A \otimes_RA,\ x \otimes y \longmapsto y \otimes x$ is the flip isomorphism. A unital associative algebra $A$ is denoted by $(A, \mu, \eta)$.
}
\subsubsection{Examples of algebras}
Clearly, very ring $R$ itself is an $R$ algebra. But more prominent examples are
\paragraph{Polynomial ring}


\subsection{Algebra on a set}

\subsubsection{Standard algebra}

An algebra, \(\Sigma \), on set \(s\) is a set of subsets of \(s\) such that:

\begin{itemize}
\item Closed under intersection: If \(a\) and \(b\) are in \(\Sigma \) then \(a\land b\) must also be in \(\Sigma \)
\item \(\forall ab [(a \in \Sigma \land b \in \Sigma )\rightarrow (a\land b \in \Sigma)]\)
\item Closed under union: If \(a\) and \(b\) are in \(\Sigma \) then \(a\lor b\) must also be in \(\Sigma \).
\item \(\forall ab [(a \in \Sigma \land b \in \Sigma )\rightarrow (a\lor b \in \Sigma)]\)
\end{itemize}

If both of these are true, then the following is also true:

\begin{itemize}
\item Closed under complement: If \(a\) is in \(\Sigma \) then \(s \backslash a\) must also be in \(\Sigma \)
\end{itemize}

We also require that the null set (and therefore the original set, null's complement) is part of the algebra.


\section{Prove circle area using integrals in Cartesian plane}
From what had been taught in a Geometry class, the area of a circle can be
calculated using $A=\pi r^2$, and it is possible to be proven using different
approaches.

As we know, circle in Cartesian plane is defined by $x^2+y^2=r^2$, where $r$
determines the radius.

\begin{figure}[H]
	\centering{
	\begin{tikzpicture}
		\draw[->] (-3,0) -- (3,0) node[right] {$x$};
		\draw[->] (0,-3) -- (0,3) node[above] {$y$};
		\draw[domain=0:6.28318,smooth,variable=\x,blue]
		plot ({2*cos(\x r)},{2*sin(\x r)});
	\end{tikzpicture}
	}
	\caption{Graph of Cartesian equation $x^2+y^2=2^2$}
	\label{circ}
\end{figure}

In this case when we solve for $y$ we can get:

$$y=\pm \sqrt{r^2-x^2}$$

Apparently, it is not a valid function, so we ignore the negative part of the
root; we eventually find the function for a semi-circle:

$$f(x)=\sqrt{r^2-x^2}$$

So if we can integrate this semi-circle, we are able to find the area of one
circle by multiplying it by a factor of $2$.

Let's begin with the integration of $f(x)$:

$$\int_{-r}^r f(x)dx=\int_{-r}^r \sqrt{r^2-x^2}dx$$

Before dealing with the bounds of the definite integral, we first cope with the
\textbf{antiderivative} of this function. When integrating with square roots, I
always think of changing the constant inside the root into one. So let's move
$r$ out of the integral:

$$\int_{-r}^r\sqrt{r^2-x^2}dx=r\cdot\int_{-r}^r\sqrt{1-(\frac{x}r)^2}dx$$

When I see one minus square of a term under a radical sign, I always think of
the \textbf{Pythagorean identity of trignonometric functions}. Therefore we can
do substitution this way:

$$
\begin{aligned}
	\frac{x}r=\sin(t) \\
	\therefore x=r\sin(t) \\
	\therefore dx=r\cos(t)dt
\end{aligned}
$$

We also need to have a formula to convert $t$ to $x$ in order to handle the
integral bounds, so we solve for $t$ below:

\begin{equation}
	t=\arcsin(\frac{x}r)
	\label{subst-back}
\end{equation}

Since we have the substitute for both $x$ and $dx$, we can plug them into the
original integral:

$$
\begin{aligned}
	r\cdot\int_{t_0}^{t_1}\sqrt{1-(\frac{x}r)^2}dx
	&=r\cdot\int_{t_0}^{t_1}\sqrt{1-\sin(t)^2}r\cos(t)dt \\
	&=r^2\int_{t_0}^{t_1}\cos(t)^2dt
\end{aligned}
$$

This time we need to eliminate the square of cosine. Since we know the
\textbf{double-angle identity of cosine} is $\cos(2\theta)=2\cos(\theta)^2-1$,
we solve for $\cos(\theta)^2$ and apply another substitution:

$$\cos(\theta)^2=\frac{\cos(2\theta)+1}2$$

$$
\begin{aligned}
	r^2\int_{t_0}^{t_1}\cos(t)^2dt
	&=r^2\int_{t_0}^{t_1}\frac{\cos(2t)+1}2dt \\
	&=\frac{1}4r^2\int_{t_0}^{t_1}(2\cos(2t)+2)dt \\
	&=\frac{1}4r^2\left[\sin(2t)+2t\right]^{t_1}_{t_0} \\
\end{aligned}
$$

Congratulations! We have successfully figured out the \textbf{antiderivative}
of this semi-circle function. It is time to cope with the \textbf{definite
integral}. Instead of making the antiderivative in terms of $x$, we can
convert the bound using Equation \ref{subst-back}

$$
\begin{aligned}
	t_0=\arcsin(\frac{-r}r)=-\frac{\pi}2 \\
	t_1=\arcsin(\frac{r}r)=\frac{\pi}2 \\
	\therefore r\cdot\int_{-r}^{r}\sqrt{1-(\frac{x}r)^2}dx
	&=r^2\cdot\int_{-\frac{\pi}2}^{\frac{\pi}2}\cos(t)^2dt \\
	&=\frac{1}4r^2\left[\sin(2t)+2t\right]^{t=\frac{\pi}2}
	_{t=-\frac{\pi}2} \\
	&=\frac{1}4r^2(2\pi) \\
	&=\frac{1}2\pi r^2
\end{aligned}
$$

We are not finished yet since this is the formula for the area of a
semi-circle; if we want full circle, we shall double it. Eventually the
formula to find the area of a circle is:

$$A=2\cdot\frac{1}2\pi r^2=\pi r^2$$


\subsection{Linear dependence}

A collection of vectors in a vector space are linearly dependent if there exist values for \(\alpha \) (other than all being \(0\)) such that:

\(\sum_i \alpha_i v_i =0\).

If no such values for \(\alpha \) exist we say the vectors are linearly independent.


%================================
\section{Homeomorphisms}
%================================

%--------------------------------
\begin{definition}
	[homeomorphisms]
	\label{def: homomorphisms}
	Let $(X, \mathcal T_X)$ and $(Y, \mathcal T_Y)$ be topological spaces. A bijection $f: X \to Y$ is called a \textit{homeomorphism} iff
	\begin{enumerate}[(i)]
		\item $f$ is a bijection;
		\item $f$ is continuous;
		\item $f^{-1}$ is continuous.
	\end{enumerate}
\end{definition}
%--------------------------------

%--------------------------------
\begin{definition}
	[homeomorphic]
	\label{def: homomorphic}
	Two topological spaces $(X, \mathcal T_X)$ and $(Y, \mathcal T_Y)$ are said to be \textit{homeomorphic} or \textit{topologically equivalent}, denoted $X \cong Y$, iff there is an homeomorphism between them.
\end{definition}
%--------------------------------

%--------------------------------
\begin{proposition}
	Two topological spaces are homeomorphic only if they have the same cardinality.

	\begin{proof}
		Let $X$ and $Y$ be two sets with $|X| < |Y|$. There is no surjection from $A$ to $B$.
	\end{proof}
\end{proposition}
%--------------------------------

%--------------------------------
\begin{example}
	$|X| = |Y|$ does not imply $(X, \mathcal T_X)$ and $(Y, \mathcal T_Y)$ are homeomorphic, even if they are finite. For example, let $X = Y = \{1, \ldots, n\}$, and let $\mathcal T_X$ be indiscrete topology on $X$ and $\mathcal T_Y = \mathcal P(X)$. There is no homeomorphism between $X$ and $Y$.

	On the other hand, even if $|X| = |Y| \ge \aleph_0$ and $\mathcal T_X$ and $\mathcal T_Y$ are induced by same metric, $X$ and $Y$ might not be homeomorphic. For example, if $\mathcal T_X$ and $\mathcal T_Y$ are both induced by standard Euclidean metric, and $X = [a, b] \subseteq \mathbb R$ and $Y = [c, d) \subseteq \mathbb R$ where $a < b$ and $c < d$. No doubt, $|X| = |Y| = \mathfrak c$, but $X$ and $Y$ are not homeomorphic.
\end{example}
%--------------------------------

%--------------------------------
\begin{example}
	$\mathbb R^n$ and $\mathbb R^m$ ($n < m$) are not homeomorphic, although $|\mathbb R^n| = |\mathbb R^m|$.
\end{example}
%--------------------------------

%--------------------------------
\begin{example}
	Let $I$ be a proper interval in $\mathbb R^n$, let $\mathcal T$ be standard Euclidean topology on $\mathbb R^n$ and let $\mathcal T_I$ be a subspace topology on $I$. $I \cong \mathbb R^n$ iff $I$ is an open interval.

	But if $\mathcal T = \mathcal P(\mathbb R^n)$, then there exists bijection $f: I \to \mathbb R^n$, for $|I| = |\mathbb R^n|$, and such $f$ can be bicontinuous, for any subset $A \subseteq I$ is also open in $\mathbb R^n$, vise versa. In this case, $I \cong \mathbb R^n$ whenever $I$ is a closed, half-close, half-open, or open interval respect to standard Euclidean metric.
\end{example}
%--------------------------------

%--------------------------------
\begin{example}
	Let $S^n$ be an $n$-dimensional sphere with center $o \in \mathbb R^{n + 1}$ and radius $r \in \mathbb R$, i.e.,
	$$
	S^n = \left\{ x \in \mathbb R^{n + 1} : \rho(o, x) = r \right\},
	$$
	where $\rho$ is the standard Euclidean metric on $\mathbb R^{n+1}$. For any $x \in S^n$, let $U = B(x, \varepsilon) \cap S^n$ where $0\le \varepsilon < \max_{x,y \in S^n} \rho(x,y)$ (here $B(x, \varepsilon) = \{x\}$ if $\varepsilon = 0$), then $S^n \setminus U \cong \mathbb R^{n}$.
\end{example}
%--------------------------------

%--------------------------------
\begin{example}
	Indeed, $S^1 \setminus \{x\} \cong \mathbb R$ where $x \in S^1$. But for any interval $I \in \mathbb R$, $S^1 \not \cong I$.
	% todo: explain!
\end{example}
%--------------------------------

%


\subsection{Identity matrix and the Kronecker delta}
\subsubsection{The Kronecker delta}

The Kronecker delta is defined as:

p\(\delta_{ij}=0\) where \(i\ne j\)

\(\delta_{ij}=1\) where \(i=j\)

We can use this to define matrices. For example for the identity matrix:

\(I_{ij}=\delta_{ij}\)

\subsubsection{Identity matrix}

A square matrix where every element is \(0\) except where \(i=j\). There is one for each square matrix.

\(I=\begin{bmatrix}1& 0&...&0\\0 & 1&...&0\\...&...&...&...\\0&0&...&1\end{bmatrix}\)


\section{Convergence in Probability}%
\label{sec:convergence_in_probability}

\begin{definition}[Convergence in Probability]
    $Y_n \convp c$ if for every $\epsilon>0$ and $\delta > 0,\
    \exists\ n_0(\epsilon, \delta)$ such that
    \begin{equation*}
        P(|Y_n - c| > \epsilon) < \delta,\
        \forall n > n_0(\epsilon, \delta)
    \end{equation*}
\end{definition}

\begin{thm}[Chebyshev Inequality]
For random variable, $Y$, $a>0$, and $c$,
    \begin{equation*}
        P(|Y-c| \ge a) \le \frac{\E (Y-c)^2}{a^2}
    \end{equation*}
\end{thm}

\begin{definition}[Markov Inequality]
    If $X$ is a non-negative random variable and $a>0$ then
    \begin{equation*}
        P( X \ge a) \le \frac{\E X}{a}
    \end{equation*}
\end{definition}

\begin{thm}
    If $\E (Y-c)^2 \to 0$, then $Y_n \convp c$.
\end{thm}

\begin{thm}
    If $X_1, \ldots, X_n$ iid, $\E X_i = \mu$, $\Var X_i = \sigma^2 < \infty$, then
    \begin{equation*}
        \bar{X} \convp \mu
    \end{equation*}
\end{thm}

\begin{thm}
    If $A_n \convp a$ and $B_n \convp b$, then
    \begin{enumerate}
        \item $A_n \pm B_n \convp a \pm b$,
        \item $A_n \cdot B_n \convp a \cdot b$.
    \end{enumerate}
\end{thm}

\begin{thm}
    If $Y_n \convp c$ and $f$ is continuous at $c$, then $f(Y_n) \convp f(c)$.
\end{thm}

\begin{definition}
    A sequence of estimators $\delta_n$ of $g(\theta)$ is \it{consistent} if
    \begin{equation*}
        \delta_n \convp g(\theta)
    \end{equation*}
\end{definition}

\begin{thm}
    If bias and variance of $\delta_n \to 0$ as $n \to \infty$, $\delta_n$
    is consistent.
\end{thm}

\begin{definition}
    $A_n = o_p (B_n)$ if $ \frac{A_n}{B_n} \convp 0$.
\end{definition}


\chapter{Large number laws (TO DO)}
\todo{write chapter}
\section{Notions of convergence}
\subsection{Almost sure convergence}
\begin{definition}
	Let $X$, $X_n$ be random variables on a probability space $\Omega$.
	We say $X_n$ \vocab{converges almost surely} to $X$ if
	\[ \mu \left( \omega \in \Omega :
		\lim_n X_n(\omega) = X(\omega) \right) = 1. \]
\end{definition}
This is a very strong notion of convergence:
it says in almost every \emph{world},
the values of $X_n$ converge to $X$.
In fact, it is almost better for me to give a \emph{non-example}.
\begin{example}
	[Non-example of almost sure convergence]
	Imagine an immortal skeleton archer is practicing shots,
	and on the $n$th shot, he scores a bulls-eye with probability
	$1 - \frac 1n$
	(which tends to $1$ because the archer improves over time).
	Let $X_n \in \{0, 1, \dots, 10\}$ be the score of the $n$th shot.

	Although the skeleton is gradually approaching perfection,
	there are \emph{almost no worlds} in which the archer
	misses only finitely many shots: that is
	\[ \mu \left( \omega \in \Omega :
		\lim_n X_n(\omega) = 10 \right) = 0. \]
\end{example}

\subsection{Convergence in probability}
Therefore, for many purposes we need a weaker notion of convergence.
\begin{definition}
	Let $X$, $X_n$ be random variables on a probability space $\Omega$.
	We say $X_n$ \vocab{converges in probability} to $X$ if
	if for every $\eps > 0$ and $\delta > 0$, we have
	\[ \mu \left( \omega \in \Omega :
			\left\lvert X_n(\omega) - X(\omega) \right\rvert < \eps
		\right) \ge 1 - \delta  \]
	for $n$ large enough (in terms of $\eps$ and $\delta$).
\end{definition}
In this sense, our skeleton archer does succeed:
for any $\delta > 0$, if $n > \delta\inv$
then the skeleton archer does hit a bulls-eye
in a $1-\delta$ fraction of the worlds.
In general, you can think of this as saying that for any $\delta > 0$,
the chance of an $\eps$-anomaly event at the $n$th stage
eventually drops below $\delta$.

\begin{remark}
	To mask $\delta$ from the definition,
	this is sometimes written instead as:
	for all $\eps$
	\[ \lim_{n \to \infty} \mu \left( \omega \in \Omega :
		\left\lvert X_n(\omega) - X(\omega) \right\rvert < \eps
		\right) = 1. \]
	I suppose it doesn't make much difference,
	though I personally don't like the asymmetry.
\end{remark}

\subsection{Convergence in law}

\section{\problemhead}
\begin{problem}
	[Quantifier hell]
	\gim
	In the definition of convergence in probability
	suppose we allowed $\delta = 0$
	(rather than $\delta > 0$).
	Show that the modified definition is
	equivalent to almost sure convergence.
	\begin{hint}
		This is actually trickier than it appears,
		you cannot just push quantifiers (contrary to the name),
		but have to focus on $\eps = 1/m$ for $m = 1, 2, \dots$.

		The problem is saying for each $\eps > 0$,
		if $n > N_\eps$, we have
		$\mu(\omega : |X(\omega)-X_n(\omega)| \le \eps) = 1$.
		For each $m$ there are some measure zero ``bad worlds'';
		take the union.
	\end{hint}
	\begin{sol}
		For each positive integer $m$,
		consider what happens when $\eps = 1/m$.
		Then, by hypothesis, there is a threshold $N_m$
		such that the \emph{anomaly set}
		\[ A_m \defeq \left\{ \omega :
			|X(\omega)-X_n(\omega)| \ge \frac 1m
			\text{ for some } n > N_m \right\} \]
		has measure $\mu(A_m) = 0$.
		Hence, the countable union $A = \bigcup_{m \ge 1} A_m$ has measure zero too.

		So the complement of $A$ has measure $1$.
		For any world $\omega \notin A$,
		we then have
		\[ \lim_n \left\lvert X(\omega) - X_n(\omega) \right\rvert = 1 \]
		because when $n > N_m$ that absolute value
		is always at most $1/m$ (as $\omega \notin A_m$).
	\end{sol}
\end{problem}

\begin{problem}
	[Almost sure convorgence is not topologizable]
	Consider the space of all random variables on $\Omega = [0,1]$.
	Prove that it's impossible to impose a metric on this space
	which makes the following statement true:
	\begin{quote}
		A sequence $X_1$, $X_2$, \dots, of converges almost surely to $X$
		if and only if $X_i$ converge to $X$ in the metric.
	\end{quote}
	\begin{sol}
		\url{https://math.stackexchange.com/a/2201906/229197}
	\end{sol}
\end{problem}


\subsection{Introduction to unconstrained optimisation}

\subsubsection{Goals}

We want to identify either the maximum or the minimum.

There exist local minima and global minima.

\subsubsection{Optimising through limits}

If we are looking to minimise a function, and the limits are \(\infty \) or \(-\infty \) then we can optimise by taking large or small values.

We can examine this for each variable.

This also applies for maximising a function.

\subsubsection{Optimisation through stationary points}

Stationary points of a function are points where marginal changes do not have an impact on the value of the function. As a result they are either local maxima or minima.

\subsubsection{Optimisation through algorithms}

If we cannot identify stationary points easily, we can instead use algorithms to identify optima.

\subsubsection{Stationary points of strictly concave and convex functions}

If a function is strictly concave it will only have one stationary point, a local, and global, maxima.

If a function is strictly convex it will only have one stationary point, a local, and global, minima.


\title{Supervised Learning (Regression)}

\subsection{Supervised Learning (Regression)}

In supervised learning, the task is to infer hidden structure from
labeled data, comprised of training examples $\{(x_n, y_n)\}$.
Regression typically means the output $y$ takes continuous values.

We demonstrate with an example in Edward.
An interactive version with Jupyter notebook is available
\href{http://nbviewer.jupyter.org/github/blei-lab/edward/blob/master/notebooks/supervised_regression.ipynb}{here}.

\subsubsection{Data}

Simulate training and test sets of $40$ data points. They comprise of
pairs of inputs $\mathbf{x}_n\in\mathbb{R}^{10}$ and outputs
$y_n\in\mathbb{R}$. They have a linear dependence with normally
distributed noise.

\begin{lstlisting}[language=Python]
def build_toy_dataset(N, w, noise_std=0.1):
  D = len(w)
  x = np.random.randn(N, D)
  y = np.dot(x, w) + np.random.normal(0, noise_std, size=N)
  return x, y

N = 40  # number of data points
D = 10  # number of features

w_true = np.random.randn(D)
X_train, y_train = build_toy_dataset(N, w_true)
X_test, y_test = build_toy_dataset(N, w_true)
\end{lstlisting}

\subsubsection{Model}

Posit the model as Bayesian linear regression \citep{murphy2012machine}.
It assumes a linear relationship between the inputs
$\mathbf{x}\in\mathbb{R}^D$ and the outputs $y\in\mathbb{R}$.

For a set of $N$ data points $(\mathbf{X},\mathbf{y})=\{(\mathbf{x}_n, y_n)\}$,
the model posits the following distributions:
\begin{align*}
  p(\mathbf{w})
  &=
  \text{Normal}(\mathbf{w} \mid \mathbf{0}, \sigma_w^2\mathbf{I}),
  \\[1.5ex]
  p(b)
  &=
  \text{Normal}(b \mid 0, \sigma_b^2),
  \\
  p(\mathbf{y} \mid \mathbf{w}, b, \mathbf{X})
  &=
  \prod_{n=1}^N
  \text{Normal}(y_n \mid \mathbf{x}_n^\top\mathbf{w} + b, \sigma_y^2).
\end{align*}
The latent variables are the linear model's weights $\mathbf{w}$ and
intercept $b$, also known as the bias.
Assume $\sigma_w^2,\sigma_b^2$ are known prior variances and $\sigma_y^2$ is a
known likelihood variance. The mean of the likelihood is given by a
linear transformation of the inputs $\mathbf{x}_n$.

Let's build the model in Edward, fixing $\sigma_w,\sigma_b,\sigma_y=1$.
\begin{lstlisting}[language=Python]
from edward.models import Normal

X = tf.placeholder(tf.float32, [N, D])
w = Normal(loc=tf.zeros(D), scale=tf.ones(D))
b = Normal(loc=tf.zeros(1), scale=tf.ones(1))
y = Normal(loc=ed.dot(X, w) + b, scale=tf.ones(N))
\end{lstlisting}
Here, we define a placeholder \texttt{X}. During inference, we pass in
the value for this placeholder according to data.

\subsubsection{Inference}

We now turn to inferring the posterior using variational inference.
Define the variational model to be a fully factorized normal across
the weights.
\begin{lstlisting}[language=Python]
qw = Normal(loc=tf.Variable(tf.random_normal([D])),
            scale=tf.nn.softplus(tf.Variable(tf.random_normal([D]))))
qb = Normal(loc=tf.Variable(tf.random_normal([1])),
            scale=tf.nn.softplus(tf.Variable(tf.random_normal([1]))))
\end{lstlisting}

Run variational inference with the Kullback-Leibler divergence, using
$250$ iterations and $5$ latent variable samples in the algorithm.
\begin{lstlisting}[language=Python]
inference = ed.KLqp({w: qw, b: qb}, data={X: X_train, y: y_train})
inference.run(n_samples=5, n_iter=250)
\end{lstlisting}
In this case \texttt{KLqp} defaults to minimizing the
$\text{KL}(q\|p)$ divergence measure using the reparameterization
gradient.
For more details on inference, see the \href{/tutorials/klqp}{$\text{KL}(q\|p)$ tutorial}.

\subsubsection{Criticism}

A standard evaluation for regression is to compare prediction accuracy on
held-out ``testing'' data. We do this by first forming the posterior predictive
distribution.
\begin{lstlisting}[language=Python]
y_post = ed.copy(y, {w: qw, b: qb})
# This is equivalent to
# y_post = Normal(loc=ed.dot(X, qw) + qb, scale=tf.ones(N))
\end{lstlisting}

With this we can evaluate various quantities using predictions from
the model (posterior predictive).
\begin{lstlisting}[language=Python]
print("Mean squared error on test data:")
print(ed.evaluate('mean_squared_error', data={X: X_test, y_post: y_test}))

print("Mean absolute error on test data:")
print(ed.evaluate('mean_absolute_error', data={X: X_test, y_post: y_test}))
\end{lstlisting}

\begin{lstlisting}
## Mean squared error on test data:
## 0.0300492
## Mean absolute error on test data:
## 0.123616
\end{lstlisting}

The trained model makes predictions with low error
(relative to the magnitude of the output).

We can also visualize the fit by comparing data generated with the
prior to data generated with the posterior (on the first feature
dimension).

\begin{lstlisting}[language=Python]
def visualise(X_data, y_data, w, b, n_samples=10):
  w_samples = w.sample(n_samples)[:, 0].eval()
  b_samples = b.sample(n_samples).eval()
  plt.scatter(X_data[:, 0], y_data)
  inputs = np.linspace(-8, 8, num=400)
  for ns in range(n_samples):
    output = inputs * w_samples[ns] + b_samples[ns]
    plt.plot(inputs, output)
\end{lstlisting}

\begin{lstlisting}[language=Python]
# Visualize samples from the prior.
visualise(X_train, y_train, w, b)
\end{lstlisting}

\includegraphics[width=450px]{/images/supervised-regression-fig0.png}

\begin{lstlisting}[language=Python]
# Visualize samples from the posterior.
visualise(X_train, y_train, qw, qb)
\end{lstlisting}

\includegraphics[width=450px]{/images/supervised-regression-fig1.png}

The model has learned a linear relationship between the
first dimension of $\mathbf{x}\in\mathbb{R}^D$ and the outputs
$y\in\mathbb{R}$.

\subsubsection{References}\label{references}


\chapter{Discrete-time Signals}

Recall from the previous meeting that a discrete-time (DT) signal \index{DT Signal} is modeled as a function $f: \mathbb{Z} \rightarrow \mathbb{C}$. We will write these as $x[n]$, $y[n]$, etc. Note $n$ is dimensionless. These are graphically plotted as stem or "lollipop" plots, as demonstrated in Fig.~\ref{fig:dtplots}.

Since the domain $\mathbb{Z}$ is usually interpreted as a time index, we will still call these {\it time-domain} signals. In the time-domain, when the co-domain is $\mathbb{R}$ we call these real DT signals. Unlike with CT signals there are no physical limitations requiring DT signals to be real, since in discrete hardware, a value at a given index can be a complex number, i.e. just a pair of numbers. However it is computationally advantageous to restrict ourselves to real arithmetic and such signals are often converted to or from CT signals, which do have to be real. For this reason real DT signals dominate in models.

\section{Primitive Models}

As with CT signals, we mathematically model DT signals by combining elementary/primitive functions, for example:
\begin{itemize}
\item polynomials: $x[n] = n$, $x[n] = n^2$, etc.
\item transendental functions: $x[n] = e^n$, $x[n] = \sin(n)$, $x[n] = \cos(n)$, etc.
\item piecewise functions, e.g.
\[
x[n] = \left\{  \begin{array}{cl}
f_1[n] & n < 0\\
f_2[n] & n \geq 0\\
\end{array}\right.
\]
\end{itemize}

\begin{example}[Unit Step]
  The DT counterpart of the CT step function is the \emph{DT Unit Step}, $u[n]$:
  \[
  u[n] = \left\{  \begin{array}{cl}
    0 & n < 0\\
    1 & n \geq 0\\
  \end{array}\right.
  \]
  Note, there are not continuity issues at $n=0$ as DT functions have discrete domains.
\end{example}

\begin{example}[Sampled Pure audio tone at "middle C"]
  A \emph{sampled} signal modeling the air pressure of a specific tone, sampled at 8kHz, might be
  \[
  x[n] = \sin\left(2\pi (261.6) \tfrac{1}{8000} n\right)
  \]
  Such DT signals are commonly used in digital music generation, storage, and playback.
\end{example}
\begin{example}[Sampled Chord]
    Similarly, the sampled chord "G", an additive mixture of tones at G, B, and D and might be modeled as
    \[
    x(t) = \sin\left(2\pi (392) \tfrac{1}{8000} n\right) + \sin\left(2\pi (494) \tfrac{1}{8000} n\right) + \sin\left(2\pi (293) \tfrac{1}{8000} n\right)
    \]
    again sampled at 8kHz. This example shows we can use addition to build-up signals to approximate real signals of interest.
\end{example}

\section{Basic Transformations}

Similar to CT signals, we can also apply transformations to DT signals to increase their modeling flexibility.

\begin{itemize}
\item magnitude scaling
\[
x_2[n] = a x_1[n]
\]
for $a \in \mathbb{R}$.
\item time differences
\[
x_2[n] = x_1[n] - x_1[n-1]
\]
\item running sums
\[
x_2[n] = \sum\limits_{m = -\infty}^{n} x_1[m]
\]
\item sums
\[
y[n] = \sum\limits_{i} x_i[n]
\]
an important example we will see is the DT Fourier series.
\item multiplication (modulation)
\[
y[n] = x_1[n] x_2[n]
\]
\item time index shift
\[
x_2[n] = x_1[n+m]
\]
\begin{itemize}
\item if $m < 0$ it is called a {\it delay}
\item if $m > 0$ it is called an {\it advance}
\end{itemize}

\item time reversal
\[
x_2[n] = x_1[-n]
\]

\item decimation
\[
y[n] = x[m n]
\]
for $m \in \mathbb{Z}^+$.
\begin{itemize}
\item e.g. for $m=2$ only keep every other sample
\item e.g. for $m=3$ only keep every third sample
\item etc.
\end{itemize}

\item interpolation
\[
y[n] = \left\{  \begin{array}{cl}
x\left[ \frac{n}{m}\right] & n = 0\; , \; \pm m, , \; \pm 2m \cdots\\
0 & \mbox{else}
\end{array}\right.
\]
When $m = 2$ this inserts a zero sample between every sample of the signal.
\end{itemize}

\section{Characterization of Signals}

There are a few basic ways of characterizing DT signals.

\begin{definition}[Causal DT Signal]
A DT signal is \emph{causal} if $x[n] = 0$ $\forall n < 0$.
\end{definition}
\begin{definition}[Anti-Causal DT Signal]
A DT signal is \emph{anti-causal} or acausal if $x[n] = 0$ $\forall n \geq 0$.
\end{definition}

A DT signal can be written as the sum of a causal and anti-causal signal.

A DT signal is periodic if $x[n] = x[n + N] \; \forall n$ for a fixed period $N \in \mathbb{Z}$.

A DT signal is even if $x[n] = x[-n] \; \forall n$.

A DT signal is odd if $x[n] = -x[-n] \; \forall n$.

Any DT signal can be written in terms of an even and odd component
\[
x[n] = x_e[n] + x_o[n]
\]
where
\[
\begin{array}{ll}
x_e[n] &= \frac{1}{2}\left\{x[n] + x[-n]\right\} \\
& \\
x_o[n] &= \frac{1}{2}\left\{x[n] - x[-n]\right\}
\end{array}
\]

Analogous to CT signals, the energy of a DT signal is
\[
E_x = \lim_{N\rightarrow\infty} \sum\limits_{-N}^N \lvert x[n]\rvert^2 \; .
\]

And the power of a DT signal is the energy averaged over an interval as that interval tends to infinity.

\[
P_x = \lim_{N\rightarrow\infty} \frac{1}{2N+1} \sum\limits_{-N}^N \lvert x[n]\rvert^2 \; .
\]

DT Signals with finite, non-zero energy and zero power are called {\it energy signals}. DT Signals with finite, non-zero power (and by implication infinite energy) are called {\it power signals}. These categories are non-exclusive, some signals are neither energy or power signals.

\section{DT Unit Impulse Function}

In DT the unit impulse function, denoted $\delta[n]$ is defined as
\[
\delta[n] = \left\{
\begin{array}{ll}
  1 & n = 0\\
  0 & \text{else}
\end{array}
\right.
\]
Note this definition is straightforward compared to the CT impulse as there are no continuity issues and it is not defined in terms of a distribution. It is typically draw as
\begin{center}
  \includegraphics[scale=0.5]{graphics/dtdelta.pdf}
\end{center}

Some useful properties of the DT impulse function are:

\begin{itemize}
\item Energy is 1: $\sum\limits_{n=-\infty}^{\infty} \delta[n] = 1$
\item Sampling: $x[n]\delta[n-n_0] = x[n_0]\delta[n-n_0]$
\item Sifting: $\sum\limits_{n=-\infty}^{\infty} x[n]\delta[n-n_0] = x[n_0]$
\end{itemize}

The impulse can be defined in terms of the step:
\[
\delta[n] = u[n] - u[n-1]
\]
and vice-versa
\[
u[n] = \sum\limits_{m=-\infty}^{n} \delta[m]
\]
or
\[
u[n] = \sum\limits_{k=0}^{\infty} \delta[n-k]
\]

\section{DT Complex Exponential}

The DT Complex Exponential is defined in a similar fashion the the CT version, but with some important differences. The general DT complex exponential is given by the expression:
\[
x[n] = Ce^{\beta n}
\]
where in general $C \in \mathbb{C}$ and $\beta \in \mathbb{C}$. It is sometimes convenient (for reasons we will see later) to write this as
\[
x[n] = C \alpha^n
\]
where $\alpha = e^{\beta}$ is a complex number $\alpha = \cos(\beta) + j\sin(\beta)$.

We now examine several special cases.

\subsection{DT Complex Exponential: real case}

Let $C$ and $\alpha$ be real, then there are four intervals of interest:

\begin{itemize}
\item $\alpha > 1$
\item $ 0 < \alpha < 1$
\item $-1 < \alpha < 0$
\item $\alpha < -1$
\end{itemize}

Each of these are shown in Fig.~\ref{fig:dtexpreal}.

\begin{figure}[ht]
\includegraphics[scale=0.5]{graphics/dtexpcase1.pdf}
\includegraphics[scale=0.5]{graphics/dtexpcase2.pdf}
\includegraphics[scale=0.5]{graphics/dtexpcase3.pdf}
\includegraphics[scale=0.5]{graphics/dtexpcase4.pdf}
\caption{ DT Complex Exponential: real case, four intervals of interest.}
\label{fig:dtexpreal}
\end{figure}

\subsection{DT Complex Exponential: sinusoidal case}

Let $C = 1$. When $\beta$ is purely imaginary, $\beta = j\omega_0$
\[
x[n] = e^{j\omega_0 n}
\]

As in CT, by Euler's identity:
\[
e^{j\omega_0 n} = \cos(\omega_0 n) + j\sin(\omega_0 n)
\]
and
\[
\Re(x[n]) = \cos(\omega_0 n) = \frac{1}{2}\left( e^{j\omega_0 n} + e^{-j\omega_0 n} \right)
\]
\[
\Im(x[n]) = \sin(\omega_0 n) = \frac{1}{2j}\left( e^{j\omega_0 n} - e^{-j\omega_0 n} \right)
\]

The energy and power are the same as for the CT complex sinusoid: $E_x = \infty$ and $P_x = 1$.

\subsection{DT Complex Exponential: sinusoidal case with phase shift}

The general DT sinusoid is

\[
x[n] = A\cos(\omega_0 n + \phi)
\]

\begin{itemize}
\item $A$ is called the amplitude
\item $\phi$ is called the phase shift
\item $\omega_0$ is now in radians (assuming $n$ is dimensionless)
\end{itemize}

\begin{center}
\includegraphics[scale=0.6]{graphics/dtsinusoid.pdf}
\end{center}

For CT sinusoids as $\omega_0$ increases the signal oscillates faster and faster. However for DT sinusoids there is a "fastest" oscillation.

\[
e^{j\omega_0 n}\rvert_{\omega_0 = \pi} = e^{j\pi n} = (-1)^n
\]

\includegraphics[scale=0.5]{graphics/dtfastsin.pdf}

\subsection{Properties of DT complex sinusoid}

If we consider two frequencies: $\omega_0$ and $\omega_0+2\pi$. In the first case:
\[
x[n] = e^{j\omega_0 n}
\]
In the second case:
\[
\begin{array}{ll}
x[n] &= e^{j(\omega_0+2\pi) n} \\
&= \underbrace{e^{j2\pi n}}_{\text{always 1}}\; e^{j\omega_0 n} \\
&= e^{j\omega_0 n}
\end{array}
\]

Thus the two are the same signal. This has important implications later in the course.

Another difference between CT and DT complex sinusoids is periodicity. Recall for a DT signal to be periodic with period $N$
\[
x[n] = x[n+N] \; \forall n
\]
Substituting the complex sinusoid
\[
e^{j\omega_0 n} = e^{j\omega_0 (n+N)} = e^{j\omega_0 n}e^{j\omega_0 N}
\]
requires $e^{j\omega_0 N} = 1$, which implies $\omega_0 N$ is a multiple of $2\pi$:
\[
\omega_0 N = 2\pi m \;\;\; m = \pm 1, \pm 2, \cdots
\]
or equivalently
\[
\frac{|\omega_0|}{2\pi} = \frac{m}{N}
\]
thus $\omega_0$ must be a rational multiple of $\pi$.

Two DT complex sinusoids are harmonics of one another is both are periodic in $N$,  i.e when

\[
x_k(t) = e^{jk\frac{2\pi}{N} n} \; \text{for} \; k = 0, \pm 1, \pm 2, \cdots
\]

This implies there are only $N$ distinct harmonics in DT.

\subsection{DT Complex Exponential: general case}

In the general case we get a sinusoid signal modulated by an exponential. Let $C = Ae^{j\phi}$ and $\beta = r + j\omega_0$, then
\[
x[n] = C e^{\beta n} =  Ae^{j\phi} e^{(r+j\omega_0)n}
\]
Expanding the terms and using Euler's identity gives:

\[
x[n] = \underbrace{Ae^{rn}\cos(\omega_0 n+\phi)}_{\Re \text{part}} + j \underbrace{Ae^{rn}\sin(\omega_0 n+\phi)}_{\Im \text{part}}
\]
Each part is a real sinusoid whose amplitude is modulated by a real exponential.

The visualization of the general case is to view the signal $x[n]$ as a vector rotating through fixed angles in the complex plane.

\begin{center}
\includegraphics[scale=0.7]{graphics/DT_complexsinusoid_visual}
\end{center}


\subsubsection{Higher Order Variation of Parameters}
\noindent
We are trying to find a solution to the nth order linear ODE
\begin{equation*}
	a_n(x)y^{(n)} + \ldots + a_0(x)y = g(x)
\end{equation*}
assuming that we already know the fundamental solutions for the corresponding homogeneous equation
\begin{equation*}
	y_h = C_1y_1 + \ldots + C_ny_n.
\end{equation*}

\noindent
For this method, we'll assume that $y$ can be written as
\begin{equation*}
	y = v_1(x)y_1 + \ldots + v_n(x)y_n
\end{equation*}
and we'll try to find $v_1, \ldots, v_n$.\\

\noindent
Since there are $n$ unknown functions, we'll need $n$ equations to find them all.
We can generate these by differentiating $y_p$.
\begin{equation*}
	y' = \left(v_1y_1' + \ldots + v_ny_n'\right) + \left(v_1'y_1 + \ldots + v_n'y_n\right)
\end{equation*}
to avoid second derivatives of $v_1, \ldots, v_n$ from entering the formula for $y''$, we also have the condition
\begin{equation*}
	v_1'y_1 + \ldots +v_n'y_n = 0.
\end{equation*}
We can now continue differentiating to get $n-2$ more equations involving $v_1', \ldots, v_n'$.
We also impose a final $n^{\text{th}}$ condition that
\begin{equation*}
	v_1'y^{(n-1)} + \ldots + v_n'y_n^{(n-1)} = g.
\end{equation*}
This gets us a system of $n$ equations,
\begin{equation*}
	\begin{cases}
		v_1'y_1 + \ldots + v_n' y_n & = 0 \\
		\vdots & \vdots \\
		v_1'y_1^{(n-2)} + \ldots + v_n'y_n^{(n-2)} & = 0 \\
		v_1'y_1^{(n-1)} + \ldots + v_n'y_n^{(n-1)} & = g
	\end{cases} .
\end{equation*}
Hopefully this system looks familiar from second order equations.\\

\noindent
We can rewrite this system in terms of matrices and vectors.
\begin{equation*}
	\begin{bmatrix}
		y_1 & \ldots & y_n \\
		\vdots & & \vdots\\
		y_1^{(n-2)} & \ldots & y_n^{(n-2)} \\
		y_1^{(n-1)} & \ldots & y_n^{(n-1)}
	\end{bmatrix} \begin{bmatrix}
		v_1' \\
		\vdots \\
		v_{n-1}' \\
		v_n'
	\end{bmatrix} = \begin{bmatrix}
		0 \\
		\vdots \\
		0 \\
		g
	\end{bmatrix}.
\end{equation*}
It's sufficient to show that a solution to this system exists if the determinant of the square matrix on the left is non-zero. The determinant of this matrix actually has a special name.

\begin{definition}
	The Wronskian of $n$ $n-1$ times differentiable functions $\left\{f_1, \ldots, f_n\right\}$ on an interval $I$ is
	\begin{equation*}
		W[f_1, \ldots, f_n](x) = \begin{vmatrix}
			f_1(x) & \ldots & f_n(x) \\
			f_1'(x) & \ldots & f_n'(x) \\
			\vdots &        & \vdots \\
			f_1^{(n-1)}(x) & \ldots & f_n^{(n-1)}(x)
		\end{vmatrix} \text{, } x \in I.
	\end{equation*}
\end{definition}

\noindent
Using the Wronskian, we can solve the system using Cramer's Rule.
\begin{equation*}
	v'_i(x) = \frac{g(x)W_i(x)}{W[y_1, \ldots, y_n](x)} \text{, } i = 1, \ldots, n,
\end{equation*}
where $W_i(x)$ is the determinant of the matrix obtained from the Wronskian $W(x)$ by replacing the $i^{\text{th}}$ column with $\text{col}[0, \ldots, 1]$. Using the cofactor expansion along this column, we can write $W_i(x)$ as
\begin{equation*}
	W_i(x) = (-1)^{n-i}W[y_1, \ldots, y_{i-1}, y_{i+1}, \ldots, y_n](x) \text{, } i = 1, \ldots, n.
\end{equation*}
Now with a solution for $v_i'$, we can integrate to get $v_i$.
\begin{equation*}
	v_i = \int{\frac{g(x)W_i(x)}{W[y_1, \ldots, y_n](x)} \mathrm{d}x}.
\end{equation*}
Now with a solution for $v_i$, we can substitute back to find $y(x)$.
\begin{equation*}
	y(x) = \sum_{i=1}^{n}{y_i(x)\int{\frac{g(x)W_i(x)}{W[y_1, \ldots, y_n](x)} \mathrm{d}x}}
\end{equation*}

\begin{example}
	Find the general solution to the equation
	\begin{equation*}
		x^3y''' + x^2y'' - 2xy' = x^3\sin{x} \text{, } x > 0
	\end{equation*}
	given that $\left\{x, x^{-1}, x^2\right\}$ is the set of fundamental solutions.
\end{example}
\noindent
First we divide by $x^3$ to get a leading coefficient of 1.
\begin{equation*}
	y''' + x^{-1}y''' - 2x^{-2}y' = \sin{x} \text{, } x > 0.
\end{equation*}
Next we calculate the $W(x)$ and each $W_i(x)$ for the fundamental solution set.
\begin{align*}
	W[x,x^{-1}, x^2](x) &= \begin{vmatrix}
		x & x^{-1} & x^2 \\
		1 & -x^{-2} & 2x \\
		0 & 2x^{-3} & 2
	\end{vmatrix} = 6x^{-1} \\
	W_1(x) &= (-1)^{3-1}W[x^{-1}, x^2](x) = (-1)^{2}\begin{vmatrix}
		x^{-1} & x^2 \\
		-x^{-2} & 2x
	\end{vmatrix} = 3 \\
	W_2(x) &= (-1)^{3-2}\begin{vmatrix}
		x & x^{2} \\
		1 & 2x
	\end{vmatrix} = -x^2 \\
	W_2(x) &= (-1)^{3-2}\begin{vmatrix}
		x & x^{-1} \\
		1 & -x^{-2}
	\end{vmatrix} = -2x^{-1}.
\end{align*}
Now we can calculate $y$.
\begin{align*}
	y(x) &= x\int{\frac{(\sin{x})^3}{-6x^{-1}} \mathrm{d}x} + x^{-1}\int{\frac{(\sin{x})(-x^2)}{-6x^{-1}} \mathrm{d}x} + x^2\int{\frac{(\sin{x})(-2x^{-1})}{-6x^{-1}} \mathrm{d}x} \\
	&= x\int{\left(\frac{-1}{2}x\sin{x}\right)\mathrm{d}x} + x^{-1}\int{\left(\frac{1}{6}x^3\sin{x}\right)\mathrm{d}x} + x^2\int{\left(\frac{1}{3}\sin{x}\right)\mathrm{d}x} \\
	&= C_1x + c_2x^{-1} + C_3x^2 + \cos{x} - x^{-1}\sin{x}
\end{align*}


\subsection{Limit \& Continuity Definitions}
\noindent
Finally, we have the proper tools to define a limit in higher dimensions.
We say that $\lim_{p\to p_0}{f(p)} = L$ if for all $N(L, \epsilon)$, there exists $N(p_0,\delta)$ such that $p \in N(p_0, \delta) \implies f(p) \in N(L, \epsilon)$.
That is, the limit of $f(p)$ as $p$ approaches $p_0$ is equal to $L$ if for all open delta neighborhoods around $L$, there exists an open delta neighborhood around $p_0$ such that $p$ being in the neighborhood around $p_0$ means $f(p)$ must be in the neighborhood around $L$.

\begin{figure}[H]
	\centering
	\includegraphics[width=0.5\textwidth]{./Images/differentialMultivariableCalculus/limit.png}
	\caption{Limit definition}
\end{figure}

\noindent
Now with a limit definition, we can define continuity at a point.
We say that a function $f : \mathbb{R}^n \to \mathbb{R}$ is continuous at $p_0$ if $\lim_{p \to p_0}{f(p)} = f(p_0)$.\\

\noindent
\begin{theorem}
	Trigonometric functions, exponentials, logarithms, and sums, products, quotients, and compositions of such functions are continuous on their domain.
\end{theorem}

\noindent
Although our definitions allow us to confirm that a value is the limit of a function, they do not give us any insight into how to find the value of the limit.
We'd need to approach our point of interest from every possible direction to see if the limit from that direction is the same as all the others.
If any two directions give different limit values, then the limit doesn't exist.
We approach a function, $f$, by composing it with a single variable path, $\vec{r}(t)$, that goes through the point of interest, and find the limit along the path.\\

\noindent
If $f$ is some surface $f(x, y)$ and $\vec{r}(t) = \langle x(t), y(t)\rangle$, then the composition of $f$ and $\vec{r}$ is $f\circ\vec{r} = f(\vec{r}(t)) = f(x(t), y(t))$.\\
For example, let's try to find
\begin{equation*}
	\lim_{(x,y) \to (0,0)}{\frac{x^2-y^3}{x^2+y^2}}.
\end{equation*}
\indent
We'll choose two paths $\vec{r_1}(t) = \langle t, 0 \rangle$ and $\vec{r_2}(t) = \langle t, t \rangle$ and find the limit as $t \to 0$ in both cases.\\
\indent
\begin{equation*}
	\lim_{t \to 0}{f(\vec{r_1}(t))} = \lim_{t \to 0}{\frac{t^2}{t^2}} = 1
\end{equation*}
\indent
\begin{equation*}
	\lim_{t \to 0}{f(\vec{r_2}(t))} = \lim{t \to 0}{\frac{t^2-t^3}{2t^2}} = \lim_{t \to 0}{\frac{1}{2} - \frac{t}{2}} = \frac{1}{2}
\end{equation*}
\indent
Since the limits on the two paths are not equal, we can say that
\begin{equation*}
	\lim_{(x,y) \to (0,0)}{\frac{x^2-y^3}{x^2+y^2}} = \text{ DNE}.
\end{equation*}


\section{Rayleigh quotient method}
\label{sec:ray}
Rayleigh's method can be extended to find the approximate value of the fundamental
natural frequency of a discrete system.
It is possible derive an expression for the approximate value of the first
natural frequency of a multi degree-of-freedom system according to Rayleigh's
method.
The kinetic and potential energies of an \(n\)-degree-of-freedom discrete system
can be expressed as:
\begin{align}\label{eq:energyequation}
  T &= \frac{1}{2}\dot{\vec{v}}^{\top}[\mathbf{M}]\dot{\vec{v}}\\[1em]
  V &= \frac{1}{2}\vec{v}^{\top}[\mathbf{K}]\vec{v}
\end{align}
To find the natural frequencies, we assume harmonic motion to be
\(\vec{x} = \vec{\mathbf{X}} \cos(\omega t)\), where \(\vec{\mathbf{X}}\)
denotes the vector of amplitudes and \(\omega\) represents the natural frequency
of vibration.
If the system is conservative, the maximum kinetic energy is equal to the
maximum potential energy. By equating \(T\) and \(V\) obtain:
\begin{equation}\label{eq:quotient}
  \omega^2 = \frac{\vec{v}^{\top}[\mathbf{K}]\vec{v}}
  {\vec{v}^{\top}[\mathbf{M}]\vec{v}}
\end{equation}
%
The right-hand side of \eqref{eq:quotient} is known as \emph{Rayleigh's quotient}
and is denoted as \(R(\vec{U})\).
% then the Rayleigh's quotient can be expressed
%as:
%\begin{align}
%  \omega^{2} =& R(\vec{X}) =
%  \frac{
%          c_{1}^{2}\,\omega_{2}^{2}\,\dot{\vec{x}}^{\top}\,[\mathbf{M}]\,\dot{\vec{x}} +
%          c_{2}^{2}\,\omega_{2}^{2}\,\dot{\vec{x}}^{\top}\,[\mathbf{M}]\,\dot{\vec{x}} +
%          \dots}
%       {
%          c_{1}^{2}\,\dot{\vec{x}}^{\top}\,[\mathbf{M}]\,\dot{\vec{x}} +
%          c_{2}^{2}\,\dot{\vec{x}}^{\top}\,[\mathbf{M}]\,\dot{\vec{x}} + \dots
%        }\\[1.5em]
%      &  R(\vec{X}) =
%      \frac{c_{1}^{2}\,\omega_{2}^{2} + c_{2}^{2}\,\omega_{2}^{2} + \dots}
%           {c_{1}^{2} + c_{2}^{2} + \dots}
%\end{align}
The results obtained with regard to the natural frequencies are as follows:
\(\omega_{1} = 8.27843\) [\si{\radian\per\second}],
\(\omega_{2} = 27.40935\) [\si{\radian\per\second}],
\(\omega_{3} = 41.87035\) [\si{\radian\per\second}].
Thus values obtained for the mode shapes are:
\begin{equation}
  \label{eq:raymodefree}
  [\mathbf{U}] = \begin{bmatrix*}[r]
	1.00000 & 1.00000 & 1.00000 \\
	0.86540 &-0.47555 &-2.44325 \\
	0.61945 &-1.28034 & 2.15533 \\
  \end{bmatrix*}
\end{equation}
The results obtained for the proportionally damped system are the natural
frequencies:
\(\omega_{1} =  8.27799\) [\si{\radian\per\second}],
\(\omega_{2} = 27.40863\) [\si{\radian\per\second}],
\(\omega_{3} = 41.91576\) [\si{\radian\per\second}].
So the values obtained for the mode shapes are:
\begin{equation}\label{eq:raymodeprop}
  [\mathbf{U}] =
	\begin{bmatrix*}[r]
  		1.00000 & 1.00000 & 1.00000 \\
		0.86499 &-0.48006 &-2.46147 \\
		0.61907 &-1.28528 & 2.16954 \\
	\end{bmatrix*}
\end{equation}
Rayleigh's quotient has a stationary value in the neighbourhood of an eigenvector.
The stationary value is actually a minimum value in the neighbourhood of the
fundamental mode.
Which means that Rayleigh s quotient is never higher than the highest eigenvalue.
Thus Rayleigh's quotient provides an upper bound for \(\omega_{1}^{2}\) lower bound
for \(\omega_{n}^{2}\).


\section*{Exercises}

\begin{ex} Let $V$ and $W$ be subspaces of $\R^{n}$ and $\R^{m}$
respectively and let $T:V\rightarrow W$ be a linear transformation. Suppose
that $\set{T\vect{v}_{1},\ldots,T\vect{v}_{r}} $ is linearly
independent. Show that it must be the case that $\set{\vect{v}_{1},\ldots,
\vect{v}_{r}} $ is also linearly independent.
\begin{sol}
If $\sum_i^r a_i \vect{v}_r =0$, then using linearity properties of $T$ we get
\[ 0 = T(0) =  T(\sum_i^r a_i \vect{v}_r) =
\sum_i^r a_i T(\vect{v}_r).\]
Since we assume that  $\set{T\vect{v}_{1},\ldots,T\vect{v}_{r}} $ is linearly
independent, we must have all $a_i=0$, and therefore we conclude that
 $\set{\vect{v}_{1},\ldots,
\vect{v}_{r}} $ is also linearly independent.
\end{sol}
\end{ex}

\begin{ex} Let
\begin{equation*}
V=\mbox{span}\set{\begin{mymatrix}{c}
1 \\
1 \\
2 \\
0
\end{mymatrix} ,\begin{mymatrix}{c}
0 \\
1 \\
1 \\
1
\end{mymatrix} ,\begin{mymatrix}{c}
1 \\
1 \\
0 \\
1
\end{mymatrix} }
\end{equation*}
Let $T\vect{x}=A\vect{x}$ where $A$ is the matrix
\begin{equation*}
\begin{mymatrix}{cccc}
1 & 1 & 1 & 1 \\
0 & 1 & 1 & 0 \\
0 & 1 & 2 & 1 \\
1 & 1 & 1 & 2
\end{mymatrix}
\end{equation*}
Give a basis for $\func{im}(T)$.
%\begin{sol}
%\end{sol}
\end{ex}

\begin{ex} Let
\begin{equation*}
V=\mbox{span}\set{\begin{mymatrix}{c}
1 \\
0 \\
0 \\
1
\end{mymatrix} ,\begin{mymatrix}{c}
1 \\
1 \\
1 \\
1
\end{mymatrix} ,\begin{mymatrix}{c}
1 \\
4 \\
4 \\
1
\end{mymatrix} }
\end{equation*}
Let $T\vect{x}=A\vect{x}$ where $A$ is the matrix
\begin{equation*}
\begin{mymatrix}{cccc}
1 & 1 & 1 & 1 \\
0 & 1 & 1 & 0 \\
0 & 1 & 2 & 1 \\
1 & 1 & 1 & 2
\end{mymatrix}
\end{equation*}
Find a basis for $\func{im}(T)$. In this case, the original
vectors do not form an independent set.

\begin{sol}
Since the third vector is a linear combinations of the first two, then
the image of the third vector will also be a linear combinations of
the image of the first two.  However the image of the first two
vectors are linearly independent (check!), and hence form a basis of
the image.

Thus a basis for $\func{im}(T) $ is:

\begin{equation*}
V=\mbox{span}\set{\begin{mymatrix}{c}
2 \\
0 \\
1 \\
3
\end{mymatrix} ,\begin{mymatrix}{c}
4 \\
2 \\
4 \\
5
\end{mymatrix}  }
\end{equation*}

\end{sol}
\end{ex}

\begin{ex} If $\set{\vect{v}_{1},\ldots,\vect{v}_{r}} $ is linearly
independent and $T$ is a one to one linear transformation, show that $
\set{T\vect{v}_{1},\ldots,T\vect{v}_{r}} $ is also linearly
independent. Give an example which shows that if $T$ is only linear, it can
happen that, although $\set{\vect{v}_{1},\ldots,\vect{v}_{r}} $ is
linearly independent, $\set{T\vect{v}_{1},\ldots,T\vect{v}_{r}} $
is not. In fact, show that it can happen that each of the $T\vect{v}_{j}$
equals 0.
%\begin{sol}
%\end{sol}
\end{ex}

\begin{ex} Let $V$ and $W$ be subspaces of $\R^{n}$ and $\R^{m}$
respectively and let $T:V\rightarrow W$ be a linear transformation. Show
that if $T$ is onto $W$ and if $\set{\vect{v}_{1},\ldots,\vect{v}
_{r}} $ is a basis for $V$, then $\sspan\set{T\vect{v}
_{1},\ldots,T\vect{v}_{r}} =W$.
%\begin{sol}
%\end{sol}
\end{ex}

\begin{ex} Define $T:\R^{4}\rightarrow \R^{3}$ as follows.
\begin{equation*}
T\vect{x}=\begin{mymatrix}{rrrr}
3 & 2 & 1 & 8 \\
2 & 2 & -2 & 6 \\
1 & 1 & -1 & 3
\end{mymatrix} \vect{x}
\end{equation*}
Find a basis for $\func{im}(T)$. Also find a basis for $\ker
(T)$.
%\begin{sol}
%\end{sol}
\end{ex}

\begin{ex} Define $T:\R^{3}\rightarrow \R^{3}$ as follows.
\begin{equation*}
T\vect{x}=\begin{mymatrix}{ccc}
1 & 2 & 0 \\
1 & 1 & 1 \\
0 & 1 & 1
\end{mymatrix} \vect{x}
\end{equation*}
where on the right, it is just matrix multiplication of the vector $\vect{x}$
which is meant. Explain why $T$ is an isomorphism of $\R^{3}$ to $
\R^{3}$.
%\begin{sol}
%\end{sol}
\end{ex}

\begin{ex} Suppose $T:\R^{3}\rightarrow \R^{3}$ is a linear
transformation given by
\begin{equation*}
T\vect{x}=A\vect{x}
\end{equation*}
where $A$ is a $3\times 3$-matrix. Show that $T$ is an isomorphism if and
only if $A$ is invertible.
%\begin{sol}
%\end{sol}
\end{ex}

\begin{ex} Suppose $T:\R^{n}\rightarrow \R^{m}$ is a linear
transformation given by
\begin{equation*}
T\vect{x}=A\vect{x}
\end{equation*}
where $A$ is an $m\times n$-matrix. Show that $T$ is never an isomorphism if
$m\neq n$. In particular, show that if $m>n$, $T$ cannot be onto and if $
m<n, $ then $T$ cannot be one to one.
%\begin{sol}
%\end{sol}
\end{ex}

\begin{ex} Define $T:\R^{2}\rightarrow \R^{3}$ as follows.
\begin{equation*}
T\vect{x}=\begin{mymatrix}{cc}
1 & 0 \\
1 & 1 \\
0 & 1
\end{mymatrix} \vect{x}
\end{equation*}
where on the right, it is just matrix multiplication of the vector $\vect{x}$
which is meant. Show that $T$ is one to one. Next let $W=\func{im}(T)$. Show that $T$ is an isomorphism of $\R^{2}$ and $\func{im}(T)$.
%\begin{sol}
%\end{sol}
\end{ex}

\begin{ex} In the above problem, find a $2\times 3$-matrix $A$ such that the
restriction of $A$ to $\func{im}(T) $ gives the same result as $
T^{-1}$ on $\func{im}(T)$. \textbf{Hint:\ }You might let $A$ be
such that
\begin{equation*}
A\begin{mymatrix}{c}
1 \\
1 \\
0
\end{mymatrix} =\begin{mymatrix}{c}
1 \\
0
\end{mymatrix} ,\ A\begin{mymatrix}{c}
0 \\
1 \\
1
\end{mymatrix} =\begin{mymatrix}{c}
0 \\
1
\end{mymatrix}
\end{equation*}
now find another vector $\vect{v}\in \R^{3}$ such that
\begin{equation*}
\set{\begin{mymatrix}{c}
1 \\
1 \\
0
\end{mymatrix} ,\begin{mymatrix}{c}
0 \\
1 \\
1
\end{mymatrix} ,\vect{v}}
\end{equation*}
is a basis. You could pick
\begin{equation*}
\vect{v}=\begin{mymatrix}{c}
0 \\
0 \\
1
\end{mymatrix}
\end{equation*}
for example. Explain why this one works or one of your choice works. Then
you could define $A\vect{v}$ to equal some vector in $\R^{2}$.
Explain why there will be more than one such matrix $A$ which will deliver
the inverse isomorphism $T^{-1}$ on $\func{im}(T)$.
%\begin{sol}
%\end{sol}
\end{ex}

\begin{ex} Now let $V$ equal $\sspan\set{\begin{mymatrix}{c}
1 \\
0 \\
1
\end{mymatrix} ,\begin{mymatrix}{c}
0 \\
1 \\
1
\end{mymatrix} } $ and let $T:V\rightarrow W$ be a linear transformation
where
\begin{equation*}
W=\sspan\set{\begin{mymatrix}{c}
1 \\
0 \\
1 \\
0
\end{mymatrix} ,\begin{mymatrix}{c}
0 \\
1 \\
1 \\
1
\end{mymatrix} }
\end{equation*}
$\ $\ and
\begin{equation*}
T\begin{mymatrix}{c}
1 \\
0 \\
1
\end{mymatrix} =\begin{mymatrix}{c}
1 \\
0 \\
1 \\
0
\end{mymatrix} ,T\begin{mymatrix}{c}
0 \\
1 \\
1
\end{mymatrix} =\begin{mymatrix}{c}
0 \\
1 \\
1 \\
1
\end{mymatrix}
\end{equation*}
Explain why $T$ is an isomorphism. Determine a matrix $A$ which, when
multiplied on the left gives the same result as $T$ on $V$ and a matrix $B$
which delivers $T^{-1}$ on $W$. \textbf{Hint:\ }You need to have
\begin{equation*}
A\begin{mymatrix}{cc}
1 & 0 \\
0 & 1 \\
1 & 1
\end{mymatrix} =\begin{mymatrix}{cc}
1 & 0 \\
0 & 1 \\
1 & 1 \\
0 & 1
\end{mymatrix}
\end{equation*}
Now enlarge $\begin{mymatrix}{c}
1 \\
0 \\
1
\end{mymatrix} ,\begin{mymatrix}{c}
0 \\
1 \\
1
\end{mymatrix} $ to obtain a basis for $\R^{3}$. You could add in $\begin{mymatrix}{c}
0 \\
0 \\
1
\end{mymatrix} $ for example, and then pick another vector in $\R^{4}$ and
let $A\begin{mymatrix}{c}
0 \\
0 \\
1
\end{mymatrix} $ equal this other vector. Then you would have
\begin{equation*}
A\begin{mymatrix}{ccc}
1 & 0 & 0 \\
0 & 1 & 0 \\
1 & 1 & 1
\end{mymatrix} =\begin{mymatrix}{ccc}
1 & 0 & 0 \\
0 & 1 & 0 \\
1 & 1 & 0 \\
0 & 1 & 1
\end{mymatrix}
\end{equation*}
This would involve picking for the new vector in $\R^{4}$ the vector
$\begin{mymatrix}{cccc}
0 & 0 & 0 & 1
\end{mymatrix} ^{T}$. Then you could find $A$. You can do something similar to find
a matrix for $T^{-1}$ denoted as $B$.
%\begin{sol}
%\end{sol}
\end{ex}


%
% Chapter 1.6
%

\section*{1.6 Limit Laws}

Suppose that \(c\) is the constant and the limits
$$\lim_{x-a}f(x) \quad \text{ and } \quad \lim_{x \to a}g(x)$$
exist. Then
$$\textbf{1. }\lim_{x \to a}[f(x)+g(x)]=\lim_{x \to a}f(x) + \lim_{x \to a}g(x)$$
$$\textbf{2. }\lim_{x \to a}[f(x)-g(x)]=\lim_{x \to a}f(x) - \lim_{x \to a}g(x)$$
$$\textbf{3. }\lim_{x \to a}[cf(x)]=c\lim_{x \to a}f(x)$$
$$\textbf{4. }\lim_{x \to a}[f(x)g(x)]=\lim_{x \to a}f(x) \times \lim_{x \to a}g(x)$$
$$\textbf{5. }\lim_{x \to a}[\frac{f(x)}{g(x)}]=\frac{\lim_{x \to a}f(x)}{\lim_{x \to a}g(x)} \quad \text{if } \lim_{x \to a}g(x) \neq 0 $$
$$\textbf{6. }\lim_{x \to a}{[f(x)]}^n={[\lim_{x \to a}f(x)]}^n \quad \text{where } n \text{ is a postitive integer.}$$
$$\textbf{7. }\lim_{x \to a}c=c$$
$$\textbf{8. }\lim_{x \to a}x=a$$
$$\textbf{9. }\lim_{x \to a}x^n=a^n \quad \text{where } n \text{ is a positive integer}$$
$$\textbf{10. }\lim_{x \to a}\sqrt[n]x=\sqrt[n]a \quad \text{where } n \text{ is a positive integer}$$
$$\textbf{11. }\lim_{x \to a}\sqrt[n]{f(x)}=\sqrt[n]{\lim_{x \to a}f(x)} \quad \text{where } n \text{ is a positive integer}$$

\begin{enumerate}
    \item The limit of the sums is the sum of the limits.
    \item The limit of the differences is the difference of the limits.
    \item The limit of a constant times a function is the constant times the limit of the function.
    \item The limit of the products is the product of the limits.
    \item The limit of the quotients is the quotient of the limits (provided that the limit of the denominator is not 0).
\end{enumerate}

\subsection*{Direct Substitution Property}

If \(f\) is a polynomial or a rational function and \(a\) is in the domain of \(f\), then
$$\lim_{x \to a}f(x)=f(a)$$

\subsection*{Sequeeze Theorem}

If \(f(x) \leq g(x) \leq h(x)\) when \(x\) is near \(a\) (except at \(a\)) and
$$\lim_{x \to a}f(x)=\lim_{x \to a}h(x) = L$$
then
$$\lim_{x \to a}g(x) = L$$


\section{Cyclic Groups}
Recall that $C_n$ is the set of $n^{th}$ roots of unity.
If we write $\xi=e^{2\pi i/n}$, the group is actually generated by $\xi$, that is, every element is of the form $\xi^k$ for some $k$.
Note that $\xi^n=\xi^0=1$.
\begin{definition}
    A group $G$ is called cyclic if there is an $a\in G$ such that every element is of the form $a^k$ for some $k$.\\
    The element $a$ is called the generator of $G$.
\end{definition}
\begin{example}
    1. The integers under addition is cyclic with generator $1$.\\
    2. The group $\mathbb Z_n$ under addition modulo $n$ is cyclic with generator $1$.
    But in fact, if we take the function $\phi(k)\to\xi^k$, this is an isomorphism and hence $C_n\cong\mathbb Z_n$.
\end{example}
\begin{theorem}[Classification of Cyclic Groups]
    A cyclic group is isomorphic to either $C_n$ for some $n\to\mathbb N$ or $\mathbb Z$.
\end{theorem}
\begin{proof}
    Let $G$ be a cyclic group and $a$ be its generator.
    Consider $S=\{k\in\mathbb N\setminus\{0\}:a^k=e\}$.
    If $S\neq\varnothing$, then let $n$ be the smallest element of $S$.
    Consider the function $\phi:C_n\to G$ by $\phi(\xi^k)=a^k$.
    We want to show that its an isomorphism.\\
    Now if $k,l<n$ are such that $k+l<n$, then $\phi(\xi^k\xi^l)=a^{k+l}=a^ka^l=\phi(\xi^k)\phi(\xi^l)$.
    On the other hand, if $k+l=n+r,0\le r<n$, then $\phi(\xi^k\xi^l)=a^{k+l}=a^{n+r}=a^r=\phi(\xi^r)=\phi(\xi^{n+r})=\phi(\xi^k)\phi(\xi^l)$.
    As $G$ is generated by $a$ and $a^n=e$, every element of $G$ is of the form $a^k$ for some $0\le k<n$, so $\phi(\xi^k)=a^k$, So $\phi$ is injective, consider the kernel of $\phi$.
    Note that if $\phi(\xi^k)=e$ then $a^k=e\implies k=0$, so $\ker\phi=\{1\}$, hence it is injective.
    So $G\cong C_n$.\\
    Now if $S=\varnothing$, then we shall show that $G\cong\mathbb Z$.
    Consider the map $\phi(k)=a^k$, then $\phi(k+l)=a^ka^l=\phi(k)\phi(l)$.
    This is surjective by the same argument as above.
    Its kernel consists of integers $k$ with $a^k=e$ but since $S$ is empty, $k=0$, so it is injective.
    Therefore $G\cong\mathbb Z$.
\end{proof}
Because of this theorem, it is convenient to write $\mathbb Z=C_\infty$.
\begin{definition}
    Let $G$ be a group and $g\in\mathbb G$, then the order of $g$ is the smallest positive integer $n$ such that $g^n=e$ if it exists.
    If there isn't such an $n$, then we say that $g$ has infinite order.\\
    We write $\operatorname{ord}(g)$ to denote the order of $g$.
\end{definition}
Consider the set generated by the powers of $g$.
It follows easily that this set is a subgroup of $G$, we denote this by $\langle g\rangle$, the subgroup generated by $g$.
It is cyclic, so it is isomorphic to $C_n$ where $n=\operatorname{ord}g$.


\section{Information Criteria}
In this section we review the concepts behind
Akaike's Information Criterion (AIC).

Akaike's original work is for
IID data, however it is extended to a regression type setting in a
straight forward way. Suppose that the conditional distribution of $Y$
given $\bx$ is know except for a $P$-dimensional parameter $\bb$. In
this case,  the probability density function of $\bY =
(Y_1,\dots,Y_n)$ can be written as
\begin{equation}
\label{parsimodel}
f_{\bY}(\by;\bX,\bb) \equiv \prod_{i=1}^n f(y_i;\bx_i,\bb)
\end{equation}
with $\bX$ the design matrix with rows $\bx_i$.

Assume that
there exists a true parameter vector $\bb^*$ defining a true
probability density denoted by
$f_{\bY}(\by;\bX,\bb^*)$. Given these assumptions, we wish to select
$\bb$, from one of the models
defined as in (\ref{modelpdef}), ``nearest'' to the true
parameter $\bb^*$ based on the observed data $\by$.
The principle behind Akaike's criterion is to
define ``nearest'' as the model that minimizes the Kullback-Leibler
Information Quantity
\begin{equation}
\label{KLdef}
\Delta(\bb^*;\bX,\bb) = \int \left\{\log
  f_\bY(\by;\bX,\bb^*) - \log
  f_\bY(\by;\bX,\bb)\right\} f_{\bY}(\by;\bX,\bb^*) \, d\by.
\end{equation}

The analytical properties of the Kullback-Leibler Information Quantity
are discussed in detail by Kullback (1959) \nocite{kull:1959}. Two
important properties
for Akaike's criterion are
\begin{enumerate}
\item $\Delta(\bb^*;\bX,\bb) > 0$ if
$f_{\bY}(\by;\bX,\bb^*) \neq f_{\bY}(\by;\bX,\bb)$
\item
$\Delta(\bb^*;\bX,\bb) = 0$  if and only if $f_{\bY}(\by;\bX,\bb^*) =
f_{\bY}(\by;\bX,\bb)$
\end{enumerate}
almost everywhere on the range of $\bY$.
The properties
mentioned suggest that finding the model that
minimizes the Kullback-Leibler
Information Quantity is an appropriate way to choose the ``nearest''
model.

Since the first term on the
right hand side of
(\ref{KLdef}) is constant over all models we consider, we may instead
maximize
\begin{eqnarray}
\nonumber
H(\bb) &=& \int \log
  f_\bY(\by;\bX,\bb) f_{\bY}(\by;\bX,\bb^*) \, d\by\\
\label{Hdef}
&=& \sum_{i=1}^n \int \log f(y_i;\bX,\bb) \, f(y_i;\bx_i,\bb^*) \, dy_i.
\end{eqnarray}

Let $\bbh_p$ be the maximum likelihood estimate under
Model(p). Akaike's procedure for model selection is based on choosing
the model which produces the estimate that maximizes
$\E_{\bb^*}\left[H(\bbh_p)\right]$ amongst all
competing models. Akaike then derives a criterion by constructing an
asymptotically unbiased estimate of $\E_{\bb^*}\left[H(\bbh_p)\right]$ based
on the observed data.

Notice that $H(\bbh_p)$ is a function,
defined by (\ref{Hdef}),
of the maximum likelihood estimate $\bbh_p$, which is a
random variable obtained from the observed data.
A natural estimator of
its expected value (under the true distribution of the data) is
obtained by substituting the empirical distribution of the data into
(\ref{Hdef})
resulting in the log likelihood equation evaluated at the maximum
likelihood estimate under model(p)
\[
l(\bbh_p) = \sum_{i=1}^n \log f(y_i;\bx_i,\bbh_p).
\]
Akaike noticed that in general  $l(\bbh_p)$ will
overestimate $\E_{\bb^*}\left[H(\bbh)\right]$.  In particular Akaike found
that under some regularity conditions
\[
E_{\bb^*}\left[l(\bbh_p) - H(\bbh_p)\right] \approx p .
\]
This suggests that larger
values of $p$ will result in smaller values
of $l(\bbh_p)$, which may be
incorrectly interpreted as a ``better''
fit, regardless of the true model. We need to ``penalize'' for larger
values of $p$ in order to obtain an unbiased estimate of
the ``closeness'' of the model.
This fact leads to the Akaike Information Criteria which is a
bias-corrected estimate given by
\begin{equation}
\label{AIC}
\mbox{AIC}(p) = - 2 l(\bbh_{p}) + 2p .
\end{equation}
See, for example, Akaike (1973) and Bozdogan (1987)  for
the details.


% !TEX root = Main.tex
\section{K-means Algorithm}
Hard assignments $Z \in \{0,1\}^{NxK}$, Centroids $U = (u_1|..|u_K) \in \mathbb{R}^{DxK}$, data $X \in \mathbb{R}^{DxN}$ \\
\textbf{Target:} $\min_{\mathbf{U}, \mathbf{Z}} J(\mathbf{U}, \mathbf{Z}) = \|\mathbf{X} - \mathbf{U} \mathbf{Z^\top}\|_F^2$\\
$= \sum_{n=1}^N \sum_{k=1}^K \mathbf{z}_{n,k} \|\mathbf{x}_n - \mathbf{u}_k\|_2^2$\\
1. \textbf{Assign} data points to closest centroid: $k^*(\mathbf{x}_n) = \argmin_k \{ \|\mathbf{x}_n - \mathbf{u}_k\|_2 \}$. Set $\mathbf{z}_{k^*,n} = 1$, and for $ l \neq k^*~ \mathbf{z}_{l,n}=0$.\\
2. \textbf{Update} centroids: $\mathbf{u}_k = \frac{\sum_{n=1}^N z_{n,k} \mathbf{x}_n}{\sum_{n=1}^N z_{n,k}}$.\\
3. Repeat until Z doesn't change anymore.\\
Computational cost: $O(K\cdot N \cdot D)$


\section{Diagonalization}

\begin{outcome}
  \begin{enumerate}
  \item Compute sums, products, and powers of diagonal matrices.
  \item Determine whether a square matrix is diagonalizable, and
    diagonalize it if possible.
  \end{enumerate}
\end{outcome}

A square matrix $D$ is called a \textbf{diagonal matrix}%
\index{diagonal matrix}%
\index{matrix!diagonal matrix}
if all entries except those on the main diagonal are zero. Such
matrices look like the following:
\begin{equation*}
  \begin{mymatrix}{ccccc}
    d_{11} & 0 & \cdots & 0 \\
    0 & d_{22} & \cdots & 0 \\
    \vdots & \vdots & \ddots & \vdots \\
    0 & 0 & \cdots & d_{nn} \\
  \end{mymatrix}.
\end{equation*}
Diagonal matrices are particularly easy to work with. For example, the
sum of two diagonal matrices is diagonal. Also, the product of two
diagonal matrices is diagonal, and is computed by taking the product
of corresponding diagonal entries.

\begin{example}{Sums, products, and powers of diagonal matrices}{sums-products-diagonal}
  Compute $A+B$, $AB$, and $A^4$, where
  \begin{equation*}
    A = \begin{mymatrix}{rrr}
      2 & 0 & 0 \\
      0 & 3 & 0 \\
      0 & 0 & 4 \\
    \end{mymatrix}
    \quad\mbox{and}\quad
    B = \begin{mymatrix}{rrr}
      1 & 0 & 0 \\
      0 & -2 & 0 \\
      0 & 0 & 2 \\
    \end{mymatrix}.
  \end{equation*}
\end{example}

\begin{solution}
  We have
  \begin{equation*}
    A+B =
    \begin{mymatrix}{ccc}
      2+1 & 0 & 0 \\
      0 & 3-2 & 0 \\
      0 & 0 & 4+2 \\
    \end{mymatrix}
    =
    \begin{mymatrix}{rrr}
      3 & 0 & 0 \\
      0 & 1 & 0 \\
      0 & 0 & 6 \\
    \end{mymatrix},
  \end{equation*}
  \begin{equation*}
    AB =
    \begin{mymatrix}{ccc}
      2\cdot 1 & 0 & 0 \\
      0 & 3\cdot(-2) & 0 \\
      0 & 0 & 4\cdot 2 \\
    \end{mymatrix}
    =
    \begin{mymatrix}{rrr}
      2 & 0 & 0 \\
      0 & -6 & 0 \\
      0 & 0 & 8 \\
    \end{mymatrix},
  \end{equation*}
  and
  \begin{equation*}
    A^4 =
    \begin{mymatrix}{ccc}
      2^4 & 0 & 0 \\
      0 & 3^4 & 0 \\
      0 & 0 & 4^4 \\
    \end{mymatrix}
    =
    \begin{mymatrix}{ccc}
      16 & 0 & 0 \\
      0 & 81 & 0 \\
      0 & 0 & 256 \\
    \end{mymatrix}.
  \end{equation*}
  Notice that all operations are computed componentwise on the
  diagonal. Therefore, multiplication of diagonal matrices is much
  simpler than multiplication of general matrices.
\end{solution}

One of the most important problem solving techniques in linear algebra
is \textbf{diagonalization}%
\index{diagonalization}%
\index{matrix!diagonalization}. In a nutshell, the point of
diagonalization is to simplify a problem by replacing an arbitrary
matrix by a diagonal matrix. We say that two square matrices $A$ and
$B$ are \textbf{similar}%
\index{similar matrices}%
\index{matrix!similar} if there exists an invertible matrix $P$ such
that $P^{-1}AP = B$. A matrix is \textbf{diagonalizable}%
\index{matrix!diagonalizable}%
\index{diagonalizable matrix} if it is similar to a diagonal matrix.
This is summarized in the following definition.

\begin{definition}{Diagonalizable matrix}{diagonalizable}
  Let $A$ be an $n\times n$-matrix. Then $A$ is said to be
  \textbf{diagonalizable}%
  \index{diagonalizable matrix} if there exists an invertible matrix
  $P$ and a diagonal matrix $D$ such that
  \begin{equation*}
    P^{-1}AP=D.
  \end{equation*}
\end{definition}

The key connection between diagonalizability, eigenvectors, and
eigenvalues is the following theorem.

\begin{theorem}{Diagonalization and eigenvectors}{eigenvectors-and-diagonalizable}
  An $n\times n$-matrix $A$ is diagonalizable if and only if $A$ has
  $n$ linearly independent eigenvectors.
  \bigskip

  Moreover, in this case, let $P$ be the invertible matrix whose
  columns are $n$ linearly independent eigenvectors of $A$, and let
  $D$ be the diagonal matrix whose diagonal entries are the
  corresponding eigenvalues. Then $P^{-1}AP=D$.
\end{theorem}

\begin{proof}
  Assume that $A$ has $n$ linearly independent eigenvectors
  $\vect{v}_1,\ldots,\vect{v}_n$. Let
  $\eigenvar_1,\ldots,\eigenvar_n$ be the corresponding eigenvalues,
  so that
  \begin{equation}\label{eqn:eigenvectors-and-diagonalizable}
    A\vect{v}_i = \eigenvar_i\vect{v}_i
  \end{equation}
  for all $i=1,\ldots,n$. Let $P$ be the matrix that has
  $\vect{v}_1,\ldots,\vect{v}_n$ as its columns. Then $P$ is
  invertible because $\vect{v}_1,\ldots,\vect{v}_n$ are linearly
  independent. Let $D$ be the diagonal matrix that has
  $\eigenvar_1,\ldots,\eigenvar_n$ as its diagonal entries.  By the
  column method of matrix multiplication, the $i\th$ column of $AP$ is
  $A\vect{v}_i$. Also by the column method of matrix multiplication,
  the $i\th$ column of $PD$ is $\eigenvar_i\vect{v}_i$.  Therefore, by
  {\eqref{eqn:eigenvectors-and-diagonalizable}}, the matrices $AP$ and
  $PD$ have the same columns, i.e.,
  \begin{equation*}
    AP = PD.
  \end{equation*}
  It follows that $P^{-1}AP = D$, as desired.

  Conversely, assume that $A$ is diagonalizable. Then there exists an
  invertible matrix $P$ and a diagonal matrix $D$ such that
  $P^{-1}AP=D$, or equivalently, $AP=PD$. Let
  $\vect{v}_1,\ldots,\vect{v}_n$ be the columns of $P$ and let
  $\eigenvar_1,\ldots,\eigenvar_n$ be the diagonal entries of $D$.
  Again we find that the $i\th$ column of $AP$ is $A\vect{v}_i$ and
  the $i\th$ column of $PD$ is $\eigenvar_i\vect{v}_i$, and therefore
  $A\vect{v}_i = \eigenvar_i\vect{v}_i$ holds for all $i$. It follows
  that $\vect{v}_1,\ldots,\vect{v}_n$ are eigenvectors of $A$. Since
  $P$ is invertible, $\vect{v}_1,\ldots,\vect{v}_n$ are linearly
  independent, so $A$ has $n$ linearly independent eigenvectors.
\end{proof}

\begin{example}{Diagonalizing a matrix}{diagonalize-matrix}
  Diagonalize the matrix
  \begin{equation*}
    A=\begin{mymatrix}{rrr}
      3  & 0 &  2 \\
      6  & 4 &  3 \\
      -4 & 0 & -3 \\
    \end{mymatrix}.
  \end{equation*}
  In other words, find an invertible matrix $P$ and a diagonal matrix
  $D$ such that $P^{-1}AP=D$.
\end{example}

\begin{solution}
  By Theorem~\ref{thm:eigenvectors-and-diagonalizable}, we use the
  eigenvectors of $A$ as the columns of $P$ and the corresponding
  eigenvalues as the diagonal entries of $D$. We already found the
  eigenvectors and -values of $A$ in
  Example~\ref{exa:finding-eigenvalues-eigenvectors}.  They were
  \begin{equation*}
    \vect{v}_1 = \begin{mymatrix}{r} -1 \\ 1 \\ 1 \end{mymatrix}.
    \quad
    \vect{v}_2 = \begin{mymatrix}{r} -1 \\ 0 \\ 2 \end{mymatrix}.
    \quad\mbox{and}\quad
    \vect{v}_3 = \begin{mymatrix}{r} 0 \\ 1 \\ 0 \end{mymatrix}.
  \end{equation*}
  with corresponding eigenvalues $\eigenvar_1=1$, $\eigenvar_2=-1$,
  and $\eigenvar_3=4$. Therefore we can use
  \begin{equation*}
    P = \begin{mymatrix}{rrr}
      -1 & -1 & 0 \\
      1  &  0 & 1 \\
      1  &  2 & 0 \\
    \end{mymatrix}
    \quad\mbox{and}\quad
    D = \begin{mymatrix}{rrr}
      1 &  0 & 0 \\
      0 & -1 & 0 \\
      0 &  0 & 4 \\
    \end{mymatrix}.
  \end{equation*}
  To double-check that $P^{-1}AP$ is indeed equal to $D$, we first
  compute the inverse of $P$:
  \begin{equation*}
    P^{-1} =
    \begin{mymatrix}{rrr}
      -2 & 0 & -1 \\
      1  & 0 &  1 \\
      2  & 1 &  1 \\
    \end{mymatrix}.
  \end{equation*}
  Then
  \begin{equation*}
    P^{-1}AP
    ~=~
    \begin{mymatrix}{rrr}
      -2 & 0 & -1 \\
      1  & 0 &  1 \\
      2  & 1 &  1 \\
    \end{mymatrix}
    \begin{mymatrix}{rrr}
      3  & 0 &  2 \\
      6  & 4 &  3 \\
      -4 & 0 & -3 \\
    \end{mymatrix}
    \begin{mymatrix}{rrr}
      -1 & -1 & 0 \\
      1  &  0 & 1 \\
      1  &  2 & 0 \\
    \end{mymatrix}
    =
    \begin{mymatrix}{rrr}
      1 &  0 & 0 \\
      0 & -1 & 0 \\
      0 &  0 & 4 \\
    \end{mymatrix}
    = D.
  \end{equation*}
  Alternatively, we could have checked that $AP=PD$, which would not
  have required computing $P^{-1}$.
\end{solution}

\begin{example}{Diagonalizing a matrix}{diagonalize-matrix2}
  Diagonalize the matrix
  \begin{equation*}
    A=\begin{mymatrix}{rrr}
      2  &  0 &  0 \\
      1  &  4 & -1 \\
      -2 & -4 &  4 \\
    \end{mymatrix}.
  \end{equation*}
\end{example}

\begin{solution}
  First, we will find the characteristic polynomial of $A$:
  \begin{eqnarray*}
    \det(A-\eigenvar I)
    &=&
        \begin{absmatrix}{ccc}
          2-\eigenvar &  0 &  0 \\
          1 &  4-\eigenvar & -1 \\
          -2 & -4 & 4-\eigenvar \\
        \end{absmatrix}
    \\\\[-1ex]
    &=& (2-\eigenvar)
        \begin{absmatrix}{cc}
          4-\eigenvar & -1 \\
          -4 & 4-\eigenvar \\
        \end{absmatrix}
    \\\\[-2ex]
    &=& (2-\eigenvar)\paren{(4-\eigenvar)(4-\eigenvar)-4}
    \\
    &=& (2-\eigenvar)(12-8\eigenvar+\eigenvar^2)
    \\
    &=& (2-\eigenvar)(2-\eigenvar)(6-\eigenvar).
  \end{eqnarray*}
  Therefore, the eigenvalues are $\eigenvar=2$ and $\eigenvar=6$.
  Next, we need to find the eigenvectors. We first find the
  eigenvectors for $\eigenvar = 2$. We solve $(A-2I)\/\vect{v} = \vect{0}$:
  \begin{equation*}
    \begin{mymatrix}{rrr|r}
      0  &  0 &  0 & 0 \\
      1  &  2 & -1 & 0 \\
      -2 & -4 &  2 & 0 \\
    \end{mymatrix}
    \roweq
    \begin{mymatrix}{rrr|r}
      1  &  2 & -1 & 0 \\
      0  &  0 &  0 & 0 \\
      0  &  0 &  0 & 0 \\
    \end{mymatrix}.
  \end{equation*}
  The general solution is
  \begin{equation*}
    t\begin{mymatrix}{r} -2 \\ 1 \\ 0 \end{mymatrix}
    + s \begin{mymatrix}{r} 1 \\ 0 \\ 1 \end{mymatrix},
  \end{equation*}
  where $t,s$ are parameters. Thus, the basic eigenvectors for
  $\eigenvar=2$ are
  \begin{equation*}
    \vect{v}_1 = \begin{mymatrix}{r} -2 \\ 1 \\ 0 \end{mymatrix}
    \quad\mbox{and}\quad
    \vect{v}_2 = \begin{mymatrix}{r}  1 \\ 0 \\ 1 \end{mymatrix}.
  \end{equation*}
  Doing a similar calculation, we find that the basic eigenvector for
  $\eigenvar=6$ is
  \begin{equation*}
    \vect{v}_3 = \begin{mymatrix}{r}  0 \\ 1 \\ -2 \end{mymatrix}.
  \end{equation*}
  By Theorem~\ref{thm:eigenvectors-and-diagonalizable}, we use the
  eigenvectors of $A$ as the columns of $P$ and the corresponding
  eigenvalues as the diagonal entries of $D$. Therefore,
  \begin{equation*}
    P=
    \begin{mymatrix}{rrr}
      -2 & 1 & 0 \\
      1 & 0 & 1 \\
      0 & 1 & -2
    \end{mymatrix}
    \quad\mbox{and}\quad
    D = \begin{mymatrix}{rrr}
      2 & 0 & 0 \\
      0 & 2 & 0 \\
      0 & 0 & 6
    \end{mymatrix}.
  \end{equation*}
  We can double-check this answer by computing
  \begin{equation*}
    \def\arraystretch{1.3}
    P^{-1}AP
    ~=~ \begin{mymatrix}{rrr}
      -\frac{1}{4} & \frac{1}{2} & \frac{1}{4} \\
      \frac{1}{2} & 1 & \frac{1}{2} \\
      \frac{1}{4} & \frac{1}{2} & -\frac{1}{4}
    \end{mymatrix} \begin{mymatrix}{rrr}
      2 & 0 & 0 \\
      1 & 4 & -1 \\
      -2 & -4 & 4
    \end{mymatrix} \begin{mymatrix}{rrr}
      -2 & 1 & 0 \\
      1 & 0 & 1 \\
      0 & 1 & -2
    \end{mymatrix} \\
    ~=~ \begin{mymatrix}{rrr}
      2 & 0 & 0 \\
      0 & 2 & 0 \\
      0 & 0 & 6
    \end{mymatrix}
    ~=~ D.
  \end{equation*}
  Notice that the eigenvalues on the main diagonal of $D$ {\em must}
  be in the same order as the corresponding eigenvectors in $P$. Since
  the eigenvectors $\vect{v}_1$ and $\vect{v}_2$ are both for the
  eigenvalue $\eigenvar=2$, the entry $2$ appears twice in the matrix
  $D$.
\end{solution}

The following example shows that not all matrices are diagonalizable.

\begin{example}{A matrix that cannot be diagonalized}{impossible-diagonalize}
  Show that the matrix $A =
    \begin{mymatrix}{rr}
      1 & 1 \\
      0 & 1
    \end{mymatrix}$ cannot be diagonalized.
\end{example}

\begin{solution}
  Through the usual procedure, we find that the characteristic
  polynomial is $(1-\eigenvar)^2$, and therefore the only eigenvalue is
  $\eigenvar=1$. To find the eigenvectors, we solve the equation
  $(A-I)\vect{v} = \vect{0}$:
  \begin{equation*}
    \begin{mymatrix}{rr|r}
      0 & 1 & 0 \\
      0 & 0 & 0 \\
    \end{mymatrix}.
  \end{equation*}
  The general solution is
  \begin{equation*}
    \vect{v} = t\begin{mymatrix}{r} 1 \\ 0 \end{mymatrix}.
  \end{equation*}
  Because the solution space is $1$-dimensional, there is only one basic
  eigenvector:
  \begin{equation*}
    \vect{v}_1 = \begin{mymatrix}{r} 1 \\ 0 \end{mymatrix}.
  \end{equation*}
  Since the matrix $A$ has only one basic eigenvector, we cannot find
  two linearly independent eigenvectors. Therefore, by
  Theorem~\ref{thm:eigenvectors-and-diagonalizable}, $A$ cannot be
  diagonalized.
\end{solution}


\subsection{Elementary matrices and row operations}

Recall from Definition~\ref{def:row-operations} that there are three
kinds of elementary row operations%
\index{matrix!row operation}%
\index{matrix!elementary row operation}%
\index{row operation}%
\index{elementary row operation} on matrices:
\begin{enumerate}
\item Switch two rows.
\item Multiply a row by a non-zero number.
\item Add a multiple of one row to another row.
\end{enumerate}
The purpose of this section is to show that each of these row
operations corresponds to a special type of invertible matrix called
an \textbf{elementary matrix}%
\index{elementary matrix}%
\index{matrix!elementary matrix}.

\begin{example}{Elementary matrix for switching two rows}{elementary-matrix-1}
  Let
  \begin{equation*}
    E ~=~ \begin{mymatrix}{lll}
      1 & 0 & 0 \\
      0 & 0 & 1 \\
      0 & 1 & 0 \\
    \end{mymatrix}.
  \end{equation*}
  What is the effect of multiplying $E$ by an arbitrary $3\times
  n$-matrix $A$?
\end{example}

\begin{solution}
  Consider an arbitrary $3\times n$-matrix
  \begin{equation*}
    A ~=~ \begin{mymatrix}{cccc}
      a_{11} & a_{12} & \cdots & a_{1n} \\
      a_{21} & a_{22} & \cdots & a_{2n} \\
      a_{31} & a_{32} & \cdots & a_{3n} \\
    \end{mymatrix}.
  \end{equation*}
  We compute the product $EA$ by the row method:
  \begin{equation*}
    EA ~=~ \begin{mymatrix}{lll}
      1 & 0 & 0 \\
      0 & 0 & 1 \\
      0 & 1 & 0 \\
    \end{mymatrix}
    \begin{mymatrix}{cccc}
      a_{11} & a_{12} & \cdots & a_{1n} \\
      a_{21} & a_{22} & \cdots & a_{2n} \\
      a_{31} & a_{32} & \cdots & a_{3n} \\
    \end{mymatrix}
    ~=~
    \begin{mymatrix}{cccc}
      a_{11} & a_{12} & \cdots & a_{1n} \\
      a_{31} & a_{32} & \cdots & a_{3n} \\
      a_{21} & a_{22} & \cdots & a_{2n} \\
    \end{mymatrix}.
  \end{equation*}
  So the effect of multiplying $A$ by $E$ on the left is exactly the
  same as switching rows 2 and 3. We say that $E$ is the
  \textbf{elementary matrix for switching rows 2 and 3}.
\end{solution}

\begin{example}{Elementary matrix for multiplying a row by a non-zero number}{elementary-matrix-2}
  Let
  \begin{equation*}
    E ~=~ \begin{mymatrix}{lll}
      1 & 0 & 0 \\
      0 & k & 0 \\
      0 & 0 & 1 \\
    \end{mymatrix}.
  \end{equation*}
  What is the effect of multiplying $E$ by an arbitrary $3\times
  n$-matrix $A$?
\end{example}

\begin{solution}
  We compute the product $EA$ by the row method:
  \begin{equation*}
    EA ~=~ \begin{mymatrix}{lll}
      1 & 0 & 0 \\
      0 & k & 0 \\
      0 & 0 & 1 \\
    \end{mymatrix}
    \begin{mymatrix}{cccc}
      a_{11} & a_{12} & \cdots & a_{1n} \\
      a_{21} & a_{22} & \cdots & a_{2n} \\
      a_{31} & a_{32} & \cdots & a_{3n} \\
    \end{mymatrix}
    ~=~
    \begin{mymatrix}{rrrr}
      a_{11} & a_{12} & \cdots & a_{1n} \\
      ka_{21} & ka_{22} & \cdots & ka_{2n} \\
      a_{31} & a_{32} & \cdots & a_{3n} \\
    \end{mymatrix}.
  \end{equation*}
  So the effect of multiplying $A$ by $E$ on the left is exactly the
  same as multiplying row 2 by the scalar $k$. We say that $E$ is the
  \textbf{elementary matrix for multiplying row 2 by $k$}.
\end{solution}

\begin{example}{Elementary matrix for adding a multiple of one row to another row}{elementary-matrix-3}
  Let
  \begin{equation*}
    E ~=~ \begin{mymatrix}{lll}
      1 & 0 & 0 \\
      0 & 1 & 0 \\
      0 & k & 1 \\
    \end{mymatrix}.
  \end{equation*}
  What is the effect of multiplying $E$ by an arbitrary $3\times
  n$-matrix $A$?
\end{example}

\begin{solution}
  Once again we compute the product $EA$:
  \begin{equation*}
    EA ~=~ \begin{mymatrix}{lll}
      1 & 0 & 0 \\
      0 & 1 & 0 \\
      0 & k & 1 \\
    \end{mymatrix}
    \begin{mymatrix}{cccc}
      a_{11} & a_{12} & \cdots & a_{1n} \\
      a_{21} & a_{22} & \cdots & a_{2n} \\
      a_{31} & a_{32} & \cdots & a_{3n} \\
    \end{mymatrix}
    ~=~
    \begin{mymatrix}{cccc}
      a_{11} & a_{12} & \cdots & a_{1n} \\
      a_{21} & a_{22} & \cdots & a_{2n} \\
      a_{31}+ka_{21} & a_{32}+ka_{22} & \cdots & a_{3n}+ka_{2n} \\
    \end{mymatrix}.
  \end{equation*}
  So the effect of multiplying $A$ by $E$ on the left is exactly the
  same as adding $k$ times row 2 to row 3. We say that $E$ is the
  \textbf{elementary matrix for adding $k$ times row 2 to row 3}.
\end{solution}

As these examples show, performing each type of elementary row
operation is the same as multiplying (on the left) by a certain
invertible matrix. These matrices are called the \textbf{elementary
  matrices}%
\index{elementary matrix}%
\index{matrix!elementary matrix}. In the
above examples, we have only considered $3\times 3$-elementary
matrices, but they exist for other sizes too. The following definition
makes this precise. It also shows how to calculate the elementary
matrix corresponding to any elementary row operation.

\begin{definition}{Elementary matrices and row operations}{elementary-matrices-and-row-operations}
  Let $E$ be an $n\times n$-matrix. Then $E$ is an \textbf{elementary
    matrix}%
  \index{elementary matrix}%
  \index{matrix!elementary matrix}
  if it is the result of applying one elementary row operation to the
  $n\times n$ identity matrix.
\end{definition}

\begin{example}{Finding an elementary matrix}{finding-elementary-matrix}
  Consider the elementary row operation of adding $5$ times row 3 to
  row 1 of a $4\times n$-matrix. Find the elementary matrix $E$
  corresponding to this row operation.
\end{example}

\begin{solution}
  Following Definition~\ref{def:elementary-matrices-and-row-operations}, all
  we have to do is apply the desired row operation to the
  $4\times 4$-identity matrix:
  \begin{equation*}
    \begin{mymatrix}{rrrr}
      1 & 0 & 0 & 0 \\
      0 & 1 & 0 & 0 \\
      0 & 0 & 1 & 0 \\
      0 & 0 & 0 & 1 \\
    \end{mymatrix}
    \quad
    \stackrel{R_1\rowop R_1+5R_3}{\roweq}
    \quad
    \begin{mymatrix}{rrrr}
      1 & 0 & 5 & 0 \\
      0 & 1 & 0 & 0 \\
      0 & 0 & 1 & 0 \\
      0 & 0 & 0 & 1 \\
    \end{mymatrix}
    ~=~ E.
  \end{equation*}
\end{solution}

We can double-check that multiplying $E$ by any $4\times n$-matrix
does indeed have the desired effect:
\begin{equation*}
  \begin{mymatrix}{rrrr}
    1 & 0 & 5 & 0 \\
    0 & 1 & 0 & 0 \\
    0 & 0 & 1 & 0 \\
    0 & 0 & 0 & 1 \\
  \end{mymatrix}
  \begin{mymatrix}{cccc}
    a_{11} & a_{12} & \cdots & a_{1n} \\
    a_{21} & a_{22} & \cdots & a_{2n} \\
    a_{31} & a_{32} & \cdots & a_{3n} \\
    a_{41} & a_{42} & \cdots & a_{4n} \\
  \end{mymatrix}
  ~=~
  \begin{mymatrix}{cccc}
    a_{11}+5a_{31} & a_{12}+5a_{32} & \cdots & a_{1n}+5a_{3n} \\
    a_{21} & a_{22} & \cdots & a_{2n} \\
    a_{31} & a_{32} & \cdots & a_{3n} \\
    a_{41} & a_{42} & \cdots & a_{4n} \\
  \end{mymatrix}.
\end{equation*}
The fact that this always works is the content of the following
theorem.

\begin{theorem}{Multiplication by an elementary matrix and row operations}{multiplication-by-elementary-matrix}
  Performing any of the three elementary row operations on a matrix $A$ is the
  same as taking the product $EA$, where $E$ is the elementary matrix
  obtained by applying the desired row operation to the identity
  matrix.
\end{theorem}


\subsection{Cauchy-Schwarz inequality}

This states that:

\(|\langle u,v\rangle |^2 \le \langle u, u\rangle \dot \langle v, v\rangle \)

Consider the vectors \(u\) and \(v\). We construct a third vector \(u-\lambda v\).
We know the length of any vector is non-negative.
\(0\le \langle u-\lambda v, u-\lambda v\rangle\)

\(0\le \langle u, u\rangle+ \langle u, -\lambda v\rangle+\langle -\lambda v, u\rangle+ \langle -\lambda v, -\lambda v\rangle\)

\(0\le \langle u, u\rangle-\bar{\lambda }\langle u, v\rangle-\lambda { \langle v, u\rangle }+ \lambda \bar{\lambda }\langle v, v\rangle\)

We now look for a value of \(\lambda \) to simplify this equation.

\(\lambda = \dfrac{\langle u,v \rangle}{\langle v, v\rangle}\)

\(0\le \langle u, u\rangle-\dfrac{\langle v,u \rangle\langle u, v\rangle}{\langle v, v\rangle}-\dfrac{\langle u,v \rangle  \langle v, u\rangle }{\langle v, v\rangle}+ \dfrac{\langle u,v \rangle}{\langle v, v\rangle}\dfrac{\langle v,u \rangle}{\langle v, v\rangle}\langle v, v\rangle\)

\(0\le \langle u, u\rangle-\dfrac{|\langle u,v \rangle|^2}{\langle v, v\rangle}\)

\(|\langle u,v \rangle|^2\ge \langle u, u\rangle\langle v, v\rangle\)


The original Bresenham Algorithms was developed by J. Bresenham in 1965~\cite{Bresenham65Line}. The goal was to build an algorithm to draw a line between to coordinates on a pixel grid. The family of Bresenham Algorithms that exist, e.g. to draw a circle were developed by others. In the following section, the Bresenham Line and Circle algorithm will be explained.
\subsubsection*{Line Drawing}
The original line drawing algorithm uses only additions, subtractions, and a multiplication with the constant value of two.

\begin{algorithm}
	\caption{Bresenham Line Drawing Algorithm Part \rom{1}}
	\label{alg:bresline}
	\KwIn{\(x_0,~ y_0,~ x_1,~ y_1\)}
	\SetKwBlock{Begin}{Begin}{}
	\SetAlgoLined
	\Begin{
		\(\varDelta_x \gets \abs{x_1 - x_0}\)\;
		\(\varDelta_y \gets \abs{y_1 - y_0}\)\;
		\(\text{step}_x \gets x_0 < x_1~ ? ~1 ~:~ -1 \)\;
		\(\text{step}_y \gets y_0 < y_1~ ?~ 1 ~:~ -1\)\;
		err \( \gets \varDelta_x + \varDelta_y \)
	}
\end{algorithm}
\begin{algorithm}
	\caption{Bresenham Line Drawing Algorithm Part \rom{2}}
	\SetAlgoVlined
	%This is to hide Begin keyword
	\SetKwBlock{Begin}{}{end}
	\Begin{
		\While{\(x_0 \neq x_1\) and \(y_0 \neq y_1\)}{
			draw Pixel at \(x_0 | y_0\) \;
			err2 \(\gets 2 \times err\)\;
			\If{err2 \(> \varDelta_y\)}{
				err \(\gets \text{err} + \varDelta_y\)\;
				\(x_0 \gets x_0 + \text{step}_x\)\;
			}
			\If{err2 \(< d_x\)}{
				err \(\gets \text{err} + \varDelta_x\)\;
				\(y_0 \gets y_0 + \text{step}_y\)\;
			}
		}
	}
\end{algorithm}

As seen in \cref{alg:bresline} the first step is to set the start coordinate. After that an error variable is set. The error variable is for evaluating, which pixel to draw next. Is the value of the error variable lesser than zero it goes one step in the x direction, is the value greater or equal than zero it goes one step in the y and one step in the x direction. It ends when the new x and y coordinate equal to the destination x and y coordinate.

\subsubsection*{Circle Drawing}
For the Circle Drawing Algorithm often referred to as the Bresenham Circle Drawing Algorithm or the Midpoint circle drawing algorithm, we first have to define smaller in a Cartesian coordinate system. Usually such a coordinate system consists of four areas called quadrants. If we now go an split each quadrant up into two area with a diagonal line with its origin at the the center of the coordinate system, we create areas called octants.

\begin{algorithm}[H]

	\KwIn{\(x_c\), \(y_c\), radius}
	\SetKwBlock{Begin}{Begin}{End}
	\SetAlgoLined
	\Begin{
	\(x \gets 0\)\;
	\(y \gets \text{radius}\)\;
	\(\text{err} \gets 3 - 2 \times \text{radius}\)\;
	\While{\(y \geqslant x\)}{
		\(x \gets x + 1\)\;
		\eIf{\(\text{err} > 0\)}{
			\(y \gets y - 1\)\;
			\(\text{err} \gets \text{err} + 4 \times (x - y) + 10\)\;
		}{
			\(\text{err} \gets \text{err} + 4 \times x + 6\)\;
		}
		Draw corresponding pixel in each quadrant\;
	}
}
	\caption{Bresenham Circle Drawing Algorithm}
	\label{alg:brescircle}
\end{algorithm}

As seen in \cref{alg:brescircle} the fundamental function of the two algorithms are the same, the main difference lies is how many pixels are drawn at the same time and the calculation of the error value.


\subsection{Performance Metrics}
We measure the accuracy of the model using two error metrics:
mean absolute error (MAE) and root mean squared error (RMSE).
These are defined as
\begin{align}
  \text{MAE} &= \frac{1}{N}\sum_{i=1}^N\left|y_i - \hat{y}_i\right|\\
  \text{RMSE} &= \sqrt{\frac{1}{N}\sum_{i=1}^N\left(y_i - \hat{y}_i\right)^2}
  \intertext{where}
  \hat{y_i} &= \text{predicted output}\nonumber\\
  y_i &= \text{true value}\nonumber
\end{align}
The MAE measures the expected error throughout the forecast horizon. The RMSE
indicates the presence of significant but infrequent errors. Since the data were
normalized by system capacity \cite{wang_quantifying_2016}, the error metrics
are easily interpretable.
In order to compare how each individual weather input changed the forecast accuracy, we calculated a ``percent improvement'' over the
univariate case (i.e. a demand prediction based only on historical demand data).
This percent improvement is calculated by
\begin{align}
  \text{\% Improvement} &= \frac{\hat{e} - e}{e}\times 100, \text{ [-]}
  \intertext{where}
  e &= \text{error from the univariate forecast}\nonumber\\
  \hat{e} &= \text{error from the duovariate forecast.}\nonumber
\end{align}
The sign indicates the direction of change in
error. Finally, in order to facilitate comparison with other work, we calculated
the normalized root mean squared error (NRMSE) by
\begin{align}
  NRMSE &= \sqrt{\frac{\sum_{i=1}^N\left(y_i - \hat{y}_i\right)^2}{\sum_{i=1}^{N}\left(y_i - \tilde{y_i}\right)^2}}
  \intertext{where}
  \tilde{y} &= \text{mean of the target set}\nonumber
\end{align}