\section{Improper Integrals}\label{sec:ImproperIntegrals}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Recall that the Fundamental Theorem of Calculus says that if $f$ is a \underline{{\bf continuous}} function on the closed interval $[a,b]$, then
$$\ds{\int_a^b f(x)~dx=F(x)\bigg|_a^b=F(b)-F(a)},$$
where $F$ is any antiderivative of $f$.

Both the {\bf continuity} condition and {\bf closed interval} must hold to use the Fundamental Theorem of Calculus, and in this case, $\ds\int_a^b f(x)\,dx$ represents the net area under $f(x)$ from $a$ to $b$:
$$\includegraphics[width=2.25in]{images2/area-under}$$

We begin with an example where blindly applying the Fundamental Theorem of Calculus can give an incorrect result.

\begin{example}{Using FTC}{Using FTC}
Explain why $\ds\int_{-1}^1\frac{1}{x^2}\,dx$ is not equal to $-2$.
\end{example}

\begin{solution}
Here is how one might proceed:
$$
\int_{-1}^1\frac{1}{x^2}\,dx
~=~ \int_{-1}^1 x^{-2}\,dx
~=~ -x^{-1}\bigg|_{-1}^1
~=~ -\frac{1}{x}\bigg|_{-1}^1
~=~ \left(-\frac{1}{1}\right) - \left(-\frac{1}{(-1)}\right)
~=~ -2$$
 However, the above answer is {\bf WRONG!}
 Since $f(x)=1/x^2$ is not continuous on $[-1,~1]$, we cannot directly apply the Fundamental Theorem of Calculus.
Intuitively, we can see why $-2$ is not the correct answer by looking at the graph of $f(x)=1/x^2$ on $[-1,~1]$.
The shaded area appears to grow without bound as seen in the figure below.
$$\includegraphics[width=2.5in]{images2/improper-integral-example-1}$$
\end{solution}

Formalizing this example leads to the concept of an improper integral.
There are two ways to extend the Fundamental Theorem of Calculus.
One is to use an {\bf infinite interval}, i.e., $[a,\infty)$, $(-\infty,b]$ or $(-\infty,\infty)$.
The second is to allow the interval $[a,b]$ to contain an infinite {\bf discontinuity} of $f(x)$.
In either case, the integral is called an {\bf improper integral}.
One of the most important applications of this concept is probability distributions.

To compute improper integrals, we use the concept of limits along with the Fundamental Theorem of Calculus.

\begin{definition}{Definitions for Improper Integrals}{Definitions for Improper Integrals}
If $f(x)$ is continuous on $[a,\infty)$, then the improper integral of $f$ over $[a,\infty)$ is:
$$\int_{a}^{\infty} f(x)\,dx:=\lim_{R\to\infty}\int_a^R f(x)\,dx.$$
If $f(x)$ is continuous on $(-\infty,b]$, then the improper integral of $f$ over $(-\infty,b]$ is:
$$\int_{-\infty}^b f(x)\,dx:=\lim_{R\to -\infty}\int_R^b f(x)\,dx.$$
\end{definition}

If the limit exists and is a finite number, we say the improper integral {\bf converges}. Otherwise, we say the  improper integral {\bf diverges}.

To get an intuitive (though not completely correct) interpretation of improper integrals, we attempt to analyze $\ds\int_a^\infty f(x)\,dx$ graphically.
 Here assume $f(x)$ is continuous on $[a,\infty)$:
$$\includegraphics[width=5in]{images2/improper-integral-theory-2}$$
 We let $R$ be a fixed number in $[a,\infty)$.
 Then by taking the limit as $R$ approaches $\infty$, we get the improper integral:
$$\int_a^\infty f(x)\,dx:=\lim_{R\to\infty}\int_a^R f(x)\,dx.$$
 We can then apply the Fundamental Theorem of Calculus to the last integral as $f(x)$ is continuous on the closed interval $[a,R]$.

We next define the improper integral for the interval $(-\infty,~\infty)$.

\begin{definition}{Definitions for Improper Integrals}{Definitions for Improper Integrals}
 If both $\ds\int_{-\infty}^a f(x)\,dx$ and $\ds\int_{a}^{\infty} f(x)\,dx$ are convergent, then the improper integral of $f$ over $(-\infty,\infty)$ is:
$$\int_{-\infty}^{\infty} f(x)\,dx:=\int_{-\infty}^a f(x)\,dx+\int_{a}^{\infty} f(x)\,dx$$
\end{definition}

The above definition requires {\bf both} of the integrals
$$\int_{-\infty}^a f(x)\,dx\qquad\mbox{and}\qquad\int_{a}^{\infty} f(x)\,dx$$
to be convergent for $\ds\int_{-\infty}^{\infty} f(x)\,dx$ to also be convergent.
 If {\bf either} of $\ds\int_{-\infty}^a f(x)\,dx$ or $\ds\int_{a}^{\infty} f(x)\,dx$ is divergent, then so is $\ds\int_{-\infty}^{\infty} f(x)\,dx$.

\begin{example}{Improper Integral}{Improper Integral}
Determine whether $\ds\int_1^\infty\frac{1}{x}\,dx$ is convergent or divergent.
\end{example}

\begin{solution}
 Using the definition for improper integrals we write this as:
$$	\int_1^\infty \frac{1}{x}\,dx= \lim_{R\to\infty} \int_1^R\frac{1}{x}\,dx
	= \lim_{R\to\infty} \ln|x|\bigg|_1^R
	=\lim_{R\to\infty} \ln|R| - \ln|1|
	= \lim_{R\to\infty} \ln|R|
	= +\infty$$
 Therefore, the integral is {\bf divergent}.
\end{solution}

\begin{example}{Improper Integral}{Improper Integral}
Determine whether $\ds\int_{-\infty}^\infty x\sin(x^2)\,dx$ is convergent or divergent.
\end{example}

\begin{solution}
 We must compute both $\ds\int_0^\infty x\sin(x^2)\,dx$ and $\ds\int_{-\infty}^0 x\sin(x^2)\,dx$.
 Note that we don't have to split the integral up at $0$, any finite value $a$ will work.
 First we compute the indefinite integral.
 Let $u=x^2$, then $du=2x\,dx$ and hence,
$$\int x\sin(x^2)\,dx=\frac{1}{2} \int \sin(u)\,du=-\frac{1}{2}\cos(x^2)+C$$
Using the definition of improper integral gives:
$$\int_0^\infty x\sin(x^2)\,dx  =  \lim_{R\to\infty} \int_0^R x\sin(x^2)\,dx
=\lim_{R\to\infty} \left[-\frac{1}{2}\cos(x^2)\right] \bigg|_0^R
=  -\frac{1}{2} \lim_{R\to\infty} \cos(R^2) +\frac{1}{2}$$
This limit does not exist since $\cos x$ {\bf oscillates} between $-1$ and $+1$.
 In particular, $\cos x$ does not approach any particular value as $x$ gets larger and larger.
 Thus, $\ds\int_0^\infty x\sin(x^2)\,dx$ diverges, and hence, $\ds\int_{-\infty}^\infty x\sin(x^2)\,dx$ diverges.
\end{solution}

When there is a discontinuity in $[a,b]$ or at an endpoint, then the improper integral is as follows.

\begin{definition}{Definitions for Improper Integrals}{Definitions for Improper Integrals}
If $f(x)$ is continuous on $(a,b]$, then the improper integral of $f$ over $(a,b]$ is:
$$\int_a^b f(x)\,dx:=\lim_{R\to a^+}\int_R^b f(x)\,dx.$$
If $f(x)$ is continuous on $[a,b)$, then the improper integral of $f$ over $[a,b)$ is:
$$\int_a^b f(x)\,dx:=\lim_{R\to b^-}\int_a^R f(x)\,dx.$$
\end{definition}

If the limit above exists and is a finite number, we say the improper integral {\bf converges}.
Otherwise, we say the improper integral {\bf diverges}.

When there is a discontinuity in the interior of $[a,b]$, we use the following definition.

\begin{definition}{Definitions for Improper Integrals}{Definitions for Improper Integrals}
If $f$ has a discontinuity at $x=c$ where $c\in[a,b]$, and both
both $\ds\int_a^c f(x)\,dx$ and $\ds\int_c^b f(x)\,dx$ are convergent, then $f$ over $[a,b]$ is:
$$\int_a^b f(x)\,dx:=\int_a^c f(x)\,dx+\int_c^b f(x)\,dx$$
\end{definition}

Again, we can get an intuitive sense of this concept by analyzing $\ds\int_a^b f(x)\,dx$ graphically.
Here assume $f(x)$ is continuous on $(a,b]$ but discontinuous at $x=a$:
$$\includegraphics[width=5in]{images2/improper-integral-theory-1}$$
 We let $R$ be a fixed number in $(a,b)$.
 Then by taking the limit as $R$ approaches $a$ from the {\bf right},  we get the improper integral:
$$\int_a^b f(x)\,dx:=\lim_{R\to a^+}\int_R^b f(x)\,dx.$$
 Now we can apply FTC to the last integral as $f(x)$ is continuous on $[R,b]$.

\begin{example}{A Divergent Integral}{A Divergent Integral}
Determine if $\ds\int_{-1}^1\frac{1}{x^2}\,dx$ is convergent or divergent.
\end{example}

\begin{solution}
 The function $f(x)=1/x^2$ has a discontinuity at $x=0$, which lies in $[-1,1]$.
 We must compute $\ds\int_{-1}^0 \frac{1}{x^2}\,dx$ and $\ds\int_0^1 \frac{1}{x^2}\,dx$. Let's start with $\ds\int_0^1 \frac{1}{x^2}\,dx$:
$$\int_0^1 \frac{1}{x^2}\,dx = \lim_{R\to 0^+} \int_R^1 \frac{1}{x^2} \,dx = \lim_{R\to 0^+} -\frac{1}{x}\bigg|_R^1
= -1 + \lim_{R\to 0^+} \frac{1}{R}$$
which diverges to $+\infty$.
 Therefore, $\ds\int_{-1}^1\frac{1}{x^2}\,dx$ is {\bf divergent} since one of $\ds\int_{-1}^0 \frac{1}{x^2}\,dx$ and $\ds\int_0^1 \frac{1}{x^2}\,dx$ is divergent.
\end{solution}

% % % % % % %
% % The following example 'Integral of the Logarithm' can be replaced with the next example if Integration by Parts is not included in the text adaptation.
\begin{example}{Integral of the Logarithm}{Integral of the Logarithm}
Determine if $\ds\int_0^1 \ln x \,dx$ is convergent or divergent. Evaluate it if it is convergent.
\end{example}

\begin{solution}
Note that $f(x)=\ln x$ is discontinuous at the endpoint $x=0$.
We first use integration by parts to compute $\ds\int\ln x\,dx$.
We let $u=\ln x$ and $dv=dx$.
Then $du=(1/x)dx$, $v=x$, giving:
\begin{eqnarray*}
\int \ln x\,dx &=& \ds x\ln x-\int x\cdot\frac{1}{x}\,dx\\
&=& x\ln x-\int 1\,dx\\
&=& x\ln x-x+C\\
\end{eqnarray*}
 Now using the definition of improper integral for $\ds\int_0^1 \ln x \,dx$:
$$
	\int_0^1 \ln x \,dx = \lim_{R\to 0^+} \int_R^1 \ln x\,dx
	=  \lim_{R\to 0^+} (x\ln x-x)\bigg|_R^1
	%=  \lim_{R\to 0^+} \left(\left[\ln 1-1\right] - \left[R\ln R-R\right]\right)
	=  -1 - \lim_{R\to 0^+}(R\ln R) + \lim_{R\to 0^+}R
$$
Note that $\ds\lim_{R\to 0^+}R=0$.
We next compute $\ds\lim_{R\to 0^+}(R\ln R)$.
First, we rewrite the expression as follows:
$$\lim_{x\to0^+}(R\ln R)=\lim_{R\to0^+}\frac{\ln R}{1/R}.$$
Now the limit is of the indeterminate type $(-\infty)/(\infty)$ and l'H\^opital's Rule can be applied.
$$\lim_{R\to0^+}(R\ln R) =\lim_{R\to0^+}\frac{\ln R}{1/R}
=\lim_{R\to0^+}\frac{1/R}{-1/R^2}
=\lim_{R\to0^+}-\frac{R^2}{R}
=\lim_{R\to0^+}(-R)
=0$$
Thus, $\ds\lim_{R\to 0^+}(R\ln R)=0$.
Thus
$$\int_0^1 \ln x \,dx = -1,$$
and the integral is convergent to $-1$.

Graphically, one might interpret this to mean that the net area under $\ln x$ on $[0,1]$ is $-1$ (the area in this case lies below the $x$-axis).
$$\includegraphics[width=2.5in]{images2/improper-integral-example-2}$$
\end{solution}

\begin{example}{Integral of a Square Root}{IntSquareRoot}
Determine if $\ds\int_0^4\frac{dx}{\sqrt{4-x}}$ is convergent or divergent. Evaluate it if it is convergent.
\end{example}
\begin{solution}
Note that $\frac{1}{\sqrt{4-x}}$ is discontinuous at the endpoint $x=4$. We use a $u$-substitution to compute $\int \frac{dx}{\sqrt{4-x}}$. We let $u=4-x$, then $du=-dx$, giving:
\begin{align*}
\ds\int\frac{dx}{\sqrt{4-x}}&=\int-\frac{du}{u^{1/2}}	\\
&=\int -u^{-1/2}\,du	\\
&=-2(u)^{1/2}+C	\\
&=-2\sqrt{4-x}+C
\end{align*}
Now using the definition of improper integrals for $\ds\int_0^4\frac{dx}{\sqrt{4-x}}$:
\[\ds\int_0^4\frac{dx}{\sqrt{4-x}}=\lim_{R\to4^{-}}(-2\sqrt{4-x})\bigg|_0^R=\lim_{R\to4^{-}}-2\sqrt{4-R}+2\sqrt{4}=4\]
\end{solution}

\begin{example}{Improper Integral}{ImpIntExample}
Determine if $\ds\int_{1}^{2}\dfrac{dx}{\left( x-1\right) ^{1/3}}$ is
convergent or divergent. Evaluate it if it is convergent.
\end{example}
\begin{solution}
Note that $f\left( x\right) =\dfrac{1}{\left( x-1\right) ^{1/3}}$
is discontinuous at the endpoint $x=1.$ We first use substitution to find
$\ds\int \dfrac{dx}{\left( x-1\right) ^{1/3}}.$ We let $u=x-1.$ Then $du=dx,$
giving%
\begin{equation*}
\int \dfrac{dx}{\left( x-1\right) ^{1/3}}=\int \frac{du}{u^{1/3}}=\int
u^{-1/3}du=\frac{3}{2}u^{2/3}+C=\frac{3}{2}\left( x-1\right) ^{2/3}+C.
\end{equation*}%
Now using the definition of improper integral for $\ds\int_{1}^{2}\dfrac{dx}{%
	\left( x-1\right) ^{1/3}}:$%
\begin{equation*}
\int_{1}^{2}\dfrac{dx}{\left( x-1\right) ^{1/3}}=\lim_{R\rightarrow
	1^{+}}\int_{R}^{2}\dfrac{dx}{\left( x-1\right) ^{1/3}}=\left.
\lim_{R\rightarrow 1^{+}}\frac{3}{2}\left( x-1\right) ^{2/3}\right\vert
_{R}^{2}=\frac{3}{2}-\lim_{R\rightarrow 1^{+}}\frac{3}{2}\left( R-1\right)
^{2/3}=\frac{3}{2},
\end{equation*}%
and the integral is convergent to $\frac{3}{2}.$ Graphically, one might
interpret this to mean that the net area under $\dfrac{1}{\left( x-1\right)
	^{1/3}}$ on $\left[ 1,2\right] $ is $\frac{3}{2}$.

$$\includegraphics[width=8cm]{images/improper-int-example}$$
\end{solution}

The following test allows us to determine convergence/divergence information about improper integrals that are hard to compute by comparing them to easier ones.
We state the test for $[a,\infty)$, but similar versions hold for the other improper integrals.

\begin{formulabox}[The Comparison Test]
Assume that $f(x)\geq g(x)\geq 0$ for $x\geq a$.
\begin{enumerate}[(i)]
\item	If $\ds\int_a^\infty f(x)\,dx$ {\bf converges}, then $\ds\int_a^\infty g(x)\,dx$ also {\bf converges}.
\item	If $\ds\int_a^\infty g(x)\,dx$ {\bf diverges}, then $\ds\int_a^\infty f(x)\,dx$ also {\bf diverges}.
\end{enumerate}
\end{formulabox}

Informally, (i) says that if $f(x)$ is larger than $g(x)$, and the area under $f(x)$ is finite (converges), then the area under $g(x)$ must also be finite (converges).
Informally, (ii) says that if $f(x)$ is larger than $g(x)$, and the area under $g(x)$ is infinite (diverges), then the area under $f(x)$ must also be infinite (diverges).

$$\includegraphics[width=6in]{images2/improper-integral-theory-3}$$

\begin{example}{Comparison Test}{Comparison Test}
Show that $\ds\int_2^\infty \frac{\cos^2x}{x^2} \,dx$ converges.
\end{example}

\begin{solution}
We use the comparison test to show that it converges.
Note that $0\leq \cos^2x\leq 1$ and hence
$$0 \leq\frac{\cos^2x}{x^2}\leq\frac{1}{x^2}.$$
Thus, taking $f(x)=1/x^2$ and $g(x)=\cos^2x / x^2$ we have $f(x)\geq g(x)\geq 0$.
One can easily see that $\ds\int_2^\infty \frac{1}{x^2}\,dx$ converges.
Therefore, $\ds\int_2^\infty \frac{\cos^2x}{x^2} \,dx$ also converges.
\end{solution}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\Opensolutionfile{solutions}[ex]
\section*{Exercises for Section \ref{sec:ImproperIntegrals}}

\begin{enumialphparenastyle}

\begin{ex}
	Determine whether $\ds\int_1^{\infty}\frac{1}{x^2}\,dx$ is convergent or divergent.
	\begin{sol}
		Converges to 1.
	\end{sol}
\end{ex}

\begin{ex}
	Determine whether $\ds\int_e^{\infty}\frac{1}{x\sqrt{\ln x}}\,dx$ is convergent or divergent.
	\begin{sol}
		Diverges.
	\end{sol}
\end{ex}

\begin{ex}
	Evaluate the improper integral $\ds\int_0^{\infty}e^{-3x}\,dx$.
	\begin{sol}
		1/3
	\end{sol}
\end{ex}

\begin{ex}
	Determine if $\ds\int_1^e\frac{1}{x(\ln x)^2}\,dx$ is convergent or divergent. Evaluate it if it is convergent.
	\begin{sol}
		Divergent.
	\end{sol}
\end{ex}

\begin{ex}
	Show that $\ds\int_0^{\infty}e^{-x}\sin^2\left(\frac{\pi x}{2}\right)\,dx$ converges.
\end{ex}

\begin{ex}
	Evaluate $\ds\int_{-\infty}^{\infty}\frac{1}{x^2+1}\,dx$ and $\ds\int_{-\infty}^{\infty}\frac{x}{x^2+1}\,dx$.
\end{ex}

\begin{ex}
	Determine whether the following improper integrals are convergent or divergent. Evaluate those that are convergent.
	\begin{enumerate}
		\item	$\int_{0}^{\infty}\dfrac{1}{x^2+1}\,dx$
		\item	$\int_{0}^{\infty}\dfrac{x}{x^2+1}\,dx$
		\item	$\int_{0}^{\infty}e^{-x}(\cos x+\sin x)\,dx$. [Hint: What is the derivative of $-e^{-x}\cos x?$]
		\item	$\int_{0}^{\pi/2}\sec^{2}x\,dx$
		\item	$\int_{0}^{4}\dfrac{1}{(4-x)^{2/5}}\,dx$
	\end{enumerate}
	\begin{sol}
		\begin{enumerate}
			\item	$\pi/2$
			\item	divergent (to $\infty$)
			\item	1
			\item	divergent (to $\infty$)
			\item	$\frac{5}{3}(4^{3/5})$
		\end{enumerate}
	\end{sol}
\end{ex}

\begin{ex}
	Prove that the integral $\int_{1}^{\infty}\dfrac{1}{x^p}\,dx$ is convergent if $p>1$ and divergent if $0<p\leq 1$.
\end{ex}

\begin{ex}
	Suppose that $p>0$. Find all values of $p$ for which $\int_{0}^{1}\dfrac{1}{x^p}\,dx$ converges.
	\begin{sol}
		$0<p<1$
	\end{sol}
\end{ex}

\begin{ex}
	Show that $\int_{1}^{\infty}\dfrac{\sin^2 x}{x(\sqrt{x}+1)}\,dx$ converges.
\end{ex}

\end{enumialphparenastyle}


\documentclass{article}

% Language setting
% Replace `english' with e.g. `spanish' to change the document language
\usepackage[english]{babel}

\usepackage{witharrows}

% Set page size and margins
% Replace `letterpaper' with `a4paper' for UK/EU standard size
\usepackage[letterpaper,top=2cm,bottom=2cm,left=3cm,right=3cm,marginparwidth=1.75cm]{geometry}

% Useful packages
\usepackage{amsmath}
\usepackage{graphicx}
\usepackage[colorlinks=true, allcolors=blue]{hyperref}

\title{Your Paper}
\author{You}

\begin{document}
\maketitle

\section{Uniswapv1}

\begin{equation}
   x*y=k \quad  \text{where:} \quad
   \begin{gathered}
   x \quad \text{is total amount of asset A} \\
   y \quad \text{is total amount of asset B} \\
   k \quad \text{is constant} \\
    \end{gathered}
\end{equation}

\begin{equation}
  (x+\Delta x)*(y-\Delta y)=k \quad  \text{where:} \quad
  \begin{gathered}
   x \quad \text{is total amount of asset A} \\
   \Delta x \quad \text{is sold amount of asset A} \\
   y \quad \text{is total amount of asset B} \\
   \Delta y \quad \text{is bought amount of asset B} \\
   k \quad \text{is constant} \\
    \end{gathered}
\end{equation}

\begin{equation}
\begin{WithArrows}
x*y = (x+\Delta x)*(y-\Delta y) \\
y-\Delta y = \dfrac{x*y}{x+\Delta x}
\end{WithArrows}
\end{equation}

\begin{equation}
\begin{WithArrows}
\Delta y &= y - \dfrac{x*y}{x+\Delta x} \\
&= \dfrac{y*(x+\Delta x)}{x+\Delta x} - \dfrac{x*y}{x+\Delta x} \\
&= \dfrac{y*x +y \Delta x - x*y}{x+\Delta x} \\
&= \dfrac{y \Delta x}{x+\Delta x}
\end{WithArrows}
\end{equation}

Let's consider that \(\Delta x_f\) is the sold amount of asset A \(\Delta x\) minus the fees\\
\begin{equation}
\begin{WithArrows}
   \Delta x_f &= \Delta  x - \Delta x*0,3\% \\
   &= \Delta x * (1-0,3\%) \\
   &= \Delta x * (\dfrac{1000}{1000}-\dfrac{3}{1000}) \\
   &= \Delta x * \dfrac{997}{1000}
 \end{WithArrows}
\end{equation}

\begin{equation}
\begin{WithArrows}
\Delta y &=  \dfrac{y \Delta x_f}{x+\Delta x_f}
\end{WithArrows}
\end{equation}

\begin{equation}
\begin{WithArrows}
\Delta y &=  \dfrac{y \Delta x_f}{x+\Delta x_f} \\
&= \dfrac{\dfrac{997*y*\Delta x}{1000}}{x+\dfrac{997*\Delta x}{1000}} \\
&= \dfrac{997 ’y \Delta x}{1000 x + 997 \Delta x}
\end{WithArrows}
\end{equation}

\end{document}


%\begin{appendices}
\chapter{Derivation of the objective function $J(\mu)$}
\label{derivation1}
The objective function to be minimized is given by
\[ \min_{k \in K_{conv}} \frac{1}{2}\sum_{i=1}^n \norm{x_i - \sum_{x_j \in B_i} k_{ij}x_j}^2 + \gamma* \sum_{i=1}^n \sum_{x_j \in B_i} k_{ij} \norm{x_i-x_j}^2 \]
Expanding the norm on the first part of the sum
\begin{equation}
\begin{split}
\min_{k \in K_{conv}} \frac{1}{2}\sum_{i=1}^n \bigg(\norm{x_i}^2 - 2*\sum_{x_j \in B_i} k_{ij} (x_i \cdot x_j)  + \\ k_i k_i^T \circ d_i d_i^T \circ X^TX \bigg) + \gamma* \sum_{i=1}^n \sum_{x_j \in B_i} k_{ij} \norm{x_i-x_j}^2
\end{split}
\label{new_obj}
\end{equation}
The notation `$\circ$' denotes elementwise multiplication of two vectors. Here the summation $\sum_{i=1}^n \norm{x_i}^2$ can be discarded, since it is independent of the optimization parameters. Substituting $X^TX = P$,
\[\sum_{x_j \in B_i} k_{ij} (x_i \cdot x_j) = k_i \circ d_i \circ p_i\]
and
\[ \sum_{x_j \in B_i} k_{ij} \norm{x_i-x_j}^2 = k_i \circ d_i \circ v_i \]
in \ref{new_obj} we will get the simplified objective function
\begin{equation}
\begin{split}
\min_{k \in K_{conv}} \sum_{i=1}^n \bigg( k_i k_i^T \circ d_i d_i^T \circ P + \\ 2\big(\gamma*k_i \circ v_i \circ d_i - k_i \circ p_i \circ d_i\big) \bigg)
\end{split}
\label{simplified}
\end{equation}
Here $p_i$ and $v_i$ are columns of $P$ and $M$ corresponding to $x_i$ respectively. Substituting $k_i = \sum_{t=1}^m \mu_tk_{t,i}$ in \ref{simplified} we will get
\[ \min_{\mu \in \Delta} \mu^T \Bigg( \sum_{t=1}^m \sum_{i=1}^n k_{t,i}k_{t,i}^T \circ d_i d_i^T \circ P \Bigg)^T \mu + z^T \mu \]
which is the objective function $J(\mu)$. Here $[z]_t = \sum_{i=1}^n (2 \gamma v_i \circ d_i - 2 p_i \circ d_i)^T \mathit{k}_{t,i} $ and $\mathit{k}_{t,i} = \Big[ k^t(x_i, x_1), \ldots, k^t(x_i, x_n) \Big]^T $ is the $i^{th}$ column of the $t^{th}$ kernel matrix.

\chapter{Derivation of cost function $\mathcal{J}(\alpha)$ in KFDA}
\label{derivation2}
The cost function is given as
\begin{equation}
\mathcal{J}(f) = \frac{f^TS_B^{\phi}f}{f^TS_W^{\phi}f}
\label{b_jw}
\end{equation}
We have
\begin{equation}
f^Tm_i^{\phi} = \frac{1}{n_i} \sum_{j=1}^n \sum_{k=1}^{n_i} \alpha_j k(x_j, x_k^i) = \alpha^T M_i
\label{b_wmi}
\end{equation}
applying \ref{b_wmi} in the numerator of \ref{b_jw} we get
\begin{equation*}
\begin{aligned}
f^TS_B^{\phi}f &= f^T(m_1^{\phi} - m_2^{\phi})(m_1^{\phi} - m_2^{\phi})^Tf \\
&= (f^T m_1^{\phi}-f^Tm_2^{\phi}) \cdot (f^Tm_1^{\phi}-f^Tm_2^{\phi}) \\
&= (\alpha^T M_1 - \alpha^T M_2) \cdot (\alpha^T M_1 - \alpha^T M_2) \\
&= \alpha^T (M_1 - M_2)(M_1 - M_2)^T \alpha \\
&= \alpha^T M \alpha
\end{aligned}
\end{equation*}
where $M = (M_1-M_2)(M_1-M_2)^T$. Applying $f = \sum_{i=1}^n \alpha_i \phi(x_i)$ in the denominator of \ref{b_jw}
\begin{equation}
f^TS_W^{\phi}f = (\sum_{i=1}^n \alpha_i \phi(x_i))^T \sum_{j=1,2} \sum_{x \in X_j} (\phi(x)-m_j^{\phi})(\phi(x)-m_j^{\phi})^T (\sum_{i=1}^n \alpha_i \phi(x_i))
\label{b_wsw}
\end{equation}
To simplify the notations, define
\[ P_{ij} =  \sum_{x \in X_i} \phi(x_j) \cdot \phi(x) \]
Then
\begin{equation}
\sum_{i=1}^n \sum_{j=1,2} \sum_{x \in X_j} \alpha_i(\phi(x_i) \cdot \phi(x)) = \alpha^TP_1 + \alpha^T P_2
\label{b_alphap}
\end{equation}
Applying \ref{b_alphap} in \ref{b_wsw} we get
\begin{equation*}
\begin{aligned}
f^TS_W^{\phi}f &= (\alpha^T P_1 - \alpha^T M_1) \cdot (\alpha^T P_1 - \alpha^T M_1) + (\alpha^T P_2 - \alpha^T M_2) \cdot (\alpha^T P_2 - \alpha^T M_2) \\
&= \alpha^T(P_1-M_1)(P_1-M_1)^T \alpha + \alpha^T(P_2-M_2)(P_2-M_2)^T \alpha \\
&= \alpha^T K_1(I - \bm{1}_{n_1})K_1^T \alpha + \alpha^T K_2(I - \bm{1}_{n_2})K_2^T \alpha \\
&= \alpha^T \Big(\sum_{i=1,2} K_i(I - \bm{1}_{n_i})K_i^T \Big) \alpha \\
&= \alpha^T N \alpha
\end{aligned}
\end{equation*}
where $K_j = \sum_{i=1}^n \sum_{x \in X_j} k(x_i, x)$, $I$ is the identity matrix, $\bm{1}_{n_j}$ is the matrix with with all entries $\frac{1}{n_j}$ and $N = \sum_{i=1,2} K_i(I - \bm{1}_{n_i})K_i^T$. Thus the cost function becomes
\[\mathcal{J}(f) = \frac{f^TS_B^{\phi}f}{f^TS_W^{\phi}f} = \frac{\alpha^T M \alpha}{\alpha^T N \alpha} \]
%\end{appendices}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% CS624: Analysis of Algorithms
% Copyright 2015 Pejman Ghorbanzade <pejman@ghorbanzade.com>
% Creative Commons Attribution-ShareAlike 4.0 International License
% More info: https://github.com/ghorbanzade/beacon
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section*{Question 7}

Suppose that, instead of sorting an array, we just require that the elements increase on average.
More precisely, we call an $n$-element array $A$, \textbf{k-sorted} if, for all $i = 1, 2, ..., n-k$ Equation \ref{eq71} holds.

\begin{equation}\label{eq71}
\frac{\sum_{j=i}^{i + k - 1} A[j]}{k} \leq \frac{\sum_{j=i+1}^{i + k} A[j]}{k}
\end{equation}

\begin{enumerate}[label=(\alph*)]
\item What does it mean for an array to be 1-sorted?
\item Give a permutation of the numbers $1$, $2$, $3$, ..., $10$, that is $2$-sorted but not sorted.
\item Prove that an $n$-element array is $k$-sorted if and only if $A[i] \leq A[i+k]$ for all $i = 1, 2, ..., n-k$.
\item Give an algorithm that $k$-sorts an $n$-element array in $\mathcal{O}(n \log (n/k))$ time.
\end{enumerate}

\subsection*{Solution}

\begin{enumerate}[label=(\alph*)]
\item Using Equation \ref{eq71} and substituting $k$ with 1 would give Equation \ref{eq72}.
\begin{equation}\label{eq72}
A[i] \leq A[i+1]
\end{equation}
Therefore, any array that is 1-sorted as actually sorted in ascending order.
\item One possible permutation of numbers $1$ to $10$ to result in a 2-sorted array is $1$, $6$, $2$, $7$, $3$, $8$, $4$, $9$, $5$ and $10$.
This is due that for any $1 \leq i < 9$, Equation \ref{eq73} would be valid.
\begin{equation}\label{eq73}
\frac{\sum_{j=i}^{i + 1} A[j]}{2} \leq \frac{\sum_{j=i+1}^{i + 2} A[j]}{2}
\end{equation}
\item Proof is given by induction on length of array $n$.
\begin{enumerate}
\item[] \textit{Base case}: ($n = 2$)\\
When length of array is 2, $k = 1$, Equation \ref{eq71} dictates that $A[i] \leq A[i+1]$ which confirms the hypothesis.
\item[] \textit{Induction Step}\\
We form inductive hypothesis as for any array of length $n$, the array is $k$-sorted if and only if $A[i] \leq A[i+k]$.
Using the inductive hypothesis, we will show that the statement would hold true for arrays of length $n + 1$.

Assuming that the $(n+1)$-element array is $k$-sorted, Equation \ref{eq71} would be available to use.
Subtracting the inductive hypothesis from Equation \ref{eq71} would directly lead to $A[j] \leq A[j+k]$ which is the right-side of the equation.

We should as well show that using $A[i] \leq A[i+1]$ and adding the inductive hypothesis, Equation \ref{eq71} would directly be obtained which shows the array is $k$-sorted.
\end{enumerate}
\item Inspired by the proposed $2$-sorted array of numbers $1$ to $10$, the algorithm to $k$-sort an $n$-element array is given as follows.

We first divide the array into $k$ subarrays.
This clearly is done in constant time and has no effect on runtime of the algorithm.
Then, we sort all $k$ subarrays using Merge Sort.
Since each array has length $\frac{n}{k}$, the runtime for sorting $k$ subarrays would be $\mathcal{O}(k\frac{n}{k}\log(\frac{n}{k}))$.
All that remains is to reconstruct the final array by picking one element from $k$ subarrays, one in turn.
This leading to a run time of $\mathcal{O}(n)$.
Therefore the total runtime of this algorithm would be $\mathcal{O}(n \log(\frac{n}{k}) + n ) = \mathcal{O}(n \log(\frac{n}{k}) )$.
\end{enumerate}


\chapter{\proj Wavelet and Multifractal Analysis}
\label{ch_fractal}
\index{fractal}
% \chapterhead{Programs}
\markright{Wavelet and Multifractal Analysis}

\section{Fractal}
\subsection{Introduction}
The word ``fractal'' was introduced by Mandelbrot
(1977) \cite{frac:mandel83}, and
comes from the Latin word {\em fractus} which means ``to break''.
According to Mandelbrot, a fractal is an object which has
a greater dimension than its topological dimension. A typical
fractal is the Cantor set.

\subsubsection*{Example: the triadic Cantor set}
\label{triadic Cantor set}
The Cantor set is built in the following way: considering a
segment of dimension $L$, we separate it into three equal parts,
and suppress the middle part. There remain two segments of size
${L \over 3}$. Repeating the process on both segments, we get
four segments of size ${3^{-2}L}$. After $n$ iterations, we have
$2^n$ segments of size ${3^{-n}L}$. The Cantor set is obtained
when $n$ tends to infinity. The set has the following properties:
\begin{enumerate}
\item It is self-similar.
\item It has a fine structure, i.e.\ detail on arbitrary small
scales.
\item It is too irregular to be described in traditional
geometrical language, both locally and globally.
\item It is obtained by successive iterations.
\item It has an infinite number of points but its length is null.
\end{enumerate}
These properties define in fact a fractal object.

A real fractal does not exist in the nature, and we always need
to indicate at which scales we are talking about fractality.

\subsection{The Hausdorff and Minkowski measure}
\label{Hausdorff}
\subsubsection*{Measure}
An object dimension describes how an object $F$ fills the space.
A simple manner of measuring the length of
curves, the area of surfaces or the volume of objects is to divide the
space into small boxes (segment in one dimension, surface in 2D, and cubes
in 3D)  of diameter $\delta$ as shown in Figure \ref{fig_mes}
\cite{frac:falconer90}. These boxes
are chosen so that their diameter is not greater than a given size $\delta$,
which corresponds to the measure resolution.
\begin{figure}[htb]
\centerline{
\hbox{
\psfig{figure=fig_frac1.ps,bbllx=5cm,bblly=12cm,bburx=16cm,bbury=18cm,height=6cm,width=12cm,clip=}
}}
\caption{Measuring the ``size'' of curves (Feder 1988).}
\index{Mallat's multiresolution}
\label{fig_mes}
\end{figure}

We consider now the quantity:
\begin{eqnarray}
L_{\delta}^d(F) = \sum diam(B_i)^d
\end{eqnarray}
where $d$ is a real number, $diam(B_i)$ is the diameter of the  box  $i$.
$L_{\delta}^s(F)$ represents an estimation of the size of $F$ at the resolution
$\delta$. Depending on the choice of the boxes, the measure is more or less
correct. Generally, it is easier to manipulate the Minkowski-Bouligand
measure which fixes all the boxes to the same size $\delta$. Using this
measure, the size of $F$ is given by:
\begin{eqnarray}
M^s(F) = \lim_{\delta \rightarrow 0} \sum diam(B_i)^s = \delta^s N_B(\delta)
\end{eqnarray}
where $N_B$ is the number of boxes needed to cover the object $F$.
Then the curves of length $L_\ast$ in Figure~\ref{fig_mes}
can be measured by finding the number $N_B(\delta)$ of line segments
 (respectively squares and cubes for the second and third object)
of length $\delta$ needed to cover the object. The three sizes are:
\begin{eqnarray}
L  &  = M^1(F) = & N_B(\delta) \delta^1 \mathop\rightarrow_{\delta \to 0}
      L_\ast \delta^0  \nonumber  \\
A  & = M^2(F) = & N_B(\delta) \delta^2 \mathop\rightarrow_{\delta \to 0}
      L_\ast \delta^1 \nonumber \\
V  & = M^3(F) = & N_B(\delta) \delta^3  \mathop\rightarrow_{\delta \to 0}
      L_\ast \delta^2
\end{eqnarray}

\subsection{The Hausdorff and Minkowski dimension}

The Hausdorff dimension $d_H$ of the set $F$ is the {\it
critical dimension} for which the measure $H^d(F)$ jumps from infinity
to zero:
\begin{eqnarray}
  H^d(F) =  \left\{ \begin{array}{ll}
                      0,      &  d > d_H, \\
                      \infty, &  d < d_H.
                   \end{array}
           \right.
 \label{eq-ch2-1}
\end{eqnarray}
But $H^{d_H}(F)$ can be finite or infinite. For a simple set
(segment, square, cube),
Hausdorff dimension is equal to the topological dimension (i.e.\ 1, 2,
or 3). This is not true for a more complex set, such as the Cantor set.
Minkowski dimension $d_M$ (and $d_H (F) \le d_M(F)$) is defined
in a similar way using Minkowski measure.

By definition, we have:
\begin{eqnarray}
M^{d_M} = \lim_{\delta \rightarrow 0} \delta^{d_M} N_B(\delta)
\end{eqnarray}
When $\delta \rightarrow 0$, we have $d_M \ln M= d_M\ln{\delta} + \ln{N_N(\delta)}$. If
$M$ is finite, the Minkowski dimension, also called box-counting,
 can be defined by
\begin{eqnarray}
 d_M = \lim_{\delta \rightarrow 0} {\ln{N_B(\delta)} \over - \ln{\delta}}
\end{eqnarray}

In the case of the Cantor set, at iteration $n$, we have $2^n$ segments
of size $3^{-n}$ ($\delta = 3^{-n}$).  When $n  \rightarrow  \infty$, we have
\begin{eqnarray}
d_M (Cantor) = {\ln{2^n} \over {- \ln{3^{-n}}}}  = {\ln{2} \over \ln{3}}
\end{eqnarray}

\section{Multifractality}
The multifractal picture is a refinement and
generalization of the fractal properties
that arise naturally in the case of self-similar distributions.
The singularity spectrum $f(\alpha)$ can be introduced as a quantity
which characterizes the degree of regularity and homogeneity of a
fractal measure.

\subsection{Definition}
\subsubsection*{H\"older exponent}
A multifractal measure describes a non-homogeneous set $A$.
Such a measure is called multifractal if it is everywhere self-similar, i.e.\
if the measure varies locally as a power law, at any point of A.
Denoting $\mu$ a measure, we call the H\"older exponent or singularity exponent
at $x_0$  the limit
\be
\alpha(x_0) =  \lim_{\delta \rightarrow 0} { \ln{\mu(B_{x_0}(\delta))}
            \over \ln{\delta} }
\ee
where $B_{r_0}$ is a box centered at $r_0$ of size $\delta$.
We have:
\be
\mu(B_{x_0}(\delta)) \propto \delta^{\alpha(x_0)}
\ee
The smaller the value $\alpha(x_0)$, the less the measure is regular around
$x_0$. For example, if $\mu$ corresponds to a Dirac distribution centered
at $0$,
then $\alpha(0) = 0$, and if $\mu$ corresponds to a Gaussian distribution, then
$\alpha(0) = -1$.

\subsubsection*{Singularity spectrum}

The singularity spectrum, associated with a measure $\mu$, is the function
which associates with $\alpha$ the fractal dimension of
any point $x_0$ such that
$\alpha(x_0) = \alpha$:
\be
f(\alpha) = d_F(\{ x_0 \in A \mid \alpha(x_0) = \alpha \})
\ee
 The function $f(\alpha)$ is usually (\cite{frac:paladin86})
a single-humped
function with the maximum at $max_{\alpha} f(\alpha) = D$,
where $D$ is the dimension of the support. In the case
of a single fractal, the function $f(\alpha)$ is reduced to
a single point: $f(\alpha) = \alpha = D$.

The singularity spectrum describes statistically the $\alpha$ exponent
distribution on the measure support. For example, if we split the
support into boxes of size $\delta$, then the number of boxes with a
measure varying as $\delta^\alpha$ for a given $\alpha$ is
\be
N_\alpha(\delta) \propto \delta^{-f(\alpha)}
\ee
$f(\alpha)$ describes the histogram of $N_\alpha(\delta)$ when
$\delta$ is small. A measure is homogeneous if its singularity spectrum
is concentrated in a single point. If $f(\alpha)$ is large, the
measure is multifractal.

\subsubsection*{Multifractal quantities}
From a practical point of view one does not determine directly the
spectrum of exponents $[f(\alpha), \alpha]$; it is more convenient to
compute its Legendre transformation $[\tau(q),q]$ given by
\be
\left\{
\begin{array}{l}
\label{mf12}
f(\alpha) = q \cdot \alpha - \tau(q)\\
\alpha    = \frac{d\tau(q)}{dq}
\end{array}
\right.
\ee
In the case of a simple fractal one has $\alpha=f(\alpha)=D$.  In
terms of the Legendre transformation this corresponds to
\be
\tau(q) = D(q-1)
\ee
i.e.\ the behavior of $\tau(q)$ versus $q$ is a straight line with
coefficient given by the fractal dimension.

\subsection{Generalized fractal dimension}

\subsubsection*{Definition}
The generalized fractal dimension, also called Reyni dimension of
order $q$, is given by:
\be
D_q = { \tau(q) \over q -1 }
\ee
$D_0$ is also called capacity dimension, and coincides with the
Hausdorff dimension. Dimensions $D_1$, and $D_2$ are respectively
called information and correlation dimension.

\subsubsection*{Partition function}
The partition function $Z$ is defined by:
\be
Z(q,\delta) = \sum_{i=1}^{N(\delta)} \mu_i^q(\delta)
\ee
where we denote $\mu_i(\delta) = \mu(B_i(\delta))$.  If the measure
$\mu$ is multifractal, $Z$ follows a power law in the limit $\delta
\rightarrow 0$.
\be
Z(q,\delta) \propto \delta^{\tau(q)}
% \label{partition function}
\ee

The box-counting method consists of calculating the partition
function, to derive from $Z$ $\tau(q)$, and to obtain, the multifractal
spectrum by a Legendre transform.

\section{Wavelets and Multifractality}

\subsection{Singularity analysis}

Let $f(x)$ be the input signal, $x_0$ the singularity location,
$\alpha(x_0)$ the H\"older exponent at the singularity point $x_0$ and
$n$ the degree of Taylor development such that
$n\leq\alpha(x_0)<n+1$. We have
\be
f(x) & = & f(x_0) + (x-x_0)\,f^{(1)}(x_0) + ... +  \\ \nonumber
     &   & \frac{(x-x_0)^n}{n!}\,f^{(n)}(x_0) + C\,|x-x_0|^{\alpha(x_0)}
\ee

Letting $\psi$ be the wavelet with $n_\psi>n$ vanishing moments, then we have
for the wavelet transform of $f(x)$ at $x_0$ when the scale goes to 0
($\psi$ is orthogonal to polynomials up to order $n$):

\be
\lim_{scale s \rightarrow 0^+} T_\psi[f](x_0,s) \sim {a^{\alpha(x_0)}}
% \label{Mallat Hwang}
\ee

One can prove that if $f$ is $C^\infty$, then we have

\be
\lim_{scale s \rightarrow 0^+} T_\psi[f](x_0,s) \sim {a^{n_\psi}}
\ee

Thus, we have
\begin{eqnarray}
\left\{  \begin{array}{ll}
 	T_\psi[f] \sim s^{n_\psi} & \mbox{ where the signal f is regular} \\
	T_\psi[f] \sim s^{\alpha} (>>s^{n_\psi}) & \mbox{ around a singular zone}
	\end{array}
	\right.
\end{eqnarray}

For a fixed scale s, $T_\psi[f](.,s)$ will be greater when the signal is
singular. This local maximum is organized in maxima lines (function of s)
which
converges, when s goes to 0, to a singularity of the signal.
Mallat and Hwang (1992) demonstrate that to recover the H\"older exponent
$\alpha(x_0)$ at $x_0$, one need only study the  wavelet transform along these
lines of maxima which converge (when the scale goes to 0) towards the singularity
point $x_0$.

Along this maxima line l we have
\be
T_\psi[f](b,s) \sim a^{\alpha(x_0)}, \,\,(b,a)\in l, s\rightarrow 0^+
\ee
Figure~\ref{fig_2} displays the function $f(x)=K(x-x_0)^{0.4}$ with a singular
point at $x_0$. The H\"older exponent at $x_0$ is equal to 0.4.
In Figure~\ref{fig_3}, we display the wavelet transform of $f(x)$ with a wavelet
$\psi$ which is the first derivative of a Gaussian. In Figure~\ref{fig_4}, we
display $log_2 {|T_\psi[f](x,s)|}$ as a  function of $\log_2(s)$.

\begin{figure}[htb]
\centerline{
\vbox{
\psfig{figure=singul.ps,height=5cm,width=8cm}
}}
\caption{Function $f(x)=K(x-x_0)^{0.4}$ \cite{frac:arneodo90}.}
\label{fig_2}
\end{figure}

\begin{figure}[htb]
\centerline{
\vbox{
\psfig{figure=WT.ps,height=5cm,width=8cm}
}}
\caption{
Wavelet transform of $f(x)$ with a wavelet $\psi$ which is the first
derivative of
a Gaussian. The small scales are at the top. The maxima line converges to the
singularity point at $x_0$.}
\label{fig_3}
\end{figure}

\begin{figure}[htb]
\centerline{
\vbox{
\psfig{figure=holder_expo.ps,height=5cm,width=8cm}
}}
\caption{
Estimation of the H\"older exponent $\alpha(x_0)$ at
$x_0$ by computing the slope of $\log_2 {|T_\psi[f](x,s)|}$ versus
$\log_2(s)$ along a maxima line which converges to $x_0$.}
\index{Mallat's multiresolution}
\label{fig_4}
\end{figure}

When we compute the slope of the curve $log_2 {|T_\psi[f](x,s)|}$
versus $\log_2(s)$ along a maxima line which converges at $x_0$,
we obtain an estimation of the H\"older exponent (in this case
$\alpha(x_0) \approx 0.4$
which corresponds to the theoretical value).

\subsection{Wavelet transform of multifractal signals }

The estimation of the H\"older exponent by this method
 becomes inaccurate
in the case of multifractal signals \cite{frac:arneodo90}.
We need to use a more global
method. One can define the wavelet-based partition function by

\be
Z(q,s) = \sum_{b_i} |T_\psi[\mu](b_i,s)|^q
\ee
where ${(b_i,s)}_i$ are all local maxima at scale $s$.

Let $\tau(q)$ be the scaling exponent. We can prove that we have
\be
Z(q,s) \sim a^{\tau(q)}
\ee

We can then calculate the singularity spectrum $D(\alpha)$ by its Legendre
transformation
\be
D(\alpha) = \underset{q}{\min} (q\alpha-\tau(q))
\ee

This method is called the {\em Wavelet Transform Modulus Maxima} (WTMM).

\subsection{Numerical applications of WWTM method}

The calculation of the singularity spectrum of signal f proceeds as follows:
\begin{list}{--}{\itemindent=-5mm \itemsep=-1mm}
\item compute the wavelet transform and the modulus maxima $T_\psi[f]$ for
all $(s,q)$. We chain all maxima across scale
lines of maxima
\item compute the partition
function $Z(q,s) = \sum_{b_i} |T_\psi[f](b_i,s)|^q$
\item compute $\tau(q)$ with
$\log_2{Z(q,s)}\,\approx\,\tau(q)\log_2(s)+C(q)$
\item compute $D(\alpha) = \underset{q}{\min}  (q\alpha-\tau(q))$
\end{list}

\subsubsection*{The triadic Cantor set}

Definition : The measure associated with the triadic Cantor set is defined by
$f(x) = \int_0^x d\mu$ where $\mu$ is the uniform measure lying on the triadic
Cantor set described in section \ref{triadic Cantor set}. To compute $f(x)$, we used the next recursive
function called the Devil's Staircase function
(which looks like a staircase whose steps are
uncountable and infinitely small, see Figure~\ref{fig_5}).

\begin{eqnarray}
f(x)=	\left\{   \begin{array}{ll}
	p_1f(3x)       & \mbox{ if $x \in [0,\frac{1}{3}]$}\\
	p_1            & \mbox{ if $x \in [\frac{1}{3}, \frac{2}{3}]$ }\\
	p_1+p_2f(3x-2) & \mbox{ if $x \in [\frac{2}{3}, 1]$}
	\end{array}
	\right.
% \label{recursive Cantor func}
\end{eqnarray}

This is a continuous function that increases  from 0 to 1
on [0,1]. The recursive
construction of $f(x)$ implies that $f(x)$ is self-similar.

\begin{figure}[htb]
\centerline{
\hbox{
	\psfig{figure=DSC3.ps,height=5cm,width=8cm}
    \psfig{figure=DSC3ParFun.ps,height=5cm,width=8cm}
}}
\caption{Classical Devil's Staircase function (associated with
triadic Cantor set) with $p_1=0.5$ and $p_2=0.5$
(left) and partition function $Z(q,s)$ for several
values of $q$ (right). The
wavelet transform is calculated with $\psi$ equal to the
first derivative of a Gaussian.}
\label{fig_5}
\end{figure}

\begin{figure}[htb]
\centerline{
\hbox{
	\psfig{figure=DSC3ScaExp.ps,height=5cm,width=8cm}
    \psfig{figure=DSC3SinSpe.ps,height=5cm,width=8cm}
}}
\caption{Scaling Exponent estimation $\tau(q)$ (left) and Singularity
Spectrum $D(\alpha)$ (right). The Scaling Exponent curve  corresponds
 to the theoretical curve
$\tau(q)=(q-1)\log_2(2)/\log_2(3)$. Points of the Singularity Spectrum
where $q \neq \infty(\max\,q)$ and  $q \neq-\infty(\min\,q)$ are reduced
to a single point
($\alpha=\log_2(2)/\log(3), D(\alpha)=\log _2(2)/\log _2(3))$. This point
corresponds to
the Hausdorff dimension of the triadic Cantor Set.}
\label{fig_6}
\end{figure}

\subsubsection*{The generalized Devil's Staircase with $p_1=0.4$ and $p_2=0.6$}

One can prove \cite{frac:arneodo90} that the theoretical singularity spectrum $D(\alpha)$
of the generalized Devil's Staircase function $f(x)=\int_0^x{d\mu}$ verifies
the following:
\begin{itemize}
\item The singular spectrum is a convex curve with a maximum value
$\alpha_{max}$ which corresponds to the fractal dimension of the
support of the measure ($\mu$).
\item The theoretical support of $D(\alpha)$ is reduced at the interval
$[\alpha_{min},\alpha_{max}]$:

\begin{eqnarray}
\left\{ \begin{array}{l}
 \alpha_{min}=\min({\frac{\ln p_1}{\ln (1/3)}, \frac{\ln p_2}{\ln (1/3)}})\\
 \alpha_{min}=\min({\frac{\ln p_1}{\ln (1/3)}, \frac{\ln p_2}{\ln (1/3)}})
	\end{array}
	\right.
\end{eqnarray}
\end{itemize}

Figure~\ref{fig_7} displays the generalized Devil's Staircase and its
partition
function $Z(q,s)$. In Figure~\ref{fig_8}, we can see the Scaling exponent and
the Singularity spectrum. This one is in perfect ``accord'' with
the theoretical
values: bell curve, $D(\alpha)_{max}=\log_2(2)/\log_2(3)$,
$\alpha_{min}\approx 0.47$ and $\alpha_{max}\approx 0.83$

\begin{figure}[htb]
\centerline{
\hbox{
    \psfig{figure=DSC3b.ps,height=5cm,width=8cm}
    \psfig{figure=DSC3bParFun.ps,height=5cm,width=8cm}
}}
\caption{Devil's Staircase function with $p_1=0.4$ and $p_2=0.6$
(left) and partition function $Z(q,s)$ for several values of $q$ (right).
The wavelet transform is calculated
with $\psi$ equal to the first derivative of a Gaussian.}
\label{fig_7}
\end{figure}

\begin{figure}[htb]
\centerline{
\hbox{
	\psfig{figure=DSC3bScaExp.ps,height=5cm,width=8cm}
    \psfig{figure=DSC3bSinSpe.ps,height=5cm,width=8cm}
}}
\caption{Scaling Exponent estimation $\tau(q)$ (left) and Singularity
Spectrum $D(\alpha)$ (right). The theoretical maximum value of $D(\alpha)$ is
obtained for $\alpha_{max D}=\log_2(2)/\log_2(3)$ and
$D(\alpha_{max D})=\log_2(2)/\log_2(3)$.}
\label{fig_8}
\end{figure}

\newpage
\section{Program}
\subsection{Devil's Staircase: mf1d\_create\_ds}
\index{mf1d\_create\_ds}
Program {\em mf1d\_create\_ds} creates a Devil's Staircase function lying on the
Cantor sets measure. If the ``-b'' option is set, a border is added, which
is equal to 0 on the left side, and to 1 on the right side. The size of the
border is ${N\over 3}$ where $N$ is the number of points. Hence the output
signal size becomes $N + {2N\over 3} = {5N\over 3}$. It is recommended to
choose a number of points which can be divided by 3.

{\bf
\begin{center}
USAGE:  mf1d\_create\_ds options image out
\end{center}}
where options are
\begin{itemize}
\item {\bf [-n number\_of\_points]} \\
Number of points of Devil's Staircase. Default is 243.
\item {\bf [-p Prob1]} \\
Coefficient $p_1$ of the first interval of the Devil's Staircase.
$p_2$ is equal to $1 - p_1$. \\
Default value is $0.5$.
\item {\bf [-b]} \\
Add a border. Default is no.
\item {\bf [-v]} \\
Verbose. Default is False.
\end{itemize}
\subsubsection*{Examples:}
\begin{itemize}
\item mf1d\_create\_ds NameOut \\
Create a Devil's Staircase  with all default values (243 points,
probabilities equal to $0.5$, ...).
\item mf1d\_create\_ds -p 0.4 NameOut \\
Ditto, but probabilities $p_1$ and $p_2$ are respectively equal to $0.4$ and
$0.6$.
\end{itemize}

\subsection{Chain maxima wavelet coefficients: mf1d\_chain}
\index{mf1d\_chain}

Program {\em mf1d\_chain} calculates the wavelet transform of a signal,
finds its maxima wavelet coefficients, and chains them. Finally, chains which
do not respect some criteria are deleted. Two files are created: the first
contains the maxima, and the second the skeleton.
{\bf
\begin{center}
 USAGE:  mf1d\_chain options  signal\_in
\end{center}}
where options are
\begin{itemize}
\item {\bf [-t type\_of\_transform]} \\
0: Gaussian derivative wavelet transform \\
1: Mexican hat wavelet transform  \\
Default is 0.
\item {\bf [-r min\_length\_chain]} \\
Remove chains with length $<$ min\_length. Default is 0.
\item {\bf [-s  dynamique(\%)]} \\
Remove points of the skeleton map
if level $<$ dynamique $*$ max(current scale). \\
Default is 0.
% \item {\bf [-c]} \\
% Draw chains of max. \\
% Default is false.
\item {\bf [-v]} \\
Verbose
\end{itemize}
Wavelet transform file name is {\bf WTMM\_signal\_in.fits}. \\
Skeleton file name is {\bf WTMMskel\_signal\_in.fits}.
\subsubsection*{Example:}
\begin{itemize}
\item mf1d\_chain Data01.fits \\
 Create the files{\bf WTMM\_Data01.fits} and {\bf WTMMskel\_Data01.fits}.
\end{itemize}

\subsection{Partition function: mf1d\_repart}
\index{mf1d\_repart}
Program {\em mf1d\_repart} computes the partition function. It needs the two
output files of the {\em mf1d\_chain} program (i.e.\ WTMMxxx.fits
and WTMMskelxxx.fits), and creates three files:
\begin{itemize}
\item {\bf q\_FileNameOut.fits}: which contains the one-dimensional array $Q$
of the $q$ values for which the
partition function is calculated.
\item {\bf s\_FileNameOut.fits}: which contains the one-dimensional array $S$
of the $\log_2 s$ values for which the
partition function is calculated.
\item {\bf Z\_FileNameOut.fits}: which contains the partition functions. It
is a two dimensional array. $Z(i,j)$ is the partition function value
 for $q = Q(i)$ and $s = S(j)$ ($i$ indexing the x-axis, and $j$ the y-axis).
\end{itemize}
{\bf
\begin{center}
USAGE: mf1d\_repart options MaxFileName MaxSupFileName FileNameOut
\end{center}}
where options are
\begin{itemize}
\item {\bf [-a q\_minq]} \\
Value min of vector $q$. Default is -10.00.
\item {\bf [-b q\_max]} \\
Value max of vector $q$. Default is +10.00.
\item {\bf [-c number\_of\_q]} \\
Number of values of vector $q$. Default is 20.
\end{itemize}
$Z(q,s)$ file name is {\bf Z\_FileNameOut.fits} \\
$q$ file name is {\bf q\_FileNameOut.fits} \\
$s$ file name is {\bf s\_FileNameOut.fits}. \\
\subsubsection*{Example:}
\begin{itemize}
\item mf1d\_chain Data01.fits \\
      mf1d\_repart  WTMM\_Data01.fits WTMMskel\_Data01.fits Repart.fits  \\
 Create the files {\bf Z\_Data01.fits}, {\bf q\_Data01.fits}, and
 {\bf s\_Data0.fits}.
\end{itemize}

\subsection{Multifractal analysis: mf\_analyse}
\index{mf\_analyse}
Program {\em mf\_analyse} realizes the multifractal analysis. It
 computes the scaling
exponent estimation $\tau(q)$ and the singularity spectrum $\alpha(q)$.
It needs the three files created by {\em mf1d\_repart}. Four files are
created:
\begin{itemize}
\item {\bf X\_Tau.fits}: which contains the x-coordinates of the
$\tau(q)$ curve.
\item {\bf Y\_Tau.fits}: which contains the y-coordinates of the
$\tau(q)$ curve.
\item {\bf X\_FracSpectrum.fits}: which contains the x-coordinates of the
singularity spectrum curve (i.e. $\alpha$ values).
\item {\bf Y\_FracSpectrum.fits}: which contains y-coordinates of the
singularity spectrum curve (i.e. $D(\alpha)$ values).
\end{itemize}

{\bf \begin{center}
 USAGE:  mf\_analyse options FileNameIn
\end{center}}
where options are
\begin{itemize}
\item {\bf [-d $\alpha_{min}$]} \\
Value min of vector $\alpha$. Default is 0.00.
\item {\bf [-e $\alpha_{max}$]} \\
Value max of vector $\alpha$. Default is 1.00.
\item {\bf [-f number\_of\_$\alpha$]} \\
Number of values of vector $\alpha$. Default is 20.
\item {\bf [-m ind\_scale\_min]} \\
ind scale min for computing $\tau$. Default is 30.
\item {\bf [-M ind\_scale\_max]} \\
ind scale max forn computing $\tau$. Default is scale\_max-2.
\end{itemize}
$Z(q,s)$ are read in {\bf Z\_FileNameIn.fits} file \\
$q$ are read in {\bf q\_FileNameIn.fits} file\\
$s$ are read in {\bf s\_FileNameIn.fits} file .
\subsubsection*{Example:}
\begin{itemize}
\item mf1d\_chain Data01.fits \\
      mf1d\_repart WTMM\_Data01.fits WTMMskel\_Data01.fits Repart  \\
      mf\_analyse  Repart \\
  Create the files  {\bf X\_Tau.fits},{\bf Y\_Tau.fits},{\bf X\_FracSpectrum.fits},{\bf Y\_FracSpectrum.fits},
  and {\bf Y\_TauTheo.fits}.
\end{itemize}

\subsection{Examples}

\subsubsection*{Fractal analysis of Devil's Staircase function}
Create Devil's Staircase function with $p_1=.6$ and $p_2=.4$.
\begin{verbatim}
mf1d_create_ds  -p .4 -b DSC3
\end{verbatim}
Compute the wavelet transform WTMM\_DSC3.fits, the skeleton WTMMskel\_DSC3.fits
(default transform: $\psi$ is the first derivative of a Gaussian).
\begin{verbatim}
mf1d_chain DSC3
\end{verbatim}
Compute the partition function $Z(q,s)$ for $q\in[-10.,10.]$ with 60 values,
write the $Z(q,s)$ to file Z\_DSC3.fits, the $s$(scale) file to s\_DSC3.fits and the
$q$ file to q\_DSC3.fits.
\begin{verbatim}
mf1d_repart -c 60 WTMM_DSC3 WTMMskel_DSC3 DSC3
\end{verbatim}
Compute the scaling exponent $\tau(q)$, write it in X\_Tau.fits and Y\_Tau.fits,
compute the singularity spectrum $\alpha(q)$, write it in X\_FracSpectrum.fits
and Y\_FracSpectrum.fits (100 values are used for $\alpha$).
\begin{verbatim}
mf_analyse -f 100 DSC3
\end{verbatim}

\subsubsection*{Fractal analysis}

Compute the wavelet transform WTMM\_DSC3.fits, the skeleton WTMMskel\_DSC3.fits
(use the Mexican hat for $\psi$)
\begin{verbatim}
mf1d_chain -t 1 signal
\end{verbatim}
Compute the partition function $Z(q,s)$ for $q\in[-5.,5.]$ with 100 values,
write the $Z(q,s)$ in file Z\_signal.fits, the $s$(scale) file in s\_signal and the
$q$ file in q\_signal.
\begin{verbatim}
mf1d_repart -a -5. -b 5. -c 100 WTMM_signal WTMMskel_signal sigout
\end{verbatim}
Compute the scaling exponent $\tau(q)$, write it in X\_Tau.fits and Y\_Tau.fits,
compute the singularity spectrum $\alpha(q)$, write it in X\_FracSpectrum.fits
and Y\_FracSpectrum.fits ($\alpha \in [0,2]$, 100 values are used).
\begin{verbatim}
mf_analyse -d 0. -e 2. -f 100 sigout
\end{verbatim}


\documentclass[letterpaper]{article}

\usepackage{fullpage}
\usepackage{nopageno}
\usepackage{amsmath}
\usepackage{amssymb}
\allowdisplaybreaks

\newcommand{\abs}[1]{\left\lvert #1 \right\rvert}

\begin{document}
\title{Notes}
\date{September 8, 2014}
\maketitle
\section*{excercise}
if $a_n\le b_n\forall n$ then $\lim_{n\to\infty}a_n\le\lim_{n\to\infty}b_n$
\begin{align*}
  a_n&\le b_n\\
  0&\le b_n-a_n\\
  \lim_{n\to\infty}0&\le\lim_{n\to\infty}(b_n-a_n)\\
  0&\le \lim_{n\to\infty}b_n-\lim_{n\to\infty}a_n\\
  \lim_{n\to\infty}a_n&\le\lim_{n\to\infty}b_n
\end{align*}

\section*{monotone sequences}
definition:

a sequence is increasing iff $a_{n+1}\ge a_n$ for every $n\in\mathbb{N}$ strictly increasing ...decreasing if $a_{n+1}\ge a_n$ for every $n\in\mathbb{N}$, strictly decreasing.... monotone if is it any of these types

\section*{theorem 2.6.1}
an increasing sequence that is bounded above is convergent. a decreasing sequence that is bounded below is convergent.
\subsection*{proof}
we are given $\{a_n\}_{n=1}^\infty$, increasing $a_n\le a_n+1\forall n\in\mathbb{N}$ and bounded above. since it is bdd above it has a supremum $L$. we prove that $\lim_{n\to\infty}a_n=L$.
\begin{align*}
  L&=\sup\{a_n:n\in\mathbb{N}\}
\end{align*}
$L$ is the least upper bound. if $M<L$ M cannot be an upper bound.

Need: $\lim_{n\to\infty}a_n=L$ ie $\forall\epsilon>0,\exists N\in\mathbb{N}$ such that $|a_n-L|<\epsilon\forall n\ge N$.

$\forall a_n,a_n\le L$

$\forall \epsilon>0\exists N$
$L-\epsilon<a_N\le L\Rightarrow|a_N-L|<\epsilon$. If $n\ge N, L\ge a_n\ge a_N>L-\epsilon$ because $\{a_n\}$ is increasing. hence $a_n-L|<\epsilon, \forall n\ge N\to\lim_{n\to\infty}=L\Box$
\section*{example}
let $0<x_1<1$ and define sequence $x_n$ recursively by $x_{n+1}=1-\sqrt{1-x_n}$. prove that $\{x_n\}$ has a limit and find its value

need to show it's monotone and bounded

if $0<x_n<1$ then $0<1-\sqrt{1-x_n}<1$

\begin{align*}
  \sqrt{1-x_n}&=1-x_{n+1}\\
  1-x_n&=(1-x_{n+1})^2\\
  1-x_n&=(1-x_{n+1})^2\le 1-x_{n+1}\\
  x_{n+1}&\le x_n\\
\end{align*}

sequence is bounded and decreasing so it has a limit

\begin{align*}
  x_{n+1}=1-\sqrt{-x_{n}}\\
  n\to\infty\\
  L=1-\sqrt{1-L}\\
  \sqrt{1-L}=1-L\\
  1-L=1-2L+L^2\\
  L^2-L=0\\
  L=0\text{ or }1
\end{align*}
Limit is 0 since sequence is decreasing

\section*{example}
let $7x_{n+1}=x_n^3+6,n\ge1$. study whether the limit eists and find it's value if it doest for $x_1=\frac{1}{2},\frac{3}{2},\frac{5}{2}$

\begin{align*}
  x_{n+1}=\frac{x_n^3}{7}+\frac{6}{7}
\end{align*}
if $0<x_n^3\le1$ then $0<x_{n+1}\le1$
\begin{align*}
  \frac{x_n^3+6}{7}\le\ge x_n\\
  x_n^3-7x_n+6=0=(x_n-1)(x_n^2+x_n-6)=(x_n-1)(x_n-2)(x_n-3)
\end{align*}
look at graph, between -3 and 1 equation is positive so $\frac{x_n^3+6}{7}\ge x_n$. between 1 and 2 $\frac{x_n^3+6}{7}\le x_n$ and above 2 $\frac{x_n^3+6}{7}\ge x_n$

for $x_1=1/2$ the sequence is increasing and bounded above by 1, for $x_1=3/2$ the sequence is decreasing. for $x_1=5/2$ the sequence is increaasing
\begin{align*}
  x_1=1/2\\
  7L=L^3+6\\
  0=L^3+7L+6\\
  L=-3,1,2\\
  L=1\text{ because it is bounded above by 1 and is increasing}
\end{align*}

for $x_1=3/2$. possibilities are 1,2,-3, it's between 1 and 2 so it will be 1. assume $1<x_n<2$
\begin{align*}
  1<x_n^3<8\\
  7<x_n^3+6<14\\
  1<\frac{x_n^3+6}{7}<2
\end{align*}
so it's bounded therefore the limit exists and is one

for $x_1=5/2$ it is increasing and greater than 3 possible limits
\end{document}


\section{Task D}
\label{sec:task-d}

The \textit{constitutive} component of the tangent matrix relating node \(a\) to
node \(b\) is
\begin{equation} \tag{9.35}
  \label{eq:tang-mat-const}
  \left[ \uubar{\bm{K}}_{c,ab} \right]_{ij} =
  \int_{v^{(e)}} \sum_{k,l=1}^{3} \frac{\partial N_{a}}{\partial x_{k}}
  c_{ijkl} \frac{\partial N_{b}}{\partial x_{l}} \, dv, \quad i,j = 1,2,3
\end{equation}
The 4th order elasticity tensor in \textit{spatial} configuration can be computed
by pushing forward its counterpart in the \textit{material} configuration
\(\partial \utilde{S} / \partial \utilde{E}\):
\begin{equation}
  \label{eq:elast-tensor-Euler}
  \uutilde{c} = J^{-1} \utilde{F} \overline{\otimes} \utilde{F} \cdot
  \frac{\partial \utilde{S}}{\partial \utilde{E}} \cdot
    \utilde{F}^{T} \overline{\otimes} \utilde{F}^{T}
\end{equation}

For a CST element of thickness \(t\) equation \eqref{eq:tang-mat-const} can be
rewritten as
\begin{equation}
  \label{eq:tang-mat-const-CST}
  \uubar{\bm{K}}_{c,ab} \approx \sum_{i=1}^{\text{nip}} W_{i} \ubar{\nabla} N_{a}
  \cdot \uubar{\bm{D}} \cdot \ubar{\nabla} N_{b} t
  \det \left( \frac{\partial \ubar{\bm{x}}}{\partial \ubar{\bm{\xi}}} \right),
\end{equation}
where \(\uutilde{c}\) was rearranged into matrix \(\uubar{\bm{D}}\) that
facilitates computations in the matrix form for 2D case:
\begin{equation}
  \label{eq:D-matrix}
  \uubar{\bm{D}} = \left[
    \begin{array}{c c c}
      c_{1111} & c_{1122} & c_{1112} \\
               & c_{2222} & c_{2212} \\
      \text{sym} &  & c_{1212}
	\end{array} \right]
\end{equation}

The components of the \textit{initial stress} matrix are as follows:
\begin{equation} \tag{9.44c}
  \label{eq:tang-mat-init}
  \left[ \uubar{\bm{K}}_{\sigma,ab} \right]_{ij} =
  \int_{v^{(e)}} \sum_{k,l=1}^{3} \frac{\partial N_{a}}{\partial x_{k}}
  \sigma_{kl} \frac{\partial N_{b}}{\partial x_{l}} \delta_{ij} \, dv,
  \quad i,j = 1,2,3
\end{equation}
Then for a CST element this becomes
\begin{equation}
  \label{eq:tang-mat-init-CST}
  \uubar{\bm{K}}_{\sigma,ab} \approx \sum_{i=1}^{\text{nip}} W_{i} \ubar{\nabla} N_{a}
  \cdot \utilde{\sigma} \cdot \ubar{\nabla} N_{b} \utilde{I} t
  \det \left( \frac{\partial \ubar{\bm{x}}}{\partial \ubar{\bm{\xi}}} \right)
\end{equation}

The total tangent matrix is then
\begin{equation}
  \label{eq:tang-mat}
  \uubar{\bm{K}}_{ab} = \uubar{\bm{K}}_{c,ab} + \uubar{\bm{K}}_{\sigma,ab}
\end{equation}

The Matlab implementation of this task can be found in
\texttt{get\_tangent\_matrices.m} (see section \ref{app:matlab-code}).

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "../main"
%%% End:


\subsection{Step 1}
Let's start by rewriting our ADI equation for the Step 1 (given in class 13 slide 13), for an staggered mesh for the vertical velocity $v$:
\begin{align*}
-d_1v_{i-1,j+1/2}^{n+1/2}+(1+2d_1)v_{i,j+1/2}^{n+1/2}-d_1v_{i+1,j+1/2}^{n+1/2}=d_2v_{i,j+3/2}^{n}+(1-2d_2)v_{i,j+1/2}^{n}+d_2v_{i,j-1/2}^{n},
\end{align*}
which has the form
\begin{align*}
av_{i-1,j+1/2}^{n}+bv_{i,j+1/2}^{n}+cv_{i+1,j+1/2}^{n}=d,
\end{align*}
a tridiagonal system. The staggered mesh for $v$ gives us a matrix $v$ that is $(M+2)\times(N+1)$. We only need to solve the previous equation for the interior with two $for$ loops in \textsl{Matlab},
\begin{verbatim}
for j=2:N
for i=2:M+1
		...
	end
end
\end{verbatim}.
We will now include the boundary conditions for each case.
\subsubsection*{Case $j=2$ and $i=2$}
In this case the general equation becomes
\begin{align*}
-d_1v_{1,5/2}^{n+1/2}+(1+2d_1)v_{2,5/2}^{n+1/2}-d_1v_{3,5/2}^{n+1/2}=d_2v_{2,7/2}^{n}+(1-2d_2)v_{2,5/2}^{n}+d_2v_{2,3/2}^{n},
\end{align*}
The value $v_{1,5/2}^{n+1/2}=-v_{2,5/2}^{n+1/2}$ by the boundary conditions and $v_{2,3/2}^{n}=0$ since it corresponds to the bottom wall. Thus the equation yields
\begin{align*}
(1+3d_1)v_{2,5/2}^{n+1/2}-d_1v_{3,5/2}^{n+1/2}=d_2v_{2,7/2}^{n}+(1-2d_2)v_{2,5/2}^{n},
\end{align*}
giving $a=0$ and $b=1+3d_1$ for this case, $d$ is given by the right hand side.
\subsubsection*{Case $j=2$ and $i\in [3,M]$}
In this case the general equation becomes
\begin{align*}
-d_1v_{i-1,5/2}^{n+1/2}+(1+2d_1)v_{i,5/2}^{n+1/2}-d_1v_{i+1,5/2}^{n+1/2}=d_2v_{i,7/2}^{n}+(1-2d_2)v_{i,5/2}^{n}+d_2v_{i,3/2}^{n},
\end{align*}
Like in the previous case, the value $v_{i,3/2}^{n}=0$ and $d$ is the right hand side of the previous equation.

\subsubsection*{Case $j=2$ and $i=M+1$}
In this case the general equation becomes
\begin{align*}
-d_1v_{M,5/2}^{n+1/2}+(1+2d_1)v_{M+1,5/2}^{n+1/2}-d_1v_{M+2,5/2}^{n+1/2}=d_2v_{M+1,7/2}^{n}+(1-2d_2)v_{M+1,5/2}^{n}+d_2v_{M+1,3/2}^{n},
\end{align*}
Like in the previous case, the value $v_{M+1,3/2}^{n}=0$. The value of $v_{M+2,5/2}^{n+1/2}=-v_{M+1,5/2}^{n+1/2}$ and the previous equation yields
\begin{align*}
-d_1v_{M,5/2}^{n+1/2}+(1+3d_1)v_{M+1,5/2}^{n+1/2}=d_2v_{M+1,7/2}^{n}+(1-2d_2)v_{M+1,5/2}^{n}.
\end{align*}
Thus, $c=0$, $b=1+3d_1$ and $d$ is the right hand side of the previous equation.

\subsubsection*{Case $j\in[3,N-1]$ and $i=2$}
In this case the general equation becomes
\begin{align*}
-d_1v_{1,j+1/2}^{n+1/2}+(1+2d_1)v_{2,j+1/2}^{n+1/2}-d_1v_{3,j+1/2}^{n+1/2}=d_2v_{2,j+3/2}^{n}+(1-2d_2)v_{2,j+1/2}^{n}+d_2v_{2,j-1/2}^{n}.
\end{align*}
The value $v_{1,j+1/2}^{n+1/2}=-v_{2,j+1/2}^{n+1/2}$ by the boundary condition. Thus the equation yields
\begin{align*}
(1+3d_1)v_{2,j+1/2}^{n+1/2}-d_1v_{3,j+1/2}^{n+1/2}=d_2v_{2,j+3/2}^{n}+(1-2d_2)v_{2,j+1/2}^{n}+d_2v_{2,j-1/2}^{n},
\end{align*}
giving $a=0$, $b=1+3d_1$ and $d$ given by the right hand side.

\subsubsection*{Case $j\in[3,N-1]$ and $i\in[3,M]$}
In this case the general equation does not include any boundary conditions, therefore it is unaltered.

\subsubsection*{Case $j\in[3,N-1]$ and $i=M+1$}
In this case the general equation becomes
\begin{align*}
-d_1v_{M,j+1/2}^{n+1/2}+(1+2d_1)v_{M+1,j+1/2}^{n+1/2}-d_1v_{M+2,j+1/2}^{n+1/2}=d_2v_{M+1,j+3/2}^{n}&+(1-2d_2)v_{M+1,j+1/2}^{n}\\&+d_2v_{M+1,j-1/2}^{n}.
\end{align*}
The value $v_{M+2,j+1/2}^{n+1/2}=-v_{M+1,j+1/2}^{n+1/2}$ by the boundary condition. Thus the equation yields
\begin{align*}
-d_1v_{M,j+1/2}^{n+1/2}+(1+3d_1)v_{M+1,j+1/2}^{n+1/2}=d_2v_{M+1,j+3/2}^{n}+(1-2d_2)v_{M+1,j+1/2}^{n}+d_2v_{M+1,j-1/2}^{n},
\end{align*}
giving $c=0$, $b=1+3d_1$ and $d$ given by the right hand side.

\subsubsection*{Case $j=N$ and $i=2$}
In this case the general equation becomes
\begin{align*}
-d_1v_{1,N+1/2}^{n+1/2}+(1+2d_1)v_{2,N+1/2}^{n+1/2}-d_1v_{3,N+1/2}^{n+1/2}=d_2v_{2,N+3/2}^{n}+(1-2d_2)v_{2,N+1/2}^{n}+d_2v_{2,N-1/2}^{n},
\end{align*}
The value $v_{1,N+1/2}^{n+1/2}=-v_{2,N+1/2}^{n+1/2}$ by the boundary conditions and $v_{2,N+3/2}^{n}=0$ since it corresponds to the top wall. Thus the equation yields
\begin{align*}
(1+3d_1)v_{2,N+1/2}^{n+1/2}-d_1v_{3,N+1/2}^{n+1/2}=(1-2d_2)v_{2,N+1/2}^{n}+d_2v_{2,N-1/2}^{n},
\end{align*}
giving $a=0$ and $b=1+3d_1$ for this case, $d$ is given by the right hand side.

\subsubsection*{Case $j=N$ and $i\in [3,M]$}
In this case the general equation becomes
\begin{align*}
-d_1v_{i-1,N+1/2}^{n+1/2}+(1+2d_1)v_{i,N+1/2}^{n+1/2}-d_1v_{i+1,N+1/2}^{n+1/2}=d_2v_{i,N+3/2}^{n}+(1-2d_2)v_{i,N+1/2}^{n}+d_2v_{i,N-1/2}^{n},
\end{align*}
Like in the previous case, the value $v_{i,N+3/2}^{n}=0$ and $d$ is the right hand side of the previous equation.

\subsubsection*{Case $j=N$ and $i=M+1$}
In this case the general equation becomes
\begin{align*}
-d_1v_{M,N+1/2}^{n+1/2}+(1+2d_1)v_{M+1,N+1/2}^{n+1/2}-d_1v_{M+2,N+1/2}^{n+1/2}=d_2v_{M+1,N+3/2}^{n}&+(1-2d_2)v_{M+1,N+1/2}^{n}\\&+d_2v_{M+1,N-1/2}^{n}.
\end{align*}
By the boundary conditions, $v_{M+2,N+1/2}^{n+1/2}=-v_{M+1,N+1/2}^{n+1/2}$ and $v_{M+1,N+3/2}^{n}=0$ and the previous equation yields
\begin{align*}
-d_1v_{M,N+1/2}^{n+1/2}+(1+3d_1)v_{M+1,N+1/2}^{n+1/2}=(1-2d_2)v_{M+1,N+1/2}^{n}+d_2v_{M+1,N-1/2}^{n},
\end{align*}
Thus, $c=0$, $b=1+3d_1$ and $d$ is the right hand side of the previous equation.

\subsection{Step 2}
We start by rewriting our ADI equation for the Step 1 (given in class 13 slide 14), for an staggered mesh for the vertical velocity $v$:
\begin{align*}
-d_2v_{i,j-1/2}^{n+1}+(1+2d_2)v_{i,j+1/2}^{n+1}-d_2v_{i,j+3/2}^{n+1}=d_1v_{i+1,j+1/2}^{n+1/2}+(1-2d_1)v_{i,j+1/2}^{n+1/2}+d_1v_{i-1,j+1/2}^{n+1/2}
\end{align*}
which has the form
\begin{align*}
av_{i,j-1/2}^{n+1}+bv_{i,j+1/2}^{n+1}+cv_{i,j+3/2}^{n+1}=d,
\end{align*}
a tridiagonal system. The staggered mesh for $v$ gives vs a matrix $v$ that is $(M+2)\times(N+1)$. We only need to solve the previous equation for the interior with two $for$ loops in \textsl{Matlab},
\begin{verbatim}
for j=2:M+1
for i=2:N
		...
	end
end
\end{verbatim}.
We will now include the boundary conditions for each case.

\subsubsection*{Case $i=2$ and $j=2$}
In this case the general equation becomes
\begin{align*}
-d_2v_{2,3/2}^{n+1}+(1+2d_2)v_{2,5/2}^{n+1}-d_2v_{2,7/2}^{n+1}=d_1v_{3,5/2}^{n+1/2}+(1-2d_1)v_{2,5/2}^{n+1/2}+d_1v_{1,5/2}^{n+1/2}
\end{align*}
By the boundary conditions, $v_{2,3/2}^{n+1}=0$ and $v_{1,5/2}^{n+1/2}=-v_{2,5/2}^{n+1/2}$. Thus $a=0$ and $d$ is given by the right hand side.

\subsubsection*{Case $i=2$ and $j\in[3,N-1]$}
In this case the general equation becomes
\begin{align*}
-d_2v_{2,j-1/2}^{n+1}+(1+2d_2)v_{2,j+1/2}^{n+1}-d_2v_{2,j+3/2}^{n+1}=d_1v_{3,j+1/2}^{n+1/2}+(1-2d_1)v_{2,j+1/2}^{n+1/2}+d_1v_{1,j+1/2}^{n+1/2}
\end{align*}
By the boundary conditions, $v_{1,j+1/2}^{n+1/2}=-v_{2,j+1/2}^{n+1/2}$ and $d$ is given by the right hand side.

\subsubsection*{Case $i=2$ and $j=N$}
In this case the general equation becomes
\begin{align*}
-d_2v_{2,N-1/2}^{n+1}+(1+2d_2)v_{2,N+1/2}^{n+1}-d_2v_{2,N+3/2}^{n+1}=d_1v_{3,N+1/2}^{n+1/2}+(1-2d_1)v_{2,N+1/2}^{n+1/2}+d_1v_{1,N+1/2}^{n+1/2}
\end{align*}
By the boundary conditions, $v_{2,N+3/2}^{n+1}=0$ and $v_{1,N+1/2}^{n+1/2}=-v_{2,N+1/2}^{n+1/2}$. Thus $c=0$ and $d$ is given by the right hand side.

\subsubsection*{Case $i\in[3,M]$ and $j=2$}
In this case the general equation becomes
\begin{align*}
-d_2v_{i,3/2}^{n+1}+(1+2d_2)v_{i,5/2}^{n+1}-d_2v_{i,7/2}^{n+1}=d_1v_{i+1,5/2}^{n+1/2}+(1-2d_1)v_{i,5/2}^{n+1/2}+d_1v_{i-1,5/2}^{n+1/2}
\end{align*}
By the boundary conditions, $v_{i,3/2}^{n+1}$ will be zero at the walls and will depend on $v_{i,5/2}^{n+1}$ and $v_{i,7/2}^{n+1}$ for the points at the outlet. Thus, $a=0$ and the values of $b$ and $c$ depend on $i$, $d$ is given by the right hand side.

\subsubsection*{Case $i\in[3,M]$ and $j\in[3,N-1]$}
In this case the general equation does not include any boundary conditions, therefore it is unaltered.

\subsubsection*{Case $i\in[3,M]$ and $j=N$}
In this case the general equation becomes
\begin{align*}
-d_2v_{i,N-1/2}^{n+1}+(1+2d_2)v_{i,N+1/2}^{n+1}-d_2v_{i,N+3/2}^{n+1}=d_1v_{i+1,N+1/2}^{n+1/2}+(1-2d_1)v_{i,N+1/2}^{n+1/2}+d_1v_{i-1,N+1/2}^{n+1/2}
\end{align*}
By the boundary conditions, the value of $v_{i,N+3/2}^{n+1}$ will depend on $i$. We will pass it to the right hand side and have
\begin{align*}
-d_2v_{i,N-1/2}^{n+1}+(1+2d_2)v_{i,N+1/2}^{n+1}=d_1v_{i+1,N+1/2}^{n+1/2}+(1-2d_1)v_{i,N+1/2}^{n+1/2}+d_1v_{i-1,N+1/2}^{n+1/2}+d_2v_{i,N+3/2}^{n+1}.
\end{align*}
Thus, $c=0$ and $d$ is given by the right hand side.

\subsubsection*{Case $i=M+1$ and $j=2$}
In this case the general equation becomes
\begin{align*}
-d_2v_{M+1,3/2}^{n+1}+(1+2d_2)v_{M+1,5/2}^{n+1}-d_2v_{M+1,7/2}^{n+1}=d_1v_{M+2,5/2}^{n+1/2}+(1-2d_1)v_{M+1,5/2}^{n+1/2}+d_1v_{M,5/2}^{n+1/2}
\end{align*}
By the boundary conditions, $v_{M+1,3/2}^{n+1}=0$ and $v_{M+2,5/2}^{n+1/2}=-v_{M+1,5/2}^{n+1/2}$. Thus, $a=0$ and $d$ is given by the right hand side.

\subsubsection*{Case $i=M+1$ and $j\in[3,N-1]$}
In this case the general equation becomes
\begin{align*}
-d_2v_{M+1,j-1/2}^{n+1}+(1+2d_2)v_{M+1,j+1/2}^{n+1}-d_2v_{M+1,j+3/2}^{n+1}=d_1v_{M+2,j+1/2}^{n+1/2}&+(1-2d_1)v_{M+1,j+1/2}^{n+1/2}\\&+d_1v_{M,j+1/2}^{n+1/2}
\end{align*}
By the boundary conditions, $v_{M+2,j+1/2}^{n+1/2}=-v_{M+1,j+1/2}^{n+1/2}$ and $d$ is given by the right hand side.

\subsubsection*{Case $i=M+1$ and $j=N$}
In this case the general equation becomes
\begin{align*}
-d_2v_{M+1,N-1/2}^{n+1}+(1+2d_2)v_{M+1,N+1/2}^{n+1}-d_2v_{M+1,N+3/2}^{n+1}=d_1v_{M+2,N+1/2}^{n+1/2}&+(1-2d_1)v_{M+1,N+1/2}^{n+1/2}\\+d_1v_{M,N+1/2}^{n+1/2}
\end{align*}
By the boundary conditions, $v_{M+1,N+3/2}^{n+1}=0$ and $v_{M+2,N+1/2}^{n+1/2}=-v_{M+1,N+1/2}^{n+1/2}$. Thus, $c=0$ and $d$ is given by the right hand side.


\subsection{Vertices and edges}

A graph is a set of vertices \(V\), a set of edges E which are subset of pairs from V.

undirected so each edge is a set

\subsubsection{Degree of a vertex}

The degree of a vertex is the number of edges connections to it.


\section{The wedge and smash product of pointed types}

\begin{defn}
Let $A$ and $B$ be pointed types.
\begin{enumerate}
\item We define the \define{wedge} $A\vee B$ of $A$ and $B$ to be the pushout
\begin{equation*}
\begin{tikzcd}
\unit \arrow[r] \arrow[d] & B \arrow[d] \\
A \arrow[r] & A\vee B
\end{tikzcd}
\end{equation*}
\item We define the \define{smash product} $A\wedge B$ of $A$ and $B$ to be the cofiber of the cogap map of the square
\begin{equation*}
\begin{tikzcd}
\unit \arrow[r] \arrow[d] & B \arrow[d] \\
A \arrow[r] & A\times B
\end{tikzcd}
\end{equation*}
That is, the smash product is defined as the cofiber of the canonical map $A\vee B\to A\times B$.
\end{enumerate}
\end{defn}

For any two pointed types $A$ and $B$, there is a pointed map
\begin{equation*}
\mathsf{pair}_\ast : A \to_\ast (B\to_\ast A\wedge B).
\end{equation*}

\begin{thm}\label{thm:smash_adj}
Let $A$, $B$, and $X$ be pointed types. Then the pointed map
\begin{equation*}
(A\wedge B \to_\ast X)\to_\ast (A \to_\ast (B\to_\ast X))
\end{equation*}
given by $f\mapsto f\mathbin{\circ_\ast}\mathsf{pair}_\ast$ is an equivalence.
Moreover, these equivalences are natural in $A$, $B$, and $X$ in the sense that...
\end{thm}

\begin{cor}
For any $m,n:\N$ we have an equivalence
\begin{equation*}
\eqv{{\sphere{m}}\wedge{\sphere{n}}}{\sphere{m+n}}.
\end{equation*}
\end{cor}

\begin{proof}
We have
\begin{align*}
({\sphere{m}}\wedge{\sphere{n}}\to_\ast X) & \eqvsym (\sphere{m}\to_\ast (\sphere{n}\to_\ast X)) \\
& \eqvsym \loopspace[m]{\loopspace[n]{X}} \\
& \eqvsym \loopspace[m+n]{X} \\
& \eqvsym (\sphere{m+n}\to_\ast X)
\end{align*}
By the naturality of the equivalences in \cref{thm:smash_adj} it follows that the composite equivalence is given by precomposition by the pointed map
\begin{equation*}
\sphere{m}\wedge\sphere{n}\to_\ast \sphere{m+n}
\end{equation*}
that corresponds to the identity map $\sphere{m+n}\to_\ast \sphere{m+n}$. Thus it follows by \cref{ex:yoneda_ptd_types} that this pointed map is an equivalence.
\end{proof}

\begin{thm}
Given two pointed spaces, there is an equivalence
\begin{equation*}
\eqv{\join{X}{Y}}{\susp(X\wedge Y)}.
\end{equation*}
\end{thm}

\begin{exercises}
\exercise
\begin{subexenum}
\item Show that $Y$ is equivalent to the mapping cone of $X\to X\vee Y$.
\item Show that the pushout of $X \leftarrow X\vee Y \rightarrow Y$ is contractible.
\end{subexenum}
\exercise Let $A$ and $B$ be pointed types. Show that the square
\begin{equation*}
\begin{tikzcd}
A+B \arrow[r] \arrow[d] & A\times B \arrow[d] \\
1+1 \arrow[r] & A\wedge B
\end{tikzcd}
\end{equation*}
is cocartesian.
\exercise Show that if
\begin{equation*}
\begin{tikzcd}
S_1 \arrow[r] \arrow[d] & Y_1 \arrow[d] & S_2 \arrow[r] \arrow[d] & Y_2 \arrow[d] \\
X_1 \arrow[r] & Z_1 & X_2 \arrow[r] & Z_2
\end{tikzcd}
\end{equation*}
are pushout squares, where all types, maps and homotopies are pointed, then so is
\begin{equation*}
\begin{tikzcd}
S_1\vee S_2 \arrow[r] \arrow[d] & Y_1\vee Y_2 \arrow[d] \\
X_1 \vee X_2 \arrow[r] & Z_1\vee Z_2.
\end{tikzcd}
\end{equation*}
\exercise Show that if
\begin{equation*}
\begin{tikzcd}
S \arrow[r] \arrow[d] & Y \arrow[d] \\
X \arrow[r] & Z
\end{tikzcd}
\end{equation*}
is a cocartesian square of pointed spaces, then the cofiber of $X\vee Y\to Z$ is equivalent to $\susp(S)$.
\exercise Show that there is an equivalence
\begin{equation*}
\eqv{\susp(X\times Y)}{\susp(X\vee Y)\vee \susp(X\wedge Y)}
\end{equation*}
\exercise Show that $\susp(X\vee Y)$ is a retract of $\susp(X\times Y)$.
\exercise Show that if $f:A\to X$ is a constant of pointed spaces, then $\eqv{M_f}{X\vee \susp(A)}$.
\exercise Show that the cofiber of the diagonal $\delta:\sphere{1}\to \sphere{1}\times\sphere{1}$ is equivalent to $\sphere{2}\vee\sphere{2}$.
\exercise Show that $\eqv{\mathsf{Fin}(n+1)\wedge \mathsf{Fin}(m+1)}{\mathsf{Fin}(n\cdot m)+\unit}$.
\end{exercises}


\subsection{Lipschitz continuity}\label{subsec:lipschitz_continuity}

\begin{definition}\label{def:lipschitz_continuity}
  Let \( f: X \to Y \) be a function between metric spaces.

  \begin{thmenum}
    \thmitem{def:lipschitz_continuity/holder} We say that \( f: X \to Y \) is \term{H\"older continuous} at \( x \in X \) with constant \( L \geq 0 \) and exponent \( \alpha > 0 \) if
    \begin{equation*}
      \rho_Y(f(x_1), f(x_2)) \leq L \rho_X(x_1, x_2)^\alpha \quad\forall x_1, x_2 \in X.
    \end{equation*}

    We refer to the smallest such constant, if any, as \enquote{the} H\"older constant.

    \thmitem{def:lipschitz_continuity/locally_holder} We say that \( f \) is \term{locally H\"older continuous} if every point has a neighborhood where \( f \) is H\"older continuous with the same exponent, but possibly with with a different constant.

    \thmitem{def:lipschitz_continuity/lipschitz} If \( \alpha = 1 \), we say that \( f \) is \term{Lipschitz continuous}.

    \thmitem{def:lipschitz_continuity/contraction} If \( X = Y \) and if \( f \) is Lipschitz with constant \( L < 1 \), we call \( f \) a \term{contraction mapping}.

    \thmitem{def:lipschitz_continuity/calm}\cite[53]{DontchevRockafellar2014} We say that \( f \) is \term{calm} at \( x \) if it satisfies the Lipschitz condition with one of the points fixed:
    \begin{equation*}
      \rho_Y(f(x), f(x')) \leq L \rho_X(x, x') \quad\forall x' \in X.
    \end{equation*}
  \end{thmenum}
\end{definition}

\begin{proposition}\label{thm:holder_map_is_uniformly_continuous}
  A H\"older map is uniformly continuous.
\end{proposition}
\begin{proof}
  Let \( f: X \to Y \) be a H\"older map with constant \( L \) and exponent \( \alpha \).

  Fix \( \varepsilon > 0 \). Then is enough to choose \( \delta < \sqrt[\alpha]{\frac \varepsilon L} \), so that
  \begin{equation*}
    \rho_X(x_1, x_2) < \delta \implies \rho_Y(f(x_1), f(x_2)) \leq L \rho_X(x_1, x_2)^\alpha < L \delta^\alpha < \varepsilon.
  \end{equation*}

  This implies uniform continuity.
\end{proof}

\begin{corollary}\label{thm:locally_holder_map_is_continuous}
  A locally H\"older map is continuous.
\end{corollary}

\begin{theorem}[Banach's fixed point theorem]\label{thm:banach_fixed_point_theorem}\mcite[exer. 4.3.J]{Engelking1989}
  A contraction \hyperref[def:lipschitz_continuity/contraction]{mapping} in a \hyperref[def:complete_metric_space]{complete metric space} has a unique fixed \hyperref[def:fixed_point]{point}.
\end{theorem}
\begin{proof}
  Let \( f: X \to X \) be a contraction mapping. Fix any point \( x_0 \in X \) and inductively define the sequence
  \begin{equation*}
    x_{k+1} \coloneqq f(x_k), k = 1, 2, \ldots
  \end{equation*}

  Fix \( \varepsilon > 0 \). Since \( L < 1 \), there exists an index \( k_0 > \log_L(\varepsilon) \) such that for positive integers \( m \) and \( k > k_0 \),
  \begin{balign*}
    \rho(x_k, x_{k+m})
     & =
    \rho(f^k(x_0), f^{k+m}(x_0))
    \leq \\ &\leq
    L^k \rho(x_0, x_m)
    <    \\ &<
    \varepsilon \rho(x_0, x_m).
  \end{balign*}

  Note that
  \begin{balign*}
    \rho(x_0, x_m)
     & \leq
    \sum_{i=1}^m \rho(x_{i-1}, x_i)
    \leq    \\ &\leq
    \rho(x_0, x_1) \sum_{i=1}^m L^{i-1}
    =       \\ &=
    \rho(x_0, x_1) \frac {1 - L^m} {1 - L}
    \leq    \\ &\leq
    \rho(x_0, x_1) \frac 1 {1 - L}.
  \end{balign*}

  Thus,
  \begin{equation*}
    \rho(x_k, x_{k+m}) < \frac {\varepsilon \rho(x_0, x_1)} {1 - L}.
  \end{equation*}

  The constant on the right is linear in \( \varepsilon \) and does not depend on \( k \) or \( m \), hence \( \{ x_k \}_{k=0}^\infty \) is a fundamental sequence. Since \( X \) is complete, the sequence has a limit \( x \).

  Because of the continuity of \( f \) (see \fullref{thm:holder_map_is_uniformly_continuous}),
  \begin{equation*}
    f(x) = f(\lim_{k \to \infty} x_k) = \lim_{k \to \infty} f(x_k) = \lim_{k \to \infty} x_{k+1} = x.
  \end{equation*}
\end{proof}


\subsection{Particle Swarm Optimization (PSO)}
PSO is an algorithm that simulates the sociological behavior of bird flocking. In the search space, we define a swarm $X$, which has many particles $x^p(t) \in \{x^1(t), x^2(t), \ldots, \gamma\}$ where $\gamma$ is a number of particles and $t$ is the $t$th iteration among $T$ total iterations. Each particle $x^p$ consists of many elements, $x^p_i$ where $i = 1,2,3, \ldots, \kappa$, with $\kappa$ is the dimension of a particle.

Firstly, the particles of the swarm are initialized and evaluated by a predefined fitness function. The objective of the PSO is to minimize these fitness function values. In order to move the particle in the search space, a velocity $v^p_j(t)$ is defined. It is the flight speed of the $j$th element of the $p$th particle at the $t$th iteration. Additionally, $x^p_j(t)$ is the current position of that element. The velocity and current position is then calculated by the following formulae:
\begin{equation}
v_{j}^{p}(t) = 2 \cdot rand() \cdot (pbest_{j}^{p} - x_{j}^{p}(t-1))
+ 2 \cdot rand() \cdot (gbest_{j} - x_{j}^{p}(t-1))
\end{equation}

\begin{equation}
x^p_j(t) = x^p_j(t - 1) + v^p_j(t)
\end{equation}

Another improved version of PSO which is introduced in \cite{eberhart2000comparing}, added two factors: the constriction and inertia which change the velocity calculation to:

\begin{equation}
v_{j}^{p}(t) = k \cdot \{w \cdot v^p_j(t-1) + \varphi_1 \cdot rand() \cdot (pbest_{j}^{p} - x_{j}^{p}(t-1))
+ \varphi_2 \cdot rand() \cdot (gbest_{j} - x_{j}^{p}(t-1)) \}
\end{equation}

where $k$ is the constriction factor and $w$ is the inertia factor, $\varphi_1$ and $\varphi_2$ are constants.

Even with the improved version of PSO, when the particle coincides with the global best position, the particle will move away from this point if the two factors above are different from zero. Furthermore, the particles will stop moving when their velocities are close to zero and they catch up with the global best position. This phenomena is called $stagination$ \cite{eberhart1998comparison}. To overcome this phenomena, the mutation process of GA has been added to PSO to form up a Hybrid PSO with Mutation (HPSOM). We will discuss this mutation process among with the Wavelet Mutation in the next section.

\subsection{Hybrid PSO with Wavelet Mutation}
The initial version of PSO with mutation process works as follows: a random particle is selected to be moved to another position in the search space by using mutation under the following operation:
\begin{equation}
mut(x_j) = x_j - \omega, r < 0
\end{equation}
\begin{equation}
mut(x_j) = x_j + \omega, r \geq 0
\end{equation}
where $x_j$ is a randomly chosen element of the particle above, $\omega$ is randomly generated in the range of $[0, 0.1 x (para^j_{max} - para^j_{min})]$, $r$ is a random number between +1 and -1, $para^j_{max}$ and $para^j_{min}$ are the upper and lower bounds of each particle element. Therefore, we can see that $\omega$ represents $1/10$ of the search space.

This version of PSO can overcome the $stagination$ phenomena. However, it can easily be seen that the mutating space is fixed by $\omega$. It is not always the best strategy for using the same size mutation space all the time. Therefore, in \cite{ling2008hybrid}, the author proposed an improvement method to dynamically adjust the mutating size $\omega$ based on the wavelet theory as follows.

\subsubsection{Hybrid PSO with Wavelet Mutation}
\
Each particle element has a chance to mutate which is controlled by a probability of mutation $p_m \in [0, 1]$. For each element, we generate a random number between 0 and 1 $r$, if $r \leq p_m$, the mutation will take place on that element. In details, let $x^p(t) = [x^p_1(t), x^p_2(t), \ldots, x^p_k(t)]$ be the current selected particle, with $x^p_j(t)$ is its randomly chosen element, $x^p_j(t) \in [param^j_{min}, param^j_{min}]$. After the mutation process, the resulting particle is $\bar{x}^p(t)$

\[ \bar{x}^p(t) = \left\{ \begin{array}{ll}
         x^p_j(t) + \sigma \times (para^j_{max} - x^p_j(t)) & \mbox{if $\sigma > 0$} \\
        x^p_j(t) + \sigma \times (x^p_j(t) - para^j_{min}) & \mbox{if $\sigma \leq 0$}.\end{array} \right. \]

where $j \in 1,2,3, \ldots k$ is the dimension of the particle and:
\begin{equation}
	\sigma = \frac{1}{\sqrt{a}}\psi(\frac{\varphi}{a})
\end{equation}

where $\psi(x)$ is a wavelet - a continuous time function (see Figure \ref{fig:fig1}) represents a certain seismic signal that satisfies two properties:

\begin{equation}
	\int_{-\infty}^{+\infty}\psi(x)dx = 0
\end{equation}

and

\begin{equation}
	\int_{-\infty}^{+\infty}|\psi(x)|^2dx < 0
\end{equation}

\begin{figure} [H]
\centering
\includegraphics[width=60.72mm]{resources/morlet1}
\caption{The original form of the Morlet wavelet}\label{fig:fig1}
\end{figure}

In this paper, we adapt this approach (HPSOWM) and transform the real-valued particles into binary-based ones. This will be discussed in the next section.


\chapter{Principal Components Analysis \label{chapter:pca}}

Clustering (Chapter~\ref{chapter:clustering}) is one approach to uncovering latent structure in data. Another are matrix decomposition methods, the most famous of which is \textbf{principal components analysis (PCA)}. PCA is a powerful statistical tool for analyzing and visualizing datasets. It has been independently discovered several times over the course of history, and can be formulated mathematically a few different ways.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Principal Components: What are they?}

You can think of PCA as a rotation of the feature space onto a set of axes that most efficiently represent the data. The \textbf{principal components} are these axes, or \textbf{basis vectors}. Here are pictures of the first two principal components for a small dataset. These were borrowed from Andrew Ng's lecture notes for CS229 at Stanford University.

\begin{center}
\includegraphics[width=0.7\textwidth]{img/l04-ang-pc1.png}
\includegraphics[width=0.7\textwidth]{img/l04-ang-pc2.png}
\end{center}

\begin{question}{}
What do you notice about principal components 1 (top) and 2 (bottom)? Think in terms of (a) the variance (spread) of the data, and (b) the projection errors (dotted lines).
\end{question}

Mathematically, PCA is a linear projection of $p$-dimensional datapoints, $x^{(1)}, \dots, x^{(n)}$ onto a $k$-dimensional space ($k \leq p$) defined by basis vectors $u_1, \dots, u_k$ such that:
\begin{itemize}
\item the projection maximizes the variance from the original dataset that is retained
\item the projection minimizes projection error (square loss)
\item the basis vectors, $\left\{u_1, \dots, u_k\right\}$, are orthogonal (i.e., perpendicular)
\end{itemize}

The number of distinct principal components is the smaller of the number of original variables ($p$) or the number of observations ($n$) minus one.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Definitions}

\subsection{Feature Vectors}

Say you have $n$ samples in a dataset, and each has dimensionality $p$. Let $x^{(i)}$ be the vector of $p$ features for the $i$th person. We write

$$ x^{(i)} = \begin{bmatrix}
           x_{1}^{(i)} \\
           x_{2}^{(i)} \\
           \vdots \\
           x_{p}^{(i)}
         \end{bmatrix} $$

\noindent and we can write our entire dataset as a $n \times p$ matrix like this:

$$ X = \begin{bmatrix}
x_1^{(1)} & x_2^{(1)} & \dots & x_p^{(1)} \\
x_1^{(2)} & x_2^{(2)} & \dots & x_p^{(2)} \\
\vdots & \vdots & \ddots & \vdots \\
x_1^{(n)} & x_2^{(n)} & \dots & x_p^{(n)} \\
\end{bmatrix} $$
such that the $i$th row of the matrix is the vector of features for the $i$th person. This is exactly the same notation that we have used in previous chapters.

\subsection{Variance and Covariance \label{ssect:varcovar}}

The \textbf{sample covariance} of two features $x_j$ and $x_k$ is given by
$$ \text{cov}(x_j, x_k) = S_{jk} = \frac{1}{n-1} \sum_{i=1}^{n}\left(x_j^{(i)}-\overline{x_j}\right) \left(x_k^{(i)}-\overline{x_k}\right) $$
where the bar ($\overline{x_j}$) refers to the mean of $x_j$ across the entire dataset. We can also write it in matrix format as
$$ S = \frac{1}{n-1} \sum_{i=1}^n (x^{(i)} - \overline{x})(x^{(i)} - \overline{x})^T. $$

The covariance can be any real number between positive and negative infinity. One weird thing about it is that it has units - the product of the units of the two features. This means that the covariance depends on the scale that is chosen for each feature, which is somewhat arbitrary. The \textbf{sample variance} of a given feature is just the covariance in cases where $j = k$ (the covariance of a feature with itself).

The \textbf{correlation} (technically the \textbf{Pearson correlation}) is a unitless, normalized version of the covariance and has the form
$$ \text{cor}(x_j, x_k)=\frac{\sum_{i=1}^{n}(x_j^{(i)}-\overline{x_j})(x_k^{(i)}-\overline{x_k})}{\sqrt{\sum_{i=1}^{n}(x_j^{(i)}-\overline{x_j})^2}{\sqrt{\sum_{i=1}^{n}(x_k^{(i)}-\overline{x_k})^2}}}. $$

\subsection{Orthogonality}

Two vectors are \textbf{orthogonal} if their \textbf{inner product}, or \textbf{dot product}, is zero. This is defined as
$$ \langle x, y \rangle = x \cdot y = \sum_{i=1}^n x_i y_i $$
where the sum is over the vector components. If this mathematical notation makes you uncomfortable, just think of ``orthogonality'' as ``perpendicularity'' and you'll be able to understand it visually.

In PCA, the vectors whose inner products we are concerned with will \emph{not} be the feature vectors for individual samples. They will be, instead, the \emph{columns} of $X$ -- the values of a single feature for all $n$ samples.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Example: Running PCA in R}

Let's say you have measurements of weight, height, and IQ ($p=3$) for $n=100$ men. Here are the first $10$ rows of the $100$ row dataset:

\begin{center}
\begin{tabular}{cccc}
Patient ID ($i$) & Height (in) & Weight (lbs) & IQ \\
\midrule
1 & 74.9 & 155.6 & 109 \\
2 & 66.5 & 171.6 & 117 \\
3 & 74.4 & 175.2 & 91 \\
4 & 65.8 & 151.3 & 84 \\
5 & 70.2 & 162.7 & 107 \\
6 & 78.8 & 206.4 & 117 \\
7 & 69.1 & 162.7 & 96 \\
8 & 70.6 & 150.3 & 110 \\
9 & 65.2 & 156.2 & 121 \\
10 & 66.8 & 153.3 & 93 \\
\end{tabular}
\end{center}

\noindent Here are some scatterplots showing pairwise comparisons of the three features:

\begin{center}
\includegraphics[width=0.32\textwidth]{img/l04-example-1.png}
\includegraphics[width=0.32\textwidth]{img/l04-example-2.png}
\includegraphics[width=0.32\textwidth]{img/l04-example-3.png}
\end{center}

\noindent The \textbf{correlation matrix} for these data looks like this:

\begin{verbatim}
       height weight     iq
height  1.000  0.790 -0.103
weight  0.790  1.000 -0.078
iq     -0.103 -0.078  1.000
\end{verbatim}

\vspace{2mm}

\begin{question}{}
Looking at the correlation matrix, which features are most tightly correlated? What does this imply about the direction of the first principal component, PC1?
\end{question}

\subsection{Centering and Scaling}

PCA is sensitive to the relative scales of the different variables in the original dataset. For this reason, datasets are usually \emph{scaled} and \emph{centered} before PCA is performed. All this means is that for each column (feature) of the dataset, we subtract its mean and divide by its standard deviation. The transformed column will have mean 0 and standard deviation 1.

If the data are scaled and centered, we can rewrite $S$ (the sample covariance matrix from Section~\ref{ssect:varcovar}) as $X^T X/(n-1)$.

\vspace{2mm}

\begin{question}{}
What would happen to the principal components if you didn't center and scale the data?
\end{question}

\begin{question}{}
Why do you think the interpretation of the principal components becomes more difficult if you have features measured on lots of different scales (e.g., some categorical, some numeric/roughly normal, some numeric/highly skewed)?
\end{question}

\subsection{Input and Output}

In R, we can run PCA on our dataset, \texttt{d}, using the following command:
\begin{center}
\verb|p <- prcomp(d[,2:4], center = TRUE, scale. = TRUE, rank. = 3)|
\end{center}
where \texttt{rank.} is an optional parameter indicating the number, $k$, of principal components desired. The output looks like this:

\begin{multicols}{2}
{\scriptsize
\begin{verbatim}
> p$sdev
[1] 1.3454074 0.9899765 0.4580672

> p$rotation
              PC1         PC2         PC3
height  0.6996236 -0.09446119 -0.70823998
weight  0.6971414 -0.12698944  0.70559726
iq     -0.1565906 -0.98739595 -0.02299201

> p$center
   height    weight        iq
 70.74468 174.43346 101.12000

> p$scale
   height    weight        iq
 3.914989 13.853272 14.183701

> p$x
                PC1          PC2          PC3
  [1,] -0.189268854  1.976180297 -0.416704298
  [2,]  1.794925031  0.710160037  0.575478783
  [3,] -0.404617344  0.283418147  0.151536071
  [4,] -0.471675025 -0.710146894  0.347006397
  [5,] -1.073877096 -0.234684815 -0.638243675
  [6,] -0.654663317  0.043481120 -0.107082277
  [7,] -1.705841441  0.368990046 -0.794047654
  [8,]  2.311353981 -0.501565781  0.033112309
  [9,] -0.006387264 -0.535422493 -1.126177513
 [10,] -1.557136911  1.399023167 -0.045281059
 [11,]  2.632974868 -1.046037442 -0.239934797
 [12,] -0.978337630 -0.193476727  0.007814938
 [13,] -0.625980903 -0.456274882 -0.305349616
 (continued)
\end{verbatim}
}
\end{multicols}

\begin{question}{}
Describe/draw the directions of the three principal component vectors, PC1, PC2, and PC3, in the coordinate system of the original predictors, height, weight, and IQ.
\end{question}

\begin{question}{}
Here is a picture of the flow cytometry dataset we first encountered in Chapter~\ref{chapter:clustering}.
\begin{center}
\includegraphics[width=0.7\textwidth]{img/biomarker-data-big-no-labels.png}
\end{center}
What would PC1 and PC2 look like for this dataset? (Why is there no PC3?) How could PCA help you separate the two clusters?
\end{question}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Applications of PCA}

\subsection{Eigenfaces}

Eigenfaces were an early application of PCA to computer vision, specifically image search and retrieval. Here's what you do to create a set of eigenfaces:
\begin{enumerate}
\item Prepare training set of images taken under the same lighting conditions, with mouths and eyes aligned, resampled to a common pixel resolution.
\item Generate a vector for each image by concatenating the pixel intensities across the rows. So if the image is of dimension $r \times c$, the image vector will have $p = rc$ features.
\item Create the data matrix from the $n$ vectors.
\item Center the data (Why wouldn't we scale the data?).
\item Calculate the eigenvectors and eigenvalues of the sample covariance matrix. Each eigenvector corresponds to one eigenface.
\end{enumerate}

Some real eigenfaces are shown below \footnote{Figure details: (top) some of the original faces from the training set (there were 86 images total); (bottom) the eigenfaces corresponding to the 18 largest eigenvalues of the covariance matrix. From https://www.clear.rice.edu/elec301/Projects99/faces/images.html.}.

\begin{center}
\includegraphics[width=0.8\textwidth]{img/l04-eigenfaces-1.png}\\[5mm]
\includegraphics[width=0.8\textwidth]{img/l04-eigenfaces-2.png}
\end{center}

\begin{question}{}
What is $X$ for the eigenfaces problem? What are the principal components? How could you use the principal components to match a new face to an existing database of faces?
\end{question}

\subsection{Image Compression}

PCA can also be used for image compression. In that case, the raw pixels for the image are the data matrix, $X$ (so each ``sample'' is one row of the image and each ``feature'' is one column of the image). Generally the data are centered but not scaled here (Why?). A subset of the principal components corresponding to the $r$ largest eigenvalues are used to reconstruct the image. The picture below illustrates the process\footnote{Figure: Using PCA for image compression. The image is recomposed by adding together many gridlike images like that shown in (a). From \texttt{https://www.projectrhea.org/rhea/index.php/PCA\_Theory\_Examples}.}.

\begin{center}
\includegraphics[width=\textwidth]{img/l04-compression.png}
\end{center}

\begin{question}{}
What is $X$ for the image compression problem? What are the principal components? How does using PCA help compress the image?
\end{question}

\subsection{Genetic Ancestry}

There are many awesome population genetics papers that use PCA to locate individuals within the ``space'' of possible genetic mutations. Here's some text from one, a figure from which is shown below\footnote{From European Journal of Human Genetics (2016) 24, 931-936 (2016).}. Here's a quote from the original paper:

\begin{quote}
Here, we analyse genome-wide variation in 173 Armenians and compare them with 78 other worldwide populations. We find that Armenians form a distinctive cluster linking the Near East, Europe, and the Caucasus. We show that Armenian diversity can be explained by several mixtures of Eurasian populations that occurred between ~3000 and ~2000 BCE, a period characterized by major population migrations after the domestication of the horse, appearance of chariots, and the rise of advanced civilizations in the Near East. However, genetic signals of population mixture cease after ~1200 BCE when Bronze Age civilizations in the Eastern Mediterranean world suddenly and violently collapsed. Armenians have since remained isolated and genetic structure within the population developed ~500 years ago when Armenia was divided between the Ottomans and the Safavid Empire in Iran.
\end{quote}

\begin{center}
\includegraphics[width=\textwidth]{img/l04-genetics.jpg}
\end{center}

\begin{question}{}
What is $X$ for the genetic ancestry problem? What are the principal components? How were the principal components used to produce the figure above?
\end{question}

\subsection{Topic Modeling}

PCA also has many uses in natural language processing. It forms the basis for latent semantic indexing (see Deerwester 1990) and can also be used for topic modeling, a form of mixture model (see Section~\ref{sect:mixturemodels})\footnote{Figure: Topic modeling French crime fiction. From https://dragonfly.hypotheses.org/530.}.

\begin{center}
\includegraphics[width=0.7\textwidth]{img/l04-topics.png}
\end{center}

\begin{question}{}
What is $X$ for the topic modeling problem? For the latent semantic indexing problem? What do the principal components represent?
\end{question}

\begin{question}{}
Think of 2-3 different unsupervised learning problems from biology or medicine where PCA makes sense, conceptually at least, for modeling the data. How would you set up the data matrix in each case? What would the principal components correspond to in the data?
\end{question}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Technical Details (Advanced)}

\subsection{Eigenvalues and Eigenvectors \label{ssect:eig}}

One interpretation of matrix multiplication, $Ax$, where $A$ is an $m \times m$ square matrix and $x$ a vector of length $m$, is that $A$ linearly transforms $x$ by rotating it, changing its magnitude, or both. An \textbf{eigenvector} of $A$ is a vector that, when multiplied by $A$, grows or shrinks in magnitude but does not rotate. The multiplier of its magnitude is given by the corresponding \textbf{eigenvalue}. There are $m$ different eigenvalues and eigenvectors for an $m \times m$ matrix, although the eigenvalues may not all be distinct. The set of all eigenvalues of $A$ is called the \textbf{spectrum} of $A$.

For all eigenvalue-eigenvector pairs, the relationship $Av = \lambda v$, where $v$ is an eigenvector and $\lambda$ its corresponding eigenvalue, must hold. To find the $\lambda$s and $v$s analytically, we can set the determinant $ | A - \lambda I | $, called the \textbf{characteristic polynomial}, equal to zero and solve for the eigenvalue and eigenvector corresponding to each root. Note that any scalar multiple of an eigenvector is itself an eigenvector; usually we restrict eigenvectors to have magnitude $1$ for this reason.

If a matrix is \textbf{symmetric} and \textbf{positive semidefinite} (all positive or zero eigenvalues), all of its eigenvectors will be orthogonal.

\subsection{PCA: Eigendecomposition Version}

The first way we can find the principal components is by performing an eigen-decomposition identical to the one we performed in Section~\ref{ssect:eig} on the sample covariance matrix. The principal components will be the eigenvectors of this matrix, and the corresponding eigenvalues tell you how much of the dataset's overall variance is accounted for by each eigenvector.

One useful fact about the covariance matrix is that, because it is a symmetric matrix, it can be diagonalized:
\begin{equation} S = \frac{1}{n-1} X^T X = V L V^T \label{eqn:eigendiag} \end{equation}
where $V$ is a matrix of the $p$ eigenvectors of $S$ (each column is an eigenvector) and $L$ is a diagonal matrix with the eigenvalues of $S$ along the diagonal. This fact will be important later.

\subsection{PCA: Singular Value Decomposition (SVD) Version}

Although the eigendecomposition of the covariance matrix is generally the way PCA is presented, in practice most software uses another matrix decomposition called the \textbf{singular value decomposition (SVD)}, even though it is slower to compute, because it's more numerically stable.

Here's why the two methods are equivalent. Any matrix, $M$, with real or complex values (we'll focus on real valued matrices here) has an SVD of the form\footnote{This is in contrast with the eigendecomposition, which works only on positive semidefinite matrices.}:
$$ M = U D V^T $$
where $U$ is an orthogonal matrix, meaning that $U^TU = I$, $D$ is diagonal and all of its nonzero elements are non-negative real numbers called \textbf{singular values}, and $V$ is also orthogonal. For a square matrix, the SVD can be viewed as ``breaking up'' the transformation described by a matrix into three separate transformations: a rotation/reflection, a scaling, and another rotation/reflection\footnote{Figure: Graphical representation of the SVD of a square matrix, $M$. Note that the illustrator here uses $\Sigma$ instead of $D$ for the middle, diagonal matrix. Attribution: By Georg-Johann (Own work), via Wikimedia Commons.}:

\begin{center}
\includegraphics[width=0.5\textwidth]{img/l04-svd.pdf}
\end{center}

\noindent If we perform the SVD on $X$, we obtain a decomposition
$$ X = U D V^T $$
where $D$ is a diagonal matrix with singular values $d_1, \dots, d_p$. Knowing this, we can rewrite our covariance matrix as
\begin{align*} S &= \frac{1}{n-1} X^T X \\
&= \frac{1}{n-1} (UDV^T)^T U D V^T \\
&= \frac{1}{n-1} V D U^T U D V^T \\
&= V \frac{D^2}{n-1} V^T. \end{align*}

Comparing this to the form from Equation~\ref{eqn:eigendiag}, we see that the eigenvectors correspond to the right singular values of the SVD, and the eigenvalues are related to the singular values on the diagonal of $D$ by $\lambda_i = d_i^2/(n-1)$.

There are a lot of interpretations of the SVD. The one I like best is that the SVD provides a ``nearest orthogonal matrix'' to the original matrix. It's like the data have a coordinate system of orthogonal axes in which they most like to live, and the SVD tells you what that system is.