%!TEX root = guide2.0.tex

\hypertarget{monte-carlo-methods-in-bayesian-analysis}{}
\pdfbookmark[0]{Monte Carlo Methods in Bayesian Analysis}{monte-carlo-methods-in-bayesian-analysis}

\section{Monte Carlo Methods in Bayesian Analysis}

Bayesian analysis often requires integration over multiple dimensions that is intractable both via analytic methods or standard methods of numerical integration. However, it is often possible to compute these integrals by simulating (drawing samples) from posterior distributions. For example, consider the expected value of a random variable $\mathbf{x}$:

\[
E[{\bf x}] = \int {\bf x} f({\bf x}) d{\bf x}, \qquad
{\bf x} = \{x_1,...,x_k\}
\]

\noindent where $k$ (the dimension of vector $x$) is perhaps very large. If we can produce a reasonable number of random vectors $\{{\bf x_i}\}$, we can use these values to approximate the unknown integral. This process is known as {\em Monte Carlo integration}. In general, MC integration allows integrals against probability density functions:

\[
I = \int h(\mathbf{x}) f(\mathbf{x}) \mathbf{dx}
\]

\noindent to be estimated by finite sums:

\[
\hat{I} = \frac{1}{n}\sum_{i=1}^n h(\mathbf{x}_i),
\]

\noindent where $\mathbf{x}_i$ is a sample from $f$. This estimate is valid and useful because:

\begin{itemize}
\item
By the strong law of large numbers:
\[\hat{I} \rightarrow I   \mbox{   with probability 1}\]
\item
Simulation error can be measured and controlled:
\[Var(\hat{I}) = \frac{1}{n(n-1)}\sum_{i=1}^n (h(\mathbf{x}_i)-\hat{I})^2\]
\end{itemize}

Why is this relevant to Bayesian analysis? If we replace $f(\mathbf{x})$ with a posterior, $f(\theta|d)$ and make $h(\theta)$ an interesting function of the unknown parameter, the resulting expectation is that of the posterior of $h(\theta)$:

\[
E[h(\theta)|d] = \int f(\theta|d) h(\theta) d\theta \approx \frac{1}{n}\sum_{i=1}^n h(\theta)
\]

%___________________________________________________________________________

\hypertarget{rejection-sampling}{}
\pdfbookmark[1]{Rejection Sampling}{rejection-sampling}
\subsection{Rejection Sampling}

Though Monte Carlo integration allows us to estimate integrals that are unassailable by analysis and standard numerical methods, it relies on the ability to draw samples from the posterior distribution. For known parametric forms, this is not a problem; probability integral transforms or bivariate techniques (e.g Box-Muller method) may be used to obtain samples from uniform pseudo-random variates generated from a computer. Often, however, we cannot readily generate random values from non-standard posteriors. In such instances, we can use rejection sampling to generate samples.

\begin{figure}[ht]
        \begin{center}
        \includegraphics[scale=0.4]{reject.png}
    \end{center}
    \caption{Rejection sampling of a bounded form. Area is estimated by the ratio of accepted (open squares) to total points, multiplied by the rectangle area.}
    \label{fig:bound}
\end{figure}

Posit a function, $f(x)$ which can be evaluated for any value on the support of $x:S_x = [A,B]$, but may not be integrable or easily sampled from. If we can calculate the maximum  value of $f(x)$, we can then define a rectangle that is guaranteed to contain all possible values $(x,f(x))$. It is then trivial to generate points over the box and enumerate the values that fall under the curve (Figure \ref{fig:bound}).

\[
\frac{\mbox{Points under curve}}{\mbox{Points generated}} \times \mbox{box area} = \lim_{n \to \infty} \int_A^B f(x) dx
\]

\begin{figure}[h]
        \begin{center}
        \includegraphics[scale=0.4]{envelope.png}
    \end{center}
    \caption{Rejection sampling of an unbounded form using an enveloping distribution.}
    \label{fig:unbound}
\end{figure}

\noindent This approach is useful, for example, in estimating the normalizing constant for posterior distributions.

If $f(x)$ has unbounded support (i.e. infinite tails), such as a Gaussian distribution, a bounding box is no longer appropriate. We must specify a majorizing (or, enveloping) function, $g(x)$, which implies:

\[
g(x) \ge  f(x) \qquad\forall x \in (-\infty,\infty)
\]

Having done this, we can now sample ${x_i}$ from $g(x)$ and accept or reject each of these values based upon $f(x_i)$. Specifically, for each draw $x_i$, we also draw a uniform random variate $u_i$ and accept $x_i$ if $u_i < f(x_i)/cg(x_i)$, where $c$ is a constant (Figure \ref{fig:unbound}). This approach is made more efficient by choosing an enveloping distribution that is ``close'' to the target distribution, thus maximizing the number of accepted points. Further improvement is gained by using optimized algorithms such as importance sampling which, as the name implies, samples more frequently from important areas of the distribution.

Rejection sampling is usually subject to declining performance as the dimension of the parameter space increases, so it is used less frequently than MCMC for evaluation of posterior distributions \citep{Gamerman:1997tb}.

%___________________________________________________________________________

\hypertarget{markov-chains}{}
\pdfbookmark[0]{Markov Chains}{markov-chains}
\section{Markov Chains}

A Markov chain is a special type of \emph{stochastic process}. The standard definition of a stochastic process is an ordered collection of random variables:

\[
\{X_t:t \in T\}
\]

\noindent where $t$ is frequently (but not necessarily) a time index. If we think of $X_t$ as a state $X$ at time $t$, and invoke the following dependence condition on each state:

\[
Pr(X_{t+1}=x_{t+1} | X_t=x_t, X_{t-1}=x_{t-1},\ldots,X_0=x_0) = Pr(X_{t+1}=x_{t+1} | X_t=x_t)
\]

\noindent then the stochastic process is known as a Markov chain. This conditioning specifies that the future depends on the current state, but not past states. Thus, the Markov chain wanders about the state space, remembering only where it has just been in the last time step. The collection of transition probabilities is sometimes called a \emph{transition matrix} when dealing with discrete states, or more generally, a \emph{transition kernel}.

\noindent In the context of Markov chain Monte Carlo, it is useful to think of the Markovian property as ``mild non-independence''. MCMC allows us to indirectly generate independent samples from a particular posterior distribution.

%___________________________________________________________________________

\hypertarget{jargon-busting}{}
\pdfbookmark[1]{Jargon-busting}{jargon-busting}
\subsection{Jargon-busting}

Before we move on, it is important to define some general properties of Markov chains. They are frequently encountered in the MCMC literature, and some will help us decide whether MCMC is producing a useful sample from the posterior.

\begin{itemize}
\item \emph{Homogeneity}: A Markov chain is homogeneous at step $t$ if the transition probabilities are independent of time $t$.
\item \emph{Irreducibility}: A Markov chain is irreducible if every state is accessible in one or more steps from any other state. That is, the chain contains no absorbing states. This implies that there is a non-zero probability of eventually reaching state $k$ from any other state in the chain.
\item \emph{Recurrence}: States which are visited repeatedly are \emph{recurrent}. If the expected time to return to a particular state is bounded, this is known as \emph{positive recurrence}, otherwise the recurrent state is \emph{null recurrent}. Further, a chain is \emph{Harris recurrent} when it visits all states $X \in S$ infinitely often in the limit as $t \to \infty$; this is an important characteristic when dealing with unbounded, continuous state spaces. Whenever a chain ends up in a closed, irreducible set of Harris recurrent states, it stays there forever and visits every state with probability one.
\item \emph{Stationarity}: A stationary Markov chain produces the same marginal distribution when multiplied by the transition kernel.  Thus, if $P$ is some $n \times n$ transition matrix:

\[{\bf \pi P} = {\bf \pi}\]

\noindent for Markov chain $\pi$. Thus, $\pi$ is no longer subscripted, and is referred to as the \emph{limiting distribution} of the chain. In MCMC, the chain explores the state space according to its limiting marginal distribution.
\item \emph{Ergodicity}: Ergodicity is an emergent property of Markov chains which are irreducible, positive Harris recurrent and aperiodic. Ergodicity is defined as:

\[
\lim_{n \to \infty} Pr^{(n)}(\theta_i \rightarrow \theta_j) = \pi(\theta) \quad \forall \theta_i, \theta_j \in \Theta
\]

\noindent or in words, after many steps the marginal distribution of the chain is the same at one step as at all other steps. This implies that our Markov chain, which we recall is dependent, can generate samples that are independent if we wait long enough between samples. If it means anything to you, ergodicity is the analogue of the strong law of large numbers for Markov chains. For example, take values $\theta_{i+1},\ldots,\theta_{i+n}$ from a chain that has reached an ergodic state. A statistic of interest can then be estimated by:

\[
\frac{1}{n}\sum_{j=i+1}^{i+n} h(\theta_j) \approx \int f(\theta) h(\theta) d\theta
\]

\end{itemize}

%___________________________________________________________________________

\hypertarget{reversible-markov-chains}{}
\pdfbookmark[0]{Why MCMC Works: Reversible Markov Chains}{reversible-markov-chains}
\section{Why MCMC Works: Reversible Markov Chains}

Markov chain Monte Carlo simulates a Markov chain for which some function of interest (\emph{e.g.} the joint distribution of the parameters of some model) is the unique, invariant limiting distribution. An invariant distribution with respect to some Markov chain with transition kernel $Pr(y \mid x)$ implies that:
\[
\int_x Pr(y \mid x) \pi(x) dx = \pi(y).
\]

Invariance is guaranteed for any \textbf{reversible} Markov chain. Consider a Markov chain in reverse sequence: $\{\theta^{(n)},\theta^{(n-1)},...,\theta^{(0)}\}$. This sequence is still Markovian, because:
\[
Pr(\theta^{(k)}=y \mid \theta^{(k+1)}=x,\theta^{(k+2)}=x_1,\ldots ) = Pr(\theta^{(k)}=y \mid \theta^{(k+1)}=x)
\]
Forward and reverse transition probabilities may be related through Bayes theorem:
\begin{eqnarray}
Pr(\theta^{(k)}=y \mid \theta^{(k+1)}=x) &=& \frac{Pr(\theta^{(k+1)}=x \mid \theta^{(k)}=y) Pr(\theta^{(k)}=y)}{Pr(\theta^{(k+1)}=x)} \nonumber \\
&=& \frac{Pr(\theta^{(k+1)}=x \mid \theta^{(k)}=y) \pi^{(k)}(y)}{\pi^{(k+1)}(x)} \nonumber
\end{eqnarray}

\[
\frac{Pr(\theta^{(k+1)}=x \mid \theta^{(k)}=y) \pi^{(k)}(y)}{\pi^{(k+1)}(x)}
\]

\noindent Though not homogeneous in general, $\pi$ becomes homogeneous if \textbf{Do you ever call the stationary distribution itself homogeneous?}:
\begin{itemize}
\item $n \rightarrow \infty$
\item $\pi^{(0)}=\pi$ for some $i < k$ \textbf{Is it meant to be $\pi^(i)$, and }
\end{itemize}

\noindent If this chain is homogeneous it is called reversible, because it satisfies the \textbf{detailed balance equation}:
\[
\pi(x)Pr(y \mid x) = \pi(y) Pr(x \mid y)
\]
Reversibility is important because it has the effect of balancing movement through the entire state space. When a Markov chain is reversible, $\pi$ is the unique, invariant, stationary distribution of that chain.
Hence, if $\pi$ is of interest, we need only find the reversible Markov chain for which $\pi$ is the limiting distribution. This is what MCMC does!

%___________________________________________________________________________

\hypertarget{gibbs-sampling}{}
\pdfbookmark[0]{Gibbs Sampling}{gibbs-sampling}
\section{Gibbs Sampling}

The Gibbs sampler is the simplest and most prevalent MCMC algorithm. If a posterior has $k$ parameters to be estimated, we may condition each parameter on current values of the other $k-1$ parameters, and sample from the resultant distributional form (usually easier), and repeat this operation on the other parameters in turn. This procedure generates samples from the posterior distribution. Note that we have now combined Markov chains (conditional independence) and Monte Carlo techniques (estimation by simulation) to yield Markov chain Monte Carlo.

Here is a stereotypical Gibbs sampling algorithm:

\newcounter{lcount}
\begin{list}{\arabic{lcount}}
{\usecounter{lcount}}
\item Choose starting values for states (parameters): ${\bf \theta} = [\theta_1^{(0)},\theta_2^{(0)},\ldots,\theta_k^{(0)}]$
\item Initialize counter $j=1$
\item Draw the following values from each of the $k$ conditional distributions:
\begin{eqnarray*}
\theta_1^{(j)} &\sim& \pi(\theta_1 | \theta_2^{(j-1)},\theta_3^{(j-1)},\ldots,\theta_{k-1}^{(j-1)},\theta_k^{(j-1)}) \\
\theta_2^{(j)} &\sim& \pi(\theta_2 | \theta_1^{(j)},\theta_3^{(j-1)},\ldots,\theta_{k-1}^{(j-1)},\theta_k^{(j-1)}) \\
\theta_3^{(j)} &\sim& \pi(\theta_3 | \theta_1^{(j)},\theta_2^{(j)},\ldots,\theta_{k-1}^{(j-1)},\theta_k^{(j-1)}) \\
\vdots \\
\theta_{k-1}^{(j)} &\sim& \pi(\theta_{k-1} | \theta_1^{(j)},\theta_2^{(j)},\ldots,\theta_{k-2}^{(j)},\theta_k^{(j-1)}) \\
\theta_k^{(j)} &\sim& \pi(\theta_k | \theta_1^{(j)},\theta_2^{(j)},\theta_4^{(j)},\ldots,\theta_{k-2}^{(j)},\theta_{k-1}^{(j)})
\end{eqnarray*}
\item Increment $j$ and repeat until convergence occurs.
\end{list}

As we can see from the algorithm, each distribution is conditioned on the last iteration of its chain values, constituting a Markov chain as advertised. The Gibbs sampler has all of the important properties outlined in the previous section: it is aperiodic, homogeneous and ergodic. Once the sampler converges, all subsequent samples are from the target distribution. This convergence occurs at a geometric rate.

\hypertarget{the-metropolis-hastings-algorithm}{}
\pdfbookmark[0]{The Metropolis-Hastings Algorithm}{the-metropolis-hastings-algorithm}
\section{The Metropolis-Hastings Algorithm}

The key to success in applying the Gibbs sampler to the estimation of Bayesian posteriors is being able to specify the form of the complete conditionals of ${\bf \theta}$. In fact, the algorithm cannot be implemented without them. Of course, the posterior conditionals cannot always be neatly specified. In contrast to the Gibbs algorithm, the Metropolis-Hastings algorithm generates candidate state transitions from an alternate distribution, and accepts or rejects each candidate probabilistically.

Let us first consider a simple Metropolis-Hastings algorithm for a single parameter, $\theta$. We will use a standard sampling distribution, referred to as the \emph{proposal distribution}, to produce candidate variables $q_t(\theta^{\prime} | \theta)$. That is, the generated value, $\theta^{\prime}$, is a \emph{possible} next value for $\theta$ at step $t+1$. We also need to be able to calculate the probability of moving back to the original value from the candidate, or $q_t(\theta | \theta^{\prime})$. These probabilistic ingredients are used to define an \emph{acceptance ratio}:

\[
a(\theta^{\prime},\theta) = \frac{q_t(\theta^{\prime} | \theta) \pi(\theta^{\prime})}{q_t(\theta | \theta^{\prime}) \pi(\theta)}
\]

\noindent The value of $\theta^{(t+1)}$ is then determined by:

\[
\theta^{(t+1)} = \left\{\begin{array}{l@{\quad \mbox{with prob.} \quad}l}\theta^{\prime} & \min(a(\theta^{\prime},\theta),1) \\ \theta^{(t)} & 1 - \min(a(\theta^{\prime},\theta),1) \end{array}\right.
\]

\noindent This transition kernel implies that movement is not guaranteed at every step. It only occurs if the suggested transition is likely based on the acceptance ratio.

A single iteration of the Metropolis-Hastings algorithm proceeds as follows:

\newcounter{lcount2}
\begin{list}{\arabic{lcount2}}
{\usecounter{lcount2}}
\item Sample $\theta^{\prime}$ from $q(\theta^{\prime} | \theta^{(t)})$.
\item Generate a Uniform[0,1] random variate $u$.
\item If $a(\theta^{\prime},\theta) > u$ then $\theta^{(t+1)} = \theta^{\prime}$, otherwise $\theta^{(t+1)} = \theta^{(t)}$.
\end{list}

\noindent The original form of the algorithm specified by Metropolis required that $q_t(\theta^{\prime} | \theta) = q_t(\theta | \theta^{\prime})$, which reduces $a(\theta^{\prime},\theta)$ to $\pi(\theta^{\prime})/\pi(\theta)$, but this is not necessary. In either case, the state moves to high-density points in the distribution with high probability, and to low-density points with low probability. After convergence, the Metropolis-Hastings algorithm describes the full target posterior density, so all points are recurrent.

%___________________________________________________________________________

\hypertarget{random-walk-metropolis-hastings}{}
\pdfbookmark[1]{Random-walk Metropolis-Hastings}{random-walk-metropolis-hastings}
\subsection{Random-walk Metropolis-Hastings}

A practical implementation of the Metropolis-Hastings algorithm makes use of a random-walk proposal. Recall that a random walk is a Markov chain that evolves according to:

\begin{eqnarray*}
\theta^{(t+1)} &=& \theta^{(t)} + \epsilon_t \\
\epsilon_t &\sim& f(\phi)
\end{eqnarray*}

As applied to the MCMC sampling, the random walk is used as a proposal distribution, whereby dependent proposals are generated according to:

\[
q(\theta^{\prime} | \theta^{(t)}) = f(\theta^{\prime} - \theta^{(t)}) = \theta^{(t)} + \epsilon_t
\]

Generally, the density generating $\epsilon_t$ is symmetric about zero, resulting in a symmetric chain. Chain symmetry implies that $q(\theta^{\prime} | \theta^{(t)}) = q(\theta^{(t)} | \theta^{\prime})$, which reduces the Metropolis-Hastings acceptance ratio to:

\[
a(\theta^{\prime},\theta) = \frac{\pi(\theta^{\prime})}{\pi(\theta)}
\]

The choice of the random walk distribution for $\epsilon_t$ is frequently a normal or Student's $t$ density, but it may be any distribution that generates an irreducible proposal chain.

An important consideration is the specification of the scale parameter for the random walk error distribution. Large values produce random walk steps that are highly exploratory, but tend to produce proposal values in the tails of the target distribution, potentially resulting in very small acceptance rates. Conversely, small values tend to be accepted more frequently, since they tend to produce proposals close to the current parameter value, but may result in chains that mix very slowly. Some simulation studies suggest optimal acceptance rates in the range of 20-50\%. It is often worthwhile to optimize the proposal variance by iteratively adjusting its value, according to observed acceptance rates early in the MCMC simulation \citep{Gamerman:1997tb}.


%!TEX root =  ../main.tex

\subsection{Polynomial Long Division}

\objective{Use and known when to use polynomial and synthetic division}

First, we recall long division.  Surprisingly, it can actually be more difficult than polynomial long division at times, so we should pick an easy example:

\vspace{5mm}

\longdiv{11}{3678}

We restrict our vision to the left two digits of the dividend, as we know that 11 will go 3 times into
36.  We write a 3 above the 6, write 33 below the 36 and subtract it.  Next, we bring down the 7
and do the same manner of work on 37.  The process repeats, but moves progressively right
and we get an answer of 334, but with a remainder of 4.  Since we most want a number that
when multiplied by 11 obtains 3678, we should write the answer as $334+\frac{4}{11}$ because
$\left(334+\frac{4}{11}\right)(11) = 3678$.

Now, polynomial long division is no different.  We start at the left edge of the dividend, and use
only the leading term of the divisor.

\polylongdiv[stage=3]{x^3-x^2+x-1}{x+1}

We only regarded the leading (left) edge of the larger polynomial.  In this case, that is $x^3$.
We might ask ourselves, ``What do I need to multiply $x$ by to get to $x^3$?''.
The answer is $x^2$.   That term is the first in our quotient, and we multiply it by
the entire divisor.  We subtract, and bring down the next term.

\polylongdiv[stage=4]{x^3-x^2+x-1}{x+1}

This process continues, only working on the left-edge as we go down.

\polylongdiv[stage=13]{x^3-x^2+x-1}{x+1}

The final answer is $x^3-x^2+x-1 = (x+1)\left(x^2-2x+3-\cfrac{4}{x+1}\right)$.

\subsection{Synthetic Division}
One might ask, ``Isn't there a faster way?'' and indeed there is.  Let us first consider
how we might compactify the polynomial.  Consider the first two terms, and look for
a common factor:
$$
(x^3-x^2)+x-1
$$
Obviously, we can take out an $x^2$ from the first two terms.  Having condensed the
first two terms into one (because multiplication does not increase the term count),
now consider the greatest common factor of the new first two term:
$$
\left[x^2(x-1)+x\right]-1
$$
Every term has at least one $x$, so that can be safely factored out.  Look at what we have made,
called \textbf{nested form}:
$$
x\left[(x-1)x+1\right]-1
$$
Substituting any value into this function now only involves multiplication and addition/subtraction,
ever exponents.  This process can be represented in half-box, called \textbf{synthetic division}

\polyhornerscheme[showvar=true,resultbottomrule=true,resultleftrule=true,resultrightrule=true,x=-1]{x^3-x^2+x-1}

Study this carefully, and notice how it is the same answer as the coefficients of the polynomial
long division problem.  There are lots of things to notice.  The technique with synthetic division is
to remember that the first number falls straight down, but from then on we  ``multiply up, add down''.
It is helpful to draw a box around the final number, to remember that it is special: it is the remainder.
Reading right-to-left: 3 is the $x^0$ coefficient, $-2$ is on $x^1$, 1 is on $x^2$.
In the long division, we divided by $x + 1$.  Here, we are plugging in a value, so it's $x + 1 = 0$,
which is $x=-1$.

\begin{example}[Synthetic Division]
\exProblem
Given $g(x) = x^3-7x+6$, find $g(2)$.  What does this tell you about $\frac{g(x)}{x-2}$?

\exSolution
First, we write our box and the $x=$ we want to test:\\

\polyhornerscheme[showvar=true,stage=1,x=2]{x^3-7x+6}
(Note the zero.  Failure to notice this is crucial mistake!)

The first number passes down freely:
\polyhornerscheme[showvar=true,tutor=true,stage=2,x=2]{x^3-7x+6}

Then we ``multiply up...
\polyhornerscheme[showvar=true,tutor=true,stage=3,x=2]{x^3-7x+6}

..., add down''
\polyhornerscheme[showvar=true,tutor=true,stage=4,x=2]{x^3-7x+6}

and so on and so forth:
\polyhornerscheme[showvar=true,resultbottomrule=true,resultleftrule=true,resultrightrule=true,x=2]{x^3-7x+6}

The fact that the remainder is 0 is significant.  $(x-2)$ is a factor in $x^3-7x+6$.  We can write
this as $(x-2)(x^2+2x-3)=x^3-7x+6$.  The quadratic factors easily into (x+3)(x-1), leaving the
entire cubic equation solved.
\end{example}

\subsection{Complex Zeros}
What if nothing so need exists as a solution?  We might slightly alter our $g(x)$ to be
$x^3-2x^2-7x+2$.  There is no easy factorization (grouping won't work).  We are forced
to use more complicated algorithms.  In the past, mathematicians invented very complicated
methods for find, eliminating, and testing solutions to polynomial equations.  You will research
one of these in problem number one of the exercises.  But in our age, why not use a computer?
If done intelligently, we can get the same level of precision as in the past.

For example, if you take our new $g(x)$ and put it in your TI-8*, you can see --- either in the
TABLE or on the graph --- that it passes through (-3,0), which should lead you to want to
factor out (x+3).  You can use synthetic division as so:

\polyhornerscheme[showvar=true,resultbottomrule=true,resultleftrule=true,resultrightrule=true,x=-3]{x^3-2x^2-7x+24}

This proves that $(x+3)(x^2-5x+8)=x^3-2x^2-7x+24$.  However, now we have a new problem: how
can we solve $x^2-5x+8=0$?  It is not factorable.  Well, any quadratic can be solved with
the quadratic formula, so that will have to do.

$$
x=\frac{5\pm\sqrt{(-5)^2-4(1)(8)}}{2\cdot{}1}
=\frac{5\pm i\sqrt{7}}{2}
$$

As you can see, unfactorable quadratics will always have some quantity inside the square root,
once plugged into the quadratic formula.  This means they will produce either a pair of
irrational solutions, or else a pair of complex (imaginary) ones.  This has other to prove
(elsewhere) that such solutions \emph{always} occur in pairs.

For example, if we are building a polynomial, and we know solutions include 2 and $3+2i$,
then we can be sure that $3-2i$ is also a solution.  What factor would produce such zeros?
Well, x-2 is an obvious one, but what about the more difficult one?  If $x=3 \pm 2i$, what
would have lead to two possibilities like that?

\begin{align*}
  x & = 3 \pm 2i\\
  x - 3 & =  2i \\
  (x-3)^2 & = -5\\
  x^2-6x+9 &= -5\\
  x^2-6x+14 &= 0
\end{align*}

Since plus-or-minus solution come from square rooting, the inverse operation must be to
square both sides.  $(+2i)(+2i) = (-2i)(-2i) = -5$, so writing plus or minus does not matter.


\chapter{Approximation properties of width-bounded ReLU DNN}
In this part, we will show a universal approximation theorem for width-bounded ReLU networks: width-(d+4) ReLU networks, where $d$ is the input dimension.

\begin{theorem}
	\label{width-bdd}
	For any Lebesgue-integrable function $f:\mathbb{R}^d\to\mathbb{R}$ and any $\epsilon>0$, there exists a fully-connected ReLU network $\mathcal{A}$ with width at most $d+4$, such that the function
	$F_{\mathcal{A}}$ represented by this network satisfies
	\begin{equation}
	\int_{\mathbb{R}^d}|f(x)-F_{\mathcal{A}}(x) |dx<\epsilon
	\end{equation}
\end{theorem}
The proof of the above theorem involves the following three steps,
\begin{enumerate}
	\item Show that $f$ can be approximated by finite weighted sum of indicator functions on d-dimensional cubes.
	\item  Use ReLU network to approximate an indicator function on an d-dimensional cube.
	\item  Show that the constructed ReLU network can ``store" the quantites and sum them up.
\end{enumerate}
 The first step can be shown by the knowledge from real analysis.

 As for the second step, let us consider the easy case in 1D to demonstrate the idea. Given $[a,b]$, the goal now is to use ReLU function to approximate the indicator function $\chi_{[a,b]}$.

 For $1/2>\delta>0$, consider the interval $[a+\delta(b-a),b-\delta(b-a)]$, we can construct a ReLU network $r(x)$ such that
 $$
 r(x)=1\quad x\in[a+\delta(b-a),b-\delta(b-a)],\quad r(x)=0\quad x\notin[a,b]
 $$
 Define
 $$
 r(x)=ReLU(ReLU(1-ReLU(\frac{x-(b-\delta(b-a))}{\delta(b-a)}))-ReLU(1-ReLU(\frac{x-a}{\delta(b-a)})))
 $$
 Actually the last ReLU can be removed, but in order to be consistent with the structure of ReLU DNN, we write one ReLU function here to indicate one hidden layer.

With the above 1D example, now we can construct the ReLU network to approximate an indicator function on an d-dimensional cube. The key idea here is to do the procedure above one dimension by one dimension. Assume the cube is $[a_1,b_1]\times[a_2,b_2]\times\cdots\times[a_d,b_d]$ and suppose now we have constructed $r_k$ to approximate the indicator function on $[a_1,b_1]\times[a_2,b_2]\times\cdots\times[a_k,b_k]$. Now we define:
\begin{equation}
\begin{aligned}
&r_{k+1}(x_1,\dots,x_{k+1})\\
=&ReLU(ReLU(r_k(x_1,\dots,x_k)-ReLU(\frac{x_{k+1}-(b_{k+1}-\delta(b_{k+1}-a_{k+1}))}{\delta(b_{k+1}-a_{k+1})}))\\
&-ReLU(r_k(x_1,\dots,x_k)-ReLU(\frac{x_{k+1}-a_{k+1}}{\delta(b_{k+1}-a_{k+1})})))
\end{aligned}
\end{equation}
It is easy to check that $r_{k+1}$ can be used to approximate the indicator function on $[a_1,b_1]\times[a_2,b_2]\times\cdots\times[a_{k+1},b_{k+1}]$.

Now we proceed to the third step, where we combine all the indicator functions together with their weights. To do this, we need to consider again the $d+4$-width.

The first $d$ nodes in hidden layer are used to pass the information of the input. For the $d+1$-th and $d+2$-th nodes, we use them to store the information we get from $r_d$. And as for the last two nodes, they are used for the computation of $r_k$ functions. See \cite{lu2017expressive} for detailed graph.


%---------------------------------------------------------------------------------
\chapter{Numerical methods}
\label{chap:numerical-methods}
%---------------------------------------------------------------------------------
\section{Ordinary differential equation}
An ordinary differential equation (ODE) is an equation that contains derivatives of single independent variable functions. Examples of ODEs would be $\frac{df}{dx}=-x$ and $\frac{df}{dx} + \frac{dg}{dx} = 4x$, where $x$ is an independent variable, $f$ and $g$ are functions of $x$. In general, ODEs are used to describe changes. One of the simplest example is speed, change in distance travelled per unit time.

There are different ways to describe an ODE. The order of an ODE is the order of the highest derivative in an ODE. The ODE $\frac{d^3f}{dx^3} = -1$ has order 3. Other than the order, ODEs can be classified into linear or non-linear ODE. Linear ODEs are functions that can be expressed as linear combinations of derivatives, while non-linear ODEs cannot. The equation, $\frac{df}{dx} \frac{dg}{dx} = 1$, for example, is non-linear ODE; the equation $\frac{df}{dx} + \frac{dg}{dx} = 1$ is linear. Linear ODE can further be categorised into homogeneous and non-homogeneous ODEs. In a general linear ODE, $a_0(x)f(x)+a_1(x)f'(x)+...+a_n(x)f^{(n)}(x)+b(x) = 0$, the function is homogeneous if $b(x) = 0$ and non-homogeneous if $b(x) \neq 0$.

ODEs are widely used in biological problem. An example of biological problem would be the SEIR model. The SEIR model describes the spread of a disease in a population with time $t$. The model compartmentalises the population into groups of susceptibles ($S$), exposed ($E$), infectious ($I$) and recovered ($R$). Transitions of individuals from groups are described by a system of ODEs, with independent variable time $t$. It is defined as
\begin{align}
\label{eqn:SEIR-model}
    \frac{dS(t)}{dt} &= -\beta S(t)I(t),  \\
    \frac{dE(t)}{dt} &= \beta S(t)I(t) - \kappa E(t), \\
    \frac{dI(t)}{dt} &= \kappa E(t) - \gamma I(t), \\
    \frac{dR(t)}{dt} &= \gamma I(t) \label{eqn:SEIR-end}
\end{align}
where the parameters $\beta$, $\kappa$ and $\gamma$ are infection rate, incubation rate and recovery rate respectively. The incidence cases, $N$ at time, $t$, is defined as the difference between the current total number of infected and recovered and that of the previous time, according to the equation
\begin{equation}
    N(t) = I(t) + R(t) - I(t-1) - R(t-1)
\end{equation}
Figure \ref{fig:SEIR_simulation} shows the incidence cases simulated by the SEIR model.

\begin{figure}
    \includegraphics[width=0.95\columnwidth]{SEIR_simulation}
    \caption{Simulation of the SEIR model. The bar graph shows the incidence number, while the line graph shows the number of individuals in each group. All values shown are in fractions of the whole population.}
    \label{fig:SEIR_simulation}
\end{figure}

Another example is the modelling of excitable systems. The heart muscle cells and nerve cells are excitable systems. The Hodgkin \& Huxley model models the ionic currents across the cell membrane, to explain the action potential of nerve cells\cite{Keener2009}.
Their model is defined as
\begin{equation}
\label{model:HH}
    C_m \frac{dV}{dt} = -g_{\text{eff}}(V - V_{\text{eq}}) + I_{\text{app}} \\
\end{equation}
where $C_m$ is membrane capacitance, $V$ is membrane potential, $g_{\text{eff}}$ is sum of conductance for all ion channels, $V_{\text{eq}}$ is membrane resting potential and $I_{\text{app}}$ is applied current. Here, the independent variable is also time $t$. The value of potential is obtained by subtracting the membrane resting potential from the membrane potential, that is $V - V_{\text{eq}}$. Figure \ref{fig:AP_simulation} shows the simulated action potential of the Hodgkin and Huxley model.

\begin{figure}
\centering
    \includegraphics[width=0.5\columnwidth]{AP_simulation}
    \caption{The action potential in the Hodgkin and Huxley model, adapted from \cite{Keener2009}.}
    \label{fig:AP_simulation}
\end{figure}

\section{Initial value problem}
Given an ODE system, we would like to solve for the function. As an example, given an ODE $\frac{df(x)}{dx} = -f(x)$, we would like to know the function $f$. To solve ODE systems, initial conditions are required to pinpoint the solution of interest, as there are infinitely possible solutions to an ODE. The ODEs, together with the initial conditions, set up the initial value problem.

A general form of an initial value problem is as follows:
\begin{align}
\label{eqn:initial_value_start}
    y'&=f(x,y)\\
    y(x_0) &= y_0 \label{eqn:initial_value}
\end{align}
for $x \in [x_0, X_M]$, where Eq.~\eqref{eqn:initial_value} describes the initial condition. For example, initial values of $S(0)=0.8, E(0)=0, I(0)=0.1$ and $R(0)=0$ are used to solve the SEIR model in Figure \ref{fig:SEIR_simulation}.

\section{Numerical method}
Many ODE systems cannot be solved analytically. Some examples would be the given SEIR Model, Eqs.~\eqref{eqn:SEIR-model}-\eqref{eqn:SEIR-end} and Hodgkin \& Huxley Model, Eq.~\eqref{model:HH}. Solutions to such models have to be estimated by numerical methods.

Throughout the report, the following notation will be used:
\begin{itemize}
    \item $y_n$ - numerical approximation of $y(x_n)$
    \item $y(x_n)$ - analytical solution at mesh point $x_n$
    \item $x_n$ - mesh points of defined range, where
        \begin{align}
            x_n &= x_0 + nh\\
            h &= \frac{(X_M - x_0)}{N}
        \end{align}
        for $n = 0,\dots, N$
    \item $h$ - step size
\end{itemize}

\subsection{One-step methods}
\label{subsec:one-step-method}
The simplest numerical method in solving ODEs is Euler's explicit method. Intuitively, Euler's explicit method assumes for a small step size $h$, the solution of the function can be estimated by its tangent line. As seen in Figure \ref{fig:Euler_method}, following the tangent line at point $x_0$, the difference between the $y$ value for point $x_0$ and point $x_1$ is $hF(x_0,y_0)$. So, the estimated value of $y_1$ is $y_0 + hF(x_0,y_0)$, where $y_0$ is the value of solution curve at $x_0$.

\begin{figure}
\centering
    \includegraphics[width=0.5\columnwidth]{Euler_method_img}
    \caption{Intuition of Euler's method, adapted from \cite{CalcWorkshop2019}.}
    \label{fig:Euler_method}
\end{figure}

Euler's explicit method has the following definition,
\begin{equation}
    y_{n+1} = y_n + hf(x_n,y_n)\\
\end{equation}
Starting from the initial value, the solution at the subsequent mesh point is estimated to follow a straight line with gradient as given.

The implementation is as follows:

\begin{algorithm}[H]
    \SetAlgoLined
    \SetKwInOut{Output}{Output}
    \Output{mesh points and its numerical solution}
    solution = [initial value]\;
    meshpoints = [starting point]\;
    \For{$n$ \text{from} 1 \text{to total number of mesh points}}{
    solution.append(solution[$n-1$] + step size $\times$ $f$(meshpoints[$n-1$], solution[$n-1$])\;
    meshpoints.append(meshpoints[$n-1$] + $n$ $\times$ step size)\;
    }
    \Return{meshpoints and solution}\;
\caption{Euler's explicit method}
\end{algorithm}

A vector containing the numerical solution and a vector of mesh points are initialised with their respective initial values. At each iteration, the mesh point and the numerical solution at the mesh point are calculated. The series of mesh points and its solution are returned. The mesh points are returned for consistency of the software, where all methods return the same outputs. Adaptive methods does not have fixed step sizes, thus it is important to return the mesh points.

Truncation error of the numerical methods is defined to be the difference between the exact solution and the numerical solution, assuming the exact solution at the previous mesh point is known. We have that the truncation error for Euler's explicit method is

\begin{equation}
\label{eqn:trun_err_def}
    T_n \defeq \frac{y(x_{n+1}) - y(x_{n})}{h} - f(x_n, y(x_n))
\end{equation}
According to Taylor's series expansion, we have
\begin{equation}
\label{eqn:Taylor_expansion}
    y(x_n + h) = y(x_n) + hy'(x_n) + \frac{1}{2}hy''(\xi_n)
\end{equation}
for some $\xi_n \in (x_n, x_{n+1})$. Substitute (\ref{eqn:Taylor_expansion}) to (\ref{eqn:trun_err_def}), noting that $f(x_n, y(x_n)) = y'(x_n)$, we get
\begin{equation}
    T_n = \frac{1}{2}hy''(\xi_n)
\end{equation}
Therefore, the truncation error for Euler's explicit method varies linearly with the step size.

A test problem
\begin{align}
\label{eqn:example_model}
    f(x,y) &= -y \\
    y(0) &= 1 \label{eqn:example-end}
\end{align}
for $x \in [0, 5]$ is used throughout this report to check that the implementation is in line with the theory. The analytical solution to this problem is $y = e^{-x}$. Some examples of solutions to the given test problem Eqs.~\eqref{eqn:example_model}-\eqref{eqn:example-end} can be found via this link: \href{https://nbviewer.jupyter.org/github/FarmHJ/numerical-solver/blob/main/examples/solver_convergence.ipynb}{\underline{\emph{test problem notebook}}}. This Jupyter Notebook is a tutorial example that demonstrates the use of the fully tested package I developed on the test problem.

With the test problem Eqs.~\eqref{eqn:example_model}-\eqref{eqn:example-end} as a reference, the truncation error versus step size graph is shown in Figure \ref{fig:Euler_explicit_error_behaviour}. The computation and comparison of one-step methods and their truncation error can be found at this link: \href{https://nbviewer.jupyter.org/github/FarmHJ/numerical-solver/blob/main/examples/Onestep_methods_convergence.ipynb}{\underline{\emph{error behaviour of one-step methods}}}. The truncation error is computed by taking the difference between the exact solution and the numerical solution at a randomly chosen $x$ value of 3. The truncation error is computed for solutions obtained by using different step sizes. Figure \ref{fig:Euler_explicit_error_behaviour} is plotted in logarithmic scale for both variables. It can be observed that $\log |T_n|$ increases linearly with $\log h$. The line gradient of 1 shows that $|T_n| \propto h$, which matches the theoretical prediction.

\begin{figure}
    \includegraphics[width=0.95\columnwidth]{Euler_explicit_error_behaviour}
    \caption{Truncation error of Euler's explicit method for various step sizes. The test problem Eqs.~\eqref{eqn:example_model}-\eqref{eqn:example-end} is solved with Euler's explicit method at different step sizes. The error is the absolute difference between exact solution and numerical solution at $x=3$.}
    \label{fig:Euler_explicit_error_behaviour}
\end{figure}

Other than Euler's explicit method, the other one-step methods implemented are Euler's implicit method, the trapezium rule method and the four-stage explicit Runge-Kutta method. Euler's implicit method is defined to be
\begin{equation}
    y_{n+1} = y_n + hf(x_{n+1},y_{n+1}),\\
\end{equation}
while the trapezium rule method is
\begin{equation}
    y_{n+1} = y_n + \frac{1}{2}h[f(x_n,y_n) + f(x_{n+1},y_{n+1})].\\
\end{equation}
The definition of the four-stage Runge-Kutta method is
\begin{equation}
    y_{n+1} = y_n + \frac{1}{6}h(k_1 + 2k_2 + 2k_3 + k4)
\end{equation}
where
\begin{align}
    k_1 &= f(x_n, y_n) \\
    k_2 &= f(x_n + \frac{1}{2}h, y_n + \frac{1}{2}hk_1) \\
    k_3 &= f(x_n + \frac{1}{2}h, y_n + \frac{1}{2}hk_2) \\
    k_4 &= f(x_n + h, y_n + hk_3).
\end{align}

Using the same definition for truncation error as in Eq.~\eqref{eqn:trun_err_def}, we have that the truncation error of Euler's implicit method, the trapezium rule method and the four-stage Runge-Kutta method are $T_n = -\frac{1}{2}hy''(\xi_n)$ for $\xi_n \in (x_n, x_{n+1})$, $T_n = -\frac{1}{12}h^2y^{(3)}(\xi_n)$ for $\xi_n \in (x_n, x_{n+1})$ and $T_n = \mathcal{O}(h^4)$ respectively.

These methods are tested on the test problem Eqs.~\eqref{eqn:example_model}-\eqref{eqn:example-end}, and the truncation error behaviour graph is shown in Figure \ref{fig:onestep_error_behaviour}. Since the graph is plotted in logarithmic scale, the gradient of the lines shows the order of accuracy of the methods. The gradient for Euler's explicit, Euler's implicit, the trapezium rule method and the four-stage Runge-Kutta method are 1, 1, 2 and 4 respectively. The gradients on the graph match the theoretical behaviour of truncation error. In Figure \ref{fig:onestep_error_behaviour}, the truncation error behaviour of Euler's implicit method overlaps that of Euler's explicit method because both have the same order of accuracy.

\begin{figure}
    \includegraphics[width=0.95\columnwidth]{onestep_error_behaviour}
    \caption{Truncation error of Euler's explicit method, Euler's implicit method, the trapezium rule method and the four-stage Runge-Kutta method for various step sizes. The test problem Eqs.~\eqref{eqn:example_model}-\eqref{eqn:example-end} is solved with Euler's explicit method, Euler's implicit method, the trapezium rule method and the four-stage Runge-Kutta method at different step sizes. The error is the absolute difference between exact solution and numerical solution at $x=3$. The line for Euler's implicit method overlaps the line for Euler's explicit method because both have the same order of accuracy. The truncation errors follow theoretical prediction.}
    \label{fig:onestep_error_behaviour}
\end{figure}

\subsection{Predictor-corrector methods}
\label{sec:predictor-corrector}
For implicit one-step methods, the fixed point iteration algorithm is used to obtain the numerical solution. Implicit functions cannot be solved with a general method, therefore a general algorithm, the fixed point iteration, is used to estimate the solution to the implicit functions. Initial guesses for this algorithm are taken as the numerical solution at previous mesh point. However, in practice, the computational cost of the fixed point algorithm is too high. The predictor-corrector method suggests a more carefully chosen initial guess for the implicit methods. An explicit numerical method is used as a predictor for the initial guess of an implicit method. The initial guess is then used for the iterations in the algorithm to solve an implicit function. The implicit method that refines the solution is known as the corrector method.

The Euler-Trapezoidal method is a predictor-corrector method that uses Euler's explicit method as the predictor and the trapezium rule method as the corrector. In this implementation, the trapezium rule method corrector is iterated until a set of conditions are satisfied. The conditions are set to be the difference between the current iteration and the previous iteration is lesser than a given threshold value or the number of iterations exceeds a certain amount. Figure \ref{fig:predictor_corrector_error_behaviour} shows the graph of $\log |T_n|$ against $\log h$ for the Euler-Trapezoidal method.

\begin{figure}
    \includegraphics[width=0.95\columnwidth]{predictor_corrector_error_behaviour}
    \caption{Error of Euler-Trapezoidal method for various step sizes. The test problem Eqs.~\eqref{eqn:example_model}-\eqref{eqn:example-end} is solved with Euler-Trapezoidal method at different step sizes. The error is the absolute difference between exact solution and numerical solution at $x=3$.}
    \label{fig:predictor_corrector_error_behaviour}
\end{figure}

\subsection{Adaptive method}
\label{sec:adaptive-method}
In some types of problem, the solution to the problem exhibits a behaviour where step sizes have to be sufficiently small to obtain a stable solution. In other words, the solution changes rapidly as a function of the independent variable in some regions of the solution domain. Such problems are called stiff problems. In order to obtain a stable solution for stiff problems, the computational cost would be high. Moreover, such a stable solution would have resolution higher than required for practical purposes in some regions.

Adaptive methods try to achieve the desired accuracy with low computational cost. The main idea of an adaptive method is to control the precision at each mesh point and from there assumes the precision of the solution is within the desired range. To achieve this, the error at each mesh point is estimated. If the error is larger than a threshold value, a smaller step size is chosen. These steps are repeated until the error is smaller than the given threshold value.

The adaptive methods implemented in my package are based on the BS23 (Bogacki and Shampine) and RKF45 (Runge-Kutta-Fehlberg) method, referenced from \cite{Forsythe1977}. A lower order method is used to estimate the solution, while a higher order method is used to estimate the error.

Absolute tolerance and relative tolerance are used in the implementation of the adaptive methods to control the error. Similarly, the implemented adaptive method is used to solve the test problem Eqs.~\eqref{eqn:example_model}-\eqref{eqn:example-end} and tested for convergence. Two comparisons were made, one based on absolute tolerance and the other on relative tolerance. As shown in Figure \ref{fig:ode45_absolute_sum_error_behaviour} and Figure \ref{fig:ode45_relative_sum_error_behaviour}, the error of the method is smaller for smaller tolerance, regardless of absolute tolerance or relative tolerance.

\begin{figure}
    \includegraphics[width=0.95\columnwidth]{ode45_absolute_sum_error_behaviour}
    \caption{Total error of RKF45 method for various absolute tolerance values. The test problem Eqs.~\eqref{eqn:example_model}-\eqref{eqn:example-end} is solved with RKF45 method at different absolute tolerance values. The relative tolerance is fixed at $1^{-15}$. The error is the sum of absolute difference between exact solution and numerical solution at all mesh points.}
    \label{fig:ode45_absolute_sum_error_behaviour}
\end{figure}

\begin{figure}
    \includegraphics[width=0.95\columnwidth]{ode45_relative_sum_error_behaviour}
    \caption{Total error of RKF45 method for various relative tolerance values. The test problem Eqs.~\eqref{eqn:example_model}-\eqref{eqn:example-end} is solved with RKF45 method at different relative tolerance values. The absolute tolerance is fixed at $1^{-15}$. The error is the sum of absolute difference between exact solution and numerical solution at all mesh points.}
    \label{fig:ode45_relative_sum_error_behaviour}
\end{figure}


\documentclass{article}
\usepackage{amsmath}
\usepackage{geometry}
\usepackage[utf8]{inputenc}
\usepackage{enumitem}

\setlength{\parindent}{0pt}
\setlength{\parskip}{1em}

\begin{document}

\section{Chebyshev's inequality}
For real numbers $a_1 \geq \dots \geq a_n$ and $b_1 \geq \dots \geq b_n$
$$\frac{1}{n} \sum_{i=1}^{n} a_ib_i
\geq
\left(\frac{1}{n}\sum_{i=1}^{n}a_i\right)
\left(\frac{1}{n}\sum_{i=1}^{n}b_i\right)
\geq
\frac{1}{n} \sum_{i=1}^{n} a_ib_{n+1-i} $$

\begin{enumerate}[resume]

	\item % Võrratused 25
	Prove Chebyshev's inequality by proving the following equation.

	Prove that
	$$
	\frac{1}{2} \sum_{i=1}^{n} \sum_{j=1}^{n}(a_i-a_j)(b_i-b_j)
	=
	n \sum_{k=1}^{n}a_k b_k - \sum_{k=1}^{n}a_k \sum_{k=1}^{n}b_k
	$$
	When does the equality hold?

	\item
	For triangle with angles $\alpha,\beta,\gamma$ and opposite sides with lengths $a,b,c$ respectively prove that
	$$\frac{\pi}{3} \leq \frac{\alpha a + \beta b + \gamma c}{a+b+c} \leq \frac{\pi}{2} $$

	\item % nesbitt inequality
	Prove that for positive real numbers
	$$ \frac{a}{b+c}+\frac{b}{a+c}+ \frac{c}{a+b} \geq \frac{3}{2}$$

	\item % https://artofproblemsolving.com/community/c1642h1004527s3_chebyshevs_inequality
	Prove that for positive real numbers
		$$ \frac{(b+c)(b^2+c^2)}{a} + \frac{(a+c)(a^2+c^2)}{b}+\frac{(a+b)(a^2+b^2)}{c}
		\geq
		4 (a^2+b^2+c^2)
		$$

	\item %https://artofproblemsolving.com/community/c1642h1004527s3_chebyshevs_inequality

	Prove that for positive real numbers
	$$\frac{x}{x^2+yz}+\frac{y}{y^2+xz}+\frac{z}{z^2+xy} \leq \frac{x^2+y^2+z^2}{2xyz} $$

	\item % vv2010-3
	Triangle has angles $\alpha,\beta,\gamma$, perimeter $2p$ and radius of circumcircle $R$. Prove that

	$$\cot^2 \alpha + \cot^2 \beta + \cot^2 \gamma  \geq 3 \left(\frac{9R^2}{p^2}-1\right)$$
	When does the equality hold?

\end{enumerate}

\newpage
\section{Rearrangement inequality}

For real numbers $a_1 \geq \dots \geq a_n$ and $b_1 \geq \dots \geq b_n$ and a permutation $a'_1,\dots,a'_n$ of $a_1,\dots,a_n$

$$a_1b_1+\dots+a_nb_n \geq a'_1b_1 + \dots + a'_nb_n \geq a_1b_n+ \dots+ a_nb_1 $$

\begin{enumerate}[resume]
	\item
	Prove rearrangement inequality. When does the equality hold?

	\item
	For real numbers $a_1,\dots,a_n$ and for its permutation $a'_1,\dots,a'_n$, prove that
	$$\sum_{i=1}^{n} \frac{a_i}{a'_i} \geq n $$

	\item
	Prove the mean inequalities by using rearrangement inequality
	$$	SM \geq AM \geq GM \geq HM$$

	\item
	Prove Cauchy inequality using rearrangement inequality.

	\item
	Prove Chebyshev's inequality using rearrangement inequality.

\end{enumerate}

\end{document}


\subsection{Gaussian Mixture Models}
\subsubsection{Mixture models}

We have a latent variable which is part of the process

The variable is distributed according to parametric distribution, but parameters are different for differnet latent classes.

There are \(K\) latent classes, and so \(K\) sets of parameters.

The population is weighted into the \(K\) classes.

We have a distribution, but we have different parameters for the distribution for different populations.

For example we could observe the height of men and women, where both are normally distributed but with different parameters.

Where there is a normal distribution, this is a Gaussian mixture model.

If there is more than one variable to observe, this is a multivariate Gaussian mixture model.

\subsubsection{Gaussian Mixture Models (GMM)}

In a Gaussian Mixture Model each non latent variable has a normal distriubtion with a mean and variance. For multiple variables there is a covariance matrix.


\subsection{Introduction}

If we have a partitioned linear regression model:

\(\mathbf y=\mathbf X\theta+\mathbf Z\beta+\mathbf \mu\)

Use the annihilator matrix:

\(M_X\mathbf y=M_X\mathbf X\theta+M_X\mathbf Z\beta+M_X\mathbf \mu\)

\(M_X\mathbf y=M_X\mathbf Z\beta+M_X\mathbf \mu\)

We can then estimate \(\beta \).

Frisch-Waugh-Lovell theorem says that this is the same estimate as the original regression.


\section{Principle of Least Action; Noether's Theorem}
Consider a particle in $\mathbb R^3$ with kinetic energy and potential energy denotes by $T,V$ respectively.
Define the Lagrangian to be $L(\underline{x},\underline{\dot{x}},t)=T-V$ and the action
$$S[\underline{x}]=\int_{t_1}^{t_2}L\,\mathrm dt$$
Hamilton's Principle (or Principle of Least Action) postulates that the motion of the particle is a stationary point of the action functional $S$, i.e. $\underline{x}$ satisfies the Euler-Lagrange equations with the integrand being $L$.
\begin{example}
    Take the kinetic energy to be $T=m|\underline{\dot{x}}|^2/2$ and $V=V(\underline{x})$, then the Euler-Lagrange equation gives
    $$\frac{\mathrm d}{\mathrm dt}\frac{\partial f}{\partial \dot{x}_i}-\frac{\partial L}{\partial x_i}=0$$
    for any $i$.
    This simplifies to
    $$m\ddot{x}_i=-\frac{\partial V}{\partial x_i}$$
    which is just Newton's Second Law.
\end{example}
\begin{example}[Central Force in Two Dimensions]
    We have
    $$L=\frac{1}{2}m(\dot{r}^2+r^2\dot{\theta}^2)-V(r)$$
    So the Euler-Lagrange equations give
    $$\frac{\mathrm d}{\mathrm dt}\frac{\partial L}{\partial \dot{r}}-\frac{\partial L}{\partial r}=0=\frac{\mathrm d}{\mathrm dt}\frac{\partial L}{\partial \dot{\theta}}-\frac{\partial L}{\partial \theta}=\frac{\mathrm d}{\mathrm dt}\frac{\partial L}{\partial \dot{\theta}}$$
    The first observation is that the rightmost expression vanishes, hence
    $$mr^2\dot{\theta}=\frac{\partial L}{\partial\dot{\theta}}=\text{const.}$$
    which is just the conservation of angular momentum.
    Now $\partial L/\partial t=0$, so by our previous discussion of the first integrals,
    $$\dot{r}\frac{\partial L}{\partial \dot{r}}+\dot{\theta}\frac{\partial L}{\partial\dot{\theta}}-L=\text{const.}$$
    So by substitution and a bit of simplification, we are left with
    $$\frac{1}{2}m\dot{r}^2+\frac{1}{2}mr^2\dot{\theta}^2+V(r)=\text{const.}$$
    Note that the left hand side is precisely $L+2V=E+V$, so this can be viewed as the conservation of energy.
\end{example}
\begin{example}[Configuration Space and Generalised Coordinates]
    Consider $N$ particles in $\mathbb R^3$.
    We can put the information of the coordinates of the particles as a $3N$-tuple, which allows us to take the $N$ particles as a single particle in $\mathbb R^{3N}$.
    The coordinates of all $N$ particles ordered in the obvious way is called a generalised coordinate, and the particle system described in $\mathbb R^{3N}$ makes it a configuration space.
    So the motion of the particles can be characterised by a path (i.e. the motion of one single particle) in $\mathbb R^{3N}$, i.e. $t\mapsto (q_i,\dot{q}_i,t)$ where $q_i$ is the $i^{th}$ generalised coordinate, $i=1,\ldots,3N$.
    The Lagrangian considered this way is then $L=L(q_i,\dot{q}_i,t)$ which we can solve the corresponding Euler-Lagrange equation for.
    There is a perspective of classical dynamics that takes this point of view to solve for the motions of the system of particles.
\end{example}
Consider the functional
$$F[\underline{y}]=\int_\alpha^\beta f(y_i,y_i^\prime,x)\,\mathrm dx,i=1,\ldots,n$$
Consider a $1$-parameter family of transformations
$$y_i(x)\mapsto Y_i(s,x),s\in\mathbb R$$
We call this a continuous symmetry (or simply symmetry) of a Lagrangian if
$$\frac{\mathrm d}{\mathrm ds}f(Y_i(s,x),Y^\prime(s,x),x)=0$$
With these set-ups, we have
\begin{theorem}[Noether's Theorem, simple version]
    Given a continuous symmetry $Y_i(s,x)$ of $f$ with $Y_i(x,0)=y_i(x)$ for all $i$, the quantity
    $$\sum_i\frac{\partial f}{\partial y_i}\left.\frac{\partial Y_i}{\partial s}\right|_{s=0}$$
    is a first integral of the Euler-Lagrange equations (i.e. is constant).
\end{theorem}
\begin{proof}
    Below by $f$ we mean $f(Y_i(s,x),Y^\prime(s,x),x)$.
    Summation convention is used.
    \begin{align*}
        0&=\left.\frac{\mathrm df}{\mathrm ds}\right|_{s=0}\\
        &=\frac{\partial f}{\partial y_i}\left.\frac{\partial Y_i}{\partial s}\right|_{s=0}+\frac{\partial f}{\partial y_i^\prime}\left.\frac{\partial Y_i^\prime}{\partial s}\right|_{s=0}\\
        &=\left.\frac{\partial Y_i}{\partial s}\right|_{s=0}\frac{\mathrm d}{\mathrm dx}\frac{\partial f}{\partial y_i}+\frac{\partial f}{\partial y_i^\prime}\left.\frac{\mathrm d}{\mathrm dx}\frac{\partial Y_i}{\partial s}\right|_{s=0}\\
        &=\frac{\mathrm d}{\mathrm dx}\left( \frac{\partial f}{\partial y_i}\left.\frac{\partial Y_i}{\partial s}\right|_{s=0} \right)
    \end{align*}
    The theorem follows.
\end{proof}
\begin{example}
    Consider
    $$f=\frac{1}{2}(y^\prime)^2+\frac{1}{2}(z^\prime)^2-V(y-z)$$
    which is the Lagrangian of a potential that depends only on difference of the components.
    Consider $Y=y+s, Z=z+s$, then $Y^\prime=y^\prime$ and $Z^\prime=z^\prime$ and $V(Y-Z)=V(y-z)$, so indeed it is a continuous symmetry.
    Noether's Theorem then implies that
    $$\left.\left( \frac{\partial f}{\partial y^\prime}\frac{\partial Y}{\partial s}+\frac{\partial f}{\partial z^\prime}\frac{\partial Z}{\partial s} \right)\right|_{s=0}=y^\prime+z^\prime$$
    is constant.
    This is just saying the conservation of momentum in the $y+z$ direction.
\end{example}
\begin{example}
    Back to our previous discussion on central force, then the transformation $\Theta=\theta+s,R=r$ is certainly a continous symmetry.
    Therefore we have the conserved quantity
    $$\left( \frac{\partial L}{\partial\dot{\theta}}\frac{\partial\Theta}{\partial s}+\frac{\partial L}{\partial\dot{r}}\frac{\partial R}{\partial s} \right)_{s=0}=mr^2\dot{\theta}$$
    which is just the magnitude of angular momentum.
    In general, isotopy of space gives rise to rotational invariants of the Lagrangian.
\end{example}


\section{Context-free grammar}

Context-free grammars are part of the Chomsky hierarchy \cite{grammars_hierarchy, grammars} and can be defined as a quadruple $G = (N, \Sigma, P, S)$ that consists of the following components:

\begin{itemize}
    \item[$-$] $N$ is a finite set of \textit{non-terminal} variables.
    \item[$-$] $\Sigma$ is a finite set of \textit{terminal} variables.
    \item[$-$] $P$ is a finite set of \textit{production rules}, also called \textit{rewrite rules}, it specifies a symbol substitution that can be recursively performed to generate new symbol sequences. In an unrestricted grammar, which is the most general form of the Chomsky hierarchy, a production is of the form $u \to v$ where $u$ and $v$ are arbitrary strings of \textit{terminals} and \textit{non-terminals}.
    \item[$-$] $S \in N$ is a distinguished symbol that is the \textit{start symbol}, also called the \textit{sentence symbol}.
\end{itemize}

The language $L(G)$ is the language generated by the grammar G \cite{intro_automata}.
To generate a string of the language, one begins with a string consisting of only a single start symbol, and then successively applies the rules (any number of times, in any order) to rewrite this string.
This stops when we obtain a string containing only terminals.
The language consists of all the strings that can be generated in this manner.

The same process can be applied backward to check weither a given string can be generated by the grammar or not.

\subsection{Description of the Chomsky normal-form}

Here is an example of a context-free grammar in the Chomsky normal-form:

\begin{align*}
&S \to AB|BC\\
&A \to BA|a\\
&B \to CC|b\\
&C \to AB|a
\end{align*}

The grammar can be described as a finite set of production rules, in those productions we can find four types of symbols:

\begin{itemize}
    \item[$-$] The `$\to$' symbol is what seperates the left-hand from the right-hand(s) of the rules.
    \item[$-$] In this grammar every single uppercase letter is a \textit{non-terminal} variable.
    \item[$-$] The \textit{terminals} are primitive symbols, they can only be found on the right-hand of a production.
    \item[$-$] A left-hand is constitued of one \textit{non-terminal}, it can relate to one or several right-hands seperated by the `$|$' symbol. A right-hand is constitued either by one \textit{terminal} or a pair of \textit{non-terminals}.
\end{itemize}

Here S, A, B, and C are \textit{non-terminals}, a and b are \textit{terminals} and S is the \textit{start symbol}.

\subsection{Convert grammar to Chomsky normal-form}

Sometimes it is needed to convert a context-free grammar to the Chomsky normal-form.
Later in this report is presented an algorithm that follows the next pseudo-code in order to automaticly convert a given context-free grammar into the Chomsky normal-form.
\\
\\
For example this is how to convert the following grammar:

\begin{align*}
&S \to S+P|P\\
&P \to P*C|C\\
&C \to (S)|0|1\\
\end{align*}

\begin{enumerate}
    \item The first thing to do is to eliminate the \textit{start symbol} from the right-hands of the grammar by replacing the \textit{start symbol} by a new one (some labels have been modified):
        \begin{align*}
        &S \to A\\
        &A \to A+B|B\\
        &B \to B*C|C\\
        &C \to (A)|0|1\\
        \end{align*}

    \item Then the goal is to replace the \textit{terminals} that are not the only symbols on the right-hand by a new \textit{non-terminal}:
        \begin{align*}
        &S \to A        &P \to +\\
        &A \to APB|B    &M \to *\\
        &B \to BMC|C    &L \to (\\
        &C \to LAR|0|1  &R \to )\\
        \end{align*}

    \item The next step is to delete the right-hands with more than 2 \textit{non-terminals} by adding \textit{non-terminals} that will relate to a pair:
        \begin{align*}
        &S \to A        &F \to AR\\
        &A \to AD|B     &P \to +\\
        &B \to BE|C     &M \to *\\
        &C \to LF|0|1   &L \to (\\
        &D \to PB       &R \to )\\
        &E \to MC\\
        \end{align*}

    \item At last it is needed to eliminate the unit rules, those are the right-hands in which there is one \textit{non-terminal} alone:
        \begin{align*}
        &S \to AD|BE|LG|0|1 &F \to AR\\
        &A \to AD|BE|LG|0|1 &P \to +\\
        &B \to BE|LG|0|1    &M \to *\\
        &C \to LF|0|1       &L \to (\\
        &D \to PB           &R \to )\\
        &E \to MC\\
        \end{align*}

\end{enumerate}

\section{CYK algorithm}

The CYK algorithm is a parser for context-free grammar, it is named after its inventors, John Cocke, Daniel Younger and Tadao Kasami.
The algorithm takes a string and a context-free grammar as input and determines if the string is part of the language of the grammar or not.
It uses the backward process explained in the presentation of section 1.1.

Context-free grammar parsers can be used for example in computer sciences to check code structure in compilers or in biology for DNA and RNA strings analysis.

\section{Diverse development approaches}

The goal of the assignment is to implement CYK parsers using different development approaches in order to compare the efficiency of those.
In this section the three used development methods are briefly presented.

\subsection{Divide and conquer}

Divide and conquer is an algorithmic strategy that consists in recursively breaking down a problem into several others, and do so until the problems become easy enough to solve.
The obtained results are then combined in order to solve the initial problem.
\\
\\
Those are the steps of the general structure of that strategy:

\begin{enumerate}
    \item Divide the problem instance given as input into smaller instances.
    \item Recursively compute the results for those smaller instances.
    \item Assemble the recursively obtained results into a result for the entire input and return it.
\end{enumerate}

\subsubsection{Recurrence relation}

It is possible to find a function that satisfies the recurrence relation of the divide and conquer strategy.
Let's suppose that a recursive algorithm divides a problem of size $n$ into $a$ sub-problems, where each sub-problem is of size $\dfrac{n}{b}$
Suppose also that $g(n)$ is the total time needed to create the sub-problems and combine their results.

Then we can define $f(n)$, being the number of operations needed to solve the problem of size $n$, as:
$$
f(n) = a * f(\dfrac{n}{b}) + g(n)
$$

\subsubsection{Master theorem}

The Master theorem for the divide and conquer recurrences provides an asymptotic analysis of the complexity, using the Big O notation.
It allows one to solve $f(n)$ depending on three different cases.
\\
\\
Now that we have the function $f(n)$ we suppose that $g(n) = \Theta(n^d)$.
\\
\\
Then the master theorem is:
$$
f(n) =
\begin{cases}
    \Theta(n^d) &\text{if } a < b^d\\
    \Theta(n^dlog(n)) &\text{if } a = b^d\\
    \Theta(n^{log_b(a)}) &\text{if } a > b^d\\
\end{cases}
$$

Here the first case is when it is possible to ignore the solving of the subproblems before the creation and combination of the sub-problems. The running time is then the time needed to create and combine the sub-problems, $g(n) = \Theta(n^d)$.

The second case is when it is not possible to ignore anything.

The final case is when it is possible to ignore the creation and combination of the sub-problems. The running time is then the time needed to solve $a^{log_b(n)}$ subproblems of size 1.

\subsection{Dynamic programing}

Dynamic programing is an algorithmic paradigm that solves a given complex problem by breaking it down into subproblems and storing the results of those subproblems to avoid computing the same results again.

\subsubsection{Top-down approach}

The top-down approach is similar to the divide and conquer approach, the initial problem is recursively broken down into several sub-problems.
The difference is that in this approach a global variable memorize every computed result.
In that way when the program runs into a known problem the memorized result is directly returned.
The program generally computes less operations and goes less deep in the recursion than the divide and conquer method.

\subsubsection{Bottom-up approach}

The bottom-up approach consists in solving the simplest sub-problems first, memorizing every computed result, and then going up solving bigger and bigger sub-problems until it can solve the initial problem.
This approach is a way to avoid recursion, which saves some memory, but on the other hand it can sometimes solve sub-problems that will never be used to solve the initial problem, which is a waste of time.

A good analogy for the bottom-up design are building blocks, indeed you have to start assembling the most little pieces together in order to construct bigger ones that you will then assemble to get the final result.


\section{Continuity Properties}
These properties should look very similar to the properties of limits.
Let $f$ and $g$ be continuous functions at $c$.
\begin{align*}
	\textbf{Sum and Difference Rule: }& f \pm g \text{ is continuous at } c. \\
	\textbf{Product Rule: }& f \cdot g \text{ is continuous at } c. \\
	\textbf{Constant Multiple Rule: }& kf \text{ is continous at } c \text{ for all real } k. \\
	\textbf{Quotient Rule: }& \frac{f}{g} \text{ is continuous at } c \text{ as long as the value of the extended function of } \\
		g \text{ at } c \text{ is not } 0. \\
	\textbf{Composition Rule: }& f \circ g \text{ is continuous at } c \text{ if } f \text{ is continuous at } g(c). \\
	\textbf{Absolute Value Rule: }& \abs{f} \text{ is continuous at } c.
\end{align*}

The following types of functions are continuous on their domains
\begin{itemize}
	\item Polynomials
	\item Rational functions, except where the denominator is 0
	\item Trigonometric functions where defined
\end{itemize}

\begin{example}
	Show that the following function is continuous.
	\begin{equation*}
		f(x) = \tan{\left(\frac{x^2}{x^2+4}\right)}.
	\end{equation*}
\end{example}
\begin{answer}
	We can write $f$ as the composition of $\tan{x}$ and $\frac{x^2}{x^2+4}$.
	$\tan{x}$ is continuous on its domain because it is a trigonometric function.
	The only points not in its domain are $(2n+1)\frac{\pi}{2}$, where $n$ is an integer.
	$\frac{x^2}{x^2+4}$ is a rational function, but it's denominator is never $0$, so it is continuous over all real numbers.
	Now, we just need to check that all points in the range of $\frac{x^2}{x^2+4}$ are in the domain of $\tan{x}$.
	The range of $\frac{x^2}{x^2+4}$ is $[0,1)$.
	None of the points in this interval are not in the domain of $\tan{x}$, so the composition is continuous over all real numbers.
\end{answer}


\section{Spectral Theory}

\subsection{Exercise 1}
(a) We can re-express $h$ as a linear combination of its eigenvectors,
from which we can see that $A^n$ causes all of these components to go to 0.

(b) Same as in part (a), except all of the components now go to $\infty$.

\subsection{Exercise 2}
$A (A^N f) = A(a^N f + Na^{N - 1} h) = a^{N+1} f + a^N h + Na^N h$.

\subsection{Exercise 3}
Suppose $q(A) = \sum_{i = 0}^N q_i A^i$. Then $q_i A^i f = q_i a^i f + q_i i a^{i-1} h$ by
Exercise 2. From the linearity of the derivative it then follows that $q(A) f = q(a) f + q'(a) h$.

\subsection{Exercise 4}
Applying Lemma 9 to $p_1 ... p_k$ and $p_{k+1}$ gives the desired result.

\subsection{Exercise 5}
$(A - aI)^d x = 0 \implies A (A - aI)^d x = 0 \implies (A - aI)^d (Ax) = 0 \implies Ax \in N_d$.

\subsection{Exercise 6}
Since $m_A$ divides the characteristic polynomial of $A$ (by definition of $d_i$), $m_A (A) = 0$ by Cayley-Hamilton.
Suppose there is some polynomial $q(A) = 0$ with $\deg(q) < \deg(m_A)$.
The roots of $q$ must contain all of the eigenvalues of $A$, since for any eigenvector $h$ we have that
$q(A)h = q(a_h)h$ (where $a_h$ denotes the eigenvalue associated with $h$). Thus, the roots of $q$ can
only differ in multiplicity from $m_A$. However, if any root of $q$ has multiplicity $d_i' < d_i$, then
we can choose an element $x \in N_{d_i}$ such that $q(A)x \neq 0$, which is a contradiction (since
$N_{d_i'} \subset N_{d_i}$).

\subsection{Exercise 7}
The columns of $A$ are $Ax_i$.

\subsection{Exercise 8}
By induction.

\subsection{Exercise 9}
The minimal polynomial of $A$ divides the minimal polynomial of $A^{\top}$, and vice versa,
so they must be the same. Thus, the indices of each eigenvalue of $A$ and $A^{\top}$ must be the same,
which means we can apply Theorem 12 to see that they are similar.

\subsection{Exercise 10}
$(\xi^{(i)}, x) = \sum k_j (\xi^{(i)}, x^{(j)}) = k_i (\xi^{(i)}, x^{(i)})$ since
$(\xi^{(i)}, x^{(j)}) = 0$ by Theorem 17.


\chapter{Mathematical Tools}

This appendix covers some of the basic mathematics used in econometrics. We briefly discuss the properties of summation operators, study the properties of linear and some nonlinear equations, and review the ratios and percentages. We also introduce some special functions that are common in econometrics applications, including quadratic functions and natural logarithms. The first four sections require only basic algebraic techniques. The fifth section briefly reviews differential Calculus Although Calculus is not necessary to understand much of this book, it is used in some of the end-of-chapter appendices and in some of the more advanced topics in part 3.

\section{Summation Operator and Description Statistics}

\textbf{Summation Operator} is an abbreviation used to express the summation of numbers, it plays an important role in statistics and econometrics analysis. If $\{x_i: i=1, 2, \ldots, n\}$ is a sequence of $n$ numbers, the summation of the $n$ numbers is:

\begin{equation}
\sum_{i=1}^n x_i \equiv x_1 + x_2 +\cdots + x_n
\end{equation}


%! Author = tstreule

\section{General}

\subsection{SigSys Basics \textnormal{\hfill $\textrm{SNR}\uparrow \hat{=} \textrm{resolution}\downarrow$ \hfill $x\leftrightarrow t$, $k\leftrightarrow f$}}
%
IFFT:\hfill
\highlight{$\displaystyle f(x,y) = \frac{1}{2\pi} \iint \mathcal{F}[f](k_x, k_y) \eu^{\iu(k_x x + k_y y)} \diff k_x \diff k_y$}\\
FFT:\hfill
\highlight{$\displaystyle \mathcal{F}[f](k_x,k_y) = \frac{1}{2\pi} \iint f(x,y) \eu^{-\iu(k_x x + k_y y)} \diff x \diff y$}
\\
\includegraphics[width=\linewidth]{Basics_Comb}
\\
Prevent \textbf{aliasing}: \fbox{$\Delta x<\frac{2\pi}{\textrm{BW}}$} $\leftrightarrow$ \fbox{$\Delta k<\frac{2\pi}{\textrm{FOV}}$}
\begin{minipage}{.25\linewidth}sampl. res.\\ $\leftrightarrow$ image size\end{minipage}
\\
\highlight{$\displaystyle \mathcal{F}[f](k_m) = \underbrace{ \eu^{-\iu k_m x_0}\vspace{-1mm} }_{ \vspace{-1mm}\textrm{shift} } \sum \limits_{n=0}^{N-1} f(x_n) \eu^{-\iu\frac{2 \pi m n}{N}}$}
normally: $x_0 = 0$

\textbf{Noise}:
\begin{minipage}[t]{.88\textwidth}
    \textbf{Gaussian}:
    $\highlight{P_n(\eta) = \frac{1}{\sqrt{2 \pi \sigma^2}} e^{-\frac{\eta^2}{2 \sigma^2}}} \mathrm{FWHM} = 2\sqrt{2\ln 2}$\\
    \textbf{Poisson}:
    \highlight{$\displaystyle P_\lambda(x) = \frac{\lambda^x}{x!}\eu^{-\lambda}$} ($\to$ discrete events)\\
    \fbox{$\textrm{SNR} = \abs{S[f](x,y)} \; \frac{1}{\sigma}$}
    $\leftrightarrow$
    \fbox{$\textrm{CNR}\ped{AB} = \abs{ \textrm{SNR}\ped{A}-\textrm{SNR}\ped{B} }$}
\end{minipage}


\section{Points and vectors}

In this section, we will first introduce complex numbers, because they are a useful way to think about and represent 2D points, especially when rotations are involved. We will then present two different ways to represent points in code: one by creating our own structure, the other by using the C++ built-in \lstinline|complex| type. Either can be used to run the code samples in this book, though \lstinline|complex| requires less typing.

\subsection{Complex numbers}\label{complex}
Complex numbers are an extension of the real numbers with a new unit, the \term{imaginary unit}, noted $i$. A complex number is usually written as $a + bi$ (for $a,b \in\real$) and we can interpret it geometrically as point $(a,b)$ in the two-dimensional plane, or as a vector with components $\vv{v}=(a,b)$. We will sometimes use all these notations interchangeably. The set of complex numbers is written as $\complex$.

\centerFig{point0}

\subsubsection{Basic operations}
Complex numbers are added, subtracted and multiplied by scalars as if $i$ were an unknown variable. Those operations are equivalent to the same operations on vectors.
\begin{align*}
(a+bi) + (c+di) &= (a+c) + (b+d)i &\mbox{(addition)} \\
(a+bi) - (c+di) &= (a-c) + (b-d)i &\mbox{(subtraction)} \\
k(a+bi) &= (ka) + (kb)i &\mbox{(multiplication by scalar)}
\end{align*}
Geometrically, adding or subtracting two complex numbers $\vv{v}=(a,b)$ and $\vv{w}=(c,d)$ corresponds to making $\vv{w}$ or its opposite start at the end of $\vv{v}$, while multiplying $\vv{v}$ by a positive real $k$ corresponds to multiplying its length by $k$ but keeping the same direction.

\centerFig{point1}
\centerFig{point5}

\subsubsection{Polar form}\label{polar-form}
The polar form is another way to represent complex numbers. To denote a complex $\vv{v} = a+bi$, instead of looking at the real and complex parts, we look at the \term{absolute value} $r$, the distance from the origin (the length of vector $\vv{v}$), and the \term{argument} $\phi$, the amplitude of the angle that $\vv{v}$ forms with the positive real axis.

\centerFig{point2}

For a given complex number $a+bi$, we can compute its polar form as
\begin{align*}
r &= |a+bi| = \sqrt{a^2+b^2}\\
\phi &= \arg(a+bi) = \atanTwo(b,a)
\end{align*}
and conversely, a complex number with polar coordinates $r,\phi$ can be written
\[r\cos\phi+(r\sin\phi)i = r(\cos\phi + i\sin\phi) \eqqcolon r\cis\phi\]
where $\cis\phi = \cos\phi+i\sin\phi$ is the unit vector that forms an angle of amplitude $\phi$ with the positive real axis.

Note that this is not a one-to-one mapping. Firstly, adding or subtracting $\turn$ from $\phi$ doesn't change the point being represented; to solve this problem, $\phi$ is generally taken in $(-\half,\half]$. Secondly, when $r=0$, all values of $\phi$ represent the same point.

\subsubsection{Multiplication}

Complex multiplication is easiest to understand using the polar form. When multiplying two complex numbers, their absolute values are multiplies, while their arguments are added. In other words,
\[(r_1\cis\phi_1) \ast (r_2\cis\phi_2) = (r_1r_2) \cis (\phi_1+\phi_2).\]

In the illustration below, $|\vv{v}\ast\vv{w}| = |\vv{v}||\vv{w}|$ and the angle between the $x$-axis and $\vv{v}$ is the same as the angle between $\vv{w}$ and $\vv{v}\ast\vv{w}$.

\centerFig{point3}

Remarkably, multiplication is also very simple to compute from the coordinates: it works a bit like polynomial multiplication, except that we transform $i^2$ into $-1$.
\begin{align*}
(a+bi)\ast(c+di) &= ac + a(di) + (bi)c + (bi)(di)\\
&= ac + adi + bci + (bd)i^2\\
&= ac + (ad+bc)i + (bd)(-1)\\
&= (ac-bd) + (ad+bc)i
\end{align*}

\exoWithSolution{
    Prove that $(r_1\cis\phi_1) \ast (r_2\cis\phi_2) = (r_1r_2) \cis (\phi_1+\phi_2)$ using this new definition of product.
}{
    \protect \begin{align*}
        (r_1\cis\phi_1)&\ast(r_2\cis\phi_2)\\
        &=\left(r_1\cos\phi_1 + (r_1\sin\phi_1)i\right)
            \ast \left(r_2\cos\phi_2 + (r_2\sin\phi_2)i\right)\\
        &= r_1r_2[(\cos\phi_1\cos\phi_2 - \sin\phi_1\sin\phi_2)\\
        &\qquad + (\cos\phi_1\sin\phi_2 + \sin\phi_1\cos\phi_2)i] \\
        &= r_1r_2(\cos(\phi_1 + \phi_2) + i\sin(\phi_1+\phi_2)) \\
        &= r_1r_2 \cis(\phi_1 + \phi_2)
    \protect \end{align*}
}{polar-multiplication}

Another way to explain complex multiplication is to say that multiplying a number by $r\cis\phi$ will scale it by $r$ and rotate it by $\phi$ counterclockwise. For example, multiplying a number by $\frac{1}{2}i = \frac{1}{2}\cis\quarter$ will divide its length by 2 and rotate it $90\degree$ counterclockwise.

\centerFig{point4}

%This also makes defining division very intuitive: just \emph{divide} the absolute values and \emph{subtract} the arguments: $(r_1\cis\phi_1) / (r_2\cis\phi_2) = \frac{r_1}{r_2} \cis(\phi_1-\phi_2)$. Without going through polar coordinates, it can also be written as
%\[z/w = \frac{z\ast\conj{w}}{\norm{w}^2}\]
%where $\conj{w}$ is the \term{complex conjugate} of $w$, an operation that changes the sign of the complex part: $\conj{a+bi} = a-bi$.

\subsection{Point representation}\label{ss:point-representation}
In this section we explain how to implement the point structure that we will use throughout the rest of the book. The code is only available in C++ at the moment, but should be easy to translate in most languages.% Our long-term plan is to offer code snippets in other languages as well.

\subsubsection{With a custom structure}
Let us first declare the basic operations: addition, subtraction, and multiplication/division by a scalar.
\begin{lstlisting}
typedef double T;
struct pt {
    T x,y;
    pt operator+(pt p) {return {x+p.x, y+p.y};}
    pt operator-(pt p) {return {x-p.x, y-p.y};}
    pt operator*(T d) {return {x*d, y*d};}
    pt operator/(T d) {return {x/d, y/d};} // only for floating-point
};
\end{lstlisting}
For generality, we declare type \lstinline|T|: the type of the the coordinates. Generally, either \lstinline|double| or \lstinline|long long| (for exact computations with integers) is appropriate. \lstinline|long double| can also be very useful if extra precision is required. The cases where integers cannot be used are often quite clear (e.g. division by scalar, rotation by arbitrary angle).

We define some comparators for convenience:
\begin{lstlisting}
bool operator==(pt a, pt b) {return a.x == b.x && a.y == b.y;}
bool operator!=(pt a, pt b) {return !(a == b);}
\end{lstlisting}
Note that there is no obvious way to define a $<$ operator on 2D points, so we will only define it as needed.

Here are some functions linked to the absolute value:
\begin{lstlisting}
T sq(pt p) {return p.x*p.x + p.y*p.y;}
double abs(pt p) {return sqrt(sq(p));}
\end{lstlisting}
The squared absolute value \lstinline|sq()| can be used to compute and compare distances quickly and exactly if the coordinates are integers. We use \lstinline|double| for \lstinline|abs()| because it will return floating-point values even for integer coordinates (if you are using \lstinline|long double| you should probably change it to \lstinline|long double|).

We also declare a way to print out points, for debugging purposes:
\begin{lstlisting}
ostream& operator<<(ostream& os, pt p) {
    return os << "("<< p.x << "," << p.y << ")";
}
\end{lstlisting}

Some example usage:
\begin{lstlisting}
pt a{3,4}, b{2,-1};
cout << a+b << " " << a-b << "\n"; // (5,3) (1,5)
cout << a*-1 << " " << b/2 << "\n"; // (-3,-4) (1.5,2)
\end{lstlisting}

We also define a signum function, which will be useful for several applications. It returns \lstinline|-1| for negative numbers, \lstinline|0| for zero, and \lstinline|1| for positive numbers.
\begin{lstlisting}
template <typename T> int sgn(T x) {
    return (T(0) < x) - (x < T(0));
}
\end{lstlisting}

\subsubsection{With the C++ \lstinline|complex| structure}
Using the \lstinline|complex| type in C++ can be a very practical choice in contests such as ACM-ICPC where everything must be typed from scratch, as many of the operations we need are already implemented and ready to use.

The code below defines a \lstinline|pt| with similar functionality.
\begin{lstlisting}
typedef double T;
typedef complex<T> pt;
#define x real()
#define y imag()
\end{lstlisting}

\begin{warning}
As with the custom structure, you should choose the appropriate coordinate type for \lstinline|T|. However, be warned that if you define it as an integral type like \lstinline|long long|, some functions which should always return floating-point numbers (like \lstinline|abs()| and \lstinline|arg()|) will be truncated to integers.
\end{warning}

The macros \lstinline|x| and \lstinline|y| are shortcuts for accessing the real and imaginary parts of a number, which are used as $x$- and $y$-coordinates:
\begin{lstlisting}
pt p{3,-4};
cout << p.x << " " << p.y << "\n"; // 3 -4
// Can be printed out of the box
cout << p << "\n"; // (3,-4)
\end{lstlisting}

Note that the coordinates can't be modified individually:
\begin{lstlisting}
pt p{-3,2};
p.x = 1; // doesn't compile
p = {1,2}; // correct
\end{lstlisting}

We can perform all the operations that we have with the custom structure and then some more. Of course, we can also use complex multiplication and division. Note however that we can only multiply/divide by scalars of type \lstinline|T| (so if \lstinline|T| is \lstinline|double|, then \lstinline|int| will not work).
\begin{lstlisting}
pt a{3,1}, b{1,-2};
a += 2.0*b; // a = (5,-3)
cout << a*b << " " << a/-b << "\n"; // (-1,-13) (-2.2,-1.4)
\end{lstlisting}

There are also useful methods for dealing with polar coordinates:
\begin{lstlisting}
pt p{4,3};
// Get the absolute value and argument of point (in [-pi,pi])
cout << abs(p) << " " << arg(p) << "\n"; // 5 0.643501
// Make a point from polar coordinates
cout << polar(2.0, -M_PI/2) << "\n"; // (1.41421,-1.41421)
\end{lstlisting}

\begin{warning}
The \lstinline|complex| library provides function \lstinline|norm|, which is mostly equivalent to the \lstinline|sq| that we defined earlier. However, it is not guaranteed to be exact for \lstinline|double|: for example, the following expression evaluates to \lstinline|false|.
\begin{lstlisting}
norm(complex<double>(2.0,1.0)) == 5.0
\end{lstlisting}

Therefore, to be safe you should implement a separate \lstinline|sq()| function as for the custom structure (or you can wait use function \lstinline|dot()| that we will define later).
\end{warning}


\subsection{Dot Products}
\noindent
A dot product is a way of multiplying two vectors so that the result is a scalar. $\vec{a}\cdot\vec{b} = \norm{\vec{a}}\norm{\vec{b}}\cos{\theta}$ where $\theta$ is the angle between $\vec{a}$ and $\vec{b}$. One way to think of the dot product is as a measure of how much two vectors point in the same direction.\\
We can also show using the law of cosines that $\vec{a}\cdot\vec{b} = a_1b_1+a_2b_2+...+a_nb_n$.\\
Knowing the lengths of two vectors and their dot product we can calculate the angle between them as
\begin{equation*}
	\theta = \arccos{\left(\frac{\vec{a} \cdot \vec{b}}{\norm{\vec{a}} \norm{\vec{b}}}\right)}
\end{equation*}

\begin{figure}[h]
	\centering
	\includegraphics[scale=0.33]{Images/backgroundReview/DotProduct}
\end{figure}

\noindent
Although similar to scalar multiplication, dot products have some properties that set them apart.
\begin{itemize}
	\item Commutative
		\begin{equation*}
			\vec{a}\cdot\vec{b} = \vec{b}\cdot\vec{a}
		\end{equation*}
		the same as scalar multiplication.
	\item Distributive
		\begin{equation*}
			\vec{a}\cdot\left(\vec{b}+\vec{c}\right) = \vec{a}\cdot\vec{b}+\vec{a}\cdot\vec{c}
		\end{equation*}
		the same as scalar multiplication.
	\item \textbf{NOT} Associative $\left(\vec{a}\cdot\vec{b}\right)\cdot\vec{c}$ is a nonsense expression. However, like scalar multiplication, dot products are scalar associative.
	\begin{equation*}
		\left(c\cdot\vec{a}\right)\cdot\vec{b} = \vec{a}\cdot\left(c\cdot\vec{b}\right)
	\end{equation*}
\end{itemize}


\paragraph{\underline{Hypothesis tests for the mean}}
$\mu$

\vspace{5pt}
\noindent \textbf{One-sample z-test}
$H_0: \mu=\mu_0$

We use it to check hypothesis that mean $\mu$ is equal to the specific value $\mu_0$. Critical
region is: $ \mathcal{K}_\alpha = (-\infty, k] \cup [k, +\infty) $

\vspace{5pt}
\textbf{\em Model 1}:
$X$ i.i.d. $\distnormal(\mu, \sigma)$, known $\sigma$

\vspace{-15pt}
\begin{gather*}
T=\frac{\bar{X}-\mu_0}{\sigma}\sqrt{n}
\overset{H_0}{\longrightarrow} \distnormal(0,1)
\end{gather*}

\vspace{-20pt}
\begin{gather*}
k = z_{1-\frac{\alpha}{2}}
\end{gather*}

\textbf{\em Model 2}:
$X$ i.i.d. ?, large $n$

\vspace{-15pt}
\begin{gather*}
T=\frac{\bar{X}-\mu_0}{s_X}\sqrt{n}
\overset{H_0}{\underset{n \rightarrow \infty}{\longrightarrow}} \distnormal(0,1)
\end{gather*}

\vspace{-15pt}
\begin{gather*}
k = \diststudentt^{[n-1]}_{1-\frac{\alpha}{2}}
\end{gather*}

\noindent \textbf{One-sample t-test}
$X$ i.i.d. $\distnormal(\mu, \sigma)$, $\sigma=?$, $H_0: \mu=\mu_0$

\vspace{-15pt}
\begin{gather*}
z=\frac{\bar{X}-\mu_0}{s_X}\sqrt{n}
\overset{H_0}{\longrightarrow} \diststudentt^{[n-1]}
\end{gather*}

\vspace{-15pt}
\begin{gather*}
k = z_{1-\frac{\alpha}{2}}
\end{gather*}

\noindent \textbf{Two-sample z-test}

We use it to check a hypothesis that means of two samples are equal. The test can be applied in
following 2 cases:

\vspace{10pt}
\textbf{\em Model 1}:
$X$, $Y$ i.i.d. $\distnormal(\mu_{X,Y}, \sigma_{X,Y})$,

known $\sigma_X$ and $\sigma_Y$, $H_0: \mu_X=\mu_Y$

\vspace{-15pt}
\begin{gather*}
a_I=\frac{\sigma_I^2}{n_I} \;,\;
z=\frac{\bar{X}-\bar{Y}}{ \sqrt{a_X+a_Y} }
\longrightarrow \distnormal(0,1)
\end{gather*}

\textbf{\em Model 2}:
$X$, $Y$ i.i.d. ?, large $n_{X,Y}$,

$H_0: \mu_X=\mu_Y$

\vspace{-15pt}
\begin{gather*}
a_I=\frac{s_X^2}{n_X} \;,\;
z=\frac{\bar{X}-\bar{Y}}{ \sqrt{a_X+a_Y} }
\underset{n \rightarrow \infty}{\longrightarrow} \distnormal(0,1)
\end{gather*}

\noindent \textbf{Two-sample t-test}

\vspace{10pt}
\textbf{\em Model 1}:
$X$, $Y$ i.i.d. $\distnormal(\mu_{X,Y}, \sigma_{X,Y})$,

$\sigma_X=\sigma_Y=?$, $H_0: \mu_X=\mu_Y$,
$d = n_X+n_Y-2$, $a_i = (n_i - 1)s_i^2$

\vspace{-15pt}
\begin{gather*}
\dot{s}^2 = \frac{a_X + a_Y}{d} \;,\;
b_I=\frac{\dot{s}^2}{n_I}
\end{gather*}

\[ t=\frac{\bar{X}-\bar{Y}}{ \sqrt{b_X+b_Y} }
\overset{H_0}{\longrightarrow} \diststudentt^{[d]} \]

\textbf{\em Model 2}:
$X$, $Y$ i.i.d. $N(\mu_{X,Y}, \sigma_{X,Y})$,

$\sigma_X=?$ and $\sigma_Y=?$, $H_0: \mu_X=\mu_Y$

\vspace{-15pt}
\begin{gather*}
a_i = \frac{s_i^2}{n_i} \;,\;
d = \frac{(a_X + a_Y)^2}{ \frac{a_X^2}{n_X-1}+\frac{a_Y^2}{n_Y-1} }
\end{gather*}

\vspace{-15pt}
\begin{gather*}
t=\frac{\bar{X}-\bar{Y}}{ \sqrt{ a_X + a_Y } }
\overset{H_0}{\longrightarrow} \diststudentt^{[d]}
\end{gather*}

\vfill
\columnbreak

\noindent \textbf{Paired t-test}
$X$, $Y$ i.i.d. $\distnormal(\mu_{X,Y}, \sigma_{X,Y})$,

$\sigma_X=?$ and $\sigma_Y=?$, $H_0: \mu_X=\mu_Y$,

$\forall i: (X_i,Y_i)$ are dep., $Z=X-Y$

\vspace{-15pt}
\begin{gather*}
t= \frac{\bar{Z}-0}{s_Z}\sqrt{n}
\overset{H_0}{\longrightarrow} \diststudentt^{[n-1]}
\end{gather*}

It is usually used when identical samples are taken at two points in time, and we wish to check
whether there are significant differences between them.

\paragraph{\underline{Hypothesis tests for variance}}
$\sigma^2$

\vspace{5pt}
\noindent \textbf{One-sample test}
$X$ i.i.d. $\distnormal(\mu, \sigma)$, $H_0: \sigma^2=\sigma_0^2$

We use this test to test a hypothesis that $\sigma^2$ is equal to a specific value $\sigma_0^2$.
$ \mathcal{K}_\alpha = (0, \distchisquare_{k, \frac{\alpha}{2}}] \cup [\distchisquare_{k, 1-\frac{\alpha}{2}}, +\infty) $

\vspace{5pt}
\textbf{\em Model 1}:
known $\mu$

\vspace{-15pt}
\begin{gather*}
T = \frac{n s_X^2}{\sigma_0^2}
\overset{H_0}{\longrightarrow} \distchisquare_{n-1}
\end{gather*}

\vspace{-15pt}
\begin{gather*}
k = n
\end{gather*}

\textbf{\em Model 2}:
$\mu=?$

\vspace{-15pt}
\begin{gather*}
T = \frac{(n-1)s_X^2}{\sigma_0^2}
\overset{H_0}{\longrightarrow} \distchisquare_{n-1}
\end{gather*}

\vspace{-15pt}
\begin{gather*}
k = n-1
\end{gather*}

\noindent \textbf{F-test}
$X$, $Y$ i.i.d. $\distnormal(\mu_{X,Y}, \sigma_{X,Y})$,

$\mu_X=?$ and $\mu_Y=?$, $H_0: \sigma_X^2=\sigma_Y^2$

We use this test to test a hypothesis that two variances $\sigma_X^2$ and $\sigma_Y^2$ (for samples
$X$ and $Y$ respectively) are equal.

\vspace{-15pt}
\begin{gather*}
F = \frac{s_X^2}{s_Y^2} \overset{H_0}{\longrightarrow} \distf^{[n_1-1,n_2-2]}
\end{gather*}

\paragraph{\underline{Hypothesis tests for proportion}} $p$

\vspace{5pt}
\noindent \textbf{One-sample z-test}
$X$ i.i.d. $Bern(p)$, $H_0: p=p_0$

We use this test to test a hypothesis that $p$ is equal to a specific value $p_0$. It applies to two
such models:

\vspace{10pt}
\textbf{\em Model 1}:
$n_{X,Y}\geq100$

\vspace{-15pt}
\begin{gather*}
z=\frac{\hat{p}-p_0}{\sqrt{p_0(1-p_0)}}\sqrt{n}
\overset{H_0}{\underset{n \rightarrow \infty}{\longrightarrow}} \distnormal(0,1)
\end{gather*}

\textbf{\em Model 2}:
$n_{X,Y}<100$, $A = \arcsin$

\vspace{-15pt}
\begin{gather*}
z=2\left(A{\sqrt{\hat{p}}}-A{\sqrt{p_0}} \right)\sqrt{n}
\longrightarrow \distnormal(0,1)
\end{gather*}

\noindent \textbf{Two-sample z-test}
$X$, $Y$ i.i.d. $\distbernoulli(p_{X,Y})$, $H_0: p_X=p_Y$

We use this test to test a hypothesis that parameters $p_X$ and $p_Y$ (which are empirical success
rates for samples $X$ and $Y$ respectively) are equal. It applies to two such models:

\vspace{10pt}
\textbf{\em Model 1}:
$n_{X,Y}\geq100$, $ k_I = \Sigma_{i=1}^{n_I} I_i $

$ \dot{p}=\frac{k_X+k_Y}{n_X+n_Y} \mbox{ , } \dot{n}=\frac{n_X n_Y}{n_X+n_Y} $

\vspace{-15pt}
\begin{gather*}
z=\frac{\hat{p_X}-\hat{p_Y}}{\sqrt{\dot{p}(1-\dot{p})}}\sqrt{\dot{n}}
\underset{n_{X,Y} \rightarrow \infty}{\longrightarrow} \distnormal(0,1)
\end{gather*}

\textbf{\em Model 2}:
$n_{X,Y}<100$, $A \equiv \arcsin$

\vspace{-15pt}
\begin{gather*}
z=2\left(A{\sqrt{\hat{p_X}}}-A{\sqrt{\hat{p_Y}}} \right)\sqrt{\dot{n}}
\longrightarrow \distnormal(0,1)
\end{gather*}

\vfill


\paragraph{Simplicial complexes.}
A simplicial complex is a collection of finite sets closed under taking subsets.
We call a member of a simplicial complex $K$ a \emph{simplex} of \emph{dimension $p$} if it has cardinality $p+1$, and denote the set of all such $p$-simplices $K_p$.
A $p$-simplex has $p+1$ \emph{faces} of dimension $p-1$, namely the subsets omitting one element. We denote these $[v_0,\dotsc,\hat{v}_i,\dotsc, v_p]$ when omitting the $i$'th element.
If a simplex $\sigma$ is a face of $\tau$, we say that $\tau$ is a \emph{coface} of $\sigma$. While this definition is entirely combinatorial, there is a geometric interpretation, and it will make sense to refer to and think of $0$-simplices as \emph{vertices}, $1$-simplices as \emph{edges}, $2$-simplices as \emph{triangles}, $3$-simplices as \emph{tetrahedra}, and so forth (see Figure~\ref{fig:data2complex}, (b)).

Let $C^p(K)$ be the set of functions $K_p\to\RR$, with the obvious vector space structure. These \emph{$p$-cochains} will encode our data. Define the linear \emph{coboundary} maps $\delta^p:C^p(K)\to C^{p+1}(K)$ by
\begin{equation*}
\delta^p(f)([v_0,\dotsc,v_{p+1}]) = \sum_{i=0}^{p+1} (-1)^i f([v_0,\dotsc,\hat{v}_i,\dotsc,v_{p+1}]).
\end{equation*}
Observe that this definition can be thought of in geometric terms: The support of $\delta^p(f)$ is contained in the set of $(p+1)$-simplices that are cofaces of the $p$-simplices that make up the support of $f$.

\begin{figure}[htpb]
%\begin{table*}[!t]
\savebox{\tempbox}{% compute size of tabulat
\scriptsize{
\begin{tabular}{llr}
    \toprule
    Papers & Authors & Citations \\
    \midrule
    Paper I & A, B, C  & 100  \\
    Paper II &  A, B & 50\\
    Paper III & A, D & 10\\
    Paper IV & C, D & 4\\
    \bottomrule
  \end{tabular}
}}%
\settowidth{\tempwidth}{\usebox{\tempbox}}%
\hfil\begin{minipage}[b]{\tempwidth}%
\raisebox{-\height}{\usebox{\tempbox}}%
%\vspace{-7pt}
\scriptsize{\caption*{(a)}}%
\label{table:data}%
\end{minipage}%
\savebox{\tempbox}{
\input{figures/coauthorship_complex.tex}
}%
\settowidth{\tempwidth}{\usebox{\tempbox}}%
\hfil\begin{minipage}[b]{\tempwidth}%
\raisebox{-\height}{\usebox{\tempbox}}%
\vspace{-3pt}
\scriptsize{\captionof*{figure}{(b)}}%
\label{fig:co-authoship-complex}%
\end{minipage}%
%\vspace{5pt}
%\end{table*}
\savebox{\tempbox}{\scriptsize{
\begin{blockarray}{cccccc}
\tiny{AB} & \tiny{AC} & \tiny{AD} & \tiny{BC} & \tiny{CD} \\
\begin{block}{(ccccc)c}
  3 & 0 & 1 & 0 & 0 & \tiny{AB} \\
  0 & 3 & 1 & 0 & -1 & \tiny{AC} \\
  1 & 1 & 2 & 0 & 1 & \tiny{AD} \\
  0 & 0 & 0 & 3 & -1 & \tiny{BC}\\
  0 & -1 & 1 & -1 & 2 & \tiny{CD}\\
\end{block}
\end{blockarray}}}%
\settowidth{\tempwidth}{\usebox{\tempbox}}%
\hfil\begin{minipage}[b]{\tempwidth}%
\raisebox{-\height}{\usebox{\tempbox}}%
\vspace{-7pt}
\scriptsize{\captionof*{figure}{(c)}}%
\end{minipage}%

%\end{table*}
\caption{Constructing a simplicial complex from data. (a)~Coauthorship data. (b)~Coauthorship complex with corresponding cochains from the data. (c)~Degree-$1$ Laplacian $L_1$ of the coauthorship complex.}\label{fig:data2complex}
\end{figure}

\paragraph{Simplicial Laplacians.}
We are in this paper concerned with finite abstract simplicial complexes, although our method is applicable to a much broader setting, e.g.\ CW-complexes. In analogy with Hodge--de Rham theory~\cite{madsen1997calculus}, we define the \emph{degree-$i$ simplicial Laplacian} of a simplicial complex $K$ as the linear map
\begin{align*}
  &\lap_i:C^i(K)\to C^i(K) \\
  &\lap_i = \lapu_i + \lapd_i = \delta^{i\ast}\circ\delta^{i} + \delta^{i-1}\circ\delta^{i-1\ast},
\end{align*}
where $\delta^{i\ast}$ is the adjoint of the coboundary with respect to the inner product (typically the one making the indicator function basis orthonormal). In most practical applications, the coboundary can be represented as a sparse matrix $B_i$ and the Laplacians can be efficiently computed as $L_i=B_i\transpose B_{i}+B_{i-1}B_{i-1}\transpose$. The matrices $L_0$ and $B_0$ are the classic graph Laplacian and incidence matrix. Note that the Laplacians carry valuable topological information about the complex: The kernel of the $k$-Laplacian is isomorphic to the $k$-(co)homology of its associated simplicial complex~\cite{eckmann1944,horak2013spectra}\footnote{In other words, the number of zero-eigenvalues of the $k$-Laplacian corresponds to the number of $k$-dimensional holes in the simplicial complex.}.