\documentclass[12pt, a4paper]{scrartcl}
\usepackage[utf8]{inputenc}
\usepackage{graphicx}
\usepackage{amsmath, amsthm, amssymb, textcomp}
\usepackage{setspace}
\usepackage{paralist}
\usepackage{graphicx}
\usepackage{caption}
\graphicspath{{WSK_im/}} %Graphic is in a folder named WSK_im in the currend directory
\usepackage{float}
\usepackage{authblk}
\renewcommand\Authfont{\fontsize{12}{14.4}\selectfont}
\title{Bayesian probability theory - Lesson 8:\\
Continuous probability distributions and invariance}

\author{Wolfgang von der Linden, Gerhard Dorn, Johanna Moser}
\date{Transcript}

\begin{document}
\setlength{\parindent}{0pt}
\maketitle
\onehalfspacing

Welcome to unit 8 of the course on Bayesian probability theory. My name
is Wolfgang von der Linden and I will enable you to help Captain Bayes and
her crew to deal with \textbf{continuous variables} and to learn how to\textit{assign prior probabilites}.
\begin{itemize}\item We will learn how to work with \textbf{probability densities} instead of probability mass functions.
\item We will learn how to use the \textbf{transformation law} to locate the lighthouse and the principle of \textbf{transformation invariance} will allow us to derive \textbf{Jeffrey’s prior}
\item We will study the probability distribution of the first digits of mathematical tables.
\end{itemize}

\section*{Continuous variables}
Up to this point we have actually only introduced the rules of probability
theory for \textbf{discrete variables}. But you certainly realized that \textbf{continuous
variables} have also crept into the problems of the last unit, for example in the star
path problem or the fishing ground problem. We even mentioned probability
densities. \\
The values of the parameters in the continuous case can take on
\textit{any real number} in a given interval or even on the entire real axis.
Now it’s time to extend the rules of probability theory to continuous variables.\\

Let’s start with the following consideration: Assume we are interested in the height of trees. Typically it ranges between 0 and and a maximum height $h_{max}$. To get started we discretize the possible heights into $N$ intervals of equal length. For simplicity, we assume that all heights are equally probable. Therefore,
the probability that the height $l$ of a randomly selected tree lies in
the interval $I_i$ is the same for all intervals.
\[P(\Delta I_i)=\frac 1N = \frac{\Delta l}{l_{max}}\]%

Two things are important to note here:\\
a) If we increase the number of intervals the probability goes to zero. So
the \textit{probability for a continuous variable to have a precise value is zero} and
hence it makes no sense to speak of probabilities or to use a probability mass
function to assign probabilities to certain variables.\\

\fbox{\parbox{\linewidth}{\textbf{Question 1.} Which of the following statements are true?\\
a) The height of trees shall be uniform in the interval [0,30]. Then, the probability for a tree to have a height of 20 meters is $\frac{1}{30}$.\\
b) The height of trees shall be uniform in the interval [0,30]. Then, the probability to have a tree of 15 to 17 meters is $\frac{1}{15}$.\\
c) The height of trees shall be uniform in the interval [0,30]. Then, the probability to have a tree of 14 to 16 meters is $\frac{1}{15}$.\\
d) Assuming a unifor mistribution with the probability given for an interval $P(\Delta I)=A$, the probability for $P(\Delta I/2)=A/2$.
}}
\\

b) We see that the probability is \textit{proportional to the interval size} $\Delta l$ .\\
Both points stay valid in the case of non-equidistant intervals and also if the height
distribution of trees is not uniform.\\

In general, we find for the probability, that the height of a tree lies in a certain interval, is \textit{proportional} to the length of this interval. The proportionality function is the \textbf{probability density} $p(l)$.
 It is the probability to find a tree with a length in the interval $[l,l+\Delta l]$ divided by the interval length.
 \begin{equation*}\boxed{p(l)=\frac{P(\Delta I_l)}{\Delta l}
}\end{equation*}\\
The probability for the height of a tree to fall into an interval can also be obtained
form the difference
\[P(\Delta I_l)=P(\text{height}\leq l+\Delta l)-P(\text{height} \leq l)\]

We introduce a special function $P(\text{height}\leq l)=F(l)$ for these probabilities, which are called \textbf{cumulative distribution functions}. %
The reason for this term becomes clear when we rewrite $p(l)$ in
terms of the cdf:\\
\[p(l)=\frac{F(l+\Delta l)-F(l)}{\Delta l}\]

For infinitesimal interval length the pdf turns into the \textit{derivative} of the cdf
and in turn the cdf is the integral of the pdf.\\
 \begin{equation*}\boxed{F(l)=\int_0^lp(l')\text{d}l'
 }\end{equation*}\\

That explains why it is called cumulative. We also see that the normalization
implies that the cdf approaches 1 when the length approaches the upper
limit of the the range of this random variable.\\

\fbox{\parbox{\linewidth}{\textbf{Question 2.} Which of the following statements are true?\\
a) If the pdf is Gaussian, then the cdf is also Gaussian.\\
b) If the pmf is given, then the pdf can be obtained by the derivative.\\
c) If I want to know the probability that a tree is at max 20 meters high, I can use the cdf $F(20)$.\\
d) The cdf has the following form: $F(x)\propto x^2$. Then the pdf reads $p(x)\propto x$.
}}
\\

To keep the notation clear, we denote pdfs with the lowercase letter $p$ and
stick with the uppercase letter $P$ for the probability mass functions.\\

\section*{Calculations with continuous variables}
The sum and product rule are the same for continuous and discrete variables.
But the normalization and the marginalization rule are slightly different, as we do have integrals instead of sums. %8_1
 \begin{figure}[H]
	\centering
	\includegraphics[width=0.75\textwidth]{8_1.png}
\end{figure}
Here, the difference between the four marginalization formulas is that capital letters stand for discrete lower case letters for continuous variables. Similarly, the mean value of functions $f(x)$ of continuous random variables x are also converted to integrals. \\
\begin{equation*}\boxed{\langle f(X)\rangle = \sum_i f(X_i)P(X_i)\qquad \langle f(x)\rangle = \int \text{d}x f(x)p(x)
}\end{equation*}\\
Here, the probability mass function is replaced by the pdf.\\

Now there is a specialty of continuous variables: You can introduce \textbf{transformations} like $l\mapsto m(l)=l^3$, which for a spherical tree would be proportional
to the mass and if one is interested in the mass then the corresponding pdf would be $p(m)$. Then a question occurs naturally: \textit{If we know the pdf $p(l)$ for the length what is the pdf $p(m)$ for the mass?}\\

The probability, that the height of a tree lies in the interval $[l,l+\Delta l]$, is given by d$P$.
The mass of these trees lies in the interval $[m,m+\Delta m]$. Since they are the same trees, we can express dP in terms of the mass and finally, due to the unique relation ``$m$ is a unique function of $l$ and $l$ is a unique function of $m$'' (i.e. bijection), we obtain the very important transformation law.\\
\begin{equation*}\boxed{p(m)=p(l(m))\left|\frac{\text{d}l(m)}{\text{d}m}\right|
}\end{equation*}\\

For the case of the uniform height distribution from above, $p(l)$ is constant, we
obtain the pdf for the mass. \[p(m)=\frac{1}{l_{\text{max}}}\frac{\text{d}(m^{1/3})}{\text{d}m}=\frac{1}{3l_{\text{max}}}m^{-2/3}\]

\fbox{\parbox{\linewidth}{\textbf{Question 3.} Which of the following transformations are correct?\\
a) $p(x)=\frac 1x$ and $y(x)=\frac 1x$. Then, the pdf of $y$ is $p(y)=\frac{1}{y^3}$\\
b) $p(x)=\frac 1x$ and $y(x)=\frac 1x$. Then, the pdf of $y$ is $p(y)=\frac{1}{y^2}$\\
c) $p(x)=x^2$ and $y(x)=\tan(x)$. Then, the pdf of $y$ is $p(y)=\arctan(y)^2\frac{1}{1+y^2}$
}}
\\

%8_2
 \begin{figure}[H]
	\centering
	\includegraphics[width=0.75\textwidth]{8_2.png}
\end{figure}
So instead of being uniform, the pdf for the mass is sharply peaked at zero.
That brings us to a very fundamental question: \textit{How do we describe ignorant pdfs?}\\

In the discrete case the law of indifference implied equal probability but in the
case of continuous variables ignorance does not necessarily imply a uniform pdf.
As a further example, let us consider a non-negative quantity $x$ that is known
to have a uniform pdf in the interval $[0,x_{\text{max}}]$. But we are interested in the
pdf of a variable $y=\exp(x)$. The desired pdf is proportional
to $\frac{1}{y}$. So we have found, if a variable is \textit{linear on a logarithmic scale $\log(y)=kx+d$},
then it's pdf has a \textit{$\frac 1y$ dependance on the linear scale}.\\

Finally, we want to consider the transformation law for the \textbf{multivariate case}.
Let $\vec{x}$ be a set of variables with $N$ elements and $\vec{y}$ a new set of variables with the invertible transformations:
$\vec{y}(\vec{x})$ is a unique function of $\vec{x}$ and $\vec{x}(\vec{y})$ in turn is a unique function of $\vec{y}$. Then the pdfs are related by the following expression, where the last factor is the \textbf{Jacobi-determinant}.\\
\begin{equation*}\boxed{p(\vec{y})=p(\vec{x}(\vec{y}))\left|\frac{\text{d}x_i}{\text{d}y_j}\right|
}\end{equation*}\\

\fbox{\parbox{\linewidth}{\textbf{Question 4.} We look at the polar transformation of a uniform pdf:\\
Given is a uniform pdf of $\vec{x}=(x_1,x_2), \vec{y}=(r,\phi)$ with $x_1=r\cos(\phi),x_2=r\sin(\phi)$. Which of the following statements are true?\\
a) The pdf of $\vec{y}$ is given by $p(r,\phi)\propto r$\\
b) If $p(\vec{x})=p(x_1,x_2)\propto x_1+x_2$, then $p(\vec{y})=p(r,\phi)=r^2(\cos(\phi)+\sin(\phi))$\\
c) The determinant of the Jacobi Matrix is given by $r^2(\sin^2(\phi)+\cos^2(\phi)=r^2$\\
d) The Jacobi matrix is given by $\left( {\begin{array}{cc}
   \cos(\phi) & -r\sin(\phi)\\
   \sin(\phi) & r\cos(\phi) \\
  \end{array} } \right)$
}}
\\

\section*{Bertrand paradox - ignorant priors}
The crew of Captain Bayes was pondering about the so-called \textbf{Bertrand paradox} which is defined as follows:\\
\textit{Given a circle of radius R and a random set of lines that cross the circle,
what is the probability that a random line has a distance from the center that is less than half the radius?}\\%8_3
 \begin{figure}[H]
	\centering
	\includegraphics[width=0.75\textwidth]{8_3.png}
\end{figure}

The line cuts the circle at two points $P_1$ and $P_2$.
We can define these endpoints by the angles $\varphi_1$ and $\varphi_2$ formed by the \textit{radial lines to the endpoints with the x-axis}. These are two independent variables that uniquely define the line.\\%8_4
 \begin{figure}[H]
	\centering
	\includegraphics[width=0.4\textwidth]{8_4.png}
\end{figure}

The line is also uniquely defined by the midpoint $P$ of the two endpoints $P_1$ and $P_2$.
This involves again two
independent variables, namely the $x$ and $y$ coordinate of the point. Finally, for solving the problem, it is enough to know the distance $r$ of the line from the center of the circle. \\%8_5
 \begin{figure}[H]
	\centering
	\includegraphics[width=0.4\textwidth]{8_5.png}
\end{figure}

Now think of how you would simulate the problem on the computer.
Bernoulli came up with the idea to choose the points $P_1$ and $P_2$ at random
on the circle. That is equivalent to choosing the two angles $\varphi_1$ and $\varphi_2$
at random. That in turn implies that the \textit{pdf of the intermediate angle is
uniform.}\\

In this variable, the distance of the line to the center is given by $R\cos(\varphi/2)$
 and the
condition, that the line has a distance less than $R$ half from the center, is given by \[R\cos(\varphi/2)\leq R/2\]

\fbox{\parbox{\linewidth}{\textbf{Question 5.} What is the range of $\varphi\in(0,\pi]$ fulfilling the inequality $R\cos(\varphi/2)\leq R/2$?\\
a) $\varphi \in [\pi/4,\pi/3]$\\
b) $\varphi \in [0,\pi/2]$\\
c) $\varphi \in [0,2\pi/3]$\\
d) $\varphi \in [2\pi/3,\pi]$\\
}}
\\

which implies $\varphi < \frac{\pi}{6}$. Hence the desired probability is $\frac 13$.\\%8_6
 \begin{figure}[H]
	\centering
	\includegraphics[width=0.75\textwidth]{8_6.png}
\end{figure}

Another way of generating the line at random was proposed in the adventure
by Pascal. She generates the midpoint at random, which means the points
are \textit{uniformly distributed within the circle}. Then the desired probability is
the probability that the point lands in the circle of radius $\frac R2$. This is a nice
illustration of the transformation law for two variables.\\
Instead of the Cartesian coordinates $x$ and $y$ we use spherical coordinates $r$ and $\varphi$. The transformation law yields for the correctly normalized pdf the following expression. %8_7
 \begin{figure}[H]
	\centering
	\includegraphics[width=0.75\textwidth]{8_7.png}
\end{figure}
The desired result requires that $r<\frac R2$ and the corresponding probability gives $\frac 14$.

Finally, Laplace suggested to generate the radial distance uniformly between 0 and $R$ resulting in a constant pdf. In that case the probability is $0.5$.
The question is now, \textit{which algorithm describes the physical experiment?}\\

\fbox{\parbox{\linewidth}{\textbf{Question 6.} Guess who is right! What is the probability for a free day (= that the line does not cut the inner circle)? Do you know why?\\
a) Laplace: It is a 50\% chance to have a free day.\\
b) Pascal: It is a 75\% chance to have a free day.\\
c) Bernoulli: It is a 67\% chance to have a free day.
}}
\\

This question is already on the mind of scientists for generations and still is.
We can phrase this problem more generally: If we know nothing about the
pdf apart from the definition of the setup - so there are no experimental
data or additional theoretical constraints - what is the correct way to find an
un-informative prior? So how do we assign probabilities in the continuous way?
Such an object is also called an \textbf{ignorant prior}.\\

For ignorant priors Ed Jaynes came up with the following idea. \textit{If there is a
continuous and invertible map $\vec{T}(\vec{x})$ from one set of variables $\vec{x}$ to a
new set $\vec{y}$, then the corresponding pdfs should be the same, i.e $p(\vec{y})=p(\vec{x})$.} In the case of the die it
means that putting the pips on different sites of the cube should not modify the probability mass function.
In other words, \textit{the probability mass function should be invariant against
permuting the indices}, which results in the well-known uniform distribution.
Since the mapping in the continuous case is assumed to be continuous as
well, we can introduce a parameter$\epsilon$ such that for $\epsilon\rightarrow 0$ the mapping is the
identity. \[\lim_{\epsilon\rightarrow 0}\vec{T}_{\epsilon}(\vec{x})=\vec{x}\]

Using the transformation law from above and the invariance properties of
the pdf, Jaynes came up with the following equation for the pdf:%8_8
 \begin{figure}[H]
	\centering
	\includegraphics[width=0.75\textwidth]{8_8.png}
\end{figure}

Let’s illustrate this approach for a \textbf{scale variable} $y=\alpha x$. A scale variable $x$ is a
\textit{non-negative quantity for which the pdf should be scale invariant}. In this case, the infinitesimal transformation has the following form:\[T_{\epsilon}=(1+\epsilon)x\]
Jaynes’ equation can be solved by elementary tools
and the result is
\begin{equation*}\boxed{p(x)=\frac 1x
}\end{equation*}\\
Note that this is a so-called \textbf{improper pdf}, as it \textit{cannot be normalized}. In the
frame of Bayesian probability theory, such a prior is not forbidden as long as
the resulting posterior is normalizable.
This scale-invariant pdf is often called \textbf{Jeffreys’ prior}, although it is only a
special application of Jeffreys’ idea.\\

The uninformative prior pdf of parameters a depends of course on the
meaning of the parameters. In many cases these are uniquely determined
by the likelihood function $p(d|a)$. In that case, a pdf that is invariant against reparameterization is given by $p$ of a is given by the determinant of g (see picture).%8_9
 \begin{figure}[H]
	\centering
	\includegraphics[width=0.75\textwidth]{8_9.png}
\end{figure}

A typical example of the scale variable is the standard deviation $\sigma$ that enters
a Gaussian likelihood. This is a nice and informative exercise for the application of Jeffrey’s prior. The result is the same as that obtained by Jaynes’
approach $p(\sigma)=\frac{1}{\sigma}$\\

Ed Jaynes applied the principle of transformation invariance to the Bertrand paradox.
He used the following transformations under which the problem is invariant:%
\begin{enumerate}
	\item Translation of the center of the circle
	\item Rescaling of the radius
	\item Rotation of the circle about the center
\end{enumerate}
The solution of his equation yields the same result that Laplace proposed :\\
\[p(r)=\text{const}=\frac 1R\]

There is another way to approach the Bertrand paradox. Since the lines
are drawn randomly and without knowing the position of the circle, we can
equally well draw the line first and then randomly place the circle on the
plane. The circle is uniquely defined by its center and therefore this is equivalent to randomly choosing the coordinates $x$ and $y$ of the center. If we define
the direction of the line as the $x$-axis, then the $x$-coordinate of the center is
irrelevant. Consequently, the distance of the line from the center of the circle is given by the absolute value of $y$. As the coordinates of the center are
chosen at random, the distance is a uniform random variable corroborating
Jaynes' result.\\%

\section*{Maximum Entropy}
We have encountered the \textbf{maximum entropy principle} already in the discrete case. It
was introduced to obtain the probability mass function in the case of testable
information given for instance in the form of mean values. Maximum entropy can also
be invoked for pdfs given testable information for instance in the following form:\[\int \text{d}x f_i(x)p(x)=\mu_i\]

The only complication in the continuous case is that the entropy is not entirely what you would expect.\\
\begin{equation*}\boxed{S(p)=-\int dx p(x)ln\left(\frac{p(x)}{p_0(x)}\right)
}\end{equation*}\\

The function $p_0(x)$ is sometimes called the \textbf{default model}, because the MaxEnt
solution without any constraints apart from normalization yields $p(x)=p_0(x)$.\\
The deeper reason why we have a default model in the continuous case and
not in the discrete case is that the transition by discretization from the
continuous to the discrete case is not unique.
The default model plays the role of an ignorant prior.
Otherwise, the implementation of testable information in the maximum entropy procedure is the same as in the discrete case.
On passing we want to discuss a simple application from statistical physics.
We want to infer the pdf $p(x)$ of particles at the height $x$ above the earth.
The mean potential energy of these particles is proportional to the mean
height.\[E=mg\int x p(x)\text{d}x \propto \langle x\rangle\]

In a canonical ensemble the mean energy is given by $k_BT$
So the constraint has the following form: \[ \langle x \rangle = \int xp(x)\text{d}x = \frac{k_BT}{mg}\]

Maximizing the entropy yields the \textbf{Barometric formula} $P\propto \exp(-\alpha x)$.
Note that $x$ is \textit{not
a scale variable here as there is an upper limit for it}. Therefore, a uniform prior
is adequate, which is corroborated by the experimental observation for free
particles without potential energy.\\

In closing, it should be mentioned that this form of entropy is also called
 \textbf{cross-entropy} or in other context the  \textbf{Kullback-Leibler distance}.
 \[d_{KL}(p,q)=\int_{-\infty}^{+\infty}p(x)log\left(\frac{p(x)}{q(x)}\right)\text{d}x\]
 It can be
considered as distance between distributions $p(x)$ and $p_0(x)$, and it plays
an important role in machine learning.\\

\fbox{\parbox{\linewidth}{\textbf{Question 7.} Which of the following statements is true?\\
a) We need a default model $p_0$ in the entropy $S$ in the continuous case, since the mapping from the continuous to the discrete case is not unique.\\
b) The height variable $x$ when deriving the barometric formula is uniform since the earth has a finite atmosphere.\\
c) We need a default model $p_0$ of the entropy $S$ in the continuous case, since variables are scale invariant.
}}
\\

\section*{The lighthouse problem}
The lighthouse problem that Lyra addressed in the adventure is another
example of the  \textbf{transformation of coordinates}. Let $x_0$ and $y_0$ be the coordinates
of the lighthouse and let the coast line define the $x$-axis with the origin of
the coordinate system being on the coast line.
The lighthouse emits light signals that are uniformly distributed in the rotation angle $\varphi$ about the lighthouse axis. We are interested in the pdf for a
light signal being detected at point $\boldsymbol{x}$ on the coast line.
The following relation exists between the coastline position x and the
angle $\varphi$, and the transformation law yields the desired pdf for $\boldsymbol{x}$ from the uniform pdf for
the angle.\\ %8_10
 \begin{figure}[H]
	\centering
	\includegraphics[width=0.75\textwidth]{8_10.png}
\end{figure}

$p(\boldsymbol{x}|x_0,y_0)$ is the probability to find a light signal at position $\boldsymbol{x}$ given the parameters
$x_0,y_0$ and therefore it represents the likelihood of the problem.
We assume for simplicity that the data points $\boldsymbol{x}_i$ are uncorrelated. Bayes
theorem then yields for the desired pdf for the position of the lighthouse the following form.
\[p(x_0,y_0|\boldsymbol{x})=\frac 1Z \prod_i p(\boldsymbol{x}_i|x_0,y_0)p(x_0,y_0)\]
The prior probability depends on our knowledge about lighthouses.
\textit{In the Pluto notebook you can gain experience with this problem and see that
it is indeed possible to locate the lighthouse on the basis of the distribution
of the light ﬂashes on the coastline.}

\section*{Important distributions}
There are many important distributions for continuous variables. Here we
only want to mention the most popular ones.
We have repeatedly used Bayes’ theorem to compute the posterior pdf for a
parameter a given some data d.
Let’s first ignore the prior $p(a)$, then the posterior is proportional to the likelihood $p(a|\boldsymbol{d})\propto p(\boldsymbol{d}|a)$.
In case of a counting experiment with a Poisson likelihood the parameter
$a$ corresponds to the mean $\mu$ and the data are the counts $\boldsymbol{N}$. Then the
posterior is of the following form:
\begin{equation*}\boxed{p(\mu|\boldsymbol{N})\propto e^{-\mu}\mu^\boldsymbol{N}
}\end{equation*}\\
This function is a  \textbf{Gamma-distribution}, that is defined for the range $x\in [0,\infty)$
and with the correct normalization it reads.%8_11
 \begin{figure}[H]
	\centering
	\includegraphics[width=0.75\textwidth]{8_11.png}
\end{figure}
It has a \textit{power-law dependence for small x} and \textit{decays exponentially for large
x}. Mean and variance are given by the following expressions.
\begin{equation*}\boxed{\langle x\rangle = \frac{\alpha}{\beta} \qquad \text{var}(x)=\frac{\alpha}{\beta^2}
}\end{equation*}\\
They can easily be derived from the Gamma function which is part of
the normalization.
\[\Gamma(\alpha):=\int_0^{\infty}t^{\alpha-1}e^{-t}\text{d}t\]
The Gamma function has a very convenient recursion relation
and for an integer argument it is related to the factorial: \[\Gamma(N)=(N-1)! \quad N\in\mathbb{N}\]

Moreover, we want to mention that $\Gamma(\frac 12)=\sqrt{\pi}$.
Special cases of the Gamma distribution are the  $\mathbf{\chi^2}=\Gamma(\frac n2, \frac 12)$ \textbf{distribution}
and the  \textbf{exponential distribution} $\Gamma(1, \beta)$.\\%

So far we have only discussed the likelihood which lead to the Gamma distribution. If we include an arbitrary prior, then the posterior will be different.
For that reason it is common practice to also use a Gamma distribution for
the prior. This is not a significant restriction, as prior knowledge is typically
vague and the Gamma distribution is very ﬂexible and can cover a wide range
of prior knowledge. Such a so-called  \textbf{conjugate prior} has the advantage that
the posterior is still a Gamma distribution and mean and covariance can be
computed analytically.\\

Now we turn to a Bernoulli experiment with sample size $N$. Here, the parameter a
corresponds to the intrinsic probability $q$ and the data is the number $K$ of
occurrences of the outcome of interest, so $p(a|d)=p(q|K,N)$. Then the posterior is of the following form:
\begin{equation*}\boxed{p(q|K,N))\propto q^K(1-q)^{N-K}
}\end{equation*}\\
This function is a  \textbf{Beta-distribution}, which has the range$x\in [0,1]$ and the
normalized distribution reads.%8_12
 \begin{figure}[H]
	\centering
	\includegraphics[width=0.75\textwidth]{8_12.png}
\end{figure}
Mean and variance
can be determined by means of the beta function.
It also enters the normalization of the beta distribution.
In case of a Bernoulli experiment the conjugate prior is therefore a Beta
distribution.\\

Finally, we want to mention the  \textbf{Gaussian distribution}, which we have already
encountered several times. The Gaussian is of central importance for the
description of experimental noise and the range of the random variable in
the one-dimensional case is the entire real axis. The normalized Gaussian has a mean and a variance which is given by the following expressions.\\%8_13
 \begin{figure}[H]
	\centering
	\includegraphics[width=0.75\textwidth]{8_13.png}
\end{figure}
\textit{See the Pluto notebook about popular distributions and study the possible
shapes that emerge by varying the parameters.}\\

\fbox{\parbox{\linewidth}{\textbf{Question 8.} Match the distributions: \{Gauss, Gamma, Beta\}\\
The domain of the .............. distribution is $x\in[0,1]$, as it describes the probability of a probability.\\
The ......... distribution is the conjugate prior of a Poisson Likelihood.\\
The .......... distribution is parametrized by mean and variance.
}}
\\

\section*{Benford's law}
Lyra in the adventure also mentioned a very interesting discovery that \textit{the
leading digits of large numbers form a pattern} no matter what is considered: %8_14
 \begin{figure}[H]
	\centering
	\includegraphics[width=0.75\textwidth]{8_14.png}
\end{figure}
For the amount of collected shells, bottles, coloured stones per day, the leading numbers all behave the same and
they are  \textbf{not} uniformly distributed. This is in contradiction to the principle
of indifference.\\
This phenomenon has to do with  \textbf{scale invariances}. We have seen before that
the pdf of scale invariant quantities $x$ is given by Jeffrey’s prior $p(x)\propto \frac 12$.
Put differently the pdf for log($x$) is uniform, so for $y=\log_{10}(x)\rightarrow p(y)= \text{const}$.
If you recall the intervals on logarithmic axes you may have noticed that the
interval sizes \textit{decrease} from 1 to 9. %
Let’s consider numbers x between 1 and 9.99999$\ldots$

The leading digit corresponds to the interval $x \in [1,2)$.
Then the probability is $P_1\propto \int_1^2\frac{\text{d}x}{x}\ln(2)-\ln(1)$.
In general, the probability that the leading digit has the value $I$ is proportional to $\log(I+1)-\log(I)$.
With the correct normalization for all possible digits 1 up to 9 we have the probability that the leading digit is $I$ given by $\log_{10}(I+1)-\log_{10}(I)$
This distribution is depicted in the figure.\\%8_15
 \begin{figure}[H]
	\centering
	\includegraphics[width=0.75\textwidth]{8_15.png}
\end{figure}

The same result is obtained if the numbers are in the interval 10 to 99.9999
or a 100 to 999.999 and so on.
Actually, the same law holds true also for \textit{counting experiments when the
numbers vary over many orders of magnitude}. Also in this case the probability mass function is \textit{uniform on a logarithmic scale} or  \textbf{log-uniform} as it is
sometimes called.\\

\fbox{\parbox{\linewidth}{\textbf{Question 9.} Which of the following variables could be scale invariant?\\
a) Rain drops per day/km$^2$\\
b) The results of repeated dice throws, using a 100 faced die\\
c) The number of sand grains in your shoes
}}
\\

\textit{You will find a Pluto notebook to perform random experiments with log-uniform distributions.}\\

This concludes lesson 8. We have learned about continuous variables and
how to assign prior probabilities using the principle of transformation invariance or the maximum entropy principle. We have discussed Bertrand's paradox and know now why Benford’s law
describes the leading digit of counting lists and measurement tables.
Check out the interactive Pluto notebooks to solve the lighthouse problem,
to do your own Benford computer experiments, and to experiment with
various popular distributions. Feel free to ask questions in the forum and feel encouraged to test your knowledge in the quiz.

\vspace{2cm}
\begin{minipage}[t]{1\textwidth}
	\raggedleft
	\centering
	\includegraphics[width = 0.20\textwidth]{CC-BY_icon}
	\vspace{0.2cm}

	\centering
	{\large ITPCP, TU Graz} \\
	https://creativecommons.org/licenses/by/4.0/legalcode
\end{minipage}
\end{document}


\documentclass[10pt,fleqn]{article}
\usepackage{amsmath}
\usepackage{cancel}
\usepackage[letterpaper, landscape, margin=1in]{geometry}

\setlength{\parindent}{0pt}

\begin{document}

\section{Isotropic visco-acoustic constant density second order self-adjoint system}
John Washbourne, Ken Bube\\
September 20, 2013

\section{Introduction}
This note shows the derivation of \textit{time update equations} and the linearization
for the isotropic constant density self-adjoint system. We implement attenuation with a
monochromatic approximation to Maxwell bodies, and use this attenuation model to
implement zero outgoing absorbing boundary conditions on the exterior of the modeling
domain.
\vspace{10pt}

The time update equations are used to advance solutions in time, expressing the pressure
wavefield at time $p_{(t + \Delta)}$ as a function of $p_{(t - \Delta)}$ and $p_{(t)}$.

\subsection{Symbols}
\begin{center}
	\begin{tabular}{ll}                                                                      \\[-10pt]
		$\partial_t$    & $\displaystyle \frac{\partial }{\partial t}$            \\[10pt]
		$\nabla p $     & Laplacian: $ \left(
			\displaystyle \frac{\partial^2 }{\partial x^2} +
			\displaystyle \frac{\partial^2 }{\partial y^2} +
			\displaystyle \frac{\partial^2 }{\partial z^2} \right) p $                \\[15pt]
		$\Delta$        & Temporal sampling rate                                  \\[10pt]
		$\omega$        & reference frequency for attenuation                     \\[10pt]
		$Q$             & attenuation at frequency $\omega$                       \\[10pt]
		$P$             & Pressure wavefields                                     \\[10pt]
		$S(x,y,z,t)$    & Pressure source term                                    \\[10pt]
		$b$             & buoyancy = $\displaystyle 1/\rho $ (reciprocal density) \\[10pt]
		$ \{\ V_p\ \} $ & Material parameters                                     \\[10pt]
	\end{tabular}
\end{center}

\newpage
\subsection{Modeling system}
Equation \ref{eq:system} shows the modeling system with absorbing boundaries implemented
using amplitude only (dissipation only, no dispersion) Q. \vspace{10pt}

We apply the time derivative in the term $\displaystyle \frac{\omega}{Q} \partial_t p$
using a backward one-sided numerical difference. We tested both forward one-sided and
centered difference alternatives and found them to be less stable.

\begin{equation}
	\frac{1}{v^2} \left( \partial_t^2 p + \frac{\omega}{Q} \partial_t p \right) = \nabla p + S
	\label{eq:system}
\end{equation}

\section{Time update equations}

\subsection{Time update numerical difference formulas, first and second order}
\begin{equation}
	\partial_t p = \frac{1}{\Delta} \left[ p_{(t)} - p_{(t - \Delta)} \right]
	\label{eq:diff1}
\end{equation}

\begin{equation}
	\begin{aligned}
		\partial_t^2 p & = \frac{1}{\Delta^2} \left[ p_{(t+\Delta)} - 2 p_{(t)} + p_{(t - \Delta)} \right] \\[10pt]
		p_{(t+\Delta)} & = \Delta^2 \partial_t^2 p + 2 p_{(t)} - p_{(t - \Delta)}
	\end{aligned}
	\label{eq:diff2}
\end{equation}

\subsection{Rearrange equation \ref{eq:system} for $\partial_t^2 p$}
\begin{equation}
	\partial_t^2 p = v^2 \left( \nabla p + S \right) - \frac{\omega}{Q} \partial_t p
\end{equation}

\subsection{Apply equations \ref{eq:diff1} and \ref{eq:diff2}, and rearrange}
\begin{equation}
	p_{(t+\Delta)} = \Delta^2 v^2 \left( \nabla p + S \right) - \Delta \frac{\omega}{Q} \left[ p_{(t)} - p_{(t - \Delta)} \right]
	+ 2 p_{(t)} - p_{(t - \Delta)}
\end{equation}

\newpage
\section{Linearization and Born modeling equation}

\subsection{Nonlinear modeling equation}
\begin{equation}
	\frac{1}{v^2} \left( \partial_t^2 p + \frac{\omega}{Q} \partial_t p \right) = \nabla p + S
\end{equation}

\subsection{Taylor expand $\displaystyle \frac{1}{v^2} \rightarrow \left( \frac{1}{v_0^2} - \frac{2}{v_0^3} \delta v \right) $
	and replace $\displaystyle p \rightarrow \displaystyle (p_0 + \delta p)$}
\begin{equation}
	\left( \frac{1}{v_0^2} - \frac{2}{v_0^3} \delta v \right) \left( \partial_t^2 p_0 + \partial_t^2 \delta p +
	\frac{\omega}{Q} \partial_t p_0 + \frac{\omega}{Q} \partial_t \delta p \right) =
	\nabla p_0 + \nabla \delta p + S
\end{equation}

\subsection{Expand}
\begin{equation}
	\frac{1}{v_0^2} \partial_t^2 p_0
	+ \frac{1}{v_0^2} \partial_t^2 \delta p
	+ \frac{1}{v_0^2} \frac{\omega}{Q} \partial_t p_0
	+ \frac{1}{v_0^2} \frac{\omega}{Q} \partial_t \delta p
	- \frac{2}{v_0^3} \delta v \partial_t^2 p_0
	- \frac{2}{v_0^3} \delta v \partial_t^2 \delta p
	- \frac{2}{v_0^3} \delta v \frac{\omega}{Q} \partial_t p_0
	- \frac{2}{v_0^3} \delta v \frac{\omega}{Q} \partial_t \delta p
	= \nabla p_0 + \nabla \delta p + S
\end{equation}

\subsection{Cancel reference terms}
\begin{equation}
	\cancel{ \frac{1}{v_0^2} \partial_t^2 p_0 }
	+ \frac{1}{v_0^2} \partial_t^2 \delta p
	+ \cancel{ \frac{1}{v_0^2} \frac{\omega}{Q} \partial_t p_0 }
	+ \frac{1}{v_0^2} \frac{\omega}{Q} \partial_t \delta p
	- \frac{2}{v_0^3} \delta v \partial_t^2 p_0
	- \frac{2}{v_0^3} \delta v \partial_t^2 \delta p
	- \frac{2}{v_0^3} \delta v \frac{\omega}{Q} \partial_t p_0
	- \frac{2}{v_0^3} \delta v \frac{\omega}{Q} \partial_t \delta p
	= \cancel{ \nabla p_0 } + \nabla \delta p + \cancel{ S }
\end{equation}

\subsection{Zero terms second order in perturbations $\delta p,\delta v$ }
\begin{equation}
	\frac{1}{v_0^2} \partial_t^2 \delta p
	+ \frac{1}{v_0^2} \frac{\omega}{Q} \partial_t \delta p
	- \frac{2}{v_0^3} \delta v \partial_t^2 p_0
	- \cancel{ \frac{2}{v_0^3} \delta v \partial_t^2 \delta p }
	- \frac{2}{v_0^3} \delta v \frac{\omega}{Q} \partial_t p_0
	- \cancel{ \frac{2}{v_0^3} \delta v \frac{\omega}{Q} \partial_t \delta p }
	= \nabla \delta p
\end{equation}

\subsection{Rearrange for the Born modeling equation}
\begin{equation}
	\begin{aligned}
		\frac{1}{v_0^2} \left( \partial_t^2 \delta p + \frac{\omega}{Q} \partial_t \delta p \right)
		 & = \nabla \delta p + \frac{2}{v_0^3} \delta v \left( \partial_t^2 p_0 + \frac{\omega}{Q} \partial_t p_0 \right) \\[10pt]
		 & = \nabla \delta p + \frac{2}{v_0^3} \delta v \left( v_0^2 \nabla p_0 + v_0^2 S \right)                         \\[10pt]
	\end{aligned}
\end{equation}

Note: it is procedurally simplest to serialize the quantity
$\displaystyle \left( v_0^2 \nabla p_0 + v_0^2 S \right)$ when we perform nonlinear
forward modeling, and use that field as the Born source for the linearized forward and
adjoint operators.
\end{document}


\section{Undirected graphs}


\section{Part 6 Classifying Gamma-ray Bursts}

The file of the functions used for this exercise is:

\lstinputlisting{part_six.py}

The result of this function is shown in Figures \ref{fig:mse}, \ref{fig:misclassify}, \ref{fig:predction}.
Figure \ref{fig:mse} shows the Mean Square Error for the 100000 epochs the network was trained on. The network as a whole
takes a lot longer to train, but would take too long for the 10 minute limit, and so the pre-trained weights, and training history, are loaded from disk.
Figure \ref{fig:misclassify} is similar for the misclassification of the training examples. In this case,
all exampels were used for training and predictions, and the network looked at the redshift and the mass of the host galaxy to determine if
a GRB was long or short.

\begin{figure}[h!]
  \centering
  \includegraphics[width=0.9\linewidth]{./plots/GRB_MSError.png}
  \caption{Mean Square Error during training on GRB.}
  \label{fig:mse}
\end{figure}

\begin{figure}[h!]
  \centering
  \includegraphics[width=0.9\linewidth]{./plots/GRB_Misclassification.png}
  \caption{GRB Misclassification Rate over training.}
  \label{fig:misclassify}
\end{figure}

\begin{figure}[h!]
  \centering
  \includegraphics[width=0.9\linewidth]{./plots/GRB_Histogram.png}
  \caption{GRB Histogram Real Class vs Predicted}
  \label{fig:predction}
\end{figure}

As can be seen in \ref{fig:predction}, the logistic regression does not do a great
job of separating the two classes, and there does not seem to be enough
information for the network to properly learn a hyperplane that can
separate the two classes. The network seems to only predict that a GRB is a long GRB without predicting any short ones.
The reason for this could be that there is a significant amount of missing data, and many more long GRBs vs short GRBs in
the dataset. This is supported by the mean square error loss, and misclassification rate, which show the network quickly learning
to predict all long GRBs and then stay fairly consistent after that.

Missing data is not
changed at all, with the idea that the network will learn to ignore the -1. values as not
being correlated with the short or long classification. From the Universal Approximation Function,
neural networks can approximate any function, and so this network could theoretically learn to differentiate between
the long and short GRBs given enough time, or enough examples, which were not available for this project.


%!TEX root = paper.tex
%
% Riemann
%

\subsection{Riemann: A Community-Driven Collection of Approximate Riemann
Solvers}\label{sec:riemann}

The methods implemented in \clawpack, and all modern Godunov-type
methods for hyperbolic PDEs, are based on the solution of Riemann
problems as discussed in \cref{sec:hyp}.  Whereas most existing codes
for hyperbolic PDEs use Riemann solvers to compute fluxes, \clawpack
Riemann solvers instead compute the waves (or discontinuities) that
make up the Riemann solution.  In the unsplit algorithm, \clawpack
also makes use of {\em transverse} Riemann solvers, responsible for
computing transport between cells that are only corner (in 2d) or edge
(in 3d) adjacent.

For nonlinear systems, the exact solution of the Riemann problem is
computationally costly and may involve both discontinuities (shocks and contact
waves) and rarefactions.  It is almost always preferable to employ inexact
Riemann solvers that approximate the solution using discontinuities only, with
an appropriate entropy condition.  The solvers available in \clawpack are
all approximate solvers, although one could easily implement their own exact
solver and make it available in the format needed by \clawpack routines.

A common feature in all packages in the \clawpack suite is the
use of a standard interface for Fortran Riemann solver routines.  This ensures that new
solvers or solver improvements developed for one package can immediately
be used by all packages.  To further facilitate this sharing and to avoid
duplication, Riemann solvers are (with rare exceptions) not maintained under
the other packages but are collected in a single repository named
\texttt{riemann}.
Users who develop new solvers for \clawpack are encouraged to submit them to the
Riemann repository.

In the Fortran-based packages (Classic, AMRClaw, and GeoClaw) the Riemann solver
is selected at compile-time by modifying a problem-specific Makefile. In
\pyclaw, the Riemann solver to be used is selected at run-time.  This is made
possible by compiling all of the Riemann solvers (when \pyclaw is installed) and
generating Python wrappers with \texttt{f2py}.  For \pyclaw, \texttt{riemann}
also
provides metadata (such as the number of equations, the number of waves,
and the names of the conserved quantities) for each solver so that setup is
made more transparent.


\section{Results}
\label{Resul}

The following Section summarises the work by presenting the obtained results. Comments on the goodness of such results are also present, together with possible future developments.
The popularity distribution computed through Eq. (\ref{Popc}) shows a gaussian-like distribution centred in 0 (Fig. \ref{PopDist}), meaning that on average comments are neutral, with lateral tails of comments with strong polarity, both positive and negative. Recommendations play a small role, because of some outliers causing the normalised recommendation count to be quite a small number. It is fundamental for this distribution to be unbiased as it will be used to estimate controversy.

\begin{figure}[b]
\centering
    \begin{subfigure}{0.5\tw}
    \centering
    \includegraphics[height=4.5cm]{Pictures/PopularityDist.pdf}
    \caption{Popularity distribution}
    \label{PopDist}
    \end{subfigure}%
\hfill
    \begin{subfigure}{0.5\tw}
    \centering
    \includegraphics[height=4.5cm]{Pictures/articleScoreDist.pdf}
    \caption{Article score distribution.}
    \label{AScore}
    \end{subfigure}
\caption{Preliminary distributions.}
\end{figure}

The article score distribution shows a similar behaviour; it is reported in Fig. \ref{AScore} even though it will not be included in further analyses. Again, the article score calculated with Eq. \ref{ASeq} shows a quite symmetrical distribution, meaning that the estimator is consistent and the articles are equally distributed among the polarity spectrum.

\subsection{Strategy 1 (linear)}
\label{Strategy1}
The first strategy to estimate controversy in news articles (Eq. \ref{Str1}) was to multiply the number of comments related to each article with the IQR of the popularity distribution for such comments (labelled as \textit{debate}). This calculation lead to an index of controversy peaked near 0. Furthermore, the controversy distribution looks very similar to the comment number distribution (see Fig. \ref{Str1Dist}). This suggests that popularity and controversy are strongly correlated, while debate plays an insignificant role in the compute.

\begin{figure}[h]
\centering
\includegraphics[width=\tw]{Pictures/Strat1Dist.pdf}
\caption{Popularity, Debate and Controversy distributions for Strategy 1}
\label{Str1Dist}
\centering
\includegraphics[width=\tw]{Pictures/Strat1Corr.pdf}
\caption{Correlation plots for Strategy 1.}
\label{Str1Corr}
\end{figure}

To corroborate this hypothesis, the correlation scatter plots for the three variables were built, and Fig. \ref{Str1Corr} shows a very strong linear correlation between popularity and controversy. This correlation rows against the thesis of the project, which is trying to differentiate the so-called "hot topics" from the most controversial ones. For this reason, another strategy was implemented to see if some other definition of controversy can detach more efficiently from the concept of popularity.

To go deeper into the analysis, the controversy scores for each articles were averaged among articles of the same section. This grouping was made to see if any article section is more controversial than others. Unfortunately, Strategy 1 did not prove to be useful in this case, as the most controversial topics are also the most popular one. Figure \ref{RG1} shows how, for each section, the controversy and the popularity score look very similar to each other (after proper normalisation).

\begin{figure}[htb]
\centering
\includegraphics[width=\tw]{Pictures/Strat1SN.pdf}
\caption{Most controversial sections for Strategy 1}
\label{RG1}
\centering
\includegraphics[width=\tw]{Pictures/Strat1ND.pdf}
\caption{Most controversial desks for Strategy 1}
\label{RG1nd}
\end{figure}

After that, articles were also grouped by the "new desk" index, to see if there is any category that stands out as the most controversial. Again, results (Fig. \ref{RG1nd}) indicate a high similarity between controversy and popularity, with "National" desk as the most popular desk.

%\begin{figure}[hbt]

%\end{figure}

\subsection{Strategy 2 (log)}

Strategy 2 consisted on applying the natural logarithm to the number of comments under each article, in order to dampen the strong dependencies between the former and controversy. Thanks to this new strategy, results change significantly, and they are reported in the same format as Section \ref{Strategy1}.
First of all, the popularity distribution (hence the controversy index) are represented in their new shapes, while the debate distribution obviously does not change (Fig. \ref{Str2Dist}).
By looking at the correlation plots, it can be seen that the strong linear correlation between the two distributions has been removed (Fig. \ref{Str2Corr}). The problem is now that distributions are not smooth for small values of popularity, because of the logarithm. However, the distribution of controversy looks smoother and more balanced compared to the one adopted in Strategy 1.

\begin{figure}[h]
\centering
\includegraphics[width=\tw]{Pictures/Strat2Dist.pdf}
\caption{Popularity, Debate and Controversy distributions for Strategy 2.}
\label{Str2Dist}
\end{figure}

\begin{figure}[h]
\centering
\includegraphics[width=\tw]{Pictures/Strat2Corr.pdf}
\caption{Correlation plots for Strategy 2.}
\label{Str2Corr}
\end{figure}

If we look at the averages on categories (Fig. \ref{RG2} and \ref{RG2nd}), it can be seen that popularity and controversy are now two very different quantities. The bar plots both for section name and desk show different behaviour between the two data series. However, the problem with Strategy 2 is that the categories labelled as "highly controversial" are quite unexpected, such as Cycling and Entertainment. On the other hand, controversial categories in Strategy 1 were Politics, Editorials and Middle East (categories one would expect to be controversial).

\begin{figure}[h]
\centering
\includegraphics[width=\tw]{Pictures/Strat2SN.pdf}
\caption{Most controversial sections for Strategy 2}
\label{RG2}
\end{figure}

\begin{figure}[h]
\centering
\includegraphics[width=\tw]{Pictures/Strat2ND.pdf}
\caption{Most controversial desks for Strategy 2}
\label{RG2nd}
\end{figure}

\subsection{Keyword extraction}
Lastly, a feature extraction was performed to check which words were common in comments related to the most controversial articles (top 10\%). A corpus vectorisation was performed via a {\tt TfidfVectorizer}, and then a $\chi2$ analysis was invoked to select words with higher frequency. Since TF-IDF tends to boost the score of words which are very uncommon in the corpus, among the "top controversial words" there were many nonsensical words that occurred only once in the whole corpus, probably originated by typos of the comment authors. For this reason, only the words with a match in the AFINN dictionary were kept. Figure \ref{TopWords} depicts the words selected by the $\chi^2$ analysis. An ample fraction of the word list is made of polarised words, indicating that polarity is at the basis of controversy.

\begin{figure}[h!]
\centering
\includegraphics[width=\linewidth]{Pictures/TopWs.png}
\caption{Most controversial words in the comment dataset.}
\label{TopWords}
\end{figure}

\section{Conclusions}

The aim of this project was to define controversy in news articles by analysing the comments below the articles. Two different strategies were adopted, each one with its pros and cons.
Strategy 1 defined controversy through a linear relationship; it was the one who guaranteed the results with most common sense (Politics and Middle East were among the top categories), but it failed to detach from popularity.
On the other hand, strategy 2 guaranteed a significant shift from popularity and removed linear correlations, but the results leave us with some doubts.
In any case, it was proved that in either case polarity in the comments is the key to analyse controversy.
Some future developments to this work could be:
\begin{itemize}
\item Perform aspect-based sentiment analysis to go deeper into the comments, and analyse which aspects within a given comment generated controversy
\item Using topic modelling, extract the article section name from the comments, since nearly half of the articles in the dataset were provided with "Unknown" section name
\item Demonstrate the robustness of this controversy estimator by changing the lexicon (for example, WordNet could be used instead of AFINN) and look for similar behaviour of the controversy distribution
\item Find another strategy to remove correlation between popularity and controversy without the need to apply a logarithm
\item Train a classifier able to predict controversy in other datasets of news and comments
\end{itemize}


\section{Structural and Selection Verbs}

Structural verbs, such as {\em catenate}, {\em laminate},
{\em rotate}, and {\em reverse}, and selection verbs, such as
{\em from} (indexing), {\em merge}, {\em take}, and {\em drop},
typically use word-at-a-time algorithms on Boolean (and other) data.
This is facilitated by use of a Boolean vector copying verb,
able to handle source and sink array indices that are not
on word or byte boundaries.
For example, a Boolean catenate, such as {\apl z\qlarrow\0x,y},
often requires copying of elements of {\apl y} into {\apl z}
without overwriting elements of {\apl x} that have already
been copied into {\apl z}. The standard libraries supplied with C
do not have movers that do this. The earliest such mover
that the author is aware of is Luther J. Woodrum's {\em ljwmove},
used in \ibmapl. The author wrote {\em rbemove}, a generalized stride-one
copy verb, including an extended version of
Woodrum's algorithm. This allowed it to be used safely for
indexed assign into an extant Boolean array, without overwriting
adjacent array elements.\footnote{Contemporary microprocessor architectures
usually include bit-manipulation instructions, which may simplify
writing such a mover.} The {\em rbemove} verb also provided
support for type conversions on stride-one data, thereby
simplifying primitives such as catenate.

For verbs operating on non-trailing array axes, further
performance improvements can often be made by moving entire
sub-arrays at once. {\em E.g.}, a first-axis rotate, such as
{\apl 1\qrotf\02~3~4\qrho\qiota\024} can move 12
adjacent array elements at once:
\medskip

{\apl T\qlarrow\02~3~4\qrho\qiota\024\\
~~0~~1~~2~~3\\
~~4~~5~~6~~7\\
~~8~~9~10~11\\
~\\
12~13~14~15\\
16~17~18~19\\
20~21~22~23\\}

{\apl 1\qrotf\0T\\
12~13~14~15\\
16~17~18~19\\
20~21~22~23\\
~\\
~~0~~1~~2~~3\\
~~4~~5~~6~~7\\
~~8~~9~10~11\\}

\medskip

In 1979, the author, in the course of implementing enclosed arrays
for SHARP APL, designed new algorithms for rotate and
reverse that worked in this manner.~\cite{RBernecky:fastrotrev}
The algorithm for last-axis reverse on Booleans reversed a byte at a time
of an entire matrix row, via table lookup on byte-aligned data:

\medskip
{\apl RevTab\qlbr\0uint8~\qomega\qrbr}
\medskip

\noindent This result was not guaranteed to be aligned properly, so
it then bit-aligned the resulting vector into the final result,
a word at a time. The {\apl uint8} verb, shown in Section~\ref{utilityverbs},
can be considered a typecast that
converts its Boolean argument into unsigned, 8-bit integers.
The performance of the non-optimized and optimized rotate code
is shown in Figure~\ref{matrix_rotate_first_axis.jpg}.

\subsection{Reshape}

The reshape verb supports argument element reuse for array generation.
For example:

\medskip
{~~~~~~\apl 8\qrho\01~0~0}\\
{\apl 1~0~0~1~0~0~1~0}
\medskip

\noindent Larry Breed designed and implemented SIMD reshape-with-reuse
by copying the right argument to the result,
then repeatedly catenating that result to itself, doubling its
length at each step, until its tail was byte-aligned,
at which point an overlapped move (``smear") of the argument to
its tail would generate the final result.~\cite{LMBreed:pc2014}

Dyalog APL does not use the smear approach. Rather, it performs
repeated catenation until the final result is generated.

%FIXME fetch bit from word (32 times), store into word, with
%flag bit signaling sink word full


% $Id:$ %
\screenshot{main_menu/images/ss-recording-settings}{The recording settings screen}{}

\note{To change the location where recordings are stored open the
      \setting{Context Menu} (see \reference{ref:Contextmenu}) on the directory
      where you want to store them in the \setting{File Browser} and select
      \setting{Set As Recording Directory}.}

\opt{masf}{
  \section{Quality}
    Choose the quality here (0 to 7). Default is 5, best quality is 7,
    smallest file size is 0. This setting affects how much your sound
    sample will be compressed. Higher quality settings result in larger
    MP3 files.

    The quality setting is just a way of selecting an average bit rate,
    or number of  bits per second, for a recording.  When  this setting
    is lowered, recordings are compressed more (meaning worse sound quality),
    and the average bitrate changes as follows.

  \begin{table}[h!]
      \begin{rbtabular}{0.75\textwidth}{lX}%
        {\emph{Frequency} & \emph{Bitrate} (Kbit/s) -- quality 0$\rightarrow$7}{}{}
        44100~Hz stereo   & 75, 80, 90, 100, 120, 140, 160, 170 \\
        22050~Hz stereo   & 39, 41, 45, 50,  60,  80,  110, 130 \\
        44100~Hz mono     & 65, 68, 73, 80,  90,  105, 125, 140 \\
        22050~Hz mono     & 35, 38, 40, 45,  50,  60,  75,  90 \\
      \end{rbtabular}
  \end{table}
}

\opt{swcodec}{
  \section{Format}
    Choose which format to save your recording in. The available choices are
    the two uncompressed formats \setting{PCM Wave} and \setting{AIFF}, the
    losslessly compressed \setting{WavPack} and the lossy
    \setting{MPEG Layer~3}.

  \section{Encoder Settings (MP3 only)}
    This sets the bitrate when using the \setting{MPEG Layer~3} format.
}

  \section{Frequency}
   \nopt{ipodnano,ipodcolor,ipod4g}{
    Choose the recording frequency (sample rate).
    \opt{c200,e200}
        {24~kHz, 22.05~kHz and 16~kHz}
    \opt{iriverh100,iriverh300}
        {44.1~kHz, 22.05~kHz and 11.025~kHz}
    \opt{iaudiom3,iaudiom5,iaudiox5,vibe500}
        {88.2~kHz, 44.1~kHz, 22.05~kHz and 11.025~kHz}
    \opt{masf}
        {48~kHz, 44.1~kHz, 32~kHz, 24~kHz, 22.05~kHz and 16~kHz}
    \opt{iriverh10}
        {96~kHz, 88.2~kHz, 48~kHz, 44.1~kHz, 32~kHz and 8~kHz}
    \opt{samsungyh}
        {48~kHz, 44.1~kHz, 32~kHz, 24~kHz, 22.05~kHz, 16~kHz, %
        11.025~kHz and 8~kHz}
    \opt{gigabeats,ipodvideo}
        {48~kHz, 44.1~kHz, 32~kHz, 24~kHz, 22.05~kHz, 16~kHz, 12~kHz, %
        11.025~kHz and 8~kHz}
    \opt{sansaAMS,ipodnano2g,fuzeplus}
        {96~kHz, 88.2~kHz, 64~kHz, 48~kHz, 44.1~kHz, 32~kHz, 24~kHz, %
        22.05~kHz, 16~kHz, 12~kHz, 11.025~kHz and 8~kHz}
    are available. Higher sample rates use up more disk space, but give better
    sound quality.
    \opt{swcodec}{\note{The 11.025~kHz setting is not available when using %
      \setting{MPEG Layer~3} format.}
    }
    \opt{masf}{
      The frequency setting also determines which version of the MPEG standard
      the sound is recorded using:\\
      MPEG~1 for 48~kHz, 44.1~kHz and 32~kHz.\\
      MPEG~2 for 24~kHz, 22.05~kHz and 16~kHz.\\
    }
    \opt{recording_digital}
      {\note{You cannot change the sample rate for digital recordings.}
    }
   } % nopt ipodnano,ipodcolor,ipod4g
   \opt{sansa}{
     \nopt{sansaAMS,c200,e200,fuzeplus}
      Recordings can only be made at a 22.05~kHz frequency (sample rate)
      on this \dap.
   } % prepared for all sansas not handled above, e.g. sansaview
   \opt{ipodnano,ipodcolor,ipod4g}{
      Recordings can only be made at a 44.1~kHz frequency (sample rate)
      on this \dap.
   } % opt ipodnano1g

\section{Source}
  Choose the source of the recording. The options are:
  \opt{recording_digital}{\setting{Digital} (SPDIF), }%
  \opt{recording_mic}{\setting{Microphone}}%
  \opt{recording_hwcodec}{ and \setting{Line~In}.}%
  \opt{recording_swcodec}{%
      \nopt{radio}{\opt{recording_linein}{ and \setting{Line~In}.}}%
      \opt{radio}{%
          \opt{recording_linein}{%
              \opt{recording_mic}{, }%
              \setting{Line~In} and %
          }%
          \nopt{recording_linein}{%
              \opt{recording_mic}{ and }%
          }%
          \setting{FM~Radio}. For more information on recording from the
          radio see \reference{ref:FMradio}.%
      }
  }

\section{Channels}
  This allows you to select mono or stereo recording. Please note that
  for mono recording, only the left channel is recorded. Mono recordings
  are usually somewhat smaller than stereo.

\opt{swcodec}{
  \section{Mono Mode}
    When configured to record to mono and the source is a stereo signal, use this
    setting to configure how the mono signal is created. Options are L, R and L+R.
}

\opt{masf}{
  \section{Independent Frames}
    The independent frames option tells the \dap{} to encode with the bit
    reservoir disabled, so the frames are independent of each other. This
    makes a file easier to edit.
}

\section{File Split Options}
  This sub menu contains options for file splitting, which can be used to split
  up long recordings into manageable pieces. The splits are seamless (frame
  accurate), no audio is lost at the split point. The break between recordings
  is only the time required to stop and restart the recording, on the order of
  2 -- 4 seconds.
  \begin{description}
    \item[Split Measure.]
      This option controls whether to split the recording when the
      \setting{Split Filesize} is reached or when the
      \setting{Split Time} has elapsed.

    \item[What to do when Splitting.]
      This controls what will happen when the splitting condition is
      fulfilled the two available options here are
      \setting{Start a new file} or \setting{Stop recording}.

    \item[Split Time.]
      Set the time to record between each split, if time is used as
      \setting{Split Measure}.\\
      Options (hours:minutes between splits): Off, 00:05, 00:10, 00:15, 00:30,
      1:00, 1:14 (74 minute CD), 1:20 (80 minute CD), 2:00, 4:00, 8:00, 10:00,
      12:00, 18:00, 24:00.

    \item[Split Filesize.]
      Set the filesize to record between each split, if filesize is used as
      \setting{Split Measure}.

  \end{description}

\section{Prerecord Time}
    This setting buffers a small amount of audio so that when the record button
    is pressed, the recording will begin from that number of seconds earlier.
    This is useful for ensuring that a recording begins before a cue that is
    being waited for.

\section{Clear Recording Directory}
    Resets the location where the recorded files are saved to the root of your
    \daps{} drive.

\nopt{ondio}{
  \section{Clipping Light}
    Causes the backlight to flash on when clipping has been detected.\\
    Options: \setting{Off}, \setting{Main unit only},
    \setting{Main and remote unit}, \setting{Remote unit only}.
}
\section{Trigger}
  When you record a source you often are only interested in the sound and not
  the silence in between. The recording trigger provides you with a
  tool to automatically distinguish between sound and silence and record the
  sound only.  Unfortunately it is not very easy to make this distinction between
  silence and sound because you hardly ever encounter real silence. There always
  are background noises. What is considered as background noise depends on the
  situation. For example during a lecture the very low noise of rustling paper
  might be considered as background noise. During a rock concert the murmur of
  the audience might be considered background noise which is much louder compared
  to rustling paper. Also the duration of the signal matters. When you record
  speech you want to record every syllable. When you record live music you may
  not be interested in that chord the guitarist plays for two minutes before
  the show to verify his amp is turned on. The trigger features numerous
  parameters to adapt its behaviour to the desired situation.
  \begin{description}
  \item[Trigger.]
    This parameter specifies the trigger mode.  When set to \setting{Off}
    the recording must be started manually and apart from the Prerecord time no
    other parameter has any effect.  \setting{Once} will have the trigger start
    one recording only; after the recording has finished the input signal will
    not start another recording. \setting{Repeat} will have the trigger start
    multiple recordings.

  \item[Trigtype.]
    \fixme{Add description of Trigtype}
    Options: \setting{Stop}, \setting{Pause}, \setting{New File}.

  \item[Prerecord Time.]
    This specifies the time that is included into the recording before the
    trigger event occurs. This is very useful if you record a signal that
    fades in. Usually you want to set the prerecord time greater than or
    equal to the start duration. That ensures that you record the entire
    sound. Strictly speaking the prerecord time is not a special parameter
    of the trigger. It is available during normal recordings too.

  \item[Start Above.]
    The start threshold defines the minimal volume a sound must have to start the
    recording. It is displayed numerically in the line ``Start Above''. Note that
    the unit of the threshold depends on the settings of the peak meter. (i.e.
    When the peak meter displays dB you can adjust the level in dB and when the
    peak meter is set to linear the threshold is displayed as percentage.) In the
    peak meter at the bottom of the screen the start threshold is displayed
    graphically by a little triangle pointing to the right. There are two special
    values. The value \setting{Off} turns the start condition off.  With this
    setting you have to start the recording manually and the trigger only stops
    the recording according to the stop condition. The setting \setting{-inf}
    sets the trigger to the absolute minimum. This setting only makes sense when
    you record via a digital input as even the noise of the device itself would
    exceed this threshold immediately.

  \item[for at least.]
    The start duration defines the minimal duration that a signal must exceed the
    start threshold to start the recording. Depending on your situation you may
    want to set this setting to 0 (e.g. when copying a song from a commercial
    medium) or to quite big values. Because sound is not continuous by nature
    (think of percussion) negligible dropouts are tolerated during this start
    duration.

  \item[Stop Below.]
    When the sound level drops below the stop threshold the recording is stopped.
    It is displayed numerically in the line ``Stop Below''. Just like the start
    threshold the unit of the stop threshold depends on the settings of the peak
    meter. There's also a small triangular marker in the peak meter at the bottom
    of the screen. In contrast to the start threshold marker it points to the
    left. The value \setting{Off} turns the stop condition off. With this setting you
    have to stop the recording manually.

  \item[for at least.]
    This time specifies the duration the signal must drop below the stop
    threshold to stop the recording. By selecting high values you can ensure
    that, for example, trailing fade-outs are recorded entirely.

  \item[Presplit Gap.]
    When the signal drops below the stop threshold for the time specified by the
    presplit gap a new recording may be started when the signal raises above the
    start threshold. Thus the value of the presplit gap should be smaller than
    the stop hold time. Otherwise the recording would stop anyway and the
    presplit gap has no effect. For most uses I recommend to set this parameter
    equal to the stop hold time. Sometimes you may encounter a sound source (e.g.
    a CD) where the songs have fade outs and hardly any gaps between the tracks.
    Here you can set the stop hold time to long values to ensure that all fade
    outs are recorded completely. By specifying a short presplit gap you still
    can split the recording into separate tracks whenever the trigger start
    condition is met.

  \end{description}

More information can be found at \wikilink{VolumeTriggeredRecording}.

\opt{agc}{%
  \section{Automatic Gain Control}
    The \setting{Automatic Gain Control} has five different presets for
    automatically controlling the gain while recording.
    \begin{description}
      \item[Safety (clip).]
        This preset will lower the gain when the levels get too high (-1~dB)
        and will never increase gain.

      \item[Live (slow).]
        This preset is designed to be used for recording of live shows and has
        quite large headroom for loud parts. It heads for a nominal target peak
        level of -9~dB and will slowly increase or decrease gain to reach it.

      \item[DJ-Set (slow).]
        This preset heads for a nominal target peak level of -5~dB and will
        slowly increase or decrease gain to reach it.

      \item[Medium.]
        This preset heads for a nominal target peak level of -6~dB and will
        increase or decrease gain to reach it.

      \item[Voice (fast).]
        This preset is designed to be used for voice recording and heads for a
        nominal target peak level of -7~dB and will quickly increase or
        decrease gain to reach it.
    \end{description}

  \section{AGC clip time}
    This setting controls how long the level is too loud or soft before the
    \setting{Automatic Gain Control} kicks in.
}%


\chapter{Value-Number driven code motion} \label{ch:vdcm}
We implement an iterative bit vector data flow algorithm for PRE. We initially
implemented the flow equations from the Lazy Code Motion paper. This set
included a total of 13 bit vectors for each basic block - 2 for block local
properties ANTLOC and TRANSP, and 11 for global properties. These equations,
           however, could only be applied to single instruction basic blocks.
           We therefore, derive a new set of equations which are motived by a
           later work of the same authors of LCM
           paper\cite{Knoop:1994:OCM:183432.183443}. This set of equations
           apply to maximal basic blocks and entails a total of 19 bit vectors
           for each basic block in our current implementation - 3 for block
           local properties ANTLOC, TRANSP, XCOMP and 16 for global properties.
           We include the equations in appendix A. In appendix B, we
           provide our generalized data flow framework, and show how each PRE
           equation maps to the framework. We call the algorithm value-number driven
           because each slot in each of the bit vectors is reserved for a
           particular value number rather than a particular expression. Also,
           we make the observation that a large number of expressions in the
           program only occur once, and are not useful for PRE. Hence to
           further optimize for space and time, we only give bit vector slots to
           value numbers which have more than one expression linked to them.

% This section on computation of localized sets can be moved to the appendix / not included in 1st report
% Also, check if the definition of XCOMP in terms of TRANSP and EVAL holds.
\section{Computation of localized sets}
For each basic block there are 3 bit vectors dedicated to the block-specific
properties, namely \texttt{Transp}, \texttt{Antloc} and \texttt{Xcomp}. As mentioned before, a bit vector
is a boolean array of value numbers. Each value number is associated with
at-least two expressions from the IR. Let the leader
expression (as defined in the section on value numbering) associated with the
value number $v$ be called $L(v)$.

\begin{equation}
\begin{array}{l c l}
\texttt{Transp(v,B)} &=& \left\{
                    \begin{array}{l l}
                        false & \quad \text{iff $\exists$ x $\in$ operands of L(v) such that \texttt{Mod(x,B)} = true}\\
                        true & \quad \text{Otherwise}
                    \end{array} \right. \\
\texttt{Antloc(v,B)} &=& \texttt{Eval(v,B)} \cap \texttt{Transp(v,B)} \\
\texttt{Xcomp(v,B)} &=& \texttt{Eval(v,B)} \cap \overline{\texttt{Transp(v,B)}}\\
  &&\\
\text{where}&& \\
\quad \texttt{Eval(v,B)} &=& \text{\{v $|$ value number v computed in B\}}\\
\quad \texttt{Mod(op,B)} &=& \text{operand op modified in B}\\
\end{array}
\end{equation}

% For the final report we could include an example to show why the definitions of TRANSP, ANTLOC and XCOMP work for value-numbers.
We do not present a formal verification of the definitions of the block local
properties, but in Figure\ref{fig:2}, we motivate these using an example CFG.
\begin{figure}[htbp]
  \begin{center}
    \scalebox{.6}{\begin{tabular}{@{}p{14cm}@{} @{}p{14cm}@{}}
     \includegraphics[scale=0.6]{3} & \includegraphics[scale=0.6]{4} \\
      &\\
  \text{The motivating example} &
  \text{Lazy code motion transformation as suggested by our implementation } \\
    \end{tabular}}
  \end{center}
  \caption{\label{fig:2} }
\end{figure}

\section{Local CSE}
For our data flow equations to work correctly, a local CSE pass has to be run
on each basic block. Basically, this path removes the redundancies inside
straight line basic block code and sanitizes it for the iterative bit vector
algorithm. This idea is borrowed from \cite{Knoop:1994:OCM:183432.183443}. We perform this step before calling
our data flow framework.

\section{Status}
We summarize accomplished work in this section. We completed the value numbering pass and included some
optimizations as part of it. We then derived PRE equations for maximal basic block CFG and molded them
to work with value numbers rather than lexical names. This was followed by construction of a generalized
data flow framework which could incorporate PRE equations. More specifically, we define a single function
which can be called with equation specific parameters. We also wrote a pass to perform local CSE on each basic block.
We have tested our implementation of small code segments with good amount of success. We are able to identify
the placements in the CFG which are computationally optimal as well as lifetime optimal. As an example, we present
a fairly involved CFG we wrote using goto statements

\section{Ongoing Work and Future Tasks}
Having identified the INSERT points (where computation should be added) and REPLACE points (from where computation should be removed),
we are now working on doing the actual IR modifications. Once completed we would begin with testing on fairly large codes for 1)
correctness and 2) performance improvements. This would be followed by a comparison with the LLVM GVN-PRE pass. Time permitting,
we would like to provide support for eliminating redundancies where two expressions are lexically different and have different value numbers
(mentioned in section on PRE)


\documentclass[a4paper,12pt]{article}

% Font
\usepackage[T1]{fontenc}
\usepackage{gentium}

% Math packages
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsthm}
\usepackage{bm}

% Define symbol shortcuts
\newcommand{\cc}{\mathcal{C}}
\newcommand{\dd}{\mathcal{D}}
\newcommand{\hh}{\mathcal{H}}
\newcommand{\xx}{{\bm x}}
\newcommand{\yy}{{\bm y}}

% Math environment
\newtheorem*{thm}{Theorem}

% Better list management:
% - vertical spacing in lists
% - items in lists start with dash not bullet point.
\usepackage{enumitem}
\setlist{label=\textemdash,
  itemsep=0pt, topsep=3pt, partopsep=0pt}

% Include graphics
\usepackage{graphicx}
\usepackage{subcaption}

% Page format
\usepackage[top=2cm,left=2cm,right=2cm,bottom=2cm]{geometry}

\begin{document}
%%% HEADER
\raisebox{0.6in}[0in]{\makebox[\textwidth][r]{\it Unproofed version }}
\vspace{-0.7in}

\begin{center}
\bf\large MA2823: Foundations of Machine Learning \\
Chapter 6: Regularized Linear Regression
\end{center}

\noindent
Lecturer: Chlo\'e-Agathe Azencott
\hfill
Scribe: Adrien Galamez \\
\null \hfill Paul Magon de la Villehuchet

\noindent
\rule{\textwidth}{1pt}

\medskip

In this chapter we will see:
\begin{itemize}
\item what is regularization;
\item how to use regularization as a means to control model complexity;
\item several forms of regularizing linear regression models.
\end{itemize}

\section{Regression setting}
\subsection{Large $p$, small $n$}
This section is a reminder of the context in which we work. We consider a dataset that has $p$ features and $n$ samples. Thus, we're working with a data matrix $X$ that have $n$ rows and $p$ columns. The outcome vector $y$ describes the quantity we want to predict. The goal of linear regression model is to approximate $y$ as a linear combination of $X$.
\begin{figure}[!h]
\centerline{
\includegraphics[scale = 0.3]{figures/RegressionSettings.png}}
\caption{Figure illustrating the dimensions of the data matrix and the outcome vector.}
\label{DataMatrix}
\end{figure}

On the Figure \ref{DataMatrix}, we can see that the data matrix is taller than it is wide ($p < n$). This is a  common shape. But, conversed settings can also appear.

This configuration is called large $p$, small $n$. The data matrix is wider than it is tall. This is the kind of settings we have in genetics and neuroimaging: we have many more descriptors than we have data points. For instance, in genetics, a data set can gather informations for thousands of genes as this information is easy to access. Yet, as each row holds data gathered on a patient, we have at best hundreds of patients. Thus, we have a wide but not tall data matrix. In neuroimaging, brain images are big objets, typically thousands pixels or voxels (3D images). Yet, as brainscans are costly to obtain, we repeat the process on thousands of patient only. Thus, we have more features than observations. In this configuration, regularization is usefull.

\subsection{Linear regression}
\subsubsection{Pro and cons of least-squares fit}
Linear regression is aimed at approximating $y$ as a linear combination of $X$. In the previous chapter, we said that, if we're doing a least-squares fit (which is equivalent to Maximum Likelihood estimation under the assumption of Gaussian noise), we obtain this solution
\begin{equation}
\hat{\beta} = \arg\min_{\beta} (y-X\beta)^{T} (y-X\beta) = (X^T X)^{-1} X^T y.
\label{LR}
\end{equation}
This solution is uniquely defined when $X^T X$ is invertible, hence when $X$ has a full column rank. As $X$ is an $n \times p$ matrix, if $p$ is larger than $n$ (as in the large $p$, small $n$ configuration), $X$ can not be inverted.

This fit has several advantages.
\begin{itemize}
\item The predicted vector $\hat{\beta}$ is unbiased ($\mathbb{E}[\hat{\beta}]= \beta$).
\item If we're restricting to unbiased estimators, minimum mean squared error implies minimum variance.
\item This fit gives an explicit solution (Equation \ref{LR}).
\item If $n \gg p$, the computational time is linear in the number of samples. Indeed, the computational time is $O(\underbrace{np^2}_{\text{compute } X^T X}+\underbrace{p^3}_{\text{invert } X^T X})$.
\end{itemize}

Yet, this fit has also some drawbacks.
\begin{itemize}
\item Correlated variable leads to high variance of the estimator.
\item Prediction error increases linearly as a function of $p$.
\item The solution is hard to interpret when $p$ is large as shown in the section \ref{LargeP}.
\end{itemize}

\subsubsection{When $p$ in larger than $n$}
\label{LargeP}
When $X^T X$ is not invertible, we still have ways to find $\hat{\beta}$. In this case, we can use the pseudo-inverse of $X$. We can also use numerical methods to solve a linear system of $p$ equations such as gradient descent (aimed at minimizing a convexe function), Gaussian elimination or LU decomposition.
On the Figure \ref{LRPseudoInverse}, we can compare the predicted coefficient (thanks to a linear regression using pseudo-inverse) and the true coefficient when $p = 1000$ and $n = 10$. So, we're in the case of the large $p$, small $n$ configuration. In fact, the outcome vector $y$ was created by a linear combination of the data-matrix $X$ using the weights shown on the left hand graph.

The objectif was to retrieve those weights using a linear regression on the vectors $X$ and $y$. $10$ causal features were highlighted in orange on both graphs. A good approximation would be to use only those $10$ causal features. Nevertheless, as the right hand graph shows, the linear regression gives coefficients that have approximately the same weight. The causal features are lost entirely in the noise created by the others coefficients. Information has been lost. Regularization is a way of addressing this issue.

\begin{figure}[!h]
\centerline{
\includegraphics[scale = 0.3]{Figures/linear_regression_pseudoinverse.png}}
\caption{Example of weights obtained from linear regression when $X$ is not invertible, thus using the pseudo-inverse. On the left, the graph is showing the true coefficients of the vector $\beta$. On the right, the graph presents the predicted coefficients of the vector $\hat{\beta}$ obtained from linear regression.}
\label{LRPseudoInverse}
\end{figure}

\section{Regularization}
The Figure \ref{LRPseudoInverse} showed that the linear regression used all the features available to predict the outcome vector $y$. Yet, all the weights are very small. Thus, the solution is hard to interpret. Usually, we prefer having a small subset of features with strong weights. This is one of the advantages of regularization. Moreover, the more we have variables in our model, the more complex our model is. And, the more complex our model is, the more chances we have to overfit our data. Thus, we want to find a way to simplify our model.

 So, instead of minimizing only the sum of squared error, the idea of regularization is to minimize
 \[ \text{Sum of squared error} + \lambda \text{ prenalty on model complexity}.\]
Thus, when using regularization, the estimator is biased (if $\lambda \not = 0$). Yet, because the model will be less complex, we'll have a smaller variance. We're willing to accept a biased estimator in exchange for smaller variance. So, there's a tradeoff to make between bias and variance. $\lambda$ can be set by cross-validation. \label{CVlambda}

This method is also called \emph{shrinkage} in the context of linear regression. Indeed, this is going to shrink the weights of the model. Thus, the final model is simpler.

The following sections are presenting examples of regularization technique.

\subsection{Ridge regression}
\subsubsection{Ridge estimator}
Instead of minimizing our sum of squared error, ridge regression is aimed at minimizing $||y - X\beta||_2^2 + \lambda ||\beta||_2^2$. Thus, the ridge estimator is given by
\[ \hat{\beta}_\text{ridge} = \arg \min_\beta ||y - X\beta||_2^2 + \lambda ||\beta||_2^2. \]

\begin{thm} The ridge regression estimator is given by
\[ \hat{\beta}_\text{ridge} = (X^T X+ \lambda I)^{-1} X^T y. \]
\end{thm}

\begin{proof}
We consider the function $f$ defined by
\[ f(\beta) = ||y - X\beta||_2^2 + \lambda ||\beta||_2^2.\]
In order to minimize this function, we take its gradient.
\[ \nabla_{\beta}f(\beta) = -2X^T (y-X\beta) + 2\lambda \beta.\]
Then, $\beta_\text{ridge}$ is defined as $\nabla_{\beta}f(\beta_{\text{ridge}}) = 0$. Thus,
\[ (X^TX+\lambda I)\beta_\text{ridge} = X^T y.\]
If $\lambda > 0$, $(X^T X + \lambda I)$ is invertible, then
\[ \hat{\beta}_\text{ridge} = (X^T X+ \lambda I)^{-1} X^T y. \]
\end{proof}

\subsubsection{Solution path}
\label{SectionSolutionPathRR}
We've said in section \ref{CVlambda} that we can find the optimal value of $\lambda$ by cross-validation. We can compute solution paths (Figure \ref{SolutionPathRR}) that is a plot showing how the feature coefficient evolves when we decrease the value of $\lambda$. Indeed, when $\lambda = 0$, there's no regularization anymore. And, if $\lambda \gg 1$, we only want to minimize the model complexity. So, all coefficient are then equal to $0$. Then, on the Figure \ref{SolutionPathRR}, on the left hand, we start with the model in which all coefficients are equal to zero and we end, on the right hand, with the linear regression without any regularization model.

The vertical red line shows the value of $\lambda$ that was obtained by cross-validation. The goal is to choose the value of $\lambda$ that gives the best generalization on another set of data.
\begin{figure}[!h]
\centerline{
\includegraphics[width = 0.7\textwidth]{figures/solution_path_ridge_regression.png}}
\caption{Ridge regression solution path}
\label{SolutionPathRR}
\end{figure}

\subsubsection{Standardization}
The goal of this section is to describe what will happen to our model if we multiply our features by a constant.
\begin{enumerate}
\item \textbf{Standard linear regression}~: Without any regularization, we multiply, in the data matrix $X$, the column $j$ by a real $c \not = 0$. So,
\[\forall i \in [|0;n-1|], x_{ij} \to c x_{ij}.\]
Then, the corresponding weight $\beta_j$ is going to be divided by $c$.
\[ \beta_j \to \frac{1}{c} \beta_j.\]
Thus, the weight is different but the solution is similar.
\item \textbf{Ridge regression}~: When we multiply, in the data matrix $X$, the column $j$ by a real $c \not = 0$, we can't know what will happen because of the penalization term $\lambda \beta_j^2$. So, it's important to use standardized feature \emph{before} regularizing linear regression.
\end{enumerate}
\subsubsection{Advantages and drawbacks}
Finally,
\begin{enumerate}
\item \textbf{Advantages}~: Ridge regression has several advantages
\begin{itemize}
\item Correlated variables get similar weights.
\item Identical variables get identical weights.
\item An analytical solution is provided
\end{itemize}
\item \textbf{Drawbacks}~: And it also have drawbacks
\begin{itemize}
\item Ridge regression shrinks coefficients but does \emph{not} result in a sparse model. A model is sparse when many coefficient get a weight of $0$. Then, when a model is sparse, many coefficient can be eliminated from the model.
\end{itemize}
\end{enumerate}

\subsection{Lasso regression}
\subsubsection{Lasso estimator}
Instead of using the $L^2$ penalty, like the Ridge estimator, the Lasso estimator is based on a $L^1$ penalty. Hence we have to minimize : $||y - X\beta||_2^2 + \lambda ||\beta||_1$.
\begin{equation}
\hat{\beta}_\text{lasso} = \arg \min_\beta ||y - X\beta||_2^2 + \lambda ||\beta||_1.
\label{DefLagrange}
\end{equation}

Unlike the Ridge estimator, there is no explicit solution. We can however, transform this problem into a quadratic problem.

\begin{thm} There is a bijection between $\lambda$ and $t$ such as ${\beta}_\text{lasso}$ is solution of the quadratic problem :
\begin{equation}
\hat{\beta}_\text{lasso} = \arg \min_\beta ||y - X\beta||_2^2 \quad s.t \quad ||\beta||_1\leq t.
\label{DefQuadratic}
\end{equation}
\end{thm}

\begin{proof}
We consider the functions $f$ and $g$ defined by
\[ f(\beta) = ||y - X\beta||_2^2.\]
\[ g(\beta) = ||\beta||_1 - t.\]
From (\ref{DefQuadratic}) to (\ref{DefLagrange}). Let $H$ be the feasible region, i.e, $\{ \beta$ $, g(\beta) \leq 0\}$
\[ \hat{\beta}_\text{0} = \arg \min_{\beta \in H} ||y - X\beta||_2^2 \quad.\]
Two cases are possible:\\
\begin{itemize}
\item Case 1 : the unconstrained minimum lies in the feasible region. Problem is solved.
\item Case 2 : the unconstrained minimum does not lie in the feasible region. \\
In this case, the solution is the intersection of the iso-contours of $f$ with the boundary of $H$, as shown in the next figure.
\end{itemize}
~\\
\begin{figure}[!h]
\centerline{
\includegraphics[width = 0.7\textwidth]{figures/sol_lasso.png}}
\caption{Solution of Lasso-estimator}
\label{SolutionLasso}
\end{figure}
~\\
Furthermore, we know that the gradient of $g(\beta)$ is orthogonal to the boundary of $H$.
The gradient of $f(\beta)$ is also orthogonal to the iso-contours of $f$.\\
Hence, at $\beta_{0}$, the gradient of $f$ and $g$ are parallel.\\
We also know that $\nabla_{\beta}g(\beta)$ points towards the unconstrained minimum of $f$ while $\nabla_{\beta}f(\beta)$ points in the opposite direction.\\
~\\
\begin{figure}[!h]
\centerline{
\includegraphics[width = 0.7\textwidth]{figures/sol_lasso2.png}}
\caption{Gradients of $f$ and $g$ and solution of Lasso-estimator}
\label{SolutionLasso2}
\end{figure}
~\\
We can conclude from previous results that $\nabla_{\beta}f(\beta) = -\lambda \nabla_{\beta}g(\beta)$.\\
The Lagrangian of the Lasso-problem : $L(\beta) = f(\beta) + \lambda g(\beta)$ is minimized at $\beta_{0}$. \\

From (\ref{DefLagrange}) to (\ref{DefQuadratic}). Let us fix $\lambda$ and define $\beta_*  = \arg \min_\beta f(\beta) + \lambda ||\beta||_1^2$. Let us define $t = ||\beta_*||_1^2.$\\
Then, for any $\beta, f(\beta_*)+ \lambda ||\beta_*||_1^2 \leq f(\beta)+\lambda||\beta||_1^2$. Hence $f(\beta_*)\leq f(\beta)+\lambda(||\beta||_1^2 - t).$\\
Under the constraint that $||\beta||_1^2 \leq t, \lambda(||\beta||_1^2 - t) \leq 0$. Hence $f(\beta_*) \leq f(\beta)$, i.e. $\beta_* = \beta_{lasso}$\\

Thus, the two problems are equivalent.
\end{proof}
\subsubsection{Solution path}
As we did before in Section \ref{SectionSolutionPathRR} for the Ridge Regression estimator, we can also compute solution paths for the Lasso estimator (Figure \ref{SolutionPathLasso}). On the left hand of the graph, we start with high values of $\lambda$ that give very simple models. We move towards lower values of $\lambda$ that gives more complex models. Yet, there's one difference with the ridge regression solution path. In fact, we see here that our features ``enter the model one by one'': their weight is equal to $0$ for all values of $\lambda$ lower to a certain $\lambda_0$. For instance, here, when the model complexity increases, features ``enter'' the model in this order: \texttt{lcavol}, \texttt{lweight}, \texttt{svi}, \ldots, \texttt{gleason}. Thus, at every stage, we have a \emph{sparse model}. For example, on the vertical red line, we have a model with only $5$ parameters that are not null (\texttt{lcavol}, \texttt{svi}, \texttt{lweight}, \texttt{pgg45}, \texttt{lbph}).

\subsubsection{Forward stepwise regression}
We just saw that, in the lasso regularization, the features are ``entering'' the model one by one. Then, we can also get the lasso solution by using forward stepwise regression: the model is built sequentially by adding one variable at a time. Then, this a greedy method.
\begin{itemize}
\item We start with the intercept only.
\item At each step, the variable that most improves the fit is added. All variables are tested and the one that gives the better fit is kept.
\item We stop when $||\beta ||_1 \leq t$.
\end{itemize}

\subsubsection{Least Angle Regression}
The computation of the lasso solution is a quadratic problem, and can be tackled by standard algorithms. But, the least angle regression algorithm is a better approach. In the previous section, when a variable and its weight was added in the model, and this weight was never updated. The least angle regression procedure follow the same general scheme, but doesn't add a feature fully into the model. Here, at each step, ``only as much of a variable as needed'' is added. In fact, the coefficient of that feature is increased only until that feature is no longer the one most correlated with the residual $r$. This algorithms can be described as follow
\begin{enumerate}
\item Standardize the features to have mean zero and unit norm. Start with the residual $r = y - \bar{y}$ and $\beta_1, \ldots, \beta_p = 0$. The residual is ``what is left to explain from our model''. The perfect model would have a residual that is equal to $0$.
\item Find the one feature that explains $r$ the most. In other words, find the feature $x_j$ most correlated with $r$.
\item Update the weight $\beta_j$ until $x_j$ is no longer the one most correlated with $r$: move $\beta_j$ from $0$ towards in the direction of the sign of its correlation with $y$. In fact, move $\beta_j$ from $0$ towards its least-squares coefficient $\langle x_j, r \rangle$ until some other competitor $x_k$ has as much correlation with the current residual as does $x_j$. So
\[ \beta_j + \alpha \frac{1}{\sum_{i=1}^n (x_j^i)^2} \sum_{i=1}^n x_j^i r^i \to \beta_j,\]
\[ (y - \bar{y}) - \beta_j x_j \to r,\]
where $\alpha$ is called the step size.
\item Move $\beta_j$ and $\beta_k$ in the direction defined by their joint least squares coefficient of the current residual on $\langle x_j, x_k \rangle$ until some other competitor $x_l$ has a much correlation with the current residual $r = (y-\bar{y} - \beta_j x_j - \beta_k x_k)$.
\item If a non-zero coefficient hits zero, drop its variable from the active set of variables and recompute the current joint least squares direction.
\item Continue in this way until all $p$ predictors have been entered.
\end{enumerate}

\begin{figure}[!h]
\centerline{
\includegraphics[width = 0.7\textwidth]{figures/solution_path_lasso.png}}
\caption{Ridge regression solution path}
\label{SolutionPathLasso}
\end{figure}

\subsection{Elastic Net}
\subsubsection{Elastic Net estimator}
The Elastic Net estimator is a convex combination of Lasso and Ridge. It is defined by :
\[ \hat{\beta}_\text{enet} = \arg \min_\beta ||y - X\beta||_2^2 + \lambda(\alpha||\beta||_2^2 + (1 - \alpha)||\beta||_1) \]
The main advantage is to combine both :
\begin{itemize}
\item It selects variables like the Lasso-estimator.
\item It shrinks together correlated and identical variables like the Ridge-estimator
\end{itemize}
Of course, this cames with the cost of having two parameters ($\lambda$, $\alpha$) to set instead of one.

\subsubsection{Comparison of Elastic Net-estimator and Lasso-estimator}
The Figure \ref{Lassoenet} represents results of both Lasso-estimator and Elastic Net-estimator on Leukemia data. We can see that Elastic Net results in more non-zero coefficients than Lasso, but with smaller amplitudes.

\begin{figure}[!h]
\centerline{
\includegraphics[width = 0.9\textwidth]{figures/lassoenet.png}}
\caption{Comparison of weights on Lasso-estimator and Elastic Net estimator}
\label{Lassoenet}
\end{figure}

\subsection{$L^q$ regression}
\subsubsection{$L^q$ estimator}
The $L^q$ estimator is a generalization of the Ridge and Lasso estimator : the penalty is based on the $L^q$ norm of $\beta$.
\[||\beta||_q^q = \sum_{i = 1}^{n}{\beta_{i}^q}\]
The $L^q$ estimator is then defined by :
\[ \hat{\beta}_\text{$L^q$} = \arg \min_\beta ||y - X\beta||_2^2 + \lambda ||\beta||_q^q. \]
We have the same theorem to transform this problem into a quadratic problem with constraints.\\

\begin{thm} There is a bijection between $\lambda$ and $t$ such that ${\beta}_\text{$L^q$}$ is solution of the quadratic problem :
\[ \hat{\beta}_\text{$L^q$} = \arg \min_\beta ||y - X\beta||_2^2 \quad s.t \quad ||\beta||_q^q\leq t. \]
\end{thm}

\begin{proof}
The proof follow the same steps than the previous one.
\end{proof}

The Figure \ref{Bound2} represents the feasible sets for different values of $q$.
\begin{figure}[!h]
\centerline{
\includegraphics[width = 0.7\textwidth]{figures/bound.png}}
\caption{Boundaries of the feasible set with different values of q}
\label{Bound2}
\end{figure}

\subsubsection{Comparison between Lasso and Ridge}
Because the feasible depends on the value of $q$, the Ridge $(q = 2)$ and Lasso $(q = 1)$ estimator will be different. The feasible set will be a square in one case and a circle in the other. This helps understand why the $L^1$ norm gives sparse models. Indeed, the $L^1$ ball has corners. This means that when we're moving from the unconstrained minimum towards the ball, we're more likely to ``hit'' a corner of the ball. And, on a corner of the ball, only one feature has a weight non equal to zero.

\begin{figure}[!h]
\centerline{
\includegraphics[width = 0.7\textwidth]{figures/comparison.png}}
\caption{Comparison of the Lasso-estimator and the Ridge-estimator}
\label{Comparison}
\end{figure}

\subsection{Structured Regularization}
\subsubsection{Group Lasso}
The \emph{group Lasso} is a variant of Lasso. All $p$ variables are partitioned in $K$ predefined groups of variables that are known to work together and expected to work together and thus be all active or inactive together. For instance, genes belonging to the same biological pathway can form a group.

Let $X_k$ be the features belonging to group $k$. Let $p_k$ be the size of group $k$. We have : $\sum_{k = 1}^K{p_k} = p$. We can define the Group Lasso-estimator:
\[ \hat{\beta}_\text{grlasso} = \arg \min_\beta ||y - \sum_{k = 1}^K{X_k\beta_k}||_2^2 + \lambda \sum_{k = 1}^K{\sqrt{p_k}||\beta_k||_2}\]

\subsubsection{Other structured penalties}
Other regularizations exist :
\begin{itemize}
\item Overlapping groups
\item Graphs
\item Trees
\item Multiple related Tasks
\end{itemize}

\section{Conclusion}
We've replaced the minimization of the sum of squared errors (that gave us the solution of the linear regression problem) by minimizing the sum of squared errors \emph{plus} a regularizer that penalizes the model complexity. We've seen 3 major forms of regularizers:
\begin{itemize}
\item Ridge regression ($L^2$ norm)
	\begin{itemize}
	\item gives similar weights to similar variables,
	\item doesn't really give sparse solution,
	\item but has the advantage of having an explicit solution.
	\end{itemize}
\item Lasso regression ($L^1$ norm)
	\begin{itemize}
	\item randomly picks one of several correlated variables,
	\item gives sparse solution,
	\item can be implemented by the Least Angle Regression algorithm.
	\end{itemize}
\item Elastic net (combining $L^1$ and $L^2$ norm)
	\begin{itemize}
	\item selects variables like the lasso,
	\item shrinks together the coefficients of correlated variables.
	\end{itemize}
\end{itemize}

\end{document}


\section{The Models}
In the nature, the process of fungi decomposing ground litter and woody fibers involves complex chemical reactions~\cite{chemicalreactions}. In order to visualize this objective process mathematically, we simplified the analysis of dead wood and leaves to the study of cylindrical woody fibers and plane irregular ground litter.
% 第一节
\input{4/1.tex}
% 第二节
\input{4/2.tex}
% 第三节
\input{4/3.tex}
% 第四节
\input{4/4.tex}
% 第五节
\input{4/5.tex}
% 第六节
\input{4/6.tex}


% ---------------------------------------------------------------
% Document configuration setup.
%
% Sets up things like document format, page size and the libraries to use later
% ---------------------------------------------------------------

\documentclass[preview]{standalone} % we only want the diagram: this crops the pdf to the size of the content

%\documentclass[11pt, oneside]{article}   	% use "amsart" instead of "article" for AMSLaTeX format
\usepackage{geometry}                		% See geometry.pdf to learn the layout options. There are lots.
\geometry{letterpaper}                   		% ... or a4paper or a5paper or ...
%\geometry{landscape}                		% Activate for for rotated page geometry
%\usepackage[parfill]{parskip}    		% Activate to begin paragraphs with an empty line rather than an indent
\usepackage{graphicx}				% Use pdf, png, jpg, or eps§ with pdflatex; use eps in DVI mode
								% TeX will automatically convert eps --> pdf in pdflatex
\usepackage{amssymb}
\usepackage{tikz}
\usepackage{amsmath}
\usetikzlibrary{arrows}
%\title{Brief Article}
%\author{The Author}
%\date{}							% Activate to display a given date or no date

% ---------------------------------------------------------------
% Start of the document itself
% ---------------------------------------------------------------
\begin{document}
%\maketitle
\thispagestyle{empty} % don't print page numbers - we only want the diagram
% ---------------------------------------------------------------
% tikz is the LaTeX library for drawing 2D figures.  Use the
% arrows library for drawing the number lines
% ---------------------------------------------------------------
\usetikzlibrary{arrows}
%\section{Hundredths}
%\subsection{}
% Frac1 b6 diagram hundredths {{ value }} %\the\numexpr\y+1
\begin{tikzpicture} [>=triangle 45, scale = 0.1]%use to change style of arrow head and scale of pic
\foreach \x in  {0,..., 107}%this tells you at which points to draw the lines and labels
\draw[shift={(\x,0)},color=black] (0pt,20pt) -- (0pt,-20pt);%this tells you the vert lines color and size
\foreach \x in  {0,..., 99}
\draw[shift={(\x,1.8)}] {rectangle(1,5.8)};
\foreach \z in  {0,10,20,30,40,50,60,70,80,90,100}%this tells you at which points to draw the lines and labels
\draw[shift={(\z,0)},color=black] (0pt,40pt) -- (0pt,-40pt);%this tells you the vert lines color and size
\draw[very thick, ->] (0,0) -- (110,0) ;
%\foreach \z in {0, 100}
%\draw[shift={(\z,0.3)}, color=black] {rectangle(1,0.7)};
\foreach \x in {0,100}
\draw[shift={(\x,0)},color=black] (0pt,0pt) -- (0pt,-12pt) node[below]{\LARGE\the\numexpr\x/100\relax}; %this tells you the labels under the vert lines
\foreach \x in {0,...,{{ value - 1 }} }
\draw[shift={(\x,1.8)}, fill=red] {rectangle(1,5.8)};
%
\end{tikzpicture}

\end{document}