\documentclass[11pt]{article}

\usepackage{amsfonts}
\usepackage{fancyvrb}
\usepackage{url}

\setlength{\oddsidemargin}{0in}
\setlength{\evensidemargin}{0in}
\setlength{\textwidth}{6.5in}
\setlength{\topmargin}{0in}
\setlength{\headsep}{0.5in}
\setlength{\textheight}{8.5in}
\setcounter{page}{1}
%\pagestyle{empty}
%\hbadness=10000

\begin{document}
\huge
\noindent
{Discrete Optimization Assignment:}
\vspace{0.25cm}

\noindent
{\bf Knapsack}
\normalsize

\section{Problem Statement}

In this assignment you will design an algorithm to solve the infamous {\em Knapsack Problem}, which plagues Indiana Jones.  You are provided with a knapsack with limited space and a collection of items with different values and weights.  Your task is to maximize the value of items packed into your knapsack without exceeding its total capacity.

\section{Assignment}

Write an algorithm to solve the knapsack problem.  The problem is mathematically formulated in the following way.  Given $n$ items to choose from, each item $i \in 0 \ldots n-1$ has a value $v_i$ and a weight $w_i$.  The knapsack has a limited capacity $K$.  Let $x_i$ be a variable that is $1$ if you choose to take item $i$ and $0$ if you leave item $i$ behind. Then the knapsack problem is formalized as the following optimization problem,
$$
\begin{array}{ll}
\mbox{maximize:} & \displaystyle \sum_{i \in 0 \ldots n-1} v_i x_i \\
\mbox{subject to:} & \\
     & \displaystyle \sum_{i \in 0 \ldots n-1} w_i x_i \leq K \\
     & x_i \in \{0,1\} \;\;\; (i \in 0 \ldots n-1)
\end{array}
$$

\section{Data Format Specification}

A knapsack input contains $n+1$ lines.  The first line contains two integers, the first is the number of items in the problem, $n$.  The second number is the capacity of the knapsack, $K$.  The remaining lines present the data for each of the items.  Each line, $i \in 0 \ldots n-1$ contains two integers, the item's value $v_i$ followed by its weight $w_i$.

\vspace{0.2cm}
\noindent
Input Format
\vspace{-0.2cm}
\begin{Verbatim}[frame=single]
n K
v_0 w_0
v_1 w_1
...
v_n-1 w_n-1
\end{Verbatim}
%
The output contains a knapsack solution and is made of two lines.  The first line contains two values $obj$ and $opt$.  $obj$ is the total value of the items selected to go into the knapsack (i.e. the objective value).  $opt$ should be $1$ if your algorithm proved optimality and $0$ otherwise.  The next line is a list of $n$ $0/1$-values, one for each of the $x_i$ variables.  This line encodes the solution.
\clearpage
%
\vspace{0.2cm}
\noindent
Output Format
\vspace{-0.2cm}
\begin{Verbatim}[frame=single]
obj opt
x_0 x_1 x_2 ... x_n-1
\end{Verbatim}
%
It is essential that the value order in the solution output matches the value order of the input.  Otherwise the grader will misinterpret the output.

\paragraph{Examples} \mbox{}
%\vspace{0.1cm}

\vspace{0.2cm}
\noindent
Input Example
\vspace{-0.2cm}
\begin{Verbatim}[frame=single]
4 11
8 4
10 5
15 8
4 3
\end{Verbatim}

\vspace{0.2cm}
\noindent
Output Example
\vspace{-0.2cm}
\begin{Verbatim}[frame=single]
19 0
0 0 1 1
\end{Verbatim}

\section{Instructions}

%Edit \texttt{solver.py} and modify the \texttt{solveIt(inputData)} function to solve the knapsack problem.  The function argument, \texttt{inputData}, contains a knapsack problem in the format described above.  The return value of \texttt{solveIt} is a solution to the problem in the output format described above.  Your \texttt{solveIt} implementation can be tested with the command,
%\[
%\texttt{python ./solver.py ./data/<inputFileName>}
%\]
%You should limit the \texttt{solveIt} method to terminate within 5 hours, otherwise the submission will not be eligible for full credit.  You may choose to implement your solver directly in python or modify the \texttt{solveIt()} function to call an external application.

\input{instructions.tex}

\paragraph{Resources}
You will find several knapsack instances in the \texttt{data} directory provided with the handout.  \texttt{Solver.java} and \texttt{solverJava.py} are included in this assignment to demonstrate how to implement your assignment in another language.  If you wish to use the Java solver, you will need to rename \texttt{solverJava.py} to \texttt{solver.py}.
%An example output file, \texttt{blabla.out}, is also provided.

%\section{Remarks}

\input{handin.tex}

\input{grading.tex}

\input{collaboration.tex}

%\paragraph{Questions} Please contact the class GTA Carleton (cjc@cs.brown.edu).

\input{warnings.tex}

%\paragraph{Warnings}
%
%\begin{enumerate}
%\item It is recommended you do not modify the \texttt{data} directory.  Modifying the files in the data directory risks
%making your assignment submissions incorrect.
%\item You cannot rename the \texttt{solver.py} file.
%\item \texttt{solver.py} must remain in the same directory as  \texttt{submit.pyc}.
%\end{enumerate}

\input{techReqs.tex}

\end{document}


\subsection{Functions of the vertices}
\label{sec:vertices}

%------------------------------------------------------------------

\subsubsection{Vertex surround angle}
\label{sec:vertex_surround_angle}

\paragraph{Non-boundary case}
\label{sec:non_boundary_vertex_surround_angle}

\begin{figure}[!htp]
\centering
\begin{verbatim}
                v2/e2 o           o v1/e1
                       \   f1    /
                        \       /
                     f2  \ g1  /   f0
                          \ ^ /
                       g2( \ / )g0
             v3/e3 o--------o--------o v0/e0
                       g3( / \ )gn-1
                          / v \
                     f3  /     \  fn-1
                        /       \
                 v4/e4 o   ...   o vn-1/en-1
\end{verbatim}
\caption{Vertex surround angle.
\label{fig:vertex_surround_angle}}
\end{figure}

Consider the non-boundary vertex
labeled $\v$ in figure \ref{fig:vertex_surround_angle}.
$\v$ has degree $n$;
the incident edges are labeled $\e_0, \e_1, \ldots, \e_{n-1}$;
the incident faces are labeled $\f_0, \f_1, \ldots, \f_{n-1}$;
and the neighboring vertices are labeled $\v_0, \v_1, \ldots, \v_{n-1}$.
The angle between edges $\e_j$ and $\e_{j + 1 \bmod n}$ is $\gamma_j$.

$\gamma_j = \gamma(\f_j,\v)$ is the {\em corner angle} in face $\f_j$
of vertex $\v$.

The {\em surround angle} of $\v$ is the sum of all its corner angles:
\begin{equation}
\alpha(\v) = \sum_{j=0}^{n-1} \gamma(\f_j,\v),
\end{equation}

If the neighborhood of $\v$ is flat, then $\alpha(\v)=2\pi$.
A neighborhood with $\alpha(\v) < 2\pi$, corresponds very roughly
to positive Gaussian curvature,
and, $\alpha(\v) > 2\pi$,
is analogous to negative Gaussian curvature.
However, it's important to remember that, unlike the case
with smooth surfaces, where the most complicated local
shape is a saddle,
the crinkling of a vertex neighborhood can be arbitrarily complicated,
for any non-zero surround angle.

Although a mesh with all surround angles of $2\pi$ is not guaranteed
to be flat,
a mesh with surround angles different from $2\pi$ is guaranteed
to be not-flat,
and, in some sense,
the more different the surround angles are from $2\pi$
the more non-flat is the mesh.
This leads to the following as a mesh roughness measure:
\begin{eqnarray}
f(\M)
& = & \sum_{\v \in \V(\M)} \left[ \alpha(\v) - 2\pi \right]^2
\\
& = & \sum_{\v \in \V(\M)}
\left[ \left(\sum_{\f \in \F(\v)} \gamma(\f,\v)\right)
 - 2\pi \right]^2
\nonumber
\end{eqnarray}
The natural implementation of this function is to first compute
all the corner angles and cache them with the faces,
then compute the surround angles and cache with each vertex,
and then do a lookup while computing the sum.

In computing the gradient, it's simplest to re-arrange the
order of summation:
\begin{eqnarray}
\Ga{f}
& = & \Ga{ \left[ \sum_{\v \in \V(\M)} \left[ \alpha(\v) - 2\pi \right]^2 \right]}
\\
& = & 2 \sum_{\v \in \V(\M)} \left[ \alpha(\v) - 2\pi \right] \Ga{\alpha(\v)}
\nonumber
\\
& = & 2 \sum_{\v \in \V(\M)} \left[ \alpha(\v) - 2\pi \right]
\sum_{\f \in \F(\v)} \Ga{\gamma(\f,\v)}
\nonumber
\\
& = & 2 \sum_{\v \in \V(\M)}
\sum_{\f \in \F(\v)}
\left[ \alpha(\v) - 2\pi \right]
\Ga{\gamma(\f,\v)}
\nonumber
\\
& = & 2
\sum_{\f \in \F(\M)}
\sum_{\v \in \V(\f)}
\left[ \alpha(\v) - 2\pi \right]
\Ga{\gamma(\f,\v)}
\nonumber
\\
& = & 2
\sum_{\f \in \F(\M)}
\sum_{i=0}^2
\left[ \alpha(\v_i(\f)) - 2\pi \right]
\Ga{\gamma(\p_i(\f),\p_{i+1 \% 2}(\f),\p_{i+2 \% 3}(\f))}
\nonumber
\end{eqnarray}

Consider the partial gradient with respect to
location of one of the vertices:
\begin{eqnarray}
\label{eq:surround_angle_partial_gradient}
\Gc{\p_j}{f}{q}
& = 2 \sum_{\f \in \F(\M)} &
\sum_{i=0}^2
\left[ \alpha(\v_i(\f)) - 2\pi \right]
\Gc{\p_j}{\gamma(\p_i(\f),\p_{i+1 \% 2}(\f),\p_{i+2 \% 3}(\f))}{\q}
\\
& = 2 \sum_{\f \in \F(\v_j)} &
\sum_{i=0}^2
\left[ \alpha(\v_i(\f)) - 2\pi \right]
\Gc{\p_j}{\gamma(\p_i(\f),\p_{i+1 \% 2}(\f),\p_{i+2 \% 3}(\f))}{\q}
\nonumber
\\
& = 2 \sum_{\f \in \F(\v_j)} & \left(
\left[ \alpha({\mathrm p}(\v_j,\f) - 2\pi \right]
\Gc{\p_j}{\gamma({\mathrm p}(\p_j,\f),\p_j,{\mathrm s}(\p_j,\f))}{\q}
\right.
\nonumber
\\
& & +
\left[ \alpha(\v_j) - 2\pi \right]
\Gc{\p_j}{\gamma(\p_j,{\mathrm s}(\p_j,\f),{\mathrm p}(\p_j,\f)}{\q}
\nonumber
\\
& & +
\left.
\left[ \alpha({\mathrm s}(\v_j,\f) - 2\pi \right]
\Gc{\p_j}{\gamma({\mathrm s}(\p_j,\f),{\mathrm p}(\p_j,\f),\p_j)}{\q}
\right)
\nonumber
\end{eqnarray}
where ${\mathrm p}(\v,\f)$ is $\v$'s {\em predecessor},
the vertex that comes before $\v$ in the oriented face $\f$,
${\mathrm s}(\v_j,\f)$ the {\em successor}
the vertex that comes after $\v$,
and similarly for the vertex positions $\p$.

Consider the term corresponding to a particular face $\f$ in
the sum in equation \ref{eq:surround_angle_partial_gradient}.
Call that face's vertices and positions $\v_0, \v_1, \v_2$
and $\p_0, \p_1, \p_2$.
\begin{eqnarray}
\Gc{\p_0}{f_{\f}}{q}
& = &
\left[ \alpha(\v_0) - 2\pi \right] \Gc{\p_0}{\gamma(\p_0,\p_1,\p_2)}{\q}
\\
& + &
\left[ \alpha(\v_1) - 2\pi \right] \Gc{\p_0}{\gamma(\p_1,\p_2,\p_0)}{\q}
\nonumber
\\
& + &
\left[ \alpha(\v_2) - 2\pi \right] \Gc{\p_0}{\gamma(\p_2,\p_0,\p_1)}{\q}
\nonumber
\\
&  &
\nonumber
\\
& = &
\left[ \alpha(\v_0) - 2\pi \right]
{{(\q_1 - \q_0) \perp (\q_2 - \q_0) + (\q_2 - \q_0) \perp (\q_1 - \q_0)}
\over
{
\sqrt{
\| \q_2 - \q_0 \|^2 \| \q_1 - \q_0 \|^2
-
\left( ( \q_1 -\q_0 ) \bullet ( \q_2 -\q_0 ) \right)^2
}
}}
\nonumber
\\
& - &
\left[ \alpha(\v_1) - 2\pi \right]
{{(\q_0 - \q_1) \perp (\q_2 - \q_1)}
\over
{
\sqrt{
\| \q_0 - \q_1 \|^2 \| \q_2 - \q_1 \|^2
-
\left( ( \q_0 -\q_1 ) \bullet ( \q_2 - \q_1 ) \right)^2
}
}}
\nonumber
\\
& - &
\left[ \alpha(\v_2) - 2\pi \right]
{{(\q_0 - \q_2) \perp (\q_1 - \q_2)}
\over
{
\sqrt{
\| \q_0 - \q_2 \|^2 \| \q_1 - \q_2 \|^2
-
\left( ( \q_0 -\q_2 ) \bullet ( \q_1 -\q_2 ) \right)^2
}
} }
\nonumber
\end{eqnarray}


\subsection{Units}

A unit is an element of a ring which has a multiplicative inverse.
\subsubsection{Examples}

The ring of integers with addition and multiplication, only \(-1\) and \(1\) are units, as both have multiplicative inverses in the ring.


\section{Methodologies}
\label{sec:methodologies}

%Describe correlation matrix (methodology 1) in mathematics
In \textbf{Methodology$_1$} we compute the autocorrelation matrix, also called series correlation, see \cite{Devore-Berk}, between features to get a first view of the available data and the relationships between them.  To do this we use the autocorrelation matrix that explains the degree of dependence between the values of our features.
Given a sequence $x$, the autocorrelation is given by $$R_x_i=Corr(x,x)_i=\sum_{i=0}^{\infty}x_i x_{i+1}.$$

In this project we did not use this formula but simply applied the .corr() function of Pandas.
\bigskip

%Describe conditional probability calculation (methodology 2) in mathematics
In \textbf{Methodology$_2$}, we calculate the conditional probability of a response variable given an explanatory variable, $\dfrac{P(A\cup B)}{P(B)}$, where deeper explanation can be found at \cite{Walsh-John-B}. This calculation assumes only a relationship between the single explanatory variable and the response and therefore we note that this calculation does \emph{not} model joint probability distributions, because the number of features makes it infeasible to model all combinations of them. While this assumption is not true in practice (see discussion of methodology 1 in Section \ref{sec:Results} for correlation between features), we can nonetheless analyze the likelihood of a given outcome from the perspective of only knowing a single feature (e.g. the chance of death given that one has pneumonia). To give a bound of estimates, we perform this calculation on samples of our dataset, using a sample of a size $N = \dfrac{Length(Dataset)}{4}$. The data is sampled with replacement ("bootstrapped") and the proability is calculated on this sample of the data. We repeat this process 1000 times, and observe the quantiles of the resulting probability estimates to characterize the effect of each explanatory variable on the response. This process is repeated for synthetic, observational, and treatment data, and the details of the calculation $\dfrac{P(A\cup B)}{P(B)}$ is performed using the Pandas library (see the github repository for details of the groupby and division functions).

\bigskip
%Describe logistic regression (methodology 3) in mathematics
In \textbf{Methodology$_3$} we use a logistic regression model to estimate the probability that the input data ($\textbf{x}$) leads to outcomes ($y_i$), for $i=1, 2, ..., m$, where $m$ is the number of possible labels for the outcome classes and $y_i \in [0, 1]$. In the data we use, an outcome of $1$ corresponds to death, and $0$ to survival. The probability of the two outcomes is then as follows: $$\mathbb{P}(y_i=1|x_i, \beta) = \frac{e^{\beta^T x_i}}{1+e^{\beta^T x_i}}$$
$$\mathbb{P}(y_i=0|x_i, \beta) = 1 - \mathbb{P}(y_i = 1|x_i, \beta),$$

\noindent where $\beta$ are the coefficients that will be estimated by the model, see \cite{Geron-Aurelien}.

We also apply a regularization parameter $\lambda \geq 0$, which scales the model parameters. Then, the cost function of the model will be: $$C(\boldsymbol{\beta}) = (\boldsymbol{z} - \boldsymbol{X\beta})^T(\boldsymbol{z} - \boldsymbol{X\beta}) + \lambda \boldsymbol{\beta}^T\boldsymbol{\beta}$$

Where \textbf{X} is the input data and $z$ is the output (true) labels.

We train the logistic regression models on a balanced set of data points sampled from the full data, in order to ensure that the classifier properly learns to predict each class (rather than e.g. predicting only $Death = 0$ because the vast majority of patients survive). After this selection, we define a pipeline for feature selection and hyper-parameter tuning which is applied in randomized-search cross-validation ($CV=500$). This is to ensure that the model parameters are accurate and that we obtain a range of values with which to estimate them.

The logistic regression models will then learn coefficient values from the data corresponding to each feature, where higher coefficient values are predictive of death and negative coefficient values are predictive of survival. Lastly, we select the features with the highest and lowest (i.e. greatest absolute value) coefficients to determine which of them are most predictive of symptoms, for both vaccine side-effects as well as treatment effectiveness.

We implement the model pipeline, logistic regression model itself, and the cross-validation using the \texttt{statsmodels} package. This is convenient because the library provides functionality to display a model summary, including summary statistics for coefficient values including mean, median, standard deviation error, p-values, and confidence intervals (the last of which we detail in Section \ref{sec:Results}). The full printout of these results can be obtained by running the code contained in the GitHub repository.


%
% Chapter 3.4
%

\section*{3.4 Limits at Infinity}

Let \(f\) be a function defined on some interval \((a, \infty)\). Then
$$ \lim_{x \to \infty}f(x) = L $$
means that the values of \(f(x)\) can be made arbitrarily close to \(L\) by requiring \(x\) to be sufficiently large.
\\\\
Let \(f\) be a function defined on some interval \((-\infty, a)\). Then
$$ \lim_{x \to -\infty}f(x) = L $$
means that the values of \(f(x)\) can be made arbitrarily close to \(L\) by requiring \(x\) to be sufficiently large negative.

\subsection*{Horizontal Asymptotes}

The line \(y=L\) is called a \textbf{horizontal asymptote} of the curve \(y=f(x)\) if either
$$\lim_{x \to \infty}f(x)=L \quad \text{or} \quad \lim_{x \to -\infty}f(x)=L$$

\subsubsection*{Theorem}

If \(r>0\) is a rational number, then
$$ \lim_{x \to \infty}\frac{1}{x^r}=0 $$
If \(r>0\) is a rational number such that \(x^r\) is defined for all \(x\), then
$$ \lim_{x \to -\infty}\frac{1}{x^r}=0 $$

\subsection*{Definition of a Limit at Positive Infinity}

Let \(f\) be a function defined on some interval \((a, \infty)\). Then
$$ \lim_{x \to \infty}f(x)=L$$
means that for every \(\epsilon>0\) there is a corresponding number \(N\) such that
$$ \text{if} \quad x>N \quad \text{then} \quad \left| f(x)-L \right| <\epsilon $$

\subsection*{Definition of a Limit at Negative Infinity}

Let \(f\) be a function defined on some interval \((-\infty, a)\). Then
$$ \lim_{x \to -\infty}f(x)=L$$
means that for every \(\epsilon>0\) there is a corresponding number \(N\) such that
$$ \text{if} \quad x<N \quad \text{then} \quad \left| f(x)-L \right| <\epsilon$$

\subsection*{Definition of an Infinite Limit at Infinity}

Let \(f\) be a function defined on some interval \((a, \infty)\). Then
$$ \lim_{x \to \infty}f(x)=\infty$$
means that for every positive number \(M\) there is a corresponding positive number \(N\) such that
$$ \text{if} \quad x>N \quad \text{then} \quad f(x)>M$$


\section{Green function method}
Let's do an example.
Consider the differential equation
\begin{align}
    \frac{d^2y}{dx^2} + y &= f(x)
\end{align}
Boundary conditions $y=0$ at $x=0$
and $y=0$ at $x=\frac{\pi}{2}$.
Introduce the green function $G(x, x')$ which satisfies
\begin{align}
    \frac{d^2 G}{dx^2} + G &=
    \delta\left( x - x' \right)
\end{align}
Why introduce $G$?
In general,
$G$ is easier to find than $y$.
Once $G$ has been determined,
we can obtain $y$ with
\begin{align}
    y(x) &=
    \int_{0}^{\pi/2}
    G\left( x, x' \right)
    f\left( x' \right)\,
    dx'
\end{align}
To see that this is a solution,
substitute it into the original ODE.
\begin{align}
    \frac{dd^2 y}{dx^2} + y
    &=
    \int_{0}^{\pi/2}
    \underbrace{\left\{
    \frac{d^2 G}{dx^2} + G
    \right\}}_{
    \delta\left( x - x' \right)
    }
    f\left( x' \right)
    \,dx'\\
    &=
    f(x).
\end{align}
It also satisfies boundary conditions
\begin{align}
    y(x=0) &=
    \int_{0}^{\pi/2}
    \underbrace{\left.G\left( x, x' \right)\right|_{x=0}}_{=0}
    f\left( x' \right)\,dx' = 0
\end{align}
and
\begin{align}
    y\left( x=\frac{\pi}{2} \right)
    &=
    \int_{0}^{\pi/2}
    \underbrace{\left.G\left( x, x' \right)\right|_{x=\pi/2}}_{=0}
    f\left( x' \right)
    \,dx'
    =0
\end{align}
But how do we find $G$?
\begin{align}
    \frac{d^2G}{dx^2} + G &= \delta\left( x - x' \right)
\end{align}
Away from $x=x'$,
\begin{align}
    \frac{d^2 G}{dx^2 + G} = 0
\end{align}
The solutions are of the form
\begin{align}
    G &=
    A\sin(x) + B\cos(x)
\end{align}
In general the solution for $x<x'$ and $x>x'$ will be different
because they must satisfy different boundary conditions at each end.
\begin{align}
    G &=
    \begin{cases}
        G_{<} = A_{<}\sin(x) + B_{<}\cos(x) & \text{for } x < x'\\
        G_{>} = A_{>}\sin(x) + B_{>}\cos(x) & \text{for } x > x'\\
    \end{cases}
\end{align}
Requiring $G=0$ at $x=0$,
we get $B_{<}=0$.
Requiring $G=0$ at $x=\frac{\pi}{2}$,
we get $A_{>}=0$.
Now we need match the two pieces across $x'$.

In quantum mechanics have you solved for a delta function potential?
Now let's look at this differential equation here.

I claim that this is what we mean.
I want $G$ to be continuous across the boundary,
but $dG/dx$ is not necessarily continuous.
That's the way this works for this example.

Requiring that $G$ be continuous at $x=x'$,
we have
\begin{align}
    G_{<}\left( x = x' \right)
    =
    G_{>}\left( x = x' \right)
\end{align}
we get
\begin{align}
    A_{,}\sin\left( x' \right)
    &=
    B_{>}\cos\left( x' \right)
\end{align}
But we need one more condition,
and that has to come from the derivatives.
So what we do is integrate that differential equation for $G$ across $x=x'$.
\begin{align}
    \lim_{\epsilon\to 0}
    \int_{x'=\epsilon}^{x' + \epsilon}dx\left\{
    \frac{d^2 G}{dx^2} + \underbrace{G}_{O(\epsilon)}
    \right\}
    &=
    \underbrace{\int_{x' - \epsilon}^{x' + \epsilon}
    dx\,
    \delta\left( x - x' \right)}_{1}
\end{align}
which gives
\begin{align}
    \left.\frac{dG}{dx}\right|_{x'-\epsilon}^{x'+\epsilon}
    = 1
\end{align}
which means
\begin{align}
    \left. \frac{dG_{>}}{dx}\right|_{x=x'}
    -
    \left. \frac{dG_{<}}{dx}\right|_{x=x'}
    &= 1\\
    -B_{<}\sin\left( x' \right) - A_{<}\cos\left( x' \right) &= 1
\end{align}
When all the dust settles,
the solution is
\begin{align}
    A_{<} &= -\cos\left( x' \right)\\
    B_{>} &= -\sin\left( x' \right)
\end{align}
Hence
\begin{align}
    G\left( x, x' \right)
    &=
    \begin{cases}
        G_{<}\left( x, x' \right)
        =
        -\sin\left( x \right)\cos\left( x' \right)
        &\text{for } x<x'\\
        G_{>}\left( x, x' \right)
        =
        -\cos\left( x \right)\sin\left( x' \right)
        &\text{for } x> x'
    \end{cases}
\end{align}
Notice that
\begin{align}
    G\left( x, x' \right) &= G\left( x', x \right)
\end{align}
You should check that your Greens function has this property.
If it doesn't your probably screwed up.
It's a powerful consistency check.

Some people like to write it in shorthand
\begin{align}
    G\left( x, x' \right)
    &=
    -\sin\left( x_{<} \right)\cos\left( x_{>} \right)
\end{align}
where $x_{<}$ refers to the smaller of $x$ and $x'$
and $x_{.}$ refers to the larger of $x$ and $x'$.
To get the final solution
\begin{align}
    y(x) &=
    \int_{0}^{\pi/2}dx'\,
    G\left( x, x' \right)
    f\left( x' \right)\\
    &=
    \int_{0}^{x}
    dx'
    f\left( x' \right)
    G_{>}\left( x, x' \right)
    +
    \int_{0}^{\pi/2}dx\,
    f\left( x' \right)
    G_{<}\left( x, x' \right)\\
    &= -\cos(\pi)\left[
    \int_{0}^{\pi}dx'\,
    f\left( x' \right)
    \sin\left( x' \right)
    \right]
    -
    \sin\left( x \right)
    \left[
    \int_{x}^{\pi/2}dx'\,
    f\left( x' \right)
    \cos\left( x' \right)
    \right]
\end{align}
So that's the solution.
This is a standard way to solve inhomogeneous differential equations.

You've actually been using this method for a long time.
The potential from a point charge admits an interpretation as the Green function
of the Laplacian operator $\nabla^2$.
Consider the Poisson equation
\begin{align}
    \nabla^2\phi &=
    -\frac{\rho}{\epsilon_0}
\end{align}
For a point charge $q$ located at $\vec{x}'$,
\begin{align}
    \rho\left( \vec{x}' \right)
    &=
    q\delta^{(3)}\left( \vec{x} - \vec{x}' \right).
\end{align}
The resulting potential $\hat{\phi}$ satisfies
\begin{align}
    \nabla^2\hat{\phi}
    &=
    -\frac{q}{\epsilon_0}
    \delta^{(3)}\left( \vec{x} - \vec{x}' \right)
\end{align}
with
\begin{align}
    \hat{\phi}
    &=
    \frac{1}{4\pi\epsilon_0}
    \frac{q}{\left| \vec{x} - \vec{x}'\right|}
\end{align}
By definition,
the Green function for the Laplacian operator is required to satisfy
\begin{align}
    \nabla^2 G
    &=
    -4\pi \delta^{(3)}\left( \vec{x} - \vec{x}' \right)
\end{align}
This Poisson equations is the kind of equation we're trying to solve.
We set up a Greens function,
replacing the source term by a delta function,
just the way we did earlier.
We use the same differential operator,
but we replace the arbitrary source by a $\delta$ function.
The only difference is the factor of $-4\pi$.
Previously we set it to 1,
which is more standard,
but in electrostatics,
it's just a convention we put a $-4\pi$ there.

Comparing we see that
\begin{align}
    G\left( \vec{x}, \vec{x}' \right)
    &=
    \frac{4\pi\epsilon_0}{q}\hat{\phi}
    =
    \frac{1}{\left|\vec{x} - \vec{x}'\right|}
\end{align}
Then the solution of the Poisson equation is
\begin{align}
    \phi\left( \vec{x} \right)
    &=
    -\frac{1}{4\pi}
    \int d^3\vec{x}\,
    G\left( x, x' \right)
    \left\{
    -\frac{\rho(x)}{\epsilon_0}
    \right\}\\
    &=
    \frac{1}{4\pi\epsilon_0}
    \int d^3x
    \frac{\rho(x)}{\left|\vec{x} - \vec{x}'\right|}
\end{align}

Let's summarize the results.
Consider the example we had
\begin{align}
    y'' + y &= f(x)\\
    G'' + G &= \delta\left( x - x' \right)\\
    G\left( x, x' \right) &= -\sin\left( x_{<} \right)\cos\left( x_{>}
    \right)\\
    y &= \int dx\, f\left( x' \right) G\left( x, x' \right)
\end{align}
The same method applies for electrostatics.
\begin{align}
    \nabla^2 \phi &=
    -\frac{\rho\left( \vec{x} \right)}{\epsilon_0}\\
    \nabla^2 G &=
    -4\pi \delta^{(3)}\left( \vec{x} - \vec{x}' \right)\\
    G\left( \vec{x}, \vec{x}' \right)
    &=
    \frac{1}{\left|\vec{x} - \vec{x}'\right|}\\
    \phi &=
    -\frac{1}{4\pi}\int d^3\vec{x}\,
    G\left( \vec{x}, \vec{x}' \right)
    \left\{
    \frac{-\rho\left( \vec{x}' \right)}{\epsilon_0}
    \right\}
\end{align}
So this is the general method for solving linear inhomogeneous differential
equations.
$G\left( \vec{x}, \vec{x}' \right)$
is the potential from a charge $q=4\pi\epsilon_0$
located at $\vec{x}=\vec{x}'$.

Consider a charge $q$ near a grounded conducting sphere.
For a point charge $q$,
\begin{align}
    \phi(\vec{x})
    &=
    \frac{1}{4\pi\epsilon_0}
    \left\{
    \frac{q}{\left|\vec{x} - \vec{x}'\right|}
    -
    \frac{aq}{\left|\vec{x}'\right|}
    \frac{1}{\left|\vec{x} - \frac{a^2}{\left|\vec{x}'\right|^2}\vec{x}'\right|}
    \right\}
\end{align}
which we obtained by the method of images.
Now consider
\begin{align}
    \nabla^2 \phi
    &=
    -\frac{q\delta^{(3)}\left( \vec{x} - \vec{x}' \right)}{\epsilon_0}
\end{align}
with boundary conditions
$\phi=0$ at $\left|\vec{x}\right|=a$.
$\phi=0$ at $\left|\vec{x}\right|=\infty$.

Then the Green function is
\begin{align}
    G\left( \vec{x}, \vec{x}' \right)
    &=
    \left\{
    \frac{1}{\left|\vec{x} - \vec{x}'\right|}
    -
    \frac{a}{\left|\vec{x}'\right|}
    \frac{1}{\left|
    \vec{x}
    -
    \frac{a^2}{\left|\vec{x}\right|^2}
    \vec{x}'
    \right|}
    \right\}
\end{align}
such that for a more general charge distribution,
\begin{align}
    \phi(\vec{x})
    &=
    \frac{1}{4\pi\epsilon_0}
    \int d^3\vec{x}'
    G\left( \vec{x}, \vec{x}' \right)
    \rho\left(\vec{x}'\right)
\end{align}

We need to generalize the Green function method to problems where $\phI$ or
$\vec{E}\cdot\hat{r}$ are specified on boundaries.
Start from Green's theorem.
\begin{align}
    \int_V
    \left(
    \phi \nabla^2 \psi
    - \psi \nabla^2 \phi
    \right)
    dV
    &=
    \oint_S
    \left[
    \phi\left( \vec{\nabla}\psi\cdot\hat{n} \right)
    - \psi\left( \vec{\nabla}\phi\cdot\hat{n} \right)
    \right]
    dS
\end{align}
To prove this is straightforward.
\begin{align}
    \psi\nabla^2 \psi &=
    \vec{\nabla}\cdot\left(
    \phi\vec{\nabla}\psi
    \right)
    - \vec{\nabla}\psi
    \cdot \vec{\nabla} \psi\\
    \psi\nabla^2\phi
    &=
    \vec{\nabla}\cdot\left(
    \psi\vec{\nabla}\phi
    \right)
    -
    \vec{\nabla}\phi
    \cdot\vec{\nabla}\cdot\vec{\nabla}\psi
\end{align}
Subtract
\begin{align}
    \phi\nabla^2\psi
    - \psi\nabla^2\phi
    &=
    \vec{\nabla}\cdot\left(
    \phi\vec{\nabla}\psi
    -
    \psi\vec{\nabla}\phi
    \right)
\end{align}
Integrate over volume and apply divergence theorem.

For $\phi$ choose the potential.
For $\psi$ choose the Green function $G\left( \vec{x}, \vec{x}' \right)$.

\begin{align}
    \vec{\nabla}'^2 G\left( \vec{x}, \vec{x}' \right)
    &= -4\pi \delta^{(3)}\left( \vec{x}' - \vec{x} \right)
\end{align}
Then
\begin{align}
    \int_V \left[
    \phi\left( \vec{x}' \right)
    \underbrace{
    {\nabla'}^{2}
    \psi\left( \vec{x}' \right)
    }_{-4\pi\delta^{3}\left( \vec{x}' - \vec{x} \right)}
    -
    \psi\left( \vec{x}' \right)
    \underbrace{{\nabla'}^2 \phi\left( \vec{x}' \right)}_{
    -\rho\left( \vec{x}' \right)/\epsilon_0
    }
    \right]
    d^3\vec{x}'
    &=
    \int \left[
    \phi\left( \vec{x}' \right)
    \left(
    \vec{\nabla}' \psi\left( \vec{x}' \right) \cdot \hat{n}'
    \right)
    -
    \psi\left( \vec{x}' \right)
    \left(
    \vec{\nabla}' \phi\left( \vec{x}' \right) \cdot \hat{n}'
    \right)
    \right]
    dS'
\end{align}
So
\begin{align}
    \phi\left( \vec{x} \right)
    &=
    \frac{1}{4\pi\epsilon_0}
    \int_V
    \rho\left( \vec{x}' \right)
    G\left( \vec{x}, \vec{x}' \right)
    d^3\vec{x}'
    +
    \frac{1}{4\pi}
    \oint_S
    \left[
    G\left( \vec{x}, \vec{x}' \right)
    \left(
    \vec{\nabla}' \phi\left( \vec{x}' \right)\cdot\hat{n}'
    \right)
    -
    \phi\left( \vec{x}' \right)
    \left(
    \vec{\nabla}' G\left( \vec{x}, \vec{x}' \right)
    \cdot \hat{n}'
    \right)
    \right]
    dS'
\end{align}


\subsection{Principled statistical inference}
\begin{frame}{Principle I: the sufficiency principle}
Sufficiency plays a central role in all of Statistics.
\begin{defn}[Sufficient statistic]
 Let $x \sim f(x \mid \theta)$.
 We say $T : \mathcal{X} \to \mathbb{R}$ is a \textbf{sufficient statistic} for the parameter $\theta$ if $\pr(X = x \mid T(x), \theta)$ is independent of $\theta$.
\end{defn}
This is the basis for a cornerstone of Statistics,
\begin{theo}[Factorisation theorem]
 Under mild regularity conditions, we can write:
 $$ f(x \mid \theta) = g(T(x) \mid \theta) h(x \mid T(x)).$$
\end{theo}
We can now state
\begin{idea}[Sufficiency principle (SP)]
\label{idea:SP}
 For $x, y \in \mathcal{X}$, if $T$ is sufficient for $\theta$ and $T(x) = T(y)$, then $x$ and $y$ should lead to the same inferences about $\theta$.
\end{idea}
\end{frame}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{frame}[allowframebreaks]{Principle II: the Likelihood principle}
The Likelihood Principle (LP) is a key concept in Statistics, of particular Bayesian Statistics.
\begin{idea}[Likelihood Principle]
\label{idea:LP}
 The information brought by an observation $x \in \mathcal{X}$ about a parameter $\theta \in \boldsymbol{\Theta}$ is \textbf{completely} contained in the likelihood function $l(\theta \mid x) \propto f(x \mid \theta)$.
\end{idea}
\begin{example}[Uma vez Flamengo...]
 Suppose a pollster is interested in estimating the fraction $\theta$ of football fans that cheer for Clube de Regatas do Flamengo (CRF).
 They survey $n=12$ people and get $x=9$ supporters and $y=3$ ``antis''.
 Consider the following two designs:
 \begin{itemize}
  \item[i)] Survey $12$ people and record the number of supporters;
  \item[ii)] Survey until they get $y=3$.
 \end{itemize}
The likelihoods for both surveys are, respectively,
\begin{align*}
x \sim \operatorname{Binomial}(n, \theta) \implies l_1(\theta \mid x, n) &= \binom{n}{x} \theta^{x}(1-\theta)^{n-x},\\
n \sim \operatorname{Negative Binomial}(y, 1-\theta) \implies l2(\theta \mid n, y) &=  \binom{n-1}{y-1}y (1-\theta)^{n-y} \theta^y,
\end{align*}
hence
\begin{equation*}
 l_1(\theta) \propto l_2(\theta) \propto \theta^{3}(1-\theta)^9.
\end{equation*}
Therefore, we say that these two experiments bring exactly the same information about $\theta$.
\end{example}
A generalised version of the LP can be stated as follows:
\begin{theorem}[\textbf{Likelihood Proportionality Theorem}~\citep{Goncalves2019}]
 Let  $\Theta$ be a nonempty set and $\mathcal{P} = \{ P_\theta; \theta \in \Theta \}$ be a family of probability measures on $(\Omega, \mathcal{A})$ and $\nu_1$ and $\nu_2$ be $\sigma$-finite measures on $(\Omega, \mathcal{A})$.
 Suppose $P \ll \nu_1$ and $P \ll \nu_2$ for all $P \in \mathcal{P}$.
 Then there exists  a measurable set $A \in \mathcal{A}$  such that $P_\theta(A) = 1$ for all $\theta \in \Theta$ and there exist $f_{1,\theta} \in \left[ \frac{dP_\theta}{d\nu_1}\right]$ and $f_{2,\theta} \in \left[ \frac{dP_\theta}{d\nu_2}\right]$ and a measurable function $h$ such that
 \begin{equation*}
  f_{1,\theta}(\omega) = h(\omega)f_{2,\theta}(\omega), \forall\, \theta \in \Theta\, \forall\, \omega \in A.
 \end{equation*}
\end{theorem}
\end{frame}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{frame}{Principle III: stopping rule principle}
A subject of contention between inference paradigms is the role of stopping rules in the inferences drawn.
\begin{idea}[Stopping rule principle (SRP)]
\label{idea:SRP}
Let $\tau$ be a stopping rule directing a series of experiments $\mathcal{E}_1, \mathcal{E}_2, \ldots$, which generates data $\boldsymbol{x} = (x_1, x_2, \ldots)$.
Inferences about $\theta$ should depend on $\tau$ only through $\boldsymbol{x}$.
\end{idea}
\begin{example}[Finite stopping rules]
 Suppose experiment $\mathcal{E}_i$ leads to the observation of $x_i \sim f(x_i \mid \theta)$ and let $\mathcal{A}_i \subset \mathcal{X}_1 \times \ldots \times \mathcal{X}_i$ be a sequence of events.
 Define
 $$ \tau := \inf \left\{ n : (x_1, \ldots, x_n) \in \mathcal{A}_n \right\}.$$
 It can be shown that $\pr(\tau < \infty) = 1$ (exercise 1.20 BC).
\end{example}
\end{frame}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{frame}{Principle IV: the conditionality principle}
We will now state one of the main ingredients of the derivation of the LP.
The Conditionality Principle (CP) is a statement about the permissible inferences from randomised experiments.
\begin{idea}[Conditionality Principle]
\label{idea:CP}
 Let $\mathcal{E}_1$ and $\mathcal{E}_2$ be two experiments about $\theta$.
 Let $Z \sim \operatorname{Bernoulli}(p)$ and
 \begin{itemize}
  \item If $Z=1$, perform $\mathcal{E}_1$ to generate $x_1 \sim f_1(x_1 \mid \theta)$;
  \item If $Z=0$ perform $\mathcal{E}_2$ to generate $x_2 \sim f_2(x_2 \mid \theta)$.
 \end{itemize}
Inferences about $\theta$ should depend \textbf{only} on the selected experiment, $\mathcal{E}_i$.
\end{idea}
\end{frame}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{frame}{Deriving the Likelihood Principle}
\cite{Birnbaum1962} showed that the simpler and mostly uncontroversial Sufficiency and Conditionality principles lead to the Likelihood Principle.
\begin{theo}[Birnbaum's theorem~\citep{Birnbaum1962}]
\label{thm:Birnbaum}
 \begin{equation}
  \operatorname{SP} + \operatorname{CP} \implies \operatorname{LP}.
 \end{equation}
\end{theo}
\begin{proof}
 Sketch:
 \begin{itemize}
  \item Define a function $\operatorname{EV}(\mathcal{E}, x)$ to quantify the evidence about $\theta$ brought by data $x$ from experiment $\mathcal{E}$ and consider a randomised experiment $\mathcal{E}^*$ in which $\mathcal{E}_1$ and $\mathcal{E}_2$ are performed with probability $p$;
  \item Show that CP implies
  $\operatorname{EV}(\mathcal{E}^*, (j, x_j)) = \operatorname{EV}(\mathcal{E}_j, x_j), j = 1, 2$;
  \item Show that SP implies
  $\operatorname{EV}(\mathcal{E}^*, (1, x_1)) = \operatorname{EV}(\mathcal{E}^*, (2, x_2))$ when
  $$ l(\theta \mid x_1) = c l(\theta \mid x_2).$$
 \end{itemize}
\end{proof}
See~\cite{Robert2007}, pg.18 for a complete proof.
\end{frame}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{frame}{Recommended reading}
\begin{itemize}
 \item[\faBook] \cite{Robert2007} Ch. 1;
 \item[\faForward] Next lecture: \cite{Robert2007} Ch. 2 and $^\ast$ \cite{Schervish2012} Ch.3;
%  \item {\large\textbf{Recommended exercises}}
%  \begin{itemize}
%   \item[\faBookmark] \cite{Robert2007}.
%   \begin{itemize}
%    \item Sections.
%    \item $^\ast$ Sections .
%   \end{itemize}
%   \end{itemize}
 \end{itemize}
\end{frame}


\chapter{Models, Statistical Inference and Learning}

\begin{ex}
  Let $X_1,\ldots,X_n\sim\text{Poisson}(\lambda)$ and let
  $\widehat{\lambda}=n^{-1}\sum_{i=1}^nX_i$. Then
  \[
    \bias{\widehat{\lambda}}
    =\E{n^{-1}\sum_{i=1}^nX_i}-\lambda
    =n^{-1}\sum_{i=1}^n\E{X_i}-\lambda
    =0,
  \]
  \[
    \var{\widehat{\lambda}}
    =\var{\frac{1}{n}\sum_{i=1}^n X_i}
    =\frac{1}{n^2}\sum_{i=1}^n \var{X_i}
    =\frac{n\lambda}{n^2}=\frac{\lambda}{n},
  \]
  and therefore
  \[
    \se(\widehat{\lambda})
    =\sqrt{\frac{\lambda}{n}},
  \]
  and
  \[
    \mse{\widehat{\lambda}}=
    0+\var{\widehat{\lambda}}=\frac{\lambda}{n}.
  \]
\end{ex}

\begin{ex}
  Let $X_1,\ldots,X_n\sim\text{Uniform}(0,\theta)$ and let
  $\widehat{\theta} = \max\{X_1,\ldots, X_n\}$. Note that
  \[
    \P{\max\{X_1,\ldots,X_n\}\leq x}
    =\P{X_1\leq x}\cdots\P{X_n\leq x}
    =x^n/\theta^n
  \]
  for $0\leq x\leq\theta$.

  Therefore,
  \[
    f_{\widehat{\theta}}(x)
    =\begin{cases}
      nx^{n-1}/\theta^n & 0\leq x\leq 1,    \\
      0                 & \text{otherwise},
    \end{cases}
  \]
  and so
  \[
    \E{\widehat{\theta}}
    =\int_0^\theta\!xnx^{n-1}/\theta^n\,\d{x}
    =\frac{n}{n+1}\theta.
  \]
  Hence,
  \[
    \bias{\widehat{\theta}}
    =\frac{n}{n+1}\theta-\theta=-\frac{\theta}{n+1}.
  \]
  Moreover,
  \[
    \E{\widehat{\theta}^2}
    =\int_0^\theta\!x^2nx^{n-1}/\theta^n\,\d{x}
    =\frac{n}{n+2}\theta^2,
  \]
  and therefore
  \[
    \var{\widehat{\theta}}
    =\E{\widehat{\theta}^2}
    -\E{\widehat{\theta}}^2
    =\frac{n}{n+2}\theta^2
    -\frac{n^2}{(n+1)^2}\theta^2
    =\frac{n}{(n+1)^2(n+2)}\theta^2,
  \]
  \[
    \se(\widehat{\theta})
    =\frac{\theta}{n+1}\sqrt{\frac{n}{n+2}},
  \]
  and
  \[
    \mse{\widehat{\theta}}
    =\left[\bias{\widehat{\theta}}\right]^2+\var{\widehat{\theta}}
    =\frac{2\theta^2}{(n+1)(n+2)}.
  \]
\end{ex}

\begin{ex}
  Let $X_1,\ldots,X_n\sim\text{Uniform}(0,\theta)$ and let
  $\widehat{\theta} = 2\overline{X}_n$. Then
  \[
    \E{\widehat{\theta}}
    =\E{\frac{2}{n}\sum_{i=1}^n X_i}
    =\frac{2}{n}\sum_{i=1}^n \E{X_i}
    =\frac{2}{n}\cdot \frac{n\theta}{2}=\theta,
  \]
  and
  \[
    \bias{\widehat{\theta}}
    =\E{\widehat{\theta}}-\theta
    =0.
  \]
  Likewise,
  \[
    \var{\widehat{\theta}}
    =\var{\frac{1}{n}\sum_{i=1}^n X_i}
    =\frac{1}{n^2}\sum_{i=1}^n \var{X_i}
    =\frac{n\theta^2}{12n^2}
    =\frac{\theta^2}{12n},
  \]
  \[
    \se(\widehat{\theta})
    =\frac{\theta}{\sqrt{12 n}},
  \]
  and therefore
  \[
    \mse{\widehat{\theta}}
    =\left[\bias{\widehat{\theta}}\right]^2+\var{\widehat{\theta}}
    =\frac{\theta^2}{12n}.
  \]
\end{ex}


\subsection{Determinants of 2x2 and 3x3 matrices}

\subsubsection{The determinant of a 2x2 matrix}

\(M=\begin{bmatrix}a & b\\c & d\end{bmatrix}\)

\(|M|=ad-bc\)

\subsubsection{The determinant of a 3x3 matrix}

\(M=\begin{bmatrix}a & b & c\\d & e & f\\g & h & i\end{bmatrix}\)

\(|M|=aei+bfg+cdh-ceg-dbi-afh\)


\subsubsection{Random Forest Regression}
\label{rf}

\cite{breiman1984} introduce the classification and regression tree
    (CART) model that is built around the idea that a single binary
    decision tree maps learned combinations of intervals of the feature
    columns to a label.
Thus, each sample in the training set is associated with one leaf node that
    is reached by following the tree from its root and branching along the
    arcs according to some learned splitting rule per intermediate node that
    compares the sample's realization for the feature specified by the rule to
    the learned decision rule.
While such models are computationally fast and offer a high degree of
    interpretability, they tend to overfit strongly to the training set as
    the splitting rules are not limited to any functional form (e.g., linear)
    in the relationship between the features and the labels.
In the regression case, it is common to maximize the variance reduction $I_V$
    from a parent node $N$ to its two children, $C1$ and $C2$, as the
    splitting rule.
\cite{breiman1984} formulate this as follows:
$$
I_V(N)
=
\frac{1}{|S_N|^2} \sum_{i \in S_N} \sum_{j \in S_N}
    \frac{1}{2} (y_i - y_j)^2
- \left(
    \frac{1}{|S_{C1}|^2} \sum_{i \in S_{C1}} \sum_{j \in S_{C1}}
        \frac{1}{2} (y_i - y_j)^2
    +
    \frac{1}{|S_{C2}|^2} \sum_{i \in S_{C2}} \sum_{j \in S_{C2}}
        \frac{1}{2} (y_i - y_j)^2
\right)
$$
$S_N$, $S_{C1}$, and $S_{C2}$ are the index sets of the samples in $N$, $C1$,
    and $C2$.

\cite{ho1998} and then \cite{breiman2001} generalize this method by combining
    many CART models into one forest of trees where every single tree is
    a randomized variant of the others.
Randomization is achieved at two steps in the training process:
First, each tree receives a distinct training set resampled with replacement
    from the original training set, an idea also called bootstrap
    aggregation.
Second, at each node a random subset of the features is used to grow the tree.
Trees can be fitted in parallel speeding up the training significantly.
For prediction at the tree level, the average of all the samples at a
    particular leaf node is used.
Then, the individual values are combined into one value by averaging again
    across the trees.
Due to the randomization, the trees are decorrelated offsetting the
    overfitting.
Another measure to counter overfitting is pruning the tree, either by
    specifying the maximum depth of a tree or the minimum number of samples
    at leaf nodes.

The forecaster must tune the structure of the forest.
Parameters include the number of trees in the forest, the size of the random
    subset of features, and the pruning criteria.
The parameters are optimized via grid search: We train many models with
    parameters chosen from a pre-defined list of values and select the best
    one by CV.
RFs are a convenient ML method for any dataset as decision trees do not
    make any assumptions about the relationship between features and labels.
\cite{herrera2010} use RFs to predict the hourly demand for water in an urban
    context, a similar application as the one in this paper, and find that RFs
    work well with time series type of data.


\subsection{Dissimilarity metrics}

The reconstruction of the puzzle is based on a dissimilarity metric between all the pieces. The higher the dissimilarity between two given pieces, the lower priority is given to the fact that these pieces are placed next to each other. Several dissimilarity metrics are introduced.

\autoref{fig:side} depicts the working principle of the dissimilarity computation. The intensity of the red channel for the right side of a piece of the puzzle is plotted along with the left side of two others pieces. One can see that the matching piece has nearly the same profile as the starting piece (nearly superimposed), while the third one is very different. Of course, the difference between matching and non matching pieces are not always that obvious.

\begin{figure}[H]
    \centering
    \input{fig/side.tex}
    \caption{Illustration of the principle of dissimilarity computation by showing the red channel intensities for three pieces.}
    \label{fig:side}
\end{figure}

\paragraph{Sum of Squared Distances (SSD)} \mbox{}

The SSD between two pieces $x_i$ and $x_j$ is obtained by summing up the squared differences of the pixels along the pieces' juxtaposing boundaries, for each color channel (red, green and blue) \cite{robust}. For a piece $x$ of size $M \times M$, $x(m,n,c)$ denotes the value of the pixel at the coordinates $(m,n)$ in the color channel $c$. The SSD between $x_i$ and $x_j$, $x_i$ being placed at the left of $x_j$, can be expressed as

\begin{equation}
    D_{LR,SSD}(x_i, x_j) = \sum_{c=1}^{3} \sum_{k=1}^{M} \left( x_i(k,M,c) - x_j(k,M,c)\right)^2
\end{equation}

This can be extended to the cases where $x_i$ is placed on the right, on the top or on the bottom of $x_j$.

\paragraph{$(L_p)^q$ norm} \mbox{}

The SSD is actually the $L_2$ norm of the two vectors that form the boundaries of the two pieces. The $(L_p)^q$ can also be used, and it is defind as

\begin{equation}\label{eq:lpq}
    \hspace{-0.3cm}
    D_{LR,pq}(x_i, x_j) = \left( \sum_{c=1}^{3} \sum_{k=1}^{M} \left( \left| x_i(k,M,c) - x_j(k,M,c)\right|\right)^p \right)^{\frac{p}{q}}
\end{equation}

Hence, the SSD is a particular case of $(L_p)^q$ norm with $p=q=2$. It was experimentally found in \cite{greedy} that the good results were obtained for $p=3/10$ and $q=1/16$, they will thus be used in the solver. Instinctively, it is due to the fact that the $L_2$ norm penalizes a lot large boundary differences even though such differences may occur in natural images.

\paragraph{Mahalanobis Gradient Compatibility (MGC)}

While the first two metrics penalize differences in the pixels intensities, the MGC (introduced in \cite{Gallagher}) instead penalizes differences in the intensity of the gradient. If a piece has a strong gradient near its edge, it is expected that the juxtaposed piece will continue the gradient. The penalty for a deviation is computed with the Mahalanobis distance.

Looking at the dissimilarity between the right side of $x_i$ and the left side of $x_j$, let $\mu_{iL}(c)$ be the average color difference in the color channel $c$ between the two last columns of $x_i$:

\begin{equation}
    \mu_{iL}(c) = \frac{1}{M} \sum_{k=1}^{M} x_i(k,M,c) - x_i(k,M-1,c)
\end{equation}

The array of gradients of dimension $M$ by 3 $G_{ijLR}$ is then introduced, with $G_{ijLR}(k,c)$ denoting the color difference between the right side of $x_i$ and the left side of $x_j$ for the color channel $c$ at the row $k$:

\begin{equation}
    G_{ijLR}(k,c) = x_j(k,1,c) - x_i(k,M,c)
\end{equation}

By denoting $S_{iL}$ the 3 by 3 covariance matrix obtained from the gradient difference at the right of $x_i$, the gradient dissimilarity is given by:

\begin{gather}
    D'_{LR}(x_i,x_j) = \sum_{k=1}^{M} D'_{LR,k}\\
    D'_{LR,k} = \left( G_{ijLR}(k) - \mu_{iL} \right) S_{iL}^{-1} \left( G_{ijLR}(k) - \mu_{iL} \right)^T
\end{gather}

The final MGC symmetric dissimilarity metric for placing the piece $x_i$ at the left of $x_j$ is given by:

\begin{equation}
    D_{LR,MGC}(x_i,x_j) = D'_{LR}(x_i,x_j) +  D'_{RL}(x_j,x_i)
\end{equation}

\paragraph{Combining MGC and SSD (M+S)} \mbox{}

It was proposed in \cite{robust} that the SSD and the MGC convey complementary information so that they could be combined to provide more accuracy. The M+S dissimilarity metric is hence defined as the weighed product of the SSD and the MGC:

\begin{equation}\label{eq:ms}
    D_{M+S}(x_i,x_j) = D_{MGC}(x_i,x_j) \times \left( D_{SSD}(x_i,x_j) \right)^{\frac{1}{r}}
\end{equation}

The value of the weighing parameter $r$ giving the best results is investigated later.

%The obtained dissimilarities are then normalized \cite{robust} \cite{Gallagher}. For a given piece, all its dissimilarities are divided by the

\paragraph{Combining MGC and $(L_p)^q$ (M+pq)} \mbox{}

In the same idea, the MGC and the $(L_p)^q$ can be combined in a weighed product. This method will be referred to as the M+pq in the latter.

\begin{equation}\label{eq:mpq}
    D_{M+pq}(x_i,x_j) = D_{MGC}(x_i,x_j) \times \left( D_{pq}(x_i,x_j) \right)^{\frac{1}{r}}
\end{equation}

\subsection{Placement algorithm}

Once the dissimilarity is evaluated between every piece of the puzzle, for every orientation (top, bottom, right, left), the pieces should be placed accordingly to these measures. The implemented algorithm is a two step iterative method, and it is inspired from what was proposed in \cite{greedy}. An illustration of its functioning is depicted in \autoref{fig:placement}.

The starting piece is chosen to be the one having the lowest dissimilarity with any other piece of the puzzle. During the first step, all the pieces that have already been placed are scanned. The next piece that will be placed on the puzzle  is the one that has the lowest dissimilarity with any of the already placed ones. Piece by piece, the puzzle is then reconstructed. When a piece has to be placed in between multiple others, the dissimilarity with all of these is taken into account.

The second step starts when the puzzle is full. The algorithm then looks for \textit{symmetric matches} in all the juxtaposed pieces. Two pieces $x_i$ and $x_j$ are symmetrically matched if the best possible match for $x_i$ in every possible orientation is $x_j$ and the best possible match for $x_j$ is $x_i$ in the opposite orientation. For example, if the pieces labeled $x_3$ and $x_7$ are juxtaposed at the end of the first step, they will be considered as symmetric matches if the best possible match for $x_3$ is $x_7$ on its right, and the best match for $x_7$ is $x_3$ on its left. From these symmetric matches, a \textit{segment} is extracted from the image. It is defined as the biggest portion of the puzzle only composed of symmetrically matched pieces, and will serve as starting point for the next iteration. The iteration process stops when the result after the reconstruction step is not changing anymore.

\begin{figure}[H]
    \centering
%     \subfloat[The puzzle is constructed piece by piece.]{\includegraphics[height=0.25\textheight]{fig/algo_start.eps}
% \label{fig:placement1}}
% \hfil
%     \subfloat[The puzzle is constructed piece by piece.]{\includegraphics[height=0.25\textheight]{fig/algo_start.eps}
% \label{fig:placement1}}
% \hfil
%     \subfloat[The puzzle is constructed piece by piece.]{\includegraphics[height=0.25\textheight]{fig/algo_start.eps}
% \label{fig:placement1}}
% \hfil
%     \subfloat[The puzzle is constructed piece by piece.]{\includegraphics[height=0.25\textheight]{fig/algo_start.eps}
% \label{fig:placement1}}
% \hfil
    \begin{subfigure}{0.25\textheight}
        \centering
        \includegraphics[width=\textwidth]{fig/algo_start.eps}
        \caption{The puzzle is constructed piece by piece.}
        \label{fig:placement1}
    \end{subfigure}%

    \begin{subfigure}[b]{0.25\textheight}
        \centering
        \includegraphics[width=\textwidth]{fig/algo_fin1.eps}
        \caption{At the end of the first placement phase, errors are present.}
        \label{fig:placement2}
    \end{subfigure}%

    \begin{subfigure}[b]{0.25\textheight}
        \centering
        \includegraphics[width=\textwidth]{fig/algo_cut.eps}
        \caption{The biggest segment is extracted from the previous image and will serve as start for the next iteration.}
        \label{fig:placement3}
    \end{subfigure}%

    \begin{subfigure}[b]{0.25\textheight}
        \centering
        \includegraphics[width=\textwidth]{fig/algo_fin2.eps}
        \caption{The reconstruction is nearly perfect after two iterations.}
        \label{fig:placement4}
    \end{subfigure}

    \caption{Illustration of the reconstruction algorithm}
    \label{fig:placement}
\end{figure}

The Matlab implementation of the algorithm was optimized using matrix computations, and the full solving of a 432 parts puzzle takes less than 10 seconds.


%================================
\section{Review of Metric Spaces}
%================================

%--------------------------------
\begin{definition}
	Let $X$ be a set. A \textit{metric} on $X$ is a function $\rho: X \times X \to \mathbb R_{\ge 0}$, such that $\forall x, y, z \in X$, the following (metric axioms) holds:
	\begin{enumerate}[M1.]
		\item $\rho(x,y) = 0 \iff x = y$ (identity of indiscernibles);
		\item $\rho(x,y) = \rho(y,x)$ (symmetry).
		\item $\rho(x,y) + \rho(y,z) \ge \rho(x,z)$ (triangle inequality);
	\end{enumerate}
	A \textit{metric space} is a set together with a metric on it, or more formally, a pair $(X,\rho)$ where $X$ is a set and $\rho$ is a metric on $X$.
\end{definition}
%--------------------------------

%--------------------------------
\begin{example} \
	\begin{enumerate}
		%--------------------------------
		\item The function $\rho_p: \mathbb R^n \times \mathbb R^n \to \mathbb R_{\ge 0}$ defined by $\forall p \in \overline{\mathbb R}_{\ge 1}$, $\forall x,y \in \mathbb R^n$,
			$$
			\rho_p (x,y) = \left( \sum_{i = 1}^n |x_i - y_i|^p \right)^\frac{1}{p},
			$$
			is a metric on $\mathbb R^n$. Clearly it satisfies identity of indiscernibles and symmetry. For triangle inequality, it is suggested by Minkowski inequality.

			Given $x \in \mathbb R^3$, $r \in \mathbb R_{\ge 0}$, and
			$$
			B_\rho = \left\{ y \in \mathbb R^3 \;|\; \rho (x,y) \le r \right\}.
			$$

			$\forall p, q \in \overline{\mathbb R}_{\ge 1}$, it is true that, $\forall x,y \in \mathbb R^n$,
			$$
			p \le q \implies \rho_p(x,y) \ge \rho_{q}(x,y).
			$$
			Thus, $B_p \subseteq B_q$.

			Geometrically, as $p = 1$, $B$ is a octahedron in $\mathbb R^3$ with center $x$ and radius $r$; as $p = 2$, $B$ is a sphere in $\mathbb R^3$ with center $x$ and radius $r$. It is easy to observe that as $p \to \infty$, $B$ tends to a cube in $\mathbb R^3$ with center $x$ and edge length $2r$; i.e.,
			$$
			\rho_\infty(x,y) = \lim_{p \to \infty} \rho_p (x,y) = \sup_{i \in \{1, \ldots, n\}} |x_i - y_i|.
			$$
		%--------------------------------

		%--------------------------------
		\item Let $f: (X, \rho) \to \mathbb R^n$ with $X \subseteq \mathbb R^m$ be a continuous map on $X$. Let $x, y \in X$, then $\rho': f[X] \times f[X] \to \mathbb R_{\ge 0}$ defined by
			$$
			\rho_p'(x,y) = \int_0^1 f(\ell(t))d_ps(t)
			$$
			where
			$$
			\ell (t) = x + t(y - a)
			$$
			and
			$$
			d_p s(t) = \left( \sum_{i = 1}^m \left|\frac{dg_i}{dt}(t)\right|^p\right)^\frac{1}{p} dt.
			$$
			with $p \ge \overline{\mathbb R}_{\ge 1}$ is a metric on $f[X]$.

			Fix $x$ and given $r \in \mathbb R_{\ge 0}$, the set
			$$
			B_p = \left\{ y \in \mathbb R^m : \rho_p'(x,y) \le r \right\}
			$$
			describes a set ``attached'' on $f[X]$ with center $x$.
			If $p = 2$, $m = 2$ and $n = 3$, and $f: [0, 2 \pi) \times [0, 2 \pi) \to \mathbb R^3$ is defined by
			$$
			f(\lambda, \phi) = \begin{cases}
				r \cos \lambda \sin \phi, \\
				r \sin \lambda \sin \phi, \\
				r \cos \phi,
			\end{cases}
			$$
			then $\rho_2'$ here is a \textit{great circle metric} defined by
			$$
			\rho_2' (x,y) = r\arccos(\sin x_\phi \sin y_\phi + \cos x_\phi \cos y_\phi \cos(x_\lambda - y_\lambda)).
			$$
		%--------------------------------

		%--------------------------------
		\item Let $a,b \in \mathbb R$ with $a \le b$, and $p \in \overline{\mathbb R}_{\ge 1}$, and $C[a,b]$ denote the set of continuous function $[a,b] \to \mathbb R$.

			Then $d_p$ defined by $\forall f, g \in C[a,b]$,
			$$
			\rho_{p}(f,g) = \left( \int_a^b |f - g|^{p} \right)^\frac{1}{p}
			$$
			is a metric on $C[a,b]$.

			Similar to $\rho_p$ on $\mathbb R^n$,
			$$
			B_{p} = \left\{ g \;|\; \rho(f, g) \le r \right\}
			$$
			defines a set with ``center'' $f$ and ``radius'' $r \in \mathbb R_{\ge 0}$.

			It also implies that, on $C[a,b]$, $\forall p, q \in \overline{\mathbb R}_{\ge 1}$, $\forall x,y \in \mathbb R^n$
			$$
			p \le q \implies d_p(f,g) \ge d_q(f,g),
			$$
			and, naturally, $B_p \subseteq B_q$. This is a straight corollary from the same case of $d_p$ on $\mathbb R^n$.
		%--------------------------------

		%--------------------------------
		\item Let $A$ be a set. The \textit{Hamming metric} $\rho$ on a set $A^n$ is given by $\forall x,y \in A^n$
			$$
			\rho(x,y) = \# \left\{ i \in \{1, \ldots, n\} : x_i \ne y_i \right\}.
			$$
			An example from Wikipedia. The word ``karolin'' and ``kathrin'' can be considered as tuples
			$$
			x = ( \mathrm{ k, a, r, o, l, i, n } ), \; y = ( \mathrm{k, a, t, h, r, i, n} ).
			$$
			For all $i \in \{0, \ldots, 6\} \setminus \{ 0, 1, 4, 6 \}$, $x_i \ne y_i$, and $\# (\{0, \ldots, 6\} \setminus \{ 0, 1, 4, 6 \}) = 3$, thus
			$$
			\rho(x,y) = 3.
			$$
		%--------------------------------

		%--------------------------------
		\item Let $(M, \rho)$ be a metric space (for example, $\rho = \rho_2$ on $\mathbb R^n$), and $X, Y \in \mathcal P(M)$. The Hausdorff metric $\rho_\mathrm{H}$ on $\mathcal P(M)$ is defined by
			$$
			\rho_\mathrm{H}(X,Y) = \max \left\{ \sup_{x \in X} \rho(x,Y), \sup_{y \in Y} \rho(X,y) \right\},
			$$
			where $\rho(a, B) = \inf_{b \in B} \rho(a,b)$ for all $B \in \mathcal P(M)$ and $a \in M$.

			This metric can be used to measure how close two figures (as sets of points) are.
		%--------------------------------
	\end{enumerate}
\end{example}
%--------------------------------

%--------------------------------
\begin{definition}
	Let $X$ be a metric space, let $x \in X$, and $\varepsilon > 0$. The \textit{open ball with center $x$ and radius $\varepsilon$}, or more briefly the \textit{open $\varepsilon$-ball about $x$} is the subset
	$$
	B(x, \varepsilon) = \{ y \in X: \rho (x,y) \le \varepsilon \} \subseteq X.
	$$
	Similarly, the \textit{closed $\varepsilon$-ball around $x$} is
	$$
	\overline B (x, \varepsilon) = \{ y \in X: \rho(x,y) \le \varepsilon \} \subseteq X.
	$$
\end{definition}
%--------------------------------

%--------------------------------
\begin{note}
	Clearly, the word ``ball'' does not mean it should look like a ball. Clearly, for all $x \in \mathbb R^3$, the ball $\{ y \in \mathbb R^3 : \rho_\infty (x,y) < 1 \}$ is a cube without its surface.

	And it is interesting to think that on $C[a,b]$ with conditions above,
	$$
	\{ g \in C[a,b] : \rho_p(f,g) < 1 \}
	$$
	defines a open ball in $C[a,b]$.

\end{note}
%--------------------------------

%--------------------------------
\begin{note}
	For hamming metric $\rho$ with conditions above, for $\varepsilon \in \mathbb R_{(0,1)}$, the ball
	$$
	\{ y \in A^n : \rho(x,y) < 1 \} = \{x\}.
	$$
	is a singleton.
\end{note}
%--------------------------------

%--------------------------------
\begin{definition}
	Let $X$ be a metric space.
	\begin{enumerate}[(i)]
		\item A subset $U$ of $X$ is \textit{open in $X$} (or an \textit{open subset of $X$}) iff
		$$
		\forall u \in U, \ \exists \varepsilon \in \mathbb R_{> 0} , \;\; B(u, \varepsilon ) \subseteq U.
		$$
		\item A subset $V$ is \textit{closed in $X$} iff $X\setminus V$ is open in $X$.
	\end{enumerate}
\end{definition}
%--------------------------------

%--------------------------------
\begin{note}
	Equivalently, $U$ is open in $X$ iff $\exists \varepsilon \in \mathbb R_{>0}$,
	$$
	U = \bigcup_{x \in U} B(x, \varepsilon );
	$$
	and $V$ is closed in $X$ iff
	$$
	V = X \setminus \bigcup_{x \in U} B(x, \varepsilon) = \bigcap_{x \in U} (X \setminus B(x, \varepsilon)),
	$$
	by De Morgan's Law.
\end{note}
%--------------------------------

%--------------------------------
\begin{definition}
	Let $X$ be a metric space, let $\{x_n\}_{n = 1}^{\infty}$ be a sequence in $X$ and let $x \in X$. Then $\{x_n\}$ \textit{converges} in $X$ iff
	$$
	\exists x \in X, \lim_{n \to \infty} d(x_n, x) = 0.
	$$
	Explicitly, then, $\{ x_n \}$ converges to $x$ iff
	$$
	\forall \varepsilon \in \mathbb R_{> 0}, \exists N \in \mathbb N_{\ge 1}, \forall n \in \mathbb N_{\ge N}, \quad d(x_n , x) < \varepsilon.
	$$
\end{definition}
%--------------------------------

%--------------------------------
\begin{note} \
	\begin{enumerate}
		\item Equivalently, $\{x_n\}$ converges in $X$ iff
			$$
			\exists x \in X, \ \forall \varepsilon \in \mathbb R_{>0}, \# (\{ x_n \} \cap B(x,\varepsilon)) = \aleph_0 \land \# ( \{x_n\} \setminus B(x, \varepsilon)) < \aleph_0.
			$$
			In other words, $B(x, \varepsilon)$ contains all but finitely many $x_n$.
		\item Let $X \subseteq S$. $\{x_n\}$ converges to $x \in S$ does not means it need to converge in $X$. For example $\mathbb Q \subseteq \mathbb R$, the sequence
			$$
			\left\{ x_n = \frac{1}{x} + r : r^2 = 2 \right\}_{n \in \mathbb N}
			$$
			does converge to $\sqrt{2} \in \mathbb R$, but $\sqrt{2} \notin \mathbb Q$, so $\{x_n\}$ converges in $\mathbb R$, but does not converge in $\mathbb Q$.
	\end{enumerate}
\end{note}
%--------------------------------

%--------------------------------
\begin{lemma}
	Let $X$ be a metric space and $V \subseteq X$. Then $V$ is closed in $X$ iff
	$$
	\forall \{x_n\}_{n = 1}^\infty \subseteq V, \ \forall x \in X, \quad \lim_{n \to \infty} d(x_n, x) = 0 \implies x \in V.
	$$
\end{lemma}
%--------------------------------

%--------------------------------
\begin{proof}
	Suppose $V$ is closed in $X$, then $X \setminus V$ is open in $X$. Suppose $\exists x \in X \setminus V$, such that $\exists \{x_n\}_{n = 1}^\infty \subseteq V$, $\{x_n\}$ converges to $x$, then $\forall \varepsilon \in \mathbb R_{> 0}$, $B(x, \varepsilon) \cap \{x_n\} \ne \emptyset$. $\{x_n\} \subseteq V$, so $B(x, \varepsilon) \cap V \ne \emptyset$. This implies that $X \setminus V$ is not open, then $V$ is not closed (for if $V$ is closed, then $X \setminus V$ is open). It is contradicted to the assumption.

	Now, suppose $V$ is not closed in $X$, then $X \setminus V$ is not open. Then, $\exists p \in X \setminus V$, such that $\forall \varepsilon \in \mathbb R_{> 0}$, $B(p, \varepsilon) \cap V \ne \emptyset$. This implies there are some $\{x_n\}_{n = 1}^\infty \subseteq V$, such that $B(p, \varepsilon)$ contains all but finite elements in $\{x_n\}$. Thus, $\{x_n\}$ converges to $p \in X \setminus V$, contradicting to the conditions.
\end{proof}
%--------------------------------

%--------------------------------
\begin{lemma}
	\label{lem: open sets are closed under blah blah blah}
	Let $X$ be a metric space, and $\mathcal T$ be the family of open subsets of $X$. Then,
	\begin{enumerate}[(i)]
		\item $\mathcal T$ is closed under arbitrary union.
		\item $\mathcal T$ is closed under finite intersection.
		\item $\emptyset, X \in \mathcal T$.
	\end{enumerate}
\end{lemma}
%--------------------------------

%--------------------------------
\begin{proof} \
	\begin{enumerate}
		\item Let $I$ be an index set. For all $i \in I$, let $U_i \in \mathcal T$. Then for some $\varepsilon \in \mathbb R_{> 0}$,
			$$
			U_i = \bigcup_{x \in U_i} B(x, \varepsilon).
			$$
			Let $U = \bigcup_{i \in I} U_i$, then we have,
			$$
			U = \bigcup_{i \in I} \bigcup_{x \in U_i} B(x, \varepsilon) = \bigcup_{x \in U} B(x, \varepsilon)
			$$
			for some $\varepsilon \in \mathbb R_{> 0}$.

		\item Let $\mathcal C$ be the family of closed subsets of $X$, and let $U, V \in \mathcal C$. Then for all $\{u_n\}_{n = 1}^\infty \subseteq U$, $\forall u \in X$, $\{u_n\}$ converges to $u$ implies that $u \in U$. It also holds for $U \cup V \supseteq U$. Similarly, for all $\{v_m\}_{m = 1}^\infty$, $\forall v \in X$, $\{v_m\}$ converges to $v$ implies $v \in V$. It also holds for $U \cup V \supseteq V$. Thus $U \cup V$ is closed.

			Then, $X \setminus (U \cup V) = (X \setminus U) \cap (X \setminus V)$ is open, where $X \setminus U$ and $X \setminus V$ are open for $U$ and $V$ are closed.

		\item $\emptyset = \bigcup_{i \in \emptyset} U_i$ for all $U_i \in \mathcal T$, so $\emptyset$ is open. $\emptyset = U \cap V$ for all mutually disjoint closed subsets $U, V \subseteq X$, so $\emptyset$ is closed, so $X = X \setminus \emptyset$ is open.
	\end{enumerate}
\end{proof}
%--------------------------------

%--------------------------------
\begin{lemma}
	Let $X$ be a metric space, and $\mathcal C$ be the family of all closed subsets of $X$. Then,
	\begin{enumerate}[(i)]
		\item $\mathcal C$ is closed under arbitrary intersection.
		\item $\mathcal C$ is closed under finite union.
		\item $\emptyset, X \in \mathcal C$.
	\end{enumerate}
\end{lemma}
%--------------------------------

%--------------------------------
\begin{proof}
	Let $\mathcal T$ be the family of all open subset of $X$, and let $I$ be any index set.
	\begin{enumerate}
		\item It has been proved that $\mathcal T$ is closed under arbitrary union, so by De Morgan's law, for any $i \in I$, if $U_i \in \mathcal T$, then
			$$
			X \setminus \bigcup_{i \in I} U_i = \bigcap_{i \in I} (X \setminus U_i) \in \mathcal C.
			$$
		\item It has been proved in Lemma \ref{lem: open sets are closed under blah blah blah}.
		\item It has been proved that $\emptyset$ is open in $X$. So $X = X \setminus \emptyset$ is closed in $X$.
	\end{enumerate}
\end{proof}
%--------------------------------

%--------------------------------
\begin{definition}
	\label{def: continuous function}
	Let $(X, \rho)$ and $(Y, \rho')$ be metric spaces. A function $f: (X, \rho) \to (Y, \rho)$ is \textit{continuous} on a point $p \in X$ iff
	$$
	\forall \varepsilon \in \mathbb R_{> 0}, \ \exists \delta \in \mathbb R_{> 0}, \ \forall x \in B(p, \delta), \quad f(x) \in B(f(p), \varepsilon).
	$$
\end{definition}
%--------------------------------

%--------------------------------
\begin{note} \
	\begin{enumerate}
		\item If $\rho$ is a discrete metric on $X$, then $B(p, \delta) = \{p\}$ for all $\delta$. Then, by definition, for all $\varepsilon$, $f(x) \in B(f(p), \varepsilon)$. So $f$ is continuous everywhere.

		\item On the contrary, if $\rho'$ is a discrete metric on $Y$, but for all $p\in X$, $\rho$ suffices for all $\delta \in \mathbb R_{> 0}$, $\# B(p, \delta) \ge \aleph_0$, then for some $\varepsilon \in \mathbb R_{>0}$, for all $\delta \in \mathbb R_{> 0}$, there exists $x \in B(p, \delta)$, such that $f(x) \notin B(f(p), \varepsilon)$. Thus $f$ is not continuous on such $p$.
	\end{enumerate}
\end{note}
%--------------------------------

%--------------------------------
\begin{lemma}
	Let $(X, \rho)$ and $(Y, \rho')$ be metric spaces and let $f: (X, \rho) \to (Y, \rho)$ be a function. The following are equivalent:
	\begin{enumerate}[(i)]
		\item $f$ is continuous on $X$;
		\item for all open $U \subseteq Y$, the preimage $f^{-1}[U] \subseteq X$ is open;
		\item for all closed $V \subseteq Y$, the preimage $f^{-1}[V] \subseteq X$ is closed.
	\end{enumerate}
\end{lemma}
%--------------------------------

%


\documentclass{article}
\usepackage{graphicx}
\usepackage{titletoc}
\usepackage{titlesec}
\usepackage{geometry}
\usepackage{fontspec, xunicode, xltxtra}
\usepackage{float}
\usepackage{cite}
\usepackage{amsmath}
\usepackage{listings}
\usepackage{titletoc}
\usepackage{bm}

\geometry{left=3cm,right=3cm,top=3cm,bottom=3cm}
\DeclareMathOperator*{\argmin}{argmin}
\DeclareMathOperator*{\argmax}{argmax}
\DeclareMathOperator*{\var}{var}
\DeclareMathOperator*{\expec}{E}

\begin{document}
\title{\textsf{Homework 5 for Pattern Recognition}}
\author{Fan JIN\quad (2015011506)}
\maketitle

\section*{Question 1.1}
{
    The optimal hyperplane satisfies $$f(x) = \sum_{i=1}^{n}{\alpha_i^{*} y_i K(x, x_i)} + b = 0.$$
    \begin{itemize}
        \item \textbf{(b)(d)(f) are of Gaussian kernels.}\quad This is obvious since the support vectors are distributed not only near the hyperplane, but also on the other side away from the hyperplane. This is a property of the Gaussian kernels. Moreover, \textbf{(d)} corresponds to $\sigma=0.1$ because it tends to overfit the positive samples. \textbf{(b)} must be with $\sigma=1$, as it has a straighter separate line. And \textbf{(f)} is in the middle, with $\sigma=0.5$.

        \item \textbf{(c) is of linear kernels.}\quad The hyperplane would follow a linear function of $x$ if the kernel $K(x, x_i)$ is linear. Thus, the straight line indicates a linear kernel in \textbf{(c)}.

        \item \textbf{(a) is of quadratic kernels.}\quad The The hyperplane would follow a quadratic function of $x$ if the kernel $K(x, x_i)$ is quadratic. This can be shown using the fact that $$(w^T x)^2 = (w^T x)^T (w^T x) = x^T (w w^T) x,$$ which is a quadratic form. The parabola thus indicates a quadratic kernel in \textbf{(a)}.

        \item \textbf{(e) is of cubic kernels.}\quad Similar to the quadratic kernels, a cubic hyperplane may come from the linear combination of many cubic terms with respect to $x$.
    \end{itemize}
}

\section*{Question 1.2}
{
    We prefer the linear kernel since the samples are linearly separable. The linear kernel, as long as feasible, means \emph{mathematical simplicity, intuitive interpretability, simple calulation,} as well as \emph{property of superpostion}. For example, the linear dot product can be interpreted as projection to a certain direction in the space, while nonlinear ones hardly have such an intuitive demonstration.
}

\section*{Question 2}
{
    \subsection*{SVM-based classifiers}
    {
        Thanks to the ``Generate Code'' function of the Classification Learner toolbox in MATLAB, I saved the code of all the 6 SVM kernels in 6 function files: Linear, Quadratic, Cubic, Gaussian Fine ($\sigma=0.1$), Gaussian Medium ($\sigma=0.5$), and Gaussian Coarse ($\sigma=1$). See Table 1 for the results.

        \begin{table}[!hbp]
            \centering
            \begin{tabular}{|c|c|c|}
            \hline
            Method & Testing accuracy & Training speed \\
            \hline
            \hline
            Linear SVM & 0.975389 & 1\textasciitilde 3 mins\\
            \hline
            Quadratic SVM & 0.982923 & 1\textasciitilde 3 mins\\
            \hline
            Cubic SVM & 0.983928 & 1\textasciitilde 3 mins\\
            \hline
            Gaussian Fine SVM & 0.546459 & 1\textasciitilde 3 mins\\
            \hline
            Gaussian Medium SVM & 0.976896 & 1\textasciitilde 3 mins\\
            \hline
            Gaussian Coarse SVM & 0.967855 & 1\textasciitilde 3 mins\\
            \hline
            \hline
            Fully Connected NN & 0.979407 & 30 secs\\
            \hline
            Logistic Regression & 0.952788 & 5\textasciitilde 10 mins\\
            \hline
            \hline
            Naive Bayes Classifier & 0.760924 & 5 secs\\
            \hline
            \end{tabular}
            \caption{Accuracy on testing set}
        \end{table}
    }

    \subsection*{FCNN-based classifiers}
    {
        In homework 4, we tried different sizes of the hidden layer, ranging from $5$, $10$, $20$, $40$, to $100$. Here is a copy of our conclusion in homework 4:

        \begin{itemize}
            \item \textbf{Confusion matrix:}\quad With more hidden nodes, the confusion rate drops accordingly. See the table below.
            \begin{table}[htb!]
                \centering
                \begin{tabular}{|c|c|}
                \hline
                Hidden nodes & Total confusion rate \\
                \hline
                5 & 11.5\% \\
                \hline
                10 & 8.9\% \\
                \hline
                20 & 5.6\% \\
                \hline
                40 & 2.9\% \\
                \hline
                100 & 1.9\% \\
                \hline
                \end{tabular}
                \caption{Confusion rates}
            \end{table}

            \item \textbf{Performance:}\quad With more hidden nodes, the error rates drop at both validation set and testing set. Meanwhile, it takes a longer time before it stops and converges, in spite of fewer iterations. Another interpretation: The network has better capability of fitting the pattern with more hidden nodes, and therefore the training process is smoother and we need fewer iterations in training. But it takes more time since we have much more parameters to train, which leads to a longer time for each iteration step.
            \begin{table}[htb!]
                \centering
                \begin{tabular}{|c|c|}
                \hline
                Hidden nodes & Total training epochs \\
                \hline
                5 & 500 \\
                \hline
                10 & 185 \\
                \hline
                20 & 125 \\
                \hline
                40 & 125 \\
                \hline
                100 & 110 \\
                \hline
                \end{tabular}
                \caption{Training iterations}
            \end{table}

            \item \textbf{ROC curve:}\quad With more hidden nodes, the ROC curve is closer to the left top corner, which indicates a better performance of classification.
        \end{itemize}

        It is obvious that a size of $100$ hidden nodes makes the best performance. The accuracy on the testing set is attached to Table 1.
    }

    \subsection*{Logistic-regression-based classifiers}
    {
        We use the logistic regression classifier in the Classification Learner toolbox. The accuracy on the testing set is attached to Table 1.
    }

    \subsection*{Naive Bayes classifiers}
    {
        There is no toolbox provided, so we manually call ``fitcnb'' function in MATLAB to train a naive Bayes model. The accuracy on the testing set is attached to Table 1. The training speed is ultrafast, only seconds.
    }

    \subsection*{Comparison}
    {
        We obtain the testing accuracy, as well as the training speed, in Table 1.

        \begin{itemize}
            \item \textbf{SVM:}\quad The SVM-based classifier with a cubic kernel function achieves the highest accuracy on the testing set. The SVM with a fine Gaussian kernel has been overfitting the data, since $\sigma=0.1$ is too small and it makes the exponential term decay too fast. However, it takes minutes to train the model, which is not so competitive as the FCNN in terms of training speed.

            \item \textbf{FCNN and LR:}\quad Fully connected neural network with a hidden layer of 100 nodes also attains a high accuracy, with a training process faster than the SVMs above. It overwhelms the logistic regression model, which can be interpreted as a FCNN with no hidden layer. Here we see the crutial importance of the hidden layer. The additional layer provides a high capacity when fitting nonlinear function.

            \item \textbf{Naive Bayes:}\quad The accuracy is not satisfactory, but it is super fast in training, much faster than other methods. I suppose that possible ways to improve this include feature selection and PCA. It also helps if we use advanced Bayes classifiers in place of the Naive Bayes classifier.
        \end{itemize}

    }
}

\section*{Source Code}
{
    Please download the souece code from http://39.106.23.58/files/PR5\_2015011506.7z

    For Question 2, please run ``main.m''. It may take minutes to train the network, but the result is reproducible because of the random seed.

    For each model, I clicked ``Generate code'' button to transcript my operations into MATLAB codes, and stored each of them in the corresponding ``.m'' file. These files include:
    \begin{itemize}
        \item trainClassifierCubic.m
        \item trainClassifierLinear.m
        \item trainClassifierQuadratic.m
        \item trainClassifierGaussianFine.m
        \item trainClassifierGaussianMedium.m
        \item trainClassifierGaussianCoarse.m
        \item trainClassifierFullyConnected.m
        \item trainClassifierLogistic.m
    \end{itemize}
    Thus, the steps above can be easily reproduced without using the GUI of the toolbox.

}

\clearpage
\end{document}