\documentclass[a4paper]{article}

\def\npart{IV}

\def\ntitle{Geometric Aspects of p-adic Hodge Theory}
\def\nlecturer{T.\ Csige}

\def\nterm{Michaelmas}
\def\nyear{2020}

\input{header}

\newcommand{\tilt}{\flat} % tilting
\newcommand{\perf}{\mathrm{perf}}
%\DeclareMathOperator{\perf}{perf} % perfection
\renewcommand{\c}[1]{\mathbf{#1}}
\newcommand{\Mod}{{\c{Mod}}}
\DeclareMathOperator{\Tor}{Tor} % torsion
\DeclareMathOperator{\Ext}{Ext} % extension
\newcommand{\sh}[1]{\mathcal{#1}} % sheaf
\DeclareMathOperator{\Spa}{Spa}
\renewcommand*{\O}{\mathcal{O}}

\newtheorem*{construction}{Construction}

\iffalse
\renewcommand*{\P}{\mathbb{P}}
\newcommand{\sh}[1]{\mathcal{#1}} % sheaf
\renewcommand*{\O}{\mathcal{O}}
\let\Sp\Relax
\DeclareMathOperator{\Sp}{Sp} % maximum spectrum
\DeclareMathOperator{\Max}{Max}
\DeclareMathOperator{\Spf}{Spf}
\DeclareMathOperator{\Spa}{Spa}
\DeclareMathOperator{\supp}{supp} % support of a valuation
\fi

\begin{document}

\input{titlepage}

\tableofcontents

\section{Introduction}

Course structure:
\begin{enumerate}
\item introduction
\item Hodge-Tate decomposition for abelian varieties with good reduction
\item Hodge-Tate decomposition in generale (pro-étale cohomology)
\item integral aspects
\item some additional topics (Hodge-Tate decomposition theorem for rigid analytic varieties)
\end{enumerate}

\subsection{Hodge decomposition over \(\C\)}

Let \(X\) be a smooth projective variety over \(\C\). The \emph{Hodge decomposition} is a direct sum decomposition for all \(n \geq 0\)
\[
  H_{\mathrm{sing}}^n(X^{\mathrm{an}}, \C) = \bigoplus_{p + q = n} H^{p, q}
\]
where LHS is the singular cohomology of the \(\C\)-analytic manifold \(X^{\mathrm{an}}\) (the complex analytification) and on RHS
\[
  H^{p, q} = H^q(X^{\mathrm{an}}, \Omega_{X^{\mathrm{an}}}^p)
\]
with \(\Omega_{X^{\mathrm{an}}}^p\) denoting the sheaf of holomorphic \(p\)-forms. Moreover, complex conjugation acts on
\[
  H^n_{\mathrm{sing}}(X^{\mathrm{an}}, \C) \cong H_{\mathrm{sing}}^n(X^{\mathrm{an}}, \Q) \otimes \C
\]
via its action on \(\C\) and \(H^{p, q} = \overline{H^{q, p}}\). This is called a \emph{pure structure of weight \(n\)}.

These are proven via identifying \(H^{p, q}\) with Dolbeault cohomology and using the (very deep) theory of harmonic forms. However, part of the theory can be understood purely algebraically. It is known that \(H^n_{\mathrm{sing}}(X^{\mathrm{an}}, \C)\) gives the cohomology of the constant sheaf \(\C\) on \(X^{\mathrm{an}}\). On the other hand, consider the de Rham complex
\[
  \Omega_{X^{\mathrm{an}}}^\bullet = \O_{X^{\mathrm{an}}} \xrightarrow{\d} \Omega_{X^{\mathrm{an}}}^1 \xrightarrow{\d} \Omega_{X^{\mathrm{an}}}^2 \to \cdots
\]
Here \(\d\) is the usual derivation and the higher \(\d\)'s are given by
\[
  \d (\omega_1 \wedge \omega_2) = \d \omega_1 \wedge\omega_2 + (-1)^p \omega_1 \wedge\d \omega_2
\]
for \(\omega_1 \in \Omega_{X^{\mathrm{an}}}^p, \omega_2 \in \Omega_{X^{\mathrm{an}}}^q\). Taking hypercohomology
\[
  H^n_{\mathrm{dR}}(X^{\mathrm{an}}) := \H(X^{\mathrm{an}}, \Omega_{X^{\mathrm{an}}}^\bullet)
\]
we get the so-called de Rham cohomology group.

Embedding the constant sheaf \(\C\) into \(\O_{X^{\mathrm{an}}}\) induces a map \(\C \to \Omega_{X^{\mathrm{an}}}^\bullet\) of complexes of sheaves. The (holomorphic) Poincaré lemma states that this map is a quasi-isomorphism of sheaves. More precisely, one can cover \(X^{\mathrm{an}}\) by open balls and for any open ball \(U \subseteq X^{\mathrm{an}}\), the complex
\[
    0 \to \C \to \O_{X^{\mathrm{an}}}(U) \xrightarrow{\d} \Omega^1_{X^{\mathrm{an}}} \to \cdots
\]
is exact: any closed differential form can be integrated on an open ball. Thus
\[
  H^n_{\mathrm{sing}}(X^{\mathrm{an}}, \C) \cong H^n_{\mathrm{dR}}(X^{\mathrm{an}}).
\]
This is the comparison theorem between singular and de Rham cohomology.

Now the complex \(\Omega_{X^{\mathrm{an}}}^\bullet\) has a decreasing filtration of subcomplexes
\[
  \Omega_{X^{\mathrm{an}}}^{\geq p} := 0 \to \cdots \to 0 \to \Omega^p_{X^{\mathrm{an}}} \xrightarrow{\d} \Omega_{X^{\mathrm{an}}}^{p + 1} \xrightarrow{\d} \cdots
\]
We have that \(\operatorname{gr}^p \Omega^\bullet_{X^{\mathrm{an}}} \cong \Omega^p_{X^{\mathrm{an}}}\). It is well-known that there is a convergent spectral sequence associated to \(\Omega_{X^{\mathrm{an}}}^\bullet\) with the filtration above, called the \emph{Hodge to de Rham spectral sequence}
\[
  E_1^{pq} = H^q(X^{\mathrm{an}}, \Omega^p_{X^{\mathrm{an}}}) \Rightarrow H^{p + q}_{\mathrm{dR}}(X^{\mathrm{an}}).
\]
The filtration on \(H^n_{\mathrm{dR}}(X^{\mathrm{an}})\) given by the spectral sequence is called the \emph{Hodge filtration}.

Fact: the Hodge to de Rham spectral sequence degenerates at \(E_1\). This together with the comparison theorem gives the Hodge decomposition
\[
  H^n_{\mathrm{sing}}(X^{\mathrm{an}}, \C) = \bigoplus_{p + q = n} H^q(X^{\mathrm{an}}, \Omega_{X^{\mathrm{an}}}^p)
\]
for all \(n\).

\subsection{Algebraisation}

On a complex variety \(X\) we may consider the algebraic de Rham complex
\[
  \Omega_X^\bullet := \O_X \xrightarrow{\d} \Omega_X^1 \xrightarrow{\d} \cdots
\]
For \(X\) smooth these are locally free sheaves. The same way as above, we get the algebraic Hodge to de Rham spectral sequence
\[
  E_1^{pq} = H^q(X, \Omega_X^p) \Rightarrow H^{p + q}_{\mathrm{dR}}(X).
\]
Here we use the Zariski topology.

There are two natural maps
\begin{align*}
  H^q(X, \Omega_X^p) &\to H^q(X^{\mathrm{an}}, \Omega_{X^{\mathrm{an}}}^p) \\
  H^{p +q}_{\mathrm{dR}}(X) &\to H^{p +q}_{\mathrm{dR}}(X^{\mathrm{an}})
\end{align*}
all compatible with the maps in the above spectral sequences. By GAGA the first one is an isomorphism, and by a theorem of Grothendieck the second is also an isomorphism. Hence degeneration of the analytic Hodge to de Rham is equivalent to the degeneration of the algebraic counterpart.

However, there is no algebraic Poincaré lemma, the algebraic de Rham complex is not a resolution of \(\C\) and anyway the sheaf cohomology of \(\C\) is trivial in the Zariski topology.

\subsection{The case of a \(p\)-adic base field}

Let \(p\) be a prime. Recall \(\C_p\) is the completion of the algebraic closure \(\overline Q_p\) of \(\Q_p\). The Galois group \(\gal(\overline \Q_p/\Q_p)\) acts on \(\C_p\) by continuity. Let \(K\) be a finite extension of \(\Q_p\).

\printindex
\end{document}


% !TeX spellcheck = de_DE

\chapter{Conclusion and Future Work}
\label{chap:conclusion}

This thesis proposed a framework for automatic 3D lane marking reconstruction using multi-view aerial imagery. Standard line detection algorithms are applied to extract lane markings on the aerial images. By exploiting the use of linear regression in image space and with the combination of collinearity condition, lane markings are reconstructed based on their detected positions in images and the viewing geometry. Without the utilization of the neighboring textures, the approach requires initial approximate 3D lines and is highly dependent on quality of pre-known image orientations. Nevertheless, it is robust to partial occlusions of the targeted lines on images and is applicable to the (quasi-)infinite line features as well as in cases of lowly textured neighboring and it improves the DSM at lowly textured road surfaces.

From simulated experiments in \cref{sec:simulation}, some conclusions are given:
\begin{itemize}
	\item Given approximations with 2 meters bias in Z-direction and sub-pixel random noises in the image coordinates of the detected/measured 2D lines, the proposed approach correctly refined the 3D positions of line segments.

	\item With the same line orientations in 3D space, the configuration strength increases with the increase of amount of covering images from different views.

	\item The reconstructed line segments have higher priori precision in horizontal direction than in vertical direction.
\end{itemize}

From experimental results of true data in \cref{sec:truedata}, some conclusions are given:
\begin{itemize}
	\item The DSM profile, i.e. the initial 3D line approximation, is significantly and systematically dozens of centimeters far away from the reconstructed line segments. This indicates the necessity of reconstrucion based on detected 2D lines in image space and the viewing geometry, instead of simply reducing DSM noise by applying mean filter or such.

	\item The LS-estimated precision of the measurements, involving the lane marking extraction quality as well as the quality of image orientation parameters, is better than 1 pixel.

	\item The configuration defect %which is mentioned in \cref{subsec:LSadj}
	barely happens: 1.when the linear regression functional model between image coordinates $x$ and $y$ are properly set up ---according to the characteristics of the detected lane-lines in image space. 2.when the flight configuration guarantees stereo views whose base-lines are perpendicular to lane-lines orientations in 3D.

	\item The theoretical precision of the reconstructed line-nodes is within 2.5 cm in vertical direction and within 5 mm in horizontal direction.
%	\item  %The configuration defect in object space %which is mentioned in \cref{subsec:LSadj}
%	%may happen in some of the stereo views in the same strip with the adopted flight configuration in this work.% of flying along the motorways, the detected lane lines lie mainly in flight direction on images.
%	With the adopted flight configuration in this work, the stereo pairs in the same strip, where the lane markings lie nearly on the epipolar plane, barely contribute on increasing the configuration strength for lane marking reconstruction.
%	%This indirectly indicates that, under this kind of image configuration, increasing forward overlapping rate does not really increase the configuration strength for 3D lane markings reconstruction.

\end{itemize}

%using precisely detected lines with sub-pixel precision as well the bundle adjusted image orientations, the lane markings can be reconstructed with ???centimeter accuracy??? using well configured aerial images.

%With the extracted line in the form of sets of points with sub-pixel accuracy and EOIO accuracy???, linear regression can be applied on reconstructing the 3D line segment position.

%image distortion vs straight line approximation

Some general conclusions are:
\begin{itemize}
	\item Image configuration plays an important roll in 3D reconstruction. In the case of this work, being covered by more views whose base-lines are as perpendicular as possible to lane-marking directions, would improve the reconstruction result.

	\item Lane markings can be exploited to provide information on refining SGM-generated DSM.% Reconstruction based on imaging geometry is necessary.

	\item The proposed reconstruction workflow relies on initial approximation with enough accuracy, i.e. a global terrain model like SRTM would not be sufficient as starting height, as the error could be several meters in particular at roads.
\end{itemize}

\section*{Future Work}
\label{chap:futurework}

%Future improvements could be achieved by:

%having crab angles of $\approx45\degree$ in flight.
%Theoretically this kind of configuration defect may be solved.

%increasing the oblique angle
The proposed approach addresses 3D line features reconstruction problems without resorting to their appearances in images. Similar cases are railways. Appling this framwork to such cases may be done in the future. Additionally, it worths trying the framework with images taken from drones instead of from helicopters.


\section{Maximising the likelihood function}


% !TEX root = scombinatorics.tex
\documentclass[scombinatorics.tex]{subfiles}
\begin{document}
\chapter*{References}
\addcontentsline{toc}{chapter}{References}

\begin{biblist}[]\normalsize

  \bib{AK1}{article}{
    author={Alon, N.},
    author={Kleitman, D. J.},
    title={A purely combinatorial proof of the Hadwiger Debrunner $(p,q)$
    conjecture},
    % note={The Wilf Festschrift (Philadelphia, PA, 1996)},
    journal={Electron. J. Combin.},
    volume={4},
    date={1997},
    % number={2},
    % pages={Research Paper 1, approx. 8},
    % review={\MR{1444148}},
 }

 \bib{AK2}{article}{
    author={Alon, Noga},
    author={Kleitman, Daniel J.},
    title={Piercing convex sets and the Hadwiger-Debrunner $(p,q)$-problem},
    journal={Adv. Math.},
    volume={96},
    date={1992},
    % number={1},
    % pages={103--112},
    % issn={0001-8708},
    % review={\MR{1185788}},
    % doi={10.1016/0001-8708(92)90052-M},
 }

\bib{ARS}{article}{
   author={Anstee, R. P.},
   author={R\'onyai, Lajos},
   author={Sali, Attila},
   title={Shattering news},
   journal={Graphs Combin.},
   volume={18},
   date={2002},
%  number={1},
%  pages={59--73},
%  issn={0911-0119},
%  doi={10.1007/s003730200003},
}
\bib{DGL}{book}{
   author={Devroye, Luc},
   author={Gy\"{o}rfi, L\'{a}szl\'{o}},
   author={Lugosi, G\'{a}bor},
   title={A probabilistic theory of pattern recognition},
   publisher={Springer-Verlag},
   date={1996},
}
\bib{DL}{book}{
   author={Devroye, Luc},
   author={Lugosi, G\'{a}bor},
   title={Combinatorial methods in density estimation},
   series={Springer Series in Statistics},
   publisher={Springer-Verlag},
   date={2001},
}
\bib{gowers}{article}{
   author={Gowers, Timothy},
   title={\href{https://gowers.wordpress.com/2008/07/31/dimension-arguments-in-combinatorics}{Dimension arguments in combinatorics}},
   journal={Gowers's Weblog},
   date={2008}
}
\bib{hodges}{article}{
  author={Hodges, Wilfrid},
  title={Encoding orders and trees in binary relations},
  journal={Mathematika},
  volume={28},
  date={1981},
  % number={1},
  pages={67--71},
  % issn={0025-5793},
  % review={\MR{632796}},
  % doi={10.1112/S0025579300015357},
}
\bib{kalai}{article}{
  author={Kalai, Gil} ,
  title={\href{https://gilkalai.wordpress.com/2008/09/28/extremal-combinatorics-iii-some-basic-theorems}{Extremal Combinatorics III: Some Basic Theorems}},
  journal={Combinatorics and more},
  date={2008}
}
\bib{matousek}{article}{
   author={Matou\v{s}ek, Ji\v{r}\'{\i}},
   title={Bounded VC-dimension implies a fractional Helly theorem},
   journal={Discrete Comput. Geom.},
   volume={31},
   date={2004},
  %  number={2},
  %  pages={251--255},
  %  issn={0179-5376},
  %  review={\MR{2060639}},
  %  doi={10.1007/s00454-003-2859-z},
}
\bib{LPmatousek}{book}{
  author={Matousek, J\v{\i}ri},
  author={Gartner, Bernd},
  title={Understanding and Using Linear Programming},
  publisher={Springer-Verlag},
  date={2007},
  pages={vi+222},
}
\bib{pajor}{book}{
   author={Pajor, Alain},
   title={Sous-espaces $l^n_1$ des espaces de Banach},
%   language={French},
   series={Travaux en Cours [Works in Progress]},
   volume={16},
%   note={With an introduction by Gilles Pisier},
   publisher={Hermann, Paris},
   date={1985},
%   pages={xii+112},
%   isbn={2-7056-6021-6}
}
\bib{sauer}{article}{
   author={Sauer, N.},
   title={On the density of families of sets},
   journal={J. Combinatorial Theory Ser. A},
   volume={13},
   date={1972},
   pages={145--147},
}
\bib{shelah72}{article}{
   author={Shelah, Saharon},
   title={A combinatorial problem; stability and order for models and theories in infinitary languages},
   journal={Pacific J. Math.},
   volume={41},
   date={1972},
   pages={247--261},
}
\bib{VC}{article}{
   author={Vapnik, V. N.},
   author={Chervonenkis, A. Ya.},
   title={On the uniform convergence of relative frequencies of events to their probabilities},
   note={Reprint of Theor. Probability Appl. {\bf 16} (1971), 264--280},
   conference={title={Measures of complexity},},
   book={publisher={Springer, Cham},},
   date={2015},
   pages={11--30},
}
\end{biblist}
\end{document}


\documentclass[11pt]{article}

\input{preamble}

%\usepackage{tikz}
%\usepackage{tikz-qtree}
\usepackage{qtree}
\usepackage{xfrac}

\title{Week 4: Naive Bayes, Entropy and Backpropagation}
\author{\url{http://mlvu.github.io}}

\begin{document}

\maketitle

\section{preliminaries: probability}

Before we start, it's helpful to review your general grasp of probability. If the exercises below give you any trouble we recommend that you follow the links provided to brush up a little.

\qu
\begin{enumerate}
\item In a few sentences, explain the difference between the Frequentist and the Bayesian interpretation of probability. \ans{A frequentist considers a probability an objective value: a property of the universe that can be measured by repeated experimentation, like measuring the probability that a bent coin lands heads, by flipping it repeatedly. A Bayesian considers probability an expression of uncertain belief. A Bayesian can talk about the probability that ``John is having an affair''. To a frequentist this is nonsense, because John is either having an affair or he isn't.}{}
\item What is the difference between a sample space and an event space? \ans{A sample space is a space of individual things that can happen (like a die landing on a 6) and an event space is a space of sets of things from the sample space (like a die landing on an even number). Technically, an event space should be a sigma-algebra of a sample space, but for discrete probability distributions, we can think of the event space as a the powerset of the sample space. For continuous probability distributions, we can ignore the technical details.}{}
\item What is the difference between a probability distribution and a probability density function? \ans{A probability function assigns probabilities to events. If the sample space is continuous (like in the case of a normal distribution) all atomic events like ``X is exactly 2.1" will have zero probability. Instead, we define a \emph{probability density function}. We then get the probability of a subset of the sample space, like ``X is between 1.95 and 2.15'', by integrating over the probability density function.}{ }
\end{enumerate}

\qu In the following, $p$ is a  probability function and $A$ and $B$ are random variables. Which of the following are true?
\begin{enumerate}
	\item Joint probability is symmetric: $p(A, B) = p(B, A)$. \ans{True}{}
	\item Conditional probability is symmetric $p(A \mid B) = p(B \mid A)$. \ans{False. This is not true in general (although it may be true for specific $A$ and $B$).}{}
	\item Two random variables $X$ and $Y$ are conditionally independent on a third $Z$. Once we know $X$ and $Y$, we also know the value of $Z$. \ans{False.}{}
	\item Two random variables $X$ and $Y$ are conditionally independent on a third $Z$. Once we know $Z$, also knowing $X$ will tell us nothing extra about $Y$. \ans{True: conditional independence means that given the value of the conditional, $X$ and $Y$ are independent (so knowing the value of one reveals nothing about the other).}{}
\end{enumerate}

\qu Assume that the probability that a given patient has diabetes is $0.01$. We have a test for diabetes with a false positive rate of $0.05$: if a patient has no diabetes, the test diagnoses it 5\% of the time. The false negative rate is $0.1$.
You are a doctor, and you administer the test to a patient (knowing nothing else). The test says she has diabetes. What is the probability that she doesn't? Hint: this is a question about Bayes' rule.
Reflect on the result. Is this what you would've expected? If not, where does the unexpected result come from?

\ans{Let's write down the given probabilities first. We'll use $D$ and $\neg D$ for the events that the patient has and doesn't have diabetes respectively. We'll use $T$ for the event of the test \emph{saying} that the patient has diabetes and $\neg T$ for the test saying that she doesn't.
We are given $p(D) = 0.01$, $\oc{p(T \mid \neg D)} = 0.05$ and $p(\neg T \mid D) = 0.1$. We can derive $p(\neg T\mid \neg D) = 0.95$ and $p(T \mid D) = 0.9$.
Now, we want to know $p(\neg D \mid T)$. We'll use Bayes' rule to reverse the conditional probability:
\begin{align*}
p(\neg D \mid T) &= \frac{\oc{p(T \mid \neg D)} \bc{p(\neg D)} }{p(T)} \\
&= \frac{\oc{p(T \mid \neg D)} \bc{p(\neg D)}}{p(T, D) + p(T, \neg D)} \\
&= \frac{\oc{p(T \mid \neg D)} \bc{p(\neg D)}}{p(T \mid D)p(D) + p(T \mid \neg D)\bc{p(\neg D)}} \\
&= \frac{\oc{0.05} \times \bc{0.99}}{0.9 \times 0.01 + \oc{0.05} \times \bc{0.99}} \\
&= \frac{\oc{5} \times \bc{99}}{90 + \oc{5}\times \bc{99}} \approx 0.85
\end{align*}
Whether you expected this result is up to you, of course, but it's certainly not what you want form a test like this. And yet, the false positive and false negative rates don't seem extremely high. Inn the last line, I've multiplied everything by 100 so you can see where the imbalance comes from: \bc{the high prior probability that someone doesn't have diabetes} dominates (in other words, we have high lass imbalance).
In order for the test to be reliable, the \oc{false positive rate} needs to be low enough to make the numerator small in proportion to the denominator.\footnotemark

\footnotetext{There's a subtle difference between the phrase \emph{false positive rate} as we use it here and as we've used it previously. What we described previously (like in lecture 3) is an estimate of the fpr, computed from the test set. What we discuss here is the ``actual'' fpr: an unknown quantity that we would normally only be able to estimate from data.}
}{}

More practice? Follow these links:
\begin{enumerate}
\item \url{https://seeing-theory.brown.edu/} (especially this one)
\item \url{https://betterexplained.com/articles/a-brief-introduction-to-probability-statistics/}
\item \url{https://www.khanacademy.org/math/probability/probability-geometry/probability-basics/a/probability-the-basics}
\item \url{http://dept.stat.lsa.umich.edu/~moulib/probrefresh.pdf}
\end{enumerate}

\section{Naive Bayes}

The following dataset represents a spam classification problem: we observe 8 emails and measure two binary features. The first is 0 if the word "pill" occurs in the e-mail and the second is 1 if the word ``meeting'' occurs.

\begin{center}
	\begin{tabular}{c c c c}
		 ``pill'' &``meeting'' & label\\
		\hline
		  \bc{T} & F & \rc{Spam} \\
		  \bc{T} & F & \rc{Spam} \\
		  F & \bc{T} & \rc{Spam} \\
		  \bc{T} & F & \rc{Spam} \\
		  F & F & \gc{Ham} \\
		  F & F & \gc{Ham} \\
		  F & \bc{T} & \gc{Ham} \\
		  \bc{T} & \bc{T} & \gc{Ham} \\
		\hline
	\end{tabular}
\end{center}

\qu We will build a naive Bayes classifier for this data. What is the defining property of naive Bayes? Why is it called ``naive''?

\ans{The naive Bayes classifier assumes that the features are independent, \emph{conditional on the class}. It is called naive, because this assumption is usually untrue. The resulting classifier nevertheless works well in many cases.}{}

\qu We build a naive Bayes classifier on this data, as described in the lecture. We get an email that contains both words. Which class does the classifier assign?

\ans{
Call the class $Y$ and the features $X_p$ and $X_m$. The class probabilities are
\[
p(Y\mid X_p, X_m) = \frac{p(X_p, X_m\mid Y)p(Y)}{p(X_p, X_m)} \p
\]
The class choice is
\[
\argmax_Y p(Y\mid X_p, X_m) = \argmax_Y p(X_p, X_m\mid Y)p(Y) \p
\]

The \emph{naive Bayes assumption} allows us to break the the probabilities up.

\[
\argmax_Y p(Y\mid X_p, X_m) = \argmax_Y p(X_p\mid Y) p(X_m\mid Y)p(Y) \p
\]

We estimate the the class prior $P(Y)$ from the data (i.e. 0.5 for each class). We estimate the likelihood of the data given the class based on the relative frequencies in the data (i.e. $p(X_p\mid \text{\rc{Spam}}) = \frac{3}{4}$). Since we only care about the class with the maximal probability, we can use Bayes rule without computing the denominator.

This gives us:
\begin{align*}
p(\text{Spam}\mid X_p=1, X_m=1) &\propto p(X_p=1\mid \text{Spam})\;p(X_m=1\mid \text{\rc{Spam}})\;p(\text{Spam})\\
					 &= \frac{3}{4}\frac{1}{4}\frac{1}{2} = \frac{3}{32}\\
p(\text{Ham}\mid X_p=1, X_m=1) &\propto p(X_p=1\mid \text{\gc{Ham}})\;p(X_m=1\mid \text{\gc{Ham}})\;p(\text{\gc{Ham}})\\
					 &= \frac{1}{4}\frac{2}{4}\frac{1}{2} = \frac{2}{32}\\
\end{align*}
\rc{Spam} gives a higher value (these are not probabilities, just values proportional to the true probabilities), so the classifier classifies the email as \rc{Spam}.
}{}

\qu Which probabilities does the classifier assign to each class?
\ans{
For full class probabilities, we need to apply Bayes' theorem including its denominator:
\[
p(Y\mid X_p, X_m) = \frac{p(X_p, X_m\mid Y)p(Y)}{p(X_p, X_m)} \p
\]
The denominator expands as
\[
p(X_p, X_m) = \sum_{Y\in \{\text{Spam}, \text{Ham}\}} p(X_p, X_m\mid Y)p(Y)
\]

These are the two unnormalized probabilities ($\frac{3}{32}$ and $\frac{2}{32}$) that we've calculated already. Computing the proper probabilities is equivalent to normalizing the unnormalized ones. Thus, the class probabilities assigned are $\frac{3}{5}$ for \rc{Spam} and $\frac{2}{5}$ for \gc{Ham}.
}{}

To improve our accuracy, we add another feature:
\begin{center}
	\begin{tabular}{c c c c}
		``pill'' & ``meeting'' & ``hello'' & label\\
		\hline
		  \bc{T} & F & F & \rc{Spam} \\
		  \bc{T} & F & F & \rc{Spam} \\
		  F & \bc{T} & F & \rc{Spam} \\
		  \bc{T} & F & F & \rc{Spam} \\
		  F & F & F & \gc{Ham} \\
		  F & F & F & \gc{Ham} \\
		  F & \bc{T} & F & \gc{Ham} \\
		  \bc{T} & \bc{T} & \bc{T} & \gc{Ham} \\
		\hline
	\end{tabular}
\end{center}

\qu For the class Spam, there are no emails recorded that contain the word ``hello''. Why is this a problem?

\ans{This makes our estimate of the probability $p(X_h=1\mid \text{\rc{Spam}})$ zero. This causes the entire class probability to collapse to 0 even if the other features make Spam very likely.}

What solution is suggested in the slides?

\ans{To add \emph{pseudo-observations}, so that for each feature each value is observed at least once for each class.}

\qu Implement this solution, and give the class probabilities for an email containing all three words.

\ans{
After adding the pseudo observations, our dataset looks like this:
\begin{center}
	\begin{tabular}{c c c c}
		 ``pill'' & ``meeting'' & ``hello'' & label\\
		\hline
		  \bc{T} & F & F & \rc{Spam} \\
		  \bc{T} & F & F & \rc{Spam} \\
		  F & \bc{T} & F & \rc{Spam} \\
		  \bc{T} & F & F & \rc{Spam} \\
		  F & F & F & \gc{Ham} \\
		  F & F & F & \gc{Ham} \\
		  F & \bc{T} & F & \gc{Ham} \\
		  \bc{T} & \bc{T} & \bc{T} & \gc{Ham} \\
		  \hline
		  \bc{T} & \bc{T} & \bc{T} & \rc{Spam} \\
		  F & F & F & \rc{Spam} \\
		  \bc{T} & \bc{T} & \bc{T} & \gc{Ham} \\
		  F & F & F & \gc{Ham} \\
		\hline
	\end{tabular}
\end{center}

Note that we don't add every possible combination of features (that would be 8 extra instances per class). We just make sure that for every feature there is one email that has F for that feature and \rc{Spam} as a class, one email that has \bc{T} for that feature and \rc{Spam} as a class and the same for \gc{Ham}.

This gives us:
\begin{align*}
p(&\text{\rc{Spam}}\mid X_p=1, X_m=1, X_h=1) \\
&\propto p(X_p=1\mid \text{\rc{Spam}})\;p(X_m=1\mid \text{\rc{Spam}})\;p(X_h=1\mid \text{\rc{Spam}})\;p(\text{\rc{Spam}})\\
					 &= \frac{4}{6}\frac{2}{6}\frac{1}{6}\frac{1}{2} = \frac{8}{6^32}\\
p(&\text{\gc{Ham}}\mid X_p=1, X_m=1, X_h=1) \\
&\propto p(X_p=1\mid \text{\gc{Ham}})\;p(X_m=1\mid \text{\gc{Ham}})\;p(X_h=1\mid \text{\gc{Ham}})\;p(\text{\gc{Ham}})\\
					 &= \frac{2}{6}\frac{3}{6}\frac{2}{6}\frac{1}{2} = \frac{12}{6^32}
\end{align*}

Normalizing these gives us $\frac{2}{5}$ for \rc{Spam} and $\frac{3}{5}$ for \gc{Ham}.
}{}

\section{Entropy}

We define two probability distributions $p$ and $q$ on a set of four outcomes $\{a, b, c, d\}$.

\begin{center}
	\begin{tabular}{c c c c}
		 $p(a)$ & $p(b)$ & $p(c)$ & $p(d)$\\
		\hline
		  $\sfrac{1}{4}$ & $\sfrac{1}{4}$ & $\sfrac{1}{4}$ & $\sfrac{1}{4}$ \\
	\end{tabular}
\end{center}

\begin{center}
	\begin{tabular}{c c c c}
		 $q(a)$ & $q(b)$ & $q(c)$ & $q(d)$\\
		\hline
		  $\sfrac{1}{2}$ & $\sfrac{1}{4}$ & $\sfrac{1}{8}$ & $\sfrac{1}{8}$ \\
	\end{tabular}
\end{center}

\qu Could you simulate sampling from these distributions using coinflips as described in the lecture?

\ans{Yes. $p$ can be simulated by assigning each unique sequence of two coinflips to one of the outcomes. For $q$, we can assign the following sequences to each outcome:

\begin{center}
	\begin{tabular}{c c c c}
		 $q(a)$ & $q(b)$ & $q(c)$ & $q(d)$\\
		\hline
		  $H$ & $TH$ & $TTH$ & $TTT$ \\
	\end{tabular}
\end{center}
}

\qu Compute the entropy of $p$ and $q$.

\ans{
\begin{align*}
H(p) &= - \sum_{y \in \{a, b, c, d\}} p(y) \log_2 p(y)  \\
&= - 4\left(\frac{1}{4} \log \frac{1}{4}\right) = - \log 4\times -1 = \log 4 = 2\
\end{align*}

\begin{align*}
H(q) &= - \sum_{y \in \{a, b, c, d\}} q(y) \log_2 q(y)  \\
&= - \frac{1}{2} \log \frac{1}{2} -  \frac{1}{4} \log \frac{1}{4} - \frac{1}{8} \log \frac{1}{8} - \frac{1}{8} \log \frac{1}{8}\\
&= \frac{1}{2} + \frac{2}{4} + \frac{3}{8} + \frac{3}{8} = \frac{4}{8} + \frac{4}{8} + \frac{3}{8} + \frac{3}{8} = 1.75
\end{align*}
}

The entropy of $q$ is lower than the entropy of $p$. What does this tell you about the difference between the two distributions?

\ans{This means that $p$ is more \emph{uniform} than $q$. In other words, we have more \emph{information} about what the outcome of a sample from $q$ will be than we do about the outcome of a sample from $p$.}

\qu In the definition of entropy that we use (information entropy), the logarithms have base 2 (i.e. $\log_2$ instead of $\log_{10}$ or $\ln$). This follows directly from our decision to model probability distributions with coinflips. How?

\ans{The logarithm is defined as the expected code length under an idealized optimal binary code. If a binary code assigns a codeword of length of $L$ to an outcome, the probability of sampling that outcome by generating a random code by flipping a coin is $\left (\frac{1}{2}\right)^L = 2^-L$. If we reverse this, to find the optimal code belonging to an event with probability $p(x)$, we get $L = -\log_2p(x)$}{}

\section{Backpropagation}

We will practice the backpropagation algorithm as described in the slides. We will use a neural network defined by the following function:
\begin{align*}
\rc{y} &= \oc{v_1}h_1 + \oc{v_2}h2 \\
\yc{h_1} &= \sigma(\gc{k_1}) \\
\yc{h_2} &= \sigma(\gc{k_2}) \\
\gc{k_1} &= \oc{w_1}x \\
\gc{k_2} &= \oc{w_2}x
\end{align*}

Where $\sigma$ represents the logistic sigmoid. The network has a single input node $x$ and a single output node $\rc{y}$. The weights are $\oc{w_1}$, $\oc{w_2}$, $\oc{v_1}$ and $\oc{v_2}$. We've left out bias nodes to keep things simple.

\qu Draw this network.

{\centering
\includegraphics[width=0.7\linewidth]{{w4.network}.pdf}
}

We will use stochastic gradient descent to train this network. That means we define the loss function for a single instance. We will use basic least-squares loss to train this network for regression. Our loss function for instance $x$ is
\[
\text{loss}_x(\oc{w_1}, \oc{w_2}, \oc{v_1}, \oc{v_2}) = \frac{1}{2}\left(\rc{y}-t\right)^2
\]

where $t$ is the target value provided by the dataset, and $y$ is the output of the network.

We see each line in the definition above as a module, with some inputs and some outputs. Using the chain rule, we will express the derivative with respect to weight $w_1$. We will use only \emph{local derivatives}, expressing the derivative for each module with respect to its inputs, but not working out the derivative beyond that.

\qu Fill in the gaps:
\begin{align}
\frac{\kp \text{loss}}{\kp \oc{w_1}}
 &= \frac{\kp \text{loss}}{\kp \rc{y}}\frac{\kp \rc{y}}{\kp \yc{h_1}} \ans{\frac{\kp \yc{h_1}}{\kp \gc{k_1}} }{\ldots}\frac{\kp \gc{k_1}}{\kp \oc{w_1}} \notag \\
 &= (\rc{y} - t) \times \ans{\oc{v_1}}{\ldots} \times \sigma(\gc{k_1})(1-\sigma(\gc{k_1})) \times \ans{x}{\ldots} \label{line:chain}
\end{align}

\qu  The network has a diamond shape, just as shown in the slide when explaining the multivariate chain rule (in the \emph{Deep Learning 1} lecture). However, in this case, we don't need the multivariate chain rule. Why not? In what kind of situation would the multivariate chain rule be required?

\ans{The multivariate chain rule is needed when the output depends on one of the variables for which we are taking the derivative (like $\oc{w_1}$) along multiple paths in the computation graph. In this case we have a diamond because $\rc{y}$ depends on $x$ along two paths, but we never take the derivative with respect  to $x$ (the input), only with respect to the weights.

In a way, we \emph{are} using the multivariate chain rule, if we write
\[
\frac{\kp l}{\kp \oc{w_1}} = \frac{\kp \oc{v_1}\yc{h_1}}{\kp \oc{w_1}} + \frac{\kp \oc{v_2}\yc{h_2}}{\kp \oc{w_1}}
\]
(because the sum rule is an instance of the multivariate chain rule) but the second term becomes zero because the numerator is constant with respect to $w_1$.

The multivariate chain rule comes into effect when the output depends on a \emph{weight} along multiple paths in the computation graph. This happens, for instance, when we compute the loss function over a batch of multiple data points. }{}

We could take the formulation above and fill in the symbolic expressions for $y$, $k_1$, etc, and come up with a general symbolic formula for the derivative for all inputs. However, that is usually expensive to do for large neural nets. Instead, we leave it as is, and fill in the \emph{numeric} values for these variables for a specific input and for specific weights.

\qu Assume that the input is $x = 1$, with target output $t=\frac{1}{2}$ and that \oc{all weights} are set at $1$. Do a \emph{forward pass}: compute the loss and all intermediate values $\gc{k_1}$, $\gc{k_2}$, $\yc{h_1}$, $\yc{h_2}$ and $\rc{y}$.

To simplify calculation, you can use the approximation $\sigma(1) = \frac{3}{4}$.

\ans{
\begin{align*}
\gc{k_1} &= 1 \\
\gc{k_2} &= 1 \\
\yc{h_1} &= \sfrac{3}{4} \\
\yc{h_2} &= \sfrac{3}{4} \\
\rc{y} &= \sfrac{3}{2} \\
\text{loss} &= \frac{1}{2} \\
\end{align*}
}{}

Now we do the \emph{backward pass}. Fill these intermediate values in to the loss function decomposed by the chain rule  from line \ref{line:chain}, and compute the derivative with respect to $w_1$ for these inputs and weights.

\ans{
\begin{align*}
\frac{\kp \text{loss}}{\kp \oc{w_1}} = 1 \times 1 \times \frac{3}{4}\frac{1}{4} \times 1 = \frac{3}{16}
\end{align*}
}{}

\end{document}


\documentclass[a4paper,12pt]{article}
\usepackage[left=2.5cm,right=2.5cm,top=2.5cm,bottom=2.5cm]{geometry}
\usepackage{color}
\usepackage[usenames,dvipsnames]{xcolor}
\usepackage{amsmath,amssymb,amsthm,algorithm,algorithmic,graphicx,yhmath,url,enumitem,lscape}
\usepackage{wrapfig,subfigure}

\newcounter{problem}
\newenvironment{problem}{\refstepcounter{problem} \noindent {\bf Problem \arabic{problem}}}{\vspace{0.5cm}}
\newenvironment{solution}{\vspace{0.3cm} \par \noindent {\bf Solution}}{}
\newenvironment{verification}{\vspace{0.3cm} \par \noindent {\bf Verification}}{}
\newenvironment{hint}{\vspace{0.3cm} \par {\bf Hint:}}{}

\newcounter{remark}
\newenvironment{remark}{\refstepcounter{remark} \vspace{0.3cm} \par \noindent {\bf Remark \arabic{remark}}}{\vspace{0.3cm}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\N}{\mathbb{N}}
\newcommand{\Rn}{\mathbb{R}^n}
\newcommand{\Rnn}{\mathbb{R}^{n \times n}}
\newcommand{\bes}{\begin{equation*}}
\newcommand{\ees}{\end{equation*}}
\newcommand{\be}{\begin{equation}}
\newcommand{\ee}{\end{equation}}
\newcommand{\eps}{\epsilon}
\newcommand{\fl}{\text{fl}}

\title{Teknisk vetenskabliga ber{\"a}kningar, Fall 2018 \\ Lab Session 9}
\author{Carl Christian Kjelgaard Mikkelsen}

\begin{document}
\maketitle
\tableofcontents
\section{Introduction}

This note contains the list of problems for our lab session
\begin{center}
Wednesday, January 9th, 2019, (kl. 13.00-16.00), Room MA416-426.
\end{center}

\section{The problems}

\begin{problem} Copy the function {\tt forward.m} into {\tt work/MyForward.m}
  \begin{enumerate}
  \item Extend {\tt MyForward} to the point where it also returns the exact flop count in a variable {\tt count}.
  \item Derive and verify a formula for the exact flop count.
  \item Extend {\tt MyForward} to the point where it can solve a lower unit triangular linear system $LX=F$ where $F$ is an $m$ by $n$ matrix.
  \item Construct a minimal working example {\tt work/MyForwardMWE.m} which verifies that {\tt MyForward} is working as advertised. For each column, the script must compute and display the normwise relative error given by
    \bes
    \frac{\| X(.,j) - \hat{X}(:,j) \|_\infty}{\|X(:,j)\|_\infty}
    \ees
    as well as the normwise relative residual given by
     \bes
    \frac{\| F(.,j) - L \hat{X}(:,j) \|_\infty}{\|F(:,j)\|_\infty}
    \ees
  \end{enumerate}
 Naturally, large values of the normwise relative error are unfortunate, but the normwise relative residual should be small for forward substitution.
\end{problem}

\begin{problem} Copy the function {\tt backward.m} into {\tt work/MyBackward.m}
  \begin{enumerate}
  \item Extend {\tt MyBackward} to the point where it also returns the exact flop count in a variable {\tt count}.
  \item Derive and verify a formula for the exact flop count.
  \item Extend {\tt MyBackward} to the point where it can solve a non-singular upper triangular linear system $UX=F$ where $F$ is an $m$ by $n$ matrix.
  \item Construct a minimal working example {\tt work/MyBackward MWE.m} which verifies that {\tt MyBackward} is working as advertised. For each column, the script must compute and display the normwise relative error given by
    \bes
    \frac{\| X(.,j) - \hat{X}(:,j) \|_\infty}{\|X(:,j)\|_\infty}
    \ees
    as well as the normwise relative residual given by
     \bes
    \frac{\| F(.,j) - U \hat{X}(:,j) \|_\infty}{\|F(:,j)\|_\infty}
    \ees
  \end{enumerate}
 Naturally, large values of the normwise relative error are unfortunate, but the normwise relative residual should be small for backward substitution.
\end{problem}

\begin{problem} Copy the function {\tt factor.m} into {\tt work/MyFactor.m}
  \begin{enumerate}
  \item Extend {\tt MyFactor} to the point where it also returns the exact flop count in a variable {\tt count}.
  \item Derive and verify a formula for the exact flop count\footnote{It is possible that this question and answer can be helpful: \url{http://math.stackexchange.com/questions/1640730/order-of-lu-factorisation/1641606#1641606}}.
  \item Develop a minimal working example {\tt MyFactorMWE} which computes an LU factorization $PA=LU$ of a random matrix $A$. The script must compute and display the normwise relative residual given by
    \bes
    \frac{\| PA - LU\|_\infty}{\|A\|_\infty}
    \ees
    In exact arithmetic, this number should be zero. In floating point arithmetic, we can expect a small value, and large values will almost certainly be caused by programming errors.
    \end{enumerate}
    \begin{remark} {\tt MyFactor} will not return the matrix $P$ explicitly. However, the array {\tt sigma} contains the necessary information. Specifically, if $B = PA$, then MATLAB can construct $B$ using {\tt B=A(sigma,:)}.
    \end{remark}
\end{problem}

\begin{problem} Develop a function {\tt work/MyGauss.m} which uses the functions {\tt MyFactor}, {\tt MyForward}, {\tt MyBackward} to solve a linear system $AX=F$ where $A$ is a non-singular $m$ by $m$ matrix and $F$ is an $m$ by $n$. Develop a minimal working example {\tt work/MyGaussMWE.m} which verifies that {\tt MyGauss} is working as advertised. In particular, the script must compute the normwise relative error and the normwise relative residual for each column. Large normwise relative errors are possible, but large normwise relative residuals strongly suggest programming errors.
\end{problem}

\end{document}


\chapter{Multi-layer Kernels in Structured Output Spaces}
\label{chap_struct}

This chapter focuses on evaluating multi-layer arc-cosine kernels on structured output prediction problems. The contents of this chapter are organized as follows; section \ref{chap5_intro} gives a brief description about the large margin formulation of pattern recognition problem on structured output spaces, section \ref{chap5_exp} talks about the result of empirical study on multi-class and multi-label classification problems and section \ref{chap5_conc} concludes the chapter.

\section{SVM in Structured Output Spaces}
\label{chap5_intro}
Typically machine learning algorithms are designed to produce flat real valued outputs; in the case of classification problems the output is a class label, for regression the output is a real number. For structured output learning algorithms, the output space has structured and interdependent variables, which is usually stored and processed as multi-dimensional arrays. For example, in the case of natural language parsing the output is a parse tree, in the case of image segmentation the output is the four 2D cordinates of bounding box surrounding the object. Two popular algorithms which works well in this domain are Conditional Random Fields(CRF\nomenclature{CRF}{Conditional Random Field}) proposed by \cite{crf} et al. and Structural SVMs proposed by \cite{joachims_struct} et al. In this project, we did our study on structural SVMs.

Structural SVMs are first introduced by \cite{joachims_struct} et al. in 2005, and  studied by \cite{joachims_cutting} et al. for simplifying the optimization problem while dealing with exponentially huge number of constraints. In particular, the number of constraints in the formulation of StructSVM is equal to the cardinality of the output space(which is exponential or even infinite). So decomposition methods like SMO\nomenclature{SMO}{Sequential Minimal Optimization} which process each constraints explicitly is not suitable for these kind of problems. \cite{joachims_cutting} et al. proved that the optimization problem can be solved efficiently using cutting plane algorithm proposed by \cite{cutting_plane}. They also proved that the number of iterations are independent of the number of training examples and provided an upper bound on the number of iterations.

\subsection{StructSVM : Formulation}
Structured output prediction describes the problem of learning a function
\[ h: \mathcal{X} \longrightarrow \mathcal{Y} \]
where $\mathcal{X}$ is the input space and $\mathcal{Y}$ is the output space(structured). To learn $h$, we assume that a training sample of input-output pairs
\[ S = ((x_1, y_1), \ldots, (x_n, y_n)) \in (\mathcal{X} \times \mathcal{Y})^n \]
is available and drawn i.i.d from a joint distribution P($\mathcal{X},\mathcal{Y}$). Following empirical risk minimization principle, we will find an $h \in \mathcal{H}$ that minimizes the empirical risk
\[ R_s^{\bigtriangleup} = \frac{1}{n} \sum_{i=1}^n \bigtriangleup(y_i, h(x_i)) \]
Here $\bigtriangleup(y, \overline{y})$ denotes the \textbf{loss} associated with predicting $\overline{y}$ when $y$ is the correct output. The formulation assumes that the loss function is arbitrary and should satisfy the follwing requirments.
\begin{equation*}
\bigtriangleup(y, \overline{y}) =
\begin{cases}
 > 0 \textrm{ for } y \neq \overline{y} \\
 = 0 \textrm{ for } y = \overline{y}
 \end{cases}
\end{equation*}
StructSVM selects an $h \in \mathcal{H}$ that minimizes a regularized empirical risk on $S$. The general idea here is to learn a discriminant function $\mathnormal{f} : \mathcal{X} \times \mathcal{Y} \longrightarrow \mathbb{R}$ over input-output pairs from which one derives a prediction by maximizing $\mathnormal{f}$ over all $y \in \mathcal{Y}$ for a given input $x$.
\[ h_w(x) =  \underset{y \in \mathcal{Y}}{\arg\max} \mathnormal{f}_w(x,y) \]
We assume that $\mathnormal{f}_w(x,y)$ is linear in some combined feature space relating x and y, denoted as $\Psi(x,y)$.
\[ \mathnormal{f}_w(x,y) = (w \cdot \Psi(x,y)) \]
Here $w \in \mathbb{R}^N$ is the parameter vector. Intuitively we can think of $\mathnormal{f}_w(x,y)$ as a compatibility function that measures how well the output $y$ matches the given input $x$(\cite{joachims_cutting} et al.). This combined feature representation is required in the formulation, since we assumed that the sample $S$ is drawn from a joint distribution P($\mathcal{X},\mathcal{Y}$). Depending upon the structure of the output space, $\Psi(x,y)$ is defined separately for different problem instances.

\subsection{Margin Rescaling(MR) Formulation}
In order to take the loss into consideration, we modify the soft-margin formulation used in SVMs. The soft-margin formulation is given by
\[\underset{w, \xi \geq 0}{\min} \quad \frac{1}{2} \norm{w}^2 + \frac{C}{n} \sum_{i=1}^n \xi_i \]
\[ \textrm{s.t } \forall i \textrm{, } \forall \overline{y} \in \mathcal{Y}\setminus y_i \textrm{  :  } w^T[\Psi(x_i, y_i) - \Psi(x_i, \overline{y})] \geq 1 - \xi_i \textrm{, } \xi_i \geq 0 \]
Here $\xi_i$ is the slack variable and $C$ is the regularization parameter.
\[ \xi_i = \max\{ 0, \max_{y \in \mathcal{Y}\\y_i}(1 - w^T[\Psi(x_i, y_i) - \Psi(x_i, \overline{y})]) \} \]
As we have mentioned previously, this optimization problem is intractable for decomposition methods like SMO, since we have $\mathcal{O}(n|\mathcal{Y}|)$ constraints in the formulation. In Margin Rescaling\nomenclature{MR}{Margin Rescaling} formulation, the margin is adjusted according to the loss. In particular, we adjust the position of the hinge by keeping its slope fixed. The loss in MR formulation is computed as
\[ \bigtriangleup_{MR}(y, h_w(x)) =  \underset{\overline{y} \in \mathcal{Y}}{\max}\{ \bigtriangleup(y, \overline{y}) - (w^T[\Psi(x, y) - \Psi(x, \overline{y})]) \}  \textrm{ } \geq  \bigtriangleup(y, h_w(x))\]
and slack is obtained as $\xi = \max\{0, \bigtriangleup_{MR}(y, h_w(x))\}$. This leads to the following formulation
\[\underset{w, \xi \geq 0}{\min} \quad \frac{1}{2} \norm{w}^2 + \frac{C}{n} \sum_{i=1}^n \xi_i \]
\[ \textrm{s.t } \forall \overline{y_1} \in \mathcal{Y} \textrm{  :  } w^T[\Psi(x_1, y_1) - \Psi(x_1, \overline{y_1})] \geq \bigtriangleup(y_1, \overline{y_1}) - \xi_1 \]
\[ \vdots \]
\[ \textrm{s.t } \forall \overline{y_n} \in \mathcal{Y} \textrm{  :  } w^T[\Psi(x_n, y_n) - \Psi(x_n, \overline{y_n})] \geq \bigtriangleup(y_n, \overline{y_n}) - \xi_n \]
Intuitively, the constraints ensures that the score of the correct label $w^T\Psi(x_i, y_i)$ must be greater than all other scores $w^T\Psi(x_i, \overline{y_i}) \textrm{, } \forall \overline{y_i} \in \mathcal{Y} \setminus y_i$ by a required margin. In MR formulation, the margin is $\bigtriangleup(y_i, \overline{y_i})$.

\subsection{Slack Rescaling(SR) Formulation}
In Slack Rescaling\nomenclature{SR}{Slack Rescaling} formulation, the slack variables are rescaled according to the loss. In particular, the slope of the hinge loss function is adjusted while keeping its position fixed. In SR formulation the margin is 1. The loss in SR formulation is computed as
\[ \bigtriangleup_{SR}(y, h_w(x)) =  \underset{\overline{y} \in \mathcal{Y}}{\max}\{ \bigtriangleup(y, \overline{y})(1 - (w^T[\Psi(x, y) - \Psi(x, \overline{y})) \} \]
and slack is obtained as $\xi = \max\{0, \bigtriangleup_{SR}(y, h_w(x))\}$. This leads to the following formulation
\[\underset{w, \xi \geq 0}{\min} \quad \frac{1}{2} \norm{w}^2 + \frac{C}{n} \sum_{i=1}^n \xi_i \]
\[ \textrm{s.t } \forall \overline{y_1} \in \mathcal{Y} \textrm{  :  } w^T[\Psi(x_1, y_1) - \Psi(x_1, \overline{y_1})] \geq 1 - \frac{\xi_1}{\bigtriangleup(y_1, \overline{y_1})} \]
\[ \vdots \]
\[ \textrm{s.t } \forall \overline{y_n} \in \mathcal{Y} \textrm{  :  } w^T[\Psi(x_n, y_n) - \Psi(x_n, \overline{y_n})] \geq 1 - \frac{\xi_n}{\bigtriangleup(y_n, \overline{y_n})} \]
Both of the above formulation has n slack variables, hence it is called n-slack formulation. These formulations can be converted into 1-slack formulation by summing up all the slack variables(\cite{joachims_cutting} et al.). n-slack formumations have $\mathcal{O}(n|\mathcal{Y}|)$ constraints.

The solution space of this problem is a compact polyhedral convex set. The cutting plane algorithm finds the most violating constraint corresponding to each training example and add it to the working set. After each addition to the working set, we find a solution across all the constraints in working set. This effectively shrinks the size of the version space in a speedy manner. As the iteration continues, the number of constraint violations decreases and the algorithm converges. Every single cut in the convex set corresponds to a constraint violation. Instead of doing a step by step updation, the cutting plane algorithm cuts down a portion of the version space which results in faster convergence.
\section{Experiments}
\label{chap5_exp}
Empirical study was conducted on multi-class and multi-label classification problems. For multi-class problems the loss function used was the absolute difference between labels, and for multi-label problems loss function was the hamming distance between labels expressed in binary form. Since the output space was finite, we used exhaustive search over $\mathcal{Y}$ in both cases, while finding the most violated constraints. Multi-label and multi-class classification problems are the simplest problem instances that can be studied using StructSVM, since their output space is finite.

The combined feature map $\Psi(x,y)$ was constructed as follows. Let $x \in \mathbb{R}^d$ and $k$ be the number of classes. Suppose $x$ is represented as $1:x_1, \ldots, d:x_d$. Then for multi-class problems $\Psi(x,y)$ was obtained by shifting the indices by $(y-1) \times d$ positions; i.e.,
\[ \Psi(x,y) = (y-1) \times d+1:x_1, \ldots, (y-1) \times d+d:x_d \]
For multi-label classification problems, we took the binary representation of $y$ and from that we extracted all bit positions that are ON. Then $\Psi(x,y)$ is computed by applying the same shifting to all the extracted indices.

Implementation was done using \cite{svm_struct} library, by modifying its API functions for multi-label and multi-class problems. Table \ref{chap5_tab1} lists the results of empirical study (value shown is the loss in percentage). The synthetic dataset was a multi-class problem instance available in \cite{svm_struct} library. Here the comparison was made between multi-layer (arc-cosine)kernel machines and commonly used single layer kernel machines.

\renewcommand{\arraystretch}{1.2}
\begin{table}
\centering
\begin{tabular}{|c|c|c|}
	\hline
	\textbf{Dataset} & \textbf{Arc-Cosine Kernel} & \textbf{Other Kernel(best)}\\
	\hline
	Scene Segentation & 30.35 & 30.60 \\
	(multilabel - 6 class) & & \\
	\hline
	Vehicle Dataset & 26.48 & 24.90 \\
	(multiclass - 4 class) & & \\
	\hline
	Iris Dataset & 1.67 & 3.33 \\
	(multiclass - 3 class) & & \\
	\hline
	Breast Cancer Wiscosin & 0.98 & 0.98 \\
	(binary) & & \\
	\hline
	Synthetic Data & 33.85 & 32.55 \\
	(multiclass - 7 class) & & \\
	\hline
\end{tabular}
\caption{Performance comparison of multi-layer arc-cosine kernel to other kernels in StructSVM framework.}
\label{chap5_tab1}
\end{table}
\renewcommand{\arraystretch}{1}

\section{Conclusion}
\label{chap5_conc}
In this chapter, we studied multi-layer kernels in structured output spaces. The experimental study was done on multi-label and multi-class problem instances. The results are competetive with single layer kernel machines. Multi-layer architectures are found to be effective in complex pattern recognition tasks. Hence the discriminating power of these multi-layer kernels must be tested in more complex structured output spaces on problems like natural language parsing, protein sequence alignment prediction etc.


%!TEX root = main.tex
\section{Data}

The basis of a realistic simulation of global virus outbreak is data on
\begin{itemize}
	\item Geographical population densities
	\item Travel connections (commuting and airports)
\end{itemize}

\subsection{Population}
The population data used in this report comes from NASA \cite{nasa-population}. The data is downloadable in a GeoTIFF file-format which is an image file with population encoded in the pixels.

\begin{figure}[H]
	\centering
	\includegraphics[width=1.0 \textwidth]{plots/nasa_population}
	\includegraphics[width=4cm]{plots/nasa_population_colorbar}
	\caption{Plot of population in the world, the used resolution is $0.1 \times 0.1$ degrees per pixel ($3600 \times 1800$).}
\end{figure}

The population dataset encodes $99999.0$ as water and $1.0369266$ as unknown population on land. Thus to get a population map, a raster mapping of $(x_i < 99999.0 \wedge x_i > 1.0369266) \cdot x_i$ was applied. After this transformation the total world population is just $60,031,128$ (without filtering it is $434,130,384,806$). This is obviously wrong, thus the data is upscaled such the sum is 7.4 billion people, which is the estimated number of people in March 2016 \cite{wiki-world-population}.

\subsection{Airport}

The airport connection data was taken from OpenFlights \cite{openflights}. The dataset contains 8021 airports with 37181 airline connections as well as the plane types flying the connection (this could potentially be used for estimating passenger counts). Some airports did not have any connections and where thus removed. After this filtering 3256 airports where left.

Plotting the airport connections we see that Europe, East Coast US and East Cost China are the most connected regions. So if a virus spreads takes hold in these areas we would expect it to cause a global outbreak quickly.

\begin{figure}[H]
	\centering
	\includegraphics[width=1.0 \textwidth]{plots/airport_connections.pdf}
	\caption{Plot of all airport connections in the dataset. Because there are so many connections each connection is plotted with a low alpha.}
\end{figure}

\subsection{Aggression}

Based on the location of the airports a Voronoi partition of the of the earths surface is made. From this partitioning one now has geographical regions. The total population of each region can be found by summing over the appropriate pixels in the population dataset. This aggression is the same as done in the GLEaM paper \cite{GLEaM}.

\begin{figure}[H]
	\centering
	\includegraphics[width=1.0 \textwidth]{plots/voronoi.pdf}
	\caption{Voronoi tessellation based on airport locations. Airports are marked with a dot, region boundaries with lines and dashed lines.}
\end{figure}


\subsection{Coercible}
We now discuss a possible solution to the problems encountered when defining the sample $ex3_wrong$. We give a predicate that expresses the relation of coercive subtyping:
\begin{lstlisting}
class Coercible a b where
  coerce :: a -> b
\end{lstlisting}

\subsection{Coercion for References}
We wish to instance the coercion predicate to references. References are:
\begin{itemize}
\item covariant in the referenced type
\item contravariant in the state type
\end{itemize}
This happens because a reference to some $a$ can be used whenever a reference to an $a$ such that $a \le a'$ is expected, and also (as seen in the third example above), a reference that works on a state $s'$ can be used whenever a state $s$ such that $s \le s'$ is available. Of course, the fact that references express not only reading values and states but also writing will make this operation relatively tricky.

At the moment we will only focus on expressing the coercion relation for the state of the reference; the coercion relation for the value of the reference will be discussed together with inheritance.

The kind of operation that we wish to perform when coercing a reference to work on a larger memory is summarized in Figure \ref{fig:ref_coerce}. Whenever we wish to perform some operation on a reference to the smaller memory, we will:
\begin{itemize}
\item take only the first part of the (larger) input memory
\item perform the operation on the obtained smaller memory through the original reference we have coerced
\item replace the first part of the (larger) input memory with the (smaller) modified memory
\end{itemize}

\begin{figure}[h]
\centerline{\psfig{file=heap_upcasting.png,height=5cm}} \caption{Coercing references.\label{fig:ref_coerce}}
\end{figure}

We instance the coercion predicate for references to perform a single step of coercion, that is for the case when we have a reference to a memory $tl$ and we want to use it where we expect a memory $Cons\ h\ tl$:
\begin{lstlisting}
instance HList tl => Coercible (Ref St tl a) (Ref St (Malloc h tl) a) where
  coerce ref =
    StRef (St(\(Malloc h tl) ->
                          let (res, tl') = get ref tl
                          in (res, h `Malloc` tl')))
          (\v -> St(\(Malloc h tl) ->
            let ((),tl') = set ref tl v
            in ((),h `Malloc` tl')))
    where get (StRef (St g) _) = g
          set (StRef _ s) = \st -> \v ->
                                         let (St s') = s v
                                         in s' st
\end{lstlisting}

Now we can finally rewrite the example above to make use of our new coercion operator:
\begin{lstlisting}
ex3 :: forall m0 m1 m2 st . (HList m0, State st, m1 ~ Malloc Int m0, m2 ~ Malloc String m1, Monad (st m2 m2),
                             Coercible (Ref st m1 Int) (Ref st m2 Int)) => st m0 m2 String
ex3 = do+ i <- new 10
          s <- new "Hello"
          ((coerce i) :: Ref St (String `Malloc` Int `Malloc` Nil) Int) *= (+2)
          s *= (++ " World")
          eval s

res3 = runSt ex3 Nil
\end{lstlisting}


\documentclass[../main.tex]{subfiles}

\begin{document}
\begin{center}
	\centering
	\includegraphics[width=\linewidth,height=4cm]{comparison2.png}
	\captionof{figure}{\label{fig:cvae} Overview of scene sampling and CVAE distribution learning.}
\end{center}

\subsubsection{Overview}
The proposed approach consists of image to image comparison with conditional
variational autoencoders (CVAE) \cite{kingma2014semi}, as shown in \textbf{Fig.}
\ref{fig:cvae}. The CVAE is a semi-supervised method for approximating the
underlying generative model that produces a set of images and their
corresponding class labels in terms of the so-called unobserved latent
variables. Each of the input images is described in terms of a probability
distribution over the latent variables and the classes.

Their approach consists of using the probability distributions calculated by
the CVAE for each image as a descriptor. The comparison between an image query
and the 3D scene renderings is with respect to the probability distributions
obtained from the CVAE. The method consists of data pre-processing, training
and retrieval described in the following subsections.

\subsubsection{Data Preprocessing}
Thirteen renderings are obtained for each of the 3D scenes. Each of the 3D
scenes has a predefined view when loaded into the SketchUp software. This view
is saved as a 2D view together with twelve views at different angles around the
scene as in \cite{Su2015}.
The training data set consists of the 3D scene renderings together with the
training images. All images are resized to a resolution of $64\times64$ and all
pixel values are normalized to the interval $[0,1]$. Image data augmentation is
carried out by performing a horizontal flip to all images. The corresponding
data space is $X = [0,1]^{64\times 64 \times 3}$, while the 3 represents the
color space.

\subsubsection{Training}
The CVAE consists of an encoder and a decoder neural network. The encoder network calculates from an image $x\in X$ the parameters of a probability distribution over the latent space $Z = \mathbb{R}^d$ and over the thirty class values in $Y = \{1,2,3,\ldots, 30\}$. The decoder network calculates from a latent variable $z\in Z$ and a class $y\in Y$, the parameters of a distribution over the data space $X$ .

The distributions for the encoder correspond to a normal distribution over $Z$
and a categorical distribution over $Y$. A normal distribution over $X$ is
chosen for the decoder. The probabilistic model used corresponds to the M2
model described in the article \cite{kingma2014semi}. Both the encoding and
decoding neural networks are convolutional.

The CVAE is fed with batches of labeled images during training. The loss function is the sum of the negative Evidence Lower Bound (ELBO) and a classification loss. The ELBO is approximated by means of the parametrization trick described in \cite{kingma2014semi, kingma2013auto} and represents the variational inference objective. The classification loss for their encoding distributions over $Y$ corresponds to the cross entropy between the probability distribution over $Y$ with respect to the input label.

\subsubsection{Retrieval}
After training, an image $x\in X$ can be described as a conditional joint
distribution over $Z\times Y$. The density $q_\phi(z|x)$ corresponds to a
normal distribution and $q_\phi(y|x)$ to a categorical distribution over $Y$,
where $\phi$ represents the weights of the encoder neural network. The joint
density corresponds to $q_\phi(z,y|x) = q_\phi(z|x) q_\phi(y|x)$.

The similarity $D$ between an input query image $x^*\in X$ and a 3D scene in
terms of its $N$ rendered images $S = \{x_r\}_{r=1}^{N}$ is given by the
minimum symmetrized cross entropy $H_s$ between the query and the rendered
images' probability distributions (see \textbf{Fig.} \ref{fig:cvae}).

\begin{multline}
 D(x^*, S)\min_{r\in\{1,2,\ldots,13\} }H_s(q_\phi(z|x^*),q_\phi(z|x_{r}))\\
+\alpha H_s(q_\phi(y|x^*),q_\phi(y|x_{r})).
\end{multline}

They have used the parameter $\alpha = 64\times 64 \times 3$ to increase the importance of label matching. A ranking of 3D scenes is obtained for each query according to this similarity.

\subsubsection{Five Runs}
They have sent five submissions corresponding to methods who differ only on the architecture of the encoding and decoding neural networks. These are described as follows:
\begin{enumerate}
    \item \textbf{CVAE-(1,2,3,4)}: CVAE with different CNN architectures for the encoder and decoder.
    \item \textbf{CVAE-VGG}: CVAE with features from pre-trained VGG \cite{gkallia2017keras_places365} on the Places data set \cite{Places88} as part of the encoder.

\end{enumerate}

\end{document}


\section{Exploratory Data Analysis}

% As part of the EDA, we checked the class distribution, the distribution of features amongst the different classes and analysed the first two principle components of the data.

% As part of our exploratory data analysis (EDA), we first examined scaled features and how well they each divide glass fragments based on their class. Second, we looked at a scatter plot of the first two principal components, which explain $51 \%$ of variance to examine the same thing as in the first example. Our overall goal was to use this analysis for our decisions during modelling part of the project.
\subsection{Class Distribution}
Figure \ref{class_distribution} shows the class distribution in the training and test split. The distribution of classes were roughly the same amongst the two splits. This was important to notice, since evaluating a classifier that was trained on data with a class distribution highly different from the testing set, would result in an inaccurate estimations for the out-of-sample performance.
Apart from that, a strong imbalance in the six observed classes became obvious. While the two majority classes 1 and 2 made up more than 65\% of the entire data, the underrepresented classes 3, 5 and 6 all combined accounted for less than 25\% of all data points. This imbalance might pose challenges when it comes to building a classifier correctly classifying the minority classes.
\newline

\begin{figure}[ht]
\centering
\includegraphics[scale=0.6]{figures/class_distribution.pdf}
\caption{Class Distribution in (a) Training Split and (b) Test Split}
\captionsetup{justification=centering,margin=2cm}
\label{class_distribution}
\end{figure}

\begin{comment}
\subsection{Feature Distribution}
Table \ref{feature_distribution} shows the five-number summary of each feature computed from the training and test split combined. It becomes evident, that all types of glass fragments are primarily composed of the chemical compound \textit{silicone} (between $\sim 69\%$ and $\sim 75\%$) and \textit{sodium} (between $\sim 10\%$ and $\sim 17\%$). Interestingly, the RI, which was expected to have a high variance amongst different types of glass fragments, has a small difference between the maximum and minimum value. Looking at the feature distribution for each class will reveal, how well each feature separates the six classes.
\newline

\begin{table}[!ht]
  \small
  \centering
\begin{tabular}{ c | c c c c c}
 \toprule
 Feature &  Min  & Lower Quartile  &  Median  &.  Upper Quartile  &   Max\\
 \midrule
 RI   &   1.51  &  1.52   & 1.52  & 1.52  & 1.53 \\
 Na   &   10.73 &  12.91  & 13.30 & 13.83 & 17.38 \\
 Mg   &   0.00  &  2.12   & 3.48  & 3.60  & 4.49 \\
 Al   &   0.29  &  1.19   & 1.36  & 1.63  & 3.50 \\
 Si   &   69.81 &  72.28  & 72.79 & 73.09 & 75.41 \\
 K    &   0.00  &  0.12   & 0.56  & 0.61  & 6.21 \\
 Ca   &   5.43  &  8.24   & 8.60  & 9.17  & 16.19 \\
 Ba   &   0.00  &  0.00   & 0.00  & 0.00  & 3.15 \\
 Fe   &   0.00  &  0.00   & 0.00  & 0.10  & 0.51 \\
 \bottomrule
\end{tabular}
\captionsetup{justification=centering,margin=2cm}
\caption{Five-Number-Summaries for all Features}
\label{feature_distribution}
\end{table}
\end{comment}

\subsection{Distribution of Features For Each Class}
Figure \ref{feature_distribution_classes} displays the distribution of each feature in each class, which gives an intuition of how well each feature separates the classes.

\begin{figure}[ht]
\centering
\includegraphics[scale=0.3]{figures/fdist_violin.jpg}
\captionsetup{justification=centering,margin=2cm}
\caption{Standardised Distribution of Features for each Class}
\label{feature_distribution_classes}
\end{figure}

The nine plots show that the different features are differently well-suited for classifying the glass fragments. Features such as \textit{silicone} do not seem to perform well, as there is little variation in the distribution of the feature in the six glass types. On the opposite, features such as \textit{calcium}, \textit{barium}, \textit{magnesium} or the \textit{RI}, seem to be better suited, as they separate subsets of types of glass fragments. However, it can be summarised, that there does not exist a single feature, that can be used to perfectly separate the classes. It is therefore likely that the classifier will need to be trained on a combination of features.

\subsection{Principal Component Analysis}
%Principle component analysis (PCA) is an unsupervised data exploration tool to transform a set of features into a set of representative features - called principle components - that collectively explain most of the variability in the original set.
Figure \ref{pca} visualises the first two principle components of the data set. The visualisation provides an understanding of the difficulty of the classification problem.

It is evident, that the classification problem is not trivial. The first two principle components reveal, that glass types for similar real-world purposes, ie. class 1, 2 and 3 all being windows, have related feature distributions and are thus overlapping in the first two principle components. This is likely to pose a challenge to the classifier. In contrast,  class $7$ (headlamp) is well-separated and is therefore expected to be easier to classify correctly.

Another challenge might be the under-representation of classes 3, 5 and 6. It will be difficult for the model to learn the patterns in the data for these classes, especially in the case of class 3 being in the middle of the point clouds of the majority classes 1 and 2.

\begin{figure}[ht]
\centering
\includegraphics[scale=0.35]{figures/pca_best2_scatter.jpg}
\captionsetup{justification=centering,margin=2cm}
\caption{Scatter Plot of first two Principal Components of Training Split}
\label{pca}
\end{figure}
\newpage