% !TEX root = ../zeth-protocol-specification.tex

\section{Receiving \zethnotes}\label{zeth-protocol:zeth-receive}

In order to confirm the reception of \zethnotes, $\zparty{R}$ must listen to the events (\cref{sssec:ethereum-events}) of type $\mixEventDType$ emitted by the processing of $\zethTx$, and try to decrypt the ciphertexts using $\zparty{R}.\privaddr.\skenc$ to see if he is the recipient of a \zeth~payment. If the decryption is successful ($\zparty{R}$ is the recipient of a payment), $\zparty{R}$ must verify that the $\zethnote$ recovered is the opening of a commitment in the Merkle tree of $\mixer$. If not, $\zparty{R}$ rejects the (invalid) payment.

We describe below the steps that $\zparty{R}$ needs to carry out for all events $\evMixOut{} \in \mixEventDType$ emitted by \mixer{}, in order to receive payments:

\begin{enumerate}
    \item\label{zeth-receive:check-commit-tree} Compute the new root $\mkroot'$ of the Merkle tree of commitments, after adding the new values $\evMixOut.\cms{}$. If this value does not match the new root $\evMixOut.\mkroot$ emitted by \mixer{}, abort.
    \item\label{zeth-receive:decrypt} Try to decrypt the ciphertexts:
        \begin{align*}
            & zn_j = \encscheme.\dec(\zparty{R}.\privaddr.\skenc, \evMixOut.\ciphers[j])
        \end{align*}
    \item For each successful decryption, let $j$ be the index of the decrypted ciphertext:
        \begin{enumerate}
            \item\label{zeth-receive:parse-plaintext} Check whether the recovered plaintext $zn_j$ is a well-formed $\zethnote$. Abort if it is not well-formed.
            \item\label{zeth-receive:check-note-data} Check that the recovered \zethnote $zn_j$ is the opening of the corresponding commitment $\evMixOut{}.\cms{}[j]$:
                \[
                    \evMixOut.\cms{}[j] = \comm.\commit{zn_j.\apk, zn_j.\rrho, zn_j.\notev}{zn_j.\noter}
                \]
              Abort if the note is not a valid opening.
            \item Additionally, if sender \zparty{S}, and recipient \zparty{R} had a contractual agreement, then \zparty{R} needs to check that the terms of this agreement are fulfilled by all the recovered $\zethnotes$, abort otherwise.
        \end{enumerate}
\end{enumerate}

Note that Steps \ref{zeth-receive:check-commit-tree} and \ref{zeth-receive:check-note-data} are required to ensure that data decrypted by $\zparty{R}$ exactly matches the data committed to in \mixer{}. In particular, Step \ref{zeth-receive:check-commit-tree} requires \zparty{R} to maintain or have access to some representation of the Merkle tree of commitments. See~\cref{client-security:syncing:note-management} for further details.


\documentclass[Physics.tex]{subfiles}
\begin{document}
\chapter{Quantum Physics}
\section{Wave-particle duality}
Classical particles exhibit wavelike properties when conditions are appropriate, while classical waves i.e. light can exhibit particle-like properties. This is wave-particle duality. For any particle, the de Broglie wavelength \begin{equation}\lambda = \frac{h}{p}\end{equation} Wave properties start to show at lengths of similar order of magnitude as the wavelength of a particle.

Phenomena like the photoelectric effect show that light has a particulate nature, while phenomena like interference and diffraction show that light has a wave nature.

A \sldef{photon} is a discrete quantum of electromagnetic energy. Photons have energy dependent on their frequency and momentum \begin{align}E &= hf = \frac{hc}{\lambda}\\\left|\mathbf{p}\right| &= \frac{E}{c} = \frac{h}{\lambda}\end{align}
\subsection{Electron diffraction}
When electrons are fired at appropriate velocities at a metal foil, they diffract to form a pattern of rings which are similar to the pattern obtained when X-rays are fired at similar foil.

In this case, the metal crystals in the foil act as a diffraction grating for the electrons to diffract. A normal grating will not work as the slit is too large -- as in superposition, the slit must be about the same order as the wavelength of the particles.
\section{Photoelectric effect}
The \sldef{photoelectric effect} is the phenomenon where electrons are emitted from a clean metal surface when electromagnetic radiation of sufficiently high frequency is incident on the surface. The emission of photons is instantaneous once suitable light strikes the surface. Electrons emitted through the photoelectric effect are \sldef{photoelectrons}, and the current created by the movement of electrons is the \sldef{photocurrent}.

The effect occurs because electrons absorb photons and convert their energy into kinetic energy, with which the electrons escape. The absorption of a photon in the photoelectric effect is an all-or-nothing process: either all the energy of a photon is absorbed by one electron, or none.

The \sldef{work function} of a material is the minimum energy necessary to remove a free electron from the surface of the material. This energy is used in overcoming attractive forces between an electron and the material.

For photoelectrons to be emitted, incident photons must have energy of at least the work function of the metal i.e. the photons must have frequency higher than a threshold frequency, or wavelength shorter than a threshold wavelength.

Since an electron absorbs all the energy of a photon, and the only energy loss is the work function of the metal, we have \begin{equation}hf = \Phi + T_\text{max}\label{eq:18.pe.coe}\end{equation} i.e. the maximum kinetic energy of the photoelectrons depends only on the incident frequency and the work function.

If photoelectrons are decelerated through a potential difference of \(V\), then the minimum potential to stop all photoelectrons is when \[T_\text{max} = eV\] and substituting into \eqref{eq:18.pe.coe} we get \begin{equation}eV_s = hf - \Phi\end{equation} where \(V_s\) is the stopping potential.

The intensity of a beam of photons is \[I = \frac{E_\text{total}}{tA} = \frac{N}{tA}hf = nhf\qquad n = \frac{N}{tA}\] thus an increase in intensity, holding \(f\) constant, means an increase in \(n\) i.e. there are more incident photons per unit area per unit time, and correspondingly there will be more photoelectrons and a higher photocurrent.
\section{Atomic structure}
Isolated atoms have discrete electron energy levels i.e. electrons in an atom can only possess certain specific energies. Similar to the potential of attractive fields, the energies of energy levels are negative, and infinity has zero energy.

Energy levels can be mostly taken to correspond to quantum electron shells, where \(n=1\) is the innermost quantum shell and thus the lowest energy level, and so on for \(n \in \mathbb{Z}^+\).

The \sldef{ground state} of an atom is when all its electrons are at their lowest possible energy levels. The ionisation energy of an atom is the energy needed to remove an electron from the outermost energy level containing an electron to infinity (to form an ion).

Electrons can be excited from one energy level to a higher one by the absorption of a photon with energy equal to the difference in energy between the two energy levels, or through energy imparted by collision with a high-energy particle, which can have any energy as long as it is sufficient. (The particle can retain some energy and fly off.)

Electrons can de-excite from one energy level to a lower one by the emission of a photon with energy equal to the difference in energy between the two energy levels.
\section{Line spectra}
Discrete electron energy levels in isolated atoms means that an electron in an atom can only have specific energies. When an electron falls from one energy level to another, energy is released in the form of a photon with energy equal to the difference in energy between the two levels. Since the energy levels are discrete, photons of only certain frequencies can be released, hence forming a line spectrum.

For an \sldef{emission spectrum}, atoms are first excited using a high voltage or by heating. When the atoms go back to the ground state, either directly or via intermediate energy levels, photons of only certain frequencies are emitted due to the discrete energy levels, so only certain frequencies of light are observed, forming an emission spectrum i.e. discrete bright coloured lines on a dark background.

For an \sldef{absorption spectrum}, white light is used to excite atoms. Incident photons with energies exactly equal to the difference between two of an atom's energy levels are absorbed. Since energy levels are discrete, only photons of certain frequencies are absorbed. When the atoms go back to the ground state, photons of the same frequencies are emitted but in all directions. As these frequencies of light now have much lower intensity, they account for the dark lines in the absorption spectrum i.e. discrete dark lines on a continuous spectrum.
\section{X-ray production}
X-rays can be produced when a beam of high-energy electrons, created through thermionic emission by heating a metal cathode and then accelerated through a potential difference, strikes a metal anode. This produces an X-ray spectrum consisting of the continuous spectrum and the characteristic spectrum.

The \sldef{continuous spectrum} is produced when emitted electrons accelerate suddenly due to attraction from metal nuclei in the anode, emitting photons with energy equal to the change in kinetic energy, while some electrons will come to a stop and so emit photons with energy equal to their kinetic energy. Radiation produced in this way is known as bremsstrahlung i.e. braking radiation.

As electrons approach the target at different positions and angles, there is a distribution of photon energies and thus wavelengths, forming a continuous spectrum. The photon of highest energy is formed by electrons that come to a complete stop, and so we have \begin{equation}eV = hf_\text{max}\end{equation} This maximum energy is independent of the target material.

The \sldef{characteristic spectrum} is formed when emitted electrons collide into electrons bound to target atoms and cause them to be ejected, leaving vacancies that are then filled by electrons in higher energy levels, emitting a photon due to the transition, with energy equal to the difference in energy between the energy levels, creating sharp peaks in the X-ray spectrum.

When the vacancy is formed in the K (\(n=1\)) shell and is filled by an electron from the L (\(n=2\)) shell, the photon is termed a K\textsubscript{\upalpha} X-ray. If the same vacancy is filled by an electron from the M shell, the photon is termed a K\textsubscript{\upbeta} X-ray.

Generally, K\textsubscript{\upalpha} X-rays have higher intensity than K\textsubscript{\upbeta} X-rays as electrons in the L shell are closer to the K shell, so there is higher probability of a K shell vacancy being filled by an L electron versus an M electron. Characteristic X-rays have shorter wavelengths for targets of higher proton number, as the energy differences between shells increase.
\section{Heisenberg's uncertainty principle}
Quantum mechanics tells us that there is a limit to how precisely we can know a particle's momentum and position, given by Heisenberg's uncertainty principle. This principle also relates energy and time e.g. in the case of energy emitted by de-excitation of atoms over a specified time interval. \begin{align}\Delta x\Delta p &\geq \frac{\hbar}{2} = \frac{h}{4\pi}\\\Delta E\Delta t &\geq \frac{\hbar}{2} = \frac{h}{4\pi}\end{align}
\section{Wavefunctions}
The Schr\"odinger's equation describes quantum systems much like Newton's laws describe classical systems. Where Newton's laws deals with position and time, Schr\"odinger's equation deals with wavefunctions \(\Psi\).

The wavefunction contains all the information about a quantum system at all times. The wavefunction gives us the probability density function weighing all measurements made of the system, and from this we can derive the probability of finding a particle between positions \(a\) and \(b\): \begin{equation}P(a \leq \mathbf{x} \leq b) = \int_a^b\left|\Psi\right|^2\slid\mathbf{x}\end{equation}
\section{Quantum tunneling}
Suppose an electron is travelling in a straight line towards a region of negative potential. The electron must do work to pass this region, and so this region is a potential barrier to the electron.

Clasically, if the electron does not have enough energy to overcome this barrier, it will never do so. Quantum mechanically, there is a probability for the electron to tunnel through the barrier even when its energy is lower than that of the potential barrier, akin to when some light passes through an interface between two materials while some light is reflected back into the material in which the incident beam is in. This is \sldef{quantum tunneling}.

When this happens, the matter wave before the barrier becomes a standing wave arising from the interference between the incident and reflected matter wave, while the wavefunction after the barrier is a wave of lower amplitude but equal energy (frequency), indicating a small probability of the electron being transmitted. Within the barrier, the wavefunction undergoes exponential decay.

The probability that an incident electron tunnels the barrier through is known as the \sldef{transmission coefficient} \(T\), while the probability that it reflects is known as the \sldef{reflection coefficient} \(R\). Since the electron must either be reflected or transmitted, \[R+T=1\]

\(T\) in this syllabus is given as \begin{equation}T \propto \exp(-2kd)\qquad k = \sqrt{\frac{8\pi^2m(U-E)}{h^2}}\end{equation}
\end{document}


\documentclass[10pt]{article}
\usepackage[a4paper, margin=1in]{geometry}
\usepackage{hyperref}
\usepackage{setspace}
\usepackage[style=iso]{datetime2}
\usepackage{amsmath}
\usepackage{eufrak}
\usepackage{amssymb}
\usepackage{mathtools}

\newcommand{\defeq}{\vcentcolon=}

\title{Introduction to Automatic Differentiation Variational Inference}
\author{Harshad Deo \\
  \href{mailto:harshad@moreficent.com}{harshad@moreficent.com} \\
  \href{mailto:harshad@simianquant.com}{harshad@simianquant.com}
}
\date{}

\setlength{\parindent}{0cm}

\hypersetup{
  colorlinks,
  citecolor=blue,
  filecolor=blue,
  linkcolor=blue,
  urlcolor=blue
}

\setlength{\parskip}{\baselineskip}%

\begin{document}

\maketitle

Bayesian methods allow us to capture uncertainty while drawing inferences and making predictions from data. Instead of
point estimates, they provide the machinery to generate a probability distribution over the inference targets. Aside from
guarding against over-zealous interpretations of the analysis, quantifying the uncertainty is useful when the utility of
the inference target is a non-linear function of its value. For example, if two assets, say a stock or a house, have the
same expected return but the second has a much higher variance (i.e.\ is more risky), most people would invest in the
first.

Formally, given observations $x$ and latent variables $\theta$, Bayesian methods attempt to construct the posterior
distribution of the latent variables given the data, i.e.\ $p(\theta|x)$. For most practical problems, it is intractable
to calculate the true posterior, therefore in practice, Bayesian methods are a suite of algorithms that attempt to
approximate the true posterior. There are two families of approaches that are used:

\begin{enumerate}
  \item Those that construct a stationary stochastic process whose outputs are samples from the joint posterior, like
    Markov Chain Monte Carlo and Hamiltonian Monte Carlo. These do not scale to large datasets and are tricky at
    best to parallelize.
  \item Those that choose the best approximation to the true posterior from a known family of distributions. This method
    of inferring the true posterior is called variational inference (VI).
\end{enumerate}

This monograph presents Automatic Differentiation Variational Inference (ADVI), a new VI algorithm
developed by Kucukelbir at al \footnote{Kucukelbir, Alp, et al. ``Automatic differentiation variational inference."
The Journal of Machine Learning Research 18.1 (2017): 430-474.} that overcomes the shortcomings of earlier VI approaches.
Subsequent monographs will develop simplifying assumptions that allow complex models to be calibrated to large datasets.

\section*{Applicability}

Variational approaches haven't been frequently used in practice because each problem required the construction of a new
variational scheme and optimization procecure. ADVI aims to automate this process.

\begin{enumerate}
  \item ADVI transforms any applicable probability model into one over unconstrained real latent variables. Since all the
  latent variables are defined in the same space, ADVI can use a single variational family for all models.
  \item ADVI expresses the gradient of the objective function as an expectation over a standard gaussian. This allows
  the gradient to be efficiently computed using Monte Carlo sampling.
\end{enumerate}

ADVI is applicable to all differentiable probability models. Given a probability parameterized by a $K$ dimensional
latent space $\theta$, and its support given by:

\begin{equation}
  supp(p(\theta)) = \{\theta | \theta \in \mathbb{R}^K \, \text{and} \, p(\theta) > 0 \} \subseteq \mathbb{R} ^ K \notag
\end{equation}

ADVI is applicable if $\theta$ is continuous and the gradient of the log joint density, $\nabla_{\theta}\log p(x, \theta)$,
is defined within the support of the prior.

\section*{Objective Function}

Considering a family of approximating variational densities $q(\theta ; \phi)$, parameterized by $\phi \in \Phi$,
variational inference finds the member of the family that minimizes the KL divergence to the posterior:

\begin{equation*}
  \phi^* = \underset{\phi \in \Phi}{\text{argmin}}\, KL(q(\theta ; \phi) || p(\theta | x)) \quad \text{such that} \quad supp(q(\theta ; \phi)) \subseteq supp(p(\theta | x))
\end{equation*}

The optimised $q(\theta; \phi^*)$ serves as an approximation to the posterior.

\begin{align*}
  KL(q(\theta ; \phi) || p(\theta | x)) \defeq &\mathbb{E}_{q(\theta)}\big[\log q(\theta;\phi) - \log p(\theta|x)  \big] \\
  =& \mathbb{E}_{q(\theta)}\big[\log q(\theta;\phi) - \log p(\theta, x) + \log p(x)\big] \\
  =& \mathbb{E}_{q(\theta)}\big[\log q(\theta;\phi) - \log p(\theta, x) \big] + \log p(x)
\end{align*}

Since the KL divergence is itself dependent on the posterior density, it cannot be directly minimized. Instead, we try
to maximise the Evidence Lower Bound (ELBO), given by:

\begin{equation*}
  \mathcal{L}(\phi) = \mathbb{E}_{q(\theta)}\big[\log p(x, \theta) - \log q(\theta;\phi) \big]
\end{equation*}

The negative of the ELBO is equal to the KL divergence, upto an additive constant. Therefore maximising the ELBO is
equivalent to minimising the KL divergence. Therefore the target objective function is given by:

\begin{equation*}
  \phi^* = \underset{\phi \in \Phi}{\text{argmax}}\, \mathcal{L}(\phi) \quad \text{such that} \quad supp(q(\theta ; \phi)) \subseteq supp(p(\theta | x))
\end{equation*}

\section*{Unconstrained Objective Function}

The first step is to remove the domain constraint of the objective function. Define a bijective differentiable function $T$:

\begin{align*}
  &T : supp(p(\theta)) \to \mathbb{R}^K \\
  &\zeta = T(\theta)
\end{align*}

The transformed joint density now has the representation:

\begin{equation*}
  p(x, \zeta) = p\big(x, T^{-1}(\zeta)\big)|\det J_{T^{-1}}(\zeta)|
\end{equation*}

To illustrate, consider the three most common constraint transforms as they are implemented in Stan:\footnote{Stan Development Team. 2020. Stan Modeling Language Users Guide and Reference Manual, 2.25. https://mc-stan.org}

\begin{itemize}
  \item $\theta > a \implies \zeta(\theta) = \log(\theta - a)$
  \item $\theta < b \implies \zeta(\theta) = \log(b - \theta)$
  \item $a < \theta < b \implies \zeta(\theta) = \log\big(\frac{\theta - a}{b - \theta}\big)$
\end{itemize}

With this transformation, the unconstrained objective function can be expressed as:

\begin{equation*}
  \phi^* = \underset{\phi \in \Phi}{\text{argmax}} \, \mathbb{E}_{q(\zeta; \phi)}\big[\log p(x, T^{-1}(\zeta)) + \log |\det J_{T^{-1}}(\zeta)| - \log q(\zeta; \phi) \big]
\end{equation*}

\section*{Variational Approximation}

ADVI uses a multivariate normal distribution to approximate the distribution of the transformed latent variables ($\zeta$). This
implicitly induces a non-Gaussian distribution in the original latent space ($\theta$). The approximation can be implemented in
two main ways:

\begin{enumerate}
  \item \textbf{Mean-field Gaussian}, in which all the transformed latent variables are assumed to be independent
  \item \textbf{Full-rank Gaussian}, in which all the transformed variables are allowed to be correlated
\end{enumerate}

The number of parameters of a Full-rank Gaussian approximation ($\phi$) grow quadratically with the dimensionality of the
latent space ($K$), making this approach unfeasable for large models. Since the focus of this repository is on techniques
to fit complex models to large datasets, subsequent monographs will focus on the mean field approximation. For the sake of
generality, this monograph presents the full rank approach. Therefore the variational approximation is given by:

\begin{align*}
  q(\zeta, \phi) \defeq& \mathcal{N}\big(\zeta; \mu, \Sigma ^ 2\big) \\
  \Phi =& \{\mu_1, \ldots, \mu_K, l_{\{1, 1\}}, \ldots, l_{\{K, K\}} \} = \mathbb{R}^{K ( K + 1) / 2}
\end{align*}

Where $l_{\{i, j\}}$ is a Cholesky factor of the covariance matrix. With this approximation, the optimization objective
can be expressed as:

\begin{equation*}
  \phi^* = \underset{\phi \in \Phi}{\text{argmax}} \, \mathbb{E}_{q(\zeta; \phi)}\big[\log p(x, T^{-1}(\zeta)) + \log |\det J_{T^{-1}}(\zeta)|\big] - \mathbb{E}_{q(\zeta; \phi)}\big[\log q(\zeta; \phi) \big]
\end{equation*}

The second term is the entropy of a multivariate Gaussian. This can be simplified to:

\begin{equation*}
  \mathbb{E}_q[\log q] = \frac{K}{2} + \frac{K}{2}\ln(2\pi) + \frac{1}{2}\ln\det \Sigma^2
\end{equation*}

Therefore the optimization objective can be simplified to:

\begin{equation*}
  \phi^* = \underset{\phi \in \Phi}{\text{argmax}} \, \mathbb{E}_{q(\zeta; \phi)}\big[\log p(x, T^{-1}(\zeta)) + \log |\det J_{T^{-1}}(\zeta)|\big] + \frac{1}{2}\ln \det \Sigma ^ 2
\end{equation*}

The final step is to convert the expectation over $q(\zeta; \phi)$ into one over a standard normal variable. To achieve
this, consider a transformation $S_{\phi}$ that aborbs the variational parameters $\phi$\footnote{This is variously
referred to as elliptical standardization, coordinate transformation, invertable transformation and the re-parameterization
trick}. In general, this is given by:

\begin{align*}
  \eta &= S_{\phi}(\zeta) = L^{-1} (\zeta - \mu), \quad \text{where $L$ is a cholesky decomposition of $\Sigma$} \\
  q(\eta) &= \prod_{k=1}^{K}\mathcal{N}(\eta_k; 0, 1)
\end{align*}

With this transformation, the objective function is given by:

\begin{equation*}
  \phi^* = \underset{\phi}{\text{argmax}} \, \mathbb{E}_{\mathcal{N}(\eta; 0, 1)}\big[\log p(x, T^{-1}(S_\phi^{-1}(\eta))) + \log |\det J_{T^{-1}}(S_\phi^{-1}(\eta))|\big] + \frac{1}{2}\ln \det \Sigma ^ 2
\end{equation*}

Since the expectation in the objective is no longer dependent on the optimization target ($\phi$), the gradient of the expectation
equals the expectation of the gradient, and can therefore be efficiently calculated using Monte Carlo integration. In the paper,
the authors argue that, in practice, a single sample is sufficient. The objective function can now be solved by standard procedures
like gradient ascent.

\end{document}


%!TEX root = soutenance_lei_2018.tex
\subsection{Panoramic view of the heat flux inside the vehicle}
\frame{\tableofcontents[currentsection,currentsubsection]}

\begin{frame}{Outline}
 As a truck is normally huge, the homography transformation would then have a large deviation in the final thermal images.\\
 \pause
 ==> Take photo inside the truck ?\\
 \pause
 ==> \alert{Panoramic view!}
\end{frame}

\begin{frame}{Experimental Set-up}
    \begin{figure}[ht]
        \centering
        \includegraphics[scale=0.3]{img/ch3/Camera_setup.jpg}
        \caption{Pan-tilt device for experiment test}
    \end{figure}
\end{frame}

\begin{frame}{Image processing--\small{spherical projection}}
    \begin{figure}
        % \centering
        \hspace*{-15pt}
        \includegraphics[scale=0.238]{img/ch3/Sph_1.jpg}
        \includegraphics[scale=0.238]{img/ch3/Sph_2.jpg}
        \caption{Spherical projection.}
        \label{Sph_pro}
    \end{figure}
\end{frame}

\begin{frame}{Image processing--\small{spherical projection}}
Therefore, given the focal length $ f $ and the image coordinates $ (x, y) $, the corresponding spherical coordinates $ (x', y') $ are:

    \begin{align*}
        x'={} f \cdot tan(\dfrac{x-x_c}{f})+x_c \notag \\
        y'=f \cdot \frac{tan(\dfrac{y-y_c}{f})}{cos(\dfrac{x-x_c}{f})} +y_c
    \end{align*}
where $ (x_c,y_c) $ are the center coordinates of the spherical image.

\end{frame}

\begin{frame}{Image processing--\small{spherical projection}}
The original image and its spherical projection are then:
    \begin{figure}
        % \centering
        \hspace*{-15pt}
        \includegraphics[scale=0.336]{img/ch3/img7.jpg}
        \includegraphics[scale=0.336]{img/ch3/sph_img7.jpg}
        \caption{Original IR image (left) and its spherical projection (right).}
        \label{Orig_sph}
    \end{figure}
\end{frame}

\begin{frame}{Image projection--\small{Harris corner}}
    \begin{figure}
        %   \centering
        \hspace*{-15pt}
        \includegraphics[scale=0.50]{img/ch3/Harris1.png}
        \includegraphics[scale=0.50]{img/ch3/Harris2.png}
        \caption{Harris corners detected (green crosses) in the figure above and its precedent image of the whole series. }
        % \label{Harris}
    \end{figure}
    % In these two continuous images, the IR camera scanned from top to bottom inside the vehicle and there is a ``clear" (from our vision) overlapping part (left bottom part in the first and left top part in the second) in these two images. However, from the Harris corners detection results, all the features obtained could not be matched correctly.
\end{frame}

\begin{frame}{Image translation and stitching}
    \begin{figure}
        \centering
        \vspace*{-10pt}
        \includegraphics[scale=0.3]{img/ch3/img_trs.jpg}\\
        \pause
        \includegraphics[scale=0.3]{img/ch3/img_sti.jpg}
    \end{figure}
\end{frame}

\begin{frame}{Temperature panorama}
    \begin{figure}
        \hspace*{-15pt}
        \includegraphics[scale=0.222]{img/ch3/Pano_T_Final.jpg}
    \end{figure}
    \centering
    Mean value: $ \overline{T} = 305.94$ K $= 32.79$ °C
\end{frame}

\begin{frame}{Heat flux meter}
    \begin{figure}
    \centering
    \vspace*{-18pt}
    \includegraphics[scale=0.31]{img/ch3/QIRT2015Aisa_Fig8.png}
    \end{figure}
    % \pause
    Mean value after 10 hours (steady condition): $q_{ref}=11.460\; W/m^2$
\end{frame}

\begin{frame}{Heat flux panorama}
    \begin{figure}
        \hspace*{-15pt}
        \includegraphics[scale=0.222]{img/ch3/Pano_Q_Final.jpg}
    \end{figure}
    \centering
    Mean value: $\bar{q}=11.724\; W/m^2$
\end{frame}

\begin{frame}{Results \& discussion}
 Therefore, the final K-value from IR thermography is:

    \begin{equation*}
    K_{th}=\frac{\bar{q}}{\Delta ̅\theta} =\frac{11.724\; W/m^2}{(32.79-7.5)\;K}=0.464\; W/K\cdot m^2
    \end{equation*}

\pause
Comparing with ATP standard:
    \begin{table}[ht]
        \centering
        \caption{ATP test results.}
        \begin{tabular}{l|r}
            \hline
            Fans Power [W] (Mean over 6 hours) & 144 \\
            \hline
            Heaters Power [W] (Mean over 6 hours) &   988\\
            \hline
            Internal temperature [°C] (Mean over 6 hours) &   32.5\\
            \hline
            External temperature [°C] (Mean over 6 hours) &   7.5\\
            \hline
            K-Value [ W/(K m$^2 $)] & 0.46 \\
            \hline
        \end{tabular}
        % \label{ATP_res}
    \end{table}
\pause
The error between these two results is then:

    \begin{equation*}
        e=  \frac{|K_{th}-K|}{K}=\frac{0.464-0.46}{0.46}=0.0087=0.87\%
    \end{equation*}
\end{frame}

\begin{frame}{Results \& discussion}
    \begin{itemize}[<+->]
    \pause
    \large
        \item Thermal resistance model works well
        \item Favorable result in panoramic view compared to ATP
        \item Uniform and repeatable texture inside the truck toughened the automatic feature detection and image stitching
        \item Amelioration in need for image processing with advanced feature detection and description
    \end{itemize}
\end{frame}


\subsection{Seasonal ARIMA}


\chapter{One Stage Detector}
After getting stuck for trying to develop a solution that would first do object detection, and then afterward classify the segmented areas separately, it was decided to try a different approach. The one stage detector  \textit{You Only Look Once} method was tried to great success. This chapter will describe how it works, sections \ref{sec:yolo} \ref{sec:mAP}, how the models were created and trained, section \ref{sec:turi}, and then an evaluation of the resulting models in section \ref{sec:turiResults}.

There are other methods of doing object detection and classification, such as \textit{Mask R-CNN} and \textit{Faster R-CNN}, however, they are outside of the scope of this project.
\section{YOLO}
\label{sec:yolo}
YOLO, or You Only Look Once, is a one stage detector approach to object detection and
 recognition \cite{YOLO1}. It takes an image and predicts both bounding boxes and the probabilities of the classes being within these bounding boxes in one run, hence its name. It was designed to be fast and usable in real-time scenarios. Since YOLO sees the entire image during training and testing, it receives contextual information about the classes and reduces error with matching background patches for objects.

 The architecture for \textit{YOLO} consist mainly as a convolutional neural network, with 24 convolutional layers and two fully connected layers. There was also a smaller neural network trained called \textit{Fast YOLO} trained which were only 9 layers which was designed to create an even faster system for object detection.

 The system first divides the input image into an  $S \times S$  grid, where each cell predicts $B$ amount of bounding boxes respectively confidence scores for the boxes. Each bounding box prediction consists of 5 separate predictions: the \textit{x, y} coordinates, represented as their position relative to the the grid cell, width and height relative to the entire image and a confidence value. The confidence values are there to reflect how certain the model is that there exists an object within the box, i.e. ideally, confidence should be zero when no object, and the intersection over union between ground truth and the predicted box if there is. Each grid cell also predicts $C$ probabilities for each class, conditioned that there's an object within the boxes.

 Intersection over union, also known as the Jaccard index, is a way of measuring similarity between sets. It can be written as
 \[
IOU(A,B) =  \frac{|A\cap B |}{|A\cup B|} =\frac{|A\cap B|}{|A| + |B| - |A \cap B|}
, \quad 0\leq IOU(A,B) \leq 1
 \]
 if $A,B$ are two finite sets. If the two sets are equal, then \[ IOU(A,A) =  \frac{|A\cap A |}{|A\cup A|} = \frac{|A\cap A|}{|A| + |A| - |A \cap A|}  = \frac{|A|}{|A|} = 1 \]

 When testing, these scores are then combined to give the class specific confidence values for each of the boxes, thus it receives both the confidence of the class being in the box, and how well the box fits the object. Figure \ref{fig:YOLO_stages} summarizes the flow in YOLO.
\begin{figure}[hbtp]
\begin{center}
\includegraphics[width = 0.75\textwidth]{./Images/YOLO_stages.PNG}
\caption{The stages of YOLO. First dives into $S \times S$ grid. Separately predicts bounding boxes with respective confidence, as well as class probability. Then, combines the two to form the final predictions. Image taken from \cite{YOLO1}.}
\label{fig:YOLO_stages}
\end{center}
\end{figure}

Comparisons done by the team working on \textit{YOLO} found that, compared to other real-time systems at the time, both \textit{Fast YOLO} and \textit{YOLO} outperformed them all with \textit{Fast YOLO} being the fastest, but \textit{YOLO} being more accurate on the \textit{PASCAL VOC} data sets \cite{PASCAL}.

Throughout the years Redmon et al. has been working on updating the design pattern of the
YOLO network. First in 2016 when they introduced \textit{YOLOv2} and then April 2018 with \textit{YOLOv3} \cite{YOLO2}\cite{YOLO3}. \textit{YOLOv2} added a few concepts to the system to make it even more fast and accurate than the earlier iteration. It removed the fully connected layers and it was now possible to train on several different input image resolutions and it could now also predict many more bounding boxes than its predecessor. \textit{YOLOv3} added some changes which improved its ability to detect small objects, with the trade off of having a bit worse performance when finding larger sized objects.

\section{Mean Average Precision}
\label{sec:mAP}
There is quite a difference when it comes to evaluating a model that does object detection,  compared to normal image classification. When doing the latter, accuracy tells you how often the model makes the correct prediction. However, with object detection it is done differently. The common metric is \textit{mean Average Precision} (mAP).  This metric makes use of the previously discussed \textit{IoU} (Intersection over Union). The IoU and the classification determines if the prediction is deemed correct. The classification obviously has to be the same as the ground truth, and the IoU has to be over a set threshold, these are known as true positives. A predicted box that does not pass the threshold is known as a false positive. If there are more than one prediction that passes the threshold, only one will get seen as a true positive, and the other as false positive. When a prediction was not made for a ground truth box, it is called a false negative \cite{turiAdvanced}.

From the false positives, true positives, and the false negatives, one can calculate recall as well as precision scores \cite{rafaelpadilla}.  Precision is how good the model is to identify only the relevant data, and can be written as
\[ \texttt{precision} = \frac{\texttt{True Positives}}{\texttt{True Positives + False positives }}\]
Recall is how good the model is to find all the relevant data, and is written as
\[ \texttt{recall} = \frac{\texttt{True Positives}}{\texttt{True Positives +False Negatives }}\]

 When a classification is made you also get a confidence value. One only calls it a prediction if the predictions confidence passes a set threshold. However, if one calculates the recall and precision for all possible thresholds, a Recall x Precision curve can be made. From this curve, one gets the mAP by taking the area under the curve.

There are different standards when calculating the mAP. The PASCAL VOC Challenge uses a mean average precision with a static threshold of 50\%
 \cite{PASCAL}. This is commonly called \textit{mean\_average\_precision\_50}. Another common standard is the one used by COCO  \cite{COCO}. This one calculates the mAP at IoU threshold from 50\%  to 95\%, with increments of 5\% every iteration, and averages them all together. This is quite strangely known as just \textit{mean\_average\_precision}. The latter mentioned method is the one this report will use for evaluation, since it puts more value on localization than the PASCAL VOC method.

\section{Turi Create}
\label{sec:turi}
%Write about the testing an implementation of Turi
Apple has a licenced toolkit for development of custom machine learning models called Turi Create \cite{turiCreate}. This was created to help developers easily implement their own ideas into an app. It includes a methods object detection.

In this project, the object detection method included in Turi Create was tested. First, one
 has to create ground truth data for every image that was trained and tested on. This was
 done using Simple Image Annotator \cite{simpleImage}, where one has to draw bounding
  boxes for the objects in the images and label them; the data output is in the \textit{.csv} format.  Turi Create uses a different format for annotations called \textit{.sframe},
  however, they provide a simple Python script which automatically does the conversion.

Turi then trains a model using a re-implementation of the TinyYOLO network. It also utilizes transfer learning by starting with an image classifier that has been trained on the imageNet dataset and then does \textit{end-to-end finetuning} to change the network to a \textit{one-stage detector}.

Several models were trained, with different amount of training data, to evaluate how many
 were needed to get a decent result. The Turi website states that at least 30 samples of
each class is needed to generate a good enough model. Hence, we tested for even lower
samples, as well as much more. Lowest amount of training images were around 50, and
highest  were around 1000. The \textit{mean average precision} was measured for each model using 200 testing images, and then plotted to create a graph showing how it
 changed, depending on the amount of training data was used. The results can be seen in section \ref{sec:turiResults}. The best performing model was then to be used in the application.

The code in appendix D.1. shows how the model was created using Turi in python.

The results could also be further inspected by drawing the newly predicted bounding boxes on top of the original image, from this one can visualize how the model actually performed. This was done using code in appendix D.2. Some of the resulting images from different models can be seen in section \ref{sec:turiResults}.

\subsection{Importing model to application}
After training and evaluating that the model lives up to a standard worth including in this
 project, it was converted to the \textit{.mlmodel} format used in Swift. When inputting a frame from the
application into the model, the model outputs a multi array containing an array for each
  \textit{object} found. These predictions have not been \textit{non-max suppressed}, since that functionality is lost during the conversion from the model used in python, to the .mlmodel format. Thus, it has to be reimplemented in the application itself.
  A non-max supression threshold of $0.5$ was used as it is considered a traditional standard \cite{nms}. The code in appendix D.3. shows how non-max suppression was implemented.

\section{Results from doing object detection with Turi}
\label{sec:turiResults}
Because of the vast amount of possible use cases it was decided to scale the objective down. The model was trained only on a few floor backgrounds.

The mean average precision from the different models that were trained were observed and can be read in table \ref{table:mAP}. These values were plotted, resulting in the graph shown in figure \ref{fig:mAPResult}.

\begin{table}[h]
\centering
\begin{tabular}{ |c|c|c| }
 \hline
 Amount of training images & Percentage of total amount & mean\_average\_precision  \\
 \hline
 $\approx 50$& 5\% & 0.17064 \\
 \hline
 $\approx 100$& 10\% & 0.30188 \\
\hline
  $\approx 140$& 15\% &  0.30342 \\
 \hline
 $\approx 185$& 20\% & 0.39269 \\
 \hline
$\approx 280$ &  30\% & 0.41588 \\
 \hline
 $\approx 370$& 40\% & 0.40595 \\
 \hline
 $\approx 415$& 45\% & 0.49917 \\
 \hline
 $\approx 450$& 50\% & 0.47397 \\
\hline
$\approx 570$ & 60\% & 0.50454 \\
\hline
 $\approx 700$& 75\% & 0.54433 \\
 \hline
$926$ & 100\% & 0.57618 \\
 \hline
\end{tabular}
\caption{Mean average precision depending on the amount of training data used.}
\label{table:mAP}
\end{table}

\begin{figure}[h]
\begin{center}
\includegraphics[width = 0.9\textwidth]{./Images/mapPlot.png}
\caption{Mean Average Precision from table \ref{table:mAP} plotted against the amount of training data used. 100\% corresponds to 926 images.}
\label{fig:mAPResult}
\end{center}
\end{figure}

\begin{figure}[h]
\begin{center}
\includegraphics[width = 0.3\textwidth, height = 0.3\textwidth]{./Images/turi100-1.png}
\includegraphics[width = 0.3\textwidth, height = 0.3\textwidth]{./Images/turi100-2.png}
\includegraphics[width = 0.3\textwidth, height = 0.3\textwidth]{./Images/turi100-3.png}
\includegraphics[width = 0.3\textwidth, height = 0.3\textwidth]{./Images/turi015-1.png}
\includegraphics[width = 0.3\textwidth, height = 0.3\textwidth]{./Images/turi015-2.png}
\includegraphics[width = 0.3\textwidth, height = 0.3\textwidth]{./Images/turi015-3.png}
\caption{Images showing how the same three images were classified differently using different models. The top row is classified using the model that was trained on all of the training set. The bottom row shows the images that were trained on only $ 15 \% $ of the training set. }
\label{fig:mAPExamples}
\end{center}
\end{figure}

When training on different amount of data it can appear as if more data gives a worse
result sometimes. That can be explained in two ways. The first explanation is that when
training the models the data was randomly split from the full training data set. That
means that sometimes there can be more "good" data and sometimes less in the set.
Some images have lots of objects in them and others have fewer. Thus, if you have a
data set with images with lots of objects in them it will likely give better results.
An important thing to mention however is that for the model to perform well it also needs
areas in the image of no object.

The other explanation is that when training, the most optimal model for that data set is
not always, if ever, returned. Take into account early stopping for example.
Sometimes the early stopping can happen on a good place and sometimes it could have
given a better result if the training session would have kept going for a little bit. We did not have a way to use an evaluation set during training, and as such, could not receive training and validation curves. Hence, we can not know how if we trained for too long or too little.

The curve from the tests, fig. \ref{fig:mAPResult}, shows no sign of saturation yet, thus even more data would
surely improve the performance. The reason for not testing with more data is because
of the difficulty of gathering more data (taking photos in different environments and
adding ground truth data) and the time it took for training.
As of now, the time for training with 926 images on a MacBook Pro 15 inch 2017 with a 2,8 GHz
Intel Core i7 processor is over 24 hours. With half the amount it takes about 17 hours. We would have trained with a GPU, however, it was at the time of training not supported on our hardware.\\

From the graph there is a small plateau between 185 to 370 images. Around that
performance (0.4 mAP) is a sort of minimum for the model to perform well.
For our 6 classes that would correlate to needing around 30-60 images per class.

From the plot in figure \ref{fig:mAPResult} one can see a sharp ascension up until around  20\% of the training set, or 185 images, were used to train the model. This corresponds to roughly 30 images per class. Afterwards, it starts to even out. Hence, the claim that at least 30 images are need to train a somewhat decent model is correct. However, as seen, more data usually generates even better result.

\newpage


% Author: Richard Glenie, Date: 31/03/18
\documentclass[referee]{rgclass}
\begin{document}
\title{Alternating renewal process with continuous-time transition intensities fit by approximate marginal likelihood}
\author{Richard Glennie}
\affil{Unversity of St Andrews}

%\abstract{
%
%}

\maketitle

\lhead{CTMC}
\rhead{R. Glennie}

% Background
\section{Introduction}

\citet{schliep2018alternating} propose an alternating renewal process with transition intensities modelled as Gaussian processes with a Mat\'ern covariance. They fit this model using MCMC and a nearest neighbour Gaussian prior, for computational feasibility. Here, we propose the same theoretical model but, diverging from \citet{schliep2018alternating}, model the transition intensities within the standard generalized additive modelling framework: this allows for the intensities to depend smoothly on covariates other than time and allows these smooths to be selected from the variety of options available, including Gaussian processes (e.g., with Mat\'ern covariance), thin plate splines,  thin plate regression splines with or without shrinkage, tensor splines, and any other basis-penalty smooths (such as B-splines or P-splines). Together with this extension, we fit the models by restricted marginal likelihood (REML) rather than MCMC; REML makes use of the Laplace approximation and is equivalent to empirical Bayes estimation. The advantage of REML is that it provides faster inference and makes larger analyses more computationally feasible.

\section{Methods}
\label{sec:mod}

An alternating renewal process \citep{norris1998markov} is a stochastic process $(X_t)_t$ that can take only two distinct values; therefore, the process alternates between these two states. Without loss of generality, we label the two states $D$ and $U$, assume that $X_0 = D$, and that $X_T = U$ where $T > 0$.  Observations from this process can then take the form $(t_i, d_i, u_i)$ for $i = 1,\ldots,n$ where $n$ is the number of times the process switches to state $D$ (setting $t_0 = 0, t_{n + 1} = T$), $d_i$ is the time spent in state $D$ during $[t_i, t_{i+1}]$ and $u_i$ is the time spent in state $U$ during $[t_i, t_{i+1}]$.

We assume this process is Markov; hence, the time spent in any state is a survival process. Let $\lambda_R:[0,T] \to [0,\infty]$ be the intensity of switching to state $R \in \{D, U\}$ over time and let $S_R(u, v) = \exp\{-\int_u^v \lambda_R(t)\;\mathrm{d}t\}$ be the survival function for state $R$, i.e., the probability of not switching from state $R$ in the time interval $[u, v]$.

 It follows that, conditional on \(X_0 = D\) and \(X_T = U\),
\begin{itemize}
\item $[t_0, d_0, u_0] = \lambda_U(t_0 + d_0)S_U(t_0, t_0 + d_0)$;
\item $[t_i, d_i, u_i] = \lambda_D(t_i)\lambda_U(t_i + d_i)S_D(t_i - u_{i - 1}, t_i)S_U(t_i, t_i + d_i)$ for $0 < i < n$;
\item $[t_n, d_n, u_n] = \lambda_D(t_n)S_D(t_n - u_{n - 1}, t_n)S_U(t_n, t_n + d_n)S_D(T - u_n, T)$.
\end{itemize}

Independence of observations follows from the Markov assumption and so $[\bm{t}, \bm{d}, \bm{u}] = \prod_{i = 1}^n [t_i, d_i, u_i]$ where $\bm{t}, \bm{d}, \bm{u}$ are vectors with $i^{\text{th}}$ entry $t_i, d_i, u_i$, respectively.

Clearly the model is specified once a form for $\lambda_R$ is specified for each state $R$. Here, we specify each intensity as a generalized additive model. Let $\log\{\lambda_R(t)\}  =  \beta_0 + \sum_{j = 1}^J \beta_jz_j(t) + \sum_{k  = 1}^K f_k(t)$ where
\begin{itemize}
\item $\beta_0$ is the intercept;
\item $z_1, \ldots, z_J$ are functions defined on \([0,T]\) which are observed --- these are covariates;
\item $\beta_1, \ldots, \beta_J$ are linear effects of the covariates;
\item $f_1, \ldots, f_K$ are non-parametric smooths.
\end{itemize}

The covariates $z_1, \ldots, z_J$ are assumed to be known for all times $t \in [0,T]$. It is likely in practice that some covariates will only be observed on state switches or some covariates may only have meaning for each $i$ and not all times. It may also be the case that a covariate is constant over $[0,T]$, but varies between individuals (e.g., sex) In both cases, we propose two solutions: (1) fit a smooth to these covariates and use the model predictions (in continuous time) as the covariate, preferably bootstrapping/posterior simulating and fitting with multiple realisations of the covariate in order to incorporate uncertainty in the reconstruction of the covariate; (2) assume the covariate is constant within each interval $[t_i, t_{i + 1}]$ for each $i$: this can either be interpreted as a piecewise constant approximation (rather than a smooth one as proposed in (1)) or can be interpreted as implying that the effect the covariate has on durations within each state is through its value at each state switch: this may not be unreasonable if information at the time of state switching has more bearing on the behaviour of the process than the covariate value between switches.

The specification of $f_1, \ldots, f_K$ is described fully in \citet{wood2017generalized}. Specifications require a smoothing penalty and either a selected or induced basis. The smoothing penalty is equivalent to a prior on each smooth specifying that smoother functions (those with a lower penalty) are more likely to reflect the true underlying process. For example, a common smoothing penalty is the integrated squared second derivative $\mathcal{P}(f_k) = \rho_k\int \{f_k''(u)\}^2 \;\mathrm{d}u$ with smoothing parameter $\rho_k > 0$. A basis can be deduced from the chosen penalty or specified such that $f_k(u) = \sum_{b = 1}^B \alpha_{k,b}\phi_{k,b}(u)$ for parameters $\bm\alpha_k = (\alpha_{k,1}, \ldots, \alpha_{k,B})$ and known basis functions $\phi_{k,1}, \ldots, \phi_{k,B}$. Notice, we allow smooths to be defined over variables other than time, such as time-varying covariates.

Ultimately, we assume that the smooth formulation reduces to the specification that $\bm{\alpha}_k$ has a (possibly improper) multivariate Normal prior with mean zero vector and (possibly singular) precision matrix $\rho_k\bm{S}_k$ where $\bm{S}_k$, termed the smoothing penalty matrix, is a $B \times B$ matrix. For the squared second derviative penalty $S$ has $(i,j)^{\text{th}}$ entry $\int \phi_{k,i}''(u)\phi_{j,k}''(u)\;\mathrm{d}u$. Let $\bm\rho$ be the vector of smoothing parameters, which may contain more or less than $K$ as each smooth, in general, can have no penalty or more than one penalty.

Given this specification for the intensities, let $\bm\theta$ be the vector of linear effects $\beta_0, \ldots, \beta_J$ and smoothing parameters $\bm\rho$.  The parameters such as $\alpha_{k, b}$ are treated as random effects and inference on $\bm\theta$ is based on the marginal likelihood. The full likelihood is a penalized likelihood; for example, where each smooth $f_k$ has a single penalty with smoothing penalty matrix $S_k$, the full likelihood has the form $\mathcal{L}(\bm\theta, \bm\alpha_1, \ldots, \bm\alpha_K) = [\bm{t}, \bm{d}, \bm{u}]\exp\left\{-\sum_{k = 1}^K \rho_k\bm\alpha_k^\intercal S_k \bm\alpha_k\right\}$, the likelihood of the parameters given the data multiplied by an improper prior.

The marginal likelihood is then $\mathcal{L}_m(\bm\theta) = \int \mathcal{L}(\bm\theta, \bm\alpha_1, \ldots, \bm\alpha_K) \;\mathrm{d}\bm\alpha_1, \ldots,d\bm\alpha_K$. The marginal likelihood can be approximately calculated using the Laplace approximation to perform the required integration. Point estimates can then be obtained by maximising the marginal likelihood and variance in these point estimators can be estimated using the generalized delta method or posterior simulation \citep{wood2016smoothing, wood2017generalized}, incorporating uncertainty in both linear effects and smoothing parameters.

The model is implemented in R \citep{R} using the Template Model Builder package \citep{TMB} to perform the Laplace approximation. The survival functions are approximated by Riemann sums.

 \subsection{AIC calculation}

For the smooth components in the model, the effective degrees of freedom were calculated to use for AIC calculation. Following from \citep[][Section 5.4.2]{wood2017generalized} we write the effective degrees of freedom as:
\begin{equation}
\text{trace}\left(\mathbf{F}\right) = \text{trace}\left((\mathbf{X}^\intercal \mathbf{X} + \lambda \mathbf{S})^{-1} \mathbf{X}^\intercal \mathbf{X}\right),
\end{equation}
where $\mathbf{F}$ is the influence matrix for the smooth. We need to calculate $\mathbf{F}$ for each part of the model (dive and surface), then add-in the degrees of freedom from fixed effects to get a total degrees of freedom for the model. This can then be used in the usual AIC formula.
% Reference the fancy version from Wood 2016/2017 and Kass and Steffey

\bibliographystyle{chicago}
\bibliography{ctmc_refs.bib}

\end{document}