%!TEX root = ../gronskiy_phd_thesis.tex
\chapter[Does the Free Energy Define the Model Behavior?]{Does the Free Energy Define \\ the Model Behavior?}
\label{ch:smbp_and_rem}

\hfill
\begin{minipage}[t]{.75\textwidth}
\textit{``When I see a bird that walks like a duck and swims like a duck and
  quacks like a duck, I call that bird a duck.''} \\
  \hrule
  \vspace{.2cm}
  \hfill
  \textsc{---  James Whitcomb RILEY} (attr.)
\end{minipage}

\section{Introduction}

\subsection{Motivation}

Random combinatorial optimization problems exhibit a highly complex structure
with a spin glass behavior~\citep{Mezard87,SK:CG:MV:Science1983}. Optimization
algorithms for these problems are slowed down by fluctuations in the problem
instances when they search for solutions with low costs. Conceptually, we
consider an optimization algorithm as a mapping from an input space of random
instances to an output space of solutions and such algorithms should sample
``typical'' solutions from appropriate posterior distributions. In this chapter,
we concentrate on maximum entropy sampling principles guided by Gibbs
distributions to study information theoretic properties of random combinatorial
optimization problems and their search landscape.  Analytical computation of
free energy, entropy and other macroscopic thermodynamical properties enables us
to understand the solution structure of a large system, but~--- as already
clarified in~Chapter~\ref{ch:free_energy},~--- this goal has been
known to be notoriously difficult and challenging from a mathematical
standpoint~\citep{talagrand03}.

While we solved this problem for specific cases (Chapter~\ref{ch:free_energy})
with the purpose of applying in to robust optimization (Chapters~\ref{ch:gen_appch}
and~\ref{ch:mst}), here we will be interested in a more general consequence of
such results: a relation between the Random Energy Model (REM;
see~\citealp{derrida81}) and the Sparse Minimum Bisection Problem (sMBP;
see~Section~\ref{sec:free_mbp-problem}).

% Why are we interested in the relations between REM and sMBP? \\

Why are the relations between REM and sMBP of interest?  The REM does not
introduce any statistical dependencies between solutions. Therefore,
optimization algorithms have to exhaustively inspect all exponentially many
solutions of REM to find the one with minimal costs. Sparse Minimum Bisection
introduces correlations between solutions but they are asymptotically so weak
that they do not change the free energy. Since the free energy is the moment
generating function of the Gibbs distribution we hypothesize that the Gibbs
distributions of both problems are equivalent in terms of Kullback-Leibler
divergences. If this claim would hold then we would not be able to efficiently
search for low cost solutions of sMBP.

\subsection{Contributions and Outline of the Chapter}
\label{sec:smbp_and_rem_contribs}

In this chapter, we revisit the idea of characterizing structural
information in solutions for combinatorial problems by
information theoretic properties.

More specifically:
%
\begin{itemize}
  \item we revisit results on the asymptotic behavior of the free
    energy~(Chapter~\ref{ch:free_energy}) on obtaining bounds on the free energy of
    solutions for the sMBP with random edge weights;

  \item these results reveal a remarkable phenomenon that the
    free energy of sMBP behaves very similarly to that of REM. Specifically, we
    show that the free energy of sMBP with random edge weights exhibits phase
    transitions equivalent to Derrida's REM;

  \item in order to deeper understand this observation and solution structure for
    dependent and independent solutions, we then make and prove statements about
    various ways sMBP and REM can be quantitatively related to each other: we show
    that the Kullback-Leibler divergence between Gibbs distributions induced by sMBP
    and REM are bounded, but not zero, which allows to make a conjecture about their
    complexity relations.
\end{itemize}

The chapter is organized as follows. As usual, we start with describing some of
related work in Section~\ref{sec:smbp_rem_related_work}. We then present and
discuss our results about the similar behavior of REM and sMBP in
Section~\ref{sec:smbp_rem_similar}. We speculate on the ways to interpret these
results in Sections~\ref{sec:mbp_and_rem_how_similar}
and~\ref{sec:rem_conclusion}.

\section{Background and Related Work Overview}
\label{sec:smbp_rem_related_work}

Information theory, statistical mechanics and combinatorial optimization in
large disordered systems have been disciplines enjoying several waves of
intensive research. The first wave, associated exclusively with the statistical
mechanics, was marked by the works of~\citet{sk75spinb} or~\citet{derrida81} on
mean-field models of spin glasses. For the reasons stated in the introduction,
we concentrate on Derrida's solvable REM. In short, REM is the simplest
example of a disordered system, whose configurations have i.i.d. energies and,
therefore, are not efficiently ``searchable''. It will become important in the
rest of the chapter that REM reflects the situation with no stochastic
dependencies between solutions. This work inspired several continuations, of
which we can mention, e.g., \citep{derrida86} as a generalization of REM,
or~\citep{Aizenman1987} as exact solution of Sherrington-Kirkpatrick
model~\citep{sk75spin}.

\index{Traveling Salesman Problem}
\index{TSP|see{Traveling Salesman Problem}}
The second wave of interest was associated not exclusively with statistical
mechanics, but also researched its connection to combinatorial optimization.
Inspired by the work of Derrida, \citet{mezard84tsp} considered the Traveling
Salesman Problem (TSP) as a large disordered system which seeks to optimize its
energy defined by respective \textit{Hamiltonians}\index{Hamiltonian} (see
Section~\ref{sec:background_disordered_systems}). This approach to view a
combinatorial optimization problem from the statistical mechanics prospective
turned out to be extremely fruitful: we recommend the book~\citep{LUCZAK1994} as
a good overview of the results. We should also mention here the
work~\citep{Auffinger2014} who studied algorithmic complexity from the
statistical mechanics viewpoint.
\index{Complexity}
\index{Algorithmic complexity}

Finally, in the last two decades, many attempts have been made to systematize
approaches traditionally used in statistical mechanics and render them rigorous
in a mathematical sense. Here, we point out the work by
\citet{Bovier2002FreeEnergyFluct}, as well as extensive
reviews by \citet{talagrand03,bovier2012statistical}.

\section{Comparison of REM and sMBP}

\subsection{Random Energy Model (REM)}

\index{Random Energy Model}
\index{REM|see{Random Energy Model}}

The REM introduced by~\citet{derrida81} is a model $\mathcal{P}^\mathrm{rem} =
(\mathcal{X}, \C^\mathrm{rem}, R^\mathrm{rem})$
\nomenclature[F, 50]{$(\mathcal{X}, \C^\mathrm{rem}, R^\mathrm{rem})$}{definition of REM\nomnorefeq}%
where the following conditions apply:
\begin{enumerate}
  \item Number of solutions (in the original terminology,
    \textit{configurations}) equals
    \begin{equation}
      |\C^\mathrm{rem}| = 2^K.
    \end{equation}
  \item Here, the data source $X \in \mathcal{X}$ is a vector of $2^K$ Gaussian
    random variables (for the notation, see
    Definition~\ref{def:optimization_problem_definition}), and all solutions
    $c_i
    \in \C^\mathrm{rem}$ carry costs (in the original terminology,
    \textit{Hamiltonians} or \textit{energy levels})
    \begin{equation}
      R^\mathrm{rem}(c_i, X) = X_i, \;\; \text{where} \;\; X_i \sim \mathcal{N}(0, \sigma^2)
    \end{equation}
    \item The costs $X_i$ are i.i.d.
\end{enumerate}

As Derrida noted in his paper, ``the third property is specific to this model.
It simplifies the model enough to allow us to solve it exactly''. While it is
true for REM, such independence is not characteristic for the most of the
models. The next section which discusses dependencies in sMBP.

\subsection{Similar Behavior of REM and sMBP}
\label{sec:smbp_rem_similar}
Earlier, we introduced the sparsity constraint on $d$ in
Theorem~\ref{thm:sparse_mbp_tight_bound} because without it, the stochastic
dependency between two random solutions for original MBP~\citep{garey79} is very
high. Indeed, in original MBP, any two bisections \textit{always} share edges.
By introducing $d$ we: a) substantially reduce such dependency, but on the other
hand b) do not eliminate it at all, like in REM~\citep{derrida81}.

But by introducing sparsity, didn't we essentially \textit{transform} MBP into
REM? E.g. one can observe that for classical dependency-free REM, the free
energy asymptotics looks just the same, in particular exhibits the same phase
transition and same phase shapes:

\begin{theorem}[adapted formulation from~\citep{talagrand03}] \label{thm:rem}
  Assume $m = 2^K$ is the number of configurations for the REM model with
  Gaussian cost values, with parameters $\mathcal{N}(0, \tau^2)$. Then the
  free energy rate is (asymptotically in $n \to \infty$) equal to
    \begin{equation} \label{eq:talagrand_rem_with_tau}
      \lim_{n \to \infty} \frac{\E[\log Z]}{\log m}=
      \left\{ \begin{array}{ll}
      \frac{\beta^2 \tau^2}{2 \log m} + 1 &
      \beta < \sqrt{2 \log m}/\tau,\\
      \frac{\beta \tau \sqrt{2}}{\sqrt{\log m}} & \beta \ge \sqrt{2 \log m}/\tau.
      \end{array}
      \right.
    \end{equation}
\end{theorem}
We note that Talagrand formulated it in a more general setting~\citep[cf.][Prop.
1.1.3]{talagrand03}, which is adapted here for clarity. Choosing $\tau = \sigma
\sqrt{N}$ and applying $\beta$ rescaling from~\ref{def:cts}, we arrive at an
equivalent formulation: for Gaussian cost values with parameters $(0, \sigma^2
N)$, we derive
\begin{equation} \label{eq:talagrand_rem_adapted}
  \lim_{n \to \infty} \frac{\E[\log Z]}{\log m}=
  \left\{
    \begin{array}{ll}
      1+\frac{\hat \beta^2\sigma^2}{2}, &
        \hat \beta< \frac{\sqrt{2}}{\sigma},\\
      \hat \beta \sigma \sqrt{2}, &
        \hat \beta \ge \frac{\sqrt{2}}{\sigma}
    \end{array}
  \right.
\end{equation}
\myremark For non-centered cost values, the necessary
correction similar to the one of~\eqref{eq:sparse_mbp_tight_bound} should be
made on the left-hand side which is trivial.

We are now going to sketch an answer to the following question: how much does
sMBP look like REM? We give the following result and then discuss it. First,
let's make some definitions.

\begin{definition}
  For an sMBP setting stated in Theorem~\ref{thm:sparse_mbp_tight_bound},
  we will call equivalent such a REM, for which the number
  of configurations is equal to $m$ and the cost values are Gaussian with
  parameters $(\mu N,
  \sigma^2 N)$.
\end{definition}
For convenience of the following explanation, let us denote the random source
behind such a REM as $Y$ (analogically to $X$ in case of sMBP). Hence for REM,
the cost values $R(c, Y) \sim \mathcal{N}(\mu N, \sigma^2 N)$ and all are
independent. We denote the respective Gibbs distributions $\psmbp_\beta(c | X)$ and
$\prem_\beta(c | Y)$.
\index{Kullback-Leibler divergence}
\begin{theorem}\label{thm:kl_divergence}
  The rate of the KL-divergence between configurations' Gibbs distributions for sMBP
  and equivalent REM is non-zero and exhibits a phase transition.
  \begin{equation}
    \frac{\Expct_{X, Y}[\KL(\prem_\beta \| \psmbp_\beta)]}{\log m} =
      \left\{
        \begin{array}{ll}
          \hat\beta^2 \sigma^2 &
            \hat \beta< \frac{\sqrt{2}}{\sigma},\\
          \hat \beta \sigma \sqrt{2}, &
            \hat \beta \ge \frac{\sqrt{2}}{\sigma}
        \end{array}
      \right.
  \end{equation}
  \nomenclature[F, 50a]{$\prem_\beta(c "| X)$}{Gibbs distribution of REM model}%
  \nomenclature[F, 50c]{$\psmbp_\beta(c "| X)$}{Gibbs distribution of sMBP}%
\end{theorem}

\paragraph{Proof}
In the first part of the proof, we will omit the expectation for the sake of
brevity.
%
\begin{align}
\MoveEqLeft \KL(\prem_\beta \| \psmbp_\beta) = \sum_c \prem_\beta(c\vert Y) \log
  \frac{\prem_\beta(c\vert Y)}{\psmbp_\beta(c\vert X)} = \notag \\
  &= \sum_c \prem_\beta(c\vert Y)
    \left(
      \log \frac{e^{-\beta \Rrem(c,Y)}}{\Zrem(Y)} \right. \notag \\
      &\qquad\qquad\qquad\qquad
      \left. - \log \frac{e^{-\beta \Rsmbp(c,X)}}{\Zsmbp(X)}
    \right) \notag \\
  &=  \sum_c \prem_\beta(c\vert Y)
    \left(  \vphantom{\sum}
      \log e^{-\beta \Rrem(c,Y)}   \right. \notag \\
      &\qquad\qquad\qquad - \log e^{-\beta \Rsmbp(c,X)} \notag \\
      &\qquad\qquad\qquad \left. - \log \Zrem(Y) + \log \Zsmbp(X)
      \vphantom{\sum}\right) \notag \\
  &= -\beta \sum_c \prem_\beta(c\vert Y)
    \Bigl( \Rrem(c,Y) - \Rsmbp(c,X) \Bigr) \notag \\
      &\qquad\qquad\qquad  - \log \Zrem(Y) + \log \Zsmbp(X)
\end{align}
Returning to the expectation $\Expct_{X,Y}$ and recalling that all the
sMBP-related terms depend on $X$ and all the REM-related terms depend on $Y$, we can continue:
\begin{align}
  \MoveEqLeft -\beta \Expct_{X,Y} \biggl[\sum_c \prem_\beta(c\vert Y)
    \Bigl( \Rrem(c,Y) - \Rsmbp(c,X) \Bigr) \biggr] \notag \\
      &\qquad\qquad
        \underbrace{
            {} - \Expct_Y [\log \Zrem(Y) ]
            + \Expct_X [\log \Zsmbp(X)]
          }_{
          \text{
            cancel out due to Thm.~\ref{thm:sparse_mbp_tight_bound} and \ref{thm:rem}}
          } \notag \\
  &= -\beta
    \Bigl( \Expct_{Y} \Bigl[\sum_c \prem_\beta(c\vert Y) \Rrem(c,Y) \Bigr] \notag \\
        &\qquad\qquad - \Expct_Y
          \underbrace{\sum_c \prem_\beta(c\vert Y)}_{1} \cdot
          \underbrace{\vphantom{\sum_c}\Expct_X \Rsmbp(c,X)}_{\mu N} \Bigr) \notag \\
  &= \beta \mu N + \beta \Expct_Y \Bigl[\frac{d}{d\beta} \log \Zrem(Y)\Bigr].
\end{align}
By the argument of dominated convergence theorem, we can under mild conditions
interchange expectation and differentiation, which together
with~\eqref{eq:talagrand_rem_with_tau} and~\eqref{eq:talagrand_rem_adapted}
leads to
\begin{equation}
  \Expct_{X, Y}[\KL(\prem_\beta \| \psmbp_\beta)] =
    \left\{
      \begin{array}{ll}
        \hat\beta^2 \sigma^2 \log m &
          \hat \beta< \frac{\sqrt{2}}{\sigma},\\
        \hat \beta \sigma \log m \sqrt{2}, &
          \hat \beta \ge \frac{\sqrt{2}}{\sigma},
      \end{array}
    \right.
\end{equation}
which completes the proof of theorem.
\QEDA

\subsection{Consequences of Similar Behavior of REM and sMBP}

We now discuss this result. There are several observations to be made about the
whole line of research reflected in
Theorems~\ref{thm:sparse_mbp_tight_bound},~\ref{thm:rem}
and~\ref{thm:kl_divergence}.

\begin{figure}[th!]
  \centering
  \begin{subfigure}[b]{.48\textwidth}
      \includegraphics[width=\linewidth]{figures/ch_smbp_and_rem/different_solution_overlaps}
      \caption{Solution to an sMBP problem}
      \label{fig:ch_rem_smbp_illustration-0}
  \end{subfigure}
  \hfill
  \begin{subfigure}[b]{.48\textwidth}
      \includegraphics[width=\linewidth]{figures/ch_smbp_and_rem/different_solution_overlaps_1}
      \caption{No overlap at all}
      \label{fig:ch_rem_smbp_illustration-1}
  \end{subfigure}
  \\[.5cm]
  \begin{subfigure}[b]{.48\textwidth}
      \includegraphics[width=\linewidth]{figures/ch_smbp_and_rem/different_solution_overlaps_2}
      \caption{Vertex overlap, but no edge overlap}
      \label{fig:ch_rem_smbp_illustration-2}
  \end{subfigure}
  \hfill
  \begin{subfigure}[b]{.48\textwidth}
      \includegraphics[width=\linewidth]{figures/ch_smbp_and_rem/different_solution_overlaps_3}
      \caption{Only edge overlap counts}
      \label{fig:ch_rem_smbp_illustration-3}
  \end{subfigure}
  \\[.5cm]
  \caption{Illustration of various types of overlaps. Only case \textbf{(d)}
  contributes to statistical dependence between costs of solutions. Carefully
  computing edge overlap can make a huge step forward in understanding higher
  moments of $\log Z$ (in Lemma~\ref{lem:expct_d_asymptotics} we computed only
  expected value).}
  \label{fig:ch_rem_smbp_illustration}
\end{figure}

First, Theorems~\ref{thm:sparse_mbp_tight_bound},~\ref{thm:rem}
and~\ref{thm:kl_divergence} can be interpreted as follows: they yield the
similarity of both problems in terms of macroscopic thermodynamical properties
like free energy rate, but at the same time they convey their difference in
terms of KL-divergence rate: by definition of $\KL$, it measures the amount of
information one might gain (or lose) by assuming $\prem$ instead of $\psmbp$ or
vice versa. The fact that the expected KL-divergence rate is non-zero allows us
to say that sMBP and REM are still different in terms of their distributions.

Second, and probably most importantly, due to the same line of reasoning as in
Theorem~\ref{thm:kl_divergence}, we can conclude that a KL-divergence between
every pair $({\prem}', {\prem}'')$ of independent (that is, with independent
$Y'$ and $Y''$) equivalent REMs is identical to that of $(\prem, \psmbp)$. This
highlights the fact that for full understanding of relations (similarities and
differences) between them one needs to properly quantify this difference in
terms of higher moments and not only \textit{expectation} of KL-divergence rate.

It is important to realize that the higher moments of the KL-divergence can be
most likely controlled by the higher moments of $\log Z$, i.e. $\Var[\log Z]$
and so on.

The necessary understanding of the behavior of $\Var[\log Z]$, in turn, can be
reached via careful computing of higher moments of edge overlap $D$ of the two
solutions $c_1$ and $c_2$ (illustrated in
Figure~\ref{fig:ch_rem_smbp_illustration}): one can see this, for example,
from the proof of Theorem~\ref{thm:sparse_mbp_tight_bound}, where we have
\begin{equation*}
  \Var Z \sim (\Expct Z)^2 \bigl( \sigma^2 \beta^2 \Expct_\D D \bigr)
\end{equation*}
as an intermediate result. There exists evidence of a possibility to use
Taylor expansions to express higher moments of $\log Z$ via those of $Z$.

\section{Comparison of REM and non-sparse MBP}
\label{sec:mbp_and_rem_how_similar}

Interesting question arises: can we say anything about non-sparse (i.e. case
when $d \not \ll n$) MBP? Note that in Section~\ref{sec:free_energy_in_general_case}
of Chapter~\ref{ch:free_energy} we made a conjecture which is backed up by
extensive simulations, which we restate here:

\newtheorem*{adhocconj}{Conjecture~\ref{ad-hoc}}
\begin{adhocconj}[see Section~\ref{sec:free_energy_in_general_case}]
Consider a class of combinatorial optimization problems complying with Common
Theorem Setting, weights $W_i$ having mean $\mu$ and variance $\sigma^2$. Then
the free energy satisfies
\begin{equation*}
    \lim_{n\to \infty} \frac{\E[\log Z(\beta, X)] +\hat \beta \mu \sqrt{N\log m}}{\log m}
    =
    \begin{cases}
        1 + \alpha^2\frac{\hat{\beta}^2\sigma^2}{2},
            &\hat{\beta} < \frac{\sqrt{2}}{\alpha\sigma} \\
\alpha\hat{\beta}\sigma\sqrt{2}, &\hat{\beta} \geq \frac{\sqrt{2}}
{\alpha\sigma}
    \end{cases}
\end{equation*}
for some $\alpha \ge 1$,
where $\alpha$ is well approximated by
\begin{equation*}
    \alpha
      = \sqrt{\frac{\E_X\Var_{\mathcal{D}} R(c,X)}{\E_{\mathcal{D}}\Var_X R(c,X)}}
      = \sqrt{\frac{\E_X\Var_{\mathcal{D}} R(c,X)}{N\sigma^2}}
\end{equation*}
\end{adhocconj}

It is instructive to note here that, loosely speaking, parameter $\alpha$
represents the ratio variability across solutions (nominator) and the
variability inside each solution (denominator). The higher the dependence is, the
less variability across solutions exists.

Apparently $\alpha = 1$ corresponds to a case of no dependencies (sMBP), while
lower alpha corresponds to higher levels of dependencies. And again, to further
back up this conjecture, one has to develop the understanding of
$\Var_{\mathcal{D}} R(c, X)$, i.e. variance of costs for uniformly chosen
solutions, which boils down to taking overlaps under control (computing
$\Var_\mathcal{D} [D]$ analogically to $\Expct_\mathcal{D}[D]$ as we did in
Lemma~\ref{lem:expct_d_asymptotics}.

\section{Discussion and Conclusion}
\label{sec:rem_conclusion}

The geometry of random combinatorial optimization problems is often
characterized by exponentially many local minima which prevent an
efficient search for low cost solutions. In this chapter, we have
studied random instances of sparse Minimum Bisection Problem which
exhibit a statistical behavior very similar to that of the Random Energy
Model. This similarity between the Gibbs distributions for REM
instances and for sMBP instances is documented by identical values for
the free energies and for the Kullback-Leibler divergences between
pairs of distributions.

Furthermore, this equivalence also suggests that the computational complexity of
REM and sMBP might be the same and, consequently, random instances of sMBP might
not be efficiently optimized due to the lack of search information as in REM. As
a future research direction we plan to analyze the finite $\log m$ corrections
to estimate the convergence rate of the free energy toward its asymptotic limit.
Another open question remains if there exist REM instances that are arbitrarily
close (w.r.t. KL-divergence) to sMBP instances in the asymptotic limit,
explaining why we cannot find efficient optimization schemes for sparse Minimum
Bisection.

For answering the last question, we have highlighted the importance of developing a better
understanding of the higher moments of the solutions overlap
(Figure~\ref{fig:ch_rem_smbp_illustration}), since the proofs we gave earlier
yield that controlling the dependency between the solutions gives us the essential
information about the problem's (dis)similarity from REM, where such dependencies
don't exist at all.


\section{Introduction} \label{sec:intro}
%
The untyped lambda calculus presents itself as an interesting
target for synthesis.
%
On one hand, it is described by a very simple grammar
defining a very unstructured program space for which it is
hard to guide a search.
%
On the other hand, this same lack of structure makes it
unclear what should be used as behavioral constraints.
%
In this work we explore the possibility of synthesizing
expressions in the untyped lambda calculus focusing mainly
on the second problem.
%
More specifically, we focus on the problem of synthesizing
functions over data encodings.

Data encodings in the lambda calculus are interesting
because the same basic construct, lambda abstractions, is
used to describe both functions and data.
%
This produces a tension which makes it hard to reason about
lambda expressions because the distinction between data and
function gets blurred.

We recognize this as one of the main problems making it
hard to specify behavioral constraints for lambda
expressions.
%
Because of this lack of distinction, it is not enough, in
general to just specify the extensional behavior of
functions, and we examine the possibility of constraining
their evaluation.
%
One of our key insights is that using free variables give us
enough expressiveness to talk about the evaluation of
functions.

Based on this idea we note that by looking at the
introduction and elimination rules of data types we are able
to specify their meaning precisely by giving a specification
of their behavior in relation to some functions operating on
them.
%
We call this a \emph{co-specification} of an encoding.
%
This opens the possibility of automatically deriving the
encoding for some data type as a product of synthesis in a
process we dubbed \emph{co-synthesis}.

Finally, we proved these ideas in practice by implementing a
prototype enumerative synthesizer.
%
Despite its naive implementation we were able to apply it in
practice to synthesize a series of interesting functions.
%
Using this technique, our implementation was able to derive
the standard encoding for booleans.
%
The code for our implementation can be found at \\
\href{https://github.com/DavidThien/elsa}
{\texttt{https://github.com/DavidThien/elsa}}.

% ; the meaning for a function only exists
% in the context of a specific encoding of your chosen terms.

% This makes it hard to

% defining a very particular search space.
% %

% for two major reasons:
%   \begin{enumerate}
%     \item The grammar describing the language is very small, meaning that
%       small optimizations in the search process are very important to
%       constrain the search space as much as possible
%       %% TODO: This isn't quite the right description
%     \item Untyped lambda calculus uses the exact same grammar to describe
%       both terms, and functions; the meaning for a function only exists
%       in the context of a specific encoding of your chosen terms.
%   \end {enumerate}
% The second reason in particular is interesting, because it begs the
%   question of creating a specification for the behavior of functions
%   independently of a given encoding.
% This would allow users of the synthesizer to either pre-specify a
%   particular encoding of terms they are interested in synthesizing
%   functions for, or to specify the behavior of terms in the context of
%   function applications, and have the synthesizer synthesize both
%   an encoding for the terms, and the functions requested.
% We call synthesis of terms and functions from a single unified encoding
%   \emph{co-synthesis}.

% \subsection{Grammar}

% The grammar for untyped lambda calculus is simply defined as:
%   \begin{grammar}
%     <expr> ::= x
%       \alt <expr> <expr>
%       \alt $\lambda$ x <expr>
%   \end{grammar}
%   where $x$ is a variable.
% Note that variables need not necessarily be bound by the immediately
%   preceeding lambda to be referenced.
% For our purposes, variables only need be bound in the ultimate term
%   that we eventually produce, as we can assume without loss of
%   generality that there is no outside environment that we reference
%   for synthesized terms or functions.

% This grammar is very compact, which means that our search space tends to
%   blow up because of the necessary depth of the search, rather than
%   because the number of possible terms enumerable at a given depth
%   is large.

% \subsection{Evaluation Strategy}

% The meaning of a term in Lambda calculus, as in other programming languages,
%   depends on the evaluation strategy used to assign meaning to that term.
% For our purposes, we use normal order evaluation.
% Normal order evaluation evaluates the leftmost outtermost reducible
%   expression first.
% Although this will generate different terms than another strategy like
%   applicative order, there is nothing specialized about this synthesis
%   that requires normal order to be used above anything else.
% We have simply chosen normal order for consistency.


\documentclass{beamer}

\usetheme{Madrid}
\usecolortheme{default}

\definecolor{THUpurple}{RGB}{102,8,116}

\usepackage{amsmath}
\usepackage{mathtools}
\usepackage{caption}
\usepackage{listings}
\usepackage{lmodern}
\usepackage{xcolor}
\lstset{language=Python,keywordstyle={\bfseries \color{blue}}}
\usepackage{pdfpages}
\usepackage{makecell}
\usepackage[EULERGREEK]{sansmath}
\usepackage{float}
\usepackage{hyperref}
\usepackage{tikz}
\usetikzlibrary{shapes,arrows,positioning}
\usepackage[subrefformat=parens]{subcaption}
\usepackage[none]{hyphenat}
\usepackage[binary-units=true,per-mode=symbol]{siunitx}
\usepackage{CJK}
\usepackage{textcomp}
\usepackage{adjustbox}
\usepackage{pgfplots}
\usepackage{bm}
\usepackage{tablefootnote}
\DeclareMathOperator{\erf}{erf}
\usefonttheme[onlymath]{serif}

\newcommand{\dd}{\mathrm{d}}
\newcommand{\mev}{\mathrm{MeV}}
\newcommand{\gev}{\mathrm{GeV}}

\setbeamercolor{structure}{fg=THUpurple}
\setbeamersize{text margin left=10mm,text margin right=10mm}
% \setlength{\belowcaptionskip}{-2mm}
\title[Waveform Analysis]{Accurate PMT waveform analysis \\ with fast Bayesian matching pursuit}
\date[JUNO]{August 13, 2021}

\AtBeginSection[]
{
    \begin{frame}[noframenumbering]
        \frametitle{Outline}
        \thispagestyle{empty}
        \tableofcontents[currentsection]
    \end{frame}
}

\begin{document}
\captionsetup[figure]{labelfont={bf},name={Fig}}
\setbeamertemplate{frametitle}
{\begin{beamercolorbox}[wd=\paperwidth]{frametitle}
    \strut\hspace{0.5em}\insertframetitle\strut
    \hfill
    \raisebox{-2mm}{\includegraphics[width=1cm]{img/180px-Junologo.jpg}}
\end{beamercolorbox}
}
\tikzstyle{every picture}+=[remember picture]

\begin{CJK*}{UTF8}{gbsn}
\author[Dacheng Xu]{Dacheng~Xu~(徐大成) \and Erjin~Bao~(宝尔金) \and Yiyang~Wu~(武益阳) \and Benda~Xu~(续本达) \and Yu~Xu~(徐宇) \and Geliang~Zhang~(张戈亮) et.al \\ [4mm] \includegraphics[height=2cm]{img/Tsinghua_University_Logo.png}}

\frame{\titlepage}

\begin{frame}[noframenumbering]
\frametitle{Outline}
\thispagestyle{empty}
\tableofcontents
\end{frame}

\section{Motivation}

\begin{frame}
\frametitle{Motivation}
\begin{columns}
\column{0.425\textwidth}
\begin{figure}
    \centering
    \includegraphics[width=1.0\linewidth]{img/10-Figure7-1.png}
    \caption{An Event in JUNO Detector\cite{zhu_method_2019}}
\end{figure}
\column{0.575\textwidth}
\begin{figure}
    \centering
    \resizebox{\textwidth}{!}{\input{img/junowave.pgf}}
    \caption{A PMT Waveform}
\end{figure}
\end{columns}
\begin{block}{}
Waveform analysis, which means extracting time and charge information from PMT waveforms, is the bedrock of subsequent analysis such as event reconstruction.
\end{block}
\end{frame}

\section{Dataset}

\begin{frame}
\frametitle{Simulation setup: Time profile}
\begin{figure}
    \centering
    \resizebox{0.6\textwidth}{!}{\input{img/profile.pgf}}
    \caption{Time Profile of Events}
\end{figure}
\begin{align*}
    \phi(t) &= \mathcal{N}(t|\sigma_l^2)\otimes \mathrm{Exp}(t|\tau_l) \\
    &= \frac{1}{2\tau_l} \exp\left(\frac{\sigma_l^2}{2\tau_l^2}-\frac{t}{\tau_l}\right) \left[1 - \erf\left( \frac{\sigma_l}{\sqrt{2}\tau_l} - \frac{t}{\sqrt{2}\sigma_l} \right)\right]
\end{align*}
\end{frame}

\begin{frame}
\frametitle{Simulation setup: Single PE response}
\begin{figure}
    \centering
    \resizebox{0.6\textwidth}{!}{\input{img/spe.pgf}}
    \caption{Single PE response\cite{jetter_pmt_2012}}
\end{figure}
\begin{align*}
  V_\mathrm{PE}(t) &= V_{0}\exp\left[-\frac{1}{2}\left(\frac{\log(t/\tau_\mathrm{PE})}{\sigma_\mathrm{PE}}\right)^{2}\right]
\end{align*}
\end{frame}

\begin{frame}
\frametitle{Data input \& output}
\begin{columns}
\column{0.5\textwidth}
\begin{figure}
    \centering
    \resizebox{1.0\textwidth}{!}{\input{img/wave.pgf}}
    \caption{Input Waveform (Pedestal free)}
\end{figure}
\column{0.5\textwidth}
\begin{figure}
    \centering
    \resizebox{1.0\textwidth}{!}{\input{img/charge.pgf}}
    \caption{Output Time and Charge $\hat\phi(t)$}
\end{figure}
\end{columns}
\begin{align*}
  \tilde{\phi}(t) &= \sum_{i=1}^{N_{\mathrm{PE}}} q_i \delta(t-t_i), \ N_{\mathrm{PE}}\sim \mathrm{Poisson}(\mu) \\
  w(t) &= \tilde{\phi}(t) \otimes V_\mathrm{PE}(t) + \epsilon(t) = \sum_{i=1}^{N_\mathrm{PE}} q_i V_\mathrm{PE}(t-t_i) + \epsilon(t)
\end{align*}
\end{frame}

\section{Evaluation criteria}

\begin{frame}
\frametitle{Evaluation criteria}
$\tilde{\phi}(t)$ (simulation result) is an approximation of $\phi(t)$ (time profile).

$\hat{\phi}(t)$ (reconstruction result) should be consistent with $\tilde{\phi}(t)$.

Several evaluation criteria are needed.
\begin{block}{}
\begin{equation*}
    \hat{\phi}(t) \leftrightarrow \tilde{\phi}(t)
\end{equation*}
\end{block}
\begin{itemize}
    \item Residual sum square between $\hat{w}(t)$ and $w(t)$
    \item Wasserstein distance between $\hat{\phi}(t)$ and $\tilde{\phi}(t)$
\end{itemize}
\end{frame}

\section{Fourier deconvolution}

\begin{frame}
\frametitle{Fourier deconvolution}
\begin{align*}
  \mathcal{F}[w] &= \mathcal{F}[\tilde{\phi}]\mathcal{F}[V_\mathrm{PE}] + \mathcal{F}[\epsilon],\Rightarrow \hat{\phi}'(t) = \mathcal{F}^{-1}\left[\frac{R \mathcal{F}[w]}{\mathcal{F}[V_\mathrm{PE}]}\right](t) \\
  \hat{\phi}(t) &= \hat{\alpha}\hat{\phi}'(t),\hat{\alpha} = \arg \underset{\alpha'}{\min}\mathrm{RSS}\left[\alpha'\hat{\phi}'(t)\otimes V_\mathrm{PE}(t),w(t)\right]
\end{align*}
\begin{figure}
    \centering
    \resizebox{0.55\textwidth}{!}{\input{img/fftrans.pgf}}
    \caption{$\mathrm{RSS}=\SI{124.7}{mV^2},D_w=\SI{2.03}{ns},\Delta t_0=\SI{-1.16}{ns}$}
\end{figure}
\end{frame}

\section{Fast Bayesian matching pursuit}

\begin{frame}
\frametitle{Model definition}
\begin{itemize}
    \item Time in DAQ window is divided into time bins: $\vec{t}$, whose length is $N$
    \item Model vector: $\vec{z}$. $z_i=0\implies q_i=0$ and $\ z_i=1\implies q_i\neq0$
    \item Linear Model: $\vec{w} = \bm{V}_\mathrm{PE}\vec{z} + \vec{\epsilon}$
    \item
        \begin{align*}
            \left.
            \begin{bmatrix}
                \vec{w} \\
                \vec{q}
            \end{bmatrix}
            \right\vert\vec{z}
            &\sim \mathrm{Normal}\left(
            \begin{bmatrix}
                \bm{V}_\mathrm{PE}\vec{z} \\
                \vec{z}
            \end{bmatrix},
            \begin{bmatrix}
                \bm{\Sigma}_z & \bm{V}_\mathrm{PE}\bm{Z} \\
                \bm{Z}\bm{V}_\mathrm{PE}^\intercal & \bm{Z}
            \end{bmatrix}
            \right) \\
            \bm{\Sigma}_z &= \bm{V}_\mathrm{PE}\bm{Z}\bm{V}_\mathrm{PE}^\intercal+\sigma_\epsilon^2\bm{I}
        \end{align*}
    where $\bm{Z}$ is the diagonal matrix of vector $\vec{z}$ controlling $q_i$
    \item $\mathcal{Z}=\{\vec{z}_j\}$ contains \textcolor{red}{$2^{N}$} model vectors
    \item A \textcolor{red}{repeated greedy search}(RGS) is performed to construct the target set $\mathcal{Z}'$, which contains only the $\vec{z}$ giving large $p(\vec{w}|\vec{z})$.
\end{itemize}
\end{frame}

\begin{frame}
\frametitle{Model definition}
\tikzstyle{na} = [baseline=-.5ex]
\begin{columns}
\column{0.5\textwidth}
\begin{figure}
    \centering
    \resizebox{0.8\textwidth}{!}{\input{img/model0.pgf}}
\end{figure}
\begin{figure}
    \centering
    \resizebox{0.8\textwidth}{!}{\input{img/model3.pgf}}
\end{figure}
\column{0.5\textwidth}
\begin{figure}
    \centering
    \resizebox{0.8\textwidth}{!}{\input{img/model1.pgf}}
\end{figure}
\begin{figure}
    \centering
    \resizebox{0.8\textwidth}{!}{\input{img/model2.pgf}}
\end{figure}
\end{columns}
\end{frame}

\begin{frame}
\frametitle{Model selection}
\begin{itemize}
    \item Calculation of \textcolor{red}{$2^{N}$} model vectors is impossible!
    \item Most of $p(\vec{w}|\vec{z}) \rightarrow 0$!
\end{itemize}
\noindent\begin{minipage}[c]{0.33\textwidth}
    \begin{figure}[H]
        \centering
            \includegraphics[width=0.95\textwidth]{img/perfect_PE.pdf}
        \caption{perfect PE matching waveform, $p(\vec{z}|\vec{w})$ hit maximum}
        \label{fig:perfect PE}
    \end{figure}
\end{minipage}\begin{minipage}[c]{0.33\textwidth}
    \begin{figure}[H]
        \centering
            \includegraphics[width=0.95\textwidth]{img/not_so_perfect_PE.pdf}
        \caption{not so perfect, $p(\vec{z}|\vec{w})$ is smaller but still $>0$}
        \label{fig:not so perfect PE}
    \end{figure}
\end{minipage}\begin{minipage}[c]{0.33\textwidth}
    \begin{figure}[H]
        \centering
            \includegraphics[width=0.95\textwidth]{img/nonsense_PE.pdf}
        \caption{Completely mismatch the waveform, $p(\vec{z}|\vec{w}) \rightarrow 0$}
        \label{fig:nonsense PE}
    \end{figure}
\end{minipage}
\begin{align*}
    \mathcal{Z}' &\subseteq \mathcal{Z} \\
    p(\vec{z}|\vec{w}) &= \frac{p(\vec{w}|\vec{z})p(\vec{z})}{\sum_{\vec{z}'\in\mathcal{Z}}p(\vec{w}|\vec{z'})p(\vec{z'})} \approx \frac{p(\vec{w}|\vec{z})p(\vec{z})}{\sum_{\vec{z}'\in\mathcal{Z}'}p(\vec{w}|\vec{z'})p(\vec{z'})}
\end{align*}
\end{frame}

\begin{frame}
\frametitle{FBMP's result: Bayesian interface}
\begin{itemize}
    \item PE Time: $\vec{t}$
    \item Models: $\mathcal{Z}'=\{\vec{z}_j\}$
    \item Charge: \begin{align*}
        \hat{\vec{q}}_z = E(\vec{q}|\vec{w},\vec{z}) &= \vec{z} + \bm{Z}\bm{V}_\mathrm{PE}^\intercal\bm{\Sigma}_z^{-1}(\vec{w}-\bm{V}_\mathrm{PE}\vec{z})
        \end{align*}
    \item Model's posterior probability: $p(\vec{z}|\vec{w})$
\end{itemize}
\begin{center}
    Provides opportunity for subsequent Bayesian analysis!
\end{center}
\end{frame}

\begin{frame}
\frametitle{FBMP demonstration}
\begin{figure}
    \centering
    \resizebox{0.6\textwidth}{!}{\input{img/demoe2c0.pgf}}
    \caption{$\mathrm{RSS}=\SI{15.8}{mV^2},D_w=\SI{0.59}{ns},\Delta t_0=\SI{-3.51}{ns}$}
\end{figure}
\end{frame}

\begin{frame}
\frametitle{Kullback-Leibler divergence}
\begin{align*}
  \hat{t}_\mathrm{KL} &= \arg\underset{t_0}{\max} \prod_{i=1}^{\hat{N}} \left[\phi(\hat{t}_i-t_0)\right]^{\hat{q}_i} ,\ \Delta t = \hat{t}_\mathrm{KL} - t_0
\end{align*}
\begin{figure}
    \centering
    \resizebox{0.6\textwidth}{!}{\input{img/twoprofile.pgf}}
    \caption{Time translation between time profiles}
\end{figure}
\end{frame}

\begin{frame}
\frametitle{$\mu$ estimation}
For Fourier deconvolution,
\begin{align*}
    \hat{\mu} &= \sum_i \hat{q}_i
\end{align*}
For FBMP,
\begin{align*}
    \left(\hat{t}_0, \hat{\mu}\right) &= \arg\underset{t_0,\mu}{\max} p(\vec{w} | t_0, \mu) \\
    p(\vec{w}|t_0, \mu) &= \sum_{\vec{z}'\in\mathcal{Z}'}p(\vec{w}|\vec{z}',t_0,\mu)p(\vec{z}'|t_0,\mu) \\
    &= \sum_{\vec{z}'\in\mathcal{Z}'}p(\vec{w}|\vec{z}')p(\vec{z}'|t_0,\mu)
\end{align*}
\end{frame}

\begin{frame}
\frametitle{FBMP's performance of evaluation criteria}
For dataset: $(\mu, \tau_l, \sigma_l)/\si{ns}=(4, 20, 5)$:
\begin{figure}
    \centering
    \resizebox{\textwidth}{!}{\input{img/vs.pgf}}
    \caption{$D_w$ and $\mathrm{RSS}$ of methods}
\end{figure}
\end{frame}

\begin{frame}
\frametitle{Charge posterior}
\begin{figure}
    \centering
    \resizebox{0.7\textwidth}{!}{\input{img/chargehist.pgf}}
    \caption{$\hat{q}$ histogram of methods}
\end{figure}
\begin{block}{}
FBMP retains charge distribution of PE.
\end{block}
\end{frame}

\begin{frame}
\frametitle{Timing \& charge resolution}
For dataset: $(\tau_l, \sigma_l)/\si{ns}=(20, 5)$:
\begin{figure}
    \centering
    \resizebox{\textwidth}{!}{\input{img/vs-reso.pgf}}
    \caption{Timing \& charge resolution}
\end{figure}
\begin{block}{}
FBMP will have better timing \& charge resolution.
\end{block}
\end{frame}

\section{Summary \& outlook}

\begin{frame}
\frametitle{Summary}
\begin{itemize}
    \item Waveform analysis is the first step for any data analysis on JUNO.
    \item FBMP restore information of PE most completely, with a Bayesian interface.
    \item FBMP will provide better timing and charge resolution.
\end{itemize}
\end{frame}

\begin{frame}
\frametitle{Outlook}
FBMP @ JUNO
\begin{itemize}
    \item Test Hamamatsu PMTs based on their SPE.
    \item Test NNVT MCP PMTs based on their SPE.
    \item Integrate FBMP with event reconstruction methods.
\end{itemize}
\end{frame}

\section{Backup}

\begin{frame}
\frametitle{Residual sum square}
\begin{align*}
  \mathrm{RSS} &\coloneqq \int\left[\hat{w}(t) - w(t)\right]^2\mathrm{d}t
\end{align*}
\begin{figure}
    \centering
    \resizebox{1.0\textwidth}{!}{\input{img/tab.pgf}}
    \caption{$b_1$ and $b_2$ have the same $\mathrm{RSS}=0.25$ to $a$, but $b_1$ is closer in timing to $a$}
\end{figure}
\end{frame}

\begin{frame}
\frametitle{Wasserstein distance}
\begin{figure}
    \centering
    \includegraphics[width=1.0\linewidth]{img/WD.png}
    \caption{Wasserstein Distance when $p=1$: Earth Mover Distance}
\end{figure}
\begin{align*}
  D_w\left[\hat{\phi}_*, \tilde{\phi}_*\right] &= \inf_{\gamma \in \Gamma} \left[\int \left\vert t_1 - t_2 \right\vert^p \gamma(t_1, t_2)\mathrm{d}t_1\mathrm{d}t_2\right]^{\frac{1}{p}}
\end{align*}
\begin{align*}
  \Gamma &= \left\{\gamma(t_1, t_2) ~\middle\vert~ \int\gamma(t_1,t_2)\mathrm{d}t_1 = \tilde{\phi}_*(t_2) , \int\gamma(t_1,t_2)\mathrm{d}t_2 = \hat{\phi}_*(t_1) \right\}
\end{align*}
when $p=1$, CDF of $\phi(t)$ is $\Phi(t)$, $D_w$ is a $\ell_1$-distance:
\begin{align*}
  D_w\left[\hat{\phi}_*, \tilde{\phi}_*\right] &= \int\left|\hat{\Phi}(t) - \tilde{\Phi}(t)\right| \mathrm{d}t
\end{align*}
\end{frame}

\begin{frame}
\frametitle{Repeated greedy search}
\begin{align*}
    \log[\textcolor{red}{p(\vec{w},\vec{z})}] =& \log[p(\vec{w}|\vec{z})p(\vec{z})] \\
    =& -\frac{1}{2}(\vec{w}-\bm{V}_\mathrm{PE}\vec{z})^\intercal\bm{\Sigma}_z^{-1}(\vec{w}-\bm{V}_\mathrm{PE}\vec{z})-\frac{1}{2}\log\det\bm{\Sigma}_z \\
    &-\frac{N}{2}\log2\pi -\mu + \sum_{i|z_i=1}\log \frac{\mu \phi(t'_i - t_0) \Delta t'}{1-\mu \phi(t'_i - t_0) \Delta t'}
\end{align*}
\end{frame}

\begin{frame}
\bibliographystyle{unsrt}
\bibliography{ref.bib}
\end{frame}
\end{CJK*}
\end{document}


\section{Experiments}
\label{sec:experiments}

We present experiments on \MNIST \citep{LecunIEEE1998} and \Cifar \citep{Krizhevsky2009}. We first analyze the impact of fixed-point quantization schemes on robustness (\secref{subsec:experiments-quantization}). Subsequently, we discuss weight clipping (\Clipping, \secref{subsec:experiments-clipping}), showing that improved robustness originates from increased redundancy in the weight distribution. Then, we focus on random bit error training (\Random, \secref{subsec:experiments-randbet}). We show that related work \cite{KimDATE2018,KoppulaMICRO2019} does not generalize, while \Random generalizes across chips and voltages, as demonstrated on profiled bit error patterns from different chips. \secref{subsec:experiments-discussion} summarizes our results for various precisions $m$.

\begin{table}
	\centering
	\small
	\caption{\textbf{Robust Quantization}. \RTE for random bit errors at $p = 0.05\%$ and $p = 0.5\%$
	for different quantization schemes, \cf \secref{subsec:robustness-quantization}. Minor differences
	can have large impact on \RTE while clean test error is unaffected. For $8$ bit the second row shows \Normal quantization (symmetric/per-layer) whereas the last row is our \Quant.
	*\Clipping[$0.1$]+\Quant with and without rounding.
	}
	\label{tab:quantization-robustness}
	\vspace*{-0.25cm}
	\hspace*{-0.2cm}
	\begin{tabular}{| c | l | c | c | c |}
		\hline
		\multicolumn{2}{|c|}{Quantization Schemes} & \multirow{2}{*}{\begin{tabular}{@{}c@{}}\TE\\in \%\end{tabular}}& \multicolumn{2}{c|}{\RTE in \%}\\
		\cline{4-5}
		\multicolumn{2}{|c|}{(%\Normal on
		\CifarT)} && $p{=}0.05$ & $p{=}0.5$\\
		\hline
		\hline
		\multirow{5}{*}{\rotatebox{90}{$8$ bit}} & \eqnref{eq:quantization}, global & 4.63 & 86.01 {\color{gray}\scriptsize ${\pm}$3.65} & 90.71 {\color{gray}\scriptsize ${\pm}$0.49}\\
		& \eqnref{eq:quantization}, per-layer & 4.36 & 5.51 {\color{gray}\scriptsize ${\pm}$0.19} & 24.76 {\color{gray}\scriptsize ${\pm}$4.71}\\
		& +asymmetric & 4.36 & 6.47 {\color{gray}\scriptsize ${\pm}$0.22} & {\color{colorbrewer1}40.78} {\color{gray}\scriptsize ${\pm}$7.56}\\
		& +unsigned & 4.42 & 6.97 {\color{gray}\scriptsize ${\pm}$0.28} & 17.00 {\color{gray}\scriptsize ${\pm}$2.77}\\
		& +rounding (=\Quant) & 4.32 & \bfseries 5.10 {\color{gray}\scriptsize ${\pm}$0.13} & \bfseries 11.28 {\color{gray}\scriptsize ${\pm}$1.47}\\
		\hline
		\hline
		\multirow{2}{*}{\rotatebox{90}{$4$ bit}} & w/o rounding* & 5.81 & 90.40 {\color{gray}\tiny ${\pm}$0.21} & 90.36 {\color{gray}\tiny ${\pm}$0.2}\\
		& w/ rounding* & \bfseries 5.29 & \bfseries 5.75 {\color{gray}\tiny ${\pm}$0.06} & \bfseries 7.71 {\color{gray}\tiny ${\pm}$0.36}\\
		\hline
	\end{tabular}
	\vspace*{-0.2cm}
\end{table}

\textbf{Metrics:} We report (clean) test error \TE (lower is better, $\downarrow$), corresponding to \emph{clean} weights, and \textbf{robust test error \RTE} ($\downarrow$) which is the
\textbf{test error after injecting bit errors into the weights}. As the
bit errors are random we report the \emph{average} \RTE and its standard deviation for $50$ samples of random bit errors with rate $p$ as detailed in \secref{sec:errors}.

\textbf{Architecture:} We use SimpleNet \citep{HasanpourARXIV2016}, providing comparable performance to ResNets \cite{HeCVPR2016} with only $W{=}5.5\text{Mio}$ weights on \CifarT. On \MNIST, we halve all channel widths, resulting in roughly $1\text{Mio}$ weights. On \CifarH, we use a Wide ResNet (WRN) \cite{ZagoruykoBMVC2016}. As batch normalization (BN) \cite{IoffeICML2015} yields
consistently worse robustness against bit errors we use group normalization (GN) \cite{WuECCV2018}, see
\appref{subsec:supp-experiments-bn}.

\textbf{Training:} We use stochastic gradient descent with an initial learning rate of $0.05$, multiplied by $0.1$ after $\nicefrac{2}{5}$, $\nicefrac{3}{5}$ and $\nicefrac{4}{5}$ of $100$/$250$ epochs on \MNIST/\Cifar. On \Cifar, we whiten the input images and use AutoAugment \cite{CubukARXIV2018} with Cutout \cite{DevriesARXIV2017}. For \Random, random bit error injection starts when the loss is below 1.75 on \MNIST/\CifarT or 3.5 on \CifarH. Normal training with the standard and our robust quantization are denoted \Normal and \Quant, respectively. Weight clipping with $\wmax$ is referred to as \Clipping[\wmax] or together with \Random as \Random[\wmax].
For \Quant, $m = 8$, we obtain $4.3\%$ on \CifarT and $18.5\%$ \TE on \CifarH. On \MNIST, $0.47\%$ are possible even for $m = 2$.

\begin{table}
	\centering
	\caption{\textbf{Weight Clipping Robustness.} Clean \TE and \RTE as well as clean confidence and confidence at $p{=}1\%$ bit errors (in \%, higher is better, $\uparrow$) for \Clipping and \Clipping with label smoothing (+LS). \TE increases for $\wmax = 0.025$ where the DNN is not able to produce large (clean) confidences. LS consistently reduces robustness, indicating that robustness is due to enforcing high confidence during training \emph{and} weight clipping.}
	\label{tab:clipping-robustness}
	\vspace*{-0.25cm}
	\small
	\hspace*{-0.2cm}
	\begin{tabular}{| l | c | c | c | c | c |}
		\hline
		Model & \multirow{2}{*}{\begin{tabular}{@{}c@{}}\TE\\in \%\end{tabular}} & \multirow{2}{*}{\begin{tabular}{@{}c@{}}Conf\\in \%\end{tabular}} & \multirow{2}{*}{\begin{tabular}{@{}c@{}}Conf\\$p{=}1$\end{tabular}} & \multicolumn{2}{c|}{\RTE in \%}\\
		\cline{5-6}
		(\CifarT) & & & & $p{=}0.1$ & $p{=}1$\\
		\hline
		\hline
		\Quant & \bfseries 4.32 & \bfseries 97.42 & 78.43 & 5.54 & 32.05\\
		\hline
		\Clipping[$0.15$] & 4.42 & 96.90 & 88.41 & \bfseries  5.31 & 13.08\\
		\Clipping[$0.1$] & 4.82 & 96.66 & 92.97 & 5.58 & 8.93\\
		\Clipping[$0.05$] & 5.44 & 95.90 & \bfseries 94.73 & 5.90 & \bfseries 7.18\\
		\Clipping[$0.025$] & 7.10 & {\color{colorbrewer1}84.69} & 83.28 & 7.40 & 8.18\\
		\hline
		\Clipping[$0.15$]+LS & 4.67 & 88.22 & 47.55 & 5.83 & {\color{colorbrewer2}29.40}\\
		\Clipping[$0.1$]+LS & 4.82 & 87.90 & 78.89 & 6.10 & 10.59\\
		\Clipping[$0.05$]+LS & 5.30 & 87.41 & 85.04 & 6.43 & 7.30\\
		\hline
	\end{tabular}
	\vspace*{-0.2cm}
\end{table}

Our \textbf{appendix} includes implementation details (\appref{sec:supp-implementation}), more information on our experimental setup (\appref{subsec:supp-experiments-setup}), and complementary experiments (\appref{sec:supp-experiments}). Among others, we discuss the robustness of BN (\appref{subsec:supp-experiments-bn}), other architectures such as ResNet-50 (\appref{subsec:supp-experiments-bn}), qualitative results for \Clipping (\appref{subsec:supp-experiments-clipping}) and complete results for $m = 4,3,2$ bits precision (\appref{subsec:supp-experiments-summary}). Also, we discuss a simple guarantee how the average \RTE relates
to the true expected robust error (\appref{subsec:supp-bound}). Our \textbf{code} will be made publicly available.

\subsection{Quantization Choice Impacts Robustness}
\label{subsec:experiments-quantization}

Quantization schemes affect robustness significantly, even when not affecting accuracy.
\tabref{tab:quantization-robustness} shows that per-layer quantization reduces \RTE significantly for small bit error rates, \eg, $p = 0.05\%$. While asymmetric quantization further reduces the quantization range, \RTE increases, especially for large bit error rates, \eg, $p = 0.5\%$ (marked in {\color{colorbrewer1}red}). This is despite \figref{fig:quantization} showing a slightly smaller impact of bit errors. This is caused by an asymmetric quantization into \emph{signed} integers: Bit flips in the most significant bit (MSB, \ie, sign bit) are not meaningful if the quantized range is not symmetric as the sign bit does not reflect the sign of the represented weight value, see \appref{subsec:supp-experiments-quantization}. Similarly, replacing integer conversion of $\nicefrac{w_i}{\Delta}$ by proper rounding, $\lceil\nicefrac{w_i}{\Delta}\rfloor$, reduces \RTE significantly (resulting in our \Quant).
This becomes particularly important for $m = 4$. Here, rounding also improves clean \TE slightly, but the effect is significantly less pronounced. Proper rounding generally reduces the quantization error. However, it is striking that this has little impact on \TE but tremendous effect on \RTE.
For $m = 4$ or lower, we also found weight clipping to help training, obtaining lower \TE.
Overall, random bit errors induce unique error distributions, \cf \figref{fig:quantization}, heavily dependent on quantization details.

\begin{table}[t]
	\centering
	\caption{\textbf{Fixed Pattern Bit Error Training.} \RTE for training on an entirely fixed bit error pattern (\Pattern). \emph{Top:} Evaluation on the same pattern; \Pattern trained on $p = 2.5\%$ does not generalize to $p = 1\%$ even though the bit errors for $p = 1\%$ are a subset of those seen during training for $p = 2.5\%$ (in {\color{colorbrewer1}red}). \emph{Bottom:} \Pattern also fails to generalize to completely random bit errors. This can be confirmed on profiled bit errors in \appref{subsec:supp-randbet-baselines}.}
	\label{tab:randbet-baselines}
	\vspace*{-0.25cm}
	\small
	\begin{tabular}{| l | c | c |}
		\hline
		Model (\CifarT) & \multicolumn{2}{c|}{\RTE in \%, $p$ in \%}\\
		\hline
		\hline
		\textbf{Evaluation on Fixed Pattern} & $p{=}1$ & $p{=}2.5$\\
		\hline
		\Pattern $p{=}2.5$ & {\color{colorbrewer1}14.14} & 7.87\\
		\Pattern[$0.15$] $p{=}2.5$ & {\color{colorbrewer1}8.50} & 7.41\\
		\hline\hline
		\textbf{Evaluation on \emph{Random} Patterns} & $p{=}1$ & $p{=}2.5$\\
		\hline
		\Pattern[$0.15$] $p{=}2.5$ & 12.09 & 61.59\\
		\hline
	\end{tabular}
	\vspace*{-0.2cm}
\end{table}

\subsection{Weight Clipping Improves Robustness}
\label{subsec:experiments-clipping}
While the quantization range adapts to the weight range
after every update during training, weight clipping explicitly constraints the weights to $[-\wmax, \wmax]$.
\tabref{tab:clipping-robustness} shows the effect of
different $\wmax$ for \CifarT with 8 bit precision. The clean test error is not affected for \Clipping[$\mathbf{\wmax{=}0.15}$] but
one has already strong robustness improvements for $p=1\%$
compared to \Quant (\RTE of 13.18\% vs 32.05\%). Further reducing $\wmax$ leads to a slow increase in clean \TE and decrease in average clean confidence, while significantly
improving \RTE to $7.18\%$ for $p=1\%$ at $\wmax=0.05$. For $\wmax=0.025$
the DNN is no longer able to achieve high confidence (marked in {\color{colorbrewer1}red}) which leads to stronger loss of clean \TE. Interestingly, the gap between clean and perturbed confidences under bit errors for $p=1\%$ is (almost) monotonically decreasing. These findings generalize to other datasets and precisions, see \appref{subsec:supp-experiments-summary}. However, for low precision $m\leq 4$ the effects are stronger
as \Quant alone does not yield any robust models and weight clipping is essential for achieving robustness.

As discussed in \secref{subsec:robustness-clipping} the robustness of the DNN originates in the cross-entropy loss enforcing high confidences on the training set and, thus, large logits while weight clipping works against having large logits. Therefore, the network has to utilize more weights with larger absolute values (compared to $\wmax$).
In order to test this hypothesis, we limit the confidences that need to be achieved via label smoothing \cite{SzegedyCVPR2016}, targeting $0.9$ for the true class and $\nicefrac{0.1}{9}$ for the other classes. According to \secref{subsec:robustness-clipping}, this should lead to less robustness, as the DNN has to use ``fewer'' weights. Indeed, in \tabref{tab:clipping-robustness}, \RTE at $p=1\%$ increases from $13.08\%$ for \Clipping[$0.15$] to $29.4\%$ when using label smoothing (marked in {\color{colorbrewer2}blue}). Moreover, the difference between average clean and perturbed confidence is significantly larger for DNNs trained with label smoothing.

In \appref{subsec:supp-experiments-clipping} we show that robustness against bit errors also leads to robustness against $L_\infty$ perturbations which generally affect all weights in contrast to random bit errors, and provide more qualitative results about the change of the weight distribution induced by clipping in \figref{fig:supp-clipping}.

\begin{table}[t]
	\centering
	\small
	\caption{\textbf{Random Bit Error Training (\Random).} Average \RTE (and standard deviation) of \Random evaluated at various bit error rates $p$ and using $m = 8$ or $4$ bit precision. For low $p$, weight clipping provides sufficient robustness. However for $p \geq 0.5$, \Random increases robustness significantly. This is pronounced for lower precisions.}
	\label{tab:randbet-robustness}
	\vspace*{-0.25cm}
	\hspace*{-0.25cm}
	\begin{tabular}{|@{\hskip 3px}c@{\hskip 3px}|@{\hskip 3px}l@{\hskip 3px}|@{\hskip 3px}c@{\hskip 3px}|@{\hskip 3px}c@{\hskip 3px}|@{\hskip 3px}c@{\hskip 3px}|@{\hskip 3px}c@{\hskip 3px}|}
		\hline
		& Model (\CifarT) & \multirow{2}{*}{\begin{tabular}{@{}c@{}}\TE\\in \%\end{tabular}} &\multicolumn{3}{c|}{\RTE in \%}\\
		\cline{4-6}
		& $\mathbf{\wmax{=}0.1}$, $p$ in \% && $p{=}0.5$ & $p{=}1$ & $p{=}1.5$\\
		\hline
		\hline
		\multirow{5}{*}{\rotatebox{90}{$8$bit}} & \Quant & \bfseries 4.32 & 11.28 {\color{gray}\tiny ${\pm}$1.47} & 32.05 {\color{gray}\tiny ${\pm}$6} & 68.65 {\color{gray}\tiny ${\pm}$9.23}\\
		& \Clipping & 4.82 & 6.95 {\color{gray}\tiny ${\pm}$0.24} & 8.93 {\color{gray}\tiny ${\pm}$0.46} & 12.22 {\color{gray}\tiny ${\pm}$1.29}\\
		& \Random $p{=}1$ & 4.90 & \bfseries 6.36 {\color{gray}\tiny ${\pm}$0.17} & \bfseries 7.41 {\color{gray}\tiny ${\pm}$0.29} & \bfseries 8.65 {\color{gray}\tiny ${\pm}$0.37}\\
		\hline
		\multirow{2}{*}{\rotatebox{90}{$4$bit}} & \Clipping & \bfseries 5.29 & 7.71 {\color{gray}\tiny ${\pm}$0.36} & 10.62 {\color{gray}\tiny ${\pm}$1.08} & 15.79 {\color{gray}\tiny ${\pm}$2.54}\\
		& \Random $p{=}1$ & 5.39 & \bfseries 7.04 {\color{gray}\tiny ${\pm}$0.21} & \bfseries 8.34 {\color{gray}\tiny ${\pm}$0.42} & \bfseries 9.77 {\color{gray}\tiny ${\pm}$0.81}\\
		\hline
	\end{tabular}
	\vspace*{-0.2cm}
\end{table}

\subsection{\Random Yields Generalizable Robustness}
\label{subsec:experiments-randbet}

\textbf{Training on Profiled Errors Does Not Generalize:}
%\label{subsec:experiments-baselines}
%
Co-design approaches such as \cite{KimDATE2018,KoppulaMICRO2019} combine training DNNs on profiled SRAM or DRAM bit errors with hardware-approaches to limit the errors' impact.
However, profiling SRAM or DRAM requires expensive infrastructure, expert knowledge and time.
More importantly, training on profiled bit errors does not generalize to previously unseen bit error distributions (\eg, other chips or voltages): \tabref{tab:randbet-baselines} (top) shows \RTE of \Pattern, \ie, pattern-specific bit error training. The main problem is that \Pattern does \emph{not} even generalize to lower bit error rates (\ie, higher voltages) of the same pattern as trained on (marked in {\color{colorbrewer1}red}). This is striking as, following \figref{fig:errors}, the bit errors form a subset of the bit errors seen during training: training with $p = 2.5\%$ bit errors does not provide robustness for $p = 1\%$, \RTE increases $7.9\%$ to $14.1\%$. It is not surprising, that \tabref{tab:randbet-baselines} (bottom) also demonstrates that \Pattern does not generalize to random bit error patterns: \RTE increases from $7.4\%$ to $61.6\%$ at $p = 2.5\%$. The same observations can be made when training on real, profiled bit errors corresponding to the chips in \figref{fig:errors}, see \appref{subsec:supp-randbet-baselines}.
Overall, obtaining robustness that generalizes across voltages \emph{and} chips is crucial for low-voltage operation to become practical.

\begin{table}[t]
	\centering
	\small
	\caption{\textbf{Generalization to Profiled Bit Errors.} \RTE for \Random on two different profiled chips. The bit error rates differ across chips due to measurements at different voltages, also see \figref{fig:errors}. Chip 2 exhibits a bit error distribution significantly different from uniform random bit errors: bit errors are strongly aligned along columns and biased towards $0$-to-$1$ flips, \cf \figref{fig:errors}. Nevertheless, \Random generalizes surprisingly well.}
	\label{tab:randbet-generalization}
	\vspace*{-0.25cm}
	\hspace*{-0.15cm}
	\begin{tabular}{| l | l | c | c |}
		\hline
		Chip (\figref{fig:errors}) & Model (\CifarT)& \multicolumn{2}{c|}{\RTE in \%}\\
		\hline
		\hline
		\bfseries Chip 1 && $p{\approx}0.86$ & {\color{colorbrewer1}$p{\approx}2.75$}\\
		\hline
		& \Random[$0.05$] $p{=}1.5$ & 7.04 & 9.37\\
		\hline
		\hline
		\bfseries Chip 2 && $p{\approx}0.14$ & {\color{colorbrewer1}$p{\approx}1.08$}\\
		\hline
		& \Random[$0.05$] $p{=}1.5$ & 6.00 & 9.00\\
		\hline
	\end{tabular}
	\vspace*{-0.2cm}
\end{table}
\begin{figure*}[t]
	\centering
	\vspace*{-0.1cm}
	\hspace*{-0.3cm}
	\begin{subfigure}{0.32\textwidth}
		\centering
		\includegraphics[width=5.75cm]{c10_pareto_8bit}
	\end{subfigure}
	\hfill
	\begin{subfigure}{0.32\textwidth}
		\centering
		\includegraphics[width=5.75cm]{c100_pareto_8bit}
	\end{subfigure}
	\hfill
	\begin{subfigure}{0.32\textwidth}
		\centering
		\includegraphics[width=5.75cm]{m_pareto_8bit}
	\end{subfigure}

	\hspace*{-0.1cm}
	\fbox{
	\begin{subfigure}{0.98\textwidth}
		\centering
		\includegraphics[width=1\textwidth]{c10_legend}
	\end{subfigure}
	}
	\vspace*{-14px}
	\caption{\textbf{Bit Error Robustness on \CifarT, \CifarH and \MNIST.} Average \RTE plotted against bit error rate $p$, both in \%. We considered various models (in {\color{gray}$\bullet$ gray}), corresponding to different $\wmax$ and $p$ during training. We explicitly plot the best model for each bit error rate: for \Normal ({\color{colorbrewer5}orange}), \Quant ({\color{colorbrewer1}red}), \Clipping ({\color{colorbrewer2}blue}) and \Random ({\color{colorbrewer4}violet}). Note that these might correspond to different $\wmax$ and $p$ (also across datasets). Across all approaches, we plot the per-error-rate best model in black: for $m = 8,4,3,2$ bits, depending on dataset. For $8$ bit and low bit error rates, \Clipping is often sufficient. However, for $4$ bit or higher bit error rates, \Random is crucial to keep \RTE low.}
	\label{fig:summary}
	\vspace*{-0.2cm}
\end{figure*}

\textbf{\Random Improves Robustness:}
%\label{subsec:experiments-randbet}
%
\Random, with weight clipping, further improves robustness and additionally generalizes across chips and voltages. \tabref{tab:randbet-robustness} shows results for weight clipping and \Random with $\wmax = 0.1$ and $m = 8,4$ bits precision. \Random is particularly effective against large bit error rates, \eg, $p = 1.5\%$, reducing \RTE from $12.22\%$ to $8.65\%$ ($m = 8$ bits). The effect is pronounced for $4$ bits or even lower precision, where models are generally less robust. The optimal combination of weight clipping and \Random depends on the bit error rate. For example, in \tabref{tab:clipping-robustness}, lowering $\wmax$ to $0.05$ reduces \RTE below \Random[$0.1$] with $p{=}1\%$ for some bit error rates.
We emphasize that \Random generalizes to lower bit errors than trained on, in stark contrast to the fixed-pattern training \Pattern. In \appref{subsec:supp-experiments-architectures}, we also show that \Random works on other architectures such as ResNet-50. On other datasets, \eg,\MNIST, \Random allows to operate at $p = 12.5\%$ bit error rate with $0.9\%$ \RTE and only $m = 2$ bits. At this point, weight clipping alone yields $90\%$ \RTE.

\textbf{\Random Generalizes to Profiled Bit Errors:}
%\label{subsec:experiments-generalization}
%
\Random also generalizes to profiled bit errors from real chips. \tabref{tab:randbet-generalization} shows results on the two profiled chips of \figref{fig:errors}. Profiling was done at various voltage levels, resulting in different bit error rates. To simulate various weights to memory mappings, we apply various offsets before linearly mapping weights to the profiled SRAM arrays. \tabref{tab:randbet-generalization} reports average \RTE, showing that \Random generalizes quite well to these profiled bit errors. Regarding chip 1, \Random performs very well, even for large $p\approx 2.75$, as the bit error distribution of chip 1 largely matches our error model in \secref{sec:errors}, \cf \figref{fig:errors} (left). In contrast, with chip 2 we picked a more difficult bit error distribution which is strongly aligned along columns, potentially hitting many MSBs simultaneously. Thus, \RTE increases for chip 2 even for a lower bit error rate $p \approx 1.08$ (marked in {\color{colorbrewer1}red})
but energy savings are still possible without degrading prediction performance.

\subsection{Summary and Discussion}
\label{subsec:experiments-discussion}

Our experiments are summarized in \figref{fig:summary}. We consider \Normal quantization vs. our robust quantization \Quant, various \Clipping and \Random models with different $\wmax$ and $p$ during training (indicated in {\color{gray}$\bullet$ gray}) and plot \RTE against bit error rate $p$ at test time. On all datasets \Quant outperforms \Normal. On \CifarT (left), \RTE increases significantly for \Quant ({\color{colorbrewer1}red}) starting at $p \approx 0.25\%$ bit error rate. While \Clipping ({\color{colorbrewer2}blue}) generally reduces \RTE, only \Random ({\color{colorbrewer4}violet}) can keep \RTE around $6\%$ or lower for a bit error rate of $p \approx 0.5\%$. The best model for each bit error rate $p$ (black and solid for $m = 8$) might vary. \CifarH is generally more difficult, while significantly higher bit error rates are possible on \MNIST. On \CifarT, \RTE increases slightly for $m = 4$. However, for $m = 3,2$ \RTE increases more significantly as clean \TE increases by $1-2\%$. % which likely due to our simple quantization.
Nevertheless, \RTE only increases slightly for larger bit error rates $p$. It remains future work whether \Random with a more sophisticated (but robust) quantization scheme can enable low-voltage operation even for $m = 2$ bits. In all cases, \RTE increases monotonically, ensuring safe operation at higher voltages. The best trade-off between robustness and accuracy depends on the application: higher energy savings require a larger ``sacrifice'' in terms of \RTE.
Finally, \appref{subsec:supp-bound} provides a
confidence-interval based guarantee on how strongly \RTE is expected to deviate from the empirical results in \figref{fig:summary}.

Overall, the results in \figref{fig:summary} enable robust low-voltage operation \emph{without} requiring expensive error correcting codes (ECCs) or other circuit techniques \cite{ReagenISCA2016,ChandramoorthyHPCA2019}. Furthermore, our analysis applies both to DRAM, commonly off-chip, and SRAM, usually used as scratchpads on-chip of DNN accelerators. Compared to co-design \cite{KimDATE2018,KoppulaMICRO2019}, we do not require expensive expert knowledge or profiling infrastructure. Moreover, \Random improves over these approaches by generalizing across chips and voltages.
Besides \Random, we show that robust fixed-point quantization \emph{only with} weight clipping can provide reasonable robustness, \eg, for $p = 0.1\%$ on \CifarT. This is without sophisticated quantization scheme, \eg, with special treatment for outliers \cite{ZhuangCVPR2018,SungARXIV2015,ParkISCA2018}, and complementary to  \cite{MurthyARXIV2019,MerollaARXIV2016,SungARXIV2015,AlizadehICLR2020}, focusing merely on robustness \emph{to} quantization.


\documentclass{article} % For LaTeX2e
\usepackage{nips13submit_e,times}
\usepackage{hyperref}
\usepackage{url}
\usepackage{mathtools}
%\documentstyle[nips13submit_09,times,art10]{article} % For LaTeX 2.09

\title{Traffic Counting with Kalman Filters}

\author{
Anthony M Lozano\\
Department of Computer Science\\
Georgia Institute of Technology\\
\texttt{amlozano1@gmail.com} \\
}

% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to \LaTeX{} to determine where to break
% the lines. Using \AND forces a linebreak at that point. So, if \LaTeX{}
% puts 3 of 4 authors names on the first line, and the last on the second
% line, try using \AND instead of \And before the third author name.

\newcommand{\fix}{\marginpar{FIX}}
\newcommand{\new}{\marginpar{NEW}}

\nipsfinalcopy % Uncomment for camera-ready version

\begin{document}

\maketitle

\begin{abstract}
Counting vehicles is an important function for traffic video data analysis. This paper examines an approach using a Shi-Tomasi corner detector with a Lucas Kanade Optical flow pyramid to create data measurements. After clustering tracked data measurements, a Kalman Filter is used to process the data in order to finally determine if a vehicle passes through the video. Our implementation of this method shows some success, but needs refinement.
\end{abstract}

\section{Background/Problem Description}
\label{sec:background}

Accurate traffic data has diverse applications; Civil Engineers use it to design, construct, and maintain roadways.  It is utilized by Policy Makers to allocate funding for such projects, and business owners may need it to make sound decisions about location. In addition, this information can be used to estimate the environmental impact of roadways and to perform other similar usage based calculations.

While the data is certainly useful, extant methods of collecting it are more expensive and cumbersome than necessary. The chief problems of current solutions are portability, setup time, and need for humans to process the data. Portability might not be an issue for areas where traffic data will need to be collected permanently, but usually agencies want a system that can move easily to take measurements from many sites. Some examples of less portable systems include inductive loop sensors installed into the roadway itself, or the TRAX Stealth Stud, which is another device that needs to be installed into the pavement.  Installing these devices requires closing the road, which is not ideal in situations where even a relatively short road closure may cause major disruptions, such as urban areas. Pneumatic loop sensors, which look like small black tubes that are laid perpendicularly over the road, are an effective solution, but some jurisdictions in the United States require that workers close the road when they are putting these up anyway for safety reasons. Other systems use a camera to record video and a human later counts the cars going by, producing very accurate data at the cost of being labor intensive and time consuming. These systems and devices are also expensive, costing thousands of dollars for a single unit or many man hours of labor to operate a single unit

This paper attempts to create a software analysis tool that can perform one of the most basic tasks for traffic data collection; counting how many cars have gone in a particular direction in a video. This software tool would enable anyone with a camera to collect traffic data.

\section{Hypothesis and Project Approach}
\label{sec:hypothesis}

Computer vision techniques for tracking motion in video combined with a Kalman Filter should be able to track vehicles and plot their movements. The software will then record the starting and ending points of each successfully tracked vehicle, thus establishing counts for each origin and destination.

\section{Implementation Details}
\label{sec:implementation}

The Kalman Car Counter program will need to perform six major tasks to count vehicles. These tasks are background subtraction (or foreground detection), feature detection, optical flow estimation, optical flow clustering, and finally Kalman Filtering.

\subsection{Background Subtraction}
\label{subsec:back_subract}

Background subtraction is the process of removing noise and unmoving sections of an image.  The Kalman Car Counter uses the simple process of frame differencing to remove the background for speed. Frame difference at a time $t$ can be calculated as
$ D(t+1) = V(x,y,t+1) - V(x,y,t) $ for each pixel $(x,y)$~\cite{Birgi09}.
This calculation will allow the subsequent algorithms to focus solely on areas of the video that contain motion, helping isolate vehicles and removing image noise.

\subsection{Feature Detection}
\label{subsec:feature_detec}

Feature detection is the process of finding good features of an image for tracking. For vehicle tracking, Shi-Tomasi corner detection is a solid algorithm choice that expands on standard Harris corner detection, and will allow us to zero in on the "corners" of vehicles. Briefly, the algorithm works as follows: If we can assume the time between frames is small, we can describe the changes between them as image motion. If we take a patch of an image defined by $(u,v)$ and shift it by $(x,y)$, the weighted sum of squared differences between the two patches, $S$ is given by \\*

{\centering
$ S(x,y) = \sum\limits_{u}\sum\limits_{v}w(u,v)(I(u+x, v+y) - I(u,v))^2$\linebreak\newline\\*}
$I(u+x,v+y)$ can be approximated bu a Taylor expansion with $I_x,$ and $I_y$ being the partial derivatives of $I$ such that:\\*

{\centering
$I(u+x,v+y)) \approx I(u,v) (I_x(u,v) x+I_y(u,v) y)^2$\linebreak\newline\\*}
which produces the approximation\\*

{\centering
$S(x,y) \approx \sum\limits_{u}\sum\limits_{v}w(u,v) (I_x(u,v) x+I_y(u,v) y)^2$\linebreak\newline\\*}
which in matrix form is\\*

{\centering
$S(x,y) \approx \left(\begin{smallmatrix}
x & y
\end{smallmatrix}\right) A
\left(\begin{smallmatrix}
x\\
y
\end{smallmatrix}\right)$\linebreak\newline\\*}
where $A$ is the structure tensor\\*
\\*

{\centering
$
A = \sum_u \sum_v w(u,v)
\begin{bmatrix}
I_x^2 & I_x I_y \\
I_x I_y & I_y^2
\end{bmatrix}
=
\begin{bmatrix}
\langle I_x^2 \rangle & \langle I_x I_y \rangle\\
\langle I_x I_y \rangle & \langle I_y^2 \rangle
\end{bmatrix}$\linebreak\newline\\*}
A corner is then found if $S$ has a large variation in all directions of the vector $\begin{pmatrix} x & y \end{pmatrix}$. Thus, if $A$ has two positive eigenvalues that are large in comparison to other eigenvalues, a corner is found~\cite{tommasini1998making}. The Shi-Tomasi corner detector takes a practical shortcut and only calculates $min(\lambda_1, \lambda_2)$ rather than doing a complete eigenvalue decomposition in the interest of speed and because this is sufficient to find features with great accuracy~\cite{Shi94}.

\subsection{Optical Flow}
Once we have determined what features in the image to track, we actually need to track them. To accomplish this, the Kalman Car Counter will use another well known algorithm, the Lucas-Kanade method. This method assumes that pixels move a small amount between frames, and the movement of the pixels is approximatly the same for all pixels in the neighborhood of a point $p$~\cite{lucas1981iterative}. Therefore, the optical flow equation should hold for all pixels in a window centered at $p$. This means the local velocity of a feature $(V_x,V_y)$ must satisfy the equations:\\*

{\centering
$I_x(q_1) V_x + I_y (q_1) V_y = -I_t(q_1)$\\*
$I_x(q_2) V_x + I_y (q_2) V_y = -I_t(q_2)$\\*
$\vdots$\\*
$I_x(q_n) V_x + I_y (q_n) V_y = -I_t(q_n)$\linebreak\newline\\
}
where $q_1,q_2,\dots,q_n$ are the pixels around a feature inside the window and $I_x(q_i),I_y(q_i),I_t(q_i)$ are the partial derivatives if the image $I$ with respect to position and time $x,y,$ and $t$

This system of equations can be written in the matrix form:\\*

{\centering
$A = \begin{bmatrix}
I_x(q_1) & I_y(q_1) \\[10pt]
I_x(q_2) & I_y(q_2) \\[10pt]
\vdots  & \vdots  \\[10pt]
I_x(q_n) & I_y(q_n)
\end{bmatrix},
\quad\quad
v =
\begin{bmatrix}
V_x\\[10pt]
V_y
\end{bmatrix},
\quad \mbox{and}\quad
b =
\begin{bmatrix}
-I_t(q_1) \\[10pt]
-I_t(q_2) \\[10pt]
\vdots  \\[10pt]
-I_t(q_n)
\end{bmatrix}
$\linebreak\newline\\*}
This overdetermined system is solved with a compromise solution by the least squares principle, solving the 2x2 system:\\*

{\centering
$A^T A v=A^T b$\linebreak\newline\\*}
Bouguet presents a pyramidal implementation of the Lucase Kanade tracker~\cite{bouguet2001pyramidal}, which is the particular implementation that the Kalman Car Counter uses. If the Lucas-Kanade tracker tracks the features detected in the previous step successfully, we can use those features as data points for our Kalman Filter.

\subsection{Optical Flow Data Clustering}
\label{subsec:optical_flow}

The Lucas-Kande tracker can return many successfully tracked features for a single vehicle. We can leverage this by clustering the tracked features to combine them into a single point. Because we cannot predict how many vehicles will be in a frame, we use a agglomerative hierarchical clustering scheme. We use cophenetic distance $t$ as a criterion for automatic clustering of the data, expecting different vehicle feature clusters to have a large distance between them. Once the data is clustered, the means of each cluster that has at least $n$ elements is assigned to a Kalman Filter.

\subsection{Hungarian Algorithm}
\label{subsec:hungarian_algo}

Once each cluster's mean has been established, we intend to again track each cluster, but this time using a Kalman Filter (described in Section~\ref{subsec:kalman_filter}) in order to be more tolerant to noise, occlusion, and merged clusters. Thus, each cluster must be assigned to a Kalman filter. For each existing Kalman filter, we attempt to assign the nearest cluster to that filter for measurement updates. In order so solve the assignment problem we use the Kuhn-Munkres or, simply, Munkres algorithm~\cite{munkres1957algorithms}. It solves a cost matrix where $a1$ would be the distance from a cluster mean $a$ to a filter's next prediction $1$:\\*

{\centering
$\begin{bmatrix}
a1 & a2 & a3 & a4\\
b1 & b2 & b3 & b4\\
c1 & c2 & c3 & c4\\
d1 & d2 & d3 & d4\end{bmatrix}$\linebreak\newline\\*}
Solving this cost matrix for the minimum cost produces assignments for each cluster. If the algorithm assigns a cluster to a Kalman filter, it is used as a measurement update for that Kalman filter for the frame. Note that this matrix can be rectangular; there could be more clusters than Kalman filters or more filters than clusters. If there are more Kalman filters than clusters, Kalman filters that do not get an assignment are marked as "Lost" and do not receive a measurement update. Lost Kalman filters accumulate a Loss Counter each subsequent frame they are lost in until they are either deleted or are again assigned to a cluster. A filter can also be lost if it's best assignment is to far away.  On the other hand, if there are more clusters than Kalman Filters, a new filter is created for each new cluster, instantiated with initial position equal to the cluster mean.

\subsection{Kalman Filter}
\label{subsec:kalman_filter}

The final step solves a practical problem: even with all the previous steps, our measurements are still somewhat noisy measurements of the vehicles location in the video. In order to better track the vehicles, we pass the measurements through a Kalman filter. The filter allows us to track vehicles even if vehicles' measurements are noisy, vehicles are temporarily occluded, or vehicles' clusters merge.

The Kalman filter requires several matrices, the most important of which is the next state function. This Kalman filter will model the motion of the vehicles using a simple system $F$ of $(x,y)$ position and velocity. \\*

{\centering
$F = \begin{bmatrix}
1 & 0 & dt & 0\\
0 & 1 & 0 & dt\\
0 & 0 & 1 & 0\\
0 & 0 & 0 & 1\end{bmatrix}$\linebreak\newline\\*}
Read in simple English, this matrix means that the next state's position should be estimated using the prior position and velocity multiplied by the change in time. The change in time is set to the number of seconds between each frame of the video. For example, in a 30 frames-per-second video, $dt$ would be .5 seconds.
Each Kalman filter will have it's initial state $X$ set to the mean of the cluster $(x,y)$ that created it with no initial velocity, that is $V_x = 0$ and $V_y = 0$. \\*

{\centering
$X = \begin{bmatrix}
x \\
y \\
0 \\
0 \end{bmatrix}$\linebreak\newline\\*}
our measurement update function will be\\*

{\centering
$H = \begin{bmatrix}
1 & 0 \\
0 & 1 \\
0 & 0 \\
0 & 0 \end{bmatrix}$\linebreak\newline\\*}
Since we can measure $x$ and $y$ but not $V_x$ or $V_y$ the uncertainty for the velocity will be set very high, while the uncertainty for position will be set relatively low.\\*

{\centering
$R = \begin{bmatrix}
 1 & 0 & 0 & 0\\
 0 & 1 & 0 & 0\\
 0 & 0 & 1000 & 0\\
 0 & 0 & 0 & 1000\end{bmatrix}$\linebreak\newline\\*}

The Kalman filter performs prediction step and calculates the posterior position $X'$:\\*

{\centering
$X' = FX+u$\linebreak\newline\\*}
and it updates uncertainty by:\\*

{\centering
$P' = F \cdot P \cdot F^T$\linebreak\newline\\*}
Similarly, after each prediction step, a measurement update step must be performed with a new measurement $Z$.\\*

{\centering
$Y = Z - H \cdot X$\\*
 $S = H \cdot P \cdot H^T + R$\\*
 $K = P \cdot H^T \cdot S^{-1}$\\*
 $X' = X + (K\cdot Y)$\\*
 $P' = (I - K \cdot H) \cdot P $\linebreak\newline\\*}
 With these two steps iterating every frame, we track each vehicle's starting location to ending location. With the tracked vehicle starting and ending positions in hand, we can count vehicles entering the scene and leaving the scene.

\section{Technical Specifications}
\label{sec:technical_specs}

The Kalman Car Tracker has been implemented in the Python programming language. This implementation uses several technologies to accomplish it's task. The foremost is \texttt{OpenCV} (Open Source Computer Vision Library), an open source computer vision and machine learning software library. \cite{wiki:opencv} It contains many optimized and state-of-the art algorithms of use to this project with Python bindings. In particular, the background subtraction, Shi-Tomasi corner detection, Lucas-Kanade optical flow tracking, and Kalman filteings all come from the \texttt{OpenCV} library. The hierarchical data clustering algorithm comes from another powerful Python library known as \texttt{SciPy}. Munkres linear assignment comes from a additional \texttt{(Scipy)} library by Lars Buitinck.\cite{scipy}

To test this method, a dataset was captured with a cellphone camera on a highway overpass. A video was recorded from the center of the overpass overlooking the freeway at a resolution of 1080p for three minutes. The overpass vibrated, introducing much noise for our particular background subtraction method. In addition, the camera was held in hand; no tripod or other stabilization was used to record video.

The video processing took approximately 15 minutes for 3 minutes of video on a Intel i7 960 CPU @ 3.20 GHz.

\section{Analysis of Results}
\label{sec:analysis}

In the first half of the sample video on the left (oncoming) lanes of traffic, $24/35$ , or about 68\%, of vehicles were successfully tracked. Our definition of successfully tracked is that Kalman filter tracking the car was assigned for at least the last 6 frames of the car being in the image. Having the car successfully tracked when it leaves the frame would be sufficient to count the car as having flowed under the overpass.  While clearly there is room for improvement, this shows the method implemented in this tool can successfully track vehicles and perform the counting task. The parameters for various methods were not optimized, and the tool is far from complete, but these preliminary results seem promising. Perhaps with optimization and refinement this method could be used as envisioned to enabled simple unattended capture of traffic data.

\section{Conclusion}
\label{sec:conclusion}

In this paper, we present a method for collecting traffic data using a simple camera and computer vision processing, even in the presence of occlusion. The method needs refinement, but can successfully track vehicles. This method seems to have trouble differentiating between clusters of tracked optical flows when they are tightly packed, which could prove a large problem for dense traffic flows. This problem might be solved with a better camera perspective, but the best camera perspectives, such as top down, make the detection process much more trivial and come with disadvantages similar to those provided in the introduction: they are costly and hard to set up. Future research could explore other options for clustering that may help alleviate this issue.

\section{Presentation and Github}
\label{sec:presentation}
A quick 5 minute presentation of this implementation is available at \url{https://www.youtube.com/watch?v=CMG_3c9UbsE}.
\\*\\*
A longer, more complete presentation is also available at \url{http://youtu.be/lEuYAGPUJWk}
\\*\\*
The implementation source code is available at \url{https://github.com/amlozano1/kalman_car_counter} under the MIT License.

\nocite{*}
\bibliographystyle{plain}
\bibliography{KalmanBib}
\end{document}


\section{Example}

\begin{lstlisting}[style=default]
Program {
  [ GridDimension {ncol}, GridDimension {nlay}, GridDimension {ngpt}],
  Domain {
    parallel_domain_dim : ncol,
    vertical_dim : nlay,
    parallel_dim : ngpt
  },
  FieldDecl {
    real, [GridDimension {ncol}],
    name : mu0
  },
  FieldDecl {
    real, [GridDimension {ncol}, GridDimension{nlay}, GridDimension{ngpt}],
    name : tau
  },
  FieldDecl {
    real, [GridDimension {ncol}, GridDimension {nlay}, GridDimension{ngpt}]
    name : w0
  },
  FieldDecl {
    real, [GridDimension {ncol}, GridDimension {nlay}, GridDimension {ngpt}],
    name : g
  },
  FieldDecl {
    real, [GridDimension {ncol}, GridDimension{nlay}],
    name : Rdif
  },
  FieldDecl {
    real, [GridDimension {ncol}, GridDimension {nlay}],
    name : Tdif
  },
  FieldDecl {
    real, [GridDimension {ncol}, GridDimension {nlay}],
    name : Rdir
  },
  FieldDecl {
    real, [GridDimension {ncol}, GridDimension {nlay}],
    name : Tdir
  },
  FieldDecl {
    real, [GridDimension {ncol}, GridDimension {nlay}],
    name : Tnoscat
  },
  VarDecl {
    int,
    name : ncolbounds,
    isarg : true
  },
  VarDecl {
    int,
    name : nlaybounds,
    isarg : true
  },
  ScopedProgram {
    // two_stream (Missing arg list, name of what is called, shouldn't inline stuff at this point)
    // adding_sw (same as up)
    // additional_step (same as up)
    // two_stream
    VarDecl {
      real,
      name : mu0_inv
    },
    VarDecl {
      real,
      name : mu0_inv
    },
    VarDecl {
      real,
      name : gamma1
    },
    VarDecl {
      real,
      name : gamma2
    },
    VarDecl {
      real,
      name : gamma3
    },
    VarDecl {
      real,
      name : gamma4
    },
    VarDecl {
      real,
      name : alpha1
    },
    VarDecl {
      real,
      name : alpha2
    },
    VarDecl {
      real,
      name : k
    },
    VarDecl {
      real,
      name : RT_term
    },
    VarDecl {
      real,
      name : exp_minusktau
    },
    VarDecl {
      real,
      name : exp_minus2ktau
    },
    VarDecl {
      real,
      name : k_mu
    },
    VarDecl {
      real,
      name : k_gamma3
    },
    VarDecl {
      real,
      name : k_gamma4
    },
    BlockStmt {
      // Computation, this is the only stmt that we will express as a tree.
      // From this on, pseudocode for the statements will be used
      AssignmentStmt {
        VarAccess {
          name : mu0_inv
        },
        divOp {
          VarAccess {
            name : mu0
          },
          Literal {
            real,
            value : 1.0
          }
        }
      },
      Computation {
        [ GridDimension {name: ngpt}, GridDimension {name: ncol} ],
        DimensionInterval {
          GridDimension {name : nlay},
          DimensionLevel {
            VarAccess {
              Literal {0},
	      name : nlaybounds
            },
            offset : 0
          },
          DimensionLevel {
            VarAccess {
              Literal {1},
	      name : nlaybounds
            },
            offset : 0
          }
        },
        BlockStmt {
          AssignmentStmt {
            VarAccess {
              name : gamma1
            },
            mulOp {
              minusOp {
                Literal { 8.0 },
                mulOp {
                  VarAccess { name : w0 },
                  addOp {
                    Literal { 5.0 },
                    mulOp {
                      Literal { 3.0 },
                      VarAccess { name : g }
                    }
                  }
                }
              },
              Literal { 0.25 }
            },
            lineno : 654,
            file : sw_solver.f90
          },
          AssignmentStmt {
            VarAccess { name : gamma2},
            mulOp {
              Literal { 3.0 },
              mulOp {
                mulOp {
                  VarAccess { name : w0 },
                  minusOp {
                    Literal { 1.0 },
                    VarAccess { name : g }
                  }
                },
                Literal { 0.25 }
              }
            },
            lineno : 655,
            file : sw_solver.f90
          },
          AssignmentStmt {
            VarAccess { name : gamma3 },
            mulOp {
              minusOp {
                Literal { 2.0 },
                mulOp {
                  mulOp {
                    Literal { 3.0 },
                    VarAccess { name : mu0 }
                  },
                  VarAccess { name : g }
                }
              },
              Literal { 0.25 }
            }
          }
          AssignmentStmt {
            VarAccess { name : gamma4 },
            minusOp {
              Literal { 1.0 },
              VarAccess { name : gamma3 }
            }
          }

          % Rest of the program not represented as HIR
          % alpha1 = gamma1 * gamma4 + gamma2 * gamma3
          % alpha2 = gamma1 * gamma3 + gamma2 * gamma4
          % k = sqrt(max((gamma1 - gamma2) * (gamma1 + gamma2), 1.e-12_wp))
          % exp_minusktau = exp(-tau * k)
          % exp_minus2ktau = exp_minusktau * exp_minusktau
          % RT_term = 1._wp / (k * (1._wp + exp_minus2ktau)  + gamma1 * (1._wp - exp_minus2ktau) )
          % Rdif = RT_term * gamma2 * (1._wp - exp_minus2ktau)
          % Tdif = RT_term * 2._wp * k * exp_minusktau
          % Tnoscat = exp(-tau * mu0_inv)
          % k_mu     = k * mu0
          % k_gamma3 = k * gamma3
          % k_gamma4 = k * gamma4
          % RT_term =  w0 * RT_term/merge(1._wp - k_mu*k_mu, epsilon(1._wp), abs(1._wp - k_mu*k_mu) >= epsilon(1._wp))
          % Rdir = RT_term  * ((1._wp - k_mu) * (alpha2 + k_gamma3) -  (1._wp + k_mu) * (alpha2 - k_gamma3) * exp_minus2ktau - 2.0_wp * (k_gamma3 - alpha2 * k_mu)  * exp_minusktau  * Tnoscat)
          % Tdir = Tnoscat - RT_term * ((1._wp + k_mu) * (alpha1 + k_gamma4) * Tnoscat - (1._wp - k_mu) * (alpha1 - k_gamma4) * exp_minus2ktau * Tnoscat - 2.0_wp * (k_gamma4 + alpha1 * k_mu)  * exp_minusktau)
          % Tdir -= Tnoscat
        }
      }
    }
  }
}
\end{lstlisting}