\documentclass{mrl}
\usepackage{amsthm,amsmath}
\usepackage{listings}
\lstset{basicstyle=\ttfamily\footnotesize,breaklines=true}
\usepackage{hyperref}
\usepackage[utf8]{inputenc}
\usepackage{graphicx}
\usepackage{enumerate}
\begin{document}
\begin{frontmatter}

\begin{fmbox}
\hfill\setlength{\fboxrule}{0px}\setlength{\fboxsep}{5px}\fbox{\includegraphics[width=2in]{moneroLogo.png}}
\dochead{Research bulletin \hfill MRL-0001}
\title{A Note on Chain Reactions in Traceability in CryptoNote 2.0}
\date{12 September 2014}
\author[
   addressref={mrl},
   email={lab@monero.cc}
]{\fnm{Surae} \snm{Noether}}
\author[
   addressref={mrl},
]{\fnm{Sarang} \snm{Noether}}
\author[
   addressref={mrl},
]{\fnm{Adam} \snm{Mackenzie}}

\address[id=mrl]{
  \orgname{Monero Research Lab}
}
\end{fmbox}

\begin{abstractbox}
\begin{abstract}
This research bulletin describes a plausible attack on a ring-signature based anonymity system. We use as motivation the cryptocurrency protocol CryptoNote 2.0 ostensibly published by Nicolas van Saberhagen in 2012. It has been previously demonstrated that the untraceability obscuring a one-time key pair can be dependent upon the untraceability of all of the keys used in composing that ring signature. This allows for the possibility of chain reactions in traceability between ring signatures, causing a critical loss in untraceability across the whole network if parameters are poorly chosen and if an attacker owns a sufficient percentage of the network. The signatures are still one-time, however, and any such attack will still not necessarily violate the anonymity of users. However, such an attack could plausibly weaken the resistance CryptoNote demonstrates against blockchain analysis. This research bulletin has not undergone peer review, and reflects only the results of internal investigation.
\end{abstract}
\end{abstractbox}

\end{frontmatter}

\section{Introduction}
The CryptoNote protocol has a problem: my anonymity depends on your anonymity. If I use $5, 6$, or $18$ of your outputs when I compose my ring signature then \emph{you} can see the true signer. If you spend all $5, 6$, or $18$ outputs with no foreign outputs used as mixins in your ring signatures, you reveal yourself as the spender, and now any observer can also see the true signer of my transaction.  This may not be malicious if you spent your outputs non-anonymously for legitimate business or legal reasons. Hence, any party with a large proportion of the UTXO set may gain knowledge of the traceability of others' transactions and reveal that information to the network at will.  One may fancifully interpret this problem as an abstract, cryptocurrency implementation of Gresham's Law: bad money drives out good. If the unspent transaction output (UTXO) set is filled with a lot of transactions that aren't really anonymous, there are fewer ways to make untraceable ring signatures. At this point it \emph{must} be noted that, even in this scenario, the one-time key pairs (so-called ``stealth addresses'') used in CryptoNote protocols are not violated in this scenario, and so the anonymity of users is still not directly violated. Rather, this attack violates the untraceability \emph{between} one-time ring signatures, but this development is still somewhat worrying. Hence, even non-malicious entities can execute this attack on accident, malicious entities can spam the network to own lots of the UTXO set, and malicious entities can break untraceability for others.

An attacker's intent may be perhaps in the interest of a pump-n-dump scheme by undermining the credibility of a currency, perhaps in the interest of spying on other users, or perhaps in a disinterest in paying the extra cost of adding more foreign unspent transaction outputs to their ring signatures. In all cases, despite the \emph{a priori} hope that any particular user should always be interested in using a large number of mixins for a ring signatures as a matter of principle, both in the interest of personal security and the common good, we should have no expectation of this in practice. A version of the tragedy of the commons may take place and cause traceability throughout the network for everyone due to a subset of users.

However, we should also keep in mind the forest, not the trees: since any user can always compose a trivial ring signature with no additional outputs, immediately exposing outputs down the line, there is a second-order problem here. A new transaction has been exposed, creating exposure risk for any ring signatures that used the newly exposed transaction as an obfuscating mixin. Is it possible that a chain reaction could occur, \emph{even if a malicious entity stops actively attacking early in the history of the coin}?  This research bulletin was written to analyze this problem, determine the plausibility of a conservative route of such an attack, and assess network parameters to ensure graceful degradation of untraceability.

\section{Setup for a Passive Attack}
Everyone who has had a college probability class has come across the following, even if you don't know how to solve it:

\begin{quote}
An urn contains $N$ marbles of two possible colors. We have $N_B$ black marbles and $N_W$ white marbles, where $N_B + N_W = N$ is fixed. We draw $M$ marbles from the urn. What is the probability that all of the marbles we drew from the urn are black? What is the probability that all of the marbles are white? If $0 \leq m \leq M$, what is the probability that we have $m$ black marbles and $M-m$ white marbles?
\end{quote}

When phrased this way, we immediately jump to the hypergeometric distribution. The probability mass function \[\frac{\binom{N_B}{B}\binom{N - N_B}{M - B}}{\binom{N}{M}}\] can be used to compute the probability that all the marbles in my hand are black (set $B=M$), and we can use convenient formulae available in any number of free online textbooks, or wikipedia, or whatever, in order to compute the probability of any particular occurrence. It's one of the nicer distributions.

We play the following game: if all the marbles I draw are black, I'll toss a new black marble into the urn while I replace my handful. Otherwise, I'll toss a new white marble into the urn while I replace my handful. As it is with marbles, so it is with CryptoNote coins: It may help to envision each marble as an atomic unit (the CryptoNote analogue to a Satoshi) tracked in the UTXO set of a CryptoNote network. The color denotes who is in control of the marble; black if controlled by a malicious entity, white if otherwise.

Let's say Lisa wishes to send some money to Milhouse (which would add a marble to the bag). The UTXO set that Lisa draws from in order to make her ring signatures will contain $N$ transaction outputs, of which $N_B$ are controlled by Burns (the black marbles; assume the rest are white). Then the above formula is exactly what we need. Lisa composed her ring signature with $M \leq N$ mixins\footnote{We define a {\it mixin} as a foreign output included in the transaction.} (she drew a handful of marbles to decide the color of marble to add to the urn), and if all are controlled by Burns? Well, then, \emph{if} Burns spends his outputs with no mixins\footnote{Which, it again must be emphasized, may be an ostensibly innocent act, not an attack!}, Lisa's transaction will also be exposed. Furthermore, forget Lisa, \emph{any} ring signature that used only Burns' signatures can actually now be considered part of Burns' controlled set, in terms of untraceability, even if Burns does not control the private keys. Notice that in this scenario, we are having a chain reaction occur: Burns reveals only his own transaction outputs, but he, in turn, reveals Lisa's output.  The four assumptions we have made are

\begin{enumerate}[(i)]
\item Burns controls an initial proportion of an otherwise unknown UTXO set,
\item all new transactions generate their ring signatures drawn from the current UTXO set in a uniform, unordered way without replacement,
\item all new transactions use exactly the mandatory minimum mixin level (handful size $M$), and
\item Burns gains control of new transactions if and only if their ring signatures are solely composed with Burns-controlled transactions (i.e.\ Burns stops putting new transactions in the system at time $t=0$).
\end{enumerate}

We can change our assumptions to strengthen or weaken the attack in our scenario.  For example, the attack is weaker if some transactions use more than the mandatory minimum. The attack is stronger if Burns can also gain control of transactions by generating them himself, i.e.\ he is not some passive actor in the background. However, this seems to be a reasonable, middle-ground scenario, if we interpret it in the following way: Burns initially controls a portion of the UTXO set before a flood of ostensibly honest transactions are composed quite suddenly, so what happens? Protecting the network against a large-scale degradation in untraceability under this scenario is simply a matter of smart engineering. If we can protect the network against large-scale degradation under worse scenarios, all the better, but we shall start here. This could be considered a ``Christmas Day'' attack, both because our simulations are set up to mimic a sudden influx of economic activity before a truly malicious attacker can respond (for ease of coding) and because the attack can be executed by an ostensibly innocent individual simply spending their money non-anonymously for perfectly legitimate reasons.

Since the hypergeometric distribution is so nice, what can we expect out of this game? Well, let us analyze the results for a \emph{single new transaction}. Assuming $M$ is the minimum number of mixins across the whole network, what is the probability that all $M$ mixin signatures are controlled by Burns? If this probability is large, then new transactions are very likely to be ``controlled'' by Burns, even if he has yet to reveal the transaction outputs he controls. That is to say, the more CryptoNote transactions controlled by a single malicious party, the weaker the \emph{untraceability} trait of new transactions becomes. Consider the following numerical example.

Suppose we have $N=10^3$ transaction outputs in our anonymity set (not counting the real output belonging to Lisa), and that Burns controls $pN$ of these transactions for some $0 \leq p \leq 1$.  Denote by $\Pr(p)$ the probability that any one ring signature (using three mixins) is signed solely with outputs belonging to Burns. This probability describes, roughly, whether or not Lisa's new marble will be black or white. Table \ref{3mixins} contains sample values of $\Pr(p)$ for this example.

\begin{table}[!h]
\begin{center}
\begin{tabular}{c|c|c}
$p$ & $pN$ & $\Pr(p)$ \\ \hline
$10^{-3}$ & $1$ & $0$ \\
$10^{-2}$ & $10$ & $7.22\times 10^{-7}$ \\
$10^{-1}$ & $100$ & $9.73 \times 10^{-4}$\\
$0.5$ & $500$ & $0.125$
\end{tabular}
\caption{If Burns controls $pN$ transaction outputs from an anonymity set of $N=10^3$ outputs, we display the probability $\Pr(p)$ that Lisa's output is traceable from Burns' point of view, assuming that Lisa uses three mixins in the construction of her ring signature. That is to say, at the beginning of a large scale simulation, if Burns controls half of all transactions, we can expect him to ``see'' about $12.5\%$ of new transactions.}
\label{3mixins}
\end{center}
\end{table}

How do we interpret this table? If Burns controls only a single output in the anonymity set, there is no way three distinct outputs in a ring signature can all belong to him. If Burns controls ten outputs, there is only a probability of $0.00000722$ that Lisa's ring signature is really controlled by Burns. This is so small that on average, it would take about $13.8$ million signatures from this anonymity set to start seeing collisions. If Burns controls $100$ outputs, there is a probability of $0.000973$ that a signature made by Lisa is controlled by Burns, so we should start seeing collisions after around $100,000$ signatures. Finally, if Burns controls half of all outputs, then $12.5\%$ of all new transactions will have a ring signature composed entirely of his outputs, so Burns will only be gaining $1$ in every $8$ new transactions, and so Burns' share of the UTXO set will shrink over time unless he takes action.

In Table \ref{2mixins}, we produce the same values as for Table \ref{3mixins} but with only two mixins. In this case, if Burns controls half of the outputs, the consequence is that every one in four new outputs will really be controlled by Burns. If Burns controls $10\%$ of the outputs, only $1$ out of every $1000$ new transactions will be controlled by Burns. Clearly, not as great as $3$ mixins, but still not terrible; even if Burns controls half of all transactions, there is only a $25\%$ chance that a new ring signature will belong to Burns. Hence, his share of the UTXO set will shrink over time unless he takes action.

\begin{table}[!h]
\begin{center}
\begin{tabular}{c|c|c}
$p$ & $pN$ & $\Pr(p)$ \\ \hline
$10^{-3}$ & $1$ & $0$ \\
$10^{-2}$ & $10$ & $9.01\times 10^{-5}$ \\
$10^{-1}$ & $100$ & $9.91 \times 10^{-3}$\\
$0.5$ & $500$ & $0.25$ \\
\end{tabular}
\end{center}
\caption{If Burns controls $pN$ transaction outputs from an anonymity set of $N=10^3$ outputs, we display the probability $\Pr(p)$ that the new output is revealed during an attack, assuming that Lisa uses two mixins in the construction of her ring signature. That is to say, at the beginning of a large scale simulation, if Burns controls half of all transactions, we can expect him to ``grab'' about $25\%$ of new transactions.}

\label{2mixins}
\end{table}

\section{Monte Carlo Methods and Results}

Hold tight for some notation. We choose parameters $N_{B_0}, N_{W_0}$ and $M$, where $N_{B_0}$ denotes the initial number of black marbles owned by Burns, $N_{W_0}$ denotes the initial number of white marbles, and $M$ is the minimum mixin level. We play the game iteratively. We record the initial proportion of unspent transactions owned by burns as $P_0 = N_{B_0}/(N_{B_0} + N_{W_0})$ before beginning. On the $i^{th}$ iteration, we are adding the $N_i = (N_{B_0} + N_{W_0} + i)^{th}$ marble to the bag. We draw a uniform random number $u$ from $(0,1)$ and compare it to the hypergeometric mass function as described before. If $u \leq \binom{N_{B_i}}{M}/\binom{N_i}{M}$, then the next marble will be black, so we iterate $N_{B_i} = N_{B_{i-1}}+1$. Regardless of the choice of $u$, we iterate $N_{i+1} = N_{i} + 1$ and if we need to compute $N_W$ we can simply subtract. After some iterations, say $I$ iterations, usually enough iterations to double or quadruple the UTXO set, we stop the game and consider the proportion of black marbles, $P_F = N_{B_I}/N_I$. However, since we are only concerned with \emph{how many of the new marbles were black} (not how many in total are black), we will watch as our metric of success for Burns' attack the value $\widehat{P}_F = (N_{B_I} - N_{B_0})/I$; number of black marbles Burns gained during the game divided by number of marbles added to the bag.

The above process constitutes one simulation. For each choice of parameters $(N_{B_0}, N_{W_0}, M)$, we get a result $\widehat{P}_F$, which was determined as a result of a sequence of random experiments, and is such a random variable. So we run $37$ trials and take the sample mean and sample variance from these experiments. We initially looked at the $95\%$ confidence intervals, but they are so narrow that we may as well ignore them, in the end. We simulate a UTXO set growing from $5000$ transactions to $20000$. Hence, we plot $(20000P_F - 5000P_I)/15000$, that is, the proportion of \emph{ostensibly honest transactions that Burns grabs.} Figure \ref{fig1} illustrates our simulation results for $N_B+N_W = 5000$ and $M=1, 2, 3$, and $4$ mandatory minimum mixins in the event that the UTXO set quadruples with honest transactions (that is, $15,000$ new transactions need new ring signatures). The horizontal axis displays the initial proportion of the UTXO set controlled by Burns, varying from $0.0$ to $1.0$, the vertical axis displays the final proportion of the \emph{new} transactions in the UTXO set controlled by Burns, varying from $0.0$ to $1.0$. Note that \emph{any number above zero on the $y$-axis represents a loss of security}.

\begin{figure}[!h]
\centering
\includegraphics[width=0.95\textwidth,height=0.95\textheight,keepaspectratio]{burns.png}
\caption{This figure displays, with apologies to the color blind, the ``success of an attack'' executed by Burns. We see that, for increasing numbers of mixins, success uniformly decreases accross the board. To refresh the reader who probably jumped directly to this graph,  here's the scenario: Evil guy Burns starts with an initial proportion ($x$-axis) of the UTXO set, which has $5000$ transaction outputs inside of it. Then, a flood of $15,000$ new honest transactions hits the network, from Lisa to Milhouse and from Homer to Marge and so on, causing the UTXO set to quadruple in size. After this event, Burns could spend his initial proportion $P_I$ of the UTXO set with no mixins and consequently break the untraceability of a proportion of the $15,000$ ostensibly honest transactions. We display this proportion $\widehat{P}_F$ as a function of $P_I$. For $10$ mixins, Burns requires $55\%$ of the UTXO set to capture any of a quadrupling event at all, and requires $87\%$ of the UTXO set in order to capture even $1\%$ of a quadrupling event with these parameters.}
\label{fig1}
\end{figure}

It's clear that $\widehat{P}_F$ is monotonically increasing as a function of $P_0 = N_B/(N_B + N_W)$, but that the situation improves for greater numbers of ring signatures. Notice that there are some dynamical systems problems built into our data. If you graph $P_F$ vs. $P_I$, (rather than our $\widehat{P}_F$ vs. $P_I$) then you can interpret this as a discrete-time dynamical system's evolution function to see that Burns' share must go to zero over time for any mixin $\geq 2$. We can expect, in the $1$ mixin scenario, for the urn to stochastically jump up and down the $y=x$ line without appreciably, in expectation, moving in any direction in a random walk. However, as long as we require $M \geq 2$ mixins, any chain reaction will burn itself out, so to speak; Burns' share of the UTXO set will shrink to zero over time unless he takes action. Our results suggest that any mandatory mixin $\geq 2$ will allow the system to recover from a passive attack quite quickly. The code we used to generate data can probably be improved, but is located at \texttt{http://goo.gl/dGj5TZ} and we used Google Documents to generate figures.

\section{Critical Mass, Slow Chain Reactions, Active Attackers: Further Questions}

Recall that we set out to ask the question ``can a critical chain reaction occur?'' and the related question ``what network parameters can we choose to ensure that any plausible chain reaction burns itself out?'' So, what do we need for a critical mass? Consider the urn problem again. What happens to the composition of the urn as this game evolves, as we toss new black or white marbles into the urn as we draw more? Remember, black marbles are the ones that will be revealed; if we draw all black marbles, that means the next marble we add to the urn must be black. A critical mass scenario is one in which, if there are enough black marbles in the urn, then Burns starts grabbing more and more and more black marbles. If ever $P_F > P_I$, we could expect some sort of critical mass, but this does not occur. Except in numerical edge cases, $P_F \leq P_I$. With $2$ and more mixins, our chain reactions become truncated very rapidly.

Chain reactions act slower and slower as mixins increase. Thus, a single mixin is inappropriate for any currency problem, because an anonymous, malicious user may spam the network with transactions they control in the hopes that they eventually control greater proportions of the UTXO set. This would essentially make all transactions \emph{from then on} fully traceable (at least from Burns' point of view), and Burns doesn't have to take a single action after his initial seed transactions are planted in the UTXO set. This \emph{fixed initial cost} for the attacker leading to a never-ending stream of information in the form of traceable transactions from other users is, clearly, a catastrophic economic failure. This is so important, we are putting it in italics: \emph{any CryptoNote coin that allows for only $1$ mixin is vulnerable to a slow chain reaction in which the owner of very few private keys can violate the untraceability of much larger number of other users.} Requiring a mixin of at least $2$ for all transactions save for transactions that are willfully spent with $0$ mixins will keep these chain reactions, probabilistically, to a smaller length. Indeed, any number greater than $1$ will do, to force these chain reactions to burn themselves out, rather than to spread to the whole network, and the higher the better. Of course, a protocol-enforced, network-wide mandatory minimimum mixin of $M=10$ would, presumably, cause a blockchain bloat, which can hinder adoption, which has it's own security benefits in terms of network size. Hence, there is likely some optimal size of mandatory minimum mixin. We do no more than to suggest $M=2$ as a protocol-enforced mandatory minimum, and to advise users to use as many mixin signatures as their little hearts desire.

Note that our scenario might be different if we change parameters of our model, or if we change our assumptions to include active attackers. Let's discuss parameters real quick. We can consider the addition of each new marble as nudging Burns' \emph{passive} ownership probability one way or another. Unless Burns grabs almost all incoming transactions, his share of the UTXO set will be depleted over time. How do our parameters change this process? Recall that our parameters are:
\begin{enumerate}
\item $N_{B_0} = $ Number of initial atomic units controlled by Burns in UTXO set,
\item $N_{W_0} = $ the remaining atomic units in the UTXO set, and
\item $M = $ mandatory minimum mixin enforced by network protocol.
\end{enumerate}
However, we are really interested in $P_I = N_{B_0}/(N_{B_0} + N_{W_0})$ and $N_0 = N_{B_0} + N_{W_0}$. We have seen above how $P_I$ and $M$, in general, change the outcome of the experiment. As $N_0$ gets big, Burns' share will start to slow down. Meaning even if Burns is grabbing all of the incoming transactions, he's getting a reduced rate of returns in terms of overall proportion of the UTXO set that he owns. We can think of the UTXO set size as a mass. As it gets bigger, it gets harder to change its state.

Another ``hidden'' parameter here is ``how many new transactions are being added?'' We swept this under the rug by simply saying ``okay, the UTXO set is going to quadruple from it's initial size,'' reducing our parameter space a little bit, and in so doing, we have reduced the importance of $N_0$ as a parameter. Indeed, we know how $N$ changes, so we can easily compute slope of any statistic, $T(N)$, with respect to ``change in population size'' and call that the sensitivity:
\[S = \frac{T(N_f) - T(N_o)}{N_f - N_o} = \frac{T(4\cdot N_o) - T(N_o)}{3N_o}\]
which, when close to zero, suggests no relationship, and when far from zero suggests a strong relationship (positive or negative). We could just as easily compute the slope of the log of the statistic with respect to the log of the population, and call that the sensitivity:
\[S = \frac{\log(T(N_f)) - \log(T(N_0))}{\log(N_f)-\log(N_o)} = \frac{1}{4}\log\left(\frac{T(4\cdot N_o)}{T(N_o)}\right)\]
in this version of sensitivity, we don't even see $N_f$ as part of the formula, simply the multiplicative factor $N_f/N_o = 4$. However, we feel it is unnecessary to present a full-scale parameter-space analysis of this model.

Now let's discuss changing assumptions. One apparently small tweak we could add to this simulation is to change assumption (iv). In order to model a more aggressive attacker rather than a ``Christmas Day'' accidental reveal sort of attack, we could implement a probabilistic assumption like
\begin{enumerate}[(v)]
\item Burns gains control of a new transaction as the result of a sequence of Bernoulli trials. With a fixed probability $q$, Burns generates a transaction that he knows is his, and with probability $1-q$, a foreign transaction appears and generates a new ring signature that may be solely composed with Burns-controlled transactions (i.e.\ Burns still participates in the economy)
\end{enumerate}
or we could implement a dynamical model where a malicious Burns tries to maintain a minimum proportion of the UTXO set by spamming transactions to the network whenever his proportion drops his secretly determined percentage:
\begin{enumerate}[(v)]
\item Burns gains control of a new transaction as the result of a sequence of non-autonomous, Bernoulli-like experiments. With a probability $q(B,N)$ dependent both on the number of UTXOs controlled by Burns, $B$, and the total UTXO set, $N$, Burns generates a transaction that he knows is his, and with probability $1-q(B,N)$, a foreign transaction appears and generates a new ring signature that may be solely composed with Burns controlled transactions.
\end{enumerate}
Any function $q(B,N)$ that Burns can try to make close to $1$ when $B/N$ is smaller than Burns' preference will allow Burns to grow his share of the UTXOset. Of course, the easiest, simplest strategy for Burns would be to just spam the network as fast as possible with his own transactions. But with fees, this isn't free.

We bring up parameters and assumptions because the work is not really done on this problem. We have discovered what we, as network engineers, need to know in order to ensure graceful degradation: pick big mixins. However, there are really interesting questions still lurking under the surface. One of us solved this problem with graph theory, one of us with probability. Note that, as time goes on, a UTXO is more likely to be chosen for mixins, but is also more likely to have been exposed from previous users revealing their transactions by spending them with $0$ mixins, complicating the wisdom of choosing UTXOs for the ring signature from a uniform distribution. Technically, for any sort of ``critical mass'' sort of problem, there's going to be a steady state problem lurking under the surface, and we definitely have some dynamical systems stuff flying\footnote{See what I did there?} about in this problem.  Maybe some ambitious undergraduate wants to pick up where we leave off and contribute to the cryptocurrency community by expanding on all of this.

\begin{backmatter}
\end{backmatter}
\end{document}


\subsubsection{Threshold Value Determination}
\label{subsubsec:threshold}

Ideally, the mean absolute difference between two consecutive unchanged frames is zero.
This is not the case, however, due to image noise and other environmental influences such as \SI{50}{Hz} flickering of the ambient light intensity.
A suitable threshold value must therefore be determined.
For this purpose, the first eight mean absolute differences $\overline{\text{MD}}$ are used to determine the threshold value.
They are averaged and multiplied by a multiplier, as show in figure \ref{fig:threshold}.
At a frame rate of \SI{200}{fps} this results in a short start-up delay of \SI{40}{ms}.

\begin{figure}[h]
  \centering
  \includegraphics[width=0.75\textwidth]{threshold}
  \caption{Determination of the threshold value from eigth mean absolute differences}
  \label{fig:threshold}
\end{figure}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%2345678901234567890123456789012345678901234567890123456789012345678901234567890
%        1         2         3         4         5         6         7         8

\documentclass[letterpaper, 12 pt, conference]{ieeeconf}  % Comment this line out
                                                          % if you need a4paper
%\documentclass[a4paper, 10pt, conference]{ieeeconf}      % Use this line for a4
                                                          % paper

\IEEEoverridecommandlockouts                              % This command is only
                                                          % needed if you want to
                                                          % use the \thanks command
\overrideIEEEmargins
% See the \addtolength command later in the file to balance the column lengths
% on the last page of the document

% The following packages can be found on http:\\www.ctan.org
%\usepackage{graphics} % for pdf, bitmapped graphics files
%\usepackage{epsfig} % for postscript graphics files
%\usepackage{mathptmx} % assumes new font selection scheme installed
%\usepackage{times} % assumes new font selection scheme installed
%\usepackage{amsmath} % assumes amsmath package installed
%\usepackage{amssymb}  % assumes amsmath package installed
\usepackage{csquotes}
\usepackage{graphicx}
\usepackage{url}

\title{\LARGE \bf Image Recognition and Labeling in Reduced Dimensions}
\author{David Anderton and Ray Dedhia}

\begin{document}

\maketitle
\thispagestyle{empty}
\pagestyle{empty}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{abstract}

As the use of artificial intelligence continues to expand into the realm of image compression\footnote{See: \url{https://ai.googleblog.com/2016/09/image-compression-with-neural-networks.html}}\footnote{See: \url{http://www.wave.one/icml2017/}}, questions are being raised on how well the original artefacts and nature of the source image are preserved. The consequences of being able to utilize smaller output images that maintain the majority of pertinent information from their often significantly larger source images are huge. With smaller inputs, applications from self driving cars that rely on camera input, autonomous flight, through to training neural networks could require significantly less processing time and resources.

An interactive exploration of the results is available at the following URI: \url{https://mit18065-dimensions.herokuapp.com/}

\end{abstract}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{INTRODUCTION}

In this paper, we briefly examine how the Google Cloud Vision API responds to the content of a selection of images pre- and post-compression. We also consider a variety of compression techniques and extents. The goal: produce a broad estimate of how much an image can be compressed until the Google Cloud Vision API, as a proxy for other classification algorithms, can no longer determine useful insights about the constitution of the image.

\section{METHODOLOGY}
Our methodology was as follows:

Within our study we utilised 22 unique images, from four major
categories: six animals, six famous landmarks, five landscapes, five
street signs. Expanding this to both the color and grayscale
versions of these images, we had a resulting set of 44 source images.

After collating the source images we sent them through the Google Cloud Vision API\footnote{The Google Cloud Vision API, \url{https://cloud.google.com/vision/}, classifies images into categories and detects objects and faces in images utilising non-public machine learning models.}
and get a control set of labels for the 44 images. Then compress the images, using the five following methods:
(1) Posterization through k-means clustering,
(2) :ow-rank approximation using Principle Component Analysis (PCA),
(3) \textquote{Mean pooling} operation commonly used in neural networks,
(4) Mogrify (a command-line utility)
and (5) Dropout (randomly removing a specific percentage of the pixels).

Once the images were compressed in all of the above methods we then re-queried the Google Cloud Vision API to re-classify the images post-compression and considered the results.

\section{COMPRESSION METHODS}
\subsection{Posterization}
K-means clustering is an algorithm that groups
a set of data points into $k$ groups by mapping each data point
to one of $k$ centroids such that the distance between each data point
and its centroid is minimized. It does this by randomly generating $k$
centroids, (1) assigning each data point to the nearest centroid,
(2) updating the centroid values be setting them equal to the arithmetic
mean of the data points assigned to them in step 1, and repeating steps
1 and 2 until some stopping criteria has been met. The resulting k-means clustering implemented a posterization
effect.

We arbitrarily stopped our k-means clustering algorithm after 10 iterations as an intuitive stopping point for efficiency, and used $k=4$ and $k=8$. In Fig. 1 below are the results of putting an image of a cat through 4-means and 8-means clustering.

\newpage

\vspace*{3mm}
\begin{tabular}{c c}
	$k=4$ & $k=8$ \\
	\includegraphics[width=0.2\textwidth]{postfour} &
		\includegraphics[width=0.2\textwidth]{posteight} \\
\end{tabular}
\begin{tabular}{l}
	{\it \hspace*{4mm} Fig. 1: Image of a cat compressed using} \\
	{\it \hspace*{4mm} 4-means and 8-means clustering.}
\end{tabular}

\subsection{PCA}
The principle components of some matrix, $M$, are its left and right singular vectors, $u_i$ and $v_i$ respectively, and its singular values,$sigma$. The decomposition of a matrix into these components is referred to most commonly as the Singular Value Decomposition, or SVD.
PCA is a form of matrix analysis that utilizes the largest principle components of a matrix of data in order to create a summarization or approximate the underlying data.

By utilising the Eckart-Young theorem\footnote{See Golub et al. \url{https://www.ime.usp.br/~jstern/miscellanea/seminario/Golub87.pdf}}, the rank $k$ matrix $M_k$ closest to $M$ is the sum of the product of the $k$ largest singular values of $M$with their corresponding left and right singular vectors. That is, $M_k = \sum_{i=1}^k \sigma_i u_i v_i^T$.

Thus, PCA can be utilised to find the best low-rank approximation for $M$.
Within our study we used PCA to reduce our image matrices to rank-5, rank-10
and rank-25 matrices. Fig. 2 below shows the results of calculating the best rank-5, rank-10,
and rank-25 approximations of an image of the Tokyo Tower.

\vspace*{3mm}
\begin{tabular}{c c c}
	rank-5 & rank-10 & rank-25 \\
	\includegraphics[width=0.13\textwidth]{pcafive} &
		\includegraphics[width=0.13\textwidth]{pcaten} &
		\includegraphics[width=0.13\textwidth]{pcatwentyfive} \\
\end{tabular}
\begin{tabular}{l}
	{\it \hspace*{4mm} Fig. 2: Best rank-5, rank-10, and rank-25} \\
	{\it \hspace*{4mm} matrix approximations of an image of the} \\
	{\it \hspace*{4mm} Tokyo Tower.} \\
\end{tabular}

\subsection{Mean Pooling}
Mean pooling refers to the operation of dividing
an image into regions of size $m \times n$, calculating
the arithmetic mean of those regions, and outputting
an image consisting of just those calculated averages.\footnote{See the UFLDL Tutorial on Pooling: \url{http://ufldl.stanford.edu/tutorial/supervised/Pooling/}}

For our study we considered regions of size $16 \times 16$, $32 \times 32$ and $64 \times 64$.
In Fig. 3 below are the results of putting an image of a US traffic light sign through
a $16 \times 16$, $32 \times 32$, and $64 \times 64$ mean pooling filter.

\vspace*{3mm}
\begin{tabular}{c c c}
	$16 \times 16$ & $32 \times 32$ & $64 \times 64$ \\
	\includegraphics[width=0.13\textwidth]{poolsixteen} &
		\includegraphics[width=0.13\textwidth]{poolthirtytwo} &
		\includegraphics[width=0.13\textwidth]{poolsixtyfour} \\
\end{tabular}
\begin{tabular}{l}
    {\it \hspace*{4mm} Fig. 3: Image of a US traffic put through} \\
    {\it \hspace*{4mm} $16 \times 16$, $32 \times 32$, and $64 \times 64$ mean} \\
	{\it \hspace*{4mm} pooling filters.} \\
\end{tabular}

\subsection{Mogrify}
The mogrify command line utility, part of the Image Magick suite of tools\footnote{See ImageMagick's documentation of the mogrify command: \url{https://www.imagemagick.org/script/mogrify.php}}, can be used to perform a number of operations
on images, such as blurring, cropping, and compression.
We used mogrify to compress our images using the option {\tt -quality [num]}\footnote{See ImageMagick's documentation of the quality option: \url{https://www.imagemagick.org/script/command-line-options.php\#quality}}. For JPEG image compression the expected value of {\tt [num]} is an integer from 1 to 100. We used the value of 1, that applies the most extreme JPEG compression from mogrify.
In Fig. 4 below, an image of the ocean before and after it has been compressed by the command {\tt mogrify -quality 1} can be considered.

\vspace*{3mm}
\begin{tabular}{c c}
	original & compressed \\
	\includegraphics[width=0.2\textwidth]{ocean} &
		\includegraphics[width=0.2\textwidth]{mogrify} \\
\end{tabular}
\vspace*{3mm}
\begin{tabular}{l}
    {\it \hspace*{4mm} Fig. 4: Image of the ocean that has been} \\
    {\it \hspace*{4mm} compressed with the command} \\
	{\tt \hspace*{4mm} mogrify -quality 1.} \\
\end{tabular}

\subsection{Dropout}
Inspired by the concept of dropout layers in neural networks\footnote{See Srivastava et al.: \url{http://jmlr.org/papers/volume15/srivastava14a.old/srivastava14a.pdf}},
which randomly remove nodes in the neural network
to prevent overfitting, we implemented an algorithm
that randomly removes percentage $p$ of the pixels
in an image. It does this by removing
some percentage $p$ of the columns and the same percentage $p$ of the pixels
from each of the remaining columns.

We implemented this algorithm with $p=0.2,0.5$, where $p$ is a value between $0$ and $1$.
In Fig. 5 the results of putting an image of a tiger through dropout
with $p=0.2$ and $p=0.5$ can be considered.

\vspace*{3mm}
\begin{tabular}{c c c}
	$p=0.2$ & $p=0.5$ \\
	\includegraphics[width=0.2\textwidth]{dropouttwenty} &
		\includegraphics[width=0.2\textwidth]{dropoutfifty} \\
\end{tabular}
\begin{tabular}{l}
    {\it \hspace*{4mm} Fig. 5: Image of a tiger put through} \\
	{\it \hspace*{4mm} dropout with $p=0.2, 0.4$.} \\
\end{tabular}

\newpage

\section{RESULTS}

\subsection{Pre-Compression}

Once the collection of images in the various categories had been located through Creative Commons Image Search\footnote{See: \url{https://search.creativecommons.org/}} the images were then sent via http request to the Google Cloud Vision API and returned with an assigned set of labels and respective confidence score.
All of the labels assigned to the color images were
found to be intuitively accurate and relevant by the authors, and for each color image,
at least one label closely identified the image (e.g. \textquote{tiger}
for the image of a tiger and \textquote{tower} for the image of
the Tokyo Tower).

However, not all the grayscale versions of images were accurately classified according to the authors intuition.
Generic labels such as \textquote{black and white}, \textquote{black}, \textquote{white}, \textquote{art},
and \textquote{monochrome photography} were prevalent throughout grayscale images - a behavior noted later in this paper where the artefacts of compression were noted as an classifiable part of the image. According to the authors judgement 40.9\% of the grayscale images
were not closely identified, and 18.2\% of them were given solely inaccurate labels.

\subsection{Post-Compression}

Table 1 and 2 below display our analysis of how accurately images are labeled by the
Google Cloud Vision API after they are compressed using each
the methods detailed earlier in the paper. We define accuracy as the coincidence of labels between pre- and post-compression images.  We ignored more general
labels such as \textquote{photography,} \textquote{black,}
and \textquote{sky,} instead focusing on labels that either correctly or falsely
identify specific objects, landforms, etc. in the images.

\vspace*{8mm}

\hspace*{28mm} \underline{Color Images}

\vspace*{2mm}

\bgroup
\def\arraystretch{1.2} % stretches table vertically
\begin{tabular}{|p{0.25\linewidth}|p{0.25\linewidth}|p{0.25\linewidth}|}
\hline
{\bf method of compression} & {\bf \% images given at least one correct label}
	& {\bf \% images given "best" label} \\
\hline
Mogrify & 81.8\% & 68.2\% \\ % 18/22, 15/22
\hline
Dropout (20\%) & 77.3\% & 68.2\% \\ % 7/22, 15/22
\hline
Dropout (50\%) & 59.1\% & 36.4\% \\ % 13/22, 8/22
\hline
Pooling (16x16) & 90.9\% & 63.6\% \\ % 20/22, 14/22
\hline
Pooling (32x32) & 40.9\% & 18.2\% \\ % 9/22, 4/22
\hline
Pooling (64x64) & 4.5\% & 0\% \\ % 1/22, 0/22
\hline
PCA (25) & 95.5\% & 81.8\% \\ % 21/22, 18/22
\hline
PCA (10) & 77.3\% & 40.9\% \\ % 17/22, 9/22
\hline
PCA (5) & 27.3\% & 0\% \\ % 6/22, 0/22
\hline
Clustering (8) & 100\% & 90.9\% \\ % 22/22, 20/22
\hline
Clustering (4) & 86.4\% & 68.2\% \\ % 19/22, 15/22
\hline
\end{tabular}
\egroup

\vspace*{2mm}
\begin{tabular}{l}
{\it Table 1: Success rate for color images} \\
{\it labeled by Google Cloud Vision API} \\
{\it after image compression.} \\
\end{tabular}
\vspace*{4mm}

\newpage

\vspace*{10mm}

\hspace*{28mm} \underline{Grayscale Images}

\vspace*{2mm}

\bgroup
\def\arraystretch{1.2} % stretches table vertically
\begin{tabular}{|p{0.25\linewidth}|p{0.25\linewidth}|p{0.25\linewidth}|}
\hline
{\bf method of compression} & {\bf \% images given at least one correct label}
	& {\bf \% images given best label} \\
\hline
Mogrify & 59.1\% & 40.9\% \\ % updated
\hline
Dropout (20\%) & 63.6\% & 27.3\% \\ % updated 14/22, 6/22
\hline
Dropout (50\%) & 22.7\% & 0\% \\ % updated 5/22, 0/22
\hline
Pooling (16x16) & 31.8\% & 22.7\% \\ % updated 7/22, 5/22
\hline
Pooling (32x32) & 4.5\% & 4.5\%\\ % updated 1/22, 1/22
\hline
Pooling (64x64) & 0\% & 0\% \\ % updated 0, 0
\hline
PCA (25) & 68.2\% & 36.4\% \\ % updated 15/22, 8/22
\hline
PCA (10) & 31.8\% & 22.7\% \\ % updated 7/22, 5/22
\hline
PCA (5) & 18.2\% & 4.5\% \\ % updated 4/22, 1/22
\hline
Clustering (8) & 90.9\% & 50\% \\ % updated 20/22, 11/22
\hline
Clustering (4) & 59.1\% & 45.5\% \\ % updated 13/22, 10/22
\hline
\end{tabular}
\egroup

\vspace*{2mm}
\begin{tabular}{l}
{\it Table 2: Success rate for grayscale} \\
{\it images labeled by Google Cloud Vision} \\
{\it API after image compression.} \\
\end{tabular}

\def\arraystretch{1.0} % removes vertical stretch

\vspace*{4mm}

\section{ANALYSIS}

As expected, the less the image was compressed, the more accurate
the labels it was assigned were. However, with PCA compression often the intuitive most important entity of the image would gain a higher score. Grayscale images were also given significantly less accurate labels than the color images and, as mentioned previously, the artefacts of grayscale compression dominated the assigned labels - such as \textquote{black and white} and \textquote{monochrome photography}.

Of the four categories, the Animal dataset had the best
performance, most likely because the animal images had only
one dominant subject, making them easier for classification to occur and return an intuitively accurate label.
In addition, each form of
compression had its own set of behaviors that the authors noted.

The smoothening effect of posterization, using the k-means clustering
algorithm, was noted to occasionally removed textures that Google Cloud Vision API appeared to utilize to correctly classify parts of an image. For example,
the image of the waterfall, after it was put through the
4-means and 8-means clustering algorithm, was assigned the label \textquote{snow}
by the Google Cloud Vision API seemingly because the smoothened water color became white and looked more like snow (see Fig. 6 below).

\vspace*{3mm}
\begin{tabular}{c c}
	original & compressed ($k=8$) \\
	\includegraphics[width=0.2\textwidth]{waterfall} &
		\includegraphics[width=0.2\textwidth]{posterror} \\
\end{tabular}
\begin{tabular}{l}
	\hspace*{4mm} {\it Fig. 6: Potential over-smoothening of an} \\
	\hspace*{4mm} {\it image of a waterfall as a result of the } \\
	\hspace*{4mm} {\it 8-means clustering algorithm.} \\
\end{tabular}
\vspace*{3mm}

Low-rank image matrix approximation using PCA often resulted in
image distortion in a plaid-like pattern, especially in smaller
images (see Fig. 7 below). This resulted in misclassifications where again the artefacts of the compression began to dominate the labelling. Examples such as images in all four categories
false labels such as \textquote{cloth,} \textquote{pattern,}
and \textquote{line}. One of the more interesting instances of this was where the image of Piccadilly Circus in London became labeled as \textquote{tartan} at rank 5.

\vspace*{3mm}
\begin{tabular}{c c}
	original & rank-5 \\
	\includegraphics[width=0.2\textwidth]{waterfall} &
		\includegraphics[width=0.2\textwidth]{pcafiveerror} \\
\end{tabular}

\begin{tabular}{c c}
	rank-10 & rank-25 \\
	\includegraphics[width=0.2\textwidth]{pcatenerror} &
		\includegraphics[width=0.2\textwidth]{pcatwentyfiveerror} \\
\end{tabular}
\begin{tabular}{l}
	\hspace*{4mm} {\it Fig. 7: Distortions in the low-rank matrix } \\
	\hspace*{4mm} {\it approximations of an image of a waterfall.} \\
\end{tabular}
\vspace*{3mm}

The \textquote{mean pooling} operation with a $64 \times 64$ filter,
and, in some cases, the $32 \times 32$ filter as well,
blurred to the extent that they became unrecognizable and
Google Cloud Vision API could no longer give them labels that accurately represented the original source image's content (see
Fig. 3). When the filter being used with the \textquote{mean pooling}
operation is too large proportional to the size of the image, it seems to
remove all useful information to derive insight from in the image.

Image compression using the {\tt mogrify -quality 1} command
occasionally resulted in random bright colors showing up in the compressed
images that did not occur in the original images (see Fig. 8 below).

\vspace*{3mm}
\begin{tabular}{c c}
	original & compressed \\
	\includegraphics[width=0.2\textwidth]{mountains} &
		\includegraphics[width=0.2\textwidth]{mogrifyerror} \\
\end{tabular}
\begin{tabular}{l}
	\hspace*{4mm} {\it Fig. 8: Purple highlights appear in an} \\
	\hspace*{4mm} {\it image of mountains after compression} \\
	\hspace*{4mm} {\it using the command} \\
	\hspace*{4mm} {\tt mogrify -quality 1}. \\
\end{tabular}
\vspace*{3mm}

The dropout algorithm resulted in distortion of
the compressed images, almost as though the images
were being reflected on moving water.
Issues with classification from this could potentially be alleviated by adding a smoothening
factor to our dropout algorithm, for example by
using an algorithm such as seam-carving\footnote{For a further exploration
of seam-carving, see: \url{https://cs.brown.edu/courses/cs129/results/proj3/taox/}}

\section{CONCLUSION}

Overall, from the data, it appears that k-means clustering was
the best compression method in terms of maintaining the legibility
of the images for Google Cloud Vision API. Reflecting on the details of each compression techniques this starts to become a more natural hypothesis, as posterization generally simplifies images whilst retaining the overall dominant shapes in the images.

Our initial recommendation is that using at least 8-means clustering
to reduce dimensionality of image data can maintain the majority of labels from Google Cloud API. However as images become busier with a richer set of entities it is important that the value of $k$ is increased. The exact optimum value of $k$ could be an interesting research question to explore.

As this study was mainly based on the author's intuition in terms of defining an \textquote{appropriate} or \textquote{accurate} label future analysis should amend for this short coming - perhaps though establishing a more
specific definition of accuracy using group or mechanical turk\footnote{See: \url{https://www.mturk.com/}} rankings and labelling of images. This would provide us with a more reliable and scaleable \textquote{ground truth} for understanding the accuracy of Google Cloud Vision API's classification of
images pre- and post-compression. The table
above also does not consider a number of factors, such as
the certainty of different labels, and the number
of completely incorrect labels assigned to images. We encourage others to further this papers brief exploration and help to develop more robust understanding of how compression effects classification.

The authors also believe much can be gained from exploring the ways in which the different
image compression methods interact with, and possibly combat, adversarial attacks. Adversarial attacks are where
stickers, random noise, etc. are added to images or scenes, causing
neural networks to misclassify them or fail to detect them appropriately
\footnote{See: \url{https://openreview.net/pdf?id=S10qYwywf}}.
Initial research has shown promise that data reduction could be used a method to fight
adversarial attacks\footnote{See: \url{https://www.researchgate.net/publication/315890594\_Dimensionality\_Reduction\_as\_a\_Defense\_against\_Evasion\_Attacks\_on\_Machine\_ Learning\_Classifiers}} and, given the consequences as neural networks are deployed more routinely into our daily lives, the authors believe further investigation here is promptly needed.

% Extra example removed
%For exmaple, putting adversarial stickers on stop signs
%can cause neural networks to be unable to detect stop signs.\footnote{\url{http://bair.berkeley.edu/blog/2017/12/30/yolo-attack/}}

\addtolength{\textheight}{-12cm}   % This command serves to balance the column lengths
                                  % on the last page of the document manually. It shortens
                                  % the textheight of the last page by a suitable amount.
                                  % This command does not take effect until the next page
                                  % so it should come on the page before the last. Make
                                  % sure that you do not shorten the textheight too much.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section*{ACKNOWLEDGMENT}

We would like to thank Prof. Gilbert Strang for his continued support and interesting and informative lectures, as well as our fellow students.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{thebibliography}{99}

\bibitem{c1} Johnston, Nick and David Minnen. \textquote{Image Compression With Neural Networks.} Google AI Blog. Published 29 September 2016. \url{https://ai.googleblog.com/2016/09/image-compression-with-neural-networks.html}
\bibitem{c2} \textquote{Real-Time Adaptive Image Compression.} WaveOne, Inc. WaveOne. Published May 2017. \url{http://www.wave.one/icml2017/}
\bibitem{c3} \textquote{Cloud Vision API.} Google Cloud. \url{https://cloud.google.com/vision/}
\bibitem{c4} Golub, G. H., Alan Hoffman and G. W. Stewart. \textquote{A Generalization of the
	Eckart-Young-Mirsky Matrix Approximation Theorem.} \url{https://www.ime.usp.br/~jstern/miscellanea/seminario/Golub87.pdf}
\bibitem{c5} \textquote{Pooling.} UFLDL Tutorial. \url{http://ufldl.stanford.edu/tutorial/supervised/Pooling/}
\bibitem{c6} \textquote{mogrify.} ImageMagick. \url{https://www.imagemagick.org/script/mogrify.php}
\bibitem{c7} \textquote{command line options.} ImageMagick. \url{https://www.imagemagick.org/script/command-line-options.php\#quality}
\bibitem{c8} Srivastava, Nitish, et. al. \textquote{Dropout: A Simple Way to Prevent Neural Networks
	from Overfitting.} {\it Journal of Machine Learning Research}, 15 (2014): 1929-1958.
	Published 14 June 2014. \url{http://jmlr.org/papers/volume15/srivastava14a.old/srivastava14a.pdf}
\bibitem{c9} Tao, Xiaofeng. \textquote{Seam Carving.} \url{https://cs.brown.edu/courses/cs129/results/proj3/taox/}
\bibitem{c10} Gopalakrishnan, Soorya, et. al. \textquote{Combating Adversarial
	Attacks Using Sparse Representations.} ICLR 2018. \url{https://openreview.net/pdf?id=S10qYwywf}
\bibitem{c11} Bhagoji, Arjun, et. al. \textquote{Dimensionality Reduction as a Defense against Evasion Attacks on Machine Learning Classifiers.} \url{https://www.researchgate.net/publication/315890594\_Dimensionality\_Reduction\_as\_a\_Defense\_against\_Evasion\_Attacks\_on\_Machine\_ Learning\_Classifiers}
\bibitem{c12} Evtimov, Ivan, et. al. \textquote{Physical Adversarial Examples Against Deep
	Learning Networks.} Berkeley Artificial Intelligence Research. Published 30 December 2017. \url{http://bair.berkeley.edu/blog/2017/12/30/yolo-attack/}

\end{thebibliography}

\end{document}


\documentclass[10pt]{article}
\title{Adaptive Algebraic Multigrid with Graph Modularity Matching for Symmetric Positive Definite Matrices}

\begin{document}
\maketitle

\section{Introduction}
This library implements some tools to explore techniques of adaptive AMG.
In classical AMG, the performance of the solver depends on the construction
of the relaxation and coarse-grid correction process (cite M. BREZINA).
Convergence of the method slows when the residual approaches the rhs of the
system (the iterate is in the `near null space` of the system).
This happens when the current iterate is an approximation of a minimal
eigenvalue of the matrix. Using this near null component and utilizing graph
modularity, we can construct a new relaxation and coarse-grid correction.
This provides the adaptivity when convergence suffers. Combining multiple of
these adaptive relaxation steps can provide a composite adaptive multigrid method.

\section{Library Details}
The library is broken into 3 main modules: partitioner, preconditioner, and solver.

\subsection{Partitioner module}
Here is implemented an algorithm described in (quiring 2019) which is graph pairwise
matching algorithm that only merges vertices where the change in modularity is
positive. The only requirement for this algorithm is that the matrix has positive
row sums. To convert our system to have positive rowsums, we construct a new system
with a matching sparsity pattern using a `near null` component. The modularity
matching algorithm applied to this new system then gives a hierarchy of interpolation
matrices that can be applied the the original system to create the coarse problem.

\subsection{Preconditioner module}
This module is called preconditioner but maybe a better name could be smoothers or
methods. In this module there are functions that take a system and provide the inverse
action of some method than can be used in an iterative solver. Currently, one can build
L1 smoother, symmetric/forward/backward Gauss-Seidel, multilevel L1, and multilevel
Gauss-Seidel. All of these can be used as methods for the stationary solver but only
the symmetric methods can be used as a preconditioner in the conjugate gradient method.

\subsection{Solver module}
This module implements different iterative solvers that accept linear systems, a
method to base the solver on, and some parameters for the solver. Currently, only
the classical stationary iterative method and preconditioned conjugate gradient is
implemented. Soon I will be adding an adaptive solver to this module.

\section{Example Usage}
The library is currently packaged with a CLI binary that can show how the library
can be used to test different solvers on your systems.

\section{Goals}
Classical AMG often breaks down on highly anisotropic problems often found in finite
element methods. To address this failure, sometimes specific knowledge of the system
can be used to carefully construct a preconidtioner. The goal of this research is to
test some adaptive methods that don't require any knowledge about the system and
measure the computational complexity and memory demands of these methods on systems
where classical AMG performs poorly.

\end{document}


\documentclass{beamer}
\usepackage{csquotes}
\usepackage{tikz}
\usetikzlibrary{arrows,positioning,shapes.geometric, calc}
\usepackage{amsmath}
\usepackage{listings, xcolor}
\usepackage{lmodern}
\usepackage{adjustbox}
\usepackage{booktabs}
\usepackage{colortbl}
\usepackage{caption}
\usepackage{icomma}
\usepackage{bigstrut}
\usepackage{geometry}
\usepackage{subfigure}

\DeclareMathOperator*{\argmin}{argmin}

\usetheme{metropolis}           % Use metropolis theme
\title{$L_0$-norm and Large Scale Genetic Machine Learning}
\date{\today}
\author{Robert M. Porsch}
\institute{Center of Genomic Science}
\begin{document}
\maketitle

\begin{frame}[t]{Introduction}
  I have two objectives:
  \begin{itemize}
    \item Introduce $L_0$ for penalized regressions
    \item Data engineering for machine learning on genetic data
  \end{itemize}
  Literature used:\\
  Learning Sparse Neural Networks through $L_0$ Regularization \\
  \emph{Louizos, Max Kingma} \\
  \vspace{0.5cm}
  The Variational Garrote \\
  \emph{Kappen,  Gomez}
\end{frame}

\begin{frame}[t]{Penalized Regression}
  Let $\mathcal{D}$ the dataset consisting of $N$ input-output pairs $\{(x_1, y_1), \ldots, (x_N, y_N)\}$ and consider the following regularized minimization procedure

  \begin{equation}
    \mathcal{R}(\theta) = \frac{1}{N} ( \sum^N_{i=1} \mathcal{L}(h(x_i; \theta), y_i)) + \lambda\mathcal{P}(\theta)
  \end{equation}

  With $\theta^* = \underset{\theta}{\argmin}\{\mathcal{R}(\theta)\}$.

  $\mathcal{P}(\theta)$ is a penalization function for the parameters $\theta$.
\end{frame}

\begin{frame}[t]{Forms of $\mathcal{P}(\theta)$}
  $L_1$-norm:
  \begin{equation}
    \mathcal{P}_{L_1}(\theta) = ||\theta||_1 = \sum^{|\theta|}_{i=1} |\theta_i|
  \end{equation}
  or $L_2$-norm:
  \begin{equation}
    \mathcal{P}_{L_2}(\theta) = ||\theta||_2 =  \sum^{|\theta|}_{i=1} \theta_i^2
  \end{equation}
  When being more general. Let $p \geq 1$
  \begin{equation}
    \mathcal{P}_{L_p}(\theta) = ||\theta||_p = (\sum^{|\theta|}_{i=1} |\theta_i|^{p})^{1/p}
  \end{equation}
\end{frame}

\begin{frame}[t]{What about $p=0$?}
  There are \textbf{two} $L_0$-norms:
  \begin{itemize}
    \item $L_0$-norm (Established by Banach and not relevant here)
    \item $L_0$-`norm' (Established by David Donoho)
  \end{itemize}
  The $\L_0$-`norm' is not a real norm, we just call it because it is the limit of the $L_p$ norm:
  \begin{equation}
    \matcal{P}_{L_0}(\theta) = ||\theta||_0 = \lim_{p \rightarrow 0} (\sum^{|\theta|}_{i=1} |\theta_i|^{p})^{1/p}
  \end{equation}
  In general we define it as
  \begin{equation}
    \matcal{P}_{L_0}(\theta) = ||\theta||_0 = \sum^{|\theta|}_{j=1} \mathcal{I}[\theta_j \neq 0]
  \end{equation}
  So its just the count of non-zero parameters.
\end{frame}

\begin{frame}[t]{Some graphical representation always helps}
  \begin{figure}[htpb]
    \centering
    \includegraphics[width=0.8\linewidth]{norms.png}
    \caption{Various Norms}\label{fig:norms}
  \end{figure}
  Furthermore,
  \begin{itemize}
    \item same as the other norms, it encourages sparsity in the parameters
    \item $L_0$ does induce any shrinkage in the parameters
  \end{itemize}
\end{frame}

\begin{frame}[t]{There is a problem however \ldots}
  Optimization is computationally intractable under the $L_0$ penalty,
  \begin{itemize}
    \item non-differentiable %TODO check this again
    \item $2^{|\theta|}$ possible states
    \item NP-hard problem
  \end{itemize}
  So there is a need to relax the discrete nature of $L_0$ to allow for efficient optimizations.
\end{frame}

\begin{frame}[t]{The General Recipe}
  The first step is to reformulate the $L_0$ norm under the parameters $\theta$.
  Hence let,
  \begin{equation}
    \begin{matrix}
      \theta_j = \tilde{\theta_j}z_j, & z_j \in \{0, 1\}, & \tilde{\theta}_j \neq 0
    \end{matrix}
  \end{equation}
  Therefore $z_j$ can be considered as binary gates (parameter has an effect).

  Then we can reformulate the minimization from Eq. 1 by letting $q(z_j|\pi_j) = Bern(\pi_j)$
  \begin{equation}
    \mathcal{R}(\tilde{\theta}, \pi) = \mathbb{E}_{q(z|\pi)} [\frac{1}{N} ( \sum^N_{i=1} \mathcal{L}(h(x_i; \tilde{\theta} \otimes z), y_i)] + \lambda \sum^{|\theta|}_{j=1} \pi_j
  \end{equation}
  with  $\tilde{\theta}^*, \pi^* = \underset{{\tilde{\theta}, \pi}}{\argmin} \{\mathcal{R}(\tilde{\theta}, \pi)\}$

  However, the discrete nature of $z$ makes it still difficult to minimize $\pi$.
  A solution is to give $z$ a \textbf{hard-sigmoid} function.
\end{frame}

\begin{frame}[t]{Hard-sigmoid function}
  \small
  Let s be a continuous random variable with a distribution $q(s)$ with parameter $\phi$.
  Then we can give the gates $z$ a hard sigmoid function with:
  \begin{equation}
    \begin{align*}
      s \sim& q(s|\phi) \\
      z =& \min(1, \max(0, s))
    \end{align*}
  \end{equation}
  Then the probability of the gates being non-zero is
  \begin{equation}
    q(z \neq 0 | \delata) =  1 - Q(s\leq 0 | \phi)
  \end{equation}
  in which $Q(\cdot)$ is the cumulative distribution function of s
  \begin{equation}
    \begin{split}
      \mathcal{R}(\tilde{\theta}, \phi) = \mathbb{E}_{q(s|\phi)} [\frac{1}{N} ( \sum^N_{i=1} \mathcal{L}(h(x_i; \tilde{\theta} \otimes g(s)), y_i)] + \\
      \lambda \sum^{|\theta|}_{j=1} (1 - Q(s_j \leq 0 | \phi_j))
    \end{split}
  \end{equation}
  with $\theta^*, \phi^* = \underset{{\tilde{\theta}, \phi}}{\argmin} \{\mathcal{R}(\tilde{\theta}, \phi)\}$ and $g(\cdot) = \min(1, \max(0, \cdot))$
\end{frame}

\begin{frame}[t]{Some graphical help}
  \begin{figure}[htpb]
    \centering
    \includegraphics[width=0.5\linewidth]{sigmoids.png}
    \caption{Sigmoid functions}\label{fig:sigmoids}
  \end{figure}
  However, we still need to chose a function $q(s)$.
\end{frame}

\begin{frame}[t]{The Hard Concrete Distribution}
  The literature suggests to use a hard concrete distribution as a smoothing function $q(s)$.
  The parameters of the distribution are $\phi = (\log \alpha, \beta)$ and can be stretched to $(\gamma, \zeta)$ intervals.
  \begin{equation}
    \begin{align*}
      u \sim \mathcal{U}(0,1) \\
      s = Sigmoid((\log u - \log(1-u) + \log\alpha)/\beta) \\
      \bar{s} = s(\zeta - \gamma) + \gamma
    \end{align*}
  \end{equation}
  \begin{figure}[htpb]
    \centering
    \includegraphics[width=0.5\linewidth]{hard_concrete.png}
    \caption{Sample from the Hard Concrete Distribution}\label{fig:hard_concrete}
  \end{figure}
\end{frame}

\section{Implementation and Plumbing}

\begin{frame}[t]{Implementation}
  Use of stochastic gradient descent (SGD) with Pytorch.
  \begin{itemize}
    \item Not as optimal as coordinated descent, but reasonably fast
    \item Pytorch implementation was easier than tensorflow
    \item Works fine on test data
  \end{itemize}
  \textbf{Challenge:} \\
  The plumbing (data engineering)
  \\
  \begin{itemize}
    \item Process of the UKB in parallel efficiently
    \item Hail is limited in its flexibility
    \item Currently using Dask on our cluster
  \end{itemize}
\end{frame}

\begin{frame}[t]{Task processing}
  \begin{figure}[htpb]
    \centering
    \includegraphics[width=0.8\linewidth]{Dask_processing.png}
  \end{figure}
\end{frame}

\begin{frame}[t]{Using Dask}
  Dask lets you specify each task in a lazy fashion (computation is done when you need it).
  \begin{figure}[htpb]
    \centering
    \includegraphics[width=0.7\linewidth]{task_plot.png}
  \end{figure}
\end{frame}

\begin{frame}[t]{Some Results}
 \textbf{In process}
\end{frame}
\end{document}


\chapter{Image Fusion }\label{sec:Fusion}
%\section{Introduction}

Satellite sensors present an important diversity in terms of characteristics.
Some provide a high spatial resolution while other focus on providing several
spectral bands. The fusion process brings the information from different
sensors with different characteristics together to get the best of both
worlds.

Most of the fusion methods in the remote sensing community deal with
the {\em pansharpening technique}. This fusion combines the image from
the PANchromatic sensor of one satellite (high spatial resolution
data) with the multispectral (XS) data (lower resolution in several
spectral bands) to generate images with a high resolution and several
spectral bands. Several advantages make this situation easier:

\begin{itemize}
\item PAN and XS images are taken simultaneously from the same satellite (or
with a very short delay);
\item the imaged area is common to both scenes;
\item many satellites provide these data (SPOT 1-5, Quickbird, Pleiades)
\end{itemize}

This case is well-studied in the literature and many methods exist. Only very
few are available in OTB now but this should evolve soon.

\section{Simple Pan Sharpening}\label{secPanSharpening}

A simple way to view the pan-sharpening of data is to consider that,
at the same resolution,  the panchromatic channel is the sum of the XS
channel. After putting the two images in the same geometry, after
orthorectification (see chapter \ref{sec:Ortho}) with an oversampling of the XS image, we can proceed to the data fusion.

The idea is to apply a low pass filter to the panchromatic band to give it a spectral content (in the Fourier domain) equivalent to the XS data. Then we normalize the XS data with this low-pass panchromatic and multiply the result with the original panchromatic band.

The process is described on figure \ref{fig:PanSharpening}.

\begin{figure}[th]
  \begin{tikzpicture}[scale=0.25]
    \tiny
    \draw[fill=black!10] (-1,-12) rectangle (53,17);
	\filldraw[fill=red!50!brown,fill opacity=0.75] (3,-4) rectangle +(6,6);
	\filldraw[fill=red,fill opacity=0.75] (2,-5) rectangle +(6,6);
	\filldraw[fill=green,fill opacity=0.75] (1,-6) rectangle +(6,6);
	\filldraw[fill=blue,fill opacity=0.75] (0,-7) rectangle +(6,6);
	\node[anchor=center, text width= 1cm] (XS) at (4,-8) {XS};
	\filldraw[fill=black,fill opacity=0.75] (1,5) rectangle +(6,6);
	\node[anchor=center, text width= 1cm] (PAN) at (4,4) {PAN};

	\draw[->,thick] (7,8) --  +(6,0);

	\draw[fill=black!30,rounded corners=2pt] (13,7) rectangle +(5,2);
	\node[text width= 1.3cm] (Lowpass) at (16,8) {Low Pass};

	\draw[->,thick] (18,8) --  ++(6,0) -- ++(0,-3);
	\draw[->,thick] (9,-1) --  ++(15,0) -- ++(0,4);

	\draw[fill=black!30,rounded corners=2pt] (23,3) rectangle +(2,2);
	\node[text width= 1.3cm] (Div) at (26.5,4) {/};

	\draw[->,thick] (25,4) --  ++(6,0);
	\draw[->,thick] (10,8) --  ++(0,4) -- ++(22,0) -- ++(0,-7);

	\draw[fill=black!30,rounded corners=2pt] (31,3) rectangle +(2,2);
	\node[anchor=center, text width= 1cm] (Mult) at (33.7,4) {$\times$};

	\draw[->,thick] (33,4) --  ++(6,0);

	\filldraw[fill=red!50!brown!50!black,fill opacity=0.75] (42,4) rectangle +(6,6);
	\filldraw[fill=red!50!black,fill opacity=0.75] (41,3) rectangle +(6,6);
	\filldraw[fill=green!50!black,fill opacity=0.75] (40,2) rectangle +(6,6);
	\filldraw[fill=blue!50!black,fill opacity=0.75] (39,1) rectangle +(6,6);
	\node[anchor=center, text width= 2cm] (PANXS) at (44,0) {PAN + XS};

  \end{tikzpicture}

\itkcaption[Simple pan-sharpening]{Simple pan-sharpening procedure.}
\label{fig:PanSharpening}
\end{figure}

\input{PanSharpeningExample.tex}

\section{Bayesian Data Fusion}\label{secBayesian}

\input{BayesianFusionImageFilter.tex}