\chapter{Preliminaries and related work}
\label{preliminaries}

In this section we briefly introduce and explain the necessary theoretical background, upon which we build later on.

We first define the concepts of Machine learning (ML) and Deep neural networks (DNNs). We explain, how DNNs can learn patterns from data by using powerful gradient-based stochastic optimization algorithm called Stochastic gradient descent (SGD). Then we talk about a subset of DNNs that perform very well on image data called Convolutional neural networks (CNNs).

Finally, we describe the systemic vulnerability of DNNs and define an adversarial attack. Then we explain related terms like adversarial defenses, adversarial examples, adversarial robustness and different adversarial attack threat models and present current state of the field.

\section{Deep learning basics}

\subsection{Machine learning}
The topic of machine learning has been introduced many times over in other books and papers. \cite{10.5555/2380985}, \cite{10.5555/1162264} and \cite{Goodfellow-et-al-2016} all provide a comprehensive exposition to the field. Because of the abundance of many different kinds of resources, we don't feel the need to reinvent the wheel and come up with yet another machine learning introduction.

To put forward the most accepted definition of machine learning, the \cite{10.5555/2380985} lay it as follows: \\

“A computer program is said to learn from experience $E$ with respect to some class of tasks $T$ and performance measure $P$, if its performance at tasks in $T$, as measured by $P$, improves with experience $E$.”

\subsection{Deep neural networks}
In our case the "computer program learning from experience" will be in most cases a deep neural network (DNN). We will introduce neural networks only very briefly at a high level as more details can be easily found in \cite{Goodfellow-et-al-2016}.

DNN is in its essence a parametrized mapping $y = f(x;\theta)$, where $x$ are the network inputs, $\theta$ are the parameters and $y$ are the outputs of the network. The "network" in its name comes from the fact that the function computation is described as an evaluation of a computational directed acyclic graph (DAG), the structure of which depends on the exact network architecture.

During evaluation, each node in the graph is asociated with a number called "activation", This activation value is computed from the values of other nodes and from the relevant parameters $\theta$. It is then passed through a non-linear activation function $a(x)$. For a long time the $sigmoid(x)$ activation function was very popular. Nowdays $ReLU(x) = \max(0, x)$ (Rectified Linear Unit) is used more often as it is better suited for deeper networks.

The nodes in the DAG are sometimes referred to as "neurons", as there is an analogy to the functioning of biological neurons.

The function $f(x;\theta)$ can be often decomposed into several computational steps: $f(x;\theta) = f_n(f_{n-1}(...f_2(f_1(x; \theta_1);\theta_2)...;\theta_{n-1});\theta_n)$. In this decomposition the individual functions $f_i(x_i; \theta_i)$ are called "layers". When the number of layers is large, the network is said to be deep. The successive computations of $f_1(x_1, \theta_1)$, $f_2(x_2, \theta_2)$ up to $f_n(x_n, \theta_n)$ is called a "forward pass".

\subsection{Network training}
If we have a dataset $D$ consisting of input datapoints $x_i$ and their corresponding $y_i$ values called "labels", we want the neural network function $f(x, \theta)$ to express the relationship between $x_i$ and $y_i$. Depending on the complexity of the network, the parameters $\theta$ can be often set accordingly, such that $f(x_i, \theta) \approx y_i$. This ability to approximate the data well with the right $\theta$ is called "network's capacity". How good is the neural network's approximation is measured by a loss function $\mathcal{L}(D, f(x, \theta))$. Because the network models the relationship between $x_i$ and $y_i$, it is sometimes just called a "model".

The process of training a neural network tries to minimize the training loss by optimizing the parameters. Formally, we search for $\theta$ = $\argmin{\theta}[ \mathcal{L}(D, f(x, \theta))]$.

This optimization is in practice done iteratively using an algorithm called stochastic gradient descent (SGD). It involves computing the gradient of the loss with respect to the parameters $\theta$: $\nabla_{\theta} \mathcal{L}(D, f(x, \theta))$. It then updates the parameters in the opposite direction of the gradient to hopefully lower the loss.

The partial derivates $\frac{\partial \mathcal{L}(D, f(x, \theta))}{\partial \theta_i}$ can be efficiently computed by an algorithm called "backpropagation". It is very similar to a forward pass, but it starts from the loss and proceeds in the opposite direction of the DAG. Detailed explanation can be again found in \cite{Goodfellow-et-al-2016}. It is also icredibly well explained in the \href{https://www.youtube.com/playlist?list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi}{YouTube video series about neural networks} by 3Blue1Brown (Grant Sanderson)

\subsection{Convolutional neural networks}
Convolutional neural network (CNN) is a network using a special type of layers called "convolutional layers". CNNs were invented more than 20 years ago (\cite{LeCun1998ConvolutionalNF}) and they are powerful feature extractors compressing highly dimensional spatial inputs to a smaller feature vector. They are used mainly when dealling with images, but in general, convolutions are useful for any type of spatial data with translational invarience property. When a network is asked to perform an image classification, it is often desireable for an object in the left part of the image to be detected in the same way as an object on the right side. Convolutional layer achives this by convolving each part of the input image with the same kernel, thus sharing weights and greatly reducing the parameter count. Each convolutional kernel is sometimes referred to as "filter", because the kernel convolution operation resembles a sobol operator used in classical computer vision and image processing. CNNs usually stack multiple convolutional layers on top of each other with each subsequent layer having more filters, but reducing the spatial dimension. The spatial dimension reduction is achived by an average or max pooling layers that follow periodically after each set of convolutional layers. The final convolution outputs are either pooled together to reduce the spatial dimension to 1x1, or they are passed through a one or two densely connected layers to produce the extracted features.

\section{Related Work}
\label{related_work}

\subsection{Definitions}
As it was outlined in chapter \ref{introduction}, adversarial attacks are methods of producing adversarial examples. Given an original input $x$ and a classifier model $y = f(x), y \in C$, an adversarial example $x_{adv} = x + \delta$ is a slightly pertubed version of $x$, such that $f(x_{adv}) \neq y$. By "slightly" we mean $\lVert \delta \rVert < \epsilon$. In this work we focus on image adversarial examples achieving misclasification on image classification models, albeit the concept of adversarial attacks is applicable across different types of neural networks and different types of input data.

One can be also interested in the minimal pertubation $\delta_{min}$ needed to change the model classification:

$$\delta_{min} = \argmin{\delta} \lVert \delta \rVert \quad \text{such that} \quad f(x + \delta) \neq y$$

This brings us to the definition of robustness $r(x, f)$ given an input $x$:
$$r(x, f) = \lVert \delta_{min} \rVert$$

More informative can be the expected robustness across all of our data $D$, the global robustness of a model:

$$\rho (f) = \underset{x \sim \mathcal D}{\mathbb E} r(x, f)$$

Whether we do or do not care about the nature of misclassification distinguishes the targeted and untargeted attacks:

\begin{itemize}
    \item targeted attacks require that $f(x_{adv}) = t, t \in C$
    \item untargeted attacks only aim for unspecified misclasification - $f(x_{adv}) \neq y$
\end{itemize}

The constraint on pertuation size $\lVert \delta \rVert$ makes sure, that the adversarial example $x_{adv}$ looks almost the same as the original $x$. That said, there are many valid choices for the metric $\lVert \cdot \rVert$ that are commonly used:

\begin{itemize}
    \item $l_{0}$ - the number of non-zero components of $\delta$.
    \item $l_{1}$ - $\lVert \delta \rVert = \underset{i}{\sum} \lvert \delta_{i} \rvert$
    \item $l_{2}$ - $\lVert \delta \rVert = \sqrt{\sum \delta_{i}^2}$
    \item $l_{inf}$ - $\lVert \delta \rVert = \underset{i}{\max \delta_{i}}$
\end{itemize}

\subsection{Whitebox attacks}
Whitebox adversarial attacks assume full knowledge of the target model, which allows for efficient computation of the the gradients with respect to the input. This is in contrast with blackbox attacks \ref{blackbox_attack}, where the attacked model is available only as a blackbox and as such the gradients can be only estimated using sampling for example.

\cite{Biggio2013EvasionAA} was the first to point out the inherent vulnerability of machine learning models by attacking SVMs and multi-layer perceptrons.

In the same year \cite{szegedy2014intriguing} used an L-BFGS optimization algorithm that leverages estimates of second order partial derivates information to find minimally distorted adversarial example $x$ by solving the following:
$$\min \lVert x - x_{adv} \rVert_2^2 \quad \text{s.t.} \quad f(x_{adv}) = t \quad \text{and} \quad x_{adv} \in [0, 1]^m$$

In 2014 \cite{goodfellow2015explaining} introduced the Fast gradient sign method (FGSM) which we use in \ref{fgsm_exp}. It involves doing only one backpropagation, so it is very efficient and is often use for an adversarial training. Adversarial training is a method of dynamicaly extending the training dataset by adversarial examples generated on the fly. FGSM is well suited for this, because it's fast.

In 2015 Jacobian-based saliency map attack (JSMA) was proposed by \cite{papernot2015limitations}.

Various methods trying to reduce the susceptibility of neural networks to adversarial examples are being proposed around this time. These methods are called "adversarial defences". One of those is the previously mentioned adversarial training. Few others include for example JPG compression, stochastic augmentations, feature distilation etc.

As an attempt to break a specific "distilation defense", \cite{carlini2017evaluating} propose their C\&W attack. FGSM and L-BFGS aren't strong-enough against the distilation defense, because the gradients are orders of magnitude smaller than in the case of an undefended target.

They reframe the optimization problem as:
$$\min \lVert x - x_{adv} \rVert^2_2 + c \cdot f(x', t) \quad \text{s.t.} \quad x' \in [0, 1]^m \quad \text{where} \quad f(x', t) = (max_{i \neq t} Z(x')_i - Z(x')_t)^+$$

Their reframing allows for adaptive scaling of the objective being optimized and as such doesn't suffer as much from the small gradients. Furthemore, their objective is different from the objectives optimized previously, which also helps the C\&W attack.

For a more comprehensive overview of whitebox attacks and adversarial attacks in general we will point the reader to the review paper \cite{xu2019adversarial}.

\subsection{Transferability}
\label{transferability}
It was demonstrated \cite{tramer2017space} that adversarial examples generated by whitebox attacks have high probability of decieving another neural network. This transferability of adeversarial examples is made more severe when the two models have similar architectures and when the datasets used to train them are the same.

\subsection{Blackbox attacks}
\label{blackbox_attack}
In the blackbox setting the access to the target model is limited and only the final output is available to the attacker.

Nevertheless, the transferability proprety \ref{transferability} can be used to fool the target model without any further knowledge.

A different approach of attacking a blackbox model can be to estimate the gradients by sampling to make up for the inability to backpropage through it. (\cite{Chen2017ZOOZO}, \cite{Ilyas2018BlackboxAA})

There is yet another class of blackbox attacks. These attacks don't rely on the gradient information at all, but instead they use random search to find the adversarial pertubation. SquareAttack \cite{Andriushchenko2020SquareAA}, which we utilize in this work, is one example of such attacks.

In chapter \ref{our_approach} we talk more about the different blackbox attacks and about possible ways of combining different approaches together.

    % - maximizes classification for t and minimizes classification for all other classes
    % - so called **"margin loss"**
    % - there are many different ways to define valid loss function, but margin loss seems to work the best (probably)
%   - only difference in formulation is that L-BFGS  uses **cross-entropy** instead of **margin loss**
    % - this formulation has a nice property, that when $C(x') = t$, then $f(x', t) = 0$, and the algorithm switches to optimizing only the **distance part of the objective**
    % - efficient for finding the minimally distorted adversarial example
% - quite very strong attack, useful for benchmarking