\documentclass[main.tex]{subfiles}

\begin{document}
\sloppy
\chapter{Introduction}

The human brain has been a long standing topic of research in the scientific community. Many attempts have been made to model its behaviour using methods such as modeling the low level architecture on a neurological level \cite{frank1957perceptron} and a more abstract stochastic approach using  Bayesian inference \cite{bourlard1988auto} .

Being able to create a general form of artificial intelligence will allow for smarter robots that can learn and perform tasks in order to assist humans. However, robots are not easy to program and each task requires a specialised set of instructions for it to be executed properly. Such programs have various (dynamical) models and control policies embedded into them in order for the robot to function properly. Intrinsic system parameters, policies and controller gains are something which the human brain can learn over time \cite{friston2010free,knill2004bayesian, kawato1999internal}, yet a robot obviously has no such capability and tedious tweaking is required for everything to work properly.

There have been many attempts to reconcile the alluring anatomy of the brain and mathematical algorithms \cite{friston2003learning}, yet the most prevalent approach is generative modelling which has been widely adopted by the machine learning community. By combining the flexibility of \textit{neural networks} and probability theory these unsupervised learning algorithms can synthesise numbers, landscapes and even faces by learning latent features from training data and are called \textit{Deep Generative Models}. They are believed to hold the promise to general AI \cite{Goodfellow-et-al-2016}, yet  their application in robotics is scarce. Therefore, we are interested in applying such learning algorithms in the robotics field and more specifically in visuomotor control of an autonomous agent.

\section{Background}
In order to properly justify our research in this literature study we will define a problem space that needs solving. We will assume a 6-DOF robotic arm equipped with a camera as our autonomous agent. We wish for the agent to reach towards a visual point in space without having any knowledge of how the image is mapped to its internal kinematics. In other words, the agent has to figure out the positions of its joints based on image data which is also known as visual servoing. In addition, the agent should have some sort of \textit{generative model} which is a learned, low dimensional and stochastic representation of its kinematic model. In fact, the reason to have a low dimensional, or latent, representation is because storing all the possible states as high dimensional images makes computations difficult and impractical.

Visual servoing has been explored extensively yet the implementations are generally model based \cite{hosoda1994versatile} or use reinforcement learning \cite{levine2016learning}. In other words, so far no application exists where a complex 6-DOF agent learns a generative kinematic model.

\section{Research Goal}

This practical example will be used for exploration and justification of various techniques. A set of requirements and conditions will be presented that can help us maintaining focus. However, in order to allow for further research and extension the research question will be formulated more generally.

In this literature study we aim to answer the following research question:
\begin{quotation}
\begin{tabular}{|p{10cm}}
\textit{What unsupervised deep generative model allows an agent to infer latent system features by observing rich sensory data?}
\end{tabular}
\end{quotation}

Formulating this question in such a general way allows us to explore a wider range of literature, yet the requirements for the application keeps the research focussed. We have purposefully used the term \textit{latent system features} to indicate that the technique we propose should be valid for a wide range of applications such as both \textit{kinematics} and \textit{dynamics}. For simplicity we will not consider dynamics since robotic arms are well behaved in that respect [].

We define a set of requirements against which we can compare and contrast various techniques found in literature. Considering the learning nature of the brain using a generative model we also seek a technique that \textbf{i)} can infer a latent low dimensional representation from high dimensional sensory input such as images. Furthermore, the technique should allow us to \textbf{ii)} sample a generative model in order to make predictions. Such predictions could be used to estimate the velocity based on two images, for example.
Finally, since we are dealing with physical actuated systems the technique must also \textbf{iii)} provide tractable, robust and stable training dynamics.

In sections 2 and 3 we will discuss the background and relevance of neural networks and generative models, respectively. Section 4 investigates current state of the art techniques which have the potential of answering our research question. Section 5 compares and contrasts these methods against our set of requirements. We finally conclude with a short summary of this literature study and the chosen technique.

\chapter{Neural Networks}
Neural networks are the foundation of the machine learning field of research and have been studied for more than half a century. A wide variety of neural networks exists, each using a different architecture to perform specific tasks. We will focus on unsupervised learning which consists predominantly of optimisation problems (e.g. gradient descent).

Recall that our agent is a robotic arm supplied with a camera. In terms of this literature study we aim to find a network (or a combination thereof) that can be used in vision based applications. In addition we want our agent to learn a low dimensional representation of its kinematics from the high dimensional visual data. Therefore, the network  must perform a form a dimensionality reduction. Since the landscape of neural networks is vast it is unpractical to give an overview of all possible types. Consequently, this chapter identifies only several neural networks and aims to justify why and how they are relevant to the research goal.

It is assumed that the reader has some working knowledge on neural networks. Therefore, the basic fundamentals of neural networks will not be covered in depth.

\section{Deep Neural Network}
A neural network is a set of connected nodes (neurons) which can have multiple inputs where each node then outputs a single value. Each connection has a weight and a bias. Each node takes the sum of the weights and biases connected to that node and is fed through some activation function. A clear graphical representation can be seen in figure \ref{fig:ann}

[figure of neural network]

Single layer neural networks, which only have one input layer and one output layer are only capable of modelling linear functions. In order to model non-linear functions, which is necessary in vision applications, we have to use \textit{deep} neural networks \cite{LeCun2015}. Such networks are characterised by having one or more \textit{hidden} layers in addition to the regular input and output layer.

What makes neural networks so powerful is their capacity to optimise their weights and biases through back-propagation. By optimising a cost function through, for example, a gradient descent method we can find the weights and biases that give us the most optimal results.

Deep neural networks with back propagation give us a very strong foundation which we can extend and apply to more exotic architectures.

\section{Autoencoder}
We have seen that deep neural nets form a solid foundation for various other types of networks. One of the goals of this literature study is to find a neural network that can learn a low dimensional (latent) representation from high dimensional data (e.g. images).

An autoencoder (AE) is used for such \textit{dimensionality reduction} applications. An AE network has a symmetric hourglass shape where the input and output layers have the same amount of nodes. The layers in between keep shrinking towards the center. The goal of the autoencoder is to ensure that the output is as similar as possible to the input. The hourglass shape forces an autoencoder to only extract relevant information which can be stored in a latent space (encoding) consisting of only several neurons as seen in figure \ref{fig:ae}. The latent representation is then fed through a decoder network which reconstructs the input. The network is penalised for errors in reconstruction. More formally, an AE learns a function $h$ with weights and biases, $w$ and $b$, that tries to approximate $x$ as much as possible, as shown in equation \eqref{eq:ae_approx}.

\begin{equation}
\label{eq:ae_approx}
  h_{w,b}(x) \approx \hat{x}
\end{equation}

% TODO: Add autoencoder figure

To summarise, autoencoders enable us to learn a low dimensional representation of our high dimensional data. Once an autoencoder is trained it can also be used to reconstruct corrupt images or interpret realistic scenes from simulated training data used in transfer learning \cite{zhuang2015supervised,kandaswamy2014improving}. This can be very beneficial when training our agent in a simulation, for example, and then using the trained network on a physical system.

\section{Boltzmann Machine}
[explanation is not entirely clear here, nor is the connection to the rest of this document]

Similar to an autoencoder, a Restricted Boltzman Machine (RBM) aims to reconstruct data in an unsupervised manner. As shown in figure \ref{fig:boltzmann_machine} an RBM has a single hidden layer and a layer which functions as both the input and output layer (i.e. visible units) which depends on the training phase. Interestingly, the RBM learns a \textit{stochastic} representation of the input data instead of storing variables in a latent space like the autoencoder does. This is exceedingly interesting since the we would have to deal with the variability of natural data as seen in images \cite{chen2003continuous}.

During the forward pass of the training phase the activation value $a_j$ for a hidden node is determined by some probabilistic distribution $p(a_j|x_i;w_{ij})$ where $x_i$ is a visible node and $w_{ij}$ is the weight connecting node $i$ to node $j$. The type of this probability distribution is dependant on the implementation of the RBM.

Now on the backward pass we let the RBM reconstruct the input and compare it to the real data, just as with an autoencoder. This comparison can be done with a Kullbackk-Leibler (KL) divergence where we try to minimise the difference in the two probability distributions, as shown in figure \ref{fig:kl_div}. We keep updating the weights until the optimisation converges or a maximum number of iterations (epochs) has been reached.

Thus, an RBM is a probabilistic spin on the AE where the RBM tries to \textit{infer} latent features of the dataset in order to make predictions when presented with new but comparable data. This is also called generative learning or \textit{generative modeling} which will be explored in more detail in the next chapter. Even though this neural network is similar to an AE, the stochastic nature of an RBM can be a preferred attribute since it resembles the variability of natural data.

\section{Convolutional Neural Network}
[Will obviously be expanded but it is trivial]

Images can have very large dimensions. A simple image that is only 20x20 pixels already requires 400 input neurons. Using a Convolutional Neural Network (CNN) allows us to take a relatively large image and use a scanning window to sample the image. This greatly reduces the complexity and dimensionality of the neural networks used.

\chapter{Generative Models}
In statistics and probability theory generative models are widely used to create a probabilistic representation of a target through observed variables. These generative models are capable of generating new data points from a joint probability distribution, unlike \textit{discriminative} models which have a conditional probability distribution.  Recall that the brain also uses inference and generative modelling to make predictions of the world which makes this method an interesting area of research.

Lately, generative models have gained a lot of traction in the machine learning community due to their ability to infer features from a probabilistic dataset where not all variables can be observed \cite{Goodfellow-et-al-2016}. An agent which has the capability to infer latent features of the world or itself and make predictions by sampling a generative model could lead to more interesting behaviour in terms of autonomous systems. For example, our agent could use such a generative model to learn admissible kinematic states based on training data in the form of images.

In this chapter we will discuss the general math behind generative models, some examples on how they are being used in the machine learning community and why it is relevant in this literature study.

\section{Joint Probability Distribution}
Take $z$ to be a random \textit{unobserved} variable and $x$ to be a random \textit{observed} variable.
A generative model is capable of generating new values for combinations of $x$ and $z$ and is given by
\begin{align}
    \label{eq:generative_model}
    p(x,z) = p(x|z)p(z).
\end{align}

In words, given a prior distribution, $p(z)$ and the likelihood, $p(x|z)$ we can generate new data for combinations of $x$ and $z$. However, the question remains how we obtain $p(x|z)$, the distribution which gives us observable variables based on the hidden variables. This might seem non-intuitive since we wish to learn the latent features from observations. This is true, but in order to make \textit{predictions} on the observed variables (posterior) we need to find this likelihood $p(x|z)$ as seen in Bayes' rule
\begin{equation}
  \label{eq:bayes_rule}
  \underbrace{p(x|z)}_{\text{likelihood}} = \dfrac{\underbrace{p(z|x)}_{\text{posterior}} \underbrace{p(x)}_\text{evidence}}{\underbrace{p(z)}_\text{prior}}.
\end{equation}

In terms of machine learning we see from equation \eqref{eq:bayes_rule} that expressions where the hidden variable $z$ is given are obtained through training. On the other hand, expression with $x$ given use inference to obtain $z$.

Going back to the predictive nature of the brain, we are interested in knowing the true (unobservable) state of the world based on observed variables. Essentially, we are interested in the posterior distribution $p(z|x)$. With this we can \textit{infer} a model of the world from which we can then sample to make predictions. This is what Bayesian inference is used for.

\section{Bayesian Inference}
When dealing with a dynamical system of which one or more states can not be directly measured one could apply Bayesian inference in order to infer that hidden state based on a model. A well known example is the Kalman Filter \cite{bishop2001introduction}. We are interested in the underlaying technique of the Kalman Filter, namely Bayesian inference in order to infer hidden variables from high dimensional data. Practically speaking, for our agent we would like to infer the kinematical model based on images.

From the previous section we have seen how we can use a generative model in order to produce new values for $x$ and $z$. However, this generative model first has to be trained in order for it to produce anything meaningful. For example, a trained generative model could be able to predict the position of a 3-DOF robotic arm from two \textit{unseen} consecutive visual representations \cite{watter2015embed}.

A generative model is able to do so by sampling the posterior probability distribution which is dependent on the marginal likelihood as shown in \eqref{eq:inference-integral}.
\begin{align}
    \label{eq:inference-integral}
    p(x) = \int p(x|z)p(z)dz
\end{align}

Even though the integral in equation \eqref{eq:inference-integral} can be solved theoretically, it is far from practical when dealing with the amount of data generally used in machine learning. In other words, evaluating the marginal likelihood (or evidence) in \eqref{eq:inference-integral} is intractable leading \eqref{eq:bayes_rule} by substitution to also become intractable.

Happily, recent advances in machine learning produced solutions which can approximate the intractable true posterior $p(z|x)$ making inference possible.

\section{Variational Inference}

As we have seen Bayesian inference is a powerful tool to use statistics in order to find the probability of an outcome based on a generative model. However, the integral over the likelihood as shown in equation \eqref{eq:inference-integral} is intractable causing the true posterior, $p(z|x)$ to also be intractable. In order to infer the kinematics of our robotic agent and be able to generate new configurations we need to be able to build the generative model and approximate the posterior. Variational inference gives us the tools for that.

Let $p(z|x)$ be a probability distribution which we can not fully observe, only sample. In order to find $p(z)$ we need to sample for all combinations of $x$ and $z$ which is intractable, as we have seen. There are sampling methods such as Markov chain Monte Carlo (MCMC), yielding exact samples from the target distribution, but these are very computationally heavy and are best suited for smaller datasets \cite{blei2017variational}. Variational inference (VI) is a method that approximates the probability density and therefore does not guarantee the same accuracy as MCMC. However, the nature of VI allows for very fast and efficient optimisation techniques such as gradient search. Assuming the geometry of the distribution to be approximated is moderately complex and well behaved, the inaccuracy of VI is acceptable and its speed then outweighs the benefits of other slower methods such as MCMC.

In order to solve the intractability of \eqref{eq:inference-integral} and \eqref{eq:bayes_rule} we can introduce a recognition model $q(z|x)$. The recognition model is an approximation of the true distribution $p(z|x)$. Let the recognition model and posterior distribution be parameterised by $\phi$ and $\theta$, respectively. For example, if the posterior is a normal distribution $\mathcal{N}(\mu,\sigma^2)$ parameterised by $\theta$ then the set is given by $\theta = \{\mu, \sigma\}$. This will lead to the following notation $q_\phi ( \cdot)$ for the approximation and $p_\theta (\cdot)$ for the true distribution.

The posterior distribution is approximated by varying the parameter $\phi$ such that the KL-divergence between $q_\phi(\cdot)$ and $p_\theta(\cdot)$ are minimised up to an added constant called the lower bound (ELBO or $\mathcal{L}$). Formally we solve the optimisation problem given by

\begin{equation}
\label{eq:elbo}
  \log p(x) = \argmin_{q_{\phi}(z)} D_{KL}\infdiv{q_\phi(z)}{p_\theta(z|x)} + \mathcal{L}(\phi,\theta)
\end{equation}

where $\mathcal{L} \leq \log p_\theta(x)$ since the KL-divergence is always $ \geq 0$. This shows that this optimisation will always be an approximation since \eqref{eq:elbo} will always have an added constant in the form of the lower bound $\mathcal{L}$. Finding the right parameters now becomes gradient search problem which can be handled by neural networks for fast convergence and parallel computing \cite{rezende2014stochastic,kingma2013auto}.

\chapter{Techniques}
So far we have investigated the fundamentals of neural networks, their implementations and statistics theory in terms of generative models. We have seen that generative models give us the ability to model the variance of natural data such as images. We can use such models to then make predictions. This is exceedingly interesting in robotics where we could use state of the art machine learning (ML) techniques in the control of robots. Returning to our goal, we want our agent to be able to infer its kinematics from images during training and then perform some action when a prior is given.

In literature two very prominent techniques are apparent that can potentially help us with our desired goal. Variational Autoencoders and Generative Adversarial Networks are both neural network based generative models \cite{kingma2013auto,goodfellow2014generative}. These techniques are researched heavily in the machine learning community where they can perform a wide array of tasks such as image synthesis, action forecasting from static images \cite{walker2016uncertain} and latent dynamic embedding for control \cite{watter2015embed}. This chapter will give an overview of the workings of both techniques. Since MNIST \footnote{MNIST is a database containing images of handwritten digits from 0-9 and is widely used as a benchmarking dataset for machine learning. \url{http://yann.lecun.com/exdb/mnist/}} is a go-to benchmarking dataset for ML algorithms we will also identify the performance of the VAE and GAN based on this dataset. For each technique we will discuss their workings after which we identify publications that are relevant to our research for our autonomous agent.

\section{Variational Autoencoder}
In the previous sections we have discussed the neural network architecture for an autoencoder and variational inference in generative modelling. One of the problems in Bayesian inference is the intractability of the posterior. A variety of algorithms exist which approximate that probability density by means of variational inference. However, these methods are relatively slow or require an analytical solution of the expectations. An efficient method is proposed by \cite{kingma2013auto} which is the Variational Autoencoder (VAE), a generative modelling neural network. Its power lays in the ability to jointly learn the parametrisation of the posterior distribution $q_\phi(z|x)$ and the likelihood $p_\theta(x|z)$.

\subsection{Background}
Similarly to a regular autoencoder, the VAE tries to reconstruct the input but it does so by minimising two loss functions: the latent loss given by the same equation as seen in equation \eqref{eq:elbo}, and the generation loss which is a simple mean square error given by
\begin{equation}
  \lVert x-\hat{x} \rVert^2
\end{equation}
where $x$ is the real data and $\hat{x}$ is the generated data.

[insert relevance to RBM]

\begin{figure}[bt]
    \centering
    \includegraphics[width=0.95\textwidth]{VAE2}
    \caption{A Variational Autoencoder ingests data into an encoder network which learns $\mu$ and $\sigma$ to parameterise the recognition model $q_\phi(z|x)$. The decoder then samples from the latent space which is constrained to be $\mathcal{N}(0,1)$ and tries to reconstruct the input by minimising the KL-divergence between $q$ and $p$.}
    \label{fig:vae}
\end{figure}

Recall from the previous chapter that in variational inference we try to optimise the parameters for the recognition density $q_\phi(z|x)$ and the likelihood $p_\theta(x|z)$. The VAE consists out of two neural networks, an encoding network and a decoding network, as seen in figure \ref{fig:vae}. When a VAE is used for inferencing latent variables from images often convolutional networks are used to reduce the network size. In short, a VAE learns the following two things: the parameters for the recognition model $\phi$ and the parameters for the likelihood $\theta$ which are used to sample the latent space $z$.

The encoder network of the VAE represents the posterior approximation $q_\phi(z|x)$. The input of the network is a dataset from which we want to learn the latent features and the output is the set of parameters $\phi = \{\mu,\sigma\}$ assuming a Gaussian distribution in the dataset. More formally, the encoder network represents a probability distribution given by
\begin{equation}
  \label{eq:recognition_model}
  q_\phi(z|x) = \mathcal{N}(z|\mu(x), \sigma(x)^2)
\end{equation}

where the parameters $\mu$ and $\sigma$ can be seen as arbitrary mapping functions learned by the neural network. The objective of the encoder neural network is to minimise the KL-divergence between the learned latent space and the unit Gaussian as shown below.
\begin{equation}
  \argmin D_{KL}\infdiv{\mathcal{N}(z|\mu(x), \sigma(x)^2)}{\mathcal{N}(0,1)}
\end{equation}

This constraint forces the VAE to learn sufficiently varied latent variables and not just how to memorise the dataset.

On the right hand side of figure \ref{fig:vae} we have the decoder network that represents the likelihood $p_\theta(x|z)$. Since we let the decoder samples from the simple latent space $z \sim \mathcal{N}(0,1)$, the complexity of the dataset is instead captured by the neural network $p_\theta(x|z)$\cite{zhao2017towards}. Both $\theta$ and $\phi$ are learned by back-propagating the error from the latent and generation loss through the neural network. Recall the generative model is given by \eqref{eq:generative_model}, thus by sampling the likelihood and the latent space we can reconstruct the image.

As many other ML techniques, the VAE is benchmarked using the MNIST dataset. Given an MNIST dataset the VAE can learn the latent features of the numbers from which it can then \textit{generate} a new but similar dataset. When the latent space $z$ is projected in 2D space (which is used for demonstrating latent feature learning) we can see how the VAE interprets the numbers and how the latent features are linearly interpolated.

[MNIST figure]

\subsection{Applications}
Variational autoencoders have seen a widespread adoption in the machine learning community for learning latent features in high dimensional data. Aside from \textit{dreaming} of landscapes and handwritten digits there are many applications which demonstrate the use of VAEs and how it can be used in our research.

Recall that our agent is a robotic arm with 6 degrees of freedom and has a camera. Our goal is for that robot to learn its own internal kinematics in the form of a generative model. This would mean that once we present the agent with a prior belief (a position in space in the form of an image) we should be able to generate a trajectory towards that point \textit{without} performing direct inverse kinematics. The VAE could be a very useful tool to achieve our goal. Therefore, we will explore some examples in literature.

In a recent publication a VAE is trained with thousands of videos after which it is able to predict the trajectory of a set of pixels in a static image \cite{walker2016uncertain}. As is usual in vision applications, a convolutional neural network was used to reduce the computation time and dimensionality of the network. Considering our research, a trained VAE that can predict the next state of a robotic arm from an image can be interesting.

Another important publication shows how a Kalman filter can be learned using multiple techniques including a VAE \cite{krishnan2015deep}. The authors propose the Deep Kalman Filter (DKF) which extends the inference network with an action variable $u$ such that recognition model becomes $q_\phi(\vec{z} \mid \vec{x}, \vec{u})$ where the arrow indicates a sequence. The neural network is parametrised by $\theta = {\alpha, \beta, \kappa}$ which allow the authors to learn the mapping functions for the latent space and observation space.
\begin{equation}
  \label{eq:deep_kalman}
  \begin{split}
  z_t &\sim \mathcal{N}\left(G_\alpha (z_{t-1}, u_{t-1},\Delta_t), S_\beta(z_{t-1}, u_{t-1},\Delta_t) \right) \\
  x_t &\sim \Pi\left(F_\kappa(z_t)\right)
  \end{split}
\end{equation}

The generative model for the Kalman filter is given by equation \eqref{eq:deep_kalman} where the latent space is normally distributed by the given non linear mapping functions, $G_\alpha$ and $S_\beta$, which are dependent on the previous latent state, action and time difference. By restricting the functional forms of the mapping functions, different Kalman filters can be trained demonstrating the flexibility of this framework. This method are a derivative thereof can be extremely beneficial for our autonomous agent when control actions have to be predicted. However, this is under the assumption that the DKF is trained with both sensory observations \textit{and} corresponding actions.

Finally, another promising example is a publication that has made an effort to learn the latent dynamics of various systems on which stochastic optimal control \cite{watter2015embed} could be performed. Their method Embed to Control (E2C) uses multiple VAEs and an iterative linear quadratic regulator (iLQR) in order to learn the swing up of an inverted pendulum. In short, they accomplish this by stacking two images of a simulated pendulum which represent their states at $s_t$ and $s_{t+1}$ and training the VAE with a large dataset of such stacks. The VAE is then able to learn the state space representation of the pendulum which the iLQR controller can then use, given a desired state, to learn a swing up.

The E2C method is a promising algorithm which has the potential to be extended to larger degrees of freedom, allowing for control of complex robotic systems using raw sensory information. As long as the latent space has the Markov property, stochastic optimal control can be applied to a latentx kinematic model which has been learned in an unsupervised manner \cite{matsubara2014latent}.

\section{Generative Adversarial Networks}
[Mention that GANs can not model discrete data]

Another well known deep generative model is the generative adversarial network (GAN). It is a technique based on a zero-sum game where two neural networks are competing against each other \cite{goodfellow2014generative}. Similarly to a VAE, a GAN learns a generative model from training data as well. However, a GAN does not require an inferencing network.

\subsection{Background}

Recall from chapter 3 that the generative model is given by equation \eqref{eq:generative_model} for which we need multiple dependent equations leading to intractable calculations. Where the VAE circumvents that by using a recognition model to approximate the posterior, the GAN completely cuts out the inference step. It does so by simultaneously training two neural networks, the generator ($G$) and the discriminator ($D$), which play a minimax game as shown in figure \ref{fig:gan_minmax}. The aim of the game is for the generator to create sufficiently believable samples that the discriminator can no longer discern what is real and what is generated.

\begin{figure}[h]
    \centering
    \includegraphics[width=0.95\textwidth]{gan_minmax}
    \caption{A generative adversarial net samples from a simple gaussian distribution. Both the generator and discriminator neural nets are optimized. The Jensen-Shannon divergence is used to calculate the loss between the distributions which is then back-propagated through the networks. needs citation: https://sthalles.github.io/intro-to-gans/}
    \label{fig:gan_minmax}
\end{figure}

The generator network of a GAN is fairly simple. The GAN does not make any approximations to the posterior, unlike the VAE, instead it simply assumes the latent space distribution $p(z)$ to be random noise given by $\mathcal{N}(0,\mathbf{I})$ where $\mathbf{I}$ is the identity matrix. The dimensionality of the identity matrix and thus that of the latent space is heuristically determined.

The generator network represents a mapping to the data space as $G_\theta(z)$ where $\phi$ are the parameters of the neural network. Statistically, the generator would represent the likelihood $p(x|z)$ which combined with the prior $p(z)$ is the generative model as we've seen in equation \eqref{eq:generative_model}.

The discriminator network $D_\gamma(x)$, parameterised by $\gamma$, is a lot simpler. It outputs a probability scalar indicating whether its input $x$ came from the real data or the generative model $G_\theta(z)$. In words, the generator constantly tries to improve itself to "fool" the discriminator network. On the other hand, the discriminator network constantly tries to become better at catching the generator network. Both networks are jointly trained with the following value function $V(G,D)$:
\begin{equation}
\label{eq:gan_minimax}
  \min_{G} \max_{D} V(D,G) = \mathbb{E}_{x\sim p_{data}(x)} [\log D(x)] + \mathbb{E}_{z\sim p_z(z)}[\log(1-D(G(z)))].
\end{equation}

As with most ML techniques the GAN is also benchmarked on the MNIST dataset. We see in figure \ref{fig:gan_mnist} that the GAN performs really well in generating new handwritten digits. Also, linearly interpolating between the coordinates in $z$-space shows that the algorithm has clearly learned latent features as shown in figure \ref{fig:gan_mnist_interp} .

\begin{figure}[htb]
    \centering
    \includegraphics[width=0.95\textwidth]{gan_interp}
    \caption{Digits obtained by interpolating between latent variables.}
    \label{fig:gan_mnist_interp}
\end{figure}

\subsection{Applications}
As with many other ML techniques, the GAN too was originally benchmarked by synthesising images. Fortunately, it has seen immense adoption and widespread use in the scientific community, probably more so than the VAE. Some examples will be discusses in this section demonstrating the power of GANs.

[will be expanded with literature i'm still trying to understand]

Finally we discuss an interesting publication on using GANs for learning forward and inverse models for motor control of a 2-DOF simulated robotic arm \cite{lenninger2017generative}. The model is trained by feeding it simulated current $S_t$ and future $S_{t+1}$ states caused by an input $u_t$. The states of the links are defined as vectors are given by $S_{t} = \{X_t(\theta_t),\theta_t, \dot{\theta}_t\}$, where $X_t(\theta_t$) is the position of the end-effector. The future state $S_{t+1}$ is calculated by the simulator. During training the generator tries to reconstruct the future state and the discriminator compares it to ground truth from the simulator. Ultimately, both the forward model and the inverse model of the arm are learned under the generative model. This allows the arm to either find the control input that has caused a change in state or find the next state given a control input.

\chapter{Conclusion}
Our goal is to find an unsupervised technique which can learn latent system features to be used in robotic applications, more specifically in applications requiring vision. In this literature study we have investigated  two important machine learning techniques, namely, variational autoencoders (VAE) and generative adversarial networks (GAN). These deep generative models are actively researched in the ML community and are believed to hold the promise for true AI. Combining such a state of the art learning algorithm in the field of robotics could pave the way to fully autonomous systems. In terms of our proposed robotic agent, we seek a technique which allows it to learn its internal kinematics in an unsupervised manner after which it can perform a task given a prior belief in the form of an image.

In this literature study we have found that the variational autoencoder is the most suitable method to proceed with our research purposes. This is according to the three main criteria which have been defined in the introduction and will be discusses next.

Firstly, VAEs are explicitly designed to learn latent features from high dimensional data. Our aim is to observe raw sensory information (e.g. images) and learn a reduced representation. This \textit{inferencing} step allows us to extract meaningful information from a scene after which we can make predictions. In contrast to VAEs, GANs have no support for this requirement unless modified \cite{mescheder2017adversarial,chen2016infogan}. For example, if we wish to supply our agent with a prior belief, for a VAE we could represent that as a simple image of which the latent features can be inferred. For a GAN this becomes somewhat complicated as the prior belief has to be directly embedded into the latent space since there is no inference step.

Secondly, both VAE and GAN learn a generative model from which we can sample in order to make predictions based on a prior belief. In this case both techniques are equal as they both have a generative model which can be sampled and are flexible in terms of scalability. However, it is important to note that VAEs require differentiation through the hidden units due to the nature of the training algorithm \cite{goodfellow2014generative}. Thus VAEs can not have discrete latent variables. How this will affect the discrete nature of digital sensors such as encoders is unknown.

Thirdly, VAEs have a robust training algorithm and good convergence. Even though both VAEs and GANs have their advantages and disadvantages, their training dynamics are very different which are crucial in our robotic application. In literature we see that VAEs take longer to train than GANs. Where the GAN has $O(n)$ time-complexity the VAE requires $O(n \log n)$ for training to complete []. In addition we see that  in contrast to GANs, VAEs produce blurry images in many benchmarks such as CIFAR-10. It is still unknown what causes the blurriness. In terms of robotic application we do not know how this effect will be translated but it is imaginable that a VAE will have a larger variance between possible states as opposed to a GAN. However, this performance does not outweigh the training dynamics where the GAN falls short. Namely, GANs are notoriously hard to train \cite{Goodfellow-et-al-2016, arjovsky2017towards}. Because the GAN tries find the Nash-equilibrium, the convergence of the gradient descent can fail under high dimensional parameters \cite{goodfellow2014distinguishability}. It is possible to alleviate some of the issues with the GAN by using a modified training algorithm \cite{arjovsky2017wasserstein}. Thus, stability and convergence of a VAE during training outweigh the high performance of a GAN. In the case that the performance for sharper images is deemed necessary, there are various options available \cite{mescheder2017adversarial, rezende2015variational, salimans2015markov}.

Finally, for future work we recommend further investigation of variations of the proposed methods. Machine learning is very much an active field of research and it is quite commonplace for techniques to be combined such as VAEs with GANs \cite{mescheder2017adversarial,hu2017unifying}. In case the performance of the \textit{vanilla} methods seem to underperform, such combinations can be investigated to improve results. In addition, generative models tend to not have an interpretable visual representation \cite{feichtenhofer2018have}. Therefore, further investigation can be beneficial in order to better understand what such a model has learned.

In summary, variational autoencoders offer a flexible and novel way for an autonomous agent to learn latent features of its own internal system or its surroundings. If this method is coupled with an implementation for \textit{active} exploration of the environment, this research could lead to a novel approach for creating real autonomous agents which can learn to understand the world they live in.

\newpage
\renewcommand\bibname{References}
\bibliographystyle{unsrt}
\bibliography{references}

\end{document}


\section{Tracer particles}

Tracer particles are to track the Lagrangian evolution of a model fluid using discrete particles. In hydrodynamical simulations based on an Eulerian grid (including CASTRO), thermodynamic variables at a given time are derived by solving the equations of motion of a fluid between cells. Therefore, in this scheme, the physical quantities that we can access to are not discretized quantities at any given position, but rather average values over each cell. However, employing discrete particles, passively advected with the fluid flow, allows us to obtain local instantaneous thermodynamic variables, such as the temperature and the density, at well-defined positions, independent of the spatial resolution, i.e., the spatial cell size. This means that we can follow the evolution of the fluid at any given position and time.

\noindent \noindent CASTRO provides a tracer particle scheme with useful options. In this scheme, particles are advanced using the midpoint method either with the cell-centered velocities or the face-centered velocities (Marker-And-Cell method)\footnote{One can simplify interpolation with the cell-centered velocity. However, this can lead to decoupling of the pressure and the velocity components, possibly resulting in instability. This can be avoided with the face-centered velocity}. The number and the initial positions of particles are flexibly determined according to the purpose of a given model.

\section{Initializing the Particles}

One must include the tracer particles in the {\tt GNUmakefile} by setting

\vspace{0.1in}
\noindent{\tt USE\_PARTICLES = TRUE}.
\vspace{0.1in}

\noindent And the particles can be initialized via

\vspace{0.1in}
\noindent {\tt {\bf castro.do\_tracer\_particles }}     = 1
\vspace{0.1in}

\noindent in the {\tt {\bf inputs}} file.

If one wants to investigate the evolution of fluid motions starting from specific positions (or a certain range of area or volume), one should manually specify the positions of particles by providing an input file containing the total number and the initial positions of the particles.
The input file should be in the same directory where your {\tt inputs} file is located. The name of the input file is  determined via :

\vspace{0.1in}
\noindent {\tt {\bf particles.particle\_init\_file =}}{\em particle\_file}
\vspace{0.1in}

\noindent Here {\em particle\_file} is the user-specified name of the file. The first line in this file is
assumed to contain the number of particles.  Each line after that contains the positions in a coordinate system adopted for your model. For 3-D cartesian coordinates, \\

$x ~y ~z$ \\

For example, an input file for a model fluid with 6 particles in 2-D Cartesian coordinates may look like,\\

\begin{lstlisting}
6
3.28125e+08 9.9198e+08
5.46875e+08 9.9198e+08
7.65625e+08 9.9198e+08
9.84375e+08 9.9198e+08
1.20312e+09 9.9198e+08
1.42188e+09 9.9198e+08
\end{lstlisting}

According to this input file, the 6 particles will be positioned at the same height (same $y$ coordinate in the second column), equally spaced in $x$ direction (the first column except for the particle number on the first line) from $3.28\times10^{8} {\rm ~cm}$ to $1.42\times 10^{9} {\rm ~cm}$.

\section{Output file}
\label{particles:output_file}
\noindent The output files are stored in a directory whose name is determined by a variable {\tt particles.timestamp\_dir}. For example, if the variable is set as follows,

\vspace{0.1in}
{\tt  {\bf particles.timestamp\_dir=}} {\em particle\_dir},
\vspace{0.1in}

\noindent A directory {\em particle\_dir} is automatically made with the directories for the main CASTRO output file ({\tt plt****}) once a simulation starts and the particle output files are stored inside that directory.

\vspace{0.05in}
\noindent The name of the output file consists of {\tt Timestamp\_} along with a number at the end. The number increases (typically from 00) as more processors are involved in following the trajectories of particles. In parallel computing, a computational domain is divided according to the number of processors requested. Then each processor only follows the particles on the domain assigned to that processor and records their positions and velocities at any given time in a different output file. Since it is possible for particles to move from one domain to another during the evolution, its history can be stored in different files. More output files (with larger numbers at the end of the file name) can be produced as more processors track the particles.

\vspace{0.05in}
\noindent By default, the output file contains the positions and velocities of all particles at a given time, meaning [$3+ 2\times$dimensionality] columns. For example, for particles in a 3-D domain, the columns in the output file are,

\vspace{0.1in}
${\rm index1}~~{\rm index2}~~x~~ y~~ z~~ t~~ v_{\rm x} ~~v_{\rm y}~~ v_{\rm z}~~ [\rho ~~ T]$
\vspace{0.1in}

\noindent The first two integers correspond to the particle index and the processor number.
One should use the two numbers in order to identify a particle and extract its history (i.e., the trajectory in Figure \ref{fig:particletrajectory}).

\begin{figure}[h]
	\centering
	\includegraphics[width=2.5in]{fluid_motion}
	\includegraphics[width=2.39in]{tracer_trajectory}
	\caption{A model atmosphere (\textit{left} panel) and the trajectories of 500 particles (\textit{right} panel) following the fluid motion on the atmosphere. The particles are initially positioned at five different heights, $y=13000\mathrm{~km},~11000\mathrm{~km},~ 8000\mathrm{~km},~ 6000\mathrm{~km}, ~38000\mathrm{~km}$ (100 particles at each height). In the \textit{left} panel, the arrows roughly show the fluid motion. In the \textit{right} panel, the solid lines represent the trajectories of the particles. }
	\label{fig:particletrajectory}
\end{figure}

\noindent One can also add the last two columns $[\rho ~~ T]$, i.e., the local density and local temperature of fluid at the position of each particle by setting the following,

\vspace{0.1in}
\noindent {\tt {\bf particles.timestamp\_temperature }}= 1,\\
\noindent {\tt  {\bf  particles.timestamp\_density}}     = 1.
\vspace{0.1in}

\noindent For example, let's consider 10 particles on a domain.  If 4 out 10 particles are initially on a processor and the rest are on another processor, this means two processors are tracking the particles and two output files are produced. In the output file written by the processor with 4 particles, one can find that four lines are stored at the same time and each line corresponds to each particle info. while in the other output file for the other 6 particles, 6 lines are stored at the same time.

\vspace{0.05in}

\noindent If {\tt {\bf particles.write\_in\_plotfile=1}}, the particle data are stored in a binary file along with the main CASTRO output plotfile in directories {\tt plt*****/Tracer/}.
\vspace{0.05in}

\subsection{Run-time Screen Output}

The verbosity written to the screen at run-time is constrolled by setting:

\noindent {\bf particles.v } = 0 or 1 (default: 0)\\


\documentclass[11pt]{article}
%============Macros==================%
% you don't need to change anything. start editing from main body.
\usepackage{amsmath,amsfonts,amssymb,amsthm, cancel, float, enumitem}
\usepackage{qcircuit}
\usepackage[margin=1in]{geometry}
%--------------Cosmetic----------------%
\usepackage{mathtools}
\usepackage{hyperref}
\usepackage{fullpage}
\usepackage{microtype}
\usepackage{xspace}
\usepackage[svgnames]{xcolor}
\usepackage[sc]{mathpazo}
\usepackage{enumitem}
\setlist[enumerate]{itemsep=1pt,topsep=2pt}
\setlist[itemize]{itemsep=1pt,topsep=2pt}
%--------------Header------------------%
\def\course{CS 410/510 Introduction to Quantum Computing}
\def\term{Portland State U, Spring 2017}
\def\prof{Lecturer: Fang Song}
\newcommand{\handout}[5]{
   \renewcommand{\thepage}{\arabic{page}}
   \begin{center}
   \framebox{
      \vbox{
    \hbox to 5.78in { \hfill \large{\course} \hfill }
       \vspace{2mm}
       \hbox to 6in { {\Large \hfill #5  \hfill} }
       \vspace{2mm}
       \hbox to 6in { \term \hfill \emph{#2}}
       \hbox to 6in { {#3 \hfill \emph{#4}}}
      }
   }
   \end{center}
   \vspace*{4mm}
}
\newcommand{\lecture}[4]{\handout{#1}{#2}{#3}{#4}{{Lecture #1}}}

\def\complex{\mathbb{C}}
\def\real{\mathbb{R}}
\def\natural{\mathbb{N}}
\def\integer{\mathbb{Z}}

\newcommand{\norm}[1]{\left\lVert#1\right\rVert}
\newcommand{\snorm}[1]{\lVert#1\rVert}
\newcommand{\abs}[1]{\left\lvert #1 \right\rvert}
\newcommand{\ceil}[1]{\left\lceil #1 \right\rceil}
\newcommand{\floor}[1]{\left\lfloor #1 \right\rfloor}
\newcommand{\set}[1]{\left\{ #1 \right\}}
\newcommand{\vecb}[1]{\boldsymbol{\vec{#1}}}
\newcommand{\conj}[1]{\overline{#1}}
\newcommand{\op}[2]{#1#2#1^\dag}

\newcommand{\tr}{\operatorname{Tr}}
\newcommand{\se}{\operatorname{H}}
\newcommand{\vne}{\operatorname{S}}
\newcommand{\e}{\operatorname{E}}
\newcommand{\rank}{\operatorname{rank}}
\renewcommand{\det}{\operatorname{Det}}

\newcommand{\bra}[1]{\langle #1 \rvert}
\newcommand{\ket}[1]{\lvert #1 \rangle}
\newcommand{\bret}[2]{\langle{#1}|{#2}\rangle}
\newcommand{\kret}[2]{\ket{#1}\bra{#2}}
\newcommand{\kera}[1]{\kret{#1}{#1}}
\newcommand{\Example}[1]{{\bf Example #1}:}

\def\X{\mathcal{X}}
\def\Y{\mathcal{Y}}
\def\Z{\mathcal{Z}}
\def\W{\mathcal{W}}
\def\A{\mathcal{A}}
\def\B{\mathcal{B}}
\def\V{\mathcal{V}}
\def\U{\mathcal{U}}
\def\C{\mathcal{C}}
\def\D{\mathcal{D}}
\def\E{\mathcal{E}}
\def\F{\mathcal{F}}
\def\M{\mathcal{M}}
\def\R{\mathcal{R}}
\def\P{\mathcal{P}}
\def\Q{\mathcal{Q}}
\def\S{\mathcal{S}}
\def\T{\mathcal{T}}
\def\K{\mathcal{K}}
\def\L{\mathcal{L}}

\def\fill{   \hfill}

%=============Main Doc=================%
\begin{document}
%-----Specs: change accordingly--------%
\def\lecdate{May 18, 2017} % put lecture date here
\def\scribe{Scribe: Asher Toback} % put your name here
\def\lecnum{14} % change the lecture number

\lecture{\lecnum}{\lecdate}{\prof}{\scribe}%

\begin{center}
{\textsc{Version: \today}}
\end{center}

%\section{Introduction}
So far, we've assumed that the channels in our quantum circuits have been noiseless. Namely, that the information sent over the channel is identical to the information that is received. However, there will always be noise in real-life channels that may cause errors during transmition. In the classical setting we see these errors as bit flips in our original message. However, when we try to communicate quantum information over a quantum channel it becomes less clear what an error is, how to detect it, and how to correct it. In this paper we'll introduce the basic concepts of QECC (Quantum Error Correcting Code). We'll start by defining our Error Model, cover X-Encoding, Z-Encoding, Shor's 9-bit QECC, and arbitrary unitary QECC, and finish with a brief description of CSS Codes.

\section{Error Models}

Let's first describe how we think of communicating over a noisy channel in a classical setting. We'll be trying to communicate a bit $b\in\{0,1\}$. We assume that an error occurs on the channel with probability $p\in(0,1/2)$. Thus, we can think of our channel as:

\abovedisplayskip=0pt\relax
\[
b \xrightarrow{\text{Message Over a Binary Symmetric Channel} }
\begin{cases}
b & \text{ with probability } 1-p\\
1-b  & \text{ with probability  } p
\end{cases}
\]

In the quantum setting we similarly wish to communicate the state of our system over a channel. We know that we can represent our system state as a density matrix $\rho$. We can then view an error on the channel as some operation $\Phi$ occuring on our state $\rho$ with some probability $p\in(0,1/2)$. As this is an introduction to QECC, we'll assume that the only errors on our channel can be unitary, meaning $\Phi = U\rho U^\dagger$. Our quantum error model is thus:

\abovedisplayskip=0pt\relax
\[
\rho \xrightarrow{\text{Message Over a Quantum Binary Symmetric Channel}}
\begin{cases}
\rho & \text{ with probability } 1-p\\
\Phi\rho  & \text{ with probability  } p
\end{cases}
\]

\section{Classical Solutions}

The standard way for dealing with errors in the classical setting is simply to introduce redundancies in our message. Here is basic 3-bit encoding scheme.

{
\[0 \xrightarrow{} 000\]
\[1 \xrightarrow{} 111\]
}

To decode a message we partition our received message into three bit chunks, find what the majority of each three bit section is, and assign that majority to that section. If no errors occur it's clear that we'll receive the same message we sent after decoding. However, it's also the case that if only one error occurs on a 3 bit section, then we'll also recover the original message as if no error had occured. Thus, by a simple counting argument, we see that:

{
\[
Pr[\text{Correct Transmission}] = (1-p)^3 + 3p(1-p)^2 = 1-(3p^2 -2p^3)
\]
}

And most importantly, when $p < 1/2$:
{
\[
1-(3p^2 -2p^3) \leq 1 -p
\]
}

Thus, using this encoding scheme will help the receiver receive the correct message. Now that we've demonstrated this, it's clear that we could get a better result if we simply introduce more and more redundancies into our message. Thus, we could communicate our message correctly with as high a probability as we'd like.

It's also worth nothing that, if $p>1/2$, we can simply do the same exact process as above, but at the end of the protocol we manually flip all the bits ourselves. This is due to the fact that the channel is reliable, in that we can expect it to always apply an error. Thus, we can correct for that by simply flipping all the bits at the end. However, if $p = 1/2$, then we're out of luck. The channel is just as likely to preserve our message as it is to cause on error, so we can't rely on it at all. This is clearly the worst case and represents white noise.

\section{Quantum Bit Flip Error}

It's not immediately obvious how to even approach a quantum error. Applying a unitary operation onto a quantum state can be viewed as rotating that state in the complex plane. This means there are uncountable many different types of errors that we need to correct. And, even if we knew which error were to occur, it's not immediately obvious how we could introduce redundacies like in the classical setting. Namely, we can't clone our input.

To begin tackling these issues let's first assume that the only errors that occur on our channel are X, the Pauli Matrix that represents a bit flip in the quantum setting. As a reminder, for some qubit $\ket{\psi} = \alpha\ket{0} +  \beta\ket{1}$:

{
\[\ket{\psi} \xrightarrow{X} \alpha\ket{1} + \beta\ket{0} \]
}

This is a start, but how will we deal with no cloning? It turns out there does exist a 3-bit encoding scheme which resembles the classical one which doesn't involve cloning. Namely:

{
\[\ket0 \xrightarrow{X-Encoding} \ket{000} \]
\[\ket1 \xrightarrow{X-Encoding} \ket{111} \]
}

This encoding scheme is called X-Encoding, as it will be able to correct for X errors, which we'll show shortly. First, note that, although they look similar, this encoding scheme is not:

{
\[\ket{\psi} \not\xrightarrow{X-Encoding} \ket{\psi\psi\psi}\]
}

It instead works like the following. If the qubit $\ket{\psi} = \alpha\ket{0} +  \beta\ket{1}$, then the encoding works as follows:

{
\[\ket{\psi} \xrightarrow{X-Encoding} \alpha\ket{000} +  \beta\ket{111}\]
}

To see why these aren't the same, note that $(\ket{00} + \ket{11}) \neq (\ket0 + \ket1)^{\otimes2}$. Clearly these must be different states as the first state is maximally entangled and the other is not.

Now, to achieve X-Encoding we use the following circuit.

{
\centering
\includegraphics[scale=1]{X-Enc.png}
%\caption{}
\label{fig:Grammar}
\par
}

where the hollow circles are CNOT gates. It's left to the reader to verify that this behaves as intended. We now want to decode in the same way that we did in the classical setting. That is, we want to find the majority of the three qubits and assign these three qubits to be this majority instead. It turns out that this can be done, and it's done by the following circuit.

{
\centering
\includegraphics[scale=1]{X-Dec.png}
%\caption{}
\label{fig:Grammar}
\par
}

The last gate is the Toffoli gate, and can be thought of as a CNOT gate which depends on two control wires. Specifically, it require that both are in the state $\ket1$ in order to apply a NOT.

With these encoding and decoding schemes let's observe that they correct an X error as claimed. We'll assume that the X error occurs on the second bit of the encoding.

{
\[\alpha\ket{0} +  \beta\ket{1} \xrightarrow{X-Encoding} \alpha\ket{000} +  \beta\ket{111}\]
\[ \xrightarrow{\mathbb{I} \otimes X \otimes \mathbb{I} }\alpha\ket{010} +  \beta\ket{101}\]
\[\xrightarrow{X-Decoding} (\alpha\ket{0} +  \beta\ket{1}) \otimes \ket{10}\]
}

It's left to the reader to verify that the above mappings are indeed correct. We see then that the top wire does indeed recover the original qubit, as desired! It's left to the reader to verify that this code does correct every possible X error, as long as only one error occurs in total.

It's also worth noting that the bottom two wires hold the information of the error. That is, if you measured them, you could discover which bit that the X error occurred. These are called the $syndrome$ and will be useful for future QECC.

We've thus accomplished what we set out to do, namely fix a bit flip error in the quantum setting.

\section{Quantum Phase Flip Error}

What if an error other than a bit flip were to occur? We'll now look at the case of a Z error, or a phase flip. As a reminder, for some qubit $\ket{\psi} = \alpha\ket{0} +  \beta\ket{1}$:

{
\[\ket{\psi} \xrightarrow{Z} \alpha\ket{0} - \beta\ket{1}\]
}

To correct this error, we make an important observation. Namely: $Z\ket+ = \ket-$, $Z\ket- = \ket+$. Thus, we see that a phase flip can be thought of as a bit flip in the Hadamard basis $\{\ket+,\ket-\}$. In other words, $HZH = X$, where H is the Hadamard gate. And, since we've already corrected bit flip errors, we're essentially finished. Thus, our encoding scheme should behave as follows:

{
\[\ket0 \xrightarrow{Z-Encoding} \ket{+++} \]
\[\ket1 \xrightarrow{Z-Encoding} \ket{---} \]
}

To acheive this encoding we simply add Hadamard gates after our X-Encoding scheme:

{
\centering
\includegraphics[scale=1]{Z-Enc.png}
%\caption{}
\label{fig:Grammar}
\par
}

Our decoding scheme similarly just adds a Hadamard gate before the X-Decoding scheme:

{
\centering
\includegraphics[scale=1]{Z-Dec.png}
%\caption{}
\label{fig:Grammar}
\par
}

Just as before, let's say a Z-Error occurs on the second bit of our encoding. We then get:

{
\[\alpha\ket{0} +  \beta\ket{1} \xrightarrow{Z-Encoding} \alpha\ket{+++} +  \beta\ket{---}\]
\[ \xrightarrow{\mathbb{I} \otimes Z \otimes \mathbb{I} }\alpha\ket{+-+} +  \beta\ket{-+-}\]
\[\xrightarrow{Z-Decoding} (\alpha\ket{0} +  \beta\ket{1}) \otimes \ket{10}\]
}

Which is just what we wanted.

\section{Shor's 9-qubit QECC}

We can correct an X error and a Z error, but what if our channel's noise has the potential to apply both an X or a Z gate? Shor came up with a solution to this which is intuitive. Essentially, you feed the Z and X encodings into each other. Namely:

{
\[Shor's: \ket0 \xrightarrow{Z-Encoding} \ket{+++} \xrightarrow{X-Encoding} (1/\sqrt{2}(\ket{000} + \ket{111}))^{\otimes3} \]
\[Shor's: \ket1 \xrightarrow{Z-Encoding} \ket{---} \xrightarrow{X-Encoding} (1/\sqrt{2}(\ket{000} - \ket{111}))^{\otimes3} \]
}

You then X-decode this state, then Z-decode it. It turns out that this can correct 1 error on the 9-qubit encoding, which is left to the reader to verify. However, it's clear that it works due to linearity. The circuit that accomplishes these encoding and decoding schemes is as follows:

{
\begin{figure}
\centering
\includegraphics[scale=0.65]{ShorQECC.png}
\caption{Shor's 9-bit QECC. All pictures from Wikipedia}
\label{fig:Grammar}
\end{figure}
\par
}

Where $E$ represents the X or Z error.

\section{Arbitrary Unitary QECC}

It's natural to wonder if we can correct an error of the remaining Pauli Gate, the Y gate. Well, we know that $Y=-iXZ$. And, because we can correct $X$ and $Z$ errors, we know we can also correct Y errors.

However, we also know that $\forall U$ where $U$ is a unitary matrix, that $U = \alpha I + \beta X + \gamma Y + \delta Z$. Thus, as it turns out, we know there exists encoding and decoding schemes such that we can correct 1 arbitrary Unitary error.

\section{CSS Codes}
In this paper, we assumed that our errors were all Unitary transformations. However, we can't always make this assumption. To fix this, we use a type of QECC called CSS Codes. These general constructions are based off of classical linear ECC. The basic ideas rellies on group theory and the idea of cosets. The encoding maps certain messages into different cosets based off what the message is. We then have QECC that corrects errors just on each coset. As it turns out, given some $t\in\mathbb{N}$, we can correct up to $t$ errors using CSS Codes.

%\bibliographystyle{plain}
%\bibliography{references}
\end{document}


In the tau reconstruction, $\tauhad$ candidates are seeded from jets. Therefore, QCD events are the main source of background (BG). Nonetheless, we do not use MC simulations for these processes. Therefore, we use a \textit{data driven method} to estimate multi-jet (MJ) background. In this section we will discuss how we derived our Multi-Jet Background (MJBG) estimation.

\subsection{Multi-Jet Background}
To obtain the MJBG, a variable that in principle is supposed to be uncorrelated with the shape of the MJBG is chosen. For this study, this variable is the relative sign between the charges of the $\tauhad$ and the lepton. This defines two regions: the same sign (SS) region where $q(\tauhad)=q(l)$ and the opposite sign region where $q(\tauhad)=-q(l)$. Our estimate for the MJBG in the SS region is obtained by subtracting the signal and electroweak backgrounds (EWBG) contributions:
\begin{equation}
\text{MJBG}_{\text{SS}}=\text{Data}_{\text{SS}}-\text{Signal}_{\text{SS}}-\text{EWBG}_{\text{SS}},
\label{eq19}
\end{equation}
To study the residual charge correlation, a control region (CR) for the MJBG is defined. The estimate for the MJBG in these regions is given by eq. \ref{eq19} as well. The region described in Sec.\ref{sec3.3}, which contains all our final selected events is called the signal region (SR). In contrast to the SR, the CR is defined by events that fail the lepton isolation criteria and that fail the Tight-ID working point criterion for the $\tauhad$ candidate. In this region, tau candidates are required to have a looser tau-$pt$ of 25 GeV or more. This last choice was made to increase the number of candidates in the CR. The other kinematic features of the SR are maintained. A diagram showing the four regions just defined is shown in Fig.\ref{Fig7}. Now, if we assume that charge correlation is the same in the SR and CR, we have:
 \begin{equation}
 \frac{\text{MJBG}_{\text{SR OS}}}{\text{MJBG}_{\text{SR SS}}}=\frac{\text{MJBG}_{\text{CR OS}}}{\text{MJBG}_{\text{CR SS}}},
 \end{equation}
then,
 \begin{equation}
\text{MJBG}_{\text{SR OS}}=\text{MJBG}_{\text{SR SS}}\times \text{RQCD}\,
\label{eq36}
\end{equation}
where $\text{RQCD}\equiv\frac{\text{MJBG}_{\text{CR OS}}}{\text{MJBG}_{\text{CR SS}}}$. So, eq. \eqref{eq36} gives the estimation of MJBG on the SROS.
\begin{figure}[htbp]
	\centering
	\includegraphics[width=0.5\textwidth]{figures/Fig7.png}
	\caption{Regions defined to estimate the MJBG contribution in the SROS region. This data-driven method is also known as the ABCD method.}
	\label{Fig7}
\end{figure}
Table \ref{tab:MJ} summarises inputs into the calculation of the MJ background in the three candidate event samples.

\begin{table}[]
	\resizebox{\textwidth}{!}{%
		\begin{tabular}{ccccc}
			\rowcolor[HTML]{C0C0C0}
			\textbf{Sample} & \multicolumn{2}{c}{\cellcolor[HTML]{C0C0C0}\textbf{$\mu \tauhad$}} & \multicolumn{2}{c}{\cellcolor[HTML]{C0C0C0}\textbf{$e \tauhad$}} \\
			& 1-prong                                  & 3-prong                                 & 1-prong                                 & 3-prong                                \\ \hline
			CR OS Data      & 476.0 $\pm$ 22                                 & 270.0 $\pm$ 16                               & 62.0  $\pm$ 8                               & 39.0 $\pm$ 6                               \\
			MC              & 28.608 $\pm$ 4.068                               & 10.531 $\pm$ 1.833                              & 21.058 $\pm$ 10.808                              &              4.794 $\pm$ 1.558                 \\
			CR SS Data      & 366.0 $\pm$ 19                                 & 151.0 $\pm$ 12                                 & 56.0 $\pm$ 7                                & 28.0 $\pm$ 5                                \\
			MC              & 3.46 $\pm$  0.740                               & 0.381 $\pm$ 0.229                                 & 1.152 $\pm$ 0.458 0.444                               &                    0.843 $\pm$             \\ \hline
			RQCD            & 1.234 $\pm$ 0.081                                  & 1.723 $\pm$ 0.178                                 & 0.746 $\pm$ 0.208                                  & 1.260 $\pm$ 0.342                                \\ \hline
			SR SS Data      & 95.0 $\pm$ 10                                  & 13.0 $\pm$ 4                                  & 113.0 $\pm$ 11                                 & 18.0 $\pm$ 4                                 \\
			MC              & 57.257 $\pm$ 4.799                                & 6.761 $\pm$ 2.363                                  & 63.585 $\pm$ 4.793                               & 19.240 $\pm$ 2.651                               \\ \hline
			MJ Background   & 46.577 $\pm$  13.825	                               & 10.748 $\pm$ 13.870                                & 36.887 $\pm$  15.695                              & 0.0 $\pm$ 15.695
		\end{tabular}%
	\caption{Inputs for the calculation of the central value of the MJBG yield in the two final states, $Z\to\mu\tauhad$ and $Z\to e\tauhad$.
		The number of data and MC events in the CR OS and CR SS samples are given.
		The excess of data with respect to simulation is assumed to arise from MJBG and it is used to calculate the value of RQCD.
	}

	\label{tab:MJ}
}

\end{table}

Fig.\ref{Fig21} shows the jetRNN score distribution in the SRSS for the $Z\to\tauhad\mu$ final state. Fig. \ref{Fig12} shows the jetRNN score distributions for the events in the CRs. It can be seen that the samples are dominated by non-simulated background.

\begin{figure}[htbp]
	\centering
	\includegraphics[width=0.5\textwidth]{figures/Fig21}
	\caption{jetRNNScore distribution for 1-prong taus in the SRSS. The data to MC difference gives the shape for the MJBG contribution. After that, the shape is scaled by RQCD to obtain the MJ in the SROS.}
	\label{Fig21}
\end{figure}

\begin{figure}[htbp]
	\centering
	\subfloat[]{\label{Fig12a}{\includegraphics[width=0.50\textwidth]{figures/Fig12a.png}}}
	\subfloat[]{\label{Fig12b}{\includegraphics[width=0.50\textwidth]{figures/Fig12b.png}}}\hfill
	\subfloat[]{\label{Fig12c}{\includegraphics[width=0.50\textwidth]{figures/Fig12c.png}}}
	\subfloat[]{\label{Fig12d}{\includegraphics[width=0.50\textwidth]{figures/Fig12d.png}}}
	\caption{Left column (a,c) represents $Z\to\tauhad\mu$ and $Z\to\tauhad e$ is in the right column (b,d). The two figures on top show the CROS and the bottom ones represent the CRSS. All of the plots show the jetRNNScore for 1 prongs. In this case the events used to calculate RQCD are the ones with jetRNNScore$<0.4$.  }
	\label{Fig12}
\end{figure}

\subsection{Electroweak Background}
The electroweak background (EWBG) sources considered are $t\bar{t}$, $\Zjets$, Diboson and single top processes. For all of them, simulated samples are available and they are listed in Table \ref{Table3}. At an early stage of our analysis the $\Wjets$  simulation was checked for correctly accounting for the expected EWBG. This study is described in Appendix \ref{wjetsstudy}. Nonetheless, the final $\Wjets$ contribution to our background yield is so small that an updated study will not improve the quality and precision of our results.


\renewcommand{\nrandom}{\num{700}}
\renewcommand{\napplication}{\num{212}}
\newcommand{\ntoilet}{\num{77}}
\newcommand{\nmaxcount}{\num{26}}
\newcommand{\nsandcastle}{\num{25}}
\newcommand{\nconformant}{\num{24}}
\newcommand{\nmpec}{\num{60}}

\section{Evaluation}
\label{sect:erssat-evaluation}

We evaluated the proposed~\cref{alg:erssat} against
the state-of-the-art DPLL-based SSAT solver \dcssat~\cite{Majercik2005}
over both random $k$-CNF and application formulas.
The proposed algorithm is implemented in the \texttt{C++} language inside the \abc~\cite{ABC} environment.
The SAT solver \minisat-2.2~\cite{Een2003Solver} is used to answer satisfiability queries.
For weighted model counting,
we tried \cachet~\cite{Sang2004,Sang2005ModelCounting},
but the overall performance was not satisfactory.
Instead, we resorted to a well-developed BDD package \cudd~\cite{CUDD}.
Weight computation of a formula is fulfilled via a classic approach~\cite{Darwiche2002KnowledgeCompilation} that traverses the BDD of a formula and computes the satisfying probabilities of the BDD nodes.
Our prototyping implementation\footnote{Available at: \url{\ssatabcurl}} is named \erssat.
A bare version of \erssat without the clause-strengthening heuristics is called \erssatb in the experiments.
We used \ssatABCRevision in the experiments.

\subsection{Benchmark set}
The SSAT instances in the evaluation are hosted
in a publicly available database\footnote{Available at: \url{\ssatbenchmarkurl}}.
We used \ssatBenchRevision in the experiments.

\subsubsection{Random $k$-CNF formulas}
We generated random $k$-CNF formulas by \cnfgen~\cite{Lauria2017CNFgen}.
A collection of~\nrandom~formulas were generated with $k$,
i.e., the number of literals in a clause,
taking values from $\{3,4,5,6,7,8,9\}$,
the number of variables taking values from $\{10,20,30,40,50\}$,
and clause-to-variable ratio taking values from $\{k-1,k,k+1,k+2\}$.
Five formulas were sampled for each parameter combination.
To convert the propositional formulas into E-MAJSAT formulas,
the first half of the variables are existentially quantified,
and the rest are randomly quantified with probability $0.37$.

\subsubsection{Application formulas}
\begin{table}[ht]
    \centering
    \caption{The families of the application formulas}
    \label{tbl:exist-random-ssat-families}
    \begin{tabular}{c|c|c}
        Family               & Description                                               & Number       \\
        \hline
        \textit{Toilet-A}    & Adapted from exist-forall-exist QBFs~\cite{Narizzano2006} & \ntoilet     \\
        \textit{Conformant}  & Adapted from exist-forall-exist QBFs~\cite{Narizzano2006} & \nconformant \\
        \textit{Sand-Castle} & A probabilistic planning problem~\cite{Majercik1998}      & \nsandcastle \\
        \textit{Max-Count}   & Adapted from maximum model counting~\cite{Fremont2017}    & \nmaxcount   \\
        \textit{MPEC}        & Maximum probabilistic equivalence checking                & \nmpec       \\
    \end{tabular}
\end{table}

We collected five families of application formulas for evaluation.
Their descriptions and the numbers of the instances in each family are summarized in~\cref{tbl:exist-random-ssat-families}.
The first two families,
\textit{Toilet-A} and \textit{Conformant},
were adapted from exist-forall-exist QBFs~\cite{Narizzano2006}.
We converted the QBFs into exist-random-exist quantified SSAT formulas
by replacing their universal quantifiers with randomized ones with probabilities $0.5$.
The third family \textit{Sand-Castle} is a probabilistic conformant planning domain.
The problem can be encoded as E-MAJSAT formulas~\cite{Majercik1998}.
The family \textit{Max-Count} models the problems of maximum satisfiability, quantitative information flow, and program synthesis with maximum model counting~\cite{Fremont2017}.
We represented the maximum model counting instances as E-MAJSAT formulas.
The last family \textit{MPEC} consists of formulas that analyze the maximum probability of a probabilistic circuit to produce erroneous outputs,
as discussed in~\cref{chap:prob-design-eval}.

\subsection{Experimental setup}
Our experiments were performed on a machine with~\machineSpec.
The operating system was~\osInfo.
The programs were compiled with~\compiler.
Each SSAT-solving task was limited to a CPU core,
a CPU time of~\timelimit,
and a memory usage of~\memlimit.
To achieve reliable benchmarking,
we used a benchmarking framework \benchexec\footnote{Available at: \url{\benchexecurl}}~\cite{Benchmarking-STTT},
and assumed~\measurement.

\subsection{Results}

\subsubsection{Random $k$-CNF formulas}

\cref{fig:erssat-quantile-random} shows the quantile plots regarding CPU time and memory usage
of the SSAT instances derived from the random $k$-CNF formulas.
A data point $(x,y)$ in a quantile plot indicates that
there are $x$ formulas processed by the respective algorithm within a resource constraint of $y$.
In~\cref{fig:erssat-quantile-cputime-random},
we observe that \erssat solved a similar amount of formulas as \dcssat did.
Moreover, the clause-strengthening heuristics improve the performance of \erssat a lot,
as can be seen from the huge difference between \erssat and \erssatb.
On the other hand,
\cref{fig:erssat-quantile-memory-random} shows that \dcssat used much more memory than \erssat.
This can be attributed to the subformula caching of \dcssat.
Instead, \erssat only builds BDDs for cofactored formulas, which confined its memory footprint.

\begin{figure*}[hp]
    \centering
    \subfloat[CPU time]{
        \includegraphics{exist-random-ssat/evaluation/plots/quantile-cputime-Random.pdf}
        \label{fig:erssat-quantile-cputime-random}
    }\\
    \subfloat[Memory usage]{
        \includegraphics{exist-random-ssat/evaluation/plots/quantile-memory-Random.pdf}
        \label{fig:erssat-quantile-memory-random}
    }
    \caption{Quantile plots of random $k$-CNF formulas}
    \label{fig:erssat-quantile-random}
\end{figure*}

\subsubsection{Application formulas}

\input{exist-random-ssat/evaluation/tex/data-commands.tex}
\begin{table}[t]
    \centering
    \caption{Summary of the results for~\napplication~application formulas}
    \label{tbl:exist-random-ssat-application}
    \begin{tabular}{l|ccc}
        \toprule
        Algorithm                   & {\dcssat}                                                     & {\erssat} & {\erssatb} \\
        \midrule
        Solved formulas             & \num{\DcssatErDefaultApplicationMissingCount}
                                    & \num{\ErssatDefaultBddApplicationMissingCount}
                                    & \num{\ErssatBareBddApplicationMissingCount}                                            \\
        \qquad \textit{Toilet-A}    & \num{\dcssatToiletA}
                                    & \num{\erssatToiletA}
                                    & \num{\erssatbToiletA}                                                                  \\
        \qquad \textit{Conformant}  & \num{\dcssatconformant}
                                    & \num{\erssatconformant}
                                    & \num{\erssatbconformant}                                                               \\
        \qquad \textit{Sand-Castle} & \num{\dcssatcastle}
                                    & \num{\erssatcastle}
                                    & \num{\erssatbcastle}                                                                   \\
        \qquad \textit{Max-Count}   & \num{\dcssatMaxCount}
                                    & \num{\erssatMaxCount}
                                    & \num{\erssatbMaxCount}                                                                 \\
        \qquad \textit{MPEC}        & \num{\dcssatMPEC}
                                    & \num{\erssatMPEC}
                                    & \num{\erssatbMPEC}                                                                     \\
        Timeouts                    & \num{\DcssatErDefaultApplicationErrorTimeoutCount}
                                    & \num{\ErssatDefaultBddApplicationErrorTimeoutCount}
                                    & \num{\ErssatBareBddApplicationErrorTimeoutCount}                                       \\
        Out of memory               & \num{\DcssatErDefaultApplicationErrorOutOfMemoryCount}
                                    & \num{\ErssatDefaultBddApplicationErrorOutOfMemoryCount}
                                    & \num{\ErssatBareBddApplicationErrorOutOfMemoryCount}                                   \\
        Other inconclusive          & \num{\DcssatErDefaultApplicationErrorOtherInconclusiveCount}
                                    & \num{\ErssatDefaultBddApplicationErrorOtherInconclusiveCount}
                                    & \num{\ErssatBareBddApplicationErrorOtherInconclusiveCount}                             \\
        \bottomrule
    \end{tabular}
\end{table}

The solving results of the application formulas are summarized in~\cref{tbl:exist-random-ssat-application}.
For each compared approach,
the numbers of its exactly solved formulas,
timeouts, out of memory, and other inconclusive situations are reported.
To study the solving performance regarding different kinds of formulas,
we further report the numbers of exactly solved formulas per family.
Observe that \dcssat exactly solved the most formulas.
Its advantage mainly comes from family \textit{Sand-Castle},
where it solved \num{22} formulas,
but \erssat and \erssatb only solved \num{13} and \num{14} formulas, respectively.
We will analyze why the proposed clause-containment learning is not suitable for this family later.
To our surprise, the proposed clause-strengthening heuristics seem not very useful on the evaluated application formulas.
They even worsened the performance over formulas from the family \textit{Toilet-A}.
While \erssat and \erssatb suffered from more timeouts than \dcssat,
they did not run out of memory for any formula.
Instead, \dcssat tends to consume a lot of memory, because it memorizes many subformulas.

\begin{figure*}[hp]
    \centering
    \subfloat[CPU time]{
        \includegraphics{exist-random-ssat/evaluation/plots/quantile-cputime-Application.pdf}
        \label{fig:erssat-quantile-cputime-application}
    }\\
    \subfloat[Memory usage]{
        \includegraphics{exist-random-ssat/evaluation/plots/quantile-memory-Application.pdf}
        \label{fig:erssat-quantile-memory-application}
    }
    \caption{Quantile plots of application formulas}
    \label{fig:erssat-quantile-application}
\end{figure*}

\cref{fig:erssat-quantile-application} shows the quantile plots of the application SSAT formulas.
We can see that the clause-strengthening heuristics affected not only the effectiveness of \erssat but also its efficiency
from~\cref{fig:erssat-quantile-cputime-application}.
This phenomenon indicates that the additional effort spent to strengthen a learnt clause is not worthy.
The reason behind this phenomenon will be inspected in the following.

\begin{figure*}[hp]
    \centering
    \subfloat[\erssatb]{
        \includegraphics{exist-random-ssat/evaluation/plots/scatter-erssat.pdf}
        \label{fig:erssat-scatter-cputime-application}
    }\\
    \subfloat[\dcssat]{
        \includegraphics{exist-random-ssat/evaluation/plots/scatter-dcssat.pdf}
        \label{fig:dcssat-scatter-cputime-application}
    }
    \caption{Run-time scatter plots of application formulas with \erssat in y-axis and compared approaches in x-axis}
    \label{fig:erssat-scatter-application}
\end{figure*}

To further examine the suitability of the clause-strengthening heuristics,
we demonstrate the scatter plots with \erssat in y-axis and compared approaches in x-axis
in~\cref{fig:erssat-scatter-application}.
A data point $(x,y)$ in the plots indicates that there is a formula processed by both \erssat and a compared approach,
while \erssat took a CPU time of $y$~seconds and the other approach took a CPU time of $x$~seconds.
From~\cref{fig:erssat-scatter-cputime-application},
we find that the clause-strengthening heuristics did improve the solving of some formulas,
but more often they were an overhead to \erssatb.
\cref{fig:dcssat-scatter-cputime-application} also shows that
\dcssat was more efficient to exactly solve formulas than \erssat over the evaluated application formulas.

\subsubsection{Clause-containment learning over the \textit{Sand-Castle} problem}
As \erssat and \erssatb did not solve formulas from the probabilistic planning domain \textit{Sand-Castle} quite well,
we look into the problem and discuss our findings here.
The \textit{Sand-Castle} problem~\cite{Majercik1998} describes an agent who wants to build a sand castle on a beach.
The agent has two actions to choose from: digging a moat or erecting a castle.
A moat protects a castle from the water and increases the probability to successfully build a castle.
The agent must take a unique action at every stage.
Under the settings of conformant planning,
the agent must decide its strategy beforehand and does not have access to internal states
(whether a moat has been digged or a castle has been erected) during the execution.
Given a finite number of stages,
the problem asks to compute a strategy to maximize the chance of successfully building a castle at the last stage.
The agent's actions are encoded with existentially quantified variables,
and the nondeterminism in the state-transition mechanism is encoded with randomly quantified variables.

A formula that encodes the \textit{Sand-Castle} problem with $n$ stages has the form:
\begin{align}
    \pf=\bigwedge_{i=1}^n \pf_d^{(i)}\land\pf_e^{(i)},
\end{align}
where $\pf_d$ and $\pf_e$ are sets of clauses used to represent the state-transition mechanism
when the agent chooses to \textit{dig} a moat or \textit{erect} a castle, respectively.
As the state-transition mechanism is the same for every stage except for the variables recording the internal state,
we use the superscripts to indicate the stage indices.
We found that the \textit{Sand-Castle} problem has the following property:
the clause set $\pf_d^{(i)}$ (resp. $\pf_e^{(i)}$) will be selected
if and only if the agent chooses to dig a moat (resp. erect a castle) at stage~$i$.
In other words, each strategy of the agent (i.e., an assignment to the existentially quantified variables)
will select a distinct set of clauses.
Recall that the proposed clause-containment principle aims at blocking assignments
selecting a superset of clauses that has been selected by a previously explored assignment.
As a result, a learnt clause constructed based on this principle can only block the current assignment itself,
which means that \erssatb degenerates to merely brute-force search.
This theoretic reasoning is confirmed by the solving statistics (visible from the log files),
which show that \erssatb invoked $2^n-1$ model-counting queries.
For \erssat, the situation is worse due to partial assignment pruning,
which invokes additional model-counting queries to strengthen a learnt clause.
From the log files, we found that \erssat invoked twice numbers of model-counting queries than \erssatb,
because it had an additional trial but always ended in vain.

On the other hand, recall that \dcssat is tailored to exploit the structural characteristics of planning problems.
The \textit{Sand-Castle} formulas favors \dcssat, as the subformulas are essentially the same across the stages.
It is not surprising the formula caching and divide-and-conquer method works well with these formulas.

\subsubsection{Approximate solving}

\begin{table}[ht]
    \centering
    \scriptsize
    \caption{Results of solving the \textit{Conformant} family}
    \label{tbl:exist-random-ssat-conformant}
    \begin{adjustbox}{angle=90}
        \pgfplotstabletypeset[
            every head row/.style={before row={\toprule
                            & \multicolumn{4}{c}{\dcssat} & \multicolumn{8}{c}{\erssat} & \multicolumn{8}{c}{\erssatb}\\},after row=\midrule},
            every last row/.style={after row=\bottomrule},
            empty cells with={--},
            formula column/.list={0},
            time column/.list={1,3,7},
            prob column/.list={2,4,8},
            lbound column/.list={5,9},
            lbtime column/.list={6,10}
        ]
        {exist-random-ssat/evaluation/csv/parsed-conformant.csv}
    \end{adjustbox}
\end{table}

\begin{table}[ht]
    \centering
    \scriptsize
    \caption{Results of solving the \textit{Max-Count} family}
    \label{tbl:exist-random-ssat-maxcount}
    \begin{adjustbox}{angle=90}
        \pgfplotstabletypeset[
            every head row/.style={before row={\toprule
                            & \multicolumn{4}{c}{\dcssat} & \multicolumn{8}{c}{\erssat} & \multicolumn{8}{c}{\erssatb}\\},after row=\midrule},
            every last row/.style={after row=\bottomrule},
            empty cells with={--},
            formula column/.list={0},
            time column/.list={1,3,7},
            prob column/.list={2,4,8},
            lbound column/.list={5,9},
            lbtime column/.list={6,10}
        ]
        {exist-random-ssat/evaluation/csv/parsed-MaxCount.csv}
    \end{adjustbox}
\end{table}

\begin{table}[ht]
    \centering
    \scriptsize
    \caption{Results of solving the \textit{MPEC} family}
    \label{tbl:exist-random-ssat-mpec}
    \begin{adjustbox}{angle=90}
        \pgfplotstabletypeset[
            every head row/.style={before row={\toprule
                            & \multicolumn{4}{c}{\dcssat} & \multicolumn{8}{c}{\erssat} & \multicolumn{8}{c}{\erssatb}\\},after row=\midrule},
            every last row/.style={after row=\bottomrule},
            empty cells with={--},
            formula column/.list={0},
            time column/.list={1,3,7},
            prob column/.list={2,4,8},
            lbound column/.list={5,9},
            lbtime column/.list={6,10}
        ]
        {exist-random-ssat/evaluation/csv/parsed-MPEC.csv}
    \end{adjustbox}
\end{table}

Recall that the proposed~\cref{alg:erssat} solves an SSAT formula in a converging manner.
Instead of computing the exact satisfying probability at once,
it keeps deriving lower bounds of the satisfying probability of a formula.
This characteristic integrates exact and approximate solving into one approach.
In the following, we study the approximation ability of \erssat.
We choose families \textit{Conformant}, \textit{Max-Count}, and \textit{MPEC} for detailed investigation,
because all of the compared approaches ran out of CPU time or memory over most of their formulas.

\cref{tbl:exist-random-ssat-conformant,tbl:exist-random-ssat-maxcount,tbl:exist-random-ssat-mpec}
show the approximation results over the above three families, respectively.
For \dcssat, the CPU time and exact satisfying probability are reported.
For \erssat and \erssatb, in addition to the CPU and exact satisfying probability,
the tightest lower bound and the time elapsed to derive the lower bound are also shown in the tables.
A formula is not shown in the tables
if none of the approaches can solve it or derive a non-trivial lower bound for it.

As can be observed from the tables,
\erssat was able to derive tight lower bounds for formulas from these families,
while \dcssat suffered from timeouts over most of them.
The approximation ability of \erssat makes it useful for large formulas,
which cannot be exactly solved by the state-of-the-art approaches.

The above results on the random and application formulas suggest that:
\begin{itemize}
    \item The proposed solver \erssat achieves a similar performance as \dcssat in terms of CPU time and outperforms \dcssat in terms of memory consumption on random formulas.
    \item The proposed solver \erssat is not as good as \dcssat at exactly solving the application formulas, which can be attributed to the overhead caused by the clause-strengthening heuristics.
    \item The proposed solver \erssat is good at deriving tight lower bounds for large formulas. This approximation ability is especially valuable when the size of a formula is beyond the capability of the state-of-the-art exact solver.
\end{itemize}
To sum up, our experimental results demonstrate the unique value of the proposed clause-containment learning.