%!TEX root = ../thesis.tex
\chapter{Project Proposal}\label{proposal}

\section*{Overview}

Balls-into-bins can be seen as an abstraction of stochastic load balancing. The general goal is to allocate incoming balls (tasks) into bins (servers) achieving a sufficiently balanced load distribution. Without any constraints, this is a simple problem (e.g. round robin scheduling works  perfectly). However, for systems with many and distant servers, having a centralised load balancer that keeps track of all the load levels becomes infeasible due to networking, memory and processing complexity.  A simple alternative is called \OneChoice, where the ball is placed into a random bin. Unlike round robin, this is a stateless protocol, and it can be easily decentralised, both making it much more robust~\cite{nasir2015power}. One of the standard results in this field is that \TwoChoice, where two randomly chosen bins are queried about their load, and the ball is placed in the lesser loaded one, achieves a significant improvement over \OneChoice in minimising the maximum load~\cite{azar1999twochoice}. It turns out that such balls-into-bins protocols and results can be applied to various other computer science related problems as well apart from load balancing (of servers), for example hashing~\cite{dahlgaard2016power}, and fair cake-cutting~\cite{edmonds2006balanced}.

We consider alterations of the \TwoChoice setting, which provide potentially more realistic and more robust models of load balancing. Most of them are directly applicable, while some of them are mainly of theoretical interest. These settings often have free parameters that can be set (unlike \TwoChoice, that does not require any parameter).
The goal of this project is to learn optimal free parameters (or in general optimal protocols) for these different settings. To do this, we will mainly focus on Reinforcement Learning (RL) techniques, but conventional algorithms, such as dynamic programming will be explored as well. I will return to the motivation for choosing RL and its applicability after giving an example setting in the next paragraph.

The starting point will be the \TwoThinning process~\cite{feldheim2021thinning}. There are $n$ bins and $m=n$ balls: for each ball in the sequence, the protocol gets to know the load of a random bin chosen uniformly, and it should make a binary decision whether to allocate the ball there, or into a random bin chosen from the $n$ bins uniformly. The challenge is to find the right cut-off point for the $i$-th ball (how much load is required to choose the second option) based on the value of $i$, to minimise the maximum load after all the balls have arrived. This setting -- which is intermediate between \OneChoice and \TwoChoice~\cite{feldheim2021thinning} -- is motivated by avoiding possibly unnecessarily querying a second bin as it is in \TwoChoice. A possible RL model for this setting would have the load and current bin sample as a \textit{state}, the decision function as an \textit{action} and a function of the maximum load as a \textit{reward} that is only received at the end of the process.

The reason we chose to apply reinforcement learning (RL) techniques for these settings is that they are very hard to tackle using conventional methods, largely because of their large state space and probabilistic nature, while RL is effective in exploring large state spaces to optimise reward in a stochastic process. This combination of balls-into-bins and RL relates to recent work~\cite{mitzenmacher2020algorithms} on tackling hard combinatorial optimisation problems by Machine Learning. For some settings that we are going to investigate, there exist mathematical proofs about the optimality of some choice of free parameters, and results they achieve~\cite{feldheim2021thinning}. In these cases, our aim is to reproduce these optimal free parameters by learning. In other cases, there are no strong theoretical guarantees available so far. Here, the goal is evaluating the ``performance'' a RL algorithm can achieve, and -- where applicable -- compare it to conventional algorithms other than RL based ones, such as greedy or dynamic programming.

Due to the breadth of the project and the large variety of existing RL algorithms we will have to restrict our attention to a few of them. Because of its flexibility (e.g. model-free, immediate updates) and its wide-ranging success in similar situations~\cite{jang2019q}, we plan to start by Q-learning. We expect several issues to arise when implementing RL algorithms, for example in settings with high dimensional state spaces, deep learning versions of Q-learning might also be explored to compress the state space.

After \TwoThinning, we will look at more complicated settings. Some possible examples are listed next. There will not be time for exploring all combinations of these versions, and I will pick some of them based on the success of related previously explored versions.
\begin{itemize}
    \item
    \textbf{Extending the environment}
    \begin{itemize}
        \item
        Looking at the case with $m>n$ balls. Surprisingly, most of the theoretical results do only hold for the lightly loaded case ($m=n$)~\cite{berenbrink2006heavilyloaded}.
        \item
        \TwoChoice can be seen as sampling uniformly from the edges of the complete graph formed by the bins as nodes. However, in practice, the graph representing an underlying network might be different, as e.g., only nearby servers can be used. It turns out that the greedy strategy of \TwoChoice is -- perhaps counter-intuitively -- suboptimal for general graphs~\cite{bansal2021twochoicegraphical}.
        \item
        The balls and the bins might be heterogeneous, motivated by the different capacity/maximum load and requirements/weights of various tasks and servers. The bins might also provide noisy answers to the queries.
        \item
        In \TwoThinning if the sampled bin has too high load, the ball is placed in a random bin. Instead, the process could recursively restart by sampling another bin, until an acceptable bin is sampled or a maximum number of rounds is reached~\cite{czumaj1997randomized}.
    \end{itemize}

    \item
    \textbf{Representing the state space}

    \begin{itemize}
        \item
        Instead of allocating the $i$-th ball based only on that value of $i$, the protocol could keep track of some summary of the load levels, or even get the sampled bin's ID (before or after the decision is made). This is motivated by the fact that even though it is not feasible for the protocol (load balancer) to keep track of all the load levels, it might still store some bits of information in its memory to optimise its decisions.
        \item
        On the other hand, motivated by the reduction in communication (for potential efficiency and secrecy benefits), the protocol might be allowed to ask only one or more binary queries of the form ``is the load greater than $x$ or not?''.
        \item
        In these settings, the challenge of efficiently encoding the states for both the RL algorithm and the protocol it produces naturally arises.
    \end{itemize}
    \item
    \textbf{Changing the objective function}

    \begin{itemize}
        \item
        Depending on the exact application domain, the objective function might differ from minimising the maximum load (e.g. minimising empirical standard deviation).
        \item
        It could also take into account some cost for asking another query, aim for a continuously balanced load, or even try to reach an arbitrary target distribution.
    \end{itemize}
\end{itemize}

\section*{Evaluation}

Evaluation will mainly consist of comparing the ``performance'' of (RL/conventional) algorithms across the several combinations of the settings discussed above. The first component of the comparison will be the efficiency (mainly time and memory complexity) for finding (learning) the values of the free parameters. The second, most important component is (the efficiency and) the score (according to the corresponding objective function) of the resulting protocol, potentially compared against the mathematically proven bounds as well.

For the second type of comparisons, we will use randomised simulations. The number of runs will be chosen based on the time complexity of simulations, while adhering to the principles of statistical significance. The same applies to the number of balls and bins used, but both larger and smaller inputs will be experimented with, especially because many theoretical bounds hold only for sufficiently large $m$, so for small $m$, a RL algorithm might outperform it.

Where applicable, algorithms and protocols will be evaluated analytically too.

\section*{Possible extensions}

\begin{itemize}
    \item
    Implementing an end-to-end application of one of the settings (e.g. hashing). This would mean that the objective function would not be created intuitively (e.g. minimising maximum load), instead the application domain's objective function can be used.
    \item
    Explaining the behaviour of a trained RL model by mapping it to a conventional algorithm or interpreting the encoding of the states (that is, how it can usefully represent the relevant information about the loads without explicitly storing the entire load vector).
    \item
    Formally conjecturing optimal parameters and bounds on results deduced from learnt processes and simulations. If time permits, further extension would be proving these.
    \item
    Finding intermediate metrics for assessing how favourable the load distribution is at a moment during the process according to the final objective function. Those two can easily be different.
    \item
    Creating a series of benchmark challenges for RL methods with gradually increasing difficulty. Having mathematically proven results for some settings, the correctness and efficiency of the methods could conveniently be compared on these benchmarks.
\end{itemize}

\section*{Starting point}

Before preparation, I had no experience in RL and balls-into-bins problems. Over the summer, I did much background reading on the theory of RL, and I have read several papers about various settings of balls-into-bins, and corresponding proofs on bounds and optimality of the protocols. This also helped me gain a better understanding of the potentials of applying RL algorithms to this area.

I have not yet written any code or other material for this project at the time of this proposal. I have a solid knowledge of Python and its machine learning libraries, but I will have to familiarise myself with its specific RL tools.

\section*{Success criteria}

I will consider the project to be successful iff:

\begin{itemize}
    \item
    Some RL models have been trained and applied on various settings.
    \item
    Conventional algorithms (such as dynamic programming) have been implemented as an alternative to RL.
    \item
    A thorough evaluation has been conducted according to what is stated in the Evaluation section.
    \item
    An easy-to-use and easy-to-extend framework has been implemented, where new settings and algorithms can be tested out.
\end{itemize}

\section*{Work plan}

My work plan consists of 2-week work packages, some parts taking up multiple packages. A few of them are for contingency, revision and resting. Milestones are also attached. Throughout each package I will maintain a logbook with all my ideas, TODOs, and accomplishments which can later be used for writing the dissertation.

\begin{itemize}
    \item
    \textbf{7 Oct--20 Oct}: experiment with RL in Python and implement some settings without free parameters (e.g. \TwoChoice) as preparation.

    Milestones: none
    \item
    \textbf{21 Oct--3 Nov}: implement the first RL method for \TwoThinning, together with conventional algorithms (e.g. dynamic programming). Plan the most reasonable next setting or alternative approach to the same setting to work on, according to the results.

    Milestones: first RL model ready
    \item
    \textbf{4 Nov--17 Nov}: turn to other similar settings, potentially try multiple RL algorithms. It will also require some further background reading in the balls-into-bins literature.

    Milestones: none
    \item
    \textbf{18 Nov--1 Dec}: continue previous package

    Milestones: several settings explored, initial comparisons created
    \item
    \textbf{2 Dec--15 Dec}: if there is a straightforward way towards an extension based on the results (e.g. a formal conjecture can be made about the optimal parameters), go for that, otherwise fine-tune the previous methods. Also create a write-up for myself, supervisors, and the department.

    Milestones: progress report created
    \item
    \textbf{16 Dec--29 Dec}: revision, winter break, contingency. Possibly working on some extensions.

    Milestones: being fresh for resuming working on the project
    \item
    \textbf{30 Dec--12 Jan}: explore further settings, including those which are not simple alterations of the initial setting and might require a completely new algorithm, e.g. changing the objective function.

    Milestones: none
    \item
    \textbf{13 Jan--26 Jan}: continue previous package, start evaluating the core deliverables by analytical methods.

    Milestones: algorithms implemented for substantially different settings
    \item
    \textbf{27 Jan--9 Feb}: continue evaluating the core deliverables, with main focus on running simulations.

    Milestones: comparison of algorithms on various settings is created
    \item
    \textbf{10 Feb--23 Feb}: start writing the draft of the main body of the dissertation, including the fresh evaluations too. Continue with extensions as time permits.

    Milestones: main sections of the first draft completed
    \item
    \textbf{24 Feb--9 Mar}: based on all the work done so far including the extensions as well, write initial draft of the remaining sections too.

    Milestones: dissertation submitted for first feedback
    \item
    \textbf{10 Mar--23 Mar}: revision, Easter break, contingency

    Milestones: supervisors and friends urged for giving feedback
    \item
    \textbf{24 Mar--6 Apr}: start working on a second draft based on the feedback collected, and finish any remaining evaluation or extension still running.

    Milestones: none
    \item
    \textbf{7 Apr--20 Apr}: continue updating the second draft.

    Milestones: second draft sent in for feedback
    \item
    \textbf{21 Apr--4 May}: finalise dissertation based on the feedback.

    Milestones: final dissertation submitted
\end{itemize}

\section*{Resource declaration}

\begin{itemize}
    \item
    My personal laptop: for convenience, I am planning to use my own laptop for writing code and storing files, instead of the MCS facility. The whole project will be version controlled on git, and weekly backup copies of the repository will be made on Google Drive as well. I might create backup copies on MCS too.
    \item
    HPC facility for GPUs: for training complex neural networks efficiently (which is not part of the core deliverable), I might need strong GPUs. Therefore, together with my supervisors, we have already applied for computational credits on the HPC facility of the university.
\end{itemize}


\section{Predicate abstraction}

After the replacements carried out in the buffer size analysis section, we have a program that both encapsulates the behaviour of the chosen WMM, and that can be abstracted according to the rules described in section 4 up and until 4.4 of \cite{balletal01}. As our program is not written in C, but in SALPL, and the output language must be recognized by our model checker, Fender, we state the following alterations to the aforementioned rules: there are no procedure calls, pointers, arrays, or any variables of a type other than integer in our input program, and no variables of a type other than boolean in the output. In this section, we define the components, and lastly, the overall structure of our output boolean program. We will use the notation $\sigma[\lstinline$a$ / \lstinline$b$]$ to denote $\sigma$ with all occurences of \lstinline$b$ replaced by \lstinline$a$.\\

For the predicate abstraction itself, we declare two boolean variables for each predicate $\varphi_i \in E$: \lstinline$B_$$i$ (boolean variables) and \lstinline$T_$$i$ (temporary variables). Both of them hold the truth value of $\varphi_i$, albeit at different times. We can guarantee however, that \lstinline$B_$$i$ is always the truth value of $\varphi_i$. \lstinline$T_$$i$ on the other hand is used during parallel assignments, where we simultaneously assign values to more than one boolean variable, while still needing to read values thereof from before the commencing of the parallel assignments. These variables are also declared for the additional predicates defined below. Also, keeping in line with the notation in the paper by Ball et al. \cite{balletal01}, we will use the notation $WP(\lstinline$s$, \varphi)$ for the weakest liberal precondition of the predicate $\varphi$ with respect to the statement $s$, and $\mathcal{F}_{V}(\varphi)$ for the largest disjunction of implicative cubes for a predicate $\varphi$ over a set of boolean variables $V$.

\subsection{Variables and predicates}

Our program has a set of variables $H$, as declared in the original input program, and a set of predicates $E$ over $H$. The predicates in $E$ are sufficient to verify the program under sequential consistency constraints (i.e. without WMM-related effects). Among the newly declared local variables, i.e. all variables that weren't part of the original program, but since have been declared during the buffer size analysis stage, we differentiate between auxiliary variables, which encompass all first pointers, counters, and buffer allocators of the forms \lstinline$x_fst_$$i$, \lstinline$x_cnt_$$i$, and \lstinline$own_x_$$i$, respectively, and buffer variables, which describe all variables abstracting WMM write buffer data. All buffer variables are assigned to a single global variable at any program point. This follows from the buffer variables' inherent association with a single global variable in the case of PSO, and TSO buffer variables' only being used after allocation to a global variable. With this information, for each global variable \lstinline$x$ and each valid process number $i$, we extend our predicate set by the following predicates:

\begin{multicols}{2}

In the case of PSO:

\begin{lstlisting}[frame=single, mathescape]
(x_cnt_$i$ >= 0)
(x_cnt_$i$ <= $K$)
(x_fst_$i$ >= 0)
(x_fst_$i$ <= $K$)
\end{lstlisting}

\columnbreak

In the case of TSO:

\begin{lstlisting}[frame=single, mathescape]
(own_1_$i$ = $\rho($x$)$)
(own_1_$i$ = $\rho($x$)$)
$...$
(own_$j$_$i$ = $\rho($x$)$)
$...$
(own_$s_{i}$_$i$ = $\rho($x$)$)
\end{lstlisting}

\end{multicols}

We will handle these new predicates as we would handle the original ones. Furthermore, for every process $i$, we define the following extended predicate set for PSO:

\[E':=\{\varphi' := \varphi[\lstinline$x_$j\lstinline$_$i / \lstinline$x$] \: | \: \lstinline$x$ \in \varphi \in E \cup E' \wedge \lstinline$x$ \in H \wedge j \in [1, s_{x,i}]\},\]

while in the case of TSO, our extended predicate set is defined as follows:

\[E':=\{\varphi' := \varphi[\lstinline$buf_$j\lstinline$_$i / \lstinline$x$] \: | \: \lstinline$x$ \in \varphi \in E \cup E' \wedge \lstinline$x$ \in H \wedge j \in [1, s_{x,i}]\}.\]

Though we will handle the latter predicates the same way we would handle the others, unless stated otherwise, we will make note of them being \textit{buffer predicates} and copies of certain different predicates $\in E$.

\subsection{The choose statement}

The \lstinline$choose($$pos$\lstinline$, $$neg$\lstinline$);$ statement is defined in subsection 4.3. We will make use of it during our predicate abstraction of assignments. Paraphrased, it states for two boolean variables \lstinline$A$ and \lstinline$B$:

\begin{lstlisting}[frame=single, mathescape]
if (A $\equiv true$)
	choose(A, B) $\equiv true$
else if (B $\equiv true$)
	choose(A, B) $\equiv false$
else
	choose(A, B) $\equiv$ *
endif;
\end{lstlisting}

Note that boolean variables in Fender may have the value \lstinline$*$, i.e. indecisive.

\subsection{Weakest liberal preconditions}

As defined in subsection 4.1 of \cite{balletal01}, $WP(\lstinline$s$, \varphi)$ is defined as the weakest predicate whose truth before $s$ entails the truth of $\varphi$ after $s$ terminates (if $s$ terminates), and in the case of $s$ being an assignment statement of the form \lstinline$x = e$, it equals $\varphi[\lstinline$e$ / \lstinline$x$]$. In this work, we will only compute the weakest liberal preconditions of assignment statements.

\subsection{Largest implicative cubes}

Paraphrasing the definitions in subsection 4.1 of \cite{balletal01} to better suit the semantics of this work, we define a cube over a boolean variable set $V$ as follows: an implicative cube for a predicate $\varphi$ is a conjunction of boolean variables $\in V$ (negated or not) representing different predicates $\in E$, which implies $\varphi$. We say for a cube $C$ to be the "superset" of another cube $D$ iff $D$ can be expressed as the conjunction of $C$ and a different non-empty cube $C'$ lacking any variable appearing in $C$. Continuing with our paraphrasing, we define $\mathcal{F_V}(\varphi)$ with the size limit $L$ as the disjunction of all cubes over $V$ implying $\varphi$ with $L$ or less literals, and $\mathcal{G_V}(\varphi) := \neg\mathcal{F_V}(\neg\varphi)$. For practicality reasons, and without loss of generality, we can redefine the largest disjunction of implicative cubes as lacking any two cubes of which one is the superset of another. We construct such cube disjunctions conservatively, i.e. once we find that a cube $c$ implies a predicate, we will omit any supersets of $c$ from our disjunction, as described in subsection 5.2 of \cite{balletal01}.

\subsection{Replacement rules}

We use the replacement rules defined in section 4 of \cite{balletal01}, albeit in a reduced and slightly adapted form, as we need not consider the caveats of the C language.

\subsubsection{Local assignment-, store-, and load statements}

We reduce all three types of assignment statements to the same form \lstinline$l = r$. First, we examine the left-hand side of the assignment. If it is a buffer variable, it will be regarded as the global variable it represents for all purposes with the exception of determining the left-hand sides of all initialization-, assignment- and reset statements. Next, we build the set of relevant predicates $R$ out of any predicate that contains \lstinline$l$, save for buffer predicates. As long as we encounter predicates containing non-buffer variables, we extend $R$ by all predicates containing the new variables, which will result in the extended set of relevant predicates $R'$. All the while, we keep ignoring and omitting buffer variables and buffer predicates. Once this closure has been computed, we have all we need to replace the assignment with:

\begin{lstlisting}[frame=single, mathescape]
begin_atomic;
$[initialization\:of\:temporary\:variables]$
$[assignment\:of\:boolean\:variables]$
$[reset\:of\:temporary\:variables]$
assume(!($\mathcal{F}_{V}($false$)$));
end_atomic;
\end{lstlisting}

In this atomic block, we simulate the parallel assignment of the new boolean values to their variables. We define the \textit{initialization of temporary variables} for each predicate $\in R$ with the index $j$ as: \lstinline$load T_$$j$\lstinline$ = B_$$j$\lstinline$;$. Analogously, we define the \textit{reset of temporary variables} as: \lstinline$T_$$j$\lstinline$ = 0;$, which leaves us with the \textit{assignment of boolean variables}, which is also done for each predicate $\in R$ with the index $j$, and in accordance to subsection 4.3 of \cite{balletal01} as follows:

\begin{lstlisting}[frame=single, mathescape]
store B_$j$ = choose($\mathcal{F}_{R'}(WP($l = r$, \: \varphi_j))$, $\mathcal{F}_{R'}(WP($l = r$, \: \neg\varphi_j))$);
\end{lstlisting}

If the initial left-hand side \lstinline$l$ was a buffer variable, we proceed differently for each store order.

\pagebreak

\paragraph{PSO approach}

We scan through every cube in every disjunction, and we extend the corresponding cube disjunction

\[C \in \{\mathcal{F}_{R'}(WP(\lstinline$l = r$, \: \varphi_j)), \mathcal{F}_{R'}(WP(\lstinline$l = r$, \: \neg\varphi_j))\}\]

with the following cubes:

\[C' := \{c' := c[b'/b]\: | \: \lstinline$l$ \in b' \wedge b \mapsto b' \wedge b \in c \in C \cup C'\},\]

where by $b \mapsto b'$, we denote the fact that we can perform repeated replacements of any global variable by one of their buffer representations in a boolean term $b$, which may be a negated, or non-negated representation of a certain predicate $\varphi \in E \cup E'$, in order to receive an equivalent boolean term $b'$ with the same negation property as $b$, representing a predicate $\varphi' \in E'$, with all buffer variables in $\varphi$ and $\varphi'$ replaced by their global counterparts resulting in the same predicate. To illustrate this relation, consider the following example:\\

If \lstinline$B_1$ represents the predicate $\varphi_1 := ($\lstinline$x $$>$\lstinline$ y$$)$, \lstinline$B_2$ represents the predicate $\varphi_1' := ($\lstinline$x_1_1 $$>$\lstinline$ y$$)$, \lstinline$B_3$ represents the predicate $\varphi_1'' := ($\lstinline$x_2_1 $$>$\lstinline$ y$$)$, and \lstinline$B_4$ represents the predicate $\varphi_1''' := ($\lstinline$x_1_1 $$>$\lstinline$ y_1_1$$)$, then only the following expressions are correct over these boolean variables: \lstinline$B_1$ $\mapsto$ \lstinline$B_2$, \lstinline$!B_1$ $\mapsto$ \lstinline$!B_2$, \lstinline$B_1$ $\mapsto$ \lstinline$B_3$, \lstinline$!B_1$ $\mapsto$ \lstinline$!B_3$, \lstinline$B_1$ $\mapsto$ \lstinline$B_4$, \lstinline$!B_1$ $\mapsto$ \lstinline$!B_4$, \lstinline$B_2$ $\mapsto$ \lstinline$B_4$, \lstinline$!B_2$ $\mapsto$ \lstinline$!B_4$.

\paragraph{TSO approach}

We fetch the global variable represented by \lstinline$l $ $\equiv$ \lstinline$ buf_$$j$\lstinline$_$$i$ by climbing one step up the abstract semantic tree, which invariably brings us to an if-else statement containing the global variable \lstinline$x$ represented by \lstinline$l$ in the current program point. We have previously illustrated this information deposition by commenting the if-statement.
In the resulting pair of cube disjunctions, we first replace every boolean term containing $b$ with
%\lstinline$choose($$b[l/x]$\lstinline$ && (own_$$j$\lstinline$_$$i$\lstinline$ = $$\rho($\lstinline$x$$)$\lstinline$), own_$$j$\lstinline$_$$i$\lstinline$ = $$\rho($\lstinline$x$$)$)$
\begin{lstlisting}[frame=single, mathescape]
choose($b[$l$/$x$]$ && $\mathcal{B}($own_$j$_$i$ = $\rho($x$))$, $\mathcal{B}($own_$j$_$i$ = $\rho($x$))$)
\end{lstlisting}

$\mathcal{B}(\varphi)$ denotes the boolean variable representing $\varphi$. Then, all cubes with boolean expressions containing global variables are replicated once for every possible combination of buffer variables replacing any occurence of a global variable, with the replacements carried out the same way as above. The following example illustrates this round of extension:\\

\pagebreak

Let \lstinline$choose($$\mathcal{B}($\lstinline$buf_1_1 < y$$)$\lstinline$ && $$\mathcal{B}($\lstinline$own_1_1 = 5$$)$\lstinline$, $$\mathcal{B}($\lstinline$own_1_1 = 5$$)$\lstinline$)$ be one of the replacements made with the above step. Assume $s_{i} = 3$. Therefore, there are 3 buffer variables, 2 of which might be containing a value of \lstinline$y$ with $\rho($\lstinline$y$$) = 7$. Because of this, 2 replicas of the same cube containing this example term are generated, with this term replaced by the following two terms, respectively:

\begin{lstlisting}[frame=single, mathescape]
choose($\mathcal{B}($buf_1_1 < choose(
  $\mathcal{B}($buf_1_1 < buf_2_1$)$ && $\mathcal{B}($own_2_1 = 7$)$, $\mathcal{B}($own_2_1 = 7$)$
)$)$ && $\mathcal{B}($own_1_1 = 5$)$, $\mathcal{B}($own_1_1 = 5$)$)

choose($\mathcal{B}($buf_1_1 < choose(
  $\mathcal{B}($buf_1_1 < buf_3_1$)$ && $\mathcal{B}($own_3_1 = 7$)$, $\mathcal{B}($own_3_1 = 7$)$
)$)$ && $\mathcal{B}($own_1_1 = 5$)$, $\mathcal{B}($own_1_1 = 5$)$)
\end{lstlisting}

Both replica cubes are added to the cube disjunction.

\subsubsection{Conditionals}

We consider conditionals of the form \lstinline$if ($$\varphi$\lstinline$) $[\textit{if block}]\lstinline$ else $[\textit{else block}]\lstinline$ endif;$ and replace them as follows, using the extended relevant predicate set $R'$ of all variables featured in $\varphi$, computed the same way as described in the previous subsection:

\begin{lstlisting}[frame=single, mathescape]
if (*) goto $else\:label$;
assume($\mathcal{G}_{R'}(\varphi)$);
$[if\:block]$
goto $end\:label$;
$else\:label$: assume($\mathcal{G}_{R'}(\neg\varphi)$);
$[else\:block]$
$end\:label$: nop;
\end{lstlisting}

\subsubsection{Assertions}

For any predicate $\varphi$ in the assertion clause that doesn't contain a \lstinline$pc($$i$\lstinline$)$ counter, we compute its extended relevant predicate set $R'$, as described above, and replace those predicates with $\mathcal{F}_{R'}(\varphi)$.

\pagebreak

\subsubsection{Overall boolean program structure}

We build our final program to be verified with our model checker the following way:

\begin{lstlisting}[frame=single, mathescape]
shared B_0, $...$, B_$|E'| - 1$;
local T_0, $...$, T_$|E'| - 1$;

init
	store B_0 = *;
	$...$
	store B_$|E'| - 1$ = *;
	load T_0 = B_0;
	$...$
	load T_$|E'| - 1$ = B_$|E'| - 1$;

	$[predicate\:abstractions\:of\:initialization\:stores]$

	T_0 = 0;
	$...$
	T_$|E'| - 1$ = 0;

	assume(!($\mathcal{F}_V($false$)$));

process 1
	assume(!($\mathcal{F}_V($false$)$));
	$[predicate\:abstraction\:of\:the\:process\:block]$
$...$
$[predicate\:abstraction\:of\:the\:assertion]$
\end{lstlisting}


Principal component analysis (PCA) is a well-known dimensionality reduction technique, widely used for data pre-processing and exploratory data analysis (EDA). Although popular for the interpretability of its results and ease of implementation, PCA's ability to extract signal from high-dimensional data is demonstrably unstable \citep{Shen2013,Johnstone2009}, in that its recovered results can vary widely with perturbations of the data \citep{yu2013}. What is more, PCA is often unable to reduce the dimensionality of the data in a contextually meaningful manner \citep{ringner2008,Abid2018}. Consequently, variants of PCA have been developed in attempts to remedy these severe issues, including, among many others, sparse PCA (SPCA) \citep{Zou2006}, which increases the interpretability and stability of the principal components in high dimensions by sparsifying the loadings, and contrastive PCA (cPCA) \citep{Abid2018}, which captures relevant information in the data by eliminating technical effects through comparison to a so-called background dataset. While SPCA and cPCA have both individually proven useful in resolving distinct shortcomings of PCA, neither is capable of simultaneously tackling the issues of stability, interpretability, and relevance. We propose a combination of these techniques, \textit{sparse constrastive PCA} (scPCA), which draws on cPCA to remove unwanted technical effects and on SPCA for sparsification of the loadings, thereby extracting stable, interpretable, and relevant uncontaminated signal from high-dimensional biological data.

\subsection{Motivation}

A longstanding problem in genomics and related disciplines centers on teasing out important biological signal from technical noise, i.e., removing \textit{unwanted} variation corresponding to experimental artifacts (e.g., batch effects). A common preliminary approach for accomplishing such a task involves the application of classical PCA to capture and deflate technical noise, followed by traditional statistical inference techniques (e.g., clustering cells, testing for differences in mean gene expression levels between populations of cells) \citep{Nguyen2019}. Such an approach operates under the assumption that meaningful biological signal is not present in the leading principal components (PC), and that the removal of the variance contained therein allows recovery of the signal previously masked by technical noise. Should these assumptions prove unmet, relevant biological signal may be unintentionally discarded, or worse, technical noise may be significantly amplified.

Several more sophisticated approaches have been proposed, including the use of control genes \citep{gagnon2012using, risso2014normalization} and control samples \citep{gagnon2013removing} whose behavior is known \textit{a priori}. Unfortunately, access to such controls may be severely limited in many settings (e.g., as with prohibitively expensive assays). Alternative approaches, for use in settings where control genes or control samples are unavailable, such as surrogate variable analysis \citep{leek2007capturing}, reconstruct sources of unwanted variation that may subsequently be controlled for via covariate adjustment in a typical regression modeling framework. In the context of single-cell transcriptome sequencing (scRNA-seq) data, a class of data that has garnered much interest due to the granularity of biological information it encodes, related approaches have been combined as part of the ZINB-WaVE methodology \citep{risso2017zinb}, which relies on factor analysis for a zero-inflated negative binomial (ZINB) generalized linear model (GLM) to remove unwanted variation.

Although such approaches have proven useful, model-based techniques rely on assumptions about the data-generating process to target biological signal, warranting that caution be exercised in their use. Additionally, owing to the diversity of experimental settings in high-dimensional biology, such techniques are often targeted to specific experimental paradigms (e.g., bulk RNA-seq but \textit{not} single-cell RNA-seq). Violations of the assumptions embedded in these techniques may often be difficult -- impossible, even -- to diagnose, leading to a lack of agreement in findings between different model-based approaches when applied to the same datasets. Accordingly, \citet{zhang2014} have shown the lack of consensus among model-based differential expression techniques on RNA-seq datasets, demonstrating that their use gives rise to subjective analyses. By contrast, we propose a wholly data-driven approach to removing unwanted variation, harnessing the information contained in control samples, pre-treatment groups, or other signal-free observations, all while enhancing the interpretability and stability of findings by inducing sparsity.

The remainder of the present manuscript is organized as follows. In Section~\ref{back}, contrastive PCA, sparse PCA, and other popular dimensionality reduction techniques are briefly surveyed. Next, in Section~\ref{method}, scPCA is formally defined and its desirable properties are detailed. A simulation study and several analyses of publicly available microarray gene expression and single-cell transcriptome sequencing data are presented in Section~\ref{results} and the analysis of a protein expression dataset is detailed in Section~\ref{sup_mice}, providing a rich comparison of the proposed methodology to other popular techniques currently relied upon for the exploration of high-dimensional biological data. Finally, we conclude by reviewing the effectiveness of scPCA based on the results of these studies and discussing paths for further investigation.

\subsection{Background}\label{back}

\subsubsection{Contrastive PCA}\label{cpca}

The development of contrastive PCA was motivated by the need to detect and visualize variation in the data deemed most relevant for the scientific question of interest. Given a target dataset believed to contain biological signal of interest and a similar background dataset believed to comprise only noise (i.e., unwanted variation), the cPCA algorithm returns a subspace of the target data that contains (a portion of) the variation absent from the background data \citep{Abid2018}. Specifically, cPCA aims to identify the pertinent variation in the target data by \textit{contrasting} the covariance matrix of its features with that of the background data. For example, consider a scRNA-seq dataset whose samples are contaminated by a batch effect. Given a collection of control samples subjected to the same batch effect, cPCA may be used to remove this unwanted technical noise (see Section \ref{sim_scRNA-seq}).

Algorithmically, cPCA is very similar to PCA. Consider a column-centered target dataset $\mathbf{X}_{n \times p}$ and a column-centered background dataset $\mathbf{Y}_{m \times p}$, where $n$ and $m$ denote, respectively, the number of target and background observations (e.g., cells) and $p$ denotes the number of features (e.g., genes). Denote their empirical covariance matrices by $\mathbf{C_X}$ and $\mathbf{C_Y}$ and let $\mathbb{R}_{\text{unit}}^p=\{\mathbf{v} \in \mathbb{R}^p: \lVert\mathbf{v}\rVert_2 = 1\}$ be the set of unit vectors of length $p$ (i.e., $p$-dimensional vectors with unit norm). The variances along direction $\mathbf{v} \in \mathbb{R}_{\text{unit}}^p$ in the target and background datasets are represented by \edit{$f_{\mathbf{X}}(\mathbf{v}) = \mathbf{v}^\top\mathbf{C_X}\mathbf{v}$} and \edit{$f_{\mathbf{Y}}(\mathbf{v}) = \mathbf{v}^\top\mathbf{C_Y}\mathbf{v}$}, respectively. The most contrastive direction $\mathbf{v}^{\star}_\gamma$ for some fixed contrastive parameter $\gamma \in \mathbb{R}^+$ is found by solving
\begin{equation}
    \begin{aligned}
  \mathbf{v}^{\star}_\gamma & = \argmax_{\mathbf{v} \in \mathbb{R}_{\text{unit}}^p}
  \edit{f_{\mathbf{X}}(\mathbf{v})-
    \gamma f_{\mathbf{Y}}(\mathbf{v})} \\
  & = \argmax_{\mathbf{v} \in \mathbb{R}_{\text{unit}}^p}
    \mathbf{v}^\top(\mathbf{C_X} - \gamma\mathbf{C_Y})\mathbf{v} \\
  & = \argmax_{\mathbf{v} \in \mathbb{R}_{\text{unit}}^p}
    \mathbf{v}^\top\mathbf{C}_{\gamma}\mathbf{v},
\end{aligned}
\end{equation}
where $\mathbf{C}_{\gamma} =  \mathbf{C_X} - \gamma\mathbf{C_Y}$ is the contrastive covariance matrix \citep{Abid2018}. cPCA can therefore be performed by computing the eigenvalue decomposition of $\mathbf{C}_{\gamma}$. The eigenvectors of $\mathbf{C}_{\gamma}$ are then used to map the target data to the \textit{contrastive} principal components (cPC).

The contrastive parameter $\gamma$ quantifies the trade-off between each feature's variances in the target and background datasets. When $\gamma = 0$, cPCA reduces to PCA -- hence, only the target variance \edit{$f_{\mathbf{X}}(\mathbf{v})$} is maximized. \edit{As $\gamma$ increases, directions that reduce the background variance become more optimal and the cPCs are driven towards the null space of the background covariance matrix.} In particular, as $\gamma \rightarrow \infty$, the variance in the background data dominates the variance in the target data such that only directions spanned by the background dataset are captured. This is akin to \edit{projecting the target data on the null space of the background data and then performing PCA on the projected data} \citep{Abid2018}. The effect of the contrastive parameter is illustrated in Figure~\ref{contrastive_par} for simulated data similar to those used by \citet{Abid2018}.

Although cPCA offers a novel approach for the removal of unwanted variation, it possesses some drawbacks. In particular, no rigorous framework exists for selecting the contrastive parameter $\gamma$ in order to achieve the optimal amount of contrast between the target and background data. Indeed, \citet{Abid2018}'s \edit{semi-automated} approach to selecting an appropriate $\gamma$ relies on visual inspection. \edit{Alternatively, recent work by \citet{Fujiwara_2020} introduces a heuristic algorithm based on a one-dimensional rotation of the target data.} Additionally, as with PCA, loading vectors may be highly variable and difficult to interpret in high dimensions since they represent linear combinations of all features in the dataset. Relatedly, cPCs are not certifiably free of unwanted technical and biological effects, potentially obscuring relevant biological signal. \edit{Unwanted variation can only be removed from the target data if it is also captured by the background data.} This issue is only exacerbated as the dimension of the subspace orthogonal to the background data increases, jeopardizing the stability of the cPCs and enfeebling conclusions drawn from them. \edit{Work by \citet{Severson_2019} provides a probabilistic framework for linear and non-linear contrastive models that attempts to remedy these shortcomings, though it is accompanied by the typical computational and epistemological issues associated with Bayesian methods.}

\subsubsection{Sparse PCA}\label{spca}

In addition to being difficult to interpret, the PCs generated by applying PCA to high-dimensional data are generally unstable, that is, they are
subject to major changes under minor perturbations of the data; we refer to \citet{johnstone2018pca} for a recent review. Luckily, an abundance of techniques for sparsifying PCA loadings have been developed to mitigate these issues; we direct the interested reader to \citet{ZouReview2018} for a recent review. Here, we consider the SPCA technique developed by \citet{Zou2006}. In contrast to standard PCA, SPCA generates interpretable and stable loadings in high dimensions, with most entries of the loading matrix being zero.

SPCA was born from the geometric interpretation of PCA, which reframes PCA as a regression problem. Given a matrix $\mathbf{V}_{p \times k}$ whose columns form an orthonormal basis, the objective is to find the projection $\mathbf{P}_k = \mathbf{V}_{p \times k}\mathbf{V}_{p \times k}^\top$ producing the best linear manifold approximation of the data $\mathbf{X}_{n \times p}$. This is accomplished by minimizing the mean squared error:
\begin{equation}
\mathbf{V}_{p \times k}^{\star} = \argmin_{\mathbf{V}_{p \times k}}\sum_{i=1}^n \lVert x_i-\mathbf{V}_{p \times k} \mathbf{V}_{p \times k}^\top x_i \rVert_2^2,
\end{equation}
where $x_i$ is the $i^{\text{th}}$ row of $\mathbf{X}$ and $\mathbf{V}_{p \times k}^{\star}$ is exactly the loading matrix of the first $k$ PCs \citep{ZouReview2018}. A sparse loading matrix can be obtained by imposing an elastic net constraint on a modification of this objective function.

\citet{Zou2006} show that optimizing the following criterion provides loadings of the first $k$ sparse PCs of $\mathbf{X}$:
\begin{equation}\label{eq:spca}
\begin{aligned}
  (\mathbf{A}^\star_{p \times k}, \mathbf{B}^\star_{p \times k}) & =
    \argmin_{\mathbf{A}_{p \times k}, \mathbf{B}_{p \times k}}
    \sum_{i=1}^n \lVert x_i - \mathbf{A}_{p \times k}\mathbf{B}_{p \times k}^\top x_i \rVert_2^2 + \\
    & \quad \quad \quad \quad \quad \quad \quad \quad \lambda_0 \sum_{j=1}^k\lVert\beta_j\rVert^2_2 +
    \sum_{j=1}^k\lambda_{1,j} \lVert \beta_j \rVert_1 \\
  & \quad \; \text{subject to }\mathbf{A}_{p \times k}^\top\mathbf{A}_{p \times k} = \mathbf{I}_{k \times k},
\end{aligned}
\end{equation}
where $\beta_j$ is the $j^{\text{th}}$ column of $\mathbf{B}_{p \times k}$ and where $\lambda_0$ and $\lambda_{1,j}$ are, respectively, the $\ell_2$ and $\ell_1$ penalty parameters for the non-normalized $j^{\text{th}}$  loading $\beta_j$; as in the original SPCA manuscript the $\ell_1$ penalty is allowed to be loading-specific \citep{Zou2006}. Then, the sparse loadings are the normalized versions of $\beta_j^\star$, i.e., the vectors $\beta_j^\star/\lVert\beta_j^\star\rVert_2$.

\edit{\citet{Zou2006} also show that the full dataset need not be used to optimize the criterion;
indeed, only the Gram matrix $\mathbf{X}_{n \times p}^\top\mathbf{X}_{n \times p}$ is required.
This follows from the first term on the right-hand side of Equation \eqref{eq:spca}, which can be
rewritten as $\lVert \mathbf{X - \mathbf{X}\mathbf{B}\mathbf{A}^\top}\rVert_2^2$. Letting $\mathbf{A}_\perp$
be any orthonormal $p \times (p-k)$ matrix such that the horizontal concatenation $[\mathbf{A}; \mathbf{A}_\perp]_{p \times p}$ is orthonormal, we have, from \citet{Zou2006}'s derivation, that $\lVert \mathbf{X - \mathbf{X}\mathbf{B}\mathbf{A}^\top}\rVert_2^2 = \lVert\mathbf{XA}_\perp\rVert_2^2 + \lVert\mathbf{XA} - \mathbf{XB}\rVert_2^2$. Hence,
obtaining the optimal $\beta_j$ for a fixed $\mathbf{A}$ is equivalent to minimizing
\begin{equation}\label{eq:cov_mat_spca}
  \begin{aligned}
    & \lVert \mathbf{X}\alpha_j - \mathbf{X}\beta_j\rVert_2^2 + \lambda_0\lVert\beta_j\rVert^2_2 + \lambda_{1, j}\lVert\beta_j\rVert_1 \\
    & = (\alpha_j-\beta_j)^\top\mathbf{X}^\top\mathbf{X}(\alpha_j-\beta_j) + \lambda_0\lVert\beta_j\rVert^2_2 + \lambda_{1, j}\lVert\beta_j\rVert_1,
  \end{aligned}
\end{equation}
where $\alpha_j$ is the $j^{\text{th}}$ column of $\mathbf{A}$.}

Although SPCA provides a transparent and efficient method for the sparsification of PCA's loading matrices and hence the generation of stable principal components in high dimensions, its development stopped short of providing means by which to identify the most relevant directions of variation in the data, presenting an obstacle to its efficacious use in biological data exploration and analysis. This motivates the development of exploratory methods that build upon the strengths of both SPCA and cPCA.

\subsubsection{Other Competing Methods}\label{others}

Other general methods frequently employed to reduce the dimensionality of high-dimensional biological data include t-distributed stochastic neighbor embedding (t-SNE) \citep{Maaten08visualizingdata} and uniform manifold approximation and projection (UMAP) \citep{McInnes2018} (e.g., \citep{Amir2013, Becht2019}). Unlike PCA, SPCA, and cPCA, both are nonlinear dimensionality reduction techniques, that is, they do not enforce linear relationships between features. Such a relaxation permits the capturing of local nonlinear structures in the data that would otherwise go unnoticed, though neither approach guarantees that their low-dimensional embeddings reflect the global structure of the data. \citet{Becht2019} demonstrated the computational efficiency exhibited by these techniques in their application to large datasets, while \citet{Amir2013} and \citet{Becht2019} illustrated the stability of their findings, further increasing their popularity as methods of choice for EDA in computational biology. Yet, the flexibility and speed of t-SNE and UMAP come at a cost: \edit{(1)} these techniques are not endowed with the ease of interpretability of factor analysis methods\edit{, (2) the results are sensitive to hyperparameter values (e.g., perplexity for t-SNE and number of nearest neighbors for UMAP) and initializations, (3) identifying reasonable hyperparameters and initializations can prove difficult \citep{Kobak2019}, and (4) the results are inherently stochastic}. In lacking an interpretable link between the data's features and low-dimensional representation, their use as hypothesis-generating tools is restricted. Furthermore, like PCA, neither t-SNE nor UMAP have the ability to explicitly remove unwanted technical effects.

Though the dimensionality reduction methods discussed thus far can be applied to a variety of high-dimensional biological data, still many others have been developed expressly for use with specific high-throughput assay biotechnologies. One such method, ZINB-WaVE, relies on a zero-inflated negative binomial model to better account for the count nature, zero inflation, and over-dispersion of scRNA-seq data, and has been shown to outperform less tailored techniques such as t-SNE \citep{risso2017zinb}. Unlike more general factor analysis methods (e.g., PCA), ZINB-WaVE takes advantage of the rich annotation metadata that are often available with scRNA-seq datasets to remove sources of unwanted variation, while preserving global biological signal \citep{risso2017zinb}. Analogous to PCA, the latent factors produced by ZINB-WaVE are not sparse. Technical and biological noise may remain after taking into account known and unknown sources of unwanted variation \citep{risso2017zinb}, potentially blurring any meaningful interpretation of latent factors. Other methods for reducing the dimensionality of scRNA-seq data, such as scVI \citep{Lopez2018} and \edit{SIMLR \citep{Wang2017} attempt to identify non-linear latent variables using, respectively, a variational auto-encoder framework and a similarity-learning framework relying on multi-kernel learning}. Further discussion of such techniques lies outside the scope of the present work on account of the dissimilarity to methods inspired by factor analysis, like SPCA, cPCA, and ZINB-WaVE, which are our focus.


\documentclass[12pt, a4paper]{article}
% \usepackage{ctex}
\usepackage[margin=1in]{geometry}
\usepackage{amsmath,amsthm,amssymb}
\usepackage{bm}
\usepackage{cases}
\usepackage{graphicx}
\usepackage{hyperref}
\hypersetup{hidelinks}
\usepackage{amsfonts}
\usepackage{authblk}
\title{PRML Note\\C01 Introduction}
\author{Yang Zhao}
\affil{Department of Automation, Tsinghua University}
\date{}

\begin{document}
    \maketitle
    \begin{itemize}
        \item Generalization is a central goal in PR
        \item The Original input variables are typically
        preprocessed. The Test data must be preprocessed
        using the same steps as training data.
        \item  $\text{feature extraction}
        \begin{cases}
        &\text{PR problem easy to solve}\\
        &\text{speed up computation}
        \end{cases}$
        \item  $\text{supervised learning}
        \begin{cases}
        &\text{classification}\\
        &\text{regression}
        \end{cases}$\\
        $\text{unsupervised learning}
        \begin{cases}
        &\text{clustring}\\
        &\text{density estimation}\\
        &\text{dimensions reduction}
        \end{cases}$
    \end{itemize}
    \section{Example: Polynomial Curve Fitting}
    \begin{itemize}
        \item Linear Model:
        \begin{equation}\label{equ:square}
            y(x,\bm{w})=w_0+w_1x+w_2x^2+\cdots+w_Mx^M=\sum_{j=0}^Mw_jx^j
        \end{equation}
        $\bm{w}$ determined by
        \[
            argmin_{\bm{w}^*}E(\bm{w})=\frac{1}{2}\sum_{n=1}^N
            \{y(x_n,\bm{w})-t_n\}^2
        \]
        which is called error funtion\\
        $M$: model comparison or model selection\\
        \[
            E_{RMS}=\sqrt{\frac{2E(\bm{w}^*)}{N}}
            \]
        in which the division by $N$ allows us to compare different
        size of data sets.
        \begin{figure}[ht]
            \begin{center}
                \includegraphics[width=3in]{figures/Figure1_5.eps}
                \caption{Graphs of the RMS error evalutated on the training
                set and on an independent test set for various values of M}
            \end{center}
        \end{figure}
        \item Large value of M $\rightarrow$ Polynomials
        flexible $\rightarrow$ increasing tuned to the random
        noises on target values
        \item The larger the data set, the more complex the model that
        we can afford to fit
        \item By adopting a Bayesian approach, over-fitting can be avoided.\\
        In a Bayesian model, the effective number of parameters adapts
        automatically to the size of the data set
        \item regularization:
        \[
            \widetilde{E}(\bm{w})=\frac{1}{2}\sum_{n=1}^N\{y(x_n,\bm(w))
            -t_n\}^2+\frac{\lambda}{2}\Vert\bm{w}\Vert^2
            \]
        \begin{enumerate}
            \item shrinkage in statistics
            \item ridge regression
            \item weight decay in NN
        \end{enumerate}
        \begin{figure}[ht]
            \begin{center}
                \includegraphics[width=3in]{figures/Figure1_8.eps}
                \caption{Graphs of the RMS error evalutated on the training
                set and on an independent test set for $ln\lambda$}
            \end{center}
        \end{figure}
        \item $\Vert\bm{w}\Vert^2=\bm{w}^T\bm{w}=w_0^2+w_1^2+\cdots+w_M^2$
        \begin{enumerate}
            \item $w_0^2$ is often omitted. if not, the result will depend
            on the choice of the origin for the target variable
            \item $\frac{\lambda_1}{2}w_0^2+\frac{\lambda_2}{2}(w_1^2+
            \cdots+w_M^2)$ is also OK
        \end{enumerate}
        \item $\text{data set}\begin{cases}
            &\text{training set}\rightarrow\bm{w}\\
            &\text{validation set(hold-out set)}\\
            &\rightarrow M,\lambda
            \text{ which is too wasteful of data set}\\
            &\text{test set}
            \end{cases}$
    \end{itemize}
    \section{Probability Theory}
    \begin{itemize}
        \item the rules of Probability
        \begin{enumerate}
            \item sum rule: $P(x)=\sum_YP(x,Y)$
            \item product rule: $P(x,Y)=P(Y|x)P(x)$\\
            in which $P(x)$ is called marginal probability and $P(Y|x)$ is
            called conditional probability.
        \end{enumerate}
        \item Bayes' Theorem:
        \[
            P(Y|X)=\frac{P(X|Y)P(Y)}{P(X)}=\frac{P(X|Y)P(Y)}{\sum_Y{P(X|Y)P(Y)}}
            \]
        \item a nonlinear change of variable $x=g(y)$\\
        consider a probability density $p_x(x)$ that correspond to a density $p_y(y)$.
        Observations falling in the range $(x,x+\delta x)$ will be transformed into the
        range $(y, y+\delta y)$ where $p_x(x)\simeq p_y(y)$
        \[
            p_y(y)=p_x(x)\Big\vert\frac{dx}{dy}\Big\vert=p_x(g(y))\vert g'(y)\vert
            \]
        So, the concept of the max of a probability density is dependent on the choice
        of variable.
        \item Expectations: $\mathbb{E}[f]=\sum_xp(x)f(x)\simeq \frac{1}{N}\sum_{n=1}
        ^{N}f(x_n)$\\
        Conditional Expectations: $\mathbb{E}[f|y]=\sum_xp(x|y)f(x)$
        \item Variances: \[Var[f]=\mathbb{E}\{(f(x)-\mathbb{E}[f(x)])^2\}=
        \mathbb{E}\{f^2(x)\}-\mathbb{E}^2\{f(x)\}\]
        Corvariances: \[cov(x,y)=\mathbb{E}_{x,y}\lbrack\{x-\mathbb{E}\lbrack x\rbrack\}
        \{y-\mathbb{E}\lbrack y\rbrack\}\rbrack=\mathbb{E}_{xy}\lbrace xy\rbrace-
        \mathbb{E}\lbrace x\rbrace\mathbb{E}\lbrace y\rbrace\]
        \item a prior probability distribution $p(\bm{w})$\\
        Observed data $D=\{t_1,t_2,\cdots,t_N\}$ is expressed through the conditional
        probability $p(D|\bm{w})$
        \[
            p(\bm{w}|D)=\frac{p(D|\bm{w})P(\bm{w})}{p(D)}
            \]
        in which $p(D|\bm{w})$ is called likelihood function which expresses how
        probable the observed data set is for different settings of the parameters $\bm{w}$
        and $p(D)$ is called normalization constant which is
        $p(D)=\int p(D|\bm{w})p(\bm{w})d\bm{w}$
        \item A data set of observations $X=(x_1,x_2,\cdots,x_N)^T$,i,i,d, the likelihood
        function is
        \[
            p(X|\mu,\sigma^2)=\prod_{n=1}^N\mathcal{N}(x_n|\mu,\sigma^2)
            \]
        and the log likelihood function can be written in the form
        \[
            lnp(x|\mu,\sigma^2)=-\frac{1}{2\sigma^2}\sum_{n=1}^N(x_n-\mu)^2-\frac{N}{2}
            ln(\sigma^2)-\frac{N}{2}ln(2\pi)
            \]
        \item find the $\mu$ and $\sigma^2$ to max this likelihood function and the maximum
        likelihood solution is\\
        $ \begin{cases}
        &\mu_{ML}=\frac{1}{N}\sum_{n=1}^Nx_n\\
        &\sigma_{ML}^2=\frac{1}{N}\sum_{n=1}^{N}(x_n-\mu_{ML})^2
        \end{cases}$\\
        and the expectation of it is\\
        $\begin{cases}
        &\mathbb{E}\lbrace \mu_{ML}\rbrace=\mu\\
        &\mathbb{E}\lbrace \sigma_{ML}^2\rbrace=(\frac{N-1}{N})\sigma^2
        \end{cases}$\\
        so
        \[\widetilde{\sigma}^2=\frac{N}{N-1}\sigma_{ML}^2=\frac{1}{N-1}
        \sum_{n=1}^N(x_n-\mu_{ML})^2\]
        \item bootstrap: suppose data set consists of $N$ data points $\bm{X}=\{\bm{x}_1,
        \cdots,\bm{x}_N\}$, we can drawing $N$ data points at random from $\bm{X}$ to create
        a new data set $\bm{X}_B$. This process can be repeated $L$ times
        \item maximum posterior(MAP): suppose training data comprising $N$ input values
        $\bm{x}=(x_1,x_2,\cdots,x_N)^T$ and their corresponding target values $\bm{t}=(t_1,
        t_2,\cdots,t_N)^T$. We have
        \begin{equation}
            p(t|x,\bm{w},\beta)=\mathcal{N}(t|y(x,\bm{w}),\beta^{-1})
        \end{equation}
        where $y(x,\bm{w})$ is given by equation (\ref{equ:square}). So the likelihood function is
        given by
        \[
            p(\bm{t}|\bm{x},\bm{w},\beta)=\prod_{n=1}^N\mathcal{N}(t_n|y(x_n,\bm{w}),\beta^{-1})
            \]
        and the log likelihood function is
        \[
            lnp(\bm{t}|\bm{x},\bm{w},\beta)=-\frac{\beta}{2}\sum_{n=1}^N\{y(x_n,\bm{w})-t_n\}^2
            +\frac{N}{2}ln\beta-\frac{N}{2}ln2\pi
            \]
        If we consider a prior distribution over $\bm{w}$,
        \[
            p(\bm{w}|\alpha)=\mathcal{N}(\bm{w}|\bm{0},\alpha^{-1}\bm{I})
            \]
        then the posterior distribution for $\bm{w}$ is
        \[
            p(\bm{w}|\bm{x},\bm{t},\alpha,\beta)\propto p(\bm{t}|\bm{x},\bm{w},\beta)p(\bm{w}|\alpha)
            \]
        and we should maximize the log posterior function
        \[
            \frac{\beta}{2}\sum_{n=1}^N\{y(x_n,\bm{w})-t_n\}^2+\frac{\alpha}{2}\bm{w}^T\bm{w}
            \]
        \item Bayesian curve fitting: Given training data $\bm{x}$ and $\bm{t}$, along with a new
        point $x$, we want to predict the value of $t$,
        \[
            p(t|x,\bm{x},\bm{t})=\int p(t|x,\bm{w})p(\bm{w}|\bm{x},\bm{t})d\bm{w}
            \]
    \end{itemize}
    \section{Model Selection}
    \begin{itemize}
        \item data is plentiful: train a range of models or a model with a range of values for its
        complexity params.
        \item  data is limited: cross-validation
        \item An ideal approach:
        \begin{enumerate}
            \item only rely on the training data
            \item allow multiple hyperparameters and model types to be compared in a single
            training run
        \end{enumerate}
        therefore, find a measure of performance which depends only on the training data, not
        suffer from bias to over-fitting.
    \end{itemize}
    \section{The Curse of Dimensionality}
    \begin{itemize}
        \item Not all intuitions developed in spaces of low dimensionality will generalize to
        spaces of many dimensions
        \item real data
        \begin{enumerate}
            \item often be confined in lower effective dimensionality
            \item exhibit smooth properties
        \end{enumerate}
    \end{itemize}
    \section{Decision Theory}
    \begin{itemize}
        \item For 2-classes, minimize
        \begin{align*}
            p(mistake)&=p(x\in R_1,C_2)+p(x\in R_2, C_1)\\
            &=\int_{R_1}p(x,C_2)dx+\int_{R_2}p(x,C_1)dx
        \end{align*}
        if $p(x,C_1)>p(x,C_2)$, which is the same as $p(C_1|x)p(x)>p(C_2|x)p(x)$ i.e.
        $p(C_1|x)>p(C_2|x)$, $x\rightarrow C_1$
        \begin{figure}[ht]
            \begin{center}
                \includegraphics[width=0.6\textwidth]{figures/Figure1_24.eps}
                \caption{The joint probability of two classes}
            \end{center}
        \end{figure}
        \item For $k$-classes, maximize
        \begin{align*}
            p(correct)&=\sum_{k=1}^K\int_{R_k}p(x,C_k)dx\\
            &=\sum_{k=1}^K\int_{R_k}p(C_k|x)p(x)dx
        \end{align*}
        find the max $p(C_k|x)$ and $x\rightarrow C_k$
        \item reject option: if $p(C_k|x)<\theta$, do not classify $x$
        \begin{enumerate}
            \item $\theta=1$, reject all samples
            \item For $k$-classes, $\theta<\frac{1}{k}$, No sample is rejected
        \end{enumerate}
        \item Generative model: find the joint distribution $p(C_k,x)$ and then find
        the posterior distribution $p(C_k|x)$ to make decision using decision theory.
        This can be used to detect new data point with low probability, which is known
        as outlier detection or novelty detection.
        \item Discriminative model: find the posterior distribution directly, and use
        it to make decision.
        \item Discriminant function: only a function, the input is $x$ and the output is
        the class label.
        \item Compensating for class prior: In training data $n(C_1)=a$ and $n(C_2)=b$, if
        $a\gg b$ ,it is not likely to generalize well. From the training data we can get a
        balanced data set. From the balanced data we can get the posterior $p(C_1|x)$ and
        $p(C_2|x)$. And the goal data set we want to apply to is $n(C_1)=a'$ and $n(C_2)=b'$,
        so the posterior we use is \[\widetilde{p}(C_1|x)=\frac{p(C_1|x)}{\frac{a}{a+b}}
        \frac{a'}{a'+b'}\] and \[\widetilde{p}(C_2|x)=\frac{p(C_2|x)}{\frac{b}{a+b}}
        \frac{b'}{a'+b'}\] then normalize to ensure
         \[\widetilde{p}(C_1|x)+\widetilde{p}(C_2|x)=1\]
         Explaination:
         \[\frac{p(C_1|x)}{\frac{a}{a+b}}\propto \text{likelihood function}=p(x|C_1)\]
         \item Combining models: $p(\bm{x_I},\bm{x_B}|C_k)=p(\bm{x_I}|C_k)p(\bm{x_B}|C_k)$,
        so we can get
        \begin{align*}
            p(C_k|\bm{x_I},\bm{x_B})&\propto p(\bm{x_I},\bm{x_B}|C_k)p(C_k)\\
            &\propto p(\bm{x_I}|C_k)p(\bm{x_B}|C_k)p(C_k)\\
            &\propto \frac{p(C_k|\bm{x_I})p(C_k|\bm{x_B})}{p(C_k)}
        \end{align*}
        \item Calculus of Variations: if $J(y)=\int_{x_1}^{x_2}\mathcal{L}(x,y(x),y'(x))dx$,
        we have
        \begin{equation}
            \label{CalculusVariations}
            \frac{\partial J(y)}{\partial y}=\frac{\partial\mathcal{L}}{\partial f}-\frac{d}{dx}
            \frac{\partial\mathcal{L}}{\partial f'}
        \end{equation}
        which is called Euler-Lagrange Equation.
        \item Incuring a Loss function $\mathcal{L}(t,y(\bm{x}))$, the average, or expected,
        loss is given by
        \begin{equation}
            \label{equ:loss:expectation}
            \mathbb{E}\lbrack L\rbrack=\iint\mathcal{L}(t,y(\bm{x}))p(\bm{x},t)d\bm{x}dt
        \end{equation}
        A common choice of loss funtion is $\mathcal{L}(t,y(\bm{x}))=\{y(\bm{x}-t)\}^2$,
        using the Calculus of Variations
        \[
            \frac{\partial\mathbb{E}\lbrack L\rbrack}{\partial y(\bm{x})}=
            2\int\{y(\bm{x})-t\}p(\bm{x},t)dt=0
            \]
        So we have
        \[
            y(\bm{x})=\frac{\int tp(\bm{x},t)dt}{p(\bm{x})}\int tp(t|\bm{x})dt=
            \mathbb{E}_t\lbrack t|\bm{x}\rbrack
            \]
        \item from equation (\ref{equ:loss:expectation}), we can rewrite to
        \begin{align}
            \mathbb{E}\lbrack L\rbrack&=\iint\{y(\bm{x}-\mathbb{E}\lbrack t|\bm{x}\rbrack
            +\mathbb{E}\lbrack t|\bm{x}\rbrack-t)\}^2p(\bm{x},t)d\bm{x}dt\nonumber\\
            \label{equ:loss:expectation2}
            &=\int\{y(\bm{x})-\mathbb{E}\lbrack t|\bm{x}\rbrack\}^2p(\bm{x})d\bm{x}+
            \int Var\lbrack t|\bm{x}\rbrack p(\bm{x})d\bm{x}
        \end{align}
        The process to get the equation (\ref{equ:loss:expectation2}) is below:
        \begin{enumerate}
            \item $\{y(\bm{x}-t)\}^2=\{y(\bm{x})-\mathbb{E}\lbrack t|\bm{x}\rbrack\}^2
            +2\{y(\bm{x})-\mathbb{E}\lbrack t|\bm{x}\rbrack\}\{\mathbb{E}\lbrack t|\bm{x}
            \rbrack-t\}+\{\mathbb{E}\lbrack t|\bm{x}\rbrack-t\}^2$
            \item $p(\bm{x},t)=p(t|\bm{x})p(\bm{x})$
            \item \begin{align*}
            \mathbb{E}\lbrack L\rbrack_1&=\iint\{y(\bm{x})-\mathbb{E}\lbrack t|\bm{x}
            \rbrack\}^2p(\bm{x},t)d\bm{x}dt\\
            &=\iint\{y(\bm{x})-\mathbb{E}\lbrack t|\bm{x}\rbrack\}^2p(t|\bm{x})p(\bm{x})d\bm{x}dt\\
            &=\int\{y(\bm{x})-\mathbb{E}\lbrack t|\bm{x}\rbrack\}^2p(\bm{x})
            (\int p(t|\bm{x})dt)d\bm{x}\\
            &=\int\{y(\bm{x})-\mathbb{E}\lbrack t|\bm{x}\rbrack\}^2p(\bm{x})d\bm{x}
            \end{align*}
            \item \begin{align*}
                \mathbb{E}\lbrack L\rbrack_2&=\iint\{y(\bm{x})-\mathbb{E}\lbrack t|\bm{x}\rbrack\}
                \{\mathbb{E}\lbrack t|\bm{x}\rbrack-t\}p(t|\bm{x})p(\bm{x})d\bm{x}dt\\
                &=\int\{y(\bm{x})-\mathbb{E}\lbrack t|\bm{x}\rbrack\}p(\bm{x})
                (\int\{\mathbb{E}\lbrack t|\bm{x}\rbrack-t\}p(t|\bm{x})dt)d\bm{x}\\
                &=\int\{y(\bm{x})-\mathbb{E}\lbrack t|\bm{x}\rbrack\}p(\bm{x})
                (\mathbb{E}\lbrack t|\bm{x}\rbrack-\int tp(t|\bm{x})dt)d\bm{x}\\
                &=\int\{y(\bm{x})-\mathbb{E}\lbrack t|\bm{x}\rbrack\}p(\bm{x})
                (\mathbb{E}\lbrack t|\bm{x}\rbrack-\mathbb{E}\lbrack t|\bm{x}\rbrack)d\bm{x}\\
                &=0
            \end{align*}
            \item \begin{align*}
                \mathbb{E}\lbrack L\rbrack_3&=\iint\{\mathbb{E}\lbrack t|\bm{x}\rbrack-t\}^2
                p(t|\bm{x})p(\bm{x})d\bm{x}dt\\
                &=\iint\{\mathbb{E}^2\lbrack t|\bm{x}\rbrack-2t\mathbb{E}\lbrack t|\bm{x}
                \rbrack+t^2\}p(t|\bm{x})p(\bm{x})d\bm{x}dt\\
                &=\int\{-\mathbb{E}^2\lbrack t|\bm{x}\rbrack+\mathbb{E}\lbrack t^2|\bm{x}\rbrack\}
                p(\bm{x})d\bm{x}\\
                &=\int Var\lbrack t|\bm{x}\rbrack p(\bm{x})d\bm{x}
            \end{align*}
            \item $\mathbb{E}\lbrack L\rbrack=\mathbb{E}\lbrack L\rbrack_1+\mathbb{E}\lbrack
            L\rbrack_2+\mathbb{E}\lbrack L\rbrack_3$
        \end{enumerate}
        \item Squared loss can lead to very poor results.
        \item The Minkowski loss $\vert y(\bm{x})-t\vert^q$
    \end{itemize}
    \section{Information Theory}
    \begin{itemize}
        \item $h(x)=-log_2p(x)$
        \item the entropy of the random variable $x$(bit): $H\lbrack x\rbrack=-\sum_xp(x)log_2p(x)$
        and the entropy of the random variable $x$(nat): $H\lbrack x\rbrack=-\frac{1}{ln2}\sum_xp(x)lnp(x)$
        \item if $p(x)=0$, we have $p(x)log_2p(x)=0$
        \item The conditional entropy of $y$ given $x$:
        \[
            H\lbrack y|x\rbrack=-\iint p(y,x)lnp(y|x)dydx
        \]
        and $H\lbrack x,y\rbrack=H\lbrack y|x\rbrack+H\lbrack x\rbrack$
        \item relative entropy or KL divergence between the distribution $p(x)$ and $q(x)$
        \begin{align}
            KL(p\Vert q)&=-\int p(x)lnq(x)dx-(-\int p(x)lnp(x)dx)\nonumber\\
            &=-\int p(x)ln\Big\{\frac{q(x)}{p(x)}\Big\}dx
        \end{align}
        Note that $KL(p\Vert q)\neq KL(q\Vert p)$ and $KL(p\Vert q)\geq 0$ with equality, if and only if
        $p(x)=q(x)$
        \item the mutual information between the variables $x$ and $y$
        \begin{align}
            I\lbrack x,y\rbrack&=KL(p(x,y)\Vert p(x)p(y))\nonumber\\
            &=-\iint p(x,y)ln\Big\{\frac{p(x)p(y)}{p(x,y)}\Big\}dxdy
        \end{align}
        Note that $I\lbrack x,y\rbrack\geq 0$ with equality, if and only if $x$ and $y$ are independent
        and we have $I\lbrack x,y\rbrack=H\lbrack x\rbrack-H\lbrack x|y\rbrack
        =\lbrack y\rbrack-H\lbrack y|x\rbrack$
    \end{itemize}
    \section{Appendix}
    \begin{itemize}
        \item To get the Euler-Lagrange equation\ref{CalculusVariations}, consider the function
        \begin{equation*}
            J(y)=\int_{x_1}^{x_2}L(x,y(x),y'(x))dx
        \end{equation*}
        where
        $x_1,x_2$ are constants, $y(x)$ is twice continuously differentiable, $y'(x)=dy/dx$,
        $L(x,y(x),y'(x))$ is twice continuously differentiable with respect to its arguments $x,y,y'$.
        If the funtion $J(y)$ attains a local minimum at $f$, and $\eta$ is an arbitrary function
        that has at least one derivative and vanishes at the endpoints $x_1$ and $x_2$, then for any
        number $\epsilon$ close to 0,
        \begin{equation*}
            J(f)\le J(f+\epsilon\eta)
        \end{equation*}
        Let $\Phi(\epsilon)=J(f+\epsilon\eta)$, and the function $\Phi(\epsilon)$ has a minimum at
        $\epsilon=0$, thus
        \begin{equation*}
            \Phi'(0)=\frac{d\Phi}{d\epsilon}\Big\vert_{\epsilon=0}=\int_{x_1}^{x_2}\frac{dL}
            {d\epsilon}\Big\vert_{\epsilon=0}dx=0
        \end{equation*}
        Using the total derivative of $L(x,y,y')$
        \begin{equation*}
            \frac{dL}{d\epsilon}=\frac{\partial L}{\partial y}\frac{dy}{d\epsilon}+
            \frac{\partial L}{\partial y'}\frac{dy'}{d\epsilon}=\frac{\partial L}{\partial y}\eta
            +\frac{\partial L}{\partial y'}\eta'
        \end{equation*}
        So,
        \begin{equation*}
            \frac{dL}{d\epsilon}\Big\vert_{\epsilon=0}=\frac{\partial L}{\partial f}\eta
            +\frac{\partial L}{\partial f'}\eta'
        \end{equation*}
        $\eta=0$ at $x_1$ and $x_2$ by definition, therefore,
        \begin{align*}
            \int_{x_1}^{x_2}\frac{dL}{d\epsilon}\Big\vert_{\epsilon=0}dx=&\int_{x_1}^{x_2}\Big
            (\frac{\partial L}{\partial f}\eta+\frac{\partial L}{\partial f'}\eta'\Big)dx\\
            =&\int_{x_1}^{x_2}\Big(\frac{\partial L}{\partial f}\eta+\frac{d}{dx}\Big(\frac{\partial L}
            {\partial f'}\eta\Big)-\eta\frac{d}{dx}\frac{\partial L}{\partial f'}\Big)dx\\
            =&\int_{x_1}^{x_2}\Big(\frac{\partial L}{\partial f}\eta-\eta\frac{d}{dx}\frac{\partial L}
            {\partial f'}\Big)dx+\frac{\partial L}{\partial f'}\eta\Big\vert_{x_1}^{x_2}\\
            =&\int_{x_1}^{x_2}\eta\Big(\frac{\partial L}{\partial f}-\frac{d}{dx}\frac{\partial L}
            {\partial f'}\Big)dx=0
        \end{align*}
        For $\eta$ is the arbitrary function, so we have
        \begin{equation}
            \frac{\partial L}{\partial f}-\frac{d}{dx}\frac{\partial L}{\partial f'}=0
        \end{equation}
    \end{itemize}
\end{document}


The code for this simulation is as shown below. For easier reading only the code for the three-body simulation has been included as it contains code for the previous 2 simulations;

\section{Code}
\lstinputlisting[language=Python]{sim3d-animated.py}