\newcommand{\norm}[1]{\left\lVert#1\right\rVert}
\newcommand{\KL}[2]{D_{\mathrm{KL}} \bigl( #1 ~||~ #2 \bigr)}
\newcommand{\trans}{\mathbf{T}}
\newcommand{\qex}{Q_{\text{explore}}}
\newcommand{\qtask}{Q_{\text{task}}}
\newcommand{\tex}{\tau_{\text{explore}}}
\newcommand{\ttask}{\tau_{\text{task}}}
\newcommand{\pitask}{\pi_{\text{task}}}
\newcommand{\piexplore}{\pi_{\text{explore}}}
\newcommand{\algname}{Decoupled Exploration and Exploitation Policies} % name TBD
\newcommand{\algshort}{DEEP} % name TBD

\section{Introduction}
Recent progress in reinforcement learning (RL) for continuous control has led to significant improvements in sample complexity and performance.
While earlier on-policy algorithms required hundreds of millions of environment steps to learn, recent off-policy algorithms have brought the sample complexity of model-free RL in range of solving tasks on real robots \citep{haarnoja2018softapp}.

In parallel, a rich literature has been developed for directed exploration in deep reinforcement learning, inspired in part by the theoretical impact of exploration on sample complexity.
% \citep{Stadie2015IncentivizingEI,Osband2016DeepEV,houthooft2016vime,pathak2017curiosity,Tang2017Exploration,burda2018exploration,Fortunato2018NoisyNF,Plappert2018ParameterSN,osband2019deep,Badia2020NeverGU,Machado2020CountBasedEW,Rashid2020OptimisticEE,Dean2020SeeHE}.
The bulk of these methods fall into the family of bonus-based exploration (BBE) methods, in which a policy receives a bonus for visiting states deemed to be interesting or novel.
BBE algorithms have enabled RL to solve a variety of long-horizon, sparse-reward tasks, most notably the game Montezuma's Revenge from the Arcade Learning Environment (ALE) \citep{Bellemare2015TheAL}.

These two subfields both aim to minimize the sample complexity of model-free RL, and their methods are in principle perfectly complementary.
Off-policy algorithms extract improved policies from data collected by (notionally) arbitrary behavior, and their performance is limited only by the coverage of the data;
meanwhile exploration generates data with improved coverage.
% -- whose performance is limited only by the coverage of the data --
% from data collected by (notionally) arbitrary behavior, while exploration methods generate data with improved coverage.
% Off-policy algorithms extract improved policies from (notionally) arbitrary data, with performance in principle limited only by the coverage of the data, while exploration methods generate data with improved coverage.
However, to date the impact of directed exploration techniques on sample-efficient control has been minimal, with state of the art algorithms using undirected exploration such as maximum-entropy objectives.
In this paper we investigate the missing synergy between off-policy continuous control and directed exploration.

We find that BBE is poorly suited to the few-sample regime due to slowly-decaying bias in the learned policy and slow adaptation to the non-stationary exploration bonus.
Bias due to optimizing a reward function other than the task reward leads a policy trained with BBE to exhibit poor performance until the bonus decays toward zero.
Meanwhile, the non-stationary (continually decreasing) exploration bonus cannot necessarily be optimized by a fixed policy, violating one of the core assumptions of RL.
This leads to slow exploration as the policy adapts only gradually, especially in the off-policy case where replay buffers will contain stale rewards.
These observations underline research by \citet{Taiga2020On} showing that across the ALE, no BBE algorithm outperforms undirected $\varepsilon$-greedy exploration.

We demonstrate that bias and slow coverage are the culprits of BBE's lackluster performance by proposing a new exploration algorithm, \algname{} (\algshort{}), which addresses these limitations.
\algshort{} decouples the learning of a \emph{task policy}, which is trained to maximize the true task reward, and an \emph{exploration policy}, which maximizes only the exploration bonus.
Both policies are trained off-policy using data collected according to the product of the two policy distributions.
Unlike the policy learned by BBE, \algshort{}'s task policy is always unbiased in the sense that it reflects the current belief about the optimal action in each state.
Furthermore, this decoupling allows \algshort{} to aggressively update the exploration policy without affecting the convergence of the task policy, thereby adapting more rapidly to the changing exploration bonus.

We perform experiments using policies based on Q-learning \citep{sutton2018reinforcement,mnih2015human} on toy tasks and soft actor-critic (SAC) \citep{haarnoja2018softapp} on larger-scale tasks from the DeepMind Control Suite \citep{tassa2018deepmind}.
Our results show that on tasks with dense rewards and uniform resets, BBE often performs worse than the underlying policy-learning algorithm while \algshort{} incurs no cost for exploring.
On tasks with more natural resets and sparse rewards, \algshort{} covers the state space more rapidly than BBE and reaches peak performance in a fraction of the samples required with undirected exploration.
In total, \algshort{} strictly outperforms undirected exploration while solving many sparse environments just as fast as dense ones.

\section{Background}

\subsection{Notation}

A Markov decision process (MDP) $\mdp$ consists of a tuple $(\mathcal{S}, \mathcal{A}, P, R, \gamma)$, where $\mathcal{S}$ is the state space, $\mathcal{A}$ is the action space, $P$ is the transition function mapping $\mathcal{S} \times \mathcal{A}$ to distributions on $\mathcal{S}$, $R$ is the scalar reward function on $\mathcal{S} \times \mathcal{A}$, and $\gamma$ is the discount factor.
We use lower-case $(s, a, r)$ to refer to concrete realizations of states, actions, and rewards.
We use $\mdp_f$ to denote the MDP $\mdp$ with the original reward function $R$ replaced by another function $f$.
For convenience we assume exploration rewards are within $[0, 1]$, and we define $\bar{r} = \nicefrac{1}{1 - \gamma}$, which is the maximum discounted value possible.

\subsection{Bonus-based exploration: a recipe for exploration in deep RL} \label{sec:intrinsic_rewards}

Bonus-based exploration has emerged as the standard framework for exploration in the deep reinforcement learning community.
In this framework, an agent learns in a sequence of MDPs $\widetilde{\mdp} = \{\mdp_{\widetilde{R}_n} \}_{n=1}^N$ where the reward function $\widetilde{R}_n$ changes as a function of each transition.
A typical choice is $\widetilde{R}_n = R + R^+_n$, where $R^+_n$ is an exploration bonus which measures the ``novelty'' of a transition $(s, a, s')$ given the history of all transitions up to $n$.
After taking each transition $(s, a, s')$, the reward $\widetilde{r} = \widetilde{R}_n(s, a, s')$ is calculated and the tuple $(s, a, s', \widetilde{r})$ is added to a replay dataset $D$.
The agent optimizes its reward in this (non-stationary) MDP $\widetilde{\mdp}$ via some model-free RL algorithm operating on the replay dataset.
The realization of a particular algorithm in this family amounts to defining a novelty function and picking a model-free RL algorithm \citep{Stadie2015IncentivizingEI,houthooft2016vime,bellemare2016unifying,pathak2017curiosity,Tang2017Exploration,burda2018exploration,Machado2020CountBasedEW}.
We illustrate this recipe in \cref{alg:bbe}.

\paragraph{Pseudo-counts.}
Building upon theoretically-motivated exploration methods for discrete environments \citep{strehl2008analysis}, \citet{bellemare2016unifying} proposed to give exploration bonuses based on a \emph{pseudo-count} function $\hat N$.
A pseudo-count has two key properties.
Like a true count, a pseudo-count increases by 1 each time a state (or state-action pair) is visited.
Unlike a true count, a pseudo-count generalizes across states and actions; that is, when a state $s$ is visited, the pseudo-count for nearby states $s + \varepsilon$ may increase as well.

\begin{figure*}
% \begin{minipage}[b]{\textwidth}
\begin{algorithm}[H]
    % \small
    \centering
    \caption{Bonus-based exploration}\label{alg:bbe}
    \begin{algorithmic}[1] % The number tells where the line numbering should start
        \Require{replay dataset $D$, policy $\pi$, bonus $R^+_n$}
        \item[]
        \State $n \gets 0$
        \Repeat
            \For{one episode}
                \item[]
                \item[]
                \State Collect $(s, a, s', r) \sim P(s, \pi(s))$
                \State $\widetilde{r} \gets r + R^+_n(s, a, s')$
                \State $D \gets D \cup (s, a, s', \widetilde{r})$
                \State $R^+_{n+1} \gets \text{Update}(R^+_n, (s, a, s'))$
                \State $n \gets n + 1$
            \EndFor
            \State Train $\pi$ with samples from $D$
        \Until{$n = N$}
    \end{algorithmic}
\end{algorithm}
% \end{minipage}
\hfill

% \begin{minipage}[b]{\textwidth}
\begin{algorithm}[H]
    % \small
    \centering
    \caption{\algname{}}\label{alg:deep}
    \begin{algorithmic}[1]
        \Require{replay dataset $D$, temperature $\tau$,}
        task policy $\pitask$, exploration policy $\piexplore$, bonus $R^+_n$
        % \Require{policy $\pi$, exploration value $\qex$}
        \State $n \gets 0$
        \Repeat
        \For{one episode}
            \State Update $\piexplore$ on $\mdp_{R^+_n}$
                \State Set $\beta(a|s) \propto \pitask(a|s) \cdot \piexplore(a|s)$ %by \cref{eq:behavior_policy}
                \State Collect $(s, a, s', r) \sim P(s, \beta(s))$
                \item[]
                \State $D \gets D \cup (s, a, s', r)$
                \State $R^+_{n+1} \gets \text{Update}(R^+_n, (s, a, s'))$
                \State $n \gets n + 1$
            \EndFor
            \State Train $\pitask$ with samples from $D$
        \Until{$n = N$}
    \end{algorithmic}
\end{algorithm}
% \end{minipage}
\caption{
Comparison of classic bonus-based exploration (BBE) with our method (\algshort{}).
BBE computes exploration bonuses at the time of visiting a transition, adds them to the real rewards, and uses a replay buffer of experience to learn a policy.
\algshort{} separates the exploration policy $\piexplore$ from the task policy $\pitask$, allowing $\pitask$ to be an unbiased estimate of the optimal policy throughout training.
It always uses the \emph{current} exploration reward function $R_n^+$ when updating the exploration value function, and is fast-adapting to deal with the non-stationary bonus MDP.
% \red{(include the reward function as an input and show it updating)}
% Finally \algshort{}'s behavior policy actions and updates are optimistic, ensuring the agent will reach and take transitions it has never seen.
}
\end{figure*}

\section{Limitations of bonus-based exploration}

\begin{figure}[h]
    \centering
    \begin{subfigure}[b]{0.49\textwidth}
        \centering
        \includegraphics[width=0.8\textwidth]{figures/deep/grid40_warmstart.pdf}
        \caption{Reward}
    \end{subfigure}
    \hfill
    \begin{subfigure}[b]{0.49\textwidth}
        \centering
        \includegraphics[width=0.8\textwidth]{figures/deep/grid40_warmstart_visits.pdf}
        \caption{Coverage}
    \end{subfigure}
    \caption{Experiments in a $40 \times 40$ grid-world environment with one goal state, where learning algorithms were warm-started with 20 episodes of data from a skilled policy.
    \textbf{(a)} With enough signal to find the goal, DDQN (Double DQN, \citet{Hasselt2016DeepRL}) alone rapidly converges to the optimal policy. BBE introduces bias, causing the policy to continually explore. Our method, \algshort{}, learns the task policy just as rapidly as DDQN alone.
    \textbf{(b)} Though it performs well, DDQN simply goes to the goal during each train episode and does not explore other options. BBE continues to seek out new states at a slow but steady rate. \algshort{} explores far more than BBE during data collection despite simultaneously performing just as well as DDQN at evaluation time.}
    \label{fig:gridworld_warmstart}
\end{figure}

The bonus-based exploration algorithm, illustrated in \Cref{alg:bbe}, has two weaknesses which limit its usefulness for sample-efficient policy learning.

\paragraph{Bias with finite samples.}
Because they estimate the optimal policy on the modified MDP $\mdp'$, bonus-based exploration algorithms learn biased policies as long as the exploration bonus is nonzero.
According to theory, the exploration bonus should be scaled larger than is done in practice \citep{strehl2008analysis} and decay slower than $\nicefrac{1}{N(s)}$ \citep{Kolter2009NearBayesianEI} in order to guarantee convergence to the optimal policy.
This behavior, shown in \Cref{fig:gridworld_warmstart}, can result in slow convergence to the optimal policy and substantially biased policies after a practically feasible number of samples.

\begin{figure}[h]
    \begin{center}
        \includegraphics[width=0.35\textwidth]{figures/deep/grid40_visits_neurips.pdf}
    \end{center}
    % \caption{A test of pure exploration speed. Undirected exploration with a uniformly random policy covers the state space very slowly. BBE, with its ability to seek unknown states, is faster. However, BBE does not adapt rapidly to the changing rewards, leading it to be slower than the fast adaptation scheme we propose.}
    \caption{Pure exploration}
    \label{fig:gridworld_visits}
\end{figure}

% \caption{Two experiments in a $40 \times 40$ grid-world environment with one goal state. \textbf{(a)} Learning algorithms warm-started with 20 episodes of data from a skilled policy. Since there is plenty of signal to find the goal, DDQN without exploration rapidly converges to the optimal policy. BBE introduces bias, causing the policy to continually explore. Our method, \algshort{}, learns the task policy just as rapidly as DDQN alone.
% \textbf{(b)} A test of pure exploration speed. Undirected exploration with a uniformly random policy covers the state space very slowly. BBE, with its ability to seek unknown states, is faster. However, BBE does not adapt rapidly to the changing rewards, leading it to be slower than the fast adaptation scheme we propose.\textcolor{red}{TOBI: TODO these need to take the count function/bonus reward as input and update them.}}

\paragraph{Slow adaptation to changing rewards.}
Algorithms in this family update the policy according to the schedule of the underlying model-free RL algorithm -- for example at the end of each episode.
This works well for the stationary MDPs that these algorithms were developed for, but the modified MDP $\mdp'$ which represents the exploration problem is non-stationary.
This leads to an agent which determines the most novel state and then stays there for an entire episode.
This degenerate behavior leads to potentially exploring only a single state per episode instead of visiting a sequence of new states as the reward function evolves.\endnote{Some implementations of bonus-based exploration may update the policy within an episode, for example via a single gradient step per environment step on transitions sampled i.i.d.~from a replay.
However, such a small update is typically not enough to change the qualitative behavior of the agent and adapting to the changing MDP has not been an emphasis in prior work.}
The use of replay buffers compounds this effect, since algorithms in this family compute exploration rewards at the time the transition is collected, rather than when it is used.
An algorithm which is unaware of the non-stationary nature of the MDP will maximize the return on this mixture of reward functions rather than the reward that incorporates the current bonus, resulting in slow coverage of the environment.
\Cref{fig:gridworld_visits} shows uniform random actions, BBE, and BBE with the fast adaptation scheme we propose in \Cref{sec:fast_adaptation} all exploring in a $40 \times 40$ grid-world without rewards.
While BBE covers the state space much faster than undirected exploration, it is unnecessarily slow.
See Appendix \ref{sec:gridworld-vis} for visualizations.

\section{Decoupled exploration for sample-efficient control}
In this section, we describe a new algorithm called \algname{} (\algshort{}) which addresses the limitations of BBE.
The core insight is that by leveraging off-policy RL algorithms, \algshort{} can learn two policies from the same replay: a task policy $\pitask$, which maximizes the reward on the original MDP $\mdp_R$, and an exploration policy $\piexplore$, which maximizes only the reward on the bonus MDP $\mdp_{R^+_n}$.
This decoupling serves two purposes.
First, it enables good performance even before exploration is complete by using $\pitask$ at test time.
Second, it allows $\piexplore$ to be updated aggressively in order to more closely match the non-stationary bonus MDP; unlike $\pitask$, it is not important that $\piexplore$ converge exactly to an optimal policy.

Like BBE, \algshort{} is a family of algorithms related by their structure; a particular algorithm in this family consists of a choice of an exploration reward function and an off-policy RL algorithm for learning each policy.
Throughout this work, we use a pseudo-count based exploration reward.
For discrete tasks we use Double DQN (DDQN, \citet{Hasselt2016DeepRL}) and Boltzmann policies.
For tasks with continuous actions we use soft actor-critic (SAC) for $\pitask$ and a DDQN policy for $\piexplore$.

\subsection{Pseudo-count estimation} \label{sec:kernel_counts}
Following \citet{bellemare2016unifying}, we use an exploration reward derived from pseudo-counts.
Instead of the high-dimensional pixel observations of the ALE, Control Suite has low-dimensional ($<$100-D) observations corresponding to joint and object locations and velocities.
% This allows us to replace the pseudo-counts extracted from a density estimator used in that work with a directly specified pseudo-count function based on kernels.
This lower dimensionality renders extracting pseudo-counts from a density estimator unnecessary, and in our experiments we specify a pseudo-count function based on kernels.
For a real-valued state-action pair $x = [s, a]$ and a set of previous observations $\{x_i\}_{i=1}^n$, define the pseudo-count of $x$ and the exploration reward as
\begin{align}
    \hat N_n(x) = \sum_{i=1}^n k(x_i, x) && R^+_n(s, a) = \hat N_n \big([s, a] \big)^{-1/2}
\end{align}
where $k$ is a kernel function scaled to have a global maximum $k(x, x) = 1$.
This satisfies the key requirement for a pseudo-count function, namely that a visit to a state $x$ increases $\hat N(x)$ by 1 and $\hat N(x')$ by at most 1 for any other state $x'$.
In our experiments we use a Gaussian kernel (scaled to have a maximum value of 1) with diagonal covariance.
For implementation details see Appendix \ref{sec:count_implementation}.
% \todo{write the actual reward function}

\subsection{Separating task and exploration policies}

\algshort{} uses two separate policies.
Each is trained off-policy using transitions sampled from the replay buffer; $\pitask$ is updated using the rewards from $R$ logged in the replay, while $\piexplore$ is updated using rewards from $R^+_n$.
Since $\pitask$ is trained only on the rewards for the true task, it is unbiased in the sense that it reflects the current best estimate of the optimal policy.
This stands in contrast to BBE policies, which optimize the sum of task and exploration rewards and thus represent a biased estimate of the optimal task policy until the exploration rewards go to zero.
Our method is agnostic to the choice of algorithm and policy parameterization.
However, it will be most effective with policy learning algorithms that work well when trained far off-policy and produce high-entropy policies (e.g. policies which cover all optimal actions).
For these reasons we use the state-of-the-art maximum-entropy algorithm SAC to learn $\pitask$ in environments with continuous actions. For experiments with discrete action spaces we forego explicitly learning the task policy $\pitask$; instead we learn the task Q-function via DDQN \citep{Hasselt2016DeepRL} and define the task policy as $\pitask(a | s; Q) \propto \exp(Q(s, a) / \tau)$, where $\tau > 0$ becomes a hyperparameter -- we refer to the supplementary material for details.

\subsection{Fast-adapting exploration policy} \label{sec:fast_adaptation}
The non-stationary nature of the exploration reward function poses a challenge to typical model-free RL algorithms, which assume a fixed reward function.
BBE methods update a single policy using a replay buffer which, at a step $n$, contains rewards from a mixture of bonus reward functions $\{R_1^+, \ldots, R_n^+\}$, computed using different past novelty or count estimates.\endnote{See e.g. the code from \citet{Machado2020CountBasedEW}:  \url{https://github.com/mcmachado/count_based_exploration_sr/blob/master/function_approximation/exp_eig_sr/train.py\#L204}}
This results in slow adaptation to the non-stationary objective of exploration.
\algshort{} makes two changes to mitigate the impact of the non-stationarity in the exploration reward function.

First, \algshort{} leverages access to $R^+_n$ to compute exploration rewards when they are needed to update $\piexplore$ rather than when the transition is collected.
We choose to use Q-learning rather than a more sophisticated algorithm in order to learn $\piexplore$ as rapidly as possible; with changing rewards, using a policy to amortize the maximization of a value function as in SAC or DDPG would slow down learning.
We represent $\piexplore$ directly as a Boltzmann policy of this exploration Q-function $\qex$:
\begin{align}
    \piexplore(a \mid s) \propto \exp \left\{ \frac{\qex(s, a)}{\tex} \right\},
\end{align}
where $\tex$ is a temperature hyperparameter.

Second, by decoupling $\piexplore$ from $\pitask$, \algshort{} unlocks the ability to update $\piexplore$ more aggressively without affecting $\pitask$'s convergence to the optimal policy.
This enables the exploration policy to more rapidly adapt to the non-stationary exploration reward.
In our experiments we achieve this by using a larger learning rate and more updates per environment step than is usually done; future work might investigate more sophisticated schemes such as prioritized sweeping \citep{Moore1993PrioritizedSR} or prioritized experience replay \citep{Schaul2016PrioritizedER}.
To improve stability we use DDQN updates and clip Q targets at $\bar{r}$, the maximum discounted exploration value.
% \todo{discuss clipping targets at 100?}

\paragraph{Optimism.} \label{sec:optimistic}
Further adapting $\piexplore$ to the unique properties of the exploration reward function, we propose to leverage optimism in its updates and actions.
We propose to make $\qex$ optimistic by leveraging the pseudo-count function in a manner similar to that proposed by \citet{Rashid2020OptimisticEE}.
We assume that the value function is trustworthy for transitions with very large counts, and very untrustworthy for transitions with near-zero counts.
When the count is zero we impose an optimistic prior which assumes the transition will lead to a whole episode of novel transitions; as the count increases we interpolate between this prior and the learned value function using a weighting function:
\begin{align} \label{eq:optimism}
\qex^+(s, a) = w(s, a) \cdot \qex(s, a) + (1 - w(s, a))  \cdot \bar{r}, \qquad w(s, a) = \frac{\sqrt{N(s, a)}}{\sqrt{N(s, a) + c}}
\end{align}
where $\bar{r} = \nicefrac{1}{1-\gamma}$, is the maximum discounted return in the bonus MDP and $c$ is a small constant representing how many counts' worth of confidence to ascribe to the optimistic prior.
We use this optimistic $\qex^+$ for computing targets for Bellman updates and for computing $\piexplore$ (Eq. \ref{eq:behavior_policy}).
For details of the implementation of the fast-adapting $\piexplore$, see Appendix \ref{sec:fast_updates_appendix}.

\subsection{Product distribution behavior policy}
A good behavior policy should attempt to explore all of the transitions which are relevant for learning the optimal policy.
This entails a trade-off between taking actions which are more novel and ones which are more likely to be relevant to a high-performing policy.
\algshort{} encodes this by representing the behavior policy as a product of the task policy $\pitask$ and the pure-exploration policy $\piexplore$:
\begin{align} \label{eq:behavior_policy}
\beta(a \mid s) \propto \pitask(a \mid s) ~ \piexplore(a \mid s).
\end{align}
% where $\tex$ is a temperature hyperparameter controlling the aggressiveness of the exploration and $\qex^+$ is an optimistic version of the exploration value function (see \Cref{sec:optimistic}).

The choice to parameterize $\beta$ as a factored policy was made for its simplicity and ease of off-policy learning.
Alternative formulations for making this trade-off while preserving the unbiased task policy are possible, and we view the form of our proposed behavior policy as just one option among many.
One alternative would be interleaving the behavior of multiple policies within one episode, akin to e.g. Scheduled Auxiliary Control \citep{Riedmiller2018LearningBP}.

In order to approximately sample from this behavior policy, we use self-normalized importance sampling with $\pitask$ as the proposal distribution:
\begin{enumerate}
    \item Draw $k$ samples $a_1, \ldots, a_k$ from $\pitask$
    \item Evaluate $\piexplore(a_i \mid s)$ for each $i \in 1 \ldots k$
    \item Draw a sample from the discrete distribution $p(a_i) = \piexplore(a_i \mid s) / \sum_{i'} \piexplore(a_{i'} \mid s)$.
\end{enumerate}
Note that since the proposal distribution is $\pitask$, the $\pitask$ terms in computing weights cancel and only the $\piexplore$ terms remain.
Importance weighting is consistent in the limit of $k \to \infty$ but introduces bias towards $\pitask$ \citep{Vehtari2015ParetoSI}.
With small $k$, this bias makes it unlikely that $\beta$ will select actions that are very unlikely under $\pitask$; roughly speaking, this procedure selects the ``most exploratory'' action in the support of the task policy.
This may act as a backstop to prevent the behavior from going too far outside the task policy to be useful.
\algshort{} works best when $\pitask$ is trained in a way that preserves variance in the policy (e.g. SAC's target entropy), enabling the behavior policy to select exploratory actions. In discrete action spaces we additionally use self-normalized importance sampling -- using a uniform proposal over actions -- to obtain samples from $\pitask$ in step 1.
% and \algshort{} with a deterministic task policy would .

% If the RL algorithm for learning $\pitask$
% If the RL algorithm for learning $\pitask$
% As mentioned above, this procedure works best when $\pitask$ is trained in a way that preserves variance in the policy (e.g. by enforcing a minimum entropy in the policy).
% If the choice of RL algorithm does not give such a guarantee importance sampling based on a mix of a random proposal and the task policy could be performed.

\section{Experiments}
% \red{Goals: (1) show that BBE hurts on dense environments, and ours doesn't; (2) show that ours learns fastest on spare environments}

In this section we perform experiments to give insight into the behavior of undirected exploration, BBE, and \algshort{}.
% In particular, we investigate how each algorithm's performance changes as the task requires more exploration
First we perform a set of investigative experiments to probe how \algshort{} interacts with environments with different reward structures.
Then we perform experiments on pairs of benchmark continuous control tasks with easy and hard exploration requirements.

\begin{figure}[h]
    \centering
    \begin{subfigure}[b]{0.49\textwidth}
        \centering
        \includegraphics[width=0.8\textwidth]{figures/deep/hallway_velocity_4_distractor.pdf}
        \caption{Local optimum environment} \label{fig:hallway_local_optimum}
    \end{subfigure}
    \hfill
    \begin{subfigure}[b]{0.49\textwidth}
        \centering
        \includegraphics[width=0.8\textwidth]{figures/deep/hallway_velocity_4_inverse_distractor.pdf}
        \caption{Adversarial environment}\label{fig:inverse_distractor}
    \end{subfigure}
    \caption{Two environments illustrating different reward structures.
    \textbf{(a)} An environment with a locally-optimal goal (reward 0.1) near the start state. SAC finds this nearby goal, but doesn't explore far enough to find the real goal (reward 1.0). When trained with \algshort{}, it finds the distractor goal but moves on to the real goal.
    \textbf{(b)} An adversarial environment for \algshort{} which has a very small goal state close to the start state, making it easy to find with random actions but hard with directed exploration.}
    \label{fig:hallway}
\end{figure}

\subsection{Investigative experiments}

We construct a simple MuJoCo \citep{todorov2012mujoco} environment called Hallway to look more closely at how exploration interacts with reward structure.
This environment consists of a long narrow 2D room with the agent controlling the velocity of a small sphere, which starts each episode at one end of this hallway.
The following two experiments share dynamics and differ only in their rewards.

\paragraph{Local optima.}
A valuable role for exploration is enabling an agent to escape from locally optimal behavior.
% behavior that is locally but not globally optimal.
To test this, we add two goal states with shaped rewards to the Hallway environment.
The first is close to the start state but only provides reward at most 0.1, while the second is at the far end of the hallway but provides reward 1.0.
\Cref{fig:hallway_local_optimum} shows that exploration using \algshort{} allows the agent to quickly find its way to the faraway optimal reward while SAC gets stuck in the local optimum.

\paragraph{Limitations.}
% While \algshort{} outperforms SAC in each of the Control Suite environments, there is no such thing as a universally optimal exploration strategy.
\algshort{} covers states quickly, but there is no such thing as a universally optimal exploration strategy.
For example, there exist environments for which the random walk dynamics of undirected exploration find the optimal strategy faster than uniform state coverage.
\Cref{fig:inverse_distractor} provides one such example: a Hallway environment with a very small goal state close to the start state.
SAC discovers this goal faster than SAC + \algshort{}, though \algshort{} does eventually find it as well.

\subsection{Benchmark experiments}
Next we provide experiments based on DeepMind Control Suite \citep{tassa2018deepmind}, a standard benchmark for continuous control RL algorithms.
We introduce versions of several environments which are modified to remove the accommodations that make them solvable without exploration, then provide results of SAC, SAC + BBE, and SAC + \algshort{} on the original and modified environments.

\subsubsection{Results}
\begin{figure}[ht]
    \centering
    \begin{subfigure}[b]{0.24\textwidth}
        \centering
        \includegraphics[width=\textwidth]{figures/deep/neurips_ball_in_cup.pdf}
        \caption{Ball-in-cup}
    \end{subfigure}
    \begin{subfigure}[b]{0.24\textwidth}
        \centering
        \includegraphics[width=\textwidth]{figures/deep/neurips_ball_in_cup_explore.pdf}
        \caption{Ball-in-cup explore}
    \end{subfigure}
    % \caption{Adding \algshort{} does not result in worse performance on the original ball-in-cup environment. In a version of the environment with resets that do not provide full coverage, \algshort{} performs just as well, whereas the performance of plain SAC is greatly impaired.}
    % \label{fig:ball_in_cup}
    \hfill
    \begin{subfigure}[b]{0.24\textwidth}
        \centering
        \includegraphics[width=\textwidth]{figures/deep/neurips_reacher.pdf}
        \caption{Reacher}
    \end{subfigure}
    \begin{subfigure}[b]{0.24\textwidth}
        \centering
        \includegraphics[width=\textwidth]{figures/deep/neurips_reacher_explore.pdf}
        \caption{Reacher explore}
    \end{subfigure}
    % \caption{Results on the original Reacher environment and on a version modified to reset the arm in one quadrant and the target in the opposite quadrant.}
    % \label{fig:reacher}
    \vspace{1em}

    \begin{subfigure}[b]{0.24\textwidth}
        \centering
        \includegraphics[width=\textwidth]{figures/deep/neurips_finger.pdf}
        \caption{Finger}
    \end{subfigure}
    \begin{subfigure}[b]{0.24\textwidth}
        \centering
        \includegraphics[width=\textwidth]{figures/deep/neurips_finger_explore.pdf}
        \caption{Finger explore}
    \end{subfigure}
    \hfill
    \begin{subfigure}[b]{0.24\textwidth}
        \centering
        \includegraphics[width=\textwidth]{figures/deep/neurips_walker.pdf}
        \caption{Walker}
    \end{subfigure}
    \begin{subfigure}[b]{0.24\textwidth}
        \centering
        \includegraphics[width=\textwidth]{figures/deep/neurips_walker_explore.pdf}
        \caption{Walker explore}
    \end{subfigure}
    \caption{Results on original Control Suite environments (left in each pair) and modified versions without exploratory resets and rewards (right).
    Across the original environments, SAC + \algshort{} performs as well or better than SAC, while SAC + BBE performs much worse on some environments.
    On the exploration environments, \algshort{} + SAC learns much faster than SAC.
    BBE sometimes provides significant gains over SAC but sometimes performs worse even on exploration environments.
    % \red{(Rename algorithm and make fonts bigger)}}
    }
    \label{fig:control_suite}
\end{figure}

\subsubsection{Environments for evaluating exploration}
% The DeepMind Control Suite \citep{tassa2018deepmind} is a standard benchmark for continuous control RL algorithms.
While Control Suite has driven great progress in policy learning, it was not designed to evaluate an agent's exploration capabilities; in fact, the included environments were selected to be solvable by algorithms with only undirected exploration. From that work:
\begin{quote}
    We ran variety of learning agents (e.g. Lillicrap et al. 2015; Mnih et al. 2016) against all tasks, and iterated on each task’s design until we were satisﬁed that
    [\ldots]
    % we were satisﬁed that the physics was stable and non-exploitable, and that
    the task is solved correctly by at least one agent. \citep{tassa2018deepmind}
\end{quote}
Control Suite avoids the need for directed exploration via two mechanisms.
First, in many environments the start state distribution is sufficiently wide (e.g. uniform over reachable states) to guarantee that any policy will see high-value states.\endnote{Some environments, such as Manipulator, additionally start a fraction of episodes at the goal state.}
Second, some environments have rewards shaped to guide the agent towards better performance (e.g. a linearly increasing reward for forward walking speed).

To construct a benchmark for continuous control with exploration, we selected four environments with different objectives (manipulation and locomotion, single-objective or goal-conditional) and observation dimensions (6-24).
We then created ``exploration'' versions of these environments with restricted start state distributions and sparse rewards.
The original environments and their exploration versions together form a benchmark which measures an algorithm's exploration ability and policy convergence.
Environment details are in Appendix \ref{sec:environments_appendix} and their implementation is in the supplement.

\subsubsection{Algorithms}
We include experiments on these eight benchmark tasks with three algorithms: SAC \citep{haarnoja2018softapp} with no additional exploration; BBE with SAC for the policy learner; and \algshort{} with SAC for $\pitask$ and DDQN for $\piexplore$.
BBE and \algshort{} use the pseudo-count reward described in \Cref{sec:kernel_counts} and the SAC implementation is that of \citet{pytorch_sac} with no hyperparameter changes.

The kernel-based exploration bonus used for BBE and \algshort{} requires a scaling law to set the kernel variance as a function of the observation dimension.
We adapt the scaling relationship from \citep{Henderson2012NormalRB} (see Appendix \ref{sec:count_implementation}).
BBE has an additional hyperparameter for the scale of the bonus.
We performed a sweep with values in $\{ 10^{-2}, 10^{-1}, 1, 10 \}$ and found that $1$ performed best overall.
This setting, which makes the maximum bonus equal to the maximum environment reward, ensures that visiting a new state remains the best option until the true goal state is discovered.
Further implementation details are available in Appendix \ref{sec:appendix_benchmark_implementation}.
We additionally performed an ablation which, like \algshort{}, learns separate Q functions for the two rewards, but which learns one policy to maximizes the sum of their Q values.
In our experiments (available in Appendix \ref{sec:benchmark_results_appendix}) this ablation never outperforms BBE, so for clarity we exclude it here.
% Those results are available in Appendix \ref{sec:benchmark_results_appendix}.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.95\textwidth]{figures/deep/control_suite_summary_100.png}
    \caption{Results after 100 episodes. In this extremely sample-limited regime, exploration speed and fast policy convergence are both essential. In every environment, SAC with \algshort{} (blue, right column in each set of three) performs comparably to or better than SAC alone or SAC with BBE.}
    \label{fig:control_suite_summary}
\end{figure}

We present experiments on the original versions of four Control Suite tasks and their exploration counterparts.
The results are shown in \Cref{fig:control_suite} with the means and 95\% confidence intervals over 8 seeds.
We find that across the original environments, \algshort{} gives similar or slightly better performance to SAC, while BBE significantly impairs SAC on two of the four environments and matches SAC on the other two.
Across the exploration environments, \algshort{} gives the best performance and sample efficiency.
BBE performs better than SAC alone on two exploration environments and worse than SAC on the other two.
\Cref{fig:control_suite_summary} shows the performance of each algorithm after only 100 episodes, highlighting the substantial benefits from using \algshort{} in the few-sample regime.

Overall, SAC + \algshort{} never performs worse than SAC alone, while yielding substantial improvements in environments where rewarding states are harder to discover.
BBE's more mixed performance provides a possible explanation for the limited influence that methods of that family have had on sample-efficient continuous control, and perhaps more generally on sample-efficient RL.
Given that in this setting the addition of BBE is as likely to harm as to help, its lack of adoption is unsurprising.

\section{Related work}

\paragraph{Sample-efficient continuous control.}
% As discussed in the introduction, there has been a large amount of progress on sample efficient off-policy RL that
Our method leverages progress on sample efficient off-policy RL, as it can be combined with any off-policy algorithm.
A strong line of work has brought the sample complexity of model-free control within range of solving tasks on real robots \citep{Popov2017DataefficientDR,kalashnikov2018qt,haarnoja2018soft,fujimoto2018addressing,haarnoja2018softapp,Abdolmaleki2018MaximumAP}.

% In general, current RL algorithms can be clustered into two categories: on-policy algorithms which are reliable but require hundreds of millions of environment steps to solve tasks \citep{Mnih2016AsynchronousMF,Schulman2015TrustRP,schulman2017proximal}, and recent off-policy algorithms which have brought the sample complexity of model-free control within range of solving tasks on real robots \citep{Popov2017DataefficientDR,kalashnikov2018qt,haarnoja2018softA,fujimoto2018addressing,haarnoja2018softapp,Abdolmaleki2018MaximumAP}.

%\subsection{Exploration in deep RL}

\paragraph{Bonus-based exploration.}
There have been many bonuses proposed in the BBE framework.
Several works \citep{Stadie2015IncentivizingEI,pathak2017curiosity,burda2018exploration} propose to use prediction error of a learned model to measure a transition's novelty, with the key differences being the state representation used for making predictions.
% \citet{Stadie2015IncentivizingEI} make predictions in the raw observation space, while \citet{pathak2017curiosity} use a learned embedding from an inverse dynamics model, and \citet{burda2018exploration} use a randomly-initialized neural network to embed the states.
\citet{houthooft2016vime} propose a bonus based on the information gain of the policy.
\citet{bellemare2016unifying} and others \citep{ostrovski2017count,Tang2017Exploration} use continuous count analogues to calculate the count-based bonuses of \citet{strehl2008analysis}.
%  also use the bonus formula from \citet{strehl2008analysis}, but compute counts using a locality-sensitive hashing function to bin the states.
\citet{Machado2020CountBasedEW} use the norm of learned successor features as a bonus, and show that it implicitly counts visits.
Unlike previous work, our paper focuses on the updates and representation of the behavior policy, and \algshort{} can be used in conjunction with any of these bonuses.
Never Give Up \citep{Badia2020NeverGU} uses an episodic exploration bonus and trains policies with different bonus scales including a task policy.
% exploration bonus with an episodic structure, encouraging a behavior policy to visit each state infinitely often while rapidly adapting the reward within each episode.
% Instead of a single policy, NGU learns an ensemble of policies with different scales of the exploration bonus, allowing it to act greedily with respect to a task policy.
However, it is designed to maximize asymptotic performance rather than sample efficiency and does not learn faster than a baseline early in training.

\paragraph{Optimism.}
Classic exploration methods \citep{Kearns1998NearOptimalRL,Brafman2002RMAXA,strehl2008analysis,Jaksch2008NearoptimalRB}, depend on an optimistically-defined model.
Model-free methods with theoretical guarantees \citep{Strehl2006PACMR,Jin2018IsQP} use Q functions initialized optimistically.
% Optimism is difficult to ensure with neural networks, since even an optimistic initialization will rapidly wash out due to generalization from other observed points.
Similar to our \cref{eq:optimism}, \citet{Rashid2020OptimisticEE} propose a method for ensuring optimism in Q learning with function approximation by using a count function.
% Our method uses a similar formulation for learning an optimistic exploration Q function.
However, \algshort{} leaves the task policy unbiased in the few-sample regime by separating the exploration policy from the task policy.

\paragraph{Temporally-extended actions.}
A variety of work proposes to speed up $\epsilon$-greedy exploration via temporally-extended actions which reduce dithering.
Some methods \citep{Schoknecht2003ReinforcementLO,neunert2020continuousdiscrete} propose to bias policies towards repeating primitive actions, resulting in faster exploration without limiting expressivity.
\citet{Dabney2020TemporallyExtendedE} describe a temporally-extended version of $\epsilon$-greedy exploration which samples a random action and a random \emph{duration} for that action.
\citet{Whitney2020Dynamics-Aware} use a learned temporally-extended action space representing the reachable states within a fixed number of steps.
While these methods improve over single-step $\epsilon$-greedy, they are unable to perform directed exploration or discover faraway states.

\paragraph{Randomized value functions.}
Modern works \citep{Osband2016DeepEV,osband2019deep} extend Thompson sampling \citep{thompson1933likelihood} to neural networks and the full RL setting.
Relatedly, \citep{Fortunato2018NoisyNF,Plappert2018ParameterSN} learn noisy parameters and sample policies from them for exploration.

\section{Discussion}
In this paper we have investigated the potential for directed exploration to improve the sample efficiency of RL in continuous control.
We found that BBE suffers from bias and slow state coverage, leading to performance which is often worse than undirected exploration.
We introduced \algname{}, which separately learns an unbiased task policy and an exploration policy and combines them to select actions at training time.
\algshort{} pays no performance penalty even on dense-reward tasks and explores faster than BBE.
In our experiments, \algshort{} combined with SAC provides strictly better performance and sample efficiency than SAC alone.
We believe that with its combination of reliable and efficient policy learning across dense and sparse environments, SAC + \algshort{} provides a compelling default algorithm for practitioners.

% \clearpage
\printendnotes


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%                                                                 %
%  GEANT manual in LaTeX form                                     %
%                                                                 %
%  Michel Goossens (for translation into LaTeX)                   %
%  Version 1.00                                                   %
%  Last Mod. Jan 24 1991  1300   MG + IB                          %
%                                                                 %
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\Documentation{R.Brun, F.Bruyant}
\Submitted{01.10.84}           \Revised{26.10.93}
\Version{Geant 3.16}           \Routid{BASE090}
\Makehead{The reference systems and physical units}
\section{The {\tt MA}ster {\tt R}eference {\tt S}ystem ({\tt MARS})}

The kinematic variables of the particles transporter by {\tt GEANT}
are always referred to the so-called
{\tt MA}ster {\tt R}eference {\tt S}ystem ({\tt MARS}). This system
is implicitly defined as the local reference system of the first
volume defined, which contains all the others. This is a Cartesian
coordinate system with axis $\hat{x}, \hat{y}, \hat{z}$ where
$\hat{z} = \hat{x} \times \hat{y}$.
If the axes are labelled {\tt (X,Y,Z)}, then the point {\tt P}
is represented in fig \ref{fg:base090-1}.

\begin{figure}[hbt]
      \centering
      \epsfig{file=eps/base090-1.eps,width=10cm}
      \caption{{\tt GEANT} reference system}
      \label{fg:base090-1}
\end{figure}

Tracking is performed in the {\tt MARS} and the input position for
user routines such as the magnetic field routine is given in this
system.

\section{The local reference systems ({\tt MRS} and {\tt DRS})}

As explained in {\tt [GEOM001]}, the setup is
described via the definition of an initial volume inside which all
the others will be positioned. In {\tt GEANT} terminology, each time
a volume has contents, created either via division or by positioning
other volumes inside, it is called a {\tt MOTHER}. The volumes contained
are called {\tt DAUGHTER}s, and they, in turn, can contain volumes to
a depth of 15 levels. This is sometimes referred to as a {\it Russian doll}
geometry.

Every volume defined in {\tt GEANT} has a reference system attached to
it (see {\tt GEOM} section). When this volume has contents, this
is referred to as the {\tt M}other {\tt R}eference {\tt S}ystem
({\tt MRS}, with origin in O$_m$). Daughters
are positioned inside the mother with respect to the {\tt MRS}. The
{\tt MRS} of the first volume defined, containing all the others, is
nothing else than the {\tt MARS}.

Each one of the daughters has its own reference system, which is referred
to as the {\tt D}aughter {\tt R}eference {\tt S}ystem, or {\tt DRS} with
origin in O$_d$.

The transformation of a point from the {\tt MRS} (V$_m$)
to the {\tt DRS} (V$_d$), at any level, is performed using a rotation
matrix $[R]$ and a translation vector $T$ via the relation :
     \[ V_d  =[ R ](V_m -T) \]
The components of $T$ are the projections of the vector  $ (O_m, O_d) $
onto the {\tt MRS} axes.
The rotation matrices are computed from
the spherical angles of each of the axes of the
daughter reference systems ({\tt I, II, III})
with respect to the mother reference system ({\tt 1, 2, 3}).
The spherical angles $\Theta$ and $\Phi$ of a
direction $D$ are defined as follows :
\begin{DLtt}{MMMMM}
\item[$\Theta$]     is the angle formed by the axis 3 and D
                 ($0^{\circ}\;<\;\Theta\;<\;180^{\circ}$).
\item[$\Phi$]      is the angle formed by the axis 1 and the projection
                of D onto the plane defined by the axes 1 and 2
                 ($0^{\circ}\;<\;\Phi\;<\;360^{\circ}$).
\end{DLtt}
Examples are given in {\tt [GEOM200]}.
The various rotation matrices required for a given setup must be
defined by the user during the initialisation stage.
A number is assigned to each matrix {\tt [GEOM200]}.
The translation vector and the number of the rotation
matrix are specified by the user when the volumes are
positioned inside their mother {\tt [GEOM110]}.

\section{Physical units}

Unless otherwise specified, the following units are
used throughout the program:
centimeter, second, kilogauss, GeV, GeV c$^{-1}$ (momentum),
GeV c$^{-2}$ (mass) and degree.


\documentclass[12pt]{article}
\newcounter{problem}
\begin{document}\thispagestyle{empty}

\section*{NYU General Physics 1---Problem set 14}

\paragraph{Problem~\theproblem:}\refstepcounter{problem}%
``Surface tension'' has dimensions of force per unit length or energy
per unit area.  It is to a surface as pressure is to a volume
(pressure has dimensions of force per unit area or energy per unit
volume).  It is to a surface as tension is to a string (tension has
dimensions of force or energy per unit length).

\textsl{(a)} If you have a spherical object of radius $R$ surrounded
by a stretched surface with surface tension $\sigma$, what, by dimensional
analysis, do you expect to be the \emph{pressure} inside the object?

\textsl{(b)} The pressure inside a living cell is set by salinity.  If
the cell is in a saline-rich environment, water leaves the cell (for
the environment) and the cell shrinks.  If the cell is in a
saline-poor environment, water enters the cell.  In a very low-salt
environment, the cell keeps taking on water until the pressure is set
by the surface tension.  Using your result from part \textsl{(a)},
estimate the pressure in a 5-micron-radius cell in a low-salt
environment if the surface tension of its surface is roughly 5
nanonewton per nanometer.  Get your answer in atmospheres.

\textsl{(c)} Look up the surface tension of pure water and estimate
the gauge pressure inside a typical raindrop.  I say ``gauge
pressure'' because I mean the \emph{additional} pressure provided by
the surface over and above atmospheric pressure.  \emph{In both this
  part and the previous, you are computing gauge pressures.}

\paragraph{Problem~\theproblem:}\refstepcounter{problem}%
\textsl{(a)} Re-find the ram-pressure and viscous-drag forces we have
found previously by dimensional analysis.  Reminder: Ram pressure
depends only on the size of the object, its velocity, and the mass
density $\rho$ of the fluid; viscous-drag depends in addition on the
kinetic viscosity $\nu$ (look it up!).

\textsl{(b)} At roughly what speed does ram pressure exceed stokes
drag for a typical automobile driving through air?  You will need to
look up the viscosity of air.  Check the dimensions of the numbers you
are using!  \emph{Hint: You might want to make the ``spherical cow''
  approximation for the car; this is just rough after all!}

\textsl{(c)} What about for a red blood cell moving through plasma?
Look up what you need to know on the internet; that will include the
sizes of the cells and the properties of plasma.

\textsl{(d)} Do red blood cells float in plasma or do they sink?
Either way, find the velocity at which the viscous drag force, the
gravitational force, and the buoyant force sum to zero.  What would
be different about this velocity if the blood cells and plasma were in
a centrifuge spinning very fast?

\paragraph{Problem~\theproblem:}\refstepcounter{problem}%
[\textsl{optional}] Pick up a copy of \textit{The Making of the Atomic
  Bomb} by Richard Rhodes and read it over the break.  Send Prof Hogg
an email with your reactions to it sometime next term.  \emph{Extra
  hard:} Use what you find in the book to calculate (very roughly) what's called
critical mass (the minimum amount of uranium 235 it takes to sustain a
nuclear fission reaction).  Check your answer on Wikipedia.

\end{document}


\chapter{Bayesian fusion model}
\label{chap_BayesianFusion}

This chapter proposes a Bayesian fusion model \citep{van2015bayesian} to combine HTLS and LTHS measurements as described in section \ref{sec:probdef2}. The model takes benefit from both sources of information by searching for the most probable flow given the measurements thanks to a \textit{maximum a posteriori} estimate. Better performances are expected since space and time correlations are equally important. The model also recovers flow details inaccessible from a single source of measurement via interpolations. By integrating directly the measurements, a compromise estimate is proposed such that detailed information of the flow close to the sensors is well preserved. This approach also overcomes the limitations of simple regression, which acts as a low pass filter due to the mean square error minimization. It potentially outperforms propagation models by using the space-time measurements in a balanced manner, permitting to take benefits equally from the two sources of information. In this chapter, DNS data of isotropic turbulence described in section \ref{sec:data_isotropic} is used to investigate model performances. Further comparisons with other methods and demonstrations for DNS data of channel flow will be presented later in chapter \ref{chap_comparisons}.

\section{A probabilistic model}
Bayesian framework has been used widely from very early in communication problems \citep{van1967detection,sage1971estimation} and more recently in image processing, remote-sensing, and data fusion \citep{levitan1987maximum,ma2000simultaneous,challa2004bayesian,koks2003introduction,durrant2008multisensor,zhang2009noise,joshi2010map}. The core of the framework is Bayes' rule, which describes how the probability of a system is altered by a new evidence or measurement. One of the main application of Bayes' rule is the inference, with the aids of different Bayesian estimators.

\subsection{Bayes' rule}
Let $ \mybold{d} $ denote a vector of observations/measurements/data, and $ \h $ denote a hypothesis. This hypothesis can be a parameter or an unknown variable to estimate, with a \textit{prior distribution} $ p(\h) $. The probability of the observation vector, also called \textit{evidence}, is $ p(\mybold{d}) $. The joint probability $ p(\mybold{d},\h) $ can be expressed as \citep{kendall1987kendall}:
\begin{equation}
\begin{split}
p(\h,\mybold{d}) &= p(\h|\mybold{d}) p(\mybold{d}) \\
								 &= p(\mybold{d}|\h) p(\h)
\end{split}
\end{equation}
where $ p(.|.)  $ is the conditional distribution of one variable given the other. Rearranging the above equation, the conditional distribution of the hypothesis knowing the data is:
\begin{equation}
p(\h|\mybold{d}) =  \frac{p(\mybold{d}|\h) p(\h)}{p(\mybold{d})}
\end{equation}
where $ p(\mybold{d}) $ is a constant. This relation is referred as \textit{Bayes's rule}, which can be rewritten as:
\begin{equation}
p(\h|\mybold{d}) \propto p(\mybold{d}|\h) p(\h)
\end{equation}
or in general:
\begin{equation}
[\text{posterior distribution}] \propto [\text{likelihood}]*[\text{prior distribution}]
\end{equation}
The prior distribution $ p(\h) $ describes our knowledge of $ \h $ without observing the data $ \mybold{d} $, while $ p(\h|\mybold{d}) $ is the corresponding knowledge after observing the data. $ p(\mybold{d}|\h) $ is \textit{likelihood} function, which measures how far the hypothesis $ \h $ explain observations $ \mybold{d} $.

\subsection{Bayesian estimates}
Bayes's rule is very simple but very powerful. Inference is one of the most important  applications. In most inference problem, either maximum likelihood estimate (MLE) or maximum a posteriori (MAP) estimate is used.

\subsubsection{Maximum likelihood estimate (MLE)}
The likelihood is the probability of the measurements $ \mybold{d} $ under the hypothesis $ \h $. For given measurements, MLE finds the hypothesis $ \h $ that maximizes the likelihood function:
\begin{equation}
\hat{\h} = \argmax_{\h}{ \left\lbrace p(\mybold{d}|\h) \right\rbrace}
\end{equation}
The resulting $ \hat{\h} $ can be considered as the one that makes the measurements the most ``probable'' or ``likely''. Alternatively, the MLE is often obtained by maximizing the \textit{log-likelihood}:
\begin{equation}
\hat{\h} = \argmax_{\h}{ \left\lbrace log(p(\mybold{d}|\h)) \right\rbrace}
\end{equation}
In many situations, an explicit expression of MLE can be obtained.

\subsubsection{Maximum a posteriori estimate (MAP)}
In many cases, some knowledge of undergoing phenomena is known \textit{a priori}. Such knowledge can be from theory or empirical evidence, represented via the prior distribution $ p(\h) $. To find the most probable hypothesis $ \h $ given measurements $ \mybold{d} $ and some prior knowledge of $ \h $, the posterior probability $ p(\h|\mybold{d}) $ is maximized. Bayes' rule tells us how to incorporate prior knowledge into the maximization problem as:
\begin{equation}
\begin{split}
\hat{\h} &= \argmax_{\h}{ \left\lbrace p(\h|\mybold{d}) \right\rbrace} \\
					 &= \argmax_{\h}{ \left\lbrace p(\mybold{d}|\h) p(\h) \right\rbrace}
\end{split}
\end{equation}
MAP is different from MLE due to the presence of the prior probability. This prior plays the role of a regularization when dealing with ill-posed inverse problems.

\section{Bayesian fusion model}
\label{sec:Bayesian_fusion_model}
In a Bayesian approach, this section derives a fusion model to combine LTHS and HTLS measurements to reconstruct fully-resolved HTHS fields. This is done by considering all variables as random. The most probable HTLS is sought such that it maximizes a \textit{posterior} probability given the two measurements via a MAP estimate.

\subsection{The model}
Let $ \x $ and $ \y $ denote LTHS and HTLS measurements, and $ \z $ denote HTHS data to reconstruct. $ \x $ and $ \y $ are directly subsampled from HTHS, i.e. $ \x = \Sub_t\z $ and $ \y = \Sub_t\z $. $ \z $, $ \x $ and $ \y $ are random, zero-mean vectors of size $ \dimsh \dimth \times 1$, $ \dimsh \dimtl \times 1$ and $\dimsl \dimth \times 1$ respectively.  $ \dimsh  $ and $ \dimsl  $ are numbers of spatial points in each snapshot, while $ \dimth $ and $ \dimtl $ are numbers of snapshots. The present work is a challenging inverse problem since we consider  $ \dimsl  \ll \dimsh  $ and $ \dimtl \ll \dimth $.
Let ``s'' denote the subscript of operators performing in space, and ``t'' denote the subscript of those in time. $ \Interp $ is an cubic spline interpolator, $ \Sub $ is for subsampling, and $ \LPF $ is a low-pass filter to separate large scales from small scales. In the case of isotropic turbulence,  $ \LPF $ is the ideal Fourier filter thanks to the isotropic and periodic properties. In the case of channel flow,  $ \LPF $ is a least-square spline filter for its sharp cutoff response.

Given LR measurements of either $ \y  $ in space or $ \x  $ in time, the fully resolved vector $ \z  $ can be estimated by single interpolations. The 1D time interpolation goes from a $ \dimsh \dimtl $ to a $ \dimsh \dimth $- dimensional space, i.e. $ \x  \mymapto \hat{\z }=\Interp_t \x  $, while the 2D space interpolation goes from a $ \dimsl \dimth$ to a $ \dimsh \dimth $- dimensional space, i.e. $ \y  \mymapto \hat{\z }=\Interp_s \y $. Let $ \dimsh \dimth $- dimensional vectors $ \h_s $ and $ \h_t $ denote the information that cannot be recovered by simple interpolations $ \Interp_s $ or $ \Interp_t $. $\z  $ can be modeled in two ways:
\begin{align}
	\z  &= \Interp_t\x +\h_t \label{eq:Bayes11}\\
	\z  &= \Interp_s\y +\h_s \label{eq:Bayes12}
\end{align}
Missing information $ \h_t $ and $ \h_s $ essentially feature small scales. Using either $ \x  $ or $ \y  $, it is not possible to estimate $ \h_t $ and $ \h_s $. The idea of Bayesian fusion is to combine the two models by using $ \Interp_t\x  $ in (\ref{eq:Bayes11}) to estimate the unknown $ \h_s $ in ~(\ref{eq:Bayes12}) and vice-versa.

Let $\distr(\mybold{u}|\mybold{\mu}_{\mybold{u}},\Sigma_{\mybold{u}}) $ denote the multivariate Gaussian distribution of a $ \dimsh \dimth $ dimensional random vector $ \mybold{u} $ with mean value $ \mybold{\mu}_{\mybold{u}} $ and covariance matrix $\Sigma_{\mybold{u}}$.
%varying due to a random Gaussian noise $ \mybold{n} $.
The $ \dimsh \dimth \times \dimsh \dimth $ matrix is the expectation of $ (\mybold{u}-\mybold{\mu}_{\mybold{u}} )(\mybold{u}-\mybold{\mu}_{\mybold{u}} )^{\mytrans} $.
The probability density function (pdf) of $ \mybold{u} $ with a multivariate Gaussian distribution $ \distr(\mybold{u}|\mybold{\mu}_{\mybold{u}},\Sigma_{\mybold{u}}) $ is:
\begin{equation}
	p(\mybold{u})=\frac{1}{(2\pi)^{\dimsh \dimth/2}|\Sigma_{\mybold{u}}|^{1/2}} \myexp{-\frac{1}{2} \Mdist{\mybold{u}-\mybold{\mu}_{\mybold{u}}}{\Sigma_{\mybold{u}}}}
	\label{eq:Bayes4}
\end{equation}
where $ |.| $ denotes the matrix determinant, and $\Arrowvert \mybold{u}-\mybold{\mu}_{\mybold{u}} \Arrowvert^2_{\Sigma_{\mybold{u}}}$ is the Mahalanobis distance:
\begin{equation}
 \Mdist{\mybold{u}-\mybold{\mu}_{\mybold{u}}}{\Sigma_{\mybold{u}}} = \left(\mybold{u}-\mybold{\mu}_{\mybold{u}}\right)^{\mytrans} \Sigma_{\mybold{n}}^{-1}\left( \mybold{u}-\mybold{\mu}_{\mybold{u}}\right)
\end{equation}
Let assume that $ \Interp_t\x  $ and $ \h_t $ are approximately independent; $ \Interp_t\x  $ captures temporal large scales of $ \x  $. Similarly, $ \Interp_s\y  $ and $ \h_s $ are assumed to be approximately independent. This is not exact, since the subsamplings cause aliasing terms in each pairs of $ (\Interp_t\x , \h_t) $ and $ (\Interp_s\y , \h_s) $. Assume also that $ \h_t $ and $ \h_s $ are zero mean Gaussian processes, i.e. $ \h_t \sim \distr (0,\Sigma_{\h_t}) $ and $ \h_s \sim \distr (0,\Sigma_{\h_t}) $. Pdfs of these unknowns are modeled as:
\begin{equation}
	p(\h_t)=\frac{1}{(2\pi)^{\dimsh \dimth/2}|\Sigma_{\h_t}|^{1/2}} \myexp{ -\frac{1}{2} \Mdist{\h_t}{\Sigma_{\h_t}}}
	\label{eq:Bayes5}
\end{equation}
and similarly for $ p(\h_s) $. Posterior probabilities of $ \z  $ knowing either $ \x  $ or $ \y  $ are then modeled as:
\begin{align}
	p(\z |\x ) &\sim \distr(\z |\Interp_t\x ,\Sigma_{\h_t}) \label{eq:Bayes21} \\
	p(\z |\y ) &\sim \distr(\z |\Interp_s\y ,\Sigma_{\h_s}) \label{eq:Bayes22}
\end{align}

\subsection{\label{subsec:MAP_estimation} MAP estimation}
The present Bayesian model aims to build an estimate of $ \z  $ given $ \x  $ and $ \y  $ using the probability models (\ref{eq:Bayes21}) and (\ref{eq:Bayes22}). The model uses a MAP estimate to search the most probable $ \hat{\z } $ given $ \x $ and $ \y  $ such that $ \hat{\z } $ maximizes the posterior pdf $ p(\z |\x ,\y ) $:
\begin{equation}
 \hat{\z }=\argmax_{\z } \:  p \left(\z |\x ,\y \right)
 \label{eq:MAP1}
\end{equation}
Using Bayesian rules, one has:
\begin{equation}
p\left(\z |\x ,\y \right) \propto p\left(\x ,\y |\z  \right)p(\z )
 \label{eq:MAP2}
\end{equation}
Assuming that $ \x  $ and $ \y  $ are independent conditioned on $ \z  $, equation \ref{eq:MAP2} becomes:
\begin{equation}
p\left(\z |\x ,\y \right) \propto p(\x |\z )p(\y |\z )p(\z )
\label{eq:MAP3}
\end{equation}
In equation \ref{eq:MAP3}, the likelihood functions $ p(\x |\z ) $, $ p(\y |\z ) $ and the prior pdf $ p(\z ) $ appear, while only the posterior probabilities $ p(\z |\x ) $ and $ p(\z |\y ) $ are available in ~(\ref{eq:Bayes21}) and ~(\ref{eq:Bayes22}).

To complete the model, the likelihood functions can be expressed in term of the posterior pdfs and prior of $ \z  $ using Bayesian rules. Section \ref{subsec:Linear_Gaussian_model} will introduce an alternative way to estimate these functions from posterior pdfs using a linear Gaussian model. Since there is no evidence of a proper prior for the fully resolved velocity field $ \z $, it is natural to assume a ``\textit{noninformative}'' prior (also referred as ``\textit{vague}'' or ``\textit{flat}''). This prior assumes that all the values of $ \z  $ are equally likely \citep{gelman2014bayesian}. The estimation of $ \hat{\z } $ is now solely based on the measurements and not influenced by external information. The prior distribution therefore has no influence on the posterior pdfs.

With the assumption of a noninformative prior, $ p(\z ) $ is constant. Using Bayes rules, the relation between the likelihood function and the posterior pdf is:
\begin{equation}
p\left(\z |\x \right) \propto p\left(\x |\z \right)p(\z )
\end{equation}
Since $ p(\z ) $ is replaced by a constant, one gets $p\left(\x |\z \right) \propto p\left(\z |\x \right)$ and $p\left(\y |\z \right) \propto p\left(\z |\y \right)$.  Equation \ref{eq:MAP3} becomes:
\begin{equation}
p\left(\z |\x ,\y \right) \propto p(\z |\x )p(\z |\y )
\label{eq:MAP4}
\end{equation}
The MAP estimate is therefore:
\begin{equation}
	\hat{\z }= \argmax_{\z } \: p(\z |\x )p(\z |\y )\
\label{eq:MAP_5}
\end{equation}
Logarithms of $ p\left(\z |\x \right) $ and $ p\left(\z |\y \right) $ are:
\begin{align}
	-\ln p(\z |\x ) &= \frac{1}{2}\Mdist{\z -\Interp_t\x }{\Sigma_{\h_t}} +C_1 \label{eq:MAP61}\\
	-\ln p(\z |\y ) &= \frac{1}{2} \Mdist{\z -\Interp_s\y}{\Sigma_{\h_s}} +C_2 \label{eq:MAP62}
\end{align}
where $ C_1 $ and $ C_2 $ only depend on $ \Sigma_{\h_t} $ and $ \Sigma_{\h_s} $, and are independent of $ \x  $, $ \y  $ and $ \z  $. Solving the optimization problem \ref{eq:MAP_5} is equivalent to minimize the cost function:
\begin{equation}
	C(\z )=\frac{1}{2} \Mdist{\z -\Interp_t\x}{\Sigma_{\h_t}} + \frac{1}{2}\Mdist{\z -\Interp_s\y}{\Sigma_{\h_s}}
\label{eq:MAP7}
\end{equation}
Setting the gradient of $ C(\z ) $  to zero:
\begin{equation}
	\frac{\partial C(\z )}{\partial \z }= \Sigma^{-1}_{\h_s} \left( \z -\Interp_s\y  \right) + \Sigma^{-1}_{\h_t} \left( \z -\Interp_t\x  \right) = 0
\label{eq:MAP8}
\end{equation}
yields:
\begin{equation}
	\hat{\z } = \left( \Sigma^{-1}_{\h_t} + \Sigma^{-1}_{\h_s} \right)^{-1} \left(\Sigma^{-1}_{\h_s}\Interp_s\y +\Sigma^{-1}_{\h_t}\Interp_t\x \right)
\label{eq:MAP9}
\end{equation}
Applying the matrix inversion lemma \citep{kay1993fundamentals}:
\begin{equation}
	(A+BD^{-1}C)^{-1}=A^{-1}-A^{-1}B(D+CA^{-1}B)^{-1}CA^{-1}
\label{eq:inversion_lemma}
\end{equation}
Equation \ref{eq:MAP9} can be rewritten as:
\begin{equation}
	\hat{\z } = \left( \Sigma_{\h_t} + \Sigma_{\h_s} \right)^{-1} \left(\Sigma_{\h_t}\Interp_s\y +\Sigma_{\h_s}\Interp_t\x \right)
\label{eq:MAP10}
\end{equation}
Equation \ref{eq:MAP10} is the final form of the proposed Bayesian fusion model using a MAP estimate and assuming a noninformative prior of $ \z  $. Variance matrices $ \Sigma_{\h_t} $ and $ \Sigma_{\h_s} $ are parameters to be learned from the measurements.

\section{Equivalent models}
The above section presents a Bayesian fusion model to combine two sources of measurements. In the final formula, the prior distribution is omitted. The model is similar to some other models existing in the literature. The linear Gaussian model assumes that measurements follow Gaussian distributions to estimate the likelihood functions and prior probability. The final formula is slightly different from the Bayesian model assuming a non-informative prior. Another model is the generalized Millman formula \citep{shin2006generalized}, which is studied in the context of multi-sensor fusion. The model aims at combining information from various sensors. It becomes identical to the Bayesian fusion model when the cross-covariance between estimates are neglected. We briefly recall these models below.

\subsection{\label{subsec:Linear_Gaussian_model} Linear Gaussian model}
Let consider equations \ref{eq:Bayes11} and \ref{eq:Bayes12} as the two linear models of $ \z  $. The posterior distributions $ \distr(\z |\Interp_t\x ,\Sigma_{\h_t}) $ and $ \distr(\z |\Interp_s\y ,\Sigma_{\h_s}) $ are given in equations \ref{eq:Bayes21} and \ref{eq:Bayes22}. Distributions of the measurements $ p(\Interp_s\y ) = \distr(0,\Sigma_{\Interp_s\y }) $ and $ p(\Interp_s\y ) = \distr(0,\Sigma_{\Interp_s\y }) $ are also assumed Gaussian. Using Bayes' rule for these linear variables as described in \citet{bishop2006pattern}, appendix \ref{apx_BayesGausvars_1} derives the following formulas:
\begin{equation}
\begin{split}
		p(\Interp_t \x |\z ) &= \distr \left(\Interp_t \x |\Sigma_t \Sigma_{\h_t}^{-1} \z ,\Sigma_t \right) \nonumber\\
		p(\Interp_s \y |\z ) &= \distr \left(\Interp_s \y |\Sigma_s \Sigma_{\h_s}^{-1} \z ,\Sigma_s \right) \nonumber\\
		p(\z ) &= \distr \left(\z |0, \Sigma_{\z }\right)
\end{split}
\label{eq:MAP_linearGauss1}
\end{equation}
where:
\begin{equation}
\begin{split}
		\Sigma_t^{-1} &= \Sigma_{\h_t}^{-1} + \Sigma_{\Interp_t\x }^{-1} \nonumber\\
		\Sigma_s^{-1} &= \Sigma_{\h_s}^{-1} + \Sigma_{\Interp_s\y }^{-1} \nonumber\\
		\Sigma_{\z } &= \Sigma_{\h_t}+\Sigma_{\Interp_t\x } = \Sigma_{\h_s}+\Sigma_{\Interp_s\y }
\end{split}
\label{eq:MAP_linearGauss2}
\end{equation}
Using the above Gaussian distributions, a MAP estimate of $ \z  $ is:
\begin{eqnarray}
\begin{split}
	\hat{\z } &= \underset{\z }{argmax} \left\lbrace  p(\Interp_t\x |\z )p(\Interp_s\y |\z )p(\z ) \right\rbrace \nonumber\\
	&= \underset{\z }{argmin} \left\lbrace -\ln p(\Interp_t\x |\z ) -\ln p(\Interp_s\y |\z ) -\ln p(\z ) \right\rbrace
\end{split}
\label{eq:MAP_linearGauss3}
\end{eqnarray}
Taking the logarithm of above probabilities yields:
\begin{equation}
\begin{split}
	-2\ln p(\Interp_t\x |\z ) &= \left(\Interp_t\x -\Sigma_t \Sigma_{\h_t}^{-1}\z  \right)^{\mytrans} \Sigma^{-1}_t \left(\Interp_t\x -\Sigma_t \Sigma_{\h_t}^{-1}\z \right) \\
	&= \Mdist{\Sigma_t \Sigma_{\h_t}^{-1}\z -\Interp_t\x}{\Sigma_t}\\
	-2\ln p(\Interp_s\y |\z ) &=  \left(\Interp_s\y -\Sigma_s \Sigma_{\h_s}^{-1}\z  \right)^{\mytrans} \Sigma^{-1}_s \left(\Interp_s\y -\Sigma_s \Sigma_{\h_s}^{-1}\z \right)\\
	&= \Mdist{\Sigma_s \Sigma_{\h_s}^{-1}\z -\Interp_s\y}{\Sigma_s}\\
	-2\ln p(\z ) &= \z ^{\mytrans} \Sigma^{-1}_z \z  = \Mdist{\z}{\Sigma_{\z }}
\end{split}
\label{eq:MAP_linearGauss4}
\end{equation}
The cost function $ C(\z ) $ is defined as the sum of above terms:
\begin{equation}
	C(\z ) =\Mdist{\Sigma_t \Sigma_{\h_t}^{-1}\z -\Interp_t\x}{\Sigma_t}  + \Mdist{\Sigma_s \Sigma_{\h_s}^{-1}\z -\Interp_s\y}{\Sigma_s} + \Mdist{\z}{\Sigma_{\z }}
\label{eq:MAP_linearGauss5}
\end{equation}
 The gradient of $ C(\z ) $ is computed as:
\begin{equation}
\begin{split}
	\frac{\partial C(\z )}{2\partial \z } &= \Sigma^{-1}_t \left( \Sigma_t \Sigma_{\h_t}^{-1}\z -\Interp_t\x  \right) + \Sigma^{-1}_s \left( \Sigma_s \Sigma_{\h_s}^{-1}\z -\Interp_s\y  \right) +\Sigma^{-1}_{\z } \z  \nonumber\\
	&= \left( \Sigma_{\h_t}^{-1} + \Sigma_{\h_s}^{-1} + \Sigma^{-1}_{\z } \right) \z  - \left( \Sigma^{-1}_t\Interp_t\x  + \Sigma^{-1}_s\Interp_s\y  \right)
\end{split}
\label{eq:MAP_linearGauss6}
\end{equation}
The estimate $ \hat{\z } $ is obtained by setting this gradient to zero so that:
\begin{equation}
	\hat{\z }=\left( \Sigma_{\h_t}^{-1} + \Sigma_{\h_s}^{-1} + \Sigma^{-1}_{\z } \right)^{-1} \left(\Sigma^{-1}_t\Interp_t\x  +\Sigma^{-1}_s\Interp_s\y \right)
\label{eq:MAP_linearGauss7}
\end{equation}
This fusion formula has the same form of a weighted sum, with the weights are slightly different from the formula \ref{eq:MAP9}.

\subsection{Generalized Millman formula}
In the context of multisensor fusion, the Millman and Bar-Shalom-Campo formula is very common to combine the two estimates with either correlated or uncorrelated errors. This formula is generalized in \cite{shin2006generalized}, permitting to fuse multi-source measurements. This formula turns out to be similar in this context of Bayesian fusion when no prior knowledge is inserted into the model.

Let denote $ \left\lbrace \hat{\z }_1,...,\hat{\z }_K \right\rbrace  $ the K estimates of an unknown random vector $ \z  \in \R^n $. The objective of the fusion model is to find the best linear estimate via a set of coefficient matrices $ C_i $ (i=1,...,K):
\begin{equation}
\hat{\z }=\sum\limits_{i=1}^{K}{C_i\hat{\z }_i} \subjectto \sum\limits_{i=1}^{K}{C_i}=I_n
\end{equation}
The final form of GMF, as derived in appendix \ref{apx_BayesGausvars_2} is:
\begin{equation}
\hat{\z }=\sum\limits_{i=1}^{K}{C_i\hat{\z }_i} \subjectto \begin{cases}
\sum\limits_{i=1}^{K}{C_i}=I_n\\
\sum\limits_{i,j=1}^{K-1}{C_i} \left(\Sigma_{ij}-\Sigma_{iK}\right)+C_K\left(\Sigma_{Kj}-\Sigma_{KK}\right) = 0, \:\: j=1,...,K-1
\end{cases}
\end{equation}
In the case of fusing two estimates ($ K=2 $), this becomes the Bar-Shalom-Campo formula:
\begin{equation}
\hat{\z }=\left(\Sigma_{22}-\Sigma_{21}\right)\left(\Sigma_{11}+\Sigma_{22}-\Sigma_{12}-\Sigma_{21}\right)^{-1} \hat{\z }_1+\left(\Sigma_{11}-\Sigma_{12}\right)\left(\Sigma_{11}+\Sigma_{22}-\Sigma_{12}-\Sigma_{21}\right)^{-1}\hat{\z }_2
\end{equation}
If $ \hat{\z }_1 $ and $ \hat{\z }_2 $ are uncorrelated, i.e. $ \Sigma_{12}=\Sigma_{21}=0 $, the fusion formula becomes the Millman's formula
\begin{equation}
\hat{\z }=\Sigma_{22}\left(\Sigma_{11}+\Sigma_{22}\right)^{-1}\hat{\z }_1+\Sigma_{11}\left(\Sigma_{11}+\Sigma_{22}\right)^{-1}\hat{\z }_2
\end{equation}
which is identical to the proposed Bayesian fusion model with a non-informative prior on $ \z  $.

\section{Model simplifications}
Though providing the full theoretical estimate of $ \hat{\z } $, equation \ref{eq:MAP10} is impractical to use as is for several reasons. The full covariance matrices $ \Sigma_{\h_t} $ and $ \Sigma_{\h_s} $, representing all the sources of correlations in space and time, cannot be estimated from only the measurements $ \x  $ and $ \y $. This is because the unknown $ \h_t $ and $ \h_s $ are only accessible at the measured positions in space and time. Also, the covariance matrices of size $ \dimsh \dimth \times \dimsh \dimth $ are very large, making them very difficult to accurately estimate and to invert. Additional assumptions on the shape of $ \Sigma_{\h_t} $ and $ \Sigma_{\h_s} $ are necessary.

\subsection{\label{subsec:simplified_fusion_model} Simplified covariance matrices}
A common approach to simplify the formula \ref{eq:MAP10} is to assume $ \Sigma_{\h_s} $ and $ \Sigma_{\h_t} $ be diagonal. This simplification implies the independence between all elements of $ \h_t $ and $ \h_s $. The resulting simplified version of formula \ref{eq:MAP10} is point-wise:
\begin{equation}
	\hat{\z }[i,t]=\frac{\mybold{\sigma}^2_{\h_s}[i,t]}{\mybold{\sigma}^2_{\h_s}[i,t]+\mybold{\sigma}^2_{\h_t}[i,t]}\Interp_t\x [i,t]+\frac{\mybold{\sigma}^2_{\h_t}[i,t]}{\mybold{\sigma}^2_{\h_s}[i,t]+\mybold{\sigma}^2_{\h_t}[i,t]}\Interp_s\y [i,t]
\label{eq:MAP_simplified}
\end{equation}
where  $ [i,t] $ is the index of each point in space and time respectively. The variances $ \mybold{\sigma}^2_{\h_s} $ and $ \mybold{\sigma}^2_{\h_t} $ are functions of each position in space and time. Their estimation is detailed in the next section. Equation \ref{eq:MAP_simplified} will be used to reconstruct HTHS data. As a weighted average, it proposes a compromise estimate from the measurements. With a symmetrical form in space and time, the model uses information from both measurements to correct large scales reconstruction and recover partial information at smaller scales.

Assuming diagonal covariance matrices, the linear Gaussian model in equation \ref{eq:MAP_linearGauss7} is simplified as:
\begin{equation}
	\begin{split}
	\hat{\z }[i,t] =& \cfrac{\sigma^2_{\h_s}[i,t]+\cfrac{\sigma^2_{\h_t}[i,t]\sigma^2_{\h_s}[i,t]}{\sigma^2_{\Interp_t\x }[i,t]}}{\sigma^2_{\h_t}[i,t]+\sigma^2_{\h_s}[i,t]
	+ \cfrac{\sigma^2_{\h_t}[i,t]\sigma^2_{\h_s}[i,t]}{\sigma^2_{\z }[i,t]}}\Interp_t\x [i,t] \\
	+& \cfrac{\sigma^2_{\h_t}[i,t] +\cfrac{\sigma^2_{\h_t}[i,t]\sigma^2_{\h_s}[i,t]}{\sigma^2_{\Interp_s\y }[i,t]}}{\sigma^2_{\h_t}[i,t]+\sigma^2_{\h_s}[i,t]+\cfrac{\sigma^2_{\h_t}[i,t]\sigma^2_{\h_s}[i,t]}{\sigma^2_{\z }[i,t]}}\Interp_s\y [i,t]
	\end{split}
	\label{eq:LinearGauss_simplified}
\end{equation}
The simplified linear Gaussian model slightly modifies formula \ref{eq:MAP_simplified} by introducing the three terms: $ \cfrac{\sigma^2_{\h_t}[i,t]\sigma^2_{\h_s}[i,t]}{\sigma^2_{\z }[i,t]} $, $ \cfrac{\sigma^2_{\h_t}[i,t]\sigma^2_{\h_s}[i,t]}{\sigma^2_{\Interp_s\y }[i,t]} $ and $ \cfrac{\sigma^2_{\h_t}[i,t]\sigma^2_{\h_s}[i,t]}{\sigma^2_{\Interp_t\x }[i,t]} $. These three terms are essentially small, since variances of $ \h_s $ and $ \h_t $ alone are relatively small compared to the variances of $ \z $, $ \Interp_s\y $ and $ \Interp_t\x $ due to the power law spectrum of turbulence.

\subsection{\label{subsec:statistical_parameters_estimation} Statistical parameters estimation}
Let $\Z$, $ \X $, $ \Y $, $ \mathbf{H}_t $, $ \mathbf{H}_s $, $ \mathbf{\Gamma}_{\h_t} $ and $ \mathbf{\Gamma}_{\h_s} $ be the matrix forms of $ \z  $, $ \x  $, $ \y  $, $ \h_t $, $ \h_s $, $\mybold{\sigma}^2_{\h_t} $ and $\mybold{\sigma}^2_{\h_s} $ respectively.  $\Z$, $ \mathbf{H}_t $, $ \mathbf{H}_s $, $ \mathbf{\Gamma}_{\h_t} $ and $ \mathbf{\Gamma}_{\h_s} $ are of size $ \dimth\times \dimsh $, while $ \X $ and $ \Y $ are of size $ \dimtl \times \dimsh $ and $ \dimth\times \dimsl $. $ \mathbf{\Gamma}_{\h_t} $ and $ \mathbf{\Gamma}_{\h_s} $ are matrices of empirical variances, which are functions of space and time $ (i,t) $.

Variance matrices are estimated from $ \mathbf{H}_t $ and $ \mathbf{H}_s $, which are available at the measurements positions only. We use:
\begin{align}
	\Sub_t\mathbf{H}_s &= \X- \Interp_s\Sub_s\X \\
	\Sub_s\mathbf{H}_t &= \Y- \Interp_t\Sub_t\Y
\end{align}
where $ \Sub_t $ subsamples in time from $ \dimth  $ to $ \dimtl  $ time steps, and $ \Sub_s $ subsamples in space from $ \dimsh $ to $ \dimsl $ points. These $ \dimtl  $ instants and $ \dimsl $ positions are the same as for LTHS and HTLS measurements. Since the flow is approximately stationary and spatial interpolation is independent of time, $ \mathbf{\Gamma}_{\h_s} [i,t] $ becomes $ \mathbf{\Gamma}_{\h_s} [i] $, a function of spatial locations only. These variances are estimated by averaging over all time steps:
\begin{equation}
	\mathbf{\Gamma}_{\h_s} [i] = \cfrac{1}{\dimtl }\sum_{t=1}^{\dimtl } \left( \X[i,t]- \Interp_s\Sub_s\X[i,t] \right)^2
	\label{eq:variance_estimation_1}
\end{equation}
Variance in $ \mathbf{\Gamma}_{\h_t}$ is a function of distances $ \tau $ to the previous LTHS time step only, where $ \tau/\delta t=0,1,2,...,\dimth/\dimtl $, and $ \delta t $ is the time lag between two consecutive HTHS time steps. $ \mathbf{\Gamma}_{\h_t} $ becomes a function of space and $ \tau $ only, i.e. $ \mathbf{\Gamma}_{\h_t} [i,\tau] $. It is estimated by averaging over $ \dimtl $  blocks (of $ \dimth/\dimtl $ snapshots) bounded by two consecutive LTHS instants:
\begin{equation}
	\mathbf{\Gamma}_{\h_t} [i,\tau] = \frac{1}{\dimtl}\sum_{t_\knot}\left( \Y(i,t_\knot+\tau)- \Interp_t\Sub_t\Y(i,t_\knot+\tau)\right)^2
	\label{eq:variance_estimation_2}
\end{equation}
where $ t_\knot/\delta t=1,1+\dimth/\dimtl,1+2\dimth/\dimtl,...,1+(\dimtl -1)\dimth/\dimtl $. Along the homogeneous direction of the flow, $ \mathbf{\Gamma}_{\h_s} [i] $ and $ \mathbf{\Gamma}_{\h_t} [i,\tau] $ can be also averaged over all blocks defined by the four neighboring HTLS measurements (see figure \ref{fig:element_block}). The variances are then a function of positions only in non-homogeneous direction and relative distances to the four closest HTLS sensors. These estimated variances are rearranged into a vector form $\mybold{\sigma}^2_{\h_t}[i,t] $ and $\mybold{\sigma}^2_{\h_s}[i,t] $ to complete the fusion model using the simplified formula in equation \ref{eq:MAP_simplified}.

\section{Applying the Bayesian fusion model to DNS data of isotropic turbulence}
The Bayesian fusion model is applied to combine HTLS and LTHS measurements from simulated data of an isotropic turbulence as described in section \ref{sec:data_isotropic}. It is noticed that the streamwise direction is considered as time dimension, while the two spatial directions are spanwise and vertical. Both Bayesian fusion model (BF) with non-informative prior using formula \ref{eq:MAP_simplified} and the linear Gaussian model (LG) using formula \ref{eq:LinearGauss_simplified} are tested.

\subsection{Impact of subsampling ratios}
\label{subsubsec:impacts_of_subsampling_ratios}

A total of seven cases gathered in table \ref{tab:fusion_various_cases} are investigated. These cases correspond to different subsampling ratios. In space, the ratios $ \sqrt{\dimsh /\dimsl } = 3, 4 $ and $ 6 $ are equally applied in spanwise and vertical direction, corresponding to resolutions of HTLS fields $ \dimsl = 32 \times 32 $, $ 24 \times 24 $ and $ 16 \times 16 $ respectively. Subsampling ratios $ \dimth/\dimtl $ in time (streamwise) dimension are 4 ($ \dimtl = 37 \times 24$), 6 ($ \dimtl = 37 \times 16$) and 8 ($ \dimtl = 37 \times 12 $). The energy losses due to these subsamplings are estimated as in section \ref{sec:data_isotropic}.

These seven cases can be categorized into three groups. Case 1 and case 2 are when the subsampling ratios in time are more critical than those in space, reversely in case 3 and case 4. In such cases, either HTLS or LTHS measurements are more informative than the other. The interpolation from the measurements with smaller ratio is more accurate. In the last three cases, the subsamplings in space and time are balanced as the energy losses are approximately equal. The fusion model in such cases is expected to out-perform both interpolations by combining useful information from both sources.

The fusion model uses equation \ref{eq:MAP_simplified} to reconstruct fully resolved velocities $ \myhat{\z } $ from $ \x $ and $ \y $. Reconstructed fields are compared with the original DNS data via the normalized root mean square error (NRMSE) (equation \ref{eq:NRMSE}). The field $\z $ is more or less difficult to estimate depending on the considered instant and position with respect to available measurements. To qualify, two types of NRMSE, the mean NRMSE $ \overline{\epsilon} $ and the maximum NRMSE $ \epsilon_{max} $, are estimated using equation \ref{eq:NRMSE}. $ \overline{\epsilon} $ is estimated over the set of points $ \varmathbb{J}$ including all space-time positions. It represents how far the reconstructed field departs from ground truth in order to evaluate reconstruction accuracy. $ \epsilon_{max} $ is estimated using all blocks (in time and in spanwise directions) between HTLS sensors (see figure \ref{fig:element_block}). The set $ \varmathbb{J} $ includes all points at the local coordinate $ (\Delta y/2, \Delta z/2, \Delta T/2 ) $. Errors of single interpolations from either measurements of HTLS or LTHS are also estimated for comparisons.

\begin{table}
	\caption{\label{tab:fusion_various_cases}
	Configuration parameters of seven testing cases. The subsampling ratios of HTLS measurements are $ \sqrt{\dimsh /\dimsl } $ and equal in both spatial directions. The ratios of LTHS measurements in time are $ \dimth/\dimtl $. The normalized energy losses in space $\Delta\kappa_s$ and in time $\Delta\kappa_t$ are defined in equation \ref{eq:RMS_losses}.}
	\vspace{.5cm}
	\centering
	\begin{tabular}{ccS[table-format=1.0]S[table-format=1.0]cS[table-format=1.2]S[table-format=1.2]}
		\toprule
		\multirow{2}{*}{Case}&\multicolumn{1}{c}{}&\multicolumn{2}{c}{Subsampling ratios}&\multicolumn{1}{c}{}&\multicolumn{2}{c}{Energy loss}\\
		\cmidrule{3-4} \cmidrule{6-7}
		 & & {$\sqrt{\dimsh /\dimsl }$} & $\dimth/\dimtl$ & & {$\Delta\kappa_s(\%)$} & {$\Delta\kappa_t (\%)$}\\
		\midrule
		1 &  & 3  & 8 & & 1.03 & 6.53 \\ %\addlinespace
		2 &  & 4  & 8 & & 2.63 & 6.53 \\ %\addlinespace
		3 &  & 6  & 4 & & 7.29 & 1.23 \\ %\addlinespace
		4 &  & 6  & 6 & & 7.29 & 1.23 \\ %\addlinespace
		\myrowcolour
		5 &  & 3  & 4 & & 1.03 & 1.23 \\ %\addlinespace
		\myrowcolour
		6 &  & 4  & 6 & & 2.63 & 3.56 \\ %\addlinespace
		\myrowcolour
		7 &  & 6  & 8 & & 7.29 & 6.53 \\ %\addlinespace
		\bottomrule
	\end{tabular}
\end{table}

\begin{table}
\caption{\label{tab:fusion_average_errors}
NRMSEs of all scales reconstruction errors for the seven cases presented in table \ref{tab:fusion_various_cases}. $\overline{\epsilon}$ and $\epsilon_{max}$ are the average and maximum NRMSE defined by equation \ref{eq:NRMSE}. $\overline{\epsilon}$ is averaged over all fields, while $\epsilon_{max}$ is computed for the most difficult positions in space and time (the most remote from all nearby measurements at local coordinates $ (\Delta y/2, \Delta z/2, \Delta T/2) $, see figure \ref{fig:element_block}). These errors are shown for reconstructed fields by spatial interpolation $ \Interp_s \y $, time interpolation $ \Interp_t \x $, linear Gaussian model (LG) using formula \ref{eq:LinearGauss_simplified} and Bayesian fusion model (BF) using formula \ref{eq:MAP_simplified}. The smallest errors in each cases are boldfaced.}
\vspace{.5cm}
\centering
	\begin{tabular}{ccccccccccc}
		\toprule \multirow{2}{*}{Case}&\multicolumn{1}{c}{}&\multicolumn{4}{c}{$\overline{\epsilon}$}&\multicolumn{1}{c}{}&\multicolumn{4}{c}{$\epsilon_{max}$}\\
		\cmidrule{3-6} \cmidrule{8-11}
		 & & {$\Interp_s\y $} & {$\Interp_t\x $} & {LG} & {BF} & & {$\Interp_s\y $} & {$\Interp_t\x $} & {LG} & {BF}\\
		\midrule
		1 & & 0.19 & 0.31 & {\textbf{0.16}} & {\textbf{0.16}} & & 0.22 & 0.42 &  {0.23} &  {\textbf{0.22}}\\
		2 & & 0.28 & 0.31 & {\textbf{0.21}} & {\textbf{0.21}} & & 0.35 & 0.42 & {0.33} &  {\textbf{0.32}}\\
		3 & & 0.43 & 0.13 & {\textbf{0.13}} & {\textbf{0.13}} & & 0.53 & {\textbf{0.19}} & {0.20} &  {\textbf{0.19}}\\
		4 & & 0.43 & 0.23 & {0.22} & {\textbf{0.21}} & & 0.53 & 0.32 & {0.33} &  {\textbf{0.32}}\\
		\myrowcolour
		5 & & 0.19 & 0.13 & {\textbf{0.11}} & {\textbf{0.11}} & & 0.22 & 0.19 &  {\textbf{0.17}} &  {\textbf{0.17}}\\
		\myrowcolour
		6 & & 0.28 & 0.23 & {\textbf{0.18}} & {\textbf{0.18}} & & 0.35 & 0.28 &  {\textbf{0.26}} &  {\textbf{0.26}}\\
		\myrowcolour
		7 & & 0.43 & 0.31 & {0.27} & {\textbf{0.26}} & & 0.52 & 0.42&  {0.41} &  {\textbf{0.40}} \\ \bottomrule
	\end{tabular}
\end{table}

Table \ref{tab:fusion_average_errors} shows reconstruction errors in all cases. In cases 1 and 2, the energy losses due to subsampling in time are much higher than in space, and vice-versa in cases 3 and 4. In these cases, the model gives similar errors compared to the best interpolation, with smaller $ \overline{\epsilon} $ and comparable $ \epsilon_{max} $. In cases 5 to 7, the losses are due to both the subsamplings in space and time in a balanced manner. The two proposed models give similar accuracy, with $ \overline{\epsilon} $ reduced by 15$ \% $ to 25$ \% $ and $ \epsilon_{max} $ reduced by 5$ \% $ to 15$ \% $ compared to the best single interpolation.

Improvements were expected from the weighted average in equation \ref{eq:MAP_simplified} since it uses variances $ \mybold{\sigma}^2_s[i,t] $ and $\mybold{\sigma}^2_t[i,t] $ as parameters of the flow physics, and $ \Interp_t\x  $ and $ \Interp_s\y  $ as the specific flow information. The proposed model imposes the reconstruction to be consistent with measurements at nearby positions and proposes compromise estimates elsewhere. Simple interpolations use either HTLS or LTHS measurements only, losing information from the other sources.

\subsection{Large and small scales reconstruction}
\label{subsubsec:large_and_small_scales_reconstruction}
\begin{table}
\centering
\caption{\label{tab:results_largescales}
Average NRMSEs $ \overline{\epsilon} $ of large and small scales (same notations in table \ref{tab:fusion_average_errors}).}
\vspace{.5cm}
	\begin{tabular}{ccccccccccc}
		\toprule \multirow{2}{*}{Case}&\multicolumn{1}{c}{}&\multicolumn{4}{c}{Large scales}&\multicolumn{1}{c}{}&\multicolumn{4}{c}{Small scales}\\
		\cmidrule{3-6} \cmidrule{8-11}
		 & & {$\Interp_s\y $} & {$\Interp_t\x $} & {LG} & {BF} & & {$\Interp_s\y $} & {$\Interp_t\x $} & {LG} & {BF}\\
		\midrule
		\myrowcolour
		5 & & 0.16 & 0.12 & {\textbf{0.09}} & {\textbf{0.09}} & & 0.98 & 0.67 & {\textbf{0.64}} & {\textbf{0.64}}\\
		\myrowcolour
		6 & & 0.23 & 0.19 & {\textbf{0.14}} & {\textbf{0.14}} & & 0.98 & 0.80 & {0.71} & {\textbf{0.70}}\\
		\myrowcolour
		7 & & 0.34 & 0.22 & {0.20} & {\textbf{0.19}} & & 0.98 & 0.81 &  {\textbf{0.72}} &  {\textbf{0.72}}\\
		\bottomrule
	\end{tabular}
\end{table}

In cases 1 to 4, both fusion models perform as the best interpolation with small improvements. This is expected since one measurement of HTLS or LTHS is much better resolved than the other. Cases 5 to 7  are the most interesting since energy losses due to subsampling in space and time are comparable. The model brings complementary information from both measurements and improves the reconstruction.

The reconstructions of large and small scales are investigated in details for these three cases. Spatial 2D ideal Fourier filters $ \LPF_s $ are used to separate large scales from small scales. These filters define different cutoff wave numbers based on the subsampling ratios. The reconstructed large scales by all methods are compared to the reference $ \LPF_s\z $. Small scales are estimated using $\Interp-\LPF_s $ where $ \Interp $ is the identity matrix. Table~\ref{tab:results_largescales} shows NRMSEs estimated using equation \ref{eq:NRMSE} but normalized by the RMS of either $ \LPF_s\z $ or $(\Interp-\LPF_s)\z  $.

The proposed fusion models recover part of small scales from complementary measurements. They correct also large-scale information compared to the spatial interpolation, since subsampling degrades those scales and introduces some aliasing effects. Results are shown in table \ref{tab:results_largescales} for the most favorable cases. Fusion models give smallest error in all cases and at both large and small scales. Errors of large scale reconstructions are reduced by 20 to 50 $ \% $ compared to the best single interpolation, while these improvements are smaller for small scales reconstruction (5 to 25 $ \% $). Note that the reconstruction error remains very high for small scales. Also, a good reconstruction of large scales is surprisingly challenging despite the theoretical claims that large scales should be accurately reconstructed by interpolations.

\section{Concluding remarks}
This chapter proposes a Bayesian fusion model using a MAP estimate to reconstruct high resolution velocities of a turbulent channel flow from low resolution measurements in space and time. It searches for the most probable field given available measurements. This approach yields a simple but efficient weighted average formula in equation~(\ref{eq:MAP_simplified}). Weighting coefficients are learned from measurements and encode the physics of the flow. The fusion of available measurements improves the large scale reconstruction and recovers some details at small-scale.

Numerical experiments using a DNS database of isotropic turbulence at a moderate Reynolds number illustrate the efficiency and robustness of the proposed method. Low resolution measurements are extracted to learn model parameters, while original data are used as the ground truth to estimate reconstruction errors. The model is tested in various cases with different subsampling ratios. Results are compared against cubic spline interpolations. Bayesian fusion always produces the most accurate reconstruction. The best results are obtained when missing spatial and temporal information are of the same order of magnitude. The fusion model provides a better large scale reconstruction compared to the interpolation. Small scales below the cutoff frequency are more difficult to recover. The error remains high, since only a certain amount of small scale details are reconstructed. This is one more reason in favor of using either some relevant physical model or some  sophisticated learning procedure to "guess" small scales.

Next chapter will compare the performances of various methods we have presented. The analyses are on both the two datasets of isotropic turbulence and channel flow.


    \filetitle{!log\_variables}{Restrict the steady state of some of the variables to be positive}{sstatelang/logvariables}

	\paragraph{Syntax}

 \begin{verbatim}
 !log_variables
     Variable_Name, Variable_Name
     Variable_Name
 \end{verbatim}

 \paragraph{Syntax with inverted list}

 \begin{verbatim}
 !log_variables
     !all_but
     Variable_Name, Variable_Name
     Variable_Name
 \end{verbatim}

 \paragraph{Description}

 \paragraph{Example}