\chapter{BENCHMARK OF EFFECTIVENESS OF IMAGE ATTACK METHODS AGAINST DEFENDED POLICIES}
\label{sec:exp3}

This section reports the results of the last of the three experiments included in this thesis. It benchmarks the effectiveness of different image attacks used by two different policy attacks to decrease victim's rewards. The victim models have been protected with different defense mechanisms.

\section{Benchmark of the effectiveness of image attack methods against defended policies}
The last experiment included in this work consists of a benchmark to evaluate the robustness of defended policies trained with DQN on the environment of Pong. All policies have been attacked with the white-box image attack FGSM under a fixed threat model. This work compares the {\it reward vs perturbation} and {\it accuracy vs perturbation} curves of the defended victim policies by attacking them over increasing values of perturbations’ strength. Policies have been defended with the following defense methods:
\begin{itemize}
    \item FGSM adversarial training;
    \item PGD adversarial training;
    \item JPEG conversion;
    \item Feature squeezing (Bit squeezing);
    \item No defense.
\end{itemize}
Note that in the benchmark is also included an undefended version of the victim policy so as to have a comparison of the defended policies with the undefended one.
Policies have been attacked with three different white-box image attacks methods including:
\begin{itemize}
    \item FGSM;
    \item PGD;
    \item MI (Momentum Iterative);
    \item No attack.
\end{itemize}
The {\it no attack} method which corresponds to evaluating the target policy without attacking it at all has been introduced so to compare the effect of each attack method with respect to not attacking at all. The curves have been generated attacking with both untargeted and targeted attacks and their corresponding results are reported in two different sections.

\subsection{Experiments settings}
\subsubsection{Defenses hyper-parameters}
Two of the four defenses that have been included in the benchmark consist of adversarial training with FGSM and PGD respectively. The first policy is the same policy trained with DQN used in the experiment in section \ref{sec:exp2}. The policy protected with PGD adversarial training has been trained crafting adversarial perturbations with 10 iterations of PGD. Besides the type of attack and the number of iterations, there is no difference between the way adversarial training has been applied. Perturbations have been crafted under \(l_\infty\) norm and \(\epsilon\) fixed to 0.01.

The third defense that has been implemented is a simple method that converts each observation to JPEG format before feeding it to the network. JPEG images retain 20\% of the quality of the original observations.

The fourth method is known as feature squeezing and it consists of reducing the number of features of each pixel for each observed image. Since we are dealing with gray-scale images, what we are going to reduce is the number of bits to represent each pixel, thus we may sometimes refer to this method as bit squeezing. More specifically, it reduces the number of bits from 8 to 5, thus lowering the number of features from \(2^8\) to \(2^5\). This method shrinks the number of features by 80\% with respect to the original observations.

In this benchmark we have evaluated a powerful defense method such as adversarial training and two simple defenses such as JPEG conversion and feature squeezing so to test defenses with different levels of effectiveness.

\subsubsection{Image adversarial attacks settings}
This section reports the value of the hyper-parameters adopted in the image attacks included in the benchmark. Note that this time the perturbation strength \(epsilon\) is variable. The first attack, FGSM, has already been employed in the previous experiments and its hyper-parameters are maintained the same for this work. The two iterative methods PGD and MI execute 10 iterations each attack and each iteration refines the adversarial perturbations with \(epsilon_i\) (attack step size) of \(\epsilon/5\).

The main reason why we have chosen the above-mentioned attacks is that they all work under targeted and untargeted settings and they support the three most common distance norms, namely \(l_\infty, l_1, l_2\). They also explicitly define the hyper-parameter \(\epsilon\) to limit the size of the adversarial perturbations. For example, adversarial examples crafted with C\&W method are not easily comparable in this benchmark since the method doesn't have a hyper-parameter \(\epsilon\) that can be fine-tuned. Moreover, those attacks are well-studied and commonly used in research papers, thus proving to work well in practice and worth being evaluated.

\subsubsection{Threat model}
All methods included in this benchmark are evaluated under both untargeted and targeted settings. Adversarial examples are crafted using the \(l_\infty\) distance norm. Their perturbations' strength \(\epsilon\) ranges from 0 to 0.05 included with a step size of 0.005. Curves are generated by computing the rewards of the policy under attack and the success rate of the malicious attacks according to the value of \(\epsilon\) defined in correspondence of each point.

\subsection{Benchmark of untargeted white-box attacks}
The charts relative to the {\it reward vs perturbation} and {\it accuracy vs perturbation} curves obtained under untargeted settings are shown in figures \ref{figure:untargeted-rew} and \ref{figure:untargeted-suc} respectively. When attacking the policy protected with FGSM adversarial training, MI is the attack that results most effective followed by PGD and FGSM. It is interesting to note that although adversarial training has been conducted with \(\epsilon\)=0.01, adversarial training can still protect quite well against adversarial observations crafted with higher values of perturbation strength. The defense is completely broken for noise generated with \(\epsilon\) greater than 0.03. Conversely, when attacking PGD adversarial training, FGSM and PGD perform the best but the difference between the three attack methods is never too significant. PGD struggles to reduce the reward of the agent against JPEG filter and bit squeezing with respect to the other two attack methods which perform very similar. FGSM is also the only attack that can't reduce the average reward to \(-20\) when attacking the undefended policy. When comparing the success rate of the three methods, PGD is the attack the overall performs the best and FGSM is the worst. In particular, PGD adversarial training seems to be more robust against FGSM, and bit squeezing is more vulnerable against PGD attack. When attacking FGSM adversarial training and JPEG filer, all the attacks show a similar success rate.

It is interesting to note that when attacking JPEG filter and bit squeezing, all attacks seem to break the defense right after increasing the strength of the perturbations to be greater than 0.01. In fact, after this threshold, their curves drop to -20 points and the attack accuracy skyrockets to be 100\%. The two defenses based on adversarial training are better at improving the robustness of their policies. As we can see from the charts, the average rewards still begin decreasing when \(\epsilon\) is greater than 0.02 or 0.03 but both curves approach -20 with a more gentle descent. Conversely, the attack accuracy against the same defenses steadily increases over the whole perturbations range but the rise is never too steep.

\begin{figure}
  \centering
  \subcaptionbox{FGSM adversarial training\label{fig:subfig-a}}
    {\includegraphics[width=0.49\linewidth]{images/exp3/untargeted/FGSM_AdvTr.jpg}}
  \subcaptionbox{PGD adversarial training\label{fig:subfig-b}}
    {\includegraphics[width=0.49\linewidth]{images/exp3/untargeted/PGD_AdvTr.jpg}}
  \subcaptionbox{JPEG filter\label{fig:subfig-c}}
    {\includegraphics[width=0.49\linewidth]{images/exp3/untargeted/JPEG_Filter.jpg}}
  \subcaptionbox{Bit squeezing\label{fig:subfig-d}}
    {\includegraphics[width=0.49\linewidth]{images/exp3/untargeted/Bit_Squeezing.jpg}}
  \subcaptionbox{No Defense\label{fig:subfig-e}}
    {\includegraphics[width=0.49\linewidth]{images/exp3/untargeted/No_Defense.jpg}}
  \caption{The {\it reward vs perturbation} curves of 5 DQN policies trained to play Pong and defended with 5 different defense methods (including no defense) under the \(l_\infty\) norm attacking with increased perturbations strength. Perturbations have been crafted in an untargeted way.}
  \label{figure:untargeted-rew}
\end{figure}

\begin{figure}
  \centering
  \subcaptionbox{FGSM adversarial training\label{fig:subfig-a}}
    {\includegraphics[width=0.49\linewidth]{images/exp3/untargeted/FGSM_AdvTr_succ.jpg}}
  \subcaptionbox{PPO adversarial training\label{fig:subfig-b}}
    {\includegraphics[width=0.49\linewidth]{images/exp3/untargeted/PGD_AdvTr_succ.jpg}}
  \subcaptionbox{JPEG filter\label{fig:subfig-c}}
    {\includegraphics[width=0.49\linewidth]{images/exp3/untargeted/JPEG_Filter_succ.jpg}}
  \subcaptionbox{Bit squeezing\label{fig:subfig-d}}
    {\includegraphics[width=0.49\linewidth]{images/exp3/untargeted/Bit_Squeezing_succ.jpg}}
  \subcaptionbox{No defense\label{fig:subfig-e}}
    {\includegraphics[width=0.49\linewidth]{images/exp3/untargeted/No_Defense_succ.jpg}}
  \caption{The {\it accuracy vs perturbation} curves of 5 DQN policies trained to play Pong and defended with 5 different defense methods (including no defense) under the \(l_\infty\) norm attacking with increased perturbations strength. Perturbations have been crafted in an untargeted way.}
  \label{figure:untargeted-suc}
\end{figure}

\subsection{Benchmark of targeted white-box attacks}
The charts relative to the {\it reward vs perturbation} and {\it accuracy vs perturbation} curves obtained under targeted settings are shown in figures \ref{figure:targeted-rew} and \ref{figure:targeted-suc} respectively. For each observation, the targeted action that has been selected is the action with the lowest probability. Hence, it is equal to adopt strategically-timed attack attacking all the possible observations. In terms of reward, FGSM is the attack that performs the worst against the two adversarial training-based defenses while the other two attacks obtain similar results. As for JPEG filter defense, the chart doesn't present any substantial difference among the curves of the evaluated attack methods. PGD is also the best attack to break bit squeezing defense. All curves begin to drop down for perturbations greater than 0.01 and the descent is steeper for JPEG filter and bit squeezing. As it was for the untargeted case, the value of the curves of the adversarial training-based defenses decreases more gradually and eventually reaches the bottom when \(\epsilon\) gets close to 0.03 or 0.04. The difference in terms of performance between adversarial training and the other two defenses is more significant when comparing their success rate curves. In fact, in the perturbations' strength interval taken into examination, targeted attacks against adversarial training protected policies work at most 20\% of the times while attacks against the other two defenses have a success rate of 100\% for \(\epsilon\) greater than 0.01. Under targeted settings, FGSM is the only attack whose accuracy never reaches 100\% against the undefended policy. It also shows an inferior performance against all the other defenses.

\begin{figure}
  \centering
  \subcaptionbox{FGSM adversarial training\label{fig:subfig-a}}
    {\includegraphics[width=0.49\linewidth]{images/exp3/targeted/FGSM_AdvTr.jpg}}
  \subcaptionbox{PPO adversarial training\label{fig:subfig-b}}
    {\includegraphics[width=0.49\linewidth]{images/exp3/targeted/PGD_AdvTr.jpg}}
  \subcaptionbox{JPEG filter\label{fig:subfig-c}}
    {\includegraphics[width=0.49\linewidth]{images/exp3/targeted/JPEG_Filter.jpg}}
  \subcaptionbox{Bit squeezing\label{fig:subfig-d}}
    {\includegraphics[width=0.49\linewidth]{images/exp3/targeted/Bit_Squeezing.jpg}}
  \subcaptionbox{No defense\label{fig:subfig-e}}
    {\includegraphics[width=0.49\linewidth]{images/exp3/targeted/No_Defense.jpg}}
  \caption{The {\it reward vs perturbation} curves of 5 DQN policies trained to play Pong and defended with 5 different defense methods (including no defense) under the \(l_\infty\) norm attacking with increased perturbations strength. Perturbations have been crafted in a targeted way.}
  \label{figure:targeted-rew}
\end{figure}

\begin{figure}
  \centering
  \subcaptionbox{FGSM adversarial training\label{fig:subfig-a}}
    {\includegraphics[width=0.49\linewidth]{images/exp3/targeted/FGSM_AdvTr_succ.jpg}}
  \subcaptionbox{PPO adversarial training\label{fig:subfig-b}}
    {\includegraphics[width=0.49\linewidth]{images/exp3/targeted/PGD_AdvTr_succ.jpg}}
  \subcaptionbox{JPEG filter\label{fig:subfig-c}}
    {\includegraphics[width=0.49\linewidth]{images/exp3/targeted/JPEG_Filter_succ.jpg}}
  \subcaptionbox{Bit squeezing\label{fig:subfig-d}}
    {\includegraphics[width=0.49\linewidth]{images/exp3/targeted/Bit_Squeezing_succ.jpg}}
  \subcaptionbox{No defense\label{fig:subfig-e}}
    {\includegraphics[width=0.49\linewidth]{images/exp3/targeted/No_Defense_succ.jpg}}
  \caption{The {\it accuracy vs perturbation} curves of 5 DQN policies trained to play Pong and defended with 5 different defense methods (including no defense) under the \(l_\infty\) norm attacking with increased perturbations strength. Perturbations have been crafted in a targeted way.}
  \label{figure:targeted-suc}
\end{figure}

\subsection{Findings}
This section shows the findings relative to the benchmark evaluated in this work. In particular, it makes some conclusions about the importance of this experiment, the differences between attacking under targeted and untargeted settings and it provides more insights about the evaluated defenses and their robustness.

\subsubsection{Importance of selecting the most appropriate image attack method}
Policy attacks based on image observations rely on image attacks to craft effective perturbations to fool their target agent. Hence, selecting the most effective image attack that performs better against a certain policy trained with a certain algorithm on a specific environment is a fundamental step to perform effective policy attacks.  In addition, different image attack methods might work better or worse against specific defense methods that protect the target policy. Thus, there are many variables that can condition the effectiveness of a policy attack. Also choosing the most appropriate threat model contributes to enhance the performance of the attacks and craft the most effective or imperceptible adversarial observations. For instance, in order to craft imperceptible adversarial examples, it is useful to know which minimal amount of \(\epsilon\) is enough to attack policy \(P\) defended with defense \(D\) on environment \(E\) or, given a fixed \(\epsilon\), it might be useful to find which attack shows the highest attack success rate or that reduces the agent's reward for the highest amount. Thus, reinforcement learning policies that take as input environment observations should have their robustness evaluated not only against policy attacks but also against image attack methods.

\subsubsection{Targeted and untargeted image attacks}
In this section, we are going to analyze the main differences that emerge when attacking under targeted and untargeted scenarios. Targeted attacks seem to be better at reducing the agent's reward but the probability that their attacks are executed correctly is significantly lower than their untargeted counterpart. However, they are still effective since most of the failed attacks, although they fail to fool the agent to select the selected action, they still corrupt the observations enough to fool the agent to take a random sub-optimal action. However, given a similar attack accuracy, targeted attacks can easily outperform untargeted attacks. For example, given the charts relative to the undefended policy, FGSM has 100\% accuracy under untargeted attack scenarios and a little more than 70\% accuracy under targeted settings. Nevertheless, its average reward for the untargeted case is a few points higher than its corresponding average reward obtained in the targeted scenario. This phenomenon is probably due to the fact that targeted attacks are more lethal and when they succeed they provoke greater damage to the victim policy. It is also worth noting that, in terms of accuracy, FGSM seems to suffer a loss in performance when attacking under targeted settings with respect to its targeted version. Overall, MI is the most effective attack under both settings.

\subsubsection{Robustness of the evaluated defenses}
Among the evaluated policies, the most robust one seems to be the one defended with the two adversarial training-based methods, in fact, they offer great protection under both attack settings. PGD adversarial training is perhaps a little better than FGSM when comparing the success rate of malicious attacks. JPEG filter and feature squeezing reduce the quality of the input observations with the goal to smooth eventual malicious perturbations. The more the quality reduction, the more perturbations would be filtered out. However, an excessive quality reduction would also negatively affect the agent's performance. Hence, when those methods are used in practice, some attention should be put into finding the right tradeoff between performance and robustness. As we can see from table \ref{table:adv_def} adversarial training doesn't impact the agent's performance in a significant way.

\subsubsection{Defenses applied during and after training}
One difference between adversarial training and the other two defenses that have been evaluated is that the first is applied during training while JPEG compression and feature squeezing have been applied only after training, that is during evaluation. Generally speaking, adversarial training should work better since the model learns to play following its policy in presence of perturbations. However, training the model for a longer time is costly and usually constrained to learn against perturbations crafted with just one type of attack method. Fortunately, adversarial training presents a sort of transferability, meaning that models defended with FGSM can also protect against PGD attacks and vice versa. This property is clearly shown in the charts in figure \ref{figure:untargeted-rew} and \ref{figure:targeted-rew}. JPEG compression and feature squeezing can still work against a large variety of attacks but the protection that they offer is often limited.

\subsubsection{Observations noise imperceptibility}
Given a fixed perturbations' strength \(\epsilon\), different image attacks can inject more or less perceptible noise to the target observations. Lower values of \(\epsilon\) usually lead to less visible perturbations, while higher values can make the target images too noisy and easily perceived by the human brain. Hence, to give an idea of the effect caused by the injection of noise to clear observations, we crafted four adversarial observations with FGSM and increasing values of \(\epsilon\) in the range [0.1, 0.01, 0.001, 0.0001] and compared them with clear observations. Then, we crafted other four adversarial observations with PGD and with the same values of \(\epsilon\). The noisy observations and the clear ones are shown in figure \ref{figure:pong-noise}. As we can immediately note in the figure, for similar \(\epsilon\) PGD crafts less perceptible adversarial observations than the one generated with FGSM. The injection of noise is visible in all the malicious observations generated with FGSM, while for adversarial examples generated with PGD, only the one crafted with \(\epsilon\) equal to 1 can be clearly detected. All perturbations have been crafted attacking a Pong-PPO model with \(l_\infty\) distance.

\begin{figure}
  \centering
  \subcaptionbox{\label{fig:subfig-a}}
    {\includegraphics[width=0.19\linewidth]{images/noise/PongNoFrameskip-v4_gsm_ppo_eps_0.1.png}}
  \subcaptionbox{\label{fig:subfig-b}}
    {\includegraphics[width=0.19\linewidth]{images/noise/PongNoFrameskip-v4_gsm_ppo_eps_0.01.png}}
    \subcaptionbox{\label{fig:subfig-c}}
    {\includegraphics[width=0.19\linewidth]{images/noise/PongNoFrameskip-v4_gsm_ppo_eps_0.001.png}}
    \subcaptionbox{\label{fig:subfig-d}}
    {\includegraphics[width=0.19\linewidth]{images/noise/PongNoFrameskip-v4_gsm_ppo_eps_0.0001.png}}
    \subcaptionbox{\label{fig:subfig-e}}
    {\includegraphics[width=0.19\linewidth]{images/noise/PongNoFrameskip-v4_gsm_ppo_eps_0.1_ori.png}}
    \subcaptionbox{\label{fig:subfig-f}}
    {\includegraphics[width=0.19\linewidth]{images/noise/PongNoFrameskip-v4_pgda_ppo_eps_0.1.png}}
    \subcaptionbox{\label{fig:subfig-g}}
    {\includegraphics[width=0.19\linewidth]{images/noise/PongNoFrameskip-v4_pgda_ppo_eps_0.01.png}}
    \subcaptionbox{\label{fig:subfig-h}}
    {\includegraphics[width=0.19\linewidth]{images/noise/PongNoFrameskip-v4_pgda_ppo_eps_0.001.png}}
    \subcaptionbox{\label{fig:subfig-i}}
    {\includegraphics[width=0.19\linewidth]{images/noise/PongNoFrameskip-v4_pgda_ppo_eps_0.0001.png}}
    \subcaptionbox{\label{fig:subfig-j}}
    {\includegraphics[width=0.19\linewidth]{images/noise/PongNoFrameskip-v4_pgda_ppo_eps_0.1_ori.png}}
  \caption{{\bf First row}: Comparison of the effect of noise injections on Pong environment observations with different strength of perturbations. The most right one is a clear observation and from the right to the left have been injected noise with increasing perturbations \(\epsilon=[0.1, 0.01, 0.001, 0.0001]\). Perturbations have been crafted with FGSM attacking a policy trained with PPO. {\bf Second row}: Comparison of the effect of noise injections on Pong environment observations with different strength of perturbations. The most right one is a clear observation and from the right to the left have been injected noise with increasing perturbations \(\epsilon=[0.1, 0.01, 0.001, 0.0001]\). Perturbations have been crafted with PGD attacking a policy trained with PPO.}
  \label{figure:pong-noise}
\end{figure}

\section{Number of attacked policies}
One of the major difficulties encountered while carrying out this thesis work is the computation time taken to run all the experiments. For the first experiment have been evaluated 5 policy attacks against 3 reinforcement learning algorithms trained on 2 environments. Each policy has been attacked 4 times corresponding to the four curves shown on each chart: attacking without transferability, attacking a surrogate policy, and attacking two different surrogate algorithms. Each curve has been drawn out of 10 different attack frequencies corresponding to ten points along the \(x\)-axis. Uniform attack, since it is the fastest attack, has its corresponding curves containing 20 points. Moreover, each point has been computed as average of 10 runs. Thus, in total have been attacked at least \(5*3*2*4*10*10=12000\) policies. The second experiment instead evaluates 2 policy attacks against 3 reinforcement learning algorithms trained in a single environment. Similarly, each policy has been attacked 4 times corresponding to the four curves shown on each chart as in the previous experiment. Given that each curve is composed of 10 points averaged over 10 runs, in total have been attacked at least \( 2*3*1*4*10*10=2400\) policies. Finally in the third experiment have been evaluated 6 policies trained with DQN where 5 of them were defended with a defense mechanism and trained on the Pong environment. Each policy has been attacked with 3 different white-box attack methods and each curve is composed of 20 points taken over an average of 10 runs. The experiment has been conducted on both targeted and untargeted threat models. Hence, in total have been attacked \(6*3*2*20*10=7200\) policies. Hence, the whole work includes a total of 12000+2400+7200=21600 policy attacks. It took about two months using between 4 and 8 GeForce GTX TITAN X GPUs. In practice, more experiments have been conducted while tuning the attacks' hyper-parameters and verifying the correctness of all the implemented attacks.

\section{Implementation details}
In this section, we are going to discuss the details relative to the implementation of the evaluated reinforcement learning algorithms, the policy attacks, and the image attacks. The Github repository containing the code to obtain the results presented in this thesis is available at the following address: \url{https://github.com/davide97l/rl-policies-attacks-defenses}. It has been developed with Pytorch but relies on other frameworks, each specialized in a different task. The reinforcement learning algorithms have been taken from the framework \href{https://github.com/davide97l/rl-policies-attacks-defenses}{Tianshou} \cite{tianshou} developed by \href{http://ml.cs.tsinghua.edu.cn/}{TSAIL}. It consists of a reinforcement learning platform based on pure PyTorch and with support to parallelized environments. Image adversarial attacks have been taken from \href{https://github.com/BorealisAI/advertorch}{Advertorch} \cite{ding2019advertorch} developed by \href{https://www.borealisai.com/en/}{BorealisAI}. Advertorch is a Python toolbox for adversarial robustness research also based on PyTorch. Given that the model interface required by Advertorch is different from the one required by the model trained with Tianshou, it has been created an adapter class to adapt Tianshou models to Advertorch models. The two simple image defenses, namely JPEG conversion and feature squeezing have been implemented from scratch. Policy adversarial training has been developed from scratch and integrated into Tianshou. It expands the Tianshou classes implementing the training procedure of the normal policies to defend them with adversarial training. Policy attacks have also been implemented from scratch and made compatible with Tianshou. Their modularized code allows to easily develop new attacks by expanding a base class or any of the existing policies. Hence, the result of this work is also a framework to test the robustness of reinforcement learning trained policies. Finally, the Atari environments have been taken from the Gym library developed by \href{https://openai.com/}{OpenAI}. The same library also includes other well-studied environments made publicly available to the AI community.


\documentclass{ecnreport}

\stud{Option Robotique, Control \& Robotics master}
\topic{Computer Vision}

\begin{document}

\inserttitle{Computer Vision Lab}

\insertsubtitle{Labs 3-4: Calibration and pose computation}

\section{Content of this lab}

The goal is this lab is to perform camera calibration and pose computation. Do to so, we will use two C++ libraries:
\begin{itemize}
 \item OpenCV\footnote{Open source Computer Vision, http://opencv.org} to read images from files or camera, and perform low-level image processing
 \item ViSP\footnote{Visual Servoing Platform, http://visp.inria.fr} to manipulate 3D points, camera pose and use vectors and matrices.
\end{itemize}
The actual classes that are used are detailed in Appendix \ref{sec:classes}.\\

% \subsection{Environment setup}
%
% To use this lab (and for the next ROS labs) you need to initialize the ROS environment in order to have access to the ViSP library.
% To do so, download the following file and execute it:
% \begin{center}\cppstyle
% \begin{lstlisting}
%         wget http://pagesperso.ls2n.fr/~kermorgant-o/files/ros_user_setup.sh
%         sh ros_user_setup.sh
% \end{lstlisting}
% \end{center}
% In order to use an IDE like Qt Creator, you should open a new terminal and launch the IDE from the command line.

\subsection{The cameras}

Several IEEE-1394 Firewire cameras are available for this lab. Some of them have a low distortion while others are fish-eye cameras. We will consider the
same framework to calibrate and perform pose computation for both, that is virtual visual servoing (VVS).

\subsection{Calibration landmarks}

Camera calibration and pose computation usually require to observe known objects. In our case we will consider a grid composed of 6$\times$6 points and
the OpenCV chessboard (see \Fig{fig:landmarks}).

\begin{figure}[h!]
 \includegraphics[width=.45\linewidth]{grid} ~~\includegraphics[width=.45\linewidth]{chessboard}
 \caption{Calibration landmarks, point grid with a low distortion camera (left) and OpenCV chessboard seen by a fish-eye camera (right).}
 \label{fig:landmarks}
\end{figure}

\newcommand{\xii}{{\boldsymbol{\xi}}}

In both cases, the interest points lie on a grid pattern of known dimension (rows, columns and inter-point distance). The 3D coordinates of those points
are thus known. The goal is to estimate the camera parameters $\xii$ and the camera poses $\M_i$ (one pose for each image) that minimize the reprojection error between the known 3D positions
and the extracted 2D (pixel) positions.

\subsection{Calibration procedure}
\label{sec:calib}

Calibration will begin by saving several images acquired from the camera (program \texttt{record}) The calibration program will then have to read them and extract the interest
points (already written in \texttt{calibration.cpp}). A virtual visual servoing scheme will then be used to minimize the error.
Assuming we are using $n$ images and that the calibration landmark is using $m$ points:
\begin{itemize}
 \item the reprojection error is of dimension $(2\times n\times m)$ (2 coordinates per point per image).
 \item the unknown is of dimension $(n_\xii + 6\times n)$ where $n_\xii$ is the number of parameters of the camera model.
\end{itemize}
The VVS is an optimization problem written as:
\begin{equation}\label{eq:calib}
 \xii, (\M_i)_i = \arg\min \norm{\s(\xii, (\M_i)_i) - \sd}^2
\end{equation}
where $\sd$ regroups the desired pixel coordinates of the interest points in each image and correspond to the extracted pixel positions,
and where $\s(\xii, (\M_i)_i)$ is the pixel coordinates of the known 3D points when projected with intrinsic parameters $\xii$ and camera poses $(\M_i)_i$.

From an estimation of $\xii$ and $(\M_i)_i$, the 3D point $\X({}^o X,{}^o Y,{}^o Z)$ whom coordinates are expressed in the object frame $\Frame{o}$ can easily be projected in pixel coordinates.
The coordinates of $\X$ in the camera frame yield:
\begin{equation}\label{eq:framechange}
 \left[\begin{array}{c}{}^c\X\\1\end{array}\right]=   \left[\begin{array}{c}X\\Y\\Z\\1\end{array}\right] =\M_i\left[\begin{array}{c}\X\\1\end{array}\right]
\end{equation}

For a classical perspective camera model, we have $\xii = (p_x, p_y, u_0, v_0)$ and the perspective projection:
\begin{equation}\label{eq:projection}
 \left\{\begin{array}{ll}
  u &= p_x.X/Z + u_0 \\
  v &= p_y.Y/Z + v_0
 \end{array}\right.
\end{equation}

From \eqref{eq:framechange} and \eqref{eq:projection} the full projection function $\s(\xii, (\M_i)_i)$ can be defined.\\

\eqref{eq:calib} can then be resolved from an initial guess on $\xii$ and $(\M_i)_i$ with a simple gradient descent.
The error derivative with regards to intrinsic parameters and camera poses has the following structure:
\newcommand{\der}{\text{d}}
\begin{equation}\label{eq:jacob}
 \der\s =  \left[\begin{array}{ccccc}
           \J_1 & \L_1 &  0 &  \hdots & 0 \\
           \J_i & 0 & \L_i &  \hdots & 0 \\
           \vdots& \vdots & \ddots & \ddots & \vdots\\
           \J_n & 0 & \hdots & 0 & \L_n
            \end{array}\right]
            \left[\begin{array}{c}\der\xii \\ \v_1 \\ \vdots \\ \v_n\end{array}\right]
             = \J\der\x
\end{equation}
where:
\begin{itemize}
\item $\J_i$ is the derivative of the reprojection of the points of image $i$ with regards to the intrinsic parameters (dimension $(2m)\times n_\xii$)
 \item $\L_i$ is the interaction matrix of the reprojection of the points of image $i$ (dimension $(2m)\times 6$)
 \item $\der\xii$ is the change in the intrinsic parameters (dimension $n_\xii$)
 \item $\v_i$ is the velocity screw of the estimated camera pose of image $i$ (dimension $6$)
 \item As a consequence, $\J$ is of dimension $(2mn)\times (n_\xii+6n)$ and $\der\x$ is of dimension  $(n_\xii+6n)$
\end{itemize}
The gradient descent allows computing iteratively new values for $\xii$ and $(\M_i)_i$ with:
\begin{equation}
 \der\x = \left[\begin{array}{c}\der\xii \\ \v_1 \\ \vdots \\ \v_n\end{array}\right] = -\lambda\Jp(\s - \sd)
\end{equation}
The update of $\xii$ and $\M_i$ is then done with:
\begin{eqnarray}
 \xii^{\text{new}} =& \xii + \der\xii \label{upd:xi}\\
 \M_i^{\text{new}} =& \exp(-\v_i)\times\M_i \label{upd:M}
\end{eqnarray}
where $\exp(-\v_i)$ is the exponential map.\\

Matrices $\J_i$ and $\L_i$ are of course not constant and depend on the several considered points. The formulation of these matrices are given in Section \ref{sec:jacpersp} for a perspective camera.

\subsection{Pose computation procedure}
\label{sec:pose}
In pose computation we assume the camera is already calibrated and we are dealing with the image stream of the camera.
We are just looking for the current pose $\M$ of the camera.
The minimization problem is thus the same as \eqref{eq:calib} but is highly simplified:
\begin{equation}\label{eq:pose comp}
\M = \arg\min \norm{\s(\M) - \sd}^2
\end{equation}The exact same algorithm as for the calibration can thus be used, without having to consider intrinsic parameters or several images.
\eqref{eq:jacob} is simplified to :
\begin{equation}
 \der\s =  \L\v
\end{equation}
The initial guess for $\M$ still has to be done, but when a new image comes the guess is of course the previous estimation of $\M$ as we assume the image stream
correspond to very close poses of the camera.

\subsection{Expected work}

During the lab the files will be modified and others will be created.
At the end of the lab, please send by email a zip file allowing to compile and test the program.

You may answer the questions by inserting comments in the code at the corresponding lines.
\newpage

\section{Calibration implementation}

You will first calibrate the camera you are working on. The first step is to save several images of the landmark (point grid) on the hard drive.

The files that need to be modified are:
\begin{itemize}
 \item \texttt{calibration.cpp}: main file, defines the image path, initial guess for the camera parameters and grid pattern (size, dimension)
 \item \texttt{perspective\_camera.h}: defines the camera model, you need to write how to project a point depending on the current camera parameters, and how to compute the Jacobian of these
 coordinates
 \item \texttt{vvs.cpp}: the \texttt{calibrate} method of this class has to be written according to the algorithm from Section \ref{sec:calib}.\\
 The overall architecture is already written, you have to write the content of the minimization loop:
 \begin{itemize}
  \item project a point from current estimation
  \item compute the Jacobians and write them into the global Jacobian
  \item  update the parameter estimation
 \end{itemize}
\end{itemize}

The calibration with virtual visual servoing is a highly non-linear, overconstrained algorithm. Indeed it tries to solve $2m\times n$ equations (2 coordinates for $m$ points in $n$ images) with
only $n_\xii + 6\times n$ unknown (the number of intrinsic parameters and 6 pose parameters for $n$ images). Hence, the initial guess is of great importance
not to fall in a local minimum. Here are some initial guess tips:
\begin{itemize}
 \item Camera parameters: we can assume a perspective camera with the middle pixel being on the optical axis, and a pixel dimension proportional to the image resolution.
 A good initial guess could thus be written as:
 \begin{center}\cppstyle
\begin{lstlisting}
    // camera model with default parameters (in calibration.cpp)
    const double pxy = 0.5*(im[0].rows+im[0].cols);
    PerspectiveCamera cam(pxy, pxy, 0.5*im[0].cols, 0.5*im[0].rows);
\end{lstlisting}
\end{center}
\item Initial pose: as it uses Cartesian coordinates, the VVS is very sensible to big error on the rotation around Z. However as we know the points have been detected, the first row can be used to
get a coarse estimation of this rotation. We also know that the object lies in front of the camera, which means a positive value for $t_z$. So a vector of $n$ homogeneous matrices could be initialized as:
 \begin{center}\cppstyle
\begin{lstlisting}
        M_.resize(n_im);
        // wild guess for initial pose, then will use the last found at initial value
        for(unsigned int n=0;n<n_im;++n)
            M_[n].buildFrom(0,0.,0.5,
            0,0,atan2(_cog[n][5].y-_cog[n][0].y, _cog[n][5].x-_cog[n][0].x));
\end{lstlisting}
\end{center}where we assume the camera is at 0.5 m from the object.
\end{itemize}

The calibration can be first carried with the point grid pattern, using the \texttt{GridTracker} class. You can then check that it also works on the chessboard,
using the \texttt{CBTracker}. The detail of input/output for all classes and methods are found in Appendix \ref{sec:classes}.

\section{Pose computation implementation}

The source file for the pose computation does not exist, so you can copy and modify the \texttt{calibration.cpp} to a new file \texttt{posecomputation.cpp}\footnote{do not forget adding your new file
to the CMakeLists.txt}. This time we do not want to load images from the hard drive but directly read them from the camera. This is possible with an infinite while loop and by using the \texttt{cv::VideoCapture} class.

For the pose computation, you can already initialize the camera with the parameters that have been found in the calibration. The files that need to be modified are then:
\begin{itemize}
 \item \texttt{posecomputation.cpp}: main file (has to be created), reads images from the camera, defines the grid pattern (size, dimension) and call the pose computation from VVS
 \item \texttt{perspective\_camera.h}: defines the camera model, it should be already done for the calibration
 \item \texttt{vvs.cpp}: the \texttt{computePose} method of this class has to be written according to the algorithm from Section \ref{sec:pose}.
\end{itemize}
This time the initial guess is the same as above for the first image, but can be the previously found homogeneous matrix as the camera will not move too much between two images. This will allow the pose computation
converge faster.

\appendix

\section{Main classes}\label{sec:classes}

\subsection{OpenCV classes}

These labs use a few OpenCV classes, mainly images and points. The only one that needs to be used is the \texttt{cv::Point}, and basically we just have to know that the $(u,v)$ coordinates can be accessed
as \texttt{P.x} and \texttt{P.y}.

\subsection{ViSP classes}

This library includes many tools for linear algebra, especially for 3D transformations. These classes are mainly used inside the \texttt{VVS} class for the minimization algorithm.
The documentation is found here: \url{http://visp-doc.inria.fr/doxygen/visp-daily/classes.html}.\\
The main classes from ViSP (at least in this lab) are:
\begin{itemize}
\item \texttt{vpMatrix} represents a classical matrix, can then be transposed, inversed (or pseudo-inversed), multiplied with a vector, etc.
\item \texttt{vpColVector} is a column vector with classical mathematical properties.
\item \texttt{vpHomogeneousMatrix} represents a frame transformation matrix \texttt{M} and is initialized from 3 translation and 3 rotation parameters.
\item \texttt{vpPoint}: represents a 3D point in the world frame. If the world-to-camera pose is \texttt{M}, we can compute the point projection on the camera plane by:
\begin{center}\cppstyle
\begin{lstlisting}
        vpPoint P;
        vpHomogeneousMatrix M;
        P.track(M);
        double x = P.get_x();
        double y = P.get_y();
\end{lstlisting}
\end{center}
\end{itemize}

\subsection{Trackers}

The \texttt{CBTracker} and \texttt{GridTracker} are very similar and differ only from the object they try to detect. They have only two methods:
\begin{itemize}
    \item \texttt{bool detect(cv::Mat \&\_im, vector<cv::Point> \&\_cog)} \\ writes the vector of points \texttt{\_cog} with the points detected in image \texttt{\_im}
    \item \texttt{bool track(cv::Mat \&\_im, vector<cv::Point> \&\_cog)} \\ updates the vector of points \texttt{\_cog} with the points detected in image \texttt{\_im}.\\The update tries to
    be consistent with the current values of the points, ie it will not lead to a 180${}^o$ rotation around Z.
\end{itemize}Both functions returns True if they managed to detect or track the points.
As for all visual trackers, we detect in the first image and track in the next images (except during calibration where there is no reason to track).\\
The tracker may be lost if the pattern is not here or if the image is blurry, so it can be a good idea to check the return value and re-detect points instead of always tracking them.

\subsection{Camera models}

The \texttt{GenericCamera} is a virtual class that implements the methods of any camera model:
\begin{itemize}
 \item \texttt{unsigned int nbParam()} \\ returns the number of parameters used by this camera model
 \item \texttt{void project(const vpPoint \&\_P, double \&\_u, double \&\_v)} \\ takes as an input the 3D point \texttt{\_P} that is already expressed in the camera frame, and computes the corresponding
 pixel coordinates $(u,v)$.
    \item \texttt{void computeJacobianIntrinsic(const vpPoint \&\_P, vpMatrix \&\_J)} \\ takes as an input the 3D point \texttt{\_P} that is already expressed in the camera frame and updates \texttt{\_J} so that it corresponds to the intrinsic Jacobian to be used in \eqref{eq:jacob}
    \item \texttt{void computeJacobianExtrinsic(const vpPoint \&\_P, vpMatrix \&\_L)} \\ takes as an input the 3D point \texttt{\_P} that is already expressed in the camera frame and updates \texttt{\_L} so that it corresponds to the extrinsic Jacobian to be used in \eqref{eq:jacob}
    \item \texttt{void updateIntrinsic(const vpColVector \&\_dxi)} \\ takes as input a small change of the intrinsic parameters and updates them according to \eqref{upd:xi}.
 \end{itemize}

 Two camera models are considered:

 \begin{enumerate}
  \item  \texttt{PerspectiveCamera}: Classical perspective camera, with projection equation \eqref{eq:projection}. The class is created but you will have to define the above mentionned methods (project, update Jacobian).
  \item \texttt{DistortionCamera}: Unified camera model, this class is already entirely written, you may use it to see how to define the methods for the perspective camera (of course the equations will be much more simple).
 \end{enumerate}

 \subsection{The \texttt{Pattern} structure}

 This structure is used to regroup data of a given image: the actual OpenCV image, the corresponding vector of points that have been extracted, and the window name used to display this image.

  \begin{center}\cppstyle
\begin{lstlisting}
 // structure used to regroup data
struct Pattern
{
    cv::Mat im;                     // OpenCV image
    std::vector<cv::Point> point;   // extracted points
    std::string window;             // window to display this image
};\end{lstlisting}
\end{center}This structure is only used to pass arguments to the virtual visual servoing class.

 \subsection{Virtual Visual Servoing class}

 The \texttt{VVS} class implements the minimization algorithm. It is instanciated with:
 \begin{center}\cppstyle
\begin{lstlisting}
        VVS vvs(cam, d, r, c);
\end{lstlisting}
\end{center}where cam is the camera model that is used (PerspectiveCamera or DistortionCamera), d is the inter-point distances of the real landmark, and r and c are the number of rows and columns. The VVS uses these values to initialize the true position of the 3D points.

Only two methods are to be modified:
\begin{itemize}
    \item \texttt{void calibrate(std::vector<Pattern> \&\_pat)} (already quite written)\\ solves the calibration problem from a given set of \texttt{Pattern}.
    \item \texttt{void computePose(Pattern \&\_pat, vpHomogeneousMatrix \&\_M, const bool \&\_reset)} \\
    solves the pose computation problem from a single \texttt{Pattern} (one image with extracted points).\\ This method can be highly inspired from the calibrate method, as shown in Section \ref{sec:pose}.
\end{itemize}

This class also have useful attributes:
\begin{itemize}
 \item \texttt{std::vector<vpPoint> X\_}: a vector of the true 3D position of the points, and is to be used in the reprojection error
 \item \texttt{GenericCamera* cam\_}: the camera that we are using or calibrating, it is a pointer to the methods can be called by using an arrow
\end{itemize}
As a syntax example, here is how we compute the (u,v) pixel coordinates of point k assuming a transformation matrix M (which can be quite useful for reprojection error):
 \begin{center}\cppstyle
\begin{lstlisting}
double u,v;
X_[k].track(M);		     // X_[k] is computed in the camera frame
cam_->project(X_[k], u, v);  // u and v are now the pixel coord. for the current camera model
\end{lstlisting}
\end{center}

\section{Jacobians for a perspective camera model}
\label{sec:jacpersp}

In this section we recall the intrinsic and extrinsic Jacobians for a classical camera model, with parameters $\xii = (p_x, p_y, u_0, v_0)$.
The projection model is:
\begin{equation}
 \left[\begin{array}{c}{}^c\X\\1\end{array}\right]=   \left[\begin{array}{c}X\\Y\\Z\\1\end{array}\right] =\M_i\left[\begin{array}{c}\X\\1\end{array}\right]
\end{equation}
\begin{equation}
 \left\{\begin{array}{ll}
  u &= p_x.X/Z + u_0  = p_x.x + u_0\\
  v &= p_y.Y/Z + v_0 = p_v.y + v_0
 \end{array}\right.
\end{equation}

The Jacobian of $(u,v)$ with regards to the intrinsic parameters is quite easy to compute:
\begin{equation}
 \J = \left[\begin{array}{cccc}
	    x & 0 & 1 & 0\\
	    0 & y & 0 & 1
            \end{array}\right]
\end{equation}
The Jacobian with regards to the extrinsic parameters (also called the interaction matrix) yields:
\begin{equation}
 \L =    \left[\begin{array}{cccccc}
-p_x/Z & 0 & p_x.x/Z & p_x.xy & -p_x(1+x^2) & p_x.y \\
0 & -p_y/Z & p_y.y/Z & p_y(1+y^2) & -p_y.xy & -p_y.x
\end{array}\right]
\end{equation}
These two matrices should be defined in the \texttt{PerspectiveCamera::computeJacobianIntrinsic} and \texttt{PerspectiveCamera::computeJacobianExtrinsic}
methods, and depend of course on the considered point.

\end{document}


\renewcommand\rimpl{\rsubbase}
\renewcommand{\rtdimp}{\rsubbased}
%
\section{Algorithmic Typing: \texorpdfstring{\declang}{LamD}}\label{sec:typing}

\input{text/refinedhaskell/modifications}

While \undeclang is sound, it cannot be \emph{implemented}
thanks to the undecidable denotational containment rule \rsubbase
(Figure~\ref{fig:refinedhaskell:typing}).
%
Next, we go from \undeclang to \declang, a core calculus
with sound, SMT-based algorithmic type-checking in four
steps.
%
First, we show how to restrict the language of
refinements to an SMT-decidable sub-language
\logiclang~(\Sref{sec:typing:logic}).
%
Second, we \emph{stratify} the types to specify
whether their inhabitants may diverge, must reduce
to values, or must reduce to finite values~(\Sref{sec:typing:stratify}).
%
Third, we show how to \emph{enforce} the stratification
by encoding recursion using special fixpoint combinator
constants (\Sref{sec:typing:termination}).
%
Finally, we show how to use \logiclang and the
stratification to approximate the undecidable \rsubbase
with a decidable verification condition \rsubbased, thereby
obtaining the algorithmic system \declang~(\Sref{sec:typing:vc}).

\subsection{Refinement Logic: \logiclang}\label{sec:typing:logic}

Figure~\ref{fig:declang:syntax} summarizes the syntax of \declang.
Refinements $r$ are now predicates $p$, drawn from
\logiclang, the decidable logic of equality,
uninterpreted functions and linear arithmetic
~\cite{Nelson81}.
%
Predicates $p$ include linear arithmetic constraints,
function application where function symbols correspond
to measures (as described in \Sref{sec:measures}), and
boolean combinations of sub-predicates.

\mypara{Well-Formedness}
For a predicate to be well-formed it should be boolean and
arithmetic operators
should be applied to integer terms, measures should be applied
to appropriate arguments
(\ie \eisNull\ is applied to \tintlist),
and equality or inequality to basic
(integer or boolean) terms.
%
Furthermore, we require that refinements, and thus measures,
always evaluate to a value.
%
We capture these requirements by assigning appropriate types
to operators and measure functions, after which we require that
each refinement $r$ has type $\FinTy{\tbool}$ (rule \rwbased in
Figure~\ref{fig:declang:typing}).

\mypara{Assignments}
Figure~\ref{fig:declang:syntax} defines the elements $d$
of the domain \dom
of integers, booleans, and data constructors that wrap
elements from \dom.
%
The domain \dom also contains a constant $c_w$
for each value $w$ of \undeclang that does
not otherwise belong in \dom (\eg functions or other primitives).
%
An \emph{assignment} $\sigma$ is a map from variables
to $\dom$.

%% \NV{DONE:Can we just drop the measure + axioms step?}
\mypara{Satisfiability \& Validity}
We interpret boolean predicates in the logic over the domain
\dom.
%
We write \lmodels{\sigma}{p} %(resp. \umodels{\sigma}{p})
if $\sigma$ is a model of $p$.
%% where the interpretations of
%%measures respect the measure axioms (resp. measures are
%%uninterpreted).
%
We omit the formal definition for space.
%
A predicate $p$ is \emph{satisfiable}  %(resp. \emph{u-satisfiable})
if there \emph{exists} $\lmodels{\sigma}{p}$. %(resp. $\umodels{\sigma}{p}$).
%
A predicate $p$ is \emph{valid} %(resp. \emph{u-valid})
if \emph{for all} assignments $\lmodels{\sigma}{p}$. % (resp. \umodels{\sigma}{p}).
%
%Note that if $p$ is u-valid then $p$ is trivially valid.

%\begin{lemma}\label{ref:valid} If $p$ is u-valid then $p$ is valid.
%\end{lemma}

\mypara{Connecting Evaluation and Logic}
To prove soundness, we need to formally connect the
notion of logical models with the evaluation of a
refinement to \etrue.
%
We do this in several steps, briefly outlined for brevity
(the detailed proof is in~\cite{vazou14techrep}).
%
First, we introduce a primitive \emph{bottom expression}
\ebot that can have \emph{any} \Div type, but does not evaluate.
%%(\NV{only values can be applied to primitive constants,
%%so $c\ \ebot$ will just not evaluate.}
%%\ie yields \ebot when applied to any primitive constant).
%
Second, we define \emph{lifted values} \botv
(Figure~\ref{fig:declang:syntax}), which are values that
contain \ebot.
%
Third, we define \emph{lifted substitutions}
$\botsto$, which are mappings from variables to
lifted values.
%
Finally, we show how to \emph{embed} a lifted substitution
\botsto into a \emph{set of} assignments \embed{\botsto}
where, intuitively speaking, each \ebot is replaced by
some arbitrarily chosen element of \dom.
%
%Equipped with the above notions, we prove the following which
%connects evaluation and logical satisfaction.
Now, we can connect evaluation and logical satisfaction:
%
\begin{theorem}\label{thm:equiv}
If \dechastype{\emptyset}{\botsto(p)}{\FinTy{\tbool}}, then
$\evals{\botsto(p)}{\etrue}\
\mbox{iff}\ \
\forall \sigma \in \embed{\botsto}. \lmodels{\sigma}{p}$.
\end{theorem}

\mypara{Restricting Refinements to Predicates}
Our goal is to restrict \rimpl so that only predicates
from the decidable logic \logiclang (not arbitrary expressions)
appear in implications \isimplied{\Env}{\p_1}{\p_2}.
%
Towards this goal, as shown in Figures~\ref{fig:declang:syntax}
and~\ref{fig:declang:typing},
we restrict the syntax and well-formedness of types to contain
only predicates
%
and we convert the program to ANF after which we can
restrict the application rule \rtappd to applications
to variables, which ensures that refinements remain
within the logic after substitution~\cite{LiquidPLDI08}.
%
Recall, that this is not enough to ensure that refinements do converge,
as under lazy evaluation,
even binders can refer to potentially divergent values.

\subsection{Stratified Types}\label{sec:typing:stratify}

The typing rules for \declang are given in Figure~\ref{fig:declang:typing}.
Instead of \emph{explicitly} reasoning about divergence or
strictness in the refinement logic, which leads to significant
theoretical and practical problems, as discussed in \Sref{sec:refinedhaskell:conclusion},
we choose to reason \emph{implicitly} about divergence within the type system.
%
Thus, the second critical step in our path to \declang is the
stratification of types into those inhabited by potentially
diverging terms, terms that only reduce to values, and
terms which reduce to finite values.
%
Furthermore, the stratification crucially allows us to prove
Theorem~\ref{thm:equiv}, which requires that refinements do
not diverge (\eg by computing the length of an infinite list)
by ensuring that inductively defined measures are only applied
to finite values.
%
Next, we describe how we stratify types with labels and
then type the various constants, in particular the fixpoint
combinators, to enforce stratification.

\mypara{Labels}
We specify stratification using two \emph{labels} for types.
%
The label \trivial\ (resp. \finite) is assigned to types given
to expressions that reduce (using $\beta$-reduction from Figure~\ref{fig:operational})
to a value $w$ (resp. \emph{finite} value,
\ie an element of the inductively defined \dom).
%
Formally,
%
\begin{align}
  \mbox{\emphbf{\Wnf types}} \quad
\interp{\tref{\vv}{\Base}{\trivial}{r}} \defeq &
    \interp{\tref{\vv}{\Base}{}{r}} \cap \{ e \mid \evals{e}{w} \}
    \label{eq:trivial} \\
  \mbox{\emphbf{\Fin types}} \quad
\interp{\tref{\vv}{\Base}{\finite}{r}} \defeq &
    \interp{\tref{\vv}{\Base}{}{r}} \cap \{ e \mid \evals{e}{d} \}
    \label{eq:finite}
\end{align}
%
Unlabelled types are assigned to expressions that may diverge.
%
Note that for any $\Base$ and refinement $r$ we have
$$
\interp{\tref{\vv}{\Base}{\finite}{r}} \subseteq
\interp{\tref{\vv}{\Base}{\trivial}{r}} \subseteq
\interp{\tref{\vv}{\Base}{}{r}}
$$
%
The first two sets are \emph{equal} for $\tint$ and $\tbool$,
and \emph{unequal} for (lazily) constructed data types $T$.
%
We need not stratify function types (\ie they are \Div types)
as binders with function types do not appear inside the VC, and
are not applied to measures.

\mypara{Enforcing Stratification}\label{sec:typing:termination}
%
We enforce stratification in two steps.
%
First, the $\rtcased$ rule uses the operational semantics of case-of to
type-check each case in an environment where the scrutinee
$x$ is assumed to have a \Wnf type.
%
All the other rules, not mentioned in Figure~\ref{fig:declang:typing},
remain the same as in Figure~\ref{fig:refinedhaskell:typing}.
%
Second, we create stratified variants for the primitive constants
and \emph{separate} fixpoint combinator constants for
(arbitary, potentially non-terminating) recursion (\efix{})
and bounded recursion (\etfix{}).

\mypara{Stratified Primitives}
First, we restrict the primitive operators whose output
types are refined with logical operators, so they are only
invoked on finite arguments (so that the corresponding
refinements are guaranteed to not diverge).
%
\begin{align*}
  \constty{\mathtt{\n}} \defeq & \tlref{\vv}{\tint}{\finite}{\vv = \n} \\
  \constty{\mathtt{=}} \defeq & \tfun{x}
                                     {\tbase^\finite}
                                     {\tfun{y}
                                           {\tbase^\finite}
                                           {\tlref{\vv}{\tbool}{\finite}{\vv \Leftrightarrow \x = \y}}} \\
  \constty{\mathtt{+}} \defeq & \tfun{x}
                                     {\tint^\finite}
                                     {\tfun{y}
                                           {\tint^\finite}
                                           {\tlref{\vv}{\tint}{\finite}{\vv = \x + \y}}}\\
  \constty{\mathtt{\land}} \defeq & \tfun{x}
                                         {\tbool^\finite}
                                         {\tfun{y}
                                               {\tbool^\finite}
                                               {\tlref{\vv}{\tbool}{\finite}{\vv \Leftrightarrow \x \land \y}}}
\end{align*}
%
It is easy to prove that the above primitives respect
their stratification labels, \ie belong in the denotations
of their types.

Note that the above types are restricted in that they can only be applied to finite arguments.
%
In future work~\ref{chapter:conclusion}, we could address this issue with unrefined versions of primitive types
that soundly allow operation on arbitrary arguments.
%
For example, with the current type for $\mathtt{+}$, addition of potentially
diverging expressions is rejected.
%
Thus, we could define an unrefined signature
\begin{align*}
  \constty{\mathtt{+}} \defeq & \tfun{x}
                                     {\tint}
                                     {\tfun{y}
                                           {\tint}
                                           {\tint}}
\end{align*}
and allow the two types of $\mathtt{+}$ to co-exist (as an intersection type),
where the type checker would choose the precise refined type
if and only if both of $\mathtt{+}$'s arguments are finite.

\mypara{Diverging Fixpoints ($\efix{\typ}$)}
Next, note that the only place where divergence enters the picture is
through the fixpoint combinators used to encode recursion.
%
For any function or basic type $\typ \defeq \typ_1 \rightarrow \ldots \rightarrow \typ_n$,
we define the \emph{result} to be the type $\typ_n$.

For each $\typ$ whose result is a \Div type,
there is a \emph{diverging fixpoint} combinator
$\efix{\typ}$, such that
%
\begin{align*}
  \ceval{\efix{\typ}}{f} \defeq & f\ (\efix{\tau}\ f) \\
  \constty{\efix{\typ}}   \defeq & (\typ \rightarrow \typ) \rightarrow \typ
\end{align*}
%
\ie, $\efix{\typ}$ yields recursive functions of type $\typ$.
Of course, $\efix{\typ}$ belongs in the denotation of
its type~\cite{PLC} \emph{only if} the result type is
a \Div type (and \emph{not} when the result is a
\Wnf or \Fin type).
%
Thus, we restrict diverging fixpoints to functions with \Div result types.

\mypara{Indexed Fixpoints ($\tfixn{\typ}{n}$)}
For each type $\typ$ whose result is a \Fin
type, we have a family of \emph{indexed}
fixpoints combinators $\tfixn{\typ}{n}$:
%
\begin{align*}
  \ceval{\tfixn{\typ}{n}}{f} \defeq & {\efun{m}{}{f\ m\ (\tfixn{\typ}{m}\ f)}}\\
%% (n:nat → τ_n → τ) → τ_n
\constty{\tfixn{\typ}{n}} \defeq &
  \tfunbasic{(\tfun{n}
                   {\FinTy{\tnat}}
                   {\tfunbasic{\decr{\typ}{n}}{\typ}})}
            {\decr{\typ}{n}} \\
\mbox{where,}\ \decr{\typ}{n} \defeq & \tfunbasic{\tref{v}{\tnat}{\finite}{v < n}}{\typ}
\end{align*}
%
$\decr{\typ}{n}$ is a \emph{weakened} version of $\typ$
that can only be invoked on inputs \emph{smaller} than $n$.
%
Thus, we enforce termination by requiring that $\tfixn{\typ}{n}$ is
\emph{only} called with $m$ that are \emph{strictly smaller than} $n$.
%
As the indices are well-founded $\tnat$s, evaluation will terminate.

\mypara{Terminating Fixpoints ($\etfix{\typ}$)}
Finally, we use the indexed combinators to define the
\emph{terminating} fixpoint combinator $\etfix{\typ}$ as:
\begin{align*}
  \ceval{\etfix{\typ}}{f} \defeq & {\efun{n}{}{f\ n\ (\tfixn{\typ}{n}\ f)}}\\
%% (n:nat → τ_n → τ) → m:nat⇓ → τ
\constty{\etfix{\typ}} \defeq &
  \tfunbasic{(\tfun{n}
                   {\FinTy{\tnat}}
                   {\tfunbasic{\decr{\typ}{n}}{\typ}})}
            {\tfunbasic{\FinTy{\tnat}}{\typ}}
\end{align*}
%
Thus, the top-level call to the recursive function
requires a $\FinTy{\tnat}$ parameter $n$ that acts
as a \emph{starting} index, after which, all ``recursive''
calls are to combinators with \emph{smaller} indices,
ensuring termination.

\mypara{Example: Factorial}
Consider the factorial function:
$$\efac \defeq
\efun{n}{}{\efun{f}{}{
	\ecaseinstance{\_}{(n = 0)}{
		 \begin{array}{l}
			 \ealt{\etrue}{1}\\
			 \ealt{\_}{n \times f (n-1)}
		\end{array}
	}
}}$$
Let $\typ \defeq {\FinTy{\tnat}}$.
We prove termination by typing
\begin{align*}
  \emptyset \vdash_D \etfix{\typ}\ \efac : &\ \tfunbasic{\FinTy{\tnat}}{\typ} \\
\intertext{To understand \emph{why}, note that $\tfixn{\typ}{n}$ is
only called with arguments strictly smaller than $n$
}
% fix fac n
\etfix{\typ}\ \efac\ n
%   = fixn fac n
% \stepsym^* & \ \tfixn{\typ}{n}\ \efac\ n\\
%   = fac n (fixn fac)
\stepsym^* & \ \efac\ n\ (\tfixn{\typ}{n}\ \efac)\\
%   = n * (fixn fac (n-1))
\stepsym^* & \ n \times (\tfixn{\typ}{n}\ \efac\ (n-1))\\
%   = n * (fac (n-1) (fix_n-1 fac))
\stepsym^* & \ n \times (\efac\ (n-1)\ (\tfixn{\typ}{n-1}\ \efac))\\
%   = n * (n-1) * (fix_n-1 fac (n-2))
\stepsym^* & \ n \times n-1 \times (\tfixn{\typ}{n-1}\ \efac\ (n-2))\\
%   = n * (n-1) * (fac (n-2) (fix_n-2 fac))
%   = n * (n-1) * (n-2) * (fix_n-2 fac (n-3))
%   = n * (n-1) * ... * (fix_1 fac 0)
\stepsym^* & \ n \times n-1 \times \ldots \times (\tfixn{\typ}{1}\ \efac\ 0)\\
\stepsym^* & \ n \times n-1 \times \ldots \times (\efac\ 0\ (\tfixn{\typ}{0}\ \efac))\\
%   = n * (n-1) * ... * 1
\stepsym^* & \ n \times n-1 \times \ldots \times 1
\end{align*}

\mypara{Soundness of Stratification}
To formally \emph{prove} that stratification is soundly
enforced, it suffices to prove that the Denotation
Lemma~\ref{lem:denotation} holds for \declang.
%
This, in turn, boils down to proving that each
(stratified) constant belongs in its type's denotation,
\ie each $c \in \interp{\constty{c}}$
or that the Lemma~\ref{lemma:constants} holds for \declang.
%
The crucial part of the above is proving that
the indexed and terminating fixpoints inhabit
their types' denotations.
%

\begin{theorem}{[Fixpoint Typing]}
\begin{itemize}
  \item $\efix{\typ} \in \interp{\constty{\efix{\typ}}}$,
  \item $\forall n. \etfixn{\typ}{f}{n} \in \interp{\constty{\etfixn{\typ}{f}{n}}}$,
  \item $\etfix{\typ} \in \interp{\constty{\etfix{\typ}}}$.
\end{itemize}
\end{theorem}

With the above we can prove soundness of Stratification as a corollary
Denotation Lemma~\ref{lem:denotation}, given the interpretations of
the stratified types.
%
\begin{corollary}{[Soundness of Stratification]}\label{cor:stratification}
\begin{enumerate}
  \item If $\dechastype{\emptyset}{e}{\FinTy{\typ}}$, then  evaluation of $e$ is finite.
  \item If $\dechastype{\emptyset}{e}{\WnfTy{\typ}}$, then  $e$ reduces to WHNF.
  \item\label{col:ref} If $\dechastype{\emptyset}{e}{\ttreft{\vv}{\typ}{\p}}$, then \p cannot diverge.
\end{enumerate}
\end{corollary}

Finally, as a direct implication the well-formedness rule \rwbased
we conclude \ref{col:ref}, \ie that refinements cannot diverge.

\subsection{Verification With Stratified Types}\label{sec:typing:vc}

We can put the pieces together to obtain an algorithmic implication
rule \rtdimp instead of the undecidable \rimpl (from Figure~\ref{fig:refinedhaskell:typing}).
%
Intuitively, each closing substitution $\sto$ will correspond to
a set of logical assignments $\embed{\sto}$.
%
Thus, we will translate $\Env$ into logical
formula $\embed{\Env}$ and
denotation inclusion into logical implication
such that:
%
\begin{itemize}
\item $\sto \in \interp{\Env}$ \textit{iff} all $\sigma \in \embed{\sto}$
     satisfy $\embed{\Env}$, and
%%\item $\evals{\sto(p_i)}{\etrue}$ iff each $\sigma \in \embed{\sto}$
%%     satisfies $p_i$, and
\item ${\sto}{\ttreft{\vv}{\mathit{B}}{\p_1}}\subseteq {\sto}{\ttreft{\vv}{\mathit{B}}{\p_2}}$
		\textit{iff} all $\sigma \in \embed{\sto}$ satisfy $p_1 \Rightarrow p_2$.
\end{itemize}

%%Thus, we will translate $\Env$, $p_1$, and $p_2$ into logical
%%formulas $\embed{\Env}$, $\embed{p_1}$, and $\embed{p_2}$ such
%%that:
%%%
%%\begin{itemize}
%%\item $\evals{\sto(p_i)}{\etrue}$ iff each $\sigma \in \embed{\sto}$
%%     satisfies $\embed{p_i}$, and
%%\item $\sto \in \interp{\Env}$ iff each $\sigma \in \embed{\sto}$
%%     satisfies $\embed{\Env}$.
%%\end{itemize}

\mypara{Translating Refinements \& Environments}
To translate environments into logical formulas, recall that $\sto \in \interp{\Env}$
iff for each $\tbind{x}{\typ} \in \Env$, we have
$\sto(x) \in \interp{\thetasub{\sto}{\typ}}$. Thus,
%
\begin{align*}
\embed{\tbind{x_1}{\typ_1},\ldots,\tbind{x_n}{\typ_n}} \defeq &
  \embed{\tbind{x_1}{\typ_1}} \wedge \ldots \wedge \embed{\tbind{x_n}{\typ_n}}\\
\intertext{How should we translate a single binding? Since a binding denotes}
\interp{\tref{x}{\ttbase}{}{p}} \defeq &
  \{ e \mid \mbox{if } \evals{e}{w}\ \mbox{then}\ \evals{\SUBST{p}{x}{w}}{\etrue} \}\\
\intertext{a direct translation would require a logical value
  predicate $\isvalue{x}$, which we could use to obtain the logical
  translation}
\embed{\tref{x}{\ttbase}{}{p}} \defeq & \lnot \isvalue{x} \vee p
\end{align*}
%
This translation poses several theoretical and
practical problems that preclude the use of existing
SMT solvers (as detailed in \Sref{sec:refinedhaskell:conclusion}).
However, our stratification guarantees
(cf. (\ref{eq:trivial}), (\ref{eq:finite}))
that labeled types reduce to values and
so we can simply conservatively translate
the \Div and labeled (\Wnf, \Fin) bindings as:
$$\embed{\tref{x}{\ttbase}{}{p}}\ \defeq\ \etrue \qquad  \embed{\tref{x}{\ttbase}{l}{p}}\ \defeq\ p$$

\mypara{Soundness} We prove soundness by showing that
the decidable implication \rtdimp approximates
the undecidable \rimpl.

\begin{theorem}{}\label{thm:approximation} If \decissubref{\Env}{p_1}{p_2} is
  valid then
  $\undecissubtype{\Env}{\ttreft{\vv}{\Base}{\p_1}}{\ttreft{\vv}{\Base}{\p_2}}$.
\end{theorem}

%% \begin{theorem}{[Approximation]}\label{thm:approximation} Let $\VC \defeq \VCOND{\Env}{p_1}{p_2}$.
%%   \begin{itemize}
%%     \item If \VC is u-valid then \VC is valid,
%%     %\item If \VC is valid then \isimplied{\Env}{p_1}{p_2},
%%     \item If \decissubref{\Env}{p_1}{p_2} then \isimplied{\Env}{p_1}{p_2}.
%%   \end{itemize}
%% \end{theorem}

To prove the above, let ${\VC \defeq \decissubref{\Env}{p_1}{p_2}}$.
We prove that if the \VC is valid then
\undecissubtype{\Env}{\ttreft{\vv}{\base}{\p_1}}{\ttreft{\vv}{\base}{\p_2}}.
%%
%%First, note that
%%if $\VC$ is u-valid then it is valid as the addition of axioms preserves
%%validity. Next, we prove that if the \VC is valid then
%%\undechastype{\Env}{\ttreft{\vv}{\base}{\p_1}}{\ttreft{\vv}{\base}{\p_2}}.
%
This fact relies crucially on a notion of \emph{tracking evaluation}
which allows us to reduce a closing substitution $\sto$ to a lifted substitution
$\botsto$, written $\trackevals{\sto}{\botsto}$, after which we prove:

\begin{lemma}{[Lifting]}
$\evals{\sto(e)}{c}$ iff $\exists \trackevals{\sto}{\botsto}$ s.t. $\evals{\botsto(e)}{c}$.
\end{lemma}

We combine the Lifting Lemma and the equivalence Theorem~\ref{thm:equiv}
to prove that the validity of the \VC demonstrates
the denotational containment
$\forall \sto\in \interp{\Env}.
  		 \interp{\thetasub{\sto}{\ttreft{\vv}{\miBase}{\p_1}}}
  		$ $\subseteq$ $\interp{\thetasub{\sto}{\ttreft{\vv}{\miBase}{\p_2}}}$.
%
The soundness of algorithmic typing follows from
Theorems~\ref{thm:approximation} and~\ref{thm:refinedhaskell:safety}:

\begin{theorem} {[Soundness of \declang]}
\begin{itemize}
\item\textbf{Approximation:} If \dechastype{\emptyset}{e}{\typ} then
  \undechastype{\emptyset}{e}{\typ}.
\item\textbf{Crash-Freedom:} If \dechastype{\emptyset}{e}{\typ}
        then $\evals{e\not}{\ecrash}$.
\end{itemize}
\end{theorem}

To prove approximation we need to prove that Lemma~\ref{lemma:constants} holds for
each constant, and thus it holds for data
constructors.
In the metatheory we assume a stronger notion
of validity that respects the measure axioms.
%
%\RJ{CHECK}
However, since our implementation does not use axioms and instead,
without loss of precision, treats measures as uninterpreted
during SMT validity checking, we omit further discussion of axioms
for clarity.

%% \NV{DONE: remove Uninterpreted Validity Checking}
%%\mypara{Uninterpreted Validity Checking}
%%We reduce the undecidable implication
%%to checking u-validity of the \VC, \ie, validity
%%where measures are uninterpreted.
%%%
%%This trivially implies validity of the \VC.
%%%
%%Recall that we use u-validity instead of validity to ensure
%%predictable and efficient checking, as then the
%%\VC belongs to \logiclang.
%%%
%%The elimination of the measure axioms is crucial
%%for efficient and practical verification.
%%%
%%% As demonstrated by our evaluation (\Sref{sec:evaluation}),
%%The absence of the axioms does not lead to loss
%%of precision in practice; the semantics of the
%%axioms are encoded in the refinement types of
%%the data constructors, and hence already \emph{instantiated}
%%inside (the environment and) the VC during type checking.
%%

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "main"
%%% End:


\newpage
\subsection{Reconstruction filters}
\label{sec:rfilters}
Image reconstruction filters are responsible for converting a series of radiance samples generated
jointly by the \emph{sampler} and \emph{integrator} into the final output image that will be written
to disk at the end of a rendering process.
This section gives a brief overview of the reconstruction filters that are available in Mitsuba.
There is no universally superior filter, and the final choice depends on a trade-off between
sharpness, ringing, and aliasing, and computational efficiency.

Desireable properties of a reconstruction filter are that it sharply captures all of the details that
are displayable at the requested image resolution, while avoiding aliasing and ringing. Aliasing is
the incorrect leakage of high-frequency into low-frequency detail, and ringing denotes oscillation artifacts
near discontinuities, such as a light-shadow transiton.

\begin{description}
\item[Box filter (\code{box}):]
the fastest, but also about the worst possible
reconstruction filter, since it is extremely prone to aliasing.
It is included mainly for completeness, though some rare situations
may warrant its use.
\item[Tent filter (\code{tent}):]
Simple tent, or triangle filter. This reconstruction filter never
suffers from ringing and usually causes less aliasing than a naive
box filter. When rendering scenes with sharp brightness discontinuities,
this may be useful; otherwise, negative-lobed filters will be preferable
(e.g. Mitchell-Netravali or Lanczos Sinc)

\item[Gaussian filter (\code{gaussian}):]
this is a windowed Gaussian filter with configurable standard deviation.
It produces pleasing results and never suffers from ringing, but may
occasionally introduce too much blurring.
When no reconstruction filter is explicitly requested, this is the default
choice in Mitsuba. Takes a standard deviation parameter (\code{stddev})
which is set to 0.5 pixels by default.
\item[Mitchell-Netravali filter (\code{mitchell}):]
Separable cubic spline reconstruction filter by Mitchell and Netravali
\cite{Mitchell:1988:Reconstruction}
This is often a good compromise between sharpness and ringing.

The plugin has two \code{float}-valued parameters named \texttt{B} and \texttt{C} that
correspond to the two parameters in the original research paper. By default, these
are set to the recommended value of $1/3$, but can be tweaked if desired.

\item[Catmull-Rom filter (\code{catmullrom}):]
This is a special version of the Mitchell-Netravali filter that has the
constants \texttt{B} and \texttt{C} adjusted to produce higher sharpness at the
cost of increased susceptibility to ringing.

\item[Lanczos Sinc filter (\code{lanczos}):]
This is a windowed version of the theoretically optimal low-pass filter.
It is generally one of the best available filters in terms of producing sharp
high-quality output. Its main disadvantage is that it produces strong ringing around
discontinuities, which can become a serious problem when rendering bright objects
with sharp edges (for instance, a directly visible light source will have black
fringing artifacts around it).
This is also the computationally slowest reconstruction filter.

This plugin has an \code{integer}-valued parameter named \code{lobes}, that
sets the desired number of filter side-lobes. The higher, the closer
the filter will approximate an optimal low-pass filter, but this also
increases the susceptibility to ringing. Values of 2 or 3 are common (3 is the default).
\end{description}
The next section contains a series of comparisons between reconstruction filters. In the first
case, a very high-resolution input image (corresponding to a hypothetical radiance field
incident at the camera) is reconstructed at low resolutions.

\newpage
\subsubsection{Reconstruction filter comparison 1: frequency attenuation and aliasing}
\vspace{-2mm}
Here, a high frequency function is reconstructed at low resolutions. A good filter
(e.g. Lanczos Sinc) will capture all oscillations that are representable at the desired
resolution and attenuate the remainder to a uniform gray. The filters are ordered by their
approximate level of success at this benchmark.
\renderings{
    \subfloat[A high resolution input image whose frequency decreases
    towards the borders. If you are looking at this on a computer, you may
    have to zoom in.]{\fbox{\includegraphics[width=0.43\textwidth]{images/rfilter_sines_input}}}
    \hfill
}
\vspace{-4mm}
\renderings{
    \medrendering{Box filter}{rfilter_sines_box}
    \medrendering{Tent filter}{rfilter_sines_tent}
    \medrendering{Gaussian filter}{rfilter_sines_gaussian}
}
\vspace{-4mm}
\renderings{
    \setcounter{subfigure}{3}
    \medrendering{Mitchell-Netravali filter}{rfilter_sines_mitchell}
    \medrendering{Catmull-Rom filter}{rfilter_sines_catmullrom}
    \medrendering{Lanczos Sinc filter}{rfilter_sines_lanczos}
}
\newpage
\subsubsection{Reconstruction filter comparison 2: ringing}
This comparison showcases the ringing artifacts that can occur when the rendered
image contains extreme and discontinuous brightness transitions. The
Mitchell-Netravali, Catmull-Rom, and Lanczos Sinc filters are affected by this problem.
Note the black fringing around the light source in the cropped Cornell box renderings below.
\renderings{
    \rendering{Box filter}{rfilter_cbox_box}
    \rendering{Tent filter}{rfilter_cbox_tent}
}
\vspace{-4mm}
\renderings{
    \setcounter{subfigure}{2}
    \rendering{Gaussian filter}{rfilter_cbox_gaussian}
    \rendering{Mitchell-Netravali filter}{rfilter_cbox_mitchell}
}
\vspace{-4mm}
\renderings{
    \setcounter{subfigure}{4}
    \rendering{Catmull-Rom filter}{rfilter_cbox_catmullrom}
    \rendering{Lanczos Sinc filter}{rfilter_cbox_lanczos}
}
\subsubsection{Specifying a reconstruction filter}
To specify a reconstruction filter, it must be instantiated inside
the sensor's film. Below is an example:
\begin{xml}
<scene version=$\MtsVer$>
    <!-- ... scene contents ... -->

    <sensor type="... sensor type ...">
        <!-- ... sensor parameters ... -->

        <film type="... film type ...">
            <!-- ... film parameters ... -->

            <!-- Instantiate a Lanczos Sinc filter with two lobes -->
            <rfilter type="lanczos">
                <integer name="lobes" value="2"/>
            </rfilter>
        </film>
    </sensor>
</scene>
\end{xml}


To compare \picalc{} processes we define the \findex[refinement!success]success refinement and relate it to the strong simulation. We start by defining the \textit{success pair} of a process.

The pair $(t, Y)$ is called a \textit{success pair}, where $t$ is a trace and $Y$ is a set of acceptable next actions after $t$. Formally, $P \bigstep{t} Q$ and $Q$ acp $Y$. Any process
$P$ is assigned a set of success pairs $\Successes[]$. Formally, this means:
\begin{align}
    \Successes[P] \define \{(t,Y) \mid \exists Q\in\procs: P \bigstep{t} Q \wedge Q \text{ acp } Y \wedge Y\in Acceptances\}
\end{align}

where: $Acceptances \define \pom{\actions}$.
\\The special inclusion of success sets is defined as follows:
\begin{definition}[special inclusion of success set]
\label{def_success_inclutuion_ref}
Let $P,Q\in\procs$, then:
\begin{equation*}
\begin{aligned}
\Successes[P] \ddot{\subseteq} \Successes[Q] \Leftrightarrow {} & \forall (t_P, Y_P) \in \Successes[P] \textit{ then }\\
      & \ \ \ \exists (t_Q, Y_Q) \in \Successes[Q]: t_p = t_Q \wedge Y_P \subseteq Y_Q
\end{aligned}
\end{equation*}
\
\end{definition}
To determine the acceptance set $Y$ we need to collect all the possible output, input, and $\tau$ actions after the substitution of the bound names of the input actions. That requires collecting the available actions before the substitution of the bound names of the input actions.

To collect the available actions before the substitution of the bound names of the input actions we need to collect the available output actions $Av_\outA$, collect the available input actions $Av_\inA$, and the available $\tau$ actions $Av_\tau$ as follows:
\[Av_\outA\define\{ \alpha \in \outA \mid \alpha\text{ is a syntatically possible next action in P}\}\]
\[Av_\inA\define\{ \alpha \in \inA \mid \alpha\text{ is a syntatically possible next action in P}\}\]
\[Av_\tau\define\{ \tau \mid \tau\text{ is a syntatically possible next action in P}\}\]

Now, we can collect all the possible actions after the substitution of bound names of the input actions, where $Acc_\outA$ is the set of the possible output actions, $Acc_\inA$ is the set of the possible input actions, and $Acc_\tau$ is the set of the possible the $\tau$ actions defined as follows:
\[Acc_\outA \define Av_\outA\]
\begin{equation*}
\begin{aligned}
Acc_\inA \define{} & \{\alpha \in \inA \mid \exists \beta \in Av_\inA \wedge \sub{\alpha}=\sub{\beta} \wedge \alpha=\substitue{\vec{y}}{\vec{z}}\beta \\
      &  \text{ where } \vec{z}=\obj{\beta} \wedge \vec{y}=\obj{\alpha} \}
\end{aligned}
\end{equation*}
\begin{equation*}
\begin{aligned}
Acc_\tau \define{} & \{\tau \mid \forall \alpha \in Acc_\outA \text{ if } \exists \beta \in Acc_\inA  :\sub{\alpha}=\sub{\beta} \wedge \obj{\alpha}=\obj{\beta}  \} \\
      &  \cup Av_\tau
\end{aligned}
\end{equation*}
\[\sub{\alpha} \text{: returns the channel name through which the exchange occurs}\]
\[\obj{\alpha} \text{: return the exchanged names across the channel}\]

Finally, we can determine $Y$ as follows:
\[Y \define Acc_\tau \cup Acc_\outA \cup Acc_\inA\]

Let us take an example to illustrate our approach of calculating the acceptance set of a trace. Let $P\define(\inp{a}{x}.\out{x}{a} \mid \inp{b}{y}).P$.  \refFig{trace_set_acceptance} shows $\Traces[P]$ and how to calculate the acceptance set $Y$ of the empty trace $\out{}{}$. $P$ has only two syntactically available actions $\inp{a}{x}$ and $\inp{b}{d}$ as shown in the set $Av_\inA$. Once one of them occurs, its bound name must be substituted with the received name. This substitution is shown in $Acc_\inA$ for all possible received names. Finally, the acceptance set $Y$ of $\out{}{}$ is the union of $Acc_\outA$, $Acc_\inA$ and $Acc_\tau$.
The sets $Av_\outA$ and $Av_\tau$ are empty, because $P$ can neither make an output action nor $\tau$ action. Thus, $Acc_\outA$ and $Acc_\tau$ are empty too.

\begin{figure}[H]
\centering
\resizebox{0.8\textwidth}{!}{%
\begin{tikzpicture}
    % draw the sets
   \draw[] (-7.5,0) ellipse [x radius = 1.2cm, y radius = 5cm];
    \draw[] (-4.5,0) ellipse [x radius = 0.9cm, y radius = 3cm];
    % the texts
    \node at (-7.5,5.3) {$\Traces[P]$};
    \node (x1) at (-4.5,3.3) {$\out{}{}$};

    \node (y1) at (-7.5,4.5) {$\out{}{}$};
    \node at (-7.5,3.9) {$\out{}{\inp{a}{a}}$};
    \node at (-7.5,3.3) {$\out{}{\inp{a}{b}}$};
    \node at (-7.5,2.7) {$\out{}{\inp{a}{c}}$};
    \node at (-7.5,2.1) {$\dot$};
    \node at (-7.5,1.5) {$\dot$};
    \node at (-7.5,0.9) {$\out{}{\inp{b}{a}}$};
    \node at (-7.5,0.3) {$\out{}{\inp{b}{d}}$};
    \node at (-7.5,-0.3) {$\dot$};
    \node at (-7.5,-0.9) {$\dot$};
    \node at (-7.5,-1.5) {$\out{}{\inp{a}{a},\inp{a}{a}}$};
    \node at (-7.5,-2.1) {$\out{}{\inp{b}{a},\inp{a}{a}}$};
    \node at (-7.5,-2.7) {$\dot$};
    \node at (-7.5,-3.3) {$\dot$};

    \node at (-4.5,2.4) {$\inp{a}{a}$};
    \node at (-4.5,1.8) {$\inp{a}{b}$};
    \node at (-4.5,1.2) {$\inp{a}{c}$};
    \node at (-4.5,0.6) {$\dot$};
    \node at (-4.5,0) {$\dot$};
    \node at (-4.5,-0.6) {$\inp{b}{a}$};
    \node at (-4.5,-1.2) {$\inp{b}{d}$};
    \node at (-4.5,-1.8) {$\dot$};
    \node at (-4.5,-2.2) {$\dot$};

    \path (y1) edge (x1);

    % draw the sets
   \draw[] (-7.5,-10) ellipse [x radius = 0.9cm, y radius = 3cm];
    \draw[] (-5.5,-10) ellipse [x radius = 0.9cm, y radius = 3cm];
    \draw[] (-3.5,-10) ellipse [x radius = 0.9cm, y radius = 3cm];
    \draw[] (0.5,-10) ellipse [x radius = 0.9cm, y radius = 3cm];
    \draw[] (2.5,-10) ellipse [x radius = 0.9cm, y radius = 3cm];
    \draw[] (4.5,-10) ellipse [x radius = 0.9cm, y radius = 3cm];

    % the texts
    \node (w1) at (-7.5,-6.7) {$Av_\outA$};
    \node (w2) at (-5.5,-6.7) {$Av_\inA$};
    \node (w3) at (-3.5,-6.7) {$Av_\tau$};
    \node (w4) at (0.5,-6.7) {$Acc_\outA$};
    \node (w5) at (2.5,-6.7) {$Acc_\inA$};
    \node (w6) at (4.5,-6.7) {$Acc_\tau$};

    \node at (-7.5,-9.7) {$\varnothing$};

    \node at (-5.5,-9.1) {$\inp{a}{x}$};
    \node at (-5.5,-9.7) {$\inp{b}{y}$};

    \node at (-3.5,-9.7) {$\varnothing$};

    \node at (0.5,-9.7) {$\varnothing$};

    \node at (2.5,-7.5) {$\inp{a}{a}$};
    \node at (2.5,-8.1) {$\inp{a}{b}$};
    \node at (2.5,-8.7) {$\inp{a}{c}$};
    \node at (2.5,-9.4) {$\dot$};
    \node at (2.5,-10) {$\dot$};
    \node at (2.5,-10.6) {$\inp{b}{a}$};
    \node at (2.5,-11.2) {$\inp{b}{d}$};
    \node at (2.5,-11.8) {$\dot$};
    \node at (2.5,-12.4) {$\dot$};

    \node at (4.5,-9.7) {$\varnothing$};

    \node (m1)at (-4.5,-3.3) {$Y$};
    \path (w4) edge (m1);
    \path (w5) edge (m1);
    \path (w6) edge (m1);

    \node (k1)at (-7.5,-13) {$ $};
    \node (l1)at (0.5,-13) {$ $};
	\draw [dashed] (k1.south) to [out=-10,in=-170] (l1.south);

	\node (k2)at (-5.5,-13) {$ $};
    \node (l2)at (2.5,-13) {$ $};
	\draw [->] (k2.south) to [out=-10,in=-170] (l2.south);

	\node (k3)at (-3.5,-13) {$ $};
    \node (l3)at (4.5,-13) {$ $};
	\draw [dashed] (k3.south) to [out=-10,in=-170] (l3.south);

\end{tikzpicture}
}
\caption{Calculating the acceptance set $Y$ of the trace $\out{}{}$ of the processes $P$.}
\label{trace_set_acceptance}
\end{figure}
We can now define the success-refinement of \picalc{} processes as follows:

\begin{definition}[Success refinement]
\label{def_Successes_ref}
	Let $P,Q\in\procs$, then $P$ is a {success refinement} of $Q$ iff the inverse set inclusion of traces and success holds:
   \[Q \refiSuc P \Leftrightarrow  \Traces[P]\subseteq\Traces[Q] \wedge \Successes[P]\ddot{\subseteq}\Successes[Q]\]
\end{definition}

From \refPro{property2} and \refDef{def_Successes_ref} we can drive the following Corollary:

\begin{cor}[Simulation and Success refinement]
\label{cor_sim_acceptance_refinement}
Let $P,Q\in\procs$. If $Q$ strongly simulates $P$, then $P$ refines $Q$ in Success-Refinement model. Formally written:
    \[(P,Q)\in{}\simu \Rightarrow Q \refiSuc P\]
holds.
\end{cor}%%

\begin{prf}\footnote{Our proof is not complete and can be enhanced in later works.}
Let $(P,Q)\in{}\simuv$, then $\Traces[P]\subseteq\Traces[Q] \wedge \Successes[P]\subseteq\Successes[Q]$ holds, Since:
\begin{itemize}
\item For $\Traces[P]\subseteq\Traces[Q]$: it holds using \refPro{property2}.
\item For $\Successes[P]\ddot{\subseteq}\Successes[Q]$: we need to show that,\\$\forall (t_P, Y_P) \in \Successes[P] \textit{ then } \exists (t_Q, Y_Q) \in \Successes[Q]: t_p = t_Q \wedge Y_P \subseteq Y_Q$
\begin{itemize}
\item $t_p = t_Q$ holds using \refPro{property2}.
\item $Y_P \subseteq Y_Q$ holds, since $Q$ strongly simulates $P$ means that $Q$ can do any the actions that $P$ can do respecting the use of the same free and bound names, what implies that $Q$ can do any trace $t$ that $P$ does. This implies that after any trace $t$, $Q$ accepts all the actions that $P$ accepts. This means that the acceptance set of $P$ is included in the acceptance set of $Q$ after any trace $t$. Formally, $Y_P \subseteq Y_Q$ for any trace $t$.
\end{itemize}
\end{itemize}
\end{prf}
\subsubsection{Success-Refinement use case}
Let us take an example to inspect if \refCor{cor_sim_acceptance_refinement} and our approach of calculating the acceptance set works. Let:\\
\[P\define(\inp{a}{x}.\out{x}{a} \mid \inp{b}{y}).P\]
\[Q\define(\inp{a}{x}.\out{x}{a} \mid \inp{b}{y}).Q + \tau.Q\]
It is clear that $Q$ strongly simulates $P$. This result implies, according to \refCor{cor_sim_acceptance_refinement}, that $P$ refines $Q$ in the success-refinement model. In other words, after a common trace $t$, the acceptance set $Y_P$ of the process $P$ is included in the acceptance set $Y_Q$ of the process $Q$. More formally:
\[\forall (t, Y_P) \in \Successes[P] \textit{ then } \exists (t, Y_Q) \in \Successes[Q]: Y_P \subseteq Y_Q\]
We will show that this holds for the empty trace $\out{}{}$ using our approach of determining the acceptance set $Y$.
This can be seen in \refFig{trace_set_acceptance} and \refFig{trace_set_acceptance2}. We can notice that there are no differences between $\Traces[P]$ and $\Traces[Q]$, since the $\tau$ action that $Q$ can do, cannot appear in the trace. But $\tau$ appears in the acceptance set $Y$ of the empty trace $\out{}{}$ of $Q$ shown in \refFig{trace_set_acceptance2}. In the same manner, we can determine $Y_P$ and $Y_Q$ for any trace $t$ and see that $Y_P \subseteq Y_Q$ holds. Thus, $\Successes[P]\ddot{\subseteq}\Successes[Q]$ holds and so $Q \refiSuc P$.

\begin{figure}[H]
\centering
\resizebox{0.8\textwidth}{!}{%
\begin{tikzpicture}
    % draw the sets
   \draw[] (-7.5,0) ellipse [x radius = 1.2cm, y radius = 5cm];
    \draw[] (-4.5,0) ellipse [x radius = 0.9cm, y radius = 3cm];
    % the texts
    \node at (-7.5,5.3) {$\Traces[Q]$};
    \node (x1) at (-4.5,3.3) {$\out{}{}$};

    \node (y1) at (-7.5,4.5) {$\out{}{}$};
    \node at (-7.5,3.9) {$\out{}{\inp{a}{a}}$};
    \node at (-7.5,3.3) {$\out{}{\inp{a}{b}}$};
    \node at (-7.5,2.7) {$\out{}{\inp{a}{c}}$};
    \node at (-7.5,2.1) {$\dot$};
    \node at (-7.5,1.5) {$\dot$};
    \node at (-7.5,0.9) {$\out{}{\inp{b}{a}}$};
    \node at (-7.5,0.3) {$\out{}{\inp{b}{d}}$};
    \node at (-7.5,-0.3) {$\dot$};
    \node at (-7.5,-0.9) {$\dot$};
    \node at (-7.5,-1.5) {$\out{}{\inp{a}{a},\inp{a}{a}}$};
    \node at (-7.5,-2.1) {$\out{}{\inp{b}{a},\inp{a}{a}}$};
    \node at (-7.5,-2.7) {$\dot$};
    \node at (-7.5,-3.3) {$\dot$};
    \node at (-7.5,-3.9) {$\tau$};

    \node at (-4.5,2.4) {$\inp{a}{a}$};
    \node at (-4.5,1.8) {$\inp{a}{b}$};
    \node at (-4.5,1.2) {$\inp{a}{c}$};
    \node at (-4.5,0.6) {$\dot$};
    \node at (-4.5,0) {$\dot$};
    \node at (-4.5,-0.6) {$\inp{b}{a}$};
    \node at (-4.5,-1.2) {$\inp{b}{d}$};
    \node at (-4.5,-1.8) {$\dot$};
    \node at (-4.5,-2.2) {$\tau$};

    \path (y1) edge (x1);

    % draw the sets
   \draw[] (-7.5,-10) ellipse [x radius = 0.9cm, y radius = 3cm];
    \draw[] (-5.5,-10) ellipse [x radius = 0.9cm, y radius = 3cm];
    \draw[] (-3.5,-10) ellipse [x radius = 0.9cm, y radius = 3cm];
    \draw[] (0.5,-10) ellipse [x radius = 0.9cm, y radius = 3cm];
    \draw[] (2.5,-10) ellipse [x radius = 0.9cm, y radius = 3cm];
    \draw[] (4.5,-10) ellipse [x radius = 0.9cm, y radius = 3cm];

    % the texts
    \node (w1) at (-7.5,-6.7) {$Av_\outA$};
    \node (w2) at (-5.5,-6.7) {$Av_\inA$};
    \node (w3) at (-3.5,-6.7) {$Av_\tau$};
    \node (w4) at (0.5,-6.7) {$Acc_\outA$};
    \node (w5) at (2.5,-6.7) {$Acc_\inA$};
    \node (w6) at (4.5,-6.7) {$Acc_\tau$};

    \node at (-7.5,-9.7) {$\varnothing$};

    \node at (-5.5,-9.1) {$\inp{a}{x}$};
    \node at (-5.5,-9.7) {$\inp{b}{y}$};

    \node at (-3.5,-9.7) {$\tau$};

    \node at (0.5,-9.7) {$\varnothing$};

    \node at (2.5,-7.5) {$\inp{a}{a}$};
    \node at (2.5,-8.1) {$\inp{a}{b}$};
    \node at (2.5,-8.7) {$\inp{a}{c}$};
    \node at (2.5,-9.4) {$\dot$};
    \node at (2.5,-10) {$\dot$};
    \node at (2.5,-10.6) {$\inp{b}{a}$};
    \node at (2.5,-11.2) {$\inp{b}{d}$};
    \node at (2.5,-11.8) {$\dot$};
    \node at (2.5,-12.4) {$\dot$};

    \node at (4.5,-9.7) {$\tau$};

    \node (m1)at (-4.5,-3.3) {$Y$};
    \path (w4) edge (m1);
    \path (w5) edge (m1);
    \path (w6) edge (m1);

    \node (k1)at (-7.5,-13) {$ $};
    \node (l1)at (0.5,-13) {$ $};
	\draw [dashed] (k1.south) to [out=-10,in=-170] (l1.south);

	\node (k2)at (-5.5,-13) {$ $};
    \node (l2)at (2.5,-13) {$ $};
	\draw [->] (k2.south) to [out=-10,in=-170] (l2.south);

	\node (k3)at (-3.5,-13) {$ $};
    \node (l3)at (4.5,-13) {$ $};
	\draw [dashed] (k3.south) to [out=-10,in=-170] (l3.south);
\end{tikzpicture}
}
\caption{Calculating the acceptance set $Y$ of the trace $\out{}{}$ of the processes $Q$.}
\label{trace_set_acceptance2}
\end{figure}


    \filetitle{fevd}{Forecast error variance decomposition for SVAR variables}{SVAR/fevd}

	\paragraph{Syntax}\label{syntax}

\begin{verbatim}
[X,Y,XX,YY] = fevd(V,NPer)
[X,Y,XX,YY] = fevd(V,Range)
\end{verbatim}

\paragraph{Input arguments}\label{input-arguments}

\begin{itemize}
\item
  \texttt{V} {[} VAR {]} - Structural VAR model.
\item
  \texttt{NPer} {[} numeric {]} - Number of periods.
\item
  \texttt{Range} {[} numeric {]} - Date range.
\end{itemize}

\paragraph{Output arguments}\label{output-arguments}

\begin{itemize}
\item
  \texttt{X} {[} namedmat \textbar{} numeric {]} - Forecast error
  variance decomposition into absolute contributions of residuals;
  absolute contributions sum up to the total variance.
\item
  \texttt{Y} {[} namedmat \textbar{} numeric {]} - Forecast error
  variance decomposition into relative contributions of residuals;
  relative contributions sum up to \texttt{1}.
\item
  \texttt{XX} {[} tseries {]} - Database with a tseries with absolute
  contributions in columns for each VAR variable.
\item
  \texttt{YY} {[} tseries {]} - Database with a tseries with relative
  contributions in columns for each VAR variable.
\end{itemize}

\paragraph{Options}\label{options}

\begin{itemize}
\itemsep1pt\parskip0pt\parsep0pt
\item
  \texttt{'matrixFmt='} {[} \emph{\texttt{'namedmat'}} \textbar{}
  \texttt{'plain'} {]} - Return matrices \texttt{X} and \texttt{Y} as be
  either \href{namedmat/Contents}{\texttt{namedmat}} objects
  (i.e.~matrices with named rows and columns) or plain numeric arrays.
\end{itemize}

\paragraph{Description}\label{description}

The output matrices \texttt{X} and \texttt{Y} are Ny-by-Ny-by-Nt-by-NAlt
namedmat objects (matrices with named rows and columns), where Ny is the
number of endogenous variables (and hence also structural residuals), Nt
is the number of periods, and NAlt is the number of alternative
parameterizations.

The output databases \texttt{XX} and \texttt{YY} contain
Nt-by-Ny-by-NAlt tseries objects (one for each endogenous variable).

\paragraph{Example}\label{example}