\section{Evaluation}
\label{sect:prob-evaluation}

We evaluated various techniques to solve the proposed PPE formulation.
Specifically, we focus on PEC and MPEC problems,
as equivalence checking is widely encountered in automated design verification.

For the worst-case analysis (i.e., MPEC), two SSAT-based techniques were evaluated.
The signal-probability approach via BDD-based SSAT solving
is implemented in the \texttt{C} language inside the \abc~\cite{ABC} environment.
The BDD operations are handled by the \cudd~\cite{CUDD} package.
Our prototyping implementation\footnote{Available at: \url{\ssatabcurl}}
of the signal-probability approach is named \bddsp.
A version of \bddsp without variable reordering during the BDD construction
is called \bddspnr in the experiments.
We used \ssatABCRevision in the experiments.
For the CNF-based SSAT approach,
we employ the state-of-the-art DPLL-based SSAT solver \dcssat~\cite{Majercik2005},
which was kindly provided by its author Majercik.

For the average-case analysis (i.e., PEC),
in addition to the above two SSAT-based solutions,
two techniques based on model counting were also evaluated.
A classic exact model counter \cachet~\cite{Sang2004,Sang2005ModelCounting} is used
for the solution via exact weighted model counting.
Thanks to the proposed formula rewriting
that converts an instance of weighted model counting into an unweighted one,
we are able to leverage the latest developments of approximate model counting.
A state-of-the-art approximate model counter \approxmc-4.0.1~\cite{Chakraborty2013,Chakraborty2016}
is applied to solve the converted instances.
To achieve a relatively fair comparison with the other exact techniques,
we set the epsilon parameter to $0.99$ and the delta parameter to $0.01$ for \approxmc.

\subsection{Benchmark set}
\begin{table}[ht]
    \centering
    \footnotesize
    \caption{Circuit statistics of ISCAS benchmark suite}
    \label{tbl:prob-design-eval-iscas}
    \pgfplotstabletypeset[
        every head row/.style={before row=\toprule,after row=\midrule},
        every last row/.style={after row=\bottomrule},
        circuit column/.list={0},
        pi column/.list={1},
        po column/.list={2},
        and column/.list={3},
        level column/.list={4}
    ]
    {prob-design-eval/evaluation/csv/parsed-ISCAS.csv}
\end{table}

Combinational circuits from the ISCAS\,'85~\cite{ISCAS85-benchmark}
and EPFL~\cite{EPFL-benchmark} benchmark suites were used in the experiments.
The circuit statistics are shown in~\cref{tbl:prob-design-eval-iscas,tbl:prob-design-eval-epfl}.
We represent the circuits as and-inverter graphs (AIGs).
Although the AIG representation is chosen in our experiments,
the proposed formulation and solutions are applicable to other circuit representations,
such as FPGA LUT-based or standard cell-based designs.

\begin{table}[hp]
    \centering
    \footnotesize
    \caption{Circuit statistics of EPFL benchmark suite}
    \label{tbl:prob-design-eval-epfl}
    \pgfplotstabletypeset[
        every head row/.style={before row=\toprule,after row=\midrule},
        every last row/.style={after row=\bottomrule},
        circuit column/.list={0},
        pi column/.list={1},
        po column/.list={2},
        and column/.list={3},
        level column/.list={4}
    ]
    {prob-design-eval/evaluation/csv/parsed-EPFL.csv}
\end{table}

\begin{table}[ht]
    \centering
    \footnotesize
    \caption{Miter statistics of ISCAS benchmark suite ($\dr=0.01$)}
    \label{tbl:prob-design-eval-miter-iscas-0.01}
    \pgfplotstabletypeset[
        every head row/.style={before row=\toprule,after row=\midrule},
        every last row/.style={after row=\bottomrule},
        circuit column/.list={0},
        pi column/.list={1},
        ai column/.list={2},
        and column/.list={3},
        level column/.list={4}
    ]
    {prob-design-eval/evaluation/csv/parsed-ISCAS-Miter-D-0.01.csv}
\end{table}

In order to generate circuits with probabilistic errors,
we specify an error rate $\er$ of a probabilistic gate and
a defection rate $\dr$ of an entire design (i.e., the ratio of erroneous gates to all gates in a design).
We fixed both parameters to constants for evaluation
and selected probabilistic gates at random according to the defection rate.
In our experiments, $\er$ was set to $0.125$, and $\dr$ was set to $0.01$ and $0.1$.
To solve probabilistic equivalence checking,
a miter circuit that compares a design with probabilistic errors against its error-free counterpart is built.
Since the two designs are structurally identical except for the probabilistic gates,
logic synthesis may achieve substantial reduction when the number of erroneous nodes is small.
In our evaluation,
we synthesized a miter with the script \texttt{resyn2} in \abc before applying the discussed techniques.
The statistics of the synthesized miters are shown in~\cref{tbl:prob-design-eval-miter-iscas-0.01,tbl:prob-design-eval-miter-iscas-0.10,tbl:prob-design-eval-miter-epfl-0.01,tbl:prob-design-eval-miter-epfl-0.10}.
Note that the numbers of auxiliary inputs (AIs) in these tables are the numbers of erroneous gates in the circuits.
Therefore, a circuit would have more AIs if the specified defection rate is larger.

\begin{table}[ht]
    \centering
    \footnotesize
    \caption{Miter statistics of ISCAS benchmark suite ($\dr=0.1$)}
    \label{tbl:prob-design-eval-miter-iscas-0.10}
    \pgfplotstabletypeset[
        every head row/.style={before row=\toprule,after row=\midrule},
        every last row/.style={after row=\bottomrule},
        circuit column/.list={0},
        pi column/.list={1},
        ai column/.list={2},
        and column/.list={3},
        level column/.list={4}
    ]
    {prob-design-eval/evaluation/csv/parsed-ISCAS-Miter-D-0.10.csv}
\end{table}

\begin{table}[hp]
    \centering
    \footnotesize
    \caption{Miter statistics of EPFL benchmark suite ($\dr=0.01$)}
    \label{tbl:prob-design-eval-miter-epfl-0.01}
    \pgfplotstabletypeset[
        every head row/.style={before row=\toprule,after row=\midrule},
        every last row/.style={after row=\bottomrule},
        circuit column/.list={0},
        pi column/.list={1},
        ai column/.list={2},
        and column/.list={3},
        level column/.list={4}
    ]
    {prob-design-eval/evaluation/csv/parsed-EPFL-Miter-D-0.01.csv}
\end{table}

\begin{table}[hp]
    \centering
    \footnotesize
    \caption{Miter statistics of EPFL benchmark suite ($\dr=0.1$)}
    \label{tbl:prob-design-eval-miter-epfl-0.10}
    \pgfplotstabletypeset[
        every head row/.style={before row=\toprule,after row=\midrule},
        every last row/.style={after row=\bottomrule},
        circuit column/.list={0},
        pi column/.list={1},
        ai column/.list={2},
        and column/.list={3},
        level column/.list={4}
    ]
    {prob-design-eval/evaluation/csv/parsed-EPFL-Miter-D-0.10.csv}
\end{table}

\subsection{Experimental setup}
Our experiments were performed on a machine with~\machineSpec.
The operating system was~\osInfo.
The programs were compiled with~\compiler.
Each PEC or MPEC task was limited to a CPU core,
a CPU time of~\timelimit,
and a memory usage of~\memlimit.
To achieve reliable benchmarking,
we used a benchmarking framework \benchexec\footnote{Available at: \url{\benchexecurl}}~\cite{Benchmarking-STTT},
and assumed~\measurement.

\subsection{Results}

\subsubsection{PEC instances}
The solving results of the PEC instances with the defection rate $\dr$ equal $0.01$ and $0.1$
are shown in~\cref{tbl:prob-design-eval-pec-0.01,tbl:prob-design-eval-pec-0.10}, respectively.
The average probabilities for the two circuits to have different outputs and the CPU time are reported.
The symbol ``-'' in an entry of the tables indicates the execution ran out of computational resource.
An instance that cannot be solved by any technique is not shown in the tables.

We observe the following phenomenons from these tables.
First of all, the defection rate strongly influences the performance of all methods.
As discussed above, the number of AIs is proportional to the defection rate.
More AIs result in more inputs to a miter and more variables in a formula.
Therefore, all of the proposed techniques performed worse when $\dr=0.1$.
Second, \bddsp solved many instances in the shortest time among all compared methods.
Recall that the complexity of the proposed BDD-based SSAT solving is linear to the numbers of BDD nodes.
The satisfying probability is readily available as long as the BDD of a miter can be built.
On the other hand, \dcssat and \cachet are less competitive in the evaluation.
Both of them failed to handle circuits with more than a thousand gates.
Finally, \approxmc achieved the best scalability.
Although it tends to spend more time for small- and medium-sized circuits than \bddsp did,
it uniquely solved three instances when $\dr=0.01$ and four instances when $\dr=0.1$.
Enabling any model counter to deal with weighted model counting,
the proposed formula-rewriting technique demonstrates its value
in improving the scalability to solve the PEC problem.

\begin{table}[hp]
    \centering
    \footnotesize
    \caption{Solving PEC by various techniques ($\dr=0.01$)}
    \label{tbl:prob-design-eval-pec-0.01}
    \pgfplotstabletypeset[
        every head row/.style={before row={\toprule
                        & \multicolumn{4}{c}{\bddsp} & \multicolumn{4}{c}{\dcssat} & \multicolumn{4}{c}{\cachet} & \multicolumn{4}{c}{\approxmc}\\},after row=\midrule},
        every last row/.style={after row=\bottomrule},
        empty cells with={--},
        circuit column/.list={0},
        time column/.list={1,3,5,7},
        prob column/.list={2,4,6,8}
    ]
    {prob-design-eval/evaluation/csv/parsed-PEC-D-0.01.csv}
\end{table}

\begin{table}[ht]
    \centering
    \footnotesize
    \caption{Solving PEC by various techniques ($\dr=0.1$)}
    \label{tbl:prob-design-eval-pec-0.10}
    \pgfplotstabletypeset[
        every head row/.style={before row={\toprule
                        & \multicolumn{4}{c}{\bddsp} & \multicolumn{4}{c}{\dcssat} & \multicolumn{4}{c}{\cachet} & \multicolumn{4}{c}{\approxmc}\\},after row=\midrule},
        every last row/.style={after row=\bottomrule},
        empty cells with={--},
        circuit column/.list={0},
        time column/.list={1,3,5,7},
        prob column/.list={2,4,6,8}
    ]
    {prob-design-eval/evaluation/csv/parsed-PEC-D-0.10.csv}
\end{table}

\subsubsection{MPEC instances}
\begin{table}[ht]
    \centering
    \footnotesize
    \caption{Solving MPEC by various techniques ($\dr=0.01$)}
    \label{tbl:prob-design-eval-mpec-0.01}
    \pgfplotstabletypeset[
        every head row/.style={before row={\toprule
                        & \multicolumn{4}{c}{\bddsp} & \multicolumn{4}{c}{\bddspnr} & \multicolumn{4}{c}{\dcssat}\\},after row=\midrule},
        every last row/.style={after row=\bottomrule},
        empty cells with={--},
        circuit column/.list={0},
        time column/.list={1,3,5},
        prob column/.list={2,4,6}
    ]
    {prob-design-eval/evaluation/csv/parsed-MPEC-D-0.01.csv}
\end{table}

\begin{table}[ht]
    \centering
    \footnotesize
    \caption{Solving MPEC by various techniques ($\dr=0.1$)}
    \label{tbl:prob-design-eval-mpec-0.10}
    \pgfplotstabletypeset[
        every head row/.style={before row={\toprule
                        & \multicolumn{4}{c}{\bddsp} & \multicolumn{4}{c}{\bddspnr} & \multicolumn{4}{c}{\dcssat}\\},after row=\midrule},
        every last row/.style={after row=\bottomrule},
        empty cells with={--},
        circuit column/.list={0},
        time column/.list={1,3,5},
        prob column/.list={2,4,6}
    ]
    {prob-design-eval/evaluation/csv/parsed-MPEC-D-0.10.csv}
\end{table}

The solving results of the MPEC instances with the defection rate $\dr$ equal $0.01$ and $0.1$
are shown in~\cref{tbl:prob-design-eval-mpec-0.01,tbl:prob-design-eval-mpec-0.10}, respectively.
The maximum probabilities for the two circuits to have different outputs and the CPU time are reported.
The symbol ``-'' in an entry of the tables indicates the execution ran out of computational resource.
An instance that cannot be solved by any technique is not shown in the tables.

We included \bddspnr (i.e., \bddsp without variable reordering) in the MPEC experiments.
In order to build a smaller BDD,
the \cudd package automatically reorders variables during the BDD construction by default.
Variable reordering usually achieves better results in our empirical experience.
For PEC instance, since both the PI and the AI variables are randomly quantified,
the variable order in BDD construction is irrelevant to signal probability computation.
Therefore, we always enabled this option in the PEC experiments.

However, variable reordering is problematic under the MPEC scenario.
For MPEC instances, since the PI variables are existentially quantified and the AI variables are randomly quantified,
the PI variables must be placed before the AI variables.
The default reordering, nevertheless, might violate such structure.
To make variable reordering respect the quantification prefix,
we classify PI and AI variables into two separate groups and
require that variable reordering can only happen within each group.
In the following experiments, we use \bddspnr as a baseline
to evaluate the performance of the constrained variable reordering.

From these tables, we observe the following phenomenons.
First, the property violation probabilities between the average case and the worst case
can differ for several orders of magnitude.
For example, the difference for circuit \texttt{router} when $\dr=0.01$ is about three orders of magnitude.
This observation shows the criticality of the MPPE formulation for scenarios
where the worst-case analysis is concerned.
Second, same as the PEC experiments, \bddsp performed much better than \dcssat did.
Regarding the reordering of variables,
the performance difference is not very consistent.
There are cases which can be solved by \bddsp within seconds but cannot be solved by \bddspnr, and vice versa.
Constrained variable reordering is still useful in general.
However, if the original variable order happens to be good enough for BDD construction,
enabling the constrained reordering would just incur overhead for run-time.

The above results on the PEC and MPEC instances suggest that:
\begin{itemize}
    \item The proposed BDD-based SSAT solver \bddsp achieves the best performance for small- and medium-sized circuits, whose BDDs can be constructed easily.
    \item The state-of-the-art approximate model counter \approxmc has the best scalability in our evaluation, which shows the unique value of the proposed formula rewriting.
    \item The exact weighted model counter \cachet and the CNF-based SSAT solver \dcssat do not scale well when the circuit size grows.
\end{itemize}


\chapter{Conclusions and Future Work}
This chapter is dedicated to drawing conclusions based on results found and make recommendation on future iterations of this project. Since the underlying methodology is quite novel this system should serve as the foundational step to a fully automated motion capture system.

\section{Conclusion}
The aim of this project was threefold: design a novel wearable data capture system using cameras and sensors to record various parameters of a running bipedal subject, develop a kinematic model to interpret the gathered data points, and design an estimation algorithm to process the captured data. These goals were achieved within certain constraints as discussed in the results section.

The system proved less accurate than modern wearable sensor suits,

Due to the relatively small dataset used in this project it serves as a proof of concept and a foundational work that can be iteratively improved.

\section{Future work}
The system was originally designed with four cameras due to the availability of equipment and the assumption that stereo vision would be implemented. For this a set of front and rear mounted cameras would be necessary. Originally there was an intention to reduce the total amount of cameras.

The iterative design process would have used the following roadmap. Initially using 4 cameras and an smartphone as a sensor. Then reducing the system to two cameras (one mounted to the back of the runner and one mounted to the chest of the runner) and a smartphone as a sensor. The next iteration would use the the smartphone camera at the front and a single camera at the back whilst using the smartphone as a sensor as well. This system greatly reduces the cost of the original design philosophy, even given that a powerful and expensive modern smartphone would be needed. The final iteration would use only the smartphone at the front as a single camera and sensor. For this method to work the estimation algorithm would need to better understand the periodic motion of the human gait and the model would need to increase in complexity.

There is however a fundamental shortcoming in the proposed methodologies. As discussed the uncertainty in the camera pose creates difficulties when trying to align the different frames of reference. This can be rectified by using purpose built IMUs attached to the cameras to better estimate their pose and incorporate it into the kinematic equations.

Due to the labour intensive approach taken to image processing a further avenue for improvement is the automation of feature detection. The progression would start from the current implementation of markers and the use of a semi-automatic toolbox to identify critical points on the image. Next, computer algorithms should be created to automate the image processing with markers. This could allow for much longer runs and larger datasets to be studied, introducing elements such as fatigue and other running modifiers. The next iterative step would be to remove the markers from the runner such that the set up time of the system is reduced. This is a difficult problem to solve using classical image processing as the variables relating to the runner and the environment are not constant. Perhaps a neural network can be trained to identify the different elements of the lower limbs.

These improvements would decrease the overall cost of the system and optimize the the process substantially. This decrease in hardware does imply that the complexity of the underlying algorithms and models would increase. This trade-off can be considered for future developments.

Finally the results obtained have not been verified against a true data source. An experiment to verify these results was designed, where external cameras would generate three dimensional positional data of the markers. This can be compared to the data generated from the EKF as applied to the wearable motion capture system by creating visual animations from both sources and comparing them. The system can also be verified by using a full body sensor based motion capture suit as designed by XSens.


\documentclass[]{article}
\usepackage{lmodern}
\usepackage{amssymb,amsmath}
\usepackage{ifxetex,ifluatex}
\usepackage{fixltx2e} % provides \textsubscript
\ifnum 0\ifxetex 1\fi\ifluatex 1\fi=0 % if pdftex
  \usepackage[T1]{fontenc}
  \usepackage[utf8]{inputenc}
\else % if luatex or xelatex
  \ifxetex
    \usepackage{mathspec}
    \usepackage{xltxtra,xunicode}
  \else
    \usepackage{fontspec}
  \fi
  \defaultfontfeatures{Mapping=tex-text,Scale=MatchLowercase}
  \newcommand{\euro}{€}
\fi
% use upquote if available, for straight quotes in verbatim environments
\IfFileExists{upquote.sty}{\usepackage{upquote}}{}
% use microtype if available
\IfFileExists{microtype.sty}{%
\usepackage{microtype}
\UseMicrotypeSet[protrusion]{basicmath} % disable protrusion for tt fonts
}{}
\ifxetex
  \usepackage[setpagesize=false, % page size defined by xetex
              unicode=false, % unicode breaks when used with xetex
              xetex]{hyperref}
\else
  \usepackage[unicode=true]{hyperref}
\fi
\hypersetup{breaklinks=true,
            bookmarks=true,
            pdfauthor={},
            pdftitle={Foo},
            colorlinks=true,
            citecolor=blue,
            urlcolor=blue,
            linkcolor=magenta,
            pdfborder={0 0 0}}
\urlstyle{same}  % don't use monospace font for urls
\setlength{\parindent}{0pt}
\setlength{\parskip}{6pt plus 2pt minus 1pt}
\setlength{\emergencystretch}{3em}  % prevent overfull lines
\setcounter{secnumdepth}{5}

\title{Foo}
\date{}
\usepackage{graphicx}
\usepackage[all]{hypcap}
\usepackage{tikz}
\usepackage{standalone}
\usepackage[sort&compress, numbers]{natbib}
\usepackage{minted}

\begin{document}
\maketitle

\section{Math}\label{math}

\begin{equation}\label{eq:ellipse}
Ax^2 + Bxy + Cy^2 + Dx + Ey + F = 0
\end{equation}

Refering equation \eqref{eq:ellipse}, \(\forall n, x_i^n = x_i\).

\section{Image}\label{image}

See following figure(\autoref{fig:xor}).

\begin{figure}[h]\centering\includegraphics[width=\textwidth]{.//assets/images/xor.png}\caption{Figure Example}\label{fig:xor}\end{figure}

\section{Tex2img}\label{tex2img}

See following figure(\autoref{fig:test}).

\begin{figure}[h]\centering\input{./misc/test.tex}\caption{Test tex figure}\label{fig:test}\end{figure}

\section{Gist}\label{gist}

\inputminted[mathescape, linenos, frame=lines, framesep=2mm]{Python}{.gist-cache/cache.7aeefc0de1bb10005514355a5c4a5dfe-1c74b8336494cb0e9c6d-xor-5d.py}

\section{Post Link}\label{post-link}

\href{http://localhost:4000/2015/04/29/foo/}{Foo}

Awesome paper\cite{mikolov2013efficient}.

\bibliographystyle{unsrt}

\bibliography{assets/printables/references}

\renewcommand{\thefootnote}{}

\footnotetext{Online version at \url{http://localhost:4000/2015/04/29/foo/}}

\end{document}