\documentclass[12pt]{article}

\usepackage{amsmath,amssymb,amsfonts}
\usepackage{psfrag}
\usepackage{color}
\definecolor{darkblue}{rgb}{0.1,0.1,.7}
\usepackage[colorlinks, linkcolor=darkblue, citecolor=darkblue, urlcolor=darkblue, linktocpage]{hyperref}
\usepackage[square, comma, compress,numbers]{natbib}
\usepackage[]{amsmath}
\usepackage[]{graphicx}
\usepackage[]{latexsym}
\usepackage{geometry}
\usepackage{amscd}
\usepackage[all,cmtip]{xy}
\usepackage{mathrsfs}
\usepackage[margin=10pt,font=small,labelfont=bf]{caption}
\geometry{verbose,letterpaper,tmargin=2.5cm,bmargin=2.5cm,lmargin=2.6cm,rmargin=2.6cm}
\usepackage{dsdshorthand}
\usepackage{changepage}
\usepackage{listings}
\setlength{\parskip}{0.1in}
\hyphenpenalty=1000

\numberwithin{equation}{section}

\renewcommand{\be}{\begin{eqnarray}}
\renewcommand{\ee}{\end{eqnarray}}
\newcommand\nn{\nonumber}
\newcommand\cS{\mathcal{S}}
\newcommand\cR{\mathcal{R}}
\newcommand\SDPAGMP{\texttt{SDPA-GMP}}
\newcommand\SDPB{\texttt{SDPB}}
\newcommand\repl[1]{$\langle$\textrm{\em #1}$\rangle$}
\newcommand\defn[1]{\textrm{\em #1}\ $\equiv$}

\begin{document}

{\Large
\begin{center}
{\bf Outer Limits \\\vspace{.1in}}
\end{center}
}
\begin{center}
\noindent \today
\end{center}
\tableofcontents
\section{Introduction}
\label{sec:introduction}

\texttt{Outer\_Limits} is an experimental arbitrary-precision
semidefinite program solver, much like \texttt{SDPB}, but with a
markedly different strategy.  \texttt{SDPB} will find a solution by
adding up polynomial matrices which are each guaranteed to be
semidefinite.  This sum becomes more and more ill-conditioned matrix
as \texttt{SDPB} iterates towards a solution.  This ill-conditioned
matrix must be inverted in order to continue to the next iteration,
causing numerical problems.  \texttt{SDPB} overcomes this by requiring
very large precision arithmetic.

\texttt{Outer\_Limits}, on the other hand, starts by enforcing positive
semidefiniteness only at a finite set of points.  These are constant
constraints, so there is no longer a problem with ill-conditioned
matrices.  This allows \texttt{Outer\_Limits} to find a solution using
much lower precision.

Initially, this will find a solution that is, in general, not positive
semidefinite everywhere.  So \texttt{Outer\_Limits} will add more
points to the places where the function is not positive semidefinite
and find another solution.  This iterative procedure continues until
the solution is positive everywhere.

This strategy also allows \texttt{Outer\_Limits} to be more easily
applied to general problems.  \texttt{SDPB} requires that all inputs
be approximated as polynomials.  \texttt{Outer\_Limits}, in contrast,
only requires evaluations of the functions at zeros of Chebyshev
polynomials.

\texttt{Outer\_Limits} is bundled with \texttt{SDPB}.  See
\texttt{Install.md} for up-to-date instructions on getting pre-made
binaries or building from source.

\section{Semidefinite Problems}
\label{sec:semidefinite}

Consider a system with $J$ blocks.  Within each block, there is a set
of inequality constraints on an $N$-dimensional vector $y$.  In
addition, there is a known vector $b$, independent of the blocks.  The
goal of semidefinite programming is to find a solution to these
constraints while trying to maximize the quantity $b\cdot y$.

Writing this more formally, consider a collection of symmetric $R_{j}\times R_{j}$ matrices
\begin{eqnarray}
\label{eq:sdp}
M_j^n(\Delta) &=& \begin{pmatrix}
m_{j,11}^{n}(\Delta) & \dots & m_{j,1m_j}^{n}(\Delta)\\
\vdots & \ddots & \vdots\\
m_{j,R_j1}^{n}(\Delta) & \dots & m^{n}_{j,R_jR_j}(\Delta)
\end{pmatrix}
\end{eqnarray}
where $0 \leq n \leq N$ and $1 \leq j \leq J$, and each element
$m_{j,rs}^{n}(\Delta)$ is an arbitrary function in $\Delta$.  Define
the matrix $F_j\left(\Delta\right)$
\begin{equation}
F_j\left(\Delta\right)=M^0_j(\Delta)+\sum_{n=1}^{N} y_n M^n_j(\Delta).
\end{equation}
Then given $b\in \R^N$, \texttt{Outer\_Limits} will solve for coefficients $y_n$
that
\begin{eqnarray}
\label{eq:SDPconstraints}
\begin{array}{ll}
\textrm{maximize} & b_0+b\.y\quad\textrm{over}\quad y \in \R^N,\\
\textrm{such that} & F_j\left(\Delta\right) \succeq 0 \quad \textrm{for all $\Delta\geq 0$ and $1 \leq j \leq J$}.
\end{array}
\end{eqnarray}
The notation $F_j\left(\Delta\right)\succeq 0$ means $F_j\left(\Delta\right)$ is positive semidefinite.

\section{Input}
\label{sec:input}

Since \texttt{Outer\_Limits} solves these problems on a computer,
there are inevitable approximations that must occur.  The main one is
that the functions $m_{j,rs}^{n}(\Delta)$ must be well approximated by
a series of Chebyshev polynomials.  Specifically, the input to
\texttt{Outer\_Limits} is not the functions $m_{j,rs}^{n}(\Delta)$,
but rather their values at Chebyshev zeros and their relative sizes
near zero and at infinity.

The example in the \texttt{SDPB} Manual is
\begin{eqnarray}
  f_1 & = & 1 + \Delta^4, \\
  f_2 & = & \Delta^2 + \frac{\Delta^4}{12}.
\end{eqnarray}
This is a system with a single block ($J=1$) and two weights ($N=2$).
We will approximate these functions by Chebyshev polynomials covering
$0$ to $\Delta_{j,max}$.  So we need to decide on a $\Delta_{j,max}$
that is sufficient to cover all of the places where the functions
might go to zero.  In general, we want to choose a large enough
$\Delta_{j,max}$ such that the functions are dominated by the largest
terms at infinity.  In this case, the functions $f_1$ and $f_2$ have
polynomial terms that are of equal size at $\Delta=1$ and
$\Delta=\sqrt{12}\simeq 3.464$ respectively.  So to be conservative,
we choose $\Delta_{j,max}=100$.  Currently, \texttt{Outer\_Limits}
requires $\Delta_{j,max}$ to be the same for each function in a block.

Now we need to evaluate $f_1$ and $f_2$ at scaled and translated zeros
of Chebyshev polynomials.  This means that for an approximation with
$N$ points, we need values at
\begin{eqnarray}
\Delta_{j,k} =  \frac{\Delta_{j,max}}{2} \left(1 + \cos\left({\frac{\pi \left( k + \frac{1}{2} \right)}{N}}\right)\right), \textrm{ } 0 \le k < N
\end{eqnarray}
Since $f_1$ and $f_2$ are polynomials, we could use $N=5$ to get a
perfect approximation.  For this exercise, we will pretend we do not
know that and choose $N=10$.  \texttt{Outer\_Limits}
currently requires $N$ to be the same for each function in a block.

After generating the value of these functions at these points, we must
characterize the relative behavior of the different functions near
zero and at infinity.  This characterization turns out to be important
for computing high precision answers for the objective.

In this case, the characterization is straightforward.  $f_1$ and
$f_2$ are both degree=4 polynomials.  Near zero, $f_1$ goes to a
constant term and $f_2$ goes to zero.  So we can use $f_{1}(0)=1$ and
$f_{2}(0)=0$ for the limiting behavior near zero.  If $f_1$ did not
have a constant term, then we would have to compare the relative sizes
of the lowest degree non-zero coefficient.  At infinity, we only need
to know the relative sizes of the coefficients of $\Delta^4$: $1$ and
$1/12 \sim 0.083333333333333$.

Now we can construct the \texttt{functions} input file shown in
listing~\ref{lst:functions}.

\begin{lstlisting}[
  caption={\texttt{functions} input file.  The first block is $f_1$, and the second block is $f_2$.},
  label=lst:functions,
  mathescape,
  columns=fullflexible,
  frame=single,
  escapeinside=@@,
  basicstyle=\small\ttfamily\selectfont,
]
{
 "objective": ["0", "-1"],
 "normalization": ["1", "0"],
 "functions":
 [
  [
   [
    [
     {
      "max_delta" : "100",
      "epsilon_value" : "1",
      "infinity_value" : "1",
      "chebyshev_values" :
      [
       "1.1435973374045645814919250171",
       "883.02729988511593507402953969",
       "45996.705504467834968336421068",
       "555496.84436116128227452603002",
       "3164867.0107610102130932508775",
       "11178001.455036876356972361751",
       "27933561.314132283966434754990",
       "53079005.294495532165031663578",
       "79919384.706784958935035978021",
       "97560312.498026486725689472807"
      ]
     },
     {
      "max_delta" : "100",
      "epsilon_value" : "0",
      "infinity_value" : "0.08333333333333333333333333333333333333333",
      "chebyshev_values" :
      [
       "0.3909088380370479104722553418",
       "103.20121941417842758517038120",
       "4047.4415527729152920238132784",
       "47036.636299366781486947578226",
       "265517.84125951408811424149736",
       "934843.38959957190879426845671",
       "2333081.9137776891397492374085",
       "4430535.8917805604180413095200",
       "6668888.4064183873419762575987",
       "8139903.2205172185244035518177"
      ]
     }
    ]
   ]
  ]
 ]
}
\end{lstlisting}

The exact format is specified in the JSON schema
\texttt{docs/functions\_schema.json}.  At its heart, there is a set of
nested arrays.  The outermost array corresponds to the $j$ index of
$m_{j,rs}^{n}(\Delta)$, followed by $r$, $s$, and $n$.  The toy
example has $N=2$, $J=1$, and $R_{1}=1$, so $j=r=s=1$ and $n=1$ or
$2$.

The last thing to do is to create a \texttt{points} file that contains
the values of $\Delta$ where \texttt{Outer\_Limits} will initially
apply positivity constraints.  \texttt{Outer\_Limits} will also apply
constraints at epsilon and infinity.  At a minimum, you will need to supply enough
points such that there are at least as many constraints as there are
degrees of freedom in $y$.

For the toy problem, there are only two functions, so in principle you
only need two points.  Epsilon and infinity are always used, so you
not add any more points.  In practice, \texttt{Outer\_Limits} will get stuck if
you do not exceed the minimum by a healthy margin.

With that in mind, we choose the two points, $0.1$ and $1$,
resulting in the \texttt{points} input file in listing \ref{lst:points}.
Needless to say, the choice of points will depend strongly on the
problem you are solving.

\begin{lstlisting}[
  caption={\texttt{poinst} input file},
  label=lst:points,
  mathescape,
  columns=fullflexible,
  frame=single,
  escapeinside=@@,
  basicstyle=\small\ttfamily\selectfont,
]
{
 "points": [["0.1", "1"]]
}
\end{lstlisting}

The format for this file is specified in the JSON schema
\texttt{docs/points\_schema.json}.  The outer array corresponds to the
$j$ index of $m_{j,rs}^{n}(\Delta)$, and the inner array is a list
of points for that index.

\section{Running \texttt{Outer\_Limits}}
\label{sec:running}

\subsection{Initial Iteration and Scaling}
\label{subsec:initial}

When \texttt{Outer\_Limits} is running, it will first try to solve
equation \ref{eq:SDPconstraints}, but with a duality gap of 1.1.  This
very relaxed constraint on the duality gap allows it to quickly get a
very approximate solution.  Also, \texttt{Outer\_Limits} is only
enforcing the solution at the initial points via constant constraints.

One wrinkle here is that \texttt{Outer\_Limits} applies several
transformations in order to scale the problem before making use of
\texttt{SDPB}.  In the usual formulation, \texttt{SDPB} maximizes
\begin{eqnarray}
  b\cdot y
\end{eqnarray}
subject to the constraints
\begin{eqnarray}
  Y + B\cdot y & = & c\\
  Y \succeq 0
\end{eqnarray}
$B$ is a matrix corresponding to $M^n_j$, and $c$ is $M^0_j$, both evaluated
at the initial values of $\Delta$.  However, the relative sizes
of $c$ and the columns of $B$ can be wildly different, leading to
numerical difficulties.  To ameliorate this, we first find $c_{\max}$,
the largest value of $|c|$.  If $c$ is completely zero, we use
$c_{\max}=1$.

Next, we compute the singular value decomposition of $B$
\begin{eqnarray}
  B = U \Sigma V^T.
\end{eqnarray}
$U$ and $V$ are real orthogonal matrices, and $\Sigma$ is a rectangular
diagonal matrix with non-negative entries.  We define new
variables
\begin{eqnarray}
  y' & = & \Sigma V^T y/c_{\max}\\
  b' & = & b V \Sigma^{\dagger}c_{\max}\\
  c' & = & c/c_{\max}\\
  Y' & = & Y/c_{\max}\\
  B' & = & B V \Sigma^{\dagger}/c_{\max}
           \label{eq:Bprime}
\end{eqnarray}
where $\Sigma^{\dagger}$ is the pseudo-inverse of $\Sigma$.  In this
case, $\Sigma^{\dagger}$ is the transpose of $\Sigma$ with the
diagonal entries inverted.  So $\Sigma^{\dagger}$ is easy to compute
once we have $\Sigma$.  The system that we initially solve with
\texttt{SDPB} routines is then
\begin{eqnarray}
  \textrm{maximize}~b'\cdot y'
\end{eqnarray}
subject to the constraints
\begin{eqnarray}
  Y' + B'\cdot y' & = & c\\
  Y' \succeq 0
\end{eqnarray}

Computing the singular value decomposition of $B$ is relatively
expensive, similar in cost to completing a single iteration of
\texttt{SDPB}.  To amortize this cost over the whole calculation, we
save the expression $V \Sigma^{\dagger}$.  Then, at each iteration, we
compute a new $B'$ using equation \ref{eq:Bprime}.  With this method,
$B'$ is only guaranteed to be unitary on the first iteration.
However, later iterations will only add rows to $B$, so it should
still work reasonably well at equalizing the columns.

\subsection{Adapted Mesh}
\label{subsec:adaptedmesh}

With an initial solution in hand, the next step is to check for
regions within each block where the functional is negative.  To enable
this, \texttt{Outer\_Limits} creates a coarse mesh of
cells over the range from $0$ to $\Delta_{j,max}$.  Within each cell,
it evaluates the functional $F_j$ using the Chebyshev approximations
for $M_j^n$ and the solution values for $y_n$.  If the functional has
too much curvature and the points are not so close together as to
cause numerical difficulties, it splits the cell into smaller cells.

To be precise, for each block, we compute the largest value of the
functional among all the Chebyshev zeros
\begin{eqnarray}
  F_{j,\max} = \max \left( \left| F_{j} \left(\Delta_{j,k} \right) \right| \right)
\end{eqnarray}
In addition, for a cell with a width $\delta$, we define an interpolated averaged value
\begin{equation}
  \overline{F}_j\left(\Delta\right) = \left( F_j\left(\Delta + \delta\right) + F_j\left(\Delta - \delta\right) \right) /2
\end{equation}
Then we split the cell if the interpolated averaged value differs too
much from the actual value at that point
\begin{eqnarray}
  \left|\overline{F}_j\left(\Delta\right) - F_j\left(\Delta\right)\right| & < & \epsilon_{\rm{relative}} \left|\overline{F}_j\left(\Delta\right) + F_j\left(\Delta\right)\right| /2\\
  \left|\overline{F}_j\left(\Delta\right) - F_j\left(\Delta\right)\right| & < & \epsilon_{\rm{absolute}} F_{j,\max}.
\end{eqnarray}
The tolerance $\epsilon_{\rm{relative}}=\frac{1}{64}$ is somewhat
arbitrary.  The goal is to catch all of the places where $F_j$ varies
strongly, and in this way we can find all of the places where it
becomes negative.

The tolerance $\epsilon_{\rm{absolute}}$ is the smallest difference from 1
that can be represented, given the working precision.  For example,
with IEEE 754 double precision floating point numbers, this is
2.22045e-16.

\subsection{Checking Positivity}
\label{subsec:checkingpositivity}

With this adapted mesh in place, \texttt{Outer\_Limits} checks every
cell for positivity. For each cell, we compute the first and
second derivative numerically
\begin{eqnarray}
  dF_{j} & = &\left( F_{j}\left( \Delta + \delta \right) - F\left( \Delta - \delta \right) \right)/\left( 2 \delta \right),\\
  d^{2}F_{j} & = & \left( F_{j}\left( \Delta + \delta \right) - 2 F\left( \Delta \right) + F\left( \Delta - \delta \right) \right)/ \delta^2.
\end{eqnarray}
Using these to form a quadratic approximation to $F_{j}$, we compute
the location and value where $F_{j}$ is minimum
\begin{eqnarray}
  \Delta_{\min} & = & \Delta - dF_{j}/d^{2}F_{j},\\
  F_{\min} & = & F\left(\Delta\right) - \left(dF_{j}\right)^{2} /\left( 2 d^{2}F_{j}\right).
\end{eqnarray}

At first, we might think that only if $F_{\min}<0$ do we need to add
$\Delta_{\min}$ to the list of constraint points.  However, if
$F_{\min}>0$, it may only be positive due to inaccuracies in the
quadratic approximation.  Conversely, if $F_{\min}<0$, it may only be
negative due to inaccuracies in the finite precision arithmetic.  To
handle these cases, we add $\Delta_{\min}$ to the list of constraint
points if and only if
\begin{eqnarray}
  F_{\min} & < & \left| F\left(\Delta\right) - \overline{F}_j\left(\Delta\right) \right|, \\
  \left|F_{\min}\right| & > & \epsilon_{\rm{absolute}}F_{j,\max}.
\end{eqnarray}

\subsection{Further Iterations and Termination Criteria}
\label{subsec:furtheriterations}

With these new points, \texttt{Outer\_Limits} creates and solves a new
system as in section \ref{subsec:initial}.  Eventually, there will be
no new points.  Then \texttt{Outer\_Limits} reduces the duality gap by
\texttt{dualityGapReduction} and iterates again until there are no new
points.  \texttt{Outer\_Limits} continues this cycle of finding new
points and reducing the duality gap until the duality gap is less than
\texttt{dualityGapThreshold}.  This guarantees that the final result
will fully satisfy all of the constraints in \ref{eq:SDPconstraints}
without having to spend a lot of effort computing highly precise
solutions for very approximate problems.

\section{Output of \SDPB}

\subsection{Terminal Output}

The output from running \texttt{Outer\_Limits} on the example problem
in section \ref{sec:input} is in listing
\ref{listing:exampleoutput}. The beginning is very similar to
\texttt{SDPB}.  When iterating, \texttt{Outer\_Limits} first prints
the current number of constraints, including all of the blocks, and
the current gap threshold.  After each iteration,
\texttt{Outer\_Limits} prints the full array of weights and the
computed value of the objective.  If \texttt{Outer\_Limits} decides
not to add points, it will reduce the duality gap threshold, print
this new threshold, and continue with the current solution.

Figures \ref{fig:plot1}, \ref{fig:plot2}, and
\ref{fig:plot3} show the behavior of $F_0$ during the iterations
as it alternates between adding points and reducing the gap threshold.

\begin{lstlisting}[
  caption={Output of \texttt{Outer\_Limits} for the input file in listing~\ref{lst:points}},
  columns=fullflexible,
  label=listing:exampleoutput,
  keepspaces=true,
  basicstyle=\scriptsize\ttfamily\selectfont,
]
Outer_Limits started at 2021-Jun-16 21:16:54
functions file  : "test/toy_functions.json"
out directory   : "test/toy_functions_out.json"

Parameters:
dualityGapReduction          = 1024
maxIterations                = 1000
maxRuntime                   = 9223372036854775807
checkpointInterval           = 3600
findPrimalFeasible           = false
findDualFeasible             = false
detectPrimalFeasibleJump     = false
detectDualFeasibleJump       = false
precision(actual)            = 128(128)
dualityGapThreshold          = 1e-10
primalErrorThreshold         = 1e-10
dualErrorThreshold           = 1e-10
initialMatrixScalePrimal     = 10
initialMatrixScaleDual       = 10
feasibleCenteringParameter   = 0.1
infeasibleCenteringParameter = 0.3
stepLengthReduction          = 0.7
maxComplementarity           = 1e+100
initialCheckpointDir         = "test/toy_functions.ck"
checkpointDir                = "test/toy_functions.ck"
writeSolution                =
verbosity                    = 1

num_constraints: 4
Threshold: 1.1
weight: [1, 17.9200220901249930755098639008252582465]
optimal: -17.9200220901249930755098639008252582465
Threshold: 0.001074218750000000086736173798840354720596
weight: [1, -1.845764788512892377453602431656624264456]
optimal: 1.845764788512892377453602431656624264454
Saving checkpoint to    : "test/toy_functions.ck"
num_constraints: 5
Threshold: 0.001074218750000000086736173798840354720596
weight: [1, -1.839611388308253894657545680555773464748]
optimal: 1.839611388308253894657545680555773464748
Threshold: 1.049041748046875084703294725430033906832e-06
weight: [1, -1.840275804402260393066356407589470103775]
optimal: 1.840275804402260393066356407589470103773
Saving checkpoint to    : "test/toy_functions.ck"
num_constraints: 6
Threshold: 1.049041748046875084703294725430033906832e-06
weight: [1, -1.840265160571370863263682954232296794857]
optimal: 1.840265160571370863263682954232296794854
Threshold: 1.024454832077026449905561255302767487141e-09
weight: [1, -1.840265762727976760495004292302021649703]
optimal: 1.840265762727976760495004292302021649702
Threshold: 1e-10
weight: [1, -1.840265763127732853472949500621348513756]
optimal: 1.840265763127732853472949500621348513754
Saving checkpoint to    : "test/toy_functions.ck"
optimal: 1.840265763127732853472949500621348513754
Saving solution to "test/toy_functions_out.json"
\end{lstlisting}

\pagebreak

\begin{figure}
\begin{center}
\includegraphics[width=0.9\textwidth]{outer_plots_1}
\end{center}
\caption{Plot of $F_{0}=y_{0}\left(1+x^4\right) + y_{1}\left(\frac{x^4}{12} + x^2\right)$
  after the first iteration, with duality gap threshold=1.1, and the second
  iteration, with duality gap threshold=$1.07\times10^{-3}$.  After the second
  iteration, \texttt{Outer\_Limits} adds a point at 1.044.}
\label{fig:plot1}
\end{figure}

\begin{figure}
\begin{center}
\includegraphics[width=0.9\textwidth]{outer_plots_2}
\end{center}
\caption{Plot of $F_{0}$
  after the third iteration, with duality gap threshold=$1.07\times10^{-3}$, and the fourth iteration, with duality gap threshold=$1.05\times10^{-6}$.  After the fourth
  iteration, \texttt{Outer\_Limits} adds a point at 1.042500.}
\label{fig:plot2}
\end{figure}

\begin{figure}
\begin{center}
\includegraphics[width=0.9\textwidth]{outer_plots_3}
\end{center}
\caption{Plot of $F_{0}$ after the fifth, sixth, and seventh
  iterations, with duality gap thresholds of $1.05\times10^{-6}$,
  $1.02\times10^{-9}$, and $1.02\times10^{-9}$.  After the sixth
  iteration, \texttt{Outer\_Limits} adds a point at 1.042497.}
\label{fig:plot3}
\end{figure}

\pagebreak

\subsection{Termination}

\texttt{Outer\_Limits} can fail when the inner \texttt{SDPB} routines
fail, so \texttt{Outer\_Limits} has the same failure termination
criteria.  \texttt{Outer\_Limits}'s only successful termination
criteria is if the duality gap is less than
\texttt{dualityGapThreshold}.

\pagebreak

\subsection{Output File}

If \texttt{Outer\_Limits} completes successfully, it writes the
solution to a single JSON file, as shown in listing
\ref{listing:solution}.

\begin{lstlisting}[
  caption={Solution output of \texttt{Outer\_Limits} for the input file in listing~\ref{lst:points}},
  columns=fullflexible,
  label=listing:solution,
  keepspaces=true,
  basicstyle=\scriptsize\ttfamily\selectfont,
]
{
  "optimal": "1.840265763127732853472949500621348513754",
  "y":
  [
    "1",
    "-1.840265763127732853472949500621348513756"
  ],
  "options":
{
    "maxIterations": "1000",
    "maxRuntime": "9223372036854775807",
    "checkpointInterval": "3600",
    "findPrimalFeasible": "false",
    "findDualFeasible": "false",
    "detectPrimalFeasibleJump": "false",
    "detectDualFeasibleJump": "false",
    "precision": "128",
    "precision_actual": "128",
    "dualityGapThreshold": "1e-10",
    "primalErrorThreshold": "1e-10",
    "dualErrorThreshold": "1e-10",
    "initialMatrixScalePrimal": "10",
    "initialMatrixScaleDual": "10",
    "feasibleCenteringParameter": "0.1",
    "infeasibleCenteringParameter": "0.3",
    "stepLengthReduction": "0.7",
    "maxComplementarity": "1e+100",
    "initialCheckpointDir": "test\/toy_functions.ck",
    "checkpointDir": "test\/toy_functions.ck",
    "functions": "test\/toy_functions.json",
    "points": "test\/toy_functions_points.json",
    "out": "test\/toy_functions_out.json",
    "dualityGapReduction": "1024",
    "writeSolution": "",
    "verbosity": "1"
}
}
\end{lstlisting}

\begin{thebibliography}{9}

\bibitem{DSD}
  David Simmons-Duffin,
  ``A Semidefinite Program Solver for the Conformal Bootstrap,"
  \href{http://arxiv.org/abs/1502.02033}{arXiv:1502.02033 [hep-th]}.

%\cite{Rychkov:2009ij}
\bibitem{Rychkov:2009ij}
  V.~S.~Rychkov and A.~Vichi,
  ``Universal Constraints on Conformal Operator Dimensions,''
  Phys.\ Rev.\ D {\bf 80}, 045006 (2009)
  \href{http://arxiv.org/abs/0905.2211}{arXiv:0905.2211 [hep-th]}.
  %%CITATION = ARXIV:0905.2211;%%
  %82 citations counted in INSPIRE as of 05 Feb 2015

%\cite{Poland:2011ey}
\bibitem{Poland:2011ey}
  D.~Poland, D.~Simmons-Duffin and A.~Vichi,
  ``Carving Out the Space of 4D CFTs,''
  JHEP {\bf 1205}, 110 (2012)
  \href{http://arXiv.org/abs/1109.5176}{arXiv:1109.5176 [hep-th]}.
  %%CITATION = ARXIV:1109.5176;%%
  %71 citations counted in INSPIRE as of 01 Feb 2015

%\cite{Kos:2013tga}
\bibitem{Kos:2013tga}
  F.~Kos, D.~Poland and D.~Simmons-Duffin,
  ``Bootstrapping the $O(N)$ vector models,''
  JHEP {\bf 1406}, 091 (2014)
  \href{http://arXiv.org/abs/1307.6856}{arXiv:1307.6856 [hep-th]}.
  %%CITATION = ARXIV:1307.6856;%%
  %29 citations counted in INSPIRE as of 01 Feb 2015

%\cite{Kos:2014bka}
\bibitem{Kos:2014bka}
  F.~Kos, D.~Poland and D.~Simmons-Duffin,
  ``Bootstrapping Mixed Correlators in the 3D Ising Model,''
  JHEP {\bf 1411}, 109 (2014)
  \href{http://arXiv.org/abs/1406.4858}{arXiv:1406.4858 [hep-th]}.
  %%CITATION = ARXIV:1406.4858;%%
  %9 citations counted in INSPIRE as of 01 Feb 2015

\bibitem{SDPA}
  M. Yamashita, K. Fujisawa, M. Fukuda, K. Nakata, and M. Nakata,
  ``A high-performance software package for semidefinite programs: SDPA 7,''
   Research Report B-463, Dept. of Mathematical and Computing Science, Tokyo Institute of Technology, Tokyo, Japan (2010).

\bibitem{SDPA2}
  M. Yamashita, K. Fujisawa, and M. Kojima,
  ``Implementation and evaluation of SDPA 6.0 (SemiDefinite Programming Algorithm 6.0),''
  Optimization Methods and Software" 18 491-505 (2003).

\bibitem{SDPAGMP}
  M. Nakata,
  ``A numerical evaluation of highly accurate multiple-precision arithmetic version of semidefinite programming solver: SDPA-GMP, -QD and -DD.,''
  2010 IEEE International Symposium on Computer-Aided Control System Design (CACSD), 29-34 Sept 2010.

\bibitem{BoostSite}
  C++ Standards Committee Library Working Group and other contributors,
  ``BOOST C++ Libraries,''
  \href{http://www.boost.org}{http://www.boost.org}.

\bibitem{libxml2}
  Gnome Project,
  Libxml2,
  \href{http://www.xmlsoft.org/}{http://www.xmlsoft.org/}

\bibitem{Elemental}
  J. Poulson, B. Marker, R. van de Geijn, J. Hammond, and N. Romero,
  ``Elemental: A new framework for distributed memory dense matrix computations, ACM Transactions on Mathematical Software,''
  ACM Trans. Math. Softw. 39 2 13:1-24 (2013),
  doi:10.1145/2427023.2427030

\bibitem{GMP}
  The GNU Multiprecision Library,
  \href{https://gmplib.org/}{https://gmplib.org/}

\bibitem{MPFR}
  The GNU MPFR Library,
  \href{https://www.mpfr.org/}{https://www.mpfr.org/}

\end{thebibliography}

\end{document}


\section{Introduction}\label{s:intro}

% Abstract
This chapter considers the problem of obtaining dense 3D reconstructions of articulated subjects from single and partially occluded views. Humans generally have no problem in predicting, at least approximately, the 3D structure of most scenes including the pose and shape of animals or other people, even from a single view.
However, the visual evidence available in a single 2D image notoriously~\citep{Faugeras01geometry} contains insufficient information for the third dimension to be recovered uniquely.
This chapter follows a growing body of work that views 3D reconstruction of articulated subjects in a probabilistic setting, and proposes that the goal of reconstruction methods should be to make the posterior distribution as sharp as possible by learning an strong prior over the space of possible solutions.
In particular, the method described in this chapter recovers \emph{several} plausible and diverse 3D reconstructions which are compatible with the input data. This is implemented using a novel multi-hypothesis neural network architecture, trained using a best-of-$M$ loss where each of the $M$ hypotheses is constrained to lie on a manifold of plausible poses by means of a generative model. Various improvements are suggested to the standard best-of-$M$ setup, to produce reconstruction sets of an arbitrary size, ensuring these are plausible and diverse by means of a learnt normalizing flow prior and discouraging mode degeneration with a multi-hypothesis reprojection loss.
A fundamental insight is that ambiguities in 3D body poses can be effectively modelled using a similar parameterization as in previous chapters, using a suitable 3D morphable model.
As well as being the first system capable of modelling ambiguities in 3D animal reconstruction, the method is shown to outperform alternative approaches in ambiguous pose recovery on standard benchmarks for 3D humans and in heavily occluded versions of these benchmarks. The chapter begins by introducing key concepts relied upon in this chapter as well as related literature, followed by a description of the method, an evaluation section and finishes with conclusions and opportunities for future work.

\input{Chapter6/FigTex/fig-splash.tex}

\subsection{Limitations of state-of-the-art 3D reconstruction techniques}

As discussed in~\Cref{chap:relwork}, recent progress in single-view 3D model reconstruction for articulated subjects has been impressive.
Methods such as 3D-Safari~\cite{xxx} and \Cref{chap:wldo} of this thesis for animals and HMR~\cite{kanazawa18end-to-end}, GraphCMR~\cite{kolotouros19convolutional} and SPIN~\cite{kolotouros19learning} for humans, formulate the task using a deep neural network that maps 2D images to the parameters of a 3D morphable model (usually SMAL~\cite{zuffi2017menagerie} for animals and SMPL~\cite{loper15smpl} for humans).
Although these methods work well in general, they are not immune from failure cases~(\Cref{fig:issues}).
Their main weakness is when processing \emph{heavily occluded images} of the subject.
When a large part of the subject is missing, say the hind legs of an animal or lower body of a sitting human, they output reconstructions that are often implausible.
Since these approaches produce only one hypothesis as output, they very likely learn to approximate the mean of the posterior distribution, which may not correspond to any plausible pose.
This can be understood with a simple thought experiment. Consider an agent which must randomly choose to travel left or right to avoid an oncoming lake. If the agent applies a policy to average these two possibilities, it will lead the agent to travel straight forwards, realising a suboptimal outcome of travelling straight into the lake.
Unfortunately, the failure modality caused by ambiguous imagery is rather common in animal and human scenes due to environmental and self-occlusion, due to environmental clutter and crowds.

% Discuss other approaches for dealing with this limitation
This chapter proposes a system that handles these existing limitations. The approach recovers 3D mesh reconstructions of complex articulated objects such as animals or humans from highly ambiguous image data, often containing significant occlusions of the object.
Clearly, it is generally impossible to reconstruct the object uniquely if too much evidence is missing; however, it is still possible to predict a \emph{set} containing all possible reconstructions (see \Cref{fig:splash}).
While ambiguous pose reconstruction has been previously investigated, this work proposes the first deep learning approach for ambiguous reconstructions of the \emph{full articulated mesh}.

\input{Chapter6/FigTex/fig-sotafailures.tex}

\subsection{Modelling ambiguities in 3D reconstruction}

% kinematic-jump-processes http://www.cs.toronto.edu/~crismin/PAPERS/kinematic.pdf
% tracking-3d-human-figures http://luthuli.cs.uiuc.edu/~daf/courses/AppCV/Papers-2/eccv00.pdf
% desity-prop https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.119.1950&rep=rep1&type=pdf
% akhter15pose-conditioned https://openaccess.thecvf.com/content_cvpr_2015/papers/Akhter_Pose-Conditioned_Joint_Angle_2015_CVPR_paper.pdf
% ligenerating https://openaccess.thecvf.com/content_CVPR_2019/papers/Li_Generating_Multiple_Hypotheses_for_3D_Human_Pose_Estimation_With_Mixture_CVPR_2019_paper.pdf
% bishop94mixture https://publications.aston.ac.uk/id/eprint/373/1/NCRG_94_004.pdf
% sharma19monocular https://openaccess.thecvf.com/content_ICCV_2019/papers/Sharma_Monocular_3D_Human_Pose_Estimation_by_Generation_and_Ordinal_Ranking_ICCV_2019_paper.pdf
% cheng19occlusion-aware https://openaccess.thecvf.com/content_ICCV_2019/papers/Cheng_Occlusion-Aware_Networks_for_3D_Human_Pose_Estimation_in_Video_ICCV_2019_paper.pdf

% rupprecht17learning: learning in an uncertain world: https://arxiv.org/pdf/1612.00197.pdf

% Resolving 3D https://arxiv.org/abs/1908.06963

% https://arxiv.org/abs/1807.01511 (deep autoencoder)

Several previous papers have looked at the problem of modelling ambiguous 3D human pose reconstructions. Early work includes~\cite{kinematic-jump-processes} who rely on known body segement lengths to resolve depth ambiguities at each joint. Sidenbldh et al.~\cite{tracking-3d-human-figures} define a probalistic method for tracking 3D articulated humans based on a prior probabilisty distribution over pose that models how humans are expected to move. Sminchisescu et al.~\cite{density-prop} describe a mixture density propagation algorithm for predicts the conditional state distributon necessary for 3D body pose tracking based on input silhouettes. More recently, ~\cite{cheng19occlusion-aware} tackle the problem of video 3D reconstruction in the presence of occlusions, and show that temporal cues can be used to disambiguate the solution.
% While our method is similar in the goal of correctly handling the prediction uncertainty, we differ by applying our method to predicting \emph{full mesh} of the human body.
% This is arguably a more challenging scenario due to the increased complexity of the desired 3D shape.

% The method is similar in that it attempts to handle prediction uncertainty, although the work in this chapter differs by predicting the \emph{full mesh} of the subject. This is arguably a more challenging scenario due to the increased complexity of the desired 3D shape.

The following paragraphs describe implementations of common approaches used to model prediction uncertainty in deep networks. Note that the methods discussed, including Mixture Density Networks~\cite{bishop94mixture,li19generating} and Conditional Variational Auto-Encoders~\cite{sharma19monocular} differ from the work in this chapter as they output a distribution over reconstructed \emph{3D locations} of a finite set of human body joints, rather than a full mesh. This chapter argues that the parameters of a 3D morphable model offers a better space for coding not just 3D shapes and poses, but also their ambiguities. In addition, predicting a full mesh is a more challenging scenario due to the increased complexity of the desired 3D shape. A later section in this chapter describes how these techniques can be extended to act as baselines for the system proposed here.
% TODO: An exception is the very recent work of BLAH, who extend the ar

% As far as we could determine, the work in this chapter is the first to model ambiguous reconstructions directly in the space of human body model parameters, although notable works such as ~\cite{xxx} have come later and are worthy of discussion.
% Ignas paper

% To be fair, we don't learn a prior directly on the SMPL parameters.

% They model distributions conditioned on the parent of each joint in the kinematic tree and show that this prior can be used to obtain improved parametric fits.
% More recently,~\citet{akhter15pose-conditioned} estimate a 3D human stick figure from 2D joint locations, making use of a prior over joint angles learnt from a motion capture dataset.

\subsubsection{Mixture Density Networks (MDN)}

Li et al. ~\cite{li19generating} use the Mixture Density Networks model of~\cite{bishop94mixture} to capture ambiguous 3D reconstructions of sparse human body keypoints (rather than a full mesh as in this work) directly in physical space.

The idea of a Mixture Density Network for articulated 3D reconstruction is to represent the probability of a 3D pose vector $x \in \R{3N}$ given an input set of features $y \in \R{M}$. $y$ is left general here, but could be for example a set of 2D keypoints, extracted image features etc. $w$ are the neural network weights. The posterior density is given as a linear combination of $M$ Gaussian functions
\begin{equation}
  p(x | y, w) = \sum_{i}^{M} \Pi_{i}(y, w)\phi_{i}(x | y, w)
\end{equation}

Here, $\Pi_{i}(y, w)$ are mixing coefficients which must satisfy $0 \leq \Pi_{i}(y) \leq 1$ and $\sum_{i=1}^{M} = 1$. $\phi(x | y, w)$ is the conditional distribution of the 3D pose $x$ for the $i^{th}$ Gaussian component.

\begin{equation}
  \phi(x | y, w) =
    \frac{
      1
    }{
      (2\pi)^{\frac{d}{2}} \sigma_{i}(y, w)^{d}
    }
    \exp{
    -\frac{
      || x - \mu_{i}(y, w) ||^{2}
    }{
      2\sigma_{i}(y, w)^{2}
    }
  }
\end{equation}

A dataset of $K$ independent and identically distributed training examples including input $X$ and 3D poses $Y$ can be used to optimize the weights $w$ of such a network. The posterior distribution of $w$ is given by

\begin{align}
  p(w | X, Y)
  &\propto p(X | Y, w)p(w, Y) \\
  &= p(w | Y) \prod_{j=1}^{K} p(x_j | y_j, w)
  % &= p(w | X) \prod_{j=1}^{K} \sum_{i=1}^{M} \Pi_{i}(x_{j}, w) \phi_{i} (y_{j} | x_{j}, w)
\end{align}

The optimal weight $w^{*}$ is obtained by minimizing the negative log-posterior

\begin{align}
  w^{*} &= \argmin_{w} -\log p(w | X, Y) \\
        &= \argmin_{w} \left[-\sum_{j=1}^{K} \log p(x_j | y_j, w)\right] - \log p(w | Y)
\end{align}

This formulation can be used to form a loss function to optimize the network.

\begin{equation}
  L = \left[-\sum_{j=1}^{K} \log p(x_j | y_j, w)\right] - \log p(w | Y)
\end{equation}

This can be deconstructed into two terms, $L = L_{3D} + L_{PRIOR}$. $L_{3D}$ tunes the network weights such that ground truth 3D poses are likely under the model, and $L_{PRIOR}$ which acts as a prior over $\mu(w, Y), \sigma(w, Y)$ and $\Pi(w, Y)$. In the Li et al.~\cite{xxx} implementation, the prior is uniform over $\mu(w, Y)$ and $\sigma(w, Y)$, and a Dirichlet~\cite{xxx} conjugate prior is used over the mixture coefficients $\Pi(w, Y)$.

For completeness, the 3D alignment term is given as
\begin{equation}
  L_{3D} = -\sum_{j=1}^{K} \log \sum_{i=1}^{M} \Pi_{i}(y_{j}, w) \phi_{i}(x_j | y_j, w)
\end{equation}

% https://arxiv.org/pdf/2103.10978.pdf
A version of this setup is used as a baseline in the later experiment section, although adapted to model the conditional probability of 3D morphable model joint angles through the generator function. Another notable work in this category (released after the content of this chapter) is that of Sengupta et al~\lazycite{Sengupta}{} who extend the MDN formulation to model ambiguity in pose and shape, operating on a collection of human images.

\subsubsection{Conditional Variational Auto-Encoder (CVAE)}

Another approach, proposed by \citet{sharma19monocular} is to learn a conditional variational auto-encoder (CVAE) to model ambiguous reconstructions as a posterior distribution.

To explain this, first consider a simple directed, latent variable model. $x$ is the observed variable, and $u$ is a latent variable, or \emph{code}, which should provide information about $x$.

% \begin{center}
% \begin{tikzpicture}
%   \SetGraphUnit{5}
%   \Vertex{x}
%   \WE(x){z}
%   \WE(z){y}
%   \Edge(z)(x)
%   \Edge(y)(z)
%   % \Edge[label = 2](B)(C)
%   % \Edge[label = 3](C)(B)
%   % \Edge[label = 4](B)(A)
%   % \Loop[dist = 4cm, dir = NO, label = 5](A.west)
%   % \Loop[dist = 4cm, dir = SO, label = 6](C.east)
%   % \tikzset{EdgeStyle/.append style = {bend left = 50}}
%   % \Edge[label = 7](A)(C)
%   % \Edge[label = 8](C)(A)
% \end{tikzpicture}
% \end{center}

\begin{center}
  \begin{tikzpicture}
    \SetGraphUnit{5}
    \Vertex{x}
    \WE(x){z}
    \Edge(z)(x)
    % \Edge[label = 2](B)(C)
    % \Edge[label = 3](C)(B)
    % \Edge[label = 4](B)(A)
    % \Loop[dist = 4cm, dir = NO, label = 5](A.west)
    % \Loop[dist = 4cm, dir = SO, label = 6](C.east)
    % \tikzset{EdgeStyle/.append style = {bend left = 50}}
    % \Edge[label = 7](A)(C)
    % \Edge[label = 8](C)(A)
  \end{tikzpicture}
  \end{center}

The joint distribution of such a model is given as

$$
  p_{\theta}(x,z) = p(x|z)p(z)
$$

% $$
%   p_{\theta}(x,z,y) = p(x|z)p(z|y)p(y)
% $$

This type of latent variable model can be viewed as a generative process for obtaining the observed data $x$. The procedure for this is:

% $$
% \begin{aligned}
%   y &\sim p(y) \\
%   z &\sim p(z|y) \\
%   x &\sim p(x|z)
% \end{aligned}
% $$

$$
\begin{aligned}
  z &\sim p(z) \\
  x &\sim p(x|z)
\end{aligned}
$$

This represents sampling a latent vector $z$ from the prior distribution $p(z)$ followed by sampling data $x$ from the conditional likelihood distribution $p(x|z)$. The variational autoencoder comprises a probabilistic encoder $p(z|x)$ that describes the distribution of the latent variables given data, and a probabilistic decoder $p(x|z)$ which describes the distribution of the data given the latent variable.

% There are two objectives. First, $p$ must be tuned such that the latent model closely matches the true underlying distribution $p_{data}$. Secondly, to infer good values for the latent variables $z$ given observed data, in other words to calculate the posterior $p(z|x)$.
% p(z|x,y)

Bayes theorem can be used to draw the link between these terms.
% $$
% \begin{aligned}
%   p(z|x,y)
%   &= \frac{p(x|y,z)p(z|y)}{p(x|y)} \\
%   &= \frac{p(x|y,z)p(z|y)}{\int p(x|y,z)p(z|y) \,dz}
% \end{aligned}
% $$
$$
\begin{aligned}
  p(z|x)
  &= \frac{p(x|z)p(z)}{p(x)} \\
  &= \frac{p(x|z)p(z)}{\int p(x|z')p(z') \,dz'}
\end{aligned}
$$

% A simple way to set up terms is to assume $p(z) = \mathcal{N}(0, I)$ is a standard Gaussian distribution and $p(x|z) = \mathcal{N}(f(z),g(z))$ is a Gaussian parameterized by outputs of neural networks $f, g$.
Unfortunately, $p(z|x)$ is generally intractable for high-dimensional $z$ since the integral is taken over all possible configurations of latent variables $z$ (or $z'$ as in the expansion above). Even estimation attempts based on Monte Carlo tend to fail due to high variance in gradient estimates. % ASK AWF!? Ask why this fails? can't you just try fewer samples? is this because the distribution is very 'isolated'

% $p(x,z,y)$
% The Kullback-Leibler (KL) divergence can be used to measure how closely $p(x,z)$ fits a training dataset. Divergence is measured between the data distribution $p_{data}(x)$ and the model's marginal distribution $p(x) = \int p(x,z) \,dz$.
%https://jaan.io/what-is-variational-autoencoder-vae-tutorial/

% Note the integral on the demoninator requires expotential time to compute, since it needs to be evaluate all configurations of latent variables.
% In practice, this expression is generally intractable for high-dimensional $z$ to compute this expression since the integral is taken integration over all possible configurations of latent variables $z$.
% Even estimation attempts based on Monte Carlo tend to fail due to high variance in gradient estimates. % ASK AWF!? Ask why this fails? can't you just try fewer samples? is this because the distribution is very 'isolated'

% This is equivalent to maximizing the marginal log-likelihood $log p(x)$ over the dataset $D$.

% $$
% \max_{p} \sum_{x \in D} \log p_{\theta}(x) = \max_{p} \sum_{x \in D} \log \int p_{\theta}(x,z) \,dz
% $$

Instead, a variational method is used to estimate the intractable posterior $p(z|x)$ using a new tractable distribution $q(z)$. To do this, Kullback-Leibler (KL) divergence can be used to measure the difference between two probability distributions. For example, it can be used to measure how closely the approximation $q(z)$ matches the true distribution $p(z|x)$
$KL(q_{\lambda}(z) \| p(z|x))$:
% $KL(q_{\lambda}(z|x,y) \| p(z|x,y))$:

The goal is to find the variational parameters for $q$ that minimize the divergence:

\begin{equation}
  q^{*}(z) = \argmin_{\lambda} KL(q(z) \| p(z|x))
\end{equation}

And looking at this term
$$
\begin{aligned}
KL(q(z) \| p(z|x))
  &= KL(q(z) \| p(z|x)) \\
  &= -\sum_{z \sim q(z)} q(z) \log \frac{p(z|x)}{q(z)} \\
  &= -\sum_{z \sim q(z)} q(z) \log \frac{p(x,z)}{q(z)} \frac{1}{p(x)} \\
  &= -\sum_{z \sim q(z)} q(z) \log \left[ \frac{p(x,z)}{q(z)} - \log p(x) \right] \\
  &= -\sum_{z \sim q(z)} q(z) \log \frac{p(x,z)}{q(z)}
  + \sum_{z \sim q(z)} q(z) \log p(x) \\
  &= -\sum_{z \sim q(z)} q(z) \log \frac{p(x,z)}{q(z)}
  + \log p(x) \underbrace{\sum_{z \sim q(z)} q(z)}_{=1} \\
\end{aligned}
$$

Notice that rearraging gives the following expression
$$
\begin{aligned}
  \log p(x) = KL(q_{\lambda}(z) \| p(z|x))
  + \underbrace{\sum_{z \sim q(z)} q(z) \log \frac{p(x,z)}{q(z)}}_{ELBO}
\end{aligned}
$$

Recall the aim of this exercise is to minimize $KL(q_{\lambda}(z) \| p(z|x))$. Since $x$ is given, $\log p(x)$ is a constant and as KL is non-negative this can be achieved by maximizing the final term, known as the Evidence Lower Bound (ELBO).
$$
\begin{aligned}
  ELBO
  &= \sum_{z \sim q(z)} q(z) \log \frac{p(x,z)}{q(z)} \\
  &= \sum_{z \sim q(z)} q(z) \log \frac{p(x|z)p(z)}{q(z)} \\
  &= \sum_{z \sim q(z)} q(z) \left[ \log p(x|z)
  + \log \frac{p(z)}{q(z)} \right] \\
  &= \sum_{z \sim q(z)} q(z) \log p(x|z)
  + \sum_{z \sim q(z)} q(z) \log \frac{p(z)}{q(z)} \\
  &= E_{z \sim q(z)} \log p(x|z) - KL(q(z) \| p(z))
\end{aligned}
$$

Peering at the two terms, one can think the first term $E_{z \sim q(z)} \log p(x|z)$ as a reconstruction `loss' which promotes $x's$ generated from $z's$ being likely. The second term $KL(q(z) \| p(z))$ is a regularizer that keeps the approximate distribution $q$ close to $p$. Further, note that the tightness of the lower bound is determined by the specific choice of $q$. In particular, the error between the original expression $\log p(x)$ and the ELBO is given by $KL(q(z) || p(z|x))$. The error is zero if these exactly match.

% $$
% \begin{aligned}
%   q^{*}_{\lambda}(z|x)
%   &= \argmin_{\lambda} KL(q_{\lambda}(z|x) \| p(z|x)) \\
%   &= \argmin_{\lambda}
%   \mathbb{E}_{z \sim q(z|x)} \left[ \log q_{\lambda}(z|x) \right]
%   -\mathbb{E}_{z \sim q(z|x)} \left[ \log \frac{p(x,z)p(z)}{p(x)} \right] \\
%   &= \argmin_{\lambda}
%   \mathbb{E}_{z \sim q(z|x)} \left[ \log q_{\lambda}(z|x) \right]
%   -\mathbb{E}_{z \sim q(z|x)} \left[ \log p(z) \right]
%   -\mathbb{E}_{z \sim q(z|x)} \left[ \log p(x|z) \right]
%   +\mathbb{E}_{z \sim q(z|x)} \left[ \log p(x) \right] \\
%   &= \argmin_{\lambda}
%   \mathbb{E}_{z \sim q(z|x)} \left[ \log q_{\lambda}(z|x) \right]
%   -\mathbb{E}_{z \sim q(z|x)} \left[ \log p(x,z) \right]
%   + \log p(x)
% \end{aligned}
% $$
% % $$
% % \begin{aligned}
% %   q^{*}_{\lambda}(z|x,y)
% %   &= \argmin_{\lambda} KL(q_{\lambda}(z|x,y) \| p(z|x,y)) \\
% %   &= \argmin_{\lambda} \mathbb{E}_{q} \left[ \log q_{\lambda}(z|x,y) \right]-\mathbb{E}_{q} \left[ \log p(x,z|y) \right] + \log p(x|y)
% % \end{aligned}
% % $$

% Unfortunately, this is again intractable as the evidence term $p(x)$ appears once more. However, note that with this definition

% $$
% \begin{aligned}
%   \log p(x|y)
%   &=
%   \mathbb{E}_{q} \left[ \log p(x,z|y) \right]
%   - \mathbb{E}_{q} \left[ \log q_{\lambda}(z|x,y) \right]
%   + KL(q_{\lambda}(z|x,y) \| p(z|x,y)) \\
%   &=
%   \underbrace{
%     \mathbb{E}_{q} \left[ q_{\lambda}(z|x,y) \log p(x|z,y) \right]
%   - KL(q_{\lambda}(z|x,y) \| p(z|y))}_{ELBO}
%   + KL(q_{\lambda}(z|x,y) \| p(z|x,y))
% \end{aligned}
% $$

% $$
% \begin{aligned}
%   \log p(x)
%   &=
%   \mathbb{E}_{q} \left[ \log p(x,z) \right]
%   - \mathbb{E}_{q} \left[ \log q_{\lambda}(z|x) \right]
%   + KL(q_{\lambda}(z|x) \| p(z|x)) \\
%   &=
%   \underbrace{
%     \mathbb{E}_{q} \left[ q_{\lambda}(z|x) \log p(x|z) \right]
%   - KL(q_{\lambda}(z|x) \| p(z))}_{ELBO}
%   + KL(q_{\lambda}(z|x) \| p(z|x))
% \end{aligned}
% $$

% This expression can be used to derive the Evidence Lower Bound (ELBO) as follows:
% \begin{equation}
%   \log p(x)
%   \geq \mathbb{E}_{q} \left[ q_{\lambda}(z|x) \log p(x|z) \right]
%   - KL(q_{\lambda}(z|x) \| p_{\theta}(z))
% \end{equation}
% \begin{equation}
%   \log p(x|y)
%   \geq \mathbb{E}_{q} \left[ q_{\lambda}(z|x,y) \log p(x|z,y) \right]
%   - KL(q_{\lambda}(z|x,y) \| p_{\theta}(z|y))
% \end{equation}

% $$
% \begin{aligned}
%   \log p_{\theta}(x|y)
%   &= \log \int p_{\theta}(x, z) \,dz \\
%   &= \log \int p_{\theta}(x, z) \frac{q_{\lambda}(z|x,y)}{q_{\lambda}(z|x,y)} \,dz \\
%   &\geq q_{\lambda}(z|x) \log \frac{p_{\theta}(x, z)}{q_{\lambda}(z|x,y)} \,dz \\
%   &=\mathbb{E}_{q_{\lambda}(z|x,y)} \left[ \log \frac{p_{\theta}(x,z)}{q_{\lambda}(z|x,y)} \,dz \right] \\
% \end{aligned}
% $$

% Note that Jensen's inequality is used in the final step. This final expression is known as the Evidence Lower Bound (ELBO).

% The tightness of the lower bound is determined by the specific choice of $q$. In particular, the error between the original objective $\log p_{\theta}(x)$ and the ELBO is given by $KL(p(z|x) || q(z|x))$. The error is zero if these exactly match.

% A careful choice of $q_{\lambda}(z)$ leads this construction to admit a tractable unbias Monte Carlo estimator:
% \begin{equation}
%   \log p_{\theta}(x) = \frac{1}{k} \sum_{i=1}^{k} \log \frac{p_{\theta}(x, z_{i})}{q_{\lambda}(z_{i}|x)}, \textrm{where } z_{i} \sim q_{\lambda}(z,x)
% \end{equation}

% Or alternatively,
% \begin{align}
%   \log p_{\theta}(x)
%   &= \int \log \left[ p_{\theta}(x, z) \frac{p_{\theta}(z)}{q_{\lambda}(z|x)} \right] q(z|x) \,dz \\
%   &= \int \log p_{\theta}(x|z)q_{\lambda}(z|x) \,dz - KL(q_{\lambda}(z|x) \| p_{\theta}(z))
% \end{align}

The next question is how to implement such a model using a deep neural network. Here,

\begin{enumerate}
  \item \textbf{Encoder} $q_{\lambda}(z|x) = \mathcal{N}(z | \mu_{\lambda}(x), \Sigma_{\lambda}(x))$ where $(\mu_{\lambda}(x),\Sigma_{\lambda}(x)) = \Phi_{\lambda}(x)$ are computed by an encoder neural network $\Phi$.
  \item \textbf{Decoder} $p_{\theta}(x|z) = \mathcal{N}(x | \mu_{\theta}(z), \Sigma_{\theta}(z))$ where $(\mu_{\theta}(z),\Sigma_{\theta}(z)) = \Psi_{\theta}(z)$ are computed by a decoder neural network $\Psi$.
  \item \textbf{Prior} $p_{\theta}(z)$ is set to a simple fixed distribution (e.g.~std Gaussian) to ensure $KL(q_{\lambda}(z|x) \| p_{\theta}(z))$ is trivial to compute.
  \item We express samples $z = \mu_{\theta}(x) + \Sigma_{\theta}(x)^{-\frac{1}{2}} \epsilon$ where $\epsilon$ is a sample from an std Gaussian, also known as *reparameterization trick*. We also use the shorthand notation $z = \Phi(x) \square \epsilon$ that captures the fact that $\mu_{\theta}(x)$ and $\Sigma_{\theta}(x)$ are estimated by the encoder $\Phi$.
  \end{enumerate}

% disambiguates the problem, The idea is to map the input to a unit Gaussian distribution which represents the space of possible outputs. A controlled version of the ground truth for the input is then used to select a point in the Gaussian, which serves to disambiguate the

% ground truth is then used  By carefully restricting information flow, the ground truth 3D output is then used to select the correct mode

% represent the space of possible outputs as a normal distribution
% The idea behind a CVAE is to represent the space of

% https://github.com/junyanz/BicycleGAN

Sharma et al.~\cite{xxx} adapt this formulation to model 3D skeletons and produce results on Human3.6M and HumanEva-I. They also propose two scoring methods to extract a single 3D reconstruction from the distribution which can be thought of as an alternative mechanism to the weighted quantizer scheme proposed here. There have also techniques~\cite{xxx} which extend these variational methods to generative adversarial netowrks. These are interesting and further reading here is available from xxx.

% Several previous papers have looked at the problem of modelling ambiguity in 3D human pose reconstruction. Early work includes~\citet{kinematic-jump-processes}, ~\citet{tracking-3d-human-figures} and \citet{density-prop}. More recently, ~\citet{akhter15pose-conditioned} learn a prior over human skeleton joint angles from a MoCap dataset.
% %(but not directly a prior on the SMPL parameters)
% % They model distributions conditioned on the parent of each joint in the kinematic tree and show that this prior can be used to obtain improved parametric fits.
% More recently,~\citet{li19generating} use the Mixture Density Networks model of~\cite{bishop94mixture} to capture ambiguous 3D reconstructions of sparse human body keypoints directly in physical space.
% \citet{sharma19monocular} learn a conditional variational auto-encoder to model ambiguous reconstructions as a posterior distribution; they also propose two scoring methods to extract a single 3D reconstruction from the distribution.

% BicycleGan etc.

\subsubsection{Min-of-$M$}

The idea of Min-of-N frameworks is to view the posterior distribution as a discrete set of $M$ hypotheses. A neural network is trained to produce $M$ hypotheses $\{f^{m}(Y_{i})\}_{m=1}^{M}$, where $Y_{i}$ is an input vector for the $i^{th}$ movie. In the literature, such multiple hypotheses networks are often trained with a so-called \emph{best-of-$M$} loss --- namely, during training, the loss is incurred only by the best of the $M$ hypothesis, back-propagating gradients from that alone~\cite{guzman2012multiple}. For example, with ground truth 3D points $X_{i}$ the min-of-$M$ loss is given as
\begin{equation}
  \mathcal{L}_\text{best}(Y;m^*)
  =
  \frac{1}{N}
  \sum_{i=1}^N
  \big \| X_{i} - f^{m_i^*}(Y_{i}) \big \|,
  ~~
  m_i^*
  =
  \operatornamewithlimits{argmin}_{m=1,\dots,M}
  \big \| X_{i} -  f^{m}(Y_{i}) \big \|,
\end{equation}

Note that this formulation can be seen as a variant of the mixture density networks as defined previously. In particular, consider an MDN with $M$ modes and reduce the covariance of each Gaussian to a single point. Further analysis of this model is left to the later method section.

This chapter opts for the \emph{best-of-$M$} approach since it has been show to outperform  alternatives (such as variational auto-encoders or mixture density networks) in tasks that are similar to our 3D human pose recovery, and which have constrained output spaces. For example, Rupprecht et al.~\cite{rupprecht17learning} demonstrate the model on classicly ambiguous tasks including human pose estimation and future prediction. The primary contribution in this chapter is the introduction of a principled multi-hypothesis framework which models ambiguities in monocular pose recovery.

% WORKS THAT USE MDNs
%% https://arxiv.org/abs/1807.01511 (deep autoencoder

% \subsubsection{Notes}

% % https://wiseodd.github.io/techblog/2016/12/10/variational-autoencoder/

% To generate the required parameters, we use a 2 stage multi-layer perceptron (MLP) $G: X \mapsto (z, \beta, \gamma, s, t)^{M}$ to generate a latent code $z_{i}$ that can be passed through the inverse flow to generate joint angles $\theta = F^{-1}(z_{i})$, and the remaining components: $\beta_{i}, \gamma_{i}$ and orthographic camera parameters $s_{i}, t_{i}$.

% Once these $M$ SMPL parameterizations have been computed, we can then generate the $M$ meshes using the standard SMPL skinning pipeline:

% $P_{i} = S((\theta_{i}, \beta_{i}, \gamma_{i})$

% We make use of a standard joint regressor $J: K \mapsto \mathbb{R}^{16\times 2}$ in order to compute the approximate 3D joint positions $K \in \mathbb{R}^{16\times 2}$ associated with the input 2D joints. Orthographic projection $\pi(s, t): \mathbb{R}^{16\times 3} \mapsto \mathbb{R}^{16\times 2}$ is then used to generate the predicted set of 2D points.
% From these diverse modes, we apply a loss to ensure they are all consistent with respect to the input set of 2D joints:

% \begin{equation}
%     L_{reproj-input} = \sum_i{|| \pi(s_{i}, t_{i})[J(P_{i})] - x||}
% \end{equation}

% We then apply a 3D joint loss in order to identify the best hypothesis among predicted modes:

% % i* = argmin_{i}{{J(Y_{i})} - y}

% \begin{equation}
%     L_{obj} = L_{joint3d} + L_{reproj-gt} + L_{vertex}
% \end{equation}

% Where

% This, and this and this are the equations.

% We consider the problem of reconstructing $x$ (3D object) form a partial observation $y$ (2D projection).
% We also assume that observations are noisy, so the observation process is described by a known conditional distribution $p(y|x)$.
% We assume that $x$ has a ``useful'' parameterization or code $z$.
% The parameterization is another random variable linked to $x$ via distribution $p(x|z)$.
% While this could be learned, we assume a parameterization is given to us (e.g. SMPL).

% The goal is to: 1) learn an ``inverse function'' $q(z|x)$ inverting the parameterization of the data $x$ and 2) learn a ``prediction'' function $q(z|y)$ that recovers $x$ in code space.
% The assumption is that expressing reconstruction ambiguities is easy in code space, so that $q(z|y)$ is far simple to model than $p(x|y)$; however, we pay the price of having to learn $q(z|x)$.

% We formulate this similar to a conditional variational autoencoder (cVAE):

% $$
% \begin{aligned}
% \log p(x|y)
% &= \log \int p(x,z|y) \,dz \\
% %&= \log \int p(x,z|y) \frac{q(z|x,y)}{q(z|x,y)} \,dz \\
% &= \log \int \underbrace{p(x|z,y) \frac{p(z|y)}{q(z|x,y)}}_{\text{$\approx$ const.}} q(z|x,y) \,dz \\
% &= \log \int p(x|z,y) \frac{p(z|y)}{q(z|x,y)} q(z|x,y) \,dz \\
% &\geq \int \log \left(p(x|z,y) \frac{p(z|y)}{q(z|x,y)}\right) q(z|x,y) \,dz \\
% &=\int \log p(x|z,y) ~q(z|x,y) \,dz \\
% &~~~~~- KL\left(q(z|x,y) \| p(z|y)\right)
% \end{aligned}
% $$

% This bound is tight whenever $q(z|x,y) \propto p(x|z,y) p(z|y)$, i.e. $q(z|x,y) = p(z|x,y)$.
% Now we make the assumption that $x$ contains a superset on the information of $y$ on $z$, i.e. $y \perp z ~| x$.
% This translates in the relations $p(x|z,y) = p(x|z)$ and $p(z|x,y) = p(z|x)$.
% This also means that we can consider $q(z|x) = q(z|x,y)$.
% With these simplification, we get:

% \begin{multline}
%   \log p(x|y) \geq \\
% \int \log p(x|z) ~q(z|x) \,dz - KL\left(q(z|x) \| p(z|y)\right)
% \end{multline}

% The bound is tight when $q(z|x) = p(z|x)$ as in a standard VAE.
% By comparisong, a standard VAE without conditioning looks like:
% \begin{multline}
%   \log p(x) \geq \\
% \int \log p(x|z) ~q(z|x) \,dz - KL\left(q(z|x) \| p(z)\right)
% \end{multline}
% So the only difference is $p(z)$ instead of $p(z|y)$ in the KL divergence (and the fact that one bounds $p(x)$ instead of $p(x|y)$.)

% This model combines:
% \begin{enumerate}
%   \item The forward parameterization $p(x|z)$ (e.g. SMPL, linear basis).
%   \item The inverse parameterization $q(z|x)$, to be learned.
%   \item The reconstruction in code space $p(z|y)$. This distribution models the ambiguity in the reconstruction, must be learned, and is assumed to be simpler to model than $p(x|y)$ directly.
% \end{enumerate}

% \subsection{Older}

% A VAE is model of a distributon $p(x)$ expressed as teh marginal of $p(x,u)$ where $u$ is a code.
% The idea is that $p(x|u)$ can be learned as a neural network and $p(u)$ is given a-priori and simple.
% Furthermore, we also learn a conditional distribution $q(u|x)$ that approximates $p(u|x)$ and helps us computing the log-likelihood of the model.

% This is done as follows:
% $$
% \begin{aligned}
% \log p(x)
% &= \log \int p(x,u) \,du \\
% &= \log \int p(x,u) \frac{q(u|x)}{q(u|x)} \,du \\
% &= \log \int p(x|u) \frac{p(u)}{q(u|x)} q(u|x) \,du \\
% &= \log \int p(x|u) \frac{p(u)}{q(u|x)} q(u|x) \,du \\
% &\geq \int \log \left(p(x|u) \frac{p(u)}{q(u|x)}\right) q(u|x) \,du \\
% &=\int \log p(x|u) ~q(u|x) \,du - KL\left(q(u|x) \| p(u)\right)
% \end{aligned}
% $$
% Note that the model is comprised by
% \begin{itemize}
% \item the encoder $q(u|x)$  (trainable)
% \item the decoder $p(x|u)$ (trainable)
% \item the prior $p(u)$ (fixed)
% \end{itemize}
% The log-likelihood of this model is
% \begin{multline}
% E_{p_{\text{gt}}(x)}[
%   \log p(x)
% ]
% \geq \\
% E_{p_{\text{gt}}(x)}
% \left[
% E_{q(u|x)}[
%   \log p(x|u)
% ]
% -
% KL\left(q(u|x) \| p(u)\right)
% \right]
% \end{multline}
% where $p_{\text{gt}}(x)$ is the true distribution of $x$.

% In a standard implementation, we do the following:

% \begin{enumerate}
% \item \textbf{Encoder} We set $q(u|x) = \mathcal{N}(u | \mu_x, \Sigma_x)$ where $(\mu_x,\Sigma_x) = \Phi(x)$ are computed by an encoder network $\Phi$.
% \item \textbf{Decoder} We set $p(x|u) = \mathcal{N}(x | \mu_u, \Sigma_u)$ where $(\mu_u,\Sigma_u) = \Psi(u)$ are computed by a decoder network $\Psi$.
% \item \textbf{Prior} We set $p(u)$ to a simple fixed distribution (e.g.~std Gaussian) so that $KL(q(u|x) \| p(u))$ is trivial to compute.
% \item We express samples $u = \mu_x + \Sigma_x^{-\frac{1}{2}} \epsilon$ where $\epsilon$ is a sample from an std Gaussian, also known as *reparameterization trick*. We also use the shorthand notation $u = \Phi(x) \square \epsilon$ that captures the fact that $\mu_x$ and $\Sigma_x$ are estimated by the encoder $\Phi$.
% \end{enumerate}

% Then, given a dataset of samples $x_1,x_2,\dots,...$, we get random noise vectors $\epsilon_1,\epsilon_2,\dots$ and use SGD to follow the gradient:
% $$
%   \nabla_{\Phi,\Psi}
%   \left[
%     \log \mathcal{N} (x_i| \Psi(\Phi(x_i) \square \epsilon_i))
%     - KL(\mathcal{N}(u| \Phi(x_i)) \| p(u))
%   \right]
% $$

% Note that this quantity can be computed in closed form.

% \subsection{Variant}

% Now we consider a variant of the model above in which $p(x|u)$ is given and fixed and $p(u)$ must be learned (which is the opposite of what we had before).

% The derivation of the bound above is the same, but the implementation is a little different. We have:

% \begin{itemize}
% \item the encoder $q(u|x)$ (trainable)
% \item the decoder $p(x|u)$ (fixed)
% \item the prior $p(u)$ (trainable)
% \end{itemize}

% \paragraph{Example: pose.} $x \in\mathbb{R}^{3\times K}$ are 3D points, $u\in\mathbb{R}^d$ a set of angles, and $p(x|u)$ the action of the kinematic tree.

% We may then build the model as follows:
% \begin{itemize}
% \item \textbf{Encoder} We set $q(u|x) = \mathcal{N}(u | \mu_x, \Sigma_x)$ where $(\mu_x,\Sigma_x) = \Phi(x)$ are computed by an encoder network $\Phi$. Same as before.

% \item \textbf{Decoder} We set $p(x|u) = \mathcal{N}(x | \mu_u, \Sigma)$ where $\mu_u = f(u)$ is computed by the a known model $f$ and $\Sigma$ is a small fixed variance.

% \item \textbf{Prior} $p(u)$ is a representation of a not-so-trivial prior distribution.

% \item We use the reparameterizatino trick as before, since that affects the encoder.
% \end{itemize}
% However, now the story is slightly different since $p(x|u)$ is fixed and $p(u)$ must be learned.
% For this, we rewrite the bound as:
% \begin{multline}
% E_{p_{\text{gt}}(x)}[
%   \log p(x)
% ]
% \geq
% \\
% E_{p_{\text{gt}}(x)}
% E_{q(u|x)}\left[
% \log p(x)
% -
% \log \frac{q(u|x)}{p(u)}
% \right]
% =
% \\
% E_{p_{\text{gt}}(x)}
% E_{q(u|x)}\left[
% \log p(x|u)
% -
% \log q(u|x)
% +
% \log p(u)
% \right]
% \end{multline}

% \paragraph{Interpretation.} In expectation under $q(u|x)$, the term $\log p(x|u)$ is a cross entropy term that encourages $x$ to be reconstructed well from the sample $u \sim q(u|x)$. The term $-log q(u|x)$ is an entropy term that prevents $q(u|x)$ from collapsing to a deterministic function. The term $\log p(u)$  is a cross-entropy that encourages $p(u)$ to match the required prior on the parameters.

% In terms of SGD, we have something like:
% $$
%   \nabla_{\Phi,p(u)}
%   \left[
%     \log \mathcal{N} (x_i| \Psi(\Phi(x_i) \square \epsilon_i))
%     + H(\mathcal{N}(u|\Phi(x_i)))
%     + \log p(u_i)
%   \right]
% $$
% where $H$ is the entropy of the Gaussian distribution $\mathcal{N}(u|\mu_{x_i}, \Sigma_{x_i})$ (also easy to compute in closed form).

% The ``tricky'' bit is to express $p(u_i)$ with an expressive and learnable model.

% \subsection{Conditional VAEs}

% We now want to modify the formulation so as to consider a partial observation $y$ and be able to compute $p(x|y)$ from it.

% \paragraph{Example: pose} We consider the case in which we have $y = \Pi(x)$ as observation, usually some form of projection or distortion of $x$ (possibly stochastic/noisy).

% \paragraph{CVAE Formulation 1.} This takes VAE and adds $|y$ everywhere, using the fact $p(x|u,y) = p(x|u)$:
% $$
% \begin{aligned}
% \log p(x|y)
% &= \log \int p(x,u|y) \,du \\
% &= \log \int p(x,u|y) \frac{q(u|y)}{q(u|y)} \,du \\
% &= \log \int p(x|u) \frac{p(u|y)}{q(u|y)} q(u|y) \,du \\
% &= \log \int p(x|u) \frac{p(u|y)}{q(u|y)} q(u|y) \,du \\
% &\geq \int \log \left(p(x|u,y) \frac{p(u|y)}{q(u|y)}\right) q(u|y) \,du \\
% &=\int \log p(x|u) ~q(u|y) \,du - KL\left(q(u|y) \| p(u|y)\right)
% \end{aligned}
% $$

% The strange thing is that this requires to use $p(u|y)$ as prior rather than $p(u).$

% \paragraph{CVAE Formulation 2.} This one seems much better: just replace $q(u|x)$ with $q(u|y)$ in the math:
% $$
% \begin{aligned}
% \log p(x)
% &= \log \int p(x,u) \,du \\
% &= \log \int p(x|u) p(u) \frac{q(u|y)}{q(u|y)}  \,du \\
% &\geq \int \log \left(\frac{p(x|u) p(u)}{q(u|y)}\right) q(u|y) \,du \\
% &=
% \int \log p(x|u)~ q(u|y) \,du +
% \int \log \frac{p(u)}{q(u|y)} q(u|y) \,du\\
% &=
% \int \log p(x|u)~ q(u|y) \,du
% -
% KL(q(u|y) \| p(u)).
% \end{aligned}
% $$

% We also make an important contribution to better optimize our multi-hypothesis predictions.

\subsection{Overcoming limitations with best-of-$M$ frameworks}

Theoretically, best-of-$M$ can minimize its loss by quantizing optimally (in the sense of minimum expected distortion) the posterior distribution, which would be desirable for coverage.
However, this is \emph{not} the only solution that optimizes the best-of-$M$ training loss, as in the end it is sufficient that \emph{one} hypothesis per training sample is close to the ground truth.
In fact, this is exactly what happens; for instance, during training hypotheses in best-of-$M$ are known to easily become degenerate and `die off', a clear symptom of this problem.

A major drawback of the \emph{best-of-$M$} approach is that it only guarantees that \emph{one} of the hypotheses lies close to the correct solution; however, it says nothing about the plausibility, or lack thereof, of the \emph{other} $M-1$ hypotheses, which can be of arbitrarily poor quality.

Not only does this mean that most of the hypotheses may be uninformative, but in an application it is impossible to determine \emph{which} hypothesis should be used, leaving open the possibility of selecting a `bad' one.
This has also a detrimental effect during learning because it makes gradients sparse as prediction errors are back-propagated only through one of the $M$ hypotheses for each training image.

In order to address these issues, a first contribution in this chapter is a \emph{hypothesis reprojection loss} that forces each member of the multi-hypothesis set to correctly reproject to 2D image keypoint annotations.
The main benefit is to constrain the \emph{whole} predicted set of meshes to be consistent with the observed image, not just the best hypothesis, also addressing gradient sparsity.

Next, another drawback of the best-of-{$M$} pipeline is to be tied to a particular value of $M$, whereas in applications it is useful to be able to tune the number of hypotheses. In the standard best-of-{$M$} implementation, this would necessitate completely retraining the network with the updated value of $M$.
Furthermore, minimizing the reprojection loss makes hypotheses geometrically consistent with the observation, but not necessarily likely.
The second contribution is therefore to improve the flexibility of best-of-$M$ models by allowing them to output any smaller number $n<M$ of hypotheses while at the same time making these hypotheses \emph{more representative of likely} poses.
The new method, referred to henceforth as $n$-quantized-best-of-$M$, does so by quantizing the best-of-$M$ model to output weighed by a \emph{explicit pose prior}, learned by means of normalizing flows.

\subsection{Methods to learn a 3D articulated pose prior}

% Harp back to the previous chapter talking about gaussian pose priors etc.

There are a number of techniques in the literature for building precise priors over human pose deformation. A common formulation, relied upon in previous chapters, is to impose a unimodal Gaussian on the 3D kinematic tree rotations. The mean and covariance parameters can either be obtained from the 3D scans used to construct the morphable model, or alternatively from a separate 3D dataset (e.g. CMU) which exhibits a wider range of motion. Alternative priors have been designed to specifically penalize ``interpenetration'', a phenomenon that a body part self-intersects the mesh or a prior that specifically penalizes ``out-of-range'' rotations.

Recently, more expressive 3D pose priors have been designed based on modern deep laerning advancements. Kanazawa et al.~\cite{xxx} propose a method that uses a per-part discriminator to ensure network-predicted 3D models lie on a manifold of plausible bodies. In addition, VAE methods have been proposed which constrain poses to a Gaussian parameterized by higher-level features. % CITE

Concurrently to the work in this chapter, priors have been built over 3D human pose using \emph{normalizing flows}. In particular, ~\cite{xu-2020-cvpr} release a prior for their new GHUM/GHUML model, and~\cite{weakly-supervised-normflow} build a prior on SMPL joint angles to constrain their weakly-supervised network. The prior introduced in this chapter differs slightly as it is learnt on 3D morphable model joint locations, rather than the Rodrigues rotations.

% In order to do so, since best-of-$M$ lacks an explicit n\emph{prior} over plausible human body poses, thus resulting in an optimal coverage of the \emph{likely} solutions.

%an arbitrary number of $n<M$ hypotheses that optimally cover the set of plausible poses.

\subsection{Sourcing datasets for training and evaluation}

The method described in this chapter focuses on modelling ambiguities in 3D reconstructions using a min-of-$M$ loss. To train such a model, gradients are propagated through the best hypothesis -- in this case, the closest to a 3D ground truth mesh. Consequently, a dataset of images with corresponding 3D annotations is required. As previously described, such datasets are limited in number and size for animals. Despite featuring only one animal subject and being captured in the controlled environment, the RGBD Dog dataset~\cite{xxx} does allow for direct evaluation on animals. However, in order to provide a fuller evaluation of the method, extensive evaluation is conducted against competitive 3D human reconstruction benchmarks.

\subsection{System overview}
To summarise, the key contributions in this chapter are as follows.
First, a multiple hypothesis network is introduced in order to deal with the challenge of 3D mesh reconstruction of articulated objects in \emph{ambiguous} scenarios.
Second, a mechanism referred to as \emph{$n$-quantized-best-of-$M$} allows best-of-$M$ models to generate an arbitrary number of $n<M$ predictions.
Third, a mode-wise re-projection loss for multi-hypothesis prediction is used to ensure that predicted hypotheses are \emph{all} consistent with the input.

Empirically, this work demonstrates the first formal study of ambiguous 3D reconstruction for an animal category. In addition, the method achieves competitive state-of-the-art monocular mesh recovery accuracy on Human3.6M, its more challenging version augmented with heavy occlusions, and the 3DPW datasets.
The method is also ablated in a formal study to validate each of the modelling choices, demonstrating their positive effect.

\input{Chapter6/FigTex/fig-method.tex}


\subsection{IPv6 hierarchical address allocation}\label{sec:mhcl}

MHCL partitions the address space available to the border router, for instance the 64 least-significant bits of the IPv6 address (or a compressed 16-bit representation of the latter), hierarchically among nodes connected to the border router through a multihop cycle-free topology (implemented by standard protocols, such as RPL~\cite{rfc6550} or CTP~\cite{Fonseca:2009}). Each node
receives an address range from its parent and partitions it among its children, until all nodes receive an address (see Figure \ref{fig:mhcl}). Since the address allocation is performed in a hierarchical way, the routing table of each node can have $k$ entries, where $k$ is the number of its (direct) children. Each routing table entry aggregates the addresses of all nodes in the subtree rooted at the corresponding child-node. However, MHCL does not handle tree topology dynamics, i. e., a failure on a node or link can cause some local topology changes, which can break the hierarchical nature of MHCL. In \cite{mhcl}, only packet loss were handled. Therefore, it is necessary to design a mechanism to cope with local topology changes able to maintain routing with low memory footprint.

\begin{figure}[!ht]
    \centering
    \includegraphics[width=1\linewidth]{./Images/mhcl.pdf}
\caption{Example of MHCL address assignment. 8-bit address space at the root and $6.25\%$ of address range reserved for future/delayed connections}
    \label{fig:mhcl}
\end{figure}

In order to decide how the available address space is partitioned, nodes need to collect information about the topology of the network. To do this, each node $n_i$ counts the total number of its descendants, i.e., the size of the subtree rooted at itself, and propagates it to its parent. Moreover, $n_i$ saves the number of descendants of each child. Once the root has received the number of descendants of each child, it partitions the available address space into $k$ slices of size proportional to the size of the subtree rooted at each child, also leaving a portion of $r\%$ as reserve. Each node $n_i$ repeats the space partitioning procedure upon receiving its own address space from the parent and sends the proportional address slices to the respective children. The idea is to allocate larger address portions to larger subtrees, which becomes important in especially large networks, because it maximizes the address space utilization. Experiments in \cite{mhcl} shown that MHCL is efficient in number of messages and time. Nevertheless, due to simulations' constraints, the results were not satisfactory and only few scenarios were evaluated. In addition, there were a lot of dependency between the proposed protocol and RPL, which can affect the results.


\documentclass{easychair}

\usepackage{doc}
\usepackage{scalerel}
\usepackage{amsfonts}

\newcommand{\easychair}{\textsf{easychair}}
\newcommand{\miktex}{MiK{\TeX}}
\newcommand{\texniccenter}{{\TeX}nicCenter}
\newcommand{\makefile}{\texttt{Makefile}}
\newcommand{\latexeditor}{LEd}
\newcommand{\emptycon}{\scaleobj{.75}\bullet}
\newcommand{\ext}{\triangleright}
\newcommand{\arri}{\Rightarrow}

\newcommand{\ToS}{\mathsf{ToS}}
\newcommand{\U}{\mathsf{U}}
\newcommand{\Code}{\mathsf{Code}}
\newcommand{\Ty}{\mathsf{Ty}}
\newcommand{\V}{\mathsf{V}}
\newcommand{\C}{\mathsf{C}}

\title{Using Two-Level Type Theory for Staged Compilation %
  \thanks{The author was supported by the European Union, co-financed
    by the European Social Fund (EFOP-3.6.3-VEKOP-16-2017-00002).}}

\author{
Andr\'as Kov\'acs
}

% Institutes for affiliations are also joined by \and,
\institute{
  E\"otv\"os Lor\'and University,
  Budapest, Hungary \\
  \email{kovacsandras@inf.elte.hu}
}

\authorrunning{Kov\'acs}
\titlerunning{Using Two-Level Type Theory for Staged Compilation}
\pagenumbering{gobble}
\begin{document}

\maketitle

Two-level type theory \cite{twolevel} (2LTT) is a system for performing certain
metatheoretic constructions and reasoning involving object-level type
theories. Such reasoning is always possible by simply embedding object theories
in metatheories, but in that case we have to explicitly handle a deluge of
technical details about the object theory, most notably substitutions. If
everything that we aim to do is natural with respect to object-level
substitution, we can instead use 2LTT, which can be viewed as a notation for
working with presheaves over the object-level category of substitutions.

Likewise in metaprogramming, there is a spectrum: we can simply write programs
which output raw source code, or use staging instead, which is safer and more
convenient, but also restricted in some ways. In the current work we observe
that 2LTT is a powerful model for generative staged compilation.

\emph{Basic rules of 2LTT}. We have universes $\U^s_{i}$, where $s \in \{0,1\}$,
denoting a \emph{stage} or level in the 2LTT sense, and $i \in \mathbb{N}$
denotes a usual level index of \emph{sizing} hierarchies. The two dimensions of
indexing are orthogonal, and we will elide the $i$ indices in the following. We
assume Russell-style universes. For each $\Gamma \vdash A : \U^0$, we have
$\Gamma \vdash \Code\,A : \U^1$. Quoting: for each $\Gamma \vdash t : A$, we
have $\Gamma \vdash\,<\!t\!> : \Code\,A$. Unquoting: for each $\Gamma \vdash t :
\Code\,A$, we have $\Gamma \vdash\,\sim\!t : A$. Moreover, quoting and unquoting
form an isomorphism up to definitional equality. $\U^0$ and $\U^1$ can be closed
under arbitrary additional type formers.

The idea of staging is the following: given a closed $A : \U^0$ with a closed $t
: A$ in 2LTT, there are unique $A'$ and $t'$ in the object theory, which become
definitionally equal to $A$ and $t$ respectively after being embedded in
2LTT. In short, every meta-level construction can be computed away, and only
object-level constructions remain in the result. Annekov et al. \cite{twolevel}
only showed mere existence of $A'$ and $t'$ (as a conservativity theorem for
2LTT). We can get unique existence as well, using the normalization of 2LTT:
by induction on the (unique) normal forms of $A$ and $t$, we can show that they
cannot contain meta-level subterms. This shows that normalization is a sound
staging algorithm, but in practice we do not want to compute full normal forms;
we want to compute \emph{meta-level redexes only}. This can be done with a
variation of standard normalization-by-evaluation \cite{abel2013normalization,
  Wieczorek:2018:CFN:3176245.3167091} which also keeps track of stages.

\subsubsection*{Applications}

\textbf{Control over inlining and compile-time computation.} We can define two
variations of the polymorphic identity function for object-level types:
\begin{alignat*}{3}
  & \mathit{id} : (A : \U^0) \to A \to A \hspace{2em}&& \mathit{id'} : (A : \Code\,\U^0) \to \Code\,\sim\!A \to \Code\,\sim\!A\\
  & \mathit{id} = \lambda\,A\,x.\,x      && \mathit{id'} = \lambda A\,x.x
\end{alignat*}
The second version is evaluated at compile time. For example,
$\sim\!(\mathit{id'}<\!\mathsf{Bool^0}\!>\,<\!\mathsf{true^0}\!>)$ can be used
in object-level code, which is computed to $\mathsf{true^0}$ by staging. We can
also freely use induction on meta-level values to generate object-level code,
including types. Hence, 2LTT supports full dependent types (with universes and
large elimination) in staging.

\textbf{Monomorphization.} We assume now that the object language is a
\emph{simple type theory}. In this case, there is no universe $\U^0$ in the
object-level, so there is no $\Code\,\U^0$, but we can still freely include a
meta-level type $\Ty^0$ whose terms are identified with object-level types. Now,
meta-level functions can be used for quantification over object-level types, as
in $\mathit{id} : (A : \Ty^0) \to \Code\,A \to \Code\,A$. However, since the
object theory is simply typed and monomorphic, all polymorphism is guaranteed to
compute away during staging.

\textbf{Control over lambda lifting and closure creation}. We assume now a
dependent type theory on both levels, but with a \emph{first-order function
type} on the object level. This is defined by splitting $\U^0$ to a universe
$\V^0$ which is closed under inductive types but not functions, and a universe
$\C^0$ which has $\V^0$ as a sub-universe, and is closed under functions with
domains in $\V^0$ and codomains in $\C^0$. This object theory supports
compilation which requires only lambda lifting, but no closures. On its own, the
object theory is fairly restricted, but together with staging we have a
remarkably expressive system. Then, we can close $\V^0$ under a separate type
former of closure-based functions, thereby formally distinguishing
lambda-liftable functions from closure-based functions. This enables typed
analysis of various optimization and fusion techniques. E.g.\ we get guaranteed
closure-freedom in code output if a certain fusion technique can be formalized
with only first-order function types. In particular, this may obviate the need
for arity analysis \cite{DBLP:journals/cl/Breitner18} in fold-based fusion.

\textbf{Memory layout control}. We assume again a dependent theory on both
levels, but now index $\U^0$ over \emph{memory layouts}. For example,
$\U^0\,\mathsf{erased}$ may contain runtime-erased types, and
$\U^0\,(\mathsf{word64} \times \mathsf{word64})$ may contain types represented
as unboxed pairs of machine words. We assume a meta-level type of
layouts. Hence, we can abstract over layouts, but after staging every layout
will be concrete and canonical in the output. This can be viewed as a more
powerful version of \emph{levity polymorphism} in GHC \cite{levity}, and a way
to retain both dependent types and non-uniform memory layouts in the object
theory.

\subsubsection*{Potential extensions}

\textbf{More stages, stage polymorphism}. The standard presheaf semantics of
2LTT can be extended to more levels in a straightforward way. It seems feasible
to also allow quantifying over all smaller levels, at a given level.

\textbf{Stage inference}. $\Code$ preserves all negative type formers up to
definitional isomorphism \cite{twolevel}, e.g. $\Code\,(A \to B) \simeq
(\Code\,A \to \Code\,B)$. This can be used to support inference for staging
annotations, by automatically inserting transports along preservation
isomorphisms during elaboration. The previous
$\sim\!(\mathit{id'}<\!\mathsf{Bool^0}\!>\,<\!\mathsf{true^0}\!>)$ example could
be simply written as $\mathit{id'}\,\mathsf{Bool^0}\,\mathsf{true^0}$ in the
surface language, and elaboration would transport $\mathit{id'}$ appropriately.
We demonstrated the practical feasibility of such stage inference in a prototype
implementation.

\textbf{Induction on $\Code$}. Basic 2LTT supports \emph{any model} of the
object theory in the presheaf semantics, it does not assume that we have
presheaves over the initial model (syntax). Hence, $\Code\,A$ is a black box
without elimination principles. However, if we are interested in staged
compilation, we can assume the object level to be syntactic and consistently add
operations on $\Code\,A$ which rely on that assumption, e.g. conversion
checking, pattern matching, or induction on normal forms of object-level
expressions.

\bibliographystyle{plain}
\bibliography{references}

\end{document}


\section{Blinking LEDs}

\project{Making LEDs blink with BASIC.}{In this project, you will learn how to make LEDs blink.}

This worksheet uses the same LED circuit that was built in Section~\ref{s:leds}.

\subsection*{Equipment Required}

The circuit built in CamJam EduKit Section~\ref{s:leds}.

\subsection*{Code---Blink Twice}

Type in the following code:
\begin{basic}
10 REM BLINK
20 REM SETUP LOCATIONS
30 UP=56577:DDR=56579
40 REM PIN VALUES FOR LEDS
50 R=1:Y=2:G=4:A=R+Y+G
60 REM SET PINS 0,1,2 FOR OUTPUT
70 POKE DDR,A
80 REM CLEAR USER PORT (ALL PINS OFF)
90 POKE UP,0
100 REM TURN LEDS ON
110 POKE UP,A
120 FOR I=1 TO 720:NEXT I:REM PAUSE FOR 1 SECOND
130 REM TURN LEDS OFF
140 POKE UP,0
150 FOR I=1 TO 720:NEXT I:REM PAUSE FOR 1 SECOND
160 REM TURN LEDS ON
170 POKE UP,A
180 FOR I=1 TO 720:NEXT I:REM PAUSE FOR 1 SECOND
190 REM TURN LEDS OFF
200 POKE UP,0
\end{basic}

Once complete, save the file as ``3 BLINK''.

\subsection*{Running the Code}

Run the code.  You will see the three LEDs turn on then turn off twice, with one second in between each change.

If your code does not run and an error is reported, edit the code again.

\subsection*{Code---Blink Forever}

Our next piece of code will flash the lights on and off forever (or until you press ``RUN/STOP'').

Type in the following code:
\begin{basic}
10 REM BLINK FOREVER
20 REM SETUP LOCATIONS
30 UP=56577:DDR=56579
40 REM PIN VALUES FOR LEDS
50 R=1:Y=2:G=4:A=R+Y+G
60 REM SET PINS 0,1,2 FOR OUTPUT
70 POKE DDR,A
80 REM CLEAR USER PORT (ALL PINS OFF)
90 POKE UP,0
100 POKE UP,A:REM TURN LEDS ON
110 FOR I=1 TO 720:NEXT I:REM PAUSE FOR 1 SECOND
120 POKE UP,0:REM TURN LEDS OFF
130 FOR I=1 TO 720:NEXT I:REM PAUSE FOR 1 SECOND
140 GOTO 100:REM LOOP FOREVER
\end{basic}

Save the program as ``3 BLINK FOREVER''.

\subsection*{Running the Code}

Run the code.  You will see the three LEDs turn on and off forever, or until you press ``RUN/STOP''.

If the LEDs are still on when you stop the program, just execute the following command:
\begin{basic}
POKE UP,0
\end{basic}

If there is an error in the code, change the code and re-save it.  Re-run to check that you have corrected the error.


\documentclass[journal,12pt,twocolumn]{IEEEtran}

\usepackage{setspace}
\usepackage{gensymb}

\singlespacing

\usepackage[cmex10]{amsmath}
%\usepackage{amsthm}
%\interdisplaylinepenalty=2500
%\savesymbol{iint}
%\usepackage{txfonts}
%\restoresymbol{TXF}{iint}
%\usepackage{wasysym}
\usepackage{amsthm}

\usepackage{mathrsfs}
\usepackage{txfonts}
\usepackage{stfloats}
\usepackage{bm}
\usepackage{cite}
\usepackage{cases}
\usepackage{subfig}

\usepackage{longtable}
\usepackage{multirow}

\usepackage{enumitem}
\usepackage{mathtools}
\usepackage{steinmetz}
\usepackage{tikz}
\usepackage{circuitikz}
\usepackage{verbatim}
\usepackage{tfrupee}
\usepackage[breaklinks=true]{hyperref}

\usepackage{tkz-euclide} %loads TikZ and tkz-base

\usetikzlibrary{calc,math}
\usepackage{listings}
    \usepackage{color}
    \usepackage{array}
    \usepackage{longtable}
    \usepackage{calc}
    \usepackage{multirow}
    \usepackage{hhline}
    \usepackage{ifthen}
    \usepackage{lscape}
\usepackage{multicol}
\usepackage{chngcntr}

\DeclareMathOperator*{\Res}{Res}

\renewcommand\thesection{\arabic{section}}
\renewcommand\thesubsection{\thesection.\arabic{subsection}}
\renewcommand\thesubsubsection{\thesubsection.\arabic{subsubsection}}

\renewcommand\thesectiondis{\arabic{section}}
\renewcommand\thesubsectiondis{\thesectiondis.\arabic{subsection}}
\renewcommand\thesubsubsectiondis{\thesubsectiondis.\arabic{subsubsection}}

\hyphenation{op-tical net-works semi-conduc-tor}
\def\inputGnumericTable{}                                 %%

\lstset{
%language=C,
frame=single,
breaklines=true,
columns=fullflexible
}

\begin{document}

\newtheorem{theorem}{Theorem}[section]
\newtheorem{problem}{Problem}
\newtheorem{proposition}{Proposition}[section]
\newtheorem{lemma}{Lemma}[section]
\newtheorem{corollary}[theorem]{Corollary}
\newtheorem{example}{Example}[section]
\newtheorem{definition}[problem]{Definition}

\newcommand{\BEQA}{\begin{eqnarray}}
\newcommand{\EEQA}{\end{eqnarray}}
\newcommand{\define}{\stackrel{\triangle}{=}}
\bibliographystyle{IEEEtran}
\providecommand{\mbf}{\mathbf}
\providecommand{\pr}[1]{\ensuremath{\Pr\left(#1\right)}}
\providecommand{\qfunc}[1]{\ensuremath{Q\left(#1\right)}}
\providecommand{\sbrak}[1]{\ensuremath{{}\left[#1\right]}}
\providecommand{\lsbrak}[1]{\ensuremath{{}\left[#1\right.}}
\providecommand{\rsbrak}[1]{\ensuremath{{}\left.#1\right]}}
\providecommand{\brak}[1]{\ensuremath{\left(#1\right)}}
\providecommand{\lbrak}[1]{\ensuremath{\left(#1\right.}}
\providecommand{\rbrak}[1]{\ensuremath{\left.#1\right)}}
\providecommand{\cbrak}[1]{\ensuremath{\left\{#1\right\}}}
\providecommand{\lcbrak}[1]{\ensuremath{\left\{#1\right.}}
\providecommand{\rcbrak}[1]{\ensuremath{\left.#1\right\}}}
\theoremstyle{remark}
\newtheorem{rem}{Remark}
\newcommand{\sgn}{\mathop{\mathrm{sgn}}}
\providecommand{\abs}[1]{\left\vert#1\right\vert}
\providecommand{\res}[1]{\Res\displaylimits_{#1}}
\providecommand{\norm}[1]{\left\lVert#1\right\rVert}
%\providecommand{\norm}[1]{\lVert#1\rVert}
\providecommand{\mtx}[1]{\mathbf{#1}}
\providecommand{\mean}[1]{E\left[ #1 \right]}
\providecommand{\fourier}{\overset{\mathcal{F}}{ \rightleftharpoons}}
%\providecommand{\hilbert}{\overset{\mathcal{H}}{ \rightleftharpoons}}
\providecommand{\system}{\overset{\mathcal{H}}{ \longleftrightarrow}}
	%\newcommand{\solution}[2]{\textbf{Solution:}{#1}}
\newcommand{\solution}{\noindent \textbf{Solution: }}
\newcommand{\cosec}{\,\text{cosec}\,}
\providecommand{\dec}[2]{\ensuremath{\overset{#1}{\underset{#2}{\gtrless}}}}
\newcommand{\myvec}[1]{\ensuremath{\begin{pmatrix}#1\end{pmatrix}}}
\newcommand{\mydet}[1]{\ensuremath{\begin{vmatrix}#1\end{vmatrix}}}
\numberwithin{equation}{subsection}
\makeatletter
\@addtoreset{figure}{problem}
\makeatother
\let\StandardTheFigure\thefigure
\let\vec\mathbf
\renewcommand{\thefigure}{\theproblem}
\def\putbox#1#2#3{\makebox[0in][l]{\makebox[#1][l]{}\raisebox{\baselineskip}[0in][0in]{\raisebox{#2}[0in][0in]{#3}}}}
     \def\rightbox#1{\makebox[0in][r]{#1}}
     \def\centbox#1{\makebox[0in]{#1}}
     \def\topbox#1{\raisebox{-\baselineskip}[0in][0in]{#1}}
     \def\midbox#1{\raisebox{-0.5\baselineskip}[0in][0in]{#1}}
\vspace{3cm}
\title{Matrix Theory (EE5609) Challenging Problem}
\author{Arkadipta De\\MTech Artificial Intelligence\\AI20MTECH14002}
\maketitle
\newpage
%\tableofcontents
\bigskip
\renewcommand{\thefigure}{\theenumi}
\renewcommand{\thetable}{\theenumi}
\begin{abstract}
This document proves that orthogonal vectors are linearly independent.
\end{abstract}
Download latex codes from
%
\begin{lstlisting}
https://github.com/Arko98/EE5609/tree/master/Challenge_2
\end{lstlisting}
%
\section{Problem}
Suppose that a set of nonzero vectors $\vec{v}_{1},\vec{v}_2,\dots\vec{v}_{n}$ are mutually orthogonal, i.e., $\vec{v_i^T}\vec{v}_{j}=0$ for $i\not=j$. Prove that these vectors are also linearly independent.
\section{Proof}
Let us consider the following linear combination
\begin{align}
c_1\vec{v_1}+c_2\vec{v_2}+\dots+c_n\vec{v_n}= 0\label{eq1}
\end{align}
We have to show that in \eqref{eq1}, $c_1=0$, $c_2=0$ and so on upto $c_n=0$.\\
We compute the dot product of \eqref{eq1} with $\vec{v_i}$ as follows -
\begin{align}
\vec{v_i^T}\brak{c_1\vec{v_1}+c_2\vec{v_2}+\dots+c_n\vec{v_n}} &=0\\
\implies c_1\vec{v_i^T}\vec{v_1}+c_2\vec{v_i^T}\vec{v_2}+\dots+c_n\vec{v_i^T}\vec{v_n} &=0\\
\intertext{As $\vec{v_i^T}\vec{v}_{j}=0$ for all $i\not=j$}
\implies c_i\vec{v_i^T}\vec{v_i} &=0\\
\implies c_i\norm{\vec{v_i}} &=0
\intertext{As the set of vectors are non zero, $\norm{\vec{v_i}} \not = 0$, hence}
\implies c_i & = 0\label{eqFinal}
\end{align}
\eqref{eqFinal} is true for all the vectors in the orthogonal set of vectors, hence,
\begin{align}
c_1 = c_2 = \dots = c_n = 0\label{eqproved}
\end{align}
Hence, from \eqref{eqproved}, the set of orthogonal vectors are linearly independent.
\end{document}