\documentclass[Thesis.tex]{subfiles}
\begin{document}
\chapter{Quantum Dots}
\label{chp:quantum-dots}

\glsresetall

We now present results for all methods discussed, applied on the example system
of \gls{qd} from \cref{sec:quantum-dots-theory}. We present first some
traditional VMC benchmarks using a typical Slater-Jastrow wave function form,
followed by the introduction of neural network-based wave functions.

\section{Benchmark}

As we shall restrict this analysis to only two interacting particles for the \gls{qd},
$\vX = (\vx_1\ \vx_2)$, our benchmark wave function is simple. We build
it up using the product of single particle ground states, multiplied
by a Pade-Jastrow correlation term:\footnote{We drop the constant factor from~\cref{eq:Phi-non-inter} because we have not normalized the wave function.}

\begin{align}
  \label{eq:qd-pade-jastrow-anzats}
  \psi_{PJ}(\vX) &= \Phi(\vX) \,J_P(\vX)\\
  &= \exp(-\alpha_G\sum_{i=1}^N \norm{\vx_i}^2 + \sum_{i < j} \frac{\alpha_{PJ}
    r_{ij}}{1 + \beta_{PJ} r_{ij}}),
\end{align}
where requirements of wave functions fix $\alpha_{PJ} = 1$~\cite{Kato-1957}, and $\alpha_G$ and $\beta_{PJ}$
are the only two variational parameters.\footnote{The specific requirement which
fixes the value of $\alpha_{PJ}$ is called Kato's cusp condition, after Japanese
mathematician T. Kato. It is a condition
on how the wave function should behave around the singularity of the Coulomb
potential, and ensures stability in our calculations.}

\subsection{Optimizing}

We have run a simple optimization of the above wave function, using initial
values of $\alpha_G = 0.5$ and $\beta_{PJ} = 1$. We used \acrlong{is} and the
ADAM optimization scheme. We used $\num{2000}$ optimization steps, each with
$\num{5000}$ \gls{mc} cycles. These values are examples, and we get
similar results for several other choices.

\begin{figure}[h]
   \centering
    \resizebox{\linewidth}{!}{%
        \input{scripts/QD-benchmark.py.tex}
    }
    \caption[Learning progression of Pade-Jastrow wave function]{\label{fig:QD-benchmark-pade-jastrow-training}Left: Performance of the
      wave function in \cref{eq:qd-pade-jastrow-anzats} as a function of
      training steps. Right: Corresponding progression of variational parameters as a function
      of training steps.\citesource{writing/scripts/QD-benchmark.py}}
\end{figure}

\begin{table}[h]
  \centering
  \caption[Energy estimates of Pade-Jastrow wave function]{Energy benchmark
using the Pade-Jastrow wave function, with $2^{22}$ samples and errors estimated
by an automated blocking algorithm by~\textcite{Jonsson-2018}. Energies in
atomic units $[\si{\au}]$. The symbols $CI_{-}^{95}$ and $CI_{+}^{95}$ represent
the lower and upper limits of a $\SI{95}{\percent}$ confidence interval, respectively. $\Std$
is the standard deviation of the energy and $\Var$ is the corresponding
variance. See \cref{fig:QD-benchmark-pade-jastrow-training} for source code
reference.}
  \input{scripts/QD-benchmark.py.table.tex}
  \label{tab:pade-jastrow-benchmark-energy}
\end{table}

\Cref{fig:QD-benchmark-pade-jastrow-training} shows the optimization as function
of percentage of training completed. We can observe that the optimizations
quickly settles down to a set of optimal values, where they only oscillate
slightly back and forth. \cref{tab:pade-jastrow-benchmark-energy} shows
statistics for the energy estimate obtained with the final parameter values. Comparing to the
analytical result of $\SI{3}{\au}$ these results are in good agreement. For
reference we have also given the results obtained without the Pade-Jastrow term,
i.e.\ the non-interacting ground state.

\subsection{Two-body Density}

Given that we now consider two interacting particles, a natural quantity to
investigate is the two-body density. Recall \cref{fig:verify-twobody} which
showed the two-body density for the non-interacting system as a simple Gaussian.
Computing $\rho(\vx_1, \vx_2)$ using $\psi_{PJ}$ with the final parameters
obtained previously we get \cref{fig:QD-benchmark-pade-jastrow-density}.

This plot shows a very different behavior. The particles no longer prefer to
(simultaneously) stay close to the center of the potential because of the Coulomb repulsion, and
the density maxima is pushed outwards. Additionally, there is a small dent in
the density when $r_1\approx r_2$, not only in the center. This effect is made
clearer by looking at the squared density. These results are quite intuitive,
while showcasing the highly non-trivial behavior that the two particles exhibit.

If we integrate out one of the particle positions, i.e.\ compute the one-body
density we get \cref{fig:QD-benchmark-pade-jastrow-onebody}. This shows that the
most probable position for any single particle is still in the center, but the
density is much more spread out to larger values of $r$.

\begin{figure}[h]
   \centering
    \resizebox{0.7\linewidth}{!}{%
        \input{scripts/QD-benchmark-density.py.tex}
    }
    \resizebox{0.7\linewidth}{!}{%
        \input{scripts/QD-benchmark-density.py.squared.tex}
    }
    \caption[Two-body density using Pade-Jastrow wave function]{\label{fig:QD-benchmark-pade-jastrow-density}Top: Two-body density
      described by $\psi_{PJ}$ with optimized parameters. Bottom: The squared
      equivalent of the top figure, included to emphasize the small decrease in
      density along the diagonal $r_1=r_2$.\citesource{writing/scripts/QD-benchmark-density.py}}
\end{figure}

\begin{figure}[h]
   \centering
    \resizebox{0.7\linewidth}{!}{%
        \input{scripts/QD-benchmark-onebody.py.tex}
    }
    \caption[One-body density using Pade-Jastrow wave function]{\label{fig:QD-benchmark-pade-jastrow-onebody}One-body density
      described by $\psi_{PJ}$ with optimized parameters. For reference, the
      exact one-body density for the non-interacting system is indicated by the
      dotted line. As expected, the Coulomb interaction pushes the particles to
      a greater distance from the origin, flattening out the density. \citesource{writing/scripts/QD-benchmark-onebody.py}}
\end{figure}
\clearpage

\section{Restricted Boltzmann Machine}

\begin{figure}[h]
  \centering
  \input{illustrations/rbm-diagram.tex}
  \caption[Illustration of a \acrfull{rbm}]{Example diagram of a Gaussian-binary \acrlong{rbm}, showed with four visible nodes
    and three hidden nodes. The red values are the parameters, and consist of
    visible layer bias, $\vb a$, hidden layer bias, $\vb b$ and connection
    weights $\mat W$. $M = \text{\# of particles}\times \text{\# of
      dimensions}=P\times D$ is the number of degrees of freedom in the system.\citesource{writing/illustrations/rbm-diagram.tex}}
  \label{fig:rbm-diagram-example}
\end{figure}

The first \gls{ml}-inspired model we have applied is a \acrfull{rbm}.
This type of model has seen a significant rise in usage
since~\textcite{Carleo602} demonstrated the \gls{rbm}'s capacity to
represent the wave function for some selected Hamiltonians. All the
Hamiltonians for which they showed successful results, however, had discrete
configuration spaces. The current system is continuous as the particles can have
any real valued coordinates, and so the type of \gls{rbm} must change as well.

While more than one choice exists, we have used a Gaussian-binary \gls{rbm}\@.
\Cref{fig:rbm-diagram-example} shows an example diagram of the network
structure. \textcite{Flugsrud-2018} has a full introduction to \glspl{rbm}, including
all details needed for \gls{vmc}\@. For our purposes it suffices to say that the
resulting wave function looks as follows:

\begin{align}
  \label{eq:rbm-def}
  \psi_{RBM}(\vX) &=
        e^{-\sum_i^{M} \frac{\qty(X_i-a_i)^2}{2\sigma^2}}
        \prod_j^N \qty(1 + e^{b_j+\sum_i^M \frac{X_iW_{ij}}{\sigma^2}}),
\end{align}
where $M = \text{\# of particles}\times \text{\# of dimensions}$ is the number of degrees of freedom and $N$ is the number
of hidden nodes. Note also that $X_i$ in the above refers to the $i$'th degree
of freedom, counting through $\mat X$ in row major order. The parameters are
$\vb a, \vb b$ and $\mat W$, and we hold $\sigma^2=1$ constant in this case.

If we set $\vb a = \vb 0$ we can recognize the first factor of~\cref{eq:rbm-def}
as the non-interacting ground state. That way we can consider the second factor
the Jastrow factor introduced by the \gls{rbm} structure. It has an unconventional
form, as it is not a pure exponential.

\subsection{Optimizing}

We produced the following results using normally distributed random
initial values for the parameters, running $\num{60000}$ optimization steps with
$\num{2000}$ \gls{mc} cycles each. We have also once again used \acrlong{is}
and ADAM\@. A new addition (not strictly necessary for similar results) is the use
of mild $L2$ regularization (\cref{sec:regularization}), which serves to drive parameters that do not
contribute towards zero. The results are similar for different hyperparameter
choices, and the above is simply one such example.

\Cref{fig:QD-rbm-training} shows the ground state energy as a function of
training steps, along with the progression of the variational
parameters. While we see a clear improvement in the initial stages, the \gls{rbm}
fails to converge as accurately as the benchmark. \cref{tab:rbm-energy-results}
shows the precise results of the final model. While slightly different results
are possible with different training settings, we have never observed the \gls{rbm}
achieve energies below $\SI{3.07}{\au}$, which is an error of about two orders
of magnitude larger than the benchmark.

\begin{figure}[h]
   \centering
    \resizebox{\linewidth}{!}{%
        \input{scripts/QD-rbm.py.tex}
    }
    \caption[Learning progression of an \acrshort{rbm} on quantum dots]{\label{fig:QD-rbm-training}Left: Performance of the
      wave function in \cref{eq:rbm-def} as a function of
      training steps. Right: Progression of variational parameters as a function
      of training steps.\citesource{writing/scripts/QD-rbm.py}}
\end{figure}

\begin{table}[h]
  \centering
  \caption[Energy estimates using an \gls{rbm} on quantum dots]{Energy using the \gls{rbm} wave function in~\cref{eq:rbm-def}, along with
the same wave function using input sorting to impose symmetry. Results obtained
from $2^{23}$ samples and errors estimated by an automated blocking
algorithm by~\textcite{Jonsson-2018}. Energies in atomic units $[\si{\au}]$. See
\cref{fig:QD-rbm-training} for source code reference.}
  \input{scripts/QD-rbm.py.table.tex}
  \label{tab:rbm-energy-results}
\end{table}

\begin{figure}[h]
   \centering
    \resizebox{\linewidth}{!}{%
        \input{scripts/QD-rbm.py.symmetry.tex}
    }
    \caption[Symmetry and weight matrix of an \acrshort{rbm} used on quantum dots]{\label{fig:QD-rbm-symmetry}Left: Permutation symmetry of the \gls{rbm} wave
function in~\cref{eq:rbm-def} as a function of training steps. Right: Color map
of the weight matrix of the \gls{rbm} after training. Weights have been scaled to
$[-1, 1]$ for simplicity. See \cref{fig:QD-rbm-training} for source code reference.}
\end{figure}

An important consideration that arises from this form of wave function model
is that it has no guarantee of satisfying the required permutation symmetry.
This is an attribute of most neural network based models. Because we know the
true ground state must have the correct symmetry, we would hope that the \gls{rbm} is
able to realize that a symmetric form is best. To this end we have defined a
metric $S(\psi)$, which has the property of being equal to 1 for symmetric
functions and $0$ for anti-symmetric ones. See \cref{app:symmetry-metric} for
its definition and details. The left plot in \cref{fig:QD-rbm-symmetry} shows a
plot of the symmetry metric of $\psi_{RBM}$ during training. Luckily we see
that the \gls{rbm}, initially starting fully non-symmetric, tends rapidly towards
$S(\psi_{RBM})=1$. Still it is not purely symmetric, and we see significant
oscillations around the maximum value.

In attempts to properly fix the symmetry of $\psi_{RBM}$, we have made two
significant attempts. The first was reducing the biases $\vb a$ to be $\vb
a\in\mathbb{R}^D$ and reducing the weights to $\mat W\in\mathbb{R}^{D\times N}$,
i.e.\ have parameters per dimension and not per degree of freedom. While this ensured
$S(\psi_{RBM})=1$, and the \gls{rbm} could still learn the ground state if the
particles where non-interacting (i.e.\ the pure Gaussian), it failed in the full
system. The next section looks closer at the weights, and explains why this
reduced form failed.

More successfully we imposed symmetry by sorting the inputs prior to feeding
them through $\psi_{RBM}$. While this was able to learn something, it stalled
out at $>\SI{3.1}{\au}$. In general it is our experience that imposing the
symmetry from the beginning of the training stops the \gls{rbm} from getting anywhere.
This is a manifestation of the classical problem in learning of balancing
exploitation and exploration, with this case suffering from a lack of exploration.

The only truly successful way we found was to train the \gls{rbm} as before, and then
apply sorted inputs on it once it had been fully trained. The results of this is
also shown in \cref{tab:rbm-energy-results} by $\psi_{SRBM}$ (symmetric \gls{rbm}). This lead to only a
marginal, not statistically significant increase in energy, and as such was a
successful way to impose the symmetry.

\subsection{Interpreting the Network}

The right plot in \cref{fig:QD-rbm-symmetry} shows a peculiar pattern emerging
from the weights $\mat W$ of the final model. Half of the weights appear
inconsequential, while the remaining come in pairs of equal value. If we
increase the number of hidden nodes, the result is always two similar columns with the extra elements
zeroing out. Exactly \emph{which} columns get the non zero values seems random,
and varies based on the random initialization of the weights.

Although not immediately obvious from the plot, there are only two unique
absolute values, i.e.\ $\pm v_1$ and $\pm v_2$. That is, in this particular
case:

\begin{align}
  \mat W = \mqty(
                    W_{1,1} & W_{1,2} & W_{1,3} & W_{1,4}\\
                    W_{2,1} & W_{2,2} & W_{2,3} & W_{2,4}\\
                    W_{3,1} & W_{3,2} & W_{3,3} & W_{3,4}\\
                    W_{4,1} & W_{4,2} & W_{4,3} & W_{4,4}
                )
                                                  \simeq \mqty(
                                                   v_1 & 0 & 0 &  v_2\\
                                                   v_2 & 0 & 0 & -v_1\\
                                                  -v_1 & 0 & 0 & -v_2\\
                                                  -v_2 & 0 & 0 &  v_1
                                                  )
\end{align}
This relation is only approximately true, as the \gls{rbm} has not learned to set the
weights exactly equal to each other, but it is close enough to show a clear
pattern.

This pattern explains why the attempt at imposing symmetry with a reduced form
of the \gls{rbm} failed. This pattern is simply not possible to express with one
weight per degree of freedom.

Having now observed this phenomena, we can seek to understand why the \gls{rbm} learns
this particular weight matrix. If we ignore the Gaussian factor of $\psi_{RBM}$
and substitute in the above matrix for $\mat W$, we get:

\begin{align}
  \begin{split}
  \psi_{RBM} \propto &\qty{1 + \exp[b_1 + \frac{1}{\sigma^2}\qty(v_1(x_1-x_2) + v_2(y_1-y_2))]}\\
                   \times &\qty{1 + \exp[b_4 + \frac{1}{\sigma^2}\qty(v_2(x_1-x_2) - v_1(y_1-y_2))]}
  \end{split}.
\end{align}
If $\vb{a}=a\vb{1}$ and $\vb{b}=b\vb{1}$ (i.e.\ biases are all equal in value),
this leads to an \gls{rbm} that is deceptively close to a fully permutation symmetric
function, broken only by the sign on $(y_1-y_2)$. The \gls{rbm} has, without direct
instructions, learned that the differences between the spacial coordinates of
the two particles are important. In contrast to $\psi_{PJ}$, which has the
distance $r_{12}$ explicitly incorporated, the \gls{rbm} has learned the closest
equivalent it has.

The fact that the \gls{rbm} is able to pick up on the importance of relative
differences in position is encouraging for further work with less constrained
neural networks. In this case, the \gls{rbm} could not learn to square the differences
(e.g.\ $(x_1 - x_2)^2$ instead of $(x_1-x_2)$), simply because it had no means
of expressing this.

\section{Neural Network}

Finally we turn our attention to the new contribution of this thesis -- applying
general neural networks as wave functions.

We have not been able to simply replace the entire wave function with a
neural network from the start. In order for \gls{vmc} optimization to work we need a
stable enough starting point to avoid immediate divergence. The better approach
is to bootstrap the network with an existing wave function as a starting point.
In our case we use the benchmark wave function $\psi_{PJ}$. There are two
options for how to do this:

\begin{enumerate}
\item Pre-train the network to emulate $\psi_{PJ}$ before starting \gls{vmc}
  optimization.
\item Use the network as an extra correlation factor multiplied with $\psi_{PJ}$.
\end{enumerate}
While there is a certain appeal to the first alternative, in the
sense that we would end up with a pure neural network wave function, the
downside is that pre-training will not be as accurate, as well as taking
some time. Small inaccuracies could also lead to instabilities from non-satisfied
cusp conditions/symmetry requirements. More importantly, we might
not know the correct form to pre-train until we have performed a \gls{vmc}
optimization to begin with. By using the second alternative, we are able to
train both network and base wave function simultaneously.

Because of this, we opted to simply treat the network as a correction factor,
aimed to fix the small discrepancy between the benchmark and the true ground
state. This approach allows us to bootstrap learning and start where existing
techniques have already taken us, and improve from there. It is also trivial to
implement in QFLOW, because of built in support for composing wave functions by
multiplication.

The proposed wave function is:

\begin{align}\label{eq:pade-dnn}
  \psi_{DNN}(\vX) &= \psi_{PJ}(\vX)\, f(\vX),
\end{align}
where $f$ represents some arbitrary neural network.

\subsection{Network Architecture}

We consider here only simple \glspl{ffnn} and leave other species of \glspl{nn} for
future research.

The size of the system determines the number of inputs.
The output layer is also essentially forced as we want a
scalar output from our network.\footnote{For applications that require complex
  wave functions, this can be easily modeled by having two outputs and
  interpret them as the real and imaginary component.} In addition, all
existing research implies that such correlation factors should be exponential.

For the hidden layers we have much more of a choice. By simple trail and error
we found $\tanh$ to be the most suitable activation function, and we have used this
in all cases. The number of nodes needs to be sufficiently large compared to the
number of inputs. This is to allow the network to learn a large amount of
different correlations, and not force it to settle right away.
The specific choices are again empirically motivated, and stem
from observing worse results with much smaller values and diminishing returns
for larger choices.

The later hidden layers should decrease in width as we approach the output, with
the idea that the network should gradually attempt to compact what it learns
into a smaller space. We have used powers of two for no particular reason other
than that it allows for even divisions.

While a large family of architectures are likely to perform well in this case,
we have settled on the following setup:

\begin{center}
  \begin{tabular}{lcc}
    \toprule
    \addlinespace
    Layer & Nodes & Activation\\
    \addlinespace
    \midrule
    \addlinespace
    \addlinespace
    Input & 4 & ---\\
    Hidden 1& 32 & $\tanh$\\
    Hidden 2& 16 & $\tanh$\\
    Output & 1 & $\exp$\\
    \addlinespace
    \addlinespace
    \bottomrule
  \end{tabular}
\end{center}

Finally, we have also tested the effect of imposing permutation symmetry on the
network by sorting the inputs, as discussed in
\cref{sec:sorting-inputs-symmetry}. The network remains unchanged, and the
symmetric version of $\psi_{DNN}$ is labeled $\psi_{SDNN}$ in all of the following.

\subsection{Optimization}

We produced the following results by optimizing $\psi_{PJ}, \psi_{DNN}$ and
$\psi_{SDNN}$ with all the same hyperparameters. We also initialized the
parameters of the two network wave functions equally for better comparison.
We used $\num{30000}$ iterations of $1000$ \gls{mc} cycles each. We did not use any
regularization in this case, and as usual we used \acrlong{is} and the
ADAM optimizer. Again, this is an example of hyperparameters, and we can
achieve good results with a range of other choices.

\Cref{fig:QD-pade-dnn-training} shows a graph of the absolute error in energy
during the course of training, for all three wave functions. To begin with,
$\psi_{PJ}$ and $\psi_{DNN}$ closely follow each other,
likely because the Pade-Jastrow factor contributes most of the improvements at
this point. Additionally, we have initialized the networks with rather small
weights, which results in the network in $\psi_{DNN}$ having a weak effect to
start. This was done to ensure stability.

While $\psi_{PJ}$ plateaus after a short amount of time, we see $\psi_{DNN}$
continues to improve beyond this point once the weights reach a critical
magnitude. In the end $\psi_{DNN}$ seems to converge at a little more than
halfway through the training, and we achieve an order of magnitude better
accuracy compared to $\psi_{PJ}$, with a corresponding reduction in the
variance.

The symmetric counterpart, $\psi_{SDNN}$, acts quite differently. Ignoring the
random spike in the beginning, we still see improvement compared to $\psi_{PJ}$,
but distinctly worse than the network without input sorting. It also appears to
have significantly more variance compared to $\psi_{DNN}$, however, the variance
estimates for the final model (\cref{tab:pade-dnn-energy-results}) disagrees. It
is possible that the variance was larger during training, and that the effect is
amplified by the logarithmic scale on the axis. Regardless, we again
conclude that imposing symmetry in this system seem to only hamper the learning.
The symmetry metric for $\psi_{DNN}$ is very close to $1$ throughout training,
so it seems the network learns the symmetry sufficiently well on its own.

\Cref{tab:pade-dnn-energy-results} shows the final energies produced from the
three wave functions after training. The results solidify the graphical
impression of~\cref{fig:QD-pade-dnn-training}, showing a clear improvement by
both networks. The result from $\psi_{DNN}$ approaches the accuracy of \gls{dmc},
a technique that can obtain exact numerical results. \textcite{Pedersen-2011}
lists $\SI{3.00000(1)}{\au}$ as the \gls{dmc} result, which has an overlapping
confidence interval with our results.

Also included is the energy obtained by using the parameters from $\psi_{DNN}$
in $\psi_{SDNN}$. Unlike for the \gls{rbm}, where this strategy barely changed the
energy, the effect was much worse for the \gls{dnn}. In this small two-body
system it seems the best strategy is to simply let the network learn the
symmetry, despite that it never learns a perfectly symmetric set of parameters.

\begin{figure}[h]
   \centering
    \resizebox{\linewidth}{!}{%
        \input{scripts/QD-pade-dnn.py.tex}
    }
    \caption[Learning progression of a neural network on quantum dots]{\label{fig:QD-pade-dnn-training}Left: Performance of the wave
function in~\cref{eq:pade-dnn} as a function of training steps. Right:
Progression of variational parameters as a function of training steps. We only
show a small selection of the total number of parameters.\citesource{writing/scripts/QD-pade-dnn.py}}
\end{figure}

\begin{table}[h]
  \centering

  \caption[Energy estimates using a neural network on quantum
  dots]{\label{tab:pade-dnn-energy-results}Energy using the neural networks
$\psi_{DNN}$ and $\psi_{SDNN}$, along with the benchmark wave function after the
same amount of optimization. $\hat\psi_{SDNN}$ shows the result of using the
parameters from $\psi_{DNN}$ with input sorting applied only after training.
Results obtained from $2^{22}$ samples and errors corrected for autocovariance by blocking. Energies in atomic units
$[\si{\au}]$. See \cref{fig:QD-pade-dnn-training} for source code reference.}
  \input{scripts/QD-pade-dnn.py.table.tex}
\end{table}

% \begin{figure}[h]
%    \centering
%     \resizebox{0.7\linewidth}{!}{%
%         \input{scripts/QD-pade-dnn.py.symmetry.tex}
%     }
%     \caption{\label{fig:QD-pade-dnn-symmetry}Permutation symmetry of
% $\psi_{DNN}$ as a function of training steps. See
% \cref{fig:QD-pade-dnn-training} for source code reference.}
% \end{figure}

\begin{figure}[h]
   \centering
    \resizebox{!}{0.35\paperheight}{%
        \input{scripts/QD-plot-parameters.py.weights-regular.tex}
    }
    \caption[Weights of a neural network trained on quantum
dots]{\label{fig:QD-pade-dnn-weights}Color map representation of the weights and
biases in the network of $\psi_{DNN}$. The left column of plots shows the weight
matrices in each layer, and the corresponding biases are in the right column.
The bottom right plot has only one color because there is only one output layer
bias. Each rectangle represents a parameter, and they have been scaled to have
the same total size. The size of the rectangles are not important. The
parameters of $\psi_{SDNN}$ are largely different, but show similar patterns.
See \cref{fig:QD-pade-dnn-training} for source code reference.}
\end{figure}

\subsection{Interpreting the Networks}

Given the apparent ability of the networks to improve upon our theoretically
motivated ansatz, it would be interesting to see if we can learn something from
the structure of the network. This is, however, notoriously difficult to do with
neural networks. In most cases the interconnections that make up the network are
treated as a black box, simply because we lack the understanding to do
anything else.

Still, we can make an attempt. \Cref{fig:QD-pade-dnn-weights} shows a color map
representation of all the weights and biases of $\psi_{DNN}$ after training.
Arguably, we should not attempt to assign physical meanings to any of the later
layers. If anything, the weight matrix of the first layer could be insightful,
as this is directly multiplied with the coordinates $\mat X$. Interestingly,
only a few columns appear to have significant values. This is very similar to the
effect seen in~\cref{fig:QD-rbm-symmetry} for the \gls{rbm}. In fact, a similar
symmetry between two of the columns is present, hinting that the network
has realized the importance of relative differences in coordinates. The
remaining columns are not as clearly zeroed out, partly due to
the lack of regularization, but also indicating that more is happening here
compared to the \gls{rbm}. While non-trivial to interpret, this points to the
networks learning, completely by themselves, some underlying physical structure
in the inputs.

\subsection{Complexity}

While the neural network has demonstrated its capacity to learn an improved
correlation function, this does come at a considerable computational cost.

The benchmark function has complexity $\mathcal{O}(N^2)$ for $N$ particles,
rooted in the double sum in~\cref{eq:qd-pade-jastrow-anzats}. Thanks to
analytic expressions for the derivatives, the otherwise computationally expensive Laplace operator
is also calculated in $\mathcal{O}(N^2)$ time.

The complexity of the neural network is a little harder to pin down, because of
the inherent flexibility in how the network can be structured.
It computes a series of matrix-vector multiplications, both during
evaluation and for its derivatives. The input layer needs to scale equal to $N$,
while the other layers could in principle be constant. This means we could say
that the network only scales linearly, $\mathcal{O}(N)$, with increasing number
of particles. Still, the \say{constant} part of the computation time is very
large, and this kind of complexity analysis is slightly unfair.

In practice we found that we needed the second layer to have more nodes than we
have inputs. We have not determined a strict relationship, but either
$\mathcal{O}(N)$ or $\mathcal{O}(N^2)$ seems reasonable for the number of nodes
in the second layer. Assuming the network shrinks from there on, the dominant
operation is a matrix-vector operation with matrix dimensions of order $(N\times
N)$ or $(N\times N^2)$, which works out to a complexity of $\mathcal{O}(N^2)$ or
$\mathcal{O}(N^3)$. In this more realistic view, the networks are not improving
upon the complexity of the benchmark, but rather the opposite is true.

The networks also depend highly on the depth (number of layers) as
well as obviously the sizes of subsequent layers. We should also note that while
asymptotic cost might be equal to the benchmark, a more precise count of the
required operations would show a substantial disparity. Furthermore, we have
found that the \glspl{nn} generally require more training iterations compared to
the benchmark, which further compounds the computational cost.

Nevertheless, a constant cost difference can always be brought down with
optimized code, more and better hardware, introducing \glspl{gpu} etc. And while
computing power may no longer increase exponentially each year, it continues to
grow. Large scale use of networks for quantum mechanics might therefore be well
within reach.

\subsection{Other Observables}

So far we have only investigated the energies produced by the different wave
functions. Other observables of interest could be the mean distance from the origin,
$\expval{r}$, the mean squared distance from the origin, $\expval{r^2}$, and the mean
inter-particle distance, $\expval{r_{12}}$. These are all quantities that should
depend on the correlation between the particles.
\Cref{tab:QD-mean-distance-metrics} shows the three observables as predicted
from the three wave functions trained previously.

\begin{table}[h]
  \centering
  \caption[Radial metrics of different wave functions on quantum dots]{Average distances predicted by the different wave functions. Results
    obtained by \gls{mci} using \gls{is} and $2^{24}$
    samples. The first row shows the corresponding values for a single particle
    in an ideal harmonic oscillator, with the values coming from the analytic
    expressions $\expval{r}=\flatfrac{\sqrt\pi}{2\sqrt\omega}$ and
    $\expval{r^2}=\omega^{-1}$. While the differences between $\psi_{PJ}$ and
    the networks are small, the inter-particle distance shows the largest
    difference. Distances in dimensionless units of $a_{ho}$.\citesource{writing/scripts/QD-other-metrics.py}}
  \input{scripts/QD-other-metrics.py.fixed-table.tex}
  \label{tab:QD-mean-distance-metrics}
\end{table}
We see first that, as one would expect, the predicted radii are larger when we
have multiple interacting particles, as opposed to the ideal non-interacting
case. Less clear is the differences between the benchmark wave function
$\psi_{PJ}$ and the networks. In general, the results
indicate that $\psi_{PJ}$ slightly underestimates the effect of the Coulomb
repulsion with smaller estimates for all three quantities. While $\expval{r}$
and $\expval{r^2}$ are only borderline significantly different from the networks, we see a
larger difference in $\expval{r_{12}}$. This might
be because $\expval{r_{12}}$ is the quantity which is most directly
connected to the strength of the interaction

The two network types seem to be equivalent, with the observed differences well
within statistical expectations.

\end{document}


\documentclass[simplex.tex]{subfiles}
% NO NEED TO INPUT PREAMBLES HERE
% packages are inherited; you can compile this on its own
%\usepackage{caption}
%\usepackage{subcaption}

\begin{document}
\subsection{Law of Large Graphs}
We note that low-rank methods can often be more easily interpreted.
By representing a low-rank matrix in terms of the latent position, where each vertex is represented as a vector in $\Re^d$ and the entries of the matrix are given by the inner products of these vectors, one can analyze and visualize the geometry of these vectors in order to interpret how each vertex is behaving in the context of the larger graph.
Now we take the CoRR dataset experiment as an example and consider the same sample of size $M=5$ based on the Desikan atlas. Our estimator $\hat{P}$ is based on the estimated latent positions $\hat{X} \in \mathbb{R}^{N\times d}$, where $N=70$ is the number of vertices and $d=11$ is the dimension selected by the Zhu and Ghodsi's method.
We color the brain using the first 5 dimensions of $\hat{X}$ as in Fig.~\ref{fig:eigenvector_brain}.
From the figures, we can see the embeddings have its own neuro-meaning, for example there is a clear distinction of the left and right hemisphere as conveyed in the second dimension. Also, the first dimension provides an average level of the entire brain. We are still exploring the interpretation other dimensions are providing.

\begin{figure}
\centering
\begin{subfigure}[t]{0.6\textwidth}
\caption{1st dimension}
\vspace*{-15pt}
\begin{center}
  \includegraphics[height=.33\linewidth]{../../figs/desikan1a.png}\hspace{-20pt}
  \includegraphics[height=.33\linewidth]{../../figs/desikan1b.png}\hspace{-20pt}
  \includegraphics[height=.33\linewidth]{../../figs/desikan1c.png}
\end{center}
\end{subfigure}\\
\vspace*{5pt}
\begin{subfigure}[t]{0.6\textwidth}
\caption{2nd dimension}
\vspace*{-15pt}
\begin{center}
  \includegraphics[height=.33\linewidth]{../../figs/desikan2a.png}\hspace{-20pt}
  \includegraphics[height=.33\linewidth]{../../figs/desikan2b.png}\hspace{-20pt}
  \includegraphics[height=.33\linewidth]{../../figs/desikan2c.png}
\end{center}
\end{subfigure}\\
\vspace*{5pt}
\begin{subfigure}[t]{0.6\textwidth}
\caption{3rd dimension}
\vspace*{-15pt}
\begin{center}
  \includegraphics[height=.33\linewidth]{../../figs/desikan3a.png}\hspace{-20pt}
  \includegraphics[height=.33\linewidth]{../../figs/desikan3b.png}\hspace{-20pt}
  \includegraphics[height=.33\linewidth]{../../figs/desikan3c.png}
\end{center}
\end{subfigure}\\
\vspace*{5pt}
\begin{subfigure}[t]{0.6\textwidth}
\caption{4th dimension}
\vspace*{-15pt}
\begin{center}
  \includegraphics[height=.33\linewidth]{../../figs/desikan4a.png}\hspace{-20pt}
  \includegraphics[height=.33\linewidth]{../../figs/desikan4b.png}\hspace{-20pt}
  \includegraphics[height=.33\linewidth]{../../figs/desikan4c.png}
\end{center}
\end{subfigure}\\
\vspace*{5pt}
\begin{subfigure}[t]{0.6\textwidth}
\caption{5th dimension}
\vspace*{-15pt}
\begin{center}
  \includegraphics[height=.33\linewidth]{../../figs/desikan5a.png}\hspace{-20pt}
  \includegraphics[height=.33\linewidth]{../../figs/desikan5b.png}\hspace{-20pt}
  \includegraphics[height=.33\linewidth]{../../figs/desikan5c.png}
\end{center}
\end{subfigure}\\
\caption{\textbf{Brain plots colored by the first 5 dimensions of $\hat{X}$ for the Desikan atlas respectively.} We plot the brain using the first 5 dimension of $\hat{X}$ . From the figures, we can see the embeddings have its own neuro-meaning, for example there is a clear distinction of the left and right hemisphere as conveyed in the second dimension. Also, the first dimension provides an average level of the entire brain.}
\label{fig:eigenvector_brain}
\end{figure}

%
\clearpage
\end{document}


In this section we present a slightly more formal treatment of inference systems. Moreover, we give a clearer syntax for expressing operators and rules. The formalism which we will use, which is also a programming language, is \textit{Meta-Casanova}. Meta-Casanova falls under the broader category of \textit{logic programming languages}, but given its structure and some of its applications it can also be considered a \textit{meta-compiler}.

\subsection{Syntax for symbol definitions}
Defining symbols in Meta-Casanova requires drawing a distinction: some symbols are considered \textit{data}, that is they are just a way to structure information, while other symbols are considered \textit{functions}, that is they are a way to define transformations from propositions to propositions.

\paragraph{Data symbols}
Data symbols are defined with the following syntax:

\begin{lstlisting}[mathescape=true]
Data [$\alpha_1$] "$\alpha_2$" [$\alpha_3$] Priority $\alpha_4$ Type $\alpha_5$
\end{lstlisting}

The $\alpha_i$'s are the arguments of the symbols:
\begin{description}
\item[$\alpha_1$] are the arguments that come left of the symbol
\item[$\alpha_2$] is the name of the symbol
\item[$\alpha_3$] are the arguments that come right of the symbol
\item[$\alpha_4$] is the priority of the symbol
\item[$\alpha_5$] is the type of the symbol
\end{description}

Examples of such a definition could be:

\begin{lstlisting}
Data [] "TRUE" [] Priority 10000 Type Expr
Data [Expr] "|" [Expr] Priority 10 Type Expr
\end{lstlisting}

the above come from the boolean expressions sample, which will be discussed in depth in a further chapter. The first line defines the \texttt{TRUE} symbols, which has no arguments left or right (it is just a value), has a very high priority (which does not matter given that this symbol has no parameters), and which is of type \texttt{Value}. The second line defines an \textit{or}-style operator that takes an \texttt{Expr} to the left, an \texttt{Expr} to the right, has a priority of 10, and is an expression itself.

With the definitions above we can write a proposition such as \texttt{TRUE | TRUE}, which would be perfectly valid. Associativity is by default to the right, thus \texttt{TRUE | TRUE | TRUE} is equivalent to \texttt{TRUE | (TRUE | TRUE)}. We can modify the default associativity by adding a last parameter of:

\begin{lstlisting}
Data [Expr] "|" [Expr] Priority 10 Type Expr Associativity Left
\end{lstlisting}

We might even go as fast as wanting to re-define \texttt{TRUE} to a new type, because indeed we could say that \texttt{TRUE} is more a value than an expression:

\begin{lstlisting}
Data [] "TRUE" [] Priority 10000 Type Value
\end{lstlisting}

With the above definition it becomes impossible to write expressions such as \texttt{TRUE | TRUE}: the compiler will refuse this as invalid code, because \texttt{|} expects arguments of type \texttt{Expr} but it is instead given arguments of type \texttt{Value}. We can say:

\begin{lstlisting}
Value is Expr
\end{lstlisting}

to inform the compiler that whenever an \texttt{Expr} is expected, then we can also give a \texttt{Value} without issues. This can be considered a form of \textit{inheritance} such as that found and used in object-oriented languages.

\paragraph{Function symbols}
Function symbols are defined with the following syntax:

\begin{lstlisting}[mathescape=true]
Func [$\alpha_1$] "$\alpha_2$" [$\alpha_3$] Priority $\alpha_4$ Type $\alpha_5$ => $\alpha_6$
\end{lstlisting}

These symbols are defined almost identically to data symbols, with the difference that they also specify what is the type ($\alpha_6$) of the data they will return as a result of transformation.

An example of such a definition could be a function that, given a boolean expression, return the corresponding value:

\begin{lstlisting}
Func [] "eval" [Expr] Priority 1 Type Expr => Value
\end{lstlisting}

It will now be possible to write \texttt{eval (TRUE | TRUE)}. Of course so far we have only defined the aacceptable \textit{shape} of terms, but we have not yet said anything about how computations happen.

\subsection{Syntax and semantics of rules}
Rules define how propositions that begin with a \textit{functions} process the \textit{data} that they have as parameters. The rules of an inference system are syntactically quite simple. The simplicity comes from the fact that a rule syntax is only made up of two operators and two additional concepts. The two operators are:

\begin{itemize}
\item the \textbf{horizontal bar} \texttt{------------}
\item the \textbf{arrow} \texttt{=>}
\end{itemize}

Both horizontal bar and implication arrow can be read out loud as ``therefore'', as they both capture a form of implication. The horizontal bar separates the main implication from its premises, and thus operates at a higher level of precedence and abstraction with respect to the arrow. A single inference rule will consist of a series of \textbf{premises} which are separated from the \textbf{main proposition} by the horizontal bar:

\begin{lstlisting}
PREMISE 1
PREMISE 2
...
PREMISE N
-----------------
MAIN PROPOSITION
\end{lstlisting}

Both individual premises and the main proposition are defined as the \textbf{input} sequence of symbols, separated from the \textbf{output sequence of symbols} by the implication arrow:

\begin{lstlisting}
INPUT SYMBOLS => OUTPUT SYMBOLS
\end{lstlisting}

A pattern consists of a series of keywords, operators, and variables, recursively \textbf{applied to each other}. Applied in this case is meant in the sense of function application, as in \texttt{f(3)} where \texttt{f} is applied to three, or \texttt{3+x}, where \texttt{+} is applied to symbol \texttt{3} and variable \texttt{x}.

For example, with reference to the example above, we might want to define a rule that has no premises as:

\begin{lstlisting}
------------------------
eval (TRUE | x) => TRUE
\end{lstlisting}

The rule above tells us that whenever we encounter \texttt{eval (TRUE | x)} for some \texttt{x} which is left unspecified, then we can directly return \texttt{TRUE}.

A rule with premises, which therefore shall require further nested computation, could be:

\begin{lstlisting}
eval x => TRUE
------------------------
eval (FALSE | x) => TRUE
\end{lstlisting}

The rule above tells us that whenever we encounter \texttt{eval (FALSE | x)} for some \texttt{x} which is left unspecified, then we need to evaluate premise \texttt{eval x => TRUE}; this premise begins by evaluating \texttt{eval x}, and if it returns \texttt{TRUE} then we proceed to return \texttt{TRUE} as the final result. If the premise returns something else than \texttt{TRUE}, then execution of the rule is interrupted (it is indeed a failure) and no result is returned. To ensure termination we could add another complementary rule:

\begin{lstlisting}
eval x => FALSE
------------------------
eval (FALSE | x) => FALSE
\end{lstlisting}

Of course we could merge the two rules above into a single rule so that the result of the premise is directly propagated:

\begin{lstlisting}
eval x => y
----------------------
eval (FALSE | x) => y
\end{lstlisting}

\paragraph{Main input}
The overall input sequence of symbols of the inference system, which can be considered as the \textit{main question} to answer or the \textit{main program} to execute, is just a sequence of symbols without variables such as \texttt{eval TRUE} or \texttt{eval (TRUE | TRUE)}. Meta-Casanova will return a list of the sequences of symbols that are obtained as a result of applying the rules of the system to the input; the list will either be:

\begin{itemize}
\item empty if no rule could be applied at some point; this means that the input is \textit{not supported} by our rules
\item one or more results if the input can be processed without issues
\item undefined if the program keeps looping forever
\end{itemize}

Note that the possiblity to return multiple results is very important to support algorithms such as \textit{backtracking} or similar searches, where multiple possible branches of derivation are followed simultaneously to see how many of them will reach an answer. This specific aspect means that the language behaviour can be considered as a sort of path-finder


\chapter{Use of non linear optimization in MMVII}

This chapter is targeted for programmer that will increase MMVII, it mainly describe classes
interfaces in headers and a detailled example of using this classes in a cpp file.
 It's not usefull for advanced users; on the other hand,for programmer who will maintain the core of MMVII,
more detailled documentation will have to be written.

At this step, the chapter may no theoreticall presentation, it assume that reader is familliar
with least square, linearization, Gauss-Newton, Schurr complements \dots

%---------------------------------------------
%---------------------------------------------
%---------------------------------------------

\section{Introduction}

%---------------------------------------------

\subsection{Quick formalization}

This chapter present the classes used in MMVII for non linear optimisation using some Gauss-Newton methods.
By non linear optimisation, we means minimize $F(X)$ with :

\begin{equation}
      F(X) = \sum_{j=1,M} w_j f_j(X)^2  \;  X \in \RR^n  \label{EqNLOInit}
\end{equation}

The assumpution on the problem are classical :

\begin{itemize}
   \item we have an initial guess $X_0$ on $X_{min}$  , this guess is sufficiently
         good for the local minimal we will (hopefully) find to be a global minimum;

   \item the function $f_j$ are differentiable (some weaker hypothesi are also possible)

\end{itemize}

The service offered by MMVII are typically those a Gauss-Newton iterations :
use current estimation $\tilde X$ of $X_min$ for linearizing the $f_j$ and
solve equation $\ref{EqNLOInit}$ by least square methods.

%---------------------------------------------

\subsection{2d-Triangularton problem in general}

We will illustrate the presentation by a detailled example that is used for testing correctness of the
implementation. Also relatively basic, this example should be sufficient for a first
understanding and utilisation of the MMVII classes. The $2d$-triangulation problem we treat
here is the following :

\begin{itemize}
    \item we have a set of points $P_1 \dots P_n$ , each $P_k$ belongs to $\RR^2$;

    \item we ignorate the exact values of $P_k$ but we have an initial estimation  $P^0_k$
          which is not too bad, whatever it means;

    \item for a certain number of pair $P_i,P_j$ we have a measurement of the distance $d_{ij}$ between
          $P_i$ and $P_j$,
\end{itemize}

The triangulation problem  consist to use the measurement of distances for recovering the unknown
value on $P_i$.  Let $M$ be the  number of pair for which we have measurement
Let $X \in \RR^{2n} = \{P_1 \dots P_n\}$ be the unkown vector,
we typically want to find $X$ that minmize $D$ defined by :

\begin{equation}
      D(X) = \sum_{j=1,M} (d_{ij} -d(P_i-P_j))^2  \label{EqConsDist}
\end{equation}

Of course as the fonction $D$ is invariant to any isometry of the plane, the minimization
of $D$ would be an ill-posed problem, as for any set of points, its image by a global rotation
will give the same value. To ensure uniqueness of the solution  will force the value of
a certain number of coordinates.

%---------------------------------------------

\subsection{2d-Triangularion in this example}

As we want the example to be as simple as possible, and also we want to use it for checking
the correctness, the data will be different from a real one , the figure~\ref{fig:NetFull} illustrates
this network:

\begin{itemize}
    \item the "real" value of points will be positionned on a regular grid, typically we
           will have $(2*N+1)^2$ points, each point having integers corrdinate
           $(x,y) \in [-N,N]^2$

    \item we will have measure between pair of point that  are $8$-neighoor i.e
          $max(|x-x'|,|y-y'|)\leq 1 $ , this is sufficient for the solution to be unique
          up a rotation ;

    \item in real life  $d_{ij}$ would be noisy, but here in this simulation example we
          use the exact value because, to check correctness of the library, we want to check
          that we are able to recover the exact values of coordinates.
\end{itemize}

Regarding the arbirtrary constraint we will keep it as simple as possible, so
considering the two points of the grid  $A=(0,0)$ and $B=(0,1)$ , we will add the following
constraint to the minimization :

\begin{equation}
      x_A=0  \;  y_A=0  \;  x_B=0 \label{Eq:FixVarAB}
\end{equation}

The two first constraint freeze the solution in translation, while the last one freeze the rotation.

\begin{figure}
\centering
\includegraphics[width=12cm]{Methods/T90-NetFull.JPG}\caption{Network, case w/o schurr}\label{fig:NetFull}
\end{figure}

%---------------------------------------------

\subsection{Use of schurr complement}

Schurr complement is an efficient way to treat the case where there is a subset of variable that
occurs in \ref{EqNLOInit}, but for which we are not specially interested in exact values ,
 however but we want to take them into account rigourously in
the computation of the others. Once we know
\emph{all} the equations where such  subset of variable is involved, schurr
complement offer a way to eliminate them without altering the value of the minimum.

A typicall case is in bundle adjsutment, when we are interested only to the value
of camera parameters and not by the value of $3d$ points involved by the projection
equation. In this case, suppose we will typically  have $1000$ camera and $1000000$ points,
doing schurr elimination can reduce the system from $3000000$ unkwnon to $6000$.

In our toy example, the schurr complement will be purely artificiall. When the
{\tt WithSchurr}  option is activated, the network is modified in the following
way , the figure~\ref{fig:NetSchurr} illustrates

\begin{itemize}
    \item all the points on the line $x=1$ are considered as temporay variable,
          i.e variable that we want to eliminate, we will not  know their value at the
          end of the computation;

    \item there will be no conexion between points of line $x=1$ (for example $(1,2)$
          and $(1,3)$ that were connected in previous case are non longer
\end{itemize}

\begin{figure}
\centering
\includegraphics[width=12cm]{Methods/T90-NetSchurr.JPG}\caption{Network, case with schurr}\label{fig:NetSchurr}
\end{figure}

%---------------------------------------------

\subsection{Files of MMVII involved}

We make a quick enumeration of files of MMVII that are to access for understanding :

\begin{itemize}
    \item {\tt include/MMVII\_SysSurR.h} contains the declaration of the class relatives to
          non linear optimisation, from the user point of view the main class
          is {\tt  cResolSysNonLinear};

    \item {\tt src/Bench/BenchResolSysNonLinear.cpp} contains the code for the $2d$-triangulation
          example;

    \item {\tt src/SymbDerGen/Formulas\_Geom2D.h } and {\tt src/SymbDerGen/GenerateCodes.cpp }
          contains the code for generating the automatic differentiation

    \item {\tt src/GeneratedCodes/CodeGen\_cDist2DConsVal.cpp } contains the code for
          generating the automatic differentiation , also it is theoretically not necessary
          to see it for using it, it's no harm to be curious and not use it as a black box;
          by the way in real case use, it will not be necessary to look at the code that you
          will generate.
\end{itemize}

%---------------------------------------------
%---------------------------------------------
%---------------------------------------------

\section{Computing functions and their derivatives}

%---------------------------------------------
\subsection{Motivation}

For Gauss Newton iteration, in equation \ref{EqNLOInit}, we will need to compute not only
the $f_j$ but also all its partial derivative $\frac{\partial f_j}{\partial x_i}$
relatively to all variable $x_i$ used in $f_j$.  There are many way to do it,
and to be honest  all have  pro and cons :

\begin{itemize}
    \item numerical derivative are the most simple, but they are slow, and can be unacurrate
          when the "small" value are not correctly choosen;

    \item hand crafted derivative are accurate and can be fast, however they can be complicated
          to write for real case formula (like arrise in bundled adjustement) and can be
          a nightmare to maintain when a "small" change occurs in the formulas;

    \item jet derivative, like used it Ceres, are almost as simple to use as numericall, once
          you have the library,  and much faster and accurate as numericall ones; they are much
          easier to use and maintain as  hand crafted ones, but can be also slower than
          handcrafted;

    \item symbolic derivative and code generation, can be a bit more complicated to use
          than jet, especially the first time; however after  this quick learning step
          they are as easy as jets to use,  easy to maintain and, in some case, much
          faster.
\end{itemize}

The solution implemenented in MMVII in based on symbolic derivative with code generation.
This choice was made because it is believed that programmer that will goes into the
core of MMVII have rather high requirement in code efficiency and are ready to pay a
small learning steps.

%---------------------------------------------
%\section{Generating the code}

          %  - - - - - - - - - - - - - - - - - - -

\subsection{class for specifying the formulas}

In this  example, we have only one formula we need to compute and derivate, it's
the formula of equation~\ref{EqConsDist}.  First introduce a bit of MMVII's jargon  :

\begin{itemize}
   \item in this formula we have two different kind of variables the $P_i$ and $P_j$
         and the $d_{ij}$;

   \item for the $P_i$ themselves we have no direct observation , we only have initial value,
         and our target is precisely to compute their values; they will be named {\tt unkowns};

   \item for the $d_{ij}$ we know their values (it could be with some uncertaincy, but that'
         another story) and dont try to compute it, they will be name {\tt observations}.
\end{itemize}

The class {\tt cDist2DConservation} in file  {\tt src/SymbDerGen/Formulas\_Geom2D.h } contains
all the information that are required for generating code. It must define $5$ members :

\begin{itemize}
   \item a contructor, that does nothing for such a basic example

   \item a method {\tt VNamesUnknowns} that return the vector of names of unkown ,
         here we have $2$ points and $2$ coordinates $x,y$ for each points so the
         vector has a length of $4$;  the names will be used for code generation;

   \item a method {\tt VNamesObs} that return the vector of names of observations ,
         here we have a single observation which is the targeted distance between
         $P_1$ and $P_2$;  the names will be used for code generation;

   \item a method {\tt FormulaName}  that return the name of the formula itself,
         the method will be used for the name of class and files containing the generated
          code;

   \item a method {\tt formula}  this is the core of the class, this method
         take as input a vector of unknown and a vector of observation and return
         a vector that is the result of the  computation  (typically a residual
         when used in context of gauss newton);  the type of {\tt tUk} of the
         i/o vectors and of temporary variable, is not very important at this step,
         it's suffice to know that it is a type on which most current mathematicall
         operation are defined (and can be completed when missing), they represent
         mathematicall formula as symbolic tree; see \RefFantome for detailled
         explanation on the type {\tt cFormula<Type>} defined in

\end{itemize}

Regarding the $3$ methods that  return names, as they are used as \CPP identifier
its important that they contains only valid character (alpha numeric and {\tt\_} ,
no {\tt + - ...}),  also it's important that  inside a vector they are unique.
Apart of that, their exact name is unimportant, but giving names semantically
meaningfull is usefull if a human want to read the generated code (something we
generally dont do, but something we will do here).

\begin{equation}
      D(X) = \sum_{j=1,M} (\frac{d(P_i-P_j)}{d_{ij}} - 1)^2  \label{EqConsDistHom}
\end{equation}

Regarding the method  {\tt formula}, it's here a direct implementation
of residual used in  equation~\ref{EqConsDistHom}, which
is non dimensionnal variant of equation~\ref{EqConsDist}. Some comment :

\begin{itemize}
   \item we make an intensive use of the {\tt auto} type specification which
         is convenient here;

   \item the less intuitive part is probably {\tt  CreateCste(1.0,x1);},  in fact
         when we need to create a constant  we must indicate the type of symbolic
         formula to the constant is create with adequate type, the {\tt x1} parameter
         is just used to indicate the type in the template function {\tt CreateCste},
         putting {\tt x2}, {\tt y1} or any other would have exactly the same effect;

   \item the function return a vector of formula, which here is of size $1$, and not
         single value, because in general, when we want to compute several values (like
         $x$ and $y$ residual in BA)  its more efficient and convenient to group
         a multiple residual in a single formula that generating different formulas;

\end{itemize}

As a slightly more complex example, the reader can investigate example {\tt cRatioDist2DConservation}
in the same file. This example correspond to the case what we want to preserve is not the distances
but the angles or ratio of distance. For each triangle $P1,P2,P3$, we have the initial
distance $D_{i,j}$ as observation and  we write :

\begin{equation}
      \frac{d(P_i,P_j)}{D_{ij}} - \frac{d(P_i,P_k)}{D_{ik}} = 0 \label{EqRatioDist}
\end{equation}

In this cas we have $3$ equation and the function has $3$  values corresponding to the
$3$ possible combination of  equation~\ref{EqRatioDist}.

          %  - - - - - - - - - - - - - - - - - - -

\subsection{Generating the code}

Once the class {\tt cDist2DConservation} has been created, the things to do for generating the
code is quite simple :

\begin{itemize}
    \item   add in file  {\tt GenerateCodes.cpp } a call to   the method {\tt GenCodesFormula}
            with an object of type {\tt cDist2DConservation}, note that we call it twice,
            the second boolean parameter indicating if we want to  compute the derivate;
            in fact sometime we will be interested only by the function and will not want
            to pay the price for the derivate;

    \item   compile with {\tt make}, execute a call to {\tt MMVII} command {\tt GenCodeSymDer},
            and compile again.
\end{itemize}

Also it's generally not necessary, we invite here the curious reader to give a look
at generated code.  All the such codes are located in {\tt src/GeneratedCodes/} ,
and here in the files  {\tt CodeGen\_cDist2DConsVal.cpp}, w/o derivative, and
{\tt CodeGen\_cDist2DConsVDer.cpp}, with derivatives, the declaration of the classes
are in corresponding header file ({\tt CodeGen\_cDist2DConsVal.h} \dots).
Note that the name used for file ans classes come from the result of {\tt FormulaName}.

Lets make a quick comment on {\tt CodeGen\_cDist2DConsVDer.cpp} :

\begin{itemize}
  \item  the computation is made in a loop {\tt for (size\_t aK=0;....)}, this
         allow a paralelization of the code when high performance computation is
         required;

  \item  the name used for unkowns and observation can be recoginzed as local
         variable at the begining of the loop;

  \item  after the code is a very monotonous code with basic instructions as
         {\tt Fx = Fy op Fz;}  or {\tt Fx=Op(Fy);}  ,  this is not the way
         human would write code ...  however it is not made to be read and you may
         think of it as "high level assembleur code";

  \item  by the way the code is relatively optmized to avoid multiple redundant computation,
         for example you can see that formula {\tt F8}  that represent {\tt y1-y2} is computed
         once and used three time;

  \item  the result of one iteration contain $5$ value here, because we have the value of the function
         itself and the value of the derivative relative to the $4$ variable ($x$ and $y$ of
         $P_1$ and $P_2$) ;

  \item  there is a high level interface for extracting independantly value and derivatives,
         but we will not need it here, as everything will be encapsulated in the main
         class {\tt cResolSysNonLinear};

  \item  give a look at file {\tt CodeGen\_cRatioDist2DConsVDer.cpp}, in this case we have $21$
         values ; $21 = 3 * (1 + 3*2)$ , the ratio return a vector of $3$ values, and
         for each value we compute the value itself and its derivatives relative to the $6$
         variable ($x,y$ for $3$ points), again there is a high level interface for extracting
         these values.
\end{itemize}

          %  - - - - - - - - - - - - - - - - - - -

\subsection{Creating the object}

\label{CreateCalc}

For computig the value and derivative of {\tt cDist2DConservation} in our \CPP program
 a possible, and classical way, is to include the file declaring the class
(i.e. {\tt CodeGen\_cDist2DConsVal.h}) and explicitely create a {\tt cDist2DConsVDer}.

However the optimizer is done to work with any object deriving from the abstract mother
class of all object resulting from code generation :  {\tt cCompiledCalculator<double>};
so the optimizer only manipulate pointers on such class and the exact class is not used.
There is a function for creating an object from its name, and we create an allocator
from this name with the functio {\tt  EqConsDist(bool WithDerive,int aSzBuf)}.

The definition in {\tt GenerateCodes.cpp }  is pretty basic :

\tiny
\begin{lstlisting}
cCalculator<double> * EqConsDist(bool WithDerive,int aSzBuf)
{
    return cName2Calc<double>::CalcFromName(NameFormula(cDist2DConservation(),WithDerive),aSzBuf);
}
\end{lstlisting}
\normalsize

And the declaration in a header file (here {\tt MMVII\_PhgrDist.h} ) :

\tiny
\begin{lstlisting}
NS\_SymbolicDerivative::cCalculator<double> * EqConsDist(bool WithDerive,int aSzBuf);
\end{lstlisting}
\normalsize

Now from the user side, the only things we need to do is calling  {\tt EqConsDist}. This
correspond in file {\tt BenchResolSysNonLinear.cpp}  to line :

\tiny
\begin{lstlisting}
     mCalcD =  EqConsDist(true,1);
\end{lstlisting}
\normalsize

%---------------------------------------------
%---------------------------------------------
%---------------------------------------------

\section{Non linear system}

\subsection{Introduction}

The class for solving  non linear system is the template class
{\tt  cResolSysNonLinear<Type>}.  This class furnish an easy interface
for computing fonctions and their derivatives (via object resulting from code generation )
ans weighted least  square in the aim of solving  non linear system.

The template of the class refers to the way linearized equation are store and solved.
Probably {\tt tREAL8=double}  would be a good default values, while {\tt tREAL16}
could be used when high accuracy is required.

The main operations that can be done with such solver are :

\begin{itemize}
   \item create a solver with an initial value of the unknown and parameter for
          specifying the adjoint least square solver;

   \item add directly  an equation on subset of unkowns \emph{w/o} temporary unknowns  ;

   \item add  an equation with a subset of unkowns \emph{with} temporary unknowns  to
         a structure that accumulate them;  then  add this structure to the solver;

   \item add equation fixing a given variable;

   \item acces to current solution;

   \item compute the next current solution and reinitialisze the solver.

\end{itemize}

          %  - - - - - - - - - - - - - - - - - - -

\subsection{Least square solving and constructor}

In {\tt BenchResolSysNonLinear.cpp} the call to these constructors can be found
after tags {\tt  BASIC:CONSTRUCTOR} and {\tt LEASTSQ:CONSTRUCTOR}.
The dense vector is created from a standard vector.

In {\tt MMVII} the core calculation of matrix algebra is realized
by eigen library. The services offered by {\tt MMVII} is essentially
an interface for storing the data and for a more homegeneous integration
in {\tt MMVII} "philosophy".

This said, in {\tt MMVII} there is  different way to handle least square system, each
one correspond to a value of enumeration {\bf \tt eModeSSR} :

\begin{enumerate}

    \item{\bf \tt eSSR\_LsqDense :}
          uses dense matrix for storing the normal matrix; for system not so spare,
          or for small system, this is most efficient  way; typically if you have
          (many) thousands of limited size system, this is the mode you must use; the solving
          is made by calling \emph{"ldlt"} method of eigen (i.e. Robust Cholesky
          decomposition of a matrix with pivoting); this classes use schur-complement
          for handling temporay unknown;

    \item {\bf \tt eSSR\_LsqNormSparse :}
          uses sparse matrix for storing the normal matrix; this is probably the more
          general way and the more efficient in time and storage for most case in
          photogrammetry involving many pose estimation, the
          draw-back is that using normal equation increase the conditionning of the
          system, the potential problem increasing with number of unknown;
          solving is made calling  {\tt SimplicialCholesky} decomposition of eigen;

    \item {\bf \tt eSSR\_LsqSparseGC :}
          uses sparse representation that memorize all the individual obsertion;
          \emph{do  not} use normal equation, the solving is made using
          {\tt LeastSquaresConjugateGradient} decomposition of eigen;
          \emph{do  not} use Schurr-complement for temporary unknown , they
          are processed like other unknowns; according to eigen documentation
          this method is the more robust for poorly conditionned system;
          compared to {\bf \tt eSSR\_LsqNormSparse :}, for big
          sparse system with a high proportion of temporary unknonws,
          it's certainly less memory  efficient and it's probaly
          less CPU-efficient (but intensive test remain to be done);

\end{enumerate}

The basic constructor take as parameter an enum value for specifying
the least square solver and an initial value :

\tiny
\begin{lstlisting}
       cResolSysNonLinear(eModeSSR,const tDVect & aInitSol);
\end{lstlisting}
\normalsize

It is also possible to create a solver with  an explicit least square solver.
This is usefull especially with {\bf \tt eSSR\_LsqNormSparse} because the memory
allocation (still in construction) is more complex and may require more parametrisation
from the user.

          %  - - - - - - - - - - - - - - - - - - -

\subsection{Adding a basic equation}

The tag  {\tt  BASIC:CALC} in {\tt BenchResolSysNonLinear.cpp} contain an example of such use.
The method for adding an observation is named {\tt CalcAndAddObs} :

\tiny
\begin{lstlisting}
    void   CalcAndAddObs(tCalc * aCalc,const tVectInd & aVI,const tStdVect& aVObs,const tResidualW & aWeighter= tResidualW());
\end{lstlisting}
\normalsize

The $4$ parameters are :

\begin{itemize}
   \item {\tt aCalc} is a calculator as described in~\ref{CreateCalc};

   \item {\tt aVI} is a std::vector of  int that contains the index of unknowns used;

   \item {\tt aVObs} is a std::vector of  {\tt double} that contains the observation;

   \item {\tt aWeighter} is an object used for computing the weight of the observation,
         the default value  associate a constant weight $1$, we will not discuss more
         this parameter at this step.

\end{itemize}

What is done is what can be expected :

\begin{enumerate}
   \item  use  vector {\tt aVI} and {\tt aVObs}  to fill the parameters
          of the functor {\tt aCalc}; for {\tt VI} the index are used read the
          values of the current unknown

   \item  execute the computation of {\tt aCalc} , that must  have been created
          with the {\tt WithDer} option at {\tt true};

   \item  use the result of differentiation and the weighting computed by {\tt aWeighter}
          to add a linearized equation in the least square system.
\end{enumerate}

          %  - - - - - - - - - - - - - - - - - - -

\subsection{Adding with schurr complement}

The tag  {\tt  SCHURR:CALC} in {\tt BenchResolSysNonLinear.cpp} contain an example of such use.
Adding equation with temporary variables, is slightly more complex, as the elimination
can only be done once we have all the equations involving a given subset of unknowns.
So the computation is done in $2$ steps : (1) accumulation in a structure
with {\tt AddEq2Subst}, (2) using this structure
for adding the equation on unknowns after having done the elimination of temporary unknowns with
{\tt  AddObsWithTmpUK}

The structure is {\tt cSetIORSNL\_SameTmp<Type>}, the method for accumulating
equation has the signature :

\tiny
\begin{lstlisting}
    void  AddEq2Subst (tSetIO\_ST & aSetIO,tCalc *,const tVectInd &,const tStdVect& aVTmp,
                       const tStdVect& aVObs,const tResidualW & aWeighter= tResidualW());
\end{lstlisting}
\normalsize

In this method {\tt aVObs} and {\tt aWeighter} are identic to the equivalent in
{\tt CalcAndAddObs}. For the others :

\begin{enumerate}
   \item the first one {\tt cSetIORSNL\_SameTmp<Type>}, is the accumulating structure;

   \item  the vector {\tt aVI} contains the  unknown and temporary unkown,
          as temporay unknowns have no numbering they are declared with
          conventionnal value {\tt -1},  standard unknown are processed as before;
         see {\tt aVIndMixt} after tag {\tt SCHURR:CALC}

   \item  the vector {\tt aVTmp} contains the  values of temporary unknowns;
\end{enumerate}

Once the equation have been accumulated, it is sufficient to call {\tt AddObsWithTmpUK}
with the structure as parameter.

          %  - - - - - - - - - - - - - - - - - - -

\subsection{Weighted equation fixing a variable}

The tag  {\tt  EQ:FIXVAR} in {\tt BenchResolSysNonLinear.cpp} contain an example .

It currently happen that the solution we compute is undetermined
up to  certain transformation.  If we do nothing, the least square
system will be not inversible and this will create problems.
A current way  to overcome this difficulty is to fix a set
of arbitrary variables.  In our case, as said in~\ref{Eq:FixVarAB},
we fix arbirtrarily $x_A,y_A,x_B$.

The method {\tt AddEqFixVar} can be used for that, it takes $3$ parameters :
the number $k$ of the var, the value $V$ we want to assign, and the weight $w$
of the equation.  It simply add the following term  to the minimization :

\begin{equation}
      w (X_k -V)^2
\end{equation}

The variant {\tt AddEqFixCurVar} fix the value of $x_k$ to its current value.

          %  - - - - - - - - - - - - - - - - - - -

\subsection{Access to current solution}

There is two methods for accessing to the current solution of the system :

\begin{itemize}
       \item {\tt CurGlobSol()} : return globally the current solution (as a dense vector);

       \item {\tt CurSol(int k)} : return  the current value of $x_k$;

\end{itemize}

          %  - - - - - - - - - - - - - - - - - - -

\subsection{Compute next iteration}

Once we have accumulated all the observation at a given step, what we classically
want to do is to :

\begin{itemize}
       \item use these observation  to have a better estimation of the solution by solving
             the least square system;

       \item supress  these observations of the system (reset it) because, due to linearisation,
             they were approximation, and we expect now to have a better approximation;

       \item use the computed solution in the next iteration as estimation for the linearization.
\end{itemize}

This is done by the method {\tt SolveUpdateReset() ;}.